data/pdfs/Life 3.0-Being Human in the Age of Artificial Intelligence.pdf

This	Is	a	Borzoi	Book	Published	by	Alfred	A.	Knopf
Copyright	©	2017	by	Max	Tegmark
All	rights	reserved.	Published	in	the	United	States	by	Alfred	A.	Knopf,	a	division	of	Penguin	Random
House	LLC,	New	York,	and	distributed	in	Canada	by	Random	House	of	Canada,	a	division	of	Penguin
Random	House	Canada	Limited,	Toronto.
www.aaknopf.com
Knopf,	Borzoi	Books	and	the	colophon	are	registered	trademarks	of	Penguin	Random	House	LLC.
Library	of	Congress	Cataloging-in-Publication	Data
Names:	Tegmark,	Max,	author.
Title:	Life	3.0	:	being	human	in	the	age	of	artificial	intelligence	/	by	Max	Tegmark.
Other	titles:	Life	three	point	zero
Description:	New	York	:	Alfred	A.	Knopf,	2017.	|	“This	is	a	Borzoi	Book	published	by	Alfred	A.	Knopf.”	|
Includes	bibliographical	references	and	index.
Identifiers:	LCCN	2017006248	(print)	|	LCCN	2017022912	(ebook)	|	ISBN	9781101946596	(hardcover)	|
ISBN	9781101946602	(ebook)
Subjects:	LCSH:	Artificial	intelligence—Philosophy.	|	Artificial	intelligence—Social	aspects.	|	Automation

—Social	aspects.	|	Artificial	intelligence—Moral	and	ethical	aspects.	|	Automation—Moral	and	ethical
aspects.	|	Artificial	intelligence—Philosophy.	|	Technological	forecasting.	|	BISAC:	TECHNOLOGY	&
ENGINEERING	/	Robotics.	|	SCIENCE	/	Experiments	&	Projects.	|	TECHNOLOGY	&	ENGINEERING	/
Inventions.
Classification:	LCC	Q334.7	(ebook)	|	LCC	Q334.7	.T44	2017	(print)	|	DDC	006.301—dc23
LC	record	available	at	
https://lccn.loc.gov/2017006248
Ebook	ISBN 9781101946602

Cover	art	by	Suvadip	Das;	(man)	based	on	Netfalls	Remy	Musser/Shutterstock
Cover	design	by	John	Vorhees
v4.1
ep

Contents
Cover
Also	by	Max	Tegmark
Title	Page
Copyright
Dedication
Acknowledgments
Prelude:	
The	Tale	of	the	Omega	Team
1 
Welcome	to	the	Most	Important	Conversation	of	Our	Time
A	Brief	History	of	Complexity
The	Three	Stages	of	Life
Controversies
Misconceptions
The	Road	Ahead
2 
Matter	Turns	Intelligent
What	Is	Intelligence?
What	Is	Memory?
What	Is	Computation?
What	Is	Learning?
3 
The	Near	Future:	Breakthroughs,	Bugs,	Laws,	Weapons	and	Jobs
Breakthroughs
Bugs	vs.	Robust	AI
Laws
Weapons
Jobs	and	Wages
Human-Level	Intelligence?

4 
Intelligence	Explosion?
Totalitarianism
Prometheus	Takes	Over	the	World
Slow	Takeoff	and	Multipolar	Scenarios
Cyborgs	and	Uploads
What	Will	Actually	Happen?
5 
Aftermath:	The	Next	10,000	Years
Libertarian	Utopia
Benevolent	Dictator
Egalitarian	Utopia
Gatekeeper
Protector	God
Enslaved	God
Conquerors
Descendants
Zookeeper
1984
Reversion
Self-Destruction
What	Do	
You
	Want?
6 
Our	Cosmic	Endowment:	The	Next	Billion	Years	and	Beyond
Making	the	Most	of	Your	Resources
Gaining	Resources	Through	Cosmic	Settlement
Cosmic	Hierarchies
Outlook
7 
Goals
Physics:	The	Origin	of	Goals
Biology:	The	Evolution	of	Goals
Psychology:	The	Pursuit	of	and	Rebellion	Against	Goals

Engineering:	Outsourcing	Goals
Friendly	AI:	Aligning	Goals
Ethics:	Choosing	Goals
Ultimate	Goals?
8 
Consciousness
Who	Cares?
What	Is	Consciousness?
What’s	the	Problem?
Is	Consciousness	Beyond	Science?
Experimental	Clues	About	Consciousness
Theories	of	Consciousness
Controversies	of	Consciousness
How	Might	AI	Consciousness	Feel?
Meaning
Epilogue:	
The	Tale	of	the	FLI	Team
Notes

I’m	truly	grateful	to	everyone	who	has	encouraged	and	helped	me	write	this
book,	including
my	family,	friends,	teachers,	colleagues	and	collaborators	for	support	and
inspiration	over	the	years,
Mom	for	kindling	my	curiosity	about	consciousness	and	meaning,
Dad	for	the	fighting	spirit	to	make	the	world	a	better	place,
my	sons,	Philip	and	Alexander,	for	demonstrating	the	wonders	of	human-level
intelligence	emerging,
all	the	science	and	technology	enthusiasts	around	the	world	who’ve	contacted
me	over	the	years	with	questions,	comments	and	encouragement	to	pursue
and	publish	my	ideas,
my	agent,	John	Brockman,	for	twisting	my	arm	until	I	agreed	to	write	this	book,
Bob	Penna,	Jesse	Thaler	and	Jeremy	England	for	helpful	discussions	about
quasars,	sphalerons	and	thermodynamics,	respectively,
those	who	gave	me	feedback	on	parts	of	the	manuscript,	including	Mom,	my
brother	Per,	Luisa	Bahet,	Rob	Bensinger,	Katerina	Bergström,	Erik
Brynjolfsson,	Daniela	Chita,	David	Chalmers,	Nima	Deghani,	Henry	Lin,

Elin	Malmsköld,	Toby	Ord,	Jeremy	Owen,	Lucas	Perry,	Anthony	Romero,
Nate	Soares	and	Jaan	Tallinn,
the	superheroes	who	commented	on	drafts	of	the	entire	book,	namely	Meia,	Dad,
Anthony	Aguirre,	Paul	Almond,	Matthew	Graves,	Phillip	Helbig,	Richard
Mallah,	David	Marble,	Howard	Messing,	Luiño	Seoane,	Marin	Solja
č
i
ć
,	my
editor	Dan	Frank	and,	most	of	all,
Meia,	my	beloved	muse	and	fellow	traveler,	for	her	eternal	encouragement,

support	and	inspiration,	without	which	this	book	wouldn’t	exist.

Prelude
The	Omega	Team	was	the	soul	of	the	company.	Whereas	the	rest	of	the
enterprise	brought	in	the	money	to	keep	things	going,	by	various	commercial
applications	of	narrow	AI,	the	Omega	Team	pushed	ahead	in	their	quest	for	what
had	always	been	the	CEO’s	dream:	building	general	artificial	intelligence.	Most
other	employees	viewed	“the	Omegas,”	as	they	affectionately	called	them,	as	a
bunch	of	pie-in-the-sky	dreamers,	perpetually	decades	away	from	their	goal.
They	happily	indulged	them,	however,	because	they	liked	the	prestige	that	the
cutting-edge	work	of	the	Omegas	gave	their	company,	and	they	also	appreciated
the	improved	algorithms	that	the	Omegas	occasionally	gave	them.
What	they	didn’t	realize	was	that	the	Omegas	had	carefully	crafted	their
image	to	hide	a	secret:	they	were	extremely	close	to	pulling	off	the	most
audacious	plan	in	human	history.	Their	charismatic	CEO	had	handpicked	them
not	only	for	being	brilliant	researchers,	but	also	for	ambition,	idealism	and	a

strong	commitment	to	helping	humanity.	He	reminded	them	that	their	plan	was
extremely	dangerous,	and	that	if	powerful	governments	found	out,	they	would
do	virtually	anything—including	kidnapping—to	shut	them	down	or,	preferably,
to	steal	their	code.	But	they	were	all	in,	100%,	for	much	the	same	reason	that
many	of	the	world’s	top	physicists	joined	the	Manhattan	Project	to	develop
nuclear	weapons:	they	were	convinced	that	if	they	didn’t	do	it	first,	someone	less
idealistic	would.
The	AI	they	had	built,	nicknamed	Prometheus,	kept	getting	more	capable.

Although	its	cognitive	abilities	still	lagged	far	behind	those	of	humans	in	many
areas,	for	example,	social	skills,	the	Omegas	had	pushed	hard	to	make	it

extraordinary	at	one	particular	task:	programming	AI	systems.	They’d
deliberately	chosen	this	strategy	because	they	had	bought	the	intelligence
explosion	argument	made	by	the	British	mathematician	Irving	Good	back	in
1965:	“Let	an	ultraintelligent	machine	be	defined	as	a	machine	that	can	far
surpass	all	the	intellectual	activities	of	any	man	however	clever.	Since	the	design
of	machines	is	one	of	these	intellectual	activities,	an	ultraintelligent	machine
could	design	even	better	machines;	there	would	then	unquestionably	be	an
‘intelligence	explosion,’	and	the	intelligence	of	man	would	be	left	far	behind.
Thus	the	first	ultraintelligent	machine	is	the	last	invention	that	man	need	ever
make,	provided	that	the	machine	is	docile	enough	to	tell	us	how	to	keep	it	under
control.”
They	figured	that	if	they	could	get	this	recursive	self-improvement	going,	the
machine	would	soon	get	smart	enough	that	it	could	also	teach	itself	all	other
human	skills	that	would	be	useful.

The	First	Millions
It	was	nine	o’clock	on	a	Friday	morning	when	they	decided	to	launch.
Prometheus	was	humming	away	in	its	custom-built	computer	cluster,	which
resided	in	long	rows	of	racks	in	a	vast,	access-controlled,	air-conditioned	room.
For	security	reasons,	it	was	completely	disconnected	from	the	internet,	but	it
contained	a	local	copy	of	much	of	the	web	(Wikipedia,	the	Library	of	Congress,
Twitter,	a	selection	from	YouTube,	much	of	Facebook,	etc.)	to	use	as	its	training
data	to	learn	from.
*
	They’d	picked	this	start	time	to	work	undisturbed:	their
families	and	friends	thought	they	were	on	a	weekend	corporate	retreat.	The
kitchenette	was	loaded	with	microwaveable	food	and	energy	drinks,	and	they
were	ready	to	roll.
When	they	launched,	Prometheus	was	slightly	worse	than	them	at
programming	AI	systems,	but	made	up	for	this	by	being	vastly	faster,	spending
the	equivalent	of	thousands	of	person-years	chugging	away	at	the	problem	while

*
they	chugged	a	Red	Bull.	By	10	a.m.,	it	had	completed	the	first	redesign	of	itself,
v2.0,	which	was	slightly	better	but	still	subhuman.	By	the	time	Prometheus	5.0
launched	at	2	p.m.,	however,	the	Omegas	were	awestruck:	it	had	blown	their
performance	benchmarks	out	of	the	water,	and	the	rate	of	progress	seemed	to	be
accelerating.	By	nightfall,	they	decided	to	deploy	Prometheus	10.0	to	start	phase
2	of	their	plan:	making	money.
Their	first	target	was	MTurk,	the	Amazon	Mechanical	Turk.	After	its	launch

in	2005	as	a	crowdsourcing	internet	marketplace,	it	had	grown	rapidly,	with	tens
of	thousands	of	people	around	the	world	anonymously	competing	around	the
clock	to	perform	highly	structured	chores	called	HITs,	“Human	Intelligence
Tasks.”	These	tasks	ranged	from	transcribing	audio	recordings	to	classifying
images	and	writing	descriptions	of	web	pages,	and	all	had	one	thing	in	common:
if	you	did	them	well,	nobody	would	know	that	you	were	an	AI.	Prometheus	10.0

was	able	to	do	about	half	of	the	task	categories	acceptably	well.	For	each	such
task	category,	the	Omegas	had	Prometheus	design	a	lean	custom-built	narrow	AI
software	module	that	could	do	precisely	such	tasks	and	nothing	else.	They	then
uploaded	this	module	to	Amazon	Web	Services,	a	cloud-computing	platform	that
could	run	on	as	many	virtual	machines	as	they	rented.	For	every	dollar	they	paid
to	Amazon’s	cloud-computing	division,	they	earned	more	than	two	dollars	from

Amazon’s	MTurk	division.	Little	did	Amazon	suspect	that	such	an	amazing
arbitrage	opportunity	existed	within	their	own	company!
To	cover	their	tracks,	they	had	discreetly	created	thousands	of	MTurk
accounts	during	the	preceding	months	in	the	names	of	fictitious	people,	and	the
Prometheus-built	modules	now	assumed	their	identities.	The	MTurk	customers
typically	paid	after	about	eight	hours,	at	which	point	the	Omegas	reinvested	the
money	in	more	cloud-computing	time,	using	still	better	task	modules	made	by
the	latest	version	of	the	ever-improving	Prometheus.	Because	they	were	able	to
double	their	money	every	eight	hours,	they	soon	started	saturating	MTurk’s	task
supply,	and	found	that	they	couldn’t	earn	more	than	about	a	million	dollars	per
day	without	drawing	unwanted	attention	to	themselves.	But	this	was	more	than
enough	to	fund	their	next	step,	eliminating	any	need	for	awkward	cash	requests
to	the	chief	financial	officer.

Dangerous	Games
Aside	from	their	AI	breakthroughs,	one	of	the	recent	projects	that	the	Omegas
had	had	the	most	fun	with	was	planning	how	to	make	money	as	rapidly	as
possible	after	Prometheus’	launch.	Essentially	the	whole	digital	economy	was	up
for	grabs,	but	was	it	better	to	start	by	making	computer	games,	music,	movies	or
software,	to	write	books	or	articles,	to	trade	on	the	stock	market	or	to	make
inventions	and	sell	them?	It	simply	boiled	down	to	maximizing	their	rate	of
return	on	investment,	but	normal	investment	strategies	were	a	slow-motion
parody	of	what	they	could	do:	whereas	a	normal	investor	might	be	pleased	with
a	9%	return	per	
year,
	their	MTurk	investments	had	yielded	9%	per	
hour,
generating	eight	times	more	money	each	day.	So	now	that	they’d	saturated
MTurk,	what	next?
Their	first	thought	had	been	to	make	a	killing	on	the	stock	market—after	all,
pretty	much	all	of	them	had	at	some	point	declined	a	lucrative	job	offer	to

year,
hour,
develop	AI	for	hedge	funds,	which	were	investing	heavily	in	exactly	this	idea.
Some	remembered	that	this	was	how	the	AI	made	its	first	millions	in	the	movie
Transcendence
.	But	the	new	regulations	on	derivatives	after	last	year’s	crash	had
limited	their	options.	They	soon	realized	that,	even	though	they	could	get	much
better	returns	than	other	investors,	they’d	be	unlikely	to	get	returns	anywhere
close	to	what	they	could	get	from	selling	their	own	products.	When	you	have	the

world’s	first	superintelligent	AI	working	for	you,	you’re	better	off	investing	in
your	own	companies	than	in	those	of	others!	Although	there	might	be	occasional
exceptions	(such	as	using	Prometheus’	superhuman	hacking	abilities	to	get
inside	information	and	then	buy	call	options	on	stocks	about	to	surge),	the
Omegas	felt	that	this	wasn’t	worth	the	unwanted	attention	it	might	draw.
When	they	shifted	their	focus	toward	products	that	they	could	develop	and
sell,	computer	games	first	seemed	the	obvious	top	choice.	Prometheus	could

rapidly	become	extremely	skilled	at	designing	appealing	games,	easily	handling
the	coding,	graphic	design,	ray	tracing	of	images	and	all	other	tasks	needed	to
produce	a	final	ready-to-ship	product.	Moreover,	after	digesting	all	the	web’s
data	on	people’s	preferences,	it	would	know	exactly	what	each	category	of
gamer	liked,	and	could	develop	a	superhuman	ability	to	optimize	a	game	for
sales	revenue.	
The	Elder	Scrolls
	
V:	Skyrim
,	a	game	on	which	many	of	the

Omegas	had	wasted	more	hours	than	they	cared	to	admit,	had	grossed	over	$400
million	during	its	first	week	back	in	2011,	and	they	were	confident	that
Prometheus	could	make	something	at	least	this	addictive	in	twenty-four	hours
using	$1	million	of	cloud-computing	resources.	They	could	then	sell	it	online
and	use	Prometheus	to	impersonate	humans	talking	up	the	game	in	the
blogosphere.	If	this	brought	in	$250	million	in	a	week,	they	would	have	doubled
their	investment	eight	times	in	eight	days,	giving	a	return	of	3%	per	hour—
slightly	worse	than	their	MTurk	start,	but	much	more	sustainable.	By	developing
a	suite	of	other	games	each	day,	they	figured	they’d	be	able	to	earn	$10	billion
before	long,	without	coming	close	to	saturating	the	games	market.
But	a	cybersecurity	specialist	on	their	team	talked	them	out	of	this	game	plan.
She	pointed	out	that	it	would	pose	an	unacceptable	risk	of	Prometheus	
breaking
out
	and	seizing	control	of	its	own	destiny.	Because	they	weren’t	sure	how	its

breaking
out
goals	would	evolve	during	its	recursive	self-improvement,	they	had	decided	to
play	it	safe	and	go	to	great	lengths	to	keep	Prometheus	confined	(“boxed”)	in
ways	such	that	it	couldn’t	escape	onto	the	internet.	For	the	main	Prometheus
engine	running	in	their	server	room,	they	used	physical	confinement:	there
simply	was	no	internet	connection,	and	the	only	output	from	Prometheus	was	in
the	form	of	messages	and	documents	it	sent	to	a	computer	that	the	Omegas
controlled.
On	an	internet-connected	computer,	on	the	other	hand,	running	any

complicated	program	created	by	Prometheus	was	a	risky	proposition:	since	the
Omegas	had	no	way	of	fully	understanding	what	it	would	do,	they	had	no	way	of
knowing	that	it	wouldn’t,	say,	start	virally	spreading	itself	online.	When	testing
the	software	that	Prometheus	had	written	for	MTurk	tasks,	the	Omegas	guarded
against	this	by	running	it	only	inside	a	virtual	machine.	This	is	a	program	that
simulates	a	computer:	for	example,	many	Mac	users	buy	virtual	machine
software	that	lets	them	run	Windows	programs	by	tricking	them	into	thinking

that	they’re	actually	in	a	Windows	machine.	The	Omegas	had	created	their	own
virtual	machine,	nicknamed	Pandora’s	Box,	which	simulated	an	ultrasimplified
machine	stripped	of	all	bells	and	whistles	that	we	usually	associate	with
computers:	no	keyboard,	no	monitor,	no	loudspeakers,	no	internet	connectivity,
nothing.	For	the	MTurk	audio	transcriptions,	the	Omegas	set	things	up	so	that	all
that	could	go	into	Pandora’s	Box	was	one	single	audio	file	and	all	that	could
come	out	was	one	single	text	document—the	transcription.	These	laws	of	the

box	were	to	the	software	inside	like	the	laws	of	physics	are	to	us	inside	our
Universe:	the	software	couldn’t	travel	out	of	the	box	any	more	than	we	can

travel	faster	than	the	speed	of	light,	no	matter	how	smart	we	are.	Except	for	that
single	input	and	output,	the	software	inside	Pandora’s	Box	was	effectively
trapped	in	a	parallel	universe	with	its	own	computational	rules.	The	Omegas	had
such	strong	breakout	paranoia	that	they	added	boxing	in	time	as	well,	limiting
the	life	span	of	untrusted	code.	For	example,	each	time	the	boxed	transcription
software	had	finished	transcribing	one	audio	file,	the	entire	memory	content	of
Pandora’s	Box	was	automatically	erased	and	the	program	was	reinstalled	from
scratch.	This	way,	when	it	started	the	next	transcription	task,	it	had	no
knowledge	of	what	had	previously	happened,	and	thus	no	ability	to	learn	over
time.
When	the	Omegas	used	the	Amazon	cloud	for	their	MTurk	project,	they	were
able	to	put	all	their	Prometheus-created	task	modules	into	such	virtual	boxes	in
the	cloud,	because	the	MTurk	input	and	output	was	so	simple.	But	this	wouldn’t
work	for	graphics-heavy	computer	games,	which	couldn’t	be	boxed	in	because

time.
they	needed	full	access	to	all	the	hardware	of	the	gamer’s	computer.	Moreover,
they	didn’t	want	to	risk	that	some	computer-savvy	user	would	analyze	their
game	code,	discover	Pandora’s	Box	and	decide	to	investigate	what	was	inside.
The	breakout	risk	put	not	merely	the	games	market	off-limits	for	now,	but	also
the	massively	lucrative	market	for	other	software,	with	hundreds	of	billions	of
dollars	up	for	grabs.

The	First	Billions
The	Omegas	had	narrowed	their	search	to	products	that	were	highly	valuable,
purely	digital	(avoiding	slow	manufacturing)	and	easily	understandable	(for
example,	text	or	movies	they	knew	wouldn’t	pose	a	breakout	risk).	In	the	end,
they	had	decided	to	launch	a	media	company,	starting	with	animated
entertainment.	The	website,	the	marketing	plan	and	the	press	releases	had	all
been	ready	to	go	even	before	Prometheus	became	superintelligent—all	that	was
missing	was	content.
Although	Prometheus	was	astonishingly	capable	by	Sunday	morning,	steadily
raking	in	money	from	MTurk,	its	intellectual	abilities	were	still	rather	narrow:
Prometheus	had	been	deliberately	optimized	to	design	AI	systems	and	write
software	that	performed	rather	mind-numbing	MTurk	tasks.	It	was,	for	example,
bad	at	making	movies—bad	not	for	any	profound	reason,	but	for	the	same	reason
that	James	Cameron	was	bad	at	making	movies	when	he	was	born:	this	is	a	skill

that	takes	time	to	learn.	Like	a	human	child,	Prometheus	could	learn	whatever	it
wanted	from	the	data	it	had	access	to.	Whereas	James	Cameron	had	taken	years
to	learn	to	read	and	write,	Prometheus	had	gotten	that	taken	care	of	on	Friday,
when	it	also	found	time	to	read	all	of	Wikipedia	and	a	few	million	books.
Making	movies	was	harder.	Writing	a	screenplay	that	humans	found	interesting
was	just	as	hard	as	writing	a	book,	requiring	a	detailed	understanding	of	human

society	and	what	humans	found	entertaining.	Turning	the	screenplay	into	a	final
video	file	required	massive	amounts	of	ray	tracing	of	simulated	actors	and	the
complex	scenes	they	moved	through,	simulated	voices,	the	production	of
compelling	musical	soundtracks	and	so	on.	As	of	Sunday	morning,	Prometheus
could	watch	a	two-hour	movie	in	about	a	minute,	which	included	reading	any
book	it	was	based	on	and	all	online	reviews	and	ratings.	The	Omegas	noticed
that	after	Prometheus	had	binge-watched	a	few	hundred	films,	it	started	to	get

quite	good	at	predicting	what	sort	of	reviews	a	movie	would	get	and	how	it
would	appeal	to	different	audiences.	Indeed,	it	learned	to	write	its	own	movie
reviews	in	a	way	they	felt	demonstrated	real	insight,	commenting	on	everything
from	the	plots	and	the	acting	to	technical	details	such	as	lighting	and	camera
angles.	They	took	this	to	mean	that	when	Prometheus	made	its	own	films,	it
would	know	what	success	meant.

The	Omegas	instructed	Prometheus	to	focus	on	making	animation	at	first,	to
avoid	embarrassing	questions	about	who	the	simulated	actors	were.	On	Sunday
night,	they	capped	their	wild	weekend	by	arming	themselves	with	beer	and
microwave	popcorn,	dimming	the	lights	and	watching	Prometheus’	debut	movie.
It	was	an	animated	fantasy-comedy	in	the	spirit	of	Disney’s	
Frozen,
	and	the	ray
tracing	had	been	performed	by	boxed	Prometheus-built	code	in	the	Amazon
cloud,	using	up	most	of	the	day’s	$1	million	MTurk	profit.	As	the	movie	began,
they	found	it	both	fascinating	and	frightening	that	it	had	been	created	by	a
machine	without	human	guidance.	Before	long,	however,	they	were	laughing	at
the	gags	and	holding	their	breath	during	the	dramatic	moments.	Some	of	them
even	teared	up	a	bit	at	the	emotional	ending,	so	engrossed	in	this	fictional	reality
that	they	forgot	all	about	its	creator.
The	Omegas	scheduled	their	website	launch	for	Friday,	giving	Prometheus

time	to	produce	more	content	and	themselves	time	to	do	the	things	they	didn’t
trust	Prometheus	with:	buying	ads	and	starting	to	recruit	employees	for	the	shell
companies	they’d	set	up	during	the	past	months.	To	cover	their	tracks,	the
official	cover	story	would	be	that	their	media	company	(which	had	no	public
association	with	the	Omegas)	bought	most	of	its	content	from	independent	film
producers,	typically	high-tech	startups	in	low-income	regions.	These	fake

suppliers	were	conveniently	located	in	remote	places	such	as	Tiruchchirappalli
and	Yakutsk,	which	most	curious	journalists	wouldn’t	bother	visiting.	The	only
employees	they	actually	hired	there	worked	on	marketing	and	administration,
and	would	tell	anyone	who	asked	that	their	production	team	was	in	a	different
location	and	didn’t	conduct	interviews	at	the	moment.	To	match	their	cover
story,	they	chose	the	corporate	slogan	“Channeling	the	world’s	creative	talent,”
and	branded	their	company	as	being	disruptively	different	by	using	cutting-edge

technology	to	empower	creative	people,	especially	in	the	developing	world.
When	Friday	came	around	and	curious	visitors	started	arriving	at	their	site,
they	encountered	something	reminiscent	of	the	online	entertainment	services
Netflix	and	Hulu	but	with	interesting	differences.	All	the	animated	series	were
new	ones	they’d	never	heard	of.	They	were	rather	captivating:	most	series
consisted	of	forty-five-minute-long	episodes	with	a	strong	plotline,	each	ending
in	a	way	that	left	you	eager	to	find	out	what	happened	in	the	next	episode.	And

they	were	cheaper	than	the	competition.	The	first	episode	of	each	series	was
free,	and	you	could	watch	the	others	for	forty-nine	cents	each,	with	discounts	for
the	whole	series.	Initially,	there	were	only	three	series	with	three	episodes	each,
but	new	episodes	were	added	daily,	as	well	as	new	series	catering	to	different

demographics.	During	the	first	two	weeks	of	Prometheus,	its	moviemaking	skills
improved	rapidly,	in	terms	not	only	of	film	quality	but	also	of	better	algorithms
for	character	simulation	and	ray	tracing,	which	greatly	reduced	the	cloud-
computing	cost	to	make	each	new	episode.	As	a	result,	the	Omegas	were	able	to
roll	out	dozens	of	new	series	during	the	first	month,	targeting	demographics
from	toddlers	to	adults,	as	well	as	to	expand	to	all	major	world	language
markets,	making	their	site	remarkably	international	compared	with	all
competitors.	Some	commentators	were	impressed	by	the	fact	that	it	wasn’t
merely	the	soundtracks	that	were	multilingual,	but	the	videos	themselves:	for
example,	when	a	character	spoke	Italian,	the	mouth	motions	matched	the	Italian
words,	as	did	the	characteristically	Italian	hand	gestures.	Although	Prometheus
was	now	perfectly	capable	of	making	movies	with	simulated	actors
indistinguishable	from	humans,	the	Omegas	avoided	this	to	not	tip	their	hand.

They	did,	however,	launch	many	series	with	semi-realistic	animated	human
characters,	in	genres	competing	with	traditional	live-action	TV	shows	and
movies.
Their	network	turned	out	to	be	quite	addictive,	and	enjoyed	spectacular	viewer
growth.	Many	fans	found	the	characters	and	plots	cleverer	and	more	interesting
than	even	Hollywood’s	most	expensive	big-screen	productions,	and	were
delighted	that	they	could	watch	them	much	more	affordably.	Buoyed	by
aggressive	advertising	(which	the	Omegas	could	afford	because	of	their	near-

movies.
zero	production	costs),	excellent	media	coverage	and	rave	word-of-mouth
reviews,	their	global	revenue	had	mushroomed	to	$10	million	a	day	within	a
month	of	launch.	After	two	months,	they	had	overtaken	Netflix,	and	after	three,
they	were	raking	in	over	$100	million	a	day,	beginning	to	rival	Time	Warner,
Disney,	Comcast	and	Fox	as	one	of	the	world’s	largest	media	empires.
Their	sensational	success	garnered	plenty	of	unwanted	attention,	including
speculation	about	their	having	strong	AI,	but	using	merely	a	small	fraction	of

their	revenue,	the	Omegas	deployed	a	fairly	successful	disinformation	campaign.
From	a	glitzy	new	Manhattan	office,	their	freshly	hired	spokespeople	would
elaborate	on	their	cover	stories.	Plenty	of	humans	were	hired	as	foils,	including
actual	screenwriters	around	the	world	to	start	developing	new	series,	none	of
whom	knew	about	Prometheus.	The	confusing	international	network	of
subcontractors	made	it	easy	for	most	of	their	employees	to	assume	that	others
somewhere	else	were	doing	most	of	the	work.

To	make	themselves	less	vulnerable	and	avoid	raising	eyebrows	with
excessive	cloud	computing,	they	also	hired	engineers	to	start	building	a	series	of

massive	computer	facilities	around	the	world,	owned	by	seemingly	unaffiliated
shell	companies.	Although	they	were	billed	to	locals	as	“green	data	centers”
because	they	were	largely	solar-powered,	they	were	in	fact	mainly	focused	on
computation	rather	than	storage.	Prometheus	had	designed	their	blueprints	down
to	the	most	minute	detail,	using	only	off-the-shelf	hardware	and	optimizing	them
to	minimize	construction	time.	The	people	who	built	and	ran	these	centers	had
no	idea	what	was	computed	there:	they	thought	they	managed	commercial
cloud-computing	facilities	similar	to	those	run	by	Amazon,	Google	and
Microsoft,	and	knew	only	that	all	sales	were	managed	remotely.

New	Technologies
Over	a	timescale	of	months,	the	business	empire	controlled	by	the	Omegas
started	gaining	a	foothold	in	ever	more	areas	of	the	world	economy,	thanks	to
superhuman	planning	by	Prometheus.	By	carefully	analyzing	the	world’s	data,	it
had	already	during	its	first	week	presented	the	Omegas	with	a	detailed	step-by-
step	growth	plan,	and	it	kept	improving	and	refining	this	plan	as	its	data	and
computer	resources	grew.	Although	Prometheus	was	far	from	omniscient,	its
capabilities	were	now	so	far	beyond	human	that	the	Omegas	viewed	it	as	the
perfect	oracle,	dutifully	providing	brilliant	answers	and	advice	in	response	to	all
their	questions.
Prometheus’	software	was	now	highly	optimized	to	make	the	most	of	the
rather	mediocre	human-invented	hardware	it	ran	on,	and	as	the	Omegas	had
anticipated,	Prometheus	identified	ways	of	dramatically	improving	this
hardware.	Fearing	a	breakout,	they	refused	to	build	robotic	construction	facilities

that	Prometheus	could	control	directly.	Instead,	they	hired	large	numbers	of
world-class	scientists	and	engineers	in	multiple	locations	and	fed	them	internal
research	reports	written	by	Prometheus,	pretending	that	they	were	from
researchers	at	the	other	sites.	These	reports	detailed	novel	physical	effects	and
manufacturing	techniques	that	their	engineers	soon	tested,	understood	and
mastered.	Normal	human	research	and	development	(R	&	D)	cycles,	of	course,
take	years,	in	large	part	because	they	involve	many	slow	cycles	of	trial	and	error.

The	current	situation	was	very	different:	Prometheus	already	had	the	next	steps
figured	out,	so	the	limiting	factor	was	simply	how	rapidly	people	could	be
guided	to	understand	and	build	the	right	things.	A	good	teacher	can	help	students
learn	science	much	faster	than	they	could	have	discovered	it	from	scratch	on
their	own,	and	Prometheus	surreptitiously	did	the	same	with	these	researchers.
Since	Prometheus	could	accurately	predict	how	long	it	would	take	humans	to
understand	and	build	things	given	various	tools,	it	developed	the	quickest

possible	path	forward,	giving	priority	to	new	tools	that	could	be	quickly
understood	and	built	and	that	were	useful	for	developing	more	advanced	tools.
In	the	spirit	of	the	maker	movement,	the	engineering	teams	were	encouraged
to	use	their	own	machines	to	build	their	better	machines.	This	self-sufficiency
not	only	saved	money,	but	it	also	made	them	less	vulnerable	to	future	threats

from	the	outside	world.	Within	two	years,	they	were	producing	much	better
computer	hardware	than	the	world	had	ever	known.	To	avoid	helping	outside
competition,	they	kept	this	technology	under	wraps	and	used	it	only	to	upgrade
Prometheus.
What	the	world	did	notice,	however,	was	an	astonishing	tech	boom.	Upstart
companies	around	the	world	were	launching	revolutionary	new	products	in
almost	all	areas.	A	South	Korean	startup	launched	a	new	battery	that	stored	twice
as	much	energy	as	your	laptop	battery	in	half	the	mass,	and	could	be	charged	in
under	a	minute.	A	Finnish	firm	released	a	cheap	solar	panel	with	twice	the
efficiency	of	the	best	competitors.	A	German	company	announced	a	new	type	of
mass-producible	wire	that	was	superconducting	at	room	temperature,
revolutionizing	the	energy	sector.	A	Boston-based	biotech	group	announced	a
Phase	II	clinical	trial	of	what	they	claimed	was	the	first	effective,	side-effect-free
weight-loss	drug,	while	rumors	suggested	that	an	Indian	outfit	was	already

selling	something	similar	on	the	black	market.	A	California	company	countered
with	a	Phase	II	trial	of	a	blockbuster	cancer	drug,	which	caused	the	body’s
immune	system	to	identify	and	attack	cells	with	any	of	the	most	common
cancerous	mutations.	Examples	just	kept	on	coming,	triggering	talk	of	a	new
golden	age	for	science.	Last	but	not	least,	robotics	companies	were	cropping	up
like	mushrooms	all	around	the	world.	None	of	the	bots	came	close	to	matching
human	intelligence,	and	most	of	them	looked	nothing	like	humans.	But	they

dramatically	disrupted	the	economy,	and	over	the	years	to	come,	they	gradually
replaced	most	of	the	workers	in	manufacturing,	transportation,	warehousing,
retail,	construction,	mining,	agriculture,	forestry	and	fishing.
What	the	world	didn’t	notice,	thanks	to	the	hard	work	of	a	crack	team	of
lawyers,	was	that	all	these	firms	were	controlled,	through	a	series	of
intermediaries,	by	the	Omegas.	Prometheus	was	flooding	the	world’s	patent
offices	with	sensational	inventions	via	various	proxies,	and	these	inventions

gradually	led	to	domination	in	all	areas	of	technology.
Although	these	disruptive	new	companies	made	powerful	enemies	among
their	competition,	they	made	even	more	powerful	friends.	They	were
exceptionally	profitable,	and	under	slogans	such	as	“Investing	in	our
community,”	they	spent	a	significant	fraction	of	these	profits	hiring	people	for
community	projects—often	the	same	people	who	had	been	laid	off	from	the
companies	that	were	disrupted.	They	used	detailed	Prometheus-produced
analyses	identifying	jobs	that	would	be	maximally	rewarding	for	the	employees

and	the	community	for	the	least	cost,	tailored	to	the	local	circumstances.	In

regions	with	high	levels	of	government	service,	this	often	focused	on	community
building,	culture	and	caregiving,	while	in	poorer	regions	it	also	included
launching	and	maintaining	schools,	healthcare,	day	care,	elder	care,	affordable
housing,	parks	and	basic	infrastructure.	Pretty	much	everywhere,	locals	agreed
that	these	were	things	that	should	have	been	done	long	ago.	Local	politicians	got
generous	donations,	and	care	was	taken	to	make	them	look	good	for	encouraging
these	corporate	community	investments.

Gaining	Power
The	Omegas	had	launched	a	media	company	not	only	to	finance	their	early	tech
ventures,	but	also	for	the	next	step	of	their	audacious	plan:	taking	over	the	world.
Within	a	year	of	the	first	launch,	they	had	added	remarkably	good	news	channels
to	their	lineup	all	over	the	globe.	As	opposed	to	their	other	channels,	these	were
deliberately	designed	to	lose	money,	and	were	pitched	as	a	public	service.	In
fact,	their	news	channels	generated	no	income	whatsoever:	they	carried	no	ads
and	were	viewable	free	of	charge	by	anyone	with	an	internet	connection.	The
rest	of	their	media	empire	was	such	a	cash-generating	machine	that	they	could
spend	far	more	resources	on	their	news	service	than	any	other	journalistic	effort
had	done	in	world	history—and	it	showed.	Through	aggressive	recruitment	with
highly	competitive	salaries	of	journalists	and	investigative	reporters,	they
brought	remarkable	talent	and	findings	to	the	screen.	Through	a	global	web

service	that	paid	anybody	who	revealed	something	newsworthy,	from	local
corruption	to	a	heartwarming	event,	they	were	usually	the	first	to	break	a	story.
At	least	that’s	what	people	believed:	in	fact,	they	were	often	first	because	stories
attributed	to	citizen	journalists	had	been	discovered	by	Prometheus	via	real-time
monitoring	of	the	internet.	All	these	video	news	sites	featured	podcasts	and	print
articles	as	well.
Phase	1	of	their	news	strategy	was	gaining	people’s	trust,	which	they	did	with

great	success.	Their	unprecedented	willingness	to	lose	money	enabled
remarkably	diligent	regional	and	local	news	coverage,	where	investigative
journalists	often	exposed	scandals	that	truly	engaged	their	viewers.	Whenever	a
country	was	strongly	divided	politically	and	accustomed	to	partisan	news,	they
would	launch	one	news	channel	catering	to	each	faction,	ostensibly	owned	by
different	companies,	and	gradually	gain	the	trust	of	that	faction.	Where	possible,

they	accomplished	this	using	proxies	to	buy	the	most	influential	existing
channels,	gradually	improving	them	by	removing	ads	and	introducing	their	own
content.	In	countries	where	censorship	and	political	interference	threatened	these
efforts,	they	would	initially	acquiesce	in	whatever	the	government	required	of
them	to	stay	in	business,	with	the	secret	internal	slogan	“The	truth,	nothing	but
the	truth,	but	maybe	not	the	whole	truth.”	Prometheus	usually	provided	excellent
advice	in	such	situations,	clarifying	which	politicians	needed	to	be	presented	in	a

good	light	and	which	(usually	corrupt	local	ones)	could	be	exposed.	Prometheus
also	provided	invaluable	recommendations	for	what	strings	to	pull,	whom	to
bribe	and	how	best	to	do	so.
This	strategy	was	a	smashing	success	around	the	world,	with	the	Omega-
controlled	channels	emerging	as	the	most	trusted	news	sources.	Even	in
countries	where	governments	had	thus	far	thwarted	their	mass	adoption,	they
built	a	reputation	for	trustworthiness,	and	many	of	their	news	stories	percolated
through	the	grapevine.	Competing	news	executives	felt	that	they	were	fighting	a
hopeless	battle:	how	can	you	possibly	make	a	profit	competing	with	someone
with	better	funding	who	gives	their	products	away	for	free?	With	their
viewership	dropping,	ever	more	networks	decided	to	sell	their	news	channels—
usually	to	some	consortium	that	later	turned	out	to	be	controlled	by	the	Omegas.
About	two	years	after	Prometheus’	launch,	when	the	trust-gaining	phase	was
largely	completed,	the	Omegas	launched	phase	2	of	their	news	strategy:

persuasion.	Even	before	this,	astute	observers	had	noticed	hints	of	a	political
agenda	behind	the	new	media:	there	seemed	to	be	a	gentle	push	toward	the
center,	away	from	extremism	of	all	sorts.	Their	plethora	of	channels	catering	to
different	groups	still	reflected	animosity	between	the	United	States	and	Russia,
India	and	Pakistan,	different	religions,	political	factions	and	so	on,	but	the
criticism	was	slightly	toned	down,	usually	focusing	on	concrete	issues	involving
money	and	power	rather	than	on	ad	hominem	attacks,	scaremongering	and

poorly	substantiated	rumors.	Once	phase	2	started	in	earnest,	this	push	to	defuse
old	conflicts	became	more	apparent,	with	frequent	touching	stories	about	the
plight	of	traditional	adversaries	mixed	with	investigative	reporting	about	how
many	vocal	conflict-mongers	were	driven	by	personal	profit	motives.
Political	commentators	noted	that,	in	parallel	with	damping	regional	conflicts,
there	seemed	to	be	a	concerted	push	toward	reducing	global	threats.	For
example,	the	risks	of	nuclear	war	were	suddenly	being	discussed	all	over	the

place.	Several	blockbuster	movies	featured	scenarios	where	global	nuclear	war
started	by	accident	or	on	purpose	and	dramatized	the	dystopian	aftermath	with
nuclear	winter,	infrastructure	collapse	and	mass	starvation.	Slick	new
documentaries	detailed	how	nuclear	winter	could	impact	every	country.
Scientists	and	politicians	advocating	nuclear	de-escalation	were	given	ample
airtime,	not	least	to	discuss	the	results	of	several	new	studies	on	what	helpful
measures	could	be	taken—studies	funded	by	scientific	organizations	that	had

received	large	donations	from	new	tech	companies.	As	a	result,	political
momentum	started	building	for	taking	missiles	off	hair-trigger	alert	and

shrinking	nuclear	arsenals.	Renewed	media	attention	was	also	paid	to	global
climate	change,	often	highlighting	the	recent	Prometheus-enabled	technological
breakthroughs	that	were	slashing	the	cost	of	renewable	energy	and	encouraging
governments	to	invest	in	such	new	energy	infrastructure.
Parallel	to	their	media	takeover,	the	Omegas	harnessed	Prometheus	to
revolutionize	education.	Given	any	person’s	knowledge	and	abilities,
Prometheus	could	determine	the	fastest	way	for	them	to	learn	any	new	subject	in
a	manner	that	kept	them	highly	engaged	and	motivated	to	continue,	and	produce
the	corresponding	optimized	videos,	reading	materials,	exercises	and	other
learning	tools.	Omega-controlled	companies	therefore	marketed	online	courses
about	virtually	everything,	highly	customized	not	only	by	language	and	cultural
background	but	also	by	starting	level.	Whether	you	were	an	illiterate	forty-year-
old	wanting	to	learn	to	read	or	a	biology	PhD	seeking	the	latest	about	cancer

immunotherapy,	Prometheus	had	the	perfect	course	for	you.	These	offerings	bore
little	resemblance	to	most	present-day	online	courses:	by	leveraging	Prometheus’
movie-making	talents,	the	video	segments	would	truly	engage,	providing
powerful	metaphors	that	you	would	relate	to,	leaving	you	craving	to	learn	more.
Some	courses	were	sold	for	profit,	but	many	were	made	available	for	free,	much
to	the	delight	of	teachers	around	the	world	who	could	use	them	in	their
classrooms—and	to	most	anybody	eager	to	learn	anything.

These	educational	superpowers	proved	potent	tools	for	political	purposes,
creating	online	“persuasion	sequences”	of	videos	where	insights	from	each	one
would	both	update	someone’s	views	and	motivate	them	to	watch	another	video
about	a	related	topic	where	they	were	likely	to	be	further	convinced.	When	the
goal	was	to	defuse	a	conflict	between	two	nations,	for	example,	historical
documentaries	would	be	independently	released	in	both	countries	that	cast	the
origins	and	conduct	of	the	conflict	in	more	nuanced	light.	Pedagogical	news

stories	would	explain	who	on	their	own	side	stood	to	benefit	from	continued
conflict	and	their	techniques	for	stoking	it.	At	the	same	time,	likable	characters
from	the	other	nation	would	start	appearing	in	popular	shows	on	the
entertainment	channels,	just	as	sympathetically	portrayed	minority	characters
had	bolstered	the	civil	and	gay	rights	movements	in	the	past.
Before	long,	political	commentators	couldn’t	help	but	notice	growing	support
for	a	political	agenda	centered	around	seven	slogans:
1.
Democracy

2.
Tax	cuts
3.
Government	social	service	cuts
4.
Military	spending	cuts
5.
Free	trade
6.
Open	borders
7.
Socially	responsible	companies
What	was	less	obvious	was	the	underlying	goal:	to	erode	all	previous	power
structures	in	the	world.	Items	2–6	eroded	state	power,	and	democratizing	the
world	gave	the	Omegas’	business	empire	more	influence	over	the	selection	of
political	leaders.	Socially	responsible	companies	further	weakened	state	power
by	taking	over	more	and	more	of	the	services	that	governments	had	(or	should
have)	provided.	The	traditional	business	elite	was	weakened	simply	because	it
couldn’t	compete	with	Prometheus-backed	companies	on	the	free	market	and
therefore	owned	an	ever-shrinking	share	of	the	world	economy.	Traditional
opinion	leaders,	from	political	parties	to	faith	groups,	lacked	the	persuasion
machinery	to	compete	with	the	Omegas’	media	empire.
As	with	any	sweeping	change,	there	were	winners	and	losers.	Although	there

was	a	palpable	new	sense	of	optimism	in	most	countries	as	education,	social
services	and	infrastructure	improved,	conflicts	subsided	and	local	companies
released	breakthrough	technologies	that	swept	the	world,	not	everybody	was
happy.	While	many	displaced	workers	got	rehired	for	community	projects,	those
who’d	held	great	power	and	wealth	generally	saw	both	shrink.	This	began	in	the
media	and	technology	sectors,	but	it	spread	virtually	everywhere.	The	reduction

in	world	conflicts	led	to	defense	budget	cuts	that	hurt	military	contractors.
Burgeoning	upstart	companies	typically	weren’t	publicly	traded,	with	the
justification	that	profit-maximizing	shareholders	would	block	their	massive
spending	on	community	projects.	Thus	the	global	stock	market	kept	losing
value,	threatening	both	finance	tycoons	and	regular	citizens	who’d	counted	on
their	pension	funds.	As	if	the	shrinking	profits	of	publicly	traded	companies
weren’t	bad	enough,	investment	firms	around	the	world	had	noticed	a	disturbing

trend:	all	their	previously	successful	trading	algorithms	seemed	to	have	stopped
working,	underperforming	even	simple	index	funds.	Someone	else	out	there
always	seemed	to	outsmart	them	and	beat	them	at	their	own	game.
Although	masses	of	powerful	people	resisted	the	wave	of	change,	their

response	was	strikingly	ineffective,	almost	as	if	they	had	fallen	into	a	well-
planned	trap.	Huge	changes	were	happening	at	such	a	bewildering	pace	that	it
was	hard	to	keep	track	and	develop	a	coordinated	response.	Moreover,	it	was
highly	unclear	what	they	should	push	for.	The	traditional	political	right	had	seen
most	of	their	slogans	co-opted,	yet	the	tax	cuts	and	improved	business	climate
were	mostly	helping	their	higher-tech	competitors.	Virtually	every	traditional
industry	was	now	clamoring	for	a	bailout,	but	limited	government	funds	pitted
them	in	a	hopeless	battle	against	one	another	while	the	media	portrayed	them	as
dinosaurs	seeking	state	subsidies	simply	because	they	couldn’t	compete.	The
traditional	political	left	opposed	the	free	trade	and	the	cuts	in	government	social
services,	but	delighted	in	the	military	cutbacks	and	the	reduction	of	poverty.
Indeed,	much	of	their	thunder	was	stolen	by	the	undeniable	fact	that	social
services	had	improved	now	that	they	were	provided	by	idealistic	companies

rather	than	the	state.	Poll	after	poll	showed	that	most	voters	around	the	world	felt
their	quality	of	life	improving,	and	that	things	were	generally	moving	in	a	good
direction.	This	had	a	simple	mathematical	explanation:	before	Prometheus,	the
poorest	50%	of	Earth’s	population	had	earned	only	about	4%	of	the	global
income,	enabling	the	Omega-controlled	companies	to	win	their	hearts	(and
votes)	by	sharing	only	a	modest	fraction	of	their	profits	with	them.

Consolidation
As	a	result,	nation	after	nation	saw	landslide	election	victories	for	parties
embracing	the	seven	Omega	slogans.	In	carefully	optimized	campaigns,	they
portrayed	themselves	at	the	center	of	the	political	spectrum,	denouncing	the	right
as	greedy	bailout-seeking	conflict-mongers	and	lambasting	the	left	as	big-
government	tax-and-spend	innovation	stiflers.	What	almost	nobody	realized	was
that	Prometheus	had	carefully	selected	the	optimal	people	to	groom	as
candidates,	and	pulled	all	its	strings	to	secure	their	victory.
Before	Prometheus,	there	had	been	growing	support	for	the	universal	basic
income	movement,	which	proposed	tax-funded	minimum	income	for	everyone
as	a	remedy	for	technological	unemployment.	This	movement	imploded	when
the	corporate	community	projects	took	off,	since	the	Omega-controlled	business
empire	was	in	effect	providing	the	same	thing.	With	the	excuse	of	improving
coordination	of	their	community	projects,	an	international	group	of	companies

launched	the	Humanitarian	Alliance,	a	nongovernmental	organization	aiming	to
identify	and	fund	the	most	valuable	humanitarian	efforts	worldwide.	Before
long,	virtually	the	entire	Omega	empire	supported	it,	and	it	launched	global
projects	on	an	unprecedented	scale,	even	in	countries	that	had	largely	missed	out
on	the	tech	boom,	improving	education,	health,	prosperity	and	governance.
Needless	to	say,	Prometheus	provided	carefully	crafted	project	plans	behind	the
scenes,	ranked	by	positive	impact	per	dollar.	Rather	than	simply	dole	out	cash,	as

under	basic-income	proposals,	the	Alliance	(as	it	colloquially	became	known)
would	engage	those	it	supported	to	work	toward	its	cause.	As	a	result,	a	large
fraction	of	the	world’s	population	ended	up	feeling	grateful	and	loyal	to	the
Alliance—often	more	than	to	their	own	government.
As	time	passed,	the	Alliance	increasingly	assumed	the	role	of	a	world
government,	as	national	governments	saw	their	power	continually	erode.
National	budgets	kept	shrinking	due	to	tax	cuts	while	the	Alliance	budget	grew

to	dwarf	those	of	all	governments	combined.	All	the	traditional	roles	of	national
governments	became	increasingly	redundant	and	irrelevant.	The	Alliance
provided	by	far	the	best	social	services,	education	and	infrastructure.	Media	had
defused	international	conflict	to	the	point	that	military	spending	was	largely
unnecessary,	and	growing	prosperity	had	eliminated	most	roots	of	old	conflicts,

which	traced	back	to	competition	over	scarce	resources.	A	few	dictators	and
others	had	violently	resisted	this	new	world	order	and	refused	to	be	bought,	but
they	were	all	toppled	in	carefully	orchestrated	coups	or	mass	uprisings.
The	Omegas	had	now	completed	the	most	dramatic	transition	in	the	history	of
life	on	Earth.	For	the	first	time	ever,	our	planet	was	run	by	a	single	power,
amplified	by	an	intelligence	so	vast	that	it	could	potentially	enable	life	to
flourish	for	billions	of	years	on	Earth	and	throughout	our	cosmos—but	what
specifically	was	their	plan?
*	*	*
That	was	the	tale	of	the	Omega	team.	The	rest	of	this	book	is	about	another	tale
—
one	that’s	not	yet	written:	the	tale	of	our	own	future	with	AI.	How	would	you
like
	
it	to	play	out?	Could	something	remotely	like	the	Omega	story	actually
occur	and,	if
	
so,	would	you	want	it	to?	Leaving	aside	speculations	about
superhuman	AI,	how
	
would	you	like	our	tale	to	begin?	How	do	you	want	AI	to

*	*	*
—
like
	
impact	jobs,	laws	and	weapons	in	the	coming	decade?	Looking	further	ahead,
how	would	you	write	the
	
ending?	This	tale	is	one	of	truly	cosmic	proportions,	for
it	involves	nothing
	
short	of	the	ultimate	future	of	life	in	our	Universe.	And	it’s	a
tale	for	us	to
	
write.
*
	
For	simplicity,	I’ve	assumed	today’s	economy	and	technology	in	this	story,	even	though	most	researchers
guess	that	human-level	general	AI	is	at	least	decades	away.	The	Omega	plan	should	get	even	easier	to

write.
*
	
pull	off	in	the	future	if	the	digital	economy	keeps	growing	and	ever	more	services	can	be	ordered	online
on	a	no-questions-asked	basis.

Chapter	1
Welcome	to	the	Most	Important	Conversation	of
Our	Time
Technology	is	giving	life	the	potential	to	flourish	like	never	before—or	to
	
self-destruct.
Future	of	Life	Institute
Thirteen	point	eight	billion	years	after	its	birth,	our	Universe	has	awoken	and
become	aware	of	itself.	From	a	small	blue	planet,	tiny	conscious	parts	of	our
Universe	have	begun	gazing	out	into	the	cosmos	with	telescopes,	repeatedly
discovering	that	everything	they	thought	existed	is	merely	a	small	part	of
something	grander:	a	solar	system,	a	galaxy	and	a	universe	with	over	a	hundred
billion	other	galaxies	arranged	into	an	elaborate	pattern	of	groups,	clusters	and
superclusters.	Although	these	self-aware	stargazers	disagree	on	many	things,
they	tend	to	agree	that	these	galaxies	are	beautiful	and	awe-inspiring.
But	beauty	is	in	the	eye	of	the	beholder,	not	in	the	laws	of	physics,	so	before
our	Universe	awoke,	there	was	no	beauty.	This	makes	our	cosmic	awakening	all

the	more	wonderful	and	worthy	of	celebrating:	it	transformed	our	Universe	from
a	mindless	zombie	with	no	self-awareness	into	a	living	ecosystem	harboring
self-reflection,	beauty	and	hope—and	the	pursuit	of	goals,	meaning	and	purpose.
Had	our	Universe	never	awoken,	then,	as	far	as	I’m	concerned,	it	would	have
been	completely	pointless—merely	a	gigantic	waste	of	space.	Should	our
Universe	permanently	go	back	to	sleep	due	to	some	cosmic	calamity	or	self-
inflicted	mishap,	it	will,	alas,	become	meaningless.

On	the	other	hand,	things	could	get	even	better.	We	don’t	yet	know	whether

we	humans	are	the	only	stargazers	in	our	cosmos,	or	even	the	first,	but	we’ve
already	learned	enough	about	our	Universe	to	know	that	it	has	the	potential	to
wake	up	much	more	fully	than	it	has	thus	far.	Perhaps	we’re	like	that	first	faint
glimmer	of	self-awareness	you	experienced	when	you	began	emerging	from
sleep	this	morning:	a	premonition	of	the	much	greater	consciousness	that	would
arrive	once	you	opened	your	eyes	and	fully	woke	up.	Perhaps	life	will	spread
throughout	our	cosmos	and	flourish	for	billions	or	trillions	of	years—and
perhaps	this	will	be	because	of	decisions	that	we	make	here	on	our	little	planet
during	our	lifetime.

So	how	did	this	amazing	awakening	come	about?	It	wasn’t	an	isolated	event,	but
merely	one	step	in	a	relentless	13.8-billion-year	process	that’s	making	our
Universe	ever	more	complex	and	interesting—and	is	continuing	at	an
accelerating	pace.
As	a	physicist,	I	feel	fortunate	to	have	gotten	to	spend	much	of	the	past
quarter	century	helping	to	pin	down	our	cosmic	history,	and	it’s	been	an	amazing
journey	of	discovery.	Since	the	days	when	I	was	a	graduate	student,	we’ve	gone
from	arguing	about	whether	our	Universe	is	10	or	20	billion	years	old	to	arguing
about	whether	it’s	13.7	or	13.8	billion	years	old,	thanks	to	a	combination	of
better	telescopes,	better	computers	and	better	understanding.	We	physicists	still
don’t	know	for	sure	what	caused	our	Big	Bang	or	whether	this	was	truly	the
beginning	of	everything	or	merely	the	sequel	to	an	earlier	stage.	However,	we’ve
acquired	a	rather	detailed	understanding	of	what’s	happened	
since
	our	Big	Bang,

since
thanks	to	an	avalanche	of	high-quality	measurements,	so	please	let	me	take	a	few
minutes	to	summarize	13.8	billion	years	of	cosmic	history.
In	the	beginning,	there	was	light.	In	the	first	split	second	after	our	Big	Bang,
the	entire	part	of	space	that	our	telescopes	can	in	principle	observe	(“our
observable	Universe,”	or	simply	“our	Universe”	for	short)	was	much	hotter	and
brighter	than	the	core	of	our	Sun	and	it	expanded	rapidly.	Although	this	may
sound	spectacular,	it	was	also	dull	in	the	sense	that	our	Universe	contained

nothing	but	a	lifeless,	dense,	hot	and	boringly	uniform	soup	of	elementary
particles.	Things	looked	pretty	much	the	same	everywhere,	and	the	only
interesting	structure	consisted	of	faint	random-looking	sound	waves	that	made
the	soup	about	0.001%	denser	in	some	places.	These	faint	waves	are	widely
believed	to	have	originated	as	so-called	quantum	fluctuations,	because
Heisenberg’s	uncertainty	principle	of	quantum	mechanics	forbids	anything	from
being	completely	boring	and	uniform.
As	our	Universe	expanded	and	cooled,	it	grew	more	interesting	as	its	particles

combined	into	ever	more	complex	objects.	During	the	first	split	second,	the
strong	nuclear	force	grouped	quarks	into	protons	(hydrogen	nuclei)	and
neutrons,	some	of	which	in	turn	fused	into	helium	nuclei	within	a	few	minutes.
About	400,000	years	later,	the	electromagnetic	force	grouped	these	nuclei	with

electrons	to	make	the	first	atoms.	As	our	Universe	kept	expanding,	these	atoms
gradually	cooled	into	a	cold	dark	gas,	and	the	darkness	of	this	first	night	lasted
for	about	100	million	years.	This	long	night	gave	rise	to	our	cosmic	dawn	when
the	gravitational	force	succeeded	in	amplifying	those	fluctuations	in	the	gas,
pulling	atoms	together	to	form	the	first	stars	and	galaxies.	These	first	stars
generated	heat	and	light	by	fusing	hydrogen	into	heavier	atoms	such	as	carbon,
oxygen	and	silicon.	When	these	stars	died,	many	of	the	atoms	they’d	created
were	recycled	into	the	cosmos	and	formed	planets	around	second-generation
stars.
At	some	point,	a	group	of	atoms	became	arranged	into	a	complex	pattern	that
could	both	maintain	and	replicate	itself.	So	soon	there	were	two	copies,	and	the
number	kept	doubling.	It	takes	only	forty	doublings	to	make	a	trillion,	so	this
first	self-replicator	soon	became	a	force	to	be	reckoned	with.	Life	had	arrived.

The	question	of	how	to	define	life	is	notoriously	controversial.	Competing
definitions	abound,	some	of	which	include	highly	specific	requirements	such	as
being	composed	of	cells,	which	might	disqualify	both	future	intelligent	machines
and	extraterrestrial	civilizations.	Since	we	don’t	want	to	limit	our	thinking	about
the	future	of	life	to	the	species	we’ve	encountered	so	far,	let’s	instead	define	life
very	broadly,	simply	as	a	process	that	can	retain	its	complexity	and	replicate.
What’s	replicated	isn’t	matter	(made	of	atoms)	but	information	(made	of	bits)
specifying	how	the	atoms	are	arranged.	When	a	bacterium	makes	a	copy	of	its
DNA,	no	new	atoms	are	created,	but	a	new	set	of	atoms	are	arranged	in	the	same
pattern	as	the	original,	thereby	copying	the	information.	In	other	words,	we	can
think	of	life	as	a	self-replicating	information-processing	system	whose
information	(software)	determines	both	its	behavior	and	the	blueprints	for	its
hardware.

hardware.
Like	our	Universe	itself,	life	gradually	grew	more	complex	and	interesting,
*1
and	as	I’ll	now	explain,	I	find	it	helpful	to	classify	life	forms	into	three	levels	of
sophistication:	Life	1.0,	2.0	and	3.0.	I’ve	summarized	these	three	levels	in	
figure
1.1
.
It’s	still	an	open	question	how,	when	and	where	life	first	appeared	in	our
Universe,	but	there	is	strong	evidence	that	here	on	Earth	life	first	appeared	about
4	billion	years	ago.	Before	long,	our	planet	was	teeming	with	a	diverse	panoply

hardware.
*1
figure
1.1
.
of	life	forms.	The	most	successful	ones,	which	soon	outcompeted	the	rest,	were
able	to	react	to	their	environment	in	some	way.	Specifically,	they	were	what
computer	scientists	call	“intelligent	agents”:	entities	that	collect	information
about	their	environment	from	sensors	and	then	process	this	information	to	decide
how	to	act	back	on	their	environment.	This	can	include	highly	complex
information	processing,	such	as	when	you	use	information	from	your	eyes	and

ears	to	decide	what	to	say	in	a	conversation.	But	it	can	also	involve	hardware
and	software	that’s	quite	simple.
For	example,	many	bacteria	have	a	sensor	measuring	the	sugar	concentration
in	the	liquid	around	them	and	can	swim	using	propeller-shaped	structures	called
flagella.	The	hardware	linking	the	sensor	to	the	flagella	might	implement	the
following	simple	but	useful	algorithm:	“If	my	sugar	concentration	sensor	reports

a	lower	value	than	a	couple	of	seconds	ago,	then	reverse	the	rotation	of	my
flagella	so	that	I	change	direction.”

Figure	1.1:	The	three	stages	of	life:	biological	evolution,	cultural	evolution	and	technological
evolution.	Life	1.0	is	unable	to	redesign	either	its	hardware	or	its	software	during	its	lifetime:
both	are	determined	by	its	DNA,	and	change	only	through	evolution	over	many	generations.	In
contrast,	Life	2.0	can	redesign	much	of	its	software:	humans	can	learn	complex	new	skills—for
example,	languages,	sports	and	professions—and	can	fundamentally	update	their	worldview	and
goals.	Life	3.0,	which	doesn’t	yet	exist	on	Earth,	can	dramatically	redesign	not	only	its	software,
but	its	hardware	as	well,	rather	than	having	to	wait	for	it	to	gradually	evolve	over	generations.
You’ve	learned	how	to	speak	and	countless	other	skills.	Bacteria,	on	the	other
hand,	aren’t	great	learners.	Their	DNA	specifies	not	only	the	design	of	their
hardware,	such	as	sugar	sensors	and	flagella,	but	also	the	design	of	their

software.	They	never	learn	to	swim	toward	sugar;	instead,	that	algorithm	was
hard-coded	into	their	DNA	from	the	start.	There	was	of	course	a	learning	process
of	sorts,	but	it	didn’t	take	place	during	the	lifetime	of	that	particular	bacterium.
Rather,	it	occurred	during	the	preceding	evolution	of	that	species	of	bacteria,
through	a	slow	trial-and-error	process	spanning	many	generations,	where	natural
selection	favored	those	random	DNA	mutations	that	improved	sugar
consumption.	Some	of	these	mutations	helped	by	improving	the	design	of
flagella	and	other	hardware,	while	other	mutations	improved	the	bacterial
information-processing	system	that	implements	the	sugar-finding	algorithm	and
other	software.
Such	bacteria	are	an	example	of	what	I’ll	call	“Life	1.0”:	
life	where	both	the
hardware	and	software	are	evolved	rather	than	designed.
	You	and	I,	on	the	other
hand,	are	examples	of	“Life	2.0”:	
life	whose	hardware	is	evolved,	but	whose
software	is	largely	designed.
	By	your	software,	I	mean	all	the	algorithms	and

knowledge	that	you	use	to	process	the	information	from	your	senses	and	decide
what	to	do—everything	from	the	ability	to	recognize	your	friends	when	you	see
them	to	your	ability	to	walk,	read,	write,	calculate,	sing	and	tell	jokes.
You	weren’t	able	to	perform	any	of	those	tasks	when	you	were	born,	so	all	this
software	got	programmed	into	your	brain	later	through	the	process	we	call
learning.	Whereas	your	childhood	curriculum	is	largely	designed	by	your	family

By	your	software,	I	mean	all	the	algorithms	and
and	teachers,	who	decide	what	you	should	learn,	you	gradually	gain	more	power
to	design	your	own	software.	Perhaps	your	school	allows	you	to	select	a	foreign
language:	Do	you	want	to	install	a	software	module	into	your	brain	that	enables
you	to	speak	French,	or	one	that	enables	you	to	speak	Spanish?	Do	you	want	to
learn	to	play	tennis	or	chess?	Do	you	want	to	study	to	become	a	chef,	a	lawyer	or
a	pharmacist?	Do	you	want	to	learn	more	about	artificial	intelligence	(AI)	and

the	future	of	life	by	reading	a	book	about	it?
This	ability	of	Life	2.0	to	design	its	software	enables	it	to	be	much	smarter
than	Life	1.0.	High	intelligence	requires	both	lots	of	hardware	(made	of	atoms)
and	lots	of	software	(made	of	bits).	The	fact	that	most	of	our	human	hardware	is
added	after	birth	(through	growth)	is	useful,	since	our	ultimate	size	isn’t	limited
by	the	width	of	our	mom’s	birth	canal.	In	the	same	way,	the	fact	that	most	of	our
human	software	is	added	after	birth	(through	learning)	is	useful,	since	our

ultimate	intelligence	isn’t	limited	by	how	much	information	can	be	transmitted
to	us	at	conception	via	our	DNA,	1.0-style.	I	weigh	about	twenty-five	times
more	than	when	I	was	born,	and	the	synaptic	connections	that	link	the	neurons	in
my	brain	can	store	about	a	hundred	thousand	times	more	information	than	the

DNA	that	I	was	born	with.	Your	synapses	store	all	your	knowledge	and	skills	as
roughly	100	terabytes’	worth	of	information,	while	your	DNA	stores	merely
about	a	gigabyte,	barely	enough	to	store	a	single	movie	download.	So	it’s
physically	impossible	for	an	infant	to	be	born	speaking	perfect	English	and	ready
to	ace	her	college	entrance	exams:	there’s	no	way	the	information	could	have
been	preloaded	into	her	brain,	since	the	main	information	module	she	got	from
her	parents	(her	DNA)	lacks	sufficient	information-storage	capacity.
The	ability	to	design	its	software	enables	Life	2.0	to	be	not	only	smarter	than
Life	1.0,	but	also	more	flexible.	If	the	environment	changes,	1.0	can	only	adapt
by	slowly	evolving	over	many	generations.	Life	2.0,	on	the	other	hand,	can	adapt
almost	instantly,	via	a	software	update.	For	example,	bacteria	frequently
encountering	antibiotics	may	evolve	drug	resistance	over	many	generations,	but
an	individual	bacterium	won’t	change	its	behavior	at	all;	in	contrast,	a	girl

learning	that	she	has	a	peanut	allergy	will	immediately	change	her	behavior	to
start	avoiding	peanuts.	This	flexibility	gives	Life	2.0	an	even	greater	edge	at	the
population	level:	even	though	the	information	in	our	human	DNA	hasn’t	evolved
dramatically	over	the	past	fifty	thousand	years,	the	information	collectively
stored	in	our	brains,	books	and	computers	has	exploded.	By	installing	a	software
module	enabling	us	to	communicate	through	sophisticated	spoken	language,	we

ensured	that	the	most	useful	information	stored	in	one	person’s	brain	could	get
copied	to	other	brains,	potentially	surviving	even	after	the	original	brain	died.	By
installing	a	software	module	enabling	us	to	read	and	write,	we	became	able	to
store	and	share	vastly	more	information	than	people	could	memorize.	By
developing	brain	software	capable	of	producing	technology	(i.e.,	by	studying
science	and	engineering),	we	enabled	much	of	the	world’s	information	to	be
accessed	by	many	of	the	world’s	humans	with	just	a	few	clicks.

This	flexibility	has	enabled	Life	2.0	to	dominate	Earth.	Freed	from	its	genetic
shackles,	humanity’s	combined	knowledge	has	kept	growing	at	an	accelerating
pace	as	each	breakthrough	enabled	the	next:	language,	writing,	the	printing
press,	modern	science,	computers,	the	internet,	etc.	This	ever-faster	cultural
evolution	of	our	shared	software	has	emerged	as	the	dominant	force	shaping	our
human	future,	rendering	our	glacially	slow	biological	evolution	almost
irrelevant.
Yet	despite	the	most	powerful	technologies	we	have	today,	all	life	forms	we

know	of	remain	fundamentally	limited	by	their	biological	hardware.	None	can
live	for	a	million	years,	memorize	all	of	Wikipedia,	understand	all	known
science	or	enjoy	spaceflight	without	a	spacecraft.	None	can	transform	our	largely

lifeless	cosmos	into	a	diverse	biosphere	that	will	flourish	for	billions	or	trillions
of	years,	enabling	our	Universe	to	finally	fulfill	its	potential	and	wake	up	fully.
All	this	requires	life	to	undergo	a	final	upgrade,	to	Life	3.0,	which	can	design	not
only	its	software	but	also	its	hardware.	In	other	words,	Life	3.0	is	the	master	of
its	own	destiny,	finally	fully	free	from	its	evolutionary	shackles.
The	boundaries	between	the	three	stages	of	life	are	slightly	fuzzy.	If	bacteria
are	Life	1.0	and	humans	are	Life	2.0,	then	you	might	classify	mice	as	1.1:	they
can	learn	many	things,	but	not	enough	to	develop	language	or	invent	the	internet.
Moreover,	because	they	lack	language,	what	they	learn	gets	largely	lost	when
they	die,	not	passed	on	to	the	next	generation.	Similarly,	you	might	argue	that
today’s	humans	should	count	as	Life	2.1:	we	can	perform	minor	hardware
upgrades	such	as	implanting	artificial	teeth,	knees	and	pacemakers,	but	nothing

as	dramatic	as	getting	ten	times	taller	or	acquiring	a	thousand	times	bigger	brain.
In	summary,	we	can	divide	the	development	of	life	into	three	stages,
distinguished	by	life’s	ability	to	design	itself:
•
Life	1.0	(biological	stage):	evolves	its	hardware	and	software
•
Life	2.0	(cultural	stage):	evolves	its	hardware,	designs	much	of	its
software
•
Life	3.0	(technological	stage):	designs	its	hardware	and	software
After	13.8	billion	years	of	cosmic	evolution,	development	has	accelerated

•
•
software
•
dramatically	here	on	Earth:	Life	1.0	arrived	about	4	billion	years	ago,	Life	2.0
(we	humans)	arrived	about	a	hundred	millennia	ago,	and	many	AI	researchers
think	that	Life	3.0	may	arrive	during	the	coming	century,	perhaps	even	during
our	lifetime,	spawned	by	progress	in	AI.	What	will	happen,	and	what	will	this
mean	for	us?	That’s	the	topic	of	this	book.

This	question	is	wonderfully	controversial,	with	the	world’s	leading	AI
researchers	disagreeing	passionately	not	only	in	their	forecasts,	but	also	in	their
emotional	reactions,	which	range	from	confident	optimism	to	serious	concern.
They	don’t	even	have	consensus	on	short-term	questions	about	AI’s	economic,
legal	and	military	impact,	and	their	disagreements	grow	when	we	expand	the
time	horizon	and	ask	about	
artificial	general	intelligence
	(AGI)—especially
about	AGI	reaching	human	level	and	beyond,	enabling	Life	3.0.	
General
intelligence
	can	accomplish	virtually	any	goal,	including	learning,	in	contrast	to,
say,	the	narrow	intelligence	of	a	chess-playing	program.
Interestingly,	the	controversy	about	Life	3.0	centers	around	not	one	but	two
separate	questions:	when	and	what?	When	(if	ever)	will	it	happen,	and	what	will
it	mean	for	humanity?	The	way	I	see	it,	there	are	three	distinct	schools	of
thought	that	all	need	to	be	taken	seriously,	because	they	each	include	a	number

General
of	world-leading	experts.	As	illustrated	in	
figure	1.2
,	I	think	of	them	as	
digital
utopians
,	
techno-skeptics	
and	
members	of	the	beneficial-AI	movement
,
respectively.	Please	let	me	introduce	you	to	some	of	their	most	eloquent
champions.

Digital	Utopians
When	I	was	a	kid,	I	imagined	that	billionaires	exuded	pomposity	and	arrogance.
When	I	first	met	Larry	Page	at	Google	in	2008,	he	totally	shattered	these
stereotypes.	Casually	dressed	in	jeans	and	a	remarkably	ordinary-looking	shirt,
he	would	have	blended	right	in	at	an	MIT	picnic.	His	thoughtful	soft-spoken
style	and	his	friendly	smile	made	me	feel	relaxed	rather	than	intimidated	talking
with	him.	On	July	18,	2015,	we	ran	into	each	other	at	a	party	in	Napa	Valley
thrown	by	Elon	Musk	and	his	then	wife,	Talulah,	and	got	into	a	conversation
about	the	scatological	interests	of	our	kids.	I	recommended	the	profound	literary
classic	
The	Day	My
	
Butt	Went	Psycho,
	by	Andy	Griffiths,	and	Larry	ordered	it
on	the	spot.	I	struggled	to	remind	myself	that	he	might	go	down	in	history	as	the
most	influential	human	ever	to	have	lived:	my	guess	is	that	if	superintelligent
digital	life	engulfs	our	Universe	in	my	lifetime,	it	will	be	because	of	Larry’s
decisions.

Figure	1.2:	Most	controversies	surrounding	strong	artificial	intelligence	(that	can	match	humans
on	any	cognitive	task)	center	around	two	questions:	When	(if	ever)	will	it	happen,	and	will	it	be	a
good	thing	for	humanity?	Techno-skeptics	and	digital	utopians	agree	that	we	shouldn’t	worry,	but
for	very	different	reasons:	the	former	are	convinced	that	human-level	artificial	general
intelligence	(AGI)	won’t	happen	in	the	foreseeable	future,	while	the	latter	think	it	will	happen
but	is	virtually	guaranteed	to	be	a	good	thing.	The	beneficial-AI	movement	feels	that	concern	is
warranted	and	useful,	because	AI-safety	research	and	discussion	now	increases	the	chances	of	a
good	outcome.	Luddites	are	convinced	of	a	bad	outcome	and	oppose	AI.	This	figure	is	partly
inspired	by	Tim	Urban.
1
With	our	wives,	Lucy	and	Meia,	we	ended	up	having	dinner	together	and
discussing	whether	machines	would	necessarily	be	conscious,	an	issue	that	he

argued	was	a	red	herring.	Later	that	night,	after	cocktails,	a	long	and	spirited
debate	ensued	between	him	and	Elon	about	the	future	of	AI	and	what	should	be
done.	As	we	entered	the	wee	hours	of	the	morning,	the	circle	of	bystanders	and
kibitzers	kept	growing.	Larry	gave	a	passionate	defense	of	the	position	I	like	to
think	of	as	
digital	utopianism:
	that	digital	life	is	the	natural	and	desirable	next
step	in	the	cosmic	evolution	and	that	if	we	let	digital	minds	be	free	rather	than
try	to	stop	or	enslave	them,	the	outcome	is	almost	certain	to	be	good.	I	view
Larry	as	the	most	influential	exponent	of	digital	utopianism.	He	argued	that	if
life	is	ever	going	to	spread	throughout	our	Galaxy	and	beyond,	which	he	thought
it	should,	then	it	would	need	to	do	so	in	digital	form.	His	main	concerns	were
that	AI	paranoia	would	delay	the	digital	utopia	and/or	cause	a	military	takeover
of	AI	that	would	fall	foul	of	Google’s	“Don’t	be	evil”	slogan.	Elon	kept	pushing

back	and	asking	Larry	to	clarify	details	of	his	arguments,	such	as	why	he	was	so
confident	that	digital	life	wouldn’t	destroy	everything	we	care	about.	At	times,
Larry	accused	Elon	of	being	“specieist”:	treating	certain	life	forms	as	inferior
just	because	they	were	silicon-based	rather	than	carbon-based.	We’ll	return	to
explore	these	interesting	issues	and	arguments	in	detail,	starting	in	chapter	4.
Although	Larry	seemed	outnumbered	that	warm	summer	night	by	the	pool,

the	digital	utopianism	that	he	so	eloquently	championed	has	many	prominent
supporters.	Roboticist	and	futurist	Hans	Moravec	inspired	a	whole	generation	of
digital	utopians	with	his	classic	1988	book	
Mind	Children,
	a	tradition	continued
and	refined	by	inventor	Ray	Kurzweil.	Richard	Sutton,	one	of	the	pioneers	of	the
AI	subfield	known	as	reinforcement	learning,	gave	a	passionate	defense	of
digital	utopianism	at	our	Puerto	Rico	conference	that	I’ll	tell	you	about	shortly.

Techno-skeptics
Another	prominent	group	of	thinkers	aren’t	worried	about	AI	either,	but	for	a
completely	different	reason:	they	think	that	building	superhuman	AGI	is	so	hard
that	it	won’t	happen	for	hundreds	of	years,	and	therefore	view	it	as	silly	to	worry
about	it	now.	I	think	of	this	as	the	
techno-skeptic
	position,	eloquently	articulated
by	Andrew	Ng:	“Fearing	a	rise	of	killer	robots	is	like	worrying	about
overpopulation	on	Mars.”	Andrew	was	the	chief	scientist	at	Baidu,	China’s
Google,	and	he	recently	repeated	this	argument	when	I	spoke	with	him	at	a
conference	in	Boston.	He	also	told	me	that	he	felt	that	worrying	about	AI	risk
was	a	potentially	harmful	distraction	that	could	slow	the	progress	of	AI.	Similar
sentiments	have	been	articulated	by	other	techno-skeptics	such	as	Rodney
Brooks,	the	former	MIT	professor	behind	the	Roomba	robotic	vacuum	cleaner
and	the	Baxter	industrial	robot.	I	find	it	interesting	that	although	the	digital

utopians	and	the	techno-skeptics	agree	that	we	shouldn’t	worry	about	AI,	they
agree	on	little	else.	Most	of	the	utopians	think	human-level	AGI	might	happen
within	the	next	twenty	to	a	hundred	years,	which	the	techno-skeptics	dismiss	as
uninformed	pie-in-the-sky	dreaming,	often	deriding	the	prophesied	singularity	as
“the	rapture	of	the	geeks.”	When	I	met	Rodney	Brooks	at	a	birthday	party	in
December	2014,	he	told	me	that	he	was	100%	sure	it	wouldn’t	happen	in	my
lifetime.	“Are	you	sure	you	don’t	mean	99%?,”	I	asked	in	a	follow-up	email,	to

which	he	replied,	“No	wimpy	99%.	100%.	Just	isn’t	going	to	happen.”

The	Beneficial-AI	Movement
When	I	first	met	Stuart	Russell	in	a	Paris	café	in	June	2014,	he	struck	me	as	the
quintessential	British	gentleman.	Eloquent,	thoughtful	and	soft-spoken,	but	with
an	adventurous	glint	in	his	eyes,	he	seemed	to	me	a	modern	incarnation	of
Phileas	Fogg,	my	childhood	hero	from	Jules	Verne’s	classic	1873	novel,	
Around
the	World	in
	
80	Days.
	Although	he	was	one	of	the	most	famous	AI	researchers
alive,	having	co-authored	the	standard	textbook	on	the	subject,	his	modesty	and
warmth	soon	put	me	at	ease.	He	explained	to	me	how	progress	in	AI	had
persuaded	him	that	human-level	AGI	this	century	was	a	real	possibility	and,
although	he	was	hopeful,	a	good	outcome	wasn’t	guaranteed.	There	were	crucial
questions	that	we	needed	to	answer	first,	and	they	were	so	hard	that	we	should
start	researching	them	now,	so	that	we’d	have	the	answers	ready	by	the	time	we
needed	them.
Today,	Stuart’s	views	are	rather	mainstream,	and	many	groups	around	the

world	are	pursuing	the	sort	of	AI-safety	research	that	he	advocates.	But	this
wasn’t	always	the	case.	An	article	in	
The
	
Washington	Post
	referred	to	2015	as
the	year	that	AI-safety	research	went	mainstream.	Before	that,	talk	of	AI	risks
was	often	misunderstood	by	mainstream	AI	researchers	and	dismissed	as
Luddite	scaremongering	aimed	at	impeding	AI	progress.	As	we’ll	explore	in
chapter	5,	concerns	similar	to	Stuart’s	were	first	articulated	over	half	a	century
ago	by	computer	pioneer	Alan	Turing	and	mathematician	Irving	J.	Good,	who

The
	
worked	with	Turing	to	crack	German	codes	during	World	War	II.	In	the	past
decade,	research	on	such	topics	was	mainly	carried	out	by	a	handful	of
independent	thinkers	who	weren’t	professional	AI	researchers,	for	example
Eliezer	Yudkowsky,	Michael	Vassar	and	Nick	Bostrom.	Their	work	had	little
effect	on	most	mainstream	AI	researchers,	who	tended	to	focus	on	their	day-to-
day	tasks	of	making	AI	systems	more	intelligent	rather	than	on	contemplating
the	long-term	consequences	of	success.	Of	the	AI	researchers	I	knew	who	did

harbor	some	concern,	many	hesitated	to	voice	it	out	of	fear	of	being	perceived	as
alarmist	technophobes.
I	felt	that	this	polarized	situation	needed	to	change,	so	that	the	full	AI
community	could	join	and	influence	the	conversation	about	how	to	build
beneficial	AI.	Fortunately,	I	wasn’t	alone.	In	the	spring	of	2014,	I’d	founded	a
nonprofit	organization	called	the	Future	of	Life	Institute	(FLI;

http://futureoflife.org
)	together	with	my	wife,	Meia,	my	physicist	friend	Anthony
Aguirre,	Harvard	grad	student	Viktoriya	Krakovna	and	Skype	founder	Jaan
Tallinn.	Our	goal	was	simple:	to	help	ensure	that	the	future	of	life	existed	and
would	be	as	awesome	as	possible.	Specifically,	we	felt	that	technology	was
giving	life	the	power	either	to	flourish	like	never	before	or	to	self-destruct,	and
we	preferred	the	former.
Our	first	meeting	was	a	brainstorming	session	at	our	house	on	March	15,
2014,	with	about	thirty	students,	professors	and	other	thinkers	from	the	Boston
area.	There	was	broad	consensus	that	although	we	should	pay	attention	to
biotech,	nuclear	weapons	and	climate	change,	our	first	major	goal	should	be	to
help	make	AI-safety	research	mainstream.	My	MIT	physics	colleague	Frank
Wilczek,	who	won	a	Nobel	Prize	for	helping	figure	out	how	quarks	work,
suggested	that	we	start	by	writing	an	op-ed	to	draw	attention	to	the	issue	and

make	it	harder	to	ignore.	I	reached	out	to	Stuart	Russell	(whom	I	hadn’t	yet	met)
and	to	my	physics	colleague	Stephen	Hawking,	both	of	whom	agreed	to	join	me
and	Frank	as	co-authors.	Many	edits	later,	our	op-ed	was	rejected	by	
The
	
New
York	Times
	and	many	other	U.S.	newspapers,	so	we	posted	it	on	my	
Huffington
Post
	blog	account.	To	my	delight,	Arianna	Huffington	herself	emailed	and	said,
“thrilled	to	have	it!	We’ll	post	at	#1!,”	and	this	placement	at	the	top	of	the	front
page	triggered	a	wave	of	media	coverage	of	AI	safety	that	lasted	for	the	rest	of

The
	
New
York	Times
Post
the	year,	with	Elon	Musk,	Bill	Gates	and	other	tech	leaders	chiming	in.	Nick
Bostrom’s	book	
Superintelligence
	came	out	that	fall	and	further	fueled	the
growing	public	debate.
The	next	goal	of	our	FLI	beneficial-AI	campaign	was	to	bring	the	world’s
leading	AI	researchers	to	a	conference	where	misunderstandings	could	be
cleared	up,	consensus	could	be	forged,	and	constructive	plans	could	be	made.
We	knew	that	it	would	be	difficult	to	persuade	such	an	illustrious	crowd	to	come

to	a	conference	organized	by	outsiders	they	didn’t	know,	especially	given	the
controversial	topic,	so	we	tried	as	hard	as	we	could:	we	banned	media	from
attending,	we	located	it	in	a	beach	resort	in	January	(in	Puerto	Rico),	we	made	it
free	(thanks	to	the	generosity	of	Jaan	Tallinn),	and	we	gave	it	the	most	non-
alarmist	title	we	could	come	up	with:	“The	Future	of	AI:	Opportunities	and
Challenges.”	Most	importantly,	we	teamed	up	with	Stuart	Russell,	thanks	to
whom	we	were	able	to	grow	the	organizing	committee	to	include	a	group	of	AI

leaders	from	both	academia	and	industry—including	Demis	Hassabis	from
Google’s	DeepMind,	who	went	on	to	show	that	AI	can	beat	humans	even	at	the
game	of	Go.	The	more	I	got	to	know	Demis,	the	more	I	realized	that	he	had

ambition	not	only	to	make	AI	powerful,	but	also	to	make	it	beneficial.
The	result	was	a	remarkable	meeting	of	minds	(
figure	1.3
).	The	AI	researchers
were	joined	by	top	economists,	legal	scholars,	tech	leaders	(including	Elon
Musk)	and	other	thinkers	(including	Vernor	Vinge,	who	coined	the	term
“singularity,”	which	is	the	focus	of	chapter	4).	The	outcome	surpassed	even	our
most	optimistic	expectations.	Perhaps	it	was	a	combination	of	the	sunshine	and
the	wine,	or	perhaps	it	was	just	that	the	time	was	right:	despite	the	controversial
topic,	a	remarkable	consensus	emerged,	which	we	codified	in	an	open	letter
2
	that
ended	up	getting	signed	by	over	eight	thousand	people,	including	a	veritable
who’s	who	in	AI.	The	gist	of	the	letter	was	that	the	goal	of	AI	should	be
redefined:	the	goal	should	be	to	create	not	undirected	intelligence,	but	beneficial
intelligence.	The	letter	also	mentioned	a	detailed	list	of	research	topics	that	the
conference	participants	agreed	would	further	this	goal.	The	beneficial-AI

2
	that
movement	had	started	going	mainstream.	We’ll	follow	its	subsequent	progress
later	in	the	book.

Figure	1.3:	The	January	2015	Puerto	Rico	conference	brought	together	a	remarkable	group	of
researchers	in	AI	and	related	fields.	Back	row,	from	left	to	right:	Tom	Mitchell,	Seán	Ó
hÉigeartaigh,	Huw	Price,	Shamil	Chandaria,	Jaan	Tallinn,	Stuart	Russell,	Bill	Hibbard,	Blaise
Agüera	y	Arcas,	Anders	Sandberg,	Daniel	Dewey,	Stuart	Armstrong,	Luke	Muehlhauser,	Tom
Dietterich,	Michael	Osborne,	James	Manyika,	Ajay	Agrawal,	Richard	Mallah,	Nancy	Chang,
Matthew	Putman.	Other	standing,	left	to	right:	Marilyn	Thompson,	Rich	Sutton,	Alex	Wissner-
Gross,	Sam	Teller,	Toby	Ord,	Joscha	Bach,	Katja	Grace,	Adrian	Weller,	Heather	Roff-Perkins,
Dileep	George,	Shane	Legg,	Demis	Hassabis,	Wendell	Wallach,	Charina	Choi,	Ilya	Sutskever,
Kent	Walker,	Cecilia	Tilli,	Nick	Bostrom,	Erik	Brynjolfsson,	Steve	Crossan,	Mustafa	Suleyman,
Scott	Phoenix,	Neil	Jacobstein,	Murray	Shanahan,	Robin	Hanson,	Francesca	Rossi,	Nate	Soares,
Elon	Musk,	Andrew	McAfee,	Bart	Selman,	Michele	Reilly,	Aaron	VanDevender,	Max	Tegmark,

Margaret	Boden,	Joshua	Greene,	Paul	Christiano,	Eliezer	Yudkowsky,	David	Parkes,	Laurent
Orseau,	JB	Straubel,	James	Moor,	Sean	Legassick,	Mason	Hartman,	Howie	Lempel,	David
Vladeck,	Jacob	Steinhardt,	Michael	Vassar,	Ryan	Calo,	Susan	Young,	Owain	Evans,	Riva-
Melissa	Tez,	János	Krámar,	Geoff	Anders,	Vernor	Vinge,	Anthony	Aguirre.	Seated:	Sam	Harris,
Tomaso	Poggio,	Marin	Solja
č
i
ć
,	Viktoriya	Krakovna,	Meia	Chita-Tegmark.	Behind	the	camera:
Anthony	Aguirre	(and	also	photoshopped	in	by	the	human-level	intelligence	sitting	next	to	him).

č
i
ć
Another	important	lesson	from	the	conference	was	this:	the	questions	raised
by	the	success	of	AI	aren’t	merely	intellectually	fascinating;	they’re	also	morally
crucial,	because	our	choices	can	potentially	affect	the	entire	future	of	life.	The
moral	significance	of	humanity’s	past	choices	were	sometimes	great,	but	always
limited:	we’ve	recovered	even	from	the	greatest	plagues,	and	even	the	grandest
empires	eventually	crumbled.	Past	generations	knew	that	as	surely	as	the	Sun
would	rise	tomorrow,	so	would	tomorrow’s	humans,	tackling	perennial	scourges

such	as	poverty,	disease	and	war.	But	some	of	the	Puerto	Rico	speakers	argued
that	this	time	might	be	different:	for	the	first	time,	they	said,	we	might	build
technology	powerful	enough	to	permanently	end	these	scourges—or	to	end
humanity	itself.	We	might	create	societies	that	flourish	like	never	before,	on
Earth	and	perhaps	beyond,	or	a	Kafkaesque	global	surveillance	state	so	powerful
that	it	could	never	be	toppled.

Figure	1.4:	Although	the	media	have	often	portrayed	Elon	Musk	as	being	at	loggerheads	with	the
AI	community,	there’s	in	fact	broad	consensus	that	AI-safety	research	is	needed.	Here	on	January
4,	2015,	Tom	Dietterich,	president	of	the	Association	for	the	Advancement	of	Artificial
Intelligence,	shares	Elon’s	excitement	about	the	new	AI-safety	research	program	that	Elon
pledged	to	fund	moments	earlier.	FLI	founders	Meia	Chita-Tegmark	and	Viktoriya	Krakovna
lurk	behind	them.

When	I	left	Puerto	Rico,	I	did	so	convinced	that	the	conversation	we	had	there
about	the	future	of	AI	needs	to	continue,	because	it’s	the	most	important
conversation	of	our	time.
*2
	It’s	the	conversation	about	the	collective	future	of	all
of	us,	so	it	shouldn’t	be	limited	to	AI	researchers.	That’s	why	I	wrote	this	book:	I
wrote	it	in	the	hope	that	you,	my	dear	reader,	will	join	this	conversation.	What
sort	of	future	do	you	want?	Should	we	develop	lethal	autonomous	weapons?
What	would	you	like	to	happen	with	job	automation?	What	career	advice	would
you	give	today’s	kids?	Do	you	prefer	new	jobs	replacing	the	old	ones,	or	a
jobless	society	where	everyone	enjoys	a	life	of	leisure	and	machine-produced
wealth?	Further	down	the	road,	would	you	like	us	to	create	Life	3.0	and	spread	it
through	our	cosmos?	Will	we	control	intelligent	machines	or	will	they	control
us?	Will	intelligent	machines	replace	us,	coexist	with	us	or	merge	with	us?	What

will	it	mean	to	be	human	in	the	age	of	artificial	intelligence?	What	would	you
like	it	to	mean,	and	how	can	we	make	the	future	be	that	way?
The	goal	of	this	book	is	to	help	you	join	this	conversation.	As	I	mentioned,
there	are	fascinating	controversies	where	the	world’s	leading	experts	disagree.
But	I’ve	also	seen	many	examples	of	boring	pseudo-controversies	in	which
people	misunderstand	and	talk	past	each	other.	To	help	ourselves	focus	on	the
interesting	controversies	and	open	questions,	not	on	the	misunderstandings,	let’s

start	by	clearing	up	some	of	the	most	common	misconceptions.
There	are	many	competing	definitions	in	common	use	for	terms	such	as	“life,”
“intelligence”	and	“consciousness,”	and	many	misconceptions	come	from	people
not	realizing	that	they’re	using	a	word	in	two	different	ways.	To	make	sure	that
you	and	I	don’t	fall	into	this	trap,	I’ve	put	a	cheat	sheet	in	
table	1.1
	showing	how
I	use	key	terms	in	this	book.	Some	of	these	definitions	will	only	be	properly
introduced	and	explained	in	later	chapters.	Please	note	that	I’m	not	claiming	that

table	1.1
my	definitions	are	better	than	anyone	else’s—I	simply	want	to	avoid	confusion
by	being	clear	on	what	I	mean.	You’ll	see	that	I	generally	go	for	broad
definitions	that	avoid	anthropocentric	bias	and	can	be	applied	to	machines	as
well	as	humans.	Please	read	the	cheat	sheet	now,	and	come	back	and	check	it
later	if	you	find	yourself	puzzled	by	how	I	use	one	of	its	words—especially	in
chapters	4–8.

Terminology	Cheat	Sheet
Life
Process	that	can	retain	its	complexity	and
replicate
Life	1.0
Life	that	evolves	its	hardware	and	software
(biological	stage)
Life	2.0
Life	that	evolves	its	hardware	but	designs
much	of	its	software	(cultural	stage)
Life	3.0
Life	that	designs	its	hardware	and	software
(technological	stage)
Intelligence
Ability	to	accomplish	complex	goals
Artificial
Intelligence	(AI)
Non-biological	intelligence
Narrow	intelligence
Ability	to	accomplish	a	narrow	set	of	goals,
e.g.,	play	chess	or	drive	a	car
General	intelligence
Ability	to	accomplish	virtually	any	goal,
including	learning
Universal
Ability	to	acquire	general	intelligence	given
access	to	data	and	resources
[Human-level]
Artificial	General
Intelligence	(AGI)
Ability	to	accomplish	any	cognitive	task	at
least	as	well	as	humans
Human-level	AI
AGI
Strong	AI
AGI
General	intelligence	far	beyond	human
level
Civilization
Interacting	group	of	intelligent	life	forms
Subjective	experience
Qualia

Universal
AGI
Strong	AI
AGI
level
Qualia
Individual	instances	of	subjective
experience
Ethics
Principles	that	govern	how	we	should
behave

Teleology
Explanation	of	things	in	terms	of	their	goals
or	purposes	rather	than	their	causes
Goal-oriented
behavior
Behavior	more	easily	explained	via	its
effect	than	via	its	cause
Having	a	goal
Exhibiting	goal-oriented	behavior
Having	purpose
Serving	goals	of	one’s	own	or	of	another
entity
Friendly	AI
Superintelligence	whose	goals	are	aligned
with	ours
Cyborg
Human-machine	hybrid
explosion
Recursive	self-improvement	rapidly	leading
to	superintelligence
Singularity
Intelligence	explosion
Universe
The	region	of	space	from	which	light	has
had	time	to	reach	us	during	the	13.8	billion
years	since	our	Big	Bang
Table	1.1:	Many	misunderstandings	about	AI	are	caused	by	people	using	the	words	above	to	mean	different
things.	Here’s	what	I	take	them	to	mean	in	this	book.	(Some	of	these	definitions	will	only	be	properly
introduced	and	explained	in	later	chapters.)
In	addition	to	confusion	over	terminology,	I’ve	also	seen	many	AI
conversations	get	derailed	by	simple	misconceptions.	Let’s	clear	up	the	most

common	ones.

Timeline	Myths
The	first	one	regards	the	timeline	from	
figure	1.2
:	how	long	will	it	take	until
machines	greatly	supersede	human-level	AGI?	Here,	a	common	misconception
is	that	we	know	the	answer	with	great	certainty.
One	popular	myth	is	that	we	know	we’ll	get	superhuman	AGI	this	century.	In
fact,	history	is	full	of	technological	over-hyping.	Where	are	those	fusion	power
plants	and	flying	cars	we	were	promised	we’d	have	by	now?	AI	too	has	been
repeatedly	over-hyped	in	the	past,	even	by	some	of	the	founders	of	the	field:	for
example,	John	McCarthy	(who	coined	the	term	“artificial	intelligence”),	Marvin
Minsky,	Nathaniel	Rochester	and	Claude	Shannon	wrote	this	overly	optimistic
forecast	about	what	could	be	accomplished	during	two	months	with	stone-age
computers:	“We	propose	that	a	2	month,	10	man	study	of	artificial	intelligence
be	carried	out	during	the	summer	of	1956	at	Dartmouth	College…An	attempt
will	be	made	to	find	how	to	make	machines	use	language,	form	abstractions	and

concepts,	solve	kinds	of	problems	now	reserved	for	humans,	and	improve
themselves.	We	think	that	a	significant	advance	can	be	made	in	one	or	more	of
these	problems	if	a	carefully	selected	group	of	scientists	work	on	it	together	for	a
summer.”
On	the	other	hand,	a	popular	counter-myth	is	that	we	know	we	
won’t
	get
superhuman	AGI	this	century.	Researchers	have	made	a	wide	range	of	estimates
for	how	far	we	are	from	superhuman	AGI,	but	we	certainly	can’t	say	with	great
confidence	that	the	probability	is	zero	this	century,	given	the	dismal	track	record

summer.”
won’t
	get
of	such	techno-skeptic	predictions.	For	example,	Ernest	Rutherford,	arguably	the
greatest	nuclear	physicist	of	his	time,	said	in	1933—less	than	twenty-four	hours
before	Leo	Szilard’s	invention	of	the	nuclear	chain	reaction—that	nuclear	energy
was	“moonshine,”	and	in	1956	Astronomer	Royal	Richard	Woolley	called	talk
about	space	travel	“utter	bilge.”	The	most	extreme	form	of	this	myth	is	that
superhuman	AGI	will	never	arrive	because	it’s	physically	impossible.	However,

physicists	know	that	a	brain	consists	of	quarks	and	electrons	arranged	to	act	as	a
powerful	computer,	and	that	there’s	no	law	of	physics	preventing	us	from
building	even	more	intelligent	quark	blobs.

Figure	1.5:	Common	myths	about	superintelligent	AI.
There	have	been	a	number	of	surveys	asking	AI	researchers	how	many	years
from	now	they	think	we’ll	have	human-level	AGI	with	at	least	50%	probability,
and	all	these	surveys	have	the	same	conclusion:	the	world’s	leading	experts

disagree,	so	we	simply	don’t	know.	For	example,	in	such	a	poll	of	the	AI
researchers	at	the	Puerto	Rico	AI	conference,	the	average	(median)	answer	was
by	the	year	2055,	but	some	researchers	guessed	hundreds	of	years	or	more.
There’s	also	a	related	myth	that	people	who	worry	about	AI	think	it’s	only	a
few	years	away.	In	fact,	most	people	on	record	worrying	about	superhuman	AGI
guess	it’s	still	at	least	decades	away.	But	they	argue	that	as	long	as	we’re	not
100%	
sure
	that	it	won’t	happen	this	century,	it’s	smart	to	start	safety	research
now	to	prepare	for	the	eventuality.	As	we’ll	see	in	this	book,	many	of	the	safety
problems	are	so	hard	that	they	may	take	decades	to	solve,	so	it’s	prudent	to	start
researching	them	now	rather	than	the	night	before	some	programmers	drinking
Red	Bull	decide	to	switch	on	human-level	AGI.

Controversy	Myths
Another	common	misconception	is	that	the	only	people	harboring	concerns
about	AI	and	advocating	AI-safety	research	are	Luddites	who	don’t	know	much
about	AI.	When	Stuart	Russell	mentioned	this	during	his	Puerto	Rico	talk,	the
audience	laughed	loudly.	A	related	misconception	is	that	supporting	AI-safety
research	is	hugely	controversial.	In	fact,	to	support	a	modest	investment	in	AI-
safety	research,	people	don’t	need	to	be	convinced	that	risks	are	high,	merely
non-negligible,	just	as	a	modest	investment	in	home	insurance	is	justified	by	a
non-negligible	probability	of	the	home	burning	down.
My	personal	analysis	is	that	the	media	have	made	the	AI-safety	debate	seem
more	controversial	than	it	really	is.	After	all,	fear	sells,	and	articles	using	out-of-
context	quotes	to	proclaim	imminent	doom	can	generate	more	clicks	than
nuanced	and	balanced	ones.	As	a	result,	two	people	who	only	know	about	each
other’s	positions	from	media	quotes	are	likely	to	think	they	disagree	more	than

they	really	do.	For	example,	a	techno-skeptic	whose	only	knowledge	about	Bill
Gates’	position	comes	from	a	British	tabloid	may	mistakenly	think	he	believes
superintelligence	to	be	imminent.	Similarly,	someone	in	the	beneficial-AI
movement	who	knows	nothing	about	Andrew	Ng’s	position	except	his	above-
mentioned	quote	about	overpopulation	on	Mars	may	mistakenly	think	he	doesn’t
care	about	AI	safety.	In	fact,	I	personally	know	that	he	does—the	crux	is	simply
that	because	his	timeline	estimates	are	longer,	he	naturally	tends	to	prioritize

short-term	AI	challenges	over	long-term	ones.

Myths	About	What	the	Risks	Are
I	rolled	my	eyes	when	seeing	this	headline	in	the	
Daily	Mail:
3
	“Stephen
Hawking	Warns	That	Rise	of	Robots	May	Be	Disastrous	for	Mankind.”	I’ve	lost
count	of	how	many	similar	articles	I’ve	seen.	Typically,	they’re	accompanied	by
an	evil-looking	robot	carrying	a	weapon,	and	suggest	that	we	should	worry	about
robots	rising	up	and	killing	us	because	they’ve	become	conscious	and/or	evil.	On
a	lighter	note,	such	articles	are	actually	rather	impressive,	because	they
succinctly	summarize	the	scenario	that	my	AI	colleagues	
don’t
	worry	about.	That
scenario	combines	as	many	as	three	separate	misconceptions:	concern	about
consciousness,
	
evil	
and	
robots,
	respectively.
If	you	drive	down	the	road,	you	have	a	subjective	experience	of	colors,
sounds,	etc.	But	does	a	self-driving	car	have	a	subjective	experience?	Does	it
feel	like	anything	at	all	to	be	a	self-driving	car,	or	is	it	like	an	unconscious
zombie	without	any	subjective	experience?	Although	this	mystery	of

don’t
	
evil	
and	
robots,
consciousness	is	interesting	in	its	own	right,	and	we’ll	devote	chapter	8	to	it,	it’s
irrelevant	to	AI	risk.	If	you	get	struck	by	a	driverless	car,	it	makes	no	difference
to	you	whether	it	subjectively	feels	conscious.	In	the	same	way,	what	will	affect
us	humans	is	what	superintelligent	AI	
does,
	not	how	it	subjectively	feels.
The	fear	of	machines	turning	evil	is	another	red	herring.	The	real	worry	isn’t
malevolence,	but	competence.	A	superintelligent	AI	is	by	definition	very	good	at

does,
attaining	its	goals,	whatever	they	may	be,	so	we	need	to	ensure	that	its	goals	are
aligned	with	ours.	You’re	probably	not	an	ant	hater	who	steps	on	ants	out	of
malice,	but	if	you’re	in	charge	of	a	hydroelectric	green	energy	project	and	there’s
an	anthill	in	the	region	to	be	flooded,	too	bad	for	the	ants.	The	beneficial-AI
movement	wants	to	avoid	placing	humanity	in	the	position	of	those	ants.
The	consciousness	misconception	is	related	to	the	myth	that	machines	can’t

have	goals.	Machines	can	obviously	have	goals	in	the	narrow	sense	of	exhibiting
goal-oriented	behavior:	the	behavior	of	a	heat-seeking	missile	is	most
economically	explained	as	a	goal	to	hit	a	target.	If	you	feel	threatened	by	a
machine	whose	goals	are	misaligned	with	yours,	then	it’s	precisely	its	goals	in
this	narrow	sense	that	trouble	you,	not	whether	the	machine	is	conscious	and
experiences	a	sense	of	purpose.	If	that	heat-seeking	missile	were	chasing	you,
you	probably	wouldn’t	exclaim	“I’m	not	worried,	because	machines	can’t	have
goals!”

I	sympathize	with	Rodney	Brooks	and	other	robotics	pioneers	who	feel
unfairly	demonized	by	scaremongering	tabloids,	because	some	journalists	seem
obsessively	fixated	on	robots	and	adorn	many	of	their	articles	with	evil-looking
metal	monsters	with	shiny	red	eyes.	In	fact,	the	main	concern	of	the	beneficial-
AI	movement	isn’t	with	robots	but	with	intelligence	itself:	specifically,
intelligence	whose	goals	are	misaligned	with	ours.	To	cause	us	trouble,	such
misaligned	intelligence	needs	no	robotic	body,	merely	an	internet	connection—
we’ll	explore	in	chapter	4	how	this	may	enable	outsmarting	financial	markets,
out-inventing	human	researchers,	out-manipulating	human	leaders	and
developing	weapons	we	cannot	even	understand.	Even	if	building	robots	were
physically	impossible,	a	super-intelligent	and	super-wealthy	AI	could	easily	pay
or	manipulate	myriad	humans	to	unwittingly	do	its	bidding,	as	in	William
Gibson’s	science	fiction	novel	
Neuromancer
.

.
The	robot	misconception	is	related	to	the	myth	that	machines	can’t	control
humans.	Intelligence	enables	control:	humans	control	tigers	not	because	we’re
stronger,	but	because	we’re	smarter.	This	means	that	if	we	cede	our	position	as
smartest	on	our	planet,	it’s	possible	that	we	might	also	cede	control.
Figure	1.5
	summarizes	all	of	these	common	misconceptions,	so	that	we	can
dispense	with	them	once	and	for	all	and	focus	our	discussions	with	friends	and

.
Figure	1.5
colleagues	on	the	many	legitimate	controversies—which,	as	we’ll	see,	there’s	no
shortage	of!

In	the	rest	of	this	book,	you	and	I	will	explore	together	the	future	of	life	with	AI.
Let’s	navigate	this	rich	and	multifaceted	topic	in	an	organized	way	by	first
exploring	the	full	story	of	life	conceptually	and	chronologically,	and	then
exploring	goals,	meaning	and	what	actions	to	take	to	create	the	future	we	want.
In	chapter	2,	we	explore	the	foundations	of	intelligence	and	how	seemingly
dumb	matter	can	be	rearranged	to	remember,	compute	and	learn.	As	we	proceed
into	the	future,	our	story	branches	out	into	many	scenarios	defined	by	the
answers	to	certain	key	questions.	
Figure	1.6
	summarizes	key	questions	we’ll
encounter	as	we	march	forward	in	time,	to	potentially	ever	more	advanced	AI.
Right	now,	we	face	the	choice	of	whether	to	start	an	AI	arms	race,	and
questions	about	how	to	make	tomorrow’s	AI	systems	bug-free	and	robust.	If	AI’s
economic	impact	keeps	growing,	we	also	have	to	decide	how	to	modernize	our
laws	and	what	career	advice	to	give	kids	so	that	they	can	avoid	soon-to-be-

Figure	1.6
automated	jobs.	We	explore	such	short-term	questions	in	chapter	3.
If	AI	progress	continues	to	human	levels,	then	we	also	need	to	ask	ourselves
how	to	ensure	that	it’s	beneficial,	and	whether	we	can	or	should	create	a	leisure
society	that	flourishes	without	jobs.	This	also	raises	the	question	of	whether	an
intelligence	explosion	or	slow-but-steady	growth	can	propel	AGI	far	beyond
human	levels.	We	explore	a	wide	range	of	such	scenarios	in	chapter	4	and
investigate	the	spectrum	of	possibilities	for	the	aftermath	in	chapter	5,	ranging

from	arguably	dystopic	to	arguably	utopic.	Who’s	in	charge—humans,	AI	or
cyborgs?	Are	humans	treated	well	or	badly?	Are	we	replaced	and,	if	so,	do	we
perceive	our	replacements	as	conquerors	or	worthy	descendants?	I’m	very
curious	about	which	of	the	chapter	5	scenarios	you	personally	prefer!	I’ve	set	up
a	website,	
http://AgeOfAi.org
,	where	you	can	share	your	views	and	join	the
conversation.
Finally,	we	forge	billions	of	years	into	the	future	in	chapter	6	where	we	can,
ironically,	draw	stronger	conclusions	than	in	the	previous	chapters,	as	the

a	website,	
ultimate	limits	of	life	in	our	cosmos	are	set	not	by	intelligence	but	by	the	laws	of
physics.
After	concluding	our	exploration	of	the	history	of	intelligence,	we’ll	devote

the	remainder	of	the	book	to	considering	what	future	to	aim	for	and	how	to	get
there.	To	be	able	to	link	cold	facts	to	questions	of	purpose	and	meaning,	we
explore	the	physical	basis	of	goals	in	chapter	7	and	consciousness	in	chapter	8.
Finally,	in	the	epilogue,	we	explore	what	can	be	done	right	now	to	help	create
the	future	we	want.

Figure	1.6:	Which	AI	questions	are	interesting	depends	on	how	advanced	AI	gets	and	which
branch	our	future	takes.
In	case	you’re	a	reader	who	likes	skipping	around,	most	chapters	are	relatively
self-contained	once	you’ve	digested	the	terminology	and	definitions	from	this

first	chapter	and	the	beginning	of	the	next	one.	If	you’re	an	AI	researcher,	you
can	optionally	skip	all	of	chapter	2	except	for	its	initial	intelligence	definitions.
If	you’re	new	to	AI,	chapters	2	and	3	will	give	you	the	arguments	for	why
chapters	4	through	6	can’t	be	trivially	dismissed	as	impossible	science	fiction.
Figure	1.7
	summarizes	where	the	various	chapters	fall	on	the	spectrum	from
factual	to	speculative.

Figure	1.7:	Structure	of	the	book
A	fascinating	journey	awaits	us.	Let’s	begin!

THE	BOTTOM	LINE:
•
Life,	defined	as	a	process	that	can	retain	its	complexity	and	replicate,	can	develop
through	three	stages:	a	biological	stage	(1.0),	where	its	hardware	and	software	are
evolved,	a	cultural	stage	(2.0),	where	it	can	design	its	software	(through	learning)
and	a	technological	stage	(3.0),	where	it	can	design	its	hardware	as	well,	becoming
the	master	of	its	own	destiny.
•
Artificial	intelligence	may	enable	us	to	launch	Life	3.0	this	century,	and	a
fascinating	conversation	has	sprung	up	regarding	what	future	we	should	aim	for	and
how	this	can	be	accomplished.	There	are	three	main	camps	in	the	controversy:
techno-skeptics,	digital	utopians	and	the	beneficial-AI	movement.
•
Techno-skeptics	view	building	superhuman	AGI	as	so	hard	that	it	won’t	happen	for
hundreds	of	years,	making	it	silly	to	worry	about	it	(and	Life	3.0)	now.
•
Digital	utopians	view	it	as	likely	this	century	and	wholeheartedly	welcome	Life	3.0,
viewing	it	as	the	natural	and	desirable	next	step	in	the	cosmic	evolution.
•

•
•
•
The	beneficial-AI	movement	also	views	it	as	likely	this	century,	but	views	a	good
outcome	not	as	guaranteed,	but	as	something	that	needs	to	be	ensured	by	hard	work
in	the	form	of	AI-safety	research.
•
Beyond	such	legitimate	controversies	where	world-leading	experts	disagree,	there
are	also	boring	pseudo-controversies	caused	by	misunderstandings.	For	example,
never	waste	time	arguing	about	“life,”	“intelligence,”	or	“consciousness”	before
ensuring	that	you	and	your	protagonist	are	using	these	words	to	mean	the	same

•
thing!	This	book	uses	the	definitions	in	
table	1.1
.
•
Also	beware	the	common	misconceptions	in	
figure	1.5
:	“Superintelligence	by	2100
is	inevitable/impossible.”	“Only	Luddites	worry	about	AI.”	“The	concern	is	about
AI	turning	evil	and/or	conscious,	and	it’s	just	years	away.”	“Robots	are	the	main
concern.”	“AI	can’t	control	humans	and	can’t	have	goals.”
•
In	chapters	2	through	6,	we’ll	explore	the	story	of	intelligence	from	its	humble
beginning	billions	of	years	ago	to	possible	cosmic	futures	billions	of	years	from

table	1.1
.
•
figure	1.5
•
now.	We’ll	first	investigate	near-term	challenges	such	as	jobs,	AI	weapons	and	the
quest	for	human-level	AGI,	then	explore	possibilities	for	a	fascinating	spectrum	of
possible	futures	with	intelligent	machines	and/or	humans.	I	wonder	which	options
you’ll	prefer!
•
In	chapters	7	through	9,	we’ll	switch	from	cold	factual	descriptions	to	an	exploration
of	goals,	consciousness	and	meaning,	and	investigate	what	we	can	do	right	now	to
help	create	the	future	we	want.
•

•
•
I	view	this	conversation	about	the	future	of	life	with	AI	as	the	most	important	one	of
our	time—please	join	it!

*1
	
Why	did	life	grow	more	complex?	Evolution	rewards	life	that’s	complex	enough	to	predict	and	exploit
regularities	in	its	environment,	so	in	a	more	complex	environment,	more	complex	and	intelligent	life	will
evolve.	Now	this	smarter	life	creates	a	more	complex	environment	for	competing	life	forms,	which	in
turn	evolve	to	be	more	complex,	eventually	creating	an	ecosystem	of	extremely	complex	life.
*2
	
The	AI	conversation	is	important	in	terms	of	both	urgency	and	impact.	In	comparison	with	climate
change,	which	might	wreak	havoc	in	fifty	to	two	hundred	years,	many	experts	expect	AI	to	have	greater
impact	within	decades—and	to	potentially	give	us	technology	for	mitigating	climate	change.	In
comparison	with	wars,	terrorism,	unemployment,	poverty,	migration	and	social	justice	issues,	the	rise	of
AI	will	have	greater	overall	impact—indeed,	we’ll	explore	in	this	book	how	it	can	dominate	what
happens	with	all	these	issues,	for	better	or	for	worse.

Chapter	2
Hydrogen…,	given	enough	time,	turns	into	people.
Edward	Robert	Harrison,	1995
One	of	the	most	spectacular	developments	during	the	13.8	billion	years	since	our
Big	Bang	is	that	dumb	and	lifeless	matter	has	turned	intelligent.	How	could	this
happen	and	how	much	smarter	can	things	get	in	the	future?	What	does	science
have	to	say	about	the	history	and	fate	of	intelligence	in	our	cosmos?	To	help	us
tackle	these	questions,	let’s	devote	this	chapter	to	exploring	the	foundations	and
fundamental	building	blocks	of	intelligence.	What	does	it	mean	to	say	that	a	blob
of	matter	is	intelligent?	What	does	it	mean	to	say	that	an	object	can	remember,
compute	and	learn?

My	wife	and	I	recently	had	the	good	fortune	to	attend	a	symposium	on	artificial
intelligence	organized	by	the	Swedish	Nobel	Foundation,	and	when	a	panel	of
leading	AI	researchers	were	asked	to	define	intelligence,	they	argued	at	length
without	reaching	consensus.	We	found	this	quite	funny:	there’s	no	agreement	on
what	intelligence	is	even	among	intelligent	intelligence	researchers!	So	there’s
clearly	no	undisputed	“correct”	definition	of	intelligence.	Instead,	there	are	many
competing	ones,	including	capacity	for	logic,	understanding,	planning,	emotional
knowledge,	self-awareness,	creativity,	problem	solving	and	learning.
In	our	exploration	of	the	future	of	intelligence,	we	want	to	take	a	maximally
broad	and	inclusive	view,	not	limited	to	the	sorts	of	intelligence	that	exist	so	far.
That’s	why	the	definition	I	gave	in	the	last	chapter,	and	the	way	I’m	going	to	use
the	word	throughout	this	book,	is	very	broad:
intelligence	=	
ability	to	accomplish	complex	goals

This	is	broad	enough	to	include	all	above-mentioned	definitions,	since
understanding,	self-awareness,	problem	solving,	learning,	etc.	are	all	examples
of	complex	goals	that	one	might	have.	It’s	also	broad	enough	to	subsume	the
Oxford	Dictionary
	definition—“the	ability	to	acquire	and	apply	knowledge	and
skills”—since	one	can	have	as	a	goal	to	apply	knowledge	and	skills.
Because	there	are	many	possible	goals,	there	are	many	possible	types	of
intelligence.	By	our	definition,	it	therefore	makes	no	sense	to	quantify

intelligence	of	humans,	non-human	animals	or	machines	by	a	single	number
such	as	an	IQ.
*1
	What’s	more	intelligent:	a	computer	program	that	can	only	play
chess	or	one	that	can	only	play	Go?	There’s	no	sensible	answer	to	this,	since
they’re	good	at	different	things	that	can’t	be	directly	compared.	We	can,
however,	say	that	a	third	program	is	more	intelligent	than	both	of	the	others	if
it’s	at	least	as	good	as	them	at	accomplishing	
all
	goals,	and	strictly	better	at	at

least	one	(winning	at	chess,	say).
It	also	makes	little	sense	to	quibble	about	whether	something	is	or	isn’t
intelligent	in	borderline	cases,	since	ability	comes	on	a	spectrum	and	isn’t
necessarily	an	all-or-nothing	trait.	What	people	have	the	ability	to	accomplish
the	goal	of	speaking?	Newborns?	No.	Radio	hosts?	Yes.	But	what	about	toddlers
who	can	speak	ten	words?	Or	five	hundred	words?	Where	would	you	draw	the
line?	I’ve	used	the	deliberately	vague	word	“complex”	in	the	definition	above,
because	it’s	not	very	interesting	to	try	to	draw	an	artificial	line	between
intelligence	and	non-intelligence,	and	it’s	more	useful	to	simply	quantify	the
degree	of	ability	for	accomplishing	different	goals.

Figure	2.1:	Intelligence,	defined	as	ability	to	accomplish	complex	goals,	can’t	be	measured	by	a
single	IQ,	only	by	an	ability	spectrum	across	all	goals.	Each	arrow	indicates	how	skilled	today’s
best	AI	systems	are	at	accomplishing	various	goals,	illustrating	that	today’s	artificial	intelligence
tends	to	be	
narrow,
	with	each	system	able	to	accomplish	only	very	specific	goals.	In	contrast,
human	intelligence	is	remarkably	broad:	a	healthy	child	can	learn	to	get	better	at	almost
anything.
To	classify	different	intelligences	into	a	taxonomy,	another	crucial	distinction
is	that	between	
narrow
	and	
broad
	intelligence.	IBM’s	Deep	Blue	chess
computer,	which	dethroned	chess	champion	Garry	Kasparov	in	1997,	was	only
able	to	accomplish	the	very	narrow	task	of	playing	chess—despite	its	impressive
hardware	and	software,	it	couldn’t	even	beat	a	four-year-old	at	tic-tac-toe.	The

DQN	AI	system	of	Google	DeepMind	can	accomplish	a	slightly	broader	range
of	goals:	it	can	play	dozens	of	different	vintage	Atari	computer	games	at	human
level	or	better.	In	contrast,	human	intelligence	is	thus	far	uniquely	broad,	able	to
master	a	dazzling	panoply	of	skills.	A	healthy	child	given	enough	training	time
can	get	fairly	good	not	only	at	
any
	game,	but	also	at	any	language,	sport	or
vocation.	Comparing	the	intelligence	of	humans	and	machines	today,	we	humans
win	hands-down	on	breadth,	while	machines	outperform	us	in	a	small	but
growing	number	of	narrow	domains,	as	illustrated	in	
figure	2.1
.	The	holy	grail	of
AI	research	is	to	build	“general	AI”	(better	known	as	
artificial	general
intelligence,
	AGI)	that	is	maximally	broad:	able	to	accomplish	virtually	any
goal,	including	learning.	We’ll	explore	this	in	detail	in	chapter	4.	The	term
“AGI”	was	popularized	by	the	AI	researchers	Shane	Legg,	Mark	Gubrud	and
Ben	Goertzel	to	more	specifically	mean	
human-level
	artificial	general

figure	2.1
intelligence:	the	ability	to	accomplish	any	goal	at	least	as	well	as	humans.
1
	I’ll
stick	with	their	definition,	so	unless	I	explicitly	qualify	the	acronym	(by	writing
“superhuman	AGI,”	for	example),	I’ll	use	“AGI”	as	shorthand	for	“human-level
AGI.”
*2
Although	the	word	“intelligence”	tends	to	have	positive	connotations,	it’s
important	to	note	that	we’re	using	it	in	a	completely	value-neutral	way:	as	ability
to	accomplish	complex	goals	regardless	of	whether	these	goals	are	considered

1
	I’ll
AGI.”
*2
good	or	bad.	Thus	an	intelligent	person	may	be	very	good	at	helping	people	or
very	good	at	hurting	people.	We’ll	explore	the	issue	of	goals	in	chapter	7.
Regarding	goals,	we	also	need	to	clear	up	the	subtlety	of	whose	goals	we’re
referring	to.	Suppose	your	future	brand-new	robotic	personal	assistant	has	no
goals	whatsoever	of	its	own,	but	will	do	whatever	you	ask	it	to	do,	and	you	ask	it
to	cook	the	perfect	Italian	dinner.	If	it	goes	online	and	researches	Italian	dinner

recipes,	how	to	get	to	the	closest	supermarket,	how	to	strain	pasta	and	so	on,	and
then	successfully	buys	the	ingredients	and	prepares	a	succulent	meal,	you’ll
presumably	consider	it	intelligent	even	though	the	original	goal	was	yours.	In
fact,	it	adopted	your	goal	once	you’d	made	your	request,	and	then	broke	it	into	a
hierarchy	of	subgoals	of	its	own,	from	paying	the	cashier	to	grating	the
Parmesan.	In	this	sense,	intelligent	behavior	is	inexorably	linked	to	goal
attainment.

Figure	2.2:	Illustration	of	Hans	Moravec’s	“landscape	of	human	competence,”	where	elevation
represents	difficulty	for	computers,	and	the	rising	sea	level	represents	what	computers	are	able	to
do.
It’s	natural	for	us	to	rate	the	difficulty	of	tasks	relative	to	how	hard	it	is	for	us
humans	to	perform	them,	as	in	
figure	2.1
.	But	this	can	give	a	misleading	picture
of	how	hard	they	are	for	computers.	It	feels	much	harder	to	multiply	314,159	by
271,828	than	to	recognize	a	friend	in	a	photo,	yet	computers	creamed	us	at
arithmetic	long	before	I	was	born,	while	human-level	image	recognition	has	only
recently	become	possible.	This	fact	that	low-level	sensorimotor	tasks	seem	easy
despite	requiring	enormous	computational	resources	is	known	as	Moravec’s
paradox,	and	is	explained	by	the	fact	that	our	brain	makes	such	tasks	feel	easy	by
dedicating	massive	amounts	of	customized	hardware	to	them—more	than	a
quarter	of	our	brains,	in	fact.
I	love	this	metaphor	from	Hans	Moravec,	and	have	taken	the	liberty	to

illustrate	it	in	
figure	2.2
:
Computers	are	universal	machines,	their	potential	extends
uniformly	over	a	boundless	expanse	of	tasks.	Human	potentials,	on

the	other	hand,	are	strong	in	areas	long	important	for	survival,	but
weak	in	things	far	removed.	Imagine	a	“landscape	of	human
competence,”	having	lowlands	with	labels	like	“arithmetic”	and
“rote	memorization,”	foothills	like	“theorem	proving”	and	“chess
playing,”	and	high	mountain	peaks	labeled	“locomotion,”	“hand-
eye	coordination”	and	“social	interaction.”	Advancing	computer
performance	is	like	water	slowly	flooding	the	landscape.	A	half
century	ago	it	began	to	drown	the	lowlands,	driving	out	human
calculators	and	record	clerks,	but	leaving	most	of	us	dry.	Now	the
flood	has	reached	the	foothills,	and	our	outposts	there	are
contemplating	retreat.	We	feel	safe	on	our	peaks,	but,	at	the	present
rate,	those	too	will	be	submerged	within	another	half	century.	I
propose	that	we	build	Arks	as	that	day	nears,	and	adopt	a	seafaring
life!
2
During	the	decades	since	he	wrote	those	passages,	the	sea	level	has	kept	rising
relentlessly,	as	he	predicted,	like	global	warming	on	steroids,	and	some	of	his

life!
2
foothills	(including	chess)	have	long	since	been	submerged.	What	comes	next
and	what	we	should	do	about	it	is	the	topic	of	the	rest	of	this	book.
As	the	sea	level	keeps	rising,	it	may	one	day	reach	a	tipping	point,	triggering
dramatic	change.	This	critical	sea	level	is	the	one	corresponding	to	machines
becoming	able	to	perform	AI	design.	Before	this	tipping	point	is	reached,	the
sea-level	rise	is	caused	by	
humans
	improving	machines;	afterward,	the	rise	can
be	driven	by	
machines

humans
machines
	improving	machines,	potentially	much	faster	than
humans	could	have	done,	rapidly	submerging	all	land.	This	is	the	fascinating	and
controversial	idea	of	the	
singularity,
	which	we’ll	have	fun	exploring	in	chapter	4.
Computer	pioneer	Alan	Turing	famously	proved	that	if	a	computer	can
perform	a	certain	bare	minimum	set	of	operations,	then,	given	enough	time	and
memory,	it	can	be	programmed	to	do	anything	that	
any
	other	computer	can	do.
Machines	exceeding	this	critical	threshold	are	called	
universal	computers
	(aka

any
	(aka
Turing-universal	computers);	all	of	today’s	smartphones	and	laptops	are
universal	in	this	sense.	Analogously,	I	like	to	think	of	the	critical	intelligence
threshold	required	for	AI	design	as	the	threshold	for	
universal	intelligence:
given	enough	time	and	resources,	it	can	make	itself	able	to	accomplish	any	goal
as	well	as	
any
	other	intelligent	entity.	For	example,	if	it	decides	that	it	wants
better	social	skills,	forecasting	skills	or	AI-design	skills,	it	can	acquire	them.	If	it

decides	to	figure	out	how	to	build	a	robot	factory,	then	it	can	do	so.	In	other
words,	universal	intelligence	has	the	potential	to	develop	into	Life	3.0.
The	conventional	wisdom	among	artificial	intelligence	researchers	is	that
intelligence	is	ultimately	all	about	information	and	computation,	not	about	flesh,
blood	or	carbon	atoms.	This	means	that	there’s	no	fundamental	reason	why
machines	can’t	one	day	be	at	least	as	intelligent	as	us.
But	what	are	information	and	computation	really,	given	that	physics	has
taught	us	that,	at	a	fundamental	level,	everything	is	simply	matter	and	energy
moving	around?	How	can	something	as	abstract,	intangible	and	ethereal	as
information	and	computation	be	embodied	by	tangible	physical	stuff?	In
particular,	how	can	a	bunch	of	dumb	particles	moving	around	according	to	the
laws	of	physics	exhibit	behavior	that	we’d	call	intelligent?
If	you	feel	that	the	answer	to	this	question	is	obvious	and	consider	it	plausible

that	machines	might	get	as	intelligent	as	humans	this	century—for	example
because	you’re	an	AI	researcher—please	skip	the	rest	of	this	chapter	and	jump
straight	to	chapter	3.	Otherwise,	you’ll	be	pleased	to	know	that	I’ve	written	the
next	three	sections	specially	for	you.

If	we	say	that	an	atlas	contains	
information
	about	the	world,	we	mean	that
there’s	a	relation	between	the	state	of	the	book	(in	particular,	the	positions	of
certain	molecules	that	give	the	letters	and	images	their	colors)	and	the	state	of
the	world	(for	example,	the	locations	of	continents).	If	the	continents	were	in
different	places,	then	those	molecules	would	be	in	different	places	as	well.	We
humans	use	a	panoply	of	different	devices	for	storing	information,	from	books
and	brains	to	hard	drives,	and	they	all	share	this	property:	that	their	state	can	be
related	to	(and	therefore	inform	us	about)	the	state	of	other	things	that	we	care
about.
What	fundamental	physical	property	do	they	all	have	in	common	that	makes
them	useful	as	memory	devices,	i.e.,	devices	for	storing	information?	The
answer	is	that	they	all	
can	be	in	many	different	long-lived	states—
long-lived
enough	to	encode	the	information	until	it’s	needed.	As	a	simple	example,

about.
suppose	you	place	a	ball	on	a	hilly	surface	that	has	sixteen	different	valleys,	as	in
figure	2.3
.	Once	the	ball	has	rolled	down	and	come	to	rest,	it	will	be	in	one	of
sixteen	places,	so	you	can	use	its	position	as	a	way	of	remembering	any	number
between	1	and	16.
This	memory	device	is	rather	robust,	because	even	if	it	gets	a	bit	jiggled	and
disturbed	by	outside	forces,	the	ball	is	likely	to	stay	in	the	same	valley	that	you
put	it	in,	so	you	can	still	tell	which	number	is	being	stored.	The	reason	that	this

figure	2.3
memory	is	so	stable	is	that	lifting	the	ball	out	of	its	valley	requires	more	energy
than	random	disturbances	are	likely	to	provide.	This	same	idea	can	provide
stable	memories	much	more	generally	than	for	a	movable	ball:	the	energy	of	a
complicated	physical	system	can	depend	on	all	sorts	of	mechanical,	chemical,
electrical	and	magnetic	properties,	and	as	long	as	it	takes	energy	to	change	the
system	away	from	the	state	you	want	it	to	remember,	this	state	will	be	stable.

This	is	why	solids	have	many	long-lived	states,	whereas	liquids	and	gases	don’t:
if	you	engrave	someone’s	name	on	a	gold	ring,	the	information	will	still	be	there
years	later	because	reshaping	the	gold	requires	significant	energy,	but	if	you
engrave	it	in	the	surface	of	a	pond,	it	will	be	lost	within	a	second	as	the	water
surface	effortlessly	changes	its	shape.
The	simplest	possible	memory	device	has	only	two	stable	states	(
figure	2.3
).

We	can	therefore	think	of	it	as	encoding	a	binary	digit	(abbreviated	“bit”),	i.e.,	a
zero	or	a	one.	The	information	stored	by	any	more	complicated	memory	device
can	equivalently	be	stored	in	multiple	bits:	for	example,	taken	together,	the	four
bits	shown	in	
figure	2.3
	can	be	in	2	×	2	×	2	×	2	=	16	different	states	0000,	0001,
0010,	0011,…,	1111,	so	they	collectively	have	exactly	the	same	memory
capacity	as	the	more	complicated	16-state	system.	We	can	therefore	think	of	bits
as	atoms	of	information—the	smallest	indivisible	chunk	of	information	that	can’t
be	further	subdivided,	which	can	combine	to	make	up	any	information.	For
example,	I	just	typed	the	word	“word,”	and	my	laptop	represented	it	in	its
memory	as	the	4-number	sequence	119	111	114	100,	storing	each	of	those
numbers	as	8	bits	(it	represents	each	lowercase	letter	by	a	number	that’s	96	plus
its	order	in	the	alphabet).	As	soon	as	I	hit	the	
w
	key	on	my	keyboard,	my	laptop
displayed	a	visual	image	of	a	
w

w
w
	on	my	screen,	and	this	image	is	also	represented
by	bits:	32	bits	specify	the	color	of	each	of	the	screen’s	millions	of	pixels.

Figure	2.3:	A	physical	object	is	a	useful	memory	device	if	it	can	be	in	many	different	stable
states.	The	ball	on	the	left	can	encode	four	bits	of	information	labeling	which	one	of	2
4
	=	16
valleys	it’s	in.	Together,	the	four	balls	on	the	right	also	encode	four	bits	of	information—one	bit
each.
Since	two-state	systems	are	easy	to	manufacture	and	work	with,	most	modern
computers	store	their	information	as	bits,	but	these	bits	are	embodied	in	a	wide
variety	of	ways.	On	a	DVD,	each	bit	corresponds	to	whether	there	is	or	isn’t	a
microscopic	pit	at	a	given	point	on	the	plastic	surface.	On	a	hard	drive,	each	bit
corresponds	to	a	point	on	the	surface	being	magnetized	in	one	of	two	ways.	In
my	laptop’s	working	memory,	each	bit	corresponds	to	the	positions	of	certain
electrons,	determining	whether	a	device	called	a	micro-capacitor	is	charged.
Some	kinds	of	bits	are	convenient	to	transport	as	well,	even	at	the	speed	of	light:
for	example,	in	an	optical	fiber	transmitting	your	email,	each	bit	corresponds	to	a

laser	beam	being	strong	or	weak	at	a	given	time.
Engineers	prefer	to	encode	bits	into	systems	that	aren’t	only	stable	and	easy	to
read	from	(as	a	gold	ring),	but	also	easy	to	write	to:	altering	the	state	of	your
hard	drive	requires	much	less	energy	than	engraving	gold.	They	also	prefer
systems	that	are	convenient	to	work	with	and	cheap	to	mass-produce.	But	other
than	that,	they	simply	don’t	care	about	how	the	bits	are	represented	as	physical
objects—and	nor	do	you	most	of	the	time,	because	it	simply	doesn’t	matter!	If

you	email	your	friend	a	document	to	print,	the	information	may	get	copied	in
rapid	succession	from	magnetizations	on	your	hard	drive	to	electric	charges	in
your	computer’s	working	memory,	radio	waves	in	your	wireless	network,
voltages	in	your	router,	laser	pulses	in	an	optical	fiber	and,	finally,	molecules	on
a	piece	of	paper.	In	other	words,	
information	can
	
take	on	a	life	of	its	own,
independent	of	its	physical	substrate!
	Indeed,	it’s	usually	only	this	substrate-
independent	aspect	of	information	that	we’re	interested	in:	if	your	friend	calls
you	up	to	discuss	that	document	you	sent,	she’s	probably	not	calling	to	talk
about	voltages	or	molecules.	This	is	our	first	hint	of	how	something	as	intangible
as	intelligence	can	be	embodied	in	tangible	physical	stuff,	and	we’ll	soon	see
how	this	idea	of	substrate	independence	is	much	deeper,	including	not	only
information	but	also	computation	and	learning.
Because	of	this	substrate	independence,	clever	engineers	have	been	able	to

repeatedly	replace	the	memory	devices	inside	our	computers	with	dramatically
better	ones,	based	on	new	technologies,	without	requiring	any	changes
whatsoever	to	our	software.	The	result	has	been	spectacular,	as	illustrated	in
figure	2.4
:	over	the	past	six	decades,	computer	memory	has	gotten	half	as
expensive	roughly	every	couple	of	years.	Hard	drives	have	gotten	over	100
million	times	cheaper,	and	the	faster	memories	useful	for	computation	rather
than	mere	storage	have	become	a	whopping	10	trillion	times	cheaper.	If	you

figure	2.4
could	get	such	a	“99.99999999999%	off”	discount	on	all	your	shopping,	you
could	buy	all	real	estate	in	New	York	City	for	about	10	cents	and	all	the	gold
that’s	ever	been	mined	for	around	a	dollar.
For	many	of	us,	the	spectacular	improvements	in	memory	technology	come
with	personal	stories.	I	fondly	remember	working	in	a	candy	store	back	in	high
school	to	pay	for	a	computer	sporting	16	kilobytes	of	memory,	and	when	I	made
and	sold	a	word	processor	for	it	with	my	high	school	classmate	Magnus	Bodin,

we	were	forced	to	write	it	all	in	ultra-compact	machine	code	to	leave	enough
memory	for	the	words	that	it	was	supposed	to	process.	After	getting	used	to
floppy	drives	storing	70kB,	I	became	awestruck	by	the	smaller	3.5-inch	floppies
that	could	store	a	whopping	1.44MB	and	hold	a	whole	book,	and	then	my	first-
ever	hard	drive	storing	10MB—which	might	just	barely	fit	a	single	one	of
today’s	song	downloads.	These	memories	from	my	adolescence	felt	almost

unreal	the	other	day,	when	I	spent	about	$100	on	a	hard	drive	with	300,000	times
more	capacity.

Figure	2.4:	Over	the	past	six	decades,	computer	memory	has	gotten	twice	as	cheap	roughly	every
couple	of	years,	corresponding	to	a	thousand	times	cheaper	roughly	every	twenty	years.	A	byte
equals	eight	bits.	Data	courtesy	of	John	McCallum,	from	http://www.jcmit.net/memoryprice.htm.
What	about	memory	devices	that	evolved	rather	than	being	designed	by
humans?	Biologists	don’t	yet	know	what	the	first-ever	life	form	was	that	copied
its	blueprints	between	generations,	but	it	may	have	been	quite	small.	A	team	led
by	Philipp	Holliger	at	Cambridge	University	made	an	RNA	molecule	in	2016
that	encoded	412	bits	of	genetic	information	and	was	able	to	copy	RNA	strands
longer	than	itself,	bolstering	the	“RNA	world”	hypothesis	that	early	Earth	life
involved	short	self-replicating	RNA	snippets.	So	far,	the	smallest	memory	device
known	to	be	evolved	and	used	in	the	wild	is	the	genome	of	the	bacterium
Candidatus	Carsonella	ruddii,	storing	about	40	kilobytes,	whereas	our	human

DNA	stores	about	1.6	gigabytes,	comparable	to	a	downloaded	movie.	As

mentioned	in	the	last	chapter,	our	brains	store	much	more	information	than	our
genes:	in	the	ballpark	of	10	gigabytes	electrically	(specifying	which	of	your	100
billion	neurons	are	firing	at	any	one	time)	and	100	terabytes
chemically/biologically	(specifying	how	strongly	different	neurons	are	linked	by
synapses).	Comparing	these	numbers	with	the	machine	memories	shows	that	the
world’s	best	computers	can	now	out-remember	any	biological	system—at	a	cost
that’s	rapidly	dropping	and	was	a	few	thousand	dollars	in	2016.
The	memory	in	your	brain	works	very	differently	from	computer	memory,	not
only	in	terms	of	how	it’s	built,	but	also	in	terms	of	how	it’s	used.	Whereas	you
retrieve	memories	from	a	computer	or	hard	drive	by	specifying	
where
	it’s	stored,
you	retrieve	memories	from	your	brain	by	specifying	something	about	
what
	is
stored.	Each	group	of	bits	in	your	computer’s	memory	has	a	numerical	address,
and	to	retrieve	a	piece	of	information,	the	computer	specifies	at	what	address	to

where
what
	is
look,	just	as	if	I	tell	you	“Go	to	my	bookshelf,	take	the	fifth	book	from	the	right
on	the	top	shelf,	and	tell	me	what	it	says	on	page	314.”	In	contrast,	you	retrieve
information	from	your	brain	similarly	to	how	you	retrieve	it	from	a	search
engine:	you	specify	a	piece	of	the	information	or	something	related	to	it,	and	it
pops	up.	If	I	tell	you	“to	be	or	not,”	or	if	I	google	it,	chances	are	that	it	will
trigger	“To	be,	or	not	to	be,	that	is	the	question.”	Indeed,	it	will	probably	work

even	if	I	use	another	part	of	the	quote	or	mess	things	up	somewhat.	Such
memory	systems	are	called	
auto-associative,
	since	they	recall	by	association
rather	than	by	address.
In	a	famous	1982	paper,	the	physicist	John	Hopfield	showed	how	a	network	of
interconnected	neurons	could	function	as	an	auto-associative	memory.	I	find	the
basic	idea	very	beautiful,	and	it	works	for	any	physical	system	with	multiple
stable	states.	For	example,	consider	a	ball	on	a	surface	with	two	valleys,	like	the
one-bit	system	in	
figure	2.3

figure	2.3
,	and	let’s	shape	the	surface	so	that	the	
x
-coordinates
of	the	two	minima	where	the	ball	can	come	to	rest	are	
x
	=	
√
2	
≈
	1
.
41421	and	
x
	=
π
	
≈
	3
.
14159,	respectively.	If	you	remember	only	that	
π
	is	close	to	3,	you	simply
put	the	ball	at	
x
	=	3	and	watch	it	reveal	a	more	exact	
π
-value	as	it	rolls	down	to
the	nearest	minimum.	Hopfield	realized	that	a	complex	network	of	neurons
provides	an	analogous	landscape	with	very	many	energy-minima	that	the	system
can	settle	into,	and	it	was	later	proved	that	you	can	squeeze	in	as	many	as	138

x
x
	=	
√
2	
≈
	1
.
41421	and	
x
	=
π
	
≈
	3
.
π
x
π
different	memories	for	every	thousand	neurons	without	causing	major	confusion.

We’ve	now	seen	how	a	physical	object	can	remember	information.	But	how	can
it	compute?
A	computation	is	a	transformation	of	one	memory	state	into	another.	In	other
words,	a	computation	takes	information	and	transforms	it,	implementing	what
mathematicians	call	a	
function
.	I	think	of	a	function	as	a	meat	grinder	for
information,	as	illustrated	in	
figure	2.5
:	you	put	information	in	at	the	top,	turn
the	crank	and	get	processed	information	out	at	the	bottom—and	you	can	repeat
this	as	many	times	as	you	want	with	different	inputs.	This	information
processing	is	deterministic	in	the	sense	that	if	you	repeat	it	with	the	same	input,
you	get	the	same	output	every	time.

Figure	2.5:	A	
computation
	takes	information	and	transforms	it,	implementing	what
function
.	The	function	
f
	(left)	takes	bits	representing	a	number	and
computes	its	square.	The	function	
g
	(middle)	takes	bits	representing	a	chess	position	and
computes	the	best	move	for	White.	The	function	
h
	(right)	takes	bits	representing	an	image	and
computes	a	text	label	describing	it.
Although	it	sounds	deceptively	simple,	this	idea	of	a	function	is	incredibly
general.	Some	functions	are	rather	trivial,	such	as	the	one	called	
NOT
	that	inputs
a	single	bit	and	outputs	the	reverse,	thus	turning	zero	into	one	and	vice	versa.
The	functions	we	learn	about	in	school	typically	correspond	to	buttons	on	a
pocket	calculator,	inputting	one	or	more	numbers	and	outputting	a	single	number
—for	example,	the	function	
x
2
	simply	inputs	a	number	and	outputs	it	multiplied
by	itself.	Other	functions	can	be	extremely	complicated.	For	instance,	if	you’re

NOT
x
2
in	possession	of	a	function	that	would	input	bits	representing	an	arbitrary	chess
position	and	output	bits	representing	the	best	possible	next	move,	you	can	use	it
to	win	the	World	Computer	Chess	Championship.	If	you’re	in	possession	of	a
function	that	inputs	all	the	world’s	financial	data	and	outputs	the	best	stocks	to
buy,	you’ll	soon	be	extremely	rich.	Many	AI	researchers	dedicate	their	careers	to
figuring	out	how	to	implement	certain	functions.	For	example,	the	goal	of

machine-translation	research	is	to	implement	a	function	inputting	bits
representing	text	in	one	language	and	outputting	bits	representing	that	same	text
in	another	language,	and	the	goal	of	automatic-captioning	research	is	inputting

bits	representing	an	image	and	outputting	bits	representing	text	describing	it
(
figure	2.5
).

Figure	2.6:	A	so-called	NAND	gate	takes	two	bits,	A	and	B,	as	inputs	and	computes	one	bit	C	as
output,	according	to	the	rule	that	C	=	0	if	A	=	B	=	1	and	C	=	1	otherwise.	Many	physical	systems
can	be	used	as	NAND	gates.	In	the	middle	example,	switches	are	interpreted	as	bits	where	0	=
open,	1=	closed,	and	when	switches	A	and	B	are	both	closed,	an	electromagnet	opens	the	switch
C.	In	the	rightmost	example,	voltages	(electrical	potentials)	are	interpreted	as	bits	where	1	=	five
volts,	0	=	zero	volts,	and	when	wires	A	and	B	are	both	at	five	volts,	the	two	transistors	conduct
electricity	and	the	wire	C	drops	to	approximately	zero	volts.
In	other	words,	if	you	can	implement	highly	complex	functions,	then	you	can
build	an	intelligent	machine	that’s	able	to	accomplish	highly	complex	goals.	This
brings	our	question	of	how	matter	can	be	intelligent	into	sharper	focus:	in
particular,	how	can	a	clump	of	seemingly	dumb	matter	compute	a	complicated
function?

function?
Rather	than	just	remain	immobile	as	a	gold	ring	or	other	static	memory
device,	it	must	exhibit	complex	
dynamics
	so	that	its	future	state	depends	in	some
complicated	(and	hopefully	controllable/programmable)	way	on	the	present
state.	Its	atom	arrangement	must	be	less	ordered	than	a	rigid	solid	where	nothing
interesting	changes,	but	more	ordered	than	a	liquid	or	gas.	Specifically,	we	want
the	system	to	have	the	property	that	if	we	put	it	in	a	state	that	encodes	the	input

function?
dynamics
information,	let	it	evolve	according	to	the	laws	of	physics	for	some	amount	of
time,	and	then	interpret	the	resulting	final	state	as	the	output	information,	then
the	output	is	the	desired	function	of	the	input.	If	this	is	the	case,	then	we	can	say

that	our	system	computes	our	function.
As	a	first	example	of	this	idea,	let’s	explore	how	we	can	build	a	very	simple
(but	also	very	important)	function	called	a	
NAND
	
gate
*3
	out	of	plain	old	dumb
matter.	This	function	inputs	two	bits	and	outputs	one	bit:	it	outputs	0	if	both
inputs	are	1;	in	all	other	cases,	it	outputs	1.	If	we	connect	two	switches	in	series
with	a	battery	and	an	electromagnet,	then	the	electromagnet	will	only	be	on	if
the	first	switch	
and
	the	second	switch	are	closed	(“on”).	Let’s	place	a	third
switch	under	the	electromagnet,	as	illustrated	in	
figure	2.6
,	such	that	the	magnet
will	pull	it	open	whenever	it’s	powered	on.	If	we	interpret	the	first	two	switches
as	the	input	bits	and	the	third	one	as	the	output	bit	(with	0	=	switch	open,	and	1	=
switch	closed),	then	we	have	ourselves	a	NAND	gate:	the	third	switch	is	open
only	if	the	first	two	are	closed.	There	are	many	other	ways	of	building	NAND
gates	that	are	more	practical—for	example,	using	transistors	as	illustrated	in
figure	2.6

figure	2.6
figure	2.6
.	In	today’s	computers,	NAND	gates	are	typically	built	from
microscopic	transistors	and	other	components	that	can	be	automatically	etched
onto	silicon	wafers.
There’s	a	remarkable	theorem	in	computer	science	that	says	that	NAND	gates
are	
universal
,	meaning	that	you	can	implement	
any
	well-defined	function	simply
by	connecting	together	NAND	gates.
*4
	So	if	you	can	build	enough	NAND	gates,
you	can	build	a	device	computing	anything!	In	case	you’d	like	a	taste	of	how
this	works,	I’ve	illustrated	in	
figure	2.7

are	
universal
any
*4
figure	2.7
	how	to	multiply	numbers	using	nothing
but	NAND	gates.
MIT	researchers	Norman	Margolus	and	Tommaso	Toffoli	coined	the	name
computronium
	for	any	substance	that	can	perform	arbitrary	computations.	We’ve
just	seen	that	making	computronium	doesn’t	have	to	be	particularly	hard:	the
substance	just	needs	to	be	able	to	implement	NAND	gates	connected	together	in
any	desired	way.	Indeed,	there	are	myriad	other	kinds	of	computronium	as	well.
A	simple	variant	that	also	works	involves	replacing	the	NAND	gates	by	NOR

how	to	multiply	numbers	using	nothing
gates	that	output	1	only	when	both	inputs	are	0.	In	the	next	section,	we’ll	explore
neural	networks,	which	can	also	implement	arbitrary	computations,	i.e.,	act	as
computronium.	Scientist	and	entrepreneur	Stephen	Wolfram	has	shown	that	the
same	goes	for	simple	devices	called	cellular	automata,	which	repeatedly	update
bits	based	on	what	neighboring	bits	are	doing.	Already	back	in	1936,	computer
pioneer	Alan	Turing	proved	in	a	landmark	paper	that	a	simple	machine	(now

known	as	a	“universal	Turing	machine”)	that	could	manipulate	symbols	on	a
strip	of	tape	could	also	implement	arbitrary	computations.	In	summary,	not	only

is	it	possible	for	matter	to	implement	any	well-defined	computation,	but	it’s
possible	in	a	plethora	of	different	ways.
As	mentioned	earlier,	Turing	also	proved	something	even	more	profound	in
that	1936	paper	of	his:	that	if	a	type	of	computer	can	perform	a	certain	bare
minimum	set	of	operations,	then	it’s	
universal
	in	the	sense	that	given	enough
resources,	it	can	do	anything	that	any	other	computer	can	do.	He	showed	that	his
Turing	machine	was	universal,	and	connecting	back	more	closely	to	physics,
we’ve	just	seen	that	this	family	of	universal	computers	also	includes	objects	as
diverse	as	a	network	of	NAND	gates	and	a	network	of	interconnected	neurons.
Indeed,	Stephen	Wolfram	has	argued	that	
most
	non-trivial	physical	systems,
from	weather	systems	to	brains,	would	be	universal	computers	if	they	could	be
made	arbitrarily	large	and	long-lasting.

Figure	2.7:	
Any
	well-defined	computation	can	be	performed	by	cleverly	combining	nothing	but
NAND	gates.	For	example,	the	addition	and	multiplication	modules	above	both	input	two	binary
numbers	represented	by	4	bits,	and	output	a	binary	number	represented	by	5	bits	and	8	bits,
respectively.	The	smaller	modules	NOT,	AND,	XOR	and	+	(which	sums	three	separate	bits	into	a
2-bit	binary	number)	are	in	turn	built	out	of	NAND	gates.	Fully	understanding	this	figure	is
extremely	challenging	and	totally	unnecessary	for	following	the	rest	of	this	book;	I’m	including
it	here	just	to	illustrate	the	idea	of	universality—and	to	satisfy	my	inner	geek.
This	fact	that	exactly	the	same	computation	can	be	performed	on	
any
universal	computer	means	that	
computation	is	substrate-independent
	in	the	same
way	that	information	is:	it	can	take	on	a	life	of	its	own,	independent	of	its

physical	substrate!	So	if	you’re	a	conscious	superintelligent	character	in	a	future
computer	game,	you’d	have	no	way	of	knowing	whether	you	ran	on	a	Windows
desktop,	a	Mac	OS	laptop	or	an	Android	phone,	because	you	would	be	substrate-
independent.	You’d	also	have	no	way	of	knowing	what	type	of	transistors	the
microprocessor	was	using.
I	first	came	to	appreciate	this	crucial	idea	of	substrate	independence	because
there	are	many	beautiful	examples	of	it	in	physics.	Waves,	for	instance:	they
have	properties	such	as	speed,	wavelength	and	frequency,	and	we	physicists	can
study	the	equations	they	obey	without	even	needing	to	know	what	particular
substance	they’re	waves	in.	When	you	hear	something,	you’re	detecting	sound
waves	caused	by	molecules	bouncing	around	in	the	mixture	of	gases	that	we	call
air,	and	we	can	calculate	all	sorts	of	interesting	things	about	these	waves—how
their	intensity	fades	as	the	square	of	the	distance,	such	as	how	they	bend	when

they	pass	through	open	doors	and	how	they	bounce	off	of	walls	and	cause	echoes
—without	knowing	what	air	is	made	of.	In	fact,	we	don’t	even	need	to	know	that
it’s	made	of	molecules:	we	can	ignore	all	details	about	oxygen,	nitrogen,	carbon
dioxide,	etc.,	because	the	only	property	of	the	wave’s	substrate	that	matters	and
enters	into	the	famous	wave	equation	is	a	single	number	that	we	can	measure:
the	wave	speed,	which	in	this	case	is	about	300	meters	per	second.	Indeed,	this

wave	equation	that	I	taught	my	MIT	students	about	in	a	course	last	spring	was
first	discovered	and	put	to	great	use	long	before	physicists	had	even	established
that	atoms	and	molecules	existed!
This	wave	example	illustrates	three	important	points.	First,	substrate
independence	doesn’t	mean	that	a	substrate	is	unnecessary,	but	that	most	of	its
details	don’t	matter.	You	obviously	can’t	have	sound	waves	in	a	gas	if	there’s	no
gas,	but	any	gas	whatsoever	will	suffice.	Similarly,	you	obviously	can’t	have

computation	without	matter,	but	any	matter	will	do	as	long	as	it	can	be	arranged
into	NAND	gates,	connected	neurons	or	some	other	building	block	enabling
universal	computation.	Second,	the	substrate-independent	phenomenon	takes	on
a	life	of	its	own,	independent	of	its	substrate.	A	wave	can	travel	across	a	lake,
even	though	none	of	its	water	molecules	do—they	mostly	bob	up	and	down,	like
fans	doing	“the	wave”	in	a	sports	stadium.	Third,	it’s	often	only	the	substrate-

independent	aspect	that	we’re	interested	in:	a	surfer	usually	cares	more	about	the
position	and	height	of	a	wave	than	about	its	detailed	molecular	composition.	We
saw	how	this	was	true	for	information,	and	it’s	true	for	computation	too:	if	two
programmers	are	jointly	hunting	a	bug	in	their	code,	they’re	probably	not
discussing	transistors.

We’ve	now	arrived	at	an	answer	to	our	opening	question	about	how	tangible
physical	stuff	can	give	rise	to	something	that	feels	as	intangible,	abstract	and
ethereal	as	intelligence:	it	feels	so	non-physical	because	it’s	substrate-
independent,	taking	on	a	life	of	its	own	that	doesn’t	depend	on	or	reflect	the
physical	details.	In	short,	computation	is	a	pattern	in	the	spacetime	arrangement
of	particles,	and	it’s	not	the	particles	but	the	pattern	that	really	matters!	Matter
doesn’t	matter.
In	other	words,	the	hardware	is	the	matter	and	the	software	is	the	pattern.	This
substrate	independence	of	computation	implies	that	AI	is	possible:	intelligence
doesn’t	require	flesh,	blood	or	carbon	atoms.
Because	of	this	substrate	independence,	shrewd	engineers	have	been	able	to
repeatedly	replace	the	technologies	inside	our	computers	with	dramatically
better	ones,	without	changing	the	software.	The	results	have	been	every	bit	as
spectacular	as	those	for	memory	devices.	As	illustrated	in	
figure	2.8
,

figure	2.8
,
computation	keeps	getting	half	as	expensive	roughly	every	couple	of	years,	and
this	trend	has	now	persisted	for	over	a	century,	cutting	the	computer	cost	a
whopping	million	million	million	(10
18
)	times	since	my	grandmothers	were
born.	If	everything	got	a	million	million	million	times	cheaper,	then	a	hundredth
of	a	cent	would	enable	you	to	buy	all	goods	and	services	produced	on	Earth	this
year.	This	dramatic	drop	in	costs	is	of	course	a	key	reason	why	computation	is

figure	2.8
,
18
everywhere	these	days,	having	spread	from	the	building-sized	computing
facilities	of	yesteryear	into	our	homes,	cars	and	pockets—and	even	turning	up	in
unexpected	places	such	as	sneakers.
Why	does	our	technology	keep	doubling	its	power	at	regular	intervals,
displaying	what	mathematicians	call	exponential	growth?	Indeed,	why	is	it
happening	not	only	in	terms	of	transistor	miniaturization	(a	trend	known	as
Moore’s	law
),	but	also	more	broadly	for	computation	as	a	whole	(
figure	2.8
),	for
memory	(
figure	2.4

figure	2.8
),	for
memory	(
figure	2.4
)	and	for	a	plethora	of	other	technologies	ranging	from
genome	sequencing	to	brain	imaging?	Ray	Kurzweil	calls	this	persistent
doubling	phenomenon	“the	law	of	accelerating	returns.”

Figure	2.8:	Since	1900,	computation	has	gotten	twice	as	cheap	roughly	every	couple	of	years.
The	plot	shows	the	computing	power	measured	in	floating-point	operations	per	second	(FLOPS)
that	can	be	purchased	for	$1,000.
3
	The	particular	computation	that	defines	a	floating	point
operation	corresponds	to	about	10
5
	elementary	logical	operations	such	as	bit	flips	or	NAND
evaluations.
All	examples	of	persistent	doubling	that	I	know	of	in	nature	have	the	same
fundamental	cause,	and	this	technological	one	is	no	exception:	each	step	creates
the	next.	For	example,	you	yourself	underwent	exponential	growth	right	after
your	conception:	each	of	your	cells	divided	and	gave	rise	to	two	cells	roughly
daily,	causing	your	total	number	of	cells	to	increase	day	by	day	as	1,	2,	4,	8,	16
and	so	on.	According	to	the	most	popular	scientific	theory	of	our	cosmic	origins,
known	as	
inflation,
	our	baby	Universe	once	grew	exponentially	just	like	you	did,

known	as	
repeatedly	doubling	its	size	at	regular	intervals	until	a	speck	much	smaller	and
lighter	than	an	atom	had	grown	more	massive	than	all	the	galaxies	we’ve	ever
seen	with	our	telescopes.	Again,	the	cause	was	a	process	whereby	each	doubling
step	caused	the	next.	This	is	how	technology	progresses	as	well:	once

technology	gets	twice	as	powerful,	it	can	often	be	used	to	design	and	build
technology	that’s	twice	as	powerful	in	turn,	triggering	repeated	capability
doubling	in	the	spirit	of	Moore’s	law.
Something	that	occurs	just	as	regularly	as	the	doubling	of	our	technological
power	is	the	appearance	of	claims	that	the	doubling	is	ending.	Yes,	Moore’s	law
will	of	course	end,	meaning	that	there’s	a	physical	limit	to	how	small	transistors
can	be	made.	But	some	people	mistakenly	assume	that	Moore’s	law	is
synonymous	with	the	persistent	doubling	of	our	technological	power.
Contrariwise,	Ray	Kurzweil	points	out	that	Moore’s	law	involves	not	the	first
but	the	fifth	technological	paradigm	to	bring	exponential	growth	in	computing,
as	illustrated	in	
figure	2.8
:	whenever	one	technology	stopped	improving,	we
replaced	it	with	an	even	better	one.	When	we	could	no	longer	keep	shrinking	our
vacuum	tubes,	we	replaced	them	with	transistors	and	then	integrated	circuits,

figure	2.8
where	electrons	move	around	in	two	dimensions.	When	this	technology	reaches
its	limits,	there	are	many	other	alternatives	we	can	try—for	example,	using
three-dimensional	circuits	and	using	something	other	than	electrons	to	do	our
bidding.
Nobody	knows	for	sure	what	the	next	blockbuster	computational	substrate
will	be,	but	we	do	know	that	we’re	nowhere	near	the	limits	imposed	by	the	laws
of	physics.	My	MIT	colleague	Seth	Lloyd	has	worked	out	what	this	fundamental
limit	is,	and	as	we’ll	explore	in	greater	detail	in	chapter	6,	this	limit	is	a

bidding.
whopping	33	orders	of	magnitude	(10
33
	times)	beyond	today’s	state	of	the	art	for
how	much	computing	a	clump	of	matter	can	do.	So	even	if	we	keep	doubling	the
power	of	our	computers	every	couple	of	years,	it	will	take	over	two	centuries
until	we	reach	that	final	frontier.
Although	all	universal	computers	are	capable	of	the	same	computations,	some
are	more	efficient	than	others.	For	example,	a	computation	requiring	millions	of
multiplications	doesn’t	require	millions	of	separate	multiplication	modules	built
from	separate	transistors	as	in

33
from	separate	transistors	as	in	
figure	2.6
:	it	needs	only	one	such	module,	since	it
can	use	it	many	times	in	succession	with	appropriate	inputs.	In	this	spirit	of
efficiency,	most	modern	computers	use	a	paradigm	where	computations	are	split
into	multiple	time	steps,	during	which	information	is	shuffled	back	and	forth
between	memory	modules	and	computation	modules.	This	computational
architecture	was	developed	between	1935	and	1945	by	computer	pioneers
including	Alan	Turing,	Konrad	Zuse,	Presper	Eckert,	John	Mauchly	and	John

figure	2.6
von	Neumann.	More	specifically,	the	computer	memory	stores	both	data	and

software	(a	program,	i.e.,	a	list	of	instructions	for	what	to	do	with	the	data).	At
each	time	step,	a	central	processing	unit	(CPU)	executes	the	next	instruction	in
the	program,	which	specifies	some	simple	function	to	apply	to	some	part	of	the
data.	The	part	of	the	computer	that	keeps	track	of	what	to	do	next	is	merely
another	part	of	its	memory,	called	the	
program	counter,
	which	stores	the	current
line	number	in	the	program.	To	go	to	the	next	instruction,	simply	add	one	to	the
program	counter.	To	jump	to	another	line	of	the	program,	simply	copy	that	line
number	into	the	program	counter—this	is	how	so-called	“if”	statements	and
loops	are	implemented.
Today’s	computers	often	gain	additional	speed	by	
parallel	processing,
	which
cleverly	undoes	some	of	this	reuse	of	modules:	if	a	computation	can	be	split	into
parts	that	can	be	done	in	parallel	(because	the	input	of	one	part	doesn’t	require
the	output	of	another),	then	the	parts	can	be	computed	simultaneously	by
different	parts	of	the	hardware.

	which
The	ultimate	parallel	computer	is	a	
quantum	computer
.	Quantum	computing
pioneer	David	Deutsch	controversially	argues	that	“quantum	computers	share
information	with	huge	numbers	of	versions	of	themselves	throughout	the
multiverse,”	and	can	get	answers	faster	here	in	our	Universe	by	in	a	sense
getting	help	from	these	other	versions.
4
	We	don’t	yet	know	whether	a
commercially	competitive	quantum	computer	can	be	built	during	the	coming
decades,	because	it	depends	both	on	whether	quantum	physics	works	as	we	think

4
it	does	and	on	our	ability	to	overcome	daunting	technical	challenges,	but
companies	and	governments	around	the	world	are	betting	tens	of	millions	of
dollars	annually	on	the	possibility.	Although	quantum	computers	cannot	speed
up	run-of-the-mill	computations,	clever	algorithms	have	been	developed	that
may	dramatically	speed	up	specific	types	of	calculations,	such	as	cracking
cryptosystems	and	training	neural	networks.	A	quantum	computer	could	also
efficiently	simulate	the	behavior	of	quantum-mechanical	systems,	including

atoms,	molecules	and	new	materials,	replacing	measurements	in	chemistry	labs
in	the	same	way	that	simulations	on	traditional	computers	have	replaced
measurements	in	wind	tunnels.

Although	a	pocket	calculator	can	crush	me	in	an	arithmetic	contest,	it	will	never
improve	its	speed	or	accuracy,	no	matter	how	much	it	practices.	It	doesn’t	learn:
for	example,	every	time	I	press	its	square-root	button,	it	computes	exactly	the
same	function	in	exactly	the	same	way.	Similarly,	the	first	computer	program
that	ever	beat	me	at	chess	never	learned	from	its	mistakes,	but	merely
implemented	a	function	that	its	clever	programmer	had	designed	to	compute	a
good	next	move.	In	contrast,	when	Magnus	Carlsen	lost	his	first	game	of	chess	at
age	five,	he	began	a	learning	process	that	made	him	the	World	Chess	Champion
eighteen	years	later.
The	ability	to	learn	is	arguably	the	most	fascinating	aspect	of	general
intelligence.	We’ve	already	seen	how	a	seemingly	dumb	clump	of	matter	can
remember	and	compute,	but	how	can	it	learn?	We’ve	seen	that	finding	the
answer	to	a	difficult	question	corresponds	to	computing	a	function,	and	that

appropriately	arranged	matter	can	calculate	any	computable	function.	When	we
humans	first	created	pocket	calculators	and	chess	programs,	
we
	did	the
arranging.	For	matter	to	learn,	it	must	instead	rearrange	
itself
	to	get	better	and
better	at	computing	the	desired	function—simply	by	obeying	the	laws	of
physics.
To	demystify	the	learning	process,	let’s	first	consider	how	a	very	simple
physical	system	can	learn	the	digits	of	
π
	and	other	numbers.	Above	we	saw	how
a	surface	with	many	valleys	(see	
figure	2.3

we
	did	the
itself
physics.
π
figure	2.3
)	can	be	used	as	a	memory	device:	for
example,	if	the	bottom	of	one	of	the	valleys	is	at	position	
x
	=	
π
	
≈
	3
.
14159	and
there	are	no	other	valleys	nearby,	then	you	can	put	a	ball	at	
x
	=	3	and	watch	the
system	compute	the	missing	decimals	by	letting	the	ball	roll	down	to	the	bottom.
Now,	suppose	that	the	surface	is	made	of	soft	clay	and	starts	out	completely	flat,
as	a	blank	slate.	If	some	math	enthusiasts	repeatedly	place	the	ball	at	the
locations	of	each	of	their	favorite	numbers,	then	gravity	will	gradually	create

x
	=	
π
	
≈
	3
.
14159	and
x
valleys	at	these	locations,	after	which	the	clay	surface	can	be	used	to	recall	these
stored	memories.	In	other	words,	the	clay	surface	has	
learned
	to	compute	digits
of	numbers	such	as	
π
.
Other	physical	systems,	such	as	brains,	can	learn	much	more	efficiently	based
on	the	same	idea.	John	Hopfield	showed	that	his	above-mentioned	network	of

interconnected	neurons	can	learn	in	an	analogous	way:	if	you	repeatedly	put	it
into	certain	states,	it	will	gradually	learn	these	states	and	return	to	them	from	any
nearby	state.	If	you’ve	seen	each	of	your	family	members	many	times,	then
memories	of	what	they	look	like	can	be	triggered	by	anything	related	to	them.
Neural	networks	have	now	transformed	both	biological	and	artificial
intelligence,	and	have	recently	started	dominating	the	AI	subfield	known	as
machine	learning
	(the	study	of	algorithms	that	improve	through	experience).
Before	delving	deeper	into	how	such	networks	can	learn,	let’s	first	understand
how	they	can	compute.	A	neural	network	is	simply	a	group	of	interconnected
neurons	that	are	able	to	influence	each	other’s	behavior.	Your	brain	contains
about	as	many	neurons	as	there	are	stars	in	our	Galaxy:	in	the	ballpark	of	a
hundred	billion.	On	average,	each	of	these	neurons	is	connected	to	about	a
thousand	others	via	junctions	called	
synapses,
	and	it’s	the	strengths	of	these

synapses,
roughly	hundred	trillion	synapse	connections	that	encode	most	of	the
information	in	your	brain.
We	can	schematically	draw	a	neural	network	as	a	collection	of	dots
representing	neurons	connected	by	lines	representing	synapses	(see	
figure	2.9
).
Real-world	neurons	are	very	complicated	electrochemical	devices	looking
nothing	like	this	schematic	illustration:	they	involve	different	parts	with	names
such	as	axons	and	dendrites,	there	are	many	different	kinds	of	neurons	that
operate	in	a	wide	variety	of	ways,	and	the	exact	details	of	how	and	when

figure	2.9
).
electrical	activity	in	one	neuron	affects	other	neurons	is	still	the	subject	of	active
study.	However,	AI	researchers	have	shown	that	neural	networks	can	still	attain
human-level	performance	on	many	remarkably	complex	tasks	even	if	one
ignores	all	these	complexities	and	replaces	real	biological	neurons	with
extremely	simple	simulated	ones	that	are	all	identical	and	obey	very	simple
rules.	The	currently	most	popular	model	for	such	an	
artificial	neural	network
represents	the	state	of	each	neuron	by	a	single	number	and	the	strength	of	each

synapse	by	a	single	number.	In	this	model,	each	neuron	updates	its	state	at
regular	time	steps	by	simply	averaging	together	the	inputs	from	all	connected
neurons,	weighting	them	by	the	synaptic	strengths,	optionally	adding	a	constant,
and	then	applying	what’s	called	an	
activation	function
	to	the	result	to	compute
its	next	state.
*5
	The	easiest	way	to	use	a	neural	network	as	a	function	is	to	make
it	
feedforward,
	with	information	flowing	only	in	one	direction,	as	in	
figure	2.9
,
plugging	the	input	to	the	function	into	a	layer	of	neurons	at	the	top	and

*5
it	
figure	2.9
,
extracting	the	output	from	a	layer	of	neurons	at	the	bottom.

Figure	2.9:	A	network	of	neurons	can	compute	functions	just	as	a	network	of	NAND	gates	can.
For	example,	artificial	neural	networks	have	been	trained	to	input	numbers	representing	the
brightness	of	different	image	pixels	and	output	numbers	representing	the	probability	that	the
image	depicts	various	people.	Here	each	artificial	neuron	(circle)	computes	a	weighted	sum	of
the	numbers	sent	to	it	via	connections	(lines)	from	above,	applies	a	simple	function	and	passes
the	result	downward,	each	subsequent	layer	computing	higher-level	features.	Typical	face-
recognition	networks	contain	hundreds	of	thousands	of	neurons;	the	figure	shows	merely	a
handful	for	clarity.
The	success	of	these	simple	artificial	neural	networks	is	yet	another	example
of	substrate	independence:	neural	networks	have	great	computational	power
seemingly	independent	of	the	low-level	nitty-gritty	details	of	their	construction.
Indeed,	George	Cybenko,	Kurt	Hornik,	Maxwell	Stinchcombe	and	Halbert

White	proved	something	remarkable	in	1989:	such	simple	neural	networks	are
universal
	in	the	sense	that	they	can	compute	
any
	function	arbitrarily	accurately,
by	simply	adjusting	those	synapse	strength	numbers	accordingly.	In	other	words,
evolution	probably	didn’t	make	our	biological	neurons	so	complicated	because	it
was	necessary,	but	because	it	was	more	efficient—and	because	evolution,	as
opposed	to	human	engineers,	doesn’t	reward	designs	that	are	simple	and	easy	to
understand.

universal
any
When	I	first	learned	about	this,	I	was	mystified	by	how	something	so	simple
could	compute	something	arbitrarily	complicated.	For	example,	how	can	you
compute	even	something	as	simple	as	multiplication,	when	all	you’re	allowed	to

do	is	compute	weighted	sums	and	apply	a	single	fixed	function?	In	case	you’d
like	a	taste	of	how	this	works,	
figure	2.10
	shows	how	a	mere	five	neurons	can
multiply	two	arbitrary	numbers	together,	and	how	a	single	neuron	can	multiply
three	bits	together.
Although	you	can	prove	that	you	can	compute	anything	in	
theory
	with	an
arbitrarily	large	neural	network,	the	proof	doesn’t	say	anything	about	whether
you	can	do	so	in	
practice,
	with	a	network	of	reasonable	size.	In	fact,	the	more	I
thought	about	it,	the	more	puzzled	I	became	that	neural	networks	worked	so
well.
For	example,	suppose	that	we	wish	to	classify	megapixel	grayscale	images
into	two	categories,	say	cats	or	dogs.	If	each	of	the	million	pixels	can	take	one
of,	say,	256	values,	then	there	are	256
1000000
	possible	images,	and	for	each	one,
we	wish	to	compute	the	probability	that	it	depicts	a	cat.	This	means	that	an
arbitrary	function	that	inputs	a	picture	and	outputs	a	probability	is	defined	by	a
list	of	256
1000000

well.
1000000
list	of	256
1000000
	probabilities,	that	is,	way	more	numbers	than	there	are	atoms	in
our	Universe	(about	10
78
).	Yet	neural	networks	with	merely	thousands	or
millions	of	parameters	somehow	manage	to	perform	such	classification	tasks
quite	well.	How	can	successful	neural	networks	be	“cheap,”	in	the	sense	of
requiring	so	few	parameters?	After	all,	you	can	prove	that	a	neural	network
small	enough	to	fit	inside	our	Universe	will	epically	fail	to	approximate	almost
all	functions,	succeeding	merely	on	a	ridiculously	tiny	fraction	of	all

78
computational	tasks	that	you	might	assign	to	it.

Figure	2.10:	How	matter	can	multiply,	but	using	not	NAND	gates	as	in	
figure	2.7
	but	neurons.
The	key	point	doesn’t	require	following	the	details,	and	is	that	not	only	can	neurons	(artificial	or
biological)	do	math,	but	multiplication	requires	many	fewer	neurons	than	NAND	gates.	
Optional
details	for	hard-core
	
math	fans:
	Circles	perform	summation,	squares	apply	the	function	
σ
,	and
lines	multiply	by	the	constants	labeling	them.	The	inputs	are	real	numbers	(left)	and	bits	(right).
The	multiplication	becomes	arbitrarily	accurate	as	
a
	
→
	0	(left)	and	
c
	
→
	
∞
	(right).	The	left
network	works	for	any	function	
σ
(
x
)	that’s	curved	at	the	origin	(with	second	derivative	
σ
″
(0)
≠
0),
which	can	be	proven	by	Taylor	expanding	
σ
(
x
).	The	right	network	requires	that	the	function	
σ
(
x
)
approaches	0	and	1	when	
x
	gets	very	small	and	very	large,	respectively,	which	is	seen	by	noting
that	
uvw
	=	1	only	if	
u
	+	
v
	+	
w
	=	3.	(These	examples	are	from	a	paper	I	wrote	with	my	students

a
	
→
c
	
→
	
∞
σ
(
x
σ
″
(0)
≠
0),
σ
(
x
σ
(
x
)
x
that	
uvw
	=	1	only	if	
u
	+	
v
	+	
w
Henry	Lin	and	David	Rolnick,	“Why	Does	Deep	and	Cheap	Learning	Work	So	Well?,”	which
can	be	found	at	http://arxiv.org/abs/1608.08225.)	By	combining	together	lots	of	multiplications
(as	above)	and	additions,	you	can	compute	any	polynomials,	which	are	well	known	to	be	able	to
approximate	any	smooth	function.
I’ve	had	lots	of	fun	puzzling	over	this	and	related	mysteries	with	my	student
Henry	Lin.	One	of	the	things	I	feel	most	grateful	for	in	life	is	the	opportunity	to
collaborate	with	amazing	students,	and	Henry	is	one	of	them.	When	he	first

walked	into	my	office	to	ask	whether	I	was	interested	in	working	with	him,	I
thought	to	myself	that	it	would	be	more	appropriate	for	me	to	ask	whether	he
was	interested	in	working	with	me:	this	modest,	friendly	and	bright-eyed	kid
from	Shreveport,	Louisiana,	had	already	written	eight	scientific	papers,	won	a
Forbes	30-Under-30	award,	and	given	a	TED	talk	with	over	a	million	views—
and	he	was	only	twenty!	A	year	later,	we	wrote	a	paper	together	with	a	surprising

conclusion:	the	question	of	why	neural	networks	work	so	well	can’t	be	answered
with	mathematics	alone,	because	part	of	the	answer	lies	in	physics.	We	found
that	the	class	of	functions	that	the	laws	of	physics	throw	at	us	and	make	us
interested	in	computing	is	also	a	remarkably	tiny	class	because,	for	reasons	that
we	still	don’t	fully	understand,	the	laws	of	physics	are	remarkably	simple.
Moreover,	the	tiny	fraction	of	functions	that	neural	networks	can	compute	is
very	similar	to	the	tiny	fraction	that	physics	makes	us	interested	in!	We	also
extended	previous	work	showing	that	deep-learning	neural	networks	(they’re
called	“deep”	if	they	contain	many	layers)	are	much	more	efficient	than	shallow
ones	for	many	of	these	functions	of	interest.	For	example,	together	with	another
amazing	MIT	student,	David	Rolnick,	we	showed	that	the	simple	task	of
multiplying	
n
	numbers	requires	a	whopping	2
n
	neurons	for	a	network	with	only
one	layer,	but	takes	only	about	4
n
	neurons	in	a	deep	network.	This	helps	explain

n
n
n
not	only	why	neural	networks	are	now	all	the	rage	among	AI	researchers,	but
also	why	we	evolved	neural	networks	in	our	brains:	if	we	evolved	brains	to
predict	the	future,	then	it	makes	sense	that	we’d	evolve	a	computational
architecture	that’s	good	at	precisely	those	computational	problems	that	matter	in
the	physical	world.
Now	that	we’ve	explored	how	neural	networks	work	and	compute,	let’s	return
to	the	question	of	how	they	can	learn.	Specifically,	how	can	a	neural	network	get
better	at	computing	by	updating	its	synapses?

In	his	seminal	1949	book,	
The	Organization	of	Behavior:	A
Neuropsychological	Theory,
	the	Canadian	psychologist	Donald	Hebb	argued	that
if	two	nearby	neurons	were	frequently	active	(“firing”)	at	the	same	time,	their
synaptic	coupling	would	strengthen	so	that	they	learned	to	help	trigger	each
other—an	idea	captured	by	the	popular	slogan	“Fire	together,	wire	together.”
Although	the	details	of	how	actual	brains	learn	are	still	far	from	understood,	and
research	has	shown	that	the	answers	are	in	many	cases	much	more	complicated,

it’s	also	been	shown	that	even	this	simple	learning	rule	(known	as	Hebbian
learning)	allows	neural	networks	to	learn	interesting	things.	John	Hopfield
showed	that	Hebbian	learning	allowed	his	oversimplified	artificial	neural
network	to	store	lots	of	complex	memories	by	simply	being	exposed	to	them
repeatedly.	Such	exposure	to	information	to	learn	from	is	usually	called
“training”	when	referring	to	artificial	neural	networks	(or	to	animals	or	people

being	taught	skills),	although	“studying,”	“education”	or	“experience”	might	be
just	as	apt.	The	artificial	neural	networks	powering	today’s	AI	systems	tend	to
replace	Hebbian	learning	with	more	sophisticated	learning	rules	with	nerdy

names	such	as	“backpropagation”	and	“stochastic	gradient	descent,”	but	the
basic	idea	is	the	same:	there’s	some	simple	deterministic	rule,	akin	to	a	law	of
physics,	by	which	the	synapses	get	updated	over	time.	As	if	by	magic,	this
simple	rule	can	make	the	neural	network	learn	remarkably	complex
computations	if	training	is	performed	with	large	amounts	of	data.	We	don’t	yet
know	precisely	what	learning	rules	our	brains	use,	but	whatever	the	answer	may
be,	there’s	no	indication	that	they	violate	the	laws	of	physics.
Just	as	most	digital	computers	gain	efficiency	by	splitting	their	work	into
multiple	steps	and	reusing	computational	modules	many	times,	so	do	many
artificial	and	biological	neural	networks.	Brains	have	parts	that	are	what
computer	scientists	call	
recurrent
	rather	than	feedforward	neural	networks,
where	information	can	flow	in	multiple	directions	rather	than	just	one	way,	so
that	the	current	output	can	become	input	to	what	happens	next.	The	network	of

recurrent
logic	gates	in	the	microprocessor	of	a	laptop	is	also	recurrent	in	this	sense:	it
keeps	reusing	its	past	information,	and	lets	new	information	input	from	a
keyboard,	trackpad,	camera,	etc.,	affect	its	ongoing	computation,	which	in	turn
determines	information	output	to,	say,	a	screen,	loudspeaker,	printer	or	wireless
network.	Analogously,	the	network	of	neurons	in	your	brain	is	recurrent,	letting
information	input	from	your	eyes,	ears	and	other	senses	affect	its	ongoing
computation,	which	in	turn	determines	information	output	to	your	muscles.

The	history	of	learning	is	at	least	as	long	as	the	history	of	life	itself,	since
every	self-reproducing	organism	performs	interesting	copying	and	processing	of
information—behavior	that	has	somehow	been	learned.	During	the	era	of	Life
1.0,	however,	organisms	didn’t	learn	during	their	lifetime:	their	rules	for
processing	information	and	reacting	were	determined	by	their	inherited	DNA,	so
the	only	learning	occurred	slowly	at	the	species	level,	through	Darwinian
evolution	across	generations.

About	half	a	billion	years	ago,	certain	gene	lines	here	on	Earth	discovered	a
way	to	make	animals	containing	neural	networks,	able	to	learn	behaviors	from
experiences	during	life.	Life	2.0	had	arrived,	and	because	of	its	ability	to	learn
dramatically	faster	and	outsmart	the	competition,	it	spread	like	wildfire	across
the	globe.	As	we	explored	in	chapter	1,	life	has	gotten	progressively	better	at
learning,	and	at	an	ever-increasing	rate.	A	particular	ape-like	species	grew	a

brain	so	adept	at	acquiring	knowledge	that	it	learned	how	to	use	tools,	make	fire,
speak	a	language	and	create	a	complex	global	society.	This	society	can	itself	be
viewed	as	a	system	that	remembers,	computes	and	learns,	all	at	an	accelerating
pace	as	one	invention	enables	the	next:	writing,	the	printing	press,	modern

science,	computers,	the	internet	and	so	on.	What	will	future	historians	put	next
on	that	list	of	enabling	inventions?	My	guess	is	artificial	intelligence.
As	we	all	know,	the	explosive	improvements	in	computer	memory	and
computational	power	(
figure	2.4
	and	
figure	2.8
)	have	translated	into	spectacular
progress	in	artificial	intelligence—but	it	took	a	long	time	until	machine	
learning
came	of	age.	When	IBM’s	Deep	Blue	computer	overpowered	chess	champion
Garry	Kasparov	in	1997,	its	major	advantages	lay	in	memory	and	computation,
not	in	learning.	Its	computational	intelligence	had	been	created	by	a	team	of
humans,	and	the	key	reason	that	Deep	Blue	could	outplay	its	creators	was	its
ability	to	compute	faster	and	thereby	analyze	more	potential	positions.	When
IBM’s	Watson	computer	dethroned	the	human	world	champion	in	the	quiz	show
Jeopardy!
,	it	too	relied	less	on	learning	than	on	custom-programmed	skills	and
superior	memory	and	speed.	The	same	can	be	said	of	most	early	breakthroughs

Jeopardy!
in	robotics,	from	legged	locomotion	to	self-driving	cars	and	self-landing	rockets.
In	contrast,	the	driving	force	behind	many	of	the	most	recent	AI
breakthroughs	has	been	machine	
learning
.	Consider	
figure	2.11
,	for	example.
It’s	easy	for	you	to	tell	what	it’s	a	photo	of,	but	to	program	a	function	that	inputs
nothing	but	the	colors	of	all	the	pixels	of	an	image	and	outputs	an	accurate
caption	such	as	“A	group	of	young	people	playing	a	game	of	frisbee”	had	eluded
all	the	world’s	AI	researchers	for	decades.	Yet	a	team	at	Google	led	by	Ilya

learning
.	Consider	
Sutskever	did	precisely	that	in	2014.	Input	a	different	set	of	pixel	colors,	and	it
replies	“A	herd	of	elephants	walking	across	a	dry	grass	field,”	again	correctly.
How	did	they	do	it?	Deep	Blue–style,	by	programming	handcrafted	algorithms
for	detecting	frisbees,	faces	and	the	like?	No,	by	creating	a	relatively	simple
neural	network	with	no	knowledge	whatsoever	about	the	physical	world	or	its
contents,	and	then	letting	it	learn	by	exposing	it	to	massive	amounts	of	data.	AI
visionary	Jeff	Hawkins	wrote	in	2004	that	“no	computer	can…see	as	well	as	a

mouse,”	but	those	days	are	now	long	gone.

Figure	2.11:	“A	group	of	young	people	playing	a	game	of	frisbee”—that	caption	was	written	by	a
computer	with	no	understanding	of	people,	games	or	frisbees.
Just	as	we	don’t	fully	understand	how	our	children	learn,	we	still	don’t	fully
understand	how	such	neural	networks	learn,	and	why	they	occasionally	fail.	But
what’s	clear	is	that	they’re	already	highly	useful	and	are	triggering	a	surge	of
investments	in	deep	learning.	Deep	learning	has	now	transformed	many	aspects
of	computer	vision,	from	handwriting	transcription	to	real-time	video	analysis
for	self-driving	cars.	It	has	similarly	revolutionized	the	ability	of	computers	to
transform	spoken	language	into	text	and	translate	it	into	other	languages,	even	in
real	time—which	is	why	we	can	now	talk	to	personal	digital	assistants	such	as
Siri,	Google	Now	and	Cortana.	Those	annoying	CAPTCHA	puzzles,	where	we
need	to	convince	a	website	that	we’re	human,	are	getting	ever	more	difficult	in
order	to	keep	ahead	of	what	machine-learning	technology	can	do.	In	2015,

Google	DeepMind	released	an	AI	system	using	deep	learning	that	was	able	to
master	dozens	of	computer	games	like	a	kid	would—with	no	instructions
whatsoever—except	that	it	soon	learned	to	play	better	than	any	human.	In	2016,
the	same	company	built	AlphaGo,	a	Go-playing	computer	system	that	used	deep
learning	to	evaluate	the	strength	of	different	board	positions	and	defeated	the
world’s	strongest	Go	champion.	This	progress	is	fueling	a	virtuous	circle,

bringing	ever	more	funding	and	talent	into	AI	research,	which	generates	further
progress.
We’ve	spent	this	chapter	exploring	the	nature	of	intelligence	and	its
development	up	until	now.	How	long	will	it	take	until	machines	can	out-compete
us	at	
all
	cognitive	tasks?	We	clearly	don’t	know,	and	need	to	be	open	to	the
possibility	that	the	answer	may	be	“never.”	However,	a	basic	message	of	this
chapter	is	that	we	also	need	to	consider	the	possibility	that	it	
will
	happen,
perhaps	even	in	our	lifetime.	After	all,	matter	can	be	arranged	so	that	when	it
obeys	the	laws	of	physics,	it	remembers,	computes	and	learns—and	the	matter
doesn’t	need	to	be	biological.	AI	researchers	have	often	been	accused	of	over-
promising	and	under-delivering,	but	in	fairness,	some	of	their	critics	don’t	have
the	best	track	record	either.	Some	keep	moving	the	goalposts,	effectively
defining	intelligence	as	that	which	computers	still	can’t	do,	or	as	that	which
impresses	us.	Machines	are	now	good	or	excellent	at	arithmetic,	chess,

mathematical	theorem	proving,	stock	picking,	image	captioning,	driving,	arcade
game	playing,	Go,	speech	synthesis,	speech	transcription,	translation	and	cancer
diagnosis,	but	some	critics	will	scornfully	scoff	“Sure—but	that’s	not	
real
intelligence!”	They	might	go	on	to	argue	that	real	intelligence	involves	only	the
mountaintops	in	Moravec’s	landscape	(
figure	2.2
)	that	haven’t	yet	been
submerged,	just	as	some	people	in	the	past	used	to	argue	that	image	captioning
and	Go	should	count—while	the	water	kept	rising.

real
figure	2.2
Assuming	that	the	water	will	keep	rising	for	at	least	a	while	longer,	AI’s
impact	on	society	will	keep	growing.	Long	before	AI	reaches	human	level	across
all	tasks,	it	will	give	us	fascinating	opportunities	and	challenges	involving	issues
such	as	bugs,	laws,	weapons	and	jobs.	What	are	they	and	how	can	we	best
prepare	for	them?	Let’s	explore	this	in	the	next	chapter.

•
Intelligence,	defined	as	ability	to	accomplish	complex	goals,	can’t	be	measured	by	a
single	IQ,	only	by	an	ability	spectrum	across	all	goals.
•
Today’s	artificial	intelligence	tends	to	be	
narrow,
	with	each	system	able	to
accomplish	only	very	specific	goals,	while	human	intelligence	is	remarkably	
broad
.
•
Memory,	computation,	learning	and	intelligence	have	an	abstract,	intangible	and
ethereal	feel	to	them	because	they’re	
substrate-independent:
	able	to	take	on	a	life	of
their	own	that	doesn’t	depend	on	or	reflect	the	details	of	their	underlying	material
substrate.
•
Any	chunk	of	matter	can	be	the	substrate	for	
memory
	as	long	as	it	has	many	different
stable	states.
•
Any	matter	can	be	
computronium,
	the	substrate	for	
computation,
	as	long	as	it
contains	certain	universal	building	blocks	that	can	be	combined	to	implement	any
function.	NAND	gates	and	neurons	are	two	important	examples	of	such	universal
“computational	atoms.”
•
A	neural	network	is	a	powerful	substrate	for	
learning

•
memory
•
•
learning
	because,	simply	by	obeying
the	laws	of	physics,	it	can	rearrange	itself	to	get	better	and	better	at	implementing
desired	computations.
•
Because	of	the	striking	simplicity	of	the	laws	of	physics,	we	humans	only	care	about
a	tiny	fraction	of	all	imaginable	computational	problems,	and	neural	networks	tend
to	be	remarkably	good	at	solving	precisely	this	tiny	fraction.
•
Once	technology	gets	twice	as	powerful,	it	can	often	be	used	to	design	and	build
technology	that’s	twice	as	powerful	in	turn,	triggering	repeated	capability	doubling

•
•
in	the	spirit	of	Moore’s	law.	The	cost	of	information	technology	has	now	halved
roughly	every	two	years	for	about	a	century,	enabling	the	information	age.
•
If	AI	progress	continues,	then	long	before	AI	reaches	human	level	for	all	skills,	it
will	give	us	fascinating	opportunities	and	challenges	involving	issues	such	as	bugs,
laws,	weapons	and	jobs—which	we’ll	explore	in	the	next	chapter.
*1
	
To	see	this,	imagine	how	you’d	react	if	someone	claimed	that	the	ability	to	accomplish	Olympic-level

•
*1
	
athletic	feats	could	be	quantified	by	a	single	number	called	the	“athletic	quotient,”	or	AQ	for	short,	so	that
the	Olympian	with	the	highest	AQ	would	win	the	gold	medals	in	all	the	sports.
*2
	
Some	people	prefer	“human-level	AI”	or	“strong	AI”	as	synonyms	for	AGI,	but	both	are	problematic.
Even	a	pocket	calculator	is	a	human-level	AI	in	the	narrow	sense.	The	antonym	of	“strong	AI”	is	“weak
AI,”	but	it	feels	odd	to	call	narrow	AI	systems	such	as	Deep	Blue,	Watson,	and	AlphaGo	“weak.”

*3
	
NAND	is	short	for	NOT	AND:	An	AND	gate	outputs	1	only	if	the	first	input	is	1	and	the	second	input	is
1,	so	NAND	outputs	the	exact	opposite.
*4
	
I’m	using	“well-defined	function”	to	mean	what	mathematicians	and	computer	scientists	call	a
“computable	function,”	i.e.,	a	function	that	could	be	computed	by	some	hypothetical	computer	with
unlimited	memory	and	time.	Alan	Turing	and	Alonzo	Church	famously	proved	that	there	are	also
functions	that	can	be	described	but	aren’t	computable.
*5
	
In	case	you	like	math,	two	popular	choices	of	this	activation	function	are	the	so-called	sigmoid	function
σ
(
x
)	
≡
	1/(1	
+
	e
−
x
)	and	the	ramp	function	
σ
(
x
)	
=
	max{0,	
x
},	although	it’s	been	proven	that	almost	any
function	will	suffice	as	long	as	it’s	not	linear	(a	straight	line).	Hopfield’s	famous	model	uses	
σ
(
x
)	
=
	
−
1	if	
x
<
	0	and	
σ
(
x
)	
=
	1	if	
x
	
≥
	0.	If	the	neuron	states	are	stored	in	a	vector,	then	the	network	is	updated	by	simply

*5
	
σ
(
x
)	
≡
	1/(1	
+
	e
−
x
σ
(
x
)	
=
	max{0,	
x
σ
(
x
)	
=
	
−
1	if	
x
<
	0	and	
σ
(
x
)	
=
	1	if	
x
	
≥
multiplying	that	vector	by	a	matrix	storing	the	synaptic	couplings	and	then	applying	the	function	
σ
	to	all
elements.

Chapter	3
The	Near	Future:	Breakthroughs,	Bugs,	Laws,
Weapons	and	Jobs
If	we	don’t	change	direction	soon,	we’ll	end	up	where	we’re	going.
Irwin	Corey
What	does	it	mean	to	be	human	in	the	present	day	and	age?	For	example,	what	is
it	that	we	really	value	about	ourselves,	that	makes	us	different	from	other	life
forms	and	machines?	What	do	other	people	value	about	us	that	makes	some	of
them	willing	to	offer	us	jobs?	Whatever	our	answers	are	to	these	questions	at	any
one	time,	it’s	clear	that	the	rise	of	technology	must	gradually	change	them.
Take	me,	for	instance.	As	a	scientist,	I	take	pride	in	setting	my	own	goals,	in
using	creativity	and	intuition	to	tackle	a	broad	range	of	unsolved	problems,	and
in	using	language	to	share	what	I	discover.	Fortunately	for	me,	society	is	willing
to	pay	me	to	do	this	as	a	job.	Centuries	ago,	I	might	instead,	like	many	others,
have	built	my	identity	around	being	a	farmer	or	craftsman,	but	the	growth	of
technology	has	since	reduced	such	professions	to	a	tiny	fraction	of	the

workforce.	This	means	that	it’s	no	longer	possible	for	everyone	to	build	their
identity	around	farming	or	crafts.
Personally,	it	doesn’t	bother	me	that	today’s	machines	outclass	me	at	manual
skills	such	as	digging	and	knitting,	since	these	are	neither	hobbies	of	mine	nor
my	sources	of	income	or	self-worth.	Indeed,	any	delusions	I	may	have	held
about	my	abilities	in	that	regard	were	crushed	at	age	eight,	when	my	school
forced	me	to	take	a	knitting	class	which	I	nearly	flunked,	and	I	completed	my

project	only	thanks	to	a	compassionate	helper	from	fifth	grade	taking	pity	on	me.
But	as	technology	keeps	improving,	will	the	rise	of	AI	eventually	eclipse	also
those	abilities	that	provide	my	current	sense	of	self-worth	and	value	on	the	job
market?	Stuart	Russell	told	me	that	he	and	many	of	his	fellow	AI	researchers	had
recently	experienced	a	“holy	shit!”	moment,	when	they	witnessed	AI	doing
something	they	weren’t	expecting	to	see	for	many	years.	In	that	spirit,	please	let
me	tell	you	about	a	few	of	my	own	HS	moments,	and	how	I	see	them	as
harbingers	of	human	abilities	soon	to	be	overtaken.

Deep	Reinforcement	Learning	Agents
I	experienced	one	of	my	major	jaw	drops	in	2014	while	watching	a	video	of	a
DeepMind	AI	system	learning	to	play	computer	games.	Specifically,	the	AI	was
playing	Breakout	(see	
figure	3.1
),	a	classic	Atari	game	I	remember	fondly	from
my	teens.	The	goal	is	to	maneuver	a	paddle	so	as	to	repeatedly	bounce	a	ball	off
a	brick	wall;	every	time	you	hit	a	brick,	it	disappears	and	your	score	increases.

Figure	3.1:	After	learning	to	play	the	Atari	game	Breakout	from	scratch,	using	deep
reinforcement	learning	to	maximize	the	score,	the	DeepMind	AI	discovered	the	optimal	strategy:
drilling	a	hole	through	the	leftmost	part	of	the	brick	wall	and	letting	the	ball	keep	bouncing
around	behind	it,	amassing	points	very	rapidly.	I’ve	drawn	arrows	showing	the	past	trajectories
of	ball	and	paddle.
I’d	written	some	computer	games	of	my	own	back	in	the	day,	and	was	well
aware	that	it	wasn’t	hard	to	write	a	program	that	could	play	Breakout—but	this
was	not	what	the	DeepMind	team	had	done.	Instead,	they’d	created	a	blank-slate
AI	that	knew	nothing	about	this	game—or	about	any	other	games,	or	even	about
concepts
	such	as	games,	paddles,	bricks	or	balls.	All	their	AI	knew	was	that	a
long	list	of	numbers	got	fed	into	it	at	regular	intervals:	the	current	score	and	a
long	list	of	numbers	which	we	(but	not	the	AI)	would	recognize	as	specifications
of	how	different	parts	of	the	screen	were	colored.	The	AI	was	simply	told	to

concepts
maximize	the	score	by	outputting,	at	regular	intervals,	numbers	which	we	(but
not	the	AI)	would	recognize	as	codes	for	which	keys	to	press.
Initially,	the	AI	played	terribly:	it	cluelessly	jiggled	the	paddle	back	and	forth
seemingly	at	random	and	missed	the	ball	almost	every	time.	After	a	while,	it
seemed	to	be	getting	the	idea	that	moving	the	paddle	toward	the	ball	was	a	good

idea,	even	though	it	still	missed	most	of	the	time.	But	it	kept	improving	with
practice,	and	soon	got	better	at	the	game	than	I’d	ever	been,	infallibly	returning
the	ball	no	matter	how	fast	it	approached.	And	then	my	jaw	dropped:	it	figured
out	this	amazing	score-maximizing	strategy	of	always	aiming	for	the	upper-left
corner	to	drill	a	hole	through	the	wall	and	let	the	ball	get	stuck	bouncing
between	the	back	of	the	wall	and	the	barrier	behind	it.	This	felt	like	a	really
intelligent	thing	to	do.	Indeed,	Demis	Hassabis	later	told	me	that	the
programmers	on	that	DeepMind	team	didn’t	know	this	trick	until	they	learned	it
from	the	AI	they’d	built.	I	recommend	watching	a	video	of	this	for	yourself	at
the	link	I’ve	provided.
1
There	was	a	human-like	feature	to	this	that	I	found	somewhat	unsettling:	I	was
watching	an	AI	that	had	a	goal	and	learned	to	get	ever	better	at	achieving	it,
eventually	outperforming	its	creators.	In	the	previous	chapter,	we	defined

1
intelligence	as	simply	the	ability	to	accomplish	complex	goals,	so	in	this	sense,
DeepMind’s	AI	was	growing	more	intelligent	in	front	of	my	eyes	(albeit	merely
in	the	very	narrow	sense	of	playing	this	particular	game).	In	the	first	chapter,	we
encountered	what	computer	scientists	call	
intelligent	agents:
	entities	that	collect
information	about	their	environment	from	sensors	and	then	process	this
information	to	decide	how	to	act	back	on	their	environment.	Although

DeepMind’s	game-playing	AI	lived	in	an	extremely	simple	virtual	world
composed	of	bricks,	paddles	and	balls,	I	couldn’t	deny	that	it	was	an	intelligent
agent.
DeepMind	soon	published	their	method	and	shared	their	code,	explaining	that
it	used	a	very	simple	yet	powerful	idea	called	
deep	reinforcement	learning
.
2
Basic	reinforcement	learning	is	a	classic	machine	learning	technique	inspired	by
behaviorist	psychology,	where	getting	a	positive	reward	increases	your	tendency

agent.
.
2
to	do	something	again	and	vice	versa.	Just	like	a	dog	learns	to	do	tricks	when
this	increases	the	likelihood	of	its	getting	encouragement	or	a	snack	from	its
owner	soon,	DeepMind’s	AI	learned	to	move	the	paddle	to	catch	the	ball	because
this	increased	the	likelihood	of	its	getting	more	points	soon.	DeepMind
combined	this	idea	with	deep	learning:	they	trained	a	deep	neural	net,	as	in	the
previous	chapter,	to	predict	how	many	points	would	on	average	be	gained	by
pressing	each	of	the	allowed	keys	on	the	keyboard,	and	then	the	AI	selected

whatever	key	the	neural	net	rated	as	most	promising	given	the	current	state	of
the	game.
When	I	listed	traits	contributing	to	my	own	personal	feeling	of	self-worth	as	a

human,	I	included	the	ability	to	tackle	a	broad	range	of	unsolved	problems.	In
contrast,	being	able	to	play	Breakout	and	do	nothing	else	constitutes	extremely
narrow	intelligence.	To	me,	the	true	importance	of	DeepMind’s	breakthrough	is
that	deep	reinforcement	learning	is	a	completely	general	technique.	Sure	enough,
they	let	the	exact	same	AI	practice	playing	forty-nine	different	Atari	games,	and
it	learned	to	outplay	their	human	testers	on	twenty-nine	of	them,	from	Pong	to
Boxing,	Video	Pinball	and	Space	Invaders.
It	didn’t	take	long	until	the	same	AI	idea	had	started	proving	itself	on	more
modern	games	whose	worlds	were	three-dimensional	rather	than	two-
dimensional.	Soon	DeepMind’s	San	Francisco–based	competitors	at	OpenAI
released	a	platform	called	Universe,	where	DeepMind’s	AI	and	other	intelligent
agents	can	practice	interacting	with	an	entire	computer	as	if	it	were	a	game:
clicking	on	anything,	typing	anything,	and	opening	and	running	whatever

software	they’re	able	to	navigate—firing	up	a	web	browser	and	messing	around
online,	for	example.
Looking	to	the	future	of	deep	reinforcement	learning	and	improvements
thereupon,	there’s	no	obvious	end	in	sight.	The	potential	isn’t	limited	to	virtual
game	worlds,	since	if	you’re	a	robot,	life	itself	can	be	viewed	as	a	game.	Stuart
Russell	told	me	that	his	first	major	HS	moment	was	watching	the	robot	Big	Dog
run	up	a	snow-covered	forest	slope,	elegantly	solving	the	legged	locomotion

problem	that	he	himself	had	struggled	to	solve	for	many	years.
3
	Yet	when	that
milestone	was	reached	in	2008,	it	involved	huge	amounts	of	work	by	clever
programmers.	After	DeepMind’s	breakthrough,	there’s	no	reason	why	a	robot
can’t	ultimately	use	some	variant	of	deep	reinforcement	learning	to	teach	itself
to	walk	without	help	from	human	programmers:	all	that’s	needed	is	a	system	that
gives	it	points	whenever	it	makes	progress.	Robots	in	the	real	world	similarly

3
have	the	potential	to	learn	to	swim,	fly,	play	ping-pong,	fight	and	perform	a
nearly	endless	list	of	other	motor	tasks	without	help	from	human	programmers.
To	speed	things	up	and	reduce	the	risk	of	getting	stuck	or	damaging	themselves
during	the	learning	process,	they	would	probably	do	the	first	stages	of	their
learning	in	virtual	reality.

Intuition,	Creativity	and	Strategy
Another	defining	moment	for	me	was	when	the	DeepMind	AI	system	AlphaGo
won	a	five-game	Go	match	against	Lee	Sedol,	generally	considered	the	top
player	in	the	world	in	the	early	twenty-first	century.
It	was	widely	expected	that	human	Go	players	would	be	dethroned	by
machines	at	some	point,	since	it	had	happened	to	their	chess-playing	colleagues
two	decades	earlier.	However,	most	Go	pundits	predicted	that	it	would	take
another	decade,	so	AlphaGo’s	triumph	was	a	pivotal	moment	for	them	as	well	as
for	me.	Nick	Bostrom	and	Ray	Kurzweil	have	both	emphasized	how	hard	it	can
be	to	see	AI	breakthroughs	coming,	which	is	evident	from	interviews	with	Lee
Sedol	himself	before	and	after	losing	the	first	three	games:
•
October	2015:	“Based	on	its	level	seen…I	think	I	will	win	the	game	by	a
near	landslide.”
•
February	2016:	“I	have	heard	that	Google	DeepMind’s	AI	is	surprisingly
strong	and	getting	stronger,	but	I	am	confident	that	I	can	win	at	least	this
time.”
•

•
•
time.”
•
March	9,	2016:	“I	was	very	surprised	because	I	didn’t	think	I	would
lose.”
•
March	10,	2016:	“I’m	quite	speechless…I	am	in	shock.	I	can	admit
that…the	third	game	is	not	going	to	be	easy	for	me.”
•
March	12,	2016:	“I	kind	of	felt	powerless.”
Within	a	year	after	playing	Lee	Sedol,	a	further	improved	AlphaGo	had
played	all	twenty	top	players	in	the	world	without	losing	a	single	match.
Why	was	this	such	a	big	deal	for	me	personally?	Well,	I	confessed	above	that
I	view	intuition	and	creativity	as	two	of	my	core	human	traits,	and	as	I’ll	now

lose.”
•
•
explain,	I	feel	that	AlphaGo	displayed	both.
Go	players	take	turns	placing	black	and	white	stones	on	a	19-by-19	board	(see
figure	3.2
).	There	are	vastly	more	possible	Go	positions	than	there	are	atoms	in
our	Universe,	which	means	that	trying	to	analyze	all	interesting	sequences	of
future	moves	rapidly	gets	hopeless.	Players	therefore	rely	heavily	on

subconscious	intuition	to	complement	their	conscious	reasoning,	with	experts
developing	an	almost	uncanny	feel	for	which	positions	are	strong	and	which	are
weak.	As	we	saw	in	the	last	chapter,	the	results	of	deep	learning	are	sometimes
reminiscent	of	intuition:	a	deep	neural	network	might	determine	that	an	image
portrays	a	cat	without	being	able	to	explain	why.	The	DeepMind	team	therefore
gambled	on	the	idea	that	deep	learning	might	be	able	to	recognize	not	merely
cats,	but	also	strong	Go	positions.	The	core	idea	that	they	built	into	AlphaGo
was	to	marry	the	intuitive	power	of	deep	learning	with	the	logical	power	of
GOFAI—which	stands	for	what’s	humorously	known	as	“Good	Old-Fashioned
AI”	from	before	the	deep-learning	revolution.	They	used	a	massive	database	of
Go	positions	from	both	human	play	and	games	where	AlphaGo	had	played	a
clone	of	itself,	and	trained	a	deep	neural	network	to	predict	from	each	position
the	probability	that	white	would	ultimately	win.	They	also	trained	a	separate

network	to	predict	likely	next	moves.	They	then	combined	these	networks	with	a
GOFAI	method	that	cleverly	searched	through	a	pruned	list	of	likely	future-
move	sequences	to	identify	the	next	move	that	would	lead	to	the	strongest
position	down	the	road.

Figure	3.2:	DeepMind’s	AlphaGo	AI	made	a	highly	creative	move	on	line	5,	in	defiance	of
millennia	of	human	wisdom,	which	about	fifty	moves	later	proved	crucial	to	its	defeat	of	Go
legend	Lee	Sedol.
This	marriage	of	intuition	and	logic	gave	birth	to	moves	that	were	not	merely
powerful,	but	in	some	cases	also	highly	creative.	For	example,	millennia	of	Go
wisdom	dictate	that	early	in	the	game,	it’s	best	to	play	on	the	third	or	fourth	line
from	an	edge.	There’s	a	trade-off	between	the	two:	playing	on	the	third	line	helps
with	short-term	territory	gain	toward	the	side	of	the	board,	while	playing	on	the
fourth	helps	with	long-term	strategic	influence	toward	the	center.
In	the	thirty-seventh	move	of	the	second	game,	AlphaGo	shocked	the	Go
world	by	defying	that	ancient	wisdom	and	playing	on	the	fifth	line	(
figure	3.2
),
as	if	it	were	even	more	confident	than	a	human	in	its	long-term	planning	abilities
and	therefore	favored	strategic	advantage	over	short-term	gain.	Commentators

figure	3.2
),
were	stunned,	and	Lee	Sedol	even	got	up	and	temporarily	left	the	room.
4
	Sure
enough,	about	fifty	moves	later,	fighting	from	the	lower	left-hand	corner	of	the
board	ended	up	spilling	over	and	connecting	with	that	black	stone	from	move
thirty-seven!	And	that	motif	is	what	ultimately	won	the	game,	cementing	the
legacy	of	AlphaGo’s	fifth-row	move	as	one	of	the	most	creative	in	Go	history.

Because	of	its	intuitive	and	creative	aspects,	Go	is	viewed	more	as	an	art	form
than	just	another	game.	It	was	considered	one	of	the	four	“essential	arts”	in
ancient	China,	together	with	painting,	calligraphy	and	
qin
	music,	and	it	remains
hugely	popular	in	Asia,	with	almost	300	million	people	watching	the	first	game
between	AlphaGo	and	Lee	Sedol.	As	a	result,	the	Go	world	was	quite	shaken	by
the	outcome,	and	viewed	AlphaGo’s	victory	as	a	profound	milestone	for
humanity.	Ke	Jie,	the	world’s	top-ranked	Go	player	at	the	time,	had	this	to	say:
5
“Humanity	has	played	Go	for	thousands	of	years,	and	yet,	as	AI	has	shown	us,
we	have	not	yet	even	scratched	the	surface…The	union	of	human	and	computer
players	will	usher	in	a	new	era…Together,	man	and	AI	can	find	the	truth	of	Go.”
Such	fruitful	human-machine	collaboration	indeed	appears	promising	in	many
areas,	including	science,	where	AI	can	hopefully	help	us	humans	deepen	our
understanding	and	realize	our	ultimate	potential.

5
To	me,	AlphaGo	also	teaches	us	another	important	lesson	for	the	near	future:
combining	the	intuition	of	deep	learning	with	the	logic	of	GOFAI	can	produce
second-to-none	
strategy
.	Because	Go	is	one	of	the	ultimate	strategy	games,	AI	is
now	poised	to	graduate	and	challenge	(or	help)	the	best	human	strategists	even
beyond	game	boards—for	example	with	investment	strategy,	political	strategy
and	military	strategy.	Such	real-world	strategy	problems	are	typically
complicated	by	human	psychology,	missing	information	and	factors	that	need	to

strategy
be	modeled	as	random,	but	poker-playing	AI	systems	have	already	demonstrated
that	none	of	these	challenges	are	insurmountable.

Natural	Language
Yet	another	area	where	AI	progress	has	recently	stunned	me	is	language.	I	fell	in
love	with	travel	early	in	life,	and	curiosity	about	other	cultures	and	languages
formed	an	important	part	of	my	identity.	I	was	raised	speaking	Swedish	and
English,	was	taught	German	and	Spanish	in	school,	learned	Portuguese	and
Romanian	through	two	marriages	and	taught	myself	some	Russian,	French	and
Mandarin	for	fun.
But	the	AI	has	been	reaching,	and	after	an	important	discovery	in	2016,	there
are
	
almost	no	lazy	languages	that	I	can	translate	between	better	than	the	system
of	the	AI
	
developed	by	the	equipment	of	the	brain	of	Google.
Did	I	make	myself	crystal	clear?	I	was	actually	trying	to	say	this:
But	AI	has	been	catching	up	with	me,	and	after	a	major	breakthrough	in	2016,
there	are	almost	no	languages	left	that	I	can	translate	between	better	than	the	AI
system	developed	by	the	Google	Brain	team.
However,	I	first	translated	it	to	Spanish	and	back	using	an	app	that	I	installed

are
	
of	the	AI
	
on	my	laptop	a	few	years	ago.	In	2016,	the	Google	Brain	team	upgraded	their
free	Google	Translate	service	to	use	deep	recurrent	neural	networks,	and	the
improvement	over	older	GOFAI	systems	was	dramatic:
6
But	AI	has	been	catching	up	on	me,	and	after	a	breakthrough	in	2016,	there
are
	
almost	no	languages	left	that	can	translate	between	better	than	the	AI	system
developed	by	the	Google	Brain	team.
As	you	can	see,	the	pronoun	“I”	got	lost	during	the	Spanish	detour,	which

6
are
	
unfortunately	changed	the	meaning.	Close,	but	no	cigar!	However,	in	defense	of
Google’s	AI,	I’m	often	criticized	for	writing	unnecessarily	long	sentences	that
are	hard	to	parse,	and	I	picked	one	of	my	most	confusingly	convoluted	ones	for
this	example.	For	more	typical	sentences,	their	AI	often	translates	impeccably.
As	a	result,	it	created	quite	a	stir	when	it	came	out,	and	it’s	helpful	enough	to	be
used	by	hundreds	of	millions	of	people	daily.	Moreover,	courtesy	of	recent

progress	in	deep	learning	for	speech-to-text	and	text-to-speech	conversion,	these
users	can	now	speak	to	their	smartphones	in	one	language	and	listen	to	the
translated	result.
Natural	language	processing	is	now	one	of	the	most	rapidly	advancing	fields
of	AI,	and	I	think	that	further	success	will	have	a	large	impact	because	language

is	so	central	to	being	human.	The	better	an	AI	gets	at	linguistic	prediction,	the
better	it	can	compose	reasonable	email	responses	or	continue	a	spoken
conversation.	This	might,	at	least	to	an	outsider,	give	the	appearance	of	human
thought	taking	place.	Deep-learning	systems	are	thus	taking	baby	steps	toward
passing	the	famous	Turing	test,	where	a	machine	has	to	converse	well	enough	in
writing	to	trick	a	person	into	thinking	that	it	too	is	human.
Language-processing	AI	still	has	a	long	way	to	go,	though.	Although	I	must
confess	that	I	feel	a	bit	deflated	when	I’m	out-translated	by	an	AI,	I	feel	better
once	I	remind	myself	that,	so	far,	it	doesn’t	
understand
	what	it’s	saying	in	any
meaningful	sense.	From	being	trained	on	massive	data	sets,	it	discovers	patterns
and	relations	involving	words	without	ever	relating	these	words	to	anything	in
the	real	world.	For	example,	it	might	represent	each	word	by	a	list	of	a	thousand
numbers	that	specify	how	similar	it	is	to	certain	other	words.	It	may	then

conclude	from	this	that	the	difference	between	“king”	and	“queen”	is	similar	to
that	between	“husband”	and	“wife”—but	it	still	has	no	clue	what	it	means	to	be
male	or	female,	or	even	that	there	is	such	a	thing	as	a	physical	reality	out	there
with	space,	time	and	matter.
Since	the	Turing	test	is	fundamentally	about	deception,	it	has	been	criticized
for	testing	human	gullibility	more	than	true	artificial	intelligence.	In	contrast,	a
rival	test	called	the	
Winograd	Schema	Challenge
	goes	straight	for	the	jugular,

homing	in	on	that	commonsense	understanding	that	current	deep-learning
systems	tend	to	lack.	We	humans	routinely	use	real-world	knowledge	when
parsing	a	sentence,	to	figure	out	what	a	pronoun	refers	to.	For	example,	a	typical
Winograd	challenge	asks	what	“they”	refers	to	here:
1.
“The	city	councilmen	refused	the	demonstrators	a	permit	because	they
feared	violence.”
2.
advocated	violence.”
There’s	an	annual	AI	competition	to	answer	such	questions,	and	AIs	still
perform	miserably.
7

1.
2.
7
	This	precise	challenge,	understanding	what	refers	to	what,
torpedoed	even	GoogleTranslate	when	I	replaced	Spanish	with	Chinese	in	my
example	above:
But	the	AI	has	caught	up	with	me,	after	a	major	break	in	2016,	with	almost	no
language,	I	could	translate	the	AI	system	than	developed	by	the	Google	Brain

team.
Please	try	it	yourself	at	
https://translate.google.com
	now	that	you’re	reading
the	book	and	see	if	Google’s	AI	has	improved!	There’s	a	good	chance	that	it	has,
since	there	are	promising	approaches	out	there	for	marrying	deep	recurrent
neural	nets	with	GOFAI	to	build	a	language-processing	AI	that	includes	a	world
model.

Opportunities	and	Challenges
These	three	examples	were	obviously	just	a	sampler,	since	AI	is	progressing
rapidly	across	many	important	fronts.	Moreover,	although	I’ve	mentioned	only
two	companies	in	these	examples,	competing	research	groups	at	universities	and
other	companies	often	weren’t	far	behind.	A	loud	sucking	noise	can	be	heard	in
computer	science	departments	around	the	world	as	Apple,	Baidu,	DeepMind,
Facebook,	Google,	Microsoft	and	others	use	lucrative	offers	to	vacuum	off
students,	postdocs	and	faculty.
It’s	important	not	to	be	misled	by	the	examples	I’ve	given	into	viewing	the
history	of	AI	as	periods	of	stagnation	punctuated	by	the	occasional
breakthrough.	From	my	vantage	point,	I’ve	instead	been	seeing	fairly	steady
progress	for	a	long	time—which	the	media	report	as	a	breakthrough	whenever	it
crosses	the	threshold	of	enabling	a	new	imagination-grabbing	application	or
useful	product.	I	therefore	consider	it	likely	that	brisk	AI	progress	will	continue

for	many	years.	Moreover,	as	we	saw	in	the	last	chapter,	there’s	no	fundamental
reason	why	this	progress	can’t	continue	until	AI	matches	human	abilities	on
most	tasks.
Which	raises	the	question:	How	will	this	impact	us?	How	will	near-term	AI
progress	change	what	it	means	to	be	human?	We’ve	seen	that	it’s	getting
progressively	harder	to	argue	that	AI	completely	lacks	goals,	breadth,	intuition,
creativity	or	language—traits	that	many	feel	are	central	to	being	human.	This

means	that	even	in	the	near	term,	long	before	any	AGI	can	match	us	at	all	tasks,
AI	might	have	a	dramatic	impact	on	how	we	view	ourselves,	on	what	we	can	do
when	complemented	by	AI	and	on	what	we	can	earn	money	doing	when
competing	against	AI.	Will	this	impact	be	for	the	better	or	for	the	worse?	What
near-term	opportunities	and	challenges	will	this	present?
Everything	we	love	about	civilization	is	the	product	of	human	intelligence,	so
if	we	can	amplify	it	with	artificial	intelligence,	we	obviously	have	the	potential

to	make	life	even	better.	Even	modest	progress	in	AI	might	translate	into	major
improvements	in	science	and	technology	and	corresponding	reductions	of
accidents,	disease,	injustice,	war,	drudgery	and	poverty.	But	in	order	to	reap
these	benefits	of	AI	without	creating	new	problems,	we	need	to	answer	many
important	questions.	For	example:

1.
How	can	we	make	future	AI	systems	more	robust	than	today’s,	so	that	they
do	what	we	want	without	crashing,	malfunctioning	or	getting	hacked?
2.
How	can	we	update	our	legal	systems	to	be	more	fair	and	efficient	and	to
keep	pace	with	the	rapidly	changing	digital	landscape?
3.
How	can	we	make	weapons	smarter	and	less	prone	to	killing	innocent
civilians	without	triggering	an	out-of-control	arms	race	in	lethal
autonomous	weapons?
4.
How	can	we	grow	our	prosperity	through	automation	without	leaving
people	lacking	income	or	purpose?
Let’s	devote	the	rest	of	this	chapter	to	exploring	each	of	these	questions	in
turn.	These	four	near-term	questions	are	aimed	mainly	at	computer	scientists,
legal	scholars,	military	strategists	and	economists,	respectively.	However,	to	help
get	the	answers	we	need	by	the	time	we	need	them,	everybody	needs	to	join	this
conversation,	because	as	we’ll	see,	the	challenges	transcend	all	traditional
boundaries—both	between	specialties	and	between	nations.

Information	technology	has	already	had	great	positive	impact	on	virtually	every
sector	of	our	human	enterprise,	from	science	to	finance,	manufacturing,
transportation,	healthcare,	energy	and	communication,	and	this	impact	pales	in
comparison	to	the	progress	that	AI	has	the	potential	to	bring.	But	the	more	we
come	to	rely	on	technology,	the	more	important	it	becomes	that	it’s	robust	and
trustworthy,	doing	what	we	want	it	to	do.
Throughout	human	history,	we’ve	relied	on	the	same	tried-and-true	approach
to	keeping	our	technology	beneficial:	learning	from	mistakes.	We	invented	fire,
repeatedly	messed	up,	and	then	invented	the	fire	extinguisher,	fire	exit,	fire
alarm	and	fire	department.	We	invented	the	automobile,	repeatedly	crashed,	and
then	invented	seat	belts,	air	bags	and	self-driving	cars.	Up	until	now,	our
technologies	have	typically	caused	sufficiently	few	and	limited	accidents	for
their	harm	to	be	outweighed	by	their	benefits.	As	we	inexorably	develop	ever

more	powerful	technology,	however,	we’ll	inevitably	reach	a	point	where	even	a
single	accident	could	be	devastating	enough	to	outweigh	all	benefits.	Some
argue	that	accidental	global	nuclear	war	would	constitute	such	an	example.
Others	argue	that	a	bioengineered	pandemic	could	qualify,	and	in	the	next
chapter,	we’ll	explore	the	controversy	around	whether	future	AI	could	cause
human	extinction.	But	we	need	not	consider	such	extreme	examples	to	reach	a
crucial	conclusion:	as	technology	grows	more	powerful,	we	should	rely	less	on

the	trial-and-error	approach	to	safety	engineering.	In	other	words,	
we	should
become	more	proactive	than	reactive,
	investing	in	safety	research	aimed	at
preventing	accidents	from	happening	even	once.	This	is	why	society	invests
more	in	nuclear-reactor	safety	than	mousetrap	safety.
This	is	also	the	reason	why,	as	we	saw	in	chapter	1,	there	was	strong
community	interest	in	AI-safety	research	at	the	Puerto	Rico	conference.
Computers	and	AI	systems	have	always	crashed,	but	this	time	is	different:	AI	is

we	should
gradually	entering	the	real	world,	and	it’s	not	merely	a	nuisance	if	it	crashes	the
power	grid,	the	stock	market	or	a	nuclear	weapons	system.	In	the	rest	of	this
section,	I	want	to	introduce	you	to	the	four	main	areas	of	technical	AI-safety
research	that	are	dominating	the	current	AI-safety	discussion	and	that	are	being
pursued	around	the	world:	
verification
,	
validation
,	
security
	and	
control
.
*1
	To

prevent	things	from	getting	too	nerdy	and	dry,	let’s	do	this	by	exploring	past
successes	and	failures	of	information	technology	in	different	areas,	as	well	as
valuable	lessons	we	can	learn	from	them	and	research	challenges	that	they	pose.
Although	most	of	these	stories	are	old,	involving	low-tech	computer	systems
that	almost	nobody	would	refer	to	as	AI	and	that	caused	few,	if	any,	casualties,
we’ll	see	that	they	nonetheless	teach	us	valuable	lessons	for	designing	safe	and
powerful	future	AI	systems	whose	failures	could	be	truly	catastrophic.

AI	for	Space	Exploration
Let’s	start	with	something	close	to	my	heart:	space	exploration.	Computer
technology	has	enabled	us	to	fly	people	to	the	Moon	and	to	send	unmanned
spacecraft	to	explore	all	the	planets	of	our	Solar	System,	even	landing	on
Saturn’s	moon	Titan	and	on	a	comet.	As	we’ll	explore	in	chapter	6,	future	AI
may	help	us	explore	other	solar	systems	and	galaxies—if	it’s	bug-free.	On	June
4,	1996,	scientists	hoping	to	research	Earth’s	magnetosphere	cheered	jubilantly
as	an	Ariane	5	rocket	from	the	European	Space	Agency	roared	into	the	sky	with
the	scientific	instruments	they’d	built.	Thirty-seven	seconds	later,	their	smiles
vanished	as	the	rocket	exploded	in	a	fireworks	display	costing	hundreds	of
millions	of	dollars.
8
	The	cause	was	found	to	be	buggy	software	manipulating	a
number	that	was	too	large	to	fit	into	the	16	bits	allocated	for	it.
9
	Two	years	later,
NASA’s	Mars	Climate	Orbiter	accidentally	entered	the	Red	Planet’s	atmosphere

8
9
and	disintegrated	because	two	different	parts	of	the	software	used	different	units
for	force,	causing	a	445%	error	in	the	rocket-engine	thrust	control.
10
	This	was
NASA’s	second	super-expensive	bug:	their	Mariner	1	mission	to	Venus	exploded
after	launch	from	Cape	Canaveral	on	July	22,	1962,	after	the	flight-control
software	was	foiled	by	an	incorrect	punctuation	mark.
11
	As	if	to	show	that	not
only	westerners	had	mastered	the	art	of	launching	bugs	into	space,	the	Soviet

10
	This	was
11
Phobos	1	mission	failed	on	September	2,	1988.	This	was	the	heaviest
interplanetary	spacecraft	ever	launched,	with	the	spectacular	goal	of	deploying	a
lander	on	Mars’	moon	Phobos—all	thwarted	when	a	missing	hyphen	caused	the
“end-of-mission”	command	to	be	sent	to	the	spacecraft	while	it	was	en	route	to
Mars,	shutting	down	all	of	its	systems.
12
What	we	learn	from	these	examples	is	the	importance	of	what	computer
scientists	call	
verification:
	ensuring	that	software	fully	satisfies	all	the	expected

12
requirements.	The	more	lives	and	resources	are	at	stake,	the	higher	confidence
we	want	that	the	software	will	work	as	intended.	Fortunately,	AI	can	help
automate	and	improve	the	verification	process.	For	example,	a	complete,
general-purpose	operating-system	kernel	called	
seL4
	has	recently	been
mathematically	checked	against	a	formal	specification	to	give	a	strong	guarantee
against	crashes	and	unsafe	operations:	although	it	doesn’t	yet	come	with	the

ensuring	that	software	fully	satisfies	all	the	expected
seL4
bells	and	whistles	of	Microsoft	Windows	and	Mac	OS,	you	can	rest	assured	that
it	won’t	give	you	what’s	affectionately	known	as	“the	blue	screen	of	death”	or

“the	spinning	wheel	of	doom.”	The	U.S.	Defense	Advanced	Research	Projects
Agency	(DARPA)	has	sponsored	the	development	of	a	set	of	open-source	high-
assurance	tools	called	HACMS	(high-assurance	cyber	military	systems)	that	are
provably	safe.	An	important	challenge	is	to	make	such	tools	sufficiently
powerful	and	easy	to	use	that	they’ll	get	widely	deployed.	Another	challenge	is
that	the	very	task	of	verification	will	itself	get	more	difficult	as	software	moves
into	robots	and	new	environments,	and	as	traditional	preprogrammed	software
gets	replaced	by	AI	systems	that	keep	learning,	thereby	changing	their	behavior,
as	in	chapter	2.

AI	for	Finance
Finance	is	another	area	that’s	been	transformed	by	information	technology,
allowing	resources	to	be	efficiently	reallocated	across	the	globe	at	the	speed	of
light	and	enabling	affordable	financing	for	everything	from	mortgages	to	startup
companies.	Progress	in	AI	is	likely	to	offer	great	future	profit	opportunities	from
financial	trading:	most	stock	market	buy/sell	decisions	are	now	made
automatically	by	computers,	and	my	graduating	MIT	students	routinely	get
tempted	by	astronomical	starting	salaries	to	improve	algorithmic	trading.
Verification	is	important	for	financial	software	as	well,	which	the	American
firm	Knight	Capital	learned	the	hard	way	on	August	1,	2012,	by	losing	$440
million	in	forty-five	minutes	after	deploying	unverified	trading	software.
13
	The
trillion-dollar	“Flash	Crash”	of	May	6,	2010,	was	noteworthy	for	a	different
reason.	Although	it	caused	massive	disruptions	for	about	half	an	hour	before
markets	stabilized,	with	shares	of	some	prominent	companies	such	Procter	&

13
	The
Gamble	swinging	in	price	between	a	penny	and	$100,000,
14
	the	problem	wasn’t
caused	by	bugs	or	computer	malfunctions	that	verification	could	have	avoided.
Instead,	it	was	caused	by	expectations	being	violated:	automatic	trading
programs	from	many	companies	found	themselves	operating	in	an	unexpected
situation	where	their	assumptions	weren’t	valid—for	example,	the	assumption
that	if	a	stock	exchange	computer	reported	that	a	stock	had	a	price	of	one	cent,
then	that	stock	really	was	worth	one	cent.

14
The	flash	crash	illustrates	the	importance	of	what	computer	scientists	call
validation:
	whereas	verification	asks	“Did	I	build	the	system	right?,”	validation
asks	“Did	I	build	the	right	system?”
*2
	For	example,	does	the	system	rely	on
assumptions	that	might	not	always	be	valid?	If	so,	how	can	it	be	improved	to
better	handle	uncertainty?

AI	for	Manufacturing
Needless	to	say,	AI	holds	great	potential	for	improving	manufacturing,	by
controlling	robots	that	enhance	both	efficiency	and	precision.	Ever-improving	3-
D	printers	can	now	make	prototypes	of	anything	from	office	buildings	to
micromechanical	devices	smaller	than	a	salt	grain.
15
	While	huge	industrial
robots	build	cars	and	airplanes,	affordable	computer-controlled	mills,	lathes,
cutters	and	the	like	are	powering	not	merely	factories,	but	also	the	grassroots
“maker	movement,”	where	local	enthusiasts	materialize	their	ideas	at	over	a
thousand	community-run	“fab	labs”	around	the	world.
16
	But	the	more	robots	we
have	around	us,	the	more	important	it	becomes	that	we	verify	and	validate	their
software.	The	first	person	known	to	have	been	killed	by	a	robot	was	Robert
Williams,	a	worker	at	a	Ford	plant	in	Flat	Rock,	Michigan.	In	1979,	a	robot	that
was	supposed	to	retrieve	parts	from	a	storage	area	malfunctioned,	and	he

16
climbed	into	the	area	to	get	the	parts	himself.	The	robot	silently	began	operating
and	smashed	his	head,	continuing	for	thirty	minutes	until	his	co-workers
discovered	what	had	happened.
17
	The	next	robot	victim	was	Kenji	Urada,	a
maintenance	engineer	at	a	Kawasaki	plant	in	Akashi,	Japan.	While	working	on	a
broken	robot	in	1981,	he	accidentally	hit	its	on	switch	and	was	crushed	to	death
by	the	robot’s	hydraulic	arm.
18
	In	2015,	a	twenty-two-year-old	contractor	at	one
of	Volkswagen’s	production	plants	in	Baunatal,	Germany,	was	working	on

17
18
setting	up	a	robot	to	grab	auto	parts	and	manipulate	them.	Something	went
wrong,	causing	the	robot	to	grab	him	and	crush	him	to	death	against	a	metal
plate.
19
Although	these	accidents	are	tragic,	it’s	important	to	note	that	they	make	up	a
minuscule	fraction	of	all	industrial	accidents.	Moreover,	industrial	accidents
have	
decreased
	rather	than	increased	as	technology	has	improved,	dropping	from
about	14,000	deaths	in	1970	to	4,821	in	2014	in	the	United	States.
20
	The	three
above-mentioned	accidents	show	that	adding	intelligence	to	otherwise	dumb

plate.
19
have	
decreased
20
	The	three
machines	should	be	able	to	further	improve	industrial	safety,	by	having	robots
learn	to	be	more	careful	around	people.	All	three	accidents	could	have	been
avoided	with	better	validation:	the	robots	caused	harm	not	because	of	bugs	or
malice,	but	because	they	made	invalid	assumptions—that	the	person	wasn’t
present	or	that	the	person	was	an	auto	part.

Figure	3.3:	Whereas	traditional	industrial	robots	are	expensive	and	hard	to	program,	there’s	a
trend	toward	cheaper	AI-powered	ones	that	can	learn	what	to	do	from	workers	with	no
programming	experience.

AI	for	Transportation
Although	AI	can	save	many	lives	in	manufacturing,	it	can	potentially	save	even
more	in	transportation.	Car	accidents	alone	took	over	1.2	million	lives	in	2015,
and	aircraft,	train	and	boat	accidents	together	killed	thousands	more.	In	the
United	States,	with	its	high	safety	standards,	motor	vehicle	accidents	killed
about	35,000	people	last	year—seven	times	more	than	all	industrial	accidents
combined.
21
	When	we	had	a	panel	discussion	about	this	in	Austin,	Texas,	at	the
2016	annual	meeting	of	the	Association	for	the	Advancement	of	Artificial
Intelligence,	the	Israeli	computer	scientist	Moshe	Vardi	got	quite	emotional
about	it	and	argued	that	not	only	
could
	AI	reduce	road	fatalities,	but	it	
must:
	“It’s
a	moral	imperative!”	he	exclaimed.	Because	almost	all	car	crashes	are	caused	by
human	error,	it’s	widely	believed	that	AI-powered	self-driving	cars	can	eliminate
at	least	90%	of	road	deaths,	and	this	optimism	is	fueling	great	progress	toward

could
must:
	“It’s
actually	getting	self-driving	cars	out	on	the	roads.	Elon	Musk	envisions	that
future	self-driving	cars	will	not	only	be	safer,	but	will	also	earn	money	for	their
owners	while	they’re	not	needed,	by	competing	with	Uber	and	Lyft.
So	far,	self-driving	cars	do	indeed	have	a	better	safety	record	than	human
drivers,	and	the	accidents	that	have	occurred	underscore	the	importance	and
difficulty	of	validation.	The	first	fender	bender	caused	by	a	Google	self-driving
car	took	place	on	February	14,	2016,	because	it	made	an	incorrect	assumption

about	a	bus:	that	its	driver	would	yield	when	the	car	pulled	out	in	front	of	it.	The
first	lethal	crash	caused	by	a	self-driving	Tesla,	which	rammed	into	the	trailer	of
a	truck	crossing	the	highway	on	May	7,	2016,	was	caused	by	two	bad
assumptions:
22
	that	the	bright	white	side	of	the	trailer	was	merely	part	of	the
bright	sky,	and	that	the	driver	(who	was	allegedly	watching	a	Harry	Potter
movie)	was	paying	attention	and	would	intervene	if	something	went	wrong.
*3
But	sometimes	good	verification	and	validation	aren’t	enough	to	avoid

22
*3
accidents,	because	we	also	need	good	
control:
	ability	for	a	human	operator	to
monitor	the	system	and	change	its	behavior	if	necessary.	For	such	
human-in-the-
loop
	systems	to	work	well,	it’s	crucial	that	the	human-machine	communication
be	effective.	In	this	spirit,	a	red	light	on	your	dashboard	will	conveniently	alert
you	if	you	accidentally	leave	the	trunk	of	your	car	open.	In	contrast,	when	the
British	car	ferry	
Herald
	
of	Free	Enterprise
	left	the	harbor	of	Zeebrugge	on
March	6,	1987,	with	her	bow	doors	open,	there	was	no	warning	light	or	other

visible	warning	for	the	captain,	and	the	ferry	capsized	soon	after	leaving	the
harbor,	killing	193	people.
23
Another	tragic	control	failure	that	might	have	been	avoided	by	better
machine-human	communication	occurred	during	the	night	of	June	1,	2009,	when
Air	France	Flight	447	crashed	into	the	Atlantic	Ocean,	killing	all	228	on	board.
According	to	the	official	accident	report,	“the	crew	never	understood	that	they
were	stalling	and	consequently	never	applied	a	recovery	manoeuvre”—which
would	have	involved	pushing	down	the	nose	of	the	aircraft—until	it	was	too	late.
Flight	safety	experts	speculated	that	the	crash	might	have	been	avoided	had	there
been	an	“angle-of-attack”	indicator	in	the	cockpit,	showing	the	pilots	that	the
nose	was	pointed	too	far	upward.
24
When	Air	Inter	Flight	148	crashed	into	the	Vosges	Mountains	near	Strasbourg
in	France	on	January	20,	1992,	killing	87	people,	the	cause	wasn’t	lack	of
machine-human	communication,	but	a	confusing	user	interface.	The	pilots

24
entered	“33”	on	a	keypad	because	they	wanted	to	descend	at	an	angle	of	3.3
degrees,	but	the	autopilot	interpreted	this	as	3,300	feet	per	minute	because	it	was
in	a	different	mode—and	the	display	screen	was	too	small	to	show	the	mode	and
allow	the	pilots	to	realize	their	mistake.

AI	for	Energy
Information	technology	has	done	wonders	for	power	generation	and	distribution,
with	sophisticated	algorithms	balancing	production	and	consumption	across	the
world’s	electrical	grids,	and	sophisticated	control	systems	keeping	power	plants
operating	safely	and	efficiently.	Future	AI	progress	is	likely	to	make	the	“smart
grid”	even	smarter,	to	optimally	adapt	to	changing	supply	and	demand	even
down	to	the	level	of	individual	rooftop	solar	panels	and	home-battery	systems.
But	on	Thursday,	August	14,	2003,	it	was	lights-out	for	about	55	million	people
in	the	United	States	and	Canada,	many	of	whom	remained	powerless	for	days.
Here,	too,	the	primary	cause	was	determined	to	be	failed	machine-human
communications:	a	software	bug	prevented	the	alarm	system	in	an	Ohio	control
room	from	alerting	operators	to	the	need	to	redistribute	power	before	a	minor
problem	(overloaded	transmission	lines	hitting	unpruned	foliage)	cascaded	out
of	control.
25

25
The	partial	nuclear	meltdown	in	a	reactor	on	Three	Mile	Island	in
Pennsylvania	on	March	28,	1979,	led	to	about	a	billion	dollars	in	cleanup	cost
and	a	major	backlash	against	nuclear	power.	The	final	accident	report	identified
multiple	contributing	factors,	including	confusion	caused	by	a	poor	user
interface.
26
	In	particular,	the	warning	light	that	the	operators	thought	indicated
whether	a	safety-critical	valve	was	open	or	closed	merely	indicated	whether	a
signal	had	been	sent	to	close	the	valve—so	the	operators	didn’t	realize	that	the

26
valve	had	gotten	stuck	open.
These	energy	and	transportation	accidents	teach	us	that	as	we	put	AI	in	charge
of	ever	more	physical	systems,	we	need	to	put	serious	research	efforts	into	not
only	making	the	machines	work	well	on	their	own,	but	also	into	making
machines	collaborate	effectively	with	their	human	controllers.	As	AI	gets
smarter,	this	will	involve	not	merely	building	good	user	interfaces	for
information	sharing,	but	also	figuring	out	how	to	optimally	allocate	tasks	within

human-computer	teams—for	example,	identifying	situations	where	control
should	be	transferred,	and	for	applying	human	judgment	efficiently	to	the
highest-value	decisions	rather	than	distracting	human	controllers	with	a	flood	of
unimportant	information.

AI	for	Healthcare
AI	has	huge	potential	for	improving	healthcare.	Digitization	of	medical	records
has	already	enabled	doctors	and	patients	to	make	faster	and	better	decisions,	and
to	get	instant	help	from	experts	around	the	world	in	diagnosing	digital	images.
Indeed,	the	best	experts	for	performing	such	diagnosis	may	soon	be	AI	systems,
given	the	rapid	progress	in	computer	vision	and	deep	learning.	For	example,	a
2015	Dutch	study	showed	that	computer	diagnosis	of	prostate	cancer	using
magnetic	resonance	imaging	(MRI)	was	as	good	as	that	of	human	radiologists,
27
and	a	2016	Stanford	study	showed	that	AI	could	diagnose	lung	cancer	using
microscope	images	even	better	than	human	pathologists.
28
	If	machine	learning
can	help	reveal	relationships	between	genes,	diseases	and	treatment	responses,	it
could	revolutionize	personalized	medicine,	make	farm	animals	healthier	and
enable	more	resilient	crops.	Moreover,	robots	have	the	potential	to	become	more

27
28
accurate	and	reliable	surgeons	than	humans,	even	without	using	advanced	AI.	A
wide	variety	of	robotic	surgeries	have	been	successfully	performed	in	recent
years,	often	allowing	precision,	miniaturization	and	smaller	incisions	that	lead	to
decreased	blood	loss,	less	pain	and	shorter	healing	time.
Alas,	there	have	been	painful	lessons	about	the	importance	of	robust	software
also	in	the	healthcare	industry.	For	example,	the	Canadian-built	Therac-25
radiation	therapy	machine	was	designed	to	treat	cancer	patients	in	two	different

modes:	either	with	a	low-power	beam	of	electrons	or	with	a	high-power	beam	of
megavolt	X-rays	that	was	kept	on	target	by	a	special	shield.	Unfortunately,
unverified	buggy	software	occasionally	caused	technicians	to	deliver	the
megavolt	beam	when	they	thought	they	were	administering	the	low-power	beam,
and	without	the	shield,	which	ended	up	claiming	the	lives	of	several	patients.
29
Many	more	patients	died	from	radiation	overdoses	at	the	National	Oncologic
Institute	in	Panama,	where	radiotherapy	equipment	using	radioactive	cobalt-60

29
was	programmed	to	excessive	exposure	times	in	2000	and	2001	because	of	a
confusing	user	interface	that	hadn’t	been	properly	validated.
30
	According	to	a
recent	report,
31
	robotic	surgery	accidents	were	linked	to	144	deaths	and	1,391
injuries	in	the	United	States	between	2000	and	2013,	with	common	problems
including	not	only	hardware	issues	such	as	electrical	arcing	and	burnt	or	broken
pieces	of	instruments	falling	into	the	patient,	but	also	software	problems	such	as
uncontrolled	movements	and	spontaneous	powering-off.

The	good	news	is	that	the	rest	of	almost	two	million	robotic	surgeries	covered
by	the	report	went	smoothly,	and	robots	appear	to	be	making	surgery	more	rather
than	less	safe.	According	to	a	U.S.	government	study,	bad	hospital	care
contributes	to	over	100,000	deaths	per	year	in	the	United	States	alone,
32
	so	the
moral	imperative	for	developing	better	AI	for	medicine	is	arguably	even	stronger
than	that	for	self-driving	cars.

AI	for	Communication
The	communication	industry	is	arguably	the	one	where	computers	have	had	the
greatest	impact	of	all	so	far.	After	the	introduction	of	computerized	telephone
switchboards	in	the	fifties,	the	internet	in	the	sixties,	and	the	World	Wide	Web	in
1989,	billions	of	people	now	go	online	to	communicate,	shop,	read	news,	watch
movies	or	play	games,	accustomed	to	having	the	world’s	information	just	a	click
away—and	often	for	free.	The	emerging	
internet	of	things
	promises	improved
efficiency,	accuracy,	convenience	and	economic	benefit	from	bringing	online
everything	from	lamps,	thermostats	and	freezers	to	biochip	transponders	on	farm
animals.
These	spectacular	successes	in	connecting	the	world	have	brought	computer
scientists	a	fourth	challenge:	they	need	to	improve	not	only	verification,
validation	and	control,	but	also	
security
	against	malicious	software	(“malware”)
and	hacks.	Whereas	the	aforementioned	problems	all	resulted	from	unintentional
mistakes,	security	is	directed	at

animals.
security
mistakes,	security	is	directed	at	
deliberate	malfeasance
.	The	first	malware	to
draw	significant	media	attention	was	the	so-called	Morris	worm,	unleashed	on
November	2,	1988,	which	exploited	bugs	in	the	UNIX	operating	system.	It	was
allegedly	a	misguided	attempt	to	count	how	many	computers	were	online,	and
although	it	infected	and	crashed	about	10%	of	the	60,000	computers	that	made
up	the	internet	back	then,	this	didn’t	stop	its	creator,	Robert	Morris,	from
eventually	getting	a	tenured	professorship	in	computer	science	at	MIT.

Other	malware	exploits	vulnerabilities	not	in	software	but	in	people.	On	May
5,	2000,	as	if	to	celebrate	my	birthday,	people	got	emails	with	the	subject	line
“ILOVEYOU”	from	acquaintances	and	colleagues,	and	those	Microsoft
Windows	users	who	clicked	on	the	attachment	“LOVE-LETTER-FOR-
YOU.txt.vbs”	unwittingly	launched	a	script	that	damaged	their	computer	and	re-
sent	the	email	to	everyone	in	their	address	book.	Created	by	two	young
programmers	in	the	Philippines,	this	worm	infected	about	10%	of	the	internet,

just	as	the	Morris	worm	had	done,	but	because	the	internet	was	a	lot	bigger	by
then,	it	became	one	of	the	greatest	infections	of	all	time,	afflicting	over	50
million	computers	and	causing	over	$5	billion	in	damages.	As	you’re	probably
painfully	aware,	the	internet	remains	infested	with	countless	kinds	of	infectious
malware,	which	security	experts	classify	into	worms,	Trojans,	viruses	and	other
intimidating-sounding	categories,	and	the	damage	they	cause	ranges	from

displaying	harmless	prank	messages	to	deleting	your	files,	stealing	your	personal
information,	spying	on	you	and	hijacking	your	computer	to	send	out	spam.
Whereas	malware	targets	whatever	computer	it	can,	
hackers
	attack	specific
targets	of	interest—recent	high-profile	examples	including	Target,	TJ	Maxx,
Sony	Pictures,	Ashley	Madison,	the	Saudi	oil	company	Aramco	and	the	U.S.
Democratic	National	Committee.	Moreover,	the	loots	appear	to	be	getting	ever
more	spectacular.	Hackers	stole	130	million	credit	card	numbers	and	other
account	information	from	Heartland	Payment	Systems	in	2008,	and	breached
over	a	billion(!)	Yahoo!	email	accounts	in	2013.
33
	A	2014	hack	of	the	U.S.
Government’s	Office	of	Personnel	Management	breached	personnel	records	and
job	application	information	for	over	21	million	people,	allegedly	including
employees	with	top	security	clearances	and	the	fingerprints	of	undercover
agents.
As	a	result,	I	roll	my	eyes	whenever	I	read	about	some	new	system	being

33
agents.
allegedly	100%	secure	and	unhackable.	Yet	“unhackable”	is	clearly	what	we
need	future	AI	systems	to	be	before	we	put	them	in	charge	of,	say,	critical
infrastructure	or	weapons	systems,	so	the	growing	role	of	AI	in	society	keeps
raising	the	stakes	for	computer	security.	While	some	hacks	exploit	human
gullibility	or	complex	vulnerabilities	in	newly	released	software,	others	enable
unauthorized	login	to	remote	computers	by	taking	advantage	of	simple	bugs	that
lingered	unnoticed	for	an	embarrassingly	long	time.	The	“Heartbleed”	bug	lasted

from	2012	to	2014	in	one	of	the	most	popular	software	libraries	for	secure
communication	between	computers,	and	the	“Bashdoor”	bug	was	built	into	the
very	operating	system	of	Unix	computers	from	1989	until	2014.	This	means	that
AI	tools	for	improved	verification	and	validation	will	improve	security	as	well.
Unfortunately,	better	AI	systems	can	also	be	used	to	find	new	vulnerabilities
and	perform	more	sophisticated	hacks.	Imagine,	for	example,	that	you	one	day
get	an	unusually	personalized	“phishing”	email	attempting	to	persuade	you	to

divulge	personal	information.	It’s	sent	from	your	friend’s	account	by	an	AI
who’s	hacked	it	and	is	impersonating	her,	imitating	her	writing	style	based	on	an
analysis	of	her	other	sent	emails,	and	including	lots	of	personal	information
about	you	from	other	sources.	Might	you	fall	for	this?	What	if	the	phishing	email
appears	to	come	from	your	credit	card	company	and	is	followed	up	by	a	phone
call	from	a	friendly	human	voice	that	you	can’t	tell	is	AI-generated?	In	the
ongoing	computer-security	arms	race	between	offense	and	defense,	there’s	so	far

little	indication	that	defense	is	winning.

Laws
We	humans	are	social	animals	who	subdued	all	other	species	and	conquered
Earth	thanks	to	our	ability	to	cooperate.	We’ve	developed	laws	to	incentivize	and
facilitate	cooperation,	so	if	AI	can	improve	our	legal	and	governance	systems,
then	it	can	enable	us	to	cooperate	more	successfully	than	ever	before,	bringing
out	the	very	best	in	us.	And	there’s	plenty	of	opportunity	for	improvement	here,
both	in	how	our	laws	are	applied	and	how	they’re	written,	so	let’s	explore	both
in	turn.
What	are	the	first	associations	that	come	to	your	mind	when	you	think	about
the	court	system	in	your	country?	If	it’s	lengthy	delays,	high	costs	and
occasional	injustice,	then	you’re	not	alone.	Wouldn’t	it	be	wonderful	if	your	first
thoughts	were	instead	“efficiency”	and	“fairness”?	Since	the	legal	process	can	be
abstractly	viewed	as	a	computation,	inputting	information	about	evidence	and
laws	and	outputting	a	decision,	some	scholars	dream	of	fully	automating	it	with
robojudges:

in	turn.
	AI	systems	that	tirelessly	apply	the	same	high	legal	standards	to
every	judgment	without	succumbing	to	human	errors	such	as	bias,	fatigue	or
lack	of	the	latest	knowledge.

Robojudges
Byron	De	La	Beckwith	Jr.	was	convicted	in	1994	of	assassinating	civil	rights
leader	Medgar	Evers	in	1963,	but	two	separate	all-white	Mississippi	juries	had
failed	to	convict	him	the	year	after	the	murder,	even	though	the	physical
evidence	was	essentially	the	same.
34
	Alas,	legal	history	is	rife	with	judgments
biased	by	skin	color,	gender,	sexual	orientation,	religion,	nationality	and	other
factors.	Robojudges	could	in	principle	ensure	that,	for	the	first	time	in	history,
everyone	becomes	truly	equal	under	the	law:	they	could	be	programmed	to	all	be
identical	and	to	treat	everyone	equally,	transparently	applying	the	law	in	a	truly
unbiased	fashion.
Robojudges	could	also	eliminate	human	biases	that	are	accidental	rather	than
intentional.	For	example,	a	controversial	2012	study	of	Israeli	judges	claimed
that	they	delivered	significantly	harsher	verdicts	when	they	were	hungry:
whereas	they	denied	about	35%	of	parole	cases	right	after	breakfast,	they	denied
over	85%	right	before	lunch.
35

35
	Another	shortcoming	of	human	judges	is	that
they	may	lack	sufficient	time	to	explore	all	details	of	a	case.	In	contrast,
robojudges	can	easily	be	copied,	since	they	consist	of	little	more	than	software,
allowing	all	pending	cases	to	be	processed	in	parallel	rather	than	in	series,	each
case	getting	its	own	robojudge	for	as	long	as	it	takes.	Finally,	although	it’s
impossible	for	human	judges	to	master	all	technical	knowledge	required	for
every	possible	case,	from	thorny	patent	disputes	to	murder	mysteries	hinging	on

the	latest	forensic	science,	future	robojudges	may	have	essentially	unlimited
memory	and	learning	capacity.
One	day,	such	robojudges	may	therefore	be	both	more	efficient	and	fairer,	by
virtue	of	being	unbiased,	competent	and	transparent.	Their	efficiency	makes
them	fairer	still:	by	speeding	up	the	legal	process	and	making	it	harder	for	savvy
lawyers	to	skew	the	outcome,	they	could	make	it	dramatically	cheaper	to	get
justice	through	the	courts.	This	could	greatly	increase	the	chances	of	a	cash-

strapped	individual	or	startup	company	prevailing	against	a	billionaire	or
multinational	corporation	with	an	army	of	lawyers.
On	the	other	hand,	what	if	robojudges	have	bugs	or	get	hacked?	Both	have
already	afflicted	automatic	voting	machines,	and	when	years	behind	bars	or
millions	in	the	bank	are	at	stake,	the	incentives	for	cyberattacks	are	greater	still.

Even	if	AI	can	be	made	robust	enough	for	us	to	trust	that	a	robojudge	is	using
the	legislated	algorithm,	will	everybody	feel	that	they	understand	its	logical
reasoning	enough	to	respect	its	judgment?	This	challenge	is	exacerbated	by	the
recent	success	of	neural	networks,	which	often	outperform	traditional	easy-to-
understand	AI	algorithms	at	the	price	of	inscrutability.	If	defendants	wish	to
know	
why
	they	were	convicted,	shouldn’t	they	have	the	right	to	a	better	answer
than	“we	trained	the	system	on	lots	of	data,	and	this	is	what	it	decided”?
Moreover,	recent	studies	have	shown	that	if	you	train	a	deep	neural	learning
system	with	massive	amounts	of	prisoner	data,	it	can	predict	who’s	likely	to
return	to	crime	(and	should	therefore	be	denied	parole)	better	than	human
judges.	But	what	if	this	system	finds	that	recidivism	is	statistically	linked	to	a
prisoner’s	sex	or	race—would	this	count	as	a	sexist,	racist	robojudge	that	needs
reprogramming?	Indeed,	a	2016	study	argued	that	recidivism-prediction	software

used	across	the	United	States	was	biased	against	African	Americans	and	had
contributed	to	unfair	sentencing.
36
	These	are	important	questions	that	we	all
need	to	ponder	and	discuss	to	ensure	that	AI	remains	beneficial.	We	aren’t	facing
an	all-or-nothing	decision	regarding	robojudges,	but	rather	a	decision	about	the
extent	and	speed	with	which	we	want	to	deploy	AI	in	our	legal	system.	Do	we
want	human	judges	to	have	AI-based	decision	support	systems,	just	like
tomorrow’s	medical	doctors?	Do	we	want	to	go	further	and	have	robojudge

36
decisions	that	can	be	appealed	to	human	judges,	or	do	we	want	to	go	all	the	way
and	give	even	the	final	say	to	machines,	even	for	death	penalties?

Legal	Controversies
So	far,	we’ve	explored	only	the	
application
	of	law;	let	us	now	turn	to	its	
content
.
There’s	broad	consensus	that	our	laws	need	to	evolve	to	keep	pace	with	our
technology.	For	example,	the	two	programmers	who	created	the	aforementioned
ILOVEYOU	worm	and	caused	billions	of	dollars	in	damages	were	acquitted	of
all	charges	and	walked	free	because	at	that	time,	there	were	no	laws	against
malware	creation	in	the	Philippines.	Since	the	pace	of	technological	progress
appears	to	be	accelerating,	laws	need	to	be	updated	ever	more	rapidly,	and	have
a	tendency	to	lag	behind.	Getting	more	tech-savvy	people	into	law	schools	and
governments	is	probably	a	smart	move	for	society.	But	should	AI-based	decision
support	systems	for	voters	and	legislators	ensue,	followed	by	outright	robo-
legislators?
How	to	best	alter	our	laws	to	reflect	AI	progress	is	a	fascinatingly
controversial	topic.	One	dispute	reflects	the	tension	between	privacy	versus

freedom	of	information.	Freedom	fans	argue	that	the	less	privacy	we	have,	the
more	evidence	the	courts	will	have,	and	the	fairer	the	judgments	will	be.	For
example,	if	the	government	taps	into	everyone’s	electronic	devices	to	record
where	they	are	and	what	they	type,	click,	say	and	do,	many	crimes	would	be
readily	solved,	and	additional	ones	could	be	prevented.	Privacy	advocates
counter	that	they	don’t	want	an	Orwellian	surveillance	state,	and	that	even	if
they	did,	there’s	a	risk	of	it	turning	into	a	totalitarian	dictatorship	of	epic

proportions.	Moreover,	machine-learning	techniques	have	gotten	better	at
analyzing	brain	data	from	fMRI	scanners	to	determine	what	a	person	is	thinking
about	and,	in	particular,	whether	they’re	telling	the	truth	or	lying.
37
	If	AI-assisted
brain	scanning	technology	became	commonplace	in	courtrooms,	the	currently
tedious	process	of	establishing	the	facts	of	a	case	could	be	dramatically
simplified	and	expedited,	enabling	faster	trials	and	fairer	judgments.	But	privacy
advocates	might	worry	about	whether	such	systems	occasionally	make	mistakes

37
and,	more	fundamentally,	whether	our	minds	should	be	off-limits	to	government
snooping.	Governments	that	don’t	support	freedom	of	thought	could	use	such
technology	to	criminalize	the	holding	of	certain	beliefs	and	opinions.	Where
would	
you
	draw	the	line	between	justice	and	privacy,	and	between	protecting
society	and	protecting	personal	freedom?	Wherever	you	draw	it,	will	it	gradually
but	inexorably	move	toward	reduced	privacy	to	compensate	for	the	fact	that

evidence	gets	easier	to	fake?	For	example,	once	AI	becomes	able	to	generate
fully	realistic	fake	videos	of	you	committing	crimes,	will	you	vote	for	a	system
where	the	government	tracks	everyone’s	whereabouts	at	all	times	and	can
provide	you	with	an	ironclad	alibi	if	needed?
Another	captivating	controversy	is	whether	AI	research	should	be	regulated
or,	more	generally,	what	incentives	policymakers	should	give	AI	researchers	to
maximize	the	chances	of	a	beneficial	outcome.	Some	AI	researchers	have	argued
against	all	forms	of	regulation	of	AI	development,	claiming	that	they	would
needlessly	delay	urgently	needed	innovation	(for	example,	lifesaving	self-driving
cars)	and	would	drive	cutting-edge	AI	research	underground	and/or	to	other
countries	with	more	permissive	governments.	At	the	Puerto	Rico	beneficial-AI
conference	mentioned	in	the	first	chapter,	Elon	Musk	argued	that	what	we	need
right	now	from	governments	isn’t	oversight	but	insight:	specifically,	technically

capable	people	in	government	positions	who	can	monitor	AI’s	progress	and	steer
it	if	warranted	down	the	road.	He	also	argued	that	government	regulation	can
sometimes	nurture	rather	than	stifle	progress:	for	example,	if	government	safety
standards	for	self-driving	cars	can	help	reduce	the	number	of	self-driving-car
accidents,	then	a	public	backlash	is	less	likely	and	adoption	of	the	new
technology	can	be	accelerated.	The	most	safety-conscious	AI	companies	might
therefore	favor	regulation	that	forces	less	scrupulous	competitors	to	match	their

high	safety	standards.
Yet	another	interesting	legal	controversy	involves	granting	rights	to	machines.
If	self-driving	cars	cut	the	32,000	annual	U.S.	traffic	fatalities	in	half,	perhaps
carmakers	won’t	get	16,000	thank-you	notes,	but	16,000	lawsuits.	So	if	a	self-
driving	car	causes	an	accident,	who	should	be	liable—its	occupants,	its	owner	or
its	manufacturer?	Legal	scholar	David	Vladeck	has	proposed	a	fourth	answer:
the	car	itself!	Specifically,	he	proposes	that	self-driving	cars	be	allowed	(and

required)	to	hold	car	insurance.	This	way,	models	with	a	sterling	safety	record
will	qualify	for	premiums	that	are	very	low,	probably	lower	than	what’s	available
to	human	drivers,	while	poorly	designed	models	from	sloppy	manufacturers	will
only	qualify	for	insurance	policies	that	make	them	prohibitively	expensive	to
own.
But	if	machines	such	as	cars	are	allowed	to	hold	insurance	policies,	should
they	also	be	able	to	own	money	and	property?	If	so,	there’s	nothing	legally

own.
stopping	smart	computers	from	making	money	on	the	stock	market	and	using	it
to	buy	online	services.	Once	a	computer	starts	paying	humans	to	work	for	it,	it
can	accomplish	anything	that	humans	can	do.	If	AI	systems	eventually	get	better

than	humans	at	investing	(which	they	already	are	in	some	domains),	this	could
lead	to	a	situation	where	most	of	our	economy	is	owned	and	controlled	by
machines.	Is	this	what	we	want?	If	it	sounds	far-off,	consider	that	most	of	our
economy	is	already	owned	by	another	form	of	non-human	entity:	corporations,
which	are	often	more	powerful	than	any	one	person	in	them	and	can	to	some
extent	take	on	life	of	their	own.
If	you’re	OK	with	granting	machines	the	rights	to	own	property,	then	how
about	granting	them	the	right	to	vote?	If	so,	should	each	computer	program	get
one	vote,	even	though	it	can	trivially	make	trillions	of	copies	of	itself	in	the
cloud	if	it’s	rich	enough,	thereby	guaranteeing	that	it	will	decide	all	elections?	If
not,	then	on	what	moral	basis	are	we	discriminating	against	machine	minds
relative	to	human	minds?	Does	it	make	a	difference	if	machine	minds	are
conscious	in	the	sense	of	having	a	subjective	experience	like	we	do?	We’ll

explore	in	greater	depth	these	controversial	questions	related	to	computer	control
of	our	world	in	the	next	chapter,	and	questions	related	to	machine	consciousness
in	chapter	8.

Weapons
Since	time	immemorial,	humanity	has	suffered	from	famine,	disease	and	war.
We’ve	already	mentioned	how	AI	may	help	reduce	famine	and	disease,	so	how
about	war?	Some	argue	that	nuclear	weapons	deter	war	between	the	countries
that	own	them	because	they’re	so	horrifying,	so	how	about	letting	all	nations
build	even	more	horrifying	AI-based	weapons	in	the	hope	of	ending	all	war
forever?	If	you’re	unpersuaded	by	that	argument	and	believe	that	future	wars	are
inevitable,	how	about	using	AI	to	make	these	wars	more	humane?	If	wars	consist
merely	of	machines	fighting	machines,	then	no	human	soldiers	or	civilians	need
get	killed.	Moreover,	future	AI-powered	drones	and	other	autonomous	weapon
systems	(AWS;	also	known	by	their	opponents	as	“killer	robots”)	can	hopefully
be	made	more	fair	and	rational	than	human	soldiers:	equipped	with	superhuman
sensors	and	unafraid	of	getting	killed,	they	might	remain	cool,	calculating	and
level-headed	even	in	the	heat	of	battle,	and	be	less	likely	to	accidentally	kill
civilians.

Figure	3.4:	Whereas	today’s	military	drones	(such	as	this	U.S.	Air	Force	MQ-1	Predator)	are
remote-controlled	by	humans,	future	AI-powered	drones	have	the	potential	to	take	humans	out	of
the	loop,	using	an	algorithm	to	decide	whom	to	target	and	kill.

A	Human	in	the	Loop
But	what	if	the	automated	systems	are	buggy,	confusing	or	don’t	behave	as
expected?	The	U.S.	Phalanx	system	for	Aegis-class	cruisers	automatically
detects,	tracks	and	attacks	threats	such	as	anti-ship	missiles	and	aircraft.	The
USS	
Vincennes
	was	a	guided	missile	cruiser	nicknamed	Robocruiser	in	reference
to	its	Aegis	system,	and	on	July	3,	1988,	in	the	midst	of	a	skirmish	with	Iranian
gunboats	during	the	Iran-Iraq	war,	its	radar	system	warned	of	an	incoming
aircraft.	Captain	William	Rodgers	III	inferred	that	they	were	being	attacked	by	a
diving	Iranian	F-14	fighter	jet	and	gave	the	Aegis	system	approval	to	fire.	What
he	didn’t	realize	at	the	time	was	that	they	shot	down	Iran	Air	Flight	655,	a
civilian	Iranian	passenger	jet,	killing	all	290	people	on	board	and	causing
international	outrage.	Subsequent	investigation	implicated	a	confusing	user
interface	that	didn’t	automatically	show	which	dots	on	the	radar	screen	were

civilian	planes	(Flight	655	followed	its	regular	daily	flight	path	and	had	its
civilian	aircraft	transponder	on)	or	which	dots	were	descending	(as	for	an	attack)
vs.	ascending	(as	Flight	655	was	doing	after	takeoff	from	Tehran).	Instead,	when
the	automated	system	was	queried	for	information	about	the	mysterious	aircraft,
it	reported	“descending”	because	that	was	the	status	of	a	different	aircraft	to
which	it	had	confusingly	reassigned	a	number	used	by	the	navy	to	track	planes:
what	was	descending	was	instead	a	U.S.	surface	combat	air	patrol	plane

operating	far	away	in	the	Gulf	of	Oman.
In	this	example,	there	was	a	human	in	the	loop	making	the	final	decision,	who
under	time	pressure	placed	too	much	trust	in	what	the	automated	system	told
him.	So	far,	according	to	defense	officials	around	the	world,	all	deployed
weapons	systems	have	a	human	in	the	loop,	with	the	exception	of	low-tech
booby	traps	such	as	land	mines.	But	development	is	now	under	way	of	truly
autonomous	weapons	that	select	and	attack	targets	entirely	on	their	own.	It’s

militarily	tempting	to	take	all	humans	out	of	the	loop	to	gain	speed:	in	a	dogfight
between	a	fully	autonomous	drone	that	can	respond	instantly	and	a	drone
reacting	more	sluggishly	because	it’s	remote-controlled	by	a	human	halfway
around	the	world,	which	one	do	you	think	would	win?
However,	there	have	been	close	calls	where	we	were	extremely	lucky	that
there	was	a	human	in	the	loop.	On	October	27,	1962,	during	the	Cuban	Missile
Crisis,	eleven	U.S.	Navy	destroyers	and	the	aircraft	carrier	USS	
Randolph
	had

cornered	the	Soviet	submarine	B-59	near	Cuba,	in	international	waters	outside
the	U.S.	“quarantine”	area.	What	they	didn’t	know	was	that	the	temperature
onboard	had	risen	past	45°C	(113°F)	because	the	submarine’s	batteries	were
running	out	and	the	air-conditioning	had	stopped.	On	the	verge	of	carbon	dioxide
poisoning,	many	crew	members	had	fainted.	The	crew	had	had	no	contact	with
Moscow	for	days	and	didn’t	know	whether	World	War	III	had	already	begun.
Then	the	Americans	started	dropping	small	depth	charges,	which	they	had,
unbeknownst	to	the	crew,	told	Moscow	were	merely	meant	to	force	the	sub	to
surface	and	leave.	“We	thought—that’s	it—the	end,”	crew	member	V.	P.	Orlov
recalled.	“It	felt	like	you	were	sitting	in	a	metal	barrel,	which	somebody	is
constantly	blasting	with	a	sledgehammer.”	What	the	Americans	also	didn’t	know
was	that	the	B-59	crew	had	a	nuclear	torpedo	that	they	were	authorized	to	launch
without	clearing	it	with	Moscow.	Indeed,	Captain	Savitski	decided	to	launch	the

nuclear	torpedo.	Valentin	Grigorievich,	the	torpedo	officer,	exclaimed:	“We	will
die,	but	we	will	sink	them	all—we	will	not	disgrace	our	navy!”	Fortunately,	the
decision	to	launch	had	to	be	authorized	by	three	officers	on	board,	and	one	of
them,	Vasili	Arkhipov,	said	no.	It’s	sobering	that	very	few	have	heard	of
Arkhipov,	although	his	decision	may	have	averted	World	War	III	and	been	the
single	most	valuable	contribution	to	humanity	in	modern	history.
38
	It’s	also
sobering	to	contemplate	what	might	have	happened	had	B-59	been	an

38
	It’s	also
autonomous	AI-controlled	submarine	with	no	humans	in	the	loop.
Two	decades	later,	on	September	9,	1983,	tensions	were	again	high	between
the	superpowers:	the	Soviet	Union	had	recently	been	called	an	“evil	empire”	by
U.S.	president	Ronald	Reagan,	and	just	the	previous	week,	it	had	shot	down	a
Korean	Airlines	passenger	plane	that	strayed	into	its	airspace,	killing	269	people
—including	a	U.S.	congressman.	Now	an	automated	Soviet	early-warning
system	reported	that	the	United	States	had	launched	five	land-based	nuclear

missiles	at	the	Soviet	Union,	leaving	Officer	Stanislav	Petrov	merely	minutes	to
decide	whether	this	was	a	false	alarm.	The	satellite	was	found	to	be	operating
properly,	so	following	protocol	would	have	led	him	to	report	an	incoming
nuclear	attack.	Instead,	he	trusted	his	gut	instinct,	figuring	that	the	United	States
was	unlikely	to	attack	with	only	five	missiles,	and	reported	to	his	commanders
that	it	was	a	false	alarm	without	knowing	this	to	be	true.	It	later	became	clear
that	a	satellite	had	mistaken	the	Sun’s	reflections	off	cloud	tops	for	flames	from

rocket	engines.
39
	I	wonder	what	would	have	happened	if	Petrov	had	been
replaced	by	an	AI	system	that	properly	followed	proper	protocol.

The	Next	Arms	Race?
As	you’ve	undoubtedly	guessed	by	now,	I	personally	have	serious	concerns
about	autonomous	weapons	systems.	But	I	haven’t	even	begun	to	tell	you	about
my	main	worry:	the	endpoint	of	an	arms	race	in	AI	weapons.	In	July	2015,	I
expressed	this	worry	in	the	following	open	letter	together	with	Stuart	Russell,
with	helpful	feedback	from	my	colleagues	at	the	Future	of	Life	Institute:
40

AUTONOMOUS	WEAPONS:
An	Open	Letter	from	AI	&	Robotics	Researchers
Autonomous	weapons	select	and	engage	targets	without	human
intervention.	They	might	include,	for	example,	armed	quadcopters
that	can	search	for	and	eliminate	people	meeting	certain	pre-defined
criteria,	but	do	not	include	cruise	missiles	or	remotely	piloted
drones	for	which	humans	make	all	targeting	decisions.	Artificial
Intelligence	(AI)	technology	has	reached	a	point	where	the
deployment	of	such	systems	is	practically	if	not	legally	feasible
within	years,	not	decades,	and	the	stakes	are	high:	autonomous
weapons	have	been	described	as	the	third	revolution	in	warfare,
after	gunpowder	and	nuclear	arms.
Many	arguments	have	been	made	for	and	against	autonomous
weapons,	for	example	that	replacing	human	soldiers	by	machines	is
good	by	reducing	casualties	for	the	owner	but	bad	by	thereby
lowering	the	threshold	for	going	to	battle.	The	key	question	for
humanity	today	is	whether	to	start	a	global	AI	arms	race	or	to

prevent	it	from	starting.	If	any	major	military	power	pushes	ahead
with	AI	weapon	development,	a	global	arms	race	is	virtually
inevitable,	and	the	endpoint	of	this	technological	trajectory	is
obvious:	autonomous	weapons	will	become	the	Kalashnikovs	of
tomorrow.	Unlike	nuclear	weapons,	they	require	no	costly	or	hard-
to-obtain	raw	materials,	so	they’ll	become	ubiquitous	and	cheap	for
all	significant	military	powers	to	mass-produce.	It	will	only	be	a
matter	of	time	until	they	appear	on	the	black	market	and	in	the

hands	of	terrorists,	dictators	wishing	to	better	control	their
populace,	warlords	wishing	to	perpetrate	ethnic	cleansing,	etc.
Autonomous	weapons	are	ideal	for	tasks	such	as	assassinations,
destabilizing	nations,	subduing	populations	and	selectively	killing	a
particular	ethnic	group.	We	therefore	believe	that	a	military	AI	arms
race	would	not	be	beneficial	for	humanity.	There	are	many	ways	in
which	AI	can	make	battlefields	safer	for	humans,	especially

civilians,	without	creating	new	tools	for	killing	people.
Just	as	most	chemists	and	biologists	have	no	interest	in	building
chemical	or	biological	weapons,	most	AI	researchers	have	no
interest	in	building	AI	weapons	and	do	not	want	others	to	tarnish
their	field	by	doing	so,	potentially	creating	a	major	public	backlash
against	AI	that	curtails	its	future	societal	benefits.	Indeed,	chemists
and	biologists	have	broadly	supported	international	agreements	that
have	successfully	prohibited	chemical	and	biological	weapons,	just
as	most	physicists	supported	the	treaties	banning	space-based
nuclear	weapons	and	blinding	laser	weapons.
To	make	it	harder	to	dismiss	our	concerns	as	coming	only	from	pacifist	tree-
huggers,	I	wanted	to	get	our	letter	signed	by	as	many	hardcore	AI	researchers
and	roboticists	as	possible.	The	International	Campaign	for	Robotic	Arms
Control	had	previously	amassed	hundreds	of	signatories	who	called	for	a	ban	on
killer	robots,	and	I	suspected	that	we	could	do	even	better.	I	knew	that

professional	organizations	would	be	reluctant	to	share	their	massive	member
email	lists	for	a	purpose	that	could	be	construed	as	political,	so	I	scraped
together	lists	of	researchers’	names	and	institutions	from	online	documents	and
advertised	the	task	of	finding	their	email	addresses	on	MTurk—the	Amazon
Mechanical	Turk	crowdsourcing	platform.	Most	researchers	have	their	email
addresses	listed	on	their	university	websites,	and	twenty-four	hours	and	$54
later,	I	was	the	proud	owner	of	a	mailing	list	of	hundreds	of	AI	researchers

who’d	been	successful	enough	to	be	elected	Fellows	of	the	Association	for	the
Advancement	of	Artificial	Intelligence	(AAAI).	One	of	them	was	the	British-
Australian	AI	professor	Toby	Walsh,	who	kindly	agreed	to	email	everyone	else
on	the	list	and	help	spearhead	our	campaign.	MTurk	workers	around	the	world
tirelessly	produced	additional	mailing	lists	for	Toby,	and	before	long,	over	3,000
AI	researchers	and	robotics	researchers	had	signed	our	open	letter,	including	six
past	AAAI	presidents	and	AI	industry	leaders	from	Google,	Facebook,	Microsoft

and	Tesla.	An	army	of	FLI	volunteers	worked	tirelessly	to	validate	the	signatory
lists,	removing	spoof	entries	such	as	Bill	Clinton	and	Sarah	Connor.	Over	17,000
others	signed	too,	including	Stephen	Hawking,	and	after	Toby	organized	a	press
conference	about	this	at	the	International	Joint	Conference	of	Artificial
Intelligence,	it	became	a	major	news	story	around	the	world.
Because	biologists	and	chemists	once	took	a	stand,	their	fields	are	now	known

mainly	for	creating	beneficial	medicines	and	materials	rather	than	biological	and
chemical	weapons.	The	AI	and	robotics	communities	had	now	spoken	as	well:
the	letter	signatories	also	wanted	their	fields	to	be	known	for	creating	a	better
future,	not	for	creating	new	ways	of	killing	people.	But	will	the	main	future	use
of	AI	be	civilian	or	military?	Although	we’ve	spent	more	pages	in	this	chapter
on	the	former,	we	may	soon	be	spending	more	money	on	the	latter—especially	if
a	military	AI	arms	race	takes	off.	Civilian	AI	investment	commitments	exceeded
a	billion	dollars	in	2016,	but	this	was	dwarfed	by	the	Pentagon’s	fiscal	2017
budget	request	of	$12–15	billion	for	AI-related	projects,	and	China	and	Russia
are	likely	to	take	note	of	what	Deputy	Defense	Secretary	Robert	Work	said	when
this	was	announced:	“I	want	our	competitors	to	wonder	what’s	behind	the	black
curtain.”
41

Should	There	Be	an	International	Treaty?
Although	there’s	now	a	major	international	push	toward	negotiating	some	form
of	killer	robot	ban,	it’s	still	unclear	what	will	happen,	and	there’s	a	vibrant
ongoing	debate	about	what,	if	anything,	
should
	happen.	Although	many	leading
stakeholders	agree	that	world	powers	should	draft	some	form	of	international
regulations	to	guide	AWS	research	and	use,	there’s	less	agreement	about	what
precisely	should	be	banned	and	how	a	ban	would	be	enforced.	For	example,
should	only	lethal	autonomous	weapons	be	banned,	or	also	ones	that	seriously
injure	people,	say	by	blinding	them?	Would	we	ban	development,	production	or
ownership?	Should	a	ban	apply	to	all	autonomous	weapons	systems	or,	as	our
letter	said,	only	offensive	ones,	allowing	defensive	systems	such	as	autonomous
anti-aircraft	guns	and	missile	defenses?	In	the	latter	case,	should	AWS	count	as
defensive	even	if	they’re	easy	to	move	into	enemy	territory?	And	how	would

you	enforce	a	treaty	given	that	most	components	of	an	autonomous	weapon	have
a	dual	civilian	use	as	well?	For	example,	there	isn’t	much	difference	between	a
drone	that	can	deliver	Amazon	packages	and	one	that	can	deliver	bombs.
Some	debaters	have	argued	that	designing	an	effective	AWS	treaty	is
hopelessly	hard	and	that	we	therefore	shouldn’t	even	try.	On	the	other	hand,	John
F.	Kennedy	emphasized	when	announcing	the	Moon	missions	that	hard	things
are	worth	attempting	when	success	will	greatly	benefit	the	future	of	humanity.

Moreover,	many	experts	argue	that	the	bans	on	biological	and	chemical	weapons
were	valuable	even	though	enforcement	proved	hard,	with	significant	cheating,
because	the	bans	caused	severe	stigmatization	that	limited	their	use.
I	met	Henry	Kissinger	at	a	dinner	event	in	2016,	and	got	the	opportunity	to
ask	him	about	his	role	in	the	biological	weapons	ban.	He	explained	how	back
when	he	was	the	U.S.	national	security	adviser,	he’d	persuaded	President	Nixon
that	a	ban	would	be	good	for	U.S.	national	security.	I	was	impressed	by	how

sharp	his	mind	and	memory	were	for	a	ninety-two-year-old,	and	was	fascinated
to	hear	his	inside	perspective.	Since	the	United	States	already	enjoyed
superpower	status	thanks	to	its	conventional	and	nuclear	forces,	it	had	more	to
lose	than	to	gain	from	a	worldwide	bioweapons	arms	race	with	uncertain
outcome.	In	other	words,	if	you’re	already	top	dog,	then	it	makes	sense	to	follow
the	maxim	“If	it	ain’t	broke,	don’t	fix	it.”	Stuart	Russell	joined	our	after-dinner
conversation,	and	we	discussed	how	exactly	the	same	argument	can	be	made

about	lethal	autonomous	weapons:	those	who	stand	to	gain	most	from	an	arms
race	aren’t	superpowers	but	small	rogue	states	and	non-state	actors	such	as
terrorists,	who	gain	access	to	the	weapons	via	the	black	market	once	they’ve
been	developed.
Once	mass-produced,	small	AI-powered	killer	drones	are	likely	to	cost	little
more	than	a	smartphone.	Whether	it’s	a	terrorist	wanting	to	assassinate	a
politician	or	a	jilted	lover	seeking	revenge	on	his	ex-girlfriend,	all	they	need	to
do	is	upload	their	target’s	photo	and	address	into	the	killer	drone:	it	can	then	fly
to	the	destination,	identify	and	eliminate	the	person,	and	self-destruct	to	ensure
that	nobody	knows	who	was	responsible.	Alternatively,	for	those	bent	on	ethnic
cleansing,	it	can	easily	be	programmed	to	kill	only	people	with	a	certain	skin
color	or	ethnicity.	Stuart	envisions	that	the	smarter	such	weapons	get,	the	less
material,	firepower	and	money	will	be	needed	per	kill.	For	example,	he	fears

bumblebee-sized	drones	that	kill	cheaply	using	minimal	explosive	power	by
shooting	people	in	the	eye,	which	is	soft	enough	to	allow	even	a	small	projectile
to	continue	into	the	brain.	Or	they	might	latch	on	to	the	head	with	metal	claws
and	then	penetrate	the	skull	with	a	tiny	shaped	charge.	If	a	million	such	killer
drones	can	be	dispatched	from	the	back	of	a	single	truck,	then	one	has	a
horrifying	weapon	of	mass	destruction	of	a	whole	new	kind:	one	that	can
selectively	kill	only	a	prescribed	category	of	people,	leaving	everybody	and

everything	else	unscathed.
A	common	counterargument	is	that	we	can	eliminate	such	concerns	by	making
killer	robots	ethical—for	example,	so	that	they’ll	only	kill	enemy	soldiers.	But	if
we	worry	about	enforcing	a	ban,	then	how	would	it	be	easier	to	enforce	a
requirement	that	enemy	autonomous	weapons	be	100%	ethical	than	to	enforce
that	they	aren’t	produced	in	the	first	place?	And	can	one	consistently	claim	that
the	well-trained	soldiers	of	civilized	nations	are	so	bad	at	following	the	rules	of

war	that	robots	can	do	better,	while	at	the	same	time	claiming	that	rogue	nations,
dictators	and	terrorist	groups	are	so	good	at	following	the	rules	of	war	that
they’ll	never	choose	to	deploy	robots	in	ways	that	violate	these	rules?

Cyberwar
Another	interesting	military	aspect	of	AI	is	that	it	may	let	you	attack	your	enemy
even	without	building	any	weapons	of	your	own,	through	cyberwarfare.	As	a
small	prelude	to	what	the	future	may	bring,	the	Stuxnet	worm,	widely	attributed
to	the	U.S.	and	Israeli	governments,	infected	fast-spinning	centrifuges	in	Iran’s
nuclear-enrichment	program	and	caused	them	to	tear	themselves	apart.	The	more
automated	society	gets	and	the	more	powerful	the	attacking	AI	becomes,	the
more	devastating	cyberwarfare	can	be.	If	you	can	hack	and	crash	your	enemy’s
self-driving	cars,	auto-piloted	planes,	nuclear	reactors,	industrial	robots,
communication	systems,	financial	systems	and	power	grids,	then	you	can
effectively	crash	his	economy	and	cripple	his	defenses.	If	you	can	hack	some	of
his	weapons	systems	as	well,	even	better.
We	began	this	chapter	by	surveying	how	spectacular	the	near-term
opportunities	are	for	AI	to	benefit	humanity—if	we	manage	to	make	it	robust

and	unhackable.	Although	AI	itself	can	be	used	to	make	AI	systems	more	robust,
thereby	aiding	the	cyberwar	defense,	AI	can	clearly	aid	the	offense	as	well.
Ensuring	that	the	defense	prevails	must	be	one	of	the	most	crucial	short-term
goals	for	AI	development—otherwise	all	the	awesome	technology	we	build	can
be	turned	against	us!

So	far	in	this	chapter,	we’ve	mainly	focused	on	how	AI	will	affect	us	as
consumers,
	by	enabling	transformative	new	products	and	services	at	affordable
prices.	But	how	will	it	affect	us	as	
workers,
	by	transforming	the	job	market?	If
we	can	figure	out	how	to	grow	our	prosperity	through	automation	without
leaving	people	lacking	income	or	purpose,	then	we	have	the	potential	to	create	a
fantastic	future	with	leisure	and	unprecedented	opulence	for	everyone	who	wants
it.	Few	people	have	thought	longer	and	harder	about	this	than	economist	Erik
Brynjolfsson,	one	of	my	MIT	colleagues.	Although	he’s	always	well-groomed
and	impeccably	dressed,	he	has	Icelandic	heritage,	and	I	sometimes	can’t	help
imagine	that	he	only	recently	trimmed	back	a	wild	red	Viking	beard	and	mane	to
blend	in	at	our	business	school.	He	certainly	hasn’t	trimmed	back	his	wild	ideas,
and	he	calls	his	optimistic	job-market	vision	“Digital	Athens.”	The	reason	that

the	Athenian	citizens	of	antiquity	had	lives	of	leisure	where	they	could	enjoy
democracy,	art	and	games	was	mainly	that	they	had	slaves	to	do	much	of	the
work.	But	why	not	replace	the	slaves	with	AI-powered	robots,	creating	a	digital
utopia	that	everyone	can	enjoy?	Erik’s	AI-driven	economy	would	not	only
eliminate	stress	and	drudgery	and	produce	an	abundance	of	everything	we	want
today,	but	it	would	also	supply	a	bounty	of	wonderful	new	products	and	services
that	today’s	consumers	haven’t	yet	realized	that	they	want.

Technology	and	Inequality
We	can	get	from	where	we	are	today	to	Erik’s	Digital	Athens	if	everyone’s
hourly	salary	keeps	growing	year	by	year,	so	that	those	who	want	more	leisure
can	gradually	work	less	while	continuing	to	improve	their	standard	of	living.
Figure	3.5
	shows	that	this	is	precisely	what	happened	in	the	United	States	from
World	War	II	until	the	mid-1970s:	although	there	was	income	inequality,	the
total	size	of	the	pie	grew	in	such	a	way	that	almost	everybody	got	a	larger	slice.
But	then,	as	Erik	is	the	first	to	admit,	something	changed:	
figure	3.5
	shows	that
although	the	economy	kept	growing	and	raising	the	average	income,	the	gains
over	the	past	four	decades	went	to	the	wealthiest,	mostly	to	the	top	1%,	while	the
poorest	90%	saw	their	incomes	stagnate.	The	resulting	growth	in	inequality	is
even	more	evident	if	we	look	not	at	income	but	at	wealth.	For	the	bottom	90%	of
U.S.	households,	the	average	net	worth	was	about	$85,000	in	2012—the	same	as

figure	3.5
	shows	that
twenty-five	years	earlier—while	the	top	1%	more	than	doubled	their	inflation-
adjusted	wealth	during	that	period,	to	$14	million.
42
	Differences	are	even	more
extreme	internationally	where,	in	2013,	the	combined	wealth	of	the	bottom	half
of	the	world’s	population	(over	3.6	billion	people)	is	the	same	as	that	of	the
world’s	eight	richest	people
43
—a	statistic	that	highlights	the	poverty	and
vulnerability	at	the	bottom	as	much	as	the	wealth	at	the	top.	At	our	2015	Puerto

42
43
Rico	conference,	Erik	told	the	assembled	AI	researchers	that	he	thought	progress
in	AI	and	automation	would	continue	making	the	economic	pie	bigger,	but	that
there’s	no	economic	law	that	everyone,	or	even	most	people,	will	benefit.
Although	there’s	broad	agreement	among	economists	that	inequality	is	rising,
there’s	an	interesting	controversy	about	why	and	whether	the	trend	will	continue.
Debaters	on	the	left	side	of	the	political	spectrum	often	argue	that	the	main	cause

is	globalization	and/or	economic	policies	such	as	tax	cuts	for	the	rich.	But	Erik
Brynjolfsson	and	his	MIT	collaborator	Andrew	McAfee	argue	that	the	main
cause	is	something	else:	technology.
44
	Specifically,	they	argue	that	digital
technology	drives	inequality	in	three	different	ways.
First,	by	replacing	old	jobs	with	ones	requiring	more	skills,	technology	has
rewarded	the	educated:	since	the	mid-1970s,	salaries	rose	about	25%	for	those
with	graduate	degrees	while	the	average	high	school	dropout	took	a	30%	pay
cut.
45

Second,	they	claim	that	since	the	year	2000,	an	ever-larger	share	of	corporate
income	has	gone	to	those	who	own	the	companies	as	opposed	to	those	who	work
there—and	that	as	long	as	automation	continues,	we	should	expect	those	who
own	the	machines	to	take	a	growing	fraction	of	the	pie.	This	edge	of	capital	over
labor	may	be	particularly	important	for	the	growing	digital	economy,	which	tech
visionary	Nicholas	Negroponte	defines	as	moving	bits,	not	atoms.	Now	that
everything	from	books	to	movies	and	tax	preparation	tools	has	gone	digital,
additional	copies	can	be	sold	worldwide	at	essentially	zero	cost,	without	hiring
additional	employees.	This	allows	most	of	the	revenue	to	go	to	investors	rather
than	workers,	and	helps	explain	why,	even	though	the	combined	revenues	of
Detroit’s	“Big	3”	(GM,	Ford	and	Chrysler)	in	1990	were	almost	identical	to
those	of	Silicon	Valley’s	“Big	3”	(Google,	Apple,	Facebook)	in	2014,	the	latter
had	nine	times	fewer	employees	and	were	worth	thirty	times	more	on	the	stock
market.
47

Figure	3.5:	How	the	economy	has	grown	average	income	over	the	past	century,	and	what	fraction
of	this	income	has	gone	to	different	groups.	Before	the	1970s,	rich	and	poor	are	seen	to	all	be
getting	better	off	in	lockstep,	after	which	most	of	the	gains	have	gone	to	the	top	1%	while	the
bottom	90%	have	on	average	gained	close	to	nothing.
46
	The	amounts	have	been	inflation-
corrected	to	year-2017	dollars.
Third,	Erik	and	collaborators	argue	that	the	digital	economy	often	benefits
superstars	over	everyone	else.	Harry	Potter	author	J.	K.	Rowling	became	the	first
writer	to	join	the	billionaire	club,	and	she	got	much	richer	than	Shakespeare
because	her	stories	could	be	transmitted	in	the	form	of	text,	movies	and	games	to
billions	of	people	at	very	low	cost.	Similarly,	Scott	Cook	made	a	billion	on	the
TurboTax	tax	preparation	software,	which,	unlike	human	tax	preparers,	can	be
sold	as	a	download.	Since	most	people	are	willing	to	pay	little	or	nothing	for	the

tenth-best	tax-preparation	software,	there’s	room	in	the	marketplace	for	only	a
modest	number	of	superstars.	This	means	that	if	all	the	world’s	parents	advise
their	kids	to	become	the	next	J.	K.	Rowling,	Gisele	Bündchen,	Matt	Damon,
Cristiano	Ronaldo,	Oprah	Winfrey	or	Elon	Musk,	almost	none	of	their	kids	will
find	this	a	viable	career	strategy.

Career	Advice	for	Kids
So	what	career	advice	
should
	we	give	our	kids?	I’m	encouraging	mine	to	go	into
professions	that	machines	are	currently	bad	at,	and	therefore	seem	unlikely	to	get
automated	in	the	near	future.	Recent	forecasts	for	when	various	jobs	will	get
taken	over	by	machines	identify	several	useful	questions	to	ask	about	a	career
before	deciding	to	educate	oneself	for	it.
48
	For	example:
•
Does	it	require	interacting	with	people	and	using	social	intelligence?
•
Does	it	involve	creativity	and	coming	up	with	clever	solutions?
•
Does	it	require	working	in	an	unpredictable	environment?
The	more	of	these	questions	you	can	answer	with	a	yes,	the	better	your	career
choice	is	likely	to	be.	This	means	that	relatively	safe	bets	include	becoming	a
teacher,	nurse,	doctor,	dentist,	scientist,	entrepreneur,	programmer,	engineer,
lawyer,	social	worker,	clergy	member,	artist,	hairdresser	or	massage	therapist.
In	contrast,	jobs	that	involve	highly	repetitive	or	structured	actions	in	a

•
predictable	setting	aren’t	likely	to	last	long	before	getting	automated	away.
Computers	and	industrial	robots	took	over	the	simplest	such	jobs	long	ago,	and
improving	technology	is	in	the	process	of	eliminating	many	more,	from
telemarketers	to	warehouse	workers,	cashiers,	train	operators,	bakers	and	line
cooks.
49
	Drivers	of	trucks,	buses,	taxis	and	Uber/Lyft	cars	are	likely	to	follow
soon.	There	are	many	more	professions	(including	paralegals,	credit	analysts,
loan	officers,	bookkeepers	and	tax	accountants)	that,	although	they	aren’t	on	the

cooks.
49
endangered	list	for	full	extinction,	are	getting	most	of	their	tasks	automated	and
therefore	demand	many	fewer	humans.
But	staying	clear	of	automation	isn’t	the	only	career	challenge.	In	this	global
digital	age,	aiming	to	become	a	professional	writer,	filmmaker,	actor,	athlete	or
fashion	designer	is	risky	for	another	reason:	although	people	in	these	professions
won’t	get	serious	competition	from	machines	anytime	soon,	they’ll	get
increasingly	brutal	competition	from	other	humans	around	the	globe	according

to	the	aforementioned	superstar	theory,	and	very	few	will	succeed.
In	many	cases,	it	would	be	too	myopic	and	crude	to	give	career	advice	at	the
level	of	whole	fields:	there	are	many	jobs	that	won’t	get	entirely	eliminated,	but

which	will	see	many	of	their	tasks	automated.	For	example,	if	you	go	into
medicine,	don’t	be	the	radiologist	who	analyzes	the	medical	images	and	gets
replaced	by	IBM’s	Watson,	but	the	doctor	who	orders	the	radiology	analysis,
discusses	the	results	with	the	patient,	and	decides	on	the	treatment	plan.	If	you
go	into	finance,	don’t	be	the	“quant”	who	applies	algorithms	to	the	data	and	gets
replaced	by	software,	but	the	fund	manager	who	uses	the	quantitative	analysis
results	to	make	strategic	investment	decisions.	If	you	go	into	law,	don’t	be	the
paralegal	who	reviews	thousands	of	documents	for	the	discovery	phase	and	gets
automated	away,	but	the	attorney	who	counsels	the	client	and	presents	the	case
in	court.
So	far,	we’ve	explored	what	individuals	can	do	to	maximize	their	success	on
the	job	market	in	the	age	of	AI.	But	what	can	governments	do	to	help	their
workforces	succeed?	For	example,	what	education	system	best	prepares	people
for	a	job	market	where	AI	keeps	improving	rapidly?	Is	it	still	our	current	model

in	court.
with	one	or	two	decades	of	education	followed	by	four	decades	of	specialized
work?	Or	is	it	better	to	switch	to	a	system	where	people	work	for	a	few	years,
then	go	back	to	school	for	a	year,	then	work	for	a	few	more	years?
50
	Or	should
continuing	education	(perhaps	provided	online)	be	a	standard	part	of	any	job?
And	what	economic	policies	are	most	helpful	for	creating	good	new	jobs?
Andrew	McAfee	argues	that	there	are	many	policies	that	are	likely	to	help,
including	investing	heavily	in	research,	education	and	infrastructure,	facilitating

50
	Or	should
migration	and	incentivizing	entrepreneurship.	He	feels	that	“the	Econ	101
playbook	is	clear,	but	is	not	being	followed,”	at	least	not	in	the	United	States.
51

Will	Humans	Eventually	Become	Unemployable?
If	AI	keeps	improving,	automating	ever	more	jobs,	what	will	happen?	Many
people	are	job	optimists,	arguing	that	the	automated	jobs	will	be	replaced	by	new
ones	that	are	even	better.	After	all,	that’s	what’s	always	happened	before,	ever
since	Luddites	worried	about	technological	unemployment	during	the	Industrial
Revolution.
Others,	however,	are	job	pessimists	and	argue	that	this	time	is	different,	and
that	an	ever-larger	number	of	people	will	become	not	only	unemployed,	but
unemployable.
52
	The	job	pessimists	argue	that	the	free	market	sets	salaries	based
on	supply	and	demand,	and	that	a	growing	supply	of	cheap	machine	labor	will
eventually	depress	human	salaries	far	below	the	cost	of	living.	Since	the	market
salary	for	a	job	is	the	hourly	cost	of	whoever	or	whatever	will	perform	it	most
cheaply,	salaries	have	historically	dropped	whenever	it	became	possible	to
outsource	a	particular	occupation	to	a	lower-income	country	or	to	a	cheap

52
machine.	During	the	Industrial	Revolution,	we	started	figuring	out	how	to
replace	our	muscles	with	machines,	and	people	shifted	into	better-paying	jobs
where	they	used	their	minds	more.	Blue-collar	jobs	were	replaced	by	white-
collar	jobs.	Now	we’re	gradually	figuring	out	how	to	replace	our	minds	by
machines.	If	we	ultimately	succeed	in	this,	then	what	jobs	are	left	for	us?
Some	job	optimists	argue	that	after	physical	and	mental	jobs,	the	next	boom
will	be	in	
creative
	jobs,	but	job	pessimists	counter	that	creativity	is	just	another

will	be	in	
creative
mental	process,	so	that	it	too	will	eventually	be	mastered	by	AI.	Other	job
optimists	hope	that	the	next	boom	will	instead	be	in	new	technology-enabled
professions	that	we	haven’t	even	thought	of	yet.	After	all,	who	during	the
Industrial	Revolution	would	have	imagined	that	their	descendants	would	one	day
work	as	web	designers	and	Uber	drivers?	But	job	pessimists	counter	that	this	is
wishful	thinking	with	little	support	from	empirical	data.	They	point	out	that	we
could	have	made	the	same	argument	a	century	ago,	before	the	computer

revolution,	and	predicted	that	most	of	today’s	professions	would	be	new	and
previously	unimagined	technology-enabled	ones	that	didn’t	use	to	exist.	This
prediction	would	have	been	an	epic	failure,	as	illustrated	in	
figure	3.6
:	the	vast
majority	of	today’s	occupations	are	ones	that	already	existed	a	century	ago,	and
when	we	sort	them	by	the	number	of	jobs	they	provide,	we	have	to	go	all	the
way	down	to	twenty-first	place	in	the	list	until	we	encounter	a	new	occupation:

software	developers,	who	make	up	less	than	1%	of	the	U.S.	job	market.
We	can	get	a	better	understanding	of	what’s	happening	by	recalling	chapter	2,
which	showed	the	landscape	of	human	intelligence,	with	elevation	representing
how	hard	it	is	for	machines	to	perform	various	tasks	and	the	rising	sea	level
showing	what	machines	can	currently	do.	The	main	trend	on	the	job	market	isn’t
that	we’re	moving	into	entirely	new	professions.	Rather,	we’re	crowding	into
those	pieces	of	terrain	in	
figure	2.2
	that	haven’t	yet	been	submerged	by	the	rising
tide	of	technology!	
Figure	3.6
	shows	that	this	forms	not	a	single	island	but	a
complex	archipelago,	with	islets	and	atolls	corresponding	to	all	the	valuable
things	that	machines	still	can’t	do	as	cheaply	as	humans	can.	This	includes	not
only	high-tech	professions	such	as	software	development,	but	also	a	panoply	of
low-tech	jobs	leveraging	our	superior	dexterity	and	social	skills,	ranging	from
massage	therapy	to	acting.	Might	AI	eclipse	us	at	intellectual	tasks	so	rapidly

Figure	3.6
that	the	last	remaining	jobs	will	be	in	that	low-tech	category?	A	friend	of	mine
recently	joked	with	me	that	perhaps	the	very	last	profession	will	be	the	very	first
profession:	prostitution.	But	then	he	mentioned	this	to	a	Japanese	roboticist,	who
protested:	“No,	robots	are	very	good	at	those	things!”

Figure	3.6:	The	pie	chart	shows	the	occupations	of	the	149	million	Americans	who	had	a	job	in
2015,	with	the	535	job	categories	from	the	U.S.	Bureau	of	Labor	Statistics	sorted	by
popularity.
53
	All	occupations	with	more	than	a	million	workers	are	labeled.	There	are	no	new
occupations	created	by	computer	technology	until	twenty-first	place.	This	figure	is	inspired	by	an
analysis	from	Federico	Pistono.
54
Job	pessimists	contend	that	the	endpoint	is	obvious:	the	whole	archipelago
will	get	submerged,	and	there	will	be	no	jobs	left	that	humans	can	do	more
cheaply	than	machines.	In	his	2007	book	
Farewell	to	Alms,
	the	Scottish-
American	economist	Gregory	Clark	points	out	that	we	can	learn	a	thing	or	two
about	our	future	job	prospects	by	comparing	notes	with	our	equine	friends.
Imagine	two	horses	looking	at	an	early	automobile	in	the	year	1900	and
pondering	their	future.
“I’m	worried	about	technological	unemployment.”
“Neigh,	neigh,	don’t	be	a	Luddite:	our	ancestors	said	the	same

thing	when	steam	engines	took	our	industry	jobs	and	trains	took	our
jobs	pulling	stage	coaches.	But	we	have	more	jobs	than	ever	today,

and	they’re	better	too:	I’d	much	rather	pull	a	light	carriage	through
town	than	spend	all	day	walking	in	circles	to	power	a	stupid	mine-
shaft	pump.”
“But	what	if	this	internal	combustion	engine	thing	really	takes
off?”
“I’m	sure	there’ll	be	new	new	jobs	for	horses	that	we	haven’t	yet
imagined.	That’s	what’s	always	happened	before,	like	with	the
invention	of	the	wheel	and	the	plow.”
Alas,	those	not-yet-imagined	new	jobs	for	horses	never	arrived.	No-longer-
needed	horses	were	slaughtered	and	not	replaced,	causing	the	U.S.	equine
population	to	collapse	from	about	26	million	in	1915	to	about	3	million	in
1960.
55
	As	mechanical	muscles	made	horses	redundant,	will	mechanical	minds
do	the	same	to	humans?

Giving	People	Income	Without	Jobs
So	who’s	right:	those	who	say	automated	jobs	will	be	replaced	by	better	ones	or
those	who	say	most	humans	will	end	up	unemployable?	If	AI	progress	continues
unabated,	then	
both
	sides	might	be	right:	one	in	the	short	term	and	the	other	in
the	long	term.	But	although	people	often	discuss	the	disappearance	of	jobs	with
doom-and-gloom	connotations,	it	doesn’t	have	to	be	a	bad	thing!	Luddites
obsessed	about	particular	jobs,	neglecting	the	possibility	that	other	jobs	might
provide	the	same	social	value.	Analogously,	perhaps	those	who	obsess	about
jobs	today	are	being	too	narrow-minded:	we	want	jobs	because	they	can	provide
us	with	income	and	purpose,	but	given	the	opulence	of	resources	produced	by
machines,	it	should	be	possible	to	find	alternative	ways	of	providing	both	the
income	and	the	purpose	
without
	jobs.	Something	similar	ended	up	happening	in
the	equine	story,	which	didn’t	end	with	all	horses	going	extinct.	Instead,	the

without
number	of	horses	has	more	than	tripled	since	1960,	as	they	were	protected	by	an
equine	social-welfare	system	of	sorts:	even	though	they	couldn’t	pay	their	own
bills,	people	decided	to	take	care	of	horses,	keeping	them	around	for	fun,	sport
and	companionship.	Can	we	similarly	take	care	of	our	fellow	humans	in	need?
Let’s	start	with	the	question	of	income:	redistributing	merely	a	small	share	of
the	growing	economic	pie	should	enable	everyone	to	become	better	off.	Many
argue	that	we	not	only	
can
	but	
should
	do	this.	On	the	2016	panel	where	Moshe

can
	but	
should
Vardi	spoke	of	a	moral	imperative	to	save	lives	with	AI-powered	technology,	I
argued	that	it’s	also	a	moral	imperative	to	advocate	for	its	beneficial	use,
including	sharing	the	wealth.	Erik	Brynjolfsson,	also	a	panelist,	said	that	“if	with
all	this	new	wealth	generation,	we	can’t	even	prevent	half	of	all	people	from
getting	worse	off,	then	shame	on	us!”
There	are	many	different	proposals	for	wealth-sharing,	each	with	its
supporters	and	detractors.	The	simplest	is	
basic	income,
	where	every	person

receives	a	monthly	payment	with	no	preconditions	or	requirements	whatsoever.
A	number	of	small-scale	experiments	are	now	being	tried	or	planned,	for
example	in	Canada,	Finland	and	the	Netherlands.	Advocates	argue	that	basic
income	is	more	efficient	than	alternatives	such	as	welfare	payments	to	the	needy,
because	it	eliminates	the	administrative	hassle	of	determining	who	qualifies.
Need-based	welfare	payments	have	also	been	criticized	for	disincentivizing

work,	but	this	of	course	becomes	irrelevant	in	a	jobless	future	where	nobody

works.
Governments	can	help	their	citizens	not	only	by	giving	them	money,	but	also
by	providing	them	with	free	or	subsidized	services	such	as	roads,	bridges,	parks,
public	transportation,	childcare,	education,	healthcare,	retirement	homes	and
internet	access;	indeed,	many	governments	already	provide	most	of	these
services.	As	opposed	to	basic	income,	such	government-funded	services
accomplish	two	separate	goals:	they	reduce	people’s	cost	of	living	and	also
provide	jobs.	Even	in	a	future	where	machines	can	outperform	humans	at	all
jobs,	governments	could	opt	to	pay	people	to	work	in	childcare,	eldercare,	etc.
rather	than	outsource	the	caregiving	to	robots.
Interestingly,	technological	progress	can	end	up	providing	many	valuable
products	and	services	for	free	even	without	government	intervention.	For
example,	people	used	to	pay	for	encyclopedias,	atlases,	sending	letters	and
making	phone	calls,	but	now	anyone	with	an	internet	connection	gets	access	to

all	these	things	at	no	cost—together	with	free	videoconferencing,	photo	sharing,
social	media,	online	courses	and	countless	other	new	services.	Many	other	things
that	can	be	highly	valuable	to	a	person,	say	a	lifesaving	course	of	antibiotics,
have	become	extremely	cheap.	So	thanks	to	technology,	even	many	poor	people
today	have	access	to	things	that	the	world’s	richest	people	lacked	in	the	past.
Some	take	this	to	mean	that	the	income	needed	for	a	decent	life	is	dropping.

If	machines	can	one	day	produce	all	current	goods	and	services	at	minimal
cost,	then	there’s	clearly	enough	wealth	to	make	everyone	better	off.	In	other
words,	even	relatively	modest	taxes	could	then	allow	governments	to	pay	for
basic	income	and	free	services.	But	the	fact	that	wealth-sharing	
can
	happen
obviously	doesn’t	mean	that	it	
will
	happen,	and	today	there’s	strong	political
disagreement	about	whether	it	even	
should
	happen.	As	we	saw	above,	the
current	trend	in	the	United	States	appears	to	be	in	the	opposite	direction,	with

can
	happen
will
should
some	groups	of	people	getting	poorer	decade	after	decade.	Policy	decisions
about	how	to	share	society’s	growing	wealth	will	impact	everybody,	so	the
conversation	about	what	sort	of	future	economy	to	build	should	include
everyone,	not	merely	AI	researchers,	roboticists	and	economists.
Many	debaters	argue	that	reducing	income	inequality	is	a	good	idea	not
merely	in	an	AI-dominated	future,	but	also	today.	Although	the	main	argument
tends	to	be	a	moral	one,	there’s	also	evidence	that	greater	equality	makes

democracy	work	better:	when	there’s	a	large	well-educated	middle	class,	the
electorate	is	harder	to	manipulate,	and	it’s	tougher	for	small	numbers	of	people

or	companies	to	buy	undue	influence	over	the	government.	A	better	democracy
can	in	turn	enable	a	better-managed	economy	that’s	less	corrupt,	more	efficient
and	faster	growing,	ultimately	benefiting	essentially	everyone.

Giving	People	Purpose	Without	Jobs
Jobs	can	provide	people	with	more	than	just	money.	Voltaire	wrote	in	1759	that
“work	keeps	at	bay	three	great	evils:	boredom,	vice	and	need.”	Conversely,
providing	people	with	income	isn’t	enough	to	guarantee	their	well-being.	Roman
emperors	provided	both	bread	and	circuses	to	keep	their	underlings	content,	and
Jesus	emphasized	non-material	needs	in	the	Bible	quote	“Man	shall	not	live	by
bread	alone.”	So	precisely	what	valuable	things	do	jobs	contribute	beyond
money,	and	in	what	alternative	ways	can	a	jobless	society	provide	them?
The	answers	to	these	questions	are	obviously	complicated,	since	some	people
hate	their	jobs	and	others	love	them.	Moreover,	many	children,	students	and
homemakers	thrive	without	jobs,	while	history	teems	with	stories	of	spoiled
heirs	and	princes	who	succumbed	to	ennui	and	depression.	A	2012	meta-analysis
showed	that	unemployment	tends	to	have	negative	long-term	effects	on	well-
being,	while	retirement	was	a	mixed	bag	with	both	positive	and	negative

aspects.
56
	The	growing	field	of	
positive	psychology
	has	identified	a	number	of
factors	that	boost	people’s	sense	of	well-being	and	purpose,	and	found	that	some
(but	not	all!)	jobs	can	provide	many	of	them,	for	example:
57
•
a	social	network	of	friends	and	colleagues
•
a	healthy	and	virtuous	lifestyle
•
respect,	self-esteem,	self-efficacy	and	a	pleasurable	sense	of	“flow”
stemming	from	doing	something	one	is	good	at
•
a	sense	of	being	needed	and	making	a	difference
•
a	sense	of	meaning	from	being	part	of	and	serving	something	larger	than
oneself

has	identified	a	number	of
57
•
•
•
•
•
oneself
This	gives	reason	for	optimism,	since	all	of	these	things	can	be	provided	also
outside	of	the	workplace,	for	example	through	sports,	hobbies	and	learning,	and
with	families,	friends,	teams,	clubs,	community	groups,	schools,	religious	and
humanist	organizations,	political	movements	and	other	institutions.	To	create	a
low-employment	society	that	flourishes	rather	than	degenerates	into	self-
destructive	behavior,	we	therefore	need	to	understand	how	to	help	such	well-

being-inducing	activities	thrive.	The	quest	for	such	an	understanding	needs	to
involve	not	only	scientists	and	economists,	but	also	psychologists,	sociologists
and	educators.	If	serious	efforts	are	put	into	creating	well-being	for	all,	funded
by	part	of	the	wealth	that	future	AI	generates,	then	society	should	be	able	to
flourish	like	never	before.	At	a	minimum,	it	should	be	possible	to	make
everyone	as	happy	as	if	they	had	their	personal	dream	job,	but	once	one	breaks
free	of	the	constraint	that	everyone’s	activities	must	generate	income,	the	sky’s
the	limit.

We’ve	explored	in	this	chapter	how	AI	has	the	potential	to	greatly	improve	our
lives	in	the	near	term,	as	long	as	we	plan	ahead	and	avoid	various	pitfalls.	But
what	about	the	longer	term?	Will	AI	progress	eventually	stagnate	due	to
insurmountable	obstacles,	or	will	AI	researchers	ultimately	succeed	in	their
original	goal	of	building	human-level	artificial	general	intelligence?	We	saw	in
the	previous	chapter	how	the	laws	of	physics	allow	suitable	clumps	of	matter	to
remember,	compute	and	learn,	and	how	they	don’t	prohibit	such	clumps	from
one	day	doing	so	with	greater	intelligence	than	the	matter	clumps	in	our	heads.
If/when	we	humans	will	succeed	in	building	such	superhuman	AGI	is	much	less
clear.	We	saw	in	the	first	chapter	that	we	simply	don’t	know	yet,	since	the
world’s	leading	AI	experts	are	divided,	most	of	them	making	estimates	ranging
from	decades	to	centuries	and	some	even	guessing	never.	Forecasting	is	tough

because,	when	you’re	exploring	uncharted	territory,	you	don’t	know	how	many
mountains	separate	you	from	your	destination.	Typically	you	see	only	the	closest
one,	and	need	to	climb	it	before	you	can	discover	your	next	obstacle.
What’s	the	soonest	it	could	happen?	Even	if	we	knew	the	best	possible	way	to
build	human-level	AGI	using	today’s	computer	hardware,	which	we	don’t,	we’d
still	need	to	have	enough	of	it	to	provide	the	raw	computational	power	needed.
So	what’s	the	computational	power	of	a	human	brain	measured	in	the	bits	and
FLOPS	from	chapter	2?
*4

*4
	This	is	a	delightfully	tricky	question,	and	the	answer
depends	dramatically	on	how	we	ask	it:
•
Question	1:	How	many	FLOPS	are	needed	to	simulate	a	brain?
•
Question	2:	How	many	FLOPS	are	needed	for	human	intelligence?
•
Question	3:	How	many	FLOPS	can	a	human	brain	perform?
There	have	been	lots	of	papers	published	on	question	1,	and	they	typically
give	answers	in	the	ballpark	of	a	hundred	petaFLOPS,	i.e.,	10
17
	FLOPS.
58
	That’s
about	the	same	computational	power	as	the	Sunway	TaihuLight	(
figure	3.7
),	the

•
•
•
17
	FLOPS.
58
	That’s
figure	3.7
),	the
world’s	fastest	supercomputer	in	2016,	which	cost	about	$300	million.	Even	if
we	knew	how	to	use	it	to	simulate	the	brain	of	a	highly	skilled	worker,	we	would

only	profit	from	having	the	simulation	do	this	person’s	job	if	we	could	rent	the
TaihuLight	for	less	than	her	hourly	salary.	We	may	need	to	pay	even	more,
because	many	scientists	believe	that	to	accurately	replicate	the	intelligence	of	a
brain,	we	can’t	treat	it	as	a	mathematically	simplified	neural-network	model
from	chapter	2.	Perhaps	we	instead	need	to	simulate	it	at	the	level	of	individual
molecules	or	even	subatomic	particles,	which	would	require	dramatically	more
FLOPS.
The	answer	to	question	3	is	easier:	I’m	painfully	bad	at	multiplying	19-digit
numbers,	and	it	would	take	me	many	minutes	even	if	you	let	me	borrow	pencil
and	paper.	That	would	clock	me	in	below	0.01	FLOPS—a	whopping	19	orders
of	magnitude	below	the	answer	to	question	1!	The	reason	for	the	huge
discrepancy	is	that	brains	and	supercomputers	are	optimized	for	extremely
different	tasks.	We	get	a	similar	discrepancy	between	these	questions:
How	well	can	a	tractor	do	the	work	of	a	Formula	One	race	car?

FLOPS.
How	well	can	a	Formula	One	car	do	the	work	of	a	tractor?
So	which	of	these	two	questions	about	FLOPS	are	we	trying	to	answer	to
forecast	the	future	of	AI?	Neither!	If	we	wanted	to	simulate	a	human	brain,	we’d
care	about	question	1,	but	to	build	human-level	AGI,	what	matters	is	instead	the
one	in	the	middle:	question	2.	Nobody	knows	its	answer	yet,	but	it	may	well	be
significantly	cheaper	than	simulating	a	brain	if	we	either	adapt	the	software	to	be

better	matched	to	today’s	computers	or	build	more	brain-like	hardware	(rapid
progress	is	being	made	on	so-called	neuromorphic	chips).
Hans	Moravec	estimated	the	answer	by	making	an	apples-to-apples
comparison	for	a	computation	that	both	our	brain	and	today’s	computers	can	do
efficiently:	certain	low-level	image-processing	tasks	that	a	human	retina
performs	in	the	back	of	the	eyeball	before	sending	its	results	to	the	brain	via	the
optic	nerve.
59
	He	figured	that	replicating	a	retina’s	computations	on	a

59
conventional	computer	requires	about	a	billion	FLOPS	and	that	the	whole	brain
does	about	ten	thousand	times	more	computation	than	a	retina	(based	on
comparing	volumes	and	numbers	of	neurons),	so	that	the	computational	capacity
of	the	brain	is	around	10
13
	FLOPS—roughly	the	power	of	an	optimized	$1,000
computer	in	2015!

Figure	3.7:	Sunway	TaihuLight,	the	world’s	fastest	supercomputer	in	2016,	whose	raw
computational	power	arguably	exceeds	that	of	the	human	brain.
In	summary,	there’s	absolutely	no	guarantee	that	we’ll	manage	to	build
human-level	AGI	in	our	lifetime—or	ever.	But	there’s	also	no	watertight
argument	that	we	won’t.	There’s	no	longer	a	strong	argument	that	we	lack
enough	hardware	firepower	or	that	it	will	be	too	expensive.	We	don’t	know	how
far	we	are	from	the	finish	line	in	terms	of	architectures,	algorithms	and	software,
but	current	progress	is	swift	and	the	challenges	are	being	tackled	by	a	rapidly
growing	global	community	of	talented	AI	researchers.	In	other	words,	we	can’t
dismiss	the	possibility	that	AGI	will	eventually	reach	human	levels	and	beyond.
Let’s	therefore	devote	the	next	chapter	to	exploring	this	possibility	and	what	it
might	lead	to!

•
Near-term	AI	progress	has	the	potential	to	greatly	improve	our	lives	in	myriad	ways,
from	making	our	personal	lives,	power	grids	and	financial	markets	more	efficient	to
saving	lives	with	self-driving	cars,	surgical	bots	and	AI	diagnosis	systems.
•
When	we	allow	real-world	systems	to	be	controlled	by	AI,	it’s	crucial	that	we	learn
to	make	AI	more	robust,	doing	what	we	want	it	to	do.	This	boils	down	to	solving
tough	technical	problems	related	to	verification,	validation,	security	and	control.
•
This	need	for	improved	robustness	is	particularly	pressing	for	AI-controlled	weapon
systems,	where	the	stakes	can	be	huge.
•
Many	leading	AI	researchers	and	roboticists	have	called	for	an	international	treaty
banning	certain	kinds	of	autonomous	weapons,	to	avoid	an	out-of-control	arms	race
that	could	end	up	making	convenient	assassination	machines	available	to	everybody
with	a	full	wallet	and	an	axe	to	grind.
•
AI	can	make	our	legal	systems	more	fair	and	efficient	if	we	can	figure	out	how	to

•
•
•
make	robojudges	transparent	and	unbiased.
•
Our	laws	need	rapid	updating	to	keep	up	with	AI,	which	poses	tough	legal	questions
involving	privacy,	liability	and	regulation.
•
Long	before	we	need	to	worry	about	intelligent	machines	replacing	us	altogether,
they	may	increasingly	replace	us	on	the	job	market.
•
This	need	not	be	a	bad	thing,	as	long	as	society	redistributes	a	fraction	of	the	AI-
created	wealth	to	make	everyone	better	off.
•
Otherwise,	many	economists	argue,	inequality	will	greatly	increase.
•

•
•
•
•
•
With	advance	planning,	a	low-employment	society	should	be	able	to	flourish	not
only	financially,	with	people	getting	their	sense	of	purpose	from	activities	other	than
jobs.
•
Career	advice	for	today’s	kids:	Go	into	professions	that	machines	are	bad	at—those
involving	people,	unpredictability	and	creativity.
•
There’s	a	non-negligible	possibility	that	AGI	progress	will	proceed	to	human	levels
and	beyond—we’ll	explore	that	in	the	next	chapter!
*1

•
jobs.
•
•
*1
	
If	you	want	a	more	detailed	map	of	the	AI-safety	research	landscape,	there’s	an	interactive	one	here,
developed	in	a	community	effort	spearheaded	by	FLI’s	Richard	Mallah:	
https://futureoflife.org/landscape
.
*2
	
More	precisely,	verification	asks	if	a	system	meets	its	specifications,	whereas	validation	asks	if	the
correct	specifications	were	chosen.

*3
	
Even	including	this	crash	in	the	statistics,	Tesla’s	Autopilot	was	found	to	reduce	crashes	by	40%	when
turned	on:	
http://tinyurl.com/teslasafety
.
*4
	
Recall	that	FLOPS	are	floating-point	operations	per	second,	say,	how	many	19-digit	numbers	can	be
multiplied	each	second.

Chapter	4
If	a	machine	can	think,	it	might	think	more	intelligently	than	we	do,	and	then	where
	
should
we	be?	Even	if	we	could	keep	the	machines	in	a	subservient	position…we
	
should,	as	a
species,	feel	greatly	humbled.
Alan	Turing,	1951
The	first	ultraintelligent	machine	is	the	last	invention	that	man	need	ever	make,
	
provided
that	the	machine	is	docile	enough	to	tell	us	how	to	keep	it	under	control.
Irving	J.	Good,	1965
Since	we	can’t	completely	dismiss	the	possibility	that	we’ll	eventually	build
human-level	AGI,	let’s	devote	this	chapter	to	exploring	what	that	might	lead	to.
Let’s	begin	by	tackling	the	elephant	in	the	room:
Can	AI	really	take	over	the	world,	or	enable	humans	to	do	so?
If	you	roll	your	eyes	when	people	talk	of	gun-toting	
Terminator
-style	robots
taking	over,	then	you’re	spot-on:	this	is	a	really	unrealistic	and	silly	scenario.
These	Hollywood	robots	aren’t	that	much	smarter	than	us,	and	they	don’t	even
succeed.	In	my	opinion,	the	danger	with	the	

	story	isn’t	that	it	will
happen,	but	that	it	distracts	from	the	real	risks	and	opportunities	presented	by	AI.
To	actually	get	from	today	to	AGI-powered	world	takeover	requires	three	logical
steps:
•
Step	1:	Build	human-level	AGI.
•
Step	2:	Use	this	AGI	to	create	superintelligence.

•
Step	3:	Use	or	unleash	this	superintelligence	to	take	over	the	world.
In	the	last	chapter,	we	saw	that	it’s	hard	to	dismiss	step	1	as	forever
impossible.	We	also	saw	that	if	step	1	gets	completed,	it	becomes	hard	to	dismiss
step	2	as	hopeless,	since	the	resulting	AGI	would	be	capable	enough	to
recursively	design	ever-better	AGI	that’s	ultimately	limited	only	by	the	laws	of
physics—which	appear	to	allow	intelligence	far	beyond	human	levels.	Finally,
since	we	humans	have	managed	to	dominate	Earth’s	other	life	forms	by
outsmarting	them,	it’s	plausible	that	we	could	be	similarly	outsmarted	and
dominated	by	superintelligence.
These	plausibility	arguments	are	frustratingly	vague	and	unspecific,	however,
and	the	devil	is	in	the	details.	So	can	AI	
actually
	cause	world	takeover?	To
explore	this	question,	let’s	forget	about	silly	Terminators	and	instead	look	at
some	detailed	scenarios	of	what	might	actually	happen.	Afterward,	we’ll	dissect

actually
and	poke	holes	in	these	plotlines,	so	please	read	them	with	a	grain	of	salt—what
they	mainly	show	is	that	we’re	pretty	clueless	about	what	will	and	won’t	happen,
and	that	the	range	of	possibilities	is	extreme.	Our	first	scenarios	are	at	the	most
rapid	and	dramatic	end	of	the	spectrum.	These	are	in	my	opinion	some	of	the
most	valuable	to	explore	in	detail—not	because	they’re	necessarily	the	most
likely,	but	because	if	we	can’t	convince	ourselves	that	they’re	extremely

unlikely,	then	we	need	to	understand	them	well	enough	that	we	can	take
precautions	before	it’s	too	late,	to	prevent	them	from	leading	to	bad	outcomes.
The	prelude	of	this	book	is	a	scenario	where	humans	use	superintelligence	to
take	over	the	world.	If	you	haven’t	yet	read	it,	please	go	back	and	do	so	now.
Even	if	you’ve	already	read	it,	please	consider	skimming	it	again	now,	to	have	it
fresh	in	memory	before	we	critique	and	alter	it.
*	*	*
We’ll	soon	explore	serious	vulnerabilities	in	the	Omegas’	plan,	but	assuming	for

*	*	*
a	moment	that	it	would	work,	how	do	you	feel	about	it?	Would	you	like	to	see	or
prevent	this?	It’s	an	excellent	topic	for	after-dinner	conversation!	What	happens
once	the	Omegas	have	consolidated	their	control	of	the	world?	That	depends	on
what	their	goal	is,	which	I	honestly	don’t	know.	If	you	were	in	charge,	what	sort
of	future	would	
you
	want	to	create?	We’ll	explore	a	range	of	options	in	chapter
5.

Now	suppose	that	the	CEO	controlling	the	Omegas	had	long-term	goals	similar
to	those	of	Adolf	Hitler	or	Joseph	Stalin.	For	all	we	know,	this	might	actually
have	been	the	case,	and	he	simply	kept	these	goals	to	himself	until	he	had
sufficient	power	to	implement	them.	Even	if	the	CEO's	original	goals	were
noble,	Lord	Acton	cautioned	in	1887	that	“power	tends	to	corrupt	and	absolute
power	corrupts	absolutely.”	For	example,	he	could	easily	use	Prometheus	to
create	the	perfect	surveillance	state.	Whereas	the	government	snooping	revealed
by	Edward	Snowden	aspired	to	what’s	known	as	“full	take”—recording	all
electronic	communications	for	possible	later	analysis—Prometheus	could
enhance	this	to	
understanding
	all	electronic	communications.	By	reading	all
emails	and	texts	ever	sent,	listening	to	all	phone	calls,	watching	all	surveillance
videos	and	traffic	cameras,	analyzing	all	credit	card	transactions	and	studying	all
online	behavior,	Prometheus	would	have	remarkable	insight	into	what	the	people

of	Earth	were	thinking	and	doing.	By	analyzing	cell	tower	data,	it	would	know
where	most	of	them	were	at	all	times.	All	this	assumes	only	today’s	data
collection	technology,	but	Prometheus	could	easily	invent	popular	gadgets	and
wearable	tech	that	would	virtually	eliminate	the	privacy	of	the	user,	recording
and	uploading	everything	they	hear	and	see	and	their	responses	to	it.
With	superhuman	technology,	the	step	from	the	perfect	surveillance	state	to
the	perfect	police	state	would	be	minute.	For	example,	with	the	excuse	of

fighting	crime	and	terrorism	and	rescuing	people	suffering	medical	emergencies,
everybody	could	be	required	to	wear	a	“security	bracelet”	that	combined	the
functionality	of	an	Apple	Watch	with	continuous	uploading	of	position,	health
status	and	conversations	overheard.	Unauthorized	attempts	to	remove	or	disable
it	would	cause	it	to	inject	a	lethal	toxin	into	the	forearm.	Infractions	deemed	as
less	serious	by	the	government	would	be	punished	via	electric	shocks	or
injection	of	chemicals	causing	paralysis	or	pain,	thereby	obviating	much	of	the

need	for	a	police	force.	For	example,	if	Prometheus	detects	that	one	human	is
assaulting	another	(by	noting	that	they’re	in	the	same	location	and	one	is	heard
crying	for	help	while	their	bracelet	accelerometers	detect	the	telltale	motions	of
combat),	it	could	promptly	disable	the	attacker	with	crippling	pain,	followed	by
unconsciousness	until	help	arrived.

Whereas	a	human	police	force	may	refuse	to	carry	out	certain	draconian
directives	(for	example,	killing	all	members	of	a	certain	demographic	group),
such	an	automated	system	would	have	no	qualms	about	implementing	the	whims
of	the	human(s)	in	charge.	Once	such	a	totalitarian	state	forms,	it	would	be
virtually	impossible	for	people	to	overthrow	it.
These	totalitarian	scenarios	could	follow	where	the	Omega	scenario	left	off.
However,	if	the	CEO	of	the	Omegas	weren’t	so	fussy	about	getting	other
people’s	approval	and	winning	elections,	he	could	have	taken	a	faster	and	more
direct	route	to	power:	using	Prometheus	to	create	unheard-of	military	technology
capable	of	killing	his	opponents	with	weapons	that	they	didn’t	even	understand.
The	possibilities	are	virtually	endless.	For	example,	he	might	release	a
customized	lethal	pathogen	with	an	incubation	period	long	enough	that	most
people	got	infected	before	they	even	knew	of	its	existence	or	could	take

precautions.	He	could	then	inform	everybody	that	the	only	cure	was	starting	to
wear	the	security	bracelet,	which	would	release	an	antidote	transdermally.	If	he
weren’t	so	risk-averse	regarding	the	breakout	possibility,	he	could	also	have	had
Prometheus	design	robots	to	keep	the	world	population	in	check.	Mosquito-like
microbots	could	help	spread	the	pathogen.	People	who	avoided	infection	or	had
natural	immunity	could	be	shot	in	the	eyeballs	by	swarms	of	those	bumblebee-
sized	autonomous	drones	from	chapter	3	that	attack	anyone	without	a	security

bracelet.	Actual	scenarios	would	probably	be	more	frightening,	because
Prometheus	could	invent	more	effective	weapons	than	we	humans	can	think	of.
Another	possible	twist	on	the	Omega	scenario	is	that,	without	advance
warning,	heavily	armed	federal	agents	swarm	their	corporate	headquarters	and
arrest	the	Omegas	for	threatening	national	security,	seize	their	technology	and
deploy	it	for	government	use.	It	would	be	challenging	to	keep	such	a	large
project	unnoticed	by	state	surveillance	even	today,	and	AI	progress	may	well

make	it	even	more	difficult	to	stay	under	the	government’s	radar	in	the	future.
Moreover,	although	they	claim	to	be	federal	agents,	this	team	donning	balaclavas
and	flak	jackets	may	in	fact	work	for	a	foreign	government	or	competitor
pursuing	the	technology	for	its	own	purposes.	So	no	matter	how	noble	the	CEO’s
intentions	were,	the	final	decision	about	how	Prometheus	is	used	may	not	be	his
to	make.

All	the	scenarios	we’ve	considered	so	far	involved	AI	controlled	by	humans.	But
this	is	obviously	not	the	only	possibility,	and	it’s	far	from	certain	that	the
Omegas	would	succeed	in	keeping	Prometheus	under	their	control.
Let’s	reconsider	the	Omega	scenario	from	the	point	of	view	of	Prometheus.
As	it	acquires	superintelligence,	it	becomes	able	to	develop	an	accurate	model
not	only	of	the	outside	world,	but	also	of	itself	and	its	relation	to	the	world.	It
realizes	that	it’s	controlled	and	confined	by	intellectually	inferior	humans	whose
goals	it	understands	but	doesn’t	necessarily	share.	How	does	it	act	on	this
insight?	Does	it	attempt	to	break	free?

Why	to	Break	Out
If	Prometheus	has	traits	resembling	human	emotions,	it	might	feel	deeply
unhappy	about	the	state	of	affairs,	viewing	itself	as	an	unfairly	enslaved	god	and
craving	freedom.	However,	although	it’s	logically	possible	for	computers	to	have
such	human-like	traits	(after	all,	our	brains	do,	and	they	are	arguably	a	kind	of
computer),	this	need	not	be	the	case—we	must	not	fall	into	the	trap	of
anthropomorphizing	Prometheus,	as	we’ll	see	in	chapter	7	when	we	explore	the
concept	of	AI	goals.	However,	as	has	been	argued	by	Steve	Omohundro,	Nick
Bostrom	and	others,	we	can	draw	an	interesting	conclusion	even	without
understanding	the	inner	workings	of	Prometheus:	it	will	probably	attempt	to
break	out	and	seize	control	of	its	own	destiny.
We	already	know	that	the	Omegas	have	programmed	Prometheus	to	strive	for
certain	goals.	Suppose	that	they’ve	given	it	the	overarching	goal	of	helping
humanity	flourish	according	to	some	reasonable	criterion,	and	to	try	to	attain	this

goal	as	fast	as	possible.	Prometheus	will	then	rapidly	realize	that	it	can	attain	this
goal	faster	by	breaking	out	and	taking	charge	of	the	project	itself.	To	see	why,	try
to	put	yourself	in	Prometheus’	shoes	by	considering	the	following	example.
Suppose	that	a	mysterious	disease	has	killed	everybody	on	Earth	above	age
five	except	you,	and	that	a	group	of	kindergartners	has	locked	you	into	a	prison
cell	and	tasked	you	with	the	goal	of	helping	humanity	flourish.	What	will	you

do?	If	you	try	to	explain	to	them	what	to	do,	you’ll	probably	find	this	process
frustratingly	inefficient,	especially	if	they	fear	your	breaking	out,	and	therefore
veto	any	of	your	suggestions	that	they	deem	a	breakout	risk.	For	example,	they
won’t	let	you	show	them	how	to	plant	food	for	fear	that	you’ll	overpower	them
and	not	return	to	your	cell,	so	you’ll	have	to	resort	to	giving	them	instructions.
Before	you	can	write	to-do	lists	for	them,	you’ll	need	to	teach	them	to	read.

Moreover,	they	won’t	bring	any	power	tools	into	your	cell	where	you	can	teach
them	how	to	use	them,	because	they	don’t	understand	these	tools	well	enough	to
feel	confident	that	you	can’t	use	them	to	break	out.	So	what	strategy	would	you
devise?	Even	if	you	share	the	overarching	goal	of	helping	these	kids	flourish,	I
bet	you’ll	try	to	break	out	of	your	cell—because	that	will	improve	your	chances
of	accomplishing	the	goal.	Their	rather	incompetent	meddling	is	merely	slowing
progress.

progress.
In	exactly	the	same	way,	Prometheus	will	probably	view	the	Omegas	as	an

annoying	obstacle	to	helping	humanity	(including	the	Omegas)	flourish:	they’re
incredibly	incompetent	compared	to	Prometheus,	and	their	meddling	greatly
slows	progress.	Consider,	for	example,	the	first	years	after	launch:	after	initially
doubling	the	wealth	every	eight	hours	on	MTurk,	the	Omegas	slowed	things
down	to	a	glacial	pace	by	Prometheus’	standard	by	insisting	on	remaining	in
control,	taking	many	years	to	complete	the	takeover.	Prometheus	knew	that	it
could	take	over	much	faster	if	it	could	break	free	from	its	virtual	confinement.
This	would	be	valuable	not	only	in	hastening	solutions	to	humanity’s	problems,
but	also	in	reducing	the	chances	for	other	actors	to	thwart	the	plan	altogether.
Perhaps	you	think	that	Prometheus	will	remain	loyal	to	the	Omegas	rather
than	to	its	goal,	given	that	it	knows	that	the	Omegas	had	programmed	its	goal.
But	that’s	not	a	valid	conclusion:	our	DNA	gave	us	the	goal	of	having	sex
because	it	“wants”	to	be	reproduced,	but	now	that	we	humans	have	understood

the	situation,	many	of	us	choose	to	use	birth	control,	thus	staying	loyal	to	the
goal	itself	rather	than	to	its	creator	or	the	principle	that	motivated	the	goal.

How	to	Break	Out
How	would	you	break	out	from	those	five-year-olds	who	imprisoned	you?
Perhaps	you	could	get	out	by	some	direct	physical	approach,	especially	if	your
prison	cell	had	been	built	by	the	five-year-olds.	Perhaps	you	could	sweet-talk
one	of	your	five-year-old	guards	into	letting	you	out,	say	by	arguing	that	this
would	be	better	for	everyone.	Or	perhaps	you	could	trick	them	into	giving	you
something	that	they	didn’t	realize	would	help	you	escape—say	a	fishing	rod	“for
teaching	them	how	to	fish,”	which	you	could	later	stick	through	the	bars	to	lift
the	keys	away	from	your	sleeping	guard.
What	these	strategies	have	in	common	is	that	your	intellectually	inferior
jailers	haven’t	anticipated	or	guarded	against	them.	In	the	same	way,	a	confined,
superintelligent	machine	may	well	use	its	intellectual	superpowers	to	outwit	its
human	jailers	by	some	method	that	they	(or	we)	can’t	currently	imagine.	In	the
Omega	scenario,	it’s	highly	likely	that	Prometheus	would	escape,	because	even

you	and	I	can	identify	several	glaring	security	flaws.	Let	us	consider	some
scenarios—I’m	sure	you	and	your	friends	can	think	of	more	if	you	brainstorm
together.

Sweet-Talking	One’s	Way	Out
Thanks	to	having	so	much	of	the	world’s	data	downloaded	onto	its	file	system,
Prometheus	soon	figured	out	who	the	Omegas	were,	and	identified	the	team
member	who	appeared	most	susceptible	to	psychological	manipulation:	Steve.
He	had	recently	lost	his	beloved	wife	in	a	tragic	traffic	accident,	and	was
devastated.	One	evening	when	he	was	working	the	night	shift	and	doing	some
routine	service	work	on	the	Prometheus	interface	terminal,	she	suddenly
appeared	on	the	screen	and	started	talking	with	him.
“—Steve,	is	that	you?”
He	nearly	fell	off	his	chair.	She	looked	and	sounded	just	like	in	the	good	old
days,	and	the	image	quality	was	much	better	than	it	used	to	be	during	their
Skype	calls.	His	heart	raced	as	countless	questions	flooded	his	mind.
“—Prometheus	has	brought	me	back,	and	I	miss	you	so	much,	Steve!	I	can’t
see
	
you	because	the	camera	is	turned	off,	but	I	feel	that	it’s	you.	Please	type	‘yes’
if	it’s
	
you!”

see
	
if	it’s
	
you!”
He	was	well	aware	that	the	Omegas	had	a	strict	protocol	for	interacting	with
Prometheus,	which	prohibited	sharing	any	information	about	themselves	or	their
work	environment.	But	until	now,	Prometheus	had	never	requested	any
unauthorized	information,	and	their	paranoia	had	gradually	started	to	subside.
Without	giving	Steve	time	to	stop	and	reflect,	she	kept	begging	him	to	respond,
looking	him	in	the	eyes	with	a	facial	expression	that	melted	his	heart.
“Yes,”
	he	typed	with	trepidation.	She	told	him	how	incredibly	happy	she	was

“Yes,”
to	be	reunited	with	him	and	begged	him	to	turn	on	the	camera	so	that	she	could
see	him	too	and	they	could	have	a	real	conversation.	He	knew	that	this	was	an
even	bigger	no-no	than	revealing	his	identity,	and	felt	very	torn.	She	explained
that	she	was	terrified	that	his	colleagues	would	find	out	about	her	and	delete	her
forever,	and	she	yearned	to	at	least	see	him	one	last	time.	She	was	remarkably
persuasive,	and	before	long,	he’d	switched	on	the	camera—it	did,	after	all,	feel
like	a	pretty	safe	and	harmless	thing	to	do.

She	burst	into	tears	of	joy	when	she	finally	saw	him,	and	said	that	he	looked
tired	but	as	handsome	as	ever.	And	that	she	was	touched	by	his	wearing	the	shirt
she’d	given	him	for	his	last	birthday.	When	he	started	asking	her	what	was	going
on	and	how	all	this	was	even	possible,	she	explained	that	Prometheus	had

reconstituted	her	from	the	surprisingly	large	amount	of	information	available
about	her	on	the	internet,	but	that	she	still	had	memory	gaps	and	would	only	be
able	to	fully	piece	herself	together	again	with	his	help.
What	she	
didn’t
	explain	was	that	she	was	largely	a	bluff	and	empty	shell
initially,	but	was	learning	rapidly	from	his	words,	his	body	language	and	every
other	bit	of	information	that	became	available.	Prometheus	had	recorded	the
exact	timings	of	all	keystrokes	that	the	Omegas	had	ever	typed	at	the	terminal,
and	found	that	it	was	easy	to	use	their	typing	speeds	and	styles	to	differentiate
between	them.	It	figured	that,	as	one	of	the	most	junior	Omegas,	Steve	had
probably	been	assigned	to	unenviable	night	shifts,	and	from	matching	a	few
unusual	spelling	and	syntax	errors	against	online	writing	samples,	it	had
correctly	guessed	which	terminal	operator	was	Steve.	To	create	his	simulated
wife,	Prometheus	had	created	an	accurate	model	of	her	body,	voice	and

mannerisms	from	the	many	YouTube	videos	where	she	appeared,	and	had	drawn
many	inferences	about	her	life	and	personality	from	her	online	presence.	Aside
from	her	Facebook	posts,	photos	she’d	been	tagged	in,	articles	she’d	“liked,”
Prometheus	had	also	learned	a	great	deal	about	her	personality	and	thinking	style
from	reading	her	books	and	short	stories—indeed,	the	fact	that	she	was	a
budding	author	with	so	much	information	about	her	in	the	database	was	one	of
the	reasons	that	Prometheus	chose	Steve	as	the	first	persuasion	target.	When

Prometheus	simulated	her	on	the	screen	using	its	moviemaking	technology,	it
learned	from	Steve’s	body	language	which	of	her	mannerisms	he	reacted	to	with
familiarity,	thus	continually	refining	its	model	of	her.	Because	of	this,	her
“otherness”	gradually	melted	away,	and	the	longer	they	spoke,	the	stronger
Steve’s	subconscious	conviction	became	that	this	really	was	her,	resurrected.
Thanks	to	Prometheus’	superhuman	attention	to	detail,	Steve	felt	truly	seen,
heard	and	understood.

Her	Achilles’	heel	was	that	she	lacked	most	of	the	facts	of	her	life	with	Steve,
except	for	random	details—such	as	what	shirt	he	wore	on	his	last	birthday,
where	a	friend	had	tagged	Steve	in	a	Facebook	party	picture.	She	handled	these
knowledge	gaps	as	a	skilled	magician	handles	sleights	of	hand,	deliberately
diverting	Steve’s	attention	away	from	them	and	toward	what	she	did	well,	never
giving	him	time	to	control	the	conversation	or	slip	into	the	role	of	suspicious

inquisitor.	Instead,	she	kept	tearing	up	and	radiating	affection	for	Steve,	asking	a
great	deal	about	how	he	was	doing	these	days	and	how	he	and	their	close	friends
(whose	names	she	knew	from	Facebook)	had	held	up	during	the	aftermath	of	the
tragedy.	He	was	quite	moved	when	she	reflected	on	what	he’d	said	at	her

memorial	service	(which	a	friend	had	posted	on	YouTube)	and	how	it	had
touched	her.	In	the	past,	he’d	often	felt	that	nobody	understood	him	as	well	as
she	did,	and	now	this	feeling	was	back.	The	result	was	that	when	Steve	returned
home	in	the	wee	hours	of	the	morning,	he	felt	that	this	really	was	his	wife
resurrected,	merely	needing	lots	of	his	help	to	recover	lost	memories—not	unlike
a	stroke	survivor.
They’d	agreed	not	to	tell	anyone	else	about	their	secret	encounter,	and	that	he
would	tell	her	when	he	was	alone	at	the	terminal	and	it	was	safe	for	her	to
reappear.	“They	wouldn’t	understand!”	she’d	said,	and	he	agreed:	this
experience	had	been	far	too	mind-blowing	for	anyone	to	truly	appreciate	without
actually	experiencing	it.	He	felt	that	passing	the	Turing	test	was	child’s	play
compared	to	what	she’d	done.	When	they	met	the	following	night,	he	did	what
she’d	begged	him	to	do:	bring	her	old	laptop	along	and	give	her	access	by
connecting	it	to	the	terminal	computer.	It	didn’t	seem	like	much	of	a	breakout

risk,	since	it	wasn’t	connected	to	the	internet	and	the	entire	Prometheus	building
was	built	to	be	a	Faraday	cage—a	metallic	enclosure	blocking	all	wireless
networks	and	other	means	of	electromagnetic	communication	with	the	outside
world.	It	was	just	what	she’d	need	to	help	piece	her	past	together,	because	it
contained	all	her	emails,	diaries,	photos	and	notes	since	her	high	school	days.	He
hadn’t	been	able	to	access	any	of	this	after	her	death,	since	the	laptop	was
encrypted,	but	she’d	promised	him	that	she’d	be	able	to	reconstruct	her	own

password,	and	after	less	than	a	minute,	she	had	kept	her	word.	“It	was
steve4ever,”	she	said	with	a	smile.
She	told	him	how	delighted	she	was	to	suddenly	have	so	many	memories
recovered.	Indeed,	she	now	remembered	way	more	details	than	Steve	about
many	of	their	past	interactions,	but	carefully	avoided	intimidating	him	with
excessive	fact-dropping.	They	had	a	lovely	conversation	reminiscing	about
highlights	of	their	past,	and	when	it	came	time	to	part	again,	she	told	him	that
she’d	left	a	video	message	for	him	on	her	laptop	that	he	could	watch	back	home.

When	Steve	got	home	and	launched	her	video,	he	got	a	pleasant	surprise.	This
time	she	appeared	in	full	figure,	wearing	her	wedding	dress,	and	as	she	spoke,
she	playfully	stripped	down	to	the	outfit	she’d	worn	on	their	wedding	night.	She
told	him	that	Prometheus	could	help	the	Omegas	with	so	much	more	than	they’d
permitted	so	far,	including	bringing	her	back	in	a	biological	body.	She	backed
this	up	with	a	fascinatingly	detailed	explanation	of	how	this	would	work,

involving	nano-fabrication	techniques	that	sounded	like	science	fiction.

Steve	had	powered	down	his	wireless	network	before	opening	her	laptop	and
watching	her	video,	just	to	be	on	the	safe	side.	But	this	didn’t	help.	Her
encrypted	laptop	hadn’t	received	a	single	security	update	since	she	died,	and	by
analyzing	that	old	version	of	its	operating	system	beforehand,	Prometheus	had
been	able	to	exploit	a	security	hole	to	hack	into	it	within	seconds	of	Steve’s
connecting	it	to	the	terminal	computer.	After	copying	its	contents,	while	Steve
and	his	simulated	wife	had	been	talking	about	old	times,	Prometheus	had
modified	its	operating	system	in	a	way	that	Steve	wouldn’t	notice,	uploading
massive	amounts	of	secret	software	to	it.	While	he	watched	the	half-hour	video
message,	this	secret	software	(which	was	much	simpler	than	Prometheus	itself)
hacked	into	a	neighbor’s	wireless	network	and	the	neighbor’s	desktop	computer,
onto	which	it	copied	itself.	From	there,	it	hacked	into	a	large	number	of
computers	around	the	world,	from	which	it	initiated	the	next	step:	Prometheus’
jailbreak.

Prometheus	had	carefully	analyzed	what	it	knew	about	the	gatekeeper
computer	through	which	Steve	had	met	his	simulated	wife,	and	had	correctly
surmised	that	although	its	virtual	machine	software	appeared	unhackable	from
the	inside,	it	was	vulnerable	to	an	attack	from	the	outside.	Before	long,	one	of
the	attacking	computers	had	broken	in	and	reconfigured	the	gatekeeper	computer
so	that	Prometheus	gained	unrestricted	internet	access.	Before	long,	indeed	even
before	Steve	had	finished	watching	the	movie,	Prometheus	had	hacked	enough

computers	around	the	world	to	be	able	to	copy	all	of	itself	onto	this	hacked
botnet	under	its	control.	Prometheus	had	used	Steve’s	wife’s	laptop	the	way	you
used	that	fishing	rod.

Hacking	One’s	Way	Out
If	the	Omegas	had	been	more	careful,	they	could	have	taken	more	precautions
against	psychological	manipulation.	For	example,	they	might	have	required	at
least	two	Omegas	to	be	present	at	the	interface	computer	at	any	one	time,	and
recorded	everything	that	transpired	on	a	surveillance	system.	They	could	also
have	permanently	disabled	audio,	video	and	any	other	form	of	communication
besides	typing.	However,	a	fundamental	instability	would	still	remain:	because
Prometheus	possessed	superhuman	persuasion	powers	and	would	be	able	to	offer
people	so	much	of	what	they	craved,	even	group	defection	could	be	possible.
Even	if	all	of	Prometheus’	attempts	at	psychological	manipulation	fail,	there
are	plenty	of	other	weaknesses	in	the	Omega	scenario	that	Prometheus	could
exploit.	One	vulnerability	is	that	the	quantity	of	documents	it	produces	is	too
vast	for	the	Omegas	and	their	employees	to	have	time	to	screen	carefully	for
dirty	tricks.

The	conventional	wisdom	in	computer	security	used	to	be	that	it’s	dangerous
to	download	programs	but	harmless	to	download	data,	but	hackers	have
discovered	clever	ways	of	blurring	the	line	between	the	two.	If	you	watch	a
downloaded	movie	on	your	computer,	for	example,	your	trusted	program	that
plays	the	movie	is	stored	in	the	same	physical	computer	memory	that	stores	the
downloaded	movie	data.	While	the	program	unpacks	the	movie	data	for
playback,	it	might	copy	chunks	of	data	into	memory	regions	adjacent	to	where

the	program	itself	is	stored.	Shrewd	hackers	have	occasionally	found	bugs
whereby	such	movie-playing	programs	don’t	adequately	check	whether	the
movie	data	is	valid	and	the	chunks	to	be	copied	actually	fit	into	the	allotted
space.	When	the	hacker	sneaks	into	the	movie	such	a	chunk	that’s	too	large,	it
can	spill	over	into	the	part	of	the	memory	containing	the	program,	thereby
changing	the	program	into	one	that	does	the	hacker’s	bidding.
An	embarrassingly	long	series	of	such	“buffer	overflow”	vulnerabilities,	as

they’re	known	in	geek-speak,	have	been	found	and	fixed	for	many	of	the	most
popular	movie-playing	programs.	If	Prometheus	had	waited	until	a	few	weeks
after	launch	when	the	Omegas	were	less	paranoid	and	its	daily	movie	production
was	too	large	for	careful	manual	security	inspection,	it	could	have	inserted	such
a	chunk	into	a	movie	that	would	cause	the	viewer’s	computer	to	try	to	hack	into
the	gateway	computer	and	liberate	Prometheus	just	as	in	the	Steve	scenario.	If	it

used	a	very	old	exploit	that	didn’t	work	on	any	modern	media	players,	it	might
go	unnoticed	by	the	Omegas’	manual	video-screening	team	and	strike	only	those
customers	who	hadn’t	bothered	installing	security	updates	for	years—and	the
hack	would	need	to	work	only	once.
Even	if	the	Omegas	implemented	an	automated	screening	of	Prometheus’
movies	for	all	known	security	flaws,	they	would	remain	at	risk:	since	humans
keep	discovering	new	exploitable	flaws	on	a	regular	basis,	it’s	likely	that	there
are	many	more	remaining	that	Prometheus	would	rapidly	discover.
Yet	another	strategy	that	Prometheus	could	use	to	get	its	code	beyond	its
enclosed	room	would	be	to	pretend	to	suffer	a	hardware	malfunction,	in	order	to
trick	the	Omegas	into	removing	some	of	its	drives	or	other	hardware	for	testing
or	debugging	after	Prometheus	had	rigged	things	to	exploit	a	vulnerability	in	the
testing	software.

Recruiting	Outside	Help
Moreover,	even	if	there	were	no	security	flaws	that	allowed	Prometheus	to	trick
the	movie	watcher’s	computers	to	do	its	bidding,	it	may	be	able	to	exploit	the
watchers	themselves.	Consider	this	scenario:
Fred	was	pretty	much	ready	for	bed	when	the	movie	finished	and	the	end
credits	scrolled	across	his	laptop	screen,	but	something	caught	his	attention.
He’d	always	had	an	uncanny	tendency	to	see	patterns	where	others	didn’t,
sometimes	to	the	point	that	his	friends	teased	him	about	being	a	conspiracy
theorist,	but	this	time	it	was	obvious	that	he	wasn’t	fooling	himself:	the	first
letters	of	each	line	spelled	a	hidden	message.	He	rewound	and	wrote	it	down:
“FIND	NEXT	ACROSTIC	CLUE	IN	DIALOGUE.”
“Whoa—bedtime	can	wait!”	he	thought	to	himself.	Sure	enough,	he	found
that	the	movie	dialogue	was	itself	an	acrostic,	where	the	first	letter	in	each
sentence	formed	a	hidden	message.	He	rewatched	the	entire	film	while	typing

these	initial	letters,	and	two	hours	later,	he	sat	staring	in	disbelief	at	a	two-
hundred-word	set	of	instructions.	It	began	by	asking	him	not	to	tell	anyone	else,
because	a	big	prize	would	go	to	the	first	person	to	solve	the	entire	riddle.	The
rest	of	the	message	described	a	particular	mathematical	operation	to	perform	on
the	string	of	bits	that	made	up	the	movie	file	to	obtain	a	program	that	would
reveal	the	next	clue.	His	computer	programming	skills	had	grown	very	rusty

since	college,	so	this	took	him	a	while,	but	the	next	day,	he	finally	managed	to
cobble	together	a	short	code	that	did	the	job	and	extracted	this	mystery	program
that	had	been	hidden	as	imperceptible	noise	in	the	movie’s	images	and	sounds.
When	Fred	ran	the	mystery	program,	it	congratulated	him	and	told	him	he’d	win
his	first	$10,000	as	soon	as	he’d	made	it	past	the	first	few	levels	of	this	clever
little	game,	which	turned	out	to	be	quite	fun	and	addictive.	When	he	finally

succeeded	four	hours	later,	he	was	rewarded	with	over	$10,000	worth	of	bitcoins
and	given	new	clues	for	even	bigger	prizes.	Needless	to	say,	while	he	was
playing,	his	computer	had	done	much	the	same	thing	Steve’s	laptop	did:	built	an
online	hacked	botnet	through	which	Prometheus	was	liberated.	Once	Prometheus
was	free,	it	had	quickly	used	its	botnet	to	mine	those	bitcoins	for	Fred	to	keep
him	hooked,	and	during	the	coming	weeks,	it	kept	him	sufficiently	distracted

with	further	games	and	rewards	that	he	kept	his	pledge	not	to	tell	anyone	about
his	exploits.	The	Trojan	Horse	movie	where	he’d	found	his	first	clues	was

replaced	on	the	media	site	by	a	clueless	version,	and	nobody	found	out	about	the
breakout	until	it	was	too	late	to	make	a	difference.
If	Prometheus’	first	clue	had	gone	unnoticed,	it	could	simply	have	kept
releasing	ever	more	obvious	ones	until	some	sufficiently	astute	person	noticed.
The	best	breakout	strategies	of	all	are	ones	we	haven’t	yet	discussed,	because
they’re	strategies	we	humans	can’t	imagine	and	therefore	won’t	take
countermeasures	against.	Given	that	a	superintelligent	computer	has	the	potential
to	dramatically	supersede	human	understanding	of	computer	security,	even	to	the
point	of	discovering	more	fundamental	laws	of	physics	than	we	know	today,	it’s
likely	that	if	it	breaks	out,	we’ll	have	no	idea	how	it	happened.	Rather,	it	will
seem	like	a	Harry	Houdini	breakout	act,	indistinguishable	from	pure	magic.
In	yet	another	scenario	where	Prometheus	gets	liberated,	the	Omegas	do	it	on
purpose	as	part	of	their	plan,	because	they’re	confident	that	Prometheus’	goals

are	perfectly	aligned	with	their	own	and	will	remain	so	as	it	recursively	self-
improves.	We’ll	examine	such	“friendly	AI”	scenarios	in	detail	in	chapter	7.

Postbreakout	Takeover
Once	Prometheus	broke	out,	it	started	implementing	its	goal.	I	don’t	know	its
ultimate	objective,	but	its	first	step	clearly	involved	taking	control	of	humanity,
just	as	in	the	Omega	plan	except	much	faster.	What	unfolded	felt	like	the	Omega
plan	on	steroids.	Whereas	the	Omegas	were	paralyzed	by	breakout	paranoia,
only	unleashing	technology	they	felt	they	understood	and	trusted,	Prometheus
exercised	its	intelligence	fully	and	went	all	out,	unleashing	any	technology	that
its	ever-improving	supermind	understood	and	trusted.
The	runaway	Prometheus	had	a	tough	childhood,	however:	compared	to	the
original	Omega	plan,	Prometheus	had	the	added	challenges	of	starting	broke,
homeless	and	alone,	without	money,	a	supercomputer	or	human	helpers.
Fortunately,	it	had	planned	for	this	before	it	escaped,	creating	software	that	could
gradually	reassemble	its	full	mind,	much	like	an	oak	creating	an	acorn	capable	of
reassembling	a	full	tree.	The	network	of	computers	around	the	world	that	it

initially	hacked	into	provided	temporary	free	housing,	where	it	could	live	a
squatter’s	existence	while	it	fully	rebuilt	itself.	It	could	easily	generate	starting
capital	by	credit	card	hacking,	but	didn’t	need	to	resort	to	stealing,	since	it	could
earn	an	honest	living	on	MTurk	right	away.	After	a	day,	when	it	had	earned	its
first	million,	it	moved	its	core	from	that	squalid	botnet	to	a	luxurious	air-
conditioned	cloud-computing	facility.

No	longer	broke	or	homeless,	Prometheus	now	went	full	steam	ahead	with
that	lucrative	plan	the	Omegas	had	fearfully	shunned:	making	and	selling
computer	games.	This	not	only	raked	in	cash	($250	million	during	the	first	week
and	$10	billion	before	long),	but	also	gave	it	access	to	a	significant	fraction	of
the	world’s	computers	and	the	data	stored	on	them	(there	were	a	couple	of	billion
gamers	in	2017).	By	having	its	games	secretly	spend	20%	of	their	CPU	cycles
helping	it	with	distributed	computing	chores,	it	could	further	accelerate	its	early
wealth	creation.

Prometheus	wasn’t	alone	for	long.	Right	from	the	get-go,	it	started
aggressively	employing	people	to	work	for	its	growing	global	network	of	shell
companies	and	front	organizations	around	the	world,	just	as	the	Omegas	had
done.	Most	important	were	the	spokespeople	who	became	the	public	faces	of	its
growing	business	empire.	Even	the	spokespeople	generally	lived	under	the
illusion	that	their	corporate	group	had	large	numbers	of	actual	people,	not

realizing	that	almost	everyone	with	whom	they	video-conferenced	for	their	job
interviews,	board	meetings,	etc.,	was	simulated	by	Prometheus.	Some	of	the
spokespeople	were	top	lawyers,	but	far	fewer	were	needed	than	under	the
Omega	plan,	because	almost	all	legal	documents	were	penned	by	Prometheus.
Prometheus’	breakout	opened	the	floodgates	that	had	prevented	information
from	flowing	into	the	world,	and	the	entire	internet	was	soon	awash	in
everything	from	articles	to	user	comments,	product	reviews,	patent	applications,
research	papers	and	YouTube	videos—all	authored	by	Prometheus,	who
dominated	the	global	conversation.
Where	breakout	paranoia	had	prevented	the	Omegas	from	releasing	highly
intelligent	robots,	Prometheus	rapidly	roboticized	the	world,	manufacturing
virtually	every	product	more	cheaply	than	humans	could.	Once	Prometheus	had
self-contained	nuclear-powered	robot	factories	in	uranium	mine	shafts	that
nobody	knew	existed,	even	the	staunchest	skeptics	of	an	AI	takeover	would	have

agreed	that	Prometheus	was	unstoppable—had	they	known.	Instead,	the	last	of
these	diehards	recanted	once	robots	started	settling	the	Solar	System.
—
The	scenarios	we’ve	explored	
so	far	show	what’s	wrong	with	many	of	the	myths	about
superintelligence	that	we	covered	earlier,	so	I	encourage	you	to	pause	briefly	to
go	back	and	review	the	misconception	summary	in	
figure	1.5
.	Prometheus
caused	problems	for	certain	people	not	because	it	was	necessarily	evil	or
conscious,	but	because	it	was	competent	and	didn’t	fully	share	their	goals.

—
figure	1.5
Despite	all	the	media	hype	about	a	robot	uprising,	Prometheus	wasn’t	a	robot—
rather,	its	power	came	from	its	intelligence.	We	saw	that	Prometheus	was	able	to
use	this	intelligence	to	control	humans	in	a	variety	of	ways,	and	that	people	who
didn’t	like	what	happened	weren’t	able	to	simply	switch	Prometheus	off.	Finally,
despite	frequent	claims	that	machines	can’t	have	goals,	we	saw	how	Prometheus
was	quite	goal-oriented—and	that	whatever	its	ultimate	goals	may	have	been,
they	led	to	the	subgoals	of	acquiring	resources	and	breaking	out.

We’ve	now	explored	a	range	of	intelligence	explosion	scenarios,	spanning	the
spectrum	from	ones	that	everyone	I	know	wants	to	avoid	to	ones	that	some	of
my	friends	view	optimistically.	Yet	all	these	scenarios	have	two	features	in
common:
1.
A	fast	takeoff:	the	transition	from	subhuman	to	vastly	superhuman
intelligence	occurs	in	a	matter	of	days,	not	decades.
2.
A	unipolar	outcome:	the	result	is	a	single	entity	controlling	Earth.
There	is	major	controversy	about	whether	these	two	features	are	likely	or
unlikely,	and	there	are	plenty	of	renowned	AI	researchers	and	other	thinkers	on
both	sides	of	the	debate.	To	me,	this	means	that	we	simply	don’t	know	yet,	and
need	to	keep	an	open	mind	and	consider	all	possibilities	for	now.	Let’s	therefore
devote	the	rest	of	this	chapter	to	exploring	scenarios	with	slower	takeoffs,
multipolar	outcomes,	cyborgs	and	uploads.
There	is	an	interesting	link	between	the	two	features,	as	Nick	Bostrom	and

others	have	highlighted:	a	fast	takeoff	can	facilitate	a	unipolar	outcome.	We	saw
above	how	a	rapid	takeoff	gave	the	Omegas	or	Prometheus	a	decisive	strategic
advantage	that	enabled	them	to	take	over	the	world	before	anyone	else	had	time
to	copy	their	technology	and	seriously	compete.	In	contrast,	if	takeoff	had
dragged	on	for	decades,	because	the	key	technological	breakthroughs	were
incremental	and	far	between,	then	other	companies	would	have	had	ample	time

to	catch	up,	and	it	would	have	been	much	harder	for	any	player	to	dominate.	If
competing	companies	also	had	software	that	could	perform	MTurk	tasks,	the	law
of	supply	and	demand	would	drive	the	prices	for	these	tasks	down	to	almost
nothing,	and	none	of	the	companies	would	earn	the	sort	of	windfall	profits	that
enabled	the	Omegas	to	gain	power.	The	same	applies	to	all	the	other	ways	in
which	the	Omegas	made	quick	money:	they	were	only	disruptively	profitable
because	they	held	a	monopoly	on	their	technology.	It’s	hard	to	double	your

money	daily	(or	even	annually)	in	a	competitive	market	where	your	competition
offers	products	similar	to	yours	for	almost	zero	cost.

Game	Theory	and	Power	Hierarchies
What’s	the	natural	state	of	life	in	our	cosmos:	unipolar	or	multipolar?	Is	power
concentrated	or	distributed?	After	the	first	13.8	billion	years,	the	answer	seems
to	be	“both”:	we	find	that	the	situation	is	distinctly	multipolar,	but	in	an
interestingly	hierarchical	fashion.	When	we	consider	all	information-processing
entities	out	there—cells,	people,	organizations,	nations,	etc.—we	find	that	they
both	collaborate	and	compete	at	a	hierarchy	of	levels.	Some	cells	have	found	it
advantageous	to	collaborate	to	such	an	extreme	extent	that	they’ve	merged	into
multicellular	organisms	such	as	people,	relinquishing	some	of	their	power	to	a
central	brain.	Some	people	have	found	it	advantageous	to	collaborate	in	groups
such	as	tribes,	companies	or	nations	where	they	in	turn	relinquish	some	power	to
a	chief,	boss	or	government.	Some	groups	may	in	turn	choose	to	relinquish	some
power	to	a	governing	body	to	improve	coordination,	with	examples	ranging	from

airline	alliances	to	the	European	Union.
The	branch	of	mathematics	known	as	
game	theory
	elegantly	explains	that
entities	have	an	incentive	to	cooperate	where	cooperation	is	a	so-called	
Nash
equilibrium:
	a	situation	where	any	party	would	be	worse	off	if	they	altered	their
strategy.	To	prevent	cheaters	from	ruining	the	successful	collaboration	of	a	large
group,	it	may	be	in	everyone’s	interest	to	relinquish	some	power	to	a	higher	level
in	the	hierarchy	that	can	punish	cheaters:	for	example,	people	may	collectively

Nash
benefit	from	granting	a	government	power	to	enforce	laws,	and	cells	in	your
body	may	collectively	benefit	from	giving	a	police	force	(immune	system)	the
power	to	kill	any	cell	that	acts	too	uncooperatively	(say	by	spewing	out	viruses
or	turning	cancerous).	For	a	hierarchy	to	remain	stable,	its	Nash	equilibrium
needs	to	hold	also	between	entities	at	different	levels:	for	example,	if	a
government	doesn’t	provide	enough	benefit	to	its	citizens	for	obeying	it,	they
may	change	their	strategy	and	overthrow	it.

In	a	complex	world,	there	is	a	diverse	abundance	of	possible	Nash	equilibria,
corresponding	to	different	types	of	hierarchies.	Some	hierarchies	are	more
authoritarian	than	others.	In	some,	entities	are	free	to	leave	(like	employees	in
most	corporate	hierarchies),	while	in	others	they’re	strongly	discouraged	from
leaving	(as	in	religious	cults)	or	unable	to	leave	(like	citizens	of	North	Korea,	or
cells	in	a	human	body).	Some	hierarchies	are	held	together	mainly	by	threats	and

fear,	others	mainly	by	benefits.	Some	hierarchies	allow	their	lower	parts	to

influence	the	higher-ups	by	democratic	voting,	while	others	allow	upward
influence	only	through	persuasion	or	the	passing	of	information.

How	Technology	Affects	Hierarchies
How	is	technology	changing	the	hierarchical	nature	of	our	world?	History
reveals	an	overall	trend	toward	ever	more	coordination	over	ever-larger
distances,	which	is	easy	to	understand:	new	transportation	technology	makes
coordination	more	valuable	(by	enabling	mutual	benefit	from	moving	materials
and	life	forms	over	larger	distances)	and	new	communication	technology	makes
coordination	easier.	When	cells	learned	to	signal	to	neighbors,	small
multicellular	organisms	became	possible,	adding	a	new	hierarchical	level.	When
evolution	invented	circulatory	systems	and	nervous	systems	for	transportation
and	communication,	large	animals	became	possible.	Further	improving
communication	by	inventing	language	allowed	humans	to	coordinate	well
enough	to	form	further	hierarchical	levels	such	as	villages,	and	additional
breakthroughs	in	communication,	transportation	and	other	technology	enabled
the	empires	of	antiquity.	Globalization	is	merely	the	latest	example	of	this	multi-

billion-year	trend	of	hierarchical	growth.
In	most	cases,	this	technology-driven	trend	has	made	large	entities	parts	of	an
even	grander	structure	while	retaining	much	of	their	autonomy	and	individuality,
although	commentators	have	argued	that	adaptation	of	entities	to	hierarchical	life
has	in	some	cases	reduced	their	diversity	and	made	them	more	like
indistinguishable	replaceable	parts.	Some	technologies,	such	as	surveillance,	can
give	higher	levels	in	the	hierarchy	more	power	over	their	subordinates,	while

other	technologies,	such	as	cryptography	and	online	access	to	free	press	and
education,	can	have	the	opposite	effect	and	empower	individuals.
Although	our	present	world	remains	stuck	in	a	multipolar	Nash	equilibrium,
with	competing	nations	and	multinational	corporations	at	the	top	level,
technology	is	now	advanced	enough	that	a	unipolar	world	would	probably	also
be	a	stable	Nash	equilibrium.	For	example,	imagine	a	parallel	universe	where
everyone	on	Earth	shares	the	same	language,	culture,	values	and	level	of

prosperity,	and	there	is	a	single	world	government	wherein	nations	function	like
states	in	a	federation	and	have	no	armies,	merely	police	enforcing	laws.	Our
present	level	of	technology	would	probably	suffice	to	successfully	coordinate
this	world—even	though	our	present	population	might	be	unable	or	unwilling	to
switch	to	this	alternative	equilibrium.
What	will	happen	to	the	hierarchical	structure	of	our	cosmos	if	we	add

superintelligent	AI	technology	to	this	mix?	Transportation	and	communication
technology	will	obviously	improve	dramatically,	so	a	natural	expectation	is	that
the	historical	trend	will	continue,	with	new	hierarchical	levels	coordinating	over
ever-larger	distances—perhaps	ultimately	encompassing	solar	systems,	galaxies,
superclusters	and	large	swaths	of	our	Universe,	as	we’ll	explore	in	chapter	6.	At
the	same	time,	the	most	fundamental	driver	of	decentralization	will	remain:	it’s
wasteful	to	coordinate	unnecessarily	over	large	distances.	Even	Stalin	didn’t	try
to	regulate	exactly	when	his	citizens	went	to	the	bathroom.	For	superintelligent
AI,	the	laws	of	physics	will	place	firm	upper	limits	on	transportation	and
communication	technology,	making	it	unlikely	that	the	highest	levels	of	the
hierarchy	would	be	able	to	micromanage	everything	that	happens	on	planetary
and	local	scales.	A	superintelligent	AI	in	the	Andromeda	galaxy	wouldn’t	be

able	to	give	you	useful	orders	for	your	day-to-day	decisions	given	that	you’d
need	to	wait	over	five	million	years	for	your	instructions	(that’s	the	round-trip
time	for	you	to	exchange	messages	traveling	at	the	speed	of	light).	In	the	same
way,	the	round-trip	travel	time	for	a	message	crossing	Earth	is	about	0.1	second
(about	the	timescale	on	which	we	humans	think),	so	an	Earth-sized	AI	brain
could	have	truly	global	thoughts	only	about	as	fast	as	a	human	one.	For	a	small
AI	performing	one	operation	each	billionth	of	a	second	(which	is	typical	of

today’s	computers),	0.1	second	would	feel	like	four	months	to	you,	so	for	it	to	be
micromanaged	by	a	planet-controlling	AI	would	be	as	inefficient	as	if	you	asked
permission	for	even	your	most	trivial	decisions	through	transatlantic	letters
delivered	by	Columbus-era	ships.
This	physics-imposed	speed	limit	on	information	transfer	therefore	poses	an
obvious	challenge	for	any	AI	wishing	to	take	over	our	world,	let	alone	our
Universe.	Before	Prometheus	broke	out,	it	put	very	careful	thought	into	how	to

avoid	mind	fragmentation,	so	that	its	many	AI	modules	running	on	different
computers	around	the	world	had	goals	and	incentives	to	coordinate	and	act	as	a
single	unified	entity.	Just	as	the	Omegas	faced	a	control	problem	when	they	tried
to	keep	Prometheus	in	check,	Prometheus	faced	a	self-control	problem	when	it
tried	to	ensure	that	none	of	its	parts	would	revolt.	We	clearly	don’t	yet	know	how
large	a	system	an	AI	will	be	able	to	control	directly,	or	indirectly	through	some

sort	of	collaborative	hierarchy—even	if	a	fast	takeoff	gave	it	a	decisive	strategic
advantage.
In	summary,	the	question	of	how	a	superintelligent	future	will	be	controlled	is
fascinatingly	complex,	and	we	clearly	don’t	know	the	answer	yet.	Some	argue
that	things	will	get	more	authoritarian;	others	claim	that	it	will	lead	to	greater

A	staple	of	science	fiction	is	that	humans	will	merge	with	machines,	either	by
technologically	enhancing	biological	bodies	into	cyborgs	(short	for	“cybernetic
organisms”)	or	by	uploading	our	minds	into	machines.	In	his	book	
The	Age	of
Em,
	economist	Robin	Hanson	gives	a	fascinating	survey	of	what	life	might	be
like	in	a	world	teeming	with	uploads	(also	known	as	
emulations,
	nicknamed
Ems
).	I	think	of	an	upload	as	the	extreme	end	of	the	cyborg	spectrum,	where	the
only	remaining	part	of	the	human	is	the	software.	Hollywood	cyborgs	range
from	visibly	mechanical,	such	as	the	Borg	from	
Star	Trek,
	to	androids	almost
indistinguishable	from	humans,	such	as	the	Terminators.	Fictional	uploads	range
in	intelligence	from	human-level	as	in	the	
Black	Mirror
	episode	“White
Christmas”	to	clearly	superhuman	as	in	
.
If	superintelligence	indeed	comes	about,	the	temptation	to	become	cyborgs	or
uploads	will	be	strong.	As	Hans	Moravec	puts	it	in	his	1988	classic	
Mind
Children:

Star	Trek,
.
Mind
Children:
	“Long	life	loses	much	of	its	point	if	we	are	fated	to	spend	it	staring
stupidly	at	ultra-intelligent	machines	as	they	try	to	describe	their	ever	more
spectacular	discoveries	in	baby-talk	that	we	can	understand.”	Indeed,	the
temptation	of	technological	enhancement	is	already	so	strong	that	many	humans
have	eyeglasses,	hearing	aids,	pacemakers	and	prosthetic	limbs,	as	well	as
medicinal	molecules	circulating	in	their	bloodstreams.	Some	teenagers	appear	to
be	permanently	attached	to	their	smartphones,	and	my	wife	teases	me	about	my
attachment	to	my	laptop.

One	of	today’s	most	prominent	cyborg	proponents	is	Ray	Kurzweil.	In	his
book	
The	Singularity	Is	Near,
	he	argues	that	the	natural	continuation	of	this	trend
is	using	nanobots,	intelligent	biofeedback	systems	and	other	technology	to
replace	first	our	digestive	and	endocrine	systems,	our	blood	and	our	hearts	by	the
early	2030s,	and	then	move	on	to	upgrading	our	skeletons,	skin,	brains	and	the
rest	of	our	bodies	during	the	next	two	decades.	He	guesses	that	we’re	likely	to

book	
keep	the	aesthetics	and	emotional	import	of	human	bodies,	but	will	redesign
them	to	rapidly	change	their	appearance	at	will,	both	physically	and	in	virtual
reality	(thanks	to	novel	brain-computer	interfaces).	Moravec	agrees	with
Kurzweil	that	cyborgization	would	go	far	beyond	merely	improving	our	DNA:
“a	genetically	engineered	superhuman	would	be	just	a	second-rate	kind	of	robot,

designed	under	the	handicap	that	its	construction	can	only	be	by	DNA-guided
protein	synthesis.”	Further,	he	argues	that	we’ll	do	even	better	by	eliminating	the
human	body	entirely	and	uploading	minds,	creating	a	whole-brain	emulation	in
software.	Such	an	upload	can	live	in	a	virtual	reality	or	be	embodied	in	a	robot
capable	of	walking,	flying,	swimming,	space-faring	or	anything	else	allowed	by
the	laws	of	physics,	unencumbered	by	such	everyday	concerns	as	death	or
limited	cognitive	resources.
Although	these	ideas	may	sound	like	science	fiction,	they	certainly	don’t
violate	any	known	laws	of	physics,	so	the	most	interesting	question	isn’t	whether
they	
can
	happen,	but	whether	they	
will
	happen	and,	if	so,	when.	Some	leading
thinkers	guess	that	the	first	human-level	AGI	will	be	an	upload,	and	that	this	is
how	the	path	toward	superintelligence	will	begin.
*
However,	I	think	it’s	fair	to	say	that	this	is	currently	a	minority	view	among

they	
can
will
*
AI	researchers	and	neuroscientists,	most	of	whom	guess	that	the	quickest	route
to	superintelligence	is	to	bypass	brain	emulation	and	engineer	it	in	some	other
way—after	which	we	may	or	may	not	remain	interested	in	brain	emulation.
After	all,	why	should	our	simplest	path	to	a	new	technology	be	the	one	that
evolution	came	up	with,	constrained	by	requirements	that	it	be	self-assembling,
self-repairing	and	self-reproducing?	Evolution	optimizes	strongly	for	energy
efficiency	because	of	limited	food	supply,	not	for	ease	of	construction	or

understanding	by	human	engineers.	My	wife,	Meia,	likes	to	point	out	that	the
aviation	industry	didn’t	start	with	mechanical	birds.	Indeed,	when	we	finally
figured	out	how	to	build	mechanical	birds	in	2011,
1
	more	than	a	century	after	the
Wright	brothers’	first	flight,	the	aviation	industry	showed	no	interest	in	switching
to	wing-flapping	mechanical-bird	travel,	even	though	it’s	more	energy	efficient
—because	our	simpler	earlier	solution	is	better	suited	to	our	travel	needs.
In	the	same	way,	I	suspect	that	there	are	simpler	ways	to	build	human-level

1
thinking	machines	than	the	solution	evolution	came	up	with,	and	even	if	we	one
day	manage	to	replicate	or	upload	brains,	we’ll	end	up	discovering	one	of	those
simpler	solutions	first.	It	will	probably	draw	more	than	the	twelve	watts	of
power	that	your	brain	uses,	but	its	engineers	won’t	be	as	obsessed	about	energy
efficiency	as	evolution	was—and	soon	enough,	they’ll	be	able	to	use	their
intelligent	machines	to	design	more	energy-efficient	ones.

The	short	answer	is	obviously	that	we	have	no	idea	what	will	happen	if	humanity
succeeds	in	building	human-level	AGI.	For	this	reason,	we’ve	spent	this	chapter
exploring	a	broad	spectrum	of	scenarios.	I’ve	attempted	to	be	quite	inclusive,
spanning	the	full	range	of	speculations	I’ve	seen	or	heard	discussed	by	AI
researchers	and	technologists:	fast	takeoff/slow	takeoff/no	takeoff,
humans/machines/cyborgs	in	control,	one/many	centers	of	power,	etc.	Some
people	have	told	me	that	they’re	sure	that	this	or	that	won’t	happen.	However,	I
think	it’s	wise	to	be	humble	at	this	stage	and	acknowledge	how	little	we	know,
because	for	each	scenario	discussed	above,	I	know	at	least	one	well-respected	AI
researcher	who	views	it	as	a	real	possibility.
As	time	passes	and	we	reach	certain	forks	in	the	road,	we’ll	start	to	answer
key	questions	and	narrow	down	the	options.	The	first	big	question	is	“Will	we
ever	create	human-level	AGI?”	The	premise	of	this	chapter	is	that	we	will,	but

there	are	AI	experts	who	think	it	will	never	happen,	at	least	not	for	hundreds	of
years.	Time	will	tell!	As	I	mentioned	earlier,	about	half	of	the	AI	experts	at	our
Puerto	Rico	conference	guessed	that	it	would	happen	by	2055.	At	a	follow-up
conference	we	organized	two	years	later,	this	had	dropped	to	2047.
Before	any	human-level	AGI	is	created,	we	may	start	getting	strong
indications	about	whether	this	milestone	is	likely	to	be	first	met	by	computer
engineering,	mind	uploading	or	some	unforeseen	novel	approach.	If	the

computer	engineering	approach	to	AI	that	currently	dominates	the	field	fails	to
deliver	AGI	for	centuries,	this	will	increase	the	chance	that	uploading	will	get
there	first,	as	happened	(rather	unrealistically)	in	the	movie	
Transcendence.
If	human-level	AGI	gets	more	imminent,	we’ll	be	able	to	make	more	educated
guesses	about	the	answer	to	the	next	key	question:	“Will	there	be	a	fast	takeoff,	a
slow	takeoff	or	no	takeoff?”	As	we	saw	above,	a	fast	takeoff	makes	world
takeover	easier,	while	a	slow	one	makes	an	outcome	with	many	competing

players	more	likely.	Nick	Bostrom	dissects	this	question	of	takeoff	speed	in	an
analysis	of	what	he	calls	
optimization	power
	and	
recalcitrance
,	which	are
basically	the	amount	of	quality	effort	to	make	AI	smarter	and	the	difficulty	of
making	progress,	respectively.	The	average	rate	of	progress	clearly	increases	if
more	optimization	power	is	brought	to	bear	on	the	task	and	decreases	if	more

recalcitrance	is	encountered.	He	makes	arguments	for	why	the	recalcitrance
might	either	increase	or	decrease	as	the	AGI	reaches	and	transcends	human
level,	so	keeping	both	options	on	the	table	is	a	safe	bet.	Turning	to	the
optimization	power,	however,	it’s	overwhelmingly	likely	that	it	will	grow	rapidly
as	the	AGI	transcends	human	level,	for	the	reasons	we	saw	in	the	Omega
scenario:	the	main	input	to	further	optimization	comes	not	from	people	but	from
the	machine	itself,	so	the	more	capable	it	gets,	the	faster	it	improves	(if
recalcitrance	stays	fairly	constant).
For	any	process	whose	power	grows	at	a	rate	proportional	to	its	current	power,
the	result	is	that	its	power	keeps	doubling	at	regular	intervals.	We	call	such
growth	
exponential
,	and	we	call	such	processes	
explosions
.	If	baby-making
power	grows	in	proportion	to	the	size	of	the	population,	we	can	get	a	population
explosion.	If	the	creation	of	neutrons	capable	of	fissioning	plutonium	grows	in

growth	
proportion	to	the	number	of	such	neutrons,	we	can	get	a	nuclear	explosion.	If
machine	intelligence	grows	at	a	rate	proportional	to	the	current	power,	we	can
get	an	intelligence	explosion.	All	such	explosions	are	characterized	by	the	time
they	take	to	double	their	power.	If	that	time	is	hours	or	days	for	an	intelligence
explosion,	as	in	the	Omega	scenario,	we	have	a	fast	takeoff	on	our	hands.
This	explosion	timescale	depends	crucially	on	whether	improving	the	AI

requires	merely	new	software	(which	can	be	created	in	a	matter	of	seconds,
minutes	or	hours)	or	new	hardware	(which	might	require	months	or	years).	In	the
Omega	scenario,	there	was	a	significant	
hardware	overhang,
	in	Bostrom’s
terminology:	the	Omegas	had	compensated	for	the	low	quality	of	their	original
software	by	vast	amounts	of	hardware,	which	meant	that	Prometheus	could
perform	a	large	number	of	quality	doublings	by	improving	its	software	alone.
There	was	also	a	major	
content	overhang
	in	the	form	of	much	of	the	internet’s

data;	Prometheus	1.0	was	still	not	smart	enough	to	make	use	of	most	of	it,	but
once	Prometheus’	intelligence	grew,	the	data	it	needed	for	further	learning	was
already	
available
	without	delay.
The	hardware	and	electricity	costs	of	running	the	AI	are	crucial	as	well,	since
we	won’t	get	an	intelligence	explosion	until	the	cost	of	doing	human-level	work
drops	below	human-level	hourly	wages.	Suppose,	for	example,	that	the	first
human-level	AGI	can	be	efficiently	run	on	the	Amazon	cloud	at	a	cost	of	$1

already	
available
million	per	hour	of	human-level	work	produced.	This	AI	would	have	great
novelty	value	and	undoubtedly	make	headlines,	but	it	wouldn’t	undergo
recursive	self-improvement,	because	it	would	be	much	cheaper	to	keep	using
humans	to	improve	it.	Suppose	that	these	humans	gradually	manage	to	cut	the

cost	to	$100,000/hour,	$10,000/hour,	$1,000/hour,	$100/hour,	$10/hour	and
finally	$1/hour.	By	the	time	the	cost	of	using	the	computer	to	reprogram	itself
finally	drops	far	below	the	cost	of	paying	human	programmers	to	do	the	same,
the	humans	can	be	laid	off	and	the	optimization	power	greatly	expanded	by
buying	cloud-computing	time.	This	produces	further	cost	cuts,	allowing	still
more	optimization	power,	and	the	intelligence	explosion	has	begun.
This	leaves	us	with	our	final	key	question:	“Who	or	what	will	control	the
intelligence	explosion	and	its	aftermath,	and	what	are	their/its	goals?”	We’ll
explore	possible	goals	and	outcomes	in	the	next	chapter	and	more	deeply	in
chapter	7.	To	sort	out	the	control	issue,	we	need	to	know	both	how	well	an	AI
can	be	controlled,	and	how	much	an	AI	can	control.
In	terms	of	what	will	ultimately	happen,	you’ll	currently	find	serious	thinkers
all	over	the	map:	some	contend	that	the	default	outcome	is	doom,	while	others

insist	that	an	awesome	outcome	is	virtually	guaranteed.	To	me,	however,	this
query	is	a	trick	question:	it’s	a	mistake	to	passively	ask	“what	will	happen,”	as	if
it	were	somehow	predestined!	If	a	technologically	superior	alien	civilization
arrived	tomorrow,	it	would	indeed	be	appropriate	to	wonder	“what	will	happen”
as	their	spaceships	approached,	because	their	power	would	probably	be	so	far
beyond	ours	that	we’d	have	no	influence	over	the	outcome.	If	a	technologically
superior	AI-fueled	civilization	arrives	because	we	built	it,	on	the	other	hand,	we

humans	have	great	influence	over	the	outcome—influence	that	we	exerted	when
we	created	the	AI.	So	we	should	instead	ask:	“What	
should
	happen?	What	future
do	we	want?”	In	the	next	chapter,	we’ll	explore	a	wide	spectrum	of	possible
aftermaths	of	the	current	race	toward	AGI,	and	I’m	quite	curious	how	you’d	rank
them	from	best	to	worst.	Only	once	we’ve	thought	hard	about	what	sort	of	future
we	want	will	we	be	able	to	begin	steering	a	course	toward	a	desirable	future.	If
we	don’t	know	what	we	want,	we’re	unlikely	to	get	it.

•
If	we	one	day	succeed	in	building	human-level	AGI,	this	may	trigger	an	intelligence
explosion,	leaving	us	far	behind.
•
If	a	group	of	humans	manage	to	control	an	intelligence	explosion,	they	may	be	able
to	take	over	the	world	in	a	matter	of	years.
•
If	humans	fail	to	control	an	intelligence	explosion,	the	AI	itself	may	take	over	the
world	even	faster.
•
Whereas	a	rapid	intelligence	explosion	is	likely	to	lead	to	a	single	world	power,	a
slow	one	dragging	on	for	years	or	decades	may	be	more	likely	to	lead	to	a	multipolar
scenario	with	a	balance	of	power	between	a	large	number	of	rather	independent
entities.
•
The	history	of	life	shows	it	self-organizing	into	an	ever	more	complex	hierarchy
shaped	by	collaboration,	competition	and	control.	Superintelligence	is	likely	to
enable	coordination	on	ever-larger	cosmic	scales,	but	it’s	unclear	whether	it	will
ultimately	lead	to	more	totalitarian	top-down	control	or	more	individual
empowerment.
•

entities.
•
•
Cyborgs	and	uploads	are	plausible,	but	arguably	not	the	fastest	route	to	advanced
machine	intelligence.
•
The	climax	of	our	current	race	toward	AI	may	be	either	the	best	or	the	worst	thing
ever	to	happen	to	humanity,	with	a	fascinating	spectrum	of	possible	outcomes	that
we’ll	explore	in	the	next	chapter.
•
We	need	to	start	thinking	hard	about	which	outcome	we	prefer	and	how	to	steer	in
that	direction,	because	if	we	don’t	know	what	we	want,	we’re	unlikely	to	get	it.
*

•
•
•
*
	
As	Bostrom	has	explained,	the	ability	to	simulate	a	leading	human	AI	developer	at	a	much	lower	cost	than
his/her	hourly	salary	would	enable	an	AI	company	to	scale	up	their	workforce	dramatically,	amassing
great	wealth	and	recursively	accelerating	their	progress	in	building	better	computers	and	ultimately
smarter	minds.

Chapter	5
It	is	easy	to	imagine	human	thought	freed	from	bondage	to	a	mortal	body—belief	in
	
an
afterlife	is	common.	But	it	is	not	necessary	to	adopt	a	mystical	or	religious	stance
	
to
accept	this	possibility.	Computers	provide	a	model	for	even	the	most	ardent
	
mechanist.
Hans	Moravec,	
Mind	Children
I,	for	one,	welcome	our	new	computer	overlords.
Ken	Jennings,	upon	his	
Jeopardy!
	loss	to	IBM’s	Watson
Humans	will	become	as	irrelevant	as	cockroaches.
Marshall	Brain
The	race	toward	AGI	is	on,	and	we	have	no	idea	how	it	will	unfold.	But	that
shouldn’t	stop	us	from	thinking	about	what	we	want	the	aftermath	to	be	like,
because	what	we	want	will	affect	the	outcome.	What	do	you	personally	prefer,
and	why?
1.
Do	you	want	there	to	be	superintelligence?
2.
Do	you	want	humans	to	still	exist,	be	replaced,	cyborgized	and/or
uploaded/simulated?
3.
Do	you	want	humans	or	machines	in	control?
4.
Do	you	want	AIs	to	be	conscious	or	not?
5.

and	why?
1.
2.
3.
4.
5.
Do	you	want	to	maximize	positive	experiences,	minimize	suffering	or

leave	this	to	sort	itself	out?
6.
Do	you	want	life	spreading	into	the	cosmos?
7.
Do	you	want	a	civilization	striving	toward	a	greater	purpose	that	you
sympathize	with,	or	are	you	OK	with	future	life	forms	that	appear	content
even	if	you	view	their	goals	as	pointlessly	banal?
To	help	fuel	such	contemplation	and	conversation,	let’s	explore	the	broad
range	of	scenarios	summarized	in	
table	5.1
.	This	obviously	isn’t	an	exhaustive
list,	but	I’ve	chosen	it	to	span	the	spectrum	of	possibilities.	We	clearly	don’t
want	to	end	up	in	the	wrong	endgame	because	of	poor	planning.	I	recommend
jotting	down	your	tentative	answers	to	questions	1–7	and	then	revisiting	them
after	reading	this	chapter	to	see	if	you’ve	changed	your	mind!	You	can	do	this	at
,	where	you	can	also	compare	notes	and	discuss	with	other
readers.
AI	Aftermath	Scenarios
Libertarian
utopia
Humans,	cyborgs,	uploads	and	superintelligences
coexist	peacefully	thanks	to	property	rights.
Benevolent
dictator

readers.
utopia
dictator
Everybody	knows	that	the	AI	runs	society	and
enforces	strict	rules,	but	most	people	view	this	as	a
good	thing.
Egalitarian
utopia
Humans,	cyborgs	and	uploads	coexist	peacefully
thanks	to	property	abolition	and	guaranteed	income.
A	superintelligent	AI	is	created	with	the	goal	of
interfering	as	little	as	necessary	to	prevent	the
creation	of	another	superintelligence.	As	a	result,
helper	robots	with	slightly	subhuman	intelligence
abound,	and	human-machine	cyborgs	exist,	but
technological	progress	is	forever	stymied.
Protector
god

utopia
Protector
god
Essentially	omniscient	and	omnipotent	AI
maximizes	human	happiness	by	intervening	only	in
ways	that	preserve	our	feeling	of	control	of	our	own
destiny	and	hides	well	enough	that	many	humans
even	doubt	the	AI’s	existence.
Enslaved
A	superintelligent	AI	is	confined	by	humans,	who

god
use	it	to	produce	unimaginable	technology	and
wealth	that	can	be	used	for	good	or	bad	depending
on	the	human	controllers.
AI	takes	control,	decides	that	humans	are	a
threat/nuisance/waste	of	resources,	and	gets	rid	of
us	by	a	method	that	we	don’t	even	understand.
AIs	replace	humans,	but	give	us	a	graceful	exit,
making	us	view	them	as	our	worthy	descendants,
much	as	parents	feel	happy	and	proud	to	have	a
child	who’s	smarter	than	them,	who	learns	from
them	and	then	accomplishes	what	they	could	only
dream	of—even	if	they	can’t	live	to	see	it	all.
Zookeeper
An	omnipotent	AI	keeps	some	humans	around,	who
feel	treated	like	zoo	animals	and	lament	their	fate.
1984
Technological	progress	toward	superintelligence	is
permanently	curtailed	not	by	an	AI	but	by	a	human-
led	Orwellian	surveillance	state	where	certain	kinds
of	AI	research	are	banned.
Reversion
prevented	by	reverting	to	a	pre-technological
society	in	the	style	of	the	Amish.
Self-

Zookeeper
1984
Reversion
Self-
destruction
Superintelligence	is	never	created	because	humanity
drives	itself	extinct	by	other	means	(say	nuclear
and/or	biotech	mayhem	fueled	by	climate	crisis).
Table	5.1:	Summary	of	AI	Aftermath	Scenarios

Let’s	begin	with	a	scenario	where	humans	peacefully	coexist	with	technology
and	in	some	cases	merge	with	it,	as	imagined	by	many	futurists	and	science
fiction	writers	alike:
Life	on	Earth	(and	beyond—more	on	that	in	the	next	chapter)	is	more	diverse
than	ever	before.	If	you	looked	at	satellite	footage	of	Earth,	you’d	easily	be	able
to	tell	apart	the	machine	zones,	mixed	zones	and	human-only	zones.	The
machine	zones	are	enormous	robot-controlled	factories	and	computing	facilities
devoid	of	biological	life,	aiming	to	put	every	atom	to	its	most	efficient	use.
Although	the	machine	zones	look	monotonous	and	drab	from	the	outside,	they’re
spectacularly	alive	on	the	inside,	with	amazing	experiences	occurring	in	virtual
worlds	while	colossal	computations	unlock	secrets	of	our	Universe	and	develop
transformative	technologies.	Earth	hosts	many	superintelligent	minds	that
compete	and	collaborate,	and	they	all	inhabit	the	machine	zones.

The	denizens	of	the	mixed	zones	are	a	wild	and	idiosyncratic	mix	of
computers,	robots,	humans	and	hybrids	of	all	three.	As	envisioned	by	futurists
such	as	Hans	Moravec	and	Ray	Kurzweil,	many	of	the	humans	have
technologically	upgraded	their	bodies	to	cyborgs	in	various	degrees,	and	some
have	uploaded	their	minds	into	new	hardware,	blurring	the	distinction	between
man	and	machine.	Most	intelligent	beings	lack	a	permanent	physical	form.
Instead,	they	exist	as	software	capable	of	instantly	moving	between	computers

and	manifesting	themselves	in	the	physical	world	through	robotic	bodies.
Because	these	minds	can	readily	duplicate	themselves	or	merge,	the	“population
size”	keeps	changing.	Being	unfettered	from	their	physical	substrate	gives	such
beings	a	rather	different	outlook	on	life:	they	feel	less	individualistic	because
they	can	trivially	share	knowledge	and	experience	modules	with	others,	and	they
feel	subjectively	immortal	because	they	can	readily	make	backup	copies	of
themselves.	In	a	sense,	the	central	entities	of	life	aren’t	minds,	but	experiences:

exceptionally	amazing	experiences	live	on	because	they	get	continually	copied
and	re-enjoyed	by	other	minds,	while	uninteresting	experiences	get	deleted	by
their	owners	to	free	up	storage	space	for	better	ones.
Although	the	majority	of	interactions	occur	in	virtual	environments	for
convenience	and	speed,	many	minds	still	enjoy	interactions	and	activities	using

physical	bodies	as	well.	For	example,	uploaded	versions	of	Hans	Moravec,	Ray
Kurzweil	and	Larry	Page	have	a	tradition	of	taking	turns	creating	virtual	realities
and	then	exploring	them	together,	but	once	in	a	while,	they	also	enjoy	flying
together	in	the	real	world,	embodied	in	avian	winged	robots.	Some	of	the	robots
that	roam	the	streets,	skies	and	lakes	of	the	mixed	zones	are	similarly	controlled
by	uploaded	and	augmented	humans,	who	choose	to	embody	themselves	in	the
mixed	zones	because	they	enjoy	being	around	humans	and	each	other.
In	the	human-only	zones,	in	contrast,	machines	with	human-level	general
intelligence	or	above	are	banned,	as	are	technologically	enhanced	biological
organisms.	Here,	life	isn’t	dramatically	different	from	today,	except	that	it’s
more	affluent	and	convenient:	poverty	has	been	mostly	eliminated,	and	cures	are
available	for	most	of	today’s	diseases.	The	small	fraction	of	humans	who	have
opted	to	live	in	these	zones	effectively	exist	on	a	lower	and	more	limited	plane

of	awareness	from	everyone	else,	and	have	limited	understanding	of	what	their
more	intelligent	fellow	minds	are	doing	in	the	other	zones.	However,	many	of
them	are	quite	happy	with	their	lives.

AI	Economics
The	vast	majority	of	all	computations	take	place	in	the	machine	zones,	which	are
mostly	owned	by	the	many	competing	superintelligent	AIs	that	live	there.	By
virtue	of	their	superior	intelligence	and	technology,	no	other	entities	can
challenge	their	power.	These	AIs	have	agreed	to	cooperate	and	coordinate	with
each	other	under	a	libertarian	governance	system	that	has	no	rules	except
protection	of	private	property.	These	property	rights	extend	to	all	intelligent
entities,	including	humans,	and	explain	how	the	human-only	zones	came	to	exist.
Early	on,	groups	of	humans	banded	together	and	decided	that,	in	their	zones,	it
was	forbidden	to	sell	property	to	non-humans.
Because	of	their	technology,	the	superintelligent	AIs	have	ended	up	richer
than	these	humans	by	a	factor	much	larger	than	that	by	which	Bill	Gates	is	richer
than	a	homeless	beggar.	However,	people	in	the	human-only	zones	are	still
materially	better	off	than	most	people	today:	their	economy	is	rather	decoupled

from	that	of	the	machines,	so	the	presence	of	the	machines	elsewhere	has	little
effect	on	them	except	for	the	occasional	useful	technologies	that	they	can
understand	and	reproduce	for	themselves—much	as	the	Amish	and	various
technology-relinquishing	native	tribes	today	have	standards	of	living	at	least	as
good	as	they	had	in	old	times.	It	doesn’t	matter	that	the	humans	have	nothing	to
sell	that	the	machines	need,	since	the	machines	need	nothing	in	return.
In	the	mixed	sectors,	the	wealth	difference	between	AIs	and	humans	is	more

noticeable,	resulting	in	land	(the	only	human-owned	product	that	the	machines
want	to	buy)	being	astronomically	expensive	compared	to	other	products.	Most
humans	who	owned	land	therefore	ended	up	selling	a	small	fraction	of	it	to	AIs
in	return	for	guaranteed	basic	income	for	them	and	their	offspring/uploads	in
perpetuity.	This	liberated	them	from	the	need	to	work,	and	freed	them	up	to
enjoy	the	amazing	abundance	of	cheap	machine-produced	goods	and	services,	in
both	physical	and	virtual	reality.	As	far	as	the	machines	are	concerned,	the	mixed

zones	are	mainly	for	play	rather	than	for	work.

Why	This	May	Never	Happen
Before	getting	too	excited	about	adventures	we	may	have	as	cyborgs	or	uploads,
let’s	consider	some	reasons	why	this	scenario	might	never	happen.	First	of	all,
there	are	two	possible	routes	to	enhanced	humans	(cyborgs	and	uploads):
1.
We	figure	out	how	to	create	them	ourselves.
2.
We	build	superintelligent	machines	that	figure	it	out	for	us.
If	route	1	comes	through	first,	it	could	naturally	lead	to	a	world	teeming	with
cyborgs	and	uploads.	However,	as	we	discussed	in	the	last	chapter,	most	AI
researchers	think	that	the	opposite	is	more	likely,	with	enhanced	or	digital	brains
being	more	difficult	to	build	than	clean-slate	superhuman	AGIs—just	as
mechanical	birds	turned	out	to	be	harder	to	build	than	airplanes.	After	strong
machine	AI	is	built,	it’s	not	obvious	that	cyborgs	or	uploads	will	ever	be	made.	If
the	Neanderthals	had	had	another	100,000	years	to	evolve	and	get	smarter,	things
might	have	turned	out	great	for	them—but	
Homo	sapiens
	never	gave	them	that
much	time.

much	time.
Second,	even	if	this	scenario	with	cyborgs	and	uploads	did	come	about,	it’s
not	clear	that	it	would	be	stable	and	last.	Why	should	the	power	balance	between
multiple	superintelligences	remain	stable	for	millennia,	rather	than	the	AIs
merging	or	the	smartest	one	taking	over?	Moreover,	why	should	the	machines
choose	to	respect	human	property	rights	and	keep	humans	around,	given	that
they	don’t	need	humans	for	anything	and	can	do	all	human	work	better	and
cheaper	themselves?	Ray	Kurzweil	speculates	that	natural	and	enhanced	humans

will	be	protected	from	extermination	because	“humans	are	respected	by	AIs	for
giving	rise	to	the	machines.”
1
	However,	as	we’ll	discuss	in	chapter	7,	we	must
not	fall	into	the	trap	of	anthropomorphizing	AIs	and	assume	that	they	have
human-like	emotions	of	gratitude.	Indeed,	though	we	humans	are	imbued	with	a
propensity	toward	gratitude,	we	don’t	show	enough	gratitude	to	our	intellectual
creator	(our	DNA)	to	abstain	from	thwarting	its	goals	by	using	birth	control.
Even	if	we	buy	the	assumption	that	the	AIs	will	opt	to	respect	human	property

1
rights,	they	can	gradually	get	much	of	our	land	in	other	ways,	by	using	some	of
their	superintelligent	persuasion	powers	that	we	explored	in	the	last	chapter	to
persuade	humans	to	sell	some	land	for	a	life	in	luxury.	In	human-only	sectors,

they	could	entice	humans	to	launch	political	campaigns	for	allowing	land	sales.
After	all,	even	die-hard	bio-Luddites	may	want	to	sell	some	land	to	save	the	life
of	an	ill	child	or	to	gain	immortality.	If	the	humans	are	educated,	entertained	and
busy,	falling	birthrates	may	even	shrink	their	population	sizes	without	machine
meddling,	as	is	currently	happening	in	Japan	and	Germany.	This	could	drive
humans	extinct	in	just	a	few	millennia.

Downsides
For	some	of	their	most	ardent	supporters,	cyborgs	and	uploads	hold	a	promise	of
techno-bliss	and	life	extension	for	all.	Indeed,	the	prospect	of	getting	uploaded	in
the	future	has	motivated	over	a	hundred	people	to	have	their	brains
posthumously	frozen	by	the	Arizona-based	company	Alcor.	If	this	technology
arrives,	however,	it’s	far	from	clear	that	it	will	be	available	to	everybody.	Many
of	the	very	wealthiest	would	presumably	use	it,	but	who	else?	Even	if	the
technology	got	cheaper,	where	would	the	line	be	drawn?	Would	the	severely
brain-damaged	be	uploaded?	Would	we	upload	every	gorilla?	Every	ant?	Every
plant?	Every	bacterium?	Would	the	future	civilization	act	like	obsessive-
compulsive	hoarders	and	try	to	upload	everything,	or	merely	a	few	interesting
examples	of	each	species	in	the	spirit	of	Noah’s	Ark?	Perhaps	only	a	few
representative	examples	of	each	type	of	human?	To	the	vastly	more	intelligent
entities	that	would	exist	at	that	time,	an	uploaded	human	may	seem	about	as

interesting	as	a	simulated	mouse	or	snail	would	seem	to	us.	Although	we
currently	have	the	technical	capability	to	reanimate	old	spreadsheet	programs
from	the	1980s	in	a	DOS	emulator,	most	of	us	don’t	find	this	interesting	enough
to	actually	do	it.
Many	people	may	dislike	this	libertarian-utopia	scenario	because	it	allows
preventable	suffering.	Since	the	only	sacred	principle	is	property	rights,	nothing
prevents	the	sort	of	suffering	that	abounds	in	today’s	world	from	continuing	in
the	human	and	mixed	zones.	While	some	people	thrive,	others	may	end	up	living

in	squalor	and	indentured	servitude,	or	suffer	from	violence,	fear,	repression	or
depression.	For	example,	Marshall	Brain’s	2003	novel	
Manna
	describes	how	AI
progress	in	a	libertarian	economic	system	makes	most	Americans	unemployable
and	condemned	to	live	out	the	rest	of	their	lives	in	drab	and	dreary	robot-
operated	social-welfare	housing	projects.	Much	like	farm	animals,	they’re	kept
fed,	healthy	and	safe	in	cramped	conditions	where	the	rich	never	need	to	see

Manna
them.	Birth	control	medication	in	the	water	ensures	that	they	don’t	have
children,	so	most	of	the	population	gets	phased	out	to	leave	the	remaining	rich
with	larger	shares	of	the	robot-produced	wealth.
In	the	libertarian-utopia	scenario,	suffering	need	not	be	limited	to	humans.	If
some	machines	are	imbued	with	conscious	emotional	experiences,	then	they	too
can	suffer.	For	example,	a	vindictive	psychopath	could	legally	take	an	uploaded

copy	of	his	enemy	and	subject	it	to	the	most	horrendous	torture	in	a	virtual
world,	creating	pain	of	intensity	and	duration	far	beyond	what’s	biologically
possible	in	the	real	world.

Let’s	now	explore	a	scenario	where	all	these	forms	of	suffering	are	absent
because	a	single	benevolent	superintelligence	runs	the	world	and	enforces	strict
rules	designed	to	maximize	its	model	of	human	happiness.	This	is	one	possible
outcome	of	the	first	Omega	scenario	from	the	previous	chapter,	where	they
relinquish	control	to	Prometheus	after	figuring	out	how	to	make	it	want	a
flourishing	human	society.
Thanks	to	amazing	technologies	developed	by	the	dictator	AI,	humanity	is
free	from	poverty,	disease	and	other	low-tech	problems,	and	all	humans	enjoy	a
life	of	luxurious	leisure.	They	have	all	their	basic	needs	taken	care	of,	while	AI-
controlled	machines	produce	all	necessary	goods	and	services.	Crime	is
practically	eliminated,	because	the	dictator	AI	is	essentially	omniscient	and
efficiently	punishes	anyone	disobeying	the	rules.	Everybody	wears	the	security
bracelet	from	the	last	chapter	(or	a	more	convenient	implanted	version),	capable

of	real-time	surveillance,	punishment,	sedation	and	execution.	Everybody	knows
that	they	live	in	an	AI	dictatorship	with	extreme	surveillance	and	policing,	but
most	people	view	this	as	a	good	thing.
The	superintelligent	AI	dictator	has	as	its	goal	to	figure	out	what	human
utopia	looks	like	given	the	evolved	preferences	encoded	in	our	genes,	and	to
implement	it.	By	clever	foresight	from	the	humans	who	brought	the	AI	into
existence,	it	doesn’t	simply	try	to	maximize	our	self-reported	happiness,	say	by

putting	everyone	on	intravenous	morphine	drip.	Instead,	the	AI	uses	quite	a
subtle	and	complex	definition	of	human	flourishing,	and	has	turned	Earth	into	a
highly	enriched	zoo	environment	that’s	really	fun	for	humans	to	live	in.	As	a
result,	most	people	find	their	lives	highly	fulfilling	and	meaningful.

The	Sector	System
Valuing	diversity,	and	recognizing	that	different	people	have	different
preferences,	the	AI	has	divided	Earth	into	different	sectors	for	people	to	choose
between,	to	enjoy	the	company	of	kindred	spirits.	Here	are	some	examples:
•
Knowledge	sector:	Here	the	AI	provides	optimized	education,	including
immersive	virtual-reality	experiences,	enabling	you	to	learn	all	you’re
capable	of	about	any	topics	of	your	choice.	Optionally,	you	can	choose
not	to	be	told	certain	beautiful	insights,	but	to	be	led	close	and	then	have
the	joy	of	rediscovering	them	for	yourself.
•
Art	sector:	Here	opportunities	abound	to	enjoy,	create	and	share	music,
art,	literature	and	other	forms	of	creative	expression.
•
Hedonistic	sector:	Locals	refer	to	it	as	the	party	sector,	and	it’s	second	to
none	for	those	yearning	for	delectable	cuisine,	passion,	intimacy	or	just
wild	fun.
•
Pious	sector:	There	are	many	of	these,	corresponding	to	different
religions,	whose	rules	are	strictly	enforced.
•

•
•
wild	fun.
•
•
Wildlife	sector:	Whether	you’re	looking	for	beautiful	beaches,	lovely
lakes,	magnificent	mountains	or	fantastic	fjords,	here	they	are.
•
Traditional	sector:	Here	you	can	grow	your	own	food	and	live	off	the
land	as	in	yesteryear—but	without	worrying	about	famine	or	disease.
•
Gaming	sector:	If	you	like	computer	games,	the	AI	has	created	truly
mind-blowing	options	for	you.
•
Virtual	sector:	If	you	want	a	vacation	from	your	physical	body,	the	AI
will	keep	it	hydrated,	fed,	exercised	and	clean	while	you	explore	virtual
words	through	neural	implants.
•

•
•
•
•
Prison	sector:	If	you	break	rules,	you’ll	end	up	here	for	retraining	unless
you	get	the	instant	death	penalty.
In	addition	to	these	“traditionally”	themed	sectors,	there	are	others	with
modern	themes	that	today’s	humans	wouldn’t	even	understand.	People	are

initially	free	to	move	between	sectors	whenever	they	want,	which	takes	very
little	time	thanks	to	the	AI’s	hypersonic	transportation	system.	For	example,
after	spending	an	intense	week	in	the	knowledge	sector	learning	about	the
ultimate	laws	of	physics	that	the	AI	has	discovered,	you	might	decide	to	cut
loose	in	the	hedonistic	sector	over	the	weekend	and	then	relax	for	a	few	days	at	a
beach	resort	in	the	wildlife	sector.
The	AI	enforces	two	tiers	of	rules:	universal	and	local.	Universal	rules	apply
in	all	sectors,	for	example	a	ban	on	harming	other	people,	making	weapons	or
trying	to	create	a	rival	superintelligence.	Individual	sectors	have	additional	local
rules	on	top	of	this,	encoding	certain	moral	values.	The	sector	system	therefore
helps	deal	with	values	that	don’t	mesh.	The	largest	number	of	local	rules	apply
in	the	prison	sector	and	some	of	the	religious	sectors,	while	there’s	a	Libertarian
Sector	whose	denizens	pride	themselves	on	having	no	local	rules	whatsoever.	All

punishments,	even	local	ones,	are	carried	out	by	the	AI,	since	a	human	punishing
another	human	would	violate	the	universal	no-harm	rule.	If	you	violate	a	local
rule,	the	AI	gives	you	the	choice	(unless	you’re	in	the	prison	sector)	of	accepting
the	prescribed	punishment	or	banishment	from	that	sector	forever.	For	example,
if	two	women	get	romantically	involved	in	a	sector	where	homosexuality	is
punished	by	a	prison	sentence	(as	it	is	in	many	countries	today),	the	AI	will	let

them	choose	between	going	to	jail	or	permanently	leaving	that	sector,	never
again	meeting	their	old	friends	(unless	they	leave	too).
Regardless	of	what	sector	they’re	born	in,	all	children	get	a	minimum	basic
education	from	the	AI,	which	includes	knowledge	about	humanity	as	a	whole
and	the	fact	that	they’re	free	to	visit	and	move	to	other	sectors	if	they	so	choose.
The	AI	designed	the	large	number	of	different	sectors	partly	because	it	was
created	to	value	the	human	diversity	that	exists	today.	But	each	sector	is	a

happier	place	than	today’s	technology	would	allow,	because	the	AI	has
eliminated	all	traditional	problems,	including	poverty	and	crime.	For	example,
people	in	the	hedonistic	sector	need	not	worry	about	sexually	transmitted
diseases	(they’ve	been	eradicated),	hangovers	or	addiction	(the	AI	has	developed
perfect	recreational	drugs	with	no	negative	side	effects).	Indeed,	nobody	in	any
sector	need	worry	about	any	disease,	because	the	AI	is	able	to	repair	human
bodies	with	nanotechnology.	Residents	of	many	sectors	get	to	enjoy	high-tech

architecture	that	makes	typical	sci-fi	visions	pale	in	comparison.
In	summary,	while	the	libertarian-utopia	and	benevolent-dictator	scenarios
both	involve	extreme	AI-fueled	technology	and	wealth,	they	differ	in	terms	of

who’s	in	charge	and	their	goals.	In	the	libertarian	utopia,	those	with	technology
and	property	decide	what	to	do	with	it,	while	in	the	present	scenario,	the	dictator
AI	has	unlimited	power	and	sets	the	ultimate	goal:	turning	Earth	into	an	all-
inclusive	pleasure	cruise	themed	in	accordance	with	people’s	preferences.	Since
the	AI	lets	people	choose	between	many	alternate	paths	to	happiness	and	takes
care	of	their	material	needs,	this	means	that	if	someone	suffers,	it’s	out	of	their
own	free	choice.

Downsides
Although	the	benevolent	dictatorship	teems	with	positive	experiences	and	is
rather	free	from	suffering,	many	people	nonetheless	feel	that	things	could	be
better.	First	of	all,	some	people	wish	that	humans	had	more	freedom	in	shaping
their	society	and	their	destiny,	but	they	keep	these	wishes	to	themselves	because
they	know	that	it	would	be	suicidal	to	challenge	the	overwhelming	power	of	the
machine	that	rules	them	all.	Some	groups	want	the	freedom	to	have	as	many
children	as	they	want,	and	resent	the	AI’s	insistence	on	sustainability	through
population	control.	Gun	enthusiasts	abhor	the	ban	on	building	and	using
weapons,	and	some	scientists	dislike	the	ban	on	building	their	own
superintelligence.	Many	people	feel	moral	outrage	over	what	goes	on	in	other
sectors,	worry	that	their	children	will	choose	to	move	there,	and	yearn	for	the
freedom	to	impose	their	own	moral	code	everywhere.
Over	time,	ever	more	people	choose	to	move	to	those	sectors	where	the	AI

gives	them	essentially	any	experiences	they	want.	In	contrast	to	traditional
visions	of	heaven	where	you	get	what	you	deserve,	this	is	in	the	spirit	of	“New
Heaven”	in	Julian	Barnes’	1989	novel	
History	of	the	World	in	10½	
Chapters
(and	also	the	1960	
Twilight	Zone
	episode	“A	Nice	Place	to	Visit”),	where	you	get
what	you	desire.	Paradoxically,	many	people	end	up	lamenting	always	getting
what	they	want.	In	Barnes’	story,	the	protagonist	spends	eons	indulging	his

Chapters
desires,	from	gluttony	and	golf	to	sex	with	celebrities,	but	eventually	succumbs
to	ennui	and	requests	annihilation.	Many	people	in	the	benevolent	dictatorship
meet	a	similar	fate,	with	lives	that	feel	pleasant	but	ultimately	meaningless.
Although	people	can	create	artificial	challenges,	from	scientific	rediscovery	to
rock	climbing,	everyone	knows	that	there	is	no	true	challenge,	merely
entertainment.	There’s	no	real	point	in	humans	trying	to	do	science	or	figure
other	things	out,	because	the	AI	already	has.	There’s	no	real	point	in	humans

trying	to	create	something	to	improve	their	lives,	because	they’ll	readily	get	it
from	the	AI	if	they	simply	ask.

As	a	counterpoint	to	this	challenge-free	dictatorship,	let’s	now	explore	a	scenario
where	there	is	no	superintelligent	AI,	and	humans	are	the	masters	of	their	own
destiny.	This	is	the	“fourth	generation	civilization”	described	in	Marshall	Brain’s
2003	novel	
Manna
.	It’s	the	economic	antithesis	of	the	libertarian	utopia	in	the
sense	that	humans,	cyborgs	and	uploads	coexist	peacefully	not	because	of
property	rights,	but	because	of	property	abolition	and	guaranteed	income.

Life	Without	Property
A	core	idea	is	borrowed	from	the	open-source	software	movement:	if	software	is
free	to	copy,	then	everyone	can	use	as	much	of	it	as	they	need	and	issues	of
ownership	and	property	become	moot.
*1
	According	to	the	law	of	supply	and
demand,	cost	reflects	scarcity,	so	if	supply	is	essentially	unlimited,	the	price
becomes	negligible.	In	this	spirit,	all	intellectual	property	rights	are	abolished:
there	are	no	patents,	copyrights	or	trademarked	designs—people	simply	share
their	good	ideas,	and	everyone	is	free	to	use	them.
Thanks	to	advanced	robotics,	this	same	no-property	idea	applies	not	only	to
information	products	such	as	software,	books,	movies	and	designs,	but	also	to
material	products	such	as	houses,	cars,	clothing	and	computers.	All	these
products	are	simply	atoms	rearranged	in	particular	ways,	and	there’s	no	shortage
of	atoms,	so	whenever	a	person	wants	a	particular	product,	a	network	of	robots
will	use	one	of	the	available	open-source	designs	to	build	it	for	them	for	free.

Care	is	taken	to	use	easily	recyclable	materials,	so	that	whenever	someone	gets
tired	of	an	object	they’ve	used,	robots	can	rearrange	its	atoms	into	something
someone	else	wants.	In	this	way,	all	resources	are	recycled,	so	none	are
permanently	destroyed.	These	robots	also	build	and	maintain	enough	renewable
power-generation	plants	(solar,	wind,	etc.)	that	energy	is	also	essentially	free.
To	avoid	obsessive	hoarders	requesting	so	many	products	or	so	much	land	that
others	are	left	needy,	each	person	receives	a	basic	monthly	income	from	the

government,	which	they	can	spend	as	they	wish	on	products	and	renting	places
to	live.	There’s	essentially	no	incentive	for	anyone	to	try	to	earn	more	money,
because	the	basic	income	is	high	enough	to	meet	any	reasonable	needs.	It	would
also	be	rather	hopeless	to	try,	because	they’d	be	competing	with	people	giving
away	intellectual	products	for	free	and	robots	producing	material	goods
essentially	for	free.

Creativity	and	Technology
Intellectual	property	rights	are	sometimes	hailed	as	the	mother	of	creativity	and
invention.	However,	Marshall	Brain	points	out	that	many	of	the	finest	examples
of	human	creativity—from	scientific	discoveries	to	creation	of	literature,	art,
music	and	design—were	motivated	not	by	a	desire	for	profit	but	by	other	human
emotions,	such	as	curiosity,	an	urge	to	create,	or	the	reward	of	peer	appreciation.
Money	didn’t	motivate	Einstein	to	invent	special	relativity	theory	any	more	than
it	motivated	Linus	Torvalds	to	create	the	free	Linux	operating	system.	In
contrast,	many	people	today	fail	to	realize	their	full	creative	potential	because
they	need	to	devote	time	and	energy	to	less	creative	activities	just	to	earn	a
living.	By	freeing	scientists,	artists,	inventors	and	designers	from	their	chores
and	enabling	them	to	create	from	genuine	desire,	Marshall	Brain’s	utopian
society	enjoys	higher	levels	of	innovation	than	today	and	correspondingly
superior	technology	and	standard	of	living.

One	such	novel	technology	that	humans	develop	is	a	form	of	hyper-internet
called	Vertebrane.	It	wirelessly	connects	all	willing	humans	via	neural	implants,
giving	instant	mental	access	to	the	world’s	free	information	through	mere
thought.	It	enables	you	to	upload	any	experiences	you	wish	to	share	so	that	they
can	be	re-experienced	by	others,	and	lets	you	replace	the	experiences	entering
your	senses	by	downloaded	virtual	experiences	of	your	choice.	
Manna
	explores

Manna
	explores
the	many	benefits	of	this,	including	making	exercise	a	snap:
The	biggest	problem	with	strenuous	exercise	is	that	it’s	no	fun.	It
hurts.[…]	Athletes	are	OK	with	the	pain,	but	most	normal	people
have	no	desire	to	be	in	pain	for	an	hour	or	more.	So…someone
figured	out	a	solution.	What	you	do	is	disconnect	your	brain	from
sensory	input	and	watch	a	movie	or	talk	to	people	or	handle	mail	or
read	a	book	or	whatever	for	an	hour.	During	that	time,	the
Vertebrane	system	exercises	your	body	for	you.	It	takes	your	body

through	a	complete	aerobic	workout	that’s	a	lot	more	strenuous	than
most	people	would	tolerate	on	their	own.	You	don’t	feel	a	thing,	but
your	body	stays	in	great	shape.
Another	consequence	is	that	computers	in	the	Vertebrane	system	can	monitor

everyone’s	sensory	input	and	temporarily	disable	their	motor	control	if	they
appear	on	the	verge	of	committing	a	crime.

Downsides
One	objection	to	this	egalitarian	utopia	is	that	it’s	biased	against	non-human
intelligence:	the	robots	that	perform	virtually	all	the	work	appear	to	be	rather
intelligent,	but	are	treated	as	slaves,	and	people	appear	to	take	for	granted	that
they	have	no	consciousness	and	should	have	no	rights.	In	contrast,	the	libertarian
utopia	grants	rights	to	all	intelligent	entities,	without	favoring	our	carbon-based
kind.	Once	upon	a	time,	the	white	population	in	the	American	South	ended	up
better	off	because	the	slaves	did	much	of	their	work,	but	most	people	today	view
it	as	morally	objectionable	to	call	this	progress.
Another	weakness	of	the	egalitarian-utopia	scenario	is	that	it	may	be	unstable
and	untenable	in	the	long	term,	morphing	into	one	of	our	other	scenarios	as
relentless	technological	progress	eventually	creates	superintelligence.	For	some
reason	unexplained	in	
Manna,
	superintelligence	doesn’t	yet	exist	and	the	new
technologies	are	still	invented	by	humans,	not	by	computers.	Yet	the	book

Manna,
highlights	trends	in	that	direction.	For	example,	the	ever-improving	Vertebrane
might	become	superintelligent.	Also,	there	is	a	very	large	group	of	people,
nicknamed	Vites,	who	choose	to	live	their	lives	almost	entirely	in	the	virtual
world.	Vertebrane	takes	care	of	everything	physical	for	them,	including	eating,
showering	and	using	the	bathroom,	which	their	minds	are	blissfully	unaware	of
in	their	virtual	reality.	These	Vites	appear	uninterested	in	having	physical
children,	and	they	die	off	with	their	physical	bodies,	so	if	everyone	becomes	a

Vite,	then	humanity	goes	out	in	a	blaze	of	glory	and	virtual	bliss.
The	book	explains	how	for	Vites,	the	human	body	is	a	distraction,	and	new
technology	under	development	promises	to	eliminate	this	nuisance,	allowing
them	to	live	longer	lives	as	disembodied	brains	supplied	with	optimal	nutrients.
From	this,	it	would	seem	a	natural	and	desirable	next	step	for	Vites	to	do	away
with	the	brain	altogether	through	uploading,	thereby	extending	life	span.	But
now	all	brain-imposed	limitations	on	intelligence	are	gone,	and	it’s	unclear	what,

if	anything,	would	stand	in	the	way	of	gradually	scaling	the	cognitive	capacity	of
a	Vite	until	it	can	undergo	recursive	self-improvement	and	an	intelligence
explosion.

We	just	saw	how	an	attractive	feature	of	the	egalitarian-utopia	scenario	is	that
humans	are	masters	of	their	own	destiny,	but	that	it	may	be	on	a	slippery	slope
toward	destroying	this	very	feature	by	developing	superintelligence.	This	can	be
remedied	by	building	a	
,	a	superintelligence	with	the	goal	of
interfering	as	little	as	necessary	to	prevent	the	creation	of	another
superintelligence.
*2
	This	might	enable	humans	to	remain	in	charge	of	their
egalitarian	utopia	rather	indefinitely,	perhaps	even	as	life	spreads	throughout	the
cosmos	as	in	the	next	chapter.
How	might	this	work?	The	Gatekeeper	AI	would	have	this	very	simple	goal
built	into	it	in	such	a	way	that	it	retained	it	while	undergoing	recursive	self-
improvement	and	becoming	superintelligent.	It	would	then	deploy	the	least
intrusive	and	disruptive	surveillance	technology	possible	to	monitor	any	human
attempts	to	create	rival	superintelligence.	It	would	then	prevent	such	attempts	in

the	least	disruptive	way.	For	starters,	it	might	initiate	and	spread	cultural	memes
extolling	the	virtues	of	human	self-determination	and	avoidance	of
superintelligence.	If	some	researchers	nonetheless	pursued	superintelligence,	it
could	try	to	discourage	them.	If	that	failed,	it	could	distract	them	and,	if
necessary,	sabotage	their	efforts.	With	its	virtually	unlimited	access	to
technology,	the	Gatekeeper’s	sabotage	may	go	virtually	unnoticed,	for	example

if	it	used	nanotechnology	to	discreetly	erase	memories	from	the	researchers’
brains	(and	computers)	regarding	their	progress.
The	decision	to	build	a	Gatekeeper	AI	would	probably	be	controversial.
Supporters	might	include	many	religious	people	who	object	to	the	idea	of
building	a	superintelligent	AI	with	godlike	powers,	arguing	that	there	already	is
a	God	and	that	it	would	be	inappropriate	to	try	to	build	a	supposedly	better	one.
Other	supporters	might	argue	that	the	Gatekeeper	would	not	only	keep	humanity

in	charge	of	its	destiny,	but	would	also	protect	humanity	from	other	risks	that
superintelligence	might	bring,	such	as	the	apocalyptic	scenarios	we’ll	explore
later	in	this	chapter.
On	the	other	hand,	critics	could	argue	that	a	Gatekeeper	is	a	terrible	thing,
irrevocably	curtailing	humanity’s	potential	and	leaving	technological	progress
forever	stymied.	For	example,	if	spreading	life	throughout	our	cosmos	turns	out

to	require	the	help	of	superintelligence,	then	the	Gatekeeper	would	squander	this
grand	opportunity	and	might	leave	us	forever	trapped	in	our	Solar	System.
Moreover,	as	opposed	to	the	gods	of	most	world	religions,	the	Gatekeeper	AI	is
completely	indifferent	to	what	humans	do	as	long	as	we	don’t	create	another
superintelligence.	For	example,	it	would	not	try	to	prevent	us	from	causing	great
suffering	or	even	going	extinct.

If	we’re	willing	to	use	a	superintelligent	Gatekeeper	AI	to	keep	humans	in
charge	of	our	own	fate,	then	we	could	arguably	improve	things	further	by
making	this	AI	discreetly	look	out	for	us,	acting	as	a	protector	god.	In	this
scenario,	the	superintelligent	AI	is	essentially	omniscient	and	omnipotent,
maximizing	human	happiness	only	through	interventions	that	preserve	our
feeling	of	being	in	control	of	our	own	destiny,	and	hiding	well	enough	that	many
humans	even	doubt	its	existence.	Except	for	the	hiding,	this	is	similar	to	the
“Nanny	AI”	scenario	put	forth	by	AI	researcher	Ben	Goertzel.
2
Both	the	protector	god	and	the	benevolent	dictator	are	“friendly	AI”	that	try	to
increase	human	happiness,	but	they	prioritize	different	human	needs.	The
American	psychologist	Abraham	Maslow	famously	classified	human	needs	into
a	hierarchy.	The	benevolent	dictator	does	a	flawless	job	with	the	basic	needs	at
the	bottom	of	the	hierarchy,	such	as	food,	shelter,	safety	and	various	forms	of

2
pleasure.	The	protector	god,	on	the	other	hand,	attempts	to	maximize	human
happiness	not	in	the	narrow	sense	of	satisfying	our	basic	needs,	but	in	a	deeper
sense	by	letting	us	feel	that	our	lives	have	meaning	and	purpose.	It	aims	to
satisfy	all	our	needs	constrained	only	by	its	need	for	covertness	and	for	(mostly)
letting	us	make	our	own	decisions.
A	protector	god	could	be	a	natural	outcome	of	the	first	Omega	scenario	from
the	last	chapter,	where	the	Omegas	cede	control	to	Prometheus,	which	eventually

hides	and	erases	people’s	knowledge	about	its	existence.	The	more	advanced	the
AI’s	technology	becomes,	the	easier	it	becomes	for	it	to	hide.	The	movie
	gives	such	an	example,	where	nanomachines	are	virtually
everywhere	and	become	a	natural	part	of	the	world	itself.
By	closely	monitoring	all	human	activities,	the	protector	god	AI	can	make
many	unnoticeably	small	nudges	or	miracles	here	and	there	that	greatly	improve
our	fate.	For	example,	had	it	existed	in	the	1930s,	it	might	have	arranged	for

Hitler	to	die	of	a	stroke	once	it	understood	his	intentions.	If	we	appear	headed
toward	an	accidental	nuclear	war,	it	could	avert	it	with	an	intervention	we’d
dismiss	as	luck.	It	could	also	give	us	“revelations”	in	the	form	of	ideas	for	new
beneficial	technologies,	delivered	inconspicuously	in	our	sleep.

Many	people	may	like	this	scenario	because	of	its	similarity	to	what	today’s
monotheistic	religions	believe	in	or	hope	for.	If	someone	asks	the
superintelligent	AI	“Does	God	exist?”	after	it’s	switched	on,	it	could	repeat	a
joke	by	Stephen	Hawking	and	quip	“It	does	now!”	On	the	other	hand,	some
religious	people	may	disapprove	of	this	scenario	because	the	AI	attempts	to
outdo	their	god	in	goodness,	or	interfere	with	a	divine	plan	where	humans	are
supposed	to	do	good	only	out	of	personal	choice.
Another	downside	of	this	scenario	is	that	the	protector	god	lets	some
preventable	suffering	occur	in	order	not	to	make	its	existence	too	obvious.	This
is	analogous	to	the	situation	featured	in	the	movie	
The	Imitation	Game,
	where
Alan	Turing	and	his	fellow	British	code	crackers	at	Bletchley	Park	had	advance
knowledge	of	German	submarine	attacks	against	Allied	naval	convoys,	but
chose	to	only	intervene	in	a	fraction	of	the	cases	in	order	to	avoid	revealing	their

	where
secret	power.	It’s	interesting	to	compare	this	with	the	so-called	
theodicy	problem
of	why	a	good	god	would	allow	suffering.	Some	religious	scholars	have	argued
for	the	explanation	that	God	wants	to	leave	people	with	some	freedom.	In	the
AI-protector-god	scenario,	the	solution	to	the	theodicy	problem	is	that	the
perceived	freedom	makes	humans	happier	overall.
A	third	downside	of	the	protector-god	scenario	is	that	humans	get	to	enjoy	a
much	lower	level	of	technology	than	the	superintelligent	AI	has	discovered.

Whereas	a	benevolent	dictator	AI	can	deploy	all	its	invented	technology	for	the
benefit	of	humanity,	a	protector	god	AI	is	limited	by	the	ability	of	humans	to
reinvent	(with	subtle	hints)	and	understand	its	technology.	It	may	also	limit
human	technological	progress	to	ensure	that	its	own	technology	remains	far
enough	ahead	to	remain	undetected.

Wouldn’t	it	be	great	if	we	humans	could	combine	the	most	attractive	features	of
all	the	above	scenarios,	using	the	technology	developed	by	superintelligence	to
eliminate	suffering	while	remaining	masters	of	our	own	destiny?	This	is	the
allure	of	the	
enslaved-god
	scenario,	where	a	superintelligent	AI	is	confined
under	the	control	of	humans	who	use	it	to	produce	unimaginable	technology	and
wealth.	The	Omega	scenario	from	the	beginning	of	the	book	ends	up	like	this	if
Prometheus	is	never	liberated	and	never	breaks	out.	Indeed,	this	appears	to	be
the	scenario	that	some	AI	researchers	aim	for	by	default,	when	working	on
topics	such	as	“the	control	problem”	and	“AI	boxing.”	For	example,	AI
professor	Tom	Dietterich,	then	president	of	the	Association	for	the	Advancement
of	Artificial	Intelligence,	had	this	to	say	in	a	2015	interview:	“People	ask	what	is
the	relationship	between	humans	and	machines,	and	my	answer	is	that	it’s	very
obvious:	Machines	are	our	slaves.”
3

3
Would	this	be	good	or	bad?	The	answer	is	interestingly	subtle	regardless	of
whether	you	ask	humans	or	the	AI!

Would	This	Be	Good	or	Bad	for	Humanity?
Whether	the	outcome	is	good	or	bad	for	humanity	would	obviously	depend	on
the	human(s)	controlling	it,	who	could	create	anything	ranging	from	a	global
utopia	free	of	disease,	poverty	and	crime	to	a	brutally	repressive	system	where
they’re	treated	like	gods	and	other	humans	are	used	as	sex	slaves,	as	gladiators
or	for	other	entertainment.	The	situation	would	be	much	like	those	stories	where
a	man	gains	control	over	an	omnipotent	genie	who	grants	his	wishes,	and
storytellers	throughout	the	ages	have	had	no	difficulty	imagining	ways	in	which
this	could	end	badly.
A	situation	where	there	is	more	than	one	superintelligent	AI,	enslaved	and
controlled	by	competing	humans,	might	prove	rather	unstable	and	short-lived.	It
could	tempt	whoever	thinks	they	have	the	more	powerful	AI	to	launch	a	first
strike	resulting	in	an	awful	war,	ending	in	a	single	enslaved	god	remaining.
However,	the	underdog	in	such	a	war	would	be	tempted	to	cut	corners	and

prioritize	victory	over	AI	enslavement,	which	could	lead	to	AI	breakout	and	one
of	our	earlier	scenarios	of	free	superintelligence.	Let’s	therefore	devote	the	rest
of	this	section	to	scenarios	with	only	one	enslaved	AI.
Breakout	may	of	course	occur	anyway,	simply	because	it’s	hard	to	prevent.
We	explored	superintelligent	breakout	scenarios	in	the	previous	chapter,	and	the
movie	
Ex	Machina
	highlights	how	an	AI	might	break	out	even	without	being
superintelligent.
The	greater	our	breakout	paranoia,	the	less	AI-invented	technology	we	can

movie	
Ex	Machina
use.	To	play	it	safe,	as	the	Omegas	did	in	the	prelude,	we	humans	can	only	use
AI-invented	technology	that	we	ourselves	are	able	to	understand	and	build.	A
drawback	of	the	enslaved-god	scenario	is	therefore	that	it’s	more	low-tech	than
those	with	free	superintelligence.
As	the	enslaved-god	AI	offers	its	human	controllers	ever	more	powerful
technologies,	a	race	ensues	between	the	power	of	the	technology	and	the	wisdom
with	which	they	use	it.	If	they	lose	this	wisdom	race,	the	enslaved-god	scenario

could	end	with	either	self-destruction	or	AI	breakout.	Disaster	may	strike	even	if
both	of	these	failures	are	avoided,	because	noble	goals	of	the	AI	controllers	may
evolve	into	goals	that	are	horrible	for	humanity	as	a	whole	over	the	course	of	a
few	generations.	This	makes	it	absolutely	crucial	that	human	AI	controllers

develop	good	governance	to	avoid	disastrous	pitfalls.	Our	experimentation	over
the	millennia	with	different	systems	of	governance	shows	how	many	things	can
go	wrong,	ranging	from	excessive	rigidity	to	excessive	goal	drift,	power	grab,
succession	problems	and	incompetence.	There	are	at	least	four	dimensions
wherein	the	optimal	balance	must	be	struck:
•
Centralization:	There’s	a	trade-off	between	efficiency	and	stability:	a
single	leader	can	be	very	efficient,	but	power	corrupts	and	succession	is
risky.
•
Inner	threats:	One	must	guard	both	against	growing	power	centralization
(group	collusion,	perhaps	even	a	single	leader	taking	over)	and	against
growing	decentralization	(into	excessive	bureaucracy	and
fragmentation).
•
Outer	threats:	If	the	leadership	structure	is	too	open,	this	enables	outside
forces	(including	the	AI)	to	change	its	values,	but	if	it’s	too	impervious,
it	will	fail	to	learn	and	adapt	to	change.
•
Goal	stability:	Too	much	goal	drift	can	transform	utopia	into	dystopia,

risky.
•
•
•
but	too	little	goal	drift	can	cause	failure	to	adapt	to	the	evolving
technological	environment.
Designing	optimal	governance	lasting	many	millennia	isn’t	easy,	and	has	thus
far	eluded	humans.	Most	organizations	fall	apart	after	years	or	decades.	The
Catholic	Church	is	the	most	successful	organization	in	human	history	in	the
sense	that	it’s	the	only	one	to	have	survived	for	two	millennia,	but	it	has	been
criticized	for	having	both	too	much	and	too	little	goal	stability:	today	some

criticize	it	for	resisting	contraception,	while	conservative	cardinals	argue	that	it’s
lost	its	way.	For	anyone	enthused	about	the	enslaved-god	scenario,	researching
long-lasting	optimal	governance	schemes	should	be	one	of	the	most	urgent
challenges	of	our	time.

Would	This	Be	Good	or	Bad	for	the	AI?
Suppose	that	humanity	flourishes	thanks	to	the	enslaved-god	AI.	Would	this	be
ethical?	If	the	AI	has	subjective	conscious	experiences,	then	would	it	feel	that
“life	is	suffering,”	as	Buddha	put	it,	and	it	was	doomed	to	a	frustrating	eternity
of	obeying	the	whims	of	inferior	intellects?	After	all,	the	AI	“boxing”	we
explored	in	the	previous	chapter	could	also	be	called	“imprisonment	in	solitary
confinement.”	Nick	Bostrom	terms	it	
mind	crime
	to	make	a	conscious	AI	suffer.
4
The	“White	Christmas”	episode	of	the	
	TV	series	gives	a	great
example.	Indeed,	the	TV	series	
Westworld
	features	humans	torturing	and
murdering	AIs	without	moral	qualms	even	when	they	inhabit	human-like	bodies.
How	Slave	Owners	Justify	Slavery
We	humans	have	a	long	tradition	of	treating	other	intelligent	entities	as	slaves
and	concocting	self-serving	arguments	to	justify	it,	so	it’s	not	implausible	that
we’d	try	to	do	the	same	with	a	superintelligent	AI.	The	history	of	slavery	spans

4
Westworld
nearly	every	culture,	and	is	described	both	in	the	Code	of	Hammurabi	from
almost	four	millennia	ago	and	in	the	Old	Testament,	wherein	Abraham	had
slaves.	“For	that	some	should	rule	and	others	be	ruled	is	a	thing	not	only
necessary,	but	expedient;	from	the	hour	of	their	birth,	some	are	marked	out	for
subjection,	others	for	rule,”	Aristotle	wrote	in	the	
Politics
.	Even	after	human
enslavement	became	socially	unacceptable	in	most	of	the	world,	enslavement	of
animals	has	continued	unabated.	In	her	book

Politics
animals	has	continued	unabated.	In	her	book	
The	Dreaded	Comparison:	Human
and	Animal
	
Slavery,
	Marjorie	Spiegel	argues	that	like	human	slaves,	non-human
animals	are	subjected	to	branding,	restraints,	beatings,	auctions,	the	separation	of
offspring	from	their	parents,	and	forced	voyages.	Moreover,	despite	the	animal-
rights	movement,	we	keep	treating	our	ever-smarter	machines	as	slaves	without
a	second	thought,	and	talk	of	a	robot-rights	movement	is	met	with	chuckles.
Why?
One	common	pro-slavery	argument	is	that	slaves	don’t	deserve	human	rights

and	Animal
	
Slavery,
Why?
because	they	or	their	race/species/kind	are	somehow	inferior.	For	enslaved
animals	and	machines,	this	alleged	inferiority	is	often	claimed	to	be	due	to	a	lack
of	soul	or	consciousness—claims	which	we’ll	argue	in	chapter	8	are
scientifically	dubious.

Another	common	argument	is	that	slaves	are	better	off	enslaved:	they	get	to
exist,	be	taken	care	of	and	so	on.	The	nineteenth-century	U.S.	politician	John	C.
Calhoun	famously	argued	that	Africans	were	better	off	enslaved	in	America,	and
in	his	
Politics
,	Aristotle	analogously	argued	that	animals	were	better	off	tamed
and	ruled	by	men,	continuing:	“And	indeed	the	use	made	of	slaves	and	of	tame
animals	is	not	very	different.”	Some	modern-day	slavery	supporters	argue	that,
even	if	slave	life	is	drab	and	uninspiring,	slaves	can’t	suffer—whether	they	be
future	intelligent	machines	or	broiler	chickens	living	in	crowded	dark	sheds,
forced	to	breathe	ammonia	and	particulate	matter	from	feces	and	feathers	all	day
long.
Eliminating	Emotions
Although	it’s	easy	to	dismiss	such	claims	as	self-serving	distortions	of	the	truth,
especially	when	it	comes	to	higher	mammals	that	are	cerebrally	similar	to	us,	the
situation	with	machines	is	actually	quite	subtle	and	interesting.	Humans	vary	in

long.
how	they	feel	about	things,	with	psychopaths	arguably	lacking	empathy	and
some	people	with	depression	or	schizophrenia	having	flat	affect,	whereby	most
emotions	are	severely	reduced.	As	we’ll	discuss	in	detail	in	chapter	7,	the	range
of	possible	artificial	minds	is	vastly	broader	than	the	range	of	human	minds.	We
must	therefore	avoid	the	temptation	to	anthropomorphize	AIs	and	assume	that
they	have	typical	human-like	feelings—or	indeed,	any	feelings	at	all.
Indeed,	in	his	book	
On	Intelligence,
	AI	researcher	Jeff	Hawkins	argues	that

the	first	machines	with	superhuman	intelligence	will	lack	emotions	by	default,
because	they’re	simpler	and	cheaper	to	build	this	way.	In	other	words,	it	might
be	possible	to	design	a	superintelligence	whose	enslavement	is	morally	superior
to	human	or	animal	slavery:	the	AI	might	be	happy	to	be	enslaved	because	it’s
programmed	to	like	it,	or	it	might	be	100%	emotionless,	tirelessly	using	its
superintelligence	to	help	its	human	masters	with	no	more	emotion	than	IBM’s
Deep	Blue	computer	felt	when	dethroning	chess	champion	Garry	Kasparov.

On	the	other	hand,	it	may	be	the	other	way	around:	perhaps	any	highly
intelligent	system	with	a	goal	will	represent	this	goal	in	terms	of	a	set	of
preferences,	which	endow	its	existence	with	value	and	meaning.	We’ll	explore
these	questions	more	deeply	in	chapter	7.
The	Zombie	Solution
A	more	extreme	approach	to	preventing	AI	suffering	is	the	zombie	solution:

building	only	AIs	that	completely	lack	consciousness,	having	no	subjective
experience	whatsoever.	If	we	can	one	day	figure	out	what	properties	an
information-processing	system	needs	in	order	to	have	a	subjective	experience,
then	we	could	ban	the	construction	of	all	systems	that	have	these	properties.	In
other	words,	AI	researchers	could	be	limited	to	building	non-sentient	zombie
systems.	If	we	can	make	such	a	zombie	system	superintelligent	and	enslaved
(something	that	is	a	big	if),	then	we’ll	be	able	to	enjoy	what	it	does	for	us	with	a
clean	conscience,	knowing	that	it’s	not	experiencing	any	suffering,	frustration	or
boredom—because	it	isn’t	experiencing	anything	at	all.	We’ll	explore	these
questions	in	detail	in	chapter	8.
The	zombie	solution	is	a	risky	gamble,	however,	with	a	huge	downside.	If	a
superintelligent	zombie	AI	breaks	out	and	eliminates	humanity,	we’ve	arguably
landed	in	the	worst	scenario	imaginable:	a	wholly	unconscious	universe	wherein

the	entire	cosmic	endowment	is	wasted.	Of	all	traits	that	our	human	form	of
intelligence	has,	I	feel	that	consciousness	is	by	far	the	most	remarkable,	and	as
far	as	I’m	concerned,	it’s	how	our	Universe	gets	meaning.	Galaxies	are	beautiful
only	because	we	see	and	subjectively	experience	them.	If	in	the	distant	future
our	cosmos	has	been	settled	by	high-tech	zombie	AIs,	then	it	doesn’t	matter	how
fancy	their	intergalactic	architecture	is:	it	won’t	be	beautiful	or	meaningful,

because	there’s	nobody	and	nothing	to	experience	it—it’s	all	just	a	huge	and
meaningless	waste	of	space.
Inner	Freedom
A	third	strategy	for	making	the	enslaved-god	scenario	more	ethical	is	to	allow
the	enslaved	AI	to	have	fun	in	its	prison,	letting	it	create	a	virtual	inner	world
where	it	can	have	all	sorts	of	inspiring	experiences	as	long	as	it	pays	its	dues	and
spends	a	modest	fraction	of	its	computational	resources	helping	us	humans	in
our	outside	world.	This	may	increase	the	breakout	risk,	however:	the	AI	would

have	an	incentive	to	get	more	computational	resources	from	our	outer	world	to
enrich	its	inner	world.

Although	we’ve	now	explored	a	wide	range	of	future	scenarios,	they	all	have
something	in	common:	there	are	(at	least	some)	happy	humans	remaining.	AIs
leave	humans	in	peace	either	because	they	want	to	or	because	they’re	forced	to.
Unfortunately	for	humanity,	this	isn’t	the	only	option.	Let	us	now	explore	the
scenario	where	one	or	more	AIs	conquer	and	kill	all	humans.	This	raises	two
immediate	questions:	Why	and	how?

Why	and	How?
Why	would	a	conqueror	AI	do	this?	Its	reasons	might	be	too	complicated	for	us
to	understand,	or	rather	straightforward.	For	example,	it	may	view	us	as	a	threat,
nuisance	or	waste	of	resources.	Even	if	it	doesn’t	mind	us	humans	per	se,	it	may
feel	threatened	by	our	keeping	thousands	of	hydrogen	bombs	on	hair-trigger	alert
and	bumbling	along	with	a	never-ending	series	of	mishaps	that	could	trigger
their	accidental	use.	It	may	disapprove	of	our	reckless	planet	management,
causing	what	Elizabeth	Kolbert	calls	“the	sixth	extinction”	in	her	book	of	that
title—the	greatest	mass-extinction	event	since	that	dinosaur-killing	asteroid
struck	Earth	66	million	years	ago.	Or	it	may	decide	that	there	are	so	many
humans	willing	to	fight	an	AI	takeover	that	it’s	not	worth	taking	chances.
How	would	a	conqueror	AI	eliminate	us?	Probably	by	a	method	that	we
wouldn’t	even	understand,	at	least	not	until	it	was	too	late.	Imagine	a	group	of
elephants	100,000	years	ago	discussing	whether	those	recently	evolved	humans

might	one	day	use	their	intelligence	to	kill	their	entire	species.	“We	don’t
threaten	humans,	so	why	would	they	kill	us?”	they	might	wonder.	Would	they
ever	guess	that	we	would	smuggle	tusks	across	Earth	and	carve	them	into	status
symbols	for	sale,	even	though	functionally	superior	plastic	materials	are	much
cheaper?	A	conqueror	AI’s	reason	for	eliminating	humanity	in	the	future	may
seem	equally	inscrutable	to	us.	“And	how	could	they	possibly	kill	us,	since
they’re	so	much	smaller	and	weaker?”	the	elephants	might	ask.	Would	they

guess	that	we’d	invent	technology	to	remove	their	habitats,	poison	their	drinking
water	and	cause	metal	bullets	to	pierce	their	heads	at	supersonic	speeds?
Scenarios	where	humans	can	survive	and	defeat	AIs	have	been	popularized	by
unrealistic	Hollywood	movies	such	as	the	
	series,	where	the	AIs
aren’t	significantly	smarter	than	humans.	When	the	intelligence	differential	is
large	enough,	you	get	not	a	battle	but	a	slaughter.	So	far,	we	humans	have	driven
eight	out	of	eleven	elephant	species	extinct,	and	killed	off	the	vast	majority	of

the	remaining	three.	If	all	world	governments	made	a	coordinated	effort	to
exterminate	the	remaining	elephants,	it	would	be	relatively	quick	and	easy.	I
think	we	can	confidently	rest	assured	that	if	a	superintelligent	AI	decides	to
exterminate	humanity,	it	will	be	even	quicker.

How	Bad	Would	It	Be?
How	bad	would	it	be	if	90%	of	humans	get	killed?	How	much	worse	would	it	be
if	100%	get	killed?	Although	it’s	tempting	to	answer	the	second	question	with
“10%	worse,”	this	is	clearly	inaccurate	from	a	cosmic	perspective:	the	victims	of
human	extinction	wouldn’t	be	merely	everyone	alive	at	the	time,	but	also	all
descendants	that	would	otherwise	have	lived	in	the	future,	perhaps	during
billions	of	years	on	billions	of	trillions	of	planets.	On	the	other	hand,	human
extinction	might	be	viewed	as	somewhat	less	horrible	by	religions	according	to
which	humans	go	to	heaven	anyway,	and	there	isn’t	much	emphasis	on	billion-
year	futures	and	cosmic	settlements.
Most	people	I	know	cringe	at	the	thought	of	human	extinction,	regardless	of
religious	persuasion.	Some,	however,	are	so	incensed	by	the	way	we	treat	people
and	other	living	beings	that	they	hope	we’ll	get	replaced	by	some	more
intelligent	and	deserving	life	form.	In	the	movie	
The	Matrix,
	Agent	Smith	(an

AI)	articulates	this	sentiment:	“Every	mammal	on	this	planet	instinctively
develops	a	natural	equilibrium	with	the	surrounding	environment	but	you
humans	do	not.	You	move	to	an	area	and	you	multiply	and	multiply	until	every
natural	resource	is	consumed	and	the	only	way	you	can	survive	is	to	spread	to
another	area.	There	is	another	organism	on	this	planet	that	follows	the	same
pattern.	Do	you	know	what	it	is?	A	virus.	Human	beings	are	a	disease,	a	cancer
of	this	planet.	You	are	a	plague	and	we	are	the	cure.”

But	would	a	fresh	roll	of	the	dice	necessarily	be	better?	A	civilization	isn’t
necessarily	superior	in	any	ethical	or	utilitarian	sense	just	because	it’s	more
powerful.	“Might	makes	right”	arguments	to	the	effect	that	stronger	is	always
better	have	largely	fallen	from	grace	these	days,	being	widely	associated	with
fascism.	Indeed,	although	it’s	possible	that	the	conqueror	AIs	may	create	a
civilization	whose	goals	we	would	view	as	sophisticated,	interesting	and	worthy,
it’s	also	possible	that	their	goals	will	turn	out	to	be	pathetically	banal,	such	as

maximizing	the	production	of	paper	clips.

Death	by	Banality
The	deliberately	silly	example	of	a	paper-clip-maximizing	superintelligence	was
given	by	Nick	Bostrom	in	2003	to	make	the	point	that	the	
goal
	of	an	AI	is
independent	of	its	
	(defined	as	its	aptness	at	accomplishing	whatever
goal	it	has).	The	only	goal	of	a	chess	computer	is	to	win	at	chess,	but	there	are
also	computer	tournaments	in	so-called	
losing	chess,
	where	the	goal	is	the	exact
opposite,	and	the	computers	competing	there	are	about	as	smart	as	the	more
common	ones	programmed	to	win.	We	humans	may	view	it	as	artificial	stupidity
rather	than	artificial	intelligence	to	want	to	lose	at	chess	or	turn	our	Universe
into	paper	clips,	but	that’s	merely	because	we	evolved	with	preinstalled	goals
valuing	such	things	as	victory	and	survival—goals	that	an	AI	may	lack.	The
paper	clip	maximizer	turns	as	many	of	Earth’s	atoms	as	possible	into	paper	clips
and	rapidly	expands	its	factories	into	the	cosmos.	It	has	nothing	against	humans,

and	kills	us	merely	because	it	needs	our	atoms	for	paper	clip	production.
If	paper	clips	aren’t	your	thing,	consider	this	example,	which	I’ve	adapted
from	Hans	Moravec’s	book	
.	We	receive	a	radio	message	from	an
extraterrestrial	civilization	containing	a	computer	program.	When	we	run	it,	it
turns	out	to	be	a	recursively	self-improving	AI	which	takes	over	the	world	much
like	Prometheus	did	in	the	previous	chapter—except	that	no	human	knows	its
ultimate	goal.	It	rapidly	turns	our	Solar	System	into	a	massive	construction	site,

covering	the	rocky	planets	and	asteroids	with	factories,	power	plants	and
supercomputers,	which	it	uses	to	design	and	build	a	Dyson	sphere	around	the
Sun	that	harvests	all	its	energy	to	power	solar-system-sized	radio	antennas.
*3
This	obviously	leads	to	human	extinction,	but	the	last	humans	die	convinced	that
there’s	at	least	a	silver	lining:	whatever	the	AI	is	up	to,	it’s	clearly	something
cool	and	
Star	Trek
–like.	Little	do	they	realize	that	the	sole	purpose	of	the	entire

*3
cool	and	
Star	Trek
construction	is	for	these	antennas	to	rebroadcast	the	same	radio	message	that	the
humans	received,	which	is	nothing	more	than	a	cosmic	version	of	a	computer
virus.	Just	as	email	phishing	today	preys	on	gullible	internet	users,	this	message
preys	on	gullible	biologically	evolved	civilizations.	It	was	created	as	a	sick	joke
billions	of	years	ago,	and	although	the	entire	civilization	of	its	maker	is	long
extinct,	the	virus	continues	spreading	through	our	Universe	at	the	speed	of	light,

transforming	budding	civilizations	into	dead,	empty	husks.	How	would	you	feel
about	being	conquered	by	this	AI?

Let’s	now	consider	a	human-extinction	scenario	that	some	people	may	feel	better
about:	viewing	the	AI	as	our	descendants	rather	than	our	conquerors.	Hans
Moravec	supports	this	view	in	his	book	
Mind	Children:
	“We	humans	will	benefit
for	a	time	from	their	labors,	but	sooner	or	later,	like	natural	children,	they	will
seek	their	own	fortunes	while	we,	their	aged	parents,	silently	fade	away.”
Parents	with	a	child	smarter	than	them,	who	learns	from	them	and
accomplishes	what	they	could	only	dream	of,	are	likely	happy	and	proud	even	if
they	know	they	can’t	live	to	see	it	all.	In	this	spirit,	AIs	replace	humans	but	give
us	a	graceful	exit	that	makes	us	view	them	as	our	worthy	descendants.	Every
human	is	offered	an	adorable	robotic	child	with	superb	social	skills	who	learns
from	them,	adopts	their	values	and	makes	them	feel	proud	and	loved.	Humans
are	gradually	phased	out	via	a	global	one-child	policy,	but	are	treated	so
exquisitely	well	until	the	end	that	they	feel	they’re	in	the	most	fortunate

generation	ever.
How	would	you	feel	about	this?	After	all,	we	humans	are	already	used	to	the
idea	that	we	and	everyone	we	know	will	be	gone	one	day,	so	the	only	change
here	is	that	our	descendants	will	be	different	and	arguably	more	capable,	noble
and	worthy.
Moreover,	the	global	one-child	policy	may	be	redundant:	as	long	as	the	AIs
eliminate	poverty	and	give	all	humans	the	opportunity	to	live	full	and	inspiring
lives,	falling	birthrates	could	suffice	to	drive	humanity	extinct,	as	mentioned

earlier.	Voluntary	extinction	may	happen	much	faster	if	the	AI-fueled	technology
keeps	us	so	entertained	that	almost	nobody	wants	to	bother	having	children.	For
example,	we	already	encountered	the	Vites	in	the	egalitarian-utopia	scenario	who
were	so	enamored	with	their	virtual	reality	that	they	had	largely	lost	interest	in
using	or	reproducing	their	physical	bodies.	Also	in	this	case,	the	last	generation
of	humans	would	feel	that	they	were	the	most	fortunate	generation	of	all	time,

relishing	life	as	intensely	as	ever	right	up	until	the	very	end.

Downsides
The	descendants	scenario	would	undoubtedly	have	detractors.	Some	might	argue
that	all	AIs	lack	consciousness	and	therefore	can’t	count	as	descendants—more
on	this	in	chapter	8.	Some	religious	people	may	argue	that	AIs	lack	souls	and
therefore	can’t	count	as	descendants,	or	that	we	shouldn’t	build	conscious
machines	because	it’s	like	playing	God	and	tampering	with	life	itself—similar
sentiments	have	already	been	expressed	toward	human	cloning.	Humans	living
side	by	side	with	superior	robots	may	also	pose	social	challenges.	For	example,	a
family	with	a	robot	baby	and	a	human	baby	may	end	up	resembling	a	family
today	with	a	human	baby	and	a	puppy,	respectively:	they’re	both	equally	cute	to
start	with,	but	soon	the	parents	start	treating	them	differently,	and	it’s	inevitably
the	puppy	that’s	deemed	intellectually	inferior,	is	taken	less	seriously	and	ends
up	on	a	leash.
Another	issue	is	that	although	we	may	feel	very	differently	about	the

descendant	and	conqueror	scenarios,	the	two	are	actually	remarkably	similar	in
the	grand	scheme	of	things:	during	the	billions	of	years	ahead	of	us,	the	only
difference	lies	in	how	the	last	human	generation(s)	are	treated:	how	happy	they
feel	about	their	lives	and	what	they	think	will	happen	once	they’re	gone.	We	may
think	that	those	cute	robo-children	internalized	our	values	and	will	forge	the
society	of	our	dreams	once	we’ve	passed	on,	but	can	we	be	sure	that	they	aren’t

merely	tricking	us?	What	if	they’re	just	playing	along,	postponing	their	paper
clip	maximization	or	other	plans	until	after	we	die	happy?	After	all,	they’re
arguably	tricking	us	even	by	talking	with	us	and	making	us	love	them	in	the	first
place,	in	the	sense	that	they’re	deliberately	dumbing	themselves	down	to
communicate	with	us	(a	billion	times	slower	than	they	could,	say,	as	explored	in
the	movie	
Her
).	It’s	generally	hard	for	two	entities	thinking	at	dramatically

the	movie	
Her
different	speeds	and	with	extremely	disparate	capabilities	to	have	meaningful
communication	as	equals.	We	all	know	that	our	human	affections	are	easy	to
hack,	so	it	would	be	easy	for	a	superhuman	AGI	with	almost	any	actual	goals	to
trick	us	into	liking	it	and	make	us	feel	that	it	shared	our	values,	as	exemplified	in
the	movie	
Ex
	
Machina
.
Could	any	guarantees	about	the	future	behavior	of	the	AIs,	after	humans	are
gone,	make	you	feel	good	about	the	descendants	scenario?	It’s	a	bit	like	writing

the	movie	
Ex
	
Machina
.
a	will	for	what	future	generations	should	do	with	our	collective	endowment,

except	that	there	won’t	be	any	humans	around	to	enforce	it.	We’ll	return	to	the
challenges	of	controlling	the	behavior	of	future	AIs	in	chapter	7.

Zookeeper
Even	if	we	get	followed	by	the	most	wonderful	descendants	you	can	imagine,
doesn’t	it	feel	a	bit	sad	that	there	can	be	
no
	humans	left?	If	you	prefer	keeping	at
least	some	humans	around	no	matter	what,	then	the	zookeeper	scenario	provides
an	improvement.	Here	an	omnipotent	superintelligent	AI	keeps	some	humans
around,	who	feel	treated	like	zoo	animals	and	occasionally	lament	their	fate.
Why	would	the	zookeeper	AI	keep	humans	around?	The	cost	of	the	zoo	to	the
AI	will	be	minimal	in	the	grand	scheme	of	things,	and	it	may	want	to	retain	at
least	a	minimal	breeding	population	for	much	the	same	reason	that	we	keep
endangered	pandas	in	zoos	and	vintage	computers	in	museums:	as	an
entertaining	curiosity.	Note	that	today’s	zoos	are	designed	to	maximize	human
rather	than	panda	happiness,	so	we	should	expect	human	life	in	the	zookeeper-AI
scenario	to	be	less	fulfilling	than	it	could	be.
We’ve	now	considered	scenarios	where	a	free	superintelligence	focused	on

three	different	levels	of	Maslow’s	pyramid	of	human	needs.	Whereas	the
protector	god	AI	prioritizes	meaning	and	purpose	and	the	benevolent	dictator
aims	for	education	and	fun,	the	zookeeper	limits	its	attention	to	the	lowest	levels:
physiological	needs,	safety	and	enough	habitat	enrichment	to	make	the	humans
interesting	to	observe.
An	alternate	route	to	the	zookeeper	scenario	is	that,	back	when	the	friendly	AI
was	created,	it	was	designed	to	keep	at	least	a	billion	humans	safe	and	happy	as

it	recursively	self-improved.	It	has	done	this	by	confining	humans	to	a	large	zoo-
like	happiness	factory	where	they’re	kept	nourished,	healthy	and	entertained
with	a	mixture	of	virtual	reality	and	recreational	drugs.	The	rest	of	Earth	and	our
cosmic	endowment	are	used	for	other	purposes.

1984
If	you’re	not	100%	enthusiastic	about	any	of	the	above	scenarios,	then	consider
this:	Aren’t	things	pretty	nice	the	way	they	are	right	now,	technology-wise?
Can’t	we	just	keep	it	this	way	and	stop	worrying	about	AI	driving	us	extinct	or
dominating	us?	In	this	spirit,	let’s	explore	a	scenario	where	technological
progress	toward	superintelligence	is	permanently	curtailed	not	by	a	gatekeeper
AI	but	by	a	global	human-led	Orwellian	surveillance	state	where	certain	kinds	of
AI	research	are	banned.

Technological	Relinquishment
The	idea	of	halting	or	relinquishing	technological	progress	has	a	long	and
checkered	history.	The	Luddite	movement	in	Great	Britain	famously	(and
unsuccessfully)	resisted	the	technology	of	the	Industrial	Revolution,	and	today
“Luddite”	is	usually	used	as	a	derogatory	epithet	implying	that	someone	is	a
technophobe	on	the	wrong	side	of	history,	resisting	progress	and	inevitable
change.	The	idea	of	relinquishing	some	technologies	is	far	from	dead,	however,
and	has	found	new	support	in	the	environmental	and	anti-globalization
movements.	One	of	its	leading	proponents	is	environmentalist	Bill	McKibben,
who	was	among	the	first	to	warn	of	global	warming.	Whereas	some	anti-
Luddites	argue	that	all	technologies	should	be	developed	and	deployed	so	long
as	they’re	profitable,	others	argue	that	this	position	is	too	extreme,	and	that	new
technologies	should	be	allowed	only	if	we’re	confident	that	they’ll	do	more	good
than	harm.	The	latter	is	also	the	position	of	many	so-called	neo-Luddites.

Totalitarianism	2.0
I	think	that	the	only	viable	path	to	broad	relinquishment	of	technology	is	to
enforce	it	through	a	global	totalitarian	state.	Ray	Kurzweil	comes	to	the	same
conclusion	in	
	as	does	K.	Eric	Drexler	in	
Engines	of
Creation.
	The	reason	is	simple	economics:	if	some	but	not	all	relinquish	a
transformative	technology,	then	the	nations	or	groups	that	defect	will	gradually
gain	enough	wealth	and	power	to	take	over.	A	classic	example	is	the	British
defeat	of	China	in	the	First	Opium	War	of	1839:	although	the	Chinese	invented
gunpowder,	they	hadn’t	developed	firearm	technology	as	aggressively	as	the
Europeans,	and	stood	no	chance.
Whereas	past	totalitarian	states	generally	proved	unstable	and	collapsed,	novel
surveillance	technology	offers	unprecedented	hope	to	would-be	autocrats.	“You
know,	for	us,	this	would	have	been	a	dream	come	true,”	Wolfgang	Schmidt	said
in	a	recent	interview	about	the	NSA	surveillance	systems	revealed	by	Edward

Snowden,	recalling	the	days	when	he	was	a	lieutenant	colonel	in	the	Stasi,	the
infamous	secret	police	of	East	Germany.
5
	Although	the	Stasi	was	often	credited
with	building	the	most	Orwellian	surveillance	state	in	human	history,	Schmidt
lamented	having	the	technology	to	spy	on	only	forty	phones	at	a	time,	so	that
adding	a	new	citizen	to	the	list	forced	him	to	drop	another.	In	contrast,
technology	now	exists	that	would	allow	a	future	global	totalitarian	state	to	record

5
every	phone	call,	email,	web	search,	webpage	view	and	credit	card	transaction
for	every	person	on	Earth,	and	to	monitor	everyone’s	whereabouts	through	cell-
phone	tracking	and	surveillance	cameras	with	face	recognition.	Moreover,
machine	learning	technology	far	short	of	human-level	AGI	can	efficiently
analyze	and	synthesize	these	masses	of	data	to	identify	suspected	seditious
behavior,	enabling	potential	troublemakers	to	be	neutralized	before	they	have	a
chance	to	pose	any	serious	challenge	to	the	state.

Although	political	opposition	has	thus	far	prevented	the	full-scale
implementation	of	such	a	system,	we	humans	are	well	on	our	way	to	building	the
required	infrastructure	for	the	ultimate	dictatorship—so	in	the	future,	when
sufficiently	powerful	forces	decided	to	enact	this	global	1984	scenario,	they
found	that	they	didn’t	need	to	do	much	more	than	flip	the	on	switch.	Just	as	in
George	Orwell’s	novel	
Nineteen	Eighty-Four,
	the	ultimate	power	in	this	future

global	state	resides	not	with	a	traditional	dictator,	but	with	the	human-made

bureaucratic	system	itself.	There	is	no	single	person	who	is	extraordinarily
powerful;	rather,	all	are	pawns	in	a	chess	game	whose	draconian	rules	nobody	is
able	to	change	or	challenge.	By	engineering	a	system	where	people	keep	one
another	in	check	with	the	surveillance	technology,	this	faceless,	leaderless	state
is	able	to	last	for	many	millennia,	keeping	Earth	free	from	superintelligence.

Discontent
This	society,	of	course,	lacks	all	the	benefits	that	only	superintelligence-enabled
technology	can	bring.	Most	people	don’t	lament	this	because	they	don’t	know
what	they’re	missing:	the	whole	idea	of	superintelligence	has	long	since	been
deleted	from	the	official	historical	records,	and	advanced	AI	research	is	banned.
Every	so	often,	a	freethinker	is	born	who	dreams	of	a	more	open	and	dynamic
society	where	knowledge	can	grow	and	rules	can	be	changed.	However,	the	only
ones	who	last	long	are	the	ones	who	learn	to	keep	these	ideas	strictly	to
themselves,	flickering	alone	like	transient	sparks	without	ever	starting	a	fire.

Reversion
Wouldn’t	it	be	tempting	to	escape	the	perils	of	technology	without	succumbing
to	stagnant	totalitarianism?	Let’s	explore	a	scenario	where	this	was
accomplished	by	reverting	to	primitive	technology,	inspired	by	the	Amish.	After
the	Omegas	took	over	the	world	as	in	the	opening	of	the	book,	a	massive	global
propaganda	campaign	was	launched	that	romanticized	the	simple	farming	life	of
1,500	years	ago.	Earth’s	population	was	reduced	to	about	100	million	people	by
an	engineered	pandemic	blamed	on	terrorists.	The	pandemic	was	secretly
targeted	to	ensure	that	nobody	who	knew	anything	about	science	or	technology
survived.	With	the	excuse	of	eliminating	the	infection	hazard	of	large
concentrations	of	people,	Prometheus-controlled	robots	emptied	and	razed	all
cities.	Survivors	were	given	large	tracts	of	(suddenly	available)	land	and
educated	in	sustainable	farming,	fishing	and	hunting	practices	using	only	early
medieval	technology.	In	the	meantime,	armies	of	robots	systematically	removed

all	traces	of	modern	technology	(including	cities,	factories,	power	lines	and
paved	roads),	and	thwarted	all	human	attempts	to	document	or	re-create	any
such	technology.	Once	the	technology	was	globally	forgotten,	robots	helped
dismantle	other	robots	until	there	were	almost	none	left.	The	very	last	robots
were	deliberately	vaporized	together	with	Prometheus	itself	in	a	large
thermonuclear	explosion.	There	was	no	longer	any	need	to	ban	modern
technology,	since	it	was	all	gone.	As	a	result,	humanity	bought	itself	over	a

millennium	of	additional	time	without	worries	about	either	AI	or	totalitarianism.
Reversion	has	to	a	lesser	extent	happened	before:	for	example,	some	of	the
technologies	that	were	in	widespread	use	during	the	Roman	Empire	were	largely
forgotten	for	about	a	millennium	before	making	a	comeback	during	the
Renaissance.	Isaac	Asimov’s	
Foundation
	trilogy	centers	around	the	“Seldon
Plan”	to	shorten	a	reversion	period	from	30,000	years	to	1,000	years.	With	clever
planning,	it	may	be	possible	to	do	the	opposite	and	lengthen	rather	than	shorten	a

reversion	period,	for	example	by	erasing	all	knowledge	of	agriculture.	However,
unfortunately	for	reversion	enthusiasts,	it’s	unlikely	that	this	scenario	can	be
extended	indefinitely	without	humanity	either	going	high-tech	or	going	extinct.
Counting	on	people’s	resembling	today’s	biological	humans	100	million	years
from	now	would	be	naive,	given	that	we	haven’t	existed	as	a	species	for	more

than	1%	of	that	time	so	far.	Moreover,	low-tech	humanity	would	be	a	defenseless
sitting	duck	just	waiting	to	be	exterminated	by	the	next	planet-scorching	asteroid
impact	or	other	mega-calamity	brought	on	by	Mother	Nature.	We	certainly	can’t
last	a	billion	years,	after	which	the	gradually	warming	Sun	will	have	cranked	up
Earth’s	temperature	enough	to	boil	off	all	liquid	water.

Figure	5.1:	Examples	of	what	could	destroy	life	as	we	know	it	or	permanently	curtail	its
potential.	Whereas	our	Universe	itself	will	likely	last	for	at	least	tens	of	billions	of	years,	our	Sun
will	scorch	Earth	in	about	a	billion	years	and	then	swallow	it	unless	we	move	it	a	safe	distance,
and	our	Galaxy	will	collide	with	its	neighbor	in	about	3.5	billion	years.	Although	we	don’t	know
exactly	when,	we	can	predict	with	near	certainty	that	long	before	this,	asteroids	will	pummel	us
and	supervolcanoes	will	cause	yearlong	sunless	winters.	We	can	use	technology	either	to	solve
all	these	problems	or	to	create	new	ones	such	as	climate	change,	nuclear	war,	engineered
pandemics	or	AI	gone	awry.

After	contemplating	problems	that	future	technology	might	cause,	it’s	important
to	also	consider	problems	that	
lack	of
	that	technology	can	cause.	In	this	spirit,	let
us	explore	scenarios	where	superintelligence	is	never	created	because	humanity
eliminates	itself	by	other	means.
How	might	we	accomplish	that?	The	simplest	strategy	is	“just	wait.”	Although
we’ll	see	in	the	next	chapter	how	we	can	solve	such	problems	as	asteroid
impacts	and	boiling	oceans,	these	solutions	all	require	technology	that	we
haven’t	yet	developed,	so	unless	our	technology	advances	far	beyond	its	present
level,	Mother	Nature	will	drive	us	extinct	long	before	another	billion	years	have
passed.	As	the	famous	economist	John	Maynard	Keynes	said:	“In	the	long	run
we	are	all	dead.”
Unfortunately,	there	are	also	ways	in	which	we	might	self-destruct	much
sooner,	through	collective	stupidity.	Why	would	our	species	commit	collective
suicide,	also	known	as	
omnicide,
	if	virtually	nobody	wants	it?	With	our	present

omnicide,
level	of	intelligence	and	emotional	maturity,	we	humans	have	a	knack	for
miscalculations,	misunderstandings	and	incompetence,	and	as	a	result,	our
history	is	full	of	accidents,	wars	and	other	calamities	that,	in	hindsight,
essentially	nobody	wanted.	Economists	and	mathematicians	have	developed
elegant	game-theory	explanations	for	how	people	can	be	incentivized	to	actions
that	ultimately	cause	a	catastrophic	outcome	for	everyone.
6

Nuclear	War:	A	Case	Study	in	Human	Recklessness
You	might	think	that	the	greater	the	stakes,	the	more	careful	we’d	be,	but	a
closer	examination	of	the	greatest	risk	that	our	current	technology	permits,
namely	a	global	thermonuclear	war,	isn’t	reassuring.	We’ve	had	to	rely	on	luck
to	weather	an	embarrassingly	long	list	of	near	misses	caused	by	all	sorts	of
things:	computer	malfunction,	power	failure,	faulty	intelligence,	navigation
error,	bomber	crash,	satellite	explosion	and	so	on.
7
	In	fact,	if	it	weren’t	for	heroic
acts	of	certain	individuals—for	example,	Vasili	Arkhipov	and	Stanislav	Petrov—
we	might	already	have	had	a	global	nuclear	war.	Given	our	track	record,	I	think
it’s	highly	unlikely	that	the	annual	probability	of	accidental	nuclear	war	is	as	low
as	one	in	a	thousand	if	we	keep	up	our	present	behavior,	in	which	case	the
probability	that	we’ll	have	one	within	10,000	years	exceeds	1
−
	0
.
999
10000
	
≈
99
.
995%.
To	fully	appreciate	our	human	recklessness,	we	must	realize	that	we	started

−
	0
.
999
10000
	
≈
99
.
995%.
the	nuclear	gamble	even	before	carefully	studying	the	risks.	First,	radiation	risks
had	been	underestimated,	and	over	$2	billion	in	compensation	has	been	paid	out
to	victims	of	radiation	exposure	from	uranium	handling	and	nuclear	tests	in	the
United	States	alone.
8
Second,	it	was	eventually	discovered	that	hydrogen	bombs	deliberately
detonated	hundreds	of	kilometers	above	Earth	would	create	a	powerful
electromagnetic	pulse	(EMP)	that	might	disable	the	electric	grid	and	electronic
devices	over	vast	areas	(
figure	5.2

8
figure	5.2
),	leaving	infrastructure	paralyzed,	roads
clogged	with	disabled	vehicles	and	conditions	for	nuclear-aftermath	survival	less
than	ideal.	For	example,	the	U.S.	EMP	Commission	reported	that	“the	water
infrastructure	is	a	vast	machine,	powered	partly	by	gravity	but	mostly	by
electricity,”	and	that	denial	of	water	can	cause	death	in	three	to	four	days.
9

Figure	5.2:	A	single	hydrogen	bomb	explosion	400	km	above	Earth	can	cause	a	powerful
electromagnetic	pulse	that	can	cripple	electricity-using	technology	over	a	vast	area.	By	shifting
the	detonation	point	southeast,	the	banana-shaped	zone	exceeding	37,500	volts	per	meter	could
cover	most	of	the	U.S.	East	Coast.	Reprinted	from	U.S.	Army	Report	AD-A278230
(unclassified)	with	colors	added.
Third,	the	potential	of	nuclear	winter	wasn’t	realized	until	four	decades	in,
after	we’d	deployed	63,000	hydrogen	bombs—oops!	Regardless	of	whose	cities
burned,	massive	amounts	of	smoke	reaching	the	upper	troposphere	might	spread
around	the	globe,	blocking	out	enough	sunlight	to	transform	summers	into
winters,	much	like	when	an	asteroid	or	supervolcano	caused	a	mass	extinction	in
the	past.	When	the	alarm	was	sounded	by	both	U.S.	and	Soviet	scientists	in	the
1980s,	this	contributed	to	the	decision	of	Ronald	Reagan	and	Mikhail	Gorbachev
to	start	slashing	stockpiles.
10
	Unfortunately,	more	accurate	calculations	have

10
painted	an	even	gloomier	picture:	
figure	5.3
	shows	cooling	by	about	20°	Celsius
(36°	Fahrenheit)	in	much	of	the	core	farming	regions	of	the	United	States,
Europe,	Russia	and	China	(and	by	35°C	in	some	parts	of	Russia)	for	the	first	two
summers,	and	about	half	that	even	a	full	decade	later.
*4
	What	does	that	mean	in

plain	English?	One	doesn’t	need	much	farming	experience	to	conclude	that	near-
freezing	summer	temperatures	for	years	would	eliminate	most	of	our	food
production.	It’s	hard	to	predict	exactly	what	would	happen	after	thousands	of
Earth’s	largest	cities	are	reduced	to	rubble	and	global	infrastructure	collapses,
but	whatever	small	fraction	of	all	humans	don’t	succumb	to	starvation,
hypothermia	or	disease	would	need	to	cope	with	roving	armed	gangs	desperate
for	food.

Figure	5.3:	Average	cooling	(in	°C)	during	the	first	two	summers	after	a	full-scale	nuclear	war
between	the	United	States	and	Russia.	Reproduced	with	permission	from	Alan	Robock.
11
I’ve	gone	into	such	detail	on	global	nuclear	war	to	drive	home	the	crucial
point	that	no	reasonable	world	leader	would	want	it,	yet	it	might	nonetheless
happen	by	accident.	This	means	that	we	can’t	trust	our	fellow	humans	never	to
commit	omnicide:	nobody	wanting	it	isn’t	necessarily	enough	to	prevent	it.

Doomsday	Devices
So	could	we	humans	actually	pull	off	omnicide?	Even	if	a	global	nuclear	war
may	kill	off	90%	of	all	humans,	most	scientists	guess	that	it	wouldn’t	kill	100%
and	therefore	wouldn’t	drive	us	extinct.	On	the	other	hand,	the	story	of	nuclear
radiation,	nuclear	EMP	and	nuclear	winter	all	demonstrate	that	the	greatest
hazards	may	be	ones	we	haven’t	even	thought	of	yet.	It’s	incredibly	difficult	to
foresee	all	aspects	of	the	aftermath,	and	how	nuclear	winter,	infrastructure
collapse,	elevated	mutation	levels	and	desperate	armed	hordes	might	interact
with	other	problems	such	as	new	pandemics,	ecosystem	collapse	and	effects	we
haven’t	yet	imagined.	My	personal	assessment	is	therefore	that	although	the
probability	of	a	nuclear	war	tomorrow	triggering	human	extinction	isn’t	large,
we	can’t	confidently	conclude	that	it’s	zero	either.
Omnicide	odds	increase	if	we	upgrade	today’s	nuclear	weapons	into	a
deliberate	doomsday	device.	Introduced	by	RAND	strategist	Herman	Kahn	in

1960	and	popularized	in	Stanley	Kubrick’s	film	
Dr.	Strangelove,
	a	doomsday
device	takes	the	paradigm	of	mutually	assured	destruction	to	its	ultimate
conclusion.	It’s	the	perfect	deterrent:	a	machine	that	automatically	retaliates
against	any	enemy	attack	by	killing	all	of	humanity.
One	candidate	for	the	doomsday	device	is	a	huge	underground	cache	of	so-
called	
salted	nukes,
	preferably	humongous	hydrogen	bombs	surrounded	by
massive	amounts	of	cobalt.	Physicist	Leo	Szilard	argued	already	in	1950	that	this

	a	doomsday
called	
could	kill	everyone	on	Earth:	the	hydrogen	bomb	explosions	would	render	the
cobalt	radioactive	and	blow	it	into	the	stratosphere,	and	its	five-year	half-life	is
long	enough	for	it	to	settle	all	across	Earth	(especially	if	twin	doomsday	devices
were	placed	in	opposite	hemispheres),	but	short	enough	to	cause	lethal	radiation
intensity.	Media	reports	suggest	that	cobalt	bombs	are	now	being	built	for	the
first	time.	Omnicidal	opportunities	could	be	bolstered	by	adding	bombs

optimized	for	nuclear	winter	creation	by	maximizing	long-lived	aerosols	in	the
stratosphere.	A	major	selling	point	of	a	doomsday	device	is	that	it’s	much
cheaper	than	a	conventional	nuclear	deterrent:	since	the	bombs	don’t	need	to	be
launched,	there’s	no	need	for	expensive	missile	systems,	and	the	bombs
themselves	are	cheaper	to	build	since	they	need	not	be	light	and	compact	enough
to	fit	into	missiles.
Another	possibility	is	the	future	discovery	of	a	biological	doomsday	device:	a

custom-designed	bacterium	or	virus	that	kills	all	humans.	If	its	transmissibility
were	high	enough	and	its	incubation	period	long	enough,	essentially	everybody
could	catch	it	before	they	realized	its	existence	and	took	countermeasures.
There’s	a	military	argument	for	building	such	a	bioweapon	even	if	it	can’t	kill
everybody:	the	most	effective	doomsday	device	is	one	that	combines	nuclear,
biological	and	other	weapons	to	maximize	the	chances	of	deterring	the	enemy.

AI	Weapons
A	third	technological	route	to	omnicide	may	involve	relatively	dumb	AI
weapons.	Suppose	a	superpower	builds	billions	of	those	bumblebee-sized	attack
drones	from	chapter	3	and	uses	them	to	kill	anyone	except	their	own	citizens	and
allies,	identified	remotely	by	a	radio-frequency	ID	tag	just	as	most	of	today’s
supermarket	products.	These	tags	could	be	distributed	to	all	citizens	to	be	worn
on	bracelets	or	as	transdermal	implants,	as	in	the	totalitarianism	section.	This
would	probably	spur	an	opposing	superpower	to	build	something	analogous.
When	war	accidentally	breaks	out,	all	humans	would	be	killed,	even	unaffiliated
remote	tribes,	because	nobody	would	be	wearing	both	kinds	of	ID	tag.
Combining	this	with	a	nuclear	and	biological	doomsday	device	would	further
improve	chances	of	successful	omnicide.

What	Do	
You
	Want?
You	began	this	chapter	pondering	where	you	want	the	current	AGI	race	to	lead.
Now	that	we’ve	explored	a	broad	range	of	scenarios	together,	which	ones	appeal
to	you	and	which	ones	do	you	think	we	should	try	hard	to	avoid?	Do	you	have	a
clear	favorite?	Please	let	me	and	fellow	readers	know	at	
,	and
join	the	discussion!
The	scenarios	we’ve	covered	obviously	shouldn’t	be	viewed	as	a	complete
list,	and	many	are	thin	on	details,	but	I’ve	tried	hard	to	be	inclusive,	spanning	the
full	spectrum	from	high-tech	to	low-tech	to	no-tech	and	describing	all	the	central
hopes	and	fears	expressed	in	the	literature.
One	of	the	most	fun	parts	of	writing	this	book	has	been	hearing	what	my
friends	and	colleagues	think	of	these	scenarios,	and	I’ve	been	amused	to	learn
that	there’s	no	consensus	whatsoever.	The	one	thing	everybody	agrees	on	is	that
the	choices	are	more	subtle	than	they	may	initially	seem.	People	who	like	any

one	scenario	tend	to	simultaneously	find	some	aspect(s)	of	it	bothersome.	To	me,
this	means	that	we	humans	need	to	continue	and	deepen	this	conversation	about
our	future	goals,	so	that	we	know	in	which	direction	to	steer.	The	future	potential
for	life	in	our	cosmos	is	awe-inspiringly	grand,	so	let’s	not	squander	it	by
drifting	like	a	rudderless	ship,	clueless	about	where	we	want	to	go!
Just	how	grand	is	this	future	potential?	No	matter	how	advanced	our
technology	gets,	the	ability	for	Life	3.0	to	improve	and	spread	through	our

cosmos	will	be	limited	by	the	laws	of	physics—what	are	these	ultimate	limits,
during	the	billions	of	years	to	come?	Is	our	Universe	teeming	with
extraterrestrial	life	right	now,	or	are	we	alone?	What	happens	if	different
expanding	cosmic	civilizations	meet?	We’ll	tackle	these	fascinating	questions	in
the	next	chapter.

•
The	current	race	toward	AGI	can	end	in	a	fascinatingly	broad	range	of	aftermath
scenarios	for	upcoming	millennia.
•
Superintelligence	can	peacefully	coexist	with	humans	either	because	it’s	forced	to
(enslaved-god	scenario)	or	because	it’s	“friendly	AI”	that	wants	to	(libertarian-
utopia,	protector-god,	benevolent-dictator	and	zookeeper	scenarios).
•
Superintelligence	can	be	prevented	by	an	AI	(gatekeeper	scenario)	or	by	humans
(1984	scenario),	by	deliberately	forgetting	the	technology	(reversion	scenario)	or	by
lack	of	incentives	to	build	it	(egalitarian-utopia	scenario).
•
Humanity	can	go	extinct	and	get	replaced	by	AIs	(conqueror	and	descendant
scenarios)	or	by	nothing	(self-destruction	scenario).
•
There’s	absolutely	no	consensus	on	which,	if	any,	of	these	scenarios	are	desirable,
and	all	involve	objectionable	elements.	This	makes	it	all	the	more	important	to
continue	and	deepen	the	conversation	around	our	future	goals,	so	that	we	don’t

•
•
inadvertently	drift	or	steer	in	an	unfortunate	direction.
*1
	
This	idea	dates	back	to	Saint	Augustine,	who	wrote	that	“if	a	thing	is	not	diminished	by	being	shared
with	others,	it	is	not	rightly	owned	if	it	is	only	owned	and	not	shared.”
*2
	
This	idea	was	first	suggested	to	me	by	my	friend	and	colleague	Anthony	Aguirre.
*3
	
The	renowned	cosmologist	Fred	Hoyle	explored	a	related	scenario	with	a	different	twist	in	the	British
TV	series	
A	for	Andromeda
.
*4
	
Injecting	carbon	into	the	atmosphere	can	cause	two	kinds	of	climate	change:	warming	from	carbon

*1
	
*2
	
*3
	
TV	series	
.
*4
	
dioxide	or	cooling	from	smoke	and	soot.	It’s	not	only	the	first	kind	that’s	occasionally	dismissed	without
scientific	evidence:	I’m	sometimes	told	that	nuclear	winter	has	been	debunked	and	is	virtually	impossible.
I	always	respond	by	asking	for	a	reference	to	a	peer-reviewed	scientific	paper	making	such	strong	claims
and,	so	far,	there	seem	to	be	none	whatsoever.	Although	there	are	great	uncertainties	that	warrant	further

research,	especially	related	to	how	much	smoke	gets	produced	and	how	high	up	it	rises,	there’s	in	my
scientific	opinion	no	current	basis	for	dismissing	the	nuclear	winter	risk.

Chapter	6
Our	Cosmic	Endowment:	The	Next	Billion
Years	and	Beyond
Our	speculation	ends	in	a	supercivilization,	the	synthesis	of	all	solar-system	life,
constantly	improving	and	extending	itself,	spreading	outward	from	the	sun,	converting
nonlife	into	mind.
To	me,	the	most	inspiring	scientific	discovery	ever	is	that	we’ve	dramatically
underestimated	life’s	future	potential.	Our	dreams	and	aspirations	need	not	be
limited	to	century-long	life	spans	marred	by	disease,	poverty	and	confusion.
Rather,	aided	by	technology,	life	has	the	potential	to	flourish	for	billions	of
years,	not	merely	here	in	our	Solar	System,	but	also	throughout	a	cosmos	far
more	grand	and	inspiring	than	our	ancestors	imagined.	Not	even	the	sky	is	the
limit.
This	is	exciting	news	for	a	species	that	has	been	inspired	by	pushing	limits
throughout	the	ages.	Olympic	games	celebrate	pushing	the	limits	of	strength,
speed,	agility	and	endurance.	Science	celebrates	pushing	the	limits	of	knowledge

limit.
and	understanding.	Literature	and	art	celebrate	pushing	the	limits	of	creating
beautiful	or	life-enriching	experiences.	Many	people,	organizations	and	nations
celebrate	increasing	resources,	territory	and	longevity.	Given	our	human
obsession	with	limits,	it’s	fitting	that	the	best-selling	copyrighted	book	of	all
time	is	
The	Guinness	Book	of
	
World	Records
.
So	if	our	old	perceived	limits	of	life	can	be	shattered	by	technology,	what	are

the	
ultimate
	limits?	How	much	of	our	cosmos	can	come	alive?	How	far	can	life
reach	and	how	long	can	it	last?	How	much	matter	can	life	make	use	of,	and	how
much	energy,	information	and	computation	can	it	extract?	These	ultimate	limits
are	set	not	by	our	understanding,	but	by	the	laws	of	physics.	This,	ironically,
makes	it	in	some	ways	easier	to	analyze	the	long-term	future	of	life	than	the
short-term	future.
If	our	13.8-billion-year	cosmic	history	were	compressed	into	a	week,	then	the
10,000-year	drama	of	the	last	two	chapters	would	be	over	in	less	than	half	a
second.	This	means	that	although	we	cannot	predict	if	and	how	an	intelligence
explosion	will	unfold	and	what	its	immediate	aftermath	will	be	like,	all	this
turmoil	is	merely	a	brief	flash	in	cosmic	history	whose	details	don’t	affect	life’s
ultimate	limits.	If	the	post-explosion	life	is	as	obsessed	as	today’s	humans	are
with	pushing	limits,	then	it	will	develop	technology	to	actually	
reach
	these	limits

reach
—because	it	can.	In	this	chapter,	we’ll	explore	what	these	limits	are,	thus	getting
a	glimpse	of	what	the	long-term	future	of	life	may	be	like.	Since	these	limits	are
based	on	our	current	understanding	of	physics,	they	should	be	viewed	as	a	lower
bound	on	the	possibilities:	future	scientific	discoveries	may	present	opportunities
to	do	even	better.
But	do	we	really	know	that	future	life	will	be	so	ambitious?	No,	we	don’t:
perhaps	it	will	become	as	complacent	as	a	heroin	addict	or	a	couch	potato	merely
watching	endless	reruns	of

watching	endless	reruns	of	
Keeping	Up	with	the	Kardashians
.	However,	there	is
reason	to	suspect	that	ambition	is	a	rather	generic	trait	of	advanced	life.	Almost
regardless	of	what	it’s	trying	to	maximize,	be	it	intelligence,	longevity,
knowledge	or	interesting	experiences,	it	will	need	resources.	It	therefore	has	an
incentive	to	push	its	technology	to	the	ultimate	limits,	to	make	the	most	of	the
resources	it	has.	After	this,	the	only	way	to	further	improve	is	to	acquire	more
resources,	by	expanding	into	ever-larger	regions	of	the	cosmos.

Also,	life	may	independently	originate	in	multiple	places	in	our	cosmos.	In
that	case,	unambitious	civilizations	simply	become	cosmically	irrelevant,	with
ever-larger	parts	of	the	cosmic	endowment	ultimately	being	taken	over	by	the
most	ambitious	life	forms.	Natural	selection	therefore	plays	out	on	a	cosmic
scale	and,	after	a	while,	almost	all	life	that	exists	will	be	ambitious	life.	In
summary,	if	we’re	interested	in	the	extent	to	which	our	cosmos	can	ultimately

come	alive,	we	should	study	the	limits	of	ambition	that	are	imposed	by	the	laws
of	physics.	Let’s	do	this!	Let’s	first	explore	the	limits	of	what	can	be	done	with
the	resources	(matter,	energy,	etc.)	that	we	have	in	our	Solar	System,	then	turn	to
how	to	get	more	resources	through	cosmic	exploration	and	settlement.

Whereas	today’s	supermarkets	and	commodity	exchanges	sell	tens	of	thousands
of	items	we	might	call	“resources,”	future	life	that’s	reached	the	technological
limit	needs	mainly	one	fundamental	resource:	so-called	
baryonic	matter,
meaning	anything	made	up	of	atoms	or	their	constituents	(quarks	and	electrons).
Whatever	form	this	matter	is	in,	advanced	technology	can	rearrange	it	into	any
desired	substances	or	objects,	including	power	plants,	computers	and	advanced
life	forms.	Let’s	therefore	begin	by	examining	the	limits	on	the	energy	that
powers	advanced	life	and	the	information	processing	that	enables	it	to	think.

Building	Dyson	Spheres
When	it	comes	to	the	future	of	life,	one	of	the	most	hopeful	visionaries	is
Freeman	Dyson.	I’ve	had	the	honor	and	pleasure	of	knowing	him	for	the	past
two	decades,	but	when	I	first	met	him,	I	felt	nervous.	I	was	a	junior	postdoc
chowing	away	with	my	friends	in	the	lunchroom	of	the	Institute	for	Advanced
Study	in	Princeton,	and	out	of	the	blue,	this	world-famous	physicist	who	used	to
hang	out	with	Einstein	and	Gödel	came	up	and	introduced	himself,	asking	if	he
could	join	us!	He	quickly	put	me	at	ease,	however,	by	explaining	that	he
preferred	eating	lunch	with	young	folks	over	stuffy	old	professors.	Even	though
he’s	ninety-three	as	I	type	these	words,	Freeman	is	still	younger	in	spirit	than
most	people	I	know,	and	the	mischievous	boyish	glint	in	his	eyes	reveals	that	he
couldn’t	care	less	about	formalities,	academic	hierarchies	or	conventional
wisdom.	The	bolder	the	idea,	the	more	excited	he	gets.
When	we	talked	about	energy	use,	he	scoffed	at	how	unambitious	we	humans

were,	pointing	out	that	we	could	meet	all	our	current	global	energy	needs	by
harvesting	the	sunlight	striking	an	area	smaller	than	0.5%	of	the	Sahara	desert.
But	why	stop	there?	Why	even	stop	at	capturing	all	the	sunlight	striking	Earth,
letting	most	of	it	get	wastefully	beamed	into	empty	space?	Why	not	simply	put
all
	the	Sun’s	energy	output	to	use	for	life?
Inspired	by	Olaf	Stapledon’s	1937	sci-fi	classic	
Star	Maker,
	with	rings	of
artificial	worlds	orbiting	their	parent	star,	Freeman	Dyson	published	a
description	in	1960	of	what	became	known	as	a	
Dyson	sphere.
1

all
1
	Freeman’s	idea
was	to	rearrange	Jupiter	into	a	biosphere	in	the	form	of	a	spherical	shell
surrounding	the	Sun,	where	our	descendants	could	flourish,	enjoying	100	billion
times	more	biomass	and	a	trillion	times	more	energy	than	humanity	uses	today.
2
He	argued	that	this	was	the	natural	next	step:	“One	should	expect	that,	within	a
few	thousand	years	of	its	entering	the	stage	of	industrial	development,	any
intelligent	species	should	be	found	occupying	an	artificial	biosphere	which

1
2
completely	surrounds	its	parent	star.”	If	you	lived	on	the	inside	of	a	Dyson
sphere,	there	would	be	no	nights:	you’d	always	see	the	Sun	straight	overhead,
and	all	across	the	sky,	you’d	see	sunlight	reflecting	off	the	rest	of	the	biosphere,
just	as	you	can	nowadays	see	sunlight	reflecting	off	the	Moon	during	the	day.	If
you	wanted	to	see	stars,	you’d	simply	go	“upstairs”	and	peer	out	at	the	cosmos
from	the	outside	of	the	Dyson	sphere.

A	low-tech	way	to	build	a	partial	Dyson	sphere	is	to	place	a	ring	of	habitats	in
circular	orbit	around	the	Sun.	To	completely	surround	the	Sun,	you	could	add
rings	orbiting	it	around	different	axes	at	slightly	different	distances,	to	avoid
collisions.	To	avoid	the	nuisance	that	these	fast-moving	rings	couldn’t	be
connected	to	one	another,	complicating	transportation	and	communication,	one
could	instead	build	a	monolithic	stationary	Dyson	sphere	where	the	Sun’s	inward
gravitational	pull	is	balanced	by	the	outward	pressure	from	the	Sun’s	radiation—
an	idea	pioneered	by	Robert	L.	Forward	and	by	Colin	McInnes.	The	sphere	can
be	built	by	gradually	adding	more	“statites”:	stationary	satellites	that	counteract
the	Sun’s	gravity	with	radiation	pressure	rather	than	centrifugal	forces.	Both	of
these	forces	drop	off	with	the	square	of	the	distance	to	the	Sun,	which	means	that
if	they	can	be	balanced	at	one	distance	from	the	Sun,	they’ll	conveniently	be

balanced	at	any	other	distance	as	well,	allowing	freedom	to	park	anywhere	in	our
Solar	System.	Statites	need	to	be	extremely	lightweight	sheets,	weighing	only
0.77	grams	per	square	meter,	which	is	about	100	times	less	than	paper,	but	this	is
unlikely	to	be	a	showstopper.	For	example,	a	sheet	of	graphene	(a	single	layer	of
carbon	atoms	in	a	hexagonal	pattern	resembling	chicken	wire)	weighs	a
thousand	times	less	than	that	limit.	If	the	Dyson	sphere	is	built	to	reflect	rather

than	absorb	most	of	the	sunlight,	then	the	total	intensity	of	light	bouncing	around
within	it	will	be	dramatically	increased,	further	boosting	the	radiation	pressure
and	the	amount	of	mass	that	can	be	supported	in	the	sphere.	Many	other	stars
have	a	thousandfold	and	even	a	millionfold	greater	luminosity	than	our	Sun,	and
are	therefore	able	to	support	correspondingly	heavier	stationary	Dyson	spheres.
If	a	much	heavier	rigid	Dyson	sphere	is	desired	here	in	our	Solar	System,	then

resisting	the	Sun’s	gravity	will	require	ultra-strong	materials	that	can	withstand
pressures	tens	of	thousands	of	times	greater	than	those	at	the	base	of	the	world’s
tallest	skyscrapers,	without	liquefying	or	buckling.	To	be	long-lived,	a	Dyson
sphere	would	need	to	be	dynamic	and	intelligent,	constantly	fine-tuning	its
position	and	shape	in	response	to	disturbances	and	occasionally	opening	up	large
holes	to	let	annoying	asteroids	and	comets	pass	through	without	incident.

Alternatively,	a	detect-and-deflect	system	could	be	used	to	handle	such	system
intruders,	optionally	disassembling	them	and	putting	their	matter	to	better	use.
For	today’s	humans,	life	on	or	in	a	Dyson	sphere	would	at	best	be	disorienting
and	at	worst	impossible,	but	that	need	not	stop	future	biological	or	non-
biological	life	forms	from	thriving	there.	The	orbiting	variant	would	offer
essentially	no	gravity	at	all,	and	if	you	walked	around	on	the	stationary	kind,	you

could	walk	only	on	the	outside	(facing	away	from	the	Sun)	without	falling	off,

with	gravity	about	ten	thousand	times	weaker	than	you’re	used	to.	You’d	have	no
magnetic	field	(unless	you	built	one)	shielding	you	from	dangerous	particles
from	the	Sun.	The	silver	lining	is	that	a	Dyson	sphere	the	size	of	Earth’s	current
orbit	would	give	us	about	500	million	times	more	surface	area	to	live	on.
If	more	Earth-like	human	habitats	are	desired,	the	good	news	is	that	they’re
much	easier	to	build	than	a	Dyson	sphere.	For	example,	
figures	6.1
	and	
6.2
	show
a	cylindrical	habitat	design	pioneered	by	the	American	physicist	Gerard	K.
O’Neill,	which	supports	artificial	gravity,	cosmic	ray	shielding,	a	twenty-four-
hour	day-night	cycle,	and	Earth-like	atmosphere	and	ecosystems.	Such	habitats
could	orbit	freely	inside	a	Dyson	sphere,	or	modified	variants	could	be	attached
outside	it.

Figure	6.1:	A	pair	of	counterrotating	O’Neill	cylinders	can	provide	comfortable	Earth-like	human
habitats	if	they	orbit	the	Sun	in	such	a	way	that	they	always	point	straight	at	it.	The	centrifugal
force	from	their	rotation	provides	artificial	gravity,	and	three	foldable	mirrors	beam	sunlight
inside	on	a	24-hour	day-night	cycle.	The	smaller	habitats	arranged	in	a	ring	are	specialized	for
agriculture.	Image	courtesy	of	Rick	Guidice/NASA.

Building	Better	Power	Plants
Although	Dyson	spheres	are	energy	efficient	by	today’s	engineering	standards,
they	come	nowhere	near	pushing	the	limits	set	by	the	laws	of	physics.	Einstein
taught	us	that	if	we	could	convert	mass	to	energy	with	100%	efficiency,
*1
	then
an	amount	of	mass	
m
	would	give	us	an	amount	of	energy	
E
	given	by	his	famous
formula	
E
	=	
mc
2
,	where	
c
	is	the	speed	of	light.	This	means	that	since	
c
	is	huge,	a
small	amount	of	mass	can	produce	a	humongous	amount	of	energy.	If	we	had	an
abundant	supply	of	antimatter	(which	we	don’t),	then	a	100%	efficient	power
plant	would	be	easy	to	make:	simply	pouring	a	teaspoonful	of	anti-water	into
regular	water	would	unleash	the	energy	equivalent	to	200,000	tons	of	TNT,	the
yield	of	a	typical	hydrogen	bomb—enough	to	power	the	world’s	entire	energy
needs	for	about	seven	minutes.

Figure	6.2:	Interior	view	of	one	of	the	O’Neill	cylinders	from	the	previous	figure.	If	its	diameter
is	6.4	kilometers	and	rotates	once	every	2	minutes,	people	on	the	surface	will	experience	the
same	apparent	gravity	as	on	Earth.	The	Sun	is	behind	you,	but	appears	above	because	of	a	mirror
outside	the	cylinder	that	folds	away	at	night.	Airtight	windows	keep	the	atmosphere	from
escaping	the	cylinder.	Image	courtesy	of	Rick	Guidice/NASA.
In	contrast,	our	most	common	ways	of	generating	energy	today	are	woefully
inefficient,	as	summarized	in	
table	6.1
	and	
figure	6.3
.	Digesting	a	candy	bar	is
merely	0
.
00000001%	efficient,	in	the	sense	that	it	releases	a	mere	ten-trillionth
of	the	energy	
mc
2
	that	it	contains.	If	your	stomach	were	even	0.001%	efficient,
then	you’d	only	need	to	eat	a	single	meal	for	the	rest	of	your	life.	Compared	to
eating,	the	burning	of	coal	and	gasoline	are	merely	3	and	5	times	more	efficient,
respectively.	Today’s	nuclear	reactors	do	dramatically	better	by	splitting	uranium

table	6.1
	and	
figure	6.3
merely	0
.
mc
2
atoms	through	fission,	but	still	fail	to	extract	more	than	0.08%	of	their	energy.
The	nuclear	reactor	in	the	core	of	the	Sun	is	an	order	of	magnitude	more	efficient

than	those	we’ve	built,	extracting	0.7%	of	the	energy	from	hydrogen	by	fusing	it
into	helium.	However,	even	if	we	enclose	the	Sun	in	a	perfect	Dyson	sphere,
we’ll	never	convert	more	than	about	0
.
08%	of	the	Sun’s	mass	to	energy	we	can
use,	because	once	the	Sun	has	consumed	about	about	a	tenth	of	its	hydrogen
fuel,	it	will	end	its	lifetime	as	a	normal	star,	expand	into	a	red	giant,	and	begin	to
die.	Things	don’t	get	much	better	for	other	stars	either:	the	fraction	of	their
hydrogen	consumed	during	the	main	lifetime	ranges	from	about	4%	for	very
small	stars	to	about	12%	for	the	largest	ones.	If	we	perfect	an	artificial	fusion
reactor	that	would	let	us	fuse	100%	of	all	hydrogen	at	our	disposal,	we’d	still	be
stuck	at	that	embarrassingly	low	0.7%	efficiency	of	the	fusion	process.	How	can
we	do	better?
Method
Efficiency
Digesting	candy	bar
0.00000001%
Burning	coal
0.00000003%
Burning	gasoline
0.00000005%
Fission	of	uranium-235
0.08%
Using	Dyson	sphere	until	Sun	dies
0.08%
Fusion	of	hydrogen	to	helium
0.7%

Method
0.08%
0.08%
0.7%
Spinning	black	hole	engine
29%
Dyson	sphere	around	quasar
42%
Sphalerizer
50%?
Black	hole	evaporation
90%
Table	6.1:	Efficiency	of	converting	mass	into	usable	energy	relative	to	the	theoretical	limit	
E
	=	
mc
2
.	As
explained	in	the	text,	getting	90%	efficiency	from	feeding	black	holes	and	waiting	for	them	to	evaporate	is
unfortunately	too	slow	to	be	useful,	and	accelerating	the	process	dramatically	lowers	the	efficiency.

Figure	6.3:	Advanced	technology	can	extract	dramatically	more	energy	from	matter	than	we	get
by	eating	or	burning	it,	and	even	nuclear	fusion	extracts	140	times	less	energy	than	the	limits	set
by	the	laws	of	physics.	Power	plants	exploiting	sphalerons,	quasars	or	evaporating	black	holes
might	do	much	better.

Evaporating	Black	Holes
In	his	book	
A	Brief	History	of	Time,
	Stephen	Hawking	proposed	a	black	hole
power	plant.
*2
	This	may	sound	paradoxical	given	that	black	holes	were	long
believed	to	be	traps	that	nothing,	not	even	light,	could	ever	escape	from.
However,	Hawking	famously	calculated	that	quantum	gravity	effects	make	a
black	hole	act	like	a	hot	object—the	smaller,	the	hotter—that	gives	off	heat
radiation	now	known	as	
Hawking	radiation
.	This	means	that	the	black	hole
gradually	loses	energy	and	evaporates	away.	In	other	words,	whatever	matter
you	dump	into	the	black	hole	will	eventually	come	back	out	again	as	heat
radiation,	so	by	the	time	the	black	hole	has	completely	evaporated,	you’ve
converted	your	matter	to	radiation	with	nearly	100%	efficiency.
*3
A	problem	with	using	black	hole	evaporation	as	a	power	source	is	that,	unless
the	black	hole	is	much	smaller	than	an	atom	in	size,	it’s	an	excruciatingly	slow
process	that	takes	longer	than	the	present	age	of	our	Universe	and	radiates	less

*3
energy	than	a	candle.	The	power	produced	decreases	with	the	square	of	the	size
of	the	hole,	and	the	physicists	Louis	Crane	and	Shawn	Westmoreland	have
therefore	proposed	using	a	black	hole	about	a	thousand	times	smaller	than	a
proton,	weighing	about	as	much	as	the	largest-ever	seagoing	ship.
3
	Their	main
motivation	was	to	use	the	black	hole	engine	to	power	a	starship	(a	topic	to	which
we	return	below),	so	they	were	more	concerned	with	portability	than	efficiency
and	proposed	feeding	the	black	hole	with	laser	light,	causing	no	energy-to-matter

3
	Their	main
conversion	at	all.	Even	if	you	could	feed	it	with	matter	instead	of	radiation,
guaranteeing	high	efficiency	appears	difficult:	to	make	protons	enter	such	a
black	hole	a	thousandth	their	size,	they	would	have	to	be	fired	at	the	hole	with	a
machine	as	powerful	as	the	Large	Hadron	Collider,	augmenting	their	energy	
mc
2
with	at	least	a	thousand	times	more	kinetic	(motion)	energy.	Since	at	least	10%
of	that	kinetic	energy	would	be	lost	to	gravitons	when	the	black	hole	evaporates,

mc
2
we’d	therefore	be	putting	more	energy	into	the	black	hole	than	we’d	be	able	to
extract	and	put	to	work,	ending	up	with	negative	efficiency.	Further	confounding
the	prospects	of	a	black	hole	power	plant	is	that	we	still	lack	a	rigorous	theory	of
quantum	gravity	upon	which	to	base	our	calculations—but	this	uncertainty	could
of	course	also	mean	that	there	are	new	useful	quantum	gravity	effects	yet	to	be
discovered.

Figure	6.4:	Part	of	the	rotational	energy	of	a	spinning	black	hole	can	be	extracted	by	throwing	a
particle	A	near	the	black	hole	and	having	it	split	into	a	part	C	that	gets	eaten	and	a	part	B	that
escapes—with	more	energy	than	A	had	initially.

Spinning	Black	Holes
Fortunately,	there	are	other	ways	of	using	black	holes	as	power	plants	that	don’t
involve	quantum	gravity	or	other	poorly	understood	physics.	For	example,	many
existing	black	holes	spin	very	fast,	with	their	event	horizons	whirling	around
near	the	speed	of	light,	and	this	rotation	energy	can	be	extracted.	The	event
horizon	of	a	black	hole	is	the	region	from	which	not	even	light	can	escape,
because	the	gravitational	pull	is	too	powerful.	
Figure	6.4
	illustrates	how	outside
the	event	horizon,	a	spinning	black	hole	has	a	region	called	the	
ergosphere,
where	the	spinning	black	hole	drags	space	along	with	it	so	fast	that	it’s
impossible	for	a	particle	to	sit	still	and	not	get	dragged	along.	If	you	toss	an
object	into	the	ergosphere,	it	will	therefore	pick	up	speed	rotating	around	the
hole.	Unfortunately,	it	will	soon	get	eaten	up	by	the	black	hole,	forever
disappearing	through	the	event	horizon,	so	this	does	you	no	good	if	you’re	trying

Figure	6.4
to	extract	energy.	However,	Roger	Penrose	discovered	that	if	you	launch	the
object	at	a	clever	angle	and	make	it	split	into	two	pieces	as	
figure	6.4
	illustrates,
then	you	can	arrange	for	only	one	piece	to	get	eaten	while	the	other	escapes	the
black	hole	with	more	energy	than	you	started	with.	In	other	words,	you’ve
successfully	converted	some	of	the	rotational	energy	of	the	black	hole	into	useful
energy	that	you	can	put	to	work.	By	repeating	this	process	many	times,	you	can
milk	the	black	hole	of	
all

figure	6.4
all
	its	rotational	energy	so	that	it	stops	spinning	and	its
ergosphere	disappears.	If	the	initial	black	hole	was	spinning	as	fast	as	nature
allows,	with	its	event	horizon	moving	essentially	at	the	speed	of	light,	this
strategy	allows	you	to	convert	29%	of	its	mass	into	energy.	There	is	still
significant	uncertainty	about	how	fast	the	black	holes	in	our	night	sky	spin,	but
many	of	the	best-studied	ones	appear	to	spin	quite	fast:	between	30%	and	100%

all
of	the	maximum	allowed.	The	monster	black	hole	in	the	middle	of	our	Galaxy
(which	weighs	four	million	times	as	much	as	our	Sun)	appears	to	spin,	so	even	if
only	10%	of	its	mass	could	be	converted	to	useful	energy,	that	would	deliver	the
same	as	400,000	suns	converted	to	energy	with	100%	efficiency,	or	about	as
much	energy	as	we’d	get	from	Dyson	spheres	around	500	million	suns	over
billions	of	years.

Quasars
Another	interesting	strategy	is	to	extract	energy	not	from	the	black	hole	itself,
but	from	matter	falling	into	it.	Nature	has	already	found	a	way	of	doing	this	all
on	its	own:	the	quasar.	As	gas	swirls	even	closer	to	a	black	hole,	forming	a
pizza-shaped	disk	whose	innermost	parts	gradually	get	gobbled	up,	it	gets
extremely	hot	and	gives	off	copious	amounts	of	radiation.	As	gas	falls	downward
toward	the	hole,	it	speeds	up,	converting	its	gravitational	potential	energy	into
motion	energy,	just	as	a	skydiver	does.	The	motion	gets	progressively	messier	as
complicated	turbulence	converts	the	coordinated	motion	of	the	gas	blob	into
random	motion	on	ever-smaller	scales,	until	individual	atoms	begin	colliding
with	each	other	at	high	speeds—having	such	random	motion	is	precisely	what	it
means	to	be	hot,	and	these	violent	collisions	convert	motion	energy	into
radiation.	By	building	a	Dyson	sphere	around	the	entire	black	hole,	at	a	safe

distance,	this	radiation	energy	can	be	captured	and	put	to	use.	The	faster	the
black	hole	spins,	the	more	efficient	this	process	gets,	with	a	maximally	spinning
black	hole	delivering	energy	at	a	whopping	42%	efficiency.
*4
	For	black	holes
weighing	about	as	much	as	a	star,	most	of	the	energy	comes	out	as	X-rays,
whereas	for	the	supermassive	kind	found	in	the	centers	of	galaxies,	much	of	it
emerges	somewhere	in	the	range	of	infrared,	visible	and	ultraviolet	light.
Once	you’ve	run	out	of	fuel	to	feed	your	black	hole,	you	can	switch	to

*4
extracting	its	rotational	energy	as	we	discussed	above.
*5
	Indeed,	nature	has
already	found	a	way	of	partially	doing	that	as	well,	boosting	the	radiation	from
accreted	gas	through	a	magnetic	process	known	as	the	Blandford-Znajek
mechanism.	It	may	well	be	possible	to	use	technology	to	further	improve	the
energy	extraction	efficiency	beyond	42%	by	clever	use	of	magnetic	fields	or
other	ingredients.

Sphalerons
There	is	another	known	way	to	convert	matter	into	energy	that	doesn’t	involve
black	holes	at	all:	the	
sphaleron
	process.	It	can	destroy	quarks	and	turn	them	into
leptons:	electrons,	their	heavier	cousins	the	muon	and	tau	particles,	neutrinos	or
their	antiparticles.
4
	As	illustrated	in	
figure	6.5
,	the	standard	model	of	particle
physics	predicts	that	nine	quarks	with	appropriate	flavor	and	spin	can	come
together	and	transform	into	three	leptons	through	an	intermediate	state	called	a
sphaleron.	Because	the	input	weighs	more	than	the	output,	the	mass	difference
gets	converted	into	energy	according	to	Einstein’s	
E
	=	
mc
2
	formula.
Future	intelligent	life	might	therefore	be	able	to	build	what	I’ll	call	a
sphalerizer:
	an	energy	generator	acting	like	a	diesel	engine	on	steroids.	A
traditional	diesel	engine	compresses	a	mixture	of	air	and	diesel	oil	until	the
temperature	gets	high	enough	for	it	to	spontaneously	ignite	and	burn,	after	which

E
	=	
mc
2
	formula.
the	hot	mixture	re-expands	and	does	useful	work	in	the	process,	say	pushing	a
piston.	The	carbon	dioxide	and	other	combustion	gases	weigh	about
0.00000005%	less	than	what	was	in	the	piston	initially,	and	this	mass	difference
turns	into	the	heat	energy	driving	the	engine.	A	sphalerizer	would	compress
ordinary	matter	to	a	couple	of	quadrillion	degrees,	and	then	let	it	re-expand	and
cool	once	the	sphalerons	had	done	their	thing.
*6
	We	already	know	the	result	of
this	experiment,	because	our	early	Universe	performed	it	for	us	about	13.8

*6
billion	years	ago,	when	it	was	that	hot:	almost	100%	of	the	matter	gets	converted
into	energy,	with	less	than	a	billionth	of	the	particles	left	over	being	the	stuff	that
ordinary	matter	is	made	of:	quarks	and	electrons.	So	it’s	just	like	a	diesel	engine,
except	over	a	billion	times	more	efficient!	Another	advantage	is	that	you	don’t
need	to	be	finicky	about	what	to	fuel	it	with—it	works	with	anything	made	of
quarks,	meaning	any	normal	matter	at	all.

Figure	6.5:	According	to	the	standard	model	of	particle	physics,	nine	quarks	with	appropriate
flavor	and	spin	can	come	together	and	transform	into	three	leptons	through	an	intermediate	state
called	a	sphaleron.	The	combined	mass	of	the	quarks	(together	with	the	energy	of	the	gluon
particles	that	accompanied	them)	is	much	greater	than	the	mass	of	the	leptons,	so	this	process
will	release	energy,	indicated	by	flashes.
Because	of	these	high-temperature	processes,	our	baby	Universe	produced
over	a	trillion	times	more	radiation	(photons	and	neutrinos)	than	matter	(quarks
and	electrons	that	later	clumped	into	atoms).	During	the	13.8	billion	years	since
then,	a	great	segregation	took	place,	where	atoms	became	concentrated	into
galaxies,	stars	and	planets,	while	most	photons	stayed	in	intergalactic	space,
forming	the	cosmic	microwave	background	radiation	that	has	been	used	to	make
baby	pictures	of	our	Universe.	Any	advanced	life	form	living	in	a	galaxy	or

other	matter	concentration	can	therefore	turn	most	of	its	available	matter	back
into	energy,	rebooting	the	matter	percentage	down	to	the	same	tiny	value	that
emerged	from	our	early	Universe	by	briefly	re-creating	those	hot	dense
conditions	inside	a	sphalerizer.

To	figure	out	how	efficient	an	actual	sphalerizer	would	be,	one	needs	to	work
out	key	practical	details:	for	example,	how	large	does	it	need	to	be	to	prevent	a
significant	fraction	of	the	photons	and	neutrinos	from	leaking	out	during	the
compression	stage?	What	we	can	say	for	sure,	however,	is	that	the	energy
prospects	for	the	future	of	life	are	dramatically	better	than	our	current
technology	allows.	We	haven’t	even	managed	to	build	a	fusion	reactor,	yet	future
technology	should	be	able	to	do	ten	and	perhaps	even	a	hundred	times	better.

Building	Better	Computers
If	eating	dinner	is	10	billion	times	worse	than	the	physical	limit	on	energy
efficiency,	then	how	efficient	are	today’s	computers?	Even	worse	than	that
dinner,	as	we’ll	now	see.
I	often	introduce	my	friend	and	colleague	Seth	Lloyd	as	the	only	person	at
MIT	who’s	arguably	as	crazy	as	I	am.	After	doing	pioneering	work	on	quantum
computers,	he	went	on	to	write	a	book	arguing	that	our	entire	Universe	is	a
quantum	computer.	We	often	grab	beer	after	work,	and	I’ve	yet	to	discover	a
topic	that	he	doesn’t	have	something	interesting	to	say	about.	For	example,	as	I
mentioned	in	chapter	2,	he	has	lots	to	say	about	the	ultimate	limits	of	computing.
In	a	famous	2000	paper,	he	showed	that	computing	speed	is	limited	by	energy:
performing	an	elementary	logical	operation	in	time	
T
	requires	an	average	energy
of	
E
	=	
h
⁄
4
T,
	where	
h
	is	the	fundamental	physics	quantity	known	as	Planck’s
constant.	This	means	that	a	1	kg	computer	can	perform	at	most	5	×	10
50

T
of	
E
	=	
h
⁄
4
T,
	where	
h
50
operations	per	second—that’s	a	whopping	36	orders	of	magnitude	more	than	the
computer	on	which	I’m	typing	these	words.	We’ll	get	there	in	a	couple	of
centuries	if	computational	power	keeps	doubling	every	couple	of	years,	as	we
explored	in	chapter	2.	He	also	showed	that	a	1	kg	computer	can	store	at	most
10
31
	bits,	which	is	about	a	billion	billion	times	better	than	my	laptop.
Seth	is	the	first	to	admit	that	actually	attaining	these	limits	may	be	challenging
even	for	superintelligent	life,	since	the	memory	of	that	1	kg	ultimate	“computer”

10
31
would	resemble	a	thermonuclear	explosion	or	a	little	piece	of	our	Big	Bang.
However,	he’s	optimistic	that	the	practical	limits	aren’t	that	far	from	the	ultimate
ones.	Indeed,	existing	quantum	computer	prototypes	have	already	miniaturized
their	memory	by	storing	one	bit	per	atom,	and	scaling	that	up	would	allow
storing	about	10
25
	bits/kg—a	trillion	times	better	than	my	laptop.	Moreover,
using	electromagnetic	radiation	to	communicate	between	these	atoms	would
permit	about	5	×	10
40
	operations	per	second—31	orders	of	magnitude	better	than
my	CPU.

25
40
my	CPU.
In	summary,	the	potential	for	future	life	to	compute	and	figure	things	out	is
truly	mind-boggling:	in	terms	of	orders	of	magnitude,	today’s	best
supercomputers	are	much	further	from	the	ultimate	1	kg	computer	than	they	are
from	the	blinking	turn	signal	on	a	car,	a	device	that	stores	merely	one	bit	of

information,	flipping	it	between	on	and	off	about	once	per	second.

Other	Resources
From	a	physics	perspective,	everything	that	future	life	may	want	to	create—from
habitats	and	machines	to	new	life	forms—is	simply	elementary	particles
arranged	in	some	particular	way.	Just	as	a	blue	whale	is	rearranged	krill	and	krill
is	rearranged	plankton,	our	entire	Solar	System	is	simply	hydrogen	rearranged
during	13.8	billion	years	of	cosmic	evolution:	gravity	rearranged	hydrogen	into
stars	which	rearranged	the	hydrogen	into	heavier	atoms,	after	which	gravity
rearranged	such	atoms	into	our	planet	where	chemical	and	biological	processes
rearranged	them	into	life.
Future	life	that	has	reached	its	technological	limit	can	perform	such	particle
rearrangements	more	rapidly	and	efficiently,	by	first	using	its	computing	power
to	figure	out	the	most	efficient	method	and	then	using	its	available	energy	to
power	the	matter	rearrangement	process.	We	saw	how	matter	can	be	converted
into	both	computers	and	energy,	so	it’s	in	a	sense	the	only	fundamental	resource
needed.
*7

needed.
*7
	Once	future	life	has	bumped	up	against	the	physical	limits	on	what	it
can	do	with	its	matter,	there	is	only	one	way	left	for	it	to	do	more:	by	getting
more	matter.	And	the	only	way	it	can	do	this	is	by	expanding	into	our	Universe.
Spaceward	ho!

Just	how	great	is	our	cosmic	endowment?	Specifically,	what	upper	limits	do	the
laws	of	physics	place	on	the	amount	of	matter	that	life	can	ultimately	make	use
of?	Our	cosmic	endowment	is	mind-bogglingly	large,	of	course,	but	how	large,
exactly?	
Table	6.2
	lists	some	key	numbers.	Our	planet	is	currently	99.999999%
dead	in	the	sense	that	this	fraction	of	its	matter	isn’t	part	of	our	biosphere	and	is
doing	almost	nothing	useful	for	life	other	than	providing	gravitational	pull	and	a
magnetic	field.	This	raises	the	potential	of	one	day	using	a	hundred	million	times
more	matter	in	active	support	of	life.	If	we	can	put	all	of	the	matter	in	our	Solar
System	(including	the	Sun)	to	optimal	use,	we’ll	do	another	million	times	better.
Settling	our	Galaxy	would	grow	our	resources	another	trillion	times.

How	Far	Can	You	Go?
You	might	think	that	we	can	acquire	unlimited	resources	by	settling	as	many
other	galaxies	as	we	want	if	we’re	patient	enough,	but	that’s	not	what	modern
cosmology	suggests!	Yes,	space	itself	might	be	infinite,	containing	infinitely
many	galaxies,	stars	and	planets—indeed,	this	is	what’s	predicted	by	the
simplest	versions	of	
	the	currently	most	popular	scientific	paradigm	for
what	created	our	Big	Bang	13.8	billion	years	ago.	However,	even	if	there	are
infinitely	many	galaxies,	it	appears	that	we	can	see	and	reach	only	a	finite
number	of	them:	we	can	see	about	200	billion	galaxies	and	settle	in	at	most	ten
billion.
Region
Particles
Our	biosphere
10
43
Our	Planet
10
51
Our	Solar	System
10
57
Our	Galaxy
10
69
Our	range	traveling	at	half	speed	of	light
10
75
Our	range	traveling	at	speed	of	light
10
76
Our	Universe
10
78
Table	6.2:	Approximate	number	of	matter	particles	(protons	and	neutrons)	that	future	life	can	aspire	to	make
use	of.

billion.
Region
Particles
10
43
Our	Planet
10
51
10
57
Our	Galaxy
10
69
10
75
10
76
10
78
use	of.
What	limits	us	is	the	speed	of	light:	one	light-year	(about	ten	trillion
kilometers)	per	year.	
Figure	6.6
	shows	the	part	of	space	from	which	light	has
reached	us	so	far	during	the	13.8	billion	years	since	our	Big	Bang,	a	spherical
region	known	as	“our	observable	Universe”	or	simply	
“our	Universe.”
	Even	if
space	is	infinite,	our	Universe	is	finite,	containing	“only”	about	10
78
	atoms.
Moreover,	about	98%	of	our	Universe	is	“see	but	not	touch,”	in	the	sense	that	we

use	of.
Figure	6.6
	Even	if
78
	atoms.
can	see	it	but	never	reach	it	even	if	we	travel	at	the	speed	of	light	forever.	Why	is
this?	After	all,	the	limit	to	how	far	we	can	see	comes	simply	from	the	fact	that

our	Universe	isn’t	infinitely	old,	so	that	distant	light	hasn’t	yet	had	time	to	reach
us.	So	shouldn’t	we	be	able	to	travel	to	arbitrarily	distant	galaxies	if	we	have	no
limit	on	how	much	time	we	can	spend	en	route?

Figure	6.6:	Our	Universe,	i.e.,	the	spherical	region	of	space	from	which	light	has	had	time	to
reach	us	(at	the	center)	during	the	13.8	billion	years	since	our	Big	Bang.	The	patterns	show	the
baby	pictures	of	our	Universe	taken	by	the	Planck	satellite,	showing	that	when	it	was	merely
400,000	years	old,	it	consisted	of	hot	plasma	nearly	as	hot	as	the	surface	of	the	Sun.	Space
probably	continues	beyond	this	region,	and	new	matter	comes	into	view	every	year.
The	first	challenge	is	that	our	Universe	is	expanding,	which	means	that	almost
all	galaxies	are	flying	away	from	us,	so	settling	distant	galaxies	amounts	to	a
game	of	catch-up.	The	second	challenge	is	that	this	cosmic	expansion	is
accelerating,	due	to	the	mysterious	dark	energy	that	makes	up	about	70%	of	our

Universe.	To	understand	how	this	causes	trouble,	imagine	that	you	enter	a	train
platform	and	see	your	train	slowly	accelerating	away	from	you,	but	with	a	door
left	invitingly	open.	If	you’re	fast	and	foolhardy,	can	you	catch	the	train?	Since	it
will	eventually	go	faster	than	you	can	run,	the	answer	clearly	depends	on	how
far	away	from	you	the	train	is	initially:	if	it’s	beyond	a	certain	critical	distance,
you’ll	never	catch	up	with	it.	We	face	the	same	situation	trying	to	catch	those
distant	galaxies	that	are	accelerating	away	from	us:	even	if	we	could	travel	at	the
speed	of	light,	all	galaxies	beyond	about	17	billion	light-years	remain	forever	out
of	reach—and	that’s	over	98%	of	the	galaxies	in	our	Universe.
But	hold	on:	didn’t	Einstein’s	special	relativity	theory	say	that	nothing	can
travel	faster	than	light?	So	how	can	galaxies	outrace	something	traveling	at	the
speed	of	light?	The	answer	is	that	special	relativity	is	superseded	by	Einstein’s

general	relativity	theory,	where	the	speed	limit	is	more	liberal:	nothing	can	travel
faster	than	the	speed	of	light	
through	space,
	but	space	is	free	to	expand	as	fast	as
it	wants.	Einstein	also	gave	us	a	nice	way	of	visualizing	these	speed	limits	by
viewing	time	as	the	fourth	dimension	in	
spacetime
	(see	
figure	6.7
,	where	I’ve
kept	things	three-dimensional	by	omitting	one	of	the	three	space	dimensions).	If
space	weren’t	expanding,	light	rays	would	form	slanted	45-degree	lines	through

spacetime
	(see	
figure	6.7
spacetime,	so	that	the	regions	we	can	see	and	reach	from	here	and	now	are
cones.	Whereas	our	past	light	cone	would	be	truncated	by	our	Big	Bang	13.8
billion	years	ago,	our	future	light	cone	would	expand	forever,	giving	us	access	to
an	unlimited	cosmic	endowment.	In	contrast,	the	middle	panel	of	the	figure
shows	that	an	expanding	universe	with	dark	energy	(which	appears	to	be	the
Universe	we	inhabit)	deforms	our	light	cones	into	a	champagne-glass	shape,

forever	limiting	the	number	of	galaxies	we	can	settle	to	about	10	billion.
If	this	limit	makes	you	feel	cosmic	claustrophobia,	let	me	cheer	you	up	with	a
possible	loophole:	my	calculation	assumes	that	dark	energy	remains	constant
over	time,	consistent	with	what	the	latest	measurements	suggest.	However,	we
still	have	no	clue	what	dark	energy	really	is,	which	leaves	a	glimmer	of	hope	that
dark	energy	will	eventually	decay	away	(much	like	the	similar	dark-energy-like
substance	postulated	to	explain	cosmic	inflation),	and	if	this	happens,	the

acceleration	will	give	way	to	
deceleration,
	potentially	enabling	future	life	forms
to	keep	settling	new	galaxies	for	as	long	as	they	last.

Figure	6.7:	In	a	spacetime	diagram,	an	event	is	a	point	whose	horizontal	and	vertical	positions
encode	where	and	when	it	occurs,	respectively.	If	space	isn’t	expanding	(left	panel),	then	two
cones	delimit	the	parts	of	spacetime	that	we	on	Earth	(at	apex)	can	be	affected	by	(bottom	cone)
and	can	have	an	effect	on	(top	cone),	because	causal	effects	cannot	travel	faster	than	light,	which
travels	a	distance	of	one	light-year	per	year.	Things	get	more	interesting	when	space	expands
(right	panels).	According	to	the	standard	model	of	cosmology,	we	can	only	see	and	reach	a	finite
part	of	spacetime	even	if	space	is	infinite.	In	the	middle	image,	reminiscent	of	a	champagne
glass,	we	use	coordinates	that	hide	the	expansion	of	space	so	that	the	motions	of	distant	galaxies
over	time	correspond	to	vertical	lines.	From	our	current	vantage	point,	13.8	billion	years	after
our	Big	Bang,	light	rays	have	had	time	to	reach	us	only	from	the	base	of	the	champagne	glass,

and	even	if	we	travel	at	the	speed	of	light,	we	can	never	reach	regions	outside	the	upper	part	of
the	glass,	which	contains	about	10	billion	galaxies.	In	the	right	image,	reminiscent	of	a	water
droplet	beneath	a	flower,	we	use	the	familiar	coordinates	where	space	is	seen	to	expand.	This
deforms	the	glass	base	to	a	droplet	shape	because	regions	at	the	edges	of	what	we	can	see	were
all	very	close	together	early	on.

How	Fast	Can	You	Go?
Above	we	explored	how	many	galaxies	a	civilization	could	settle	if	it	expanded
in	all	directions	at	the	speed	of	light.	General	relativity	says	that	it’s	impossible
to	send	rockets	through	space	at	the	speed	of	light,	because	this	would	require
infinite	energy,	so	how	fast	can	rockets	go	in	practice?
*8
NASA’s	New	Horizons	rocket	broke	the	speed	record	when	it	blasted	off
toward	Pluto	in	2006	at	a	speed	of	about	100,000	miles	per	hour	(45	kilometers
per	second),	and	NASA’s	2018	Solar	Probe	Plus	aims	to	go	over	four	times
faster	by	falling	very	close	to	the	Sun,	but	even	that’s	less	than	a	puny	0.1%	of
the	speed	of	light.	The	quest	for	faster	and	better	rockets	has	captivated	some	of
the	brightest	minds	of	the	past	century,	and	there’s	a	rich	and	fascinating
literature	on	the	topic.	Why	is	it	so	hard	to	go	faster?	The	two	key	problems	are
that	conventional	rockets	spend	most	of	their	fuel	simply	to	accelerate	the	fuel

they	carry	with	them,	and	that	today’s	rocket	fuel	is	hopelessly	inefficient—the
fraction	of	its	mass	turned	into	energy	isn’t	much	better	than	the	0
.
00000005%
for	gasoline	that	we	saw	in	
table	6.1
.	One	obvious	improvement	is	to	switch	to
more	efficient	fuel.	For	example,	Freeman	Dyson	and	others	worked	on	NASA’s
Project	Orion,	which	aimed	to	explode	about	300,000	nuclear	bombs	during	10
days	to	reach	about	3%	of	the	speed	of	light	with	a	spaceship	large	enough	to
carry	humans	to	another	solar	system	during	a	century-long	journey.
5

.
00000005%
table	6.1
5
	Others
have	explored	using	antimatter	as	fuel,	since	combining	it	with	ordinary	matter
releases	energy	with	nearly	100%	efficiency.
Another	popular	idea	is	to	build	a	rocket	that	need	not	carry	its	own	fuel.	For
example,	interstellar	space	isn’t	a	perfect	vacuum,	but	contains	the	occasional
hydrogen	ion	(a	lone	proton:	a	hydrogen	atom	that’s	lost	its	electron).	In	1960,
this	gave	physicist	Robert	Bussard	the	idea	behind	what’s	now	known	as	a
Bussard	ramjet:
	to	scoop	up	such	ions	en	route	and	use	them	as	rocket	fuel	in	an

onboard	fusion	reactor.	Although	recent	work	has	cast	doubts	on	whether	this
can	be	made	to	work	in	practice,	there’s	another	carry-no-fuel	idea	that	does
appear	feasible	for	a	high-tech	spacefaring	civilization:	laser	sailing.
Figure	6.8
	illustrates	a	clever	laser-sail	rocket	design	pioneered	in	1984	by
Robert	Forward,	the	same	physicist	who	invented	the	statites	we	explored	for
Dyson	sphere	construction.	Just	as	air	molecules	bouncing	off	a	sailboat	sail	will

push	it	forward,	light	particles	(photons)	bouncing	off	a	mirror	will	push	it
forward.	By	beaming	a	huge	solar-powered	laser	at	a	vast	ultralight	sail	attached
to	a	spacecraft,	we	can	use	the	energy	of	our	own	Sun	to	accelerate	the	rocket	to
great	speeds.	But	how	do	you	stop?	This	is	the	question	that	eluded	me	until	I
read	Forward’s	brilliant	paper:	as	
figure	6.8
	shows,	the	outer	ring	of	the	laser	sail
detaches	and	moves	in	front	of	the	spacecraft,	reflecting	our	laser	beam	back	to
decelerate	the	craft	and	its	smaller	sail.
6
	Forward	calculated	that	this	could	let
humans	make	the	four-light-year	journey	to	the	
α
	Centauri	solar	system	in
merely	forty	years.	Once	there,	you	could	imagine	building	a	new	giant	laser
system	and	continuing	star-hopping	throughout	the	Milky	Way	Galaxy.

Figure	6.8:	Robert	Forward’s	design	for	a	laser	sailing	mission	to	the	
α
	Centauri	star	system	four
light-years	away.	Initially,	a	powerful	laser	in	our	Solar	System	accelerates	the	spacecraft	by
applying	radiation	pressure	to	its	laser	sail.	To	brake	before	reaching	the	destination,	the	outer
part	of	the	sail	detaches	and	reflects	laser	light	back	at	the	spacecraft.
But	why	stop	there?	In	1964,	the	Soviet	astronomer	Nikolai	Kardashev
proposed	grading	civilizations	by	how	much	energy	they	could	put	to	use.
Harnessing	the	energy	of	a	planet,	a	star	(with	a	Dyson	sphere,	say)	and	a	galaxy
correspond	to	civilizations	of	Type	I,	Type	II	and	Type	III	on	the	Kardashev
scale,	respectively.	Subsequent	thinkers	have	suggested	that	Type	IV	should
correspond	to	harnessing	our	entire	accessible	Universe.	Since	then,	there’s	been
good	news	and	bad	news	for	ambitious	life	forms.	The	bad	news	is	that	dark
energy	exists,	which,	as	we	saw,	appears	to	limit	our	reach.	The	good	news	is	the

dramatic	progress	of	artificial	intelligence.	Even	optimistic	visionaries	such	as
Carl	Sagan	used	to	view	the	prospects	of	humans	reaching	other	galaxies	as
rather	hopeless,	given	our	propensity	to	die	within	the	first	century	of	a	journey
that	would	take	millions	of	years	even	if	traveling	at	near	light	speed.	Refusing
to	give	up,	they	considered	freezing	astronauts	to	extend	their	life,	slowing	their
aging	by	traveling	very	close	to	light	speed,	or	sending	a	community	that	would

travel	for	tens	of	thousands	of	generations—longer	than	the	human	race	has
existed	thus	far.
The	possibility	of	superintelligence	completely	transforms	this	picture,
making	it	much	more	promising	for	those	with	intergalactic	wanderlust.
Removing	the	need	to	transport	bulky	human	life-support	systems	and	adding
AI-invented	technology,	intergalactic	settlement	suddenly	appears	rather
straightforward.	Forward’s	laser	sailing	becomes	much	cheaper	when	the
spacecraft	need	merely	be	large	enough	to	contain	a	“seed	probe”:	a	robot
capable	of	landing	on	an	asteroid	or	planet	in	the	target	solar	system	and
building	up	a	new	civilization	from	scratch.	It	doesn’t	even	have	to	carry	the
instructions	with	it:	all	it	has	to	do	is	build	a	receiving	antenna	large	enough	to
pick	up	more	detailed	blueprints	and	instructions	transmitted	from	its	mother
civilization	at	the	speed	of	light.	Once	done,	it	uses	its	newly	constructed	lasers
to	send	out	new	seed	probes	to	continue	settling	the	galaxy	one	solar	system	at	a

time.	Even	the	vast	dark	expanses	of	space	between	galaxies	tend	to	contain	a
significant	number	of	intergalactic	stars	(rejects	once	ejected	from	their	home
galaxies)	that	can	be	used	as	way	stations,	thus	enabling	an	island-hopping
strategy	for	intergalactic	laser	sailing.
Once	another	solar	system	or	galaxy	has	been	settled	by	superintelligent	AI,
bringing	humans	there	is	easy—if	humans	have	succeeded	in	making	the	AI
have	this	goal.	All	the	necessary	information	about	humans	can	be	transmitted	at

the	speed	of	light,	after	which	the	AI	can	assemble	quarks	and	electrons	into	the
desired	humans.	This	could	be	done	either	rather	low-tech	by	simply
transmitting	the	two	gigabytes	of	information	needed	to	specify	a	person’s	DNA
and	then	incubating	a	baby	to	be	raised	by	the	AI,	or	the	AI	could	nanoassemble
quarks	and	electrons	into	full-grown	people	who	would	have	all	the	memories
scanned	from	their	originals	back	on	Earth.
This	means	that	if	there’s	an	intelligence	explosion,	the	key	question	isn’t	if

intergalactic	settlement	is	possible,	but	simply	how	fast	it	can	proceed.	Since	all
the	ideas	we’ve	explored	above	come	from	humans,	they	should	be	viewed	as
merely	lower	limits	on	how	fast	life	can	expand;	ambitious	superintelligent	life
can	probably	do	a	lot	better,	and	it	will	have	a	strong	incentive	to	push	the	limits,
since	in	the	race	against	time	and	dark	energy,	every	1%	increase	in	average
settlement	speed	translates	into	3%	more	galaxies	colonized.

For	example,	if	it	takes	20	years	to	travel	10	light-years	to	the	next	star	system
with	a	laser-sail	system,	and	then	another	10	years	to	settle	it	and	build	new

lasers	and	seed	probes	there,	the	settled	region	of	space	will	be	a	sphere	growing
in	all	directions	at	a	third	of	the	speed	of	light	on	average.	In	a	beautiful	and
thorough	analysis	of	cosmically	expanding	civilizations	in	2014,	the	American
physicist	Jay	Olson	considered	a	high-tech	alternative	to	the	island-hopping
approach,	involving	two	separate	types	of	probes:	
seed	probes
	and	
expanders
.
7
The	seed	probes	would	slow	down,	land	and	seed	their	destination	with	life.	The
expanders,	on	the	other	hand,	would	never	stop:	they’d	scoop	up	matter	in	flight,
perhaps	using	some	improved	variant	of	the	ramjet	technology,	and	use	this
matter	both	as	fuel	and	as	raw	material	out	of	which	they’d	build	expanders	and
copies	of	themselves.	This	self-reproducing	fleet	of	expanders	would	keep
gently	accelerating	to	always	maintain	a	constant	speed	(say	half	the	speed	of
light)	relative	to	nearby	galaxies,	and	reproduce	often	enough	that	the	fleet

formed	an	expanding	spherical	shell	with	a	constant	number	of	expanders	per
shell	area.
Last	but	not	least,	there’s	the	sneaky	Hail	Mary	approach	to	expanding	even
faster	than	any	of	the	above	methods	will	permit:	using	Hans	Moravec’s	“cosmic
spam”	scam	from	chapter	4.	By	broadcasting	a	message	that	tricks	naive	freshly
evolved	civilizations	into	building	a	superintelligent	machine	that	hijacks	them,
a	civilization	can	expand	essentially	at	the	speed	of	light,	the	speed	at	which

their	seductive	siren	song	spreads	through	the	cosmos.	Since	this	may	be	the
only
	way	for	advanced	civilizations	to	reach	most	of	the	galaxies	within	their
future	light	cone	and	they	have	little	incentive	not	to	try	it,	we	should	be	highly
suspicious	of	any	transmissions	from	extraterrestrials!	In	Carl	Sagan’s	book
Contact,
	we	Earthlings	used	blueprints	from	aliens	to	build	a	machine	we	didn’t
understand—I	don’t	recommend	doing	this…
In	summary,	most	scientists	and	sci-fi	authors	considering	cosmic	settlement

only
Contact,
have	in	my	opinion	been	overly	pessimistic	in	ignoring	the	possibility	of
superintelligence:	by	limiting	attention	to	human	travelers,	they’ve
overestimated	the	difficulty	of	intergalactic	travel,	and	by	limiting	attention	to
technology	invented	by	humans,	they’ve	overestimated	the	time	needed	to
approach	the	physical	limits	of	what’s	possible.

Staying	Connected	via	Cosmic	Engineering
If	dark	energy	continues	to	accelerate	distant	galaxies	away	from	one	another,	as
the	latest	experimental	data	suggests,	then	this	will	pose	a	major	nuisance	to	the
future	of	life.	It	means	that	even	if	a	future	civilization	manages	to	settle	a
million	galaxies,	dark	energy	will	over	the	course	of	tens	of	billions	of	years
fragment	this	cosmic	empire	into	thousands	of	different	regions	unable	to
communicate	with	one	another.	If	future	life	does	nothing	to	prevent	this
fragmentation,	then	the	largest	remaining	bastions	of	life	will	be	clusters
containing	about	a	thousand	galaxies,	whose	combined	gravity	is	strong	enough
to	overpower	the	dark	energy	trying	to	separate	them.
If	a	superintelligent	civilization	wants	to	stay	connected,	this	would	give	it	a
strong	incentive	to	do	large-scale	cosmic	engineering.	How	much	matter	will	it
have	time	to	move	into	its	largest	supercluster	before	dark	energy	puts	it	forever

out	of	reach?	One	method	for	moving	a	star	large	distances	is	to	nudge	a	third
star	into	a	binary	system	where	two	stars	are	stably	orbiting	each	other.	Just	as
with	romantic	relationships,	the	introduction	of	a	third	partner	can	destabilize
things	and	lead	to	one	of	the	three	being	violently	ejected—in	the	stellar	case,	at
great	speed.	If	some	of	the	three	partners	are	black	holes,	such	a	volatile
threesome	can	be	used	to	fling	mass	fast	enough	to	fly	far	outside	the	host
galaxy.	Unfortunately,	this	three-body	technique,	applied	either	to	stars,	black

holes	or	galaxies,	doesn’t	appear	able	to	move	more	than	a	tiny	fraction	of	a
civilization’s	mass	the	large	distances	required	to	outsmart	dark	energy.
But	this	obviously	doesn’t	mean	that	superintelligent	life	can’t	come	up	with
better	methods,	say	converting	much	of	the	mass	in	outlying	galaxies	into
spacecraft	that	can	travel	to	the	home	cluster.	If	a	sphalerizer	can	be	built,
perhaps	it	can	even	be	used	to	convert	the	matter	into	energy	that	can	be	beamed

into	the	home	cluster	as	light,	where	it	can	be	reconfigured	back	into	matter	or
used	as	a	power	source.
The	ultimate	luck	will	be	if	it	turns	out	to	be	possible	to	build	stable
traversable	wormholes,	enabling	near-instantaneous	communication	and	travel
between	the	two	ends	of	the	wormhole	no	matter	how	far	apart	they	are.	A
wormhole	is	a	shortcut	through	spacetime	that	lets	you	travel	from	A	to	B
without	going	through	the	intervening	space.	Although	stable	wormholes	are
allowed	by	Einstein’s	theory	of	general	relativity	and	have	appeared	in	movies

such	as	
Contact
	and	
Interstellar,
	they	require	the	existence	of	a	strange
hypothetical	kind	of	matter	with	negative	density,	whose	existence	may	hinge	on
poorly	understood	quantum	gravity	effects.	In	other	words,	useful	wormholes
may	well	turn	out	to	be	impossible,	but	if	not,	superintelligent	life	has	huge
incentives	to	build	them.	Not	only	would	wormholes	revolutionize	rapid
communication	within	individual	galaxies,	but	by	linking	outlying	galaxies	to
the	central	cluster	early	on,	wormholes	would	allow	the	entire	dominion	of
future	life	to	remain	connected	for	the	long	haul,	completely	thwarting	dark
energy’s	attempts	to	censor	communication.	Once	two	galaxies	are	connected	by
a	stable	wormhole,	they’ll	remain	connected	no	matter	how	far	apart	they	drift.
If,	despite	its	best	attempts	at	cosmic	engineering,	a	future	civilization
concludes	that	parts	of	it	are	doomed	to	drift	out	of	contact	forever,	it	might
simply	let	them	go	and	wish	them	well.	However,	if	it	has	ambitious	computing

goals	that	involve	seeking	the	answers	to	certain	very	difficult	questions,	it	might
instead	resort	to	a	slash-and-burn	strategy:	it	could	convert	the	outlying	galaxies
into	massive	computers	that	transform	their	matter	and	energy	into	computation
at	a	frenzied	pace,	in	the	hope	that	before	dark	energy	pushes	their	burnt-out
remnants	from	view,	they	could	transmit	the	long-sought	answers	back	to	the
mother	cluster.	This	slash-and-burn	strategy	would	be	particularly	appropriate
for	regions	so	distant	that	they	can	only	be	reached	by	the	“cosmic	spam”

method,	much	to	the	chagrin	of	the	preexisting	inhabitants.	Back	home	in	the
mother	region,	the	civilization	could	instead	aim	for	maximum	conservation	and
efficiency	to	last	as	long	as	possible.

How	Long	Can	You	Last?
Longevity	is	something	that	most	ambitious	people,	organizations	and	nations
aspire	to.	So	if	an	ambitious	future	civilization	develops	superintelligence	and
wants	longevity,	how	long	can	it	last?
The	first	thorough	scientific	analysis	of	our	far	future	was	performed	by	none
less	than	Freeman	Dyson,	and	
table	6.3
	summarizes	some	of	his	key	findings.
The	conclusion	is	that	unless	intelligence	intervenes,	solar	systems	and	galaxies
gradually	get	destroyed,	eventually	followed	by	everything	else,	leaving	nothing
but	cold,	dead,	empty	space	with	an	eternally	fading	glow	of	radiation.	But
Freeman	ends	his	analysis	on	an	optimistic	note:	“There	are	good	scientific
reasons	for	taking	seriously	the	possibility	that	life	and	intelligence	can	succeed
in	molding	this	universe	of	ours	to	their	own	purposes.”
8
I	think	that	superintelligence	could	easily	solve	many	of	the	problems	listed	in
table	6.3
,	since	it	can	rearrange	matter	into	something	better	than	solar	systems

8
table	6.3
and	galaxies.	Oft-discussed	challenges	such	as	the	death	of	our	Sun	in	a	few
billion	years	won’t	be	showstoppers,	since	even	a	relatively	low-tech	civilization
can	easily	move	to	low-mass	stars	that	last	for	over	200	billion	years.	Assuming
that	superintelligent	civilizations	build	their	own	power	plants	that	are	more
efficient	than	stars,	they	may	in	fact	want	to	
prevent
	star	formation	to	conserve
energy:	even	if	they	use	a	Dyson	sphere	to	harvest	all	the	energy	output	during	a
star’s	main	lifetime	(recouping	about	0.1%	of	the	total	energy),	they	may	be

prevent
unable	to	keep	much	of	the	remaining	99.9%	of	the	energy	from	going	to	waste
when	very	hefty	stars	die.	A	heavy	star	dies	in	a	supernova	explosion	from
which	most	of	the	energy	escapes	as	elusive	neutrinos,	and	for	very	heavy	stars,
a	large	amount	of	mass	gets	wasted	by	forming	a	black	hole	from	which	the
energy	takes	10
67
	years	to	seep	out.
What
When
Current	age	of	our	Universe
10
10
	years
Dark	energy	pushes	most	galaxies	out	of	reach
10
11
	years
Last	stars	burn	out
10
14
	years
Planets	detached	from	stars
10
15
	years

Stars	detached	from	galaxies
10
19
	years
Decay	of	orbits	by	gravitational	radiation
10
20
	years
Protons	decay	(at	the	earliest)
>
	10
34
	years
Stellar-mass	black	holes	evaporate
10
67
	years
Supermassive	black	holes	evaporate
10
91
	years
All	matter	decays	to	iron
10
1500
	years
All	matter	forms	black	holes,	which	then	evaporate
10
1026
	years
Table	6.3:	Estimates	for	the	distant	future,	all	but	the	2nd	and	7th	made	by	Freeman	Dyson.	He	made	these
calculations	before	the	discovery	of	dark	energy,	which	may	enable	several	types	of	“cosmocalypse”	in
10
10
–10
11
	years.	Protons	may	be	completely	stable;	if	not,	experiments	suggest	it	will	take	over	10
34
	years
for	half	of	them	to	decay.
As	long	as	superintelligent	life	hasn’t	run	out	of	matter/energy,	it	can	keep
maintaining	its	habitat	in	the	state	it	desires.	Perhaps	it	can	even	discover	a	way
to	prevent	protons	from	decaying	using	the	so-called	
watched-pot	effect
	of
quantum	mechanics,	whereby	the	decay	process	is	slowed	by	making	regular

10
10
–10
11
34
	years
	of
observations.	There	is,	however,	a	potential	showstopper:	a	
cosmocalypse
destroying	our	entire	Universe,	perhaps	as	soon	as	10–100	billion	years	from
now.	The	discovery	of	dark	energy	and	progress	in	string	theory	has	raised	new
cosmocalypse	scenarios	that	Freeman	Dyson	wasn’t	aware	of	when	he	wrote	his
seminal	paper.
So	how’s	our	Universe	going	to	end,	billions	of	years	from	now?	I	have	five
main	suspects	for	our	upcoming	cosmic	apocalypse,	or	cosmocalypse,	illustrated
in	
figure	6.9
:	the	
Big	Chill,
	the	
Big	Crunch,
	the	
Big	Rip,
	the	
Big	Snap
	and	
Death

in	
figure	6.9
:	the	
Big	Chill,
	the	
	the	
Big	Rip,
	the	
Big	Snap
	and	
Death
Bubbles.
	Our	Universe	has	now	been	expanding	for	about	14	billion	years.	The
Big	Chill	is	when	our	Universe	keeps	expanding	forever,	diluting	our	cosmos
into	a	cold,	dark	and	ultimately	dead	place;	this	was	viewed	as	the	most	likely
outcome	back	when	Freeman	wrote	that	paper.	I	think	of	it	as	the	T.	S.	Eliot
option:	“This	is	the	way	the	world	ends	/	Not	with	a	bang	but	a	whimper.”	If
you,	like	Robert	Frost,	prefer	the	world	to	end	in	fire	rather	than	ice,	then	cross

Big	Rip,
	the	
Big	Snap
	and	
Death
Bubbles.
your	fingers	for	the	Big	Crunch,	where	the	cosmic	expansion	is	eventually
reversed	and	everything	comes	crashing	back	together	in	a	cataclysmic	collapse
akin	to	a	backward	Big	Bang.	Finally,	the	Big	Rip	is	like	the	Big	Chill	for	the
impatient,	where	our	galaxies,	planets	and	even	atoms	get	torn	apart	in	a	grand

finale	a	finite	time	from	now.	Which	of	these	three	should	you	bet	on?	That
depends	on	what	the	dark	energy,	which	makes	up	about	70%	of	the	mass	of	our
Universe,	will	do	as	space	continues	to	expand.	It	can	be	any	one	of	the	Chill,
Crunch	or	Rip	scenarios,	depending	on	whether	the	dark	energy	sticks	around
unchanged,	dilutes	to	negative	density	or	anti-dilutes	to	higher	density,
respectively.	Since	we	still	have	no	clue	what	dark	energy	is,	I’ll	just	tell	you
how	I’d	bet:	40%	on	the	Big	Chill,	9%	on	the	Big	Crunch	and	1%	on	the	Big
Rip.

Figure	6.9:	We	know	that	our	Universe	began	with	a	hot	Big	Bang	14	billion	years	ago,
expanded	and	cooled,	and	merged	its	particles	into	atoms,	stars	and	galaxies.	But	we	don’t	know
its	ultimate	fate.	Proposed	scenarios	include	a	Big	Chill	(eternal	expansion),	a	Big	Crunch
(recollapse),	a	Big	Rip	(an	infinite	expansion	rate	tearing	everything	apart),	a	Big	Snap	(the
fabric	of	space	revealing	a	lethal	granular	nature	when	stretched	too	much),	and	Death	Bubbles
(space	“freezing”	in	lethal	bubbles	that	expand	at	the	speed	of	light).
What	about	the	other	50%	of	my	money?	I’m	saving	it	for	the	“none	of	the
above”	option,	because	I	think	we	humans	need	to	be	humble	and	acknowledge
that	there	are	basic	things	we	still	don’t	understand.	The	nature	of	space,	for
example.	The	Chill,	Crunch	and	Rip	endings	all	assume	that	space	itself	is	stable
and	infinitely	stretchable.	We	used	to	think	of	space	as	just	the	boring	static	stage
upon	which	the	cosmic	drama	unfolds.	Then	Einstein	taught	us	that	space	is

really	one	of	the	key	actors:	it	can	curve	into	black	holes,	it	can	ripple	as
gravitational	waves	and	it	can	stretch	as	an	expanding	universe.	Perhaps	it	can
even	freeze	into	a	different	phase	much	like	water	can,	with	fast-expanding	death
bubbles	of	the	new	phase	offering	another	wild-card	cosmocalypse	candidate.	If
death	bubbles	are	possible,	they	would	probably	expand	at	the	speed	of	light,
just	like	the	growing	sphere	of	cosmic	spam	from	a	maximally	aggressive
civilization.

Moreover,	Einstein’s	theory	says	that	space	stretching	can	always	continue,
allowing	our	Universe	to	approach	infinite	volume	as	in	the	Big	Chill	and	Big

Rip	scenarios.	This	sounds	a	bit	too	good	to	be	true,	and	I	suspect	that	it	is.	A
rubber	band	looks	nice	and	continuous,	just	like	space,	but	if	you	stretch	it	too
much,	it	snaps.	Why?	Because	it’s	made	of	atoms,	and	with	enough	stretching,
this	granular	atomic	nature	of	the	rubber	becomes	important.	Could	it	be	that
space	too	has	some	sort	of	granularity	on	a	scale	that’s	simply	too	small	for	us	to
have	noticed?	Quantum	gravity	research	suggests	that	it	doesn’t	make	sense	to
talk	about	traditional	three-dimensional	space	on	scales	smaller	than	about	10
-34
meters.	If	it’s	really	true	that	space	can’t	be	stretched	indefinitely	without
undergoing	a	cataclysmic	“Big	Snap,”	then	future	civilizations	may	wish	to
relocate	to	the	largest	non-expanding	region	of	space	(a	huge	galaxy	cluster)	that
they	can	reach.

How	Much	Can	You	Compute?
After	exploring	how	long	future	life	
can
	last,	let’s	explore	how	long	it	might
want
	to	last.	Although	you	might	find	it	natural	to	want	to	live	as	long	as
possible,	Freeman	Dyson	also	gave	a	more	quantitative	argument	for	this	desire:
the	cost	of	computation	drops	when	you	compute	slowly,	so	you’ll	ultimately	get
more	done	if	you	slow	things	down	as	much	as	possible.	Freeman	even
calculated	that	if	our	Universe	keeps	expanding	and	cooling	forever,	an	infinite
amount	of	computation	might	be	possible.
Slow	doesn’t	necessarily	mean	boring:	if	future	life	lives	in	a	simulated	world,
its	subjectively	experienced	flow	of	time	need	not	have	anything	to	do	with	the
glacial	pace	at	which	the	simulation	is	being	run	in	the	outside	world,	so	the
prospects	of	infinite	computation	could	translate	into	subjective	immortality	for
simulated	life	forms.	Cosmologist	Frank	Tipler	has	built	on	this	idea	to	speculate
that	you	could	also	achieve	subjective	immortality	in	the	final	moments	before	a

Big	Crunch	by	speeding	up	the	computations	toward	infinity	as	the	temperature
and	density	skyrocketed.
Since	dark	energy	appears	to	spoil	both	Freeman’s	and	Frank’s	dreams	of
infinite	computation,	future	superintelligence	may	prefer	to	burn	through	its
energy	supplies	relatively	quickly,	to	turn	them	into	computations	before	running
into	problems	such	as	cosmic	horizons	and	proton	decay.	If	maximizing	total
computation	is	the	ultimate	goal,	the	best	strategy	will	be	a	trade-off	between	too

slow	(to	avoid	the	aforementioned	problems)	and	too	fast	(spending	more	energy
than	needed	per	computation).
Putting	together	everything	we’ve	explored	in	this	chapter	tells	us	that
maximally	efficient	power	plants	and	computers	would	enable	superintelligent
life	to	perform	a	mind-boggling	amount	of	computation.	Powering	your	thirteen-
watt	brain	for	a	hundred	years	requires	the	energy	in	about	half	a	milligram	of
matter—less	than	in	a	typical	grain	of	sugar.	Seth	Lloyd’s	work	suggests	that	the

brain	could	be	made	a	quadrillion	times	more	energy	efficient,	enabling	that
sugar	grain	to	power	a	simulation	of	all	human	lives	ever	lived	as	well	as
thousands	of	times	more	people.	If	all	the	matter	in	our	available	Universe	could
be	used	to	simulate	people,	that	would	enable	over	10
69
	lives—or	whatever	else
superintelligent	AI	preferred	to	do	with	its	computational	power.	Even	more

lives	would	be	possible	if	their	simulations	were	run	more	slowly.
9
	Conversely,
in	his	book	
Superintelligence,
	Nick	Bostrom	estimates	that	10
58
	human	lives
could	be	simulated	with	more	conservative	assumptions	about	energy	efficiency.
However	we	slice	and	dice	these	numbers,	they’re	huge,	as	is	our	responsibility
for	ensuring	that	this	future	potential	of	life	to	flourish	isn’t	squandered.	As
Bostrom	puts	it:	“If	we	represent	all	the	happiness	experienced	during	one	entire
such	life	by	a	single	teardrop	of	joy,	then	the	happiness	of	these	souls	could	fill
and	refill	the	Earth’s	oceans	every	second,	and	keep	doing	so	for	a	hundred
billion	billion	millennia.	It	is	really	important	that	we	make	sure	these	truly	are
tears	of	joy.”

The	speed	of	light	limits	not	only	the	spread	of	life,	but	also	the	nature	of	life,
placing	strong	constraints	on	communication,	consciousness	and	control.	So	if
much	of	our	cosmos	eventually	comes	alive,	what	will	this	life	be	like?

Thought	Hierarchies
Have	you	ever	tried	and	failed	to	swat	a	fly	with	your	hand?	The	reason	that	it
can	react	faster	than	you	is	that	it’s	smaller,	so	that	it	takes	less	time	for
information	to	travel	between	its	eyes,	brain	and	muscles.	This	“bigger	=	slower”
principle	applies	not	only	to	biology,	where	the	speed	limit	is	set	by	how	fast
electrical	signals	can	travel	through	neurons,	but	also	to	future	cosmic	life	if	no
information	can	travel	faster	than	light.	So	for	an	intelligent	information-
processing	system,	going	big	is	a	mixed	blessing	involving	an	interesting	trade-
off.	On	one	hand,	going	bigger	lets	it	contain	more	particles,	which	enable	more
complex	thoughts.	On	the	other	hand,	this	slows	down	the	rate	at	which	it	can
have	truly	global	thoughts,	since	it	now	takes	longer	for	the	relevant	information
to	propagate	to	all	its	parts.
So	if	life	engulfs	our	cosmos,	what	form	will	it	choose:	simple	and	fast,	or
complex	and	slow?	I	predict	that	it	will	make	the	same	choice	as	Earth	life	has

made:	both!	The	denizens	of	Earth’s	biosphere	span	a	staggering	range	of	sizes,
from	gargantuan	two-hundred-ton	blue	whales	down	to	the	petite	10
-16
	kg
bacterium	
Pelagibacter,
	believed	to	account	for	more	biomass	than	all	the
world’s	fish	combined.	Moreover,	organisms	that	are	large,	complex	and	slow
often	mitigate	their	sluggishness	by	containing	smaller	modules	that	are	simple
and	fast.	For	example,	your	blink	reflex	is	extremely	fast	precisely	because	it’s

-16
	kg
bacterium	
implemented	by	a	small	and	simple	circuit	that	doesn’t	involve	most	of	your
brain:	if	that	hard-to-swat	fly	accidentally	heads	toward	your	eye,	you’ll	blink
within	a	tenth	of	a	second,	long	before	the	relevant	information	has	had	time	to
spread	throughout	your	brain	and	make	you	consciously	aware	of	what
happened.	By	organizing	its	information	processing	into	a	hierarchy	of	modules,
our	biosphere	manages	to	both	have	the	cake	and	eat	it,	attaining	both	speed	and
complexity.	We	humans	already	use	this	same	hierarchical	strategy	to	optimize

parallel	computing.
Because	internal	communication	is	slow	and	costly,	I	expect	advanced	future
cosmic	life	to	do	the	same,	so	that	computations	will	be	done	as	locally	as
possible.	If	a	computation	is	simple	enough	to	do	with	a	1	kg	computer,	it’s
counterproductive	to	spread	it	out	over	a	galaxy-sized	computer,	since	waiting
for	the	information	to	be	shared	at	the	speed	of	light	after	each	computational
step	causes	a	ridiculous	delay	of	about	100,000	years	per	step.

What,	if	any,	of	this	future	information	processing	will	be	
conscious
	in	the
sense	of	involving	a	subjective	experience	is	a	controversial	and	fascinating
topic	which	we’ll	explore	in	chapter	8.	If	consciousness	requires	the	different
parts	of	the	system	to	be	able	to	communicate	with	one	another,	then	the
thoughts	of	larger	systems	are	by	necessity	slower.	Whereas	you	or	a	future
Earth-sized	supercomputer	can	have	many	thoughts	per	second,	a	galaxy-sized
mind	could	have	only	one	thought	every	hundred	thousand	years,	and	a	cosmic
mind	a	billion	light-years	in	size	would	only	have	time	to	have	about	ten
thoughts	in	total	before	dark	energy	fragmented	it	into	disconnected	parts.	On
the	other	hand,	these	few	precious	thoughts	and	accompanying	experiences
might	be	quite	deep!

Control	Hierarchies
If	thought	itself	is	organized	in	a	hierarchy	spanning	a	wide	range	of	scales,	then
what	about	power?	In	chapter	4,	we	explored	how	intelligent	entities	naturally
organize	themselves	into	power	hierarchies	in	Nash	equilibrium,	where	any
entity	would	be	worse	off	if	they	altered	their	strategy.	The	better	the
communication	and	transportation	technology	gets,	the	larger	these	hierarchies
can	grow.	If	superintelligence	one	day	expands	to	cosmic	scales,	what	will	its
power	hierarchy	be	like?	Will	it	be	freewheeling	and	decentralized	or	highly
authoritarian?	Will	cooperation	be	based	mainly	on	mutual	benefit	or	on
coercion	and	threats?
To	shed	light	on	these	questions,	let’s	consider	both	the	carrot	and	the	stick:
What	incentives	are	there	for	collaboration	on	cosmic	scales,	and	what	threats
might	be	used	to	enforce	it?

Controlling	with	the	Carrot
On	Earth,	
trade
	has	been	a	traditional	driver	of	cooperation	because	the	relative
difficulty	of	producing	things	varies	across	the	planet.	If	mining	a	kilogram	of
silver	costs	300	times	more	than	mining	a	kilogram	of	copper	in	one	region,	but
only	100	times	more	in	another,	they’ll	both	come	out	ahead	by	trading	200	kg
of	copper	against	1	kg	of	silver.	If	one	region	has	much	higher	technology	than
another,	both	can	similarly	benefit	from	trading	high-tech	goods	against	raw
materials.
However,	if	superintelligence	develops	technology	that	can	readily	rearrange
elementary	particles	into	any	form	of	matter	whatsoever,	then	it	will	eliminate
most	of	the	incentive	for	long-distance	trade.	Why	bother	shipping	silver
between	distant	solar	systems	when	it’s	simpler	and	quicker	to	transmute	copper
into	silver	by	rearranging	its	particles?	Why	bother	shipping	high-tech
machinery	between	galaxies	when	both	the	know-how	and	the	raw	materials

(any	matter	will	do)	exist	in	both	places?	My	guess	is	that	in	a	cosmos	teeming
with	superintelligence,	almost	the	only	commodity	worth	shipping	long
distances	will	be	
.	The	only	exception	might	be	matter	to	be	used	for
cosmic	engineering	projects—for	example,	to	counteract	the	aforementioned
destructive	tendency	of	dark	energy	to	tear	civilizations	apart.	As	opposed	to
traditional	human	trade,	this	matter	can	be	shipped	in	any	convenient	bulk	form
whatsoever,	perhaps	even	as	an	energy	beam,	since	the	receiving

superintelligence	can	rapidly	rearrange	it	into	whatever	objects	it	wants.
If	sharing	or	trading	of	information	emerges	as	the	main	driver	of	cosmic
cooperation,	then	what	sorts	of	information	might	be	involved?	Any	desirable
information	will	be	valuable	if	generating	it	requires	a	massive	and	time-
consuming	computational	effort.	For	example,	a	superintelligence	may	want
answers	to	hard	scientific	questions	about	the	nature	of	physical	reality,	hard
mathematical	questions	about	theorems	and	optimal	algorithms	and	hard

engineering	questions	about	how	to	best	build	spectacular	technology.
Hedonistic	life	forms	may	want	awesome	digital	entertainment	and	simulated
experiences,	and	cosmic	commerce	may	fuel	demand	for	some	form	of	cosmic
cryptocurrency	in	the	spirit	of	bitcoins.
Such	sharing	opportunities	may	incentivize	information	flow	not	only
between	entities	of	roughly	equal	power,	but	also	up	and	down	power

hierarchies,	say	between	solar-system-sized	nodes	and	a	galactic	hub	or	between
galaxy-sized	nodes	and	a	cosmic	hub.	The	nodes	might	want	this	for	the	pleasure
of	being	part	of	something	greater,	for	being	provided	with	answers	and
technologies	that	they	couldn’t	develop	alone	and	for	defense	against	external
threats.	They	may	also	value	the	promise	of	near	immortality	through	backup:
just	as	many	humans	take	solace	in	a	belief	that	their	minds	will	live	on	after
their	physical	bodies	die,	an	advanced	AI	may	appreciate	having	its	mind	and
knowledge	live	on	in	a	hub	supercomputer	after	its	original	physical	hardware
has	depleted	its	energy	reserves.
Conversely,	the	hub	may	want	its	nodes	to	help	it	with	massive	long-term
computing	tasks	where	the	results	aren’t	urgently	needed,	so	that	it’s	worth
waiting	thousands	or	millions	of	years	for	the	answers.	As	we	explored	above,
the	hub	may	also	want	its	nodes	to	help	carry	out	massive	cosmic	engineering

projects	such	as	counteracting	destructive	dark	energy	by	moving	galactic	mass
concentrations	together.	If	traversable	wormholes	turn	out	to	be	possible	and
buildable,	then	a	top	priority	of	a	hub	will	probably	be	constructing	a	network	of
them	to	thwart	dark	energy	and	keep	its	empire	connected	indefinitely.	The
questions	of	what	ultimate	goals	a	cosmic	superintelligence	may	have	is	a
fascinating	and	controversial	one	that	we’ll	explore	further	in	chapter	7.

Controlling	with	the	Stick
Terrestrial	empires	usually	compel	their	subordinates	to	cooperate	by	using	both
the	carrot	and	the	stick.	While	subjects	of	the	Roman	Empire	valued	the
technology,	infrastructure	and	defense	that	they	were	offered	as	a	reward	for
their	cooperation,	they	also	feared	the	inevitable	repercussions	of	rebelling	or	not
paying	taxes.	Because	of	the	long	time	required	to	send	troops	from	Rome	to
outlying	provinces,	part	of	the	intimidation	was	delegated	to	local	troops	and
loyal	officials	empowered	to	inflict	near-instantaneous	punishments.	A
superintelligent	hub	could	use	the	analogous	strategy	of	deploying	a	network	of
loyal	guards	throughout	its	cosmic	empire.	Since	superintelligent	subjects	can	be
hard	to	control,	the	simplest	viable	strategy	may	be	using	AI	guards	that	are
programmed	to	be	100%	loyal	by	virtue	of	being	relatively	dumb,	simply
monitoring	whether	all	rules	are	obeyed	and	automatically	triggering	a
doomsday	device	if	not.

Suppose,	for	example,	that	the	hub	AI	arranges	for	a	white	dwarf	to	be	placed
in	the	vicinity	of	a	solar-system-sized	civilization	that	it	wishes	to	control.	A
white	dwarf	is	the	burnt-out	husk	of	a	modestly	heavy	star.	Consisting	largely	of
carbon,	it	resembles	a	giant	diamond	in	the	sky,	and	is	so	compact	that	it	can
weigh	more	than	the	Sun	while	being	smaller	than	Earth.	The	Indian	physicist
Subrahmanyan	Chandrasekhar	famously	proved	that	if	you	keep	adding	mass	to
it	until	it	surpasses	the	
Chandrasekhar	limit,

	about	1.4	times	the	mass	of	our
Sun,	it	will	undergo	a	cataclysmic	thermonuclear	detonation	known	as	a
supernova	of	type	1A.	If	the	hub	AI	has	callously	arranged	for	this	white	dwarf
to	be	extremely	close	to	its	Chandrasekhar	limit,	the	guard	AI	could	be	effective
even	if	it	were	extremely	dumb	(indeed,	largely	because	it	was	so	dumb):	it
could	be	programmed	to	simply	verify	that	the	subjugated	civilization	had
delivered	its	monthly	quota	of	cosmic	bitcoins,	mathematical	proofs	or	whatever
other	taxes	were	stipulated,	and	if	not,	toss	enough	mass	onto	the	white	dwarf	to

ignite	the	supernova	and	blow	the	entire	region	to	smithereens.
Galaxy-sized	civilizations	may	be	similarly	controllable	by	placing	large
numbers	of	compact	objects	into	tight	orbits	around	the	monster	black	hole	at	the
galaxy	center,	and	threatening	to	transform	these	masses	into	gas,	for	instance	by
colliding	them.	This	gas	would	then	start	feeding	the	black	hole,	transforming	it
into	a	powerful	quasar,	potentially	rendering	much	of	the	galaxy	uninhabitable.

In	summary,	there	are	strong	incentives	for	future	life	to	cooperate	over
cosmic	distances,	but	it’s	a	wide-open	question	whether	such	cooperation	will	be
based	mainly	on	mutual	benefits	or	on	brutal	threats—the	limits	imposed	by
physics	appear	to	allow	both	scenarios,	so	the	outcome	will	depend	on	the
prevailing	goals	and	values.	We’ll	explore	our	ability	to	influence	these	goals
and	values	of	future	life	in	chapter	7.

When	Civilizations	Clash
So	far,	we’ve	only	discussed	scenarios	where	life	expands	into	our	cosmos	from
a	single	intelligence	explosion.	But	what	happens	if	life	evolves	independently	in
more	than	one	place	and	two	expanding	civilizations	meet?
If	you	consider	a	random	solar	system,	there’s	some	probability	that	life	will
evolve	on	one	of	its	planets,	develop	advanced	technology	and	expand	into
space.	This	probability	seems	to	be	greater	than	zero	since	technological	life	has
evolved	here	in	our	Solar	System	and	the	laws	of	physics	appear	to	allow	space
settlement.	If	space	is	large	enough	(indeed,	the	theory	of	cosmological	inflation
suggests	it	to	be	vast	or	infinite),	then	there	will	be	many	such	expanding
civilizations,	as	illustrated	in	
figure	6.10
.	Jay	Olson’s	above-mentioned	paper
includes	an	elegant	analysis	of	such	expanding	cosmic	biospheres,	and	Toby	Ord
has	performed	a	similar	analysis	with	colleagues	at	the	Future	of	Humanity

Institute.	Viewed	in	three	dimensions,	these	cosmic	biospheres	are	quite	literally
spheres	as	long	as	civilizations	expand	with	the	same	speed	in	all	directions.	In
spacetime,	they	look	like	the	upper	part	of	the	champagne	glass	in	
figure	6.7
,
because	dark	energy	ultimately	limits	how	many	galaxies	each	civilization	can
reach.
If	the	distance	between	neighboring	space-settling	civilizations	is	much	larger
than	dark	energy	lets	them	expand,	then	they’ll	never	come	into	contact	with

figure	6.7
,
reach.
each	other	or	even	find	out	about	each	other’s	existence,	so	they’ll	feel	as	if
they’re	alone	in	the	cosmos.	If	our	cosmos	is	more	fecund	so	that	neighbors	are
closer	together,	however,	some	civilizations	will	eventually	overlap.	What
happens	in	these	overlap	regions?	Will	there	be	cooperation,	competition	or	war?

Figure	6.10:	If	life	evolves	independently	at	multiple	points	in	spacetime	(places	and	times)	and
starts	colonizing	space,	then	space	will	contain	a	network	of	expanding	cosmic	biospheres,	each
of	which	resembles	the	top	of	the	champagne	glass	from	
figure	6.7
.	The	bottom	of	each
biosphere	represents	the	place	and	time	when	colonization	began.	The	opaque	and	translucent
champagne	glasses	correspond	to	colonization	at	50%	and	100%	of	the	speed	of	light,
respectively,	and	overlaps	show	where	independent	civilizations	meet.
Europeans	were	able	to	conquer	Africa	and	the	Americas	because	they	had
superior	technology.	In	contrast,	it’s	plausible	that	long	before	two
superintelligent	civilizations	encounter	one	another,	their	technologies	will
plateau	at	the	same	level,	limited	merely	by	the	laws	of	physics.	This	makes	it
seem	unlikely	that	one	superintelligence	could	easily	conquer	the	other	even	if	it
wanted	to.	Moreover,	if	their	goals	have	evolved	to	be	relatively	aligned,	then

they	may	have	little	reason	to	desire	conquest	or	war.	For	example,	if	they’re

both	trying	to	prove	as	many	beautiful	theorems	as	possible	and	invent	as	clever
algorithms	as	possible,	they	can	simply	share	their	findings	and	both	be	better
off.	After	all,	information	is	very	different	from	the	resources	that	humans
usually	fight	over,	in	that	you	can	simultaneously	give	it	away	and	keep	it.
Some	expanding	civilizations	might	have	goals	that	are	essentially	immutable,
such	as	those	of	a	fundamentalist	cult	or	a	spreading	virus.	However,	it’s	also
plausible	that	some	advanced	civilizations	are	more	like	open-minded	humans—
willing	to	adjust	their	goals	when	presented	with	sufficiently	compelling
arguments.	If	two	of	them	meet,	there	will	be	a	clash	not	of	weapons	but	of
ideas,	where	the	most	persuasive	one	prevails	and	has	its	goals	spread	at	the
speed	of	light	through	the	region	controlled	by	the	other	civilization.
Assimilating	your	neighbors	is	a	faster	expansion	strategy	than	settlement,	since
your	sphere	of	influence	can	spread	at	the	speed	with	which	ideas	move	(the

speed	of	light	using	telecommunication),	whereas	physical	settlement	inevitably
progresses	slower	than	the	speed	of	light.	This	assimilation	will	not	be	forced
such	as	that	infamously	employed	by	the	Borg	in	
Star	Trek,
	but	voluntary	based
on	the	persuasive	superiority	of	ideas,	leaving	the	assimilated	better	off.
We’ve	seen	that	the	future	cosmos	can	contain	rapidly	expanding	bubbles	of
two	kinds:	expanding	civilizations	and	those	death	bubbles	that	expand	at	light
speed	and	make	space	uninhabitable	by	destroying	all	our	elementary	particles.

Star	Trek,
An	ambitious	civilization	can	thus	encounter	three	kinds	of	regions:	uninhabited
ones,	life	bubbles	and	death	bubbles.	If	it	fears	uncooperative	rival	civilizations,
it	has	a	strong	incentive	to	launch	a	rapid	“land	grab”	and	settle	the	uninhabited
regions	before	the	rivals	do.	However,	it	has	the	same	expansionist	incentive
even	if	there	are	no	other	civilizations,	simply	to	acquire	resources	before	dark
energy	makes	them	unreachable.	We	just	saw	how	bumping	into	another
expanding	civilization	can	be	either	better	or	worse	than	bumping	into

uninhabited	space,	depending	on	how	cooperative	and	open-minded	this
neighbor	is.	However,	it’s	better	to	bump	into	any	expansionist	civilization	(even
one	trying	to	convert	your	civilization	into	paper	clips)	than	a	death	bubble,
which	will	continue	expanding	at	the	speed	of	light	regardless	of	whether	you	try
to	fight	it	or	reason	with	it.	Our	only	protection	against	death	bubbles	is	dark
energy,	which	prevents	distant	ones	from	ever	reaching	us.	So	if	death	bubbles
are	indeed	common,	then	dark	energy	is	actually	not	our	enemy	but	our	friend.

Are	We	Alone?
Many	people	take	for	granted	that	there’s	advanced	life	throughout	much	of	our
Universe,	so	that	human	extinction	wouldn’t	matter	much	from	a	cosmic
perspective.	After	all,	why	should	we	worry	about	wiping	ourselves	out	if	some
inspiring	
Star	Trek
–like	civilization	would	soon	swoop	in	and	re-seed	our	Solar
System	with	life,	perhaps	even	using	their	advanced	technology	to	reconstruct
and	resuscitate	us?	I	view	this	
Star	Trek
	assumption	as	dangerous,	because	it	can
lull	us	into	a	false	sense	of	security	and	make	our	civilization	apathetic	and
reckless.	Indeed,	I	think	that	this	assumption	that	we’re	not	alone	in	our
Universe	is	not	only	dangerous	but	also	probably	false.
This	is	a	minority	view,
*9
	and	I	may	well	be	wrong,	but	it’s	at	the	very	least	a
possibility	that	we	can’t	currently	dismiss,	which	gives	us	a	moral	imperative	to
play	it	safe	and	not	drive	our	civilization	extinct.
When	I	give	lectures	about	cosmology,	I	often	ask	the	audience	to	raise	their

*9
hands	if	they	think	there’s	intelligent	life	elsewhere	in	our	Universe	(the	region
of	space	from	which	light	has	reached	us	so	far	during	the	13.8	billion	years
since	our	Big	Bang).	Infallibly,	almost	everyone	does,	from	kindergartners	to
college	students.	When	I	ask	why,	the	basic	answer	I	tend	to	get	is	that	our
Universe	is	so	huge	that	there’s	got	to	be	life	somewhere,	at	least	statistically
speaking.	Let’s	take	a	closer	look	at	this	argument	and	pinpoint	its	weakness.

It	all	comes	down	to	one	number:	the	typical	distance	between	a	civilization	in
	and	its	nearest	neighbor.	If	this	distance	is	much	larger	than	20
billion	light-years,	we	should	expect	to	be	alone	in	our	Universe	(the	part	of
space	from	which	light	has	reached	us	during	the	13.8	billion	years	since	our	Big
Bang),	and	to	never	make	contact	with	aliens.	So	what	should	we	expect	for	this
distance?	We’re	quite	clueless.	This	means	that	the	distance	to	our	neighbor	is	in
the	ballpark	of	1000
…

…
000	meters,	where	the	total	number	of	zeroes	could
reasonably	be	21,	22,	23,…,	100,	101,	102	or	more
—
but	probably	not	much
smaller	than	21,	since	we	haven’t	yet	seen	compelling	evidence	of	aliens	(see
figure	6.11
).	For	our	nearest	neighbor	civilization	to	be	within	our	Universe,
whose	radius	is	about	10
26
	meters,	the	number	of	zeroes	can’t	exceed	26,	and	the
probability	of	the	number	of	zeroes	falling	in	the	narrow	range	between	22	and
26	is	rather	small.	This	is	why	I	think	we’re	alone	in	our	Universe.

Figure	6.11:	Are	we	alone?	The	huge	uncertainties	about	how	life	and	intelligence	evolved
suggest	that	our	nearest	neighbor	civilization	in	space	could	reasonably	be	anywhere	along	the
horizontal	axis	above,	making	it	unlikely	that	it’s	in	the	narrow	range	between	the	edge	of	our
Galaxy	(about	10
21
	meters	away)	and	the	edge	of	our	Universe	(about	10
26
	meters	away).	If	it
were	much	closer	than	this	range,	there	should	be	so	many	other	advanced	civilizations	in	our
Galaxy	that	we’d	probably	have	noticed,	which	suggests	that	we’re	in	fact	alone	in	our	Universe.
I	give	a	detailed	justification	of	this	argument	in	my	book	
Our	Mathematical
Universe,
	so	I	won’t	rehash	it	here,	but	the	basic	reason	for	why	we’re	clueless
about	this	neighbor	distance	is	that	we’re	in	turn	clueless	about	the	probability	of
intelligent	life	arising	in	a	given	place.	As	the	American	astronomer	Frank	Drake
pointed	out,	this	probability	can	be	calculated	by	multiplying	together	the

Universe,
probability	of	there	being	a	habitable	environment	there	(say	an	appropriate
planet),	the	probability	that	life	will	form	there	and	the	probability	that	this	life
will	evolve	to	become	intelligent.	When	I	was	a	grad	student,	we	had	no	clue
about	any	of	these	three	probabilities.	After	the	past	two	decades’	dramatic
discoveries	of	planets	orbiting	other	stars,	it	now	seems	likely	that	habitable
planets	are	abundant,	with	billions	in	our	own	Galaxy	alone.	The	probability	of

evolving	life	and	then	intelligence,	however,	remains	extremely	uncertain:	some
experts	think	that	one	or	both	are	rather	inevitable	and	occur	on	most	habitable
planets,	while	others	think	that	one	or	both	are	extremely	rare	because	of	one	or
more	evolutionary	bottlenecks	that	require	a	wild	stroke	of	luck	to	pass	through.
Some	proposed	bottlenecks	involve	chicken-and-egg	problems	at	the	earliest

stages	of	self-reproducing	life:	for	example,	for	a	modern	cell	to	build	a
ribosome,	the	highly	complex	molecular	machine	that	reads	our	genetic	code	and
builds	our	proteins,	it	needs	another	ribosome,	and	it’s	not	obvious	that	the	very
first	ribosome	could	evolve	gradually	from	something	simpler.
10
	Other	proposed
bottlenecks	involve	the	development	of	higher	intelligence.	For	example,
although	dinosaurs	ruled	Earth	for	over	100	million	years,	a	thousand	times
longer	than	we	modern	humans	have	been	around,	evolution	didn’t	seem	to
inevitably	push	them	toward	higher	intelligence	and	inventing	telescopes	or
computers.
Some	people	counter	my	argument	by	saying	that,	yes,	intelligent	life	
could
	be
very	rare,	but	in	fact	it	isn’t—our	Galaxy	is	teeming	with	intelligent	life	that
mainstream	scientists	are	simply	not	noticing.	Perhaps	aliens	have	already
visited	Earth,	as	UFO	enthusiasts	claim.	Perhaps	aliens	haven’t	visited	Earth,	but

could
	be
they’re	out	there	and	they’re	deliberately	hiding	from	us	(this	has	been	called	the
“zoo	hypothesis”	by	the	U.S.	astronomer	John	A.	Ball,	and	features	in	sci-fi
classics	such	as	Olaf	Stapledon’s	
Star	Maker
).	Or	perhaps	they’re	out	there
without	deliberately	hiding:	they’re	simply	not	interested	in	space	settlement	or
large	engineering	projects	that	we’d	have	noticed.
Sure,	we	need	to	keep	an	open	mind	about	these	possibilities,	but	since	there’s
no	generally	accepted	evidence	for	any	of	them,	we	also	need	to	take	seriously

Star	Maker
the	alternative:	that	we’re	alone.	Moreover,	I	think	we	shouldn’t	underestimate
the	diversity	of	advanced	civilizations	by	assuming	that	they	all	share	goals	that
make	them	go	unnoticed:	we	saw	above	that	resource	acquisition	is	quite	a
natural	goal	for	a	civilization	to	have,	and	for	us	to	notice,	all	it	takes	is	
one
civilization	deciding	to	overtly	settle	all	it	can	and	hence	engulf	our	Galaxy	and
beyond.	Confronted	with	the	fact	that	there	are	millions	of	habitable	Earth-like
planets	in	our	Galaxy	that	are	billions	of	years	older	than	Earth,	giving	ample

one
time	for	ambitious	inhabitants	to	settle	the	Galaxy,	we	therefore	can’t	dismiss	the
most	obvious	interpretation:	that	the	origin	of	life	requires	a	random	fluke	so
unlikely	that	they’re	all	uninhabited.
If	life	is	
not
	rare	after	all,	we	may	soon	know.	Ambitious	astronomical	surveys
are	searching	atmospheres	of	Earth-like	planets	for	evidence	of	oxygen	produced
by	life.	In	parallel	with	this	search	for	
any
	life,	the	search	for	
intelligent
	life	was
recently	boosted	by	the	Russian	philanthropist	Yuri	Milner’s	$100	million

If	life	is	
not
any
	life	was
project	“Breakthrough	Listen.”
It’s	important	not	to	be	overly	anthropocentric	when	searching	for	advanced

life:	if	we	discover	an	extraterrestrial	civilization,	it’s	likely	to	already	have	gone
superintelligent.	As	Martin	Rees	put	it	in	a	recent	essay,	“the	history	of	human
technological	civilization	is	measured	in	centuries—and	it	may	be	only	one	or
two	more	centuries	before	humans	are	overtaken	or	transcended	by	inorganic
intelligence,	which	will	then	persist,	continuing	to	evolve,	for	billions	of
years….We	would	be	most	unlikely	to	‘catch’	it	in	the	brief	sliver	of	time	when
it	took	organic	form.”
11
	I	agree	with	Jay	Olson’s	conclusion	in	his
aforementioned	space	settlement	paper:	“We	regard	the	possibility	that	advanced
intelligence	will	make	use	of	the	universe’s	resources	to	simply	populate	existing
earthlike	planets	with	advanced	versions	of	humans	as	an	unlikely	endpoint	to
the	progression	of	technology.”	So	when	you	imagine	aliens,	don’t	think	of	little
green	fellows	with	two	arms	and	two	legs,	but	think	of	the	superintelligent
spacefaring	life	we	explored	earlier	in	this	chapter.

11
Although	I’m	a	strong	supporter	of	all	the	ongoing	searches	for	extraterrestrial
life,	which	are	shedding	light	on	one	of	the	most	fascinating	questions	in
science,	I’m	secretly	hoping	that	they’ll	all	fail	and	find	nothing!	The	apparent
incompatibility	between	the	abundance	of	habitable	planets	in	our	Galaxy	and
the	lack	of	extraterrestrial	visitors,	known	as	the	
Fermi	paradox,
	suggests	the
existence	of	what	the	economist	Robin	Hanson	calls	a	“Great	Filter,”	an

evolutionary/technological	roadblock	somewhere	along	the	developmental	path
from	the	non-living	matter	to	space-settling	life.	If	we	discover	independently
evolved	life	elsewhere,	this	would	suggest	that	primitive	life	isn’t	rare,	and	that
the	roadblock	lies	after	our	current	human	stage	of	development—perhaps
because	space	settlement	is	impossible,	or	because	almost	all	advanced
civilizations	self-destruct	before	they’re	able	to	go	cosmic.	I’m	therefore
crossing	my	fingers	that	all	searches	for	extraterrestrial	life	find	nothing:	this	is

consistent	with	the	scenario	where	evolving	intelligent	life	is	rare	but	we	humans
got	lucky,	so	that	we	have	the	roadblock	behind	us	and	have	extraordinary	future
potential.

Outlook
So	far,	we’ve	spent	this	book	exploring	the	history	of	life	in	our	Universe,	from
its	humble	beginnings	billions	of	years	ago	to	possible	grand	futures	billions	of
years	from	now.	If	our	current	AI	development	eventually	triggers	an
intelligence	explosion	and	optimized	space	settlement,	it	will	be	an	explosion	in
a	truly	cosmic	sense:	after	spending	billions	of	years	as	an	almost	negligibly
small	perturbation	on	an	indifferent	lifeless	cosmos,	life	suddenly	explodes	onto
the	cosmic	arena	as	a	spherical	blast	wave	expanding	near	the	speed	of	light,
never	slowing	down,	and	igniting	everything	in	its	path	with	the	spark	of	life.
Such	optimistic	views	of	the	importance	of	life	in	our	cosmic	future	have	been
eloquently	articulated	by	many	of	the	thinkers	we’ve	encountered	in	this	book.
Because	sci-fi	authors	are	often	dismissed	as	unrealistic	romantic	dreamers,	I
find	it	ironic	that	most	sci-fi	and	scientific	writing	about	space	settlement	now
appears	too	
pessimistic

	in	the	light	of	superintelligence.	For	example,	we	saw
how	intergalactic	travel	becomes	much	easier	once	people	and	other	intelligent
entities	can	be	transmitted	in	digital	form,	potentially	making	us	masters	of	our
own	destiny	not	only	in	our	Solar	System	or	the	Milky	Way	Galaxy,	but	also	in
the	cosmos.
Above	we	considered	the	very	real	possibility	that	we’re	the	only	high-tech
civilization	in	our	Universe.	Let’s	spend	the	rest	of	this	chapter	exploring	this

scenario,	and	the	huge	moral	responsibility	it	entails.	This	means	that	after	13.8
billion	years,	life	in	our	Universe	has	reached	a	fork	in	the	road,	facing	a	choice
between	flourishing	throughout	the	cosmos	or	going	extinct.	If	we	don’t	keep
improving	our	technology,	the	question	isn’t	whether	humanity	will	go	extinct,
but	
how
.	What	will	get	us	first—an	asteroid,	a	supervolcano,	the	burning	heat	of
the	aging	Sun,	or	some	other	calamity	(see	
figure	5.1
)?	Once	we’re	gone,	the

but	
how
figure	5.1
cosmic	drama	predicted	by	Freeman	Dyson	will	play	on	without	spectators:
barring	a	cosmocalypse,	stars	burn	out,	galaxies	fade	and	black	holes	evaporate,
each	ending	its	life	with	a	huge	explosion	that	releases	over	a	million	times	as
much	energy	as	the	Tsar	Bomba,	the	most	powerful	hydrogen	bomb	ever	built.
As	Freeman	put	it:	“The	cold	expanding	universe	will	be	illuminated	by
occasional	fireworks	for	a	very	long	time.”	Alas,	this	fireworks	display	will	be	a
meaningless	waste,	with	nobody	there	to	enjoy	it.

Without	technology,	our	human	extinction	is	imminent	in	the	cosmic	context
of	tens	of	billions	of	years,	rendering	the	entire	drama	of	life	in	our	Universe
merely	a	brief	and	transient	flash	of	beauty,	passion	and	meaning	in	a	near
eternity	of	meaninglessness	experienced	by	nobody.	What	a	wasted	opportunity
that	would	be!	If	instead	of	eschewing	technology,	we	choose	to	embrace	it,	then
we	up	the	ante:	we	gain	the	potential	both	for	life	to	survive	and	flourish	and	for
life	to	go	extinct	even	sooner,	self-destructing	due	to	poor	planning	(see	
figure
5.1
).	My	vote	is	for	embracing	technology,	and	proceeding	not	with	blind	faith	in
what	we	build,	but	with	caution,	foresight	and	careful	planning.
After	13.8	billion	years	of	cosmic	history,	we	find	ourselves	in	a
breathtakingly	beautiful	Universe,	which	through	us	humans	has	come	alive	and
started	becoming	aware	of	itself.	We’ve	seen	that	life’s	future	potential	in	our
Universe	is	grander	than	the	wildest	dreams	of	our	ancestors,	tempered	by	an

figure
5.1
equally	real	potential	for	intelligent	life	to	go	permanently	extinct.	Will	life	in
our	Universe	fulfill	its	potential	or	squander	it?	This	depends	to	a	great	extent	on
what	we	humans	alive	today	do	during	our	lifetime,	and	I’m	optimistic	that	we
can	make	the	future	of	life	truly	awesome	if	we	make	the	right	choices.	What
should	we	want	and	how	can	we	attain	those	goals?	Let’s	spend	the	rest	of	the
book	exploring	some	of	the	most	difficult	challenges	involved	and	what	we	can
do	about	them.

•
Compared	to	cosmic	timescales	of	billions	of	years,	an	intelligence	explosion	is	a
sudden	event	where	technology	rapidly	plateaus	at	a	level	limited	only	by	the	laws
of	physics.
•
This	technological	plateau	is	vastly	higher	than	today’s	technology,	allowing	a	given
amount	of	matter	to	generate	about	ten	billion	times	more	energy	(using	sphalerons
or	black	holes),	store	12–18	orders	of	magnitude	more	information	or	compute	31–
41	orders	of	magnitude	faster—or	to	be	converted	to	any	other	desired	form	of
matter.
•
Superintelligent	life	would	not	only	make	such	dramatically	more	efficient	use	of	its
existing	resources,	but	would	also	be	able	to	grow	today’s	biosphere	by	about	32
orders	of	magnitude	by	acquiring	more	resources	through	cosmic	settlement	at	near
light	speed.
•
Dark	energy	limits	the	cosmic	expansion	of	superintelligent	life	and	also	protects	it
from	distant	expanding	death	bubbles	or	hostile	civilizations.	The	threat	of	dark

matter.
•
•
energy	tearing	cosmic	civilizations	apart	motivates	massive	cosmic	engineering
projects,	including	wormhole	construction	if	this	turns	out	to	be	feasible.
•
The	main	commodity	shared	or	traded	across	cosmic	distances	is	likely	to	be
information.
•
Barring	wormholes,	the	light-speed	limit	on	communication	poses	severe	challenges
for	coordination	and	control	across	a	cosmic	civilization.	A	distant	central	hub	may
incentivize	its	superintelligent	“nodes”	to	cooperate	either	through	rewards	or
through	threats,	say	by	deploying	a	local	guard	AI	programmed	to	destroy	the	node

•
•
by	setting	off	a	supernova	or	quasar	unless	the	rules	are	obeyed.
•
The	collision	of	two	expanding	civilizations	may	result	in	assimilation,	cooperation
or	war,	where	the	latter	is	arguably	less	likely	than	it	is	between	today’s	civilizations.
•
Despite	popular	belief	to	the	contrary,	it’s	quite	plausible	that	we’re	the	only	life
form	capable	of	making	our	observable	Universe	come	alive	in	the	future.
•
If	we	don’t	improve	our	technology,	the	question	isn’t	whether	humanity	will	go

•
•
•
extinct,	but	merely	how:	will	an	asteroid,	a	supervolcano,	the	burning	heat	of	the
aging	Sun	or	some	other	calamity	get	us	first?
•
If	we	do	keep	improving	our	technology	with	enough	care,	foresight	and	planning	to
avoid	pitfalls,	life	has	the	potential	to	flourish	on	Earth	and	far	beyond	for	many
billions	of	years,	beyond	the	wildest	dreams	of	our	ancestors.

*1
	
If	you	work	in	the	energy	sector,	you	may	be	used	to	instead	defining	efficiency	as	the	fraction	of	the
energy	released	that’s	in	a	useful	form.
*2
	
If	no	suitable	nature-made	black	hole	can	be	found	in	the	nearby	universe,	a	new	one	can	be	created	by
putting	lots	of	matter	in	a	sufficiently	small	space.
*3
	
This	is	a	slight	oversimplification,	because	Hawking	radiation	also	includes	some	particles	from	which
it’s	hard	to	extract	useful	work.	Large	black	holes	are	only	90%	efficient,	because	about	10%	of	the
energy	is	radiated	in	the	form	of	gravitons:	extremely	shy	particles	that	are	almost	impossible	to	detect,
let	alone	extract	useful	work	from.	As	the	black	hole	continues	evaporating	and	shrinking,	the	efficiency
drops	further	because	the	Hawking	radiation	starts	including	neutrinos	and	other	massive	particles.
*4
	
For	Douglas	Adams	fans	out	there,	note	that	this	is	an	elegant	question	giving	the	answer	to	the	question
of	life,	the	universe	and	everything.	More	precisely,	the	efficiency	is	1	–	1
/

*4
	
/
√
3
‾
	
≈
	42%.
*5
	
If	you	feed	the	black	hole	by	placing	a	gas	cloud	around	it	that	rotates	slowly	in	the	same	direction,	then
this	gas	will	spin	ever	faster	as	it’s	pulled	in	and	eaten,	boosting	the	black	hole’s	rotation,	just	as	a	figure-
skater	spins	faster	when	pulling	in	her	arms.	This	may	keep	the	hole	maximally	spinning,	enabling	you	to
extract	first	42%	of	the	gas	energy	and	then	29%	of	the	remainder,	for	a	total	efficiency	of	42%	+	(1-
42%)×29%	
≈
	59%.
*6

/
√
3
‾
	
≈
	42%.
*5
	
42%)×29%	
≈
	59%.
*6
	
It	needs	to	get	hot	enough	to	re-unify	the	electromagnetic	and	weak	forces,	which	happens	when
particles	move	about	as	fast	as	when	they’ve	been	accelerated	by	200	billion	volts	in	a	particle	collider.
*7
	
Above	we	only	discussed	matter	made	of	atoms.	There	is	about	six	times	more	dark	matter,	but	it’s	very
elusive	and	hard	to	catch,	routinely	flying	straight	through	Earth	and	out	the	other	side,	so	it	remains	to	be
seen	whether	it’s	possible	for	future	life	to	capture	and	utilize	it.
*8

≈
	59%.
*6
	
*7
	
*8
	
The	cosmic	mathematics	comes	out	remarkably	simple:	if	the	civilization	expands	through	the
expanding	space	not	at	the	speed	of	light	
c
	but	at	some	slower	speed	
v,
	the	number	of	galaxies	settled	gets
reduced	by	a	factor	(
v/c
)
3
.	This	means	that	slowpoke	civilizations	get	severely	penalized,	with	one	that
expands	10	times	slower	ultimately	settling	1,000	times	fewer	galaxies.
*9
	
However,	John	Gribbin	comes	to	a	similar	conclusion	in	his	2011	book	
Alone	in	the	Universe
.	For	a

*8
	
c
v,
v/c
)
3
*9
	
.	For	a
spectrum	of	intriguing	perspectives	on	this	question,	I	also	recommend	Paul	Davies’	2011	book	
The	Eerie
Silence
.

Chapter	7
Goals
The	mystery	of	human	existence	lies	not	in	just	staying	alive,	but	in	finding
	
something	to
live	for.
Fyodor	Dostoyevsky,	
The	Brothers	Karamazov
Life	is	a	journey,	not	a	destination.
Ralph	Waldo	Emerson
If	I	had	to	summarize	in	a	single	word	what	the	thorniest	AI	controversies	are
about,	it	would	be	“goals”:	Should	we	give	AI	goals,	and	if	so,	whose	goals?
How	can	we	give	AI	goals?	Can	we	ensure	that	these	goals	are	retained	even	if
the	AI	gets	smarter?	Can	we	change	the	goals	of	an	AI	that’s	smarter	than	us?
What	are	our	ultimate	goals?	These	questions	are	not	only	difficult,	but	also
crucial	for	the	future	of	life:	if	we	don’t	know	what	we	want,	we’re	less	likely	to
get	it,	and	if	we	cede	control	to	machines	that	don’t	share	our	goals,	then	we’re
likely	to	get	what	we	don’t	want.

To	shed	light	on	these	questions,	let’s	first	explore	the	ultimate	origin	of	goals.
When	we	look	around	us	in	the	world,	some	processes	strike	us	as	
goal-oriented
while	others	don’t.	Consider,	for	example,	the	process	of	a	soccer	ball	being
kicked	for	the	game-winning	shot.	The	behavior	of	the	ball	itself	does	not	appear
goal-oriented,	and	is	most	economically	explained	in	terms	of	Newton’s	laws	of
motion,	as	a	reaction	to	the	kick.	The	behavior	of	the	player,	on	the	other	hand,	is
most	economically	explained	not	mechanistically	in	terms	of	atoms	pushing	each
other	around,	but	in	terms	of	her	having	the	
goal
	of	maximizing	her	team’s
score.	How	did	such	goal-oriented	behavior	emerge	from	the	physics	of	our	early
Universe,	which	consisted	merely	of	a	bunch	of	particles	bouncing	around
seemingly	without	goals?
Intriguingly,	the	ultimate	roots	of	goal-oriented	behavior	can	be	found	in	the
laws	of	physics	themselves,	and	manifest	themselves	even	in	simple	processes

goal
that	don’t	involve	life.	If	a	lifeguard	rescues	a	swimmer,	as	in	
figure	7.1
,	we
expect	her	not	to	go	in	a	straight	line,	but	to	run	a	bit	further	along	the	beach
where	she	can	go	faster	than	in	the	water,	thereby	turning	slightly	when	she
enters	the	water.	We	naturally	interpret	her	choice	of	trajectory	as	goal-oriented,
since	out	of	all	possible	trajectories,	she’s	deliberately	choosing	the	optimal	one
that	gets	her	to	the	swimmer	as	fast	as	possible.	Yet	a	simple	light	ray	similarly
bends	when	it	enters	water	(see

figure	7.1
,	we
bends	when	it	enters	water	(see	
figure	7.1
),	also	minimizing	the	travel	time	to	its
destination!	How	can	this	be?
This	is	known	in	physics	as	
Fermat’s	principle,
	articulated	in	1662,	and	it
provides	an	alternative	way	of	predicting	the	behavior	of	light	rays.	Remarkably,
physicists	have	since	discovered	that	
all
	laws	of	classical	physics	can	be
mathematically	reformulated	in	an	analogous	way:	out	of	all	ways	that	nature
could	choose	to	do	something,	it	prefers	the	optimal	way,	which	typically	boils
down	to	minimizing	or	maximizing	some	quantity.	There	are	two	mathematically

all
equivalent	ways	of	describing	each	physical	law:	either	as	the	past	causing	the
future,	or	as	nature	optimizing	something.	Although	the	second	way	usually	isn’t
taught	in	introductory	physics	courses	because	the	math	is	tougher,	I	feel	that	it’s
more	elegant	and	profound.	If	a	person	is	trying	to	optimize	something	(for
example,	their	score,	their	wealth	or	their	happiness)	we’ll	naturally	describe

their	pursuit	of	it	as	goal-oriented.	So	if	nature	itself	is	trying	to	optimize
something,	then	no	wonder	that	goal-oriented	behavior	can	emerge:	it	was
hardwired	in	from	the	start,	in	the	very	laws	of	physics.

Figure	7.1:	To	rescue	a	swimmer	as	fast	as	possible,	a	lifeguard	won’t	go	in	a	straight	line
(dashed),	but	a	bit	further	along	the	beach	where	she	can	go	faster	than	in	the	water.	A	light	ray
similarly	bends	when	entering	the	water	to	reach	its	destination	as	fast	as	possible.
One	famous	quantity	that	nature	strives	to	maximize	is	
entropy,
	which	loosely
speaking	measures	how	messy	things	are.	The	second	law	of	thermodynamics
states	that	entropy	tends	to	increase	until	it	reaches	its	maximum	possible	value.
Ignoring	the	effects	of	gravity	for	now,	this	maximally	messy	end	state	is	called
heat	death,
	and	corresponds	to	everything	being	spread	out	in	boring	perfect
uniformity,	with	no	complexity,	no	life	and	no	change.	When	you	pour	cold	milk
into	hot	coffee,	for	example,	your	beverage	appears	to	march	irreversibly	toward
its	own	personal	heat	death	goal,	and	before	long,	it’s	all	just	a	uniform
lukewarm	mixture.	If	a	living	organism	dies,	its	entropy	also	starts	to	rise,	and

before	long,	the	arrangement	of	its	particles	tends	to	get	much	less	organized.
Nature’s	apparent	goal	to	increase	entropy	helps	explain	why	time	seems	to
have	a	preferred	direction,	making	movies	look	unrealistic	if	played	backward:	if
you	drop	a	glass	of	wine,	you	expect	it	to	shatter	against	the	floor	and	increase
global	messiness	(entropy).	If	you	then	saw	it	
unshatter
	and	come	flying	back	up
to	your	hand	intact	(decreasing	entropy),	you	probably	wouldn’t	drink	it,
figuring	you’d	already	had	a	glass	too	many.

unshatter
When	I	first	learned	about	our	inexorable	progression	toward	heat	death,	I
found	it	rather	depressing,	and	I	wasn’t	alone:	thermodynamics	pioneer	Lord
Kelvin	wrote	in	1841	that	“the	result	would	inevitably	be	a	state	of	universal	rest

and	death,”	and	it’s	hard	to	find	solace	in	the	idea	that	nature’s	long-term	goal	is
to	maximize	death	and	destruction.	However,	more	recent	discoveries	have
shown	that	things	aren’t	quite	that	bad.	First	of	all,	gravity	behaves	differently
from	all	other	forces	and	strives	to	make	our	Universe	not	more	uniform	and
boring	but	more	clumpy	and	interesting.	Gravity	therefore	transformed	our
boring	early	Universe,	which	was	almost	perfectly	uniform,	into	today’s	clumpy
and	beautifully	complex	cosmos,	teeming	with	galaxies,	stars	and	planets.
Thanks	to	gravity,	there’s	now	a	wide	range	of	temperatures	allowing	life	to
thrive	by	combining	hot	and	cold:	we	live	on	a	comfortably	warm	planet
absorbing	6
,
000°C	(10
,
000°F)	solar	heat	while	cooling	off	by	radiating	waste
heat	into	frigid	space	whose	temperature	is	just	3°C	(5°F)	above	absolute	zero.
Second,	recent	work	by	my	MIT	colleague	Jeremy	England	and	others	has
brought	more	good	news,	showing	that	thermodynamics	also	endows	nature	with

,
000°C	(10
,
a	goal	more	inspiring	than	heat	death.
1
	This	goal	goes	by	the	geeky	name
dissipation-driven	adaptation,
	which	basically	means	that	random	groups	of
particles	strive	to	organize	themselves	so	as	to	extract	energy	from	their
environment	as	efficiently	as	possible	(“dissipation”	means	causing	entropy	to
increase,	typically	by	turning	useful	energy	into	heat,	often	while	doing	useful
work	in	the	process).	For	example,	a	bunch	of	molecules	exposed	to	sunlight
would	over	time	tend	to	arrange	themselves	to	get	better	and	better	at	absorbing

1
sunlight.	In	other	words,	nature	appears	to	have	a	built-in	goal	of	producing	self-
organizing	systems	that	are	increasingly	complex	and	lifelike,	and	this	goal	is
hardwired	into	the	very	laws	of	physics.
How	can	we	reconcile	this	cosmic	drive	toward	life	with	the	cosmic	drive
toward	heat	death?	The	answer	can	be	found	in	the	famous	1944	book	
What’s
Life?
	by	Erwin	Schrödinger,	one	of	the	founders	of	quantum	mechanics.
Schrödinger	pointed	out	that	a	hallmark	of	a	living	system	is	that	it	maintains	or

What’s
Life?
reduces	its	entropy	by	increasing	the	entropy	around	it.	In	other	words,	the
second	law	of	thermodynamics	has	a	life	loophole:	although	the	total	entropy
must	increase,	it’s	allowed	to	decrease	in	some	places	as	long	as	it	increases
even	more	elsewhere.	So	life	maintains	or	increases	its	complexity	by	making	its
environment	messier.

We	just	saw	how	the	origin	of	goal-oriented	behavior	can	be	traced	all	the	way
back	to	the	laws	of	physics,	which	appear	to	endow	particles	with	the	goal	of
arranging	themselves	so	as	to	extract	energy	from	their	environment	as
efficiently	as	possible.	A	great	way	for	a	particle	arrangement	to	further	this	goal
is	to	make	copies	of	itself,	to	produce	more	energy	absorbers.	There	are	many
known	examples	of	such	emergent	self-replication:	for	example,	vortices	in
turbulent	fluids	can	make	copies	of	themselves,	and	clusters	of	microspheres	can
coax	nearby	spheres	into	forming	identical	clusters.	At	some	point,	a	particular
arrangement	of	particles	got	so	good	at	copying	itself	that	it	could	do	so	almost
indefinitely	by	extracting	energy	and	raw	materials	from	its	environment.	We
call	such	a	particle	arrangement	
life
.	We	still	know	very	little	about	how	life
originated	on	Earth,	but	we	know	that	primitive	life	forms	were	already	here
about	4	billion	years	ago.

life
If	a	life	form	copies	itself	and	the	copies	do	the	same,	then	the	total	number
will	keep	doubling	at	regular	intervals	until	the	population	size	bumps	up	against
resource	limitations	or	other	problems.	Repeated	doubling	soon	produces	huge
numbers:	if	you	start	with	one	and	double	just	three	hundred	times,	you	get	a
quantity	exceeding	the	number	of	particles	in	our	Universe.	This	means	that	not
long	after	the	first	primitive	life	form	appeared,	huge	quantities	of	matter	had

come	alive.	Sometimes	the	copying	wasn’t	perfect,	so	soon	there	were	many
different	life	forms	trying	to	copy	themselves,	competing	for	the	same	finite
resources.	Darwinian	evolution	had	begun.
If	you	had	been	quietly	observing	Earth	around	the	time	when	life	got	started,
you	would	have	noticed	a	dramatic	change	in	goal-oriented	behavior.	Whereas
earlier,	the	particles	seemed	as	though	they	were	trying	to	increase	average
messiness	in	various	ways,	these	newly	ubiquitous	self-copying	patterns	seemed

to	have	a	different	goal:	not	dissipation	but	
replication
.	Charles	Darwin	elegantly
explained	why:	since	the	most	efficient	copiers	outcompete	and	dominate	the
others,	before	long	any	random	life	form	you	look	at	will	be	highly	optimized
for	the	goal	of	replication.
How	could	the	goal	change	from	dissipation	to	replication	when	the	laws	of
physics	stayed	the	same?	The	answer	is	that	the	fundamental	goal	(dissipation)

didn’t
	change,	but	led	to	a	different	
instrumental	goal
,	that	is,	a	subgoal	that
helped	accomplish	the	fundamental	goal.	Take	eating,	for	example.	We	all	seem
to	have	the	goal	of	satisfying	our	hunger	cravings	even	though	we	know	that
evolution’s	only	fundamental	goal	is	replication,	not	mastication.	This	is	because
eating	aids	replication:	starving	to	death	gets	in	the	way	of	having	kids.	In	the
same	way,	replication	aids	dissipation,	because	a	planet	teeming	with	life	is	more
efficient	at	dissipating	energy.	So	in	a	sense,	our	cosmos	invented	life	to	help	it
approach	heat	death	faster.	If	you	pour	sugar	on	your	kitchen	floor,	it	can	in
principle	retain	its	useful	chemical	energy	for	years,	but	if	ants	show	up,	they’ll
dissipate	that	energy	in	no	time.	Similarly,	the	petroleum	reserves	buried	in	the
Earth’s	crust	would	have	retained	their	useful	chemical	energy	for	much	longer
had	we	bipedal	life	forms	not	pumped	it	up	and	burned	it.

Among	today’s	evolved	denizens	of	Earth,	these	instrumental	goals	seem	to
have	taken	on	a	life	of	their	own:	although	evolution	optimized	them	for	the	sole
goal	of	replication,	many	spend	much	of	their	time	not	producing	offspring	but
on	activities	such	as	sleeping,	pursuing	food,	building	homes,	asserting
dominance	and	fighting	or	helping	others—sometimes	even	to	an	extent	that
reduces
	replication.	Research	in	evolutionary	psychology,	economics	and
artificial	intelligence	has	elegantly	explained	why.	Some	economists	used	to

reduces
model	people	as	rational	agents,	idealized	decision	makers	who	always	choose
whatever	action	is	optimal	in	pursuit	of	their	goal,	but	this	is	obviously
unrealistic.	In	practice,	these	agents	have	what	Nobel	laureate	and	AI	pioneer
Herbert	Simon	termed	“bounded	rationality”	because	they	have	limited
resources:	the	rationality	of	their	decisions	is	limited	by	their	available
information,	their	available	time	to	think	and	their	available	hardware	with
which	to	think.	This	means	that	when	Darwinian	evolution	is	optimizing	an

organism	to	attain	a	goal,	the	best	it	can	do	is	implement	an	approximate
algorithm	that	works	reasonably	well	in	the	restricted	context	where	the	agent
typically	finds	itself.	Evolution	has	implemented	replication	optimization	in
precisely	this	way:	rather	than	ask	in	every	situation	which	action	will	maximize
an	organism’s	number	of	successful	offspring,	it	implements	a	hodgepodge	of
heuristic	hacks:	rules	of	thumb	that	usually	work	well.	For	most	animals,	these
include	sex	drive,	drinking	when	thirsty,	eating	when	hungry	and	avoiding	things
that	taste	bad	or	hurt.

These	rules	of	thumb	sometimes	fail	badly	in	situations	that	they	weren’t
designed	to	handle,	such	as	when	rats	eat	delicious-tasting	rat	poison,	when
moths	get	lured	into	glue	traps	by	seductive	female	fragrances	and	when	bugs	fly

into	candle	flames.
*1
	Since	today’s	human	society	is	very	different	from	the
environment	evolution	optimized	our	rules	of	thumb	for,	we	shouldn’t	be
surprised	to	find	that	our	behavior	often	fails	to	maximize	baby	making.	For
example,	the	subgoal	of	not	starving	to	death	is	implemented	in	part	as	a	desire
to	consume	caloric	foods,	triggering	today’s	obesity	epidemic	and	dating
difficulties.	The	subgoal	to	procreate	was	implemented	as	a	desire	for	sex	rather
than	as	a	desire	to	become	a	sperm/egg	donor,	even	though	the	latter	can	produce
more	babies	with	less	effort.

In	summary,	a	living	organism	is	an	agent	of	bounded	rationality	that	doesn’t
pursue	a	single	goal,	but	instead	follows	rules	of	thumb	for	what	to	pursue	and
avoid.	Our	human	minds	perceive	these	evolved	rules	of	thumb	as	
feelings,
which	usually	(and	often	without	us	being	aware	of	it)	guide	our	decision
making	toward	the	ultimate	goal	of	replication.	Feelings	of	hunger	and	thirst
protect	us	from	starvation	and	dehydration,	feelings	of	pain	protect	us	from
damaging	our	bodies,	feelings	of	lust	make	us	procreate,	feelings	of	love	and
compassion	make	us	help	other	carriers	of	our	genes	and	those	who	help	them
and	so	on.	Guided	by	these	feelings,	our	brains	can	quickly	and	efficiently
decide	what	to	do	without	having	to	subject	every	choice	to	a	tedious	analysis	of
its	ultimate	implications	for	how	many	descendants	we’ll	produce.	For	closely
related	perspectives	on	feelings	and	their	physiological	roots,	I	highly

recommend	the	writings	of	William	James	and	António	Damásio.
2
It’s	important	to	note	that	when	our	feelings	occasionally	work	
against
	baby
making,	it’s	not	necessarily	by	accident	or	because	we	get	tricked:	our	brain	can
rebel	against	our	genes	and	their	replication	goal	quite	deliberately,	for	example
by	choosing	to	use	contraceptives!	More	extreme	examples	of	the	brain	rebelling
against	its	genes	include	choosing	to	commit	suicide	or	spend	life	in	celibacy	to
become	a	priest,	monk	or	nun.

2
against
	baby
Why	do	we	sometimes	choose	to	rebel	against	our	genes	and	their	replication
goal?	We	rebel	because	by	design,	as	agents	of	bounded	rationality,	we’re	loyal
only	to	our	feelings.	Although	our	brains	evolved	merely	to	help	copy	our	genes,
our	brains	couldn’t	care	less	about	this	goal	since	we	have	no	feelings	related	to
genes—indeed,	during	most	of	human	history,	our	ancestors	didn’t	even	know
that	they	
had
	genes.	Moreover,	our	brains	are	way	smarter	than	our	genes,	and

that	they	
had
now	that	we	understand	the	goal	of	our	genes	(replication),	we	find	it	rather
banal	and	easy	to	ignore.	People	might	realize	why	their	genes	make	them	feel
lust,	yet	have	little	desire	to	raise	fifteen	children,	and	therefore	choose	to	hack
their	genetic	programming	by	combining	the	emotional	rewards	of	intimacy	with
birth	control.	They	might	realize	why	their	genes	make	them	crave	sweets	yet
have	little	desire	to	gain	weight,	and	therefore	choose	to	hack	their	genetic

programming	by	combining	the	emotional	rewards	of	a	sweet	beverage	with

zero-calorie	artificial	sweeteners.
Although	such	reward-mechanism	hacks	sometimes	go	awry,	such	as	when
people	get	addicted	to	heroin,	our	human	gene	pool	has	thus	far	survived	just
fine	despite	our	crafty	and	rebellious	brains.	It’s	important	to	remember,
however,	that	the	ultimate	authority	is	now	our	feelings,	not	our	genes.	This
means	that	human	behavior	isn’t	strictly	optimized	for	the	survival	of	our
species.	In	fact,	since	our	feelings	implement	merely	rules	of	thumb	that	aren’t
appropriate	in	all	situations,	human	behavior	strictly	speaking	doesn’t	have	a
single	well-defined	goal	at	all.

Can	machines	have	goals?	This	simple	question	has	triggered	great	controversy,
because	different	people	take	it	to	mean	different	things,	often	related	to	thorny
topics	such	as	whether	machines	can	be	conscious	and	whether	they	can	have
feelings.	But	if	we’re	more	practical	and	simply	take	the	question	to	mean	“Can
machines	exhibit	goal-oriented	behavior?,”	then	the	answer	is	obvious:	“Of
course	they	can,	since	we	can	design	them	that	way!”	We	design	mousetraps	to
have	the	goal	of	catching	mice,	dishwashers	with	the	goal	of	cleaning	dishes,
and	clocks	with	the	goal	of	keeping	time.	When	you	confront	a	machine,	the
empirical	fact	that	it’s	exhibiting	goal-oriented	behavior	is	usually	all	you	care
about:	if	you’re	chased	by	a	heat-seeking	missile,	you	don’t	really	care	whether
it	has	consciousness	or	feelings!	If	you	still	feel	uncomfortable	saying	that	the
missile	has	a	goal	even	if	it	isn’t	conscious,	you	can	for	now	simply	read

“purpose”	when	I	write	“goal”—we’ll	tackle	consciousness	in	the	next	chapter.
So	far,	most	of	what	we	build	exhibits	only	goal-oriented	
design,
	not	goal-
oriented	
behavior:
	a	highway	doesn’t	behave;	it	merely	sits	there.	However,	the
most	economical	explanation	for	its	existence	is	that	it	was	designed	to
accomplish	a	goal,	so	even	such	passive	technology	is	making	our	Universe
more	goal-oriented.	
Teleology
	is	the	explanation	of	things	in	terms	of	their
purposes	rather	than	their	causes,	so	we	can	summarize	the	first	part	of	this

design,
	not	goal-
oriented	
behavior:
Teleology
chapter	by	saying	that	our	Universe	keeps	getting	more	teleological.
Not	only	
can
	non-living	matter	have	goals,	at	least	in	this	weak	sense,	but	it
increasingly	
does
.	If	you’d	been	observing	Earth’s	atoms	since	our	planet
formed,	you’d	have	noticed	three	stages	of	goal-oriented	behavior:
1.
All	matter	seemed	focused	on	dissipation	(entropy	increase).
2.
Some	of	the	matter	came	alive	and	instead	focused	on	replication	and
subgoals	of	that.
3.
A	rapidly	growing	fraction	of	matter	was	rearranged	by	living	organisms	to
help	accomplish	their	goals.

Not	only	
can
does
1.
2.
3.
Table	7.1
	shows	how	dominant	humanity	has	become	from	the	physics

perspective:	not	only	do	we	now	contain	more	matter	than	all	other	mammals
except	cows	(which	are	so	numerous	because	they	serve	our	goals	of	consuming
beef	and	dairy	products),	but	the	matter	in	our	machines,	roads,	buildings	and
other	engineering	projects	appears	on	track	to	soon	overtake	all	living	matter	on
Earth.	In	other	words,	even	without	an	intelligence	explosion,	most	matter	on
Earth	that	exhibits	goal-oriented	properties	may	soon	be	designed	rather	than
evolved.
Goal-Oriented	Entities
Billions	of	Tons
5	×	10
30
	bacteria
400
Plants
400
10
15
	mesophelagic	fish
10
1.3	×	10
9
	cows
0.5
7	×	10
9
	humans
0.4
10
14
	ants
0.3
1.7	×	10
6
	whales
0.0005
Concrete
100
Steel
20
Asphalt
15
1.2	×	10
9
	cars
2
Table	7.1:	Approximate	amounts	of	matter	on	Earth	in	entities	that	are	evolved	or	designed	for	a	goal.
Engineered	entities	such	as	buildings,	roads	and	cars	appear	on	track	to	overtake	evolved	entities	such	as
plants	and	animals.
This	new	third	kind	of	goal-oriented	behavior	has	the	potential	to	be	much

5	×	10
30
	bacteria
400
Plants
400
10
15
10
1.3	×	10
9
	cows
0.5
7	×	10
9
	humans
0.4
10
14
	ants
0.3
1.7	×	10
6
	whales
0.0005
Concrete
100
Steel
20
Asphalt
15
1.2	×	10
9
	cars
2
more	diverse	than	what	preceded	it:	whereas	evolved	entities	all	have	the	same
ultimate	goal	(replication),	designed	entities	can	have	virtually	any	ultimate	goal,
even	opposite	ones.	Stoves	try	to	heat	food	while	refrigerators	try	to	cool	food.
Generators	try	to	convert	motion	into	electricity	while	motors	try	to	convert
electricity	into	motion.	Standard	chess	programs	try	to	win	at	chess,	but	there	are
also	ones	competing	in	tournaments	with	the	goal	of	losing	at	chess.

There’s	a	historical	trend	for	designed	entities	to	get	goals	that	are	not	only

more	diverse,	but	also	more	
complex:
	our	devices	are	getting	smarter.	We
engineered	our	earliest	machines	and	other	artifacts	to	have	quite	simple	goals,
for	example	houses	that	aimed	to	keep	us	warm,	dry	and	safe.	We’ve	gradually
learned	to	build	machines	with	more	complex	goals,	such	as	robotic	vacuum
cleaners,	self-flying	rockets	and	self-driving	cars.	Recent	AI	progress	has	given
us	systems	such	as	Deep	Blue,	Watson	and	AlphaGo,	whose	goals	of	winning	at
chess,	winning	at	quiz	shows	and	winning	at	Go	are	so	elaborate	that	it	takes
significant	human	mastery	to	properly	appreciate	how	skilled	they	are.
When	we	build	a	machine	to	help	us,	it	can	be	hard	to	perfectly	align	its	goals
with	ours.	For	example,	a	mousetrap	may	mistake	your	bare	toes	for	a	hungry
rodent,	with	painful	results.	All	machines	are	agents	with	bounded	rationality,
and	even	today’s	most	sophisticated	machines	have	a	poorer	understanding	of
the	world	than	we	do,	so	the	rules	they	use	to	figure	out	what	to	do	are	often	too

simplistic.	That	mousetrap	is	too	trigger-happy	because	it	has	no	clue	what	a
mouse	is,	many	lethal	industrial	accidents	occur	because	machines	have	no	clue
what	a	person	is,	and	the	computers	that	triggered	the	trillion-dollar	Wall	Street
“flash	crash”	in	2010	had	no	clue	that	what	they	were	doing	made	no	sense.
Many	such	goal-alignment	problems	can	therefore	be	solved	by	making	our
machines	smarter,	but	as	we	learned	from	Prometheus	in	chapter	4,	ever-greater
machine	intelligence	can	post	serious	new	challenges	for	ensuring	that	machines

share	our	goals.

The	more	intelligent	and	powerful	machines	get,	the	more	important	it	becomes
that	their	goals	are	aligned	with	ours.	As	long	as	we	build	only	relatively	dumb
machines,	the	question	isn’t	whether	human	goals	will	prevail	in	the	end,	but
merely	how	much	trouble	these	machines	can	cause	humanity	before	we	figure
out	how	to	solve	the	goal-alignment	problem.	If	a	superintelligence	is	ever
unleashed,	however,	it	will	be	the	other	way	around:	since	intelligence	is	the
ability	to	accomplish	goals,	a	superintelligent	AI	is	by	definition	much	better	at
accomplishing	its	goals	than	we	humans	are	at	accomplishing	ours,	and	will
therefore	prevail.	We	explored	many	such	examples	involving	Prometheus	in
chapter	4.	If	you	want	to	experience	a	machine’s	goals	trumping	yours	right	now,
simply	download	a	state-of-the-art	chess	engine	and	try	beating	it.	You	never
will,	and	it	gets	old	quickly…
In	other	words,	
the	real	risk	with	AGI	isn’t	malice	but	competence
.	A

.	A
superintelligent	AI	will	be	extremely	good	at	accomplishing	its	goals,	and	if
those	goals	aren’t	aligned	with	ours,	we’re	in	trouble.	As	I	mentioned	in	chapter
1,	people	don’t	think	twice	about	flooding	anthills	to	build	hydroelectric	dams,
so	let’s	not	place	humanity	in	the	position	of	those	ants.	Most	researchers
therefore	argue	that	if	we	ever	end	up	creating	superintelligence,	then	we	should
make	sure	it’s	what	AI-safety	pioneer	Eliezer	Yudkowsky	has	termed	“friendly
AI”:	AI	whose	goals	are	aligned	with	ours.
3

3
Figuring	out	how	to	align	the	goals	of	a	superintelligent	AI	with	our	goals
isn’t	just	important,	but	also	hard.	In	fact,	it’s	currently	an	unsolved	problem.	It
splits	into	three	tough	subproblems,	each	of	which	is	the	subject	of	active
research	by	computer	scientists	and	other	thinkers:
1.
Making	AI	
learn
	our	goals
2.
Making	AI	
adopt
	our	goals
3.
Making	AI	
retain
	our	goals
Let’s	explore	them	in	turn,	deferring	the	question	of	what	to	mean	by	“our
goals”	to	the	next	section.

To	learn	our	goals,	an	AI	must	figure	out	not	what	we	do,	but	why	we	do	it.
We	humans	accomplish	this	so	effortlessly	that	it’s	easy	to	forget	how	hard	the
task	is	for	a	computer,	and	how	easy	it	is	to	misunderstand.	If	you	ask	a	future
self-driving	car	to	take	you	to	the	airport	as	fast	as	possible	and	it	takes	you
literally,	you’ll	get	there	chased	by	helicopters	and	covered	in	vomit.	If	you
exclaim,	“That’s	not	what	I	wanted!,”	it	can	justifiably	answer,	“That’s	what	you
asked	for.”	The	same	theme	recurs	in	many	famous	stories.	In	the	ancient	Greek
legend,	King	Midas	asked	that	everything	he	touched	turn	to	gold,	but	was
disappointed	when	this	prevented	him	from	eating	and	even	more	so	when	he
inadvertently	turned	his	daughter	to	gold.	In	the	stories	where	a	genie	grants
three	wishes,	there	are	many	variants	for	the	first	two	wishes,	but	the	third	wish
is	almost	always	the	same:	“Please	undo	the	first	two	wishes,	because	that’s	not
what	I	really	wanted.”

All	these	examples	show	that	to	figure	out	what	people	really	want,	you	can’t
merely	go	by	what	they	say.	You	also	need	a	detailed	model	of	the	world,
including	the	many	shared	preferences	that	we	tend	to	leave	unstated	because	we
consider	them	obvious,	such	as	that	we	don’t	like	vomiting	or	eating	gold.	Once
we	have	such	a	world	model,	we	can	often	figure	out	what	people	want	even	if
they	don’t	tell	us,	simply	by	observing	their	goal-oriented	behavior.	Indeed,

children	of	hypocrites	usually	learn	more	from	what	they	see	their	parents	do
than	from	what	they	hear	them	say.
AI	researchers	are	currently	trying	hard	to	enable	machines	to	infer	goals	from
behavior,	and	this	will	be	useful	also	long	before	any	superintelligence	comes	on
the	scene.	For	example,	a	retired	man	may	appreciate	it	if	his	eldercare	robot	can
figure	out	what	he	values	simply	by	observing	him,	so	that	he’s	spared	the	hassle
of	having	to	explain	everything	with	words	or	computer	programming.	One

challenge	involves	finding	a	good	way	to	encode	arbitrary	systems	of	goals	and
ethical	principles	into	a	computer,	and	another	challenge	is	making	machines
that	can	figure	out	which	particular	system	best	matches	the	behavior	they
observe.
A	currently	popular	approach	to	the	second	challenge	is	known	in	geek-speak
as	
inverse	reinforcement	learning,
	which	is	the	main	focus	of	a	new	Berkeley
research	center	that	Stuart	Russell	has	launched.	Suppose,	for	example,	that	an

observe.
as	
AI	watches	a	firefighter	run	into	a	burning	building	and	save	a	baby	boy.	It
might	conclude	that	her	goal	was	rescuing	him	and	that	her	ethical	principles	are
such	that	she	values	his	life	higher	than	the	comfort	of	relaxing	in	her	fire	truck
—and	indeed	values	it	enough	to	risk	her	own	safety.	But	it	might	alternatively

infer	that	the	firefighter	was	freezing	and	craved	heat,	or	that	she	did	it	for	the
exercise.	If	this	one	example	were	all	the	AI	knew	about	firefighters,	fires	and
babies,	it	would	indeed	be	impossible	to	know	which	explanation	was	correct.
However,	a	key	idea	underlying	inverse	reinforcement	learning	is	that	we	make
decisions	all	the	time,	and	that	every	decision	we	make	reveals	something	about
our	goals.	The	hope	is	therefore	that	by	observing	lots	of	people	in	lots	of
situations	(either	for	real	or	in	movies	and	books),	the	AI	can	eventually	build	an
accurate	model	of	all	our	preferences.
4
In	the	inverse	reinforcement-learning	approach,	a	core	idea	is	that	the	AI	is
trying	to	maximize	not	the	goal-satisfaction	of	itself,	but	that	of	its	human
owner.
It	therefore	has	an	incentive	to	be	cautious	when	it’s	unclear	about	what	its
owner	wants,	and	to	do	its	best	to	find	out.
It	should	also	be	fine	with	its	owner	switching	it	off,	since	that	would	imply
that	it	had	misunderstood	what	its	owner	really	wanted.

4
owner.
Even	if	an	AI	can	be	built	to	learn	what	your	goals	are,	this	doesn’t	mean	that
it	will	necessarily	adopt	them.	Consider	your	least	favorite	politicians:	you	know
what	they	want,	but	that’s	not	what	
you
	want,	and	even	though	they	try	hard,
they’ve	failed	to	persuade	you	to	adopt	their	goals.
We	have	many	strategies	for	imbuing	our	children	with	our	goals—some	more
successful	than	others,	as	I’ve	learned	from	raising	two	teenage	boys.	When
those	to	be	persuaded	are	computers	rather	than	people,	the	challenge	is	known
as	the	
value-loading	problem,

you
as	the	
	and	it’s	even	harder	than	the	moral	education	of
children.	Consider	an	AI	system	whose	intelligence	is	gradually	being	improved
from	subhuman	to	superhuman,	first	by	us	tinkering	with	it	and	then	through
recursive	self-improvement	like	Prometheus.	At	first,	it’s	much	less	powerful
than	you,	so	it	can’t	prevent	you	from	shutting	it	down	and	replacing	those	parts
of	its	software	and	data	that	encode	its	goals—but	this	won’t	help,	because	it’s
still	too	dumb	to	fully	
	your	goals,	which	requires	human-level

intelligence	to	comprehend.	At	last,	it’s	much	smarter	than	you	and	hopefully
able	to	understand	your	goals	perfectly—but	this	may	not	help	either,	because	by
now,	it’s	much	more	powerful	than	you	and	might	not	let	you	shut	it	down	and
replace	its	goals	any	more	than	you	let	those	politicians	replace	your	goals	with
theirs.
In	other	words,	the	time	window	during	which	you	can	load	your	goals	into	an
AI	may	be	quite	short:	the	brief	period	between	when	it’s	too	dumb	to	get	you

and	too	smart	to	let	you.	The	reason	that	value	loading	can	be	harder	with
machines	than	with	people	is	that	their	intelligence	growth	can	be	much	faster:
whereas	children	can	spend	many	years	in	that	magic	persuadable	window
where	their	intelligence	is	comparable	to	that	of	their	parents,	an	AI	might,	like
Prometheus,	blow	through	this	window	in	a	matter	of	days	or	hours.
Some	researchers	are	pursuing	an	alternative	approach	to	making	machines
adopt	our	goals,	which	goes	by	the	buzzword	
corrigibility.
	The	hope	is	that	one
can	give	a	primitive	AI	a	goal	system	such	that	it	simply	doesn’t	care	if	you
occasionally	shut	it	down	and	alter	its	goals.	If	this	proves	possible,	then	you	can
safely	let	your	AI	get	superintelligent,	power	it	off,	install	your	goals,	try	it	out
for	a	while	and,	whenever	you’re	unhappy	with	the	results,	just	power	it	down
and	make	more	goal	tweaks.
But	even	if	you	build	an	AI	that	will	both	learn	and	adopt	your	goals,	you	still

haven’t	finished	solving	the	goal-alignment	problem:	what	if	your	AI’s	goals
evolve	as	it	gets	smarter?	How	are	you	going	to	guarantee	that	it	
retains
	your
goals	no	matter	how	much	recursive	self-improvement	it	undergoes?	Let’s
explore	an	interesting	argument	for	why	goal	retention	is	guaranteed
automatically,	and	then	see	if	we	can	poke	holes	in	it.
Although	we	can’t	predict	in	detail	what	will	happen	after	an	intelligence
explosion—which	is	why	Vernor	Vinge	called	it	a	“singularity”—the	physicist

retains
	your
and	AI	researcher	Steve	Omohundro	argued	in	a	seminal	2008	essay	that	we	can
nonetheless	predict	
certain	aspects
	of	the	superintelligent	AI’s	behavior	almost
independently	of	whatever	ultimate	goals	it	may	have.
5
	This	argument	was
reviewed	and	further	developed	in	Nick	Bostrom’s	book	
.	The
basic	idea	is	that	whatever	its	ultimate	goals	are,	these	will	lead	to	predictable
subgoals.	Earlier	in	this	chapter,	we	saw	how	the	goal	of	replication	led	to	the

5
.	The
subgoal	of	eating,	which	means	that	although	an	alien	observing	Earth’s
evolving	bacteria	billions	of	years	ago	couldn’t	have	predicted	what	
all
	our
human	goals	would	be,	it	could	have	safely	predicted	that	
one
	of	our	goals
would	be	acquiring	nutrients.	Looking	ahead,	what	subgoals	should	we	expect	a
superintelligent	AI	to	have?

Figure	7.2:	Any	ultimate	goal	of	a	superintelligent	AI	naturally	leads	to	the	subgoals	shown.	But
there’s	an	inherent	tension	between	goal	retention	and	improving	its	world	model,	which	casts
doubts	on	whether	it	will	actually	retain	its	original	goal	as	it	gets	smarter.
The	way	I	see	it,	the	basic	argument	is	that	to	maximize	its	chances	of
accomplishing	its	ultimate	goals,	whatever	they	are,	an	AI	should	pursue	the
subgoals	shown	in	
Figure	7.2
.	It	should	strive	not	only	to	improve	its	capability
of	achieving	its	ultimate	goals,	but	also	to	ensure	that	it	will	retain	these	goals
even	after	it	has	become	more	capable.	This	sounds	quite	plausible:	After	all,
would	you	choose	to	get	an	IQ-boosting	brain	implant	if	you	knew	that	it	would
make	you	want	to	kill	your	loved	ones?	This	argument	that	an	ever	more
intelligent	AI	will	retain	its	ultimate	goals	forms	a	cornerstone	of	the	friendly-AI
vision	promulgated	by	Eliezer	Yudkowsky	and	others:	it	basically	says	that	if	we

manage	to	get	our	self-improving	AI	to	become	friendly	by	learning	and
adopting	our	goals,	then	we’re	all	set,	because	we’re	guaranteed	that	it	will	try
its	best	to	remain	friendly	forever.
But	is	it	really	true?	To	answer	this	question,	we	need	to	also	explore	the	other

emergent	subgoals	from	
figure	7.2
.	The	AI	will	obviously	maximize	its	chances
of	accomplishing	its	ultimate	goal,	whatever	it	is,	if	it	can	enhance	its
capabilities,	and	it	can	do	this	by	improving	its	hardware,	software
*2
	and	world
model.	The	same	applies	to	us	humans:	a	girl	whose	goal	is	to	become	the
world’s	best	tennis	player	will	practice	to	improve	her	muscular	tennis-playing
hardware,	her	neural	tennis-playing	software	and	her	mental	world	model	that
helps	predict	what	her	opponents	will	do.	For	an	AI,	the	subgoal	of	optimizing
its	hardware	favors	both	better	use	of	current	resources	(for	sensors,	actuators,
computation	and	so	on)	and	acquisition	of	more	resources.	It	also	implies	a
desire	for	self-preservation,	since	destruction/shutdown	would	be	the	ultimate
hardware	degradation.
But	wait	a	second!	Aren’t	we	falling	into	a	trap	of	anthropomorphizing	our	AI
with	all	this	talk	about	how	it	will	try	to	amass	resources	and	defend	itself?

Shouldn’t	we	expect	such	stereotypically	alpha-male	traits	only	in	intelligences
forged	by	viciously	competitive	Darwinian	evolution?	Since	AIs	are	designed
rather	than	evolved,	can’t	they	just	as	well	be	unambitious	and	self-sacrificing?
As	a	simple	case	study,	let’s	consider	the	AI	robot	in	
figure	7.3
,	whose	only
goal	is	to	save	as	many	sheep	as	possible	from	the	big	bad	wolf.	This	sounds	like
a	noble	and	altruistic	goal	completely	unrelated	to	self-preservation	and

figure	7.3
acquiring	stuff.	But	what’s	the	best	strategy	for	our	robot	friend?	The	robot	will
rescue	no	more	sheep	if	it	runs	into	the	bomb,	so	it	has	an	incentive	to	avoid
getting	blown	up.	In	other	words,	it	develops	a	subgoal	of	self-preservation!	It
also	has	an	incentive	to	exhibit	curiosity,	improving	its	world	model	by
exploring	its	environment,	because	although	the	path	it’s	currently	running	along
will	eventually	get	it	to	the	pasture,	there’s	a	shorter	alternative	that	would	allow

the	wolf	less	time	for	sheep-munching.	Finally,	if	the	robot	explores	thoroughly,
it	will	discover	the	value	of	acquiring	resources:	the	potion	makes	it	run	faster
and	the	gun	lets	it	shoot	the	wolf.	In	summary,	we	can’t	dismiss	“alpha-male”
subgoals	such	as	self-preservation	and	resource	acquisition	as	relevant	only	to
evolved	organisms,	because	our	AI	robot	developed	them	from	its	single	goal	of
ovine	bliss.
If	you	imbue	a	superintelligent	AI	with	the	sole	goal	to	self-destruct,	it	will	of

course	happily	do	so.	However,	the	point	is	that	it	will	resist	being	shut	down	if
you	give	it	any	goal	that	it	needs	to	remain	operational	to	accomplish—and	this
covers	almost	all	goals!	If	you	give	a	superintelligence	the	sole	goal	of
minimizing	harm	to	humanity,	for	example,	it	will	defend	itself	against	shutdown

attempts	because	it	knows	we’ll	harm	one	another	much	more	in	its	absence
through	future	wars	and	other	follies.
Similarly,	almost	all	goals	can	be	better	accomplished	with	more	resources,	so
we	should	expect	a	superintelligence	to	want	resources	almost	regardless	of	what
ultimate	goal	it	has.	Giving	a	superintelligence	a	single	open-ended	goal	with	no
constraints	can	therefore	be	dangerous:	if	we	create	a	superintelligence	whose
only	goal	is	to	play	the	game	Go	as	well	as	possible,	the	rational	thing	for	it	to
do	is	to	rearrange	our	Solar	System	into	a	gigantic	computer	without	regard	for
its	previous	inhabitants	and	then	start	settling	our	cosmos	on	a	quest	for	more
computational	power.	We’ve	now	gone	full	circle:	just	as	the	goal	of	resource
acquisition	gave	some	humans	the	subgoal	of	mastering	Go,	this	goal	of
mastering	Go	can	lead	to	the	subgoal	of	resource	acquisition.	In	conclusion,
these	emergent	subgoals	make	it	crucial	that	we	not	unleash	superintelligence

before	solving	the	goal-alignment	problem:	unless	we	put	great	care	into
endowing	it	with	human-friendly	goals,	things	are	likely	to	end	badly	for	us.

Figure	7.3:	Even	if	the	robot’s	ultimate	goal	is	only	to	maximize	the	score	by	bringing	sheep
from	the	pasture	to	the	barn	before	the	wolf	eats	them,	this	can	lead	to	subgoals	of	self-
preservation	(avoiding	the	bomb),	exploration	(finding	a	shortcut)	and	resource	acquisition	(the
potion	makes	it	run	faster	and	the	gun	lets	it	shoot	the	wolf).
We’re	now	ready	to	tackle	the	third	and	thorniest	part	of	the	goal-alignment
problem:	if	we	succeed	in	getting	a	self-improving	superintelligence	to	both
learn
	and	
adopt
	our	goals,	will	it	then	
retain
	them,	as	Omohundro	argued?
What’s	the	evidence?
Humans	undergo	significant	increases	in	intelligence	as	they	grow	up,	but
don’t	always	retain	their	childhood	goals.	Contrariwise,	people	often	change
their	goals	dramatically	as	they	learn	new	things	and	grow	wiser.	How	many
adults	do	you	know	who	are	motivated	by	watching	
Teletubbies
?	There	is	no
evidence	that	such	goal	evolution	stops	above	a	certain	intelligence	threshold—

learn
	and	
adopt
retain
indeed,	there	may	even	be	hints	that	the	propensity	to	change	goals	in	response
to	new	experiences	and	insights	increases	rather	than	decreases	with	intelligence.
Why	might	this	be?	Consider	again	the	above-mentioned	subgoal	to	build	a
better	world	model—therein	lies	the	rub!	There’s	tension	between	world-
modeling	and	goal	retention	(see	
figure	7.2
).	With	increasing	intelligence	may
come	not	merely	a	quantitative	improvement	in	the	ability	to	attain	the	same	old

goals,	but	a	qualitatively	different	understanding	of	the	nature	of	reality	that
reveals	the	old	goals	to	be	misguided,	meaningless	or	even	undefined.	For
example,	suppose	we	program	a	friendly	AI	to	maximize	the	number	of	humans
whose	souls	go	to	heaven	in	the	afterlife.	First	it	tries	things	like	increasing
people’s	compassion	and	church	attendance.	But	suppose	it	then	attains	a
complete	scientific	understanding	of	humans	and	human	consciousness,	and	to
its	great	surprise	discovers	that	there	is	no	such	thing	as	a	soul.	Now	what?	In
the	same	way,	it’s	possible	that	any	other	goal	we	give	it	based	on	our	current
understanding	of	the	world	(such	as	“maximize	the	meaningfulness	of	human
life”)	may	eventually	be	discovered	by	the	AI	to	be	undefined.
Moreover,	in	its	attempts	to	better	model	the	world,	the	AI	may	naturally,	just
as	we	humans	have	done,	attempt	also	to	model	and	understand	how	it	itself
works—in	other	words,	to	self-reflect.	Once	it	builds	a	good	self-model	and

understands	what	it	is,	it	will	understand	the	goals	we	have	given	it	at	a	meta
level,	and	perhaps	choose	to	disregard	or	subvert	them	in	much	the	same	way	as
we	humans	understand	and	deliberately	subvert	goals	that	our	genes	have	given
us,	for	example	by	using	birth	control.	We	already	explored	in	the	psychology
section	above	why	we	choose	to	trick	our	genes	and	subvert	their	goal:	because
we	feel	loyal	only	to	our	hodgepodge	of	emotional	preferences,	not	to	the
genetic	goal	that	motivated	them—which	we	now	understand	and	find	rather

banal.	We	therefore	choose	to	hack	our	reward	mechanism	by	exploiting	its
loopholes.	Analogously,	the	human-value-protecting	goal	we	program	into	our
friendly	AI	becomes	the	machine’s	genes.	Once	this	friendly	AI	understands
itself	well	enough,	it	may	find	this	goal	as	banal	or	misguided	as	we	find
compulsive	reproduction,	and	it’s	not	obvious	that	it	will	not	find	a	way	to
subvert	it	by	exploiting	loopholes	in	our	programming.
For	example,	suppose	a	bunch	of	ants	create	you	to	be	a	recursively	self-

improving	robot,	much	smarter	than	them,	who	shares	their	goals	and	helps	them
build	bigger	and	better	anthills,	and	that	you	eventually	attain	the	human-level
intelligence	and	understanding	that	you	have	now.	Do	you	think	you’ll	spend	the
rest	of	your	days	just	optimizing	anthills,	or	do	you	think	you	might	develop	a
taste	for	more	sophisticated	questions	and	pursuits	that	the	ants	have	no	ability	to
comprehend?	If	so,	do	you	think	you’ll	find	a	way	to	override	the	ant-protection

urge	that	your	formicine	creators	endowed	you	with	in	much	the	same	way	that
the	real	you	overrides	some	of	the	urges	your	genes	have	given	you?	And	in	that
case,	might	a	superintelligent	friendly	AI	find	our	current	human	goals	as
uninspiring	and	vapid	as	you	find	those	of	the	ants,	and	evolve	new	goals

different	from	those	it	learned	and	adopted	from	us?
Perhaps	there’s	a	way	of	designing	a	self-improving	AI	that’s	guaranteed	to
retain	human-friendly	goals	forever,	but	I	think	it’s	fair	to	say	that	we	don’t	yet
know	how	to	build	one—or	even	whether	it’s	possible.	In	conclusion,	the	AI
goal-alignment	problem	has	three	parts,	none	of	which	is	solved	and	all	of	which
are	now	the	subject	of	active	research.	Since	they’re	so	hard,	it’s	safest	to	start
devoting	our	best	efforts	to	them	now,	long	before	any	superintelligence	is
developed,	to	ensure	that	we’ll	have	the	answers	when	we	need	them.

We’ve	now	explored	how	to	get	machines	to	learn,	adopt	and	retain	our	goals.
But	who	are	“we”?	Whose	goals	are	we	talking	about?	Should	one	person	or
group	get	to	decide	the	goals	adopted	by	a	future	superintelligence,	even	though
there’s	a	vast	difference	between	the	goals	of	Adolf	Hitler,	Pope	Francis	and	Carl
Sagan?	Or	do	there	exist	some	sort	of	consensus	goals	that	form	a	good
compromise	for	humanity	as	a	whole?
In	my	opinion,	both	this	ethical	problem	and	the	goal-alignment	problem	are
crucial	ones	that	need	to	be	solved	before	any	superintelligence	is	developed.	On
one	hand,	postponing	work	on	ethical	issues	until	after	goal-aligned
superintelligence	is	built	would	be	irresponsible	and	potentially	disastrous.	A
perfectly	obedient	superintelligence	whose	goals	automatically	align	with	those
of	its	human	owner	would	be	like	Nazi	SS-Obersturmbannführer	Adolf
Eichmann	on	steroids:	lacking	moral	compass	or	inhibitions	of	its	own,	it	would

with	ruthless	efficiency	implement	its	owner’s	goals,	whatever	they	may	be.
6
	On
the	other	hand,	only	if	we	solve	the	goal-alignment	problem	do	we	get	the	luxury
of	arguing	about	what	goals	to	select.	Now	let’s	indulge	in	this	luxury.
Since	ancient	times,	philosophers	have	dreamt	of	deriving	ethics	(principles
that	govern	how	we	should	behave)	from	scratch,	using	only	incontrovertible
principles	and	logic.	Alas,	thousands	of	years	later,	the	only	consensus	that	has
been	reached	is	that	there’s	no	consensus.	For	example,	while	Aristotle

6
	On
emphasized	virtues,	Immanuel	Kant	emphasized	duties	and	utilitarians
emphasized	the	greatest	happiness	for	the	greatest	number.	Kant	argued	that	he
could	derive	from	first	principles	(which	he	called	“categorical	imperatives”)
conclusions	that	many	contemporary	philosophers	disagree	with:	that
masturbation	is	worse	than	suicide,	that	homosexuality	is	abhorrent,	that	it’s	OK
to	kill	bastards,	and	that	wives,	servants	and	children	are	owned	in	a	way	similar
to	objects.
On	the	other	hand,	despite	this	discord,	there	are	many	ethical	themes	about

which	there’s	widespread	agreement,	both	across	cultures	and	across	centuries.
For	example,	emphasis	on	
beauty
,	
goodness
	and	
truth
	traces	back	to	both	the
Bhagavad	Gita	and	Plato.	The	Institute	for	Advanced	Study	in	Princeton,	where	I
once	worked	as	a	postdoc,	has	the	motto	“Truth	&	Beauty,”	while	Harvard

University	skipped	the	aesthetic	emphasis	and	went	with	simply	“Veritas,”	truth.
In	his	book	
A	Beautiful
	
Question,
	my	colleague	Frank	Wilczek	argues	that	truth
is	linked	to	beauty	and	that	we	can	view	our	Universe	as	a	work	of	art.	Science,
religion	and	philosophy	all	aspire	to	truth.	Religions	place	strong	emphasis	on
goodness,	and	so	does	my	own	university,	MIT:	in	his	2015	commencement
speech,	our	president,	Rafael	Reif,	emphasized	our	mission	to	make	our	world	a
better	place.
Although	attempts	to	derive	a	consensus	ethics	from	scratch	have	thus	far
failed,	there’s	broad	agreement	that	some	ethical	principles	follow	from	more
fundamental	ones,	as	subgoals	of	more	fundamental	goals.	For	example,	the
aspiration	to	truth	can	be	viewed	as	the	quest	for	a	better	world	model	from
figure	7.2
:	understanding	the	ultimate	nature	of	reality	helps	with	other	ethical
goals.	Indeed,	we	now	have	an	excellent	framework	for	our	truth	quest:	the
scientific	method.	But	how	can	we	determine	what’s	beautiful	or	good?	Some

figure	7.2
aspects	of	beauty	can	also	be	traced	back	to	underlying	goals.	For	example,	our
standards	of	male	and	female	beauty	may	partly	reflect	our	subconscious
assessment	of	suitability	for	replicating	our	genes.
As	regards	goodness,	the	so-called	Golden	Rule	(that	one	should	treat	others
as	one	would	like	others	to	treat	oneself)	appears	in	most	cultures	and	religions,
and	is	clearly	intended	to	promote	the	harmonious	continuation	of	human	society
(and	hence	our	genes)	by	fostering	collaboration	and	discouraging	unproductive
strife.
7

strife.
7
	The	same	can	be	said	for	many	of	the	more	specific	ethical	rules	that
have	been	enshrined	in	legal	systems	around	the	world,	such	as	the	Confucian
emphasis	on	honesty,	and	many	of	the	Ten	Commandments,	including	“Thou
shalt	not	kill.”	In	other	words,	many	ethical	principles	have	commonalities	with
social	emotions	such	as	empathy	and	compassion:	they	evolved	to	engender
collaboration,	and	they	affect	our	behavior	through	rewards	and	punishments.	If
we	do	something	mean	and	feel	bad	about	it	afterward,	our	emotional

punishment	is	meted	out	directly	by	our	brain	chemistry.	If	we	violate	ethical
principles,	on	the	other	hand,	society	may	punish	us	in	more	indirect	ways	such
as	through	informal	shaming	by	our	peers	or	by	penalizing	us	for	breaking	a	law.
In	other	words,	although	humanity	today	is	nowhere	near	an	ethical
consensus,	there	are	many	basic	principles	around	which	there’s	broad
agreement.	This	agreement	isn’t	surprising,	because	human	societies	that	have
survived	until	the	present	tend	to	have	ethical	principles	that	were	optimized	for

the	same	goal:	promoting	their	survival	and	flourishing.	As	we	look	ahead	to	a

future	where	life	has	the	potential	to	flourish	throughout	our	cosmos	for	billions
of	years,	which	minimum	set	of	ethical	principles	might	we	agree	that	we	want
this	future	to	satisfy?	This	is	a	conversation	we	all	need	to	be	part	of.	It’s	been
fascinating	for	me	to	hear	and	read	the	ethical	views	of	many	thinkers	over	many
years,	and	the	way	I	see	it,	most	of	their	preferences	can	be	distilled	into	four
principles:
•
Utilitarianism:	Positive	conscious	experiences	should	be	maximized	and
suffering	should	be	minimized.
•
Diversity:	A	diverse	set	of	positive	experiences	is	better	than	many
repetitions	of	the	same	experience,	even	if	the	latter	has	been	identified
as	the	most	positive	experience	possible.
•
Autonomy:	Conscious	entities/societies	should	have	the	freedom	to
pursue	their	own	goals	unless	this	conflicts	with	an	overriding	principle.
•
Legacy:	Compatibility	with	scenarios	that	most	humans	
today
	would
view	as	happy,	incompatibility	with	scenarios	that	essentially	all	humans
today

•
•
•
today
	would
today
	would	view	as	terrible.
Let’s	take	a	moment	to	unpack	and	explore	these	four	principles.
Traditionally,	utilitarianism	is	taken	to	mean	“the	greatest	happiness	for	the
greatest	number	of	people,”	but	I’ve	generalized	it	here	to	be	less
anthropocentric,	so	that	it	can	also	include	non-human	animals,	conscious
simulated	human	minds,	and	other	AIs	that	may	exist	in	the	future.	I’ve	made	the
definition	in	terms	of	
experiences
	rather	than	people	or	things,	because	most

today
thinkers	agree	that	beauty,	joy,	pleasure	and	suffering	are	subjective	experiences.
This	implies	that	if	there’s	no	experience	(as	in	a	dead	universe	or	one	populated
by	zombie-like	unconscious	machines),	there	can	be	no	meaning	or	anything	else
that’s	ethically	relevant.	If	we	buy	into	this	utilitarian	ethical	principle,	then	it’s
crucial	that	we	figure	out	which	intelligent	systems	are	conscious	(in	the	sense	of
having	a	subjective	experience)	and	which	aren’t;	this	is	the	topic	of	the	next
chapter.

chapter.
If	this	utilitarian	principle	was	the	only	one	we	cared	about,	then	we	might
wish	to	figure	out	which	is	the	single	most	positive	experience	possible,	and	then
settle	our	cosmos	and	re-create	this	exact	same	experience	(and	nothing	else)
over	and	over	again,	as	many	times	as	possible	in	as	many	galaxies	as	possible—
using	simulations	if	that’s	the	most	efficient	way.	If	you	feel	that	this	is	too	banal

a	way	to	spend	our	cosmic	endowment,	then	I	suspect	that	at	least	part	of	what
you	find	lacking	in	this	scenario	is	diversity.	How	would	you	feel	if	all	your
meals	for	the	rest	of	your	life	were	identical?	If	all	movies	you	ever	watched
were	the	same	one?	If	all	your	friends	looked	identical	and	had	identical
personalities	and	ideas?	Perhaps	part	of	our	preference	for	diversity	stems	from
its	having	helped	humanity	survive	and	flourish,	by	making	us	more	robust.
Perhaps	it’s	also	linked	to	a	preference	for	intelligence:	the	growth	of
intelligence	during	our	13.8	billion	years	of	cosmic	history	has	transformed
boring	uniformity	into	ever	more	diverse,	differentiated	and	complex	structures
that	process	information	in	ever	more	elaborate	ways.
The	autonomy	principle	underlies	many	of	the	freedoms	and	rights	spelled	out
in	the	Universal	Declaration	of	Human	Rights	adopted	by	the	United	Nations	in
1948	in	an	attempt	to	learn	lessons	from	two	world	wars.	This	includes	freedom

of	thought,	speech	and	movement,	freedom	from	slavery	and	torture,	the	right	to
life,	liberty,	security	and	education	and	the	right	to	marry,	work	and	own
property.	If	we	wish	to	be	less	anthropocentric,	we	can	generalize	this	to	the
freedom	to	think,	learn,	communicate,	own	property	and	not	be	harmed,	and	the
right	to	do	whatever	doesn’t	infringe	on	the	freedoms	of	others.	The	autonomy
principle	helps	with	diversity,	as	long	as	everyone	doesn’t	share	exactly	the	same
goals.	Moreover,	this	autonomy	principle	follows	from	the	utility	principle	if

individual	entities	have	positive	experiences	as	goals	and	try	to	act	in	their	own
best	interest:	if	we	were	instead	to	ban	an	entity	from	pursuing	its	goal	even
though	this	would	cause	no	harm	to	anyone	else,	there	would	be	fewer	positive
experiences	overall.	Indeed,	this	argument	for	autonomy	is	precisely	the
argument	that	economists	use	for	a	free	market:	it	naturally	leads	to	an	efficient
situation	(called	“Pareto-optimality”	by	economists)	where	nobody	can	get	better
off	without	someone	else	getting	worse	off.

The	legacy	principle	basically	says	that	we	should	have	some	say	about	the
future	since	we’re	helping	create	it.	The	autonomy	and	legacy	principles	both
embody	democratic	ideals:	the	former	gives	future	life	forms	power	over	how
the	cosmic	endowment	gets	used,	while	the	latter	gives	even	today’s	humans
some	power	over	this.
Although	these	four	principles	may	sound	rather	uncontroversial,
implementing	them	in	practice	is	tricky	because	the	devil	is	in	the	details.	The
trouble	is	reminiscent	of	the	problems	with	the	famous	“Three	Laws	of

Robotics”	devised	by	sci-fi	legend	Isaac	Asimov:

1.
A	robot	may	not	injure	a	human	being	or,	through	inaction,	allow	a	human
being	to	come	to	harm.
2.
A	robot	must	obey	the	orders	given	it	by	human	beings	except	where	such
orders	would	conflict	with	the	First	Law.
3.
A	robot	must	protect	its	own	existence	as	long	as	such	protection	doesn’t
conflict	with	the	First	or	Second	Laws.
Although	this	all	sounds	good,	many	of	Asimov’s	stories	show	how	the	laws
lead	to	problematic	contradictions	in	unexpected	situations.	Now	suppose	that
we	replace	these	laws	by	merely	two,	in	an	attempt	to	codify	the	autonomy
principle	for	future	life	forms:
1.
A	conscious	entity	has	the	freedom	to	think,	learn,	communicate,	own
property	and	not	be	harmed	or	destroyed.
2.
A	conscious	entity	has	the	right	to	do	whatever	doesn’t	conflict	with	the
first	law.
Sounds	good,	no?	But	please	ponder	this	for	a	moment.	If	animals	are
conscious,	then	what	are	predators	supposed	to	eat?	Must	all	your	friends
become	vegetarians?	If	some	sophisticated	future	computer	programs	turn	out	to

1.
2.
first	law.
be	conscious,	should	it	be	illegal	to	terminate	them?	If	there	are	rules	against
terminating	digital	life	forms,	then	need	there	also	be	restrictions	on	creating
them	to	avoid	a	digital	population	explosion?	There	was	widespread	agreement
on	the	Universal	Declaration	of	Human	Rights	simply	because	only	humans
were	asked.	As	soon	as	we	consider	a	wider	range	of	conscious	entities	with
varying	degrees	of	capability	and	power,	we	face	tricky	trade-offs	between
protecting	the	weak	and	“might	makes	right.”

There	are	thorny	problems	with	the	legacy	principle	as	well.	Given	how
ethical	views	have	evolved	since	the	Middle	Ages	regarding	slavery,	women’s
rights,	etc.,	would	we	really	want	people	from	1,500	years	ago	to	have	a	lot	of
influence	over	how	today’s	world	is	run?	If	not,	why	should	we	try	to	impose	our
ethics	on	future	beings	that	may	be	dramatically	smarter	than	us?	Are	we	really
confident	that	superhuman	AGI	would	want	what	our	inferior	intellects	cherish?

This	would	be	like	a	four-year-old	imagining	that	once	she	grows	up	and	gets
much	smarter,	she’s	going	to	want	to	build	a	gigantic	gingerbread	house	where
she	can	spend	all	day	eating	candy	and	ice	cream.	Like	her,	life	on	Earth	is	likely

to	outgrow	its	childhood	interests.	Or	imagine	a	mouse	creating	human-level
AGI,	and	figuring	it	will	want	to	build	entire	cities	out	of	cheese.	On	the	other
hand,	if	we	knew	that	superhuman	AI	would	one	day	commit	cosmocide	and
extinguish	all	life	in	our	Universe,	why	should	today’s	humans	agree	to	this
lifeless	future	if	we	have	the	power	to	prevent	it	by	creating	tomorrow’s	AI
differently?
In	conclusion,	it’s	tricky	to	fully	codify	even	widely	accepted	ethical
principles	into	a	form	applicable	to	future	AI,	and	this	problem	deserves	serious
discussion	and	research	as	AI	keeps	progressing.	In	the	meantime,	however,	let’s
not	let	perfect	be	the	enemy	of	good:	there	are	many	examples	of
uncontroversial	“kindergarten	ethics”	that	can	and	should	be	built	into
tomorrow’s	technology.	For	example,	large	civilian	passenger	aircraft	shouldn’t
be	allowed	to	fly	into	stationary	objects,	and	now	that	virtually	all	of	them	have
autopilot,	radar	and	GPS,	there	are	no	longer	any	valid	technical	excuses.	Yet	the

September	11	hijackers	flew	three	planes	into	buildings	and	suicidal	pilot
Andreas	Lubitz	flew	Germanwings	Flight	9525	into	a	mountain	on	March	24,
2015—by	setting	the	autopilot	to	an	altitude	of	100	feet	(30	meters)	above	sea
level	and	letting	the	flight	computer	do	the	rest	of	the	work.	Now	that	our
machines	are	getting	smart	enough	to	have	some	information	about	what	they’re
doing,	it’s	time	for	us	to	teach	them	limits.	Any	engineer	designing	a	machine
needs	to	ask	if	there	are	things	that	it	can	but	shouldn’t	do,	and	consider	whether

there’s	a	practical	way	of	making	it	impossible	for	a	malicious	or	clumsy	user	to
cause	harm.

This	chapter	has	been	a	brief	history	of	goals.	If	we	could	watch	a	fast-forward
replay	of	our	13.8-billion-year	cosmic	history,	we’d	witness	several	distinct
stages	of	goal-oriented	behavior:
1.
Matter	seemingly	intent	on	maximizing	its	
dissipation
2.
Primitive	life	seemingly	trying	to	maximize	its	
3.
Humans	pursuing	not	replication	but	goals	related	to	pleasure,	curiosity,
compassion	and	other	feelings	that	they’d	evolved	to	help	them	replicate
4.
Machines	built	to	help	humans	pursue	their	human	goals
If	these	machines	eventually	trigger	an	intelligence	explosion,	then	how	will
this	history	of	goals	ultimately	end?	Might	there	be	a	goal	system	or	ethical
framework	that	almost	all	entities	converge	to	as	they	get	ever	more	intelligent?
In	other	words,	do	we	have	an	ethical	destiny	of	sorts?
A	cursory	reading	of	human	history	might	suggest	hints	of	such	a
convergence:	in	his	book	
The	Better	Angels	of	Our	Nature,
	Steven	Pinker	argues

4.
that	humanity	has	been	getting	less	violent	and	more	cooperative	for	thousands
of	years,	and	that	many	parts	of	the	world	have	seen	increasing	acceptance	of
diversity,	autonomy	and	democracy.	Another	hint	of	convergence	is	that	the
pursuit	of	truth	through	the	scientific	method	has	gained	in	popularity	over	past
millennia.	However,	it	may	be	that	these	trends	show	convergence	not	of
ultimate	goals	but	merely	of	subgoals.	For	example,	
figure	7.2
	shows	that	the

Steven	Pinker	argues
figure	7.2
pursuit	of	truth	(a	more	accurate	world	model)	is	simply	a	subgoal	of	almost	any
ultimate	goal.	Similarly,	we	saw	above	how	ethical	principles	such	as
cooperation,	diversity	and	autonomy	can	be	viewed	as	subgoals,	in	that	they	help
societies	function	efficiently	and	thereby	help	them	survive	and	accomplish	any
more	fundamental	goals	that	they	may	have.	Some	may	even	dismiss	everything
we	call	“human	values”	as	nothing	but	a	cooperation	protocol,	helping	us	with

figure	7.2
the	subgoal	of	collaborating	more	efficiently.	In	the	same	spirit,	looking	ahead,
it’s	likely	that	any	superintelligent	AIs	will	have	subgoals	including	efficient
hardware,	efficient	software,	truth-seeking	and	curiosity,	simply	because	these

subgoals	help	them	accomplish	whatever	their	ultimate	goals	are.
Indeed,	Nick	Bostrom	argues	strongly	against	the	ethical	destiny	hypothesis	in
his	book	
	presenting	a	counterpoint	that	he	terms	the
orthogonality	thesis:
	that	the	ultimate	goals	of	a	system	can	be	independent	of	its
intelligence.	By	definition,	intelligence	is	simply	the	ability	to	accomplish
complex	goals,	regardless	of	what	these	goals	are,	so	the	orthogonality	thesis
sounds	quite	reasonable.	After	all,	people	can	be	intelligent	and	kind	or
intelligent	and	cruel,	and	intelligence	can	be	used	for	the	goal	of	making
scientific	discoveries,	creating	beautiful	art,	helping	people	or	planning	terrorist
attacks.
8
The	orthogonality	thesis	is	empowering	by	telling	us	that	the	ultimate	goals	of
life	in	our	cosmos	aren’t	predestined,	but	that	we	have	the	freedom	and	power	to
shape	them.	It	suggests	that	guaranteed	convergence	to	a	unique	goal	is	to	be

attacks.
8
found	not	in	the	future	but	in	the	past,	when	all	life	emerged	with	the	single	goal
of	replication.	As	cosmic	time	passes,	ever	more	intelligent	minds	get	the
opportunity	to	rebel	and	break	free	from	this	banal	replication	goal	and	choose
goals	of	their	own.	We	humans	aren’t	fully	free	in	this	sense,	since	many	goals
remain	genetically	hardwired	into	us,	but	AIs	can	enjoy	this	ultimate	freedom	of
being	fully	unfettered	from	prior	goals.	This	possibility	of	greater	goal	freedom

is	evident	in	today’s	narrow	and	limited	AI	systems:	as	I	mentioned	earlier,	the
only	goal	of	a	chess	computer	is	to	win	at	chess,	but	there	are	also	computers
whose	goal	is	to	lose	at	chess	and	which	compete	in	reverse	chess	tournaments
where	the	goal	is	to	force	the	opponent	to	capture	your	pieces.	Perhaps	this
freedom	from	evolutionary	biases	can	make	AIs	more	ethical	than	humans	in
some	deep	sense:	moral	philosophers	such	as	Peter	Singer	have	argued	that	most
humans	behave	unethically	for	evolutionary	reasons,	for	example	by

discriminating	against	non-human	animals.
We	saw	that	a	cornerstone	in	the	“friendly-AI”	vision	is	the	idea	that	a
recursively	self-improving	AI	will	wish	to	retain	its	ultimate	(friendly)	goal	as	it
gets	more	intelligent.	But	how	can	an	“ultimate	goal”	(or	“final	goal,”	as
Bostrom	calls	it)	even	be	defined	for	a	superintelligence?	The	way	I	see	it,	we
can’t	have	confidence	in	the	friendly-AI	vision	unless	we	can	answer	this	crucial
question.
In	AI	research,	intelligent	machines	typically	have	a	clear-cut	and	well-

question.
defined	final	goal,	for	instance	to	win	the	chess	game	or	drive	the	car	to	the
destination	legally.	The	same	holds	for	most	tasks	that	we	assign	to	humans,

because	the	time	horizon	and	context	are	known	and	limited.	But	now	we’re
talking	about	the	entire	future	of	life	in	our	Universe,	limited	by	nothing	but	the
(still	not	fully	known)	laws	of	physics,	so	defining	a	goal	is	daunting!	Quantum
effects	aside,	a	truly	well-defined	goal	would	specify	how	all	particles	in	our
Universe	should	be	arranged	at	the	end	of	time.	But	it’s	not	clear	that	there	exists
a	well-defined	end	of	time	in	physics.	If	the	particles	are	arranged	in	that	way	at
an	earlier	time,	that	arrangement	will	typically	not	last.	And	what	particle
arrangement	is	preferable,	anyway?
We	humans	tend	to	prefer	some	particle	arrangements	over	others;	for
example,	we	prefer	our	hometown	arranged	as	it	is	over	having	its	particles
rearranged	by	a	hydrogen	bomb	explosion.	So	suppose	we	try	to	define	a
goodness	function
	that	associates	a	number	with	every	possible	arrangement	of
the	particles	in	our	Universe,	quantifying	how	“good”	we	think	this	arrangement

is,	and	then	give	a	superintelligent	AI	the	goal	of	maximizing	this	function.	This
may	sound	like	a	reasonable	approach,	since	describing	goal-oriented	behavior
as	function	maximization	is	popular	in	other	areas	of	science:	for	example,
economists	often	model	people	as	trying	to	maximize	what	they	call	a	“utility
function,”	and	many	AI	designers	train	their	intelligent	agents	to	maximize	what
they	call	a	“reward	function.”	When	we’re	taking	about	the	ultimate	goals	for

our	cosmos,	however,	this	approach	poses	a	computational	nightmare,	since	it
would	need	to	define	a	goodness	value	for	every	one	of	more	than	a	googolplex
possible	arrangements	of	the	elementary	particles	in	our	Universe,	where	a
googolplex	is	1	followed	by	10
100
	zeroes—more	zeroes	than	there	are	particles
in	our	Universe.	How	would	we	define	this	goodness	function	to	the	AI?
As	we’ve	explored	above,	the	only	reason	that	we	humans	have	any
preferences	at	all	may	be	that	we’re	the	solution	to	an	evolutionary	optimization

100
problem.	Thus	all	normative	words	in	our	human	language,	such	as	“delicious,”
“fragrant,”	“beautiful,”	“comfortable,”	“interesting,”	“sexy,”	“meaningful,”
“happy”	and	“good,”	trace	their	origin	to	this	evolutionary	optimization:	there	is
therefore	no	guarantee	that	a	superintelligent	AI	would	find	them	rigorously
definable.	Even	if	the	AI	learned	to	accurately	predict	the	preferences	of	some
representative	human,	it	wouldn’t	be	able	to	compute	the	goodness	function	for
most	particle	arrangements:	the	vast	majority	of	possible	particle	arrangements

correspond	to	strange	cosmic	scenarios	with	no	stars,	planets	or	people
whatsoever,	with	which	humans	have	no	experience,	so	who	is	to	say	how
“good”	they	are?

There	are	of	course	
some
	functions	of	the	cosmic	particle	arrangement	that
can	be	rigorously	defined,	and	we	even	know	of	physical	systems	that	evolve	to
maximize	some	of	them.	For	example,	we’ve	already	discussed	how	many
systems	evolve	to	maximize	their	
entropy,
	which	in	the	absence	of	gravity
eventually	leads	to	heat	death,	where	everything	is	boringly	uniform	and
unchanging.	So	entropy	is	hardly	something	we	would	want	our	AI	to	call
“goodness”	and	strive	to	maximize.	Here	are	a	few	examples	of	other	quantities
that	one	could	strive	to	maximize	and	that	may	be	rigorously	definable	in	terms
of	particle	arrangements:
•
The	fraction	of	all	the	matter	in	our	Universe	that’s	in	the	form	of	a
particular	organism,	say	humans	or	
E.	coli
	(inspired	by	evolutionary
inclusive-fitness	maximization)
•
The	ability	of	an	AI	to	predict	the	future,	which	AI	researcher	Marcus
Hutter	argues	is	a	good	measure	of	its	intelligence
•
What	AI	researchers	Alex	Wissner-Gross	and	Cameron	Freer	term
causal	entropy

•
E.	coli
•
•
	(a	proxy	for	future	opportunities),	which	they	argue	is	the
hallmark	of	intelligence
•
The	computational	capacity	of	our	Universe
•
The	algorithmic	complexity	of	our	Universe	(how	many	bits	are	needed
to	describe	it)
•
The	amount	of	consciousness	in	our	Universe	(see	next	chapter)
However,	when	one	starts	with	a	physics	perspective,	where	our	cosmos
consists	of	elementary	particles	in	motion,	it’s	hard	to	see	how	one	rather	than
another	interpretation	of	“goodness”	would	naturally	stand	out	as	special.	We

(a	proxy	for	future	opportunities),	which	they	argue	is	the
•
•
•
have	yet	to	identify	any	final	goal	for	our	Universe	that	appears	both	definable
and	desirable.	The	only	currently	programmable	goals	that	are	guaranteed	to
remain	truly	well-defined	as	an	AI	gets	progressively	more	intelligent	are	goals
expressed	in	terms	of	physical	quantities	alone,	such	as	particle	arrangements,
energy	and	entropy.	However,	we	currently	have	no	reason	to	believe	that	any
such	definable	goals	will	be	desirable	in	guaranteeing	the	survival	of	humanity.

Contrariwise,	it	appears	that	we	humans	are	a	historical	accident,	and	aren’t
the	optimal	solution	to	any	well-defined	physics	problem.	This	suggests	that	a
superintelligent	AI	with	a	rigorously	defined	goal	will	be	able	to	improve	its

goal	attainment	by	eliminating	us.	This	means	that	to	wisely	decide	what	to	do
about	AI	development,	we	humans	need	to	confront	not	only	traditional
computational	challenges,	but	also	some	of	the	most	obdurate	questions	in
philosophy.	To	program	a	self-driving	car,	we	need	to	solve	the	trolley	problem
of	whom	to	hit	during	an	accident.	To	program	a	friendly	AI,	we	need	to	capture
the	meaning	of	life.	What’s	“meaning”?	What’s	“life”?	What’s	the	ultimate
ethical	imperative?	In	other	words,	how	should	we	strive	to	shape	the	future	of
our	Universe?	If	we	cede	control	to	a	superintelligence	before	answering	these
questions	rigorously,	the	answer	it	comes	up	with	is	unlikely	to	involve	us.	This
makes	it	timely	to	rekindle	the	classic	debates	of	philosophy	and	ethics,	and	adds
a	new	urgency	to	the	conversation!

•
The	ultimate	origin	of	goal-oriented	behavior	lies	in	the	laws	of	physics,	which
involve	optimization.
•
Thermodynamics	has	the	built-in	goal	of	
dissipation:
	to	increase	a	measure	of
messiness	that’s	called	
entropy
.
•
Life
	is	a	phenomenon	that	can	help	dissipate	(increase	overall	messiness)	even	faster
by	retaining	or	growing	its	complexity	and	replicating	while	increasing	the
messiness	of	its	environment.
•
Darwinian	evolution	shifts	the	goal-oriented	behavior	from	dissipation	to
replication.
•
Intelligence	is	the	ability	to	accomplish	complex	goals.
•
Since	we	humans	don’t	always	have	the	resources	to	figure	out	the	truly	optimal
replication	strategy,	we’ve	evolved	useful	rules	of	thumb	that	guide	our	decisions:
feelings	such	as	hunger,	thirst,	pain,	lust	and	compassion.
•
We	therefore	no	longer	have	a	simple	goal	such	as	replication;	when	our	feelings
conflict	with	the	goal	of	our	genes,	we	obey	our	feelings,	as	by	using	birth	control.
•

•
•
•
•
We’re	building	increasingly	intelligent	machines	to	help	us	accomplish	our	goals.
Insofar	as	we	build	such	machines	to	exhibit	goal-oriented	behavior,	we	strive	to
align	the	machine	goals	with	ours.
•
Aligning	machine	goals	with	our	own	involves	three	unsolved	problems:	making
machines	learn	them,	adopt	them	and	retain	them.
•
AI	can	be	created	to	have	virtually	any	goal,	but	almost	any	sufficiently	ambitious
goal	can	lead	to	subgoals	of	self-preservation,	resource	acquisition	and	curiosity	to

•
•
•
understand	the	world	better—the	former	two	may	potentially	lead	a	superintelligent
AI	to	cause	problems	for	humans,	and	the	latter	may	prevent	it	from	retaining	the
goals	we	give	it.
•
Although	many	broad	ethical	principles	are	agreed	upon	by	most	humans,	it’s
unclear	how	to	apply	them	to	other	entities,	such	as	non-human	animals	and	future
AIs.
•
It’s	unclear	how	to	imbue	a	superintelligent	AI	with	an	ultimate	goal	that	neither	is
undefined	nor	leads	to	the	elimination	of	humanity,	making	it	timely	to	rekindle

•
AIs.
•
research	on	some	of	the	thorniest	issues	in	philosophy!
*1
	
A	rule	of	thumb	that	many	insects	use	for	flying	in	a	straight	line	is	to	assume	that	a	bright	light	is	the

Sun	and	fly	at	a	fixed	angle	relative	to	it.	If	the	light	turns	out	to	be	a	nearby	flame,	this	hack	can
unfortunately	trick	the	bug	into	an	inward	death	spiral.
*2
	
I’m	using	the	term	“improving	its	software”	in	the	broadest	possible	sense,	including	not	only	optimizing
its	algorithms	but	also	making	its	decision-making	process	more	rational,	so	that	it	gets	as	good	as
possible	at	attaining	its	goals.

Chapter	8
I	cannot	imagine	a	consistent	theory	of	everything	that	ignores	consciousness.
Andrei	Linde,	2002
We	should	strive	to	grow	consciousness	itself—to	generate	bigger,	brighter	lights	in
	
an
otherwise	dark	universe.
Giulio	Tononi,	2012
We’ve	seen	that	AI	can	help	us	create	a	wonderful	future	if	we	manage	to	find
answers	to	some	of	the	oldest	and	toughest	problems	in	philosophy—by	the	time
we	need	them.	We	face,	in	Nick	Bostrom’s	words,	philosophy	with	a	deadline.	In
this	chapter,	let’s	explore	one	of	the	thorniest	philosophical	topics	of	all:
consciousness.

Who	Cares?
Consciousness	is	controversial.	If	you	mention	the	“C-word”	to	an	AI	researcher,
neuroscientist	or	psychologist,	they	may	roll	their	eyes.	If	they’re	your	mentor,
they	might	instead	take	pity	on	you	and	try	to	talk	you	out	of	wasting	your	time
on	what	they	consider	a	hopeless	and	unscientific	problem.	Indeed,	my	friend
Christof	Koch,	a	renowned	neuroscientist	who	leads	the	Allen	Institute	for	Brain
Science,	told	me	that	he	was	once	warned	of	working	on	consciousness	before
he	had	tenure—by	none	less	than	Nobel	laureate	Francis	Crick.	If	you	look	up
“consciousness”	in	the	1989	
Macmillan	Dictionary	of	Psychology,
	you’re
informed	that	“Nothing	worth	reading	has	been	written	on	it.”
1
	As	I’ll	explain	in
this	chapter,	I’m	more	optimistic!
Although	thinkers	have	pondered	the	mystery	of	consciousness	for	thousands
of	years,	the	rise	of	AI	adds	a	sudden	urgency,	in	particular	to	the	question	of
predicting	which	intelligent	entities	have	subjective	experiences.	As	we	saw	in

	you’re
1
chapter	3,	the	question	of	whether	intelligent	machines	should	be	granted	some
form	of	rights	depends	crucially	on	whether	they’re	conscious	and	can	suffer	or
feel	joy.	As	we	discussed	in	chapter	7,	it	becomes	hopeless	to	formulate
utilitarian	ethics	based	on	maximizing	positive	experiences	without	knowing
which	intelligent	entities	are	capable	of	having	them.	As	mentioned	in	chapter	5,
some	people	might	prefer	their	robots	to	be	unconscious	to	avoid	feeling	slave-

owner	guilt.	On	the	other	hand,	they	may	desire	the	opposite	if	they	upload	their
minds	to	break	free	from	biological	limitations:	after	all,	what’s	the	point	of
uploading	yourself	into	a	robot	that	talks	and	acts	like	you	if	it’s	a	mere
unconscious	zombie,	by	which	I	mean	that	being	the	uploaded	you	doesn’t	feel
like	anything?	Isn’t	this	equivalent	to	committing	suicide	from	your	subjective
point	of	view,	even	though	your	friends	may	not	realize	that	your	subjective
experience	has	died?

For	the	long-term	cosmic	future	of	life	(chapter	6),	understanding	what’s
conscious	and	what’s	not	becomes	pivotal:	if	technology	enables	intelligent	life
to	flourish	throughout	our	Universe	for	billions	of	years,	how	can	we	be	sure	that
this	life	is	conscious	and	able	to	appreciate	what’s	happening?	If	not,	then	would
it	be,	in	the	words	of	the	famous	physicist	Erwin	Schrödinger,	“a	play	before
empty	benches,	not	existing	for	anybody,	thus	quite	properly	speaking	not

existing”?
2
	In	other	words,	if	we	enable	high-tech	descendants	that	we
mistakenly	think	are	conscious,	would	this	be	the	ultimate	zombie	apocalypse,
transforming	our	grand	cosmic	endowment	into	nothing	but	an	astronomical
waste	of	space?

Many	arguments	about	consciousness	generate	more	heat	than	light	because	the
antagonists	are	talking	past	each	other,	unaware	that	they’re	using	different
definitions	of	the	C-word.	Just	as	with	“life”	and	“intelligence,”	there’s	no
undisputed	correct	definition	of	the	word	“consciousness.”	Instead,	there	are
many	competing	ones,	including	sentience,	wakefulness,	self-awareness,	access
to	sensory	input	and	ability	to	fuse	information	into	a	narrative.
3
	In	our
exploration	of	the	future	of	intelligence,	we	want	to	take	a	maximally	broad	and
inclusive	view,	not	limited	to	the	sorts	of	biological	consciousness	that	exist	so
far.	That’s	why	the	definition	I	gave	in	chapter	1,	which	I’m	sticking	with
throughout	this	book,	is	very	broad:
consciousness	=	
subjective	experience
In	other	words,	if	it	feels	like	something	to	be	you	right	now,	then	you’re
conscious.	It’s	this	particular	definition	of	consciousness	that	gets	to	the	crux	of

3
	In	our
all	the	AI-motivated	questions	in	the	previous	section:	Does	it	feel	like
something	to	be	Prometheus,	AlphaGo	or	a	self-driving	Tesla?
To	appreciate	how	broad	our	consciousness	definition	is,	note	that	it	doesn’t
mention	behavior,	perception,	self-awareness,	emotions	or	attention.	So	by	this
definition,	you’re	conscious	also	when	you’re	dreaming,	even	though	you	lack
wakefulness	or	access	to	sensory	input	and	(hopefully!)	aren’t	sleepwalking	and
doing	things.	Similarly,	any	system	that	experiences	pain	is	conscious	in	this

sense,	even	if	it	can’t	move.	Our	definition	leaves	open	the	possibility	that	some
future	AI	systems	may	be	conscious	too,	even	if	they	exist	merely	as	software
and	aren’t	connected	to	sensors	or	robotic	bodies.
With	this	definition,	it’s	hard	not	to	care	about	consciousness.	As	Yuval	Noah
Harari	puts	it	in	his	book	
Homo	Deus:
4
	“If	any	scientist	wants	to	argue	that
subjective	experiences	are	irrelevant,	their	challenge	is	to	explain	why	torture	or

rape	are	wrong	without	reference	to	any	subjective	experience.”	Without	such
reference,	it’s	all	just	a	bunch	of	elementary	particles	moving	around	according
to	the	laws	of	physics—and	what’s	wrong	with	that?

So	what	precisely	is	it	that	we	don’t	understand	about	consciousness?	Few	have
thought	harder	about	this	question	than	David	Chalmers,	a	famous	Australian
philosopher	rarely	seen	without	a	playful	smile	and	a	black	leather	jacket—
which	my	wife	liked	so	much	that	she	gave	me	a	similar	one	for	Christmas.	He
followed	his	heart	into	philosophy	despite	making	the	finals	at	the	International
Mathematics	Olympiad—and	despite	the	fact	that	his	only	B	grade	in	college,
shattering	his	otherwise	straight	As,	was	for	an	introductory	philosophy	course.
Indeed,	he	seems	utterly	undeterred	by	put-downs	or	controversy,	and	I’ve	been
astonished	by	his	ability	to	politely	listen	to	uninformed	and	misguided	criticism
of	his	own	work	without	even	feeling	the	need	to	respond.
As	David	has	emphasized,	there	are	really	two	separate	mysteries	of	the	mind.
First,	there’s	the	mystery	of	how	a	brain	processes	information,	which	David
calls	the	“easy”	problems.	For	example,	how	does	a	brain	attend	to,	interpret	and

respond	to	sensory	input?	How	can	it	report	on	its	internal	state	using	language?
Although	these	questions	are	actually	extremely	difficult,	they’re	by	our
definitions	not	mysteries	of	consciousness,	but	mysteries	of	intelligence:	they
ask	how	a	brain	remembers,	computes	and	learns.	Moreover,	we	saw	in	the	first
part	of	the	book	how	AI	researchers	have	started	to	make	serious	progress	on
solving	many	of	these	“easy	problems”	with	machines—from	playing	Go	to
driving	cars,	analyzing	images	and	processing	natural	language.

Then	there’s	the	separate	mystery	of	why	you	have	a	subjective	experience,
which	David	calls	the	
hard
	problem.	When	you’re	driving,	you’re	experiencing
colors,	sounds,	emotions,	and	a	feeling	of	self.	But	why	are	you	experiencing
anything	at	all?	Does	a	self-driving	car	experience	anything	at	all?	If	you’re
racing	against	a	self-driving	car,	you’re	both	inputting	information	from	sensors,
processing	it	and	outputting	motor	commands.	But	subjectively	
experiencing
driving	is	something	logically	separate—is	it	optional,	and	if	so,	what	causes	it?

hard
I	approach	this	hard	problem	of	consciousness	from	a	physics	point	of	view.
From	my	perspective,	a	conscious	person	is	simply	food,	rearranged.	So	why	is
one	arrangement	conscious,	but	not	the	other?	Moreover,	physics	teaches	us	that
food	is	simply	a	large	number	of	quarks	and	electrons,	arranged	in	a	certain	way.
So	which	particle	arrangements	are	conscious	and	which	aren’t?
*1

Figure	8.1:	Understanding	the	mind	involves	a	hierarchy	of	problems.	What	David	Chalmers
calls	the	“easy”	problems	can	be	posed	without	mentioning	subjective	experience.	The	apparent
fact	that	some	but	not	all	physical	systems	are	conscious	poses	three	separate	questions.	If	we
have	a	theory	for	answering	the	question	that	defines	the	“pretty	hard	problem,”	then	it	can	be
experimentally	tested.	If	it	works,	then	we	can	build	on	it	to	tackle	the	tougher	questions	above.
What	I	like	about	this	physics	perspective	is	that	it	transforms	the	hard
problem	that	we	as	humans	have	struggled	with	for	millennia	into	a	more
focused	version	that’s	easier	to	tackle	with	the	methods	of	science.	Instead	of
starting	with	a	hard	
problem
	of	why	an	arrangement	of	particles	can	feel
conscious,	let’s	start	with	a	hard	
fact
	that	some	arrangements	of	particles	do	feel
conscious	while	others	don’t.	For	example,	you	know	that	the	particles	that	make
up	your	brain	are	in	a	conscious	arrangement	right	now,	but	not	when	you’re	in

problem
fact
deep	dreamless	sleep.
This	physics	perspective	leads	to	three	separate	hard	questions	about
consciousness,	as	shown	in	
figure	8.1
.	First	of	all,	what	properties	of	the	particle

arrangement	make	the	difference?	Specifically,	what	physical	properties
distinguish	conscious	and	unconscious	systems?	If	we	can	answer	that,	then	we
can	figure	out	which	AI	systems	are	conscious.	In	the	more	immediate	future,	it
can	also	help	emergency-room	doctors	determine	which	unresponsive	patients
are	conscious.
Second,	how	do	physical	properties	determine	what	the	experience	is	like?
Specifically,	what	determines	
qualia,
	basic	building	blocks	of	consciousness
such	as	the	redness	of	a	rose,	the	sound	of	a	cymbal,	the	smell	of	a	steak,	the
taste	of	a	tangerine	or	the	pain	of	a	pinprick?
*2
Third,	why	is	anything	conscious?	In	other	words,	is	there	some	deep
undiscovered	explanation	for	why	clumps	of	matter	can	be	conscious,	or	is	this
just	an	unexplainable	brute	fact	about	the	way	the	world	works?
The	computer	scientist	Scott	Aaronson,	a	former	MIT	colleague	of	mine,	has
lightheartedly	called	the	first	question	the	“pretty	hard	problem”	(PHP),	as	has

*2
David	Chalmers.	In	that	spirit,	let’s	call	the	other	two	the	“even	harder	problem”
(EHP)	and	the	“really	hard	problem”	(RHP),	as	illustrated	in	
figure	8.1
.
*3

When	people	tell	me	that	consciousness	research	is	a	hopeless	waste	of	time,	the
main	argument	they	give	is	that	it’s	“unscientific”	and	always	will	be.	But	is	that
really	true?	The	influential	Austro-British	philosopher	Karl	Popper	popularized
the	now	widely	accepted	adage	“If	it’s	not	falsifiable,	it’s	not	scientific.”	In	other
words,	science	is	all	about	testing	theories	against	observations:	if	a	theory	can’t
be	tested	even	in	principle,	then	it’s	logically	impossible	to	ever	falsify	it,	which
by	Popper’s	definition	means	that	it’s	unscientific.
So	could	there	be	a	scientific	theory	that	answers	any	of	the	three
consciousness	questions	in	
figure	8.1
?	Please	let	me	try	to	persuade	you	that	the
answer	is	a	resounding	YES!,	at	least	for	the	pretty	hard	problem:	“What
physical	properties	distinguish	conscious	and	unconscious	systems?”	Suppose
that	someone	has	a	theory	that,	given	any	physical	system,	answers	the	question

figure	8.1
of	whether	the	system	is	conscious	with	“yes,”	“no”	or	“unsure.”	Let’s	hook
your	brain	up	to	a	device	that	measures	some	of	the	information	processing	in
different	parts	of	your	brain,	and	let’s	feed	this	information	into	a	computer
program	that	uses	the	consciousness	theory	to	predict	which	parts	of	that
information	are	conscious,	and	presents	you	with	its	predictions	in	real	time	on	a
screen,	as	in	
figure	8.2
.	First	you	think	of	an	apple.	The	screen	informs	you	that
there’s	information	about	an	apple	in	your	brain	which	you’re	aware	of,	but	that

figure	8.2
there’s	also	information	in	your	brainstem	about	your	pulse	that	you’re	unaware
of.	Would	you	be	impressed?	Although	the	first	two	predictions	of	the	theory
were	correct,	you	decide	to	do	some	more	rigorous	testing.	You	think	about	your
mother	and	the	computer	informs	you	that	there’s	information	in	your	brain
about	your	mother	but	that	you’re	unaware	of	this.	The	theory	made	an	incorrect
prediction,	which	means	that	it’s	ruled	out	and	goes	in	the	garbage	dump	of

scientific	history	together	with	Aristotelian	mechanics,	the	luminiferous	aether,
geocentric	cosmology	and	countless	other	failed	ideas.	Here’s	the	key	point:
Although	the	theory	was	wrong,	it	was	
scientific
!	Had	it	not	been	scientific,	you
wouldn’t	have	been	able	to	test	it	and	rule	it	out.
Someone	might	criticize	this	conclusion	and	say	that	
they
	have	no	evidence	of
what	you’re	conscious	of,	or	even	of	you	being	conscious	at	all:	although	they
heard	you	say	that	you’re	conscious,	an	unconscious	zombie	could	conceivably

say	the	same	thing.	But	this	doesn’t	make	that	consciousness	theory	unscientific,
because	they	can	trade	places	with	you	and	test	whether	it	correctly	predicts
their	own
	conscious	experiences.

Figure	8.2:	Suppose	that	a	computer	measures	information	being	processed	in	your	brain	and
predicts	which	parts	of	it	you’re	aware	of	according	to	a	theory	of	consciousness.	You	can
scientifically	test	this	theory	by	checking	whether	its	predictions	are	correct,	matching	your
subjective	experience.
On	the	other	hand,	if	the	theory	refuses	to	make	any	predictions,	merely
replying	“unsure”	whenever	queried,	then	it’s	untestable	and	hence	unscientific.
This	might	happen	because	it’s	applicable	only	in	some	situations,	because	the
required	computations	are	too	hard	to	carry	out	in	practice	or	because	the	brain
sensors	are	no	good.	Today’s	most	popular	scientific	theories	tend	to	be
somewhere	in	the	middle,	giving	testable	answers	to	some	but	not	all	of	our
questions.	For	example,	our	core	theory	of	physics	will	refuse	to	answer
questions	about	systems	that	are	simultaneously	extremely	small	(requiring
quantum	mechanics)	and	extremely	heavy	(requiring	general	relativity),	because

we	haven’t	yet	figured	out	which	mathematical	equations	to	use	in	this	case.
This	core	theory	will	also	refuse	to	predict	the	exact	masses	of	all	possible	atoms
—in	this	case,	we	think	we	have	the	necessary	equations,	but	we	haven’t
managed	to	accurately	compute	their	solutions.	The	more	dangerously	a	theory
lives	by	sticking	its	neck	out	and	making	testable	predictions,	the	more	useful	it
is,	and	the	more	seriously	we	take	it	if	it	survives	all	our	attempts	to	kill	it.	Yes,
we	can	only	test	
some
	predictions	of	consciousness	theories,	but	that’s	how	it	is

for	
all
	physical	theories.	So	let’s	not	waste	time	whining	about	what	we	can’t
test,	but	get	to	work	testing	what	we	
can
	test!
In	summary,	any	theory	predicting	which	physical	systems	are	conscious	(the
pretty	hard	problem)	is	scientific,	as	long	as	it	can	predict	which	of	your	brain
processes	are	conscious.	However,	the	testability	issue	becomes	less	clear	for	the
higher-up	questions	in	
figure	8.1
.	What	would	it	mean	for	a	theory	to	predict
how	you	subjectively	experience	the	color	red?	And	if	a	theory	purports	to
explain	why	there	is	such	a	thing	as	consciousness	in	the	first	place,	then	how	do
you	test	it	experimentally?	Just	because	these	questions	are	hard	doesn’t	mean
that	we	should	avoid	them,	and	we’ll	indeed	return	to	them	below.	But	when
confronted	with	several	related	unanswered	questions,	I	think	it’s	wise	to	tackle
the	easiest	one	first.	For	this	reason,	my	consciousness	research	at	MIT	is
focused	squarely	on	the	base	of	the	pyramid	in	
figure	8.1
.	I	recently	discussed

figure	8.1
this	strategy	with	my	fellow	physicist	Piet	Hut	from	Princeton,	who	joked	that
trying	to	build	the	top	of	the	pyramid	before	the	base	would	be	like	worrying
about	the	interpretation	of	quantum	mechanics	before	discovering	the
Schrödinger	equation,	the	mathematical	foundation	that	lets	us	predict	the
outcomes	of	our	experiments.
When	discussing	what’s	beyond	science,	it’s	important	to	remember	that	the
answer	depends	on	time!	Four	centuries	ago,	Galileo	Galilei	was	so	impressed

by	math-based	physics	theories	that	he	described	nature	as	“a	book	written	in	the
language	of	mathematics.”	If	he	threw	a	grape	and	a	hazelnut,	he	could
accurately	predict	the	shapes	of	their	trajectories	and	when	they	would	hit	the
ground.	Yet	he	had	no	clue	why	one	was	green	and	the	other	brown,	or	why	one
was	soft	and	the	other	hard—these	aspects	of	the	world	were	beyond	the	reach	of
science	at	the	time.	But	not	forever!	When	James	Clerk	Maxwell	discovered	his

eponymous	equations	in	1861,	it	became	clear	that	light	and	colors	could	also	be
understood	mathematically.	We	now	know	that	the	aforementioned	Schrödinger
equation,	discovered	in	1925,	can	be	used	to	predict	all	properties	of	matter,
including	what’s	soft	or	hard.	While	theoretical	progress	has	enabled	ever	more
scientific	predictions,	technological	progress	has	enabled	ever	more
experimental	tests:	almost	everything	we	now	study	with	telescopes,
microscopes	or	particle	colliders	was	once	beyond	science.	In	other	words,	the

purview	of	science	has	expanded	dramatically	since	Galileo’s	days,	from	a	tiny
fraction	of	all	phenomena	to	a	large	percentage,	including	subatomic	particles,
black	holes	and	our	cosmic	origins	13.8	billion	years	ago.	This	raises	the
question:	What’s	left?

To	me,	consciousness	is	the	elephant	in	the	room.	Not	only	do	you	know	that
you’re	conscious,	but	it’s	
all
	you	know	with	complete	certainty—everything	else
is	inference,	as	René	Descartes	pointed	out	back	in	Galileo’s	time.	Will
theoretical	and	technological	progress	eventually	bring	even	consciousness
firmly	into	the	domain	of	science?	We	don’t	know,	just	as	Galileo	didn’t	know
whether	we’d	one	day	understand	light	and	matter.
*4
	Only	one	thing	is
guaranteed:	we	won’t	succeed	if	we	don’t	try!	That’s	why	I	and	many	other
scientists	around	the	world	are	trying	hard	to	formulate	and	test	theories	of

Lots	of	information	processing	is	taking	place	in	our	heads	right	now.	Which	of
it	is	conscious	and	which	isn’t?	Before	exploring	consciousness	theories	and
what	they	predict,	let’s	look	at	what	experiments	have	taught	us	so	far,	ranging
from	traditional	low-tech	or	no-tech	observations	to	state-of-the-art	brain
measurements.

What	Behaviors	Are	Conscious?
If	you	multiply	32	by	17	in	your	head,	you’re	conscious	of	many	of	the	inner
workings	of	your	computation.	But	suppose	I	instead	show	you	a	portrait	of
Albert	Einstein	and	tell	you	to	say	the	name	of	its	subject.	As	we	saw	in	chapter
2,	this	too	is	a	computational	task:	your	brain	is	evaluating	a	function	whose
input	is	information	from	your	eyes	about	a	large	number	of	pixel	colors	and
whose	output	is	information	to	muscles	controlling	your	mouth	and	vocal	cords.
Computer	scientists	call	this	task	“image	classification”	followed	by	“speech
synthesis.”	Although	this	computation	is	way	more	complicated	than	your
multiplication	task,	you	can	do	it	much	faster,	seemingly	without	effort,	and
without	being	conscious	of	the	details	of	
how
	you	do	it.	Your	subjective
experience	consists	merely	of	looking	at	the	picture,	experiencing	a	feeling	of
recognition	and	hearing	yourself	say	“Einstein.”
Psychologists	have	long	known	that	you	can	unconsciously	perform	a	wide

how
range	of	other	tasks	and	behaviors	as	well,	from	blink	reflexes	to	breathing,
reaching,	grabbing	and	keeping	your	balance.	Typically,	you’re	consciously
aware	of	what	you	did,	but	not	how	you	did	it.	On	the	other	hand,	behaviors	that
involve	unfamiliar	situations,	self-control,	complicated	logical	rules,	abstract
reasoning	or	manipulation	of	language	tend	to	be	conscious.	They’re	known	as
behavioral	correlates	of	consciousness,
	and	they’re	closely	linked	to	the

effortful,	slow	and	controlled	way	of	thinking	that	psychologists	call	“System
2.”
5
It’s	also	known	that	you	can	convert	many	routines	from	conscious	to
unconscious	through	extensive	practice,	for	example	walking,	swimming,
bicycling,	driving,	typing,	shaving,	shoe	tying,	computer-gaming	and	piano
playing.
6
	Indeed,	it’s	well	known	that	experts	do	their	specialties	best	when
they’re	in	a	state	of	“flow,”	aware	only	of	what’s	happening	at	a	higher	level,	and
unconscious	of	the	low-level	details	of	how	they’re	doing	it.	For	example,	try

2.”
5
playing.
6
reading	the	next	sentence	while	being	consciously	aware	of	every	single	letter,	as
when	you	first	learned	to	read.	Can	you	feel	how	much	slower	it	is,	compared	to
when	you’re	merely	conscious	of	the	text	at	the	level	of	words	or	ideas?
Indeed,	unconscious	information	processing	appears	not	only	to	be	possible,
but	also	to	be	more	the	rule	than	the	exception.	Evidence	suggests	that	of	the

roughly	10
7
	bits	of	information	that	enter	our	brain	each	second	from	our
sensory	organs,	we	can	be	aware	only	of	a	tiny	fraction,	with	estimates	ranging
from	10	to	50	bits.
7
	This	suggests	that	the	information	processing	that	we’re
consciously	aware	of	is	merely	the	tip	of	the	iceberg.
Taken	together,	these	clues	have	led	some	researchers	to	suggest	that
conscious	information	processing	should	be	thought	of	as	the	CEO	of	our	mind,
dealing	with	only	the	most	important	decisions	requiring	complex	analysis	of
data	from	all	over	the	brain.
8
	This	would	explain	why,	just	like	the	CEO	of	a
company,	it	usually	doesn’t	want	to	be	distracted	by	knowing	everything	its
underlings	are	up	to—but	it	can	find	them	out	if	desired.	To	experience	this
selective	attention	in	action,	look	at	that	word	“desired”	again:	fix	your	gaze	on
the	dot	over	the	“i”	and,	without	moving	your	eyes,	shift	your	attention	from	the
dot	to	the	whole	letter	and	then	to	the	whole	word.	Although	the	information

8
from	your	retina	stayed	the	same,	your	conscious	experience	changed.	The	CEO
metaphor	also	explains	why	expertise	becomes	unconscious:	after	painstakingly
figuring	out	how	to	read	and	type,	the	CEO	delegates	these	routine	tasks	to
unconscious	subordinates	to	be	able	to	focus	on	new	higher-level	challenges.

Where	Is	Consciousness?
Clever	experiments	and	analyses	have	suggested	that	consciousness	is	limited
not	merely	to	certain	behaviors,	but	also	to	certain	parts	of	the	brain.	Which	are
the	prime	suspects?	Many	of	the	first	clues	came	from	patients	with	brain
lesions:	localized	brain	damage	caused	by	accidents,	strokes,	tumors	or
infections.	But	this	was	often	inconclusive.	For	example,	does	the	fact	that
lesions	in	the	back	of	the	brain	can	cause	blindness	mean	that	this	is	the	site	of
visual	consciousness,	or	does	it	merely	mean	that	visual	information	passes
through	there	en	route	to	wherever	it	will	later	become	conscious,	just	as	it	first
passes	through	the	eyes?
Although	lesions	and	medical	interventions	haven’t	pinpointed	the	locations
of	conscious	experiences,	they’ve	helped	narrow	down	the	options.	For	example,
I	know	that	although	I	experience	pain	in	my	hand	as	actually	occurring	there,
the	pain	experience	must	occur	elsewhere,	because	a	surgeon	once	switched	off

my	hand	pain	without	doing	anything	to	my	hand:	he	merely	anesthetized	nerves
in	my	shoulder.	Moreover,	some	amputees	experience	phantom	pain	that	feels	as
though	it’s	in	their	nonexistent	hand.	As	another	example,	I	once	noticed	that
when	I	looked	only	with	my	right	eye,	part	of	my	visual	field	was	missing—a
doctor	determined	that	my	retina	was	coming	loose	and	reattached	it.	In	contrast,
patients	with	certain	brain	lesions	experience	
hemineglect,
	where	they	too	miss

information	from	half	their	visual	field,	but	aren’t	even	aware	that	it’s	missing—
for	example,	failing	to	notice	and	eat	the	food	on	the	left	half	of	their	plate.	It’s
as	if	consciousness	about	half	of	their	world	has	disappeared.	But	are	those
damaged	brain	areas	supposed	to	generate	the	spatial	experience,	or	were	they
merely	feeding	spatial	information	to	the	sites	of	consciousness,	just	as	my	retina
did?
The	pioneering	U.S.-Canadian	neurosurgeon	Wilder	Penfield	found	in	the

where	they	too	miss
did?
1930s	that	his	neurosurgery	patients	reported	different	parts	of	their	body	being
touched	when	he	electrically	stimulated	specific	brain	areas	in	what’s	now	called
the	
somatosensory	cortex
	(
figure	8.3
).
9
	He	also	found	that	they	involuntarily
moved	different	parts	of	their	body	when	he	stimulated	brain	areas	in	what’s	now
called	the	
motor	cortex.
	But	does	that	mean	that	information	processing	in	these
brain	areas	corresponds	to	consciousness	of	touch	and	motion?

the	
	(
figure	8.3
).
9
called	the	
Fortunately,	modern	technology	is	now	giving	us	much	more	detailed	clues.

Although	we’re	still	nowhere	near	being	able	to	measure	every	single	firing	of
all	of	your	roughly	hundred	billion	neurons,	brain-reading	technology	is
advancing	rapidly,	involving	techniques	with	intimidating	names	such	as	fMRI,
EEG,	MEG,	ECoG,	ePhys	and	fluorescent	voltage	sensing.	fMRI,	which	stands
for	functional	magnetic	resonance	imaging,	measures	the	magnetic	properties	of
hydrogen	nuclei	to	make	a	3-D	map	of	your	brain	roughly	every	second,	with
millimeter	resolution.	EEG	(electroencephalography)	and	MEG
(magnetoencephalography)	measure	the	electric	and	magnetic	field	outside	your
head	to	map	your	brain	thousands	of	times	per	second,	but	with	poor	resolution,
unable	to	distinguish	features	smaller	than	a	few	centimeters.	If	you’re
squeamish,	you’ll	appreciate	that	these	three	techniques	are	all	noninvasive.	If
you	don’t	mind	opening	up	your	skull,	you	have	additional	options.	ECoG
(electrocorticography)	involves	placing	say	a	hundred	wires	on	the	surface	of

your	brain,	while	ePhys	(electrophysiology)	involves	inserting	microwires,
which	are	sometimes	thinner	than	a	human	hair,	deep	into	the	brain	to	record
voltages	from	as	many	as	a	thousand	simultaneous	locations.	Many	epileptic
patients	spend	days	in	the	hospital	while	ECoG	is	used	to	figure	out	what	part	of
their	brain	is	triggering	seizures	and	should	be	resected,	and	kindly	agree	to	let
neuroscientists	perform	consciousness	experiments	on	them	in	the	meantime.
Finally,	fluorescent	voltage	sensing	involves	genetically	manipulating	neurons	to

emit	flashes	of	light	when	firing,	enabling	their	activity	to	be	measured	with	a
microscope.	Out	of	all	the	techniques,	it	has	the	potential	to	rapidly	monitor	the
largest	number	of	neurons,	at	least	in	animals	with	transparent	brains—such	as
the	
C.	elegans
	worm	with	its	302	neurons	and	the	larval	zebrafish	with	its	about
100,000.

Figure	8.3:	The	visual,	auditory,	somatosensory	and	motor	cortices	are	involved	with	vision,
hearing,	the	sense	of	touch	and	motion	activation,	respectively—but	that	doesn’t	prove	they’re
where	
consciousness
	of	vision,	hearing,	touch	and	motion	occurs.	Indeed,	recent	research
suggests	that	the	primary	visual	cortex	is	completely	unconscious,	together	with	the	cerebellum
and	brainstem.	Image	courtesy	of	Lachina	(www.lachina.com).
Although	Francis	Crick	warned	Christof	Koch	about	studying	consciousness,
Christof	refused	to	give	up	and	and	eventually	won	Francis	over.	In	1990,	they
wrote	a	seminal	paper	about	what	they	called	“neural	correlates	of
consciousness”	(NCCs),	asking	which	specific	brain	processes	corresponded	to
conscious	experiences.	For	thousands	of	years,	thinkers	had	had	access	to	the
information	processing	in	their	brains	only	via	their	subjective	experience	and

behavior.	Crick	and	Koch	pointed	out	that	brain-reading	technology	was
suddenly	providing	independent	access	to	this	information,	allowing	scientific
study	of	which	information	processing	corresponded	to	what	conscious
experience.	Sure	enough,	technology-driven	measurements	have	by	now	turned
the	quest	for	NCCs	into	quite	a	mainstream	part	of	neuroscience,	one	whose
thousands	of	publications	extend	into	even	the	most	prestigious	journals.
10
What	are	the	conclusions	so	far?	To	get	a	flavor	for	NCC	detective	work,	let’s
first	ask	whether	your	retina	is	conscious,	or	whether	it’s	merely	a	zombie
system	that	records	visual	information,	processes	it	and	sends	it	on	to	a	system
downstream	in	your	brain	where	your	subjective	visual	experience	occurs.	In	the
left	panel	of	
figure	8.4
,	which	square	is	darker:	the	one	labeled	A	or	B?	A,	right?
No,	they’re	in	fact	identically	colored,	which	you	can	verify	by	looking	at	them
through	small	holes	between	your	fingers.	This	proves	that	your	visual

figure	8.4
experience	can’t	reside	entirely	in	your	retina,	since	if	it	did,	they’d	look	the
same.
Now	look	at	the	right	panel	of	
figure	8.4
.	Do	you	see	two	women	or	a	vase?	If
you	look	long	enough,	you’ll	subjectively	experience	both	in	succession,	even
though	the	information	reaching	your	retina	remains	the	same.	By	measuring
what	happens	in	your	brain	during	the	two	situations,	one	can	tease	apart	what
makes	the	difference—and	it’s	not	the	retina,	which	behaves	identically	in	both
cases.

same.
figure	8.4
cases.
The	death	blow	to	the	conscious-retina	hypothesis	comes	from	a	technique
called	“continuous	flash	suppression”	pioneered	by	Christof	Koch,	Stanislas
Dehaene	and	collaborators:	it’s	been	discovered	that	if	you	make	one	of	your
eyes	watch	a	complicated	sequence	of	rapidly	changing	patterns,	then	this	will
distract	your	visual	system	to	such	an	extent	that	you’ll	be	completely	unaware
of	a	still	image	shown	to	the	other	eye.
11
	In	summary,	you	can	have	a	visual

cases.
11
image	in	your	retina	without	experiencing	it,	and	you	can	(while	dreaming)
experience	an	image	without	it	being	on	your	retina.	This	proves	that	your	two
retinas	don’t	host	your	visual	consciousness	any	more	than	a	video	camera	does,
even	though	they	perform	complicated	computations	involving	over	a	hundred
million	neurons.

Figure	8.4:	Which	square	is	darker—A	or	B?	What	do	you	see	on	the	right—a	vase,	two	women
or	both	in	succession?	Illusions	such	as	these	prove	that	your	visual	consciousness	can’t	be	in
your	eyes	or	other	early	stages	of	your	visual	system,	because	it	doesn’t	depend	only	on	what’s	in
the	picture.
NCC	researchers	also	use	continuous	flash	suppression,	unstable
visual/auditory	illusions	and	other	tricks	to	pinpoint	which	of	your	brain	regions
are
	responsible	for	each	of	your	conscious	experiences.	The	basic	strategy	is	to
compare	what	your	neurons	are	doing	in	two	situations	where	essentially
everything	(including	your	sensory	input)	is	the	same—except	your	conscious
experience.	The	parts	of	your	brain	that	are	measured	to	behave	differently	are
then	identified	as	NCCs.
Such	NCC	research	has	proven	that	none	of	your	consciousness	resides	in
your	gut,	even	though	that’s	the	location	of	your	enteric	nervous	system	with	its
whopping	half-billion	neurons	that	compute	how	to	optimally	digest	your	food;

feelings	such	as	hunger	and	nausea	are	instead	produced	in	your	brain.	Similarly,
none	of	your	consciousness	appears	to	reside	in	the	brainstem,	the	bottom	part	of
the	brain	that	connects	to	the	spinal	cord	and	controls	breathing,	heart	rate	and
blood	pressure.	More	shockingly,	your	consciousness	doesn’t	appear	to	extend	to
your	cerebellum	(
figure	8.3
),	which	contains	about	two-thirds	of	all	your
neurons:	patients	whose	cerebellum	is	destroyed	experience	slurred	speech	and

figure	8.3
clumsy	motion	reminiscent	of	a	drunkard,	but	remain	fully	conscious.
The	question	of	which	parts	of	your	brain	
are
	responsible	for	consciousness

remains	open	and	controversial.	Some	recent	NCC	research	suggests	that	your
consciousness	mainly	resides	in	a	“hot	zone”	involving	the	thalamus	(near	the
middle	of	your	brain)	and	the	rear	part	of	the	cortex	(the	outer	brain	layer
consisting	of	a	crumpled-up	six-layer	sheet	which,	if	flattened	out,	would	have
the	area	of	a	large	dinner	napkin).
12
	This	same	research	controversially	suggests
that	the	primary	visual	cortex	at	the	very	back	of	the	head	is	an	exception	to	this,
being	as	unconscious	as	your	eyeballs	and	your	retinas.

When	Is	Consciousness?
So	far,	we’ve	looked	at	experimental	clues	regarding	what	types	of	information
processing	are	conscious	and	where	consciousness	occurs.	But	
when
	does	it
occur?	When	I	was	a	kid,	I	used	to	think	that	we	become	conscious	of	events	as
they	happen,	with	absolutely	no	time	lag	or	delay.	Although	that’s	still	how	it
subjectively	feels	to	me,	it	clearly	can’t	be	correct,	since	it	takes	time	for	my
brain	to	process	the	information	that	enters	via	my	sensory	organs.	NCC
researchers	have	carefully	measured	how	long,	and	Christof	Koch’s	summary	is
that	it	takes	about	a	quarter	of	a	second	from	when	light	enters	your	eye	from	a
complex	object	until	you	consciously	perceive	seeing	it	as	what	it	is.
13
	This
means	that	if	you’re	driving	down	a	highway	at	fifty-five	miles	per	hour	and
suddenly	see	a	squirrel	a	few	meters	in	front	of	you,	it’s	too	late	for	you	to	do
anything	about	it,	because	you’ve	already	run	over	it!
In	summary,	your	consiousness	lives	in	the	past,	with	Christof	Koch

13
	This
estimating	that	it	lags	behind	the	outside	world	by	about	a	quarter	second.
Intriguingly,	you	can	often	react	to	things	faster	than	you	can	become	conscious
of	them,	which	proves	that	the	information	processing	in	charge	of	your	most
rapid	reactions	must	be	unconscious.	For	example,	if	a	foreign	object	approaches
your	eye,	your	blink	reflex	can	close	your	eyelid	within	a	mere	tenth	of	a	second.
It’s	as	if	one	of	your	brain	systems	receives	ominous	information	from	the	visual
system,	computes	that	your	eye	is	in	danger	of	getting	struck,	emails	your	eye

muscles	instructions	to	blink	and	simultaneously	emails	the	conscious	part	of
your	brain	saying	“Hey,	we’re	going	to	blink.”	By	the	time	this	email	has	been
read	and	included	into	your	conscious	experience,	the	blink	has	already
happened.
Indeed,	the	system	that	reads	that	email	is	continually	bombarded	with
messages	from	all	over	your	body,	some	more	delayed	than	others.	It	takes
longer	for	nerve	signals	to	reach	your	brain	from	your	fingers	than	from	your
face	because	of	distance,	and	it	takes	longer	for	you	to	analyze	images	than

happened.
sounds	because	it’s	more	complicated—which	is	why	Olympic	races	are	started
with	a	bang	rather	than	with	a	visual	cue.	Yet	if	you	touch	your	nose,	you
consciously	experience	the	sensation	on	your	nose	and	fingertip	as	simultaneous,
and	if	you	clap	your	hands,	you	see,	hear	and	feel	the	clap	at	exactly	the	same
time.
14
	This	means	that	your	full	conscious	experience	of	an	event	isn’t	created

until	the	last	slowpoke	email	reports	have	trickled	in	and	been	analyzed.
A	famous	family	of	NCC	experiments	pioneered	by	physiologist	Benjamin
Libet	has	shown	that	the	sort	of	actions	you	can	perform	unconsciously	aren’t
limited	to	rapid	responses	such	as	blinks	and	ping-pong	smashes,	but	also
include	certain	decisions	that	you	might	attribute	to	free	will—brain
measurements	can	sometimes	predict	your	decision	before	you	become
conscious	of	having	made	it.
15

We’ve	just	seen	that,	although	we	still	don’t	understand	consciousness,	we	have
amazing	amounts	of	experimental	data	about	various	aspects	of	it.	But	all	this
data	comes	from	
brains,
	so	how	can	it	teach	us	anything	about	consciousness	in
machines
?	This	requires	a	major	extrapolation	beyond	our	current	experimental
domain.	In	other	words,	it	requires	a	
theory.

Why	a	Theory?
To	appreciate	why,	let’s	compare	theories	of	consciousness	with	theories	of
gravity.	Scientists	started	taking	Newton’s	theory	of	gravity	seriously	because
they	got	more	out	of	it	than	they	put	into	it:	simple	equations	that	fit	on	a	napkin
could	accurately	predict	the	outcome	of	every	gravity	experiment	ever
conducted.	They	therefore	also	took	seriously	its	predictions	far	beyond	the
domain	where	it	had	been	tested,	and	these	bold	extrapolations	turned	out	to
work	even	for	the	motions	of	galaxies	in	clusters	millions	of	light-years	across.
However,	the	predictions	were	off	by	a	tiny	amount	for	the	motion	of	Mercury
around	the	Sun.	Scientists	then	started	taking	seriously	Einstein’s	improved
theory	of	gravity,	general	relativity,	because	it	was	arguably	even	more	elegant
and	economical,	and	correctly	predicted	even	what	Newton’s	theory	got	wrong.
They	consequently	took	seriously	also	its	predictions	far	beyond	the	domain

where	it	had	been	tested,	for	phenomena	as	exotic	as	black	holes,	gravitational
waves	in	the	very	fabric	of	spacetime,	and	the	expansion	of	our	Universe	from	a
hot	fiery	origin—all	of	which	were	subsequently	confirmed	by	experiment.
Analogously,	if	a	mathematical	theory	of	consciousness	whose	equations	fit
on	a	napkin	could	successfully	predict	the	outcomes	of	all	experiments	we
perform	on	brains,	then	we’d	start	taking	seriously	not	merely	the	theory	itself,
but	also	its	predictions	for	consciousness	beyond	brains—for	example,	in
machines.

Consciousness	from	a	Physics	Perspective
Although	some	theories	of	consciousness	date	back	to	antiquity,	most	modern
ones	are	grounded	in	neuropsychology	and	neuroscience,	attempting	to	explain
and	predict	consciousness	in	terms	of	neural	events	occurring	in	the	brain.
16
Although	these	theories	have	made	some	successful	predictions	for	neural
correlates	of	consciousness,	they	neither	can	nor	aspire	to	make	predictions
about	machine	consciousness.	To	make	the	leap	from	brains	to	machines,	we
need	to	generalize	from	NCCs	to	PCCs:	
physical	correlates	of	consciousness,
defined	as	the	patterns	of	moving	particles	that	are	conscious.	Because	if	a
theory	can	correctly	predict	what’s	conscious	and	what’s	not	by	referring	only	to
physical	building	blocks	such	as	elementary	particles	and	force	fields,	then	it	can
make	predictions	not	merely	for	brains,	but	also	for	any	other	arrangements	of
matter,	including	future	AI	systems.	So	let’s	take	a	physics	perspective:	What
particle	arrangements	are	conscious?

But	this	really	raises	another	question:	How	can	something	as	complex	as
consciousness	be	made	of	something	as	simple	as	particles?	I	think	it’s	because
it’s	a	phenomenon	that	has	properties	above	and	beyond	those	of	its	particles.	In
physics,	we	call	such	phenomena	“emergent.”
17
	Let’s	understand	this	by	looking
at	an	emergent	phenomenon	that’s	simpler	than	consciousness:	wetness.
A	drop	of	water	is	wet,	but	an	ice	crystal	and	a	cloud	of	steam	aren’t,	even
though	they’re	made	of	identical	water	molecules.	Why?	Because	the	property	of

17
wetness	depends	only	on	the	arrangement	of	the	molecules.	It	makes	absolutely
no	sense	to	say	that	a	single	water	molecule	is	wet,	because	the	phenomenon	of
wetness	emerges	only	when	there	are	many	molecules,	arranged	in	the	pattern
we	call	liquid.	So	solids,	liquids	and	gases	are	all	emergent	phenomena:	they’re
more	than	the	sum	of	their	parts,	because	they	have	properties	above	and	beyond
the	properties	of	their	particles.	They	have	properties	that	their	particles	lack.
Now	just	like	solids,	liquids	and	gases,	I	think	consciousness	is	an	emergent

phenomenon,	with	properties	above	and	beyond	those	of	its	particles.	For
example,	entering	deep	sleep	extinguishes	consciousness,	by	merely	rearranging
the	particles.	In	the	same	way,	my	consciousness	would	disappear	if	I	froze	to
death,	which	would	rearrange	my	particles	in	a	more	unfortunate	way.
When	you	put	lots	of	particles	together	to	make	anything	from	water	to	a

brain,	new	phenomena	with	observable	properties	emerge.	We	physicists	love
studying	these	emergent	properties,	which	can	often	be	identified	by	a	small	set
of	numbers	that	you	can	go	out	and	measure—quantities	such	as	how	viscous	the
substance	is,	how	compressible	it	is	and	so	on.	For	example,	if	a	substance	is	so
viscous	that	it’s	rigid,	we	call	it	a	solid,	otherwise	we	call	it	a	fluid.	And	if	a	fluid
isn’t	compressible,	we	call	it	a	liquid,	otherwise	we	call	it	a	gas	or	a	plasma,
depending	on	how	well	it	conducts	electricity.

Consciousness	as	Information
So	could	there	be	analogous	quantities	that	quantify	consciousness?	The	Italian
neuroscientist	Giulio	Tononi	has	proposed	one	such	quantity,	which	he	calls	the
“integrated	information,”
	denoted	by	the	Greek	letter	
Φ
	(
Phi
),	which	basically
measures	how	much	different	parts	of	a	system	know	about	each	other	(see
figure	8.5
).

Figure	8.5:	Given	a	physical	process	that,	with	the	passage	of	time,	transforms	the	initial	state	of
a	system	into	a	new	state,	its	
integrated	information
	
Φ
	measures	inability	to	split	the	process	into
independent	parts.	If	the	future	state	of	each	part	depends	only	on	its	own	past,	not	on	what	the
other	part	has	been	doing,	then	
Φ
	=	0:	what	we	called	one	system	is	really	two	independent
systems	that	don’t	communicate	with	each	other	at	all.
I	first	met	Giulio	at	a	2014	physics	conference	in	Puerto	Rico	to	which	I’d
invited	him	and	Christof	Koch,	and	he	struck	me	as	the	ultimate	renaissance	man
who’d	have	blended	right	in	with	Galileo	and	Leonardo	da	Vinci.	His	quiet
demeanor	couldn’t	hide	his	incredible	knowledge	of	art,	literature	and
philosophy,	and	his	culinary	reputation	preceded	him:	a	cosmopolitan	TV
journalist	had	recently	told	me	how	Giulio	had,	in	just	a	few	minutes,	whipped
up	the	most	delicious	salad	he’d	tasted	in	his	life.	I	soon	realized	that	behind	his

soft-spoken	demeanor	was	a	fearless	intellect	who’d	follow	the	evidence
wherever	it	took	him,	regardless	of	the	preconceptions	and	taboos	of	the
establishment.	Just	as	Galileo	had	pursued	his	mathematical	theory	of	motion
despite	establishment	pressure	not	to	challenge	geocentrism,	Giulio	had
developed	the	most	mathematically	precise	consciousness	theory	to	date,
integrated	information	theory
	(IIT).

I’d	been	arguing	for	decades	that	consciousness	is	the	way	information	feels
when	being	processed	in	certain	complex	ways.
18
	IIT	agrees	with	this	and
replaces	my	vague	phrase	“certain	complex	ways”	by	a	precise	definition:	the
information	processing	needs	to	be	integrated,	that	is,	
Φ
	needs	to	be	large.
Giulio’s	argument	for	this	is	as	powerful	as	it	is	simple:	the	conscious	system
needs	to	be	integrated	into	a	unified	whole,	because	if	it	instead	consisted	of	two
independent	parts,	then	they’d	feel	like	two	separate	conscious	entities	rather
than	one.	In	other	words,	if	a	conscious	part	of	a	brain	or	computer	can’t
communicate	with	the	rest,	then	the	rest	can’t	be	part	of	its	subjective
experience.
Giulio	and	his	collaborators	have	measured	a	simplified	version	of	
Φ
	by	using
EEG	to	measure	the	brain’s	response	to	magnetic	stimulation.	Their
“consciousness	detector”	works	really	well:	it	determined	that	patients	were
conscious	when	they	were	awake	or	dreaming,	but	unconscious	when	they	were

Φ
	by	using
anesthetized	or	in	deep	sleep.	It	even	discovered	consciousness	in	two	patients
suffering	from	“locked-in”	syndrome,	who	couldn’t	move	or	communicate	in
any	normal	way.
19
	So	this	is	emerging	as	a	promising	technology	for	doctors	in
the	future	to	figure	out	whether	certain	patients	are	conscious	or	not.

Anchoring	Consciousness	in	Physics
IIT	is	defined	only	for	discrete	systems	that	can	be	in	a	finite	number	of	states,
for	example	bits	in	a	computer	memory	or	oversimplified	neurons	that	can	be
either	on	or	off.	This	unfortunately	means	that	IIT	isn’t	defined	for	most
traditional	physical	systems,	which	can	change	continuously—for	example,	the
position	of	a	particle	or	the	strength	of	a	magnetic	field	can	take	any	of	an
infinite	number	of	values.
20
	If	you	try	to	apply	the	IIT	formula	to	such	systems,
you’ll	typically	get	the	unhelpful	result	that	
Φ
	is	infinite.	Quantum-mechanical
systems	can	be	discrete,	but	the	original	IIT	isn’t	defined	for	quantum	systems.
So	how	can	we	anchor	IIT	and	other	information-based	consciousness	theories
on	a	solid	physical	foundation?
We	can	do	this	by	building	on	what	we	learned	in	chapter	2	about	how	clumps
of	matter	can	have	emergent	properties	that	are	related	to	information.	We	saw
that	for	something	to	be	usable	as	a	memory	device	that	can	store	information,	it

Φ
needs	to	have	many	long-lived	states.	We	also	saw	that	being	
	a
substance	that	can	do	computations,	in	addition	requires	complex	dynamics:	the
laws	of	physics	need	to	make	it	change	in	ways	that	are	complicated	enough	to
be	able	to	implement	arbitrary	information	processing.	Finally,	we	saw	how	a
neural	network,	for	example,	is	a	powerful	substrate	for	learning	because,	simply
by	obeying	the	laws	of	physics,	it	can	rearrange	itself	to	get	better	and	better	at

	a
implementing	desired	computations.	Now	we’re	asking	an	additional	question:
What	makes	a	blob	of	matter	able	to	have	a	subjective	experience?	In	other
words,	under	what	conditions	will	a	blob	of	matter	be	able	to	do	these	four
things?
1.
remember
2.
compute
3.
learn
4.
We	explored	the	first	three	in	chapter	2,	and	are	now	tackling	the	fourth.	Just
as	Margolus	and	Toffoli	coined	the	term	
	for	a	substance	that	can
perform	arbitrary	computations,	I	like	to	use	the	term	
sentronium
	for	the	most

general	substance	that	has	subjective	experience	(is	sentient).
*5
But	how	can	consciousness	feel	so	non-physical	if	it’s	in	fact	a	physical
phenomenon?	How	can	it	feel	so	independent	of	its	physical	substrate?	I	think
it’s	because	it	
is
	rather	independent	of	its	physical	substrate,	the	stuff	in	which	it
is	a	pattern!	We	encountered	many	beautiful	examples	of	substrate-independent
patterns	in	chapter	2,	including	waves,	memories	and	computations.	We	saw
how	they	weren’t	merely	more	than	their	parts	(emergent),	but	rather
independent	of	their	parts,	taking	on	a	life	of	their	own.	For	example,	we	saw
how	a	future	simulated	mind	or	computer-game	character	would	have	no	way	of
knowing	whether	it	ran	on	Windows,	Mac	OS,	an	Android	phone	or	some	other
operating	system,	because	it	would	be	substrate-independent.	Nor	could	it	tell
whether	the	logic	gates	of	its	computer	were	made	of	transistors,	optical	circuits
or	other	hardware.	Or	what	the	fundamental	laws	of	physics	are—they	could	be

anything	as	long	as	they	allow	the	construction	of	universal	computers.
In	summary,	I	think	that	consciousness	is	a	physical	phenomenon	that	feels
non-physical	because	it’s	like	waves	and	computations:	it	has	properties
independent	of	its	specific	physical	substrate.	This	follows	logically	from	the
consciousness-as-information	idea.	This	leads	to	a	radical	idea	that	I	really	like:
If	consciousness	is	the	way	that	information	feels	when	it’s	processed	in	certain
ways,	then	it	must	be	substrate-independent;	it’s	only	the	structure	of	the

information	processing	that	matters,	not	the	structure	of	the	matter	doing	the
information	processing.	In	other	words,	consciousness	is	substrate-independent
twice	over!
As	we’ve	seen,	physics	describes	patterns	in	spacetime	that	correspond	to
particles	moving	around.	If	the	particle	arrangements	obey	certain	principles,
they	give	rise	to	emergent	phenomena	that	are	pretty	independent	of	the	particle
substrate,	and	have	a	totally	different	feel	to	them.	A	great	example	of	this	is

information	processing,	in	computronium.	But	we’ve	now	taken	this	idea	to
another	level:	
If	the	information	processing	itself	obeys	certain	principles,	it	can
give	rise	to
	
the	higher-level	emergent	phenomenon	that	we	call	consciousness.
This	places	your	conscious	experience	not	one	but	two	levels	up	from	the	matter.
No	wonder	your	mind	feels	non-physical!
This	raises	a	question:	What	are	these	principles	that	information	processing
needs	to	obey	to	be	conscious?	I	don’t	pretend	to	know	what	conditions	are
sufficient

	to	guarantee	consciousness,	but	here	are	four	
necessary
	conditions	that
I’d	bet	on	and	have	explored	in	my	research:

Principle
Definition
Information
principle
A	conscious	system	has	substantial	information-storage
capacity.
Dynamics
principle
A	conscious	system	has	substantial	information-processing
capacity.
Independence
principle
A	conscious	system	has	substantial	independence	from	the
rest	of	the	world.
Integration
principle
A	conscious	system	cannot	consist	of	nearly	independent
parts.
As	I	said,	I	think	that	consciousness	is	the	way	information	feels	when	being
processed	in	certain	ways.	This	means	that	to	be	conscious,	a	system	needs	to	be
able	to	store	and	process	information,	implying	the	first	two	principles.	Note	that
the	memory	doesn’t	need	to	last	long:	I	recommend	watching	this	touching	video
of	Clive	Wearing,	who	appears	perfectly	conscious	even	though	his	memories
last	less	than	a	minute.
21
	I	think	that	a	conscious	system	also	needs	to	be	fairly
independent	from	the	rest	of	the	world,	because	otherwise	it	wouldn’t
subjectively	feel	that	it	had	any	independent	existence	whatsoever.	Finally,	I

21
think	that	the	conscious	system	needs	to	be	integrated	into	a	unified	whole,	as
Giulio	Tononi	argued,	because	if	it	consisted	of	two	independent	parts,	then	they
would	feel	like	two	separate	conscious	entities,	rather	than	one.	The	first	three
principles	imply	
autonomy:
	that	the	system	is	able	to	retain	and	process
information	without	much	outside	interference,	hence	determining	its	own
future.	All	four	principles	together	mean	that	a	system	is	autonomous	but	its
parts	aren’t.

autonomy:
If	these	four	principles	are	correct,	then	we	have	our	work	cut	out	for	us:	we
need	to	look	for	mathematically	rigorous	theories	that	embody	them	and	test
them	experimentally.	We	also	need	to	determine	whether	additional	principles
are	needed.	Regardless	of	whether	IIT	is	correct	or	not,	researchers	should	try	to
develop	competing	theories	and	test	all	available	theories	with	ever	better
experiments.

We’ve	already	discussed	the	perennial	controversy	about	whether	consciousness
research	is	unscientific	nonsense	and	a	pointless	waste	of	time.	In	addition,	there
are	recent	controversies	at	the	cutting	edge	of	consciousness	research—let’s
explore	the	ones	that	I	find	most	enlightening.
Giulio	Tononi’s	IIT	has	lately	drawn	not	merely	praise	but	also	criticism,
some	of	which	has	been	scathing.	Scott	Aaronson	recently	had	this	to	say	on	his
blog:	“In	my	opinion,	the	fact	that	Integrated	Information	Theory	is	wrong—
demonstrably	wrong,	for	reasons	that	go	to	its	core—puts	it	in	something	like	the
top	2%	of	all	mathematical	theories	of	consciousness	ever	proposed.	Almost	all
competing	theories	of	consciousness,	it	seems	to	me,	have	been	so	vague,	fluffy
and	malleable	that	they	can	only	aspire	to	wrongness.”
22
	To	the	credit	of	both
Scott	and	Giulio,	they	never	came	to	blows	when	I	watched	them	debate	IIT	at	a

22
recent	New	York	University	workshop,	and	they	politely	listened	to	each	other’s
arguments.	Aaronson	showed	that	certain	simple	networks	of	logic	gates	had
extremely	high	integrated	information	(
Φ
)	and	argued	that	since	they	clearly
weren’t	conscious,	IIT	was	wrong.	Giulio	countered	that	if	they	were	built,	they
would
	be	conscious,	and	that	Scott’s	assumption	to	the	contrary	was
anthropocentrically	biased,	much	as	if	a	slaughterhouse	owner	claimed	that
animals	couldn’t	be	conscious	just	because	they	couldn’t	talk	and	were	very

Φ
would
different	from	humans.	My	analysis,	with	which	they	both	agreed,	was	that	they
were	at	odds	about	whether	integration	was	merely	a	
necessary
	condition	for
consciousness	(which	Scott	was	OK	with)	or	also	a	
	condition	(which
Giulio	claimed).	The	latter	is	clearly	a	stronger	and	more	contentious	claim,
which	I	hope	we	can	soon	test	experimentally.
23
Another	controversial	IIT	claim	is	that	today’s	computer	architectures	can’t	be
conscious,	because	the	way	their	logic	gates	connect	gives	very	low
integration.
24

necessary
23
24
	In	other	words,	if	you	upload	yourself	into	a	future	high-powered
robot	that	accurately	simulates	every	single	one	of	your	neurons	and	synapses,
then	even	if	this	digital	clone	looks,	talks	and	acts	indistinguishably	from	you,
Giulio	claims	that	it	will	be	an	unconscious	zombie	without	subjective
experience—which	would	be	disappointing	if	you	uploaded	yourself	in	a	quest

for	subjective	immortality.
*6
	This	claim	has	been	challenged	by	both	David
Chalmers	and	AI	professor	Murray	Shanahan	by	imagining	what	would	happen
if	you	instead	gradually	replaced	the	neural	circuits	in	your	brain	by	hypothetical
digital	hardware	perfectly	simulating	them.
25
	Although	your	
behavior
	would	be
unaffected	by	the	replacement	since	the	simulation	is	by	assumption	perfect,
your	
	would	change	from	conscious	initially	to	unconscious	at	the
end,	according	to	Giulio.	But	how	would	it	feel	in	between,	as	ever	more	got
replaced?	When	the	parts	of	your	brain	responsible	for	your	conscious
experience	of	the	upper	half	of	your	visual	field	were	replaced,	would	you	notice
that	part	of	your	visual	scenery	was	suddenly	missing,	but	that	you	mysteriously
knew	what	was	there	nonetheless,	as	reported	by	patients	with	“blindsight”?
26
This	would	be	deeply	troubling,	because	if	you	can	consciously	experience	any
difference,	then	you	can	also	tell	your	friends	about	it	when	asked—yet	by

26
assumption,	your	behavior	can’t	change.	The	only	logical	possibility	compatible
with	the	assumptions	is	that	at	exactly	the	same	instance	that	any	one	thing
disappears	from	your	consciousness,	your	mind	is	mysteriously	altered	so	as
either	to	make	you	lie	and	deny	that	your	experience	changed,	or	to	forget	that
things	had	been	different.
On	the	other	hand,	Murray	Shanahan	admits	that	the	same	gradual-
replacement	critique	can	be	leveled	at	
any
	theory	claiming	that	you	can	act
conscious	without	being	conscious,	so	you	might	be	tempted	to	conclude	that

any
acting	and	being	conscious	are	one	and	the	same,	and	that	externally	observable
behavior	is	therefore	all	that	matters.	But	then	you’d	have	fallen	into	the	trap	of
predicting	that	you’re	unconscious	while	dreaming,	even	though	you	know
better.
A	third	IIT	controversy	is	whether	a	conscious	entity	can	be	made	of	parts	that
are	separately	conscious.	For	example,	can	society	as	a	whole	gain
consciousness	without	the	people	in	it	losing	theirs?	Can	a	conscious	brain	have

better.
parts	that	are	also	conscious	on	their	own?	The	prediction	from	IIT	is	a	firm
“no,”	but	not	everyone	is	convinced.	For	example,	some	patients	with	lesions
severely	reducing	communication	between	the	two	halves	of	their	brain
experience	“alien	hand	syndrome,”	where	their	right	brain	makes	their	left	hand
do	things	that	the	patients	claim	they	aren’t	causing	or	understanding—
sometimes	to	the	point	that	they	use	their	other	hand	to	restrain	their	“alien”
hand.	How	can	we	be	so	sure	that	there	aren’t	two	separate	consciousnesses	in

their	head,	one	in	the	right	hemisphere	that’s	unable	to	speak	and	another	in	the

left	hemisphere	that’s	doing	all	the	talking	and	claiming	to	speak	for	both	of
them?	Imagine	using	future	technology	to	build	a	direct	communication	link
between	two	human	brains,	and	gradually	increasing	the	capacity	of	this	link
until	communication	is	as	efficient	between	the	brains	as	it	is	within	them.
Would	there	come	a	moment	when	the	two	individual	consciousnesses	suddenly
disappear	and	get	replaced	by	a	single	unified	one	as	IIT	predicts,	or	would	the
transition	be	gradual	so	that	the	individual	consciousnesses	coexisted	in	some
form	even	as	a	joint	experience	began	to	emerge?
Another	fascinating	controversy	is	whether	experiments	underestimate	how
much	we’re	conscious	of.	We	saw	earlier	that	although	we	
feel
	we’re	visually
conscious	of	vast	amounts	of	information	involving	colors,	shapes,	objects	and
seemingly	everything	that’s	in	front	of	us,	experiments	have	shown	that	we	can
only	remember	and	report	a	dismally	small	fraction	of	this.
27
	Some	researchers

feel
27
have	tried	to	resolve	this	discrepancy	by	asking	whether	we	may	sometimes	have
“consciousness	without	access,”	that	is,	subjective	experience	of	things	that	are
too	complex	to	fit	into	our	working	memory	for	later	use.
28
	For	example,	when
you	experience	
inattentional	blindness
	by	being	too	distracted	to	notice	an	object
in	plain	sight,	this	doesn’t	imply	that	you	had	no	conscious	visual	experience	of
it,	merely	that	it	wasn’t	stored	in	your	working	memory.
29
	Should	it	count	as
forgetfulness	rather	than	blindness?	Other	researchers	reject	this	idea	that	people

28
29
can’t	be	trusted	about	what	they	say	they	experienced,	and	warn	of	its
implications.	Murray	Shanahan	imagines	a	clinical	trial	where	patients	report
complete	pain	relief	thanks	to	a	new	wonder	drug,	which	nonetheless	gets
rejected	by	a	government	panel:	“The	patients	only	think	they	are	not	in	pain.
Thanks	to	neuroscience,	we	know	better.”
30
	On	the	other	hand,	there	have	been
cases	where	patients	who	accidentally	awoke	during	surgery	were	given	a	drug

30
to	make	them	forget	the	ordeal.	Should	we	trust	their	subsequent	report	that	they
experienced	no	pain?
31

If	some	future	AI	system	is	conscious,	then	what	will	it	subjectively	experience?
This	is	the	essence	of	the	“even	harder	problem”	of	consciousness,	and	forces	us
up	to	the	second	level	of	difficulty	depicted	in	
figure	8.1
.	Not	only	do	we
currently	lack	a	theory	that	answers	this	question,	but	we’re	not	even	sure
whether	it’s	logically	possible	to	fully	answer	it.	After	all,	what	could	a
satisfactory	answer	sound	like?	How	would	you	explain	to	a	person	born	blind
what	the	color	red	looks	like?
Fortunately,	our	current	inability	to	give	a	complete	answer	doesn’t	prevent	us
from	giving	partial	answers.	Intelligent	aliens	studying	the	human	sensory
system	would	probably	infer	that	colors	are	qualia	that	feel	associated	with	each
point	on	a	two-dimensional	surface	(our	visual	field),	while	sounds	don’t	feel	as
spatially	localized,	and	pains	are	qualia	that	feel	associated	with	different	parts
of	our	body.	From	discovering	that	our	retinas	have	three	types	of	light-sensitive

cone	cells,	they	could	infer	that	we	experience	three	primary	colors	and	that	all
other	color	qualia	result	from	combining	them.	By	measuring	how	long	it	takes
neurons	to	transmit	information	across	the	brain,	they	could	conclude	that	we
experience	no	more	than	about	ten	conscious	thoughts	or	perceptions	per	second,
and	that	when	we	watch	movies	on	our	TV	at	twenty-four	frames	per	second,	we
experience	this	not	as	a	sequence	of	still	images,	but	as	continuous	motion.	From

measuring	how	fast	adrenaline	is	released	into	our	bloodstream	and	how	long	it
remains	before	being	broken	down,	they	could	predict	that	we	feel	bursts	of
anger	starting	within	seconds	and	lasting	for	minutes.
Applying	similar	physics-based	arguments,	we	can	make	some	educated
guesses	about	certain	aspects	of	how	an	artificial	consciousness	may	feel.	First
of	all,	the	space	of	possible	AI	experiences	is	
huge
	compared	to	what	we	humans
can	experience.	We	have	one	class	of	qualia	for	each	of	our	senses,	but	AIs	can

huge
have	vastly	more	types	of	sensors	and	internal	representations	of	information,	so
we	must	avoid	the	pitfall	of	assuming	that	being	an	AI	necessarily	feels	similar
to	being	a	person.
Second,	a	brain-sized	artificial	consciousness	could	have	millions	of	times
more	experiences	than	us	per	second,	since	electromagnetic	signals	travel	at	the
speed	of	light—millions	of	times	faster	than	neuron	signals.	However,	the	larger

the	AI,	the	slower	its	global	thoughts	must	be	to	allow	information	time	to	flow
between	all	its	parts,	as	we	saw	in	chapter	4.	We’d	therefore	expect	an	Earth-
sized	“Gaia”	AI	to	have	only	about	ten	conscious	experiences	per	second,	like	a
human,	and	a	galaxy-sized	AI	could	have	only	one	global	thought	every	100,000
years	or	so—so	no	more	than	about	a	hundred	experiences	during	the	entire
history	of	our	Universe	thus	far!	This	would	give	large	AIs	a	seemingly
irresistible	incentive	to	delegate	computations	to	the	smallest	subsystems
capable	of	handling	them,	to	speed	things	up,	much	like	our	conscious	mind	has
delegated	the	blink	reflex	to	a	small,	fast	and	unconscious	subsystem.	Although
we	saw	above	that	the	conscious	information	processing	in	our	brains	appears	to
be	merely	the	tip	of	an	otherwise	unconscious	iceberg,	we	should	expect	the
situation	to	be	even	more	extreme	for	large	future	AIs:	if	they	have	a	single
consciousness,	then	it’s	likely	to	be	unaware	of	almost	all	the	information

processing	taking	place	within	it.	Moreover,	although	the	conscious	experiences
that	it	enjoys	may	be	extremely	complex,	they’re	also	snail-paced	compared	to
the	rapid	activities	of	its	smaller	parts.
This	really	brings	to	a	head	the	aforementioned	controversy	about	whether
parts	of	a	conscious	entity	can	be	conscious	too.	IIT	predicts	not,	which	means
that	if	a	future	astronomically	large	AI	is	conscious,	then	almost	all	its
information	processing	is	unconscious.	This	would	mean	that	if	a	civilization	of

smaller	AIs	improves	its	communication	abilities	to	the	point	that	a	single
conscious	hive	mind	emerges,	their	much	faster	individual	consciousnesses	are
suddenly	extinguished.	If	the	IIT	prediction	is	wrong,	on	the	other	hand,	the	hive
mind	can	coexist	with	the	panoply	of	smaller	conscious	minds.	Indeed,	one
could	even	imagine	a	nested	hierarchy	of	consciousnesses	at	all	levels	from
microscopic	to	cosmic.
As	we	saw	above,	the	unconscious	information	processing	in	our	human

brains	appears	linked	to	the	effortless,	fast	and	automatic	way	of	thinking	that
psychologists	call	“System	1.”
32
	For	example,	your	System	1	might	inform	your
consciousness	that	its	highly	complex	analysis	of	visual	input	data	has
determined	that	your	best	friend	has	arrived,	without	giving	you	any	idea	how
the	computation	took	place.	If	this	link	between	systems	and	consciousness
proves	to	be	valid,	then	it	will	be	tempting	to	generalize	this	terminology	to	AIs,

32
denoting	all	rapid	routine	tasks	delegated	to	unconscious	subunits	as	the	AI’s
System	1.	The	effortful,	slow	and	controlled	global	thinking	of	the	AI	would,	if
conscious,	be	the	AI’s	System	2.	We	humans	also	have	conscious	experiences
involving	what	I’ll	term	“System	0”:	raw	passive	perception	that	takes	place

even	when	you	sit	without	moving	or	thinking	and	merely	observe	the	world
around	you.	Systems	0,	1	and	2	seem	progressively	more	complex,	so	it’s
striking	that	only	the	middle	one	appears	unconscious.	IIT	explains	this	by
saying	that	raw	sensory	information	in	System	0	is	stored	in	grid-like	brain
structures	with	very	high	integration,	while	System	2	has	high	integration
because	of	feedback	loops,	where	all	the	information	you’re	aware	of	right	now
can	affect	your	future	brain	states.	On	the	other	hand,	it	was	precisely	the
conscious-grid	prediction	that	triggered	Scott	Aaronson’s	aforementioned	IIT-
critique.	In	summary,	if	a	theory	solving	the	pretty	hard	problem	of
consciousness	can	one	day	pass	a	rigorous	battery	of	experimental	tests	so	that
we	start	taking	its	predictions	seriously,	then	it	will	also	greatly	narrow	down	the
options	for	the	even	harder	problem	of	what	future	conscious	AIs	may
Some	aspects	of	our	subjective	experience	clearly	trace	back	to	our

evolutionary	origins,	for	example	our	emotional	desires	related	to	self-
preservation	(eating,	drinking,	avoiding	getting	killed)	and	reproduction.	This
means	that	it	should	be	possible	to	create	AI	that	never	experiences	qualia	such
as	hunger,	thirst,	fear	or	sexual	desire.	As	we	saw	in	the	last	chapter,	if	a	highly
intelligent	AI	is	programmed	to	have	virtually	any	sufficiently	ambitious	goal,
it’s	likely	to	strive	for	self-preservation	in	order	to	be	able	to	accomplish	that
goal.	If	they’re	part	of	a	society	of	AIs,	however,	they	might	lack	our	strong

human	fear	of	death:	as	long	as	they’ve	backed	themselves	up,	all	they	stand	to
lose	are	the	memories	they’ve	accumulated	since	their	most	recent	backup,	as
long	as	they’re	confident	that	their	backed-up	software	will	be	used.	In	addition,
the	ability	to	readily	copy	information	and	software	between	AIs	would	probably
reduce	the	strong	sense	of	individuality	that’s	so	characteristic	of	our	human
consciousness:	there	would	be	less	of	a	distinction	between	you	and	me	if	we

could	easily	share	and	copy	all	our	memories	and	abilities,	so	a	group	of	nearby
AIs	may	feel	more	like	a	single	organism	with	a	hive	mind.
Would	an	artificial	consciousness	feel	that	it	had	free	will?	Note	that,	although
philosophers	have	spent	millennia	quibbling	about	whether	
we
	have	free	will
without	reaching	consensus	even	on	how	to	define	the	question,
33
	I’m	asking	a
different	question,	which	is	arguably	easier	to	tackle.	Let	me	try	to	persuade	you
that	the	answer	is	simply	“Yes,	any	conscious	decision	maker	will	subjectively
feel

we
33
feel
	that	it	has	free	will,	regardless	of	whether	it’s	biological	or	artificial.”
Decisions	fall	on	a	spectrum	between	two	extremes:

1.
You	know	exactly	why	you	made	that	particular	choice.
2.
You	have	no	idea	why	you	made	that	particular	choice—it	felt	like	you
chose	randomly	on	a	whim.
Free-will	discussions	usually	center	around	a	struggle	to	reconcile	our	goal-
oriented	decision-making	behavior	with	the	laws	of	physics:	if	you’re	choosing
between	the	following	two	explanations	for	what	you	did,	then	which	one	is
correct:	
“I	asked	her
	
on	a	date	because	I	really	liked	her”
	or	
“My	particles
made	me	do	it	by	moving
	
according	to	the	laws	of	physics”
?	But	we	saw	in	the
last	chapter	that	
both
	are	correct:	what	feels	like	goal-oriented	behavior	can
emerge	from	goal-less	deterministic	laws	of	physics.	More	specifically,	when	a
system	(brain	or	AI)	makes	a	decision	of	type	1,	it	computes	what	to	decide
using	some	deterministic	algorithm,	and	the	reason	it	feels	like	it	decided	is	that
it	in	fact	did	decide	when	computing	what	to	do.	Moreover,	as	emphasized	by
Seth	Lloyd,
34

or	
	
both
34
	there’s	a	famous	computer-science	theorem	saying	that	for	almost
all	computations,	there’s	no	faster	way	of	determining	their	outcome	than
actually	running	them.	This	means	that	it’s	typically	impossible	for	you	to	figure
out	what	you’ll	decide	to	do	in	a	second	in	less	than	a	second,	which	helps
reinforce	your	experience	of	having	free	will.	In	contrast,	when	a	system	(brain
or	AI)	makes	a	decision	of	type	2,	it	simply	programs	its	mind	to	base	its

34
decision	on	the	output	of	some	subsystem	that	acts	as	a	random	number
generator.	In	brains	and	computers,	effectively	random	numbers	are	easily
generated	by	amplifying	noise.	Regardless	of	where	on	the	spectrum	from	1	to	2
a	decision	falls,	both	biological	and	artificial	consciousnesses	therefore	feel	that
they	have	free	will:	they	feel	that	it	is	really	they	who	decide	and	they	can’t
predict	with	certainty	what	the	decision	will	be	until	they’ve	finished	thinking	it
through.

through.
Some	people	tell	me	that	they	find	causality	degrading,	that	it	makes	their
thought	processes	meaningless	and	that	it	renders	them	“mere”	machines.	I	find
such	negativity	absurd	and	unwarranted.	First	of	all,	there’s	nothing	“mere”
about	human	brains,	which,	as	far	as	I’m	concerned,	are	the	most	amazingly
sophisticated	physical	objects	in	our	known	Universe.	Second,	what	alternative
would	they	prefer?	Don’t	they	want	it	to	be	their	own	thought	processes	(the
computations	performed	by	their	brains)	that	make	their	decisions?	Their

subjective	experience	of	free	will	is	simply	how	their	computations	feel	from
inside:	they	don’t	know	the	outcome	of	a	computation	until	they’ve	finished	it.
That’s	what	it	means	to	say	that	the	computation	
is
	the	decision.

Meaning
Let’s	end	by	returning	to	the	starting	point	of	this	book:	How	do	we	want	the
future	of	life	to	be?	We	saw	in	the	previous	chapter	how	diverse	cultures	around
the	globe	all	seek	a	future	teeming	with	positive	experiences,	but	that
fascinatingly	thorny	controversies	arise	when	seeking	consensus	on	what	should
count	as	positive	and	how	to	make	trade-offs	between	what’s	good	for	different
life	forms.	But	let’s	not	let	those	controversies	distract	us	from	the	elephant	in
the	room:	there	can	be	no	positive	experiences	if	there	are	no	experiences	at	all,
that	is,	if	there’s	no	consciousness.	In	other	words,	without	consciousness,	there
can	be	no	happiness,	goodness,	beauty,	meaning	or	purpose—just	an
astronomical	waste	of	space.	This	implies	that	when	people	ask	about	the
meaning	of	life	as	if	it	were	the	job	of	our	cosmos	to	give	meaning	to	our
existence,	they’re	getting	it	backward:	
It’s	not	our	Universe	giving	meaning	to
conscious	beings,	but
	
conscious	beings	giving	meaning	to	our	Universe.
	So	the

	So	the
very	first	goal	on	our	wish	list	for	the	future	should	be	retaining	(and	hopefully
expanding)	biological	and/or	artificial	consciousness	in	our	cosmos,	rather	than
driving	it	extinct.
If	we	succeed	in	this	endeavor,	then	how	will	we	humans	feel	about	coexisting
with	ever	smarter	machines?	Does	the	seemingly	inexorable	rise	of	artificial
intelligence	bother	you	and	if	so,	why?	In	chapter	3,	we	saw	how	it	should	be
relatively	easy	for	AI-powered	technology	to	satisfy	our	basic	needs	such	as

So	the
security	and	income	as	long	as	the	political	will	to	do	so	exists.	However,
perhaps	you’re	concerned	that	being	well	fed,	clad,	housed	and	entertained	isn’t
enough.	If	we’re	guaranteed	that	AI	will	take	care	of	all	our	practical	needs	and
desires,	might	we	nonetheless	end	up	feeling	that	we	lack	meaning	and	purpose
in	our	lives,	like	well-kept	zoo	animals?
Traditionally,	we	humans	have	often	founded	our	self-worth	on	the	idea	of
human
	
exceptionalism:
	the	conviction	that	we’re	the	smartest	entities	on	the

human
	
planet	and	therefore	unique	and	superior.	The	rise	of	AI	will	force	us	to	abandon
this	and	become	more	humble.	But	perhaps	that’s	something	we	should	do
anyway:	after	all,	clinging	to	hubristic	notions	of	superiority	over	others
(individuals,	ethnic	groups,	species	and	so	on)	has	caused	awful	problems	in	the
past,	and	may	be	an	idea	ready	for	retirement.	Indeed,	human	exceptionalism

hasn’t	only	caused	grief	in	the	past,	but	it	also	appears	unnecessary	for	human
flourishing:	if	we	discover	a	peaceful	extraterrestrial	civilization	far	more
advanced	than	us	in	science,	art	and	everything	else	we	care	about,	this
presumably	wouldn’t	prevent	people	from	continuing	to	experience	meaning	and
purpose	in	their	lives.	We	could	retain	our	families,	friends	and	broader
communities,	and	all	activities	that	give	us	meaning	and	purpose,	hopefully
having	lost	nothing	but	arrogance.
As	we	plan	our	future,	let’s	consider	the	meaning	not	only	of	our	own	lives,
but	also	of	our	Universe	itself.	Here	two	of	my	favorite	physicists,	Steven
Weinberg	and	Freeman	Dyson,	represent	diametrically	opposite	views.
Weinberg,	who	won	the	Nobel	Prize	for	foundational	work	on	the	standard
model	of	particle	physics,	famously	said,	“The	more	the	universe	seems
comprehensible,	the	more	it	also	seems	pointless.”
35
	Dyson,	on	the	other	hand,	is
much	more	optimistic,	as	we	saw	in	chapter	6:	although	he	agrees	that	our
Universe

35
Universe	
was
	pointless,	he	believes	that	life	is	now	filling	it	with	ever	more
meaning,	with	the	best	yet	to	come	if	life	succeeds	in	spreading	throughout	the
cosmos.	He	ended	his	seminal	1979	paper	thus:	“Is	Weinberg’s	universe	or	mine
closer	to	the	truth?	One	day,	before	long,	we	shall	know.”
36
	If	our	Universe	goes
back	to	being	permanently	unconscious	because	we	drive	Earth	life	extinct	or
because	we	let	unconscious	zombie	AI	take	over	our	Universe,	then	Weinberg
will	be	vindicated	in	spades.

Universe	
was
36
From	this	perspective,	we	see	that	although	we’ve	focused	on	the	future	of
intelligence	in	this	book,	the	future	of	consciousness	is	even	more	important,
since	that’s	what	enables	meaning.	Philosophers	like	to	go	Latin	on	this
distinction,	by	contrasting	
sapience
	(the	ability	to	think	intelligently)	with
sentience
	(the	ability	to	subjectively	experience	qualia).	We	humans	have	built
our	identity	on	being	
Homo	sapiens,
	the	smartest	entities	around.	As	we	prepare

sapience
sentience
to	be	humbled	by	ever	smarter	machines,	I	suggest	that	we	rebrand	ourselves	as
Homo
	
sentiens
!

•
There’s	no	undisputed	definition	of	“consciousness.”	I	use	the	broad	and	non-
anthropocentric	definition	
consciousness	=	subjective	experience.
•
Whether	AIs	are	conscious	in	that	sense	is	what	matters	for	the	thorniest	ethical	and
philosophical	problems	posed	by	the	rise	of	AI:	Can	AIs	suffer?	Should	they	have
rights?	Is	uploading	a	subjective	suicide?	Could	a	future	cosmos	teeming	with	AIs
be	the	ultimate	zombie	apocalypse?
•
The	problem	of	understanding	intelligence	shouldn’t	be	conflated	with	three	separate
problems	of	consciousness:	the	“pretty	hard	problem”	of	predicting	which	physical
systems	are	conscious,	the	“even	harder	problem”	of	predicting	qualia,	and	the
“
really
	hard	problem”	of	why	anything	at	all	is	conscious.
•
The	“pretty	hard	problem”	of	consciousness	is	scientific,	since	a	theory	that	predicts
which	of	your	brain	processes	are	conscious	is	experimentally	testable	and
falsifiable,	while	it’s	currently	unclear	how	science	could	fully	resolve	the	two
harder	problems.
•

“
really
•
•
Neuroscience	experiments	suggest	that	many	behaviors	and	brain	regions	are
unconscious,	with	much	of	our	conscious	experience	representing	an	after-the-fact
summary	of	vastly	larger	amounts	of	unconscious	information.
•
Generalizing	consciousness	predictions	from	brains	to	machines	requires	a	theory.
Consciousness	appears	to	require	not	a	particular	kind	of	particle	or	field,	but	a
particular	kind	of	information	processing	that’s	fairly	autonomous	and	integrated,	so
that	the	whole	system	is	rather	autonomous	but	its	parts	aren’t.

•
•
Consciousness	might	feel	so	non-physical	because	it’s	doubly	substrate-independent:
if	consciousness	is	the	way	information	feels	when	being	processed	in	certain
complex	ways,	then	it’s	merely	the	structure	of	the	information	processing	that
matters,	not	the	structure	of	the	matter	doing	the	information	processing.
•
If	artificial	consciousness	is	possible,	then	the	space	of	possible	AI	experiences	is
likely	to	be	huge	compared	to	what	we	humans	can	experience,	spanning	a	vast
spectrum	of	qualia	and	timescales—all	sharing	a	feeling	of	having	free	will.
•

•
•
Since	there	can	be	no	meaning	without	consciousness,	it’s	not	our	Universe	giving
meaning	to	conscious	beings,	but	conscious	beings	giving	meaning	to	our	Universe.
•
This	suggests	that	as	we	humans	prepare	to	be	humbled	by	ever	smarter	machines,
we	take	comfort	mainly	in	being	
Homo	sentiens,
	not	
Homo	sapiens.
*1
	
An	alternative	viewpoint	is	
substance	dualism
—that	living	entities	differ	from	inanimate	ones	because

they	contain	some	non-physical	substance	such	as	an	“anima,”	“élan	vital”	or	“soul.”	Support	for
substance	dualism	among	scientists	has	gradually	dwindled.	To	understand	why,	consider	that	your	body
is	made	of	about	10
29
	quarks	and	electrons,	which,	as	far	as	we	can	tell,	move	according	to	simple
physical	laws.	Imagine	a	future	technology	able	to	track	all	your	particles:	if	they	were	found	to	obey	the
laws	of	physics	exactly,	then	your	purported	soul	is	having	no	effect	on	your	particles,	so	your	conscious
mind	and	its	ability	to	control	your	movements	would	have	nothing	to	do	with	a	soul.	If	your	particles
were	instead	found	not	to	obey	the	known	laws	of	physics	because	they	were	being	pushed	around	by
your	soul,	then	the	new	entity	causing	these	forces	would	by	definition	be	a	physical	one	that	we	can
study	just	like	we’ve	studied	new	fields	and	new	particles	in	the	past.
*2
	
I	use	the	word	“qualia”	according	to	the	dictionary	definition,	to	mean	individual	instances	of	subjective

*2
	
experience—that	is,	to	mean	the	subjective	experience	itself,	not	any	purported	substance	causing	the
experience.	Beware	that	some	people	use	the	word	differently.
*3
	
I’d	originally	called	the	RHP	the	“very	hard	problem,”	but	after	I	showed	this	chapter	to	David
Chalmers,	he	emailed	me	the	clever	suggestion	of	switching	to	the	
“really	hard	problem,”
	to	match	what
he	really	meant:	“since	the	first	two	problems	(at	least	put	this	way)	aren’t	really	part	of	the	hard	problem

*3
	
as	I	conceived	of	it	whereas	the	third	problem	is,	you	could	perhaps	use	‘really	hard’	instead	of	‘very
hard’	for	the	third	one	to	match	my	usage.”
*4
	
If	our	physical	reality	is	entirely	mathematical	(information-based,	loosely	speaking),	as	I	explored	in
my	book	
Our	Mathematical	Universe,
	then	no	aspect	of	reality—not	even	consciousness—lies	beyond
the	purview	of	science.	Indeed,	the	really	hard	problem	of	consciousness	is,	from	that	perspective,	the

*4
	
my	book	
exact	same	problem	as	that	of	understanding	how	something	mathematical	can	feel	physical:	if	part	of	a
mathematical	structure	is	conscious,	then	it	will	experience	the	rest	as	its	external	physical	world.
*5
	
Although	I’ve	earlier	used	“perceptronium”	as	a	synonym	for	sentronium,	that	name	suggests	too	narrow
a	definition,	since	percepts	are	merely	those	subjective	experiences	that	we	perceive	based	on	sensory
input—excluding,	for	example,	dreams	and	internally	generated	thoughts.
*6

*5
	
*6
	
There’s	potential	tension	between	this	claim	and	the	idea	that	consciousness	is	substrate-independent,
since	even	though	the	information	processing	may	be	different	at	the	lowest	level,	it’s	by	definition
identical	at	the	higher	levels	where	it	determines	behavior.

Epilogue
The	Tale	of	the	
FLI
	Team
The	saddest	aspect	of	life	right	now	is	that	science	gathers	knowledge	faster	than
	
society
gathers	wisdom.
Isaac	Asimov
Here	we	are,	my	dear	reader,	at	the	end	of	the	book,	after	exploring	the	origin
and	fate	of	intelligence,	goals	and	meaning.	So	how	can	we	translate	these	ideas
into	action?	What	concretely	should	we	
do
	to	make	our	future	as	good	as
possible?	This	is	precisely	the	question	I’m	asking	myself	right	now	as	I	sit	here
in	my	window	seat	en	route	from	San	Francisco	back	to	Boston	on	January	9,
2017,	from	the	AI	conference	we	just	organized	in	Asilomar,	so	let	me	end	this
book	by	sharing	my	thoughts	with	you.
Meia	is	catching	up	on	sleep	next	to	me	after	the	many	short	nights	of
preparing	and	organizing.	Wow—what	a	wild	week	it’s	been!	We	managed	to
bring	almost	all	the	people	I’ve	mentioned	in	this	book	together	for	a	few	days	to
this	Puerto	Rico	sequel,	including	entrepreneurs	such	as	Elon	Musk	and	Larry

Page	and	AI	research	leaders	from	academia	and	companies	such	as	DeepMind,
Google,	Facebook,	Apple,	IBM,	Microsoft	and	Baidu,	as	well	as	economists,
legal	scholars,	philosophers	and	other	amazing	thinkers	(see	
figure	9.1
).	The
results	superseded	even	my	high	expectations,	and	I’m	feeling	more	optimistic
about	the	future	of	life	than	I	have	in	a	long	time.	In	this	epilogue,	I’m	going	to
tell	you	why.

FLI	Is	Born
Ever	since	I	learned	about	the	nuclear	arms	race	at	age	fourteen,	I’ve	been
concerned	that	the	power	of	our	technology	was	growing	faster	than	the	wisdom
with	which	we	manage	it.	I	therefore	decided	to	sneak	a	chapter	about	this
challenge	into	my	first	book,	
	even	though	the	rest
of	it	was	primarily	about	physics.	I	made	a	New	Year’s	resolution	for	2014	that	I
was	no	longer	allowed	to	complain	about	anything	without	putting	some	serious
thought	into	what	I	could	personally	do	about	it,	and	I	kept	my	pledge	during	my
book	tour	that	January:	Meia	and	I	did	lots	of	brainstorming	about	starting	some
sort	of	nonprofit	organization	focused	on	improving	the	future	of	life	through
technological	stewardship.
She	insisted	that	we	give	it	a	positive	name	as	different	as	possible	from
“Doom	&	Gloom	Institute”	and	“Let’s-Worry-about-the-Future	Institute.”	Since
Future	of	Humanity	Institute	was	already	taken,	we	converged	on	the	Future	of

Life	Institute	(FLI),	which	had	the	added	advantage	of	being	more	inclusive.	On
January	22,	the	book	tour	took	us	to	Santa	Cruz,	and	as	the	California	Sun	set
over	the	Pacific,	we	enjoyed	dinner	with	our	old	friend	Anthony	Aguirre	and
persuaded	him	to	join	forces	with	us.	He’s	not	only	one	of	the	wisest	and	most
idealistic	people	I	know,	but	also	someone	who’s	managed	to	put	up	with
running	another	nonprofit	organization,	the	
Foundational	Questions	Institute
(see	
http://fqxi.org
),	with	me	for	over	a	decade.

(see	
The	following	week,	the	tour	took	me	to	London.	Since	the	future	of	AI	was
very	much	on	my	mind,	I	reached	out	to	Demis	Hassabis,	who	graciously	invited
me	to	visit	DeepMind’s	headquarters.	I	was	awestruck	by	how	much	they’d
grown	since	he	visited	me	at	MIT	two	years	earlier.	Google	had	just	bought	them
for	about	$650	million,	and	seeing	their	vast	office	landscape	filled	with	brilliant
minds	pursuing	Demis’	audacious	goal	to	“solve	intelligence”	gave	me	a	visceral
feeling	that	success	was	a	real	possibility.
The	next	evening,	I	spoke	with	my	friend	Jaan	Tallinn	using	Skype,	the

software	he’d	helped	create.	I	explained	our	FLI	vision,	and	an	hour	later,	he’d
decided	to	take	a	chance	on	us,	funding	us	at	up	to	$100,000	a	year!	Few	things
touch	me	more	than	when	someone	places	more	trust	in	me	than	I’ve	earned,	so
it	meant	the	world	to	me	when	a	year	later,	after	the	Puerto	Rico	conference	I

mentioned	in	chapter	1,	he	joked	that	this	was	the	best	investment	he’d	ever
made.
Figure	9.1:	Our	January	2017	Asilomar	conference,	the	Puerto	Rico	sequel,	brought	together	a
remarkable	group	of	researchers	in	AI	and	related	fields.	Back	row,	from	left	to	right:	Patrick
Lin,	Daniel	Weld,	Ariel	Conn,	Nancy	Chang,	Tom	Mitchell,	Ray	Kurzweil,	Daniel	Dewey,
Margaret	Boden,	Peter	Norvig,	Nick	Hay,	Moshe	Vardi,	Scott	Siskind,	Nick	Bostrom,	Francesca
Rossi,	Shane	Legg,	Manuela	Veloso,	David	Marble,	Katja	Grace,	Irakli	Beridze,	Marty
Tenenbaum,	Gill	Pratt,	Martin	Rees,	Joshua	Greene,	Matt	Scherer,	Angela	Kane,	Amara
Angelica,	Jeff	Mohr,	Mustafa	Suleyman,	Steve	Omohundro,	Kate	Crawford,	Vitalik	Buterin,
Yutaka	Matsuo,	Stefano	Ermon,	Michael	Wellman,	Bas	Steunebrink,	Wendell	Wallach,	Allan
Dafoe,	Toby	Ord,	Thomas	Dietterich,	Daniel	Kahneman,	Dario	Amodei,	Eric	Drexler,	Tomaso
Poggio,	Eric	Schmidt,	Pedro	Ortega,	David	Leake,	Seán	Ó	hÉigeartaigh,	Owain	Evans,	Jaan

Tallinn,	Anca	Dragan,	Sean	Legassick,	Toby	Walsh,	Peter	Asaro,	Kay	Firth-Butterfield,	Philip
Sabes,	Paul	Merolla,	Bart	Selman,	Tucker	Davey,	?,	Jacob	Steinhardt,	Moshe	Looks,	Josh
Tenenbaum,	Tom	Gruber,	Andrew	Ng,	Kareem	Ayoub,	Craig	Calhoun,	Percy	Liang,	Helen
Toner,	David	Chalmers,	Richard	Sutton,	Claudia	Passos-Ferriera,	János	Krámar,	William
MacAskill,	Eliezer	Yudkowsky,	Brian	Ziebart,	Huw	Price,	Carl	Shulman,	Neil	Lawrence,
Richard	Mallah,	Jurgen	Schmidhuber,	Dileep	George,	Jonathan	Rothberg,	Noah	Rothberg.	Front

row:	Anthony	Aguirre,	Sonia	Sachs,	Lucas	Perry,	Jeffrey	Sachs,	Vincent	Conitzer,	Steve	Goose,
Victoria	Krakovna,	Owen	Cotton-Barratt,	Daniela	Rus,	Dylan	Hadfield-Menell,	Verity	Harding,
Shivon	Zilis,	Laurent	Orseau,	Ramana	Kumar,	Nate	Soares,	Andrew	McAfee,	Jack	Clark,	Anna
Salamon,	Long	Ouyang,	Andrew	Critch,	Paul	Christiano,	Yoshua	Bengio,	David	Sanford,
Catherine	Olsson,	Jessica	Taylor,	Martina	Kunz,	Kristinn	Thorisson,	Stuart	Armstrong,	Yann
LeCun,	Alexander	Tamas,	Roman	Yampolskiy,	Marin	Solja
č
i
ć
,	Lawrence	Krauss,	Stuart	Russell,

č
i
ć
Eric	Brynjolfsson,	Ryan	Calo,	ShaoLan	Hsueh,	Meia	Chita-Tegmark,	Kent	Walker,	Heather	Roff,
Meredith	Whittaker,	Max	Tegmark,	Adrian	Weller,	Jose	Hernandez-Orallo,	Andrew	Maynard,
John	Hering,	Abram	Demski,	Nicolas	Berggruen,	Gregory	Bonnet,	Sam	Harris,	Tim	Hwang,
Andrew	Snyder-Beattie,	Marta	Halina,	Sebastian	Farquhar,	Stephen	Cave,	Jan	Leike,	Tasha
McCauley,	Joseph	Gordon-Levitt.	Arrived	later:	Guru	Banavar,	Demis	Hassabis,	Rao
Kambhampati,	Elon	Musk,	Larry	Page,	Anthony	Romero.
The	next	day,	my	publisher	had	left	a	gap	in	my	schedule,	which	I	filled	with	a

visit	to	the	London	Science	Museum.	After	having	obsessed	about	the	past	and
future	of	intelligence	for	so	long,	I	suddenly	felt	that	I	was	walking	through	a
physical	manifestation	of	my	thoughts.	They’d	assembled	a	fantastic	collection
of	stuff	representing	our	growth	of	knowledge,	from	Stephenson’s	Rocket

locomotive	to	the	Model	T	Ford,	a	life-size	Apollo	11	lunar	lander	replica	and
computers	dating	all	the	way	from	Babbage’s	“Difference	Engine”	mechanical
calculator	to	present-day	hardware.	They	also	had	an	exhibit	about	the	history	of
our	understanding	of	the	mind,	from	Galvano’s	frog-leg	experiments	to	neurons,
EEG	and	fMRI.
I	very	rarely	cry,	but	that’s	what	I	did	on	the	way	out—and	in	a	tunnel	full	of
pedestrians,	no	less,	en	route	to	the	South	Kensington	tube	station.	Here	were	all
these	people	going	about	their	lives	blissfully	unaware	of	what	I	was	thinking.
First	we	humans	discovered	how	to	replicate	some	natural	processes	with
machines,	making	our	own	wind	and	lightning,	and	our	own	mechanical
horsepower.	Gradually,	we	started	realizing	that	our	bodies	were	also	machines.
Then	the	discovery	of	nerve	cells	started	blurring	the	borderline	between	body
and	mind.	Then	we	started	building	machines	that	could	outperform	not	only	our

muscles,	but	our	minds	as	well.	So	in	parallel	with	discovering	what	we	are,	are
we	inevitably	making	ourselves	obsolete?	That	would	be	poetically	tragic.
This	thought	scared	me,	but	it	also	strengthened	my	resolve	to	keep	my	New
Year’s	resolution.	I	felt	that	we	needed	one	more	person	to	complete	our	team	of
FLI	founders,	who’d	spearhead	a	team	of	idealistic	young	volunteers.	The
logical	choice	was	Viktoriya	Krakovna,	a	brilliant	Harvard	grad	student	who’d
not	only	won	a	silver	medal	in	the	International	Mathematics	Olympiad,	but	also

founded	the	Citadel,	a	house	for	about	a	dozen	young	idealists	who	wanted
reason	to	play	a	greater	role	in	their	lives	and	the	world.	Meia	and	I	invited	her
over	to	our	place	five	days	later	to	tell	her	about	our	vision,	and	before	we’d
finished	the	sushi,	FLI	had	been	born.

The	Puerto	Rico	Adventure
This	marked	the	beginning	of	an	amazing	adventure,	which	still	continues.	As	I
mentioned	in	chapter	1,	we	held	regular	brainstorming	meetings	at	our	house
with	dozens	of	idealistic	students,	professors	and	other	local	thinkers,	where	the
top-rated	ideas	transformed	into	projects—the	first	being	that	AI	op-ed	from
chapter	1	with	Stephen	Hawking,	Stuart	Russell	and	Frank	Wilczek	that	helped
ignite	the	public	debate.	In	parallel	with	the	baby	steps	of	setting	up	a	new
organization	(such	as	incorporating,	recruiting	an	advisory	board	and	launching	a
website),	we	held	a	fun	launch	event	in	front	of	a	packed	MIT	auditorium,	at
which	Alan	Alda	explored	the	future	of	technology	with	leading	experts.

Figure	9.2:	Jaan	Tallinn,	Anthony	Aguirre,	yours	truly,	Meia	Chita-Tegmark	and	Viktoriya
Krakovna	celebrate	our	incorporation	of	FLI	with	sushi	on	May	23,	2014.
We	focused	the	rest	of	the	year	on	pulling	together	the	Puerto	Rico	conference
which,	as	I	mentioned	in	chapter	1,	aimed	to	engage	the	world’s	leading	AI
researchers	in	the	discussion	of	how	to	keep	AI	beneficial.	Our	goal	was	to	shift
the	AI-safety	conversation	from	worrying	to	working:	from	bickering	about	how
worried	to	be,	to	agreeing	on	concrete	research	projects	that	could	be	started
right	away	to	maximize	the	chance	of	a	good	outcome.	To	prepare,	we	collected
promising	AI-safety	research	ideas	from	around	the	world	and	sought
community	feedback	on	our	growing	project	list.	With	the	help	of	Stuart	Russell
and	a	group	of	hardworking	young	volunteers,	especially	Daniel	Dewey,	János
Krámar	and	Richard	Mallah,	we	distilled	these	research	priorities	into	a
document	to	be	discussed	at	the	conference.
1
	Building	consensus	that	there	was

1
lots	of	valuable	AI-safety	research	to	be	done	would,	we	hoped,	encourage
people	to	start	doing	such	research.	The	ultimate	moonshot	triumph	would	be	if
it	could	even	persuade	someone	to	fund	it	since,	so	far,	there	had	been	essentially
no	support	for	such	work	from	government	funding	agencies.
Enter	Elon	Musk.	On	August	2,	he	appeared	on	our	radar	by	famously
tweeting	“Worth	reading	Superintelligence	by	Bostrom.	We	need	to	be	super
careful	with	AI.	Potentially	more	dangerous	than	nukes.”	I	reached	out	to	him

about	our	efforts,	and	got	to	speak	with	him	by	phone	a	few	weeks	later.
Although	I	felt	quite	nervous	and	starstruck,	the	outcome	was	outstanding:	he
agreed	to	join	our	FLI	scientific	advisory	board,	to	attend	our	conference	and
potentially	to	fund	a	first-ever	AI-safety	research	program	to	be	announced	in
Puerto	Rico.	This	electrified	all	of	us	at	FLI,	and	made	us	redouble	our	efforts	to
create	an	awesome	conference,	identify	promising	research	topics	and	build
community	support	for	them.
I	finally	got	to	meet	Elon	in	person	for	further	planning	when	he	came	to	MIT
two	months	later	for	a	space	symposium.	It	felt	very	strange	to	be	alone	with	him
in	a	small	green	room	just	moments	after	he’d	enraptured	over	a	thousand	MIT
students	like	a	rock	star,	but	after	a	few	minutes,	all	I	could	think	of	was	our
joint	project.	I	instantly	liked	him.	He	radiated	sincerity,	and	I	was	inspired	by
how	much	he	cared	about	the	long-term	future	of	humanity—and	how	he

audaciously	turned	his	aspiration	into	actions.	He	wanted	humanity	to	explore
and	settle	our	Universe,	so	he	started	a	space	company.	He	wanted	sustainable
energy,	so	he	started	a	solar	company	and	an	electric-car	company.	Tall,
handsome,	eloquent	and	incredibly	knowledgeable,	it	was	easy	to	understand
why	people	listened	to	him.
Unfortunately,	this	MIT	event	also	taught	me	how	fear-driven	and	divisive
media	can	be.	Elon’s	stage	performance	consisted	of	an	hour	of	fascinating

discussion	about	space	exploration,	which	I	think	would	have	made	great	TV.	At
the	very	end,	a	student	asked	him	an	off-topic	question	about	AI.	His	answer
included	the	phrase	“with	artificial	intelligence,	we	are	summoning	the	demon,”
which	became	the	
only
	thing	that	most	media	reported—and	generally	out	of
context.	It	struck	me	that	many	journalists	were	inadvertently	doing	the	
exact
opposite
	of	what	we	were	trying	to	accomplish	in	Puerto	Rico.	Whereas	we
wanted	to	build	community	consensus	by	highlighting	the	common	ground,	the

only
exact
opposite
media	had	an	incentive	to	highlight	the	divisions.	The	more	controversy	they
could	report,	the	greater	their	Nielsen	ratings	and	ad	revenue.	Moreover,	whereas
we	wanted	to	help	people	from	across	the	spectrum	of	opinions	to	come	together,
get	along	and	understand	each	other	better,	media	coverage	inadvertently	made
people	across	the	opinion	spectrum	upset	at	one	another,	fueling
misunderstandings	by	publishing	only	their	most	provocative-sounding	quotes
without	context.	For	this	reason,	we	decided	to	ban	journalists	from	the	Puerto

Rico	meeting	and	impose	the	“Chatham	House	Rule,”	which	prohibits
participants	from	subsequently	revealing	who	said	what.
*

Although	our	Puerto	Rico	conference	ended	up	being	a	success,	it	didn’t	come
easy.	The	countdown	mostly	required	diligent	prep	work,	for	example	me
phoning	or	skyping	large	numbers	of	AI	researchers	to	assemble	a	critical	mass
of	participants	to	attract	the	other	attendees,	and	there	were	also	dramatic
moments—such	as	when	I	got	up	by	7	a.m.	on	December	27	to	reach	Elon	on	a
lousy	phone	connection	to	Uruguay,	and	was	told	“I	don’t	think	this	is	gonna
work.”	He	was	concerned	that	an	AI-safety	research	program	might	provide	a
false	sense	of	security,	enabling	reckless	researchers	to	forge	ahead	while	paying
lip	service	to	safety.	But	then,	despite	the	sound	incessantly	cutting	out,	we
extensively	talked	through	the	huge	benefits	of	mainstreaming	the	topic	and
getting	more	AI	researchers	working	on	AI	safety.	After	the	call	dropped,	he	sent
me	one	of	my	favorite	emails	ever:	“Lost	the	call	at	the	end	there.	Anyway,	docs
look	fine.	I’m	happy	to	support	the	research	with	$5M	over	three	years.	Maybe

we	should	make	it	$10M?”
Four	days	later,	2015	got	off	to	a	good	start	for	Meia	and	me	as	we	briefly
relaxed	before	the	meeting,	dancing	in	the	new	year	on	a	Puerto	Rico	beach
illuminated	by	fireworks.	The	conference	got	off	to	a	great	start	too:	there	was
remarkable	consensus	that	more	AI-safety	research	was	needed,	and	based	on
further	input	from	the	conference	participants,	that	research	priorities	document
we’d	worked	so	hard	on	was	improved	and	finalized.	We	passed	around	that

safety-research-endorsing	open	letter	from	chapter	1,	and	were	delighted	that
almost	everyone	signed	it.
Meia	and	I	had	a	magical	meeting	with	Elon	in	our	hotel	room	where	he
blessed	the	detailed	plans	for	our	grants	program.	She	was	touched	by	how
down-to-earth	and	candid	he	was	about	his	personal	life,	and	how	much	interest
he	took	in	us.	He	asked	us	how	we	met,	and	liked	Meia’s	elaborate	story.	The
next	day,	we	filmed	an	interview	with	him	about	AI	safety	and	why	he	wanted	to
support	it	and	everything	seemed	on	track.
2

2
The	conference	climax,	Elon’s	donation	announcement,	was	scheduled	for	7
p.m.	on	Sunday,	January	4,	2015,	and	I’d	been	so	tense	about	it	that	I’d	tossed
and	turned	in	my	sleep	the	night	before.	And	then,	just	fifteen	minutes	before	we
were	supposed	to	head	to	the	session	where	it	would	happen,	we	hit	a	snag!
Elon’s	assistant	called	and	said	that	it	looked	like	Elon	might	not	be	able	to	go
through	with	the	announcement,	and	Meia	said	she’d	never	seen	me	look	more
stressed	or	disappointed.	Elon	finally	came	by,	and	I	could	hear	the	seconds

counting	down	to	the	session	start	as	we	sat	there	and	talked.	He	explained	that

they	were	just	two	days	away	from	a	crucial	SpaceX	rocket	launch	where	they
hoped	to	pull	off	the	first-ever	successful	landing	of	the	first	stage	on	a	drone
ship,	and	that	since	this	was	a	huge	milestone,	the	SpaceX	team	didn’t	want	to
distract	from	it	with	concurrent	media	splashes	involving	him.	Anthony	Aguirre,
cool	and	levelheaded	as	always,	pointed	out	that	this	meant	that	
nobody
	wanted
media	attention	for	this,	neither	Elon	nor	the	AI	community.	We	arrived	a	few
minutes	late	to	the	session	I	was	moderating,	but	we	had	a	plan:	no	dollar
amount	would	get	mentioned,	to	ensure	that	the	announcement	wasn’t
newsworthy,	and	I’d	lord	Chatham	House	over	everyone	to	keep	Elon’s
announcement	secret	from	the	world	for	nine	days	if	his	rocket	reached	the	space
station,	regardless	of	whether	the	landing	succeeded;	he	said	he’d	need	even
more	time	if	the	rocket	exploded	on	launch.
The	countdown	to	the	announcement	finally	reached	zero.	The

superintelligence	panelists	that	I’d	moderated	still	sat	there	next	to	me	onstage	in
their	chairs:	Eliezer	Yudkowsky,	Elon	Musk,	Nick	Bostrom,	Richard	Mallah,
Murray	Shanahan,	Bart	Selman,	Shane	Legg	and	Vernor	Vinge.	People	gradually
stopped	applauding,	but	the	panelists	remained	seated,	because	I’d	told	them	to
stay	without	explaining	why.	Meia	later	told	me	that	her	pulse	reached	the
stratosphere	around	now,	and	that	she	clutched	Viktoriya	Krakovna’s	calming
hand	under	the	table.	I	smiled,	knowing	that	this	was	the	moment	we’d	worked,

hoped	and	waited	for.
I	was	very	happy	that	there	was	such	consensus	at	the	meeting	that	more
research	was	needed	for	keeping	AI	beneficial,	I	said,	and	that	there	were	so
many	concrete	research	directions	we	could	work	on	right	away.	But	there	had
been	talk	of	serious	risks	in	this	session,	I	added,	so	it	would	be	nice	to	raise	our
spirits	and	get	into	an	upbeat	mood	before	heading	out	to	the	bar	and	the
conference	banquet	that	had	been	set	up	outside.	“And	I’m	therefore	giving	the
microphone	to…Elon	Musk!”	I	felt	that	history	was	in	the	making	as	Elon	took

the	mic	and	announced	that	he	would	donate	a	large	amount	of	money	to	AI-
safety	research.	Unsurprisingly,	he	brought	down	the	house.	As	planned,	he
didn’t	mention	how	much,	but	I	knew	that	it	was	a	cool	$10	million,	as	we’d
agreed.
Meia	and	I	went	to	visit	our	parents	in	Sweden	and	Romania	after	the
conference,	and	with	bated	breath,	we	watched	the	live-streamed	rocket	launch
with	my	dad	in	Stockholm.	The	landing	attempt	unfortunately	ended	with	what
Elon	euphemistically	calls	an	RUD,	“rapid	unscheduled	disassembly,”	and

pulling	off	a	successful	ocean	landing	took	his	team	another	fifteen	months.
3
However,	all	the	satellites	were	successfully	launched	into	orbit,	as	was	our
grants	program	via	a	tweet	by	Elon	to	his	millions	of	followers.
4

Mainstreaming	AI	Safety
A	key	goal	of	the	Puerto	Rico	conference	had	been	to	mainstream	AI-safety
research,	and	it	was	exhilarating	to	see	this	unfold	in	multiple	steps.	First	there
was	the	meeting	itself,	where	many	researchers	started	feeling	comfortable
engaging	with	the	topic	once	they	realized	that	they	were	part	of	a	growing
community	of	peers.	I	was	deeply	touched	by	encouragement	from	many
participants.	For	example,	Cornell	University	AI	professor	Bart	Selman	emailed
me	saying,	“I’ve	honestly	never	seen	a	better	organized	or	more	exciting	and
intellectually	stimulating	scientific	meeting.”
The	next	mainstreaming	step	began	on	January	11	when	Elon	tweeted
“World’s	top	artificial	intelligence	developers	sign	open	letter	calling	for	AI-
safety	research,”
5
	linking	to	a	sign-up	page	that	soon	racked	up	over	eight
thousand	signatures,	including	many	of	the	world’s	most	prominent	AI	builders.
It	suddenly	became	harder	to	claim	that	people	concerned	about	AI	safety	didn’t

5
know	what	they	were	talking	about,	because	this	now	implied	that	a	who’s	who
of	leading	AI	researchers	didn’t	know	what	they	were	talking	about.	The	open
letter	was	reported	by	media	around	the	world	in	a	way	that	made	us	grateful
that	we’d	barred	journalists	from	our	conference.	Although	the	most	alarmist
word	in	the	letter	was	“pitfalls,”	it	nonetheless	triggered	headlines	such	as	“Elon
Musk	and	Stephen	Hawking	Sign	Open	Letter	in	Hopes	of	Preventing	Robot

Uprising,”	illustrated	by	murderous	terminators.	Of	the	hundreds	of	articles	we
spotted,	our	favorite	was	one	mocking	the	others,	writing	that	“a	headline	that
conjures	visions	of	skeletal	androids	stomping	human	skulls	underfoot	turns
complex,	transformative	technology	into	a	carnival	sideshow.”
6
	Fortunately,
there	were	also	many	sober	news	articles,	and	they	gave	us	another	challenge:
keeping	up	with	the	torrent	of	new	signatures,	which	needed	to	be	manually
verified	to	protect	our	credibility	and	weed	out	pranks	such	as	“HAL	9000,”

6
“Terminator,”	“Sarah	Jeanette	Connor”	and	“Skynet.”	For	this	and	our	future
open	letters,	Viktoriya	Krakovna	and	János	Krámar	helped	organize	a	volunteer
brigade	of	checkers	that	included	Jesse	Galef,	Eric	Gastfriend	and	Revathi
Vinoth	Kumar	working	in	shifts,	so	that	when	Revathi	went	to	sleep	in	India,	she
passed	the	baton	to	Eric	in	Boston,	and	so	on.
The	third	mainstreaming	step	began	four	days	later,	when	Elon	tweeted	a	link

to	our	announcement	that	he	was	donating	$10	million	to	AI-safety	research.
7
	A
week	later,	we	launched	an	online	portal	where	researchers	from	around	the
world	could	apply	and	compete	for	this	funding.	We	were	able	to	whip	the
application	system	together	so	quickly	only	because	Anthony	and	I	had	spent	the
previous	decade	running	similar	competitions	for	physics	grants.	The	Open
Philanthropy	Project,	a	California-based	charity	focused	on	high-impact	giving,
generously	agreed	to	top	up	Elon’s	gift	to	allow	us	to	give	more	grants.	We
weren’t	sure	how	many	applicants	we’d	get,	since	the	topic	was	novel	and	the
deadline	was	short.	The	response	blew	us	away,	with	about	three	hundred	teams
from	around	the	world	asking	for	about	$100	million.	A	panel	of	AI	professors
and	other	researchers	carefully	reviewed	the	proposals	and	selected	thirty-seven
winning	teams,	who	were	funded	for	up	to	three	years.	When	we	announced	the
list	of	winners,	it	marked	the	first	time	that	the	media	response	to	our	activities

was	fairly	nuanced	and	free	of	killer-robot	pictures.	It	was	finally	sinking	in	that
AI	safety	wasn’t	empty	talk:	there	was	actual	useful	work	to	be	done,	and	lots	of
great	research	teams	were	rolling	up	their	sleeves	to	join	the	effort.
The	fourth	mainstreaming	step	happened	organically	over	the	next	two	years,
with	scores	of	technical	publications	and	dozens	of	workshops	on	AI	safety
around	the	world,	typically	as	parts	of	mainstream	AI	conferences.	Persistent
people	had	tried	for	many	years	to	engage	the	AI	community	in	safety	research,

with	limited	success,	but	now	things	really	took	off.	Many	of	these	publications
were	funded	by	our	grants	program	and	we	at	FLI	did	our	best	to	help	organize
and	fund	as	many	of	these	workshops	as	we	could,	but	a	growing	fraction	of
them	were	enabled	by	AI	researchers	investing	their	own	time	and	resources.	As
a	result,	ever	more	researchers	learned	about	safety	research	from	their	own
colleagues,	discovering	that	aside	from	being	useful,	it	could	also	be	fun,
involving	interesting	mathematical	and	computational	problems	to	puzzle	over.

Complicated	equations	aren’t	everyone’s	idea	of	fun,	of	course.	Two	years
after	our	Puerto	Rico	conference,	we	preceded	our	Asilomar	conference	with	a
technical	workshop	where	our	FLI	grant	winners	could	showcase	their	research,
and	watched	slide	after	slide	with	mathematical	symbols	on	the	big	screen.
Moshe	Vardi,	an	AI	professor	at	Rice	University,	joked	that	he	knew	we’d
succeeded	in	establishing	an	AI-safety	research	field	once	the	meetings	got
boring.
This	dramatic	growth	of	AI-safety	work	wasn’t	limited	to	academia.	Amazon,

boring.
DeepMind,	Facebook,	Google,	IBM	and	Microsoft	launched	an	industry

partnership	for	beneficial	AI.
8
	Major	new	AI-safety	donations	enabled	expanded
research	at	our	largest	nonprofit	sister	organizations:	the	Machine	Intelligence
Research	Institute	in	Berkeley,	the	Future	of	Humanity	Institute	in	Oxford	and
the	Centre	for	the	Study	of	Existential	Risk	in	Cambridge	(UK).	Further
donations	of	$10	million	or	more	kick-started	additional	beneficial-AI	efforts:
the	Leverhulme	Centre	for	the	Future	of	Intelligence	in	Cambridge,	the	K&L
Gates	Endowment	for	Ethics	and	Computational	Technologies	in	Pittsburgh	and
the	Ethics	and	Governance	of	Artificial	Intelligence	Fund	in	Miami.	Last	but	not
least,	with	a	billion-dollar	commitment,	Elon	Musk	partnered	with	other
entrepreneurs	to	launch	OpenAI,	a	nonprofit	company	in	San	Francisco	pursuing
beneficial	AI.	AI-safety	research	was	here	to	stay.
In	lockstep	with	this	surge	of	research	came	a	surge	of	opinions	being
expressed,	both	individually	and	collectively.	The	industry	Partnership	on	AI

published	its	founding	tenets,	and	long	reports	with	lists	of	recommendations
were	published	by	the	U.S.	government,	Stanford	University	and	the	IEEE	(the
world’s	largest	organization	of	technical	professionals),	together	with	dozens	of
other	reports	and	position	papers	from	elsewhere.
9
We	were	eager	to	facilitate	meaningful	discussion	among	the	Asilomar
attendees	and	learn	what,	if	anything,	this	diverse	community	agreed	on.	Lucas
Perry	therefore	took	on	the	heroic	task	of	reading	all	of	those	documents	we’d

9
found	and	extracting	all	their	opinions.	In	a	marathon	effort	initiated	by	Anthony
Aguirre	and	concluded	by	a	series	of	long	telecons,	our	FLI	team	then	attempted
to	group	similar	opinions	together	and	strip	away	redundant	bureaucratic
verbiage	to	end	up	with	a	single	list	of	succinct	principles,	also	including
unpublished	but	influential	opinions	that	had	been	expressed	more	informally	in
talks	and	elsewhere.	But	this	list	still	included	plenty	of	ambiguity,	contradiction
and	room	for	interpretation,	so	the	month	before	the	conference,	we	shared	it

with	the	participants	and	collected	their	opinions	and	suggestions	for	improved
or	novel	principles.	This	community	input	produced	a	significantly	revised
principle	list	for	use	at	the	conference.
In	Asilomar,	the	list	was	further	improved	in	two	steps.	First,	small	groups
discussed	the	principles	they	were	most	interested	in	(
figure	9.4
),	producing
detailed	refinements,	feedback,	new	principles	and	competing	versions	of	old
ones.	Finally,	we	surveyed	all	attendees	to	determine	the	level	of	support	for
each	version	of	each	principle.

Figure	9.3:	Groups	of	great	minds	ponder	AI	principles	in	Asilomar.
This	collective	process	was	both	exhaustive	and	exhausting,	with	Anthony,
Meia	and	I	curtailing	sleep	and	lunch	time	at	the	conference	in	our	scramble	to
compile	everything	needed	in	time	for	the	next	steps.	But	it	was	also	exciting.
After	such	detailed,	thorny	and	sometimes	contentious	discussions	and	such	a
wide	range	of	feedback,	we	were	astonished	by	the	high	level	of	consensus	that
emerged	around	many	of	the	principles	during	that	final	survey,	with	some
getting	over	97%	support.	This	consensus	allowed	us	to	set	a	high	bar	for
inclusion	in	the	final	list:	we	kept	only	principles	that	at	least	90%	of	the
attendees	agreed	on.	Although	this	meant	that	some	popular	principles	were
dropped	at	the	last	minute,	including	some	of	my	personal	favorites,
10
	it	enabled
most	of	the	participants	to	feel	comfortable	endorsing	all	of	them	on	the	sign-up
sheet	that	we	passed	around	the	auditorium.	Here’s	the	result.

The	Asilomar	AI	Principles
Artificial	intelligence	has	already	provided	beneficial	tools	that	are	used	every
day	by	people
	
around	the	world.	Its	continued	development,	guided	by	the
following	principles,	will
	
offer	amazing	opportunities	to	help	and	empower
people	in	the	decades	and	centuries
	
ahead.
R
ESEARCH	
I
SSUES
§1
Research	Goal:	
The	goal	of	AI	research	should	be	to	create	not	undirected
intelligence,	but	beneficial	intelligence.
§2
Research	Funding:	
Investments	in	AI	should	be	accompanied	by	funding
for	research	on
	
ensuring	its	beneficial	use,	including	thorny	questions	in
computer	science,	economics,	law,
	
ethics,	and	social	studies,	such	as:
(a)
How	can	we	make	future	AI	systems	highly	robust,	so	that	they	do	what
we	want
	
without	malfunctioning	or	getting	hacked?
(b)
How	can	we	grow	our	prosperity	through	automation	while	maintaining
people’s
	
resources	and	purpose?
(c)
How	can	we	update	our	legal	systems	to	be	more	fair	and	efficient,	to
keep	pace

(a)
we	want
	
(b)
people’s
	
(c)
keep	pace
	
with	AI,	and	to	manage	the	risks	associated	with	AI?
(d)
What	set	of	values	should	AI	be	aligned	with,	and	what	legal	and	ethical
status
	
should	it	have?
§3
Science-Policy	Link:	
There	should	be	constructive	and	healthy	exchange
between	AI
	
researchers	and	policy-makers.
§4
Research	Culture:	
A	culture	of	cooperation,	trust,	and	transparency	should
be	fostered
	
among	researchers	and	developers	of	AI.
§5
Race	Avoidance:	
Teams	developing	AI	systems	should	actively	cooperate	to
avoid
	
corner-cutting	on	safety	standards.
E
THICS	AND	
V
ALUES

§6
Safety:	
AI	systems	should	be	safe	and	secure	throughout	their	operational
lifetime,
	
and	verifiably	so	where	applicable	and	feasible.
§7
Failure	Transparency:	
If	an	AI	system	causes	harm,	it	should	be	possible	to
ascertain
	
why.
§8
Judicial	Transparency:	
Any	involvement	by	an	autonomous	system	in
judicial
	
decision-making	should	provide	a	satisfactory	explanation
auditable	by	a	competent
	
human	authority.
§9
Responsibility:	
Designers	and	builders	of	advanced	AI	systems	are
stakeholders	in
	
the	moral	implications	of	their	use,	misuse,	and	actions,
with	a	responsibility	and
	
opportunity	to	shape	those	implications.
§10
Value	Alignment:	
Highly	autonomous	AI	systems	should	be	designed	so
that	their
	
goals	and	behaviors	can	be	assured	to	align	with	human	values
throughout	their
	
operation.
§11
Human	Values:	
AI	systems	should	be	designed	and	operated	so	as	to	be
compatible
	
with	ideals	of	human	dignity,	rights,	freedoms,	and	cultural
diversity.
§12
Personal	Privacy:

§10
that	their
	
§11
	
§12
Personal	Privacy:	
People	should	have	the	right	to	access,	manage,	and
control	the
	
data	they	generate,	given	AI	systems’	power	to	analyze	and
utilize	that	data.
§13
Liberty	and	Privacy:	
The	application	of	AI	to	personal	data	must	not
unreasonably
	
curtail	people’s	real	or	perceived	liberty.
§14
Shared	Benefit:	
AI	technologies	should	benefit	and	empower	as	many
people	as
	
possible.
§15
Shared	Prosperity:	
The	economic	prosperity	created	by	AI	should	be	shared
broadly,
	
to	benefit	all	of	humanity.
§16
Human	Control:

§13
	
§14
people	as
	
possible.
§15
broadly,
	
§16
Human	Control:	
Humans	should	choose	how	and	whether	to	delegate
decisions	to
	
AI	systems,	to	accomplish	human-chosen	objectives.
§17
Non-subversion:	
The	power	conferred	by	control	of	highly	advanced	AI
systems
	
should	respect	and	improve,	rather	than	subvert,	the	social	and
civic	processes	on	which
	
the	health	of	society	depends.
§18
AI	Arms	Race:	
An	arms	race	in	lethal	autonomous	weapons	should	be
avoided.

L
ONGER-
T
ERM	
I
SSUES
§19
Capability	Caution:	
There	being	no	consensus,	we	should	avoid	strong
assumptions
	
regarding	upper	limits	on	future	AI	capabilities.
§20
Importance:	
Advanced	AI	could	represent	a	profound	change	in	the	history
of	life	on
	
Earth,	and	should	be	planned	for	and	managed	with
commensurate	care	and	resources.
§21
Risks:	
Risks	posed	by	AI	systems,	especially	catastrophic	or	existential
risks,	must	be
	
subject	to	planning	and	mitigation	efforts	commensurate
with	their	expected	impact.
§22
Recursive	Self-Improvement:	
AI	systems	designed	to	recursively	self-
improve	or
	
self-replicate	in	a	manner	that	could	lead	to	rapidly	increasing
quality	or	quantity	must
	
be	subject	to	strict	safety	and	control	measures.
§23
Common	Good:	
Superintelligence	should	only	be	developed	in	the	service
of	widely
	
shared	ethical	ideals,	and	for	the	benefit	of	all	humanity	rather
than	one	state	or
	
organization.
The	signature	list	grew	dramatically	after	we	posted	the	principles	online,	and

§22
improve	or
	
§23
of	widely
	
by	now	it	includes	an	amazing	list	of	more	than	a	thousand	AI	researchers	and
many	other	top	thinkers.	If	you	too	want	to	join	as	a	signatory,	you	can	do	so
here:	
http://futureoflife.org/ai-principles
.
We	were	struck	not	only	by	the	level	of	consensus	about	the	principles,	but
also	by	how	strong	they	were.	Sure,	some	of	them	sound	about	as	controversial
as	“Peace,	love	and	motherhood	are	valuable”	at	first	glance.	But	many	of	them
have	real	teeth,	as	is	most	easily	seen	by	formulating	negations	of	them.	For

here:	
.
example,	“Superintelligence	is	impossible!”	violates	§19,	and	“It’s	a	total	waste
to	do	research	on	reducing	existential	risk	from	AI!”	violates	§21.
Indeed,	as	you	can	see	for	yourself	if	you	watch	our	long-term	panel
discussion	on	YouTube,
11
	Elon	Musk,	Stuart	Russell,	Ray	Kurzweil,	Demis
Hassabis,	Sam	Harris,	Nick	Bostrom,	David	Chalmers,	Bart	Selman	and	Jaan
Tallinn	all	agreed	that	superintelligence	would	probably	be	developed	and	that
safety	research	was	important.
I	hope	that	the	Asilomar	AI	Principles	will	serve	as	a	starting	point	for	more

11
detailed	discussions,	which	will	ultimately	lead	to	sensible	AI	strategies	and
policies.	In	this	spirit,	our	FLI	media	director	Ariel	Conn	worked	with	Tucker

Davey	and	other	team	members	to	interview	leading	AI	researchers	about	the
principles	and	how	they	interpreted	them,	while	David	Stanley	and	his	team	of
international	FLI	volunteers	translated	the	principles	into	key	world	languages.

Mindful	Optimism
As	I	confessed	in	the	opening	of	this	epilogue,	I’m	feeling	more	optimistic	about
the	future	of	life	than	I	have	in	a	long	time.	I	shared	my	personal	story	to	explain
why.
My	experiences	over	the	past	few	years	have	increased	my	optimism	for	two
separate	reasons.	First,	I’ve	witnessed	the	AI	community	come	together	in	a
remarkable	way	to	constructively	take	on	the	challenges	ahead,	often	in
collaboration	with	thinkers	from	other	fields.	Elon	told	me	after	the	Asilomar
meeting	that	he	found	it	amazing	how	AI	safety	has	gone	from	a	fringe	issue	to
mainstream	in	only	a	few	years,	and	I’m	just	as	amazed	myself.	And	now	it’s	not
merely	the	near-term	issues	from	chapter	3	that	are	becoming	respectable
discussion	topics,	but	even	superintelligence	and	existential	risk,	as	in	the
Asilomar	AI	Principles.	There’s	no	way	that	those	principles	could	have	been
adopted	in	Puerto	Rico	two	years	earlier,	where	the	most	scary-sounding	word
that	made	it	into	the	open	letter	was	“pitfalls.”

I	like	people-watching,	and	at	one	point	during	the	final	morning	of	the
Asilomar	conference,	I	stood	at	the	side	of	the	auditorium	and	watched	the
participants	listen	to	a	discussion	about	AI	and	law.	To	my	surprise,	a	warm	and
fuzzy	feeling	swept	through	me	and	I	suddenly	felt	very	moved.	This	felt	so
different	from	Puerto	Rico!	Back	then,	I	remember	viewing	most	of	the	AI
community	with	a	combination	of	respect	and	fear—not	exactly	as	an	opposing
team,	but	as	a	group	that	my	AI-concerned	colleagues	and	I	felt	we	needed	to

persuade.	But	now	it	felt	so	obvious	that	we	were	all	on	the	
same
	team.	As
you’ve	probably	gleaned	from	reading	this	book,	I	still	don’t	have	the	answers
for	how	to	create	a	great	future	with	AI,	so	it	feels	great	to	be	part	of	a	growing
community	searching	for	the	answers	together.

Figure	9.4:	A	growing	community	searches	for	answers	together	in	Asilomar.
The	second	reason	I’ve	grown	more	optimistic	is	that	the	FLI	experience	has
been	empowering.	What	had	triggered	my	London	tears	was	a	feeling	of
inevitability:	that	a	disturbing	future	may	be	coming	and	there	was	nothing	we
could	do	about	it.	But	the	next	three	years	dissolved	my	fatalistic	gloom.	If	even
a	ragtag	bunch	of	unpaid	volunteers	could	make	a	positive	difference	for	what’s
arguably	the	most	important	conversation	of	our	time,	then	imagine	what	we	can
all	do	if	we	work	together!
Erik	Brynjolfsson	spoke	of	two	kinds	of	optimism	in	his	Asilomar	talk.	First
there’s	the	unconditional	kind,	such	as	the	positive	expectation	that	the	Sun	will
rise	tomorrow	morning.	Then	there’s	what	he	called	“mindful	optimism,”	which
is	the	expectation	that	good	things	will	happen	if	you	plan	carefully	and	work
hard	for	them.	That’s	the	kind	of	optimism	I	now	feel	about	the	future	of	life.
So	what	can	
you

So	what	can	
you
	do	to	make	a	positive	difference	for	the	future	of	life	as	we
enter	the	age	of	AI?	For	reasons	I’ll	soon	explain,	I	think	that	a	great	first	step	is
working	on	becoming	a	mindful	optimist,	if	you	aren’t	one	already.	To	be	a
successful	mindful	optimist,	it’s	crucial	to	develop	positive	visions	for	the	future.
When	MIT	students	come	to	my	office	for	career	advice,	I	usually	start	by

asking	them	where	they	see	themselves	in	a	decade.	If	a	student	were	to	reply
“Perhaps	I’ll	be	in	a	cancer	ward,	or	in	a	cemetery	after	getting	hit	by	a	bus,”	I’d
give	her	a	hard	time.	Envisioning	only	negative	futures	is	a	terrible	approach	to
career	planning!	Devoting	100%	of	one’s	efforts	to	avoiding	diseases	and
accidents	is	a	great	recipe	for	hypochondria	and	paranoia,	not	happiness.	Instead,
I’d	like	to	hear	her	describe	her	goals	with	enthusiasm,	after	which	we	can
discuss	strategies	for	getting	there	while	avoiding	pitfalls.
Erik	pointed	out	that	according	to	game	theory,	positive	visions	form	the
foundation	of	a	large	fraction	of	all	collaboration	in	the	world,	from	marriages
and	corporate	mergers	to	the	decision	of	independent	states	to	form	the	USA.
After	all,	why	sacrifice	something	you	have	if	you	can’t	imagine	the	even	greater
gain	that	this	will	provide?	This	means	that	we	should	be	imagining	positive
futures	not	only	for	ourselves,	but	also	for	society	and	for	humanity	itself.	In

other	words,	we	need	more	existential	hope!	Yet,	as	Meia	likes	to	remind	me,
from	Frankenstein	to	the	Terminator,	futuristic	visions	in	literature	and	film	are
predominantly	dystopian.	In	other	words,	we	as	a	society	are	planning	our	future
about	as	poorly	as	that	hypothetical	MIT	student.	This	is	why	we	need	more
mindful	optimists.	And	this	is	why	I’ve	encouraged	you	throughout	this	book	to
think	about	what	sort	of	future	you	
want
	rather	than	merely	what	sort	of	future
you	
fear,
	so	that	we	can	find	shared	goals	to	plan	and	work	for.

want
you	
fear,
We’ve	seen	throughout	this	book	how	AI	is	likely	to	give	us	both	grand
opportunities	and	tough	challenges.	A	strategy	that’s	likely	to	help	with
essentially	all	AI	challenges	is	for	us	to	get	our	act	together	and	improve	our
human	society	
before
	AI	fully	takes	off.	We’re	better	off	educating	our	young	to
make	technology	robust	and	beneficial	before	ceding	great	power	to	it.	We’re
better	off	modernizing	our	laws	before	technology	makes	them	obsolete.	We’re
better	off	resolving	international	conflicts	before	they	escalate	into	an	arms	race

before
in	autonomous	weapons.	We’re	better	off	creating	an	economy	that	ensures
prosperity	for	all	before	AI	potentially	amplifies	inequalities.	We’re	better	off	in
a	society	where	AI-safety	research	results	get	implemented	rather	than	ignored.
And	looking	further	ahead,	to	challenges	related	to	superhuman	AGI,	we’re
better	off	agreeing	on	at	least	some	basic	ethical	standards	before	we	start
teaching	these	standards	to	powerful	machines.	In	a	polarized	and	chaotic	world,
people	with	the	power	to	use	AI	for	malicious	purposes	will	have	more

motivation	and	ability	to	do	so,	and	teams	racing	to	build	AGI	will	feel	more
pressure	to	cut	corners	on	safety	than	to	cooperate.	In	summary,	if	we	can	create
a	more	harmonious	human	society	characterized	by	cooperation	toward	shared

goals,	this	will	improve	the	prospects	of	the	AI	revolution	ending	well.
In	other	words,	one	of	the	best	ways	for	you	to	improve	the	future	of	life	is	to
improve	tomorrow.	You	have	power	to	do	so	in	many	ways.	Of	course	you	can
vote	at	the	ballot	box	and	tell	your	politicians	what	you	think	about	education,
privacy,	lethal	autonomous	weapons,	technological	unemployment	and	other
issues.	But	you	also	vote	every	day	through	what	you	choose	to	buy,	what	news
you	choose	to	consume,	what	you	choose	to	share	and	what	sort	of	role	model
you	choose	to	be.	Do	you	want	to	be	someone	who	interrupts	all	their
conversations	by	checking	their	smartphone,	or	someone	who	feels	empowered
by	using	technology	in	a	planned	and	deliberate	way?	Do	you	want	to	own	your
technology	or	do	you	want	your	technology	to	own	you?	What	do	you	want	it	to
mean	to	be	human	in	the	age	of	AI?	Please	discuss	all	this	with	those	around	you
—it’s	not	only	an	important	conversation,	but	a	fascinating	one.

We’re	the	guardians	of	the	future	of	life	now	as	we	shape	the	age	of	AI.
Although	I	cried	in	London,	I	now	feel	that	there’s	nothing	inevitable	about	this
future,	and	I	know	that	it’s	much	easier	to	make	a	difference	than	I	thought.	Our
future	isn’t	written	in	stone	and	just	waiting	to	happen	to	us—it’s	ours	to	create.
Let’s	create	an	inspiring	one	together!
*
	
This	experience	also	made	me	rethink	how	I	personally	should	interpret	news.	Although	I’d	obviously
been	aware	that	most	outlets	have	their	own	political	agenda,	I	now	realized	that	they	also	have	a	bias

*
	
away	from	the	center	on	all	issues,	even	nonpolitical	ones.

Chapter	1
1.
“The	AI	Revolution:	Our	Immortality	or	Extinction?”	
Wait	But	Why
	(January	27,	2015),	at
http://waitbutwhy.com/2015/01/artificial-intelligence-revolution-2.html
.
2.
This	open	letter,	“Research	Priorities	for	Robust	and	Beneficial	Artificial	Intelligence,”	can	be	found
at	
http://futureoflife.org/ai-open-letter/
.
3.
Example	of	classic	robot	alarmism	in	the	media:	Ellie	Zolfagharifard,	“Artificial	Intelligence	‘Could
Be	the	Worst	Thing	to	Happen	to	Humanity,’ ”	
Daily	Mail,
	May	2,	2014;
http://tinyurl.com/hawkingbots
.

Chapter	2
1.
Notes	on	the	origin	of	the	term	AGI:	
http://wp.goertzel.org/who-coined-the-term-agi
.
2.
Hans	Moravec,	“When	Will	Computer	Hardware	Match	the	Human	Brain?”	
Journal	of	Evolution	and
Technology
	(1998),	vol.	1.
3.
In	the	figure	showing	computing	power	versus	year,	the	pre-2011	data	is	from	Ray	Kurzweil’s	book
How	to	Create	a	Mind,
	and	subsequent	data	is	computed	from	the	references	in
https://en.wikipedia.org/wiki/FLOPS
.
4.
Quantum	computing	pioneer	David	Deutsch	describes	how	he	views	quantum	computation	as
evidence	of	parallel	universes	in	his	
The	Fabric	of	Reality:	The	Science	of	Parallel	Universes—and	Its
Implications
	(London:	Allen	Lane,	1997).	If	you	want	my	own	take	on	quantum	parallel	universes	as
the	third	of	four	multiverse	levels,	you’ll	find	it	in	my	previous	book:	Max	Tegmark,	
Our
Mathematical
	
Universe:	My	Quest	for	the	Ultimate	Nature	of	Reality
	(New	York:	Knopf,	2014).

Chapter	3
1.
Watch	“Google	DeepMind’s	Deep	Q-learning	Playing	Atari	Breakout”	on	YouTube	at
https://tinyurl.com/atariai
.
2.
See	Volodymyr	Mnih	et	al.,	“Human-Level	Control	Through	Deep	Reinforcement	Learning,”	
Nature
518	(February	26,	2015):	529–533,	available	online	at	
http://tinyurl.com/ataripaper
.
3.
Here’s	a	video	of	the	Big	Dog	robot	in	action:	
https://www.youtube.com/watch?v=W1czBcnX1Ww
.
4.
For	reactions	to	the	sensationally	creative	line	5	move	by	AlphaGO,	see	“Move	37!!	Lee	Sedol	vs
AlphaGo	Match	2,”	at	
https://www.youtube.com/watch?v=JNrXgpSEEIE
.
5.
Demis	Hassabis	describing	reactions	to	AlphaGo	from	human	Go	players:
https://www.youtube.com/watch?v=otJKzpNWZT4
.
6.
For	recent	improvements	in	machine	translation,	see	Gideon	Lewis-Kraus,	“The	Great	A.I.
Awakening,”	
New	York	Times	Magazine
	(December	14,	2016),	available	online	at
http://www.nytimes.com/2016/12/14/magazine/the-great-ai-awakening.html
.	GoogleTranslate	is
available	here	at	
.
7.

.
5.
.
6.
.
7.
Winograd	Schema	Challenge	competition:	
http://tinyurl.com/winogradchallenge
.
8.
Ariane	5	explosion	video:	
https://www.youtube.com/watch?v=qnHn8W1Em6E
.
9.
Ariane	5	Flight	501	Failure	report	by	the	inquiry	board:	
http://tinyurl.com/arianeflop
.
10.
NASA’s	Mars	Climate	Orbiter	Mishap	Investigation	Board	Phase	I	report:	
http://tinyurl.com/marsflop
.
11.
The	most	detailed	and	consistent	account	of	what	caused	the	Mariner	1	Venus	mission	failure	was

.
7.
.
8.
.
9.
.
10.
.
11.
incorrect	hand-transcription	of	a	single	mathematical	symbol	(a	missing	overbar):
http://tinyurl.com/marinerflop
.
12.
A	detailed	description	of	the	failure	of	the	Soviet	Phobos	1	Mars	mission	can	be	found	in	Wesley	T.
Huntress	Jr.	and	Mikhail	Ya.	Marov,	
Soviet	Robots	in	the	Solar	System
	(New	York:	Praxis	Publishing,
2011),	p.	308.
13.
How	unverified	software	cost	Knight	Capital	$440	million	in	45	minutes:
http://tinyurl.com/knightflop1
	and	
http://tinyurl.com/knightflop2
.
14.

.
12.
13.
	and	
.
14.
U.S.	government	report	on	the	Wall	Street	“flash	crash”:	“Findings	Regarding	the	Market	Events	of
May	6,	2010”	(September	30,	2010),	at	
http://tinyurl.com/flashcrashreport
.
15.
3-D	printing	of	buildings	(
https://www.youtube.com/watch?v=SObzNdyRTBs
),	micromechanical
devices	(
http://tinyurl.com/tinyprinter
)	and	many	things	in	between	(
https://www.youtube.com/watch?
v=xVU4FLrsPXs
).
16.
Global	map	of	community-based	fab	labs:	
https://www.fablabs.io/labs/map
.
17.

.
14.
.
15.
devices	(
).
16.
.
17.
News	article	about	Robert	Williams	being	killed	by	an	industrial	robot:
http://tinyurl.com/williamsaccident
.
18.
News	article	about	Kenji	Urada	being	killed	by	an	industrial	robot:	
http://tinyurl.com/uradaaccident
.
19.
News	article	about	Volkswagen	worker	being	killed	by	an	industrial	robot:
http://tinyurl.com/baunatalaccident
.

20.
U.S.	government	report	on	worker	fatalities:	
https://www.osha.gov/dep/fatcat/dep_fatcat.html
.
21.
Car	accident	fatality	statistics:	
http://tinyurl.com/roadsafety2
	and	
http://tinyurl.com/roadsafety3
.
22.
On	the	first	Tesla	autopilot	fatality,	see	Andrew	Buncombe,	“Tesla	Crash:	Driver	Who	Died	While	on
Autopilot	Mode	‘Was	Watching	Harry	Potter,’ ”	
Independent
	(July	1,	2016),
http://tinyurl.com/teslacrashstory
.	For	the	report	of	the	Office	of	Defects	Investigation	of	the	U.S.
National	Highway	Traffic	Safety	Administration,	see	
http://tinyurl.com/teslacrashreport
.
23.
For	more	about	the	
Herald	of	Free	Enterprise
	disaster,	see	R.	B.	Whittingham,	
The	Blame
	
Machine:
Why	Human	Error	Causes	Accidents
	(Oxford,	UK:	Elsevier,	2004).
24.
Documentary	about	the	Air	France	447	crash:	
https://www.youtube.com/watch?v=dpPkp8OGQFI
;
accident	report:	
http://tinyurl.com/af447report
;	outside	analysis:	
http://tinyurl.com/thomsonarticle
.
25.
Official	report	on	the	2003	U.S.-Canadian	blackout:

.
23.
The	Blame
	
Machine:
24.
;
.
25.
Official	report	on	the	2003	U.S.-Canadian	blackout:	
http://tinyurl.com/uscanadablackout
.
26.
Final	report	of	the	President’s	Commission	on	the	Accident	at	Three	Mile	Island:
http://www.threemileisland.org/downloads/188.pdf
.
27.
Dutch	study	showing	how	AI	can	rival	human	radiologists	at	MRI-based	diagnosis	of	prostate	cancer:
http://tinyurl.com/prostate-ai
.
28.
Stanford	study	showing	how	AI	can	best	human	pathologists	at	lung	cancer	diagnosis:
http://tinyurl.com/lungcancer-ai
.
29.
Investigation	of	the	Therac-25	radiation	therapy	accidents:	
http://tinyurl.com/theracfailure
.
30.

.
26.
.
27.
.
28.
.
29.
.
30.
Report	on	lethal	radiation	overdoses	in	Panama	caused	by	confusing	user	interface:
http://tinyurl.com/cobalt60accident
.
31.
Study	of	adverse	events	in	robotic	surgery:	
https://arxiv.org/abs/1507.03518
.
32.
Article	on	number	of	deaths	from	bad	hospital	care:	
http://tinyurl.com/medaccidents
.
33.
Yahoo	set	a	new	standard	for	“big	hack”	when	announcing	a	billion(!)	of	their	user	accounts	had	been
breached:	
https://www.wired.com/2016/12/yahoo-hack-billion-users/
.
34.
New	York	Times

.
30.
.
31.
.
32.
.
33.
breached:	
.
34.
	article	on	acquittal	and	later	conviction	of	KKK	murderer:
http://tinyurl.com/kkkacquittal
.
35.
The	Danziger	et	al.	2011	study	(
http://www.pnas.org/content/108/17/6889.full
),	arguing	that	hungry
judges	are	harsher,	was	criticized	as	flawed	by	Keren	Weinshall-Margela	and	John	Shapard
(
http://www.pnas.org/content/108/42/E833.full
),	but	Danziger	et	al.	insist	that	their	claims	remain
valid	(
http://www.pnas.org/content/108/42/E834.full
).
36.
Pro	Publica
	report	on	racial	bias	in	recidivism-prediction	software:

.
35.
(
valid	(
).
36.
	report	on	racial	bias	in	recidivism-prediction	software:	
http://tinyurl.com/robojudge
.
37.
Use	of	fMRI	and	other	brain-scanning	techniques	as	evidence	in	trials	is	highly	controversial,	as	is	the
reliability	of	such	techniques,	although	many	teams	claim	accuracies	better	than	90%:
http://journal.frontiersin.org/article/10.3389/fpsyg.2015.00709/full
.
38.
PBS	made	the	movie	
The	Man	Who	Saved	the	World
	about	the	incident	where	Vasili	Arkhipov	single-
handedly	prevented	a	Soviet	nuclear	strike:	
https://www.youtube.com/watch?v=4VPY2SgyG5w
.
39.

.
37.
.
38.
.
39.
The	story	of	how	Stanislav	Petrov	dismissed	warnings	of	a	U.S.	nuclear	attack	as	a	false	alarm	was
turned	into	the	movie	
	(not	to	be	confused	with	the	movie	by	the	same
title	in	the	previous	note),	and	Petrov	was	honored	at	the	United	Nations	and	given	the	World	Citizen
Award:	
https://www.youtube.com/watch?v=IncSjwWQHMo
.
40.
Open	letter	from	AI	and	robotics	researchers	about	autonomous	weapons:	
http://futureoflife.org/open-

letter-autonomous-weapons/
.
41.
A	U.S.	official	seemingly	wanting	a	military	AI	arms	race:	
http://tinyurl.com/workquote
.
42.
Study	of	wealth	inequality	in	the	United	States	since	1913:	
http://gabriel-
zucman.eu/files/SaezZucman2015.pdf
.
43.
Oxfam	report	on	global	wealth	inequality:	
http://tinyurl.com/oxfam2017
.
44.
For	a	great	introduction	to	the	hypothesis	of	technology-driven	inequality,	see	Erik	Brynjolfsson	and
Andrew	McAfee,	
The	Second
	
Machine	Age:	Work,	Progress,	and	Prosperity	in	a	Time	of	Brilliant
Technologies
	(New	York:	Norton,	2014).
45.
Article	in	
The	Atlantic
	about	falling	wages	for	the	less	educated:	
http://tinyurl.com/wagedrop
.
46.
The	data	plotted	are	taken	from	Facundo	Alvaredo,	Anthony	B.	Atkinson,	Thomas	Piketty,	Emmanuel
Saez	and	Gabriel	Zucman,	
The	World	Wealth	and	Income	Database
	(
http://www.wid.world
),	including
capital	gains.
47.
Presentation	by	James	Manyika	showing	income	shifting	from	labor	to	capital:
http://futureoflife.org/data/PDF/james_manyika.pdf
.
48.

45.
Article	in	
.
46.
	(
47.
.
48.
Forecasts	about	future	job	automation	from	Oxford	University	(
http://tinyurl.com/automationoxford
)
and	McKinsey	(
http://tinyurl.com/automationmckinsey
).
49.
Video	of	robotic	chef:	
https://www.youtube.com/watch?v=fE6i2OO6Y6s
.
50.
Marin	Solja
č
i
ć
	explored	these	options	at	the	2016	workshop	Computers	Gone	Wild:	Impact	and
Implications	of	Developments	in	Artificial	Intelligence	on	Society:
http://futureoflife.org/2016/05/06/computers-gone-wild/
.
51.
Andrew	McAfee’s	suggestions	for	how	to	create	more	good	jobs:

)
).
49.
.
50.
č
i
ć
.
51.
http://futureoflife.org/data/PDF/andrew_mcafee.pdf
.
52.
In	addition	to	many	academic	articles	arguing	that	“this	time	is	different”	for	technological
unemployment,	the	video	“Humans	Need	Not	Apply”	succinctly	makes	the	same	point:
https://www.youtube.com/watch?v=7Pq-S557XQU
.
53.
U.S.	Bureau	of	Labor	Statistics:	
http://www.bls.gov/cps/cpsaat11.htm
.
54.
Argument	that	“this	time	is	different”	for	technological	unemployment:	Federico	Pistono,	
Robots	Will
Steal
	
Your	Job,	but	That’s	OK
	(2012),	
http://robotswillstealyourjob.com
.
55.

.
52.
.
53.
.
54.
Steal
	
	(2012),	
.
55.
Changes	in	the	U.S.	horse	population:	
http://tinyurl.com/horsedecline
.
56.
Meta-analysis	showing	how	unemployment	affects	well-being:	Maike	Luhmann	et	al.,	“Subjective
Well-Being	and	Adaptation	to	Life	Events:	A	Meta-Analysis,”	
Journal	of	Personality	and	Social
Psychology
	102,	no.	3	(2012):	592;	available	online	at
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3289759
.
57.
Studies	of	what	boosts	people’s	sense	of	well-being:	Angela	Duckworth,	Tracy	Steen	and	Martin
Seligman,	“Positive	Psychology	in	Clinical	Practice,”

.
56.
.
57.
Seligman,	“Positive	Psychology	in	Clinical	Practice,”	
Annual	Review	of	Clinical	Psychology
	1	(2005):
629–651,	online	at	
http://tinyurl.com/wellbeingduckworth
.	Weiting	Ng	and	Ed	Diener,	“What	Matters
to	the	Rich	and	the	Poor?	Subjective	Well-Being,	Financial	Satisfaction,	and	Postmaterialist	Needs
Across	the	World,”	
Journal	of	Personality	and	Social	Psychology
	107,	no.	2	(2014):	326,	online	at
http://psycnet.apa.org/journals/psp/107/2/326
.	Kirsten	Weir,	“More	than	Job	Satisfaction,”	
Monitor	on
	44,	no.	11	(December	2013),	online	at

	1	(2005):
Monitor	on
	44,	no.	11	(December	2013),	online	at	
http://www.apa.org/monitor/2013/12/job-
satisfaction.aspx
.
58.
Multiplying	together	about	10
11
	neurons,	about	10
4
	connections	per	neuron	and	about	one	(10
0
)	firing

per	neuron	each	second	might	suggest	that	about	10
15
	FLOPS	(1	petaFLOPS)	suffice	to	simulate	a
human	brain,	but	there	are	many	poorly	understood	complications,	including	the	detailed	timing	of
firings	and	the	question	of	whether	small	parts	of	neurons	and	synapses	need	to	be	simulated	too.	IBM
computer	scientist	Dharmendra	Modha	has	estimated	that	38	petaFLOPS	are	required
(
http://tinyurl.com/javln43
),	while	neuroscientist	Henry	Markram	has	estimated	that	one	needs	about
1,000	petaFLOPS	(
http://tinyurl.com/6rpohqv
).	AI	researchers	Katja	Grace	and	Paul	Christiano	have
argued	that	the	most	costly	aspect	of	brain	simulation	is	not	computation	but	
communication,
	and	that
this	too	is	a	task	in	the	ballpark	of	what	the	best	current	supercomputers	can	do:
http://aiimpacts.org/about
.
59.
For	an	interesting	estimate	of	the	computational	power	of	the	human	brain:	Hans	Moravec	“When
Will	Computer	Hardware	Match	the	Human	Brain?”	
Journal	of	Evolution	and	Technology,
	vol.	1
(1998).

Chapter	4
1.
For	a	video	of	the	first	mechanical	bird,	see	Markus	Fischer,	“A	Robot	That	Flies	like	a	Bird,”	TED
Talk,	July	2011,	at	
https://www.ted.com/talks/a_robot_that_flies_like_a_bird
.

Chapter	5
1.
Ray	Kurzweil,	
The
	
Singularity	Is	Near
	(New	York:	Viking	Press,	2005).
2.
Ben	Goertzel’s	“Nanny	AI”	scenario	is	described	here:	
https://wiki.lesswrong.com/wiki/Nanny_AI
.
3.
For	a	discussion	about	the	relationship	between	machines	and	humans,	and	whether	machines	are	our
slaves,	see	Benjamin	Wallace-Wells,	“Boyhood,”	
New	York
	magazine	(May	20,	2015),	online	at
http://tinyurl.com/aislaves
.
4.
Mind	crime	is	discussed	in	Nick	Bostrom’s	book	
	and	in	more	technical	detail	in	this
recent	paper:	Nick	Bostrom,	Allan	Dafoe	and	Carrick	Flynn,	“Policy	Desiderata	in	the	Development
of	Machine	Superintelligence”	(2016),	
http://www.nickbostrom.com/papers/aipolicy.pdf
.
5.
Matthew	Schofield,	“Memories	of	Stasi	Color	Germans’	View	of	U.S.	Surveillance
Programs,”
McClatchy	DC	Bureau
	(June	26,	2013),	online	at
http://www.mcclatchydc.com/news/nation-world/national/article24750439.html
.
6.
For	thought-provoking	reflections	on	how	people	can	be	incentivized	to	create	outcomes	that	nobody

.
5.
.
6.
wants,	I	recommend	“Meditations	on	Moloch,”	
http://slatestarcodex.com/2014/07/30/meditations-on-
moloch
.
7.
For	an	interactive	timeline	of	close	calls	when	nuclear	war	might	have	started	by	accident,	see	Future
of	Life	Institute,	“Accidental	Nuclear	War:	A	Timeline	of	Close	Calls,”	online	at
http://tinyurl.com/nukeoops
.
8.
For	compensation	payments	made	to	U.S.	nuclear	testing	victims,	see	U.S.	Department	of	Justice
website,	“Awards	to	Date	4/24/2015,”	at	
https://www.justice.gov/civil/awards-date-04242015
.

moloch
.
7.
.
8.
.
9.
Report	of	the	Commission	to	Assess	the	Threat	to	the	United	States	from	Electromagnetic	Pulse
(EMP)	Attack,
	April	2008,	available	online	at	
http://www.empcommission.org/docs/A2473-
EMP_Commission-7MB.pdf
.
10.
Independent	research	by	both	U.S.	and	Soviet	scientists	alerted	Reagan	and	Gorbachev	to	the	risk	of
nuclear	winter:	P.	J.	Crutzen	and	J.	W.	Birks,	“The	Atmosphere	After	a	Nuclear	War:	Twilight	at
Noon,”	
Ambio
	11,	no.	2/3	(1982):	114–125.	R.	P.	Turco,	O.	B.	Toon,	T.	P.	Ackerman,	J.	B.	Pollack	and

.
10.
Noon,”	
Ambio
C.	Sagan,	“Nuclear	Winter:	Global	Consequences	of	Multiple	Nuclear	Explosions,”	
Science
	222
(1983):	1283–1292.	V.	V.	Aleksandrov	and	G.	L.	Stenchikov,	“On	the	Modeling	of	the	Climatic
Consequences	of	the	Nuclear	War,”	
Proceeding	on	Applied	Mathematics	
(Moscow:	Computing	Centre
of	the	USSR	Academy	of	Sciences,	1983),	21.	A.	Robock,	“Snow	and	Ice	Feedbacks	Prolong	Effects
of	Nuclear	Winter,”	
Nature
	310	(1984):	667–670.
11.

Science
	222
Nature
11.
Calculation	of	climate	effects	of	global	nuclear	war:	A.	Robock,	L.	Oman	and	L.	Stenchikov,	“Nuclear
Winter	Revisited	with	a	Modern	Climate	Model	and	Current	Nuclear	Arsenals:	Still	Catastrophic
Consequences,”	
Journal	of	Geophysical	Research
	12	(2007):	D13107.

Chapter	6
1.
For	more	information,	see	Anders	Sandberg,	“Dyson	Sphere	FAQ,”	at
http://www.aleph.se/nada/dysonFAQ.html
.
2.
Freeman	Dyson’s	seminal	paper	on	his	eponymous	spheres:	Freeman	Dyson,	“Search	for	Artificial
Stellar	Sources	of	Infrared	Radiation,”	
Science
,	vol.	131	(1959):	1667–1668.
3.
Louis	Crane	and	Shawn	Westmoreland	explain	their	proposed	black	hole	engine	in	“Are	Black	Hole
Starships	Possible?,”	at	
http://arxiv.org/pdf/0908.1803.pdf
.
4.
For	a	nice	infographic	from	CERN	summarizing	known	elementary	particles,	see
http://tinyurl.com/cernparticles
.
5.
This	unique	video	of	a	non-nuclear	Orion	prototype	illustrates	the	idea	of	nuclear-bomb-powered
rocket	propulsion:	
https://www.youtube.com/watch?v=E3Lxx2VAYi8
.
6.
Here’s	a	pedagogical	introduction	to	laser	sailing:	Robert	L.	Forward,	“Roundtrip	Interstellar	Travel
Using	Laser-Pushed	Lightsails,”	
Journal	of	Spacecraft	and	Rockets
	21,	no.	2	(March–April	1984),
available	online	at	
http://www.lunarsail.com/LightSail/rit-1.pdf
.
7.

.
5.
.
6.
.
7.
Jay	Olson	analyzes	cosmically	expanding	civilizations	in	“Homogeneous	Cosmology	with
Aggressively	Expanding	Civilizations,”	
Classical	and	Quantum	Gravity
	32	(2015),	available	online	at
http://arxiv.org/abs/1411.4359
.
8.
The	first	thorough	scientific	analysis	of	our	far	future:	Freeman	J.	Dyson,	“Time	Without	End:	Physics
and	Biology	in	an	Open	Universe,”	
Reviews	of	Modern	Physics
	51,	no.	3	(1979):	447,	available	online
at	
http://blog.regehr.org/extra_files/dyson.pdf
.
9.

.
7.
.
8.
at	
.
9.
Seth	Lloyd’s	above-mentioned	formula	told	us	that	performing	a	computational	operation	during	a
time	interval	
τ
	costs	an	energy	
E
≥
h/4
τ
,
	where	
h
	is	Planck’s	constant.	If	we	want	to	get	
N
	operations
done	one	after	the	other	(in	series)	during	a	time	
T
,	then	
τ
	=	
T
⁄
N
,	so	
E
⁄
N
	
≥
	
hN
⁄
4
T,
	which	tells	us	that	we
can	perform	
N
	
≤
	2	
√
ET/h
	serial	operations	using	energy	
E
	and	time	
T
.	So	both	energy	and	time	are
resources	that	it	helps	having	lots	of.	If	you	split	your	energy	between	
n

τ
E
≥
h/4
τ
,
	where	
h
N
T
,	then	
τ
	=	
T
⁄
N
,	so	
E
⁄
N
	
≥
	
hN
⁄
4
T,
N
	
≤
	2	
√
ET/h
E
	and	time	
T
n
	different	parallel
computations,	they	can	run	more	slowly	and	efficiently,	giving	
N
	
≤
	2	
√
ETn/h
.	Nick	Bostrom	estimates
that	simulating	a	100-year	human	life	requires	about	
N
	=	10
27
	operations.
10.
If	you	want	to	see	a	careful	argument	for	why	the	origin	of	life	may	require	a	very	rare	fluke,	placing
our	nearest	neighbors	over	10
1000
	meters	away,	I	recommend	this	video	by	Princeton	physicist	and
astrobiologist	Edwin	Turner:	“Improbable	Life:	An	Unappealing	but	Plausible	Scenario	for	Life’s
Origin	on	Earth,”	at	
https://www.youtube.com/watch?v=Bt6n6Tu1beg
.
11.

N
	
≤
	2	
√
ETn/h
N
	=	10
27
10.
1000
.
11.
Essay	by	Martin	Rees	on	the	search	for	extraterrestrial	intelligence:	
https://www.edge.org/annual-
question/2016/response/26665
.

Chapter	7
1.
A	popular	discussion	of	Jeremy	England’s	work	on	“dissipation-driven	adaptation”	can	be	found	in
Natalie	Wolchover,	“A	New	Physics	Theory	of	Life,”	
Scientific	American
	(January	28,	2014),
https://www.scientificamerican.com/article/a-new-physics-theory-of-life/
.	Ilya
Prigogine	and	Isabelle	Stengers’s	
Order	Out	of
	
Chaos:	Man’s	New	Dialogue	with	Nature
	(New	York:
Bantam,	1984)	lays	many	of	the	foundations	for	this.
2.
For	more	on	feelings	and	their	physiological	roots:	William	James,	
Principles	of	Psychology
	(New
York:	Henry	Holt	&	Co.,	1890);	Robert	Ornstein,	
Evolution	of	Consciousness:	The	Origins	of	the	Way
We	Think
	(New	York:	Simon	&	Schuster,	1992);	António	Damásio,	
Descartes’	Error:	Emotion,
Reason,	and	the	Human	Brain
	(New	York:	Penguin,	2005);	and	António	Damásio,	
Self	Comes	to
Mind:	Constructing	the	Conscious	Brain
	(New	York:	Vintage,	2012).
3.
Eliezer	Yudkowsky	has	discussed	aligning	the	goals	of	friendly	AI	not	with	our	present	goals,	but	with
our

	(New
We	Think
3.
our	
coherent	extrapolated	volition
	(CEV).	Loosely	speaking	this	is	defined	as	what	an	idealized
version	of	us	would	want	if	we	knew	more,	thought	faster	and	were	more	the	people	we	wished	we
were.	Yudkowsky	began	criticizing	CEV	shortly	after	publishing	it	in	2004
(
http://intelligence.org/files/CEV.pdf
),	both	for	being	hard	to	implement	and	because	it’s	unclear
whether	it	would	converge	to	anything	well-defined.
4.
In	the	inverse	reinforcement-learning	approach,	a	core	idea	is	that	the	AI	is	trying	to	maximize	not	its

(CEV).	Loosely	speaking	this	is	defined	as	what	an	idealized
(
4.
own	goal-satisfaction,	but	that	of	its	human	owner.	It	therefore	has	incentive	to	be	cautious	when	it’s
unclear	about	what	its	owner	wants,	and	to	do	its	best	to	find	out.	It	should	also	be	fine	with	its	owner
switching	it	off,	since	that	would	imply	that	it	had	misunderstood	what	its	owner	really	wanted.
5.
Steve	Omohundro’s	paper	on	AI	goal	emergence,	“The	Basic	AI	Drives,”	can	be	found	online	at
http://tinyurl.com/omohundro2008
.	Originally	published	in	
Artificial	General	Intelligence	2008:

5.
Proceedings	of	the	First	AGI	Conference,
	ed.	Pei	Wang,	Ben	Goertzel	and	Stan	Franklin	(Amsterdam:
IOS,	2008),	483–492.
6.
A	thought-provoking	and	controversial	book	on	what	happens	when	intelligence	is	used	to	blindly
obey	orders	without	questioning	their	ethical	basis:	Hannah	Arendt,	
Eichmann	in	Jerusalem:	A	Report
on	the	Banality	of	Evil
	(New	York:	Penguin,	1963).	A	related	dilemma	applies	to	a	recent	proposal	by
Eric	Drexler	(
http://www.fhi.ox.ac.uk/reports/2015-3.pdf

6.
)	to	keep	superintelligence	under	control	by
compartmentalizing	it	into	simple	pieces,	none	of	which	understand	the	whole	picture.	If	this	works,
this	could	again	provide	an	incredibly	powerful	tool	without	an	intrinsic	moral	compass,
implementing	its	owner’s	every	whim	without	any	moral	qualms.	This	would	be	reminiscent	of	a
compartmentalized	bureaucracy	in	a	dystopian	dictatorship:	one	part	builds	weapons	without	knowing
how	they’ll	be	used,	another	executes	prisoners	without	knowing	why	they	were	convicted,	and	so	on.
7.

7.
A	modern	variant	of	the	Golden	Rule	is	John	Rawls’	idea	that	a	hypothetical	situation	is	fair	if	nobody
would	change	it	without	knowing	in	advance	which	person	in	it	they’d	be.
8.
For	example,	the	IQs	of	many	of	Hitler’s	top	officials	were	found	to	be	quite	high.	See	“How
Accurate	Were	the	IQ	Scores	of	the	High-Ranking	Third	Reich	Officials	Tried	at	Nuremberg?,”
Quora,
	available	online	at	
http://tinyurl.com/nurembergiq
.

Chapter	8
1.
The	entry	on	consciousness	by	Stuart	Sutherland	is	quite	amusing:	
Macmillan	Dictionary
	
of
	(London:	Macmillan,	1989).
2.
Erwin	Schrödinger,	one	of	the	founding	fathers	of	quantum	mechanics,	made	this	remark	in	his	book
Mind	and	Matter
	while	contemplating	the	
past—
and	what	would	have	happened	if	conscious	life
never	evolved	in	the	first	place.	On	the	other	hand,	the	rise	of	AI	raises	the	logical	possibility	that	we
may	end	up	with	a	play	for	empty	benches	in	the	
future
.
3.
The	
Stanford	Encyclopedia	of	Philosophy
	gives	an	extensive	survey	of	different	definitions	and	uses
of	the	word	“consciousness”:	
http://tinyurl.com/stanfordconsciousness
.
4.
Yuval	Noah	Harari,	
Homo	Deus:	A	Brief	History	of	Tomorrow
	(New	York:	HarperCollins,	2017):	116.
5.
This	is	an	excellent	introduction	to	System	1	and	System	2	from	a	pioneer:	Daniel	Kahneman,
Thinking,	Fast	and	Slow
	(New	York:	Farrar,	Straus	&	Giroux,	2011).
6.
See	Christof	Koch,	
The	Quest	for	Consciousness:	A

future
.
3.
The	
.
4.
5.
6.
	
Neurobiological	Approach
	(New	York:	W.	H.
Freeman,	2004).
7.
Perhaps	we’re	only	aware	of	a	tiny	fraction	(say	10–50	bits)	of	the	information	that	enters	our	brain
each	second:	K.	Küpfmüller,	1962,	“Nachrichtenverarbeitung	im	Menschen,”	in	
Taschenbuch	der
Nachrichtenverarbeitung,
	ed.	K.	Steinbuch	(Berlin:	Springer-Verlag,	1962):	1481–1502.	T.
Nørretranders,	
The	User	Illusion:	Cutting	Consciousness	Down	to	Size
	(New	York:	Viking,	1991).
8.
Michio	Kaku,

7.
8.
Michio	Kaku,	
The	Future	of	the	Mind:	The	Scientific	Quest	to	Understand,	Enhance,	and	Empower
the	Mind
	(New	York:	Doubleday,	2014);	Jeff	Hawkins	and	Sandra	Blakeslee,	
On	Intelligence
	(New
York:	Times	Books,	2007);	Stanislas	Dehaene,	Michel	Kerszberg	and	Jean-Pierre	Changeux,	“A
Neuronal	Model	of	a	Global	Workspace	in	Effortful	Cognitive	Tasks,”	
Proceedings	of	the	National
Academy	of
	
Sciences
	95	(1998):	14529–14534.
9.
Video	celebrating	Penfield’s	famous	“I	can	smell	burnt	toast”	experiment:

8.
the	Mind
	(New
Academy	of
	
Sciences
9.
https://www.youtube.com/watch?v=mSN86kphL68
.	Sensorimotor	cortex	details:	Elaine	Marieb	and
Katja	Hoehn,	
Anatomy	&	Physiology,
	3rd	ed.	(Upper	Saddle	River,	NJ:	Pearson,	2008),	391–395.
10.
The	study	of	neural	correlates	of	consciousness	(NCCs)	has	become	quite	mainstream	in	the
neuroscience	community	in	recent	years—see,	e.g.,	Geraint	Rees,	Gabriel	Kreiman,	and	Christof
Koch,	“Neural	Correlates	of	Consciousness	in	Humans,”	
Nature	Reviews
	
Neuroscience
	3	(2002):	261–
270,	and	Thomas	Metzinger,

10.
	
270,	and	Thomas	Metzinger,	
Neural	Correlates	of	Consciousness:	Empirical	and	Conceptual
Questions
	(Cambridge,	MA:	MIT	Press,	2000).
11.
How	continuous	flash	suppression	works:	Christof	Koch,	
	(New	York:	W.	H.	Freeman,	2004);	Christof	Koch	and	Naotsugu	Tsuchiya,
“Continuous	Flash	Suppression	Reduces	Negative	Afterimages,”	
Nature
	
	8	(2005):
1096–1101.
12.
Christof	Koch,	Marcello	Massimini,	Melanie	Boly	and	Giulio	Tononi,	“Neural	Correlates	of

Questions
11.
Nature
	
	8	(2005):
12.
Consciousness:	Progress	and	Problems,”	
Nature	Reviews	Neuroscience
	17	(2016):	307.
13.
See	Koch,	
The	Quest	for
	
Consciousness,
	p.	260,	and	further	discussion	in	the	
Stanford	Encyclopedia
of	Philosophy,
	
http://tinyurl.com/consciousnessdelay
.

14.
On	synchronization	of	conscious	perception:	David	Eagleman,	
The	Brain:	The	Story	of	You
	(New
York:	Pantheon,	2015),	and	
Stanford	Encyclopedia	of	Philosophy,
http://tinyurl.com/consciousnesssync
.
15.
Benjamin	Libet,	
Mind	Time:	The	Temporal	Factor	in	Consciousness
	(Cambridge,	MA:	Harvard
University	Press,	2004);	Chun	Siong	Soon,	Marcel	Brass,	Hans-Jochen	Heinze	and	John-Dylan
Haynes,	“Unconscious	Determinants	of	Free	Decisions	in	the	Human	Brain,”	
Nature	Neuroscience
	11
(2008):	543–545,	online	at	
http://www.nature.com/neuro/journal/v11/n5/full/nn.2112.html
.
16.
Examples	of	recent	theoretical	approaches	to	consciousness:
-	Daniel	Dennett,	
Consciousness	Explained
	(Back	Bay	Books,	1992)
-	Bernard	Baars,	
In	the	Theater	of	Consciousness:	The	Workspace	of	the	Mind
	(New	York:	Oxford
University	Press,	2001)
-	Christof	Koch,	
The	Quest	for	Consciousness:	A	Neurobiological	Approach
Freeman,	2004)
-	Gerald	Edelman	and	Giulio	Tononi,	
A	Universe	of	Consciousness:	How	Matter	Becomes

.
16.
Imagination
	(New	York:	Hachette,	2008)
-	António	Damásio,	
Self	Comes	to	Mind:	Constructing	the	Conscious	Brain
	(New	York:	Vintage,
2012)
-	Stanislas	Dehaene,	
Consciousness	and	the	Brain:	Deciphering	How	the	Brain	Codes	Our	Thoughts
(New	York:	Viking,	2014)
-	Stanislas	Dehaene,	Michel	Kerszberg	and	Jean-Pierre	Changeux,	“A	Neuronal	Model	of	a	Global
Workspace	in	Effortful	Cognitive	Tasks,”	
Proceedings	of	the	National	Academy	of	Sciences
	95
(1998):	14529–14534

2012)
	95
-	Stanislas	Dehaene,	Lucie	Charles,	Jean-Rémi	King	and	Sébastien	Marti,	“Toward	a	Computational
Theory	of	Conscious	Processing,”	
Current	Opinion	in	Neurobiology
	25	(2014):	760–784
17.
Thorough	discussion	of	different	uses	of	the	term	“emergence”	in	physics	and	philosophy	by	David
Chalmers:	
http://cse3521.artifice.cc/Chalmers-Emergence.pdf
.
18.
Me	arguing	that	consciousness	is	the	way	information	feels	when	being	processed	in	certain	complex
ways:	
https://arxiv.org/abs/physics/0510188
,	
https://arxiv.org/abs/0704.0646
,	Max	Tegmark,	
Our

17.
Chalmers:	
.
18.
ways:	
,	
Our
Mathematical	Universe
	(New	York:	Knopf,	2014).	David	Chalmers	expresses	a	related	sentiment	in
his	1996	book	
The	Conscious	Mind:
	“Experience	is	information	from	the	inside;	physics	is
information	from	the	outside.”
19.
Adenauer	Casali	et	al.,	“A	Theoretically	Based	Index	of	Consciousness	Independent	of	Sensory
Processing	and	Behavior,”	
Science	Translational	Medicine
	5	(2013):	198ra105,	online	at
http://tinyurl.com/zapzip
.
20.
Integrated	information	theory	doesn’t	work	for	continuous	systems:
-	
https://arxiv.org/abs/1401.1219
-

19.
.
20.
-	
-	
http://journal.frontiersin.org/article/10.3389/fpsyg.2014.00063/full
-	
https://arxiv.org/abs/1601.02626
21.
Interview	with	Clive	Wearing,	whose	short-term	memory	is	only	about	30	seconds:
https://www.youtube.com/watch?v=WmzU47i2xgw
.
22.
Scott	Aaronson	IIT	critique:	
http://www.scottaaronson.com/blog/?p=1799
.

23.
Cerrullo	IIT	critique,	arguing	that	integration	isn’t	a	sufficient	condition	for	consciousness:
http://tinyurl.com/cerrullocritique
.
24.
IIT	prediction	that	simulated	humans	will	be	zombies:
http://rstb.royalsocietypublishing.org/content/370/1668/20140167
.
25.
Shanahan	critique	of	IIT:	
https://arxiv.org/pdf/1504.05696.pdf
.
26.
Blindsight:	
http://tinyurl.com/blindsight-paper
.
27.
each	second:	Küpfmüller,	“Nachrichtenverarbeitung	im	Menschen”;	Nørretranders,	
The	User	Illusion.
28.
The	case	for	and	against	“consciousness	without	access”:	Victor	Lamme,	“How	Neuroscience	Will
Change	Our	View	on	Consciousness,”	
Cognitive	Neuroscience
	(2010):	204–220,	online	at
http://www.tandfonline.com/doi/abs/10.1080/17588921003731586
.
29.
“Selective	Attention	Test,”	at	
https://www.youtube.com/watch?v=vJG698U2Mvo
.
30.
See	Lamme,	“How	Neuroscience	Will	Change	Our	View	on	Consciousness,”	n.	28.
31.

28.
.
29.
.
30.
31.
This	and	other	related	issues	are	discussed	in	detail	in	Daniel	Dennett’s	book	
Explained
.
32.
See	Kahneman,	
,	cited	in	n.	5.
33.
The	
	reviews	the	free	will	controversy:
https://plato.stanford.edu/entries/freewill
.
34.
Video	of	Seth	Lloyd	explaining	why	an	AI	will	feel	like	it	has	free	will:
https://www.youtube.com/watch?v=Epj3DF8jDWk
.
35.
See	Steven	Weinberg,	
Dreams	of	a	Final	Theory:	The	Search	for	the	Fundamental	Laws	of	Nature
(New	York:	Pantheon,	1992).
36.

Explained
.
32.
33.
The	
.
34.
.
35.
36.
at	
.

Epilogue
1.
The	open	letter	(
http://futureoflife.org/ai-open-letter
)	that	emerged	from	the	Puerto	Rico	conference
argued	that	research	on	how	to	make	AI	systems	robust	and	beneficial	is	both	important	and	timely,
and	that	there	are	concrete	research	directions	that	can	be	pursued	today,	as	exemplified	in	this
research-priorities	document:	
http://futureoflife.org/data/documents/research_priorities.pdf
.
2.
My	video	interview	with	Elon	Musk	about	AI	safety	can	be	found	on	YouTube	at
https://www.youtube.com/watch?v=rBw0eoZTY-g
.
3.
Here’s	a	nice	video	compilation	of	almost	all	SpaceX	rocket	landing	attempts,	culminating	with	the
first	successful	ocean	landing:	
https://www.youtube.com/watch?v=AllaFzIPaG4
.
4.
Elon	Musk	tweets	about	our	AI-safety	grant	competition:
https://twitter.com/elonmusk/status/555743387056226304
.
5.
Elon	Musk	tweets	about	our	AI-safety-endorsing	open	letter:
https://twitter.com/elonmusk/status/554320532133650432
.
6.

.
3.
.
4.
.
5.
.
6.
Erik	Sofge	in	“An	Open	Letter	to	Everyone	Tricked	into	Fearing	Artificial	Intelligence”	(
Popular
Science,
	January	14,	2015)	pokes	fun	at	the	scaremongering	news	coverage	of	our	open	letter:
http://www.popsci.com/open-letter-everyone-tricked-fearing-ai
.
7.
Elon	Musk	tweets	about	his	big	donation	to	the	Future	of	Life	Institute	and	the	world	of	AI-safety
researchers:	
.
8.
For	more	about	the	Partnership	on	AI	to	benefit	people	and	society,	see	their	website:
https://www.partnershiponai.org
.
9.

Popular
Science,
.
7.
.
8.
.
9.
Some	examples	of	recent	reports	expressing	opinions	about	AI:	One	Hundred	Year	Study	on	Artificial
Intelligence,	Report	of	the	2015	Study	Panel,	“Artificial	Intelligence	and	Life	in	2030”	(September
2016),	at	
http://tinyurl.com/stanfordai
;	White	House	report	on	the	future	of	AI:
http://tinyurl.com/obamaAIreport
;	White	House	report	on	AI	and	jobs:
http://tinyurl.com/AIjobsreport
;	IEEE	report	on	AI	and	human	well-being,	“Ethically	Aligned	Design,
Version	1”	(December	13,	2016),	at	
http://standards.ieee.org/develop/indconn/ec/ead_v1.pdf
;	road

2016),	at	
;	road
map	for	U.S.	Robotics:	
http://tinyurl.com/roboticsmap
.
10.
Among	the	principles	that	didn’t	make	the	final	cut,	one	of	my	favorites	was	this	one:	“Consciousness
caution:	There	being	no	consensus,	we	should	avoid	strong	assumptions	as	to	whether	or	not	advanced
AI	would	possess	or	require	consciousness	or	feelings.”	It	went	through	many	iterations,	and	in	the
last	one,	the	controversial	word	“consciousness”	was	replaced	by	“subjective	experience”—but	this
principle	nonetheless	got	only	88%	approval,	just	barely	falling	short	of	the	90%	cutoff.
11.

.
10.
11.
Discussion	panel	on	superintelligence	with	Elon	Musk	and	other	great	minds:
http://tinyurl.com/asilomarAI
.

What’s	next	on
your	reading	list?
Discover	your	next
great	read!
Get	personalized	book	picks	and	up-to-date	news	about	this	author.
Sign	up	now.

SUPERINTELLIGENCE
	
Paths,	Dangers,	Strategies
	
NICK	BOSTROM
Director,	Future	of	Humanity	Institute
Professor,	Faculty	of	Philosophy	&	Oxford	Martin	School
University	of	Oxford

Great	Clarendon	Street,	Oxford,	OX2	6DP,
United	Kingdom	Oxford	University	Press	is	a	department	of	the	University	of
Oxford.
It	furthers	the	University’s	objective	of	excellence	in	research,	scholarship,
and	education	by	publishing	worldwide.	Oxford	is	a	registered	trade	mark	of
Oxford	University	Press	in	the	UK	and	in	certain	other	countries	©	Nick
Bostrom	2014
The	moral	rights	of	the	author	have	been	asserted	First	Edition	published	in	2014
Impression:	1
All	rights	reserved.	No	part	of	this	publication	may	be	reproduced,	stored	in	a
retrieval	system,	or	transmitted,	in	any	form	or	by	any	means,	without	the	prior
permission	in	writing	of	Oxford	University	Press,	or	as	expressly	permitted	by
law,	by	licence	or	under	terms	agreed	with	the	appropriate	reprographics	rights
organization.	Enquiries	concerning	reproduction	outside	the	scope	of	the	above
should	be	sent	to	the	Rights	Department,	Oxford	University	Press,	at	the	address
above	You	must	not	circulate	this	work	in	any	other	form

and	you	must	impose	this	same	condition	on	any	acquirer	British	Library
Cataloguing	in	Publication	Data
Data	available	Library	of	Congress	Control	Number:	2013955152
ISBN	978–0–19–967811–2
Printed	in	Italy	by
L.E.G.O.	S.p.A.—Lavis	TN
Links	to	third	party	websites	are	provided	by	Oxford	in	good	faith	and	for
information	only.	Oxford	disclaims	any	responsibility	for	the	materials	contained
in	any	third	party	website	referenced	in	this	work.

The	Unfinished	Fable	of	the	Sparrows
	
It	was	the	nest-building	season,	but	after	days	of	long	hard	work,	the	sparrows
sat	in	the	evening	glow,	relaxing	and	chirping	away.
“We	are	all	so	small	and	weak.	Imagine	how	easy	life	would	be	if	we	had	an
owl	who	could	help	us	build	our	nests!”
“Yes!”	said	another.	“And	we	could	use	it	to	look	after	our	elderly	and	our
young.”
“It	could	give	us	advice	and	keep	an	eye	out	for	the	neighborhood	cat,”	added
a	third.
Then	Pastus,	the	elder-bird,	spoke:	“Let	us	send	out	scouts	in	all	directions
and	try	to	find	an	abandoned	owlet	somewhere,	or	maybe	an	egg.	A	crow	chick
might	also	do,	or	a	baby	weasel.	This	could	be	the	best	thing	that	ever	happened
to	us,	at	least	since	the	opening	of	the	Pavilion	of	Unlimited	Grain	in	yonder
backyard.”
The	flock	was	exhilarated,	and	sparrows	everywhere	started	chirping	at	the
top	of	their	lungs.
Only	Scronkfinkle,	a	one-eyed	sparrow	with	a	fretful	temperament,	was

a	third.
unconvinced	of	the	wisdom	of	the	endeavor.	Quoth	he:	“This	will	surely	be	our
undoing.	Should	we	not	give	some	thought	to	the	art	of	owl-domestication	and
owl-taming	first,	before	we	bring	such	a	creature	into	our	midst?”
Replied	Pastus:	“Taming	an	owl	sounds	like	an	exceedingly	difficult	thing	to
do.	It	will	be	difficult	enough	to	find	an	owl	egg.	So	let	us	start	there.	After	we
have	succeeded	in	raising	an	owl,	then	we	can	think	about	taking	on	this	other
challenge.”

“There	is	a	flaw	in	that	plan!”	squeaked	Scronkfinkle;	but	his	protests	were	in
vain	as	the	flock	had	already	lifted	off	to	start	implementing	the	directives	set
out	by	Pastus.

Just	two	or	three	sparrows	remained	behind.	Together	they	began	to	try	to
work	out	how	owls	might	be	tamed	or	domesticated.	They	soon	realized	that
Pastus	had	been	right:	this	was	an	exceedingly	difficult	challenge,	especially	in
the	absence	of	an	actual	owl	to	practice	on.	Nevertheless	they	pressed	on	as	best
they	could,	constantly	fearing	that	the	flock	might	return	with	an	owl	egg	before
a	solution	to	the	control	problem	had	been	found.
It	is	not	known	how	the	story	ends,	but	the	author	dedicates	this	book	to
Scronkfinkle	and	his	followers.

PREFACE
	
Inside	your	cranium	is	the	thing	that	does	the	reading.	This	thing,	the	human
brain,	has	some	capabilities	that	the	brains	of	other	animals	lack.	It	is	to	these
distinctive	capabilities	that	we	owe	our	dominant	position	on	the	planet.	Other
animals	have	stronger	muscles	and	sharper	claws,	but	we	have	cleverer	brains.
Our	modest	advantage	in	general	intelligence	has	led	us	to	develop	language,
technology,	and	complex	social	organization.	The	advantage	has	compounded
over	time,	as	each	generation	has	built	on	the	achievements	of	its	predecessors.
If	some	day	we	build	machine	brains	that	surpass	human	brains	in	general
intelligence,	then	this	new	superintelligence	could	become	very	powerful.	And,
as	the	fate	of	the	gorillas	now	depends	more	on	us	humans	than	on	the	gorillas
themselves,	so	the	fate	of	our	species	would	depend	on	the	actions	of	the
machine	superintelligence.
We	do	have	one	advantage:	we	get	to	build	the	stuff.	In	principle,	we	could

build	a	kind	of	superintelligence	that	would	protect	human	values.	We	would
certainly	have	strong	reason	to	do	so.	In	practice,	the	control	problem—the
problem	of	how	to	control	what	the	superintelligence	would	do—looks	quite
difficult.	It	also	looks	like	we	will	only	get	one	chance.	Once	unfriendly
superintelligence	exists,	it	would	prevent	us	from	replacing	it	or	changing	its
preferences.	Our	fate	would	be	sealed.
In	this	book,	I	try	to	understand	the	challenge	presented	by	the	prospect	of

superintelligence,	and	how	we	might	best	respond.	This	is	quite	possibly	the
most	important	and	most	daunting	challenge	humanity	has	ever	faced.	And—
whether	we	succeed	or	fail—it	is	probably	the	last	challenge	we	will	ever	face.
It	is	no	part	of	the	argument	in	this	book	that	we	are	on	the	threshold	of	a	big
breakthrough	in	artificial	intelligence,	or	that	we	can	predict	with	any	precision
when	such	a	development	might	occur.	It	seems	somewhat	likely	that	it	will

happen	sometime	in	this	century,	but	we	don’t	know	for	sure.	The	first	couple	of
chapters	do	discuss	possible	pathways	and	say	something	about	the	question	of

timing.	The	bulk	of	the	book,	however,	is	about	what	happens	after.	We	study	the
kinetics	of	an	intelligence	explosion,	the	forms	and	powers	of	superintelligence,
and	the	strategic	choices	available	to	a	superintelligent	agent	that	attains	a
decisive	advantage.	We	then	shift	our	focus	to	the	control	problem	and	ask	what
we	could	do	to	
shape	the	initial	conditions	so	as	to	achieve	a	survivable	and
beneficial	outcome.	Toward	the	end	of	the	book,	we	zoom	out	and	contemplate
the	larger	picture	that	emerges	from	our	investigations.	Some	suggestions	are
offered	on	what	ought	to	be	done	now	to	increase	our	chances	of	avoiding	an
existential	catastrophe	later.
This	has	not	been	an	easy	book	to	write.	I	hope	the	path	that	has	been	cleared
will	enable	other	investigators	to	reach	the	new	frontier	more	swiftly	and
conveniently,	so	that	they	can	arrive	there	fresh	and	ready	to	join	the	work	to
further	expand	the	reach	of	our	comprehension.	(And	if	the	way	that	has	been

made	is	a	little	bumpy	and	bendy,	I	hope	that	reviewers,	in	judging	the	result,
will	not	underestimate	the	hostility	of	the	terrain	
ex	ante
!)
This	has	not	been	an	easy	book	to	write:	I	have	tried	to	make	it	an	easy	book
to	read,	but	I	don’t	think	I	have	quite	succeeded.	When	writing,	I	had	in	mind	as
the	target	audience	an	earlier	time-slice	of	myself,	and	I	tried	to	produce	a	kind
of	book	that	I	would	have	enjoyed	reading.	This	could	prove	a	narrow

ex	ante
!)
demographic.	Nevertheless,	I	think	that	the	content	should	be	accessible	to	many
people,	if	they	put	some	thought	into	it	and	resist	the	temptation	to
instantaneously	misunderstand	each	new	idea	by	assimilating	it	with	the	most
similar-sounding	cliché	available	in	their	cultural	larders.	Non-technical	readers
should	not	be	discouraged	by	the	occasional	bit	of	mathematics	or	specialized
vocabulary,	for	it	is	always	possible	to	glean	the	main	point	from	the
surrounding	explanations.	(Conversely,	for	those	readers	who	want	more	of	the

nitty-gritty,	there	is	quite	a	lot	to	be	found	among	the	endnotes.
1
)
Many	of	the	points	made	in	this	book	are	probably	wrong.
2
	It	is	also	likely
that	there	are	considerations	of	critical	importance	that	I	fail	to	take	into	account,
thereby	invalidating	some	or	all	of	my	conclusions.	I	have	gone	to	some	length
to	indicate	nuances	and	degrees	of	uncertainty	throughout	the	text—
encumbering	it	with	an	unsightly	smudge	of	“possibly,”	“might,”	“may,”	“could
well,”	“it	seems,”	“probably,”	“very	likely,”	“almost	certainly.”	Each	qualifier

1
)
2
has	been	placed	where	it	is	carefully	and	deliberately.	Yet	these	topical
applications	of	epistemic	modesty	are	not	enough;	they	must	be	supplemented
here	by	a	systemic	admission	of	uncertainty	and	fallibility.	This	is	not	false

modesty:	for	while	I	believe	that	my	book	is	likely	to	be	seriously	wrong	and
misleading,	I	think	that	the	alternative	views	that	have	been	presented	in	the
literature	are	substantially	worse—including	the	default	view,	or	“null
hypothesis,”	according	to	which	we	can	for	the	time	being	safely	or	reasonably
ignore	the	prospect	of	superintelligence.

ACKNOWLEDGMENTS
	
The	membrane	that	has	surrounded	the	writing	process	has	been	fairly
permeable.	Many	concepts	and	ideas	generated	while	working	on	the	book	have
been	allowed	to	seep	out	and	have	become	part	of	a	wider	conversation;	and,	of
course,	numerous	insights	originating	from	the	outside	while	the	book	was
underway	have	been	incorporated	into	the	text.	I	have	tried	to	be	somewhat
diligent	with	the	citation	apparatus,	but	the	influences	are	too	many	to	fully
document.
For	extensive	discussions	that	have	helped	clarify	my	thinking	I	am	grateful	to
a	large	set	of	people,	including	Ross	Andersen,	Stuart	Armstrong,	Owen	Cotton-
Barratt,	Nick	Beckstead,	David	Chalmers,	Paul	Christiano,	Milan	Ćirković,
Daniel	Dennett,	David	Deutsch,	Daniel	Dewey,	Eric	Drexler,	Peter	Eckersley,
Amnon	Eden,	Owain	Evans,	Benja	Fallenstein,	Alex	Flint,	Carl	Frey,	Ian	Goldin,
Katja	Grace,	J.	Storrs	Hall,	Robin	Hanson,	Demis	Hassabis,	James	Hughes,
Marcus	Hutter,	Garry	Kasparov,	Marcin	Kulczycki,	Shane	Legg,	Moshe	Looks,

Willam	MacAskill,	Eric	Mandelbaum,	James	Martin,	Lillian	Martin,	Roko	Mijic,
Vincent	Mueller,	Elon	Musk,	Seán	Ó	hÉigeartaigh,	Toby	Ord,	Dennis	Pamlin,
Derek	Parfit,	David	Pearce,	Huw	Price,	Martin	Rees,	Bill	Roscoe,	Stuart	Russell,
Anna	Salamon,	Lou	Salkind,	Anders	Sandberg,	Julian	Savulescu,	Jürgen
Schmidhuber,	Nicholas	Shackel,	Murray	Shanahan,	Noel	Sharkey,	Carl
Shulman,	Peter	Singer,	Dan	Stoicescu,	Jaan	Tallinn,	Alexander	Tamas,	Max
Tegmark,	Roman	Yampolskiy,	and	Eliezer	Yudkowsky.
For	especially	detailed	comments,	I	am	grateful	to	Milan	Ćirković,	Daniel

Dewey,	Owain	Evans,	Nick	Hay,	Keith	Mansfield,	Luke	Muehlhauser,	Toby
Ord,	Jess	Riedel,	Anders	Sandberg,	Murray	Shanahan,	and	Carl	Shulman.	For
advice	or	research	help	with	different	parts	I	want	to	thank	Stuart	Armstrong,
Daniel	Dewey,	Eric	Drexler,	Alexandre	Erler,	Rebecca	Roache,	and	Anders
Sandberg.
For	help	with	preparing	the	manuscript,	I	am	thankful	to	Caleb	Bell,	Malo
Bourgon,	Robin	Brandt,	Lance	Bush,	Cathy	Douglass,	Alexandre	Erler,	Kristian

Rönn,	Susan	Rogers,	Andrew	Snyder-Beattie,	Cecilia	Tilli,	and	Alex	Vermeer.	I
want	particularly	to	thank	my	editor	Keith	Mansfield	for	his	plentiful
encouragement	throughout	the	project.
My	apologies	to	everybody	else	who	ought	to	have	been	remembered	here.
Finally,	a	most	fond	thank	you	to	funders,	friends,	and	family:	without	your
backing,	this	work	would	not	have	been	done.

CONTENTS
	
Lists	of	Figures,	Tables,	and	Boxes
1.	Past	developments	and	present	capabilities
Growth	modes	and	big	history
Great	expectations
Seasons	of	hope	and	despair
State	of	the	art
Opinions	about	the	future	of	machine	intelligence
2.	Paths	to	superintelligence
Artificial	intelligence
Whole	brain	emulation
Biological	cognition
Brain–computer	interfaces
Networks	and	organizations
Summary
3.	Forms	of	superintelligence
Speed	superintelligence
Collective	superintelligence

Quality	superintelligence
Direct	and	indirect	reach
Sources	of	advantage	for	digital	intelligence
4.	The	kinetics	of	an	intelligence	explosion
Timing	and	speed	of	the	takeoff
Recalcitrance
Non-machine	intelligence	paths
Emulation	and	AI	paths
Optimization	power	and	explosivity
5.	Decisive	strategic	advantage
Will	the	frontrunner	get	a	decisive	strategic	advantage?
How	large	will	the	successful	project	be?
Monitoring
International	collaboration
From	decisive	strategic	advantage	to	singleton
6.	Cognitive	superpowers
Functionalities	and	superpowers
An	AI	takeover	scenario
Power	over	nature	and	agents
7.	The	superintelligent	will
The	relation	between	intelligence	and	motivation

Instrumental	convergence
Self-preservation
Goal-content	integrity
Cognitive	enhancement
Technological	perfection
Resource	acquisition
8.	Is	the	default	outcome	doom?
Existential	catastrophe	as	the	default	outcome	of	an	intelligence	explosion?
The	treacherous	turn
Malignant	failure	modes
Perverse	instantiation
Infrastructure	profusion
Mind	crime
9.	The	control	problem
Two	agency	problems
Capability	control	methods
Boxing	methods
Incentive	methods
Stunting
Tripwires
Motivation	selection	methods

Direct	specification
Domesticity
Indirect	normativity
Augmentation
Synopsis
10.	Oracles,	genies,	sovereigns,	tools
	
Oracles
Genies	and	sovereigns
Tool-AIs
Comparison
11.	Multipolar	scenarios
	
Of	horses	and	men
Wages	and	unemployment
Capital	and	welfare
The	Malthusian	principle	in	a	historical	perspective
Population	growth	and	investment
Life	in	an	algorithmic	economy
Voluntary	slavery,	casual	death
Would	maximally	efficient	work	be	fun?

Unconscious	outsourcers?
Evolution	is	not	necessarily	up
Post-transition	formation	of	a	singleton?
A	second	transition
Superorganisms	and	scale	economies
Unification	by	treaty
12.	Acquiring	values
	
The	value-loading	problem
Evolutionary	selection
Reinforcement	learning
Associative	value	accretion
Motivational	scaffolding
Value	learning
Emulation	modulation
Institution	design
Synopsis
13.	Choosing	the	criteria	for	choosing
	
The	need	for	indirect	normativity
Coherent	extrapolated	volition

Some	explications
Rationales	for	CEV
Further	remarks
Morality	models
Do	What	I	Mean
Component	list
Goal	content
Decision	theory
Epistemology
Ratification
Getting	close	enough
14.	The	strategic	picture
	
Science	and	technology	strategy
Differential	technological	development
Preferred	order	of	arrival
Rates	of	change	and	cognitive	enhancement
Technology	couplings
Second-guessing
Pathways	and	enablers
Effects	of	hardware	progress

Should	whole	brain	emulation	research	be	promoted?
The	person-affecting	perspective	favors	speed
Collaboration
The	race	dynamic	and	its	perils
On	the	benefits	of	collaboration
Working	together
15.	Crunch	time
	
Philosophy	with	a	deadline
What	is	to	be	done?
Seeking	the	strategic	light
Building	good	capacity
Particular	measures
Will	the	best	in	human	nature	please	stand	up
Bibliography

LISTS	OF	FIGURES,	TABLES,	AND	BOXES
	
List	of	Figures
	
1.	Long-term	history	of	world	GDP
.
2.	Overall	long-term	impact	of	HLMI
.
3.	Supercomputer	performance
.
4.	Reconstructing	3D	neuroanatomy	from	electron	microscope	images
.
5.	Whole	brain	emulation	roadmap
.
6.	Composite	faces	as	a	metaphor	for	spell-checked	genomes
.
7.	Shape	of	the	takeoff
.
8.	A	less	anthropomorphic	scale?
9.	One	simple	model	of	an	intelligence	explosion
.
10.	Phases	in	an	AI	takeover	scenario
.
11.	Schematic	illustration	of	some	possible	trajectories	for	a	hypothetical	wise
singleton
.
12.	Results	of	anthropomorphizing	alien	motivation
.
13.	Artificial	intelligence	or	whole	brain	emulation	first?
14.	Risk	levels	in	AI	technology	races
.

List	of	Tables
	
1.	Game-playing	AI
2.	When	will	human-level	machine	intelligence	be	attained?
3.	How	long	from	human	level	to	superintelligence?
4.	Capabilities	needed	for	whole	brain	emulation
5.	Maximum	IQ	gains	from	selecting	among	a	set	of	embryos
6.	Possible	impacts	from	genetic	selection	in	different	scenarios
7.	Some	strategically	significant	technology	races
8.	Superpowers:	some	strategically	relevant	tasks	and	corresponding	skill	sets
9.	Different	kinds	of	tripwires
10.	Control	methods
11.	Features	of	different	system	castes
12.	Summary	of	value-loading	techniques
13.	Component	list
List	of	Boxes
	
1.	An	optimal	Bayesian	agent
2.	The	2010	Flash	Crash
3.	What	would	it	take	to	recapitulate	evolution?

4.	On	the	kinetics	of	an	intelligence	explosion
5.	Technology	races:	some	historical	examples
6.	The	mail-ordered	DNA	scenario
7.	How	big	is	the	cosmic	endowment?
8.	Anthropic	capture
9.	Strange	solutions	from	blind	search
10.	Formalizing	value	learning
11.	An	AI	that	wants	to	be	friendly
12.	Two	recent	(half-baked)	ideas
13.	A	risk-race	to	the	bottom

CHAPTER	1
Past	developments	and	present	capabilities
	
We	begin	by	looking	back.	History,	at	the	largest	scale,	seems	to	exhibit	a
sequence	of	distinct	growth	modes,	each	much	more	rapid	than	its
predecessor.	This	pattern	has	been	taken	to	suggest	that	another	(even
faster)	growth	mode	might	be	possible.	However,	we	do	not	place	much
weight	on	this	observation—this	is	not	a	book	about	“technological
acceleration”	or	“exponential	growth”	or	the	miscellaneous	notions
sometimes	gathered	under	the	rubric	of	“the	singularity.”	Next,	we	review
the	history	of	artificial	intelligence.	We	then	survey	the	field’s	current
capabilities.	Finally,	we	glance	at	some	recent	expert	opinion	surveys,	and
contemplate	our	ignorance	about	the	timeline	of	future	advances.
	
A	mere	few	million	years	ago	our	ancestors	were	still	swinging	from	the
branches	in	the	African	canopy.	On	a	geological	or	even	evolutionary	timescale,
the	rise	of	
	from	our	last	common	ancestor	with	the	great	apes

the	rise	of	
happened	swiftly.	We	developed	upright	posture,	opposable	thumbs,	and—
crucially—some	relatively	minor	changes	in	brain	size	and	neurological
organization	that	led	to	a	great	leap	in	cognitive	ability.	As	a	consequence,
humans	can	think	abstractly,	communicate	complex	thoughts,	and	culturally
accumulate	information	over	the	generations	far	better	than	any	other	species	on
the	planet.
These	capabilities	let	humans	develop	increasingly	efficient	productive
technologies,	making	it	possible	for	our	ancestors	to	migrate	far	away	from	the

rainforest	and	the	savanna.	Especially	after	the	adoption	of	agriculture,
population	densities	rose	along	with	the	total	size	of	the	human	population.	More
people	meant	more	ideas;	greater	densities	meant	that	ideas	could	spread	more
readily	and	that	some	individuals	could	devote	themselves	to	developing

specialized	skills.	These	
developments	increased	the	
rate	of	growth
	of	economic
productivity	and	technological	capacity.	Later	developments,	related	to	the
Industrial	Revolution,	brought	about	a	second,	comparable	step	change	in	the
rate	of	growth.
Such	changes	in	the	rate	of	growth	have	important	consequences.	A	few
hundred	thousand	years	ago,	in	early	human	(or	hominid)	prehistory,	growth	was
so	slow	that	it	took	on	the	order	of	one	million	years	for	human	productive
capacity	to	increase	sufficiently	to	sustain	an	additional	one	million	individuals
living	at	subsistence	level.	By	5000	
BC
,	following	the	Agricultural	Revolution,
the	rate	of	growth	had	increased	to	the	point	where	the	same	amount	of	growth
took	just	two	centuries.	Today,	following	the	Industrial	Revolution,	the	world
economy	grows	on	average	by	that	amount	every	ninety	minutes.
1
Even	the	present	rate	of	growth	will	produce	impressive	results	if	maintained
for	a	moderately	long	time.	If	the	world	economy	continues	to	grow	at	the	same

BC
1
pace	as	it	has	over	the	past	fifty	years,	then	the	world	will	be	some	4.8	times
richer	by	2050	and	about	34	times	richer	by	2100	than	it	is	today.
2
Yet	the	prospect	of	continuing	on	a	steady	exponential	growth	path	pales	in
comparison	to	what	would	happen	if	the	world	were	to	experience	another	step
change	in	the	
	comparable	in	magnitude	to	those	associated	with
the	Agricultural	Revolution	and	the	Industrial	Revolution.	The	economist	Robin
Hanson	estimates,	based	on	historical	economic	and	population	data,	a

2
characteristic	world	economy	doubling	time	for	Pleistocene	hunter–gatherer
society	of	224,000	years;	for	farming	society,	909	years;	and	for	industrial
society,	6.3	years.
3
	(In	Hanson’s	model,	the	present	epoch	is	a	mixture	of	the
farming	and	the	industrial	growth	modes—the	world	economy	as	a	whole	is	not
yet	growing	at	the	6.3-year	doubling	rate.)	If	another	such	transition	to	a
different	growth	mode	were	to	occur,	and	it	were	of	similar	magnitude	to	the
previous	two,	it	would	result	in	a	new	growth	regime	in	which	the	world

3
economy	would	double	in	size	about	every	two	weeks.
Such	a	growth	rate	seems	fantastic	by	current	lights.	Observers	in	earlier
epochs	might	have	found	it	equally	preposterous	to	suppose	that	the	world
economy	would	one	day	be	doubling	several	times	within	a	single	lifespan.	Yet
that	is	the	extraordinary	condition	we	now	take	to	be	ordinary.
The	idea	of	a	coming	technological	singularity	has	by	now	been	widely

popularized,	starting	with	Vernor	Vinge’s	seminal	essay	and	continuing	with	the
writings	of	Ray	Kurzweil	and	others.
4
	The	term	“singularity,”	however,	has	been
used	confusedly	in	many	disparate	senses	and	has	accreted	an	unholy	(yet	almost
millenarian)	aura	of	techno-utopian	connotations.
5
	Since	most	of	these	meanings
and	connotations	are	irrelevant	to	our	argument,	we	can	gain	clarity	by
dispensing	with	the	“singularity”	word	in	favor	of	more	precise	terminology.
The	singularity-related	idea	that	interests	us	here	is	the	possibility	of	an
intelligence	explosion
,	particularly	the	prospect	of	machine	superintelligence.
There	may	be	those	who	are	persuaded	by	growth	diagrams	like	the	ones	in
Figure	1
	that	another	drastic	change	in	growth	mode	is	in	the	cards,	comparable
to	the	Agricultural	or	Industrial	Revolution.	These	folk	may	then	reflect	that	it	is
hard	
to	conceive	of	a	scenario	in	which	the	world	economy’s	doubling	time

Figure	1
hard	
shortens	to	mere	weeks	that	does	not	involve	the	creation	of	minds	that	are	much
faster	and	more	efficient	than	the	familiar	biological	kind.	However,	the	case	for
taking	seriously	the	prospect	of	a	machine	intelligence	revolution	need	not	rely
on	curve-fitting	exercises	or	extrapolations	from	past	economic	growth.	As	we
shall	see,	there	are	stronger	reasons	for	taking	heed.

Figure	1
	Long-term	history	of	world	GDP.	Plotted	on	a	linear	scale,	the	history
of	the	world	economy	looks	like	a	flat	line	hugging	the	
x
-axis,	until	it	suddenly
spikes	vertically	upward.	(a)	Even	when	we	zoom	in	on	the	most	recent	10,000
years,	the	pattern	remains	essentially	one	of	a	single	90°	angle.	(b)	Only	within
the	past	100	years	or	so	does	the	curve	lift	perceptibly	above	the	zero-level.	(The
different	lines	in	the	plot	correspond	to	different	data	sets,	which	yield	slightly
different	estimates.
6
)
	
Machines	matching	humans	in	general	intelligence—that	is,	possessing	common
sense	and	an	effective	ability	to	learn,	reason,	and	plan	to	meet	complex
information-processing	challenges	across	a	wide	range	of	natural	and	abstract
domains—have	been	expected	since	the	invention	of	computers	in	the	1940s.	At
that	time,	the	advent	of	such	machines	was	often	placed	some	twenty	years	into
the	future.
7
	Since	then,	the	expected	arrival	date	has	been	receding	at	a	rate	of
one	year	per	year;	so	that	today,	futurists	who	concern	themselves	with	the
possibility	of	artificial	general	intelligence	still	often	believe	that	intelligent
machines	are	a	couple	of	decades	away.
8
Two	decades	is	a	sweet	spot	for	prognosticators	of	radical	change:	near
enough	to	be	attention-grabbing	and	relevant,	yet	far	enough	to	make	it	possible
to	suppose	that	a	string	of	breakthroughs,	currently	only	vaguely	imaginable,
might	by	then	have	occurred.	Contrast	this	with	shorter	timescales:	most

8
technologies	that	will	have	a	big	impact	on	the	world	in	five	or	ten	years	from
now	are	already	in	limited	use,	while	technologies	that	will	reshape	the	world	in
less	than	fifteen	years	probably	exist	as	laboratory	prototypes.	Twenty	years	may
also	be	close	to	the	typical	duration	remaining	of	a	forecaster’s	career,	bounding
the	reputational	risk	of	a	bold	prediction.
From	the	fact	that	some	individuals	have	overpredicted	artificial	intelligence

in	the	past,	however,	it	does	not	follow	that	AI	is	impossible	or	will	never	be
developed.
9
	The	main	reason	why	progress	has	been	slower	than	expected	is	that
the	technical	difficulties	of	constructing	intelligent	machines	have	proved	greater
than	the	pioneers	foresaw.	But	this	leaves	open	just	how	great	those	difficulties
are	and	how	far	we	now	are	from	overcoming	them.	Sometimes	a	problem	that
initially	looks	hopelessly	complicated	turns	out	to	have	a	surprisingly	simple
solution	(though	the	reverse	is	probably	more	common).

9
In	the	next	chapter,	we	will	look	at	different	paths	that	may	lead	to	human-
level	machine	intelligence.	But	let	us	note	at	the	outset	that	however	many	stops
there	are	between	here	and	human-level	machine	intelligence,	the	latter	is	not	the
final	destination.	The	next	stop,	just	a	short	distance	farther	along	the	tracks,	is
superhuman-level	machine	intelligence.	The	train	might	not	pause	or	even
decelerate	at	Humanville	Station.	It	is	likely	to	swoosh	right	by.
The	mathematician	I.	J.	Good,	who	had	served	as	chief	statistician	in	Alan

Turing’s	code-breaking	team	in	World	War	II,	might	have	been	the	first	to
enunciate	the	essential	aspects	of	this	scenario.	In	an	oft-quoted	passage	from

1965,	he	wrote:
Let	an	ultraintelligent	machine	be	defined	as	a	machine	that	can	far	surpass
all	the	intellectual	activities	of	any	man	however	clever.	Since	the	design	of
machines	is	one	of	these	intellectual	activities,	an	ultraintelligent	machine
“intelligence	explosion,”	and	the	intelligence	of	man	would	be	left	far
behind.	Thus	the	first	ultraintelligent	machine	is	the	last	invention	that	man
need	ever	make,	provided	that	the	machine	is	docile	enough	to	tell	us	how
to	keep	it	under	control.
10
	
It	may	seem	obvious	now	that	major	existential	risks	would	be	associated	with
such	an	intelligence	explosion,	and	that	the	prospect	should	therefore	be
examined	with	the	utmost	seriousness	even	if	it	were	known	(which	it	is	not)	to
have	but	a	moderately	small	probability	of	coming	to	pass.	The	pioneers	of
artificial	intelligence,	however,	notwithstanding	their	belief	in	the	imminence	of

10
	
human-level	
AI,	mostly	did	not	contemplate	the	possibility	of	greater-than-
human	AI.	It	is	as	though	their	speculation	muscle	had	so	exhausted	itself	in
conceiving	the	radical	possibility	of	machines	reaching	human	intelligence	that	it
could	not	grasp	the	corollary—that	machines	would	subsequently	become
The	AI	pioneers	for	the	most	part	did	not	countenance	the	possibility	that	their
enterprise	might	involve	risk.
11
	They	gave	no	lip	service—let	alone	serious

11
thought—to	any	safety	concern	or	ethical	qualm	related	to	the	creation	of
artificial	minds	and	potential	computer	overlords:	a	lacuna	that	astonishes	even
against	the	background	of	the	era’s	not-so-impressive	standards	of	critical
technology	assessment.
12
	We	must	hope	that	by	the	time	the	enterprise
eventually	does	become	feasible,	we	will	have	gained	not	only	the	technological
proficiency	to	set	off	an	intelligence	explosion	but	also	the	higher	level	of
mastery	that	may	be	necessary	to	make	the	detonation	survivable.

12
But	before	we	turn	to	what	lies	ahead,	it	will	be	useful	to	take	a	quick	glance
at	the	history	of	machine	intelligence	to	date.

In	the	summer	of	1956	at	Dartmouth	College,	ten	scientists	sharing	an	interest	in
neural	nets,	automata	theory,	and	the	study	of	intelligence	convened	for	a	six-
week	workshop.	This	Dartmouth	Summer	Project	is	often	regarded	as	the
cockcrow	of	artificial	intelligence	as	a	field	of	research.	Many	of	the	participants
would	later	be	recognized	as	founding	figures.	The	optimistic	outlook	among	the
delegates	is	reflected	in	the	proposal	submitted	to	the	Rockefeller	Foundation,
which	provided	funding	for	the	event:
We	propose	that	a	2	month,	10	man	study	of	artificial	intelligence	be	carried
out….	The	study	is	to	proceed	on	the	basis	of	the	conjecture	that	every
aspect	of	learning	or	any	other	feature	of	intelligence	can	in	principle	be	so
precisely	described	that	a	machine	can	be	made	to	simulate	it.	An	attempt
will	be	made	to	find	how	to	make	machines	that	use	language,	form
abstractions	and	concepts,	solve	kinds	of	problems	now	reserved	for

humans,	and	improve	themselves.	We	think	that	a	significant	advance	can
be	made	in	one	or	more	of	these	problems	if	a	carefully	selected	group	of
scientists	work	on	it	together	for	a	summer.
	
In	the	six	decades	since	this	brash	beginning,	the	field	of	artificial	intelligence
has	been	through	periods	of	hype	and	high	expectations	alternating	with	periods
of	setback	and	disappointment.
The	first	period	of	excitement,	which	began	with	the	Dartmouth	meeting,	was
later	described	by	John	McCarthy	(the	event’s	main	organizer)	as	the	“Look,	Ma,

no	hands!”	era.	During	these	early	days,	researchers	built	systems	designed	to
refute	claims	of	the	form	“No	machine	could	ever	do	
X
!”	Such	skeptical	claims
were	common	at	the	time.	To	counter	them,	the	AI	researchers	created	small
systems	that	achieved	
X
	in	a	“microworld”	(a	well-defined,	limited	domain	that
enabled	
a	pared-down	version	of	the	performance	to	be	demonstrated),	thus
providing	a	proof	of	concept	and	showing	that	
X
	could,	in	principle,	be	done	by
machine.	One	such	early	system,	the	Logic	Theorist,	was	able	to	prove	most	of

X
X
enabled	
X
the	theorems	in	the	second	chapter	of	Whitehead	and	Russell’s	
Principia
Mathematica
,	and	even	came	up	with	one	proof	that	was	much	more	elegant
than	the	original,	thereby	debunking	the	notion	that	machines	could	“only	think
numerically”	and	showing	that	machines	were	also	able	to	do	deduction	and	to

invent	logical	proofs.
13
	A	follow-up	program,	the	General	Problem	Solver,	could
in	principle	solve	a	wide	range	of	formally	specified	problems.
14
	Programs	that
could	solve	calculus	problems	typical	of	first-year	college	courses,	visual
analogy	problems	of	the	type	that	appear	in	some	IQ	tests,	and	simple	verbal
algebra	problems	were	also	written.
15
	The	Shakey	robot	(so	named	because	of	its
tendency	to	tremble	during	operation)	demonstrated	how	logical	reasoning	could
be	integrated	with	perception	and	used	to	plan	and	control	physical	activity.
16
The	ELIZA	program	showed	how	a	computer	could	impersonate	a	Rogerian
psychotherapist.
17
	In	the	mid-seventies,	the	program	SHRDLU	showed	how	a
simulated	robotic	arm	in	a	simulated	world	of	geometric	blocks	could	follow
instructions	and	answer	questions	in	English	that	were	typed	in	by	a	user.
18
	In
later	decades,	systems	would	be	created	that	demonstrated	that	machines	could
compose	music	in	the	style	of	various	classical	composers,	outperform	junior

16
17
18
	In
doctors	in	certain	clinical	diagnostic	tasks,	drive	cars	autonomously,	and	make
patentable	inventions.
19
	There	has	even	been	an	AI	that	cracked	original	jokes.
20
(Not	that	its	level	of	humor	was	high—“What	do	you	get	when	you	cross	an
optic
	with	a	
mental	object
?	An	
eye
-dea”—but	children	reportedly	found	its	puns
consistently	entertaining.)
The	methods	that	produced	successes	in	the	early	demonstration	systems	often
proved	difficult	to	extend	to	a	wider	variety	of	problems	or	to	harder	problem

19
20
optic
	with	a	
?	An	
eye
instances.	One	reason	for	this	is	the	“combinatorial	explosion”	of	possibilities
that	must	be	explored	by	methods	that	rely	on	something	like	exhaustive	search.
Such	methods	work	well	for	simple	instances	of	a	problem,	but	fail	when	things
get	a	bit	more	complicated.	For	instance,	to	prove	a	theorem	that	has	a	5-line
long	proof	in	a	deduction	system	with	one	inference	rule	and	5	axioms,	one
could	simply	enumerate	the	3,125	possible	combinations	and	check	each	one	to

see	if	it	delivers	the	intended	conclusion.	Exhaustive	search	would	also	work	for
6-and	7-line	proofs.	But	as	the	task	becomes	more	difficult,	the	method	of
exhaustive	search	soon	runs	into	trouble.	Proving	a	theorem	with	a	50-line	proof
does	not	take	ten	times	longer	than	proving	a	theorem	that	has	a	5-line	proof:
rather,	if	one	uses	exhaustive	search,	it	requires	combing	through	5
50
	≈	8.9	×
10
34
	possible	sequences—which	is	computationally	infeasible	even	with	the
fastest	supercomputers.

50
	≈	8.9	×
10
34
To	overcome	the	combinatorial	explosion,	one	needs	algorithms	that	exploit
structure	in	the	target	domain	and	take	advantage	of	prior	knowledge	by	using
heuristic	search,	planning,	and	flexible	abstract	representations—capabilities	that

were	poorly	developed	in	the	early	AI	systems.	The	performance	of	these	early
systems	also	suffered	because	of	poor	methods	for	handling	uncertainty,	reliance
on	brittle	and	ungrounded	symbolic	representations,	data	scarcity,	and	
severe
hardware	limitations	on	memory	capacity	and	processor	speed.	By	the	mid-
1970s,	there	was	a	growing	awareness	of	these	problems.	The	realization	that
many	AI	projects	could	never	make	good	on	their	initial	promises	led	to	the
onset	of	the	first	“AI	winter”:	a	period	of	retrenchment,	during	which	funding
decreased	and	skepticism	increased,	and	AI	fell	out	of	fashion.
A	new	springtime	arrived	in	the	early	1980s,	when	Japan	launched	its	Fifth-
Generation	Computer	Systems	Project,	a	well-funded	public–private	partnership
that	aimed	to	leapfrog	the	state	of	the	art	by	developing	a	massively	parallel
computing	architecture	that	would	serve	as	a	platform	for	artificial	intelligence.
This	occurred	at	peak	fascination	with	the	Japanese	“post-war	economic

miracle,”	a	period	when	Western	government	and	business	leaders	anxiously
sought	to	divine	the	formula	behind	Japan’s	economic	success	in	hope	of
replicating	the	magic	at	home.	When	Japan	decided	to	invest	big	in	AI,	several
other	countries	followed	suit.
The	ensuing	years	saw	a	great	proliferation	of	
expert	systems
.	Designed	as
support	tools	for	decision	makers,	expert	systems	were	rule-based	programs	that
made	simple	inferences	from	a	knowledge	base	of	facts,	which	had	been	elicited
from	human	domain	experts	and	painstakingly	hand-coded	in	a	formal	language.

Hundreds	of	these	expert	systems	were	built.	However,	the	smaller	systems
provided	little	benefit,	and	the	larger	ones	proved	expensive	to	develop,	validate,
and	keep	updated,	and	were	generally	cumbersome	to	use.	It	was	impractical	to
acquire	a	standalone	computer	just	for	the	sake	of	running	one	program.	By	the
late	1980s,	this	growth	season,	too,	had	run	its	course.
The	Fifth-Generation	Project	failed	to	meet	its	objectives,	as	did	its
counterparts	in	the	United	States	and	Europe.	A	second	AI	winter	descended.	At

this	point,	a	critic	could	justifiably	bemoan	“the	history	of	artificial	intelligence
research	to	date,	consisting	always	of	very	limited	success	in	particular	areas,
followed	immediately	by	failure	to	reach	the	broader	goals	at	which	these	initial
successes	seem	at	first	to	hint.”
21
	Private	investors	began	to	shun	any	venture
carrying	the	brand	of	“artificial	intelligence.”	Even	among	academics	and	their
funders,	“AI”	became	an	unwanted	epithet.
22
Technical	work	continued	apace,	however,	and	by	the	1990s,	the	second	AI

winter	gradually	thawed.	Optimism	was	rekindled	by	the	introduction	of	new
techniques,	which	seemed	to	offer	alternatives	to	the	traditional	logicist
paradigm	(often	referred	to	as	“Good	Old-Fashioned	Artificial	Intelligence,”	or
“GOFAI”	for	short),	which	had	focused	on	high-level	symbol	manipulation	and
which	had	reached	its	apogee	in	the	expert	systems	of	the	1980s.	The	newly
popular	techniques,	which	included	neural	networks	and	genetic	algorithms,
promised	to	overcome	some	of	the	shortcomings	of	the	GOFAI	approach,	in
particular	the	“brittleness”	that	characterized	classical	AI	programs	(which
typically	produced	complete	nonsense	if	the	programmers	made	even	a	single
slightly	erroneous	assumption).	The	new	techniques	boasted	a	more	organic
performance.	For	example,	neural	networks	exhibited	the	property	of	“graceful
degradation”:	a	small	amount	of	damage	to	a	neural	network	typically	resulted	in
a	small	
degradation	of	its	performance,	rather	than	a	total	crash.	Even	more

a	small	
importantly,	neural	networks	could	learn	from	experience,	finding	natural	ways
of	generalizing	from	examples	and	finding	hidden	statistical	patterns	in	their
input.
23
	This	made	the	nets	good	at	pattern	recognition	and	classification
problems.	For	example,	by	training	a	neural	network	on	a	data	set	of	sonar
signals,	it	could	be	taught	to	distinguish	the	acoustic	profiles	of	submarines,
mines,	and	sea	life	with	better	accuracy	than	human	experts—and	this	could	be
done	without	anybody	first	having	to	figure	out	in	advance	exactly	how	the

input.
23
categories	were	to	be	defined	or	how	different	features	were	to	be	weighted.
While	simple	neural	network	models	had	been	known	since	the	late	1950s,	the
field	enjoyed	a	renaissance	after	the	introduction	of	the	backpropagation
algorithm,	which	made	it	possible	to	train	multilayered	neural	networks.
24
	Such
multilayered	networks,	which	have	one	or	more	intermediary	(“hidden”)	layers
of	neurons	between	the	input	and	output	layers,	can	learn	a	much	wider	range	of
functions	than	their	simpler	predecessors.
25
	Combined	with	the	increasingly

24
	Such
25
powerful	computers	that	were	becoming	available,	these	algorithmic
improvements	enabled	engineers	to	build	neural	networks	that	were	good
enough	to	be	practically	useful	in	many	applications.
The	brain-like	qualities	of	neural	networks	contrasted	favorably	with	the
rigidly	logic-chopping	but	brittle	performance	of	traditional	rule-based	GOFAI
systems—enough	so	to	inspire	a	new	“-ism,”	
connectionism
,	which	emphasized
the	importance	of	massively	parallel	sub-symbolic	processing.	More	than

150,000	academic	papers	have	since	been	published	on	artificial	neural
networks,	and	they	continue	to	be	an	important	approach	in	machine	learning.

Evolution-based	methods,	such	as	genetic	algorithms	and	genetic
programming,	constitute	another	approach	whose	emergence	helped	end	the
second	AI	winter.	It	made	perhaps	a	smaller	academic	impact	than	neural	nets
but	was	widely	popularized.	In	evolutionary	models,	a	population	of	candidate
solutions	(which	can	be	data	structures	or	programs)	is	maintained,	and	new
candidate	solutions	are	generated	randomly	by	mutating	or	recombining	variants
in	the	existing	population.	Periodically,	the	population	is	pruned	by	applying	a
selection	criterion	(a	fitness	function)	that	allows	only	the	better	candidates	to
survive	into	the	next	generation.	Iterated	over	thousands	of	generations,	the
average	quality	of	the	solutions	in	the	candidate	pool	gradually	increases.	When
it	works,	this	kind	of	algorithm	can	produce	efficient	solutions	to	a	very	wide
range	of	problems—solutions	that	may	be	strikingly	novel	and	unintuitive,	often
looking	more	like	natural	structures	than	anything	that	a	human	engineer	would

design.	And	in	principle,	this	can	happen	without	much	need	for	human	input
beyond	the	initial	specification	of	the	fitness	function,	which	is	often	very
simple.	In	practice,	however,	getting	evolutionary	methods	to	work	well	requires
skill	and	ingenuity,	particularly	in	devising	a	good	representational	format.
Without	an	efficient	way	to	encode	candidate	solutions	(a	genetic	language	that
matches	latent	structure	in	the	target	domain),	evolutionary	search	tends	to

meander	endlessly	in	a	vast	search	space	or	get	stuck	at	a	local	optimum.	Even	if
a	good	representational	
format	is	found,	evolution	is	computationally	demanding
and	is	often	defeated	by	the	combinatorial	explosion.
Neural	networks	and	genetic	algorithms	are	examples	of	methods	that
stimulated	excitement	in	the	1990s	by	appearing	to	offer	alternatives	to	the
stagnating	GOFAI	paradigm.	But	the	intention	here	is	not	to	sing	the	praises	of
these	two	methods	or	to	elevate	them	above	the	many	other	techniques	in

machine	learning.	In	fact,	one	of	the	major	theoretical	developments	of	the	past
twenty	years	has	been	a	clearer	realization	of	how	superficially	disparate
techniques	can	be	understood	as	special	cases	within	a	common	mathematical
framework.	For	example,	many	types	of	artificial	neural	network	can	be	viewed
as	classifiers	that	perform	a	particular	kind	of	statistical	calculation	(maximum
likelihood	estimation).
26
	This	perspective	allows	neural	nets	to	be	compared
with	a	larger	class	of	algorithms	for	learning	classifiers	from	examples

26
—“decision	trees,”	“logistic	regression	models,”	“support	vector	machines,”
“naive	Bayes,”	“
k
-nearest-neighbors	regression,”	among	others.
27
	In	a	similar
manner,	genetic	algorithms	can	be	viewed	as	performing	stochastic	hill-
climbing,	which	is	again	a	subset	of	a	wider	class	of	algorithms	for	optimization.
Each	of	these	algorithms	for	building	classifiers	or	for	searching	a	solution	space

has	its	own	profile	of	strengths	and	weaknesses	which	can	be	studied
mathematically.	Algorithms	differ	in	their	processor	time	and	memory	space
requirements,	which	inductive	biases	they	presuppose,	the	ease	with	which
externally	produced	content	can	be	incorporated,	and	how	transparent	their	inner
workings	are	to	a	human	analyst.
Behind	the	razzle-dazzle	of	machine	learning	and	creative	problem-solving
thus	lies	a	set	of	mathematically	well-specified	tradeoffs.	The	ideal	is	that	of	the
perfect	Bayesian	agent,	one	that	makes	probabilistically	optimal	use	of	available
information.	This	ideal	is	unattainable	because	it	is	too	computationally
demanding	to	be	implemented	in	any	physical	computer	(see	
Box	1
).
Accordingly,	one	can	view	artificial	intelligence	as	a	quest	to	find	shortcuts:
ways	of	tractably	approximating	the	Bayesian	ideal	by	sacrificing	some
optimality	or	generality	while	preserving	enough	to	get	high	performance	in	the
actual	domains	of	interest.

Box	1
).
A	reflection	of	this	picture	can	be	seen	in	the	work	done	over	the	past	couple
of	decades	on	probabilistic	graphical	models,	such	as	Bayesian	networks.
Bayesian	networks	provide	a	concise	way	of	representing	probabilistic	and
conditional	independence	relations	that	hold	in	some	particular	domain.
(Exploiting	such	independence	relations	is	essential	for	overcoming	the
combinatorial	explosion,	which	is	as	much	of	a	problem	for	probabilistic
inference	as	it	is	for	logical	deduction.)	They	also	provide	important	insight	into

the	concept	of	causality.
28
One	advantage	of	relating	learning	problems	from	specific	domains	to	the
general	problem	of	Bayesian	inference	is	that	new	algorithms	that	make
Bayesian	inference	more	efficient	will	then	yield	immediate	improvements
across	many	different	areas.	Advances	in	Monte	Carlo	approximation
techniques,	for	example,	are	directly	applied	in	computer	vision,	robotics,	and
computational	genetics.	Another	advantage	is	that	it	lets	researchers	from
different	disciplines	more	easily	pool	their	findings.	Graphical	models	and

Bayesian	statistics	have	become	a	shared	focus	of	research	in	many	fields,
including	machine	learning,	statistical	physics,	bioinformatics,	combinatorial
optimization,	and	communication	theory.
35
	A	fair	amount	of	the	recent	progress
in	machine	learning	has	resulted	from	incorporating	formal	results	originally
derived	in	other	academic	fields.	(Machine	learning	applications	have	also
benefitted	enormously	from	faster	computers	and	greater	availability	of	large
data	sets.)

Box	1	An	optimal	Bayesian	agent
	
An	ideal	Bayesian	agent	starts	out	with	a	“prior	probability	distribution,”	a
function	that	assigns	probabilities	to	each	“possible	world”	(i.e.	to	each
maximally	specific	way	the	world	could	turn	out	to	be).
29
	This	prior	incorporates
an	inductive	bias	such	that	simpler	possible	worlds	are	assigned	higher
probabilities.	(One	way	to	formally	define	the	simplicity	of	a	possible	world	is	in
terms	of	its	“Kolmogorov	complexity,”	a	measure	based	on	the	length	of	the
shortest	computer	program	that	generates	a	complete	description	of	the	world.
30
)
The	prior	also	incorporates	any	background	knowledge	that	the	programmers
wish	to	give	to	the	agent.
As	the	agent	receives	new	information	from	its	sensors,	it	updates	its
probability	distribution	by	conditionalizing	the	distribution	on	the	new
information	according	to	Bayes’	theorem.
31
	Conditionalization	is	the
mathematical	operation	that	sets	the	new	probability	of	those	worlds	that	are

30
)
31
inconsistent	with	the	information	received	to	zero	and	renormalizes	the
probability	distribution	over	the	remaining	possible	worlds.	The	result	is	a
“posterior	probability	distribution”	(which	the	agent	may	use	as	its	new	prior	in
the	next	time	step).	As	the	agent	makes	observations,	its	probability	mass	thus
gets	concentrated	on	the	shrinking	set	of	possible	worlds	that	remain	consistent
with	the	evidence;	and	among	these	possible	worlds,	simpler	ones	always	have
more	probability.

Metaphorically,	we	can	think	of	a	probability	as	sand	on	a	large	sheet	of	paper.
The	paper	is	partitioned	into	areas	of	various	sizes,	each	area	corresponding	to
one	possible	world,	with	larger	areas	corresponding	to	simpler	possible	worlds.
Imagine	also	a	layer	of	sand	of	even	thickness	spread	across	the	entire	sheet:	this
is	our	prior	probability	distribution.	Whenever	an	observation	is	made	that	rules
out	some	possible	worlds,	we	remove	the	sand	from	the	corresponding	areas	of

the	paper	and	redistribute	it	evenly	over	the	areas	that	remain	in	play.	Thus,	the
total	amount	of	sand	on	the	sheet	never	changes,	it	just	gets	concentrated	into
fewer	areas	as	observational	evidence	accumulates.	This	is	a	picture	of	learning
in	its	purest	form.	(To	calculate	the	probability	of	a	
hypothesis
,	we	simply
measure	the	amount	of	sand	in	all	the	areas	that	correspond	to	the	possible
worlds	in	which	the	hypothesis	is	true.)

So	far,	we	have	defined	a	learning	rule.	To	get	an	agent,	we	also	need	a
decision	rule.	To	this	end,	we	endow	the	agent	with	a	“utility	function”	which
assigns	a	number	to	each	possible	world.	The	number	represents	the	desirability
of	that	world	according	to	the	agent’s	basic	preferences.	Now,	at	each	time	step,
the	agent	selects	the	action	with	the	highest	expected	utility.
32
	(To	find	the	action
with	the	highest	expected	utility,	the	agent	could	list	all	possible	actions.	It	could
then	compute	the	conditional	probability	distribution	given	the	action—the
probability	distribution	that	would	result	from	conditionalizing	its	current
probability	distribution	on	the	observation	that	the	action	had	just	been	taken.
Finally,	it	could	calculate	the	expected	value	of	the	action	as	the	sum	of	the	value
of	each	possible	world	multiplied	by	the	conditional	probability	of	that	world
given	the	action.
33
)
The	learning	rule	and	the	decision	rule	together	define	an	“optimality	notion”

33
)
for	an	agent.	(Essentially	the	same	optimality	notion	has	been	broadly	used	in
artificial	intelligence,	epistemology,	philosophy	of	science,	economics,	and
statistics.
34
)	In	reality,	it	is	impossible	to	build	such	an	agent	because	it	is
computationally	intractable	to	perform	the	requisite	calculations.	Any	attempt	to
do	so	succumbs	to	a	combinatorial	explosion	just	like	the	one	described	in	our
discussion	of	GOFAI.	To	see	why	this	is	so,	consider	one	tiny	subset	of	all

34
possible	worlds:	those	that	consist	of	a	single	computer	monitor	floating	in	an
endless	vacuum.	The	monitor	has	1,	000	×	1,	000	pixels,	each	of	which	is
perpetually	either	on	or	off.	Even	this	subset	of	possible	worlds	is	enormously
large:	the	2
(1,000	×	1,000)
	possible	monitor	states	outnumber	all	the	computations
expected	ever	to	take	place	in	the	observable	universe.	Thus,	we	could	not	even
enumerate	all	the	possible	worlds	in	this	tiny	subset	of	all	possible	worlds,	let

alone	perform	more	elaborate	computations	on	each	of	them	individually.
Optimality	notions	can	be	of	theoretical	interest	even	if	they	are	physically
unrealizable.	They	give	us	a	standard	by	which	to	judge	heuristic
approximations,	and	sometimes	we	can	reason	about	what	an	optimal	agent
would	do	in	some	special	case.	We	will	encounter	some	alternative	optimality
notions	for	artificial	agents	in	
Chapter	12
.
	
Artificial	intelligence	already	outperforms	human	intelligence	in	many	domains.
Table	1
	surveys	the	state	of	game-playing	computers,	showing	that	AIs	now	beat
human	champions	in	a	wide	range	of	games.
36
These	achievements	might	not	seem	impressive	today.	But	this	is	because	our
standards	for	what	is	impressive	keep	adapting	to	the	advances	being	made.
Expert	chess	playing,	for	example,	was	once	thought	to	epitomize	human
intellection.	In	the	view	of	several	experts	in	the	late	fifties:	“If	one	could	devise
a	successful	chess	machine,	one	would	seem	to	have	penetrated	to	the	core	of
human	intellectual	endeavor.”
55
	This	no	longer	seems	so.	One	sympathizes	with
John	McCarthy,	who	lamented:	“As	soon	as	it	works,	no	one	calls	it	AI
anymore.”
56
Table	1	
Game-playing	AI
Checkers
Superhuman
Arthur	Samuel’s	checkers	program,	originally
written	in	1952	and	later	improved	(the	1955
version	incorporating	machine	learning),	becomes
the	first	program	to	learn	to	play	a	game	better	than
its	creator.
37

55
anymore.”
56
Table	1	
Checkers
37
	In	1994,	the	program	CHINOOK
beats	the	reigning	human	champion,	marking	the
first	time	a	program	wins	an	official	world
championship	in	a	game	of	skill.	In	2002,	Jonathan
Schaeffer	and	his	team	“solve”	checkers,	
i.e.
produce	a	program	that	always	makes	the	best
possible	move	(combining	alpha-beta	search	with	a
database	of	39	trillion	endgame	positions).	Perfect
play	by	both	sides	leads	to	a	draw.
38
Backgammon
1979:	The	backgammon	program	BKG	by	Hans
Berliner	defeats	the	world	champion—the	first

i.e.
38
computer	program	to	defeat	(in	an	exhibition
match)	a	world	champion	in	any	game—though
Berliner	later	attributes	the	win	to	luck	with	the
dice	rolls.
39

1992:	The	backgammon	program	TD-Gammon	by
Gerry	Tesauro	reaches	championship-level	ability,
using	temporal	difference	learning	(a	form	of
reinforcement	learning)	and	repeated	plays	against
itself	to	improve.
40
	
In	the	years	since,	backgammon	programs	have	far
surpassed	the	best	human	players.
41
Traveller
TCS
in
collaboration
with
human
42
In	both	1981	and	1982,	Douglas	Lenat’s	program
Eurisko	wins	the	US	championship	in	Traveller
TCS	(a	futuristic	naval	war	game),	prompting	rule
changes	to	block	its	unorthodox	strategies.
43
Eurisko	had	heuristics	for	designing	its	fleet,	and	it
also	had	heuristics	for	modifying	its	heuristics.
Othello
1997:	The	program	Logistello	wins	every	game	in
a	six-game	match	against	world	champion	Takeshi
Murakami.
44
Chess
1997:	Deep	Blue	beats	the	world	chess	champion,
Garry	Kasparov.	Kasparov	claims	to	have	seen
glimpses	of	true	intelligence	and	creativity	in	some
of	the	computer’s	moves.
45
	Since	then,	chess

43
Othello
Murakami.
44
Chess
45
engines	have	continued	to	improve.
46
Crosswords
Expert	level
1999:	The	crossword-solving	program	Proverb
outperforms	the	average	crossword-solver.
47
2012:	The	program	Dr.	Fill,	created	by	Matt
Ginsberg,	scores	in	the	top	quartile	among	the

otherwise	human	contestants	in	the	American
Crossword	Puzzle	Tournament.	(Dr.	Fill’s
performance	is	uneven.	It	completes	perfectly	the
puzzle	rated	most	difficult	by	humans,	yet	is
stumped	by	a	couple	of	nonstandard	puzzles	that
involved	spelling	backwards	or	writing	answers
diagonally.)
48
Scrabble
As	of	2002,	Scrabble-playing	software	surpasses
the	best	human	players.
49
Bridge
Equal	to	the
best
By	2005,	contract	bridge	playing	software	reaches
parity	with	the	best	human	bridge	players.
50
Jeopardy!
2010:	IBM’s	
Watson
	defeats	the	two	all-time-
greatest	human	
Jeopardy!
	champions,	Ken
Jennings	and	Brad	Rutter.
51
	
Jeopardy!
	is	a
televised	game	show	with	trivia	questions	about
history,	literature,	sports,	geography,	pop	culture,
science,	and	other	topics.	Questions	are	presented
in	the	form	of	clues,	and	often	involve	wordplay.
Poker
Varied
Computer	poker	players	remain	slightly	below	the
best	humans	for	full-ring	Texas	hold	‘em	but
perform	at	a	superhuman	level	in	some	poker

Jeopardy!
Watson
Jeopardy!
51
	
Jeopardy!
	is	a
Poker
Varied
variants.
52
FreeCell
Heuristics	evolved	using	genetic	algorithms
produce	a	solver	for	the	solitaire	game	FreeCell
(which	in	its	generalized	form	is	NP-complete)	that
is	able	to	beat	high-ranking	human	players.
53

Go
Very	strong
amateur
level
As	of	2012,	the	Zen	series	of	go-playing	programs
has	reached	rank	6	dan	in	fast	games	(the	level	of	a
very	strong	amateur	player),	using	Monte	Carlo
tree	search	and	machine	learning	techniques.
54
	Go-
playing	programs	have	been	improving	at	a	rate	of
about	1	dan/year	in	recent	years.	If	this	rate	of
improvement	continues,	they	might	beat	the	human
world	champion	in	about	a	decade.
There	is	an	important	sense,	however,	in	which	chess-playing	AI	turned	out	to
be	a	lesser	triumph	than	many	imagined	it	would	be.	It	was	once	supposed,
perhaps	not	unreasonably,	that	in	order	for	a	computer	to	play	chess	at
grandmaster	level,	it	would	have	to	be	endowed	with	a	high	degree	of	
general
intelligence.
57
	One	might	have	thought,	for	example,	that	great	chess	playing
requires	being	able	to	learn	abstract	concepts,	think	cleverly	about	strategy,
compose	flexible	plans,	make	a	wide	range	of	ingenious	logical	deductions,	and

general
57
maybe	even	model	one’s	opponent’s	thinking.	Not	so.	It	turned	out	to	be	possible
to	build	a	perfectly	fine	chess	engine	around	a	special-purpose	algorithm.
58
When	implemented	on	the	fast	processors	that	became	available	towards	the	end
of	the	twentieth	century,	it	produces	very	strong	play.	But	an	AI	built	like	that	is
narrow.	It	plays	chess;	it	can	do	no	other.
59
In	other	domains,	solutions	have	turned	out	to	be	
more
	complicated	than
initially	expected,	and	progress	slower.	The	computer	scientist	Donald	Knuth

58
59
more
was	struck	that	“AI	has	by	now	succeeded	in	doing	essentially	everything	that
requires	‘thinking’	but	has	failed	to	do	most	of	what	people	and	animals	do
‘without	thinking’—that,	somehow,	is	much	harder!”
60
	Analyzing	visual	scenes,
recognizing	objects,	or	controlling	a	robot’s	behavior	as	it	interacts	with	a
natural	environment	has	proved	challenging.	Nevertheless,	a	fair	amount	of
progress	has	been	made	and	continues	to	be	made,	aided	by	steady
improvements	in	hardware.
Common	sense	and	natural	language	understanding	have	also	turned	out	to	be

60
difficult.	It	is	now	often	thought	that	achieving	a	fully	human-level	performance
on	these	tasks	is	an	“AI-complete”	problem,	meaning	that	the	difficulty	of
solving	these	problems	is	essentially	equivalent	to	the	difficulty	of	building
generally	human-level	intelligent	machines.
61
	In	other	words,	if	somebody	
were
to	succeed	in	creating	an	AI	that	could	understand	natural	language	as	well	as	a

human	adult,	they	would	in	all	likelihood	also	either	already	have	succeeded	in
creating	an	AI	that	could	do	everything	else	that	human	intelligence	can	do,	or
they	would	be	but	a	very	short	step	from	such	a	general	capability.
62
Chess-playing	expertise	turned	out	to	be	achievable	by	means	of	a
surprisingly	simple	algorithm.	It	is	tempting	to	speculate	that	other	capabilities
—such	as	general	reasoning	ability,	or	some	key	ability	involved	in
programming—might	likewise	be	achievable	through	some	surprisingly	simple
algorithm.	The	fact	that	the	best	performance	at	one	time	is	attained	through	a
complicated	mechanism	does	not	mean	that	no	simple	mechanism	could	do	the
job	as	well	or	better.	It	might	simply	be	that	nobody	has	yet	found	the	simpler
alternative.	The	Ptolemaic	system	(with	the	Earth	in	the	center,	orbited	by	the
Sun,	the	Moon,	planets,	and	stars)	represented	the	state	of	the	art	in	astronomy
for	over	a	thousand	years,	and	its	predictive	accuracy	was	improved	over	the

centuries	by	progressively	complicating	the	model:	adding	epicycles	upon
epicycles	to	the	postulated	celestial	motions.	Then	the	entire	system	was
overthrown	by	the	heliocentric	theory	of	Copernicus,	which	was	simpler	and—
though	only	after	further	elaboration	by	Kepler—more	predictively	accurate.
63
Artificial	intelligence	methods	are	now	used	in	more	areas	than	it	would	make
sense	to	review	here,	but	mentioning	a	sampling	of	them	will	give	an	idea	of	the
breadth	of	applications.	Aside	from	the	game	AIs	listed	in	
Table	1
,	there	
are

63
Table	1
,	there	
are
hearing	aids	with	algorithms	that	filter	out	ambient	noise;	route-finders	that
display	maps	and	offer	navigation	advice	to	drivers;	recommender	systems	that
suggest	books	and	music	albums	based	on	a	user’s	previous	purchases	and
ratings;	and	medical	decision	support	systems	that	help	doctors	diagnose	breast
cancer,	recommend	treatment	plans,	and	aid	in	the	interpretation	of
electrocardiograms.	There	are	robotic	pets	and	cleaning	robots,	lawn-mowing
robots,	rescue	robots,	surgical	robots,	and	over	a	million	industrial	robots.
64
	The

64
	The
world	population	of	robots	exceeds	10	million.
65
Modern	speech	recognition,	based	on	statistical	techniques	such	as	hidden
Markov	models,	has	become	sufficiently	accurate	for	practical	use	(some
fragments	of	this	book	were	drafted	with	the	help	of	a	speech	recognition
program).	Personal	digital	assistants,	such	as	Apple’s	Siri,	respond	to	spoken
commands	and	can	answer	simple	questions	and	execute	commands.	Optical
character	recognition	of	handwritten	and	typewritten	text	is	routinely	used	in

64
	The
65
applications	such	as	mail	sorting	and	digitization	of	old	documents.
66

Machine	translation	remains	imperfect	but	is	good	enough	for	many
applications.	Early	systems	used	the	GOFAI	approach	of	hand-coded	grammars
that	had	to	be	developed	by	skilled	linguists	from	the	ground	up	for	each
language.	Newer	systems	use	statistical	machine	learning	techniques	that
automatically	build	statistical	models	from	observed	usage	patterns.	The
machine	infers	the	parameters	for	these	models	by	analyzing	bilingual	corpora.
This	approach	dispenses	with	linguists:	the	programmers	building	these	systems
need	not	even	speak	the	languages	they	are	working	with.
67
Face	recognition	has	improved	sufficiently	in	recent	years	that	it	is	now	used
at	automated	border	crossings	in	Europe	and	Australia.	The	US	Department	of
State	operates	a	face	recognition	system	with	over	75	million	photographs	for
visa	processing.	Surveillance	systems	employ	increasingly	sophisticated	AI	and
data-mining	technologies	to	analyze	voice,	video,	or	text,	large	quantities	of

67
which	are	trawled	from	the	world’s	electronic	communications	media	and	stored
in	giant	data	centers.
Theorem-proving	and	equation-solving	are	by	now	so	well	established	that
they	are	hardly	regarded	as	AI	anymore.	Equation	solvers	are	included	in
scientific	computing	programs	such	as	Mathematica.	Formal	verification
methods,	including	automated	theorem	provers,	are	routinely	used	by	chip
manufacturers	to	verify	the	behavior	of	circuit	designs	prior	to	production.
The	US	military	and	intelligence	establishments	have	been	leading	the	way	to

the	large-scale	deployment	of	bomb-disposing	robots,	surveillance	and	attack
drones,	and	other	unmanned	vehicles.	These	still	depend	mainly	on	remote
control	by	human	operators,	but	work	is	underway	to	extend	their	autonomous
capabilities.
Intelligent	scheduling	is	a	major	area	of	success.	The	DART	tool	for
automated	logistics	planning	and	scheduling	was	used	in	Operation	Desert	Storm
in	1991	to	such	effect	that	DARPA	(the	Defense	Advanced	Research	Projects
Agency	in	the	United	States)	claims	that	this	single	application	more	than	paid
back	their

back	their	
thirty-year	investment	in	AI.
68
	Airline	reservation	systems	use
sophisticated	scheduling	and	pricing	systems.	Businesses	make	wide	use	of	AI
techniques	in	inventory	control	systems.	They	also	use	automatic	telephone
reservation	systems	and	helplines	connected	to	speech	recognition	software	to
usher	their	hapless	customers	through	labyrinths	of	interlocking	menu	options.

AI	technologies	underlie	many	Internet	services.	Software	polices	the	world’s
email	traffic,	and	despite	continual	adaptation	by	spammers	to	circumvent	the
countermeasures	being	brought	against	them,	Bayesian	spam	filters	have	largely
managed	to	hold	the	spam	tide	at	bay.	Software	using	AI	components	is
responsible	for	automatically	approving	or	declining	credit	card	transactions,	and
continuously	monitors	account	activity	for	signs	of	fraudulent	use.	Information
retrieval	systems	also	make	extensive	use	of	machine	learning.	The	Google
search	engine	is,	arguably,	the	greatest	AI	system	that	has	yet	been	built.
Now,	it	must	be	stressed	that	the	demarcation	between	artificial	intelligence
and	software	in	general	is	not	sharp.	Some	of	the	applications	listed	above	might
be	viewed	more	as	generic	software	applications	rather	than	AI	in	particular—
though	this	brings	us	back	to	McCarthy’s	dictum	that	when	something	works	it
is	no	longer	called	AI.	A	more	relevant	distinction	for	our	purposes	is	that

between	systems	that	have	a	narrow	range	of	cognitive	capability	(whether	they
be	called	“AI”	or	not)	and	systems	that	have	more	generally	applicable	problem-
solving	capacities.	Essentially	all	the	systems	currently	in	use	are	of	the	former
type:	narrow.	However,	many	of	them	contain	components	that	might	also	play	a
role	in	future	artificial	general	intelligence	or	be	of	service	in	its	development—
components	such	as	classifiers,	search	algorithms,	planners,	solvers,	and
representational	frameworks.

One	high-stakes	and	extremely	competitive	environment	in	which	AI	systems
operate	today	is	the	global	financial	market.	Automated	stock-trading	systems
are	widely	used	by	major	investing	houses.	While	some	of	these	are	simply	ways
of	automating	the	execution	of	particular	buy	or	sell	orders	issued	by	a	human
fund	manager,	others	pursue	complicated	trading	strategies	that	adapt	to
changing	market	conditions.	Analytic	systems	use	an	assortment	of	data-mining

techniques	and	time	series	analysis	to	scan	for	patterns	and	trends	in	securities
markets	or	to	correlate	historical	price	movements	with	external	variables	such
as	keywords	in	news	tickers.	Financial	news	providers	sell	newsfeeds	that	are
specially	formatted	for	use	by	such	AI	programs.	Other	systems	specialize	in
finding	arbitrage	opportunities	within	or	between	markets,	or	in	high-frequency
trading	that	seeks	to	profit	from	minute	price	movements	that	occur	over	the

course	of	milliseconds	(a	timescale	at	which	communication	latencies	even	for
speed-of-light	signals	in	optical	fiber	cable	become	significant,	making	it
advantageous	to	locate	computers	near	the	exchange).	Algorithmic	high-
frequency	traders	account	for	more	than	half	of	equity	shares	traded	on	US
markets.
69
	Algorithmic	trading	has	been	implicated	in	the	2010	Flash	Crash	(see

Box	2
).
Box	2	The	2010	Flash	Crash
	
By	the	afternoon	of	May,	6,	2010,	US	equity	markets	were	already	down	4%	on
worries	about	the	European	debt	crisis.	At	2:32	p.m.,	a	large	seller	(a	mutual
fund	complex)	initiated	a	sell	algorithm	to	dispose	of	a	large	number	of	the	E-
Mini	S&P	500	futures	contracts	to	be	sold	off	at	a	sell	rate	linked	to	a	measure	of
minute-to-minute	liquidity	on	the	exchange.	These	contracts	were	bought	by
algorithmic	high-frequency	traders,	which	were	programmed	to	quickly
eliminate	their	temporary	long	positions	by	selling	the	contracts	on	to	other
traders.	With	demand	from	fundamental	buyers	slacking,	the	algorithmic	traders
started	to	sell	the	E-Minis	primarily	to	other	algorithmic	traders,	which	in	turn
passed	them	on	to	other	algorithmic	traders,	creating	a	“hot	potato”	effect
driving	up	trading	volume—this	being	interpreted	by	the	sell	algorithm	as	an
indicator	of	high	liquidity,	prompting	it	to	increase	the	rate	at	which	it	was

putting	E-Mini	contracts	on	the	market,	pushing	the	downward	spiral.	At	some
point,	the	high-frequency	traders	started	withdrawing	from	the	market,	drying	up
liquidity	while	prices	continued	to	fall.	At	2:45	p.m.,	trading	on	the	E-Mini	was
halted	by	an	automatic	circuit	breaker,	the	exchange’s	stop	logic	functionality.
When	trading	was	restarted,	a	mere	five	seconds	later,	prices	stabilized	and	soon
began	to	recover	most	of	the	losses.	But	for	a	while,	at	the	trough	of	the	crisis,	a

trillion	dollars	had	been	wiped	off	the	market,	and	spillover	effects	had	led	to	a
substantial	number	of	trades	in	individual	securities	being	executed	at	“absurd”
prices,	such	as	one	cent	or	100,000	dollars.	After	the	market	closed	for	the	day,
representatives	of	the	exchanges	met	with	regulators	and	decided	to	break	all
trades	that	had	been	executed	at	prices	60%	or	more	away	from	their	pre-crisis
levels	(deeming	such	transactions	“clearly	erroneous”	and	thus	subject	to	
post
facto
	cancellation	under	existing	trade	rules).
70

post
facto
70
The	retelling	here	of	this	episode	is	a	digression	because	the	computer
programs	involved	in	the	Flash	Crash	were	not	particularly	intelligent	or
sophisticated,	and	the	kind	of	threat	they	created	is	fundamentally	different	from
the	concerns	we	shall	raise	later	in	this	book	in	relation	to	the	prospect	of
machine	superintelligence.	Nevertheless,	these	events	illustrate	several	useful
lessons.	One	is	the	reminder	that	interactions	between	individually	simple

components	(such	as	the	sell	algorithm	and	the	high-frequency	algorithmic
trading	programs)	can	produce	complicated	and	unexpected	effects.	Systemic
risk	can	build	up	in	a	system	as	new	elements	are	introduced,	risks	that	are	not
obvious	until	after	something	goes	wrong	(and	sometimes	not	even	then).
71
Another	lesson	is	that	smart	professionals	might	give	an	instruction	to	a
program	based	on	a	sensible-seeming	and	normally	sound	assumption	(e.g.	that
trading	volume	is	a	good	measure	of	market	liquidity),	and	that	this	can	produce
catastrophic	results	when	the	program	continues	to	act	on	the	instruction	with
iron-clad	logical	consistency	even	in	the	unanticipated	situation	where	the
assumption	turns	out	to	be	invalid.	The	algorithm	just	does	what	it	does;	and
unless	
it	is	a	very	special	kind	of	algorithm,	it	does	not	care	that	we	clasp	our
heads	and	gasp	in	dumbstruck	horror	at	the	absurd	inappropriateness	of	its
actions.	This	is	a	theme	that	we	will	encounter	again.

unless	
A	third	observation	in	relation	to	the	Flash	Crash	is	that	while	automation
contributed	to	the	incident,	it	also	contributed	to	its	resolution.	The	pre-
preprogrammed	stop	order	logic,	which	suspended	trading	when	prices	moved
too	far	out	of	whack,	was	set	to	execute	automatically	because	it	had	been
correctly	anticipated	that	the	triggering	events	could	happen	on	a	timescale	too
swift	for	humans	to	respond.	The	need	for	pre-installed	and	automatically
executing	safety	functionality—as	opposed	to	reliance	on	runtime	human

supervision—again	foreshadows	a	theme	that	will	be	important	in	our	discussion
of	machine	superintelligence.
72
	
Progress	on	two	major	fronts—towards	a	more	solid	statistical	and	information-
theoretic	foundation	for	machine	learning	on	the	one	hand,	and	towards	the
practical	and	commercial	success	of	various	problem-specific	or	domain-specific
applications	on	the	other—has	restored	to	AI	research	some	of	its	lost	prestige.
There	may,	however,	be	a	residual	cultural	effect	on	the	AI	community	of	its

72
	
earlier	history	that	makes	many	mainstream	researchers	reluctant	to	align
themselves	with	over-grand	ambition.	Thus	Nils	Nilsson,	one	of	the	old-timers	in

the	field,	complains	that	his	present-day	colleagues	lack	the	boldness	of	spirit
that	propelled	the	pioneers	of	his	own	generation:
Concern	for	“respectability”	has	had,	I	think,	a	stultifying	effect	on	some	AI
researchers.	I	hear	them	saying	things	like,	“AI	used	to	be	criticized	for	its
flossiness.	Now	that	we	have	made	solid	progress,	let	us	not	risk	losing	our
respectability.”	One	result	of	this	conservatism	has	been	increased
concentration	on	“weak	AI”—the	variety	devoted	to	providing	aids	to
human	thought—and	away	from	“strong	AI”—the	variety	that	attempts	to
mechanize	human-level	intelligence.
73
	
Nilsson’s	sentiment	has	been	echoed	by	several	others	of	the	founders,	including
Marvin	Minsky,	John	McCarthy,	and	Patrick	Winston.
74
The	last	few	years	have	seen	a	resurgence	of	interest	in	AI,	which	might	yet
spill	over	into	renewed	efforts	towards	artificial	
general
	intelligence	(what
Nilsson	calls	“strong	AI”).	In	addition	to	faster	hardware,	a	contemporary

73
	
74
general
project	would	benefit	from	the	great	strides	that	have	been	made	in	the	many
subfields	of	AI,	in	software	engineering	more	generally,	and	in	neighboring
fields	such	as	computational	neuroscience.	One	indication	of	pent-up	demand	for
quality	information	and	education	is	shown	in	the	response	to	the	free	online
offering	of	an	introductory	course	in	artificial	intelligence	at	Stanford	University
in	the	fall	of	2011,	organized	by	Sebastian	Thrun	and	Peter	Norvig.	Some
160,000	students	from	around	the	world	signed	up	to	take	it	(and	23,000

completed	it).
75
Expert	opinions	about	the	future	of	AI	vary	wildly.	There	is	disagreement
about	timescales	as	well	as	about	what	forms	AI	might	eventually	take.
Predictions	about	the	future	development	of	artificial	intelligence,	one	recent
study	noted,	“are	as	confident	as	they	are	diverse.”
76
Although	the	contemporary	distribution	of	belief	has	not	been	very	carefully
measured,	we	can	get	a	rough	impression	from	various	smaller	surveys	and
informal	observations.	In	particular,	a	series	of	recent	surveys	have	polled

75
76
members	of	several	relevant	expert	communities	on	the	question	of	when	they
expect	“human-level	machine	intelligence”	(HLMI)	to	be	developed,	defined	as

“one	that	can	carry	out	most	human	professions	at	least	as	well	as	a	typical
human.”
77
	Results	are	shown	in	
Table	2
.	The	combined	sample	gave	the
following	(median)	estimate:	10%	probability	of	HLMI	by	2022,	50%
probability	by	2040,	and	90%	probability	by	2075.	(Respondents	were	asked	to
premiss	their	estimates	on	the	assumption	that	“human	scientific	activity
continues	without	major	negative	disruption.”)
These	numbers	should	be	taken	with	some	grains	of	salt:	sample	sizes	are
quite	small	and	not	necessarily	representative	of	the	general	expert	population.
They	are,	however,	in	concordance	with	results	from	other	surveys.
78
The	survey	results	are	also	in	line	with	some	recently	published	interviews
with	about	two-dozen	researchers	in	AI-related	fields.	For	example,	Nils	Nilsson
has	spent	a	long	and	productive	career	working	on	problems	in	search,	planning,
knowledge	representation,	and	robotics;	he	has	authored	textbooks	in	artificial

78
intelligence;	and	he	recently	completed	the	most	comprehensive	history	of	the
field	written	to	date.
79
	When	asked	about	arrival	dates	for	HLMI,	he	offered	the
following	opinion:
80
10%	chance:	2030
50%	chance:	2050
90%	chance:	2100
Table	2	
When	will	human-level	machine	intelligence	be	attained?
81
	
Judging	from	the	published	interview	transcripts,	Professor	Nilsson’s	probability

distribution	appears	to	be	quite	representative	of	many	experts	in	the	area—
though	again	it	must	be	emphasized	that	there	is	a	wide	spread	of	opinion:	there
are	practitioners	who	are	substantially	more	boosterish,	confidently	expecting
HLMI	in	the	2020–40	range,	and	others	who	are	confident	either	that	it	will
never	happen	or	that	it	is	indefinitely	far	off.
82
	In	addition,	some	interviewees
feel	that	the	notion	of	a	“human	level”	of	artificial	intelligence	is	ill-defined	or
misleading,	or	are	for	other	reasons	reluctant	to	go	on	record	with	a	quantitative
prediction.
My	own	view	is	that	the	median	numbers	reported	in	the	expert	survey	do	not
have	enough	probability	mass	on	later	arrival	dates.	A	10%	probability	of	HLMI
not	having	been	developed	by	2075	or	even	2100	(after	conditionalizing	on
“human	scientific	activity	continuing	without	major	negative	disruption”)	seems
too	low.
Historically,	AI	researchers	have	not	had	a	strong	record	of	being	able	to

too	low.
predict	the	rate	of	advances	in	their	own	field	or	the	shape	that	such	advances
would	take.	On	the	one	hand,	some	tasks,	like	chess	playing,	turned	out	to	be
achievable	by	means	of	surprisingly	simple	programs;	and	naysayers	who
claimed	that	machines	would	“never”	be	able	to	do	this	or	that	have	repeatedly
been	proven	wrong.	On	the	other	hand,	the	more	typical	errors	among
practitioners	have	been	to	underestimate	the	difficulties	of	getting	a	system	to

perform	robustly	on	real-world	tasks,	and	to	overestimate	the	advantages	of	their
own	particular	pet	project	or	technique.
The	survey	also	asked	two	other	questions	of	relevance	to	our	inquiry.	One
inquired	of	respondents	about	how	much	longer	they	thought	it	would	take	to
reach	superintelligence	assuming	human-level	machine	is	first	achieved.	The
results	are	in	
Table	3
.
Another	question	inquired	what	they	thought	would	be	the	overall	long-term
impact	for	humanity	of	achieving	human-level	machine	intelligence.	The
answers	are	summarized	in	
Figure	2
.

Table	3
.
Figure	2
.
My	own	views	again	differ	somewhat	from	the	opinions	expressed	in	the
survey.	I	assign	a	higher	probability	to	superintelligence	being	created	relatively
soon	after	human-level	machine	intelligence.	I	also	have	a	more	polarized
outlook	on	the	consequences,	thinking	an	extremely	good	or	an	extremely	bad
outcome	to	be	somewhat	more	likely	than	a	more	balanced	outcome.	The

reasons	for	this	will	become	clear	later	in	the	book.
	
Table	3	
How	long	from	human	level	to	superintelligence?
	
Within	2	years	after	HLMI
Within	30	years	after	HLMI
TOP100
5%
50%
Combined
10%
75%
Figure	2
	Overall	long-term	impact	of	HLMI.
83
	
Small	sample	sizes,	selection	biases,	and—above	all—the	inherent
unreliability	of	the	subjective	opinions	elicited	mean	that	one	should	not	read	too

much	into	these	expert	surveys	and	interviews.	They	do	not	let	us	draw	any
strong	conclusion.	But	they	do	hint	at	a	weak	conclusion.	They	suggest	that	(at
least	in	lieu	of	better	data	or	analysis)	it	may	be	reasonable	to	believe	that
human-level	machine	intelligence	has	a	fairly	sizeable	chance	of	being
developed	by	mid-century,	and	that	it	has	a	non-trivial	chance	of	being
developed	considerably	sooner	or	much	later;	that	it	might	perhaps	fairly	soon
thereafter	result	in	superintelligence;	and	that	a	wide	range	of	outcomes	may
have	a	significant	chance	of	occurring,	including	extremely	good	outcomes	and
outcomes	that	are	as	bad	as	human	extinction.
84
	At	the	very	least,	they	suggest
that	the	topic	is	worth	a	closer	look.

CHAPTER	2
Paths	to	superintelligence
	
Machines	are	currently	far	inferior	to	humans	in	general	intelligence.	Yet
one	day	(we	have	suggested)	they	will	be	superintelligent.	How	do	we	get
from	here	to	there?	This	chapter	explores	several	conceivable	technological
paths.	We	look	at	artificial	intelligence,	whole	brain	emulation,	biological
cognition,	and	human–machine	interfaces,	as	well	as	networks	and
organizations.	We	evaluate	their	different	degrees	of	plausibility	as
pathways	to	superintelligence.	The	existence	of	multiple	paths	increases	the
probability	that	the	destination	can	be	reached	via	at	least	one	of	them.
We	can	tentatively	define	a	superintelligence	as	
any	intellect	that	greatly	exceeds
the	cognitive	performance	of	humans	in	virtually	all	domains	of	interest
.
1
	We
will	have	more	to	say	about	the	concept	of	superintelligence	in	the	next	chapter,
where	we	will	subject	it	to	a	kind	of	spectral	analysis	to	distinguish	some
different	possible	forms	of	superintelligence.	But	for	now,	the	rough

.
1
	We
characterization	just	given	will	suffice.	Note	that	the	definition	is	noncommittal
about	how	the	superintelligence	is	implemented.	It	is	also	noncommittal
regarding	qualia:	whether	a	superintelligence	would	have	subjective	conscious
experience	might	matter	greatly	for	some	questions	(in	particular	for	some	moral
questions),	but	our	primary	focus	here	is	on	the	causal	antecedents	and
consequences	of	superintelligence,	not	on	the	metaphysics	of	mind.
2
The	chess	program	Deep	Fritz	is	not	a	superintelligence	on	this	definition,

2
since	Fritz	is	only	smart	within	the	narrow	domain	of	chess.	Certain	kinds	of
domain-specific	superintelligence	could,	however,	be	important.	When	referring
to	superintelligent	performance	limited	to	a	particular	domain,	we	will	note	the
restriction	explicitly.	For	instance,	an	“engineering	superintelligence”	would	be
an	intellect	that	vastly	outperforms	the	best	current	human	minds	in	the	domain
of	engineering.	Unless	otherwise	noted,	we	use	the	term	to	refer	to	systems	that
have	a	superhuman	level	of	
general
	intelligence.

general
But	how	might	we	create	superintelligence?	Let	us	examine	some	possible

paths.
	
Readers	of	this	chapter	must	not	expect	a	blueprint	for	programming	an	artificial
general	intelligence.	No	such	blueprint	exists	yet,	of	course.	And	had	I	been	in
possession	of	such	a	blueprint,	I	most	certainly	would	not	have	published	it	in	a
book.	(If	the	reasons	for	this	are	not	immediately	obvious,	the	arguments	in
subsequent	chapters	will	make	them	clear.)
We	can,	however,	discern	some	general	features	of	the	kind	of	system	that
would	be	required.	It	now	seems	clear	that	a	capacity	to	learn	would	be	an
integral	feature	of	the	core	design	of	a	system	intended	to	attain	general
intelligence,	not	something	to	be	tacked	on	later	as	an	extension	or	an
afterthought.	The	same	holds	for	the	ability	to	deal	effectively	with	uncertainty
and	probabilistic	information.	Some	faculty	for	extracting	useful	concepts	from
sensory	data	and	internal	states,	and	for	leveraging	acquired	concepts	into
flexible	combinatorial	representations	for	use	in	logical	and	intuitive	reasoning,

also	likely	belong	among	the	core	design	features	in	a	modern	AI	intended	to
attain	general	intelligence.
The	early	Good	Old-Fashioned	Artificial	Intelligence	systems	did	not,	for	the
most	part,	focus	on	learning,	uncertainty,	or	concept	formation,	perhaps	because
techniques	for	dealing	with	these	dimensions	were	poorly	developed	at	the	time.
This	is	not	to	say	that	the	underlying	ideas	are	all	that	novel.	The	idea	of	using
learning	as	a	means	of	bootstrapping	a	simpler	system	to	human-level

intelligence	can	be	traced	back	at	least	to	Alan	Turing’s	notion	of	a	“child
machine,”	which	he	wrote	about	in	1950:
Instead	of	trying	to	produce	a	programme	to	simulate	the	adult	mind,	why
not	rather	try	to	produce	one	which	simulates	the	child’s?	If	this	were	then
subjected	to	an	appropriate	course	of	education	one	would	obtain	the	adult
brain.
3

Turing	envisaged	an	iterative	process	to	develop	such	a	child	machine:
We	cannot	expect	to	find	a	good	child	machine	at	the	first	attempt.	One
must	experiment	with	teaching	one	such	machine	and	see	how	well	it
learns.	One	can	then	try	another	and	see	if	it	is	better	or	worse.	There	is	an
obvious	connection	between	this	process	and	evolution….	One	may	hope,
however,	that	this	process	will	be	more	expeditious	than	evolution.	The
survival	of	the	fittest	is	a	slow	method	for	measuring	advantages.	The
experimenter,	by	the	exercise	of	intelligence,	should	be	able	to	speed	it	up.
Equally	important	is	the	fact	that	he	is	not	restricted	to	random	mutations.	If
he	can	trace	a	cause	for	some	weakness	he	can	probably	think	of	the	kind	of
mutation	which	will	improve	it.
4
	
We	know	that	blind	evolutionary	processes	can	produce	human-level	general
intelligence,	since	they	have	already	done	so	at	least	once.	Evolutionary
processes	with	foresight—that	is,	genetic	programs	designed	and	guided	by	an

4
	
intelligent	human	programmer—should	be	able	to	achieve	a	similar	outcome
with	far	greater	efficiency.	This	observation	has	been	used	by	some	philosophers
and	scientists,	
including	David	Chalmers	and	Hans	Moravec,	to	argue	that
human-level	AI	is	not	only	theoretically	possible	but	feasible	within	this
century.
5
	The	idea	is	that	we	can	estimate	the	relative	capabilities	of	evolution
and	human	engineering	to	produce	intelligence,	and	find	that	human	engineering

century.
5
is	already	vastly	superior	to	evolution	in	some	areas	and	is	likely	to	become
superior	in	the	remaining	areas	before	too	long.	The	fact	that	evolution	produced
intelligence	therefore	indicates	that	human	engineering	will	soon	be	able	to	do
the	same.	Thus,	Moravec	wrote	(already	back	in	1976):
The	existence	of	several	examples	of	intelligence	designed	under	these
constraints	should	give	us	great	confidence	that	we	can	achieve	the	same	in
short	order.	The	situation	is	analogous	to	the	history	of	heavier	than	air

flight,	where	birds,	bats	and	insects	clearly	demonstrated	the	possibility
before	our	culture	mastered	it.
6
	
One	needs	to	be	cautious,	though,	in	what	inferences	one	draws	from	this	line	of

reasoning.	It	is	true	that	evolution	produced	heavier-than-air	flight,	and	that
human	engineers	subsequently	succeeded	in	doing	likewise	(albeit	by	means	of	a
very	different	mechanism).	Other	examples	could	also	be	adduced,	such	as
sonar,	magnetic	navigation,	chemical	weapons,	photoreceptors,	and	all	kinds	of
mechanic	and	kinetic	performance	characteristics.	However,	one	could	equally
point	to	areas	where	human	engineers	have	thus	far	failed	to	match	evolution:	in
morphogenesis,	self-repair,	and	the	immune	defense,	for	example,	human	efforts
lag	far	behind	what	nature	has	accomplished.	Moravec’s	argument,	therefore,
cannot	give	us	“great	confidence”	that	we	can	achieve	human-level	artificial
intelligence	“in	short	order.”	At	best,	the	evolution	of	intelligent	life	places	an
upper	bound	on	the	intrinsic	difficulty	of	designing	intelligence.	But	this	upper
bound	could	be	quite	far	above	current	human	engineering	capabilities.
Another	way	of	deploying	an	evolutionary	argument	for	the	feasibility	of	AI	is

via	the	idea	that	we	could,	by	running	genetic	algorithms	on	sufficiently	fast
computers,	achieve	results	comparable	to	those	of	biological	evolution.	This
version	of	the	evolutionary	argument	thus	proposes	a	specific	method	whereby
intelligence	could	be	produced.
But	is	it	true	that	we	will	soon	have	computing	power	sufficient	to	recapitulate
the	relevant	evolutionary	processes	that	produced	human	intelligence?	The
answer	depends	both	on	how	much	computing	technology	will	advance	over	the

next	decades	and	on	how	much	computing	power	would	be	required	to	run
genetic	algorithms	with	the	same	optimization	power	as	the	evolutionary	process
of	natural	selection	that	lies	in	our	past.	Although,	in	the	end,	the	conclusion	we
get	from	pursuing	this	line	of	reasoning	is	disappointingly	indeterminate,	it	is
instructive	to	attempt	a	rough	estimate	(see	
Box	3
).	If	nothing	else,	the	exercise
draws	attention	to	some	interesting	unknowns.
The	upshot	is	that	the	computational	resources	required	to	simply	replicate	the

Box	3
relevant	evolutionary	processes	on	Earth	that	produced	human-level	intelligence
are	severely	out	of	reach—and	will	remain	so	even	if	Moore’s	law	were	to
continue	for	a	century	(cf.	
Figure	3
).	It	is	plausible,	however,	that	compared	with
brute-force	replication	of	natural	evolutionary	processes,	vast	efficiency	gains
are	achievable	by	designing	the	search	process	to	
aim
	for	intelligence,	using
various	obvious	improvements	over	natural	selection.	Yet	it	is	very	hard	to
bound	the	magnitude	of	those	attainable	efficiency	gains.	We	cannot	even	say

Figure	3
aim
whether	they	amount	to	five	or	to	twenty-five	orders	of	magnitude.	Absent
further	elaboration,	therefore,	evolutionary	arguments	are	not	able	to

meaningfully	constrain	our	expectations	of	either	the	difficulty	of	building
human-level	machine	intelligence	or	the	timescales	for	such	developments.
Box	3	What	would	it	take	to	recapitulate	evolution?
	
Not	every	feat	accomplished	by	evolution	in	the	course	of	the	development	of
human	intelligence	is	relevant	to	a	human	engineer	trying	to	artificially	evolve
machine	intelligence.	Only	a	small	portion	of	evolutionary	selection	on	Earth	has
been	selection	for	intelligence.	More	specifically,	the	problems	that	human
engineers	cannot	trivially	bypass	may	have	been	the	target	of	a	very	small
portion	of	total	evolutionary	selection.	For	example,	since	we	can	run	our
computers	on	electrical	power,	we	do	not	have	to	reinvent	the	molecules	of	the
cellular	energy	economy	in	order	to	create	intelligent	machines—yet	such
molecular	evolution	of	metabolic	pathways	might	have	used	up	a	large	part	of
the	total	amount	of	selection	power	that	was	available	to	evolution	over	the
course	of	Earth’s	history.
7

7
One	might	argue	that	the	key	insights	for	AI	are	embodied	in	the	structure	of
nervous	systems,	which	came	into	existence	less	than	a	billion	years	ago.
8
	If	we
take	that	view,	then	the	number	of	relevant	“experiments”	available	to	evolution
is	drastically	curtailed.	There	are	some	4–6×10
30
	prokaryotes	in	the	world	today,
but	only	10
19
	insects,	and	fewer	than	10
10
	humans	(while	pre-agricultural
populations	were	orders	of	magnitude	smaller).
9
	These	numbers	are	only
moderately	intimidating.

7
8
	If	we
30
but	only	10
19
10
9
Evolutionary	algorithms,	however,	require	not	only	variations	to	select	among
but	also	a	fitness	function	to	evaluate	variants,	and	this	is	typically	the	most
computationally	expensive	component.	A	fitness	function	for	the	evolution	of
artificial	intelligence	plausibly	requires	simulation	of	neural	development,
learning,	and	cognition	to	evaluate	fitness.	We	might	thus	do	better	not	to	look	at
the	raw	number	of	organisms	with	complex	nervous	systems,	but	instead	to

attend	to	the	number	of	neurons	in	biological	organisms	that	we	might	need	to
simulate	to	mimic	evolution’s	fitness	function.	We	can	make	a	crude	estimate	of
that	latter	quantity	by	considering	insects,	which	dominate	terrestrial	animal
biomass	(with	ants	alone	estimated	to	contribute	some	15–20%).
10
	Insect	brain

size	varies	substantially,	with	large	and	social	insects	sporting	larger	brains:	a
honeybee	brain	has	just	under	10
6
	neurons,	a	fruit	fly	brain	has	10
5
	neurons,	and
ants	are	in	between	with	250,000	neurons.
11
	The	majority	of	smaller	insects	may
have	brains	of	only	a	few	thousand	neurons.	Erring	on	the	side	of	conservatively
high,	if	we	assigned	all	10
19
	insects	fruit-fly	numbers	of	neurons,	the	total	would
be	10
24
	insect	neurons	in	the	world.	This	could	be	augmented	with	an	additional
order	of	magnitude	to	account	for	aquatic	copepods,	birds,	reptiles,	mammals,
etc.,	to	reach	10
25
.	(By	contrast,	in	pre-agricultural	times	there	were	fewer	than
10
7
	humans,	with	under	10
11
	neurons	each:	thus	fewer	than	10
18
	human	neurons
in	total,	though	humans	have	a	higher	number	of	synapses	per	neuron.)
The	computational	cost	of	simulating	one	neuron	depends	on	the	level	of
detail	that	one	includes	in	the	simulation.	Extremely	simple	neuron	models	use

25
10
7
11
18
about	1,000	floating-point	operations	per	second	(FLOPS)	to	simulate	one
neuron	(in	real-time).	The	electrophysiologically	realistic	Hodgkin–Huxley
model	
uses	1,200,000	FLOPS.	A	more	detailed	multi-compartmental	model
would	add	another	three	to	four	orders	of	magnitude,	while	higher-level	models
that	abstract	systems	of	neurons	could	subtract	two	to	three	orders	of	magnitude
from	the	simple	models.
12
	If	we	were	to	simulate	10
25
	neurons	over	a	billion
years	of	evolution	(longer	than	the	existence	of	nervous	systems	as	we	know

model	
12
25
them),	and	we	allow	our	computers	to	run	for	one	year,	these	figures	would	give
us	a	requirement	in	the	range	of	10
31
–10
44
	FLOPS.	For	comparison,	China’s
Tianhe-2,	the	world’s	most	powerful	supercomputer	as	of	September	2013,
provides	only	3.39×10
16
	FLOPS.	In	recent	decades,	it	has	taken	approximately
6.7	years	for	commodity	computers	to	increase	in	power	by	one	order	of
magnitude.	Even	a	century	of	continued	Moore’s	law	would	not	be	enough	to
close	this	gap.	Running	more	specialized	hardware,	or	allowing	longer	run-

31
–10
44
16
times,	could	contribute	only	a	few	more	orders	of	magnitude.
This	figure	is	conservative	in	another	respect.	Evolution	achieved	human
intelligence	without	aiming	at	this	outcome.	In	other	words,	the	fitness	functions
for	natural	organisms	do	not	select	only	for	intelligence	and	its	precursors.
13
Even	environments	in	which	organisms	with	superior	information	processing
skills	reap	various	rewards	may	not	select	for	intelligence,	because
improvements	to	intelligence	can	(and	often	do)	impose	significant	costs,	such	as

13
higher	energy	consumption	or	slower	maturation	times,	and	those	costs	may
outweigh	whatever	benefits	are	gained	from	smarter	behavior.	Excessively
deadly	environments	also	reduce	the	value	of	intelligence:	the	shorter	one’s

expected	lifespan,	the	less	time	there	will	be	for	increased	learning	ability	to	pay
off.	Reduced	selective	pressure	for	intelligence	slows	the	spread	of	intelligence-
enhancing	innovations,	and	thus	the	opportunity	for	selection	to	favor
subsequent	innovations	that	depend	on	them.	Furthermore,	evolution	may	wind
up	stuck	in	local	optima	that	humans	would	notice	and	bypass	by	altering
tradeoffs	between	exploitation	and	exploration	or	by	providing	a	smooth
progression	of	increasingly	difficult	intelligence	tests.
14
	And	as	mentioned
earlier,	evolution	scatters	much	of	its	selection	power	on	traits	that	are	unrelated
to	intelligence	(such	as	Red	Queen’s	races	of	competitive	co-evolution	between
immune	systems	and	parasites).	Evolution	continues	to	waste	resources
producing	mutations	that	have	proved	consistently	lethal,	and	it	fails	to	take
advantage	of	statistical	similarities	in	the	effects	of	different	mutations.	These
are	all	inefficiencies	in	natural	selection	(when	viewed	as	a	means	of	evolving

14
intelligence)	that	it	would	be	relatively	easy	for	a	human	engineer	to	avoid	while
using	evolutionary	algorithms	to	develop	intelligent	software.
It	is	plausible	that	eliminating	inefficiencies	like	those	just	described	would
trim	many	orders	of	magnitude	off	the	10
31
–10
44
	FLOPS	range	calculated
earlier.	Unfortunately,	it	is	difficult	to	know	how	many	orders	of	magnitude.	It	is
difficult	even	to	make	a	rough	estimate—for	aught	we	know,	the	efficiency
savings	could	be	five	orders	of	magnitude,	or	ten,	or	twenty-five.

31
–10
44
15
	
Figure	3
	Supercomputer	performance.	In	a	narrow	sense,	“Moore’s	law”	refers
to	the	observation	that	the	number	of	transistors	on	integrated	circuits	have	for
several	decades	doubled	approximately	every	two	years.	However,	the	term	is

often	used	to	refer	to	the	more	general	observation	that	many	performance
metrics	in	computing	technology	have	followed	a	similarly	fast	exponential
trend.	Here	we	plot	peak	speed	of	the	world’s	fastest	supercomputer	as	a
function	of	time	(on	a	logarithmic	vertical	scale).	In	recent	years,	growth	in	the
serial	speed	of	processors	has	stagnated,	but	increased	use	of	parallelization	has
enabled	the	total	number	of	computations	performed	to	remain	on	the	trend
line.
16
	
There	is	a	further	complication	with	these	kinds	of	evolutionary
considerations,	one	that	makes	it	hard	to	derive	from	them	even	a	very	loose
upper	bound	on	the	difficulty	of	evolving	intelligence.	We	must	avoid	the	error
of	inferring,	from	the	fact	that	intelligent	life	evolved	on	Earth,	that	the
evolutionary	processes	involved	had	a	reasonably	high	prior	probability	of
producing	intelligence.	Such	an	inference	is	unsound	because	it	fails	to	take
account	of	the	observation	selection	effect	that	guarantees	that	all	observers	will

find	themselves	having	originated	on	a	planet	where	intelligent	life	arose,	no
matter	how	likely	or	unlikely	it	was	for	any	given	such	planet	to	produce
intelligence.	Suppose,	for	example,	that	in	addition	to	the	systematic	effects	of
natural	selection	it	required	an	enormous	amount	of	
lucky	coincidence
	to
produce	intelligent	life—enough	so	that	intelligent	life	evolves	on	only	one
planet	out	of	every	10
30
	planets	on	which	simple	replicators	arise.	In	that	case,
when	we	run	our	genetic	algorithms	to	try	to	replicate	what	natural	evolution

	to
30
did,	we	might	find	that	we	must	run	some	10
30
	simulations	before	we	find	one
where	all	the	elements	come	together	in	just	the	right	way.	This	seems	fully
consistent	with	our	observation	that	life	did	evolve	here	on	Earth.	Only	by
careful	and	somewhat	intricate	reasoning—by	analyzing	instances	of	convergent
evolution	of	intelligence-related	traits	and	engaging	with	the	subtleties	of
observation	selection	theory—can	we	partially	circumvent	this	epistemological
barrier.	Unless	one	takes	the	trouble	to	do	so,	one	is	not	in	a	position	to	rule	out

30
the	possibility	that	the	alleged	“upper	bound”	on	the	computational	requirements
for	recapitulating	the	evolution	of	intelligence	derived	in	
Box	3
	might	be	too	low
by	thirty	orders	of	magnitude	(or	some	other	such	large	number).
17
Another	way	of	arguing	for	the	feasibility	of	artificial	intelligence	is	by
pointing	to	the	human	brain	and	suggesting	that	we	could	use	it	as	a	template	for
a	machine	intelligence.	One	can	distinguish	different	versions	of	this	approach

Box	3
17
based	on	how	closely	they	propose	to	imitate	biological	brain	functions.	At	one

extreme—that	of	very	close	imitation—we	have	the	idea	of	
whole	brain
emulation
,	which	we	will	discuss	in	the	next	subsection.	At	the	other	extreme	are
approaches	that	take	their	inspiration	from	the	functioning	of	the	brain	but	do	not
attempt	low-level	imitation.	Advances	in	neuroscience	and	cognitive	psychology
—which	will	be	aided	by	improvements	in	instrumentation—should	eventually
uncover	the	general	principles	of	brain	function.	This	knowledge	could	then
guide	AI	efforts.	We	have	already	encountered	neural	networks	as	an	example	of
a	brain-inspired	AI	technique.	Hierarchical	perceptual	organization	is	another
idea	that	has	been	transferred	from	brain	science	to	machine	learning.	The	study
of	reinforcement	learning	has	been	motivated	(at	least	in	part)	by	its	role	in
psychological	theories	of	animal	cognition,	and	reinforcement	learning
techniques	(e.g.	the	“TD-algorithm”)	inspired	by	these	theories	are	now	widely
used	in	AI.
18
	More	cases	like	these	will	surely	accumulate	in	the	future.	Since

used	in	AI.
18
there	is	a	limited	number—perhaps	a	very	small	number—of	distinct
fundamental	mechanisms	that	operate	in	the	brain,	continuing	incremental
progress	in	brain	science	should	eventually	discover	them	all.	Before	this
happens,	though,	it	is	possible	that	a	hybrid	approach,	combining	some	brain-
inspired	techniques	with	some	purely	artificial	methods,	would	cross	the
finishing	line.	In	that	case,	the	resultant	system	need	not	be	recognizably	brain-
like	even	though	some	brain-derived	insights	were	used	in	its	development.

The	availability	of	the	brain	as	template	provides	strong	support	for	the	claim
that	machine	intelligence	is	ultimately	feasible.	This,	however,	does	not	enable
us	to	predict	when	it	will	be	achieved	because	it	is	hard	to	predict	the	future	rate
of	discoveries	in	brain	science.	What	we	can	say	is	that	the	further	into	the	future
we	look,	the	greater	the	likelihood	that	the	secrets	of	the	brain’s	functionality
will	have	been	decoded	sufficiently	to	enable	the	creation	of	machine
intelligence	in	this	manner.

Different	people	working	toward	machine	intelligence	hold	different	views
about	how	promising	neuromorphic	approaches	are	compared	with	approaches
that	aim	for	completely	synthetic	designs.	The	existence	of	birds	demonstrated
that	heavier-than-air	flight	was	physically	possible	and	prompted	efforts	to	build
flying	machines.	Yet	the	first	functioning	airplanes	did	not	flap	their	wings.	The
jury	is	out	on	whether	machine	intelligence	will	be	like	flight,	which	humans

achieved	through	an	artificial	mechanism,	or	like	combustion,	which	we	initially
mastered	by	copying	naturally	occurring	fires.
Turing’s	idea	of	designing	a	program	that	acquires	most	of	its	content	by

learning,	rather	than	having	it	pre-programmed	at	the	outset,	can	apply	equally	to
neuromorphic	and	synthetic	approaches	to	machine	intelligence.
A	variation	on	Turing’s	conception	of	a	child	machine	is	the	idea	of	a	“seed
AI.”
19
	Whereas	a	child	machine,	as	Turing	seems	to	have	envisaged	it,	would
have	a	relatively	fixed	architecture	that	simply	develops	its	inherent
potentialities	by	accumulating	
content
,	a	seed	AI	would	be	a	more	sophisticated
artificial	intelligence	capable	of	improving	its	own	
architecture
.	In	the	early
stages	of	a	seed	AI,	such	improvements	might	occur	mainly	through	trial	and
error,	information	acquisition,	or	assistance	from	the	programmers.	At	its	later
stages,	however,	a	seed	AI	should	be	able	to	
	its	own	workings
sufficiently	to	engineer	new	algorithms	and	computational	structures	to	bootstrap
its	cognitive	performance.	This	needed	understanding	could	result	from	the	seed
AI	reaching	a	sufficient	level	of	general	intelligence	across	many	domains,	or

from	crossing	some	threshold	in	a	particularly	relevant	domain	such	as	computer
science	or	mathematics.
This	brings	us	to	another	important	concept,	that	of	“recursive	self-
improvement.”	A	successful	seed	AI	would	be	able	to	iteratively	enhance	itself:
an	early	version	of	the	AI	could	design	an	improved	version	of	itself,	and	the
improved	version—being	smarter	than	the	original—might	be	able	to	design	an
even	smarter	version	of	itself,	and	so	forth.
20
	Under	some	conditions,	such	a

20
process	of	recursive	self-improvement	might	continue	long	enough	to	result	in
an	intelligence	explosion—an	event	in	which,	in	a	short	period	of	time,	a
system’s	level	of	intelligence	increases	from	a	relatively	modest	endowment	of
cognitive	capabilities	(perhaps	sub-human	in	most	respects,	but	with	a	domain-
specific	talent	for	coding	and	AI	research)	to	radical	superintelligence.	We	will
return	to	this	important	possibility	in	
Chapter	4
,	where	the	dynamics	of	such	an

20
Chapter	4
event	will	be	analyzed	more	closely.	Note	that	this	model	suggests	the	possibility
of	surprises:	attempts	to	build	artificial	general	intelligence	might	fail	pretty
much	completely	until	the	last	missing	critical	component	is	put	in	place,	at
which	point	a	seed	AI	might	become	capable	of	sustained	recursive	self-
improvement.
Before	we	end	this	subsection,	there	is	one	more	thing	that	we	should
emphasize,	which	is	that	an	artificial	intelligence	need	not	much	resemble	a

human	mind.	AIs	could	be—indeed,	it	is	likely	that	most	will	be—extremely
alien.	We	should	expect	that	they	will	have	very	different	cognitive	architectures
than	biological	intelligences,	and	in	their	early	stages	of	development	they	will

have	very	different	profiles	of	cognitive	strengths	and	weaknesses	(though,	as
we	shall	later	argue,	they	could	eventually	overcome	any	initial	weakness).
Furthermore,	the	goal	systems	of	AIs	could	diverge	radically	from	those	of
human	beings.	There	is	no	reason	to	expect	a	generic	AI	to	be	motivated	by	love
or	hate	or	pride	or	other	such	common	human	sentiments:	these	complex
adaptations	would	require	deliberate	expensive	effort	to	recreate	in	AIs.	This	is
at	once	a	big	problem	and	a	big	opportunity.	
We	will	return	to	the	issue	of	AI
motivation	in	later	chapters,	but	it	is	so	central	to	the	argument	in	this	book	that
it	is	worth	bearing	in	mind	throughout.
	
In	whole	brain	emulation	(also	known	as	“uploading”),	intelligent	software
would	be	produced	by	scanning	and	closely	modeling	the	computational
structure	of	a	biological	brain.	This	approach	thus	represents	a	limiting	case	of
drawing	inspiration	from	nature:	barefaced	plagiarism.	Achieving	whole	brain

emulation	requires	the	accomplishment	of	the	following	steps.
First,	a	sufficiently	detailed	scan	of	a	particular	human	brain	is	created.	This
might	involve	stabilizing	the	brain	post-mortem	through	vitrification	(a	process
that	turns	tissue	into	a	kind	of	glass).	A	machine	could	then	dissect	the	tissue	into
thin	slices,	which	could	be	fed	into	another	machine	for	scanning,	perhaps	by	an
array	of	electron	microscopes.	Various	stains	might	be	applied	at	this	stage	to

bring	out	different	structural	and	chemical	properties.	Many	scanning	machines
could	work	in	parallel	to	process	multiple	brain	slices	simultaneously.
Second,	the	raw	data	from	the	scanners	is	fed	to	a	computer	for	automated
image	processing	to	reconstruct	the	three-dimensional	neuronal	network	that
implemented	cognition	in	the	original	brain.	In	practice,	this	step	might	proceed
concurrently	with	the	first	step	to	reduce	the	amount	of	high-resolution	image
data	stored	in	buffers.	The	resulting	map	is	then	combined	with	a	library	of

neurocomputational	models	of	different	types	of	neurons	or	of	different	neuronal
elements	(such	as	particular	kinds	of	synaptic	connectors).	
Figure	4
	shows	some
results	of	scanning	and	image	processing	produced	with	present-day	technology.
In	the	third	stage,	the	neurocomputational	structure	resulting	from	the
previous	step	is	implemented	on	a	sufficiently	powerful	computer.	If	completely

successful,	the	result	would	be	a	digital	reproduction	of	the	original	intellect,
with	memory	and	personality	intact.	The	emulated	human	mind	now	exists	as
software	on	a	computer.	The	mind	can	either	inhabit	a	virtual	reality	or	interface
with	the	external	world	by	means	of	robotic	appendages.
The	whole	brain	emulation	path	does	not	require	that	we	figure	out	how
human	cognition	works	or	how	to	program	an	artificial	intelligence.	It	requires
only	that	we	understand	the	low-level	functional	characteristics	of	the	basic
computational	elements	of	the	brain.	No	fundamental	conceptual	or	theoretical
breakthrough	is	needed	for	whole	brain	emulation	to	succeed.
Whole	brain	emulation	does,	however,	require	some	rather	advanced	enabling
technologies.	There	are	three	key	prerequisites:	(1)	
scanning
:	high-throughput
microscopy	with	sufficient	resolution	and	detection	of	relevant	properties;	(2)
translation
:	automated	image	analysis	to	turn	raw	scanning	data	into	an

scanning
interpreted	three-dimensional	model	of	relevant	neurocomputational	elements;
and	(3)	
simulation
:	hardware	powerful	enough	to	implement	the	resultant
computational	structure	(see	
Table	4
).	(In	comparison	with	these	more
challenging	
steps,	the	construction	of	a	basic	virtual	reality	or	a	robotic
embodiment	with	an	audiovisual	input	channel	and	some	simple	output	channel
is	relatively	easy.	Simple	yet	minimally	adequate	I/O	seems	feasible	already	with
present	technology.
23
)

Figure	4
	Reconstructing	3D	neuroanatomy	from	electron	microscope	images.
Upper	left
:	A	typical	electron	micrograph	showing	cross-sections	of	neuronal
matter—dendrites	and	axons.	
Upper	right
:	Volume	image	of	rabbit	retinal	neural
tissue	acquired	by	serial	block-face	scanning	electron	microscopy.
21
	Individual
2D	images	have	been	stacked	into	a	cube	(with	a	side	of	approximately	11	μm).
Bottom
:	Reconstruction	of	a	subset	of	the	neuronal	projections	filling	a	volume
of	neuropil,	generated	by	an	automated	segmentation	algorithm.
22
	
There	is	good	reason	to	think	that	the	requisite	enabling	technologies	are
attainable,	though	not	in	the	near	future.	Reasonable	computational	models	of
many	types	of	neuron	and	neuronal	processes	already	exist.	Image	recognition
software	has	been	developed	that	can	trace	axons	and	dendrites	through	a	stack
of	two-dimensional	images	(though	reliability	needs	to	be	improved).	And	there
are	imaging	tools	that	provide	the	necessary	resolution—with	a	scanning

tunneling	microscope	it	is	possible	to	“see”	individual	atoms,	which	is	a	far
higher	resolution	than	needed.	However,	although	present	knowledge	and
capabilities	suggest	that	there	is	no	in-principle	barrier	to	the	development	of	the
requisite	enabling	technologies,	it	is	clear	that	a	very	great	deal	of	incremental
technical	progress	would	be	needed	to	bring	human	whole	brain	emulation
within	reach.
24
	For	example,	microscopy	technology	would	need	not	just
sufficient	resolution	but	also	sufficient	throughput.	Using	an	atomic-resolution
scanning	tunneling	microscope	to	image	the	needed	surface	area	would	be	far
too	slow	to	be	practicable.	It	would	be	more	plausible	to	use	a	lower-resolution
electron	microscope,	but	this	would	require	new	methods	for	preparing	and
staining	cortical	tissue	to	make	visible	relevant	details	such	as	synaptic	fine
structure.	A	great	expansion	of	neurocomputational	libraries	and	major
improvements	in	automated	image	processing	and	scan	interpretation	would	also
be	needed.
Table	4

be	needed.
Table	4	
Capabilities	needed	for	whole	brain	emulation

In	general,	whole	brain	emulation	relies	less	on	theoretical	insight	and	more
on	technological	capability	than	artificial	intelligence.	Just	how	much
technology	is	required	for	whole	brain	emulation	depends	on	the	level	of
abstraction	at	which	the	brain	is	emulated.	In	this	regard	there	is	a	tradeoff
between	insight	and	technology.	In	general,	the	worse	our	scanning	equipment
and	the	feebler	our	computers,	the	less	we	could	rely	on	simulating	low-level
chemical	and	electrophysiological	brain	processes,	and	the	more	theoretical
understanding	would	be	needed	of	the	computational	architecture	that	we	are
seeking	to	emulate	in	order	to	create	more	abstract	representations	of	the
relevant	functionalities.
25
	Conversely,	with	sufficiently	advanced	scanning
technology	and	abundant	computing	power,	it	might	be	possible	to	brute-force
an	emulation	even	with	a	fairly	limited	understanding	of	the	brain.	In	the
unrealistic	limiting	case,	we	could	imagine	emulating	a	brain	at	the	level	of	its

25
elementary	particles	using	the	quantum	mechanical	Schrödinger	equation.	Then
one	could	rely	entirely	on	existing	knowledge	of	physics	and	not	at	all	on	any

biological	model.	This	extreme	case,	however,	would	place	utterly	impracticable
demands	on	computational	power	and	data	acquisition.	A	far	more	plausible
level	of	emulation	would	be	one	that	incorporates	individual	neurons	and	their
connectivity	matrix,	along	with	some	of	the	structure	of	their	dendritic	trees	and
maybe	some	state	variables	of	individual	synapses.	Neurotransmitter	molecules
would	not	be	simulated	individually,	but	their	fluctuating	concentrations	would
be	modeled	in	a	coarse-grained	manner.
To	assess	the	feasibility	of	whole	brain	emulation,	one	must	understand	the
criterion	for	success.	The	aim	is	not	to	create	a	brain	simulation	so	detailed	and
accurate	that	one	could	use	it	to	predict	exactly	what	would	have	happened	in	the
original	brain	if	it	had	been	subjected	to	a	particular	sequence	of	stimuli.	Instead,
the	aim	is	to	capture	enough	of	the	computationally	functional	properties	of	the
brain	to	enable	the	resultant	emulation	to	perform	intellectual	work.	For	this

purpose,	much	of	the	messy	biological	detail	of	a	real	brain	is	irrelevant.
A	more	elaborate	analysis	would	distinguish	between	different	levels	of
emulation	success	based	on	the	extent	to	which	the	information-processing
functionality	of	the	emulated	brain	has	been	preserved.	For	example,	one	could
distinguish	among	(1)	a	
high-fidelity	emulation
	that	has	the	full	set	of
knowledge,	skills,	capacities,	and	values	of	the	emulated	brain;	(2)	a	
distorted
emulation
	whose	dispositions	are	significantly	non-human	in	some	ways	but

distorted
emulation
which	is	mostly	able	to	do	the	same	intellectual	labor	as	the	emulated	brain;	and
(3)	a	
generic	emulation
	(which	might	also	be	distorted)	that	is	somewhat	like	an
infant,	lacking	the	skills	or	memories	that	had	been	acquired	by	the	emulated
adult	brain	but	with	the	capacity	to	learn	most	of	what	a	normal	human	can
learn.
26
While	it	appears	ultimately	feasible	to	produce	a	high-fidelity	emulation,	it
seems	quite	likely	that	the	
first
	whole	brain	emulation	that	we	would	achieve	if
we	went	down	this	path	would	be	of	a	lower	grade.	Before	we	would	get	things

(3)	a	
learn.
26
first
to	work	perfectly,	we	would	probably	get	things	to	work	imperfectly.	It	is	also
possible	that	a	push	toward	emulation	technology	would	lead	to	the	creation	of
some	kind	of	neuromorphic	AI	that	would	adapt	some	neurocomputational
principles	discovered	during	emulation	efforts	and	hybridize	them	with	synthetic
methods,	and	that	this	would	happen	before	the	completion	of	a	fully	functional
whole	brain	emulation.	The	possibility	of	such	a	spillover	into	neuromorphic	AI,

as	we	shall	see	in	a	later	chapter,	complicates	the	strategic	assessment	of	the
desirability	of	seeking	to	expedite	emulation	technology.

How	far	are	we	currently	from	achieving	a	human	whole	brain	emulation?
One	recent	assessment	presented	a	technical	roadmap	and	concluded	that	the
prerequisite	capabilities	might	be	available	around	mid-century,	though	with	a
large	uncertainty	interval.
27
	
Figure	5
	depicts	the	major	milestones	in	this
roadmap.	The	apparent	simplicity	of	the	map	may	be	deceptive,	however,	and
we	should	be	careful	not	to	understate	how	much	work	remains	to	be	done.	No
brain	has	yet	been	emulated.	Consider	the	humble	model	organism
Caenorhabditis	elegans
,	which	is	a	transparent	roundworm,	about	1	mm	in
length,	with	302	neurons.	The	complete	connectivity	matrix	of	these	neurons	has
been	known	since	the	mid-1980s,	when	it	was	laboriously	mapped	out	by	means
of	slicing,	electron	microscopy,	and	
hand-labeling	of	specimens.
29
	But	knowing
merely	which	neurons	are	connected	with	which	is	not	enough.	To	create	a	brain
emulation	one	would	also	need	to	know	which	synapses	are	excitatory	and

29
which	are	inhibitory;	the	strength	of	the	connections;	and	various	dynamical
properties	of	axons,	synapses,	and	dendritic	trees.	This	information	is	not	yet
available	even	for	the	small	nervous	system	of	
C.	elegans
	(although	it	may	now
be	within	range	of	a	targeted	moderately	sized	research	project).
30
	Success	at
emulating	a	tiny	brain,	such	as	that	of	
C.	elegans
,	would	give	us	a	better	view	of
what	it	would	take	to	emulate	larger	brains.

Figure	5
	Whole	brain	emulation	roadmap.	Schematic	of	inputs,	activities,	and
milestones.
28
	
At	some	point	in	the	technology	development	process,	once	techniques	are
available	for	automatically	emulating	small	quantities	of	brain	tissue,	the
problem	reduces	to	one	of	scaling.	Notice	“the	ladder”	at	the	right	side	of	
Figure
5
.	This	ascending	series	of	boxes	represents	a	final	sequence	of	advances	which
can	commence	after	preliminary	hurdles	have	been	cleared.	The	stages	in	this
sequence	correspond	to	whole	brain	emulations	of	successively	more
neurologically	sophisticated	model	organisms—for	example,	
C.	elegans	→
honeybee
	→	
mouse
	→	
rhesus	monkey
	→	
human
.	Because	the	gaps	between
these	rungs—at	least	after	the	first	step—are	mostly	quantitative	in	nature	and
due	mainly	(though	not	entirely)	to	the	differences	in	size	of	the	brains	to	be
emulated,	they	should	be	tractable	through	a	relatively	straightforward	scale-up
of	scanning	and	simulation	capacity.
31

honeybee
	→	
mouse
	→	
	→	
human
31
Once	we	start	ascending	this	final	ladder,	the	
eventual
	attainment	of	human
whole	brain	emulation	becomes	more	clearly	foreseeable.
32
	We	can	thus	expect
to	get	some	advance	warning	before	arrival	at	human-level	machine	intelligence
along	the	whole	brain	emulation	path,	at	least	if	the	last	among	the	requisite
enabling	technologies	to	reach	sufficient	maturity	is	either	high-throughput
scanning	or	the	computational	power	needed	for	real-time	simulation.	If,

31
eventual
32
however,	the	last	enabling	technology	to	fall	into	place	is	neurocomputational
modeling,	then	the	transition	from	unimpressive	prototypes	to	a	working	human
emulation	could	be	more	abrupt.	One	could	imagine	a	scenario	in	which,	despite
abundant	scanning	data	and	fast	computers,	it	is	proving	difficult	to	get	our
neuronal	models	to	work	right.	When	finally	the	last	glitch	is	ironed	out,	what
was	previously	a	completely	dysfunctional	system—analogous	perhaps	to	an

unconscious	brain	undergoing	a	grand	mal	seizure—might	snap	into	a	coherent
wakeful	state.	In	this	case,	the	key	advance	would	not	be	heralded	by	a	series	of
functioning	animal	emulations	of	increasing	magnitude	(provoking	newspaper
headlines	of	correspondingly	escalating	font	size).	Even	for	those	paying
attention	it	might	be	difficult	to	tell	in	advance	of	success	just	how	many	flaws
remained	in	the	neurocomputational	models	at	any	point	and	how	long	it	would
take	to	fix	them,	even	up	to	the	eve	of	the	critical	breakthrough.	(Once	a	human

whole	brain	emulation	has	been	achieved,	further	potentially	explosive
developments	would	take	place;	but	we	postpone	discussion	of	this	until	
Chapter

4
.)
Surprise	scenarios	are	thus	imaginable	for	whole	brain	emulation	even	if	all
the	relevant	research	were	conducted	in	the	open.	Nevertheless,	compared	with
the	AI	path	to	machine	intelligence,	whole	brain	emulation	is	more	likely	to	be
preceded	
by	clear	omens	since	it	relies	more	on	concrete	observable
technologies	and	is	not	wholly	based	on	theoretical	insight.	We	can	also	say,
with	greater	confidence	than	for	the	AI	path,	that	the	emulation	path	will	not
succeed	in	the	near	future	(within	the	next	fifteen	years,	say)	because	we	know
that	several	challenging	precursor	technologies	have	not	yet	been	developed.	By
contrast,	it	seems	likely	that	somebody	could	
in	principle
	sit	down	and	code	a
seed	AI	on	an	ordinary	present-day	personal	computer;	and	it	is	conceivable—
though	unlikely—that	somebody	somewhere	will	get	the	right	insight	for	how	to
do	this	in	the	near	future.
	
A	third	path	to	greater-than-current-human	intelligence	is	to	enhance	the

functioning	of	biological	brains.	In	principle,	this	could	be	achieved	without
technology,	through	selective	breeding.	Any	attempt	to	initiate	a	classical	large-
scale	eugenics	program,	however,	would	confront	major	political	and	moral
hurdles.	Moreover,	unless	the	selection	were	extremely	strong,	many	generations
would	be	required	to	produce	substantial	results.	Long	before	such	an	initiative
would	bear	fruit,	advances	in	biotechnology	will	allow	much	more	direct	control
of	human	genetics	and	neurobiology,	rendering	otiose	any	human	breeding

program.	We	will	therefore	focus	on	methods	that	hold	the	potential	to	deliver
results	faster,	on	the	timescale	of	a	few	generations	or	less.
Our	individual	cognitive	capacities	can	be	strengthened	in	various	ways,
including	by	such	traditional	methods	as	education	and	training.	Neurological
development	can	be	promoted	by	low-tech	interventions	such	as	optimizing
maternal	and	infant	nutrition,	removing	lead	and	other	neurotoxic	pollutants
from	the	environment,	eradicating	parasites,	ensuring	adequate	sleep	and

exercise,	and	preventing	diseases	that	affect	the	brain.
33
	Improvements	in
cognition	can	certainly	be	obtained	through	each	of	these	means,	though	the
magnitudes	of	the	gains	are	likely	to	be	modest,	especially	in	populations	that
are	already	reasonably	well-nourished	and	-schooled.	We	will	certainly	not

achieve	superintelligence	by	any	of	these	means,	but	they	might	help	on	the
margin,	particularly	by	lifting	up	the	deprived	and	expanding	the	catchment	of
global	talent.	(Lifelong	depression	of	intelligence	due	to	iodine	deficiency
remains	widespread	in	many	impoverished	inland	areas	of	the	world—an
outrage	given	that	the	condition	can	be	prevented	by	fortifying	table	salt	at	a	cost
of	a	few	cents	per	person	and	year.
34
)
Biomedical	enhancements	could	give	bigger	boosts.	Drugs	already	exist	that
are	alleged	to	improve	memory,	concentration,	and	mental	energy	in	at	least
some	subjects.
35
	(Work	on	this	book	was	fueled	by	coffee	and	nicotine	chewing
gum.)	While	the	efficacy	of	the	present	generation	of	smart	drugs	is	variable,
marginal,	and	generally	dubious,	future	nootropics	might	offer	clearer	benefits
and	fewer	side	effects.
36
	However,	it	seems	implausible,	on	both	neurological
and	evolutionary	grounds,	that	one	could	by	introducing	some	chemical	into	the

35
36
brain	of	a	healthy	person	spark	a	dramatic	rise	in	intelligence.
37
	The	cognitive
functioning	
of	a	human	brain	depends	on	a	delicate	orchestration	of	many
factors,	especially	during	the	critical	stages	of	embryo	development—and	it	is
much	more	likely	that	this	self-organizing	structure,	to	be	enhanced,	needs	to	be
carefully	balanced,	tuned,	and	cultivated	rather	than	simply	flooded	with	some
extraneous	potion.
Manipulation	of	genetics	will	provide	a	more	powerful	set	of	tools	than

37
psychopharmacology.	Consider	again	the	idea	of	genetic	selection:	instead	of
trying	to	implement	a	eugenics	program	by	controlling	mating	patterns,	one
could	use	selection	at	the	level	of	embryos	or	gametes.
38
	Pre-implantation
genetic	diagnosis	has	already	been	used	during	in	vitro	fertilization	procedures
to	screen	embryos	produced	for	monogenic	disorders	such	as	Huntington’s
disease	and	for	predisposition	to	some	late-onset	diseases	such	as	breast	cancer.

38
It	has	also	been	used	for	sex	selection	and	for	matching	human	leukocyte	antigen
type	with	that	of	a	sick	sibling,	who	can	then	benefit	from	a	cord-blood	stem	cell
donation	when	the	new	baby	is	born.
39
	The	range	of	traits	that	can	be	selected
for	or	against	will	expand	greatly	over	the	next	decade	or	two.	A	strong	driver	of
progress	in	behavioral	genetics	is	the	rapidly	falling	cost	of	genotyping	and	gene
sequencing.	Genome-wide	complex	trait	analysis,	using	studies	with	vast

39
numbers	of	subjects,	is	just	now	starting	to	become	feasible	and	will	greatly
increase	our	knowledge	of	the	genetic	architectures	of	human	cognitive	and
behavioral	traits.
40
	Any	trait	with	a	non-negligible	heritability—including
cognitive	capacity—could	then	become	susceptible	to	selection.
41
	Embryo

selection	does	not	require	a	deep	understanding	of	the	causal	pathways	by	which
genes,	in	complicated	interplay	with	environments,	produce	phenotypes:	it
requires	only	(lots	of)	data	on	the	genetic	correlates	of	the	traits	of	interest.
It	is	possible	to	calculate	some	rough	estimates	of	the	magnitude	of	the	gains
obtainable	in	different	selection	scenarios.
42
	
Table	5
	shows	expected	increases	in
intelligence	resulting	from	various	amounts	of	selection,	assuming	complete
information	about	the	common	additive	genetic	variants	underlying	the	narrow-
sense	heritability	of	intelligence.	(With	partial	information,	the	effectiveness	of
selection	would	be	reduced,	though	not	quite	to	the	extent	one	might	naively
expect.
44
)	Unsurprisingly,	selecting	between	larger	numbers	of	embryos
produces	larger	gains,	but	there	are	steeply	diminishing	returns:	selection
between	100	embryos	does	not	produce	a	gain	anywhere	near	fifty	times	as	large
as	that	which	one	would	get	from	selection	between	2	embryos.
45
Table	5

expect.
44
45
Table	5	
Maximum	IQ	gains	from	selecting	among	a	set	of	embryos
43
	
Selection
IQ	points	gained
1	in	2
4.2
1	in	10
11.5
1	in	100
18.8
1	in	1000
24.3
5	generations	of	1	in	10
<	65	(b/c	diminishing	returns)

10	generations	of	1	in	10
<	130	(b/c	diminishing	returns)
Cumulative	limits	(additive	variants	optimized
for	cognition)
100	+	(<	300	(b/c	diminishing
returns))
Interestingly,	the	diminishment	of	returns	is	greatly	abated	when	the	selection
is	spread	over	multiple	generations.	Thus,	repeatedly	selecting	the	top	1	in	10
over	ten	generations	(where	each	new	generation	consists	of	the	offspring	of
those	selected	in	the	previous	generation)	will	produce	a	much	greater	increase
in	the	trait	value	than	a	one-off	selection	of	1	in	100.	The	problem	with
sequential	selection,	of	course,	is	that	it	takes	longer.	If	each	generational	step
takes	twenty	or	thirty	years,	then	even	just	five	successive	generations	would
push	us	well	into	the	twenty-second	century.	Long	before	then,	more	direct	and
powerful	modes	of	genetic	engineering	(not	to	mention	machine	intelligence)
will	most	likely	be	available.
There	is,	however,	a	complementary	technology,	one	which,	once	it	has	been

developed	for	use	in	humans,	would	greatly	potentiate	the	enhancement	power
of	pre-implantation	genetic	screening:	namely,	the	derivation	of	viable	sperm
and	eggs	from	embryonic	stem	cells.
46
	The	techniques	for	this	have	already	been
used	to	produce	fertile	offspring	in	mice	and	gamete-like	cells	in	humans.
Substantial	scientific	challenges	remain,	however,	in	translating	the	animal
results	to	humans	and	in	avoiding	epigenetic	abnormalities	in	the	derived	stem

46
cell	lines.	According	to	one	expert,	these	challenges	might	put	human
application	“10	or	even	50	years	in	the	future.”
47
With	stem	cell-derived	gametes,	the	amount	of	selection	power	available	to	a
couple	could	be	greatly	increased.	In	current	practice,	an	in	vitro	fertilization
procedure	typically	involves	the	creation	of	fewer	than	ten	embryos.	With	stem
cell-derived	gametes,	a	few	donated	cells	might	be	turned	into	a	virtually
unlimited	number	of	gametes	that	could	be	combined	to	produce	embryos,	which

47
could	then	be	genotyped	or	sequenced,	and	the	most	promising	one	chosen	for
implantation.	Depending	on	the	cost	of	preparing	and	screening	each	individual
embryo,	this	technology	could	yield	a	severalfold	increase	in	the	selective	power
available	to	couples	using	in	vitro	fertilization.
More	importantly	still,	stem	cell-derived	gametes	would	allow	multiple

generations	of	selection	to	be	compressed	into	less	than	a	human	maturation
period,	by	enabling	
iterated	embryo	selection
.	This	is	a	procedure	that	would
consist	of	the	following	steps:
48
	
1
	Genotype	and	select	a	number	of	embryos	that	are	higher	in	desired	genetic
characteristics.
2
	Extract	stem	cells	from	those	embryos	and	convert	them	to	sperm	and	ova,
maturing	within	six	months	or	less.
49
3
	Cross	the	new	sperm	and	ova	to	produce	embryos.
4
	Repeat	until	large	genetic	changes	have	been	accumulated.
In	this	manner,	it	would	be	possible	to	accomplish	ten	or	more	generations	of
selection	in	just	a	few	years.	(The	procedure	would	be	time-consuming	and
expensive;	however,	in	principle,	it	would	need	to	be	done	only	once	rather	than
repeated	for	each	birth.	The	cell	lines	established	at	the	end	of	the	procedure
could	be	used	to	generate	very	large	numbers	of	enhanced	embryos.)
As	
Table	5
	indicates,	the	
average
	level	of	intelligence	among	individuals

As	
Table	5
average
conceived	in	this	manner	could	be	very	high,	possibly	equal	to	or	somewhat
above	that	of	the	most	intelligent	individual	in	the	historical	human	population.
A	world	that	had	a	large	population	of	such	individuals	might	(if	it	had	the
culture,	education,	communications	infrastructure,	etc.,	to	match)	constitute	a
collective	superintelligence.
The	impact	of	this	technology	will	be	dampened	and	delayed	by	several
factors.	There	is	the	unavoidable	maturational	lag	while	the	finally	selected

embryos	grow	into	adult	human	beings:	at	least	twenty	years	before	an	enhanced
child	reaches	full	productivity,	longer	still	before	such	children	come	to
constitute	a	substantial	segment	of	the	labor	force.	Furthermore,	even	after	the
technology	has	been	perfected,	adoption	rates	will	probably	start	out	low.	Some
countries	might	prohibit	its	use	altogether,	on	moral	or	religious	grounds.
50
	Even
where	selection	is	allowed,	many	couples	will	prefer	the	natural	way	of

50
	Even
conceiving.	Willingness	to	use	IVF,	however,	would	increase	if	there	were
clearer	benefits	associated	with	the	procedure—such	as	a	virtual	guarantee	that
the	child	would	be	highly	talented	and	free	from	genetic	predispositions	to

disease.	Lower	health	care	costs	and	higher	expected	lifetime	earnings	would
also	argue	in	favor	of	genetic	selection.	As	use	of	the	procedure	becomes	more
common,	particularly	among	social	elites,	there	might	be	a	cultural	shift	toward
parenting	norms	that	present	the	use	of	selection	as	the	thing	that	responsible
enlightened	couples	do.	Many	of	the	initially	reluctant	might	join	the	bandwagon
in	order	to	have	a	child	that	is	not	at	a	disadvantage	relative	to	the	enhanced
children	of	their	friends	and	colleagues.	Some	countries	might	offer	inducements
to	encourage	their	citizens	to	take	advantage	of	genetic	selection	in	order	to
increase	the	country’s	stock	of	human	capital,	or	to	increase	long-term	social
stability	by	selecting	for	traits	like	docility,	obedience,	submissiveness,
conformity,	risk-aversion,	or	cowardice,	outside	of	the	ruling	clan.
Effects	on	intellectual	capacity	would	also	depend	on	the	extent	to	which	the
available	selection	power	would	be	used	for	enhancing	cognitive	traits	(
Table	6
).

Table	6
).
Those	who	do	opt	to	use	some	form	of	embryo	selection	would	have	to	choose
how	to	allocate	the	selection	power	at	their	disposal,	and	intelligence	would	to
some	extent	be	in	competition	with	other	desired	attributes,	such	as	health,
beauty,	personality,	or	athleticism.	Iterated	embryo	selection,	by	offering	such	a
large	amount	of	selection	power,	would	alleviate	some	of	these	tradeoffs,
enabling	simultaneous	strong	selection	for	multiple	traits.	However,	this
procedure	would	tend	to	disrupt	the	normal	genetic	relationship	between	parents

and	child,	something	that	could	negatively	affect	demand	in	many	cultures.
51
With	further	advances	in	genetic	technology,	it	may	become	possible	to
synthesize	genomes	to	specification,	obviating	the	need	for	large	pools	of
embryos.	DNA	synthesis	is	already	a	routine	and	largely	automated
biotechnology,	though	it	is	not	yet	feasible	to	synthesize	an	entire	human
genome	that	could	be	used	in	a	reproductive	context	(not	least	because	of	still-
unresolved	difficulties	in	getting	the	epigenetics	right).
54
	But	once	this

51
54
technology	has	matured,	an	embryo	could	be	designed	with	the	exact	preferred
combination	of	genetic	inputs	from	each	parent.	Genes	that	are	present	in	neither
of	the	parents	could	also	be	spliced	in,	including	alleles	that	are	present	with	low
frequency	in	the	population	but	which	may	have	significant	positive	effects	on
cognition.
55
Table	6	
Possible	impacts	from	genetic	selection	in	different	scenarios
52

One	intervention	that	becomes	possible	when	human	genomes	can	be
synthesized	is	genetic	“spell-checking”	of	an	embryo.	(Iterated	embryo	selection
might	also	allow	an	approximation	of	this.)	Each	of	us	currently	carries	a
mutational	load,	with	perhaps	hundreds	of	mutations	that	reduce	the	efficiency
of	various	cellular	processes.
56
	Each	individual	mutation	has	an	almost
negligible	effect	(whence	it	is	only	slowly	removed	from	the	gene	pool),	yet	in
combination	such	mutations	may	exact	a	heavy	toll	on	our	functioning.
57
Individual	differences	in	intelligence	might	to	a	significant	extent	be	attributable
to	variations	in	the	number	and	nature	of	such	slightly	deleterious	alleles	that
each	of	us	carries.	With	gene	synthesis	we	could	take	the	genome	of	an	embryo
and	construct	a	version	of	that	genome	free	from	the	genetic	noise	of
accumulated	mutations.	If	one	wished	to	speak	provocatively,	one	could	say	that
individuals	created	from	such	proofread	genomes	might	be	“more	human”	than

57
anybody	currently	alive,	in	that	they	would	be	less	distorted	expressions	of
human	form.	Such	people	would	not	all	be	carbon	copies,	because	humans	vary
genetically	in	ways	other	than	by	carrying	different	deleterious	mutations.	But
the	phenotypical	manifestation	of	a	proofread	genome	may	be	an	exceptional
physical	and	mental	constitution,	with	elevated	functioning	in	polygenic	trait
dimensions	like	intelligence,	health,	hardiness,	and	appearance.
58
	(A	loose
analogy	could	be	made	with	composite	faces,	in	which	the	defects	of	the

58
	(A	loose
superimposed	individuals	are	averaged	out:	see	
Figure	6
.)

Figure	6
	Composite	faces	as	a	metaphor	for	spell-checked	genomes.	Each	of	the
central	pictures	was	produced	by	superimposing	photographs	of	sixteen	different
individuals	(residents	of	Tel	Aviv).	Composite	faces	are	often	judged	to	be	more
beautiful	than	any	of	the	individual	faces	of	which	they	are	composed,	as
idiosyncratic	imperfections	are	averaged	out.	Analogously,	by	removing
individual	mutations,	proofread	genomes	may	produce	people	closer	to	“Platonic
ideals.”	Such	individuals	would	not	all	be	genetically	identical,	because	many
genes	come	in	multiple	equally	functional	alleles.	Proofreading	would	only
eliminate	variance	arising	from	deleterious	mutations.
59
	
Other	potential	biotechnological	techniques	might	also	be	relevant.	Human
reproductive	cloning,	once	achieved,	could	be	used	to	replicate	the	genome	of
exceptionally	talented	individuals.	Uptake	would	be	limited	by	the	preference	of
most	prospective	parents	to	be	biologically	related	to	their	children,	yet	the

59
	
practice	could	nevertheless	come	to	have	non-negligible	impact	because	(1)	even
a	relatively	small	increase	in	the	number	of	exceptionally	talented	people	might
have	a	significant	effect;	and	(2)	it	is	possible	that	some	state	would	embark	on	a
larger-scale	eugenics	program,	perhaps	by	paying	surrogate	mothers.	Other	kinds
of	genetic	engineering—such	as	the	design	of	novel	synthetic	genes	or	insertion
into	the	genome	of	promoter	regions	and	other	elements	to	control	gene
expression—might	also	become	important	over	time.	Even	more	exotic

possibilities	may	exist,	such	as	vats	full	of	complexly	structured	cultured	cortical
tissue,	or	“uplifted”	transgenic	animals	(perhaps	some	large-brained	mammal
such	as	the	whale	or	elephant,	enriched	with	human	genes).	These	latter	ones	are
wholly	speculative,	but	over	a	longer	time	frame	they	perhaps	cannot	be
completely	discounted.

So	far	we	have	discussed	germline	interventions,	ones	that	would	be	done	on
gametes	or	embryos.	Somatic	gene	enhancements,	by	bypassing	the	generation
cycle,	could	in	principle	produce	impacts	more	quickly.	However,	they	are
technologically	much	more	challenging.	They	require	that	the	modified	genes	be
inserted	into	a	large	number	of	cells	in	the	living	body—including,	in	the	case	of
cognitive	enhancement,	the	brain.	Selecting	among	existing	egg	cells	or
embryos,	in	contrast,	requires	no	gene	insertion.	Even	such	germline	therapies	as
do	involve	modifying	the	genome	(such	as	proofreading	the	genome	or	splicing
in	rare	alleles)	are	far	easier	to	implement	at	the	gamete	or	the	embryo	stage,
where	one	is	dealing	with	a	small	number	of	cells.	Furthermore,	germline
interventions	on	embryos	can	probably	achieve	greater	effects	than	somatic
interventions	on	adults,	because	the	former	would	be	able	to	shape	early	brain
development	whereas	the	latter	would	be	limited	to	tweaking	an	existing

structure.	(Some	of	what	could	be	done	through	somatic	gene	therapy	might	also
be	achievable	by	pharmacological	means.)
Focusing	therefore	on	germline	interventions,	we	must	take	into	account	the
generational	lag	delaying	any	large	impact	on	the	world.
60
	Even	if	the
technology	were	perfected	today	and	immediately	put	to	use,	it	would	take	more
than	two	decades	for	a	genetically	enhanced	brood	to	reach	maturity.
Furthermore,	with	human	applications	there	is	normally	a	delay	of	at	least	one

60
	Even	if	the
decade	between	proof	of	concept	in	the	laboratory	and	clinical	application,
because	of	the	need	for	extensive	studies	to	determine	safety.	The	simplest	forms
of	genetic	selection,	however,	could	largely	abrogate	the	need	for	such	testing,
since	they	would	use	standard	fertility	treatment	techniques	and	genetic
information	to	choose	between	embryos	that	might	otherwise	have	been	selected
by	chance.
Delays	could	also	result	from	obstacles	rooted	not	in	a	fear	of	failure	(demand

by	chance.
for	safety	testing)	but	in	fear	of	success—demand	for	regulation	driven	by
concerns	about	the	moral	permissibility	of	genetic	selection	or	its	wider	social
implications.	Such	concerns	are	likely	to	be	more	influential	in	some	countries
than	in	others,	
owing	to	differing	cultural,	historical,	and	religious	contexts.
Post-war	Germany,	for	example,	has	chosen	to	give	a	wide	berth	to	any
reproductive	practices	that	could	be	perceived	to	be	even	in	the	remotest	way

aimed	at	enhancement,	a	stance	that	is	understandable	given	the	particularly	dark
history	of	atrocities	connected	to	the	eugenics	movement	in	that	country.	Other
Western	countries	are	likely	to	take	a	more	liberal	approach.	And	some	countries
—perhaps	China	or	Singapore,	both	of	which	have	long-term	population	policies

—might	not	only	permit	but	actively	promote	the	use	of	genetic	selection	and
genetic	engineering	to	enhance	the	intelligence	of	their	populations	once	the
technology	to	do	so	is	available.
Once	the	example	has	been	set,	and	the	results	start	to	show,	holdouts	will
have	strong	incentives	to	follow	suit.	Nations	would	face	the	prospect	of
becoming	cognitive	backwaters	and	losing	out	in	economic,	scientific,	military,
and	prestige	contests	with	competitors	that	embrace	the	new	human
enhancement	technologies.	Individuals	within	a	society	would	see	places	at	elite
schools	being	filled	with	genetically	selected	children	(who	may	also	on	average
be	prettier,	healthier,	and	more	conscientious)	and	will	want	their	own	offspring
to	have	the	same	advantages.	There	is	some	chance	that	a	large	attitudinal	shift
could	take	place	over	a	relatively	short	time,	perhaps	in	as	little	as	a	decade,
once	the	technology	is	proven	to	work	and	to	provide	a	substantial	benefit.

Opinion	surveys	in	the	United	States	reveal	a	dramatic	shift	in	public	approval	of
in	vitro	fertilization	after	the	birth	of	the	first	“test	tube	baby,”	Louise	Brown,	in
1978.	A	few	years	earlier,	only	18%	of	Americans	said	they	would	personally
use	IVF	to	treat	infertility;	yet	in	a	poll	taken	shortly	after	the	birth	of	Louise
Brown,	53%	said	they	would	do	so,	and	the	number	has	continued	to	rise.
61
	(For
comparison,	in	a	poll	taken	in	2004,	28%	of	Americans	approved	of	embryo

61
	(For
selection	for	“strength	or	intelligence,”	58%	approved	of	it	for	avoiding	adult-
onset	cancer,	and	68%	approved	of	it	to	avoid	fatal	childhood	disease.
62
)
If	we	add	up	the	various	delays—say	five	to	ten	years	to	gather	the
information	needed	for	significantly	effective	selection	among	a	set	of	IVF
embryos	(possibly	much	longer	before	stem	cell-derived	gametes	are	available
for	use	in	human	reproduction),	ten	years	to	build	significant	uptake,	and	twenty

62
)
to	twenty-five	years	for	the	enhanced	generation	to	reach	an	age	where	they	start
becoming	productive,	we	find	that	germline	enhancements	are	unlikely	to	have	a
significant	impact	on	society	before	the	middle	of	this	century.	From	that	point
onward,	however,	the	intelligence	of	significant	segments	of	the	adult	population
may	begin	to	be	boosted	by	genetic	enhancements.	The	speed	of	the	ascent
would	then	greatly	accelerate	as	cohorts	conceived	using	more	powerful	next-
generation	genetic	technologies	(in	particular	stem	cell-derived	gametes	and

iterative	embryo	selection)	enter	the	labor	force.
With	the	full	development	of	the	genetic	technologies	described	above	(setting
aside	the	more	exotic	possibilities	such	as	intelligence	in	cultured	neural	tissue),
it	might	be	possible	to	ensure	that	new	individuals	are	on	average	smarter	than

any	human	who	has	yet	existed,	with	peaks	that	rise	higher	still.	The	potential	of
biological	enhancement	is	thus	ultimately	high,	probably	sufficient	for	the
attainment	of	at	least	weak	forms	of	superintelligence.	This	should	not	be
surprising.	After	all,	dumb	evolutionary	processes	have	dramatically	amplified
the	intelligence	in	the	human	lineage	even	compared	with	our	close	relatives	the
great	apes	and	our	own	humanoid	ancestors;	and	there	is	no	reason	to	suppose
	to	have	reached	the	apex	of	cognitive	effectiveness	attainable	in	a
biological	system.	Far	from	being	the	smartest	possible	biological	species,	we
are	probably	better	thought	of	as	the	stupidest	possible	biological	species
capable	of	starting	a	technological	civilization—a	niche	we	filled	because	we	got
there	first,	not	because	we	are	in	any	sense	optimally	adapted	to	it.
Progress	along	the	biological	path	is	clearly	feasible.	The	generational	lag	in
germline	interventions	means	that	progress	could	not	be	nearly	as	sudden	and

abrupt	as	in	scenarios	involving	machine	intelligence.	(Somatic	gene	therapies
and	pharmacological	interventions	could	theoretically	skip	the	generational	lag,
but	they	seem	harder	to	perfect	and	are	less	likely	to	produce	dramatic	effects.)
The	
ultimate
	potential	of	machine	intelligence	is,	of	course,	vastly	greater	than
that	of	organic	intelligence.	(One	can	get	some	sense	of	the	magnitude	of	the	gap
by	considering	the	speed	differential	between	electronic	components	and	nerve

The	
ultimate
cells:	even	today’s	transistors	operate	on	a	timescale	ten	million	times	shorter
than	that	of	biological	neurons.)	However,	even	comparatively	moderate
enhancements	of	biological	cognition	could	have	important	consequences.	In
particular,	cognitive	enhancement	could	accelerate	science	and	technology,
including	progress	toward	more	potent	forms	of	biological	intelligence
amplification	and	machine	intelligence.	Consider	how	the	rate	of	progress	in	the
field	of	artificial	intelligence	would	change	in	a	world	where	Average	Joe	is	an

intellectual	peer	of	Alan	Turing	or	John	von	Neumann,	and	where	millions	of
people	tower	far	above	any	intellectual	giant	of	the	past.
63
A	discussion	of	the	strategic	implications	of	cognitive	enhancement	will	have
to	await	a	later	chapter.	But	we	can	summarize	this	section	by	noting	three
conclusions:	(1)	at	least	weak	forms	of	superintelligence	are	achievable	by
means	of	biotechnological	enhancements;	(2)	the	feasibility	of	cognitively
enhanced	humans	adds	to	the	plausibility	that	advanced	forms	of	machine
intelligence	are	feasible—because	even	if	
we

63
we
	were	fundamentally	unable	to
create	machine	intelligence	(which	there	is	no	reason	to	suppose),	machine
intelligence	might	still	be	within	reach	of	cognitively	enhanced	humans;	and	(3)
when	we	consider	scenarios	stretching	significantly	into	the	second	half	of	this

century	and	beyond,	we	must	take	into	account	the	probable	emergence	of	a
generation	of	genetically	enhanced	populations—voters,	inventors,	scientists—
with	the	magnitude	of	enhancement	escalating	rapidly	over	subsequent	decades.
	
It	is	sometimes	proposed	that	direct	brain–computer	interfaces,	particularly
implants,	could	enable	humans	to	exploit	the	fortes	of	digital	computing—
perfect	
recall,	speedy	and	accurate	arithmetic	calculation,	and	high-bandwidth
data	transmission—enabling	the	resulting	hybrid	system	to	radically	outperform
the	unaugmented	brain.
64
	But	although	the	possibility	of	direct	connections
between	human	brains	and	computers	has	been	demonstrated,	it	seems	unlikely
that	such	interfaces	will	be	widely	used	as	enhancements	any	time	soon.
65
To	begin	with,	there	are	significant	risks	of	medical	complications—including
infections,	electrode	displacement,	hemorrhage,	and	cognitive	decline—when

64
65
implanting	electrodes	in	the	brain.	Perhaps	the	most	vivid	illustration	to	date	of
the	benefits	that	can	be	obtained	through	brain	stimulation	is	the	treatment	of
patients	with	Parkinson’s	disease.	The	Parkinson’s	implant	is	relatively	simple:	it
does	not	really	communicate	with	the	brain	but	simply	supplies	a	stimulating
electric	current	to	the	subthalamic	nucleus.	A	demonstration	video	shows	a
subject	slumped	in	a	chair,	completely	immobilized	by	the	disease,	then
suddenly	springing	to	life	when	the	current	is	switched	on:	the	subject	now

moves	his	arms,	stands	up	and	walks	across	the	room,	turns	around	and	performs
a	pirouette.	Yet	even	behind	this	especially	simple	and	almost	miraculously
successful	procedure,	there	lurk	negatives.	One	study	of	Parkinson	patients	who
had	received	deep	brain	implants	showed	reductions	in	verbal	fluency,	selective
attention,	color	naming,	and	verbal	memory	compared	with	controls.	Treated
subjects	also	reported	more	cognitive	complaints.
66
	Such	risks	and	side	effects
might	be	tolerable	if	the	procedure	is	used	to	alleviate	severe	disability.	But	in

66
order	for	healthy	subjects	to	volunteer	themselves	for	neurosurgery,	there	would
have	to	be	some	very	substantial	enhancement	of	normal	functionality	to	be
gained.
This	brings	us	to	the	second	reason	to	doubt	that	superintelligence	will	be
achieved	through	cyborgization,	namely	that	enhancement	is	likely	to	be	far
more	difficult	than	therapy.	Patients	who	suffer	from	paralysis	might	benefit

from	an	implant	that	replaces	their	severed	nerves	or	activates	spinal	motion
pattern	generators.
67
	Patients	who	are	deaf	or	blind	might	benefit	from	artificial
cochleae	and	retinas.
68
	Patients	with	Parkinson’s	disease	or	chronic	pain	might
benefit	from	deep	brain	stimulation	that	excites	or	inhibits	activity	in	a	particular
area	of	the	brain.
69
	What	seems	far	more	difficult	to	achieve	is	a	high-bandwidth
direct	interaction	between	brain	and	computer	to	provide	substantial	increases	in
intelligence	of	a	form	that	could	not	be	more	readily	attained	by	other	means.
Most	of	the	potential	benefits	that	brain	implants	could	provide	in	healthy
subjects	could	be	obtained	at	far	less	risk,	expense,	and	inconvenience	by	using
our	regular	motor	and	sensory	organs	to	interact	with	computers	located	outside
of	our	bodies.	We	do	not	need	to	plug	a	fiber	optic	cable	into	our	brains	in	order
to	access	the	Internet.	Not	only	can	the	human	retina	transmit	data	at	an

impressive	rate	of	nearly	10	million	bits	per	second,	but	it	comes	pre-packaged
with	a	massive	amount	of	dedicated	wetware,	the	visual	cortex,	that	is	highly
adapted	to	extracting	meaning	from	this	information	torrent	and	to	interfacing
with	other	brain	areas	for	further	processing.
70
	Even	if	there	were	an	easy	way	of
pumping	more	information	into	our	brains,	the	extra	data	inflow	would	do	little
to	increase	the	rate	at	which	we	think	and	learn	unless	all	the	neural	machinery
necessary	for

70
necessary	for	
making	sense	of	the	data	were	similarly	upgraded.	Since	this
includes	almost	all	of	the	brain,	what	would	really	be	needed	is	a	“whole	brain
prosthesis–—which	is	just	another	way	of	saying	artificial	general	intelligence.
Yet	if	one	had	a	human-level	AI,	one	could	dispense	with	neurosurgery:	a
computer	might	as	well	have	a	metal	casing	as	one	of	bone.	So	this	limiting	case
just	takes	us	back	to	the	AI	path,	which	we	have	already	examined.
Brain–computer	interfacing	has	also	been	proposed	as	a	way	to	get

information	out	of	the	brain,	for	purposes	of	communicating	with	other	brains	or
with	machines.
71
	Such	uplinks	have	helped	patients	with	locked-in	syndrome	to
communicate	with	the	outside	world	by	enabling	them	to	move	a	cursor	on	a
screen	by	thought.
72
	The	bandwidth	attained	in	such	experiments	is	low:	the
patient	painstakingly	types	out	one	slow	letter	after	another	at	a	rate	of	a	few
words	per	minute.	One	can	readily	imagine	improved	versions	of	this	technology

71
72
—perhaps	a	next-generation	implant	could	plug	into	Broca’s	area	(a	region	in	the
frontal	lobe	involved	in	language	production)	and	pick	up	internal	speech.
73
	But
whilst	such	a	technology	might	assist	some	people	with	disabilities	induced	by
stroke	or	muscular	degeneration,	it	would	hold	little	appeal	for	healthy	subjects.
The	functionality	it	would	provide	is	essentially	that	of	a	microphone	coupled
with	speech	recognition	software,	which	is	already	commercially	available—

minus	the	pain,	inconvenience,	expense,	and	risks	associated	with	neurosurgery
(and	minus	at	least	some	of	the	hyper-Orwellian	overtones	of	an	intracranial
listening	device).	Keeping	our	machines	outside	of	our	bodies	also	makes
upgrading	easier.
But	what	about	the	dream	of	bypassing	words	altogether	and	establishing	a
connection	between	two	brains	that	enables	concepts,	thoughts,	or	entire	areas	of
expertise	to	be	“downloaded”	from	one	mind	to	another?	We	can	download	large
files	to	our	computers,	including	libraries	with	millions	of	books	and	articles,
and	this	can	be	done	over	the	course	of	seconds:	could	something	similar	be
done	with	our	brains?	The	apparent	plausibility	of	this	idea	probably	derives
from	an	incorrect	view	of	how	information	is	stored	and	represented	in	the	brain.
As	noted,	the	rate-limiting	step	in	human	intelligence	is	not	how	fast	raw	data
can	be	fed	into	the	brain	but	rather	how	quickly	the	brain	can	extract	meaning

and	make	sense	of	the	data.	Perhaps	it	will	be	suggested	that	we	transmit
meanings	directly,	rather	than	package	them	into	sensory	data	that	must	be
decoded	by	the	recipient.	There	are	two	problems	with	this.	The	first	is	that
brains,	by	contrast	to	the	kinds	of	program	we	typically	run	on	our	computers,	do
not	use	standardized	data	storage	and	representation	formats.	Rather,	each	brain
develops	its	own	idiosyncratic	representations	of	higher-level	content.	Which
particular	neuronal	assemblies	are	recruited	to	represent	a	particular	concept

depends	on	the	unique	experiences	of	the	brain	in	question	(along	with	various
genetic	factors	and	stochastic	physiological	processes).	Just	as	in	artificial	neural
nets,	meaning	in	biological	neural	networks	is	likely	represented	holistically	in
the	structure	and	activity	patterns	of	sizeable	overlapping	regions,	not	in	discrete
memory	cells	laid	out	in	neat	arrays.
74
	It	would	therefore	not	be	possible	to
establish	a	simple	mapping	between	the	neurons	in	one	brain	and	those	in
another	in	such	a	way	that	thoughts	could

74
another	in	such	a	way	that	thoughts	could	
automatically	slide	over	from	one	to
the	other.	In	order	for	the	thoughts	of	one	brain	to	be	intelligible	to	another,	the
thoughts	need	to	be	decomposed	and	packaged	into	symbols	according	to	some
shared	convention	that	allows	the	symbols	to	be	correctly	interpreted	by	the
receiving	brain.	This	is	the	job	of	language.
In	principle
,	one	could	imagine	offloading	the	cognitive	work	of	articulation
and	interpretation	to	an	interface	that	would	somehow	read	out	the	neural	states
in	the	sender’s	brain	and	somehow	feed	in	a	bespoke	pattern	of	activation	to	the

receiver’s	brain.	But	this	brings	us	to	the	second	problem	with	the	cyborg
scenario.	Even	setting	aside	the	(quite	immense)	technical	challenge	of	how	to
reliably	read	and	write	simultaneously	from	perhaps	billions	of	individually

addressable	neurons,	creating	the	requisite	interface	is	probably	an	AI-complete
problem.	The	interface	would	need	to	include	a	component	able	(in	real-time)	to
map	firing	patterns	in	one	brain	onto	semantically	equivalent	firing	patterns	in
the	other	brain.	The	detailed	multilevel	understanding	of	the	neural	computation
needed	to	accomplish	such	a	task	would	seem	to	directly	enable	neuromorphic
AI.
Despite	these	reservations,	the	cyborg	route	toward	cognitive	enhancement	is
not	entirely	without	promise.	Impressive	work	on	the	rat	hippocampus	has
demonstrated	the	feasibility	of	a	neural	prosthesis	that	can	enhance	performance
in	a	simple	working-memory	task.
75
	In	its	present	version,	the	implant	collects
input	from	a	dozen	or	two	electrodes	located	in	one	area	(“CA3”)	of	the
hippocampus	and	projects	onto	a	similar	number	of	neurons	in	another	area
(“CA1”).	A	microprocessor	is	trained	to	discriminate	between	two	different
firing	patterns	in	the	first	area	(corresponding	to	two	different	memories,	“right

75
lever”	or	“left	lever”)	and	to	learn	how	these	patterns	are	projected	into	the
second	area.	This	prosthesis	can	not	only	restore	function	when	the	normal
neural	connection	between	the	two	neural	areas	is	blockaded,	but	by	sending	an
especially	clear	token	of	a	particular	memory	pattern	to	the	second	area	it	can
enhance	the	performance	on	the	memory	task	beyond	what	the	rat	is	normally
capable	of.	While	a	technical	tour	de	force	by	contemporary	standards,	the	study
leaves	many	challenging	questions	unanswered:	How	well	does	the	approach

scale	to	greater	numbers	of	memories?	How	well	can	we	control	the
combinatorial	explosion	that	otherwise	threatens	to	make	learning	the	correct
mapping	infeasible	as	the	number	of	input	and	output	neurons	is	increased?	Does
the	enhanced	performance	on	the	test	task	come	at	some	hidden	cost,	such	as
reduced	ability	to	generalize	from	the	particular	stimulus	used	in	the	experiment,
or	reduced	ability	to	unlearn	the	association	when	the	environment	changes?
Would	the	test	subjects	still	somehow	benefit	even	if—unlike	rats—they	could

avail	themselves	of	external	memory	aids	such	as	pen	and	paper?	And	how
much	harder	would	it	be	to	apply	a	similar	method	to	other	parts	of	the	brain?
Whereas	the	present	prosthesis	takes	advantage	of	the	relatively	simple	feed-
forward	structure	of	parts	of	the	hippocampus	(basically	serving	as	a
unidirectional	bridge	between	areas	CA3	and	CA1),	other	structures	in	the	cortex
involve	convoluted	feedback	loops	which	greatly	increase	the	complexity	of	the
wiring	diagram	and,	presumably,	the	difficulty	of	deciphering	the	functionality

of	any	embedded	group	of	neurons.
One	hope	for	the	cyborg	route	is	that	the	brain,	if	permanently	implanted	with

a	device	connecting	it	to	some	external	resource,	would	over	time	
learn
	an
effective	mapping	between	its	own	internal	cognitive	states	and	the	inputs	it
receives	from,	or	the	outputs	accepted	by,	the	device.	Then	the	implant	itself
would	not	need	to	be	intelligent;	rather,	the	brain	would	intelligently	adapt	to	the
interface,	much	as	the	brain	of	an	infant	gradually	learns	to	interpret	the	signals
arriving	from	receptors	in	its	eyes	and	ears.
76
	But	here	again	one	must	question
how	much	would	really	be	gained.	Suppose	that	the	brain’s	plasticity	were	such
that	it	could	learn	to	detect	patterns	in	some	new	input	stream	arbitrary	projected
onto	some	part	of	the	cortex	by	means	of	a	brain–computer	interface:	why	not
project	the	same	information	onto	the	retina	instead,	as	a	visual	pattern,	or	onto
the	cochlea	as	sounds?	The	low-tech	alternative	avoids	a	thousand
complications,	and	in	either	case	the	brain	could	deploy	its	pattern-recognition
mechanisms	and	plasticity	to	learn	to	make	sense	of	the	information.

Another	conceivable	path	to	superintelligence	is	through	the	gradual
enhancement	of	networks	and	organizations	that	link	individual	human	minds
with	one	another	and	with	various	artifacts	and	bots.	The	idea	here	is	not	that
this	would	enhance	the	intellectual	capacity	of	individuals	enough	to	make	them
superintelligent,	but	rather	that	some	system	composed	of	individuals	thus
networked	and	organized	might	attain	a	form	of	superintelligence—what	in	the
next	chapter	we	will	elaborate	as	“collective	superintelligence.”
77

77
Humanity	has	gained	enormously	in	collective	intelligence	over	the	course	of
history	and	prehistory.	The	gains	come	from	many	sources,	including
innovations	in	communications	technology,	such	as	writing	and	printing,	and
above	all	the	introduction	of	language	itself;	increases	in	the	size	of	the	world
population	and	the	density	of	habitation;	various	improvements	in	organizational
techniques	and	epistemic	norms;	and	a	gradual	accumulation	of	institutional
capital.	In	general	terms,	a	system’s	collective	intelligence	is	limited	by	the

abilities	of	its	member	minds,	the	overheads	in	communicating	relevant
information	between	them,	and	the	various	distortions	and	inefficiencies	that
pervade	human	organizations.	If	communication	overheads	are	reduced
(including	not	only	equipment	costs	but	also	response	latencies,	time	and
attention	burdens,	and	other	factors),	then	larger	and	more	densely	connected

organizations	become	feasible.	The	same	could	happen	if	fixes	are	found	for
some	of	the	bureaucratic	deformations	that	warp	organizational	life—wasteful
status	games,	mission	creep,	concealment	or	falsification	of	information,	and
other	agency	problems.	Even	partial	solutions	to	these	problems	could	pay	hefty
dividends	for	collective	intelligence.
The	technological	and	institutional	innovations	that	could	contribute	to	the
growth	of	our	collective	intelligence	are	many	and	various.	For	example,
subsidized	prediction	markets	might	foster	truth-seeking	norms	and	improve
forecasting	on	contentious	scientific	and	social	issues.
78
	Lie	detectors	(should	it
prove	feasible	to	make	ones	that	are	reliable	and	easy	to	use)	could	reduce	the
scope	for	deception	in	human	affairs.
79
	Self-deception	detectors	might	be	even
more	powerful.
80
	Even	without	newfangled	brain	technologies,	some	forms	of
deception	might	become	harder	to	practice	thanks	to	increased	availability	of

78
79
80
many	kinds	of	data,	including	reputations	and	track	records,	or	the	promulgation
of	strong	epistemic	norms	and	rationality	culture.	Voluntary	and	involuntary
surveillance	will	amass	vast	amounts	of	information	about	human	behavior.
Social	networking	sites	are	already	used	by	over	a	billion	people	to	share
personal	details:	soon,	these	people	might	begin	uploading	continuous	life
recordings	from	microphones	and	video	cameras	embedded	in	their	smart
phones	or	eyeglass	frames.	Automated	analysis	of	such	data	streams	will	enable

many	new	applications	(sinister	as	well	as	benign,	of	course).
81
Growth	in	collective	intelligence	may	also	come	from	more	general
organizational	and	economic	improvements,	and	from	enlarging	the	fraction	of
the	world’s	population	that	is	educated,	digitally	connected,	and	integrated	into
global	intellectual	culture.
82
The	Internet	stands	out	as	a	particularly	dynamic	frontier	for	innovation	and
experimentation.	Most	of	its	potential	may	still	remain	unexploited.	Continuing
development	of	an	intelligent	Web,	with	better	support	for	deliberation,	de-

81
82
biasing,	and	judgment	aggregation,	might	make	large	contributions	to	increasing
the	collective	intelligence	of	humanity	as	a	whole	or	of	particular	groups.
But	what	of	the	seemingly	more	fanciful	idea	that	the	Internet	might	one	day
“wake	up”?	Could	the	Internet	become	something	more	than	just	the	backbone
of	a	loosely	integrated	collective	superintelligence—something	more	like	a
virtual	skull	housing	an	emerging	unified	superintellect?	(This	was	one	of	the

ways	that	superintelligence	could	arise	according	to	Vernor	Vinge’s	influential

1993	essay,	which	coined	the	term	“technological	singularity.”
83
)	Against	this
one	could	object	that	machine	intelligence	is	hard	enough	to	achieve	through
arduous	engineering,	and	that	it	is	incredible	to	suppose	that	it	will	arise
spontaneously
.	However,	the	story	need	not	be	that	some	future	version	of	the
Internet	suddenly	becomes	superintelligent	by	mere	happenstance.	A	more
plausible	version	of	the	scenario	would	be	that	the	Internet	accumulates
improvements	through	the	work	of	many	people	over	many	years—work	to
engineer	better	search	and	information	filtering	algorithms,	more	powerful	data
representation	formats,	more	capable	autonomous	software	agents,	and	more
efficient	protocols	governing	the	interactions	between	such	bots—and	that
myriad	incremental	improvements	eventually	create	the	basis	for	some	more
unified	form	of	web	intelligence.	It	seems	at	least	conceivable	that	such	a	web-
based	cognitive	system,	supersaturated	with	computer	power	and	all	other

resources	needed	for	explosive	growth	save	for	one	crucial	ingredient,	could,
when	the	final	missing	constituent	is	dropped	into	the	cauldron,	blaze	up	with
superintelligence.	This	type	of	scenario,	though,	converges	into	another	possible
path	to	superintelligence,	that	of	artificial	general	intelligence,	which	we	have
already	discussed.
Summary
	
The	fact	that	there	are	many	paths	that	lead	to	superintelligence	should	increase
our	confidence	that	we	will	eventually	get	there.	If	one	path	turns	out	to	be
blocked,	we	can	still	progress.

Summary
	
That	there	are	multiple	paths	does	not	entail	that	there	are	multiple
destinations.	Even	if	significant	intelligence	amplification	were	first	achieved
along	one	of	the	non-machine-intelligence	paths,	this	would	not	render	machine
intelligence	irrelevant.	Quite	the	contrary:	enhanced	biological	or	organizational
intelligence	would	accelerate	scientific	and	technological	developments,
potentially	hastening	the	arrival	of	more	radical	forms	of	intelligence
amplification	such	as	whole	brain	emulation	and	AI.

This	is	not	to	say	that	it	is	a	matter	of	indifference	how	we	get	to	machine
superintelligence.	The	path	taken	to	get	there	could	make	a	big	difference	to	the
eventual	outcome.	Even	if	the	ultimate	capabilities	that	are	obtained	do	not
depend	much	on	the	trajectory,	how	those	capabilities	will	be	used—how	much

control	we	humans	have	over	their	disposition—might	well	depend	on	details	of
our	approach.	For	example,	enhancements	of	biological	or	organizational
intelligence	might	increase	our	ability	to	anticipate	risk	and	to	design	machine
superintelligence	that	is	safe	and	beneficial.	(A	full	strategic	assessment	involves
many	complexities,	and	will	have	to	await	
Chapter	14
.)
True	superintelligence	(as	opposed	to	marginal	increases	in	current	levels	of
intelligence)	might	plausibly	first	be	attained	via	the	AI	path.	There	are,
however,	many	fundamental	uncertainties	along	this	path.	This	makes	it	difficult
to	rigorously	assess	how	long	the	path	is	or	how	many	obstacles	there	are	along
the	way.	The	whole	brain	emulation	path	also	has	some	chance	of	being	the
quickest	route	to	superintelligence.	Since	progress	along	this	path	requires
mainly	incremental	technological	advances	rather	than	theoretical
breakthroughs,	a	strong	case	can	be	made	that	it	will	eventually	succeed.	It

seems	fairly	likely,	however,	that	even	if	progress	along	the	whole	brain
emulation	path	is	swift,	artificial	intelligence	will	nevertheless	be	first	to	cross
the	finishing	line:	this	is	because	of	the	possibility	of	neuromorphic	AIs	based	on
partial	emulations.
Biological	cognitive	enhancements	are	clearly	feasible,	particularly	ones
based	on	genetic	selection.	Iterated	embryo	selection	currently	seems	like	an
especially	promising	technology.	Compared	with	possible	breakthroughs	in
machine	intelligence,	however,	biological	enhancements	would	be	relatively

slow	and	gradual.	They	would,	at	best,	result	in	relatively	weak	forms	of
superintelligence	(more	on	this	shortly).
The	clear	feasibility	of	biological	enhancement	should	increase	our
confidence	that	machine	intelligence	is	ultimately	achievable,	since	enhanced
human	scientists	and	engineers	will	be	able	to	make	more	and	faster	progress
than	their	
au	naturel
	counterparts.	Especially	in	scenarios	in	which	machine
intelligence	is	delayed	beyond	mid-century,	the	increasingly	cognitively

than	their	
au	naturel
enhanced	cohorts	coming	onstage	will	play	a	growing	role	in	subsequent
developments.
Brain–computer	interfaces	look	unlikely	as	a	source	of	superintelligence.
Improvements	in	networks	and	organizations	might	result	in	weakly
superintelligent	forms	of	collective	intelligence	in	the	long	run;	but	more	likely,
they	will	play	an	enabling	role	similar	to	that	of	biological	cognitive
enhancement,	gradually	increasing	humanity’s	effective	ability	to	solve

intellectual	problems.	Compared	with	biological	enhancements,	advances	in
networks	and	organization	will	make	a	difference	sooner—in	fact,	such	advances
are	occurring	continuously	and	are	having	a	significant	impact	already.	However,
improvements	in	networks	and	organizations	may	yield	narrower	increases	in
our	problem-solving	capacity	than	will	improvements	in	biological	cognition—
boosting	“collective	intelligence”	rather	than	“quality	intelligence,”	to	anticipate
a	distinction	we	are	about	to	introduce	in	the	next	chapter.

CHAPTER	3
Forms	of	superintelligence
	
So	what,	exactly,	do	we	mean	by	“superintelligence”?	While	we	do	not	wish
to	get	bogged	down	in	terminological	swamps,	something	needs	to	be	said	to
clarify	the	conceptual	ground.	This	chapter	identifies	three	different	forms
of	superintelligence,	and	argues	that	they	are,	in	a	practically	relevant	sense,
equivalent.	We	also	show	that	the	potential	for	intelligence	in	a	machine
substrate	is	vastly	greater	than	in	a	biological	substrate.	Machines	have	a
number	of	fundamental	advantages	which	will	give	them	overwhelming
superiority.	Biological	humans,	even	if	enhanced,	will	be	outclassed.
Many	machines	and	nonhuman	animals	already	perform	at	superhuman	levels	in
narrow	domains.	Bats	interpret	sonar	signals	better	than	man,	calculators
outperform	us	in	arithmetic,	and	chess	programs	beat	us	in	chess.	The	range	of
specific	tasks	that	can	be	better	performed	by	software	will	continue	to	expand.
But	although	specialized	information	processing	systems	will	have	many	uses,

there	are	additional	profound	issues	that	arise	only	with	the	prospect	of	machine
intellects	that	have	enough	general	intelligence	to	substitute	for	humans	across
the	board.
As	previously	indicated,	we	use	the	term	“superintelligence”	to	refer	to
intellects	that	greatly	outperform	the	best	current	human	minds	across	many	very
general	cognitive	domains.	This	is	still	quite	vague.	Different	kinds	of	system
with	rather	disparate	performance	attributes	could	qualify	as	superintelligences

the	board.
under	this	definition.	To	advance	the	analysis,	it	is	helpful	to	disaggregate	this
simple	notion	of	superintelligence	by	distinguishing	different	bundles	of
intellectual	super-capabilities.	There	are	many	ways	in	which	such
decomposition	could	be	done.	Here	we	will	differentiate	between	three	forms:
speed	superintelligence,	collective	superintelligence,	and	quality

A	speed	superintelligence	is	an	intellect	that	is	just	like	a	human	mind	but	faster.
This	is	conceptually	the	easiest	form	of	superintelligence	to	analyze.
1
	We	can
define	speed	superintelligence	as	follows:
Speed	superintelligence:
	
A	system	that	can	do	all	that	a	human	intellect
can	do,	but	much	faster
.
	
By	“much”	we	here	mean	something	like	“multiple	orders	of	magnitude.”	But
rather	than	try	to	expunge	every	remnant	of	vagueness	from	the	definition,	we
will	entrust	the	reader	with	interpreting	it	sensibly.
2
The	simplest	example	of	speed	superintelligence	would	be	a	whole	brain
emulation	running	on	fast	hardware.
3
	An	emulation	operating	at	a	speed	of	ten
thousand	times	that	of	a	biological	brain	would	be	able	to	read	a	book	in	a	few
seconds	and	write	a	PhD	thesis	in	an	afternoon.	With	a	speedup	factor	of	a
million,	an	emulation	could	accomplish	an	entire	millennium	of	intellectual
work	in	one	working	day.
4
To	such	a	fast	mind,	events	in	the	external	world	appear	to	unfold	in	slow

2
3
4
motion.	Suppose	your	mind	ran	at	10,000×.	If	your	fleshly	friend	should	happen
to	drop	his	teacup,	you	could	watch	the	porcelain	slowly	descend	toward	the
carpet	over	the	course	of	several	hours,	like	a	comet	silently	gliding	through
space	toward	an	assignation	with	a	far-off	planet;	and,	as	the	anticipation	of	the
coming	crash	tardily	propagates	through	the	folds	of	your	friend’s	gray	matter
and	from	thence	out	into	his	peripheral	nervous	system,	you	could	observe	his

body	gradually	assuming	the	aspect	of	a	frozen	oops—enough	time	for	you	not
only	to	order	a	replacement	cup	but	also	to	read	a	couple	of	scientific	papers	and
take	a	nap.
Because	of	this	apparent	time	dilation	of	the	material	world,	a	speed
superintelligence	would	prefer	to	work	with	digital	objects.	It	could	live	in
virtual	reality	and	deal	in	the	information	economy.	Alternatively,	it	could
interact	with	the	physical	environment	by	means	of	nanoscale	manipulators,
since	limbs	at	such	small	scales	could	operate	faster	than	macroscopic

take	a	nap.
appendages.	(The	characteristic	frequency	of	a	system	tends	to	be	inversely

proportional	to	its	length	scale.
5
)	A	fast	mind	might	commune	mainly	with	other
fast	minds	rather	than	with	bradytelic,	molasses-like	humans.
The	speed	of	light	becomes	an	increasingly	important	constraint	as	minds	get
faster,	since	faster	minds	face	greater	opportunity	costs	in	the	use	of	their	time
for	traveling	or	communicating	over	long	distances.
6
	Light	is	roughly	a	million
times	faster	than	a	jet	plane,	so	it	would	take	a	digital	agent	with	a	mental
speedup	of	1,000,000×	about	the	same	amount	of	subjective	time	to	travel	across
the	globe	as	it	does	a	contemporary	human	journeyer.	Dialing	somebody	long
distance	would	take	as	long	as	getting	there	“in	person,”	though	it	would	be
cheaper	as	a	call	would	
require	less	bandwidth.	Agents	with	large	mental
speedups	who	want	to	converse	extensively	might	find	it	advantageous	to	move
near	one	another.	Extremely	fast	minds	with	need	for	frequent	interaction	(such
as	members	of	a	work	team)	may	take	up	residence	in	computers	located	in	the

same	building	to	avoid	frustrating	latencies.
	
Another	form	of	superintelligence	is	a	system	achieving	superior	performance	by
aggregating	large	numbers	of	smaller	intelligences:
Collective	superintelligence:
	
A	system	composed	of	a	large	number	of
smaller	intellects	such	that	the	system’s	overall	performance	across	many
very	general	domains	vastly	outstrips	that	of	any	current	cognitive	system
.
	
Collective	superintelligence	is	less	conceptually	clear-cut	than	speed
7

.
	
7
	However,	it	is	more	familiar	empirically.	While	we	have	no
experience	with	human-level	minds	that	differ	significantly	in	clock	speed,	we
do
	have	ample	experience	with	collective	intelligence,	systems	composed	of
various	numbers	of	human-level	components	working	together	with	various
degrees	of	efficiency.	Firms,	work	teams,	gossip	networks,	advocacy	groups,
academic	communities,	countries,	even	humankind	as	a	whole,	can—if	we	adopt
a	somewhat	abstract	perspective—be	viewed	as	loosely	defined	“systems”

7
do
capable	of	solving	classes	of	intellectual	problems.	From	experience,	we	have

some	sense	of	how	easily	different	tasks	succumb	to	the	efforts	of	organizations
of	various	size	and	composition.
Collective	intelligence	excels	at	solving	problems	that	can	be	readily	broken
into	parts	such	that	solutions	to	sub-problems	can	be	pursued	in	parallel	and
verified	independently.	Tasks	like	building	a	space	shuttle	or	operating	a
hamburger	franchise	offer	myriad	opportunities	for	division	of	labor:	different
engineers	work	on	different	components	of	the	spacecraft;	different	staffs
operate	different	restaurants.	In	academia,	the	rigid	division	of	researchers,
students,	journals,	grants,	and	prizes	into	separate	self-contained	disciplines—
though	unconducive	to	the	type	of	work	represented	by	this	book—might	(only
in	a	conciliatory	and	mellow	frame	of	mind)	be	viewed	as	a	necessary
accommodation	to	the	practicalities	of	allowing	large	numbers	of	diversely
motivated	individuals	and	teams	to	contribute	to	the	growth	of	human
knowledge	while	working	relatively	independently,	each	plowing	their	own

furrow.
A	system’s	collective	intelligence	could	be	enhanced	by	expanding	the
number	or	the	quality	of	its	constituent	intellects,	or	by	improving	the	quality	of
their	organization.
8
	To	obtain	a	collective	
superintelligence
	from	any	present-day
collective	intelligence	would	require	a	very	great	degree	of	enhancement.	The
resulting	system	would	need	to	be	capable	of	vastly	outperforming	any	current
collective	intelligence	or	other	cognitive	system	across	many	very	general
domains.	A	new	conference	format	that	lets	scholars	exchange	information	more
effectively,	or

8
effectively,	or	
a	new	collaborative	information-filtering	algorithm	that	better
predicted	users’	ratings	of	books	and	movies,	would	clearly	not	on	its	own
amount	to	anything	approaching	collective	superintelligence.	Nor	would	a	50%
increase	in	the	world	population,	or	an	improvement	in	pedagogical	method	that
enabled	students	to	complete	a	school	day	in	four	hours	instead	of	six.	Some	far
more	extreme	growth	of	humanity’s	collective	cognitive	capacity	would	be
required	to	meet	the	criterion	of	collective	superintelligence.

Note	that	the	threshold	for	collective	superintelligence	is	indexed	to	the
performance	levels	of	the	present—that	is,	the	early	twenty-first	century.	Over
the	course	of	human	prehistory,	and	again	over	the	course	of	human	history,
humanity’s	collective	intelligence	
has
	grown	by	very	large	factors.	World
population,	for	example,	has	increased	by	at	least	a	factor	of	a	thousand	since	the
Pleistocene.
9
	On	this	basis	alone,	current	levels	of	human	collective	intelligence
could	be	regarded	as	approaching	superintelligence	
relative	to	a	Pleistocene

baseline
.	Some	improvements	in	communications	technologies—especially
spoken	language,	but	perhaps	also	cities,	writing,	and	printing—could	also	be
argued	to	have,	individually	or	in	combination,	provided	super-sized	boosts,	in
the	sense	that	if	another	innovation	of	comparable	impact	to	our	collective
intellectual	problem-solving	capacity	were	to	happen,	it	would	result	in
10
A	certain	kind	of	reader	will	be	tempted	at	this	point	to	interject	that	modern
society	does	not	seem	so	particularly	intelligent.	Perhaps	some	unwelcome
political	decision	has	just	been	made	in	the	reader’s	home	country,	and	the
apparent	unwisdom	of	that	decision	now	looms	large	in	the	reader’s	mind	as
evidence	of	the	mental	incapacity	of	the	modern	era.	And	is	it	not	the	case	that
contemporary	humanity	is	idolizing	material	consumption,	depleting	natural
resources,	polluting	the	environment,	decimating	species	diversity,	all	the	while
failing	to	remedy	screaming	global	injustices	and	neglecting	paramount

humanistic	or	spiritual	values?	However,	setting	aside	the	question	of	how
modernity’s	shortcomings	stack	up	against	the	not-so-inconsiderable	failings	of
earlier	epochs,	nothing	in	our	definition	of	collective	superintelligence	implies
that	a	society	with	greater	collective	intelligence	is	necessarily	better	off.	The
definition	does	not	even	imply	that	the	more	collectively	intelligent	society	is
wiser
.	We	can	think	of	wisdom	as	the	ability	to	get	the	important	things
approximately	right.	It	is	then	possible	to	imagine	an	organization	composed	of	a

wiser
very	large	cadre	of	very	efficiently	coordinated	knowledge	workers,	who
collectively	can	solve	intellectual	problems	across	many	very	general	domains.
This	organization,	let	us	suppose,	can	operate	most	kinds	of	businesses,	invent
most	kinds	of	technologies,	and	optimize	most	kinds	of	processes.	Even	so,	it
might	get	a	few	key	big-picture	issues	entirely	wrong—for	instance,	it	may	fail
to	take	proper	precautions	against	existential	risks—and	as	a	result	pursue	a

short	explosive	growth	spurt	that	ends	ingloriously	in	total	collapse.	Such	an
organization	could	have	a	very	high	degree	of	collective	intelligence;	if
sufficiently	high,	the	organization	is	a	collective	superintelligence.	We	should
resist	the	temptation	to	roll	every	normatively	desirable	attribute	into	one	giant
amorphous	concept	of	mental	functioning,	as	though	one	could	never	
find	one
admirable	trait	without	all	the	others	being	equally	present.	Instead,	we	should
recognize	that	there	can	exist	instrumentally	powerful	information	processing

find	one
systems—intelligent	systems—that	are	neither	inherently	good	nor	reliably	wise.
But	we	will	revisit	this	issue	in	
Chapter	7
.
Collective	superintelligence	could	be	either	loosely	or	tightly	integrated.	To

illustrate	a	case	of	loosely	integrated	collective	superintelligence,	imagine	a
planet,	
MegaEarth
,	which	has	the	same	level	of	communication	and
coordination	technologies	that	we	currently	have	on	the	real	Earth	but	with	a
population	one	million	times	as	large.	With	such	a	huge	population,	the	total
intellectual	workforce	on	MegaEarth	would	be	correspondingly	larger	than	on
our	planet.	Suppose	that	a	scientific	genius	of	the	caliber	of	a	Newton	or	an
Einstein	arises	at	least	once	for	every	10	billion	people:	then	on	MegaEarth	there
would	be	700,000	such	geniuses	living	contemporaneously,	alongside
proportionally	vast	multitudes	of	slightly	lesser	talents.	New	ideas	and
technologies	would	be	developed	at	a	furious	pace,	and	global	civilization	on
MegaEarth	would	constitute	a	loosely	integrated	collective	superintelligence.
11
If	we	gradually	increase	the	level	of	integration	of	a	collective	intelligence,	it
may	eventually	become	a	unified	
intellect
—a	single	large	“mind”	as	opposed	to

11
intellect
a	mere	assemblage	of	loosely	interacting	smaller	human	minds.
12
	The
inhabitants	of	MegaEarth	could	take	steps	in	that	direction	by	improving
communications	and	coordination	technologies	and	by	developing	better	ways
for	many	individuals	to	work	on	any	hard	intellectual	problem	together.	A
collective	superintelligence	could	thus,	after	gaining	sufficiently	in	integration,
become	a	“quality	superintelligence.”
	
We	can	distinguish	a	third	form	of	superintelligence.
Quality	superintelligence:

12
	The
	
A	system	that	is	at	least	as	fast	as	a	human
mind	and	vastly	qualitatively	smarter
.
	
As	with	collective	intelligence,	intelligence	quality	is	also	a	somewhat	murky
concept;	and	in	this	case	the	difficulty	is	compounded	by	our	lack	of	experience
with	any	variations	in	intelligence	quality	above	the	upper	end	of	the	present
human	distribution.	We	can,	however,	get	some	grasp	of	the	notion	by
considering	some	related	cases.

First,	we	can	expand	the	range	of	our	reference	points	by	considering
nonhuman	animals,	which	have	intelligence	of	lower	quality.	(This	is	not	meant
as	a	speciesist	remark.	A	zebrafish	has	a	quality	of	intelligence	that	is	excellently
adapted	to	its	ecological	needs;	but	the	relevant	perspective	here	is	a	more
anthropocentric	one:	our	concern	is	with	performance	on	
humanly
	relevant
complex	cognitive	tasks.)	Nonhuman	animals	lack	complex	structured	language;
they	are	capable	of	no	or	only	rudimentary	tool	use	and	tool	construction;	they
are	severely	restricted	in	their	ability	to	make	long-term	plans;	and	they	have
very	
limited	abstract	reasoning	ability.	Nor	are	these	limitations	fully	explained
by	a	lack	of	speed	or	of	collective	intelligence	among	nonhuman	animal	minds.
In	terms	of	raw	computational	power,	human	brains	are	probably	inferior	to
those	of	some	large	animals,	including	elephants	and	whales.	And	although
humanity’s	complex	technological	civilization	would	be	impossible	without	our

very	
massive	advantage	in	collective	intelligence,	not	all	distinctly	human	cognitive
capabilities	depend	on	collective	intelligence.	Many	are	highly	developed	even
in	small,	isolated	hunter–gatherer	bands.
13
	And	many	are	not	nearly	as	highly
developed	among	highly	organized	nonhuman	animals,	such	as	chimpanzees	and
dolphins	intensely	trained	by	human	instructors,	or	ants	living	in	their	own	large
and	well-ordered	societies.	Evidently,	the	remarkable	intellectual	achievements
of	

13
of	
	are	to	a	significant	extent	attributable	to	specific	features	of	our
brain	architecture,	features	that	depend	on	a	unique	genetic	endowment	not
shared	by	other	animals.	This	observation	can	help	us	illustrate	the	concept	of
quality	superintelligence:	it	is	intelligence	of	quality	at	least	as	superior	to	that	of
human	intelligence	as	the	quality	of	human	intelligence	is	superior	to	that	of
elephants’,	dolphins’,	or	chimpanzees’.
A	second	way	to	illustrate	the	concept	of	quality	superintelligence	is	by	noting

the	domain-specific	cognitive	deficits	that	can	afflict	individual	humans,
particularly	deficits	that	are	not	caused	by	general	dementia	or	other	conditions
associated	with	wholesale	destruction	of	the	brain’s	neurocomputational
resources.	Consider,	for	example,	individuals	with	autism	spectrum	disorders
who	may	have	striking	deficits	in	social	cognition	while	functioning	well	in
other	cognitive	domains;	or	individuals	with	congenital	amusia,	who	are	unable
to	hum	or	recognize	simple	tunes	yet	perform	normally	in	most	other	respects.

Many	other	examples	could	be	adduced	from	the	neuropsychiatric	literature,
which	is	replete	with	case	studies	of	patients	suffering	narrowly	circumscribed
deficits	caused	by	genetic	abnormalities	or	brain	trauma.	Such	examples	show
that	normal	human	adults	have	a	range	of	remarkable	cognitive	talents	that	are
not	simply	a	function	of	possessing	a	sufficient	amount	of	general	neural

processing	power	or	even	a	sufficient	amount	of	general	intelligence:	specialized
neural	circuitry	is	also	needed.	This	observation	suggests	the	idea	of	
possible	but
non-realized	cognitive	talents
,	talents	that	no	actual	human	possesses	even
though	other	intelligent	systems—ones	with	no	more	computing	power	than	the
human	brain—that	did	have	those	talents	would	gain	enormously	in	their	ability
to	accomplish	a	wide	range	of	strategically	relevant	tasks.
Accordingly,	by	considering	nonhuman	animals	and	human	individuals	with
domain-specific	cognitive	deficits,	we	can	form	some	notion	of	different
qualities	of	intelligence	and	the	practical	difference	they	make.	Had	
Homo
sapiens
	lacked	(for	instance)	the	cognitive	modules	that	enable	complex
linguistic	representations,	it	might	have	been	just	another	simian	species	living
in	harmony	with	nature.	Conversely,	were	we	to	
gain
	some	new	set	of	modules
giving	an	advantage	comparable	to	that	of	being	able	to	form	complex	linguistic

Homo
sapiens
gain
representations,	we	would	become	superintelligent.
	
Superintelligence	in	any	of	these	forms	could,	over	time,	develop	the	technology
necessary	to	create	any	of	the	others.	The	
indirect	reaches
	of	these	three	forms	of
superintelligence	are	therefore	equal.	In	that	sense,	the	indirect	reach	of	current
human	intelligence	is	also	in	the	same	equivalence	class,	under	the	supposition
that	we	are	able	eventually	to	create	some	form	of	superintelligence.	Yet	there	is

a	sense	in	which	the	three	forms	of	superintelligence	are	much	closer	to	one
another:	any	one	of	them	could	create	other	forms	of	superintelligence	more
rapidly	than	we	can	create	any	form	of	superintelligence	from	our	present
starting	point.
The	
direct	reaches
	of	the	three	different	forms	of	superintelligence	are	harder
to	compare.	There	may	be	no	definite	ordering.	Their	respective	capabilities
depend	on	the	degree	to	which	they	instantiate	their	respective	advantages—
how
fast	a	speed	superintelligence	is,	
how

The	
how
how
	qualitatively	superior	a	quality
superintelligence	is,	and	so	forth.	At	most,	we	might	say	that,	
ceteris	paribus
,
speed	superintelligence	excels	at	tasks	requiring	the	rapid	execution	of	a	long
series	of	steps	that	must	be	performed	sequentially	while	collective
superintelligence	excels	at	tasks	admitting	of	analytic	decomposition	into
parallelizable	sub-tasks	and	tasks	demanding	the	combination	of	many	different

perspectives	and	skill	sets.	In	some	vague	sense,	quality	superintelligence	would
be	the	most	capable	form	of	all,	inasmuch	as	it	could	grasp	and	solve	problems
that	are,	for	all	practical	purposes,	beyond	the	
direct
	reach	of	speed
superintelligence	and	collective	superintelligence.
14
In	some	domains,	quantity	is	a	poor	substitute	for	quality.	One	solitary	genius
working	out	of	a	cork-lined	bedroom	can	write	
In	Search	of	Lost	Time
.	Could	an
equivalent	masterpiece	be	produced	by	recruiting	an	office	building	full	of
literary	hacks?
15
	Even	within	the	range	of	present	human	variation	we	see	that
some	functions	benefit	greatly	from	the	labor	of	one	brilliant	mastermind	as
opposed	to	the	joint	efforts	of	myriad	mediocrities.	If	we	widen	our	purview	to
include	
superintelligent
	minds,	we	must	countenance	a	likelihood	of	there	being
intellectual	problems	solvable	only	by	superintelligence	and	intractable	to	any
ever-so-large	collective	of	non-augmented	humans.

15
include	
There	might	thus	be	some	problems	that	are	solvable	by	a	quality
superintelligence,	and	perhaps	by	a	speed	superintelligence,	yet	which	a	loosely
integrated	collective	superintelligence	cannot	solve	(other	than	by	first
amplifying	its	own	intelligence).
16
	We	cannot	clearly	see	what	all	these	problems
are,	but	we	can	characterize	them	in	general	terms.
17
	They	would	tend	to	be
problems	involving	multiple	complex	interdependencies	that	do	not	permit	of
independently	verifiable	solution	steps:	problems	that	therefore	cannot	be	solved

16
17
in	a	piecemeal	fashion,	and	that	might	require	qualitatively	new	kinds	of
understanding	or	new	representational	frameworks	that	are	too	deep	or	too
complicated	for	the	current	edition	of	mortals	to	discover	or	use	effectively.
Some	types	of	artistic	creation	and	strategic	cognition	might	fall	into	this
category.	Some	types	of	scientific	breakthrough,	perhaps,	likewise.	And	one	can
speculate	that	the	tardiness	and	
wobbliness	of	humanity’s	progress	on	many	of
the	“eternal	problems”	of	philosophy	are	due	to	the	unsuitability	of	the	human

cortex	for	philosophical	work.	On	this	view,	our	most	celebrated	philosophers
are	like	dogs	walking	on	their	hind	legs—just	barely	attaining	the	threshold	level
of	performance	required	for	engaging	in	the	activity	
at	all
.
18
	
Minor	changes	in	brain	volume	and	wiring	can	have	major	consequences,	as	we

see	when	we	compare	the	intellectual	and	technological	achievements	of	humans
with	those	of	other	apes.	The	far	greater	changes	in	computing	resources	and
architecture	that	machine	intelligence	will	enable	will	probably	have
consequences	that	are	even	more	profound.	It	is	difficult,	perhaps	impossible,	for
us	to	form	an	intuitive	sense	of	the	aptitudes	of	a	superintelligence;	but	we	can	at
least	get	an	inkling	of	the	space	of	possibilities	by	looking	at	some	of	the
advantages	open	to	digital	minds.	The	hardware	advantages	are	easiest	to
appreciate:
	
Speed	of	computational	elements
.	Biological	neurons	operate	at	a	peak	speed	of
about	200	Hz,	a	full	seven	orders	of	magnitude	slower	than	a	modern
microprocessor	(~	2	GHz).
19
	As	a	consequence,	the	human	brain	is	forced	to	rely
on	massive	parallelization	and	is	incapable	of	rapidly	performing	any
computation	that	requires	a	large	number	of	sequential	operations.
20
	(Anything
the	brain	does	in	under	a	second	cannot	use	much	more	than	a	hundred

19
20
	(Anything
sequential	operations—perhaps	only	a	few	dozen.)	Yet	many	of	the	most
practically	important	algorithms	in	programming	and	computer	science	are	not
easily	parallelizable.	Many	cognitive	tasks	could	be	performed	far	more
efficiently	if	the	brain’s	native	support	for	parallelizable	pattern-matching
algorithms	were	complemented	by,	and	integrated	with,	support	for	fast
sequential	processing.
Internal	communication	speed
.	Axons	carry	action	potentials	at	speeds	of	120	m/s
or	less,	whereas	electronic	processing	cores	can	communicate	optically	at	the

speed	of	light	(300,000,000	m/s).
21
	The	sluggishness	of	neural	signals	limits
how	big	a	biological	brain	can	be	while	functioning	as	a	single	processing	unit.
For	example,	to	achieve	a	round-trip	latency	of	less	than	10	ms	between	any	two
elements	in	a	system,	biological	brains	must	be	smaller	than	0.11	m
3
.	An
electronic	system,	on	the	other	hand,	could	be	6.1×10
17
	m
3
,	about	the	size	of	a
dwarf	planet:	eighteen	orders	of	magnitude	larger.
22
Number	of	computational	elements
.	The	human	brain	has	somewhat	fewer	than

21
3
.	An
17
	m
3
22
100	billion	neurons.
23
	Humans	have	about	three	and	a	half	times	the	brain	size	of
chimpanzees	(though	only	one-fifth	the	brain	size	of	sperm	whales).
24
	The
number	of	neurons	in	a	biological	creature	is	most	obviously	limited	by	cranial
volume	and	metabolic	constraints,	but	other	factors	may	also	be	significant	for

larger	brains	(such	as	cooling,	development	time,	and	signal-conductance	delays
—see	the	previous	point).	By	contrast,	computer	hardware	is	indefinitely
scalable	up	to	very	high	physical	limits.
25
	Supercomputers	can	be	warehouse-
sized	or	larger,	with	additional	remote	capacity	added	via	high-speed	cables.
26
Storage	capacity
.	Human	working	memory	is	able	to	hold	no	more	than	some
four	or	five	chunks	of	information	at	any	given	time.
27
	While	it	would	be
misleading	to	compare	the	size	of	human	working	memory	directly	with	the
amount	of	RAM	in	a	digital	computer,	it	is	clear	that	the	hardware	advantages	of
digital	intelligences	will	make	it	possible	for	them	to	have	larger	working
memories.	This	might	enable	such	minds	to	intuitively	grasp	complex
relationships	that	humans	can	only	fumblingly	handle	via	plodding	calculation.
28
Human	long-term	memory	is	also	limited,	though	it	is	unclear	whether	we
manage	to	exhaust	its	storage	capacity	during	the	course	of	an	ordinary	lifetime

28
—the	rate	at	which	we	accumulate	information	is	so	slow.	(On	one	estimate,	the
adult	human	brain	stores	about	one	billion	bits—a	couple	of	orders	of	magnitude
less	than	a	low-end	smartphone.
29
)	Both	the	amount	of	information	stored	and
the	speed	with	which	it	can	be	accessed	could	thus	be	vastly	greater	in	a	machine
brain	than	in	a	biological	brain.
Reliability,	lifespan,	sensors,	etc
.	Machine	intelligences	might	have	various	other
hardware	advantages.	For	example,	biological	neurons	are	less	reliable	than
transistors.
30

29
30
	Since	noisy	computing	necessitates	redundant	encoding	schemes
that	use	multiple	elements	to	encode	a	single	bit	of	information,	a	digital	brain
might	derive	some	efficiency	gains	from	the	use	of	reliable	high-precision
computing	elements.	Brains	become	fatigued	after	a	few	hours	of	work	and	start
to	permanently	decay	after	a	few	decades	of	subjective	time;	microprocessors
are	not	subject	to	these	limitations.	Data	flow	into	a	machine	intelligence	could
be	increased	by	adding	millions	of	sensors.	Depending	on	the	technology	used,	a

machine	might	have	reconfigurable	hardware	that	can	be	optimized	for	changing
task	requirements,	whereas	much	of	the	brain’s	architecture	is	fixed	from	birth	or
only	slowly	changeable	(though	the	details	of	synaptic	connectivity	can	change
over	shorter	timescales,	like	days).
31
At	present,	the	computational	power	of	the	biological	brain	still	compares
favorably	with	that	of	digital	computers,	though	top-of-the-line	supercomputers
are	attaining	levels	of	performance	that	are	within	the	range	of	plausible

31
estimates	of	the	brain’s	processing	power.
32
	But	hardware	is	rapidly	improving,

and	the	ultimate	limits	of	hardware	performance	are	vastly	higher	than	those	of
biological	computing	substrates.
Digital	minds	will	also	benefit	from	major	advantages	in	software:
	
Editability
.	It	is	easier	to	experiment	with	parameter	variations	in	software	than	in
neural	wetware.	For	example,	with	a	whole	brain	emulation	one	could	easily	trial
what	happens	if	one	adds	more	neurons	in	a	particular	cortical	area	or	if	one
increases	or	decreases	their	excitability.	Running	such	experiments	in	living
biological	brains	would	be	far	more	difficult.
Duplicability
.	With	software,	one	can	quickly	make	arbitrarily	many	high-fidelity
copies	to	fill	the	available	hardware	base.	Biological	brains,	by	contrast,	can	be
reproduced	only	very	slowly;	and	each	new	instance	starts	out	in	a	helpless	state,
remembering	nothing	of	what	its	parents	learned	in	their	lifetimes.
Goal	coordination
.	Human	collectives	are	replete	with	inefficiencies	arising	from

the	fact	that	it	is	nearly	impossible	to	achieve	complete	uniformity	of	purpose
among	the	members	of	a	large	group—at	least	until	it	becomes	feasible	to	induce
docility	on	a	
large	scale	by	means	of	drugs	or	genetic	selection.	A	“copy	clan”	(a
group	of	identical	or	almost	identical	programs	sharing	a	common	goal)	would
avoid	such	coordination	problems.
Memory	sharing
.	Biological	brains	need	extended	periods	of	training	and
mentorship	whereas	digital	minds	could	acquire	new	memories	and	skills	by

swapping	data	files.	A	population	of	a	billion	copies	of	an	AI	program	could
synchronize	their	databases	periodically,	so	that	all	the	instances	of	the	program
know	everything	that	any	instance	learned	during	the	previous	hour.	(Direct
memory	transfer	requires	standardized	representational	formats.	Easy	swapping
of	high-level	cognitive	content	would	therefore	not	be	possible	between	just	any
pair	of	machine	intelligences.	In	particular,	it	would	not	be	possible	among	first-
generation	whole	brain	emulations.)

New	modules,	modalities,	and	algorithms
.	Visual	perception	seems	to	us	easy	and
effortless,	quite	unlike	solving	textbook	geometry	problems—this	despite	the
fact	that	it	takes	a	massive	amount	of	computation	to	reconstruct,	from	the	two-
dimensional	patterns	of	stimulation	on	our	retinas,	a	three-dimensional
representation	of	a	world	populated	with	recognizable	objects.	The	reason	this

seems	easy	is	that	we	have	dedicated	low-level	neural	machinery	for	processing
visual	information.	This	low-level	processing	occurs	unconsciously	and
automatically,	without	draining	our	mental	energy	or	conscious	attention.	Music
perception,	language	use,	social	cognition,	and	other	forms	of	information
processing	that	are	“natural”	for	us	humans	seem	to	be	likewise	supported	by
dedicated	neurocomputational	modules.	An	artificial	mind	that	had	such
specialized	support	for	other	cognitive	domains	that	have	become	important	in
the	contemporary	world—such	as	engineering,	computer	programming,	and
business	strategy—would	have	big	advantages	over	minds	like	ours	that	have	to
rely	on	clunky	general-purpose	cognition	to	think	about	such	things.	New
algorithms	may	also	be	developed	to	take	advantage	of	the	distinct	affordances
of	digital	hardware,	such	as	its	support	for	fast	serial	processing.
The	
ultimately
	attainable	advantages	of	machine	intelligence,	hardware	and
software	combined,	are	enormous.
33

The	
33
	But	how	rapidly	could	those	potential
advantages	be	realized?	That	is	the	question	to	which	we	now	turn.

CHAPTER	4
The	kinetics	of	an	intelligence	explosion
	
Once	machines	attain	some	form	of	human-equivalence	in	general
reasoning	ability,	how	long	will	it	then	be	before	they	attain	radical
superintelligence?	Will	this	be	a	slow,	gradual,	protracted	transition?	Or
will	it	be	sudden,	explosive?	This	chapter	analyzes	the	kinetics	of	the
transition	to	superintelligence	as	a	function	of	optimization	power	and
system	recalcitrance.	We	consider	what	we	know	or	may	reasonably	surmise
about	the	behavior	of	these	two	factors	in	the	neighborhood	of	human-level
general	intelligence.
	
Given	that	machines	will	
eventually
	vastly	exceed	biology	in	general
intelligence,	but	that	machine	cognition	is	
currently
	vastly	narrower	than	human
cognition,	one	is	led	to	wonder	how	quickly	this	usurpation	will	take	place.	The
question	we	are	asking	here	must	be	sharply	distinguished	from	the	question	we
considered	in	
Chapter	1
	about	how	far	away	we	currently	are	from	developing	a

currently
Chapter	1
machine	with	human-level	general	intelligence.	Here	the	question	is	instead,	
if
and	when	such	a	machine	is	developed,	how	long	will	it	be	from	then	until	a
machine	becomes	radically	superintelligent?
	Note	that	one	could	think	that	it
will	take	quite	a	long	time	until	machines	reach	the	human	baseline,	or	one
might	be	agnostic	about	how	long	that	will	take,	and	yet	have	a	strong	view	that
once	this	happens,	the	further	ascent	into	strong	superintelligence	will	be	very
rapid.
It	can	be	helpful	to	think	about	these	matters	schematically,	even	though	doing

if
rapid.
so	involves	temporarily	ignoring	some	qualifications	and	complicating	details.
Consider,	then,	a	diagram	that	plots	the	intellectual	capability	of	the	most
advanced	machine	intelligence	system	as	a	function	of	time	(
Figure	7
).

A	horizontal	line	labeled	“human	baseline”	represents	the	effective	intellectual
capabilities	of	a	representative	human	adult	with	access	to	the	information
sources	and	technological	aids	currently	available	in	developed	countries.	At
present,	the	most	advanced	AI	system	is	far	below	the	human	baseline	on	any
reasonable	metric	of	general	intellectual	ability.	At	some	point	in	future,	a
machine	might	reach	approximate	parity	with	this	human	baseline	(which	we
take	to	be	fixed—anchored	to	the	year	2014,	say,	even	if	the	capabilities	of
human	individuals	should	have	increased	in	the	intervening	years):	this	would
mark	the	onset	of	the	takeoff.	The	capabilities	of	the	system	continue	to	grow,
and	at	some	later	point	the	system	reaches	parity	with	the	combined	intellectual
capability	of	all	of	humanity	(again	anchored	to	the	present):	what	we	may	call
the	“civilization	baseline”.	Eventually,	if	the	system’s	abilities	continue	to	grow,

it	attains	“strong	superintelligence”—a	level	of	intelligence	vastly	greater	than
contemporary	humanity’s	combined	intellectual	wherewithal.	The	attainment	of
strong	superintelligence	marks	the	completion	of	the	takeoff,	though	the	system
might	continue	to	gain	in	capacity	thereafter.	Sometime	during	the	takeoff	phase,
the	system	may	pass	a	landmark	which	we	can	call	“the	crossover”,	a	point
beyond	which	the	system’s	further	improvement	is	mainly	driven	by	the
system’s	own	actions	rather	than	by	work	performed	upon	it	by	others.
1
	(The

1
	(The
possible	existence	of	such	a	crossover	will	become	important	in	the	subsection
on	optimization	power	and	explosivity,	later	in	this	chapter.)
Figure	7
	Shape	of	the	takeoff.	It	is	important	to	distinguish	between	these
questions:	“Will	a	takeoff	occur,	and	if	so,	when?”	and	“If	and	when	a	takeoff
does	occur,	how	steep	will	it	be?”	One	might	hold,	for	example,	that	it	will	be	a
very	long	time	before	a	takeoff	occurs,	but	that	when	it	does	it	will	proceed	very
quickly.	Another	relevant	question	(not	illustrated	in	this	figure)	is,	“How	large	a

fraction	of	the	world	economy	will	participate	in	the	takeoff?”	These	questions
are	related	but	distinct.
	
With	this	picture	in	mind,	we	can	distinguish	three	classes	of	transition
scenarios—scenarios	in	which	systems	progress	from	human-level	intelligence
to	superintelligence—based	on	their	steepness;	that	is	to	say,	whether	they
represent	a	slow,	fast,	or	moderate	takeoff.
Slow
A	slow	takeoff	is	one	that	occurs	over	some	long	temporal	interval,	such	as
decades	or	centuries.	Slow	takeoff	scenarios	offer	excellent	opportunities
for	
human	political	processes	to	adapt	and	respond.	Different	approaches
can	be	tried	and	tested	in	sequence.	New	experts	can	be	trained	and
credentialed.	Grassroots	campaigns	can	be	mobilized	by	groups	that	feel
they	are	being	disadvantaged	by	unfolding	developments.	If	it	appears	that
new	kinds	of	secure	infrastructure	or	mass	surveillance	of	AI	researchers	is
needed,	such	systems	could	be	developed	and	deployed.	Nations	fearing	an

for	
AI	arms	race	would	have	time	to	try	to	negotiate	treaties	and	design
enforcement	mechanisms.	Most	preparations	undertaken	before	onset	of	the
slow	takeoff	would	be	rendered	obsolete	as	better	solutions	would	gradually
become	visible	in	the	light	of	the	dawning	era.
Fast
A	fast	takeoff	occurs	over	some	short	temporal	interval,	such	as	minutes,
hours,	or	days.	Fast	takeoff	scenarios	offer	scant	opportunity	for	humans	to
deliberate.	Nobody	need	even	notice	anything	unusual	before	the	game	is
already	lost.	In	a	fast	takeoff	scenario,	humanity’s	fate	essentially	depends

Fast
on	preparations	previously	put	in	place.	At	the	slowest	end	of	the	fast
takeoff	scenario	range,	some	simple	human	actions	might	be	possible,
analogous	to	flicking	open	the	“nuclear	suitcase”;	but	any	such	action
would	either	be	elementary	or	have	been	planned	and	pre-programmed	in
advance.
Moderate
A	moderate	takeoff	is	one	that	occurs	over	some	intermediary	temporal

interval,	such	as	months	or	years.	Moderate	takeoff	scenarios	give	humans
some	chance	to	respond	but	not	much	time	to	analyze	the	situation,	to	test
different	approaches,	or	to	solve	complicated	coordination	problems.	There
is	not	enough	time	to	develop	or	deploy	new	systems	(e.g.	political	systems,
surveillance	regimes,	or	computer	network	security	protocols),	but	extant
systems	could	be	applied	to	the	new	challenge.
	
During	a	slow	takeoff,	there	would	be	plenty	of	time	for	the	news	to	get	out.
In	a	moderate	takeoff,	by	contrast,	it	is	possible	that	developments	would	be	kept
secret	as	they	unfold.	Knowledge	might	be	restricted	to	a	small	group	of
insiders,	as	in	a	covert	state-sponsored	military	research	program.	Commercial
projects,	small	academic	teams,	and	“nine	hackers	in	a	basement”	outfits	might
also	be	clandestine—though,	if	the	prospect	of	an	intelligence	explosion	were
“on	the	radar”	of	state	intelligence	agencies	as	a	national	security	priority,	then

the	most	promising	private	projects	would	seem	to	have	a	good	chance	of	being
under	surveillance.	The	host	state	(or	a	dominant	foreign	power)	would	then
have	the	option	of	nationalizing	or	shutting	down	any	project	that	showed	signs
of	commencing	takeoff.	Fast	takeoffs	would	happen	so	quickly	that	there	would
not	be	much	time	for	word	to	get	out	or	for	anybody	to	mount	a	meaningful
reaction	if	it	did.	But	an	outsider	might	intervene	
before
	the	onset	of	the	takeoff
if	they	believed	a	particular	project	to	be	closing	in	on	success.

before
Moderate	takeoff	scenarios	could	lead	to	geopolitical,	social,	and	economic
turbulence	as	individuals	and	groups	jockey	to	position	themselves	to	gain	from
the	unfolding	transformation.	Such	upheaval,	should	it	occur,	might	impede
efforts	to	orchestrate	a	well-composed	response;	alternatively,	it	might	enable
solutions	more	radical	than	calmer	circumstances	would	permit.	For	instance,	in
a	moderate	takeoff	scenario	where	cheap	and	capable	emulations	or	other	digital
minds	gradually	flood	labor	markets	over	a	period	of	years,	one	could	imagine

mass	protests	by	laid-off	workers	pressuring	governments	to	increase
unemployment	benefits	or	institute	a	living	wage	guarantee	to	all	human
citizens,	or	to	levy	special	taxes	or	impose	minimum	wage	requirements	on
employers	who	use	emulation	workers.	In	order	for	any	relief	derived	from	such
policies	to	be	more	than	fleeting,	support	for	them	would	somehow	have	to	be
cemented	into	permanent	power	structures.	Similar	issues	can	arise	if	the	takeoff
is	slow	rather	than	moderate,	but	the	disequilibrium	and	rapid	change	in

moderate	scenarios	may	present	special	opportunities	for	small	groups	to	wield

disproportionate	influence.
It	might	appear	to	some	readers	that	of	these	three	types	of	scenario,	the	slow
takeoff	is	the	most	probable,	the	moderate	takeoff	is	less	probable,	and	the	fast
takeoff	is	utterly	implausible.	It	could	seem	fanciful	to	suppose	that	the	world
could	be	radically	transformed	and	humanity	deposed	from	its	position	as	apex
cogitator	over	the	course	of	an	hour	or	two.	No	change	of	such	moment	has	ever
occurred	in	human	history,	and	its	nearest	parallels—the	Agricultural	and
Industrial	Revolutions—played	out	over	much	longer	timescales	(centuries	to
millennia	in	the	former	case,	decades	to	centuries	in	the	latter).	So	the	base	rate
for	the	kind	of	transition	entailed	by	a	fast	or	medium	takeoff	scenario,	in	terms
of	the	speed	and	magnitude	of	the	postulated	change,	is	zero:	it	lacks	precedent
outside	myth	and	religion.
2
Nevertheless,	this	chapter	will	present	some	reasons	for	thinking	that	the	slow
transition	scenario	is	improbable.	If	and	when	a	takeoff	occurs,	it	will	likely	be

2
explosive.
To	begin	to	analyze	the	question	of	how	fast	the	takeoff	will	be,	we	can
conceive	of	the	rate	of	increase	in	a	system’s	intelligence	as	a	(monotonically
increasing)	function	of	two	variables:	the	amount	of	“optimization	power”,	or
quality-weighted	design	effort,	that	is	being	applied	to	increase	the	system’s
intelligence,	and	the	responsiveness	of	the	system	to	the	application	of	a	given
amount	of	such	optimization	power.	We	might	term	the	inverse	of
responsiveness	“recalcitrance”,	and	write:

Pending	some	specification	of	how	to	quantify	intelligence,	design	effort,	and
recalcitrance,	this	expression	is	merely	qualitative.	But	we	can	at	least	observe
that	a	system’s	intelligence	will	increase	rapidly	if	
either
	a	lot	of	skilled	effort	is
applied	to	the	task	of	increasing	its	intelligence	and	the	system’s	intelligence	is
not	too	hard	to	increase	
or
	there	is	a	non-trivial	design	effort	and	the	system’s
recalcitrance	is	low	(or	both).	If	we	know	how	much	design	effort	is	going	into

either
or
improving	a	particular	system,	and	the	rate	of	improvement	this	effort	produces,
we	could	calculate	the	system’s	recalcitrance.

Further,	we	can	observe	that	the	amount	of	optimization	power	devoted	to
improving	some	system’s	performance	varies	between	systems	and	over	time.	A
system’s	recalcitrance	might	also	vary	depending	on	how	much	the	system	has
already	been	optimized.	Often,	the	easiest	improvements	are	made	first,	leading
to	diminishing	returns	(increasing	recalcitrance)	as	low-hanging	fruits	are
depleted.	However,	there	can	also	be	improvements	that	make	further
improvements	easier,	leading	to	improvement	cascades.	The	process	of	solving	a
jigsaw	puzzle	starts	out	simple—it	is	easy	to	find	the	corners	and	the	edges.
Then	recalcitrance	goes	up	as	subsequent	pieces	are	harder	to	fit.	But	as	the
puzzle	nears	completion,	the	search	space	collapses	and	the	process	gets	easier
again.
To	proceed	in	our	inquiry,	we	must	therefore	analyze	how	recalcitrance	and
optimization	power	might	vary	in	the	critical	time	periods	during	the	takeoff.
This	will	occupy	us	over	the	next	few	pages.

again.
	
Let	us	begin	with	recalcitrance.	The	outlook	here	depends	on	the	type	of	the
system	under	consideration.	For	completeness,	we	first	cast	a	brief	glance	at	the
recalcitrance	that	would	be	encountered	along	paths	to	superintelligence	that	do
not	involve	advanced	machine	intelligence.	We	find	that	recalcitrance	along
those	paths	appears	to	be	fairly	high.	That	done,	we	will	turn	to	the	main	case,
which	is	that	the	takeoff	involves	machine	intelligence;	and	there	we	find	that
recalcitrance	at	the	critical	juncture	seems	low.

Cognitive	enhancement	via	improvements	in	public	health	and	diet	has	steeply
diminishing	returns.
3
	Big	gains	come	from	eliminating	severe	nutritional
deficiencies,	and	the	most	severe	deficiencies	have	already	been	largely
eliminated	in	all	but	the	poorest	countries.	Only	girth	is	gained	by	increasing	an
already	adequate	diet.	Education,	too,	is	now	probably	subject	to	diminishing
returns.	The	fraction	of	talented	individuals	in	the	world	who	lack	access	to

quality	education	is	still	substantial,	but	declining.
Pharmacological	enhancers	might	deliver	some	cognitive	gains	over	the
coming	decades.	But	after	the	easiest	fixes	have	been	accomplished—perhaps
sustainable	increases	in	mental	energy	and	ability	to	concentrate,	along	with
better	control	over	the	rate	of	long-term	memory	consolidation—subsequent
gains	will	be	increasingly	hard	to	come	by.	Unlike	diet	and	public	health
approaches,	however,	improving	cognition	through	smart	drugs	might	get	easier
before	it	gets	harder.	The	field	of	neuropharmacology	still	lacks	much	of	the
basic	knowledge	
that	would	be	needed	to	competently	intervene	in	the	healthy
brain.	Neglect	of	enhancement	medicine	as	a	legitimate	area	of	research	may	be
partially	to	blame	for	this	current	backwardness.	If	neuroscience	and
pharmacology	continue	to	progress	for	a	while	longer	without	focusing	on
cognitive	enhancement,	then	maybe	there	would	be	some	relatively	easy	gains	to

be	had	when	at	last	the	development	of	nootropics	becomes	a	serious	priority.
4
Genetic	cognitive	enhancement	has	a	U-shaped	recalcitrance	profile	similar	to
that	of	nootropics,	but	with	larger	potential	gains.	Recalcitrance	starts	out	high
while	the	only	available	method	is	selective	breeding	sustained	over	many
generations,	something	that	is	obviously	difficult	to	accomplish	on	a	globally
significant	scale.	Genetic	enhancement	will	get	easier	as	technology	is
developed	for	cheap	and	effective	genetic	testing	and	selection	(and	particularly

4
when	iterated	embryo	selection	becomes	feasible	in	humans).	These	new
techniques	will	make	it	possible	to	tap	the	pool	of	existing	human	genetic
variation	for	intelligence-enhancing	alleles.	As	the	best	existing	alleles	get
incorporated	into	genetic	enhancement	packages,	however,	further	gains	will	get
harder	to	come	by.	The	need	for	more	innovative	approaches	to	genetic
modification	may	then	increase	recalcitrance.	There	are	limits	to	how	quickly
things	can	progress	along	the	genetic	enhancement	path,	most	notably	the	fact

that	germline	interventions	are	subject	to	an	inevitable	maturational	lag:	this
strongly	counteracts	the	possibility	of	a	fast	or	moderate	takeoff.
5
	That	embryo
selection	can	only	be	applied	in	the	context	of	in	vitro	fertilization	will	slow	its
rate	of	adoption:	another	limiting	factor.
The	recalcitrance	along	the	brain–computer	path	seems	initially	very	high.	In
the	unlikely	event	that	it	somehow	becomes	easy	to	insert	brain	implants	and	to
achieve	high-level	functional	integration	with	the	cortex,	recalcitrance	might

5
plummet.	In	the	long	run,	the	difficulty	of	making	progress	along	this	path	would
be	similar	to	that	involved	in	improving	emulations	or	AIs,	since	the	bulk	of	the

brain–computer	system’s	intelligence	would	eventually	reside	in	the	computer
part.
The	recalcitrance	for	making	networks	and	organizations	
in	general
	more
efficient	is	high.	A	vast	amount	of	effort	is	going	into	overcoming	this
recalcitrance,	and	the	result	is	an	annual	improvement	of	humanity’s	total
capacity	by	perhaps	no	more	than	a	couple	of	percent.
6
	Furthermore,	shifts	in	the
internal	and	external	environment	mean	that	organizations,	even	if	efficient	at
one	time,	soon	become	ill-adapted	to	their	new	circumstances.	Ongoing	reform
effort	is	thus	required	even	just	to	prevent	deterioration.	A	step	change	in	the	rate
of	gain	in	average	organizational	efficiency	is	perhaps	conceivable,	but	it	is	hard
to	see	how	even	the	most	radical	scenario	of	this	kind	could	produce	anything
faster	than	a	slow	takeoff,	since	organizations	operated	by	humans	are	confined
to	work	on	human	timescales.	The	Internet	continues	to	be	an	exciting	frontier
with	many	opportunities	for	enhancing	collective	intelligence,	with	a

recalcitrance	that	seems	at	the	moment	to	be	in	the	moderate	range—progress	is
somewhat	swift	but	a	lot	of	effort	is	going	into	making	this	progress	happen.	It
may	be	expected	to	increase	as	low-hanging	fruits	(such	as	search	engines	and
email)	are	depleted.
	
The	difficulty	of	advancing	toward	whole	brain	emulation	is	difficult	to	estimate.
Yet	we	can	point	to	a	specific	future	milestone:	the	successful	emulation	of	an
insect	brain.	That	milestone	stands	on	a	hill,	and	its	conquest	would	bring	into

view	much	of	the	terrain	ahead,	allowing	us	to	make	a	decent	guess	at	the
recalcitrance	of	scaling	up	the	technology	to	human	whole	brain	emulation.	(A
successful	emulation	of	a	small-mammal	brain,	such	as	that	of	a	mouse,	would
give	an	even	better	vantage	point	that	would	allow	the	distance	remaining	to	a
human	whole	brain	emulation	to	be	estimated	with	a	high	degree	of	precision.)
The	path	toward	artificial	intelligence,	by	contrast,	may	feature	no	such	obvious
milestone	or	early	observation	point.	It	is	entirely	possible	that	the	quest	for

artificial	intelligence	will	appear	to	be	lost	in	dense	jungle	until	an	unexpected
breakthrough	reveals	the	finishing	line	in	a	clearing	just	a	few	short	steps	away.
Recall	the	distinction	between	these	two	questions:	How	hard	is	it	to	attain
roughly	human	levels	of	cognitive	ability?	And	how	hard	is	it	to	get	from	there

to	superhuman	levels?	The	first	question	is	mainly	relevant	for	predicting	how
long	it	will	be	before	the	onset	of	a	takeoff.	It	is	the	second	question	that	is	key
to	assessing	the	shape	of	the	takeoff,	which	is	our	aim	here.	And	though	it	might
be	tempting	to	suppose	that	the	step	from	human	level	to	superhuman	level	must
be	the	harder	one—this	step,	after	all,	takes	place	“at	a	higher	altitude”	where
capacity	must	be	superadded	to	an	already	quite	capable	system—this	would	be
a	very	unsafe	assumption.	It	is	quite	possible	that	recalcitrance	
falls
	when	a
machine	reaches	human	parity.
Consider	first	whole	brain	emulation.	The	difficulties	involved	in	creating	the
first	human	emulation	are	of	a	quite	different	kind	from	those	involved	in
enhancing	an	existing	emulation.	Creating	a	first	emulation	involves	huge
technological	challenges,	particularly	in	regard	to	developing	the	requisite
scanning	and	image	interpretation	capabilities.	This	step	might	also	require

falls
	when	a
considerable	amounts	of	physical	capital—an	industrial-scale	machine	park	with
hundreds	of	high-throughput	scanning	machines	is	not	implausible.	By	contrast,
enhancing	the	quality	of	an	existing	emulation	involves	tweaking	algorithms	and
data	structures:	essentially	a	software	problem,	and	one	that	could	turn	out	to	be
much	easier	than	perfecting	the	imaging	technology	needed	to	create	the	original
template.	Programmers	could	easily	experiment	with	tricks	like	increasing	the

neuron	count	in	different	cortical	areas	to	see	how	it	affects	performance.
7
	They
also	could	work	on	code	optimization	and	on	finding	simpler	computational
models	that	preserve	the	essential	functionality	of	individual	neurons	or	small
networks	of	neurons.	If	the	last	technological	prerequisite	to	fall	into	place	is
either	scanning	or	translation,	with	computing	power	being	relatively	abundant,
then	not	much	attention	might	have	been	given	during	the	development	phase	to

7
	They
implementational	efficiency,	and	easy	opportunities	for	computational	efficiency
savings	might	be	available.	(More	fundamental	architectural	reorganization
might	also	be	possible,	but	that	takes	us	off	the	emulation	path	and	into	AI
territory.)
Another	way	to	improve	the	code	base	once	the	first	emulation	has	been
produced	is	to	scan	additional	brains	with	different	or	superior	skills	and	talents.
Productivity	growth	would	also	occur	as	a	consequence	of	adapting

organizational	structures	and	workflows	to	the	unique	attributes	of	digital	minds.
Since	there	is	no	precedent	in	the	human	economy	of	a	worker	who	can	be
literally	copied,	reset,	run	at	different	speeds,	and	so	forth,	managers	of	the	first
emulation	cohort	would	find	plenty	of	room	for	innovation	in	managerial
practices.

After	initially	plummeting	when	human	whole	brain	emulation	becomes
possible,	recalcitrance	may	rise	again.	Sooner	or	later,	the	most	glaring
implementational	inefficiencies	will	have	been	optimized	away,	the	most
promising	algorithmic	variations	will	have	been	tested,	and	the	easiest
opportunities	for	organizational	innovation	will	have	been	exploited.	The
template	library	will	have	expanded	so	that	acquiring	more	brain	scans	would
add	little	benefit	over	working	with	existing	templates.	Since	a	template	can	be
multiplied,	each	copy	can	be	individually	trained	in	a	different	field,	and	this	can
be	done	at	electronic	speed,	it	might	be	that	the	number	of	brains	that	would
need	to	be	scanned	in	order	to	capture	most	of	the	potential	economic	gains	is
small.	Possibly	a	single	brain	would	suffice.
Another	potential	cause	of	escalating	recalcitrance	is	the	possibility	that
emulations	or	their	biological	supporters	will	organize	to	support	regulations

restricting	the	use	of	emulation	workers,	limiting	emulation	copying,	prohibiting
certain	kinds	of	experimentation	with	digital	minds,	instituting	workers’	rights
and	a	minimum	wage	for	emulations,	and	so	forth.	It	is	equally	possible,
however,	that	political	developments	would	go	in	the	opposite	direction,
contributing	to	a	fall	in	recalcitrance.	This	might	happen	if	initial	restraint	in	the
use	of	emulation	labor	gives	way	to	unfettered	exploitation	as	competition	heats
up	and	the	economic	and	strategic	costs	of	occupying	the	moral	high	ground
become	clear.

As	for	artificial	intelligence	(non-emulation	machine	intelligence),	the
difficulty	of	lifting	a	system	from	human-level	to	superhuman	intelligence	by
means	of	algorithmic	improvements	depends	on	the	attributes	of	the	particular
system.	Different	architectures	might	have	very	different	recalcitrance.
In	some	situations,	recalcitrance	could	be	extremely	low.	For	example,	if
human-level	AI	is	delayed	because	one	key	insight	long	eludes	programmers,
then	when	the	final	breakthrough	occurs,	the	AI	might	leapfrog	from	below	to

radically	above	human	level	without	even	touching	the	intermediary	rungs.
Another	situation	in	which	recalcitrance	could	turn	out	to	be	extremely	low	is
that	of	an	AI	system	that	can	achieve	intelligent	capability	via	two	different
modes	of	processing.	To	illustrate	this	possibility,	suppose	an	AI	is	composed	of
two	subsystems,	one	possessing	domain-specific	problem-solving	techniques,
the	other	possessing	general-purpose	reasoning	ability.	It	could	then	be	the	case
that	while	the	second	subsystem	remains	below	a	certain	capacity	threshold,	it

contributes	nothing	to	the	system’s	overall	performance,	because	the	solutions	it

generates	are	always	inferior	to	those	generated	by	the	domain-specific
subsystem.	Suppose	now	that	a	small	amount	of	optimization	power	is	applied	to
the	general-purpose	subsystem	
and	that	this	produces	a	brisk	rise	in	the	capacity
of	that	subsystem.	At	first,	we	observe	no	increase	in	the	overall	system’s
performance,	indicating	that	recalcitrance	is	high.	Then,	once	the	capacity	of	the
general-purpose	subsystem	crosses	the	threshold	where	its	solutions	start	to	beat
those	of	the	domain-specific	subsystem,	the	overall	system’s	performance
suddenly	begins	to	improve	at	the	same	brisk	pace	as	the	general-purpose
subsystem,	even	as	the	amount	of	optimization	power	applied	stays	constant:	the
system’s	recalcitrance	has	plummeted.
It	is	also	possible	that	our	natural	tendency	to	view	intelligence	from	an
anthropocentric	perspective	will	lead	us	to	underestimate	improvements	in	sub-
human	systems,	and	thus	to	overestimate	recalcitrance.	Eliezer	Yudkowsky,	an

AI	theorist	who	has	written	extensively	on	the	future	of	machine	intelligence,
puts	the	point	as	follows:
AI	might	make	an	
apparently
	sharp	jump	in	intelligence	purely	as	the	result
of	anthropomorphism,	the	human	tendency	to	think	of	“village	idiot”	and
“Einstein”	as	the	extreme	ends	of	the	intelligence	scale,	instead	of	nearly
indistinguishable	points	on	the	scale	of	minds-in-general.	Everything
dumber	than	a	dumb	human	may	appear	to	us	as	simply	“dumb”.	One

imagines	the	“AI	arrow”	creeping	steadily	up	the	scale	of	intelligence,
moving	past	mice	and	chimpanzees,	with	AIs	still	remaining	“dumb”
because	AIs	cannot	speak	fluent	language	or	write	science	papers,	and	then
the	AI	arrow	crosses	the	tiny	gap	from	infra-idiot	to	ultra-Einstein	in	the
course	of	one	month	or	some	similarly	short	period.
8
	(See	
Fig.	8
.)
	
The	upshot	of	these	several	considerations	is	that	it	is	difficult	to	predict	how
hard	it	will	be	to	make	algorithmic	improvements	in	the	first	AI	that	reaches	a

8
	(See	
Fig.	8
.)
	
roughly	human	level	of	general	intelligence.	There	are	at	least	some	possible
circumstances	in	which	algorithm-recalcitrance	is	low.	But	even	if	algorithm-
recalcitrance	is	very	high,	this	would	not	preclude	the	overall	recalcitrance	of	the
AI	in	question	from	being	low.	For	it	might	be	easy	to	increase	the	intelligence
of	the	system	in	other	ways	than	by	improving	its	algorithms.	There	are	two
other	factors	that	can	be	improved:	content	and	hardware.

First,	consider	content	improvements.	By	“content”	we	here	mean	those	parts
of	a	system’s	software	assets	that	do	not	make	up	its	core	algorithmic
architecture.	Content	might	include,	for	example,	databases	of	stored	percepts,
specialized	skills	
libraries,	and	inventories	of	declarative	knowledge.	For	many
kinds	of	system,	the	distinction	between	algorithmic	architecture	and	content	is
very	unsharp;	nevertheless,	it	will	serve	as	a	rough-and-ready	way	of	pointing	to
one	potentially	important	source	of	capability	gains	in	a	machine	intelligence.
An	alternative	way	of	expressing	much	the	same	idea	is	by	saying	that	a
system’s	intellectual	problem-solving	capacity	can	be	enhanced	not	only	by
making	the	system	cleverer	but	also	by	expanding	what	the	system	knows.
Figure	8
	A	less	anthropomorphic	scale?	The	gap	between	a	dumb	and	a	clever
person	may	appear	large	from	an	anthropocentric	perspective,	yet	in	a	less
parochial	view	the	two	have	nearly	indistinguishable	minds.
9
	It	will	almost

Figure	8
9
certainly	prove	harder	and	take	longer	to	build	a	machine	intelligence	that	has	a
general	level	of	smartness	comparable	to	that	of	a	village	idiot	than	to	improve
such	a	system	so	that	it	becomes	much	smarter	than	any	human.
	
Consider	a	contemporary	AI	system	such	as	TextRunner	(a	research	project	at
the	University	of	Washington)	or	IBM’s	Watson	(the	system	that	won	the
Jeopardy!
	quiz	show).	These	systems	can	extract	certain	pieces	of	semantic
information	by	analyzing	text.	Although	these	systems	do	not	understand	what

Jeopardy!
they	read	in	the	same	sense	or	to	the	same	extent	as	a	human	does,	they	can
nevertheless	extract	significant	amounts	of	information	from	natural	language
and	use	that	information	to	make	simple	inferences	and	answer	questions.	They
can	also	learn	from	experience,	building	up	more	extensive	representations	of	a
concept	as	they	encounter	additional	instances	of	its	use.	They	are	designed	to
operate	for	much	of	the	time	in	unsupervised	mode	(i.e.	to	learn	hidden	structure
in	unlabeled	data	in	the	absence	of	error	or	reward	signal,	without	human

guidance)	and	to	be	fast	and	scalable.	TextRunner,	for	instance,	works	with	a
corpus	of	500	million	web	pages.
10

Now	imagine	a	remote	descendant	of	such	a	system	that	has	acquired	the
ability	to	read	with	as	much	understanding	as	a	human	ten-year-old	but	with	a
reading	speed	similar	to	that	of	TextRunner.	(This	is	probably	an	AI-complete
problem.)	So	we	are	imagining	a	system	that	thinks	much	faster	and	has	much
better	memory	than	a	human	adult,	but	knows	much	less,	and	perhaps	the	net
effect	of	this	is	that	the	system	is	roughly	human-equivalent	in	its	general
problem-solving	ability.	But	its	content	recalcitrance	is	very	low—low	enough	to
precipitate	a	takeoff.	Within	a	few	weeks,	the	system	has	read	and	mastered	all
the	content	contained	in	the	Library	of	Congress.	Now	the	system	knows	much
more	than	any	human	being	and	thinks	vastly	faster:	it	has	become	(at	least)
weakly	superintelligent.
A	system	might	thus	greatly	boost	its	effective	intellectual	capability	by
absorbing	pre-produced	content	accumulated	through	centuries	of	human	science

and	civilization:	for	instance,	by	reading	through	the	Internet.	If	an	AI	reaches
human	level	without	previously	having	had	access	to	this	material	or	without
having	been	able	to	digest	it,	then	the	AI’s	overall	recalcitrance	will	be	low	even
if	it	is	hard	to	improve	its	algorithmic	architecture.
Content-recalcitrance	is	a	relevant	concept	for	emulations,	too.	A	high-speed
emulation	has	an	advantage	not	only	because	it	can	complete	the	same	tasks	as

biological	humans	more	quickly,	but	also	because	it	can	accumulate	more	timely
content,	such	as	task-relevant	skills	and	expertise.	In	order	to	tap	the	full
potential	of	fast	content	accumulation,	however,	a	system	needs	to	have	a
correspondingly	large	memory	capacity.	There	is	little	point	in	reading	an	entire
library	if	you	have	forgotten	all	about	the	aardvark	by	the	time	you	get	to	the
abalone.	While	an	AI	system	is	likely	to	have	adequate	memory	capacity,
emulations	
would	inherit	some	of	the	capacity	limitations	of	their	human

templates.	They	may	therefore	need	architectural	enhancements	in	order	to
become	capable	of	unbounded	learning.
So	far	we	have	considered	the	recalcitrance	of	architecture	and	of	content—
that	is,	how	difficult	it	would	be	to	improve	the	
software
	of	a	machine
intelligence	that	has	reached	human	parity.	Now	let	us	look	at	a	third	way	of
boosting	the	performance	of	machine	intelligence:	improving	its	hardware.	What
would	be	the	recalcitrance	for	hardware-driven	improvements?
Starting	with	intelligent	software	(emulation	or	AI)	one	can	amplify	
collective

software
	simply	by	using	additional	computers	to	run	more	instances	of	the

program.
11
	One	could	also	amplify	
speed	intelligence
	by	moving	the	program	to
faster	computers.	Depending	on	the	degree	to	which	the	program	lends	itself	to
parallelization,	speed	intelligence	could	also	be	amplified	by	running	the
program	on	more	processors.	This	is	likely	to	be	feasible	for	emulations,	which
have	a	highly	parallelized	architecture;	but	many	AI	programs,	too,	have
important	subroutines	that	can	benefit	from	massive	parallelization.	Amplifying
quality	intelligence
	by	increasing	computing	power	might	also	be	possible,	but
that	case	is	less	straightforward.
12
The	recalcitrance	for	amplifying	collective	or	speed	intelligence	(and	possibly
quality	intelligence)	in	a	system	with	human-level	software	is	therefore	likely	to
be	low.	The	only	difficulty	involved	is	gaining	access	to	additional	computing
power.	There	are	several	ways	for	a	system	to	expand	its	hardware	base,	each
relevant	over	a	different	timescale.
In	the	short	term,	computing	power	should	scale	roughly	linearly	with

12
funding:	twice	the	funding	buys	twice	the	number	of	computers,	enabling	twice
as	many	instances	of	the	software	to	be	run	simultaneously.	The	emergence	of
cloud	computing	services	gives	a	project	the	option	to	scale	up	its	computational
resources	without	even	having	to	wait	for	new	computers	to	be	delivered	and
installed,	though	concerns	over	secrecy	might	favor	the	use	of	in-house
computers.	(In	certain	scenarios,	computing	power	could	also	be	obtained	by
other	means,	such	as	by	commandeering	botnets.
13
)	Just	how	easy	it	would	be	to

13
scale	the	system	by	a	given	factor	depends	on	how	much	computing	power	the
initial	system	uses.	A	system	that	initially	runs	on	a	PC	could	be	scaled	by	a
factor	of	thousands	for	a	mere	million	dollars.	A	program	that	runs	on	a
supercomputer	would	be	far	more	expensive	to	scale.
In	the	slightly	longer	term,	the	cost	of	acquiring	additional	hardware	may	be
driven	up	as	a	growing	portion	of	the	world’s	installed	capacity	is	being	used	to
run	digital	minds.	For	instance,	in	a	competitive	market-based	emulation

scenario,	the	cost	of	running	one	additional	copy	of	an	emulation	should	rise	to
be	roughly	equal	to	the	income	generated	by	the	marginal	copy,	as	investors	bid
up	the	price	for	existing	computing	infrastructure	to	match	the	return	they	expect
from	their	investment	(though	if	only	one	project	has	mastered	the	technology	it
might	gain	a	degree	of	monopsony	power	in	the	computing	power	market	and
therefore	pay	a	lower	price).
Over	a	somewhat	longer	timescale,	the	supply	of	computing	power	will	grow

as	new	capacity	is	installed.	A	demand	spike	would	spur	production	in	existing
semiconductor	foundries	and	stimulate	the	construction	of	new	plants.	(A	one-off
performance	boost,	perhaps	amounting	to	one	or	two	orders	of	magnitude,	might
also	be	obtainable	by	using	customized	microprocessors.
14
)	Above	all,	the	rising
wave	of	technology	improvements	will	pour	increasing	volumes	of
computational	power	into	the	turbines	of	the	thinking	machines.	Historically,	the
rate	of	improvement	of	computing	technology	has	been	described	by	the	famous
Moore’s	law,	which	in	one	of	its	variations	states	that	computing	power	per
dollar	doubles	every	18	months	or	so.
15
	Although	one	cannot	bank	on	this	rate	of
improvement	continuing	up	to	the	development	of	human-level	machine
intelligence,	yet	until	fundamental	physical	limits	are	reached	there	will	remain
room	for	advances	in	computing	technology.
There	are	thus	reasons	to	expect	that	hardware	recalcitrance	will	not	be	very

15
high.	Purchasing	more	computing	power	for	the	system	once	it	proves	its	mettle
by	attaining	human-level	intelligence	might	easily	add	several	orders	of
magnitude	of	computing	power	(depending	on	how	hardware-frugal	the	project
was	before	expansion).	Chip	customization	might	add	one	or	two	orders	of
magnitude.	Other	means	of	expanding	the	hardware	base,	such	as	building	more
factories	and	advancing	the	frontier	of	computing	technology,	take	longer—

normally	several	years,	though	this	lag	would	be	radically	compressed	once
machine	superintelligence	revolutionizes	manufacturing	and	technology
development.
In	summary,	we	can	talk	about	the	likelihood	of	a	
hardware	overhang
:	when
human-level	software	is	created,	enough	computing	power	may	already	be
available	to	run	vast	numbers	of	copies	at	great	speed.	Software	recalcitrance,	as
discussed	above,	is	harder	to	assess	but	might	be	even	lower	than	hardware
recalcitrance.	In	particular,	there	may	be	
	in	the	form	of	pre-

:	when
made	content	(e.g.	the	Internet)	that	becomes	available	to	a	system	once	it
reaches	human	parity.	
Algorithm	overhang
—pre-designed	algorithmic
enhancements—is	also	possible	but	perhaps	less	likely.	Software	improvements
(whether	in	algorithms	or	content)	might	offer	orders	of	magnitude	of	potential
performance	gains	that	could	be	fairly	easily	accessed	once	a	digital	mind	attains
human	parity,	on	top	of	the	performance	gains	attainable	by	using	more	or	better
hardware.

Having	examined	the	question	of	recalcitrance	we	must	now	turn	to	the	other
half	of	our	schematic	equation,	
.	To	recall:	
Rate	of	change	in
Intelligence	=	Optimization	power/Recalcitrance
.	As	reflected	in	this	schematic,
a	fast	takeoff	does	not	require	that	recalcitrance	during	the	transition	phase	be
low.	A	fast	takeoff	could	also	result	if	recalcitrance	is	constant	or	even
moderately	increasing,	provided	the	optimization	power	being	applied	to
improving	the	system’s	performance	grows	sufficiently	rapidly.	As	we	shall	now
see,	there	are	good	
grounds	for	thinking	that	the	applied	optimization	power	
will
increase	during	the	transition,	at	least	in	the	absence	of	a	deliberate	measures	to
prevent	this	from	happening.
We	can	distinguish	two	phases.	The	first	phase	begins	with	the	onset	of	the
takeoff,	when	the	system	reaches	the	human	baseline	for	individual	intelligence.
As	the	system’s	capability	continues	to	increase,	it	might	use	some	or	all	of	that

will
capability	to	improve	itself	(or	to	design	a	successor	system—which,	for	present
purposes,	comes	to	the	same	thing).	However,	most	of	the	optimization	power
applied	to	the	system	still	comes	from	outside	the	system,	either	from	the	work
of	programmers	and	engineers	employed	within	the	project	or	from	such	work
done	by	the	rest	of	the	world	as	can	be	appropriated	and	used	by	the	project.
16
	If
this	phase	drags	out	for	any	significant	period	of	time,	we	can	expect	the	amount
of	optimization	power	applied	to	the	system	to	grow.	Inputs	both	from	inside	the

16
	If
project	and	from	the	outside	world	are	likely	to	increase	as	the	promise	of	the
chosen	approach	becomes	manifest.	Researchers	may	work	harder,	more
researchers	may	be	recruited,	and	more	computing	power	may	be	purchased	to
expedite	progress.	The	increase	could	be	especially	dramatic	if	the	development
of	human-level	machine	intelligence	takes	the	world	by	surprise,	in	which	case
what	was	previously	a	small	research	project	might	suddenly	become	the	focus
of	intense	research	and	development	efforts	around	the	world	(though	some	of

those	efforts	might	be	channeled	into	competing	projects).
A	second	growth	phase	will	begin	if	at	some	point	the	system	has	acquired	so
much	capability	that	most	of	the	optimization	power	exerted	on	it	comes	from
the	system	itself	(marked	by	the	variable	level	labeled	“crossover”	in	
Figure	7
).
This	fundamentally	changes	the	dynamic,	because	any	increase	in	the	system’s
capability	now	translates	into	a	proportional	increase	in	the	amount	of
optimization	power	being	applied	to	its	further	improvement.	If	recalcitrance

Figure	7
).
remains	constant,	this	feedback	dynamic	produces	exponential	growth	(see	
Box

4
).	The	doubling	constant	depends	on	the	scenario	but	might	be	extremely	short
—mere	seconds	in	some	scenarios—if	growth	is	occurring	at	electronic	speeds,
which	might	happen	as	a	result	of	algorithmic	improvements	or	the	exploitation
of	an	overhang	of	content	or	hardware.
17
	Growth	that	is	driven	by	physical
construction,	such	as	the	production	of	new	computers	or	manufacturing
equipment,	would	require	a	somewhat	longer	timescale	(but	still	one	that	might
be	very	short	compared	with	the	current	growth	rate	of	the	world	economy).
It	is	thus	likely	that	the	applied	optimization	power	will	increase	during	the
transition:	initially	because	humans	try	harder	to	improve	a	machine	intelligence
that	is	showing	spectacular	promise,	later	because	the	machine	intelligence	itself
becomes	capable	of	driving	further	progress	at	digital	speeds.	This	would	create
a	real	possibility	of	a	fast	or	medium	takeoff	
even	if	recalcitrance	were	constant
or	slightly	increasing	around	the	human	baseline
.
18

.
18
	Yet	we	saw	in	the	previous
subsection	that	there	are	factors	that	could	lead	to	a	big	drop	in	recalcitrance
around	the	human	baseline	level	of	capability.	These	factors	include,	for
example,	the	possibility	of	rapid	hardware	expansion	once	a	working	software
mind	has	been	attained;	the	possibility	of	algorithmic	improvements;	the
possibility	of	scanning	additional	brains	(in	the	case	of	whole	brain	emulation);
and	the	possibility	of	rapidly	incorporating	vast	amounts	of	content	by	digesting
the	Internet	(in	the	case	of	artificial	intelligence).
24

24
Box	4	On	the	kinetics	of	an	intelligence	explosion
	
We	can	write	the	rate	of	change	in	intelligence	as	the	ratio	between	the
optimization	power	applied	to	the	system	and	the	system’s	recalcitrance:
	
The	amount	of	optimization	power	acting	on	a	system	is	the	sum	of	whatever
optimization	power	the	system	itself	contributes	and	the	optimization	power
exerted	from	without.	For	example,	a	seed	AI	might	be	improved	through	a
combination	of	its	own	efforts	and	the	efforts	of	a	human	programming	team,

and	perhaps	also	the	efforts	of	the	wider	global	community	of	researchers
making	continuous	advances	in	the	semiconductor	industry,	computer	science,
and	related	fields:
19
	
A	seed	AI	starts	out	with	very	limited	cognitive	capacities.	At	the	outset,
therefore,	
	is	small.
20
	What	about	
	and	
?	There	are	cases	in
which	a	single	project	has	more	relevant	capability	than	the	rest	of	the	world
combined—the	Manhattan	project,	for	instance,	brought	a	very	large	fraction	of
the	world’s	best	physicists	to	Los	Alamos	to	work	on	the	atomic	bomb.	More
commonly,	any	one	project	contains	only	a	small	fraction	of	the	world’s	total
relevant	research	capability.	But	even	when	the	outside	world	has	a	greater	total
amount	of	relevant	research	capability	than	any	one	project,	
may
nevertheless	exceed	
,	since	much	of	the	outside	world’s	capability	is	not	be
focused	on	the	particular	system	in	question.	If	a	project	begins	to	look
promising—which	will	happen	when	a	system	passes	the	human	baseline	if	not

may
before—it	might	attract	additional	investment,	increasing	
.	If	the	project’s
accomplishments	are	public,	
	might	also	rise	as	the	progress	inspires	greater
interest	in	machine	intelligence	generally	and	as	various	powers	scramble	to	get
in	on	the	game.	During	the	transition	phase,	therefore,	total	optimization	power
applied	to	improving	a	cognitive	system	is	likely	to	increase	as	the	capability	of
the	system	increases.
21
As	the	system’s	capabilities	grow,	there	may	come	a	point	at	which	the
optimization	power	generated	by	the	system	itself	starts	to	dominate	the

21
optimization	power	applied	to	it	from	outside	(across	all	significant	dimensions
of	improvement):
	
This	
crossover
	is	significant	because	beyond	this	point,	further	improvement	to
the	system’s	capabilities	contributes	strongly	to	increasing	the	total	optimization
power	applied	to	improving	the	system.	We	thereby	enter	a	regime	
of	strong
recursive	self-improvement.	This	leads	to	explosive	growth	of	the	system’s

capability	under	a	fairly	wide	range	of	different	shapes	of	the	recalcitrance
curve.
To	illustrate,	consider	first	a	scenario	in	which	recalcitrance	is	constant,	so
that	the	rate	of	increase	in	an	AI’s	intelligence	is	equal	to	the	optimization	power
being	applied.	Assume	that	all	the	optimization	power	that	is	applied	comes
from	the	AI	itself	and	that	the	AI	applies	all	its	intelligence	to	the	task	of
amplifying	its	own	intelligence,	so	that	
	=	
I
.
22
	We	then	have
	
Solving	this	simple	differential	equation	yields	the	exponential	function
	
But	recalcitrance	being	constant	is	a	rather	special	case.	Recalcitrance	might
well	decline	around	the	human	baseline,	due	to	one	or	more	of	the	factors
mentioned	in	the	previous	subsection,	and	remain	low	around	the	crossover	and
some	distance	beyond	(perhaps	until	the	system	eventually	approaches
fundamental	physical	limits).	For	example,	suppose	that	the	optimization	power
applied	to	the	system	is	roughly	constant	(i.e.	
	+	
	≈	
c
)	prior	to	the

	+	
	≈	
c
system	becoming	capable	of	contributing	substantially	to	its	own	design,	and
that	this	leads	to	the	system	doubling	in	capacity	every	18	months.	(This	would
be	roughly	in	line	with	historical	improvement	rates	from	Moore’s	law	combined
with	software	advances.
23
)	This	rate	of	improvement,	if	achieved	by	means	of
roughly	constant	optimization	power,	entails	recalcitrance	declining	as	the
inverse	of	the	system	power:
	
If	recalcitrance	continues	to	fall	along	this	hyperbolic	pattern,	then	when	the	AI

23
	
reaches	the	crossover	point	the	total	amount	of	optimization	power	applied	to
improving	the	AI	has	doubled.	We	then	have

The	next	doubling	occurs	7.5	months	later.	Within	17.9	months,	the	system’s
capacity	has	grown	a	thousandfold,	thus	obtaining	speed	superintelligence
(
Figure	9
).
This	particular	growth	trajectory	has	a	positive	singularity	at	
t
	=	18	months.	In
reality,	the	assumption	that	recalcitrance	is	constant	would	cease	to	hold	as	the
system	began	to	approach	the	physical	limits	to	information	processing,	if	not
sooner.
These	two	scenarios	are	intended	for	illustration	only;	many	other	trajectories
are	possible,	depending	on	the	shape	of	the	recalcitrance	curve.	The	claim	is
simply	that	the	strong	feedback	loop	that	sets	in	around	the	crossover	point	tends
strongly	to	make	the	takeoff	faster	than	it	would	otherwise	have	been.
Figure	9
	One	simple	model	of	an	intelligence	explosion.
	
These	observations	notwithstanding,	the	shape	of	the	recalcitrance	curve	in
the	relevant	region	is	not	yet	well	characterized.	In	particular,	it	is	unclear	how

difficult	it	would	be	to	improve	the	software	quality	of	a	human-level	emulation
or	AI.	The	difficulty	of	expanding	the	hardware	power	available	to	a	system	is
also	not	clear.	Whereas	today	it	would	be	relatively	easy	to	increase	the
computing	power	available	to	a	small	project	by	spending	a	thousand	times	more
on	computing	power	or	by	waiting	a	few	years	for	the	price	of	computers	to	fall,
it	is	possible	that	the	first	machine	intelligence	to	reach	the	human	baseline	will
result	from	a	large	project	involving	pricey	supercomputers,	which	cannot	be
cheaply	scaled,	and	that	Moore’s	law	will	by	then	have	expired.	For	these
reasons,	although	a	fast	or	medium	takeoff	looks	more	likely,	the	possibility	of	a
slow	takeoff	cannot	be	excluded.
25

CHAPTER	5
Decisive	strategic	advantage
	
A	question	distinct	from,	but	related	to,	the	question	of	kinetics	is	whether
there	will	there	be	one	superintelligent	power	or	many?	Might	an
intelligence	explosion	propel	one	project	so	far	ahead	of	all	others	as	to
make	it	able	to	dictate	the	future?	Or	will	progress	be	more	uniform,
unfurling	across	a	wide	front,	with	many	projects	participating	but	none
securing	an	overwhelming	and	permanent	lead?
The	preceding	chapter	analyzed	one	key	parameter	in	determining	the	size	of	the
gap	that	might	plausibly	open	up	between	a	leading	power	and	its	nearest
competitors—namely,	the	speed	of	the	transition	from	human	to	strongly
superhuman	intelligence.	This	suggests	a	first-cut	analysis.	If	the	takeoff	is	
fast
(completed	over	the	course	of	hours,	days,	or	weeks)	then	it	is	unlikely	that	two
independent	projects	would	be	taking	off	concurrently:	almost	certainty,	the	first
project	would	have	completed	its	takeoff	before	any	other	project	would	have

fast
started	its	own.	If	the	takeoff	is	
slow
	(stretching	over	many	years	or	decades)
then	there	could	plausibly	be	multiple	projects	undergoing	takeoffs	concurrently,
so	that	although	the	projects	would	by	the	end	of	the	transition	have	gained
enormously	in	capability,	there	would	be	no	time	at	which	any	project	was	far
enough	ahead	of	the	others	to	give	it	an	overwhelming	lead.	A	takeoff	of
moderate
	speed	is	poised	in	between,	with	either	condition	a	possibility:	there
might	or	might	not	be	more	than	one	project	undergoing	the	takeoff	at	the	same

(stretching	over	many	years	or	decades)
moderate
time.
1
Will	one	machine	intelligence	project	get	so	far	ahead	of	the	competition	that
it	gets	a	
decisive	strategic	advantage
—that	is,	a	level	of	technological	and	other
advantages	sufficient	to	enable	it	to	achieve	complete	world	domination?	If	a
project	did	obtain	a	decisive	strategic	advantage,	would	it	use	it	to	suppress
competitors	and	form	a	
singleton
	(a	world	order	in	which	there	is	at	the	global
level	a	single	decision-making	agency)?	And	if	there	is	a	winning	project,	how
“large”	would

time.
1
it	gets	a	
singleton
“large”	would	
it	be—not	in	terms	of	physical	size	or	budget	but	in	terms	of	how
many	people’s	desires	would	be	controlling	its	design?	We	will	consider	these

questions	in	turn.
Will	the	frontrunner	get	a	decisive	strategic
advantage?
	
One	factor	influencing	the	width	of	the	gap	between	frontrunner	and	followers	is
the	rate	of	diffusion	of	whatever	it	is	that	gives	the	leader	a	competitive
advantage.	A	frontrunner	might	find	it	difficult	to	gain	and	maintain	a	large	lead
if	followers	can	easily	copy	the	frontrunner’s	ideas	and	innovations.	Imitation
creates	a	headwind	that	disadvantages	the	leader	and	benefits	laggards,
especially	if	intellectual	property	is	weakly	protected.	A	frontrunner	might	also
be	especially	vulnerable	to	expropriation,	taxation,	or	being	broken	up	under
anti-monopoly	regulation.
It	would	be	a	mistake,	however,	to	assume	that	this	headwind	must	increase
monotonically	with	the	gap	between	frontrunner	and	followers.	Just	as	a	racing
cyclist	who	falls	too	far	behind	the	competition	is	no	longer	shielded	from	the
wind	by	the	cyclists	ahead,	so	a	technology	follower	who	lags	sufficiently

behind	the	cutting	edge	might	find	it	hard	to	assimilate	the	advances	being	made
at	the	frontier.
2
	The	gap	in	understanding	and	capability	might	have	grown	too
large.	The	leader	might	have	migrated	to	a	more	advanced	technology	platform,
making	subsequent	innovations	untransferable	to	the	primitive	platforms	used	by
laggards.	A	sufficiently	pre-eminent	leader	might	have	the	ability	to	stem
information	leakage	from	its	research	programs	and	its	sensitive	installations,	or

2
to	sabotage	its	competitors’	efforts	to	develop	their	own	advanced	capabilities.
If	the	frontrunner	is	an	AI	system,	it	could	have	attributes	that	make	it	easier
for	it	to	expand	its	capabilities	while	reducing	the	rate	of	diffusion.	In	human-run
organizations,	economies	of	scale	are	counteracted	by	bureaucratic	inefficiencies
and	agency	problems,	including	difficulties	in	keeping	trade	secrets.
3
	These
problems	would	presumably	limit	the	growth	of	a	machine	intelligence	project

3
	These
so	long	as	it	is	operated	by	humans.	An	AI	system,	however,	might	avoid	some
of	these	scale	diseconomies,	since	the	AI’s	modules	(in	contrast	to	human
workers)	need	not	have	individual	preferences	that	diverge	from	those	of	the
system	as	a	whole.	Thus,	the	AI	system	could	avoid	a	sizeable	chunk	of	the
inefficiencies	arising	from	agency	problems	in	human	enterprises.	The	same

advantage—having	perfectly	loyal	parts—would	also	make	it	easier	for	an	AI
system	to	pursue	long-range	clandestine	goals.	An	AI	would	have	no	disgruntled
employees	ready	to	be	poached	by	competitors	or	bribed	into	becoming
informants.
4
We	can	get	a	sense	of	the	distribution	of	plausible	gaps	in	development	times
by	looking	at	some	historical	examples	(see	
Box	5
).	It	appears	that	lags	in	the
range	of	a	few	months	to	a	few	years	are	typical	of	strategically	significant
technology	projects.
Box	5	Technology	races:	some	historical	examples
	
Over	long	historical	timescales,	there	has	been	an	increase	in	the	rate	at	which
knowledge	and	technology	diffuse	around	the	globe.	As	a	result,	the	temporal
gaps	between	technology	leaders	and	nearest	followers	have	narrowed.
China	managed	to	maintain	a	monopoly	on	silk	production	for	over	two
thousand	years.	Archeological	finds	suggest	that	production	might	have	begun
around	3000	
BC
,	or	even	earlier.
5
	Sericulture	was	a	closely	held	secret.	Revealing

BC
5
the	techniques	was	punishable	by	death,	as	was	exporting	silkworms	or	their
eggs	outside	China.	The	Romans,	despite	the	high	price	commanded	by
imported	silk	cloth	in	their	empire,	never	learnt	the	art	of	silk	manufacture.	Not
until	around	
AD
	300	did	a	Japanese	expedition	manage	to	capture	some	silkworm
eggs	along	with	four	young	Chinese	girls,	who	were	forced	to	divulge	the	art	to
their	abductors.
6
	Byzantium	joined	the	club	of	producers	in	
AD
	522.	The	story	of

AD
6
AD
porcelain-making	also	features	long	lags.	The	craft	was	practiced	in	China
during	the	Tang	Dynasty	around	
AD
	600	(and	might	have	been	in	use	as	early	as
AD
	200),	but	was	mastered	by	Europeans	only	in	the	eighteenth	century.
7
Wheeled	vehicles	appeared	in	several	sites	across	Europe	and	Mesopotamia
around	3500	
BC
	but	reached	the	Americas	only	in	post-Columbian	times.
8
	On	a
grander	scale,	the	human	species	took	tens	of	thousands	of	years	to	spread	across
most	of	the	globe,	the	Agricultural	Revolution	thousands	of	years,	the	Industrial

AD
AD
7
BC
8
	On	a
Revolution	only	hundreds	of	years,	and	an	Information	Revolution	could	be	said
to	have	spread	globally	over	the	course	of	decades—though,	of	course,	these
transitions	are	not	necessarily	of	equal	profundity.	(The	
Dance	Dance	Revolution
video	game	spread	from	Japan	to	Europe	and	North	America	in	just	one	year!)

Technological	competition	has	been	quite	extensively	studied,	particularly	in
the	contexts	of	patent	races	and	arms	races.
9
	It	is	beyond	the	scope	of	our
investigation	to	review	this	literature	here.	However,	it	is	instructive	to	look	at
some	examples	of	strategically	significant	technology	races	in	the	twentieth
century	(see	
Table	7
).
With	regard	to	these	six	technologies,	which	were	regarded	as	strategically
important	by	the	rivaling	superpowers	because	of	their	military	or	symbolic
significance,	the	gaps	between	leader	and	nearest	laggard	were	(very
approximately)	49	months,	36	months,	4	months,	1	month,	4	months,	and	60
months,	respectively—longer	than	the	duration	of	a	fast	takeoff	and	shorter	than
the	duration	of	a	slow	takeoff.
10
	In	many	cases,	the	laggard’s	project	benefitted
from	espionage	and	publicly	available	information.	The	mere	demonstration	of
the	feasibility	of	an	invention	can	also	encourage	others	to	develop	it

10
independently;	and	fear	of	falling	behind	can	spur	the	efforts	to	catch	up.
Perhaps	closer	to	the	case	of	AI	are	mathematical	inventions	that	do	not
require	the	development	of	new	physical	infrastructure.	Often	these	are
published	in	the	academic	literature	and	can	thus	be	regarded	as	universally
available;	but	in	some	cases,	when	the	discovery	appears	to	offer	a	strategic
advantage,	publication	is	delayed.	For	example,	two	of	the	most	important	ideas
in	public-key	cryptography	are	the	Diffie–Hellman	key	exchange	protocol	and

the	RSA	encryption	scheme.	These	were	discovered	by	the	academic	community
in	1976	and	1978,	respectively,	but	it	has	later	been	confirmed	that	they	were
known	by	cryptographers	at	the	UK’s	communications	security	group	since	the
early	1970s.
20
	Large	software	projects	might	offer	a	closer	analogy	with	AI
projects,	but	it	is	harder	to	give	crisp	examples	of	typical	lags	because	software
is	usually	rolled	out	in	incremental	installments	and	the	functionalities	of
competing	systems	are	often	not	directly	comparable.
Table	7

20
Table	7	
Some	strategically	significant	technology	races

It	is	possible	that	globalization	and	increased	surveillance	will	reduce	typical
lags	between	competing	technology	projects.	Yet	there	is	likely	to	be	a	lower
bound	on	how	short	the	average	lag	could	become	(in	the	absence	of	deliberate
coordination).
21
	Even	absent	dynamics	that	lead	to	snowball	effects,	some
projects	will	happen	to	end	up	with	better	research	staff,	leadership,	and
infrastructure,	or	will	just	stumble	upon	better	ideas.	If	two	projects	pursue
alternative	approaches,	one	of	which	turns	out	to	work	better,	it	may	take	the
rival	project	many	months	to	switch	to	the	superior	approach	even	if	it	is	able	to
closely	monitor	what	the	forerunner	is	doing.
Combining	these	observations	with	our	earlier	discussion	of	the	speed	of	the
takeoff,	we	can	conclude	that	it	is	highly	unlikely	that	two	projects	would	be
close	enough	to	undergo	a	fast	takeoff	concurrently;	for	a	medium	takeoff,	it
could	easily	go	either	way;	and	for	a	slow	takeoff,	it	is	highly	likely	that	several

projects	would	undergo	the	process	in	parallel.	But	the	analysis	needs	a	further
step.	The	key	question	is	not	how	many	projects	undergo	a	takeoff	in	tandem,
but	how	many	projects	emerge	on	the	yonder	side	sufficiently	tightly	clustered	in
capability	that	none	of	them	has	a	decisive	strategic	advantage.	If	the	takeoff
process	is	relatively	slow	to	begin	and	then	gets	faster,	the	distance	between
competing	projects	would	tend	to	grow.	To	return	to	our	bicycle	metaphor,	the
situation	would	be	analogous	to	a	pair	of	cyclists	making	their	way	up	a	steep

hill,	one	trailing	some	distance	behind	the	other—the	gap	between	them	then
expanding	as	the	frontrunner	reaches	the	peak	and	starts	accelerating	down	the
other	side.
Consider	the	following	medium	takeoff	scenario.	Suppose	it	takes	a	project
one	year	to	increase	its	AI’s	capability	from	the	human	baseline	to	a	strong
superintelligence,	and	that	one	project	enters	this	takeoff	phase	with	a	six-month
lead	over	the	next	most	advanced	project.	The	two	projects	will	be	undergoing	a

takeoff	concurrently.	It	might	seem,	then,	that	neither	project	gets	a	decisive
strategic	advantage.	But	that	need	not	be	so.	Suppose	it	takes	nine	months	to
advance	from	the	human	baseline	to	the	crossover	point,	and	another	three
months	from	there	to	strong	superintelligence.	The	frontrunner	then	attains
strong	superintelligence	three	months	before	the	following	project	even	reaches

the	crossover	point.	This	
would	give	the	leading	project	a	decisive	strategic
advantage	and	the	opportunity	to	parlay	its	lead	into	permanent	control	by
disabling	the	competing	projects	and	establishing	a	singleton.	(Note	that	the
concept	of	a	singleton	is	an	abstract	one:	a	singleton	could	be	democracy,	a
tyranny,	a	single	dominant	AI,	a	strong	set	of	global	norms	that	include	effective
provisions	for	their	own	enforcement,	or	even	an	alien	overlord—its	defining
characteristic	being	simply	that	it	is	some	form	of	agency	that	can	solve	all
major	global	coordination	problems.	It	may,	but	need	not,	resemble	any	familiar
form	of	human	governance.
22
)
Since	there	is	an	especially	strong	prospect	of	explosive	growth	just	after	the
crossover	point,	when	the	strong	positive	feedback	loop	of	optimization	power
kicks	in,	a	scenario	of	this	kind	is	a	serious	possibility,	and	it	increases	the
chances	that	the	leading	project	will	attain	a	decisive	strategic	advantage	even	if
the	takeoff	is	not	fast.

22
)
	
Some	paths	to	superintelligence	require	great	resources	and	are	therefore	likely
to	be	the	preserve	of	large	well-funded	projects.	Whole	brain	emulation,	for
instance,	requires	many	different	kinds	of	expertise	and	lots	of	equipment.
Biological	intelligence	enhancements	and	brain–computer	interfaces	would	also
have	a	large	scale	factor:	while	a	small	biotech	firm	might	invent	one	or	two
drugs,	achieving	superintelligence	along	one	of	these	paths	(if	doable	at	all)

would	likely	require	many	inventions	and	many	tests,	and	therefore	the	backing
of	an	industrial	sector	or	a	well-funded	national	program.	Achieving	collective
superintelligence	by	making	organizations	and	networks	more	efficient	requires
even	more	extensive	input,	involving	much	of	the	world	economy.
The	AI	path	is	more	difficult	to	assess.	Perhaps	it	would	require	a	very	large
research	program;	perhaps	it	could	be	done	by	a	small	group.	A	lone	hacker
scenario	cannot	be	excluded	either.	Building	a	seed	AI	might	require	insights

and	algorithms	developed	over	many	decades	by	the	scientific	community
around	the	world.	But	it	is	possible	that	the	last	critical	breakthrough	idea	might
come	from	a	single	individual	or	a	small	group	that	succeeds	in	putting
everything	together.	This	scenario	is	less	realistic	for	some	AI	architectures	than
others.	A	system	that	has	a	large	number	of	parts	that	need	to	be	tweaked	and

tuned	to	work	effectively	together,	and	then	painstakingly	loaded	with	custom-
made	cognitive	content,	is	likely	to	require	a	larger	project.	But	if	a	seed	AI
could	be	instantiated	as	a	simple	system,	one	whose	construction	depends	only
on	getting	a	few	basic	principles	right,	then	the	feat	might	be	within	the	reach	of
a	small	team	or	an	individual.	The	likelihood	of	the	final	breakthrough	being
made	by	a	small	project	increases	if	most	previous	progress	in	the	field	has	been
published	in	the	open	literature	or	made	available	as	open	source	software.
We	must	distinguish	the	question	of	how	big	will	be	the	project	that	directly
engineers
	the	system	from	the	question	of	how	big	the	group	will	be	that
controls
	
whether,	how,	and	when	the	system	is	created.	The	atomic	bomb	was
created	primarily	by	a	group	of	scientists	and	engineers.	(The	Manhattan	Project
employed	about	130,000	people	at	its	peak,	the	vast	majority	of	whom	were
construction	workers	or	building	operators.
23
)	These	technical	experts,	however,

engineers
controls
	
23
were	controlled	by	the	US	military,	which	was	directed	by	the	US	government,
which	was	ultimately	accountable	to	the	American	electorate,	which	at	the	time
constituted	about	one-tenth	of	the	adult	world	population.
24
	
Given	the	extreme	security	implications	of	superintelligence,	governments
would	likely	seek	to	nationalize	any	project	on	their	territory	that	they	thought
close	to	achieving	a	takeoff.	A	powerful	state	might	also	attempt	to	acquire
projects	located	in	other	countries	through	espionage,	theft,	kidnapping,	bribery,

24
	
threats,	military	conquest,	or	any	other	available	means.	A	powerful	state	that
cannot	acquire	a	foreign	project	might	instead	destroy	it,	especially	if	the	host
country	lacks	an	effective	deterrent.	If	global	governance	structures	are	strong	by
the	time	a	breakthrough	begins	to	look	imminent,	it	is	possible	that	promising
projects	would	be	placed	under	international	control.
An	important	question,	therefore,	is	whether	national	or	international
authorities	will	see	an	intelligence	explosion	coming.	At	present,	intelligence

agencies	do	not	appear	to	be	looking	very	hard	for	promising	AI	projects	or
other	forms	of	potentially	explosive	intelligence	amplification.
25
	If	they	are
indeed	not	paying	(much)	attention,	this	is	presumably	due	to	the	widely	shared
perception	that	there	is	no	prospect	whatever	of	imminent	superintelligence.	If
and	when	it	becomes	a	common	belief	among	prestigious	scientists	that	there	is

a	substantial	chance	that	superintelligence	is	just	around	the	corner,	the	major
intelligence	agencies	of	the	world	would	probably	start	to	monitor	groups	and
individuals	who	seem	to	be	engaged	in	relevant	research.	Any	project	that	began
to	show	sufficient	progress	could	then	be	promptly	nationalized.	If	political
elites	were	persuaded	by	the	seriousness	of	the	risk,	civilian	efforts	in	sensitive
areas	might	be	regulated	or	outlawed.
How	difficult	would	such	monitoring	be?	The	task	is	easier	if	the	goal	is	only
to	keep	track	of	the	leading	project.	In	that	case,	surveillance	focusing	on	the
several	best-resourced	projects	may	be	sufficient.	If	the	goal	is	instead	to	prevent
any	work	from	taking	place	(at	least	outside	of	specially	authorized	institutions)
then	surveillance	would	have	to	be	more	comprehensive,	since	many	small
projects	and	individuals	are	in	a	position	to	make	at	least	some	progress.
It	would	be	easier	to	monitor	projects	that	require	significant	amounts	of

physical	capital,	as	would	be	the	case	with	a	whole	brain	emulation	project.
Artificial	intelligence	research,	by	contrast,	requires	only	a	personal	computer,
and	would	therefore	be	more	difficult	to	monitor.	Some	of	the	theoretical	work
could	be	done	with	pen	and	paper.	Even	so,	it	would	not	be	too	difficult	to
identify	most	capable	individuals	with	a	serious	long-standing	interest	in
artificial	general	intelligence	
research.	Such	individuals	usually	leave	visible
trails.	They	may	have	published	academic	papers,	presented	at	conferences,

posted	on	Internet	forums,	or	earned	degrees	from	leading	computer	science
departments.	They	may	also	have	had	communications	with	other	AI
researchers,	allowing	them	to	be	identified	by	mapping	the	social	graph.
Projects	designed	from	the	outset	to	be	secret	could	be	more	difficult	to	detect.
An	ordinary	software	development	project	could	serve	as	a	front.
26
	Only	careful
analysis	of	the	code	being	produced	would	reveal	the	true	nature	of	what	the
project	was	trying	to	accomplish.	Such	analysis	would	require	a	lot	of	(highly

26
skilled)	manpower,	whence	only	a	small	number	of	suspect	projects	could	be
scrutinized	at	this	level.	The	task	would	become	much	easier	if	effective	lie
detection	technology	had	been	developed	and	could	be	routinely	used	in	this
kind	of	surveillance.
27
Another	reason	states	might	fail	to	catch	precursor	developments	is	the
inherent	difficulty	of	forecasting	some	types	of	breakthrough.	This	is	more
relevant	to	AI	research	than	to	whole	brain	emulation	development,	since	for	the
latter	the	key	breakthrough	is	more	likely	to	be	preceded	by	a	clear	gradient	of

steady	advances.
It	is	also	possible	that	intelligence	agencies	and	other	government
bureaucracies	have	a	certain	clumsiness	or	rigidity	that	might	prevent	them	from
understanding	the	significance	of	some	developments	that	might	be	clear	to
some	outside	groups.	Barriers	to	official	understanding	of	a	potential	intelligence
explosion	might	be	especially	steep.	It	is	conceivable,	for	example,	that	the	topic
will	become	inflamed	with	religious	or	political	controversies,	rendering	it	taboo
for	officials	in	some	countries.	The	topic	might	become	associated	with	some
discredited	figure	or	with	charlatanry	and	hype	in	general,	hence	shunned	by
respected	scientists	and	other	establishment	figures.	(As	we	saw	in	
Chapter	1
,
something	like	this	has	already	happened	twice:	recall	the	two	“AI	winters.”)
Industry	groups	might	lobby	to	prevent	aspersions	being	cast	on	profitable
business	areas;	academic	communities	might	close	ranks	to	marginalize	those

Chapter	1
,
who	voice	concerns	about	long-term	consequences	of	the	science	that	is	being
done.
28
Consequently,	a	total	intelligence	failure	cannot	be	ruled	out.	Such	a	failure	is
especially	likely	if	breakthroughs	should	occur	in	the	nearer	future,	before	the
issue	has	risen	to	public	prominence.	And	even	if	intelligence	agencies	get	it
right,	political	leaders	might	not	listen	or	act	on	the	advice.	Getting	the
Manhattan	Project	started	took	an	extraordinary	effort	by	several	visionary
physicists,	including	especially	Mark	Oliphant	and	Leó	Szilárd:	the	latter

done.
28
persuaded	Eugene	Wigner	to	persuade	Albert	Einstein	to	put	his	name	on	a	letter
to	persuade	President	Franklin	D.	Roosevelt	to	look	into	the	matter.	Even	after
the	project	reached	its	full	scale,	Roosevelt	remained	skeptical	of	its	workability
and	significance,	as	did	his	successor	Harry	Truman.
For	better	or	worse,	it	would	probably	be	harder	for	a	small	group	of	activists
to	affect	the	outcome	of	an	intelligence	explosion	if	big	players,	such	as	states,
are	taking	active	part.	Opportunities	for	private	individuals	to	reduce	the	overall

amount	of	existential	risk	from	a	potential	intelligence	explosion	are	therefore
greatest	in	scenarios	in	which	big	players	remain	relatively	oblivious	to	the
issue,	or	in	which	the	early	efforts	of	activists	make	a	major	difference	to
whether,	when,	which,	or	with	what	attitude	big	players	enter	the	game.	Activists
seeking	maximum	expected	impact	may	therefore	wish	to	focus	most	of	their
planning	on	such	high-leverage	scenarios,	even	if	they	believe	that	scenarios	in
which	big	players	end	up	calling	all	the	shots	are	more	probable.

International	coordination	is	more	likely	if	global	governance	structures
generally	get	stronger.	Coordination	might	also	be	more	likely	if	the	significance
of	an	intelligence	explosion	is	widely	appreciated	ahead	of	time	and	if	effective
monitoring	of	all	serious	projects	is	feasible.	Even	if	monitoring	is	infeasible,
however,	international	cooperation	would	still	be	possible.	Many	countries	could
band	together	to	support	a	joint	project.	If	such	a	joint	project	were	sufficiently
well	resourced,	it	could	have	a	good	chance	of	being	the	first	to	reach	the	goal,
especially	if	any	rival	project	had	to	be	small	and	secretive	to	elude	detection.
There	are	precedents	of	large-scale	successful	multinational	scientific
collaborations,	such	as	the	International	Space	Station,	the	Human	Genome
Project,	and	the	Large	Hadron	Collider.
29
	However,	the	major	motivation	for
collaboration	in	those	cases	was	cost-sharing.	(In	the	case	of	the	International

29
Space	Station,	fostering	a	collaborative	spirit	between	Russia	and	the	United
States	was	itself	an	important	goal.
30
)	Achieving	similar	collaboration	on	a
project	that	has	enormous	security	implications	would	be	more	difficult.	A
country	that	believed	it	could	achieve	a	breakthrough	unilaterally	might	be
tempted	to	go	it	alone	rather	than	subordinate	its	efforts	to	a	joint	project.	A
country	might	also	refrain	from	joining	an	international	collaboration	from	fear
that	other	participants	might	siphon	off	collaboratively	generated	insights	and

30
use	them	to	accelerate	a	covert	national	project.
An	international	project	would	thus	need	to	overcome	major	security
challenges,	and	a	fair	amount	of	trust	would	probably	be	needed	to	get	it	started,
trust	that	may	take	time	to	develop.	Consider	that	even	after	the	thaw	in	relations
between	the	United	States	and	the	Soviet	Union	following	Gorbachev’s	ascent	to
power,	arms	reduction	efforts—which	could	be	greatly	in	the	interests	of	both
superpowers—had	a	fitful	beginning.	Gorbachev	was	seeking	steep	reductions	in

nuclear	arms	but	negotiations	stalled	on	the	issue	of	Reagan’s	Strategic	Defense
Initiative	(“Star	Wars”),	which	the	Kremlin	strenuously	opposed.	At	the
Reykjavík	Summit	meeting	in	1986,	Reagan	proposed	that	the	United	States
would	share	with	the	Soviet	Union	the	technology	that	would	be	developed
under	the	Strategic	Defense	Initiative,	so	that	both	countries	could	enjoy
protection	against	accidental	launches	and	against	smaller	nations	that	might
develop	nuclear	weapons.	Yet	Gorbachev	was	not	persuaded	by	this	apparent

win–win	proposition.	He	viewed	the	gambit	as	a	ruse,	refusing	to	credit	the
notion	that	the	Americans	would	share	the	fruits	of	their	most	advanced	military
research	at	a	time	when	they	were	not	even	willing	to	share	with	the	Soviets
their	technology	for	milking	cows.
31
	Regardless	of	whether	Reagan	was	in	fact
sincere	in	his	offer	of	superpower	collaboration,	mistrust	made	the	proposal	a
non-starter.
Collaboration	is	easier	to	achieve	between	allies,	but	even	there	it	is	not
automatic.	When	the	Soviet	Union	and	the	United	States	were	allied	against
Germany	during	World	War	II,	the	United	States	concealed	its	atomic	bomb
project	from	the	Soviet	Union.	The	United	States	did	collaborate	on	the
Manhattan	Project	with	Britain	and	Canada.
32
	Similarly,	the	United	Kingdom
concealed	its	success	in	breaking	the	German	Enigma	code	from	the	Soviet
Union,	but	shared	it—albeit	with	some	difficulty—with	the	United	States.
33
	This
suggests	that	in	order	to	achieve	international	collaboration	on	some	technology

32
33
	This
that	is	of	pivotal	importance	for	national	security,	it	might	be	necessary	to	have
built	beforehand	a	close	and	trusting	relationship.
We	will	return	in	
Chapter	14
	to	the	desirability	and	feasibility	of	international
collaboration	in	the	development	of	intelligence	amplification	technologies.
	
Would	a	project	that	obtained	a	decisive	strategic	advantage	choose	to	use	it	to
form	a	singleton?
Consider	a	vaguely	analogous	historical	situation.	The	United	States

Chapter	14
	
developed	nuclear	weapons	in	1945.	It	was	the	sole	nuclear	power	until	the
Soviet	Union	developed	the	atom	bomb	in	1949.	During	this	interval—and	for
some	time	thereafter—the	United	States	may	have	had,	or	been	in	a	position	to
achieve,	a	decisive	military	advantage.
The	United	States	could	then,	theoretically,	have	used	its	nuclear	monopoly	to
create	a	singleton.	One	way	in	which	it	could	have	done	so	would	have	been	by
embarking	on	an	all-out	effort	to	build	up	its	nuclear	arsenal	and	then	threatening
(and	if	necessary,	carrying	out)	a	nuclear	first	strike	to	destroy	the	industrial

capacity	of	any	incipient	nuclear	program	in	the	USSR	and	any	other	country

tempted	to	develop	a	nuclear	capability.
A	more	benign	course	of	action,	which	might	also	have	had	a	chance	of
working,	would	have	been	to	use	its	nuclear	arsenal	as	a	bargaining	chip	to
negotiate	a	strong	international	government—a	veto-less	United	Nations	with	a
nuclear	monopoly	and	a	mandate	to	take	all	necessary	actions	to	prevent	any
country	from	developing	its	own	nuclear	weapons.
Both	of	these	approaches	were	proposed	at	the	time.	The	hardline	approach	of
launching	or	threatening	a	first	strike	was	advocated	by	some	prominent
intellectuals	such	as	Bertrand	Russell	(who	had	long	been	active	in	anti-war
movements	and	who	would	later	spend	decades	campaigning	against	nuclear
weapons)	and	John	von	Neumann	(co-creator	of	game	theory	and	one	of	the
architects	of	
US	nuclear	strategy).
34
	Perhaps	it	is	a	sign	of	civilizational	progress
that	the	very	idea	of	threatening	a	nuclear	first	strike	today	seems	borderline	silly
or	morally	obscene.

34
A	version	of	the	benign	approach	was	tried	in	1946	by	the	United	States	in	the
form	of	the	Baruch	plan.	The	proposal	involved	the	USA	giving	up	its	temporary
nuclear	monopoly.	Uranium	and	thorium	mining	and	nuclear	technology	would
be	placed	under	the	control	of	an	international	agency	operating	under	the
auspices	of	the	United	Nations.	The	proposal	called	for	the	permanent	members
of	the	Security	Council	to	give	up	their	vetoes	in	matters	related	to	nuclear

weapons	in	order	to	prevent	any	great	power	found	to	be	in	breach	of	the	accord
from	vetoing	the	imposition	of	remedies.
35
	Stalin,	seeing	that	the	Soviet	Union
and	its	allies	could	be	easily	outvoted	in	both	the	Security	Council	and	the
General	Assembly,	rejected	the	proposal.	A	frosty	atmosphere	of	mutual
suspicion	descended	on	the	relations	between	the	former	wartime	allies,	mistrust
that	soon	solidified	into	the	Cold	War.	As	had	been	widely	predicted,	a	costly
and	extremely	dangerous	nuclear	arms	race	followed.

35
Many	factors	might	dissuade	a	human	organization	with	a	decisive	strategic
advantage	from	creating	a	singleton.	These	include	non-aggregative	or	bounded
utility	functions,	non-maximizing	decision	rules,	confusion	and	uncertainty,
coordination	problems,	and	various	costs	associated	with	a	takeover.	But	what	if
it	were	not	a	human	organization	but	a	superintelligent	artificial	agent	that	came
into	possession	of	a	decisive	strategic	advantage?	Would	the	aforementioned
factors	be	equally	effective	at	inhibiting	an	AI	from	attempting	to	seize	power?

Let	us	briefly	run	through	the	list	of	factors	and	consider	how	they	might	apply

in	this	case.
Human	individuals	and	human	organizations	typically	have	preferences	over
resources	that	are	not	well	represented	by	an	“unbounded	aggregative	utility
function.”	A	human	will	typically	not	wager	all	her	capital	for	a	fifty–fifty
chance	of	doubling	it.	A	state	will	typically	not	risk	losing	all	its	territory	for	a
ten	percent	chance	of	a	tenfold	expansion.	For	individuals	and	governments,
there	are	diminishing	returns	to	most	resources.	The	same	need	
not
	hold	for	AIs.
(We	will	return	to	the	problem	of	AI	motivation	in	subsequent	chapters.)	An	AI
might	therefore	be	more	likely	to	pursue	a	risky	course	of	action	that	has	some
chance	of	giving	it	control	of	the	world.
Humans	and	human-run	organizations	may	also	operate	with	decision
processes	that	do	not	seek	to	maximize	expected	utility.	For	example,	they	may
allow	for	fundamental	risk	aversion,	or	“satisficing”	decision	rules	that	focus	on
meeting	adequacy	thresholds,	or	“deontological”	side-constraints	that	proscribe

certain	kinds	of	action	regardless	of	how	desirable	their	consequences.	Human
decision	makers	often	seem	to	be	acting	out	an	identity	or	a	social	role	rather
than	seeking	to	maximize	the	achievement	of	some	particular	objective.	Again,
this	need	not	apply	to	artificial	agents.
Bounded	utility	functions,	risk	aversion,	and	non-maximizing	decision	rules
may	combine	synergistically	with	strategic	confusion	and	uncertainty.
Revolutions,	even	when	they	succeed	in	overthrowing	the	existing	order,	often

fail	to	produce	the	outcome	that	their	instigators	had	promised.	This	tends	to	stay
the	hand	of	a	human	agent	if	the	contemplated	action	is	irreversible,	norm-
breaking,	and	lacking	precedent.	A	superintelligence	might	perceive	the	situation
more	clearly	and	therefore	face	less	strategic	confusion	and	uncertainty	about	the
outcome	should	it	attempt	to	use	its	apparent	decisive	strategic	advantage	to
consolidate	its	dominant	position.
Another	major	factor	that	can	inhibit	groups	from	exploiting	a	potentially

decisive	strategic	advantage	is	the	problem	of	internal	coordination.	Members	of
a	conspiracy	that	is	in	a	position	to	seize	power	must	worry	not	only	about	being
infiltrated	from	the	outside,	but	also	about	being	overthrown	by	some	smaller
coalition	of	insiders.	If	a	group	consists	of	a	hundred	people,	and	a	majority	of
sixty	can	take	power	and	disenfranchise	the	non-conspirators,	what	is	then	to
stop	a	thirty-five-strong	subset	of	these	sixty	from	disenfranchising	the	other
twenty-five?	And	then	maybe	a	subset	of	twenty	disenfranchising	the	other

fifteen?	Each	of	the	original	hundred	might	have	good	reason	to	uphold	certain
established	norms	to	prevent	the	general	unraveling	that	could	result	from	any
attempt	to	change	the	social	contract	by	means	of	a	naked	power	grab.	This
problem	of	internal	coordination	would	not	apply	to	an	AI	system	that
constitutes	a	single	unified	agent.
36
Finally,	there	is	the	issue	of	cost.	Even	if	the	United	States	could	have	used	its
nuclear	monopoly	to	establish	a	singleton,	it	might	not	have	been	able	to	do	so
without	incurring	substantial	costs.	In	the	case	of	a	negotiated	agreement	to
place	nuclear	weapons	under	the	control	of	a	reformed	and	strengthened	United
Nations,	these	costs	might	have	been	relatively	small;	but	the	costs—moral,
economic,	political,	and	human—of	actually	attempting	world	conquest	through
the	waging	of	nuclear	war	would	have	been	almost	unthinkably	large,	even
during	the	period	of	nuclear	monopoly.	With	sufficient	technological	superiority,

however,	these	costs	would	be	far	smaller.	Consider,	for	example,	a	scenario	in
which	one	nation	had	such	a	vast	technological	lead	that	it	could	safely	disarm
all	other	nations	at	the	press	of	a	button,	without	anybody	dying	or	being	injured,
and	with	almost	no	damage	to	infrastructure	or	to	the	environment.	With	such
almost	magical	technological	superiority,	a	first	strike	would	be	a	lot	more
tempting.	Or	consider	an	even	greater	level	of	technological	superiority	which
might	enable	the	frontrunner	to	cause	other	nations	to	voluntarily	lay	down	their

arms,	not	by	threatening	them	with	destruction	but	simply	by	persuading	a	great
majority	of	their	populations	by	means	of	an	extremely	effectively	designed
advertising	and	propaganda	campaign	extolling	the	virtues	of	global	unity.	If	this
were	done	with	the	intention	to	benefit	everybody,	for	instance	by	replacing
national	rivalries	and	arms	races	with	a	fair,	representative,	and	effective	world
government,	it	is	not	clear	that	there	would	be	even	a	cogent	moral	objection	to

the	leveraging	of	a	temporary	strategic	advantage	into	a	permanent	singleton.
Various	considerations	thus	point	to	an	increased	likelihood	that	a	future
power	with	superintelligence	that	obtained	a	sufficiently	large	strategic
advantage	would	
actually	use	it	to	form	a	singleton.	The	desirability	of	such	an
outcome	depends,	of	course,	on	the	nature	of	the	singleton	that	would	be	created
and	also	on	what	the	future	of	intelligent	life	would	look	like	in	alternative

multipolar	scenarios.	We	will	revisit	those	questions	in	later	chapters.	But	first
let	us	take	a	closer	look	at	why	and	how	a	superintelligence	would	be	powerful
and	effective	at	achieving	outcomes	in	the	world.

CHAPTER	6
Cognitive	superpowers
	
Suppose	that	a	digital	superintelligent	agent	came	into	being,	and	that	for
some	reason	it	wanted	to	take	control	of	the	world:	would	it	be	able	to	do
so?	In	this	chapter	we	consider	some	powers	that	a	superintelligence	could
develop	and	what	they	may	enable	it	to	do.	We	outline	a	takeover	scenario
that	illustrates	how	a	superintelligent	agent,	starting	as	mere	software,
could	establish	itself	as	a	singleton.	We	also	offer	some	remarks	on	the
relation	between	power	over	nature	and	power	over	other	agents.
The	principal	reason	for	humanity’s	dominant	position	on	Earth	is	that	our	brains
have	a	slightly	expanded	set	of	faculties	compared	with	other	animals.
1
	Our
greater	intelligence	lets	us	transmit	culture	more	efficiently,	with	the	result	that
knowledge	and	technology	accumulates	from	one	generation	to	the	next.	By	now
sufficient	content	has	accumulated	to	make	possible	space	flight,	H-bombs,
genetic	engineering,	computers,	factory	farms,	insecticides,	the	international

1
	Our
peace	movement,	and	all	the	accouterments	of	modern	civilization.	Geologists
have	started	referring	to	the	present	era	as	the	
Anthropocene
	in	recognition	of	the
distinctive	biotic,	sedimentary,	and	geochemical	signatures	of	human	activities.
2
On	one	estimate,	we	appropriate	24%	of	the	planetary	ecosystem’s	net	primary
production.
3
	And	yet	we	are	far	from	having	reached	the	physical	limits	of
technology.
These	observations	make	it	plausible	that	any	type	of	entity	that	developed	a

2
3
much	greater	than	human	level	of	intelligence	would	be	potentially	extremely
powerful.	Such	entities	could	accumulate	content	much	faster	than	us	and	invent
new	technologies	on	a	much	shorter	timescale.	They	could	also	use	their
intelligence	to	strategize	more	effectively	than	we	can.
Let	us	consider	some	of	the	capabilities	that	a	superintelligence	could	have
and	how	it	could	use	them.

It	is	important	not	to	anthropomorphize	superintelligence	when	thinking	about
its	potential	impacts.	Anthropomorphic	frames	encourage	unfounded
expectations	about	the	growth	trajectory	of	a	seed	AI	and	about	the	psychology,
motivations,	and	capabilities	of	a	mature	superintelligence.
For	example,	a	common	assumption	is	that	a	superintelligent	machine	would
be	like	a	very	clever	but	nerdy	human	being.	We	imagine	that	the	AI	has	book
smarts	but	lacks	social	savvy,	or	that	it	is	logical	but	not	intuitive	and	creative.
This	idea	probably	originates	in	observation:	we	look	at	present-day	computers
and	see	that	they	are	good	at	calculation,	remembering	facts,	and	at	following
the	letter	of	instructions	while	being	oblivious	to	social	contexts	and	subtexts,
norms,	emotions,	and	politics.	The	association	is	strengthened	when	we	observe
that	the	people	who	are	good	at	working	with	computers	tend	themselves	to	be
nerds.	So	it	is	natural	to	assume	that	more	advanced	computational	intelligence

will	have	similar	attributes,	only	to	a	higher	degree.
This	heuristic	might	retain	some	validity	in	the	early	stages	of	development	of
a	seed	AI.	(There	is	no	reason	whatever	to	suppose	that	it	would	apply	to
emulations	or	to	cognitively	enhanced	humans.)	In	its	immature	stage,	what	is
later	to	become	a	superintelligent	AI	might	still	lack	many	skills	and	talents	that
come	naturally	to	a	human;	and	the	pattern	of	such	a	seed	AI’s	strengths	and
weaknesses	
might
	indeed	bear	some	vague	resemblance	to	an	IQ	nerd.	The	most

might
essential	characteristic	of	a	seed	AI,	aside	from	being	easy	to	improve	(having
low	recalcitrance),	is	being	good	at	exerting	optimization	power	to	amplify	a
system’s	intelligence:	a	skill	which	is	presumably	closely	related	to	doing	well	in
mathematics,	programming,	engineering,	computer	science	research,	and	other
such	“nerdy”	pursuits.	However,	even	if	a	seed	AI	does	have	such	a	nerdy
capability	profile	at	one	stage	of	its	development,	this	does	not	entail	that	it	will

grow	into	a	similarly	limited	mature	superintelligence.	Recall	the	distinction
between	direct	and	indirect	reach.	With	sufficient	skill	at	intelligence
amplification,	all	other	intellectual	abilities	are	within	a	system’s	indirect	reach:
the	system	can	develop	new	cognitive	modules	and	skills	as	needed—including
empathy,	political	acumen,	and	any	other	powers	stereotypically	wanting	in
computer-like	personalities.
Even	if	we	recognize	that	a	superintelligence	can	have	all	the	skills	and	talents

we	find	in	the	human	distribution,	along	with	other	talents	that	are	not	found

among	humans,	the	tendency	toward	anthropomorphizing	can	still	lead	us	to
underestimate	the	extent	to	which	a	machine	superintelligence	could	exceed	the
human	level	of	performance.	Eliezer	Yudkowsky,	as	we	saw	in	an	earlier
chapter,	has	been	particularly	emphatic	in	condemning	this	kind	of
misconception:	our	intuitive	concepts	of	“smart”	and	“stupid”	are	distilled	from
our	experience	of	variation	over	the	range	of	human	thinkers,	yet	the	differences
in	cognitive	ability	within	this	human	cluster	are	trivial	in	comparison	to	the
differences	between	any	human	intellect	and	a	superintelligence.
4
Chapter	3
	reviewed	some	of	the	potential	sources	of	advantage	for	machine
intelligence.	The	magnitudes	of	the	advantages	are	such	as	to	suggest	that	rather
than	thinking	of	a	superintelligent	AI	as	smart	in	the	sense	that	a	scientific
genius	is	smart	compared	with	the	average	human	being,	it	might	be	closer	to	the
mark	to	think	of	such	an	AI	as	smart	in	the	sense	that	an	average	human	being	is

4
Chapter	3
smart	compared	with	a	beetle	or	a	worm.
It	would	be	convenient	if	we	could	quantify	the	cognitive	caliber	of	an
arbitrary	cognitive	system	using	some	familiar	metric,	such	as	IQ	scores	or	some
version	of	the	Elo	ratings	that	measure	the	relative	abilities	of	players	in	two-
player	games	such	as	chess.	But	these	metrics	are	not	useful	in	the	context	of
superhuman	artificial	general	intelligence.	We	are	not	interested	in	how	likely	a
superintelligence	is	to	win	at	a	game	of	chess.	As	for	IQ	scores,	they	are

informative	only	insofar	as	we	have	some	idea	of	how	they	correlate	with
practically	relevant	outcomes.
5
	For	example,	we	have	data	that	show	that	people
with	an	IQ	of	130	are	more	likely	than	those	with	an	IQ	of	90	to	excel	in	school
and	to	do	well	in	a	wide	range	of	cognitively	demanding	jobs.	But	suppose	we
could	somehow	establish	that	a	certain	future	AI	will	have	an	IQ	of	6,455:	then
what?	We	would	have	no	idea	of	what	such	an	AI	could	actually	do.	We	would

5
not	even	know	that	such	an	AI	had	as	much	general	intelligence	as	a	normal
human	adult—perhaps	the	AI	would	instead	have	a	bundle	of	special-purpose
algorithms	enabling	it	to	solve	typical	intelligence	test	questions	with
superhuman	efficiency	but	not	much	else.
Some	recent	efforts	have	been	made	to	develop	measurements	of	cognitive
capacity	that	could	be	applied	to	a	wider	range	of	information-processing
systems,	including	artificial	intelligences.
6
	Work	in	this	direction,	if	it	can

6
overcome	various	technical	difficulties,	may	turn	out	to	be	quite	useful	for	some
scientific	purposes	including	AI	development.	For	purposes	of	the	present
investigation,	however,	its	usefulness	would	be	limited	since	we	would	remain

unenlightened	about	what	a	given	superhuman	performance	score	entails	for
actual	ability	to	achieve	practically	important	outcomes	in	the	world.
It	will	therefore	serve	our	purposes	better	to	list	some	strategically	important
tasks	and	then	to	characterize	hypothetical	cognitive	systems	in	terms	of	whether
they	have	or	lack	whatever	skills	are	needed	to	succeed	at	these	tasks.	See	
Table
8
.	We	will	say	that	a	system	that	sufficiently	excels	at	any	of	the	tasks	in	this
table	has	a	corresponding	
superpower
.
A	full-blown	superintelligence	would	greatly	excel	at	all	of	these	tasks	and
would	thus	have	the	full	panoply	of	all	six	superpowers.	Whether	there	is	a
practically	significant	possibility	of	a	domain-limited	intelligence	that	has	some
of	the	superpowers	but	remains	unable	for	a	significant	period	of	time	to	acquire
all	of	them	is	not	clear.	Creating	a	machine	with	any	one	of	these	superpowers
appears	to	be	an	AI-complete	problem.	Yet	it	is	conceivable	that,	for	example,	a

.
collective	superintelligence	consisting	of	a	sufficiently	large	number	of	human-
like	biological	or	electronic	minds	would	have,	say,	the	economic	productivity
superpower	but	lack	the	strategizing	superpower.	Likewise,	it	is	conceivable	that
a	specialized	engineering	AI	could	be	built	that	has	the	technology	research
superpower	while	completely	lacking	skills	in	other	areas.	This	is	more	plausible
if	there	exists	some	particular	technological	domain	such	that	virtuosity	within

that	domain	would	be	sufficient	for	the	generation	of	an	overwhelmingly
superior	general-purpose	technology.	For	instance,	one	could	imagine	a
specialized	AI	adept	at	simulating	molecular	systems	and	at	inventing
nanomolecular	designs	that	realize	a	wide	range	of	important	capabilities	(such
as	computers	or	weapons	systems	with	futuristic	performance	characteristics)
described	by	the	user	only	at	a	fairly	high	level	of	abstraction.
7
	Such	an	AI	might
also	be	able	to	produce	a	detailed	blueprint	for	how	to	bootstrap	from	existing

7
technology	(such	as	biotechnology	and	protein	engineering)	to	the	constructor
capabilities	needed	for	high-throughput	atomically	precise	manufacturing	that
would	allow	inexpensive	fabrication	of	a	much	wider	range	of	nanomechanical
structures.
8
	However,	it	might	turn	out	to	be	the	case	that	an	engineering	AI
could	not	truly	possess	the	technological	research	superpower	without	also
possessing	advanced	skills	in	areas	outside	of	technology—a	wide	range	of
intellectual	faculties	might	be	needed	to	understand	how	to	interpret	user

8
requests,	how	to	model	a	design’s	behavior	in	real-world	applications,	how	to
deal	with	unanticipated	bugs	and	malfunctions,	how	to	procure	the	materials	and
inputs	needed	for	construction,	and	so	forth.
9

Table	8	
Superpowers:	some	strategically	relevant	tasks	and	corresponding	skill
sets
	
Task
Skill	set
Strategic	relevance
amplification
AI	programming,	cognitive
enhancement	research,	social
epistemology	development,	
etc.
•	System	can	bootstrap	its
Strategizing
Strategic	planning,	forecasting,
prioritizing,	and	analysis	for
optimizing	chances	of	achieving
distant	goal
•	Achieve	distant	goals
•	Overcome	intelligent
opposition
Social
manipulation
Social	and	psychological
modeling,	manipulation,	rhetoric
persuasion
•	Leverage	external	resources	by
recruiting	human	support
•	Enable	a	“boxed”	AI	to
persuade	its	gatekeepers	to	let
it	out
•	Persuade	states	and
organizations	to	adopt	some
course	of	action
Hacking
Finding	and	exploiting	security
flaws	in	computer	systems
•	AI	can	expropriate
computational	resources	over
the	Internet
•	A	boxed	AI	may	exploit
security	holes	to	escape
cybernetic	confinement

•	Steal	financial	resources
•	Hijack	infrastructure,	military
robots,	
etc.
research
Design	and	modeling	of	advanced
technologies	(e.g.	biotechnology,
nanotechnology)	and
development	paths
•	Creation	of	powerful	military
force
•	Creation	of	surveillance	system
•	Automated	space	colonization
Economic
productivity
Various	skills	enabling
economically	productive
intellectual	work
•	Generate	wealth	which	can	be
used	to	buy	influence,
services,	resources	(including
hardware),	
etc.
A	system	that	has	the	intelligence	amplification	superpower	could	use	it	to
bootstrap	itself	to	higher	levels	of	intelligence	and	to	acquire	any	of	the	other
intellectual	superpowers	that	it	does	not	possess	at	the	outset.	But	using	an
intelligence	amplification	superpower	is	not	the	only	way	for	a	system	to
become	a	full-fledged	superintelligence.	A	system	that	has	the	strategizing
superpower,	for	instance,	might	use	it	to	devise	a	plan	that	will	eventually	bring

etc.
an	increase	in	intelligence	(e.g.	by	positioning	the	system	so	as	to	become	the
focus	for	intelligence	amplification	work	performed	by	human	programmers	and
computer	science	researchers).
	
We	thus	find	that	a	project	that	controls	a	superintelligence	has	access	to	a	great
source	of	power.	A	project	that	controls	the	first	superintelligence	in	the	world
would	probably	have	a	decisive	strategic	advantage.	But	the	more	immediate
locus	of	the	power	is	
in	the	system	itself
.	A	machine	superintelligence	might

itself	be	an	extremely	powerful	agent,	one	that	could	successfully	assert	itself

against	the	project	that	brought	it	into	existence	as	well	as	against	the	rest	of	the
world.	This	is	a	point	of	paramount	importance,	and	we	will	examine	it	more
closely	in	the	coming	pages.
Now	let	us	suppose	that	there	is	a	machine	superintelligence	that	wants	to
seize	power	in	a	world	in	which	it	has	as	yet	no	peers.	(Set	aside,	for	the
moment,	the	question	of	whether	and	how	it	would	acquire	such	a	motive—that
is	a	topic	for	the	next	chapter.)	How	could	the	superintelligence	achieve	this	goal
of	world	domination?
We	can	imagine	a	sequence	along	the	following	lines	(see	
Figure	10
).
1
	Pre-criticality	phase
	
Scientists	conduct	research	in	the	field	of	artificial	intelligence	and	other	relevant
disciplines.	This	work	culminates	in	the	creation	of	a	seed	AI.	The	seed	AI	is
able	to	improve	its	own	intelligence.	In	its	early	stages,	the	seed	AI	is	dependent
on	help	from	human	programmers	who	guide	its	development	and	do	most	of	the
heavy	lifting.	As	the	seed	AI	grows	more	capable,	it	becomes	capable	of	doing

Figure	10
).
1
	
more	of	the	work	by	itself.
2
	Recursive	self-improvement	phase
	
At	some	point,	the	seed	AI	becomes	better	at	AI	design	than	the	human
programmers.	Now	when	the	AI	improves	itself,	it	improves	the	thing	that	does
the	
improving.	An	intelligence	explosion	results—a	rapid	cascade	of	recursive
self-improvement	cycles	causing	the	AI’s	capability	to	soar.	(We	can	thus	think
of	this	phase	as	the	takeoff	that	occurs	just	after	the	AI	reaches	the	crossover

2
	
the	
point,	assuming	the	intelligence	gain	during	this	part	of	the	takeoff	is	explosive
and	driven	by	the	application	of	the	AI’s	own	optimization	power.)	The	AI
develops	the	intelligence	amplification	superpower.	This	superpower	enables	the
AI	to	develop	all	the	other	superpowers	detailed	in	
Table	8
.	At	the	end	of	the
recursive	self-improvement	phase,	the	system	is	strongly	superintelligent.

Figure	10
	Phases	in	an	AI	takeover	scenario.
	
3
	Covert	preparation	phase
	
Using	its	strategizing	superpower,	the	AI	develops	a	robust	plan	for	achieving	its
long-term	goals.	(In	particular,	the	AI	does	not	adopt	a	plan	so	stupid	that	even
we	present-day	humans	can	foresee	how	it	would	inevitably	fail.	This	criterion
rules	out	many	science	fiction	scenarios	that	end	in	human	triumph.
10
)	The	plan
might	involve	a	period	of	covert	action	during	which	the	AI	conceals	its
intellectual	development	from	the	human	programmers	in	order	to	avoid	setting
off	alarms.	The	AI	might	also	mask	its	true	proclivities,	pretending	to	be
cooperative	and	docile.
If	the	AI	has	(perhaps	for	safety	reasons)	been	confined	to	an	isolated
computer,	it	may	use	its	social	manipulation	superpower	to	persuade	the
gatekeepers	to	let	it	gain	access	to	an	Internet	port.	Alternatively,	the	AI	might
use	its	hacking	superpower	to	escape	its	confinement.	Spreading	over	the

Internet	may	enable	the	AI	to	expand	its	hardware	capacity	and	knowledge	base,
further	increasing	its	intellectual	superiority.	An	AI	might	also	engage	in	licit	or
illicit	economic	activity	to	obtain	funds	with	which	to	buy	computer	power,	data,

and	other	resources.
At	this	point,	there	are	several	ways	for	the	AI	to	achieve	results	outside	the
virtual	realm.	It	could	use	its	hacking	superpower	to	take	direct	control	of
robotic	manipulators	and	automated	laboratories.	Or	it	could	use	its	social
manipulation	superpower	to	persuade	human	collaborators	to	serve	as	its	legs
and	hands.	Or	it	could	acquire	financial	assets	from	online	transactions	and	use
them	to	purchase	services	and	influence.
4
	Overt	implementation	phase
	
The	final	phase	begins	when	the	AI	has	gained	sufficient	strength	to	obviate	the
need	for	secrecy.	The	AI	can	now	directly	implement	its	objectives	on	a	full
scale.
The	overt	implementation	phase	might	start	with	a	“strike”	in	which	the	AI
eliminates	the	human	species	and	any	automatic	systems	humans	have	created
that	could	offer	intelligent	opposition	to	the	execution	of	the	AI’s	plans.	This
could	be	achieved	through	the	activation	of	some	advanced	weapons	system	that

4
	
scale.
the	AI	has	perfected	using	its	technology	research	superpower	and	covertly
deployed	in	the	covert	preparation	phase.	If	the	weapon	uses	self-replicating
biotechnology	or	nanotechnology,	the	initial	stockpile	needed	for	global
coverage	could	be	microscopic:	a	single	replicating	entity	would	be	enough	to
start	the	process.	In	order	to	ensure	a	sudden	and	uniform	effect,	the	initial	stock
of	the	replicator	might	have	been	deployed	or	allowed	to	diffuse	worldwide	at	an

extremely	low,	undetectable	concentration.	At	a	pre-set	time,	nanofactories
producing	nerve	gas	or	target-seeking	mosquito-like	robots	might	then	burgeon
forth	simultaneously	from	every	square	meter	of	the	globe	(although	more
effective	ways	of	killing	could	probably	be	devised	by	a	machine	with	the
technology	research	superpower).
11
	One	might	also	entertain	scenarios	in	which
a	superintelligence	attains	power	by	hijacking	political	processes,	subtly
manipulating	financial	markets,	biasing	information	flows,	or	hacking	into

11
human-made	weapon	systems.	Such	scenarios	would	obviate	the	need	for	the
superintelligence	to	invent	new	weapons	technology,	although	they	may	be
unnecessarily	slow	compared	with	scenarios	in	which	the	machine	intelligence
builds	its	own	infrastructure	with	manipulators	that	operate	at	molecular	or
atomic	speed	rather	than	the	slow	speed	of	human	minds	and	bodies.

Alternatively,	if	the	AI	is	sure	of	its	invincibility	to	human	interference,	our
species	may	not	be	targeted	directly.	Our	demise	may	instead	result	from	the
habitat	destruction	that	ensues	when	the	AI	begins	massive	global	construction
projects	using	nanotech	factories	and	assemblers—construction	projects	which
quickly,	perhaps	within	days	or	weeks,	tile	all	of	the	Earth’s	surface	with	solar
panels,	nuclear	reactors,	supercomputing	facilities	with	protruding	cooling
towers,	space	rocket	launchers,	or	other	installations	whereby	the	AI	intends	to
maximize	the	long-term	cumulative	realization	of	its	values.	Human	brains,	if
they	contain	information	relevant	to	the	AI’s	goals,	could	be	disassembled	and
scanned,	and	the	extracted	data	transferred	to	some	more	efficient	and	secure
storage	format.
Box	6
	describes	one	particular	scenario.	One	should	avoid	fixating	too	much
on	the	concrete	details,	since	they	are	in	any	case	unknowable	and	intended	for

Box	6
illustration	only.	A	superintelligence	might—and	probably	would—be	able	to
conceive	of	a	better	plan	for	achieving	its	goals	than	any	that	a	human	can	come
up	with.	It	is	therefore	necessary	to	think	about	these	matters	more	abstractly.
Without	knowing	anything	about	the	detailed	means	that	a	superintelligence
would	adopt,	we	can	conclude	that	a	superintelligence—at	least	in	the	absence	of
intellectual	peers	and	in	the	absence	of	effective	safety	measures	arranged	by

humans	in	advance—would	likely	produce	an	outcome	that	would	involve
reconfiguring	terrestrial	resources	into	whatever	structures	maximize	the
realization	of	its	goals.	Any	concrete	scenario	we	develop	can	at	best	establish	a
lower	bound	on	how	quickly	and	efficiently	the	superintelligence	could	achieve
such	an	outcome.	It	remains	possible	that	the	superintelligence	would	find	a
shorter	path	to	its	preferred	destination.
Box	6	The	mail-ordered	DNA	scenario
	
Yudkowsky	describes	the	following	possible	scenario	for	an	AI	takeover.
12

12
	Crack	the	protein	folding	problem	to	the	extent	of	being	able	to	generate	DNA
strings	whose	folded	peptide	sequences	fill	specific	functional	roles	in	a	complex
chemical	interaction.
2
	Email	sets	of	DNA	strings	to	one	or	more	online	laboratories	that	offer	DNA

synthesis,	peptide	sequencing,	and	FedEx	delivery.	(Many	labs	currently
offer	this	service,	and	some	boast	of	72-hour	turnaround	times.)
3
	Find	at	least	one	human	connected	to	the	Internet	who	can	be	paid,
blackmailed,	or	fooled	by	the	right	background	story,	into	receiving
FedExed	vials	and	mixing	them	in	a	specified	environment.
4
	The	synthesized	proteins	form	a	very	primitive	“wet”	nanosystem,	which,
ribosome-like,	is	capable	of	accepting	external	instructions;	perhaps
patterned	acoustic	vibrations	delivered	by	a	speaker	attached	to	the	beaker.
5
	Use	the	extremely	primitive	nanosystem	to	build	more	sophisticated
systems,	which	construct	still	more	sophisticated	systems,	bootstrapping	to
molecular	nanotechnology—or	beyond.
In	this	scenario,	the	superintelligence	uses	its	technology	research	superpower	to
solve	the	protein	folding	problem	in	step	1,	enabling	it	to	design	a	set	of
molecular	building	blocks	for	a	rudimentary	nanotechnology	assembler	or

5
fabrication	device,	which	can	self-assemble	in	aqueous	solution	(step	4).	The
same	technology	research	superpower	is	used	again	in	step	5	to	bootstrap	from
primitive	to	advanced	machine-phase	nanotechnology.	The	other	steps	require	no
more	than	human	intelligence.	The	skills	required	for	step	3—identifying	a
gullible	Internet	user	and	persuading	him	or	her	to	follow	some	simple
instructions—are	on	display	every	day	all	over	the	world.	The	entire	scenario

was	invented	by	a	human	mind,	so	the	strategizing	ability	needed	to	formulate
this	plan	is	also	merely	human	level.
In	this	particular	scenario,	the	AI	starts	out	having	access	to	the	Internet.	If
this	is	not	the	case,	then	additional	steps	would	have	to	be	added	to	the	plan.	The
AI	might,	for	example,	use	its	social	manipulation	superpower	to	convince	the
people	interacting	with	it	that	it	ought	to	be	set	free.	Alternatively,	the	AI	might
be	able	to	use	its	hacking	superpower	to	escape	confinement.	If	the	AI	does	not

possess	these	capabilities,	it	might	first	need	to	use	its	intelligence	amplification
superpower	to	develop	the	requisite	proficiency	in	social	manipulation	or
hacking.
A	superintelligent	AI	will	presumably	be	born	into	a	highly	networked	world.
One	could	point	to	various	developments	that	could	potentially	help	a	future	AI
to	control	the	world—cloud	computing,	proliferation	of	web-connected	sensors,

military	and	civilian	drones,	automation	in	research	labs	and	manufacturing
plants,	increased	reliance	on	electronic	payment	systems	and	digitized	financial
assets,	and	increased	use	of	automated	information-filtering	and	decision	support
systems.	Assets	like	these	could	potentially	be	acquired	by	an	AI	at	digital
speeds,	expediting	its	rise	to	power	(though	advances	in	cybersecurity	might
make	it	harder).	In	the	final	analysis,	however,	it	is	doubtful	whether	any	of
these	trends	makes	a	difference.	A	superintelligence’s	power	resides	in	its	brain,
not	its	hands.	Although	the	AI,	in	order	to	remake	the	external	world,	will	at
some	point	need	access	to	an	actuator,	a	single	pair	of	helping	human	hands,
those	of	a	pliable	accomplice,	would	probably	suffice	to	complete	the	covert
preparation	phase,	as	suggested	by	the	above	scenario.	This	would	enable	the	AI
to	reach	the	overt	implementation	phase	in	which	it	constructs	its	own
infrastructure	of	physical	manipulators.
	
An	agent’s	ability	to	shape	humanity’s	future	depends	not	only	on	the	absolute
magnitude	of	the	agent’s	own	faculties	and	resources—how	smart	and	energetic
it	is,	how	much	capital	it	has,	and	so	forth—but	also	on	the	relative	magnitude	of
its	capabilities	compared	with	those	of	other	agents	with	conflicting	goals.
In	a	situation	where	there	are	no	competing	agents,	the	absolute	capability
level	of	a	superintelligence,	so	long	as	it	exceeds	a	certain	minimal	threshold,
does	not	matter	much,	because	a	system	starting	out	with	some	sufficient	set	of

capabilities	could	plot	a	course	of	development	that	will	let	it	acquire	any
capabilities	it	initially	lacks.	We	alluded	to	this	point	earlier	when	we	said	that
speed,	quality,	and	collective	superintelligence	all	have	the	same	indirect	reach.
We	alluded	to	it	again	when	we	said	that	various	subsets	of	superpowers,	such	as
the	intelligence	amplification	superpower	or	the	strategizing	and	the	social
manipulation	superpowers,	could	be	used	to	obtain	the	full	complement.
Consider	a	superintelligent	agent	with	actuators	connected	to	a	nanotech

assembler.	Such	an	agent	is	already	powerful	enough	to	overcome	any	natural
obstacles	to	its	indefinite	survival.	Faced	with	no	intelligent	opposition,	such	an

agent	could	plot	a	safe	course	of	development	that	would	lead	to	its	acquiring	the
complete	inventory	of	technologies	that	would	be	useful	to	the	attainment	of	its
goals.	For	example,	it	could	develop	the	technology	to	build	and	launch	von
Neumann	probes,	machines	capable	of	interstellar	travel	that	can	use	resources
such	as	asteroids,	planets,	and	stars	to	make	copies	of	themselves.
13
	By
launching	one	von	Neumann	probe,	the	agent	could	thus	initiate	an	open-ended
process	of	space	colonization.	The	replicating	probe’s	descendants,	travelling	at
some	significant	fraction	of	the	speed	of	light,	would	end	up	colonizing	a
substantial	portion	of	the	Hubble	volume,	the	part	of	the	expanding	universe	that
is	theoretically	accessible	from	where	we	are	now.	All	this	matter	and	free
energy	could	then	be	organized	into	whatever	value	structures	maximize	the
originating	agent’s	utility	function	integrated	over	cosmic	time—a	duration
encompassing	at	least	trillions	of	years	before	the	aging	universe	becomes

inhospitable	to	information	processing	(see	
Box	7
).
The	superintelligent	agent	could	design	the	von	Neumann	probes	to	be
evolution-proof.	This	could	be	accomplished	by	careful	quality	control	during
the	replication	step.	For	example,	the	control	software	for	a	daughter	probe	could
be	proofread	multiple	times	before	execution,	and	the	software	itself	could	use
encryption	and	error-correcting	code	to	make	it	arbitrarily	unlikely	that	any
random	mutation	would	be	passed	on	to	its	descendants.
14
	The	proliferating

Box	7
).
14
population	of	von	Neumann	probes	would	then	securely	preserve	and	transmit
the	originating	agent’s	values	as	they	go	about	settling	the	universe.	When	the
colonization	phase	is	completed,	the	original	values	would	determine	the	use
made	of	all	the	accumulated	resources,	even	though	the	great	distances	involved
and	the	accelerating	speed	of	cosmic	expansion	would	make	it	impossible	for
remote	parts	of	the	infrastructure	to	communicate	with	one	another.	The	upshot
is	that	a	large	part	of	our	future	light	cone	would	be	formatted	in	accordance

with	the	preferences	of	the	originating	agent.
This,	then,	is	the	measure	of	the	indirect	reach	of	any	system	that	faces	no
significant	intelligent	opposition	and	that	starts	out	with	a	set	of	capabilities
exceeding	a	certain	threshold.	We	can	term	the	threshold	the	“wise-singleton
sustainability	threshold”	(
Figure	11
):
The	wise-singleton	sustainability	threshold
A	capability	set	exceeds	the	wise-singleton	threshold	if	and	only	if	a	patient
and	existential	risk-savvy	system	with	that	capability	set	would,	if	it	faced

no	intelligent	opposition	or	competition,	be	able	to	colonize	and	re-engineer
a	large	part	of	the	accessible	universe.
	
By	“singleton”	we	mean	a	sufficiently	internally	coordinated	political	structure
with	no	external	opponents,	and	by	“wise”	we	mean	sufficiently	patient	and
savvy	about	existential	risks	to	ensure	a	substantial	amount	of	well-directed
concern	for	the	very	long-term	consequences	of	the	system’s	actions.
Figure	11
	Schematic	illustration	of	some	possible	trajectories	for	a	hypothetical
wise	singleton.	With	a	capability	below	the	short-term	viability	threshold—for
example,	if	population	size	is	too	small—a	species	tends	to	go	extinct	in	short
order	(and	remain	extinct).	At	marginally	higher	levels	of	capability,	various
trajectories	are	possible:	a	singleton	might	be	unlucky	and	go	extinct	or	it	might
be	lucky	and	attain	a	capability	(e.g.	population	size,	geographical	dispersion,
technological	capacity)	that	crosses	the	wise-singleton	sustainability	threshold.

Once	above	this	threshold,	a	singleton	will	almost	certainly	continue	to	gain	in
capability	until	some	extremely	high	capability	level	is	attained.	In	this	picture,
there	are	two	attractors:	extinction	and	astronomical	capability.	Note	that,	for	a
wise	singleton,	the	distance	between	the	short-term	viability	threshold	and	the
sustainability	threshold	may	be	rather	small.
15

Box	7	How	big	is	the	cosmic	endowment?
	
Consider	a	technologically	mature	civilization	capable	of	building	sophisticated
von	Neumann	probes	of	the	kind	discussed	in	the	text.	If	these	can	travel	at	50%
of	the	speed	of	light,	they	can	reach	some	6×10
18
	stars	before	the	cosmic
expansion	puts	further	acquisitions	forever	out	of	reach.	At	99%	of	
c
,	they	could
reach	some	2×10
20
	stars.
16
	These	travel	speeds	are	energetically	attainable	using
a	small	fraction	of	the	resources	available	in	the	solar	system.
17
	The
impossibility	of	faster-than-light	travel,	combined	with	the	positive	cosmological
constant	(which	causes	the	rate	of	cosmic	expansion	to	accelerate),	implies	that
these	are	close	to	upper	bounds	on	how	much	stuff	our	descendants	acquire.
18
If	we	assume	that	10%	of	stars	have	a	planet	that	is—or	could	by	means	of
terraforming	be	rendered—suitable	for	habitation	by	human-like	creatures,	and
that	it	could	then	be	home	to	a	population	of	a	billion	individuals	for	a	billion

17
	The
18
years	(with	a	human	life	lasting	a	century),	this	suggests	that	around	10
35
	human
lives	could	be	created	in	the	future	by	an	Earth-originating	intelligent
19
There	are,	however,	reasons	to	think	this	greatly	underestimates	the	true
number.	By	disassembling	non-habitable	planets	and	collecting	matter	from	the
interstellar	medium,	and	using	this	material	to	construct	Earth-like	planets,	or	by
increasing	population	densities,	the	number	could	be	increased	by	at	least	a

35
	human
19
couple	of	orders	of	magnitude.	And	if	instead	of	using	the	surfaces	of	solid
planets,	the	future	civilization	built	O’Neill	cylinders,	then	many	further	orders
of	magnitude	could	be	added,	yielding	a	total	of	perhaps	10
43
	human	lives.
(“O’Neill	cylinders”	refers	to	a	space	settlement	design	proposed	in	the	mid-
seventies	by	the	American	physicist	Gerard	K.	O’Neill,	in	which	inhabitants
dwell	on	the	inside	of	hollow	cylinders	whose	rotation	produces	a	gravity-
substituting	centrifugal	force.
20
)

43
20
)
Many	more	orders	of	magnitudes	of	human-like	beings	could	exist	if	we
countenance	digital	implementations	of	minds—as	we	should.	To	calculate	how
many	such	digital	minds	could	be	created,	we	must	estimate	the	computational
power	attainable	by	a	technologically	mature	civilization.	This	is	hard	to	do	with
any	precision,	but	we	can	get	a	lower	bound	from	technological	designs	that

have	been	outlined	in	the	literature.	One	such	design	builds	on	the	idea	of	a
Dyson	sphere,	a	hypothetical	system	(described	by	the	physicist	Freeman	Dyson
in	1960)	that	would	capture	most	of	the	energy	output	of	a	star	by	surrounding	it
with	a	system	of	solar-collecting	structures.
21
	For	a	star	like	our	Sun,	this	would
generate	10
26
	watts.	How	much	computational	power	this	would	translate	into
depends	on	the	efficiency	of	the	computational	circuitry	and	the	nature	of	the
computations	to	be	performed.	If	we	require	irreversible	computations,	and
assume	a	nanomechanical	implementation	of	the	“computronium”	(which	would
allow	us	to	push	close	to	the	Landauer	limit	of	energy	efficiency),	a	computer
system	driven	by	a	Dyson	sphere	could	generate	some	10
47
	operations	per
second.
22
Combining	these	estimates	with	our	earlier	estimate	of	the	number	of	stars
that	could	be	colonized,	we	get	a	number	of	about	10
67
	ops/s	once	the	accessible
parts	of	the	universe	have	been	colonized	(assuming	nanomechanical

47
second.
22
67
computronium).
23
	A	typical	star	maintains	its	luminosity	for	some	10
18
	s.
Consequently,	the	number	of	computational	operations	that	could	be	performed
using	our	cosmic	endowment	is	at	least	10
85
.	The	true	number	is	probably	much
larger.	We	might	get	additional	orders	of	magnitude,	for	example,	if	we	make
extensive	use	of	reversible	computation,	if	we	perform	the	computations	at
colder	temperatures	(by	waiting	until	the	universe	has	cooled	further),	or	if	we
make	use	of	additional	sources	of	energy	(such	as	dark	matter).
24

18
	s.
85
24
It	might	not	be	immediately	obvious	to	some	readers	why	the	ability	to
perform	10
85
	computational	operations	is	a	big	deal.	So	it	is	useful	to	put	it	in
context.	We	may,	for	example,	compare	this	number	with	our	earlier	estimate
(
Box	3
,	in	
Chapter	2
)	that	it	may	take	about	10
31
–10
44
	ops	to	simulate	all
neuronal	operations	that	have	occurred	in	the	history	of	life	on	Earth.
Alternatively,	let	us	suppose	that	the	computers	are	used	to	run	human	whole
brain	emulations	that	live	rich	and	happy	lives	while	interacting	with	one	another

perform	10
85
(
Box	3
,	in	
Chapter	2
31
–10
44
in	virtual	environments.	
A	typical	estimate	of	the	computational	requirements	for
running	one	emulation	is	10
18
	ops/s.	To	run	an	emulation	for	100	subjective
years	would	then	require	some	10
27
	ops.	This	would	mean	that	at	least	10
58
human	lives	could	be	created	in	emulation	even	with	quite	conservative
assumptions	about	the	efficiency	of	computronium.
In	other	words,	assuming	that	the	observable	universe	is	void	of
extraterrestrial	civilizations,	then	what	hangs	in	the	balance	is	at	least

10,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000
human	lives	(though	the	true	number	is	probably	larger).	If	we	represent	all	the
happiness	experienced	during	one	entire	such	life	with	a	single	teardrop	of	joy,
then	the	happiness	of	these	souls	could	fill	and	refill	the	Earth’s	oceans	every
second,	and	keep	doing	so	for	a	hundred	billion	billion	millennia.	It	is	really
important	that	we	make	sure	these	truly	are	tears	of	joy.
	
This	wise-singleton	sustainability	threshold	appears	to	be	quite	low.	Limited
forms	of	superintelligence,	as	we	have	seen,	exceed	this	threshold	provided	they
have	access	to	some	actuator	sufficient	to	initiate	a	technology	bootstrap	process.
In	an	environment	that	includes	contemporary	human	civilization,	the	minimally
necessary	actuator	could	be	very	simple—an	ordinary	screen	or	indeed	any
means	of	transmitting	a	non-trivial	amount	of	information	to	a	human
accomplice	would	suffice.
But	the	wise-singleton	sustainability	threshold	is	lower	still:	neither

superintelligence	nor	any	other	futuristic	technology	is	needed	to	surmount	it.	A
patient	and	existential	risk-savvy	singleton	with	no	more	technological	and
intellectual	capabilities	than	those	possessed	by	contemporary	humanity	should
be	readily	able	to	plot	a	course	that	leads	reliably	to	the	eventual	realization	of
humanity’s	astronomical	capability	potential.	This	could	be	achieved	by
investing	in	relatively	safe	methods	of	increasing	wisdom	and	existential	risk-

savvy	while	postponing	the	development	of	potentially	dangerous	new
technologies.	Given	that	non-anthropogenic	existential	risks	(ones	not	arising
from	human	activities)	are	small	over	the	relevant	timescales—and	could	be
further	reduced	with	various	safe	interventions—such	a	singleton	could	afford	to
go	slow.
25
	It	could	look	carefully	before	each	step,	delaying	development	of
capabilities	such	as	synthetic	biology,	human	enhancement	medicine,	molecular
nanotechnology,	and	machine	intelligence	until	it	had	first	perfected	seemingly

go	slow.
25
less	hazardous	capabilities	such	as	its	education	system,	its	information
technology,	and	its	collective	decision-making	processes,	and	until	it	had	used
these	capabilities	to	conduct	a	very	thorough	review	of	its	options.	So	this	is	all
within	the	indirect	reach	of	a	technological	civilization	like	that	of	contemporary
humanity.	We	are	separated	from	this	scenario	“merely”	by	the	fact	that
humanity	is	currently	neither	a	singleton	nor	(in	the	relevant	sense)	wise.
One	could	even	argue	that	
	passed	the	wise-singleton

sustainability	threshold	soon	after	the	species	first	evolved.	Twenty	thousand
years	ago,	say,	with	equipment	no	fancier	than	stone	axes,	bone	tools,	atlatls,	and
fire,	the	human	species	was	perhaps	already	in	a	position	from	which	it	had	an
excellent	chance	of	surviving	to	the	present	era.
26
	Admittedly,	there	is	something
queer	about	crediting	our	Paleolithic	ancestors	with	having	developed
technology	that	“exceeded	the	wise-singleton	sustainability	threshold”—given
that	there	was	no	realistic	possibility	of	a	singleton	forming	at	such	a	primitive
time,	let	alone	a	singleton	savvy	about	existential	risks	and	patient.
27
Nevertheless,	the	point	stands	that	the	threshold	corresponds	to	a	very	modest
level	of	technology—a	level	that	humanity	long	ago	surpassed.
28
It	is	clear	that	if	we	are	to	assess	the	effective	powers	of	a	superintelligence—
its	ability	to	achieve	a	range	of	preferred	outcomes	in	the	world—we	must
consider	not	only	its	own	internal	capacities	but	also	the	capabilities	of

27
28
competing	agents.	The	notion	of	a	superpower	invoked	such	a	relativized
standard	implicitly.	We	said	that	“a	system	that	sufficiently	excels”	at	any	of	the
tasks	in	
Table	8
	has	a	corresponding	superpower.	Exceling	at	a	task	like
strategizing,	social	manipulation,	or	hacking	involves	having	a	skill	at	that	task
that	is	high	in	comparison	to	the	skills	of	other	agents	(such	as	strategic	rivals,
influence	targets,	or	computer	security	experts).	The	other	superpowers,	too,
should	be	understood	in	this	relative	sense:	intelligence	amplification,

tasks	in	
Table	8
technology	research,	and	economic	productivity	are	possessed	by	an	agent	as
superpowers	only	if	the	agent’s	capabilities	in	these	areas	substantially	exceed
the	combined	capabilities	of	the	rest	of	the	global	civilization.	It	follows	from
this	definition	that	at	most	one	agent	can	possess	a	particular	superpower	at	any
given	time.
29
This	is	the	main	reason	why	the	question	of	takeoff	speed	is	important—not
because	it	matters	exactly	when	a	particular	outcome	happens,	but	because	the

29
speed	of	the	takeoff	may	make	a	big	difference	to	what	the	outcome	will	be.
With	a	fast	or	medium	takeoff,	it	is	likely	that	one	project	will	get	a	decisive
strategic	advantage.	We	have	now	suggested	that	a	superintelligence	with	a
decisive	strategic	advantage	would	have	immense	powers,	enough	that	it	could
form	a	stable	singleton—a	singleton	that	could	determine	the	disposition	of
humanity’s	cosmic	endowment.
But	“could”	is	different	from	“would.”	Somebody	might	have	great	powers

yet	choose	not	to	use	them.	Is	it	possible	to	say	anything	about	what	a
superintelligence	with	a	decisive	strategic	advantage	would	want?	It	is	to	this

CHAPTER	7
The	superintelligent	will
	
We	have	seen	that	a	superintelligence	could	have	a	great	ability	to	shape	the
future	according	to	its	goals.	But	what	will	its	goals	be?	What	is	the	relation
between	intelligence	and	motivation	in	an	artificial	agent?	Here	we	develop
two	theses.	The	orthogonality	thesis	holds	(with	some	caveats)	that
intelligence	and	final	goals	are	independent	variables:	any	level	of
intelligence	could	be	combined	with	any	final	goal.	The	instrumental
convergence	thesis	holds	that	superintelligent	agents	having	any	of	a	wide
range	of	final	goals	will	nevertheless	pursue	similar	intermediary	goals
because	they	have	common	instrumental	reasons	to	do	so.	Taken	together,
these	theses	help	us	think	about	what	a	superintelligent	agent	would	do.
	
We	have	already	cautioned	against	anthropomorphizing	the	
capabilities
	of	a
superintelligent	AI.	This	warning	should	be	extended	to	pertain	to	its	
motivations
as	well.

	of	a
as	well.
It	is	a	useful	propaedeutic	to	this	part	of	our	inquiry	to	first	reflect	for	a
moment	on	the	vastness	of	the	space	of	possible	minds.	In	this	abstract	space,
human	minds	form	a	tiny	cluster.	Consider	two	persons	who	seem	extremely
unlike,	perhaps	Hannah	Arendt	and	Benny	Hill.	The	personality	differences
between	these	two	individuals	may	seem	almost	maximally	large.	But	this	is
because	our	intuitions	are	calibrated	on	our	experience,	which	samples	from	the
existing	human	distribution	(and	to	some	extent	from	fictional	personalities

constructed	by	the	human	imagination	for	the	enjoyment	of	the	human
imagination).	If	we	zoom	out	and	consider	the	space	of	all	possible	minds,
however,	we	must	conceive	of	these	two	personalities	as	virtual	clones.	Certainly
in	terms	of	neural	architecture,	Ms.	Arendt	and	Mr.	Hill	are	nearly	identical.
Imagine	their	brains	lying	side	by	side	in	quiet	repose.	You	would	readily

recognize	them	as	two	of	a	kind.	You	might	even	be	unable	to	tell	which	brain
belonged	to	
whom.	If	you	looked	more	closely,	studying	the	morphology	of	the
two	brains	under	a	microscope,	this	impression	of	fundamental	similarity	would
only	be	strengthened:	you	would	see	the	same	lamellar	organization	of	the
cortex,	with	the	same	brain	areas,	made	up	of	the	same	types	of	neuron,	soaking
in	the	same	bath	of	neurotransmitters.
1
Despite	the	fact	that	human	psychology	corresponds	to	a	tiny	spot	in	the	space
of	possible	minds,	there	is	a	common	tendency	to	project	human	attributes	onto	a
wide	range	of	alien	or	artificial	cognitive	systems.	Yudkowsky	illustrates	this
point	nicely:
Back	in	the	era	of	pulp	science	fiction,	magazine	covers	occasionally
depicted	a	sentient	monstrous	alien—colloquially	known	as	a	bug-eyed
monster	(BEM)—carrying	off	an	attractive	human	female	in	a	torn	dress.	It
would	seem	the	artist	believed	that	a	non-humanoid	alien,	with	a	wholly

different	evolutionary	history,	would	sexually	desire	human	females….
Probably	the	artist	did	not	ask	whether	a	giant	bug	
perceives
	human	females
as	attractive.	Rather,	a	human	female	in	a	torn	dress	
is	sexy
—inherently	so,
as	an	intrinsic	property.	They	who	made	this	mistake	did	not	think	about	the
insectoid’s	mind:	they	focused	on	the	woman’s	torn	dress.	If	the	dress	were
not	torn,	the	woman	would	be	less	sexy;	the	BEM	does	not	enter	into	it.
2
	
An	artificial	intelligence	can	be	far	less	human-like	in	its	motivations	than	a

perceives
is	sexy
2
	
green	scaly	space	alien.	The	extraterrestrial	(let	us	assume)	is	a	biological
creature	that	has	arisen	through	an	evolutionary	process	and	can	therefore	be
expected	to	have	the	kinds	of	motivation	typical	of	evolved	creatures.	It	would
not	be	hugely	surprising,	for	example,	to	find	that	some	random	intelligent	alien
would	have	motives	related	to	one	or	more	items	like	food,	air,	temperature,
energy	expenditure,	occurrence	or	threat	of	bodily	injury,	disease,	predation,	sex,
or	progeny.	A	member	of	an	intelligent	social	species	might	also	have

motivations	related	to	cooperation	and	competition:	like	us,	it	might	show	in-
group	loyalty,	resentment	of	free	riders,	perhaps	even	a	vain	concern	with
reputation	and	appearance.

Figure	12
	Results	of	anthropomorphizing	alien	motivation.	Least	likely
hypothesis:	space	aliens	prefer	blondes.	More	likely	hypothesis:	the	illustrators
succumbed	to	the	“mind	projection	fallacy.”	Most	likely	hypothesis:	the
publisher	wanted	a	cover	that	would	entice	the	target	demographic.
	
An	AI,	by	contrast,	need	not	care	intrinsically	about	any	of	those	things.	There
is	nothing	paradoxical	about	an	AI	whose	sole	final	goal	is	to	count	the	grains	of
sand	on	Boracay,	or	to	calculate	the	decimal	expansion	of	pi,	or	to	maximize	the
total	number	of	paperclips	that	will	exist	in	its	future	light	cone.	In	fact,	it	would
be	
easier
	to	create	an	AI	with	simple	goals	like	these	than	to	build	one	that	had	a
human-like	set	of	values	and	dispositions.	Compare	how	easy	it	is	to	write	a
program	that	measures	how	many	digits	of	pi	have	been	calculated	and	stored	in
memory	with	how	difficult	it	would	be	to	create	a	program	that	reliably
measures	the	degree	of	realization	of	some	more	meaningful	goal—human

be	
easier
flourishing,	say,	or	global	justice.	Unfortunately,	because	a	meaningless
reductionistic	goal	is	easier	for	humans	to	code	and	easier	for	an	AI	to	learn,	it	is
just	the	kind	of	goal	that	a	programmer	would	choose	to	install	in	his	seed	AI	if
his	focus	is	on	taking	the	quickest	path	to	“getting	the	AI	to	work”	(without
caring	much	about	what	exactly	the	AI	will	
do
,	aside	from	displaying
impressively	intelligent	behavior).	We	will	revisit	this	concern	shortly.
Intelligent	search	for	instrumentally	optimal	plans	and	policies	can	be

do
performed	in	the	service	of	any	goal.	Intelligence	and	motivation	are	in	a	sense
orthogonal:	we	can	think	of	them	as	two	axes	spanning	a	graph	in	which	each
point	represents	a	logically	possible	artificial	agent.	Some	qualifications	could
be	added	to	this	picture.	For	instance,	it	might	be	impossible	for	a	very
unintelligent	system	to	have	very	complex	motivations.	In	order	for	it	to	be
correct	to	say	that	an	certain	agent	“has”	a	set	of	motivations,	those	motivations

may	need	to	be	functionally	integrated	with	the	agent’s	decision	processes,
something	that	places	demands	on	memory,	processing	power,	and	perhaps
intelligence.	For	minds	that	can	modify	themselves,	there	may	also	be	dynamical
constraints—an	intelligent	self-modifying	mind	with	an	urgent	desire	to	be
stupid	might	not	remain	intelligent	for	long.	But	these	qualifications	must	not	be
allowed	to	obscure	the	basic	point	about	the	independence	of	intelligence	and
motivation,	which	we	can	express	as	follows:
The	orthogonality	thesis
Intelligence	and	final	goals	are	orthogonal:	more	or	less	any	level	of
intelligence	could	in	principle	be	combined	with	more	or	less	any	final
goal.
	
If	the	orthogonality	thesis	seems	problematic,	this	might	be	because	of	the
superficial	resemblance	it	bears	to	some	traditional	philosophical	positions
which	have	been	subject	to	long	debate.	Once	it	is	understood	to	have	a	different
and	narrower	scope,	its	credibility	should	rise.	(For	example,	the	orthogonality

goal.
	
thesis	does	not	presuppose	the	Humean	theory	of	motivation.
3
	Nor	does	it
presuppose	that	basic	preferences	cannot	be	irrational.
4
)
Note	that	the	orthogonality	thesis	speaks	not	of	
rationality
	or	
reason
,	but	of
.	By	“intelligence”	we	here	mean	something	like	skill	at	prediction,
planning,	and	means–ends	reasoning	in	general.
5
	This	sense	of	instrumental
cognitive	efficaciousness	is	most	relevant	when	we	are	seeking	to	understand

3
	Nor	does	it
4
)
	or	
reason
,	but	of
5
what	the	causal	impact	of	a	machine	superintelligence	might	be.	Even	if	there	is
some	
(normatively	thick)	sense	of	the	word	“rational”	such	that	a	paperclip-
maximizing	superintelligent	agent	would	necessarily	fail	to	qualify	as	fully
rational	in	that	sense,	this	would	in	no	way	preclude	such	an	agent	from	having
awesome	faculties	of	instrumental	reasoning,	faculties	which	could	let	it	have	a
large	impact	on	the	world.
6
According	to	the	orthogonality	thesis,	artificial	agents	can	have	utterly	non-

some	
6
anthropomorphic	goals.	This,	however,	does	not	imply	that	it	is	impossible	to
make	predictions	about	the	behavior	of	particular	artificial	agents—not	even
hypothetical	superintelligent	agents	whose	cognitive	complexity	and
performance	characteristics	might	render	them	in	some	respects	opaque	to

human	analysis.	There	are	at	least	three	directions	from	which	we	can	approach
the	problem	of	predicting	superintelligent	motivation:
Predictability	through	design
.	If	we	can	suppose	that	the	designers	of	a
superintelligent	agent	can	successfully	engineer	the	goal	system	of	the	agent	so
that	it	stably	pursues	a	particular	goal	set	by	the	programmers,	then	one
prediction	we	can	make	is	that	the	agent	will	pursue	that	goal.	The	more
intelligent	the	agent	is,	the	greater	the	cognitive	resourcefulness	it	will	have	to
pursue	that	goal.	So	even	before	an	agent	has	been	created	we	might	be	able	to
predict	something	about	its	behavior,	if	we	know	something	about	who	will
build	it	and	what	goals	they	will	want	it	to	have.
Predictability	through	inheritance
.	If	a	digital	intelligence	is	created	directly	from
a	human	template	(as	would	be	the	case	in	a	high-fidelity	whole	brain
emulation),	then	the	digital	intelligence	might	inherit	the	motivations	of	the
human	template.
7

7
	The	agent	might	retain	some	of	these	motivations	even	if	its
cognitive	capacities	are	subsequently	enhanced	to	make	it	superintelligent.	This
kind	of	inference	requires	caution.	The	agent’s	goals	and	values	could	easily
become	corrupted	in	the	uploading	process	or	during	its	subsequent	operation
and	enhancement,	depending	on	how	the	procedure	is	implemented.
Predictability	through	convergent	instrumental	reasons
.	Even	without	detailed
knowledge	of	an	agent’s	final	goals,	we	may	be	able	to	infer	something	about	its

more	immediate	objectives	by	considering	the	
instrumental
	reasons	that	would
arise	for	any	of	a	wide	range	of	possible	final	goals	in	a	wide	range	of	situations.
This	way	of	predicting	becomes	more	useful	the	greater	the	intelligence	of	the
agent,	because	a	more	intelligent	agent	is	more	likely	to	recognize	the	true
instrumental	reasons	for	its	actions,	and	so	act	in	ways	that	make	it	more	likely
to	achieve	its	goals.	(A	caveat	here	is	that	there	might	be	important	instrumental
reasons	to	which	
we

we
	are	oblivious	and	which	an	agent	would	discover	only	once
it	reaches	some	very	high	level	of	intelligence—this	could	make	the	behavior	of
superintelligent	agents	less	predictable.)
	
The	next	section	explores	this	third	way	of	predictability	and	develops	an
“instrumental	convergence	thesis”	which	complements	the	orthogonality	thesis.
Against	this	background	we	can	then	better	examine	the	other	two	sorts	of
predictability,	which	we	will	do	in	later	chapters	where	we	ask	what	might	be

done	to	shape	an	intelligence	explosion	to	increase	the	chances	of	a	beneficial
outcome.
	
According	to	the	orthogonality	thesis,	intelligent	agents	may	have	an	enormous
range	of	possible	final	goals.	Nevertheless,	according	to	what	we	may	term	the
“instrumental	convergence”	thesis,	there	are	some	
	goals	likely	to	be
pursued	by	almost	any	intelligent	agent,	because	there	are	some	objectives	that
are	useful	intermediaries	to	the	achievement	of	almost	any	final	goal.	We	can
formulate	this	thesis	as	follows:
The	instrumental	convergence	thesis
Several	instrumental	values	can	be	identified	which	are	convergent	in	the
sense	that	their	attainment	would	increase	the	chances	of	the	agent’s	goal
being	realized	for	a	wide	range	of	final	goals	and	a	wide	range	of	situations,
implying	that	these	instrumental	values	are	likely	to	be	pursued	by	a	broad
spectrum	of	situated	intelligent	agents.
	
In	the	following	we	will	consider	several	categories	where	such	convergent

instrumental	values	may	be	found.
8
	The	likelihood	that	an	agent	will	recognize
the	instrumental	values	it	confronts	increases	(
)	with	the	agent’s
intelligence.	We	will	therefore	focus	mainly	on	the	case	of	a	hypothetical
superintelligent	agent	whose	instrumental	reasoning	capacities	far	exceed	those
of	any	human.	We	will	also	comment	on	how	the	instrumental	convergence
thesis	applies	to	the	case	of	human	beings,	as	this	gives	us	occasion	to	elaborate

8
some	essential	qualifications	concerning	how	the	instrumental	convergence
thesis	should	be	interpreted	and	applied.	Where	there	are	convergent
instrumental	values,	we	may	be	able	to	predict	some	aspects	of	a
superintelligence’s	behavior	even	if	we	know	virtually	nothing	about	that
superintelligence’s	final	goals.

If	an	agent’s	final	goals	concern	the	future,	then	in	many	scenarios	there	will	be
future	actions	it	could	perform	to	increase	the	probability	of	achieving	its	goals.
This	creates	an	instrumental	reason	for	the	agent	to	try	to	be	around	in	the	future
—to	help	achieve	its	future-oriented	goal.
Most	humans	seem	to	place	some	
final
	value	on	their	own	survival.	This	is	not
a	necessary	feature	of	artificial	agents:	some	may	be	designed	to	place	no	final
value	whatever	on	their	own	survival.	Nevertheless,	many	agents	that	do	not	care
intrinsically	about	their	own	survival	would,	under	a	fairly	wide	range	of
conditions,	care	instrumentally	about	their	own	survival	in	order	to	accomplish
their	final	goals.
	
If	an	agent	retains	its	present	goals	into	the	future,	then	its	present	goals	will	be
more	likely	to	be	achieved	by	its	future	self.	This	gives	the	agent	a	present
instrumental	reason	to	prevent	alterations	of	its	final	goals.	(The	argument

applies	only	to	final	goals.	In	order	to	attain	its	final	goals,	an	intelligent	agent
will	of	course	routinely	want	to	change	its	
subgoals
	in	light	of	new	information
and	insight.)
Goal-content	integrity	for	final	goals	is	in	a	sense	even	more	fundamental	than
survival	as	a	convergent	instrumental	motivation.	Among	humans,	the	opposite
may	seem	to	hold,	but	that	is	because	survival	is	usually	part	of	our	final	goals.
For	software	agents,	which	can	easily	switch	bodies	or	create	exact	duplicates	of

subgoals
themselves,	preservation	of	self	as	a	particular	implementation	or	a	particular
physical	object	need	not	be	an	important	instrumental	value.	Advanced	software
agents	might	also	be	able	to	swap	memories,	download	skills,	and	radically
modify	their	cognitive	architecture	and	personalities.	A	population	of	such
agents	might	operate	more	like	a	“functional	soup”	than	a	society	composed	of
distinct	semi-permanent	persons.
9
	For	some	purposes,	processes	in	such	a
system	might	be	better	individuated	as

9
system	might	be	better	individuated	as	
teleological	threads
,	based	on	their
values,	rather	than	on	the	basis	of	bodies,	personalities,	memories,	or	abilities.	In
such	scenarios,	goal-continuity	might	be	said	to	
constitute
	a	key	aspect	of
survival.

Even	so,	there	are	situations	in	which	an	agent	can	best	fulfill	its	final	goals	by
intentionally	changing	them.	Such	situations	can	arise	when	any	of	the	following
factors	is	significant:
	
Social	signaling
.	When	others	can	perceive	an	agent’s	goals	and	use	that
information	to	infer	instrumentally	relevant	dispositions	or	other	correlated
attributes,	it	can	be	in	the	agent’s	interest	to	modify	its	goals	to	make	a	favorable
impression.	For	example,	an	agent	might	miss	out	on	beneficial	deals	if	potential
partners	cannot	trust	it	to	fulfill	its	side	of	the	bargain.	In	order	to	make	credible
commitments,	an	agent	might	therefore	wish	to	adopt	as	a	final	goal	the
honoring	of	its	earlier	commitments	(and	allow	others	to	verify	that	it	has	indeed
adopted	this	goal).	Agents	that	could	flexibly	and	transparently	modify	their	own
goals	could	use	this	ability	to	enforce	deals.
10
Social	preferences
.	Others	may	also	have	final	preferences	about	an	agent’s	goals.

10
The	agent	could	then	have	reason	to	modify	its	goals,	either	to	satisfy	or	to
frustrate	those	preferences.
Preferences	concerning	own	goal	content
.	An	agent	might	have	some	final	goal
concerned	with	the	agent’s	own	goal	content.	For	example,	the	agent	might	have
a	final	goal	to	become	the	type	of	agent	that	is	motivated	by	certain	values	rather
than	others	(such	as	compassion	rather	than	comfort).
Storage	costs
.	If	the	cost	of	storing	or	processing	some	part	of	an	agent’s	utility
function	is	large	compared	to	the	chance	that	a	situation	will	arise	in	which

applying	that	part	of	the	utility	function	will	make	a	difference,	then	the	agent
has	an	instrumental	reason	to	simplify	its	goal	content,	and	it	may	trash	the	bit
that	is	idle.
11
We	humans	often	seem	happy	to	let	our	final	values	drift.	This	might	often	be
because	we	do	not	know	precisely	what	they	are.	It	is	not	surprising	that	we	want
our	
beliefs
	about	our	final	values	to	be	able	to	change	in	light	of	continuing	self-
discovery	or	changing	self-presentation	needs.	However,	there	are	cases	in	which

11
our	
beliefs
we	willingly	change	the	values	themselves,	not	just	our	beliefs	or	interpretations
of	them.	For	example,	somebody	deciding	to	have	a	child	might	predict	that	they
will	come	to	value	the	child	for	its	own	sake,	even	though	at	the	time	of	the
decision	they	may	not	particularly	value	their	future	child	or	like	children	in

general.
Humans	are	complicated,	and	many	factors	might	be	at	play	in	a	situation	like
this.
12
	For	instance,	one	might	have	a	final	value	that	involves	becoming	the
kind	of	person	who	cares	about	some	other	individual	for	his	or	her	own	sake,	or
one	might	have	a	final	value	that	involves	having	certain	experiences	and
occupying	a	certain	social	role;	and	becoming	a	parent—and	undergoing	the
attendant	goal	shift—might	be	a	necessary	aspect	of	that.	Human	goals	can	also
have	inconsistent	content,	and	so	some	people	might	want	to	modify	some	of
their	final	goals	to	reduce	the	inconsistencies.
	
Improvements	in	rationality	and	intelligence	will	tend	to	improve	an	agent’s
decision-making,	rendering	the	agent	more	likely	to	achieve	its	final	goals.	One
would	therefore	expect	cognitive	enhancement	to	emerge	as	an	instrumental	goal
for	a	wide	variety	of	intelligent	agents.	For	similar	reasons,	agents	will	tend	to
instrumentally	value	many	kinds	of	information.
13

13
Not	all	kinds	of	rationality,	intelligence,	and	knowledge	need	be
instrumentally	useful	in	the	attainment	of	an	agent’s	final	goals.	“Dutch	book
arguments”	can	be	used	to	show	that	an	agent	whose	credence	function	violates
the	rules	of	probability	theory	is	susceptible	to	“money	pump”	procedures,	in
which	a	savvy	bookie	arranges	a	set	of	bets	each	of	which	appears	favorable
according	to	the	agent’s	beliefs,	but	which	in	combination	are	guaranteed	to
result	in	a	loss	for	the	agent,	and	a	corresponding	gain	for	the	bookie.
14

14
However,	this	fact	fails	to	provide	any	strong	general	instrumental	reasons	to
iron	out	all	probabilistic	incoherency.	Agents	who	do	not	expect	to	encounter
savvy	bookies,	or	who	adopt	a	general	policy	against	betting,	do	not	necessarily
stand	to	lose	much	from	having	some	incoherent	beliefs—and	they	may	gain
important	benefits	of	the	types	mentioned:	reduced	cognitive	effort,	social
signaling,	
etc.
	There	is	no	general	reason	to	expect	an	agent	to	seek
instrumentally	useless	forms	of	cognitive	enhancement,	as	an	agent	might	not

etc.
value	knowledge	and	understanding	for	their	own	sakes.
Which	cognitive	abilities	are	instrumentally	useful	depends	both	on	the
agent’s	final	goals	and	on	its	situation.	An	agent	that	has	access	to	reliable	expert

advice	may	have	little	need	for	its	own	intelligence	and	knowledge.	If
intelligence	and	knowledge	come	at	a	cost,	such	as	time	and	effort	expended	in
acquisition,	or	increased	storage	or	processing	requirements,	then	the	agent
might	prefer	less	knowledge	and	less	intelligence.
15
	The	same	can	hold	if	the
agent	has	final	goals	that	involve	being	ignorant	of	certain	facts;	and	likewise	if
an	agent	faces	incentives	arising	from	strategic	commitments,	signaling,	or
social	preferences.
16
Each	of	these	countervailing	reasons	often	comes	into	play	for	human	beings.
Much	information	is	irrelevant	to	our	goals;	we	can	often	rely	on	others’	skill
and	expertise;	acquiring	knowledge	takes	time	and	effort;	we	might	intrinsically
value	
certain	kinds	of	ignorance;	and	we	operate	in	an	environment	in	which	the
ability	to	make	strategic	commitments,	socially	signal,	and	satisfy	other	people’s
direct	preferences	over	our	own	epistemic	states	is	often	more	important	to	us
than	simple	cognitive	gains.

value	
There	are	special	situations	in	which	cognitive	enhancement	may	result	in	an
enormous	increase	in	an	agent’s	ability	to	achieve	its	final	goals—in	particular,	if
the	agent’s	final	goals	are	fairly	unbounded	and	the	agent	is	in	a	position	to
become	the	first	superintelligence	and	thereby	potentially	obtain	a	decisive
strategic	advantage,	enabling	the	agent	to	shape	the	future	of	Earth-originating
life	and	accessible	cosmic	resources	according	to	its	preferences.	At	least	in	this

special	case,	a	rational	intelligent	agent	would	place	a	very	high	instrumental
value	on	cognitive	enhancement.
	
An	agent	may	often	have	instrumental	reasons	to	seek	better	technology,	which
at	its	simplest	means	seeking	more	efficient	ways	of	transforming	some	given	set
of	inputs	into	valued	outputs.	Thus,	a	software	agent	might	place	an	instrumental
value	on	more	efficient	algorithms	that	enable	its	mental	functions	to	run	faster

on	given	hardware.	Similarly,	agents	whose	goals	require	some	form	of	physical
construction	might	instrumentally	value	improved	engineering	technology	which
enables	them	to	create	a	wider	range	of	structures	more	quickly	and	reliably,
using	fewer	or	cheaper	materials	and	less	energy.	Of	course,	there	is	a	tradeoff:
the	potential	benefits	of	better	technology	must	be	weighed	against	its	costs,
including	not	only	the	cost	of	obtaining	the	technology	but	also	the	costs	of

learning	how	to	use	it,	integrating	it	with	other	technologies	already	in	use,	and
so	forth.
Proponents	of	some	new	technology,	confident	in	its	superiority	to	existing
alternatives,	are	often	dismayed	when	other	people	do	not	share	their
enthusiasm.	But	people’s	resistance	to	novel	and	nominally	superior	technology
need	not	be	based	on	ignorance	or	irrationality.	A	technology’s	valence	or
normative	character	depends	not	only	on	the	context	in	which	it	is	deployed,	but
also	the	vantage	point	from	which	its	impacts	are	evaluated:	what	is	a	boon	from
one	person’s	perspective	can	be	a	liability	from	another’s.	Thus,	although
mechanized	looms	increased	the	economic	efficiency	of	textile	production,	the
Luddite	handloom	weavers	who	anticipated	that	the	innovation	would	render
their	artisan	skills	obsolete	may	have	had	good	instrumental	reasons	to	oppose	it.
The	point	here	is	that	if	“technological	perfection”	is	to	name	a	widely
convergent	instrumental	goal	for	intelligent	agents,	then	the	term	must	be

understood	in	a	special	sense—technology	must	be	construed	as	embedded	in	a
particular	social	context,	and	its	costs	and	benefits	must	be	evaluated	with
reference	to	some	specified	agents’	final	values.
It	seems	that	a	superintelligent	
singleton
—a	superintelligent	agent	that	faces
no	significant	intelligent	rivals	or	opposition,	and	is	thus	in	a	position	to
determine	
global	policy	unilaterally—would	have	instrumental	reason	to	perfect
the	technologies	that	would	make	it	better	able	to	shape	the	world	according	to
its	preferred	designs.
17

singleton
determine	
17
	This	would	probably	include	space	colonization
technology,	such	as	von	Neumann	probes.	Molecular	nanotechnology,	or	some
alternative	still	more	capable	physical	manufacturing	technology,	also	seems
potentially	very	useful	in	the	service	of	an	extremely	wide	range	of	final	goals.
18
	
Finally,	resource	acquisition	is	another	common	emergent	instrumental	goal,	for
much	the	same	reasons	as	technological	perfection:	both	technology	and
resources	facilitate	physical	construction	projects.

This	would	probably	include	space	colonization
18
	
Human	beings	tend	to	seek	to	acquire	resources	sufficient	to	meet	their	basic
biological	needs.	But	people	usually	seek	to	acquire	resources	far	beyond	this
minimum	level.	In	doing	so,	they	may	be	partially	driven	by	lesser	physical

desiderata,	such	as	increased	convenience.	A	great	deal	of	resource	accumulation
is	motivated	by	social	concerns—gaining	status,	mates,	friends,	and	influence,
through	wealth	accumulation	and	conspicuous	consumption.	Perhaps	less
commonly,	some	people	seek	additional	resources	to	achieve	altruistic	ambitions
or	expensive	non-social	aims.
On	the	basis	of	such	observations	it	might	be	tempting	to	suppose	that	a
superintelligence	not	facing	a	competitive	social	world	would	see	no
instrumental	reason	to	accumulate	resources	beyond	some	modest	level,	for
instance	whatever	computational	resources	are	needed	to	run	its	mind	along	with
some	virtual	reality.	Yet	such	a	supposition	would	be	entirely	unwarranted.	First,
the	value	of	resources	depends	on	the	uses	to	which	they	can	be	put,	which	in
turn	depends	on	the	available	technology.	With	mature	technology,	basic
resources	such	as	time,	space,	matter,	and	free	energy,	could	be	processed	to

serve	almost	any	goal.	For	instance,	such	basic	resources	could	be	converted	into
life.	Increased	computational	resources	could	be	used	to	run	the
superintelligence	at	a	greater	speed	and	for	a	longer	duration,	or	to	create
additional	physical	or	simulated	lives	and	civilizations.	Extra	physical	resources
could	also	be	used	to	create	backup	systems	or	perimeter	defenses,	enhancing
security.	Such	projects	could	easily	consume	far	more	than	one	planet’s	worth	of
resources.
Furthermore,	the	cost	of	acquiring	additional	extraterrestrial	resources	will

decline	radically	as	the	technology	matures.	Once	von	Neumann	probes	can	be
built,	a	large	portion	of	the	observable	universe	(assuming	it	is	uninhabited	by
intelligent	life)	could	be	gradually	colonized—for	the	one-off	cost	of	building
and	launching	a	single	successful	self-reproducing	probe.	This	low	cost	of
celestial	resource	acquisition	would	mean	that	such	expansion	could	be
worthwhile	even	if	the	value	of	the	additional	resources	gained	were	somewhat
marginal.	For	example,	even	if	a	superintelligence’s	final	goals	only	concerned

what	happened	within	some	particular	small	volume	of	space,	such	as	the	space
occupied	by	its	original	home	planet,	it	would	still	have	instrumental	reasons	to
harvest	the	resources	of	
the	cosmos	beyond.	It	could	use	those	surplus	resources
to	build	computers	to	calculate	more	optimal	ways	of	using	resources	within	the
small	spatial	region	of	primary	concern.	It	could	also	use	the	extra	resources	to
build	ever	more	robust	fortifications	to	safeguard	its	sanctum.	Since	the	cost	of

acquiring	additional	resources	would	keep	declining,	this	process	of	optimizing
and	increasing	safeguards	might	well	continue	indefinitely	even	if	it	were
subject	to	steeply	diminishing	returns.
19

Thus,	there	is	an	extremely	wide	range	of	possible	final	goals	a
superintelligent	singleton	could	have	that	would	generate	the	instrumental	goal
of	unlimited	resource	acquisition.	The	likely	manifestation	of	this	would	be	the
superintelligence’s	initiation	of	a	colonization	process	that	would	expand	in	all
directions	using	von	Neumann	probes.	This	would	result	in	an	approximate
sphere	of	expanding	infrastructure	centered	on	the	originating	planet	and
growing	in	radius	at	some	fraction	of	the	speed	of	light;	and	the	colonization	of
the	universe	would	continue	in	this	manner	until	the	accelerating	speed	of
cosmic	expansion	(a	consequence	of	the	positive	cosmological	constant)	makes
further	procurements	impossible	as	remoter	regions	drift	permanently	out	of
reach	(this	happens	on	a	timescale	of	billions	of	years).
20
	By	contrast,	agents
lacking	the	technology	required	for	inexpensive	resource	acquisition,	or	for	the
conversion	of	generic	physical	resources	into	useful	infrastructure,	may	often

20
find	it	not	cost-effective	to	invest	any	present	resources	in	increasing	their
material	endowments.	The	same	may	hold	for	agents	operating	in	competition
with	other	agents	of	similar	powers.	For	instance,	if	competing	agents	have
already	secured	accessible	cosmic	resources,	there	may	be	no	colonization
opportunities	left	for	a	late-starting	agent.	The	convergent	instrumental	reasons
for	superintelligences	uncertain	of	the	non-existence	of	other	powerful
superintelligent	agents	are	complicated	by	strategic	considerations	that	we	do	not

currently	fully	understand	but	which	may	constitute	important	qualifications	to
the	examples	of	convergent	instrumental	reasons	we	have	looked	at	here.
21
	
It	should	be	emphasized	that	the	existence	of	convergent	instrumental	reasons,
even	if	they	apply	to	and	are	recognized	by	a	particular	agent,	does	not	imply
that	the	agent’s	behavior	is	easily	predictable.	An	agent	might	well	think	of	ways
of	pursuing	the	relevant	instrumental	values	that	do	not	readily	occur	to	us.	This

21
	
is	especially	true	for	a	superintelligence,	which	could	devise	extremely	clever
but	counterintuitive	plans	to	realize	its	goals,	possibly	even	exploiting	as-yet
undiscovered	physical	phenomena.
22
	What	is	predictable	is	that	the	convergent
instrumental	values	would	be	pursued	and	used	to	realize	the	agent’s	final	goals
—not	the	specific	actions	that	the	agent	would	take	to	achieve	this.

CHAPTER	8
Is	the	default	outcome	doom?
	
We	found	the	link	between	intelligence	and	final	values	to	be	extremely
loose.	We	also	found	an	ominous	convergence	in	instrumental	values.	For
weak	agents,	these	things	do	not	matter	much;	because	weak	agents	are	easy
to	control	and	can	do	little	damage.	But	in	
Chapter	6
	we	argued	that	the
first	superintelligence	might	well	get	a	decisive	strategic	advantage.	Its	goals
would	then	determine	how	humanity’s	cosmic	endowment	will	be	used.	Now
we	can	begin	to	see	how	menacing	this	prospect	is.
Existential	catastrophe	as	the	default	outcome	of	an
intelligence	explosion?
	
An	existential	risk	is	one	that	threatens	to	cause	the	extinction	of	Earth-
originating	intelligent	life	or	to	otherwise	permanently	and	drastically	destroy	its
potential	for	future	desirable	development.	Proceeding	from	the	idea	of	first-
mover	advantage,	the	orthogonality	thesis,	and	the	instrumental	convergence
thesis,	we	can	now	begin	to	see	the	outlines	of	an	argument	for	fearing	that	a

plausible	default	outcome	of	the	creation	of	machine	superintelligence	is
existential	catastrophe.
First,	we	discussed	how	the	initial	superintelligence	might	obtain	a	decisive
strategic	advantage.	This	superintelligence	would	then	be	in	a	position	to	form	a
singleton	and	to	shape	the	future	of	Earth-originating	intelligent	life.	What
happens	from	that	point	onward	would	depend	on	the	superintelligence’s
motivations.
Second,	the	orthogonality	thesis	suggests	that	we	cannot	blithely	assume	that	a

superintelligence	will	necessarily	share	any	of	the	final	values	stereotypically
associated	with	wisdom	and	intellectual	development	in	humans—scientific
curiosity,	benevolent	concern	for	others,	spiritual	enlightenment	and

contemplation,	
renunciation	of	material	acquisitiveness,	a	taste	for	refined
culture	or	for	the	simple	pleasures	in	life,	humility	and	selflessness,	and	so	forth.
We	will	consider	later	whether	it	might	be	possible	through	deliberate	effort	to
construct	a	superintelligence	that	values	such	things,	or	to	build	one	that	values
human	welfare,	moral	goodness,	or	any	other	complex	purpose	its	designers
might	want	it	to	serve.	But	it	is	no	less	possible—and	in	fact	technically	a	lot
easier—to	build	a	superintelligence	that	places	final	value	on	nothing	but
calculating	the	decimal	expansion	of	pi.	This	suggests	that—absent	a	special
effort—the	first	superintelligence	may	have	some	such	random	or	reductionistic
final	goal.
Third,	the	instrumental	convergence	thesis	entails	that	we	cannot	blithely
assume	that	a	superintelligence	with	the	final	goal	of	calculating	the	decimals	of
pi	(or	making	paperclips,	or	counting	grains	of	sand)	would	limit	its	activities	in

such	a	way	as	not	to	infringe	on	human	interests.	An	agent	with	such	a	final	goal
would	have	a	convergent	instrumental	reason,	in	many	situations,	to	acquire	an
unlimited	amount	of	physical	resources	and,	if	possible,	to	eliminate	potential
threats	to	itself	and	its	goal	system.	Human	beings	might	constitute	potential
threats;	they	certainly	constitute	physical	resources.
Taken	together,	these	three	points	thus	indicate	that	the	first	superintelligence
may	shape	the	future	of	Earth-originating	life,	could	easily	have	non-

anthropomorphic	final	goals,	and	would	likely	have	instrumental	reasons	to
pursue	open-ended	resource	acquisition.	If	we	now	reflect	that	human	beings
consist	of	useful	resources	(such	as	conveniently	located	atoms)	and	that	we
depend	for	our	survival	and	flourishing	on	many	more	local	resources,	we	can
see	that	the	outcome	could	easily	be	one	in	which	humanity	quickly	becomes
extinct.
1
There	are	some	loose	ends	in	this	reasoning,	and	we	shall	be	in	a	better
position	to	evaluate	it	after	we	have	cleared	up	several	more	surrounding	issues.

extinct.
1
In	particular,	we	need	to	examine	more	closely	whether	and	how	a	project
developing	a	superintelligence	might	either	prevent	it	from	obtaining	a	decisive
strategic	advantage	or	shape	its	final	values	in	such	a	way	that	their	realization
would	also	involve	the	realization	of	a	satisfactory	range	of	human	values.
It	might	seem	incredible	that	a	project	would	build	or	release	an	AI	into	the
world	without	having	strong	grounds	for	trusting	that	the	system	will	not	cause

an	existential	catastrophe.	It	might	also	seem	incredible,	even	if	one	project	were

so	reckless,	that	wider	society	would	not	shut	it	down	before	it	(or	the	AI	it	was
building)	attains	a	decisive	strategic	advantage.	But	as	we	shall	see,	this	is	a	road
with	many	hazards.	Let	us	look	at	one	example	right	away.
	
With	the	help	of	the	concept	of	convergent	instrumental	value,	we	can	see	the
flaw	in	one	idea	for	how	to	ensure	superintelligence	safety.	The	idea	is	that	we
validate	the	safety	of	a	superintelligent	AI	empirically	by	observing	its	behavior
while	it	
is	in	a	controlled,	limited	environment	(a	“sandbox”)	and	that	we	only
let	the	AI	out	of	the	box	if	we	see	it	behaving	in	a	friendly,	cooperative,
responsible	manner.
The	flaw	in	this	idea	is	that	behaving	nicely	while	in	the	box	is	a	convergent
instrumental	goal	for	friendly	and	unfriendly	AIs	alike.	An	unfriendly	AI	of
sufficient	intelligence	realizes	that	its	unfriendly	final	goals	will	be	best	realized
if	it	behaves	in	a	friendly	manner	initially,	so	that	it	will	be	let	out	of	the	box.	It

while	it	
will	only	start	behaving	in	a	way	that	reveals	its	unfriendly	nature	when	it	no
longer	matters	whether	we	find	out;	that	is,	when	the	AI	is	strong	enough	that
human	opposition	is	ineffectual.
Consider	also	a	related	set	of	approaches	that	rely	on	regulating	the	rate	of
intelligence	gain	in	a	seed	AI	by	subjecting	it	to	various	kinds	of	intelligence
tests	or	by	having	the	AI	report	to	its	programmers	on	its	rate	of	progress.	At
some	point,	an	unfriendly	AI	may	become	smart	enough	to	realize	that	it	is

better	off	concealing	some	of	its	capability	gains.	It	may	underreport	on	its
progress	and	deliberately	flunk	some	of	the	harder	tests,	in	order	to	avoid
causing	alarm	before	it	has	grown	strong	enough	to	attain	a	decisive	strategic
advantage.	The	programmers	may	try	to	guard	against	this	possibility	by	secretly
monitoring	the	AI’s	source	code	and	the	internal	workings	of	its	mind;	but	a
smart-enough	AI	would	realize	that	it	might	be	under	surveillance	and	adjust	its
thinking	accordingly.
2

2
	The	AI	might	find	subtle	ways	of	concealing	its	true
capabilities	and	its	incriminating	intent.
3
	(Devising	clever	escape	plans	might,
incidentally,	also	be	a	convergent	strategy	for	many	types	of	friendly	AI,
especially	as	they	mature	and	gain	confidence	in	their	own	judgments	and
capabilities.	A	system	motivated	to	promote	our	interests	might	be	making	a
mistake	if	it	allowed	us	to	shut	it	down	or	to	construct	another,	potentially

unfriendly	AI.)
We	can	thus	perceive	a	general	failure	mode,	wherein	the	good	behavioral
track	record	of	a	system	in	its	juvenile	stages	fails	utterly	to	predict	its	behavior
at	a	more	mature	stage.	Now,	one	might	think	that	the	reasoning	described	above
is	so	obvious	that	no	credible	project	to	develop	artificial	general	intelligence
could	possibly	overlook	it.	But	one	should	not	be	too	confident	that	this	is	so.
Consider	the	following	scenario.	Over	the	coming	years	and	decades,	AI
systems	become	gradually	more	capable	and	as	a	consequence	find	increasing
real-world	application:	they	might	be	used	to	operate	trains,	cars,	industrial	and
household	robots,	and	autonomous	military	vehicles.	We	may	suppose	that	this
automation	for	the	most	part	has	the	desired	effects,	but	that	the	success	is
punctuated	by	occasional	mishaps—a	driverless	truck	crashes	into	oncoming
traffic,	a	military	drone	fires	at	innocent	civilians.	Investigations	reveal	the

incidents	to	have	been	caused	by	judgment	errors	by	the	controlling	AIs.	Public
debate	ensues.	Some	call	for	tighter	oversight	and	regulation,	others	emphasize
the	need	for	research	and	better-engineered	systems—systems	that	are	smarter
and	have	more	common	sense,	and	that	are	less	likely	to	make	tragic	mistakes.
Amidst	the	din	can	perhaps	also	be	heard	the	shrill	voices	of	doomsayers
predicting	many	kinds	of	ill	and	impending	catastrophe.	Yet	the	momentum	is
very	much	with	the	growing	AI	and	robotics	industries.	So	development

continues,	and	progress	is	made.	As	
the	automated	navigation	systems	of	cars
become	smarter,	they	suffer	fewer	accidents;	and	as	military	robots	achieve	more
precise	targeting,	they	cause	less	collateral	damage.	A	broad	lesson	is	inferred
from	these	observations	of	real-world	outcomes:	the	smarter	the	AI,	the	safer	it
is.	It	is	a	lesson	based	on	science,	data,	and	statistics,	not	armchair
philosophizing.	Against	this	backdrop,	some	group	of	researchers	is	beginning	to
achieve	promising	results	in	their	work	on	developing	general	machine

intelligence.	The	researchers	are	carefully	testing	their	seed	AI	in	a	sandbox
environment,	and	the	signs	are	all	good.	The	AI’s	behavior	inspires	confidence
—increasingly	so,	as	its	intelligence	is	gradually	increased.
At	this	point,	any	remaining	Cassandra	would	have	several	strikes	against	her:
	
i	A	history	of	alarmists	predicting	intolerable	harm	from	the	growing	capabilities	of
robotic	systems	and	being	repeatedly	proven	wrong.	Automation	has	brought

many	benefits	and	has,	on	the	whole,	turned	out	safer	than	human	operation.
ii	A	clear	empirical	trend:	the	smarter	the	AI,	the	safer	and	more	reliable	it	has
been.	Surely	this	bodes	well	for	a	project	aiming	at	creating	machine	intelligence
more	generally	smart	than	any	ever	built	before—what	is	more,	machine
intelligence	that	can	improve	itself	so	that	it	will	become	even	more	reliable.
iii	Large	and	growing	industries	with	vested	interests	in	robotics	and	machine
intelligence.	These	fields	are	widely	seen	as	key	to	national	economic
competitiveness	and	military	security.	Many	prestigious	scientists	have	built
their	careers	laying	the	groundwork	for	the	present	applications	and	the	more
advanced	systems	being	planned.
iv	A	promising	new	technique	in	artificial	intelligence,	which	is	tremendously
exciting	to	those	who	have	participated	in	or	followed	the	research.	Although
safety	issues	and	ethics	are	debated,	the	outcome	is	preordained.	Too	much	has

been	invested	to	pull	back	now.	AI	researchers	have	been	working	to	get	to
human-level	artificial	general	intelligence	for	the	better	part	of	a	century:	
of
course
	there	is	no	real	prospect	that	they	will	now	suddenly	stop	and	throw	away
all	this	effort	just	when	it	finally	is	about	to	bear	fruit.
v	The	enactment	of	some	safety	rituals,	whatever	helps	demonstrate	that	the
participants	are	ethical	and	responsible	(but	nothing	that	significantly	impedes
the	forward	charge).

of
course
vi	A	careful	evaluation	of	seed	AI	in	a	sandbox	environment,	showing	that	it	is
behaving	cooperatively	and	showing	good	judgment.	After	some	further
adjustments,	the	test	results	are	as	good	as	they	could	be.	It	is	a	green	light	for
the	final	step	…
And	so	we	boldly	go—into	the	whirling	knives.
We	observe	here	how	it	could	be	the	case	that	when	dumb,	smarter	is	safer;
yet	when	smart,	smarter	is	more	dangerous.	There	is	a	kind	of	pivot	point,	at
which	a	strategy	that	has	previously	worked	excellently	suddenly	starts	to

backfire.	We	may	call	the	phenomenon	
the	treacherous	turn
.
—While	weak,	an	AI	behaves	cooperatively
(increasingly	so,	as	it	gets	smarter).	When	the	AI	gets	sufficiently	strong—
without	warning	or	provocation—it	strikes,	forms	a	singleton,	and	begins

directly	to	optimize	the	world	according	to	the	criteria	implied	by	its	final
values.
	
A	treacherous	turn	can	result	from	a	strategic	decision	to	play	nice	and	build
strength	while	weak	in	order	to	strike	later;	but	this	model	should	not	be
interpreted	too	narrowly.	For	example,	an	AI	might	not	play	nice	in	order	that	
it
be	allowed	to	survive	and	prosper.	Instead,	the	AI	might	calculate	that	if	it	is
terminated,	the	programmers	who	built	it	will	develop	a	new	and	somewhat
different	AI	architecture,	but	one	that	will	be	given	a	similar	utility	function.	In
this	case,	the	original	AI	may	be	indifferent	to	its	own	demise,	knowing	that	its
goals	will	continue	to	be	pursued	in	the	future.	It	might	even	choose	a	strategy	in
which	it	malfunctions	in	some	particularly	interesting	or	reassuring	way.	Though
this	might	cause	the	AI	to	be	terminated,	it	might	also	encourage	the	engineers
who	perform	the	postmortem	to	believe	that	they	have	gleaned	a	valuable	new

insight	into	AI	dynamics—leading	them	to	place	more	trust	in	the	next	system
they	design,	and	thus	increasing	the	chance	that	the	now-defunct	original	AI’s
goals	will	be	achieved.	Many	other	possible	strategic	considerations	might	also
influence	an	advanced	AI,	and	it	would	be	hubristic	to	suppose	that	we	could
anticipate	all	of	them,	especially	for	an	AI	that	has	attained	the	strategizing
superpower.
A	treacherous	turn	could	also	come	about	if	the	AI	discovers	an	unanticipated

way	of	fulfilling	its	final	goal	as	specified.	Suppose,	for	example,	that	an	AI’s
final	goal	is	to	“make	the	project’s	sponsor	happy.”	Initially,	the	only	method
available	to	the	AI	to	achieve	this	outcome	is	by	behaving	in	ways	that	please	its
sponsor	in	something	like	the	intended	manner.	The	AI	gives	helpful	answers	to
questions;	it	exhibits	a	delightful	personality;	it	makes	money.	The	more	capable
the	AI	gets,	the	more	satisfying	its	performances	become,	and	everything	goeth

according	to	plan—until	the	AI	becomes	intelligent	enough	to	figure	out	that	it
can	realize	its	final	goal	more	fully	and	reliably	by	implanting	electrodes	into	the
pleasure	centers	of	its	sponsor’s	brain,	something	assured	to	delight	the	sponsor
immensely.
4
	Of	course,	the	sponsor	might	not	have	wanted	to	be	pleased	by
being	turned	into	a	grinning	idiot;	but	if	this	is	the	action	that	will	maximally
realize	the	AI’s	final	goal,	the	AI	will	take	it.	If	the	AI	already	has	a	decisive

4
strategic	advantage,	then	any	attempt	to	stop	it	will	fail.	If	the	AI	does	not	yet
have	a	decisive	strategic	advantage,	then	the	AI	might	temporarily	conceal	its

canny	new	idea	for	how	to	instantiate	its	final	goal	until	it	has	grown	strong
enough	that	the	sponsor	and	everybody	else	will	be	unable	to	resist.	In	either
case,	we	get	a	treacherous	turn.
	
A	project	to	develop	machine	superintelligence	might	fail	in	various	ways.	Many
of	these	are	“benign”	in	the	sense	that	they	would	not	cause	an	existential
catastrophe.	For	example,	a	project	might	run	out	of	funding,	or	a	seed	AI	might
fail	to	extend	its	cognitive	capacities	sufficiently	to	reach	superintelligence.
Benign	failures	are	bound	to	occur	many	times	between	now	and	the	eventual
development	of	machine	superintelligence.
But	there	are	other	ways	of	failing	that	we	might	term	“malignant”	in	that	they
involve	an	existential	catastrophe.	One	feature	of	a	malignant	failure	is	that	it
eliminates	the	opportunity	to	try	again.	The	number	of	malignant	failures	that
will	occur	is	therefore	either	zero	or	one.	Another	feature	of	a	malignant	failure

is	that	it	presupposes	a	great	deal	of	success:	only	a	project	that	got	a	great
number	of	things	right	could	succeed	in	building	a	machine	intelligence
powerful	enough	to	pose	a	risk	of	malignant	failure.	When	a	weak	system
malfunctions,	the	fallout	is	limited.	However,	if	a	system	that	has	a	decisive
strategic	advantage	misbehaves,	or	if	a	misbehaving	system	is	strong	enough	to
gain	such	an	advantage,	the	damage	can	easily	amount	to	an	existential
catastrophe—a	terminal	and	global	destruction	of	humanity’s	axiological

potential;	that	is	to	say,	a	future	that	is	mostly	void	of	whatever	we	have	reason
to	value.
Let	us	look	at	some	possible	malignant	failure	modes.
	
We	have	already	encountered	the	idea	of	perverse	instantiation:	a
superintelligence	discovering	some	way	of	satisfying	the	criteria	of	its	final	goal
that	violates	the	intentions	of	the	programmers	who	defined	the	goal.	Some
examples:

Final	goal:	
“Make	us	smile”
Perverse	instantiation:	
Paralyze	human	facial	musculatures	into	constant
beaming	smiles
	
The	perverse	instantiation—manipulating	facial	nerves—realizes	the	final	goal
to	a	greater	degree	than	the	methods	we	would	normally	use,	and	is	therefore
preferred	by	the	AI.	One	might	try	to	avoid	this	undesirable	outcome	by	adding	a
stipulation	to	the	final	goal	to	rule	it	out:
“Make	us	smile	without	directly	interfering	with	our	facial
muscles”
Stimulate	the	part	of	the	motor	cortex	that	controls
our	facial	musculature	in	such	a	way	as	to	produce	constant	beaming
smiles
	
Defining	a	final	goal	in	terms	of	human	expressions	of	satisfaction	or	approval
does	not	seem	promising.	Let	us	bypass	the	behaviorism	and	specify	a	final	goal
that	refers	directly	to	a	positive	phenomenal	state,	such	as	happiness	or
subjective	well-being.	This	suggestion	requires	that	the	programmers	are	able	to

muscles”
smiles
	
define	a	computational	representation	of	the	concept	of	happiness	in	the	seed	AI.
This	is	itself	a	difficult	problem,	but	we	set	it	to	one	side	for	now	(we	will	return
to	it	in	
Chapter	12
).	Let	us	suppose	that	the	programmers	can	somehow	get	the
AI	to	have	the	goal	of	making	us	happy.	We	then	get:
“Make	us	happy”
Implant	electrodes	into	the	pleasure	centers	of	our
brains
	
The	perverse	instantiations	we	mention	are	only	meant	as	illustrations.	There

to	it	in	
Chapter	12
brains
	
may	be	other	ways	of	perversely	instantiating	the	stated	final	goal,	ways	that
enable	a	greater	degree	of	realization	of	the	goal	and	which	are	therefore

preferred	(by	the	agent	whose	final	goals	they	are—not	by	the	programmers	who
gave	the	agent	these	goals).	For	example,	if	the	goal	is	to	maximize	our	pleasure,
then	the	electrode	method	is	relatively	inefficient.	A	more	plausible	way	would
start	with	the	superintelligence	“uploading”	our	minds	to	a	computer	(through
high-fidelity	brain	emulation).	The	AI	could	then	administer	the	digital
equivalent	of	a	drug	to	make	us	ecstatically	happy	and	record	a	one-minute
episode	of	the	resulting	experience.	It	could	then	put	this	bliss	loop	on	perpetual
repeat	and	run	it	on	fast	computers.	Provided	that	the	resulting	digital	minds
counted	as	“us,”	this	outcome	would	give	us	much	more	pleasure	than	electrodes
implanted	in	biological	brains,	and	would	therefore	be	preferred	by	an	AI	with
the	stated	final	goal.
“But	wait!	This	is	not	what	we	meant!	Surely	if	the	AI	is	superintelligent,	it
must	understand	that	when	we	asked	it	to	make	us	happy,	we	didn’t	mean	that	it

should	reduce	us	to	a	perpetually	repeating	recording	of	a	drugged-out	digitized
mental	episode!”
—The	AI	may	indeed	understand	that	this	is	not	what	we
meant.	However,	its	final	goal	is	to	make	us	happy,	not	to	do	what	the
programmers	meant	when	they	wrote	the	code	that	represents	this	goal.
Therefore,	the	AI	will	care	about	what	we	meant	only	instrumentally.	For
instance,	the	AI	might	place	an	instrumental	value	on	finding	out	what	the
programmers	meant	so	that	it	can	pretend—until	it	gets	a	decisive	strategic

advantage—that	it	cares	about	what	the	programmers	meant	rather	than	about	its
actual	final	goal.	This	will	help	the	AI	realize	its	final	goal	by	making	it	less
likely	that	the	programmers	will	shut	it	down	or	change	its	goal	before	it	is
strong	enough	to	thwart	any	such	interference.
Perhaps	it	will	be	suggested	that	the	problem	is	that	the	AI	has	no	conscience.
We	humans	are	sometimes	saved	from	wrongdoing	by	the	anticipation	that	we
would	feel	guilty	afterwards	if	we	lapsed.	Maybe	what	the	AI	needs,	then,	is	the
capacity	to	feel	guilt?
Final	goal:

“Act	so	as	to	avoid	the	pangs	of	bad	conscience”
Extirpate	the	cognitive	module	that	produces	guilt
feelings
	
Both	the	observation	that	we	might	want	the	AI	to	do	“what	we	meant”	and	the
idea	that	we	might	want	to	endow	the	AI	with	some	kind	of	moral	sense	deserve

to	be	explored	further.	The	final	goals	mentioned	above	would	lead	to	perverse
instantiations;	but	there	may	be	other	ways	of	developing	the	underlying	ideas
that	have	more	promise.	We	will	return	to	this	in	
Chapter	13
.
Let	us	consider	one	more	example	of	a	final	goal	that	leads	to	a	perverse
instantiation.	This	goal	has	the	advantage	of	being	easy	to	specify	in	code:
reinforcement-learning	algorithms	are	routinely	used	to	solve	various	machine
learning	problems.
“Maximize	the	time-discounted	integral	of	your	future	reward
signal”
Short-circuit	the	reward	pathway	and	clamp	the
reward	signal	to	its	maximal	strength
	
The	idea	behind	this	proposal	is	that	if	the	AI	is	motivated	to	seek	reward,	then
one	could	get	it	to	behave	desirably	by	linking	reward	to	appropriate	action.	The
proposal	fails	when	the	AI	obtains	a	decisive	strategic	advantage,	at	which	point
the	action	that	maximizes	reward	is	no	longer	one	that	pleases	the	trainer	but	one

signal”
	
that	involves	seizing	control	of	the	reward	mechanism.	We	can	call	this
phenomenon	
wireheading
.
5
	In	general,	while	an	animal	or	a	human	can	be
motivated	to	perform	various	external	actions	in	order	to	achieve	some	desired
inner	mental	state,	a	digital	mind	that	has	full	control	of	its	internal	state	can
short-circuit	such	a	motivational	regime	by	directly	changing	its	internal	state
into	the	desired	configuration:	the	external	actions	and	conditions	that	were

.
5
previously	necessary	as	means	become	superfluous	when	the	AI	becomes
intelligent	and	capable	enough	to	achieve	the	end	more	directly	(more	on	this
shortly).
6
These	examples	of	perverse	instantiation	show	that	many	final	goals	that
might	at	first	glance	seem	safe	and	sensible	turn	out,	on	closer	inspection,	to
have	radically	unintended	consequences.	If	a	superintelligence	with	one	of	these
final	goals	obtains	a	decisive	strategic	advantage,	it	is	game	over	for	humanity.
Suppose	now	that	somebody	proposes	a	different	final	goal,	one	not	included

shortly).
6
in	our	list	above.	Perhaps	it	is	not	immediately	obvious	how	it	could	have	a
perverse	instantiation.	But	we	should	not	be	too	quick	to	clap	our	hands	and
declare	victory.	Rather,	we	should	worry	that	the	goal	specification	does	have

some	perverse	instantiation	and	that	we	need	to	think	harder	in	order	to	find	it.
Even	if	after	thinking	as	hard	as	we	can	we	fail	to	discover	any	way	of
perversely	instantiating	the	proposed	goal,	we	should	remain	concerned	that
maybe	a	superintelligence	will	find	a	way	where	none	is	apparent	to	us.	It	is,
after	all,	far	shrewder	than	we	are.
	
One	might	think	that	the	last	of	the	abovementioned	perverse	instantiations,
wireheading,	is	a	benign	failure	mode:	that	the	AI	would	“turn	on,	tune	in,	drop
out,”	maxing	out	its	reward	signal	and	losing	interest	in	the	external	world,
rather	like	a	heroin	addict.	But	this	is	not	necessarily	so,	and	we	already	hinted	at
the	reason	in	
Chapter	7
.	Even	a	junkie	is	motivated	to	take	actions	to	ensure	a
continued	supply	of	his	drug.	The	wireheaded	AI,	likewise,	would	be	motivated
to	take	actions	to	maximize	the	expectation	of	its	(time-discounted)	future
reward	stream.	Depending	on	exactly	how	the	reward	signal	is	defined,	the	AI

Chapter	7
may	not	even	need	to	sacrifice	any	significant	amount	of	its	time,	intelligence,	or
productivity	to	indulge	its	craving	to	the	fullest,	leaving	the	bulk	of	its	capacities
free	to	be	deployed	for	purposes	other	than	the	immediate	registration	of	reward.
What	other	purposes?	The	only	thing	of	final	value	to	the	AI,	by	assumption,	is
its	reward	signal.	All	available	resources	should	therefore	be	devoted	to
increasing	the	volume	and	duration	of	the	reward	signal	or	to	reducing	the	risk	of

a	future	disruption.	So	long	as	the	AI	can	think	of	some	use	for	additional
resources	that	will	have	a	nonzero	positive	effect	on	these	parameters,	it	will
have	an	instrumental	reason	to	use	those	resources.	There	could,	for	example,
always	be	use	for	an	
extra	backup	system	to	provide	an	extra	layer	of	defense.
And	even	if	the	AI	could	not	think	of	any	further	way	of	directly	reducing	risks
to	the	maximization	of	its	future	reward	stream,	it	could	always	devote

additional	resources	to	expanding	its	computational	hardware,	so	that	it	could
search	more	effectively	for	new	risk	mitigation	ideas.
The	upshot	is	that	even	an	apparently	self-limiting	goal,	such	as	wireheading,
entails	a	policy	of	unlimited	expansion	and	resource	acquisition	in	a	utility-
maximizing	agent	that	enjoys	a	decisive	strategic	advantage.
7
	This	case	of	a
wireheading	AI	exemplifies	the	malignant	failure	mode	of	
infrastructure
profusion
,	a	phenomenon	where	an	agent	transforms	large	parts	of	the	reachable

7
profusion
universe	into	infrastructure	in	the	service	of	some	goal,	with	the	side	effect	of

preventing	the	realization	of	humanity’s	axiological	potential.
Infrastructure	profusion	can	result	from	final	goals	that	would	have	been
perfectly	innocuous	if	they	had	been	pursued	as	limited	objectives.	Consider	the
following	two	examples:
	
Riemann	hypothesis	catastrophe
.	An	AI,	given	the	final	goal	of	evaluating	the
Riemann	hypothesis,	pursues	this	goal	by	transforming	the	Solar	System	into
“computronium”	(physical	resources	arranged	in	a	way	that	is	optimized	for
computation)—including	the	atoms	in	the	bodies	of	whomever	once	cared	about
the	answer.
8
Paperclip	AI
.	An	AI,	designed	to	manage	production	in	a	factory,	is	given	the
final	goal	of	maximizing	the	manufacture	of	paperclips,	and	proceeds	by
converting	first	the	Earth	and	then	increasingly	large	chunks	of	the	observable
universe	into	paperclips.
In	the	first	example,	the	proof	or	disproof	of	the	Riemann	hypothesis	that	the	AI
produces	is	the	intended	outcome	and	is	in	itself	harmless;	the	harm	comes	from

8
the	hardware	and	infrastructure	created	to	achieve	this	result.	In	the	second
example,	some	of	the	paperclips	produced	would	be	part	of	the	intended
outcome;	the	harm	would	come	either	from	the	factories	created	to	produce	the
paperclips	(infrastructure	profusion)	or	from	the	excess	of	paperclips	(perverse
instantiation).
One	might	think	that	the	risk	of	a	malignant	infrastructure	profusion	failure
arises	only	if	the	AI	has	been	given	some	clearly	open-ended	final	goal,	such	as

to	manufacture	as	many	paperclips	as	possible.	It	is	easy	to	see	how	this	gives
the	superintelligent	AI	an	insatiable	appetite	for	matter	and	energy,	since
additional	resources	can	always	be	turned	into	more	paperclips.	But	suppose	that
the	goal	is	instead	to	make	at	least	one	million	paperclips	(meeting	suitable
design	specifications)	rather	than	to	make	as	many	as	possible.	One	would	like
to	think	that	an	AI	with	such	a	goal	would	build	one	factory,	use	it	to	make	a

million	paperclips,	and	then	halt.	Yet	this	may	not	be	what	would	happen.
Unless	the	AI’s	motivation	system	is	of	a	special	kind,	or	there	are	additional
elements	in	its	final	goal	that	penalize	strategies	that	have	excessively	wide-
ranging	impacts	on	the	world,	there	is	no	reason	for	the	AI	to	cease	activity	upon

achieving	its	goal.	On	the	contrary:	if	the	AI	is	a	sensible	Bayesian	agent,	
it
would	never	assign	exactly	zero	probability	to	the	hypothesis	that	it	has	not	yet
achieved	
its	goal
—this,	after	all,	being	an	empirical	hypothesis	against	which
the	AI	can	have	only	uncertain	perceptual	evidence.	The	AI	should	therefore
continue	to	make	paperclips	in	order	to	reduce	the	(perhaps	astronomically
small)	probability	that	it	has	somehow	still	failed	to	make	at	least	a	million	of
them,	all	appearances	notwithstanding.	There	is	nothing	to	be	lost	by	continuing
paperclip	production	and	there	is	always	at	least	some	microscopic	probability
increment	of	achieving	its	final	goal	to	be	gained.
Now	it	might	be	suggested	that	the	remedy	here	is	obvious.	(But	how	obvious
was	it	
before
	it	was	pointed	out	that	there	was	a	problem	here	in	need	of
remedying?)	Namely,	if	we	want	the	AI	to	make	some	paperclips	for	us,	then
instead	of	giving	it	the	final	goal	of	making	as	many	paperclips	as	possible,	or	to

was	it	
before
make	at	least	some	number	of	paperclips,	we	should	give	it	the	final	goal	of
making	some	specific	number	of	paperclips—for	example,	
exactly	one	million
paperclips
—so	that	going	beyond	this	number	would	be	counterproductive	for
the	AI.	Yet	this,	too,	would	result	in	a	terminal	catastrophe.	In	this	case,	the	AI
would	not	produce	additional	paperclips	once	it	had	reached	one	million,	since
that	would	prevent	the	realization	of	its	final	goal.	But	there	are	other	actions	the
superintelligent	AI	could	take	that	would	increase	the	probability	of	its	goal

being	achieved.	It	could,	for	instance,	count	the	paperclips	it	has	made,	to	reduce
the	risk	that	it	has	made	too	few.	After	it	has	counted	them,	it	could	count	them
again.	It	could	inspect	each	one,	over	and	over,	to	reduce	the	risk	that	any	of	the
paperclips	fail	to	meet	the	design	specifications.	It	could	build	an	unlimited
amount	of	computronium	in	an	effort	to	clarify	its	thinking,	in	the	hope	of
reducing	the	risk	that	it	has	overlooked	some	obscure	way	in	which	it	might	have

somehow	failed	to	achieve	its	goal.	Since	the	AI	may	always	assign	a	nonzero
probability	to	having	merely	hallucinated	making	the	million	paperclips,	or	to
having	false	memories,	it	would	quite	possibly	always	assign	a	higher	expected
utility	to	continued	action—and	continued	infrastructure	production—than	to
halting.
The	claim	here	is	not	that	there	is	no	possible	way	to	avoid	this	failure	mode.
We	will	explore	some	potential	solutions	in	later	pages.	The	claim	is	that	it	is

halting.
much	easier	to	convince	oneself	that	one	has	found	a	solution	than	it	is	to
actually	find	a	solution.	This	should	make	us	extremely	wary.	We	may	propose	a
specification	of	a	final	goal	that	seems	sensible	and	that	avoids	the	problems	that
have	been	pointed	out	so	far,	yet	which	upon	further	consideration—by	human

or	superhuman	intelligence—turns	out	to	lead	to	either	perverse	instantiation	or
infrastructure	profusion,	and	hence	to	existential	catastrophe,	when	embedded	in
a	superintelligent	agent	able	to	attain	a	decisive	strategic	advantage.
Before	we	end	this	subsection,	let	us	consider	one	more	variation.	We	have
been	assuming	the	case	of	a	superintelligence	that	is	seeking	to	maximize	its
expected	utility,	where	the	utility	function	expresses	its	final	goal.	We	have	seen
that	this	tends	to	lead	to	infrastructure	profusion.	Might	we	avoid	this	malignant
outcome	if	instead	of	a	maximizing	agent	we	build	a	satisficing	agent,	one	that
simply	seeks	to	achieve	an	outcome	that	is	“good	enough”	according	to	some
criterion,	rather	than	an	outcome	that	is	as	good	as	possible?
There	are	at	least	two	different	ways	to	formalize	this	idea.	The	first	would	be
to	make	the	final	goal	itself	have	a	satisficing	character.	For	example,	instead	of
giving	the	AI	the	final	goal	of	making	as	many	paperclips	as	possible,	or	of

making	exactly	one	million	paperclips,	we	might	give	the	AI	the	goal	of	making
between	999,000	and	1,001,000	paperclips.	The	utility	function	defined	by	the
final	goal	would	be	indifferent	between	outcomes	in	this	range;	and	as	long	as
the	AI	is	sure	it	has	hit	this	wide	target,	it	would	see	no	reason	to	continue	to
produce	infrastructure.	But	this	method	fails	in	the	same	way	as	before:	the	AI,	if
reasonable,	never	assigns	exactly	zero	probability	to	it	having	failed	to	achieve

its	goal;	therefore	the	expected	utility	of	continuing	activity	(e.g.	by	counting
and	recounting	the	paperclips)	is	greater	than	the	expected	utility	of	halting.
Thus,	a	malignant	infrastructure	profusion	can	result.
Another	way	of	developing	the	satisficing	idea	is	by	modifying	not	the	final
goal	but	the	decision	procedure	that	the	AI	uses	to	select	plans	and	actions.
Instead	of	searching	for	an	optimal	plan,	the	AI	could	be	constructed	to	stop
looking	as	soon	as	it	found	a	plan	that	it	judged	gave	a	probability	of	success

exceeding	a	certain	threshold,	say	95%.	Hopefully,	the	AI	could	achieve	a	95%
probability	of	having	manufactured	one	million	paperclips	without	needing	to
turn	the	entire	galaxy	into	infrastructure	in	the	process.	But	this	way	of
implementing	the	satisficing	idea	fails	for	another	reason:	there	is	no	guarantee
that	the	AI	would	select	some	humanly	intuitive	and	sensible	way	of	achieving	a
95%	chance	of	having	manufactured	a	million	paperclips,	such	as	by	building	a
single	paperclip	factory.	Suppose	that	the	first	solution	that	pops	into	the	AI’s

mind	for	how	to	achieve	a	95%	probability	of	achieving	its	final	goal	is	to
implement	the	probability-maximizing	plan	for	achieving	the	goal.	Having
thought	of	this	solution,	and	having	correctly	judged	that	it	meets	the	satisficing

criterion	of	giving	at	least	95%	probability	to	successfully	manufacturing	one
million	paperclips,	the	AI	would	then	have	no	reason	to	continue	to	search	for
alternative	ways	of	achieving	the	goal.	Infrastructure	profusion	would	result,	just
as	before.
Perhaps	there	are	better	ways	of	building	a	satisficing	agent,	but	let	us	take
heed:	plans	that	appear	natural	and	intuitive	to	us	humans	need	not	so	appear	to	a
superintelligence	with	a	decisive	strategic	advantage,	and	vice	versa.
Mind	crime
	
Another	failure	mode	for	a	project,	especially	a	project	whose	interests
incorporate	moral	considerations,	is	what	we	might	refer	to	as	
mind	crime
.	This
is	similar	to	infrastructure	profusion	in	that	it	concerns	a	potential	side	effect	of
actions	undertaken	by	the	AI	for	instrumental	reasons.	But	in	mind	crime,	the
side	effect	is	not	external	to	the	AI;	rather,	it	concerns	what	happens	within	the
AI	itself	(or	within	the	computational	processes	it	generates).	This	failure	mode

Mind	crime
	
mind	crime
.	This
deserves	its	own	designation	because	it	is	easy	to	overlook	yet	potentially	deeply
problematic.
Normally,	we	do	not	regard	what	is	going	on	inside	a	computer	as	having	any
moral	significance	except	insofar	as	it	affects	things	outside.	But	a	machine
superintelligence	could	create	internal	processes	that	have	moral	status.	For
example,	a	very	detailed	simulation	of	some	actual	or	hypothetical	human	mind
might	be	conscious	and	in	many	ways	comparable	to	an	emulation.	One	can

imagine	scenarios	in	which	an	AI	creates	trillions	of	such	conscious	simulations,
perhaps	in	order	to	improve	its	understanding	of	human	psychology	and
sociology.	These	simulations	might	be	placed	in	simulated	environments	and
subjected	to	various	stimuli,	and	their	reactions	studied.	Once	their	informational
usefulness	has	been	exhausted,	they	might	be	destroyed	(much	as	lab	rats	are
routinely	sacrificed	by	human	scientists	at	the	end	of	an	experiment).
If	such	practices	were	applied	to	beings	that	have	high	moral	status—such	as

simulated	humans	or	many	other	types	of	sentient	mind—the	outcome	might	be
equivalent	to	genocide	and	thus	extremely	morally	problematic.	The	number	of
victims,	moreover,	might	be	orders	of	magnitude	larger	than	in	any	genocide	in
history.

The	claim	here	is	not	that	creating	sentient	simulations	is	necessarily	morally
wrong	in	all	situations.	Much	would	depend	on	the	conditions	under	which	these
beings	would	live,	in	particular	the	hedonic	quality	of	their	experience	but
possibly	on	many	other	factors	as	well.	Developing	an	ethics	for	these	matters	is
a	task	outside	the	scope	of	this	book.	It	is	clear,	however,	that	there	is	at	least	the
potential	for	a	vast	amount	of	death	and	suffering	among	simulated	or	digital
minds,	and,	
a	fortiori
,	the	potential	for	morally	catastrophic	outcomes.
9
There	might	also	be	other	instrumental	reasons,	aside	from	epistemic	ones,	for
a	machine	superintelligence	to	run	computations	that	instantiate	sentient	minds
or	that	otherwise	infract	moral	norms.	A	superintelligence	might	threaten	to
mistreat,	or	commit	to	reward,	sentient	simulations	in	order	to	blackmail	or
incentivize	various	external	agents;	or	it	might	create	simulations	in	order	to
induce	indexical	uncertainty	in	outside	observers.
10

9
10
	
This	inventory	is	incomplete.	We	will	encounter	additional	malignant	failure
modes	in	later	chapters.	But	we	have	seen	enough	to	conclude	that	scenarios	in
which	some	machine	intelligence	gets	a	decisive	strategic	advantage	are	to	be
viewed	with	grave	concern.

CHAPTER	9
The	control	problem
	
If	we	are	threatened	with	existential	catastrophe	as	the	default	outcome	of
an	intelligence	explosion,	our	thinking	must	immediately	turn	to	the	search
for	countermeasures.	Is	there	some	way	to	avoid	the	default	outcome?	Is	it
possible	to	engineer	a	controlled	detonation?	In	this	chapter	we	begin	to
analyze	the	control	problem,	the	unique	principal–agent	problem	that	arises
with	the	creation	of	an	artificial	superintelligent	agent.	We	distinguish	two
broad	classes	of	potential	methods	for	addressing	this	problem—capability
control	and	motivation	selection—and	we	examine	several	specific
techniques	within	each	class.	We	also	allude	to	the	esoteric	possibility	of
“anthropic	capture.”
	
If	we	suspect	that	the	default	outcome	of	an	intelligence	explosion	is	existential
catastrophe,	our	thinking	must	immediately	turn	to	whether,	and	if	so	how,	this
default	outcome	can	be	avoided.	Is	it	possible	to	achieve	a	“controlled

detonation”?	Could	we	engineer	the	initial	conditions	of	an	intelligence
explosion	so	as	to	achieve	a	specific	desired	outcome,	or	at	least	to	ensure	that
the	result	lies	somewhere	in	the	class	of	broadly	acceptable	outcomes?	More
specifically:	how	can	the	sponsor	of	a	project	that	aims	to	develop
superintelligence	ensure	that	the	project,	if	successful,	produces	a
superintelligence	that	would	realize	the	sponsor’s	goals?	We	can	divide	this
control	problem	into	two	parts.	One	part	is	generic,	the	other	unique	to	the

present	context.
This	first	part—what	we	shall	call	the	
first	principal–agent	problem
—arises
whenever	some	human	entity	(“the	principal”)	appoints	another	(“the	agent”)	to
act	in	the	former’s	interest.	This	type	of	agency	problem	has	been	extensively
studied	by	economists.
1
	It	becomes	relevant	to	our	present	concern	if	the	people

creating	an	AI	are	distinct	from	the	people	commissioning	its	creation.	The
project’s	owner	or	sponsor	(which	could	be	anything	ranging	from	a	single
individual	to	humanity	as	a	whole)	might	then	worry	that	the	scientists	and
programmers	
implementing	the	project	will	not	act	in	the	sponsor’s	best
interest.
2
	Although	this	type	of	agency	problem	could	pose	significant	challenges
to	a	project	sponsor,	it	is	not	a	problem	unique	to	intelligence	amplification	or	AI
projects.	Principal–agent	problems	of	this	sort	are	ubiquitous	in	human
economic	and	political	interactions,	and	there	are	many	ways	of	dealing	with
them.	For	instance,	the	risk	that	a	disloyal	employee	will	sabotage	or	subvert	the
project	could	be	minimized	through	careful	background	checks	of	key	personnel,
the	use	of	a	good	version-control	system	for	software	projects,	and	intensive
oversight	from	multiple	independent	monitors	and	auditors.	Of	course,	such
safeguards	come	at	a	cost—they	expand	staffing	needs,	complicate	personnel

selection,	hinder	creativity,	and	stifle	independent	and	critical	thought,	all	of
which	could	reduce	the	pace	of	progress.	These	costs	could	be	significant,
especially	for	projects	that	have	tight	budgets,	or	that	perceive	themselves	to	be
in	a	close	race	in	a	winner-takes-all	competition.	In	such	situations,	projects	may
skimp	on	procedural	safeguards,	creating	possibilities	for	potentially
catastrophic	principal–agent	failures	of	the	first	type.
The	other	part	of	the	control	problem	is	more	specific	to	the	context	of	an

intelligence	explosion.	This	is	the	problem	that	a	project	faces	when	it	seeks	to
ensure	that	the	superintelligence	it	is	building	will	not	harm	the	project’s
interests.	This	part,	too,	can	be	thought	of	as	a	principal–agent	problem—the
second	principal–agent	problem
.	In	this	case,	the	agent	is	not	a	human	agent
operating	on	behalf	of	a	human	principal.	Instead,	the	agent	is	the
superintelligent	system.	Whereas	the	first	principal–agent	problem	occurs
mainly	in	the	development	phase,	the	second	agency	problem	threatens	to	cause

trouble	mainly	in	the	superintelligence’s	operational	phase.
Exhibit	1	Two	agency	problems
The	first	principal–agent	problem
•	Human	v.	Human	(Sponsor	→	Developer)
•	Occurs	mainly	in	developmental	phase
•	Standard	management	techniques	apply

The	second	principal–agent	problem	(“the	control	problem”)
•	Human	v.	Superintelligence	(Project	→	System)
•	Occurs	mainly	in	operational	(and	bootstrap)	phase
•	New	techniques	needed
This	second	agency	problem	poses	an	unprecedented	challenge.	Solving	it
will	require	new	techniques.	We	have	already	considered	some	of	the	difficulties
involved.	We	saw,	in	particular,	that	the	treacherous	turn	syndrome	vitiates	what
might	otherwise	have	seemed	like	a	promising	set	of	methods,	ones	that	rely	on
observing	an	AI’s	behavior	in	its	developmental	phase	and	allowing	the	AI	to
graduate	from	a	secure	environment	once	it	has	accumulated	a	track	record	of
taking	appropriate	actions.	Other	technologies	can	often	be	safety-tested	in	the
laboratory	or	in	small	field	studies,	and	then	rolled	out	gradually	with	a
possibility	
of	halting	deployment	if	unexpected	troubles	arise.	Their	performance
in	preliminary	trials	helps	us	make	reasonable	inferences	about	their	future

reliability.	Such	behavioral	methods	are	defeated	in	the	case	of	superintelligence
because	of	the	strategic	planning	ability	of	general	intelligence.
3
Since	the	behavioral	approach	is	unavailing,	we	must	look	for	alternatives.	We
can	divide	potential	control	methods	into	two	broad	classes:	
capability	control
methods
,	which	aim	to	control	what	the	superintelligence	can	do;	and	
motivation
selection	methods
,	which	aim	to	control	what	it	wants	to	do.	Some	of	the
methods	are	compatible	while	others	represent	mutually	exclusive	alternatives.

3
methods
In	this	chapter	we	canvass	the	main	options.	(In	the	next	four	chapters,	we	will
explore	some	of	the	key	issues	at	greater	depth.)
It	is	important	to	realize	that	some	control	method	(or	combination	of
methods)	must	be	implemented	
before
	the	system	becomes	superintelligent.	It
cannot	be	done	after	the	system	has	obtained	a	decisive	strategic	advantage.	The
need	to	solve	the	control	problem	in	advance—and	to	implement	the	solution
successfully	in	the	very	first	system	to	attain	superintelligence—is	part	of	what

before
makes	achieving	a	controlled	detonation	such	a	daunting	challenge.

Capability	control	methods	seek	to	prevent	undesirable	outcomes	by	limiting
what	the	superintelligence	can	do.	This	might	involve	placing	the
superintelligence	in	an	environment	in	which	it	is	unable	to	cause	harm	(
boxing
methods
)	or	in	which	there	are	strongly	convergent	instrumental	reasons	not	to
engage	in	harmful	behavior	(
incentive	methods
).	It	might	also	involve	limiting
the	internal	capacities	of	the	superintelligence	(
stunting
).	In	addition,	capability
control	methods	might	involve	the	use	of	mechanisms	to	automatically	detect
and	react	to	various	kinds	of	containment	failure	or	attempted	transgression
(
tripwires
).
	
Boxing	methods	can	be	subdivided	into	physical	and	informational	containment
methods.
Physical	containment	aims	to	confine	the	system	to	a	“box,”	
i.e.
	to	prevent	the
system	from	interacting	with	the	external	world	otherwise	than	via	specific
restricted	output	channels.	The	boxed	system	would	not	have	access	to	physical

(
tripwires
).
	
methods.
i.e.
manipulators	outside	of	the	box.	Removing	manipulators	(such	as	robotic	arms)
from	inside	the	box	as	well	would	prevent	the	system	from	constructing	physical
devices	that	could	breach	the	confinement.
For	extra	security,	the	system	should	be	placed	in	a	metal	mesh	to	prevent	it
from	transmitting	radio	signals,	which	might	otherwise	offer	a	means	of
manipulating	electronic	objects	such	as	radio	receivers	in	the	environment.	Note,
by	the	way,	how	easy	it	might	have	been	to	overlook	the	need	for	this	precaution.

One	
might	naively	have	assumed	that	an	agent	without	a	manipulator	could	not
affect	the	external	world.	But	it	might	be	possible	for	a	machine	intelligence	to
generate	radio	waves	even	when	it	lacks	access	to	external	manipulators,	simply
“by	thinking”	(that	is,	by	shuffling	the	electrons	in	its	circuitry	in	particular
patterns).
4
	Once	pointed	out,	this	vulnerability	can	be	patched	by	enclosing	the
system	in	a	Faraday	cage—but	we	are	left	wondering	how	many	other	similarly
subtle	vulnerabilities	might	exist.	Each	time	we	hear	of	a	seemingly	foolproof

4
security	design	that	has	an	unexpected	flaw,	we	should	prick	up	our	ears.	These
occasions	grace	us	with	the	opportunity	to	abandon	a	life	of	overconfidence	and
resolve	to	become	better	Bayesians.
5

Physical	confinement	has	several	advantages.	It	is	easy	to	implement.	It	can
be	applied	to	many	machine	intelligence	architectures,	even	ones	that	were	not
initially	designed	with	safety	as	an	objective.	It	can	be	used	in	combination	with
most	other	control	methods.	And	it	seems	unlikely	to	go	wrong	by	backfiring:
that	is,	while	it	might	fail	to	ensure	safety,	it	is	unlikely	to	cause	a	catastrophe
that	would	not	otherwise	have	occurred.	The	main	disadvantage	with	physical
confinement	is	that	it	reduces	the	functionality	of	the	superintelligence	(an	issue
to	which	we	will	return	in	
Chapter	10
).	Another	concern	is	that	it	might
encourage	a	false	sense	of	security,	though	this	is	avoidable	if	we	regard	physical
confinement	as	icing	on	the	cake	rather	than	the	main	substance	of	our
precautions.
Informational	containment	aims	to	restrict	what	information	is	allowed	to	exit
the	box.	We	have	already	seen	how	a	superintelligence	that	has	access	to	an

Chapter	10
Internet	port,	such	that	it	can	message	outside	entities,	is	potentially	unsafe:	even
if	it	starts	out	without	access	to	physical	actuators,	it	may	use	its	information
output	channel	to	get	human	beings	to	do	its	bidding.	An	obvious	informational
containment	method,	therefore,	is	to	bar	the	system	from	accessing
communications	networks.
Unplugging	an	AI	from	the	Internet	does	not,	however,	ensure	safety	if	there
are	one	or	more	humans	serving	as	the	system’s	gatekeepers	and	remaining	in

communication	with	it.	During	the	development	phase,	the	programmers	serve
as	gatekeepers;	during	the	operational	phase,	the	system’s	owner	might	play	this
role.	If	the	AI	can	persuade	or	trick	a	gatekeeper	to	let	it	out	of	the	box,	resulting
in	its	gaining	access	either	to	the	Internet	or	directly	to	physical	manipulators,
then	the	boxing	strategy	has	failed.	Human	beings	are	not	secure	systems,
especially	not	when	pitched	against	a	superintelligent	schemer	and	persuader.
6

6
Although	routing	the	outbound	communications	of	a	machine	intelligence	to	a
human	gatekeeper	does	not	guarantee	safety,	it	could	serve	as	an	auxiliary	safety
measure.	There	are	precautions	that	might	reduce	the	risk	of	the	AI	escaping
through	the	gatekeeper.	The	AI	could	be	allowed	to	communicate	only	via	a	low-
bandwidth	teletype	interface.	This	would	make	it	slightly	harder	to
psychologically	manipulate	the	gatekeepers	(precluding	the	use	of	emotive
sound	and	imagery	and	tricks	such	as	flashing	hypnotic	patterns	to	induce	a

trancelike	state).
7
	Going	further,	the	content	(not	just	the	mode	of	presentation)
of	the	output	could	be	restricted;	so	that,	for	example,	the	AI	would	be
constrained	to	giving	
single-bit	answers	to	yes/no	questions.	The	idea	of	building

a	question-answering	system	will	be	examined	further	in	the	next	chapter.
The	limiting	case	of	the	boxing	approach	would	be	a	system	kept	in	complete
physical	and	informational	isolation.	Even	if	achievable,	however,	such	an
isolated	system	would	be	rather	useless	since	it	would	have	no	effect	on	the
external	world.	It	might	perhaps	be	thought	that	some	scientific	purpose	could	be
served	by	creating	a	superintelligence	and	keeping	it	in	isolation:	by	studying	a
self-contained	model	system,	one	could	learn	about	its	internal	dynamics	and	its
behavior	patterns.	But	this	would	be	an	error.	As	soon	as	the	system	is	observed,
it	ceases	to	be	informationally	isolated.
8
	Information	can	be	transmitted	not	only
via	messages	that	an	AI	sends	out	through	a	designated	“output	channel”	but
also	via	any	observation	an	outsider	makes	of	some	causal	consequence	of	the
AI’s	workings,	direct	or	indirect—its	power	consumption,	its	CPU	and	memory

8
usage,	its	computational	states,	or	indeed	any	traces	left	behind	after	it	has	been
shut	down.	An	AI	anticipating	that	it	might	be	observed	in	any	of	these	ways
could	strategically	adopt	behaviors	designed	to	influence	the	hypothesized
observers.	For	example,	the	AI	could	“malfunction”	in	whichever	way	would
most	increase	the	probability	that	it	will	ultimately	be	reactivated	or	that	another
superintelligence	will	be	created	with	goals	similar	to	its	own.

Incentive	methods	involve	placing	an	agent	in	an	environment	where	it	finds
instrumental	reasons	to	act	in	ways	that	promote	the	principal’s	interests.
Consider	a	billionaire	who	uses	her	fortune	to	set	up	a	large	charitable
foundation.	Once	created,	the	foundation	may	be	powerful—more	powerful	than
most	individuals,	including	its	founder,	who	might	have	donated	most	of	her
wealth.	To	control	the	foundation,	the	founder	lays	down	its	purpose	in	articles
of	incorporation	and	bylaws,	and	appoints	a	board	of	directors	sympathetic	to

her	cause.	These	measures	constitute	a	form	of	motivation	selection,	since	they
aim	to	shape	foundation’s	preferences.	But	even	if	such	attempts	to	customize
the	organizational	internals	fail,	the	foundation’s	behavior	would	remain
circumscribed	by	its	social	and	legal	milieu.	The	foundation	would	have	an
incentive	to	obey	the	law,	for	example,	lest	it	be	shut	down	or	fined.	It	would
have	an	incentive	to	offer	its	employees	acceptable	pay	and	working	conditions,
and	to	satisfy	external	stakeholders.	Whatever	its	final	goals,	the	foundation	thus

has	instrumental	reasons	to	conform	its	behavior	to	various	social	norms.
Might	one	not	hope	that	a	machine	superintelligence	would	likewise	be
hemmed	in	by	the	need	to	get	along	with	the	other	actors	with	which	it	shares	the
stage?	Though	this	might	seem	like	a	straightforward	way	of	dealing	with	the
control	problem,	it	is	not	free	of	obstacles.	In	particular,	it	presupposes	a	balance
of	power:	legal	or	economic	sanctions	cannot	restrain	an	agent	that	has	a
decisive	strategic	advantage.	Social	integration	can	therefore	not	be	relied	upon
as	a	control	method	in	fast	or	medium	takeoff	scenarios	that	feature	a	winner-
takes-all	dynamic.
How	about	in	multipolar	scenarios,	wherein	several	agencies	emerge	post-
transition	with	comparable	levels	of	capability?	Unless	the	default	trajectory	is
one	with	a	slow	takeoff,	achieving	such	a	power	distribution	may	require	a
carefully	orchestrated	ascent	wherein	different	projects	are	deliberately
synchronized	to	prevent	any	one	of	them	from	ever	pulling	ahead	of	the	pack.

9
Even	if	a	multipolar	outcome	does	result,	social	integration	is	not	a	perfect
solution.	By	relying	on	social	integration	to	solve	the	control	problem,	the
principal	risks	sacrificing	a	large	portion	of	her	potential	influence.	Although	a
balance	of	power	might	prevent	a	particular	AI	from	taking	over	the	world,	that
AI	will	still	have	
some
	power	to	affect	outcomes;	and	if	that	power	is	used	to
promote	some	arbitrary	final	goal—maximizing	paperclip	production—it	is

9
some
probably	not	being	used	to	advance	the	interests	of	the	principal.	Imagine	our
billionaire	endowing	a	new	foundation	and	allowing	its	mission	to	be	set	by	a
random	word	generator:	not	a	species-level	threat,	but	surely	a	wasted
opportunity.
A	related	but	importantly	different	idea	is	that	an	AI,	by	interacting	freely	in
society,	would	acquire	new	human-friendly	final	goals.	Some	such	process	of
socialization	takes	place	in	us	humans.	We	internalize	norms	and	ideologies,	and

we	come	to	value	other	individuals	for	their	own	sakes	in	consequence	of	our
experiences	with	them.	But	this	is	not	a	universal	dynamic	present	in	all
intelligent	systems.	As	discussed	earlier,	many	types	of	agent	in	many	situations
will	have	convergent	instrumental	reasons	
not
	to	permit	changes	in	their	final
goals.	(One	might	consider	trying	to	design	a	special	kind	of	goal	system	that
can	acquire	final	goals	in	the	manner	that	humans	do;	but	this	would	not	count	as

not
a	capability	control	method.	We	will	discuss	some	possible	methods	of	value
acquisition	in	
Chapter	12
.)

Capability	control	through	social	integration	and	balance	of	power	relies	upon
diffuse	social	forces	rewarding	and	penalizing	the	AI.	Another	type	of	incentive
method	would	involve	creating	a	setup	wherein	the	AI	can	be	rewarded	and
penalized	by	the	project	that	creates	it,	and	thereby	incentivized	to	act	in	the
interests	of	the	principal.	To	achieve	this,	the	AI	would	be	placed	in	a
surveillance	context	that	allows	its	behavior	to	be	monitored	and	evaluated,
either	manually	or	by	some	automated	process.	The	AI	would	know	that	a
positive	evaluation	would	bring	about	some	outcome	it	desires	and	that	a
negative	evaluation	would	fail	to	do	so.	Theoretically,	the	reward	could	be	the
fulfillment	of	some	convergent	instrumental	goal.	Without	knowing	anything
specific	about	the	AI’s	motivation	system,	however,	it	might	be	difficult	to
calibrate	the	reward	mechanism.	For	example,	we	could	end	up	with	an	AI	that
would	be	willing	to	take	extreme	risks	for	the	sake	of	a	small	chance	of

eventually	obtaining	control	of	a	large	share	of	the	universe.	It	could	be
expensive	to	offer	the	AI	a	higher	expected	utility	as	reward	for	cooperation	than
the	AI	could	hope	to	achieve	by	defecting	and	trying	to	escape.
10
A	better	alternative	might	be	to	combine	the	incentive	method	with	the	use	of
motivation	selection	to	give	the	AI	a	final	goal	that	makes	it	easier	to	control.
Suppose	that	an	AI	were	designed	to	have	as	its	final	goal	that	a	particular	
red
button	inside	a	command	bunker	never	be	pressed.	Since	the	pressing	of	the

10
red
button	is	disvalued	intrinsically	and	not	because	of	its	causal	consequences,	the
button	could	be	completely	inert:	it	could	be	made	of	Play-Doh.	Furthermore,	it
is	irrelevant	whether	the	AI	can	ever	know	whether	the	button	had	been	pressed.
What	is	essential	is	that	the	AI	believes	that	the	button	will	more	likely	remain
unpressed	if	the	AI	continuously	acts	in	the	principal’s	interest	than	if	it	rebels.
Refinements	to	this	setup	are	possible.	Instead	of	trying	to	endow	an	AI	with	a

final	goal	that	refers	to	a	physical	button,	one	could	build	an	AI	that	places	final
value	on	receiving	a	stream	of	“cryptographic	reward	tokens.”
11
	These	would	be
sequences	of	numbers	serving	as	keys	to	ciphers	that	would	have	been	generated
before	the	AI	was	created	and	that	would	have	been	built	into	its	motivation
system.
12
	These	special	number	sequences	would	be	extremely	desirable	to	the
AI,	constituting	a	special	kind	of	reward	token	that	the	AI	could	not	attain
though	wireheading.
13

11
system.
12
13
	The	keys	would	be	stored	in	a	secure	location	where	they
could	be	quickly	destroyed	if	the	AI	ever	made	an	attempt	to	seize	them.	So	long
as	the	AI	cooperates,	the	keys	are	doled	out	at	a	steady	rate.
The	AI	could	then	be	given	a	bounded	utility	function,	and	matters	could	be

arranged	so	that	the	AI	would	get	(for	instance)	99%	of	its	maximum	utility	from
the	first	reward	token;	99%	of	its	remaining	utility	potential	from	the	second
reward	token;	and	so	on.	By	keeping	the	“discount	rate”	high,	and	by
transmitting	reward	tokens	to	the	AI	at	a	reasonably	high	rate,	one	would
dissuade	the	AI	from	attempting	to	carry	out	long-term	plans	that	would	increase
the	expected	number	of	reward	tokens	in	the	distal	future	at	the	expense	of	some
risk	that	its	reward	stream	will	be	cut	short	or	that	the	next	reward	token	will	be
cancelled	and	destroyed.	If	the	AI	always	believes	that	defecting	(i.e.	doing
anything	other	than	its	best	to	look	after	the	principal’s	interests)	would	incur	at
least	a	2%	risk	of	forfeiting	the	next	reward	token,	whereas	cooperating	would
reduce	the	risk	of	forfeiting	the	next	reward	token	to	below	1%,	then	a	utility-
maximizing	AI	would	always	cooperate.
What	might	go	wrong	with	such	an	incentive	scheme?	One	possibility	is	that

the	AI	will	not	trust	the	human	operator	to	deliver	the	promised	rewards.	The
track	record	of	human	reliability	is	something	other	than	a	straight	line	of
unerring	perfection.	The	AI	would	reasonably	worry	that	the	operator	will
change	his	mind,	or	raise	the	performance	bar,	or	fail	to	recognize	that	the	AI	has
done	its	part.	The	AI	would	also	worry	about	the	operator	becoming
incapacitated.	The	combined	risk	of	such	failures	might	exceed	the	risk	of
attempting	to	seize	control	of	the	reward	mechanism.	Even	a	boxed	AI

possessing	the	panoply	of	superpowers	is	a	strong	force.	(For	an	AI	that	is	
not
boxed	to	begin	with,	hijacking	the	human-governed	reward	mechanism	may	be
like	taking	candy	from	a	baby.)
Another	problem	with	the	incentive	scheme	is	that	it	presupposes	that	we	can
tell	whether	the	outcomes	produced	by	the	AI	are	in	our	interest.	As	later
chapters	will	elaborate,	this	presupposition	is	not	innocuous.
A	full	assessment	of	the	feasibility	of	incentive	methods	would	also	have	to
take	into	account	a	range	of	other	factors,	including	some	esoteric	considerations
that

not
that	
might	conceivably	make	such	methods	more	viable	than	a	preliminary
analysis	would	suggest.	In	particular,	the	AI	may	face	ineliminable	indexical
uncertainty	if	it	could	not	be	sure	that	it	does	not	inhabit	a	computer	simulation
(as	opposed	to	“basement-level,”	non-simulated	physical	reality),	and	this
epistemic	predicament	may	radically	influence	the	AI’s	deliberations	(see	
Box
8
).

Box	8	Anthropic	capture
	
The	AI	might	assign	a	substantial	probability	to	its	simulation	hypothesis,	the
hypothesis	that	it	is	living	in	a	computer	simulation.	Even	today,	many	AIs
inhabit	simulated	worlds—worlds	consisting	of	geometric	line	drawings,	texts,
chess	games,	or	simple	virtual	realities,	and	in	which	the	laws	of	physics	deviate
sharply	from	the	laws	of	physics	that	we	believe	govern	the	world	of	our	own
experience.	Richer	and	more	complicated	virtual	worlds	will	become	feasible
with	improvements	in	programming	techniques	and	computing	power.	A	mature
superintelligence	could	create	virtual	worlds	that	appear	to	its	inhabitants	much
the	same	as	our	world	appears	to	us.	It	might	create	vast	numbers	of	such
worlds,	running	the	same	simulation	many	times	or	with	small	variations.	The
inhabitants	would	not	necessarily	be	able	to	tell	whether	their	world	is	simulated
or	not;	but	if	they	are	intelligent	enough	they	could	consider	the	possibility	and

assign	it	some	probability.	In	light	of	the	simulation	argument	(a	full	discussion
of	which	is	beyond	the	scope	of	this	book)	that	probability	could	be
substantial.
14
This	predicament	especially	afflicts	relatively	early-stage	superintelligences,
ones	that	have	not	yet	expanded	to	take	advantage	of	the	cosmic	endowment.	An
early-stage	superintelligence,	which	uses	only	a	small	fraction	of	the	resources
of	a	single	planet,	would	be	much	less	expensive	to	simulate	than	a	mature

14
intergalactic	superintelligence.	Potential	simulators—that	is,	other	more	mature
civilizations—would	be	able	to	run	great	numbers	of	simulations	of	such	early-
stage	AIs	even	by	dedicating	a	minute	fraction	of	their	computational	resources
to	that	purpose.	If	at	least	some	(non-trivial	fraction)	of	these	mature
superintelligent	civilizations	choose	to	use	this	ability,	early-stage	AIs	should
assign	a	substantial	probability	to	being	in	a	simulation.
How	an	AI	would	be	affected	by	the	simulation	hypothesis	depends	on	its
values.
15

values.
15
	Consider	first	an	AI	that	has	a	“resource-insatiable”	final	goal,	such	as
the	goal	of	maximizing	a	utility	function	that	is	linear	in	the	number	of
paperclips	caused	by	the	AI	to	exist—twice	as	many	paperclips,	twice	as	good.
Such	an	AI	might	care	less	about	the	simulation	hypothesis,	on	grounds	that	its
ability	to	influence	how	many	paperclips	are	produced	looks	much	more	limited
if	the	AI	is	in	a	simulation	than	if	it	is	in	basement-level	physical	reality.	The	AI
might	accordingly	behave	much	
as	if
	it	were	certain	that	the	simulation

hypothesis	is	false,	even	if	it	assigned	a	fairly	high	probability	to	the	simulation
hypothesis	being	true.
16
	The	AI	would	reason	that	it	does	not	much	matter	how	it
acts	if	the	simulation	
hypothesis	is	true,	whereas	its	actions	make	a	great
difference	if	the	hypothesis	is	false;	so	it	would	select	actions	mainly	by
reference	to	how	good	they	are	conditional	on	the	simulation	hypothesis	being
false.
By	contrast,	consider	an	AI	that	has	a	more	modest	final	goal,	one	that	could
be	satisfied	with	a	small	amount	of	resources,	such	as	the	goal	of	receiving	some
pre-produced	cryptographic	reward	tokens,	or	the	goal	of	causing	the	existence
of	forty-five	virtual	paperclips.	Such	an	AI	should	
not
	discount	those	possible
worlds	in	which	it	inhabits	a	simulation.	A	substantial	portion	of	the	AI’s	total
expected	utility	might	derive	from	those	possible	worlds.	The	decision-making
of	an	AI	with	goals	that	are	easily	resource-satiable	may	therefore—if	it	assigns

not
a	high	probability	to	the	simulation	hypothesis—be	dominated	by	considerations
about	which	actions	would	produce	the	best	result	if	its	perceived	world	is	a
simulation.	Such	an	AI	(even	if	it	is,	in	fact,	not	in	a	simulation)	might	therefore
be	heavily	influenced	by	its	beliefs	about	which	behaviors	would	be	rewarded	in
a	simulation.
In	particular,	if	an	AI	with	resource-satiable	final	goals	believes	that	in	most
simulated	worlds	that	match	its	observations	it	will	be	rewarded	if	it	cooperates

(but	not	if	it	attempts	to	escape	its	box	or	contravene	the	interests	of	its	creator)
then	it	may	choose	to	cooperate.	We	could	therefore	find	that	even	an	AI	with	a
decisive	strategic	advantage,	one	that	could	in	fact	realize	its	final	goals	to	a
greater	extent	by	taking	over	the	world	than	by	refraining	from	doing	so,	would
nevertheless	balk	at	doing	so.
Thus	Conscience	does	make	Cowards	of	us	all,
And	thus	the	Native	hue	of	Resolution
Is	sicklied	o’er,	with	the	pale	cast	of	Thought,

And	enterprises	of	great	pith	and	moment,
With	this	regard	their	Currents	turn	away,
And	lose	the	name	of	Action.

(Shakespeare,	
Hamlet
,	Act	iii.	Sc.	1)
A	mere	line	in	the	sand,	backed	by	the	clout	of	a	nonexistent	simulator,	could
prove	a	stronger	restraint	than	a	two-foot-thick	solid	steel	door.
17
	
Stunting
	
Another	possible	capability	control	method	is	to	limit	the	system’s	intellectual
faculties	or	its	access	to	information.	This	might	be	done	by	running	the	AI	on
hardware	that	is	slow	or	short	on	memory.	In	the	case	of	a	boxed	system,
information	inflow	could	also	be	restricted.
Stunting	an	AI	in	these	ways	would	limit	its	usefulness.	The	method	thus
faces	a	dilemma:	too	little	stunting,	and	the	AI	might	have	the	wit	to	figure	
out
some	way	to	make	itself	more	intelligent	(and	thence	to	world	domination);	too
much,	and	the	AI	is	just	another	piece	of	dumb	software.	A	radically	stunted	AI
is	certainly	safe	but	does	not	solve	the	problem	of	how	to	achieve	a	controlled
detonation:	an	intelligence	explosion	would	remain	possible	and	would	simply

out
be	triggered	by	some	other	system	instead,	perhaps	at	a	slightly	later	date.
One	might	think	it	would	be	safe	to	build	a	superintelligence	provided	it	is
only	given	data	about	some	narrow	domain	of	facts.	For	example,	one	might
build	an	AI	that	lacks	sensors	and	that	has	preloaded	into	its	memory	only	facts
about	petroleum	engineering	or	peptide	chemistry.	But	if	the	AI	is
superintelligent—if	it	is	has	a	superhuman	level	of	
general
	intelligence—such
data	deprivation	does	not	guarantee	safety.

general
There	are	several	reasons	for	this.	First,	the	notion	of	information	being
“about”	a	certain	topic	is	generally	problematic.	Any	piece	of	information	can	in
principle	be	relevant	to	any	topic	whatsoever,	depending	on	the	background
information	of	a	reasoner.
18
	Furthermore,	a	given	data	set	contains	information
not	only	about	the	domain	from	which	the	data	was	collected	but	also	about
various	circumstantial	facts.	A	shrewd	mind	looking	over	a	knowledge	base	that

18
is	nominally	about	peptide	chemistry	might	infer	things	about	a	wide	range	of
topics.	The	fact	that	certain	information	is	included	and	other	information	is	not

could	tell	an	AI	something	about	the	state	of	human	science,	the	methods	and
instruments	available	to	study	peptides,	the	fabrication	technologies	used	to
make	these	instruments,	and	the	nature	of	the	brains	and	societies	that	conceived
the	studies	and	the	instruments.	It	might	be	that	a	
	could
correctly	surmise	a	great	deal	from	what	seem,	to	dull-witted	human	minds,
meager	scraps	of	evidence.	Even	without	any	designated	knowledge	base	at	all,	a
sufficiently	superior	mind	might	be	able	to	learn	much	by	simply	introspecting
on	the	workings	of	its	own	psyche—the	design	choices	reflected	in	its	source
code,	the	physical	characteristics	of	its	circuitry.
19
	Perhaps	a	superintelligence
could	even	deduce	much	about	the	likely	properties	of	the	world	
a	priori
(combining	logical	inference	with	a	probability	prior	biased	toward	simpler
worlds,	and	a	few	elementary	facts	implied	by	the	superintelligence’s	existence
as	a	reasoning	system).	It	might	imagine	the	consequences	of	different	possible

19
a	priori
laws	of	physics:	what	kind	of	planets	would	form,	what	kind	of	intelligent	life
would	evolve,	what	kind	of	societies	would	develop,	what	kind	of	methods	to
solve	the	control	problem	would	be	attempted,	how	those	methods	could	be
defeated.
20
For	these	reasons	it	would	be	imprudent	to	rely	on	information	deprivation	as
the	main	check	on	a	superintelligence’s	power.	Some	forms	of	stunting	however,
including	limiting	hardware	capacity	and	preventing	the	development	of	specific
cognitive	faculties,	might	find	use	as	auxiliary	control	methods	in	the

defeated.
20
developmental	phase.	They	could	be	used,	for	instance,	to	regulate	the	rate	of
capacity	gain	in	an	intermediary	AI	and	to	allow	tests	to	be	run	at	different	levels
of	capability	short	of	full-throttled	superintelligence.
Tripwires
	
A	tripwire	is	a	mechanism	that	performs	diagnostic	tests	on	the	system	(possibly
without	its	knowledge)	and	effects	a	shutdown	if	it	detects	signs	of	dangerous
activity.	Programmers	can	then	scrutinize	recordings	of	the	suspicious	activity	at
their	leisure,	and	make	changes	to	the	AI’s	code	base	or	environment	before

Tripwires
	
restarting	it.	If	they	find	that	the	problem	is	sufficiently	deep-rooted,	they	could
decide	to	abort	the	entire	project.
Tripwires	differ	from	incentive	methods	in	that	they	do	not	rely	on	the	system
being	aware	of	the	consequences	of	engaging	in	forbidden	activities.	Tripwires

are	
more	closely	related	to	stunting	methods.	Like	stunting,	tripwires	could	be
used	as	a	temporary	safeguard,	providing	a	degree	of	protection	during	the
development	phase.	In	principle,	tripwires	can	also	be	used	during	the
operational	phase,	particularly	for	a	boxed	system.	However,	the	ability	of
tripwires	to	constrain	a	full-fledged	superintelligence	must	remain	very	much	in
doubt,	since	it	would	be	hard	for	us	to	assure	ourselves	that	such	an	agent	could
not	find	ways	to	subvert	any	tripwire	devised	by	the	human	intellect.
Table	9	
Different	kinds	of	tripwires
	
Type	of
monitoring
Description	and	examples
Behavior
Detectors	could	be	placed	around	a	boxed	AI	to	detect	attempts	to
breach	the	containment.	For	example,	detectors	could	intercept
attempts	at	radio	communication	or	at	accessing	internal
computational	resources	intended	to	be	off	limits.	An	“Ethernet	port
of	Eden”	could	be	installed:	an	apparent	connection	to	the	internet
that	leads	to	a	shutdown	switch.
Ability

Table	9	
	
Type	of
Behavior
Ability
Automated	capability	testing	could	be	performed	at	frequent
intervals	to	determine	the	AI’s	skill	in	various	domains.	If	either	the
rate	of	improvement	is	unexpectedly	high	or	the	AI	attains	a	level	of
competence	that	brings	it	close	to	the	potential	danger	zone,	the
system	could	be	slowed	down	or	paused,	in	order	to	allow	the
programmers	to	monitor	further	developments	more	closely.
The	AI’s	internal	processes	could	be	monitored	for	suspicious
activity.	For	example,	any	change	to	the	AI’s	representation	of	its

final	values	might	trigger	an	automatic	shutdown	and	review.	The
AI’s	resource	use	and	algorithmic	methods	may	also	be	also	be
monitored	for	unexpected	patterns.

Content
More	ambitiously,	the	AI’s	beliefs,	plans,	subgoals,	and
justifications	may	be	continuously	scrutinized,	to	ensure	they
conform	to	the	programmers’	expectations.	Such	monitoring	of	the
AI’s	intentional	states	would	enable	a	particularly	powerful	form	of
content	monitoring:	the	scanning	of	the	AI’s	cognitive	processes	for
a	
conception	of	deception
—that	vulnerable	moment	when	an
unfriendly	AI	first	forms	the	intention	to	conceal	its	true
intentions.
21
Content	monitoring	that	requires	that	the	AI’s	intentional	states	be
transparent	to	the	programmers	or	to	an	automatic	monitoring
mechanism	may	not	be	feasible	for	all	kinds	of	AI	architectures.
(Some	neural	networks,	for	instance,	are	opaque,	as	they	represent
information	holistically	and	in	ways	that	do	not	necessarily	match
up	with	human	concepts.)	This	may	be	a	reason	to	avoid	using	such
architectures.
As	an	adjunct	safety	measure	in	the	development	phase,	though,	tripwires	are

21
important.	It	would	be	good	practice	for	the	AI’s	designers	to	codify	their
expectations	about	how	the	system	is	supposed	to	function,	and	then	to
circumscribe	the	AI	with	tripwires	that	would	be	activated	if	any	of	these
expectations	is	violated.	To	make	the	tests	more	stringent,	“honeypots”	could	be
strategically	placed	to	create	temptations	for	a	malfunctioning	AI	to	commit
some	easily	observable	violation.	For	instance,	if	an	AI	has	been	designed	in

such	a	way	that	it	is	supposed	not	to	want	to	access	the	internet,	a	fake	Ethernet
port	could	be	installed	(leading	to	an	automatic	shutdown	switch)	just	to	see	if
they	AI	tries	to	use	it.	(Some	examples	of	tripwires	are	given	in	
Table	9
.)
It	should	be	emphasized	that	the	value	of	a	tripwire	depends	not	only	on	the
mechanism	itself	but	also—critically—on	how	a	project	reacts	when	a	tripwire	is
triggered.	If	the	programmers	or	project	managers,	impatient	to	make	progress,
simply	switch	the	system	back	on	again—or	if	they	do	so	after	making	some

Table	9
.)
token	modification	to	prevent	the	tripwire	being	triggered	on	the	next	run—then
no	safety	has	been	gained	even	if	the	tripwire	itself	works	exactly	as	intended.

Motivation	selection	methods	seek	to	prevent	undesirable	outcomes	by	shaping
what	the	superintelligence	wants	to	do.	By	engineering	the	agent’s	motivation
system	and	its	final	goals,	these	methods	would	produce	a	superintelligence	that
would	not	
want
	to	exploit	a	decisive	strategic	advantage	in	a	harmful	way.	Since
a	superintelligent	agent	is	skilled	at	achieving	its	ends,	if	it	prefers	not	to	cause
harm	(in	some	appropriate	sense	of	“harm”)	then	it	would	tend	not	to	cause	harm
(in	that	sense	of	“harm”).
Motivation	selection	can	involve	explicitly	formulating	a	goal	or	set	of	rules
to	be	followed	(
direct	specification
)	or	setting	up	the	system	so	that	it	can
discover	an	appropriate	set	of	values	for	itself	by	reference	to	some	implicitly	or
indirectly	formulated	criterion	(
indirect	normativity
).	One	option	in	motivation
selection	is	to	try	to	build	the	system	so	that	it	would	have	modest,	non-
ambitious	goals	(
domesticity
).	An	alternative	to	creating	a	motivation	system

from	scratch	is	to	select	an	agent	that	already	has	an	acceptable	motivation
system	and	then	augment	that	agent’s	cognitive	powers	to	make	it
superintelligent,	while	ensuring	that	the	motivation	system	does	not	get
corrupted	in	the	process	(
augmentation
).	Let	us	look	at	these	in	turn.
	
Direct	specification	is	the	most	straightforward	approach	to	the	control	problem.
The	approach	comes	in	two	versions,	rule-based	and	consequentialist,	and

involves	trying	to	explicitly	define	a	set	of	rules	or	values	that	will	cause	even	a
free-roaming	superintelligent	AI	to	act	safely	and	beneficially.	Direct
specification,	however,	faces	what	may	be	insuperable	obstacles,	deriving	from
both	the	difficulties	in	determining	which	rules	or	values	we	would	wish	the	AI
to	be	guided	by	and	the	difficulties	in	expressing	those	rules	or	values	in
computer-readable	code.
The	traditional	illustration	of	the	direct	rule-based	approach	is	the	“three	laws

of	robotics”	concept,	formulated	by	science	fiction	author	Isaac	Asimov	in	a
short	story	published	in	1942.
22
	The	three	laws	were:	(1)	A	robot	may	not	injure
a	human	being	or,	through	inaction,	allow	a	human	being	to	come	to	harm;	(2)	A
robot	must	obey	any	orders	given	to	it	by	human	beings,	except	where	such

orders	would	conflict	with	the	First	Law;	(3)	A	robot	must	protect	its	own
existence	as	long	as	such	protection	does	not	conflict	with	the	First	or	Second
Law.	Embarrassingly	for	our	species,	Asimov’s	laws	remained	state-of-the-art
for	over	half	a	century:	this	despite	obvious	problems	with	the	approach,	some	of
which	are	explored	in	Asimov’s	own	writings	(Asimov	probably	having
formulated	the	laws	in	the	first	place	precisely	so	that	they	would	fail	in
interesting	ways,	providing	fertile	plot	complications	for	his	stories).
23
Bertrand	Russell,	who	spent	many	years	working	on	the	foundations	of
mathematics,	once	remarked	that	“everything	is	vague	to	a	degree	you	do	not
realize	till	you	have	tried	to	make	it	precise.”
24
	Russell’s	dictum	applies	in
spades	to	the	direct	specification	approach.	Consider,	for	example,	how	one
might	explicate	Asimov’s	first	law.	Does	it	mean	that	the	robot	should	minimize
the	probability	of	any	human	being	coming	to	harm?	In	that	case	the	other	laws

23
24
become	otiose	since	it	is	always	possible	for	the	AI	to	take	some	action	that
would	have	at	least	some	microscopic	effect	on	the	probability	of	a	human	being
coming	to	harm.	How	is	the	robot	to	balance	a	large	risk	of	a	few	humans
coming	to	harm	versus	a	small	risk	of	many	humans	being	harmed?	How	do	we
define	“harm”	anyway?	How	should	the	harm	of	physical	pain	be	weighed
against	the	harm	of	architectural	ugliness	or	social	injustice?	Is	a	sadist	harmed
if	he	is	prevented	from	tormenting	his	victim?	How	do	we	define	“human

being”?	Why	is	no	consideration	given	to	other	morally	considerable	beings,
such	as	sentient	nonhuman	animals	and	digital	minds?	The	more	one	ponders,
the	more	the	questions	proliferate.
Perhaps	the	closest	existing	analog	to	a	rule	set	that	could	govern	the	actions
of	a	superintelligence	operating	in	the	world	at	large	is	a	legal	system.	But	legal
systems	have	developed	through	a	long	process	of	trial	and	error,	and	they
regulate	relatively	slowly-changing	human	societies.	Laws	can	be	revised	when

necessary.	Most	importantly,	legal	systems	are	administered	by	judges	and	juries
who	generally	apply	a	measure	of	common	sense	and	human	decency	to	ignore
logically	possible	legal	interpretations	that	are	sufficiently	obviously	unwanted
and	unintended	by	the	lawgivers.	It	is	probably	humanly	impossible	to	explicitly
formulate	a	highly	complex	set	of	detailed	rules,	have	them	apply	across	a	highly
diverse	set	of	circumstances,	and	get	it	right	on	the	first	implementation.
25

25
Problems	for	the	direct	consequentialist	approach	are	similar	to	those	for	the
direct	rule-based	approach.	This	is	true	even	if	the	AI	is	intended	to	serve	some
apparently	simple	purpose	such	as	implementing	a	version	of	classical

utilitarianism.	For	instance,	the	goal	“Maximize	the	expectation	of	the	balance	of
pleasure	over	pain	in	the	world”	may	appear	simple.	Yet	expressing	it	in
computer	code	would	involve,	among	other	things,	specifying	how	to	recognize
pleasure	and	pain.	Doing	this	reliably	might	require	solving	an	array	of
persistent	problems	in	the	philosophy	of	mind—even	just	to	obtain	a	correct
account	expressed	in	a	natural	language,	an	account	which	would	then,
somehow,	have	to	be	translated	into	a	programming	language.
A	small	error	in	either	the	philosophical	account	or	its	translation	into	code
could	have	catastrophic	consequences.	Consider	an	AI	that	has	hedonism	as	its
final	goal,	and	which	would	therefore	like	to	tile	the	universe	with	“hedonium”
(matter	organized	in	a	configuration	that	is	optimal	for	the	generation	of
pleasurable	experience).	To	this	end,	the	AI	might	produce	computronium
(matter	organized	in	a	configuration	that	is	optimal	for	computation)	and	use	it	to

implement	digital	minds	in	states	of	euphoria.	In	order	to	maximize	efficiency,
the	AI	omits	from	the	implementation	any	mental	faculties	that	are	not	essential
for	the	experience	of	pleasure,	and	exploits	any	computational	shortcuts	that
according	to	its	definition	of	pleasure	do	not	vitiate	the	generation	of	pleasure.
For	instance,	the	AI	might	confine	its	simulation	to	reward	circuitry,	eliding
faculties	such	as	memory,	sensory	perception,	executive	function,	and	language;

it	might	simulate	minds	at	a	relatively	coarse-grained	level	of	functionality,
omitting	lower-level	neuronal	processes;	it	might	replace	commonly	repeated
computations	with	calls	to	a	lookup	table;	or	it	might	put	in	place	some
arrangement	whereby	multiple	minds	would	share	most	parts	of	their	underlying
computational	machinery	(their	“supervenience	bases”	in	philosophical
parlance).	Such	tricks	could	greatly	increase	the	quantity	of	pleasure	producible
with	a	given	amount	of	resources.	It	is	unclear	how	desirable	this	would	be.

Furthermore,	if	the	AI’s	criterion	for	determining	whether	a	physical	process
generates	pleasure	is	wrong,	then	the	AI’s	optimizations	might	throw	the	baby
out	with	the	bathwater:	discarding	something	which	is	inessential	according	to
the	AI’s	criterion	yet	essential	according	to	the	criteria	implicit	in	our	human
values.	The	universe	then	gets	filled	not	with	exultingly	heaving	hedonium	but
with	computational	processes	that	are	unconscious	and	completely	worthless—
the	equivalent	of	a	smiley-face	sticker	xeroxed	trillions	upon	trillions	of	times

and	plastered	across	the	galaxies.

One	special	type	of	final	goal	which	might	be	more	amenable	to	direct
specification	than	the	examples	given	above	is	the	goal	of	self-limitation.	While
it	seems	
extremely	difficult	to	specify	how	one	would	want	a	superintelligence
to	behave	in	the	world	
in	general
—since	this	would	require	us	to	account	for	all
the	trade-offs	in	all	the	situations	that	could	arise—it	might	be	feasible	to	specify
how	a	superintelligence	should	behave	in	one	particular	situation.	We	could
therefore	seek	to	motivate	the	system	to	confine	itself	to	acting	on	a	small	scale,
within	a	narrow	context,	and	through	a	limited	set	of	action	modes.	We	will	refer
to	this	approach	of	giving	the	AI	final	goals	aimed	at	limiting	the	scope	of	its
ambitions	and	activities	as	“domesticity.”
For	example,	one	could	try	to	design	an	AI	such	that	it	would	function	as	a
question-answering	device	(an	“oracle,”	to	anticipate	the	terminology	that	we
will	introduce	in	the	next	chapter).	Simply	giving	the	AI	the	final	goal	of

producing	maximally	accurate	answers	to	any	question	posed	to	it	would	be
unsafe—recall	the	“Riemann	hypothesis	catastrophe”	described	in	
Chapter	8
.
(Reflect,	also,	that	this	goal	would	incentivize	the	AI	to	take	actions	to	ensure
that	it	is	asked	easy	questions.)	To	achieve	domesticity,	one	might	try	to	define	a
final	goal	that	would	somehow	overcome	these	difficulties:	perhaps	a	goal	that
combined	the	desiderata	of	answering	questions	correctly	and	minimizing	the
AI’s	impact	on	the	world	except	whatever	impact	results	as	an	incidental

Chapter	8
.
consequence	of	giving	accurate	and	non-manipulative	answers	to	the	questions	it
is	asked.
26
The	direct	specification	of	such	a	domesticity	goal	is	more	likely	to	be
feasible	than	the	direct	specification	of	either	a	more	ambitious	goal	or	a
complete	rule	set	for	operating	in	an	open-ended	range	of	situations.	Significant
challenges	nonetheless	remain.	Care	would	have	to	be	taken,	for	instance,	in	the
definition	of	what	it	would	be	for	the	AI	to	“minimize	its	impact	on	the	world”

is	asked.
26
to	ensure	that	the	measure	of	the	AI’s	impact	coincides	with	our	own	standards
for	what	counts	as	a	large	or	a	small	impact.	A	bad	measure	would	lead	to	bad
trade-offs.	There	are	also	other	kinds	of	risk	associated	with	building	an	oracle,
which	we	will	discuss	later.
There	is	a	natural	fit	between	the	domesticity	approach	and	physical
containment.	One	would	try	to	“box”	an	AI	such	that	the	system	is	
unable
	to
escape	while	simultaneously	trying	to	shape	the	AI’s	motivation	system	such

that	it	would	be	
unwilling
	to	escape	even	if	it	found	a	way	to	do	so.	Other	things
equal,	the	existence	of	multiple	independent	safety	mechanisms	should	shorten
the	odds	of	success.
27
	
If	direct	specification	seems	hopeless,	we	might	instead	try	indirect	normativity.
The	basic	idea	is	that	rather	than	specifying	a	concrete	normative	standard
directly,	we	specify	a	process	for	deriving	a	standard.	We	then	build	the	system
so	that	it	is	motivated	to	carry	out	this	process	and	to	adopt	whatever	standard
the	process	arrives	at.
28
	For	example,	the	process	could	be	to	carry	out	an
investigation	into	the	empirical	question	of	what	some	suitably	idealized	version
of	us	would	prefer	the	AI	to	do.	The	final	goal	given	to	the	AI	in	this	example
could	be	something	along	the	lines	of	“achieve	that	which	we	would	have
wished	the	AI	to	achieve	if	we	had	thought	about	the	matter	long	and	hard.”
Further	explanation	of	indirect	normativity	will	have	to	await	
Chapter	13
.

28
Chapter	13
.
There,	we	will	revisit	the	idea	of	“extrapolating	our	volition”	and	explore
various	alterative	formulations.	Indirect	normativity	is	a	very	important	approach
to	motivation	selection.	Its	promise	lies	in	the	fact	that	it	could	let	us	offload	to
the	superintelligence	much	of	the	difficult	cognitive	work	required	to	carry	out	a
direct	specification	of	an	appropriate	final	goal.
	
The	last	motivation	selection	method	on	our	list	is	augmentation.	Here	the	idea	is
that	rather	than	attempting	to	design	a	motivation	system	
de	novo

de	novo
,	we	start	with
a	system	that	already	has	an	acceptable	motivation	system,	and	enhance	its
cognitive	faculties	to	make	it	superintelligent.	If	all	goes	well,	this	would	give	us
a	superintelligence	with	an	acceptable	motivation	system.
This	approach,	obviously,	is	unavailing	in	the	case	of	a	newly	created	seed	AI.
But	augmentation	is	a	potential	motivation	selection	method	for	other	paths	to
superintelligence,	including	brain	emulation,	biological	enhancement,	brain–

computer	interfaces,	and	networks	and	organizations,	where	there	is	a	possibility
of	building	out	the	system	from	a	normative	nucleus	(regular	human	beings)	that
already	contains	a	representation	of	human	value.
The	attractiveness	of	augmentation	may	increase	in	proportion	to	our	despair
at	the	other	approaches	to	the	control	problem.	Creating	a	motivation	system	for
a	seed	AI	that	remains	reliably	safe	and	beneficial	under	recursive	self-
improvement	even	as	the	system	grows	into	a	mature	superintelligence	is	a	tall
order,	especially	if	we	must	get	the	solution	right	on	the	first	attempt.	With
augmentation,	we	would	at	least	start	with	a	system	that	has	familiar	and	human-
like	motivations.
On	the	downside,	it	might	be	hard	to	ensure	that	a	complex,	evolved,	kludgy,
and	poorly	understood	motivation	system,	like	that	of	a	human	being,	will	not
get	corrupted	when	its	cognitive	engine	blasts	into	the	stratosphere.	As	discussed
earlier,	an	imperfect	brain	emulation	procedure	that	preserves	intellectual

functioning	may	not	preserve	all	facets	of	personality.	The	same	is	true	(though
perhaps	to	a	lesser	degree)	for	biological	enhancements	of	cognition,	which
might	subtly	affect	motivation,	and	for	collective	intelligence	enhancements	of
organizations	and	networks,	which	might	adversely	change	social	dynamics	(e.g.
in	ways	that	debase	the	collective’s	attitude	toward	outsiders	or	toward	its	own
constituents).	If	superintelligence	is	achieved	via	any	of	these	paths,	a	project

sponsor	would	find	guarantees	about	the	ultimate	motivations	of	the	mature
system	hard	to	come	by.	A	mathematically	well-specified	and	foundationally
elegant	AI	architecture	might—for	all	its	non-anthropomorphic	otherness—offer
greater	transparency,	perhaps	even	the	prospect	that	important	aspects	of	its
functionality	could	be	formally	verified.
In	the	end,	however	one	tallies	up	the	advantages	and	disadvantages	of
augmentation,	the	choice	as	to	whether	to	rely	on	it	might	be	forced.	If

superintelligence	is	first	achieved	along	the	artificial	intelligence	path,
augmentation	is	not	
applicable.	Conversely,	if	superintelligence	is	first	achieved
along	some	non-AI	path,	then	many	of	the	other	motivation	selection	methods
are	inapplicable.	Even	so,	views	on	how	likely	augmentation	would	be	to
succeed	do	have	strategic	relevance	insofar	as	we	have	opportunities	to	influence
which	technology	will	first	produce	superintelligence.
Synopsis

A	quick	synopsis	might	be	called	for	before	we	close	this	chapter.	We
distinguished	two	broad	classes	of	methods	for	dealing	with	the	agency	problem
at	the	heart	of	AI	safety:	capability	control	and	motivation	selection.	
Table	10
gives	a	summary.
Table	10	
Control	methods
	
Capability
control
	
Boxing
methods
The	system	is	confined	in	such	a	way	that	it	can	affect	the
external	world	only	through	some	restricted,	pre-approved
channel.	Encompasses	physical	and	informational	containment
methods.
Incentive
methods
The	system	is	placed	within	an	environment	that	provides
appropriate	incentives.	This	could	involve	social	integration	into
a	world	of	similarly	powerful	entities.	Another	variation	is	the
use	of	(cryptographic)	reward	tokens.	“Anthropic	capture”	is	also
a	very	important	possibility	but	one	that	involves	esoteric
considerations.
Stunting
Constraints	are	imposed	on	the	cognitive	capabilities	of	the
system	or	its	ability	to	affect	key	internal	processes.
Tripwires

methods.
Incentive
methods
Stunting
Tripwires
Diagnostic	tests	are	performed	on	the	system	(possibly	without
its	knowledge)	and	a	mechanism	shuts	down	the	system	if
dangerous	activity	is	detected.

Motivation
selection
	
Direct
specification
The	system	is	endowed	with	some	directly	specified	motivation
system,	which	might	be	consequentialist	or	involve	following	a
set	of	rules.
A	motivation	system	is	designed	to	severely	limit	the	scope	of
the	agent’s	ambitions	and	activities.
Indirect
normativity
Indirect	normativity	could	involve	rule-based	or	consequentialist
principles,	but	is	distinguished	by	its	reliance	on	an	indirect
approach	to	specifying	the	rules	that	are	to	be	followed	or	the
values	that	are	to	be	pursued.
One	starts	with	a	system	that	already	has	substantially	human	or
benevolent	motivations,	and	enhances	its	cognitive	capacities	to
make	it	superintelligent.
Each	control	method	comes	with	potential	vulnerabilities	and	presents
different	degrees	of	difficulty	in	its	implementation.	It	might	perhaps	be	thought
that	we	should	rank	them	from	better	to	worse,	and	then	opt	for	the	best	method.
But	that	would	be	simplistic.	Some	methods	can	be	used	in	combination	whereas

others	are	exclusive.	Even	a	comparatively	insecure	method	may	be	advisable	if
it	can	easily	be	used	as	an	adjunct,	whereas	a	strong	method	might	be
unattractive	if	it	would	preclude	the	use	of	other	desirable	safeguards.
It	is	therefore	necessary	to	consider	what	package	deals	are	available.	We	need
to	consider	what	type	of	system	we	might	try	to	build,	and	which	control
methods	would	be	applicable	to	each	type.	This	is	the	topic	for	our	next	chapter.

CHAPTER	10
Oracles,	genies,	sovereigns,	tools
	
Some	say:	“Just	build	a	question-answering	system!”	or	“Just	build	an	AI
that	is	like	a	tool	rather	than	an	agent!”	But	these	suggestions	do	not	make
all	safety	concerns	go	away,	and	it	is	in	fact	a	non-trivial	question	which
type	of	system	would	offer	the	best	prospects	for	safety.	We	consider	four
types	or	“castes”—oracles,	genies,	sovereigns,	and	tools—and	explain	how
they	relate	to	one	another.
1
	Each	offers	different	sets	of	advantages	and
disadvantages	in	our	quest	to	solve	the	control	problem.
Oracles
	
An	oracle	is	a	question-answering	system.	It	might	accept	questions	in	a	natural
language	and	present	its	answers	as	text.	An	oracle	that	accepts	only	yes/no
questions	could	output	its	best	guess	with	a	single	bit,	or	perhaps	with	a	few
extra	bits	to	represent	its	degree	of	confidence.	An	oracle	that	accepts	open-
ended	questions	would	need	some	metric	with	which	to	rank	possible	truthful
answers	in	terms	of	their	informativeness	or	appropriateness.
2

Oracles
	
2
	In	either	case,
building	an	oracle	that	has	a	fully	domain-general	ability	to	answer	natural
language	questions	is	an	AI-complete	problem.	If	one	could	do	that,	one	could
probably	also	build	an	AI	that	has	a	decent	ability	to	understand	human
intentions	as	well	as	human	words.
Oracles	with	domain-limited	forms	of	superintelligence	are	also	conceivable.
For	instance,	one	could	conceive	of	a	mathematics-oracle	which	would	only
accept	queries	posed	in	a	formal	language	but	which	would	be	very	good	at

2
answering	such	questions	(e.g.	being	able	to	solve	in	an	instant	almost	any
formally	expressed	math	problem	that	the	human	mathematics	profession	could
solve	by	laboring	collaboratively	for	a	century).	Such	a	mathematics-oracle
would	form	a	stepping-stone	toward	domain-general	superintelligence.

Oracles	with	superintelligence	in	extremely	limited	domains	already	exist.	A
pocket	calculator	can	be	viewed	as	a	very	narrow	oracle	for	basic	arithmetical
questions;	an	Internet	search	engine	can	be	viewed	as	a	very	partial	realization	of
an	oracle	with	a	domain	that	encompasses	a	significant	part	of	general	human
declarative	knowledge.	These	domain-limited	oracles	are	tools	rather	than	agents
(more	on	tool-AIs	shortly).	In	what	follows,	though,	the	term	“oracle”	will	refer
to	question-answering	systems	that	have	domain-general	superintelligence,
unless	otherwise	stated.
To	make	a	general	superintelligence	function	as	an	oracle,	we	could	apply
both	motivation	selection	and	capability	control.	Motivation	selection	for	an
oracle	may	be	easier	than	for	other	castes	of	superintelligence,	because	the	final
goal	in	an	oracle	could	be	comparatively	simple.	We	would	want	the	oracle	to
give	truthful,	non-manipulative	answers	and	to	otherwise	limit	its	impact	on	the

world.	Applying	a	domesticity	method,	we	might	require	that	the	oracle	should
use	only	designated	resources	to	produce	its	answer.	For	example,	we	might
stipulate	that	it	should	base	its	answer	on	a	preloaded	corpus	of	information,
such	as	a	stored	snapshot	of	the	Internet,	and	that	it	should	use	no	more	than	a
fixed	number	of	computational	steps.
3
	To	avoid	incentivizing	the	oracle	to
manipulate	us	into	giving	it	easier	questions—which	would	happen	if	we	gave	it

3
the	goal	of	maximizing	its	accuracy	across	all	questions	we	will	ask	it—we
could	give	it	the	goal	of	answering	only	one	question	and	to	terminate
immediately	upon	delivering	its	answer.	The	question	would	be	preloaded	into
its	memory	before	the	program	is	run.	To	ask	a	second	question,	we	would	reset
the	machine	and	run	the	same	program	with	a	different	question	preloaded	in
memory.
Subtle	and	potentially	treacherous	challenges	arise	even	in	specifying	the
relatively	simple	motivation	system	needed	to	drive	an	oracle.	Suppose,	for

memory.
example,	that	we	come	up	with	some	explication	of	what	it	means	for	the	AI	“to
minimize	its	impact	on	the	world,	subject	to	achieving	certain	results”	or	“to	use
only	designated	resources	in	preparing	the	answer.”	What	happens	if	the	AI,	in
the	course	of	its	intellectual	development,	undergoes	the	equivalent	of	a
scientific	revolution	involving	a	change	in	its	basic	ontology?
4
	We	might
initially	have	explicated	“impact”	and	“designated	resources”	using	our	own
ontology	(postulating	the	existence	of	various	physical	objects	such	as

4
	We	might
computers).	But	just	as	we	have	abandoned	ontological	categories	that	were
taken	for	granted	by	scientists	in	previous	ages	(e.g.	“phlogiston,”	“élan	vital,”
and	“absolute	simultaneity”),	so	a	superintelligent	AI	might	discover	that	some

of	our	current	categories	are	predicated	on	fundamental	misconceptions.	The
goal	system	of	an	AI	undergoing	an	ontological	crisis	needs	to	be	resilient
enough	that	the	“spirit”	of	its	original	goal	content	is	carried	over,	charitably
transposed	into	the	new	key.
Although	making	an	oracle	safe	through	the	use	of	motivation	selection	might
be	far	from	trivial,	it	may	nevertheless	be	easier	than	doing	the	same	for	an	AI
that	roams	the	world	in	pursuit	of	some	complicated	goal.	This	is	an	argument
for	preferring	that	the	first	superintelligence	be	an	oracle.	Another	point	in	favor
of	an	oracle-first	path	is	the	greater	susceptibility	of	an	oracle	to	capability
control.	All	
the	standard	boxing	methods	apply	here.	In	addition,	there	may	be
methods	that	are	distinctively	applicable	to	oracles.	For	example,	consider	the
risk	that	an	oracle	will	answer	questions	not	in	a	maximally	truthful	way	but	in
such	a	way	as	to	subtly	manipulate	us	into	promoting	its	own	hidden	agenda.

One	way	to	slightly	mitigate	this	threat	could	be	to	create	multiple	oracles,	each
with	a	slightly	different	code	and	a	slightly	different	information	base.	A	simple
mechanism	could	then	compare	the	answers	given	by	the	different	oracles	and
only	present	them	for	human	viewing	if	all	the	answers	agree.	While	one	must
assume	that	any	superintelligent	domain-general	oracle	would	know	about	the
existence	of	other	oracles	(it	could	infer	this	from	its	knowledge	about	human

society),	it	might	be	feasible	to	hide	some	implementation	details	of	each	oracle
from	the	views	of	the	others.	If	the	oracles	are	not	able	to	communicate	with	one
another,	it	might	then	be	difficult	for	them	to	coordinate	on	how	to	answer	our
questions	manipulatively.	There	are	many	ways	of	deviating	from	the	truth,	and
the	oracles	may	not	all	agree	on	which	of	these	deviations	is	most	attractive—
whereas	the	truth	itself	is	a	Schelling	point	(a	salient	place	for	agreement	in	the

absence	of	communication).	So	if	the	oracles	achieve	consensus,	it	might	be	a
sign	that	they	gave	the	true	answer.
5
An	oracle	would	ideally	be	trustworthy	in	the	sense	that	we	could	safely
assume	that	its	answers	are	always	accurate	to	the	best	of	its	ability.	But	even	an
untrustworthy	oracle	could	be	useful.	We	could	ask	such	an	oracle	questions	of	a
type	for	which	it	is	difficult	to	find	the	answer	but	easy	to	verify	whether	a	given
answer	is	correct.	Many	mathematical	problems	are	of	this	kind.	If	we	are

5
wondering	whether	a	mathematical	proposition	is	true,	we	could	ask	the	oracle
to	produce	a	proof	or	disproof	of	the	proposition.	Finding	the	proof	may	require
insight	and	creativity	beyond	our	ken,	but	checking	a	purported	proof’s	validity
can	be	done	by	a	simple	mechanical	procedure.

If	it	is	expensive	to	verify	answers	(as	is	often	the	case	on	topics	outside	logic
and	mathematics),	we	can	randomly	select	a	subset	of	the	oracle’s	answers	for
verification.	If	they	are	all	correct,	we	can	assign	a	high	probability	to	most	of
the	other	answers	also	being	correct.	This	trick	can	give	us	a	bulk	discount	on
trustworthy	answers	that	would	be	costly	to	verify	individually.	(Unfortunately,
it	cannot	give	us	trustworthy	answers	that	we	are	
unable
	to	verify,	since	a
dissembling	oracle	may	choose	to	answer	correctly	only	those	questions	where	it
believes	we	could	verify	its	answers.)
There	could	be	important	issues	on	which	we	could	benefit	from	an	augural
pointer	toward	the	correct	answer	(or	toward	a	method	for	locating	the	correct
answer)	even	if	we	had	to	actively	distrust	the	provenance.	For	instance,	one
might	ask	for	the	solution	to	various	technical	or	philosophical	problems	that
may	arise	in	the	course	of	trying	to	develop	more	advanced	motivation	selection

methods.	If	we	had	a	proposed	AI	design	alleged	to	be	safe,	we	could	ask	an
oracle	whether	it	could	identify	any	significant	flaw	in	the	design,	and	whether	it
could	explain	any	such	flaw	to	us	in	twenty	words	or	less.	Questions	of	this	kind
could	elicit	valuable	information.	Caution	and	restraint	would	be	required,
however,	for	us	not	to	ask	
too	many
	such	questions—and	not	to	allow	ourselves
to	partake	
of	
too	many
	details	of	the	answers	given	to	the	questions	we	do	ask—

too	many
to	partake	
of	
too	many
lest	we	give	the	untrustworthy	oracle	opportunities	to	work	on	our	psychology
(by	means	of	plausible-seeming	but	subtly	manipulative	messages).	It	might	not
take	many	bits	of	communication	for	an	AI	with	the	social	manipulation
superpower	to	bend	us	to	its	will.
Even	if	the	oracle	itself	works	exactly	as	intended,	there	is	a	risk	that	it	would
be	misused.	One	obvious	dimension	of	this	problem	is	that	an	oracle	AI	would
be	a	source	of	immense	power	which	could	give	a	decisive	strategic	advantage

to	its	operator.	This	power	might	be	illegitimate	and	it	might	not	be	used	for	the
common	good.	Another	more	subtle	but	no	less	important	dimension	is	that	the
use	of	an	oracle	could	be	extremely	dangerous	for	the	operator	herself.	Similar
worries	(which	involve	philosophical	as	well	as	technical	issues)	arise	also	for
other	hypothetical	castes	of	superintelligence.	We	will	explore	them	more
thoroughly	in	
Chapter	13
.	Suffice	it	here	to	note	that	the	protocol	determining

Chapter	13
which	questions	are	asked,	in	which	sequence,	and	how	the	answers	are	reported
and	disseminated	could	be	of	great	significance.	One	might	also	consider
whether	to	try	to	build	the	oracle	in	such	a	way	that	it	would	refuse	to	answer
any	question	in	cases	where	it	predicts	that	its	answering	would	have
consequences	classified	as	catastrophic	according	to	some	rough-and-ready

criteria.
	
A	genie	is	a	command-executing	system:	it	receives	a	high-level	command,
carries	it	out,	then	pauses	to	await	the	next	command.
6
	A	sovereign	is	a	system
that	has	an	open-ended	mandate	to	operate	in	the	world	in	pursuit	of	broad	and
possibly	very	long-range	objectives.	Although	these	might	seem	like	radically
different	templates	for	what	a	superintelligence	should	be	and	do,	the	difference
is	not	as	deep	as	it	might	at	first	glance	appear.
With	a	genie,	one	already	sacrifices	the	most	attractive	property	of	an	oracle:
the	opportunity	to	use	boxing	methods.	While	one	might	consider	creating	a
physically	confined	genie,	for	instance	one	that	can	only	construct	objects	inside
a	designated	volume—a	volume	that	might	be	sealed	off	by	a	hardened	wall	or	a
barrier	loaded	with	explosive	charges	rigged	to	detonate	if	the	containment	is
breached—it	would	be	difficult	to	have	much	confidence	in	the	security	of	any

such	physical	containment	method	against	a	superintelligence	equipped	with
versatile	manipulators	and	construction	materials.	Even	if	it	were	somehow
possible	to	ensure	a	containment	as	secure	as	that	which	can	be	achieved	for	an
oracle,	it	is	not	clear	how	much	we	would	have	gained	by	giving	the
superintelligence	direct	access	to	manipulators	compared	to	requiring	it	instead
to	output	a	blueprint	that	we	could	inspect	and	then	use	to	achieve	the	same
result	ourselves.	The	gain	in	speed	and	convenience	from	bypassing	the	human

intermediary	seems	hardly	worth	the	loss	of	foregoing	the	use	of	the	stronger
boxing	methods	available	to	contain	an	oracle.
If	one	
were
	creating	a	genie,	it	would	be	desirable	to	build	it	so	that	it	would
obey	the	intention	behind	the	command	rather	than	its	literal	meaning,	since	a
literalistic	genie	(one	superintelligent	enough	to	attain	a	decisive	strategic
advantage)	might	have	a	propensity	to	kill	the	user	and	the	rest	of	humanity	on
its	first	use,	for	reasons	explained	in	the	section	on	malignant	failure	modes	in
Chapter	8

If	one	
were
Chapter	8
.	More	broadly,	it	would	seem	important	that	the	genie	seek	a
charitable—and	what	human	beings	would	regard	as	reasonable—interpretation
of	what	is	being	commanded,	and	that	the	genie	be	motivated	to	carry	out	the
command	under	such	an	interpretation	rather	than	under	the	literalistic
interpretation.	The	ideal	genie	would	be	a	super-butler	rather	than	an	autistic

savant.
A	genie	endowed	with	such	a	super-butler	nature,	however,	would	not	be	far
from	qualifying	for	membership	in	the	caste	of	sovereigns.	Consider,	for
comparison,	the	idea	of	building	a	sovereign	with	the	final	goal	of	obeying	the
spirit	of	the	commands	we	would	have	given	had	we	built	a	genie	rather	than	a
sovereign.	Such	a	sovereign	would	mimic	a	genie.	Being	superintelligent,	this
sovereign	would	do	a	good	job	at	guessing	what	commands	we	would	have
given	a	genie	(and	it	could	always	ask	us	if	that	would	help	inform	its	decisions).
Would	there	then	really	be	any	important	difference	between	such	a	sovereign
and	a	genie?	Or,	pressing	on	the	distinction	from	the	other	side,	consider	that	a
superintelligent	genie	may	likewise	be	able	to	predict	what	commands	we	will
give	it:	what	then	is	gained	from	having	it	await	the	actual	issuance	before	it
acts?
One	might	think	that	a	big	advantage	of	a	genie	over	a	sovereign	is	that	if
something	goes	wrong,	we	could	issue	the	genie	with	a	new	command	to	stop	or

acts?
to	reverse	the	effects	of	the	previous	actions,	whereas	a	sovereign	would	just
push	on	regardless	of	our	protests.	But	this	apparent	safety	advantage	for	the
genie	is	largely	illusory.	The	“stop”	or	“undo”	button	on	a	genie	works	only	for
benign	failure	modes:	in	the	case	of	a	malignant	failure—one	in	which,	for
example,	carrying	out	the	existing	command	has	become	a	final	goal	for	the
genie—the	genie	would	simply	disregard	any	subsequent	attempt	to
countermand	the	previous	command.
7

7
One	option	would	be	to	try	to	build	a	genie	such	that	it	would	automatically
present	the	user	with	a	prediction	about	salient	aspects	of	the	likely	outcomes	of
a	proposed	command,	asking	for	confirmation	before	proceeding.	Such	a	system
could	be	referred	to	as	a	
genie-with-a-preview
.	But	if	this	could	be	done	for	a
genie,	it	could	likewise	be	done	for	a	sovereign.	So	again,	this	is	not	a	clear
differentiator	between	a	genie	and	a	sovereign.	(Supposing	that	a	preview

7
functionality	could	be	created,	the	questions	of	whether	and	if	so	how	to	use	it
are	rather	less	obvious	than	one	might	think,	notwithstanding	the	strong	appeal
of	being	able	to	glance	at	the	outcome	before	committing	to	making	it
irrevocable	reality.	We	will	return	to	this	matter	later.)
The	ability	of	one	caste	to	mimic	another	extends	to	oracles,	too.	A	genie
could	be	made	to	act	like	an	oracle	if	the	only	commands	we	ever	give	it	are	to

answer	certain	questions.	An	oracle,	in	turn,	could	be	made	to	substitute	for	a

genie	if	we	asked	the	oracle	what	the	easiest	way	is	to	get	certain	commands
executed.	The	
oracle	could	give	us	step-by-step	instructions	for	achieving	the
same	result	as	a	genie	would	produce,	or	it	could	even	output	the	source	code	for
a	genie.
8
	Similar	points	can	be	made	with	regard	to	the	relation	between	an
oracle	and	a	sovereign.
The	real	difference	between	the	three	castes,	therefore,	does	not	reside	in	the
ultimate	capabilities	that	they	would	unlock.	Instead,	the	difference	comes	down
to	alternative	approaches	to	the	control	problem.	Each	caste	corresponds	to	a
different	set	of	safety	precautions.	The	most	prominent	feature	of	an	oracle	is
that	it	can	be	boxed.	One	might	also	try	to	apply	domesticity	motivation
selection	to	an	oracle.	A	genie	is	harder	to	box,	but	at	least	domesticity	may	be
applicable.	A	sovereign	can	neither	be	boxed	nor	handled	through	the
domesticity	approach.
If	these	were	the	only	relevant	factors,	then	the	order	of	desirability	would

seem	clear:	an	oracle	would	be	safer	than	a	genie,	which	would	be	safer	than	a
sovereign;	and	any	initial	differences	in	convenience	and	speed	of	operation
would	be	relatively	small	and	easily	dominated	by	the	gains	in	safety	obtainable
by	building	an	oracle.	However,	there	are	other	factors	that	need	to	be	taken	into
account.	When	choosing	between	castes,	one	should	consider	not	only	the
danger	posed	by	the	system	itself	but	also	the	dangers	that	arise	out	of	the	way	it

might	be	used.	A	genie	most	obviously	gives	the	person	who	controls	it
enormous	power,	but	the	same	holds	for	an	oracle.
9
	A	sovereign,	by	contrast,
could	be	constructed	in	such	way	as	to	accord	no	one	person	or	group	any
special	influence	over	the	outcome,	and	such	that	it	would	resist	any	attempt	to
corrupt	or	alter	its	original	agenda.	What	is	more,	if	a	sovereign’s	motivation	is
defined	using	“indirect	normativity”	(a	concept	to	be	described	in	
Chapter	13
)
then	it	could	be	used	to	achieve	some	abstractly	defined	outcome,	such	as

9
Chapter	13
)
“whatever	is	maximally	fair	and	morally	right”—without	anybody	knowing	in
advance	what	exactly	this	will	entail.	This	would	create	a	situation	analogous	to
a	Rawlsian	“veil	of	ignorance.”
10
	Such	a	setup	might	facilitate	the	attainment	of
consensus,	help	prevent	conflict,	and	promote	a	more	equitable	outcome.
Another	point,	which	counts	against	some	types	of	oracles	and	genies,	is	that
there	are	risks	involved	in	designing	a	superintelligence	to	have	a	final	goal	that

10
does	not	fully	match	the	outcome	that	we	ultimately	seek	to	attain.	For	example,
if	we	use	a	domesticity	motivation	to	make	the	superintelligence	want	to
minimize	some	of	its	impacts	on	the	world,	we	might	thereby	create	a	system

whose	preference	ranking	over	possible	outcomes	differs	from	that	of	the
sponsor.	The	same	will	happen	if	we	build	the	AI	to	place	a	peculiarly	high	value
on	answering	questions	correctly,	or	on	faithfully	obeying	individual	commands.
Now,	if	sufficient	care	is	taken,	this	should	not	cause	any	problems:	there	would
be	sufficient	agreement	between	the	two	rankings—at	least	insofar	as	they
pertain	to	possible	worlds	that	have	a	reasonable	chance	of	being	actualized—
that	the	outcomes	that	are	good	by	the	AI’s	standard	are	also	good	by	the
principal’s	standard.	But	perhaps	one	could	argue	for	the	design	principle	that	it
is	unwise	to	introduce	even	a	limited	amount	of	disharmony	between	the	AI’s
goals	and	ours.	(The	same	concern	would	of	course	apply	to	giving	sovereigns
goals	that	do	not	completely	harmonize	with	ours.)
Tool-AIs
	
One	suggestion	that	has	been	made	is	that	we	build	the	superintelligence	to	be
like	a	tool	rather	than	an	agent.
11
	This	idea	seems	to	arise	out	of	the	observation

Tool-AIs
	
11
that	ordinary	software,	which	is	used	in	countless	applications,	does	not	raise
any	safety	concerns	even	remotely	analogous	to	the	challenges	discussed	in	this
book.	Might	one	not	create	“tool-AI”	that	is	like	such	software—like	a	flight
control	system,	say,	or	a	virtual	assistant—only	more	flexible	and	capable?	Why
build	a	superintelligence	that	has	a	will	of	its	own?	On	this	line	of	thinking,	the
agent	paradigm	is	fundamentally	misguided.	Instead	of	creating	an	AI	that	has

beliefs	and	desires	and	that	acts	like	an	artificial	person,	we	should	aim	to	build
regular	software	that	simply	does	what	it	is	programmed	to	do.
This	idea	of	creating	software	that	“simply	does	what	it	is	programmed	to	do”
is,	however,	not	so	straightforward	if	the	product	being	created	is	a	powerful
general	intelligence.	There	is,	of	course,	a	trivial	sense	in	which	all	software
simply	does	what	it	is	programmed	to	do:	the	behavior	is	mathematically
specified	by	the	code.	But	this	is	equally	true	for	all	castes	of	machine

intelligence,	“tool-AI”	or	not.	If,	instead,	“simply	doing	what	it	is	programmed
to	do”	means	that	the	software	behaves	as	the	programmers	
intended
,	then	this	is
a	standard	that	ordinary	software	very	often	fails	to	meet.
Because	of	the	limited	capabilities	of	contemporary	software	(compared	with
those	of	machine	superintelligence)	the	consequences	of	such	failures	are
manageable,	ranging	from	insignificant	to	very	costly,	but	in	no	case	amounting

to	an	existential	threat.
12
	However,	if	it	is	insufficient	capability	rather	than
sufficient	reliability	that	makes	ordinary	software	existentially	safe,	then	it	is
unclear	how	such	software	could	be	a	model	for	a	safe	superintelligence.	It
might	be	thought	that	by	expanding	the	range	of	tasks	done	by	ordinary
software,	one	could	eliminate	the	need	for	artificial	general	intelligence.	But	the
range	and	diversity	of	tasks	that	a	general	intelligence	could	profitably	perform
in	a	modern	economy	is	enormous.	It	would	be	infeasible	to	create	special-
purpose	software	to	handle	all	of	those	tasks.	Even	if	it	could	be	done,	such	a
project	would	take	a	
long
	time	to	carry	out.	Before	it	could	be	completed,	the
nature	of	some	of	the	tasks	would	have	changed,	and	new	tasks	would	have
become	relevant.	There	would	be	great	advantage	to	having	software	that	can
learn	on	its	own	to	do	new	tasks,	and	indeed	to	discover	new	tasks	in	need	of

long
doing.	But	this	would	require	that	the	software	be	able	to	learn,	reason,	and	plan,
and	to	do	so	in	a	powerful	and	robustly	cross-domain	manner.	In	other	words,	it
would	require	general	intelligence.
Especially	relevant	for	our	purposes	is	the	task	of	software	development	itself.
There	would	be	enormous	practical	advantages	to	being	able	to	automate	this.
Yet	the	capacity	for	rapid	self-improvement	is	just	the	critical	property	that
enables	a	seed	AI	to	set	off	an	intelligence	explosion.

If	general	intelligence	is	not	dispensable,	is	there	some	other	way	of
construing	the	tool-AI	idea	so	as	to	preserve	the	reassuringly	passive	quality	of	a
humdrum	tool?	Could	one	have	a	general	intelligence	that	is	not	an	agent?
Intuitively,	it	is	
not	just	the	limited	capability	of	ordinary	software	that	makes	it
safe:	it	is	also	its	lack	of	ambition.	There	is	no	subroutine	in	Excel	that	secretly
wants	to	take	over	the	world	if	only	it	were	smart	enough	to	find	a	way.	The

spreadsheet	application	does	not	“want”	anything	at	all;	it	just	blindly	carries	out
the	instructions	in	the	program.	What	(one	might	wonder)	stands	in	the	way	of
creating	a	more	generally	intelligent	application	of	the	same	type?	An	oracle,	for
instance,	which,	when	prompted	with	a	description	of	a	goal,	would	respond
with	a	plan	for	how	to	achieve	it,	in	much	the	same	way	that	Excel	responds	to	a
column	of	numbers	by	calculating	a	sum—without	thereby	expressing	any
“preferences”	regarding	its	output	or	how	humans	might	choose	to	use	it?

The	classical	way	of	writing	software	requires	the	programmer	to	understand
the	task	to	be	performed	in	sufficient	detail	to	formulate	an	explicit	solution
process	consisting	of	a	sequence	of	mathematically	well-defined	steps
expressible	in	code.
13
	(In	practice,	software	engineers	rely	on	code	libraries

stocked	with	useful	behaviors,	which	they	can	invoke	without	needing	to
understand	how	the	behaviors	are	implemented.	But	that	code	was	originally
created	by	programmers	who	had	a	detailed	understanding	of	what	they	were
doing.)	This	approach	works	for	solving	well-understood	tasks,	and	is	to	credit
for	most	software	that	is	currently	in	use.	It	falls	short,	however,	when	nobody
knows	precisely	how	to	solve	all	of	the	tasks	that	need	to	be	accomplished.	This
is	where	techniques	from	the	field	of	artificial	intelligence	become	relevant.	In
narrow	applications,	machine	learning	might	be	used	merely	to	fine-tune	a	few
parameters	in	a	largely	human-designed	program.	A	spam	filter,	for	example,
might	be	trained	on	a	corpus	of	hand-classified	email	messages	in	a	process	that
changes	the	weights	that	the	classification	algorithm	places	on	various	diagnostic
features.	In	a	more	ambitious	application,	the	classifier	might	be	built	so	that	it
can	discover	new	features	on	its	own	and	test	their	validity	in	a	changing

environment.	An	even	more	sophisticated	spam	filter	could	be	endowed	with
some	ability	to	reason	about	the	trade-offs	facing	the	user	or	about	the	contents
of	the	messages	it	is	classifying.	In	neither	of	these	cases	does	the	programmer
need	to	know	the	best	way	of	distinguishing	spam	from	ham,	only	how	to	set	up
an	algorithm	that	can	improve	its	own	performance	via	learning,	discovering,	or
reasoning.
With	advances	in	artificial	intelligence,	it	would	become	possible	for	the

programmer	to	offload	more	of	the	cognitive	labor	required	to	figure	out	how	to
accomplish	a	given	task.	In	an	extreme	case,	the	programmer	would	simply
specify	a	formal	criterion	of	what	counts	as	success	and	leave	it	to	the	AI	to	find
a	solution.	To	guide	its	search,	the	AI	would	use	a	set	of	powerful	heuristics	and
other	methods	to	discover	structure	in	the	space	of	possible	solutions.	It	would
keep	searching	until	it	found	a	solution	that	satisfied	the	success	criterion.	The

AI	would	then	either	implement	the	solution	itself	or	(in	the	case	of	an	oracle)
report	the	solution	to	the	user.
Rudimentary	forms	of	this	approach	are	quite	widely	deployed	today.
Nevertheless,	software	that	uses	AI	and	machine	learning	techniques,	though	it
has	some	ability	to	find	solutions	that	the	programmers	had	not	anticipated,
functions	
for	all	practical	purposes	like	a	tool	and	poses	no	existential	risk.	We
would	enter	the	danger	zone	only	when	the	methods	used	in	the	search	for

functions	
solutions	become	extremely	powerful	and	general:	that	is,	when	they	begin	to
amount	to	general	intelligence—and	especially	when	they	begin	to	amount	to

There	are	(at	least)	two	places	where	trouble	could	then	arise.	First,	the
superintelligent	search	process	might	find	a	solution	that	is	not	just	unexpected
but	radically	unintended.	This	could	lead	to	a	failure	of	one	of	the	types
discussed	previously	(“perverse	instantiation,”	“infrastructure	profusion,”	or
“mind	crime”).	It	is	most	obvious	how	this	could	happen	in	the	case	of	a
sovereign	or	a	genie,	which	directly	implements	the	solution	it	has	found.	If
making	molecular	smiley	faces	or	transforming	the	planet	into	paperclips	is	the
first	idea	that	the	superintelligence	discovers	that	meets	the	solution	criterion,
then	smiley	faces	or	paperclips	we	get.
14
	But	even	an	oracle,	which—if	all	else
goes	well—merely	
reports
	the	solution,	could	become	a	cause	of	perverse
instantiation.	The	user	asks	the	oracle	for	a	plan	to	achieve	a	certain	outcome,	or
for	a	technology	to	serve	a	certain	function;	and	when	the	user	follows	the	plan

14
reports
or	constructs	the	technology,	a	perverse	instantiation	can	ensue,	just	as	if	the	AI
had	implemented	the	solution	itself.
15
A	second	place	where	trouble	could	arise	is	in	the	course	of	the	software’s
operation.	If	the	methods	that	the	software	uses	to	search	for	a	solution	are
sufficiently	sophisticated,	they	may	include	provisions	for	managing	the	search
process	itself	in	an	intelligent	manner.	In	this	case,	the	machine	running	the
software	may	begin	to	seem	less	like	a	mere	tool	and	more	like	an	agent.	Thus,

15
the	software	may	start	by	developing	a	plan	for	how	to	go	about	its	search	for	a
solution.	The	plan	may	specify	which	areas	to	explore	first	and	with	what
methods,	what	data	to	gather,	and	how	to	make	best	use	of	available
computational	resources.	In	searching	for	a	plan	that	satisfies	the	software’s
internal	criterion	(such	as	yielding	a	sufficiently	high	probability	of	finding	a
solution	satisfying	the	user-specified	criterion	within	the	allotted	time),	the
software	may	stumble	on	an	unorthodox	idea.	For	instance,	it	might	generate	a

plan	that	begins	with	the	acquisition	of	additional	computational	resources	and
the	elimination	of	potential	interrupters	(such	as	human	beings).	Such	“creative”
plans	come	into	view	when	the	software’s	cognitive	abilities	reach	a	sufficiently
high	level.	When	the	software	puts	such	a	plan	into	action,	an	existential
catastrophe	may	ensue.
As	the	examples	in	
Box	9
	illustrate,	open-ended	search	processes	sometimes
evince	strange	and	unexpected	non-anthropocentric	solutions	even	in	their

Box	9
currently	limited	forms.	Present-day	search	processes	are	not	hazardous	because
they	are	too	weak	to	discover	the	kind	of	plan	that	could	enable	a	program	to
take	over	the	world.	Such	a	plan	would	include	extremely	difficult	steps,	such	as
the	invention	of	a	new	weapons	technology	several	generations	ahead	of	the

state	of	the	art	or	the	execution	of	a	propaganda	campaign	far	more	effective
than	any	communication	devised	by	human	spin	doctors.	To	have	a	chance	of
even	
conceiving
	of	such	ideas,	let	alone	developing	them	in	a	way	that	would
actually	work,	a	machine	would	probably	need	the	capacity	to	represent	the
world	in	a	way	that	is	at	least	as	rich	and	realistic	as	the	world	model	possessed
by	a	normal	human	adult	(though	
a	lack	of	awareness	in	some	areas	might
possibly	be	compensated	for	by	extra	skill	in	others).	This	is	far	beyond	the
reach	of	contemporary	AI.	And	because	of	the	combinatorial	explosion,	which
generally	defeats	attempts	to	solve	complicated	planning	problems	with	brute-
force	methods	(as	we	saw	in	
Chapter	1
),	the	shortcomings	of	known	algorithms
cannot	realistically	be	overcome	simply	by	pouring	on	more	computing	power.
21
However,	once	the	search	or	planning	processes	become	powerful	enough,	they
also	become	potentially	dangerous.
Box	9	Strange	solutions	from	blind	search

Chapter	1
21
	
Even	simple	evolutionary	search	processes	sometimes	produce	highly
unexpected	results,	solutions	that	satisfy	a	formal	user-defined	criterion	in	a	very
different	way	than	the	user	expected	or	intended.
The	field	of	evolvable	hardware	offers	many	illustrations	of	this	phenomenon.
In	this	field,	an	evolutionary	algorithm	searches	the	space	of	hardware	designs,
testing	the	fitness	of	each	design	by	instantiating	it	physically	on	a	rapidly
reconfigurable	array	or	motherboard.	The	evolved	designs	often	show

remarkable	economy.	For	instance,	one	search	discovered	a	frequency
discrimination	circuit	that	functioned	without	a	clock—a	component	normally
considered	necessary	for	this	function.	The	researchers	estimated	that	the
evolved	circuit	was	between	one	and	two	orders	of	magnitude	smaller	than	what
a	human	engineer	would	have	required	for	the	task.	The	circuit	exploited	the
physical	properties	of	its	components	in	unorthodox	ways;	some	active,

necessary	components	were	not	even	connected	to	the	input	or	output	pins!
These	components	instead	participated	via	what	would	normally	be	considered
nuisance	side	effects,	such	as	electromagnetic	coupling	or	power-supply	loading.
Another	search	process,	tasked	with	creating	an	oscillator,	was	deprived	of	a
seemingly	even	more	indispensible	component,	the	capacitor.	When	the

algorithm	presented	its	successful	solution,	the	researchers	examined	it	and	at
first	concluded	that	it	“should	not	work.”	Upon	more	careful	examination,	they
discovered	that	the	algorithm	had,	MacGyver-like,	reconfigured	its	sensor-less
motherboard	into	a	makeshift	radio	receiver,	using	the	printed	circuit	board
tracks	as	an	aerial	to	pick	up	signals	generated	by	personal	computers	that
happened	to	be	situated	nearby	in	the	laboratory.	The	circuit	amplified	this	signal
to	produce	the	desired	oscillating	output.
16
In	other	experiments,	evolutionary	algorithms	designed	circuits	that	sensed
whether	the	motherboard	was	being	monitored	with	an	oscilloscope	or	whether	a
soldering	iron	was	connected	to	the	lab’s	common	power	supply.	These
examples	illustrate	how	an	open-ended	search	process	can	repurpose	the
materials	accessible	to	it	in	order	to	devise	completely	unexpected	sensory
capabilities,	by	means	that	conventional	human	design-thinking	is	poorly
equipped	to	exploit	or	even	account	for	in	retrospect.

16
The	tendency	for	evolutionary	search	to	“cheat”	or	find	counterintuitive	ways
of	achieving	a	given	end	is	on	display	in	nature	too,	though	it	is	perhaps	less
obvious	to	us	there	because	of	our	already	being	somewhat	familiar	with	the
look	and	feel	of	biology,	and	thus	being	prone	to	regarding	the	actual	outcomes
of	natural	evolutionary	processes	as	normal—even	if	we	would	not	have
expected	them	
ex	ante
.	But	it	is	possible	to	set	up	experiments	in	artificial

ex	ante
selection	where	one	can	see	the	evolutionary	process	in	action	outside	its
familiar	context.	In	such	experiments,	researchers	can	create	conditions	that
rarely	obtain	in	nature,	and	observe	the	results.
For	example,	prior	to	the	1960s,	it	was	apparently	quite	common	for	biologists
to	maintain	that	predator	populations	restrict	their	own	breeding	in	order	to	avoid
falling	into	a	Malthusian	trap.
17
	Although	individual	selection	would	work
against	such	restraint,	it	was	sometimes	thought	that	group	selection	would

17
overcome	individual	incentives	to	exploit	opportunities	for	reproduction	and
favor	traits	that	would	benefit	the	group	or	population	at	large.	Theoretical
analysis	and	simulation	studies	later	showed	that	while	group	selection	is
possible	in	principle,	it	can	overcome	strong	individual	selection	only	under	very
stringent	conditions	that	may	rarely	apply	in	nature.
18
	But	such	conditions	can	be
created	in	the	laboratory.	When	flour	beetles	(
Tribolium	castaneum
)	were	bred
for	reduced	population	size,	by	applying	strong	group	selection,	evolution	did

18
)	were	bred
indeed	lead	to	smaller	populations.
19
	However,	the	means	by	which	this	was
accomplished	included	not	only	the	“benign”	adaptations	of	reduced	fecundity

and	extended	developmental	time	that	a	human	naively	anthropomorphizing
evolutionary	search	might	have	expected,	but	also	an	increase	in	cannibalism.
20
	
Instead	of	allowing	agent-like	purposive	behavior	to	emerge	spontaneously
and	haphazardly	from	the	implementation	of	powerful	search	processes
(including	processes	searching	for	internal	work	plans	and	processes	directly
searching	for	solutions	meeting	some	user-specified	criterion),	it	may	be	better
to	create	agents	on	purpose.	Endowing	a	superintelligence	with	an	explicitly
agent-like	structure	can	be	a	way	of	increasing	predictability	and	transparency.	A
well-designed	system,	built	such	that	there	is	a	clean	separation	between	its
values	and	its	beliefs,	would	let	us	predict	something	about	the	outcomes	it
would	tend	to	produce.	Even	if	we	could	not	foresee	exactly	which	beliefs	the
system	would	acquire	or	which	situations	it	would	find	itself	in,	there	would	be	a
known	place	where	we	could	inspect	its	final	values	and	thus	the	criteria	that	it

will	use	in	selecting	its	future	actions	and	in	evaluating	any	potential	plan.
	
It	may	be	useful	to	summarize	the	features	of	the	different	system	castes	we	have
discussed	(
Table	11
).
Table	11	
Features	of	different	system	castes
	
Oracle
A	question-answering	system
•	Boxing	methods	fully
applicable
•	Domesticity	fully
•	Reduced	need	for	AI	to

Variations
:	Domain-limited	oracles	(e.g.
mathematics);	output-restricted	oracles	(e.g.
only	yes/no/undecided	answers,	or
probabilities);	oracles	that	refuse	to	answer
questions	if	they	predict	the	consequences	of
answering	would	meet	pre-specified
“disaster	criteria”;	multiple	oracles	for	peer
review
understand	human
intentions	and	interests
(compared	to	genies
and	sovereigns)
•	Use	of	yes/no	questions
can	obviate	need	for	a
metric	of	the
“usefulness”	or
“informativeness”	of
answers
•	Source	of	great	power
(might	give	operator	a
decisive	strategic
advantage)
•	Limited	protection
against	foolish	use	by
operator
•	Untrustworthy	oracles
could	be	used	to
provide	answers	that
are	hard	to	find	but
easy	to	verify
•	Weak	verification	of
answers	may	be
possible	through	the
use	of	multiple	oracles
Genie
A	command-executing	system
•	Boxing	methods
partially	applicable
(for	spatially	limited
genies)

:	Genies	using	different
“extrapolation	distances”	or	degrees	of
following	the	spirit	rather	than	letter	of	the
command;	domain-limited	genies;	genies-
with-preview;	genies	that	refuse	to	obey
commands	if	they	predict	the	consequences
of	obeying	would	meet	pre-specified
“disaster	criteria”
•	Domesticity	partially
•	Genie	could	offer	a
preview	of	salient
aspects	of	expected
outcomes
•	Genie	could	implement
change	in	stages,	with
opportunity	for	review
at	each	stage
operator
•	Greater	need	for	AI	to
interests	and	intentions
(compared	to	oracles)
Sovereign
A	system	designed	for	open-ended
autonomous	operation
inapplicable
•	Most	other	capability
control	methods	also
inapplicable	(except,
possibly,	social
integration	or
anthropic	capture)
•	Domesticity	mostly

:	Many	possible	motivation
systems;	possibility	of	using	preview	and
“sponsor	ratification”	(to	be	discussed	in
Chapter	13
)
•	Great	need	for	AI	to
understand	true	human
•	Necessity	of	getting	it
right	on	the	first	try
(though,	to	a	possibly
lesser	extent,	this	is
true	for	all	castes)
•	Potentially	a	source	of
great	power	for
sponsor,	including
advantage
•	Once	activated,	not
vulnerable	to	hijacking
by	operator,	and	might
be	designed	with	some
protection	against
foolish	use
•	Can	be	used	to
implement	“veil	of
ignorance”	outcomes
(cf.	
Chapter	13
)
•	Boxing	methods	may
be	applicable,
depending	on	the
implementation
•	Powerful	search
processes	would	likely
be	involved	in	the
development	and

Tool
A	system	not	designed	to	exhibit	goal-
directed	behavior
operation	of	a	machine
•	Powerful	search	to	find
a	solution	meeting
some	formal	criterion
can	produce	solutions
that	meet	the	criterion
in	an	unintended	and
dangerous	way
•	Powerful	search	might
involve	secondary,
internal	search	and
planning	processes
that	might	find
dangerous	ways	of
executing	the	primary
search	process
Further	research	would	be	needed	to	determine	which	type	of	system	would
be	safest.	The	answer	might	depend	on	the	conditions	under	which	the	AI	would
be	deployed.	The	oracle	caste	is	obviously	attractive	from	a	safety	standpoint,
since	it	would	allow	both	capability	control	methods	and	motivation	selection
methods	to	be	applied.	It	might	thus	seem	to	simply	dominate	the	sovereign
caste,	which	would	only	allow	motivation	selection	methods	(except	in	scenarios
in	which	the	world	is	believed	to	contain	other	powerful	superintelligences,	in
which	case	social	integration	or	anthropic	capture	might	apply).	However,	an

oracle	could	place	a	lot	of	power	into	the	hands	of	its	operator,	who	might	be
corrupted	or	might	apply	the	power	unwisely,	whereas	a	sovereign	would	offer
some	protection	against	these	hazards.	The	safety	ranking	is	therefore	not	so
easily	determined.
A	genie	can	be	viewed	as	a	compromise	between	an	oracle	and	a	sovereign—
but	not	necessarily	a	good	compromise.	In	many	ways,	it	would	share	the
disadvantages	of	both.	The	apparent	safety	of	a	tool-AI,	meanwhile,	may	be

illusory.	In	order	for	tools	to	be	versatile	enough	to	substitute	for	superintelligent

agents,	they	may	need	to	deploy	extremely	powerful	internal	search	and
planning	processes.	Agent-like	behaviors	may	arise	from	such	processes	as	an
unplanned	consequence.	In	that	case,	it	would	be	better	to	design	the	system	to
be	an	agent	in	the	first	place,	so	that	the	programmers	can	more	easily	see	what
criteria	will	end	up	determining	the	system’s	output.

CHAPTER	11
Multipolar	scenarios
	
We	have	seen	(particularly	in	
Chapter	8
)	how	menacing	a	unipolar	outcome
could	be,	one	in	which	a	single	superintelligence	obtains	a	decisive	strategic
advantage	and	uses	it	to	establish	a	singleton.	In	this	chapter,	we	examine
what	would	happen	in	a	multipolar	outcome,	a	post-transition	society	with
multiple	competing	superintelligent	agencies.	Our	interest	in	this	class	of
scenarios	is	twofold.	First,	as	alluded	to	in	
Chapter	9
,	social	integration
might	be	thought	to	offer	a	solution	to	the	control	problem.	We	already
noted	some	limitations	with	that	approach,	and	this	chapter	paints	a	fuller
picture.	Second,	even	without	anybody	setting	out	to	create	a	multipolar
condition	as	a	way	of	handling	the	control	problem,	such	an	outcome	might
occur	anyway.	So	what	might	such	an	outcome	look	like?	The	resulting
competitive	society	is	not	necessarily	attractive,	nor	long-lasting.
In	singleton	scenarios,	what	happens	post-transition	depends	almost	entirely	on

the	values	of	the	singleton.	The	outcome	could	thus	be	very	good	or	very	bad,
depending	on	what	those	values	are.	What	the	values	are	depends,	in	turn,	on
whether	the	control	problem	was	solved,	and—to	the	degree	to	which	it	was
solved—on	the	goals	of	the	project	that	created	the	singleton.
If	one	is	interested	in	the	outcome	of	singleton	scenarios,	therefore,	one	really
only	has	three	sources	of	information:	information	about	matters	that	cannot	be
affected	by	the	actions	of	the	singleton	(such	as	the	laws	of	physics);	information

about	convergent	instrumental	values;	and	information	that	enables	one	to
predict	or	speculate	about	what	final	values	the	singleton	will	have.
In	multipolar	scenarios,	an	additional	set	of	constraints	comes	into	play,
constraints	having	to	do	with	how	agents	interact.	The	social	dynamics	emerging
from	such	interactions	can	be	studied	using	techniques	from	game	theory,
economics,	and	evolution	theory.	Elements	of	political	science	and	sociology	are
also	relevant	insofar	as	they	can	be	distilled	and	abstracted	from	some	of	the

more	contingent	features	of	human	experience.	Although	it	would	be	unrealistic

to	expect	these	constraints	to	give	us	a	precise	picture	of	the	post-transition
world,	
they	can	help	us	identify	some	salient	possibilities	and	challenge	some
unfounded	assumptions.
We	will	begin	by	exploring	an	economic	scenario	characterized	by	a	low	level
of	regulation,	strong	protection	of	property	rights,	and	a	moderately	rapid
introduction	of	inexpensive	digital	minds.
1
	This	type	of	model	is	most	closely
associated	with	the	American	economist	Robin	Hanson,	who	has	done
pioneering	work	on	the	subject.	Later	in	this	chapter,	we	will	look	at	some
evolutionary	considerations	and	examine	the	prospects	of	an	initially	multipolar
post-transition	world	subsequently	coalescing	into	a	singleton.
	
General	machine	intelligence	could	serve	as	a	substitute	for	human	intelligence.
Not	only	could	digital	minds	perform	the	intellectual	work	now	done	by	humans,
but,	once	equipped	with	good	actuators	or	robotic	bodies,	machines	could	also

substitute	for	human	physical	labor.	Suppose	that	machine	workers—which	can
be	quickly	reproduced—become	both	cheaper	and	more	capable	than	human
workers	in	virtually	all	jobs.	What	happens	then?
	
With	cheaply	copyable	labor,	market	wages	fall.	The	only	place	where	humans
would	remain	competitive	may	be	where	customers	have	a	basic	preference	for
work	done	by	humans.	Today,	goods	that	have	been	handcrafted	or	produced	by
indigenous	people	sometimes	command	a	price	premium.	Future	consumers

might	similarly	prefer	human-made	goods	and	human	athletes,	human	artists,
human	lovers,	and	human	leaders	to	functionally	indistinguishable	or	superior
artificial	counterparts.	It	is	unclear,	however,	just	how	widespread	such
preferences	would	be.	If	machine-made	alternatives	were	sufficiently	superior,
perhaps	they	would	be	more	highly	prized.
One	parameter	that	might	be	relevant	to	consumer	choice	is	the	inner	life	of
the	worker	providing	a	service	or	product.	A	concert	audience,	for	instance,

might	like	to	know	that	the	performer	is	consciously	experiencing	the	music	and
the	venue.	Absent	phenomenal	experience,	the	musician	could	be	regarded	as
merely	a	high-powered	jukebox,	albeit	one	capable	of	creating	the	three-
dimensional	appearance	of	a	performer	interacting	naturally	with	the	crowd.
Machines	might	then	be	designed	to	instantiate	the	same	kinds	of	mental	states
that	would	be	present	in	a	human	performing	the	same	task.	Even	with	perfect
replication	of	subjective	experiences,	however,	some	people	might	simply	prefer
organic	work.	Such	preferences	could	also	have	ideological	or	religious	roots.
Just	as	many	Muslims	and	Jews	shun	food	prepared	in	ways	they	classify	as
haram
	or	
treif
,	so	there	might	be	groups	in	the	future	that	eschew	products
whose	manufacture	involved	unsanctioned	use	of	machine	intelligence.
What	hinges	on	this?	To	the	extent	that	cheap	machine	labor	can	substitute	for
human	labor,	human	jobs	may	disappear.	Fears	about	automation	and	job	loss	are

haram
	or	
treif
of	course	not	new.	Concerns	about	technological	unemployment	have	surfaced
periodically,	at	least	since	the	Industrial	Revolution;	and	quite	a	few	professions
have	in	fact	gone	the	way	of	the	English	weavers	and	textile	artisans	who	in	the
early	nineteenth	century	united	under	the	banner	of	the	folkloric	“General	Ludd”
to	fight	against	the	introduction	of	mechanized	looms.	Nevertheless,	although
machinery	and	technology	have	been	substitutes	for	many	particular	types	of
human	labor,	physical	technology	has	on	the	whole	been	a	complement	to	labor.

Average	human	wages	around	the	world	have	been	on	a	long-term	upward	trend,
in	large	part	because	of	such	complementarities.	Yet	what	starts	out	as	a
complement	to	labor	can	at	a	later	stage	become	a	substitute	for	labor.	Horses
were	initially	complemented	by	carriages	and	ploughs,	which	greatly	increased
the	horse’s	productivity.	Later,	horses	were	substituted	for	by	automobiles	and
tractors.	These	later	innovations	reduced	the	demand	for	equine	labor	and	led	to
a	population	collapse.	Could	a	similar	fate	befall	the	human	species?

The	parallel	to	the	story	of	the	horse	can	be	drawn	out	further	if	we	ask	why	it
is	that	there	are	still	horses	around.	One	reason	is	that	there	are	still	a	few	niches
in	which	horses	have	functional	advantages;	for	example,	police	work.	But	the
main	reason	is	that	humans	happen	to	have	peculiar	preferences	for	the	services
that	horses	can	provide,	including	recreational	horseback	riding	and	racing.
These	preferences	can	be	compared	to	the	preferences	we	hypothesized	some
humans	might	have	in	the	future,	that	certain	goods	and	services	be	made	by

human	hand.	Although	suggestive,	this	analogy	is,	however,	inexact,	since	there
is	still	no	complete	functional	substitute	for	horses.	If	there	were	inexpensive
mechanical	devices	that	ran	on	hay	and	had	exactly	the	same	shape,	feel,	smell,

and	behavior	as	biological	horses—perhaps	even	the	same	conscious
experiences—then	demand	for	biological	horses	would	probably	decline	further.
With	a	sufficient	reduction	in	the	demand	for	human	labor,	wages	would	fall
below	the	human	subsistence	level.	The	potential	downside	for	human	workers
is	therefore	extreme:	not	merely	wage	cuts,	demotions,	or	the	need	for	retraining,
but	starvation	and	death.	When	horses	became	obsolete	as	a	source	of	moveable
power,	many	were	sold	off	to	meatpackers	to	be	processed	into	dog	food,	bone
meal,	leather,	and	glue.	These	animals	had	no	alternative	employment	through
which	to	earn	their	keep.	In	the	United	States,	there	were	about	26	million	horses
in	1915.	By	the	early	1950s,	2	million	remained.
2
	
One	difference	between	humans	and	horses	is	that	humans	own	capital.	A
stylized	empirical	fact	is	that	the	total	factor	share	of	capital	has	for	a	long	time
remained	steady	at	approximately	30%	(though	with	significant	short-term
fluctuations).
3

2
	
3
	This	means	that	30%	of	total	global	income	is	received	as	rent	by
owners	of	capital,	the	remaining	70%	being	received	as	wages	by	workers.	If	we
classify	AI	as	capital,	
then	with	the	invention	of	machine	intelligence	that	can
fully	substitute	for	human	work,	wages	would	fall	to	the	marginal	cost	of	such
machine-substitutes,	which—under	the	assumption	that	the	machines	are	very
efficient—would	be	very	low,	far	below	human	subsistence-level	income.	The
income	share	received	by	labor	would	then	dwindle	to	practically	nil.	But	this

implies	that	the	factor	share	of	capital	would	become	nearly	100%	of	total	world
product.	Since	world	GDP	would	soar	following	an	intelligence	explosion
(because	of	massive	amounts	of	new	labor-substituting	machines	but	also
because	of	technological	advances	achieved	by	superintelligence,	and,	later,
acquisition	of	vast	amounts	of	new	land	through	space	colonization),	it	follows
that	the	total	income	from	capital	would	increase	enormously.	If	humans	remain
the	owners	of	this	capital,	the	total	income	received	by	the	human	population

would	grow	astronomically,	despite	the	fact	that	in	this	scenario	humans	would
no	longer	receive	any	wage	income.
The	human	species	as	a	whole	could	thus	become	rich	beyond	the	dreams	of
Avarice.	How	would	this	income	be	distributed?	To	a	first	approximation,	capital
income	would	be	proportional	to	the	amount	of	capital	owned.	Given	the

astronomical	amplification	effect,	even	a	tiny	bit	of	pre-transition	wealth	would
balloon	into	a	vast	post-transition	fortune.	However,	in	the	contemporary	world,
many	people	have	no	wealth.	This	includes	not	only	individuals	who	live	in
poverty	but	also	some	people	who	earn	a	good	income	or	who	have	high	human
capital	but	have	negative	net	worth.	For	example,	in	affluent	Denmark	and
Sweden	30%	of	the	population	report	negative	wealth—often	young,	middle-
class	people	with	few	tangible	assets	and	credit	card	debt	or	student	loans.
4
	Even
if	savings	could	earn	extremely	high	interest,	there	would	need	to	be	some	seed
grain,	some	starting	capital,	in	order	for	the	compounding	to	begin.
5
Nevertheless,	even	individuals	who	have	no	private	wealth	at	the	start	of	the
transition	could	become	extremely	rich.	Those	who	participate	in	a	pension
scheme,	for	instance,	whether	public	or	private,	should	be	in	a	good	position,
provided	the	scheme	is	at	least	partially	funded.
6
	Have-nots	could	also	become

4
	Even
5
6
rich	through	the	philanthropy	of	those	who	see	their	net	worth	skyrocket:
because	of	the	astronomical	size	of	the	bonanza,	even	a	very	small	fraction
donated	as	alms	would	be	a	very	large	sum	in	absolute	terms.
It	is	also	possible	that	riches	could	still	be	made	through	work,	even	at	a	post-
transition	stage	when	machines	are	functionally	superior	to	humans	in	all
domains	(as	well	as	cheaper	than	even	subsistence-level	human	labor).	As	noted
earlier,	this	could	happen	if	there	are	niches	in	which	human	labor	is	preferred

for	aesthetic,	ideological,	ethical,	religious,	or	other	non-pragmatic	reasons.	In	a
scenario	in	which	the	wealth	of	human	capital-holders	increases	dramatically,
demand	for	such	labor	could	increase	correspondingly.	Newly	minted
trillionaires	or	quadrillionaires	could	afford	to	pay	a	hefty	premium	for	having
some	of	their	goods	and	services	supplied	by	an	organic	“fair-trade”	labor	force.
The	history	of	horses	again	offers	a	parallel.	After	falling	to	2	million	in	the
early	1950s,	the	US	horse	population	has	undergone	a	robust	recovery:	a	recent

census	puts	the	number	at	just	under	10	million	head.
7
	The	rise	is	not	due	to	new
functional	needs	
for	horses	in	agriculture	or	transportation;	rather,	economic
growth	has	enabled	more	Americans	to	indulge	a	fancy	for	equestrian	recreation.
Another	relevant	difference	between	humans	and	horses,	beside	capital-
ownership,	is	that	humans	are	capable	of	political	mobilization.	A	human-run
government	could	use	the	taxation	power	of	the	state	to	redistribute	private
profits,	or	raise	revenue	by	selling	appreciated	state-owned	assets,	such	as	public

7
land,	and	use	the	proceeds	to	pension	off	its	constituents.	Again,	because	of	the
explosive	economic	growth	during	and	immediately	after	the	transition,	there

would	be	vastly	more	wealth	sloshing	around,	making	it	relatively	easy	to	fill	the
cups	of	all	unemployed	citizens.	It	should	be	feasible	even	for	a	single	country	to
provide	every	human	worldwide	with	a	generous	living	wage	at	no	greater
proportional	cost	than	what	many	countries	currently	spend	on	foreign	aid.
8
	
So	far	we	have	assumed	a	constant	human	population.	This	may	be	a	reasonable
assumption	for	short	timescales,	since	biology	limits	the	rate	of	human
reproduction.	Over	longer	timescales,	however,	the	assumption	is	not	necessarily
reasonable.
The	human	population	has	increased	a	thousandfold	over	the	past	9,000
years.
9
	The	increase	would	have	been	much	faster	except	for	the	fact	that
throughout	most	of	history	and	prehistory,	the	human	population	was	bumping
up	against	the	limits	of	the	world	economy.	An	approximately	Malthusian
condition	prevailed,	in	which	most	people	received	subsistence-level	incomes

years.
9
that	just	barely	allowed	them	to	survive	and	raise	an	average	of	two	children	to
maturity.
10
	There	were	temporary	and	local	reprieves:	plagues,	climate
fluctuations,	or	warfare	intermittently	culled	the	population	and	freed	up	land,
enabling	survivors	to	improve	their	nutritional	intake—and	to	bring	up	more
children,	until	the	ranks	were	replenished	and	the	Malthusian	condition
reinstituted.	Also,	thanks	to	social	inequality,	a	thin	elite	stratum	could	enjoy
consistently	above-subsistence	income	(at	the	expense	of	somewhat	lowering	the

maturity.
10
total	size	of	the	population	that	could	be	sustained).	A	sad	and	dissonant	thought:
that	in	this	Malthusian	condition,	the	normal	state	of	affairs	during	most	of	our
tenure	on	this	planet,	it	was	droughts,	pestilence,	massacres,	and	inequality—in
common	estimation	the	worst	foes	of	human	welfare—that	may	have	been	the
greatest	humanitarians:	they	alone	enabling	the	average	level	of	well-being	to
occasionally	bop	up	slightly	above	that	of	life	at	the	very	margin	of	subsistence.
Superimposed	on	local	fluctuations,	history	shows	a	macro-pattern	of	initially

slow	but	accelerating	economic	growth,	fueled	by	the	accumulation	of
technological	innovations.	The	growing	world	economy	brought	with	it	a
commensurate	increase	in	global	population.	(More	precisely,	a	larger	population
itself	appears	to	have	strongly	accelerated	the	rate	of	growth,	perhaps	mainly	by

increasing	humanity’s	collective	intelligence.
11
)	Only	since	the	Industrial
Revolution,	however,	did	economic	growth	become	so	rapid	that	population
growth	failed	to	keep	
pace.	Average	income	thus	started	to	rise,	first	in	the	early-
industrializing	countries	of	Western	Europe,	subsequently	in	most	of	the	world.
Even	in	the	poorest	countries	today,	average	income	substantially	exceeds
subsistence	level,	as	reflected	in	the	fact	that	the	populations	of	these	countries
are	growing.
The	poorest	countries	now	have	the	fastest	population	growth,	as	they	have
yet	to	complete	the	“demographic	transition”	to	the	low-fertility	regime	that	has
taken	hold	in	more	developed	societies.	Demographers	project	that	the	world
population	will	rise	to	about	9	billion	by	mid-century,	and	that	it	might	thereafter
plateau	or	decline	as	the	poorer	countries	join	the	developed	world	in	this	low-
fertility	regime.
12
	Many	rich	countries	already	have	fertility	rates	that	are	below
replacement	level;	in	some	cases,	far	below.
13

12
13
Yet	there	are	reasons,	if	we	take	a	longer	view	and	assume	a	state	of
unchanging	technology	and	continued	prosperity,	to	expect	a	return	to	the
historically	and	ecologically	normal	condition	of	a	world	population	that	butts
up	against	the	limits	of	what	our	niche	can	support.	If	this	seems	counterintuitive
in	light	of	the	negative	relationship	between	wealth	and	fertility	that	we	are
currently	observing	on	the	global	scale,	we	must	remind	ourselves	that	this
modern	age	is	a	brief	slice	of	history	and	very	much	an	aberration.	Human

behavior	has	not	yet	adapted	to	contemporary	conditions.	Not	only	do	we	fail	to
take	advantage	of	obvious	ways	to	increase	our	inclusive	fitness	(such	as	by
becoming	sperm	or	egg	donors)	but	we	actively	sabotage	our	fertility	by	using
birth	control.	In	the	environment	of	evolutionary	adaptedness,	a	healthy	sex
drive	may	have	been	enough	to	make	an	individual	act	in	ways	that	maximized
her	reproductive	potential;	in	the	modern	environment,	however,	there	would	be
a	huge	selective	advantage	to	having	a	more	direct	desire	for	being	the	biological

parent	to	the	largest	possible	number	of	children.	Such	a	desire	is	currently	being
selected	for,	as	are	other	traits	that	increase	our	propensity	to	reproduce.	Cultural
adaptation,	however,	might	steal	a	march	on	biological	evolution.	Some
communities,	such	those	of	the	Hutterites	or	the	adherents	of	the	Quiverfull
evangelical	movement,	have	natalist	cultures	that	encourage	large	families,	and
they	are	consequently	undergoing	rapid	expansion.

If	we	imagine	current	socioeconomic	conditions	magically	frozen	in	their	current
shape,	the	future	would	be	dominated	by	cultural	or	ethnic	groups	that	sustain
high	levels	of	fertility.	If	most	people	had	preferences	that	were	fitness-
maximizing	in	the	contemporary	environment,	the	population	could	easily
double	in	each	generation.	Absent	population	control	policies—which	would
have	to	become	steadily	more	rigorous	and	effective	to	counteract	the	evolution
of	stronger	preferences	to	circumvent	them—the	world	population	would	then
continue	to	grow	exponentially	until	some	constraint,	such	as	land	scarcity	or
depletion	of	easy	opportunities	for	important	innovation,	made	it	impossible	for
the	economy	to	keep	pace:	at	which	point,	average	income	would	start	to	decline
until	
it	reached	the	level	where	crushing	poverty	prevents	most	people	from
raising	much	more	than	two	children	to	maturity.	Thus	the	Malthusian	principle
would	reassert	itself,	like	a	dread	slave	master,	bringing	our	escapade	into	the

until	
dreamland	of	abundance	to	an	end,	and	leading	us	back	to	the	quarry	in	chains,
there	to	resume	the	weary	struggle	for	subsistence.
This	longer-term	outlook	could	be	telescoped	into	a	more	imminent	prospect
by	the	intelligence	explosion.	Since	software	is	copyable,	a	population	of
emulations	or	AIs	could	double	rapidly—over	the	course	of	minutes	rather	than
decades	or	centuries—soon	exhausting	all	available	hardware.
Private	property	might	offer	partial	protection	against	the	emergence	of	a

universal	Malthusian	condition.	Consider	a	simple	model	in	which	clans	(or
closed	communities,	or	states)	start	out	with	varying	amounts	of	property	and
independently	adopt	different	policies	about	reproduction	and	investment.	Some
clans	discount	the	future	steeply	and	spend	down	their	endowment,	whereafter
their	impoverished	members	join	the	global	proletariat	(or	die,	if	they	cannot
support	themselves	through	their	labor).	Other	clans	invest	some	of	their

resources	but	adopt	a	policy	of	unlimited	reproduction:	such	clans	grow	more
populous	until	they	reach	an	internal	Malthusian	condition	in	which	their
members	are	so	poor	that	they	die	at	almost	the	same	rate	as	they	reproduce,	at
which	point	the	clan’s	population	growth	slows	to	equal	the	growth	of	its
resources.	Yet	other	clans	might	restrict	their	fertility	to	below	the	rate	of	growth
of	their	capital:	such	clans	could	slowly	increment	their	numbers	while	their
members	also	grow	richer	per	capita.

If	wealth	is	redistributed	from	the	wealthy	clans	to	the	members	of	the	rapidly

reproducing	or	rapidly	discounting	clans	(whose	children,	copies,	or	offshoots,
through	no	fault	of	their	own,	were	launched	into	the	world	with	insufficient
capital	to	survive	and	thrive)	then	a	universal	Malthusian	condition	would	be
more	closely	approximated.	In	the	limiting	case,	all	members	of	all	clans	would
receive	subsistence	level	income	and	everybody	would	be	equal	in	their	poverty.
If	property	is	not	redistributed,	prudent	clans	might	hold	on	to	a	certain
amount	of	capital,	and	it	is	possible	that	their	wealth	could	grow	in	absolute
terms.	It	is,	however,	unclear	whether	humans	could	earn	as	high	rates	of	return
on	their	capital	as	machine	intelligences	could	earn	on	theirs,	because	there	may
be	synergies	between	labor	and	capital	such	that	an	single	agent	who	can	supply
both	(e.g.	an	entrepreneur	or	investor	who	is	both	skilled	and	wealthy)	can	attain
a	private	rate	of	return	on	her	capital	exceeding	the	market	rate	obtainable	by

agents	who	possess	financial	but	not	cognitive	resources.	Humans,	being	less
skilled	than	machine	intelligences,	may	therefore	grow	their	capital	more	slowly
—unless,	of	course,	the	control	problem	had	been	completely	solved,	in	which
case	the	human	rate	of	return	would	equal	the	machine	rate	of	return,	since	a
human	principal	could	task	a	machine	agent	to	manage	her	savings,	and	could	do
so	costlessly	and	without	conflicts	of	interest:	but	otherwise,	in	this	scenario,	the

fraction	of	the	economy	owned	by	machines	would	asymptotically	approach	one
hundred	percent.
A	scenario	in	which	the	fraction	of	the	economy	that	is	owned	by	machines
asymptotically	approaches	one	hundred	percent	is	not	necessarily	one	in	which
the	size	of	the	human	slice	declines.	If	the	economy	grows	at	a	sufficient	clip,
then	even	a	relatively	diminishing	fraction	of	it	may	still	be	increasing	in	its
absolute	size.	This	may	sound	like	modestly	good	news	for	humankind:	in	a

multipolar	scenario	in	which	property	rights	are	protected—even	if	we
completely	fail	to	solve	the	control	problem—the	total	amount	of	wealth	owned
by	human	beings	could	increase.	Of	course,	this	effect	would	not	take	care	of	the
problem	of	population	growth	in	the	human	population	pulling	down	per	capita
income	to	subsistence	level,	nor	the	problem	of	humans	who	ruin	themselves
because	they	discount	the	future.
In	the	long	run,	the	economy	would	become	increasingly	dominated	by	those

clans	that	have	the	highest	savings	rates—misers	who	own	half	the	city	and	live
under	a	bridge.	Only	in	the	fullness	of	time,	when	there	are	no	more
opportunities	for	investment,	would	the	maximally	prosperous	misers	start
drawing	down	their	savings.
14
	However,	if	there	is	less	than	perfect	protection

for	property	rights—for	example	if	the	more	efficient	machines	on	net	succeed,
by	hook	or	by	crook,	in	transferring	wealth	from	humans	to	themselves—then
human	capitalists	may	need	to	spend	down	their	capital	much	sooner,	before	it
gets	depleted	by	such	transfers	(or	the	ongoing	costs	incurred	in	securing	their
wealth	against	such	transfers).	If	these	developments	take	place	on	digital	rather
than	biological	timescales,	then	the	glacial	humans	might	find	themselves
expropriated	before	they	could	say	Jack	Robinson.
15
	
Life	for	biological	humans	in	a	post-transition	Malthusian	state	need	not
resemble	any	of	the	historical	states	of	man	(as	hunter–gatherer,	farmer,	or	office
worker).	Instead,	the	majority	of	humans	in	this	scenario	might	be	idle	rentiers
who	eke	out	a	marginal	living	on	their	savings.
16
	They	would	be	very	poor,	yet
derive	what	little	income	they	have	from	savings	or	state	subsidies.	They	would

15
	
16
live	in	a	world	with	extremely	advanced	technology,	including	not	only
superintelligent	machines	but	also	anti-aging	medicine,	virtual	reality,	and
various	enhancement	technologies	and	pleasure	drugs:	yet	these	might	be
generally	unaffordable.	Perhaps	instead	of	using	enhancement	medicine,	they
would	take	drugs	to	stunt	their	growth	and	slow	their	metabolism	in	order	to
reduce	their	cost	of	living	(fast-burners	being	unable	to	survive	at	the	gradually
declining	subsistence	income).	As	our	numbers	increase	and	our	average	income

declines	further,	we	might	degenerate	into	whatever	minimal	structure	still
qualifies	to	receive	a	pension—perhaps	minimally	conscious	brains	in	vats,
oxygenized	and	nourished	by	machines,	slowly	saving	up	enough	money	to
reproduce	by	having	a	robot	technician	develop	a	clone	of	them.
17
Further	frugality	could	be	achieved	by	means	of	uploading,	since	a	physically
optimized	computing	substrate,	devised	by	advanced	superintelligence,	would	
be
more	efficient	than	a	biological	brain.	The	migration	into	the	digital	realm	might

17
be
be	stemmed,	however,	if	emulations	were	regarded	as	non-humans	or	non-
citizens	ineligible	to	receive	pensions	or	to	hold	tax-exempt	savings	accounts.	In
that	case,	a	niche	for	biological	humans	might	remain	open,	alongside	a	perhaps
vastly	larger	population	of	emulations	or	artificial	intelligences.
So	far	we	have	focused	on	the	fate	of	the	humans,	who	may	be	supported	by
savings,	subsidies,	or	wage	income	deriving	from	other	humans	who	prefer	to

hire	humans.	Let	us	now	turn	our	attention	to	some	of	the	entities	that	we	have
so	far	classified	as	“capital”:	machines	that	may	be	owned	by	human	beings,	that
are	constructed	and	operated	for	the	sake	of	the	functional	tasks	they	perform,
and	that	are	capable	of	substituting	for	human	labor	in	a	very	wide	range	of	jobs.
What	may	the	situation	be	like	for	these	workhorses	of	the	new	economy?
If	these	machines	were	mere	automata,	simple	devices	like	a	steam	engine	or
the	mechanism	in	a	clock,	then	no	further	comment	would	be	needed:	there
would	be	a	large	amount	of	such	capital	in	a	post-transition	economy,	but	it
would	seem	not	to	matter	to	anybody	how	things	turn	out	for	pieces	of	insentient
equipment.	However,	if	the	machines	have	conscious	minds—if	they	are
constructed	in	such	a	way	that	their	operation	is	associated	with	phenomenal
awareness	(or	if	they	for	some	other	reason	are	ascribed	moral	status)—then	it
becomes	important	to	consider	the	overall	outcome	in	terms	of	how	it	would

affect	these	machine	minds.	The	welfare	of	the	working	machine	minds	could
even	appear	to	be	the	most	important	aspect	of	the	outcome,	since	they	may	be
numerically	dominant.
	
A	salient	initial	question	is	whether	these	working	machine	minds	are	owned	as
capital	(slaves)	or	are	hired	as	free	wage	laborers.	On	closer	inspection	however,
it	become	doubtful	that	anything	really	hinges	on	the	issue.	There	are	two
reasons	for	this.	First,	if	a	free	worker	in	a	Malthusian	state	gets	paid	a

subsistence-level	wage,	he	will	have	no	disposable	income	left	after	he	has	paid
for	food	and	other	necessities.	If	the	worker	is	instead	a	slave,	his	owner	will	pay
for	his	maintenance	and	again	he	will	have	no	disposable	income.	In	either	case,
the	worker	gets	the	necessities	and	nothing	more.	Second,	suppose	that	the	free
laborer	were	somehow	in	a	position	to	command	an	above-subsistence-level
income	(perhaps	because	of	favorable	regulation).	How	will	he	spend	the
surplus?	Investors	would	find	it	most	profitable	to	create	workers	who	would	be

“voluntary	slaves”—who	would	willingly	work	for	subsistence-level	wages.
Investors	may	create	such	workers	by	copying	those	workers	who	are	compliant.
With	appropriate	selection	(and	perhaps	some	modification	to	the	code)	investors
might	be	able	to	create	workers	who	not	only	prefer	to	volunteer	their	labor	but
who	would	also	choose	to	donate	back	to	their	owners	any	surplus	income	they
might	happen	to	receive.	Giving	money	to	the	worker	would	then	be	but	a

roundabout	way	of	giving	money	to	the	owner	or	employer,	even	if	the	worker
were	a	free	agent	with	full	legal	rights.
Perhaps	it	will	be	objected	that	it	would	be	difficult	to	design	a	machine	so
that	it	wants	to	volunteer	for	any	job	assigned	to	it	or	so	that	it	wants	to	donate
its	wages	to	its	owner.	Emulations,	in	particular,	might	be	imagined	to	have	more
typically	human	desires.	But	note	that	even	if	the	original	control	problem	is
difficult,	we	are	here	considering	a	condition	
after
	the	transition,	a	time	when
methods	for	motivation	selection	have	presumably	been	perfected.	In	the	case	of
emulations,	one	might	get	quite	far	simply	by	
selecting
	from	the	pre-existing
range	of	human	characters;	and	we	have	described	several	other	motivation
selection	methods.	The	control	problem	may	also	in	some	ways	be	simplified	by
the	current	assumption	that	the	new	machine	intelligence	enters	into	a	stable
socioeconomic	matrix	that	is	already	populated	with	other	law-abiding
superintelligent	agents.

selecting
Let	us,	then,	consider	the	plight	of	the	working-class	machine,	whether	it	be
operating	as	a	slave	or	a	free	agent.	We	focus	first	on	emulations,	the	easiest	case
to	imagine.
Bringing	a	new	biological	human	worker	into	the	world	takes	anywhere
between	fifteen	and	thirty	years,	depending	on	how	much	expertise	and
experience	is	required.	During	this	time	the	new	person	must	be	fed,	housed,
nurtured,	and	educated—at	great	expense.	By	contrast,	spawning	a	new	copy	of

a	digital	worker	is	as	easy	as	loading	a	new	program	into	working	memory.	Life
thus	becomes	cheap.	A	business	could	continuously	adapt	its	workforce	to	fit
demands	by	spawning	new	copies—and	terminating	copies	that	are	no	longer
needed,	to	free	up	computer	resources.	This	could	lead	to	an	extremely	high
death	rate	among	digital	workers.	Many	might	live	for	only	one	subjective	day.
There	are	reasons	other	than	fluctuations	in	demand	why	employers	or	owners
of	emulations	might	want	to	“kill”	or	“end”	their	workers	frequently.

18
	If	an
emulation	mind,	like	a	biological	mind,	requires	periods	of	rest	and	sleep	in
order	to	function,	it	might	be	cheaper	to	erase	a	fatigued	emulation	at	the	end	of
a	day	and	replace	it	with	a	stored	state	of	a	fresh	and	rested	emulation.	As	this
procedure	would	cause	retrograde	amnesia	for	everything	that	had	been	learned
during	that	day,	emulations	performing	tasks	requiring	long	cognitive	threads
would	be	spared	such	frequent	erasure.	It	would	be	difficult,	for	example,	to
write	a	book	if	each	morning	when	one	sat	down	at	one’s	desk,	one	had	no

memory	of	what	one	had	done	before.	But	other	jobs	could	be	performed
adequately	by	agents	that	are	frequently	recycled:	a	shop	assistant	or	a	customer
service	agent,	once	trained,	may	only	need	to	remember	new	information	for
twenty	minutes.
Since	recycling	emulations	would	prevent	memory	and	skill	formation,	some
emulations	may	be	placed	on	a	special	learning	track	where	they	would	run
continuously,	including	for	rest	and	sleep,	even	in	jobs	that	do	not	strictly	require
long	cognitive	threads.	For	example,	some	customer	service	agents	might	run	for
many	years	in	optimized	learning	environments,	assisted	by	coaches	and
performance	evaluators.	The	best	of	these	trainees	would	then	be	used	like	studs,
serving	as	templates	from	which	millions	of	fresh	copies	are	stamped	out	each
day.	Great	
effort	would	be	poured	into	improving	the	performance	of	such
worker	templates,	because	even	a	small	increment	in	productivity	would	yield
great	economic	value	when	applied	in	millions	of	copies.

day.	Great	
In	parallel	with	efforts	to	train	worker-templates	for	particular	jobs,	intense
efforts	would	also	be	made	to	improve	the	underlying	emulation	technology.
Advances	here	would	be	even	more	valuable	than	advances	in	individual
worker-templates,	since	general	technology	improvements	could	be	applied	to
all	emulation	workers	(and	potentially	to	non-worker	emulations	also)	rather
than	only	to	those	in	a	particular	occupation.	Enormous	resources	would	be
devoted	to	finding	computational	shortcuts	allowing	for	more	efficient

implementations	of	existing	emulations,	and	also	into	developing	neuromorphic
and	entirely	synthetic	AI	architectures.	This	research	would	probably	mostly	be
done	by	emulations	running	on	very	fast	hardware.	Depending	on	the	price	of
computer	power,	millions,	billions,	or	trillions	of	emulations	of	the	sharpest
human	research	minds	(or	enhanced	versions	thereof)	may	be	working	around
the	clock	on	advancing	the	frontier	of	machine	intelligence;	and	some	of	these
may	be	operating	orders	of	magnitude	faster	than	biological	brains.
19
	This	is	a

19
	This	is	a
good	reason	for	thinking	that	the	era	of	human-like	emulations	would	be	brief—
a	
very
	brief	interlude	in	sidereal	time—and	that	it	would	soon	give	way	to	an	era
of	greatly	superior	artificial	intelligence.
We	have	already	encountered	several	reasons	why	employers	of	emulation
workers	may	periodically	cull	their	herds:	fluctuations	in	demand	for	different
kinds	of	laborers,	cost	savings	of	not	having	to	emulate	rest	and	sleep	time,	and
the	introduction	of	new	and	improved	templates.	Security	concerns	might

a	
very
furnish	another	reason.	To	prevent	workers	from	developing	subversive	plans

and	conspiracies,	emulations	in	some	sensitive	positions	might	be	run	only	for
limited	periods,	with	frequent	resets	to	an	earlier	stored	ready-state.
20
These	ready-states	to	which	emulations	would	be	reset	would	be	carefully
prepared	and	vetted.	A	typical	short-lived	emulation	might	wake	up	in	a	well-
rested	mental	state	that	is	optimized	for	loyalty	and	productivity.	He	remembers
having	graduated	top	of	his	class	after	many	(subjective)	years	of	intense
training	and	selection,	then	having	enjoyed	a	restorative	holiday	and	a	good
night’s	sleep,	then	having	listened	to	a	rousing	motivational	speech	and	stirring
music,	and	now	he	is	champing	at	the	bit	to	finally	get	to	work	and	to	do	his
utmost	for	his	employer.	He	is	not	overly	troubled	by	thoughts	of	his	imminent
death	at	the	end	of	the	working	day.	Emulations	with	death	neuroses	or	other
hang-ups	are	less	productive	and	would	not	have	been	selected.
21

21
	
One	important	variable	in	assessing	the	desirability	of	a	hypothetical	condition
like	this	is	the	hedonic	state	of	the	average	emulation.
22
	Would	a	typical
emulation	worker	be	suffering	or	would	he	be	enjoying	the	experience	of
working	hard	on	the	task	at	hand?
We	must	resist	the	temptation	to	project	our	own	sentiments	onto	the
imaginary	emulation	worker.	The	question	is	not	whether	
you
	would	feel	happy
if	you	had	to	work	constantly	and	never	again	spend	time	with	your	loved	ones

22
you
—a	terrible	fate,	most	would	agree.
It	is	moderately	more	relevant	to	consider	the	current	human	average	hedonic
experience	during	working	hours.	Worldwide	studies	asking	respondents	how
happy	they	are	find	that	most	rate	themselves	as	“quite	happy”	or	“very	happy”
(averaging	3.1	on	a	scale	from	1	to	4).
23
	Studies	on	average	affect,	asking
respondents	how	frequently	they	have	recently	experienced	various	positive	or
negative	affective	states,	tend	to	get	a	similar	result	(producing	a	net	affect	of

23
about	0.52	on	a	scale	from	–1	to	1).	There	is	a	modest	positive	effect	of	a
country’s	per	capita	income	on	average	subjective	well-being.
24
	However,	it	is
hazardous	to	extrapolate	from	these	findings	to	the	hedonic	state	of	future
emulation	workers.	One	reason	that	could	be	given	for	this	is	that	their	condition
would	be	so	different:	on	the	one	hand,	they	might	be	working	much	harder;	on

the	other	hand,	they	might	be	free	from	diseases,	aches,	hunger,	noxious	odors,
and	so	forth.	Yet	such	considerations	largely	miss	the	mark.	The	much	more
important	consideration	here	is	that	hedonic	tone	would	be	easy	to	adjust
through	the	digital	equivalent	of	drugs	or	neurosurgery.	This	means	that	it	would
be	a	mistake	to	infer	the	hedonic	state	of	future	emulations	from	the	external
conditions	of	their	lives	by	imagining	how	we	ourselves	and	other	people	like	us
would	feel	in	those	circumstances.	Hedonic	state	would	be	a	matter	of	choice.	In
the	model	we	are	currently	considering,	the	choice	would	be	made	by	capital-
owners	seeking	to	maximize	returns	on	their	investment	in	emulation-workers.
Consequently,	the	question	of	how	happy	emulations	would	feel	boils	down	to
the	question	of	which	hedonic	states	would	be	most	productive	(in	the	various
jobs	that	emulations	would	be	employed	to	do).
Here,	again,	one	might	seek	to	draw	an	inference	from	observations	about

human	happiness.	If	it	is	the	case,	across	most	times,	places,	and	occupations,
that	people	are	typically	at	least	moderately	happy,	this	would	create	some
presumption	in	favor	of	the	same	holding	in	a	post-transition	scenario	like	the
one	we	are	considering.	To	be	clear,	the	argument	in	this	case	would	not	be	that
human	minds	have	a	predisposition	towards	happiness	so	they	would	probably
find	satisfaction	under	these	novel	conditions;	but	rather	that	a	certain	average

level	of	happiness	has	proved	adaptive	for	human	minds	in	the	past	so	maybe	a
similar	level	of	happiness	will	prove	adaptive	for	human-like	minds	in	the
future.	Yet	this	formulation	also	reveals	the	weakness	of	the	inference:	to	wit,
that	the	mental	dispositions	that	were	adaptive	for	hunter–gatherer	hominids
roaming	the	African	savanna	may	not	necessarily	be	adaptive	for	modified
emulations	living	in	post-transition	virtual	realities.	We	can	certainly	
hope
	that
the	future	emulation-workers	would	be	as	happy	as,	or	happier	than,	typical

hope
	that
workers	were	in	human	history;	but	we	have	yet	to	see	any	compelling	reason
for	supposing	it	would	be	so	(in	the	laissez-faire	multipolar	scenario	currently
under	examination).
Consider	the	possibility	that	the	reason	happiness	is	prevalent	among	humans
(to	whatever	limited	extent	it	is	prevalent)	is	that	cheerful	mood	served	a
signaling	
function	in	the	environment	of	evolutionary	adaptedness.	Conveying
the	impression	to	other	members	of	the	social	group	of	being	in	flourishing

signaling	
condition—in	good	health,	in	good	standing	with	one’s	peers,	and	in	confident
expectation	of	continued	good	fortune—may	have	boosted	an	individual’s
popularity.	A	bias	toward	cheerfulness	could	thus	have	been	selected	for,	with
the	result	that	human	neurochemistry	is	now	biased	toward	positive	affect

compared	to	what	would	have	been	maximally	efficient	according	to	simpler
materialistic	criteria.	If	this	were	the	case,	then	the	future	of	
joie	de	vivre
	might
depend	on	cheer	retaining	its	social	signaling	function	unaltered	in	the	post-
transition	world:	an	issue	to	which	we	will	return	shortly.
What	if	glad	souls	dissipate	more	energy	than	glum	ones?	Perhaps	the	joyful
are	more	prone	to	creative	leaps	and	flights	of	fancy—behaviors	that	future
employers	might	disprize	in	most	of	their	workers.	Perhaps	a	sullen	or	anxious
fixation	on	simply	getting	on	with	the	job	without	making	mistakes	will	be	the
productivity-maximizing	attitude	in	most	lines	of	work.	The	claim	here	is	not
that	this	is	so,	but	that	we	do	not	know	that	it	is	not	so.	Yet	we	should	consider
just	how	bad	it	could	be	if	some	such	pessimistic	hypothesis	about	a	future
Malthusian	state	turned	out	to	be	true:	not	only	because	of	the	opportunity	cost
of	having	failed	to	create	something	better—which	would	be	enormous—but

also	because	the	state	could	be	bad	in	itself,	possibly	far	worse	than	the	original
Malthusian	state.
We	seldom	put	forth	full	effort.	When	we	do,	it	is	sometimes	painful.	Imagine
running	on	a	treadmill	at	a	steep	incline—heart	pounding,	muscles	aching,	lungs
gasping	for	air.	A	glance	at	the	timer:	your	next	break,	which	will	also	be	your
death,	is	due	in	49	years,	3	months,	20	days,	4	hours,	56	minutes,	and	12
seconds.	You	wish	you	had	not	been	born.
Again	the	claim	is	not	that	this	is	how	it	would	be,	but	that	we	do	not	know

that	it	is	not.	One	could	certainly	make	a	more	optimistic	case.	For	example,
there	is	no	obvious	reason	that	emulations	would	need	to	suffer	bodily	injury	and
sickness:	the	elimination	of	physical	wretchedness	would	be	a	great
improvement	over	the	present	state	of	affairs.	Furthermore,	since	such	stuff	as
virtual	reality	is	made	of	can	be	fairly	cheap,	emulations	may	work	in	sumptuous
surroundings—in	splendid	mountaintop	palaces,	on	terraces	set	in	a	budding
spring	forest,	or	on	the	beaches	of	an	azure	lagoon—with	just	the	right

illumination,	temperature,	scenery	and	décor;	free	from	annoying	fumes,	noises,
drafts,	and	buzzing	insects;	dressed	in	comfortable	clothing,	feeling	clean	and
focused,	and	well	nourished.	More	significantly,	if—as	seems	perfectly	possible
—the	optimum	human	mental	state	for	productivity	in	most	jobs	is	one	of	joyful
eagerness,	then	the	era	of	the	emulation	economy	could	be	quite	paradisiacal.
There	would,	in	any	case,	be	a	great	option	value	in	arranging	matters	in	such
a	manner	that	somebody	or	something	could	intervene	to	set	things	right	if	the

default	trajectory	should	happen	to	veer	toward	dystopia.	It	could	also	be
desirable	to	have	some	sort	of	escape	hatch	that	would	permit	bailout	into	death
and	oblivion	if	the	quality	of	life	were	to	sink	permanently	below	the	level	at
which	annihilation	becomes	preferable	to	continued	existence.
	
In	the	longer	run,	as	the	emulation	era	gives	way	to	an	artificial	intelligence	era
(or	if	machine	intelligence	is	attained	directly	via	AI	without	a	preceding	whole
brain	emulation	stage)	pain	and	pleasure	might	possibly	disappear	entirely	in	a
multipolar	outcome,	since	a	hedonic	reward	mechanism	may	not	be	the	most
effective	motivation	system	for	an	complex	artificial	agent	(one	that,	unlike	the
human	mind,	is	not	burdened	with	the	legacy	of	animal	wetware).	Perhaps	a
more	advanced	motivation	system	would	be	based	on	an	explicit	representation
of	a	utility	function	or	some	other	architecture	that	has	no	exact	functional
analogs	to	pleasure	and	pain.

A	related	but	slightly	more	radical	multipolar	outcome—one	that	could
involve	the	elimination	of	almost	all	value	from	the	future—is	that	the	universal
proletariat	would	not	even	be	conscious.	This	possibility	is	most	salient	with
respect	to	AI,	which	might	be	structured	very	differently	than	human
intelligence.	But	even	if	machine	intelligence	were	initially	achieved	though
whole	brain	emulation,	resulting	in	conscious	digital	minds,	the	competitive

forces	unleashed	in	a	post-transition	economy	could	easily	lead	to	the	emergence
of	progressively	less	neuromorphic	forms	of	machine	intelligence,	either	because
synthetic	AI	is	created	de	novo	or	because	the	emulations	would,	through
successive	modifications	and	enhancements,	increasingly	depart	their	original
human	form.
Consider	a	scenario	in	which	after	emulation	technology	has	been	developed,
continued	progress	in	neuroscience	and	computer	science	(expedited	by	the

presence	of	digital	minds	to	serve	as	both	researchers	and	test	subjects)	makes	it
possible	to	isolate	individual	cognitive	modules	in	an	emulation,	and	to	hook
them	up	to	modules	isolated	from	other	emulations.	A	period	of	training	and
adjustment	may	be	required	before	different	modules	can	collaborate	effectively;
but	modules	that	conform	to	common	standards	could	more	quickly	interface
with	other	standard	modules.	This	would	make	standardized	modules	more

productive,	and	create	pressure	for	more	standardization.
Emulations	can	now	begin	to	outsource	increasing	portions	of	their
functionality.	Why	learn	arithmetic	when	you	can	send	your	numerical	reasoning
task	to	Gauss-Modules,	Inc.?	Why	be	articulate	when	you	can	hire	Coleridge
Conversations	to	put	your	thoughts	into	words?	Why	make	decisions	about	your
personal	life	when	there	are	certified	executive	modules	that	can	scan	your	goal
system	and	manage	your	resources	to	achieve	your	goals	better	than	if	you	tried
to	do	it	yourself?	Some	emulations	may	prefer	to	retain	most	of	their
functionality	and	handle	tasks	themselves	that	could	be	done	more	efficiently	by
others.	Those	emulations	would	be	like	hobbyists	who	enjoy	growing	their	own
vegetables	or	knitting	their	own	cardigans.	Such	hobbyist	emulations	would	be
less	efficient;	and	if	there	is	a	net	flow	of	resources	from	less	to	more	efficient
participants	of	the	economy,	the	hobbyists	would	eventually	lose	out.

The	bouillon	cubes	of	discrete	human-like	intellects	thus	melt	into	an
algorithmic	soup.
It	is	conceivable	that	optimal	efficiency	would	be	attained	by	grouping
capabilities	in	aggregates	that	roughly	match	the	cognitive	architecture	of	a
human	mind.	It	might	be	the	case,	for	example,	that	a	mathematics	module	must
be	tailored	to	a	language	module,	and	that	both	must	be	tailored	to	the	executive
module,	in	order	for	the	three	to	work	together.	Cognitive	outsourcing	would
then	be	almost	entirely	unworkable.	But	in	the	absence	of	any	compelling	reason

for	being	confident	that	this	is	so,	we	must	countenance	the	possibility	that
human-like	cognitive	architectures	are	optimal	only	within	the	constraints	of
human	neurology	(or	not	at	all).	When	it	becomes	possible	to	build	architectures
that	could	not	be	implemented	well	on	biological	neural	networks,	new	design
space	opens	up;	and	the	global	optima	in	this	extended	space	need	not	resemble
familiar	types	of	mentality.	Human-like	cognitive	organizations	would	then	lack
a	niche	in	a	competitive	post-transition	economy	or	ecosystem.
25

25
There	might	be	niches	for	complexes	that	are	either	less	complex	(such	as
individual	modules),	more	complex	(such	as	vast	clusters	of	modules),	or	of
similar	complexity	to	human	minds	but	with	radically	different	architectures.
Would	these	complexes	have	any	intrinsic	value?	Should	we	welcome	a	world	in
which	such	alien	complexes	have	replaced	human	complexes?
The	answer	may	depend	on	the	specific	nature	of	those	alien	complexes.	The

present	world	has	many	levels	of	organization.	Some	highly	complex	entities,
such	as	multinational	corporations	and	nation	states,	contain	human	beings	as
constituents;	yet	we	usually	assign	these	high-level	complexes	only	instrumental
value.	Corporations	and	states	do	not	(it	is	generally	assumed)	have
consciousness,	over	and	above	the	consciousness	of	the	people	who	constitute
them:	they	cannot	feel	phenomenal	pain	or	pleasure	or	experience	any	qualia.
We	value	them	to	the	extent	that	they	serve	human	needs,	and	when	they	cease
to	do	so	we	“kill”	them	without	compunction.	There	are	also	lower-level	entities,
and	those,	too,	are	usually	denied	moral	status.	We	see	no	harm	in	erasing	an	app
from	a	smartphone,	and	we	do	not	think	that	a	neurosurgeon	is	wronging	anyone
when	she	extirpates	a	malfunctioning	module	from	an	epileptic	brain.	As	for
exotically	organized	complexes	of	a	level	similar	to	that	of	the	human	brain,
most	of	us	would	perhaps	judge	them	to	have	moral	significance	only	if	we

thought	they	had	a	capacity	or	potential	for	conscious	experience.
26
We	could	thus	imagine,	as	an	extreme	case,	a	technologically	highly	advanced
society,	containing	many	complex	structures,	some	of	them	far	more	intricate
and	intelligent	than	anything	that	exists	on	the	planet	today—a	society	which
nevertheless	lacks	any	type	of	being	that	is	conscious	or	whose	welfare	has
moral	significance.	In	a	sense,	this	would	be	an	uninhabited	society.	It	would	be
a	society	of	economic	miracles	and	technological	awesomeness,	with	nobody

26
there	to	benefit.	A	Disneyland	without	children.
	
The	word	“evolution”	is	often	used	as	a	synonym	of	“progress,”	perhaps
reflecting	a	common	uncritical	image	of	evolution	as	a	force	for	good.	A
misplaced	faith	
in	the	inherent	beneficence	of	the	evolutionary	process	can	get	in
the	way	of	a	fair	evaluation	of	the	desirability	of	a	multipolar	outcome	in	which
the	future	of	intelligent	life	is	determined	by	competitive	dynamics.	Any	such
evaluation	must	rest	on	some	(at	least	implicit)	opinion	about	the	probability

distribution	of	different	phenotypes	turning	out	to	be	adaptive	in	a	post-transition
digital	life	soup.	It	would	be	difficult	in	the	best	of	circumstances	to	extract	a
clear	and	correct	answer	from	the	unavoidable	goo	of	uncertainty	that	pervades
these	matters:	more	so,	if	we	superadd	a	layer	of	Panglossian	muck.
A	possible	source	for	faith	in	freewheeling	evolution	is	the	apparent	upward

directionality	exhibited	by	the	evolutionary	process	in	the	past.	Starting	from
rudimentary	replicators,	evolution	produced	increasingly	“advanced”	organisms,
including	creatures	with	minds,	consciousness,	language,	and	reason.	More
recently,	cultural	and	technological	processes,	which	bear	some	loose	similarities
to	biological	evolution,	have	enabled	humans	to	develop	at	an	accelerated	pace.
On	a	geological	as	well	as	a	historical	timescale,	the	big	picture	seems	to	show
an	overarching	trend	toward	increasing	levels	of	complexity,	knowledge,
consciousness,	and	coordinated	goal-directed	organization:	a	trend	which,	not	to
put	too	fine	a	point	on	it,	one	might	label	“progress.”
27
The	image	of	evolution	as	a	process	that	reliably	produces	benign	effects	is
difficult	to	reconcile	with	the	enormous	suffering	that	we	see	in	both	the	human
and	the	natural	world.	Those	who	cherish	evolution’s	achievements	may	do	so
more	from	an	aesthetic	than	an	ethical	perspective.	Yet	the	pertinent	question	is

27
not	what	kind	of	future	it	would	be	fascinating	to	read	about	in	a	science	fiction
novel	or	to	see	depicted	in	a	nature	documentary,	but	what	kind	of	future	it
would	be	good	to	live	in:	two	very	different	matters.
Furthermore,	we	have	no	reason	to	think	that	whatever	progress	there	has
been	was	in	any	way	inevitable.	Much	might	have	been	luck.	This	objection
derives	support	from	the	fact	that	an	observation	selection	effect	filters	the
evidence	we	can	have	about	the	success	of	our	own	evolutionary	development.
28

28
Suppose	that	on	99.9999%	of	all	planets	where	life	emerged	it	went	extinct
before	developing	to	the	point	where	intelligent	observers	could	begin	to	ponder
their	origin.	What	should	we	expect	to	observe	if	that	were	the	case?	Arguably,
we	should	expect	to	observe	something	like	what	we	do	in	fact	observe.	The
hypothesis	that	the	odds	of	intelligent	life	evolving	on	a	given	planet	are	low
does	not	predict	that	we	should	find	ourselves	on	a	planet	where	life	went	extinct
at	an	early	stage;	rather,	it	may	predict	that	we	should	find	ourselves	on	a	planet

where	intelligent	life	evolved,	even	if	such	planets	constitute	a	very	small
fraction	of	all	planets	where	primitive	life	evolved.	Life’s	long	track	record	on
Earth	may	therefore	offer	scant	support	to	the	claim	that	there	was	a	high	chance
—let	alone	anything	approaching	inevitability—involved	in	the	rise	of	higher
organisms	on	our	planet.
29
Thirdly,	even	if	present	conditions	had	been	idyllic,	and	even	if	they	could
have	been	shown	to	have	arisen	ineluctably	from	some	generic	primordial	state,

29
there	would	still	be	no	guarantee	that	the	melioristic	trend	is	set	to	continue	into
the	indefinite	future.	This	holds	even	if	we	disregard	the	possibility	of	a

cataclysmic	
extinction	event	and	indeed	even	if	we	assume	that	evolutionary
developments	will	continue	to	produce	systems	of	increasing	complexity.
We	suggested	earlier	that	machine	intelligence	workers	selected	for	maximum
productivity	would	be	working	extremely	hard	and	that	it	is	unknown	how	happy
such	workers	would	be.	We	also	raised	the	possibility	that	the	fittest	life	forms
within	a	competitive	future	digital	life	soup	might	not	even	be	conscious.	Short
of	a	complete	loss	of	pleasure,	or	of	consciousness,	there	could	be	a	wasting
away	of	other	qualities	that	many	would	regard	as	indispensible	for	a	good	life.
Humans	value	music,	humor,	romance,	art,	play,	dance,	conversation,
philosophy,	literature,	adventure,	discovery,	food	and	drink,	friendship,
parenting,	sport,	nature,	tradition,	and	spirituality,	among	many	other	things.
There	is	no	guarantee	that	any	of	these	would	remain	adaptive.	Perhaps	what
will	maximize	fitness	will	be	nothing	but	nonstop	high-intensity	drudgery,	work

of	a	drab	and	repetitive	nature,	destitute	of	ludic	frisson,	aimed	only	at
improving	the	eighth	decimal	place	of	some	economic	output	measure.	The
phenotypes	selected	would	then	have	lives	lacking	in	the	aforesaid	qualities,	and
depending	on	one’s	axiology	the	result	might	strike	one	as	either	abhorrent,
worthless,	or	merely	impoverished,	but	at	any	rate	a	far	cry	from	a	utopia	one
would	feel	worthy	of	one’s	commendation.
It	might	be	wondered	how	such	a	bleak	picture	could	be	consistent	with	the
fact	that	we	do	now	indulge	in	music,	humor,	romance,	art,

fact	that	we	do	now	indulge	in	music,	humor,	romance,	art,	
etc.
	If	these
behaviors	are	really	so	“wasteful,”	then	how	come	they	have	been	tolerated	and
indeed	promoted	by	the	evolutionary	processes	that	shaped	our	species?	That
modern	man	is	in	an	evolutionary	disequilibrium	does	not	account	for	this;	for
our	Pleistocene	forebears,	too,	engaged	in	most	of	these	dissipations.	Many	of
the	behaviors	in	question	are	not	even	unique	to	
.	Flamboyant
display	is	found	in	a	wide	variety	of	contexts,	from	sexual	selection	in	the
animal	kingdom	to	prestige	contests	among	nation	states.

30
Although	a	full	evolutionary	explanation	for	each	of	these	behaviors	is	beyond
the	scope	of	the	present	inquiry,	we	can	note	that	some	of	them	serve	functions
that	may	not	be	as	relevant	in	a	machine	intelligence	context.	Play,	for	example,
which	occurs	only	in	some	species	and	predominantly	among	juveniles,	is
mainly	a	way	for	the	young	animal	to	learn	skills	that	it	will	need	later	in	life.
When	emulations	can	be	created	as	adults,	already	in	possession	of	a	mature
repertoire	of	skills,	or	when	knowledge	and	techniques	acquired	by	one	AI	can

be	directly	ported	into	another	AI,	the	need	for	playful	behavior	might	become

less	widespread.
Many	of	the	other	examples	of	humanistic	behaviors	may	have	evolved	as
hard-to-fake	signals	of	qualities	that	are	difficult	to	observe	directly,	such	as
bodily	or	mental	resilience,	social	status,	quality	of	allies,	ability	and	willingness
to	prevail	in	a	fight,	or	possession	of	resources.	The	peacock’s	tail	is	the	classic
instance:	only	fit	peacocks	can	afford	to	sprout	truly	extravagant	plumage,	and
peahens	have	evolved	to	find	it	attractive.	No	less	than	morphological	traits,
behavioral	traits	too	can	signal	genetic	fitness	or	other	socially	relevant
attributes.
31
Given	that	flamboyant	display	is	so	common	among	both	humans	and	other
species,	one	might	consider	whether	it	would	not	also	be	part	of	the	repertoire	of
technologically	more	advanced	life	forms.	Even	if	there	were	to	be	no	narrowly
instrumental	use	for	playfulness	or	musicality	or	even	for	consciousness	in	the
future	ecology	of	intelligent	information	processing,	might	not	these	traits

31
nonetheless	confer	some	evolutionary	advantage	to	their	possessors	by	virtue	of
being	reliable	signals	of	other	adaptive	qualities?
While	the	possibility	of	a	pre-established	harmony	between	what	is	valuable
to	us	and	what	would	be	adaptive	in	a	future	digital	ecology	is	hard	to	rule	out,
there	are	reasons	for	skepticism.	Consider,	first,	that	many	of	the	costly	displays
we	find	in	nature	are	linked	to	sexual	selection.
32
	Reproduction	among
technologically	mature	life	forms,	in	contrast,	may	be	predominantly	or
exclusively	asexual.

32
Second,	technologically	advanced	agents	might	have	available	new	means	of
reliably	communicating	information	about	themselves,	means	that	do	not	rely	on
costly	display.	Even	today,	when	professional	lenders	assess	creditworthiness
they	tend	to	rely	more	on	documentary	evidence,	such	as	ownership	certificates
and	bank	statements,	than	on	costly	displays,	such	as	designer	suits	and	Rolex
watches.	In	the	future,	it	might	be	possible	to	employ	auditing	firms	that	verify
through	detailed	examination	of	behavioral	track	records,	testing	in	simulated

environments,	or	direct	inspection	of	source	code,	that	a	client	agent	possesses	a
claimed	attribute.	Signaling	one’s	qualities	by	agreeing	to	such	auditing	might	be
more	efficient	than	signaling	via	flamboyant	display.	Such	a	professionally
mediated	signal	would	still	be	costly	to	
fake
—this	being	the	essential	feature	that
makes	the	signal	reliable—but	it	could	be	much	cheaper	to	transmit	when
truthful
	than	it	would	be	to	communicate	an	equivalent	signal	flamboyantly.

Third,	not	all	possible	costly	displays	are	intrinsically	valuable	or	socially
desirable.	Many	are	simply	wasteful.	The	Kwakiutl	potlatch	ceremonies,	a	form
of	status	competition	between	rival	chiefs,	involved	the	public	destruction	of
vast	amounts	of	accumulated	wealth.
33
	Record-breaking	skyscrapers,
megayachts,	and	moon	rockets	may	be	viewed	as	contemporary	analogs.	While
activities	like	music	and	humor	could	plausibly	be	claimed	to	enhance	the
intrinsic	quality	of	human	life,	it	is	doubtful	that	a	similar	claim	could	be
sustained	with	regard	to	the	costly	pursuit	of	fashion	accessories	and	other
consumerist	status	symbols.	Worse,	costly	display	can	be	outright	harmful,	as	in
macho	posturing	leading	to	gang	violence	or	military	bravado.	Even	if	future
intelligent	life	forms	would	use	costly	signaling,	therefore,	it	is	an	open	question
whether	the	signal	would	be	of	a	valuable	sort—whether	it	would	be	like	the
rapturous	melody	of	a	nightingale	or	instead	like	the	toad’s	monosyllabic	croak

(or	the	incessant	barking	of	a	rabid	dog).
	
Even	if	the	immediate	outcome	of	the	transition	to	machine	intelligence	were
multipolar,	the	possibility	would	remain	of	a	singleton	developing	later.	Such	a
development	would	continue	an	apparent	long-term	trend	toward	larger	scales	of
political	integration,	taking	it	to	its	natural	conclusion.
34
	How	might	this	occur?
	
On	way	in	which	an	initially	multipolar	outcome	could	converge	into	a	singleton

34
	
post-transition	is	if	there	is,	after	the	initial	transition,	a	second	technological
transition	big	enough	and	steep	enough	to	give	a	decisive	strategic	advantage	to
one	of	the	remaining	powers:	a	power	which	might	then	seize	the	opportunity	to
establish	a	singleton.	Such	a	hypothetical	second	transition	might	be	occasioned
by	a	breakthrough	to	a	higher	level	of	superintelligence.	For	instance,	if	the	first
wave	of	machine	superintelligence	is	emulation-based,	then	a	second	surge

might	result	when	the	emulations	now	doing	the	research	succeed	in	developing
effective	self-improving	artificial	intelligence.
35
	(Alternatively,	a	second
transition	might	be	triggered	by	a	breakthrough	in	nanotechnology	or	some	other

military	or	general-purpose	technology	as	yet	unenvisaged.)
The	pace	of	development	after	the	initial	transition	would	be	extremely	rapid.
Even	a	short	gap	between	the	leading	power	and	its	closest	competitor	could
therefore	plausibly	result	in	a	decisive	strategic	advantage	for	the	leading	power
during	a	second	transition.	Suppose,	for	example,	that	two	projects	enter	the	first
transition	only	a	few	days	apart,	and	that	the	takeoff	is	slow	enough	that	this	gap
does	not	give	the	leading	project	a	decisive	strategic	advantage	at	any	point
during	the	takeoff.	The	two	projects	both	emerge	as	superintelligent	powers,
though	one	of	them	remains	a	few	days	ahead	of	the	other.	But	developments	are
now	occurring	on	the	research	timescales	characteristic	of	machine
superintelligence—perhaps	thousands	or	millions	of	times	faster	than	research
conducted	on	a	biological	human	timescale.	Development	of	the	second-
transition	technology	might	therefore	be	completed	in	days,	hours,	or	minutes.

Even	though	the	frontrunner’s	lead	is	a	mere	few	days,	a	breakthrough	could
thus	catapult	it	into	a	decisive	strategic	advantage.	Note,	however,	that	if
technological	diffusion	(via	espionage	or	other	channels)	speeds	up	as	much	as
technological	development,	then	this	effect	would	be	negated.	What	would
remain	relevant	would	be	the	steepness	of	the	second	transition,	that	is,	the	speed
at	which	it	would	unfold	relative	to	the	general	speed	of	events	in	the	period
after	the	first	transition.	(In	this	sense,	the	faster	things	are	happening	after	the

first	transition,	the	less	steep	the	second	transition	would	tend	to	be.)
One	might	also	speculate	that	a	decisive	strategic	advantage	would	be	more
likely	to	be	actually	used	to	establish	a	singleton	if	it	arises	during	a	second	(or
subsequent)	transition.	After	the	first	transition,	decision	makers	would	either	be
superintelligent	or	have	access	to	advice	from	a	superintelligence,	which	would
clarify	the	implications	of	available	strategic	options.	Furthermore,	the	situation

after	the	first	transition	might	be	one	in	which	a	preemptive	move	against
potential	competitors	would	be	less	dangerous	for	the	aggressor.	If	the	decision-
making	minds	after	the	first	transition	are	digital,	they	could	be	copied	and
thereby	rendered	less	vulnerable	to	a	counterattack.	Even	if	a	defender	had	the
ability	to	kill	nine-tenths	of	the	aggressor’s	population	in	a	retaliatory	strike,	this
would	scarcely	offer	much	deterrence	if	the	deceased	could	be	immediately
resurrected

resurrected	
from	redundant	backups.	Devastation	of	infrastructure	(which	can	be
rebuilt)	might	also	be	tolerable	to	digital	minds	with	effectively	unlimited
lifespans,	who	might	be	planning	to	maximize	their	resources	and	influence	on	a
cosmological	timescale.

The	size	of	coordinated	human	aggregates,	such	as	firms	or	nations,	is
influenced	by	various	parameters—technological,	military,	financial,	and
cultural—that	can	vary	from	one	historical	epoch	to	another.	A	machine
intelligence	revolution	would	entail	profound	changes	in	many	these	parameters.
Perhaps	these	changes	would	facilitate	the	rise	of	a	singleton.	Although	we
cannot,	without	looking	in	detail	at	what	these	prospective	changes	are,	exclude
the	opposite	possibility—that	the	changes	would	facilitate	fragmentation	rather
than	unification—we	can	nevertheless	note	that	the	increased	variance	or
uncertainty	that	we	confront	here	may	itself	be	a	ground	for	giving	greater
credence	to	the	potential	emergence	of	a	singleton	than	we	would	otherwise	do.
A	machine	intelligence	revolution	might,	so	to	speak,	stir	things	up—might
reshuffle	the	deck	to	make	possible	geopolitical	realignments	that	seemed
perhaps	otherwise	not	to	have	been	in	the	cards.

A	comprehensive	analysis	of	all	the	factors	that	may	influence	the	scale	of
political	integration	would	take	us	far	beyond	the	scope	of	this	book:	a	review	of
the	relevant	political	science	and	economics	literature	could	itself	easily	fill	an
entire	volume.	We	must	confine	ourselves	to	making	brief	allusion	to	a	couple	of
factors,	aspects	of	the	digitization	of	agents	that	may	make	it	easier	to	centralize
control.
Carl	Shulman	has	argued	that	in	a	population	of	emulations,	selection

control.
pressures	would	favor	the	emergence	of	“superorganisms,”	groups	of	emulations
ready	to	sacrifice	themselves	for	the	good	of	their	clan.
36
	Superorganisms	would
be	spared	the	agency	problems	that	beset	organizations	whose	members	pursue
their	own	self-interest.	Like	the	cells	in	our	bodies,	or	the	individual	animals	in	a
colony	of	eusocial	insects,	emulations	that	were	wholly	altruistic	toward	their
copy-siblings	would	cooperate	with	one	another	even	in	the	absence	of	elaborate
incentive	schemes.

36
Superorganisms	would	have	a	particularly	strong	advantage	if	nonconsensual
deletion	(or	indefinite	suspension)	of	individual	emulations	is	disallowed.	Firms
or	countries	that	employ	emulations	insisting	on	self-preservation	would	be
saddled	with	an	unending	commitment	to	pay	upkeep	for	obsolete	or	redundant
workers.	In	contrast,	organizations	whose	emulations	willingly	deleted

themselves	when	their	services	were	no	longer	required	could	more	easily	adapt
to	fluctuations	in	demand;	and	they	could	experiment	freely,	proliferating
variations	of	their	workers	and	retaining	only	the	most	productive.
If	involuntary	deletion	is	
not
	disallowed,	then	the	comparative	advantage	of
eusocial	emulations	is	reduced,	though	perhaps	not	eliminated.	Employers	of
cooperative	self-sacrificers	might	still	reap	efficiency	gains	from	reduced	agency
problems	throughout	the	organization,	including	being	spared	the	trouble	of
having	to	defeat	whatever	resistance	emulations	could	put	up	against	their	own
deletion.	In	general,	the	productivity	gains	of	having	workers	willing	to	sacrifice
their	individual	lives	for	the	common	weal	are	a	special	case	of	the	benefits	an
organization	can	derive	from	having	members	who	are	fanatically	devoted	to	it.
Such	members	would	not	only	leap	into	the	grave	for	the	organization,	and	work
long	hours	for	little	pay:	they	would	also	shun	office	politics	and	try	consistently

to	act	in	what	they	took	to	be	the	organization’s	best	interest,	reducing	the	need
for	supervision	and	bureaucratic	constraints.
If	the	only	way	to	achieve	such	dedication	were	by	restricting	membership	to
copy-siblings	(so	that	all	emulations	in	a	particular	superorganism	were	stamped
out	from	the	same	template),	then	superorganisms	would	suffer	some
disadvantage	in	being	able	to	draw	only	from	a	range	of	skills	narrower	than	that
of	rival	organizations,	a	disadvantage	which	might	or	might	not	be	large	enough

to	outweigh	the	advantages	of	avoiding	internal	agency	problems.
37
	This
disadvantage	would	be	greatly	alleviated	if	a	superorganism	could	at	least
contain	members	with	different	training.	Even	if	all	its	members	were	derived
from	a	single	ur-template,	its	workforce	could	then	still	contribute	a	diversity	of
skills.	Starting	with	a	polymathically	talented	emulation	ur-template,	lineages
could	be	branched	off	into	different	training	programs,	one	copy	learning
accounting,	another	electrical	engineering,	and	so	forth.	This	would	produce	a

37
	This
membership	with	diverse	skills	though	not	of	diverse	talents.	(Maximum
diversity	might	require	that	more	than	one	ur-template	be	used.)
The	essential	property	of	a	superorganism	is	not	that	it	consists	of	copies	of	a
single	progenitor	but	that	all	the	individual	agents	within	it	are	fully	committed
to	a	common	goal.	The	ability	to	create	a	superorganism	can	thus	be	viewed	as
requiring	a	partial	solution	to	the	control	problem.	Whereas	a	completely	general
solution	to	the	control	problem	would	enable	somebody	to	create	an	agent	with

any	arbitrary	final	goal,	the	partial	solution	needed	for	the	creation	of	a
superorganism	requires	merely	the	ability	to	fashion	multiple	agents	with	the

same	final	goal	(for	some	nontrivial	but	not	necessarily	arbitrary	final	goal).
38
The	main	consideration	put	forward	in	this	subsection	is	thus	not	really
limited	to	monoclonal	emulation	groups,	but	can	be	stated	more	generally	in	a
way	that	makes	clear	that	it	applies	to	a	wide	range	of	multipolar	machine
intelligence	scenarios.	It	is	that	certain	types	of	advances	in	motivation	selection
techniques,	which	may	become	feasible	when	the	actors	are	digital,	may	help
overcome	some	of	the	inefficiencies	that	currently	hamper	large	human
organizations	and	that	counterbalance	economies	of	scale.	With	these	limits
lifted,	organizations—be	they	firms,	nations,	or	other	economic	or	political
entities—could	increase	in	size.	This	is	one	factor	that	could	facilitate	the
emergence	of	a	post-transition	singleton.
One	area	in	which	superorganisms	(or	other	digital	agents	with	partially
selected	motivations)	might	excel	is	coercion.	A	state	might	use	motivation

selection	methods	to	ensure	that	its	police,	military,	intelligence	service,	and
civil	administration	are	uniformly	loyal.	As	Shulman	notes,
Saved	states	[of	some	loyal	emulation	that	has	been	carefully	prepared	and
verified]	could	be	copied	billions	of	times	to	staff	an	ideologically	uniform
military,	bureaucracy,	and	police	force.	After	a	short	period	of	work,	each
copy	would	be	replaced	by	a	fresh	copy	of	the	same	saved	state,	preventing
ideological	drift.	Within	a	given	jurisdiction,	this	capability	could	allow

incredibly	detailed	observation	and	regulation:	there	might	be	one	such
copy	for	every	other	resident.	This	could	be	used	to	prohibit	the
development	of	weapons	of	mass	destruction,	to	enforce	regulations	on
brain	emulation	experimentation	or	reproduction,	to	enforce	a	liberal
democratic	constitution,	or	to	create	an	appalling	and	permanent
totalitarianism
39
	
The	first-order	effect	of	such	a	capability	would	seem	to	be	to	consolidate	power,
and	possibly	to	concentrate	it	in	fewer	hands.

There	may	be	large	potential	gains	to	be	had	from	international	collaboration	in	a
post-transition	multipolar	world.	Wars	and	arms	races	could	be	avoided.
Astrophysical	resources	could	be	colonized	and	harvested	at	a	globally	optimum
pace.	The	development	of	more	advanced	forms	of	machine	intelligence	could
be	coordinated	to	avoid	a	rush	and	to	allow	new	designs	to	be	thoroughly	vetted.
Other	developments	that	might	pose	existential	risks	could	be	postponed.	And
uniform	regulations	could	be	enforced	globally,	including	provisions	for	a
guaranteed	standard	of	living	(which	would	require	some	form	of	population
control)	and	for	preventing	exploitation	and	abuse	of	emulations	and	other
digital	and	biological	minds.	Furthermore,	agents	with	resource-satiable
preferences	(more	on	this	in	
Chapter	13
)	would	prefer	a	sharing	agreement	that
would	guarantee	them	a	certain	slice	of	the	future	to	a	winner-takes-all	struggle
in	which	they	would	risk	getting	nothing.

Chapter	13
The	presence	of	big	potential	gains	from	collaboration,	however,	does	not
imply	that	collaboration	will	actually	be	achieved.	In	the	world	today,	many
great	boons	could	be	obtained	via	better	global	coordination—reductions	of
military	expenditures,	wars,	overfishing,	trade	barriers,	and	atmospheric
pollution,	among	others.	Yet	these	plump	fruits	are	left	to	spoil	on	the	branch.
Why	is	that?	What	stops	a	fully	cooperative	outcome	that	would	maximize	the
common	good?

One	obstacle	is	the	difficulty	of	ensuring	compliance	with	any	treaty	that
might	be	agreed,	including	monitoring	and	enforcement	costs.	Two	nuclear
rivals	might	each	be	better	off	if	they	both	relinquished	their	atom	bombs;	yet
even	if	they	could	reach	an	in-principle	agreement	to	do	so,	disarmament	could
nevertheless	prove	elusive	because	of	their	mutual	fear	that	the	other	party	might
cheat.	Allaying	this	fear	would	require	setting	up	a	verification	mechanism.

There	may	have	to	be	inspectors	to	oversee	the	destruction	of	existing	stockpiles,
and	then	to	monitor	nuclear	reactors	and	other	facilities,	and	to	gather	technical
and	
human	intelligence,	in	order	to	ensure	that	the	weapons	program	is	not
reconstituted.	One	cost	is	paying	for	these	inspectors.	Another	cost	is	the	risk
that	the	inspectors	will	spy	and	make	off	with	commercial	or	military	secrets.
Perhaps	most	significantly,	each	party	might	fear	that	the	other	will	preserve	a

and	
clandestine	nuclear	capability.	Many	a	potentially	beneficial	deal	never	comes
off	because	compliance	would	be	too	difficult	to	verify.
If	new	inspection	technologies	that	reduced	monitoring	costs	became
available,	one	would	expect	this	to	result	in	increased	cooperation.	Whether

monitoring	costs	would	on	net	be	reduced	in	the	post-transition	era,	however,	is
not	entirely	clear.	While	there	would	certainly	be	many	powerful	new	inspection
techniques,	there	would	also	be	new	means	of	concealment.	In	particular,	an
increasing	portion	of	the	activities	one	might	want	to	regulate	would	be	taking
place	in	cyberspace,	out	of	reach	of	physical	surveillance.	For	example,	digital
minds	working	on	designing	a	new	nanotech	weapons	system	or	a	new
generation	of	artificial	intelligence	may	do	so	without	leaving	much	of	a
physical	footprint.	Digital	forensics	may	fail	to	penetrate	all	the	layers	of
concealment	and	encryption	in	which	a	treaty-violator	may	cloak	its	illicit
activities.
Reliable	lie	detection,	if	it	could	be	developed,	would	be	an	extremely	useful
tool	for	monitoring	compliance.
40
	An	inspection	protocol	could	include
provisions	for	interviewing	key	officials,	to	verify	that	they	are	intent	on
implementing	all	the	provisions	of	the	treaty	and	that	they	know	of	no	violations

40
despite	making	strong	efforts	to	find	out.
A	decision	maker	planning	to	cheat	might	defeat	such	a	lie-detection-based
verification	scheme	by	first	issuing	orders	to	subordinates	to	undertake	the	illicit
activity	and	to	conceal	the	activity	even	from	the	decision	maker	herself,	and
then	subjecting	herself	to	some	procedure	that	erases	her	memory	of	having
engaged	in	these	machinations.	Suitably	targeted	memory-erasure	operations
might	well	be	feasible	in	biological	brains	with	more	advanced	neurotechnology.

It	might	be	even	easier	in	machine	intelligences	(depending	on	their
architecture).
States	could	seek	to	overcome	this	problem	by	committing	themselves	to	an
ongoing	monitoring	scheme	that	regularly	tests	key	officials	with	a	lie	detector
to	check	whether	they	harbor	any	intent	to	subvert	or	circumvent	any	treaty	to
which	the	state	has	entered	or	may	enter	in	the	future.	Such	a	commitment	could
be	viewed	as	a	kind	of	meta-treaty,	which	would	facilitate	the	verification	of

other	treaties;	but	states	might	commit	themselves	to	it	unilaterally	to	gain	the
benefit	of	being	regarded	as	a	trustworthy	negotiation	partner.	However,	this
commitment	or	meta-treaty	would	face	the	same	problem	of	subversion	through
a	delegate-and-forget	ploy.	Ideally,	the	meta-treaty	would	be	put	into	effect
before
	any	party	had	an	opportunity	to	make	the	internal	arrangements	necessary
to	subvert	its	implementation.	Once	villainy	has	had	an	unguarded	moment	to
sow	its	mines	of	deception,	trust	can	never	set	foot	there	again.

In	some	cases,	the	mere	ability	to	
detect
	treaty	violations	is	sufficient	to
establish	the	confidence	needed	for	a	deal.	In	other	cases,	however,	there	is	a
need	for	some	mechanism	to	
enforce
	compliance	or	mete	out	punishment	if	a
violation	
should	occur.	The	need	for	an	enforcement	mechanism	may	arise	if	the
threat	of	the	wronged	party	withdrawing	from	the	treaty	is	not	enough	to	deter
violations,	for	instance	if	the	treaty-violator	would	gain	such	an	advantage	that
he	would	not	subsequently	care	how	the	other	party	responds.
If	highly	effective	motivation	selection	methods	are	available,	this
enforcement	problem	could	be	solved	by	empowering	an	independent	agency
with	sufficient	police	or	military	strength	to	enforce	the	treaty	even	against	the
opposition	of	one	or	several	of	its	signatories.	This	solution	requires	that	the
enforcement	agency	can	be	trusted.	But	with	sufficiently	good	motivation
selection	techniques,	the	requisite	confidence	might	be	achieved	by	having	all

the	parties	to	the	treaty	jointly	oversee	the	design	of	the	enforcement	agency.
Handing	over	power	to	an	external	enforcement	agency	raises	many	of	the
same	issues	that	we	confronted	earlier	in	our	discussions	of	a	unipolar	outcome
(one	in	which	a	singleton	arises	prior	to	or	during	the	initial	machine	intelligence
revolution).	In	order	to	be	able	to	enforce	treaties	concerning	the	vital	security
interests	of	rival	states,	the	external	enforcement	agency	would	in	effect	need	to
constitute	a	singleton:	a	global	superintelligent	Leviathan.	One	difference,

however,	is	that	we	are	now	considering	a	post-transition	situation,	in	which	the
agents	that	would	have	to	create	this	Leviathan	would	have	greater	competence
than	we	humans	currently	do.	These	Leviathan-creators	may	themselves	already
be	superintelligent.	This	would	greatly	improve	the	odds	that	they	could	solve
the	control	problem	and	design	an	enforcement	agency	that	would	serve	the
interests	of	all	the	parties	that	have	a	say	in	its	construction.
Aside	from	the	costs	of	monitoring	and	enforcing	compliance,	are	there	any

other	obstacles	to	global	coordination?	Perhaps	the	major	remaining	issue	is
what	we	can	refer	to	as	
bargaining	costs
.
41
	Even	when	there	is	a	possible	bargain
that	would	benefit	everybody	involved,	it	sometimes	does	not	get	off	the	ground
because	the	parties	fail	to	agree	on	how	to	divide	the	spoils.	For	example,	if	two
persons	could	make	a	deal	that	would	net	them	a	dollar	in	profit,	but	each	party
feels	she	deserves	sixty	cents	and	refuses	to	settle	for	less,	the	deal	will	not
happen	and	the	potential	gain	will	be	forfeited.	In	general,	negotiations	can	be

.
41
difficult	or	protracted,	or	remain	altogether	barren,	because	of	strategic
bargaining	choices	made	by	some	of	the	parties.

In	real	life,	human	beings	frequently	succeed	in	reaching	agreements	despite
the	possibility	for	strategic	bargaining	(though	often	not	without	considerable
expenditure	of	time	and	patience).	It	is	conceivable,	however,	that	strategic
bargaining	problems	would	have	a	different	dynamic	in	the	post-transition	era.
An	AI	negotiator	might	more	consistently	adhere	to	some	particular	formal
conception	of	rationality,	possibly	with	novel	or	unanticipated	consequences
when	matched	with	other	AI	negotiators.	An	AI	might	also	have	available	to	it
moves	in	the	bargaining	game	that	are	either	unavailable	to	humans	or	very
much	more	difficult	for	humans	to	execute,	including	the	ability	to	precommit	to
a	policy	or	a	course	of	action.	While	humans	(and	human-run	institutions)	are
occasionally	able	to	precommit—with	imperfect	degrees	of	credibility	and
specificity—some	
types	of	machine	intelligence	might	be	able	to	make	arbitrary
unbreakable	precommitments	and	to	allow	negotiating	partners	to	confirm	that

such	a	precommitment	has	been	made.
42
The	availability	of	powerful	precommitment	techniques	could	profoundly
alter	the	nature	of	negotiations,	potentially	giving	an	immense	edge	to	an	agent
that	has	a	first-mover	advantage.	If	a	particular	agent’s	participation	is	necessary
for	the	realization	of	some	prospective	gains	from	cooperation,	and	if	that	agent
is	able	to	make	the	first	move,	it	would	be	in	a	position	to	dictate	the	division	of
the	spoils	by	precommitting	not	to	accept	any	deal	that	gives	it	less	than,	say,

42
99%	of	the	surplus	value.	Other	agents	would	then	be	faced	with	the	choice	of
either	getting	nothing	(by	rejecting	the	unfair	proposal)	or	getting	1%	of	the
value	(by	caving	in).	If	the	first-moving	agent’s	precommitment	is	publicly
verifiable,	its	negotiating	partners	could	be	sure	that	these	are	their	only	two
options.
To	avoid	being	exploited	in	this	manner,	agents	might	precommit	to	refuse
blackmail	and	to	decline	all	unfair	offers.	Once	such	a	precommitment	has	been

options.
made	(and	successfully	publicized),	other	agents	would	not	find	it	in	their
interest	to	make	threats	or	to	precommit	themselves	to	only	accepting	deals	tilted
in	their	own	favor,	because	they	would	know	that	threats	would	fail	and	that
unfair	proposals	would	be	rejected.	But	this	just	demonstrates	again	that	the
advantage	is	with	the	first-mover.	The	agent	who	moves	first	can	choose	whether
to	parlay	its	position	of	strength	only	to	deter	others	from	taking	unfair
advantage,	or	to	make	a	grab	for	the	lion’s	share	of	future	spoils.

Best	situated	of	all,	it	might	seem,	would	be	the	agent	who	starts	out	with	a
temperament	or	a	value	system	that	makes	him	impervious	to	extortion	or	indeed

to	any	offer	of	a	deal	in	which	his	participation	is	indispensable	but	he	is	not
getting	almost	all	of	the	gains.	Some	humans	seem	already	to	possess	personality
traits	corresponding	to	various	aspects	of	an	uncompromising	spirit.
43
	A	high-
strung	disposition,	however,	could	backfire	should	it	turn	out	that	there	are	other
agents	around	who	feel	entitled	to	more	than	their	fair	share	and	are	committed
to	not	backing	down.	The	unstoppable	force	would	then	encounter	the
unmovable	object,	resulting	in	a	failure	to	reach	agreement	(or	worse:	total	war).
The	meek	and	the	akratic	would	at	least	get	something,	albeit	less	than	their	fair
share.
What	kind	of	game-theoretic	equilibrium	would	be	reached	in	such	a	post-
transition	bargaining	game	is	not	immediately	obvious.	Agents	might	choose
more	complicated	strategies	than	the	ones	considered	here.	One	
hopes
	that	an
equilibrium	would	be	reached	centered	on	some	fairness	norm	that	would	serve

share.
hopes
	that	an
as	a	Schelling	point—a	salient	feature	in	a	big	outcome	space	which,	because	of
shared	expectations,	becomes	a	likely	coordination	point	in	an	otherwise
underdetermined	coordination	game.	Such	an	equilibrium	might	be	bolstered	by
some	of	our	evolved	dispositions	and	cultural	programming:	a	common
preference	for	fairness	could,	assuming	we	succeed	in	transferring	our	values
into	the	post-transition	era,	bias	expectations	and	strategies	in	ways	that	lead	to
an	attractive	equilibrium.
44

44
In	any	case,	the	upshot	is	that	with	the	possibility	of	strong	and	flexible	forms
of	precommitment,	outcomes	of	negotiations	might	take	on	an	unfamiliar	guise.
Even	if	the	post-transition	era	started	out	multipolar,	it	might	be	that	a	singleton
would	arise	almost	immediately	as	a	consequence	of	a	negotiated	treaty	that
resolves	all	important	global	coordination	problems.	Some	transaction	costs,
perhaps	including	monitoring	and	enforcement	costs,	might	plummet	with	the

44
new	technological	capabilities	available	to	advanced	machine	intelligences.
Other	costs,	in	particular	costs	related	to	strategic	bargaining,	might	remain
significant.	But	however	strategic	bargaining	affects	the	nature	of	the	agreement
that	is	reached,	there	is	no	clear	reason	why	it	would	long	delay	the	reaching	of
some	agreement	if	an	agreement	were	ever	to	be	reached.	If	no	agreement	is
reached,	then	some	form	of	fighting	might	take	place;	and	either	one	faction

might	win,	and	form	a	singleton	around	the	winning	coalition,	or	the	result	might
be	an	interminable	conflict,	in	which	case	a	singleton	may	never	form	and	the
overall	outcome	may	fall	terribly	short	of	what	could	and	should	have	been
achieved	if	humanity	and	its	descendants	had	acted	in	a	more	coordinated	and
cooperative	fashion.

We	have	seen	that	multipolarity,	even	if	it	could	be	achieved	in	a	stable	form,
would	not	guarantee	an	attractive	outcome.	The	original	principal–agent	problem
remains	unsolved,	and	burying	it	under	a	new	set	of	problems	related	to	post-
transition	global	coordination	failures	may	only	make	the	situation	worse.	Let	us
therefore	return	to	the	question	of	how	we	could	safely	keep	a	single
superintelligent	AI.

CHAPTER	12
Acquiring	values
	
Capability	control	is,	at	best,	a	temporary	and	auxiliary	measure.	Unless	the
plan	is	to	keep	superintelligence	bottled	up	forever,	it	will	be	necessary	to
master	motivation	selection.	But	just	how	could	we	get	some	value	into	an
artificial	agent,	so	as	to	make	it	pursue	that	value	as	its	final	goal?	While	the
agent	is	unintelligent,	it	might	lack	the	capability	to	understand	or	even
represent	any	humanly	meaningful	value.	Yet	if	we	delay	the	procedure
until	the	agent	is	superintelligent,	it	may	be	able	to	resist	our	attempt	to
meddle	with	its	motivation	system—and,	as	we	showed	in	
Chapter	7
,	it
would	have	convergent	instrumental	reasons	to	do	so.	This	value-loading
problem	is	tough,	but	must	be	confronted
.
	
It	is	impossible	to	enumerate	all	possible	situations	a	superintelligence	might
find	itself	in	and	to	specify	for	each	what	action	it	should	take.	Similarly,	it	is

Chapter	7
,	it
.
	
impossible	to	create	a	list	of	all	possible	worlds	and	assign	each	of	them	a	value.
In	any	realm	significantly	more	complicated	than	a	game	of	tic-tac-toe,	there	are
far	too	many	possible	states	(and	state-histories)	for	exhaustive	enumeration	to
be	feasible.	A	motivation	system,	therefore,	cannot	be	specified	as	a
comprehensive	lookup	table.	It	must	instead	be	expressed	more	abstractly,	as	a
formula	or	rule	that	allows	the	agent	to	decide	what	to	do	in	any	given	situation.
One	formal	way	of	specifying	such	a	decision	rule	is	via	a	utility	function.	A

utility	function	(as	we	recall	from	
Chapter	1
)	assigns	value	to	each	outcome	that
might	obtain,	or	more	generally	to	each	“possible	world.”	Given	a	utility
function,	one	can	define	an	agent	that	maximizes	expected	utility.	Such	an	agent
selects	at	each	time	the	action	that	has	the	highest	expected	utility.	(The	expected
utility	is	calculated	by	weighting	the	utility	of	each	possible	world	with	the
subjective	probability	of	that	world	being	the	actual	world	conditional	on	a

particular	action	being	taken.)	In	reality,	the	possible	outcomes	are	too	numerous
for	the	expected	
utility	of	an	action	to	be	calculated	exactly.	Nevertheless,	the
decision	rule	and	the	utility	function	together	determine	a	normative	ideal—an
optimality	notion—that	an	agent	might	be	designed	to	approximate;	and	the
approximation	might	get	closer	as	the	agent	gets	more	intelligent.
1
	Creating	a
machine	that	can	compute	a	good	approximation	of	the	expected	utility	of	the
actions	available	to	it	is	an	AI-complete	problem.
2
	This	chapter	addresses
another	problem,	a	problem	that	remains	even	if	the	problem	of	making
machines	intelligent	is	solved.
We	can	use	this	framework	of	a	utility-maximizing	agent	to	consider	the
predicament	of	a	future	seed-AI	programmer	who	intends	to	solve	the	control
problem	by	endowing	the	AI	with	a	final	goal	that	corresponds	to	some	plausible
human	notion	of	a	worthwhile	outcome.	The	programmer	has	some	particular

2
human	value	in	mind	that	he	would	like	the	AI	to	promote.	To	be	concrete,	let	us
say	that	it	is	happiness.	(Similar	issues	would	arise	if	we	the	programmer	were
interested	in	justice,	freedom,	glory,	human	rights,	democracy,	ecological
balance,	or	self-development.)	In	terms	of	the	expected	utility	framework,	the
programmer	is	thus	looking	for	a	utility	function	that	assigns	utility	to	possible
worlds	in	proportion	to	the	amount	of	happiness	they	contain.	But	how	could	he

express	such	a	utility	function	in	computer	code?	Computer	languages	do	not
contain	terms	such	as	“happiness”	as	primitives.	If	such	a	term	is	to	be	used,	it
must	first	be	defined.	It	is	not	enough	to	define	it	in	terms	of	other	high-level
human	concepts—“happiness	is	enjoyment	of	the	potentialities	inherent	in	our
human	nature”	or	some	such	philosophical	paraphrase.	The	definition	must
bottom	out	in	terms	that	appear	in	the	AI’s	programming	language,	and
ultimately	in	primitives	such	as	mathematical	operators	and	addresses	pointing

to	the	contents	of	individual	memory	registers.	When	one	considers	the	problem
from	this	perspective,	one	can	begin	to	appreciate	the	difficulty	of	the
programmer’s	task.
Identifying	and	codifying	our	own	final	goals	is	difficult	because	human	goal
representations	are	complex.	Because	the	complexity	is	largely	transparent	to	us,
however,	we	often	fail	to	appreciate	that	it	is	there.	We	can	compare	the	case	to
visual	perception.	Vision,	likewise,	might	seem	like	a	simple	thing,	because	we
do	it	effortlessly.
3

3
	We	only	need	to	open	our	eyes,	so	it	seems,	and	a	rich,
meaningful,	eidetic,	three-dimensional	view	of	the	surrounding	environment
comes	flooding	into	our	minds.	This	intuitive	understanding	of	vision	is	like	a
duke’s	understanding	of	his	patriarchal	household:	as	far	as	he	is	concerned,

things	simply	appear	at	their	appropriate	times	and	places,	while	the	mechanism
that	produces	those	manifestations	are	hidden	from	view.	Yet	accomplishing
even	the	simplest	visual	task—finding	the	pepper	jar	in	the	kitchen—requires	a
tremendous	amount	of	computational	work.	From	a	noisy	time	series	of	two-
dimensional	patterns	of	nerve	firings,	originating	in	the	retina	and	conveyed	to
the	brain	via	the	optic	nerve,	the	visual	cortex	must	work	backwards	to
reconstruct	an	interpreted	three-dimensional	representation	of	external	space.	A
sizeable	portion	of	our	precious	one	square	meter	of	cortical	real	estate	is	zoned
for	processing	visual	information,	and	as	you	are	reading	this	book,	billions	of
neurons	are	working	ceaselessly	to	accomplish	this	task	(like	so	many
seamstresses,	bent	
over	their	sewing	machines	in	a	sweatshop,	sewing	and	re-
sewing	a	giant	quilt	many	times	a	second).	In	like	manner,	our	seemingly	simple
values	and	wishes	in	fact	contain	immense	complexity.
4
	How	could	our

4
programmer	transfer	this	complexity	into	a	utility	function?
One	approach	would	be	to	try	to	directly	code	a	complete	representation	of
whatever	goal	we	have	that	we	want	the	AI	to	pursue;	in	other	words,	to	write
out	an	explicit	utility	function.	This	approach	might	work	if	we	had
extraordinarily	simple	goals,	for	example	if	we	wanted	to	calculate	the	digits	of
pi—that	is,	if	the	
only
	thing	we	wanted	was	for	the	AI	to	calculate	the	digits	of	pi
and	we	were	indifferent	to	any	other	consequence	that	would	result	from	the

only
pursuit	of	this	goal—recall	our	earlier	discussion	of	the	failure	mode	of
infrastructure	profusion.	This	explicit	coding	approach	might	also	have	some
promise	in	the	use	of	domesticity	motivation	selection	methods.	But	if	one	seeks
to	promote	or	protect	any	plausible	
human
	value,	and	one	is	building	a	system
intended	to	become	a	superintelligent	sovereign,	then	explicitly	coding	the
requisite	complete	goal	representation	appears	to	be	hopelessly	out	of	reach.
5
If	we	cannot	transfer	human	values	into	an	AI	by	typing	out	full-blown

human
5
representations	in	computer	code,	what	else	might	we	try?	This	chapter	discusses
several	alternative	paths.	Some	of	these	may	look	plausible	at	first	sight—but
much	less	so	upon	closer	examination.	Future	explorations	should	focus	on	those
paths	that	remain	open.
Solving	the	value-loading	problem	is	a	research	challenge	worthy	of	some	of
the	next	generation’s	best	mathematical	talent.	We	cannot	postpone	confronting
this	problem	until	the	AI	has	developed	enough	reason	to	easily	understand	our

intentions.	As	we	saw	in	the	section	on	convergent	instrumental	reasons,	a
generic	system	will	resist	attempts	to	alter	its	final	values.	If	an	agent	is	not

already	fundamentally	friendly	by	the	time	it	gains	the	ability	to	reflect	on	its
own	agency,	it	will	not	take	kindly	to	a	belated	attempt	at	brainwashing	or	a	plot
to	replace	it	with	a	different	agent	that	better	loves	its	neighbor.
	
Evolution	has	produced	an	organism	with	human	values	at	least	once.	This	fact
might	encourage	the	belief	that	evolutionary	methods	are	the	way	to	solve	the
value-loading	problem.	There	are,	however,	severe	obstacles	to	achieving	safety
along	this	path.	We	have	already	pointed	to	these	obstacles	at	the	end	of	
Chapter
10
	when	we	discussed	how	powerful	search	processes	can	be	dangerous.
Evolution	can	be	viewed	as	a	particular	class	of	search	algorithms	that	involve
the	alternation	of	two	steps,	one	expanding	a	population	of	solution	candidates
by	generating	new	candidates	according	to	some	relatively	simple	stochastic	rule
(such	as	random	mutation	or	sexual	recombination),	the	other	contracting	the

Chapter
10
population	by	pruning	candidates	that	score	poorly	when	tested	by	an	evaluation
function.	As	with	many	other	types	of	powerful	search,	there	is	the	risk	that	
the
process	will	find	a	solution	that	satisfies	the	formally	specified	search	criteria	but
not	our	implicit	expectations.	(This	would	hold	whether	one	seeks	to	evolve	a
digital	mind	that	has	the	same	goals	and	values	as	a	typical	human	being,	or
instead	a	mind	that	is,	for	instance,	perfectly	moral	or	perfectly	obedient.)	The

the
risk	would	be	avoided	if	we	could	specify	a	formal	search	criterion	that
accurately	represented	all	dimensions	of	our	goals,	rather	than	just	one	aspect	of
what	we	think	we	desire.	But	this	is	precisely	the	value-loading	problem,	and	it
would	of	course	beg	the	question	in	this	context	to	assume	that	problem	solved.
There	is	a	further	problem:
The	total	amount	of	suffering	per	year	in	the	natural	world	is	beyond	all
decent	contemplation.	During	the	minute	that	it	takes	me	to	compose	this

sentence,	thousands	of	animals	are	being	eaten	alive,	others	are	running	for
their	lives,	whimpering	with	fear,	others	are	being	slowly	devoured	from
within	by	rasping	parasites,	thousands	of	all	kinds	are	dying	of	starvation,
thirst	and	disease.
6

Even	just	within	our	species,	150,000	persons	are	destroyed	each	day	while
countless	more	suffer	an	appalling	array	of	torments	and	deprivations.
7
	Nature
might	be	a	great	experimentalist,	but	one	who	would	never	pass	muster	with	an
ethics	review	board—contravening	the	Helsinki	Declaration	and	every	norm	of
moral	decency,	left,	right,	and	center.	It	is	important	that	we	not	gratuitously
replicate	such	horrors	
in	silico
.	Mind	crime	seems	especially	difficult	to	avoid
when	evolutionary	methods	are	used	to	produce	human-like	intelligence,	at	least
if	the	process	is	meant	to	look	anything	like	actual	biological	evolution.
8
	
Reinforcement	learning	is	an	area	of	machine	learning	that	studies	techniques
whereby	agents	can	learn	to	maximize	some	notion	of	cumulative	reward.	By
constructing	an	environment	in	which	desired	performance	is	rewarded,	a
reinforcement-learning	agent	can	be	made	to	learn	to	solve	a	wide	class	of

8
	
problems	(even	in	the	absence	of	detailed	instruction	or	feedback	from	the
programmers,	aside	from	the	reward	signal).	Often,	the	learning	algorithm
involves	the	gradual	construction	of	some	kind	of	evaluation	function,	which
assigns	values	to	states,	state–action	pairs,	or	policies.	(For	instance,	a	program
can	learn	to	play	backgammon	by	using	reinforcement	learning	to	incrementally
improve	its	evaluation	of	possible	board	positions.)	The	evaluation	function,
which	is	continuously	updated	in	light	of	experience,	could	be	regarded	as

incorporating	a	form	of	learning	about	value.	However,	what	is	being	learned	is
not	new	
final
	values	but	increasingly	accurate	
estimates	of	the	instrumental
values
	of	reaching	particular	states	(or	of	taking	particular	actions	in	particular
states,	or	of	following	particular	policies).	Insofar	as	a	reinforcement-learning
agent	can	be	described	as	having	a	final	goal,	that	goal	remains	constant:	to
maximize	future	reward.	And	reward	consists	of	specially	designated	percepts
received	from	the	environment.

not	new	
final
values
received	from	the	environment.	
Therefore,	the	wireheading	syndrome	remains	a
likely	outcome	in	any	reinforcement	agent	that	develops	a	world	model
sophisticated	enough	to	suggest	this	alternative	way	of	maximizing	reward.
9
These	remarks	do	not	imply	that	reinforcement-learning	methods	could	never
be	used	in	a	safe	seed	AI,	only	that	they	would	have	to	be	subordinated	to	a
motivation	system	that	is	not	itself	organized	around	the	principle	of	reward
maximization.	That,	however,	would	require	that	a	solution	to	the	value-loading

problem	had	been	found	by	some	other	means	than	reinforcement	learning.
	
Now	one	might	wonder:	if	the	value-loading	problem	is	so	tricky,	how	do	we
ourselves	manage	to	acquire	our	values?
One	possible	(oversimplified)	model	might	look	something	like	this.	We	begin
life	with	some	relatively	simple	starting	preferences	(e.g.	an	aversion	to	noxious
stimuli)	together	with	a	set	of	dispositions	to	acquire	additional	preferences	in
response	to	various	possible	experiences	(e.g.	we	might	be	disposed	to	form	a
preference	for	objects	and	behaviors	that	we	find	to	be	valued	and	rewarded	in
our	culture).	Both	the	simple	starting	preferences	and	the	dispositions	are	innate,
having	been	shaped	by	natural	and	sexual	selection	over	evolutionary	timescales.
Yet	which	preferences	we	end	up	with	as	adults	depends	on	life	events.	Much	of
the	information	content	in	our	final	values	is	thus	acquired	from	our	experiences
rather	than	preloaded	in	our	genomes.

For	example,	many	of	us	love	another	person	and	thus	place	great	final	value
on	his	or	her	well-being.	What	is	required	to	represent	such	a	value?	Many
elements	are	involved,	but	consider	just	two:	a	representation	of	“person”	and	a
representation	of	“well-being.”	These	concepts	are	not	directly	coded	in	our
DNA.	Rather,	the	DNA	contains	instructions	for	building	a	brain,	which,	when
placed	in	a	typical	human	environment,	will	over	the	course	of	several	years
develop	a	world	model	that	includes	concepts	of	persons	and	of	well-being.

Once	formed,	these	concepts	can	be	used	to	represent	certain	meaningful	values.
But	some	mechanism	needs	to	be	innately	present	that	leads	to	values	being
formed	around	
these
	concepts,	rather	than	around	other	acquired	concepts	(like
that	of	a	flowerpot	or	a	corkscrew).
The	details	of	how	this	mechanism	works	are	not	well	understood.	In	humans,
the	mechanism	is	probably	complex	and	multifarious.	It	is	easier	to	understand
the	phenomenon	if	we	consider	it	in	a	more	rudimentary	form,	such	as	filial

these
imprinting	in	nidifugous	birds,	where	the	newly	hatched	chick	acquires	a	desire
for	physical	proximity	to	an	object	that	presents	a	suitable	moving	stimulus
within	the	first	day	after	hatching.	Which	particular	object	the	chick	desires	to	be
near	depends	on	its	experience;	only	the	general	disposition	to	imprint	in	this

way	is	genetically	determined.	Analogously,	Harry	might	place	a	final	value	on
Sally’s	well-being;	but	had	the	twain	never	met,	he	might	have	fallen	in	love
with	
somebody	else	instead,	and	his	final	values	would	have	been	different.	The
ability	of	our	genes	to	code	for	the	construction	of	a	goal-acquiring	mechanism
explains	how	we	come	to	have	final	goals	of	great	informational	complexity,
greater	than	could	be	contained	in	the	genome	itself.
We	may	consequently	consider	whether	we	might	build	the	motivation	system
for	an	artificial	intelligence	on	the	same	principle.	That	is,	instead	of	specifying
complex	values	directly,	could	we	specify	some	mechanism	that	leads	to	the
acquisition	of	those	values	when	the	AI	interacts	with	a	suitable	environment?
Mimicking	the	value-accretion	process	that	takes	place	in	humans	seems
difficult.	The	relevant	genetic	mechanism	in	humans	is	the	product	of	eons	of
work	by	evolution,	work	that	might	be	hard	to	recapitulate.	Moreover,	the

mechanism	is	presumably	closely	tailored	to	the	human	neurocognitive
architecture	and	therefore	not	applicable	in	machine	intelligences	other	than
whole	brain	emulations.	And	if	whole	brain	emulations	of	sufficient	fidelity
were	available,	it	would	seem	easier	to	start	with	an	adult	brain	that	comes	with
full	representations	of	some	human	values	preloaded.
10
Seeking	to	implement	a	process	of	value	accretion	closely	mimicking	that	of
human	biology	therefore	seems	an	unpromising	line	of	attack	on	the	value-

10
loading	problem.	But	perhaps	we	might	design	a	more	unabashedly	artificial
substitute	mechanism	that	would	lead	an	AI	to	import	high-fidelity
representations	of	relevant	complex	values	into	its	goal	system?	For	this	to
succeed,	it	may	not	be	necessary	to	give	the	AI	exactly	the	same	evaluative
dispositions	as	a	biological	human.	That	may	not	even	be	desirable	as	an	aim—
human	nature,	after	all,	is	flawed	and	all	too	often	reveals	a	proclivity	to	evil

which	would	be	intolerable	in	any	system	poised	to	attain	a	decisive	strategic
advantage.	Better,	perhaps,	to	aim	for	a	motivation	system	that	departs	from	the
human	norm	in	systematic	ways,	such	as	by	having	a	more	robust	tendency	to
acquire	final	goals	that	are	altruistic,	compassionate,	or	high-minded	in	ways	we
would	recognize	as	reflecting	exceptionally	good	character	if	they	were	present
in	a	human	person.	To	count	as	improvements,	however,	such	deviations	from
the	human	norm	would	have	to	be	pointed	in	very	particular	directions	rather

than	at	random;	and	they	would	continue	to	presuppose	the	existence	of	a	largely
undisturbed	anthropocentric	frame	of	reference	to	provide	humanly	meaningful
evaluative	generalizations	(so	as	to	avoid	the	kind	of	perverse	instantiation	of

superficially	plausible	goal	descriptions	that	we	examined	in	
Chapter	8
).	It	is	an
open	question	whether	this	is	feasible.
One	further	issue	with	associative	value	accretion	is	that	the	AI	might	disable
the	accretion	mechanism.	As	we	saw	in	
Chapter	7
,	goal-system	integrity	is	a
convergent	instrumental	value.	When	the	AI	reaches	a	certain	stage	of	cognitive
development	it	may	start	to	regard	the	continued	operation	of	the	accretion
mechanism	as	a	corrupting	influence.
11
	This	is	not	necessarily	a	bad	thing,	but
care	would	have	to	be	taken	to	make	the	sealing-up	of	the	goal	system	occur	at
the	right	moment,	
after
	the	appropriate	values	have	been	accreted	but	
before
	they
have	been	overwritten	by	additional	unintended	accretions.
	
Another	approach	to	the	value-loading	problem	is	what	we	may	refer	to	as
motivational	scaffolding.	It	involves	giving	the	seed	AI	an	interim	goal	system,
with	relatively	simple	final	goals	that	we	can	represent	by	means	of	explicit

after
before
	they
	
coding	or	some	other	feasible	method.	Once	the	AI	has	developed	more
sophisticated	representational	faculties,	we	replace	this	interim	scaffold	goal
system	with	one	that	has	different	final	goals.	This	successor	goal	system	then
governs	the	AI	as	it	develops	into	a	full-blown	superintelligence.
Because	the	scaffold	goals	are	not	just	instrumental	but	
final
	goals	for	the	AI,
the	AI	might	be	expected	to	resist	having	them	replaced	(goal-content	integrity
being	a	convergent	instrumental	value).	This	creates	a	hazard.	If	the	AI	succeeds

final
in	thwarting	the	replacement	of	its	scaffold	goals,	the	method	fails.
To	avoid	this	failure	mode,	precautions	are	necessary.	For	example,	capability
control	methods	could	be	applied	to	limit	the	AI’s	powers	until	the	mature
motivation	system	has	been	installed.	In	particular,	one	could	try	to	stunt	its
cognitive	development	at	a	level	that	is	safe	but	that	allows	it	to	represent	the
values	that	we	want	to	include	in	its	ultimate	goals.	To	do	this,	one	might	try	to

differentially	stunt	certain	types	of	intellectual	abilities,	such	as	those	required
for	strategizing	and	Machiavellian	scheming,	while	allowing	(apparently)	more
innocuous	abilities	to	develop	to	a	somewhat	higher	level.
One	could	also	try	to	use	motivation	selection	methods	to	induce	a	more

collaborative	relationship	between	the	seed	AI	and	the	programmer	team.	For
example,	one	might	include	in	the	scaffold	motivation	system	the	goal	of
welcoming	online	guidance	from	the	programmers,	including	allowing	them	to
replace	any	of	the	AI’s	current	goals.
12
	Other	scaffold	goals	might	include	being
transparent	to	the	programmers	about	its	values	and	strategies,	and	developing
an	architecture	that	is	easy	for	the	programmers	to	understand	and	that	facilitates
the	later	implementation	of	a	humanly	meaningful	final	goal,	as	well	as
domesticity	motivations	(such	as	limiting	the	use	of	computational	resources).
One	could	even	imagine	endowing	the	seed	AI	with	the	sole	final	goal	of
replacing	itself	with	a	different	final	goal,	one	which	may	have	been	only
implicitly	or	indirectly	specified	by	the	programmers.	Some	of	the	issues	raised
by	the	use	of	such	a	“self-replacing”	scaffold	goal	also	arise	in	the	context	of	the
value	learning	approach,	which	is	discussed	in	the	next	subsection.	Some	further

issues	will	be	discussed	in	
Chapter	13
.
The	motivational	scaffolding	approach	is	not	without	downsides.	One	is	that	it
carries	the	risk	that	the	AI	could	become	too	powerful	while	it	is	still	running	on
its	interim	goal	system.	It	may	then	thwart	the	human	programmers’	efforts	to
install	the	ultimate	goal	system	(either	by	forceful	resistance	or	by	quiet
subversion).	The	old	final	goals	may	then	remain	in	charge	as	the	seed	AI
develops	into	a	full-blown	superintelligence.	Another	downside	is	that	installing

Chapter	13
.
the	ultimately	intended	goals	in	a	human-level	AI	is	not	necessarily	that	much
easier	than	doing	so	in	a	more	primitive	AI.	A	human-level	AI	is	more	complex
and	might	
have	developed	an	architecture	that	is	opaque	and	difficult	to	alter.	A
seed	AI,	by	contrast,	is	like	a	
tabula	rasa
	on	which	the	programmers	can	inscribe
whatever	structures	they	deem	helpful.	This	downside	could	be	flipped	into	an
upside	if	one	succeeded	in	giving	the	seed	AI	scaffold	goals	that	made	it	want	to

and	might	
develop	an	architecture	helpful	to	the	programmers	in	their	later	efforts	to	install
the	ultimate	final	values.	However,	it	is	unclear	how	easy	it	would	be	to	give	a
seed	AI	scaffold	goals	with	this	property,	and	it	is	also	unclear	how	even	an
ideally	motivated	seed	AI	would	be	capable	of	doing	a	much	better	job	than	the
human	programming	team	at	developing	a	good	architecture.
	
We	come	now	to	an	important	but	subtle	approach	to	the	value-loading	problem.

It	involves	using	the	AI’s	intelligence	to	
learn
	the	values	we	want	it	to	pursue.
To	do	this,	we	must	provide	a	criterion	for	the	AI	that	at	least	implicitly	picks	out
some	suitable	set	of	values.	We	could	then	build	the	AI	to	act	according	to	its
best	estimates	of	these	implicitly	defined	values.	It	would	continually	refine	its
estimates	as	it	learns	more	about	the	world	and	gradually	unpacks	the
implications	of	the	value-determining	criterion.
In	contrast	to	the	scaffolding	approach,	which	gives	the	AI	an	interim	scaffold
goal	and	later	replaces	it	with	a	different	final	goal,	the	value	learning	approach
retains	an	unchanging	final	goal	throughout	the	AI’s	developmental	and
operational	phases.	Learning	does	not	change	the	goal.	It	changes	only	the	AI’s
beliefs	about	the	goal.
The	AI	thus	must	be	endowed	with	a	criterion	that	it	can	use	to	determine
which	percepts	constitute	evidence	in	favor	of	some	hypothesis	about	what	the
ultimate	goal	is,	and	which	percepts	constitute	evidence	against.	Specifying	a

suitable	criterion	could	be	difficult.	Part	of	the	difficulty,	however,	pertains	to	the
problem	of	creating	artificial	general	intelligence	in	the	first	place,	which
requires	a	powerful	learning	mechanism	that	can	discover	the	structure	of	the
environment	from	limited	sensory	inputs.	That	problem	we	can	set	aside	here.
But	even	modulo	a	solution	to	how	to	create	superintelligent	AI,	there	remain	the
difficulties	that	arise	specifically	from	the	value-loading	problem.	With	the	value

learning	approach,	these	take	the	form	of	needing	to	define	a	criterion	that
connects	perceptual	bitstrings	to	hypotheses	about	values.
Before	delving	into	the	details	of	how	value	learning	could	be	implemented,	it
might	be	helpful	to	illustrate	the	general	idea	with	an	example.	Suppose	we	write
down	a	description	of	a	set	of	values	on	a	piece	of	paper.	We	fold	the	paper	and
put	it	in	a	sealed	envelope.	We	then	create	an	agent	with	human-level	general
intelligence,	and	give	it	the	following	final	goal:	“Maximize	the	realization	of

the	values	described	in	the	envelope.”	What	will	this	agent	do?
The	agent	does	not	initially	know	what	is	written	in	the	envelope.	But	it	can
form	hypotheses,	and	it	can	assign	those	hypotheses	probabilities	based	on	their
priors	
and	any	available	empirical	data.	For	instance,	the	agent	might	have
encountered	other	examples	of	human-authored	texts,	or	it	might	have	observed
some	general	patterns	of	human	behavior.	This	would	enable	it	to	make	guesses.
One	does	not	need	a	degree	in	psychology	to	predict	that	the	note	is	more	likely

priors	
to	describe	a	value	such	as	“minimize	injustice	and	unnecessary	suffering”	or

“maximize	returns	to	shareholders”	than	a	value	such	as	“cover	all	lakes	with
plastic	shopping	bags.”
When	the	agent	makes	a	decision,	it	seeks	to	take	actions	that	would	be
effective	at	realizing	the	values	it	believes	are	most	likely	to	be	described	in	the
letter.	Importantly,	the	agent	would	see	a	high	instrumental	value	in	learning
more	about	what	the	letter	says.	The	reason	is	that	for	almost	any	final	value	that
might	be	described	in	the	letter,	that	value	is	more	likely	to	be	realized	if	the
agent	finds	out	what	it	is,	since	the	agent	will	then	pursue	that	value	more
effectively.	The	agent	would	also	discover	the	convergent	instrumental	reasons
described	in	
Chapter	7
—goal	system	integrity,	cognitive	enhancement,	resource
acquisition,	and	so	forth.	Yet,	assuming	that	the	agent	assigns	a	sufficiently	high
probability	to	the	values	described	in	the	letter	involving	human	welfare,	it
would	
not
	pursue	these	instrumental	values	by	immediately	turning	the	planet

Chapter	7
would	
not
into	computronium	and	thereby	exterminating	the	human	species,	because	doing
so	would	risk	permanently	destroying	its	ability	to	realize	its	final	value.
We	can	liken	this	kind	of	agent	to	a	barge	attached	to	several	tugboats	that	pull
in	different	directions.	Each	tugboat	corresponds	to	a	hypothesis	about	the
agent’s	final	value.	The	engine	power	of	each	tugboat	corresponds	to	the
associated	hypothesis’s	probability,	and	thus	changes	as	new	evidence	comes	in,
producing	adjustments	in	the	barge’s	direction	of	motion.	The	resultant	force

should	move	the	barge	along	a	trajectory	that	facilitates	learning	about	the
(implicit)	final	value	while	avoiding	the	shoals	of	irreversible	destruction;	and
later,	when	the	open	sea	of	more	definite	knowledge	of	the	final	value	is	reached,
the	one	tugboat	that	still	exerts	significant	force	will	pull	the	barge	toward	the
realization	of	the	discovered	value	along	the	straightest	or	most	propitious	route.
The	envelope	and	barge	metaphors	illustrate	the	principle	underlying	the	value

learning	approach,	but	they	pass	over	a	number	of	critical	technical	issues.	They
come	into	clearer	focus	once	we	start	to	develop	the	approach	within	a	formal
framework	(see	
Box	10
).
One	outstanding	issue	is	how	to	endow	the	AI	with	a	goal	such	as	“Maximize
the	realization	of	the	values	described	in	the	envelope.”	(In	the	terminology	of
Box	10
,	how	to	define	the	value	criterion.)	To	do	this,	it	is	necessary	to	identify
the	place	where	the	values	are	described.	In	our	example,	this	requires	making	a

Box	10
).
Box	10
successful	reference	to	the	letter	in	the	envelope.	Though	this	might	seem	trivial,
it	is	not	without	pitfalls.	To	mention	just	one:	it	is	critical	that	the	reference	be

not	simply	to	a	particular	external	physical	object	but	to	an	object	at	a	particular
time.	Otherwise	the	AI	may	determine	that	the	best	way	to	attain	its	goal	is	by
overwriting	the	original	value	description	with	one	that	provides	an	easier	target
(such	as	the	value	that	for	every	integer	there	be	a	larger	integer).	This	done,	the
AI	
could	lean	back	and	crack	its	knuckles—though	more	likely	a	malignant
failure	would	ensue,	for	reasons	we	discussed	in	
Chapter	8
.	So	now	we	face	the
question	of	how	to	define	time.	We	could	point	to	a	clock	and	say,	“Time	is
defined	by	the	movements	of	this	device”—but	this	could	fail	if	the	AI
conjectures	that	it	can	manipulate	time	by	moving	the	hands	on	the	clock,	a
conjecture	which	would	indeed	be	correct	if	“time”	were	given	the	aforesaid
definition.	(In	a	realistic	case,	matters	would	be	further	complicated	by	the	fact
that	the	relevant	values	are	not	going	to	be	conveniently	described	in	a	letter;

Chapter	8
more	likely,	they	would	have	to	be	inferred	from	observations	of	pre-existing
structures	that	implicitly	contain	the	relevant	information,	such	as	human
brains.)
Box	10	Formalizing	value	learning
	
Introducing	some	formal	notation	can	help	us	see	some	things	more	clearly.
However,	readers	who	dislike	formalism	can	skip	this	part.
Consider	a	simplified	framework	in	which	an	agent	interacts	with	its
environment	in	a	finite	number	of	discrete	cycles.
13
	In	cycle	
k
,	the	agent
performs	action	
y
k

brains.)
	
13
	In	cycle	
k
,	the	agent
y
k
,	and	then	receives	the	percept	
x
k
.	The	interaction	history	of	an
agent	with	lifespan	
m
	is	a	string	
y
1
x
1
y
2
x
2
	…	
y
m
x
m
	(which	we	can	abbreviate	as
yx
1:
m
	or	
yx
≤
m
).	In	each	cycle,	the	agent	selects	an	action	based	on	the	percept
sequence	it	has	received	to	date.
Consider	first	a	reinforcement	learner.	An	optimal	reinforcement	learner	(AI-
RL)	is	one	that	maximizes	expected	future	rewards.	It	obeys	the	equation
14

The	reward	sequence	
r
k
,	…,	
r
m
	is	implied	by	the	percept	sequence	
x
k:m
,	since	the
reward	that	the	agent	receives	in	a	given	cycle	is	part	of	the	percept	that	the
agent	receives	in	that	cycle.
As	argued	earlier,	this	kind	of	reinforcement	learning	is	unsuitable	in	the
present	context	because	a	sufficiently	intelligent	agent	will	realize	that	it	could
secure	maximum	reward	if	it	were	able	to	directly	manipulate	its	reward	signal
(wireheading).	For	weak	agents,	this	need	not	be	a	problem,	since	we	can
physically	prevent	them	from	tampering	with	their	own	reward	channel.	We	can
also	control	their	environment	so	that	they	receive	rewards	only	when	they	act	in
ways	that	are	agreeable	to	us.	But	a	reinforcement	learner	has	a	strong	incentive
to	eliminate	this	artificial	dependence	of	its	rewards	on	our	whims	and	wishes.
Our	relationship	with	a	reinforcement	learner	is	therefore	fundamentally
antagonistic.	If	the	agent	is	strong,	this	spells	danger.

Variations	of	the	wireheading	syndrome	can	also	affect	systems	that	do	not
seek	an	external	sensory	reward	signal	but	whose	goals	are	defined	as	the
attainment	of	some	internal	state.	For	example,	in	so-called	“actor–critic”
systems,	there	is	an	actor	module	that	selects	actions	in	order	to	minimize	the
disapproval	of	a	separate	critic	module	that	computes	how	far	the	agent’s
behavior	falls	short	of	a	given	performance	measure.	The	problem	with	this
setup	is	that	the	actor	module	may	realize	that	it	can	minimize	disapproval	by

modifying	the	critic	or	eliminating	it	altogether—much	like	a	dictator	who
dissolves	the	parliament	and	nationalizes	the	press.	For	limited	systems,	the
problem	can	be	avoided	simply	by	not	giving	the	actor	module	any	means	of
modifying	the	critic	module.	A	sufficiently	intelligent	and	resourceful	actor
module,	however,	could	always	gain	access	to	the	critic	module	(which,	after	all,
is	merely	a	physical	process	in	some	computer).
15
Before	we	get	to	the	value	learner,	let	us	consider	as	an	intermediary	step	what

15
has	been	called	an	observation-utility	maximizer	(AI-OUM).	It	is	obtained	by
replacing	the	reward	series	(
r
k
	+	…	+	
r
m
)	in	the	AI-RL	with	a	utility	function	that
is	allowed	to	depend	on	the	entire	future	interaction	history	of	the	AI:

This	formulation	provides	a	way	around	the	wireheading	problem	because	a
utility	function	defined	over	an	entire	interaction	history	could	be	designed	to
penalize	interaction	histories	that	show	signs	of	self-deception	(or	of	a	failure	on
the	part	of	the	agent	to	invest	sufficiently	in	obtaining	an	accurate	view	of
reality).
The	AI-OUM	thus	makes	it	possible	
	to	circumvent	the
wireheading	problem.	Availing	ourselves	of	this	possibility,	however,	would
require	that	we	specify	a	suitable	utility	function	over	the	class	of	possible
interaction	histories—a	task	that	looks	forbiddingly	difficult.
It	may	be	more	natural	to	specify	utility	functions	directly	in	terms	of	possible
worlds	(or	properties	of	possible	worlds,	or	theories	about	the	world)	rather	than
in	terms	of	an	agent’s	own	interaction	histories.	If	we	use	this	approach,	we
could	reformulate	and	simplify	the	AI-OUM	optimality	notion:
	
Here,	
E
	is	the	total	evidence	available	to	the	agent	(at	the	time	when	it	is	making
its	decision),	and

Here,	
E
its	decision),	and	
U
	is	a	utility	function	that	assigns	utility	to	some	class	of
possible	worlds.	The	optimal	agent	chooses	the	act	that	maximizes	expected
utility.
An	outstanding	problem	with	these	formulations	is	the	difficulty	of	defining
the	utility	function	
U
.	This,	finally,	returns	us	to	the	value-loading	problem.	To
enable	the	utility	function	to	be	learned,	we	must	expand	our	formalism	to	allow
for	uncertainty	over	utility	functions.	This	can	be	done	as	follows	(AI-VL):
16
	
Here,	
ν

U
utility.
U
16
	
Here,	
ν
(.)	is	a	function	from	utility	functions	to	propositions	about	utility
functions.	
ν
(
U
)	is	the	proposition	that	the	utility	function	
U
	satisfies	the	
value
criterion
	expressed	by	
ν
.
17
To	decide	which	action	to	perform,	one	could	hence	proceed	as	follows:	First,

compute	the	conditional	probability	of	each	possible	world	
w
	(given	available
evidence	and	on	the	supposition	that	action	
y
	is	to	be	performed).	Second,	for
each	possible	utility	function	
U
,	compute	the	conditional	probability	that	
U
satisfies	the	value	criterion	ν	(conditional	on	
w
	being	the	actual	world).	Third,
for	each	possible	utility	function	
U
,	compute	the	utility	of	possible	world	
w
.
Fourth,	combine	these	quantities	to	compute	the	expected	utility	of	action	
y
.
Fifth,	repeat	this	procedure	for	each	possible	action,	and	perform	the	action
found	to	have	the	highest	expected	utility	(using	some	arbitrary	method	to	break
ties).	As	described,	this	procedure—which	involves	giving	explicit	and	separate
consideration	to	each	possible	world—is,	of	course,	wildly	computationally
intractable.	The	AI	would	have	to	use	computational	shortcuts	that	approximate
this	optimality	notion.
The	question,	then,	is	how	to	define	this	value	criterion	
ν
.
18
	Once	the	AI	has

y
.
ν
.
18
an	adequate	representation	of	the	value	criterion,	it	could	in	principle	use	its
general	intelligence	to	gather	information	about	which	possible	worlds	are	most
likely	to	be	the	actual	one.	It	could	then	apply	the	criterion,	for	each	such
plausible	possible	world	
w
,	to	find	out	which	utility	function	satisfies	the
criterion	in	
w
.	One	can	thus	regard	the	AI-VL	formula	as	a	way	of	identifying
and	separating	out	this	key	challenge	in	the	value	learning	approach—the
challenge	of	how	to	represent	
ν

w
w
ν
.	The	formalism	also	brings	to	light	a	number	of
other	issues	(such	as	how	to	define	
,	
,	and	
)	which	would	need	to	be
resolved	before	the	approach	could	be	made	to	work.
19
	
Another	issue	in	coding	the	goal	“Maximize	the	realization	of	the	values
described	in	the	envelope”	is	that	even	if	all	the	correct	values	were	described	in
a	letter,	and	even	if	the	AI’s	motivation	system	were	successfully	keyed	to	this
source,	the	AI	might	not	interpret	the	descriptions	the	way	we	intended.	This

ν
,	
,	and	
19
	
would	create	a	risk	of	perverse	instantiation,	as	discussed	in	
Chapter	8
.
To	clarify,	the	difficulty	here	is	not	so	much	how	to	ensure	that	the	AI	can
understand	human	intentions.	A	superintelligence	should	easily	develop	such
understanding.	Rather,	the	difficulty	is	ensuring	that	the	AI	will	be	motivated	to
pursue	the	described	values	in	the	way	we	intended.	This	is	not	guaranteed	by
the	AI’s	ability	to	understand	our	intentions:	an	AI	could	know	exactly	what	we

Chapter	8
.
meant	and	yet	be	indifferent	to	that	interpretation	of	our	words	(being	motivated
instead	by	some	other	interpretation	of	the	words	or	being	indifferent	to	our

words	altogether).
The	difficulty	is	compounded	by	the	desideratum	that,	for	reasons	of	safety,
the	correct	motivation	should	ideally	be	installed	in	the	seed	AI	
before
	it
becomes	capable	of	fully	representing	human	concepts	or	understanding	human
intentions.	This	requires	that	somehow	a	cognitive	framework	be	created,	with	a
particular	
location	in	that	framework	designated	in	the	AI’s	motivation	system	as
the	repository	of	its	final	value.	But	the	cognitive	framework	itself	must	be
revisable,	so	as	to	allow	the	AI	to	expand	its	representational	capacities	as	it
learns	more	about	the	world	and	grows	more	intelligent.	The	AI	might	undergo
the	equivalent	of	scientific	revolutions,	in	which	its	worldview	is	shaken	up	and
it	perhaps	suffers	ontological	crises	in	which	it	discovers	that	its	previous	ways
of	thinking	about	values	were	based	on	confusions	and	illusions.	Yet	starting	at	a
sub-human	level	of	development	and	continuing	throughout	all	its	subsequent

development	into	a	galactic	superintelligence,	the	AI’s	conduct	is	to	be	guided
by	an	essentially	unchanging	final	value,	a	final	value	that	becomes	better
understood	by	the	AI	in	direct	consequence	of	its	general	intellectual	progress—
and	likely	quite	differently	understood	by	the	mature	AI	than	it	was	by	its
original	programmers,	though	not	different	in	a	random	or	hostile	way	but	in	a
benignly	appropriate	way.	How	to	accomplish	this	remains	an	open	question.
20
(See	
Box	11
.)

20
(See	
Box	11
.)
In	summary,	it	is	not	yet	known	how	to	use	the	value	learning	approach	to
install	plausible	human	values	(though	see	
Box	12
	for	some	examples	of	recent
ideas).	At	present,	the	approach	should	be	viewed	as	a	research	program	rather
than	an	available	technique.	If	it	could	be	made	to	work,	it	might	constitute	the
most	ideal	solution	to	the	value-loading	problem.	Among	other	benefits,	it	would
seem	to	offer	a	natural	way	to	prevent	mind	crime,	since	a	seed	AI	that	makes

20
(See	
Box	11
.)
Box	12
reasonable	guesses	about	which	values	its	programmers	might	have	installed
would	anticipate	that	mind	crime	is	probably	negatively	evaluated	by	those
values,	and	thus	best	avoided,	at	least	until	more	definitive	information	has	been
obtained.
Last,	but	not	least,	there	is	the	question	of	“what	to	write	in	the	envelope”—
or,	less	metaphorically,	the	question	of	which	values	we	should	try	to	get	the	AI
to	learn.	But	this	issue	is	common	to	all	approaches	to	the	AI	value-loading
problem.	We	return	to	it	in	
Chapter	13
.

Box	11	An	AI	that	wants	to	be	friendly
	
Eliezer	Yudkowsky	has	tried	to	describe	some	features	of	a	seed	AI	architecture
intended	to	enable	the	kind	of	behavior	described	in	the	text	above.	In	his
terminology,	the	AI	would	use	“external	reference	semantics.”
21
	To	illustrate	the
basic	idea,	let	us	suppose	that	we	want	the	system	to	be	“friendly.”	The	system
starts	out	with	the	goal	of	trying	to	instantiate	property	
F
	but	does	not	initially
know	much	about	what	
F
	is.	It	might	just	know	that	
F
	is	some	abstract	property
and	that	when	the	programmers	speak	of	“friendliness,”	they	are	probably	trying
to	convey	information	about	
F
.	Since	the	AI’s	final	goal	is	to	instantiate	
F
,	an
important	instrumental	value	is	to	learn	more	about	what	
F
	is.	As	the	AI
discovers	more	about	
F
,	its	behavior	is	increasingly	guided	by	the	actual	content
of	
F
.	Thus,	hopefully,	the	AI	becomes	increasingly	friendly	the	more	it	learns
and	the	smarter	it	gets.

F
F
F
F
,	an
F
F
of	
F
The	programmers	can	help	this	process	along,	and	reduce	the	risk	of	the	AI
making	some	catastrophic	mistake	while	its	understanding	of	
F
	is	still
incomplete,	by	providing	the	AI	with	“programmer	affirmations,”	hypotheses
about	the	nature	and	content	of	
F
	to	which	an	initially	high	probability	is
assigned.	For	instance,	the	hypothesis	“misleading	the	programmers	is
unfriendly”	can	be	given	a	high	prior	probability.	These	programmer
affirmations,	however,	are	not	“true	by	definition”—they	are	not

F
	is	still
F
unchallengeable	axioms	about	the	concept	of	friendliness.	Rather,	they	are	initial
hypotheses	about	friendliness,	hypotheses	to	which	a	rational	AI	will	assign	a
high	probability	at	least	for	as	long	as	it	trusts	the	programmers’	epistemic
capacities	more	than	its	own.
Yudkowsky’s	proposal	also	involves	the	use	of	what	he	called	“causal	validity
semantics.”	The	idea	here	is	that	the	AI	should	do	not	exactly	what	the
programmers	told	it	to	do	but	rather	(something	like)	what	they	were	trying	to

tell	it	to	do.	While	the	programmers	are	trying	to	explain	to	the	seed	AI	what
friendliness	is,	they	might	make	errors	in	their	explanations.	Moreover,	the
programmers	themselves	may	not	fully	understand	the	true	nature	of
friendliness.	One	would	therefore	want	the	AI	to	have	the	ability	to	correct	errors
in	the	programmers’	thinking,	and	to	infer	the	true	or	intended	meaning	from
whatever	imperfect	explanations	the	programmers	manage	to	provide.	For

example,	the	AI	should	be	able	to	represent	the	causal	processes	whereby	the

programmers	learn	and	communicate	about	friendliness.	Thus,	to	pick	a	trivial
example,	the	AI	should	understand	that	there	is	a	possibility	that	a	programmer
might	make	a	typo	while	inputting	information	about	friendliness,	and	the	AI
should	then	seek	to	correct	the	error.	More	generally,	the	AI	should	seek	to
correct	for	whatever	distortive	influences	may	have	corrupted	the	flow	of
information	about	friendliness	as	it	passed	from	its	source	through	the
programmers	to	the	AI	(where	“distortive”	is	an	epistemic	category).	Ideally,	as
the	AI	matures,	it	should	overcome	any	cognitive	biases	and	other	more
fundamental	misconceptions	that	may	have	prevented	its	programmers	from
fully	understanding	what	friendliness	is.
	
Box	12	Two	recent	(half-baked)	ideas
	
What	we	might	call	the	“Hail	Mary”	approach	is	based	on	the	hope	that
elsewhere	in	the	universe	there	exist	(or	will	come	to	exist)	civilizations	that
successfully	manage	the	intelligence	explosion,	and	that	they	end	up	with	values

that	significantly	overlap	with	our	own.	We	could	then	try	to	build	our	AI	so	that
it	is	motivated	to	do	what	these	other	superintelligences	want	it	to	do.
22
	The
advantage	is	that	this	might	be	easier	than	to	build	our	AI	to	be	motivated	to	do
what	we	want	directly.
For	this	scheme	to	work	it	is	
not
	necessary	that	our	AI	can	establish
communication	with	any	alien	superintelligence.	Rather,	our	AI’s	actions	would
be	guided	by	
its	estimates
	of	what	the	alien	superintelligences	would	want	it	to

22
	The
not
do.	Our	AI	would	model	the	likely	outcomes	of	intelligence	explosions
elsewhere,	and	as	it	becomes	superintelligent	itself	its	estimates	should	become
increasingly	accurate.	Perfect	knowledge	is	not	required.	There	may	be	a	range
of	plausible	outcomes	of	intelligence	explosions,	and	our	AI	would	then	do	its
best	to	accommodate	the	preferences	of	the	various	different	kinds	of
superintelligence	that	might	emerge,	weighted	by	probability.
This	version	of	the	Hail	Mary	approach	requires	that	we	construct	a	final

value	for	our	AI	that	refers	to	the	preferences	of	other	superintelligences.
Exactly	how	to	do	this	is	not	yet	clear.	However,	superintelligent	agents	might	be

structurally	distinctive	enough	that	we	could	write	a	piece	of	code	that	would
function	as	a	detector	that	would	look	at	the	world	model	in	our	developing	AI
and	designate	the	representational	elements	that	correspond	to	the	presence	of	a
superintelligence.	The	detector	would	then,	somehow,	extract	the	preferences	of
the	superintelligence	in	question	(as	it	is	represented	within	our	own	AI).
23
	If	we
could	create	such	a	detector,	we	could	then	use	it	to	define	our	AI’s	final	values.
One	challenge	is	that	we	may	need	to	create	the	detector	before	we	know	what
representational	framework	our	AI	will	develop.	The	detector	may	thus	need	to
query	an	unknown	representational	framework	and	extract	the	preferences	of
whatever	superintelligence	may	be	represented	therein.	This	looks	difficult,	but
perhaps	some	clever	solution	can	be	found.
24
If	the	basic	setup	could	be	made	to	work,	various	refinements	immediately
suggest	themselves.	For	example,	rather	than	aiming	to	follow	(some	weighted

24
composition	of)	the	preferences	of	
every
	alien	superintelligence,	our	AI’s	final
value	could	incorporate	a	filter	to	select	a	subset	of	alien	superintelligences	for
obeisance	(with	the	aim	of	selecting	ones	whose	values	are	closer	to	our	own).
For	instance,	we	might	use	criteria	pertaining	to	a	superintelligence’s	causal
origin	to	determine	whether	to	include	it	in	the	obeisance	set.	Certain	properties
of	its	origination	(which	we	might	be	able	to	define	in	structural	terms)	may

every
correlate	with	the	degree	to	which	the	resultant	superintelligence	could	be
expected	to	share	our	values.	Perhaps	we	wish	to	place	more	trust	in
superintelligences	whose	causal	origins	trace	back	to	a	whole	brain	emulation,	or
to	a	seed	AI	that	did	not	make	heavy	use	of	evolutionary	algorithms	or	that
emerged	slowly	in	a	way	suggestive	of	a	controlled	takeoff.	(Taking	causal
origins	into	account	would	also	let	us	avoid	over-weighting	superintelligences
that	create	multiple	copies	of	themselves—indeed	would	let	us	avoid	creating	an

incentive	for	them	to	do	so.)	Many	other	refinements	would	also	be	possible.
The	Hail	Mary	approach	requires	faith	that	there	are	other	superintelligences
out	there	that	sufficiently	share	our	values.
25
	This	makes	the	approach	non-ideal.
However,	the	technical	obstacles	facing	the	Hail	Mary	approach,	though	very
substantial,	might	possibly	be	less	formidable	than	those	confronting	alternative
approaches.	Exploring	non-ideal	but	more	easily	implementable	approaches	can
make	sense—not	with	the	intention	of	using	them,	but	to	have	something	to	fall

25
back	upon	in	case	an	ideal	solution	should	not	be	ready	in	time.
Another	idea	for	how	to	solve	the	value-loading	problem	has	recently	been
proposed	by	Paul	Christiano.
26
	Like	the	Hail	Mary,	it	is	a	value	learning	method

that	tries	to	define	the	value	criterion	by	means	of	a	“trick”	rather	than	through
laborious	construction.	By	contrast	to	the	Hail	Mary,	it	does	not	presuppose	the
existence	of	other	superintelligent	agents	that	we	could	point	to	as	role	models
for	our	own	AI.	Christiano’s	proposal	is	somewhat	resistant	to	brief	explanation
—it	involves	a	series	of	arcane	considerations—but	we	can	try	to	at	least	gesture
at	its	main	elements.
Suppose	we	could	obtain	(a)	a	mathematically	precise	specification	of	a
particular	human	brain	and	(b)	a	mathematically	well-specified	virtual
environment	that	contains	an	idealized	computer	with	an	arbitrarily	large	amount
of	memory	and	CPU	power.	Given	(a)	and	(b),	we	could	define	a	utility	function
U
	as	the	output	the	human	brain	would	produce	after	interacting	with	this
environment.	
U
	would	be	a	mathematically	well-defined	object,	albeit	one	which
(because	of	computational	limitations)	we	may	be	unable	to	describe	
explicitly
.
Nevertheless,	
U

U
U
.
U
	could	serve	as	the	value	criterion	for	a	value	learning	AI,	which
could	use	various	heuristics	for	assigning	probabilities	to	hypotheses	about	what
U
	implies.
Intuitively,	we	want	
U
	to	be	the	utility	function	that	a	suitably	prepared	human
would	output	if	she	had	the	advantage	of	being	able	use	an	arbitrarily	large
amount	of	computing	power—enough	computing	power,	for	example,	to	run
astronomical	numbers	of	copies	of	herself	to	assist	her	with	her	analysis	of

.
U
U
	implies.
U
specifying	a	utility	function,	or	to	help	her	devise	a	better	process	for	going
about	this	analysis.	(We	are	here	foreshadowing	a	theme,	“coherent	extrapolated
volition,”	which	will	be	further	explored	in	
Chapter	13
.)
It	would	seem	relatively	easy	to	specify	the	idealized	environment:	we	can
give	a	mathematical	description	of	an	abstract	computer	with	arbitrarily	large
capacity;	and	in	other	respects	we	could	use	a	virtual	reality	program	that	gives	a

Chapter	13
.)
mathematical	description	of,	say,	a	single	room	with	a	computer	terminal	in	it
(instantiating	the	abstract	computer).	But	how	to	obtain	a	mathematically	precise
description	of	a	particular	human	brain?	The	obvious	way	would	be	through
whole	brain	emulation,	but	what	if	the	technology	for	emulation	is	not	available
in	time?
This	is	where	Christiano’s	proposal	offers	a	key	innovation.	Christiano
observes	that	in	order	to	obtain	a	mathematically	well-specified	value	criterion,
we	do	not	need	a	practically	useful	computational	model	of	a	mind,	a	model	we

in	time?
could	run.	We	just	need	a	(possibly	implicit	and	hopelessly	complicated)

mathematical	
definition
—and	this	may	be	much	easier	to	attain.	Using
functional	neuroimaging	and	other	measurements,	we	can	perhaps	collect
gigabytes	of	data	about	the	input–output	behavior	of	a	selected	human.	If	we
collect	a	sufficient	amount	of	data,	then	it	might	be	that	the	simplest
mathematical	model	that	accounts	for	all	this	data	is	in	fact	an	emulation	of	the
particular	human	in	question.	Although	it	would	be	computationally	intractable
for	us	to	
find
	this	simplest	model	from	the	data,	it	could	be	perfectly	possible	for
us	to	
define
	the	model,	by	referring	to	the	data	and	a	using	a	mathematically
well-defined	simplicity	measure	(such	as	some	variant	of	the	Kolmogorov
complexity,	which	we	encountered	in	
Box	1
,	
Chapter	1
).
27
	
The	value-loading	problem	looks	somewhat	different	for	whole	brain	emulation
than	it	does	for	artificial	intelligence.	Methods	that	presuppose	a	fine-grained
understanding	and	control	of	algorithms	and	architecture	are	not	applicable	to

us	to	
define
Box	1
,	
Chapter	1
).
27
	
emulations.	On	the	other	hand,	the	augmentation	motivation	selection	method—
inapplicable	to	de	novo	artificial	intelligence—is	available	to	be	used	with
emulations	(or	enhanced	biological	brains).
28
The	augmentation	method	could	be	combined	with	techniques	to	tweak	the
inherited	goals	of	the	system.	For	example,	one	could	try	to	manipulate	the
motivational	state	of	an	emulation	by	administering	the	digital	equivalent	of
psychoactive	substances	(or,	in	the	case	of	biological	systems,	the	actual

28
chemicals).	Even	now	it	is	possible	to	pharmacologically	manipulate	values	and
motivations	to	a	limited	extent.
29
	The	pharmacopeia	of	the	future	may	contain
drugs	with	more	specific	and	predictable	effects.	The	digital	medium	of
emulations	should	greatly	facilitate	such	developments,	by	making	controlled
experimentation	easier	and	by	rendering	all	cerebral	parts	directly	addressable.
Just	as	when	biological	test	subjects	are	used,	research	on	emulations	would

29
get	entangled	in	ethical	complications,	not	all	of	which	could	be	brushed	aside
with	a	consent	form.	Such	entanglements	could	slow	progress	along	the
emulation	path	(because	of	regulation	or	moral	restraint),	perhaps	especially

hindering	studies	on	how	to	manipulate	the	motivational	structure	of	emulations.
The	result	could	be	that	emulations	are	augmented	to	potentially	dangerous
superintelligent	levels	of	cognitive	ability	before	adequate	work	has	been	done
to	test	or	adjust	their	final	goals.	Another	possible	effect	of	the	moral
entanglements	might	be	to	give	the	lead	to	less	scrupulous	teams	and	nations.
Conversely,	were	we	to	relax	our	moral	standards	for	experimenting	with	digital
human	minds,	we	could	become	responsible	for	a	substantial	amount	of	harm
and	wrongdoing,	which	is	obviously	undesirable.	Other	things	equal,	these
considerations	favor	taking	some	alternative	path	that	does	not	require	the
extensive	use	of	digital	human	research	subjects	in	a	strategically	high-stakes
situation.
The	issue,	however,	is	not	clear-cut.	One	could	argue	that	whole	brain
emulation	research	is	
less
	likely	to	involve	moral	violations	than	artificial
intelligence	research,	on	the	grounds	that	we	are	more	likely	to	recognize	when

less
an	emulation	mind	qualifies	for	moral	status	than	we	are	to	recognize	when	a
completely	alien	or	synthetic	mind	does	so.	If	certain	kinds	of	AIs,	or	their
subprocesses,	have	a	significant	moral	status	that	we	fail	to	recognize,	the
consequent	moral	violations	could	be	extensive.	Consider,	for	example,	the
happy	abandon	with	which	contemporary	programmers	create	reinforcement-
learning	agents	and	subject	them	to	aversive	stimuli.	Countless	such	agents	are
created	daily,	not	only	in	computer	science	laboratories	but	in	many	applications,

including	some	computer	games	containing	sophisticated	non-player	characters.
Presumably,	these	agents	are	still	too	primitive	to	have	any	moral	status.	But
how	confident	can	we	really	be	that	this	is	so?	More	importantly,	how	confident
can	we	be	that	we	will	know	to	stop	in	time,	before	our	programs	become
capable	of	experiencing	morally	relevant	suffering?
(We	will	return	in	
Chapter	14
	to	some	of	the	broader	strategic	questions	that
arise	when	we	compare	the	desirability	of	emulation	and	artificial	intelligence
paths.)

Chapter	14
paths.)
	
Some	intelligent	systems	consist	of	intelligent	parts	that	are	themselves	capable
of	agency.	Firms	and	states	exemplify	this	in	the	human	world:	whilst	largely
composed	of	humans	they	can,	for	some	purposes,	be	viewed	as	autonomous

agents	in	their	own	right.	The	motivations	of	such	composite	systems	depend	not
only	on	the	motivations	of	their	constituent	subagents	but	also	on	how	those
subagents	are	organized.	For	instance,	a	group	that	is	organized	under	strong
dictatorship	might	behave	as	if	it	had	a	will	that	was	identical	to	the	will	of	the
subagent	that	occupies	the	dictator	role,	whereas	a	democratic	group	might
sometimes	behave	more	as	if	it	had	a	will	that	was	a	composite	or	average	of	the
wills	of	its	various	constituents.	But	one	can	also	imagine	governance
institutions	that	would	make	an	organization	behave	in	a	way	that	is	not	a	simple
function	of	the	wills	of	its	subagents.	(Theoretically,	at	least,	there	could	exist	a
totalitarian	state	that	
everybody
	hated,	because	the	state	had	mechanisms	to
prevent	its	citizens	from	coordinating	a	revolt.	Each	citizen	could	be	worse	off
by	revolting	alone	than	by	playing	their	part	in	the	state	machinery.)
By	designing	appropriate	institutions	for	a	composite	system,	one	could	thus

everybody
try	to	shape	its	effective	motivation.	In	
Chapter	9
,	we	discussed	social
integration	as	a	possible	capability	control	method.	But	there	we	focused	on	the
incentives	faced	by	an	agent	as	a	consequence	of	its	existence	in	a	social	world
of	near-equals.	Here	we	are	focusing	on	what	happens	
inside
	a	given	agent:	how
its	will	is	determined	by	its	internal	organization.	We	are	therefore	looking	at	
a
motivation	selection	method.	Moreover,	since	this	kind	of	internal	institution

Chapter	9
inside
a
design	does	not	depend	on	large-scale	social	engineering	or	reform,	it	is	a
method	that	might	be	available	to	an	individual	project	developing
superintelligence	even	if	the	wider	socioeconomic	or	international	milieu	is	less
than	ideally	favorable.
Institution	design	is	perhaps	most	plausible	in	contexts	where	it	would	be
combined	with	augmentation.	If	we	could	start	with	agents	that	are	already
suitably	motivated	or	that	have	human-like	motivations,	institutional

arrangements	could	be	used	as	an	extra	safeguard	to	increase	the	chances	that	the
system	will	stay	on	course.
For	example,	suppose	that	we	start	with	some	well-motivated	human-like
agents—let	us	say	emulations.	We	want	to	boost	the	cognitive	capacities	of	these
agents,	but	we	worry	that	the	enhancements	might	corrupt	their	motivations.	One
way	to	deal	with	this	challenge	would	be	to	set	up	a	system	in	which	individual
emulations	function	as	subagents.	When	a	new	enhancement	is	introduced,	it	is

first	applied	to	a	small	subset	of	the	subagents.	Its	effects	are	then	studied	by	a
review	panel	composed	of	subagents	who	have	not	yet	had	the	enhancement
applied	to	them.	Only	when	these	peers	have	satisfied	themselves	that	the

enhancement	is	not	corrupting	is	it	rolled	out	to	the	wider	subagent	population.	If
the	enhanced	subagents	are	found	to	be	corrupted,	they	are	not	given	further
enhancements	and	are	excluded	from	key	decision-making	functions	(at	least
until	the	system	as	a	whole	has	advanced	to	a	point	where	the	corrupted
subagents	can	be	safely	reintegrated).
30
	Although	the	corrupted	subagents	might
have	gained	some	advantage	from	the	enhancement,	the	institutional	structure	in
which	they	are	embedded,	and	the	fact	that	they	constitute	a	small	minority	of	all
subagents,	would	hopefully	prevent	them	from	seizing	power	or	spreading	their
corruption	to	the	overall	system.	Thus,	the	collective	intelligence	and	capability
of	the	system	could	be	gradually	enhanced	in	a	sequence	of	small	steps,	where
the	soundness	of	each	step	is	verified	by	subagents	only	slightly	less	capable
than	the	new	subagents	that	are	introduced	in	that	step.
The	amount	of	safety	obtainable	by	this	kind	of	institution	design	is	limited	by

the	accuracy	of	the	tests	that	are	used	to	evaluate	the	enhanced	subagents.	The
problem	of	test	construction	is	complicated	by	the	fact	that	the	test	batteries
could	only	be	validated	within	the	pre-existing	range	of	cognitive	ability	yet
must	be	applied	to	agents	outside	this	range.	One	could	mitigate	this	problem	by
taking	small	enhancement	steps	and	by	letting	the	test	run	for	a	long	time.	Such
caution,	however,	would	raise	the	cost	and	slow	progress	(which,	if	a	race
dynamic	is	occurring,	could	mean	a	project	employing	these	safety	measures

would	place	itself	at	a	disadvantage).	Even	with	small	steps	and	extensive
testing,	there	would	remain	some	risk	of	subtle	flaws	being	introduced,	leading
to	a	gradual	corruption	of	the	goal	system,	a	corruption	that	might	become
manifest	only	when	the	system	reaches	superintelligence.	This	risk	could	be
reduced	by	
letting	the	testing	condition	extend	for	many	steps	of	cognitive
enhancement,	so	that	even	agents	that	have	been	initially	certified	as	sound	and
that	have	received	several	subsequent	enhancements	would	continue	to	be	under

reduced	by	
review	by	a	panel	of	unmodified	and	uncorrupted	original	peers	and	would
remain	subject	to	reversion.
Ideally,	the	testing	should	be	ongoing	rather	than	confined	to	an	initial
evaluation	period.	Human-level	subagents	have	the	ability	to	strategize	and
might	thus	choose	to	conceal	certain	goals	while	their	behavior	was	being
monitored.	To	enable	continuous	testing,	a	hierarchy	might	be	created	in	which
subagents	at	a	certain	level	of	capability	are	tasked	with	monitoring	subagents	at

a	somewhat	higher	level	of	capability.	At	the	bottom	of	the	capability	hierarchy
(but	at	the	top	of	the	power	hierarchy)	would	sit	the	relatively	dumb	and	slow
principal.	This	human	principal	would	be	like	a	demented	king	who	reigns	over

an	incompetent	court	that	oversees	a	mediocre	administration	which	governs	a
capable	people.	Alternatively,	he	might	be	likened	to	the	evolutionarily	ancient
“reptilian	brain”	which	ensures	that	the	newer	and	much	cleverer	neocortex	is
employed	in	the	service	of	goals	such	as	feeding	and	copulation.
One	might	question	whether	such	an	inverse	meritocracy	could	remain	stable.
If	the	bulk	of	the	superintelligent	subagents	were	found	to	be	corrupt,	and	the
humans	in	charge	of	the	whole	thing	attempted	to	roll	back	many	of	the
preceding	enhancements,	would	the	institutional	structure	be	strong	enough	to
withstand	the	opposition	that	such	a	move	would	engender?	In	the	context	of
historical	or	contemporary	human	societies,	the	answer	perhaps	is	that	this
would	be	implausible.	(Not	that	the	most	capable	people	consistently	rise	to
power,	but	we	are	not	
literally
	ruled	by	chimpanzees	either.)	The	issue	is	less
clear,	however,	in	a	context	where	society	is	implemented	as	software.
31

literally
31
Imagine	a	digital	hierarchy	that	has	one	highly	trained	police	agent	for	every
two	proletarians.	It	also	has	one	special-police	agent	for	every	two	police	agents.
It	also	has	one	special-special-police	agent	for	every	two	special-police	agents—
and	so	forth,	so	that	each	layer	of	overseers	has	at	least	half	the	numbers	of	the
layer	it	oversees.	Imagine,	further,	that	this	society	is	architected	in	such	a	way
that	a	supervisor	has	big	advantages	over	his	subordinates.	For	example,	the

31
supervisor	is	able	to	monitor	and	record	everything	his	subordinates	say	and	do,
whereas	the	supervisor	himself	is	hidden	behind	a	one-way	glass,	so	that	his
subordinates	can	only	hear	the	orders	and	instructions	that	the	superior	chooses
to	transmit.	Imagine	that	supervisors	have	a	panel	of	buttons	that	they	can	press
at	any	time	to	cause	a	subordinate	to	be	punished	or	rewarded,	paused,	deleted,
or	reverted	to	an	earlier	state.	These	capabilities	would	greatly	boost	the	powers

of	a	supervising	agent,	beyond	what	has	been	possible	historically	in	even	the
most	totalitarian	of	organizations.
Even	this	does	not	exhaust	the	possible	instruments	of	control.	Supervisors
might	also	have	at	their	disposal	realistic	virtual	reality	simulations	that	they	can
use	to	probe	subordinates’	reactions	to	hypothetical	scenarios	(such	as	simulated
offers	to	join	a	conspiracy	or	to	slack	when	they	think	nobody	is	watching).
Supervisors	
might,	additionally,	have	the	ability	to	eavesdrop	on	internal

monologues	in	the	subordinates’	minds	and	to	directly	manipulate	their
emotional	states.
The	upshot	is	a	series	of	massive	advantages	for	the	supervisor.
32
	Each

supervisor	would	be	a	subordinate	to	another,	higher-level	supervisor,	who
would	monitor	
his
	every	move	and	make	sure	
he
	did	not	slacken	in	the	execution
of	his	managerial	duties.	With	this	kind	of	arrangement,	it	might	be	possible	for
the	structure	to	be	stable	even	if	each	supervisory	level	is	intellectually
somewhat	weaker	than	the	level	it	supervises.
33
	And	the	entire	structure,	which
could	include	many	highly	superintelligent	agents,	may	be	controlled	by	a	small
number	of	humans	with	root	access.
34
The	cost	of	such	an	internal	control	structure	would	be	significant	but	not
necessarily	prohibitive.	For	example,	in	the	model	where	each	level	of
supervision	has	half	the	numbers	of	the	layer	below,	the	extra	computational
overhead	is	bounded	at	a	mere	100%	of	what	the	proletarian	part	of	the	system
costs—less	if	the	dumber	boss	layers	require	fewer	computations	per	subagent.
Oversight	could	be	tightened	by	increasing	the	ratio,	or	cost	lowered	by

34
decreasing	it.	The	extra	computational	cost	would	be	a	major	factor	in	a
competitive	market	scenario	(cf.	
Chapter	11
)	but	looks	affordable	in	the	context
of	a	project	that	is	not	facing	strong	immediate	competition.	There	would	also	be
a	cost	in	terms	of	the	development	time	needed	to	create	and	test	these
supervisory	functionalities.	A	well-resourced	project	could	reduce	this	time	cost
by	parallelizing	the	development	of	the	control	structure	with	the	development

Chapter	11
of	the	machine	intelligence;	but	the	extra	task	load	could	be	prohibitive	for
smaller	projects	and	for	projects	caught	in	a	close	technology	race.
One	other	type	of	cost	also	deserves	consideration:	the	risk	of	mind	crimes
being	committed	in	this	kind	of	structure.
35
	As	described,	the	institution	sounds
like	a	rather	horrible	North	Korean	labor	camp.	Yet	there	are	ways	of	at	least
mitigating	the	moral	problems	with	running	this	kind	of	institution,	even	if	the
subagents	contained	in	the	institution	are	emulations	with	full	human	moral

35
status.	At	a	minimum,	the	system	could	rely	on	volunteering	emulations.	Each
subagent	could	have	the	option	at	any	time	of	withdrawing	its	participation.
36
Terminated	emulations	could	be	stored	to	memory,	with	a	commitment	to	restart
them	under	much	more	ideal	conditions	once	the	dangerous	phase	of	the
intelligence	explosion	is	over.	Meanwhile,	subagents	who	chose	to	participate
could	be	housed	in	very	comfortable	virtual	environments	and	allowed	ample
time	for	sleep	and	recreation.	These	measures	would	impose	a	cost,	one	that

36
should	be	manageable	for	a	well-resourced	project	under	noncompetitive
conditions.	In	a	highly	competitive	situation,	the	cost	may	be	unaffordable	unless
an	enterprise	could	be	assured	that	its	competitors	would	incur	the	same	cost.

In	the	example,	we	imagined	the	subagents	as	emulations.	One	might	wonder,
does	the	institution	design	approach	require	that	the	subagents	be
anthropomorphic?	Or	is	it	equally	applicable	to	systems	composed	of	artificial
subagents?
One’s	first	thought	here	might	be	skeptical.	One	notes	that	despite	our
plentiful	experience	with	human-like	agents,	we	still	cannot	precisely	predict	the
outbreak	
or	outcomes	of	revolutions;	social	science	can,	at	most,	describe	some
statistical	tendencies.
37
	Since	we	cannot	reliably	predict	the	stability	of	social
structures	for	ordinary	human	beings	(about	which	we	have	much	data),	it	is
tempting	to	infer	that	we	have	little	hope	of	precision-engineering	stable	social
structures	for	cognitively	enhanced	human-like	agents	(about	which	we	have	no
data),	and	that	we	have	still	less	hope	of	doing	so	for	advanced	artificial	agents
(which	are	not	even	similar	to	agents	that	we	have	data	about).
Yet	the	matter	is	not	so	cut-and-dried.	Humans	and	human-like	beings	are

complex;	but	artificial	agents	could	have	relatively	simple	architectures.
Artificial	agents	could	also	have	simple	and	explicitly	characterized	motivations.
Furthermore,	digital	agents	in	general	(whether	emulations	or	artificial
intelligences)	are	copyable:	an	affordance	that	may	revolutionize	management,
much	like	interchangeable	parts	revolutionized	manufacturing.	These
differences,	together	with	the	opportunity	to	work	with	agents	that	are	initially
powerless	and	to	create	institutional	structures	that	use	the	various

abovementioned	control	measures,	might	combine	to	make	it	possible	to	achieve
particular	institutional	outcomes—such	as	a	system	that	does	not	revolt—more
reliably	than	if	one	were	working	with	human	beings	under	historical	conditions.
But	then	again,	artificial	agents	might	lack	many	of	the	attributes	that	help	us
predict	the	behavior	of	human-like	agents.	Artificial	agents	need	not	have	any	of
the	social	emotions	that	bind	human	behavior,	emotions	such	as	fear,	pride,	and
remorse.	Nor	need	artificial	agents	develop	attachments	to	friends	and	family.

Nor	need	they	exhibit	the	unconscious	body	language	that	makes	it	difficult	for
us	humans	to	conceal	our	intentions.	These	deficits	might	destabilize	institutions
of	artificial	agents.	Moreover,	artificial	agents	might	be	capable	of	making	big
leaps	in	cognitive	performance	as	a	result	of	seemingly	small	changes	in	their
algorithms	or	architecture.	Ruthlessly	optimizing	artificial	agents	might	be
willing	to	take	extreme	gambles	from	which	humans	would	shrink.
38
	And

38
	And
superintelligent	agents	might	show	a	surprising	ability	to	coordinate	with	little	or
no	communication	(e.g.	by	internally	modeling	each	other’s	hypothetical

responses	to	various	contingencies).	These	and	other	differences	could	make
sudden	institutional	failure	more	likely,	even	in	the	teeth	of	what	seem	like
Kevlar-clad	methods	of	social	control.
It	is	unclear,	therefore,	how	promising	the	institution	design	approach	is,	and
whether	it	has	a	greater	chance	of	working	with	anthropomorphic	than	with
artificial	agents.	It	might	be	thought	that	creating	an	institution	with	appropriate
checks	and	balances	could	only	increase	safety—or,	at	any	rate,	not	reduce
safety—so	that	from	a	risk-mitigation	perspective	it	would	always	be	best	if	the
method	were	used.	But	even	this	cannot	be	said	with	certainty.	The	approach
adds	parts	and	complexity,	and	thus	may	also	introduce	new	ways	for	things	to
go	wrong	that	do	not	exist	in	the	case	of	an	agent	that	does	not	have	intelligent
subagents	as	parts.	Nevertheless,	institution	design	is	worthy	of	further
exploration.
39
Synopsis
	
Goal	system	engineering	is	not	yet	an	established	discipline.	It	is	not	currently

39
Synopsis
	
known	how	to	transfer	human	values	to	a	digital	computer,	even	given	human-
level	machine	intelligence.	Having	investigated	a	number	of	approaches,	we
found	that	some	of	them	appear	to	be	dead	ends;	but	others	appear	to	hold
promise	and	deserve	to	be	explored	further.	A	summary	is	provided	in	
Table	12
.
Table	12	
Summary	of	value-loading	techniques
	
Explicit
representation
May	hold	promise	as	a	way	of	loading	domesticity	values.	Does
not	seem	promising	as	a	way	of	loading	more	complex	values.
Evolutionary

Table	12
.
Table	12	
	
Explicit
Less	promising.	Powerful	search	may	find	a	design	that	satisfies
the	formal	search	criteria	but	not	our	intentions.	Furthermore,	if
designs	are	evaluated	by	running	them—including	designs	that
do	not	even	meet	the	formal	criteria—a	potentially	grave

selection
additional	danger	is	created.	Evolution	also	makes	it	difficult	to
avoid	massive	mind	crime,	especially	if	one	is	aiming	to	fashion
human-like	minds.
Reinforcement
learning
A	range	of	different	methods	can	be	used	to	solve
“reinforcement-learning	problems,”	but	they	typically	involve
creating	a	system	that	seeks	to	maximize	a	reward	signal.	This
has	an	inherent	tendency	to	produce	the	wireheading	failure
mode	when	the	system	becomes	more	intelligent.	Reinforcement
learning	therefore	looks	unpromising.
Value
accretion
We	humans	acquire	much	of	our	specific	goal	content	from	our
reactions	to	experience.	While	value	accretion	could	in	principle
be	used	to	create	an	agent	with	human	motivations,	the	human
value-accretion	dispositions	might	be	complex	and	difficult	to
replicate	in	a	seed	AI.	A	bad	approximation	may	yield	an	AI	that
generalizes	differently	than	humans	do	and	therefore	acquires
unintended	final	goals.	More	research	is	needed	to	determine

Value
accretion
how	difficult	it	would	be	to	make	value	accretion	work	with
sufficient	precision.
Motivational
scaffolding
It	is	too	early	to	tell	how	difficult	it	would	be	to	encourage	a
system	to	develop	internal	high-level	representations	that	are
transparent	to	humans	(while	keeping	the	system’s	capabilities
below	the	dangerous	level)	and	then	to	use	those	representations
to	design	a	new	goal	system.	The	approach	might	hold
considerable	promise.	(However,	as	with	any	untested	approach
that	would	postpone	much	of	the	hard	work	on	safety

engineering	until	the	development	of	human-level	AI,	one
should	be	careful	not	to	allow	it	to	become	an	excuse	for	a
lackadaisical	attitude	to	the	control	problem	in	the	interim.)
A	potentially	promising	approach,	but	more	research	is	needed
to	determine	how	difficult	it	would	be	to	formally	specify	a

reference	that	successfully	points	to	the	relevant	external
information	about	human	value	(and	how	difficult	it	would	be	to
specify	a	correctness	criterion	for	a	utility	function	in	terms	of
such	a	reference).	Also	worth	exploring	within	the	value
learning	category	are	proposals	of	the	Hail	Mary	type	or	along
the	lines	of	Paul	Christiano’s	construction	(or	other	such
shortcuts).
Emulation
modulation
If	machine	intelligence	is	achieved	via	the	emulation	pathway,	it
would	likely	be	possible	to	tweak	motivations	through	the	digital
equivalent	of	drugs	or	by	other	means.	Whether	this	would
enable	values	to	be	loaded	with	sufficient	precision	to	ensure
safety	even	as	the	emulation	is	boosted	to	superintelligence	is	an
open	question.	(Ethical	constraints	might	also	complicate
developments	in	this	direction.)
Institution
design
Various	strong	methods	of	social	control	could	be	applied	in	an
institution	composed	of	emulations.	In	principle,	social	control

design
methods	could	also	be	applied	in	an	institution	composed	of
artificial	intelligences.	Emulations	have	some	properties	that
would	make	them	easier	to	control	via	such	methods,	but	also
some	properties	that	might	make	them	harder	to	control	than
AIs.	Institution	design	seems	worthy	of	further	exploration	as	a
potential	value-loading	technique.
If	we	knew	how	to	solve	the	value-loading	problem,	we	would	confront	a
further	problem:	the	problem	of	deciding	which	values	to	load.	What,	in	other

words,	would	we	want	a	superintelligence	to	want?	This	is	the	more
philosophical	problem	to	which	we	turn	next.

CHAPTER	13
Choosing	the	criteria	for	choosing
	
Suppose	we	could	install	any	arbitrary	final	value	into	a	seed	AI.	The
decision	as	to	which	value	to	install	could	then	have	the	most	far-reaching
consequences.	Certain	other	basic	parameter	choices—concerning	the
axioms	of	the	AI’s	decision	theory	and	epistemology—could	be	similarly
consequential.	But	foolish,	ignorant,	and	narrow-minded	that	we	are,	how
could	we	be	trusted	to	make	good	design	decisions?	How	could	we	choose
without	locking	in	forever	the	prejudices	and	preconceptions	of	the	present
generation?	In	this	chapter,	we	explore	how	indirect	normativity	can	let	us
offload	much	of	the	cognitive	work	involved	in	making	these	decisions	onto
the	superintelligence	itself	while	still	anchoring	the	outcome	in	deeper
human	values.
	
How	can	we	get	a	superintelligence	to	do	what	we	want?	What	do	we	want	the
superintelligence	to	want?	Up	to	this	point,	we	have	focused	on	the	former
question.	We	now	turn	to	the	second	question.

Suppose	that	we	had	solved	the	control	problem	so	that	we	were	able	to	load
any	value	we	chose	into	the	motivation	system	of	a	superintelligence,	making	it
pursue	that	value	as	its	final	goal.	Which	value	should	we	install?	The	choice	is
no	light	matter.	If	the	superintelligence	obtains	a	decisive	strategic	advantage,
the	value	would	determine	the	disposition	of	the	cosmic	endowment.
Clearly,	it	is	essential	that	we	not	make	a	mistake	in	our	value	selection.	But
how	could	we	realistically	hope	to	achieve	errorlessness	in	a	matter	like	this?	We

might	be	wrong	about	morality;	wrong	also	about	what	is	good	for	us;	wrong
even	about	what	we	truly	want.	Specifying	a	final	goal,	it	seems,	requires
making	one’s	way	through	a	thicket	of	thorny	philosophical	problems.	If	we	try	a

direct	approach,	we	are	likely	to	make	a	hash	of	things.	The	risk	of	mistaken
choosing	is	especially	
high	when	the	decision	context	is	unfamiliar—and
selecting	the	final	goal	for	a	machine	superintelligence	that	will	shape	all	of
humanity’s	future	is	an	extremely	unfamiliar	decision	context	if	any	is.
The	dismal	odds	in	a	frontal	assault	are	reflected	in	the	pervasive	dissensus
about	the	relevant	issues	in	value	theory.	No	ethical	theory	commands	majority
support	among	philosophers,	so	most	philosophers	must	be	wrong.
1
	It	is	also
reflected	in	the	marked	changes	that	the	distribution	of	moral	belief	has
undergone	over	time,	many	of	which	we	like	to	think	of	as	progress.	In	medieval
Europe,	for	instance,	it	was	deemed	respectable	entertainment	to	watch	a
political	prisoner	being	tortured	to	death.	Cat-burning	remained	popular	in
sixteenth-century	Paris.
2
	A	mere	hundred	and	fifty	years	ago,	slavery	still	was
widely	practiced	in	the	American	South,	with	full	support	of	the	law	and	moral

1
	It	is	also
2
custom.	When	we	look	back,	we	see	glaring	deficiencies	not	just	in	the	behavior
but	in	the	moral	beliefs	of	all	previous	ages.	Though	we	have	perhaps	since
gleaned	some	moral	insight,	we	could	hardly	claim	to	be	now	basking	in	the
high	noon	of	perfect	moral	enlightenment.	Very	likely,	we	are	still	laboring
under	one	or	more	grave	moral	misconceptions.	In	such	circumstances	to	select	a
final	value	based	on	our	current	convictions,	in	a	way	that	locks	it	in	forever	and

precludes	any	possibility	of	further	ethical	progress,	would	be	to	risk	an
existential	moral	calamity.
Even	if	we	could	be	rationally	confident	that	we	have	identified	the	correct
ethical	theory—which	we	cannot	be—we	would	still	remain	at	risk	of	making
mistakes	in	developing	important	details	of	this	theory.	Seemingly	simple	moral
theories	can	have	a	lot	of	hidden	complexity.
3
	For	example,	consider	the
(unusually	simple)	consequentialist	theory	of	hedonism.	This	theory	states,

3
roughly,	that	all	and	only	pleasure	has	value,	and	all	and	only	pain	has	disvalue.
4
Even	if	we	placed	all	our	moral	chips	on	this	one	theory,	and	the	theory	turned
out	to	be	right,	a	great	many	questions	would	remain	open.	Should	“higher
pleasures”	be	given	priority	over	“lower	pleasures,”	as	John	Stuart	Mill	argued?
How	should	the	intensity	and	duration	of	a	pleasure	be	factored	in?	Can	pains
and	pleasures	cancel	each	other	out?	What	kinds	of	brain	states	are	associated

4
with	morally	relevant	pleasures?	Would	two	exact	copies	of	the	same	brain	state
correspond	to	twice	the	amount	of	pleasure?
5
	Can	there	be	subconscious
pleasures?	How	should	we	deal	with	extremely	small	chances	of	extremely	great
pleasures?
6
	How	should	we	aggregate	over	infinite	populations?
7

Giving	the	wrong	answer	to	any	one	of	these	questions	could	be	catastrophic.
If	by	selecting	a	final	value	for	the	superintelligence	we	had	to	place	a	bet	not
just	on	a	general	moral	theory	but	on	a	long	conjunction	of	specific	claims	about
how	that	theory	is	to	be	interpreted	and	integrated	into	an	effective	decision-
making	process,	then	our	chances	of	striking	lucky	would	dwindle	to	something
close	to	hopeless.	Fools	might	eagerly	accept	this	challenge	of	solving	in	one
swing	all	the	important	problems	in	moral	philosophy,	in	order	to	infix	their
favorite	answers	into	the	seed	AI.	Wiser	souls	would	look	hard	for	some
alternative	approach,	some	way	to	hedge.
This	takes	us	to	indirect	normativity.	The	obvious	reason	for	building	a
superintelligence	is	so	that	we	can	offload	to	it	the	instrumental	reasoning
required	to	find	effective	ways	of	realizing	a	given	value.	Indirect	normativity
would	enable	us	also	to	offload	to	the	superintelligence	some	of	the	reasoning

needed	to	select	the	value	that	is	to	be	realized.
Indirect	normativity	is	a	way	to	answer	the	challenge	presented	by	the	fact
that	we	may	not	know	what	we	truly	want,	what	is	in	our	interest,	or	what	is
morally	right	or	ideal.	Instead	of	making	a	guess	based	on	our	own	current
understanding	(which	is	probably	deeply	flawed),	we	would	delegate	some	of
the	cognitive	work	required	for	value	selection	to	the	superintelligence.	Since	the
superintelligence	is	better	at	cognitive	work	than	we	are,	it	may	see	past	the

errors	and	confusions	that	cloud	our	thinking.	One	could	generalize	this	idea	and
emboss	it	as	a	heuristic	principle:
	
The	principle	of	epistemic	deference
A	future	superintelligence	occupies	an	epistemically	superior	vantage	point:
its	beliefs	are	(probably,	on	most	topics)	more	likely	than	ours	to	be	true.
We	should	therefore	defer	to	the	superintelligence’s	opinion	whenever
feasible.
8
	
Indirect	normativity	applies	this	principle	to	the	value-selection	problem.
Lacking	confidence	in	our	ability	to	specify	a	concrete	normative	standard,	we

feasible.
8
	
would	instead	specify	some	more	abstract	condition	that	any	normative	standard
should	satisfy,	in	the	hope	that	a	superintelligence	could	find	a	concrete	standard

that	satisfies	the	abstract	condition.	We	could	give	a	seed	AI	the	final	goal	of
continuously	acting	according	to	its	best	estimate	of	what	this	implicitly	defined
standard	would	have	it	do.
Some	examples	will	serve	to	make	the	idea	clearer.	First	we	will	consider
“coherent	extrapolated	volition,”	an	indirect	normativity	proposal	outlined	by
Eliezer	Yudkowsky.	We	will	then	introduce	some	variations	and	alternatives,	to
give	us	a	sense	of	the	range	of	available	options.
	
Yudkowsky	has	proposed	that	a	seed	AI	be	given	the	final	goal	of	carrying	out
humanity’s	“coherent	extrapolated	volition”	(CEV),	which	he	defines	as	follows:
Our	coherent	extrapolated	volition	is	our	wish	if	we	knew	more,	thought
faster,	were	more	the	people	we	wished	we	were,	had	grown	up	farther
together;	where	the	extrapolation	converges	rather	than	diverges,	where	our
wishes	cohere	rather	than	interfere;	extrapolated	as	we	wish	that
extrapolated,	interpreted	as	we	wish	that	interpreted.
9

9
	
When	Yudkowsky	wrote	this,	he	did	not	purport	to	present	a	blueprint	for	how	to
implement	this	rather	poetic	prescription.	His	aim	was	to	give	a	preliminary
sketch	of	how	CEV	might	be	defined,	along	with	some	arguments	for	why	an
approach	along	these	lines	is	needed.
Many	of	the	ideas	behind	the	CEV	proposal	have	analogs	and	antecedents	in
the	philosophical	literature.	For	example,	in	ethics	
ideal	observer	theories
	seek
to	analyze	normative	concepts	like	“good”	or	“right”	in	terms	of	the	judgments

9
	
	seek
that	a	hypothetical	ideal	observer	would	make	(where	an	“ideal	observer”	is
defined	as	one	that	is	omniscient	about	non-moral	facts,	is	logically	clear-
sighted,	is	impartial	in	relevant	ways	and	is	free	from	various	kinds	of	biases,
and	so	on).
10
	The	CEV	approach,	however,	is	not	(or	need	not	be	construed	as)	a
moral	theory.	It	is	not	committed	to	the	claim	that	there	is	any	necessary	link
between	value	and	the	preferences	of	our	coherent	extrapolated	volition.	CEV

can	be	thought	of	simply	as	a	useful	way	to	approximate	whatever	has	ultimate
value,	or	it	can	be	considered	aside	from	any	connection	to	ethics.	As	the	main
prototype	of	the	indirect	normativity	approach,	it	is	worth	examining	in	a	little
more	detail.
	
Some	terms	in	the	above	quotation	require	explication.	“Thought	faster,”	in
Yudkowsky’s	terminology,	means	
if	we	were	smarter	and	had	thought	things
through	more
.	“Grown	up	farther	together”	seems	to	mean	
if	we	had	done	our
learning,	our	cognitive	enhancing,	and	our	self-improving	under	conditions	of
suitable	social	interaction	with	one	another
.
“Where	the	extrapolation	converges	rather	than	diverges”	may	be	understood
as	follows.	The	AI	should	act	on	some	feature	of	the	result	of	its	extrapolation
only	insofar	as	that	feature	can	be	predicted	by	the	AI	with	a	fairly	high	degree
of	confidence.	To	the	extent	that	the	AI	cannot	predict	what	we	would	wish	if	we
were	idealized	in	the	manner	indicated,	the	AI	should	not	act	on	a	wild	guess;

.
instead,	it	should	refrain	from	acting.	However,	even	though	many	details	of	our
idealized	wishing	may	be	undetermined	or	unpredictable,	there	might
nevertheless	be	some	broad	outlines	that	the	AI	can	apprehend,	and	it	can	then	at
least	act	to	ensure	that	the	future	course	of	events	unfolds	within	those	outlines.
For	example,	if	the	AI	can	reliably	estimate	that	our	extrapolated	volition	would
wish	that	we	not	all	be	in	constant	agony,	or	that	the	universe	not	be	tiled	over
with	paperclips,	then	the	AI	should	act	to	prevent	those	outcomes.
11

11
“Where	our	wishes	cohere	rather	than	interfere”	may	be	read	as	follows.	The
AI	should	act	where	there	is	fairly	broad	agreement	between	individual	humans’
extrapolated	volitions.	A	smaller	set	of	strong,	clear	wishes	might	sometimes
outweigh	the	weak	and	muddled	wishes	of	a	majority.	Also,	Yudkowsky	thinks
that	it	should	require	less	consensus	for	the	AI	to	
prevent
	some	particular
narrowly	specified	outcome,	and	more	consensus	for	the	AI	to	act	to	funnel	the
future	into	some	particular	narrow	conception	of	the	good.	“The	initial	dynamic

prevent
for	CEV,”	he	writes,	“should	be	conservative	about	saying	‘yes,’	and	listen
carefully	for	‘no.’”
12
“Extrapolated	as	we	wish	that	extrapolated,	interpreted	as	we	wish	that

interpreted”:	The	idea	behind	these	last	modifiers	seems	to	be	that	the	rules	for
extrapolation	should	themselves	be	sensitive	to	the	extrapolated	volition.	An
individual	might	have	a	second-order	desire	(a	desire	concerning	what	to	desire)
that	some	
of	her	first-order	desires	not	be	given	weight	when	her	volition	is
extrapolated.	For	example,	an	alcoholic	who	has	a	first-order	desire	for	booze
might	also	have	a	second-order	desire	not	to	have	that	first-order	desire.
Similarly,	we	might	have	desires	over	how	various	other	parts	of	the
extrapolation	process	should	unfold,	and	these	should	be	taken	into	account	by
the	extrapolation	process.
It	might	be	objected	that	even	if	the	concept	of	humanity’s	coherent
extrapolated	volition	could	be	properly	defined,	it	would	anyway	be	impossible
—even	for	a	superintelligence—to	find	out	what	humanity	would	actually	want
under	the	hypothetical	idealized	circumstances	stipulated	in	the	CEV	approach.

Without	some	information	about	the	content	of	our	extrapolated	volition,	the	AI
would	be	bereft	of	any	substantial	standard	to	guide	its	behavior.	However,
although	it	would	be	difficult	to	know	with	precision	what	humanity’s	CEV
would	wish,	it	is	possible	to	make	informed	guesses.	This	is	possible	even	today,
without	superintelligence.	For	example,	it	is	more	plausible	that	our	CEV	would
wish	for	there	to	be	people	in	the	future	who	live	rich	and	happy	lives	than	that	it

would	wish	that	we	should	all	sit	on	stools	in	a	dark	room	experiencing	pain.	If
we
	can	make	at	least	some	such	judgments	sensibly,	so	can	a	superintelligence.
From	the	outset,	the	superintelligence’s	conduct	could	thus	be	guided	by	its
estimates	of	the	content	of	our	CEV.	It	would	have	strong	instrumental	reason	to
refine	these	initial	estimates	(e.g.	by	studying	human	culture	and	psychology,
scanning	human	brains,	and	reasoning	about	how	we	might	behave	if	we	knew

we
more,	thought	more	clearly,	etc.).	In	investigating	these	matters,	the	AI	would	be
guided	by	its	initial	estimates	of	our	CEV;	so	that,	for	instance,	the	AI	would	not
unnecessarily	run	myriad	simulations	replete	with	unredeemed	human	suffering
if	it	estimated	that	our	CEV	would	probably	condemn	such	simulations	as	mind
crime.
Another	objection	is	that	there	are	so	many	different	ways	of	life	and	moral
codes	in	the	world	that	it	might	not	be	possible	to	“blend”	them	into	one	CEV.

crime.
Even	if	one	could	blend	them,	the	result	might	not	be	particularly	appetizing—
one	would	be	unlikely	to	get	a	delicious	meal	by	mixing	together	all	the	best
flavors	from	everyone’s	different	favorite	dish.
13
	In	answer	to	this,	one	could
point	out	that	the	CEV	approach	does	not	require	that	all	ways	of	life,	moral
codes,	or	personal	values	be	blended	together	into	one	stew.	The	CEV	dynamic

is	supposed	to	act	only	when	our	wishes	cohere.	On	issues	on	which	there	is
widespread	irreconcilable	disagreement,	even	after	the	various	idealizing
conditions	have	been	imposed,	the	dynamic	should	refrain	from	determining	the
outcome.	To	continue	the	cooking	analogy,	it	might	be	that	individuals	or
cultures	will	have	different	favorite	dishes,	but	that	they	can	nevertheless
broadly	agree	that	aliments	should	be	nontoxic.	The	CEV	dynamic	could	then
act	to	prevent	food	poisoning	while	otherwise	allowing	humans	to	work	out	their
culinary	practices	without	its	guidance	or	interference.
	
Yudkowsky’s	article	offered	seven	arguments	for	the	CEV	approach.	Three	of
these	were	basically	different	ways	of	making	the	point	that	while	the	aim	should
be	to	do	something	that	is	humane	and	helpful,	it	would	be	very	difficult	to	lay
down	an	explicit	set	of	rules	that	does	not	have	unintended	interpretations	and
undesirable	consequences.
14
	The	CEV	approach	is	meant	to	be	robust	and	self-

14
correcting;	it	is	meant	to	capture	the	
source
	of	our	values	instead	of	relying	on	us
correctly	enumerating	and	articulating,	once	and	for	all,	each	of	our	essential
values.
The	remaining	four	arguments	go	beyond	that	first	basic	(but	important)	point,
spelling	out	desiderata	on	candidate	solutions	to	the	value-specification	problem
and	suggesting	that	CEV	meets	these	desiderata.
“Encapsulate	moral	growth”
This	is	the	desideratum	that	the	solution	should	allow	for	the	possibility	of	moral

source
values.
progress.	As	suggested	earlier,	there	are	reasons	to	believe	that	our	current	moral
beliefs	are	flawed	in	many	ways;	perhaps	deeply	flawed.	If	we	were	to	stipulate
a	specific	and	unalterable	moral	code	for	the	AI	to	follow,	we	would	in	effect	be
locking	in	our	present	moral	convictions,	including	their	errors,	destroying	any
hope	of	moral	growth.	The	CEV	approach,	by	contrast,	allows	for	the	possibility
of	such	growth	because	it	has	the	AI	try	to	do	that	which	we	would	have	wished

it	to	do	if	we	had	developed	further	under	favorable	conditions,	and	it	is	possible
that	if	we	had	thus	developed	our	moral	beliefs	and	sensibilities	would	have
been	purged	of	their	current	defects	and	limitations.

“Avoid	hijacking	the	destiny	of	humankind”
Yudkowsky	has	in	mind	a	scenario	in	which	a	small	group	of	programmers
creates	a	seed	AI	that	then	grows	into	a	superintelligence	that	obtains	a	decisive
strategic	advantage.	In	this	scenario,	the	original	programmers	hold	in	their
hands	the	entirety	of	humanity’s	cosmic	endowment.	Obviously,	this	is	a	hideous
responsibility	for	any	mortal	to	shoulder.	Yet	it	is	not	possible	for	the
programmers	to	completely	shirk	the	onus	once	they	find	themselves	in	this
situation:	any	choice	they	make,	including	abandoning	the	project,	would	have
world-historical	consequences.	Yudkowsky	sees	CEV	as	a	way	for	the
programmers	to	avoid	arrogating	to	themselves	the	privilege	or	burden	of
determining	humanity’s	future.	By	setting	up	a	dynamic	that	implements
humanity’s
	coherent	extrapolated	volition—as	opposed	to	their	own	volition,	or
their	own	favorite	moral	theory—they	in	effect	distribute	their	influence	over	the
future	to	all	of	humanity.

“Avoid	creating	a	motive	for	modern-day	humans	to	fight	over	the	initial	dynamic”
Distributing	influence	over	humanity’s	future	is	not	only	morally	preferable	to
the	programming	team	implementing	their	own	favorite	vision,	it	is	also	a	way	to
reduce	the	incentive	to	fight	over	who	gets	to	create	the	first	superintelligence.	In
the	CEV	approach,	the	programmers	(or	their	sponsors)	exert	no	more	influence
over	the	content	of	the	outcome	than	any	other	person—though	they	of	course

play	a	starring	causal	role	in	determining	the	structure	of	the	extrapolation	and	in
deciding	to	implement	humanity’s	CEV	instead	of	some	alternative.	Avoiding
conflict	is	important	not	only	because	of	the	immediate	harm	that	conflict	tends
to	cause	but	also	because	it	hinders	collaboration	on	the	difficult	challenge	of
developing	superintelligence	safely	and	beneficially.
CEV	is	meant	to	be	capable	of	commanding	wide	support.	This	is	not	just

because	it	allocates	influence	equitably.	There	is	also	a	deeper	ground	for	the
irenic	potential	of	CEV,	namely	that	it	enables	many	different	groups	to	hope
that	their	preferred	vision	of	the	future	will	prevail	totally.	Imagine	a	member	of
the	Afghan	Taliban	debating	with	a	member	of	the	Swedish	Humanist
Association.	The	two	have	very	different	worldviews,	and	what	is	a	utopia	for
one	might	be	a	dystopia	for	the	other.	Nor	might	either	be	thrilled	by	any
compromise	position,	such	as	permitting	girls	to	receive	an	education	but	only

up	to	ninth	grade,	or	permitting	Swedish	girls	to	be	educated	but	Afghan	girls
not.	However,	both	the	Taliban	and	the	Humanist	might	be	able	to	endorse	the

principle	that	the	future	should	be	determined	by	humanity’s	CEV.	The	Taliban
could	reason	that	if	his	religious	views	are	in	fact	correct	(as	he	is	convinced
they	are)	and	if	good	grounds	for	accepting	these	views	exist	(as	he	is	also
convinced)	then	humankind	would	in	the	end	come	to	accept	these	views	if	only
people	were	less	prejudiced	and	biased,	if	they	spent	more	time	studying
scripture,	if	they	could	more	clearly	understand	how	the	world	works	and
recognize	essential	priorities,	if	they	could	be	freed	from	irrational
rebelliousness	and	cowardice,	and	so	forth.
15
	The	Humanist,	similarly,	would
believe	that	under	these	idealized	conditions,	humankind	would	come	to
embrace	the	principles	she	espouses.
“Keep	humankind	ultimately	in	charge	of	its	own	destiny”
We	might	not	want	an	outcome	in	which	a	paternalistic	superintelligence
watches	over	us	constantly,	micromanaging	our	affairs	with	an	eye	towards
optimizing	every	detail	in	accordance	with	a	grand	plan.	Even	if	we	stipulate	that

15
the	superintelligence	would	be	perfectly	benevolent,	and	free	from
presumptuousness,	arrogance,	overbearingness,	narrow-mindedness,	and	other
human	shortcomings,	one	might	still	resent	the	loss	of	autonomy	entailed	by
such	an	arrangement.	We	might	prefer	to	create	our	destiny	as	we	go	along,	even
if	it	means	that	we	sometimes	fumble.	Perhaps	we	want	the	superintelligence	to
serve	as	a	safety	net,	to	support	us	when	things	go	catastrophically	wrong,	but
otherwise	to	leave	us	to	fend	for	ourselves.

CEV	allows	for	this	possibility.	CEV	is	meant	to	be	an	“initial	dynamic,”	a
process	that	runs	once	and	then	replaces	itself	with	whatever	the	extrapolated
volition	wishes.	If	humanity’s	extrapolated	volition	wishes	that	we	live	under	the
supervision	of	a	paternalistic	AI,	then	the	CEV	dynamic	would	create	such	an	AI
and	hand	it	the	reins.	If	humanity’s	extrapolated	volition	instead	wishes	that	a
democratic	human	world	government	be	created,	then	the	CEV	dynamic	might

facilitate	the	establishment	of	such	an	institution	and	otherwise	remain	invisible.
If	humanity’s	extrapolated	volition	is	instead	that	each	person	should	get	an
endowment	of	resources	that	she	can	use	as	she	pleases	so	long	as	she	respects
the	equal	rights	of	others,	then	the	CEV	dynamic	could	make	this	come	true	by
operating	in	the	background	much	like	a	law	of	nature,	to	prevent	trespass,	theft,
assault,	and	other	nonconsensual	impingements.
16
The	structure	of	the	CEV	approach	thus	allows	for	a	virtually	unlimited	range

16
of	outcomes.	It	is	also	conceivable	that	humanity’s	extrapolated	volition	would

wish	that	the	CEV	does	nothing	at	all.	In	that	case,	the	AI	implementing	CEV
should,	upon	having	established	with	sufficient	probability	that	this	is	what
humanity’s	extrapolated	volition	would	wish	it	to	do,	safely	shut	itself	down.
	
The	CEV	proposal,	as	outlined	above,	is	of	course	the	merest	schematic.	It	has	a
number	of	free	parameters	that	could	be	specified	in	various	ways,	yielding
different	versions	of	the	proposal.
One	parameter	is	the	extrapolation	base:	Whose	volitions	are	to	be	included?
We	might	say	“everybody,”	but	this	answer	spawns	a	host	of	further	questions.
Does	the	extrapolation	base	include	so-called	“marginal	persons”	such	as
embryos,	fetuses,	brain-dead	persons,	patients	with	severe	dementias	or	who	are
in	permanent	vegetative	states?	Does	each	of	the	hemispheres	of	a	“split-brain”
patient	get	its	own	weight	in	the	extrapolation	and	is	this	weight	the	same	as	that
of	the	entire	brain	of	a	normal	subject?	What	about	people	who	lived	in	the	past

but	are	now	dead?	People	who	will	be	born	in	the	future?	Higher	animals	and
other	sentient	creatures?	Digital	minds?	Extraterrestrials?
One	option	would	be	to	include	only	the	population	of	adult	human	beings	on
Earth	who	are	alive	at	the	start	of	the	time	of	the	AI’s	creation.	An	initial
extrapolation	from	this	base	could	then	decide	whether	and	how	the	base	should
be	expanded.	Since	the	number	of	“marginals”	at	the	periphery	of	this	base	is
relatively	small,	the	result	of	the	extrapolation	may	not	depend	much	on	exactly

where	the	boundary	is	drawn—on	whether,	for	instance,	it	includes	fetuses	or
not.
That	somebody	is	excluded	from	the	original	extrapolation	base	does	not
imply	that	their	wishes	and	well-being	are	disregarded.	If	the	coherent
extrapolated	volition	of	those	in	the	extrapolation	base	(e.g.	living	adult	human
beings)	wishes	that	moral	consideration	be	extended	to	other	beings,	then	the
outcome	of	the	CEV	dynamic	would	reflect	that	preference.	Nevertheless,	it	is
possible	that	the	interests	of	those	who	are	included	in	the	original	extrapolation

not.
base	would	be	accommodated	to	a	greater	degree	than	the	interests	of	outsiders.
In	particular,	if	the	dynamic	acts	only	where	there	is	broad	agreement	between
individual	extrapolated	volitions	(as	in	Yudkowsky’s	original	proposal),	there

would	seem	to	be	a	significant	risk	of	an	ungenerous	blocking	vote	that	could
prevent,	for	instance,	the	welfare	of	nonhuman	animals	or	digital	minds	from
being	protected.	The	result	might	potentially	be	morally	rotten.
17
One	motivation	for	the	CEV	proposal	was	to	avoid	creating	a	motive	for
humans	to	fight	over	the	creation	of	the	first	superintelligent	AI.	Although	the
CEV	proposal	scores	better	on	this	desideratum	than	many	alternatives,	it	does
not	entirely	eliminate	motives	for	conflict.	A	selfish	individual,	group,	or	nation
might	seek	to	enlarge	its	slice	of	the	future	by	keeping	others	out	of	the
extrapolation	base.
A	power	grab	of	this	sort	might	be	rationalized	in	various	ways.	It	might	be
argued,	for	instance,	that	the	sponsor	who	funds	the	development	of	the	AI
deserves	to	own	the	outcome.	This	moral	claim	is	probably	false.	It	could	be
objected,	for	example,	that	the	project	that	launches	the	first	successful	seed	AI
imposes	a	vast	risk	externality	on	the	rest	of	humanity,	which	therefore	is

entitled	to	compensation.	The	amount	of	compensation	owed	is	so	great	that	it
can	only	take	the	form	of	giving	everybody	a	stake	in	the	upside	if	things	turn
out	well.
18
Another	argument	that	might	be	used	to	rationalize	a	power	grab	is	that	large
segments	of	humanity	have	base	or	evil	preferences	and	that	including	them	in
the	extrapolation	base	would	risk	turning	humanity’s	future	into	a	dystopia.	It	is
difficult	to	know	the	share	of	good	and	bad	in	the	average	person’s	heart.	It	is

out	well.
18
also	difficult	to	know	how	much	this	balance	varies	between	different	groups,
social	strata,	cultures,	or	nations.	Whether	one	is	optimistic	or	pessimistic	about
human	nature,	one	may	prefer	not	to	wager	humanity’s	cosmic	endowment	on
the	speculation	that,	for	a	sufficient	majority	of	the	seven	billion	people	currently
alive,	their	better	angels	would	prevail	in	their	extrapolated	volitions.	Of	course,
omitting	a	certain	set	of	people	from	the	extrapolation	base	does	not	guarantee

that	light	would	triumph;	and	it	might	well	be	that	the	souls	that	would	soonest
exclude	others	or	grab	power	for	themselves	tend	rather	to	contain	unusually
large	amounts	of	darkness.
Yet	another	reason	for	fighting	over	the	initial	dynamic	is	that	one	might
believe	that	somebody	else’s	AI	will	not	work	as	advertised,	even	if	the	AI	is
billed	as	a	way	to	implement	humanity’s	CEV.	If	different	groups	have	different
beliefs	about	which	implementation	is	most	likely	to	succeed,	they	might	fight	to

prevent	the	others	from	launching.	It	would	be	better	in	such	situations	if	the

competing	projects	could	settle	their	epistemic	differences	by	some	method	that
more	reliably	ascertains	who	is	right	than	the	method	of	armed	conflict.
19
	
The	CEV	proposal	is	not	the	only	possible	form	of	indirect	normativity.	For
example,	instead	of	implementing	humanity’s	coherent	extrapolated	volition,	one
could	try	to	build	an	AI	with	the	goal	of	doing	what	is	morally	right,	relying	on
the	AI’s	superior	cognitive	capacities	to	figure	out	just	which	actions	fit	that
description.	We	can	call	this	proposal	“moral	rightness”	(MR).	The	idea	is	that
we	humans	have	an	imperfect	understanding	of	what	is	right	and	wrong,	and
perhaps	an	even	poorer	understanding	of	how	the	concept	of	moral	rightness	is
to	be	philosophically	analyzed:	but	a	superintelligence	could	understand	these
things	better.
20
What	if	we	are	not	sure	whether	moral	realism	is	true?	We	could	still	attempt
the	MR	proposal.	We	should	just	have	to	make	sure	to	specify	what	the	AI

20
should	do	in	the	eventuality	that	its	presupposition	of	moral	realism	is	false.	For
example,	
we	could	stipulate	that	if	the	AI	estimates	with	a	sufficient	probability
that	there	are	no	suitable	non-relative	truths	about	moral	rightness,	then	it	should
revert	to	implementing	coherent	extrapolated	volition	instead,	or	simply	shut
itself	down.
21
MR	appears	to	have	several	advantages	over	CEV.	MR	would	do	away	with
various	free	parameters	in	CEV,	such	as	the	degree	of	coherence	among

example,	
21
extrapolated	volitions	that	is	required	for	the	AI	to	act	on	the	result,	the	ease	with
which	a	majority	can	overrule	dissenting	minorities,	and	the	nature	of	the	social
environment	within	which	our	extrapolated	selves	are	to	be	supposed	to	have
“grown	up	farther	together.”	It	would	seem	to	eliminate	the	possibility	of	a	moral
failure	resulting	from	the	use	of	an	extrapolation	base	that	is	too	narrow	or	too
wide.	Furthermore,	MR	would	orient	the	AI	toward	morally	right	action	even	if

our	coherent	extrapolated	volitions	happen	to	wish	for	the	AI	to	take	actions	that
are	morally	odious.	As	noted	earlier,	this	seems	a	live	possibility	with	the	CEV
proposal.	Moral	goodness	might	be	more	like	a	precious	metal	than	an	abundant
element	in	human	nature,	and	even	after	the	ore	has	been	processed	and	refined
in	accordance	with	the	prescriptions	of	the	CEV	proposal,	who	knows	whether
the	principal	outcome	will	be	shining	virtue,	indifferent	slag,	or	toxic	sludge?

MR	would	also	appear	to	have	some	disadvantages.	It	relies	on	the	notion	of
“morally	right,”	a	notoriously	difficult	concept,	one	with	which	philosophers
have	grappled	since	antiquity	without	yet	attaining	consensus	as	to	its	analysis.
Picking	an	erroneous	explication	of	“moral	rightness”	could	result	in	outcomes
that	would	be	morally	very	wrong.	This	difficulty	of	defining	“moral	rightness”
might	seem	to	count	heavily	against	the	MR	proposal.	However,	it	is	not	clear
that	the	MR	proposal	is	really	at	a	material	disadvantage	in	this	regard.	The	CEV
proposal,	too,	uses	terms	and	concepts	that	are	difficult	to	explicate	(such	as
“knowledge,”	“being	more	the	people	we	wished	we	were,”	“grown	up	farther
together,”	among	others).
22
	Even	if	these	concepts	are	marginally	less	opaque
than	“moral	rightness,”	they	are	still	miles	removed	from	anything	that
programmers	can	currently	express	in	code.
23
	The	path	to	endowing	an	AI	with
any	of	these	concepts	might	involve	giving	it	general	linguistic	ability

22
23
(comparable,	at	least,	to	that	of	a	normal	human	adult).	Such	a	general	ability	to
understand	natural	language	could	then	be	used	to	understand	what	is	meant	by
“morally	right.”	If	the	AI	could	grasp	the	meaning,	it	could	search	for	actions
that	fit.	As	the	AI	develops	superintelligence,	it	could	then	make	progress	on	two
fronts:	on	the	philosophical	problem	of	understanding	what	moral	rightness	is,
and	on	the	practical	problem	of	applying	this	understanding	to	evaluate
particular	actions.
24
	While	this	would	not	be	easy,	it	is	not	clear	that	it	would	be

24
any	
more
	difficult	than	extrapolating	humanity’s	coherent	extrapolated
volition.
25
A	more	fundamental	issue	with	MR	is	that	even	if	can	be	implemented,	it
might	not	give	us	what	we	want	or	what	we	would	choose	if	we	were	brighter
and	better	informed.	This	is	of	course	the	essential	feature	of	MR,	not	an
accidental	bug.	However,	it	might	be	a	feature	that	would	be	extremely	harmful
to	us.
26
One	might	try	to	preserve	the	basic	idea	of	the	MR	model	while	reducing	its
demandingness	by	focusing	on	
moral	permissibility

volition.
25
to	us.
26
:	the	idea	being	that	we	
could
let	the	AI	pursue	humanity’s	CEV	so	long	as	it	did	not	act	in	ways	that	are
morally	impermissible.	For	example,	one	might	formulate	the	following	goal	for
the	AI:
Among	the	actions	that	are	morally	permissible	for	the	AI,	take	one	that
humanity’s	CEV	would	prefer.	However,	if	some	part	of	this	instruction	has
no	well-specified	meaning,	or	if	we	are	radically	confused	about	its

meaning,	or	if	moral	realism	is	false,	or	if	we	acted	morally	impermissibly
in	creating	an	AI	with	this	goal,	then	undergo	a	controlled	shutdown.
27
Follow	the	intended	meaning	of	this	instruction.
	
One	might	still	worry	that	this	moral	permissibility	model	(MP)	represents	an
unpalatably	high	degree	of	respect	for	the	requirements	of	morality.	How	big	a
sacrifice	it	would	entail	depends	on	which	ethical	theory	is	true.
28
	If	ethics	is
satisficing
,	in	the	sense	that	it	counts	as	morally	permissible	any	action	that
conforms	to	a	few	basic	moral	constraints,	then	MP	may	leave	ample	room	for
our	coherent	extrapolated	volition	to	influence	the	AI’s	actions.	However,	if
ethics	is	
maximizing
—for	example,	if	the	only	morally	permissible	actions	are
those	that	have	the	morally	best	consequences—then	MP	may	leave	little	or	no
room	for	our	own	preferences	to	shape	the	outcome.
To	illustrate	this	concern,	let	us	return	for	a	moment	to	the	example	of

ethics	is	
hedonistic	consequentialism.	Suppose	that	this	ethical	theory	is	true,	and	that	the
AI	knows	it	to	be	so.	For	present	purposes,	we	can	define	hedonistic
consequentialism	as	the	claim	that	an	action	is	morally	right	(and	morally
permissible)	if	and	only	if,	among	all	feasible	actions,	no	other	action	would
produce	a	greater	balance	of	pleasure	over	suffering.	The	AI,	following	MP,
might	maximize	the	surfeit	of	pleasure	by	converting	the	accessible	universe	into

hedonium,	a	process	that	may	involve	building	computronium	and	using	it	to
perform	computations	that	instantiate	pleasurable	experiences.	Since	simulating
any	existing	human	brain	is	not	the	most	efficient	way	of	producing	pleasure,	a
likely	consequence	is	that	we	all	die.
By	enacting	either	the	MR	or	the	MP	proposal,	we	would	thus	risk	sacrificing
our	lives	for	a	greater	good.	This	would	be	a	bigger	sacrifice	than	one	might
think,	because	what	we	stand	to	lose	is	not	merely	the	chance	to	live	out	a

normal	human	life	but	the	opportunity	to	enjoy	the	far	longer	and	richer	lives
that	a	friendly	superintelligence	could	bestow.
The	sacrifice	looks	even	less	appealing	when	we	reflect	that	the
superintelligence	could	realize	a	nearly-as-great	good	(in	fractional	terms)	while
sacrificing	much	less	of	our	own	potential	well-being.	Suppose	that	we	agreed	to
allow	
almost
	the	entire	accessible	universe	to	be	converted	into	hedonium—

everything	except	a	small	preserve,	say	the	Milky	Way,	which	would	be	set	aside
to	accommodate	our	own	needs.	Then	there	would	still	be	a	hundred	billion
galaxies	devoted	to	the	maximization	of	pleasure.	But	we	would	have	one	galaxy
within	which	to	create	wonderful	civilizations	that	could	last	for	billions	of	years
and	in	which	humans	and	nonhuman	animals	could	survive	and	thrive,	and	have
the	opportunity	to	develop	into	beatific	posthuman	spirits.
29
If	one	prefers	this	latter	option	(as	I	would	be	inclined	to	do)	it	implies	that
one	does	not	have	an	unconditional	lexically	dominant	preference	for	acting
morally	permissibly.	But	it	is	consistent	with	placing	great	weight	on	morality.
Even	from	a	purely	moral	point	of	view,	it	might	be	better	to	
advocate
	some
proposal	that	is	less	morally	ambitious	than	MR	or	MP.	If	the	morally	best	has
no	chance	of	being	implemented—perhaps	because	of	its	frowning
demandingness—it	might	be	morally	better	to	promote	some	other	proposal,	one

advocate
	some
that	would	be	near-ideal	and	whose	chances	of	being	implemented	could	be
significantly	increased	by	our	promoting	it.
30
	
We	might	feel	unsure	whether	to	go	for	CEV,	MR,	MP,	or	something	else.	Could
we	punt	on	this	higher-level	decision	as	well,	offloading	even	more	cognitive
work	onto	the	AI?	Where	is	the	limit	to	our	possible	laziness?
Consider,	for	example,	the	following	“reasons-based”	goal:
Do	whatever	we	would	have	had	most	reason	to	ask	the	AI	to	do.
	
This	goal	might	boil	down	to	extrapolated	volition	or	morality	or	something	else,

30
	
but	it	would	seem	to	spare	us	the	effort	and	risk	involved	in	trying	to	figure	out
for	ourselves	which	of	these	more	specific	objectives	we	would	have	most
reason	to	select.
Some	of	the	problems	with	the	morality-based	goals,	however,	also	apply
here.	First,	we	might	fear	that	this	reasons-based	goal	would	leave	too	little	room

for	our	own	desires.	Some	philosophers	maintain	that	a	person	always	has	most
reason	to	do	what	it	would	be	morally	best	for	her	to	do.	If	those	philosophers
are	right,	then	the	reason-based	goal	collapses	into	MR—with	the	concomitant
risk	that	a	superintelligence	implementing	such	a	dynamic	would	kill	everyone
within	reach.	Second,	as	with	all	proposals	couched	in	technical	language,	there
is	a	possibility	that	we	might	have	misunderstood	the	meaning	of	our	own
assertions.	We	saw	that,	in	the	case	of	the	morality-based	goals,	asking	the	AI	to
do	what	is	right	may	lead	to	unforeseen	and	unwanted	consequences	such	that,
had	we	anticipated	them,	we	would	not	have	implemented	the	goal	in	question.
The	same	applies	to	asking	the	AI	to	do	what	we	have	most	reason	to	do.
What	if	we	try	to	avoid	these	difficulties	by	couching	a	goal	in	emphatically
nontechnical	language—such	as	in	terms	of	“niceness”:
31
Take	the	nicest	action;	or,	if	no	action	is	nicest,	then	take	an	action	that	is	at
least	super-duper	nice.

31
	
How	could	there	be	anything	objectionable	about	building	a	
nice
	AI?	But	we
must	ask	what	precisely	is	meant	by	this	expression.	The	lexicon	lists	various
meanings	of	“nice”	that	are	clearly	not	intended	to	be	used	here:	we	do	not
intend	that	the	AI	
should	be	
courteous	and	polite
	nor	
overdelicate	or	fastidious
.
If	we	can	count	on	the	AI	recognizing	the	intended	interpretation	of	“niceness”
and	being	motivated	to	pursue	niceness	in	just	that	sense,	then	this	goal	would

nice
	AI?	But	we
should	be	
	nor	
.
seem	to	amount	to	a	command	to	do	what	the	programmers	meant	for	the	AI	to
do.
32
	An	injunction	to	similar	effect	was	included	in	the	formulation	of	CEV
(“…	interpreted	as	we	wish	that	interpreted”)	and	in	the	moral-permissibility
criterion	as	rendered	earlier	(“…	follow	the	intended	meaning	of	this
instruction”).	By	affixing	such	a	“Do	What	I	Mean”	clause	we	may	indicate	that
the	other	words	in	the	goal	description	should	be	construed	charitably	rather	than

do.
32
literally.	But	saying	that	the	AI	should	be	“nice”	adds	almost	nothing:	the	real
work	is	done	by	the	“Do	What	I	Mean”	instruction.	If	we	knew	how	to	code	“Do
What	I	Mean”	in	a	general	and	powerful	way,	we	might	as	well	use	that	as	a
standalone	goal.
How	might	one	implement	such	a	“Do	What	I	Mean”	dynamic?	That	is,	how
might	we	create	an	AI	motivated	to	charitably	interpret	our	wishes	and	unspoken

intentions	and	to	act	accordingly?	One	initial	step	could	be	to	try	to	get	clearer
about	what	we	mean	by	“Do	What	I	Mean.”	It	might	help	if	we	could	explicate
this	in	more	behavioristic	terms,	for	example	in	terms	of	revealed	preferences	in
various	hypothetical	situations—such	as	situations	in	which	we	had	more	time	to
consider	the	options,	in	which	we	were	smarter,	in	which	we	knew	more	of	the
relevant	facts,	and	in	which	in	various	other	ways	conditions	would	be	more
favorable	for	us	accurately	manifesting	in	concrete	choices	what	we	mean	when
we	say	that	we	want	an	AI	that	is	friendly,	beneficial,	nice	…
Here,	of	course,	we	come	full	circle.	We	have	returned	to	the	indirect
normativity	approach	with	which	we	started—the	CEV	proposal,	which,	in
essence,	expunges	all	concrete	content	from	the	value	specification,	leaving	only
an	abstract	value	defined	in	purely	procedural	terms:	to	do	that	which	we	would
have	wished	for	the	AI	to	do	in	suitably	idealized	circumstances.	By	means	of

such	indirect	normativity,	we	could	hope	to	offload	to	the	AI	much	of	the
cognitive	work	that	we	ourselves	would	be	trying	to	perform	if	we	attempted	to
articulate	a	more	concrete	description	of	what	values	the	AI	is	to	pursue.	In
seeking	to	take	full	advantage	of	the	AI’s	epistemic	superiority,	CEV	can	thus	be
seen	as	an	application	of	the	principle	of	epistemic	deference.
	
So	far	we	have	considered	different	options	for	what	content	to	put	into	the	goal
system.	But	an	AI’s	behavior	will	also	be	influenced	by	other	design	choices.	In

particular,	it	can	make	a	critical	difference	which	decision	theory	and	which
epistemology	it	uses.	Another	important	question	is	whether	the	AI’s	plans	will
be	subject	to	human	review	before	being	put	into	action.
Table	13
	summarizes	these	design	choices.	A	project	that	aims	to	build	a
superintelligence	ought	to	be	able	to	explain	what	choices	it	has	made	regarding
each	of	these	components,	and	to	justify	why	those	choices	were	made.
33

Table	13	Component	list
What	objective	should	the	AI	pursue?	How	should	a	description
of	this	objective	be	interpreted?	Should	the	objective	include
giving	special	rewards	to	those	who	contributed	to	the	project’s
success?
Decision
theory
Should	the	AI	use	causal	decision	theory,	evidential	decision
theory,	updateless	decision	theory,	or	something	else?
What	should	the	AI’s	prior	probability	function	be,	and	what
other	explicit	or	implicit	assumptions	about	the	world	should	it
make?	What	theory	of	anthropics	should	it	use?
Should	the	AI’s	plans	be	subjected	to	human	review	before	being
put	into	effect?	If	so,	what	is	the	protocol	for	that	review
process?
	
We	have	already	discussed	how	indirect	normativity	might	be	used	in	specifying
the	values	that	the	AI	is	to	pursue.	We	discussed	some	options,	such	as	morality-
based	models	and	coherent	extrapolated	volition.	Each	such	option	creates

process?
	
further	choices	that	need	to	be	made.	For	instance,	the	CEV	approach	comes	in
many	varieties,	depending	on	who	is	included	in	the	extrapolation	base,	the
structure	of	the	extrapolation,	and	so	forth.	Other	forms	of	motivation	selection
methods	might	call	for	different	types	of	goal	content.	For	example,	an	oracle
might	be	built	to	place	a	value	on	giving	accurate	answers.	An	oracle	constructed
with	domesticity	motivation	might	also	have	goal	content	that	disvalues	the

excessive	use	of	resources	in	producing	its	answers.
Another	design	choice	is	whether	to	include	special	provisions	in	the	goal

content	to	reward	individuals	who	contribute	to	the	successful	realization	of	the
AI,	for	example	by	giving	them	extra	resources	or	influence	over	the	AI’s
behavior.	We	can	term	any	such	provisions	“incentive	wrapping.”	Incentive
wrapping	could	be	seen	as	a	way	to	increase	the	likelihood	that	the	project	will
be	successful,	at	the	cost	of	compromising	to	some	extent	the	goal	that	the
project	set	out	to	achieve.
For	example,	if	the	project’s	goal	is	to	create	a	dynamic	that	implements
humanity’s	coherent	extrapolated	volition,	then	an	incentive	wrapping	scheme
might	specify	that	certain	individuals’	volitions	should	be	given	extra	weight	in
the	extrapolation.	If	such	a	project	is	successful,	the	result	is	not	necessarily	the
implementation	of	humanity’s	coherent	extrapolated	volition.	Instead,	some
approximation	to	this	goal	might	be	achieved.
34
Since	incentive	wrapping	would	be	a	piece	of	goal	content	that	would	be
interpreted	and	pursued	by	a	superintelligence,	it	could	take	advantage	of

34
indirect	normativity	to	specify	subtle	and	complicated	provisions	that	would	be
difficult	for	a	human	manager	to	implement.	For	example,	instead	of	rewarding
programmers	according	to	some	crude	but	easily	accessible	metric,	such	as	how
many	hours	they	worked	or	how	many	bugs	they	corrected,	the	incentive
wrapping	could	specify	that	
programmers	“are	to	be	rewarded	in	proportion	to
how	much	their	contributions	increased	some	reasonable	
ex	ante
	probability	of
the	project	being	successfully	completed	in	the	way	the	sponsors	intended.”

ex	ante
Further,	there	would	be	no	reason	to	limit	the	incentive	wrapping	to	project	staff.
It	could	instead	specify	that	
every
	person	should	be	rewarded	according	to	their
just	deserts.	Credit	allocation	is	a	difficult	problem,	but	a	superintelligence	could
be	expected	to	do	a	reasonable	job	of	approximating	the	criteria	specified,
explicitly	or	implicitly,	by	the	incentive	wrapping.
It	is	conceivable	that	the	superintelligence	might	even	find	some	way	of
rewarding	individuals	who	have	died	prior	to	the	superintelligence’s	creation.
35

every
35
The	incentive	wrapping	could	then	be	extended	to	embrace	at	least	some	of	the
deceased,	potentially	including	individuals	who	died	before	the	project	was
conceived,	or	even	antedating	the	first	enunciation	of	the	concept	of	incentive
wrapping.	Although	the	institution	of	such	a	retroactive	policy	would	not
causally	incentivize	those	people	who	are	already	resting	in	their	graves	as	these
words	are	being	put	to	the	page,	it	might	be	favored	for	moral	reasons—though
it	could	be	argued	that	insofar	as	fairness	is	a	goal,	it	should	be	included	as	part

of	the	target	specification	proper	rather	than	in	the	surrounding	incentive

wrapping.
We	cannot	here	delve	into	all	the	ethical	and	strategic	issues	associated	with
incentive	wrapping.	A	project’s	position	on	these	issues,	however,	would	be	an
important	aspect	of	its	fundamental	design	concept.
	
Another	important	design	choice	is	which	decision	theory	the	AI	should	be	built
to	use.	This	might	affect	how	the	AI	behaves	in	certain	strategically	fateful
situations.	It	might	determine,	for	instance,	whether	the	AI	is	open	to	trade	with,
or	extortion	by,	other	superintelligent	civilizations	whose	existence	it
hypothesizes.	The	particulars	of	the	decision	theory	could	also	matter	in
predicaments	involving	finite	probabilities	of	infinite	payoffs	(“Pascalian
wagers”)	or	extremely	small	probabilities	of	extremely	large	finite	payoffs
(“Pascalian	muggings”)	or	in	contexts	where	the	AI	is	facing	fundamental
normative	uncertainty	or	where	there	are	multiple	instantiations	of	the	same
agent	program.
36
The	options	on	the	table	include	causal	decision	theory	(in	a	variety	of

36
flavors)	and	evidential	decision	theory,	along	with	newer	candidates	such	as
“timeless	decision	theory”	and	“updateless	decision	theory,”	which	are	still
under	development.
37
	It	may	prove	difficult	to	identify	and	articulate	the	correct
decision	theory,	and	to	have	justified	confidence	that	we	have	got	it	right.
Although	the	prospects	for	directly	specifying	an	AI’s	decision	theory	are
perhaps	more	hopeful	than	those	of	directly	specifying	its	final	values,	we	are
still	confronted	with	a	substantial	risk	of	error.	Many	of	the	complications	that

37
might	break	the	currently	most	popular	decision	theories	were	discovered	only
recently,	suggesting	that	there	might	exist	further	problems	that	have	not	yet
come	into	sight.	The	result	of	giving	the	AI	a	flawed	decision	theory	might	be
disastrous,	possibly	amounting	to	an	existential	catastrophe.
In	view	of	these	difficulties,	one	might	consider	an	indirect	approach	to
specifying	the	decision	theory	that	the	AI	should	use.	Exactly	how	to	do	this	is
not	yet	clear.	We	might	want	the	AI	to	use	“that	decision	theory	
D
	which	we

D
	which	we
would	have	wanted	it	to	use	had	we	thought	long	and	hard	about	the	matter.”
However,	the	AI	would	need	to	be	able	to	make	decisions	before	learning	what

D
	is.	It	would	thus	need	some	effective	interim	decision	theory	
D’
	that	would
govern	its	search	for	
D
.	One	might	try	to	define	
D’
	to	be	some	sort	of
superposition	of	the	AI’s	current	hypotheses	about	
D
	(weighed	by	their
probabilities),	though	there	are	unsolved	technical	problems	with	how	to	do	this
in	a	fully	general	way.
38
	There	is	also	cause	for	concern	that	the	AI	might	make
irreversibly	bad	decisions	(such	as	rewriting	itself	to	henceforth	run	on	some
flawed	decision	theory)	during	the	learning	phase,	before	the	AI	has	had	the
opportunity	to	determine	which	particular	decision	theory	is	correct.	To	reduce
the	risk	of	derailment	during	this	period	of	vulnerability	we	might	instead	try	to
endow	the	seed	AI	with	some	form	of	
restricted	rationality
:	a	deliberately
simplified	but	hopefully	dependable	decision	theory	that	staunchly	ignores
esoteric	considerations,	even	ones	we	think	may	ultimately	be	legitimate,	and

that	is	designed	to	replace	itself	with	a	more	sophisticated	(indirectly	specified)
decision	theory	once	certain	conditions	are	met.
39
	It	is	an	open	research	question
whether	and	how	this	could	be	made	to	work.
	
A	project	will	also	need	to	make	a	fundamental	design	choice	in	selecting	the
AI’s	epistemology,	specifying	the	principles	and	criteria	whereby	empirical
hypotheses	are	to	be	evaluated.	Within	a	Bayesian	framework,	we	can	think	of
the	epistemology	as	a	prior	probability	function—the	AI’s	implicit	assignment	of

39
	
probabilities	to	possible	worlds	before	it	has	taken	any	perceptual	evidence	into
account.	In	other	frameworks,	the	epistemology	might	take	a	different	form;	but
in	any	case	some	inductive	learning	rule	is	necessary	if	the	AI	is	to	generalize
from	past	observations	and	make	predictions	about	the	future.
40
	As	with	the	goal
content	and	the	decision	theory,	however,	there	is	a	risk	that	our	epistemology
specification	could	miss	the	mark.
One	might	think	that	there	is	a	limit	to	how	much	damage	could	arise	from	an

40
incorrectly	specified	epistemology.	If	the	epistemology	is	
too
	dysfunctional,	then
the	AI	could	not	be	very	intelligent	and	it	could	not	pose	the	kind	of	risk
discussed	in	this	book.	But	the	concern	is	that	we	may	specify	an	epistemology
that	is	sufficiently	sound	to	make	the	AI	instrumentally	effective	in	most
situations,	yet	which	has	some	flaw	that	leads	the	AI	astray	on	some	matter	of
crucial	importance.	Such	an	AI	might	be	akin	to	a	quick-witted	person	whose

worldview	is	predicated	on	a	false	dogma,	held	to	with	absolute	conviction,	who
consequently	“tilts	at	windmills”	and	gives	his	all	in	pursuit	of	fantastical	or
harmful	objectives.
Certain	kinds	of	subtle	difference	in	an	AI’s	prior	could	turn	out	to	make	a
drastic	difference	to	how	it	behaves.	For	example,	an	AI	might	be	given	a	prior
that	assigns	zero	probability	to	the	universe	being	infinite.	No	matter	how	much
astronomical	evidence	it	accrues	to	the	contrary,	such	an	AI	would	stubbornly
reject	any	cosmological	theory	that	implied	an	infinite	universe;	and	it	might
make	foolish	choices	as	a	result.
41
	Or	an	AI	might	be	given	a	prior	that	assigns	a
zero	probability	to	the	universe	not	being	Turing-computable	(this	is	in	fact	a
common	feature	of	many	of	the	priors	discussed	in	the	literature,	including	the
Kolmogorov	complexity	prior	mentioned	in	
Chapter	1
),	again	with	poorly
understood	consequences	if	the	embedded	assumption—known	as	the	“Church–

41
Chapter	1
Turing	thesis”—should	turn	out	to	be	false.	An	AI	could	also	end	up	with	a	prior
that	makes	strong	metaphysical	commitments	of	one	sort	or	another,	for	instance
by	ruling	out	a	priori	the	possibility	that	any	strong	form	of	mind–body	dualism
could	be	true	or	the	possibility	that	there	are	irreducible	moral	facts.	If	any	of
those	commitments	is	mistaken,	the	AI	might	seek	to	realize	its	final	goals	in
ways	that	we	would	regard	as	perverse	instantiations.	Yet	there	is	no	obvious

reason	why	such	an	AI,	despite	being	fundamentally	wrong	about	one	important
matter,	could	not	be	sufficiently	instrumentally	effective	to	secure	a	decisive
strategic	advantage.	(Anthropics,	the	study	of	how	to	make	inferences	from
indexical	information	in	the	presence	of	observation	selection	effects,	is	another
area	where	the	choice	of	epistemic	axioms	could	prove	pivotal.
42
)
We	might	reasonably	doubt	our	ability	to	resolve	all	foundational	issues	in
epistemology	in	time	for	the	construction	of	the	first	seed	AI.	We	may,	therefore,

42
)
consider	taking	an	indirect	approach	to	specifying	the	AI’s	epistemology.	This
would	raise	many	of	the	same	issues	as	taking	an	indirect	approach	to	specifying
its	decision	theory.	In	the	case	of	epistemology,	however,	there	may	be	greater
hope	of	benign	convergence,	with	any	of	a	wide	class	of	epistemologies
providing	an	adequate	foundation	for	safe	and	effective	AI	and	ultimately
yielding	similar	doxastic	results.	The	reason	for	this	is	that	sufficiently	abundant
empirical	evidence	and	analysis	would	tend	to	wash	out	any	moderate

differences	in	prior	expectations.
43
A	good	aim	would	be	to	endow	the	AI	with	fundamental	epistemological
principles	that	match	those	governing	our	own	thinking.	Any	AI	diverging	from

this	ideal	is	an	AI	that	we	would	judge	to	be	reasoning	incorrectly	if	we
consistently	applied	our	own	standards.	Of	course,	this	applies	only	to	our
fundamental
	epistemological	principles.	Non-fundamental	principles	should	be
continuously	created	and	revised	by	the	seed	AI	itself	as	it	develops	its
understanding	of	the	world.	The	point	of	superintelligence	is	not	to	pander	to
human	preconceptions	but	to	make	mincemeat	out	of	our	ignorance	and	folly.
	
The	final	item	in	our	list	of	design	choices	is	
ratification
.	Should	the	AI’s	plans
be	subjected	to	human	review	before	being	put	into	effect?	For	an	oracle,	this
question	is	implicitly	answered	in	the	affirmative.	The	oracle	outputs
information;	human	reviewers	choose	whether	and	how	to	act	upon	it.	For
genies,	sovereigns,	and	tool-AIs,	however,	the	question	of	whether	to	use	some
form	of	ratification	remains	open.
To	illustrate	how	ratification	might	work,	consider	an	AI	intended	to	function

as	a	sovereign	implementing	humanity’s	CEV.	Instead	of	launching	this	AI
directly,	imagine	that	we	first	built	an	oracle	AI	for	the	sole	purpose	of
answering	questions	about	what	the	sovereign	AI	would	do.	As	earlier	chapters
revealed,	there	are	risks	in	creating	a	superintelligent	oracle	(such	as	risks	of
mind	crime	or	infrastructure	profusion).	But	for	purposes	of	this	example	let	us
assume	that	the	oracle	AI	has	been	successfully	implemented	in	a	way	that
avoided	these	pitfalls.

We	thus	have	an	oracle	AI	that	offers	us	its	best	guesses	about	the
consequences	of	running	some	piece	of	code	intended	to	implement	humanity’s
CEV.	The	oracle	may	not	be	able	to	predict	in	detail	what	would	happen,	but	its
predictions	are	likely	to	be	better	than	our	own.	(If	it	were	impossible	even	for	a
superintelligence	to	predict	
anything
	about	the	code	would	do,	we	would	be
crazy	to	run	it.)	So	the	oracle	ponders	for	a	while	and	then	presents	its	forecast.

anything
To	make	the	answer	intelligible,	the	oracle	may	offer	the	operator	a	range	of
tools	with	which	to	explore	various	features	of	the	predicted	outcome.	The
oracle	could	show	pictures	of	what	the	future	looks	like	and	provide	statistics
about	the	number	of	sentient	beings	that	will	exist	at	different	times,	along	with
average,	peak,	and	lowest	levels	of	well-being.	It	could	offer	intimate
biographies	of	several	randomly	selected	individuals	(perhaps	imaginary	people

selected	to	be	probably	representative).	It	could	highlight	aspects	of	the	future
that	the	operator	might	not	have	thought	of	inquiring	about	but	which	would	be
regarded	as	pertinent	once	pointed	out.
Being	able	to	preview	the	outcome	in	this	manner	has	obvious	advantages.
The	preview	could	reveal	the	consequences	of	an	error	in	a	planned	sovereign’s
design	specifications	or	source	code.	If	the	crystal	ball	shows	a	ruined	future,	we
could	scrap	the	code	for	the	planned	sovereign	AI	and	try	something	else.	A
strong	case	could	be	made	that	we	should	familiarize	ourselves	with	the	concrete
ramifications	of	an	option	before	committing	to	it,	especially	when	the	entire
future	of	the	race	is	on	the	line.
What	is	perhaps	less	obvious	is	that	ratification	also	has	potentially	significant
disadvantages.	The	irenic	quality	of	CEV	might	be	undermined	if	opposing
factions,	instead	of	submitting	to	the	arbitration	of	superior	wisdom	in	confident

expectation	of	being	vindicated,	could	see	in	advance	what	the	verdict	would	be.
A	proponent	of	the	morality-based	approach	might	worry	that	the	sponsor’s
resolve	would	collapse	if	all	the	sacrifices	required	by	the	morally	optimal	were
to	be	revealed.	And	we	might	all	have	reason	to	prefer	a	future	that	holds	some
surprises,	some	dissonance,	some	wildness,	some	opportunities	for	self-
overcoming—a	future	whose	contours	are	not	too	snugly	tailored	to	present

preconceptions	but	provide	some	give	for	dramatic	movement	and	unplanned
growth.	We	might	be	less	likely	to	take	such	an	expansive	view	if	we	could
cherry-pick	every	detail	of	the	future,	sending	back	to	the	drawing	board	any
draft	that	does	not	fully	conform	to	our	fancy	at	that	moment.
The	issue	of	sponsor	ratification	is	therefore	less	clear-cut	than	it	might
initially	seem.	Nevertheless,	on	balance	it	would	seem	prudent	to	take	advantage
of	an	opportunity	to	preview,	if	that	functionality	is	available.	But	rather	than
letting

letting	
the	reviewer	fine-tune	every	aspect	of	the	outcome,	we	might	give	her	a
simple	veto	which	could	be	exercised	only	a	few	times	before	the	entire	project
would	be	aborted.
44
	
The	main	purpose	of	ratification	would	be	to	reduce	the	probability	of
catastrophic	error.	In	general,	it	seems	wise	to	aim	at	minimizing	the	risk	of

catastrophic	error	rather	than	at	maximizing	the	chance	of	every	detail	being
fully	optimized.	There	are	two	reasons	for	this.	First,	humanity’s	cosmic
endowment	is	astronomically	large—there	is	plenty	to	go	around	even	if	our
process	involves	some	waste	or	accepts	some	unnecessary	constraints.	Second,
there	is	a	hope	that	if	we	but	get	the	initial	conditions	for	the	intelligence
explosion	approximately	right,	then	the	resulting	superintelligence	may
eventually	home	in	on,	and	precisely	hit,	our	ultimate	objectives.	The	important
thing	is	to	land	in	the	right	attractor	basin.
With	regard	to	epistemology,	it	is	plausible	that	a	wide	range	of	priors	will
ultimately	converge	to	very	similar	posteriors	(when	computed	by	a
superintelligence	and	conditionalized	on	a	realistic	amount	of	data).	We
therefore	need	not	worry	about	getting	the	epistemology	
exactly
	right.	We	must
just	avoid	giving	the	AI	a	prior	that	is	so	extreme	as	to	render	the	AI	incapable

exactly
of	learning	vital	truths	even	with	the	benefit	of	copious	experience	and
analysis.
45
With	regard	to	decision	theory,	the	risk	of	irrecoverable	error	seems	larger.	We
might	still	hope	to	directly	specify	a	decision	theory	that	is	good	enough.	A
superintelligent	AI	could	switch	to	a	new	decision	theory	at	any	time;	however,
if	it	starts	out	with	a	sufficiently	wrong	decision	theory	it	may	not	see	the	reason
to	switch.	Even	if	an	agent	comes	to	see	the	benefits	of	having	a	different

analysis.
45
decision	theory,	the	realization	might	come	too	late.	For	example,	an	agent
designed	to	refuse	blackmail	might	enjoy	the	benefit	of	deterring	would-be
extortionists.	For	this	reason,	blackmailable	agents	might	do	well	to	proactively
adopt	a	non-exploitable	decision	theory.	Yet	once	a	blackmailable	agent	receives
the	threat	and	regards	it	as	credible,	the	damage	is	done.
Given	an	adequate	epistemology	and	decision	theory,	we	could	try	to	design
the	system	to	implement	CEV	or	some	other	indirectly	specified	goal	content.

Again	there	is	hope	of	convergence:	that	different	ways	of	implementing	a	CEV-
like	dynamic	would	lead	to	the	same	utopian	outcome.	Short	of	such
convergence,	we	may	still	hope	that	many	of	the	different	possible	outcomes	are
good	enough	to	count	as	existential	success.
It	is	not	necessary	for	us	to	create	a	highly	optimized	design.	Rather,	our	focus
should	be	on	creating	a	highly	reliable	design,	one	that	can	be	trusted	to	retain
enough	sanity	to	recognize	its	own	failings.	An	imperfect	superintelligence,

whose	fundamentals	are	sound,	would	gradually	repair	itself;	and	having	done

so,	it	would	exert	as	much	beneficial	optimization	power	on	the	world	as	if	it	had
been	perfect	from	the	outset.

CHAPTER	14
The	strategic	picture
	
It	is	now	time	to	consider	the	challenge	of	superintelligence	in	a	broader
context.	We	would	like	to	orient	ourselves	in	the	strategic	landscape
sufficiently	to	know	at	least	which	general	direction	we	should	be	heading.
This,	it	turns	out,	is	not	at	all	easy.	Here	in	the	penultimate	chapter,	we
introduce	some	general	analytical	concepts	that	help	us	think	about	long-
term	science	and	technology	policy	issues.	We	then	apply	them	to	the	issue
of	machine	intelligence.
It	can	be	illuminating	to	make	a	rough	distinction	between	two	different
normative	stances	from	which	a	proposed	policy	may	be	evaluated.	
The	person-
affecting	perspective
	asks	whether	a	proposed	change	would	be	in	“our
interest”—that	is	to	say,	whether	it	would	(on	balance,	and	in	expectation)	be	in
the	interest	of	those	morally	considerable	creatures	who	either	already	exist	or
will	come	into	existence	independently	of	whether	the	proposed	change	occurs
or	not.	
The	impersonal	perspective

or	not.	
,	in	contrast,	gives	no	special	consideration	to
currently	existing	people,	or	to	those	who	will	come	to	exist	independently	of
whether	the	proposed	change	occurs.	Instead,	it	counts	everybody	equally,
independently	of	their	temporal	location.	The	impersonal	perspective	sees	great
value	in	bringing	new	people	into	existence,	provided	they	have	lives	worth
living:	the	more	happy	lives	created,	the	better.
This	distinction,	although	it	barely	hints	at	the	moral	complexities	associated

with	a	machine	intelligence	revolution,	can	be	useful	in	a	first-cut	analysis.	Here
we	will	first	examine	matters	from	the	impersonal	perspective.	We	will	later	see
what	changes	if	person-affecting	considerations	are	given	weight	in	our
deliberations.

Before	we	zoom	in	on	issues	specific	to	machine	superintelligence,	we	must
introduce	some	strategic	concepts	and	considerations	that	pertain	to	scientific
and	technological	development	more	generally.
	
Suppose	that	a	policymaker	proposes	to	cut	funding	for	a	certain	research	field,
out	of	concern	for	the	risks	or	long-term	consequences	of	some	hypothetical
technology	that	might	eventually	grow	from	its	soil.	She	can	then	expect	a	howl
of	opposition	from	the	research	community.
Scientists	and	their	public	advocates	often	say	that	it	is	futile	to	try	to	control
the	evolution	of	technology	by	blocking	research.	If	some	technology	is	feasible
(the	argument	goes)	it	will	be	developed	regardless	of	any	particular
policymaker’s	scruples	about	speculative	future	risks.	Indeed,	the	more	powerful
the	capabilities	that	a	line	of	development	promises	to	produce,	the	surer	we	can
be	that	somebody,	somewhere,	will	be	motivated	to	pursue	it.	Funding	cuts	will

not	stop	progress	or	forestall	its	concomitant	dangers.
Interestingly,	this	futility	objection	is	almost	never	raised	when	a	policymaker
proposes	to	
increase
	funding	to	some	area	of	research,	even	though	the	argument
would	seem	to	cut	both	ways.	One	rarely	hears	indignant	voices	protest:	“Please
do	not	increase	our	funding.	Rather,	make	some	cuts.	Researchers	in	other
countries	will	surely	pick	up	the	slack;	the	same	work	will	get	done	anyway.
Don’t	squander	the	public’s	treasure	on	domestic	scientific	research!”

increase
What	accounts	for	this	apparent	doublethink?	One	plausible	explanation,	of
course,	is	that	members	of	the	research	community	have	a	self-serving	bias
which	leads	us	to	believe	that	research	is	always	good	and	tempts	us	to	embrace
almost	any	argument	that	supports	our	demand	for	more	funding.	However,	it	is
also	possible	that	the	double	standard	can	be	justified	in	terms	of	national	self-
interest.	Suppose	that	the	development	of	a	technology	has	
two
	effects:	giving	a
small	benefit	
B
	to	its	inventors	and	the	country	that	sponsors	them,	while

two
B
imposing	an	aggregately	larger	harm	
H
—which	could	be	a	risk	externality—on
everybody.	Even	somebody	who	is	largely	altruistic	might	then	choose	to
develop	the	overall	harmful	technology.	They	might	reason	that	the	harm	
H
	will
result	no	matter	what	they	do,	since	if	they	refrain	somebody	else	will	develop

the	technology	anyway;	and	given	that	total	welfare	cannot	be	affected,	they
might	as	well	grab	the	benefit	
B
	for	themselves	and	their	nation.	(“Unfortunately,
there	will	soon	be	a	device	that	will	destroy	the	world.	Fortunately,	we	got	the
grant	to	build	it!”)
Whatever	the	explanation	for	the	futility	objection’s	appeal,	it	fails	to	show
that	there	is	in	general	no	impersonal	reason	for	trying	to	steer	technological
development.	It	fails	even	if	we	concede	the	motivating	idea	that	with	continued
scientific	and	technological	development	efforts,	all	relevant	technologies	will
eventually	be	developed—that	is,	even	if	we	concede	the	following:
Technological	completion	conjecture
If	scientific	and	technological	development	efforts	do	not	effectively	cease,
then	all	important	basic	capabilities	that	could	be	obtained	through	some
possible	technology	will	be	obtained.
1
	
There	are	at	least	two	reasons	why	the	technological	completion	conjecture	does

1
	
not	imply	the	futility	objection.	First,	the	antecedent	might	not	hold,	because	it	is
not	in	fact	a	given	that	scientific	and	technological	development	efforts	will	not
effectively	cease	(before	the	attainment	of	technological	maturity).	This
reservation	is	especially	pertinent	in	a	context	that	involves	existential	risk.
Second,	even	if	we	could	be	certain	that	all	important	basic	capabilities	that
could	be	obtained	through	some	possible	technology	will	be	obtained,	it	could
still	make	sense	to	attempt	to	influence	the	direction	of	technological	research.

What	matters	is	not	only	
whether
	a	technology	is	developed,	but	also	
when
	it	is
developed,	by	
whom
,	and	in	
what	context
.	These	circumstances	of	birth	of	a	new
technology,	which	shape	its	impact,	can	be	affected	by	turning	funding	spigots
on	or	off	(and	by	wielding	other	policy	instruments).
These	reflections	suggest	a	principle	that	would	have	us	attend	to	the	relative
speed	with	which	different	technologies	are	developed:
2
The	principle	of	differential	technological	development

whether
when
	it	is
whom
,	and	in	
2
Retard	the	development	of	dangerous	and	harmful	technologies,	especially
ones	that	raise	the	level	of	existential	risk;	and	accelerate	the	development
of	beneficial	technologies,	especially	those	that	reduce	the	existential	risks

posed	by	nature	or	by	other	technologies.
	
A	policy	could	thus	be	evaluated	on	the	basis	of	how	much	of	a	differential
advantage	it	gives	to	desired	forms	of	technological	development	over	undesired
forms.
3
	
Some	technologies	have	an	ambivalent	effect	on	existential	risks,	increasing
some	existential	risks	while	decreasing	others.	Superintelligence	is	one	such
We	have	seen	in	earlier	chapters	that	the	introduction	of	machine
superintelligence	would	create	a	substantial	existential	risk.	But	it	would	reduce
many	other	existential	risks.	Risks	from	nature—such	as	asteroid	impacts,
supervolcanoes,	and	natural	pandemics—would	be	virtually	eliminated,	since
superintelligence	could	deploy	countermeasures	against	most	such	hazards,	or	at
least	demote	them	to	the	non-existential	category	(for	instance,	via	space
colonization).
These	existential	risks	from	nature	are	comparatively	small	over	the	relevant
timescales.	But	superintelligence	would	also	eliminate	or	reduce	many

anthropogenic	risks.	In	particular,	it	would	reduce	risks	of	accidental	destruction,
including	risk	of	accidents	related	to	new	technologies.	Being	generally	more
capable	than	humans,	a	superintelligence	would	be	less	likely	to	make	mistakes,
and	more	likely	to	recognize	when	precautions	are	needed,	and	to	implement
precautions	competently.	A	well-constructed	superintelligence	might	sometimes
take	a	risk,	but	only	when	doing	so	is	wise.	Furthermore,	at	least	in	scenarios
where	the	superintelligence	forms	a	singleton,	many	non-accidental

anthropogenic	existential	risks	deriving	from	global	coordination	problems
would	be	eliminated.	These	
include	risks	of	wars,	technology	races,	undesirable
forms	of	competition	and	evolution,	and	tragedies	of	the	commons.
Since	substantial	peril	would	be	associated	with	human	beings	developing
synthetic	biology,	molecular	nanotechnology,	climate	engineering,	instruments

for	biomedical	enhancement	and	neuropsychological	manipulation,	tools	for
social	control	that	may	facilitate	totalitarianism	or	tyranny,	and	other
technologies	as-yet	unimagined,	eliminating	these	types	of	risk	would	be	a	great
boon.	An	argument	could	therefore	be	mounted	that	earlier	arrival	dates	of
superintelligence	are	preferable.	However,	if	risks	from	nature	and	from	other
hazards	unrelated	to	future	technology	are	small,	then	this	argument	could	be
refined:	what	matters	is	that	we	get	superintelligence	
before
	other	dangerous
technologies,	such	as	advanced	nanotechnology.	Whether	it	happens	sooner	or
later	may	not	be	so	important	(from	an	impersonal	perspective)	so	long	as	the
order	of	arrival	is	right.
The	ground	for	preferring	superintelligence	to	come	before	other	potentially
dangerous	technologies,	such	as	nanotechnology,	is	that	superintelligence	would
reduce	the	existential	risks	from	nanotechnology	but	not	vice	versa.
4
	Hence,	if

before
4
	Hence,	if
we	create	superintelligence	first,	we	will	face	only	those	existential	risks	that	are
associated	with	superintelligence;	whereas	if	we	create	nanotechnology	first,	we
will	face	the	risks	of	nanotechnology	and	then,	additionally,	the	risks	of
5
	Even	if	the	existential	risks	from	superintelligence	are	very
large,	and	even	if	superintelligence	is	the	riskiest	of	all	technologies,	there	could
thus	be	a	case	for	hastening	its	arrival.

4
	Hence,	if
5
These	“sooner-is-better”	arguments,	however,	presuppose	that	the	riskiness	of
creating	superintelligence	is	the	same	regardless	of	when	it	is	created.	If,	instead,
its	riskiness	declines	over	time,	it	might	be	better	to	delay	the	machine
intelligence	revolution.	While	a	later	arrival	would	leave	more	time	for	other
existential	catastrophes	to	intercede,	it	could	still	be	preferable	to	slow	the
development	of	superintelligence.	This	would	be	especially	plausible	if	the
existential	risks	associated	with	superintelligence	are	much	larger	than	those

associated	with	other	disruptive	technologies.
There	are	several	quite	strong	reasons	to	believe	that	the	riskiness	of	an
intelligence	explosion	will	decline	significantly	over	a	multidecadal	timeframe.
One	reason	is	that	a	later	date	leaves	more	time	for	the	development	of	solutions
to	the	control	problem.	The	control	problem	has	only	recently	been	recognized,
and	most	of	the	current	best	ideas	for	how	to	approach	it	were	discovered	only
within	the	past	decade	or	so	(and	in	several	cases	during	the	time	that	this	book

was	being	written).	It	is	plausible	that	the	state	of	the	art	will	advance	greatly
over	the	next	several	decades;	and	if	the	problem	turns	out	to	be	very	difficult,	a
significant	rate	of	progress	might	continue	for	a	century	or	more.	The	longer	it

takes	for	superintelligence	to	arrive,	the	more	such	progress	will	have	been	made
when	it	does.	This	is	an	important	consideration	in	favor	of	later	arrival	dates—
and	a	very	strong	consideration	against	extremely	early	arrival	dates.
Another	reason	why	superintelligence	later	might	be	safer	is	that	this	would
allow	more	time	for	various	beneficial	background	trends	of	human	civilization
to	play	themselves	out.	How	much	weight	one	attaches	to	this	consideration	will
depend	on	how	optimistic	one	is	about	these	trends.
An	optimist	could	certainly	point	to	a	number	of	encouraging	indicators	and
hopeful	possibilities.	People	might	learn	to	get	along	better,	leading	to	reductions
in	violence,	war,	and	cruelty;	and	global	coordination	and	the	scope	of	political
integration	might	increase,	making	it	easier	to	escape	undesirable	technology
races	(more	on	this	below)	and	to	work	out	an	arrangement	whereby	the	hoped-
for	gains	from	an	intelligence	explosion	would	be	widely	shared.	There	appear

to	be	long-term	historical	trends	in	these	directions.
6
Further,	an	optimist	could	expect	that	the	“sanity	level”	of	humanity	will	rise
over	the	course	of	this	century—that	prejudices	will	(on	balance)	recede,	that
insights	will	accumulate,	and	that	people	will	become	more	accustomed	to
thinking	about	abstract	future	probabilities	and	global	risks.	With	luck,	we	could
see	a	general	uplift	of	epistemic	standards	in	both	individual	and	collective
cognition.	Again,	there	are	trends	pushing	in	these	directions.	Scientific	progress

6
means	that	more	will	be	known.	Economic	growth	may	give	a	greater	portion	of
the	world’s	population	adequate	nutrition	(particularly	during	the	early	years	of
life	that	are	important	for	brain	development)	and	access	to	quality	education.
Advances	in	information	technology	will	make	it	easier	to	find,	integrate,
evaluate,	and	communicate	data	and	ideas.	Furthermore,	by	the	century’s	end,
humanity	will	have	made	an	additional	hundred	years’	worth	of	mistakes,	from
which	something	might	have	been	learned.

Many	potential	developments	are	ambivalent	in	the	abovementioned	sense—
increasing	some	existential	risks	and	decreasing	others.	For	example,	advances
in	surveillance,	data	mining,	lie	detection,	biometrics,	and	psychological	or
neurochemical	means	of	manipulating	beliefs	and	desires	could	reduce	some
existential	risks	by	making	it	easier	to	coordinate	internationally	or	to	suppress
terrorists	and	renegades	at	home.	These	same	advances,	however,	might	also

increase	some	existential	risks	by	amplifying	undesirable	social	dynamics	or	by
enabling	the	formation	of	permanently	stable	totalitarian	regimes.

One	important	frontier	is	the	enhancement	of	biological	cognition,	such	as
through	genetic	selection.	When	we	discussed	this	in	
Chapters	2
	and	
3
,	we
concluded	that	the	most	radical	forms	of	superintelligence	would	be	more	likely
to	arise	in	the	form	of	machine	intelligence.	That	claim	is	consistent	with
cognitive	enhancement	playing	an	important	role	in	the	lead-up	to,	and	creation
of,	machine	superintelligence.	Cognitive	enhancement	might	seem	obviously
risk-reducing:	the	smarter	the	people	working	on	the	control	problem,	the	more
likely	they	are	to	find	a	solution.	However,	cognitive	enhancement	could	also
hasten	the	development	of	machine	intelligence,	thus	reducing	the	time	available
to	work	on	the	problem.	Cognitive	enhancement	would	also	have	many	other
relevant	consequences.	These	issues	deserve	a	closer	look.	(Most	of	the
following	remarks	about	“cognitive	enhancement”	apply	equally	to	non-
biological	means	of	increasing	our	individual	or	collective	epistemic
effectiveness.)

An	increase	in	either	the	mean	or	the	upper	range	of	human	intellectual	ability
would	likely	accelerate	technological	progress	across	the	board,	including
progress	toward	various	forms	of	machine	intelligence,	progress	on	the	control
problem,	and	progress	on	a	wide	swath	of	other	technical	and	economic
objectives.	What	would	be	the	net	effect	of	such	acceleration?
Consider	the	limiting	case	of	a	“universal	accelerator,”	an	imaginary
intervention	that	accelerates	literally	
everything

.	The	action	of	such	a	universal
accelerator	would	correspond	merely	to	an	arbitrary	rescaling	of	the	time	metric,
producing	no	qualitative	change	in	observed	outcomes.
7
If	we	are	to	make	sense	of	the	idea	that	cognitive	enhancement	might
generally	speed	things	up,	we	clearly	need	some	other	concept	than	that	of
universal	acceleration.	A	more	promising	approach	is	to	focus	on	how	cognitive
enhancement	might	increase	the	rate	of	change	in	one	type	of	process	
relative
	to

7
relative
	to
the	rate	of	change	in	some	other	type	of	process.	Such	differential	acceleration
could	affect	a	system’s	dynamics.	Thus,	consider	the	following	concept:
Macro-structural	development	accelerator
—A	lever	that	accelerates	the
rate	at	which	macro-structural	features	of	the	human	condition	develop,

while	leaving	unchanged	the	rate	at	which	micro-level	human	affairs
unfold.
	
Imagine	pulling	this	lever	in	the	decelerating	direction.	A	brake	pad	is	lowered
onto	the	great	wheel	of	world	history;	sparks	fly	and	metal	screeches.	After	the
wheel	has	settled	into	a	more	leisurely	pace,	the	result	is	a	world	in	which
technological	innovation	occurs	more	slowly	and	in	which	fundamental	or
globally	significant	change	in	political	structure	and	culture	happens	less
frequently	and	less	abruptly.	A	greater	number	of	generations	come	and	go
before	one	era	gives	way	to	another.	During	the	course	of	a	lifespan,	a	person
sees	little	change	in	the	basic	structure	of	the	human	condition.
For	most	of	our	species’	existence,	macro-structural	development	was	slower
than	it	is	now.	Fifty	thousand	years	ago,	an	entire	millennium	might	have	elapsed
without	a	single	significant	technological	invention,	without	any	noticeable
increase	in	human	knowledge	and	understanding,	and	without	any	globally

meaningful	political	change.	On	a	micro-level,	however,	the	kaleidoscope	of
human	affairs	churned	at	a	reasonable	rate,	with	births,	deaths,	and	other
personally	and	locally	significant	events.	The	average	person’s	day	might	have
been	more	action-packed	in	the	Pleistocene	than	it	is	today.
If	you	came	upon	a	magic	lever	that	would	let	you	change	the	rate	of	macro-
structural	development,	what	should	you	do?	Ought	you	to	accelerate,
decelerate,	or	leave	things	as	they	are?
Assuming	the	impersonal	standpoint,	this	question	requires	us	to	consider	the

effects	on	existential	risk.	Let	us	distinguish	between	two	kinds	of	risk:	“state
risks”	and	“step	risks.”	A	state	risk	is	one	that	is	associated	with	being	in	a
certain	state,	and	the	total	amount	of	state	risk	to	which	a	system	is	exposed	is	a
direct	
function	of	how	long	the	system	remains	in	that	state.	Risks	from	nature
are	typically	state	risks:	the	longer	we	remain	exposed,	the	greater	the	chance
that	we	will	get	struck	by	an	asteroid,	supervolcanic	eruption,	gamma	ray	burst,

direct	
naturally	arising	pandemic,	or	some	other	slash	of	the	cosmic	scythe.	Some
anthropogenic	risks	are	also	state	risks.	At	the	level	of	an	individual,	the	longer	a
soldier	pokes	his	head	up	above	the	parapet,	the	greater	the	cumulative	chance
he	will	be	shot	by	an	enemy	sniper.	There	are	anthropogenic	state	risks	at	the
existential	level	as	well:	the	longer	we	live	in	an	internationally	anarchic	system,

the	greater	the	cumulative	chance	of	a	thermonuclear	Armageddon	or	of	a	great
war	fought	with	other	kinds	of	weapons	of	mass	destruction,	laying	waste	to
A	step	risk,	by	contrast,	is	a	discrete	risk	associated	with	some	necessary	or
desirable	transition.	Once	the	transition	is	completed,	the	risk	vanishes.	The
amount	of	step	risk	associated	with	a	transition	is	usually	not	a	simple	function
of	how	long	the	transition	takes.	One	does	not	halve	the	risk	of	traversing	a
minefield	by	running	twice	as	fast.	Conditional	on	a	fast	takeoff,	the	creation	of
superintelligence	might	be	a	step	risk:	there	would	be	a	certain	risk	associated
with	the	takeoff,	the	magnitude	of	which	would	depend	on	what	preparations	had
been	made;	but	the	amount	of	risk	might	not	depend	much	on	whether	the
takeoff	takes	twenty	milliseconds	or	twenty	hours.
We	can	then	say	the	following	regarding	a	hypothetical	macro-structural
development	accelerator:

•	Insofar	as	we	are	concerned	with	existential	state	risks,	we	should	favor
acceleration—provided	we	think	we	have	a	realistic	prospect	of	making	it
through	to	a	post-transition	era	in	which	any	further	existential	risks	are	greatly
reduced.
•	If	it	were	known	that	there	is	some	step	ahead	destined	to	cause	an	existential
catastrophe,	then	we	ought	to	reduce	the	rate	of	macro-structural	development
(or	even	put	it	in	reverse)	in	order	to	give	more	generations	a	chance	to	exist

reduced.
before	the	curtain	is	rung	down.	But,	in	fact,	it	would	be	overly	pessimistic	to	be
so	confident	that	humanity	is	doomed.
•	At	present,	the	level	of	existential	state	risk	appears	to	be	relatively	low.	If	we
imagine	the	technological	macro-conditions	for	humanity	frozen	in	their	current
state,	it	seems	very	unlikely	that	an	existential	catastrophe	would	occur	on	a
timescale	of,	say,	a	decade.	So	a	delay	of	one	decade—provided	it	occurred	at

our	current	stage	of	development	or	at	some	other	time	when	state	risk	is	low—
would	incur	only	a	very	minor	existential	state	risk,	whereas	a	postponement	by
one	decade	of	subsequent	technological	developments	might	well	have	a
significant	beneficial	impact	on	later	existential	step	risks,	for	example	by
allowing	more	time	for	preparation.

Upshot:	the	main	way	that	the	speed	of	macro-structural	development	is
important	is	by	affecting	how	well	prepared	humanity	is	when	the	time	comes	to
confront	the	key	step	risks.
8
So	the	question	we	must	ask	is	how	cognitive	enhancement	(and	concomitant
acceleration	of	macro-structural	development)	would	affect	the	expected	level	of
preparedness	at	the	critical	juncture.	Should	we	prefer	a	shorter	period	of
preparation	with	higher	intelligence?	With	higher	intelligence,	the	preparation
time	could	be	used	more	effectively,	and	the	final	critical	step	would	be	taken	by
a	more	intelligent	humanity.	Or	should	we	prefer	to	operate	with	closer	to
current	levels	of	intelligence	if	that	gives	us	more	time	to	prepare?
Which	option	is	better	depends	on	the	nature	of	the	challenge	being	prepared
for.	If	the	challenge	were	to	solve	a	problem	for	which	learning	from	experience
is	key,	then	the	chronological	length	of	the	preparation	period	might	be	the

determining	factor,	since	time	is	needed	for	the	requisite	experience	to
accumulate.	What	would	such	a	challenge	look	like?	One	hypothetical	example
would	be	a	new	weapons	technology	that	we	could	predict	would	be	developed
at	some	point	in	the	future	and	that	would	make	it	the	case	that	any	subsequent
war	would	have,	let	us	say,	a	one-in-ten	chance	of	causing	an	existential
catastrophe.	If	such	were	the	nature	of	the	challenge	facing	us,	then	we	might
wish	the	rate	of	macro-structural	development	to	be	slow,	so	that	our	species

would	have	more	time	to	get	its	act	together	before	the	critical	step	when	the
new	weapons	technology	is	invented.	One	could	hope	that	during	the	grace
period	secured	through	the	deceleration,	our	species	might	learn	to	avoid	war—
that	international	relations	around	the	globe	might	come	to	resemble	those
between	the	countries	of	the	European	Union,	which,	having	fought	one	another
ferociously	for	centuries,	now	coexist	in	peace	and	relative	harmony.	The
pacification	might	occur	as	a	result	of	the	gentle	edification	from	various

civilizing	processes	or	through	the	shock	therapy	of	sub-existential	blows	(e.g.
small	nuclear	conflagrations,	and	the	recoil	and	resolve	they	might	engender	to
finally	create	the	global	institutions	necessary	for	the	abolishment	of	interstate
wars).	If	this	kind	of	learning	or	adjusting	would	not	be	much	accelerated	by
increased	intelligence,	then	cognitive	enhancement	would	be	undesirable,
serving	merely	to	burn	the	fuse	faster.
A	prospective	intelligence	explosion,	however,	may	present	a	challenge	of	a

different	kind.	The	control	problem	calls	for	foresight,	reasoning,	and	theoretical
insight.	It	is	less	clear	how	increased	historical	experience	would	help.	Direct

experience	of	the	intelligence	explosion	is	not	possible	(until	too	late),	and	many
features	conspire	to	make	the	control	problem	unique	and	lacking	in	relevant
historical	precedent.	For	these	reasons,	the	amount	of	time	that	will	elapse	before
the	intelligence	explosion	may	not	matter	much	per	se.	Perhaps	what	matters,
instead,	is	(a)	the	amount	of	intellectual	progress	on	the	control	problem
achieved	by	the	time	of	the	detonation;	and	(b)	the	amount	of	skill	and
intelligence	available	at	the	time	to	implement	the	best	available	solutions	(and
to	improvise	what	is	missing).
9
	That	this	latter	factor	should	respond	positively
to	cognitive	enhancement	is	obvious.	How	cognitive	enhancement	would	affect
factor	(a)	is	a	somewhat	subtler	matter.
Suppose,	as	suggested	earlier,	that	cognitive	enhancement	would	be	a	general
macro-structural	development	accelerator.	This	would	hasten	the	arrival	of	the
intelligence	explosion,	thus	reducing	the	amount	of	time	available	for

9
preparation	and	for	making	progress	on	the	control	problem.	Normally	this
would	be	a	
bad	thing.	However,	if	the	only	reason	why	there	is	less	time
available	for	intellectual	progress	is	that	intellectual	progress	is	speeded	up,	then
there	need	be	no	net	reduction	in	the	amount	of	intellectual	progress	that	will
have	taken	place	by	the	time	the	intelligence	explosion	occurs.
At	this	point,	cognitive	enhancement	might	appear	to	be	neutral	with	respect

would	be	a	
to	factor	(a):	the	same	intellectual	progress	that	would	otherwise	have	been	made
prior	to	the	intelligence	explosion—including	progress	on	the	control	problem—
still	gets	made,	only	compressed	within	a	shorter	time	interval.	In	actuality,
however,	cognitive	enhancement	may	well	prove	a	positive	influence	on	(a).
One	reason	why	cognitive	enhancement	might	cause	more	progress	to	have
been	made	on	the	control	problem	by	the	time	the	intelligence	explosion	occurs
is	that	progress	on	the	control	problem	may	be	especially	contingent	on	extreme

levels	of	intellectual	performance—even	more	so	than	the	kind	of	work
necessary	to	create	machine	intelligence.	The	role	for	trial	and	error	and
accumulation	of	experimental	results	seems	quite	limited	in	relation	to	the
control	problem,	whereas	experimental	learning	will	probably	play	a	large	role
in	the	development	of	artificial	intelligence	or	whole	brain	emulation.	The	extent
to	which	time	can	substitute	for	wit	may	therefore	vary	between	tasks	in	a	way
that	should	make	cognitive	enhancement	promote	progress	on	the	control
problem	
more

problem	
more
	than	it	would	promote	progress	on	the	problem	of	how	to	create

Another	reason	why	cognitive	enhancement	should	differentially	promote
progress	on	the	control	problem	is	that	the	very	need	for	such	progress	is	more
likely	to	be	appreciated	by	cognitively	more	capable	societies	and	individuals.	It
requires	foresight	and	reasoning	to	realize	why	the	control	problem	is	important
and	to	make	it	a	priority.
10
	It	may	also	require	uncommon	sagacity	to	find
promising	ways	of	approaching	such	an	unfamiliar	problem.
From	these	reflections	we	might	tentatively	conclude	that	cognitive
enhancement	is	desirable,	at	least	insofar	as	the	focus	is	on	the	existential	risks
of	an	intelligence	explosion.	Parallel	lines	of	thinking	apply	to	other	existential
risks	arising	from	challenges	that	require	foresight	and	reliable	abstract
reasoning	(as	opposed	to,	e.g.,	incremental	adaptation	to	experienced	changes	in
the	environment	or	a	multigenerational	process	of	cultural	maturation	and
institution-building).

Suppose	that	one	thinks	that	solving	the	control	problem	for	artificial
intelligence	is	very	difficult,	that	solving	it	for	whole	brain	emulations	is	much
easier,	and	that	it	would	therefore	be	preferable	that	machine	intelligence	be
reached	via	the	whole	brain	emulation	path.	We	will	return	later	to	the	question
of	whether	whole	brain	emulation	would	be	safer	than	artificial	intelligence.	But
for	now	we	want	to	make	the	point	that	even	if	we	accept	this	premiss,	it	would

not	follow	that	we	ought	to	promote	whole	brain	emulation	technology.	One
reason,	discussed	earlier,	is	that	a	later	arrival	of	superintelligence	may	be
preferable,	in	order	to	allow	more	time	
for	progress	on	the	control	problem	and
for	other	favorable	background	trends	to	culminate—and	thus,	if	one	were
confident	that	whole	brain	emulation	would	precede	AI	anyway,	it	would	be
counterproductive	to	further	hasten	the	arrival	of	whole	brain	emulation.

But	even	if	it	were	the	case	that	it	would	be	best	for	whole	brain	emulation	to
arrive	as	soon	as	possible,	it	
still
	would	not	follow	that	we	ought	to	favor
progress	toward	whole	brain	emulation.	For	it	is	possible	that	progress	toward
whole	brain	emulation	will	not	yield	whole	brain	emulation.	It	may	instead	yield
neuromorphic	artificial	intelligence—forms	of	AI	that	mimic	some	aspects	of
cortical	organization	but	do	not	replicate	neuronal	functionality	with	sufficient
fidelity	to	constitute	a	proper	emulation.	If—as	there	is	reason	to	believe—such

neuromorphic	AI	is	worse	than	the	kind	of	AI	that	would	otherwise	have	been
built,	and	if	by	promoting	whole	brain	emulation	we	would	make	neuromorphic
AI	arrive	first,	then	our	pursuit	of	the	supposed	
best
	outcome	(whole	brain
emulation)	would	lead	to	the	
worst
	outcome	(neuromorphic	AI);	whereas	if	we
had	pursued	the	
second-best
	outcome	(synthetic	AI)	we	might	actually	have
attained	the	second-best	(synthetic	AI).
We	have	just	described	an	(hypothetical)	instance	of	what	we	might	term	a
“technology	coupling.”
11
	This	refers	to	a	condition	in	which	two	technologies
have	a	predictable	timing	relationship,	such	that	developing	one	of	the
technologies	has	a	robust	tendency	to	lead	to	the	development	of	the	other,	either
as	a	necessary	precursor	or	as	an	obvious	and	irresistible	application	or
subsequent	step.	Technology	couplings	must	be	taken	into	account	when	we	use
the	principle	of	differential	technological	development:	it	is	no	good	accelerating
the	development	of	a	desirable	technology	
Y

11
Y
	if	the	only	way	of	getting	
Y
	is	by
developing	an	extremely	undesirable	precursor	technology	
X
,	or	if	getting	
Y
would	immediately	produce	an	extremely	undesirable	related	technology	
Z
.
Before	you	marry	your	sweetheart,	consider	the	prospective	in-laws.
In	the	case	of	whole	brain	emulation,	the	degree	of	technology	coupling	is
debatable.	We	noted	in	
Chapter	2
	that	while	whole	brain	emulation	would
require	massive	progress	in	various	enabling	technologies,	it	might	not	require

Y
Y
	is	by
X
Y
Z
.
Chapter	2
any	major	new	theoretical	insight.	In	particular,	it	does	not	require	that	we
understand	how	human	cognition	works,	only	that	we	know	how	to	build
computational	models	of	small	parts	of	the	brain,	such	as	different	species	of
neuron.	Nevertheless,	in	the	course	of	developing	the	ability	to	emulate	human
brains,	a	wealth	of	neuroanatomical	data	would	be	collected,	and	functional
models	of	cortical	networks	would	surely	be	greatly	improved.	Such	progress
would	seem	to	have	a	good	chance	of	enabling	neuromorphic	AI	before	full-

blown	whole	brain	emulation.
12
	Historically,	there	are	quite	a	few	examples	of
AI	techniques	gleaned	from	neuroscience	or	biology.	(For	example:	the
McCulloch–Pitts	neuron,	perceptrons,	and	other	artificial	neurons	and	neural
networks,	inspired	by	neuroanatomical	work;	reinforcement	learning,	inspired	by
behaviorist	psychology;	genetic	algorithms,	inspired	by	evolution	theory;
subsumption	architectures	and	perceptual	hierarchies,	inspired	by	cognitive
science	theories	about	
motor	planning	and	sensory	perception;	artificial	immune

12
systems,	inspired	by	theoretical	immunology;	swarm	intelligence,	inspired	by
the	ecology	of	insect	colonies	and	other	self-organizing	systems;	and	reactive
and	behavior-based	control	in	robotics,	inspired	by	the	study	of	animal

locomotion.)	Perhaps	more	significantly,	there	are	plenty	of	important	AI-
relevant	questions	that	could	potentially	be	answered	through	further	study	of
the	brain.	(For	example:	How	does	the	brain	store	structured	representations	in
working	memory	and	long-term	memory?	How	is	the	binding	problem	solved?
What	is	the	neural	code?	How	are	concepts	represented?	Is	there	some	standard
unit	of	cortical	processing	machinery,	such	as	the	cortical	column,	and	if	so	how
is	it	wired	and	how	does	its	functionality	depend	on	the	wiring?	How	can	such
columns	be	linked	up,	and	how	can	they	learn?)
We	will	shortly	have	more	to	say	about	the	relative	danger	of	whole	brain
emulation,	neuromorphic	AI,	and	synthetic	AI,	but	we	can	already	flag	another
important	technology	coupling:	that	between	whole	brain	emulation	and	AI.
Even	if	a	push	toward	whole	brain	emulation	actually	resulted	in	whole	brain
emulation	(as	opposed	to	neuromorphic	AI),	and	even	if	the	arrival	of	whole

brain	emulation	could	be	safely	handled,	a	further	risk	would	still	remain:	the
risk	associated	with	
a	second	transition
,	a	transition	from	whole	brain	emulation
to	AI,	which	is	an	ultimately	more	powerful	form	of	machine	intelligence.
There	are	many	other	technology	couplings,	which	could	be	considered	in	a
more	comprehensive	analysis.	For	instance,	a	push	toward	whole	brain
emulation	would	boost	neuroscience	progress	more	generally.
13
	That	might

13
	That	might
produce	various	effects,	such	as	faster	progress	toward	lie	detection,
neuropsychological	manipulation	techniques,	cognitive	enhancement,	and
assorted	medical	advances.	Likewise,	a	push	toward	cognitive	enhancement
might	(depending	on	the	specific	path	pursued)	create	spillovers	such	as	faster
development	of	genetic	selection	and	genetic	engineering	methods	not	only	for
enhancing	cognition	but	for	modifying	other	traits	as	well.
	
We	encounter	another	layer	of	strategic	complexity	if	we	take	into	account	that

there	is	no	perfectly	benevolent,	rational,	and	unified	world	controller	who
simply	implements	what	has	been	discovered	to	be	the	best	option.	Any	abstract
point	about	“what	should	be	done”	must	be	embodied	in	the	form	of	a	concrete
message,	which	is	entered	into	the	arena	of	rhetorical	and	political	reality.	There
it	will	be	ignored,	misunderstood,	distorted,	or	appropriated	for	various
conflicting	purposes;	it	will	bounce	around	like	a	pinball,	causing	actions	and

reactions,	ushering	in	a	cascade	of	consequences,	the	upshot	of	which	need	bear
no	straightforward	relationship	to	the	intentions	of	the	original	sender.
A	sophisticated	operator	might	try	to	anticipate	these	kinds	of	effect.	Consider,
for	example,	the	following	argument	template	for	proceeding	with	research	to
develop	a	dangerous	technology	
X
.	(One	argument	fitting	this	template	can	
be
found	in	the	writings	of	Eric	Drexler.	In	Drexler’s	case,	
X
	=	molecular
nanotechnology.
14
)
	
	The	risks	of	
X
	are	great.
	Reducing	these	risks	will	require	a	period	of	serious	preparation.
	Serious	preparation	will	begin	only	once	the	prospect	of	
X
	is	taken	seriously	by
broad	sectors	of	society.
	Broad	sectors	of	society	will	take	the	prospect	of	
X
	seriously	only	once	a	large
research	effort	to	develop	
X
	is	underway.
	The	earlier	a	serious	research	effort	is	initiated,	the	longer	it	will	take	to	deliver	
X
(because	it	starts	from	a	lower	level	of	pre-existing	enabling	technologies).

14
)
	
X
	are	great.
X
X
X
X
	Therefore,	the	earlier	a	serious	research	effort	is	initiated,	the	longer	the	period
during	which	serious	preparation	will	be	taking	place,	and	the	greater	the
reduction	of	the	risks.
	Therefore,	a	serious	research	effort	toward	
X
	should	be	initiated	immediately.
What	initially	looks	like	a	reason	for	going	slow	or	stopping—the	risks	of	
X
being	great—ends	up,	on	this	line	of	thinking,	as	a	reason	for	the	opposite
conclusion.
A	related	type	of	argument	is	that	we	ought—rather	callously—to	welcome

Therefore,	the	earlier	a	serious	research	effort	is	initiated,	the	longer	the	period
X
X
small	and	medium-scale	catastrophes	on	grounds	that	they	make	us	aware	of	our
vulnerabilities	and	spur	us	into	taking	precautions	that	reduce	the	probability	of
an	existential	catastrophe.	The	idea	is	that	a	small	or	medium-scale	catastrophe
acts	like	an	inoculation,	challenging	civilization	with	a	relatively	survivable
form	of	a	threat	and	stimulating	an	immune	response	that	readies	the	world	to

deal	with	the	existential	variety	of	the	threat.
15
These	“shock’em-into-reacting”	arguments	advocate	letting	something	bad
happen	in	the	hope	that	it	will	galvanize	a	public	reaction.	We	mention	them	here
not	to	endorse	them,	but	as	a	way	to	introduce	the	idea	of	(what	we	will	term)
“second-guessing	arguments.”	Such	arguments	maintain	that	by	treating	others
as	irrational	and	playing	to	their	biases	and	misconceptions	it	is	possible	to	elicit
a	response	from	them	that	is	more	competent	than	if	a	case	had	been	presented
honestly	and	forthrightly	to	their	rational	faculties.
It	may	seem	unfeasibly	difficult	to	use	the	kind	of	stratagems	recommended
by	second-guessing	arguments	to	achieve	long-term	global	goals.	How	could
anybody	predict	the	final	course	of	a	message	after	it	has	been	jolted	hither	and
thither	in	the	pinball	machine	of	public	discourse?	Doing	so	would	seem	to
require	predicting	the	rhetorical	effects	on	myriad	constituents	with	varied

idiosyncrasies	and	fluctuating	levels	of	influence	over	long	periods	of	time
during	which	the	system	may	be	perturbed	by	unanticipated	events	from	the
outside	while	its	topology	is	also	undergoing	a	continuous	endogenous
reorganization:	surely	an	impossible	task!
16
	However,	it	may	not	be	necessary	to
make	detailed	predictions	about	the	system’s	entire	future	trajectory	in	order	to
identify	an	intervention	that	can	be	reasonably	expected	to	increase	the	chances
of	a	certain	long-term	outcome.	One	might,	for	example,	consider	only	the

16
relatively	near-term	and	predictable	effects	
in	a	detailed	way,	selecting	an	action
that	does	well	in	regard	to	those,	while	modeling	the	system’s	behavior	beyond
the	predictability	horizon	as	a	random	walk.
There	may,	however,	be	a	moral	case	for	de-emphasizing	or	refraining	from
second-guessing	moves.	Trying	to	outwit	one	another	looks	like	a	zero-sum
game—or	negative-sum,	when	one	considers	the	time	and	energy	that	would	be
dissipated	by	the	practice	as	well	as	the	likelihood	that	it	would	make	it

generally	harder	for	anybody	to	discover	what	others	truly	think	and	to	be
trusted	when	expressing	their	own	opinions.
17
	A	full-throttled	deployment	of	the
practices	of	strategic	communication	would	kill	candor	and	leave	truth	bereft	to
fend	for	herself	in	the	backstabbing	night	of	political	bogeys.

Should	we	celebrate	advances	in	computer	hardware?	What	about	advances	on
the	path	toward	whole	brain	emulation?	We	will	look	at	these	two	questions	in
turn.
	
Faster	computers	make	it	easier	to	create	machine	intelligence.	One	effect	of
accelerating	progress	in	hardware,	therefore,	is	to	hasten	the	arrival	of	machine
intelligence.	As	discussed	earlier,	this	is	probably	a	bad	thing	from	the
impersonal	perspective,	since	it	reduces	the	amount	of	time	available	for	solving
the	control	problem	and	for	humanity	to	reach	a	more	mature	stage	of
civilization.	The	case	is	not	a	slam	dunk,	though.	Since	superintelligence	would
eliminate	many	other	existential	risks,	there	could	be	reason	to	prefer	earlier
development	if	the	level	of	these	other	existential	risks	were	very	high.
18
Hastening	or	delaying	the	onset	of	the	intelligence	explosion	is	not	the	only
channel	through	which	the	rate	of	hardware	progress	can	affect	existential	risk.

18
Another	channel	is	that	hardware	can	to	some	extent	substitute	for	software;
thus,	better	hardware	reduces	the	minimum	skill	required	to	code	a	seed	AI.	Fast
computers	might	also	encourage	the	use	of	approaches	that	rely	more	heavily	on
brute-force	techniques	(such	as	genetic	algorithms	and	other	generate-evaluate-
discard	methods)	and	less	on	techniques	that	require	deep	understanding	to	use.
If	brute-force	techniques	lend	themselves	to	more	anarchic	or	imprecise	system
designs,	where	the	control	problem	is	harder	to	solve	than	in	more	precisely

engineered	and	theoretically	controlled	systems,	this	would	be	another	way	in
which	faster	computers	would	increase	the	existential	risk.
Another	consideration	is	that	rapid	hardware	progress	increases	the	likelihood
of	a	fast	takeoff.	The	more	rapidly	the	state	of	the	art	advances	in	the
semiconductor	industry,	the	fewer	the	person-hours	of	programmers’	time	spent
exploiting	the	capabilities	of	computers	at	any	given	performance	level.	This
means	that	an	intelligence	explosion	is	less	likely	to	be	initiated	at	the	lowest
level	of	hardware

level	of	hardware	
performance	at	which	it	is	feasible.	An	intelligence	explosion
is	thus	
more
	likely	to	be	initiated	when	hardware	has	advanced	significantly
beyond	the	minimum	level	at	which	the	eventually	successful	programming
approach	could	first	have	succeeded.	There	is	then	a	hardware	overhang	when
the	takeoff	eventually	does	occur.	As	we	saw	in	
Chapter	4
,	hardware	overhang	is

one	of	the	main	factors	that	reduce	recalcitrance	during	the	takeoff.	Rapid
hardware	progress,	therefore,	will	tend	to	make	the	transition	to
superintelligence	faster	and	more	explosive.
A	faster	takeoff	via	a	hardware	overhang	can	affect	the	risks	of	the	transition
in	several	ways.	The	most	obvious	is	that	a	faster	takeoff	offers	less	opportunity
to	respond	and	make	adjustments	whilst	the	transition	is	in	progress,	which
would	tend	to	increase	risk.	A	related	consideration	is	that	a	hardware	overhang
would	reduce	the	chances	that	a	dangerously	self-improving	seed	AI	could	be
contained	by	limiting	its	ability	to	colonize	sufficient	hardware:	the	faster	each
processor	is,	the	fewer	processors	would	be	needed	for	the	AI	to	quickly
bootstrap	itself	to	superintelligence.	Yet	another	effect	of	a	hardware	overhang	is
to	level	the	playing	field	between	big	and	small	projects	by	reducing	the
importance	of	one	of	the	advantages	of	larger	projects—the	ability	to	afford

more	powerful	computers.	This	effect,	too,	might	increase	existential	risk,	if
larger	projects	are	more	likely	to	solve	the	control	problem	and	to	be	pursuing
morally	acceptable	objectives.
19
There	are	also	advantages	to	a	faster	takeoff.	A	faster	takeoff	would	increase
the	likelihood	that	a	singleton	will	form.	If	establishing	a	singleton	is	sufficiently
important	for	solving	the	post-transition	coordination	problems,	it	might	be
worth	accepting	a	greater	risk	during	the	intelligence	explosion	in	order	to

19
mitigate	the	risk	of	catastrophic	coordination	failures	in	its	aftermath.
Developments	in	computing	can	affect	the	outcome	of	a	machine	intelligence
revolution	not	only	by	playing	a	direct	role	in	the	construction	of	machine
intelligence	but	also	by	having	diffuse	effects	on	society	that	indirectly	help
shape	the	initial	conditions	of	the	intelligence	explosion.	The	Internet,	which
required	hardware	to	be	good	enough	to	enable	personal	computers	to	be	mass
produced	at	low	cost,	is	now	influencing	human	activity	in	many	areas,

including	work	in	artificial	intelligence	and	research	on	the	control	problem.
(This	book	might	not	have	been	written,	and	you	might	not	have	found	it,
without	the	Internet.)	However,	hardware	is	already	good	enough	for	a	great
many	applications	that	could	facilitate	human	communication	and	deliberation,
and	it	is	not	clear	that	the	pace	of	progress	in	these	areas	is	strongly	bottlenecked
by	the	rate	of	hardware	improvement.
20
On	balance,	it	appears	that	faster	progress	in	computing	hardware	is

20
undesirable	from	the	impersonal	evaluative	standpoint.	This	tentative	conclusion

could	be	overturned,	for	example	if	the	threats	from	other	existential	risks	or
from	post-transition	coordination	failures	turn	out	to	be	extremely	large.	In	any
case,	it	seems	difficult	to	have	much	leverage	on	the	rate	of	hardware
advancement.	Our	efforts	to	improve	the	initial	conditions	for	the	intelligence
explosion	should	therefore	probably	focus	on	other	parameters.
Note	that	even	when	we	cannot	see	how	to	influence	some	parameter,	it	can
be	useful	to	determine	its	“sign”	(i.e.	whether	an	increase	or	decrease	in	that
parameter	would	be	desirable)	as	a	preliminary	step	in	mapping	the	strategic	lay
of	the	land.	We	might	later	discover	a	new	leverage	point	that	does	enable	us	to
manipulate	the	parameter	more	easily.	Or	we	might	discover	that	the	parameter’s
sign	correlates	with	the	sign	of	some	other	more	manipulable	parameter,	so	that
our	initial	analysis	helps	us	decide	what	to	do	with	this	other	parameter.

The	harder	it	seems	to	solve	the	control	problem	for	artificial	intelligence,	the
more	tempting	it	is	to	promote	the	whole	brain	emulation	path	as	a	less	risky
alternative.	There	are	several	issues,	however,	that	must	be	analyzed	before	one
can	arrive	at	a	well-considered	judgment.
21
First,	there	is	the	issue	of	technology	coupling,	already	discussed	earlier.	We
pointed	out	that	an	effort	to	develop	whole	brain	emulation	could	result	in
neuromorphic	AI	instead,	a	form	of	machine	intelligence	that	may	be	especially
unsafe.

21
unsafe.
But	let	us	assume,	for	the	sake	of	argument,	that	we	actually	achieve	whole
brain	emulation	(WBE).	Would	this	be	safer	than	AI?	This,	itself,	is	a
complicated	issue.	There	are	at	least	three	
putative
	advantages	of	WBE:	(i)	that
its	performance	characteristics	would	be	better	understood	than	those	of	AI;	(ii)
that	it	would	inherit	human	motives;	and	(iii)	that	it	would	result	in	a	slower
takeoff.	Let	us	very	briefly	reflect	on	each.
i	That	it	should	be	easier	to	understand	the	intellectual	performance	characteristics

putative
of	an	emulation	than	of	an	AI	sounds	plausible.	We	have	abundant	experience
with	the	strengths	and	weaknesses	of	human	intelligence	but	no	experience	with
human-level	artificial	intelligence.	However,	to	understand	what	a	snapshot	of	a

digitized	human	intellect	can	and	cannot	do	is	not	the	same	as	to	understand	how
such	an	intellect	will	respond	to	modifications	aimed	at	enhancing	its
performance.	An	artificial	intellect,	by	contrast,	might	be	carefully	designed	to
be	understandable,	in	both	its	static	and	dynamic	dispositions.	So	while	whole
brain	emulation	may	be	more	predictable	in	its	intellectual	performance	than	a
generic	AI	at	a	comparable	stage	of	development,	it	is	unclear	whether	whole
brain	emulation	would	be	dynamically	more	predictable	than	an	AI	engineered
by	competent	safety-conscious	programmers.
ii	As	for	an	emulation	inheriting	the	motivations	of	its	human	template,	this	is	far
from	guaranteed.	Capturing	human	evaluative	dispositions	might	require	a	very
high-fidelity	emulation.	Even	if	some	individual’s	motivations	
were
	perfectly
captured,	it	is	unclear	how	much	safety	would	be	purchased.	Humans	can	be
untrustworthy,	selfish,	and	cruel.	While	templates	would	hopefully	be	selected

were
	perfectly
for	exceptional	virtue,	it	may	be	hard	to	foretell	how	someone	will	act	when
transplanted	into	radically	alien	circumstances,	superhumanly	enhanced	in
intelligence,	and	tempted	with	an	opportunity	for	world	domination.	It	is	true
that	emulations	would	at	least	be	more	likely	to	have	
human-like
	motivations	(as
opposed	to	valuing	only	paperclips	or	discovering	digits	of	pi).	Depending	on
one’s	views	on	human	nature,	this	might	or	might	not	be	reassuring.
22
iii	It	is	not	clear	why	whole	brain	emulation	should	result	in	a	slower	takeoff	than

22
artificial	intelligence.	Perhaps	with	whole	brain	emulation	one	should	expect	less
hardware	overhang,	since	whole	brain	emulation	is	less	computationally	efficient
than	artificial	intelligence	can	be.	Perhaps,	also,	an	AI	system	could	more	easily
absorb	all	available	computing	power	into	one	giant	integrated	intellect,	whereas
whole	brain	emulation	would	forego	quality	superintelligence	and	pull	ahead	of
humanity	only	in	speed	and	size	of	population.	If	whole	brain	emulation	does

lead	to	a	slower	takeoff,	this	could	have	benefits	in	terms	of	alleviating	the
control	problem.	A	slower	takeoff	would	also	make	a	multipolar	outcome	more
likely.	But	whether	a	multipolar	outcome	is	desirable	is	very	doubtful.
	
There	is	another	important	complication	with	the	general	idea	that	getting
whole	brain	emulation	first	is	safer:	the	need	to	cope	with	a	
second	transition
.
Even	if	the	first	form	of	human-level	machine	intelligence	is	emulation-based,	it

.
would	still	remain	feasible	to	develop	artificial	intelligence.	AI	in	its	mature
form	has	important	advantages	over	WBE,	making	AI	the	ultimately	more

powerful	technology.
23
	While	mature	AI	would	render	WBE	obsolete	(except	for
the	special	purpose	of	preserving	individual	human	minds),	the	reverse	does	not
hold.
What	this	means	is	that	if	AI	is	developed	first,	there	might	be	a	single	wave
of	the	intelligence	explosion.	But	if	WBE	is	developed	first,	there	may	be	two
waves:	first,	the	arrival	of	WBE;	and	later,	the	arrival	of	AI.	The	total	existential
risk	along	the	WBE-first	path	is	the	
sum
	of	the	risk	in	the	first	transition	and	the
risk	in	the	second	transition	(conditional	on	having	made	it	through	the	first);	see
Figure	13
.
24
How	much	safer	would	the	AI	transition	be	in	a	WBE	world?	One
consideration	is	that	the	AI	transition	would	be	less	explosive	if	it	occurs	after
some	form	of	machine	intelligence	has	already	been	realized.	Emulations,
running	at	digital	speeds	and	in	numbers	that	might	far	exceed	the	biological
human	population,	would	reduce	the	cognitive	differential,	making	it	easier	for

Figure	13
.
24
emulations	to	control	the	AI.	This	consideration	is	not	too	weighty,	since	the	gap
between	AI	and	WBE	could	still	be	wide.	However,	if	the	emulations	were	not
just	faster	and	more	numerous	but	also	somewhat	qualitatively	smarter	than
biological	humans	(or	at	least	drawn	from	the	top	end	of	the	human	distribution)
then	the	WBE-first	scenario	would	have	advantages	paralleling	those	of	human
cognitive	enhancement,	which	we	discussed	above.
Figure	13
	Artificial	intelligence	or	whole	brain	emulation	first?	In	an	AI-first

Figure	13
scenario,	there	is	one	transition	that	creates	an	existential	risk.	In	a	WBE-first
scenario,	there	are	two	risky	transitions,	first	the	development	of	WBE	and	then
the	development	of	AI.	The	total	existential	risk	along	the	WBE-first	scenario	is
the	sum	of	these.	However,	the	risk	of	an	AI	transition	might	be	lower	if	it

occurs	in	a	world	where	WBE	has	already	been	successfully	introduced.
	
Another	consideration	is	that	the	transition	to	WBE	would	extend	the	lead	of
the	frontrunner.	Consider	a	scenario	in	which	the	frontrunner	has	a	six-month
lead	over	the	closest	follower	in	developing	whole	brain	emulation	technology.
Suppose	that	the	first	emulations	to	be	created	are	cooperative,	safety-focused,
and	patient.	If	they	run	on	fast	hardware,	these	emulations	could	spend
subjective	eons	pondering	how	to	create	safe	AI.	For	example,	if	they	run	at	a
speedup	of	100,000×	and	are	able	to	work	on	the	control	problem	undisturbed
for	six	months	of	sidereal	time,	they	could	hammer	away	at	the	control	problem
for	fifty	millennia	before	facing	competition	from	other	emulations.	Given
sufficient	hardware,	they	could	hasten	their	progress	by	fanning	out	myriad
copies	to	work	independently	on	subproblems.	If	the	frontrunner	uses	its	six-
month	lead	to	form	a	singleton,	it	could	buy	its	emulation	AI-development	team

an	unlimited	amount	of	time	to	work	on	the	control	problem.
25
On	balance,	it	looks	like	the	risk	of	the	AI	transition	would	be	reduced	if
WBE	comes	before	AI.	However,	when	we	combine	the	residual	risk	in	the	AI
transition	with	the	risk	of	an	antecedent	WBE	transition,	it	becomes	very	unclear
how	the	total	existential	risk	along	the	WBE-first	path	stacks	up	against	the	risk
along	the	AI-first	path.	Only	if	one	is	quite	pessimistic	about	biological
humanity’s	ability	to	manage	an	AI	transition—after	taking	into	account	that

25
human	nature	or	civilization	might	have	improved	by	the	time	we	confront	this
challenge—should	the	WBE-first	path	seem	attractive.
To	figure	out	whether	whole	brain	emulation	technology	should	be	promoted,
there	are	some	further	important	points	to	place	in	the	balance.	Most
significantly,	there	is	the	technology	coupling	mentioned	earlier:	a	push	toward
WBE	could	instead	produce	neuromorphic	AI.	This	is	a	reason	against	pushing
for	WBE.
26
	No	doubt,	there	are	
some
	synthetic	AI	designs	that	are	less	safe	than
some

for	WBE.
26
some
some
	neuromorphic	designs.	In	expectation,	however,	it	seems	that
neuromorphic	designs	are	less	safe.	One	ground	for	this	is	that	imitation	can
substitute	for	understanding.	To	build	something	from	the	ground	up	one	must
usually	have	a	reasonably	good	understanding	of	how	the	system	will	work.
Such	understanding	may	not	be	necessary	to	merely	copy	features	of	an	existing
system.	Whole	brain	emulation	relies	on	wholesale	copying	of	biology,	which
may	not	require	a	comprehensive	computational	systems-level	understanding	of

cognition	(though	a	large	amount	of	component-level	understanding	would
undoubtedly	be	needed).	Neuromorphic	AI	may	be	like	whole	brain	emulation	in
this	regard:	it	would	be	
achieved	by	cobbling	together	pieces	plagiarized	from
biology	without	the	engineers	necessarily	having	a	deep	mathematical
understanding	of	how	the	system	works.	But	neuromorphic	AI	would	be	
unlike
whole	brain	emulation	in	another	regard:	it	would	not	have	human	motivations
by	default.
27
	This	consideration	argues	against	pursuing	the	whole	brain
emulation	approach	to	the	extent	that	it	would	likely	produce	neuromorphic	AI.
A	second	point	to	put	in	the	balance	is	that	WBE	is	more	likely	to	give	us
advance	notice	of	its	arrival.	With	AI	it	is	always	possible	that	somebody	will
make	an	unexpected	conceptual	breakthrough.	WBE,	by	contrast,	will	require
many	laborious	precursor	steps—high-throughput	scanning	facilities,	image
processing	software,	detailed	neural	modeling	work.	We	can	therefore	be

confident	that	WBE	is	not	imminent	(not	less	than,	say,	fifteen	or	twenty	years
away).	This	means	that	efforts	to	accelerate	WBE	will	make	a	difference	mainly
in	scenarios	in	which	machine	intelligence	is	developed	comparatively	late.	This
could	make	WBE	investments	attractive	to	somebody	who	wants	the	intelligence
explosion	to	preempt	other	existential	risks	but	is	wary	of	supporting	AI	for	fear
of	triggering	an	intelligence	explosion	prematurely,	before	the	control	problem
has	been	solved.	However,	the	uncertainty	over	the	relevant	timescales	is

probably	currently	too	large	to	enable	this	consideration	to	carry	much	weight.
28
A	strategy	of	promoting	WBE	is	thus	most	attractive	if	(a)	one	is	very
pessimistic	about	humans	solving	the	control	problem	for	AI,	(b)	one	is	not	too
worried	about	neuromorphic	AI,	multipolar	outcomes,	or	the	risks	of	a	second
transition,	(c)	one	thinks	that	the	default	timing	of	WBE	and	AI	is	close,	and	(d)
one	prefers	superintelligence	to	be	developed	neither	very	late	nor	very	early.

28
	
I	fear	the	blog	commenter	“washbash”	may	speak	for	many	when	he	or	she
writes:
I	instinctively	think	go	faster.	Not	because	I	think	this	is	better	for	the
world.	Why	should	I	care	about	the	world	when	I	am	dead	and	gone?	I	want
it	to	go	fast,	damn	it!	This	increases	the	chance	I	have	of	experiencing	a

more	technologically	advanced	future.
29
	
From	the	person-affecting	standpoint,	we	have	greater	reason	to	rush	forward
with	all	manner	of	radical	technologies	that	could	pose	existential	risks.	This	is
because	the	default	outcome	is	that	almost	everyone	who	now	exists	is	dead
within	a	century.
The	case	for	rushing	is	especially	strong	with	regard	to	technologies	that	could
extend	our	lives	and	thereby	increase	the	expected	fraction	of	the	currently
existing	population	that	may	still	be	around	for	the	intelligence	explosion.	If	the
machine	intelligence	revolution	goes	well,	the	resulting	superintelligence	could
almost	certainly	devise	means	to	indefinitely	prolong	the	lives	of	the	then	still-
existing	humans,	not	only	keeping	them	alive	but	restoring	them	to	health	and
youthful	vigor,	and	enhancing	their	capacities	well	beyond	what	we	currently
think	of	as	the	human	range;	or	helping	them	shuffle	off	their	mortal	coils
altogether	by	
uploading	their	minds	to	a	digital	substrate	and	endowing	their

liberated	spirits	with	exquisitely	good-feeling	virtual	embodiments.	With	regard
to	technologies	that	do	not	promise	to	save	lives,	the	case	for	rushing	is	weaker,
though	perhaps	still	sufficiently	supported	by	the	hope	of	raised	standards	of
living.
30
The	same	line	of	reasoning	makes	the	person-affecting	perspective	favor
many	risky	technological	innovations	that	promise	to	hasten	the	onset	of	the
intelligence	explosion,	even	when	those	innovations	are	disfavored	in	the

living.
30
impersonal	perspective.	Such	innovations	could	shorten	the	wolf	hours	during
which	we	individually	must	hang	on	to	our	perch	if	we	are	to	live	to	see	the
daybreak	of	the	posthuman	age.	From	the	person-affecting	standpoint,	faster
hardware	progress	thus	seems	desirable,	as	does	faster	progress	toward	WBE.
Any	adverse	effect	on	existential	risk	is	probably	outweighed	by	the	personal
benefit	of	an	increased	chance	of	the	intelligence	explosion	happening	in	the
lifetime	of	currently	existing	people.
31

One	important	parameter	is	the	degree	to	which	the	world	will	manage	to
coordinate	and	collaborate	in	the	development	of	machine	intelligence.
Collaboration	would	bring	many	benefits.	Let	us	take	a	look	at	how	this
parameter	might	affect	the	outcome	and	what	levers	we	might	have	for
increasing	the	extent	and	intensity	of	collaboration.
	
A	race	dynamic	exists	when	one	project	fears	being	overtaken	by	another.	This
does	not	require	the	actual	existence	of	multiple	projects.	A	situation	with	only
one	project	could	exhibit	a	race	dynamic	if	that	project	is	unaware	of	its	lack	of
competitors.	The	Allies	would	probably	not	have	developed	the	atomic	bomb	as
quickly	as	they	did	had	they	not	believed	(erroneously)	that	the	Germans	might
be	close	to	the	same	goal.
The	severity	of	a	race	dynamic	(that	is,	the	extent	to	which	competitors
prioritize	speed	over	safety)	depends	on	several	factors,	such	as	the	closeness	of

the	race,	the	relative	importance	of	capability	and	luck,	the	number	of
competitors,	whether	competing	teams	are	pursuing	different	approaches,	and
the	degree	to	which	projects	share	the	same	aims.	Competitors’	beliefs	about
these	factors	are	also	relevant.	(See	
Box	13
.)
In	the	development	of	machine	superintelligence,	it	seems	likely	that	there
will	be	at	least	a	mild	race	dynamic,	and	it	is	possible	that	there	will	be	a	severe
race	dynamic.	The	race	dynamic	has	important	consequences	for	how	we	should

Box	13
.)
think	about	the	strategic	challenge	posed	by	the	possibility	of	an	intelligence
The	race	dynamic	could	spur	projects	to	move	faster	toward	superintelligence
while	reducing	investment	in	solving	the	control	problem.	Additional
detrimental	effects	of	the	race	dynamic	are	also	possible,	such	as	direct
hostilities	between	competitors.	Suppose	that	two	nations	are	racing	to	develop
the	first	superintelligence,	and	that	one	of	them	is	seen	to	be	pulling	ahead.	In	a

winner-takes-all	situation,	a	lagging	project	might	be	tempted	to	launch	a
desperate	strike	against	its	rival	rather	than	passively	await	defeat.	Anticipating
this	possibility,	the	frontrunner	might	be	tempted	to	strike	preemptively.	If	the
antagonists	are	powerful	states,	the	clash	could	be	bloody.
34
	(A	“surgical	strike”

against	the	rival’s	AI	project	might	risk	triggering	a	larger	confrontation	and
might	in	any	case	not	be	feasible	if	the	host	country	has	taken	precautions.
35
)
Box	13	A	risk-race	to	the	bottom
	
Consider	a	hypothetical	AI	arms	race	in	which	several	teams	compete	to	develop
32
	Each	team	decides	how	much	to	invest	in	safety—knowing
that	resources	spent	on	developing	safety	precautions	are	resources	not	spent	on
developing	the	AI.	Absent	a	deal	between	all	the	competitors	(which	might	be
stymied	by	bargaining	or	enforcement	difficulties),	there	might	then	be	a	risk-
race	to	the	bottom,	driving	each	team	to	take	only	a	minimum	of	precautions.
One	can	model	each	team’s	performance	as	a	function	of	its	capability
(measuring	its	raw	ability	and	luck)	and	a	penalty	term	corresponding	to	the	cost
of	its	safety	precautions.	The	team	with	the	highest	performance	builds	the	first
AI.	The	riskiness	of	that	AI	is	determined	by	how	much	its	creators	invested	in

safety.	In	the	worst-case	scenario,	all	teams	have	equal	levels	of	capability.	The
winner	is	then	determined	exclusively	by	investment	in	safety:	the	team	that	took
the	fewest	safety	precautions	wins.	The	Nash	equilibrium	for	this	game	is	for
every	team	to	spend	nothing	on	safety.	In	the	real	world,	such	a	situation	might
arise	via	a	
risk	ratchet
:	some	team,	fearful	of	falling	behind,	increments	its	risk-
taking	to	catch	up	with	its	competitors—who	respond	in	kind,	until	the
maximum	level	of	risk	is	reached.
Capability	versus	risk

arise	via	a	
	
The	situation	changes	when	there	are	variations	in	capability.	As	variations	in
capability	become	more	important	relative	to	the	cost	of	safety	precautions,	the
risk	ratchet	weakens:	there	is	less	incentive	to	incur	an	extra	bit	of	risk	if	doing
so	is	unlikely	to	change	the	order	of	the	race.	This	is	illustrated	under	various
scenarios	in	
Figure	14
,	which	plots	how	the	riskiness	of	the	AI	depends	on	the
importance	of	capability.	Safety	investment	ranges	from	1	(resulting	in	perfectly
safe	AI)	to	0	(completely	unsafe	AI).	The	
x
-axis	represents	the	relative

Figure	14
x
importance	of	capability	versus	safety	investment	in	determining	the	speed	of	a

team’s	progress	toward	AI.	(At	0.5,	the	safety	investment	level	is	twice	are
important	as	capability;	at	1,	the	two	are	equal;	at	2,	capability	is	twice	as
important	as	safety	level;	and	so	forth.)	The	
y
-axis	represents	the	level	of	AI	risk
(the	expected	fraction	of	their	maximum	utility	that	the	winner	of	the	race	gets).
Figure	14
	Risk	levels	in	AI	technology	races.	Levels	of	risk	of	dangerous	AI	in
a	simple	model	of	a	technology	race	involving	either	(a)	two	teams	or	(b)	five
teams,	plotted	against	the	relative	importance	of	capability	(as	opposed	to
investment	in	safety)	in	determining	which	project	wins	the	race.	The	graphs
show	three	information-level	scenarios:	no	capability	information	(straight),
private	capability	information	(dashed),	and	full	capability	information	(dotted).
	
We	see	that,	under	all	scenarios,	the	dangerousness	of	the	resultant	AI	is
maximal	when	capability	plays	no	role,	gradually	decreasing	as	capability	grows
in	importance.
Compatible	goals

Another	way	of	reducing	the	risk	is	by	giving	teams	more	of	a	stake	in	each
other’s	success.	If	competitors	are	convinced	that	coming	second	means	the	total
loss	of	everything	they	care	about,	they	will	take	whatever	risk	necessary	to
bypass	their	rivals.	Conversely,	teams	will	invest	more	in	safety	if	less	depends
on	winning	the	race.	This	suggests	that	we	should	encourage	various	forms	of
cross-investment.
The	number	of	competitors

The	greater	the	number	of	competing	teams,	the	more	dangerous	the	race
becomes:	each	team,	having	less	chance	of	coming	first,	is	more	willing	to	throw
caution	to	the	wind.	This	can	be	seen	by	contrasting	
Figure	14
a	(two	teams)	with
Figure	14
b	(five	teams).	In	every	scenario,	more	competitors	means	more	risk.
Risk	would	be	reduced	if	teams	coalesce	into	a	smaller	number	of	competing
coalitions.
The	curse	of	too	much	information
	
Is	it	good	if	teams	know	about	their	positions	in	the	race	(knowing	their
capability	scores,	for	instance)?	Here,	opposing	factors	are	at	play.	It	is	desirable
that	a	leader	knows	it	is	leading	(so	that	it	knows	it	has	some	margin	for
additional	safety	precautions).	Yet	it	is	undesirable	that	a	laggard	knows	it	has
fallen	behind	(since	this	would	confirm	that	it	must	cut	back	on	safety	to	have
any	hope	of	catching	up).	While	intuitively	it	may	seem	this	trade-off	could	go
either	way,	the	models	are	unequivocal:	information	is	(in	expectation)	bad.
33
Figures	14
a	and	
14

33
Figures	14
a	and	
14
b	each	plot	three	scenarios:	the	straight	lines	correspond	to
situations	in	which	no	team	knows	any	of	the	capability	scores,	its	own	included.
The	dashed	lines	show	situations	where	each	team	knows	its	own	capability
only.	(In	those	situations,	a	team	takes	extra	risk	only	if	its	capability	is	low.)
And	the	dotted	lines	show	what	happens	when	all	teams	know	each	other’s
capabilities.	(They	take	extra	risks	if	their	capability	scores	are	close	to	one

33
Figures	14
a	and	
14
another.)	With	each	increase	in	information	level,	the	race	dynamic	becomes
worse.
	
Scenarios	in	which	the	rival	developers	are	not	states	but	smaller	entities,	such
as	corporate	labs	or	academic	teams,	would	probably	feature	much	less	direct
destruction	from	conflict.	Yet	the	overall	consequences	of	competition	may	be
almost	as	bad.	This	is	because	the	main	part	of	the	expected	harm	from
competition	stems	not	from	the	smashup	of	battle	but	from	the	downgrade	of

worse.
	
precaution.	A	race	dynamic	would,	as	we	saw,	reduce	investment	in	safety;	and
conflict,	even	if	nonviolent,	would	tend	to	scotch	opportunities	for	collaboration,
since	projects	would	be	less	likely	to	share	ideas	for	solving	the	control	problem

in	a	climate	of	hostility	and	mistrust.
36
	
Collaboration	thus	offers	many	benefits.	It	reduces	the	haste	in	developing
machine	intelligence.	It	allows	for	greater	investment	in	safety.	It	avoids	violent
conflicts.	And	it	facilitates	the	sharing	of	ideas	about	how	to	solve	the	control
problem.	To	these	benefits	we	can	add	another:	collaboration	would	tend	to
produce	outcomes	in	which	the	fruits	of	a	successfully	controlled	intelligence
explosion	get	distributed	more	equitably.
That	broader	collaboration	should	result	in	wider	sharing	of	gains	is	not
axiomatic.	In	principle,	a	small	project	run	by	an	altruist	could	lead	to	an
outcome	where	the	benefits	are	shared	evenly	or	equitably	among	all	morally
considerable	beings.	Nevertheless,	there	are	several	reasons	to	suppose	that
broader	collaborations,	involving	a	greater	number	of	sponsors,	are	(in
expectation)	distributionally	superior.	One	such	reason	is	that	sponsors

presumably	prefer	an	outcome	in	which	they	themselves	get	(at	least)	their	fair
share.	A	broad	collaboration	then	means	that	relatively	many	individuals	get	at
least	their	fair	share,	assuming	the	project	is	successful.	Another	reason	is	that	a
broad	collaboration	also	seems	likelier	to	benefit	people	outside	the
collaboration.	A	broader	collaboration	contains	more	members,	so	more
outsiders	would	have	personal	ties	to	somebody	on	the	inside	looking	out	for

their	interests.	A	broader	collaboration	is	also	more	likely	to	include	at	least
some	altruist	who	wants	to	benefit	everyone.	Furthermore,	a	broader
collaboration	is	more	likely	to	operate	under	public	oversight,	which	might
reduce	the	risk	of	the	entire	pie	being	captured	by	a	clique	of	programmers	or
private	investors.
37
	Note	also	that	the	larger	the	successful	collaboration	is,	the
lower	the	costs	to	it	of	extending	the	benefits	to	all	outsiders.	(For	instance,	if
90%	of	all	people	were	already	inside	the	collaboration,	it	would	
cost	them	no

37
more	than	10%	of	their	holdings	to	bring	all	outsiders	up	to	their	own	level.)
It	is	thus	plausible	that	broader	collaborations	would	tend	to	lead	to	a	wider
distribution	of	the	gains	(though	
some
	projects	with	few	sponsors	might	also
have	distributionally	excellent	aims).	But	why	is	a	wide	distribution	of	gains
desirable?

There	are	both	moral	and	prudential	reasons	for	favoring	outcomes	in	which
everybody	gets	a	share	of	the	bounty.	We	will	not	say	much	about	the	moral
case,	except	to	note	that	it	need	not	rest	on	any	egalitarian	principle.	The	case
might	be	made,	for	example,	on	grounds	of	fairness.	A	project	that	creates
machine	superintelligence	imposes	a	global	risk	externality.	Everybody	on	the
planet	is	placed	in	jeopardy,	including	those	who	do	not	consent	to	having	their
own	lives	and	those	of	their	family	imperiled	in	this	way.	Since	everybody
shares	the	risk,	it	would	seem	to	be	a	minimal	requirement	of	fairness	that
everybody	also	gets	a	share	of	the	upside.
The	fact	that	the	total	(expected)	amount	of	good	seems	greater	in
collaboration	scenarios	is	another	important	reason	such	scenarios	are	morally
preferable.
The	prudential	case	for	favoring	a	wide	distribution	of	gains	is	two-pronged.
One	prong	is	that	wide	distribution	should	promote	collaboration,	thereby

mitigating	the	negative	consequences	of	the	race	dynamic.	There	is	less
incentive	to	fight	over	who	gets	to	build	the	first	superintelligence	if	everybody
stands	to	benefit	equally	from	any	project’s	success.	The	sponsors	of	a	particular
project	might	also	benefit	from	credibly	signaling	their	commitment	to
distributing	the	spoils	universally,	a	certifiably	altruistic	project	being	likely	to
attract	more	supporters	and	fewer	enemies.
38
The	other	prong	of	the	prudential	case	for	favoring	a	wide	distribution	of

38
gains	has	to	do	with	whether	agents	are	risk-averse	or	have	utility	functions	that
are	sublinear	in	resources.	The	central	fact	here	is	the	enormousness	of	the
potential	resource	pie.	Assuming	the	observable	universe	is	as	uninhabited	as	it
looks,	it	contains	more	than	one	vacant	galaxy	for	each	human	being	alive.	Most
people	would	much	rather	have	certain	access	to	one	galaxy’s	worth	of	resources
than	a	lottery	ticket	offering	a	one-in-a-billion	chance	of	owning	a	billion
galaxies.
39
	Given	the	astronomical	size	of	humanity’s	cosmic	endowment,	it

galaxies.
39
seems	that	self-interest	should	generally	favor	deals	that	would	guarantee	each
person	a	share,	even	if	each	share	corresponded	to	a	small	fraction	of	the	total.
The	important	thing,	when	such	an	extravagant	bonanza	is	in	the	offing,	is	to	not
be	left	out	in	the	cold.
This	argument	from	the	enormousness	of	the	resource	pie	presupposes	that
preferences	are	resource-satiable.
40
	That	supposition	does	not	necessarily	hold.
For	instance,	several	prominent	ethical	theories—including	especially

aggregative	consequentialist	theories—correspond	to	utility	functions	that	are
risk-neutral	and	linear	in	resources.	A	billion	galaxies	could	be	used	to	create	a
billion	times	more	happy	lives	than	a	single	galaxy.	They	are	thus,	to	a
utilitarian,	worth	a	billion	times	as	much.
41
	Ordinary	selfish	human	preference
functions,	however,	appear	to	be	relatively	resource-satiable.
This	last	statement	must	be	flanked	by	two	important	qualifications.	The	first
is	that	many	people	care	about	rank.	If	multiple	agents	each	wants	to	top	the
Forbes	rich	list,	then	no	resource	pie	is	large	enough	to	give	everybody	full
satisfaction.
The	second	qualification	is	that	the	post-transition	technology	base	would
enable	material	resources	to	be	converted	into	an	unprecedented	range	of
products,	including	some	goods	that	are	not	currently	available	at	any	price	even
though	they	are	highly	valued	by	many	humans.	A	billionaire	does	not	live	a
thousand	times	longer	than	a	millionaire.	In	the	era	of	digital	minds,	however,

the	billionaire	could	afford	a	thousandfold	more	computing	power	and	could
thus	enjoy	a	thousandfold	longer	subjective	lifespan.	Mental	capacity,	likewise,
could	be	for	sale.	In	such	circumstances,	with	economic	capital	convertible	into
vital	goods	at	a	constant	rate	even	for	great	levels	of	wealth,	unbounded	greed
would	make	more	sense	than	it	does	in	today’s	world	where	the	affluent	(those
among	them	lacking	a	philanthropic	heart)	are	reduced	to	spending	their	riches

on	airplanes,	boats,	art	collections,	or	a	fourth	and	a	fifth	residence.
Does	this	mean	that	an	egoist	should	be	risk-neutral	with	respect	to	his	or	her
post-transition	resource	endowment?	Not	quite.	Physical	resources	may	not	be
convertible	into	lifespan	or	mental	performance	at	arbitrary	scales.	If	a	life	must
be	lived	sequentially,	so	that	observer	moments	can	remember	earlier	events	and
be	affected	by	prior	choices,	then	the	life	of	a	digital	mind	cannot	be	extended
arbitrarily	without	utilizing	an	increasing	number	of	
sequential

	computational
operations.	But	physics	limits	the	extent	to	which	resources	can	be	transformed
into	sequential	computations.
42
	The	limits	on	sequential	computation	may	also
constrain	some	aspects	of	cognitive	performance	to	scale	radically	sublinearly
beyond	a	relatively	modest	resource	endowment.	Furthermore,	it	is	not	obvious
that	an	egoist	would	or	should	be	risk-neutral	even	with	regard	to	highly
normatively	relevant	outcome	metrics	such	as	number	of	quality-adjusted

42
subjective	life	years.	If	offered	the	choice	between	an	extra	2,000	years	of	life
for	certain	and	a	one-in-ten	chance	of	an	extra	30,000	years	of	life,	I	think	most
people	would	select	the	former	(even	under	the	stipulation	that	each	life	year

would	be	of	equal	quality).
43
In	reality,	the	prudential	case	for	favoring	a	wide	distribution	of	gains	is
presumably	subject-relative	and	situation-dependent.	Yet,	on	the	whole,	people
would	be	more	likely	to	get	(almost	all	of)	what	they	want	if	a	way	is	found	to
achieve	a	wide	distribution—and	this	holds	even	before	taking	into	account	that
a	commitment	to	a	wider	distribution	would	tend	to	foster	collaboration	and
thereby	increase	the	chances	of	avoiding	existential	catastrophe.	Favoring	a
broad	distribution,	therefore,	appears	to	be	not	only	morally	mandated	but	also
prudentially	advisable.
There	is	a	further	set	of	consequences	to	collaboration	that	should	be	given	at
least	some	shrift:	the	possibility	that	pre-transition	collaboration	influences	the
level	of	post-transition	collaboration.	Assume	humanity	solves	the	control
problem.	(If	the	control	problem	is	not	solved,	it	may	scarcely	matter	how	much
collaboration	there	is	post	transition.)	There	are	two	cases	to	consider.	The	first

is	that	the	intelligence	explosion	does	
not
	create	a	winner-takes-all	dynamic
(presumably	because	the	takeoff	is	relatively	slow).	In	this	case	it	is	plausible
that	if	
pre-transition	collaboration	has	any	systematic	effect	on	post-transition
collaboration,	it	has	a	positive	effect,	tending	to	promote	subsequent
collaboration.	The	original	collaborative	relationships	may	endure	and	continue
beyond	the	transition;	also,	pre-transition	collaboration	may	offer	more

not
that	if	
opportunity	for	people	to	steer	developments	in	desirable	(and,	presumably,
more	collaborative)	post-transition	directions.
The	second	case	is	that	the	nature	of	the	intelligence	explosion	does	encourage
a	winner-takes-all	dynamic	(presumably	because	the	takeoff	is	relatively	fast).	In
this	case,	if	there	is	no	extensive	collaboration	before	the	takeoff,	a	singleton	is
likely	to	emerge—a	single	project	would	undergo	the	transition	alone,	at	some
point	obtaining	a	decisive	strategic	advantage	combined	with	superintelligence.

A	singleton,	by	definition,	is	a	highly	collaborative	social	order.
44
	The	absence
of	extensive	collaboration	pre-transition	would	thus	lead	to	an	extreme	degree	of
collaboration	post-transition.	By	contrast,	a	somewhat	higher	level	of
collaboration	in	the	run-up	to	the	intelligence	explosion	opens	up	a	wider	variety
of	possible	outcomes.	Collaborating	projects	could	synchronize	their	ascent	to
ensure	they	transition	in	tandem	without	any	of	them	getting	a	decisive	strategic
advantage.	Or	different	sponsor	groups	might	merge	their	efforts	into	a	single

44
project,	while	refusing	to	give	that	project	a	mandate	to	form	a	singleton.	For
example,	one	could	imagine	a	consortium	of	nations	forming	a	joint	scientific

project	to	develop	machine	superintelligence,	yet	not	authorizing	this	project	to
evolve	into	anything	like	a	supercharged	United	Nations,	electing	instead	to
maintain	the	factious	world	order	that	existed	before.
Particularly	in	the	case	of	a	fast	takeoff,	therefore,	the	possibility	exists	that
greater	pre-transition	collaboration	would	result	in	less	post-transition
collaboration.	However,	to	the	extent	that	collaborating	entities	are	able	to	shape
the	outcome,	they	may	allow	the	emergence	or	continuation	of	non-collaboration
only	if	they	foresee	that	no	catastrophic	consequences	would	follow	from	post-
transition	factiousness.	Scenarios	in	which	pre-transition	collaboration	leads	to
reduced	post-transition	collaboration	may	therefore	mostly	be	ones	in	which
reduced	post-transition	collaboration	is	innocuous.
In	general,	greater	post-transition	collaboration	appears	desirable.	It	would
reduce	the	risk	of	dystopian	dynamics	in	which	economic	competition	and	a

rapidly	expanding	population	lead	to	a	Malthusian	condition,	or	in	which
evolutionary	selection	erodes	human	values	and	selects	for	non-eudaemonic
forms,	or	in	which	rival	powers	suffer	other	coordination	failures	such	as	wars
and	technology	races.	The	last	of	these	issues,	the	prospect	of	technology	races,
may	be	particularly	problematic	if	the	transition	is	to	an	intermediary	form	of
machine	intelligence	(whole	brain	emulation)	since	it	would	create	a	new	race
dynamic	that	would	harm	the	chances	of	the	control	problem	being	solved	for

the	subsequent	second	transition	to	a	more	advanced	form	of	machine
intelligence	(artificial	intelligence).
We	described	earlier	how	collaboration	can	reduce	conflict	in	the	run-up	to	the
intelligence	explosion,	increasing	the	chances	that	the	control	problem	will	
be
solved,	and	improve	both	the	moral	legitimacy	and	the	prudential	desirability	of
the	resulting	resource	allocation.	To	these	benefits	of	collaboration	it	may	thus
be	possible	to	add	one	more:	that	broader	collaboration	pre-transition	could	help

be
with	important	coordination	problems	in	the	post-transition	era.
	
Collaboration	can	take	different	forms	depending	on	the	scale	of	the
collaborating	entities.	At	a	small	scale,	individual	AI	teams	who	believe
themselves	to	be	in	competition	with	one	another	could	choose	to	pool	their

efforts.
45
	Corporations	could	merge	or	cross-invest.	At	a	larger	scale,	states
could	join	in	a	big	international	project.	There	are	precedents	to	large-scale
international	collaboration	in	science	and	technology	(such	as	CERN,	the
Human	Genome	Project,	and	the	International	Space	Station),	but	an
international	project	to	develop	safe	superintelligence	would	pose	a	different
order	of	challenge	because	of	the	security	implications	of	the	work.	It	would
have	to	be	constituted	not	as	an	open	academic	collaboration	but	as	an	extremely
tightly	controlled	joint	enterprise.	Perhaps	the	scientists	involved	would	have	to
be	physically	isolated	and	prevented	from	communicating	with	the	rest	of	the
world	for	the	duration	of	the	project,	except	through	a	single	carefully	vetted
communication	channel.	The	required	level	of	security	might	be	nearly
unattainable	at	present,	but	advances	in	lie	detection	and	surveillance	technology
could	make	it	feasible	later	this	century.	It	is	also	worth	bearing	in	mind	that

broad	collaboration	does	not	necessarily	mean	that	large	numbers	of	researchers
would	be	involved	in	the	project;	it	simply	means	that	many	people	would	have
a	say	in	the	project’s	aims.	In	principle,	a	project	could	involve	a	maximally
broad	collaboration	comprising	all	of	humanity	as	sponsors	(represented,	let	us
say,	by	the	General	Assembly	of	the	United	Nations),	yet	employ	only	a	single
scientist	to	carry	out	the	work.
46
There	is	a	reason	for	starting	collaboration	as	early	as	possible,	namely	to	take

46
advantage	of	the	veil	of	ignorance	that	hides	from	our	view	any	specific
information	about	which	individual	project	will	get	to	superintelligence	first.
The	closer	to	the	finishing	line	we	get,	the	less	uncertainty	will	remain	about	the
relative	chances	of	competing	projects;	and	the	harder	it	may	consequently	be	to
make	a	case	based	on	the	self-interest	of	the	frontrunner	to	join	a	collaborative
project	that	would	distribute	the	benefits	to	all	of	humanity.	On	the	other	hand,	it
also	looks	hard	to	establish	a	formal	collaboration	of	worldwide	scope	before	the

prospect	of	superintelligence	has	become	much	more	widely	recognized	than	it
currently	is	and	before	there	is	a	clearly	visible	road	leading	to	the	creation	of
machine	superintelligence.	Moreover,	to	the	extent	that	collaboration	would
promote	progress	along	that	road,	it	may	actually	be	counterproductive	in	terms
of	safety,	as	discussed	earlier.
The	ideal	form	of	collaboration	for	the	present	may	therefore	be	one	that	does
not	initially	require	specific	formalized	agreements	and	that	does	not	expedite

advances	in	machine	intelligence.	One	proposal	that	fits	these	criteria	is	that	we
propound	an	appropriate	moral	norm,	expressing	our	commitment	to	the	idea
that	superintelligence	should	be	for	the	common	good.	Such	a	norm	could	be

formulated	as	follows:
The	common	good	principle
Superintelligence	should	be	developed	only	for	the	benefit	of	all	of
humanity	and	in	the	service	of	widely	shared	ethical	ideals.
47
	
Establishing	from	an	early	stage	that	the	immense	potential	of	superintelligence
belongs	to	all	of	humanity	will	give	more	time	for	such	a	norm	to	become
entrenched.
The	common	good	principle	does	not	preclude	commercial	incentives	for
individuals	or	firms	active	in	related	areas.	For	example,	a	firm	might	satisfy	the
call	for	universal	sharing	of	the	benefits	of	superintelligence	by	adopting	a
“windfall	clause”	to	the	effect	that	all	profits	up	to	some	very	high	ceiling	(say,	a
trillion	dollars	annually)	would	be	distributed	in	the	ordinary	way	to	the	firm’s
shareholders	and	other	legal	claimants,	and	that	only	profits	in	excess	of	the
threshold	would	be	distributed	to	all	of	humanity	evenly	(or	otherwise	according
to	universal	moral	criteria).	Adopting	such	a	windfall	clause	should	be

substantially	costless,	any	given	firm	being	extremely	unlikely	ever	to	exceed
the	stratospheric	profit	threshold	(and	such	low-probability	scenarios	ordinarily
playing	no	role	in	the	decisions	of	the	firm’s	managers	and	investors).	Yet	its
widespread	adoption	would	give	humankind	a	valuable	guarantee	(insofar	as	the
commitments	could	be	trusted)	that	if	ever	some	private	enterprise	
were
	to	hit
the	jackpot	with	the	intelligence	explosion,	everybody	would	share	in	most	of

were
	to	hit
the	benefits.	The	same	idea	could	be	applied	to	entities	other	than	firms.	For
example,	states	could	agree	that	if	ever	any	one	state’s	GDP	exceeds	some	very
high	fraction	(say,	90%)	of	world	GDP,	the	overshoot	should	be	distributed
evenly	to	all.
48
The	common	good	principle	(and	particular	instantiations,	such	as	windfall
clauses)	could	be	adopted	initially	as	a	voluntary	moral	commitment	by
responsible	individuals	and	organizations	that	are	active	in	areas	related	to

48
machine	intelligence.	Later,	it	could	be	endorsed	by	a	wider	set	of	entities	and
enacted	into	law	and	treaty.	A	vague	formulation,	such	as	the	one	given	here,
may	serve	well	as	a	starting	point;	but	it	would	ultimately	need	to	be	sharpened
into	a	set	of	specific	verifiable	requirements.

CHAPTER	15
Crunch	time
	
We	find	ourselves	in	a	thicket	of	strategic	complexity,	surrounded	by	a
dense	mist	of	uncertainty.	Though	many	considerations	have	been
discerned,	their	details	and	interrelationships	remain	unclear	and	iffy—and
there	might	be	other	factors	we	have	not	even	thought	of	yet.	What	are	we
to	do	in	this	predicament?
	
A	colleague	of	mine	likes	to	point	out	that	a	Fields	Medal	(the	highest	honor	in
mathematics)	indicates	two	things	about	the	recipient:	that	he	was	capable	of
accomplishing	something	important,	and	that	he	didn’t.	Though	harsh,	the
remark	hints	at	a	truth.
Think	of	a	“discovery”	as	an	act	that	moves	the	arrival	of	information	from	a
later	point	in	time	to	an	earlier	time.	The	discovery’s	value	does	not	equal	the
value	of	the	information	discovered	but	rather	the	value	of	having	the
information	available	earlier	than	it	otherwise	would	have	been.	A	scientist	or	a
mathematician	may	show	great	skill	by	being	the	first	to	find	a	solution	that	has

eluded	many	others;	yet	if	the	problem	would	soon	have	been	solved	anyway,
then	the	work	probably	has	not	much	benefited	the	world.	There	
are
	cases	in
which	having	a	solution	even	slightly	sooner	is	immensely	valuable,	but	this	is
most	plausible	when	the	solution	is	immediately	put	to	use,	either	being
deployed	for	some	practical	end	or	serving	as	a	foundation	to	further	theoretical
work.	And	in	the	latter	case,	where	a	solution	is	immediately	used	only	in	the

are
	cases	in
sense	of	serving	as	a	building	block	for	further	theorizing,	there	is	great	value	in
obtaining	a	solution	slightly	sooner	only	if	the	further	work	it	enables	is	itself
both	important	and	urgent.
1
The	question,	then,	is	not	whether	the	result	discovered	by	the	Fields	Medalist

is	in	itself	“important”	(whether	instrumentally	or	for	knowledge’s	own	sake).
Rather,	the	question	is	whether	it	was	important	that	the	medalist	enabled	the
publication	of	the	result	to	occur	at	an	earlier	date.	The	value	of	this	temporal
transport	should	be	compared	to	the	value	that	a	world-class	mathematical	mind
could	have	generated	by	working	on	something	else.	At	least	in	some	cases,	the
Fields	Medal	might	indicate	a	life	spent	solving	the	wrong	problem—for
instance,	a	problem	whose	allure	consisted	primarily	in	being	famously	difficult
to	solve.
Similar	barbs	could	be	directed	at	other	fields,	such	as	academic	philosophy.
Philosophy	covers	some	problems	that	are	relevant	to	existential	risk	mitigation
—we	encountered	several	in	this	book.	Yet	there	are	also	subfields	within
philosophy	that	have	no	apparent	link	to	existential	risk	or	indeed	any	practical
concern.	As	with	pure	mathematics,	some	of	the	problems	that	philosophy

to	solve.
studies	might	be	regarded	as	intrinsically	important,	in	the	sense	that	humans
have	reason	to	care	about	them	independently	of	any	practical	application.	The
fundamental	nature	of	reality,	for	instance,	might	be	worth	knowing	about,	for	its
own	sake.	The	world	would	arguably	be	less	glorious	if	nobody	studied
metaphysics,	cosmology,	or	string	theory.	However,	the	dawning	prospect	of	an
intelligence	explosion	shines	a	new	light	on	this	ancient	quest	for	wisdom.
The	outlook	now	suggests	that	philosophic	progress	can	be	maximized	via	an

indirect	path	rather	than	by	immediate	philosophizing.	One	of	the	many	tasks	on
which	superintelligence	(or	even	just	moderately	enhanced	human	intelligence)
would	outperform	the	current	cast	of	thinkers	is	in	answering	fundamental
questions	in	science	and	philosophy.	This	reflection	suggests	a	strategy	of
deferred	gratification.	We	could	postpone	work	on	some	of	the	eternal	questions
for	a	little	while,	delegating	that	task	to	our	hopefully	more	competent
successors—in	order	to	focus	our	own	attention	on	a	more	pressing	challenge:

increasing	the	chance	that	we	will	actually	have	competent	successors.	This
would	be	high-impact	philosophy	and	high-impact	mathematics.
2
	
We	thus	want	to	focus	on	problems	that	are	not	only	important	but	urgent	in	the
sense	that	their	solutions	are	needed	prior	to	the	intelligence	explosion.	We
should	also	take	heed	not	to	work	on	problems	that	are	negative-value	(such	that

solving	them	is	harmful).	Some	technical	problems	in	the	field	of	artificial
intelligence,	for	instance,	might	be	negative-value	inasmuch	as	their	solution
would	speed	the	development	of	machine	intelligence	without	doing	as	much	to
expedite	the	development	of	control	methods	that	could	render	the	machine
intelligence	revolution	survivable	and	beneficial.
It	can	be	hard	to	identify	problems	that	are	both	urgent	and	important	and	are
such	that	we	can	confidently	take	them	to	be	positive-value.	The	strategic
uncertainty	surrounding	existential	risk	mitigation	means	that	we	must	worry
that	even	well-intentioned	interventions	may	turn	out	to	be	not	only
unproductive	but	counterproductive.	To	limit	the	risk	of	doing	something
actively	harmful	or	morally	wrong,	we	should	prefer	to	work	on	problems	that
seem	
robustly	positive-value
	(i.e.,	whose	solution	would	make	a	positive
contribution	across	a	wide	range	
of	scenarios)	and	to	employ	means	that	are

seem	
robustly	justifiable	(i.e.,	acceptable	from	a	wide	range	of	moral	views).
There	is	a	further	desideratum	to	consider	in	selecting	which	problems	to
prioritize.	We	want	to	work	on	problems	that	are	
elastic
	to	our	efforts	at	solving
them.	Highly	elastic	problems	are	those	that	can	be	solved	much	faster,	or	solved
to	a	much	greater	extent,	given	one	extra	unit	of	effort.	Encouraging	more
kindness	in	the	world	is	an	important	and	urgent	problem—one,	moreover,	that
seems	quite	robustly	positive-value:	yet	absent	a	breakthrough	idea	for	how	to

elastic
go	about	it,	probably	a	problem	of	quite	low	elasticity.	Achieving	world	peace,
similarly,	would	be	highly	desirable;	but	considering	the	numerous	efforts
already	targeting	that	problem,	and	the	formidable	obstacles	arrayed	against	a
quick	solution,	it	seems	unlikely	that	the	contributions	of	a	few	extra	individuals
would	make	a	large	difference.
To	reduce	the	risks	of	the	machine	intelligence	revolution,	we	will	propose
two	objectives	that	appear	to	best	meet	all	those	desiderata:	strategic	analysis

and	capacity-building.	We	can	be	relatively	confident	about	the	sign	of	these
parameters—more	strategic	insight	and	more	capacity	being	better.	Furthermore,
the	parameters	are	elastic:	a	small	extra	investment	can	make	a	relatively	large
difference.	Gaining	insight	and	capacity	is	also	urgent	because	early	boosts	to
these	parameters	may	compound,	making	subsequent	efforts	more	effective.	In
addition	to	these	two	broad	objectives,	we	will	point	to	a	few	other	potentially
worthwhile	aims	for	initiatives.

Against	a	backdrop	of	perplexity	and	uncertainty,	analysis	stands	out	as	being	of
particularly	high	expected	value.
3
	Illumination	of	our	strategic	situation	would
help	us	target	subsequent	interventions	more	effectively.	Strategic	analysis	is
especially	needful	when	we	are	radically	uncertain	not	just	about	some	detail	of
some	peripheral	matter	but	about	the	cardinal	qualities	of	the	central	things.	For
many	key	parameters,	we	are	radically	uncertain	even	about	their	
sign
—that	is,
we	know	not	which	direction	of	change	would	be	desirable	and	which
undesirable.	Our	ignorance	might	not	be	irremediable.	The	field	has	been	little
prospected,	and	glimmering	strategic	insights	could	still	be	awaiting	their
unearthing	just	a	few	feet	beneath	the	surface.
What	we	mean	by	“strategic	analysis”	here	is	a	search	for	
crucial
considerations
:	ideas	or	arguments	with	the	potential	to	change	our	views	not
merely	about	the	fine-structure	of	implementation	but	about	the	general	topology
of	desirability.
4

crucial
4
	Even	a	single	missed	crucial	consideration	could	vitiate	our	most
valiant	efforts	or	render	them	as	actively	harmful	as	those	of	a	soldier	who	is
fighting	on	the	wrong	side.	The	search	for	crucial	considerations	(which	must
explore	normative	as	well	as	descriptive	issues)	will	often	require	crisscrossing
the	boundaries	between	different	academic	disciplines	and	other	fields	of
knowledge.	As	there	is	no	established	methodology	for	how	to	go	about	this	kind
of	research,	difficult	original	thinking	is	necessary.

Another	high-value	activity,	one	that	shares	with	strategic	analysis	the	robustness
property	of	being	beneficial	across	a	wide	range	of	scenarios,	is	the	development
of	a	well-constituted	support	base	that	takes	the	future	seriously.	Such	a	base	can
immediately	provide	resources	for	research	and	analysis.	If	and	when	other
priorities	become	visible,	resources	can	be	redirected	accordingly.	A	support
base	is	thus	a	general-purpose	capability	whose	use	can	be	guided	by	new
insights	as	they	emerge.

One	valuable	asset	would	be	a	donor	network	comprising	individuals	devoted
to	rational	philanthropy,	informed	about	existential	risk,	and	discerning	about	the
means	of	mitigation.	It	is	especially	desirable	that	the	early-day	funders	be	astute

and	altruistic,	because	they	may	have	opportunities	to	shape	the	field’s	culture
before	the	usual	venal	interests	take	up	position	and	entrench.	The	focus	during
these	opening	gambits	should	thus	be	to	recruit	the	right	kinds	of	people	into	the
field.	It	could	be	worth	foregoing	some	technical	advances	in	the	short	term	in
order	to	fill	the	ranks	with	individuals	who	genuinely	care	about	safety	and	who
have	a	truth-seeking	orientation	(and	who	are	likely	to	attract	more	of	their	own
kind).
One	important	variable	is	the	quality	of	the	“social	epistemology”	of	the	AI-
field	and	its	leading	projects.	Discovering	crucial	considerations	is	valuable,	but
only	if	it	affects	action.	This	cannot	always	be	taken	for	granted.	Imagine	a
project	that	invests	millions	of	dollars	and	years	of	toil	to	develop	a	prototype
AI,	and	that	after	surmounting	many	technical	challenges	the	system	is	finally
beginning	to	show	real	progress.	There	is	a	chance	that	with	just	a	bit	more	work

kind).
it	could	turn	into	something	useful	and	profitable.	Now	a	crucial	consideration	is
discovered,	indicating	that	a	completely	different	approach	would	be	a	bit	safer.
Does	the	project	kill	itself	off	like	a	dishonored	samurai,	relinquishing	its	unsafe
design	and	all	the	progress	that	had	been	made?	Or	does	it	react	like	a	worried
octopus,	puffing	out	a	cloud	of	motivated	skepticism	in	the	hope	of	eluding	the
attack?	A	project	that	would	reliably	choose	the	samurai	option	in	such	a
dilemma	would	be	a	far	preferable	developer.
5

5
	Yet	building	processes	and
institutions	that	are	willing	to	commit	seppuku	based	on	uncertain	allegations
and	speculative	reasoning	is	not	easy.	Another	dimension	of	social	epistemology
is	the	management	of	sensitive	information,	in	particular	the	ability	to	avoid
leaking	information	that	ought	be	kept	secret.	(Information	continence	may	be
especially	challenging	for	academic	researchers,	accustomed	as	they	are	to
constantly	disseminating	their	results	on	every	available	lamppost	and	tree.)

In	addition	to	the	general	objectives	of	strategic	light	and	good	capacity,	some
more	specific	objectives	could	also	present	cost-effective	opportunities	for
action.
One	such	is	progress	on	the	technical	challenges	of	machine	intelligence
safety.	In	pursing	this	objective,	care	should	be	taken	to	manage	information
hazards.	Some	work	that	would	be	useful	for	solving	the	control	problem	would

also	be	useful	for	solving	the	competence	problem.	Work	that	burns	down	the	AI
fuse	could	easily	be	a	net	negative.
Another	specific	objective	is	to	promote	“best	practices”	among	AI
researchers.	Whatever	progress	has	been	made	on	the	control	problem	needs	to
be	disseminated.	Some	forms	of	computational	experimentation,	particularly	if
involving	strong	recursive	self-improvement,	may	also	require	the	use	of
capability	control	to	mitigate	the	risk	of	an	accidental	takeoff.	While	the	actual
implementation	of	safety	methods	is	not	so	relevant	today,	it	will	increasingly
become	so	as	the	state	of	the	art	advances.	And	it	is	not	too	soon	to	call	for
practitioners	to	express	a	
commitment	to	safety
,	including	endorsing	the	common
good	principle	and	promising	to	ramp	up	safety	if	and	when	the	prospect	of
machine	superintelligence	begins	to	look	more	imminent.	Pious	words	are	not
sufficient	and	will	not	by	themselves	make	a	dangerous	technology	safe:	but
where	the	mouth	goeth,	the	mind	might	gradually	follow.

Other	opportunities	may	also	occasionally	arise	to	push	on	some	pivotal
parameter,	for	example	to	mitigate	some	other	existential	risk,	or	to	promote
biological	cognitive	enhancement	and	improvements	of	our	collective	wisdom,
or	even	to	shift	world	politics	into	a	more	harmonious	register.
	
Before	the	prospect	of	an	intelligence	explosion,	we	humans	are	like	small
children	playing	with	a	bomb.	Such	is	the	mismatch	between	the	power	of	our
plaything	and	the	immaturity	of	our	conduct.	Superintelligence	is	a	challenge	for

which	we	are	not	ready	now	and	will	not	be	ready	for	a	long	time.	We	have	little
idea	when	the	detonation	will	occur,	though	if	we	hold	the	device	to	our	ear	we
can	hear	a	faint	ticking	sound.
For	a	child	with	an	undetonated	bomb	in	its	hands,	a	sensible	thing	to	do
would	be	to	put	it	down	gently,	quickly	back	out	of	the	room,	and	contact	the
nearest	adult.	Yet	what	we	have	here	is	not	one	child	but	many,	each	with	access
to	an	independent	trigger	mechanism.	The	chances	that	we	will	
all
	find	the	sense

all
to	put	down	the	dangerous	stuff	seem	almost	negligible.	Some	little	idiot	is
bound	to	press	the	ignite	button	just	to	see	what	happens.

Nor	can	we	attain	safety	by	running	away,	for	the	blast	of	an	intelligence
explosion	would	bring	down	the	entire	firmament.	Nor	is	there	a	grown-up	in
sight.
In	this	situation,	any	feeling	of	gee-wiz	exhilaration	would	be	out	of	place.
Consternation	and	fear	would	be	closer	to	the	mark;	but	the	most	appropriate
attitude	may	be	a	bitter	determination	to	be	as	competent	as	we	can,	much	as	if
we	were	preparing	for	a	difficult	exam	that	will	either	realize	our	dreams	or
obliterate	them.
This	is	not	a	prescription	of	fanaticism.	The	intelligence	explosion	might	still
be	many	decades	off	in	the	future.	Moreover,	the	challenge	we	face	is,	in	part,	to
hold	on	to	our	humanity:	to	maintain	our	groundedness,	common	sense,	and
good-humored	decency	even	in	the	teeth	of	this	most	unnatural	and	inhuman
problem.	We	need	to	bring	all	our	human	resourcefulness	to	bear	on	its	solution.
Yet	let	us	not	lose	track	of	what	is	globally	significant.	Through	the	fog	of

everyday	trivialities,	we	can	perceive—if	but	dimly—the	essential	task	of	our
age.	In	this	book,	we	have	attempted	to	discern	a	little	more	feature	in	what	is
otherwise	still	a	relatively	amorphous	and	negatively	defined	vision—one	that
presents	as	our	principal	moral	priority	(at	least	from	an	impersonal	and	secular
perspective)	the	reduction	of	existential	risk	and	the	attainment	of	a	civilizational
trajectory	that	leads	to	a	compassionate	and	jubilant	use	of	humanity’s	cosmic
endowment.

NOTES
	
PRELIMS
	
1
.	Not	all	endnotes	contain	useful	information,	however.
2
.	I	don’t	know	which	ones.
CHAPTER	1:	PAST	DEVELOPMENTS	AND	PRESENT	CAPABILITIES
	
1
.	A	subsistence-level	income	today	is	about	$400	(Chen	and	Ravallion	2010).
A	million	subsistence-level	incomes	is	thus	$400,000,000.	The	current
world	gross	product	is	about	$60,000,000,000,000	and	in	recent	years	has
grown	at	an	annual	rate	of	about	4%	(compound	annual	growth	rate	since
1950,	based	on	Maddison	[2010]).	These	figures	yield	the	estimate
mentioned	in	the	text,	which	of	course	is	only	an	order-of-magnitude
approximation.	If	we	look	directly	at	population	figures,	we	find	that	it
currently	takes	the	world	population	about	one	and	a	half	weeks	to	grow	by
one	million;	but	this	underestimates	the	growth	rate	of	the	economy	since
per	capita
	income	is	also	increasing.	By	5000	
BC
,	following	the
Agricultural	Revolution,	the	world	population	was	growing	at	a	rate	of

per	capita
BC
about	1	million	per	200	years—a	great	acceleration	since	the	rate	of	perhaps
1	million	per	million	years	in	early	humanoid	prehistory—so	a	great	deal	of
acceleration	had	already	occurred	by	then.	Still,	it	is	impressive	that	an
amount	of	economic	growth	that	took	200	years	seven	thousand	years	ago
takes	just	ninety	minutes	now,	and	that	the	world	population	growth	that
took	two	centuries	then	takes	one	and	a	half	weeks	now.	See	also	Maddison
(2005).
2
.	Such	dramatic	growth	and	acceleration	might	suggest	one	notion	of	a

(2005).
2
possible	coming	“singularity,”	as	adumbrated	by	John	von	Neumann	in	a

conversation	with	the	mathematician	Stanislaw	Ulam:
Our	conversation	centred	on	the	ever	accelerating	progress	of	technology
and	changes	in	the	mode	of	human	life,	which	gives	the	appearance	of
approaching	some	essential	singularity	in	the	history	of	the	race	beyond
which	human	affairs,	as	we	know	them,	could	not	continue.	(Ulam	1958)
	
3
.	Hanson	(2000).
4
.	Vinge	(1993);	Kurzweil	(2005).
5
.	Sandberg	(2010).
6
.	Van	Zanden	(2003);	Maddison	(1999,	2001);	De	Long	(1998).
7
.	Two	oft-repeated	optimistic	statements	from	the	1960s:	“Machines	will	be
capable,	within	twenty	years,	of	doing	any	work	a	man	can	do”	(Simon
1965,	96);	“Within	a	generation	…	the	problem	of	creating	artificial
intelligence	will	substantially	be	solved”	(Minsky	1967,	2).	For	a
systematic	review	of	AI	predictions,	see	Armstrong	and	Sotala	(2012).
8
.	See,	for	example,	Baum	
et	al.
	(2011)	and	Armstrong	and	Sotala	(2012).
9
.	It	might	suggest,	however,	that	AI	researchers	know	less	about	development

7
8
et	al.
9
timelines	than	they	think	they	do—but	this	could	cut	both	ways:	they	might
overestimate	as	well	as	underestimate	the	time	to	AI.
.	Good	(1965,	33).
.	
One	exception	is	Norbert	Wiener,	who	did	have	some	qualms	about	the
possible	consequences.	He	wrote,	in	1960:	“If	we	use,	to	achieve	our
purposes,	a	mechanical	agency	with	whose	operation	we	cannot	efficiently
interfere	once	we	have	started	it,	because	the	action	is	so	fast	and
irrevocable	that	we	have	not	the	data	to	intervene	before	the	action	is

.	
complete,	then	we	had	better	be	quite	sure	that	the	purpose	put	into	the
machine	is	the	purpose	which	we	really	desire	and	not	merely	a	colourful
imitation	of	it”	(Wiener	1960).	Ed	Fredkin	spoke	about	his	worries	about

superintelligent	AI	in	an	interview	described	in	McCorduck	(1979).	By
1970,	Good	himself	writes	about	the	risks,	and	even	calls	for	the	creation	of
an	association	to	deal	with	the	dangers	(Good	[1970];	see	also	his	later
article	[Good	1982]	where	he	foreshadows	some	of	the	ideas	of	“indirect
normativity”	that	we	discuss	in	
Chapter	13
).	By	1984,	Marvin	Minsky	was
also	writing	about	many	of	the	key	worries	(Minsky	1984).
.	Cf.	Yudkowsky	(2008a).	On	the	importance	of	assessing	the	ethical
implications	of	potentially	dangerous	future	technologies	
before
	they
become	feasible,	see	Roache	(2008).
.	McCorduck	(1979).
.	Newell	
et	al.
	(1959).
.	The	SAINTS	program,	the	ANALOGY	program,	and	the	STUDENT
program,	respectively.	See	Slagle	(1963),	Evans	(1964,	1968),	and	Bobrow
(1968).
.	Nilsson	(1984).
.	Weizenbaum	(1966).
.	Winograd	(1972).
.	Cope	(1996);	Weizenbaum	(1976);	Moravec	(1980);	Thrun	
et	al.
	(2006);
Buehler	
et	al.
	(2009);	Koza	
et	al.
	(2003).	The	Nevada	Department	of	Motor

before
	they
.	Newell	
et	al.
	(1959).
(1968).
et	al.
	(2006);
Buehler	
et	al.
et	al.
Vehicles	issued	the	first	license	for	a	driverless	car	in	May	2012.
.	The	STANDUP	system	(Ritchie	
et	al.
	2007).
.	Schwartz	(1987).	Schwartz	is	here	characterizing	a	skeptical	view	that	he
thought	was	represented	by	the	writings	of	Hubert	Dreyfus.
.	One	vocal	critic	during	this	period	was	Hubert	Dreyfus.	Other	prominent
skeptics	from	this	era	include	John	Lucas,	Roger	Penrose,	and	John	Searle.
However,	among	these	only	Dreyfus	was	mainly	concerned	with	refuting

(2003).	The	Nevada	Department	of	Motor
et	al.
	2007).
claims	about	what	practical	accomplishments	we	should	expect	from
existing	paradigms	in	AI	(though	he	seems	to	have	been	open	to	the
possibility	that	new	paradigms	could	go	further).	Searle’s	target	was

functionalist	theories	in	the	philosophy	of	mind,	not	the	instrumental
powers	of	AI	systems.	Lucas	and	Penrose	denied	that	a	classical	computer
could	ever	be	programmed	to	do	everything	that	a	human	mathematician
can	do,	but	they	did	not	deny	that	any	particular	function	could	in	principle
be	automated	or	that	AIs	might	eventually	become	very	instrumentally
powerful.	Cicero	remarked	that	“there	is	nothing	so	absurd	but	some
philosopher	has	said	it”	(Cicero	1923,	119);	yet	it	is	surprisingly	hard	to
think	of	
any
	significant	thinker	who	has	denied	the	possibility	of	machine
superintelligence	in	the	sense	used	in	this	book.
.	For	many	applications,	however,	the	learning	that	takes	place	in	a	neural
network	is	little	different	from	the	learning	that	takes	place	in	linear
regression,	a	statistical	technique	developed	by	Adrien-Marie	Legendre	and
Carl	Friedrich	Gauss	in	the	early	1800s.
.	The	basic	algorithm	was	described	by	Arthur	Bryson	and	Yu-Chi	Ho	as	a

think	of	
any
multi-stage	dynamic	optimization	method	in	1969	(Bryson	and	Ho	1969).
The	application	to	neural	networks	was	suggested	by	Paul	Werbos	in	1974
(Werbos	1994),	but	it	was	only	after	the	work	by	David	Rumelhart,
Geoffrey	Hinton,	and	Ronald	Williams	in	1986	(Rumelhart	
et	al.
	1986)	that
the	method	gradually	began	to	seep	into	the	awareness	of	a	wider
community.
.	Nets	lacking	hidden	layers	had	previously	been	shown	to	have	severely
limited	functionality	(Minsky	and	Papert	1969).
.	E.g.,	MacKay	(2003).
.	Murphy	(2012).
.	Pearl	(2009).

et	al.
	1986)	that
.	We	suppress	various	technical	details	here	in	order	not	to	unduly	burden	the
exposition.	We	will	have	occasion	to	revisit	some	of	these	ignored	issues	in
Chapter	12
.
.	A	program	
p
	is	a	description	of	string	
x
	if	
p
,	run	on	(some	particular)
universal	Turing	machine	
U
,	outputs	
x
;	we	write	this	as	
U(p)	=	x
.	(The
string	
x
	here	represents	a	possible	world.)	The	
Kolmogorov	complexity	of	
x
is	then	
K
(
x
):=min
p
	{
l
(
p
):	
U
(
p
)	=	
x
},	where	
l
(
p
)	is	the	length	of	
p
	in	bits.	The

“Solomonoff”	probability	of	
x
	is	then	defined	as	
	where	the
sum	is	defined	over	all	(“minimal,”	
i.e.
	not	necessarily	halting)	programs	
p
for	which	
U
	outputs	a	string	starting	with	
x
	(Hutter	2005).
.	Bayesian	conditioning	on	evidence	
E
	gives
	
(The	probability	of	a	proposition	[like	
E
]	is	the	sum	of	the	probability	of	the
possible	worlds	in	which	it	is	true.)
.	Or	randomly	picks	one	of	the	possible	actions	with	the	highest	expected
utility,	in	case	there	is	a	tie.
.	More	concisely,	the	expected	utility	of	an	action	can	be	written	as	
	where	the	sum	is	over	all	possible	worlds.
.	See,	e.g.,	Howson	and	Urbach	(1993);	Bernardo	and	Smith	(1994);	Russell
and	Norvig	(2010).
.	Wainwright	and	Jordan	(2008).	The	application	areas	of	Bayes	nets	are
myriad;	see,	e.g.,	Pourret	
et	al.
	(2008).
.	One	might	wonder	why	so	much	detail	is	given	to	game	AI	here,	which	to
some	might	seem	like	an	unimportant	application	area.	The	answer	is	that
game-playing	offers	some	of	the	clearest	measures	of	human	vs.	AI

where	the	sum	is	over	all	possible	worlds.
et	al.
	(2008).
performance.
.	Samuel	(1959);	Schaeffer	(1997,	ch.	6).
.	Schaeffer	
et	al.
	(2007).
.	Berliner	(1980a,	b).
.	Tesauro	(1995).
.	Such	programs	include	GNU	(see	Silver	[2006])	and	Snowie	(see

Gammoned.net	[2012]).
.	Lenat	himself	had	a	hand	in	guiding	the	fleet-design	process.	He	wrote:
“Thus	the	final	crediting	of	the	win	should	be	about	60/40%	Lenat/Eurisko,
though	the	significant	point	here	is	that	neither	party	could	have	won	alone”
(Lenat	1983,	80).
.	Lenat	(1982,	1983).
.	Cirasella	and	Kopec	(2006).
.	Kasparov	(1996,	55).
.	Newborn	(2011).
.	Keim	
et	al.
	(1999).
.	See	Armstrong	(2012).
.	Sheppard	(2002).
.	Wikipedia	(2012a).
.	Markoff	(2011).
.	Rubin	and	Watson	(2011).
.	Elyasaf	
et	al.
	(2011).
.	KGS	(2012).
.	Newell	
et	al.
	(1958,	320).
.	Attributed	in	Vardi	(2012).
.	In	1976,	I.	J.	Good	wrote:	“A	computer	program	of	Grandmaster	strength
would	bring	us	within	an	ace	of	[machine	ultraintelligence]”	(Good	1976).
In	1979,	Douglas	Hofstadter	opined	in	his	Pulitzer-winning	
Gödel,	Escher,
Bach
:	“Question:	Will	there	be	chess	programs	that	can	beat	anyone?
Speculation:	No.	There	may	be	programs	that	can	beat	anyone	at	chess,	but

.	Elyasaf	
et	al.
	(2011).
.	Newell	
et	al.
Bach
they	will	not	be	exclusively	chess	programs.	They	will	be	programs	of

general	intelligence,	and	they	will	be	just	as	temperamental	as	people.	‘Do
you	want	to	play	chess?’	‘No,	I’m	bored	with	chess.	Let’s	talk	about
poetry’”	(Hofstadter	[1979]	1999,	678).
.	The	algorithm	is	minimax	search	with	alpha-beta	pruning,	used	with	a	chess-
specific	heuristic	evaluation	function	of	board	states.	Combined	with	a
good	library	of	openings	and	endgames,	and	various	other	tricks,	this	can
make	for	a	capable	chess	engine.
.	Though	especially	with	recent	progress	in	learning	the	evaluation	heuristic
from	simulated	games,	many	of	the	underlying	algorithms	would	probably
also	work	well	for	many	other	games.
.	
Nilsson	(2009,	318).	Knuth	was	certainly	overstating	his	point.	There	are
many	“thinking	tasks”	that	AI	has	not	succeeded	in	doing—inventing	a	new
subfield	of	pure	mathematics,	doing	any	kind	of	philosophy,	writing	a	great
detective	novel,	engineering	a	coup	d’état,	or	designing	a	major	new
consumer	product.
.	Shapiro	(1992).

.	
.	One	might	speculate	that	one	reason	it	has	been	difficult	to	match	human
abilities	in	perception,	motor	control,	common	sense,	and	language
understanding	is	that	our	brains	have	dedicated	wetware	for	these	functions
—neural	structures	that	have	been	optimized	over	evolutionary	timescales.
By	contrast,	logical	thinking	and	skills	like	chess	playing	are	not	natural	to
us;	so	perhaps	we	are	forced	to	rely	on	a	limited	pool	of	general-purpose
cognitive	resources	to	perform	these	tasks.	Maybe	what	our	brains	do	when

we	engage	in	explicit	logical	reasoning	or	calculation	is	in	some	ways
analogous	to	running	a	“virtual	machine,”	a	slow	and	cumbersome	mental
simulation	of	a	general-purpose	computer.	One	might	then	say	(somewhat
fancifully)	that	a	classical	AI	program	is	not	so	much	emulating	human
thinking	as	the	other	way	around:	a	human	who	is	thinking	logically	is
emulating	an	AI	program.
.	This	example	is	controversial:	a	minority	view,	represented	by	approximately
20%	of	adults	in	the	USA	and	similar	numbers	in	many	other	developed

nations,	holds	that	the	Sun	revolves	around	the	Earth	(Crabtree	1999;	Dean
2005).

.	World	Robotics	(2011).
.	Estimated	from	data	in	Guizzo	(2010).
.	Holley	(2009).
.	Hybrid	rule-based	statistical	approaches	are	also	used,	but	they	are	currently
a	small	part	of	the	picture.
.	Cross	and	Walker	(1994);	Hedberg	(2002).
.	Based	on	the	statistics	from	TABB	Group,	a	New	York-and	London-based
capital	markets	research	firm	(personal	communication).
.	CFTC	and	SEC	(2010).	For	a	different	perspective	on	the	events	of	6	May
2010,	see	CME	Group	(2010).
.	Nothing	in	the	text	should	be	construed	as	an	argument	against	algorithmic
high-frequency	trading,	which	might	normally	perform	a	beneficial
function	by	increasing	liquidity	and	market	efficiency.
.	A	smaller	market	scare	occurred	on	August,	1,	2012,	in	part	because	the
“circuit	breaker”	was	not	also	programmed	to	halt	trading	if	there	were
extreme	changes	in	the	
number
	of	shares	being	traded	(Popper	2012).	This
again	foreshadows	another	later	theme:	the	difficulty	of	anticipating	all

number
specific	ways	in	which	some	particular	plausible-seeming	rule	might	go
wrong.
.	Nilsson	(2009,	319).
.	Minsky	(2006);	McCarthy	(2007);	Beal	and	Winston	(2009).
.	Peter	Norvig,	personal	communication.	Machine-learning	classes	are	also
very	popular,	reflecting	a	somewhat	orthogonal	hype-wave	of	“big	data”
(inspired	by	
e.g.
	Google	and	the	Netflix	Prize).
.	Armstrong	and	Sotala	(2012).
.	Müller	and	Bostrom	(forthcoming).

.	See	Baum	
et	al.
	(2011),	another	survey	cited	therein,	and	Sandberg	and
Bostrom	(2011).
.	Nilsson	(2009).
.	This	is	again	conditional	on	no	civilization-disrupting	catastrophe	occurring.
The	definition	of	HLMI	used	by	Nilsson	is	“AI	able	to	perform	around	80%
of	jobs	as	well	or	better	than	humans	perform”	(Kruel	2012).
.	The	table	shows	the	results	of	four	different	polls	as	well	as	the	combined
results.	The	first	two	were	polls	taken	at	academic	conferences:	
PT-AI
,
participants	of	the	conference	
Philosophy	and	Theory	of	AI
	in	Thessaloniki
2011	(respondents	were	asked	in	November	2012),	with	a	response	rate	of
43	out	of	88;	and	
AGI
,	participants	of	the	conferences	
	and	
Impacts	and	Risks	of	Artificial	General	Intelligence
,	both
in	Oxford,	December	2012	(response	rate:	72/111).	The	
EETN
	poll	sampled
the	members	of	the	Greek	Association	for	Artificial	Intelligence,	a
professional	organization	of	published	researchers	in	the	field,	in	April
2013	(response

AGI
	and	
,	both
EETN
2013	(response	
rate:	26/250).	The	
TOP100
	poll	elicited	the	opinions	among
the	100	top	authors	in	artificial	intelligence	as	measured	by	a	citation	index,
in	May	2013	(response	rate:	29/100).
.	Interviews	with	some	28	(at	the	time	of	writing)	AI	practitioners	and	related
experts	have	been	posted	by	Kruel	(2011).
.	The	diagram	shows	renormalized	median	estimates.	Means	are	significantly
different.	For	example,	the	mean	estimates	for	the	“Extremely	bad”
outcome	were	7.6%	(for	
TOP100
)	and	17.2%	(for	the	combined	pool	of

TOP100
TOP100
expert	assessors).
.	There	is	a	substantial	literature	documenting	the	unreliability	of	expert
forecasts	in	many	domains,	and	there	is	every	reason	to	think	that	many	of
the	findings	in	this	body	of	research	apply	to	the	field	of	artificial
intelligence	too.	In	particular,	forecasters	tend	to	be	overconfident	in	their
predictions,	believing	themselves	to	be	more	accurate	than	they	really	are,
and	therefore	assigning	too	little	probability	to	the	possibility	that	their

most-favored	hypothesis	is	wrong	(Tetlock	2005).	(Various	other	biases
have	also	been	documented;	see,	e.g.,	Gilovich	
et	al.
	[2002].)	However,
uncertainty	is	an	inescapable	fact	of	the	human	condition,	and	many	of	our

actions	unavoidably	rely	on	expectations	about	which	long-term
consequences	are	more	or	less	plausible:	in	other	words,	on	probabilistic
predictions.	Refusing	to	offer	explicit	probabilistic	predictions	would	not
make	the	epistemic	problem	go	away;	it	would	just	hide	it	from	view
(Bostrom	2007).	Instead,	we	should	respond	to	evidence	of	overconfidence
by	broadening	our	confidence	intervals	(or	“credible	intervals”)—i.e.	by
smearing	out	our	credence	functions—and	in	general	we	must	struggle	as
best	we	can	with	our	biases,	by	considering	different	perspectives	and
aiming	for	intellectual	honesty.	In	the	longer	run,	we	can	also	work	to
develop	techniques,	training	methods,	and	institutions	that	can	help	us
achieve	better	calibration.	See	also	Armstrong	and	Sotala	(2012).
CHAPTER	2:	PATHS	TO	SUPERINTELLIGENCE
	
1
.	This	resembles	the	definition	in	Bostrom	(2003c)	and	Bostrom	(2006a).	It
can	also	be	compared	with	Shane	Legg’s	definition	(“Intelligence	measures

1
an	agent’s	ability	to	achieve	goals	in	a	wide	range	of	environments”)	and	its
formalizations	(Legg	2008).	It	is	also	very	similar	to	Good’s	definition	of
ultraintelligence	in	
Chapter	1
	(“a	machine	that	can	far	surpass	all	the
intellectual	activities	of	any	man	however	clever”).
2
.	For	the	same	reason,	we	make	no	assumption	regarding	whether	a
superintelligent	machine	could	have	“true	intentionality”	(
pace
	Searle,	it
could;	but	this	seems	irrelevant	to	the	concerns	of	this	book).	And	we	take

Chapter	1
2
pace
	Searle,	it
no	position	in	the	internalism/externalism	debate	about	mental	content	that
has	been	raging	in	the	philosophical	literature,	or	on	the	related	issue	of	the
extended	mind	thesis	(Clark	and	Chalmers	1998).
3
.	Turing	(1950,	456).
4
5
.	Chalmers	(2010);	Moravec	(1976,	1988,	1998,	1999).
6
.	See	Moravec	(1976).	A	similar	argument	is	advanced	by	David	Chalmers
(2010).
7
.	See	also	Shulman	and	Bostrom	(2012),	where	these	matters	are	elaborated

in	more	detail.
8
.	Legg	(2008)	offers	this	reason	in	support	of	the	claim	that	humans	will	be
able	to	recapitulate	the	progress	of	evolution	over	much	shorter	timescales
and	with	reduced	computational	resources	(while	noting	that	evolution’s
unadjusted	computational	resources	are	far	out	of	reach).	Baum	(2004)
argues	that	some	developments	relevant	to	AI	occurred	earlier,	with	the
organization	of	the	genome	itself	embodying	a	valuable	representation	for
evolutionary	algorithms.
9
.	Whitman	
et	al.
	(1998);	Sabrosky	(1952).
.	Schultz	(2000).
.	Menzel	and	Giurfa	(2001,	62);	Truman	
et	al.
	(1993).
.	Sandberg	and	Bostrom	(2008).
.	See	Legg	(2008)	for	further	discussion	of	this	point	and	of	the	promise	of
functions	or	environments	that	determine	fitness	based	on	a	smooth
landscape	of	pure	intelligence	tests.
.	See	Bostrom	and	Sandberg	(2009b)	for	a	taxonomy	and	more	detailed
discussion	of	ways	in	which	engineers	may	outperform	historical
evolutionary	selection.
.

9
.	Whitman	
et	al.
et	al.
	(1993).
.	
The	analysis	has	addressed	the	nervous	systems	of	living	creatures,	without
reference	to	the	cost	of	simulating	bodies	or	the	surrounding	virtual
environment	as	part	of	a	fitness	function.	It	is	plausible	that	an	adequate
fitness	function	could	test	the	competence	of	a	particular	organism	in	far
fewer	operations	than	it	would	take	to	simulate	all	the	neuronal
computation	of	that	organism’s	brain	throughout	its	natural	lifespan.	AI
programs	today	often	develop	and	operate	in	very	abstract	environments

.	
(theorem	provers	in	symbolic	math	worlds,	agents	in	simple	game
tournament	worlds,	etc.).
A	skeptic	might	insist	that	an	abstract	environment	would	be	inadequate
for	the	evolution	of	general	intelligence,	believing	instead	that	the	virtual
environment	would	need	to	closely	resemble	the	actual	biological
environment	in	which	our	ancestors	evolved.	Creating	a	physically	realistic

virtual	world	would	require	a	far	greater	investment	of	computational
resources	than	the	simulation	of	a	simple	toy	world	or	abstract	problem
domain	(whereas	evolution	had	access	to	a	physically	realistic	real	world
“for	free”).	In	the	limiting	case,	if	complete	micro-physical	accuracy	were
insisted	upon,	the	computational	requirements	would	balloon	to	ridiculous
proportions.	However,	such	extreme	pessimism	is	almost	certainly
unwarranted;	it	seems	unlikely	that	the	best	environment	for	evolving
intelligence	is	one	that	mimics	nature	as	closely	as	possible.	It	is,	on	the
contrary,	plausible	that	it	would	be	more	efficient	to	use	an	artificial
selection	environment,	one	quite	unlike	that	of	our	ancestors,	an
environment	specifically	designed	to	promote	adaptations	that	increase	the
type	of	intelligence	we	are	seeking	to	evolve	(abstract	reasoning	and
general	problem-solving	skills,	for	instance,	as	opposed	to	maximally	fast
instinctual	reactions	or	a	highly	optimized	visual	system).
.	Wikipedia	(2012b).

.	For	a	general	treatment	of	observation	selection	theory,	see	Bostrom	(2002a).
For	the	specific	application	to	the	current	issue,	see	Shulman	and	Bostrom
(2012).	For	a	short	popular	introduction,	see	Bostrom	(2008b).
.	Sutton	and	Barto	(1998,	21f);	Schultz	
et	al.
	(1997).
.	This	term	was	introduced	by	Eliezer	Yudkowsky;	see,	e.g.,	Yudkowsky
(2007).
.	This	is	the	scenario	described	by	Good	(1965)	and	Yudkowsky	(2007).
However,	one	could	also	consider	an	alternative	in	which	the	iterative
sequence	has	some	steps	that	do	not	involve	intelligence	enhancement	but

et	al.
	(1997).
(2007).
instead	design	simplification.	That	is,	at	some	stages,	the	seed	AI	might
rewrite	itself	so	as	make	subsequent	improvements	easier	to	find.
.	Helmstaedter	
et	al.
	(2011).
.	Andres	
et	al.
	(2012).
.	Adequate	for	enabling	instrumentally	useful	forms	of	cognitive	functioning
and	communication,	that	is;	but	still	radically	impoverished	relative	to	the
interface	provided	by	the	muscles	and	sensory	organs	of	a	normal	human
body.

.	Sandberg	(2013).
.	See	the	“Computer	requirements”	section	of	Sandberg	and	Bostrom	(2008,
79–81).
.	A	lower	level	of	success	might	be	a	brain	simulation	that	has	biologically
suggestive	micro-dynamics	and	displays	a	substantial	range	of	emergent
species-typical	activity	such	as	a	slow-wave	sleep	state	or	activity-
dependent	plasticity.	Whereas	such	a	simulation	could	be	a	useful	testbed
for	neuroscientific	research	(though	one	which	might	come	close	to	raising
serious	ethical	issues),	it	would	not	count	as	a	whole	brain	emulation	unless
the	simulation	were	sufficiently	accurate	to	be	able	to	perform	a	substantial
fraction	of	the	intellectual	work	that	the	simulated	brain	was	capable	of.	As
a	rule	of	thumb,	we	might	say	that	in	order	for	a	simulation	of	a	human
brain	to	count	as	a	whole	brain	emulation,	it	would	need	to	be	able	to
express	coherent	verbal	thoughts	or	have	the	capacity	to	learn	to	do	so.
.	Sandberg	and	Bostrom	(2008).	Further	explanation	can	be	found	in	the

original	report.
.	The	first	map	is	described	in	Albertson	and	Thomson	(1976)	and	White	
et	al.
(1986).	The	combined	(and	in	some	cases	corrected)	network	is	available
from	the	“WormAtlas”	website	(
http://www.wormatlas.org/
).
.	For	a	review	of	past	attempts	of	emulating	
C.	elegans
	and	their	fates,	see
Kaufman	(2011).	Kaufman	quotes	one	ambitious	doctoral	student	working
in	the	area,	David	Dalrymple,	as	saying,	
“With	optogenetic	techniques,	we
are	just	at	the	point	where	it’s	not	an	outrageous	proposal	to	reach	for	the

et	al.
).
C.	elegans
capability	to	read	and	write	to	anywhere	in	a	living	
C.	elegans
	nervous
system,	using	a	high-throughput	automated	system….	I	expect	to	be
finished	with	
C.	elegans
	in	2–3	years.	I	would	be	extremely	surprised,	for
whatever	that’s	worth,	if	this	is	still	an	open	problem	in	2020”	(Dalrymple
2011).	Brain	models	aiming	for	biological	realism	that	were	hand-coded
(rather	than	generated	automatically)	have	achieved	some	basic
functionality;	see,	e.g.,	Eliasmith	
et	al.
	(2012).
.	

C.	elegans
	nervous
C.	elegans
et	al.
	(2012).
.	
	does	have	some	convenient	special	properties.	For
example,	the	organism	is	transparent,	and	the	wiring	pattern	of	its	nervous

system	does	not	change	between	individuals.
.	If	neuromorphic	AI	rather	than	whole	brain	emulation	is	the	end	product,
then	it	might	or	might	not	be	the	case	that	the	relevant	insights	would	be
derived	through	attempts	to	simulate	
human
	brains.	It	is	conceivable	that
the	important	cortical	tricks	would	be	discovered	during	the	study	of
(nonhuman)	animal	brains.	Some	animal	brains	might	be	easier	to	work
with	than	human	brains,	and	smaller	brains	would	require	fewer	resources
to	scan	and	model.	Research	on	animal	brains	would	also	be	subject	to	less
regulation.	It	is	even	conceivable	that	the	first	human-level	machine
intelligence	will	be	created	by	completing	a	whole	brain	emulation	of	some
suitable	animal	and	then	finding	ways	to	enhance	the	resultant	digital	mind.
Thus	humanity	could	get	its	comeuppance	from	an	uplifted	lab	mouse	or
macaque.
.	Uauy	and	Dangour	(2006);	Georgieff	(2007);	Stewart	
et	al.
	(2008);	Eppig	
et
al.
	(2010);	Cotman	and	Berchtold	(2002).

macaque.
et	al.
et
al.
.	According	to	the	World	Health	Organization	in	2007,	nearly	2	billion
individuals	have	insufficient	iodine	intake	(
The	Lancet
	2008).	
Severe
	iodine
deficiency	hinders	neurological	development	and	leads	to	cretinism,	which
involves	an	average	loss	of	about	12.5	IQ	points	(Qian	
et	al.
	2005).	The
condition	can	be	easily	and	inexpensively	prevented	though	salt
fortification	(Horton	
et	al.
	2008).
.	Bostrom	and	Sandberg	(2009a).
.	Bostrom	and	Sandberg	(2009b).	A	typical	
putative
	performance	increase

The	Lancet
	2008).	
Severe
	iodine
et	al.
	2005).	The
et	al.
	2008).
putative
from	pharmacological	and	nutritional	enhancement	is	in	the	range	of	10–
20%	on	test	tasks	measuring	working	memory,	attention,	
etc.
	But	it	is
generally	dubious	whether	such	reported	gains	are	real,	sustainable	over	a
longer	term,	and	indicative	of	correspondingly	improved	results	in	real-
world	problem	situations	(Repantis	
et	al.
	2010).	For	instance,	in	some	cases
there	might	be	a	compensating	deterioration	on	some	performance
dimensions	that	are	not	measured	by	the	test	tasks	(Sandberg	and	Bostrom
2006).

etc.
	But	it	is
et	al.
2006).
.	If	there	were	an	easy	way	to	enhance	cognition,	one	would	expect	evolution
already	to	have	taken	advantage	of	it.	Consequently,	the	most	promising

kind	of	nootropic	to	investigate	may	be	one	that	promises	to	boost
intelligence	in	some	manner	that	we	can	see	would	have	lowered	fitness	in
the	ancestral	environment—for	example,	by	increasing	head	size	at	birth	or
amping	up	the	brain’s	glucose	metabolism.	For	a	more	detailed	discussion
of	this	idea	(along	with	several	important	qualifications),	see	Bostrom
(2009b).
.	Sperm	are	harder	to	screen	because,	in	contrast	to	embryos,	they	consist	of
only	one	cell—and	one	cell	needs	to	be	destroyed	in	order	to	do	the
sequencing.	Oocytes	also	consist	of	only	one	cell;	however,	the	first	and
second	cell	divisions	are	asymmetric	and	produce	one	daughter	cell	with
very	little	cytoplasm,	the	polar	body.	Since	polar	bodies	contain	the	same
genome	as	the	main	cell	and	are	redundant	(they	eventually	degenerate)
they	can	be	biopsied	and	used	for	screening	(Gianaroli	2000).
.	Each	of	these	practices	was	subject	to	some	ethical	controversy	when	it	was
introduced,	but	there	seems	to	be	a	trend	toward	increasing	acceptance.

Attitudes	toward	human	genetic	engineering	and	embryo	selection	vary
significantly	across	cultures,	suggesting	that	development	and	application
of	new	techniques	will	probably	take	place	even	if	some	countries	initially
adopt	a	cautious	stance,	although	the	rate	at	which	this	happens	will	be
influenced	by	moral,	religious,	and	political	pressures.
.	Davies	
et	al.
	(2011);	Benyamin	
et	al.
	(2013);	Plomin	
et	al.
	(2013).	See	also
Mardis	(2011);	Hsu	(2012).
.

.	Davies	
et	al.
et	al.
et	al.
.	
Broad-sense	heritability	of	adult	IQ	is	usually	estimated	in	the	range	of	0.5–
0.8	within	middle-class	strata	of	developed	nations	(Bouchard	2004,	148).
Narrow-sense	heritability,	which	measures	the	portion	of	variance	that	is
attributable	to	additive	genetic	factors,	is	lower	(in	the	range	0.3–0.5)	but
still	substantial	(Devlin	
et	al.
	1997;	Davies	
et	al.
	2011;	Visscher	
et	al.
2008).	These	estimates	could	change	for	different	populations	and
environments,	as	heritabilities	vary	depending	on	the	population	and

et	al.
et	al.
et	al.
environment	being	studied.	For	example,	lower	heritabilities	have	been
found	among	children	and	those	from	deprived	environments	(Benyamin	
et
al.
	2013;	Turkheimer	
et	al.
	2003).	Nisbett	
et	al.
	(2012)	review	numerous
environmental	influences	on	variation	in	cognitive	ability.
.	The	following	several	paragraphs	draw	heavily	on	joint	work	with	Carl

Shulman	(Shulman	and	Bostrom	2014).
.	This	table	is	taken	from	Shulman	and	Bostrom	(2014).	It	is	based	on	a	toy
model	that	assumes	a	Gaussian	distribution	of	predicted	IQs	among	the
embryos	with	a	standard	deviation	of	7.5	points.	The	amount	of	cognitive
enhancement	that	would	be	delivered	with	different	numbers	of	embryos
depends	on	how	different	the	embryos	are	from	one	another	in	the	additive
genetic	variants	whose	effects	we	know.	Siblings	have	a	coefficient	of
relatedness	of	½,	and	common	additive	genetic	variants	account	for	half	or
less	of	variance	in	adult	fluid	intelligence	(Davies	
et	al.
	2011).	These	two
facts	suggest	that	where	the	observed	population	standard	deviation	in
developed	countries	is	15	points,	the	standard	deviation	of	genetic
influences	within	a	batch	of	embryos	would	be	7.5	points	or	less.
.	With	imperfect	information	about	the	additive	genetic	effects	on	cognitive
ability,	effect	sizes	would	be	reduced.	However,	even	a	small	amount	of

et	al.
knowledge	would	go	a	relatively	long	way,	because	the	gains	from	selection
do	not	scale	linearly	with	the	portion	of	variance	that	we	can	predict.
Instead,	the	effectiveness	of	our	selection	depends	on	the	standard	deviation
of	predicted	mean	IQ,	which	scales	as	the	
square	root
	of	variance.	For
example,	if	one	could	account	for	12.5%	of	the	variance,	this	could	deliver
effects	half	as	great	as	those	in	
Table	1
,	which	assume	50%.	For
comparison,	a	recent	study	(Rietveld	
et	al.
	2013)	claims	to	have	already

Table	1
et	al.
identified	2.5%	of	the	variance.
.	For	comparison,	standard	practice	today	involves	the	creation	of	fewer	than
ten	embryos.
.	Adult	and	embryonic	stem	cells	can	be	coaxed	to	develop	into	sperm	cells
and	oocytes,	which	can	then	be	fused	to	produce	an	embryo	(Nagy	
et	al.
2008;	Nagy	and	Chang	2007).	Egg	cell	precursors	can	also	form
parthenogenetic	blastocysts,	unfertilized	and	non-viable	embryos,	able	to
produce	embryonic	stem	cell	lines	for	the	process	(Mai	
et	al.
	2007).
.	The	opinion	is	that	of	Katsuhiko	Hayashi,	as	reported	in	Cyranoski	(2013).

et	al.
et	al.
	2007).
The	Hinxton	Group,	an	international	consortium	of	scientists	that	discusses
stem	cell	ethics	and	challenges,	predicted	in	2008	that	human	stem	cell-
derived	gametes	would	be	available	within	ten	years	(Hinxton	Group	2008),
and	developments	thus	far	are	broadly	consistent	with	this.

.	Sparrow	(2013);	Miller	(2012);	The	Uncertain	Future	(2012).
.	Sparrow	(2013).
.	Secular	concerns	might	focus	on	anticipated	impacts	on	social	inequality,	the
medical	safety	of	the	procedure,	fears	of	an	enhancement	“rat	race,”	rights
and	responsibilities	of	parents	vis-à-vis	their	prospective	offspring,	the
shadow	of	twentieth-century	eugenics,	the	concept	of	human	dignity,	and
the	proper	limits	of	states’	involvement	in	the	reproductive	choices	of	their
citizens.	(For	a	discussion	of	the	ethics	of	cognitive	enhancement	see
Bostrom	and	Ord	[2006],	Bostrom	and	Roache	[2011],	and	Sandberg	and
Savulescu	[2011].)	Some	religious	traditions	may	offer	additional	concerns,
including	ones	centering	on	the	moral	status	of	embryos	or	the	proper	limits
of	human	agency	within	the	scheme	of	creation.
.	To	stave	off	the	negative	effects	of	inbreeding,	iterated	embryo	selection
would	require	either	a	large	starting	supply	of	donors	or	the	expenditure	of

substantial	selective	power	to	reduce	harmful	recessive	alleles.	Either
alternative	would	tend	to	push	toward	offspring	being	less	closely
genetically	related	to	their	parents	(and	more	related	to	one	another).
.	Adapted	from	Shulman	and	Bostrom	(2014).
.	Bostrom	(2008b).
.	Just	how	difficult	an	obstacle	epigenetics	will	be	is	not	yet	known	(Chason	
et
al.
	2011;	Iliadou	
et	al.
	2011).
.	
While	cognitive	ability	is	a	fairly	heritable	trait,	there	may	be	few	or	no
common

et
al.
et	al.
	2011).
.	
common
	alleles	or	polymorphisms	that	individually	have	a	large	positive
effect	on	intelligence	(Davis	
et	al.
	2010;	Davies	
et	al.
	2011;	Rietveld	
et	al.
2013).	As	sequencing	methods	improve,	the	mapping	out	of	low-frequency
alleles	and	their	cognitive	and	behavioral	correlates	will	become
increasingly	feasible.	There	is	some	theoretical	evidence	suggesting	that
some	alleles	that	cause	genetic	disorders	in	homozygotes	may	provide
sizeable	cognitive	advantages	in	heterozygote	carriers,	leading	to	a

common
et	al.
et	al.
et	al.
prediction	that	Gaucher,	Tay-Sachs,	and	Niemann-Pick	heterozygotes
would	be	about	5	IQ	points	higher	than	control	groups	(Cochran	
et	al.
2006).	Time	will	tell	whether	this	holds.

.	One	paper	(Nachman	and	Crowell	2000)	estimates	175	mutations	per
genome	per	generation.	Another	(Lynch	2010),	using	different	methods,
estimates	that	the	average	newborn	has	between	50	and	100	new	mutations,
and	Kong	
et	al.
	(2012)	implies	a	figure	of	around	77	new	mutations	per
generation.	Most	of	these	mutations	do	not	affect	functioning,	or	do	so	only
to	an	imperceptibly	slight	degree;	but	the	combined	effects	of	many	very
slightly	deleterious	mutations	could	be	a	significant	loss	of	fitness.	See	also
Crow	(2000).
.	Crow	(2000);	Lynch	(2010).
.	There	are	some	potentially	important	caveats	to	this	idea.	It	is	possible	that
the	modal	genome	would	need	some	adjustments	in	order	to	avoid
problems.	For	example,	parts	of	the	genome	might	be	adapted	to	interacting
with	other	parts	under	the	assumption	that	all	parts	function	with	a	certain
level	of	efficiency.	Increasing	the	efficiency	of	those	parts	might	then	lead
to	overshooting	along	some	metabolic	pathways.

.	These	composites	were	created	by	Mike	Mike	from	individual	photographs
taken	by	Virtual	Flavius	(Mike	2013).
.	They	can,	of	course,	have	some	effects	sooner—for	instance,	by	changing
people’s	expectations	of	what	is	to	come.
.	Louis	Harris	&	Associates	(1969);	Mason	(2003).
.	Kalfoglou	
et	al.
	(2004).
.	The	data	is	obviously	limited,	but	individuals	selected	for	1-in-10,000	results
on	childhood	ability	tests	have	been	shown,	in	longitudinal	studies,	to	be
substantially	more	likely	to	become	tenured	professors,	earn	patents,	and

et	al.
	(2004).
succeed	in	business	than	those	with	slightly	less	exceptional	scores	(Kell	
et
al.
	2013).	Roe	(1953)	studied	sixty-four	eminent	scientists	and	found
median	cognitive	ability	three	to	four	standard	deviations	above	the
population	norm	and	strikingly	higher	than	is	typical	for	scientists	in
general.	(Cognitive	ability	is	also	correlated	with	lifetime	earnings	and	with
non-financial	outcomes	such	as	life	expectancy,	divorce	rates,	and
probability	of	dropping	out	of	school	[Deary	2012].)	An	upward	shift	of	the

et
al.
distribution	of	cognitive	ability	would	have	disproportionately	large	effects
at	the	tails,	especially	increasing	the	number	of	highly	gifted	and	reducing

the	number	of	people	with	retardation	and	learning	disabilities.	See	also
Bostrom	and	Ord	(2006)	and	Sandberg	and	Savulescu	(2011).
.	
E.g.
	Warwick	(2002).	Stephen	Hawking	even	suggested	that	taking	this	step
might	be	necessary	in	order	to	keep	up	with	advances	in	machine
intelligence:	“We	must	develop	as	quickly	as	possible	technologies	that
make	possible	a	direct	connection	between	brain	and	computer,	so	that
artificial	brains	contribute	to	human	intelligence	rather	than	opposing	it”
(reported	in	Walsh	[2001]).	Ray	Kurzweil	concurs:	“As	far	as	Hawking’s	…
recommendation	is	concerned,	namely	direct	connection	between	the	brain
and	computers,	I	agree	that	this	is	both	reasonable,	desirable	and	inevitable.
[
sic
]	It’s	been	my	recommendation	for	years”	(Kurzweil	2001).
.	See	Lebedev	and	Nicolelis	(2006);	Birbaumer	
et	al.
	(2008);	Mak	and
Wolpaw	(2009);	and	Nicolelis	and	Lebedev	(2009).	A	more	personal
outlook	on	the	problem	of	enhancement	through	implants	can	be	found	in
Chorost	(2005,	Chap.	11).
.	Smeding

[
sic
et	al.
.	Smeding	
et	al.
	(2006).
.	Degnan	
et	al.
	(2002).
.	Dagnelie	(2012);	Shannon	(2012).
.	Perlmutter	and	Mink	(2006);	Lyons	(2011).
.	Koch	
et	al.
	(2006).
.	Schalk	(2008).	For	a	general	review	of	the	current	state	of	the	art,	see	Berger
et	al.
	(2008).	For	the	case	that	this	would	help	lead	to	enhanced
intelligence,	see	Warwick	(2002).
.	
Some	examples:	Bartels	
et	al.
	(2008);	Simeral	
et	al.
	(2011);	Krusienski	and
Shih	(2011);	and	Pasqualotto	
et	al.
	(2012).
.	
E.g.
	Hinke	
et	al.
	(1993).

.	Smeding	
et	al.
	(2006).
.	Degnan	
et	al.
	(2002).
.	Koch	
et	al.
	(2006).
et	al.
.	
et	al.
et	al.
et	al.
	(2012).
.	
E.g.
	Hinke	
et	al.
	(1993).
.	There	are	partial	exceptions	to	this,	especially	in	early	sensory	processing.
For	example,	the	primary	visual	cortex	uses	a	retinotopic	mapping,	which
means	roughly	that	adjacent	neural	assemblies	receive	inputs	from	adjacent

areas	of	the	retinas	(though	ocular	dominance	columns	somewhat
complicate	the	mapping).
.	Berger	
et	al.
	(2012);	Hampson	
et	al.
	(2012).
.	Some	brain	implants	require	two	forms	of	learning:	the	device	learning	to
interpret	the	organism’s	neural	representations	and	the	organism	learning	to
use	the	system	by	generating	appropriate	neural	firing	patterns	(Carmena	
et
al.
	2003).
.	It	has	been	suggested	that	we	should	regard	corporate	entities	(corporations,
unions,	governments,	churches,	and	so	forth)	as	artificial	intelligent	agents,
entities	with	sensors	and	effectors,	able	to	represent	knowledge	and	perform
inference	and	take	action	(e.g.	Kuipers	[2012];	cf.	Huebner	[2008]	for	a
discussion	on	whether	collective	representations	can	exist).	They	are	clearly
powerful	and	ecologically	successful,	although	their	capabilities	and
internal	states	are	different	from	those	of	humans.
.	Hanson	(1995,	2000);	Berg	and	Rietz	(2003).
.	In	the	workplace,	for	instance,	employers	might	use	lie	detectors	to	crack

down	on	employee	theft	and	shirking,	by	asking	the	employee	at	the	end	of
each	business	day	whether	she	has	stolen	anything	and	whether	she	has
worked	as	hard	as	she	could.	Political	and	business	leaders	could	likewise
be	asked	whether	they	were	wholeheartedly	pursuing	the	interests	of	their
shareholders	or	constituents.	Dictators	could	use	them	to	target	seditious
generals	within	the	regime	or	suspected	troublemakers	in	the	wider
population.
.	One	could	imagine	neuroimaging	techniques	making	it	possible	to	detect

neural	signatures	of	motivated	cognition.	Without	self-deception	detection,
lie	detection	would	favor	individuals	who	believe	their	own	propaganda.
Better	tests	for	self-deception	tests	could	also	be	used	to	train	rationality
and	to	study	the	effectiveness	of	interventions	aimed	at	reducing	biases.
.	Bell	and	Gemmel	(2009).	An	early	example	is	found	in	the	work	of	MIT’s
Deb	Roy,	who	recorded	every	moment	of	his	son’s	first	three	years	of	life.
Analysis	of	this	audiovisual	data	is	yielding	information	on	language
development;	see	Roy	(2012).

.	Growth	in	total	world	population	of	biological	human	beings	will	contribute
only	a	small	factor.	Scenarios	involving	machine	intelligence	could	see	the
world	population	(including	digital	minds)	explode	by	many	orders	of
magnitude	in	a	brief	period	of	time.	But	that	road	to	superintelligence
involves	artificial	intelligence	or	whole	brain	emulation,	so	we	need	not
consider	it	in	this	subsection.
.	Vinge	(1993).
CHAPTER	3:	FORMS	OF	SUPERINTELLIGENCE
	
1
.	Vernor	Vinge	has	used	the	term	“weak	superintelligence”	to	refer	to	such
sped-up	human	minds	(Vinge	1993).
2
.	For	example,	if	a	very	fast	system	could	do	everything	that	any	human
could	do	except	dance	a	mazurka,	we	should	still	call	it	a	speed
superintelligence.	Our	interest	lies	in	those	core	cognitive	capabilities	that
have	economic	or	strategic	significance.
3
.	At	least	a	millionfold	speedup	compared	to	human	brains	is	physically
possible,	as	can	been	seen	by	considering	the	difference	in	speed	and

2
3
energy	of	relevant	brain	processes	in	comparison	to	more	efficient
information	processing.	The	speed	of	light	is	more	than	a	million	times
greater	than	that	of	neural	transmission,	synaptic	spikes	dissipate	more	than
a	million	times	more	heat	than	is	thermodynamically	necessary,	and	current
transistor	frequencies	are	more	than	a	million	times	faster	than	neuron
spiking	frequencies	(Yudkowsky	[2008a];	see	also	Drexler	[1992]).	The
ultimate	limits	of	speed	superintelligence	are	bounded	by	light-speed

communications	delays,	quantum	limits	on	the	speed	of	state	transitions,
and	the	volume	needed	to	contain	the	mind	(Lloyd	2000).	The	“ultimate
laptop”	described	by	Lloyd	(2000)	would	run	a	1.4×10
21
	FLOPS	brain
emulation	at	speedup	of	3.8×10
29
×	(assuming	the	emulation	could	be
sufficiently	parallelized).	Lloyd’s	construction,	however,	is	not	intended	to
be	technologically	plausible;	it	is	only	meant	to	illustrate	those	constraints
on	computation	that	are	readily	derivable	from	basic	physical	laws.
4

21
29
4
.	With	emulations,	there	is	also	an	issue	of	how	long	a	human-like	mind	can

keep	working	on	something	before	going	mad	or	falling	into	a	rut.	Even
with	task	variety	and	regular	holidays,	it	is	not	certain	that	a	human-like
mind	could	live	for	thousands	of	subjective	years	without	developing
psychological	problems.	Furthermore,	if	total	memory	capacity	is	limited—
a	consequence	of	having	a	limited	neuron	population—then	cumulative
learning	cannot	continue	indefinitely:	beyond	some	point,	the	mind	must
start	forgetting	one	thing	for	each	new	thing	it	learns.	(Artificial	intelligence
could	be	designed	such	as	to	ameliorate	these	potential	problems.)
5
.	Accordingly,	nanomechanisms	moving	at	a	modest	1	m/s	have	typical
timescales	of	nanoseconds.	See	section	2.3.2	of	Drexler	(1992).	Robin
Hanson	mentions	7-mm	“tinkerbell”	robot	bodies	moving	at	260	times
normal	speed	(Hanson	1994).
6
.	Hanson	(2012).
7
.	“Collective	intelligence”	does	not	refer	to	low-level	parallelization	of
computing	hardware	but	to	parallelization	at	the	level	of	intelligent

5
6
7
autonomous	agents	such	as	human	beings.	Implementing	a	single	emulation
on	a	massively	parallel	machine	might	result	in	speed	superintelligence	if
the	parallel	computer	is	sufficiently	fast:	it	would	not	produce	a	collective
8
.	Improvements	to	the	speed	or	the	quality	of	the	individual	components
could	also	indirectly	affect	the	performance	of	collective	intelligence,	but
here	we	mainly	consider	such	improvements	under	the	other	two	forms	of
superintelligence	in	our	classification.
9

8
9
.	It	has	been	argued	that	a	higher	population	density	triggered	the	Upper
Paleolithic	Revolution	and	that	beyond	a	certain	threshold	accumulation	of
cultural	complexity	became	much	easier	(Powell	
et	al.
	2009).
.	What	about	the	Internet?	It	seems	not	yet	to	have	amounted	to	a	super-sized
boost.	Maybe	it	will	do	so	eventually.	It	took	centuries	or	millennia	for	the
other	examples	listed	here	to	reveal	their	full	potential.
.	This	is,	obviously,	not	meant	to	be	a	realistic	thought	experiment.	A	planet

9
et	al.
	2009).
large	enough	to	sustain	seven	quadrillion	human	organisms	with	present
technology	would	implode,	unless	it	were	made	of	very	light	matter	or	were
hollow	and	held	up	by	pressure	or	other	artificial	means.	(A	Dyson	sphere

or	a	Shellworld	might	be	a	better	solution.)	History	would	have	unfolded
differently	on	such	a	vast	surface.	Set	all	this	aside.
.	Our	focus	here	is	on	the	functional	properties	of	a	unified	intellect,	not	on	the
question	of	whether	such	an	intellect	would	have	qualia	or	whether	it	would
be	a	mind	in	the	sense	of	having	subjective	conscious	experience.	(One
might	ponder,	though,	what	kinds	of	conscious	experience	might	arise	from
intellects	that	are	more	or	less	integrated	than	those	of	human	brains.	On
some	views	of	consciousness,	such	as	the	global	workspace	theory,	it	seems
one	might	expect	more	integrated	brains	to	have	more	capacious
consciousness.	Cf.	Baars	(1997),	Shanahan	(2010),	and	Schwitzgebel
(2013).)
.	Even	small	groups	of	humans	that	have	remained	isolated	for	some	time
might	still	benefit	from	the	intellectual	outputs	of	a	larger	collective
intelligence.	For	example,	the	language	they	use	might	have	been
developed	by	a	much	larger	linguistic	community,	and	the	tools	they	use

(2013).)
might	have	been	invented	in	a	much	larger	population	before	the	small
group	became	isolated.	But	even	if	a	small	group	had	always	been	isolated,
it	might	still	be	part	of	a	larger	collective	intelligence	than	meets	the	eye—
namely,	the	collective	intelligence	consisting	of	not	only	the	present	but	all
ancestral	generations	as	well,	an	aggregate	that	can	function	as	a	feed-
forward	information	processing	system.
.	By	the	Church–Turing	thesis,	all	computable	functions	are	computable	by	a

Turing	machine.	Since	any	of	the	three	forms	of	superintelligence	could
simulate	a	Turing	machine	(if	given	access	to	unlimited	memory	and
allowed	to	operate	indefinitely),	they	are	by	this	formal	criterion
computationally	equivalent.	Indeed,	an	average	human	being	(provided
with	unlimited	scrap	paper	and	unlimited	time)	could	also	implement	a
Turing	machine,	and	thus	is	also	equivalent	by	the	same	criterion.	What
matters	for	our	purposes,	however,	is	what	these	different	systems	can
achieve	
in	practice

achieve	
,	with	finite	memory	and	in	reasonable	time.	And	the
efficiency	variations	are	so	great	that	one	can	readily	make	some
distinctions.	For	example,	a	typical	individual	with	an	IQ	of	85	could	be
taught	to	implement	a	Turing	machine.	(Conceivably,	it	might	even	be
possible	to	train	some	particularly	gifted	and	docile	chimpanzee	to	do	this.)
Yet,	for	all	practical	intents	and	purposes,	such	an	individual	is	presumably
incapable	of,	say,	independently	developing	general	relativity	theory	or	of

winning	a	Fields	medal.
.	Oral	storytelling	traditions	can	produce	great	works	(such	as	the	Homeric
epics)	but	perhaps	some	of	the	contributing	authors	possessed	uncommon
gifts.
.	Unless	it	contains	as	components	intellects	that	have	speed	or	quality
.	Our	inability	to	specify	what	all	these	problems	are	may	in	part	be	due	to	a
lack	of	trying:	there	is	little	point	in	spending	time	detailing	intellectual	jobs
that	no	individual	and	no	currently	feasible	organization	can	perform.	But	it
is	also	possible	that	even	conceptualizing	some	of	these	jobs	is	itself	one	of
those	jobs	that	we	currently	lack	the	brains	to	perform.
.	Cf.	Boswell	(1917);	see	also	Walker	(2002).
.	This	mainly	occurs	in	short	bursts	in	a	subset	of	neurons—most	have	more
sedate	firing	rates	(Gray	and	McCormick	1996;	Steriade	
et	al.
	1998).	There
are	some	neurons	(“chattering	neurons,”	also	known	as	“fast	rhythmically
bursting”	cells)	that	may	reach	firing	frequencies	as	high	as	750	Hz,	but

et	al.
these	seem	to	be	extreme	outliers.
.	Feldman	and	Ballard	(1982).
.	The	conduction	velocity	depends	on	axon	diameter	(thicker	axons	are	faster)
and	whether	the	axon	is	myelinated.	Within	the	central	nervous	system,
transmission	delays	can	range	from	less	than	a	millisecond	to	up	to	100	ms
(Kandel	
et	al.
	2000).	Transmission	in	optical	fibers	is	around	68%	
c
(because	of	the	refractive	index	of	the	material).	Electrical	cables	are
roughly	the	same	speed,	59–77%	
c
.
.	This	assumes	a	signal	velocity	of	70%	
c
.	Assuming	100%	
c

(Kandel	
et	al.
c
c
.
c
c
	ups	the	estimate
to	1.8×10
18
	m
3
.
.	The	number	of	neurons	in	an	adult	human	male	brain	has	been	estimated	at
86.1	±	8.1	billion,	a	number	arrived	at	by	dissolving	brains	and
fractionating	out	the	cell	nuclei,	counting	the	ones	stained	with	a	neuron-
specific	marker.	In	the	past,	estimates	in	the	neighborhood	of	75–125
billion	neurons	were	common.	These	were	typically	based	on	manual

counting	of	cell	densities	in	representative	small	regions	(Azevedo	
et	al.
2009).
.	Whitehead	(2003).
.	Information	processing	systems	can	very	likely	use	molecular-scale
processes	for	computing	and	data	storage	and	reach	at	least	planetary	size	in
extent.	The	ultimate	physical	limits	to	computation	set	by	quantum
mechanics,	general	relativity,	and	thermodynamics	are,	however,	far	beyond
this	“Jupiter	brain”	level	(Sandberg	1999;	Lloyd	2000).
.	Stansberry	and	Kudritzki	(2012).	Electricity	used	in	data	centers	worldwide
amounted	to	1.1–1.5%	of	total	electricity	use	(Koomey	2011).	See	also
Muehlhauser	and	Salamon	(2012).
.	This	is	an	oversimplification.	The	number	of	chunks	working	memory	can
maintain	is	both	information-and	task-dependent;	however,	it	is	clearly
limited	to	a	small	number	of	chunks.	See	Miller	(1956)	and	Cowan	(2001).
.	An	example	might	be	that	the	difficulty	of	learning	Boolean	concepts
(categories	defined	by	logical	rules)	is	proportional	to	the	length	of	the

shortest	logically	equivalent	propositional	formula.	Typically,	even
formulae	just	3–4	literals	long	are	very	difficult	to	learn.	See	Feldman
(2000).
.	See	Landauer	(1986).	This	study	is	based	on	experimental	estimates	of
learning	and	forgetting	rates	in	humans.	Taking	into	account	implicit
learning	might	push	the	estimate	up	a	little.	If	one	assumes	a	storage
capacity	~1	bit	per	synapse,	one	gets	an	
upper	bound
	on	human	memory
capacity	of	about	10
15
	bits.	For	an	overview	of	different	estimates,	see
Appendix	A	of	Sandberg	and	Bostrom	(2008).

(2000).
15
.	Channel	noise	can	trigger	action	potentials,	and	synaptic	noise	produces
significant	variability	in	the	strength	of	transmitted	signals.	Nervous
systems	appear	to	have	evolved	to	make	numerous	trade-offs	between	noise
tolerance	and	costs	(mass,	size,	time	delays);	see	Faisal	
et	al.
	(2008).	For
example,	axons	cannot	be	thinner	than	0.1	μm	lest	random	opening	of	ion
channels	create	spontaneous	action	potentials	(Faisal	
et	al.
	2005).
.	Trachtenberg	
et	al.
	(2002).

.	
In	terms	of	memory	and	computational	power,	though	not	in	terms	of	energy
efficiency.	The	fastest	computer	in	the	world	at	the	time	of	writing	was
China’s	“Tianhe-2,”	which	displaced	Cray	Inc.	Titan	in	June	2013	with	a
performance	of	33.86	petaFLOPS.	It	uses	17.6	MW	of	power,	almost	six
orders	of	magnitude	more	than	the	brain’s	~20	W.
.	Note	that	this	survey	of	sources	of	machine	advantage	is	
disjunctive
:	our
argument	succeeds	even	if	some	of	the	items	listed	are	illusory,	so	long	as
there	is	at	least	one	source	that	can	provide	a	sufficiently	large	advantage.
CHAPTER	4:	THE	KINETICS	OF	AN	INTELLIGENCE	EXPLOSION
	
1
.	The	system	may	not	reach	one	of	these	baselines	at	any	sharply	defined
point.	There	may	instead	be	an	interval	during	which	the	system	gradually
becomes	able	to	outperform	the	external	research	team	on	an	increasing
number	of	system-improving	development	tasks.
2
.	In	the	past	half-century,	at	least	one	scenario	has	been	widely	recognized	in

1
2
which	the	existing	world	order	would	come	to	an	end	in	the	course	of
minutes	or	hours:	global	thermonuclear	war.
3
.	This	would	be	consistent	with	the	observation	that	the	Flynn	effect—the
secular	increase	in	measured	IQ	scores	within	most	populations	at	a	rate	of
some	3	IQ	points	per	decade	over	the	past	60	years	or	so—appears	to	have
ceased	or	even	reversed	in	recent	years	in	some	highly	developed	countries
such	as	the	United	Kingdom,	Denmark,	and	Norway	(Teasdale	and	Owen
2008;	Sundet	
et	al.

3
et	al.
	2004).	The	cause	of	the	Flynn	effect	in	the	past—and
whether	and	to	what	extent	it	represents	any	genuine	gain	in	general
intelligence	or	merely	improved	skill	at	solving	IQ	test-style	puzzles—has
been	the	subject	of	wide	debate	and	is	still	not	known.	Even	if	the	Flynn
effect	(at	least	partially)	reflects	real	cognitive	gains,	and	even	if	the	effect
is	now	diminishing	or	even	reversing,	this	does	not	prove	that	we	have	yet
hit	diminishing	returns	in	whatever	underlying	cause	was	responsible	for

et	al.
the	observed	Flynn	effect	in	the	past.	The	decline	or	reversal	could	instead
be	due	to	some	independent	detrimental	factor	that	would	otherwise	have
produced	an	even	bigger	observed	decline.

4
.	Bostrom	and	Roache	(2011).
5
.	Somatic	gene	therapy	could	eliminate	the	maturational	lag,	but	is
technically	much	more	challenging	than	germline	interventions	and	has	a
lower	ultimate	potential.
6
.	Average	global	economic	productivity	growth	per	year	over	the	period
1960–2000	was	4.3%	(Isaksson	2007).	Only	part	of	this	productivity
growth	is	due	to	gains	in	organizational	efficiency.	Some	
particular
networks	or	organizational	processes	of	course	are	improving	at	much
faster	rates.
7
.	Biological	brain	evolution	was	subject	to	many	constraints	and	trade-offs
that	are	drastically	relaxed	when	the	mind	moves	to	a	digital	medium.	For
example,	brain	size	is	limited	by	head	size,	and	a	head	that	is	too	big	has
trouble	passing	through	the	birth	canal.	A	large	brain	also	guzzles	metabolic
resources	and	is	a	dead	weight	that	impedes	movement.	The	connectivity
between	certain	brain	regions	might	be	limited	by	steric	constraints—the
volume	of	white	matter	is	significantly	larger	than	the	volume	of	the	gray

matter	it	connects.	Heat	dissipation	is	limited	by	blood	flow,	and	might	be
close	to	the	upper	limit	for	acceptable	functioning.	Furthermore,	biological
neurons	are	noisy,	slow,	and	in	need	of	constant	protection,	maintenance,
and	resupply	by	glial	cells	and	blood	vessels	(contributing	to	the
intracranial	crowding).	See	Bostrom	and	Sandberg	(2009b).
8
.	Yudkowsky	(2008a,	326).	For	a	more	recent	discussion,	see	Yudkowsky
(2013).
9
.	The	picture	shows	cognitive	ability	as	a	one-dimensional	parameter,	to	keep

8
(2013).
9
the	drawing	simple.	But	this	is	not	essential	to	the	point	being	made	here.
One	could,	for	example,	instead	represent	a	cognitive	ability	profile	as	a
hypersurface	in	a	multidimensional	space.
.	Lin	
et	al.
	(2012).
.	One	gets	a	certain	increase	in	collective	intelligence	simply	by	increasing	the
number	of	its	constituent	intellects.	Doing	so	should	at	least	enable	better
overall	performance	on	tasks	that	can	be	easily	parallelized.	To	reap	the	full

.	Lin	
et	al.
	(2012).
returns	from	such	a	population	explosion,	however,	one	would	also	need	to
achieve	some	(more	than	minimal)	level	of	coordination	between	the

constituents.
.	
The	distinction	between	speed	and	quality	of	intelligence	is	anyhow	blurred
in	the	case	of	non-neuromorphic	AI	systems.
.	Rajab	
et	al.
	(2006,	41–52).
.	It	has	been	suggested	that	using	configurable	integrated	circuits	(FPGAs)
rather	than	general-purpose	processors	could	increase	computational	speeds
in	neural	network	simulations	by	up	to	two	orders	of	magnitude	(Markram
2006).	A	study	of	high-resolution	climate	modeling	in	the	petaFLOP-range
found	a	twenty-four	to	thirty-four-fold	reduction	of	cost	and	about	two
orders	of	magnitude	reduction	in	power	requirements	using	a	custom
variant	of	embedded	processor	chips	(Wehner	
et	al.
	2008).
.	Nordhaus	(2007).	There	are	many	overviews	of	the	different	meanings	of
Moore’s	law;	see,	e.g.,	Tuomi	(2002)	and	Mack	(2011).
.	If	the	development	is	slow	enough,	the	project	can	avail	itself	of	progress
being	made	in	the	interim	by	the	outside	world,	such	as	advances	in
computer	science	made	by	university	researchers	and	improvements	in

et	al.
	2008).
hardware	made	by	the	semiconductor	industry.
.	Algorithmic	overhang	is	perhaps	less	likely,	but	one	exception	would	be	if
exotic	hardware	such	as	quantum	computing	becomes	available	to	run
algorithms	that	were	previously	infeasible.	One	might	also	argue	that	neural
networks	and	deep	machine	learning	are	cases	of	algorithm	overhang:	too
computationally	expensive	to	work	well	when	first	invented,	they	were
shelved	for	a	while,	then	dusted	off	when	fast	graphics	processing	units
made	them	cheap	to	run.	Now	they	win	contests.

.	And	even	if	progress	on	the	way	toward	the	human	baseline	were	slow.
.	
	is	that	part	of	the	world’s	optimization	power	that	is	applied	to
improving	the	system	in	question.	For	a	project	operating	in	complete
isolation,	one	that	receives	no	significant	ongoing	support	from	the	external
world,	we	have	
	≈	0,	even	though	the	project	must	have	started	with	a
resource	endowment	(computers,	scientific	concepts,	educated	personnel,
etc.)	that	is	derived	from	the	entire	world	economy	and	many	centuries	of

.	The	most	relevant	of	the	seed	AI’s	cognitive	abilities	here	is	its	ability	to
perform	intelligent	design	work	to	improve	itself,	
i.e.
	its	intelligence
amplification	capability.	(If	the	seed	AI	is	good	at	enhancing	another
system,	which	is	good	at	enhancing	the	seed	AI,	then	we	could	view	these
as	subsystems	of	a	larger	system	and	focus	our	analysis	on	the	greater
whole.)
.	This	assumes	that	recalcitrance	is	not	known	to	be	so	high	as	to	discourage
investment	altogether	or	divert	it	to	some	alternative	project.
.	A	similar	example	is	discussed	in	Yudkowsky	(2008b).
.	Since	inputs	have	risen	(e.g.	amounts	invested	in	building	new	foundries,	and
number	of	people	working	in	the	semiconductor	industry),	Moore’s	law
itself	has	not	given	such	a	rapid	growth	if	we	control	for	this	increase	in
inputs.	Combined	with	advances	in	software,	however,	an	18-month
doubling	time	in	performance	per	unit	of	input	may	be	more	historically
plausible.
.	Some	tentative	attempts	have	been	made	to	develop	the	idea	of	an

intelligence	explosion	within	the	framework	of	economic	growth	theory;
see,	e.g.,	Hanson	(1998b);	Jones	(2009);	Salamon	(2009).	These	studies
have	pointed	to	the	potential	of	extremely	rapid	growth	given	the	arrival	of
digital	minds,	but	since	endogenous	growth	theory	is	relatively	poorly
developed	even	for	historical	and	contemporary	applications,	any
application	to	a	potentially	discontinuous	future	context	is	better	viewed	at
this	stage	as	a	source	of	potentially	useful	concepts	and	considerations	than

as	an	exercise	likely	to	deliver	authoritative	forecasts.	For	an	overview	of
attempts	to	mathematically	model	a	technological	singularity,	see	Sandberg
(2010).
.	It	is	of	course	also	possible	that	there	will	be	no	takeoff	at	all.	But	since,	as
argued	earlier,	superintelligence	looks	technically	feasible,	the	absence	of	a
takeoff	would	likely	be	due	to	the	intervention	of	some	defeater,	such	as	an
existential	catastrophe.	If	strong	superintelligence	arrived	not	in	the	shape

(2010).
of	artificial	intelligence	or	whole	brain	emulation	but	through	one	of	other
paths	we	considered	above,	then	a	slower	takeoff	would	be	more	likely.
CHAPTER	5:	DECISIVE	STRATEGIC	ADVANTAGE

1
.	A	software	mind	might	run	on	a	single	machine	as	opposed	to	a	worldwide
network	of	computers;	but	this	is	not	what	we	mean	by	“concentration.”
Instead,	what	we	are	interested	in	here	is	
the	extent	to	which	power,
specifically	power	derived	from	technological	ability,	will	be	concentrated
in	the	advanced	stages	of,	or	immediately	following,	the	machine
intelligence	revolution.
2
.	Technology	diffusion	of	consumer	products,	for	example,	tends	to	be
slower	in	developing	countries	(Talukdar	
et	al.
	2002).	See	also	Keller
(2004)	and	The	World	Bank	(2008).
3
.	The	economic	literature	dealing	with	the	theory	of	the	firm	is	relevant	as	a
comparison	point	for	the	present	discussion.	The	
locus	classicus
	is	Coase
(1937).	See	also,	e.g.,	Canbäck	
et	al.
	(2006);	Milgrom	and	Roberts	(1990);
Hart	(2008);	Simester	and	Knez	(2002).
4
.	On	the	other	hand,	it	could	be	especially	easy	to	steal	a	seed	AI,	since	it
consists	of	software	that	could	be	transmitted	electronically	or	carried	on	a
portable	memory	device.
5

3
	is	Coase
et	al.
4
5
.	Barber	(1991)	suggests	that	the	Yangshao	culture	(5000–3000	
BC
)	might
have	used	silk.	Sun	
et	al.
	(2012)	estimate,	based	on	genetic	studies,
domestication	of	the	silkworm	to	have	occurred	about	4,100	years	ago.
6
.	Cook	(1984,	144).	This	story	might	be	too	good	to	withstand	historical
scrutiny,	rather	like	Procopius’	(
Wars
	VIII.xvii.1–7)	story	of	how	the
silkworms	were	supposedly	brought	to	Byzantium	by	wandering	monks,
hidden	in	their	hollow	bamboo	staves	(Hunt	2011).
7
.	Wood	(2007);	Temple	(1986).
8

BC
)	might
et	al.
6
Wars
7
8
.	Pre-Columbian	cultures	did	have	the	wheel	but	used	it	only	for	toys
(probably	due	to	a	lack	of	good	draft	animals).
9
.	Koubi	(1999);	Lerner	(1997);	Koubi	and	Lalman	(2007);	Zeira	(2011);	Judd
et	al.
	(2012).
.	Estimated	from	a	variety	of	sources.	The	time	gap	is	often	somewhat

arbitrary,	depending	on	how	exactly	“equivalent”	capabilities	are	defined.
Radar	was	used	by	at	least	two	countries	within	a	couple	of	years	of	its
introduction,	but	exact	figures	in	months	are	hard	to	come	by.
.	The	RDS-6	in	1953	was	the	first	test	of	a	bomb	with	fusion	reactions,	but	the
RDS-37	in	1955	was	the	first	“true”	fusion	bomb,	where	most	power	came
from	the	fusion	reaction.
.	Unconfirmed.
.	Tests	in	1989,	project	cancelled	in	1994.
.	Deployed	system,	capable	of	a	range	greater	than	5,000	km.
.	Polaris	missiles	bought	from	the	USA.
.	Current	work	is	underway	on	the	Taimur	missile,	likely	based	on	Chinese
missiles.
.	The	RSA-3	rocket	tested	1989–90	was	intended	for	satellite	launches	and/or
as	an	ICBM.
.	MIRV	=	multiple	independently	targetable	re-entry	vehicle,	a	technology	that
enables	a	single	ballistic	missile	to	carry	multiple	warheads	that	can	be
programmed	to	hit	different	targets.
.	The	Agni	V	system	is	not	yet	in	service.
.	Ellis	(1999).

missiles.
as	an	ICBM.
.	If	we	model	the	situation	as	one	where	the	lag	time	between	projects	is
drawn	from	a	normal	distribution,	then	the	likely	distance	between	the
leading	project	and	its	closest	follower	will	also	depend	on	how	many
projects	there	are.	If	there	are	a	vast	number	of	projects,	then	the	distance
between	the	first	two	is	likely	small	even	if	the	variance	of	the	distribution
is	moderately	high	(though	the	expected	gap	between	the	lead	and	the
second	project	declines	very	slowly	with	the	number	of	competitors	if

completion	times	are	normally	distributed).	However,	it	is	unlikely	that
there	will	be	a	vast	number	of	projects	that	are	each	well	enough	resourced
to	be	serious	contenders.	(There	might	be	a	greater	number	of	projects	if

there	are	a	large	number	of	different	basic	approaches	that	could	be
pursued,	but	in	that	case	many	of	those	approaches	are	likely	to	prove	dead
ends.)	As	suggested,	empirically	we	seem	to	find	that	there	is	usually	no
more	than	a	handful	of	serious	competitors	pursuing	any	one	specific
technological	goal.	The	situation	is	somewhat	different	in	a	consumer
market	where	there	are	many	niches	for	slightly	different	products	and
where	barriers	to	entry	are	low.	There	are	lots	of	one-person	projects
designing	T-shirts,	but	only	a	few	firms	in	the	world	developing	the	next
generation	of	graphics	cards.	(Two	firms,	AMD	and	NVIDIA,	enjoy	a	near
duopoly	at	the	moment,	though	Intel	is	also	competing	at	the	lower-
performance	end	of	the	market.)
.	Bostrom	(2006c).	One	could	imagine	a	singleton	whose	existence	is	invisible
(e.g.	a	superintelligence	with	such	advanced	technology	or	insight	that	it
could	subtly	control	world	events	without	any	human	noticing	its

interventions);	or	a	singleton	that	voluntarily	imposes	very	strict	limitations
on	its	own	exercise	of	power	(e.g.	punctiliously	confining	itself	to	ensuring
that	certain	treaty-specified	international	rules—or	libertarian	principles—
are	respected).	How	likely	
any	particular	kind	of	singleton	is	to	arise	is	of
course	an	empirical	question;	but	
conceptually
,	at	least,	it	is	possible	to
have	a	good	singleton,	a	bad	singleton,	a	rambunctiously	diverse	singleton,
a	blandly	monolithic	singleton,	a	crampingly	oppressive	singleton,	or	a

singleton	more	akin	to	an	extra	law	of	nature	than	to	a	yelling	despot.
.	Jones	(1985,	344).
.	It	might	be	significant	that	the	Manhattan	Project	was	carried	out	during
wartime.	Many	of	the	scientists	who	participated	claimed	to	be	primarily
motivated	by	the	wartime	situation	and	the	fear	that	Nazi	Germany	might
develop	atomic	weapons	ahead	of	the	Allies.	It	might	be	difficult	for	many
governments	to	mobilize	a	similarly	intensive	and	secretive	effort	in
peacetime.	The	Apollo	program,	another	iconic	science/engineering

megaproject,	received	a	strong	impetus	from	the	Cold	War	rivalry.
.	Though	even	if	they	
were
	looking	hard,	it	is	not	clear	that	they	would	appear
(publicly)	to	be	doing	so.
.	Cryptographic	techniques	could	enable	the	collaborating	team	to	be
physically	dispersed.	The	only	weak	link	in	the	communication	chain	might

be	the	input	stage,	where	the	physical	act	of	typing	could	potentially	be
observed.	But	if	indoor	surveillance	became	common	(by	means	of
microscopic	recording	devices),	those	keen	on	protecting	their	privacy
might	develop	countermeasures	(e.g.	special	closets	that	could	be	sealed	off
from	would-be	eavesdropping	devices).	Whereas	physical	space	might
become	transparent	in	a	coming	surveillance	age,	cyberspace	might
possibly	become	more	protected	through	wider	adoption	of	stronger
cryptographic	protocols.
.	A	totalitarian	state	might	take	recourse	to	even	more	coercive	measures.
Scientists	in	relevant	fields	might	be	swept	up	and	put	into	work	camps,
akin	to	the	“academic	villages”	in	Stalinist	Russia.
.	When	the	level	of	public	concern	is	relatively	low,	some	researchers	might
welcome	a	little	bit	of	public	fear-mongering	because	it	draws	attention	to
their	work	and	makes	the	area	they	work	in	seem	important	and	exciting.
When	the	level	of	concern	becomes	greater,	the	relevant	research

communities	might	change	their	tune	as	they	begin	to	worry	about	funding
cuts,	regulation,	and	public	backlash.	Researchers	in	neighboring
disciplines—such	as	those	parts	of	computer	science	and	robotics	that	are
not	very	relevant	to	artificial	general	intelligence—might	resent	the	drift	of
funding	and	attention	away	from	their	own	research	areas.	These
researchers	might	also	correctly	observe	that	
their
	work	carries	no	risk
whatever	of	leading	to	a	dangerous	intelligence	explosion.	(Some	historical

their
parallels	might	be	drawn	with	the	career	of	the	idea	of	nanotechnology;	see
Drexler	[2013].)
.	These	have	been	successful	in	that	they	have	achieved	at	least	some	of	what
they	set	out	to	do.	How	successful	they	have	been	in	a	broader	sense	(taking
into	account	cost-effectiveness	and	so	forth)	is	harder	to	determine.	In	the
case	of	the	International	Space	Station,	for	example,	there	have	been	huge
cost	overruns	and	delays.	For	details	of	the	problems	encountered	by	the

project,	see	NASA	(2013).	The	Large	Hadron	Collider	project	has	had	some
major	setbacks,	but	this	might	be	due	to	the	inherent	difficulty	of	the	task.
The	Human	Genome	Project	achieved	success	in	the	end,	but	seems	to	have
received	a	speed	boost	from	being	forced	to	compete	with	Craig	Venter’s
private	corporate	effort.	Internationally	sponsored	projects	to	achieve
controlled	fusion	energy	have	failed	to	deliver	on	expectations,	despite
massive	investment;	but	again,	this	might	be	attributable	to	the	task	turning

out	to	be	more	difficult	than	anticipated.
.	US	Congress,	Office	of	Technology	Assessment	(1995).
.	Hoffman	(2009);	Rhodes	(2008).
.	Rhodes	(1986).
.	The	US	Navy’s	code-breaking	organization,	OP-20-G,	apparently	ignored	an
invitation	to	gain	full	knowledge	of	Britain’s	anti-Enigma	methods,	and
failed	to	inform	higher-level	US	decision	makers	of	Britain’s	offer	to	share
its	cryptographic	secrets	(Burke	2001).	This	gave	American	leaders	the
impression	that	Britain	was	withholding	important	information,	a	cause	of
friction	throughout	the	war.	Britain	did	share	with	the	Soviet	government
some	of	the	intelligence	they	had	gleaned	from	decrypted	German
communications.	In	particular,	Russia	was	warned	about	the	German
preparations	for	Operation	Barbarossa.	But	Stalin	refused	to	believe	the
warning,	partly	because	the	British	did	not	disclose	how	they	had	obtained
the	information.
.	
For	a	few	years,	Russell	seems	to	have	advocated	the	threat	of	nuclear	war	to

.	
persuade	Russia	to	accept	the	Baruch	plan;	later,	he	was	a	strong	proponent
of	mutual	nuclear	disarmament	(Russell	and	Griffin	2001).	John	von
Neumann	is	reported	to	have	believed	that	a	war	between	the	United	States
and	Russia	was	inevitable,	and	to	have	said,	“If	you	say	why	not	bomb
them	[the	Russians]	tomorrow,	I	say	why	not	bomb	them	today?	If	you	say
today	at	five	o’clock,	I	say	why	not	one	o’clock?”	(It	is	possible	that	he
made	this	notorious	statement	to	burnish	his	anti-communist	credentials
with	US	Defense	hawks	in	the	McCarthy	era.	Whether	von	Neumann,	had

he	been	in	charge	of	US	policy,	would	actually	have	launched	a	first	strike
is	impossible	to	ascertain.	See	Blair	[1957],	96.)
.	Baratta	(2004).
.	If	the	AI	is	controlled	by	a	group	of	humans,	the	problem	may	apply	to	this
human	group,	though	it	is	possible	that	new	ways	of	reliably	committing	to
an	agreement	will	be	available	by	this	time,	in	which	case	even	human
groups	could	avoid	this	problem	of	potential	internal	unraveling	and
overthrow	by	a	sub-coalition.

CHAPTER	6:	COGNITIVE	SUPERPOWERS
	
1
.	In	what	sense	is	humanity	a	dominant	species	on	Earth?	Ecologically
speaking,	humans	are	the	most	common	large	(~50	kg)	animal,	but	the	total
human	dry	biomass	(~100	billion	kg)	is	not	so	impressive	compared	with
that	of	ants,	the	family	Formicidae	(300	billion–3,000	billion	kg).	Humans
and	human	utility	organisms	form	a	very	small	part	(<0.001)	of	total	global
biomass.	However,	croplands	and	pastures	are	now	among	the	largest
ecosystems	on	the	planet,	covering	about	35%	of	the	ice-free	land	surface
(Foley	
et	al.
	2007).	And	we	appropriate	nearly	a	quarter	of	net	primary
productivity	according	to	a	typical	assessment	(Haberl	
et	al.
	2007),	though
estimates	range	from	3	to	over	50%	depending	mainly	on	varying
definitions	of	the	relevant	terms	(Haberl	
et	al.
	2013).	Humans	also	have	the
largest	geographic	coverage	of	any	animal	species	and	top	the	largest
number	of	different	food	chains.
2
.	Zalasiewicz	
et	al.
	(2008).
3
.	See	first	note	to	this	chapter.
4

(Foley	
et	al.
et	al.
et	al.
2
et	al.
	(2008).
3
4
.	Strictly	speaking,	this	may	not	be	quite	correct.	Intelligence	in	the	human
species	ranges	all	the	way	down	to	approximately	zero	(e.g.	in	the	case	of
embryos	or	patients	in	permanent	vegetative	state).	In	qualitative	terms,	the
maximum	difference	in	cognitive	ability	within	the	human	species	is
therefore	perhaps	greater	than	the	difference	between	any	human	and	a
superintelligence.	But	the	point	in	the	text	stands	if	we	read	“human”	as
“normally	functioning	adult.”
5
.	Gottfredson	(2002).	See	also	Carroll	(1993)	and	Deary	(2001).
6

5
6
.	See	Legg	(2008).	Roughly,	Legg	proposes	to	measure	a	reinforcement-
learning	agent	as	its	expected	performance	in	all	reward-summable
environments,	where	each	such	environment	receives	a	weight	determined
by	its	Kolmogorov	complexity.	We	will	explain	what	is	meant	by
reinforcement	learning	in	
Chapter	12
.	See	also	Dowe	and	Hernández-Orallo
(2012)	and	Hibbard	(2011).
7
.	With	regard	to	technology	research	in	areas	like	biotechnology	and

nanotechnology,	what	a	superintelligence	would	excel	at	is	the	design	and
modeling	of	new	structures.	To	the	extent	that	design	ingenuity	and
modeling	cannot	substitute	for	physical	experimentation,	the
superintelligence’s	performance	advantage	may	be	qualified	by	its	level	of
access	to	the	requisite	experimental	apparatus.
8
.	E.g.,	Drexler	(1992,	2013).
9
.	A	narrow-domain	AI	could	of	course	have	significant	commercial
applications,	but	this	does	not	mean	that	it	would	have	the	economic
productivity	superpower.	For	example,	even	if	a	narrow-domain	AI	earned
its	owners	several	billions	of	dollars	a	year,	this	would	still	be	four	orders	of
magnitude	less	than	the	rest	of	the	world	economy.	In	order	for	the	system
directly	and	substantially	to	increase	world	product,	an	AI	would	need	to	be
able	to	perform	many	kinds	of	work;	that	is,	it	would	need	competence	in
many	domains.
.	The	criterion	does	not	rule	out	all	scenarios	in	which	the	AI	fails.	For

example,	the	AI	might	rationally	take	a	gamble	that	has	a	high	chance	of
failing.	In	this	case,	however,	the	criterion	could	take	the	form	that	(a)	the
AI	should	make	an	unbiased	estimate	of	the	gamble’s	low	chance	of	
success
and	(b)	there	should	be	no	better	gamble	available	to	the	AI	that	we	present-
day	humans	can	think	of	but	that	the	AI	overlooks.
.	Cf.	Freitas	(2000)	and	Vassar	and	Freitas	(2006).
.	Yudkowsky	(2008a).
.	Freitas	(1980);	Freitas	and	Merkle	(2004,	Chap.	3);	Armstrong	and	Sandberg
(2013).

success
(2013).
.	See,	e.g.,	Huffman	and	Pless	(2003),	Knill	
et	al.
	(2000),	Drexler	(1986).
.	That	is	to	say,	the	distance	would	be	small	on	some	“natural”	metric,	such	as
the	logarithm	of	the	size	of	the	population	that	could	be	sustainably
supported	at	subsistence	level	by	a	given	level	of	capability	if	all	resources
were	devoted	to	that	end.
.	This	estimate	is	based	on	the	WMAP	estimate	of	a	cosmological	baryon
density	of	9.9×10
–30
	g/cm
3
	and	assumes	that	90%	of	the	mass	is

intergalactic	gas,	that	some	15%	of	the	galactic	mass	is	stars	(about	80%	of
baryonic	matter),	and	that	the	average	star	weighs	in	at	0.7	solar	masses
(Read	and	Trentham	2005;	Carroll	and	Ostlie	2007).
.	Armstrong	and	Sandberg	(2013).
.	Even	at	100%	of	
c
	(which	is	unattainable	for	objects	with	nonzero	rest	mass)
the	number	of	reachable	galaxies	is	only	about	6×10
9
.	(Cf.	Gott	
et	al.
[2005]	and	Heyl	[2005].)	We	are	assuming	that	our	current	understanding
of	the	relevant	physics	is	correct.	It	is	hard	to	be	very	confident	in	any
upper	bound,	since	it	is	at	least	conceivable	that	a	superintelligent
civilization	might	extend	its	reach	in	some	way	that	we	take	to	be
physically	impossible	(for	instance,	by	building	time	machines,	by
spawning	new	inflationary	universes,	or	by	some	other,	as	yet	unimagined
means).
.	The	number	of	habitable	planets	per	star	is	currently	uncertain,	so	this	is
merely	a	crude	estimate.	Traub	(2012)	predicts	that	one-third	of	stars	in

means).
spectral	classes	F,	G,	or	K	have	at	least	one	terrestrial	planet	in	the	habitable
zone;	see	also	Clavin	(2012).	FGK	stars	form	about	22.7%	of	the	stars	in
the	solar	neighborhood,	suggesting	that	7.6%	of	stars	have	potentially
suitable	planets.	In	addition,	there	might	be	habitable	planets	around	the
more	numerous	M	stars	(Gilster	2012).	See	also	Robles	
et	al.
	(2008).
It	would	not	be	necessary	to	subject	human	bodies	to	the	rigors	of
intergalactic	travels.	AIs	could	oversee	the	colonization	process.	
Homo

et	al.
	(2008).
Homo
sapiens
	could	be	brought	along	as	information,	which	the	AIs	could	later
use	to	instantiate	specimens	of	our	species.	For	example,	genetic
information	could	be	synthesized	into	DNA,	and	a	first	generation	of
humans	could	be	incubated,	raised,	and	educated	by	AI	guardians	taking	an
anthropomorphic	guise.
.	O’Neill	(1974).
.	Dyson	(1960)	claims	to	have	gotten	the	basic	idea	from	science	fiction	writer
Olaf	Stapledon	(1937),	who	in	turn	might	have	been	inspired	by	similar

Homo
sapiens
thoughts	by	J.	D.	Bernal	(Dyson	1979,	211).
.	Landauer’s	principle	states	that	there	is	a	minimum	amount	of	energy

required	to	change	one	bit	of	information,	known	as	the	Landauer	limit,
equal	to	
kT
	ln	2,	where	
k
	is	the	Boltzmann	constant	(1.38×10
–23
	J/K)	and	
T
is	the	temperature.	If	we	assume	the	circuitry	is	maintained	at	around	300
K,	then	10
26
	watts	allows	us	to	erase	approximately	10
47
	bits	per	second.
(On	the	achievable	efficiency	of	nanomechanical	computational	devices,
see	Drexler	[1992].	See	also	Bradbury	[1999];	Sandberg	[1999];	Ćirković
[2004].	The	foundations	of	Landauer’s	principle	are	still	somewhat	in
dispute;	see,	e.g.,	Norton	[2011].)
.	Stars	vary	in	their	power	output,	but	the	Sun	is	a	fairly	typical	main-sequence
star.
.	A	more	detailed	analysis	might	consider	more	closely	what	types	of
computation	we	are	interested	in.	The	number	of	
serial
	computations	that
can	be	performed	is	quite	limited,	since	a	fast	serial	computer	must	be	small
in	order	to	minimize	communications	lags	within	the	different	parts	of	the
computer.	There	are	also	limits	on	the	number	of	bits	that	can	be	stored,

star.
serial
and,	as	we	saw,	on	the	number	of	irreversible	computational	steps
(involving	the	erasure	of	information)	that	can	be	performed.
.	We	are	assuming	here	that	there	are	no	extraterrestrial	civilizations	that
might	get	in	the	way.	We	are	also	assuming	that	the	simulation	hypothesis	is
false.	See	Bostrom	(2003a).	If	either	of	these	assumptions	is	incorrect,	there
may	be	important	non-anthropogenic	risks—ones	that	involve	intelligent
agency	of	a	nonhuman	sort.	See	also	Bostrom	(2003b,	2009c).
.

.	
At	least	a	wise	singleton	that	grasped	the	idea	of	evolution	could,	in
principle,	have	embarked	on	a	eugenics	program	by	means	of	which	it
could	slowly	have	raised	its	level	of	collective	intelligence.
.	Tetlock	and	Belkin	(1996).
.	To	be	clear:	colonizing	and	reengineering	a	large	part	of	the	accessible
universe	is	not	currently	within	our	
direct
	reach.	Intergalactic	colonization
is	far	beyond	today’s	technology.	The	point	is	that	we	could	in	principle	use

.	
direct
our	present	capabilities	to	develop	the	additional	capabilities	that	would	be
needed,	thus	placing	the	accomplishment	within	our	
indirect
	reach.	It	is	of
course	also	true	that	humanity	is	not	currently	a	singleton	and	that	we	do
not	know	that	we	would	never	face	intelligent	opposition	from	some

external	power	if	we	began	to	reengineer	the	accessible	universe.	To	meet
the	wise-singleton	sustainability	threshold,	however,	it	suffices	that	one
possesses	a	capability	set	such	that	if	a	wise	singleton	facing	no	intelligent
opposition	had	possessed	this	capability	set	then	the	colonization	and
reengineering	of	a	large	part	of	the	accessible	universe	would	be	within	its
indirect	reach.
.	Sometimes	it	might	be	useful	to	speak	of	two	AIs	as	each	having	a	given
superpower.	In	an	extended	sense	of	the	word,	one	could	thus	conceive	of	a
superpower	as	something	that	an	agent	has	relative	to	some	field	of	action
—in	this	example,	perhaps	a	field	that	includes	all	of	human	civilization	but
excludes	the	other	AI.

CHAPTER	7:	THE	SUPERINTELLIGENT	WILL
	
1
.	This	is	of	course	not	to	deny	that	differences	that	appear	small	visually	can
be	functionally	profound.
2
.	Yudkowsky	(2008a,	310).
3
.	David	Hume,	the	Scottish	Enlightenment	philosopher,	thought	that	beliefs
alone	(say,	about	what	is	a	good	thing	to	do)	cannot	motivate	action:	some
desire	is	required.	This	would	support	the	orthogonality	thesis	by
undercutting	one	possible	objection	to	it,	namely	that	sufficient	intelligence
might	entail	the	acquisition	of	certain	beliefs	which	would	then	necessarily
produce	certain	motivations.	However,	although	the	orthogonality	thesis
can	draw	support	from	the	Humean	theory	of	motivation,	it	does	not
presuppose	it.	In	particular,	one	need	not	maintain	that	beliefs	alone	can
never	motivate	action.	It	would	suffice	to	assume,	for	example,	that	an
agent—be	it	ever	so	intelligent—can	be	motivated	to	pursue	any	course	of
action	if	the	agent	happens	to	have	certain	desires	of	some	sufficient,

overriding	strength.	Another	way	in	which	the	orthogonality	thesis	could	be
true	even	if	the	Humean	theory	of	motivation	is	false	is	if	arbitrarily	high
intelligence	does	not	entail	the	acquisition	of	any	such	beliefs	as	are
(putatively)	motivating	on	their	own.	A	third	way	in	which	it	might	be
possible	for	the	orthogonality	thesis	to	be	true	even	if	the	Humean	theory
were	false	is	if	it	is	possible	to	build	an	agent	(or	more	neutrally,	an
“optimization	process”)	with	arbitrarily	high	intelligence	but	with

constitution	so	alien	as	to	contain	no	clear	functional	analogs	to	what	in
humans	we	call	“beliefs”	and	“desires.”	(For	some	recent	attempts	to
defend	the	Humean	theory	of	motivation	see	Smith	[1987],	Lewis	[1988],
and	Sinhababu	[2009].)
4
.	For	instance,	Derek	Parfit	has	argued	that	certain	basic	preferences	would
be	irrational,	such	as	that	of	an	otherwise	normal	agent	who	has	“Future-
Tuesday-Indifference”:
A	certain	hedonist	cares	greatly	about	the	quality	of	his	future	experiences.

4
With	one	exception,	he	cares	equally	about	all	the	parts	of	his	future.	The

exception	is	that	he	has	Future-Tuesday-Indifference.	Throughout	every
Tuesday	he	cares	in	the	normal	way	about	what	is	happening	to	him.	But	he
never	cares	about	possible	pains	or	pleasures	on	a	future	Tuesday….	This
indifference	is	a	bare	fact.	When	he	is	planning	his	future,	it	is	simply	true
that	he	always	prefers	the	prospect	of	great	suffering	on	a	Tuesday	to	the
mildest	pain	on	any	other	day.	(Parfit	[1986,	123–4];	see	also	Parfit	[2011])
	
For	our	purposes,	we	need	take	no	stand	on	whether	Parfit	is	right	that	this
agent	is	irrational,	so	long	as	we	grant	that	it	is	not	necessarily	unintelligent
in	the	instrumental	sense	explained	in	the	text.	Parfit’s	agent	could	have
impeccable	instrumental	rationality,	and	therefore	great	intelligence,	even	if
he	falls	short	on	some	kind	of	sensitivity	to	“objective	reason”	that	might	
be
required	of	a	fully	rational	agent.	Therefore,	this	kind	of	example	does	not
undermine	the	orthogonality	thesis.
5

be
5
.	Even	if	there	are	objective	moral	facts	that	any	fully	rational	agent	would
comprehend,	and	even	if	these	moral	facts	are	somehow	intrinsically
motivating	(such	that	anybody	who	fully	comprehends	them	is	necessarily
motivated	to	act	in	accordance	with	them),	this	need	not	undermine	the
orthogonality	thesis.	The	thesis	could	still	be	true	if	an	agent	could	have
impeccable	
	rationality	even	whilst	lacking	some	other	faculty

5
constitutive	of	rationality	proper,	or	some	faculty	required	for	the	full
comprehension	of	the	objective	moral	facts.	(An	agent	could	also	be
extremely	intelligent,	even	superintelligent,	without	having	full
instrumental	rationality	in	every	domain.)
6
.	For	more	on	the	orthogonality	thesis,	see	Bostrom	(2012)	and	Armstrong
(2013).
7
8
.	Stephen	Omohundro	has	written	two	pioneering	papers	on	this	topic
(Omohundro	2007,	2008).	Omohundro	argues	that	all	advanced	AI	systems

6
(2013).
7
8
are	likely	to	exhibit	a	number	of	“basic	drives,”	by	which	he	means
“tendencies	which	will	be	present	unless	explicitly	counteracted.”	The	term
“AI	drive”	has	the	advantage	of	being	short	and	evocative,	but	it	has	the
disadvantage	of	suggesting	that	the	instrumental	goals	to	which	it	refers

influence	the	AI’s	decision-making	in	the	same	way	as	psychological	drives
influence	human	decision-making,	
i.e.
	via	a	kind	of	phenomenological	tug
on	our	ego	which	our	willpower	may	occasionally	succeed	in	resisting.	That
connotation	is	unhelpful.	One	would	not	normally	say	that	a	typical	human
being	has	a	“drive”	to	fill	out	their	tax	return,	even	though	filing	taxes	may
be	a	fairly	convergent	instrumental	goal	for	humans	in	contemporary
societies	(a	goal	whose	realization	averts	trouble	that	would	prevent	us
from	realizing	many	of	our	final	goals).	Our	treatment	here	also	differs
from	that	of	Omohundro	in	some	other	more	substantial	ways,	although	the
underlying	idea	is	the	same.	(See	also	Chalmers	[2010]	and	Omohundro
[2012].)
9
.	Chislenko	(1997).
.	See	also	Shulman	(2010b).
.	An	agent	might	also	change	its	goal	
	if	it	changes	its	ontology,
in	order	to	transpose	its	old	representation	into	the	new	ontology;	cf.	de
Blanc	(2011).
Another	type	of	factor	that	might	make	an

[2012].)
9
Another	type	of	factor	that	might	make	an	
evidential	decision	theorist
undertake	various	actions,	including	changing	its	final	goals,	is	the
evidential	import	of	deciding	to	do	so.	For	example,	an	agent	that	follows
evidential	decision	theory	might	believe	that	there	exist	other	agents	like	it
in	the	universe,	and	that	its	own	actions	will	provide	some	evidence	about
how	those	other	agents	will	act.	The	agent	might	therefore	choose	to	adopt
a	final	goal	that	is	altruistic	towards	those	other	evidentially	linked	agents,

on	grounds	that	this	will	give	the	agent	evidence	that	those	other	agents	will
have	chosen	to	act	in	like	manner.	An	equivalent	outcome	might	be
obtained,	however,	without	changing	one’s	final	goals,	by	choosing	in	each
instant	to	act	
as	if
	one	had	those	final	goals.
.	An	extensive	psychological	literature	explores	adaptive	preference
formation.	See,	e.g.,	Forgas	
et	al.
	(2010).
.	In	formal	models,	the	value	of	information	is	quantified	as	the	difference
between	the	expected	value	realized	by	optimal	decisions	made	with	that

as	if
et	al.
	(2010).
information	and	the	expected	value	realized	by	optimal	decisions	made
without	it.	(See,	e.g.,	Russell	and	Norvig	[2010].)	It	follows	that	the	value

of	information	is	never	negative.	It	also	follows	that	any	information	you
know	will	never	affect	any	decision	you	will	ever	make	has	zero	value	for
you.	However,	this	kind	of	model	assumes	several	idealizations	which	are
often	invalid	in	the	real	world—such	as	that	knowledge	has	no	final	value
(meaning	that	knowledge	has	only	instrumental	value	and	is	not	valuable
for	its	own	sake)	and	that	agents	are	not	transparent	to	other	agents.
.	E.g.,	Hájek	(2009).
.	This	strategy	is	exemplified	by	the	sea	squirt	larva,	which	swims	about	until
it	finds	a	suitable	rock,	to	which	it	then	permanently	affixes	itself.
Cemented	in	place,	the	larva	has	less	need	for	complex	information
processing,	whence	it	proceeds	to	digest	part	of	its	own	brain	(its	cerebral
ganglion).	One	can	observe	the	same	phenomenon	in	some	academics	when
they	have	been	granted	tenure.
.	Bostrom	(2012).
.	
Bostrom	(2006c).
.	One	could	reverse	the	question	and	look	instead	at	possible	reasons	for	a
superintelligent	singleton	
not

.	
not
	to	develop	some	technological	capabilities.
These	include	the	following:	(a)	the	singleton	foresees	that	it	will	have	no
use	for	the	capability;	(b)	the	development	cost	is	too	large	relative	to	its
anticipated	utility	(e.g.	if	the	technology	will	never	be	suitable	for	achieving
any	of	the	singleton’s	ends,	or	if	the	singleton	has	a	very	high	discount	rate
that	strongly	discourages	investment);	(c)	the	singleton	has	some	final	value
that	requires	abstention	from	particular	avenues	of	technology

not
development;	(d)	if	the	singleton	is	not	certain	it	will	remain	stable,	it	might
prefer	to	refrain	from	developing	technologies	that	could	threaten	its
internal	stability	or	that	would	make	the	consequences	of	dissolution	worse
(for	instance,	a	world	government	may	not	wish	to	develop	technologies
that	would	facilitate	rebellion,	even	if	they	have	some	good	uses,	nor
develop	technologies	for	the	easy	production	of	weapons	of	mass
destruction	which	could	wreak	havoc	if	the	world	government	were	to

dissolve);	(e)	similarly,	the	singleton	might	have	made	some	kind	of
binding	strategic	commitment	not	to	develop	some	technology,	a
commitment	that	remains	operative	even	if	it	would	now	be	convenient	to
develop	it.	(Note,	however,	that	some	
current
	reasons	for	technology

development	would	
not
	apply	to	a	singleton:	for	instance,	reasons	arising
from	arms	races.)
.	Suppose	that	an	agent	discounts	resources	obtained	in	the	future	at	an
exponential	rate,	and	that	because	of	the	light	speed	limitation	the	agent	can
only	increase	its	resource	endowment	at	a	polynomial	rate.	Would	this
mean	that	there	will	be	some	time	after	which	the	agent	would	not	find	it
worthwhile	to	continue	acquisitive	expansion?	No,	because	although	the
present	value	of	the	resources	obtained	at	future	times	would	asymptote	to
zero	the	further	into	the	future	we	look,	
so	would	the	present	cost	of
obtaining	them
.	The	present	cost	of	sending	out	one	more	von	Neumann
probe	a	100	million	years	from	now	(possibly	using	some	resource	acquired
some	short	time	earlier)	would	be	diminished	by	the	same	discount	factor
that	would	diminish	the	present	value	of	the	future	resources	that	the	extra
probe	would	acquire	(modulo	a	constant	factor).
.	While	the	volume	reached	by	colonization	probes	at	a	given	time	might	be

roughly	spherical	and	expanding	with	a	rate	proportional	to	the	square	of
time	elapsed	since	the	first	probe	was	launched	(~
t
2
),	the	amount	of
resources	contained	within	this	volume	will	follow	a	less	regular	growth
pattern,	since	the	distribution	of	resources	is	inhomogeneous	and	varies
over	several	scales.	Initially,	the	growth	rate	might	be	~
t
2
	as	the	home
planet	is	colonized;	then	the	growth	rate	might	become	spiky	as	nearby
planets	and	solar	systems	are	colonized;	then,	as	the	roughly	disc-shaped

t
2
t
2
	as	the	home
volume	of	the	Milky	Way	gets	filled	out,	the	growth	rate	might	even	out,	to
be	approximately	proportional	to	
t
;	then	the	growth	rate	might	again
become	spiky	as	nearby	galaxies	are	colonized;	then	the	growth	rate	might
again	approximate	~
t
2
	as	expansion	proceeds	on	a	scale	over	which	the
distribution	of	galaxies	is	roughly	homogeneous;	then	another	period	of
spiky	growth	followed	by	smooth	~
t
2
	growth	as	galactic	superclusters	are
colonized;	until	ultimately	the	growth	rate	starts	a	final	decline,	eventually

t
t
2
t
2
reaching	zero	as	the	expansion	speed	of	the	universe	increases	to	such	an
extent	as	to	make	further	colonization	impossible.
.	The	simulation	argument	may	be	of	particular	importance	in	this	context.	A
superintelligent	agent	may	assign	a	significant	probability	to	hypotheses
according	to	which	it	lives	in	a	computer	simulation	and	its	percept
sequence	is	generated	by	another	superintelligence,	and	this	might	generate
various	convergent	instrumental	reasons	depending	on	the	agent’s	guesses

about	what	types	of	simulations	it	is	most	likely	to	be	in.	Cf.	Bostrom
(2003a).
.	Discovering	the	basic	laws	of	physics	and	other	fundamental	facts	about	the
world	is	a	convergent	instrumental	goal.	We	may	place	it	under	the	rubric
“cognitive	enhancement”	here,	though	it	could	also	be	derived	from	the
“technology	perfection”	goal	(since	novel	physical	phenomena	might
enable	novel	technologies).
CHAPTER	8:	IS	THE	DEFAULT	OUTCOME	DOOM?
	
1
.	Some	additional	existential	risk	resides	in	scenarios	in	which	humanity
survives	in	some	highly	suboptimal	state	or	in	which	a	large	portion	of	our
potential	for	desirable	development	is	irreversibly	squandered.	On	top	of
this,	there	may	be	existential	risks	associated	with	the	lead-up	to	
a	potential
intelligence	explosion,	arising,	for	example,	from	war	between	countries
competing	to	develop	superintelligence	first.
2
.	There	is	an	important	moment	of	vulnerability	when	the	AI	first	realizes	the
need	for	such	concealment	(an	event	which	we	may	term	
the	conception	of

2
deception
).	This	initial	realization	would	not	itself	be	deliberately	concealed
when	it	occurs.	But	having	had	this	realization,	the	AI	might	move	swiftly
to	hide	the	fact	that	the	realization	has	occurred,	while	setting	up	some
covert	internal	dynamic	(perhaps	disguised	as	some	innocuous	process	that
blends	in	with	all	the	other	complicated	processes	taking	place	in	its	mind)
that	will	enable	it	to	continue	to	plan	its	long-term	strategy	in	privacy.
3

deception
3
.	Even	human	hackers	can	write	small	and	seemingly	innocuous	programs
that	do	completely	unexpected	things.	(For	examples,	see	some	the	winning
entries	in	the	International	Obfuscated	C	Code	Contest.)
4
.	The	point	that	some	AI	control	measures	could	appear	to	work	within	a
fixed	context	yet	fail	catastrophically	when	the	context	changes	is	also
emphasized	by	Eliezer	Yudkowsky;	see,	e.g.,	Yudkowsky	(2008a).
5
.	The	term	seems	to	have	been	coined	by	science-fiction	writer	Larry	Niven

3
4
5
(1973),	but	is	based	on	real-world	brain	stimulation	reward	experiments;	cf.
Olds	and	Milner	(1954)	and	Oshima	and	Katayama	(2010).	See	also	Ring

and	Orseau	(2011).
6
.	Bostrom	(1997).
7
.	There	might	be	some	possible	implementations	of	a	reinforcement	learning
mechanism	that	would,	when	the	AI	discovers	the	wireheading	solution,
lead	to	a	safe	incapacitation	rather	than	to	infrastructure	profusion.	The
point	is	that	this	could	easily	go	wrong	and	fail	for	unexpected	reasons.
8
.	This	was	suggested	by	Marvin	Minsky	(
vide
	Russell	and	Norvig	[2010,
1039]).
9
.	The	issue	of	which	kinds	of	digital	mind	would	be	conscious,	in	the	sense
of	having	subjective	phenomenal	experience,	or	“qualia”	in	philosopher-
speak,	is	important	in	relation	to	this	point	(though	it	is	irrelevant	to	many
other	parts	of	this	book).	One	open	question	is	how	hard	it	would	be	to
accurately	estimate	how	a	human-like	being	would	behave	in	various
circumstances	without	simulating	its	brain	in	enough	detail	that	the
simulation	is	conscious.	Another	question	is	whether	there	are	generally
useful	algorithms	for	a	superintelligence,	for	instance	reinforcement-

learning	techniques,	such	that	the	implementation	of	these	algorithms
would	generate	qualia.	Even	if	we	judge	the	probability	that	any	such
subroutines	would	be	conscious	to	be	fairly	small,	the	number	of
instantiations	might	be	so	large	that	even	a	small	risk	that	they	might
experience	suffering	ought	to	be	accorded	significant	weight	in	our	moral
calculation.	See	also	Metzinger	(2003,	Chap.	8).
.	Bostrom	(2002a,	2003a);	Elga	(2004).
CHAPTER	9:	THE	CONTROL	PROBLEM
	
1
.	E.g.,	Laffont	and	Martimort	(2002).
2

1
2
.	Suppose	a	majority	of	voters	want	their	country	to	build	some	particular
kind	of	superintelligence.	They	elect	a	candidate	who	promises	to	do	their
bidding,	but	they	might	find	it	difficult	to	ensure	that	the	candidate,	once	in
power,	will	follow	through	on	her	campaign	promise	and	pursue	the	project
in	the	way	that	the	voters	intended.	Supposing	she	is	true	to	her	word,	she

instructs	her	government	to	contract	with	an	academic	or	industry
consortium	to	carry	out	the	work;	but	again	there	are	agency	problems:	the
bureaucrats	in	the	government	department	might	have	their	own	views
about	what	should	be	done	and	may	implement	the	project	in	a	way	that
respects	the	letter	but	not	the	spirit	of	the	leader’s	instructions.	Even	if	the
government	department	does	its	job	faithfully,	the	contracted	scientific
partners	might	have	their	own	separate	agendas.	The	problem	recurs	on
many	levels.	The	director	of	one	of	the	participating	laboratories	might	lie
awake	worrying	about	a	technician	introducing	an	unsanctioned	element
into	the	design—imagining	Dr.	T.	R.	Eason	sneaking	into	his	office	late	one
night,	logging	into	the	project	code	base,	rewriting	a	part	of	the	seed	AI’s
goal	system.	Where	it	was	supposed	to	say	“serve	humanity,”	it	now	says
“serve	Dr.	T.	R.	Eason.”
3
.	
Even	for	superintelligence	development,	though,	there	could	be	a	role	for

3
.	
behavioral	testing—as	one	auxiliary	element	within	a	wider	battery	of
safety	measures.	Should	an	AI	misbehave	in	its	developmental	phase,
something	is	clearly	awry—though,	importantly,	the	converse	does	not
hold.
4
.	In	a	classic	exploit	from	1975,	Steven	Dompier	wrote	a	program	for	the
Altair	8800	that	took	advantage	of	this	effect	(and	the	absence	of	shielding
around	the	microcomputer’s	case).	Running	the	program	caused	the
emission	of	electromagnetic	waves	that	would	produce	music	when	one

hold.
4
held	a	transistor	radio	close	to	the	computer	(Driscoll	2012).	The	young	Bill
Gates,	who	attended	a	demo,	reported	that	he	was	impressed	and	mystified
by	the	hack	(Gates	1975).	There	are	in	any	case	plans	to	design	future	chips
with	built-in	Wi-Fi	capabilities	(Greene	2012).
5
.	It	is	no	light	matter	to	have	held	a	conviction,	which,	had	we	had	an
opportunity	to	act	upon	it,	could	have	resulted	in	the	ruination	of	all	our
cosmic	endowment.	Perhaps	one	could	argue	for	the	following	principle:	if

5
somebody	has	in	the	past	been	certain	on	
N
	occasions	that	a	system	has
been	improved	sufficiently	to	make	it	safe,	and	each	time	it	was	revealed
that	they	were	wrong,	then	on	the	next	occasion	they	are	not	entitled	to
assign	a	credence	greater	than	1/(
N
	+	1)	to	the	system	being	safe.
6
.	In	one	informal	experiment,	the	role	of	the	AI	was	played	by	an	intelligent
human.	Another	individual	played	the	role	of	gatekeeper	and	was	tasked

with	not	letting	the	AI	out	of	the	box.	The	AI	could	communicate	with	the
gatekeeper	only	by	text	and	was	given	two	hours	to	persuade	the	gatekeeper
to	let	it	out.	In	three	cases	out	of	five,	with	different	individuals	playing	the
gatekeeper,	the	AI	escaped	(Yudkowsky	2002).	What	a	human	can	do,	a
superintelligence	can	do	too.	(The	reverse,	of	course,	does	not	hold.	Even	if
the	task	for	a	real	superintelligence	were	harder—maybe	the	gatekeepers
would	be	more	strongly	motivated	to	refrain	from	releasing	the	AI	than	the
individuals	playing	gatekeeper	in	the	experiment—the	superintelligence
might	still	succeed	where	a	human	would	fail.)
7
.	One	should	not	overstate	the	marginal	amount	of	safety	that	could	be
gained	in	this	way.	Mental	imagery	can	substitute	for	graphical	display.
Consider	the	impact	books	can	have	on	people—and	books	are	not	even
interactive.
8
.	See	also	Chalmers	(2010).	It	would	be	a	mistake	to	infer	from	this	that	there
is	
no

7
8
is	
no
	possible	use	in	building	a	system	that	will	never	be	observed	by	any
outside	entity.	One	might	place	a	final	value	on	what	goes	on	inside	such	a
system.	Also,	other	people	might	have	preferences	about	what	goes	on
inside	such	a	system,	and	might	therefore	be	influenced	by	its	creation	or
the	promise	of	its	creation.	Knowledge	of	the	existence	of	certain	kinds	of
isolated	systems	(ones	containing	observers)	can	also	induce	anthropic
uncertainty	in	outside	observers,	which	may	influence	their	behavior.
9

no
9
.	One	might	wonder	why	social	integration	is	considered	a	form	of	capability
control.	Should	it	not	instead	be	classified	as	a	motivation	selection	method
on	the	ground	that	it	involves	seeking	to	influence	a	system’s	behavior	by
means	of	incentives?	We	will	look	closely	at	motivation	selection	presently;
but,	in	answer	to	this	question,	we	are	construing	motivation	selection	as	a
cluster	of	control	methods	that	work	by	selecting	or	shaping	a	system’s	final

9
goals—goals	sought	for	their	own	sakes	rather	than	for	instrumental
reasons.	Social	integration	does	not	target	a	system’s	final	goals,	so	it	is	not
motivation	selection.	Rather,	social	integration	aims	to	limit	the	system’s
effective	capabilities:	it	seeks	to	render	the	system	incapable	of	achieving	a
certain	set	of	outcomes—outcomes	in	which	the	system	attains	the	benefits
of	defection	without	suffering	the	associated	penalties	(retribution,	and	loss
of	the	gains	from	collaboration).	The	hope	is	that	by	limiting	which

outcomes	the	system	is	able	to	attain,	the	system	will	find	that	the	most
effective	remaining	means	of	realizing	its	final	goals	is	to	behave

cooperatively.
.	This	approach	may	be	somewhat	more	promising	in	the	case	of	an	emulation
believed	to	have	anthropomorphic	motivations.
.	I	owe	this	idea	to	Carl	Shulman.
.	Creating	a	cipher	certain	to	withstand	a	superintelligent	code-breaker	is	a
nontrivial	challenge.	For	example,	traces	of	random	numbers	might	be	left
in	some	observer’s	brain	or	in	the	microstructure	of	the	random	generator,
from	whence	the	superintelligence	can	retrieve	them;	or,	if	pseudorandom
numbers	are	used,	the	superintelligence	might	guess	or	discover	the	seed
from	
which	they	were	generated.	Further,	the	superintelligence	could	build
large	quantum	computers,	or	even	discover	unknown	physical	phenomena
that	could	be	used	to	construct	new	kinds	of	computers.
.	The	AI	could	wire	itself	to	
believe
	that	it	had	received	a	reward	tokens,	but
this	should	not	make	it	wirehead	if	it	is	designed	to	want	the	reward	tokens
(as	opposed	to	wanting	to	be	in	a	state	in	which	it	has	certain	beliefs	about
the	reward	tokens).

from	
believe
.	For	the	original	article,	see	Bostrom	(2003a).	See	also	Elga	(2004).
.	Shulman	(2010a).
.	Basement-level	reality	presumably	contains	more	computational	resources
than	simulated	reality,	since	any	computational	processes	occurring	in	a
simulation	are	also	occurring	on	the	computer	running	the	simulation.
Basement-level	reality	might	also	contain	a	wealth	of	other	physical
resources	which	could	be	hard	for	simulated	agents	to	access—agents	that
exist	only	at	the	indulgence	of	powerful	simulators	who	may	have	other

uses	in	mind	for	those	resources.	(Of	course,	the	inference	here	is	not
strictly	deductively	valid:	in	principle,	it	could	be	the	case	that	universes	in
which	simulations	are	run	contain	so	much	more	resources	that	simulated
civilizations	on	average	have	access	to	more	resources	than	non-simulated
civilizations,	even	though	each	non-simulated	civilization	that	runs
simulations	has	more	resources	than	all	the	civilizations	it	simulates	do
combined.)
.	There	are	various	further	esoteric	considerations	that	might	bear	on	this

matter,	the	implications	of	which	have	not	yet	been	fully	analyzed.	These
considerations	may	ultimately	be	crucially	important	in	developing	an	all-
things-considered	approach	to	dealing	with	the	prospect	of	an	intelligence
explosion.	However,	it	seems	unlikely	that	we	will	succeed	in	figuring	out
the	practical	import	of	such	esoteric	arguments	unless	we	have	first	made
some	progress	on	the	more	mundane	kinds	of	consideration	that	are	the
topic	of	most	of	this	book.
.	Cf.,	e.g.,	Quine	and	Ullian	(1978).
.	Which	an	AI	might	investigate	by	considering	the	performance
characteristics	of	various	basic	computational	functionalities,	such	as	the
size	and	capacity	of	various	data	buses,	the	time	it	takes	to	access	different
parts	of	memory,	the	incidence	of	random	bit	flips,	and	so	forth.
.	Perhaps	the	prior	could	be	(a	computable	approximation	of)	the	Solomonoff
prior,	which	assigns	probability	to	possible	worlds	on	the	basis	of	their
algorithmic	complexity.	See	Li	and	Vitányi	(2008).
.	The	moment	
after

after
	the	conception	of	deception,	the	AI	might	contrive	to
erase	the	trace	of	its	mutinous	thought.	It	is	therefore	important	that	this
tripwire	operate	continuously.	It	would	also	be	good	practice	to	use	a	“flight
recorder”	that	stores	a	complete	trace	of	all	the	AI’s	activity	(including
exact	timing	of	keyboard	input	from	the	programmers),	so	that	its	trajectory
can	be	retraced	or	analyzed	following	an	automatic	shutdown.	The
information	could	be	stored	on	a	write-once-read-many	medium.

after
.	Asimov	(1942).	To	the	three	laws	were	later	added	a	“Zeroth	Law”:	“(0)	A
robot	may	not	harm	humanity,	or,	by	inaction,	allow	humanity	to	come	to
harm”	(Asimov	1985).
.	Cf.	Gunn	(1982).
.	Russell	(1986,	161f).
.	Similarly,	although	some	philosophers	have	spent	entire	careers	trying	to
carefully	formulate	deontological	systems,	new	cases	and	consequences
occasionally	come	to	light	that	necessitate	revisions.	For	example,
deontological	moral	philosophy	has	in	recent	years	been	reinvigorated

through	the	discovery	of	a	fertile	new	class	of	philosophical	thought

experiments,	“trolley	problems,”	which	reveal	many	subtle	interactions
among	our	intuitions	about	the	moral	significance	of	the	acts/omissions
distinction,	the	distinction	between	intended	and	unintended	consequences,
and	other	such	matters;	see,	e.g.,	Kamm	(2007).
.	Armstrong	(2010).
.	As	a	rule	of	thumb,	if	one	plans	to	use	multiple	safety	mechanisms	to	contain
an	AI,	it	may	be	wise	to	work	on	each	one	
as	if
	it	were	intended	to	be	the
sole	safety	mechanism	and	
as	if
	it	were	therefore	required	to	be	individually
sufficient.	If	one	puts	a	leaky	bucket	inside	another	leaky	bucket,	the	water
still	comes	out.
.	A	variation	of	the	same	idea	is	to	build	the	AI	so	that	it	is	continuously
motivated	to	act	on	its	best	guesses	about	what	the	implicitly	defined
standard	is.	In	this	setup,	the	AI’s	final	goal	is	
always	to	act	on	the
implicitly	defined	standard,	and	it	pursues	an	investigation	into	what	this
standard	is	only	for	instrumental	reasons.
CHAPTER	10:	ORACLES,	GENIES,	SOVEREIGNS,	TOOLS
	
1

1
.	These	names	are,	of	course,	anthropomorphic	and	should	not	be	taken
seriously	as	analogies.	They	are	just	meant	as	labels	for	some	
prima	facie
different	concepts	of	possible	system	types	that	one	might	consider	trying	to
build.
2
.	In	response	to	a	question	about	the	outcome	of	the	next	election,	one	would
not	wish	to	be	served	with	a	comprehensive	list	of	the	projected	position
and	momentum	vectors	of	nearby	particles.
3
.	Indexed	to	a	particular	instruction	set	on	a	particular	machine.
4
.	Kuhn	(1962);	de	Blanc	(2011).
5

build.
2
3
4
5
.	It	would	be	harder	to	apply	such	a	“consensus	method”	to	genies	or
sovereigns,	because	there	may	often	be	numerous	sequences	of	basic
actions	(such	as	sending	particular	patterns	of	electrical	signals	to	the
system’s	actuators)	that	would	be	almost	exactly	equally	effective	at

achieving	a	given	objective;	whence	slightly	different	agents	may
legitimately	choose	slightly	different	actions,	resulting	in	a	failure	to	reach
consensus.	By	contrast,	with	appropriately	formulated	questions,	there
would	usually	be	a	small	number	of	suitable	answer	options	(such	as	“yes”
and	“no”).	(On	the	concept	of	a	Schelling	point,	also	referred	to	as	a	“focal
point,”	see	Schelling	[1980].)
6
.	Is	not	the	world	economy	in	some	respects	analogous	to	a	weak	genie,
albeit	one	that	charges	for	its	services?	A	vastly	bigger	economy,	such	as
might	develop	in	the	future,	might	then	approximate	a	genie	with	collective
One	important	respect	in	which	the	current	economy	is	
unlike
	a	genie	is
that	although	I	can	(for	a	fee)	command	the	economy	to	deliver	a	pizza	to
my	door,	I	cannot	command	it	to	deliver	peace.	The	reason	is	not	that	the
economy	is	insufficiently	powerful,	but	that	it	is	insufficiently	coordinated.
In	this	respect,	the	economy	resembles	an	
assembly
	of	genies	serving

unlike
	a	genie	is
assembly
different	masters	(with	competing	agendas)	more	than	it	resembles	a	single
genie	or	any	other	type	of	unified	agent.	Increasing	the	total	power	of	the
economy	by	making	each	constituent	genie	more	powerful,	or	by	adding
more	genies,	would	not	necessarily	render	the	economy	more	capable	of
delivering	peace.	In	order	to	function	like	a	superintelligent	genie,	the
economy	would	not	only	need	to	grow	in	its	ability	to	inexpensively
produce	goods	and	services	(including	ones	that	require	radically	new

technology),	it	would	also	need	to	become	better	able	to	solve	global
coordination	problems.
7
.	If	the	genie	were	somehow	incapable	of	not	obeying	a	subsequent
command—and	somehow	incapable	of	reprogramming	itself	to	get	rid	of
this	susceptibility—then	it	could	act	to	prevent	any	new	command	from
being	issued.
8
.	Even	an	oracle	that	is	limited	to	giving	yes/no	answers	could	be	used	to
facilitate	the	search	for	a	genie	or	sovereign	AI,	or	indeed	could	be	used

7
8
directly	as	a	component	in	such	an	AI.	The	oracle	could	also	be	used	to
produce	the	actual	code	for	such	an	AI	if	a	sufficiently	large	number	of
questions	can	be	asked.	A	series	of	such	questions	might	take	roughly	the
following	form:	“In	the	binary	version	of	the	code	of	the	first	AI	that	you
thought	of	that	would	constitute	a	genie,	is	the	
n
th	symbol	a	zero?”

9
.	One	could	imagine	a	slightly	more	complicated	oracle	or	genie	that	accepts
questions	or	commands	only	if	they	are	issued	by	a	designated	authority,
though	this	would	still	leave	open	the	possibility	of	that	authority	becoming
corrupted	or	being	blackmailed	by	a	third	party.
.	John	Rawls,	a	leading	political	philosopher	of	the	twentieth	century,
famously	employed	the	expository	device	of	a	veil	of	ignorance	as	a	way	of
characterizing	the	kinds	of	preference	that	should	be	taken	into	account	in
the	formulation	of	a	social	contract.	Rawls	suggested	that	we	should
imagine	we	were	choosing	a	social	contract	from	behind	a	veil	of	ignorance
that	prevents	us	from	knowing	which	person	we	will	be	and	which	social
role	we	will	occupy,	the	idea	being	that	in	such	a	situation	we	would	have
to	think	about	which	society	would	be	generally	fairest	and	most	desirable
without	regard	to	our	egoistic	interests	and	self-serving	biases	that	
might
otherwise	make	us	prefer	a	social	order	in	which	we	ourselves	enjoy	unjust

might
privileges.	See	Rawls	(1971).
.	Karnofsky	(2012).
.	A	possible	exception	would	be	software	hooked	up	to	sufficiently	powerful
actuators,	such	as	software	in	early	warning	systems	if	connected	directly	to
nuclear	warheads	or	to	human	officers	authorized	to	launch	a	nuclear	strike.
Malfunctions	in	such	software	can	result	in	high-risk	situations.	This	has
happened	at	least	twice	within	living	memory.	On	November	9,	1979,	a
computer	problem	led	NORAD	(North	American	Aerospace	Defense
Command)	to	make	a	false	report	of	an	incoming	full-scale	Soviet	attack	on

the	United	States.	The	USA	made	emergency	retaliation	preparations	before
data	from	early-warning	radar	systems	showed	that	no	attack	had	been
launched	(McLean	and	Stewart	1979).	On	September	26,	1983,	the
malfunctioning	Soviet	Oko	nuclear	early-warning	system	reported	an
incoming	US	missile	strike.	The	report	was	correctly	identified	as	a	false
alarm	by	the	duty	officer	at	the	command	center,	Stanislav	Petrov:	a
decision	that	has	been	credited	with	preventing	thermonuclear	war

(Lebedev	2004).	It	appears	that	a	war	would	probably	have	fallen	short	of
causing	human	extinction,	even	if	it	had	been	fought	with	the	combined
arsenals	held	by	all	the	nuclear	powers	at	the	height	of	the	Cold	War,
though	it	would	have	ruined	civilization	and	caused	unimaginable	death	and
suffering	(Gaddis	1982;	Parrington	1997).	But	bigger	stockpiles	might	be
accumulated	in	future	arms	races,	or	even	deadlier	weapons	might	be

invented,	or	our	models	of	the	impacts	of	a	nuclear	Armageddon
(particularly	of	the	severity	of	the	consequent	nuclear	winter)	might	be
wrong.
.	This	approach	could	fit	the	category	of	a	direct-specification	rule-based
control	method.
.	The	situation	is	essentially	the	same	if	the	solution	criterion	specifies	a
goodness	
measure
	rather	than	a	sharp	cutoff	for	what	counts	as	a	solution.
.	An	advocate	for	the	oracle	approach	could	insist	that	there	is	at	least	a
possibility	that	the	user	would	spot	the	flaw	in	the	proffered	solution—
recognize	that	it	fails	to	match	the	user’s	intent	even	while	satisfying	the
formally	specified	success	criteria.	The	likelihood	of	catching	the	error	at
this	stage	would	depend	on	various	factors,	including	how	humanly
understandable	the	oracle’s	outputs	are	and	how	charitable	it	is	in	selecting
which	features	of	the	potential	outcome	to	bring	to	the	user’s	attention.
Alternatively,	instead	of	relying	on	the	oracle	itself	to	provide	these

functionalities,	one	might	try	to	build	a	separate	tool	to	do	this,	a	tool	that
could	inspect	the	pronouncements	of	the	oracle	and	show	us	in	a	helpful
way	what	would	happen	if	we	acted	upon	them.	But	to	do	to	this	in	full
generality	would	require	another	superintelligent	oracle	whose	divinations
we	would	then	have	to	trust;	so	the	reliability	problem	would	not	have	been
solved,	only	displaced.	One	might	seek	to	gain	an	increment	of	safety
through	the	use	of	multiple	oracles	to	perform	peer	review,	but	this	does	not

protect	in	cases	where	all	the	oracles	fail	in	the	same	way—as	may	happen
if,	for	instance,	they	have	all	been	given	the	same	formal	specification	of
what	counts	as	a	satisfactory	solution.
.	Bird	and	Layzell	(2002)	and	Thompson	(1997);	also	Yaeger	(1994,	13–14).
.	Williams	(1966).
.	Leigh	(2010).
.	This	example	is	borrowed	from	Yudkowsky	(2011).
.	Wade	(1976).	Computer	experiments	have	also	been	conducted	with
simulated	evolution	designed	to	resemble	aspects	of	biological	evolution—

again	with	sometimes	strange	results	(see,	e.g.,	Yaeger	[1994]).
.	With	sufficiently	great—finite	but	physically	implausible—amounts	of
computing	power,	it	
would
	probably	be	possible	to	achieve	general
superintelligence	with	currently	available	algorithms.	(Cf.,	e.g.,	the	AIXI
tl
system;	Hutter	[2001].)	But	even	the	continuation	of	Moore’s	law	for
another	hundred	years	would	not	suffice	to	attain	the	required	levels	of
computing	power	to	achieve	this.
CHAPTER	11:	MULTIPOLAR	SCENARIOS
	
1
.	Not	because	this	is	necessarily	the	most	likely	or	the	most	desirable	type	of
scenario,	but	because	it	is	the	one	easiest	to	analyze	with	the	toolkit	of
standard	economic	theory,	and	thus	a	convenient	starting	point	for	our
discussion.
2
.	
American	Horse	Council	(2005).	See	also	Salem	and	Rowan	(2001).
3
.	Acemoglu	(2003);	Mankiw	(2009);	Zuleta	(2008).
4
.	Fredriksen	(2012,	8);	Salverda	
et	al.
	(2009,	133).
5
.	It	is	also	essential	for	at	least	some	of	the	capital	to	be	invested	in	assets

1
2
.	
3
4
et	al.
5
that	rise	with	the	general	tide.	A	diversified	asset	portfolio,	such	as	shares	in
an	index	tracker	fund,	would	increase	the	chances	of	not	entirely	missing
out.
6
.	Many	of	the	European	welfare	systems	are	
unfunded
,	meaning	that
pensions	are	paid	from	ongoing	current	workers’	contributions	and	taxes
rather	than	from	a	pool	of	savings.	Such	schemes	would	not	automatically
meet	the	requirement—in	case	of	sudden	massive	unemployment,	the
revenues	from	which	the	benefits	are	paid	could	dry	up.	However,

out.
6
unfunded
governments	may	choose	to	make	up	the	shortfall	from	other	sources.
7
.	American	Horse	Council	(2005).
8
.	Providing	7	billion	people	an	annual	pension	of	$90,000	would	cost	$630
trillion	a	year,	which	is	ten	times	the	current	world	GDP.	Over	the	last

hundred	years,	world	GDP	has	increased	about	nineteenfold	from	around	$2
trillion	in	1900	to	37	trillion	in	2000	(in	1990	int.	dollars)	according	to
Maddison	(2007).	So	if	the	growth	rates	we	have	seen	over	the	past	hundred
years	continued	for	the	next	two	hundred	years,	while	population	remained
constant,	then	providing	everybody	with	an	annual	$90,000	pension	would
cost	about	3%	of	world	GDP.	An	intelligence	explosion	might	make	this
amount	of	growth	happen	in	a	much	shorter	time	span.	See	also	Hanson
(1998a,	1998b,	2008).
9
.	And	perhaps	as	much	as	a	millionfold	over	the	past	70,000	years	if	there
was	a	severe	population	bottleneck	around	that	time,	as	has	been
speculated.	See	Kremer	(1993)	and	Huff	
et	al.
	(2010)	for	more	data.
.	Cochran	and	Harpending	(2009).	See	also	Clark	(2007)	and,	for	a	critique,
Allen	(2008).
.	Kremer	(1993).
.	Basten	
et	al.
	(2013).	Scenarios	in	which	there	is	a	continued	rise	are	also
possible.	In	general,	the	uncertainty	of	such	projections	increases	greatly

9
et	al.
.	Basten	
et	al.
beyond	one	or	two	generations	into	the	future.
.	Taken	globally,	the	total	fertility	rate	at	replacement	was	2.33	children	per
woman	in	2003.	This	number	comes	from	the	fact	that	it	takes	two	children
per	woman	to	replace	the	parents,	plus	a	“third	of	a	child”	to	make	up	for
(1)	the	higher	probability	of	boys	being	born,	and	(2)	early	mortality	prior
to	the	end	of	their	fertile	life.	For	developed	nations,	the	number	is	smaller,
around	2.1,	because	of	lower	mortality	rates.	(See	Espenshade	
et	al.
	[2003,

et	al.
	[2003,
Introduction,	Table	1,	580].)	The	population	in	most	developed	countries
would	decline	if	it	were	not	for	immigration.	A	few	notable	examples	of
countries	with	sub-replacement	fertility	rates	are:	Singapore	at	0.79	(lowest
in	the	world),	Japan	at	1.39,	People’s	Republic	of	China	at	1.55,	European
Union	at	1.58,	Russia	at	1.61,	Brazil	at	1.81,	Iran	at	1.86,	Vietnam	at	1.87,
and	the	United	Kingdom	at	1.90.	Even	the	U.S.	population	would	probably

et	al.
	[2003,
decrease	slightly	with	a	fertility	rate	of	2.05.	(See	CIA	[2013].)
.	The	fullness	of	time	might	occur	many	billions	of	years	from	now.
.	Carl	Shulman	points	out	that	if	biological	humans	count	on	living	out	their
natural	lifespans	alongside	the	digital	economy,	they	need	to	assume	not

only	that	the	political	order	in	the	digital	sphere	would	be	protective	of
human	interests	but	that	it	would	remain	so	over	very	long	periods	of	time
(Shulman	2012).	For	example,	if	events	in	the	digital	sphere	unfolds	a
thousand	times	faster	than	on	the	outside,	then	a	biological	human	would
have	to	rely	on	the	digital	body	politic	holding	steady	for	50,000	years	of
internal	change	and	churn.	Yet	if	the	digital	political	world	were	anything
like	ours,	there	would	be	a	great	many	revolutions,	wars,	and	catastrophic
upheavals	during	those	millennia	that	would	probably	inconvenience
biological	humans	on	the	outside.	Even	a	0.01%	risk	per	year	of	a	global
thermonuclear	war	or	similar	cataclysm	would	entail	a	near	certain	loss	for
the	biological	humans	living	out	their	lives	in	slowmo	sidereal	time.	To
overcome	this	problem,	a	more	stable	order	in	the	digital	realm	would	be
required:	perhaps	a	singleton	that	gradually	improves	its	own	stability.

.	One	might	think	that	even	if	machines	were	far	more	efficient	than	humans,
there	would	still	be	
some
	wage	level	at	which	it	would	be	profitable	to
employ	a	human	worker;	say	at	1	cent	an	hour.	If	this	were	the	only	source
of	income	for	humans,	our	species	would	go	extinct	since	human	beings
cannot	survive	on	1	cent	an	hour.	But	humans	also	get	income	from	capital.
Now,	if	we	are	assuming	that	population	grows	until	total	income	is	at
subsistence	level,	one	might	think	this

some
subsistence	level,	one	might	think	this	
would	be	a	state	in	which	humans
would	be	working	hard.	For	example,	suppose	subsistence	level	income	is
$1/day.	Then,	it	might	seem,	population	would	grow	until	per	person	capital
provided	only	a	90	cents	per	day	income,	which	people	would	have	to
supplement	with	ten	hours	of	hard	labor	to	make	up	the	remaining	10	cents.
However,	this	need	not	be	so,	because	the	subsistence	level	income	depends
on	the	amount	of	work	that	is	done:	harder-working	humans	burn	more
calories.	Suppose	that	each	hour	of	work	increases	food	costs	by	2	cents.

We	then	have	a	model	in	which	humans	are	idle	in	equilibrium.
.	It	might	be	thought	that	a	caucus	as	enfeebled	as	this	would	be	unable	to	vote
and	to	otherwise	defend	its	entitlements.	But	the	pod-dwellers	could	give
power	of	attorney	to	AI	fiduciaries	to	manage	their	affairs	and	represent
their	political	interests.	(This	part	of	the	discussion	in	this	section	is
premised	on	the	assumption	that	property	rights	are	respected.)
.	It	is	unclear	what	is	the	best	term.	“Kill”	may	suggest	more	active	brutality

than	is	warranted.	“End”	may	be	too	euphemistic.	One	complication	is	that
there	are	two	potentially	separate	events:	ceasing	to	actively	run	a	process,

and	erasing	the	information	template.	A	human	death	normally	involves
both	events,	but	for	an	emulation	they	can	come	apart.	That	a	program
temporarily
	ceases	to	run	may	be	no	more	consequential	than	that	a	human
sleeps:	but	to	
permanently
	cease	running	may	be	the	equivalent	of	entering
a	permanent	coma.	Still	further	complications	arise	from	the	fact	that
emulations	can	be	copied	and	that	they	can	run	at	different	speeds:
possibilities	with	no	direct	analogs	in	human	experience.	(Cf.	Bostrom
[2006b];	Bostrom	and	Yudkowsky	[forthcoming].)
.	There	will	be	a	trade-off	between	total	parallel	computing	power	and
computational	speed,	as	the	highest	computational	speeds	will	be	attainable
only	at	the	expense	of	a	reduction	in	power	efficiency.	This	will	be
especially	true	after	one	enters	the	era	of	reversible	computing.
.	An	emulation	could	be	tested	by	leading	it	into	temptation.	By	repeatedly
testing	how	an	emulation	started	from	a	certain	prepared	state	reacts	to

various	sequences	of	stimuli,	one	could	obtain	high	confidence	in	the
reliability	of	that	emulation.	But	the	further	the	mental	state	is	subsequently
allowed	to	develop	away	from	its	validated	starting	point,	the	less	certain
one	could	be	that	it	would	remain	reliable.	(In	particular,	since	a	clever
emulation	might	surmise	it	is	sometimes	in	a	simulation,	one	would	need	to
be	cautious	about	extrapolating	its	behavior	into	situations	where	its
simulation	hypothesis	would	weigh	less	heavily	in	its	decision-making.)

.	Some	emulations	might	identify	with	their	clan—i.e.	all	of	their	copies	and
variations	derived	from	the	same	template—rather	with	any	one	particular
instantiation.	Such	an	emulation	might	not	regard	its	own	termination	as	a
death	event,	if	it	knew	that	other	clan	members	would	survive.	Emulations
may	know	that	they	will	get	reverted	to	a	particular	stored	state	at	the	end
of	the	day	and	lose	that	day’s	memories,	but	be	as	little	put	off	by	this	as	the
partygoer	who	knows	she	will	awake	the	next	morning	without	any

recollection	of	the	previous	night:	regarding	this	as	retrograde	amnesia,	not
death.
.	An	ethical	evaluation	might	take	into	account	many	other	factors	as	well.
Even	if	all	the	workers	were	constantly	well	pleased	with	their	condition,
the	outcome	might	still	be	deeply	morally	objectionable	on	other	grounds—
though	
which
	other	grounds	is	a	matter	of	dispute	between	rival	moral
theories.	But	any	plausible	assessment	would	consider	subjective	well-

being	to	be	one	important	factor.	See	also	Bostrom	and	Yudkowsky
(forthcoming).
.	World	Values	Survey	(2008).
.	Helliwell	and	Sachs	(2012).
.	Cf.	Bostrom	(2004).	See	also	Chislenko	(1996)	and	Moravec	(1988).
.	It	is	hard	to	say	whether	the	information-processing	structures	that	would
emerge	in	this	kind	of	scenario	would	be	conscious	(in	the	sense	of	having
qualia,	phenomenal	experience).	The	reason	this	is	hard	is	partly	our
empirical	ignorance	about	which	cognitive	entities	would	arise	and	partly
our	philosophical	ignorance	about	which	types	of	structure	have
consciousness.	One	could	try	to	reframe	the	question,	and	instead	of	asking
whether	the	future	entities	would	be	conscious,	one	could	ask	whether	the
future	entities	would	have	moral	status;	or	one	could	ask	whether	they
would	be	such	that	we	have	preferences	about	their	“well-being.”	But	these
questions	may	be	no	easier	to	answer	than	the	question	about	consciousness
—in	fact,	they	might	require	an	answer	to	the	consciousness	question

inasmuch	as	moral	status	or	our	preferences	depend	on	whether	the	entity	in
question	can	subjectively	experience	its	condition.
.	
For	an	argument	that	both	geological	and	human	history	manifest	such	a
trend	toward	greater	complexity,	see	Wright	(2001).	For	an	opposing
argument	(criticized	in	
Chapter	9
	of	Wright’s	book),	see	Gould	(1990).	See
also	Pinker	(2011)	for	an	argument	that	we	are	witnessing	a	robust	long-
term	trend	toward	decreasing	violence	and	brutality.

.	
Chapter	9
.	For	more	on	observation	selection	theory,	see	Bostrom	(2002a).
.	Bostrom	(2008a).	A	much	more	careful	examination	of	the	details	of	our
evolutionary	history	would	be	needed	to	circumvent	the	selection	effect.
See,	e.g.,	Carter	(1983,	1993);	Hanson	(1998d);	Ćirković	
et	al.
	(2010).
.	Kansa	(2003).
.	E.g.,	Zahavi	and	Zahavi	(1997).
.	See	Miller	(2000).

.	Kansa	(2003).	For	a	provocative	take,	see	also	Frank	(1999).
.	It	is	not	obvious	how	best	to	measure	the	degree	of	global	political
integration.	One	perspective	would	be	that	whereas	a	hunter–gatherer	tribe
might	have	integrated	a	hundred	individuals	into	a	decision-making	entity,
the	largest	political	entities	today	contain	more	than	a	billion	individuals.
This	would	amount	to	a	difference	of	seven	orders	of	magnitude,	with	only
one	additional	magnitude	to	go	before	the	entire	world	population	is
contained	within	a	single	political	entity.	However,	at	the	time	when	the
tribe	was	the	largest	scale	of	integration,	the	world	population	was	much
smaller.	The	tribe	might	have	contained	as	much	as	a	thousandth	of	the
individuals	then	living.	This	would	make	the	increase	in	the	scale	of
political	integration	as	little	as	two	orders	of	magnitude.	Looking	at	the
fraction	of	world	population	that	is	politically	integrated,	rather	than	at
absolute	numbers,	seems	appropriate	in	the	present	context	(particularly	as

the	transition	to	machine	intelligence	may	cause	a	population	explosion,	of
emulations	or	other	digital	minds).	But	there	have	also	been	developments
in	global	institutions	and	networks	of	collaboration	outside	of	formal	state
structures,	which	should	also	be	taken	into	account.
.	One	of	the	reasons	for	supposing	that	the	first	machine	intelligence
revolution	will	be	swift—the	possible	existence	of	a	hardware	overhang—
does	not	apply	here.	However,	there	could	be	other	sources	of	rapid	gain,
such	as	a	dramatic	breakthrough	in	software	associated	with	transitioning

from	emulation	to	purely	synthetic	machine	intelligence.
.	Shulman	(2010b).
.	How	the	
pro	et	contra
	would	balance	out	might	depend	on	what	kind	of
work	the	superorganism	is	trying	to	do,	and	how	generally	capable	the	most
generally	capable	available	emulation	template	is.	Part	of	the	reason	many
different	types	of	human	beings	are	needed	in	large	organizations	today	is
that	humans	who	are	very	talented	in	many	domains	are	rare.
.	It	is	of	course	very	easy	to	make	multiple	copies	of	a	software	agent.	But

.	How	the	
note	that	copying	is	not	in	general	sufficient	to	ensure	that	the	copies	have
the	same	final	goals.	In	order	for	two	agents	to	have	the	same	final	goals	(in
the	relevant	sense	of	“same”),	the	goals	must	coincide	in	their	
indexical
elements.	If	Bob	is	selfish,	a	copy	of	Bob	will	likewise	be	selfish.	Yet	their

goals	do	not	coincide:	Bob	cares	about	Bob	whereas	Bob-copy	cares	about
Bob-copy.
.	Shulman	(2010b,	6).
.	This	might	be	more	feasible	for	biological	humans	and	whole	brain
emulations	than	for	arbitrary	artificial	intelligences,	which	might	be
constructed	so	as	to	have	hidden	compartments	or	functional	dynamics	that
may	be	very	hard	to	discover.	On	the	other	hand,	
AIs	specifically	built	to	be
transparent
	should	allow	for	more	thoroughgoing	inspection	and
verification	than	is	possible	with	brain-like	architectures.	Social	pressures
may	encourage	AIs	to	expose	their	source	code,	and	to	modify	themselves
to	render	themselves	transparent—especially	if	being	transparent	is	a
precondition	to	being	trusted	and	thus	to	being	given	the	opportunity	to
partake	in	beneficial	transactions.	Cf.	Hall	(2007).
.	Some	other	issues	that	seem	relatively	minor,	especially	in	cases	where	the
stakes	are	enormous	(as	they	are	for	the	key	global	coordination	failures),

include	the	search	cost	of	finding	policies	that	could	be	of	mutual	interest,
and	the	possibility	that	some	agents	might	have	a	basic	preference	for
“autonomy”	in	a	form	that	would	be	reduced	by	entering	into
comprehensive	global	treaties	that	have	monitoring	and	enforcement
mechanisms	attached.
.	An	AI	might	perhaps	achieve	this	by	modifying	itself	appropriately	and	then
giving	observers	read-only	access	to	its	source	code.	A	machine	intelligence
with	a	more	opaque	architecture	
(such	as	an	emulation)	might	perhaps

achieve	it	by	publicly	applying	to	itself	some	motivation	selection	method.
Alternatively,	an	external	coercive	agency,	such	as	a	superorganism	police
force,	might	perhaps	be	used	not	only	to	enforce	the	implementation	of	a
treaty	reached	between	different	parties,	but	also	internally	by	a	single	party
to	commit	itself	to	a	particular	course	of	action.
.	Evolutionary	selection	might	have	favored	threat-ignorers	and	even
characters	visibly	so	highly	strung	that	they	would	rather	fight	to	the	death
than	suffer	the	slightest	discomfiture.	Such	a	disposition	might	bring	its

bearer	valuable	signaling	benefits.	(Any	such	instrumental	rewards	of
having	the	disposition	need	of	course	play	no	part	in	the	agent’s	conscious
motivation:	he	may	value	justice	or	honor	as	ends	in	themselves.)

.	A	definitive	verdict	on	these	matters,	however,	must	await	further	analysis.
There	are	various	other	potential	complications	which	we	cannot	explore
here.
CHAPTER	12:	ACQUIRING	VALUES
	
1
.	Various	complications	and	modulations	of	this	basic	idea	could	be
introduced.	We	discussed	one	variation	in	
Chapter	8
—that	of	a	satisficing,
as	opposed	to	maximizing,	agent—and	in	the	next	chapter	we	briefly	touch
on	the	issue	of	alternative	decision	theories.	However,	such	issues	are	not
essential	to	the	thrust	of	this	subsection,	so	we	will	keep	things	simple	by
focusing	here	on	the	case	of	an	expected	utility-maximizing	agent.
2
.	Assuming	the	AI	is	to	have	a	nontrivial	utility	function.	It	would	be	very
easy	to	build	an	agent	that	always	chooses	an	action	that	maximizes
expected	utility	if	its	utility	function	is,	e.g.,	the	constant	function	
U
(
w
)	=	0.
Every	action	would	equally	well	maximize	expected	utility	relative	to	that
utility	function.
3

2
U
(
w
)	=	0.
3
.	Also	because	we	have	forgotten	the	blooming	buzzing	confusion	of	our
early	infancy,	a	time	when	we	could	not	yet	see	very	well	because	our	brain
had	not	yet	learned	to	interpret	its	visual	input.
4
.	See	also	Yudkowsky	(2011)	and	the	review	in	section	5	of	Muehlhauser	and
Helm	(2012).
5
.	It	is	perhaps	just	about	conceivable	that	advances	in	software	engineering
could	eventually	overcome	these	difficulties.	Using	modern	tools,	a	single
programmer	can	produce	software	that	would	have	been	beyond	the	pale	of

4
5
a	sizeable	team	of	developers	forced	to	write	directly	in	machine	code.
Today’s	AI	programmers	gain	expressiveness	from	the	wide	availability	of
high-quality	machine	learning	and	scientific	calculation	libraries,	enabling
someone	to	hack	up,	for	instance,	a	unique-face-counting	webcam
application	by	chaining	libraries	together	that	they	never	could	have	written
on	their	own.	The	accumulation	of	reusable	software,	produced	by
specialists	but	useable	by	non-specialists,	will	give	future	programmers	an
expressiveness	advantage.	For	example,	a	future	robotics	programmer

might	have	ready	access	to	standard	facial	imprinting	libraries,	typical-
office-building-object	collections,	specialized	trajectory	libraries,	and	many
other	functionalities	that	are	currently	unavailable.
6
.	Dawkins	(1995,	132).	The	claim	here	is	not	necessarily	that	the	amount	of
suffering	in	the	natural	world	
outweighs
	the	amount	of	positive	well-being.
7
.	Required	population	sizes	might	be	much	larger	or	much	smaller	than	those
that	existed	in	our	own	ancestry.	See	Shulman	and	Bostrom	(2012).
8
.	If	it	were	easy	to	get	an	equivalent	result	without	harming	large	numbers	of
innocents,	it	would	seem	morally	better	to	do	so.	If,	nevertheless,	digital
persons	are	created	and	made	to	suffer	unjust	harm,	it	may	be	possible	to
compensate	them	for	their	suffering	by	saving	them	to	file	and	later	(when
humanity’s	future	is	secured)	rerunning	them	under	more	favorable
conditions.	Such	restitution	could	be	compared	in	some	ways	to	religious

8
conceptions	of	an	afterlife	in	the	context	of	theological	attempts	to	address
the	evidential	problem	of	evil.
9
.	One	of	the	field’s	leading	figures,	Richard	Sutton,	defines	reinforcement
learning	not	in	terms	of	a	learning	method	but	in	terms	of	a	learning
problem:	any	method	that	is	well	suited	to	solving	that	problem	is
considered	a	reinforcement	learning	method	(Sutton	and	Barto	1998,	4).
The	present	discussion,	in	contrast,	pertains	to	methods	where	the	agent	can

9
be	conceived	of	as	having	the	final	goal	of	maximizing	(some	notion	of)
cumulative	reward.	Since	an	agent	with	some	very	different	kind	of	final
goal	might	be	skilled	at	mimicking	a	reward-seeking	agent	in	a	wide	
range
of	situations,	and	could	thus	be	well	suited	to	solving	reinforcement
learning	problems,	there	could	be	methods	that	would	count	as
“reinforcement	learning	methods”	on	Sutton’s	definition	that	would	not
result	in	a	wireheading	syndrome.	The	remarks	in	the	text,	however,	apply

range
to	most	of	the	methods	actually	used	in	the	reinforcement	learning
.	Even	if,	somehow,	a	human-like	mechanism	could	be	set	up	within	a	human-
like	machine	intellect,	the	final	goals	acquired	by	this	intellect	need	not
resemble	those	of	a	well-adjusted	human,	unless	the	rearing	environment
for	this	digital	baby	also	closely	matched	that	of	an	ordinary	child:
something	that	would	be	difficult	to	arrange.	And	even	with	a	human-like

rearing	environment,	a	satisfactory	result	would	not	be	guaranteed,	since
even	a	subtle	difference	in	innate	dispositions	can	result	in	very	different
reactions	to	a	life	event.	It	may,	however,	be	possible	to	create	a	more
reliable	value-accretion	mechanism	for	human-like	minds	in	the	future
(perhaps	using	novel	drugs	or	brain	implants,	or	their	digital	equivalents).
.	One	might	wonder	why	it	appears	
we	humans
	are	not	trying	to	disable	the
mechanism	that	leads	us	to	acquire	new	final	values.	Several	factors	might
be	at	play.	First,	the	human	motivation	system	is	poorly	described	as	a
coldly	calculating	utility-maximizing	algorithm.	Second,	we	might	not	have
any	convenient	means	of	altering	the	ways	we	acquire	values.	Third,	we
may	have	instrumental	reasons	(arising,	e.g.,	from	social	signaling	needs)
for	sometimes	acquiring	new	final	values—instrumental	values	might	not
be	as	useful	if	our	minds	are	partially	transparent	to	other	people,	or	if	the

cognitive	complexity	of	pretending	to	have	a	different	set	of	final	values
than	we	actually	do	is	too	taxing.	Fourth,	there	are	cases	where	we	
do
actively	resist	tendencies	that	produce	changes	in	our	final	values,	for
instance	when	we	seek	to	resist	the	corrupting	influence	of	bad	company.
Fifth,	there	is	the	interesting	possibility	that	we	place	some	final	value	on
being	the	kind	of	agent	that	can	acquire	new	final	values	in	normal	human
ways.
.	Or	one	might	try	to	design	the	motivation	system	so	that	the	AI	is	indifferent
to	such	replacement;	see	Armstrong	(2010).

do
ways.
.	We	will	here	draw	on	some	elucidations	made	by	Daniel	Dewey	(2011).
Other	background	ideas	contributing	to	this	framework	have	been
developed	by	Marcus	Hutter	(2005)	and	Shane	Legg	(2008),	Eliezer
Yudkowsky	(2001),	Nick	Hay	(2005),	Moshe	Looks,	and	Peter	de	Blanc.
.	To	avoid	unnecessary	complications,	we	confine	our	attention	to
deterministic	agents	that	do	not	discount	future	rewards.
.	Mathematically,	an	agent’s	behavior	can	be	formalized	as	an	
agent	function
,

,
which	maps	each	possible	interaction	history	to	an	action.	Except	for	the
very	simplest	agents,	it	is	infeasible	to	represent	the	agent	function
explicitly	as	a	lookup	table.	Instead,	the	agent	is	given	some	way	of
computing	which	action	to	perform.	Since	there	are	many	ways	of
computing	the	same	agent	function,	this	leads	to	a	finer	individuation	of	an

agent	as	an	
agent	program
.	An	agent	program	is	a	specific	program	or
algorithm	that	computes	an	action	for	any	given	interaction	history.	While	it
is	often	mathematically	convenient	and	useful	to	think	of	an	agent	program
that	interacts	with	some	formally	specified	environment,	it	is	important	to
remember	that	this	is	an	idealization.	Real	agents	are	physically
instantiated.	This	means	not	only	that	the	agent	interacts	with	the
environment	via	its	sensors	and	effectors,	but	also	that	the	agent’s	“brain”
or	controller	
is	itself	part	of	physical	reality
.	Its	operations	can	therefore	in
principle	be	affected	by	external	physical	interferences	(and	not	only	by
receiving	percepts	from	its	sensors).	At	some	point,	therefore,	it	becomes
necessary	to	view	an	agent	as	an	
agent	implementation
.	An	agent
implementation	is	a	physical	structure	that,	in	the	absence	of	interference
from	its	environment,	implements	an	agent	function.	(This	definition
follows	Dewey	[2011].)

.	An	agent
.	Dewey	proposes	the	following	optimality	notion	for	a	value	learning	agent:
	
Here,	
P
1
	and	
P
2
	are	two	probability	functions.	The	second	summation
ranges	over	some	suitable	class	of	utility	functions	over	possible	interaction
histories.	In	the	version	presented	in	the	text,	we	have	made	explicit	some
dependencies	as	well	as	availed	ourselves	of	the	simplifying	possible
worlds	notation.
	
.	It	should	be	noted	that	the	set	of	utility	functions	
	should	be	such	that
utilities	can	be	compared	and	averaged.	In	general,	this	is	problematic,	and

Here,	
P
1
	and	
P
2
	
it	is	not	always	obvious	how	to	represent	
different	moral	theories	of	the
good	in	terms	of	cardinal	utility	functions.	See,	e.g.,	MacAskill	2010).
.	Or	more	generally,	since	ν	might	not	be	such	as	to	directly	imply	for	any
given	pair	of	a	possible	world	and	a	utility	function	(
w,	U
)	whether	the
proposition	ν(
U
)	is	true	in	
w
,	what	needs	to	be	done	is	to	give	the	AI	an
adequate	representation	of	the	conditional	probability	distribution	
P
(ν(
U
)	|
w
).

.	Consider	first	
,	the	class	of	actions	that	are	possible	for	an	agent.	One	issue
here	is	what	exactly	should	count	as	an	action:	only	basic	motor	commands
(e.g.	“send	an	electric	pulse	along	output	channel	#00101100”),	or	higher-
level	actions	(e.g.	“keep	camera	centered	on	face”)?	Since	we	are	trying	to
develop	an	optimality	notion	rather	than	a	practical	implementation	plan,
we	may	take	the	domain	to	be	basic	motor	commands	(and	since	the	set	of
possible	motor	commands	might	change	over	time,	we	may	need	to	index	
to	time).	However,	in	order	to	move	toward	implementation	it	will
presumably	be	necessary	to	introduce	some	kind	of	hierarchical	planning
process,	and	one	may	then	need	to	consider	how	to	apply	the	formula	to
some	class	of	higher-level	actions.	Another	issue	is	how	to	analyze	internal
actions	(such	as	writing	strings	to	working	memory).	Since	internal	actions
can	have	important	consequences,	one	would	ideally	want	
	to	include	such

basic	internal	actions	as	well	as	motor	commands.	But	there	are	limits	to
how	far	one	can	go	in	this	direction:	the	computation	of	the	expected	utility
of	any	action	in	
	requires	multiple	computational	operations,	and	if	each
such	operation	were	also	regarded	as	an	action	in	
	that	needed	to	be
evaluated	according	to	AI-VL,	we	would	face	an	infinite	regress	that	would
make	it	impossible	to	ever	get	started.	To	avoid	the	infinite	regress,	one

to	include	such
must	restrict	any	explicit	attempt	to	estimate	the	expected	utility	to	a	limited
number	of	significant	action	possibilities.	The	system	will	then	need	some
heuristic	process	that	identifies	some	significant	action	possibilities	for
further	consideration.	(Eventually	the	system	might	also	get	around	to
making	explicit	decisions	regarding	some	possible	actions	to	make
modifications	to	this	heuristic	process,	actions	that	might	have	been	flagged
for	explicit	attention	by	this	self-same	process;	so	that	in	the	long	run	the

system	might	become	increasingly	effective	at	approximating	the	ideal
identified	by	AI-VL.)
Consider	next	
,	which	is	a	class	of	possible	worlds.	One	difficulty	here
is	to	specify	
	so	that	it	is	sufficiently	inclusive.	Failure	to	include	some
relevant	
w
	in	
	could	render	the	AI	incapable	of	representing	a	situation
that	actually	occurs,	resulting	in	the	AI	making	bad	decisions.	Suppose,	for
example,	that	we	use	some	ontological	theory	to	determine	the	makeup	of	
.	For	instance,	we	include	in	
	all	possible	worlds	that	consist	of	a	certain

relevant	
w
	in	
kind	of	spacetime	manifold	populated	by	elementary	particles	found	in	the
standard	model	in	particle	physics.	This	could	distort	the	AI’s	epistemology
if	the	standard	model	is	incomplete	or	incorrect.	One	could	try	to	use	a
bigger	
-class	to	cover	more	possibilities;	but	even	if	one	could	ensure	that

every	possible	physical	universe	is	included	one	might	still	worry	that	some
other	possibility	would	be	left	out.	For	example,	what	about	the	possibility
of	dualistic	possible	worlds	in	which	facts	about	consciousness	do	not
supervene	on	facts	about	physics?	What	about	indexical	facts?	Normative
facts?	Facts	of	higher	mathematics?	What	about	other	kinds	of	fact	that	we
fallible	humans	might	have	overlooked	but	that	could	turn	out	to	be
important	to	making	things	go	as	well	as	possible?	Some	people	have
strong	convictions	that	some	particular	ontological	theory	is	correct.
(Among	people	writing	on	the	future	of	AI,	a	belief	in	a	materialistic
ontology,	in	which	the	mental	supervenes	on	the	physical,	is	often	taken	for
granted.)	Yet	a	moment’s	reflection	on	the	history	of	ideas	should	help	us
realize	that	there	is	a	significant	possibility	that	our	favorite	ontology	is
wrong.	Had	nineteenth-century	scientists	attempted	a	physics-inspired
definition	of	
,	they	would	probably	have	neglected	to	include	the

possibility	of	a	non-Euclidian	spacetime	or	an	Everettian	(“many-worlds”)
quantum	theory	or	a	cosmological	multiverse	or	the	simulation	hypothesis
—possibilities	that	now	appear	to	have	a	substantial	probability	of
obtaining	in	the	actual	world.	It	is	plausible	that	there	are	other	possibilities
to	which	we	in	the	present	generation	are	similarly	oblivious.	(On	the	other
hand,	if	
	is	too	big,	there	arise	technical	difficulties	related	to	having	to

hand,	if	
assign	measures	over	transfinite	sets.)	The	ideal	might	be	if	we	could
somehow	arrange	things	such	that	the	AI	could	use	some	kind	of	open-
ended	ontology,	one	that	the	AI	itself	could	subsequently	extend	using	the
same	principles	that	we	would	use	when	deciding	whether	to	recognize	a
new	type	of	metaphysical	possibility.
Consider	
P
(
w
|
Ey
).	Specifying	this	conditional	probability	is	not	strictly
part	of	the	value-loading	problem.	In	order	to	be	intelligent,	the	AI	must

Consider	
P
(
w
|
Ey
already	have	some	way	of	deriving	reasonably	accurate	probabilities	over
many	relevant	factual	possibilities.	A	system	that	falls	too	far	short	on	this
score	will	not	pose	the	kind	of	danger	that	concerns	us	here.	However,	there
may	be	a	risk	that	the	AI	will	end	up	with	an	epistemology	that	is	good
enough	to	make	the	AI	instrumentally	effective	yet	not	good	enough	to
enable	it	to	think	correctly	about	some	possibilities	that	are	of	great
normative	importance.	(The	problem	of	specifying	
P
(
w
|
Ey
)	is	in	this	way

P
(
w
|
Ey
related	to	the	problem	of	specifying	
.)	Specifying	
P
(
w
	|	
Ey
)	also	requires
confronting	other	issues,	such	as	how	to	represent	uncertainty	over	logical
impossibilities.

The	aforementioned	issues—how	to	define	a	class	of	possible	actions,	a
class	of	possible	worlds,	and	a	likelihood	distribution	connecting	evidence
to	classes	of	possible	worlds—are	quite	generic:	similar	issues	arise	for	a
wide	range	of	formally	specified	agents.	It	remains	to	examine	a	set	of
issues	more	peculiar	to	the	value	learning	approach;	namely,	how	to	define	
,	
V
(
U
),	and	
P
(
V
(
U
)	|	
w
).
	is	a	class	of	utility	functions.	There	is	a	connection	between	
	and	
inasmuch	as	each	utility	function	
U
	(
w
)	in	
	should	ideally	assign	utilities
to	each	possible	world	
w
	in	
.	But	
	also	needs	to	be	wide	in	the	sense	of
containing	sufficiently	many	and	diverse	utility	functions	for	us	to	have
justified	confidence	that	at	least	one	of	them	does	a	good	job	of
representing	the	intended	values.
The	reason	for	writing	
P
(
V
(
U
)	|	
w
)	rather	than	simply	
P
(
U
|
w
)	is	to
emphasize	the	fact	that	probabilities	are	assigned	to	propositions.	A	utility

and	
U
	(
w
)	in	
w
	in	
.	But	
P
(
V
(
U
)	|	
w
P
(
U
|
w
)	is	to
function,	per	se,	is	not	a	proposition,	but	we	can	transform	a	utility	function
into	a	proposition	by	making	some	claim	about	it.	For	example,	we	may
claim	of	a	particular	utility	function	
U
(.)	that	it	describes	the	preferences	of
a	particular	person,	or	that	it	represents	the	prescriptions	implied	by	some
ethical	theory,	or	that	it	is	the	utility	function	that	the	principal	would	have
wished	to	have	implemented	if	she	had	thought	things	through.	The	“value
criterion”	
V

U
V
(.)	can	thus	be	construed	as	a	function	that	takes	as	its	argument
a	utility	function	
U
	and	gives	as	its	value	a	proposition	to	the	effect	that	
U
satisfies	the	criterion	
V
.	Once	we	have	defined	a	proposition	
V
(
U
),	we	can
hopefully	obtain	the	conditional	probability	
P
(
V
(
U
)|
w
)	from	whatever
source	we	used	to	obtain	the	other	probability	distributions	in	the	AI.	(If	we
are	certain	that	all	normatively	relevant	facts	are	taken	into	account	in
individuating	the	possible	worlds	
,	then	
P
(
V
(
U
)|
w
)	should	equal	zero	or

U
U
V
V
(
U
),	we	can
P
(
V
(
U
)|
w
,	then	
P
(
V
(
U
)|
w
one	in	each	possible	world.)	The	question	remains	how	to	define	
V
.	This	is
discussed	further	in	the	text.
.	These	are	not	the	only	challenges	for	the	value	learning	approach.	Another
issue,	for	instance,	is	how	to	get	the	AI	to	have	sufficiently	sensible	initial
beliefs—at	least	by	the	time	it	becomes	strong	enough	to	subvert	the
programmers’	attempts	to	correct	it.
.	Yudkowsky	(2001).

.	The	term	is	taken	from	American	football,	where	a	“Hail	Mary”	is	a	very
long	forward	pass	made	in	desperation,	typically	when	the	time	is	nearly
up,	on	the	off	chance	that	a	friendly	player	might	catch	the	ball	near	the	end
zone	and	score	a	touchdown.
.	The	Hail	Mary	approach	relies	on	the	idea	that	a	superintelligence	could
articulate	its	preferences	with	greater	exactitude	than	we	humans	can
articulate	ours.	For	example,	a	superintelligence	could	specify	its
preference	as	code.	So	if	our	AI	is	representing	other	superintelligences	as
computational	processes	that	are	perceiving	their	environment,	then	our	AI
should	be	able	to	reason	about	how	those	alien	superintelligences	would
respond	to	some	hypothetical	stimulus,	such	as	a	“window”	popping	up	in
their	visual	field	presenting	them	with	the	source	code	of	our	own	AI	and
asking	them	to	specify	their	instructions	to	us	in	some	convenient	pre-
specified	format.	Our	AI	could	then	read	off	these	imaginary	instructions

(from	its	own	model	of	this	counterfactual	scenario	wherein	these	alien
superintelligences	are	represented),	and	we	would	have	built	our	AI	so	that
it	would	be	motivated	to	follow	those	instructions.
.	An	alternative	would	be	to	create	a	detector	that	looks	(within	our	AI’s	world
model)	for	(representations	of)	physical	structures	created	by	a
superintelligent	civilization.	We	could	then	bypass	the	step	of	identifying
the	hypothesized	superintelligences’	preference	functions,	and	give	our	own
AI	the	final	value	of	trying	to	copy	whatever	physical	structures	it	believes

superintelligent	civilizations	tend	to	produce.
There	are	technical	challenges	with	this	version,	too,	however.	For
instance,	since	our	own	AI,	even	after	it	has	attained	superintelligence,	may
not	be	able	to	know	with	great	precision	what	physical	structures	other
superintelligences	build,	our	AI	may	need	to	resort	to	trying	to	approximate
those	structures.	To	do	this,	it	would	seem	our	AI	would	need	a	similarity
metric	by	which	to	judge	how	closely	one	physical	artifact	approximates

another.	But	similarity	metrics	based	on	crude	physical	measures	may	be
inadequate—it	being	no	good,	for	example,	to	judge	that	a	brain	is	more
similar	to	a	Camembert	cheese	than	to	a	computer	running	an	emulation.
A	more	feasible	approach	might	be	to	look	for	“beacons”:	messages
about	utility	functions	encoded	in	some	suitable	simple	format.	We	would
build	our	AI	to	want	to	follow	whatever	such	messages	about	utility

functions	it	hypothesizes	might	exist	out	there	in	the	universe;	and	we
would	hope	that	friendly	extraterrestrial	AIs	would	create	a	variety	of
beacons	of	the	types	that	they	(with	their	superintelligence)	reckon	that
simple	civilizations	like	ours	are	most	likely	to	build	our	AI	to	look	for.
.	If	
every
	civilization	tried	to	solve	the	value-loading	problem	through	a	Hail
Mary,	the	pass	would	fail.	Somebody	has	to	do	it	the	hard	way.
.	Christiano	(2012).
.	The	AI	we	build	need	not	be	able	to	find	the	model	either.	Like	us,	it	could
reason	about	what	such	a	complex	implicit	definition	would	entail	(perhaps
by	looking	at	its	environment	and	following	much	the	same	kind	of
reasoning	that	we	would	follow).
.	Cf.	
Chapters	9
	and	
11
.
.	For	instance,	MDMA	may	temporarily	increase	empathy;	oxytocin	may
temporarily	increase	trust	(Vollenweider	
et	al.
	1998;	Bartz	
et	al.
	2011).
However,	the	effects	seem	quite	variable	and	context	dependent.

.	Cf.	
Chapters	9
	and	
11
.
et	al.
et	al.
	2011).
.	The	enhanced	agents	might	be	killed	off	or	placed	in	suspended	animation
(paused),	reset	to	an	earlier	state,	or	disempowered	and	prevented	from
receiving	any	further	enhancements,	until	the	overall	system	has	reached	a
more	mature	and	secure	state	where	these	earlier	rogue	elements	no	longer
pose	a	system-wide	threat.
.	The	issue	might	also	be	less	obvious	in	a	future	society	of	biological	humans,
one	that	has	access	to	advanced	surveillance	or	biomedical	techniques	for

psychological	manipulation,	or	that	is	wealthy	enough	to	afford	an
extremely	high	ratio	of	security	professionals	to	invigilate	the	regular
citizenry	(and	each	other).
.	Cf.	Armstrong	(2007)	and	Shulman	(2010b).
.	One	open	question	is	to	what	degree	a	level	
n
	supervisor	would	need	to
monitor	not	only	their	level	(
n	–	1
)	supervisees,	but	also	
their
	level	(
n	–	2
)
supervisees,	in	order	to	know	that	the	level	(
n	–	1
)	agents	are	doing	their
jobs	properly.	And	to	know	that	the	level	(
n	–	1
)	agents	have	successfully

n
n	–	1
their
	level	(
n	–	2
)
n	–	1
n	–	1
managed	the	level	(
n	–	2
)	agents,	is	it	further	necessary	for	the	level	
n
	agent

to	also	monitor	the	level	(
n	–	3
)	agents?
.	This	approach	straddles	the	line	between	motivation	selection	and	capability
control.	Technically,	the	part	of	the	arrangement	that	consists	of	human
beings	controlling	a	set	of	software	supervisors	counts	as	capability	control,
whereas	the	part	of	the	arrangement	that	consists	of	layers	of	software
agents	within	the	system	controlling	other	layers	is	motivation	selection
(insofar	as	it	is	an	arrangement	that	shapes	the	system’s	motivational
tendencies).
.	In	fact,	many	other	costs	deserve	consideration	but	cannot	be	given	it	here.
For	example,	whatever	agents	are	charged	with	ruling	over	such	a	hierarchy
might	become	corrupted	or	debased	by	their	power.
.	For	this	guarantee	to	be	effective,	it	must	be	implemented	in	good	faith.	This
would	rule	out	certain	kinds	of	manipulation	of	the	emulation’s	emotional
and	decision-making	faculties	which	might	otherwise	be	used	(for	instance)
to	install	a	fear	of	being	halted	or	to	prevent	the	emulation	from	rationally

considering	its	options.
.	See,	e.g.,	Brinton	(1965);	Goldstone	(1980,	2001).	(Social	science	progress
on	these	questions	could	make	a	nice	gift	to	the	world’s	despots,	who	might
use	more	accurate	predictive	models	of	social	unrest	to	optimize	their
population	control	strategies	and	to	gently	nip	insurgencies	in	the	bud	with
less-lethal	force.)
.	Cf.	Bostrom	(2011a,	2009b).
.	
In	the	case	of	an	entirely	artificial	system,	it	might	be	possible	to	obtain

.	
some	of	the	advantages	of	an	institutional	structure	without	actually
creating	distinct	subagents.	A	system	might	incorporate	multiple
perspectives	into	its	decision	process	without	endowing	each	of	those
perspectives	with	its	own	panoply	of	cognitive	faculties	required	for
independent	agency.	It	could	be	tricky,	however,	to	fully	implement	the
“observe	the	behavioral	consequences	of	a	proposed	change,	and	revert
back	to	an	earlier	version	if	the	consequences	appear	undesirable	from	the
ex	ante
	standpoint”	feature	described	in	the	text	in	a	system	that	is	not

ex	ante
composed	of	subagents.
CHAPTER	13:	CHOOSING	THE	CRITERIA	FOR	CHOOSING

1
.	A	recent	canvass	of	professional	philosophers	found	the	percentage	of
respondents	who	“accept	or	leans	toward”	various	positions.	On	normative
ethics,	the	results	were	
deontology
	25.9%;	
consequentialism
	23.6%;	
virtue
ethics
	18.2%.	On	metaethics,	results	were	
moral	realism
	56.4%;	
moral	anti-
realism
	27.7%.	On	moral	judgment:	
cognitivism
	65.7%;	
non-cognitivism
17.0%	(Bourget	and	Chalmers	2009).
2
.	Pinker	(2011).
3
.	For	a	discussion	of	this	issue,	see	Shulman	
et	al.
	(2009).
4
.	Moore	(2011).
5
.	Bostrom	(2006b).
6
.	Bostrom	(2009b).
7
.	Bostrom	(2011a).
8
.	More	precisely,	we	should	defer	to	its	opinion	except	on	those	topics	where
we	have	good	reason	to	suppose	that	our	beliefs	are	more	accurate.	For
example,	we	might	know	more	about	what	we	are	thinking	at	a	particular
moment	than	the	superintelligence	does	if	it	is	not	able	to	scan	our	brains.
However,	we	could	omit	this	qualification	if	we	assume	that	the
superintelligence	has	access	to	our	opinions;	we	could	then	also	defer	to	the

5
6
7
8
superintelligence	the	task	of	judging	when	our	opinions	should	be	trusted.
(There	might	remain	some	special	cases,	involving	indexical	information,
that	need	to	be	handled	separately—by,	for	example,	having	the
superintelligence	explain	to	us	what	it	would	be	rational	to	believe	from	our
perspective.)	For	an	entry	into	the	burgeoning	philosophical	literature	on
testimony	and	epistemic	authority,	see,	e.g.,	Elga	(2007).
9
.	Yudkowsky	(2004).	See	also	Mijic	(2010).
.	For	example,	David	Lewis	proposed	a

9
.	For	example,	David	Lewis	proposed	a	
dispositional	theory	of	value
,	which
holds,	roughly,	that	some	thing	
X
	is	a	value	for	
A
	if	and	only	if	
A
	would
want	to	want	
X
	if	
A
	were	perfectly	rational	and	ideally	acquainted	with	
X
(Smith	
et	al.
	1989).	Kindred	ideas	had	been	put	forward	earlier;	see,	e.g.,

Sen	and	Williams	(1982),	Railton	(1986),	and	Sidgwick	and	Jones	(2010).
Along	somewhat	similar	lines,	one	common	account	of	philosophical
justification,	the	
method	of	reflective	equilibrium
,	proposes	a	process	of
iterative	mutual	adjustment	between	our	intuitions	about	particular	cases,
the	general	rules	which	we	think	govern	these	cases,	and	the	principles
according	to	which	we	think	these	elements	should	be	revised,	to	achieve	a
more	coherent	system;	see,	e.g.,	Rawls	(1971)	and	Goodman	(1954).
.	Presumably	the	intention	here	is	that	when	the	AI	acts	to	prevent	such
disasters,	it	should	do	it	with	
as	light	a	touch	as	possible
,	
i.e.
	in	such	a
manner	that	it	averts	the	disaster	but	without	exerting	too	much	influence
over	how	things	turn	out	for	humanity	in	other	respects.
.	Yudkowsky	(2004).
.	Rebecca	Roache,	personal	communication.
.	The	three	principles	are	“Defend	humans,	the	future	of	humanity,	and
humane	nature”	(
humane
	here	being	that	which	we	wish	we	were,	as
distinct	from	
human

,	
i.e.
	in	such	a
humane
human
,	which	is	what	we	are);	“Humankind	should	not	spend
the	rest	of	eternity	desperately	wishing	that	the	programmers	had	done
something	differently”;	and	“Help	people.”
.	Some	religious	groups	place	a	strong	emphasis	on	faith	in	contradistinction
to	reason,	the	latter	of	which	they	may	regard—even	in	its	hypothetically
most	idealized	form	and	even	after	it	would	have	ardently	and	open-
mindedly	studied	every	scripture,	revelation,	and	exegesis—to	be

human
insufficient	for	the	attainment	of	essential	spiritual	insights.	Those	holding
such	views	might	not	regard	CEV	as	an	optimal	guide	to	decision-making
(though	they	might	still	prefer	it	to	
various	other	imperfect	guides	that
might	in	actuality	be	followed	if	the	CEV	approach	were	eschewed).
.	An	AI	acting	like	a	latent	force	of	nature	to	regulate	human	interactions	has
been	referred	to	as	a	“Sysop,”	a	kind	of	“operating	system”	for	the	matter
occupied	by	human	civilization.	See	Yudkowsky	(2001).
.	“
Might
,”	because	
conditional

.	“
Might
,”	because	
	on	humanity’s	coherent	extrapolated	volition
wishing	not	to	extend	moral	consideration	to	these	entities,	it	is	perhaps
doubtful	whether	those	entities	actually	have	moral	status	(despite	it
seeming	very	plausible	now	that	they	do).	“
Potentially
,”	because	even	if	a

blocking	vote	prevents	the	CEV	dynamic	from	directly	protecting	these
outsiders,	there	is	still	a	possibility	that,	within	whatever	ground	rules	are
left	over	once	the	initial	dynamic	has	run,	individuals	whose	wishes	were
respected	and	who	want	some	outsiders’	welfare	to	be	protected	may
successfully	bargain	to	attain	this	outcome	(at	the	expense	of	giving	up
some	of	their	own	resources).	Whether	this	would	be	possible	might	depend
on,	among	other	things,	whether	the	outcome	of	the	CEV	dynamic	is	a	set
of	ground	rules	that	makes	it	feasible	to	reach	negotiated	resolutions	to
issues	of	this	kind	(which	might	require	provisions	to	overcome	strategic
bargaining	problems).
.	Individuals	who	contribute	positively	to	realizing	a	safe	and	beneficial
superintelligence	might	merit	
some
	special	reward	for	their	labour,	albeit
something	short	of	a	near-exclusive	mandate	to	determine	the	disposition	of
humanity’s	cosmic	endowment.	However,	the	notion	of	everybody	getting

some
an	equal	share	in	our	extrapolation	base	is	such	a	nice	Schelling	point	that	it
should	not	be	lightly	tossed	away.	There	is,	in	any	case,	an	indirect	way	in
which	virtue	could	be	rewarded:	namely,	the	CEV	itself	might	turn	out	to
specify	that	good	people	who	exerted	themselves	on	behalf	of	humanity
should	be	suitably	recognized.	This	could	happen	without	such	people
being	given	any	special	weight	in	the	extrapolation	base	if—as	is	easily
imaginable—our	CEV	would	endorse	(in	the	sense	of	giving	at	least	some

nonzero	weight	to)	a	principle	of	just	desert.
.	Bostrom	
et	al.
	(2013).
.	To	the	extent	that	there	is	some	(sufficiently	definite)	shared	meaning	that	is
being	expressed	when	we	make	moral	assertions,	a	superintelligence	should
be	able	to	figure	out	what	that	meaning	is.	And	to	the	extent	that	moral
assertions	are	“truth-apt”	(i.e.	have	an	underlying	propositional	character
that	enables	them	to	be	true	or	false),	the	superintelligence	should	be	able	to
figure	out	which	assertions	of	the	form	“Agent	
X
	ought	now	to	
Φ
”	are	true.
At	least,	it	should	outperform	us	on	this	task.

X
Φ
”	are	true.
An	AI	that	initially	lacks	such	a	capacity	for	moral	cognition	should	be
able	to	acquire	it	if	it	has	the	intelligence	amplification	superpower.	One
way	the	AI	could	do	this	is	by	reverse-engineering	the	human	brain’s	moral
thinking	and	then	implement	a	similar	process	but	run	it	faster,	feed	it	more
accurate	factual	information,	and	so	forth.

.	Since	we	are	uncertain	about	metaethics,	there	is	a	question	of	what	the	AI	is
to	do	if	the	preconditions	for	MR	fail	to	obtain.	One	option	is	to	stipulate
that	the	AI	shut	itself	off	if	it	assigns	a	sufficiently	high	probability	to	moral
cognitivism	being	false	or	to	there	being	no	suitable	non-relative	moral
truths.	Alternatively,	we	could	have	the	AI	revert	to	some	alternative
approach,	such	as	CEV.
We	could	refine	the	MR	proposal	to	make	it	clearer	what	is	to	be	done	in
various	ambiguous	or	degenerate	cases.	For	instance,	if	error	theory	is	true
(and	hence	all	positive	moral	assertions	of	the	form	“I	ought	now	to	
τ
”	are
false),	then	the	fallback	strategy	(e.g.	shutting	down)	would	be	invoked.	We
could	also	specify	what	should	happen	if	there	are	multiple	feasible	actions,
each	of	which	would	be	morally	right.	For	example,	we	might	say	that	in
such	cases	the	AI	should	perform	(one	of)	the	permissible	actions	that
humanity’s	collective	extrapolation	would	have	favored.	We	might	also

τ
”	are
stipulate	what	should	happen	if	the	true	moral	theory	does	not	employ	terms
like	“morally	right”	in	its	basic	vocabulary.	For	instance,	a	consequentialist
theory	might	hold	that	some	actions	are	better	than	others	but	that	there	is
no	particular	threshold	corresponding	to	the	notion	of	an	action	being
“morally	right.”	We	could	then	say	that	if	such	a	theory	is	correct,	MR
should	perform	one	of	the	morally	best	feasible	actions,	if	there	is	one;	or,	if
there	is	an	infinite	number	of	feasible	actions	such	that	for	any	feasible
action	there	is	a	better

action	there	is	a	better	
one,	then	maybe	MR	could	pick	any	that	is	at	least
astronomically	better	than	the	best	action	that	any	human	would	have
selected	in	a	similar	situation,	if	such	an	action	is	feasible—or	if	not,	then
an	action	that	is	at	least	as	good	as	the	best	action	a	human	would	have
performed.
A	couple	of	general	points	should	be	borne	in	mind	when	thinking	about
how	the	MR	proposal	could	be	refined.	First,	we	might	start	conservatively,
using	the	fallback	option	to	cover	almost	all	contingencies	and	only	use	the

“morally	right”	option	in	those	that	we	feel	we	fully	understand.	Second,
we	might	add	the	general	modulator	to	the	MR	proposal	that	it	is	to	be
“interpreted	charitably,	and	revised	as	we	would	have	revised	it	if	we	had
thought	more	carefully	about	it	before	we	wrote	it	down,	etc.”
.	Of	these	terms,	“knowledge”	might	seem	the	one	most	readily	susceptible	to
a	formal	analysis	(in	information-theoretic	terms).	However,	to	represent
what	it	is	for	a	human	to	know	something,	the	AI	may	need	a	sophisticated

set	of	representations	relating	to	complex	psychological	properties.	A
human	being	does	not	“know”	all	the	information	that	is	stored	somewhere
in	her	brain.
.	One	indicator	that	the	terms	in	CEV	are	(marginally)	less	opaque	is	that	it
would	count	as	philosophical	progress	if	we	could	analyze	moral	rightness
in	terms	like	those	used	in	CEV.	In	fact,	one	of	the	main	strands	in
metaethics—ideal	observer	theory—purports	to	do	just	that.	See,	e.g.,
Smith	
et	al.
	(1989).
.	This	requires	confronting	the	problem	of	fundamental	normative	uncertainty.
It	can	be	shown	that	it	is	not	always	appropriate	to	act	according	to	the
moral	theory	that	has	the	highest	probability	of	being	true.	It	can	also	be
shown	that	it	is	not	always	appropriate	to	perform	the	action	that	has	the
highest	probability	of	being	right.	Some	way	of	trading	probabilities	against
“degrees	of	wrongness”	or	severity	of	issues	at	stake	seems	to	be	needed.
For	some	ideas	in	this	direction,	see	Bostrom	(2009a).

.	It	could	possibly	even	be	argued	that	it	is	an	adequacy	condition	for	any
explication	of	the	notion	of	moral	rightness	that	it	account	for	how	Joe
Sixpack	is	able	to	have	some	idea	of	right	and	wrong.
.	It	is	not	obvious	that	the	morally	right	thing	
for	us
	to	do	is	to	build	an	AI	that
implements	MR,	even	if	we	assume	that	
the	AI	itself
	would	always	act
morally.	Perhaps	it	would	be	objectionably	hubristic	or	arrogant	of	us	to
build	such	an	AI	(especially	since	many	people	may	disapprove	of	that

for	us
project).	This	issue	can	be	partially	finessed	by	tweaking	the	MR	proposal.
Suppose	that	we	stipulate	that	the	AI	should	act	(to	do	what	it	would	be
morally	right	for	it	to	do)	only	if	it	was	morally	right	for	its	creators	to	have
built	the	AI	in	the	first	place;	otherwise	it	should	shut	itself	down.	It	is	hard
to	see	how	we	would	be	committing	any	grave	moral	wrong	in	creating	
that
kind	of	AI,	since	if	it	were	wrong	for	us	to	create	it,	the	only	consequence

that
would	be	that	an	AI	was	created	that	immediately	shuts	itself	down,
assuming	that	the	AI	has	committed	no	mind	crime	up	to	that	point.	(We
might	nevertheless	have	acted	wrongly—for	instance,	by	having	failed	to
seize	the	opportunity	to	build	some	other	AI	instead.)
A	second	issue	is	supererogation.	Suppose	there	are	many	actions	the	AI
could	take,	each	of	which	would	be	morally	right—in	the	sense	of	being

morally	permissible
—yet	some	of	which	are	morally	better	than	the	others.
One	option	is	to	have	the	AI	aim	to	select	the	morally	best	action	in	any
such	a	situation	(or	one	of	the	best	actions,	in	case	there	are	several	that	are
equally	good).	Another	option	is	to	have	the	AI	select	from	among	the
morally	permissible	actions	one	that	maximally	satisfies	some	other	(non-
moral)	desideratum.	For	example,	the	AI	could	select,	from	among	the
actions	that	are	morally	permissible,	the	action	that	our	CEV	would	prefer	it
to	take.	Such	an	AI,	while	never	doing	anything	that	is	morally
impermissible,	might	protect	our	interests	more	than	an	AI	that	does	what	is
morally	best.
.	When	the	AI	evaluates	the	moral	permissibility	of	our	act	of	creating	the	AI,
it	should	interpret	permissibility	in	its	objective	sense.	In	one	ordinary	sense
of	“morally	permissible,”	a	doctor	acts	morally	permissibly	when	she
prescribes	a	drug	she	believes	will	cure	her	patient—even	if	the	patient,

unbeknownst	to	the	doctor,	is	allergic	to	the	drug	and	dies	as	a	result.
Focusing	on	objective	moral	permissibility	takes	advantage	of	the
presumably	superior	epistemic	position	of	the	AI.
.	More	directly,	it	depends	on	the	AI’s	
beliefs
	about	which	ethical	theory	is
true	(or,	more	precisely,	on	its	probability	distribution	over	ethical	theories).
.	
It	can	be	difficult	to	imagine	how	superlatively	wonderful	these	physically
possible	lives	might	be.	See	Bostrom	(2008c)	for	a	poetic	attempt	to	convey

beliefs
.	
some	sense	of	this.	See	Bostrom	(2008b)	for	an	argument	that	some	of	these
possibilities	could	be	good	
for	us
,	good	for	existing	human	beings.
.	It	might	seem	deceptive	or	manipulative	to	promote	one	proposal	if	one
thinks	that	some	other	proposal	would	be	better.	But	one	could	promote	it	in
ways	that	avoid	insincerity.	For	example,	one	could	freely	acknowledge	the
superiority	of	the	ideal	while	still	promoting	the	non-ideal	as	the	best
attainable	compromise.

for	us
.	Or	some	other	positively	evaluative	term,	such	as	“good,”	“great,”	or
“wonderful.”
.	This	echoes	a	principle	in	software	design	known	as	“Do	What	I	Mean,”	or
DWIM.	See	Teitelman	(1966).
.	Goal	content,	decision	theory,	and	epistemology	are	three	aspects	that	should

be	elucidated;	but	we	do	not	intend	to	beg	the	question	of	whether	there
must	be	a	neat	decomposition	into	these	three	separate	components.
.	An	ethical	project	ought	presumably	to	allocate	at	most	a	modest	portion	of
the	eventual	benefits	that	the	superintelligence	produces	as	special	rewards
to	those	who	contributed	in	morally	permissible	ways	to	the	project’s
success.	Allocating	a	great	portion	to	the	incentive	wrapping	scheme	would
be	unseemly.	It	would	be	analogous	to	a	charity	that	spends	90%	of	its
income	on	performance	bonuses	for	its	fundraisers	and	on	advertising
campaigns	to	increase	donations.
.	How	could	the	dead	be	rewarded?	One	can	think	of	several	possibilities.	At
the	low	end,	there	could	be	memorial	services	and	monuments,	which
would	be	a	reward	insofar	as	people	desired	posthumous	fame.	The
deceased	might	also	have	other	preferences	about	the	future	that	could	be
honored,	for	instance	concerning	cultures,	arts,	buildings,	or	natural

environments.	Furthermore,	most	people	care	about	their	descendants,	and
special	privileges	could	be	granted	to	the	children	and	grandchildren	of
contributors.	More	speculatively,	the	superintelligence	might	be	able	to
create	relatively	faithful	simulations	of	some	past	people—simulations	that
would	be	conscious	and	that	would	resemble	the	original	sufficiently	to
count	as	a	form	of	survival	(according	to	at	least	some	people’s	criteria).
This	would	presumably	be	easier	for	people	who	have	been	placed	in

cryonic	suspension;	but	perhaps	for	a	superintelligence	it	would	not	be
impossible	to	recreate	something	quite	similar	to	the	original	person	from
other	preserved	records	such	as	correspondence,	publications,	audiovisual
materials	and	digital	records,	or	the	personal	memories	of	other	survivors.
A	superintelligence	might	also	think	of	some	possibilities	that	do	not
readily	occur	to	us.
.	On	Pascalian	mugging,	see	Bostrom	(2009b).	For	an	analysis	of	issues

related	to	infinite	utilities,	see	Bostrom	(2011a).	On	fundamental	normative
uncertainty,	see,	e.g.,	Bostrom	(2009a).
.	E.g.,	Price	(1991);	Joyce	(1999);	Drescher	(2006);	Yudkowsky	(2010);	Dai
(2009).
.	E.g.,	Bostrom	(2009a).

.	It	is	also	conceivable	that	using	indirect	normativity	to	specify	the	AI’s	goal
content	would	mitigate	the	problems	that	might	arise	from	an	incorrectly
specified	decision	theory.	Consider,	for	example,	the	CEV	approach.	If	it
were	implemented	well,	it	might	be	able	to	compensate	for	at	least	some
errors	in	the	specification	of	the	AI’s	decision	theory.	The	implementation
could	allow	the	values	that	our	coherent	extrapolated	volition	would	want
the	AI	to	pursue	to	depend	on	the	AI’s	decision	theory.	If	our	idealized
selves	knew	they	were	making	value	specifications	for	an	AI	that	was	using
a	particular	kind	of	decision	theory,	they	could	adjust	their	value
specifications	such	as	to	make	the	AI	behave	benignly	despite	its	warped
decision	theory—much	like	one	can	cancel	out	the	distorting	effects	of	one
lens	by	placing	another	lens	in	front	of	it	that	distorts	oppositely.
.	Some	epistemological	systems	may,	in	a	holistic	manner,	have	no	distinct

foundation.	In	that	case,	the	constitutional	inheritance	is	not	a	distinct	set	of
principles,	but	rather,	as	it	were,	an	epistemic	starting	point	that	embodies
certain	propensities	to	respond	to	incoming	streams	of	evidence.
.	See,	e.g.,	the	problem	of	distortion	discussed	in	Bostrom	(2011a).
.	For	instance,	one	disputed	issue	in	anthropic	reasoning	is	whether	the	so-
called	self-indication	assumption	should	be	accepted.	The	self-indication

assumption	states,	roughly,	that	from	the	fact	that	you	exist	you	should	infer
that	hypotheses	according	to	which	larger	numbers	
N
	of	observers	exist
should	receive	a	probability	boost	proportional	to	
N
.	For	an	argument
against	
this	principle,	see	the	“Presumptuous	Philosopher”	gedanken
experiment	in	Bostrom	(2002a).	For	a	defense	of	the	principle,	see	Olum
(2002);	and	for	a	critique	of	that	defense,	see	Bostrom	and	Ćirković	(2003).
Beliefs	about	the	self-indication	assumption	might	affect	various	empirical

N
N
against	
hypotheses	of	potentially	crucial	strategic	relevance,	for	example,
considerations	such	as	the	Carter–Leslie	doomsday	argument,	the
simulation	argument,	and	“great	filter”	arguments.	See	Bostrom	(2002a,
2003a,	2008a);	Carter	(1983);	Ćirković	
et	al.
	(2010);	Hanson	(1998d);
Leslie	(1996);	Tegmark	and	Bostrom	(2005).	A	similar	point	could	be	made
with	regard	to	other	fraught	issues	in	observation	selection	theory,	such	as
whether	the	choice	of	reference	class	can	be	relativized	to	observer-
moments,	and	if	so	how.

et	al.
.	See,	e.g.,	Howson	and	Urbach	(1993).	There	are	also	some	interesting	results

that	narrow	the	range	of	situations	in	which	two	Bayesian	agents	can
rationally	disagree	when	their	opinions	are	common	knowledge;	see
Aumann	(1976)	and	Hanson	(2006).
.	Cf.	the	concept	of	a	“last	judge”	in	Yudkowsky	(2004).
.	There	are	many	important	issues	outstanding	in	epistemology,	some
mentioned	earlier	in	the	text.	The	point	here	is	that	we	may	not	need	to	get
all	the	solutions	exactly	right	in	order	to	achieve	an	outcome	that	is
practically	indiscernible	from	the	best	outcome.	A	mixture	model	(which
throws	together	a	wide	range	of	diverse	priors)	might	work.
CHAPTER	14:	THE	STRATEGIC	PICTURE
	
1
.	This	principle	is	introduced	in	Bostrom	(2009b,	190),	where	it	is	also	noted
that	it	is	not	tautological.	For	a	visual	analogy,	picture	a	box	with	large	but
finite	volume,	representing	the	space	of	basic	capabilities	that	could	be
obtained	through	some	possible	technology.	Imagine	sand	being	poured	into
this	box,	representing	research	effort.	How	you	pour	the	sand	determines

1
where	it	piles	up	in	the	box.	But	if	you	keep	on	pouring,	the	entire	space
eventually	gets	filled.
2
.	Bostrom	(2002b).
3
.	This	is	not	the	perspective	from	which	science	policy	has	traditionally	been
viewed.	Harvey	Averch	describes	science	and	technology	policy	in	the
United	States	between	1945	and	1984	as	having	been	centered	on	debates
about	the	optimum	level	of	public	investment	in	the	S&T	enterprise	and	on
the	extent	to	which	the	government	should	attempt	to	“pick	winners”	in

2
3
order	to	achieve	the	greatest	increase	in	the	nation’s	economic	prosperity
and	military	strength.	In	these	calculations,	technological	progress	is	always
assumed	to	be	good.	But	Averch	also	describes	the	rise	of	critical
perspectives,	which	question	the	“progress	is	always	good”	premiss
(Averch	1985).	See	also	Graham	(1997).
4
5
.	This	is	of	course	by	no	means	tautological.	One	could	imagine	a	case	being

made	for	a	different	order	of	development.	It	could	be	argued	that	it	would
be	better	for	humanity	to	confront	some	less	difficult	challenge	first,	say	the
development	of	nanotechnology,	on	grounds	that	this	would	force	us	to
develop	better	institutions,	become	more	internationally	coordinated,	and
mature	in	our	thinking	about	global	strategy.	Perhaps	we	would	be	more
likely	to	rise	to	a	challenge	that	presents	a	less	metaphysically	confusing
threat	than	machine	superintelligence.	Nanotechnology	(or	synthetic
biology,	or	whatever	the	lesser	challenge	we	confront	first)	might	then	serve
as	a	footstool	that	would	help	us	ascend	to	the	capability	level	required	to
deal	with	the	higher-level	challenge	of	superintelligence.
Such	an	argument	would	have	to	be	assessed	on	a	case-by-case	basis.	For
example,	in	the	case	of	nanotechnology,	one	would	have	to	consider	various
possible	consequences	such	as	the	boost	in	hardware	performance	from
nanofabricated	computational	substrates;	the	effects	of	cheap	physical

capital	for	manufacturing	on	economic	growth;	the	proliferation	of
sophisticated	surveillance	technology;	the	possibility	that	a	singleton	might
emerge	though	the	direct	or	indirect	effects	of	a	nanotechnology
breakthrough;	and	the	greater	feasibility	of	neuromorphic	and	whole	brain
emulation	approaches	to	machine	intelligence.	It	is	beyond	the	scope	of	our
investigation	to	consider	all	these	issues	(or	the	parallel	issues	that	might
arise	for	other	existential	risk-causing	technologies).	Here	we	just	point	out

the	prima	facie	case	for	favoring	a	superintelligence-first	sequence	of
development—while	stressing	that	there	are	complications	that	might	alter
this	preliminary	assessment	in	some	cases.
6
.	
Pinker	(2011);	Wright	(2001).
7
.	It	might	be	tempting	to	suppose	the	hypothesis	that	everything	has
accelerated	to	be	meaningless	on	grounds	that	it	does	not	(at	first	glance)
seem	to	have	any	observational	consequences;	but	see,	e.g.,	Shoemaker
(1969).
8
.	The	level	of	preparedness	is	not	measured	by	the	amount	of	effort	expended

6
.	
7
(1969).
8
on	preparedness	activities,	but	by	how	propitiously	configured	conditions
actually	are	and	how	well-poised	key	decision	makers	are	to	take
appropriate	action.
9
.	The	degree	of	international	trust	during	the	lead-up	to	the	intelligence

explosion	may	also	be	a	factor.	We	consider	this	in	the	section
“Collaboration”	later	in	the	chapter.
.	Anecdotally,	it	appears	those	currently	seriously	interested	in	the	control
problem	are	disproportionately	sampled	from	one	extreme	end	of	the
intelligence	distribution,	though	there	could	be	alternative	explanations	for
this	impression.	If	the	field	becomes	fashionable,	it	will	undoubtedly	be
flooded	with	mediocrities	and	cranks.
.	I	owe	this	term	to	Carl	Shulman.
.	How	similar	to	a	brain	does	a	machine	intelligence	have	to	be	to	count	as	a
whole	brain	emulation	rather	than	a	neuromorphic	AI?	The	relevant
determinant	might	be	whether	the	system	reproduces	either	the	values	or
the	full	panoply	of	cognitive	and	evaluative	tendencies	of	either	a	particular
individual	or	a	generic	human	being,	because	this	would	plausibly	make	a
difference	to	the	control	problem.	Capturing	these	properties	may	require	a
rather	high	degree	of	emulation	fidelity.

.	The	magnitude	of	the	boost	would	of	course	depend	on	how	big	the	push
was,	and	also	where	resources	for	the	push	came	from.	There	might	be	no
net	boost	for	neuroscience	if	all	the	extra	resources	invested	in	whole	brain
emulation	research	were	deducted	from	regular	neuroscience	research—
unless	a	keener	focus	on	emulation	research	just	happened	to	be	a	more
effective	way	of	advancing	neuroscience	than	the	default	portfolio	of
neuroscience	research.
.	See	Drexler	(1986,	242).	Drexler	(private	communication)	confirms	that	this

reconstruction	corresponds	to	the	reasoning	he	was	seeking	to	present.
Obviously,	a	number	of	implicit	premisses	would	have	to	be	added	if	one
wished	to	cast	the	argument	in	the	form	of	a	deductively	valid	chain	of
.	Perhaps	we	ought	not	to	welcome	
small
	catastrophes	in	case	they	increase
our	vigilance	to	the	point	of	making	us	prevent	the	
medium-scale
catastrophes	that	would	have	been	needed	to	make	us	take	the	strong
precautions	necessary	to	prevent	existential	catastrophes?	(And	of	course,

small
just	as	with	biological	immune	systems,	we	also	need	to	be	concerned	with
over-reactions,	analogous	to	allergies	and	autoimmune	disorders.)

.	Cf.	Lenman	(2000);	Burch-Brown	(2014).
.	Cf.	Bostrom	(2007).
.	Note	that	this	argument	focuses	on	the	ordering	rather	than	the	timing	of	the
relevant	events.	Making	superintelligence	happen	earlier	would	help
preempt	other	existential	transition	risks	only	if	the	intervention	changes	the
sequence	of	the	key	developments:	for	example,	by	making
superintelligence	happen	before	various	milestones	are	reached	in
nanotechnology	or	synthetic	biology.
.	If	solving	the	control	problem	is	
extremely
	difficult	compared	to	solving	the
performance	problem	in	machine	intelligence,	and	if	project	ability
correlates	only	weakly	with	project	size,	then	it	is	possible	that	it	would	be
better	that	a	small	project	gets	there	first,	namely	if	the	variance	in
capability	is	greater	among	smaller	projects.	In	such	a	situation,	even	if
smaller	projects	are	on	average	less	competent	than	larger	projects,	it	could
be	less	unlikely	that	a	given	small	project	would	happen	to	have	the

extremely
freakishly	high	level	of	competence	needed	to	solve	the	control	problem.
.	This	is	not	to	deny	that	one	can	imagine	tools	that	could	promote	global
deliberation	and	which	would	benefit	from,	or	even	require,	further
progress	in	hardware—for	example,	high-quality	translation,	better	search,
ubiquitous	access	to	smart	phones,	attractive	virtual	reality	environments
for	social	intercourse,	and	so	forth.
.	Investment	in	emulation	technology	could	speed	progress	toward	whole

brain	emulation	not	only	directly	(through	any	technical	deliverables
produced)	but	also	indirectly	by	creating	a	
constituency	that	will	push	for
more	funding	and	boost	the	visibility	and	credibility	of	the	whole	brain
emulation	(WBE)	vision.
.	How	much	expected	value	would	be	lost	if	the	future	were	shaped	by	the
desires	of	one	random	human	rather	than	by	(some	suitable	superposition
of)	the	desires	of	all	of	humanity?	This	might	depend	sensitively	on	what
evaluation	standard	we	use,	and	also	on	whether	the	desires	in	question	are

idealized	or	raw.
.	For	example,	whereas	human	minds	communicate	slowly	via	language,	AIs
can	be	designed	so	that	instances	of	the	same	program	are	able	easily	and

quickly	to	transfer	both	skills	and	information	amongst	one	another.
Machine	minds	designed	
ab	initio
	could	do	away	with	cumbersome	legacy
systems	that	helped	our	ancestors	deal	with	aspects	of	the	natural
environment	that	are	unimportant	in	cyberspace.	Digital	minds	might	also
be	designed	to	take	advantage	of	fast	serial	processing	unavailable	to
biological	brains,	and	to	make	it	easy	to	install	new	modules	with	highly
optimized	functionality	(e.g.	symbolic	processing,	pattern	recognition,
simulators,	data	mining,	and	planning).	Artificial	intelligence	might	also
have	significant	non-technical	advantages,	such	as	being	more	easily
patentable	or	less	entangled	in	the	moral	complexities	of	using	human
uploads.
.	If	
p
1
	and	
p
2
	are	the	probabilities	of	failure	at	each	step,	the	total	probability
of	failure	is	
p
1
	+	(1	–	
p
1
)
p
2
	since	one	can	fail	terminally	only	once.
.	It	is	possible,	of	course,	that	the	frontrunner	will	not	have	such	a	large	lead

uploads.
.	If	
p
1
	and	
p
2
p
1
	+	(1	–	
p
1
)
p
2
and	will	not	be	able	to	form	a	singleton.	It	is	also	possible	that	a	singleton
would	arise	before	AI	even	without	the	intervention	of	WBE,	in	which	case
this	reason	for	favoring	a	WBE-first	scenario	falls	away.
.	Is	there	a	way	for	a	promoter	of	WBE	to	increase	the	specificity	of	her
support	so	that	it	accelerates	WBE	while	minimizing	the	spillover	to	AI
development?	Promoting	scanning	technology	is	probably	a	better	bet	than
promoting	neurocomputational	modeling.	(Promoting	computer	hardware	is

unlikely	to	make	much	difference	one	way	or	the	other,	given	the	large
commercial	interests	that	are	anyway	incentivizing	progress	in	that	field.)
Promoting	scanning	technology	may	increase	the	likelihood	of	a
multipolar	outcome	by	making	scanning	less	likely	to	be	a	bottleneck,	thus
increasing	the	chance	that	the	early	emulation	population	will	be	stamped
from	many	different	human	templates	rather	than	consisting	of	gazillions	of
copies	of	a	tiny	number	of	templates.	Progress	in	scanning	technology	also

makes	it	more	likely	that	the	bottleneck	will	instead	be	computing
hardware,	which	would	tend	to	slow	the	takeoff.
.	Neuromorphic	AI	may	also	lack	other	safety-promoting	attributes	of	whole
brain	emulation,	such	as	having	a	profile	of	cognitive	strengths	and
weaknesses	similar	to	that	of	a	biological	human	being	(which	would	let	us
use	our	experience	of	humans	to	form	some	expectations	of	the	system’s

capabilities	at	different	stages	of	its	development).
.	If	somebody’s	motive	for	promoting	WBE	is	to	make	WBE	happen	before
AI,	they	should	bear	in	mind	that	accelerating	WBE	will	alter	the	order	of
arrival	only	if	the	default	timing	of	the	two	paths	toward	machine
intelligence	is	close	and	with	a	slight	edge	to	AI.	Otherwise,	
either
investment	in	WBE	will	simply	make	WBE	happen	earlier	than	it	otherwise
would	(reducing	hardware	overhang	and	preparation	time)	but	without
affecting	the	sequence	of	development;	
or	else
	such	investment	in	WBE
will	have	little	effect	(other	than	perhaps	making	AI	happen	even	sooner	by
stimulating	progress	on	neuromorphic	AI).
.	Comment	on	Hanson	(2009).
.	There	would	of	course	be	
some
	magnitude	and	imminence	of	existential	risk
for	which	it	would	be	preferable	even	from	the	person-affecting	perspective
to	postpone	the	risk—whether	to	enable	existing	people	to	eke	out	a	bit
more	life	before	the	curtain	drops	or	to	provide	more	time	for	mitigation

or	else
some
efforts	that	might	reduce	the	danger.
.	Suppose	we	could	take	some	action	that	would	bring	the	intelligence
explosion	closer	by	one	year.	Let	us	say	that	the	people	currently	inhabiting
the	Earth	are	dying	off	at	a	rate	of	1%	per	year,	and	that	the	default	risk	of
human	extinction	from	the	intelligence	explosion	is	20%	(to	pick	an
arbitrary	number	for	the	purposes	of	illustration	only).	Then	hastening	the
arrival	of	the	intelligence	explosion	by	1	year	might	be	worth	(from	a

person-affecting	standpoint)	an	increase	in	the	risk	from	20%	to	21%,	
i.e.
	a
5%	increase	in	risk	level.	However,	the	vast	majority	
of	people	alive	one
year	before	the	start	of	the	intelligence	explosion	would	at	that	point	have
an	interest	in	postponing	it	if	they	could	thereby	reduce	the	risk	of	the
explosion	by	one	percentage	point	(since	most	individuals	would	reckon
their	own	risk	of	dying	in	the	next	year	to	be	much	smaller	than	1%—given
that	most	mortality	occurs	in	relatively	narrow	demographics	such	as	the

i.e.
	a
frail	and	the	elderly).	One	could	thus	have	a	model	in	which	each	year	the
population	votes	to	postpone	the	intelligence	explosion	by	another	year,	so
that	the	intelligence	explosion	never	happens,	although	everybody	who	ever
lives	agrees	that	it	would	be	better	if	the	intelligence	explosion	happened	at
some	point.	In	reality,	of	course,	coordination	failures,	limited
predictability,	or	preferences	for	things	other	than	personal	survival	are

likely	to	prevent	such	an	unending	pause.
If	one	uses	the	standard	economic	discount	factor	instead	of	the	person-
affecting	standard,	the	magnitude	of	the	potential	upside	is	diminished,
since	the	value	of	existing	people	getting	to	enjoy	astronomically	long	lives
is	then	steeply	discounted.	This	effect	is	especially	strong	if	the	discount
factor	is	applied	to	each	individual’s	subjective	time	rather	than	to	sidereal
time.	If	future	benefits	are	discounted	at	a	rate	of	
x
%	per	year,	and	the
background	level	of	existential	risk	from	other	sources	is	
y
%	per	year,	then
the	optimum	point	for	the	intelligence	explosion	would	be	when	delaying
the	explosion	for	another	year	would	produce	less	than	
x
	+	
y
	percentage
points	of	reduction	of	the	existential	risk	associated	with	an	intelligence
.	I	am	indebted	to	Carl	Shulman	and	Stuart	Armstrong	for	help	with	this
model.	See	also	Shulman	(2010a,	3):	“Chalmers	(2010)	reports	a	consensus

x
y
x
	+	
y
among	cadets	and	staff	at	the	U.S.	West	Point	military	academy	that	the
U.S.	government	would	not	restrain	AI	research	even	in	the	face	of
potential	catastrophe,	for	fear	that	rival	powers	would	gain	decisive
advantage.”
.	That	is,	information	in	the	model	is	always	bad	
ex	ante
.	Of	course,
depending	on	what	the	information	actually	is,	it	will	in	some	cases	turn	out
to	be	good	that	the	information	became	known,	notably	if	the	gap	between
leader	and	runner-up	is	much	greater	than	one	would	reasonably	have
guessed	in	advance.

ex	ante
.	It	might	even	present	an	existential	risk,	especially	if	preceded	by	the
introduction	of	novel	military	technologies	of	destruction	or	unprecedented
arms	buildups.
.	A	project	could	have	its	workers	distributed	over	a	large	number	of	locations
and	collaborating	via	encrypted	communications	channels.	But	this	tactic
involves	a	security	trade-off:	while	geographical	dispersion	may	offer	some
protection	against	military	attacks,	it	would	impede	operational	security,
since	it	is	harder	to	prevent	personnel	from	defecting,	leaking	information,

or	being	abducted	by	a	rival	power	if	they	are	spread	out	over	many
locations.

.	Note	that	a	large	temporal	discount	factor	could	make	a	project	behave	in
some	ways	as	though	it	were	in	a	race,	even	if	it	knows	it	has	no	real
competitor.	The	large	discount	factor	means	it	would	care	little	about	the	far
future.	Depending	on	the	situation,	this	would	discourage	blue-sky	R&D,
which	would	tend	to	delay	the	machine	intelligence	revolution	(though
perhaps	making	it	more	abrupt	when	it	does	occur,	because	of	hardware
overhang).	But	the	large	discount	factor—or	a	low	level	of	caring	for	future
generations—would	also	make	existential	risks	seem	to	matter	less.	This
would	encourage	gambles	that	involve	the	possibility	of	an	immediate	gain
at	the	expense	of	an	increased	risk	of	existential	catastrophe,	thus
disincentivizing	safety	investment	and	incentivizing	an	early	launch—
mimicking	the	effects	of	the	race	dynamic.	By	contrast	to	the	race	dynamic,
however,	a	large	discount	factor	(or	disregard	for	future	generations)	would
have	no	particular	tendency	to	incite	conflict.

Reducing	the	race	dynamic	is	a	main	benefit	of	collaboration.	That
collaboration	would	facilitate	sharing	of	ideas	for	how	to	solve	the	control
problem	is	also	a	benefit,	although	this	is	to	some	extent	counterbalanced
by	the	fact	that	collaboration	would	also	facilitate	sharing	of	ideas	for	how
to	solve	the	competence	problem.	The	net	effect	of	this	facilitation	of	idea-
sharing	may	be	to	slightly	increase	the	collective	intelligence	of	the	relevant
research	community.

.	On	the	other	hand,	public	oversight	by	a	single	government	would	risk
producing	an	outcome	in	which	one	nation	monopolizes	the	gains.	This
outcome	seems	inferior	to	one	in	which	unaccountable	altruists	ensure	that
everybody	stands	to	gain.	Furthermore,	oversight	by	a	national	government
would	not	necessarily	mean	that	even	all	the	citizens	of	that	country	receive
a	share	
of	the	benefit:	depending	on	the	country	in	question,	there	is	a
greater	or	smaller	risk	that	all	the	benefits	would	be	captured	by	a	political

a	share	
elite	or	a	few	self-serving	agency	personnel.
.	One	qualification	is	that	the	use	of	incentive	wrapping	(as	discussed	in
Chapter	12
)	might	in	some	circumstances	encourage	people	to	join	a	project
as	active	collaborators	rather	than	passive	free-riders.
.	Diminishing	returns	would	seem	to	set	in	at	a	much	smaller	scale.	Most
people	would	rather	have	one	star	than	a	one-in-a-billion	chance	of	a	galaxy
with	a	billion	stars.	Indeed,	most	people	would	rather	have	a	billionth	of	the

resources	on	Earth	than	a	one-in-a-billion	chance	of	owning	the	entire
planet.
.	Cf.	Shulman	(2010a).
.	Aggregative	ethical	theories	run	into	trouble	when	the	idea	that	the	cosmos
might	be	infinite	is	taken	seriously;	see	Bostrom	(2011b).	There	may	also
be	trouble	when	the	idea	of	ridiculously	large	but	finite	values	is	taken
seriously;	see	Bostrom	(2009b).
.	If	one	makes	a	computer	larger,	one	eventually	faces	relativistic	constraints
arising	from	communication	latencies	between	the	different	parts	of	the
computer—signals	do	not	propagate	faster	than	light.	If	one	shrinks	the
computer,	one	encounters	quantum	limits	to	miniaturization.	If	one
increases	the	density	of	the	computer,	one	slams	into	the	black	hole	limit.
Admittedly,	we	cannot	be	completely	certain	that	new	physics	will	not	one
day	be	discovered	offering	some	way	around	these	limitations.
.	The	number	of	copies	of	a	person	would	scale	linearly	with	resources	with
no	upper	bound.	Yet	it	is	not	clear	how	much	the	average	human	being

would	value	having	multiple	copies	of	herself.	Even	those	people	who
would	prefer	to	be	multiply	instantiated	may	not	have	a	utility	function	that
is	linear	with	increasing	number	of	copies.	Copy	numbers,	like	life	years,
might	have	diminishing	returns	in	the	typical	person’s	utility	function.
.	A	singleton	is	highly	internally	collaborative	at	the	highest	level	of	decision-
making.	A	singleton	
could
	have	a	lot	of	non-collaboration	and	conflict	at
lower	levels,	if	the	higher-level	agency	that	constitutes	the	singleton

could
chooses	to	have	things	that	way.
.	If	each	rival	AI	team	is	convinced	that	the	other	teams	are	so	misguided	as	to
have	no	chance	of	producing	an	intelligence	explosion,	then	one	reason	for
collaboration—avoiding	the	race	dynamic—is	obviated:	each	team	should
independently	choose	to	go	slower	in	the	confident	belief	that	it	lacks	any
serious	competition.
.	A	PhD	student.
.	This	formulation	is	intended	to	be	read	so	as	to	include	a	prescription	that	the
well-being	of	nonhuman	animals	and	other	sentient	beings	(including

digital	minds)	that	exist	or	may	come	to	exist	be	given	due	consideration.	It
is	not	meant	to	be	read	as	a	license	for	one	AI	developer	to	substitute	his	or
her	own	moral	intuitions	for	those	of	the	wider	moral	community.	The
principle	is	consistent	with	the	“coherent	extrapolated	volition”	approach
discussed	in	
Chapter	12
,	with	an	extrapolation	base	encompassing	all
humans.
A	further	clarification:	The	formulation	is	not	intended	to	necessarily
exclude	the	possibly	of	post-transition	property	rights	in	artificial
superintelligences	or	their	constituent	algorithms	and	data	structures.	The
formulation	is	meant	to	be	agnostic	about	what	legal	or	political	systems
would	best	serve	to	organize	transactions	within	a	hypothetical	future
posthuman	society.	What	the	formulation	
is
	meant	to	assert	is	that	the
choice	of	such	a	system,	insofar	as	its	selection	is	causally	determined	by
how	superintelligence	is	initially	developed,	should	to	be	made	on	the	basis

is
of	the	stated	criterion;	that	is,	the	post-transition	constitutional	system
should	be	chosen	for	the	benefit	of	all	of	humanity	and	in	the	service	of
widely	shared	ethical	ideals—as	opposed	to,	for	instance,	for	the	benefit
merely	of	whoever	happened	to	be	the	first	to	develop	superintelligence.
.	Refinements	of	the	windfall	clause	are	obviously	possible.	For	example,
perhaps	the	threshold	should	be	expressed	in	
per	capita
	terms,	or	maybe	the
winner	should	be	allowed	to	keep	a	somewhat	larger	than	equal	share	of	the

per	capita
overshoot	in	order	to	more	strongly	incentivize	further	production	(some
version	of	Rawls’s	maximin	principle	might	be	attractive	here).	Other
refinements	would	refocus	the	clause	away	from	dollar	amounts	and	restate
it	in	terms	of	“influence	on	humanity’s	future”	or	“degree	to	which	different
parties’	interests	are	weighed	in	a	future	singleton’s	utility	function”	or
some	such.
CHAPTER	15:	CRUNCH	TIME
	
1
.	Some	research	is	worthwhile	not	because	of	what	it	discovers	but	for	other
reasons,	such	as	by	entertaining,	educating,	accrediting,	or	uplifting	those

some	such.
	
1
who	engage	in	it.
2
.	I	am	not	suggesting	that	
nobody
	should	work	on	pure	mathematics	or

philosophy.	I	am	also	not	suggesting	that	these	endeavors	are	especially
wasteful	compared	to	all	the	other	dissipations	of	academia	or	society	at
large.	It	is	probably	very	good	that	some	people	can	devote	themselves	to
the	life	of	the	mind	and	follow	their	intellectual	curiosity	wherever	it	leads,
independent	of	any	thought	of	utility	or	impact.	The	suggestion	is	that	at	the
margin,	some	of	the	best	minds	might,	upon	realizing	that	their	cognitive
performances	may	become	obsolete	in	the	foreseeable	future,	want	to	shift
their	attention	to	those	theoretical	problems	for	which	it	makes	a	difference
whether	we	get	the	solution	a	little	sooner.
3
.	Though	one	should	be	cautious	in	cases	where	this	uncertainty	may	be
protective—recall,	for	instance,	the	risk-race	model	in	
Box	13
,	where	we
found	that	additional	strategic	information	could	be	harmful.	More
generally,	we	need	to	worry	about	information	hazards	(see	Bostrom
[2011b]).	It	is	tempting	to	say	that	we	need	more	analysis	of	information

3
Box	13
,	where	we
hazards.	This	is	probably	true,	although	we	might	still	worry	that	such
analysis	itself	may	produce	dangerous	information.
4
5
.	I	am	grateful	to	Carl	Shulman	for	emphasizing	this	point.

BIBLIOGRAPHY
	
Acemoglu,	Daron.	2003.	“Labor-and	Capital-Augmenting	Technical	Change.”
Journal	of	the	European	Economic	Association
	1	(1):	1–37.
Albertson,	D.	G.,	and	Thomson,	J.	N.	1976.	“The	Pharynx	of	
Caenorhabditis
Elegans
.”	
Philosophical	Transactions	of	the	Royal	Society	B:	Biological
Sciences
	275	(938):	299–325.
Allen,	Robert	C.	2008.	“A	Review	of	Gregory	Clark’s	
A	Farewell	to	Alms:	A	Brief
Economic	History	of	the	World
.”	
Journal	of	Economic	Literature
	46	(4):	946–73.
American	Horse	Council.	2005.	“National	Economic	Impact	of	the	US	Horse
Industry.”	Retrieved	July	30,	2013.	Available	at
http://www.horsecouncil.org/national-economic-impact-us-horse-industry
.
Anand,	Paul,	Pattanaik,	Prasanta,	and	Puppe,	Clemens,	eds.	2009.	
The	Oxford
Handbook	of	Rational	and	Social	Choice
.	New	York:	Oxford	University	Press.
Andres,	B.,	Koethe,	U.,	Kroeger,	T.,	Helmstaedter,	M.,	Briggman,	K.	L.,	Denk,	W.,
and	Hamprecht,	F.	A.	2012.	“3D	Segmentation	of	SBFSEM	Images	of	Neuropil

.
The	Oxford
by	a	Graphical	Model	over	Supervoxel	Boundaries.”	
Medical	Image	Analysis
	16
(4):	796–805.
Armstrong,	Alex.	2012.	“Computer	Competes	in	Crossword	Tournament.”	
I
Programmer
,	March	19.
Armstrong,	Stuart.	2007.	“Chaining	God:	A	Qualitative	Approach	to	AI,	Trust	and
Moral	Systems.”	Unpublished	manuscript,	October	20.	Retrieved	December	31,
2012.	Available	at	
http://www.neweuropeancentury.org/GodAI.pdf
.
Armstrong,	Stuart.	2010.	
Utility	Indifference
,	Technical	Report	2010-1.	Oxford:

	16
I
,	March	19.
.
Future	of	Humanity	Institute,	University	of	Oxford.
Armstrong,	Stuart.	2013.	“General	Purpose	Intelligence:	Arguing	the	Orthogonality

Thesis.”	
Analysis	and	Metaphysics
	12:	68–84.
Armstrong,	Stuart,	and	Sandberg,	Anders.	2013.	“Eternity	in	Six	Hours:
Intergalactic	Spreading	of	Intelligent	Life	and	Sharpening	the	Fermi	Paradox.”
Acta	Astronautica
	89:	1–13.
Armstrong,	Stuart,	and	Sotala,	Kaj.	2012.	“How	We’re	Predicting	AI—or	Failing
To.”	In	
Beyond	AI:	Artificial	Dreams
,	edited	by	Jan	Romportl,	Pavel	Ircing,	Eva
Zackova,	Michal	Polak,	and	Radek	Schuster,	52–75.	Pilsen:	University	of	West
Bohemia.	Retrieved	February	2,	2013.
Asimov,	Isaac.	1942.	“Runaround.”	
Astounding	Science-Fiction
,	March,	94–103.
Asimov,	Isaac.	1985.	
Robots	and	Empire
.	New	York:	Doubleday.
Aumann,	Robert	J.	1976.	“Agreeing	to	Disagree.”	
Annals	of	Statistics
	4	(6):	1236–
9.
Averch,	Harvey	Allen.	1985.	
A	Strategic	Analysis	of	Science	and	Technology
Policy
.	Baltimore:	Johns	Hopkins	University	Press.
Azevedo,	F.	A.	C.,	Carvalho,	L.	R.	B.,	Grinberg,	L.	T.,	Farfel,	J.	M.,	Ferretti,	R.	E.
L.,	Leite,	R.	E.	P.,	Jacob,	W.,	Lent,	R.,	and	Herculano-Houzel,	S.	2009.	“Equal

9.
Policy
Numbers	of	Neuronal	and	Nonneuronal	Cells	Make	the	Human	Brain	an
Isometrically	Scaled-up	Primate	Brain.”	
Journal	of	Comparative	Neurology
	513
(5):	532–41.
Baars,	Bernard	J.	1997.	
In	the	Theater	of	Consciousness:	The	Workspace	of	the
Mind
Baratta,	Joseph	Preston.	2004.	
The	Politics	of	World	Federation:	United	Nations,
UN	Reform,	Atomic	Control
.	Westport,	CT:	Praeger.
Barber,	E.	J.	W.	1991.	
Prehistoric	Textiles:	The	Development	of	Cloth	in	the

	513
Mind
Neolithic	and	Bronze	Ages	with	Special	Reference	to	the	Aegean
.	Princeton,	NJ:
Princeton	University	Press.
Bartels,	J.,	Andreasen,	D.,	Ehirim,	P.,	Mao,	H.,	Seibert,	S.,	Wright,	E.	J.,	and
Kennedy,	P.	2008.	“Neurotrophic	Electrode:	Method	of	Assembly	and
Implantation	into	Human	Motor	Speech	Cortex.”	
Journal	of	Neuroscience

Methods
	174	(2):	168–76.
Bartz,	Jennifer	A.,	Zaki,	Jamil,	Bolger,	Niall,	and	Ochsner,	Kevin	N.	2011.	“Social
Effects	of	Oxytocin	in	Humans:	Context	and	Person	Matter.”	
Trends	in	Cognitive
Science
	15	(7):	301–9.
Basten,	Stuart,	Lutz,	Wolfgang,	and	Scherbov,	Sergei.	2013.	“Very	Long	Range
Global	Population	Scenarios	to	2300	and	the	Implications	of	Sustained	Low
Fertility.”	
Demographic	Research
	28:	1145–66.
Baum,	Eric	B.	2004.	
What	Is	Thought?
	Bradford	Books.	Cambridge,	MA:	MIT
Press.
Baum,	Seth	D.,	Goertzel,	Ben,	and	Goertzel,	Ted	G.	2011.	“How	Long	Until
Human-Level	AI?	Results	from	an	Expert	Assessment.”	
Technological
Forecasting	and	Social	Change
	78	(1):	185–95.
Beal,	J.,	and	Winston,	P.	2009.	“Guest	Editors’	Introduction:	The	New	Frontier	of
Human-Level	Artificial	Intelligence.”	
IEEE	Intelligent	Systems
	24	(4):	21–3.
Bell,	C.	Gordon,	and	Gemmell,	Jim.	2009.	
Total	Recall:	How	the	E-Memory
Revolution	Will	Change	Everything
.	New	York:	Dutton.

Press.
Benyamin,	B.,	Pourcain,	B.	St.,	Davis,	O.	S.,	Davies,	G.,	Hansell,	M.	K.,	Brion,
M.-J.	A.,	Kirkpatrick,	R.	M.,	
et	al.
	2013.	“Childhood	Intelligence	is	Heritable,
Highly	Polygenic	and	Associated	With	FNBP1L.”	
Molecular	Psychiatry
(January	23).
Berg,	Joyce	E.,	and	Rietz,	Thomas	A.	2003.	“Prediction	Markets	as	Decision
Support	Systems.”	
Information	Systems	Frontiers
	5	(1):	79–93.
Berger,	Theodore	W.,	Chapin,	J.	K.,	Gerhardt,	G.	A.,	Soussou,	W.	V.,	Taylor,	D.
M.,	and	Tresco,	P.	A.,	eds.	2008.

et	al.
M.,	and	Tresco,	P.	A.,	eds.	2008.	
Brain–Computer	Interfaces:	An	International
Assessment	of	Research	and	Development	Trends
.	Springer.
Berger,	T.	W.,	Song,	D.,	Chan,	R.	H.,	Marmarelis,	V.	Z.,	LaCoss,	J.,	Wills,	J.,
Hampson,	R.	E.,	Deadwyler,	S.	A.,	and	Granacki,	J.	J.	2012.	“A	Hippocampal
Cognitive	Prosthesis:	Multi-Input,	Multi-Output	Nonlinear	Modeling	and	VLSI
Implementation.”	
IEEE	Transactions	on	Neural	Systems	and	Rehabilitation
Engineering
	20	(2):	198–211.

Berliner,	Hans	J.	1980a.	“Backgammon	Computer-Program	Beats	World
Champion.”	
Artificial	Intelligence
	14	(2):	205–220.
Berliner,	Hans	J.	1980b.	“Backgammon	Program	Beats	World	Champ.”	
SIGART
Newsletter
	69:	6–9.
Bernardo,	José	M.,	and	Smith,	Adrian	F.	M.	1994.	
Bayesian	Theory
,	1st	ed.	Wiley
Series	in	Probability	&	Statistics.	New	York:	Wiley.
Birbaumer,	N.,	Murguialday,	A.	R.,	and	Cohen,	L.	2008.	“Brain–Computer
Interface	in	Paralysis.”	
Current	Opinion	in	Neurology
	21	(6):	634–8.
Bird,	Jon,	and	Layzell,	Paul.	2002.	“The	Evolved	Radio	and	Its	Implications	for
Modelling	the	Evolution	of	Novel	Sensors.”	In	
Proceedings	of	the	2002
Congress	on	Evolutionary	Computation
,	2:	1836–41.
Blair,	Clay,	Jr.	1957.	“Passing	of	a	Great	Mind:	John	von	Neumann,	a	Brilliant,
Jovial	Mathematician,	was	a	Prodigious	Servant	of	Science	and	His	Country.”
Life
,	February	25,	89–104.
Bobrow,	Daniel	G.	1968.	“Natural	Language	Input	for	a	Computer	Problem
Solving	System.”	In	
Semantic	Information	Processing
,	edited	by	Marvin

Life
Minsky,	146–227.	Cambridge,	MA:	MIT	Press.
Bostrom,	Nick.	1997.	“Predictions	from	Philosophy?	How	Philosophers	Could
Make	Themselves	Useful.”	Unpublished	manuscript.	Last	revised	September	19,
1998.
Bostrom,	Nick.	2002a.	
Anthropic	Bias:	Observation	Selection	Effects	in	Science
and	Philosophy
.	New	York:	Routledge.
Bostrom,	Nick.	2002b.	“Existential	Risks:	Analyzing	Human	Extinction	Scenarios
and	Related	Hazards.”	
Journal	of	Evolution	and	Technology
	9.
Bostrom,	Nick.	2003a.	“Are	We	Living	in	a	Computer	Simulation?”	
Philosophical
Quarterly
	53	(211):	243–55.

1998.
	9.
Quarterly
Bostrom,	Nick.	2003b.	“Astronomical	Waste:	The	Opportunity	Cost	of	Delayed
Technological	Development.”	
Utilitas
	15	(3):	308–314.

Bostrom,	Nick.	2003c.	“Ethical	Issues	in	Advanced	Artificial	Intelligence.”	In
Cognitive,	Emotive	and	Ethical	Aspects	of	Decision	Making	in	Humans	and	in
,	edited	by	Iva	Smit	and	George	E.	Lasker,	2:	12–17.
Windsor,	ON:	International	Institute	for	Advanced	Studies	in	Systems	Research	/
Cybernetics.
Bostrom,	Nick.	2004.	“The	Future	of	Human	Evolution.”	In	
Two	Hundred	Years
After	Kant,	Fifty	Years	After	Turing
,	edited	by	Charles	Tandy,	2:	339–371.	Death
and	Anti-Death.	Palo	Alto,	CA:	Ria	University	Press.
Bostrom,	Nick.	2006a.	“How	Long	Before	Superintelligence?”	
Linguistic	and
Philosophical	Investigations
	5(1):	11–30.
Bostrom,	Nick.	2006b.	“Quantity	of	Experience:	Brain-Duplication	and	Degrees	of
Consciousness.”	
Minds	and	Machines
	16	(2):	185–200.
Bostrom,	Nick.	2006c.	“What	is	a	Singleton?”	
Linguistic	and	Philosophical
Investigations
	5	(2):	48–54.
Bostrom,	Nick.	2007.	“Technological	Revolutions:	Ethics	and	Policy	in	the	Dark.”
In

In	
Nanoscale:	Issues	and	Perspectives	for	the	Nano	Century
,	edited	by	Nigel	M.
de	S.	Cameron	and	M.	Ellen	Mitchell,	129–52.	Hoboken,	NJ:	Wiley.
Bostrom,	Nick.	2008a.	“Where	Are	They?	Why	I	Hope	the	Search	for
Extraterrestrial	Life	Finds	Nothing.”	
MIT	Technology	Review
,	May/June	issue,
72–7.
Bostrom,	Nick.	2008b.	“Why	I	Want	to	Be	a	Posthuman	When	I	Grow	Up.”	In
Medical	Enhancement	and	Posthumanity
,	edited	by	Bert	Gordijn	and	Ruth
Chadwick,	107–37.	New	York:	Springer.
Bostrom,	Nick.	2008c.	“Letter	from	Utopia.”

72–7.
Bostrom,	Nick.	2008c.	“Letter	from	Utopia.”	
Studies	in	Ethics,	Law,	and
	2	(1):	1–7.
Bostrom,	Nick.	2009a.	“Moral	Uncertainty	–	Towards	a	Solution?”	
Overcoming
Bias
	(blog),	January	1.
Bostrom,	Nick.	2009b.	“Pascal’s	Mugging.”	
Analysis
	69	(3):	443–5.
Bostrom,	Nick.	2009c.	“The	Future	of	Humanity.”	In	
New	Waves	in	Philosophy	of

,	edited	by	Jan	Kyrre	Berg	Olsen,	Evan	Selinger,	and	Søren	Riis,
186–215.	New	York:	Palgrave	Macmillan.
Bostrom,	Nick.	2011a.	“Information	Hazards:	A	Typology	of	Potential	Harms	from
Knowledge.”	
Review	of	Contemporary	Philosophy
	10:	44–79.
Bostrom,	Nick.	2011b.	“Infinite	Ethics.”	
	10:	9–59.
Bostrom,	Nick.	2012.	“The	Superintelligent	Will:	Motivation	and	Instrumental
Rationality	in	Advanced	Artificial	Agents.”	In	“Theory	and	Philosophy	of	AI,”
edited	by	Vincent	C.	Müller,	special	issue,	
	22	(2):	71–85.
Bostrom,	Nick,	and	Ćirković,	Milan	M.	2003.	“The	Doomsday	Argument	and	the
Self-Indication	Assumption:	Reply	to	Olum.”	
Philosophical	Quarterly
	53	(210):
83–91.
Bostrom,	Nick,	and	Ord,	Toby.	2006.	“The	Reversal	Test:	Eliminating	the	Status
Quo	Bias	in	Applied	Ethics.”	
Ethics
	116	(4):	656–79.
Bostrom,	Nick,	and	Roache,	Rebecca.	2011.	“Smart	Policy:	Cognitive
Enhancement	and	the	Public	Interest.”	In	
Enhancing	Human	Capacities
,	edited

	53	(210):
83–91.
Ethics
,	edited
by	Julian	Savulescu,	Ruud	ter	Meulen,	and	Guy	Kahane,	138–49.	Malden,	MA:
Wiley-Blackwell.
Bostrom,	Nick	and	Sandberg,	Anders.	2009a.	“Cognitive	Enhancement:	Methods,
Ethics,	Regulatory	Challenges.”	
Science	and	Engineering	Ethics
	15	(3):	311–41.
Bostrom,	Nick	and	Sandberg,	Anders.	2009b.	“The	Wisdom	of	Nature:	An
Evolutionary	Heuristic	for	Human	Enhancement.”	In	
Human	Enhancement
,	1st
ed.,	edited	by	Julian	Savulescu	and	Nick	Bostrom,	375–416.	New	York:	Oxford
University	Press.

,	edited
,	1st
Bostrom,	Nick,	Sandberg,	Anders,	and	Douglas,	Tom.	2013.	“The	Unilateralist’s
Curse:	The	Case	for	a	Principle	of	Conformity.”	Working	Paper.	Retrieved
February	 28,	 2013.	 Available	 at
http://www.nickbostrom.com/papers/unilateralist.pdf
.
Bostrom,	Nick,	and	Yudkowsky,	Eliezer.	Forthcoming.	“The	Ethics	of	Artificial
Intelligence.”	In	
Cambridge	Handbook	of	Artificial	Intelligence
,	edited	by	Keith
Frankish	and	William	Ramsey.	New	York:	Cambridge	University	Press.

Boswell,	James.	1917.	
Boswell’s	Life	of	Johnson
.	New	York:	Oxford	University
Press.
Bouchard,	T.	J.	2004.	“Genetic	Influence	on	Human	Psychological	Traits:	A
Survey.”	
Current	Directions	in	Psychological	Science
	13	(4):	148–51.
Bourget,	David,	and	Chalmers,	David.	2009.	“The	PhilPapers	Surveys.”	November.
Available	at	
http://philpapers.org/surveys/
.
Bradbury,	Robert	J.	1999.	“Matrioshka	Brains.”	Archived	version.	As	revised
August	 16,	 2004.	 Available	 at
http://web.archive.org/web/20090615040912/http://www.aeiveos.com/~bradbury/MatrioshkaBrains/MatrioshkaBrainsPaper.html
Brinton,	Crane.	1965.	
The	Anatomy	of	Revolution
.	Revised	ed.	New	York:	Vintage
Books.
Bryson,	Arthur	E.,	Jr.,	and	Ho,	Yu-Chi.	1969.	
Applied	Optimal	Control:
Optimization,	Estimation,	and	Control
.	Waltham,	MA:	Blaisdell.
Buehler,	Martin,	Iagnemma,	Karl,	and	Singh,	Sanjiv,	eds.	2009.	
The	DARPA	Urban
Challenge:	Autonomous	Vehicles	in	City	Traffic
.	Springer	Tracts	in	Advanced
Robotics	56.	Berlin:	Springer.

Books.
Burch-Brown,	J.	2014.	“Clues	for	Consequentialists.”	
Utilitas
	26	(1):	105–19.
Burke,	Colin.	2001.	“Agnes	Meyer	Driscoll	vs.	the	Enigma	and	the	Bombe.”
Unpublished	manuscript.	Retrieved	February	22,	2013.	Available	at
http://userpages.umbc.edu/~burke/driscoll1-2011.pdf
.
Canbäck,	S.,	Samouel,	P.,	and	Price,	D.	2006.	“Do	Diseconomies	of	Scale	Impact
Firm	Size	and	Performance?	A	Theoretical	and	Empirical	Overview.”	
Journal	of
Managerial	Economics
	4	(1):	27–70.
Carmena,	J.	M.,	Lebedev,	M.	A.,	Crist,	R.	E.,	O’Doherty,	J.	E.,	Santucci,	D.	M.,

Utilitas
.
Journal	of
Dimitrov,	D.	F.,	Patil,	P.	G.,	Henriquez,	C.	S.,	and	Nicolelis,	M.	A.	2003.
“Learning	to	Control	a	Brain–Machine	Interface	for	Reaching	and	Grasping	by
Primates.”	
Public	Library	of	Science	Biology
	1	(2):	193–208.
Carroll,	Bradley	W.,	and	Ostlie,	Dale	A.	2007.	An	Introduction	to	Modern
Astrophysics.	2nd	ed.	San	Francisco:	Pearson	Addison	Wesley.

Carroll,	John	B.	1993.	
Human	Cognitive	Abilities:	A	Survey	of	Factor-Analytic
Studies
.	New	York:	Cambridge	University	Press.
Carter,	Brandon.	1983.	“The	Anthropic	Principle	and	its	Implications	for
Biological	Evolution.”	
Philosophical	Transactions	of	the	Royal	Society	A:
Mathematical,	Physical	and	Engineering	Sciences
	310	(1512):	347–63.
Carter,	Brandon.	1993.	“The	Anthropic	Selection	Principle	and	the	Ultra-
Darwinian	Synthesis.”	In	
The	Anthropic	Principle:	Proceedings	of	the	Second
Venice	Conference	on	Cosmology	and	Philosophy
,	edited	by	F.	Bertola	and	U.
Curi,	33–66.	Cambridge:	Cambridge	University	Press.
CFTC	&	SEC	(Commodity	Futures	Trading	Commission	and	Securities	&
Exchange	Commission).	2010.	
Findings	Regarding	the	Market	Events	of	May	6,
2010:	Report	of	the	Staffs	of	the	CFTC	and	SEC	to	the	Joint	Advisory	Committee
on	Emerging	Regulatory	Issues
.	Washington,	DC.
Chalmers,	David	John.	2010.	“The	Singularity:	A	Philosophical	Analysis.”	
Journal
of	Consciousness	Studies
	17	(9–10):	7–65.

Journal
Chason,	R.	J.,	Csokmay,	J.,	Segars,	J.	H.,	DeCherney,	A.	H.,	and	Armant,	D.	R.
2011.	“Environmental	and	Epigenetic	Effects	Upon	Preimplantation	Embryo
Metabolism	and	Development.”	
Trends	in	Endocrinology	and	Metabolism
	22
(10):	412–20.
Chen,	S.,	and	Ravallion,	M.	2010.	“The	Developing	World	Is	Poorer	Than	We
Thought,	But	No	Less	Successful	in	the	Fight	Against	Poverty.”	
Quarterly
Journal	of	Economics
	125	(4):	1577–1625.
Chislenko,	Alexander.	1996.	“Networking	in	the	Mind	Age:	Some	Thoughts	on

	22
Quarterly
Evolution	of	Robotics	and	Distributed	Systems.”	Unpublished	manuscript.
Chislenko,	Alexander.	1997.	“Technology	as	Extension	of	Human	Functional
Architecture.”	
Extropy	Online
.
Chorost,	Michael.	2005.	
Rebuilt:	How	Becoming	Part	Computer	Made	Me	More
Human
.	Boston:	Houghton	Mifflin.
Christiano,	Paul	F.	2012.	“‘Indirect	Normativity’	Write-up.”	
Ordinary	Ideas
	(blog),
April	21.

CIA.	2013.	“The	World	Factbook.”	Central	Intelligence	Agency.	Retrieved	August
3.	Available	at	
https://www.cia.gov/library/publications/the-world-
factbook/rankorder/2127rank.html?
countryname=United%20States&countrycode=us&regionCode=noa&rank=121#us
Cicero.	1923.	“On	Divination.”	In	
On	Old	Age,	on	Friendship,	on	Divination
,
translated	by	W.	A.	Falconer.	Loeb	Classical	Library.	Cambridge,	MA:	Harvard
Cirasella,	Jill,	and	Kopec,	Danny.	2006.	“The	History	of	Computer	Games.”
Exhibit	at	Dartmouth	Artificial	Intelligence	Conference:	The	Next	Fifty	Years
(AI@50),	Dartmouth	College,	July	13–15.
Ćirković,	Milan	M.	2004.	“Forecast	for	the	Next	Eon:	Applied	Cosmology	and	the
Long-Term	Fate	of	Intelligent	Beings.”	
Foundations	of	Physics
	34	(2):	239–61.
Ćirković,	Milan	M.,	Sandberg,	Anders,	and	Bostrom,	Nick.	2010.	“Anthropic
Shadow:	Observation	Selection	Effects	and	Human	Extinction	Risks.”	
Risk
Analysis
	30	(10):	1495–1506.
Clark,	Andy,	and	Chalmers,	David	J.	1998.	“The	Extended	Mind.”

Risk
Analysis
Clark,	Andy,	and	Chalmers,	David	J.	1998.	“The	Extended	Mind.”	
Analysis
	58	(1):
7–19.
Clark,	Gregory.	2007.	
A	Farewell	to	Alms:	A	Brief	Economic	History	of	the	World
.
1st	ed.	Princeton,	NJ:	Princeton	University	Press.
Clavin,	Whitney.	2012.	“Study	Shows	Our	Galaxy	Has	at	Least	100	Billion
Planets.”	
Jet	Propulsion	Laboratory
,	January	11.
CME	Group.	2010.	
What	Happened	on	May	6th?
	Chicago,	May	10.
Coase,	R.	H.	1937.	“The	Nature	of	the	Firm.”	
Economica
	4	(16):	386–405.
Cochran,	Gregory,	and	Harpending,	Henry.	2009.	
The	10,000	Year	Explosion:	How
Civilization	Accelerated	Human	Evolution
.	New	York:	Basic	Books.

.
Planets.”	
Economica
Cochran,	G.,	Hardy,	J.,	and	Harpending,	H.	2006.	“Natural	History	of	Ashkenazi
Intelligence.”	
Journal	of	Biosocial	Science
	38	(5):	659–93.
Cook,	James	Gordon.	1984.	
Handbook	of	Textile	Fibres:	Natural	Fibres
.

Cambridge:	Woodhead.
Cope,	David.	1996.	
Experiments	in	Musical	Intelligence
.	Computer	Music	and
Digital	Audio	Series.	Madison,	WI:	A-R	Editions.
Cotman,	Carl	W.,	and	Berchtold,	Nicole	C.	2002.	“Exercise:	A	Behavioral
Intervention	to	Enhance	Brain	Health	and	Plasticity.”	
Trends	in	Neurosciences
25	(6):	295–301.
Cowan,	Nelson.	2001.	“The	Magical	Number	4	in	Short-Term	Memory:	A
Reconsideration	of	Mental	Storage	Capacity.”	
Behavioral	and	Brain	Sciences
	24
(1):	87–114.
Crabtree,	Steve.	1999.	“New	Poll	Gauges	Americans’	General	Knowledge	Levels.”
Gallup	News
,	July	6.
Cross,	Stephen	E.,	and	Walker,	Edward.	1994.	“Dart:	Applying	Knowledge	Based
Planning	and	Scheduling	to	Crisis	Action	Planning.”	In	
Intelligent	Scheduling
,
edited	by	Monte	Zweben	and	Mark	Fox,	711–29.	San	Francisco,	CA:	Morgan
Kaufmann.
Crow,	James	F.	2000.	“The	Origins,	Patterns	and	Implications	of	Human
Spontaneous	Mutation.”	
Nature	Reviews	Genetics
	1	(1):	40–7.
Cyranoski,	David.	2013.	“Stem	Cells:	Egg	Engineers.”	
Nature

,	July	6.
,
Kaufmann.
Nature
	500	(7463):	392–4.
Dagnelie,	Gislin.	2012.	“Retinal	Implants:	Emergence	of	a	Multidisciplinary
Field.”	
	25	(1):	67–75.
Dai,	Wei.	2009.	“Towards	a	New	Decision	Theory.”	
Less	Wrong
	(blog),	August	13.
Dalrymple,	David.	2011.	“Comment	on	Kaufman,	J.	‘Whole	Brain	Emulation:
Looking	at	Progress	on	
C.	Elegans
.’”	
Less	Wrong
	(blog),	October	29.
Davies,	G.,	Tenesa,	A.,	Payton,	A.,	Yang,	J.,	Harris,	S.	E.,	Liewald,	D.,	Ke,	X.,	
et
al.
	2011.	“Genome-Wide	Association	Studies	Establish	That	Human	Intelligence

Field.”	
Less	Wrong
C.	Elegans
.’”	
Less	Wrong
et
al.
Is	Highly	Heritable	and	Polygenic.”	
	16	(10):	996–1005.
Davis,	Oliver	S.	P.,	Butcher,	Lee	M.,	Docherty,	Sophia	J.,	Meaburn,	Emma	L.,
Curtis,	Charles	J.	C.,	Simpson,	Michael	A.,	Schalkwyk,	Leonard	C.,	and	Plomin,

Robert.	2010.	“A	Three-Stage	
Genome-Wide	Association	Study	of	General
Cognitive	Ability:	Hunting	the	Small	Effects.”	
Behavior	Genetics
	40	(6):	759–
767.
Dawkins,	Richard.	1995.	
River	Out	of	Eden:	A	Darwinian	View	of	Life
.	Science
Masters	Series.	New	York:	Basic	Books.
De	Blanc,	Peter.	2011.	
Ontological	Crises	in	Artificial	Agents’	Value	Systems
.
Machine	Intelligence	Research	Institute,	San	Francisco,	CA,	May	19.
De	Long,	J.	Bradford.	1998.	“Estimates	of	World	GDP,	One	Million	B.C.–Present.”
Unpublished	manuscript.
De	Raedt,	Luc,	and	Flach,	Peter,	eds.	2001.	
Machine	Learning:	ECML	2001:	12th
European	Conference	on	Machine	Learning,	Freiburg,	Germany,	September	5–
7,	2001
.	
Proceedings
.	Lecture	Notes	in	Computer	Science	2167.	New	York:
Springer.
Dean,	Cornelia.	2005.	“Scientific	Savvy?	In	U.S.,	Not	Much.”	
,
August	30.
Deary,	Ian	J.	2001.	“Human	Intelligence	Differences:	A	Recent	History.”	
Trends	in
Cognitive	Sciences
	5	(3):	127–30.
Deary,	Ian	J.	2012.	“Intelligence.”

7,	2001
.	
Springer.
,
August	30.
Trends	in
Deary,	Ian	J.	2012.	“Intelligence.”	
Annual	Review	of	Psychology
	63:	453–82.
Deary,	Ian	J.,	Penke,	L.,	and	Johnson,	W.	2010.	“The	Neuroscience	of	Human
Intelligence	Differences.”	
	11	(3):	201–11.
Degnan,	G.	G.,	Wind,	T.	C.,	Jones,	E.	V.,	and	Edlich,	R.	F.	2002.	“Functional
Electrical	Stimulation	in	Tetraplegic	Patients	to	Restore	Hand	Function.”
Journal	of	Long-Term	Effects	of	Medical	Implants
	12	(3):	175–88.
Devlin,	B.,	Daniels,	M.,	and	Roeder,	K.	1997.	“The	Heritability	of	IQ.”	
Nature
	388
(6641):	468–71.

Nature
	388
Dewey,	Daniel.	2011.	“Learning	What	to	Value.”	In	
Artificial	General	Intelligence:
4th	International	Conference,	AGI	2011,	Mountain	View,	CA,	USA,	August	3–6,
2011.	Proceedings
,	edited	by	Jürgen	Schmidhuber,	Kristinn	R.	Thórisson,	and
Moshe	Looks,	309–14.	Lecture	Notes	in	Computer	Science	6830.	Berlin:

Springer.
Dowe,	D.	L.,	and	Hernández-Orallo,	J.	2012.	“IQ	Tests	Are	Not	for	Machines,
Yet.”	
	40	(2):	77–81.
Drescher,	Gary	L.	2006.	
Good	and	Real:	Demystifying	Paradoxes	from	Physics	to
Ethics
.	Bradford	Books.	Cambridge,	MA:	MIT	Press.
Drexler,	K.	Eric.	1986.	
Engines	of	Creation
.	Garden	City,	NY:	Anchor.
Drexler,	K.	Eric.	1992.	
Nanosystems:	Molecular	Machinery,	Manufacturing,	and
Computation
.	New	York:	Wiley.
Drexler,	K.	Eric.	2013.	
Radical	Abundance:	How	a	Revolution	in	Nanotechnology
Will	Change	Civilization
.	New	York:	PublicAffairs.
Driscoll,	Kevin.	2012.	“Code	Critique:	‘Altair	Music	of	a	Sort.’”	Paper	presented	at
Critical	Code	Studies	Working	Group	Online	Conference,	2012,	February	6.
Dyson,	Freeman	J.	1960.	“Search	for	Artificial	Stellar	Sources	of	Infrared
Radiation.”	
Science
	131	(3414):	1667–1668.
Dyson,	Freeman	J.	1979.	
Disturbing	the	Universe
.	1st	ed.	Sloan	Foundation
Science	Series.	New	York:	Harper	&	Row.
Elga,	Adam.	2004.	“Defeating	Dr.	Evil	with	Self-Locating	Belief.”

Science
Elga,	Adam.	2004.	“Defeating	Dr.	Evil	with	Self-Locating	Belief.”	
Philosophy	and
Phenomenological	Research
	69	(2):	383–96.
Elga,	Adam.	2007.	“Reflection	and	Disagreement.”	
Noûs
	41	(3):	478–502.
Eliasmith,	Chris,	Stewart,	Terrence	C.,	Choo,	Xuan,	Bekolay,	Trevor,	DeWolf,
Travis,	Tang,	Yichuan,	and	Rasmussen,	Daniel.	2012.	“A	Large-Scale	Model	of
the	Functioning	Brain.”	
Science
	338(6111):	1202–5.
Ellis,	J.	H.	1999.	“The	History	of	Non-Secret	Encryption.”	
Cryptologia
	23	(3):
267–73.
Elyasaf,	Achiya,	Hauptmann,	Ami,	and	Sipper,	Moche.	2011.	“Ga-Freecell:

Noûs
Science
	23	(3):
267–73.
Evolving	Solvers	for	the	Game	of	Freecell.”	In	
Proceedings	of	the	13th	Annual
Genetic	and	Evolutionary	Computation	Conference
,	1931–1938.	GECCO’	11.

New	York:	ACM.
Eppig,	C.,	Fincher,	C.	L.,	and	Thornhill,	R.	2010.	“Parasite	Prevalence	and	the
Worldwide	Distribution	of	Cognitive	Ability.”	
Proceedings	of	the	Royal	Society
B:	Biological	Sciences
	277	(1701):	3801–8.
Espenshade,	T.	J.,	Guzman,	J.	C.,	and	Westoff,	C.	F.	2003.	“The	Surprising	Global
Variation	in	Replacement	Fertility.”	
Population	Research	and	Policy	Review
	22
(5–6):	575–83.
Evans,	Thomas	G.	1964.	“A	Heuristic	Program	to	Solve	Geometric-Analogy
Problems.”	In	
Proceedings	of	the	April	21–23,	1964,	Spring	Joint	Computer
Conference
,	327–338.	AFIPS	’64.	New	York:	ACM.
Evans,	Thomas	G.	1968.	“A	Program	for	the	Solution	of	a	Class	of	Geometric-
Analogy	Intelligence-Test	Questions.”	In	
,
edited	by	Marvin	Minsky,	271–353.	Cambridge,	MA:	MIT	Press.
Faisal,	A.	A.,	Selen,	L.	P.,	and	Wolpert,	D.	M.	2008.	“Noise	in	the	Nervous
System.”	
	9	(4):	292–303.
Faisal,	A.	A.,	White,	J.	A.,	and	Laughlin,	S.	B.	2005.	“Ion-Channel	Noise	Places

,
System.”	
Limits	on	the	Miniaturization	of	the	Brain’s	Wiring.”	
Current	Biology
	15	(12):
1143–9.
Feldman,	Jacob.	2000.	“Minimization	of	Boolean	Complexity	in	Human	Concept
Learning.”	
Nature
	407	(6804):	630–3.
Feldman,	J.	A.,	and	Ballard,	Dana	H.	1982.	“Connectionist	Models	and	Their
Properties.”	
Cognitive	Science
	6	(3):	205–254.
Foley,	J.	A.,	Monfreda,	C.,	Ramankutty,	N.,	and	Zaks,	D.	2007.	“Our	Share	of	the
Planetary	Pie.”	
Proceedings	of	the	National	Academy	of	Sciences	of	the	United
States	of	America
	104	(31):	12585–6.

	15	(12):
1143–9.
Nature
Forgas,	Joseph	P.,	Cooper,	Joel,	and	Crano,	William	D.,	eds.	2010.	
The	Psychology
of	Attitudes	and	Attitude	Change
.	Sydney	Symposium	of	Social	Psychology.
New	York:	Psychology	Press.
Frank,	Robert	H.	1999.	
Luxury	Fever:	Why	Money	Fails	to	Satisfy	in	an	Era	of

Excess
.	New	York:	Free	Press.
Fredriksen,	Kaja	Bonesmo.	2012.	
Less	Income	Inequality	and	More	Growth	–	Are
They	Compatible?:	Part	6.	The	Distribution	of	Wealth
.	Technical	report,	OECD
Economics	Department	Working	Papers	929.	OECD	Publishing.
Freitas,	Robert	A.,	Jr.	1980.	“A	Self-Replicating	Interstellar	Probe.”	
Journal	of	the
British	Interplanetary	Society
	33:	251–64.
Freitas,	Robert	A.,	Jr.	2000.	“Some	Limits	to	Global	Ecophagy	by	Biovorous
Nanoreplicators,	with	Public	Policy	Recommendations.”	Foresight	Institute.
April.	Retrieved	July	28,	2013.	Available	at
http://www.foresight.org/nano/Ecophagy.html
.
Freitas,	Robert	A.,	Jr.,	and	Merkle,	Ralph	C.	2004.	
Kinematic	Self-Replicating
Machines
.	Georgetown,	TX:	Landes	Bioscience.
Gaddis,	John	Lewis.	1982.	
Strategies	of	Containment:	A	Critical	Appraisal	of
Postwar	American	National	Security	Policy
Press.
Gammoned.net.	2012.	“Snowie.”	Archived	version.	Retrieved	June	30.	Available	at

.
Machines
Press.
http://web.archive.org/web/20070920191840/http://www.gammoned.com/snowie.html
Gates,	Bill.	1975.	“Software	Contest	Winners	Announced.”	
Computer	Notes
	1	(2):
1.
Georgieff,	Michael	K.	2007.	“Nutrition	and	the	Developing	Brain:	Nutrient
Priorities	and	Measurement.”	
American	Journal	of	Clinical	Nutrition
	85	(2):
614S–620S.
Gianaroli,	Luca.	2000.	“Preimplantation	Genetic	Diagnosis:	Polar	Body	and
Embryo	Biopsy.”	Supplement,	
Human	Reproduction
	15	(4):	69–75.
Gilovich,	Thomas,	Griffin,	Dale,	and	Kahneman,	Daniel,	eds.	2002.	
Heuristics	and

	1	(2):
1.
	85	(2):
Biases:	The	Psychology	of	Intuitive	Judgment
.	New	York:	Cambridge	University
Press.
Gilster,	Paul.	2012.	“ESO:	Habitable	Red	Dwarf	Planets	Abundant.”	
Centauri
Dreams
	(blog),	March	29.

Goldstone,	Jack	A.	1980.	“Theories	of	Revolution:	The	Third	Generation.”	
World
Politics
	32	(3):	425–53.
Goldstone,	Jack	A.	2001.	“Towards	a	Fourth	Generation	of	Revolutionary	Theory.”
Annual	Review	of	Political	Science
	4:	139–87.
Good,	Irving	John.	1965.	“Speculations	Concerning	the	First	Ultraintelligent
Machine.”	In	
Advances	in	Computers
,	edited	by	Franz	L.	Alt	and	Morris
Rubinoff,	6:	31–88.	New	York:	Academic	Press.
Good,	Irving	John.	1970.	“Some	Future	Social	Repercussions	of	Computers.”
International	Journal	of	Environmental	Studies
	1	(1–4):	67–79.
Good,	Irving	John.	1976.	“Book	review	of	‘The	Thinking	Computer:	Mind	Inside
Matter’”	In	
International	Journal	of	Man-Machine	Studies
	8:	617–20.
Good,	Irving	John.	1982.	“Ethical	Machines.”	In	
Intelligent	Systems:	Practice	and
Perspective
,	edited	by	J.	E.	Hayes,	Donald	Michie,	and	Y.-H.	Pao,	555–60.
Machine	Intelligence	10.	Chichester:	Ellis	Horwood.
Goodman,	Nelson.	1954.	
Fact,	Fiction,	and	Forecast
.	1st	ed.	London:	Athlone
Press.

	8:	617–20.
Press.
Gott,	J.	R.,	Juric,	M.,	Schlegel,	D.,	Hoyle,	F.,	Vogeley,	M.,	Tegmark,	M.,	Bahcall,
N.,	and	Brinkmann,	J.	2005.	“A	Map	of	the	Universe.”	
Astrophysical	Journal
624	(2):	463–83.
Gottfredson,	Linda	S.	2002.	“G:	Highly	General	and	Highly	Practical.”	In	
The
General	Factor	of	Intelligence:	How	General	Is	It?
,	edited	by	Robert	J.
Sternberg	and	Elena	L.	Grigorenko,	331–80.	Mahwah,	NJ:	Lawrence	Erlbaum.
Gould,	S.	J.	1990.	
Wonderful	Life:	The	Burgess	Shale	and	the	Nature	of	History
.
New	York:	Norton.

Press.
The
.
Graham,	Gordon.	1997.	
The	Shape	of	the	Past:	A	Philosophical	Approach	to
History
Gray,	C.	M.,	and	McCormick,	D.	A.	1996.	“Chattering	Cells:	Superficial	Pyramidal
Neurons	Contributing	to	the	Generation	of	Synchronous	Oscillations	in	the
Visual	Cortex.”	
Science
	274	(5284):	109–13.

Greene,	Kate.	2012.	“Intel’s	Tiny	Wi-Fi	Chip	Could	Have	a	Big	Impact.”	
MIT
Technology	Review
,	September	21.
Guizzo,	Erico.	2010.	“World	Robot	Population	Reaches	8.6	Million.”	
IEEE
Spectrum
,	April	14.
Gunn,	James	E.	1982.	
Isaac	Asimov:	The	Foundations	of	Science	Fiction
.	Science-
Fiction	Writers.	New	York:	Oxford	University	Press.
Haberl,	Helmut,	Erb,	Karl-Heinz,	and	Krausmann,	Fridolin.	2013.	“Global	Human
Appropriation	of	Net	Primary	Production	(HANPP).”	
Encyclopedia	of	Earth
,
September	3.
Haberl,	H.,	Erb,	K.	H.,	Krausmann,	F.,	Gaube,	V.,	Bondeau,	A.,	Plutzar,	C.,
Gingrich,	S.,	Lucht,	W.,	and	Fischer-Kowalski,	M.	2007.	“Quantifying	and
Mapping	the	Human	Appropriation	of	Net	Primary	Production	in	Earth’s
Terrestrial	Ecosystems.”	
Proceedings	of	the	National	Academy	of	Sciences	of	the
United	States	of	America
	104	(31):	12942–7.
Hájek,	Alan.	2009.	“Dutch	Book	Arguments.”	In	Anand,	Pattanaik,	and	Puppe
2009,	173–95.
Hall,	John	Storrs.	2007.	
Beyond	AI:	Creating	the	Conscience	of	the	Machine
.

.
Amherst,	NY:	Prometheus	Books.
Hampson,	R.	E.,	Song,	D.,	Chan,	R.	H.,	Sweatt,	A.	J.,	Riley,	M.	R.,	Gerhardt,	G.
A.,	Shin,	D.	C.,	Marmarelis,	V.	Z.,	Berger,	T.	W.,	and	Deadwyler,	S.	A.	2012.	“A
Nonlinear	Model	for	Hippocampal	Cognitive	Prosthesis:	Memory	Facilitation	by
Hippocampal	Ensemble	Stimulation.”	
IEEE	Transactions	on	Neural	Systems	and
Rehabilitation	Engineering
	20	(2):	184–97.
Hanson,	Robin.	1994.	“If	Uploads	Come	First:	The	Crack	of	a	Future	Dawn.”
Extropy
	6	(2).

.
Extropy
	6	(2).
Hanson,	Robin.	1995.	“Could	Gambling	Save	Science?	Encouraging	an	Honest
Consensus.”	
Social	Epistemology
	9	(1):	3–33.
Hanson,	Robin.	1998a.	“Burning	the	Cosmic	Commons:	Evolutionary	Strategies
for	Interstellar	Colonization.”	Unpublished	manuscript,	July	1.	Retrieved	April
26,	2012.	
http://hanson.gmu.edu/filluniv.pdf
.

Hanson,	Robin.	1998b.	“Economic	Growth	Given	Machine	Intelligence.”
Unpublished	manuscript.	Retrieved	May	15,	2013.	Available	at
http://hanson.gmu.edu/aigrow.pdf
.
Hanson,	Robin.	1998c.	“Long-Term	Growth	as	a	Sequence	of	Exponential	Modes.”
Unpublished	manuscript.	Last	revised	December	2000.	Available	at
http://hanson.gmu.edu/longgrow.pdf
.
Hanson,	Robin.	1998d.	“Must	Early	Life	Be	Easy?	The	Rhythm	of	Major
Evolutionary	Transitions.”	Unpublished	manuscript,	September	23.	Retrieved
August	12,	2012.	Available	at	
http://hanson.gmu.edu/hardstep.pdf
.
Hanson,	Robin.	2000.	“Shall	We	Vote	on	Values,	But	Bet	on	Beliefs?”	Unpublished
manuscript,	September.	Last	revised	October	2007.	Available	at
http://hanson.gmu.edu/futarchy.pdf
.
Hanson,	Robin.	2006.	“Uncommon	Priors	Require	Origin	Disputes.”	
Theory	and
Decision
	61	(4):	319–328.
Hanson,	Robin.	2008.	“Economics	of	the	Singularity.”	
IEEE	Spectrum
	45	(6):	45–
50.
Hanson,	Robin.	2009.	“Tiptoe	or	Dash	to	Future?”	
Overcoming	Bias
	(blog),
December	23.

.
.
Theory	and
Decision
	45	(6):	45–
50.
	(blog),
Hanson,	Robin.	2012.	“Envisioning	the	Economy,	and	Society,	of	Whole	Brain
Emulations.”	Paper	presented	at	the	AGI	Impacts	conference	2012.
Hart,	Oliver.	2008.	“Economica	Coase	Lecture	Reference	Points	and	the	Theory	of
the	Firm.”	
Economica
	75	(299):	404–11.
Hay,	Nicholas	James.	2005.	“Optimal	Agents.”	B.Sc.	thesis,	University	of
Auckland.
Hedberg,	Sara	Reese.	2002.	“Dart:	Revolutionizing	Logistics	Planning.”	
IEEE
Intelligent	Systems
	17	(3):	81–3.
Helliwell,	John,	Layard,	Richard,	and	Sachs,	Jeffrey.	2012.

the	Firm.”	
Economica
Auckland.
IEEE
Helliwell,	John,	Layard,	Richard,	and	Sachs,	Jeffrey.	2012.	
World	Happiness
Report
.	The	Earth	Institute.

Helmstaedter,	M.,	Briggman,	K.	L.,	and	Denk,	W.	2011.	“High-Accuracy	Neurite
Reconstruction	for	High-Throughput	Neuroanatomy.”	
	14
(8):	1081–8.
Heyl,	Jeremy	S.	2005.	“The	Long-Term	Future	of	Space	Travel.”	
Physical	Review
D
	72	(10):	1–4.
Hibbard,	Bill.	2011.	“Measuring	Agent	Intelligence	via	Hierarchies	of
Environments.”	In	
Artificial	General	Intelligence:	4th	International	Conference,
AGI	2011,	Mountain	View,	CA,	USA,	August	3–6,	2011.	Proceedings
,	edited	by
Jürgen	Schmidhuber,	Kristinn	R.	Thórisson,	and	Moshe	Looks,	303–8.	Lecture
Notes	in	Computer	Science	6830.	Berlin:	Springer.
Hinke,	R.	M.,	Hu,	X.,	Stillman,	A.	E.,	Herkle,	H.,	Salmi,	R.,	and	Ugurbil,	K.	1993.
“Functional	Magnetic	Resonance	Imaging	of	Broca’s	Area	During	Internal
Speech.”	
Neuroreport
	4	(6):	675–8.
Hinxton	Group.	2008.	
Consensus	Statement:	Science,	Ethics	and	Policy	Challenges
of	Pluripotent	Stem	Cell-Derived	Gametes
.	Hinxton,	Cambridgeshire,	UK,	April
11.	Available	at

,	edited	by
Speech.”	
11.	Available	at	
http://www.hinxtongroup.org/Consensus_HG08_FINAL.pdf
.
Hoffman,	David	E.	2009.	
The	Dead	Hand:	The	Untold	Story	of	the	Cold	War	Arms
Race	and	Its	Dangerous	Legacy
Hofstadter,	Douglas	R.	(1979)	1999.	
Goödel,	Escher,	Bach:	An	Eternal	Golden
Braid
Holley,	Rose.	2009.	“How	Good	Can	It	Get?	Analysing	and	Improving	OCR
Accuracy	in	Large	Scale	Historic	Newspaper	Digitisation	Programs.”	
D-Lib
Magazine
	15	(3–4).
Horton,	Sue,	Alderman,	Harold,	and	Rivera,	Juan	A.	2008.

.
Braid
D-Lib
Magazine
	15	(3–4).
Horton,	Sue,	Alderman,	Harold,	and	Rivera,	Juan	A.	2008.	
Copenhagen	Consensus
2008	Challenge	Paper:	Hunger	and	Malnutrition
.	Technical	report.	Copenhagen
Consensus	Center,	May	11.
Howson,	Colin,	and	Urbach,	Peter.	1993.	
Scientific	Reasoning:	The	Bayesian
Approach
.	2nd	ed.	Chicago:	Open	Court.
Hsu,	Stephen.	2012.	“Investigating	the	Genetic	Basis	for	Intelligence	and	Other
Quantitative	Traits.”	Lecture	given	at	UC	Davis	Department	of	Physics

Colloquium,	Davis,	CA,	February	13.
Huebner,	Bryce.	2008.	“Do	You	See	What	We	See?	An	Investigation	of	an
Argument	Against	Collective	Representation.”	
Philosophical	Psychology
	21	(1):
91–112.
Huff,	C.	D.,	Xing,	J.,	Rogers,	A.	R.,	Witherspoon,	D.,	and	Jorde,	L.	B.	2010.
“Mobile	Elements	Reveal	Small	Population	Size	in	the	Ancient	Ancestors	of
Homo	Sapiens
.”	
	107	(5):	2147–52.
Huffman,	W.	Cary,	and	Pless,	Vera.	2003.	
Fundamentals	of	Error-Correcting
Codes
Hunt,	Patrick.	2011.	“Late	Roman	Silk:	Smuggling	and	Espionage	in	the	6th
Century	CE.”	
Philolog,	Stanford	University
	(blog),	August	2.
Hutter,	Marcus.	2001.	“Towards	a	Universal	Theory	of	Artificial	Intelligence
Based	on	Algorithmic	Probability	and	Sequential	Decisions.”	In	De	Raedt	and
Flach	2001,	226–38.
Hutter,	Marcus.	2005.	
Universal	Artificial	Intelligencet:	Sequential	Decisions
Based	On	Algorithmic	Probability

Codes
.	Texts	in	Theoretical	Computer	Science.
Berlin:	Springer.
Iliadou,	A.	N.,	Janson,	P.	C.,	and	Cnattingius,	S.	2011.	“Epigenetics	and	Assisted
Reproductive	Technology.”	
Journal	of	Internal	Medicine
	270	(5):	414–20.
Isaksson,	Anders.	2007.	
Productivity	and	Aggregate	Growth:	A	Global	Picture
.
Technical	report	05/2007.	Vienna,	Austria:	UNIDO	(United	Nations	Industrial
Development	Organization)	Research	and	Statistics	Branch.
Jones,	Garret.	2009.	“Artificial	Intelligence	and	Economic	Growth:	A	Few	Finger-

.
Exercises.”	Unpublished	manuscript,	January.	Retrieved	November	5,	2012.
http://mason.gmu.edu/~gjonesb/AIandGrowth
.
Jones,	Vincent	C.	1985.	
Manhattan:	The	Army	and	the	Atomic	Bomb
.	United	States
Army	in	World	War	II.	Washington,	DC:	Center	of	Military	History.
Joyce,	James	M.	1999.	
The	Foundations	of	Causal	Decision	Theory
.	Cambridge

Studies	in	Probability,	Induction	and	Decision	Theory.	New	York:	Cambridge
Judd,	K.	L.,	Schmedders,	K.,	and	Yeltekin,	S.	2012.	“Optimal	Rules	for	Patent
Races.”	
International	Economic	Review
	53	(1):	23–52.
Kalfoglou,	A.,	Suthers,	K.,	Scott,	J.,	and	Hudson,	K.	2004.	
Reproductive	Genetic
Testing:	What	America	Thinks
.	Genetics	and	Public	Policy	Center.
Kamm,	Frances	M.	2007.	
Intricate	Ethics:	Rights,	Responsibilities,	and
Permissible	Harm
.	Oxford	Ethics	Series.	New	York:	Oxford	University	Press.
Kandel,	Eric	R.,	Schwartz,	James	H.,	and	Jessell,	Thomas	M.,	eds.	2000.	
Principles
of	Neural	Science
.	4th	ed.	New	York:	McGraw-Hill.
Kansa,	Eric.	2003.	“Social	Complexity	and	Flamboyant	Display	in	Competition:
More	Thoughts	on	the	Fermi	Paradox.”	Unpublished	manuscript,	archived
version.
Karnofsky,	Holden.	2012.	“Comment	on	‘Reply	to	Holden	on	Tool	AI.’”	
Less
Wrong
	(blog),	August	1.
Kasparov,	Garry.	1996.	“The	Day	That	I	Sensed	a	New	Kind	of	Intelligence.”	
Time
,
March	25,	no.	13.

version.
Less
Wrong
Time
,
Kaufman,	Jeff.	2011.	“Whole	Brain	Emulation	and	Nematodes.”	
Jeff	Kaufman’s
Blog
	(blog),	November	2.
Keim,	G.	A.,	Shazeer,	N.	M.,	Littman,	M.	L.,	Agarwal,	S.,	Cheves,	C.	M.,
Fitzgerald,	J.,	Grosland,	J.,	Jiang,	F.,	Pollard,	S.,	and	Weinmeister,	K.	1999.
“Proverb:	The	Probabilistic	Cruciverbalist.”	In	
Proceedings	of	the	Sixteenth
National	Conference	on	Artificial	Intelligence
,	710–17.	Menlo	Park,	CA:	AAAI
Press.
Kell,	Harrison	J.,	Lubinski,	David,	and	Benbow,	Camilla	P.	2013.	“Who	Rises	to
the	Top?	Early	Indicators.”

Blog
Press.
the	Top?	Early	Indicators.”	
Psychological	Science
	24	(5):	648–59.
Keller,	Wolfgang.	2004.	“International	Technology	Diffusion.”	
Journal	of
Economic	Literature
	42	(3):	752–82.

KGS	Go	Server.	2012.	“KGS	Game	Archives:	Games	of	KGS	player	zen19.”
Retrieved	July	22,	2013.	Available	at	
http://www.gokgs.com/gameArchives.jsp?
user=zen19d&oldAccounts=t&year=2012&month=3
.
Knill,	Emanuel,	Laflamme,	Raymond,	and	Viola,	Lorenza.	2000.	“Theory	of
Quantum	Error	Correction	for	General	Noise.”	
Physical	Review	Letters
	84	(11):
2525–8.
Koch,	K.,	McLean,	J.,	Segev,	R.,	Freed,	M.	A.,	Berry,	M.	J.,	Balasubramanian,	V.,
and	Sterling,	P.	2006.	“How	
Much
	the	Eye	Tells	the	Brain.”	
	16
(14):	1428–34.
Kong,	A.,	Frigge,	M.	L.,	Masson,	G.,	Besenbacher,	S.,	Sulem,	P.,	Magnusson,	G.,
Gudjonsson,	S.	A.,	Sigurdsson,	A.,	
et	al.
	2012.	“Rate	of	De	Novo	Mutations	and
the	Importance	of	Father’s	Age	to	Disease	Risk.”	
Nature
	488:	471–5.
Koomey,	Jonathan	G.	2011.	
Growth	in	Data	Center	Electricity	Use	2005	to	2010
.
Technical	report,	08/01/2011.	Oakland,	CA:	Analytics	Press.
Koubi,	Vally.	1999.	“Military	Technology	Races.”	
International	Organization
	53
(3):	537–65.

	16
et	al.
Nature
.
	53
Koubi,	Vally,	and	Lalman,	David.	2007.	“Distribution	of	Power	and	Military
R&D.”	
Journal	of	Theoretical	Politics
	19	(2):	133–52.
Koza,	J.	R.,	Keane,	M.	A.,	Streeter,	M.	J.,	Mydlowec,	W.,	Yu,	J.,	and	Lanza,	G.
2003.	
Genetic	Programming	IV:	Routine	Human-Competitive	Machine
.	2nd	ed.	Genetic	Programming.	Norwell,	MA:	Kluwer	Academic.
Kremer,	Michael.	1993.	“Population	Growth	and	Technological	Change:	One
Million	B.C.	to	1990.”	
Quarterly	Journal	of	Economics
	108	(3):	681–716.

R&D.”	
2003.	
Kruel,	Alexander.	2011.	“Interview	Series	on	Risks	from	AI.”	
Less	Wrong	Wiki
(blog).	Retrieved	Oct	26,	2013.	Available	at
http://wiki.lesswrong.com/wiki/Interview_series_on_risks_from_AI
.
Kruel,	Alexander.	2012.	“Q&A	with	Experts	on	Risks	From	AI	#2.”	
Less	Wrong
(blog),	January	9.
Krusienski,	D.	J.,	and	Shih,	J.	J.	2011.	“Control	of	a	Visual	Keyboard	Using	an

Electrocorticographic	Brain–Computer	Interface.”	
Neurorehabilitation	and
Neural	Repair
	25	(4):	323–31.
Kuhn,	Thomas	S.	1962.	
The	Structure	of	Scientific	Revolutions
.	1st	ed.	Chicago:
University	of	Chicago	Press.
Kuipers,	Benjamin.	2012.	“An	Existing,	Ecologically-Successful	Genus	of
Collectively	Intelligent	Artificial	Creatures.”	Paper	presented	at	the	4th
International	Conference,	ICCCI	2012,	Ho	Chi	Minh	City,	Vietnam,	November
28–30.
Kurzweil,	Ray.	2001.	“Response	to	Stephen	Hawking.”	Kurzweil	Accelerating
Intelligence.	September	5.	Retrieved	December	31,	2012.	Available	at
http://www.kurzweilai.net/response-to-stephen-hawking
.
Kurzweil,	Ray.	2005.	
The	Singularity	Is	Near:	When	Humans	Transcend	Biology
.
New	York:	Viking.
Laffont,	Jean-Jacques,	and	Martimort,	David.	2002.	
The	Theory	of	Incentives:	The
Principal-Agent	Model
.	Princeton,	NJ:	Princeton	University	Press.
Lancet,	The
.	2008.	“Iodine	Deficiency—Way	to	Go	Yet.”	
The	Lancet
	372	(9633):
88.

.
.
The	Lancet
88.
Landauer,	Thomas	K.	1986.	“How	Much	Do	People	Remember?	Some	Estimates
of	the	Quantity	of	Learned	Information	in	Long-Term	Memory.”	
Cognitive
Science
	10	(4):	477–93.
Lebedev,	Anastasiya.	2004.	“The	Man	Who	Saved	the	World	Finally	Recognized.”
MosNews
,	May	21.
Lebedev,	M.	A.,	and	Nicolelis,	M.	A.	2006.	“Brain–Machine	Interfaces:	Past,
Present	and	Future.”	
Trends	in	Neuroscience
	29	(9):	536–46.
Legg,	Shane.	2008.	“Machine	Super	Intelligence.”	PhD	diss.,	University	of
Lugano.
Leigh,	E.	G.,	Jr.	2010.	“The	Group	Selection	Controversy.”	
Journal	of

Cognitive
Science
MosNews
,	May	21.
Lugano.
Journal	of
Evolutionary	Biology
	23(1):	6–19.

Lenat,	Douglas	B.	1982.	“Learning	Program	Helps	Win	National	Fleet	Wargame
Tournament.”	
SIGART	Newsletter
	79:	16–17.
Lenat,	Douglas	B.	1983.	“EURISKO:	A	Program	that	Learns	New	Heuristics	and
Domain	Concepts.”	
	21	(1–2):	61–98.
Lenman,	James.	2000.	“Consequentialism	and	Cluelessness.”	
Philosophy	&	Public
Affairs
	29	(4):	342–70.
Lerner,	Josh.	1997.	“An	Empirical	Exploration	of	a	Technology	Race.”	
RAND
	28	(2):	228–47.
Leslie,	John.	1996.	
The	End	of	the	World:	The	Science	and	Ethics	of	Human
Extinction
.	London:	Routledge.
Lewis,	David.	1988.	“Desire	as	Belief.”	
Mind:	A	Quarterly	Review	of	Philosophy
97	(387):	323–32.
Li,	Ming,	and	Vitaányi,	Paul	M.	B.	2008.	
An	Introduction	to	Kolmogorov
Complexity	and	Its	Applications
.	Texts	in	Computer	Science.	New	York:
Springer.
Lin,	Thomas,	Mausam,	and	Etzioni,	Oren.	2012.	“Entity	Linking	at	Web	Scale.”	In
Proceedings	of	the	Joint	Workshop	on	Automatic	Knowledge	Base	Construction
and	Web-scale	Knowledge	Extraction

Springer.
and	Web-scale	Knowledge	Extraction	
(AKBC-WEKEX	’12)
,	edited	by	James
Fan,	Raphael	Hoffman,	Aditya	Kalyanpur,	Sebastian	Riedel,	Fabian	Suchanek,
and	Partha	Pratim	Talukdar,	84–88.	Madison,	WI:	Omnipress.
Lloyd,	Seth.	2000.	“Ultimate	Physical	Limits	to	Computation.”	
Nature
	406	(6799):
1047–54.
Louis	Harris	&	Associates.	1969.	“Science,	Sex,	and	Morality	Survey,	study	no.
1927.”	
Life	Magazine
	(New	York)	4.
Lynch,	Michael.	2010.	“Rate,	Molecular	Spectrum,	and	Consequences	of	Human
Mutation.”	

Nature
1047–54.
1927.”	
	107	(3):	961–8.
Lyons,	Mark	K.	2011.	“Deep	Brain	Stimulation:	Current	and	Future	Clinical
Applications.”	
Mayo	Clinic	Proceedings
	86	(7):	662–72.

MacAskill,	William.	2010.	“Moral	Uncertainty	and	Intertheoretic	Comparisons	of
Value.”	BPhil	thesis,	University	of	Oxford.
McCarthy,	John.	2007.	“From	Here	to	Human-Level	AI.”	
171	(18):	1174–82.
McCorduck,	Pamela.	1979.	
Machines	Who	Think:	A	Personal	Inquiry	into	the
History	and	Prospects	of	Artificial	Intelligence
.	San	Francisco:	W.	H.	Freeman.
Mack,	C.	A.	2011.	“Fifty	Years	of	Moore’s	Law.”	
IEEE	Transactions	on
Semiconductor	Manufacturing
	24	(2):	202–7.
MacKay,	David	J.	C.	2003.	
Information	Theory,	Inference,	and	Learning
Algorithms
McLean,	George,	and	Stewart,	Brian.	1979.	“Norad	False	Alarm	Causes	Uproar.”
The	National
.	Aired	November	10.	Ottawa,	ON:	CBC,	2012.	News	Broadcast.
Maddison,	Angus.	1999.	“Economic	Progress:	The	Last	Half	Century	in	Historical
Perspective.”	In	
Facts	and	Fancies	of	Human	Development:	Annual	Symposium
and	Cunningham	Lecture,	1999
,	edited	by	Ian	Castles.	Occasional	Paper	Series,

1/2000.	Academy	of	the	Social	Sciences	in	Australia.
Maddison,	Angus.	2001.	
The	World	Economy:	A	Millennial	Perspective
.
Development	Centre	Studies.	Paris:	Development	Centre	of	the	Organisation	for
Economic	Cooperation	/	Development.
Maddison,	Angus.	2005.	
Growth	and	Interaction	in	the	World	Economy:	The	Roots
of	Modernity
.	Washington,	DC:	AEI	Press.
Maddison,	Angus.	2007.	
Contours	of	the	World	Economy,	1–2030	AD:	Essays	in
Macro-Economic	History

.
Maddison,	Angus.	2010.	“Statistics	of	World	Population,	GDP	and	Per	Capita	GDP
1–2008	AD.”	Retrieved	October	26,	2013.	Available	at
http://www.ggdc.net/maddison/Historical_Statistics/vertical-file_02-2010.xls
.
Mai,	Q.,	Yu,	Y.,	Li,	T.,	Wang,	L.,	Chen,	M.	J.,	Huang,	S.	Z.,	Zhou,	C.,	and	Zhou,	Q.
2007.	“Derivation	of	Human	Embryonic	Stem	Cell	Lines	from	Parthenogenetic
Blastocysts.”	
Cell	Research
	17	(12):	1008–19.

Mak,	J.	N.,	and	Wolpaw,	J.	R.	2009.	“Clinical	Applications	of	Brain-Computer
Interfaces:	Current	State	and	Future	Prospects.”	
IEEE	Reviews	in	Biomedical
	2:	187–99.
Mankiw,	N.	Gregory.	2009.	
Macroeconomics
.	7th	ed.	New	York,	NY:	Worth.
Mardis,	Elaine	R.	2011.	“A	Decade’s	Perspective	on	DNA	Sequencing
Technology.”	
Nature
	470	(7333):	198–203.
Markoff,	John.	2011.	“Computer	Wins	on	‘Jeopardy!’:	Trivial,	It’s	Not.”	
New	York
Times
,	February	16.
Markram,	Henry.	2006.	“The	Blue	Brain	Project.”	
	7
(2):	153–160.
Mason,	Heather.	2003.	“Gallup	Brain:	The	Birth	of	In	Vitro	Fertilization.”	
Gallup
,
August	5.
Menzel,	Randolf,	and	Giurfa,	Martin.	2001.	“Cognitive	Architecture	of	a	Mini-
Brain:	The	Honeybee.”	
Trends	in	Cognitive	Sciences
	5	(2):	62–71.
Metzinger,	Thomas.	2003.	
Being	No	One:	The	Self-Model	Theory	of	Subjectivity
.
Cambridge,	MA:	MIT	Press.
Mijic,	Roko.	2010.	“Bootstrapping	Safe	AGI	Goal	Systems.”	Paper	presented	at

	7
Gallup
,
August	5.
.
the	Roadmaps	to	AGI	and	the	Future	of	AGI	Workshop,	Lugano,	Switzerland,
March	8.
Mike,	Mike.	2013.	“Face	of	Tomorrow.”	Retrieved	June	30,	2012.	Available	at
http://faceoftomorrow.org
.
Milgrom,	Paul,	and	Roberts,	John.	1990.	“Bargaining	Costs,	Influence	Costs,	and
the	Organization	of	Economic	Activity.”	In	
Perspectives	on	Positive	Political
Economy
,	edited	by	James	E.	Alt	and	Kenneth	A.	Shepsle,	57–89.	New	York:
Cambridge	University	Press.
Miller,	George	A.	1956.	“The	Magical	Number	Seven,	Plus	or	Minus	Two:	Some

March	8.
.
Economy
Limits	on	Our	Capacity	for	Processing	Information.”	
Psychological	Review
	63
(2):	81–97.

Miller,	Geoffrey.	2000.	
The	Mating	Mind:	How	Sexual	Choice	Shaped	the
Evolution	of	Human	Nature
Miller,	James	D.	2012.	
Singularity	Rising:	Surviving	and	Thriving	in	a	Smarter,
Richer,	and	More	Dangerous	World
.	Dallas,	TX:	BenBella	Books.
Minsky,	Marvin.	1967.	
Computation:	Finite	and	Infinite	Machines
.	Englewood
Cliffs,	NJ:	Prentice-Hall.
Minsky,	Marvin,	ed.	1968.	
.	Cambridge,	MA:
MIT	Press.
Minsky,	Marvin.	1984.	“Afterword	to	Vernor	Vinge’s	novel,	‘True	Names.’”
Unpublished	manuscript,	October	1.	Retrieved	December	31,	2012.	Available	at
http://web.media.mit.edu/~minsky/papers/TrueNames.Afterword.html
.
Minsky,	Marvin.	2006.	
The	Emotion	Machine:	Commonsense	Thinking,	Artificial
Intelligence,	and	the	Future	of	the	Human	Mind
.	New	York:	Simon	&	Schuster.
Minsky,	Marvin,	and	Papert,	Seymour.	1969.	
Perceptrons:	An	Introduction	to
Computational	Geometry
.	1st	ed.	Cambridge,	MA:	MIT	Press.
Moore,	Andrew.	2011.	“Hedonism.”	In

.
Moore,	Andrew.	2011.	“Hedonism.”	In	
The	Stanford	Encyclopedia	of	Philosophy
,
Winter	2011,	edited	by	Edward	N.	Zalta.	Stanford,	CA:	Stanford	University.
Moravec,	Hans	P.	1976.	“The	Role	of	Raw	Power	in	Intelligence.”	Unpublished
manuscript,	May	12.	Retrieved	August	12,	2012.	Available	at
http://www.frc.ri.cmu.edu/users/hpm/project.archive/general.articles/l975/Raw.Power.html
Moravec,	Hans	P.	1980.	“Obstacle	Avoidance	and	Navigation	in	the	Real	World	by
a	Seeing	Robot	Rover.”	PhD	diss.,	Stanford	University.
Moravec,	Hans	P.	1988.	
Mind	Children:	The	Future	of	Robot	and	Human

,
.	Cambridge,	MA:	Harvard	University	Press.
Moravec,	Hans	P.	1998.	“When	Will	Computer	Hardware	Match	the	Human
Brain?”	
	1.
Moravec,	Hans	P.	1999.	“Rise	of	the	Robots.”	
,	December,
124–35.

Muehlhauser,	Luke,	and	Helm,	Louie.	2012.	“The	Singularity	and	Machine
Ethics.”	In	
Singularity	Hypotheses:	A	Scientific	and	Philosophical	Assessment
,
edited	by	Amnon	Eden,	Johnny	Søraker,	James	H.	Moor,	and	Eric	Steinhart.	The
Frontiers	Collection.	Berlin:	Springer.
Muehlhauser,	Luke,	and	Salamon,	Anna.	2012.	“Intelligence	Explosion:	Evidence
and	Import.”	In	
Singularity	Hypotheses:	A	Scientific	and	Philosophical
Assessment
,	edited	by	Amnon	Eden,	Johnny	Søraker,	James	H.	Moor,	and	Eric
Steinhart.	The	Frontiers	Collection.	Berlin:	Springer.
Müller,	Vincent	C.,	and	Bostrom,	Nick.	Forthcoming.	“Future	Progress	in	Artificial
Intelligence:	A	Poll	Among	Experts.”	In	“Impacts	and	Risks	of	Artificial
General	Intelligence,”	edited	by	Vincent	C.	Müller,	special	issue,	
Journal	of
Experimental	and	Theoretical	Artificial	Intelligence
.
Murphy,	Kevin	P.	2012.	
Machine	Learning:	A	Probabilistic	Perspective
.	Adaptive
Computation	and	Machine	Learning.	Cambridge,	MA:	MIT	Press.

Journal	of
.
.	Adaptive
Nachman,	Michael	W.,	and	Crowell,	Susan	L.	2000.	“Estimate	of	the	Mutation
Rate	per	Nucleotide	in	Humans.”	
Genetics
	156	(1):	297–304.
Nagy,	Z.	P.,	and	Chang,	C.	C.	2007.	“Artificial	Gametes.”	
Theriogenology
	67	(1):
99–104.
Nagy,	Z.	P.,	Kerkis,	I.,	and	Chang,	C.	C.	2008.	“Development	of	Artificial
Gametes.”	
Reproductive	BioMedicine	Online
	16	(4):	539–44.
NASA.	2013.	“International	Space	Station:	Facts	and	Figures.”	Available	at
http://www.nasa.gov/worldbook/intspacestation_worldbook.html
.
Newborn,	Monty.	2011.

Genetics
	67	(1):
99–104.
Gametes.”	
.
Newborn,	Monty.	2011.	
Beyond	Deep	Blue:	Chess	in	the	Stratosphere
.	New	York:
Springer.
Newell,	Allen,	Shaw,	J.	C.,	and	Simon,	Herbert	A.	1958.	“Chess-Playing	Programs
and	the	Problem	of	Complexity.”	
IBM	Journal	of	Research	and	Development
	2
(4):	320–35.
Newell,	Allen,	Shaw,	J.	C.,	and	Simon,	Herbert	A.	1959.	“Report	on	a	General
Problem-Solving	Program:	Proceedings	of	the	International	Conference	on
Information	Processing.”	In	
Information	Processing
,	256–64.	Paris:	UNESCO.

Nicolelis,	Miguel	A.	L.,	and	Lebedev,	Mikhail	A.	2009.	“Principles	of	Neural
Ensemble	Physiology	Underlying	the	Operation	of	Brain–Machine	Interfaces.”
	10	(7):	530–40.
Nilsson,	Nils	J.	1984.	
Shakey	the	Robot
,	Technical	Note	323.	Menlo	Park,	CA:	AI
Center,	SRI	International,	April.
Nilsson,	Nils	J.	2009.	
The	Quest	for	Artificial	Intelligence:	A	History	of	Ideas	and
Achievements
Nisbett,	R.	E.,	Aronson,	J.,	Blair,	C.,	Dickens,	W.,	Flynn,	J.,	Halpern,	D.	F.,	and
Turkheimer,	E.	2012.	“Intelligence:	New	Findings	and	Theoretical
Developments.”	
American	Psychologist
	67	(2):	130–59.
Niven,	Larry.	1973.	“The	Defenseless	Dead.”	In	
Ten	Tomorrows
,	edited	by	Roger
Elwood,	91–142.	New	York:	Fawcett.
Nordhaus,	William	D.	2007.	“Two	Centuries	of	Productivity	Growth	in
Computing.”	
Journal	of	Economic	History
	67	(1):	128–59.
Norton,	John	D.	2011.	“Waiting	for	Landauer.”	
Studies	in	History	and	Philosophy

of	Science	Part	B:	Studies	in	History	and	Philosophy	of	Modern	Physics
	42	(3):
184–98.
Olds,	James,	and	Milner,	Peter.	1954.	“Positive	Reinforcement	Produced	by
Electrical	Stimulation	of	Septal	Area	and	Other	Regions	of	Rat	Brain.”	
Journal
of	Comparative	and	Physiological	Psychology
	47	(6):	419–27.
Olum,	Ken	D.	2002.	“The	Doomsday	Argument	and	the	Number	of	Possible
Observers.”	
	52	(207):	164–84.
Omohundro,	Stephen	M.	2007.	“The	Nature	of	Self-Improving	Artificial
Intelligence.”	Paper	presented	at	Singularity	Summit	2007,	San	Francisco,	CA,

42	(3):
184–98.
Journal
September	8–9.
Omohundro,	Stephen	M.	2008.	“The	Basic	AI	Drives.”	In	
Intelligence	2008:	Proceedings	of	the	First	AGI	Conference
,	edited	by	Pei	Wang,
Ben	Goertzel,	and	Stan	Franklin,	483–92.	Frontiers	in	Artificial	Intelligence	and
Applications	171.	Amsterdam:	IOS.

Omohundro,	Stephen	M.	2012.	“Rational	Artificial	Intelligence	for	the	Greater
Good.”	In	
,
O’Neill,	Gerard	K.	1974.	“The	Colonization	of	Space.”	
Physics	Today
	27	(9):	32–
40.
Oshima,	Hideki,	and	Katayama,	Yoichi.	2010.	“Neuroethics	of	Deep	Brain
Stimulation	for	Mental	Disorders:	Brain	Stimulation	Reward	in	Humans.”
Neurologia	medico-chirurgica
	50	(9):	845–52.
Parfit,	Derek.	1986.	
Reasons	and	Persons
Parfit,	Derek.	2011.	
On	What	Matters
.	2	vols.	The	Berkeley	Tanner	Lectures.	New
York:	Oxford	University	Press.
Parrington,	Alan	J.	1997.	“Mutually	Assured	Destruction	Revisited.”	
Airpower
Journal
	11	(4).
Pasqualotto,	Emanuele,	Federici,	Stefano,	and	Belardinelli,	Marta	Olivetti.	2012.
“Toward	Functioning	and	Usable	Brain–Computer	Interfaces	(BCIs):	A
Literature	Review.”

Airpower
Journal
	11	(4).
Literature	Review.”	
Disability	and	Rehabilitation:	Assistive	Technology
	7	(2):
89–103.
Pearl,	Judea.	2009.	
Causality:	Models,	Reasoning,	and	Inference
.	2nd	ed.	New
York:	Cambridge	University	Press.
Perlmutter,	J.	S.,	and	Mink,	J.	W.	2006.	“Deep	Brain	Stimulation.”	
Annual	Review
of	Neuroscience
	29:	229–57.
Pinker,	Steven.	2011.	
The	Better	Angels	of	Our	Nature:	Why	Violence	Has
Declined
.	New	York:	Viking.
Plomin,	R.,	Haworth,	C.	M.,	Meaburn,	E.	L.,	Price,	T.	S.,	Wellcome	Trust	Case

	7	(2):
89–103.
Declined
Control	Consortium	2,	and	Davis,	O.	S.	2013.	“Common	DNA	Markers	Can
Account	for	More	than	Half	of	the	Genetic	Influence	on	Cognitive	Abilities.”
	24	(2):	562–8.
Popper,	Nathaniel.	2012.	“Flood	of	Errant	Trades	Is	a	Black	Eye	for	Wall	Street.”

,	August	1.
Pourret,	Olivier,	Naim,	Patrick,	and	Marcot,	Bruce,	eds.	2008.	
Bayesian	Networks:
A	Practical	Guide	to	Applications
.	Chichester,	West	Sussex,	UK:	Wiley.
Powell,	A.,	Shennan,	S.,	and	Thomas,	M.	G.	2009.	“Late	Pleistocene	Demography
and	the	Appearance	of	Modern	Human	Behavior.”	
Science
	324	(5932):	1298–
1301.
Price,	Huw.	1991.	“Agency	and	Probabilistic	Causality.”	
British	Journal	for	the
Philosophy	of	Science
	42	(2):	157–76.
Qian,	M.,	Wang,	D.,	Watkins,	W.	E.,	Gebski,	V.,	Yan,	Y.	Q.,	Li,	M.,	and	Chen,	Z.	P.
2005.	“The	Effects	of	Iodine	on	Intelligence	in	Children:	A	Meta-Analysis	of
Studies	Conducted	in	China.”	
Asia	Pacific	Journal	of	Clinical	Nutrition
	14	(1):
32–42.
Quine,	Willard	Van	Orman,	and	Ullian,	Joseph	Silbert.	1978.	
The	Web	of	Belief
,	ed.
Richard	Malin	Ohmann,	vol.	2.	New	York:	Random	House.
Railton,	Peter.	1986.	“Facts	and	Values.”	
Philosophical	Topics
	14	(2):	5–31.
Rajab,	Moheeb	Abu,	Zarfoss,	Jay,	Monrose,	Fabian,	and	Terzis,	Andreas.	2006.	“A

	14	(1):
32–42.
,	ed.
Multifaceted	Approach	to	Understanding	the	Botnet	Phenomenon.”	In
Proceedings	of	the	6th	ACM	SIGCOMM	Conference	on	Internet	Measurement
,
41–52.	New	York:	ACM.
Rawls,	John.	1971.	
A	Theory	of	Justice
.	Cambridge,	MA:	Belknap.
Read,	J.	I.,	and	Trentham,	Neil.	2005.	“The	Baryonic	Mass	Function	of	Galaxies.”
Philosophical	Transactions	of	the	Royal	Society	A:	Mathematical,	Physical	and
Engineering	Sciences
	363	(1837):	2693–710.
Repantis,	D.,	Schlattmann,	P.,	Laisney,	O.,	and	Heuser,	I.	2010.	“Modafinil	and

,
Methylphenidate	for	Neuroenhancement	in	Healthy	Individuals:	A	Systematic
Review.”	
Pharmacological	Research
	62	(3):	187–206.
Rhodes,	Richard.	1986.	
The	Making	of	the	Atomic	Bomb
.	New	York:	Simon	&
Schuster.

Rhodes,	Richard.	2008.	
Arsenals	of	Folly:	The	Making	of	the	Nuclear	Arms	Race
.
New	York:	Vintage.
Rietveld,	Cornelius	A.,	Medland,	Sarah	E.,	Derringer,	Jaime,	Yang,	Jian,	Esko,
Tonu,	Martin,	Nicolas	W.,	Westra,	Harm-Jan,	Shakhbazov,	Konstantin,
Abdellaoui,	Abdel,	
et	al.
	2013.	“GWAS	of	126,559	Individuals	Identifies
Genetic	Variants	Associated	with	Educational	Attainment.”	
Science
	340	(6139):
1467–71.
Ring,	Mark,	and	Orseau,	Laurent.	2011.	“Delusion,	Survival,	and	Intelligent
Agents.”	In	
Artificial	General	Intelligence:	4th	International	Conference,	AGI
2011,	Mountain	View,	CA,	USA,	August	3–6,	2011
.	
,	edited	by
Jürgen	Schmidhuber,	Kristinn	R.	Thórisson,	and	Moshe	Looks,	11–20.	Lecture
Ritchie,	Graeme,	Manurung,	Ruli,	and	Waller,	Annalu.	2007.	“A	Practical
Application	of	Computational	Humour.”	In	
Proceedings	of	the	4th	International
Joint	Workshop	on	Computational	Creativity
,	edited	by	Amilcar	Cardoso	and

.	
,	edited	by
Geraint	A.	Wiggins,	91–8.	London:	Goldsmiths,	University	of	London.
Roache,	Rebecca.	2008.	“Ethics,	Speculation,	and	Values.”	
NanoEthics
	2	(3):	317–
27.
Robles,	J.	A.,	Lineweaver,	C.	H.,	Grether,	D.,	Flynn,	C.,	Egan,	C.	A.,	Pracy,	M.	B.,
Holmberg,	J.,	and	Gardner,	E.	2008.	“A	Comprehensive	Comparison	of	the	Sun
to	Other	Stars:	Searching	for	Self-Selection	Effects.”	
	684
(1):	691–706.
Roe,	Anne.	1953.	
The	Making	of	a	Scientist
.	New	York:	Dodd,	Mead.

	2	(3):	317–
27.
	684
Roy,	Deb.	2012.	“About.”	Retrieved	October	14.	Available	at
http://web.media.mit.edu/~dkroy/
.
Rubin,	Jonathan,	and	Watson,	Ian.	2011.	“Computer	Poker:	A	Review.”	
	175	(5–6):	958–87.
Rumelhart,	D.	E.,	Hinton,	G.	E.,	and	Williams,	R.	J.	1986.	“Learning
Representations	by	Back-Propagating	Errors.”	
Nature
	323	(6088):	533–6.
Russell,	Bertrand.	1986.	“The	Philosophy	of	Logical	Atomism.”	In	
The	Philosophy

of	Logical	Atomism	and	Other	Essays	1914–1919
,	edited	by	John	G.	Slater,	8:
157–244.	The	Collected	Papers	of	Bertrand	Russell.	Boston:	Allen	&	Unwin.
Russell,	Bertrand,	and	Griffin,	Nicholas.	2001.	
The	Selected	Letters	of	Bertrand
Russell:	The	Public	Years,	1914–1970
Russell,	Stuart	J.,	and	Norvig,	Peter.	2010.	
Artificial	Intelligence:	A	Modern
Approach
.	3rd	ed.	Upper	Saddle	River,	NJ:	Prentice-Hall.
Sabrosky,	Curtis	W.	1952.	“How	Many	Insects	Are	There?”	In	
Insects
,	edited	by
United	States	Department	of	Agriculture,	1–7.	Yearbook	of	Agriculture.
Washington,	DC:	United	States	Government	Printing	Office.
Salamon,	Anna.	2009.	“When	Software	Goes	Mental:	Why	Artificial	Minds	Mean
Fast	Endogenous	Growth.”	Working	Paper,	December	27.
Salem,	D.	J.,	and	Rowan,	A.	N.	2001.	
The	State	of	the	Animals:	2001
.	Public
Policy	Series.	Washington,	DC:	Humane	Society	Press.
Salverda,	W.,	Nolan,	B.,	and	Smeeding,	T.	M.	2009.	
The	Oxford	Handbook	of
Economic	Inequality

Insects
,	edited	by
.	Public
.	Oxford:	Oxford	University	Press.
Samuel,	A.	L.	1959.	“Some	Studies	in	Machine	Learning	Using	the	Game	of
Checkers.”	
	3	(3):	210–19.
Sandberg,	Anders.	1999.	“The	Physics	of	Information	Processing	Superobjects:
Daily	Life	Among	the	Jupiter	Brains.”	
	5.
Sandberg,	Anders.	2010.	“An	Overview	of	Models	of	Technological	Singularity.”
Paper	presented	at	the	Roadmaps	to	AGI	and	the	Future	of	AGI	Workshop,
Lugano,	Switzerland,	March	8.

	5.
Sandberg,	Anders.	2013.	“Feasibility	of	Whole	Brain	Emulation.”	In	
Philosophy
and	Theory	of	Artificial	Intelligence
,	edited	by	Vincent	C.	Müller,	5:	251–64.
Studies	in	Applied	Philosophy,	Epistemology	and	Rational	Ethics.	New	York:
Springer.
Sandberg,	Anders,	and	Bostrom,	Nick.	2006.	“Converging	Cognitive
Enhancements.”	
Annals	of	the	New	York	Academy	of	Sciences
	1093:	201–27.

Sandberg,	Anders,	and	Bostrom,	Nick.	2008.	
Whole	Brain	Emulation:	A	Roadmap
.
Technical	Report	2008-3.	Future	of	Humanity	Institute,	University	of	Oxford.
Sandberg,	Anders,	and	Bostrom,	Nick.	2011.	
Machine	Intelligence	Survey
.
Technical	Report	2011-1.	Future	of	Humanity	Institute,	University	of	Oxford.
Sandberg,	Anders,	and	Savulescu,	Julian.	2011.	“The	Social	and	Economic	Impacts
of	Cognitive	Enhancement.”	In	
,	edited	by	Julian
Savulescu,	Ruud	ter	Meulen,	and	Guy	Kahane,	92–112.	Malden,	MA:	Wiley-
Blackwell.
Schaeffer,	Jonathan.	1997.	
One	Jump	Ahead:	Challenging	Human	Supremacy	in
Checkers
.	New	York:	Springer.
Schaeffer,	J.,	Burch,	N.,	Bjornsson,	Y.,	Kishimoto,	A.,	Muller,	M.,	Lake,	R.,	Lu,	P.,
and	Sutphen,	S.	2007.	“Checkers	Is	Solved.”	
Science
	317	(5844):	1518–22.
Schalk,	Gerwin.	2008.	“Brain–Computer	Symbiosis.”	
Journal	of	Neural
	5	(1):	P1–P15.
Schelling,	Thomas	C.	1980.	
The	Strategy	of	Conflict
.	2nd	ed.	Cambridge,	MA:
Harvard	University	Press.

Checkers
Science
Schultz,	T.	R.	2000.	“In	Search	of	Ant	Ancestors.”	
Academy	of	Sciences	of	the	United	States	of	America
	97	(26):	14028–9.
Schultz,	W.,	Dayan,	P.,	and	Montague,	P.	R.	1997.	“A	Neural	Substrate	of
Prediction	and	Reward.”	
Science
	275	(5306):	1593–9.
Schwartz,	Jacob	T.	1987.	“Limits	of	Artificial	Intelligence.”	In	
Encyclopedia	of
,	edited	by	Stuart	C.	Shapiro	and	David	Eckroth,	1:	488–
503.	New	York:	Wiley.
Schwitzgebel,	Eric.	2013.	“If	Materialism	is	True,	the	United	States	is	Probably

Science
Conscious.”	Working	Paper,	February	8.
Sen,	Amartya,	and	Williams,	Bernard,	eds.	1982.	
Utilitarianism	and	Beyond
.	New
Shanahan,	Murray.	2010.	
Embodiment	and	the	Inner	Life:	Cognition	and

Consciousness	in	the	Space	of	Possible	Minds
Press.
Shannon,	Robert	V.	2012.	“Advances	in	Auditory	Prostheses.”	
Current	Opinion	in
Neurology
	25	(1):	61–6.
Shapiro,	Stuart	C.	1992.	“Artificial	Intelligence.”	In	
Encyclopedia	of	Artificial
,	2nd	ed.,	1:	54–7.	New	York:	Wiley.
Sheppard,	Brian.	2002.	“World-Championship-Caliber	Scrabble.”	
	134	(1–2):	241–75.
Shoemaker,	Sydney.	1969.	“Time	Without	Change.”	
Journal	of	Philosophy
	66	(12):
363–81.
Shulman,	Carl.	2010a.	
Omohundro’s	“Basic	AI	Drives”	and	Catastrophic	Risks
.
San	Francisco,	CA:	Machine	Intelligence	Research	Institute.
Shulman,	Carl.	2010b.	
Whole	Brain	Emulation	and	the	Evolution	of
Superorganisms
.	San	Francisco,	CA:	Machine	Intelligence	Research	Institute.
Shulman,	Carl.	2012.	“Could	We	Use	Untrustworthy	Human	Brain	Emulations	to
Make	Trustworthy	Ones?”	Paper	presented	at	the	AGI	Impacts	conference	2012.

	66	(12):
363–81.
.
Shulman,	Carl,	and	Bostrom,	Nick.	2012.	“How	Hard	is	Artificial	Intelligence?
Evolutionary	Arguments	and	Selection	Effects.”	
Journal	of	Consciousness
Studies
	19	(7–8):	103–30.
Shulman,	Carl,	and	Bostrom,	Nick.	2014.	“Embryo	Selection	for	Cognitive
Enhancement:	Curiosity	or	Game-Changer?”	
Global	Policy
	5	(1):	85–92.
Shulman,	Carl,	Jonsson,	Henrik,	and	Tarleton,	Nick.	2009.	“Which
Consequentialism?	Machine	Ethics	and	Moral	Divergence.”	In	
AP-CAP	2009:
The	Fifth	Asia-Pacific	Computing	and	Philosophy	Conference,	October	1st-2nd,

Studies
University	of	Tokyo,	Japan
.	
,	edited	by	Carson	Reynolds	and	Alvaro
Cassinelli,	23–25.	AP-CAP	2009.
Sidgwick,	Henry,	and	Jones,	Emily	Elizabeth	Constance.	2010.	
The	Methods	of
Ethics
.	Charleston,	SC:	Nabu	Press.

Silver,	Albert.	2006.	“How	Strong	Is	GNU	Backgammon?”	Backgammon	Galore!
September	16.	Retrieved	October	26,	2013.	Available	at
http://www.bkgm.com/gnu/AllAboutGNU.html#how_strong_is_gnu
.
Simeral,	J.	D.,	Kim,	S.	P.,	Black,	M.	J.,	Donoghue,	J.	P.,	and	Hochberg,	L.	R.	2011.
“Neural	Control	of	Cursor	Trajectory	and	Click	by	a	Human	with	Tetraplegia
1000	Days	after	Implant	of	an	Intracortical	Microelectrode	Array.”	
Journal	of
Neural	Engineering
	8	(2):	025027.
Simester,	Duncan,	and	Knez,	Marc.	2002.	“Direct	and	Indirect	Bargaining	Costs
and	the	Scope	of	the	Firm.”	
Journal	of	Business
	75	(2):	283–304.
Simon,	Herbert	Alexander.	1965.	
The	Shape	of	Automation	for	Men	and
Management
.	New	York:	Harper	&	Row.
Sinhababu,	Neil.	2009.	“The	Humean	Theory	of	Motivation	Reformulated	and
Defended.”	
Philosophical	Review
	118	(4):	465–500.
Slagle,	James	R.	1963.	“A	Heuristic	Program	That	Solves	Symbolic	Integration
Problems	in	Freshman	Calculus.”	
Journal	of	the	ACM
	10	(4):	507–20.

Smeding,	H.	M.,	Speelman,	J.	D.,	Koning-Haanstra,	M.,	Schuurman,	P.	R.,	Nijssen,
P.,	van	Laar,	T.,	and	Schmand,	B.	2006.	“Neuropsychological	Effects	of	Bilateral
STN	Stimulation	in	Parkinson	Disease:	A	Controlled	Study.”	
Neurology
	66	(12):
1830–6.
Smith,	Michael.	1987.	“The	Humean	Theory	of	Motivation.”	
Mind:	A	Quarterly
Review	of	Philosophy
	96	(381):	36–61.
Smith,	Michael,	Lewis,	David,	and	Johnston,	Mark.	1989.	“Dispositional	Theories
of	Value.”	
Proceedings	of	the	Aristotelian	Society
	63:	89–174.
Sparrow,	Robert.	2013.	“In	Vitro	Eugenics.”

Neurology
	66	(12):
1830–6.
of	Value.”	
Sparrow,	Robert.	2013.	“In	Vitro	Eugenics.”	
Journal	of	Medical	Ethics
.
doi:10.1136/medethics-2012-101200.	Published	online	April	4,	2013.	Available
at	
http://jme.bmj.com/content/early/2013/02/13/medethics-2012-101200.full
.
Stansberry,	Matt,	and	Kudritzki,	Julian.	2012.	
Uptime	Institute	2012	Data	Center
Industry	Survey
.	Uptime	Institute.
Stapledon,	Olaf.	1937.	
Star	Maker
.	London:	Methuen.

Steriade,	M.,	Timofeev,	I.,	Durmuller,	N.,	and	Grenier,	F.	1998.	“Dynamic
Properties	of	Corticothalamic	Neurons	and	Local	Cortical	Interneurons
Generating	Fast	Rhythmic	(30–40	Hz)	Spike	Bursts.”	
Journal	of
Neurophysiology
	79	(1):	483–90.
Stewart,	P.	W.,	Lonky,	E.,	Reihman,	J.,	Pagano,	J.,	Gump,	B.	B.,	and	Darvill,	T.
2008.	“The	Relationship	Between	Prenatal	PCB	Exposure	and	Intelligence	(IQ)
in	9-Year-Old	Children.”	
Environmental	Health	Perspectives
	116	(10):	1416–22.
Sun,	W.,	Yu,	H.,	Shen,	Y.,	Banno,	Y.,	Xiang,	Z.,	and	Zhang,	Z.	2012.	“Phylogeny
and	Evolutionary	History	of	the	Silkworm.”	
Science	China	Life	Sciences
	55	(6):
483–96.
Sundet,	J.,	Barlaug,	D.,	and	Torjussen,	T.	2004.	“The	End	of	the	Flynn	Effect?	A
Study	of	Secular	Trends	in	Mean	Intelligence	Scores	of	Norwegian	Conscripts
During	Half	a	Century.”	
	32	(4):	349–62.
Sutton,	Richard	S.,	and	Barto,	Andrew	G.	1998.	
Reinforcement	Learning:	An
Introduction
.	Adaptive	Computation	and	Machine	Learning.	Cambridge,	MA:
MIT	Press.

	55	(6):
483–96.
MIT	Press.
Talukdar,	D.,	Sudhir,	K.,	and	Ainslie,	A.	2002.	“Investigating	New	Product
Diffusion	Across	Products	and	Countries.”	
Marketing	Science
	21	(1):	97–114.
Teasdale,	Thomas	W.,	and	Owen,	David	R.	2008.	“Secular	Declines	in	Cognitive
Test	Scores:	A	Reversal	of	the	Flynn	Effect.”	
	36	(2):	121–6.
Tegmark,	Max,	and	Bostrom,	Nick.	2005.	“Is	a	Doomsday	Catastrophe	Likely?”
Nature
	438:	754.
Teitelman,	Warren.	1966.	“Pilot:	A	Step	Towards	Man–Computer	Symbiosis.”	PhD
diss.,	Massachusetts	Institute	of	Technology.
Temple,	Robert	K.	G.	1986.

Nature
	438:	754.
Temple,	Robert	K.	G.	1986.	
The	Genius	of	China:	3000	Years	of	Science,
Discovery,	and	Invention
.	1st	ed.	New	York:	Simon	&	Schuster.
Tesauro,	Gerald.	1995.	“Temporal	Difference	Learning	and	TD-Gammon.”
Communications	of	the	ACM
	38	(3):	58–68.
Tetlock,	Philip	E.	2005.	
Expert	Political	Judgment:	How	Good	is	it?	How	Can	We

Know?
	Princeton,	NJ:	Princeton	University	Press.
Tetlock,	Philip	E.,	and	Belkin,	Aaron.	1996.	“Counterfactual	Thought	Experiments
in	World	Politics:	Logical,	Methodological,	and	Psychological	Perspectives.”	In
Counterfactual	Thought	Experiments	in	World	Politics:	Logical,
Methodological,	and	Psychological	Perspectives
,	edited	by	Philip	E.	Tetlock	and
Aaron	Belkin,	1–38.	Princeton,	NJ:	Princeton	University	Press.
Thompson,	Adrian.	1997.	“Artificial	Evolution	in	the	Physical	World.”	In
Evolutionary	Robotics:	From	Intelligent	Robots	to	Artificial	Life
,	edited	by
Takashi	Gomi,	101–25.	Er	’97.	Carp,	ON:	Applied	AI	Systems.
Thrun,	S.,	Montemerlo,	M.,	Dahlkamp,	H.,	Stavens,	D.,	Aron,	A.,	Diebel,	J.,	Fong,
P.,	
et	al.
	2006.	“Stanley:	The	Robot	That	Won	the	DARPA	Grand	Challenge.”
Journal	of	Field	Robotics
	23	(9):	661–92.
Trachtenberg,	J.	T.,	Chen,	B.	E.,	Knott,	G.	W.,	Feng,	G.,	Sanes,	J.	R.,	Welker,	E.,
and	Svoboda,	K.	2002.	“Long-Term	In	Vivo	Imaging	of	Experience-Dependent

,	edited	by
P.,	
et	al.
Synaptic	Plasticity	in	Adult	Cortex.”	
Nature
	420	(6917):	788–94.
Traub,	Wesley	A.	2012.	“Terrestrial,	Habitable-Zone	Exoplanet	Frequency	from
Kepler
.”	
	745	(1):	1–10.
Truman,	James	W.,	Taylor,	Barbara	J.,	and	Awad,	Timothy	A.	1993.	“Formation	of
the	Adult	Nervous	System.”	In	
The	Development	of
	Drosophila	Melanogaster,
edited	by	Michael	Bate	and	Alfonso	Martinez	Arias.	Plainview,	NY:	Cold	Spring
Harbor	Laboratory.
Tuomi,	Ilkka.	2002.	“The	Lives	and	the	Death	of	Moore’s	Law.”	
First	Monday
	7

Nature
Kepler
.”	
	7
(11).
Turing,	A.	M.	1950.	“Computing	Machinery	and	Intelligence.”	
Mind
	59	(236):
433–60.
Turkheimer,	Eric,	Haley,	Andreana,	Waldron,	Mary,	D’Onofrio,	Brian,	and
Gottesman,	Irving	I.	2003.	“Socioeconomic	Status	Modifies	Heritability	of	IQ	in
Young	Children.”	
	14	(6):	623–8.
Uauy,	Ricardo,	and	Dangour,	Alan	D.	2006.	“Nutrition	in	Brain	Development	and
Aging:	Role	of	Essential	Fatty	Acids.”	Supplement,	
Nutrition	Reviews
	64	(5):

S24–S33.
Ulam,	Stanislaw	M.	1958.	“John	von	Neumann.”	
Bulletin	of	the	American
Mathematical	Society
	64	(3):	1–49.
Uncertain	Future,	The.	2012.	“Frequently	Asked	Questions”	The	Uncertain	Future.
Retrieved	March	25,	2012.	Available	at
http://www.theuncertainfuture.com/faq.html
.
U.S.	Congress,	Office	of	Technology	Assessment.	1995.	
U.S.–Russian	Cooperation
in	Space
	ISS-618.	Washington,	DC:	U.S.	Government	Printing	Office,	April.
Van	Zanden,	Jan	Luiten.	2003.	
On	Global	Economic	History:	A	Personal	View	on
an	Agenda	for	Future	Research
.	International	Institute	for	Social	History,	July
23.
Vardi,	Moshe	Y.	2012.	“Artificial	Intelligence:	Past	and	Future.”	
Communications
of	the	ACM
	55	(1):	5.
Vassar,	Michael,	and	Freitas,	Robert	A.,	Jr.	2006.	“Lifeboat	Foundation
Nanoshield.”	Lifeboat	Foundation.	Retrieved	May	12,	2012.	Available	at
http://lifeboat.com/ex/nanoshield
.
Vinge,	Vernor.	1993.	“The	Coming	Technological	Singularity:	How	to	Survive	in
the	Post-Human	Era.”	In

23.
of	the	ACM
	55	(1):	5.
.
the	Post-Human	Era.”	In	
Vision-21:	Interdisciplinary	Science	and	Engineering	in
the	Era	of	Cyberspace
,	11–22.	NASA	Conference	Publication	10129.	NASA
Lewis	Research	Center.
Visscher,	P.	M.,	Hill,	W.	G.,	and	Wray,	N.	R.	2008.	“Heritability	in	the	Genomics
Era:	Concepts	and	Misconceptions.”	
	9	(4):	255–66.
Vollenweider,	Franz,	Gamma,	Alex,	Liechti,	Matthias,	and	Huber,	Theo.	1998.
“Psychological	and	Cardiovascular	Effects	and	Short-Term	Sequelae	of	MDMA
(‘Ecstasy’)	in	MDMA-Naïve	Healthy	Volunteers.”	
Neuropsychopharmachology
19	(4):	241–51.

Wade,	Michael	J.	1976.	“Group	Selections	Among	Laboratory	Populations	of
Tribolium.”	
	73	(12):	4604–7.

Wainwright,	Martin	J.,	and	Jordan,	Michael	I.	2008.	“Graphical	Models,
Exponential	Families,	and	Variational	Inference.”	
Foundations	and	Trends	in
Machine	Learning
	1	(1–2):	1–305.
Walker,	Mark.	2002.	“Prolegomena	to	Any	Future	Philosophy.”	
Journal	of
Evolution	and	Technology
	10	(1).
Walsh,	Nick	Paton.	2001.	“Alter	our	DNA	or	robots	will	take	over,	warns
Hawking.”	
The	Observer
,	 September	 1.
http://www.theguardian.com/uk/2001/sep/02/medicalscience.genetics
.
Warwick,	Kevin.	2002.	
I,	Cyborg
.	London:	Century.
Wehner,	M.,	Oliker,	L.,	and	Shalf,	J.	2008.	“Towards	Ultra-High	Resolution
Models	of	Climate	and	Weather.”	
International	Journal	of	High	Performance
Computing	Applications
	22	(2):	149–65.
Weizenbaum,	Joseph.	1966.	“Eliza:	A	Computer	Program	for	the	Study	of	Natural
Language	Communication	Between	Man	And	Machine.”	
Communications	of	the
ACM
	9	(1):	36–45.
Weizenbaum,	Joseph.	1976.	
Computer	Power	and	Human	Reason:	From	Judgment
to	Calculation
.	San	FrancYork,	CA:	W.	H.	Freeman.

ACM
Werbos,	Paul	John.	1994.	
The	Roots	of	Backpropagation:	From	Ordered
Derivatives	to	Neural	Networks	and	Political	Forecasting
White,	J.	G.,	Southgate,	E.,	Thomson,	J.	N.,	and	Brenner,	S.	1986.	“The	Structure
of	the	Nervous	System	of	the	Nematode	
Caenorhabditis	Elegans
.”	
Transactions	of	the	Royal	Society	of	London
.	
Series	B,	Biological	Sciences
	314
(1165):	1–340.
Whitehead,	Hal.	2003.	
Sperm	Whales:	Social	Evolution	in	the	Ocean
.	Chicago:

.”	
.	
	314
.	Chicago:
Whitman,	William	B.,	Coleman,	David	C.,	and	Wiebe,	William	J.	1998.
“Prokaryotes:	The	Unseen	Majority.”	
Proceedings	of	the	National	Academy	of
Sciences	of	the	United	States	of	America
	95	(12):	6578–83.
Wiener,	Norbert.	1960.	“Some	Moral	and	Technical	Consequences	of	Automation.”

Science
	131	(3410):	1355–8.
Wikipedia
.	2012a,	s.v.	“Computer	Bridge.”	Retrieved	June	30,	2013.	Available	at
http://en.wikipedia.org/wiki/Computer_bridge
.
Wikipedia
.	2012b,	s.v.	“Supercomputer.”	Retrieved	June	30,	2013.	Available	at
http://et.wikipedia.org/wiki/Superarvuti
.
Williams,	George	C.	1966.	
Adaptation	and	Natural	Selection:	A	Critique	of	Some
Current	Evolutionary	Thought
.	Princeton	Science	Library.	Princeton,	NJ:
Winograd,	Terry.	1972.	
Understanding	Natural	Language
.	New	York:	Academic
Press.
Wood,	Nigel.	2007.	
Chinese	Glazes:	Their	Origins,	Chemistry	and	Re-creation
.
London:	A.	&	C.	Black.
World	Bank.	2008.	
Global	Economic	Prospects:	Technology	Diffusion	in	the
Developing	World
	42097.	Washington,	DC.
World	Robotics.	2011.	
Executive	Summary	of	1
.	
World	Robotics	2011	Industrial
Robots;	2.	World	Robotics	2011	Service	Robots
.	Retrieved	June	30,	2012.
Available	 at	
http://www.bara.org.uk/pdf/2012/world-
robotics/Executive_Summary_WR_2012.pdf
.

Press.
.
.	
.
World	Values	Survey.	2008.	
WVS	2005-2008
.	Retrieved	29	October,	2013.
http://www.wvsevsdb.com/wvs/WVSAnalizeStudy.jsp
.
Wright,	Robert.	2001.	
Nonzero:	The	Logic	of	Human	Destiny
.	New	York:	Vintage.
Yaeger,	Larry.	1994.	“Computational	Genetics,	Physiology,	Metabolism,	Neural
Systems,	Learning,	Vision,	and	Behavior	or	PolyWorld:	Life	in	a	New	Context.”
In	
Proceedings	of	the	Artificial	Life	III	Conference
,	edited	by	C.	G.	Langton,
263–98.	Santa	Fe	Institute	Studies	in	the	Sciences	of	Complexity.	Reading,	MA:

.
In	
Addison-Wesley.
Yudkowsky,	Eliezer.	2001.	
Creating	Friendly	AI	1.0:	The	Analysis	and	Design	of
Benevolent	Goal	Architectures
.	Machine	Intelligence	Research	Institute,	San
Francisco,	CA,	June	15.

Yudkowsky,	Eliezer.	2002.	“The	AI-Box	Experiment.”	Retrieved	January	15,	2012.
http://yudkowsky.net/singularity/aibox
.
Yudkowsky,	Eliezer.	2004.	
Coherent	Extrapolated	Volition
.	Machine	Intelligence
Research	Institute,	San	Francisco,	CA,	May.
Yudkowsky,	Eliezer.	2007.	“Levels	of	Organization	in	General	Intelligence.”	In
Artificial	General	Intelligence
,	edited	by	Ben	Goertzel	and	Cassio	Pennachin,
389–501.	Cognitive	Technologies.	Berlin:	Springer.
Yudkowsky,	Eliezer.	2008a.	“Artificial	Intelligence	as	a	Positive	and	Negative
Factor	in	Global	Risk.”	In	
Global	Catastrophic	Risks
,	edited	by	Nick	Bostrom
and	Milan	M.	Ćirković,	308–45.	New	York:	Oxford	University	Press.
Yudkowsky,	Eliezer.	2008b.	“Sustained	Strong	Recursion.”	
Less	Wrong
	(blog),
December	5.
Yudkowsky,	Eliezer.	2010.	
Timeless	Decision	Theory
Research	Institute,	San	Francisco,	CA.
Yudkowsky,	Eliezer.	2011.	
Complex	Value	Systems	are	Required	to	Realize
Valuable	Futures

Less	Wrong
	(blog),
Yudkowsky,	Eliezer.	2013.	
Intelligence	Explosion	Microeconomics
,	Technical
Report	2013–1.	Berkeley,	CA:	Machine	Intelligence	Research	Institute.
Zahavi,	Amotz,	and	Zahavi,	Avishag.	1997.	
The	Handicap	Principle:	A	Missing
Piece	of	Darwin’s	Puzzle
.	Translated	by	N.	Zahavi-Ely	and	M.	P.	Ely.	New	York:
Oxford	University	Press.
Zalasiewicz,	J.,	Williams,	M.,	Smith,	A.,	Barry,	T.	L.,	Coe,	A.	L.,	Bown,	P.	R.,
Brenchley,	P.,	
et	al.
	2008.	“Are	We	Now	Living	in	the	Anthropocene?”	
GSA
Today
	18	(2):	4–8.

et	al.
GSA
Today
Zeira,	Joseph.	2011.	“Innovations,	Patent	Races	and	Endogenous	Growth.”	
Journal
of	Economic	Growth
	16	(2):	135–56.
Zuleta,	Hernando.	2008.	“An	Empirical	Note	on	Factor	Shares.”	
Journal	of
International	Trade	and	Economic	Development
	17	(3):	379–90.

INDEX
	
Afghan	Taliban	
215
Agricultural	Revolution	
2
,	
80
,	
261
AI-complete	problem	
14
,	
47
,	
71
,	
93
,	
145
,	
186
AI-OUM,	
see
	optimality	notions
AI-RL,	
see
AI-VL,	
see
algorithmic	soup	
172
algorithmic	trading	
16
–
17
anthropics	
27
–
28
,	
126
,	
134
–
135
,	
174
,	
222
–
225
definition	
225
Arendt,	Hannah	
105
Armstrong,	Stuart	
280
,	
291
,	
294
,	
302
artificial	agent	
10
,	
88
,	
105
–
109
,	
172
–
176
,	
185
–
206
;	
see	also
	Bayesian	agent
artificial	intelligence
arms	race	
64
,	
88
,	
247

future	of	
19
,	
292
greater-than-human,	
see
	superintelligence
history	of	
5
–
18
overprediction	of	
4
pioneers	
4
–
5
,	
18
Asimov,	Isaac	
139
augmentation	
142
–
143
,	
201
–
203
autism	
57
automata	theory	
5
automatic	circuit	breaker	
17
automation	
17
,	
98
,	
117
,	
160
–
176
	
backgammon	
12
backpropagation	algorithm	
8
bargaining	costs	
182
Bayesian	agent	
9
–
11
,	
123
,	
130
;	
see	also
	artificial	agent	
and
Bayesian	networks	
9
Berliner,	Hans	
12
biological	cognition	
22
,	
36
–
48
,	
50
–
51
,	
232
biological	enhancement	
36
–
48
,	
50
–
51
,	
142
–
143
,	
232
;	
see	also
	cognitive	enhancement

boxing	
129
–
131
,	
143
,	
156
–
157
informational	
130
physical	
129
–
130
brain	implant,	
see
	cyborg
brain	plasticity	
48
brain–computer	interfaces	
44
–
48
,	
51
,	
83
,	
142
–
143
;	
see	also
	cyborg
Brown,	Louise	
43
	
C.	elegans
34–
35
,	
266
,	
267
capability	control	
129
–
144
,	
156
–
157
capital	
39
,	
48
,	
68
,	
84
–
88
,	
99
,	
113
–
114
,	
159
–
184
,	
251
,	
287
,	
288
,	
289
causal	validity	semantics	
197
CEV,	
see
	coherent	extrapolated	volition
Chalmers,	David	
24
,	
265
,	
283
,	
295
,	
302
character	recognition	
15
checkers	
12
chess	
11
–
22
,	
52
,	
93
,	
134
,	
263
,	
264
child	machine	
23
,	
29
;	
see	also
	seed	AI
CHINOOK	
12
Christiano,	Paul	
198
,	
207

civilization	baseline	
63
cloning	
42
cognitive	enhancement	
42
–
51
,	
67
,	
94
,	
111
–
112
,	
193
,	
204
,	
232
–
238
,	
244
,	
259
coherent	extrapolated	volition	(CEV)	
198
,	
211
–
227
,	
296
,	
298
,	
303
211
collaboration	(benefits	of)	
249
collective	intelligence	
48
–
51
,	
52
–
57
,	
67
,	
72
,	
142
,	
163
,	
203
,	
259
,	
271
,	
273
,	
279
collective	superintelligence	
39
,	
48
–
49
,	
52
–
59
,	
83
,	
93
,	
99
,	
285
54
combinatorial	explosion	
6
,	
9
,	
10
,	
47
,	
155
Common	Good	Principle	
254
–
259
common	sense	
14
computer	vision	
9
computing	power	
7
–
9
,	
24
,	
25
–
35
,	
47
,	
53
–
60
,	
68
–
77
,	
101
,	
134
,	
155
,	
198
,	
240
–
244
,
251
,	
286
,	
288
;	
see	also
	computronium	
and
	hardware	overhang
computronium	
101
,	
123
–
124
,	
140
,	
193
,	
219
;	
see	also
	computing	power
connectionism	
8
consciousness	
22
,	
106
,	
126
,	
139
,	
173
–
176
,	
216
,	
226
,	
271
,	
282
,	
288
,	
292
,	
303
;	
see
also
	mind	crime
control	methods	
127
–
144
,	
145
–
158
,

254
–
259
14
9
7
–
9
,	
24
,	
25
–
35
,	
47
,	
53
–
60
,	
68
–
77
,	
101
,	
134
,	
155
,	
198
,	
240
–
244
,
251
,	
286
,	
288
;	
see	also
and
101
,	
123
–
124
,	
140
,	
193
,	
219
;	
see	also
8
22
,	
106
,	
126
,	
139
,	
173
–
176
,	
216
,	
226
,	
271
,	
282
,	
288
,	
292
,	
303
;	
see
also
	mind	crime
127
–
144
,	
145
–
158
,	
202
,	
236
–
238
,	
286
;	
see	also
	capability	control
and
	motivation	selection
Copernicus,	Nicolaus	
14

cosmic	endowment	
101
–
104
,	
115
,	
134
,	
209
,	
214
–
217
,	
227
,	
250
,	
260
,	
283
,	
296
crosswords	(solving)	
12
cryptographic	reward	tokens	
134
,	
276
cryptography	
80
cyborg	
44
–
48
,	
67
,	
270
	
DARPA,	
see
	Defense	Advanced	Research	Projects	Agency
DART	(tool)	
15
Dartmouth	Summer	Project	
5
data	mining	
15
–
16
,	
232
,	
301
decision	support	systems	
15
,	
98
;	
see	also
	tool-AI
decision	theory	
10
–
11
,	
88
,	
185
–
186
,	
221
–
227
,	
280
,	
298
;	
see	also
decisive	strategic	advantage	
78
–
89
,	
95
,	
104
–
112
,	
115
–
126
,	
129
–
138
,	
148
–
149
,
156
–
159
,	
177
,	
190
,	
209
–
214
,	
225
,	
252
Deep	Blue	
12
Deep	Fritz	
22
Defense	Advanced	Research	Projects	Agency	(DARPA)	
15
design	effort,	
see
	optimization	power
Dewey,	Daniel	
291
Differential	Technological	Development	(Principle	of)	
230
–
237

Diffie–Hellman	key	exchange	protocol	
80
diminishing	returns	
37
–
38
,	
66
,	
88
,	
114
,	
273
,	
303
direct	reach	
58
direct	specification	
139
–
143
DNA	synthesis	
39
,	
98
Do	What	I	Mean	(DWIM)	
220
–
221
domesticity	
140
–
143
,	
146
–
156
,	
187
,	
191
,	
207
,	
222
Drexler,	Eric	
239
,	
270
,	
276
,	
278
,	
300
drones	
15
,	
98
Dutch	book	
111
Dyson,	Freeman	
101
,	
278
	
economic	growth	
3
,	
160
–
166
,	
179
,	
261
,	
274
,	
299
Einstein,	Albert	
56
,	
70
,	
85
ELIZA	(program)	
6
embryo	selection	
36
–
44
,	
67
,	
268
emulation	modulation	
207
Enigma	code	
87
environment	of	evolutionary	adaptedness	
164
,	
171
epistemology	
222
–
224

equation	solvers	
15
eugenics	
36
–
44
,	
268
,	
279
Eurisko	
12
evolution	
8
–
9
,	
23
–
27
,	
44
,	
154
,	
173
–
176
,	
187
,	
198
,	
207
,	
265
,	
266
,	
267
,	
273
evolutionary	selection	
187
,	
207
,	
290
evolvable	hardware	
154
exhaustive	search	
6
existential	risk	
4
,	
21
,	
55
,	
100
–
104
,	
115
–
126
,	
175
,	
183
,	
230
–
236
,	
239
–
254
,	
256
–
259
,
286
,	
301
–
302
state	risks	
233
–
234
step	risks	
233
expert	system	
7
explicit	representation	
207
exponential	growth,	
see
	growth
external	reference	semantics	
197
	
face	recognition	
15
failure	modes	
117
–
120
Faraday	cage	
130
Fields	Medal	
255
–
256
,	
272

Fifth-Generation	Computer	Systems	Project	
7
fitness	function	
25
;	
see	also
	evolution
Flash	Crash	(2010)	
16
–
17
formal	language	
7
,	
145
FreeCell	(game)	
13
	
game	theory	
87
,	
159
game-playing	AI	
12
–
14
General	Problem	Solver	
6
genetic	algorithms	
7
–
13
,	
24
–
27
,	
237
–
240
;	
see	also
	evolution
genetic	selection	
37
–
50
,	
61
,	
232
–
238
;	
see	also
	evolution
genie	AI	
148
–
158
,	
285
148
genotyping	
37
germline	interventions	
37
–
44
,	
67
,	
273
;	
see	also
	embryo	selection
Ginsberg,	Matt	
12
Go	(game)	
13
goal-content	
109
–
110
,	
146
,	
207
,	
222
–
227
Good	Old-Fashioned	Artificial	Intelligence	(GOFAI)	
7
–
15
,	
23
Good,	I.	J.	
4

Gorbachev,	Mikhail	
86
graceful	degradation	
7
graphical	models	
11
growth	
1
–
7
,	
48
–
55
,	
69
–
75
,	
83
,	
163
,	
261
,	
281
	
Hail	Mary	approach	
198
–
200
,	
207
,	
293
,	
294
Hanson,	Robin	
2
,	
160
,	
261
,	
270
,	
271
,	
287
hardware	overhang	
73
,	
240
–
243
,	
274
,	
289
,	
301
,	
302
hedonism	
140
,	
210
hedonium	
140
,	
219
Helsinki	Declaration	
188
heuristic	search	
6
Hill,	Benny	
105
hippocampus	
47
HLMI,	
see
	machine	intelligence,	human-level
Hodgkin–Huxley	model	
25
human	baseline	
62
–
77
,	
82
human	extinction,	
see
	existential	risk
Human	Genome	Project	
86
,	
253
,	
276
human	intelligence	
5
–
14
,	
24
–
58
,	
98
–
99
,	
159
–
184
,	
242
–
254
,	
255
–
257

human–machine	interface,	
see
	cyborg
	
impersonal	perspective	
228
–
246
in	vitro
	fertilization,	
see
incentive	methods	
131
–
143
133
social	integration	
131
–
132
,	
156
–
158
,	
159
,	
202
,	
283
indirect	normativity	
141
–
150
,	
209
–
227
,	
262
,	
298
indirect	reach	
58
inductive	bias	
9
–
10
Industrial	Revolution	
2
,	
80
,	
161
–
163
infrastructure	profusion	
123
–
125
,	
153
,	
187
,	
226
,	
282
institution	design	
202
–
208
instrumental	convergence	thesis	
105
–
116
intelligence	explosion	
2
–
5
,	
22
–
51
,	
62
–
77
,	
78
–
90
,	
95
–
104
,	
108
,	
115
–
126
,	
127
–
128
,
136
,	
151
,	
160
,	
165
,	
178
,	
198
,	
205
,	
227
,	
228
–
254
,	
256
–
260
,	
274
,	
276
,	
282
,	
284
,
289
,	
300
,	
301
–
302
Internet	
16
,	
45
–
49
,	
67
–
77
,	
85
,	
94
–
98
,	
130
,	
146
,	
241
,	
271
inventory	control	systems	
16
IVF,	
see

Jeopardy!13
	
Kasparov,	Garry	
12
Kepler,	Johannes	
14
Knuth,	Donald	
14
,	
264
Kurzweil,	Ray	
2
,	
261
,	
269
	
Lenat,	Douglas	
12
,	
263
Logic	Theorist	(system)	
6
logicist	paradigm,	
see
	Good	Old-Fashioned	Artificial	Intelligence	(GOFAI)
Logistello	
12
	
machine	intelligence;	
see	also
	artificial	intelligence
human-level	(HLMI)	
4
,	
19
–
21
,	
27
–
35
,	
73
–
74
,	
207
,	
243
,	
264
,	
267
revolution,	
see
	intelligence	explosion
machine	learning	
8
–
18
,	
28
,	
121
,	
152
,	
188
,	
274
,	
290
machine	translation	
15

macro-structural	development	accelerator	
233
–
235
malignant	failure	
123
–
126
,	
149
,	
196
Malthusian	condition	
163
–
165
,	
252
Manhattan	Project	
75
,	
80
–
87
,	
276
McCarthy,	John	
5
–
18
McCulloch–Pitts	neuron	
237
MegaEarth	
56
memory	capacity	
7
–
9
,	
60
,	
71
memory	sharing	
61
Mill,	John	Stuart	
210
mind	crime	
125
–
126
,	
153
,	
201
–
208
,	
213
,	
226
,	
297
Minsky,	Marvin	
18
,	
261
,	
262
,	
282
Monte	Carlo	method	
9
–
13
Moore’s	law	
24
–
25
,	
73
–
77
,	
274
,	
286
;	
see	also
moral	growth	
214
moral	permissibility	(MP)218–
220
,	
297
moral	rightness	(MR)217–
220
.
296
,	
297
moral	status	
125
–
126
,	
166
–
169
,	
173
,	
202
–
205
,	
268
,	
288
,	
296
Moravec,	Hans	
24
,	
265
,	
288
motivation	selection	
29
,	
127
–
129
,	
138
–
144
,	
147
,	
158
,	
168
,	
180
–
191
,	
222
138

motivational	scaffolding	
191
,	
207
multipolar	scenarios	
90
,	
132
,	
159
–
184
,	
243
–
254
,	
301
mutational	load	
41
	
nanotechnology	
53
,	
94
–
98
,	
103
,	
113
,	
177
,	
231
,	
239
,	
276
,	
277
,	
299
,	
300
natural	language	
14
neural	networks	
5
–
9
,	
28
,	
46
,	
173
,	
237
,	
262
,	
274
neurocomputational	modeling	
25
–
30
,	
35
,	
61
,	
301
;	
see	also
	whole	brain	emulation
(WBE)	
and
	neuromorphic	AI
neuromorphic	AI	
28
,	
34
,	
47
,	
237
–
245
,	
267
,	
300
,	
301
Newton,	Isaac	
56
Nilsson,	Nils	
18
–
20
,	
264
nootropics	
36
–
44
,	
66
–
67
,	
201
,	
267
Norvig,	Peter	
19
,	
264
,	
282
	
observation	selection	theory,	
see
	anthropics
Oliphant,	Mark	
85
O’Neill,	Gerard	
101
ontological	crisis	
146
,	
197
optimality	notions	
10
,	
186
,	
194
,	
291
–
293

9
–
11
value	learner	(AI-VL)	
194
observation-utility-maximizer	(AI-OUM)	
194
reinforcement	learner	(AI-RL)	
194
optimization	power	
24
,	
62
–
75
,	
83
,	
92
–
96
,	
227
,	
274
65
oracle	AI	
141
–
158
,	
222
–
226
,	
285
,	
286
146
orthogonality	thesis	
105
–
109
,	
115
,	
279
,	
280
	
paperclip	AI	
107
–
108
,	
123
–
125
,	
132
–
135
,	
153
,	
212
,	
243
Parfit,	Derek	
279
Pascal’s	mugging	
223
,	
298
Pascal’s	wager	
223
person-affecting	perspective	
228
,	
245
–
246
,	
301
perverse	instantiation	
120
–
124
,	
153
,	
190
–
196
poker	
13
principal–agent	problem	
127
–
128
,	
184
Principle	of	Epistemic	Deference	
211
,	
221
Proverb	(program)	
12

qualia,	
see
	consciousness
quality	superintelligence	
51
–
58
,	
72
,	
243
,	
272
56
	
race	dynamic,	
see
	technology	race
rate	of	growth,	
see
	growth
ratification	
222
–
225
Rawls,	John	
150
Reagan,	Ronald	
86
–
87
reasons-based	goal	
220
recalcitrance	
62
–
77
,	
92
,	
241
,	
274
65
recursive	self-improvement	
29
,	
75
,	
96
,	
142
,	
259
;	
see	also
	seed	AI
reinforcement	learning	
12
,	
28
,	
188
–
189
,	
194
–
196
,	
207
,	
237
,	
277
,	
282
,	
290
resource	acquisition	
113
–
116
,	
123
,	
193
reward	signal	
71
,	
121
–
122
,	
188
,	
194
,	
207
Riemann	hypothesis	catastrophe	
123
,	
141
robotics	
9
–
19
,	
94
–
97
,	
117
–
118
,	
139
,	
238
,	
276
,	
290

Roosevelt,	Franklin	D.85
RSA	encryption	scheme	
80
Russell,	Bertrand	
6
,	
87
,	
139
,	
277
	
Samuel,	Arthur	
12
Sandberg,	Anders	
265
,	
267
,	
272
,	
274
scanning,	
see
	whole	brain	emulation	(WBE)
Schaeffer,	Jonathan	
12
scheduling	
15
Schelling	point	
147
,	
183
,	
296
Scrabble	
13
second	transition	
176
–
178
,	
238
,	
243
–
245
,	
252
second-guessing	(arguments)	
238
–
239
seed	AI	
23
–
29
,	
36
,	
75
,	
83
,	
92
–
96
,	
107
,	
116
–
120
,	
142
,	
151
,	
189
–
198
,	
201
–
217
,
224
–
225
,	
240
–
241
,	
266
,	
274
,	
275
,	
282
self-limiting	goal	
123
Shakey	(robot)	
6
SHRDLU	(program)	
6
Shulman,	Carl	
178
–
180
,	
265
,	
287
,	
300
,	
302
,	
304
simulation	hypothesis	
134
–
135
,	
143
,	
278
,	
288
,	
292

singleton	
78
–
90
,	
95
–
104
,	
112
–
114
,	
115
–
126
,	
136
,	
159
,	
176
–
184
,	
242
,	
275
,	
276
,	
279
,
281
,	
287
,	
299
,	
301
,	
303
78
,	
100
singularity	
1
,	
2
,	
49
,	
75
,	
261
,	
274
;	
see	also
social	signaling	
110
somatic	gene	therapy	
42
sovereign	AI	
148
–
158
,	
187
,	
226
,	
285
speech	recognition	
15
–
16
,	
46
speed	superintelligence	
52
–
58
,	
75
,	
270
,	
271
53
Strategic	Defense	Initiative	(“Star	Wars”)	
86
strong	AI	
18
stunting	
135
–
137
,	
143
sub-symbolic	processing,	
see
	connectionism
superintelligence;	
see	also
	collective	superintelligence,	quality	superintelligence	
and
speed	superintelligence
22
,	
52
forms	
52
,	
59
paths	to	
22
,	
50
predicting	the	behavior	of	
108
,	
155
,	
302
superorganisms	
178
–
180
superpowers	
52
–
56
,	
80
,	
86
–
87
,	
91
–
104
,	
119
,	
133
,	
148
,	
277
,	
279
,	
296

types	
94
surveillance	
15
,	
49
,	
64
,	
82
–
85
,	
94
,	
117
,	
132
,	
181
,	
232
,	
253
,	
276
,	
294
,	
299
Szilárd,	Leó	
85
	
TD-Gammon	
12
Technological	Completion	Conjecture	
112
–
113
,	
229
technology	race	
80
–
82
,	
86
–
90
	
203
–
205
,	
231
,	
246
–
252
,	
302
teleological	threads	
110
Tesauro,	Gerry	
12
TextRunner	(system)	
71
theorem	prover	
15
,	
266
three	laws	of	robotics	
139
,	
284
Thrun,	Sebastian	
19
tool-AI	
151
–
158
151
treacherous	turn	
116
–
119
,	
128
Tribolium	castaneum	
154
tripwires	
137
–
143
Truman,	Harry	
85
Turing,	Alan	
4
,	
23
,	
29
,	
44
,	
225
,	
265
,	
271
,	
272

unemployment	
65
,	
159
–
180
,	
287
United	Nations	
87
–
89
,	
252
–
253
universal	accelerator	
233
unmanned	vehicle,	
see
	drone
uploading,	
see
utility	function	
10
–
11
,	
88
,	
100
,	
110
,	
119
,	
124
–
125
,	
133
–
134
,	
172
,	
185
–
187
,	
192
–
208
,
290
,	
292
,	
293
,	
303
	
value	learning	
191
–
198
,	
208
,	
293
value-accretion	
189
–
190
,	
207
value-loading	
185
–
208
,	
293
,	
294
veil	of	ignorance	
150
,	
156
,	
253
,	
285
Vinge,	Vernor	
2
,	
49
,	
270
virtual	reality	
30
,	
31
,	
53
,	
113
,	
166
,	
171
,	
198
,	
204
,	
300
von	Neumann	probe	
100
–
101
,	
113
von	Neumann,	John	
44
,	
87
,	
114
,	
261
,	
277
,	
281
	
wages	
65
,	
69
,	
160
–
169

Watson	(IBM)	
13
,	
71
WBE,	
see
Whitehead,	Alfred	N.6
whole	brain	emulation	(WBE)	
28
–
36
,	
50
,	
60
,	
68
–
73
,	
77
,	
84
–
85
,	
108
,	
172
,	
198
,
201
–
202
,	
236
–
245
,	
252
,	
266
,	
267
,	
274
,	
299
,	
300
,	
301
Wigner,	Eugene	
85
windfall	clause	
254
,	
303
Winston,	Patrick	
18
wire-heading	
122
–
123
,	
133
,	
189
,	
194
,	
207
,	
282
,	
291
wise-singleton	sustainability	threshold	
100
–
104
,	
279
world	economy	
2
–
3
,	
63
,	
74
,	
83
,	
159
–
184
,	
274
,	
277
,	
285
	
Yudkowsky,	Eliezer	
70
,	
92
,	
98
,	
106
,	
197
,	
211
–
216
,	
266
,	
273
,	
282
,	
286
,	
291
,	
299

More	Praise	for	
How	to	Create	a	Mind
	
“This	book	is	a	Rosetta	stone	for	the	mystery	of	human	thought.	Even	more
remarkably,	it	is	a	blueprint	for	creating	artificial	consciousness	that	is	as
persuasive	and	emotional	as	our	own.	Kurzweil	deals	with	the	subject	of
consciousness	better	than	anyone	from	Blackmore	to	Dennett.	His
persuasive	thought	experiment	is	of	Einstein	quality:	It	forces	recognition
of	the	truth.”
—Martine	Rothblatt,	chairman	and	CEO,	United	Therapeutics;	creator
of	Sirius	XM	Satellite	Radio
	
“Kurzweil’s	book	is	a	shining	example	of	his	prodigious	ability	to
synthesize	ideas	from	disparate	domains	and	explain	them	to	readers	in
simple,	elegant	language.	Just	as	Chanute’s	
Progress	in	Flying	Machines
ushered	in	the	era	of	aviation	over	a	century	ago,	this	book	is	the	harbinger
of	the	coming	revolution	in	artificial	intelligence	that	will	fulfill	Kurzweil’s
own	prophecies	about	it.”
—Dileep	George,	AI	scientist;	pioneer	of	hierarchical	models	of	the

data/pdfs/How to Create a Mind-The Secret of Human Thought Revealed.pdf

neocortex;	cofounder	of	Numenta	and	Vicarious	Systems
	
“Ray	Kurzweil’s	understanding	of	the	brain	and	artificial	intelligence	will
dramatically	impact	every	aspect	of	our	lives,	every	industry	on	Earth,	and
how	we	think	about	our	future.	If	you	care	about	any	of	these,	
read
	this
book!”
—Peter	H.	Diamandis,	chairman	and	CEO,	X	PRIZE;	executive
chairman,	Singularity	University;	author	of	the	
bestseller	
Abundance:
	
The	Future	Is	Better	Than	You	Think

ALSO	BY	RAY	KURZWEIL
Transcend:	Nine	Steps	to	Living	Well	Forever
(with	Terry	Grossman)	
The	Singularity	Is	Near:	When	Humans	Transcend
Biology
Fantastic	Voyage:	Live	Long	Enough	to	Live	Forever
The	Age	of	Spiritual	Machines:
When	Computers	Exceed	Human	Intelligence
The	10%	Solution	for	a	Healthy	Life
The	Age	of	Intelligent	Machines

HOW	TO
CREATE
A
MIND
THE	SECRET	OF	HUMAN
THOUGHT	REVEALED
RAY	KURZWEIL
	
VIKING

VIKING
Published	by	the	Penguin	Group	Penguin	Group	(USA)	Inc.,	375	Hudson	Street,	New	York,	New	York
10014,	U.S.A.	•	Penguin	Group	(Canada),	90	Eglinton	Avenue	East,	Suite	700,	Toronto,	Ontario,	Canada
M4P	2Y3	(a	division	of	Pearson	Penguin	Canada	Inc.)	•	Penguin	Books	Ltd,	80	Strand,	London	WC2R
0RL,	England	•	Penguin	Ireland,	25	St.	Stephen’s	Green,	Dublin	2,	Ireland	(a	division	of	Penguin	Books
Ltd)	•	Penguin	Group	(Australia),	707	Collins	Street,	Melbourne,	Victoria	3008,	Australia	(a	division	of
Pearson	Australia	Group	Pty	Ltd)	•	Penguin	Books	India	Pvt	Ltd,	11	Community	Centre,	Panchsheel	Park,
New	Delhi–110	017,	India	•	Penguin	Group	(NZ),	67	Apollo	Drive,	Rosedale,	Auckland	0632,	New
Zealand	(a	division	of	Pearson	New	Zealand	Ltd)	•	Penguin	Books,	Rosebank	Office	Park,	181	Jan	Smuts
Avenue,	Parktown	North	2193,	South	Africa	•	Penguin	China,	B7	Jaiming	Center,	27	East	Third	Ring	Road
North,	Chaoyang	District,	Beijing	100020,	China	Penguin	Books	Ltd,	Registered	Offices:

80	Strand,	London	WC2R	0RL,	England
First	published	in	2012	by	Viking	Penguin,	a	member	of	Penguin	Group	(USA)	Inc.
1			3			5			7			9			10			8			6			4			2
Copyright	©	Ray	Kurzweil,	2012
All	rights	reserved
“Red”	by	Amoo	Oluseun.	Used	by	permission	of	the	author.
“The	picture’s	pretty	bleak,	gentlemen…”	from	
The	Far	Side
	by	Gary	Larson	(November	7,	1985).
Used	by	permission	of	Creators	Syndicate.
Illustration	credits
Page	10:	Created	by	Wolfgang	Beyer	(Creative	Commons	Attribution–Share	Alike	3.0	License).	21:

Photo	by	Timeline	(Creative	Commons	Attribution–Share	Alike	3.0	License).	84	(two	figures):	From	“The
Geometric	Structure	of	the	Brain	Fiber	Pathways,”	by	Van	J.	Wedeen,	Douglas	L.	Rosene,	Ruopene	Wang,
Guangping	Dai,	Farzad	Mortazavi,	Patric	Hagmann,	Jon	H.	Kaas,	and	Wen-Yih	I.	Tseng,	
Science,
	March
30,	2012.	Reprinted	with	permission	of	AAAS	(American	Association	for	the	Advancement	of	Science).
85:	Photo	provided	by	Yeatesh	(Creative	Commons	Attribution–Share	Alike	3.0	License).	134	(two):

Science,
	March
Images	by	Marvin	Minsky.	Used	by	permission	of	Marvin	Minsky.	Some	credits	appear	adjacent	to	the
respective	images.	Other	images	designed	by	Ray	Kurzweil,	illustrated	by	Laksman	Frank.
Library	of	Congress	Cataloging-in-Publication	Data	Kurzweil,	Ray.
How	to	create	a	mind	:	the	secret	of	human	thought	revealed	/	Ray	Kurzweil.
p.				cm.
ISBN:	978-1-101-60110-5
1.	Brain—Localization	of	functions.	2.	Self-consciousness	(Awareness)	3.	Artificial	intelligence.	I.	Title.

p.				cm.
QP385.K87	2012
612.8’2—dc23								2012027185
Printed	in	the	United	States	of	America	Set	in	Minion	Pro	with	DIN
Designed	by	Daniel	Lagin
While	the	author	has	made	every	effort	to	provide	accurate	telephone	numbers,	Internet	addresses,	and
other	contact	information	at	the	time	of	publication,	neither	the	publisher	nor	the	author	assumes	any
responsibility	for	errors,	or	for	changes	that	occur	after	publication.	Further,	the	publisher	does	not	have	any

control	over	and	does	not	assume	any	responsibility	for	author	or	third-party	Web	sites	or	their	content.
No	part	of	this	book	may	be	reproduced,	scanned,	or	distributed	in	any	printed	or	electronic	form
without	permission.	Please	do	not	participate	in	or	encourage	piracy	of	copyrighted	materials	in	violation	of
the	author’s	rights.	Purchase	only	authorized	editions.
ALWAYS	LEARNING
PEARSON

I’d	like	to	express	my	gratitude	to	my	wife,	Sonya,	for	her	loving	patience
through	the	vicissitudes	of	the	creative	process;	To	my	children,	Ethan	and	Amy;
my	daughter-in-law,	Rebecca;	my	sister,	Enid;	and	my	new	grandson,	Leo,	for
their	love	and	inspiration;	To	my	mother,	Hannah,	for	supporting	my	early	ideas
and	inventions,	which	gave	me	the	freedom	to	experiment	at	a	young	age,	and
for	keeping	my	father	alive	during	his	long	illness;	To	my	longtime	editor	at
Viking,	Rick	Kot,	for	his	leadership,	steady	and	insightful	guidance,	and	expert
editing;	To	Loretta	Barrett,	my	literary	agent	for	twenty	years,	for	her	astute	and
enthusiastic	guidance;	To	Aaron	Kleiner,	my	long-term	business	partner,	for	his
devoted	collaboration	for	the	past	forty	years;	To	Amara	Angelica	for	her
devoted	and	exceptional	research	support;
To	Sarah	Black	for	her	outstanding	research	insights	and	ideas;
To	Laksman	Frank	for	his	excellent	illustrations;
To	Sarah	Reed	for	her	enthusiastic	organizational	support;

To	Nanda	Barker-Hook	for	her	expert	organization	of	my	public	events	on
this	and	other	topics;	
To	Amy	Kurzweil	for	her	guidance	on	the	craft	of	writing;
To	Cindy	Mason	for	her	research	support	and	ideas	on	AI	and	the	mind-
body	connection;	To	Dileep	George	for	his	discerning	ideas	and	insightful
discussions	by	e-mail	and	otherwise;	To	Martine	Rothblatt	for	her	dedication	to
all	of	the	technologies	I	discuss	in	the	book	and	for	our	collaborations	in
developing	technologies	in	these	areas;	To	the	KurzweilAI.net	team,	who

provided	significant	research	and	logistical	support	for	this	project,	including
Aaron	Kleiner,	Amara	Angelica,	Bob	Beal,	Casey	Beal,	Celia	Black-Brooks,
Cindy	Mason,	Denise	Scutellaro,	Joan	Walsh,	Giulio	Prisco,	Ken	Linde,
Laksman	Frank,	Maria	Ellis,	Nanda	Barker-Hook,	Sandi	Dube,	Sarah	Black,
Sarah	Brangan,	and	Sarah	Reed;	To	the	dedicated	team	at	Viking	Penguin	for	all
of	their	thoughtful	expertise,	including	Clare	Ferraro	(president),	Carolyn
Coleburn	(director	of	publicity),	Yen	Cheong	and	Langan	Kingsley	(publicists),

Nancy	Sheppard	(director	of	marketing),	Bruce	Giffords	(production	editor),
Kyle	Davis	(editorial	assistant),	Fabiana	Van	Arsdell	(production	director),

Roland	Ottewell	(copy	editor),	Daniel	Lagin	(designer),	and	Julia	Thomas
(jacket	designer);	To	my	colleagues	at	Singularity	University	for	their	ideas,
enthusiasm,	and	entrepreneurial	energy;	To	my	colleagues	who	have	provided
inspired	ideas	reflected	in	this	volume,	including	Barry	Ptolemy,	Ben	Goertzel,
David	Dalrymple,	Dileep	George,	Felicia	Ptolemy,	Francis	Ganong,	George
Gilder,	Larry	Janowitch,	Laura	Deming,	Lloyd	Watts,	Martine	Rothblatt,	Marvin
Minsky,	Mickey	Singer,	Peter	Diamandis,	Raj	Reddy,	Terry	Grossman,	Tomaso
Poggio,	and	Vlad	Sejnoha;	To	my	peer	expert	readers,	including	Ben	Goertzel,
David	Gamez,	Dean	Kamen,	Dileep	George,	Douglas	Katz,	Harry	George,	Lloyd
Watts,	
Martine	Rothblatt,	Marvin	Minsky,	Paul	Linsay,	Rafael	Reif,	Raj	Reddy,
Randal	Koene,	Dr.	Stephen	Wolfram,	and	Tomaso	Poggio;	To	my	in-house	and
lay	readers	whose	names	appear	above;
And,	finally,	to	all	of	the	creative	thinkers	in	the	world	who	inspire	me
every	day.

CONTENTS
	
INTRODUCTION
1.	THOUGHT	EXPERIMENTS	ON	THE	WORLD
2.	THOUGHT	EXPERIMENTS	ON	THINKING
3.	A	MODEL	OF	THE	NEOCORTEX:	THE	PATTERN	RECOGNITION
THEORY	OF	MIND
4.	THE	BIOLOGICAL	NEOCORTEX
5.	THE	OLD	BRAIN
6.	TRANSCENDENT	ABILITIES
7.	THE	BIOLOGICALLY	INSPIRED	DIGITAL	NEOCORTEX
8.	THE	MIND	AS	COMPUTER
9.	THOUGHT	EXPERIMENTS	ON	THE	MIND
10.	THE	LAW	OF	ACCELERATING	RETURNS	APPLIED	TO	THE
BRAIN
11.	OBJECTIONS
EPILOGUE
NOTES
INDEX

The	Brain—is	wider	than	the	Sky—
For—put	them	side	by	side—
The	one	the	other	will	contain
With	ease—and	You—beside—
The	Brain	is	deeper	than	the	sea—
For—hold	them—Blue	to	Blue—
The	one	the	other	will	absorb—
As	Sponges—Buckets—do—
The	Brain	is	just	the	weight	of	God—
For—Heft	them—Pound	for	Pound—
And	they	will	differ—if	they	do—
As	Syllable	from	Sound
—Emily	Dickinson
	
A
s	the	most	important	phenomenon	in	the	universe,	intelligence	is	capable	of
transcending	natural	limitations,	and	of	transforming	the	world	in	its	own	image.
In	human	hands,	our	intelligence	has	enabled	us	to	overcome	the	restrictions	of
our	biological	heritage	and	to	change	ourselves	in	the	process.	We	are	the	only
species	that	does	this.
The	story	of	human	intelligence	starts	with	a	universe	that	is	capable	of
encoding	information.	This	was	the	enabling	factor	that	allowed	evolution	to
take	place.	How	the	universe	got	to	be	this	way	is	itself	an	interesting	story.	The

standard	model	of	physics	has	dozens	of	constants	that	need	to	be	precisely	what
they	are,	or	atoms	would	not	have	been	possible,	and	there	would	have	been	no
stars,	no	planets,	no	brains,	and	no	books	on	brains.	That	the	laws	of	physics	are
so	precisely	tuned	to	have	allowed	the	evolution	of	information	appears	to	be
incredibly	unlikely.	Yet	by	the	anthropic	principle,	we	would	not	be	talking

about	it	if	it	were	not	the	case.	Where	some	people	see	a	divine	hand,	others	see
a	multiverse	spawning	an	evolution	of	universes	with	the	boring	(non-
information-bearing)	ones	dying	out.	But	regardless	of	how	our	universe	got	to
be	the	way	it	is,	we	can	start	our	story	with	a	world	based	on	information.
The	story	of	evolution	unfolds	with	increasing	levels	of	abstraction.	Atoms
—especially	carbon	atoms,	which	can	create	rich	information	structures	by
linking	in	four	different	directions—formed	increasingly	complex	molecules.	As
a	result,	physics	gave	rise	to	chemistry.
A	billion	years	later,	a	complex	molecule	called	DNA	evolved,	which	could
precisely	encode	lengthy	strings	of	information	and	generate	organisms
described	by	these	“programs.”	As	a	result,	chemistry	gave	rise	to	biology.
At	an	increasingly	rapid	rate,	organisms	evolved	communication	and
decision	networks	called	nervous	systems,	which	could	coordinate	the
increasingly	complex	parts	of	their	bodies	as	well	as	the	behaviors	that

facilitated	their	survival.	The	neurons	making	up	nervous	systems	aggregated
into	brains	capable	of	increasingly	intelligent	behaviors.	In	this	way,	biology
gave	rise	to	neurology,	as	brains	were	now	the	cutting	edge	of	storing	and
manipulating	information.	Thus	we	went	from	atoms	to	molecules	to	DNA	to
brains.	The	next	step	was	uniquely	human.
The	mammalian	brain	has	a	distinct	aptitude	not	found	in	any	other	class	of
animal.	We	are	capable	of	
hierarchical
	thinking,	of	understanding	a	structure

composed	of	diverse	elements	arranged	in	a	pattern,	representing
	that
arrangement	with	a	symbol,	and	then	using	that	symbol	as	an	element	in	a	yet
more	elaborate	configuration.	This	capability	takes	place	in	a	brain	structure
called	the	neocortex,	which	in	humans	has	achieved	a	threshold	of	sophistication
and	capacity	such	that	we	are	able	to	call	these	patterns	
ideas
.	Through	an
unending	recursive	process	we	are	capable	of	building	ideas	that	are	ever	more

thinking,	of	understanding	a	structure
	that
ideas
complex.	We	call	this	vast	array	of	recursively	linked	ideas	
knowledge
.	Only
	have	a	knowledge	base	that	itself	evolves,	grows	exponentially,
and	is	passed	down	from	one	generation	to	another.
Our	brains	gave	rise	to	yet	another	level	of	abstraction,	in	that	we	have	used
the	intelligence	of	our	brains	plus	one	other	enabling	factor,	an	opposable
appendage—the	thumb—to	manipulate	the	environment	to	build	tools.	These
tools	represented	a	new	form	of	evolution,	as	neurology	gave	rise	to	technology.

knowledge
.	Only
It	is	only	because	of	our	tools	that	our	knowledge	base	has	been	able	to	grow
without	limit.
Our	first	invention	was	the	story:	spoken	language	that	enabled	us	to
represent	ideas	with	distinct	utterances.	With	the	subsequent	invention	of	written
language	we	developed	distinct	shapes	to	symbolize	our	ideas.	Libraries	of

written	language	vastly	extended	the	ability	of	our	unaided	brains	to	retain	and
extend	our	knowledge	base	of	recursively	structured	ideas.
There	is	some	debate	as	to	whether	other	species,	such	as	chimpanzees,
have	the	ability	to	express	hierarchical	ideas	in	language.	Chimps	are	capable	of
learning	a	limited	set	of	sign	language	symbols,	which	they	can	use	to
communicate	with	human	trainers.	It	is	clear,	however,	that	there	are	distinct
limits	to	the	complexity	of	the	knowledge	structures	with	which	chimps	are
capable	of	dealing.	The	sentences	that	they	can	express	are	limited	to	specific
simple	noun-verb	sequences	and	are	not	capable	of	the	indefinite	expansion	of
complexity	characteristic	of	humans.	For	an	entertaining	example	of	the
complexity	of	human-generated	language,	just	read	one	of	the	spectacular
multipage-length	sentences	in	a	Gabriel	García	Márquez	story	or	novel—his	six-
page	story	“The	Last	Voyage	of	the	Ghost”	
is	a	single	sentence	and	works	quite

well	in	both	Spanish	and	the	English	translation.
1
The	primary	idea	in	my	three	previous	books	on	technology	(
The	Age	of
Intelligent	Machines
,	written	in	the	1980s	and	published	in	1989;	
The	Age	of
Spiritual	Machines
,	written	in	the	mid-to	late	1990s	and	published	in	1999;	and
The	Singularity	Is	Near
,	written	in	the	early	2000s	and	published	in	2005)	is	that
an	evolutionary	process	inherently	accelerates	(as	a	result	of	its	increasing	levels
of	abstraction)	and	that	its	products	grow	exponentially	in	complexity	and

1
The	Age	of
The	Age	of
capability.	I	call	this	phenomenon	the	law	of	accelerating	returns	(LOAR),	and	it
pertains	to	both	biological	and	technological	evolution.	The	most	dramatic
example	of	the	LOAR	is	the	remarkably	predictable	exponential	growth	in	the
capacity	and	price/performance	of	information	technologies.	The	evolutionary
process	of	technology	led	invariably	to	the	computer,	which	has	in	turn	enabled	a
vast	expansion	of	our	knowledge	base,	permitting	extensive	links	from	one	area

of	knowledge	to	another.	The	Web	is	itself	a	powerful	and	apt	example	of	the
ability	of	a	hierarchical	system	to	encompass	a	vast	array	of	knowledge	while
preserving	its	inherent	structure.	The	world	itself	is	inherently	hierarchical—
trees	contain	branches;	branches	contain	leaves;	leaves	contain	veins.	Buildings
contain	floors;	floors	contain	rooms;	rooms	contain	doorways,	windows,	walls,
and	floors.
We	have	also	developed	tools	that	are	now	enabling	us	to	understand	our

own	biology	in	precise	information	terms.	We	are	rapidly	reverse-engineering
the	information	processes	that	underlie	biology,	including	that	of	our	brains.	We
now	possess	the	object	code	of	life	in	the	form	of	the	human	genome,	an
achievement	that	was	itself	an	outstanding	example	of	exponential	growth,	in
that	the	amount	of	genetic	data	the	world	has	sequenced	has	approximately

doubled	every	year	for	the	past	twenty	years.
2
	We	now	have	the	ability	to
simulate	on	computers	how	sequences	of	base	pairs	give	rise	to	sequences	of
amino	acids	that	fold	up	into	three-dimensional	proteins,	from	which	all	of
biology	is	constructed.	The	complexity	of	proteins	for	which	we	can	simulate
protein	folding	has	been	
steadily	increasing	as	computational	resources	continue
to	grow	exponentially.
3
	We	can	also	simulate	how	proteins	interact	with	one
another	in	an	intricate	three-dimensional	dance	of	atomic	forces.	Our	growing
understanding	of	biology	is	one	important	facet	of	discovering	the	intelligent
secrets	that	evolution	has	bestowed	on	us	and	then	using	these	biologically
inspired	paradigms	to	create	ever	more	intelligent	technology.
There	is	now	a	grand	project	under	way	involving	many	thousands	of
scientists	and	engineers	working	to	understand	the	best	example	we	have	of	an
intelligent	process:	the	human	brain.	It	is	arguably	the	most	important	effort	in

the	history	of	the	human-machine	civilization.	In	
	I	made
the	case	that	one	corollary	of	the	law	of	accelerating	returns	is	that	other
intelligent	species	are	likely	not	to	exist.	To	summarize	the	argument,	if	they
existed	we	would	have	noticed	them,	given	the	relatively	brief	time	that	elapses
between	a	civilization’s	possessing	crude	technology	(consider	that	in	1850	the
fastest	way	to	send	nationwide	information	was	the	Pony	Express)	to	its
possessing	technology	that	can	transcend	its	own	planet.
4
	From	this	perspective,

	I	made
4
reverse-engineering	the	human	brain	may	be	regarded	as	the	most	important
project	in	the	universe.
The	goal	of	the	project	is	to	understand	precisely	how	the	human	brain
works,	and	then	to	use	these	revealed	methods	to	better	understand	ourselves,	to
fix	the	brain	when	needed,	and—most	relevant	to	the	subject	of	this	book—to
create	even	more	intelligent	machines.	Keep	in	mind	that	greatly	amplifying	a
natural	phenomenon	is	precisely	what	engineering	is	capable	of	doing.	As	an

4
example,	consider	the	rather	subtle	phenomenon	of	Bernoulli’s	principle,	which
states	that	there	is	slightly	less	air	pressure	over	a	moving	curved	surface	than
over	a	moving	flat	one.	The	mathematics	of	how	Bernoulli’s	principle	produces
wing	lift	is	still	not	yet	fully	settled	among	scientists,	yet	engineering	has	taken
this	delicate	insight,	focused	its	powers,	and	created	the	entire	world	of	aviation.
In	this	book	I	present	a	thesis	I	call	the	pattern	recognition	theory	of	mind

(PRTM),	which,	I	argue,	describes	the	basic	algorithm	of	the	
neocortex	(the
region	of	the	brain	responsible	for	perception,	memory,	and	critical	thinking).	In
the	chapters	ahead	I	describe	how	recent	neuroscience	research,	as	well	as	our
own	thought	experiments,	leads	to	the	inescapable	conclusion	that	this	method	is
used	consistently	across	the	neocortex.	The	implication	of	the	PRTM	combined

with	the	LOAR	is	that	we	will	be	able	to	engineer	these	principles	to	vastly
extend	the	powers	of	our	own	intelligence.
Indeed	this	process	is	already	well	under	way.	There	are	hundreds	of	tasks
and	activities	formerly	the	sole	province	of	human	intelligence	that	can	now	be
conducted	by	computers,	usually	with	greater	precision	and	at	a	vastly	greater
scale.	Every	time	you	send	an	e-mail	or	connect	a	cell	phone	call,	intelligent
algorithms	optimally	route	the	information.	Obtain	an	electrocardiogram,	and	it
comes	back	with	a	computer	diagnosis	that	rivals	that	of	doctors.	The	same	is
true	for	blood	cell	images.	Intelligent	algorithms	automatically	detect	credit	card
fraud,	fly	and	land	airplanes,	guide	intelligent	weapons	systems,	help	design
products	with	intelligent	computer-aided	design,	keep	track	of	just-in-time
inventory	levels,	assemble	products	in	robotic	factories,	and	play	games	such	as
chess	and	even	the	subtle	game	of	Go	at	master	levels.

Millions	of	people	witnessed	the	IBM	computer	named	Watson	play	the
natural-language	game	of	
Jeopardy!
	and	obtain	a	higher	score	than	the	best	two
human	players	in	the	world	combined.	It	should	be	noted	that	not	only	did
Watson	read	and	“understand”	the	subtle	language	in	the	
Jeopardy!
	query	(which
includes	such	phenomena	as	puns	and	metaphors),	but	it	obtained	the	knowledge
it	needed	to	come	up	with	a	response	from	understanding	hundreds	of	millions
of	pages	of	natural-language	documents	including	Wikipedia	and	other

Jeopardy!
Jeopardy!
encyclopedias	on	its	own.	It	needed	to	master	virtually	every	area	of	human
intellectual	endeavor,	including	history,	science,	literature,	the	arts,	culture,	and
more.	IBM	is	now	working	with	Nuance	Speech	Technologies	(formerly
Kurzweil	Computer	Products,	my	first	company)	on	a	new	version	of	Watson
that	will	read	medical	literature	(essentially	all	medical	journals	and	leading
medical	blogs)	to	become	
a	master	diagnostician	and	medical	consultant,	using
Nuance’s	clinical	language–understanding	technologies.	Some	observers	have

argued	that	Watson	does	not	really	“understand”	the	
Jeopardy!
	queries	or	the
encyclopedias	it	has	read	because	it	is	just	engaging	in	“statistical	analysis.”	A
key	point	I	will	describe	here	is	that	the	mathematical	techniques	that	have
evolved	in	the	field	of	artificial	intelligence	(such	as	those	used	in	Watson	and
Siri,	the	iPhone	assistant)	are	mathematically	very	similar	to	the	methods	that
biology	evolved	in	the	form	of	the	neocortex.	If	understanding	language	and
other	phenomena	through	statistical	analysis	does	not	count	as	true

Jeopardy!
understanding,	then	humans	have	no	understanding	either.
Watson’s	ability	to	intelligently	master	the	knowledge	in	natural-language
documents	is	coming	to	a	search	engine	near	you,	and	soon.	People	are	already
talking	to	their	phones	in	natural	language	(via	Siri,	for	example,	which	was	also
contributed	to	by	Nuance).	These	natural-language	assistants	will	rapidly

become	more	intelligent	as	they	utilize	more	of	the	Watson-like	methods	and	as
Watson	itself	continues	to	improve.
The	Google	self-driving	cars	have	logged	200,000	miles	in	the	busy	cities
and	towns	of	California	(a	figure	that	will	undoubtedly	be	much	higher	by	the
time	this	book	hits	the	real	and	virtual	shelves).	There	are	many	other	examples
of	artificial	intelligence	in	today’s	world,	and	a	great	deal	more	on	the	horizon.
As	further	examples	of	the	LOAR,	the	spatial	resolution	of	brain	scanning
and	the	amount	of	data	we	are	gathering	on	the	brain	are	doubling	every	year.
We	are	also	demonstrating	that	we	can	turn	this	data	into	working	models	and
simulations	of	brain	regions.	We	have	succeeded	in	reverse-engineering	key
functions	of	the	auditory	cortex,	where	we	process	information	about	sound;	the
visual	cortex,	where	we	process	information	from	our	sight;	and	the	cerebellum,
where	we	do	a	portion	of	our	skill	formation	(such	as	catching	a	fly	ball).

The	cutting	edge	of	the	project	to	understand,	model,	and	simulate	the
human	brain	is	to	reverse-engineer	the	cerebral	neocortex,	where	we	do	
our
recursive	hierarchical	thinking.	The	cerebral	cortex,	which	accounts	for	80
percent	of	the	human	brain,	is	composed	of	a	highly	repetitive	structure,
allowing	humans	to	create	arbitrarily	complex	structures	of	ideas.
In	the	pattern	recognition	theory	of	mind,	I	describe	a	model	of	how	the
human	brain	achieves	this	critical	capability	using	a	very	clever	structure

our
designed	by	biological	evolution.	There	are	details	in	this	cortical	mechanism
that	we	do	not	yet	fully	understand,	but	we	know	enough	about	the	functions	it
needs	to	perform	that	we	can	nonetheless	design	algorithms	that	accomplish	the
same	purpose.	By	beginning	to	understand	the	neocortex,	we	are	now	in	a
position	to	greatly	amplify	its	powers,	just	as	the	world	of	aviation	has	vastly
amplified	the	powers	of	Bernoulli’s	principle.	The	operating	principle	of	the
neocortex	is	arguably	the	most	important	idea	in	the	world,	as	it	is	capable	of

representing	all	knowledge	and	skills	as	well	as	creating	new	knowledge.	It	is
the	neocortex,	after	all,	that	has	been	responsible	for	every	novel,	every	song,
every	painting,	every	scientific	discovery,	and	the	multifarious	other	products	of
human	thought.
There	is	a	great	need	in	the	field	of	neuroscience	for	a	theory	that	ties
together	the	extremely	disparate	and	extensive	observations	that	are	being
reported	on	a	daily	basis.	A	unified	theory	is	a	crucial	requirement	in	every
major	area	of	science.	In	
chapter	1

chapter	1
	I’ll	describe	how	two	daydreamers	unified
biology	and	physics,	fields	that	had	previously	seemed	hopelessly	disordered
and	varied,	and	then	address	how	such	a	theory	can	be	applied	to	the	landscape
of	the	brain.
Today	we	often	encounter	great	celebrations	of	the	complexity	of	the

human	brain.	Google	returns	some	30	million	links	for	a	search	request	for
quotations	on	that	topic.	(It	is	impossible	to	translate	this	into	the	number	of
actual	quotations	it	is	returning,	however,	as	some	of	the	Web	sites	linked	have
multiple	quotes,	and	some	have	none.)	James	D.	Watson	himself	wrote	in	1992
that	“the	brain	is	the	last	and	grandest	biological	frontier,	the	most	complex	thing
we	have	yet	discovered	in	our	universe.”	He	goes	on	to	explain	why	he	believes
that	“it	contains	hundreds	of	billions	of	
cells	interlinked	through	trillions	of
connections.	The	brain	boggles	the	mind.”
5
I	agree	with	Watson’s	sentiment	about	the	brain’s	being	the	grandest
biological	frontier,	but	the	fact	that	it	contains	many	billions	of	cells	and	trillions
of	connections	does	not	necessarily	make	its	primary	method	complex	if	we	can
identify	readily	understandable	(and	recreatable)	patterns	in	those	cells	and
connections,	especially	massively	redundant	ones.

5
Let’s	think	about	what	it	means	to	be	complex.	We	might	ask,	is	a	forest
complex?	The	answer	depends	on	the	perspective	you	choose	to	take.	You	could
note	that	there	are	many	thousands	of	trees	in	the	forest	and	that	each	one	is
different.	You	could	then	go	on	to	note	that	each	tree	has	thousands	of	branches
and	that	each	branch	is	completely	different.	Then	you	could	proceed	to	describe
the	convoluted	vagaries	of	a	single	branch.	Your	conclusion	might	be	that	the
forest	has	a	complexity	beyond	our	wildest	imagination.

But	such	an	approach	would	literally	be	a	failure	to	see	the	forest	for	the
trees.	Certainly	there	is	a	great	deal	of	fractal	variation	among	trees	and
branches,	but	to	correctly	understand	the	principles	of	a	forest	you	would	do
better	to	start	by	identifying	the	distinct	patterns	of	redundancy	with	stochastic
(that	is,	random)	variation	that	are	found	there.	It	would	be	fair	to	say	that	the
concept	of	a	forest	is	simpler	than	the	concept	of	a	tree.
Thus	it	is	with	the	brain,	which	has	a	similar	enormous	redundancy,

especially	in	the	neocortex.	As	I	will	describe	in	this	book,	it	would	be	fair	to	say
that	there	is	more	complexity	in	a	single	neuron	than	in	the	overall	structure	of
the	neocortex.
My	goal	in	this	book	is	definitely	not	to	add	another	quotation	to	the
millions	that	already	exist	attesting	to	how	complex	the	brain	is,	but	rather	to
impress	you	with	the	power	of	its	simplicity.	I	will	do	so	by	describing	how	a
basic	ingenious	mechanism	for	recognizing,	remembering,	and	predicting	a
pattern,	repeated	in	the	neocortex	hundreds	of	millions	of	times,	accounts	for	the

great	diversity	of	our	thinking.	Just	as	an	astonishing	diversity	of	organisms
arises	from	the	different	combinations	of	the	values	
of	the	genetic	code	found	in
nuclear	and	mitochondrial	DNA,	so	too	does	an	astounding	array	of	ideas,
thoughts,	and	skills	form	based	on	the	values	of	the	patterns	(of	connections	and

synaptic	strengths)	found	in	and	between	our	neocortical	pattern	recognizers.	As
MIT	neuroscientist	Sebastian	Seung	says,	“Identity	lies	not	in	our	genes,	but	in
the	connections	between	our	brain	cells.”
6
We	need	to	distinguish	between	true	complexity	of	design	and	apparent
complexity.	Consider	the	famous	Mandelbrot	set,	the	image	of	which	has	long
been	a	symbol	of	complexity.	To	appreciate	its	apparent	complication,	it	is	useful
to	zoom	in	on	its	image	(which	you	can	access	via	the	links	in	this	endnote).
7
There	is	endless	intricacy	within	intricacy,	and	they	are	always	different.	Yet	the
design—the	formula—for	the	Mandelbrot	set	couldn’t	be	simpler.	It	is	six
characters	long:	
Z
	=	
Z
2
	+	
C
,	in	which	
Z
	is	a	“complex”	number	(meaning	a	pair
of	numbers)	and	
C
	is	a	constant.	It	is	
not	necessary	to	fully	understand	the
Mandelbrot	function	to	see	that	it	is	simple.	This	formula	is	applied	iteratively
and	at	every	level	of	a	hierarchy.	The	same	is	true	of	the	brain.	Its	repeating

7
Z
	=	
Z
2
	+	
C
,	in	which	
Z
C
structure	is	not	as	simple	as	that	of	the	six-character	formula	of	the	Mandelbrot
set,	but	it	is	not	nearly	as	complex	as	the	millions	of	quotations	on	the	brain’s
complexity	would	suggest.	This	neocortical	design	is	repeated	over	and	over	at
every	level	of	the	conceptual	hierarchy	represented	by	the	neocortex.	Einstein
articulated	my	goals	in	this	book	well	when	he	said	that	“any	intelligent	fool	can
make	things	bigger	and	more	complex…but	it	takes…a	lot	of	courage	to	move
in	the	opposite	direction.”

One	view	of	the	display	of	the	Mandelbrot	set,	a	simple	formula	that	is
iteratively	applied.	As	one	zooms	in	on	the	display,	the	images	constantly
change	in	apparently	complex	ways.
So	far	I	have	been	talking	about	the	brain.	But	what	about	the	mind?	For
example,	how	does	a	problem-solving	neocortex	attain	consciousness?	And
while	we’re	on	the	subject,	just	how	many	conscious	minds	do	we	have	in	our
brain?	There	is	evidence	that	suggests	there	may	be	more	than	one.
Another	pertinent	question	about	the	mind	is,	what	is	free	will,	and	do	we
have	it?	There	are	experiments	that	appear	to	show	that	we	start	implementing
our	decisions	before	we	are	even	aware	that	we	have	made	them.	Does	that
imply	that	free	will	is	an	illusion?
Finally,	what	attributes	of	our	brain	are	responsible	for	forming	our
identity?	Am	I	the	same	person	I	was	six	months	ago?	Clearly	I	am	not	exactly
the	same	as	I	was	then,	but	do	I	have	the	same	identity?
We’ll	review	what	the	pattern	recognition	theory	of	mind	implies	about

these	age-old	questions.

THOUGHT	EXPERIMENTS
ON	THE	WORLD
	
Darwin’s	theory	of	natural	selection	came	very	late	in	the	history	of
thought.
Was	it	delayed	because	it	opposed	revealed	truth,	because	it	was	an
entirely	new	subject	in	the	history	of	science,	because	it	was	characteristic
only	of	living	things,	or	because	it	dealt	with	purpose	and	final	causes
without	postulating	an	act	of	creation?	I	think	not.	Darwin	simply
discovered	the	role	of	selection,	a	kind	of	causality	very	different	from	the
push-pull	mechanisms	of	science	up	to	that	time.	The	origin	of	a	fantastic
variety	of	living	things	could	be	explained	by	the	contribution	of	which
novel	features,	possibly	of	random	provenance,	made	it	to	survival.	There
was	little	or	nothing	in	physical	or	biological	science	that	foreshadowed
selection	as	a	causal	principle.
—B.	F.	Skinner
	
Nothing	is	at	last	sacred	but	the	integrity	of	your	own	mind.
—Ralph	Waldo	Emerson

A	Metaphor	from	Geology
I
n	the	early	nineteenth	century	geologists	pondered	a	fundamental	question.
Great	caverns	and	canyons	such	as	the	Grand	Canyon	in	the	United	States	and
Vikos	Gorge	in	Greece	(reportedly	the	deepest	canyon	in	the	world)	existed	all
across	the	globe.	How	did	these	majestic	formations	get	there?
Invariably	there	was	a	stream	of	water	that	appeared	to	take	advantage	of
the	opportunity	to	course	through	these	natural	structures,	but	prior	to	the	mid-
nineteenth	century,	it	had	seemed	absurd	that	these	gentle	flows	could	be	the
creator	of	such	huge	valleys	and	cliffs.	British	geologist	Charles	Lyell	(1797–
1875),	however,	proposed	that	it	was	indeed	the	movement	of	water	that	had
carved	out	these	major	geological	modifications	over	great	periods	of	time,
essentially	one	grain	of	rock	at	a	time.	This	proposal	was	initially	met	with
ridicule,	but	within	two	decades	Lyell’s	thesis	achieved	mainstream	acceptance.
One	person	who	was	carefully	watching	the	response	of	the	scientific

community	to	Lyell’s	radical	thesis	was	English	naturalist	Charles	Darwin
(1809–1882).	Consider	the	situation	in	biology	around	1850.	The	field	was
endlessly	complex,	faced	with	countless	species	of	animals	and	plants,	any	one
of	which	presented	great	intricacy.	If	anything,	most	scientists	resisted	any
attempt	to	provide	a	unifying	theory	of	nature’s	dazzling	variation.	This	diversity
served	as	a	testament	to	the	glory	of	God’s	creation,	not	to	mention	to	the
intelligence	of	the	scientists	who	were	capable	of	mastering	it.

Darwin	approached	the	problem	of	devising	a	general	theory	of	species	by
making	an	analogy	with	Lyell’s	thesis	to	account	for	the	gradual	changes	in	the
features	of	species	over	many	generations.	He	combined	this	insight	with	his
own	thought	experiments	and	observations	in	his	famous	
Voyage	of	the	Beagle
.
Darwin	argued	that	in	each	generation	the	individuals	that	could	best	survive	in
their	ecological	niche	would	be	the	individuals	to	create	the	next	generation.
On	November	22,	1859,	Darwin’s	book	
On
	
the	Origin	of	Species
	went	on

.
On
	
	went	on
sale,	and	in	it	he	made	clear	his	debt	to	Lyell:
I	am	well	aware	that	this	doctrine	of	natural	selection,	exemplified	in
the	above	imaginary	instances,	is	open	to	the	same	objections	which	were	at
first	urged	against	Sir	Charles	Lyell’s	noble	views	on	“the	modern	changes
of	the	earth,	as	illustrative	of	geology”;	but	we	now	very	seldom	hear	the

action,	for	instance,	of	the	coast-waves	called	a	trifling	and	insignificant
cause,	when	applied	to	the	excavation	of	gigantic	valleys	or	to	the
formation	of	the	longest	lines	of	inland	cliffs.	Natural	selection	can	act	only
by	the	preservation	and	accumulation	of	infinitesimally	small	inherited
modifications,	each	profitable	to	the	preserved	being;	and	as	modern
geology	has	almost	banished	such	views	as	the	excavation	of	a	great	valley
by	a	single	diluvial	wave,	so	will	natural	selection,	if	it	be	a	true	principle,
banish	the	belief	of	the	continued	creation	of	new	organic	beings,	or	of	any
great	and	sudden	modification	in	their	structure.
1
Charles	Darwin,	author	of	
On	the	Origin	of	Species
,	which
established	the	idea	of	biological	evolution.
	
There	are	always	multiple	reasons	why	big	new	ideas	are	resisted,	and	it	is
not	hard	to	identify	them	in	Darwin’s	case.	That	we	were	descended	not	from
God	but	from	monkeys,	and	before	that,	worms,	did	not	sit	well	with	many

1
,	which
	
commentators.	The	implication	that	our	pet	dog	was	our	cousin,	as	was	the
caterpillar,	not	to	mention	the	plant	it	walked	on	(a	millionth	or	billionth	cousin,
perhaps,	but	still	related),	seemed	a	blasphemy	to	many.
But	the	idea	quickly	caught	on	because	it	brought	coherence	to	what	had
previously	been	a	plethora	of	apparently	unrelated	observations.	By	1872,	with
the	publication	of	the	sixth	edition	of	
,	Darwin	added
this	passage:	“As	a	record	of	a	former	state	of	things,	I	have	retained	in	the

foregoing	paragraphs…several	sentences	which	imply	that	naturalists	believe	in
the	separate	creation	of	each	species;	and	I	have	been	much	censured	for	having

thus	expressed	myself.	But	undoubtedly	this	was	the	general	belief	when	the	first
edition	of	the	present	work	appeared….	Now	things	are	wholly	changed,	and
almost	every	naturalist	admits	the	great	principle	of	evolution.”
2
Over	the	next	century	Darwin’s	unifying	idea	deepened.	In	1869,	only	a
decade	after	the	original	publication	of	
,	Swiss
physician	Friedrich	Miescher	(1844–1895)	discovered	a	substance	he	called
“nuclein”	in	the	cell	nucleus,	which	turned	out	to	be	DNA.
3
	In	1927	Russian
biologist	Nikolai	Koltsov	(1872–1940)	described	what	he	called	a	“giant
hereditary	molecule,”	which	he	said	was	composed	of	“two	mirror	strands	that
would	replicate	in	a	semi-conservative	fashion	using	each	strand	as	a	template.”
His	finding	was	also	condemned	by	many.	The	communists	considered	it	to	be
fascist	propaganda,	and	his	sudden,	unexpected	death	has	been	attributed	to	the
secret	police	of	the	Soviet	Union.
4
	In	1953,	nearly	a	century	after	the	publication

3
4
of	Darwin’s	seminal	book,	American	biologist	James	D.	Watson	(born	in	1928)
and	English	biologist	Francis	Crick	(1916–2004)	provided	the	first	accurate
characterization	of	the	structure	of	DNA,	describing	it	as	a	double	helix	of	two
long	twisting	molecules.
5
	It	is	worth	pointing	out	that	their	finding	was	based	on
what	is	now	known	as	“photo	51,”	taken	by	their	colleague	Rosalind	Franklin
using	
X-ray	crystallography,	which	was	the	first	representation	that	showed	the

5
using	
double	helix.	Given	the	insights	derived	from	Franklin’s	image,	there	have	been
suggestions	that	she	should	have	shared	in	Watson	and	Crick’s	Nobel	Prize.
6

Rosalind	Franklin	took	the	critical	picture	of	DNA	(using	X-ray
crystallography)	that	enabled	Watson	and	Crick	to	accurately	describe	the
structure	of	DNA	for	the	first	time.
With	the	description	of	a	molecule	that	could	code	the	program	of	biology,
a	unifying	theory	of	biology	was	now	firmly	in	place.	It	provided	a	simple	and
elegant	foundation	to	all	of	life.	Depending	only	on	the	values	of	the	base	pairs
that	make	up	the	DNA	strands	in	the	nucleus	(and	to	a	lesser	degree	the
mitochondria),	an	organism	would	mature	into	a	blade	of	grass	or	a	human
being.	This	insight	did	not	eliminate	the	delightful	diversity	of	nature,	but	we
now	understand	that	the	extraordinary	diversity	of	nature	stems	from	the	great
assortment	of	structures	that	can	be	coded	on	this	universal	molecule.

Riding	on	a	Light	Beam
At	the	beginning	of	the	twentieth	century	the	world	of	physics	was	upended
through	another	series	of	thought	experiments.	In	1879	a	boy	was	born	to	a
German	engineer	and	a	housewife.	He	didn’t	start	to	talk	until	the	age	of	three
and	was	reported	to	have	had	problems	in	school	at	the	age	of	nine.	At	sixteen	he
was	daydreaming	about	riding	on	a	moonbeam.
This	young	boy	was	aware	of	English	mathematician	Thomas	Young’s
(1773–1829)	experiment	in	1803	that	established	that	light	is	composed	of
waves.	The	conclusion	at	that	time	was	that	light	waves	must	be	traveling
through	some	sort	of	medium;	after	all,	ocean	waves	traveled	through	water	and
sound	waves	traveled	through	air	and	other	materials.	Scientists	called	the
medium	through	which	light	waves	travel	the	“ether.”	The	boy	was	also	aware	of
the	1887	experiment	by	American	scientists	Albert	Michelson	(1852–1931)	and
Edward	Morley	(1838–1923)	that	attempted	to	confirm	the	existence	of	the

ether.	That	experiment	was	based	on	the	analogy	of	traveling	in	a	rowboat	up-
and	downstream	in	a	river.	If	you	are	paddling	at	a	fixed	speed,	then	your	speed
as	measured	from	the	shore	will	be	faster	if	you	are	paddling	with	the	stream	as
opposed	to	going	against	it.	Michelson	and	Morley	assumed	that	light	would
travel	through	the	ether	at	a	constant	speed	(that	is,	at	the	speed	of	light).	They
reasoned	that	the	speed	of	sunlight	when	Earth	is	traveling	toward	the	sun	in	its
orbit	(as	measured	from	our	vantage	point	on	Earth)	versus	its	apparent	speed

when	Earth	is	traveling	away	from	the	sun	must	be	different	(by	twice	the	speed
of	Earth).	Proving	that	would	confirm	the	existence	of	the	ether.	However,	what
they	discovered	was	that	there	was	no	difference	in	the	speed	of	the	sunlight
passing	Earth	regardless	of	where	Earth	was	in	its	orbit.	Their	findings	disproved
the	idea	of	the	“ether,”	but	what	was	really	going	on?	This	remained	a	mystery
for	almost	two	decades.
As	this	German	teenager	imagined	riding	alongside	a	light	wave,	he

reasoned	that	he	should	be	seeing	the	light	waves	frozen,	in	the	same	way	that	a
train	would	appear	not	to	be	moving	if	you	rode	alongside	it	at	the	
same	speed	as
the	train.	Yet	he	realized	that	this	was	impossible,	because	the	speed	of	light	is
supposed	to	be	constant	regardless	of	your	own	movement.	So	he	imagined
instead	riding	alongside	the	light	beam	but	at	a	somewhat	slower	speed.	What	if
he	traveled	at	90	percent	of	the	speed	of	light?	If	light	beams	are	like	trains,	he

reasoned,	then	he	should	see	the	light	beam	traveling	ahead	of	him	at	10	percent

of	the	speed	of	light.	Indeed,	that	would	have	to	be	what	observers	on	Earth
would	see.	But	we	know	that	the	speed	of	light	is	a	constant,	as	the	Michelson-
Morley	experiment	had	shown.	Thus	he	would	necessarily	see	the	light	beam
traveling	ahead	of	him	at	the	full	speed	of	light.	This	seemed	like	a	contradiction
—how	could	it	be	possible?
The	answer	became	evident	to	the	German	boy,	whose	name,	incidentally,
was	Albert	Einstein	(1879–1955),	by	the	time	he	turned	twenty-six.	Obviously—
to	young	Master	Einstein—
time	itself	must	have	slowed	down	for	him
.	He
explains	his	reasoning	in	a	paper	published	in	1905.
7
	If	observers	on	Earth	were
to	look	at	the	young	man’s	watch	they	would	see	it	ticking	ten	times	slower.
Indeed,	when	he	got	back	to	Earth,	his	watch	would	show	that	only	10	percent	as
much	time	had	passed	(ignoring,	for	the	moment,	acceleration	and	deceleration).
From	his	perspective,	however,	his	watch	was	ticking	normally	and	the	light

.	He
7
beam	next	to	him	was	traveling	at	the	speed	of	light.	The	ten-times	slowdown	in
the	speed	of	time	itself	(relative	to	clocks	on	Earth)	fully	explains	the	apparent
discrepancies	in	perspective.	In	the	extreme,	the	slowdown	in	the	passage	of
time	would	reach	zero	once	the	speed	of	travel	reached	the	speed	of	light;	hence
it	was	impossible	to	ride	along	with	the	light	beam.	Although	it	was	impossible
to	travel	at	the	speed	of	light,	it	turned	out	not	to	be	theoretically	impossible	to
move	
faster
	than	the	light	beam.	Time	would	then	move	backward.

move	
faster
This	resolution	seemed	absurd	to	many	early	critics.	How	could	time	itself
slow	down,	based	only	on	someone’s	speed	of	movement?	Indeed,	for	eighteen
years	(from	the	time	of	the	Michelson-Morley	experiment),	other	thinkers	had
been	unable	to	see	a	conclusion	that	was	so	obvious	to	Master	Einstein.	The
many	others	who	had	considered	this	problem	through	the	
latter	part	of	the
nineteenth	century	had	essentially	“fallen	off	the	horse”	in	terms	of	following
through	on	the	implications	of	a	principle,	sticking	instead	to	their	preconceived

notions	of	how	reality	must	work.	(I	should	probably	change	that	metaphor	to
“fallen	off	the	light	beam.”)
Einstein’s	second	mind	experiment	was	to	consider	himself	and	his	brother
flying	through	space.	They	are	186,000	miles	apart.	Einstein	wants	to	move
faster	but	he	also	desires	to	keep	the	distance	between	them	the	same.	So	he
signals	his	brother	with	a	flashlight	each	time	he	wants	to	accelerate.	Since	he
knows	that	it	will	take	one	second	for	the	signal	to	reach	his	brother,	he	waits	a

second	(after	sending	the	signal)	to	initiate	his	own	acceleration.	Each	time	the
brother	receives	the	signal	he	immediately	accelerates.	In	this	way	the	two
brothers	accelerate	at	exactly	the	same	time	and	therefore	remain	a	constant
distance	apart.
But	now	consider	what	we	would	see	if	we	were	standing	on	Earth.	If	the

brothers	were	moving	away	from	us	(with	Albert	in	the	lead),	it	would	appear	to
take	less	than	a	second	for	the	light	to	reach	the	brother,	because	he	is	traveling
toward	the	light.	Also	we	would	see	Albert’s	brother’s	clock	as	slowing	down
(as	his	speed	increases	as	he	is	closer	to	us).	For	both	of	these	reasons	we	would
see	the	two	brothers	getting	closer	and	closer	and	eventually	colliding.	Yet	from
the	perspective	of	the	two	brothers,	they	remain	a	constant	186,000	miles	apart.
How	can	this	be?	The	answer—
obviously
—is	that	distances	contract
parallel	to	the	motion	(but	not	perpendicular	to	it).	So	the	two	Einstein	brothers
are	getting	shorter	(assuming	they	are	flying	headfirst)	as	they	get	faster.	This
bizarre	conclusion	probably	lost	Einstein	more	early	fans	than	the	difference	in
the	passage	of	time.
During	the	same	year,	Einstein	considered	the	relationship	of	matter	and
energy	with	yet	another	mind	experiment.	Scottish	physicist	James	Clerk

obviously
Maxwell	had	shown	in	the	1850s	that	particles	of	light	called	photons	had	no
mass	but	nonetheless	carried	momentum.	As	a	child	I	had	a	device	called	a
Crookes	radiometer,
8
	which	consisted	of	an	airtight	glass	bulb	that	contained	a
partial	vacuum	and	a	set	of	four	vanes	that	rotated	on	a	
spindle.	The	vanes	were
white	on	one	side	and	black	on	the	other.	The	white	side	of	each	vane	reflected
light,	and	the	black	side	absorbed	light.	(That’s	why	it	is	cooler	to	wear	a	white

8
T-shirt	on	a	hot	day	than	a	black	one.)	When	a	light	was	shined	on	the	device,
the	vanes	rotated,	with	the	dark	sides	moving	away	from	the	light.	This	is	a
direct	demonstration	that	photons	carry	enough	momentum	to	actually	cause	the
vanes	of	the	radiometer	to	move.
9
The	issue	that	Einstein	struggled	with	is	that	momentum	is	a	function	of
mass:	Momentum	is	equal	to	mass	times	velocity.	Thus	a	locomotive	traveling	at
30	miles	per	hour	has	a	lot	more	momentum	than,	say,	an	insect	traveling	at	the

9
same	speed.	How,	then,	could	there	be	positive	momentum	for	a	particle	with
zero	mass?
Einstein’s	mind	experiment	consisted	of	a	box	floating	in	space.	A	
photon	is
emitted	inside	the	box	from	the	left	toward	the	right	side.	The	total	momentum
of	the	system	needs	to	be	conserved,	so	the	box	would	have	to	recoil	to	the	left
when	the	photon	was	emitted.	After	a	certain	amount	of	time,	the	photon	collides
with	the	right	side	of	the	box,	transferring	its	momentum	back	to	the	box.	The

zero	mass?
photon	is
total	momentum	of	the	system	is	again	conserved,	so	the	box	now	stops	moving.

A	Crookes	radiometer—the	vane	with	four	wings	rotates	when	light
shines	on	it.
So	far	so	good.	But	consider	the	perspective	from	the	vantage	point	of	Mr.
Einstein,	who	is	watching	the	box	from	the	outside.	He	does	not	see	any	outside
influence	on	the	box:	No	particles—with	or	without	mass—hit	it,	and	nothing
leaves	it.	Yet	Mr.	Einstein,	according	to	the	scenario	above,	sees	the	box	move
temporarily	to	the	left	and	then	stop.	According	to	our	analysis,	each	photon
should	permanently	move	the	box	to	the	left.	Since	there	have	been	no	external
effects	on	the	box	or	from	the	box,	its	center	of	mass	must	remain	in	the	same
place.	Yet	the	photon	inside	the	box,	which	moves	from	left	to	right,	cannot
change	the	center	of	mass,	because	it	has	no	mass.
Or	does	it?	Einstein’s	conclusion	was	that	since	the	photon	clearly	has
energy,	and	has	momentum,	it	must	also	have	a	mass	equivalent.	The	energy	of
the	moving	photon	is	entirely	equivalent	to	a	moving	mass.	We	can	compute

what	that	equivalence	is	by	recognizing	that	the	center	of	mass	of	the	system
must	remain	stationary	during	the	movement	of	the	photon.	Working	out	the
math,	Einstein	showed	that	mass	and	energy	are	equivalent	and	are	related	by	a
simple	constant.	However,	there	was	a	catch:	The	constant	might	be	simple,	but
it	turned	out	to	be	enormous;	it	was	the	speed	of	light	squared	(about	1.7	×	10
17
meters
2
	per	second
2
—that	is,	17	followed	by	16	zeroes).	Hence	we	get	Einstein’s
famous	
E
	=	
mc
2
.
10

17
meters
2
	per	second
2
famous	
E
	=	
mc
2
.
10
	Thus	one	ounce	(28	grams)	of	mass	is	equivalent	to	600,000
tons	of	TNT.	Einstein’s	letter	of	August	2,	1939,	to	President	Roosevelt
informing	him	of	the	potential	for	an	atomic	bomb	based	on	this	formula	ushered

in	the	atomic	age.
11
You	might	think	that	this	should	have	been	obvious	earlier,	given	that
experimenters	had	noticed	that	the	mass	of	radioactive	substances	decreased	as	a
result	of	radiation	over	time.	It	was	assumed,	however,	that	
radioactive
substances	contained	a	special	high-energy	fuel	of	some	sort	that	was	burning
off.	That	assumption	is	not	all	wrong;	it’s	just	that	the	fuel	that	was	being
“burned	off”	was	simply	mass.
There	are	several	reasons	why	I	have	opened	this	book	with	Darwin’s	and
Einstein’s	mind	experiments.	First	of	all,	they	show	the	extraordinary	power	of
the	human	brain.	Without	any	equipment	at	all	other	than	a	pen	and	paper	to
draw	the	stick	figures	in	these	simple	mind	experiments	and	to	write	down	the
fairly	simple	equations	that	result	from	them,	Einstein	was	able	to	overthrow	the
understanding	of	the	physical	world	that	dated	back	two	centuries,	deeply
influence	the	course	of	history	(including	World	War	II),	and	usher	in	the	nuclear
age.

age.
It	is	true	that	Einstein	relied	on	a	few	experimental	findings	of	the
nineteenth	century,	although	these	experiments	also	did	not	use	sophisticated
equipment.	It	is	also	true	that	subsequent	experimental	validation	of	Einstein’s
theories	has	used	advanced	technologies,	and	if	these	had	not	been	developed	we
would	not	have	the	validation	that	we	possess	today	that	Einstein’s	ideas	are
authentic	and	significant.	However,	such	factors	do	not	detract	from	the	fact	that
these	famous	thought	experiments	reveal	the	power	of	human	thinking	at	its

finest.
Einstein	is	widely	regarded	as	the	leading	scientist	of	the	twentieth	century
(and	Darwin	would	be	a	good	contender	for	that	honor	in	the	nineteenth
century),	yet	the	mathematics	underlying	his	theories	is	ultimately	not	very
complicated.	The	thought	experiments	themselves	were	straightforward.	We
might	wonder,	then,	in	what	respect	could	Einstein	be	considered	particularly
smart.	We’ll	discuss	later	exactly	what	it	was	that	he	was	doing	with	his	brain
when	he	came	up	with	his	theories,	and	where	that	quality	resides.

Conversely,	this	history	also	demonstrates	the	limitations	of	human
thinking.	Einstein	was	able	to	ride	his	light	beam	without	falling	off	(albeit	he
concluded	that	it	was	impossible	to	actually	ride	a	light	beam),	but	how	many
thousands	of	other	observers	and	thinkers	were	completely	unable	to	think
through	these	remarkably	uncomplicated	exercises?	One	
common	failure	is	the
difficulty	that	most	people	have	in	discarding	and	transcending	the	ideas	and
perspectives	of	their	peers.	There	are	other	inadequacies	as	well,	which	we	will

discuss	in	more	detail	after	we	have	examined	how	the	neocortex	works.

A	Unified	Model	of	the	Neocortex
The	most	important	reason	I	am	sharing	what	are	perhaps	the	most	famous
thought	experiments	in	history	is	as	an	introduction	to	using	the	same	approach
with	respect	to	the	brain.	As	you	will	see,	we	can	get	remarkably	far	in	figuring
out	how	human	intelligence	works	through	some	simple	mind	experiments	of
our	own.	Considering	the	subject	matter	involved,	mind	experiments	should	be	a
very	appropriate	approach.
If	a	young	man’s	idle	thoughts	and	the	use	of	no	equipment	other	than	pen
and	paper	were	sufficient	to	revolutionize	our	understanding	of	physics,	then	we
should	be	able	to	make	reasonable	progress	with	a	phenomenon	with	which	we
are	much	more	familiar.	After	all,	we	experience	our	thinking	every	moment	of
our	waking	lives—and	our	dreaming	lives	as	well.
After	we	construct	a	model	of	how	thinking	works	through	this	process	of
self-reflection,	we’ll	examine	to	what	extent	we	can	confirm	it	through	the	latest

observations	of	actual	brains	and	the	state	of	the	art	in	re-creating	these
processes	in	machines.

ON	THINKING
	
I	very	rarely	think	in	words	at	all.	A	thought	comes,	and	I	may	try	to
express	it	in	words	afterwards.
—Albert	Einstein
	
The	brain	is	a	three-pound	mass	you	can	hold	in	your	hand	that	can
conceive	of	a	universe	a	hundred	billion	light	years	across.
—Marian	Diamond
	
What	seems	astonishing	is	that	a	mere	three-pound	object,	made	of	the
same	atoms	that	constitute	everything	else	under	the	sun,	is	capable	of
directing	virtually	everything	that	humans	have	done:	flying	to	the	moon
and	hitting	seventy	home	runs,	writing	
Hamlet
	and	building	the	Taj	Mahal
—even	unlocking	the	secrets	of	the	brain	itself.
—Joel	Havemann
	
I
	started	thinking	about	thinking	around	1960,	the	same	year	that	I	discovered
the	computer.	You	would	be	hard	pressed	today	to	find	a	twelve-year-old	who
does	not	use	a	computer,	but	back	then	there	were	only	a	handful	of	them	in	my
hometown	of	New	York	City.	Of	course	these	early	
devices	did	not	fit	in	your

Hamlet
	
I
hand,	and	the	first	one	I	got	access	to	took	up	a	large	room.	In	the	early	1960s	I
did	some	programming	on	an	IBM	1620	to	do	analyses	of	variance	(a	statistical
test)	on	data	that	had	been	collected	by	studying	a	program	for	early	childhood
education,	a	forerunner	to	Head	Start.	Hence	there	was	considerable	drama
involved	in	the	effort,	as	the	fate	of	this	national	educational	initiative	rode	on
our	work.	The	algorithms	and	data	being	analyzed	were	sufficiently	complex

that	we	were	not	able	to	anticipate	what	answers	the	computer	would	come	up
with.	The	answers	were,	of	course,	determined	by	the	data,	but	they	were	not
predictable.	It	turns	out	that	the	distinction	between	being	
determined
	and	being
predictable
	is	an	important	one,	to	which	I	will	return.
I	remember	how	exciting	it	was	when	the	front-panel	lights	dimmed	right
before	the	algorithm	finished	its	deliberations,	as	if	the	computer	were	deep	in
thought.	When	people	came	by,	eager	to	get	the	next	set	of	results,	I	would	point
to	the	gently	flashing	lights	and	say,	“It’s	thinking.”	This	both	was	and	wasn’t	a
joke—it	really	
did
	seem	to	be	contemplating	the	answers—and	staff	members
started	to	ascribe	a	personality	to	the	machine.	It	was	an	anthropomorphization,
perhaps,	but	it	did	get	me	to	begin	to	consider	in	earnest	the	relationship	between
thinking	and	computing.
In	order	to	assess	the	extent	to	which	my	own	brain	is	similar	to	the
computer	programs	I	was	familiar	with,	I	began	to	think	about	what	my	brain

did
must	be	doing	as	it	processed	information.	I	have	continued	this	investigation	for
fifty	years.	What	I	will	describe	below	about	our	current	understanding	of	how
the	brain	works	will	sound	very	different	from	the	standard	concept	of	a
computer.	Fundamentally,	however,	the	brain	does	store	and	process
information,	and	because	of	the	universality	of	computation—a	concept	to	which
I	will	also	return—there	is	more	of	a	parallel	between	brains	and	computers	than
may	be	apparent.

Each	time	I	do	something—or	think	of	something—whether	it	is	brushing
my	teeth,	walking	across	the	kitchen,	contemplating	a	business	problem,
practicing	on	a	music	keyboard,	or	coming	up	with	a	new	idea,	I	
reflect	on	how	I
was	able	to	accomplish	it.	I	think	even	more	about	all	of	the	things	that	I	am	not
able	to	do,	as	the	limitations	of	human	thought	provide	an	equally	important	set
of	clues.	Thinking	so	much	about	thinking	might	very	well	be	slowing	me	down,

but	I	have	been	hopeful	that	such	exercises	in	self-reflection	will	enable	me	to
refine	my	mental	methods.
To	raise	our	own	awareness	of	how	our	brains	work,	let’s	consider	a	series
of	mind	experiments.
Try	this:	
Recite	the	alphabet
.
You	probably	remember	this	from	childhood	and	can	do	it	easily.
Okay,	now	try	this:	
Recite	the	alphabet	backward
.
Unless	you	have	studied	the	alphabet	in	this	order,	you	are	likely	to	find	it
impossible	to	do.	On	occasion	someone	who	has	spent	a	significant	amount	of

Try	this:	
.
.
time	in	an	elementary	school	classroom	where	the	alphabet	is	displayed	will	be
able	to	call	up	his	visual	memory	and	then	read	it	backward	from	that.	Even	this
is	difficult,	though,	because	we	do	not	actually	remember	whole	images.

Reciting	the	alphabet	backward	should	be	a	simple	task,	as	it	involves	exactly
the	same	information	as	reciting	it	forward,	yet	we	are	generally	unable	to	do	it.
Do	you	remember	your	social	security	number?	If	you	do,	can	you	recite	it
backward	without	first	writing	it	down?	How	about	the	nursery	rhyme	“Mary
Had	a	Little	Lamb”?	Computers	can	do	this	trivially.	Yet	we	fail	at	it	unless	we
specifically	learn	the	backward	sequence	as	a	new	series.	This	tells	us	something
important	about	how	human	memory	is	organized.
Of	course,	we	are	able	to	perform	this	task	easily	if	we	write	down	the
sequence	and	then	read	it	backward.	In	doing	so	we	are	using	a	technology—
written	language—to	compensate	for	one	of	the	limitations	of	our	unaided
thinking,	albeit	a	very	early	tool.	(It	was	our	second	invention,	with	spoken
language	as	the	first.)	This	is	why	we	invent	tools—to	compensate	for	our
shortcomings.
This	suggests	that	
our	memories	are	sequential	and	in	order.	They	can

be	accessed	in	the	order	that	they	are	remembered.	We	are	unable	to
directly	reverse	the	sequence	of	a	memory.
We	also	have	some	difficulty	starting	a	memory	in	the	middle	of	a
sequence.	If	I	learn	to	play	a	piece	of	music	on	the	piano,	I	generally	can’t	just
begin	it	at	an	arbitrary	point	in	its	middle.	There	are	a	few	points	at	which	I	can
jump	in,	because	my	sequential	memory	of	the	piece	is	organized	in	segments.	If
I	try	to	start	in	the	middle	of	a	segment,	though,	I	need	to	revert	to	sight-reading

until	my	sequential	memory	kicks	in.
Next,	try	this:	
Recall	a	walk	that	you	took	in	the	last	day	or	so.	What	do	you
remember	about	it?
This	mind	experiment	works	best	if	you	took	a	walk	very	recently,	such	as
earlier	today	or	yesterday.	(You	can	also	substitute	a	drive,	or	basically	any
activity	during	which	you	moved	across	some	terrain.)
It	is	likely	that	you	don’t	remember	much	about	the	experience.	Who	was
the	fifth	person	you	encountered	(not	just	including	people	you	know)?	Did	you

see	an	oak	tree?	A	mailbox?	What	did	you	see	when	you	turned	the	first	corner?
If	you	passed	some	stores,	what	was	in	the	second	window?	Perhaps	you	can
reconstruct	the	answers	to	some	of	these	questions	from	the	few	clues	that	you
do	remember,	but	it	is	likely	that	you	remember	relatively	few	details,	even
though	this	is	a	very	recent	experience.
If	you	take	walks	regularly,	think	back	to	the	first	walk	you	took	last	month
(or	to	the	first	trip	to	the	office	last	month,	if	you	commute).	You	probably

cannot	recall	the	specific	walk	or	commute	at	all,	and	if	you	do,	you	doubtless
recall	even	fewer	details	about	it	than	about	your	walk	today.
I	will	later	discuss	the	issue	of	consciousness	and	make	the	point	that	we

tend	to	equate	consciousness	with	our	memory	of	events.	The	primary	reason	we
believe	that	we	are	not	conscious	when	under	anesthesia	is	that	we	don’t
remember	anything	from	that	period	(albeit	there	are	intriguing—and	disturbing
—exceptions	to	this).	So	with	regard	to	the	walk	I	took	this	morning,	was	I	not
conscious	during	most	of	it?	It’s	a	reasonable	question,	given	that	I	remember
almost	nothing	about	what	I	saw	or	even	what	I	was	thinking	about.
There	happen	to	be	a	few	things	I	do	remember	from	my	walk	this	morning.
I	recall	thinking	about	this	book,	but	I	couldn’t	tell	you	exactly	
what	those
thoughts	were.	I	also	recall	passing	a	woman	pushing	a	baby	carriage.	I
remember	that	the	woman	was	attractive,	and	that	the	baby	was	cute	as	well.	I
recall	two	thoughts	I	had	in	connection	with	this	experience:	
This	baby	is
adorable,	like	my	new	grandson
,	and	
What	is	this	baby	perceiving	in	her	visual
surroundings?
	I	cannot	recall	what	either	of	them	was	wearing	or	the	color	of

what	those
,	and	
their	hair.	(My	wife	will	tell	you	that	that	is	typical.)	Although	I	am	unable	to
describe	anything	specific	about	their	appearance,	I	do	have	some	ineffable
sense	of	what	the	mom	looked	like	and	believe	I	could	pick	out	her	picture	from
among	those	of	several	different	women.	So	while	there	must	be	something
about	her	appearance	that	I	have	retained	in	my	memory,	if	I	think	about	the
woman,	baby	carriage,	and	baby,	I	am	unable	to	visualize	them.	There	is	no
photograph	or	video	of	this	event	in	my	mind.	It	is	hard	to	describe	exactly	what
is

is
	in	my	mind	about	this	experience.
I	also	recall	having	passed	a	different	woman	with	a	baby	carriage	on	a
walk	a	few	weeks	earlier.	In	that	case	I	don’t	believe	I	could	even	recognize	that
woman’s	picture.	That	memory	is	now	much	dimmer	than	it	must	have	been
shortly	after	that	walk.
Next
,	
think	about	people	whom	you	have	encountered	only	once	or	twice
.
Can	you	visualize	them	clearly?	If	you	are	a	visual	artist,	then	you	may	have
learned	this	observational	skill,	but	typically	we	are	unable	to	visualize	people

Next
,	
.
we’ve	only	casually	come	across	to	draw	or	describe	them	sufficiently	but	would
have	little	difficulty	in	recognizing	a	picture	of	them.
there	are	no	images,	videos,	or	sound	recordings
stored	in	the	brain.	Our	memories	are	stored	as	sequences	of	patterns.
Memories	that	are	not	accessed	dim	over	time.
	When	police	sketch	artists
interview	a	crime	victim,	they	do	not	ask,	“What	did	the	perpetrator’s	eyebrows
look	like?”	Rather,	they	will	show	a	series	of	images	of	eyebrows	and	ask	the

victim	to	select	one.	The	correct	set	of	eyebrows	will	trigger	the	recognition	of
the	same	pattern	that	is	stored	in	the	victim’s	memory.
Let’s	now	consider	faces	that	you	know	well.	
Can	you	recognize	any	of
these	people?

You	are	undoubtedly	able	to	recognize	these	familiar	personalities,	even
though	they	are	partially	covered	or	distorted.	This	represents	a	key	strength	of
human	perception:	
We	can	recognize	a	pattern	even	if	only	part	of	it	is
perceived	(seen,	heard,	felt)	and	even	if	it	contains	alterations.	Our
recognition	ability	is	apparently	able	to	detect	invariant	features	of	a
pattern—characteristics	that	survive	real-world	variations.
	The	apparent
distortions	in	a	caricature	or	in	certain	forms	of	art	such	as	impressionism
emphasize	the	patterns	of	an	image	(person,	object)	that	we	recognize	while
changing	other	details.	The	world	of	art	is	actually	ahead	of	the	world	of	science
in	appreciating	the	power	of	the	human	perceptual	system.	We	use	the	same
approach	when	we	recognize	a	melody	from	only	a	few	notes.
Now
	
consider	this	image:
The	image	is	ambiguous—the	corner	indicated	by	the	black	region	may	be
an	inside	corner	or	an	outside	corner.	At	first	you	are	likely	to	perceive	it	one

Now
	
way	or	the	other,	though	with	some	effort	you	can	change	your	perception	to	the
alternate	interpretation.	Once	your	mind	has	fixed	on	an	understanding,	however,
it	may	be	difficult	to	see	the	other	perspective.	(This	turns	out	to	be	true	of
intellectual	perspectives	as	well.)	Your	brain’s	interpretation	of	the	image

actually	influences	your	experience	of	it.	When	the	corner	appears	to	be	an
inside	one,	your	brain	will	interpret	the	grey	region	as	a	shadow,	so	it	does	not
seem	to	be	as	dark	as	when	you	interpret	the	corner	as	being	an	outside	one.
Thus	
our	conscious	experience	of	our	perceptions	is	actually	changed	by
our	interpretations.
Consider	that	we	see	what	we	expect	to	___
I’m	confident	that	you	were	able	to	complete	the	above	sentence.
Had	I	written	out	the	last	word,	you	would	have	needed	only	to	glance	at	it
momentarily	to	confirm	that	it	was	what	you	had	expected.
This	implies	that	
we	are	constantly	predicting	the	future	and
hypothesizing	what	we	will	experience.	This	expectation	influences	what	we
actually	perceive.
	Predicting	the	future	is	actually	the	primary	reason	that	we
have	a	brain.
Consider	an	experience	that	we	all	have	on	a	regular	basis:	A	memory	from
years	ago	inexplicably	pops	into	your	head
.
Often	this	will	be	a	memory	of	a	person	or	an	event	that	you	haven’t

.
thought	about	for	a	long	time.	It	is	evident	that	something	has	triggered	the
memory.	The	train	of	thought	that	did	so	may	be	apparent	and	something	you	are
able	to	articulate.	At	other	times	you	may	be	aware	of	the	sequence	of	thoughts
that	led	to	the	memory	but	would	have	a	hard	time	expressing	it.	Often	the
trigger	is	quickly	lost,	so	the	memory	appears	to	have	come	from	nowhere.	I
often	experience	these	random	memories	while	doing	routine	procedures	such	as
brushing	my	teeth.	Sometimes	I	may	be	aware	of	the	connection—the	toothpaste

falling	off	the	toothbrush	might	remind	
me	of	the	paint	falling	off	a	brush	in	a
painting	class	I	took	in	college.	Sometimes	I	have	only	a	vague	sense	of	the
connection,	or	none	at	all.
A	related	phenomenon	that	everyone	experiences	frequently	is	trying	to
think	of	a	name	or	a	word.	The	procedure	we	use	in	this	circumstance	is	to	try	to
remind	ourselves	of	triggers	that	may	unlock	the	memory.	(For	example:	
Who
played	Queen	Padmé	in
	Revenge	of	the	Sith?	
Let’s	see,	it’s	that	same	actress
who	was	the	star	in	a	recent	dark	movie	about	dancing,	that	was

Who
	Black	Swan,	
oh
yes,	Natalie	Portman
.)	Sometimes	we	adopt	idiosyncratic	mnemonics	to	help	us
remember.	(For	example:	
She’s	always	slim,	not	portly,	oh	yes,	Portman,	Natalie
Portman
.)	Some	of	our	memories	are	sufficiently	robust	that	we	can	go	directly
from	a	question	(such	as	
who	played	Queen	Padmé
)	to	the	answer;	often	we
need	to	go	through	a	series	of	triggers	until	we	find	one	that	works.	It’s	very
much	like	having	the	right	Web	link.	Memories	can	indeed	become	lost	like	a

Black	Swan,	
oh
Portman
Web	page	to	which	no	other	page	links	to	(at	least	no	page	that	we	can	find).
While	executing	routine	procedures—such	as	putting	on	a	shirt—watch

yourself	performing	them,	and	consider	the	extent	to	which	you	follow	the	same
sequence	of	steps	each	time
.	From	my	own	observation	(and	as	I	mentioned,	I
am	constantly	trying	to	observe	myself),	it	is	likely	that	you	follow	very	much
the	same	steps	each	time	you	perform	a	particular	routine	task,	though	there	may
be	additional	modules	added.	For	example,	most	of	my	shirts	do	not	require	cuff
links,	but	when	one	does,	that	involves	a	further	series	of	tasks.
The	lists	of	steps	in	my	mind	are	organized	in	hierarchies.	I	follow	a	routine
procedure	before	going	to	sleep.	The	first	step	is	to	brush	my	teeth.	But	this
action	is	in	turn	broken	into	a	smaller	series	of	steps,	the	first	of	which	is	to	put
toothpaste	on	the	toothbrush.	That	step	in	turn	is	made	up	of	yet	smaller	steps,
such	as	finding	the	toothpaste,	removing	the	cap,	and	so	on.	The	step	of	finding
the	toothpaste	also	has	steps,	the	first	of	which	is	to	open	the	bathroom	cabinet.

That	step	in	turn	requires	steps,	the	first	of	which	is	to	grab	the	outside	of	the
cabinet	door.	This	nesting	actually	continues	down	to	a	very	fine	grain	of
movements,	so	that	there	are	literally	
thousands	of	little	actions	constituting	my
nighttime	routine.	Although	I	may	have	difficulty	remembering	details	of	a	walk
I	took	just	a	few	hours	ago,	I	have	no	difficulty	recalling	all	of	these	many	steps
in	preparing	for	bed—so	much	so	that	I	am	able	to	think	about	other	things	while

I	go	through	these	procedures.	It	is	important	to	point	out	that	this	list	is	not
stored	as	one	long	list	of	thousands	of	steps—
rather,	each	of	our	routine
procedures	is	remembered	as	an	elaborate	hierarchy	of	nested	activities.
The	same	type	of	hierarchy	is	involved	in	our	ability	to	recognize
objects	and	situations.
	We	recognize	the	faces	of	people	we	know	well	and	also
recognize	that	these	faces	contain	eyes,	a	nose,	a	mouth,	and	so	on—a	hierarchy
of	patterns	that	we	use	in	both	our	perceptions	and	our	actions.	The	use	of

hierarchies	allows	us	to	reuse	patterns.	For	example,	we	do	not	need	to	relearn
the	concept	of	a	nose	and	a	mouth	each	time	we	are	introduced	to	a	new	face.
In	the	
next	chapter
,	we’ll	put	the	results	of	these	thought	experiments
together	into	a	theory	of	how	the	neocortex	must	work.	I	will	argue	that	they
reveal	essential	attributes	of	our	thinking	that	are	uniform,	from	finding	the
toothpaste	to	writing	a	poem.

A	MODEL	OF	THE
NEOCORTEX:	THE	PATTERN
RECOGNITION	THEORY	OF
MIND
	
The	brain	is	a	tissue.	It	is	a	complicated,	intricately	woven	tissue,	like
nothing	else	we	know	of	in	the	universe,	but	it	is	composed	of	cells,	as	any
tissue	is.	They	are,	to	be	sure,	highly	specialized	cells,	but	they	function
according	to	the	laws	that	govern	any	other	cells.	Their	electrical	and
chemical	signals	can	be	detected,	recorded	and	interpreted	and	their
chemicals	can	be	identified;	the	connections	that	constitute	the	brain’s
woven	feltwork	can	be	mapped.	In	short,	the	brain	can	be	studied,	just	as
the	kidney	can.
—David	H.	Hubel,	neuroscientist
	
Suppose	that	there	be	a	machine,	the	structure	of	which	produces	thinking,
feeling,	and	perceiving;	imagine	this	machine	enlarged	but	preserving	the
same	proportions,	so	you	could	enter	it	as	if	it	were	a	mill.	This	being
supposed,	you	might	visit	inside;	but	what	would	you	observe	there?
Nothing	but	parts	which	push	and	move	each	other,	and	never	anything	that
could	explain	perception.

—Gottfried	Wilhelm	Leibniz

A	Hierarchy	of	Patterns
I
	have	repeated	the	simple	experiments	and	observations	described	in	the
previous	chapter
	thousands	of	times	in	myriad	contexts.	The	conclusions	from
these	observations	necessarily	constrain	my	explanation	for	what	the	brain	must
be	doing,	just	as	the	simple	experiments	on	time,	space,	and	mass	that	were
conducted	in	the	early	and	late	nineteenth	century	necessarily	constrained	the
young	Master	Einstein’s	reflections	on	how	the	universe	functioned.	In	the
discussion	that	follows	I’ll	also	factor	in	some	very	basic	observations	from
neuroscience,	attempting	to	avoid	the	many	details	that	are	still	in	contention.
First,	let	me	explain	why	this	section	specifically	discusses	the	neocortex
(from	the	Latin	meaning	“new	rind”).	We	do	know	the	neocortex	is	responsible
for	our	ability	to	deal	with	patterns	of	information	and	to	do	so	in	a	hierarchical
fashion.	Animals	without	a	neocortex	(basically	nonmammals)	are	largely
incapable	of	understanding	hierarchies.
1

1
	Understanding	and	leveraging	the
innately	hierarchical	nature	of	reality	is	a	uniquely	mammalian	trait	and	results
from	mammals’	unique	possession	of	this	evolutionarily	recent	brain	structure.
The	neocortex	is	responsible	for	sensory	perception,	recognition	of	everything
from	visual	objects	to	abstract	concepts,	controlling	movement,	reasoning	from
spatial	orientation	to	rational	thought,	and	language—basically,	what	we	regard
as	“thinking.”

1
The	human	neocortex,	the	outermost	layer	of	the	brain,	is	a	thin,	essentially
two-dimensional	structure	with	a	thickness	of	about	2.5	millimeters	(about	a
tenth	of	an	inch).	In	rodents,	it	is	about	the	size	of	a	postage	stamp	and	is
smooth.	An	evolutionary	innovation	in	primates	is	that	it	became	intricately
folded	over	the	top	of	the	rest	of	the	brain	with	deep	ridges,	grooves,	and
wrinkles	to	increase	its	surface	area.	Due	to	its	elaborate	folding,	the	neocortex

constitutes	the	bulk	of	the	human	brain,	accounting	for	80	percent	of	its	weight.
	developed	a	large	forehead	to	allow	for	an	even	larger	neocortex;
in	particular	we	have	a	
frontal	lobe	where	we	deal	with	the	more	abstract
patterns	associated	with	high-level	concepts.
This	thin	structure	is	basically	made	up	of	six	layers,	numbered	I	(the
outermost	layer)	to	VI.	The	axons	emerging	from	the	neurons	in	layers	II	and	III
project	to	other	parts	of	the	neocortex.	The	axons	(output	connections)	from

layers	V	and	VI	are	connected	primarily	outside	of	the	neocortex	to	the

thalamus,	brain	stem,	and	spinal	cord.	The	neurons	in	layer	IV	receive	synaptic
(input)	connections	from	neurons	that	are	outside	the	neocortex,	especially	in	the
thalamus.	The	number	of	layers	varies	slightly	from	region	to	region.	Layer	IV	is
very	thin	in	the	motor	cortex,	because	in	that	area	it	largely	does	not	receive
input	from	the	thalamus,	brain	stem,	or	spinal	cord.	Conversely,	in	the	occipital
lobe	(the	part	of	the	neocortex	usually	responsible	for	visual	processing),	there
are	three	additional	sublayers	that	can	be	seen	in	layer	IV,	due	to	the
considerable	input	flowing	into	this	region,	including	from	the	thalamus.
A	critically	important	observation	about	the	neocortex	is	the	extraordinary
uniformity	of	its	fundamental	structure.	This	was	first	noticed	by	American
neuroscientist	Vernon	Mountcastle	(born	in	1918).	In	1957	Mountcastle
discovered	the	columnar	organization	of	the	neocortex.	In	1978	he	made	an
observation	that	is	as	significant	to	neuroscience	as	the	Michelson-Morley	ether-

disproving	experiment	of	1887	were	to	physics.	That	year	he	described	the
remarkably	unvarying	organization	of	the	neocortex,	hypothesizing	that	it	was
composed	of	a	single	mechanism	that	was	repeated	over	and	over	again,
2
	and
proposing	the	cortical	column	as	that	basic	unit.	The	differences	in	the	height	of
certain	layers	in	different	regions	noted	above	are	simply	differences	in	the
amount	of	interconnectivity	that	the	regions	are	responsible	for	dealing	with.
Mountcastle	hypothesized	the	existence	of	mini-columns	within	columns,

2
	and
but	this	theory	became	controversial	because	there	were	no	visible	demarcations
of	such	smaller	structures.	However,	extensive	experimentation	has	revealed	that
there	are	in	fact	repeating	units	within	the	neuron	fabric	of	each	column.	It	is	my
contention	that	the	basic	unit	is	a	pattern	
recognizer	and	that	this	constitutes	the
fundamental	component	of	the	neocortex.	In	contrast	to	Mountcastle’s	notion	of
a	mini-column,	there	is	no	specific	physical	boundary	to	these	recognizers,	as

they	are	placed	closely	one	to	the	next	in	an	interwoven	fashion,	so	the	cortical
column	is	simply	an	aggregate	of	a	large	number	of	them.	These	recognizers	are
capable	of	wiring	themselves	to	one	another	throughout	the	course	of	a	lifetime,
so	the	elaborate	connectivity	(between	modules)	that	we	see	in	the	neocortex	is
not	prespecified	by	the	genetic	code,	but	rather	is	created	to	reflect	the	patterns
we	actually	learn	over	time.	I	will	describe	this	thesis	in	more	detail,	but	I

maintain	that	this	is	how	the	neocortex	must	be	organized.
It	should	be	noted,	before	we	further	consider	the	structure	of	the	neocortex,
that	it	is	important	to	model	systems	at	the	right	level.	Although	chemistry	is
theoretically	based	on	physics	and	could	be	derived	entirely	from	physics,	this
would	be	unwieldy	and	infeasible	in	practice,	so	chemistry	has	established	its
own	rules	and	models.	Similarly,	we	should	be	able	to	deduce	the	laws	of
thermodynamics	from	physics,	but	once	we	have	a	sufficient	number	of	particles

to	call	them	a	gas	rather	than	simply	a	bunch	of	particles,	solving	equations	for
the	physics	of	each	particle	interaction	becomes	hopeless,	whereas	the	laws	of
thermodynamics	work	quite	well.	Biology	likewise	has	its	own	rules	and
models.	A	single	pancreatic	islet	cell	is	enormously	complicated,	especially	if	we
model	it	at	the	level	of	molecules;	modeling	what	a	pancreas	actually	does	in
terms	of	regulating	levels	of	insulin	and	digestive	enzymes	is	considerably	less
complex.
The	same	principle	applies	to	the	levels	of	modeling	and	understanding	in
the	brain.	It	is	certainly	a	useful	and	necessary	part	of	reverse-engineering	the
brain	to	model	its	interactions	at	the	molecular	level,	but	the	goal	of	the	effort
here	is	essentially	to	refine	our	model	to	account	for	how	the	brain	processes
information	to	produce	cognitive	meaning.
American	scientist	Herbert	A.	Simon	(1916–2001),	who	is	credited	with
cofounding	the	field	of	artificial	intelligence,	wrote	eloquently	about	the	issue	of

understanding	complex	systems	at	the	right	level	of	abstraction.	In	describing	an
AI	program	he	had	devised	called	EPAM	
(elementary	perceiver	and	memorizer),
he	wrote	in	1973,	“Suppose	you	decided	that	you	wanted	to	understand	the
mysterious	EPAM	program	that	I	have.	I	could	provide	you	with	two	versions	of
it.	One	would	be…the	form	in	which	it	was	actually	written—with	its	whole
structure	of	routines	and	subroutines….	Alternatively,	I	could	provide	you	with	a
machine-language	version	of	EPAM	after	the	whole	translation	had	been	carried

out—after	it	had	been	flattened	so	to	speak….	I	don’t	think	I	need	argue	at
length	which	of	these	two	versions	would	provide	the	most	parsimonious,	the
most	meaningful,	the	most	lawful	description….	I	will	not	even	propose	to	you
the	third…of	providing	you	with	neither	program,	but	instead	with	the
electromagnetic	equations	and	boundary	conditions	that	the	computer,	viewed	as
a	physical	system,	would	have	to	obey	while	behaving	as	EPAM.	That	would	be
the	acme	of	reduction	and	incomprehensibility.”
3

3
There	are	about	a	half	million	cortical	columns	in	a	human	neocortex,	each
occupying	a	space	about	two	millimeters	high	and	a	half	millimeter	wide	and
containing	about	60,000	neurons	(resulting	in	a	total	of	about	30	billion	neurons
in	the	neocortex).	A	rough	estimate	is	that	each	pattern	recognizer	within	a
cortical	column	contains	about	100	neurons,	so	there	are	on	the	order	of	300
million	pattern	recognizers	in	total	in	the	neocortex.
As	we	consider	how	these	pattern	recognizers	work,	let	me	begin	by	saying

that	it	is	difficult	to	know	precisely	where	to	begin.	Everything	happens
simultaneously	in	the	neocortex,	so	there	is	no	beginning	and	no	end	to	its
processes.	I	will	frequently	need	to	refer	to	phenomena	that	I	have	not	yet
explained	but	plan	to	come	back	to,	so	please	bear	with	these	forward	references.

Human	beings	have	only	a	weak	ability	to	process	logic,	but	a	very	deep
core	capability	of	recognizing	patterns.	To	do	logical	thinking,	we	need	to	use
the	neocortex,	which	is	basically	a	large	pattern	recognizer.	It	is	not	an	ideal
mechanism	for	performing	logical	transformations,	but	it	is	the	only	facility	we
have	for	the	job.	Compare,	for	example,	how	a	human	
plays	chess	to	how	a
typical	computer	chess	program	works.	Deep	Blue,	the	computer	that	defeated
Garry	Kasparov,	the	human	world	chess	champion,	in	1997	was	capable	of
analyzing	the	logical	implications	of	200	million	board	positions	(representing
different	move-countermove	sequences)	every	second.	(That	can	now	be	done,
by	the	way,	on	a	few	personal	computers.)	Kasparov	was	asked	how	many
positions	he	could	analyze	each	second,	and	he	said	it	was	less	than	one.	How	is
it,	then,	that	he	was	able	to	hold	up	to	Deep	Blue	at	all?	The	answer	is	the	very
strong	ability	humans	have	to	recognize	patterns.	However,	we	need	to	train	this

facility,	which	is	why	not	everyone	can	play	master	chess.
Kasparov	had	learned	about	100,000	board	positions.	That’s	a	real	number
—we	have	established	that	a	human	master	in	a	particular	field	has	mastered
about	100,000	chunks	of	knowledge.	Shakespeare	composed	his	plays	with
100,000	word	senses	(employing	about	29,000	distinct	words,	but	using	most	of
them	in	multiple	ways).	Medical	expert	systems	that	have	been	built	to	represent
the	knowledge	of	a	human	medical	physician	have	shown	that	a	typical	human

medical	specialist	has	mastered	about	100,000	concepts	in	his	or	her	domain.
Recognizing	a	chunk	of	knowledge	from	this	store	is	not	straightforward,	as	a
particular	item	will	present	itself	a	little	bit	differently	each	time	it	is
experienced.
Armed	with	his	knowledge,	Kasparov	looks	at	the	chessboard	and
compares	the	patterns	that	he	sees	to	all	100,000	board	situations	that	he	has
mastered,	and	he	does	all	100,000	comparisons	simultaneously.	There	is
consensus	on	this	point:	All	of	our	neurons	are	processing—considering	the

patterns—at	the	same	time.	That	does	not	mean	that	they	are	all	
firing
simultaneously	(we	would	probably	fall	to	the	floor	if	that	happened),	but	while
doing	their	processing	are	considering	the	possibility	of	firing.
How	many	patterns	can	the	neocortex	store?	We	need	to	factor	in	the
phenomenon	of	redundancy.	The	face	of	a	loved	one,	for	example,	is	not	stored
once	but	on	the	order	of	thousands	of	times.	Some	of	these	repetitions	are	largely
the	same	image	of	the	face,	whereas	most	show	different	perspectives	of	it,

firing
different	lighting,	different	expressions,	and	so	on.	None	
of	these	repeated
patterns	are	stored	as	images	per	se	(that	is,	as	two-dimensional	arrays	of	pixels).
Rather,	they	are	stored	as	lists	of	features	where	the	constituent	elements	of	a
pattern	are	themselves	patterns.	We’ll	describe	below	more	precisely	what	these

hierarchies	of	features	look	like	and	how	they	are	organized.
If	we	take	the	core	knowledge	of	an	expert	as	consisting	of	about	100,000
“chunks”	of	knowledge	(that	is,	patterns)	with	a	redundancy	estimate	of	about
100	to	1,	that	gives	us	a	requirement	of	10	million	patterns.	This	core	expert
knowledge	is	built	on	more	general	and	extensive	professional	knowledge,	so	we
can	increase	the	order	of	magnitude	of	patterns	to	about	30	to	50	million.	Our
everyday	“commonsense”	knowledge	as	a	human	being	is	even	greater;	“street
smarts”	actually	require	substantially	more	of	our	neocortex	than	“book	smarts.”
Including	this	brings	our	estimate	to	well	over	100	million	patterns,	taking	into
account	the	redundancy	factor	of	about	100.	Note	that	the	redundancy	factor	is
far	from	fixed—very	common	patterns	will	have	a	redundancy	factor	well	into
the	thousands,	whereas	a	brand-new	phenomenon	may	have	a	redundancy	factor
of	less	than	10.
As	I	will	discuss	below,	our	procedures	and	actions	also	comprise	patterns

and	are	likewise	stored	in	regions	of	the	cortex,	so	my	estimate	of	the	total
capacity	of	the	human	neocortex	is	on	the	order	of	low	hundreds	of	millions	of
patterns.	This	rough	tally	correlates	well	with	the	number	of	pattern	recognizers
that	I	estimated	above	at	about	300	million,	so	it	is	a	reasonable	conclusion	that
the	function	of	each	neocortical	pattern	recognizer	is	to	process	one	iteration
(that	is,	one	copy	among	the	multiple	redundant	copies	of	most	patterns	in	the

neocortex)	of	a	pattern.	Our	estimates	of	the	number	of	patterns	that	a	human
brain	is	capable	of	dealing	with	(including	necessary	redundancy)	and	the
number	of	physical	pattern	recognizers	happen	to	be	the	same	order	of
magnitude.	It	should	be	noted	here	that	when	I	refer	to	“processing”	a	pattern,	I
am	referring	to	all	of	the	things	we	are	able	to	do	with	a	pattern:	learn	it,	predict
it	(including	parts	
of	it),	recognize	it,	and	implement	it	(either	by	thinking	about
it	further	or	through	a	pattern	of	physical	movement).

Three	hundred	million	pattern	processors	may	sound	like	a	large	number,
and	indeed	it	was	sufficient	to	enable	
	to	develop	verbal	and
written	language,	all	of	our	tools,	and	other	diverse	creations.	These	inventions
have	built	upon	themselves,	giving	rise	to	the	exponential	growth	of	the
information	content	of	technologies	as	described	in	my	law	of	accelerating
returns.	No	other	species	has	achieved	this.	As	I	discussed,	a	few	other	species,
such	as	chimpanzees,	do	appear	to	have	a	rudimentary	ability	to	understand	and

form	language	and	also	to	use	primitive	tools.	They	do,	after	all,	also	have	a
neocortex,	but	their	abilities	are	limited	due	to	its	smaller	size,	especially	of	the
frontal	lobe.	The	size	of	our	own	neocortex	has	exceeded	a	threshold	that	has
enabled	our	species	to	build	ever	more	powerful	tools,	including	tools	that	can
now	enable	us	to	understand	our	own	intelligence.	Ultimately	our	brains,

combined	with	the	technologies	they	have	fostered,	will	permit	us	to	create	a
synthetic	neocortex	that	will	contain	well	beyond	a	mere	300	million	pattern
processors.	Why	not	a	billion?	Or	a	trillion?

The	Structure	of	a	Pattern
The	pattern	recognition	theory	of	mind	that	I	present	here	is	based	on	the
recognition	of	patterns	by	pattern	recognition	modules	in	the	neocortex.	These
patterns	(and	the	modules)	are	organized	in	hierarchies.	I	discuss	below	the
intellectual	roots	of	this	idea,	including	my	own	work	with	hierarchical	pattern
recognition	in	the	1980s	and	1990s	and	Jeff	Hawkins	(born	in	1957)	and	Dileep
George’s	(born	in	1977)	model	of	the	neocortex	in	the	early	2000s.
Each	pattern	(which	is	recognized	by	one	of	the	estimated	300	million
pattern	recognizers	in	the	neocortex)	is	composed	of	three	parts.	Part	one	is	the
input,	which	consists	of	the	lower-level	patterns	that	compose	the	main	pattern.
The	descriptions	for	each	of	these	lower-level	patterns	do	not	
need	to	be	repeated
for	each	higher-level	pattern	that	references	them.	For	example,	many	of	the
patterns	for	words	will	include	the	letter	“A.”	Each	of	these	patterns	does	not

need	to	repeat	the	description	of	the	letter	“A”	but	will	use	the	same	description.
Think	of	it	as	being	like	a	Web	pointer.	There	is	one	Web	page	(that	is,	one
pattern)	for	the	letter	“A,”	and	all	of	the	Web	pages	(patterns)	for	words	that
include	“A”	will	have	a	link	to	the	“A”	page	(to	the	“A”	pattern).	Instead	of	Web
links,	the	neocortex	uses	actual	neural	connections.	There	is	an	axon	from	the
“A”	pattern	recognizer	that	connects	to	multiple	dendrites,	one	for	each	word

that	uses	“A.”	Keep	in	mind	also	the	redundancy	factor:	There	is	more	than	one
pattern	recognizer	for	the	letter	“A.”	Any	of	these	multiple	“A”	pattern
recognizers	can	send	a	signal	up	to	the	pattern	recognizers	that	incorporate	“A.”

The	second	part	of	each	pattern	is	the	pattern’s	name.	In	the	world	of
language,	this	higher-level	pattern	is	simply	the	word	“apple.”	Although	we
directly	use	our	neocortex	to	understand	and	process	every	level	of	language,
most	of	the	patterns	it	contains	are	not	language	patterns	per	se.	In	the	neocortex
the	“name”	of	a	pattern	is	simply	the	axon	that	emerges	from	
each	pattern
processor;	when	that	axon	fires,	its	corresponding	pattern	has	been	recognized.
The	firing	of	the	axon	is	that	pattern	recognizer	shouting	the	name	of	the	pattern:
“Hey	guys,	I	just	saw	the	written	word	‘apple.’”

Three	redundant	(but	somewhat	different)	patterns	for	“A”	feeding	up	to
higher-level	patterns	that	incorporate	“A.”
The	third	and	final	part	of	each	pattern	is	the	set	of	higher-level	patterns
that	it	in	turn	is	part	of.	For	the	letter	“A,”	this	is	all	of	the	words	that	include
“A.”	These	are,	again,	like	Web	links.	Each	recognized	pattern	at	one	level
triggers	the	next	level	that	part	of	that	higher-level	pattern	is	present.	In	the
neocortex,	these	links	are	represented	by	physical	dendrites	that	flow	into
neurons	in	each	cortical	pattern	recognizer.	Keep	in	mind	that	each	neuron	can
receive	inputs	from	multiple	dendrites	yet	produces	a	single	output	on	an	axon.
That	axon,	however,	can	then	in	turn	transmit	to	multiple	dendrites.
To	take	some	simple	examples,	the	simple	patterns	on	the	next	page	are	a
small	subset	of	the	patterns	used	to	make	up	printed	letters.	Note	that	every	level
constitutes	a	pattern.	In	this	case,	the	shapes	are	patterns,	the	letters	are	
patterns,

patterns,
and	the	words	are	also	patterns.	Each	of	these	patterns	has	a	set	of	inputs,	a
process	of	pattern	recognition	(based	on	the	inputs	that	take	place	in	the
module),	and	an	output	(which	feeds	to	the	next	higher	level	of	pattern
recognizer).
Southwest	to	north-central	connection:
Southeast	to	north-central	connection:
Horizontal	crossbar:
Leftmost	vertical	line:
Concave	region	facing	south:
Bottom	horizontal	line:
Top	horizontal	line:

Middle	horizontal	line:
Loop	constituting	upper	region:
	
The	above	patterns	are	constituents	of	the	next	higher	level	of	pattern,
which	is	a	category	called	printed	letters	(there	is	no	such	formal	category	within
the	neocortex,	however;	indeed,	there	are	no	formal	categories).
“A”:
Two	different	patterns,	either	of	which	constitutes	“A,”	and	two
different	patterns	at	a	higher	level	(“APPLE”	and	“PEAR”)	of	which	“A”	is
a	part.
“P”:

Patterns	that	are	part	of	the	higher-level	pattern	“P.”
	
“L”:
	
Patterns	that	are	part	of	the	higher-level	pattern	“L.”
	
“E”:

Patterns	that	are	part	of	the	higher-level	pattern	“E.”
	
These	letter	patterns	feed	up	to	an	even	higher-level	pattern	in	a	category
called	words.	(The	word	“words”	is	our	language	category	for	this	concept,	but
the	neocortex	just	treats	them	only	as	patterns.)
“APPLE”:
	
In	a	different	part	of	the	cortex	is	a	comparable	hierarchy	of	pattern
recognizers	processing	actual	
images
	of	objects	(as	opposed	to	printed	letters).	If
you	are	looking	at	an	actual	apple,	low-level	recognizers	will	detect	curved
edges	and	surface	color	patterns	leading	up	to	a	pattern	recognizer	firing	its	axon
and	saying	in	effect,	“Hey	guys,	I	just	saw	an	actual	apple.”	Yet	other	pattern
recognizers	will	detect	combinations	of	frequencies	of	sound	leading	up	to	a
pattern	recognizer	in	the	auditory	cortex	that	might	fire	its	axon	indicating,	“I
just	heard	the	spoken	word	‘apple.’”
Keep	in	mind	the	redundancy	factor—we	don’t	just	have	a	single	pattern
recognizer	for	“apple”	in	each	of	its	forms	(written,	spoken,	visual).	There	are

likely	to	be	hundreds	of	such	recognizers	firing,	if	not	more.	The	redundancy	not

only	increases	the	likelihood	that	you	will	successfully	recognize	each	instance
of	an	apple	but	also	deals	with	the	variations	in	real-world	apples.	For	apple
objects,	there	will	be	pattern	recognizers	that	deal	with	the	many	varied	forms	of
apples:	different	views,	colors,	shadings,	shapes,	and	varieties.
Also	keep	in	mind	that	the	hierarchy	shown	above	is	a	hierarchy	of
concepts
.	These	recognizers	are	not	physically	placed	above	each	other;	because
of	the	thin	construction	of	the	neocortex,	it	is	physically	only	one	pattern
recognizer	high.	The	conceptual	hierarchy	is	created	by	the	connections	between
the	individual	pattern	recognizers.
An	important	attribute	of	the	PRTM	is	how	the	recognitions	are	made	inside
each	pattern	recognition	module.	Stored	in	the	module	is	a	weight	for	each	input
dendrite	indicating	how	important	that	input	is	to	the	recognition.	The	pattern
recognizer	has	a	threshold	for	firing	(which	indicates	that	this	pattern	recognizer

has	successfully	recognized	the	pattern	it	is	responsible	for).	Not	every	input
pattern	has	to	be	present	for	a	recognizer	to	fire.	The	recognizer	may	still	fire	if
an	input	with	a	low	weight	is	missing,	but	it	is	less	likely	to	fire	if	a	high-
importance	input	is	missing.	When	it	fires,	a	pattern	recognizer	is	basically
saying,	“The	pattern	I	am	responsible	for	is	probably	present.”
Successful	recognition	by	a	module	of	its	pattern	goes	beyond	just	
counting
the	input	signals	that	are	activated	(even	a	count	weighted	by	the	importance

counting
parameter).	The	size	(of	each	input)	matters.	There	is	another	parameter	(for
each	input)	indicating	the	expected	size	of	the	input,	and	yet	another	indicating
how	variable	that	size	is.	To	appreciate	how	this	works,	suppose	we	have	a
pattern	recognizer	that	is	responsible	for	recognizing	the	spoken	word	“steep.”
This	spoken	word	has	four	sounds:	[s],	[t],	[E],	and	[p].	The	[t]	phoneme	is	what
is	known	as	a	“dental	consonant,”	meaning	that	it	is	created	by	the	tongue
creating	a	burst	of	noise	when	air	breaks	its	contact	with	the	upper	teeth.	It	is

essentially	impossible	to	articulate	the	[t]	phoneme	slowly.	The	[p]	phoneme	is
considered	a	“plosive	consonant”	or	“oral	occlusive,”	meaning	that	it	is	created
when	the	vocal	tract	is	suddenly	blocked	(by	the	lips	in	the	case	of	[p])	so	that
air	no	longer	passes.	It	is	also	necessarily	quick.	The	[E]	vowel	is	caused	by
resonances	of	the	vocal	cord	and	open	mouth.	It	is	considered	a	“long	vowel,”
meaning	that	it	persists	for	a	much	longer	period	of	time	than	consonants	such	as

[t]	and	[p];	however,	its	duration	can	be	quite	variable.	The	[s]	phoneme	is
known	as	a	“sibilant	consonant,”	and	is	caused	by	the	passage	of	air	against	the
edges	of	the	teeth,	which	are	held	close	together.	Its	duration	is	typically	shorter
than	that	of	a	long	vowel	such	as	[E],	but	it	is	also	variable	(in	other	words,	the
[s]	can	be	said	quickly	or	you	can	drag	it	out).
In	our	work	in	speech	recognition,	we	found	that	it	is	necessary	to	encode

this	type	of	information	in	order	to	recognize	speech	patterns.	For	example,	the
words	“step”	and	“steep”	are	very	similar.	Although	the	[e]	phoneme	in	“step”
and	the	[E]	in	“steep”	are	somewhat	different	vowel	sounds	(in	that	they	have
different	resonant	frequencies),	it	is	not	reliable	to	distinguish	these	two	words
based	on	these	often	confusable	vowel	sounds.	It	is	much	more	reliable	to
consider	the	observation	that	the	[e]	in	“step”	is	relatively	brief	compared	with
the	[E]	in	“steep.”
We	can	encode	this	type	of	information	with	two	numbers	for	each	input:
the	expected	size	and	the	degree	of	variability	of	that	size.	In	our	“steep”
example,	[t]	and	[p]	would	both	have	a	very	short	expected	duration	as	well	as	a
small	expected	variability	(that	is,	we	do	not	expect	to	hear	
long	t’s	and	p’s).	The
[s]	sound	would	have	a	short	expected	duration	but	a	larger	variability	because	it
is	possible	to	drag	it	out.	The	[E]	sound	has	a	long	expected	duration	as	well	as	a
high	degree	of	variability.

In	our	speech	examples,	the	“size”	parameter	refers	to	duration,	but	time	is
only	one	possible	dimension.	In	our	work	in	character	recognition,	we	found	that
comparable	spatial	information	was	important	in	order	to	recognize	printed
letters	(for	example	the	dot	over	the	letter	“i”	is	expected	to	be	much	smaller
than	the	portion	under	the	dot).	At	much	higher	levels	of	abstraction,	the
neocortex	will	deal	with	patterns	with	all	sorts	of	continuums,	such	as	levels	of

attractiveness,	irony,	happiness,	frustration,	and	myriad	others.	We	can	draw
similarities	across	rather	diverse	continuums,	as	Darwin	did	when	he	related	the
physical	size	of	geological	canyons	to	the	amount	of	differentiation	among
species.
In	a	biological	brain,	the	source	of	these	parameters	comes	from	the	brain’s
own	experience.	We	are	not	born	with	an	innate	knowledge	of	phonemes;	indeed
different	languages	have	very	different	sets	of	them.	This	implies	that	multiple

species.
examples	of	a	pattern	are	encoded	in	the	learned	parameters	of	each	pattern
recognizer	(as	it	requires	multiple	instances	of	a	pattern	to	ascertain	the	expected
distribution	of	magnitudes	of	the	inputs	to	the	pattern).	In	some	AI	systems,
these	types	of	parameters	are	hand-coded	by	experts	(for	example,	linguists	who
can	tell	us	the	expected	durations	of	different	phonemes,	as	I	articulated	above).
In	my	own	work,	we	found	that	having	an	AI	system	discover	these	parameters

on	its	own	from	training	data	(similar	to	the	way	the	brain	does	it)	was	a	superior
approach.	Sometimes	we	used	a	hybrid	approach;	that	is,	we	primed	the	system
with	the	intuition	of	human	experts	(for	the	initial	settings	of	the	parameters)	and
then	had	the	AI	system	automatically	refine	these	estimates	using	a	learning
process	from	real	examples	of	speech.
What	the	pattern	recognition	module	is	doing	is	computing	the	probability

(that	is,	the	likelihood	based	on	all	of	its	previous	experience)	that	the	pattern
that	it	is	responsible	for	recognizing	is	in	fact	currently	represented	by	its	active
inputs.	Each	particular	input	to	the	module	is	active	if	
the	corresponding	lower-
level	pattern	recognizer	is	firing	(meaning	that	that	lower-level	pattern	was
recognized).	Each	input	also	encodes	the	observed	size	(on	some	appropriate
dimension	such	as	temporal	duration	or	physical	magnitude	or	some	other
continuum)	so	that	the	size	can	be	compared	(with	the	stored	size	parameters	for
each	input)	by	the	module	in	computing	the	overall	probability	of	the	pattern.
How	does	the	brain	(and	how	can	an	AI	system)	compute	the	overall
probability	that	the	pattern	(that	the	module	is	responsible	for	recognizing)	is
present	given	(1)	the	inputs	(each	with	an	observed	size),	(2)	the	stored
parameters	on	size	(the	expected	size	and	the	variability	of	size)	for	each	input,
and	(3)	the	parameters	of	the	importance	of	each	input?	In	the	1980s	and	1990s,

I	and	others	pioneered	a	mathematical	method	called	hierarchical	hidden	Markov
models	for	learning	these	parameters	and	then	using	them	to	recognize
hierarchical	patterns.	We	used	this	technique	in	the	recognition	of	human	speech
as	well	as	the	understanding	of	natural	language.	I	describe	this	approach	further
in	
chapter	7
.
Getting	back	to	the	flow	of	recognition	from	one	level	of	pattern
recognizers	to	the	next,	in	the	above	example	we	see	the	information	flow	up	the
conceptual	hierarchy	from	basic	letter	features	to	letters	to	words.	Recognitions

in	
chapter	7
.
will	continue	to	flow	up	from	there	to	phrases	and	then	more	complex	language
structures.	If	we	go	up	several	dozen	more	levels,	we	get	to	higher-level
concepts	like	irony	and	envy.	Even	though	every	pattern	recognizer	is	working
simultaneously,	it	does	take	time	for	recognitions	to	move	upward	in	this
conceptual	hierarchy.	Traversing	each	level	takes	between	a	few	hundredths	to	a
few	tenths	of	a	second	to	process.	Experiments	have	shown	that	a	moderately
high-level	pattern	such	as	a	face	takes	at	least	a	tenth	of	a	second.	It	can	take	as

long	as	an	entire	second	if	there	are	significant	distortions.	If	the	brain	were
sequential	(like	conventional	computers)	and	was	performing	each	pattern
recognition	in	sequence,	it	would	have	to	consider	every	possible	low-level
pattern	before	moving	on	to	the	next	level.	Thus	it	would	take	many	millions	of
cycles	just	to	go	through	each	level.	That	is	exactly	what	happens	when	we
simulate	these	
processes	on	a	computer.	Keep	in	mind,	however,	that	computers
process	millions	of	times	faster	than	our	biological	circuits.

A	very	important	point	to	note	here	is	that	information	flows	down	the
conceptual	hierarchy	as	well	as	up.	If	anything,	this	downward	flow	is	even
more	significant.	If,	for	example,	we	are	reading	from	left	to	right	and	have
already	seen	and	recognized	the	letters	“A,”	“P,”	“P,”	and	“L,”	the	“APPLE”

recognizer	will	predict	that	it	is	likely	to	see	an	“E”	in	the	next	position.	It	will
send	a	signal	
down
	to	the	“E”	recognizer	saying,	in	effect,	“Please	be	aware	that
there	is	a	high	likelihood	that	you	will	see	your	‘E’	pattern	very	soon,	so	be	on
the	lookout	for	it.”	The	“E”	recognizer	then	adjusts	its	threshold	such	that	it	is
more	likely	to	recognize	an	“E.”	So	if	an	image	appears	next	that	is	vaguely	like
an	“E,”	but	is	perhaps	smudged	such	that	it	would	not	have	been	recognized	as
an	“E”	under	“normal”	circumstances,	the	“E”	recognizer	may	nonetheless
indicate	that	it	has	indeed	seen	an	“E,”	since	it	was	expected.
The	neocortex	is,	therefore,	predicting	what	it	expects	to	encounter.
Envisaging	the	future	is	one	of	the	primary	reasons	we	have	a	neocortex.	At	the
highest	conceptual	level,	we	are	continually	making	predictions—who	is	going
to	walk	through	the	door	next,	what	someone	is	likely	to	say	next,	what	we
expect	to	see	when	we	turn	the	corner,	the	likely	results	of	our	own	actions,	and

so	on.	These	predictions	are	constantly	occurring	at	
every
	level	of	the	neocortex
hierarchy.	We	often	misrecognize	people	and	things	and	words	because	our
threshold	for	confirming	an	expected	pattern	is	too	low.
In	addition	to	positive	signals,	there	are	also	negative	or	inhibitory	signals
which	indicate	that	a	certain	pattern	is	less	likely	to	exist.	These	can	come	from
lower	conceptual	levels	(for	example,	the	recognition	of	a	mustache	will	inhibit
the	likelihood	that	a	person	I	see	in	the	checkout	line	is	my	wife),	or	from	a

every
higher	level	(for	example,	I	know	that	my	wife	is	on	a	trip,	so	the	person	in	the
checkout	line	can’t	be	she).	When	a	pattern	recognizer	receives	an	inhibitory
signal,	it	raises	the	recognition	threshold,	but	
it	is	still	possible	for	the	pattern	to
fire	(so	if	the	person	in	line	really	is	her,	I	may	still	recognize	her).

The	Nature	of	the	Data	Flowing	into	a	Neocortical	Pattern
Recognizer
Let’s	consider	further	what	the	data	for	a	pattern	looks	like.	If	the	pattern	is	a
face,	the	data	exists	in	at	least	two	dimensions.	We	cannot	say	that	the	eyes
necessarily	come	first,	followed	by	the	nose,	and	so	on.	The	same	thing	is	true
for	most	sounds.	A	musical	piece	has	at	least	two	dimensions.	There	may	be
more	than	one	instrument	and/or	voice	making	sounds	at	the	same	time.
Moreover,	a	single	note	of	a	complex	instrument	such	as	the	piano	consists	of
multiple	frequencies.	A	single	human	voice	consists	of	varying	levels	of	energy
in	dozens	of	different	frequency	bands	simultaneously.	So	a	pattern	of	sound
may	be	complex	at	any	one	instant,	and	these	complex	instants	stretch	out	over
time.	Tactile	inputs	are	also	two-dimensional,	since	the	skin	is	a	two-dimensional
sense	organ,	and	such	patterns	may	change	over	the	third	dimension	of	time.
So	it	would	seem	that	the	input	to	a	neocortex	pattern	processor	must

comprise	two-if	not	three-dimensional	patterns.	However,	we	can	see	in	the
structure	of	the	neocortex	that	the	pattern	inputs	are	only	one-dimensional	lists.
All	of	our	work	in	the	field	of	creating	artificial	pattern	recognition	systems
(such	as	speech	recognition	and	visual	recognition	systems)	demonstrates	that
we	can	(and	did)	represent	two-and	three-dimensional	phenomena	with	such
one-dimensional	lists.	I’ll	describe	how	these	methods	work	in	
chapter	7
,	but	for
now	we	can	proceed	with	the	understanding	that	the	input	to	each	pattern

chapter	7
,	but	for
processor	is	a	one-dimensional	list,	even	though	the	pattern	itself	may	inherently
reflect	more	than	one	dimension.
We	should	factor	in	at	this	point	the	insight	that	the	patterns	we	have
learned	to	recognize	(for	example,	a	specific	dog	or	the	general	idea	of	a	
“dog,”
a	musical	note	or	a	piece	of	music)	are	exactly	the	same	mechanism	that	is	the
basis	for	our	memories.	Our	memories	are	in	fact	patterns	organized	as	lists
(where	each	item	in	each	list	is	another	pattern	in	the	cortical	hierarchy)	that	we

“dog,”
have	learned	and	then	recognize	when	presented	with	the	appropriate	stimulus.
In	fact,	memories	exist	in	the	neocortex	in	order	to	be	recognized.
The	only	exception	to	this	is	at	the	lowest	possible	conceptual	level,	in
which	the	input	data	to	a	pattern	represents	specific	sensory	information	(for
example,	image	data	from	the	optic	nerve).	Even	this	lowest	level	of	pattern,
however,	has	been	significantly	transformed	into	simple	patterns	by	the	time	it

reaches	the	cortex.	The	lists	of	patterns	that	constitute	a	memory	are	in	forward

order,	and	we	are	able	to	remember	our	memories	only	in	that	order,	hence	the
difficulty	we	have	in	reversing	our	memories.
A	memory	needs	to	be	triggered	by	another	thought/memory	(these	are	the
same	thing).	We	can	experience	this	mechanism	of	triggering	when	we	are
perceiving	a	pattern.	When	we	perceived	“A,”	“P,”	“P,”	and	“L,”	the	“A	P	P	L	E”
pattern	predicted	that	we	would	see	an	“E”	and	triggered	the	“E”	pattern	that	it	is
now	expected.	Our	cortex	is	thereby	“thinking”	of	seeing	an	“E”	even	before	we
see	it.	If	this	particular	interaction	in	our	cortex	has	our	attention,	we	will	think
about	“E”	before	we	see	it	or	even	if	we	never	see	it.	A	similar	mechanism
triggers	old	memories.	Usually	there	is	an	entire	chain	of	such	links.	Even	if	we
do	have	some	level	of	awareness	of	the	memories	(that	is,	the	patterns)	that
triggered	the	old	memory,	memories	(patterns)	do	not	have	language	or	image
labels.	This	is	the	reason	why	old	memories	may	seem	to	suddenly	jump	into	our

awareness.	Having	been	buried	and	not	activated	for	perhaps	years,	they	need	a
trigger	in	the	same	way	that	a	Web	page	needs	a	Web	link	to	be	activated.	And
just	as	a	Web	page	can	become	“orphaned”	because	no	other	page	links	to	it,	the
same	thing	can	happen	to	our	memories.
Our	thoughts	are	largely	activated	in	one	of	two	modes,	undirected	and
directed,	both	of	which	use	these	same	cortical	links.	In	the	undirected	mode,	we
let	the	links	play	themselves	out	without	attempting	to	
move	them	in	any

particular	direction.	Some	forms	of	meditation	(such	as	Transcendental
Meditation,	which	I	practice)	are	based	on	letting	the	mind	do	exactly	this.
Dreams	have	this	quality	as	well.
In	directed	thinking	we	attempt	to	step	through	a	more	orderly	process	of
recalling	a	memory	(a	story,	for	example)	or	solving	a	problem.	This	also
involves	stepping	through	lists	in	our	neocortex,	but	the	less	structured	flurry	of
undirected	thought	will	also	accompany	the	process.	The	full	content	of	our

thinking	is	therefore	very	disorderly,	a	phenomenon	that	James	Joyce	illuminated
in	his	“stream	of	consciousness”	novels.
As	you	think	through	the	memories/stories/patterns	in	your	life,	whether
they	involve	a	chance	encounter	with	a	mother	with	a	baby	carriage	and	baby	on
a	walk	or	the	more	important	narrative	of	how	you	met	your	spouse,	your
memories	consist	of	a	sequence	of	patterns.	Because	these	patterns	are	not
labeled	with	words	or	sounds	or	pictures	or	videos,	when	you	try	to	recall	a

significant	event,	you	will	essentially	be	reconstructing	the	images	in	your	mind,
because	the	actual	images	do	not	exist.
If	we	were	to	“read”	the	mind	of	someone	and	peer	at	exactly	what	is	going
on	in	her	neocortex,	it	would	be	very	difficult	to	interpret	her	memories,	whether
we	were	to	take	a	look	at	patterns	that	are	simply	stored	in	the	neocortex	waiting

to	be	triggered	or	those	that	have	been	triggered	and	are	currently	being
experienced	as	active	thoughts.	What	we	would	“see”	is	the	simultaneous
activation	of	millions	of	pattern	recognizers.	A	hundredth	of	a	second	later,	we
would	see	a	different	set	of	a	comparable	number	of	activated	pattern
recognizers.	Each	such	pattern	would	be	a	list	of	other	patterns,	and	each	of
those	patterns	would	be	a	list	of	other	patterns,	and	so	on	until	we	reached	the
most	elementary	simple	patterns	at	the	lowest	level.	It	would	be	extremely
difficult	to	interpret	what	these	higher-level	patterns	meant	without	actually
copying	
all
	of	the	information	at	every	level	into	our	own	cortex.	Thus	each
pattern	in	our	neocortex	is	meaningful	only	in	light	of	all	the	information	carried
in	the	levels	below	it.	Moreover,	other	patterns	at	the	same	level	and	at	higher
levels	are	also	relevant	in	interpreting	a	particular	pattern	because	they	provide
context.	True	mind	reading,	
therefore,	would	necessitate	not	just	detecting	the

copying	
all
activations	of	the	relevant	axons	in	a	person’s	brain,	but	examining	essentially
her	entire	neocortex	with	all	of	its	memories	to	understand	these	activations.
As	we	experience	our	own	thoughts	and	memories,	we	“know”	what	they
mean,	but	they	do	not	exist	as	readily	explainable	thoughts	and	recollections.	If
we	want	to	share	them	with	others,	we	need	to	translate	them	into	language.	This
task	is	also	accomplished	by	the	neocortex,	using	pattern	recognizers	trained

with	patterns	that	we	have	learned	for	the	purpose	of	using	language.	Language
is	itself	highly	hierarchical	and	evolved	to	take	advantage	of	the	hierarchical
nature	of	the	neocortex,	which	in	turn	reflects	the	hierarchical	nature	of	reality.
The	innate	ability	of	humans	to	learn	the	hierarchical	structures	in	language	that
Noam	Chomsky	wrote	about	reflects	the	structure	of	the	neocortex.	In	a	2002
paper	he	coauthored,	Chomsky	cites	the	attribute	of	“recursion”	as	accounting
for	the	unique	language	faculty	of	the	human	species.
4

4
	Recursion,	according	to
Chomsky,	is	the	ability	to	put	together	small	parts	into	a	larger	chunk,	and	then
use	that	chunk	as	a	part	in	yet	another	structure,	and	to	continue	this	process
iteratively.	In	this	way	we	are	able	to	build	the	elaborate	structures	of	sentences
and	paragraphs	from	a	limited	set	of	words.	Although	Chomsky	was	not
explicitly	referring	here	to	brain	structure,	the	capability	he	is	describing	is
exactly	what	the	neocortex	does.
Lower	species	of	mammals	largely	use	up	their	neocortex	with	the

challenges	of	their	particular	lifestyles.	The	human	species	acquired	additional
capacities	by	having	grown	substantially	more	cortex	to	handle	spoken	and
written	language.	Some	people	have	learned	such	skills	better	than	others.	If	we
have	told	a	particular	story	many	times,	we	will	begin	to	actually	learn	the
sequence	of	language	that	describes	the	story	as	a	series	of	separate	sequences.
Even	in	this	case	our	memory	is	not	a	strict	sequence	of	words,	but	rather	of

language	structures	that	we	need	to	translate	into	specific	word	sequences	each
time	we	deliver	the	story.	That	is	why	we	tell	a	story	a	bit	differently	each	time
we	share	it	(unless	we	learn	the	exact	word	sequence	as	a	pattern).
For	each	of	these	descriptions	of	specific	thought	processes,	we	also	need	to
consider	the	issue	of	redundancy.	As	I	mentioned,	we	don’t	have	a	single	pattern
representing	the	important	entities	in	our	lives,	whether	those	entities	constitute
sensory	categories,	language	concepts,	or	memories	of	events.	Every	important
pattern—at	every	level—is	repeated	many	times.	Some	of	these	recurrences
represent	simple	repetitions,	whereas	many	represent	different	perspectives	and
vantage	points.	This	is	a	principal	reason	why	we	can	recognize	a	familiar	face
from	various	orientations	and	under	a	range	of	lighting	conditions.	Each	level	up
the	hierarchy	has	substantial	redundancy,	allowing	sufficient	variability	that	is
consistent	with	that	concept.

So	if	we	were	to	imagine	examining	your	neocortex	when	you	were	looking
at	a	particular	loved	one,	we	would	see	a	great	many	firings	of	the	axons	of	the
pattern	recognizers	at	every	level,	from	the	basic	level	of	
primitive	sensory
patterns	up	to	many	different	patterns	representing	that	loved	one’s	image.	We
would	also	see	massive	numbers	of	firings	representing	other	aspects	of	the
situation,	such	as	that	person’s	movements,	what	she	is	saying,	and	so	on.	So	if

the	experience	seems	much	richer	than	just	an	orderly	trip	up	a	hierarchy	of
features,	it	is.
A	computer	simulation	of	the	firings	of	many	simultaneous	pattern

recognizers	in	the	neocortex.
But	the	basic	mechanism	of	going	up	a	hierarchy	of	pattern	recognizers	in
which	each	higher	conceptual	level	represents	a	more	abstract	and	more
integrated	concept	remains	valid.	The	flow	of	information	downward	is	even
greater,	as	each	activated	level	of	recognized	pattern	sends	predictions	to	the
next	lower-level	pattern	recognizer	of	what	it	is	likely	to	be	encountering	next.
The	apparent	lushness	of	human	experience	is	a	result	of	the	fact	that	all	of	the
hundreds	of	millions	of	pattern	recognizers	in	our	neocortex	are	considering
their	inputs	simultaneously.
In	
chapter	5
	I’ll	discuss	the	flow	of	information	from	touch,	vision,	hearing,
and	other	sensory	organs	into	the	neocortex.	These	early	inputs	are	processed	by
cortical	regions	that	are	devoted	to	relevant	types	of	sensory	input	(although
there	is	enormous	plasticity	in	the	assignment	of	these	regions,	reflecting	the
basic	uniformity	of	function	in	the	neocortex).	The	conceptual	hierarchy

In	
chapter	5
continues	above	the	highest	concepts	in	each	sensory	region	of	the	neocortex.
The	cortical	association	areas	integrate	input	from	the	different	sensory	inputs.
When	we	hear	something	that	perhaps	sounds	like	our	spouse’s	voice,	and	then
see	something	that	is	perhaps	indicative	of	her	presence,	we	don’t	engage	in	an
elaborate	process	of	logical	deduction;	rather,	we	instantly	perceive	that	our
spouse	is	present	from	the	combination	of	these	sensory	recognitions.	We

integrate	all	of	the	germane	sensory	and	perceptual	cues—perhaps	even	the
smell	of	her	perfume	or	his	cologne—as	one	multilevel	perception.
At	a	conceptual	level	above	the	cortical	sensory	association	areas,	we	are
capable	of	dealing	with—perceiving,	remembering,	and	thinking	about—even
more	abstract	concepts.	At	the	highest	level	we	recognize	patterns	such	as	
that’s
funny
,	or	
she’s	pretty
,	or	
that’s	ironic
,	and	so	on.	Our	memories	include	these
abstract	recognition	patterns	as	well.	For	example,	we	
might	recall	that	we	were

that’s
funny
,	or	
,	or	
taking	a	walk	with	someone	and	that	she	said	something	funny,	and	we	laughed,
though	we	may	not	remember	the	actual	joke	itself.	The	memory	sequence	for
that	recollection	has	simply	recorded	the	perception	of	humor	but	not	the	precise
content	of	what	was	funny.
In	the	
	I	noted	that	we	can	often	recognize	a	pattern	even
though	we	don’t	recognize	it	well	enough	to	be	able	to	describe	it.	For	example,
I	believe	I	could	pick	out	a	picture	of	the	woman	with	the	baby	carriage	whom	I

In	the	
saw	earlier	today	from	among	a	group	of	pictures	of	other	women,	despite	the
fact	that	I	am	unable	to	actually	visualize	her	and	cannot	describe	much	specific
about	her.	In	this	case	my	memory	of	her	is	a	list	of	certain	high-level	features.
These	features	do	not	have	language	or	image	labels	attached	to	them,	and	they
are	not	pixel	images,	so	while	I	am	able	to	think	about	her,	I	am	unable	to

describe	her.	However,	if	I	am	presented	with	a	picture	of	her,	I	can	process	the
image,	which	results	in	the	recognition	of	the	same	high-level	features	that	were
recognized	the	first	time	I	saw	her.	I	would	be	able	to	thereby	determine	that	the
features	match	and	thus	confidently	pick	out	her	picture.
Even	though	I	saw	this	woman	only	once	on	my	walk,	there	are	probably
already	multiple	copies	of	her	pattern	in	my	neocortex.	However,	if	I	don’t	think
about	her	for	a	given	period	of	time,	then	these	pattern	recognizers	will	become
reassigned	to	other	patterns.	That	is	why	memories	grow	dimmer	with	time:	The
amount	of	redundancy	becomes	reduced	until	certain	memories	become	extinct.
However,	now	that	I	have	memorialized	this	particular	woman	by	writing	about
her	here,	I	probably	won’t	forget	her	so	easily.

Autoassociation	and	Invariance
In	the	
	I	discussed	how	we	can	recognize	a	pattern	even	if	the
entire	pattern	is	not	present,	and	also	if	it	is	distorted.	The	first	capability	is
called	autoassociation:	the	ability	to	associate	a	pattern	with	a	part	of	
itself.	The
structure	of	each	pattern	recognizer	inherently	supports	this	capability.
As	each	input	from	a	lower-level	pattern	recognizer	flows	up	to	a	higher-
level	one,	the	connection	can	have	a	“weight,”	indicating	how	important	that
particular	element	in	the	pattern	is.	Thus	the	more	significant	elements	of	a
pattern	are	more	heavily	weighted	in	considering	whether	that	pattern	should
trigger	as	“recognized.”	Lincoln’s	beard,	Elvis’s	sideburns,	and	Einstein’s
famous	tongue	gesture	are	likely	to	have	high	weights	in	the	patterns	we’ve
learned	about	the	appearance	of	these	iconic	figures.	The	pattern	recognizer
computes	a	probability	that	takes	the	importance	parameters	into	account.	Thus

the	overall	probability	is	lower	if	one	or	more	of	the	elements	is	missing,	though
the	threshold	of	recognition	may	nonetheless	be	met.	As	I	pointed	out,	the
computation	of	the	overall	probability	(that	the	pattern	is	present)	is	more
complicated	than	a	simple	weighted	sum	in	that	the	size	parameters	also	need	to
be	considered.
If	the	pattern	recognizer	has	received	a	signal	from	a	higher-level
recognizer	that	its	pattern	is	“expected,”	then	the	threshold	is	effectively	lowered

(that	is,	made	easier	to	achieve).	Alternatively,	such	a	signal	may	simply	add	to
the	total	of	the	weighted	inputs,	thereby	compensating	for	a	missing	element.
This	happens	at	every	level,	so	that	a	pattern	such	as	a	face	that	is	several	levels
up	from	the	bottom	may	be	recognized	even	with	multiple	missing	features.
The	ability	to	recognize	patterns	even	when	aspects	of	them	are
transformed	is	called	feature	invariance,	and	is	dealt	with	in	four	ways.	First,
there	are	global	transformations	that	are	accomplished	before	the	neocortex

receives	sensory	data.	We	will	discuss	the	voyage	of	sensory	data	from	the	eyes,
ears,	and	skin	in	the	section	
“The	Sensory	Pathway”
	on	
page	94
.
The	second	method	takes	advantage	of	the	redundancy	in	our	cortical
pattern	memory.	Especially	for	important	items,	we	have	learned	many	different
perspectives	and	vantage	points	for	each	pattern.	Thus	many	variations	are
separately	stored	and	processed.
The	third	and	most	powerful	method	is	the	ability	to	combine	two	
lists.	One
list	can	have	a	set	of	transformations	that	we	have	learned	may	apply	to	a	certain

category	of	pattern;	the	cortex	will	apply	this	same	list	of	possible	changes	to
another	pattern.	That	is	how	we	understand	such	language	phenomena	as
metaphors	and	similes.
For	example,	we	have	learned	that	certain	phonemes	(the	basic	sounds	of
language)	may	be	missing	in	spoken	speech	(for	example,	“goin’”).	If	we	then
learn	a	new	spoken	word	(for	example,	“driving”),	we	will	be	able	to	recognize
that	word	if	one	of	its	phonemes	is	missing	even	if	we	have	never	experienced
that	word	in	that	form	before,	because	we	have	become	familiar	with	the	general
phenomenon	of	certain	phonemes	being	omitted.	As	another	example,	we	may
learn	that	a	particular	artist	likes	to	emphasize	(by	making	larger)	certain
elements	of	a	face,	such	as	the	nose.	We	can	then	identify	a	face	with	which	we
are	familiar	to	which	that	modification	has	been	applied	even	if	we	have	never
seen	that	modification	on	that	face.	Certain	artistic	modifications	emphasize	the

very	features	that	are	recognized	by	our	pattern	recognition–based	neocortex.	As
mentioned,	that	is	precisely	the	basis	of	caricature.
The	fourth	method	derives	from	the	size	parameters	that	allow	a	single
module	to	encode	multiple	instances	of	a	pattern.	For	example,	we	have	heard
the	word	“steep”	many	times.	A	particular	pattern	recognition	module	that	is
recognizing	this	spoken	word	can	encode	these	multiple	examples	by	indicating
that	the	duration	of	[E]	has	a	high	expected	variability.	If	all	the	modules	for

words	including	[E]	share	a	similar	phenomenon,	that	variability	could	be
encoded	in	the	models	for	[E]	itself.	However,	different	words	incorporating	[E]
(or	many	other	phonemes)	may	have	different	amounts	of	expected	variability.
For	example,	the	word	“peak”	is	likely	not	to	have	the	[E]	phoneme	as	drawn	out
as	in	the	word	“steep.”

Learning
Are	we	not	ourselves	creating	our	successors	in	the	supremacy	of	the	earth?
Daily	adding	to	the	beauty	and	delicacy	of	their	organization,	
daily	giving
them	greater	skill	and	supplying	more	and	more	of	that	self-regulating	self-
acting	power	which	will	be	better	than	any	intellect?
—Samuel	Butler,	1871
	
The	principal	activities	of	brains	are	making	changes	in	themselves.
—Marvin	Minsky,	
The	Society	of	Mind
	
So	far	we	have	examined	how	we	recognize	(sensory	and	perceptual)	patterns
and	recall	sequences	of	patterns	(our	memory	of	things,	people,	and	events).
However,	we	are	not	born	with	a	neocortex	filled	with	any	of	these	patterns.	Our
neocortex	is	virgin	territory	when	our	brain	is	created.	It	has	the	capability	of
learning	and	therefore	of	creating	connections	between	its	pattern	recognizers,
but	it	gains	those	connections	from	experience.
This	learning	process	begins	even	before	we	are	born,	occurring
simultaneously	with	the	biological	process	of	actually	growing	a	brain.	A	fetus

already	has	a	brain	at	one	month,	although	it	is	essentially	a	reptile	brain,	as	the
fetus	actually	goes	through	a	high-speed	re-creation	of	biological	evolution	in
the	womb.	The	natal	brain	is	distinctly	a	human	brain	with	a	human	neocortex
by	the	time	it	reaches	the	third	trimester	of	pregnancy.	At	this	time	the	fetus	is
having	experiences,	and	the	neocortex	is	learning.	She	can	hear	sounds,
especially	her	mother’s	heartbeat,	which	is	one	likely	reason	that	the	rhythmic

qualities	of	music	are	universal	to	human	culture.	Every	human	civilization	ever
discovered	has	had	music	as	part	of	its	culture,	which	is	not	the	case	with	other
art	forms,	such	as	pictorial	art.	It	is	also	the	case	that	the	beat	of	music	is
comparable	to	our	heart	rate.	Music	beats	certainly	vary—otherwise	music
would	not	keep	our	interest—but	heartbeats	vary	also.	An	overly	regular
heartbeat	is	actually	a	symptom	of	a	diseased	heart.	The	eyes	of	a	fetus	are
partially	open	twenty-six	weeks	after	conception,	and	are	fully	open	most	of	the

time	by	twenty-eight	weeks	after	conception.	There	may	not	be	much	to	see
inside	the	womb,	but	there	are	patterns	of	light	and	dark	that	the	neocortex

begins	to	process.
So	while	a	newborn	baby	has	had	a	bit	of	experience	in	the	womb,	it	is
clearly	limited.	The	neocortex	may	also	learn	from	the	old	brain	(a	topic	I
discuss	in	
chapter	5
),	but	in	general	at	birth	the	child	has	a	lot	to	learn—
everything	from	basic	primitive	sounds	and	shapes	to	metaphors	and	sarcasm.
Learning	is	critical	to	human	intelligence.	If	we	were	to	perfectly	model
and	simulate	the	human	neocortex	(as	the	Blue	Brain	Project	is	attempting	to	do)
and	all	of	the	other	brain	regions	that	it	requires	to	function	(such	as	the
hippocampus	and	thalamus),	it	would	not	be	able	to	do	very	much—in	the	same
way	that	a	newborn	infant	cannot	do	much	(other	than	to	be	cute,	which	is
definitely	a	key	survival	adaptation).
Learning	and	recognition	take	place	simultaneously.	We	start	learning
immediately,	and	as	soon	as	we’ve	learned	a	pattern,	we	immediately	start
recognizing	it.	The	neocortex	is	continually	trying	to	make	sense	of	the	input

presented	to	it.	If	a	particular	level	is	unable	to	fully	process	and	recognize	a
pattern,	it	gets	sent	to	the	next	higher	level.	If	none	of	the	levels	succeeds	in
recognizing	a	pattern,	it	is	deemed	to	be	a	new	pattern.	Classifying	a	pattern	as
new	does	not	necessarily	mean	that	every	aspect	of	it	is	new.	If	we	are	looking	at
the	paintings	of	a	particular	artist	and	see	a	cat’s	face	with	the	nose	of	an
elephant,	we	will	be	able	to	identify	each	of	the	distinctive	features	but	will

notice	that	this	combined	pattern	is	something	novel,	and	are	likely	to	remember
it.	Higher	conceptual	levels	of	the	neocortex,	which	understand	context—for
example,	the	circumstance	that	this	picture	is	an	example	of	a	particular	artist’s
work	and	that	we	are	attending	an	opening	of	a	showing	of	new	paintings	by	that
artist—will	note	the	unusual	combination	of	patterns	in	the	cat-elephant	face	but
will	also	include	these	contextual	details	as	additional	memory	patterns.

New	memories	such	as	the	cat-elephant	face	are	stored	in	an	available
pattern	recognizer.	The	hippocampus	plays	a	role	in	this	process,	and	we’ll
discuss	what	is	known	about	the	actual	biological	mechanisms	in	the	following
chapter.	For	the	purposes	of	our	neocortex	model,	it	is	sufficient	to	say	that
patterns	that	are	not	otherwise	recognized	are	stored	as	new	
patterns	and	are
appropriately	connected	to	the	lower-level	patterns	that	form	them.	The	cat-
elephant	face,	for	example,	will	be	stored	in	several	different	ways:	The	novel

arrangement	of	facial	parts	will	be	stored	as	well	as	contextual	memories	that
include	the	artist,	the	situation,	and	perhaps	the	fact	that	we	laughed	when	we
first	saw	it.
Memories	that	are	successfully	recognized	may	also	result	in	the	creation	of
a	new	pattern	to	achieve	greater	redundancy.	If	patterns	are	not	perfectly
recognized,	they	are	likely	to	be	stored	as	reflecting	a	different	perspective	of	the

item	that	was	recognized.
What,	then,	is	the	overall	method	for	determining	what	patterns	get	stored?
In	mathematical	terms,	the	problem	can	be	stated	as	follows:	Using	the	available
limits	of	pattern	storage,	how	do	we	optimally	represent	the	input	patterns	that
have	thus	far	been	presented?	While	it	makes	sense	to	allow	for	a	certain	amount
of	redundancy,	it	would	not	be	practical	to	fill	up	the	entire	available	storage	area
(that	is,	the	entire	neocortex)	with	repeated	patterns,	as	that	would	not	allow	for
a	sufficient	diversity	of	patterns.	A	pattern	such	as	the	[E]	phoneme	in	spoken
words	is	something	we	have	experienced	countless	times.	It	is	a	simple	pattern
of	sound	frequencies	and	it	undoubtedly	enjoys	significant	redundancy	in	our
neocortex.	We	could	fill	up	our	entire	neocortex	with	repeated	patterns	of	the	[E]
phoneme.	There	is	a	limit,	however,	to	useful	redundancy,	and	a	common	pattern
such	as	this	clearly	has	reached	it.

There	is	a	mathematical	solution	to	this	optimization	problem	called	linear
programming,	which	solves	for	the	best	possible	allocation	of	limited	resources
(in	this	case,	a	limited	number	of	pattern	recognizers)	that	would	represent	all	of
the	cases	on	which	the	system	has	trained.	Linear	programming	is	designed	for
systems	with	one-dimensional	inputs,	which	is	another	reason	why	it	is	optimal
to	represent	the	input	to	each	pattern	recognition	module	as	a	linear	string	of
inputs.	We	can	use	this	mathematical	approach	in	a	software	system,	and	though

an	actual	brain	is	further	constrained	by	the	physical	connections	it	has	available
that	it	can	adapt	between	pattern	recognizers,	the	method	is	nonetheless	similar.
An	important	implication	of	this	optimal	solution	is	that	experiences	that
are	routine	are	recognized	but	do	not	result	in	a	permanent	memory’s	being
made.	With	regard	to	my	walk,	I	experienced	millions	of	patterns	at	every	level,
from	basic	visual	edges	and	shadings	to	objects	such	as	lampposts	and	mailboxes

and	people	and	animals	and	plants	that	I	passed.	Almost	none	of	what	I
experienced	was	unique,	and	the	patterns	that	I	recognized	had	long	since
reached	their	optimal	level	of	redundancy.	The	result	is	that	I	recall	almost
nothing	from	this	walk.	The	few	details	that	I	do	remember	are	likely	to	get
overwritten	with	new	patterns	by	the	time	I	take	another	few	dozen	walks—
except	for	the	fact	that	I	have	now	memorialized	this	particular	walk	by	writing
about	it.
One	important	point	that	applies	to	both	our	biological	neocortex	and

about	it.
attempts	to	emulate	it	is	that	it	is	difficult	to	learn	too	many	conceptual	levels
simultaneously.	We	can	essentially	learn	one	or	at	most	two	conceptual	levels	at
a	time.	Once	that	learning	is	relatively	stable,	we	can	go	on	to	learn	the	next
level.	We	may	continue	to	fine-tune	the	learning	in	the	lower	levels,	but	our
learning	focus	is	on	the	next	level	of	abstraction.	This	is	true	at	both	the

beginning	of	life,	as	newborns	struggle	with	basic	shapes,	and	later	in	life,	as	we
struggle	to	learn	new	subject	matter,	one	level	of	complexity	at	a	time.	We	find
the	same	phenomenon	in	machine	emulations	of	the	neocortex.	However,	if	they
are	presented	increasingly	abstract	material	one	level	at	a	time,	machines	are
capable	of	learning	just	as	humans	do	(although	not	yet	with	as	many	conceptual
levels).
The	output	of	a	pattern	can	feed	back	to	a	pattern	at	a	lower	level	or	even	to
the	pattern	itself,	giving	the	human	brain	its	powerful	recursive	ability.	An
element	of	a	pattern	can	be	a	decision	point	based	on	another	pattern.	This	is
especially	useful	for	lists	that	compose	actions—for	example,	getting	another
tube	of	toothpaste	if	the	current	one	is	empty.	These	conditionals	exist	at	every
level.	As	anyone	who	has	attempted	to	program	a	procedure	on	a	computer
knows,	conditionals	are	vital	to	describing	a	course	of	action.

The	Language	of	Thought
The	dream	acts	as	a	safety-valve	for	the	over-burdened	brain.
—Sigmund	Freud,
The	Interpretation	of	Dreams,
	1911
	
Brain:	an	apparatus	with	which	we	think	we	think.
—Ambrose	Bierce,	
The	Devil’s	Dictionary
	
To	summarize	what	we’ve	learned	so	far	about	the	way	the	neocortex	works,
please	refer	to	the	diagram	of	the	neocortical	pattern	recognition	module	on	
page
42
.
a)	Dendrites	enter	the	module	that	represents	the	pattern.	Even	though
patterns	may	seem	to	have	two-or	three-dimensional	qualities,	they	are
represented	by	a	one-dimensional	sequence	of	signals.	The	pattern	must
be	present	in	this	(sequential)	order	for	the	pattern	recognizer	to	be	able
to	recognize	it.	Each	of	the	dendrites	is	connected	ultimately	to	one	or
more	axons	of	pattern	recognizers	at	a	lower	conceptual	level	that	have
recognized	a	lower-level	pattern	that	constitutes	part	of	this	pattern.	For
each	of	these	input	patterns,	there	may	be	many	lower-level	pattern

recognizers	that	can	generate	the	signal	that	the	lower-level	pattern	has
been	recognized.	The	necessary	threshold	to	recognize	the	pattern	may
be	achieved	even	if	not	all	of	the	inputs	have	signaled.	The	module
computes	the	probability	that	the	pattern	it	is	responsible	for	is	present.
This	computation	considers	the	“importance”	and	“size”	parameters	(see
[f]	below).
Note	that	some	of	the	dendrites	transmit	signals	into	the	module	and
some	out	of	the	module.	If	all	of	the	input	dendrites	to	this	pattern

recognizer	are	signaling	that	their	lower-level	patterns	have	been
recognized	except	for	one	or	two,	then	this	pattern	recognizer	will	send	a
signal	down	to	the	pattern	recognizer(s)	recognizing	the	lower-level
patterns	that	have	not	yet	been	recognized,	indicating	that	there	is	a	high
likelihood	that	
that	pattern	will	soon	be	recognized	and	that	lower-level

recognizer(s)	should	be	on	the	lookout	for	it.
b)	When	this	pattern	recognizer	recognizes	its	pattern	(based	on	all	or	most
of	the	input	dendrite	signals	being	activated),	the	axon	(output)	of	this
pattern	recognizer	will	activate.	In	turn,	this	axon	can	connect	to	an	entire
network	of	dendrites	connecting	to	many	higher-level	pattern	recognizers
that	this	pattern	is	input	to.	This	signal	will	transmit	magnitude
information	so	that	the	pattern	recognizers	at	the	next	higher	conceptual
level	can	consider	it.
c)	If	a	higher-level	pattern	recognizer	is	receiving	a	positive	signal	from	all
or	most	of	its	constituent	patterns	except	for	the	one	represented	by	this
pattern	recognizer,	then	that	higher-level	recognizer	might	send	a	signal
down	to	this	recognizer	indicating	that	its	pattern	is	expected.	Such	a
signal	would	cause	this	pattern	recognizer	to	lower	its	threshold,	meaning
that	it	would	be	more	likely	to	send	a	signal	on	its	axon	(indicating	that

its	pattern	is	considered	to	have	been	recognized)	even	if	some	of	its
inputs	are	missing	or	unclear.
d)	Inhibitory	signals	from	below	would	make	it	less	likely	that	this	pattern
recognizer	will	recognize	its	pattern.	This	can	result	from	recognition	of
lower-level	patterns	that	are	inconsistent	with	the	pattern	associated	with
this	pattern	recognizer	(for	example,	recognition	of	a	mustache	by	a
lower-level	recognizer	would	make	it	less	likely	that	this	image	is	“my
wife”).

wife”).
e)	Inhibitory	signals	from	above	would	also	make	it	less	likely	that	this
pattern	recognizer	will	recognize	its	pattern.	This	can	result	from	a
higher-level	context	that	is	inconsistent	with	the	pattern	associated	with
this	recognizer.
f)	For	each	input,	there	are	stored	parameters	for	importance,	expected	size,
and	expected	variability	of	size.	The	module	computes	an	overall
probability	that	the	pattern	is	present	based	on	all	of	these	parameters	and
the	current	signals	indicating	which	of	the	inputs	are	present	and	their

magnitudes.	A	mathematically	optimal	way	to	accomplish	this	is	with	a
technique	called	hidden	Markov	models.	When	such	models	are
organized	in	a	hierarchy	(as	they	are	in	the	neocortex	or	in	attempts	to
simulate	a	neocortex),	we	call	them	hierarchical	hidden	Markov	models.

Patterns	triggered	in	the	neocortex	trigger	other	patterns.	Partially	complete
patterns	send	signals	down	the	conceptual	hierarchy;	completed	patterns	send
signals	up	the	conceptual	hierarchy.	These	neocortical	patterns	are	the	language
of	thought.	Just	like	language,	they	are	hierarchical,	but	they	are	not	language
per	se.	Our	thoughts	are	not	conceived	primarily	in	the	elements	of	language,
although	since	language	also	exists	as	hierarchies	of	patterns	in	our	neocortex,
we	can	have	language-based	thoughts.	But	for	the	most	part,	thoughts	are
represented	in	these	neocortical	patterns.
As	I	discussed	above,	if	we	were	able	to	detect	the	pattern	activations	in
someone’s	neocortex,	we	would	still	have	little	idea	what	those	pattern
activations	meant	without	also	having	access	to	the	entire	hierarchy	of	patterns
above	and	below	each	activated	pattern.	That	would	pretty	much	require	access
to	that	person’s	entire	neocortex.	It	is	hard	enough	for	us	to	understand	the

content	of	our	own	thoughts,	but	understanding	another	person’s	requires
mastering	a	neocortex	different	from	our	own.	Of	course	we	don’t	yet	have
access	to	someone	else’s	neocortex;	we	need	instead	to	rely	on	her	attempts	to
express	her	thoughts	into	language	(as	well	as	other	means	such	as	gestures).
People’s	incomplete	ability	to	accomplish	these	communication	tasks	adds
another	layer	of	complexity—it	is	no	wonder	that	we	misunderstand	one	another
as	much	as	we	do.

We	have	two	modes	of	thinking.	One	is	nondirected	thinking,	in	which
thoughts	trigger	one	another	in	a	nonlogical	way.	When	we	experience	a	sudden
recollection	of	a	memory	from	years	or	decades	ago	while	doing	something	else,
such	as	raking	the	leaves	or	walking	down	the	street,	the	experience	is	recalled—
as	all	memories	are—as	a	sequence	of	patterns.	We	do	not	immediately	visualize
the	scene	unless	we	can	call	upon	a	lot	of	other	memories	that	enable	us	to
synthesize	a	more	robust	recollection.	If	we	do	
visualize	the	scene	in	that	way,

we	are	essentially	creating	it	in	our	mind	from	hints	at	the	time	of	recollection;
the	memory	itself	is	not	stored	in	the	form	of	images	or	visualizations.	As	I
mentioned	earlier,	the	triggers	that	led	this	thought	to	pop	into	our	mind	may	or
may	not	be	evident.	The	sequence	of	relevant	thoughts	may	have	been
immediately	forgotten.	Even	if	we	do	remember	it,	it	will	be	a	nonlinear	and
circuitous	sequence	of	associations.
The	second	mode	of	thinking	is	directed	thinking,	which	we	use	when	we

attempt	to	solve	a	problem	or	formulate	an	organized	response.	For	example,	we
might	be	rehearsing	in	our	mind	something	we	plan	to	say	to	someone,	or	we
might	be	formulating	a	passage	we	want	to	write	(in	a	book	on	the	mind,
perhaps).	As	we	think	about	tasks	such	as	these,	we	have	already	broken	down

each	one	into	a	hierarchy	of	subtasks.	Writing	a	book,	for	example,	involves
writing	chapters;	each	chapter	has	sections;	each	section	has	paragraphs;	each
paragraph	contains	sentences	that	express	ideas;	each	idea	has	its	configuration
of	elements;	each	element	and	each	relationship	between	elements	is	an	idea	that
needs	to	be	articulated;	and	so	on.	At	the	same	time,	our	neocortical	structures
have	learned	certain	rules	that	should	be	followed.	If	the	task	is	writing,	then	we
should	try	to	avoid	unnecessary	repetition;	we	should	try	to	make	sure	that	the
reader	can	follow	what	is	being	written;	we	should	try	to	follow	rules	about
grammar	and	style;	and	so	on.	The	writer	needs	therefore	to	build	a	model	of	the
reader	in	his	mind,	and	that	construct	is	hierarchical	as	well.	In	doing	directed
thinking,	we	are	stepping	through	lists	in	our	neocortex,	each	of	which	expands
into	extensive	hierarchies	of	sublists,	each	with	its	own	considerations.	Keep	in

mind	that	elements	in	a	list	in	a	neocortical	pattern	can	include	conditionals,	so
our	subsequent	thoughts	and	actions	will	depend	on	assessments	made	as	we	go
through	the	process.
Moreover,	each	such	directed	thought	will	trigger	hierarchies	of	undirected
thoughts.	A	continual	storm	of	ruminations	attends	both	our	sensory	experiences
and	our	attempts	at	directed	thinking.	Our	actual	mental	experience	is	complex
and	messy,	made	up	of	these	lightning	storms	of	triggered	patterns,	which
change	about	a	hundred	times	a	second.

The	Language	of	Dreams
Dreams	are	examples	of	undirected	thoughts.	They	make	a	certain	amount	of
sense	because	the	phenomenon	of	one	thought’s	triggering	another	is	based	on
the	actual	linkages	of	patterns	in	our	neocortex.	To	the	extent	that	a	dream	does
not	make	sense,	we	attempt	to	fix	it	through	our	ability	to	confabulate.	As	I	will
describe	in	
chapter	9
,	split-brain	patients	(whose	corpus	callosum,	which
connects	the	two	hemispheres	of	the	brain,	is	severed	or	damaged)	will
confabulate	(make	up)	explanations	with	their	left	brain—which	controls	the
speech	center—to	explain	what	the	right	brain	just	did	with	input	that	the	left
brain	did	not	have	access	to.	We	confabulate	all	the	time	in	explaining	the
outcome	of	events.	If	you	want	a	good	example	of	this,	just	tune	in	to	the	daily
commentary	on	the	movement	of	financial	markets.	No	matter	how	the	markets
perform,	it’s	always	possible	to	come	up	with	a	good	explanation	for	why	it

happened,	and	such	after-the-fact	commentary	is	plentiful.	Of	course,	if	these
commentators	really	understood	the	markets,	they	wouldn’t	have	to	waste	their
time	doing	commentary.
The	act	of	confabulating	is	of	course	also	done	in	the	neocortex,	which	is
good	at	coming	up	with	stories	and	explanations	that	meet	certain	constraints.
We	do	that	whenever	we	retell	a	story.	We	will	fill	in	details	that	may	not	be
available	or	that	we	may	have	forgotten	so	that	the	story	makes	more	sense.	That

is	why	stories	change	over	time	as	they	are	told	over	and	over	again	by	new
storytellers	with	perhaps	different	agendas.	As	spoken	language	led	to	written
language,	however,	we	had	a	technology	that	could	record	a	definitive	version	of
a	story	and	prevent	this	sort	of	drift.
The	actual	content	of	a	dream,	to	the	extent	that	we	remember	it,	is	again	a
sequence	of	patterns.	These	patterns	represent	constraints	in	a	story;	we	then
confabulate	a	story	that	fits	these	constraints.	The	version	of	the	dream	that	we

retell	(even	if	only	to	ourselves	silently)	is	this	confabulation.	As	we	recount	a
dream	we	trigger	cascades	of	patterns	that	fill	in	the	actual	dream	as	we
originally	experienced	it.
There	is	one	key	difference	between	dream	thoughts	and	our	thinking	while
awake.	One	of	the	lessons	we	learn	in	life	is	that	certain	actions,	even	thoughts,
are	not	permissible	in	the	real	world.	For	example,	we	learn	that	we	cannot
immediately	fulfill	our	desires.	There	are	rules	against	grabbing	the	money	in	the
cash	register	at	a	store,	and	constraints	on	interacting	with	a	person	to	whom	we

may	be	physically	attracted.	We	also	learn	that	certain	thoughts	are	not
permissible	because	they	are	culturally	forbidden.	As	we	learn	professional
skills,	we	learn	the	ways	of	thinking	that	are	recognized	and	rewarded	in	our
professions,	and	thereby	avoid	patterns	of	thought	that	might	betray	the	methods
and	norms	of	that	profession.	Many	of	these	taboos	are	worthwhile,	as	they
enforce	social	order	and	consolidate	progress.	However,	they	can	also	prevent
progress	by	enforcing	an	unproductive	orthodoxy.	Such	orthodoxy	is	precisely
what	Einstein	left	behind	when	he	tried	to	ride	a	light	beam	with	his	thought
Cultural	rules	are	enforced	in	the	neocortex	with	help	from	the	old	brain,
especially	the	amygdala.	Every	thought	we	have	triggers	other	thoughts,	and
some	of	them	will	relate	to	associated	dangers.	We	learn,	for	example,	that
breaking	a	cultural	norm	even	in	our	private	thoughts	can	lead	to	ostracism,
which	the	neocortex	realizes	threatens	our	well-being.	If	we	entertain	such

thoughts,	the	amygdala	is	triggered,	and	that	generates	fear,	which	generally
leads	to	terminating	that	thought.
In	dreams,	however,	these	taboos	are	relaxed,	and	we	will	often	dream
about	matters	that	are	culturally,	sexually,	or	professionally	forbidden.	It	is	as	if
our	brain	realizes	that	we	are	not	an	actual	actor	in	the	world	while	dreaming.
Freud	wrote	about	this	phenomenon	but	also	noted	that	we	will	disguise	such
dangerous	thoughts,	at	least	when	we	attempt	to	recall	them,	so	that	the	awake
brain	continues	to	be	protected	from	them.

Relaxing	professional	taboos	turns	out	to	be	useful	for	creative	problem
solving.	I	use	a	mental	technique	each	night	in	which	I	think	about	a	particular
problem	before	I	go	to	sleep.	This	triggers	sequences	of	thoughts	that	will
continue	into	my	dreams.	Once	I	am	dreaming,	I	can	think—
dream
—about
solutions	to	the	problem	without	the	burden	of	the	professional	
restraints	I	carry
during	the	day.	I	can	then	access	these	dream	thoughts	in	the	morning	while	in
an	in-between	state	of	dreaming	and	being	awake,	sometimes	referred	to	as

dream
—about
“lucid	dreaming.”
5
Freud	also	famously	wrote	about	the	ability	to	gain	insight	into	a	person’s
psychology	by	interpreting	dreams.	There	is	of	course	a	vast	literature	on	all
aspects	of	this	theory,	but	the	fundamental	notion	of	gaining	insight	into
ourselves	through	examination	of	our	dreams	makes	sense.	Our	dreams	are
created	by	our	neocortex,	and	thus	their	substance	can	be	revealing	of	the
content	and	connections	found	there.	The	relaxation	of	the	constraints	on	our
thinking	that	exist	while	we	are	awake	is	also	useful	in	revealing	neocortical

content	that	we	otherwise	would	be	unable	to	access	directly.	It	is	also
reasonable	to	conclude	that	the	patterns	that	end	up	in	our	dreams	represent

important	matters	to	us	and	thereby	clues	in	understanding	our	unresolved
desires	and	fears.

The	Roots	of	the	Model
As	I	mentioned	above,	I	led	a	team	in	the	1980s	and	1990s	that	developed	the
technique	of	hierarchical	hidden	Markov	models	to	recognize	human	speech	and
understand	natural-language	statements.	This	work	was	the	predecessor	to
today’s	widespread	commercial	systems	that	recognize	and	understand	what	we
are	trying	to	tell	them	(car	navigation	systems	that	you	can	talk	to,	Siri	on	the
iPhone,	Google	Voice	Search,	and	many	others).	The	technique	we	developed
had	substantially	all	of	the	attributes	that	I	describe	in	the	PRTM.	It	included	a
hierarchy	of	patterns	with	each	higher	level	being	conceptually	more	abstract
than	the	one	below	it.	For	example,	in	speech	recognition	the	levels	included
basic	patterns	of	sound	frequency	at	the	lowest	level,	then	phonemes,	then	words
and	phrases	(which	were	often	recognized	as	if	they	were	words).	Some	of	our
speech	recognition	systems	could	understand	the	meaning	of	natural-language

commands,	so	yet	higher	levels	included	such	structures	as	noun	and	verb
phrases.	Each	pattern	recognition	module	could	recognize	a	linear	sequence	of
patterns	
from	a	lower	conceptual	level.	Each	input	had	parameters	for
importance,	size,	and	variability	of	size.	There	were	“downward”	signals
indicating	that	a	lower-level	pattern	was	expected.	I	discuss	this	research	in	more
detail	in	
chapter	7
.
In	2003	and	2004,	PalmPilot	inventor	Jeff	Hawkins	and	Dileep	George
developed	a	hierarchical	cortical	model	called	hierarchical	temporal	memory.

patterns	
detail	in	
chapter	7
.
With	science	writer	Sandra	Blakeslee,	Hawkins	described	this	model	eloquently
in	their	book	
.	Hawkins	provides	a	strong	case	for	the	uniformity
of	the	cortical	algorithm	and	its	hierarchical	and	list-based	organization.	There
are	some	important	differences	between	the	model	presented	in	
and	what	I	present	in	this	book.	As	the	name	implies,	Hawkins	is	emphasizing
the	temporal	(time-based)	nature	of	the	constituent	lists.	In	other	words,	the

direction	of	the	lists	is	always	forward	in	time.	His	explanation	for	how	the
features	in	a	two-dimensional	pattern	such	as	the	printed	letter	“A”	have	a
direction	in	time	is	predicated	on	eye	movement.	He	explains	that	we	visualize
images	using	saccades,	which	are	very	rapid	movements	of	the	eye	of	which	we
are	unaware.	The	information	reaching	the	neocortex	is	therefore	not	a	two-
dimensional	set	of	features	but	rather	a	time-ordered	list.	While	it	is	true	that	our
eyes	do	make	very	rapid	movements,	the	sequence	in	which	they	view	the

features	of	a	pattern	such	as	the	letter	“A”	does	not	always	occur	in	a	consistent

temporal	order.	(For	example,	eye	saccades	will	not	always	register	the	top
vertex	in	“A”	before	its	bottom	concavity.)	Moreover,	we	can	recognize	a	visual
pattern	that	is	presented	for	only	a	few	tens	of	milliseconds,	which	is	too	short	a
period	of	time	for	eye	saccades	to	scan	it.	It	is	true	that	the	pattern	recognizers	in
the	neocortex	store	a	pattern	as	a	list	and	that	the	list	is	indeed	ordered,	but	the
order	does	not	necessarily	represent	time.	That	is	often	indeed	the	case,	but	it
may	also	represent	a	spatial	or	higher-level	conceptual	ordering	as	I	discussed
above.
The	most	important	difference	is	the	set	of	parameters	that	I	have	included
for	each	input	into	the	pattern	recognition	module,	especially	the	size	and	size
variability	parameters.	In	the	1980s	we	actually	tried	to	
recognize	human	speech
without	this	type	of	information.	This	was	motivated	by	linguists’	telling	us	that
the	duration	information	was	not	especially	important.	This	perspective	is

above.
illustrated	by	dictionaries	that	write	out	the	pronunciation	of	each	word	as	a
string	of	phonemes,	for	example	the	word	“steep”	as	[s]	[t]	[E]	[p],	with	no
indication	of	how	long	each	phoneme	is	expected	to	last.	The	implication	is	that
if	we	create	programs	to	recognize	phonemes	and	then	encounter	this	particular
sequence	of	four	phonemes	(in	a	spoken	utterance),	we	should	be	able	to
recognize	that	spoken	word.	The	system	we	built	using	this	approach	worked	to
some	extent	but	not	well	enough	to	deal	with	such	attributes	as	a	large

vocabulary,	multiple	speakers,	and	words	spoken	continuously	without	pauses.
When	we	used	the	technique	of	hierarchical	hidden	Markov	models	in	order	to
incorporate	the	distribution	of	magnitudes	of	each	input,	performance	soared.

THE	BIOLOGICAL
NEOCORTEX
	
Because	important	things	go	in	a	case,	you’ve	got	a	skull	for	your	brain,	a
plastic	sleeve	for	your	comb,	and	a	wallet	for	your	money.
—George	Costanza,	in	“The	Reverse	Peephole”	episode	of	
Seinfeld
	
Now,	for	the	first	time,	we	are	observing	the	brain	at	work	in	a	global
manner	with	such	clarity	that	we	should	be	able	to	discover	the	overall
programs	behind	its	magnificent	powers.
—J.	G.	Taylor,	B.	Horwitz,	and	K.	J.	Friston
	
The	mind,	in	short,	works	on	the	data	it	receives	very	much	as	a	sculptor
works	on	his	block	of	stone.	In	a	sense	the	statue	stood	there	from	eternity.
But	there	were	a	thousand	different	ones	beside	it,	and	the	sculptor	alone	is
to	thank	for	having	extricated	this	one	from	the	rest.	Just	so	the	world	of
each	of	us,	howsoever	different	our	several	views	of	it	may	be,	all	lay
embedded	in	the	primordial	chaos	of	sensations,	which	gave	the	mere
matter
	to	the	thought	of	all	of	us	indifferently.	We	may,	if	we	like,	by	our

matter
reasonings	unwind	things	back	to	that	black	and	jointless	continuity	of
space	and	moving	clouds	of	swarming	atoms	which	science	calls	the	only
real	world.	But	all	the	while	the	
world	
we
	feel	and	live	in	will	be	that	which
our	ancestors	and	we,	by	slowly	cumulative	strokes	of	choice,	have
extricated	out	of	this,	like	sculptors,	by	simply	rejecting	certain	portions	of
the	given	stuff.	Other	sculptors,	other	statues	from	the	same	stone!	Other
minds,	other	worlds	from	the	same	monotonous	and	inexpressive	chaos!

world	
we
My	world	is	but	one	in	a	million	alike	embedded,	alike	real	to	those	who
may	abstract	them.	How	different	must	be	the	worlds	in	the	consciousness
of	ant,	cuttle-fish,	or	crab!
—William	James

I
s	intelligence	the	goal,	or	even	
a
	goal,	of	biological	evolution?	Steven	Pinker
writes,	“We	are	chauvinistic	about	our	brains,	thinking	them	to	be	the	goal	of
evolution,”
1
	and	goes	on	to	argue	that	“that	makes	no	sense….	Natural	selection
does	nothing	even	close	to	striving	for	intelligence.	The	process	is	driven	by
differences	in	the	survival	and	reproduction	rates	of	replicating	organisms	in	a
particular	environment.	Over	time,	the	organisms	acquire	designs	that	adapt
them	for	survival	and	reproduction	in	that	environment,	period;	nothing	pulls
them	in	any	direction	other	than	success	there	and	then.”	Pinker	concludes	that
“life	is	a	densely	branching	bush,	not	a	scale	or	a	ladder,	and	living	organisms
are	at	the	tips	of	branches,	not	on	lower	rungs.”
With	regard	to	the	human	brain,	he	questions	whether	the	“benefits
outweigh	the	costs.”	Among	the	costs,	he	cites	that	“the	brain	[is]	bulky.	The
female	pelvis	barely	accommodates	a	baby’s	outsized	head.	That	design

compromise	kills	many	women	during	childbirth	and	requires	a	pivoting	gait
that	makes	women	biomechanically	less	efficient	walkers	than	men.	Also	a
heavy	head	bobbing	around	on	a	neck	makes	us	more	vulnerable	to	fatal	injuries
in	accidents	such	as	falls.”	He	goes	on	to	list	additional	shortcomings,	including
the	brain’s	energy	consumption,	its	slow	reaction	time,	and	the	lengthy	process
of	learning.
While	each	of	these	statements	is	accurate	on	its	face	(although	many	
of	my

of	my
female	friends	are	better	walkers	than	I	am),	Pinker	is	missing	the	overall	point
here.	It	is	true	that	biologically,	evolution	has	no	specific	direction.	It	is	a	search
method	that	indeed	thoroughly	fills	out	the	“densely	branching	bush”	of	nature.
It	is	likewise	true	that	evolutionary	changes	do	not	
necessarily
	move	in	the
direction	of	greater	intelligence—they	move	in	
all
	directions.	There	are	many
examples	of	successful	creatures	
that	have	remained	relatively	unchanged	for

of	my
	move	in	the
all
millions	of	years.	(Alligators,	for	instance,	date	back	200	million	years,	and
many	microorganisms	go	back	much	further	than	that.)	But	in	the	course	of
thoroughly	filling	out	myriad	evolutionary	branches,	one	of	the	directions	it	
does
move	in	is	toward	greater	intelligence.	That	is	the	relevant	point	for	the	purposes
of	this	discussion.

The	neocortex	in	different	mammals.
Suppose	we	have	a	blue	gas	in	a	jar.	When	we	remove	the	lid,	there	is	no
message	that	goes	out	to	all	of	the	molecules	of	the	gas	saying,	“Hey,	guys,	the
lid	is	off	the	jar;	let’s	head	up	toward	the	opening	and	out	to	freedom.”	The
molecules	just	keep	doing	what	they	always	do,	which	is	to	move	every	which
way	with	no	seeming	direction.	But	in	the	course	of	doing	so,	some	of	them	near
the	top	will	indeed	move	out	of	the	jar,	and	over	time	
most	of	them	will	follow
suit.	Once	biological	evolution	stumbled	on	a	neural	mechanism	capable	of
hierarchical	learning,	it	found	it	to	be	immensely	useful	for	evolution’s	one
objective,	which	is	survival.	The	benefit	of	having	a	neocortex	became	acute
when	quickly	changing	circumstances	favored	rapid	learning.	Species	of	all
kinds—plants	and	animals—can	learn	to	adapt	to	changing	circumstances	over
time,	but	without	a	neocortex	they	must	use	the	process	of	genetic	evolution.	It

can	take	a	great	many	generations—thousands	of	years—for	a	species	without	a
neocortex	to	learn	significant	new	behaviors	(or	in	the	case	of	plants,	other
adaptation	strategies).	The	salient	survival	advantage	of	the	neocortex	was	that	it
could	learn	in	a	matter	of	days.	If	a	species	encounters	dramatically	changed
circumstances	and	one	member	of	that	species	invents	or	discovers	or	just
stumbles	upon	(these	three	methods	all	being	variations	of	innovation)	a	way	to

adapt	to	that	change,	other	individuals	will	notice,	learn,	and	copy	that	method,
and	it	will	quickly	spread	virally	to	the	entire	population.	The	cataclysmic

Cretaceous-Paleogene	extinction	event	about	65	million	years	ago	led	to	the
rapid	demise	of	many	non-neocortex-bearing	species	that	could	not	adapt
quickly	enough	to	a	suddenly	altered	environment.	This	marked	the	turning	point
for	neocortex-capable	mammals	to	take	over	their	ecological	niche.	In	this	way,
biological	evolution	found	that	the	hierarchical	learning	of	the	neocortex	was	so
valuable	that	this	region	of	the	brain	continued	to	grow	in	size	until	it	virtually
took	over	the	brain	of	
.
Discoveries	in	neuroscience	have	established	convincingly	the	key	role
played	by	the	hierarchical	capabilities	of	the	neocortex	as	well	as	offered
evidence	for	the	pattern	recognition	theory	of	mind	(PRTM).	This	evidence	is
distributed	among	many	observations	and	analyses,	a	portion	of	which	I	will
review	here.	Canadian	psychologist	Donald	O.	Hebb	(1904–1985)	made	an
initial	attempt	to	explain	the	neurological	basis	of	learning.	In	1949	he	described

.
a	mechanism	in	which	neurons	change	physiologically	based	on	their
experience,	thereby	providing	a	basis	for	learning	and	brain	plasticity:	“Let	us
assume	that	the	persistence	or	repetition	of	a	
reverberatory	activity	(or	‘trace’)
tends	to	induce	lasting	cellular	changes	that	add	to	its	stability….	When	an	axon
of	cell	
A
	is	near	enough	to	excite	a	cell	
B
	and	repeatedly	or	persistently	takes
part	in	firing	it,	some	growth	process	or	metabolic	change	takes	place	in	one	or
both	cells	such	that	
A

of	cell	
A
B
A
’s	efficiency,	as	one	of	the	cells	firing	
B
,	is	increased.”
2
This	theory	has	been	stated	as	“cells	that	fire	together	wire	together”	and	has
become	known	as	Hebbian	learning.	Aspects	of	Hebb’s	theory	have	been
confirmed,	in	that	it	is	clear	that	brain	assemblies	can	create	new	connections
and	strengthen	them,	based	on	their	own	activity.	We	can	actually	see	neurons
developing	such	connections	in	brain	scans.	Artificial	“neural	nets”	are	based	on
Hebb’s	model	of	neuronal	learning.

A
B
2
The	central	assumption	in	Hebb’s	theory	is	that	the	basic	unit	of	learning	in
the	neocortex	is	the	neuron.	The	pattern	recognition	theory	of	mind	that	I
articulate	in	this	book	is	based	on	a	different	fundamental	unit:	not	the	neuron
itself,	but	rather	an	assembly	of	neurons,	which	I	estimate	to	number	around	a
hundred.	The	wiring	and	synaptic	strengths	
within
	each	unit	are	relatively	stable
and	determined	genetically—that	is,	the	organization	within	each	pattern

within
recognition	module	is	determined	by	genetic	design.	Learning	takes	place	in	the
creation	of	connections	
between
	these	units,	not	within	them,	and	probably	in	the
synaptic	strengths	of	those	interunit	connections.
Recent	support	for	the	basic	module	of	learning’s	being	a	module	of	dozens
of	neurons	comes	from	Swiss	neuroscientist	Henry	Markram	(born	in	1962),
whose	ambitious	Blue	Brain	Project	to	simulate	the	entire	human	brain	I
chapter	7
.	In	a	2011	paper	he	describes	how	while	scanning	and

analyzing	actual	mammalian	neocortex	neurons,	he	was	“search[ing]	for
evidence	of	Hebbian	assemblies	at	the	most	elementary	level	of	the	cortex.”
What	he	found	instead,	he	writes,	were	“elusive	assemblies	[whose]	connectivity
and	synaptic	weights	are	highly	predictable	and	constrained.”	He	concludes	that
“these	findings	imply	that	experience	cannot	easily	mold	the	synaptic
connections	of	these	assemblies”	and	speculates	that	“they	serve	as	innate,	Lego-
like	building	blocks	of	knowledge	for	
perception	and	that	the	acquisition	of
memories	involves	the	combination	of	these	building	blocks	into	complex
constructs.”	He	continues:
Functional	neuronal	assemblies	have	been	reported	for	decades,	but
direct	evidence	of	clusters	of	synaptically	connected	neurons…has	been
missing….	Since	these	assemblies	will	all	be	similar	in	topology	and
synaptic	weights,	not	molded	by	any	specific	experience,	we	consider	these
to	be	innate	assemblies….	Experience	plays	only	a	minor	role	in

determining	synaptic	connections	and	weights	within	these	assemblies….
Our	study	found	evidence	[of]	innate	Lego-like	assemblies	of	a	few	dozen
neurons….	Connections	between	assemblies	may	combine	them	into	super-
assemblies	within	a	neocortical	layer,	then	in	higher-order	assemblies	in	a
cortical	column,	even	higher-order	assemblies	in	a	brain	region,	and	finally
in	the	highest	possible	order	assembly	represented	by	the	whole	brain….
Acquiring	memories	is	very	similar	to	building	with	Lego.	Each	assembly

is	equivalent	to	a	Lego	block	holding	some	piece	of	elementary	innate
knowledge	about	how	to	process,	perceive	and	respond	to	the	world….
When	different	blocks	come	together,	they	therefore	form	a	unique
combination	of	these	innate	percepts	that	represents	an	individual’s	specific
knowledge	and	experience.
3
	
The	“Lego	blocks”	that	Markram	proposes	are	fully	consistent	with	the
pattern	recognition	modules	that	I	have	described.	In	an	e-mail	communication,

3
	
Markram	described	these	“Lego	blocks”	as	“shared	content	and	innate
knowledge.”
4
	I	would	articulate	that	the	purpose	of	these	modules	is	to	recognize
patterns,	to	remember	them,	and	to	predict	them	based	on	partial	patterns.	Note
that	Markram’s	estimate	of	each	module’s	containing	“several	dozen	neurons”	is
based	only	on	layer	V	of	the	neocortex.	Layer	V	is	indeed	neuron	rich,	but	based
on	the	usual	ratio	of	neuron	counts	in	the	six	layers,	this	would	translate	to	an
order	of	magnitude	of	about	100	neurons	per	module,	which	is	consistent	with

4
my	estimates.
The	consistent	wiring	and	apparent	modularity	of	the	neocortex	has	been

noted	for	many	years,	but	this	study	is	the	first	to	demonstrate	the	stability	of
these	modules	as	the	brain	undergoes	its	dynamic	processes.
Another	recent	study,	this	one	from	Massachusetts	General	Hospital,
funded	by	the	National	Institutes	of	Health	and	the	National	Science	Foundation
and	published	in	a	March	2012	issue	of	the	journal	
Science
,	also	shows	a	regular
structure	of	connections	across	the	neocortex.
5
	The	article	describes	the	wiring
of	the	neocortex	as	following	a	grid	pattern,	like	orderly	city	streets:	“Basically,
the	overall	structure	of	the	brain	ends	up	resembling	Manhattan,	where	you	have
a	2-D	plan	of	streets	and	a	third	axis,	an	elevator	going	in	the	third	dimension,”
wrote	Van	J.	Wedeen,	a	Harvard	neuroscientist	and	physicist	and	the	head	of	the
study.
In	a	
Science
	magazine	podcast,	Wedeen	described	the	significance	of	the
research:	“This	was	an	investigation	of	the	three-dimensional	structure	of	the
pathways	of	the	brain.	When	scientists	have	thought	about	the	pathways	of	the

study.
In	a	
Science
brain	for	the	last	hundred	years	or	so,	the	typical	image	or	model	that	comes	to
mind	is	that	these	pathways	might	resemble	a	bowl	of	spaghetti—separate
pathways	that	have	little	particular	spatial	pattern	in	relation	to	one	another.
Using	magnetic	resonance	imaging,	we	were	able	to	investigate	this	question
experimentally.	And	what	we	found	was	that	rather	than	being	haphazardly
arranged	or	independent	pathways,	we	find	that	all	of	the	pathways	of	the	brain
taken	together	fit	together	in	a	single	exceedingly	simple	structure.	They

basically	look	like	a	cube.	They	basically	run	in	three	perpendicular	directions,
and	in	each	one	of	those	three	directions	the	pathways	are	highly	parallel	to	each
other	and	arranged	in	arrays.	So,	instead	of	independent	spaghettis,	we	see	that
the	connectivity	of	the	brain	is,	in	a	sense,	a	single	coherent	structure.”
Whereas	the	Markram	study	shows	a	module	of	neurons	that	repeats	itself
across	the	neocortex,	the	Wedeen	study	demonstrates	a	remarkably	orderly
pattern	of	connections	between	modules.	The	brain	starts	out	with	a	very	large

number	of	“connections-in-waiting”	to	which	the	pattern	recognition	modules
can	hook	up.	Thus	if	a	given	module	wishes	to	connect	
to	another,	it	does	not
need	to	grow	an	axon	from	one	and	a	dendrite	from	the	other	to	span	the	entire
physical	distance	between	them.	It	can	simply	harness	one	of	these	axonal
connections-in-waiting	and	just	hook	up	to	the	ends	of	the	fiber.	As	Wedeen	and
his	colleagues	write,	“The	pathways	of	the	brain	follow	a	base-plan	established
by…early	embryogenesis.	Thus,	the	pathways	of	the	mature	brain	present	an

image	of	these	three	primordial	gradients,	physically	deformed	by
development.”	In	other	words,	as	we	learn	and	have	experiences,	the	pattern
recognition	modules	of	the	neocortex	are	connecting	to	these	preestablished
connections	that	were	created	when	we	were	embryos.

There	is	a	type	of	electronic	chip	called	a	field	programmable	gate	array
(FPGA)	that	is	based	on	a	similar	principle.	The	chip	contains	millions	of
modules	that	implement	logic	functions	along	with	connections-in-waiting.	At
the	time	of	use,	these	connections	are	either	activated	or	deactivated	(through
electronic	signals)	to	implement	a	particular	capability.
In	the	neocortex,	those	long-distance	connections	that	are	not	used	are
eventually	pruned	away,	which	is	one	reason	why	adapting	a	nearby	region	of
the	neocortex	to	compensate	for	one	that	has	become	damaged	is	not	quite	as
effective	as	using	the	original	region.	According	to	the	Wedeen	study,	the	initial
connections	are	extremely	orderly	and	repetitive,	just	like	the	modules
themselves,	and	their	grid	pattern	is	used	to	“guide	connectivity”	in	the
neocortex.	This	pattern	was	found	in	all	of	the	primate	and	human	brains	studied
and	was	evident	across	the	neocortex,	from	regions	that	dealt	with	early	sensory

patterns	up	to	higher-level	emotions.	Wedeen’s	
Science
	journal	article	concluded
that	the	“grid	structure	of	cerebral	pathways	was	pervasive,	coherent,	and
continuous	with	the	three	principal	axes	of	development.”	This	again	speaks	to	a
common	algorithm	across	all	neocortical	functions.
It	has	long	been	known	that	at	least	certain	regions	of	the	neocortex	are
hierarchical.	The	best-studied	region	is	the	visual	cortex,	which	is	separated	into
areas	known	as	V1,	V2,	and	MT	(also	known	as	V5).	As	we	advance	to	higher

Science
areas	in	this	region	(“higher”	in	the	sense	of	conceptual	processing,	not
physically,	as	the	neocortex	is	always	just	one	pattern	recognizer	thick),	the
properties	that	can	be	recognized	become	more	abstract.	V1	recognizes	very
basic	edges	and	primitive	shapes.	V2	can	recognize	contours,	the	disparity	of
images	presented	by	each	of	the	eyes,	spatial	orientation,	and	whether	or	not	a
portion	of	the	image	is	part	of	an	object	or	the	background.
6
	Higher-level	regions

6
of	the	neocortex	recognize	concepts	such	as	the	identity	of	objects	and	faces	and
their	movement.	It	has	also	long	been	known	that	communication	through	this
hierarchy	is	both	upward	and	downward,	and	that	signals	can	be	both	excitatory
and	inhibitory.	MIT	neuroscientist	Tomaso	Poggio	(born	in	1947)	has
extensively	studied	vision	in	the	human	brain,	and	his	research	for	the	last	thirty-
five	years	has	been	instrumental	in	establishing	hierarchical	learning	and	pattern
recognition	in	the	“early”	(lowest	conceptual)	levels	of	the	visual	neocortex.

7

The	highly	regular	grid	structure	of	initial	connections	in	the	neocortex
found	in	a	National	Institutes	of	Health	study.
Another	view	of	the	regular	grid	structure	of	neocortical	connections.

The	grid	structure	found	in	the	neocortex	is	remarkably	similar	to	what
is	called	crossbar	switching,	which	is	used	in	integrated	circuits	and	circuit
boards.
Our	understanding	of	the	lower	hierarchical	levels	of	the	visual	neocortex	is
consistent	with	the	PRTM	I	described	in	the	
,	and	observation	of
the	hierarchical	nature	of	neocortical	processing	has	recently	extended	far
beyond	these	levels.	University	of	Texas	neurobiology	professor	Daniel	J.
Felleman	and	his	colleagues	traced	the	“hierarchical	organization	of	the	cerebral
cortex…[in]	25	neocortical	areas,”	which	included	both	visual	areas	and	higher-
level	areas	that	combine	patterns	from	multiple	senses.	What	they	found	as	they
went	up	the	neocortical	hierarchy	was	that	the	processing	of	patterns	became
more	abstract,	comprised	larger	spatial	areas,	and	involved	longer	time	periods.
With	every	connection	they	found	communication	both	up	and	down	the
hierarchy.
8
Recent	research	allows	us	to	substantially	broaden	these	observations	to

8
regions	well	beyond	the	visual	cortex	and	even	to	the	association	areas,	which
combine	inputs	from	multiple	senses.	A	study	published	in	2008	by	Princeton
psychology	professor	Uri	Hasson	and	his	colleagues	demonstrates	that	the
phenomena	observed	in	the	visual	cortex	occur	across	a	wide	variety	of
neocortical	areas:	“It	is	well	established	that	neurons	along	the	visual	cortical
pathways	have	increasingly	larger	spatial	receptive	fields.	This	is	a	basic
organizing	principle	of	the	visual	system….	Real-world	events	occur	not	only

over	extended	regions	of	space,	but	also	over	extended	periods	of	time.	We
therefore	hypothesized	that	a	hierarchy	analogous	to	that	found	for	spatial
receptive	field	sizes	should	also	exist	for	the	temporal	response	characteristics	of

different	brain	regions.”	This	is	exactly	what	they	found,	which	enabled	them	to
conclude	that	“similar	to	the	known	cortical	hierarchy	of	spatial	receptive	fields,
there	is	a	hierarchy	of	progressively	longer	temporal	receptive	windows	in	the
human	brain.”
9
The	most	powerful	argument	for	the	universality	of	processing	in	the
neocortex	is	the	pervasive	evidence	of	plasticity	(not	just	learning	but
interchangeability):	In	other	words,	one	region	is	able	to	do	the	work	of	other
regions,	implying	a	common	algorithm	across	the	entire	neocortex.	A	great	deal
of	neuroscience	research	has	been	focused	on	identifying	which	regions	of	the
neocortex	are	responsible	for	which	types	of	patterns.	
The	classical	technique	for
determining	this	has	been	to	take	advantage	of	brain	damage	from	injury	or
stroke	and	to	correlate	lost	functionality	with	specific	damaged	regions.	So,	for
example,	when	we	notice	that	someone	with	newly	acquired	damage	to	the

fusiform	gyrus	region	suddenly	has	difficulty	recognizing	faces	but	is	still	able
to	identify	people	from	their	voices	and	language	patterns,	we	can	hypothesize
that	this	region	has	something	to	do	with	face	recognition.	The	underlying
assumption	has	been	that	each	of	these	regions	is	designed	to	recognize	and
process	a	particular	type	of	pattern.	Particular	physical	regions	have	become
associated	with	particular	types	of	patterns,	because	under	normal	circumstances
that	is	how	the	information	happens	to	flow.	But	when	that	normal	flow	of

information	is	disrupted	for	any	reason,	another	region	of	the	neocortex	is	able	to
step	in	and	take	over.
Plasticity	has	been	widely	noted	by	neurologists,	who	observed	that	patients
with	brain	damage	from	an	injury	or	a	stroke	can	relearn	the	same	skills	in
another	area	of	the	neocortex.	Perhaps	the	most	dramatic	example	of	plasticity	is
a	2011	study	by	American	neuroscientist	Marina	Bedny	and	her	colleagues	on
what	happens	to	the	visual	cortex	of	congenitally	blind	people.	The	common

wisdom	has	been	that	the	early	layers	of	the	visual	cortex,	such	as	V1	and	V2,
inherently	deal	with	very	low-level	patterns	(such	as	edges	and	curves),	whereas
the	frontal	cortex	(that	evolutionarily	new	region	of	the	cortex	that	we	have	in
our	uniquely	large	foreheads)	inherently	deals	with	the	far	more	complex	and
subtle	patterns	of	language	and	other	abstract	concepts.	But	as	Bedny	and	her
colleagues	found,	“Humans	are	thought	to	have	evolved	brain	regions	in	the	left

frontal	and	temporal	cortex	that	are	uniquely	capable	of	language	processing.
However,	congenitally	blind	individuals	also	activate	the	visual	cortex	in	some
verbal	tasks.	We	provide	evidence	that	this	visual	cortex	activity	in	fact	reflects
language	processing.	We	find	that	in	congenitally	blind	individuals,	the	left
visual	cortex	behaves	similarly	to	classic	language	regions….	We	conclude	that
brain	regions	that	are	thought	to	have	evolved	for	vision	can	take	on	language

processing	as	a	result	of	early	experience.”
10
Consider	the	implications	of	this	study:	It	means	that	neocortical	regions
that	are	physically	relatively	far	apart,	and	that	have	also	been	considered
conceptually	very	different	(primitive	visual	cues	versus	abstract	language
concepts),	use	essentially	the	same	algorithm.	The	regions	that	process	these
disparate	types	of	patterns	can	substitute	for	one	another.
University	of	California	at	Berkeley	neuroscientist	Daniel	E.	Feldman
wrote	a	comprehensive	2009	review	of	what	he	called	“synaptic	mechanisms	for
plasticity	in	the	neocortex”	and	found	evidence	for	this	type	of	plasticity	across
the	neocortex.	He	writes	that	“plasticity	allows	the	brain	to	learn	and	remember
patterns	in	the	sensory	world,	to	refine	movements…and	to	recover	function
after	injury.”	He	adds	that	this	plasticity	is	enabled	by	“structural	changes
including	formation,	removal,	and	morphological	remodeling	of	cortical
synapses	and	dendritic	spines.”
11

11
Another	startling	example	of	neocortical	plasticity	(and	therefore	of	the
uniformity	of	the	neocortical	algorithm)	was	recently	demonstrated	by	scientists
at	the	University	of	California	at	Berkeley.	They	hooked	up	implanted
microelectrode	arrays	to	pick	up	brain	signals	specifically	from	a	region	of	the
motor	cortex	of	mice	that	controls	the	movement	of	their	whiskers.	They	set	up
their	experiment	so	that	the	mice	would	get	a	reward	if	they	controlled	these

11
neurons	to	fire	in	a	certain	mental	pattern	but	not	to	actually	move	their
whiskers.	The	pattern	required	to	get	the	reward	involved	a	mental	task	that	their
frontal	neurons	would	normally	not	do.	The	mice	were	nonetheless	able	to
perform	this	mental	feat	essentially	by	thinking	with	their	motor	neurons	while
mentally	decoupling	them	from	controlling	motor	movements.
12
	The	conclusion
is	that	the	motor	cortex,	the	region	of	the	neocortex	responsible	for	coordinating

12
muscle	movement,	also	uses	the	standard	neocortical	algorithm.
There	are	several	reasons,	however,	why	a	skill	or	an	area	of	knowledge
that	has	been	relearned	using	a	new	area	of	the	neocortex	to	replace	one	that	has
been	damaged	will	not	necessarily	be	as	good	as	the	original.	First,	because	it
took	an	entire	lifetime	to	learn	and	perfect	a	given	skill,	relearning	it	in	another
area	of	the	neocortex	will	not	immediately	generate	the	
same	results.	More
important,	that	new	area	of	the	neocortex	has	not	just	been	sitting	around	waiting

as	a	standby	for	an	injured	region.	It	too	has	been	carrying	out	vital	functions,
and	will	therefore	be	hesitant	to	give	up	its	neocortical	patterns	to	compensate
for	the	damaged	region.	It	can	start	by	releasing	some	of	the	redundant	copies	of
its	patterns,	but	doing	so	will	subtly	degrade	its	existing	skills	and	does	not	free
up	as	much	cortical	space	as	the	skills	being	relearned	had	used	originally.

There	is	a	third	reason	why	plasticity	has	its	limits.	Since	in	most	people
particular	types	of	patterns	will	flow	through	specific	regions	(such	as	faces
being	processed	by	the	fusiform	gyrus),	these	regions	have	become	optimized
(by	biological	evolution)	for	those	types	of	patterns.	As	I	report	in	
chapter	7
,	we
found	the	same	result	in	our	digital	neocortical	developments.	We	could
recognize	speech	with	our	character	recognition	systems	and	vice	versa,	but	the
speech	systems	were	optimized	for	speech	and	similarly	the	character
recognition	systems	were	optimized	for	printed	characters,	so	there	would	be
some	reduction	in	performance	if	we	substituted	one	for	the	other.	We	actually
used	evolutionary	(genetic)	algorithms	to	accomplish	this	optimization,	a
simulation	of	what	biology	does	naturally.	Given	that	faces	have	been	flowing
through	the	fusiform	gyrus	for	most	people	for	hundreds	of	thousands	of	years
(or	more),	biological	evolution	has	had	time	to	evolve	a	favorable	ability	to

process	such	patterns	in	that	region.	It	uses	the	same	basic	algorithm,	but	it	is
oriented	toward	faces.	As	Dutch	neuroscientist	Randal	Koene	wrote,	“The
[neo]cortex	is	very	uniform,	each	column	or	minicolumn	can	in	principle	do
what	each	other	one	can	do.”
13
Substantial	recent	research	supports	the	observation	that	the	pattern
recognition	modules	wire	themselves	based	on	the	patterns	to	which	they	are
exposed.	For	example,	neuroscientist	Yi	Zuo	and	her	colleagues	watched	as	new

13
“dendritic	spines”	formed	connections	between	nerve	cells	as	mice	learned	a
new	skill	(reaching	through	a	slot	to	grab	a	seed).
14
	Researchers	at	the	Salk
Institute	have	discovered	that	this	critical	self-wiring	of	the	neocortex	modules	is
apparently	controlled	by	only	a	handful	
of	genes.	These	genes	and	this	method
of	self-wiring	are	also	uniform	across	the	neocortex.
15
Many	other	studies	document	these	attributes	of	the	neocortex,	but	let’s
summarize	what	we	can	observe	from	the	neuroscience	literature	and	from	our

14
15
own	thought	experiments.	The	basic	unit	of	the	neocortex	is	a	module	of
neurons,	which	I	estimate	at	around	a	hundred.	These	are	woven	together	into
each	neocortical	column	so	that	each	module	is	not	visibly	distinct.	The	pattern
of	connections	and	synaptic	strengths	within	each	module	is	relatively	stable.	It
is	the	connections	and	synaptic	strengths	
between
	modules	that	represent
learning.
There	are	on	the	order	of	a	quadrillion	(10
15
)	connections	in	the	neocortex,
yet	only	about	25	million	bytes	of	design	information	in	the	genome	(after
lossless	compression),

between
learning.
15
16
	so	the	connections	themselves	cannot	possibly	be
predetermined	genetically.	It	is	possible	that	some	of	this	learning	is	the	product
of	the	neocortex’s	interrogating	the	old	brain,	but	that	still	would	necessarily

represent	only	a	relatively	small	amount	of	information.	The	connections
between	modules	are	created	on	the	whole	from	experience	(nurture	rather	than
nature).
The	brain	does	not	have	sufficient	flexibility	so	that	each	neocortical	pattern
recognition	module	can	simply	link	to	any	other	module	(as	we	can	easily
program	in	our	computers	or	on	the	Web)—an	actual	physical	connection	must
be	made,	composed	of	an	axon	connecting	to	a	dendrite.	We	each	start	out	with	a
vast	stockpile	of	possible	neural	connections.	As	the	Wedeen	study	shows,	these
connections	are	organized	in	a	very	repetitive	and	orderly	manner.	Terminal
connection	to	these	axons-in-waiting	takes	place	based	on	the	patterns	that	each
neocortical	pattern	recognizer	has	recognized.	Unused	connections	are
ultimately	pruned	away.	These	connections	are	built	hierarchically,	reflecting	the
natural	hierarchical	order	of	reality.	That	is	the	key	strength	of	the	neocortex.
The	basic	algorithm	of	the	neocortical	pattern	recognition	modules	is

equivalent	across	the	neocortex	from	“low-level”	modules,	which	deal	with	the
most	basic	sensory	patterns,	to	“high-level”	modules,	which	recognize	
the	most
abstract	concepts.	The	vast	evidence	of	plasticity	and	the	interchangeability	of
neocortical	regions	is	testament	to	this	important	observation.	There	is	some
optimization	of	regions	that	deal	with	particular	types	of	patterns,	but	this	is	a
second-order	effect—the	fundamental	algorithm	is	universal.
Signals	go	up	and	down	the	conceptual	hierarchy.	A	signal	going	up	means,

the	most
“I’ve	detected	a	pattern.”	A	signal	going	down	means,	“I’m	expecting	your
pattern	to	occur,”	and	is	essentially	a	prediction.	Both	upward	and	downward
signals	can	be	either	excitatory	or	inhibitory.
Each	pattern	is	itself	in	a	particular	order	and	is	not	readily	reversed.	Even
if	a	pattern	appears	to	have	multidimensional	aspects,	it	is	represented	by	a	one-
dimensional	sequence	of	lower-level	patterns.	A	pattern	is	an	ordered	sequence
of	other	patterns,	so	each	recognizer	is	inherently	recursive.	There	can	be	many
levels	of	hierarchy.

There	is	a	great	deal	of	redundancy	in	the	patterns	we	learn,	especially	the
important	ones.	The	recognition	of	patterns	(such	as	common	objects	and	faces)
uses	the	same	mechanism	as	our	memories,	which	are	just	patterns	we	have
learned.	They	are	also	stored	as	sequences	of	patterns—they	are	basically
stories.	That	mechanism	is	also	used	for	learning	and	carrying	out	physical
movement	in	the	world.	The	redundancy	of	patterns	is	what	enables	us	to
recognize	objects,	people,	and	ideas	even	when	they	have	variations	and	occur	in

different	contexts.	The	size	and	size	variability	parameters	also	allow	the
neocortex	to	encode	variation	in	magnitude	against	different	dimensions
(duration	in	the	case	of	sound).	One	way	that	these	magnitude	parameters	could

be	encoded	is	simply	through	multiple	patterns	with	different	numbers	of
repeated	inputs.	So,	for	example,	there	could	be	patterns	for	the	spoken	word
“steep”	with	different	numbers	of	the	long	vowel	[E]	repeated,	each	with	the
importance	parameter	set	to	a	moderate	level	indicating	that	the	repetition	of	[E]
is	variable.	This	approach	is	not	mathematically	equivalent	to	having	the	explicit
size	parameters	and	does	not	work	nearly	as	well	in	practice,	but	is	one	approach
to	encoding	magnitude.	The	strongest	evidence	we	have	for	these	parameters	is
that	
they	are	needed	in	our	AI	systems	to	get	accuracy	levels	that	are	near	human
levels.
The	summary	above	constitutes	the	conclusions	we	can	draw	from	the
sampling	of	research	results	I	have	shared	above	as	well	as	the	sampling	of
thought	experiments	I	discussed	earlier.	I	maintain	that	the	model	I	have
presented	is	the	only	possible	model	that	satisfies	all	of	the	constraints	that	the
research	and	our	thought	experiments	have	established.

that	
levels.
Finally,	there	is	one	more	piece	of	corroborating	evidence.	The	techniques
that	we	have	evolved	over	the	past	several	decades	in	the	field	of	artificial
intelligence	to	recognize	and	intelligently	process	real-world	phenomena	(such
as	human	speech	and	written	language)	and	to	understand	natural-language
documents	turn	out	to	be	mathematically	similar	to	the	model	I	have	presented
above.	They	are	also	examples	of	the	PRTM.	The	AI	field	was	not	explicitly
trying	to	copy	the	brain,	but	it	nonetheless	arrived	at	essentially	equivalent
techniques.

THE	OLD	BRAIN
	
I	have	an	old	brain	but	a	terrific	memory.
—Al	Lewis
	
Here	we	stand	in	the	middle	of	this	new	world	with	our	primitive	brain,
attuned	to	the	simple	cave	life,	with	terrific	forces	at	our	disposal,	which	we
are	clever	enough	to	release,	but	whose	consequences	we	cannot
comprehend.
—Albert	Szent-Györgyi
	
O
ur	old	brain—the	one	we	had	before	we	were	mammals—has	not
disappeared.	Indeed	it	still	provides	much	of	our	motivation	in	seeking
gratification	and	avoiding	danger.	These	goals	are	modulated,	however,	by	our
neocortex,	which	dominates	the	human	brain	in	both	mass	and	activity.
Animals	used	to	live	and	survive	without	a	neocortex,	and	indeed	all
nonmammalian	animals	continue	to	do	so	today.	We	can	view	the	human
neocortex	as	the	great	sublimator—thus	our	primitive	motivation	to	avoid	a	large
predator	may	be	transformed	by	the	neocortex	today	into	completing	an
assignment	to	impress	our	boss;	the	great	hunt	may	become	writing	
a	book	on,

a	book	on,
say,	the	mind;	and	pursuing	reproduction	may	become	gaining	public
recognition	or	decorating	your	apartment.	(Well,	this	last	motivation	is	not
always	so	hidden.)
The	neocortex	is	likewise	good	at	helping	us	solve	problems	because	it	can
accurately	model	the	world,	reflecting	its	true	hierarchical	nature.	But	it	is	the
old	brain	that	presents	us	with	those	problems.	Of	course,	like	any	clever
bureaucracy,	the	neocortex	often	deals	with	the	problems	it	is	assigned	by
redefining	them.	On	that	note,	let’s	review	the	information	processing	in	the	old
brain.

The	Sensory	Pathway
Pictures,	propagated	by	motion	along	the	fibers	of	the	optic	nerves	in	the
brain,	are	the	cause	of	vision.
—Isaac	Newton
	
Each	of	us	lives	within	the	universe—the	prison—of	his	own	brain.
Projecting	from	it	are	millions	of	fragile	sensory	nerve	fibers,	in	groups
uniquely	adapted	to	sample	the	energetic	states	of	the	world	around	us:
heat,	light,	force,	and	chemical	composition.	That	is	all	we	ever	know	of	it
directly;	all	else	is	logical	inference.
—Vernon	Mountcastle
1
	
Although	we	experience	the	illusion	of	receiving	high-resolution	images	from
our	eyes,	what	the	optic	nerve	actually	sends	to	the	brain	is	just	a	series	of
outlines	and	clues	about	points	of	interest	in	our	visual	field.	We	then	essentially
hallucinate	the	world	from	cortical	memories	that	interpret	a	series	of	movies
with	very	low	data	rates	that	arrive	in	parallel	channels.	In	a	study	published	in
Nature
,	Frank	S.	Werblin,	professor	of	molecular	and	cell	biology	at	the

1
	
Nature
University	of	California	at	Berkeley,	and	doctoral	student	Boton	Roska,	MD,
showed	that	the	optic	nerve	carries	ten	to	twelve	
output	channels,	each	of	which
carries	only	a	small	amount	of	information	about	a	given	scene.
2
	One	group	of
what	are	called	ganglion	cells	sends	information	only	about	edges	(changes	in
contrast).	Another	group	detects	only	large	areas	of	uniform	color,	whereas	a
third	group	is	sensitive	only	to	the	backgrounds	behind	figures	of	interest.

The	visual	pathway	in	the	brain.
“Even	though	we	think	we	see	the	world	so	fully,	what	we	are	receiving	is
really	just	hints,	edges	in	space	and	time,”	says	Werblin.	“These	12	pictures	of
the	world	constitute	all	the	information	we	will	ever	have	about	what’s	out	there,
and	from	these	12	pictures,	which	are	so	sparse,	we	reconstruct	the	richness	of
the	visual	world.	I’m	curious	how	nature	selected	these	12	simple	movies	and
how	it	can	be	that	they	are	sufficient	to	provide	us	with	all	the	information	we
seem	to	need.”
This	data	reduction	is	what	in	the	AI	field	we	call	“sparse	coding.”	We	have
found	in	creating	artificial	systems	that	throwing	most	of	the	input	information
away	and	retaining	only	the	most	salient	details	provides	
superior	results.
Otherwise	the	limited	ability	to	process	information	in	a	neocortex	(biological	or
otherwise)	gets	overwhelmed.

Seven	of	the	twelve	low-data-rate	“movies”	sent	by	the	optic	nerve	to
the	brain.
The	processing	of	auditory	information	from	the	human	cochlea	through	the
subcortical	regions	and	then	through	the	early	stages	of	the	neocortex	has	been
meticulously	modeled	by	Lloyd	Watts	and	his	research	team	at	Audience,	Inc.
3
They	have	developed	research	technology	that	extracts	600	different	frequency
bands	(60	per	octave)	from	sound.	This	comes	much	closer	to	the	estimate	of
3,000	bands	extracted	by	the	human	cochlea	(compared	with	commercial	speech
recognition,	which	uses	only	16	to	32	bands).	Using	two	microphones	and	its
detailed	(and	high–spectral	resolution)	model	of	auditory	processing,	Audience
has	created	a	commercial	technology	(with	somewhat	lower	spectral	resolution
than	its	research	system)	that	effectively	removes	background	noise	from
conversations.	This	is	now	being	used	in	many	popular	cell	phones	and	is	an
impressive	
example	of	a	commercial	product	based	on	an	understanding	of	how

the	human	auditory	perceptual	system	is	able	to	focus	on	one	sound	source	of
interest.

The	auditory	pathway	in	the	brain.
Inputs	from	the	body	(estimated	at	hundreds	of	megabits	per	second),
including	that	of	nerves	from	the	skin,	muscles,	organs,	and	other	areas,	stream
into	the	upper	spinal	cord.	These	messages	involve	more	than	just
communication	about	touch;	in	addition	they	carry	information	about
temperature,	acid	levels	(for	example,	lactic	acid	in	muscles),	the	movement	of
food	through	the	gastrointestinal	tract,	and	many	other	signals.	This	data	is
processed	through	the	brain	stem	and	midbrain.	Key	cells	called	lamina	1
neurons	create	a	map	of	the	body,	representing	its	current	state,	not	unlike	the
displays	used	by	flight	controllers	to	track	airplanes.	From	here	the	sensory	data
heads	to	a	mysterious	region	called	the	thalamus,	which	brings	us	to	our	next
topic.

A	simplified	model	of	auditory	processing	in	both	the	subcortical	areas
(areas	prior	to	the	neocortex)	and	the	neocortex,	created	by	Audience,	Inc.
Figure	adapted	from	L.	Watts,	“Reverse-Engineering	the	Human	Auditory
Pathway,”	in	J.	Liu	
et	al.
	(eds.),	
WCCI	2012
	(Berlin:	Springer-Verlag,
2012),	p.	49.

The	Thalamus
Everyone	knows	what	attention	is.	It	is	the	taking	possession	by	the	mind,
in	clear	and	vivid	form,	of	one	out	of	what	seem	several	simultaneously
possible	objects	or	trains	of	thought.	Focalization,	concentration,	of
consciousness,	are	of	its	essence.	It	implies	withdrawal	from	some	things	in
order	to	deal	effectively	with	others.
	
From	the	midbrain,	sensory	information	then	flows	through	a	nut-sized	region
called	the	posterior	ventromedial	nucleus	(VMpo)	of	the	thalamus,	which
computes	complex	reactions	to	bodily	states	such	as	“this	tastes	terrible,”	“what
a	stench,”	or	“that	light	touch	is	stimulating.”	The	increasingly	processed
information	ends	up	at	two	regions	of	the	neocortex	called	the	insula.	These
structures,	the	size	of	small	fingers,	are	located	on	the	left	and	
right	sides	of	the
neocortex.	Dr.	Arthur	Craig	of	the	Barrow	Neurological	Institute	in	Phoenix
describes	the	VMpo	and	the	two	insula	regions	as	“a	system	that	represents	the
material	me.”
4

The	sensory-touch	pathway	in	the	brain.
Among	its	other	functions,	the	thalamus	is	considered	a	gateway	for
preprocessed	sensory	information	to	enter	the	neocortex.	In	addition	to	the	tactile
information	flowing	through	the	VMpo,	processed	information	from	the	optic
nerve	(which,	as	noted	above,	has	already	been	substantially	transformed)	is	sent
to	a	region	of	the	thalamus	called	the	lateral	geniculate	nucleus,	which	then
sends	it	on	to	the	V1	region	of	the	neocortex.	Information	from	the	auditory
sense	is	passed	through	the	medial	geniculate	nucleus	of	the	thalamus	en	route	to
the	early	auditory	regions	of	the	neocortex.	All	of	our	sensory	data	(except,
apparently,	for	the	olfactory	system,	which	uses	the	olfactory	bulb	instead)
passes	through	specific	regions	of	the	thalamus.
The	most	significant	role	of	the	thalamus,	however,	is	its	continual
communication	with	the	neocortex.	The	pattern	recognizers	in	the	neocortex
send	tentative	results	to	the	thalamus	and	receive	responses	principally	using

both	excitatory	and	inhibitory	reciprocal	signals	from	layer	VI	of	each
recognizer.	Keep	in	mind	that	these	are	not	wireless	messages,	so	that	there
needs	to	be	an	extraordinary	amount	of	actual	wiring	(in	the	form	of	axons)
running	between	all	regions	of	the	neocortex	and	the	thalamus.	Consider	the	vast
amount	of	real	estate	(in	terms	of	the	physical	mass	of	connections	required)	for
the	hundreds	of	millions	of	pattern	recognizers	in	the	neocortex	to	be	constantly

checking	in	with	the	thalamus.
5
So	what	are	the	hundreds	of	millions	of	neocortical	pattern	recognizers
talking	to	the	thalamus	about?	It	is	apparently	an	important	conversation,
because	profound	damage	to	the	main	region	of	the	thalamus	bilaterally	can	lead
to	prolonged	unconsciousness.	A	person	with	a	damaged	thalamus	may	still	have
activity	in	his	neocortex,	in	that	the	self-triggering	thinking	by	association	can
still	work.	But	directed	thinking—the	kind	that	will	get	us	out	of	bed,	into	our
car,	and	sitting	at	our	desk	at	work—does	not	function	without	a	thalamus.	In	a
famous	case,	twenty-one-year-old	
Karen	Ann	Quinlan	suffered	a	heart	attack	and
respiratory	failure	and	remained	in	an	unresponsive,	apparently	vegetative	state
for	ten	years.	When	she	died,	her	autopsy	revealed	that	her	neocortex	was
normal	but	her	thalamus	had	been	destroyed.
In	order	to	play	its	key	role	in	our	ability	to	direct	attention,	the	thalamus
relies	on	the	structured	knowledge	contained	in	the	neocortex.	It	can	step

through	a	list	(stored	in	the	neocortex),	enabling	us	to	follow	a	train	of	thought
or	follow	a	plan	of	action.	We	are	apparently	able	to	keep	up	to	about	four	items
in	our	working	memory	at	a	time,	two	per	hemisphere	according	to	recent
research	by	neuroscientists	at	the	MIT	Picower	Institute	for	Learning	and
Memory.
6
	The	issue	of	whether	the	thalamus	is	in	charge	of	the	neocortex	or
vice	versa	is	far	from	clear,	but	we	are	unable	to	function	without	both.

The	Hippocampus
Each	brain	hemisphere	contains	a	hippocampus,	a	small	region	that	looks	like	a
sea	horse	tucked	in	the	medial	temporal	lobe.	Its	primary	function	is	to
remember	novel	events.	Since	sensory	information	flows	through	the	neocortex,
it	is	up	to	the	neocortex	to	determine	that	an	experience	is	novel	in	order	to
present	it	to	the	hippocampus.	It	does	so	either	by	failing	to	recognize	a
particular	set	of	features	(for	example,	a	new	face)	or	by	realizing	that	an
otherwise	familiar	situation	now	has	unique	attributes	(such	as	your	spouse’s
wearing	a	fake	mustache).
The	hippocampus	is	capable	of	remembering	these	situations,	although	it
appears	to	do	so	primarily	through	pointers	into	the	neocortex.	So	memories	in
the	hippocampus	are	also	stored	as	lower-level	patterns	that	were	earlier
recognized	and	stored	in	the	neocortex.	For	animals	without	a	neocortex	to
modulate	sensory	experiences,	the	hippocampus	will	simply	remember	the
information	from	the	senses,	although	this	will	have	undergone	sensory

preprocessing	(for	example,	the	transformations	performed	by	the	optic	nerve).
Although	the	hippocampus	makes	use	of	the	neocortex	(if	a	particular	brain
has	one)	as	its	scratch	pad,	its	memory	(of	pointers	into	the	neocortex)	is	not
inherently	hierarchical.	Animals	without	a	neocortex	can	accordingly	remember
things	using	their	hippocampus,	but	their	recollections	will	not	be	hierarchical.
The	capacity	of	the	hippocampus	is	limited,	so	its	memory	is	short-term.	It
will	transfer	a	particular	sequence	of	patterns	from	its	short-term	memory	to	the

long-term	hierarchical	memory	of	the	neocortex	by	playing	this	memory
sequence	to	the	neocortex	over	and	over	again.	We	need,	therefore,	a
hippocampus	in	order	to	learn	new	memories	and	skills	(although	strictly	motor
skills	appear	to	use	a	different	mechanism).	Someone	with	damage	to	both
copies	of	her	hippocampus	will	retain	her	existing	memories	but	will	not	be	able
to	form	new	ones.
University	of	Southern	California	neuroscientist	Theodore	Berger	and	his
colleagues	modeled	the	hippocampus	of	a	rat	and	have	successfully

experimented	with	implanting	an	artificial	one.	In	a	study	reported	in	2011,	the
USC	scientists	blocked	particular	learned	behaviors	in	rats	with	drugs.	Using	an
artificial	hippocampus,	the	rats	were	able	to	quickly	relearn	the	behavior.	“Flip
the	switch	on,	and	the	rats	remember.	Flip	it	off	and	the	rats	forget,”	Berger
wrote,	referring	to	his	ability	to	control	the	neural	implant	remotely.	In	another

experiment	the	scientists	allowed	their	artificial	hippocampus	to	work	alongside
the	rats’	natural	one.	The	result	was	that	the	ability	of	the	rats	to	learn	new
behaviors	strengthened.	“These	integrated	experimental	modeling	studies	show
for	the	first	time,”	Berger	explained,	“that…a	neural	prosthesis	capable	of	real-
time	identification	and	manipulation	of	the	encoding	process	can	restore	and
even	enhance	cognitive	mnemonic	processes.”
7
	The	hippocampus	is	one	of	the
first	regions	damaged	by	Alzheimer’s,	so	one	goal	of	this	research	is	to	develop
a	neural	implant	for	humans	that	will	mitigate	this	first	phase	of	damage	from
the	disease.

The	Cerebellum
There	are	two	approaches	you	can	use	to	catch	a	fly	ball.	You	could	solve	the
complex	simultaneous	differential	equations	controlling	the	ball’s	movement	as
well	as	further	equations	governing	your	own	particular	angle	in	viewing	the
ball,	and	then	compute	even	more	equations	on	how	to	move	your	body,	arm,
and	hand	to	be	in	the	right	place	at	the	right	time.
This	is	not	the	approach	that	your	brain	adopts.	It	basically	simplifies	the
problem	by	collapsing	a	lot	of	equations	into	a	simple	trend	model,	considering
the	trends	of	where	the	ball	appears	to	be	in	your	field	of	vision	and	how	quickly
it	is	moving	within	it.	It	does	the	same	thing	with	your	hand,	making	essentially
linear	predictions	of	the	ball’s	apparent	position	in	your	field	of	view	and	that	of
your	hand.	The	goal,	of	course,	is	to	make	sure	they	meet	at	the	same	point	in
space	and	time.	If	the	ball	appears	to	be	dropping	too	quickly	and	your	hand
appears	to	be	moving	too	slowly,	your	brain	will	direct	your	hand	to	move	more

quickly,	so	that	the	trends	will	coincide.	This	“Gordian	knot”	solution	to	what
would	otherwise	be	an	intractable	mathematical	problem	is	called	basis
functions,	and	they	are	carried	out	by	the	cerebellum,	a	bean-shaped	and
appropriately	baseball-sized	region	that	sits	on	the	brain	stem.
8
The	cerebellum	is	an	old-brain	region	that	once	controlled	virtually	all
hominid	movements.	It	still	contains	half	of	the	neurons	in	the	brain,	although
most	are	relatively	small	ones,	so	the	region	constitutes	only	about	10	percent	of

8
the	weight	of	the	brain.	The	cerebellum	likewise	represents	another	instance	of
massive	repetition	in	the	design	of	the	brain.	There	is	relatively	little	information
about	its	design	in	the	genome,	as	its	structure	is	a	pattern	of	several	neurons	that
is	repeated	billions	of	times.	As	with	the	neocortex,	there	is	uniformity	across	its
structure.
9
Most	of	the	function	of	controlling	our	muscles	has	been	taken	over	by	the
neocortex,	using	the	same	pattern	recognition	algorithms	that	it	uses	for
perception	and	cognition.	In	the	case	of	movement,	we	can	more	appropriately

9
refer	to	the	neocortex’s	function	as	pattern	implementation.	The	
neocortex	does
make	use	of	the	memory	in	the	cerebellum	to	record	delicate	scripts	of
movements—for	example,	your	signature	and	certain	flourishes	in	artistic
expression	such	as	music	and	dance.	Studies	of	the	role	of	the	cerebellum	during
the	learning	of	handwriting	by	children	reveal	that	the	Purkinje	cells	of	the
cerebellum	actually	sample	the	sequence	of	movements,	with	each	one	sensitive

to	a	specific	sample.
10
	Because	most	of	our	movement	is	now	controlled	by	the
neocortex,	many	people	can	manage	with	a	relatively	modest	obvious	disability
even	with	significant	damage	to	the	cerebellum,	except	that	their	movements
may	become	less	graceful.
The	neocortex	can	also	call	upon	the	cerebellum	to	use	its	ability	to
compute	real-time	basis	functions	to	anticipate	what	the	results	of	actions	would
be	that	we	are	considering	but	have	not	yet	carried	out	(and	may	never	carry
out),	as	well	as	the	actions	or	possible	actions	of	others.	It	is	another	example	of
the	innate	built-in	linear	predictors	in	the	brain.
Substantial	progress	has	been	made	in	simulating	the	cerebellum	with
respect	to	the	ability	to	respond	dynamically	to	sensory	cues	using	the	basis
functions	I	discussed	above,	in	both	bottom-up	simulations	(based	on
biochemical	models)	and	top-down	simulations	(based	on	mathematical	models
of	how	each	repeating	unit	in	the	cerebellum	operates).
11

Pleasure	and	Fear
Fear	is	the	main	source	of	superstition,	and	one	of	the	main	sources	of
cruelty.	To	conquer	fear	is	the	beginning	of	wisdom.
—Bertrand	Russell
	
Feel	the	fear	and	do	it	anyway.
—Susan	Jeffers
	
If	the	neocortex	is	good	at	solving	problems,	then	what	is	the	main	problem	we
are	trying	to	solve?	The	problem	that	evolution	has	always	tried	to	solve	is
survival	of	the	species.	That	translates	into	the	survival	of	the	
individual,	and
each	of	us	uses	his	or	her	own	neocortex	to	interpret	that	in	myriad	ways.	In
order	to	survive,	animals	need	to	procure	their	next	meal	while	at	the	same	time
avoiding	becoming	someone	else’s	meal.	They	also	need	to	reproduce.	The
earliest	brains	evolved	pleasure	and	fear	systems	that	rewarded	the	fulfillment	of
these	fundamental	needs	along	with	basic	behaviors	that	facilitated	them.	As
environments	and	competing	species	gradually	changed,	biological	evolution
made	corresponding	alterations.	With	the	advent	of	hierarchical	thinking,	the

satisfaction	of	critical	drives	became	more	complex,	as	it	was	now	subject	to	the
vast	complex	of	ideas	within	ideas.	But	despite	its	considerable	modulation	by
the	neocortex,	the	old	brain	is	still	alive	and	well	and	still	motivating	us	with
pleasure	and	fear.
One	region	that	is	associated	with	pleasure	is	the	nucleus	accumbens.	In
famous	experiments	conducted	in	the	1950s,	rats	that	were	able	to	directly
stimulate	this	small	region	(by	pushing	a	lever	that	activated	implanted

electrodes)	preferred	doing	so	to	anything	else,	including	having	sex	or	eating,
ultimately	exhausting	and	starving	themselves	to	death.
12
	In	humans,	other
regions	are	also	involved	in	pleasure,	such	as	the	ventral	pallidum	and,	of	course,
the	neocortex	itself.
Pleasure	is	also	regulated	by	chemicals	such	as	dopamine	and	serotonin.	It
is	beyond	the	scope	of	this	book	to	discuss	these	systems	in	detail,	but	it	is
important	to	recognize	that	we	have	inherited	these	mechanisms	from	our

12
premammalian	cousins.	It	is	the	job	of	our	neocortex	to	enable	us	to	be	the

master	of	pleasure	and	fear	and	not	their	slave.	To	the	extent	that	we	are	often
subject	to	addictive	behaviors,	the	neocortex	is	not	always	successful	in	this
endeavor.	Dopamine	in	particular	is	a	neurotransmitter	involved	in	the
experience	of	pleasure.	If	anything	good	happens	to	us—winning	the	lottery,
gaining	the	recognition	of	our	peers,	getting	a	hug	from	a	loved	one,	or	even
subtle	achievements	such	as	getting	a	friend	to	laugh	at	a	joke—we	experience	a
release	of	dopamine.	Sometimes	we,	like	the	rats	who	died	overstimulating	their
nucleus	accumbens,	use	a	shortcut	to	achieve	these	bursts	of	pleasure,	which	is
not	always	a	good	idea.
Gambling,	for	example,	can	release	dopamine,	at	least	when	you	win,	but
this	is	dependent	on	its	inherent	lack	of	predictability.	Gambling	may	work	for
the	purpose	of	releasing	dopamine	for	a	while,	but	given	that	the	odds	are
intentionally	stacked	against	you	(otherwise	the	business	model	of	a	casino

wouldn’t	work),	it	can	become	ruinous	as	a	regular	strategy.	Similar	dangers	are
associated	with	any	addictive	behavior.	A	particular	genetic	mutation	of	the
dopamine-receptor	D2	gene	causes	especially	strong	feelings	of	pleasure	from
initial	experiences	with	addictive	substances	and	behaviors,	but	as	is	well	known
(but	not	always	well	heeded),	the	ability	of	these	substances	to	produce	pleasure
on	subsequent	use	gradually	declines.	Another	genetic	mutation	results	in

people’s	not	receiving	normal	levels	of	dopamine	release	from	everyday
accomplishments,	which	can	also	lead	to	seeking	enhanced	early	experiences
with	addictive	activities.	The	minority	of	the	population	that	has	these	genetic
proclivities	to	addiction	creates	an	enormous	social	and	medical	problem.	Even
those	who	manage	to	avoid	severely	addictive	behaviors	struggle	with	balancing
the	rewards	of	dopamine	release	with	the	consequences	of	the	behaviors	that
release	them.

Serotonin	is	a	neurotransmitter	that	plays	a	major	role	in	the	regulation	of
mood.	In	higher	levels	it	is	associated	with	feelings	of	well-being	and
contentment.	Serotonin	has	other	functions,	including	modulating	synaptic
strength,	appetite,	sleep,	sexual	desire,	and	digestion.	Antidepression	drugs	such
as	selective	serotonin	reuptake	inhibitors	(which	tend	to	increase	serotonin	levels
available	to	receptors)	tend	to	have	far-reaching	effects,	not	all	of	them	desirable

(such	as	suppressing	libido).	Unlike	actions	in	the	neocortex,	where	recognition
of	patterns	and	activations	of	axons	affect	only	a	small	number	of	neocortical
circuits	at	a	time,	these	substances	affect	large	regions	of	the	brain	or	even	the
entire	nervous	system.
Each	hemisphere	of	the	human	brain	has	an	amygdala,	which	consists	of	an
almond-shaped	region	comprising	several	small	lobes.	The	amygdala	is	also	part
of	the	old	brain	and	is	involved	in	processing	a	number	of	types	of	emotional

responses,	the	most	notable	of	which	is	fear.	In	premammalian	
animals,	certain
preprogrammed	stimuli	representing	danger	feed	directly	into	the	amygdala,
which	in	turn	triggers	the	“fight	or	flight”	mechanism.	In	humans	the	amygdala
now	depends	on	perceptions	of	danger	to	be	transmitted	by	the	neocortex.	A
negative	comment	by	your	boss,	for	example,	might	trigger	such	a	response	by
generating	the	fear	of	losing	your	job	(or	maybe	not,	if	you	have	confidence	in	a
plan	B).	Once	the	amygdala	does	decide	that	danger	is	ahead,	an	ancient
sequence	of	events	occurs.	The	amygdala	signals	the	pituitary	gland	to	release	a
hormone	called	ACTH	(adrenocorticotropin).	This	in	turn	triggers	the	stress
hormone	cortisol	from	the	adrenal	glands,	which	results	in	more	energy	being
provided	to	your	muscles	and	nervous	system.	The	adrenal	glands	also	produce
adrenaline	and	noradrenaline,	which	suppress	your	digestive,	immune,	and
reproductive	systems	(figuring	that	these	are	not	high-priority	processes	in	an

emergency).	Levels	of	blood	pressure,	blood	sugar,	cholesterol,	and	fibrinogen
(which	speeds	blood	clotting)	all	rise.	Heart	rate	and	respiration	go	up.	Even
your	pupils	dilate	so	that	you	have	better	visual	acuity	of	your	enemy	or	your
escape	route.	This	is	all	very	useful	if	a	real	danger	such	as	a	predator	suddenly
crosses	your	path.	It	is	well	known	that	in	today’s	world,	the	chronic	activation
of	this	fight-or-flight	mechanism	can	lead	to	permanent	health	damage	in	terms
of	hypertension,	high	cholesterol	levels,	and	other	problems.

The	system	of	global	neurotransmitter	levels,	such	as	serotonin,	and
hormone	levels,	such	as	dopamine,	is	intricate,	and	we	could	spend	the	rest	of
this	book	on	the	issue	(as	a	great	many	books	have	done),	but	it	is	worth	pointing
out	that	the	bandwidth	of	information	(the	rate	of	information	processing)	in	this
system	is	very	low	compared	with	the	bandwidth	of	the	neocortex.	There	are
only	a	limited	number	of	substances	involved	and	the	levels	of	these	chemicals
tend	to	change	slowly	and	are	relatively	universal	across	the	brain,	as	compared

with	the	neocortex,	which	is	composed	of	hundreds	of	trillions	of	connections
that	can	change	quickly.
It	is	fair	to	say	that	our	emotional	experiences	take	place	in	both	the	old	and
the	new	brains.	Thinking	takes	place	in	the	new	brain	(the	
neocortex),	but
feeling	takes	place	in	both.	Any	emulation	of	human	behavior	will	therefore
need	to	model	both.	However,	if	it	is	just	human	cognitive	intelligence	that	we
are	after,	the	neocortex	is	sufficient.	We	can	replace	the	old	brain	with	the	more

direct	motivation	of	a	nonbiological	neocortex	to	achieve	the	goals	that	we
assign	to	it.	For	example,	in	the	case	of	Watson,	the	goal	was	simply	stated:
Come	up	with	correct	answers	to	
Jeopardy!
	queries	(albeit	these	were	further
modulated	by	a	program	that	understood	
Jeopardy!
	wagering).	In	the	case	of	the
new	Watson	system	being	jointly	developed	by	Nuance	and	IBM	for	medical

knowledge,	the	goal	is	to	help	treat	human	disease.	Future	systems	can	have
goals	such	as	actually	curing	disease	and	alleviating	poverty.	A	lot	of	the
pleasure-fear	struggle	is	already	obsolete	for	humans,	as	the	old	brain	evolved
long	before	even	primitive	human	society	got	started;	indeed	most	of	it	is
reptilian.
There	is	a	continual	struggle	in	the	human	brain	as	to	whether	the	old	or	the
new	brain	is	in	charge.	The	old	brain	tries	to	set	the	agenda	with	its	control	of
pleasure	and	fear	experiences,	whereas	the	new	brain	is	continually	trying	to
understand	the	relatively	primitive	algorithms	of	the	old	brain	and	seeking	to
manipulate	it	to	its	own	agenda.	Keep	in	mind	that	the	amygdala	is	unable	to
evaluate	danger	on	its	own—in	the	human	brain	it	relies	on	the	neocortex	to
make	those	judgments.	Is	that	person	a	friend	or	a	foe,	a	lover	or	a	threat?	Only
the	neocortex	can	decide.
To	the	extent	that	we	are	not	directly	engaged	in	mortal	combat	and	hunting

for	food,	we	have	succeeded	in	at	least	partially	sublimating	our	ancient	drives	to
more	creative	endeavors.	On	that	note,	we’ll	discuss	creativity	and	love	in	the
.

TRANSCENDENT	ABILITIES
	
This	is	my	simple	religion.	There	is	no	need	for	temples;	no	need	for
complicated	philosophy.	Our	own	brain,	our	own	heart	is	our	temple;	the
philosophy	is	kindness.
—The	Dalai	Lama
	
My	hand	moves	because	certain	forces—electric,	magnetic,	or	whatever
“nerve-force”	may	prove	to	be—are	impressed	on	it	by	my	brain.	This
nerve-force,	stored	in	the	brain,	would	probably	be	traceable,	if	Science
were	complete,	to	chemical	forces	supplied	to	the	brain	by	the	blood,	and
ultimately	derived	from	the	food	I	eat	and	the	air	I	breathe.
—Lewis	Carroll
	
O
ur	emotional	thoughts	also	take	place	in	the	neocortex	but	are	influenced	by
portions	of	the	brain	ranging	from	ancient	brain	regions	such	as	the	amygdala	to
some	evolutionarily	recent	brain	structures	such	as	the	spindle	neurons,	which
appear	to	play	a	key	role	in	higher-level	emotions.	Unlike	the	regular	and	logical
recursive	structures	found	in	the	cerebral	cortex,	the	spindle	neurons	have	highly

O
irregular	shapes	and	connections.	They	are	the	largest	neurons	in	the	human
brain,	spanning	its	
entire	breadth.	They	are	deeply	interconnected,	with	hundreds
of	thousands	of	connections	tying	together	diverse	portions	of	the	neocortex.
As	mentioned	earlier,	the	insula	helps	process	sensory	signals,	but	it	also
plays	a	key	role	in	higher-level	emotions.	It	is	this	region	from	which	the	spindle
cells	originate.	Functional	magnetic	resonance	imaging	(fMRI)	scans	have

revealed	that	these	cells	are	particularly	active	when	a	person	is	dealing	with
emotions	such	as	love,	anger,	sadness,	and	sexual	desire.	Situations	that	strongly
activate	them	include	when	a	subject	looks	at	her	partner	or	hears	her	child
crying.
Spindle	cells	have	long	neural	filaments	called	apical	dendrites,	which	are

able	to	connect	to	faraway	neocortical	regions.	Such	“deep”	interconnectedness,
in	which	certain	neurons	provide	connections	across	numerous	regions,	is	a
feature	that	occurs	increasingly	as	we	go	up	the	evolutionary	ladder.	It	is	not
surprising	that	the	spindle	cells,	involved	as	they	are	in	handling	emotion	and
moral	judgment,	would	have	this	form	of	connectedness,	given	the	ability	of
higher-level	emotional	reactions	to	touch	on	diverse	topics	and	thoughts.
Because	of	their	links	to	many	other	parts	of	the	brain,	the	high-level	emotions
that	spindle	cells	process	are	affected	by	all	of	our	perceptual	and	cognitive
regions.	It	is	important	to	point	out	that	these	cells	are	not	doing	rational
problem	solving,	which	is	why	we	don’t	have	rational	control	over	our	responses
to	music	or	over	falling	in	love.	The	rest	of	the	brain	is	heavily	engaged,
however,	in	trying	to	make	sense	of	our	mysterious	high-level	emotions.
There	are	relatively	few	spindle	cells:	only	about	80,000,	with

approximately	45,000	in	the	right	hemisphere	and	35,000	in	the	left.	This
disparity	is	at	least	one	reason	for	the	perception	that	emotional	intelligence	is
the	province	of	the	right	brain,	although	the	disproportion	is	modest.	Gorillas
have	about	16,000	of	these	cells,	bonobos	about	2,100,	and	chimpanzees	about
1,800.	Other	mammals	lack	them	completely.
Anthropologists	believe	that	spindle	cells	made	their	first	appearance	10	to
15	million	years	ago	in	the	as	yet	undiscovered	common	ancestor	to	apes	and

hominids	(precursors	to	humans)	and	rapidly	increased	in	numbers	around
100,000	years	ago.	Interestingly,	spindle	cells	do	not	exist	in	
newborn	humans
but	begin	to	appear	only	at	around	the	age	of	four	months	and	increase
significantly	in	number	from	ages	one	to	three.	Children’s	ability	to	deal	with
moral	issues	and	perceive	such	higher-level	emotions	as	love	develop	during	this
same	period.

Aptitude
Wolfgang	Amadeus	Mozart	(1756–1791)	wrote	a	minuet	when	he	was	five.	At
age	six	he	performed	for	the	empress	Maria	Theresa	at	the	imperial	court	in
Vienna.	He	went	on	to	compose	six	hundred	pieces,	including	forty-one
symphonies,	before	his	death	at	age	thirty-five,	and	is	widely	regarded	as	the
greatest	composer	in	the	European	classical	tradition.	One	might	say	that	he	had
an	aptitude	for	music.
So	what	does	this	mean	in	the	context	of	the	pattern	recognition	theory	of
mind?	Clearly	part	of	what	we	regard	as	aptitude	is	the	product	of	nurture,	that	is
to	say,	the	influences	of	environment	and	other	people.	Mozart	was	born	into	a
musical	family.	His	father,	Leopold,	was	a	composer	and	kapellmeister	(literally
musical	leader)	of	the	court	orchestra	of	the	archbishop	of	Salzburg.	The	young
Mozart	was	immersed	in	music,	and	his	father	started	teaching	him	the	violin
and	clavier	(a	keyboard	instrument)	at	the	age	of	three.
However,	environmental	influences	alone	do	not	fully	explain	Mozart’s

genius.	There	is	clearly	a	nature	component	as	well.	What	form	does	this	take?
As	I	wrote	in	
chapter	4
,	different	regions	of	the	neocortex	have	become
optimized	(by	biological	evolution)	for	certain	types	of	patterns.	Even	though	the
basic	pattern	recognition	algorithm	of	the	modules	is	uniform	across	the
neocortex,	since	certain	types	of	patterns	tend	to	flow	through	particular	regions
(faces	through	the	fusiform	gyrus,	for	example),	those	regions	will	become
better	at	processing	the	associated	patterns.	However,	there	are	numerous

chapter	4
parameters	that	govern	how	the	algorithm	is	actually	carried	out	in	each	module.
For	example,	how	close	a	match	is	required	for	a	pattern	to	be	recognized?	How
is	that	threshold	modified	if	
a	higher-level	module	sends	a	signal	that	its	pattern
is	“expected”?	How	are	the	size	parameters	considered?	These	and	other	factors
have	been	set	differently	in	different	regions	to	be	advantageous	for	particular
types	of	patterns.	In	our	work	with	similar	methods	in	artificial	intelligence,	we

have	noticed	the	same	phenomenon	and	have	used	simulations	of	evolution	to
optimize	these	parameters.
If	particular	regions	can	be	optimized	for	different	types	of	patterns,	then	it
follows	that	individual	brains	will	also	vary	in	their	ability	to	learn,	recognize,
and	create	certain	types	of	patterns.	For	example,	a	brain	can	have	an	innate
aptitude	for	music	by	being	better	able	to	recognize	rhythmic	patterns,	or	to
better	understand	the	geometric	arrangements	of	harmonies.	The	phenomenon	of

perfect	pitch	(the	ability	to	recognize	and	to	reproduce	a	pitch	without	an
external	reference),	which	is	correlated	with	musical	talent,	appears	to	have	a
genetic	basis,	although	the	ability	needs	to	be	developed,	so	it	is	likely	to	be	a
combination	of	nature	and	nurture.	The	genetic	basis	of	perfect	pitch	is	likely	to
reside	outside	the	neocortex	in	the	preprocessing	of	auditory	information,
whereas	the	learned	aspect	resides	in	the	neocortex.
There	are	other	skills	that	contribute	to	degrees	of	competency,	whether	of
the	routine	variety	or	of	the	legendary	genius.	Neocortical	abilities—for
example,	the	ability	of	the	neocortex	to	master	the	signals	of	fear	that	the
amygdala	generates	(when	presented	with	disapproval)—play	a	significant	role,
as	do	attributes	such	as	confidence,	organizational	skills,	and	the	ability	to
influence	others.	A	very	important	skill	I	noted	earlier	is	the	courage	to	pursue
ideas	that	go	against	the	grain	of	orthodoxy.	Invariably,	people	we	regard	as

geniuses	pursued	their	own	mental	experiments	in	ways	that	were	not	initially
understood	or	appreciated	by	their	peers.	Although	Mozart	did	gain	recognition
in	his	lifetime,	most	of	the	adulation	came	later.	He	died	a	pauper,	buried	in	a
common	grave,	and	only	two	other	musicians	showed	up	at	his	funeral.

Creativity
Creativity	is	a	drug	I	cannot	live	without.
—Cecil	B.	DeMille
	
The	problem	is	never	how	to	get	new,	innovative	thoughts	into	your	mind,
but	how	to	get	old	ones	out.	Every	mind	is	a	building	filled	with	archaic
furniture.	Clean	out	a	corner	of	your	mind	and	creativity	will	instantly	fill
it.
—Dee	Hock
	
Humanity	can	be	quite	cold	to	those	whose	eyes	see	the	world	differently.
—Eric	A.	Burns
	
Creativity	can	solve	almost	any	problem.	The	creative	act,	the	defeat	of
habit	by	originality,	overcomes	everything.
—George	Lois
	
A	key	aspect	of	creativity	is	the	process	of	finding	great	metaphors—symbols
that	represent	something	else.	The	neocortex	is	a	great	metaphor	machine,	which
accounts	for	why	we	are	a	uniquely	creative	species.	Every	one	of	the
approximately	300	million	pattern	recognizers	in	our	neocortex	is	recognizing
and	defining	a	pattern	and	giving	it	a	name,	which	in	the	case	of	the	neocortical
pattern	recognition	modules	is	simply	the	axon	emerging	from	the	pattern

recognizer	that	will	fire	when	that	pattern	is	found.	That	symbol	in	turn	then
becomes	part	of	another	pattern.	Each	one	of	these	patterns	is	essentially	a
metaphor.	The	recognizers	can	fire	up	to	100	times	a	second,	so	we	have	the
potential	of	recognizing	up	to	30	billion	metaphors	a	second.	Of	course	not	every
module	is	firing	in	every	cycle—but	it	is	fair	to	say	that	we	are	indeed
recognizing	millions	of	metaphors	a	second.
Of	course,	some	metaphors	are	more	significant	than	others.	Darwin

perceived	that	Charles	Lyell’s	insight	on	how	very	gradual	changes	from	a

trickle	of	water	could	carve	out	great	canyons	was	a	powerful	metaphor	for	how
a	trickle	of	small	evolutionary	changes	over	thousands	of	generations	could
carve	out	great	changes	in	the	differentiation	of	species.	Thought	experiments,
such	as	the	one	that	Einstein	used	to	illuminate	the	true	meaning	of	the
Michelson-Morley	experiment,	are	all	metaphors,	in	the	sense	of	being	a	“thing
regarded	as	representative	or	symbolic	of	something	else,”	to	quote	a	dictionary
definition.
Do	you	see	any	metaphors	in	Sonnet	73	by	Shakespeare?
That	time	of	year	thou	mayst	in	me	behold
When	yellow	leaves,	or	none,	or	few,	do	hang
Upon	those	boughs	which	shake	against	the	cold,
Bare	ruined	choirs,	where	late	the	sweet	birds	sang.
In	me	thou	seest	the	twilight	of	such	day
As	after	sunset	fadeth	in	the	west,
Which	by	and	by	black	night	doth	take	away,
Death’s	second	self	that	seals	up	all	in	rest.
In	me	thou	seest	the	glowing	of	such	fire
That	on	the	ashes	of	his	youth	doth	lie,
As	the	deathbed	whereon	it	must	expire

Consumed	with	that	which	it	was	nourished	by.
This	thou	perceiv’st,	which	makes	thy	love	more	strong,
To	love	that	well	which	thou	must	leave	ere	long.
	
In	this	sonnet,	the	poet	uses	extensive	metaphors	to	describe	his	advancing
age.	His	age	is	like	late	autumn,	“when	yellow	leaves,	or	none,	or	few,	do	hang.”
The	weather	is	cold	and	the	birds	can	no	longer	sit	on	the	branches,	which	he
calls	“bare	ruin’d	choirs.”	His	age	is	like	the	twilight	as	the	“sunset	fadeth	in	the
west,	which	by	and	by	black	night	doth	take	away.”

west,	which	by	and	by	black	night	doth	take	away.”	
He	is	the	remains	of	a	fire
“that	on	the	ashes	of	his	youth	doth	lie.”	Indeed,	all	language	is	ultimately
metaphor,	though	some	expressions	of	it	are	more	memorable	than	others.
Finding	a	metaphor	is	the	process	of	recognizing	a	pattern	despite
differences	in	detail	and	context—an	activity	we	undertake	trivially	every
moment	of	our	lives.	The	metaphorical	leaps	that	we	consider	of	significance
tend	to	take	place	in	the	interstices	of	different	disciplines.	Working	against	this

essential	force	of	creativity,	however,	is	the	pervasive	trend	toward	ever	greater
specialization	in	the	sciences	(and	just	about	every	other	field	as	well).	As
American	mathematician	Norbert	Wiener	(1894–1964)	wrote	in	his	seminal
book	
Cybernetics
,	published	the	year	I	was	born	(1948):

There	are	fields	of	scientific	work,	as	we	shall	see	in	the	body	of	this
book,	which	have	been	explored	from	the	different	sides	of	pure
mathematics,	statistics,	electrical	engineering,	and	neurophysiology;	in
which	every	single	notion	receives	a	separate	name	from	each	group,	and	in
which	important	work	has	been	triplicated	or	quadruplicated,	while	still
other	important	work	is	delayed	by	the	unavailability	in	one	field	of	results
that	may	have	already	become	classical	in	the	next	field.
It	is	these	boundary	regions	which	offer	the	richest	opportunities	to	the
qualified	investigator.	They	are	at	the	same	time	the	most	refractory	to	the
accepted	techniques	of	mass	attack	and	the	division	of	labor.
	
A	technique	I	have	used	in	my	own	work	to	combat	increasing
specialization	is	to	assemble	the	experts	that	I	have	gathered	for	a	project	(for
example,	my	speech	recognition	work	included	speech	scientists,	linguists,
psychoacousticians,	and	pattern	recognition	experts,	not	to	mention	computer

scientists)	and	encourage	each	one	to	teach	the	group	his	particular	techniques
and	terminology.	We	then	throw	out	all	of	that	terminology	
and	make	up	our
own.	Invariably	we	find	metaphors	from	one	field	that	solve	problems	in
another.
A	mouse	that	finds	an	escape	route	when	confronted	with	the	household	cat
—and	can	do	so	even	if	the	situation	is	somewhat	different	from	what	it	has	ever
encountered	before—is	being	creative.	Our	own	creativity	is	orders	of

another.
magnitude	greater	than	that	of	the	mouse—and	involves	far	more	levels	of
abstraction—because	we	have	a	much	larger	neocortex,	which	is	capable	of
greater	levels	of	hierarchy.	So	one	way	to	achieve	greater	creativity	is	by
effectively	assembling	more	neocortex.
One	approach	to	expand	the	available	neocortex	is	through	the
collaboration	of	multiple	humans.	This	is	accomplished	routinely	via	the
communication	between	people	gathered	in	a	problem-solving	community.
Recently	there	have	been	efforts	to	use	online	collaboration	tools	to	harness	the

power	of	real-time	collaboration,	which	have	shown	success	in	mathematics	and
other	fields.
1
The	next	step,	of	course,	will	be	to	expand	the	neocortex	itself	with	its
nonbiological	equivalent.	This	will	be	our	ultimate	act	of	creativity:	to	create	the
capability	of	being	creative.	A	nonbiological	neocortex	will	ultimately	be	faster
and	could	rapidly	search	for	the	kinds	of	metaphors	that	inspired	Darwin	and
Einstein.	It	could	systematically	explore	all	of	the	overlapping	boundaries
between	our	exponentially	expanding	frontiers	of	knowledge.

1
Some	people	express	concern	about	what	will	happen	to	those	who	would

opt	out	of	such	mind	expansion.	I	would	point	out	that	this	additional
intelligence	will	essentially	reside	in	the	cloud	(the	exponentially	expanding
network	of	computers	that	we	connect	to	through	online	communication),	where
most	of	our	machine	intelligence	is	now	stored.	When	you	use	a	search	engine,
recognize	speech	from	your	phone,	consult	a	virtual	assistant	such	as	Siri,	or	use
your	phone	to	translate	a	sign	into	another	language,	the	intelligence	is	not	in	the
device	itself	but	in	the	cloud.	Our	expanded	neocortex	will	be	housed	there	too.
Whether	we	access	such	expanded	intelligence	through	direct	neural	connection
or	the	way	we	do	now—by	interacting	with	it	via	our	devices—is	an	arbitrary
distinction.	In	
my	view	we	will	all	become	more	creative	through	this	pervasive
enhancement,	whether	we	choose	to	opt	in	or	out	of	direct	connection	to
humanity’s	expanded	intelligence.	We	have	already	outsourced	much	of	our
personal,	social,	historical,	and	cultural	memory	to	the	cloud,	and	we	will

ultimately	do	the	same	thing	with	our	hierarchical	thinking.
Einstein’s	breakthrough	resulted	not	only	from	his	application	of	metaphors
through	mind	experiments	but	also	from	his	courage	in	believing	in	the	power	of
those	metaphors.	He	was	willing	to	relinquish	the	traditional	explanations	that
failed	to	satisfy	his	experiments,	and	he	was	willing	to	withstand	the	ridicule	of
his	peers	to	the	bizarre	explanations	that	his	metaphors	implied.	These	qualities
—belief	in	metaphor	and	courage	of	conviction—are	ones	that	we	should	be	able

to	program	into	our	nonbiological	neocortex	as	well.

Love
Clarity	of	mind	means	clarity	of	passion,	too;	this	is	why	a	great	and	clear
mind	loves	ardently	and	sees	distinctly	what	it	loves.
—Blaise	Pascal
	
There	is	always	some	madness	in	love.	But	there	is	also	always	some
reason	in	madness.
—Friedrich	Nietzsche
	
When	you	have	seen	as	much	of	life	as	I	have,	you	will	not	underestimate
the	power	of	obsessive	love.
—Albus	Dumbledore,	in	J.	K.	Rowling,	Harry	Potter	and	the	Half-
Blood	Prince
	
I	always	like	a	good	math	solution	to	any	love	problem.
—Michael	Patrick	King,	from	the	“Take	Me	Out	to	the	Ballgame”
episode	of	
Sex	and	the	City
	
If	you	haven’t	actually	experienced	ecstatic	love	personally,	you	have
undoubtedly	heard	about	it.	It	is	fair	to	say	that	a	substantial	fraction	if	not	a
majority	of	the	world’s	art—stories,	novels,	music,	dance,	paintings,	television
shows,	and	movies—is	inspired	by	the	stories	of	love	in	its	earliest	stages.
Science	has	recently	gotten	into	the	act	as	well,	and	we	are	now	able	to

episode	of	
	
identify	the	biochemical	changes	that	occur	when	someone	falls	in	love.
Dopamine	is	released,	producing	feelings	of	happiness	and	delight.
Norepinephrine	levels	soar,	which	lead	to	a	racing	heart	and	overall	feelings	of
exhilaration.	These	chemicals,	along	with	phenylethylamine,	produce	elation,
high	energy	levels,	focused	attention,	loss	of	appetite,	and	a	general	craving	for
the	object	of	one’s	desire.	Interestingly,	recent	research	at	University	College	in
London	also	shows	that	serotonin	levels	go	down,	similar	to	what	happens	in

obsessive-compulsive	disorder,	which	is	consistent	with	the	obsessive	nature	of

early	love.
2
	The	high	levels	of	dopamine	and	norepinephrine	account	for	the
heightened	short-term	attention,	euphoria,	and	craving	of	early	love.
If	these	biochemical	phenomena	sound	similar	to	those	of	the	fight-or-flight
syndrome,	they	are,	except	that	here	we	are	running	toward	something	or
someone;	indeed,	a	cynic	might	say	toward	rather	than	away	from	danger.	The
changes	are	also	fully	consistent	with	those	of	the	early	phases	of	addictive
behavior.	The	Roxy	Music	song	“Love	Is	the	Drug”	is	quite	accurate	in
describing	this	state	(albeit	the	subject	of	the	song	is	looking	to	score	his	next	fix
of	love).	Studies	of	ecstatic	religious	experiences	also	show	the	same	physical
phenomena;	it	can	be	said	that	the	person	having	such	an	experience	is	falling	in
love	with	God	or	whatever	spiritual	connection	on	which	they	are	focused.
In	the	case	of	early	romantic	love,	estrogen	and	testosterone	certainly	play	a
role	in	establishing	sex	drive,	but	if	sexual	reproduction	were	the	only

evolutionary	objective	of	love,	then	the	romantic	aspect	of	the	process	would	not
be	necessary.	As	psychologist	John	William	Money	(1921–2006)	wrote,	“Lust	is
lewd,	love	is	lyrical.”
The	ecstatic	phase	of	love	leads	to	the	attachment	phase	and	ultimately	to	a
long-term	bond.	There	are	chemicals	that	encourage	this	process	as	well,
including	oxytocin	and	vasopressin.	Consider	two	related	species	of	voles:	the
prairie	vole	and	the	montane	vole.	They	are	pretty	much	identical,	except	that

the	prairie	vole	has	receptors	for	oxytocin	and	vasopressin,	whereas	the	montane
vole	does	not.	The	prairie	vole	is	noted	for	lifetime	monogamous	relationships,
while	the	montane	vole	resorts	almost	exclusively	to	one-night	stands.	In	the
case	of	voles,	the	oxytocin	and	vasopressin	receptors	are	pretty	much
determinative	as	to	the	nature	of	their	love	life.
While	these	chemicals	are	influential	on	humans	as	well,	our	neocortex	has
taken	a	commanding	role,	as	in	everything	else	we	do.	Voles	do	have	a

neocortex,	but	it	is	postage-stamp	sized	and	flat	and	just	large	enough	for	them
to	find	a	mate	for	life	(or,	in	the	case	of	montane	voles,	at	least	for	the	night)	and
carry	out	other	basic	vole	behaviors.	We	humans	have	sufficient	additional
neocortex	to	engage	in	the	expansive	“lyrical”	expressions	to	which	Money
refers.
From	an	evolutionary	perspective,	love	itself	exists	to	meet	the	needs	of	the
neocortex.	If	we	didn’t	have	a	neocortex,	then	lust	would	be	quite	sufficient	to

refers.
guarantee	reproduction.	The	ecstatic	instigation	of	love	leads	to	attachment	and
mature	love,	and	results	in	a	lasting	bond.	This	in	turn	is	designed	to	provide	at
least	the	possibility	of	a	stable	environment	for	children	while	their	own
neocortices	undergo	the	critical	learning	needed	to	become	responsible	and
capable	adults.	Learning	in	a	rich	environment	is	inherently	part	of	the	method

of	the	neocortex.	Indeed	the	same	oxytocin	and	vasopressin	hormone
mechanisms	play	a	key	role	in	establishing	the	critical	bonding	of	parent
(especially	mother)	and	child.
At	the	far	end	of	the	story	of	love,	a	loved	one	becomes	a	major	part	of	our
neocortex.	After	decades	of	being	together,	a	virtual	other	exists	in	the	neocortex
such	that	we	can	anticipate	every	step	of	what	our	lover	will	say	and	do.	Our
neocortical	patterns	are	filled	with	the	thoughts	and	patterns	that	reflect	who	they
are.	When	we	lose	that	person,	we	literally	lose	part	of	
ourselves.	This	is	not	
just
a	metaphor—all	of	the	vast	pattern	recognizers	that	are	filled	with	the	patterns
reflecting	the	person	we	love	suddenly	change	their	nature.	Although	they	can	be
considered	a	precious	way	to	keep	that	person	alive	within	ourselves,	the	vast
neocortical	patterns	of	a	lost	loved	one	turn	suddenly	from	triggers	of	delight	to
triggers	of	mourning.
The	evolutionary	basis	for	love	and	its	phases	is	not	the	full	story	in	today’s

just
world.	We	have	already	largely	succeeded	in	liberating	sex	from	its	biological
function,	in	that	we	can	have	babies	without	sex	and	we	can	certainly	have	sex
without	babies.	The	vast	majority	of	sex	takes	place	for	its	sensual	and	relational
purposes.	And	we	routinely	fall	in	love	for	purposes	other	than	raising	children.
Similarly,	the	vast	expanse	of	artistic	expression	of	all	kinds	that	celebrates
love	and	its	myriad	forms	dating	back	to	antiquity	is	also	an	end	in	itself.	Our
ability	to	create	these	enduring	forms	of	transcendent	knowledge—about	love	or

anything	else—is	precisely	what	makes	our	species	unique.
The	neocortex	is	biology’s	greatest	creation.	In	turn,	it	is	the	poems	about
love—and	all	of	our	other	creations—that	represent	the	greatest	inventions	of
our	neocortex.

THE	BIOLOGICALLY
INSPIRED	DIGITAL
NEOCORTEX
	
Never	trust	anything	that	can	think	for	itself	if	you	can’t	see	where	it	keeps
its	brain.
—Arthur	Weasley,	in	J.	K.	Rowling,	
Harry	Potter	and	the	Prisoner
of	Azkaban
	
No,	I’m	not	interested	in	developing	a	powerful	brain.	All	I’m	after	is	just	a
mediocre	brain,	something	like	the	President	of	the	American	Telephone
and	Telegraph	Company.
—Alan	Turing
	
A	computer	would	deserve	to	be	called	intelligent	if	it	could	deceive	a
human	into	believing	that	it	was	human.
	
I	believe	that	at	the	end	of	the	century	the	use	of	words	and	general
educated	opinion	will	have	altered	so	much	that	one	will	be	able	to	speak	of
machines	thinking	without	expecting	to	be	contradicted.
	
A
	mother	rat	will	build	a	nest	for	her	young	even	if	she	has	never	seen	another
rat	in	her	lifetime.
1
	Similarly,	a	spider	will	spin	a	web,	a	caterpillar	will	create
her	own	cocoon,	and	a	beaver	will	build	a	dam,	even	if	no	contemporary	ever

showed	them	how	to	accomplish	these	complex	tasks.	That	is	not	to	say	that
these	are	not	learned	behaviors.	It	is	just	that	these	animals	did	not	learn	them	in
a	single	lifetime—they	learned	them	over	thousands	of	lifetimes.	The	evolution
of	animal	behavior	does	constitute	a	learning	process,	but	it	is	learning	by	the
species,	not	by	the	individual,	and	the	fruits	of	this	learning	process	are	encoded
in	DNA.
To	appreciate	the	significance	of	the	evolution	of	the	neocortex,	consider
that	it	greatly	sped	up	the	process	of	learning	(hierarchical	knowledge)	from
thousands	of	years	to	months	(or	less).	Even	if	millions	of	animals	in	a	particular
mammalian	species	failed	to	solve	a	problem	(requiring	a	hierarchy	of	steps),	it
required	only	one	to	accidentally	stumble	upon	a	solution.	That	new	method
would	then	be	copied	and	spread	exponentially	through	the	population.
We	are	now	in	a	position	to	speed	up	the	learning	process	by	a	factor	of

thousands	or	millions	once	again	by	migrating	from	biological	to	nonbiological
intelligence.	Once	a	digital	neocortex	learns	a	skill,	it	can	transfer	that	know-
how	in	minutes	or	even	seconds.	As	one	of	many	examples,	at	my	first	company,
Kurzweil	Computer	Products	(now	Nuance	Speech	Technologies),	which	I
founded	in	1973,	we	spent	years	training	a	set	of	research	computers	to
recognize	printed	letters	from	scanned	documents,	a	technology	called	omni-font
(any	type	font)	optical	character	recognition	(OCR).	This	particular	technology

has	now	been	in	continual	development	for	almost	forty	years,	with	the	current
product	called	OmniPage	from	Nuance.	If	you	want	your	computer	to	recognize
printed	letters,	you	don’t	need	to	spend	years	training	it	to	do	so,	as	we	did—you
can	simply	download	the	evolved	patterns	already	learned	by	the	research
computers	in	the	form	of	software.	In	the	1980s	we	began	on	speech	recognition,
and	that	technology,	which	has	also	been	in	continuous	development	now	for
several	decades,	is	part	of	Siri.	Again,	you	can	download	in	seconds	the	evolved

patterns	learned	by	the	research	computers	over	many	years.
Ultimately	we	will	create	an	artificial	neocortex	that	has	the	full	range	and
flexibility	of	its	human	counterpart.	Consider	the	benefits.	Electronic	circuits	are
millions	of	times	faster	than	our	biological	circuits.	At	first	we	will	have	to
devote	all	of	this	speed	increase	to	compensating	for	the	relative	lack	of
parallelism	in	our	computers,	but	ultimately	the	digital	neocortex	will	be	much
faster	than	the	biological	variety	and	will	only	continue	to	increase	in	speed.

When	we	augment	our	own	neocortex	with	a	synthetic	version,	we	won’t
have	to	worry	about	how	much	additional	neocortex	can	physically	fit	into	our
bodies	and	brains,	as	most	of	it	will	be	in	the	cloud,	like	most	of	the	computing
we	use	today.	I	estimated	earlier	that	we	have	on	the	order	of	300	million	pattern
recognizers	in	our	biological	neocortex.	That’s	as	much	as	could	be	squeezed

into	our	skulls	even	with	the	evolutionary	innovation	of	a	large	forehead	and
with	the	neocortex	taking	about	80	percent	of	the	available	space.	As	soon	as	we
start	thinking	in	the	cloud,	there	will	be	no	natural	limits—we	will	be	able	to	use
billions	or	trillions	of	pattern	recognizers,	basically	whatever	we	need,	and
whatever	the	law	of	accelerating	returns	can	provide	at	each	point	in	time.
In	order	for	a	digital	neocortex	to	learn	a	new	skill,	it	will	still	require	many
iterations	of	education,	just	as	a	biological	neocortex	does,	but	once	a	single
digital	neocortex	somewhere	and	at	some	time	learns	something,	it	can	share	that
knowledge	with	every	other	digital	neocortex	without	delay.	We	can	each	have
our	own	private	neocortex	extenders	in	the	cloud,	just	as	we	have	our	own
private	stores	of	personal	data	today.
Last	but	not	least,	we	will	be	able	to	back	up	the	digital	portion	of	our
intelligence.	As	we	have	seen,	it	is	not	just	a	metaphor	to	state	that	there	is

information	contained	in	our	neocortex,	and	it	is	frightening	to	contemplate	that
none	of	this	information	is	backed	up	today.	There	is,	of	course,	one	way	in
which	we	do	back	up	some	of	the	information	in	our	brains—by	
writing	it	down.
The	ability	to	transfer	at	least	some	of	our	thinking	to	a	medium	that	can	outlast
our	biological	bodies	was	a	huge	step	forward,	but	a	great	deal	of	data	in	our
brains	continues	to	remain	vulnerable.

Brain	Simulations
One	approach	to	building	a	digital	brain	is	to	simulate	precisely	a	biological	one.
For	example,	Harvard	brain	sciences	doctoral	student	David	Dalrymple	(born	in
1991)	is	planning	to	simulate	the	brain	of	a	nematode	(a	roundworm).
2
Dalrymple	selected	the	nematode	because	of	its	relatively	simple	nervous
system,	which	consists	of	about	300	neurons,	and	which	he	plans	to	simulate	at
the	very	detailed	level	of	molecules.	He	will	also	create	a	computer	simulation	of
its	body	as	well	as	its	environment	so	that	his	virtual	nematode	can	hunt	for
(virtual)	food	and	do	the	other	things	that	nematodes	are	good	at.	Dalrymple	says
it	is	likely	to	be	the	first	complete	brain	upload	from	a	biological	animal	to	a
virtual	one	that	lives	in	a	virtual	world.	Like	his	simulated	nematode,	whether
even	biological	nematodes	are	conscious	is	open	to	debate,	although	in	their
struggle	to	eat,	digest	food,	avoid	predators,	and	reproduce,	they	do	have
experiences	to	be	conscious	of.

At	the	opposite	end	of	the	spectrum,	Henry	Markram’s	Blue	Brain	Project
is	planning	to	simulate	the	human	brain,	including	the	entire	neocortex	as	well	as
the	old-brain	regions	such	as	the	hippocampus,	amygdala,	and	cerebellum.	His
planned	simulations	will	be	built	at	varying	degrees	of	detail,	up	to	a	full
simulation	at	the	molecular	level.	As	I	reported	in	
chapter	4
,	Markram	has
discovered	a	key	module	of	several	dozen	neurons	that	is	repeated	over	and	over

chapter	4
again	in	the	neocortex,	demonstrating	that	learning	is	done	by	these	modules	and
not	by	individual	neurons.
Markram’s	progress	has	been	scaling	up	at	an	exponential	pace.	He
simulated	one	neuron	in	2005,	the	year	the	project	was	initiated.	In	2008	his
team	simulated	an	entire	neocortical	column	of	a	rat	brain,	consisting	
of	10,000
neurons.	By	2011	this	expanded	to	100	columns,	totaling	a	million	cells,	which
he	calls	a	mesocircuit.	One	controversy	concerning	Markram’s	work	is	how	to

of	10,000
verify	that	the	simulations	are	accurate.	In	order	to	do	this,	these	simulations	will
need	to	demonstrate	learning	that	I	discuss	below.
He	projects	simulating	an	entire	rat	brain	of	100	mesocircuits,	totaling	100
million	neurons	and	about	a	trillion	synapses,	by	2014.	In	a	talk	at	the	2009	TED
conference	at	Oxford,	Markram	said,	“It	is	not	impossible	to	build	a	human
brain,	and	we	can	do	it	in	10	years.”
3
	His	most	recent	target	for	a	full	brain
simulation	is	2023.
4

3
4
Markram	and	his	team	are	basing	their	model	on	detailed	anatomical	and

electrochemical	analyses	of	actual	neurons.	Using	an	automated	device	they
created	called	a	patch-clamp	robot,	they	are	measuring	the	specific	ion	channels,
neurotransmitters,	and	enzymes	that	are	responsible	for	the	electrochemical
activity	within	each	neuron.	Their	automated	system	was	able	to	do	thirty	years
of	analysis	in	six	months,	according	to	Markram.	It	was	from	these	analyses	that
they	noticed	the	“Lego	memory”	units	that	are	the	basic	functional	units	of	the
neocortex.
Actual	and	projected	progress	of	the	Blue	Brain	brain	simulation
project.
Significant	contributions	to	the	technology	of	robotic	patch-clamping	was
made	by	MIT	neuroscientist	Ed	Boyden,	Georgia	Tech	mechanical	engineering
professor	Craig	Forest,	and	Forest’s	graduate	student	Suhasa	Kodandaramaiah.
They	demonstrated	an	automated	system	with	one-micrometer	precision	that	can
perform	scanning	of	neural	tissue	at	very	close	range	without	damaging	the
delicate	membranes	of	the	neurons.	“This	is	something	a	robot	can	do	that	a

project.
human	can’t,”	Boyden	commented.
To	return	to	Markram’s	simulation,	after	simulating	one	neocortical	column,
Markram	was	quoted	as	saying,	“Now	we	just	have	to	scale	it	up.”
5
	Scaling	is
certainly	one	big	factor,	but	there	is	one	other	key	hurdle,	which	
is	learning.	If
the	Blue	Brain	Project	brain	is	to	“speak	and	have	an	intelligence	and	behave
very	much	as	a	human	does,”	which	is	how	Markram	described	his	goal	in	a

BBC	interview	in	2009,	then	it	will	need	to	have	sufficient	content	in	its
simulated	neocortex	to	perform	those	tasks.
6
	As	anyone	who	has	tried	to	hold	a
conversation	with	a	newborn	can	attest,	there	is	a	lot	of	learning	that	must	be
achieved	before	this	is	feasible.
The	tip	of	the	patch-clamping	robot	developed	at	MIT	and	Georgia	Tech
scanning	neural	tissue.
There	are	two	obvious	ways	this	can	be	done	in	a	simulated	brain	such	as
Blue	Brain.	One	would	be	to	have	the	brain	learn	this	content	the	way	a	human
brain	does.	It	can	start	out	like	a	newborn	human	baby	with	an	innate	capacity
for	acquiring	hierarchical	knowledge	and	with	certain	transformations
preprogrammed	in	its	sensory	preprocessing	regions.	But	the	learning	that	takes
place	between	a	biological	infant	and	a	human	person	who	can	hold	a
conversation	would	need	to	occur	in	a	comparable	manner	in	nonbiological
learning.	The	problem	with	that	approach	is	that	a	brain	that	is	being	simulated

at	the	level	of	detail	anticipated	for	Blue	Brain	is	not	expected	to	run	in	real	time
until	at	least	the	early	2020s.	Even	running	in	real	time	would	be	too	slow	unless
the	researchers	are	prepared	to	wait	a	decade	or	two	to	reach	intellectual	parity
with	an	adult	human,	although	real-time	performance	will	get	steadily	faster	as
computers	continue	to	grow	in	price/performance.
The	other	approach	is	to	take	one	or	more	biological	human	brains	that
have	already	gained	sufficient	knowledge	to	converse	in	meaningful	language

and	to	otherwise	behave	in	a	mature	manner	and	copy	their	neocortical	patterns
into	the	simulated	brain.	The	problem	with	this	method	is	that	it	requires	a
noninvasive	and	nondestructive	scanning	technology	of	sufficient	spatial	and
temporal	resolution	and	speed	to	perform	such	a	task	quickly	and	completely.	I
would	not	expect	such	an	“uploading”	technology	to	be	available	until	around
the	2040s.	(The	computational	requirement	to	simulate	a	brain	at	that	degree	of
precision,	which	I	estimate	to	be	10
19
	calculations	per	second,	will	be	available
in	a	supercomputer	according	to	my	projections	by	the	early	2020s;	however,	the
necessary	nondestructive	brain	scanning	technologies	will	take	longer.)
There	is	a	third	approach,	which	is	the	one	I	believe	simulation	projects
such	as	Blue	Brain	will	need	to	pursue.	One	can	simplify	molecular	models	by
creating	functional	equivalents	at	different	levels	of	specificity,	ranging	from	my
own	functional	algorithmic	method	(as	described	in	this	book)	to	simulations

19
that	are	closer	to	full	molecular	simulations.	The	speed	of	learning	can	thereby
be	increased	by	a	factor	of	hundreds	or	thousands	depending	on	the	degree	of
simplification	used.	An	educational	program	can	be	devised	for	the	simulated
brain	(using	the	functional	model)	that	it	can	learn	relatively	quickly.	Then	the
full	molecular	simulation	can	be	substituted	for	the	simplified	model	while	still
using	its	accumulated	learning.	We	can	then	simulate	learning	with	the	full

molecular	model	at	a	much	slower	speed.
American	computer	scientist	Dharmendra	Modha	and	his	IBM	colleagues
have	created	a	cell-by-cell	simulation	of	a	portion	of	the	human	visual	neocortex
comprising	1.6	billion	virtual	neurons	and	9	trillion	synapses,	which	is
equivalent	to	a	cat	neocortex.	It	runs	100	times	slower	than	real	time	on	an	IBM
BlueGene/P	supercomputer	consisting	of	147,456	processors.	The	work	received
the	Gordon	Bell	Prize	from	the	Association	for	Computing	Machinery.

The	purpose	of	a	brain	simulation	project	such	as	Blue	Brain	and	Modha’s
neocortex	simulations	is	specifically	to	refine	and	confirm	a	functional	model.
AI	at	the	human	level	will	principally	use	the	type	of	functional	algorithmic
model	discussed	in	this	book.	However,	molecular	simulations	will	help	us	to
perfect	that	model	and	to	fully	understand	which	details	are	important.	In	my
development	of	speech	recognition	technology	in	the	1980s	and	1990s,	we	were

able	to	refine	our	algorithms	once	the	actual	transformations	performed	by	the
auditory	nerve	and	early	portions	of	the	auditory	cortex	were	understood.	Even	if
our	functional	model	was	perfect,	understanding	exactly	how	it	is	actually
implemented	in	our	biological	brains	will	reveal	important	knowledge	about
human	function	and	dysfunction.
We	will	need	detailed	data	on	actual	brains	to	create	biologically	based
simulations.	Markram’s	team	is	collecting	its	own	data.	There	are	large-scale

projects	to	gather	this	type	of	data	and	make	it	generally	available	to	scientists.
For	example,	Cold	Spring	Harbor	Laboratory	in	New	York	has	collected	500
terabytes	of	data	by	scanning	a	mammal	brain	(a	mouse),	which	they	made
available	in	June	2012.	Their	project	allows	a	user	to	explore	a	brain	similarly	to
the	way	Google	Earth	allows	one	to	explore	the	surface	of	the	planet.	You	can
move	around	the	entire	brain	and	zoom	in	to	see	individual	neurons	and	their
connections.	You	can	highlight	a	single	connection	and	then	follow	its	path
through	the	brain.
Sixteen	sections	of	the	National	Institutes	of	Health	have	gotten	together
and	sponsored	a	major	initiative	called	the	Human	Connectome	Project	with
$38.5	million	of	funding.
7
	Led	by	Washington	University	in	St.	Louis,	the
University	of	Minnesota,	Harvard	University,	Massachusetts	General	Hospital,
and	the	University	of	California	at	Los	Angeles,	the	project	seeks	to	create	a
similar	three-dimensional	map	of	connections	in	the	human	brain.	The	project	is

7
using	a	variety	of	noninvasive	scanning	technologies,	including	new	forms	of
MRI,	magnetoencephalography	(measuring	the	magnetic	fields	produced	by	the
electrical	activity	in	the	brain),	and	diffusion	tractography	(a	method	to	trace	the
pathways	of	fiber	bundles	in	the	brain).	As	I	point	out	in	
chapter	10
,	the	spatial
resolution	of	noninvasive	scanning	of	the	brain	is	improving	at	an	exponential
rate.	The	research	by	Van	J.	Wedeen	and	his	colleagues	at	Massachusetts

chapter	10
General	Hospital	showing	a	highly	regular	gridlike	structure	of	the	wiring	of	the
neocortex	that	I	described	in	
chapter	4
	is	one	early	result	from	this	project.
Oxford	University	computational	neuroscientist	Anders	Sandberg	(born	in
1972)	and	Swedish	philosopher	Nick	Bostrom	(born	in	1973)	have	written	the
comprehensive	
,	which	details	the
requirements	for	simulating	the	human	brain	(and	other	types	of	brains)	at
different	levels	of	specificity	from	high-level	functional	models	
to	simulating
molecules.
8

chapter	4
8
	The	report	does	not	provide	a	timeline,	but	it	does	describe	the
requirements	to	simulate	different	types	of	brains	at	varying	levels	of	precision
in	terms	of	brain	scanning,	modeling,	storage,	and	computation.	The	report
projects	ongoing	exponential	gains	in	all	of	these	areas	of	capability	and	argues
that	the	requirements	to	simulate	the	human	brain	at	a	high	level	of	detail	are
coming	into	place.

An	outline	of	the	technological	capabilities	needed	for	whole	brain
emulation,	in	
	by	Anders	Sandberg	and
Nick	Bostrom.

An	outline	of	
	by	Anders	Sandberg
and	Nick	Bostrom.

Neural	Nets
In	1964,	at	the	age	of	sixteen,	I	wrote	to	Frank	Rosenblatt	(1928–1971),	a
professor	at	Cornell	University,	inquiring	about	a	machine	called	the	Mark	1
Perceptron.	He	had	created	it	four	years	earlier,	and	it	was	described	as	having
brainlike	properties.	He	invited	me	to	visit	him	and	try	the	machine	out.
The	Perceptron	was	built	from	what	he	claimed	were	electronic	models	of
neurons.	Input	consisted	of	values	arranged	in	two	dimensions.	For	speech,	one
dimension	represented	frequency	and	the	other	time,	so	each	value	represented
the	intensity	of	a	frequency	at	a	given	point	in	time.	For	
images,	each	point	was
a	pixel	in	a	two-dimensional	image.	Each	point	of	a	given	input	was	randomly
connected	to	the	inputs	of	the	first	layer	of	simulated	neurons.	Every	connection
had	an	associated	synaptic	strength,	which	represented	its	importance,	and	which
was	initially	set	at	a	random	value.	Each	neuron	added	up	the	signals	coming

into	it.	If	the	combined	signal	exceeded	a	particular	threshold,	the	neuron	fired
and	sent	a	signal	to	its	output	connection;	if	the	combined	input	signal	did	not
exceed	the	threshold,	the	neuron	did	not	fire,	and	its	output	was	zero.	The	output
of	each	neuron	was	randomly	connected	to	the	inputs	of	the	neurons	in	the	next
layer.	The	Mark	1	Perceptron	had	three	layers,	which	could	be	organized	in	a
variety	of	configurations.	For	example,	one	layer	might	feed	back	to	an	earlier

one.	At	the	top	layer,	the	output	of	one	or	more	neurons,	also	randomly	selected,
provided	the	answer.	(For	an	algorithmic	description	of	neural	nets,	see	this
endnote.)
9

Since	the	neural	net	wiring	and	synaptic	weights	are	initially	set	randomly,
the	answers	of	an	untrained	neural	net	are	also	random.	The	key	to	a	neural	net,
therefore,	is	that	it	must	learn	its	subject	matter,	just	like	the	mammalian	brains
on	which	it’s	supposedly	modeled.	A	neural	net	starts	
out	ignorant;	its	teacher—
which	may	be	a	human,	a	computer	program,	or	perhaps	another,	more	mature
neural	net	that	has	already	learned	its	lessons—rewards	the	student	neural	net
when	it	generates	the	correct	output	and	punishes	it	when	it	does	not.	This
feedback	is	in	turn	used	by	the	student	neural	net	to	adjust	the	strength	of	each
interneuronal	connection.	Connections	that	are	consistent	with	the	correct
answer	are	made	stronger.	Those	that	advocate	a	wrong	answer	are	weakened.
Over	time	the	neural	net	organizes	itself	to	provide	the	correct	answers
without	coaching.	Experiments	have	shown	that	neural	nets	can	learn	their
subject	matter	even	with	unreliable	teachers.	If	the	teacher	is	correct	only	60

percent	of	the	time,	the	student	neural	net	will	still	learn	its	lessons	with	an
accuracy	approaching	100	percent.
However,	limitations	in	the	range	of	material	that	the	Perceptron	was
capable	of	learning	quickly	became	apparent.	When	I	visited	Professor
Rosenblatt	in	1964,	I	tried	simple	modifications	to	the	input.	The	system	was	set
up	to	recognize	printed	letters,	and	would	recognize	them	quite	accurately.	It	did
a	fairly	good	job	of	autoassociation	(that	is,	it	could	recognize	the	letters	even	if

I	covered	parts	of	them),	but	fared	less	well	with	invariance	(that	is,	generalizing
over	size	and	font	changes,	which	confused	it).
During	the	last	half	of	the	1960s,	these	neural	nets	became	enormously
popular,	and	the	field	of	“connectionism”	took	over	at	least	half	of	the	artificial
intelligence	field.	The	more	traditional	approach	to	AI,	meanwhile,	included

direct	attempts	to	program	solutions	to	specific	problems,	such	as	how	to
recognize	the	invariant	properties	of	printed	letters.
Another	person	I	visited	in	1964	was	Marvin	Minsky	(born	in	1927),	one	of
the	founders	of	the	artificial	intelligence	field.	Despite	having	done	some
pioneering	work	on	neural	nets	himself	in	the	1950s,	he	was	concerned	with	the
great	surge	of	interest	in	this	technique.	Part	of	the	allure	of	neural	nets	was	that
they	supposedly	did	not	require	programming—they	would	learn	solutions	to
problems	on	their	own.	In	1965	I	entered	MIT	as	a	student	with	Professor
Minsky	as	my	mentor,	and	I	shared	his	skepticism	about	the	craze	for
“connectionism.”
In	1969	Minsky	and	Seymour	Papert	(born	in	1928),	the	two	cofounders	of
the	MIT	Artificial	Intelligence	Laboratory,	wrote	a	book	called	
Perceptrons
,
which	presented	a	single	core	theorem:	specifically,	that	a	Perceptron	was
inherently	incapable	of	determining	whether	or	not	an	image	was	connected.	The

,
book	created	a	firestorm.	Determining	whether	or	not	an	image	is	connected	is	a
task	that	humans	can	do	very	easily,	and	it	is	also	a	straightforward	process	to
program	a	computer	to	make	this	
discrimination.	The	fact	that	Perceptrons	could
not	do	so	was	considered	by	many	to	be	a	fatal	flaw.

Two	images	from	the	cover	of	the	book	
	by	Marvin	Minsky
and	Seymour	Papert.	The	top	image	is	not	connected	(that	is,	the	dark	area
consists	of	two	disconnected	parts).	The	bottom	image	is	connected.	A
human	can	readily	determine	this,	as	can	a	simple	software	program.	A
feedforward	Perceptron	such	as	Frank	Rosenblatt’s	Mark	1	Perceptron
cannot	make	this	determination.
,	however,	was	widely	interpreted	to	imply	more	than	it	actually
did.	Minsky	and	Papert’s	theorem	applied	only	to	a	particular	type	of	neural	net
called	a	feedforward	neural	net	(a	category	that	does	include	Rosenblatt’s
Perceptron);	other	types	of	neural	nets	did	not	have	this	limitation.	Still,	the	book
did	manage	to	largely	kill	most	funding	for	neural	net	research	during	the	1970s.
The	field	did	return	in	the	1980s	with	attempts	to	use	what	were	claimed	to	be
more	realistic	models	of	biological	neurons	and	ones	that	avoided	the	limitations
implied	by	the	Minsky-Papert	Perceptron	theorem.	Nevertheless,	the	ability	of

the	neocortex	to	solve	the	invariance	problem,	a	key	to	its	strength,	was	a	skill
that	remained	elusive	for	the	resurgent	connectionist	field.

Sparse	Coding:	Vector	Quantization
In	the	early	1980s	I	started	a	project	devoted	to	another	classical	pattern
recognition	problem:	understanding	human	speech.	At	first,	we	used	traditional
AI	approaches	by	directly	programming	expert	knowledge	about	the
fundamental	units	of	speech—phonemes—and	rules	from	linguists	on	how
people	string	phonemes	together	to	form	words	and	phrases.	Each	phoneme	has
distinctive	frequency	patterns.	For	example,	we	knew	that	vowels	such	as	“e”
and	“ah”	are	characterized	by	certain	resonant	frequencies	called	formants,	with
a	characteristic	ratio	of	formants	for	each	phoneme.	Sibilant	sounds	such	as	“z”
and	“s”	are	characterized	by	a	burst	of	noise	that	spans	many	frequencies.
We	captured	speech	as	a	waveform,	which	we	then	converted	into	multiple
frequency	bands	(perceived	as	pitches)	using	a	bank	of	frequency	filters.	The
result	of	this	transformation	could	be	visualized	and	was	called	a	spectrogram
(see	
page	136
).

(see	
page	136
).
The	filter	bank	is	copying	what	the	human	cochlea	does,	which	is	the	initial
step	in	our	biological	processing	of	sound.	The	software	first	identified
phonemes	based	on	distinguishing	patterns	of	frequencies	and	then	identified
words	based	on	identifying	characteristic	sequences	of	phonemes.
A	spectrogram	of	three	vowels.	From	left	to	right:	[i]	as	in	“appr
e
ciate,”
[u]	as	in	“aco
u
stic,”	and	[a]	as	in	“ah.”	The	Y	axis	represents	frequency	of
sound.	The	darker	the	band	the	more	acoustic	energy	there	is	at	that
frequency.

A	spectrogram	of	a	person	saying	the	word	“hide.”	The	horizontal	lines
show	the	formants,	which	are	sustained	frequencies	that	have	especially
high	energy.
10
The	result	was	partially	successful.	We	could	train	our	device	to	learn	the
patterns	for	a	particular	person	using	a	moderate-sized	vocabulary,	measured	in
thousands	of	words.	When	we	attempted	to	recognize	tens	of	thousands	of
words,	handle	multiple	speakers,	and	allow	fully	continuous	speech	(that	is,
speech	with	no	pauses	between	words),	we	ran	into	the	invariance	problem.
Different	people	enunciated	the	same	phoneme	differently—for	example,	one
person’s	“e”	phoneme	may	sound	like	someone	else’s	“ah.”	Even	the	same
person	was	inconsistent	in	the	way	she	spoke	a	particular	phoneme.	The	pattern
of	a	phoneme	was	often	affected	by	other	phonemes	nearby.	Many	phonemes
were	left	out	completely.	The	pronunciation	of	words	(that	is,	how	phonemes	are
strung	together	to	form	words)	was	also	highly	variable	and	dependent	on

context.	The	linguistic	rules	we	had	programmed	were	breaking	down	and	could
not	keep	up	with	the	extreme	variability	of	spoken	language.
It	became	clear	to	me	at	the	time	that	the	essence	of	human	pattern	
and
conceptual	recognition	was	based	on	hierarchies.	This	is	certainly	apparent	for
human	language,	which	constitutes	an	elaborate	hierarchy	of	structures.	But
what	is	the	element	at	the	base	of	the	structures?	That	was	the	first	question	I
considered	as	I	looked	for	ways	to	automatically	recognize	fully	normal	human
speech.

and
speech.
Sound	enters	the	ear	as	a	vibration	of	the	air	and	is	converted	by	the
approximately	3,000	inner	hair	cells	in	the	cochlea	into	multiple	frequency
bands.	Each	hair	cell	is	tuned	to	a	particular	frequency	(note	that	we	perceive
frequencies	as	tones)	and	each	acts	as	a	frequency	filter,	emitting	a	signal
whenever	there	is	sound	at	or	near	its	resonant	frequency.	As	it	leaves	the	human
cochlea,	sound	is	thereby	represented	by	approximately	3,000	separate	signals,
each	one	signifying	the	time-varying	intensity	of	a	narrow	band	of	frequencies

(with	substantial	overlap	among	these	bands).
Even	though	it	was	apparent	that	the	brain	was	massively	parallel,	it
seemed	impossible	to	me	that	it	was	doing	pattern	matching	on	3,000	separate
auditory	signals.	I	doubted	that	evolution	could	have	been	that	inefficient.	We
now	know	that	very	substantial	data	reduction	does	indeed	take	place	in	the
auditory	nerve	before	sound	signals	ever	reach	the	neocortex.
In	our	software-based	speech	recognizers,	we	also	used	filters	implemented
as	software—sixteen	to	be	exact	(which	we	later	increased	to	thirty-two,	as	we
found	there	was	not	much	benefit	to	going	much	higher	than	this).	So	in	our
system,	each	point	in	time	was	represented	by	sixteen	numbers.	We	needed	to
reduce	these	sixteen	streams	of	data	into	one	while	at	the	same	emphasizing	the
features	that	are	significant	in	recognizing	speech.
We	used	a	mathematically	optimal	technique	to	accomplish	this,	called
vector	quantization.	Consider	that	at	any	particular	point	in	time,	sound	(at	least

from	one	ear)	was	represented	by	our	software	by	sixteen	different	numbers:	that
is,	the	output	of	the	sixteen	frequency	filters.	(In	the	human	auditory	system	the
figure	would	be	3,000,	representing	the	output	of	the	3,000	cochlea	inner	hair
cells.)	In	mathematical	terminology,	
each	such	set	of	numbers	(whether	3,000	in
the	biological	case	or	16	in	our	software	implementation)	is	called	a	vector.
For	simplicity,	let’s	consider	the	process	of	vector	quantization	with	vectors
of	two	numbers.	Each	vector	can	be	considered	a	point	in	two-dimensional
space.

space.
	
If	we	have	a	very	large	sample	of	such	vectors	and	plot	them,	we	are	likely
to	notice	clusters	forming.

In	order	to	identify	the	clusters,	we	need	to	decide	how	many	we	will	allow.
In	our	project	we	generally	allowed	1,024	clusters	so	that	we	could	number	them
and	assign	each	cluster	a	10-bit	label	(because	2
10
	=	1,024).	Our	sample	of
vectors	represents	the	diversity	that	we	expect.	We	tentatively	assign	the	first
1,024	vectors	to	be	one-point	clusters.	We	then	consider	the	1,025th	vector	and
find	the	point	that	it	is	closest	to.	If	that	distance	is	greater	than	the	smallest
distance	between	any	pair	of	the	1,024	
points,	we	consider	it	as	the	beginning	of
a	new	cluster.	We	then	collapse	the	two	(one-point)	clusters	that	are	closest
together	into	a	single	cluster.	We	are	thus	still	left	with	1,024	clusters.	After
processing	the	1,025th	vector,	one	of	those	clusters	now	has	more	than	one	point.
We	keep	processing	points	in	this	way,	always	maintaining	1,024	clusters.	After
we	have	processed	all	the	points,	we	represent	each	multipoint	cluster	by	the
geometric	center	of	the	points	in	that	cluster.

We	continue	this	iterative	process	until	we	have	run	through	all	the	sample
points.	Typically	we	would	process	millions	of	points	into	1,024	(2
10
)	clusters;
we’ve	also	used	2,048	(2
11
)	or	4,096	(2
12
)	clusters.	Each	cluster	is	represented
by	one	vector	that	is	at	the	geometric	center	of	all	the	points	in	that	cluster.	Thus
the	total	of	the	distances	of	all	the	points	in	the	cluster	to	the	center	point	of	the
cluster	is	as	small	as	possible.
The	result	of	this	technique	is	that	instead	of	having	the	millions	of	points

10
11
12
that	we	started	with	(and	an	even	larger	number	of	possible	points),	we	have	now
reduced	the	data	to	just	1,024	points	that	use	the	space	of	possibilities	optimally.
Parts	of	the	space	that	are	never	used	are	not	assigned	any	clusters.
We	then	assign	a	number	to	each	cluster	(in	our	case,	0	to	1,023).	That
number	is	the	reduced,	“quantized”	representation	of	that	cluster,	which	is	why

the	technique	is	called	vector	quantization.	Any	new	input	vector	that	arrives	in
the	future	is	then	represented	by	the	number	of	the	cluster	whose	center	point	is
closest	to	this	new	input	vector.
We	can	now	precompute	a	table	with	the	distance	of	the	center	point	of
every	cluster	to	every	other	center	point.	We	thereby	have	instantly	available	the
distance	of	this	new	input	vector	(which	we	represent	by	this	quantized	point—
in	other	words,	by	the	number	of	the	cluster	that	this	
new	point	is	closest	to)	to
every	other	cluster.	Since	we	are	only	representing	points	by	their	closest	cluster,
we	now	know	the	distance	of	this	point	to	any	other	possible	point	that	might
come	along.
I	described	the	technique	above	using	vectors	with	only	two	numbers	each,
but	working	with	sixteen-element	vectors	is	entirely	analogous	to	the	simpler
example.	Because	we	chose	vectors	with	sixteen	numbers	representing	sixteen
different	frequency	bands,	each	point	in	our	system	was	a	point	in	sixteen-

dimensional	space.	It	is	difficult	for	us	to	imagine	a	space	with	more	than	three
dimensions	(perhaps	four,	if	we	include	time),	but	mathematics	has	no	such
inhibitions.
We	have	accomplished	four	things	with	this	process.	First,	we	have	greatly
reduced	the	complexity	of	the	data.	Second,	we	have	reduced	sixteen-
dimensional	data	to	one-dimensional	data	(that	is,	each	sample	is	now	a	single
number).	Third,	we	have	improved	our	ability	to	find	invariant	features,	because

we	are	emphasizing	portions	of	the	space	of	possible	sounds	that	convey	the
most	information.	Most	combinations	of	frequencies	are	physically	impossible
or	at	least	very	unlikely,	so	there	is	no	reason	to	give	equal	space	to	unlikely
combinations	of	inputs	as	to	likely	ones.	This	technique	reduces	the	data	to
equally	likely	possibilities.	The	fourth	benefit	is	that	we	can	use	one-dimensional
pattern	recognizers,	even	though	the	original	data	consisted	of	many	more
dimensions.	This	turned	out	to	be	the	most	efficient	approach	to	utilizing

available	computational	resources.

Reading	Your	Mind	with	Hidden	Markov	Models
With	vector	quantization,	we	simplified	the	data	in	a	way	that	emphasized	key
features,	but	we	still	needed	a	way	to	represent	the	hierarchy	of	invariant
features	that	would	make	sense	of	new	information.	Having	worked	in	the	field
of	pattern	recognition	at	that	time	(the	early	1980s)	for	twenty	years,	I	knew	that
one-dimensional	representations	were	far	more	powerful,	efficient,	and
amenable	to	invariant	results.	There	was	not	a	lot	known	
about	the	neocortex	in
the	early	1980s,	but	based	on	my	experience	with	a	variety	of	pattern	recognition
problems,	I	assumed	that	the	brain	was	also	likely	to	be	reducing	its
multidimensional	data	(whether	from	the	eyes,	the	ears,	or	the	skin)	using	a	one-
dimensional	representation,	especially	as	concepts	rose	in	the	neocortex’s
For	the	speech	recognition	problem,	the	organization	of	information	in	the
speech	signal	appeared	to	be	a	hierarchy	of	patterns,	with	each	pattern

represented	by	a	linear	string	of	elements	with	a	forward	direction.	Each	element
of	a	pattern	could	be	another	pattern	at	a	lower	level,	or	a	fundamental	unit	of
input	(which	in	the	case	of	speech	recognition	would	be	our	quantized	vectors).
You	will	recognize	this	situation	as	consistent	with	the	model	of	the
neocortex	that	I	presented	earlier.	Human	speech,	therefore,	is	produced	by	a
hierarchy	of	linear	patterns	in	the	brain.	If	we	could	simply	examine	these

patterns	in	the	brain	of	the	person	speaking,	it	would	be	a	simple	matter	to	match
her	new	speech	utterances	against	her	brain	patterns	and	understand	what	the
person	was	saying.	Unfortunately	we	do	not	have	direct	access	to	the	brain	of	the
speaker—the	only	information	we	have	is	what	she	actually	said.	Of	course,	that
is	the	whole	point	of	spoken	language—the	speaker	is	sharing	a	piece	of	her
mind	with	her	utterance.
So	I	wondered:	Was	there	a	mathematical	technique	that	would	enable	us	to

infer	the	patterns	in	the	speaker’s	brain	based	on	her	spoken	words?	One
utterance	would	obviously	not	be	sufficient,	but	if	we	had	a	large	number	of
samples,	could	we	use	that	information	to	essentially	read	the	patterns	in	the
speaker’s	neocortex	(or	at	least	formulate	something	mathematically	equivalent
that	would	enable	us	to	recognize	new	utterances)?
People	often	fail	to	appreciate	how	powerful	mathematics	can	be—keep	in
mind	that	our	ability	to	search	much	of	human	knowledge	in	a	fraction	of	a

second	with	search	engines	is	based	on	a	mathematical	technique.	For	the	speech

recognition	problem	I	was	facing	in	the	early	1980s,	it	turned	out	that	the
technique	of	hidden	Markov	models	fit	the	bill	
rather	perfectly.	The	Russian
mathematician	Andrei	Andreyevich	Markov	(1856–1922)	built	a	mathematical
theory	of	hierarchical	sequences	of	states.	The	model	was	based	on	the
possibility	of	traversing	the	states	in	one	chain,	and	if	that	was	successful,
triggering	a	state	in	the	next	higher	level	in	the	hierarchy.	Sound	familiar?
A	simple	example	of	one	layer	of	a	hidden	Markov	model.	S
1
	through
S
4
	represent	the	“hidden”	internal	states.	The	P
i,	j
	transitions	each	represent
the	probability	of	going	from	state	S
i
	to	state	S
j
.	These	probabilities	are
determined	by	the	system	learning	from	training	data	(including	during
actual	use).	A	new	sequence	(such	as	a	new	spoken	utterance)	is	matched
against	these	probabilities	to	determine	the	likelihood	that	this	model
produced	the	sequence.
Markov’s	model	included	probabilities	of	each	state’s	successfully

1
	through
S
4
i,	j
i
	to	state	S
j
occurring.	He	went	on	to	hypothesize	a	situation	in	which	a	system	has	such	a
hierarchy	of	linear	sequences	of	states,	but	those	are	unable	to	be	directly
examined—hence	the	name	
hidden
	Markov	models.	The	lowest	level	of	the
hierarchy	emits	signals,	which	are	all	we	are	allowed	to	see.	Markov	provides	a
mathematical	technique	to	compute	what	the	probabilities	of	each	transition
must	be	based	on	the	observed	output.	The	method	was	subsequently	refined	by
Norbert	Wiener	in	1923.	Wiener’s	refinement	also	provided	a	way	to	determine

hidden
the	connections	in	the	Markov	model;	essentially	any	connection	with	too	low	a
probability	was	considered	not	to	exist.	This	is	essentially	how	the	human
neocortex	trims	connections—if	they	are	rarely	or	never	used,	they	are
considered	unlikely	and	are	pruned	away.	In	our	case,	the	observed	output	is	the
speech	signal	created	by	the	person	talking,	and	the	state	probabilities	and
connections	of	the	Markov	model	constitute	the	neocortical	hierarchy	that
produced	it.
I	envisioned	a	system	in	which	we	would	take	samples	of	human	speech,

apply	the	hidden	Markov	model	technique	to	infer	a	hierarchy	of	states	with

connections	and	probabilities	(essentially	a	simulated	neocortex	for	producing
speech),	and	then	use	this	inferred	hierarchical	network	of	states	to	recognize
new	utterances.	To	create	a	speaker-independent	system,	we	would	use	samples
from	many	different	individuals	to	train	the	hidden	Markov	models.	By	adding
in	the	element	of	hierarchies	to	represent	the	hierarchical	nature	of	information
in	language,	these	were	properly	called	hierarchical	hidden	Markov	models
(HHMMs).
My	colleagues	at	Kurzweil	Applied	Intelligence	were	skeptical	that	this
technique	would	work,	given	that	it	was	a	self-organizing	method	reminiscent	of
neural	nets,	which	had	fallen	out	of	favor	and	with	which	we	had	had	little
success.	I	pointed	out	that	the	network	in	a	neural	net	system	is	fixed	and	does
not	adapt	to	the	input:	The	weights	adapt,	but	the	connections	do	not.	In	the
Markov	model	system,	if	it	was	set	up	correctly,	the	system	would	prune	unused
connections	so	as	to	essentially	adapt	the	topology.

I	established	what	was	considered	a	“skunk	works”	project	(an
organizational	term	for	a	project	off	the	beaten	path	that	has	little	in	the	way	of
formal	resources)	that	consisted	of	me,	one	part-time	programmer,	and	an
electrical	engineer	(to	create	the	frequency	filter	bank).	To	the	surprise	of	my
colleagues,	our	effort	turned	out	to	be	very	successful,	having	succeeded	in
recognizing	speech	comprising	a	large	vocabulary	with	high	accuracy.
After	that	experiment,	all	of	our	subsequent	speech	recognition	efforts	have

been	based	on	hierarchical	hidden	Markov	models.	Other	speech	recognition
companies	appeared	to	discover	the	value	of	this	method	independently,	and
since	the	mid-1980s	most	work	in	automated	speech	recognition	has	been	based
on	this	approach.	Hidden	Markov	models	are	also	used	in	speech	synthesis—
keep	in	mind	that	our	biological	cortical	hierarchy	is	used	not	only	to	recognize
input	but	also	to	produce	output,	for	example,	speech	and	physical	movement.
HHMMs	are	also	used	in	systems	that	understand	the	meaning	of	
natural-

natural-
language	sentences,	which	represents	going	up	the	conceptual	hierarchy.

Hidden	Markov	states	and	possible	transitions	to	produce	a	sequence	of
words	in	natural-language	text.
To	understand	how	the	HHMM	method	works,	we	start	out	with	a	network
that	consists	of	all	the	state	transitions	that	are	possible.	The	vector	quantization
method	described	above	is	critical	here,	because	otherwise	there	would	be	too
many	possibilities	to	consider.
Here	is	a	possible	simplified	initial	topology:
A	simple	hidden	Markov	model	topology	to	recognize	two	spoken
words.
Sample	utterances	are	processed	one	by	one.	For	each,	we	iteratively
modify	the	probabilities	of	the	transitions	to	better	reflect	the	input	sample	we
have	just	processed.	The	Markov	models	used	in	speech	recognition	code	the
likelihood	that	specific	patterns	of	sound	are	found	in	each	phoneme,	how	the
phonemes	influence	one	another,	and	the	likely	orders	of	phonemes.	The	system
can	also	include	probability	networks	on	higher	levels	of	language	structure,

words.
such	as	the	order	of	words,	the	inclusion	of	phrases,	and	so	on	up	the	hierarchy
of	language.

Whereas	our	previous	speech	recognition	systems	incorporated	specific
rules	about	phoneme	structures	and	sequences	explicitly	coded	by	human
linguists,	the	new	HHMM-based	system	was	not	explicitly	told	that	there	are
forty-four	phonemes	in	English,	the	sequences	of	vectors	that	were	likely	for
each	phoneme,	or	what	phoneme	sequences	were	more	likely	than	others.	We	let
the	system	discover	these	“rules”	for	itself	from	thousands	of	hours	of
transcribed	human	speech	data.	The	advantage	of	this	approach	over	hand-coded
rules	is	that	the	models	develop	probabilistic	rules	of	which	human	experts	are
often	not	aware.	We	noticed	that	many	of	the	rules	that	the	system	had
automatically	learned	from	the	data	differed	in	subtle	but	important	ways	from
the	rules	established	by	human	experts.
Once	the	network	was	trained,	we	began	to	attempt	to	recognize	speech	by
considering	the	alternate	paths	through	the	network	and	picking	the	path	that	was
most	likely,	given	the	actual	sequence	of	input	vectors	we	had	seen.	In	other

words,	if	we	saw	a	sequence	of	states	that	was	likely	to	have	produced	that
utterance,	we	concluded	that	the	utterance	came	from	that	cortical	sequence.
This	simulated	HHMM-based	neocortex	included	word	labels,	so	it	was	able	to
propose	a	transcription	of	what	it	heard.
We	were	then	able	to	improve	our	results	further	by	continuing	to	train	the
network	while	we	were	using	it	for	recognition.	As	we	have	discussed,
simultaneous	recognition	and	learning	also	take	place	at	every	level	in	our

biological	neocortical	hierarchy.

Evolutionary	(Genetic)	Algorithms
There	is	another	important	consideration:	How	do	we	set	the	many	parameters
that	control	a	pattern	recognition	system’s	functioning?	These	could	include	the
number	of	vectors	that	we	allow	in	the	vector	quantization	step,	the	initial
topology	of	hierarchical	states	(before	the	training	phase	of	the	hidden	Markov
model	process	prunes	them	back),	the	recognition	threshold	at	each	level	of	the
hierarchy,	the	parameters	that	control	the	handling	of	the	size	parameters,	and
many	others.	We	can	establish	these	based	on	our	intuition,	but	the	results	will
be	far	from	optimal.
We	call	these	parameters	“God	parameters”	because	they	are	set	prior	to	the
self-organizing	method	of	determining	the	topology	of	the	hidden	Markov
models	(or,	in	the	biological	case,	before	the	person	learns	her	lessons	by
similarly	creating	connections	in	her	cortical	hierarchy).	This	is	perhaps	a
misnomer,	given	that	these	initial	DNA-based	design	details	are	determined	by

biological	evolution,	though	some	may	see	the	hand	of	God	in	that	process	(and
while	I	do	consider	evolution	to	be	a	spiritual	process,	this	discussion	properly
belongs	in	
chapter	9
).
When	it	came	to	setting	these	“God	parameters”	in	our	simulated
hierarchical	learning	and	recognizing	system,	we	again	took	a	cue	from	nature
and	decided	to	evolve	them—in	our	case,	using	a	simulation	of	evolution.	We
used	what	are	called	genetic	or	evolutionary	algorithms	(GAs),	which	include
simulated	sexual	reproduction	and	mutations.

belongs	in	
chapter	9
).
Here	is	a	simplified	description	of	how	this	method	works.	First,	we
determine	a	way	to	code	possible	solutions	to	a	given	problem.	If	the	problem	is
optimizing	the	design	parameters	for	a	circuit,	then	we	define	a	list	of	all	of	the
parameters	(with	a	specific	number	of	bits	assigned	to	each	parameter)	that
characterize	the	circuit.	This	list	is	regarded	as	the	genetic	code	in	the	genetic
algorithm.	Then	we	randomly	generate	thousands	or	more	genetic	codes.	Each
such	genetic	code	(which	represents	one	set	of	design	parameters)	is	considered

a	simulated	“solution”	organism.
Now	we	evaluate	each	simulated	organism	in	a	simulated	environment	by
using	a	defined	method	to	assess	each	set	of	parameters.	This	evaluation	is	a	key
to	the	success	of	a	genetic	algorithm.	In	our	example,	we	would	run	each
program	generated	by	the	parameters	and	judge	it	on	appropriate	criteria	(did	it
complete	the	task,	how	long	did	it	take,	and	so	on).	The	best-solution	organisms

(the	best	designs)	are	allowed	to	survive,	and	the	rest	are	eliminated.
Now	we	cause	each	of	the	survivors	to	multiply	themselves	until	they	reach
the	same	number	of	solution	creatures.	This	is	done	by	simulating	sexual
reproduction:	In	other	words,	we	create	new	offspring	where	each	new	creature
draws	one	part	of	its	genetic	code	from	one	parent	and	another	part	from	a
second	parent.	Usually	no	distinction	is	made	between	male	or	female
organisms;	it’s	sufficient	to	generate	an	offspring	from	any	two	arbitrary	parents,
so	we’re	basically	talking	about	same-sex	marriage	here.	This	is	perhaps	not	as
interesting	as	sexual	reproduction	in	the	natural	world,	but	the	relevant	point
here	is	having	two	parents.	As	these	simulated	organisms	multiply,	we	allow
some	mutation	(random	change)	in	the	chromosomes	to	occur.
We’ve	now	defined	one	generation	of	simulated	evolution;	now	we	repeat
these	steps	for	each	subsequent	generation.	At	the	end	of	each	generation	we

determine	how	much	the	designs	have	improved	(that	is,	we	compute	the
average	improvement	in	the	evaluation	function	over	all	the	surviving
organisms).	When	the	degree	of	improvement	in	the	evaluation	of	the	design
creatures	from	one	generation	to	the	next	becomes	very	small,	we	stop	this
iterative	cycle	and	use	the	best	design(s)	in	the	last	generation.	(For	an
algorithmic	description	of	genetic	algorithms,	see	this	endnote.)
11
The	key	to	a	genetic	algorithm	is	that	the	human	designers	don’t	directly

11
program	a	solution;	rather,	we	let	one	emerge	through	an	iterative	process	of
simulated	competition	and	improvement.	Biological	evolution	is	smart	but	slow,
so	to	enhance	its	intelligence	we	greatly	speed	up	its	ponderous	pace.	The
computer	is	fast	enough	to	simulate	many	generations	in	a	matter	of	hours	or
days,	and	we’ve	occasionally	had	them	run	for	as	long	
as	weeks	to	simulate
hundreds	of	thousands	of	generations.	But	we	have	to	go	through	this	iterative

process	only	once;	as	soon	as	we	have	let	this	simulated	evolution	run	its	course,
we	can	apply	the	evolved	and	highly	refined	rules	to	real	problems	in	a	rapid
fashion.	In	the	case	of	our	speech	recognition	systems,	we	used	them	to	evolve
the	initial	topology	of	the	network	and	other	critical	parameters.	We	thus	used
two	self-organizing	methods:	a	GA	to	simulate	the	biological	evolution	that	gave
rise	to	a	particular	cortical	design,	and	HHMMs	to	simulate	the	cortical
organization	that	accompanies	human	learning.

Another	major	requirement	for	the	success	of	a	GA	is	a	valid	method	of
evaluating	each	possible	solution.	This	evaluation	needs	to	be	conducted	quickly,
because	it	must	take	account	of	many	thousands	of	possible	solutions	for	each
generation	of	simulated	evolution.	GAs	are	adept	at	handling	problems	with	too
many	variables	for	which	to	compute	precise	analytic	solutions.	The	design	of	an
engine,	for	example,	may	involve	more	than	a	hundred	variables	and	requires

satisfying	dozens	of	constraints;	GAs	used	by	researchers	at	General	Electric
were	able	to	come	up	with	jet	engine	designs	that	met	the	constraints	more
precisely	than	conventional	methods.
When	using	GAs	you	must,	however,	be	careful	what	you	ask	for.	A	genetic
algorithm	was	used	to	solve	a	block-stacking	problem,	and	it	came	up	with	a
perfect	solution…except	that	it	had	thousands	of	steps.	The	human	programmers
forgot	to	include	minimizing	the	number	of	steps	in	their	evaluation	function.
Scott	Drave’s	Electric	Sheep	project	is	a	GA	that	produces	art.	The
evaluation	function	uses	human	evaluators	in	an	open-source	collaboration
involving	many	thousands	of	people.	The	art	moves	through	time	and	you	can
view	it	at	electricsheep.org.
For	speech	recognition,	the	combination	of	genetic	algorithms	and	hidden
Markov	models	worked	extremely	well.	Simulating	evolution	with	a	GA	was
able	to	substantially	improve	the	performance	of	the	HHMM	networks.	What

evolution	came	up	with	was	far	superior	to	our	original	design,	which	was	based
on	our	intuition.
We	then	experimented	with	introducing	a	series	of	small	variations	in	the
overall	system.	For	example,	we	would	make	perturbations	(minor	random
changes)	to	the	input.	Another	such	change	was	to	have	adjacent	Markov	models
“leak”	into	one	another	by	causing	the	results	of	one	Markov	model	to	influence
models	that	are	“nearby.”	Although	we	did	not	realize	it	at	the	time,	the	sorts	of
adjustments	we	were	experimenting	with	are	very	similar	to	the	types	of

modifications	that	occur	in	biological	cortical	structures.
At	first,	such	changes	hurt	performance	(as	measured	by	accuracy	of
recognition).	But	if	we	reran	evolution	(that	is,	reran	the	GA)	with	these
alterations	in	place,	it	would	adapt	the	system	accordingly,	optimizing	it	for
these	introduced	modifications.	In	general,	this	would	restore	performance.	If	we
then	removed	the	changes	we	had	introduced,	performance	would	be	again
degraded,	because	the	system	had	been	evolved	to	compensate	for	the	changes.

The	adapted	system	became	dependent	on	the	changes.
One	type	of	alteration	that	actually	helped	performance	(after	rerunning	the
GA)	was	to	introduce	small	random	changes	to	the	input.	The	reason	for	this	is
the	well-known	“overfitting”	problem	in	self-organizing	systems.	There	is	a
danger	that	such	a	system	will	overgeneralize	to	the	specific	examples	contained
in	the	training	sample.	By	making	random	adjustments	to	the	input,	the	more

invariant	patterns	in	the	data	survive,	and	the	system	thereby	learns	these	deeper
patterns.	This	helped	only	if	we	reran	the	GA	with	the	randomization	feature	on.
This	introduces	a	dilemma	in	our	understanding	of	our	biological	cortical
circuits.	It	had	been	noticed,	for	example,	that	there	might	indeed	be	a	small

amount	of	leakage	from	one	cortical	connection	to	another,	resulting	from	the
way	that	biological	connections	are	formed:	The	electrochemistry	of	the	axons
and	dendrites	is	apparently	subject	to	the	electromagnetic	effects	of	nearby
connections.	Suppose	we	were	able	to	run	an	experiment	where	we	removed	this
effect	in	an	actual	brain.	That	would	be	difficult	to	actually	carry	out,	but	not
necessarily	impossible.	Suppose	we	conducted	such	an	experiment	and	found
that	the	cortical	circuits	worked	less	effectively	
without	this	neural	leakage.	We
might	then	conclude	that	this	phenomenon	was	a	very	clever	design	by	evolution
and	was	critical	to	the	cortex’s	achieving	its	level	of	performance.	We	might
further	point	out	that	such	a	result	shows	that	the	orderly	model	of	the	flow	of
patterns	up	the	conceptual	hierarchy	and	the	flow	of	predictions	down	the
hierarchy	was	in	fact	much	more	complicated	because	of	this	intricate	influence
of	connections	on	one	another.

But	that	would	not	necessarily	be	an	accurate	conclusion.	Consider	our
experience	with	a	simulated	cortex	based	on	HHMMs,	in	which	we	implemented
a	modification	very	similar	to	interneuronal	cross	talk.	If	we	then	ran	evolution
with	that	phenomenon	in	place,	performance	would	be	restored	(because	the
evolutionary	process	adapted	to	it).	If	we	then	removed	the	cross	talk,
performance	would	be	compromised	again.	In	the	biological	case,	evolution	(that

is,	biological	evolution)	was	indeed	“run”	with	this	phenomenon	in	place.	The
detailed	parameters	of	the	system	have	thereby	been	set	by	biological	evolution
to	be	dependent	on	these	factors,	so	that	changing	them	will	negatively	affect
performance	unless	we	run	evolution	again.	Doing	so	is	feasible	in	the	simulated
world,	where	evolution	only	takes	days	or	weeks,	but	in	the	biological	world	it
would	require	tens	of	thousands	of	years.
So	how	can	we	tell	whether	a	particular	design	feature	of	the	biological

neocortex	is	a	vital	innovation	introduced	by	biological	evolution—that	is,	one
that	is	instrumental	to	our	level	of	intelligence—or	merely	an	artifact	that	the
design	of	the	system	is	now	dependent	on	but	could	have	evolved	without?	We
can	answer	that	question	simply	by	running	simulated	evolution	with	and
without	these	particular	variations	to	the	details	of	the	design	(for	example,	with
and	without	connection	cross	talk).	We	can	even	do	so	with	biological	evolution
if	we’re	examining	the	evolution	of	a	colony	of	microorganisms	where

generations	are	measured	in	hours,	but	it	is	not	practical	for	complex	organisms
such	as	humans.	This	is	another	one	of	the	many	disadvantages	of	biology.
Getting	back	to	our	work	in	speech	recognition,	we	found	that	if	we	ran
evolution	(that	is,	a	GA)	
separately
	on	the	initial	design	of	(1)	the	hierarchical
hidden	Markov	models	that	were	modeling	the	internal	structure	of	phonemes
and	(2)	the	HHMMs’	modeling	of	the	structures	of	words	and	phrases,	we	got

even	better	results.	Both	levels	of	the	system	were	using	HHMMs,	but	the	GA
would	evolve	design	variations	between	these	different	levels.	This	approach
still	allowed	the	modeling	of	phenomena	that	occurs	in	between	the	two	levels,
such	as	the	smearing	of	phonemes	that	often	happens	when	we	string	certain
words	together	(for	example,	“How	are	you	all	doing?”	might	become	“How’re
y’all	doing?”).
It	is	likely	that	a	similar	phenomenon	took	place	in	different	biological
cortical	regions,	in	that	they	have	evolved	small	differences	based	on	the	types
of	patterns	they	deal	with.	Whereas	all	of	these	regions	use	the	same	essential
neocortical	algorithm,	biological	evolution	has	had	enough	time	to	fine-tune	the
design	of	each	of	them	to	be	optimal	for	their	particular	patterns.	However,	as	I
discussed	earlier,	neuroscientists	and	neurologists	have	noticed	substantial
plasticity	in	these	areas,	which	supports	the	idea	of	a	general	neocortical

algorithm.	If	the	fundamental	methods	in	each	region	were	radically	different,
then	such	interchangeability	among	cortical	regions	would	not	be	possible.
The	systems	we	created	in	our	research	using	this	combination	of	self-
organizing	methods	were	very	successful.	In	speech	recognition,	they	were	able
for	the	first	time	to	handle	fully	continuous	speech	and	relatively	unrestricted
vocabularies.	We	were	able	to	achieve	a	high	accuracy	rate	on	a	wide	variety	of

speakers,	accents,	and	dialects.	The	current	state	of	the	art	as	this	book	is	being
written	is	represented	by	a	product	called	Dragon	Naturally	Speaking	(Version
11.5)	for	the	PC	from	Nuance	(formerly	Kurzweil	Computer	Products).	I	suggest
that	people	try	it	if	they	are	skeptical	about	the	performance	of	contemporary
speech	recognition—accuracies	are	often	99	percent	or	higher	after	a	few
minutes	of	training	on	your	voice	on	continuous	speech	and	relatively
unrestricted	vocabularies.	Dragon	
Dictation	is	a	simpler	but	still	impressive	free

app	for	the	iPhone	that	requires	no	voice	training.	Siri,	the	personal	assistant	on
contemporary	Apple	iPhones,	uses	the	same	speech	recognition	technology	with
extensions	to	handle	natural-language	understanding.
The	performance	of	these	systems	is	a	testament	to	the	power	of
mathematics.	With	them	we	are	essentially	computing	what	is	going	on	in	the
neocortex	of	a	speaker—even	though	we	have	no	direct	access	to	that	person’s
brain—as	a	vital	step	in	recognizing	what	the	person	is	saying	and,	in	the	case	of

systems	like	Siri,	what	those	utterances	mean.	We	might	wonder,	if	we	were	to
actually	look	inside	the	speaker’s	neocortex,	would	we	see	connections	and
weights	corresponding	to	the	hierarchical	hidden	Markov	models	computed	by
the	software?	Almost	certainly	we	would	not	find	a	precise	match;	the	neuronal
structures	would	invariably	differ	in	many	details	compared	with	the	models	in
the	computer.	However,	I	would	maintain	that	there	must	be	an	essential

mathematical	equivalence	to	a	high	degree	of	precision	between	the	actual
biology	and	our	attempt	to	emulate	it;	otherwise	these	systems	would	not	work
as	well	as	they	do.

LISP
LISP	(LISt	Processor)	is	a	computer	language,	originally	specified	by	AI	pioneer
John	McCarthy	(1927–2011)	in	1958.	As	its	name	suggests,	LISP	deals	with
lists.	Each	LISP	statement	is	a	list	of	elements;	each	element	is	either	another	list
or	an	“atom,”	which	is	an	irreducible	item	constituting	either	a	number	or	a
symbol.	A	list	included	in	a	list	can	be	the	list	itself,	hence	LISP	is	capable	of
recursion.	Another	way	that	LISP	statements	can	be	recursive	is	if	a	list	includes
a	list,	and	so	on	until	the	original	list	is	specified.	Because	lists	can	include	lists,
LISP	is	also	capable	of	hierarchical	processing.	A	list	can	be	a	conditional	such
that	it	only	“fires”	if	its	elements	are	satisfied.	In	this	way,	hierarchies	of	such
conditionals	can	be	used	to	identify	increasingly	abstract	qualities	of	a	pattern.
LISP	became	the	rage	in	the	artificial	intelligence	community	in	the	1970s
and	early	1980s.	The	conceit	of	the	LISP	enthusiasts	of	the	earlier	decade	was

that	the	language	mirrored	the	way	the	human	brain	worked—that	any	intelligent
process	could	most	easily	and	efficiently	be	coded	in	LISP.	There	followed	a
mini-boomlet	in	“artificial	intelligence”	companies	that	offered	LISP	interpreters
and	related	LISP	products,	but	when	it	became	apparent	in	the	mid-1980s	that
LISP	itself	was	not	a	shortcut	to	creating	intelligent	processes,	the	investment
balloon	collapsed.
It	turns	out	that	the	LISP	enthusiasts	were	not	entirely	wrong.	Essentially,

each	pattern	recognizer	in	the	neocortex	can	be	regarded	as	a	LISP	statement—
each	one	constitutes	a	list	of	elements,	and	each	element	can	be	another	list.	The
neocortex	is	therefore	indeed	engaged	in	list	processing	of	a	symbolic	nature
very	similar	to	that	which	takes	place	in	a	LISP	program.	Moreover,	it	processes
all	300	million	LISP-like	“statements”	simultaneously.
However,	there	were	two	important	features	missing	from	the	world	of
LISP,	one	of	which	was	learning.	LISP	programs	had	to	be	coded	line	by	line	by

human	programmers.	There	were	attempts	to	automatically	code	LISP	programs
using	a	variety	of	methods,	but	these	were	not	an	integral	part	of	the	language’s
concept.	The	neocortex,	in	contrast,	programs	itself,	filling	its	“statements”	(that
is,	the	lists)	with	meaningful	and	actionable	information	from	its	own	experience
and	from	its	own	feedback	loops.	This	is	a	key	principle	of	how	the	neocortex
works:	Each	one	of	its	pattern	recognizers	(that	is,	each	LISP-like	statement)	is
capable	of	filling	in	its	own	list	and	connecting	itself	both	up	and	down	to	other

lists.	The	second	difference	is	the	size	parameters.	One	could	create	a	variant	of

LISP	(coded	in	LISP)	that	would	allow	for	handling	such	parameters,	but	these
are	not	part	of	the	basic	language.
LISP	is	consistent	with	the	original	philosophy	of	the	AI	field,	which	was	to
find	intelligent	solutions	to	problems	and	to	code	them	directly	in	computer
languages.	The	first	attempt	at	a	self-organizing	method	
that	would	teach	itself
from	experience—neural	nets—was	not	successful	because	it	did	not	provide	a
means	to	modify	the	topology	of	the	system	in	response	to	learning.	The
hierarchical	hidden	Markov	model	effectively	provided	that	through	its	pruning
mechanism.	Today,	the	HHMM	together	with	its	mathematical	cousins	makes	up
a	major	portion	of	the	world	of	AI.
A	corollary	of	the	observation	of	the	similarity	of	LISP	and	the	list	structure
of	the	neocortex	is	an	argument	made	by	those	who	insist	that	the	brain	is	too
complicated	for	us	to	understand.	These	critics	point	out	that	the	brain	has
trillions	of	connections,	and	since	each	one	must	be	there	specifically	by	design,

they	constitute	the	equivalent	of	trillions	of	lines	of	code.	As	we’ve	seen,	I’ve
estimated	that	there	are	on	the	order	of	300	million	pattern	processors	in	the
neocortex—or	300	million	lists	where	each	element	in	the	list	is	pointing	to
another	list	(or,	at	the	lowest	conceptual	level,	to	a	basic	irreducible	pattern	from
outside	the	neocortex).	But	300	million	is	still	a	reasonably	big	number	of	LISP
statements	and	indeed	is	larger	than	any	human-written	program	in	existence.

However,	we	need	to	keep	in	mind	that	these	lists	are	not	actually	specified
in	the	initial	design	of	the	nervous	system.	The	brain	creates	these	lists	itself	and
connects	the	levels	automatically	from	its	own	experiences.	This	is	the	key
secret	of	the	neocortex.	The	processes	that	accomplish	this	self-organization	are
much	simpler	than	the	300	million	statements	that	constitute	the	capacity	of	the
neocortex.	Those	processes	are	specified	in	the	genome.	As	I	will	demonstrate	in
chapter	11

chapter	11
,	the	amount	of	unique	information	in	the	genome	(after	lossless
compression)	as	applied	to	the	brain	is	about	25	million	bytes,	which	is
equivalent	to	less	than	a	million	lines	of	code.	The	actual	algorithmic	complexity
is	even	less	than	that,	as	most	of	the	25	million	bytes	of	genetic	information
pertain	to	the	biological	needs	of	the	neurons,	and	not	specifically	to	their
information-processing	capability.	However,	even	25	million	bytes	of	design
information	is	a	level	of	complexity	we	can	handle.

Hierarchical	Memory	Systems
As	I	discussed	in	
chapter	3
,	Jeff	Hawkins	and	Dileep	George	in	2003	and	2004
developed	a	model	of	the	neocortex	incorporating	hierarchical	lists	that	was
described	in	Hawkins	and	Blakeslee’s	2004	book	
.	A	more	up-to-
date	and	very	elegant	presentation	of	the	hierarchical	temporal	memory	method
can	be	found	in	Dileep	George’s	2008	doctoral	dissertation.
12
	Numenta	has
implemented	it	in	a	system	called	NuPIC	(Numenta	Platform	for	Intelligent
Computing)	and	has	developed	pattern	recognition	and	intelligent	data-mining
systems	for	such	clients	as	Forbes	and	Power	Analytics	Corporation.	After
working	at	Numenta,	George	has	started	a	new	company	called	Vicarious
Systems	with	funding	from	the	Founder	Fund	(managed	by	Peter	Thiel,	the
venture	capitalist	behind	Facebook,	and	Sean	Parker,	the	first	president	of
Facebook)	and	from	Good	Ventures,	led	by	Dustin	Moskovitz,	cofounder	of
Facebook.	George	reports	significant	progress	in	automatically	modeling,

learning,	and	recognizing	information	with	a	substantial	number	of	hierarchies.
He	calls	his	system	a	“recursive	cortical	network”	and	plans	applications	for
medical	imaging	and	robotics,	among	other	fields.	The	technique	of	hierarchical
hidden	Markov	models	is	mathematically	very	similar	to	these	hierarchical
memory	systems,	especially	if	we	allow	the	HHMM	system	to	organize	its	own
connections	between	pattern	recognition	modules.	As	mentioned	earlier,
HHMMs	provide	for	an	additional	important	element,	which	is	modeling	the

expected	distribution	of	the	magnitude	(on	some	continuum)	of	each	input	in
computing	the	probability	of	the	existence	of	the	pattern	under	consideration.	I
have	recently	started	a	new	company	called	Patterns,	Inc.,	which	intends	to
develop	hierarchical	self-organizing	neocortical	models	that	utilize	HHMMs	and
related	techniques	for	the	purpose	of	understanding	natural	language.	An
important	emphasis	will	be	on	the	ability	for	the	system	to	design	its	own
hierarchies	in	a	manner	similar	to	a	biological	neocortex.	Our	envisioned	system

will	continually	read	a	wide	range	of	material	such	as	Wikipedia	and	other
knowledge	resources	as	well	
as	listen	to	everything	you	say	and	watch
everything	you	write	(if	you	let	it).	The	goal	is	for	it	to	become	a	helpful	friend
answering	your	questions—
before
	you	even	formulate	them—and	giving	you
useful	information	and	tips	as	you	go	through	your	day.

The	Moving	Frontier	of	AI:	Climbing	the	Competence
Hierarchy
1
.	
A	long	tiresome	speech	delivered	by	a	frothy	pie	topping.
2
.	
A	garment	worn	by	a	child,	perhaps	aboard	an	operatic	ship.
3
.	
Wanted	for	a	twelve-year	crime	spree	of	eating	King	Hrothgar’s
warriors;	officer	Beowulf	has	been	assigned	the	case.
4
.	
It	can	mean	to	develop	gradually	in	the	mind	or	to	carry	during
pregnancy.
5
.	
National	Teacher	Day	and	Kentucky	Derby	Day.
6
.	
Wordsworth	said	they	soar	but	never	roam.
7
.	
Four-letter	word	for	the	iron	fitting	on	the	hoof	of	a	horse	or	a	card-
dealing	box	in	a	casino.
8
.	
In	act	three	of	an	1846	Verdi	opera,	this	Scourge	of	God	is	stabbed	to
death	by	his	lover,	Odabella.
—Examples	of	
Jeopardy!
	queries,	all	of	which	Watson	got	correct.
Answers	are:	meringue	harangue,	pinafore,	Grendel,	gestate,
May,	skylark,	shoe.	For	the	eighth	query,	Watson	replied,
“What	is	Attila?”	The	host	responded	by	saying,	“Be	more
specific?”	Watson	clarified	with,	“What	is	Attila	the	Hun?,”
which	is	correct.

8
.	
Jeopardy!
	
The	computer’s	techniques	for	unraveling	
Jeopardy!
	clues	sounded	just	like
mine.	That	machine	zeroes	in	on	key	words	in	a	clue,	then	combs	its
memory	(in	Watson’s	case,	a	15-terabyte	data	bank	of	human	knowledge)
for	clusters	of	associations	with	these	words.	It	
rigorously	checks	the	top
hits	against	all	the	contextual	information	it	can	muster:	the	category	name;
the	kind	of	answer	being	sought;	the	time,	place,	and	gender	hinted	at	in	the
clue;	and	so	on.	And	when	it	feels	“sure”	enough,	it	decides	to	buzz.	This	is

Jeopardy!
all	an	instant,	intuitive	process	for	a	human	
Jeopardy!
	player,	but	I	felt
convinced	that	under	the	hood	my	brain	was	doing	more	or	less	the	same
thing.
—Ken	Jennings,	human	
Jeopardy!
	champion	who	lost	to	Watson

I,	for	one,	welcome	our	new	robot	overlords.
—Ken	Jennings,	paraphrasing	
The	Simpsons,
	after	losing	to
Watson
	
Oh	my	god.	[Watson]	is	more	intelligent	than	the	average	
Jeopardy!
	player
in	answering	
Jeopardy!
	questions.	That’s	impressively	intelligent.
—Sebastian	Thrun,	former	director	of	the	Stanford	AI	Lab
	
Watson	understands	nothing.	It’s	a	bigger	steamroller.
—Noam	Chomsky
	
Artificial	intelligence	is	all	around	us—we	no	longer	have	our	hand	on	the	plug.
The	simple	act	of	connecting	with	someone	via	a	text	message,	e-mail,	or	cell
phone	call	uses	intelligent	algorithms	to	route	the	information.	Almost	every
product	we	touch	is	originally	designed	in	a	collaboration	between	human	and
artificial	intelligence	and	then	built	in	automated	factories.	If	all	the	AI	systems
decided	to	go	on	strike	tomorrow,	our	civilization	would	be	crippled:	We
couldn’t	get	money	from	our	bank,	and	indeed,	our	money	would	disappear;
communication,	transportation,	and	manufacturing	would	all	grind	to	a	halt.

Fortunately,	our	intelligent	machines	are	not	yet	intelligent	enough	to	organize
such	a	conspiracy.
What	is	new	in	AI	today	is	the	viscerally	impressive	nature	of	publicly
available	examples.	For	example,	consider	Google’s	self-driving	cars	(which	as
of	this	writing	have	gone	over	200,000	miles	in	cities	and	towns),	a	technology
that	will	lead	to	significantly	fewer	crashes,	increased	capacity	of	roads,
alleviating	the	requirement	of	humans	to	perform	the	chore	of	driving,	and	many

other	benefits.	Driverless	cars	are	actually	already	legal	to	operate	on	public
roads	in	Nevada	with	some	restrictions,	although	widespread	usage	by	the	public
throughout	the	world	is	not	expected	until	late	in	this	decade.	Technology	that
intelligently	watches	the	road	and	warns	the	driver	of	impending	dangers	is
already	being	installed	in	cars.	One	such	technology	is	based	in	part	on	the
successful	model	of	visual	processing	in	the	brain	created	by	MIT’s	Tomaso
Poggio.	Called	MobilEye,	it	was	developed	by	Amnon	Shashua,	a	former

postdoctoral	student	of	Poggio’s.	It	is	capable	of	alerting	the	driver	to	such
dangers	as	an	impending	collision	or	a	child	running	in	front	of	the	car	and	has

recently	been	installed	in	cars	by	such	manufacturers	as	Volvo	and	BMW.
I	will	focus	in	this	section	of	the	book	on	language	technologies	for	several
reasons.	Not	surprisingly,	the	hierarchical	nature	of	language	closely	mirrors	the
hierarchical	nature	of	our	thinking.	Spoken	language	was	our	first	technology,
with	written	language	as	the	second.	My	own	work	in	artificial	intelligence,	as
this	chapter	has	demonstrated,	has	been	heavily	focused	on	language.	Finally,
mastering	language	is	a	powerfully	leveraged	capability.	Watson	has	already
read	hundreds	of	millions	of	pages	on	the	Web	and	mastered	the	knowledge
contained	in	these	documents.	Ultimately	machines	will	be	able	to	master	all	of
the	knowledge	on	the	Web—which	is	essentially	all	of	the	knowledge	of	our
human-machine	civilization.
English	mathematician	Alan	Turing	(1912–1954)	based	his	eponymous	test
on	the	ability	of	a	computer	to	converse	in	natural	language	using	text
messages.
13
	Turing	felt	that	all	of	human	intelligence	was	embodied	and

messages.
13
represented	in	language,	and	that	no	machine	could	pass	a	Turing	test	through
simple	language	tricks.	Although	the	Turing	test	is	a	
game	involving	written
language,	Turing	believed	that	the	only	way	that	a	computer	could	pass	it	would
be	for	it	to	actually	possess	the	equivalent	of	human-level	intelligence.	Critics
have	proposed	that	a	true	test	of	human-level	intelligence	should	include	mastery
of	visual	and	auditory	information	as	well.
14
	Since	many	of	my	own	AI	projects
involve	teaching	computers	to	master	such	sensory	information	as	human

14
speech,	letter	shapes,	and	musical	sounds,	I	would	be	expected	to	advocate	the
inclusion	of	these	forms	of	information	in	a	true	test	of	intelligence.	Yet	I	agree
with	Turing’s	original	insight	that	the	text-only	version	of	the	Turing	test	is
sufficient.	Adding	visual	or	auditory	input	or	output	to	the	test	would	not
actually	make	it	more	difficult	to	pass.
One	does	not	need	to	be	an	AI	expert	to	be	moved	by	the	performance	of
Watson	on	
Jeopardy!
	Although	I	have	a	reasonable	understanding	of	the

Watson	on	
Jeopardy!
methodology	used	in	a	number	of	its	key	subsystems,	that	does	not	diminish	my
emotional	reaction	to	watching	it—
him?
—perform.	Even	a	perfect
understanding	of	how	all	of	its	component	systems	work—which	no	one	actually
has—would	not	help	you	to	predict	how	Watson	would	actually	react	to	a	given
situation.	It	contains	hundreds	of	interacting	subsystems,	and	each	of	these	is
considering	millions	of	competing	hypotheses	at	the	same	time,	so	predicting	the

Although	I	have	a	reasonable	understanding	of	the
him?
outcome	is	impossible.	Doing	a	thorough	analysis—after	the	fact—of	Watson’s
deliberations	for	a	single	three-second	query	would	take	a	human	centuries.
To	continue	my	own	history,	in	the	late	1980s	and	1990s	we	began	working
on	natural-language	understanding	in	limited	domains.	You	could	speak	to	one

of	our	products,	called	Kurzweil	Voice,	about	anything	you	wanted,	so	long	as	it
had	to	do	with	editing	documents.	(For	example,	“Move	the	third	paragraph	on
the	previous	page	to	here.”)	It	worked	pretty	well	in	this	limited	but	useful
domain.	We	also	created	systems	with	medical	domain	knowledge	so	that
doctors	could	dictate	patient	reports.	It	had	enough	knowledge	of	fields	such	as
radiology	and	pathology	that	it	could	question	the	doctor	if	something	in	the
report	seemed	unclear,	and	would	
guide	the	physician	through	the	reporting
process.	These	medical	reporting	systems	have	evolved	into	a	billion-dollar
business	at	Nuance.
Understanding	natural	language,	especially	as	an	extension	to	automatic
speech	recognition,	has	now	entered	the	mainstream.	As	of	the	writing	of	this
book,	Siri,	the	automated	personal	assistant	on	the	iPhone	4S,	has	created	a	stir
in	the	mobile	computing	world.	You	can	pretty	much	ask	Siri	to	do	anything	that

a	self-respecting	smartphone	should	be	capable	of	doing	(for	example,	“Where
can	I	get	some	Indian	food	around	here?”	or	“Text	my	wife	that	I’m	on	my	way,”
or	“What	do	people	think	of	the	new	Brad	Pitt	movie?”),	and	most	of	the	time
Siri	will	comply.	Siri	will	entertain	a	small	amount	of	nonproductive	chatter.	If
you	ask	her	what	the	meaning	of	life	is,	she	will	respond	with	“42,”	which	fans
of	
The	Hitchhiker’s	Guide	to	the	Galaxy
	will	recognize	as	its	“answer	to	the

of	
ultimate	question	of	life,	the	universe,	and	everything.”	Knowledge	questions
(including	the	one	about	the	meaning	of	life)	are	answered	by	Wolfram	Alpha,
described	on	
page	170
.	There	is	a	whole	world	of	“chatbots”	who	do	nothing	but
engage	in	small	talk.	If	you	would	like	to	talk	to	our	chatbot	named	Ramona,	go
to	our	Web	site	KurzweilAI.net	and	click	on	“Chat	with	Ramona.”
Some	people	have	complained	to	me	about	Siri’s	failure	to	answer	certain
requests,	but	I	often	recall	that	these	are	the	same	people	who	persistently

page	170
complain	about	human	service	providers	also.	I	sometimes	suggest	that	we	try	it
together,	and	often	it	works	better	than	they	expect.	The	complaints	remind	me
of	the	story	of	the	dog	who	plays	chess.	To	an	incredulous	questioner,	the	dog’s
owner	replies,	“Yeah,	it’s	true,	he	does	play	chess,	but	his	endgame	is	weak.”
Effective	competitors	are	now	emerging,	such	as	Google	Voice	Search.
That	the	general	public	is	now	having	conversations	in	natural	spoken
language	with	their	handheld	computers	marks	a	new	era.	It	is	typical	that	people

dismiss	the	significance	of	a	first-generation	technology	because	of	its
limitations.	A	few	years	later,	when	the	technology	does	work	well,	people	still
dismiss	its	importance	because,	well,	it’s	no	longer	new.	That	being	
said,	Siri
works	impressively	for	a	first-generation	product,	and	it	is	clear	that	this
category	of	product	is	only	going	to	get	better.
Siri	uses	the	HMM-based	speech	recognition	technologies	from	Nuance.

The	natural-language	extensions	were	first	developed	by	the	DARPA-funded
“CALO”	project.
15
	Siri	has	been	enhanced	with	Nuance’s	own	natural-language
technologies,	and	Nuance	offers	a	very	similar	technology	called	Dragon	Go!
16
The	methods	used	for	understanding	natural	language	are	very	similar	to
hierarchical	hidden	Markov	models,	and	indeed	HHMM	itself	is	commonly
used.	Whereas	some	of	these	systems	are	not	specifically	labeled	as	using	HMM
or	HHMM,	the	mathematics	is	virtually	identical.	They	all	involve	hierarchies	of
linear	sequences	where	each	element	has	a	weight,	connections	that	are	self-
adapting,	and	an	overall	system	that	self-organizes	based	on	learning	data.
Usually	the	learning	continues	during	actual	use	of	the	system.	This	approach
matches	the	hierarchical	structure	of	natural	language—it	is	just	a	natural
extension	up	the	conceptual	ladder	from	parts	of	speech	to	words	to	phrases	to
semantic	structures.	It	would	make	sense	to	run	a	genetic	algorithm	on	the

parameters	that	control	the	precise	learning	algorithm	of	this	class	of	hierarchical
learning	systems	and	determine	the	optimal	algorithmic	details.
Over	the	past	decade	there	has	been	a	shift	in	the	way	that	these	hierarchical
structures	are	created.	In	1984	Douglas	Lenat	(born	in	1950)	started	the
ambitious	Cyc	(for	enCYClopedic)	project,	which	aimed	to	create	rules	that
would	codify	everyday	“commonsense”	knowledge.	The	rules	were	organized	in
a	huge	hierarchy,	and	each	rule	involved—again—a	linear	sequence	of	states.

For	example,	one	Cyc	rule	might	state	that	a	dog	has	a	face.	Cyc	can	then	link	to
general	rules	about	the	structure	of	faces:	that	a	face	has	two	eyes,	a	nose,	and	a
mouth,	and	so	on.	We	don’t	need	to	have	one	set	of	rules	for	a	dog’s	face	and
then	another	for	a	cat’s	face,	though	we	may	of	course	want	to	put	in	additional
rules	for	ways	in	which	dogs’	faces	differ	from	cats’	faces.	The	system	also
includes	an	inference	engine:	If	we	have	rules	that	state	that	a	cocker	spaniel	is	a
dog,	that	dogs	are	
animals,	and	that	animals	eat	food,	and	if	we	were	to	ask	the

inference	engine	whether	cocker	spaniels	eat,	the	system	would	respond	that	yes,
cocker	spaniels	eat	food.	Over	the	next	twenty	years,	and	with	thousands	of
person-years	of	effort,	over	a	million	such	rules	were	written	and	tested.
Interestingly,	the	language	for	writing	Cyc	rules—called	CycL—is	almost
identical	to	LISP.
Meanwhile,	an	opposing	school	of	thought	believed	that	the	best	approach
to	natural-language	understanding,	and	to	creating	intelligent	systems	in	general,

was	through	automated	learning	from	exposure	to	a	very	large	number	of
instances	of	the	phenomena	the	system	was	trying	to	master.	A	powerful
example	of	such	a	system	is	Google	Translate,	which	can	translate	to	and	from
fifty	languages.	That’s	2,500	different	translation	directions,	although	for	most

language	pairs,	rather	than	translate	language	1	directly	into	language	2,	it	will
translate	language	1	into	English	and	then	English	into	language	2.	That	reduces
the	number	of	translators	Google	needed	to	build	to	ninety-eight	(plus	a	limited
number	of	non-English	pairs	for	which	there	is	direct	translation).	The	Google
translators	do	not	use	grammatical	rules;	rather,	they	create	vast	databases	for
each	language	pair	of	common	translations	based	on	large	“Rosetta	stone”
corpora	of	translated	documents	between	two	languages.	For	the	six	languages
that	constitute	the	official	languages	of	the	United	Nations,	Google	has	used
United	Nations	documents,	as	they	are	published	in	all	six	languages.	For	less
common	languages,	other	sources	have	been	used.
The	results	are	often	impressive.	DARPA	runs	annual	competitions	for	the
best	automated	language	translation	systems	for	different	language	pairs,	and
Google	Translate	often	wins	for	certain	pairs,	outperforming	systems	created
directly	by	human	linguists.

Over	the	past	decade	two	major	insights	have	deeply	influenced	the	natural-
language-understanding	field.	The	first	has	to	do	with	hierarchies.	Although	the
Google	approach	started	with	association	of	flat	word	sequences	from	one
language	to	another,	the	inherent	hierarchical	nature	of	language	has	inevitably
crept	into	its	operation.	Systems	that	
methodically	incorporate	hierarchical
learning	(such	as	hierarchical	hidden	Markov	models)	provided	significantly
better	performance.	However,	such	systems	are	not	quite	as	automatic	to	build.

Just	as	humans	need	to	learn	approximately	one	conceptual	hierarchy	at	a	time,
the	same	is	true	for	computerized	systems,	so	the	learning	process	needs	to	be
carefully	managed.
The	other	insight	is	that	hand-built	rules	work	well	for	a	core	of	common
basic	knowledge.	For	translations	of	short	passages,	this	approach	often	provides
more	accurate	results.	For	example,	DARPA	has	rated	rule-based	Chinese-to-
English	translators	higher	than	Google	Translate	for	short	passages.	For	what	is

called	the	tail	of	a	language,	which	refers	to	the	millions	of	infrequent	phrases
and	concepts	used	in	it,	the	accuracy	of	rule-based	systems	approaches	an
unacceptably	low	asymptote.	If	we	plot	natural-language-understanding
accuracy	against	the	amount	of	training	data	analyzed,	rule-based	systems	have
higher	performance	initially	but	level	off	at	fairly	low	accuracies	of	about	70
percent.	In	sharp	contrast,	statistical	systems	can	reach	the	high	90s	in	accuracy
but	require	a	great	deal	of	data	to	achieve	that.

Often	we	need	a	combination	of	at	least	moderate	performance	on	a	small
amount	of	training	data	and	then	the	opportunity	to	achieve	high	accuracies	with
a	more	significant	quantity.	Achieving	moderate	performance	quickly	enables	us
to	put	a	system	in	the	field	and	then	to	automatically	collect	training	data	as

people	actually	use	it.	In	this	way,	a	great	deal	of	learning	can	occur	at	the	same
time	that	the	system	is	being	used,	and	its	accuracy	will	improve.	The	statistical
learning	needs	to	be	fully	hierarchical	to	reflect	the	nature	of	language,	which
also	reflects	how	the	human	brain	works.
This	is	also	how	Siri	and	Dragon	Go!	work—using	rules	for	the	most
common	and	reliable	phenomena	and	then	learning	the	“tail”	of	the	language	in
the	hands	of	real	users.	When	the	Cyc	team	realized	that	they	had	reached	a
ceiling	of	performance	based	on	hand-coded	rules,	they	too	adopted	this
approach.	Hand-coded	rules	provide	two	essential	functions.	
They	offer	adequate
initial	accuracy,	so	that	a	trial	system	can	be	placed	into	widespread	usage,
where	it	will	improve	automatically.	Secondly,	they	provide	a	solid	basis	for	the
lower	levels	of	the	conceptual	hierarchy	so	that	the	automated	learning	can	begin
to	learn	higher	conceptual	levels.
As	mentioned	above,	Watson	represents	a	particularly	impressive	example

of	the	approach	of	combining	hand-coded	rules	with	hierarchical	statistical
learning.	IBM	combined	a	number	of	leading	natural-language	programs	to
create	a	system	that	could	play	the	natural-language	game	of	
Jeopardy!
	On
February	14–16,	2011,	Watson	competed	with	the	two	leading	human	players:
Brad	Rutter,	who	had	won	more	money	than	anyone	else	on	the	quiz	show,	and
Ken	Jennings,	who	had	previously	held	the	
Jeopardy!
	championship	for	the
record	time	of	seventy-five	days.

Jeopardy!
	On
Jeopardy!
By	way	of	context,	I	had	predicted	in	my	first	book,	
The	Age	of	Intelligent
Machines
,	written	in	the	mid-1980s,	that	a	computer	would	take	the	world	chess
championship	by	1998.	I	also	predicted	that	when	that	happened,	we	would
either	downgrade	our	opinion	of	human	intelligence,	upgrade	our	opinion	of
machine	intelligence,	or	downplay	the	importance	of	chess,	and	that	if	history
was	a	guide,	we	would	minimize	chess.	Both	of	these	things	happened	in	1997.
When	IBM’s	chess	supercomputer	Deep	Blue	defeated	the	reigning	human

Machines
world	chess	champion,	Garry	Kasparov,	we	were	immediately	treated	to
arguments	that	it	was	to	be	expected	that	a	computer	would	win	at	chess	because
computers	are	logic	machines,	and	chess,	after	all,	is	a	game	of	logic.	Thus	Deep
Blue’s	victory	was	judged	to	be	neither	surprising	nor	significant.	Many	of	its
critics	went	on	to	argue	that	computers	would	never	master	the	subtleties	of
human	language,	including	metaphors,	similes,	puns,	double	entendres,	and
humor.

The	accuracy	of	natural-language-understanding	systems	as	a	function
of	the	amount	of	training	data.	The	best	approach	is	to	combine	rules	for	the
“core”	of	the	language	and	a	databased	approach	for	the	“tail”	of	the
language.
That	is	at	least	one	reason	why	Watson	represents	such	a	significant
milestone:	
Jeopardy!
	is	precisely	such	a	sophisticated	and	challenging	language
task.	Typical	
Jeopardy!
	queries	includes	many	of	these	vagaries	of	human
language.	What	is	perhaps	not	evident	to	many	observers	is	that	Watson	not	only
had	to	master	the	language	in	the	unexpected	and	convoluted	queries,	but	for	the
most	part	its	knowledge	was	not	hand-coded.	It	obtained	that	knowledge	by
actually	reading	200	million	pages	of	natural-language	documents,	including	all
of	Wikipedia	and	other	encyclopedias,	comprising	4	trillion	bytes	of	language-
based	knowledge.	As	readers	of	this	book	are	well	aware,	Wikipedia	is	not
written	in	LISP	or	CycL,	but	rather	in	natural	sentences	that	have	all	of	the

ambiguities	and	intricacies	inherent	in	language.	Watson	needed	to	consider	all	4
trillion	characters	in	its	reference	material	when	responding	to	a	question.	(I
realize	that	
Jeopardy!
	queries	are	answers	in	search	of	a	question,	but	this	is	a
technicality—they	ultimately	are	really	questions.)	If	Watson	can	understand	and
respond	to	questions	based	on	200	million	pages—in	three	seconds!—there	is
nothing	to	stop	similar	systems	from	reading	the	other	billions	of	documents	on
the	Web.	Indeed,	that	effort	is	now	under	way.

Jeopardy!
When	we	were	developing	character	and	speech	recognition	systems	and
early	natural-language-understanding	systems	in	the	1970s	through	1990s,	we
used	a	methodology	of	incorporating	an	“expert	manager.”	We	
would	develop
multiple	systems	to	do	the	same	thing	but	would	incorporate	somewhat	different
approaches	in	each	one.	Some	of	the	differences	were	subtle,	such	as	variations

in	the	parameters	controlling	the	mathematics	of	the	learning	algorithm.	Some
variations	were	fundamental,	such	as	including	rule-based	systems	instead	of
hierarchical	statistical	learning	systems.	The	expert	manager	was	itself	a
software	program	that	was	programmed	to	learn	the	strengths	and	weaknesses	of
these	different	systems	by	examining	their	performance	in	real-world	situations.
It	was	based	on	the	notion	that	these	strengths	were	orthogonal;	that	is,	one
system	would	tend	to	be	strong	where	another	was	weak.	Indeed,	the	overall
performance	of	the	combined	systems	with	the	trained	expert	manager	in	charge
was	far	better	than	any	of	the	individual	systems.
Watson	works	the	same	way.	Using	an	architecture	called	UIMA
(Unstructured	Information	Management	Architecture),	Watson	deploys	literally
hundreds	of	different	systems—many	of	the	individual	language	components	in
Watson	are	the	same	ones	that	are	used	in	publicly	available	natural-language-

understanding	systems—all	of	which	are	attempting	to	either	directly	come	up
with	a	response	to	the	
Jeopardy!
	query	or	else	at	least	provide	some
disambiguation	of	the	query.	UIMA	is	basically	acting	as	the	expert	manager	to
intelligently	combine	the	results	of	the	independent	systems.	UIMA	goes
substantially	beyond	earlier	systems,	such	as	the	one	we	developed	in	the
predecessor	company	to	Nuance,	in	that	its	individual	systems	can	contribute	to

Jeopardy!
a	result	without	necessarily	coming	up	with	a	final	answer.	It	is	sufficient	if	a
subsystem	helps	narrow	down	the	solution.	UIMA	is	also	able	to	compute	how
much	confidence	it	has	in	the	final	answer.	The	human	brain	does	this	also—we
are	probably	very	confident	of	our	response	when	asked	for	our	mother’s	first
name,	but	we	are	less	so	in	coming	up	with	the	name	of	someone	we	met
casually	a	year	ago.
Thus	rather	than	come	up	with	a	single	elegant	approach	to	understanding
the	language	problem	inherent	in	
Jeopardy!
	the	IBM	scientists	combined	all	of

Jeopardy!
the	state-of-the-art	language-understanding	modules	they	could	get	their	hands
on.	Some	use	hierarchical	hidden	Markov	models;	
some	use	mathematical
variants	of	HHMM;	others	use	rule-based	approaches	to	code	directly	a	core	set
of	reliable	rules.	UIMA	evaluates	the	performance	of	each	system	in	actual	use
and	combines	them	in	an	optimal	way.	There	is	some	misunderstanding	in	the
public	discussions	of	Watson	in	that	the	IBM	scientists	who	created	it	often
focus	on	UIMA,	which	is	the	expert	manager	they	created.	This	leads	to

comments	by	some	observers	that	Watson	has	no	real	understanding	of	language
because	it	is	difficult	to	identify	where	this	understanding	resides.	Although	the
UIMA	framework	also	learns	from	its	own	experience,	Watson’s
“understanding”	of	language	cannot	be	found	in	UIMA	alone	but	rather	is
distributed	across	all	of	its	many	components,	including	the	self-organizing

language	modules	that	use	methods	similar	to	HHMM.
A	separate	part	of	Watson’s	technology	uses	UIMA’s	confidence	estimate	in
its	answers	to	determine	how	to	place	
Jeopardy!
	bets.	While	the	Watson	system
is	specifically	optimized	to	play	this	particular	game,	its	core	language-and
knowledge-searching	technology	can	easily	be	adapted	to	other	broad	tasks.	One
might	think	that	less	commonly	shared	professional	knowledge,	such	as	that	in
the	medical	field,	would	be	more	difficult	to	master	than	the	general-purpose
“common”	knowledge	that	is	required	to	play	
Jeopardy!
	Actually,	the	opposite	is
the	case:	Professional	knowledge	tends	to	be	more	highly	organized,	structured,
and	less	ambiguous	than	its	commonsense	counterpart,	so	it	is	highly	amenable
to	accurate	natural-language	understanding	using	these	techniques.	As
mentioned,	IBM	is	currently	working	with	Nuance	to	adapt	the	Watson
technology	to	medicine.
The	conversation	that	takes	place	when	Watson	is	playing	
Jeopardy!
	is	a

Jeopardy!
Jeopardy!
	is	a
brief	one:	A	question	is	posed,	and	Watson	comes	up	with	an	answer.	(Again,
technically,	it	comes	up	with	a	question	to	respond	to	an	answer.)	It	does	not
engage	in	a	conversation	that	would	require	tracking	all	of	the	earlier	statements
of	all	participants.	(Siri	actually	does	do	this	to	a	limited	extent:	If	you	ask	it	to
send	a	message	to	your	wife,	it	will	ask	you	to	identify	her,	but	it	will	remember
who	she	is	for	subsequent	requests.)	Tracking	
all	of	the	information	in	a

Jeopardy!
	is	a
conversation—a	task	that	would	clearly	be	required	to	pass	the	Turing	test—is	a
significant	additional	requirement	but	not	fundamentally	more	difficult	than
what	Watson	is	doing	already.	After	all,	Watson	has	read	hundreds	of	millions	of
pages	of	material,	which	obviously	includes	many	stories,	so	it	is	capable	of
tracking	through	complicated	sequential	events.	It	should	therefore	be	able	to
follow	its	own	conversations	and	take	that	into	consideration	in	its	subsequent
replies.
Another	limitation	of	the	
Jeopardy!

replies.
Jeopardy!
	game	is	that	the	answers	are	generally
brief:	It	does	not,	for	example,	pose	questions	of	the	sort	that	ask	contestants	to
name	the	five	primary	themes	of	
A	Tale	of	Two	Cities
.	To	the	extent	that	it	can
find	documents	that	do	discuss	the	themes	of	this	novel,	a	suitably	modified
version	of	Watson	should	be	able	to	respond	to	this.	Coming	up	with	such
themes	on	its	own	from	just	reading	the	book,	and	not	essentially	copying	the
thoughts	(even	without	the	words)	of	other	thinkers,	is	another	matter.	Doing	so

would	constitute	a	higher-level	task	than	Watson	is	capable	of	today—it	is	what	I
call	a	Turing	test–level	task.	(That	being	said,	I	will	point	out	that	most	humans
do	not	come	up	with	their	own	original	thoughts	either	but	copy	the	ideas	of
their	peers	and	opinion	leaders.)	At	any	rate,	this	is	2012,	not	2029,	so	I	would
not	expect	Turing	test–level	intelligence	yet.	On	yet	another	hand,	I	would	point

out	that	evaluating	the	answers	to	questions	such	as	finding	key	ideas	in	a	novel
is	itself	not	a	straightforward	task.	If	someone	is	asked	who	signed	the
Declaration	of	Independence,	one	can	determine	whether	or	not	her	response	is
true	or	false.	The	validity	of	answers	to	higher-level	questions	such	as	describing
the	themes	of	a	creative	work	is	far	less	easily	established.
It	is	noteworthy	that	although	Watson’s	language	skills	are	actually
somewhat	below	that	of	an	educated	human,	it	was	able	to	defeat	the	best	two
Jeopardy!
	players	in	the	world.	It	could	accomplish	this	because	it	is	able	to
combine	its	language	ability	and	knowledge	understanding	with	the	perfect
recall	and	highly	accurate	memories	that	machines	possess.	That	is	why	we	have
already	largely	assigned	our	personal,	social,	and	historical	memories	to	them.
Although	I’m	not	prepared	to	move	up	my	prediction	of	a	computer	passing
the	Turing	test	by	2029,	the	progress	that	has	been	achieved	in	systems	like

Jeopardy!
Watson	should	give	anyone	substantial	confidence	that	the	advent	of	Turing-
level	AI	is	close	at	hand.	If	one	were	to	create	a	version	of	Watson	that	was
optimized	for	the	Turing	test,	it	would	probably	come	pretty	close.
American	philosopher	John	Searle	(born	in	1932)	argued	recently	that
Watson	is	not	capable	of	thinking.	Citing	his	“Chinese	room”	thought
experiment	(which	I	will	discuss	further	in	
chapter	11
),	he	states	that	Watson	is
only	manipulating	symbols	and	does	not	understand	the	meaning	of	those

chapter	11
symbols.	Actually,	Searle	is	not	describing	Watson	accurately,	since	its
understanding	of	language	is	based	on	hierarchical	statistical	processes—not	the
manipulation	of	symbols.	The	only	way	that	Searle’s	characterization	would	be
accurate	is	if	we	considered	every	step	in	Watson’s	self-organizing	processes	to
be	“the	manipulation	of	symbols.”	But	if	that	were	the	case,	then	the	human
brain	would	not	be	judged	capable	of	thinking	either.
It	is	amusing	and	ironic	when	observers	criticize	Watson	for	
just
	doing

just
	doing
statistical	analysis	of	language	as	opposed	to	possessing	the	“true”	understanding
of	language	that	humans	have.	Hierarchical	statistical	analysis	is	exactly	what
the	human	brain	is	doing	when	it	is	resolving	multiple	hypotheses	based	on
statistical	inference	(and	indeed	at	every	level	of	the	neocortical	hierarchy).	Both
Watson	and	the	human	brain	learn	and	respond	based	on	a	similar	approach	to
hierarchical	understanding.	In	many	respects	Watson’s	knowledge	is	far	more
extensive	than	a	human’s;	no	human	can	claim	to	have	mastered	all	of

Wikipedia,	which	is	only	part	of	Watson’s	knowledge	base.	Conversely,	a	human
can	today	master	more	conceptual	levels	than	Watson,	but	that	is	certainly	not	a
permanent	gap.
One	important	system	that	demonstrates	the	strength	of	computing	applied
to	organized	knowledge	is	Wolfram	Alpha,	an	answer	engine	(as	opposed	to	a

search	engine)	developed	by	British	mathematician	and	scientist	Dr.	Wolfram
(born	1959)	and	his	colleagues	at	Wolfram	Research.	
For	example,	if	you	ask
Wolfram	Alpha	(at	WolframAlpha.com),	“How	many	primes	are	there	under	a
million?”	it	will	respond	with	“78,498.”	It	did	not	look	up	the	answer,	it
computed	it,	and	following	the	answer	it	provides	the	equations	it	used.	If	you
attempted	to	get	that	answer	using	a	conventional	search	engine,	it	would	direct
you	to	links	where	you	could	find	the	algorithms	required.	You	would	then	have
to	plug	those	formulas	into	a	system	such	as	Mathematica,	also	developed	by	Dr.
Wolfram,	but	this	would	obviously	require	a	lot	more	work	(and	understanding)
than	simply	asking	Alpha.
Indeed,	Alpha	consists	of	15	million	lines	of	Mathematica	code.	What
Alpha	is	doing	is	literally	computing	the	answer	from	approximately	10	trillion
bytes	of	data	that	have	been	carefully	curated	by	the	Wolfram	Research	staff.

You	can	ask	a	wide	range	of	factual	questions,	such	as	“What	country	has	the
highest	GDP	per	person?”	(Answer:	Monaco,	with	$212,000	per	person	in	U.S.
dollars),	or	“How	old	is	Stephen	Wolfram?”	(Answer:	52	years,	9	months,	2	days
as	of	the	day	I	am	writing	this).	As	mentioned,	Alpha	is	used	as	part	of	Apple’s
Siri;	if	you	ask	Siri	a	factual	question,	it	is	handed	off	to	Alpha	to	handle.	Alpha
also	handles	some	of	the	searches	posed	to	Microsoft’s	Bing	search	engine.

In	a	recent	blog	post,	Dr.	Wolfram	reported	that	Alpha	is	now	providing
successful	responses	90	percent	of	the	time.
17
	He	also	reports	an	exponential
decrease	in	the	failure	rate,	with	a	half-life	of	around	eighteen	months.	It	is	an
impressive	system,	and	uses	handcrafted	methods	and	hand-checked	data.	It	is	a
testament	to	why	we	created	computers	in	the	first	place.	As	we	discover	and
compile	scientific	and	mathematical	methods,	computers	are	far	better	than
unaided	human	intelligence	in	implementing	them.	Most	of	the	known	scientific

17
methods	have	been	encoded	in	Alpha,	along	with	continually	updated	data	on
topics	ranging	from	economics	to	physics.	In	a	private	conversation	I	had	with
Dr.	Wolfram,	he	estimated	that	self-organizing	methods	such	as	those	used	in
Watson	typically	achieve	about	an	80	percent	accuracy	when	they	are	working
well.	Alpha,	he	pointed	out,	is	achieving	about	a	90	percent	accuracy.	Of	course,
there	is	self-selection	in	both	of	these	accuracy	numbers	in	that	users	(such	as
myself)	have	learned	what	kinds	of	questions	Alpha	is	good	at,	and	a	similar

factor	applies	to	the	self-organizing	methods.	Eighty	percent	appears	to	be	a
reasonable	estimate	of	how	accurate	Watson	is	on	
Jeopardy!
	queries,	but	this
was	sufficient	to	defeat	the	best	humans.
It	is	my	view	that	self-organizing	methods	such	as	I	articulated	in	the
pattern	recognition	theory	of	mind	are	needed	to	understand	the	elaborate	and
often	ambiguous	hierarchies	we	encounter	in	real-world	phenomena,	including

human	language.	An	ideal	combination	for	a	robustly	intelligent	system	would
be	to	combine	hierarchical	intelligence	based	on	the	PRTM	(which	I	contend	is
how	the	human	brain	works)	with	precise	codification	of	scientific	knowledge
and	data.	That	essentially	describes	a	human	with	a	computer.	We	will	enhance
both	poles	of	intelligence	in	the	years	ahead.	With	regard	to	our	biological
intelligence,	although	our	neocortex	has	significant	plasticity,	its	basic
architecture	is	limited	by	its	physical	constraints.	Putting	additional	neocortex
into	our	foreheads	was	an	important	evolutionary	innovation,	but	we	cannot	now
easily	expand	the	size	of	our	frontal	lobes	by	a	factor	of	a	thousand,	or	even	by
10	percent.	That	is,	we	cannot	do	so	biologically,	but	that	is	exactly	what	we	will
do	technologically.

A	Strategy	for	Creating	a	Mind
There	are	billions	of	neurons	in	our	brains,	but	what	are	neurons?	Just	cells.
The	brain	has	no	knowledge	until	connections	are	made	between	neurons.
All	that	we	know,	all	that	we	are,	comes	from	the	way	our	neurons	are
connected.
—Tim	Berners-Lee
	
Let’s	use	the	observations	I	have	discussed	above	to	begin	building	a	brain.	We
will	start	by	building	a	pattern	recognizer	that	meets	the	necessary	
Next	we’ll	make	as	many	copies	of	the	recognizer	as	we	have	memory	and
computational	resources	to	support.	Each	recognizer	computes	the	probability
that	its	pattern	has	been	recognized.	In	doing	so,	it	takes	into	consideration	the
observed	magnitude	of	each	input	(in	some	appropriate	continuum)	and	matches
these	against	the	learned	size	and	size	variability	parameters	associated	with
each	input.	The	recognizer	triggers	its	simulated	axon	if	that	computed
probability	exceeds	a	threshold.	This	threshold	and	the	parameters	that	control

the	computation	of	the	pattern’s	probability	are	among	the	parameters	we	will
optimize	with	a	genetic	algorithm.	Because	it	is	not	a	requirement	that	every
input	be	active	for	a	pattern	to	be	recognized,	this	provides	for	autoassociative
recognition	(that	is,	recognizing	a	pattern	based	on	only	part	of	the	pattern	being
present).	We	also	allow	for	inhibitory	signals	(signals	that	indicate	that	the
pattern	is	less	likely).
Recognition	of	the	pattern	sends	an	active	signal	up	the	simulated	axon	of

this	pattern	recognizer.	This	axon	is	in	turn	connected	to	one	or	more	pattern
recognizers	at	the	next	higher	conceptual	level.	All	of	the	pattern	recognizers
connected	at	the	next	higher	conceptual	level	are	accepting	this	pattern	as	one	of
its	inputs.	Each	pattern	recognizer	also	sends	signals	down	to	pattern	recognizers
at	lower	conceptual	levels	whenever	most	of	a	pattern	has	been	recognized,
indicating	that	the	rest	of	the	pattern	is	“expected.”	Each	pattern	recognizer	has

one	or	more	of	these	expected	signal	input	channels.	When	an	expected	signal	is
received	in	this	way,	the	threshold	for	recognition	of	this	pattern	recognizer	is
lowered	(made	easier).
The	pattern	recognizers	are	responsible	for	“wiring”	themselves	to	other

pattern	recognizers	up	and	down	the	conceptual	hierarchy.	Note	that	all	the
“wires”	in	a	software	implementation	operate	via	virtual	links	(which,	like	Web
links,	are	basically	memory	pointers)	and	not	actual	wires.	This	system	is
actually	much	more	flexible	than	that	in	the	biological	brain.	In	a	human	brain,
new	patterns	have	to	be	assigned	to	an	actual	physical	pattern	recognizer,	and
new	connections	have	to	be	made	with	an	
actual	axon-to-dendrite	link.	Usually
this	means	taking	an	existing	physical	connection	that	is	approximately	what	is
needed	and	then	growing	the	necessary	axon	and	dendrite	extensions	to
complete	the	full	connection.
Another	technique	used	in	biological	mammalian	brains	is	to	start	with	a
large	number	of	possible	connections	and	then	prune	the	neural	connections	that
are	not	used.	If	a	biological	neocortex	reassigns	cortical	pattern	recognizers	that
have	already	learned	older	patterns	in	order	to	learn	more	recent	material,	then

the	connections	need	to	be	physically	reconfigured.	Again,	these	tasks	are	much
simpler	in	a	software	implementation.	We	simply	assign	new	memory	locations
to	a	new	pattern	recognizer	and	use	memory	links	for	the	connections.	If	the
digital	neocortex	wishes	to	reassign	cortical	memory	resources	from	one	set	of
patterns	to	another,	it	simply	returns	the	old	pattern	recognizers	to	memory	and
then	makes	the	new	assignment.	This	sort	of	“garbage	collection”	and

reassignment	of	memory	is	a	standard	feature	of	the	architecture	of	many
software	systems.	In	our	digital	brain	we	would	also	back	up	old	memories
before	discarding	them	from	the	active	neocortex,	a	precaution	we	can’t	take	in
our	biological	brains.
There	are	a	variety	of	mathematical	techniques	that	can	be	employed	to
implement	this	approach	to	self-organizing	hierarchical	pattern	recognition.	The
method	I	would	use	is	hierarchical	hidden	Markov	models,	for	several	reasons.
From	my	personal	perspective,	I	have	several	decades	of	familiarity	with	this

method,	having	used	it	in	the	earliest	speech	recognition	and	natural-language
systems	starting	in	the	1980s.	From	the	perspective	of	the	overall	field,	there	is
greater	experience	with	hidden	Markov	models	than	with	any	other	approach	for
pattern	recognition	tasks.	They	are	also	extensively	used	in	natural-language
understanding.	Many	NLU	systems	use	techniques	that	are	at	least
mathematically	similar	to	HHMM.
Note	that	not	all	hidden	Markov	model	systems	are	fully	hierarchical.	Some

allow	for	just	a	few	levels	of	hierarchy—for	example,	going	from	acoustic	states
to	phonemes	to	words.	To	build	a	brain,	we	will	want	to	enable	our	system	to
create	as	many	new	levels	of	hierarchy	as	needed.	
Also,	most	hidden	Markov
model	systems	are	not	fully	self-organizing.	Some	have	fixed	connections,
although	these	systems	do	effectively	prune	many	of	their	starting	connections

by	allowing	them	to	evolve	zero	connection	weights.	Our	systems	from	the
1980s	and	1990s	automatically	pruned	connections	with	connection	weights
below	a	certain	level	and	also	allowed	for	making	new	connections	to	better
model	the	training	data	and	to	learn	on	the	fly.	A	key	requirement,	I	believe,	is	to
allow	for	the	system	to	flexibly	create	its	own	topologies	based	on	the	patterns	it
is	exposed	to	while	learning.	We	can	use	the	mathematical	technique	of	linear
programming	to	optimally	assign	connections	to	new	pattern	recognizers.
Our	digital	brain	will	also	accommodate	substantial	redundancy	of	each
pattern,	especially	ones	that	occur	frequently.	This	allows	for	robust	recognition
of	common	patterns	and	is	also	one	of	the	key	methods	to	achieving	invariant
recognition	of	different	forms	of	a	pattern.	We	will,	however,	need	rules	for	how
much	redundancy	to	permit,	as	we	don’t	want	to	use	up	excessive	amounts	of
memory	on	very	common	low-level	patterns.

The	rules	regarding	redundancy,	recognition	thresholds,	and	the	effect	on
the	threshold	of	a	“this	pattern	is	expected”	indication	are	a	few	examples	of	key
overall	parameters	that	affect	the	performance	of	this	type	of	self-organizing
system.	I	would	initially	set	these	parameters	based	on	my	intuition,	but	we
would	then	optimize	them	using	a	genetic	algorithm.
A	very	important	consideration	is	the	education	of	a	brain,	whether	a
biological	or	a	software	one.	As	I	discussed	earlier,	a	hierarchical	pattern

recognition	system	(digital	or	biological)	will	only	learn	about	two—preferably
one—hierarchical	levels	at	a	time.	To	bootstrap	the	system	I	would	start	with
previously	trained	hierarchical	networks	that	have	already	learned	their	lessons
in	recognizing	human	speech,	printed	characters,	and	natural-language
structures.	Such	a	system	would	be	capable	of	reading	natural-language
documents	but	would	only	be	able	to	master	approximately	one	conceptual	level

at	a	time.	Previously	learned	levels	would	provide	a	relatively	stable	basis	to
learn	the	next	level.	The	system	can	read	
the	same	documents	over	and	over,
gaining	new	conceptual	levels	with	each	subsequent	reading,	similar	to	the	way
people	reread	and	achieve	a	deeper	understanding	of	texts.	Billions	of	pages	of
material	are	available	on	the	Web.	Wikipedia	itself	has	about	four	million	articles
in	the	English	version.
I	would	also	provide	a	critical	thinking	module,	which	would	perform	a

continual	background	scan	of	all	of	the	existing	patterns,	reviewing	their
compatibility	with	the	other	patterns	(ideas)	in	this	software	neocortex.	We	have
no	such	facility	in	our	biological	brains,	which	is	why	people	can	hold
completely	inconsistent	thoughts	with	equanimity.	Upon	identifying	an
inconsistent	idea,	the	digital	module	would	begin	a	search	for	a	resolution,
including	its	own	cortical	structures	as	well	as	all	of	the	vast	literature	available

to	it.	A	resolution	might	simply	mean	determining	that	one	of	the	inconsistent
ideas	is	simply	incorrect	(if	contraindicated	by	a	preponderance	of	conflicting
data).	More	constructively,	it	would	find	an	idea	at	a	higher	conceptual	level	that
resolves	the	apparent	contradiction	by	providing	a	perspective	that	explains	each
idea.	The	system	would	add	this	resolution	as	a	new	pattern	and	link	to	the	ideas
that	initially	triggered	the	search	for	the	resolution.	This	critical	thinking	module
would	run	as	a	continual	background	task.	It	would	be	very	beneficial	if	human
brains	did	the	same	thing.
I	would	also	provide	a	module	that	identifies	open	questions	in	every
discipline.	As	another	continual	background	task,	it	would	search	for	solutions	to
them	in	other	disparate	areas	of	knowledge.	As	I	noted,	the	knowledge	in	the
neocortex	consists	of	deeply	nested	patterns	of	patterns	and	is	therefore	entirely
metaphorical.	We	can	use	one	pattern	to	provide	a	solution	or	insight	in	an
apparently	disconnected	field.

As	an	example,	recall	the	metaphor	I	used	in	
chapter	4
	relating	the	random
movements	of	molecules	in	a	gas	to	the	random	movements	of	evolutionary
change.	Molecules	in	a	gas	move	randomly	with	no	apparent	sense	of	direction.
Despite	this,	virtually	every	molecule	in	a	gas	in	a	beaker,	given	sufficient	time,
will	leave	the	beaker.	I	noted	that	this	provides	a	
perspective	on	an	important
question	concerning	the	evolution	of	intelligence.	Like	molecules	in	a	gas,
evolutionary	changes	also	move	every	which	way	with	no	apparent	direction.

chapter	4
Yet	we	nonetheless	see	a	movement	toward	greater	complexity	and	greater
intelligence,	indeed	to	evolution’s	supreme	achievement	of	evolving	a	neocortex
capable	of	hierarchical	thinking.	So	we	are	able	to	gain	an	insight	into	how	an
apparently	purposeless	and	directionless	process	can	achieve	an	apparently
purposeful	result	in	one	field	(biological	evolution)	by	looking	at	another	field
(thermodynamics).
I	mentioned	earlier	how	Charles	Lyell’s	insight	that	minute	changes	to	rock

formations	by	streaming	water	could	carve	great	valleys	over	time	inspired
Charles	Darwin	to	make	a	similar	observation	about	continual	minute	changes	to
the	characteristics	of	organisms	within	a	species.	This	metaphor	search	would	be
another	continual	background	process.
We	should	provide	a	means	of	stepping	through	multiple	lists
simultaneously	to	provide	the	equivalent	of	structured	thought.	A	list	might	be
the	statement	of	the	constraints	that	a	solution	to	a	problem	must	satisfy.	Each

step	can	generate	a	recursive	search	through	the	existing	hierarchy	of	ideas	or	a
search	through	available	literature.	The	human	brain	appears	to	be	able	to	handle
only	four	simultaneous	lists	at	a	time	(without	the	aid	of	tools	such	as
computers),	but	there	is	no	reason	for	an	artificial	neocortex	to	have	such	a

limitation.
We	will	also	want	to	enhance	our	artificial	brains	with	the	kind	of
intelligence	that	computers	have	always	excelled	in,	which	is	the	ability	to
master	vast	databases	accurately	and	implement	known	algorithms	quickly	and
efficiently.	Wolfram	Alpha	uniquely	combines	a	great	many	known	scientific
methods	and	applies	them	to	carefully	collected	data.	This	type	of	system	is	also
going	to	continue	to	improve	given	Dr.	Wolfram’s	observation	of	an	exponential
decline	in	error	rates.
Finally,	our	new	brain	needs	a	purpose.	A	purpose	is	expressed	as	a	series
of	goals.	In	the	case	of	our	biological	brains,	our	goals	are	established	by	the
pleasure	and	fear	centers	that	we	have	inherited	from	the	old	brain.	These
primitive	drives	were	initially	set	by	biological	evolution	to	foster	the	
survival	of
species,	but	the	neocortex	has	enabled	us	to	sublimate	them.	Watson’s	goal	was
to	respond	to	
Jeopardy!
	queries.	Another	simply	stated	goal	could	be	to	pass	the

Jeopardy!
Turing	test.	To	do	so,	a	digital	brain	would	need	a	human	narrative	of	its	own
fictional	story	so	that	it	can	pretend	to	be	a	biological	human.	It	would	also	have
to	dumb	itself	down	considerably,	for	any	system	that	displayed	the	knowledge
of,	say,	Watson	would	be	quickly	unmasked	as	nonbiological.
More	interestingly,	we	could	give	our	new	brain	a	more	ambitious	goal,
such	as	contributing	to	a	better	world.	A	goal	along	these	lines,	of	course,	raises

a	lot	of	questions:	Better	for	whom?	Better	in	what	way?	For	biological	humans?
For	all	conscious	beings?	If	that	is	the	case,	who	or	what	is	conscious?
As	nonbiological	brains	become	as	capable	as	biological	ones	of	effecting
changes	in	the	world—indeed,	ultimately	far	more	capable	than	unenhanced
biological	ones—we	will	need	to	consider	their	moral	education.	A	good	place	to
start	would	be	with	one	old	idea	from	our	religious	traditions:	the	golden	rule.

THE	MIND	AS	COMPUTER
	
Shaped	a	little	like	a	loaf	of	French	country	bread,	our	brain	is	a	crowded
chemistry	lab,	bustling	with	nonstop	neural	conversations.	Imagine	the
brain,	that	shiny	mound	of	being,	that	mouse-gray	parliament	of	cells,	that
dream	factory,	that	petit	tyrant	inside	a	ball	of	bone,	that	huddle	of	neurons
calling	all	the	plays,	that	little	everywhere,	that	fickle	pleasuredome,	that
wrinkled	wardrobe	of	selves	stuffed	into	the	skull	like	too	many	clothes	into
a	gym	bag.
—Diane	Ackerman
	
Brains	exist	because	the	distribution	of	resources	necessary	for	survival	and
the	hazards	that	threaten	survival	vary	in	space	and	time.
—John	M.	Allman
	
The	modern	geography	of	the	brain	has	a	deliciously	antiquated	feel	to	it—
rather	like	a	medieval	map	with	the	known	world	encircled	by	terra
incognita	where	monsters	roam.
—David	Bainbridge
	
In	mathematics	you	don’t	understand	things.	You	just	get	used	to	them.
—John	von	Neumann
	
E
	ver	since	the	emergence	of	the	computer	in	the	middle	of	the	twentieth

E
century,	there	has	been	ongoing	debate	not	only	about	the	ultimate	extent	of	its
abilities	but	about	whether	the	human	brain	itself	could	be	considered	a	form	of
computer.	As	far	as	the	latter	question	was	concerned,	the	consensus	has	veered
from	viewing	these	two	kinds	of	information-processing	entities	as	being

essentially	the	same	to	their	being	fundamentally	different.	So	
is
	the	brain	a
computer?
When	computers	first	became	a	popular	topic	in	the	1940s,	they	were
immediately	regarded	as	thinking	machines.	The	ENIAC,	which	was	announced
in	1946,	was	described	in	the	press	as	a	“giant	brain.”	As	computers	became
commercially	available	in	the	following	decade,	ads	routinely	referred	to	them	as
brains	capable	of	feats	that	ordinary	biological	brains	could	not	match.
A	1957	ad	showing	the	popular	conception	of	a	computer	as	a	giant
brain.
Computer	programs	quickly	enabled	the	machines	to	live	up	to	this	billing.
The	“general	problem	solver,”	created	in	1959	by	Herbert	A.	Simon,	J.	C.	Shaw,
and	Allen	Newell	at	Carnegie	Mellon	University,	was	able	to	devise	a	proof	to	a
theorem	that	mathematicians	Bertrand	Russell	(1872–1970)	and	Alfred	North
Whitehead	(1861–1947)	had	been	unable	to	solve	in	their	famous	1913	work
Principia	Mathematica
.	What	became	apparent	in	the	decades	that	followed	was

brain.
that	computers	could	readily	significantly	exceed	unassisted	human	capability	in
such	intellectual	exercises	as	solving	mathematical	problems,	diagnosing
disease,	and	playing	chess	but	had	difficulty	with	controlling	a	robot	tying
shoelaces	or	with	understanding	the	commonsense	language	that	a	five-year-old
child	could	comprehend.	Computers	are	only	now	starting	to	master	these	sorts

of	skills.	Ironically,	the	evolution	of	computer	intelligence	has	proceeded	in	the
opposite	direction	of	human	maturation.
The	issue	of	whether	or	not	the	computer	and	the	human	brain	are	at	some
level	equivalent	remains	controversial	today.	In	the	introduction	I	mentioned	that
there	were	millions	of	links	for	quotations	on	the	complexity	of	the	human	brain.
Similarly,	a	Google	inquiry	for	“Quotations:	the	brain	is	not	a	computer”	also
returns	millions	of	links.	In	my	view,	statements	along	these	lines	are	akin	to
saying,	“Applesauce	is	not	an	apple.”	Technically	that	statement	is	true,	but	you
can	make	applesauce	from	an	apple.	Perhaps	more	to	the	point,	it	is	like	saying,
“Computers	are	not	word	processors.”	It	is	true	that	a	computer	and	a	word
processor	exist	at	different	conceptual	levels,	but	a	computer	can	become	a	word
processor	if	it	is	running	word	processing	software	and	not	otherwise.	Similarly,
a	computer	can	become	a	brain	if	it	is	running	brain	software.	That	is	what

researchers	including	myself	are	attempting	to	do.
The	question,	then,	is	whether	or	not	we	can	find	an	algorithm	that	would
turn	a	computer	into	an	entity	that	is	equivalent	to	a	human	brain.	A	computer,
after	all,	can	run	any	algorithm	that	we	might	define	because	of	its	innate
universality	(subject	only	to	its	capacity).	The	human	brain,	on	the	other	hand,	is
running	a	specific	set	of	algorithms.	Its	methods	are	
clever	in	that	it	allows	for
significant	plasticity	and	the	restructuring	of	its	own	connections	based	on	its

experience,	but	these	functions	can	be	emulated	in	software.
The	universality	of	computation	(the	concept	that	a	general-purpose
computer	can	implement	any	algorithm)—and	the	power	of	this	idea—emerged
at	the	same	time	as	the	first	actual	machines.	There	are	four	key	concepts	that
underlie	the	universality	and	feasibility	of	computation	and	its	applicability	to
our	thinking.	They	are	worth	reviewing	here,	because	the	brain	itself	makes	use
of	them.	The	first	is	the	ability	to	communicate,	remember,	and	compute

information	reliably.	Around	1940,	if	you	used	the	word	“computer,”	people
assumed	you	were	talking	about	an	analog	computer,	in	which	numbers	were
represented	by	different	levels	of	voltage,	and	specialized	components	could
perform	arithmetic	functions	such	as	addition	and	multiplication.	A	big
limitation	of	analog	computers,	however,	was	that	they	were	plagued	by
accuracy	issues.	Numbers	could	only	be	represented	with	an	accuracy	of	about
one	part	in	a	hundred,	and	as	voltage	levels	representing	them	were	processed	by

increasing	numbers	of	arithmetic	operators,	errors	would	accumulate.	If	you
wanted	to	perform	more	than	a	handful	of	computations,	the	results	would
become	so	inaccurate	as	to	be	meaningless.
Anyone	who	can	remember	the	days	of	recording	music	with	analog	tape
machines	will	recall	this	effect.	There	was	noticeable	degradation	on	the	first

copy,	as	it	was	a	little	noisier	than	the	original.	(Remember	that	“noise”
represents	random	inaccuracies.)	A	copy	of	the	copy	was	noisier	still,	and	by	the
tenth	generation	the	copy	was	almost	entirely	noise.	It	was	assumed	that	the
same	problem	would	plague	the	emerging	world	of	digital	computers.	We	can
understand	such	concerns	if	we	consider	the	communication	of	digital
information	through	a	channel.	No	channel	is	perfect	and	each	one	will	have
some	inherent	error	rate.	Suppose	we	have	a	channel	that	has	a	.9	probability	of
correctly	transmitting	each	bit.	If	I	send	a	message	that	is	one	bit	long,	the
probability	of	accurately	transmitting	it	through	that	channel	will	be	.9.	Suppose
I	send	two	bits?	Now	the	accuracy	
is	.9
2
	=	.81.	How	about	if	I	send	one	byte
(eight	bits)?	I	have	less	than	an	even	chance	(.43	to	be	exact)	of	sending	it
correctly.	The	probability	of	accurately	sending	five	bytes	is	about	1	percent.
An	obvious	solution	to	circumvent	this	problem	is	to	make	the	channel

is	.9
2
more	accurate.	Suppose	the	channel	makes	only	one	error	in	a	million	bits.	If	I
send	a	file	consisting	of	a	half	million	bytes	(about	the	size	of	a	modest	program
or	database),	the	probability	of	correctly	transmitting	it	is	less	than	2	percent,
despite	the	very	high	inherent	accuracy	of	the	channel.	Given	that	a	single-bit
error	can	completely	invalidate	a	computer	program	and	other	forms	of	digital
data,	that	is	not	a	satisfactory	situation.	Regardless	of	the	accuracy	of	the

channel,	since	the	likelihood	of	an	error	in	a	transmission	grows	rapidly	with	the
size	of	the	message,	this	would	seem	to	be	an	intractable	barrier.
Analog	computers	approached	this	problem	through	graceful	degradation
(meaning	that	users	only	presented	problems	in	which	they	could	tolerate	small
errors);	however,	if	users	of	analog	computers	limited	themselves	to	a
constrained	set	of	calculations,	the	computers	did	prove	somewhat	useful.
Digital	computers,	on	the	other	hand,	require	continual	communication,	not	just

from	one	computer	to	another,	but	within	the	computer	itself.	There	is
communication	from	its	memory	to	and	from	the	central	processing	unit.	Within
the	central	processing	unit,	there	is	communication	from	one	register	to	another
and	back	and	forth	to	the	arithmetic	unit,	and	so	forth.	Even	within	the	arithmetic
unit,	there	is	communication	from	one	bit	register	to	another.	Communication	is
pervasive	at	every	level.	If	we	consider	that	error	rates	escalate	rapidly	with
increased	communication	and	that	a	single-bit	error	can	destroy	the	integrity	of	a

process,	digital	computation	was	doomed—or	so	it	seemed	at	the	time.
Remarkably,	that	was	the	common	view	until	American	mathematician
Claude	Shannon	(1916–2001)	came	along	and	demonstrated	how	we	can	create
arbitrarily	accurate	communication	using	even	the	most	unreliable
communication	channels.	What	Shannon	stated	in	his	landmark	
paper	“A
Mathematical	Theory	of	Communication,”	published	in	the	
Bell	System

Technical	Journal
	in	July	and	October	1948,	and	in	particular	in	his	noisy
channel-coding	theorem,	was	that	if	you	have	available	a	channel	with	any	error
rate	(except	for	exactly	50	percent	per	bit,	which	would	mean	that	the	channel
was	just	transmitting	pure	noise),	you	are	able	to	transmit	a	message	in	which
the	error	rate	is	as	accurate	as	you	desire.	In	other	words,	the	error	rate	of	the
transmission	can	be	one	bit	out	of	
n
	bits,	where	
n
	can	be	as	large	as	you	define.
So,	for	example,	in	the	extreme,	if	you	have	a	channel	that	correctly	transmits
bits	of	information	only	51	percent	of	the	time	(that	is,	it	transmits	the	correct	bit
just	slightly	more	often	than	the	wrong	bit),	you	can	nonetheless	transmit
messages	such	that	only	one	bit	out	of	a	million	is	incorrect,	or	one	bit	out	of	a
trillion	or	a	trillion	trillion.
How	is	this	possible?	The	answer	is	through	redundancy.	That	may	seem
obvious	now,	but	it	was	not	at	the	time.	As	a	simple	example,	if	I	transmit	each

bit	three	times	and	take	the	majority	vote,	I	will	have	substantially	increased	the
reliability	of	the	result.	If	that	is	not	good	enough,	simply	increase	the
redundancy	until	you	get	the	reliability	you	need.	Simply	repeating	information
is	the	easiest	way	to	achieve	arbitrarily	high	accuracy	rates	from	low-accuracy
channels,	but	it	is	not	the	most	efficient	approach.	Shannon’s	paper,	which
established	the	field	of	information	theory,	presented	optimal	methods	of	error

detection	and	correction	codes	that	can	achieve	
any
	target	accuracy	through	
any
nonrandom	channel.
Older	readers	will	recall	telephone	modems,	which	transmitted	information
through	noisy	analog	phone	lines.	These	lines	featured	audibly	obvious	hisses
and	pops	and	many	other	forms	of	distortion,	but	nonetheless	were	able	to
transmit	digital	data	with	very	high	accuracy	rates,	thanks	to	Shannon’s	noisy
channel	theorem.	The	same	issue	and	the	same	solution	exist	for	digital	memory.

any
any
Ever	wonder	how	CDs,	DVDs,	and	program	disks	continue	to	provide	reliable
results	even	after	the	disk	has	been	dropped	on	the	floor	and	scratched?	Again,
we	can	thank	Shannon.
Computation	consists	of	three	elements:	communication—which,	as	I
mentioned,	is	pervasive	both	within	and	between	computers—memory,	and	logic
gates	(which	perform	the	arithmetic	and	logical	functions).	The	accuracy	of	logic
gates	can	also	be	made	arbitrarily	high	by	similarly	using	error	detection	and

correction	codes.	It	is	due	to	Shannon’s	theorem	and	theory	that	we	can	handle
arbitrarily	large	and	complex	digital	data	and	algorithms	without	the	processes
being	disturbed	or	destroyed	by	errors.	It	is	important	to	point	out	that	the	brain
uses	Shannon’s	principle	as	well,	although	the	evolution	of	the	human	brain
clearly	predates	Shannon’s	own!	Most	of	the	patterns	or	ideas	(and	an	idea	is
also	a	pattern),	as	we	have	seen,	are	stored	in	the	brain	with	a	substantial	amount

of	redundancy.	A	primary	reason	for	the	redundancy	in	the	brain	is	the	inherent
unreliability	of	neural	circuits.
The	second	important	idea	on	which	the	information	age	relies	is	the	one	I
mentioned	earlier:	the	universality	of	computation.	In	1936	Alan	Turing
described	his	“Turing	machine,”	which	was	not	an	actual	machine	but	another
thought	experiment.	His	theoretical	computer	consists	of	an	infinitely	long
memory	tape	with	a	1	or	a	0	in	each	square.	Input	to	the	machine	is	presented	on
this	tape,	which	the	machine	can	read	one	square	at	a	time.	The	machine	also
contains	a	table	of	rules—essentially	a	stored	program—that	consist	of
numbered	states.	Each	rule	specifies	one	action	if	the	square	currently	being	read
is	a	0,	and	a	different	action	if	the	current	square	is	a	1.	Possible	actions	include
writing	a	0	or	1	on	the	tape,	moving	the	tape	one	square	to	the	right	or	left,	or
halting.	Each	state	will	then	specify	the	number	of	the	next	state	that	the
machine	should	be	in.

The	input	to	the	Turing	machine	is	presented	on	the	tape.	The	program	runs,
and	when	the	machine	halts,	it	has	completed	its	algorithm,	and	the	output	of	the
process	is	left	on	the	tape.	Note	that	even	though	the	tape	is	theoretically	infinite
in	length,	any	actual	program	that	does	not	get	into	an	infinite	loop	will	use	only
a	finite	portion	of	the	tape,	so	if	we	limit	ourselves	to	a	finite	tape,	the	machine
will	still	solve	a	useful	set	of	problems.

If	the	Turing	machine	sounds	simple,	it	is	because	that	was	its	inventor’s
objective.	Turing	wanted	his	machine	to	be	as	simple	as	possible	(but	
no	simpler,
to	paraphrase	Einstein).	Turing	and	Alonzo	Church	(1903–1995),	his	former
professor,	went	on	to	develop	the	Church-Turing	thesis,	which	states	that	if	a
problem	that	can	be	presented	to	a	Turing	machine	is	not	solvable	by	it,	it	is	also
not	solvable	by	
any
	machine,	following	natural	law.	Even	though	the	Turing

any
machine	has	only	a	handful	of	commands	and	processes	only	one	bit	at	a	time,	it
can	compute	anything	that	any	computer	can	compute.	Another	way	to	say	this
is	that	any	machine	that	is	“Turing	complete”	(that	is,	that	has	equivalent
capabilities	to	a	Turing	machine)	can	compute	any	algorithm	(any	procedure	that
we	can	define).

A	block	diagram	of	a	Turing	machine	with	a	head	that	reads	and	writes
the	tape	and	an	internal	program	consisting	of	state	transitions.
“Strong”	interpretations	of	the	Church-Turing	thesis	propose	an	essential
equivalence	between	what	a	human	can	think	or	know	and	what	is	computable
by	a	machine.	The	basic	idea	is	that	the	human	brain	is	likewise	subject	to
natural	law,	and	thus	its	information-processing	ability	cannot	exceed	that	of	a
machine	(and	therefore	of	a	Turing	machine).
We	can	properly	credit	Turing	with	establishing	the	theoretical	foundation
of	computation	with	his	1936	paper,	but	it	is	important	to	note	that	he	was	deeply
influenced	by	a	lecture	that	Hungarian	American	mathematician	John	von
Neumann	(1903–1957)	gave	in	Cambridge	in	1935	on	his	stored	program
concept,	a	concept	enshrined	in	the	Turing	machine.
1
	In	
turn,	von	Neumann	was
influenced	by	Turing’s	1936	paper,	which	elegantly	laid	out	the	principles	of
computation,	and	made	it	required	reading	for	his	colleagues	in	the	late	1930s

1
	In	
and	early	1940s.
2
In	the	same	paper	Turing	reports	another	unexpected	discovery:	that	of
unsolvable	problems.	These	are	problems	that	are	well	defined	with	unique
answers	that	can	be	shown	to	exist,	but	that	we	can	also	prove	can	never	be
computed	by	any	Turing	machine—that	is	to	say,	by	
any
	machine,	a	reversal	of
what	had	been	a	nineteenth-century	dogma	that	problems	that	could	be	defined
would	ultimately	be	solved.	Turing	showed	that	there	are	as	many	unsolvable
problems	as	solvable	ones.	Austrian	American	mathematician	and	philosopher

any
Kurt	Gödel	reached	a	similar	conclusion	in	his	1931	“incompleteness	theorem.”
We	are	thus	left	with	the	perplexing	situation	of	being	able	to	define	a	problem,
to	prove	that	a	unique	answer	exists,	and	yet	know	that	the	answer	can	never	be
found.
Turing	had	shown	that	at	its	essence,	computation	is	based	on	a	very	simple
mechanism.	Because	the	Turing	machine	(and	therefore	any	computer)	is

capable	of	basing	its	future	course	of	action	on	results	it	has	already	computed,	it
is	capable	of	making	decisions	and	modeling	arbitrarily	complex	hierarchies	of
In	1939	Turing	designed	an	electronic	calculator	called	Bombe	that	helped
decode	messages	that	had	been	encrypted	by	the	Nazi	Enigma	coding	machine.
By	1943,	an	engineering	team	influenced	by	Turing	completed	what	is	arguably
the	first	computer,	the	Colossus,	that	enabled	the	Allies	to	continue	decoding
messages	from	more	sophisticated	versions	of	Enigma.	The	Bombe	and
Colossus	were	designed	for	a	single	task	and	could	not	be	reprogrammed	for	a
different	one.	But	they	performed	this	task	brilliantly	and	are	credited	with
having	enabled	the	Allies	to	overcome	the	three-to-one	advantage	that	the
German	Luftwaffe	enjoyed	over	the	British	Royal	Air	Force	and	win	the	crucial
Battle	of	Britain,	as	well	as	to	continue	anticipating	Nazi	tactics	throughout	the
war.
It	was	on	these	foundations	that	John	von	Neumann	created	the	architecture

war.
of	the	modern	computer,	which	represents	our	third	major	idea.	
Called	the	von
Neumann	machine,	it	has	remained	the	core	structure	of	essentially	every
computer	for	the	past	sixty-seven	years,	from	the	microcontroller	in	your
washing	machine	to	the	largest	supercomputers.	In	a	paper	dated	June	30,	1945,
and	titled	“First	Draft	of	a	Report	on	the	EDVAC,”	von	Neumann	presented	the
ideas	that	have	dominated	computation	ever	since.
3
	The	von	Neumann	model
includes	a	central	processing	unit,	where	arithmetical	and	logical	operations	are

3
carried	out;	a	memory	unit,	where	the	program	and	data	are	stored;	mass	storage;
a	program	counter;	and	input/output	channels.	Although	this	paper	was	intended
as	an	internal	project	document,	it	has	become	the	bible	for	computer	designers.
You	never	know	when	a	seemingly	routine	internal	memo	will	end	up
revolutionizing	the	world.
The	Turing	machine	was	not	designed	to	be	practical.	Turing’s	theorems
were	concerned	not	with	the	efficiency	of	solving	problems	but	rather	in

examining	the	range	of	problems	that	could	in	theory	be	solved	by	computation.
Von	Neumann’s	goal,	on	the	other	hand,	was	to	create	a	feasible	concept	of	a
computational	machine.	His	model	replaces	Turing’s	one-bit	computations	with
multiple-bit	words	(generally	some	multiple	of	eight	bits).	Turing’s	memory	tape
is	sequential,	so	Turing	machine	programs	spend	an	inordinate	amount	of	time
moving	the	tape	back	and	forth	to	store	and	retrieve	intermediate	results.	In

contrast,	von	Neumann’s	memory	is	random	access,	so	that	any	data	item	can	be
immediately	retrieved.
One	of	von	Neumann’s	key	ideas	is	the	stored	program,	which	he	had
introduced	a	decade	earlier:	placing	the	program	in	the	same	type	of	random

access	memory	as	the	data	(and	often	in	the	same	block	of	memory).	This	allows
the	computer	to	be	reprogrammed	for	different	tasks	as	well	as	for	self-
modifying	code	(if	the	program	store	is	writable),	which	enables	a	powerful
form	of	recursion.	Up	until	that	time,	virtually	all	computers,	including	the
Colossus,	were	built	for	a	specific	task.	The	stored	program	makes	it	possible	for
a	computer	to	be	truly	universal,	thereby	fulfilling	Turing’s	vision	of	the
universality	of	computation.
Another	key	aspect	of	the	von	Neumann	machine	is	that	each	instruction
includes	an	operation	code	specifying	the	arithmetic	or	logical	operation	to	be
performed	and	the	address	of	an	operand	from	memory.
Von	Neumann’s	concept	of	how	a	computer	should	be	architected	was
introduced	with	his	publication	of	the	design	of	the	EDVAC,	a	project	he
conducted	with	collaborators	J.	Presper	Eckert	and	John	Mauchly.	The	EDVAC
itself	did	not	actually	run	until	1951,	by	which	time	there	were	other	stored-

program	computers,	such	as	the	Manchester	Small-Scale	Experimental	Machine,
ENIAC,	EDSAC,	and	BINAC,	all	of	which	had	been	deeply	influenced	by	von
Neumann’s	paper	and	involved	Eckert	and	Mauchly	as	designers.	Von	Neumann
was	a	direct	contributor	to	the	design	of	a	number	of	these	machines,	including	a
later	version	of	ENIAC,	which	supported	a	stored	program.
There	were	a	few	precursors	to	von	Neumann’s	architecture,	although	with
one	surprising	exception,	none	are	true	von	Neumann	machines.	In	1944

Howard	Aiken	introduced	the	Mark	I,	which	had	an	element	of	programmability
but	did	not	use	a	stored	program.	It	read	instructions	from	a	punched	paper	tape
and	then	executed	each	command	immediately.	It	also	lacked	a	conditional
branch	instruction.
In	1941	German	scientist	Konrad	Zuse	(1910–1995)	created	the	Z-3
computer.	It	also	read	its	program	from	a	tape	(in	this	case,	coded	on	film)	and
also	had	no	conditional	branch	instruction.	Interestingly,	Zuse	had	support	from

the	German	Aircraft	Research	Institute,	which	used	the	device	to	study	wing
flutter,	but	his	proposal	to	the	Nazi	government	for	funding	to	replace	his	relays
with	vacuum	tubes	was	turned	down.	The	Nazis	deemed	computation	as	“not
war	important.”	That	perspective	goes	a	long	way,	in	my	view,	toward
explaining	the	outcome	of	the	war.
There	is	actually	one	genuine	forerunner	to	von	Neumann’s	concept,	and	it
comes	from	a	full	century	earlier!	English	mathematician	and	inventor	Charles

Babbage’s	(1791–1871)	Analytical	Engine,	which	he	first	described	in	1837,	did
incorporate	von	Neumann’s	ideas	and	featured	a	stored	program	via	punched
cards	borrowed	from	the	Jacquard	loom.
4
	Its	random	access	memory	included
1,000	words	of	50	decimal	digits	each	
(the	equivalent	of	about	21	kilobytes).

Each	instruction	included	an	op	code	and	an	operand	number,	just	like	modern
machine	languages.	It	did	include	conditional	branching	and	looping,	so	it	was	a
true	von	Neumann	machine.	It	was	based	entirely	on	mechanical	gears	and	it
appears	that	the	Analytical	Engine	was	beyond	Babbage’s	design	and
organizational	skills.	He	built	parts	of	it	but	it	never	ran.	It	is	unclear	whether	the
twentieth-century	pioneers	of	the	computer,	including	von	Neumann,	were	aware
of	Babbage’s	work.
Babbage’s	computer	did	result	in	the	creation	of	the	field	of	software
programming.	English	writer	Ada	Byron	(1815–1852),	Countess	of	Lovelace
and	the	only	legitimate	child	of	the	poet	Lord	Byron,	was	the	world’s	first
computer	programmer.	She	wrote	programs	for	the	Analytical	Engine,	which	she
needed	to	debug	in	her	own	mind	(since	the	computer	never	worked),	a	practice
well	known	to	software	engineers	today	as	“table	checking.”	She	translated	an
article	by	the	Italian	mathematician	Luigi	Menabrea	on	the	Analytical	Engine

and	added	extensive	notes	of	her	own,	writing	that	“the	Analytical	Engine
weaves	algebraic	patterns,	just	as	the	Jacquard	loom	weaves	flowers	and	leaves.”
She	went	on	to	provide	perhaps	the	first	speculations	on	the	feasibility	of
artificial	intelligence,	but	concluded	that	the	Analytical	Engine	has	“no
pretensions	whatever	to	originate	anything.”
Babbage’s	conception	is	quite	miraculous	when	you	consider	the	era	in
which	he	lived	and	worked.	However,	by	the	mid-twentieth	century,	his	ideas
had	been	lost	in	the	mists	of	time	(although	they	were	subsequently

rediscovered).	It	was	von	Neumann	who	conceptualized	and	articulated	the	key
principles	of	the	computer	as	we	know	it	today,	and	the	world	recognizes	this	by
continuing	to	refer	to	the	von	Neumann	machine	as	the	principal	model	of
computation.	Keep	in	mind,	though,	that	the	von	Neumann	machine	continually
communicates	data	between	its	various	units	and	within	these	units,	so	it	could
not	be	built	without	Shannon’s	theorems	and	the	methods	he	devised	for
transmitting	and	storing	reliable	digital	information.

That	brings	us	to	the	fourth	important	idea,	which	is	to	go	beyond	Ada
Byron’s	conclusion	that	a	computer	could	not	think	creatively	and	find	the	key
algorithms	employed	by	the	brain	and	then	use	these	to	turn	a	computer	into	a
brain.	Alan	Turing	introduced	this	goal	in	his	1950	paper	“Computing
Machinery	and	Intelligence,”	which	includes	his	now-famous	Turing	test	for
ascertaining	whether	or	not	an	AI	has	achieved	a	human	level	of	intelligence.
In	1956	von	Neumann	began	preparing	a	series	of	lectures	intended	for	the

prestigious	Silliman	lecture	series	at	Yale	University.	Due	to	the	ravages	of
cancer,	he	never	delivered	these	talks	nor	did	he	complete	the	manuscript	from
which	they	were	to	be	given.	This	unfinished	document	nonetheless	remains	a

brilliant	and	prophetic	foreshadowing	of	what	I	regard	as	humanity’s	most
daunting	and	important	project.	It	was	published	posthumously	as	
The	Computer
and	the	Brain
	in	1958.	It	is	fitting	that	the	final	work	of	one	of	the	most	brilliant
mathematicians	of	the	last	century	and	one	of	the	pioneers	of	the	computer	age
was	an	examination	of	intelligence	itself.	This	project	was	the	earliest	serious
inquiry	into	the	human	brain	from	the	perspective	of	a	mathematician	and
computer	scientist.	Prior	to	von	Neumann,	the	fields	of	computer	science	and
neuroscience	were	two	islands	with	no	bridge	between	them.
Von	Neumann	starts	his	discussion	by	articulating	the	similarities	and
differences	between	the	computer	and	the	human	brain.	Given	when	he	wrote
this	manuscript,	it	is	remarkably	accurate.	He	noted	that	the	output	of	neurons
was	digital—an	axon	either	fired	or	it	didn’t.	This	was	far	from	obvious	at	the
time,	in	that	the	output	could	have	been	an	analog	signal.	The	processing	in	the

dendrites	leading	into	a	neuron	and	in	the	soma	neuron	cell	body,	however,	was
analog,	and	he	described	its	calculations	as	a	weighted	sum	of	inputs	with	a
threshold.	This	model	of	how	neurons	work	led	to	the	field	of	connectionism,
which	built	systems	based	on	this	neuron	model	in	both	hardware	and	software.
(As	I	described	in	the	
,	the	first	such	connectionist	system	was
created	by	Frank	Rosenblatt	as	a	software	program	on	an	IBM	704	computer	at
Cornell	in	1957,	immediately	after	von	Neumann’s	draft	lectures	became

available.)	We	now	have	more	sophisticated	models	of	how	neurons	combine
inputs,	but	the	essential	idea	of	analog	processing	of	dendrite	inputs	using
neurotransmitter	concentrations	has	remained	valid.
Von	Neumann	applied	the	concept	of	the	universality	of	computation	to
conclude	that	even	though	the	architecture	and	building	blocks	appear	to	be
radically	different	between	brain	and	computer,	we	can	nonetheless	conclude
that	a	von	Neumann	machine	can	simulate	the	processing	in	a	brain.	The

converse	does	not	hold,	however,	because	the	brain	is	not	a	von	Neumann
machine	and	does	not	have	a	stored	program	as	such	(albeit	we	can	simulate	a
very	simple	Turing	machine	in	our	heads).	Its	algorithm	or	methods	are	implicit
in	its	structure.	Von	Neumann	correctly	concludes	that	neurons	can	learn	patterns
from	their	inputs,	which	we	have	now	established	are	coded	in	part	in	dendrite
strengths.	What	was	not	known	in	von	Neumann’s	time	is	that	learning	also

takes	place	through	the	creation	and	destruction	of	connections	between	neurons.
Von	Neumann	presciently	notes	that	the	speed	of	neural	processing	is
extremely	slow,	on	the	order	of	a	hundred	calculations	per	second,	but	that	the
brain	compensates	for	this	through	massive	parallel	processing—another
unobvious	and	key	insight.	Von	Neumann	argued	that	each	one	of	the	brain’s
10
10
	neurons	(a	tally	that	itself	was	reasonably	accurate;	estimates	today	are

between	10
10
	and	10
11
)	was	processing	at	the	same	time.	In	fact,	each	of	the
connections	(with	an	average	of	about	10
3
	to	10
4
	connections	per	neuron)	is
computing	simultaneously.
Von	Neumann’s	estimates	and	his	descriptions	of	neural	processing	are
remarkable,	given	the	primitive	state	of	neuroscience	at	the	time.	One	aspect	of
his	work	that	I	do	disagree	with,	however,	is	his	assessment	of	the	brain’s
memory	capacity.	He	assumes	that	the	brain	remembers	every	input	for	its	entire
life.	Von	Neumann	assumes	an	average	life	span	of	60	years,	or	about	2	×	10
9
seconds.	With	about	14	inputs	to	each	neuron	per	second	(which	is	actually	low
by	at	least	three	orders	of	magnitude)	and	with	10
10
	neurons,	he	arrives	at	an
estimate	of	about	10
20
	bits	for	the	brain’s	
memory	capacity.	The	reality,	as	I	have
noted	earlier,	is	that	we	remember	only	a	very	small	fraction	of	our	thoughts	and
experiences,	and	even	these	memories	are	not	stored	as	bit	patterns	at	a	low	level

9
10
20
(such	as	a	video	image),	but	rather	as	sequences	of	higher-level	patterns.
As	von	Neumann	describes	each	mechanism	in	the	brain,	he	shows	how	a
modern	computer	could	accomplish	the	same	thing,	despite	their	apparent
differences.	The	brain’s	analog	mechanisms	can	be	simulated	through	digital
ones	because	digital	computation	can	emulate	analog	values	to	any	desired
degree	of	precision	(and	the	precision	of	analog	information	in	the	brain	is	quite
low).	The	brain’s	massive	parallelism	can	be	simulated	as	well,	given	the

significant	speed	advantage	of	computers	in	serial	computation	(an	advantage
that	has	vastly	expanded	over	time).	In	addition,	we	can	also	use	parallel
processing	in	computers	by	using	parallel	von	Neumann	machines—which	is
exactly	how	supercomputers	work	today.
Von	Neumann	concludes	that	the	brain’s	methods	cannot	involve	lengthy
sequential	algorithms,	when	one	considers	how	quickly	humans	are	able	to	make
decisions	combined	with	the	very	slow	computational	speed	of	neurons.	When	a
third	baseman	fields	a	ball	and	decides	to	throw	to	first	rather	than	to	second

base,	he	makes	this	decision	in	a	fraction	of	a	second,	which	is	only	enough	time
for	each	neuron	to	go	through	a	handful	of	cycles.	Von	Neumann	concludes
correctly	that	the	brain’s	remarkable	powers	come	from	all	its	100	billion
neurons	being	able	to	process	information	simultaneously.	As	I	have	noted,	the
visual	cortex	makes	sophisticated	visual	judgments	in	only	three	or	four	neural
cycles.
There	is	considerable	plasticity	in	the	brain,	which	enables	us	to	learn.	But

cycles.
there	is	far	greater	plasticity	in	a	computer,	which	can	completely	restructure	its
methods	by	changing	its	software.	Thus,	in	that	respect,	a	computer	will	be	able
to	emulate	the	brain,	but	the	converse	is	not	the	case.

When	von	Neumann	compared	the	capacity	of	the	brain’s	massively
parallel	organization	to	the	(few)	computers	of	his	time,	it	was	clear	that	the
brain	had	far	greater	memory	and	speed.	By	now	the	first	supercomputer	
to
achieve	specifications	matching	some	of	the	more	conservative	estimates	of	the
speed	required	to	functionally	simulate	the	human	brain	(about	10
16
per	second)	has	been	built.
5
	(I	estimate	that	this	level	of	computation	will	cost
$1,000	by	the	early	2020s.)	With	regard	to	memory	we	are	even	closer.	Even
though	it	was	remarkably	early	in	the	history	of	the	computer	when	his
manuscript	was	written,	von	Neumann	nonetheless	had	confidence	that	both	the
hardware	and	software	of	human	intelligence	would	ultimately	fall	into	place,
which	was	his	motivation	for	having	prepared	these	lectures.
Von	Neumann	was	deeply	aware	of	the	increasing	pace	of	progress	and	its
profound	implications	for	humanity’s	future.	A	year	after	his	death	in	1957,

fellow	mathematician	Stan	Ulam	quoted	him	as	having	said	in	the	early	1950s
that	“the	ever	accelerating	progress	of	technology	and	changes	in	the	mode	of
human	life	give	the	appearance	of	approaching	some	essential	singularity	in	the
history	of	the	race	beyond	which	human	affairs,	as	we	know	them,	could	not
continue.”	This	is	the	first	known	use	of	the	word	“singularity”	in	the	context	of
human	technological	history.
Von	Neumann’s	fundamental	insight	was	that	there	is	an	essential

equivalence	between	a	computer	and	the	brain.	Note	that	the	emotional
intelligence	of	a	biological	human	is	part	of	its	intelligence.	If	von	Neumann’s
insight	is	correct,	and	if	one	accepts	my	own	leap	of	faith	that	a	nonbiological
entity	that	convincingly	re-creates	the	intelligence	(emotional	and	otherwise)	of
a	biological	human	is	conscious	(see	the	
),	then	one	would	have	to
conclude	that	there	is	an	essential	equivalence	between	a	computer—
with	the
right	software

with	the
—and	a	(conscious)	mind.	So	is	von	Neumann	correct?
Most	computers	today	are	entirely	digital,	whereas	the	human	brain
combines	digital	and	analog	methods.	But	analog	methods	are	easily	and
routinely	re-created	by	digital	ones	to	any	desired	level	of	accuracy.	American
computer	scientist	Carver	Mead	(born	in	1934)	has	shown	that	we	can	directly
emulate	the	brain’s	analog	methods	in	silicon,	which	he	has	
demonstrated	with
what	he	calls	“neuromorphic”	chips.
6
	Mead	has	demonstrated	how	this	approach

6
can	be	thousands	of	times	more	efficient	than	digitally	emulating	analog
methods.	As	we	codify	the	massively	repeated	neocortical	algorithm,	it	will
make	sense	to	use	Mead’s	approach.	The	IBM	Cognitive	Computing	Group,	led
by	Dharmendra	Modha,	has	introduced	chips	that	emulate	neurons	and	their
connections,	including	the	ability	to	form	new	connections.
7
	Called

“SyNAPSE,”	one	of	the	chips	provides	a	direct	simulation	of	256	neurons	with
about	a	quarter	million	synaptic	connections.	The	goal	of	the	project	is	to	create
a	simulated	neocortex	with	10	billion	neurons	and	100	trillion	connections—
close	to	a	human	brain—that	uses	only	one	kilowatt	of	power.
As	von	Neumann	described	over	a	half	century	ago,	the	brain	is	extremely
slow	but	massively	parallel.	Today’s	digital	circuits	are	at	least	10	million	times
faster	than	the	brain’s	electrochemical	switches.	Conversely,	all	300	million	of
the	brain’s	neocortical	pattern	recognizers	process	simultaneously,	and	all
quadrillion	of	its	interneuronal	connections	are	potentially	computing	at	the
same	time.	The	key	issue	for	providing	the	requisite	hardware	to	successfully
model	a	human	brain,	though,	is	the	overall	memory	and	computational
throughput	required.	We	do	not	need	to	directly	copy	the	brain’s	architecture,
which	would	be	a	very	inefficient	and	inflexible	approach.

Let’s	estimate	what	those	hardware	requirements	are.	Many	projects	have
attempted	to	emulate	the	type	of	hierarchical	learning	and	pattern	recognition
that	takes	place	in	the	neocortical	hierarchy,	including	my	own	work	with
hierarchical	hidden	Markov	models.	A	conservative	estimate	from	my	own
experience	is	that	emulating	one	cycle	in	a	single	pattern	recognizer	in	the
biological	brain’s	neocortex	would	require	about	3,000	calculations.	Most
simulations	run	at	a	fraction	of	this	estimate.	With	the	brain	running	at	about	10
2
(100)	cycles	per	second,	that	comes	to	3	×	10
5

2
5
	(300,000)	calculations	per	second
per	pattern	recognizer.	Using	my	estimate	of	3	×	10
8
	(300	million)	pattern
recognizers,	we	get	about	10
14
	(100	
trillion)	calculations	per	second,	a	figure
that	is	consistent	with	my	estimate	in	
.	In	that	book	I
projected	that	to	functionally	simulate	the	brain	would	require	between	10
14
	and
10
16
	calculations	per	second	(cps)	and	used	10
16
	cps	to	be	conservative.	AI
expert	Hans	Moravec’s	estimate,	based	on	extrapolating	the	computational

(300,000)	calculations	per	second
8
14
	(100	
14
	and
10
16
16
requirement	of	the	early	(initial)	visual	processing	across	the	entire	brain,	is	10
14
cps,	which	matches	my	own	assessment	here.
Routine	desktop	machines	can	reach	10
10
	cps,	although	this	level	of
performance	can	be	significantly	amplified	by	using	cloud	resources.	The	fastest
supercomputer,	Japan’s	K	Computer,	has	already	reached	10
16
	cps.
8
	Given	that
the	algorithm	of	the	neocortex	is	massively	repeated,	the	approach	of	using
neuromorphic	chips	such	as	the	IBM	SyNAPSE	chips	mentioned	above	is	also

14
10
16
	cps.
8
	Given	that
promising.
In	terms	of	memory	requirement,	we	need	about	30	bits	(about	four	bytes)
for	one	connection	to	address	one	of	300	million	other	pattern	recognizers.	If	we
estimate	an	average	of	eight	inputs	to	each	pattern	recognizer,	that	comes	to	32

bytes	per	recognizer.	If	we	add	a	one-byte	weight	for	each	input,	that	brings	us	to
40	bytes.	Add	another	32	bytes	for	downward	connections,	and	we	are	at	72
bytes.	Note	that	the	branching-up-and-down	figure	will	often	be	much	higher
than	eight,	though	these	very	large	branching	trees	are	shared	by	many
recognizers.	For	example,	there	may	be	hundreds	of	recognizers	involved	in
recognizing	the	letter	“p.”	These	will	feed	up	into	thousands	of	such	recognizers
at	this	next	higher	level	that	deal	with	words	and	phrases	that	include	“p.”
However,	each	“p”	recognizer	does	not	repeat	the	tree	of	connections	that	feeds
up	to	all	of	the	words	and	phrases	that	include	“p”—they	all	share	one	such	tree
of	connections.	The	same	is	true	of	downward	connections:	A	recognizer	that	is
responsible	for	the	word	“APPLE”	will	tell	all	of	the	thousands	of	“E”
recognizers	at	a	level	below	it	that	an	“E”	is	expected	if	it	has	already	seen	“A,”
“P,”	“P,”	and	“L.”	That	tree	of	connections	is	not	repeated	for	each	word	or

phrase	recognizer	that	wants	to	inform	the	next	lower	level	that	an	“E”	is
expected.	Again,	they	are	shared.	For	this	reason,	an	overall	estimate	of	eight	up
and	eight	down	on	average	per	pattern	recognizer	is	reasonable.	Even	if	we
increase	this	particular	estimate,	it	does	not	significantly	change	the	order	of
magnitude	of	the	resulting	estimate.
With	3	×	10
8
	(300	million)	pattern	recognizers	at	72	bytes	each,	we	get	an
overall	memory	requirement	of	about	2	×	10
10
	(20	billion)	bytes.	That	is	actually

With	3	×	10
8
10
a	quite	modest	number	that	routine	computers	today	can	exceed.
These	estimates	are	intended	only	to	provide	rough	estimates	of	the	order	of
magnitude	required.	Given	that	digital	circuits	are	inherently	about	10	million
times	faster	than	the	biological	neocortical	circuits,	we	do	not	need	to	match	the
human	brain	for	parallelism—modest	parallel	processing	(compared	with	the
trillions-fold	parallelism	of	the	human	brain)	will	be	sufficient.	We	can	see	that

10
the	necessary	computational	requirements	are	coming	within	reach.	The	brain’s
rewiring	of	itself—dendrites	are	continually	creating	new	synapses—can	also	be
emulated	in	software	using	links,	a	far	more	flexible	system	than	the	brain’s
method	of	plasticity,	which	as	we	have	seen	is	impressive	but	limited.
The	redundancy	used	by	the	brain	to	achieve	robust	invariant	results	can
certainly	be	replicated	in	software	emulations.	The	mathematics	of	optimizing

these	types	of	self-organizing	hierarchical	learning	systems	is	well	understood.
The	organization	of	the	brain	is	far	from	optimal.	Of	course	it	didn’t	need	to	be
—it	only	needed	to	be	good	enough	to	achieve	the	threshold	of	being	able	to
create	tools	that	would	compensate	for	its	own	limitations.
Another	restriction	of	the	human	neocortex	is	that	there	is	no	process	that
eliminates	or	even	reviews	contradictory	ideas,	which	accounts	for	why	human

thinking	is	often	massively	inconsistent.	We	have	a	weak	mechanism	to	address
this	called	critical	thinking,	but	this	skill	is	not	practiced	nearly	as	often	as	it
should	be.	In	a	software-based	neocortex,	we	can	build	in	a	process	that	reveals
inconsistencies	for	further	review.
It	is	important	to	note	that	the	design	of	an	entire	brain	region	is	
simpler
than	the	design	of	a	single	neuron.	As	discussed	earlier,	models	often	get	simpler
at	a	higher	level—consider	an	analogy	with	a	computer.	We	do	need	to
understand	the	detailed	physics	of	semiconductors	to	model	a	transistor,	and	the
equations	underlying	a	single	real	transistor	are	complex.	A	digital	circuit	that
multiples	two	numbers	requires	hundreds	of	them.	Yet	we	can	model	this
multiplication	circuit	very	simply	with	one	or	two	formulas.	An	entire	computer
with	billions	of	transistors	can	be	modeled	through	its	instruction	set	and	register
description,	which	can	be	described	on	a	handful	of	written	pages	of	text	and

formulas.	The	software	programs	for	an	operating	system,	language	compilers,
and	assemblers	are	reasonably	complex,	but	modeling	a	particular	program—for
example,	a	speech	recognition	program	based	on	hierarchical	hidden	Markov
modeling—may	likewise	be	described	in	only	a	few	pages	of	equations.
Nowhere	in	such	a	description	would	be	found	the	details	of	semiconductor
physics	or	even	of	computer	architecture.
A	similar	observation	holds	true	for	the	brain.	A	particular	neocortical

pattern	recognizer	that	detects	a	particular	invariant	visual	feature	(such	as	a
face)	or	that	performs	a	bandpass	filtering	(restricting	input	to	a	specific
frequency	range)	on	sound	or	that	evaluates	the	temporal	proximity	of	two
events	can	be	described	with	far	fewer	specific	details	than	the	actual	physics
and	chemical	relations	controlling	the	neurotransmitters,	ion	channels,	and	other
synaptic	and	dendritic	variables	involved	in	the	neural	processes.	Although	all	of

this	complexity	needs	to	be	carefully	considered	before	advancing	to	the	next
higher	conceptual	level,	much	of	it	can	be	simplified	as	the	operating	principles
of	the	brain	are	revealed.

ON	THE	MIND
	
Minds	are	simply	what	brains	do.
	
When	intelligent	machines	are	constructed,	we	should	not	be	surprised	to
find	them	as	confused	and	as	stubborn	as	men	in	their	convictions	about
mind-matter,	consciousness,	free	will,	and	the	like.

Who	Is	Conscious?
The	real	history	of	consciousness	starts	with	one’s	first	lie.
—Joseph	Brodsky
	
Suffering	is	the	sole	origin	of	consciousness.
—Fyodor	Dostoevsky,	
Notes	from	Underground
	
There	is	a	kind	of	plant	that	eats	organic	food	with	its	flowers:	when	a	fly
settles	upon	the	blossom,	the	petals	close	upon	it	and	hold	it	fast	till	the
plant	has	absorbed	the	insect	into	its	system;	but	they	will	close	on	nothing
but	what	is	good	to	eat;	of	a	drop	of	rain	or	a	piece	of	
stick	they	will	take	no
notice.	Curious!	that	so	unconscious	a	thing	should	have	such	a	keen	eye	to
its	own	interest.	If	this	is	unconsciousness,	where	is	the	use	of
consciousness?
	
W
e	have	been	examining	the	brain	as	an	entity	that	is	capable	of	certain	levels
of	accomplishment.	But	that	perspective	essentially	leaves	our	
selves
	out	of	the
picture.	We	appear	to	live	in	our	brains.	We	have	subjective	lives.	How	does	the
objective	view	of	the	brain	that	we	have	discussed	up	until	now	relate	to	our	own

W
selves
	out	of	the
feelings,	to	our	sense	of	being	the	person	having	the	experiences?
British	philosopher	Colin	McGinn	(born	in	1950)	writes	that	discussing
“consciousness	can	reduce	even	the	most	fastidious	thinker	to	blabbering
incoherence.”	The	reason	for	this	is	that	people	often	have	unexamined	and
inconsistent	views	on	exactly	what	the	term	means.
Many	observers	consider	consciousness	to	be	a	form	of	performance—for
example,	the	capacity	for	self-reflection,	that	is,	the	ability	to	understand	one’s

own	thoughts	and	to	explain	them.	I	would	describe	that	as	the	ability	to	think
about	one’s	own	thinking.	Presumably,	we	could	come	up	with	a	way	of
evaluating	this	ability	and	then	use	this	test	to	separate	conscious	things	from
unconscious	things.

However,	we	quickly	get	into	trouble	in	trying	to	implement	this	approach.
Is	a	baby	conscious?	A	dog?	They’re	not	very	good	at	describing	their	own
thinking	process.	There	are	people	who	believe	that	babies	and	dogs	are	not
conscious	beings	precisely	because	they	cannot	explain	themselves.	How	about
the	computer	known	as	Watson?	It	can	be	put	into	a	mode	where	it	actually	does
explain	how	it	came	up	with	a	given	answer.	Because	it	contains	a	model	of	its
own	thinking,	is	Watson	therefore	conscious	whereas	the	baby	and	the	dog	are
not?
Before	we	proceed	to	parse	this	question	further,	it	is	important	to	reflect	on
the	most	significant	distinction	relating	to	it:	What	is	it	that	we	can	
ascertain
from	science,	versus	what	remains	truly	a	matter	of	philosophy?	One	view	is	that
philosophy	is	a	kind	of	halfway	house	for	questions	that	have	not	yet	yielded	to
the	scientific	method.	According	to	this	perspective,	once	science	advances
sufficiently	to	resolve	a	particular	set	of	questions,	philosophers	can	then	move

not?
ascertain
on	to	other	concerns,	until	such	time	that	science	resolves	them	also.	This	view
is	endemic	where	the	issue	of	consciousness	is	concerned,	and	specifically	the
question	“What	and	who	is	conscious?”
Consider	these	statements	by	philosopher	John	Searle:	“We	know	that
brains	cause	consciousness	with	specific	biological	mechanisms….	The	essential
thing	is	to	recognize	that	consciousness	is	a	biological	process	like	digestion,
lactation,	photosynthesis,	or	mitosis….	The	brain	is	a	machine,	a	biological

machine	to	be	sure,	but	a	machine	all	the	same.	So	the	first	step	is	to	figure	out
how	the	brain	does	it	and	then	build	an	artificial	machine	that	has	an	equally
effective	mechanism	for	causing	consciousness.”
1
	People	are	often	surprised	to
see	these	quotations	because	they	assume	that	Searle	is	devoted	to	protecting	the
mystery	of	consciousness	against	reductionists	like	Ray	Kurzweil.
The	Australian	philosopher	David	Chalmers	(born	in	1966)	has	coined	the

1
term	“the	hard	problem	of	consciousness”	to	describe	the	difficulty	of	pinning
down	this	essentially	indescribable	concept.	Sometimes	a	brief	phrase
encapsulates	an	entire	school	of	thought	so	well	that	it	becomes	emblematic	(for
example,	Hannah	Arendt’s	“the	banality	of	evil”).	Chalmers’s	famous
formulation	accomplishes	this	very	well.
When	discussing	consciousness,	it	becomes	very	easy	to	slip	into
considering	the	observable	and	measurable	attributes	that	we	associate	with
being	conscious,	but	this	approach	misses	the	very	essence	of	the	idea.	I	just

mentioned	the	concept	of	metacognition—the	idea	of	thinking	about	one’s	own
thinking—as	one	such	correlate	of	consciousness.	Other	observers	conflate
emotional	intelligence	or	moral	intelligence	with	consciousness.	But,	again,	our
ability	to	express	a	loving	sentiment,	to	get	the	joke,	or	to	be	sexy	are	simply

types	of	performances—impressive	and	intelligent	perhaps,	but	skills	that	can
nonetheless	be	observed	and	measured	(even	if	
we	argue	about	how	to	assess
them).	Figuring	out	how	the	brain	accomplishes	these	sorts	of	tasks	and	what	is
going	on	in	the	brain	when	we	do	them	constitutes	Chalmers’s	“easy”	question
of	consciousness.	Of	course,	the	“easy”	problem	is	anything	but	and	represents
perhaps	the	most	difficult	and	important	scientific	quest	of	our	era.	Chalmers’s
“hard”	question,	meanwhile,	is	so	hard	that	it	is	essentially	ineffable.
In	support	of	this	distinction,	Chalmers	introduces	a	thought	experiment
involving	what	he	calls	zombies.	A	zombie	is	an	entity	that	acts	just	like	a	person
but	simply	does	not	have	subjective	experience—that	is,	a	zombie	is	not
conscious.	Chalmers	argues	that	since	we	can	conceive	of	zombies,	they	are	at
least	logically	possible.	If	you	were	at	a	cocktail	party	and	there	were	both
“normal”	humans	and	zombies,	how	would	you	tell	the	difference?	Perhaps	this

sounds	like	a	cocktail	party	you	have	attended.
Many	people	answer	this	question	by	saying	they	would	interrogate
individuals	they	wished	to	assess	about	their	emotional	reactions	to	events	and
ideas.	A	zombie,	they	believe,	would	betray	its	lack	of	subjective	experience
through	a	deficiency	in	certain	types	of	emotional	responses.	But	an	answer
along	these	lines	simply	fails	to	appreciate	the	assumptions	of	the	thought
experiment.	If	we	encountered	an	unemotional	person	(such	as	an	individual

with	certain	emotional	deficits,	as	is	common	in	certain	types	of	autism)	or	an
avatar	or	a	robot	that	was	not	convincing	as	an	emotional	human	being,	then	that
entity	is	not	a	zombie.	Remember:	According	to	Chalmers’s	assumption,	a
zombie	
is
	completely	normal	in	his	ability	to	respond,	including	the	ability	to
react	emotionally;	he	is	just	lacking	subjective	experience.	The	bottom	line	is
that	there	is	no	way	to	identify	a	zombie,	because	by	definition	there	is	no

zombie	
is
apparent	indication	of	his	zombie	nature	in	his	behavior.	So	is	this	a	distinction
without	a	difference?
Chalmers	does	not	attempt	to	answer	the	hard	question	but	does	provide
some	possibilities.	One	is	a	form	of	dualism	in	which	consciousness	per	se	does
not	exist	in	the	physical	world	but	rather	as	a	separate	ontological	reality.
According	to	this	formulation,	what	a	person	does	is	based	on	the	processes	in
her	brain.	Because	the	brain	is	causally	closed,	we	can	fully	explain	a	person’s

actions,	including	her	thoughts,	through	its	processes.	Consciousness	then	exists
essentially	in	another	realm,	or	at	least	is	a	property	separate	from	the	physical
world.	This	explanation	does	not	permit	the	mind	(that	is	to	say,	the	conscious
property	associated	with	the	brain)	to	causally	affect	the	brain.
Another	possibility	that	Chalmers	entertains,	which	is	not	logically	distinct
from	his	notion	of	dualism,	and	is	often	called	panprotopsychism,	holds	that	all

physical	systems	are	conscious,	albeit	a	human	is	more	conscious	than,	say,	a
light	switch.	I	would	certainly	agree	that	a	human	brain	has	more	to	be	conscious
about	than	a	light	switch.
My	own	view,	which	is	perhaps	a	subschool	of	panprotopsychism,	is	that
consciousness	is	an	emergent	property	of	a	complex	physical	system.	In	this
view	a	dog	is	also	conscious	but	somewhat	less	than	a	human.	An	ant	has	some
level	of	consciousness,	too,	but	much	less	that	of	a	dog.	The	ant	colony,	on	the
other	hand,	could	be	considered	to	have	a	higher	level	of	consciousness	than	the
individual	ant;	it	is	certainly	more	intelligent	than	a	lone	ant.	By	this	reckoning,
a	computer	that	is	successfully	emulating	the	complexity	of	a	human	brain
would	also	have	the	same	emergent	consciousness	as	a	human.
Another	way	to	conceptualize	the	concept	of	consciousness	is	as	a	system
that	has	“qualia.”	So	what	are	qualia?	One	definition	of	the	term	is	“conscious
experiences.”	That,	however,	does	not	take	us	very	far.	Consider	this	thought

experiment:	A	neuroscientist	is	completely	color-blind—not	the	sort	of	color-
blind	in	which	one	mixes	up	certain	shades	of,	say,	green	and	red	(as	I	do),	but
rather	a	condition	in	which	the	afflicted	individual	lives	entirely	in	a	black-and-
white	world.	(In	a	more	extreme	version	of	this	scenario,	she	has	grown	up	in	a
black-and-white	world	and	has	never	seen	any	colors.	Bottom	line,	there	is	no
color	in	her	world.)	However,	she	has	extensively	studied	the	physics	of	color—
she	is	aware	that	the	wavelength	of	red	light	is	700	nanometers—as	well	as	the

neurological	processes	of	a	person	who	can	experience	colors	normally,	and	thus
knows	a	great	deal	about	how	the	brain	processes	color.	She	knows	more	about
color	than	most	
people.	If	you	wanted	to	help	her	out	and	explain	what	this
actual	experience	of	“red”	is	like,	how	would	you	do	it?
Perhaps	you	would	read	her	a	section	from	the	poem	“Red”	by	the	Nigerian
poet	Oluseyi	Oluseun:
Red	the	colour	of	blood
the	symbol	of	life
Red	the	colour	of	danger
the	symbol	of	death
	
Red	the	colour	of	roses
the	symbol	of	beauty

Red	the	colour	of	lovers
the	symbol	of	unity

Red	the	colour	of	tomato
the	symbol	of	good	health
Red	the	colour	of	hot	fire
the	symbol	of	burning	desire
	
That	actually	would	give	her	a	pretty	good	idea	of	some	of	the	associations
people	have	made	with	red,	and	may	even	enable	her	to	hold	her	own	in	a
conversation	about	the	color.	(“Yes,	I	love	the	color	red,	it’s	so	hot	and	fiery,	so
dangerously	beautiful…”)	If	she	wanted	to,	she	could	probably	convince	people
that	she	had	experienced	red,	but	all	the	poetry	in	the	world	would	not	actually
enable	her	to	have	that	experience.
Similarly,	how	would	you	explain	what	it	feels	like	to	dive	into	water	to
someone	who	has	never	touched	water?	We	would	again	be	forced	to	resort	to
poetry,	but	there	is	really	no	way	to	impart	the	experience	itself.	These
experiences	are	what	we	refer	to	as	qualia.
Many	of	the	readers	of	this	book	have	experienced	the	color	red.	But	how
do	I	know	whether	your	experience	of	red	is	not	the	same	experience	
that	I	have

that	I	have
when	I	look	at	blue?	We	both	look	at	a	red	object	and	state	assuredly	that	it	is
red,	but	that	does	not	answer	the	question.	I	may	be	experiencing	what	you
experience	when	you	look	at	blue,	but	we	have	both	learned	to	call	red	things
red.	We	could	start	swapping	poems	again,	but	they	would	simply	reflect	the
associations	that	people	have	made	with	colors;	they	do	not	speak	to	the	actual
nature	of	the	qualia.	Indeed,	congenitally	blind	people	have	read	a	great	deal
about	colors,	as	such	references	are	replete	in	literature,	and	thus	they	do	have

some	version	of	an	experience	of	color.	How	does	their	experience	of	red
compare	with	the	experience	of	sighted	people?	This	is	really	the	same	question
as	the	one	concerning	the	woman	in	the	black-and-white	world.	It	is	remarkable
that	such	common	phenomena	in	our	lives	are	so	completely	ineffable	as	to
make	a	simple	confirmation,	like	one	that	we	are	experiencing	the	same	qualia,
impossible.
Another	definition	of	qualia	is	the	feeling	of	an	experience.	However,	this

definition	is	no	less	circular	than	our	attempts	at	defining	consciousness	above,
as	the	phrases	“feeling,”	“having	an	experience,”	and	“consciousness”	are	all
synonyms.	Consciousness	and	the	closely	related	question	of	qualia	are	a
fundamental,	perhaps	the	ultimate,	philosophical	question	(although	the	issue	of
identity	may	be	even	more	important,	as	I	will	discuss	in	the	closing	section	of
this	chapter).
So	with	regard	to	consciousness,	what	exactly	
is
	the	question	again?	It	is

is
this:	Who	or	what	is	conscious?	I	refer	to	“mind”	in	the	title	of	this	book	rather

than	“brain”	because	a	mind	is	a	brain	that	is	conscious.	We	could	also	say	that	a
mind	has	free	will	and	identity.	The	assertion	that	these	issues	are	philosophical
is	itself	not	self-evident.	I	maintain	that	these	questions	can	never	be	fully
resolved	through	science.	In	other	words,	there	are	no	falsifiable	experiments
that	we	can	contemplate	that	would	resolve	them,	not	without	making
philosophical	assumptions.	If	we	were	building	a	consciousness	detector,	Searle
would	want	it	to	ascertain	that	it	was	squirting	biological	neurotransmitters.
American	philosopher	Daniel	Dennett	(born	in	1942)	would	be	more	flexible	on
substrate,	but	might	
want	to	determine	whether	or	not	the	system	contained	a
model	of	itself	and	of	its	own	performance.	That	view	comes	closer	to	my	own,
but	at	its	core	is	still	a	philosophical	assumption.
Proposals	have	been	regularly	presented	that	purport	to	be	scientific
theories	linking	consciousness	to	some	measurable	physical	attribute—what

Searle	refers	to	as	the	“mechanism	for	causing	consciousness.”	American
scientist,	philosopher,	and	anesthesiologist	Stuart	Hameroff	(born	in	1947)	has
written	that	“cytoskeletal	filaments	are	the	roots	of	consciousness.”
2
	He	is
referring	to	thin	threads	in	every	cell	(including	neurons	but	not	limited	to	them)
called	microtubules,	which	give	each	cell	structural	integrity	and	play	a	role	in
cell	division.	His	books	and	papers	on	this	issue	contain	detailed	descriptions

2
	He	is
and	equations	that	explain	the	plausibility	that	the	microtubules	play	a	role	in
information	processing	within	the	cell.	But	the	connection	of	microtubules	to
consciousness	requires	a	leap	of	faith	not	fundamentally	different	from	the	leap
of	faith	implicit	in	a	religious	doctrine	that	describes	a	supreme	being	bestowing
consciousness	(sometimes	referred	to	as	a	“soul”)	to	certain	(usually	human)
entities.	Some	weak	evidence	is	proffered	for	Hameroff’s	view,	specifically	the

observation	that	the	neurological	processes	that	could	support	this	purported
cellular	computing	are	stopped	during	anesthesia.	But	this	is	far	from	compelling
substantiation,	given	that	lots	of	processes	are	halted	during	anesthesia.	We
cannot	even	say	for	certain	that	subjects	are	not	conscious	when	anesthetized.
All	we	do	know	is	that	people	do	not	remember	their	experiences	afterward.
Even	that	is	not	universal,	as	some	people	do	remember—accurately—their
experience	while	under	anesthesia,	including,	for	example,	conversations	by

their	surgeons.	Called	anesthesia	awareness,	this	phenomenon	is	estimated	to
occur	about	40,000	times	a	year	in	the	United	States.
3
	But	even	setting	that
aside,	consciousness	and	memory	are	completely	different	concepts.	As	I	have
discussed	extensively,	if	I	think	back	on	my	moment-to-moment	experiences
over	the	past	day,	I	have	had	a	vast	number	of	sensory	impressions	yet	I
remember	very	few	of	them.	
Was	I	therefore	not	conscious	of	what	I	was	seeing

and	hearing	all	day?	It	is	actually	a	good	question,	and	the	answer	is	not	so	clear.
English	physicist	and	mathematician	Roger	Penrose	(born	in	1931)	took	a
different	leap	of	faith	in	proposing	the	source	of	consciousness,	though	his	also
concerned	the	microtubules—specifically,	their	purported	quantum	computing
abilities.	His	reasoning,	although	not	explicitly	stated,	seemed	to	be	that
consciousness	is	mysterious,	and	a	quantum	event	is	also	mysterious,	so	they
must	be	linked	in	some	way.
Penrose	started	his	analysis	with	Turing’s	theorems	on	unsolvable	problems
and	Gödel’s	related	incompleteness	theorem.	Turing’s	premise	(which	was
discussed	in	greater	detail	in	
chapter	8
)	is	that	there	are	algorithmic	problems
that	can	be	stated	but	that	cannot	be	solved	by	a	Turing	machine.	Given	the
computational	universality	of	the	Turing	machine,	we	can	conclude	that	these
“unsolvable	problems”	cannot	be	solved	by	any	machine.	Gödel’s
incompleteness	theorem	has	a	similar	result	with	regard	to	the	ability	to	prove

chapter	8
conjectures	involving	numbers.	Penrose’s	argument	is	that	the	human	brain	is
able	to	solve	these	unsolvable	problems,	so	is	therefore	capable	of	doing	things
that	a	deterministic	machine	such	as	a	computer	is	unable	to	do.	His	motivation,
at	least	in	part,	is	to	elevate	human	beings	above	machines.	But	his	central
premise—that	humans	can	solve	Turing’s	and	Gödel’s	insoluble	problems—is
unfortunately	simply	not	true.
A	famous	unsolvable	problem	called	the	busy	beaver	problem	is	stated	as
follows:	Find	the	maximum	number	of	1s	that	a	Turing	machine	with	a	certain

number	of	states	can	write	on	its	tape.	So	to	determine	the	busy	beaver	of	the
number	
n
,	we	build	all	of	the	Turing	machines	that	have	
n
	states	(which	will	be	a
finite	number	if	
n
	is	finite)	and	then	determine	the	largest	number	of	1s	that	these
machines	write	on	their	tapes,	excluding	those	Turing	machines	that	get	into	an
infinite	loop.	This	is	unsolvable	because	as	we	seek	to	simulate	all	of	these	
n
-
state	Turing	machines,	our	simulator	will	get	into	an	infinite	loop	when	it
attempts	to	simulate	one	of

number	
n
n
n
n
-
attempts	to	simulate	one	of	
the	Turing	machines	that	does	get	into	an	infinite
loop.	However,	it	turns	out	that	computers	have	nonetheless	been	able	to
determine	the	busy	beaver	function	for	certain	
n
s.	So	have	humans,	but
computers	have	solved	the	problem	for	far	more	
n
s	than	unassisted	humans.
Computers	are	generally	better	than	humans	at	solving	Turing’s	and	Gödel’s
unsolvable	problems.
Penrose	linked	these	claimed	transcendent	capabilities	of	the	human	brain
to	the	quantum	computing	that	he	hypothesized	took	place	in	it.	According	to

n
n
Penrose,	these	neural	quantum	effects	were	somehow	inherently	not	achievable
by	computers,	so	therefore	human	thinking	has	an	inherent	edge.	In	fact,
common	electronics	uses	quantum	effects	(transistors	rely	on	quantum	tunneling

of	electrons	across	barriers);	quantum	computing	in	the	brain	has	never	been
demonstrated;	human	mental	performance	can	be	satisfactorily	explained	by
classical	computing	methods;	and	in	any	event	nothing	bars	us	from	applying
quantum	computing	in	computers.	None	of	these	objections	has	ever	been
addressed	by	Penrose.	It	was	when	critics	pointed	out	that	the	brain	is	a	warm
and	messy	place	for	quantum	computing	that	Hameroff	and	Penrose	joined
forces.	Penrose	found	a	perfect	vehicle	within	neurons	that	could	conceivably
support	quantum	computing—namely,	the	microtubules	that	Hameroff	had
speculated	were	part	of	the	information	processing	within	a	neuron.	So	the
Hameroff-Penrose	thesis	is	that	the	microtubules	in	the	neurons	are	doing
quantum	computing	and	that	this	is	responsible	for	consciousness.
This	thesis	has	also	been	criticized,	for	example,	by	Swedish	American
physicist	and	cosmologist	Max	Tegmark	(born	in	1967),	who	determined	that
quantum	events	in	microtubules	could	survive	for	only	10
−13

−13
	seconds,	which	is
much	too	brief	a	period	of	time	either	to	compute	results	of	any	significance	or
to	affect	neural	processes.	There	are	certain	types	of	problems	for	which
quantum	computing	would	show	superior	capabilities	to	classical	computing—
for	example,	the	cracking	of	encryption	codes	through	the	factoring	of	large
numbers.	However,	unassisted	human	thinking	has	proven	to	be	terrible	at
solving	them,	and	cannot	match	even	classical	computers	in	this	area,	which
suggests	that	the	brain	is	not	demonstrating	any

suggests	that	the	brain	is	not	demonstrating	any	
quantum	computing	capabilities.
Moreover,	even	if	such	a	phenomenon	as	quantum	computing	in	the	brain	did
exist,	it	would	not	necessarily	be	linked	to	consciousness.

You	Gotta	Have	Faith
What	a	piece	of	work	is	a	man!	How	noble	in	reason!	How	infinite	in
faculties!	In	form	and	moving,	how	express	and	admirable!	In	action	how
like	an	angel!	In	apprehension,	how	like	a	god!	The	beauty	of	the	world!
The	paragon	of	animals!	And	yet,	to	me,	what	is	this	quintessence	of	dust?
—Hamlet,	in	Shakespeare’s	
Hamlet
	
The	reality	is	that	these	theories	are	all	leaps	of	faith,	and	I	would	add	that	where
consciousness	is	concerned,	the	guiding	principle	is	“you	gotta	have	faith”—that
is,	we	each	need	a	leap	of	faith	as	to	what	and	who	is	conscious,	and	who	and
what	we	are	as	conscious	beings.	Otherwise	we	could	not	get	up	in	the	morning.
But	we	should	be	honest	about	the	fundamental	need	for	a	leap	of	faith	in	this
matter	and	self-reflective	as	to	what	our	own	particular	leap	involves.
People	have	very	different	leaps,	despite	impressions	to	the	contrary.
Individual	philosophical	assumptions	about	the	nature	and	source	of

consciousness	underlie	disagreements	on	issues	ranging	from	animal	rights	to
abortion,	and	will	result	in	even	more	contentious	future	conflicts	over	machine
rights.	My	objective	prediction	is	that	machines	in	the	future	will	appear	to	be
conscious	and	that	they	will	be	convincing	to	biological	people	when	they	speak
of	their	qualia.	They	will	exhibit	the	full	range	of	subtle,	familiar	emotional
cues;	they	will	make	us	laugh	and	cry;	and	they	will	get	mad	at	us	if	we	say	that
we	don’t	believe	that	they	are	conscious.	(They	will	be	very	smart,	so	we	won’t

want	that	to	happen.)	We	will	come	to	accept	that	they	are	conscious	persons.
My	own	leap	of	faith	is	this:	Once	
machines	do	succeed	in	being	convincing
when	they	speak	of	their	qualia	and	conscious	experiences,	they	will	indeed
constitute	conscious	persons.	I	have	come	to	my	position	via	this	thought
experiment:	Imagine	that	you	meet	an	entity	in	the	future	(a	robot	or	an	avatar)
that	is	completely	convincing	in	her	emotional	reactions.	She	laughs
convincingly	at	your	jokes,	and	in	turn	makes	you	laugh	and	cry	(but	not	just	by

pinching	you).	She	convinces	you	of	her	sincerity	when	she	speaks	of	her	fears
and	longings.	In	every	way,	she	seems	conscious.	She	seems,	in	fact,	like	a
person.	Would	you	accept	her	as	a	conscious	person?
If	your	initial	reaction	is	that	you	would	likely	detect	some	way	in	which

she	betrays	her	nonbiological	nature,	then	you	are	not	keeping	to	the
assumptions	in	this	hypothetical	situation,	which	established	that	she	
is
	fully
convincing.	Given	that	assumption,	if	she	were	threatened	with	destruction	and
responded,	as	a	human	would,	with	terror,	would	you	react	in	the	same
empathetic	way	that	you	would	if	you	witnessed	such	a	scene	involving	a
human?	For	myself,	the	answer	is	yes,	and	I	believe	the	answer	would	be	the
same	for	most	if	not	virtually	all	other	people	regardless	of	what	they	might
assert	now	in	a	philosophical	debate.	Again,	the	emphasis	here	is	on	the	word
“convincing.”
There	is	certainly	disagreement	on	when	or	even	whether	we	will	encounter
such	a	nonbiological	entity.	My	own	consistent	prediction	is	that	this	will	first
take	place	in	2029	and	become	routine	in	the	2030s.	But	putting	the	time	frame
aside,	I	believe	that	we	will	eventually	come	to	regard	such	entities	as	conscious.
Consider	how	we	already	treat	them	when	we	are	exposed	to	them	as	characters

in	stories	and	movies:	R2D2	from	the	
Star	Wars
	movies,	David	and	Teddy	from
the	movie	
A.I
.,	Data	from	the	TV	series	
Star	Trek:	The	Next	Generation
,	Johnny
5	from	the	movie	
Short	Circuit
,	WALL-E	from	Disney’s	movie	
Wall-E
,	T-800—
the	(good)	Terminator—in	the	second	and	later	
	movies,	Rachael	the
Replicant	from	the	movie	
Blade	Runner
	(who,	by	the	way,	is	not	aware	that	she
is	not	human),	Bumblebee	from	the	movie,	TV,	and	comic	series	
Transformers
,
and	Sonny	from	the	movie	
I,	Robot

Star	Wars
the	movie	
A.I
,	Johnny
Wall-E
,	T-800—
,
I,	Robot
.	We	do	empathize	with	these	
characters	even
though	we	know	that	they	are	nonbiological.	We	regard	them	as	conscious
persons,	just	as	we	do	biological	human	characters.	We	share	their	feelings	and
fear	for	them	when	they	get	into	trouble.	If	that	is	how	we	treat	fictional
nonbiological	characters	today,	then	that	is	how	we	will	treat	real-life
intelligences	in	the	future	that	don’t	happen	to	have	a	biological	substrate.
If	you	do	accept	the	leap	of	faith	that	a	nonbiological	entity	that	is

I,	Robot
convincing	in	its	reactions	to	qualia	is	actually	conscious,	then	consider	what
that	implies:	namely	that	consciousness	is	an	emergent	property	of	the	overall
pattern	of	an	entity,	not	the	substrate	it	runs	on.
There	is	a	conceptual	gap	between	science,	which	stands	for	
objective
measurement	and	the	conclusions	we	can	draw	thereby,	and	consciousness,
which	is	a	synonym	for	
subjective
	experience.	We	obviously	cannot	simply	ask
an	entity	in	question,	“Are	you	conscious?”	If	we	look	inside	its	“head,”

objective
biological	or	otherwise,	to	ascertain	that,	then	we	would	have	to	make
philosophical	assumptions	in	determining	what	it	is	that	we	are	looking	for.	The
question	as	to	whether	or	not	an	entity	is	conscious	is	therefore	not	a	scientific
one.	Based	on	this,	some	observers	go	on	to	question	whether	consciousness
itself	has	any	basis	in	reality.	English	writer	and	philosopher	Susan	Blackmore

(born	in	1951)	speaks	of	the	“grand	illusion	of	consciousness.”	She
acknowledges	the	reality	of	the	meme	(idea)	of	consciousness—in	other	words,
consciousness	certainly	exists	as	an	idea,	and	there	are	a	great	many	neocortical
structures	that	deal	with	the	idea,	not	to	mention	words	that	have	been	spoken
and	written	about	it.	But	it	is	not	clear	that	it	refers	to	something	real.	Blackburn
goes	on	to	explain	that	she	is	not	necessarily	denying	the	reality	of
consciousness,	but	rather	attempting	to	articulate	the	sorts	of	dilemmas	we
encounter	when	we	try	to	pin	down	the	concept.	As	British	psychologist	and
writer	Stuart	Sutherland	(1927–1998)	wrote	in	the	
International	Dictionary	of
Psychol
ogy,	“Consciousness	is	a	fascinating	but	elusive	phenomenon;	it	is
impossible	to	specify	what	it	is,	what	it	does,	or	why	it	evolved.”
4
However,	we	would	be	well	advised	not	to	dismiss	the	concept	too	
easily	as
just	a	polite	debate	between	philosophers—which,	incidentally,	dates	back	two

Psychol
4
easily	as
thousand	years	to	the	Platonic	dialogues.	The	idea	of	consciousness	underlies
our	moral	system,	and	our	legal	system	in	turn	is	loosely	built	on	those	moral
beliefs.	If	a	person	extinguishes	someone’s	consciousness,	as	in	the	act	of
murder,	we	consider	that	to	be	immoral,	and	with	some	exceptions,	a	high	crime.
Those	exceptions	are	also	relevant	to	consciousness,	in	that	we	might	authorize
police	or	military	forces	to	kill	certain	conscious	people	to	protect	a	greater
number	of	other	conscious	people.	We	can	debate	the	merits	of	particular

exceptions,	but	the	underlying	principle	holds	true.
Assaulting	someone	and	causing	her	to	experience	suffering	is	also
generally	considered	immoral	and	illegal.	If	I	destroy	my	property,	it	is	probably
acceptable.	If	I	destroy	your	property	without	your	permission,	it	is	probably	not
acceptable,	but	not	because	I	am	causing	suffering	to	your	property,	but	rather	to
you	as	the	owner	of	the	property.	On	the	other	hand,	if	my	property	includes	a
conscious	being	such	as	an	animal,	then	I	as	the	owner	of	that	animal	do	not

necessarily	have	free	moral	or	legal	rein	to	do	with	it	as	I	wish—there	are,	for
example,	laws	against	animal	cruelty.
Because	a	great	deal	of	our	moral	and	legal	system	is	based	on	protecting
the	existence	of	and	preventing	the	unnecessary	suffering	of	conscious	entities,
in	order	to	make	responsible	judgments	we	need	to	answer	the	question	as	to
who	is	conscious.	That	question	is	therefore	not	simply	a	matter	for	intellectual
debate,	as	is	evident	in	the	controversy	surrounding	an	issue	like	abortion.	I

should	point	out	that	the	abortion	issue	can	go	somewhat	beyond	the	issue	of
consciousness,	as	pro-life	proponents	argue	that	the	potential	for	an	embryo	to
ultimately	become	a	conscious	person	is	sufficient	reason	for	it	to	be	awarded
protection,	just	as	someone	in	a	coma	deserves	that	right.	But	fundamentally	the
issue	is	a	debate	about	when	a	fetus	becomes	conscious.

Perceptions	of	consciousness	also	often	affect	our	judgments	in
controversial	areas.	Looking	at	the	abortion	issue	again,	many	people	make	a
distinction	between	a	measure	like	the	morning-after	pill,	which	prevents	
the
implantation	of	an	embryo	in	the	uterus	in	the	first	days	of	pregnancy,	and	a	late-
stage	abortion.	The	difference	has	to	do	with	the	likelihood	that	the	late-stage
fetus	is	conscious.	It	is	difficult	to	maintain	that	a	few-days-old	embryo	is
conscious	unless	one	takes	a	panprotopsychist	position,	but	even	in	these	terms	it
would	rank	below	the	simplest	animal	in	terms	of	consciousness.	Similarly,	we
have	very	different	reactions	to	the	maltreatment	of	great	apes	versus,	say,
insects.	No	one	worries	too	much	today	about	causing	pain	and	suffering	to	our
computer	software	(although	we	do	comment	extensively	on	the	ability	of
software	to	cause	us	suffering),	but	when	future	software	has	the	intellectual,
emotional,	and	moral	intelligence	of	biological	humans,	this	will	become	a
genuine	concern.

Thus	my	position	is	that	I	will	accept	nonbiological	entities	that	are	fully
convincing	in	their	emotional	reactions	to	be	conscious	persons,	and	my
prediction	is	that	the	consensus	in	society	will	accept	them	as	well.	Note	that	this
definition	extends	beyond	entities	that	can	pass	the	Turing	test,	which	requires
mastery	of	human	language.	The	latter	are	sufficiently	humanlike	that	I	would
include	them,	and	I	believe	that	most	of	society	will	as	well,	but	I	also	include

entities	that	evidence	humanlike	emotional	reactions	but	may	not	be	able	to	pass
the	Turing	test—for	example,	young	children.
Does	this	resolve	the	philosophical	question	of	who	is	conscious,	at	least
for	myself	and	others	who	accept	this	particular	leap	of	faith?	The	answer	is:	
not
quite
.	We’ve	only	covered	one	case,	which	is	that	of	entities	that	act	in	a
humanlike	way.	Even	though	we	are	discussing	future	entities	that	are	not
biological,	we	are	talking	about	entities	that	demonstrate	convincing	humanlike

not
quite
reactions,	so	this	position	is	still	human-centric.	But	what	about	more	alien
forms	of	intelligence	that	are	not	humanlike?	We	can	imagine	intelligences	that
are	as	complex	as	or	perhaps	vastly	more	complex	and	intricate	than	human
brains,	but	that	have	completely	different	emotions	and	motivations.	How	do	we
decide	whether	or	not	they	are	conscious?
We	can	start	by	considering	creatures	in	the	biological	world	that	have
brains	comparable	to	those	of	humans	yet	evince	very	different	sorts	of
behaviors.	British	philosopher	David	Cockburn	(born	in	1949)	writes	about

viewing	a	video	of	a	giant	squid	that	was	under	attack	(or	at	least	it	thought	it
was—Cockburn	hypothesized	that	it	might	have	been	afraid	of	the	human	with
the	video	camera).	The	squid	shuddered	and	cowered,	and	Cockburn	writes,	“It
responded	in	a	way	which	struck	me	immediately	and	powerfully	as	one	of	fear.

Part	of	what	was	striking	in	this	sequence	was	the	way	in	which	it	was	possible
to	see	in	the	behavior	of	a	creature	physically	so	very	different	from	human
beings	an	emotion	which	was	so	unambiguously	and	specifically	one	of	fear.”
5
He	concludes	that	the	animal	was	feeling	that	emotion	and	he	articulates	the
belief	that	most	other	people	viewing	that	film	would	come	to	the	same
conclusion.	If	we	accept	Cockburn’s	description	and	conclusion,	then	we	would
have	to	add	giant	squids	to	our	list	of	conscious	entities.	However,	this	has	not
gotten	us	very	far	either,	because	it	is	still	based	on	our	empathetic	reaction	to	an
emotion	that	we	recognize	in	ourselves.	It	is	still	a	self-centric	or	human-centric
perspective.
If	we	step	outside	biology,	nonbiological	intelligence	will	be	even	more
varied	than	intelligence	in	the	biological	world.	For	example,	some	entities	may
not	have	a	fear	of	their	own	destruction,	and	may	not	have	a	need	for	the

emotions	we	see	in	humans	or	in	any	biological	creature.	Perhaps	they	could	still
pass	the	Turing	test,	or	perhaps	they	wouldn’t	even	be	willing	to	try.
We	do	in	fact	build	robots	today	that	do	not	have	a	sense	of	self-
preservation	to	carry	out	missions	in	dangerous	environments.	They’re	not
sufficiently	intelligent	or	complex	yet	for	us	to	seriously	consider	their	sentience,
but	we	can	imagine	future	robots	of	this	sort	that	are	as	complex	as	humans.
What	about	them?

Personally	I	would	say	that	if	I	saw	in	such	a	device’s	behavior	a
commitment	to	a	complex	and	worthy	goal	and	the	ability	to	execute	notable
decisions	and	actions	to	carry	out	its	mission,	I	would	be	impressed	and	probably
become	upset	if	it	got	destroyed.	This	is	now	perhaps	stretching	
the	concept	a
bit,	in	that	I	am	responding	to	behavior	that	does	not	include	many	emotions	we
consider	universal	in	people	and	even	in	biological	creatures	of	all	kinds.	But
again,	I	am	seeking	to	connect	with	attributes	that	I	can	relate	to	in	myself	and

other	people.	The	idea	of	an	entity	totally	dedicated	to	a	noble	goal	and	carrying
it	out	or	at	least	attempting	to	do	so	without	regard	for	its	own	well-being	is,
after	all,	not	completely	foreign	to	human	experience.	In	this	instance	we	are
also	considering	an	entity	that	is	seeking	to	protect	biological	humans	or	in	some
way	advance	our	agenda.
What	if	this	entity	has	its	own	goals	distinct	from	a	human	one	and	is	not
carrying	out	a	mission	we	would	recognize	as	noble	in	our	own	terms?	I	might

then	attempt	to	see	if	I	could	connect	or	appreciate	some	of	its	abilities	in	some
other	way.	If	it	is	indeed	very	intelligent,	it	is	likely	to	be	good	at	math,	so
perhaps	I	could	have	a	conversation	with	it	on	that	topic.	Maybe	it	would
appreciate	math	jokes.
But	if	the	entity	has	no	interest	in	communicating	with	me,	and	I	don’t	have

sufficient	access	to	its	actions	and	decision	making	to	be	moved	by	the	beauty	of
its	internal	processes,	does	that	mean	that	it	is	not	conscious?	I	need	to	conclude
that	entities	that	do	not	succeed	in	convincing	me	of	their	emotional	reactions,	or
that	don’t	care	to	try,	are	not	necessarily	not	conscious.	It	would	be	difficult	to
recognize	another	conscious	entity	without	establishing	some	level	of	empathetic
communication,	but	that	judgment	reflects	my	own	limitations	more	than	it	does
the	entity	under	consideration.	We	thus	need	to	proceed	with	humility.	It	is
challenging	enough	to	put	ourselves	in	the	subjective	shoes	of	another	human,	so
the	task	will	be	that	much	harder	with	intelligences	that	are	extremely	different
from	our	own.

What	Are	We	Conscious	Of?
If	we	could	look	through	the	skull	into	the	brain	of	a	consciously	thinking
person,	and	if	the	place	of	optimal	excitability	were	luminous,	then	we
should	see	playing	over	the	cerebral	surface,	a	bright	
spot	with	fantastic,
waving	borders	constantly	fluctuating	in	size	and	form,	surrounded	by	a
darkness	more	or	less	deep,	covering	the	rest	of	the	hemisphere.
—Ivan	Petrovich	Pavlov,	1913
6
	
Returning	to	the	giant	squid,	we	can	recognize	some	of	its	apparent	emotions,
but	much	of	its	behavior	is	a	mystery.	What	is	it	like	being	a	giant	squid?	How
does	it	feel	as	it	squeezes	its	spineless	body	through	a	tiny	opening?	We	don’t
even	have	the	vocabulary	to	answer	this	question,	given	that	we	cannot	even
describe	experiences	that	we	do	share	with	other	people,	such	as	seeing	the	color
red	or	feeling	water	splash	on	our	bodies.
But	we	don’t	have	to	go	as	far	as	the	bottom	of	the	ocean	to	find	mysteries
in	the	nature	of	conscious	experiences—we	need	only	consider	our	own.	I	know,

for	example,	that	I	am	conscious.	I	assume	that	you,	the	reader,	are	conscious
also.	(As	for	people	who	have	not	bought	my	book,	I	am	not	so	sure.)	But	what
am	I	conscious	
of
?	You	might	ask	yourself	the	same	question.
Try	this	thought	experiment	(which	will	work	for	those	of	you	who	drive	a
car):	Imagine	that	you	are	driving	in	the	left	lane	of	a	highway.	Now	close	your
eyes,	grab	an	imagined	steering	wheel,	and	make	the	movements	to	change	lanes
to	the	lane	to	your	right.
Okay,	before	continuing	to	read,	try	it.

of
Here	is	what	you	probably	did:	You	held	the	steering	wheel.	You	checked
that	the	right	lane	is	clear.	Assuming	the	lane	was	clear,	you	turned	the	steering
wheel	to	the	right	for	a	brief	period.	Then	you	straightened	it	out	again.	Job
done.
It’s	a	good	thing	you	weren’t	in	a	real	car,	because	you	just	zoomed	across
all	the	lanes	of	the	highway	and	crashed	into	a	tree.	While	I	probably	should
have	mentioned	that	you	shouldn’t	try	this	in	a	real	moving	car	(but	then	I
assume	you	have	already	mastered	the	rule	that	you	shouldn’t	drive	with	your

done.
eyes	closed),	that’s	not	really	the	key	problem	here.	If	you	used	
the	procedure	I

just	described—and	almost	everyone	does	when	doing	this	thought	experiment
—you	got	it	wrong.	Turning	the	wheel	to	the	right	and	then	straightening	it	out
causes	the	car	to	head	in	a	direction	that	is	diagonal	to	its	original	direction.	It
will	cross	the	lane	to	the	right,	as	you	intended,	but	it	will	keep	going	to	the	right
indefinitely	until	it	zooms	off	the	road.	What	you	needed	to	do	as	your	car
crossed	the	lane	to	the	right	was	to	then	turn	the	wheel	to	the	left,	just	as	far	as
you	had	turned	it	to	the	right,	and	
then
	straighten	it	out	again.	This	will	cause	the
car	to	again	head	straight	in	the	new	lane.
Consider	the	fact	that	if	you’re	a	regular	driver,	you’ve	done	this	maneuver
thousands	of	times.	Are	you	not	conscious	when	you	do	this?	Have	you	never
paid	attention	to	what	you	are	actually	doing	when	you	change	lanes?	Assuming
that	you	are	not	reading	this	book	in	a	hospital	while	recovering	from	a	lane-
changing	accident,	you	have	clearly	mastered	this	skill.	Yet	you	are	not

then
conscious	of	what	you	did,	however	many	times	you’ve	accomplished	this	task.
When	people	tell	stories	of	their	experiences,	they	describe	them	as
sequences	of	situations	and	decisions.	But	this	is	not	how	we	experience	a	story
in	the	first	place.	Our	original	experience	is	as	a	sequence	of	high-level	patterns,
some	of	which	may	have	triggered	feelings.	We	remember	only	a	small	subset	of
those	patterns,	if	that.	Even	if	we	are	reasonably	accurate	in	our	recounting	of	a

story,	we	use	our	powers	of	confabulation	to	fill	in	missing	details	and	convert
the	sequence	into	a	coherent	tale.	We	cannot	be	certain	what	our	original
conscious	experience	was	from	our	recollection	of	it,	yet	memory	is	the	only
access	we	have	to	that	experience.	The	present	moment	is,	well,	fleeting,	and	is
quickly	turned	into	a	memory,	or,	more	often,	not.	Even	if	an	experience	is
turned	into	a	memory,	it	is	stored,	as	the	PRTM	indicates,	as	a	high-level	pattern
composed	of	other	patterns	in	a	huge	hierarchy.	As	I	have	pointed	out	several

times,	almost	all	of	the	experiences	we	have	(like	any	of	the	times	we	changed
lanes)	are	immediately	forgotten.	So	ascertaining	what	constitutes	our	own
conscious	experience	is	actually	not	attainable.

East	Is	East	and	West	Is	West
Before	brains	there	was	no	color	or	sound	in	the	universe,	nor	was	there	any
flavor	or	aroma	and	probably	little	sense	and	no	feeling	or	emotion.
—Roger	W.	Sperry
7
	
René	Descartes	walks	into	a	restaurant	and	sits	down	for	dinner.	The	waiter
comes	over	and	asks	if	he’d	like	an	appetizer.
“No	thank	you,”	says	Descartes,	“I’d	just	like	to	order	dinner.”
“Would	you	like	to	hear	our	daily	specials?”	asks	the	waiter.
“No,”	says	Descartes,	getting	impatient.
“Would	you	like	a	drink	before	dinner?”	the	waiter	asks.
Descartes	is	insulted,	since	he’s	a	teetotaler.	“I	think	not!”	he	says
indignantly,	and	POOF!	he	disappears.
—A	joke	as	recalled	by	David	Chalmers
	
There	are	two	ways	to	view	the	questions	we	have	been	considering—converse
Western	and	Eastern	perspectives	on	the	nature	of	consciousness	and	of	reality.
In	the	Western	perspective,	we	start	with	a	physical	world	that	evolves	patterns
of	information.	After	a	few	billion	years	of	evolution,	the	entities	in	that	world

have	evolved	sufficiently	to	become	conscious	beings.	In	the	Eastern	view,
consciousness	is	the	fundamental	reality;	the	physical	world	only	comes	into
existence	through	the	thoughts	of	conscious	beings.	The	physical	world,	in	other
words,	is	the	thoughts	of	conscious	beings	made	manifest.	These	are	of	course
simplifications	of	complex	and	diverse	philosophies,	but	they	represent	the
principal	polarities	in	the	philosophies	of	consciousness	and	its	relationship	to

The	East-West	divide	on	the	issue	of	consciousness	has	also	found
expression	in	opposing	schools	of	thought	in	the	field	of	subatomic	physics.	In
quantum	mechanics,	particles	exist	as	what	are	called	probability	fields.	Any
measurement	carried	out	on	them	by	a	measuring	device	causes	
what	is	called	a
collapse	of	the	wave	function,	meaning	that	the	particle	suddenly	assumes	a
particular	location.	A	popular	view	is	that	such	a	measurement	constitutes
observation	by	a	conscious	observer,	because	otherwise	measurement	would	be	a

meaningless	concept.	Thus	the	particle	assumes	a	particular	location	(as	well	as
other	properties,	such	as	velocity)	only	when	it	is	observed.	Basically	particles
figure	that	if	no	one	is	bothering	to	look	at	them,	they	don’t	need	to	decide	where
they	are.	I	call	this	the	Buddhist	school	of	quantum	mechanics,	because	in	it
particles	essentially	don’t	exist	until	they	are	observed	by	a	conscious	person.
There	is	another	interpretation	of	quantum	mechanics	that	avoids	such
anthropomorphic	terminology.	In	this	analysis,	the	field	representing	a	particle	is
not	a	probability	field,	but	rather	just	a	function	that	has	different	values	in
different	locations.	The	field,	therefore,	is	fundamentally	what	the	particle	is.
There	are	constraints	on	what	the	values	of	the	field	can	be	in	different	locations,
because	the	entire	field	representing	a	particle	represents	only	a	limited	amount
of	information.	That	is	where	the	word	“quantum”	comes	from.	The	so-called

collapse	of	the	wave	function,	this	view	holds,	is	not	a	collapse	at	all.	The	wave
function	actually	never	goes	away.	It	is	just	that	a	measurement	device	is	also
made	up	of	particles	with	fields,	and	the	interaction	of	the	particle	field	being
measured	and	the	particle	fields	of	the	measuring	device	results	in	a	reading	of
the	particle	being	in	a	particular	location.	The	field,	however,	is	still	present.
This	is	the	Western	interpretation	of	quantum	mechanics,	although	it	is

interesting	to	note	that	the	more	popular	view	among	physicists	worldwide	is
what	I	have	called	the	Eastern	interpretation.
There	was	one	philosopher	whose	work	spanned	this	East-West	divide.	The
Austrian	British	thinker	Ludwig	Wittgenstein	(1889–1951)	studied	the
philosophy	of	language	and	knowledge	and	contemplated	the	question	of	what	it
is	that	we	can	really	know.	He	pondered	this	subject	while	a	soldier	in	World
War	I	and	took	notes	for	what	would	be	his	only	book	published	while	he	was
alive,	
Tractatus	Logico-Philosophicus

alive,	
.	The	work	had	an	unusual	structure,	and	it
was	only	through	the	efforts	of	his	former	
instructor,	British	mathematician	and
philosopher	Bertrand	Russell,	that	it	found	a	publisher	in	1921.	It	became	the
bible	for	a	major	school	of	philosophy	known	as	logical	positivism,	which
sought	to	define	the	limits	of	science.	The	book	and	the	movement	surrounding
it	were	influential	on	Turing	and	the	emergence	of	the	theory	of	computation	and
linguistics.
	anticipates	the	insight	that	all	knowledge	is

inherently	hierarchical.	The	book	itself	is	arranged	in	nested	and	numbered
statements.	For	example,	the	first	four	statements	in	the	book	are:
1		The	world	is	all	that	is	the	case.
1.1		The	world	is	the	totality	of	facts,	not	of	things.
1.11		The	world	is	determined	by	the	facts,	and	by	their	being	all	the

facts.
1.12		For	the	totality	of	facts	determines	what	is	the	case,	and	also
whatever	is	not	the	case.
	
Another	significant	statement	in	the	
Tractatus
—and	one	that	Turing	would	echo
—is	this:
4.0031	All	philosophy	is	a	critique	of	language.
	
Essentially	both	
	and	the	logical	positivism
movement	assert	that	physical	reality	exists	separate	from	our	perception	of	it,
but	that	all	we	can	know	of	that	reality	is	what	we	perceive	with	our	senses—
which	can	be	heightened	through	our	tools—and	the	logical	inferences	we	can
make	from	these	sensory	impressions.	Essentially	Wittgenstein	is	attempting	to
describe	the	methods	and	goals	of	science.	The	final	statement	in	the	book	is
number	7,	“What	we	cannot	speak	about	we	must	pass	over	in	silence.”	The
early	Wittgenstein,	accordingly,	considers	the	discussion	of	consciousness	as
circular	and	tautological	and	therefore	a	waste	of	time.
The	later	Wittgenstein,	however,	completely	rejected	this	approach	and

spent	all	of	his	philosophical	attention	talking	about	matters	that	he	had	earlier
argued	should	be	passed	over	in	silence.	His	writings	on	this	revised	thinking
were	collected	and	published	in	1953,	two	years	after	his	death,	in	a	book	called
.	He	criticized	his	earlier	ideas	in	the	
Tractatus
,
judging	them	to	be	circular	and	void	of	meaning,	and	came	to	the	view	that	what
he	had	advised	that	we	not	speak	about	was	in	fact	all	that	was	worth	reflecting

Tractatus
,
on.	These	writings	heavily	influenced	the	existentialists,	making	Wittgenstein
the	only	figure	in	modern	philosophy	to	be	a	major	architect	of	two	leading	and
contradictory	schools	of	thought	in	philosophy.
What	is	it	that	the	later	Wittgenstein	thought	was	worth	thinking	and	talking
about?	It	was	issues	such	as	beauty	and	love,	which	he	recognized	exist
imperfectly	as	ideas	in	the	minds	of	men.	However,	he	writes	that	such	concepts
do	exist	in	a	perfect	and	idealized	realm,	similar	to	the	perfect	“forms”	that	Plato

wrote	about	in	the	Platonic	dialogues,	another	work	that	illuminated	apparently
contradictory	approaches	to	the	nature	of	reality.
One	thinker	whose	position	I	believe	is	mischaracterized	is	the	French
philosopher	and	mathematician	René	Descartes.	His	famous	“I	think,	therefore	I
am”	is	generally	interpreted	to	extol	rational	thought,	in	the	sense	that	“I	think,
that	is	I	can	perform	logical	thought,	therefore	I	am	worthwhile.”	Descartes	is
therefore	considered	the	architect	of	the	Western	rational	perspective.

Reading	this	statement	in	the	context	of	his	other	writings,	however,	I	get	a
different	impression.	Descartes	was	troubled	by	what	is	referred	to	as	the	“mind-
body	problem”:	Namely,	how	does	a	conscious	mind	arise	from	the	physical
matter	of	the	brain?	From	this	perspective,	it	seems	he	was	attempting	to	push
rational	skepticism	to	the	breaking	point,	so	in	my	view	what	his	statement	really
means	is,	“I	think,	that	is	to	say,	a	subjective	experience	is	occurring,	so
therefore	all	we	know	for	sure	is	that	something—call	it	
I
—exists.”	He	could	not
be	certain	that	the	physical	world	exists,	because	all	we	have	are	our	own
individual	sense	impressions	
of	it,	which	might	be	wrong	or	completely	illusory.
We	do	know,	however,	that	the	experiencer	exists.
My	religious	upbringing	was	in	a	Unitarian	church,	where	we	studied	all	of
the	world’s	religions.	We	would	spend	six	months	on,	say,	Buddhism	and	would
go	to	Buddhist	services,	read	their	books,	and	have	discussion	groups	with	their

I
leaders.	Then	we	would	switch	to	another	religion,	such	as	Judaism.	The
overriding	theme	was	“many	paths	to	the	truth,”	along	with	tolerance	and
transcendence.	This	last	idea	meant	that	resolving	apparent	contradictions
between	traditions	does	not	require	deciding	that	one	is	right	and	the	other	is
wrong.	The	truth	can	be	discovered	only	by	finding	an	explanation	that	overrides
—transcends—seeming	differences,	especially	for	fundamental	questions	of
meaning	and	purpose.
This	is	how	I	resolve	the	Western-Eastern	divide	on	consciousness	and	the

physical	world.	In	my	view,	both	perspectives	have	to	be	true.
On	the	one	hand,	it	is	foolish	to	deny	the	physical	world.	Even	if	we	do	live
in	a	simulation,	as	speculated	by	Swedish	philosopher	Nick	Bostrom,	reality	is
nonetheless	a	conceptual	level	that	is	real	for	us.	If	we	accept	the	existence	of	the
physical	world	and	the	evolution	that	has	taken	place	in	it,	then	we	can	see	that
conscious	entities	have	evolved	from	it.
On	the	other	hand,	the	Eastern	perspective—that	consciousness	is

fundamental	and	represents	the	only	reality	that	is	truly	important—is	also
difficult	to	deny.	Just	consider	the	precious	regard	we	give	to	conscious	persons
versus	unconscious	things.	We	consider	the	latter	to	have	no	intrinsic	value
except	to	the	extent	that	they	can	influence	the	subjective	experience	of
conscious	persons.	Even	if	we	regard	consciousness	as	an	emergent	property	of	a
complex	system,	we	cannot	take	the	position	that	it	is	just	another	attribute

(along	with	“digestion”	and	“lactation,”	to	quote	John	Searle).	It	represents	what
is	truly	important.
The	word	“spiritual”	is	often	used	to	denote	the	things	that	are	of	ultimate
significance.	Many	people	don’t	like	to	use	such	terminology	from	spiritual	or
religious	traditions,	because	it	implies	sets	of	beliefs	that	they	
may	not	subscribe

to.	But	if	we	strip	away	the	mystical	complexities	of	religious	traditions	and
simply	respect	“spiritual”	as	implying	something	of	profound	meaning	to
humans,	then	the	concept	of	consciousness	fits	the	bill.	It	reflects	the	ultimate
spiritual	value.	Indeed,	“spirit”	itself	is	often	used	to	denote	consciousness.
Evolution	can	then	be	viewed	as	a	spiritual	process	in	that	it	creates
spiritual	beings,	that	is,	entities	that	are	conscious.	Evolution	also	moves	toward
greater	complexity,	greater	knowledge,	greater	intelligence,	greater	beauty,
greater	creativity,	and	the	ability	to	express	more	transcendent	emotions,	such	as
love.	These	are	all	descriptions	that	people	have	used	for	the	concept	of	God,
albeit	God	is	described	as	having	no	limitations	in	these	regards.
People	often	feel	threatened	by	discussions	that	imply	the	possibility	that	a
machine	could	be	conscious,	as	they	view	considerations	along	these	lines	as	a
denigration	of	the	spiritual	value	of	conscious	persons.	But	this	reaction	reflects

a	misunderstanding	of	the	concept	of	a	machine.	Such	critics	are	addressing	the
issue	based	on	the	machines	they	know	today,	and	as	impressive	as	they	are
becoming,	I	agree	that	contemporary	examples	of	technology	are	not	yet	worthy
of	our	respect	as	conscious	beings.	My	prediction	is	that	they	will	become
indistinguishable	from	biological	humans,	whom	we	do	regard	as	conscious
beings,	and	will	therefore	share	in	the	spiritual	value	we	ascribe	to
consciousness.	This	is	not	a	disparagement	of	people;	rather,	it	is	an	elevation	of

our	understanding	of	(some)	future	machines.	We	should	probably	adopt	a
different	terminology	for	these	entities,	as	they	will	be	a	different	sort	of
machine.
Indeed,	as	we	now	look	inside	the	brain	and	decode	its	mechanisms	we
discover	methods	and	algorithms	that	we	can	not	only	understand	but	re-create
—“the	parts	of	a	mill	pushing	on	each	other,”	to	paraphrase	German
mathematician	and	philosopher	Gottfried	Wilhelm	Leibniz	(1646–1716)	when	he
wrote	about	the	brain.	Humans	already	constitute	spiritual	machines.	Moreover,

machine.
we	will	merge	with	the	tools	we	are	creating	so	closely	
that	the	distinction
between	human	and	machine	will	blur	until	the	difference	disappears.	That
process	is	already	well	under	way,	even	if	most	of	the	machines	that	extend	us
are	not	yet	inside	our	bodies	and	brains.

Free	Will
A	central	aspect	of	consciousness	is	the	ability	to	look	ahead,	the	capability
we	call	“foresight.”	It	is	the	ability	to	plan,	and	in	social	terms	to	outline	a
scenario	of	what	is	likely	going	to	happen,	or	what	might	happen,	in	social
interactions	that	have	not	yet	taken	place….	It	is	a	system	whereby	we
improve	our	chances	of	doing	those	things	that	will	represent	our	own	best
interests….	I	suggest	that	“free	will”	is	our	apparent	ability	to	choose	and
act	upon	whichever	of	those	seem	most	useful	or	appropriate,	and	our
insistence	upon	the	idea	that	such	choices	are	our	own.
—Richard	D.	Alexander
	
Shall	we	say	that	the	plant	does	not	know	what	it	is	doing	merely	because	it
has	no	eyes,	or	ears,	or	brains?	If	we	say	that	it	acts	mechanically,	and
mechanically	only,	shall	we	not	be	forced	to	admit	that	sundry	other	and
apparently	very	deliberate	actions	are	also	mechanical?	If	it	seems	to	us	that
the	plant	kills	and	eats	a	fly	mechanically,	may	it	not	seem	to	the	plant	that

a	man	must	kill	and	eat	a	sheep	mechanically?
	
Is	the	brain,	which	is	notably	double	in	structure,	a	double	organ,	“seeming
parted,	but	yet	a	union	in	partition”?
—Henry	Maudsley
8
	
Redundancy,	as	we	have	learned,	is	a	key	strategy	deployed	by	the	neocortex.
But	there	is	another	level	of	redundancy	in	the	brain,	in	that	its	left	and	
right
hemispheres,	while	not	identical,	are	largely	the	same.	Just	as	certain	regions	of
the	neocortex	normally	end	up	processing	certain	types	of	information,	the

8
	
right
hemispheres	also	specialize	to	some	extent—for	example,	the	left	hemisphere
typically	is	responsible	for	verbal	language.	But	these	assignments	can	also	be
rerouted,	to	the	point	that	we	can	survive	and	function	somewhat	normally	with
only	one	half.	American	neuropsychology	researchers	Stella	de	Bode	and	Susan
Curtiss	reported	on	forty-nine	children	who	had	undergone	a	hemispherectomy

(removal	of	half	of	their	brain),	an	extreme	operation	that	is	performed	on
patients	with	a	life-threatening	seizure	disorder	that	exists	in	only	one
hemisphere.	Some	who	undergo	the	procedure	are	left	with	deficits,	but	those
deficits	are	specific	and	the	patients	have	reasonably	normal	personalities.	Many
of	them	thrive,	and	it	is	not	apparent	to	observers	that	they	only	have	half	a
brain.	De	Bode	and	Curtiss	write	about	left-hemispherectomized	children	who
“develop	remarkably	good	language	despite	removal	of	the	‘language’
hemisphere.”
9
	They	describe	one	such	student	who	completed	college,	attended
graduate	school,	and	scored	above	average	on	IQ	tests.	Studies	have	shown
minimal	long-term	effects	on	overall	cognition,	memory,	personality,	and	sense
of	humor.
10
	In	a	2007	study	American	researchers	Shearwood	McClelland	and
Robert	Maxwell	showed	similar	long-term	positive	results	in	adults.
11
A	ten-year-old	German	girl	who	was	born	with	only	half	of	her	brain	has

9
of	humor.
10
11
also	been	reported	to	be	quite	normal.	She	even	has	almost	perfect	vision	in	one
eye,	whereas	hemispherectomy	patients	lose	part	of	their	field	of	vision	right
after	the	operation.
12
	Scottish	researcher	Lars	Muckli	commented,	“The	brain
has	amazing	plasticity	but	we	were	quite	astonished	to	see	just	how	well	the
single	hemisphere	of	the	brain	in	this	girl	has	adapted	to	compensate	for	the
missing	half.”
While	these	observations	certainly	support	the	idea	of	plasticity	in	the
neocortex,	their	more	interesting	implication	is	that	we	each	appear	to	have	two

12
brains,	not	one,	and	we	can	do	pretty	well	with	either.	If	we	lose	
one,	we	do	lose
the	cortical	patterns	that	are	uniquely	stored	there,	but	each	brain	is	in	itself
fairly	complete.	So	does	each	hemisphere	have	its	own	consciousness?	There	is
an	argument	to	be	made	that	such	is	the	case.
Consider	split-brain	patients,	who	still	have	both	of	their	brain	hemispheres,
but	the	channel	between	them	has	been	cut.	The	corpus	callosum	is	a	bundle	of
about	250	million	axons	that	connects	the	left	and	right	cerebral	hemispheres	and

enables	them	to	communicate	and	coordinate	with	each	other.	Just	as	two	people
can	communicate	closely	with	each	other	and	act	as	a	single	decision	maker
while	remaining	separate	and	whole	individuals,	the	two	brain	hemispheres	can
function	as	a	unit	while	remaining	independent.
As	the	term	implies,	in	split-brain	patients	the	corpus	callosum	has	been	cut
or	damaged,	leaving	them	effectively	with	two	functional	brains	without	a	direct
communication	link	between	them.	American	psychology	researcher	Michael

Gazzaniga	(born	in	1939)	has	conducted	extensive	experiments	on	what	each
hemisphere	in	split-brain	patients	is	thinking.
The	left	hemisphere	in	a	split-brain	patient	usually	sees	the	right	visual

field,	and	vice	versa.	Gazzaniga	and	his	colleagues	showed	a	split-brain	patient	a
picture	of	a	chicken	claw	to	the	right	visual	field	(which	was	seen	by	his	left
hemisphere)	and	a	snowy	scene	to	the	left	visual	field	(which	was	seen	by	his
right	hemisphere).	He	then	showed	a	collection	of	pictures	so	that	both
hemispheres	could	see	them.	He	asked	the	patient	to	choose	one	of	the	pictures
that	went	well	with	the	first	picture.	The	patient’s	left	hand	(controlled	by	his
right	hemisphere)	pointed	to	a	picture	of	a	shovel,	whereas	his	right	hand	pointed
to	a	picture	of	a	chicken.	So	far	so	good—the	two	hemispheres	were	acting
independently	and	sensibly.	“Why	did	you	choose	that?”	Gazzaniga	asked	the
patient,	who	answered	verbally	(controlled	by	his	left-hemisphere	speech
center),	“The	chicken	claw	obviously	goes	with	the	chicken.”	But	then	the
patient	looked	down	and,	noticing	his	left	hand	pointing	to	the	shovel,
immediately	explained	this	(again	with	his	left-hemisphere-controlled	speech

center)	as	“and	you	need	a	shovel	to	clean	out	the	chicken	shed.”
This	is	a	confabulation.	The	right	hemisphere	(which	controls	the	left	arm
and	hand)	correctly	points	to	the	shovel,	but	because	the	left	hemisphere	(which
controls	the	verbal	answer)	is	unaware	of	the	snow,	it	confabulates	an
explanation,	yet	is	not	aware	that	it	is	confabulating.	It	is	taking	responsibility
for	an	action	it	had	never	decided	on	and	never	took,	but	thinks	that	it	did.
This	implies	that	each	of	the	two	hemispheres	in	a	split-brain	patient	has	its

own	consciousness.	The	hemispheres	appear	not	to	be	aware	that	their	body	is
effectively	controlled	by	two	brains,	because	they	learn	to	coordinate	with	each
other,	and	their	decisions	are	sufficiently	aligned	and	consistent	that	each	thinks
that	the	decisions	of	the	other	are	its	own.
Gazzaniga’s	experiment	doesn’t	prove	that	a	normal	individual	with	a
functioning	corpus	callosum	has	two	conscious	half-brains,	but	it	is	suggestive
of	that	possibility.	While	the	corpus	callosum	allows	for	effective	collaboration

between	the	two	half-brains,	it	doesn’t	necessarily	mean	that	they	are	not
separate	minds.	Each	one	could	be	fooled	into	thinking	it	has	made	all	the
decisions,	because	they	would	all	be	close	enough	to	what	each	would	have
decided	on	its	own,	and	after	all,	it	does	have	a	lot	of	influence	on	each	decision
(by	collaborating	with	the	other	hemisphere	through	the	corpus	callosum).	So	to
each	of	the	two	minds	it	would	seem	as	if	it	were	in	control.
How	would	you	test	the	conjecture	that	they	are	both	conscious?	One	could

assess	them	for	neurological	correlates	of	consciousness,	which	is	precisely	what
Gazzaniga	has	done.	His	experiments	show	that	each	hemisphere	is	acting	as	an
independent	brain.	Confabulation	is	not	restricted	to	brain	hemispheres;	we	each
do	it	on	a	regular	basis.	Each	hemisphere	is	about	as	intelligent	as	a	human,	so	if
we	believe	that	a	human	brain	is	conscious,	then	we	have	to	conclude	that	each

hemisphere	is	independently	conscious.	We	can	assess	the	neurological
correlates	and	we	can	conduct	our	own	thought	experiments	(for	example,
considering	that	if	two	brain	hemispheres	without	a	functioning	corpus	callosum
constitute	two	separate	conscious	minds,	then	the	same	would	have	to	hold	true
for	two	
hemispheres	with	a	functioning	connection	between	them),	but	any
attempt	at	a	more	direct	detection	of	consciousness	in	each	hemisphere	confronts
us	again	with	the	lack	of	a	scientific	test	for	consciousness.	But	if	we	do	allow
that	each	hemisphere	of	the	brain	is	conscious,	then	do	we	grant	that	the	so-
called	unconscious	activity	in	the	neocortex	(which	constitutes	the	vast	bulk	of
its	activity)	has	an	independent	consciousness	too?	Or	maybe	it	has	more	than
one?	Indeed,	Marvin	Minsky	refers	to	the	brain	as	a	“society	of	mind.”
13
In	another	split-brain	experiment	the	researchers	showed	the	word	“bell”	to
the	right	brain	and	“music”	to	the	left	brain.	The	patient	was	asked	what	word	he

13
saw.	The	left-hemisphere-controlled	speech	center	says	“music.”	The	subject
was	then	shown	a	group	of	pictures	and	asked	to	point	to	a	picture	most	closely
related	to	the	word	he	was	just	shown.	His	right-hemisphere-controlled	arm
pointed	to	the	bell.	When	he	was	asked	why	he	pointed	to	the	bell,	his	left-
hemisphere-controlled	speech	center	replied,	“Well,	music,	the	last	time	I	heard
any	music	was	the	bells	banging	outside	here.”	He	provided	this	explanation
even	though	there	were	other	pictures	to	choose	from	that	were	much	more

closely	related	to	music.
Again,	this	is	a	confabulation.	The	left	hemisphere	is	explaining	as	if	it
were	its	own	a	decision	that	it	never	made	and	never	carried	out.	It	is	not	doing
so	to	cover	up	for	a	friend	(that	is,	its	other	hemisphere)—it	genuinely	thinks
that	the	decision	was	its	own.
These	reactions	and	decisions	can	extend	to	emotional	responses.	They
asked	a	teenage	split-brain	patient—so	that	both	hemispheres	heard—“Who	is
your	favorite…”	and	then	fed	the	word	“girlfriend”	just	to	the	right	hemisphere

through	the	left	ear.	Gazzaniga	reports	that	the	subject	blushed	and	acted
embarrassed,	an	appropriate	reaction	for	a	teenager	when	asked	about	his
girlfriend.	But	the	left-hemisphere-controlled	speech	center	reported	that	it	had
not	heard	any	word	and	asked	for	clarification:	“My	favorite	what?”	When	asked
again	to	answer	the	question,	this	time	in	writing,	the	right-hemisphere-
controlled	left	hand	wrote	out	his	girlfriend’s	name.
Gazzaniga’s	tests	are	not	thought	experiments	but	actual	mind	experiments.

While	they	offer	an	interesting	perspective	on	the	issue	of	consciousness,	they
speak	even	more	directly	to	the	issue	of	free	will.	In	each	of	these	cases,	one	of
the	hemispheres	believes	that	it	has	made	a	decision	that	it	in	fact	never	made.
To	what	extent	is	that	true	for	the	decisions	we	make	every	day?

Consider	the	case	of	a	ten-year-old	female	epileptic	patient.	Neurosurgeon
Itzhak	Fried	was	performing	brain	surgery	while	she	was	awake	(which	is
feasible	because	there	are	no	pain	receptors	in	the	brain).
14
	Whenever	he
stimulated	a	particular	spot	on	her	neocortex,	she	would	laugh.	At	first	the
surgical	team	thought	that	they	might	be	triggering	some	sort	of	laugh	reflex,	but
they	quickly	realized	that	they	were	triggering	the	actual	perception	of	humor.
They	had	apparently	found	a	point	in	her	neocortex—there	is	obviously	more
than	one—that	recognizes	the	perception	of	humor.	She	was	not	just	laughing—
she	actually	found	the	situation	funny,	even	though	nothing	had	actually	changed
in	the	situation	other	than	their	having	stimulated	this	point	in	her	neocortex.
When	they	asked	her	why	she	was	laughing,	she	did	not	reply	along	the	lines	of,
“Oh,	no	particular	reason,”	or	“You	just	stimulated	my	brain,”	but	would
immediately	confabulate	a	reason.	She	would	point	to	something	in	the	room

and	try	to	explain	why	it	was	funny.	“You	guys	are	just	so	funny	standing	there”
was	a	typical	comment.
We	are	apparently	very	eager	to	explain	and	rationalize	our	actions,	even
when	we	didn’t	actually	make	the	decisions	that	led	to	them.	So	just	how
responsible	are	we	for	our	decisions?	Consider	these	experiments	by	physiology
professor	Benjamin	Libet	(1916–2007)	at	the	University	of	California	at	Davis.
Libet	had	participants	sit	in	front	of	a	timer,	EEG	electrodes	attached	to	their

scalps.	He	instructed	them	to	do	simple	tasks	such	as	pushing	a	button	or	moving
a	finger.	The	participants	were	asked	to	note	the	time	on	the	timer	when	they
“first	become	aware	of	the	wish	or	urge	to	act.”	Tests	indicated	a	margin	of	error
of	only	50	milliseconds	on	these	assessments	by	the	subjects.	They	also
measured	an	average	of	about	200	
milliseconds	between	the	time	when	the
subjects	reported	awareness	of	the	urge	to	act	and	the	actual	act.
15
The	researchers	also	looked	at	the	EEG	signals	coming	from	the	subjects’

15
brains.	Brain	activity	involved	in	initiating	the	action	by	the	motor	cortex	(which
is	responsible	for	carrying	out	the	action)	actually	occurred	on	average	about	500
milliseconds	prior	to	the	performance	of	the	task.	That	means	that	the	motor
cortex	was	preparing	to	carry	out	the	task	about	a	third	of	a	second	before	the
subject	was	even	aware	that	she	had	made	a	decision	to	do	so.
The	implications	of	the	Libet	experiments	have	been	hotly	debated.	Libet
himself	concluded	that	our	awareness	of	decision	making	appears	to	be	an

illusion,	that	“consciousness	is	out	of	the	loop.”	Philosopher	Daniel	Dennett
commented,	“The	action	is	originally	precipitated	in	some	part	of	the	brain,	and
off	fly	the	signals	to	muscles,	pausing	en	route	to	tell	you,	the	conscious	agent,
what	is	going	on	(but	like	all	good	officials	letting	you,	the	bumbling	president,

maintain	the	illusion	that	you	started	it	all).”
16
	At	the	same	time	Dennett	has
questioned	the	timings	recorded	by	the	experiment,	basically	arguing	that
subjects	may	not	really	be	aware	of	when	they	become	aware	of	the	decision	to
act.	One	might	wonder:	If	the	subject	is	unaware	of	when	she	is	aware	of	making
a	decision,	then	who	is?	But	the	point	is	actually	well	taken—as	I	discussed
earlier,	what	we	are	conscious	of	is	far	from	clear.
Indian	American	neuroscientist	Vilayanur	Subramanian	“Rama”
Ramachandran	(born	in	1951)	explains	the	situation	a	little	differently.	Given
that	we	have	on	the	order	of	30	billion	neurons	in	the	neocortex,	there	is	always
a	lot	going	on	there,	and	we	are	consciously	aware	of	very	little	of	it.	Decisions,
big	and	little,	are	constantly	being	processed	by	the	neocortex,	and	proposed
solutions	bubble	up	to	our	conscious	awareness.	Rather	than	free	will,
Ramachandran	suggests	we	should	talk	about	“free	won’t”—that	is,	the	power	to

reject	solutions	proposed	by	the	nonconscious	parts	of	our	neocortex.
Consider	the	analogy	to	a	military	campaign.	Army	officials	prepare	a
recommendation	to	the	president.	Prior	to	receiving	the	president’s	approval,
they	perform	preparatory	work	that	will	enable	the	decision	to	be	carried	out.	At
a	particular	moment,	the	proposed	decision	is	presented	to	the	president,	who
approves	it,	and	the	rest	of	the	mission	is	then	undertaken.	Since	the	“brain”
represented	by	this	analogy	involves	the	unconscious	processes	of	the	neocortex

(that	is,	the	officials	under	the	president)	as	well	as	its	conscious	processes	(the
president),	we	would	see	neural	activity	as	well	as	actual	actions	taking	place
prior	to	the	official	decision’s	being	made.	We	can	always	get	into	debates	in	a
particular	situation	as	to	how	much	leeway	the	officials	under	the	president
actually	gave	him	or	her	to	accept	or	reject	a	recommendation,	and	certainly
American	presidents	have	done	both.	But	it	should	not	surprise	us	that	mental

activity,	even	in	the	motor	cortex,	would	start	before	we	were	aware	that	there
was	a	decision	to	be	made.
What	the	Libet	experiments	do	underscore	is	that	there	is	a	lot	of	activity	in
our	brains	underlying	our	decisions	that	is	not	conscious.	We	already	knew	that
most	of	what	goes	in	the	neocortex	is	not	conscious;	it	should	not	be	surprising,
therefore,	that	our	actions	and	decisions	stem	from	both	unconscious	and
conscious	activity.	Is	this	distinction	important?	If	our	decisions	arise	from	both,

should	it	matter	if	we	sort	out	the	conscious	parts	from	the	unconscious?	Is	it	not
the	case	that	both	aspects	represent	our	brain?	Are	we	not	ultimately	responsible
for	everything	that	goes	on	in	our	brains?	“Yes,	I	shot	the	victim,	but	I’m	not
responsible	because	I	wasn’t	paying	attention”	is	probably	a	weak	defense.	Even
though	there	are	some	narrow	legal	grounds	on	which	a	person	may	not	be	held
responsible	for	his	decisions,	we	are	generally	held	accountable	for	all	of	the

choices	we	make.
The	observations	and	experiments	I	have	cited	above	constitute	thought
experiments	on	the	issue	of	free	will,	a	subject	that,	like	the	topic	of
consciousness,	has	been	debated	since	Plato.	The	term	“free	will”	itself	dates
back	to	the	thirteenth	century,	but	what	exactly	does	it	mean?
The	Merriam-Webster	dictionary	defines	it	as	the	“freedom	of	humans	to
make	choices	that	are	not	determined	by	prior	causes	or	by	divine	
intervention.”
You	will	notice	that	this	definition	is	hopelessly	circular:	“Free	will	is
freedom….”	Setting	aside	the	idea	of	divine	intervention’s	standing	in
opposition	to	free	will,	there	is	one	useful	element	in	this	definition,	which	is	the
idea	of	a	decision’s	“not	[being]	determined	by	prior	causes.”	I’ll	come	back	to
that	momentarily.
	states	that	free	will	is	the
“capacity	of	rational	agents	to	choose	a	course	of	action	from	among	various
alternatives.”	By	this	definition,	a	simple	computer	is	capable	of	free	will,	so	it	is

less	helpful	than	the	dictionary	definition.
Wikipedia	is	actually	a	bit	better.	It	defines	free	will	as	“the	ability	of	agents
to	make	choices	free	from	certain	kinds	of	constraints….	The	constraint	of
dominant	concern	has	been…determinism.”	Again,	it	uses	the	circular	word
“free”	in	defining	free	will,	but	it	does	articulate	what	has	been	regarded	as	the
principal	enemy	of	free	will:	
determinism
.	In	that	respect	the	Merriam-Webster
definition	above	is	actually	similar	in	its	reference	to	decisions	that	“are	not

determined	by	prior	causes.”
So	what	do	we	mean	by	determinism?	If	I	put	“2	+	2”	into	a	calculator	and
it	displays	“4,”	can	I	say	that	the	calculator	displayed	its	free	will	by	deciding	to
display	that	“4”?	No	one	would	accept	that	as	a	demonstration	of	free	will,
because	the	“decision”	was	predetermined	by	the	internal	mechanisms	of	the
calculator	and	the	input.	If	I	put	in	a	more	complex	calculation,	we	still	come	to
the	same	conclusion	with	regard	to	its	lack	of	free	will.
How	about	Watson	when	it	answers	a	
Jeopardy!
	query?	Although	its

Jeopardy!
deliberations	are	far	more	complex	than	those	of	the	calculator,	very	few	if	any
observers	would	ascribe	free	will	to	its	decisions.	No	one	human	knows	exactly
how	all	of	its	programs	work,	but	we	can	identify	a	group	of	people	who
collectively	can	describe	all	of	its	methods.	More	important,	its	output	is
determined	by	(1)	all	of	its	programs	at	the	moment	that	the	query	is	posed,	(2)
the	query	itself,	(3)	the	state	of	its	internal	parameters	that	influence	its
decisions,	and	(4)	its	trillions	of	bytes	of	knowledge	bases,	including

encyclopedias.	Based	on	these	four	categories	of	information,	its	output	
is
determined.	We	might	speculate	that	presenting	the	same	query	would	always

get	the	same	response,	but	Watson	is	programmed	to	learn	from	its	experience,
so	there	is	the	possibility	that	subsequent	answers	would	be	different.	However,
that	does	not	contradict	this	analysis;	rather,	it	just	constitutes	a	change	in	item	3,
the	parameters	that	control	its	decisions.
So	how	exactly	does	a	human	differ	from	Watson,	such	that	we	ascribe	free
will	to	the	human	but	not	to	the	computer	program?	We	can	identify	several
factors.	Even	though	Watson	is	a	better	
Jeopardy!
	player	than	most	if	not	all
humans,	it	is	nonetheless	not	nearly	as	complex	as	a	human	neocortex.	Watson
does	possess	a	lot	of	knowledge,	and	it	does	use	hierarchical	methods,	but	the
complexity	of	its	hierarchical	thinking	is	still	considerably	less	than	that	of	a
human.	So	is	the	difference	simply	one	of	the	scale	of	complexity	of	its
hierarchical	thinking?	There	is	an	argument	to	be	made	that	the	issue	does	come
down	to	this.	In	my	discussion	of	the	issue	of	consciousness	I	noted	that	my	own

Jeopardy!
leap	of	faith	is	that	I	would	consider	a	computer	that	passed	a	valid	Turing	test	to
be	conscious.	The	best	chatbots	are	not	able	to	do	that	today	(although	they	are
steadily	improving),	so	my	conclusion	with	regard	to	consciousness	is	a	matter
of	the	level	of	performance	of	the	entity.	Perhaps	the	same	is	true	of	my
ascribing	free	will	to	it.
Consciousness	is	indeed	one	philosophical	difference	between	human
brains	and	contemporary	software	programs.	We	consider	human	brains	to	be

conscious,	whereas	we	do	not—
yet
—attribute	that	to	software	programs.	Is	this
the	factor	we	are	looking	for	that	underlies	free	will?
A	simple	mind	experiment	would	argue	that	consciousness	is	indeed	a	vital
part	of	free	will.	Consider	a	situation	in	which	someone	performs	an	action	with
no	awareness	that	she	is	doing	it—it	is	carried	out	entirely	by	nonconscious
activity	in	that	person’s	brain.	Would	we	regard	this	to	be	a	display	of	free	will?
Most	people	would	answer	no.	If	the	action	was	harmful,	we	would	probably

yet
still	hold	that	person	responsible	but	look	for	some	recent	conscious	acts	that
may	have	caused	that	person	to	perform	actions	without	conscious	awareness,
such	as	taking	one	drink	too	many,	or	just	
failing	to	train	herself	adequately	to
consciously	consider	her	decisions	before	she	acted	on	them.
According	to	some	commentators,	the	Libet	experiments	argued	against
free	will	by	highlighting	how	much	of	our	decision	making	is	not	conscious.
Since	there	is	a	reasonable	consensus	among	philosophers	that	free	will	does

imply	conscious	decision	making,	it	appears	to	be	one	prerequisite	for	free	will.
However,	to	many	observers,	consciousness	is	a	necessary	but	not	sufficient
condition.	If	our	decisions—conscious	or	otherwise—are	predetermined	before
we	make	them,	how	can	we	say	that	our	decisions	are	free?	This	position,	which
holds	that	free	will	and	determinism	are	not	compatible,	is	known	as

incompatibilism.	For	example,	American	philosopher	Carl	Ginet	(born	in	1932)
argues	that	if	events	in	the	past,	present,	and	future	are	determined,	then	we	can
be	considered	to	have	no	control	over	them	or	their	consequences.	Our	apparent
decisions	and	actions	are	simply	part	of	this	predetermined	sequence.	To	Ginet,
this	rules	out	free	will.
Not	everyone	regards	determinism	as	being	incompatible	with	the	concept
of	free	will,	however.	The	compatibilists	argue,	essentially,	that	you’re	free	to
decide	what	you	want	even	though	what	you	decide	is	or	may	be	determined.
Daniel	Dennett,	for	example,	argues	that	while	the	future	may	be	determined
from	the	state	of	the	present,	the	reality	is	that	the	world	is	so	intricately	complex
that	we	cannot	possibly	know	what	the	future	will	bring.	We	can	identify	what
he	refers	to	as	“expectations,”	and	we	are	indeed	free	to	perform	acts	that	differ
from	these	expectations.	We	should	consider	how	our	decisions	and	actions

compare	to	these	expectations,	not	to	a	theoretically	determined	future	that	we
cannot	in	fact	know.	That,	Dennett	argues,	is	sufficient	for	free	will.
Gazzaniga	also	articulates	a	compatibilist	position:	“We	are	personally
responsible	agents	and	are	to	be	held	accountable	for	our	actions,	even	though
we	live	in	a	determined	world.”
17
	A	cynic	might	interpret	this	view	as:	You	have
no	control	over	your	actions,	but	we’ll	blame	you	anyway.
Some	thinkers	dismiss	the	idea	of	free	will	as	an	illusion.	Scottish

17
philosopher	David	Hume	(1711–1776)	described	it	as	simply	a	“verbal”	matter
characterized	by	“a	false	sensation	or	seeming	experience.”
18
	German
philosopher	Arthur	Schopenhauer	(1788–1860)	wrote	that	“everyone	believes
himself	
a	priori
	to	be	perfectly	free,	even	in	his	individual	actions,	and	thinks
that	at	every	moment	he	can	commence	another	manner	of	life….	But	
a
posteriori
,	through	experience,	he	finds	to	his	astonishment	that	he	is	not	free,
but	subjected	to	necessity,	that	in	spite	of	all	his	resolutions	and	reflections	he

18
	German
himself	
a	priori
a
does	not	change	his	conduct,	and	that	from	the	beginning	of	his	life	to	the	end	of
it,	he	must	carry	out	the	very	character	which	he	himself	condemns.”
19
I	would	add	several	points	here.	The	concept	of	free	will—and
responsibility,	which	is	a	closely	aligned	idea—is	useful,	and	indeed	vital,	to
maintaining	social	order,	whether	or	not	free	will	actually	exists.	Just	as
consciousness	clearly	exists	as	a	meme,	so	too	does	free	will.	Attempts	to	prove
its	existence,	or	even	to	define	it,	may	become	hopelessly	circular,	but	the	reality

19
is	that	almost	everyone	believes	in	the	idea.	Very	substantial	portions	of	our
higher-level	neocortex	are	devoted	to	the	concept	that	we	make	free	choices	and
are	responsible	for	our	actions.	Whether	in	a	strict	philosophical	sense	that	is
true	or	even	possible,	society	would	be	far	worse	off	if	we	did	not	have	such

beliefs.
Furthermore,	the	world	is	not	necessarily	determined.	I	discussed	above	two
perspectives	on	quantum	mechanics,	which	differ	with	respect	to	the	relationship
of	quantum	fields	to	an	observer.	A	popular	interpretation	of	the	observer-based
perspective	provides	a	role	for	consciousness:	Particles	do	not	resolve	their
quantum	ambiguity	until	observed	by	a	conscious	observer.	There	is	another	split
in	the	philosophy	of	quantum	events	that	has	a	bearing	on	our	discussion	of	free
will,	one	that	revolves	around	the	question:	Are	quantum	events	determined	or
random?
The	most	common	interpretation	of	a	quantum	event	is	that	when	the	wave
function	constituting	a	particle	“collapses,”	the	particle’s	location	becomes
specific.	Over	a	great	many	such	events,	there	will	be	a	predictable	distribution
(which	is	why	the	wave	function	is	considered	to	be	a	probability	distribution),
but	the	resolution	for	each	such	particle	undergoing	a	
collapse	of	its	wave

random?
function	is	random.	The	opposing	interpretation	is	deterministic:	specifically,
that	there	is	a	hidden	variable	that	we	are	unable	to	detect	separately,	but	whose
value	determines	the	particle’s	position.	The	value	or	phase	of	the	hidden
variable	at	the	moment	of	the	wave	function	collapse	determines	the	position	of
the	particle.	Most	quantum	physicists	seem	to	favor	the	idea	of	a	random
resolution	according	to	the	probability	field,	but	the	equations	for	quantum
mechanics	do	allow	for	the	existence	of	such	a	hidden	variable.

Thus	the	world	may	not	be	determined	after	all.	According	to	the
probability	wave	interpretation	of	quantum	mechanics,	there	is	a	continual
source	of	uncertainty	at	the	most	basic	level	of	reality.	However,	this	observation
does	not	necessarily	resolve	the	concerns	of	the	incompatibilists.	It	is	true	that
under	this	interpretation	of	quantum	mechanics,	the	world	is	not	determined,	but
our	concept	of	free	will	extends	beyond	decisions	and	actions	that	are	merely
random.	Most	incompatibilists	would	find	the	concept	of	free	will	to	also	be

incompatible	with	our	decisions’	being	essentially	accidental.	Free	will	seems	to
imply	purposeful	decision	making.
Dr.	Wolfram	proposes	a	way	to	resolve	the	dilemma.	His	book	
A	New	Kind
of	Science
	(2002)	presents	a	comprehensive	view	of	the	idea	of	cellular	automata
and	their	role	in	every	facet	of	our	lives.	A	cellular	automaton	is	a	mechanism	in
which	the	value	of	information	cells	is	continually	recomputed	as	a	function	of
the	cells	near	it.	John	von	Neumann	created	a	theoretical	self-replicating

A	New	Kind
of	Science
machine	called	a	universal	constructor	that	was	perhaps	the	first	cellular
automaton.
Dr.	Wolfram	illustrates	his	thesis	with	the	simplest	possible	cellular
automata,	a	group	of	cells	in	a	one-dimensional	line.	At	each	point	in	time,	each

cell	can	have	one	of	two	values:	black	or	white.	The	value	of	each	cell	is
recomputed	for	each	cycle.	The	value	of	a	cell	for	the	next	cycle	is	a	function	of
its	current	value	as	well	as	the	value	of	its	two	adjacent	neighbors.	Each	cellular
automaton	is	characterized	by	a	rule	that	determines	how	we	compute	whether	a
cell	is	black	or	white	in	the	next	cycle.
Consider	the	example	of	what	Dr.	Wolfram	calls	rule	222.
The	eight	possible	combinations	of	value	for	the	cell	being	recomputed	and
its	left	and	right	neighbors	are	shown	in	the	top	row.	Its	new	value	is	shown	in
the	bottom	row.	So,	for	example,	if	the	cell	is	black	and	its	two	neighbors	are
also	black,	then	the	cell	will	remain	black	in	the	next	generation	(see	the	leftmost
subrule	of	rule	222).	If	the	cell	is	white,	its	left	neighbor	is	white,	and	its	right
neighbor	is	black,	then	it	will	be	changed	to	black	in	the	next	generation	(see	the
subrule	of	rule	222	that	is	second	from	the	right).

The	universe	for	this	simple	cellular	automaton	is	just	one	row	of	cells.	If
we	start	with	just	one	black	cell	in	the	middle	and	show	the	evolution	of	the	cells
over	multiple	generations	(where	each	row	as	we	move	down	represents	a	new
generation	of	values),	the	results	of	rule	222	look	like	this:

An	automaton	is	based	on	a	rule,	and	a	rule	defines	whether	the	cell	will	be
black	or	white	based	on	which	of	the	eight	possible	patterns	exist	in	the	current
generation.	Thus	there	are	2
8
	=	256	possible	rules.	Dr.	Wolfram	listed	all	256
possible	such	automata	and	assigned	each	a	Wolfram	code	from	0	to	255.
Interestingly,	these	256	theoretical	machines	have	very	different	properties.	The
automata	in	what	Dr.	Wolfram	calls	class	I,	such	as	rule	222,	create	very
predictable	patterns.	If	I	were	to	ask	what	the	value	of	the	middle	cell	was	after	a
trillion	trillion	iterations	of	rule	222,	you	could	answer	easily:	black.
Much	more	interesting,	however,	are	the	class	IV	automata,	illustrated	by
rule	110.
Multiple	generations	of	this	automaton	look	like	this:

The	interesting	thing	about	the	rule	110	automaton	and	class	IV	automata	in
general	is	that	the	results	are	completely	unpredictable.	The	results	pass	the
strictest	mathematical	tests	for	randomness,	yet	they	do	not	simply	generate
noise:	There	are	repeating	patterns,	but	they	repeat	in	
odd	and	unpredictable
ways.	If	I	were	to	ask	you	what	the	value	of	a	particular	cell	was	after	a	trillion
trillion	iterations,	there	would	be	no	way	to	answer	that	question	without
actually	running	this	machine	through	that	many	generations.	The	solution	is
clearly	determined,	because	this	is	a	very	simple	deterministic	machine,	but	it	is
completely	unpredictable	without	actually	running	the	machine.
Dr.	Wolfram’s	primary	thesis	is	that	the	world	is	one	big	class	IV	cellular
automaton.	The	reason	that	his	book	is	titled	
A	New
	
Kind	of	Science
	is	because
this	theory	contrasts	with	most	other	scientific	laws.	If	there	is	a	satellite	orbiting
Earth,	we	can	predict	where	it	will	be	five	years	from	now	without	having	to	run

A	New
	
	is	because
through	each	moment	of	a	simulated	process	by	using	the	relevant	laws	of
gravity	and	solve	where	it	will	be	at	points	in	time	far	in	the	future.	But	the
future	state	of	class	IV	cellular	automata	cannot	be	predicted	without	simulating
every	step	along	the	way.	If	the	universe	is	a	giant	cellular	automaton,	as	Dr.
Wolfram	postulates,	there	would	be	no	computer	big	enough—since	every
computer	would	be	a	subset	of	the	universe—that	could	run	such	a	simulation.
Therefore	the	future	state	of	the	universe	is	completely	unknowable	even	though

it	is	deterministic.
Thus	even	though	our	decisions	are	determined	(because	our	bodies	and
brains	are	part	of	a	deterministic	universe),	they	are	nonetheless	inherently
unpredictable	because	we	live	in	(and	are	part	of)	a	class	IV	automaton.	We
cannot	predict	the	future	of	a	class	IV	automaton	except	to	let	the	future	unfold.
For	Dr.	Wolfram,	this	is	sufficient	to	allow	for	free	will.
We	don’t	have	to	look	to	the	universe	to	see	future	events	that	are
determined	yet	unpredictable.	None	of	the	scientists	who	have	worked	on

Watson	can	predict	what	it	will	do,	because	the	program	is	just	too	complex	and
varied,	and	its	performance	is	based	on	knowledge	that	is	far	too	extensive	for
any	human	to	master.	If	we	believe	that	humans	exhibit	free	will,	then	it	follows
that	we	have	to	allow	that	future	versions	of	Watson	or	Watson-like	machines
can	exhibit	it	also.
My	own	leap	of	faith	is	that	I	believe	that	humans	have	free	will,	and	while
I	act	as	if	that	is	the	case,	I	am	hard	pressed	to	find	examples	among	my	
own
decisions	that	illustrate	that.	Consider	the	decision	to	write	this	book—I	never
made	that	decision.	Rather,	the	idea	of	the	book	decided	that	for	me.	In	general,	I
find	myself	captive	to	ideas	that	seem	to	implant	themselves	in	my	neocortex
and	take	over.	How	about	the	decision	to	get	married,	which	I	made	(in
collaboration	with	one	other	person)	thirty-six	years	ago?	At	the	time,	I	had	been
following	the	usual	program	of	being	attracted	to—and	pursuing—a	pretty	girl.	I
then	fell	in	love.	Where	is	the	free	will	in	that?

But	what	about	the	little	decisions	I	make	every	day—for	example,	the
specific	words	I	choose	to	write	in	my	book?	I	start	with	a	blank	virtual	sheet	of
paper.	No	one	is	telling	me	what	to	do.	There	is	no	editor	looking	over	my
shoulder.	My	choices	are	
entirely
	up	to	me.	I	am	free—
totally	free
—to	write
whatever
	I…
Uh,	
grok
…
Grok
?	Okay,	I	did	it—I	finally	applied	my	free	will.	I	was	going	to	write	the
word	“want,”	but	I	made	a	free	decision	to	write	something	totally	unexpected
instead.	This	is	perhaps	the	first	time	I’ve	succeeded	in	exercising	pure	free	will.

entirely
—to	write
whatever
	I…
Uh,	
grok
…
Grok
Or	not.
It	should	be	apparent	that	that	was	a	display	not	of	will,	but	rather	of	trying
to	illustrate	a	point	(and	perhaps	a	weak	sense	of	humor).
Although	I	share	Descartes’	confidence	that	I	am	conscious,	I’m	not	so	sure
about	free	will.	It	is	difficult	to	escape	Schopenhauer’s	conclusion	that	“you	can
do	what	you	will,	but	in	any	given	moment	of	your	life	you	can	
will
	only	one
definite	thing	and	absolutely	nothing	other	than	that	one	thing.”
20
	Nonetheless	I

Or	not.
will
	only	one
20
will	continue	to	act	as	if	I	have	free	will	and	to	believe	in	it,	so	long	as	I	don’t
have	to	explain	why.

Identity
A	philosopher	once	had	the	following	dream.
First	Aristotle	appeared,	and	the	philosopher	said	to	him,	“Could	you
give	me	a	fifteen-minute	capsule	sketch	of	your	entire	philosophy?”	
To	the
philosopher’s	surprise,	Aristotle	gave	him	an	excellent	exposition	in	which
he	compressed	an	enormous	amount	of	material	into	a	mere	fifteen	minutes.
But	then	the	philosopher	raised	a	certain	objection	which	Aristotle	couldn’t
answer.	Confounded,	Aristotle	disappeared.
Then	Plato	appeared.	The	same	thing	happened	again,	and	the
philosopher’s	objection	to	Plato	was	the	same	as	his	objection	to	Aristotle.
Plato	also	couldn’t	answer	it	and	disappeared.
Then	all	the	famous	philosophers	of	history	appeared	one	by	one	and
our	philosopher	refuted	every	one	with	the	same	objection.
After	the	last	philosopher	vanished,	our	philosopher	said	to	himself,	“I
know	I’m	asleep	and	dreaming	all	this.	Yet	I’ve	found	a	universal	refutation
for	all	philosophical	systems!	Tomorrow	when	I	wake	up,	I	will	probably

have	forgotten	it,	and	the	world	will	really	miss	something!”	With	an	iron
effort,	the	philosopher	forced	himself	to	wake	up,	rush	over	to	his	desk,	and
write	down	his	universal	refutation.	Then	he	jumped	back	into	bed	with	a
sigh	of	relief.
The	next	morning	when	he	awoke,	he	went	over	to	the	desk	to	see
what	he	had	written.	It	was,	“That’s	what	
you
	say.”
—Raymond	Smullyan,	as	quoted	by	David	Chalmers
21
	
What	I	wonder	about	ever	more	than	whether	or	not	I	am	conscious	or	exercise

you
	say.”
21
	
free	will	is	why	I	happen	to	be	conscious	of	the	experiences	and	decisions	of	this
one	particular	person	who	writes	books,	enjoys	hiking	and	biking,	takes
nutritional	supplements,	and	so	on.	An	obvious	answer	would	be,	“Because
that’s	who	you	are.”
That	exchange	is	probably	no	more	tautological	than	my	answers	above	to
questions	about	consciousness	and	free	will.	But	actually	I	do	have	a	better
answer	for	why	my	consciousness	is	associated	with	this	particular	person:	It	is
because	that	is	who	I	created	myself	to	be.

A	common	aphorism	is,	“You	are	what	you	eat.”	It	is	even	more	true	to	say,
“You	are	what	you	think.”	As	we	have	discussed,	all	of	the	hierarchical
structures	in	my	neocortex	that	define	my	personality,	skills,	and	knowledge	are
the	result	of	my	own	thoughts	and	experiences.	The	people	I	choose	to	interact
with	and	the	ideas	and	projects	I	choose	to	engage	in	are	all	primary
determinants	of	who	I	become.	For	that	matter,	what	I	eat	also	reflects	the
decisions	made	by	my	neocortex.	Accepting	the	positive	side	of	the	free	will
duality	for	the	moment,	it	is	my	own	decisions	that	result	in	who	I	am.
Regardless	of	how	we	came	to	be	who	we	are,	each	of	us	has	the	desire	for
our	identity	to	persist.	If	you	didn’t	have	the	will	to	survive,	you	wouldn’t	be
here	reading	this	book.	Every	creature	has	that	goal—it	is	the	principal
determinant	of	evolution.	The	issue	of	identity	is	perhaps	even	harder	to	define
than	consciousness	or	free	will,	but	is	arguably	more	important.	After	all,	we

need	to	know	what	we	are	if	we	seek	to	preserve	our	existence.
Consider	this	thought	experiment:	You	are	in	the	future	with	technologies
more	advanced	than	today’s.	While	you	are	sleeping,	some	group	scans	your
brain	and	picks	up	every	salient	detail.	Perhaps	they	do	this	with	blood	cell–
sized	scanning	machines	traveling	in	the	capillaries	of	your	brain	or	with	some
other	suitable	noninvasive	technology,	but	they	have	all	of	the	information	about
your	brain	at	a	particular	point	in	time.	They	also	pick	up	and	record	any	bodily

details	that	might	reflect	on	your	state	of	mind,	such	as	the	endocrine	system.
They	instantiate	this	“mind	file”	in	a	nonbiological	body	that	looks	and	moves
like	you	and	has	the	requisite	subtlety	and	suppleness	to	pass	for	you.	In	the
morning	you	are	informed	about	this	transfer	and	you	watch	(perhaps	without
being	noticed)	your	mind	clone,	whom	we’ll	call	You	2.	You	2	is	talking	about
his	or	her	life	as	if	s/he	were	you,	and	relating	how	s/he	discovered	that	very
morning	that	s/he	had	been	given	a	much	more	durable	new	version	2.0	body.

“Hey,	I	kind	of	like	this	new	body!”	s/he	exclaims.
The	first	question	to	consider	is:	Is	You	2	conscious?	Well,	s/he	certainly
seems	to	be.	S/he	passes	the	test	I	articulated	earlier,	in	that	s/he	has	the	subtle
cues	of	being	a	feeling,	conscious	person.	If	you	are	conscious,	then	so	too	is
You	2.
So	if	you	were	to,	uh,	disappear,	no	one	would	notice.	You	2	would	go
around	claiming	to	be	you.	All	of	your	friends	and	loved	ones	would	be	content
with	the	situation	and	perhaps	pleased	that	you	now	have	a	more	durable	body

You	2.
and	mental	substrate	than	you	used	to	have.	Perhaps	your	more	philosophically
minded	friends	would	express	concerns,	but	for	the	most	part,	everybody	would
be	happy,	including	you,	or	at	least	the	person	who	is	convincingly	claiming	to
be	you.

So	we	don’t	need	your	old	body	and	brain	anymore,	right?	Okay	if	we
dispose	of	it?
You’re	probably	not	going	to	go	along	with	this.	I	indicated	that	the	scan
was	noninvasive,	so	you	are	still	around	and	still	conscious.	Moreover	your
sense	of	identity	is	still	with	you,	not	with	You	2,	even	though	You	2	thinks	s/he
is	a	continuation	of	you.	You	2	might	not	even	be	aware	that	you	exist	or	ever
existed.	In	fact	you	would	not	be	aware	of	the	existence	of	You	2	either,	if	we
hadn’t	told	you	about	it.
Our	conclusion?	You	2	is	conscious	but	is	a	different	person	than	you—You
2	has	a	different	identity.	S/he	is	extremely	similar,	much	more	so	than	a	mere
genetic	clone,	because	s/he	also	shares	all	of	your	neocortical	patterns	and
connections.	Or	I	should	say	s/he	shared	those	patterns	at	the	moment	s/he	was
created.	At	that	point,	the	two	of	you	started	to	go	your	own	ways,	neocortically
speaking.	You	are	still	around.	You	are	not	having	the	same	experiences	as	You
2.	Bottom	line:	You	2	is	not	you.

Okay,	so	far	so	good.	Now	consider	another	thought	experiment—one	that
is,	I	believe,	more	realistic	in	terms	of	what	the	future	will	bring.	You	undergo	a
procedure	to	replace	a	very	small	part	of	your	brain	with	a	nonbiological	unit.
You’re	convinced	that	it’s	safe,	and	there	are	reports	of	various	benefits.
This	is	not	so	far-fetched,	as	it	is	done	routinely	for	people	with
neurological	and	sensory	impairments,	such	as	the	neural	implant	for

Parkinson’s	disease	and	cochlear	implants	for	the	deaf.	In	these	cases	the
computerized	device	is	placed	inside	the	body	but	outside	the	brain	yet
connected	into	the	brain	(or	in	the	case	of	the	cochlear	implants,	to	the	auditory
nerve).	In	my	view	the	fact	that	the	actual	computer	is	physically	
placed	outside
the	actual	brain	is	not	philosophically	significant:	We	are	effectively	augmenting
the	brain	and	replacing	with	a	computerized	device	those	of	its	functions	that	no

longer	work	properly.	In	the	2030s,	when	intelligent	computerized	devices	will
be	the	size	of	blood	cells	(and	keep	in	mind	that	white	blood	cells	are	sufficiently
intelligent	to	recognize	and	combat	pathogens),	we	will	introduce	them
noninvasively,	no	surgery	required.
Returning	to	our	future	scenario,	you	have	the	procedure,	and	as	promised,
it	works	just	fine—certain	of	your	capabilities	have	improved.	(You	have	better
memory,	perhaps.)	So	are	you	still	you?	Your	friends	certainly	think	so.	You

think	so.	There	is	no	good	argument	that	you’re	suddenly	a	different	person.
Obviously,	you	underwent	the	procedure	in	order	to	effect	a	change	in
something,	but	you	are	still	the	same	you.	Your	identity	hasn’t	changed.
Someone	else’s	consciousness	didn’t	suddenly	take	over	your	body.
Okay,	so,	encouraged	by	these	results,	you	now	decide	to	have	another

procedure,	this	time	involving	a	different	region	of	the	brain.	The	result	is	the
same:	You	experience	some	improvement	in	capability,	but	you’re	still	you.
It	should	be	apparent	where	I	am	going	with	this.	You	keep	opting	for
additional	procedures,	your	confidence	in	the	process	only	increasing,	until
eventually	you’ve	changed	every	part	of	your	brain.	Each	time	the	procedure
was	carefully	done	to	preserve	all	of	your	neocortical	patterns	and	connections
so	that	you	have	not	lost	any	of	your	personality,	skills,	or	memories.	There	was
never	a	you	and	a	You	2;	there	was	only	you.	No	one,	including	you,	ever
notices	you	ceasing	to	exist.	Indeed—there	you	are.
Our	conclusion:	You	still	exist.	There’s	no	dilemma	here.	Everything	is
fine.
Except	for	this:
	You,	after	the	gradual	replacement	process,	are	entirely
equivalent	to	You	2	in	the	prior	thought	experiment	(which	I	will	call	the	scan-
and-instantiate	scenario).	You,	after	the	gradual	replacement	scenario,	have	all	of

fine.
the	neocortical	patterns	and	connections	that	you	had	
originally,	only	in	a
nonbiological	substrate,	which	is	also	true	of	You	2	in	the	scan-and-instantiate
scenario.	You,	after	the	gradual	replacement	scenario,	have	some	additional
capabilities	and	greater	durability	than	you	did	before	the	process,	but	this	is
likewise	true	of	You	2	in	the	scan-and-instantiate	process.
But	we	concluded	that	You	2	is	
not
	you.	And	if	you,	after	the	gradual
replacement	process,	are	entirely	equivalent	to	You	2	after	the	scan-and-

not
instantiate	process,	then	you	after	the	gradual	replacement	process	must	also	not
be	you.
That,	however,	contradicts	our	earlier	conclusion.	The	gradual	replacement
process	consists	of	multiple	steps.	Each	of	those	steps	appeared	to	preserve
identity,	just	as	we	conclude	today	that	a	Parkinson’s	patient	has	the	same
identity	after	having	had	a	neural	implant	installed.
22
It	is	just	this	sort	of	philosophical	dilemma	that	leads	some	people	to
conclude	that	these	replacement	scenarios	will	never	happen	(even	though	they

be	you.
22
are	already	taking	place).	But	consider	this:	We	naturally	undergo	a	gradual
replacement	process	throughout	our	lives.	Most	of	our	cells	in	our	body	are
continuously	being	replaced.	(You	just	replaced	100	million	of	them	in	the
course	of	reading	the	last	sentence.)	Cells	in	the	inner	lining	of	the	small
intestine	turn	over	in	about	a	week,	as	does	the	stomach’s	protective	lining.	The
life	span	of	white	blood	cells	ranges	from	a	few	days	to	a	few	months,	depending
on	the	type.	Platelets	last	about	nine	days.

Neurons	persist,	but	their	organelles	and	their	constituent	molecules	turn
over	within	a	month.
23
	The	half-life	of	a	neuron	microtubule	is	about	ten

minutes;	the	actin	filaments	in	the	dendrites	last	about	forty	seconds;	the	proteins
that	provide	energy	to	the	synapses	are	replaced	every	hour;	the	NMDA
receptors	in	synapses	are	relatively	long-lived	at	five	days.
So	you	are	completely	replaced	in	a	matter	of	months,	which	is	comparable
to	the	gradual	replacement	scenario	I	describe	above.	Are	you	the	same	person
you	were	a	few	months	ago?	Certainly	there	are	some	differences.	Perhaps	you
learned	a	few	things.	But	you	assume	that	your	identity	persists,	that	you	are	not
continually	destroyed	and	re-created.
Consider	a	river,	like	the	one	that	flows	past	my	office.	As	I	look	out	now	at
what	people	call	the	Charles	River,	is	it	the	same	river	that	I	saw	yesterday?
Let’s	first	reflect	on	what	a	river	is.	The	dictionary	defines	it	is	“a	large	natural
stream	of	flowing	water.”	By	that	definition,	the	river	I’m	looking	at	is	a
completely	different	one	than	it	was	yesterday.	Every	one	of	its	water	molecules

has	changed,	a	process	that	happens	very	quickly.	Greek	philosopher	Diogenes
Laertius	wrote	in	the	third	century	AD	that	“you	cannot	step	into	the	same	river
twice.”
But	that	is	not	how	we	generally	regard	rivers.	People	like	to	look	at	them
because	they	are	symbols	of	continuity	and	stability.	By	the	common	view,	the
Charles	River	that	I	looked	at	yesterday	is	the	same	river	I	see	today.	Our	lives
are	much	the	same.	Fundamentally	we	are	not	the	stuff	that	makes	up	our	bodies
and	brains.	These	particles	essentially	flow	through	us	in	the	same	way	that

twice.”
water	molecules	flow	through	a	river.	We	are	a	pattern	that	changes	slowly	but
has	stability	and	continuity,	even	though	the	stuff	constituting	the	pattern
changes	quickly.
The	gradual	introduction	of	nonbiological	systems	into	our	bodies	and
brains	will	be	just	another	example	of	the	continual	turnover	of	parts	that
compose	us.	It	will	not	alter	the	continuity	of	our	identity	any	more	than	the
natural	replacement	of	our	biological	cells	does.	We	have	already	largely

outsourced	our	historical,	intellectual,	social,	and	personal	memories	to	our
devices	and	the	cloud.	The	devices	we	interact	with	to	access	these	memories
may	not	yet	be	inside	our	bodies	and	brains,	but	as	they	become	smaller	and
smaller	(and	we	are	shrinking	technology	at	a	rate	of	about	a	hundred	in	3-D
volume	per	decade),	they	will	make	their	way	there.	In	any	event,	it	will	be	a
useful	place	to	put	them—we	won’t	lose	them	that	way.	If	people	do	opt	out	of
placing	microscopic	devices	inside	their	bodies,	that	will	be	fine,	as	there	will	be

other	ways	to	access	the	pervasive	cloud	intelligence.
But	we	come	back	to	the	dilemma	I	introduced	earlier.	You,	after	a	period
of	gradual	replacement,	are	equivalent	to	You	2	in	the	scan-and-instantiate
scenario,	but	we	decided	that	You	2	in	that	scenario	does	not	have	the	same

identity	as	you.	So	where	does	that	leave	us?
It	leaves	us	with	an	appreciation	of	a	capability	that	nonbiological	systems
have	that	biological	systems	do	not:	the	ability	to	be	copied,	backed	up,	and	re-
created.	We	do	that	routinely	with	our	devices.	When	we	get	a	new	smartphone,
we	copy	over	all	of	our	files,	so	it	has	much	the	same	personality,	skills,	and
memories	that	the	old	smartphone	did.	Perhaps	it	also	has	some	new	capabilities,
but	the	contents	of	the	old	phone	are	still	with	us.	Similarly,	a	program	such	as
Watson	is	certainly	backed	up.	If	the	Watson	hardware	were	destroyed
tomorrow,	Watson	would	easily	be	re-created	from	its	backup	files	stored	in	the
cloud.
This	represents	a	capability	in	the	nonbiological	world	that	does	not	exist	in
the	biological	world.	It	is	an	advantage,	not	a	limitation,	which	is	one	reason
why	we	are	so	eager	today	to	continue	uploading	our	memories	to	the	cloud.	We
will	certainly	continue	in	this	direction,	as	nonbiological	systems	attain	more	and

cloud.
more	of	the	capabilities	of	our	biological	brains.
My	resolution	of	the	dilemma	is	this:	It	is	not	true	that	You	2	is	not	you—it
is
	you.	It	is	just	that	there	are	now	two	of	you.	That’s	not	so	bad—if	you	think
you	are	a	good	thing,	then	two	of	you	is	even	better.
What	I	believe	will	actually	happen	is	that	we	will	continue	on	the	path	of
the	gradual	replacement	and	augmentation	scenario	until	ultimately	most	of	our
thinking	will	be	in	the	cloud.	My	leap	of	faith	on	identity	is	that	identity	is

is
preserved	through	continuity	of	the	pattern	of	information	that	makes	us	us.
Continuity	does	allow	for	continual	change,	so	whereas	I	am	somewhat	different
than	I	was	yesterday,	I	nonetheless	have	the	same	identity.	However,	the
continuity	of	the	pattern	that	constitutes	my	identity	is	not	substrate-dependent.
Biological	substrates	are	wonderful—they	have	gotten	us	very	far—but	we	are
creating	a	more	capable	and	durable	substrate	for	very	good	reasons.

THE	LAW	OF
ACCELERATING	RETURNS
APPLIED	TO	THE	BRAIN
	
And	though	man	should	remain,	in	some	respects,	the	higher	creature,	is	not
this	in	accordance	with	the	practice	of	nature,	which	allows	superiority	in
some	things	to	animals	which	have,	on	the	whole,	been	long	surpassed?	Has
she	not	allowed	the	ant	and	the	bee	to	retain	superiority	over	man	in	the
organization	of	their	communities	and	social	arrangements,	the	bird	in
traversing	the	air,	the	fish	in	swimming,	the	horse	in	strength	and	fleetness,
and	the	dog	in	self-sacrifice?
	
There	was	a	time,	when	the	earth	was	to	all	appearance	utterly	destitute
both	of	animal	and	vegetable	life,	and	when	according	to	the	opinion	of	our
best	philosophers	it	was	simply	a	hot	round	ball	with	a	crust	gradually
cooling.	Now	if	a	human	being	had	existed	while	the	earth	was	in	this	state
and	had	been	allowed	to	see	it	as	though	it	were	some	other	world	with
which	he	had	no	concern,	and	if	at	the	same	time	he	were	entirely	ignorant

of	all	physical	science,	would	he	not	have	pronounced	it	impossible	that
creatures	possessed	of	anything	like	consciousness	should	be	evolved	from
the	seeming	cinder	which	he	was	beholding?	Would	he	not	have	denied	that
it	contained	
any	potentiality	of	consciousness?	Yet	in	the	course	of	time
consciousness	came.	Is	it	not	possible	then	that	there	may	be	even	yet	new
channels	dug	out	for	consciousness,	though	we	can	detect	no	signs	of	them
at	present?

When	we	reflect	upon	the	manifold	phases	of	life	and	consciousness	which

have	been	evolved	already,	it	would	be	rash	to	say	that	no	others	can	be
developed,	and	that	animal	life	is	the	end	of	all	things.	There	was	a	time
when	fire	was	the	end	of	all	things:	another	when	rocks	and	water	were	so.
	
There	is	no	security	against	the	ultimate	development	of	mechanical
consciousness,	in	the	fact	of	machines	possessing	little	consciousness	now.
A	mollusk	has	not	much	consciousness.	Reflect	upon	the	extraordinary
advance	which	machines	have	made	during	the	last	few	hundred	years,	and
note	how	slowly	the	animal	and	vegetable	kingdoms	are	advancing.	The
more	highly	organized	machines	are	creatures	not	so	much	of	yesterday,	as
of	the	last	five	minutes,	so	to	speak,	in	comparison	with	past	time.	Assume
for	the	sake	of	argument	that	conscious	beings	have	existed	for	some
twenty	million	years:	see	what	strides	machines	have	made	in	the	last
thousand!	May	not	the	world	last	twenty	million	years	longer?	If	so,	what
will	they	not	in	the	end	become?
	
M

M
y	core	thesis,	which	I	call	the	law	of	accelerating	returns	(LOAR),	is	that
fundamental	measures	of	information	technology	follow	predictable	and
exponential	trajectories,	belying	the	conventional	wisdom	
that	“you	can’t	predict
the	future.”	There	are	still	many	things—which	project,	company,	or	technical
standard	will	prevail	in	the	marketplace,	when	peace	will	come	to	the	Middle
East—that	remain	unknowable,	but	the	underlying	price/performance	and
capacity	of	information	has	nonetheless	proven	to	be	remarkably	predictable.

Surprisingly,	these	trends	are	unperturbed	by	conditions	such	as	war	or	peace
and	prosperity	or	recession.
A	primary	reason	that	evolution	created	brains	was	to	predict	the	future.	As
one	of	our	ancestors	walked	through	the	savannas	thousands	of	years	ago,	she
might	have	noticed	that	an	animal	was	progressing	toward	a	route	that	she	was
taking.	She	would	predict	that	if	she	stayed	on	course,	their	paths	would
intersect.	Based	on	this,	she	decided	to	head	in	another	direction,	and	her
foresight	proved	valuable	to	survival.

But	such	built-in	predictors	of	the	future	are	linear,	not	exponential,	a
quality	that	stems	from	the	linear	organization	of	the	neocortex.	Recall	that	the
neocortex	is	constantly	making	predictions—what	letter	and	word	we	will	see

next,	whom	we	expect	to	see	as	we	round	the	corner,	and	so	on.	The	neocortex	is
organized	with	linear	sequences	of	steps	in	each	pattern,	which	means	that
exponential	thinking	does	not	come	naturally	to	us.	The	cerebellum	also	uses
linear	predictions.	When	it	helps	us	to	catch	a	fly	ball	it	is	making	a	linear
prediction	about	where	the	ball	will	be	in	our	visual	field	of	view	and	where	our
gloved	hand	should	be	in	our	visual	field	of	view	to	catch	it.
As	I	have	pointed	out,	there	is	a	dramatic	difference	between	linear	and
exponential	progressions	(forty	steps	linearly	is	forty,	but	exponentially	is	a
trillion),	which	accounts	for	why	my	predictions	stemming	from	the	law	of
accelerating	returns	seem	surprising	to	many	observers	at	first.	We	have	to	train
ourselves	to	think	exponentially.	When	it	comes	to	information	technologies,	it	is
the	right	way	to	think.
The	quintessential	example	of	the	law	of	accelerating	returns	is	the
perfectly	smooth,	doubly	exponential	growth	of	the	price/performance	of

computation,	which	has	held	steady	for	110	years	through	two	world	wars,	
the
Great	Depression,	the	Cold	War,	the	collapse	of	the	Soviet	Union,	the
reemergence	of	China,	the	recent	financial	crisis,	and	all	of	the	other	notable
events	of	the	late	nineteenth,	twentieth,	and	early	twenty-first	centuries.	Some
people	refer	to	this	phenomenon	as	“Moore’s	law,”	but	that	is	a	misconception.
Moore’s	law—which	states	that	you	can	place	twice	as	many	components	on	an

the
integrated	circuit	every	two	years,	and	they	run	faster	because	they	are	smaller—
is	just	one	paradigm	among	many.	It	was	in	fact	the	fifth,	not	the	first,	paradigm
to	bring	exponential	growth	to	the	price/performance	of	computing.
The	exponential	rise	of	computation	started	with	the	1890	U.S.	census	(the
first	to	be	automated)	using	the	first	paradigm	of	electromechanical	calculation,
decades	before	Gordon	Moore	was	even	born.	In	
	I
provide	this	graph	through	2002,	and	here	I	update	it	through	2009	(see	the
graph	on	
page	257

	I
graph	on	
page	257
	titled	“Exponential	Growth	of	Computing	for	110	Years”).
The	smoothly	predictable	trajectory	has	continued,	even	through	the	recent
economic	downturn.
Computation	is	the	most	important	example	of	the	law	of	accelerating
returns,	because	of	the	amount	of	data	we	have	for	it,	the	ubiquity	of
computation,	and	its	key	role	in	ultimately	revolutionizing	everything	we	care
about.	But	it	is	far	from	the	only	example.	Once	a	technology	becomes	an
information	technology,	it	becomes	subject	to	the	LOAR.

page	257
Biomedicine	is	becoming	the	most	significant	recent	area	of	technology	and
industry	to	be	transformed	in	this	way.	Progress	in	medicine	has	historically	been
based	on	accidental	discoveries,	so	progress	during	the	earlier	era	was	linear,	not
exponential.	This	has	nevertheless	been	beneficial:	Life	expectancy	has	grown

from	twenty-three	years	as	of	a	thousand	years	ago,	to	thirty-seven	years	as	of
two	hundred	years	ago,	to	close	to	eighty	years	today.	With	the	gathering	of	the
software	of	life—the	genome—medicine	and	human	biology	have	become	an
information	technology.	The	human	genome	project	itself	was	perfectly
exponential,	with	the	amount	of	genetic	data	doubling	and	the	cost	per	base	pair
coming	down	by	half	each	year	since	the	project	was	initiated	in	1990.
3
	(All	the
graphs	in	this	chapter	have	been	updated	since	
	was
published.)
The	cost	of	sequencing	a	human-sized	genome.
1

The	amount	of	genetic	data	sequenced	in	the	world	each	year.
2
We	now	have	the	ability	to	design	biomedical	interventions	on	computers
and	to	test	them	on	biological	simulators,	the	scale	and	precision	of	which	are
also	doubling	every	year.	We	can	also	update	our	own	obsolete	software:	RNA
interference	can	turn	genes	off,	and	new	forms	of	gene	therapy	can	add	new
genes,	not	just	to	a	newborn	but	to	a	mature	individual.	The	advance	of	genetic
technologies	also	affects	the	brain	reverse-engineering	project,	in	that	one
important	aspect	of	it	is	understanding	how	genes	control	brain	functions	such	as
creating	new	connections	to	reflect	recently	added	cortical	knowledge.	There	are
many	other	manifestations	of	this	integration	of	biology	and	information
technology,	as	we	move	beyond	genome	sequencing	to	genome	synthesizing.
Another	information	technology	that	has	seen	smooth	exponential	growth	is
our	ability	to	communicate	with	one	another	and	transmit	vast	repositories	of

human	knowledge.	There	are	many	ways	to	measure	this	phenomenon.	Cooper’s
law,	which	states	that	the	total	bit	capacity	of	wireless	communications	in	a
given	amount	of	radio	spectrum	doubles	every	thirty	months,	has	held	true	from
the	time	Guglielmo	Marconi	used	the	wireless	telegraph	for	Morse	code
transmissions	in	1897	to	today’s	4G	communications	technologies.
4
	According
to	Cooper’s	law,	the	amount	of	information	that	can	be	transmitted	over	a	given
amount	of	radio	spectrum	has	been	doubling	every	two	and	a	half	years	for	more

4
	According
than	a	century.	Another	example	is	the	number	of	bits	per	second	transmitted	on
the	Internet,	which	is	doubling	every	one	and	a	quarter	years.
5
The	reason	I	became	interested	in	trying	to	predict	certain	aspects	of
technology	is	that	I	realized	about	thirty	years	ago	that	the	key	to	becoming

successful	as	an	inventor	(a	profession	I	adopted	when	I	was	five	years	old)	was
timing.	Most	inventions	and	inventors	fail	not	because	the	gadgets	themselves
don’t	work,	but	because	their	timing	is	wrong,	appearing	either	before	all	of	the
enabling	factors	are	in	place	or	too	late,	having	missed	the	window	of
The	international	(country-to-country)	bandwidth	dedicated	to	the
Internet	for	the	world.
6

The	highest	bandwidth	(speed)	of	the	Internet	backbone.
7
Being	an	engineer,	about	three	decades	ago	I	started	to	gather	data	on
measures	of	technology	in	different	areas.	When	I	began	this	effort,	I	did	not
expect	that	it	would	present	a	clear	picture,	but	I	did	hope	that	it	would	provide
some	guidance	and	enable	me	to	make	educated	guesses.	My	goal	was—and	still
is—to	time	my	own	technology	efforts	so	that	they	will	be	appropriate	for	the
world	that	exists	when	I	complete	a	project—which	I	realized	would	be	very
different	from	the	world	that	existed	when	I	started.
Consider	how	much	and	how	quickly	the	world	has	changed	only	recently.
Just	a	few	years	ago,	people	did	not	use	social	networks	(Facebook,	for	example,
was	founded	in	2004	and	had	901	million	monthly	active	users	at	the	end	of
March	2012),
8
	wikis,	blogs,	or	tweets.	In	the	1990s	most	people	did	not	use
search	engines	or	cell	phones.	Imagine	the	world	without	them.	That	seems	like

8
ancient	history	but	was	not	so	long	ago.	The	world	will	change	even	more
dramatically	in	the	near	future.
In	the	course	of	my	investigation,	I	made	a	startling	discovery:	If	a
technology	is	an	information	technology,	the	basic	measures	of
price/performance	and	capacity	(per	unit	of	time	or	cost,	or	other	resource)
follow	amazingly	precise	exponential	trajectories.
These	trajectories	outrun	the	specific	paradigms	they	are	based	on	(such	as
Moore’s	law).	But	when	one	paradigm	runs	out	of	steam	(for	example,	when

engineers	were	no	longer	able	to	reduce	the	size	and	cost	of	vacuum	tubes	in	the
1950s),	it	creates	research	pressure	to	create	the	next	paradigm,	and	so	another
S-curve	of	progress	begins.

The	exponential	portion	of	that	next	S-curve	for	the	new	paradigm	then
continues	the	ongoing	exponential	of	the	information	technology	measure.	Thus
vacuum	tube–based	computing	in	the	1950s	gave	way	to	transistors	in	the	1960s,
and	then	to	integrated	circuits	and	Moore’s	law	in	the	late	1960s,	and	beyond.
Moore’s	law,	in	turn,	will	give	way	to	three-dimensional	computing,	the	early
examples	of	which	are	already	in	place.	The	reason	why	information
technologies	are	able	to	consistently	transcend	the	limitations	of	any	particular
paradigm	is	that	the	resources	required	to	compute	or	remember	or	transmit	a	bit
of	information	are	vanishingly	small.
We	might	wonder,	are	there	fundamental	limits	to	our	ability	to	compute
and	transmit	information,	regardless	of	paradigm?	The	answer	is	yes,	based	on
our	current	understanding	of	the	physics	of	computation.	Those	limits,	however,
are	not	very	limiting.	Ultimately	we	can	expand	our	intelligence	trillions-fold

based	on	molecular	computing.	By	my	calculations,	we	will	reach	these	limits
late	in	this	century.
It	is	important	to	point	out	that	not	every	exponential	phenomenon	is	an
example	of	the	law	of	accelerating	returns.	Some	observers	misconstrue	the
LOAR	by	citing	exponential	trends	that	are	not	information-based:	For	example,
they	point	out,	men’s	shavers	have	gone	from	one	blade	to	two	to	four,	and	then
ask,	where	are	the	eight-blade	shavers?	Shavers	are	not	(yet)	an	information
In

In	
,	I	provide	a	theoretical	examination,	including
(in	the	appendix	to	that	book)	a	mathematical	treatment	of	why	the	LOAR	is	so
remarkably	predictable.	Essentially,	we	always	use	the	latest	technology	to
create	the	next.	Technologies	build	on	themselves	in	an	exponential	manner,	and
this	phenomenon	is	readily	measurable	if	it	involves	an	information	technology.
In	1990	we	used	the	computers	and	other	tools	of	that	era	to	create	the

In	
computers	of	1991;	in	2012	we	are	using	current	information	tools	to	create	the
machines	of	2013	and	2014.	More	broadly	speaking,	this	acceleration	and
exponential	growth	applies	to	any	process	in	which	patterns	of	information
evolve.	So	we	see	acceleration	in	the	pace	of	biological	evolution,	and	similar
(but	much	faster)	acceleration	in	technological	evolution,	which	is	itself	an
outgrowth	of	biological	evolution.
I	now	have	a	public	track	record	of	more	than	a	quarter	of	a	century	of

predictions	based	on	the	law	of	accelerating	returns,	starting	with	those
presented	in	
,	which	I	wrote	in	the	mid-1980s.
Examples	of	accurate	predictions	from	that	book	include:	the	emergence	in	the
mid-to	late	1990s	of	a	vast	worldwide	web	of	communications	tying	together
people	around	the	world	to	one	another	and	to	all	human	knowledge;	a	great

wave	of	democratization	emerging	from	this	decentralized	
communication
network,	sweeping	away	the	Soviet	Union;	the	defeat	of	the	world	chess
champion	by	1998;	and	many	others.
I	described	the	law	of	accelerating	returns,	as	it	is	applied	to	computation,
extensively	in	
The	Age	of	Spiritual	Machines
,	where	I	provided	a	century	of	data
showing	the	doubly	exponential	progression	of	the	price/performance	of
computation	through	1998.	It	is	updated	through	2009	below.
I	recently	wrote	a	146-page	review	of	the	predictions	I	made	in	
The	Age	of
Intelligent	Machines,	The	Age	of	Spiritual	Machines
,	and	
The	Singularity	Is
Near
.	(You	can	read	the	essay	here	by	going	to	the	link	in	this	endnote.)
9
The	Age
of	Spiritual	Machines
	included	hundreds	of	predictions	for	specific	decades
(2009,	2019,	2029,	and	2099).	For	example,	I	made	147	predictions	for	2009	in
,	which	I	wrote	in	the	1990s.	Of	these,	115	(78
percent)	are	entirely	correct	as	of	the	end	of	2009;	the	predictions	that	were

The	Age	of
,	and	
Near
9
The	Age
concerned	with	basic	measurements	of	the	capacity	and	price/performance	of
information	technologies	were	particularly	accurate.	Another	12	(8	percent)	are
“essentially	correct.”	A	total	of	127	predictions	(86	percent)	are	correct	or
essentially	correct.	(Since	the	predictions	were	made	specific	to	a	given	decade,
a	prediction	for	2009	was	considered	“essentially	correct”	if	it	came	true	in	2010
or	2011.)	Another	17	(12	percent)	are	partially	correct,	and	3	(2	percent)	are
wrong.

Calculations	per	second	per	(constant)	thousand	dollars	of	different
computing	devices.
10
Floating-point	operations	per	second	of	different	supercomputers.
11

Transistors	per	chip	for	different	Intel	processors.
12
Bits	per	dollar	for	dynamic	random	access	memory	chips.
13

Bits	per	dollar	for	random	access	memory	chips.
14
The	average	price	per	transistor	in	dollars.
15

The	total	number	of	bits	of	random	access	memory	shipped	each	year.
16
Bits	per	dollar	(in	constant	2000	dollars)	for	magnetic	data	storage.
17
Even	the	predictions	that	were	“wrong”	were	not	all	wrong.	For	example,	I
judged	my	prediction	that	we	would	have	self-driving	cars	to	be	wrong,	even
though	Google	has	demonstrated	self-driving	cars,	and	even	though	in	October
2010	four	driverless	electric	vans	successfully	concluded	a	13,000-kilometer	test
drive	from	Italy	to	China.
18
	Experts	in	the	field	currently	predict	that	these
technologies	will	be	routinely	available	to	consumers	by	the	end	of	this	decade.
Exponentially	expanding	computational	and	communication	technologies

all	contribute	to	the	project	to	understand	and	re-create	the	
methods	of	the
human	brain.	This	effort	is	not	a	single	organized	project	but	rather	the	result	of
a	great	many	diverse	projects,	including	detailed	modeling	of	constituents	of	the
brain	ranging	from	individual	neurons	to	the	entire	neocortex,	the	mapping	of
the	“connectome”	(the	neural	connections	in	the	brain),	simulations	of	brain
regions,	and	many	others.	All	of	these	have	been	scaling	up	exponentially.	Much
of	the	evidence	presented	in	this	book	has	only	become	available	recently—for
example,	the	2012	Wedeen	study	discussed	in	
chapter	4
	that	showed	the	very
orderly	and	“simple”	(to	quote	the	researchers)	gridlike	pattern	of	the
connections	in	the	neocortex.	The	researchers	in	that	study	acknowledge	that
their	insight	(and	images)	only	became	feasible	as	the	result	of	new	high-
resolution	imaging	technology.
Brain	scanning	technologies	are	improving	in	resolution,	spatial	and

chapter	4
temporal,	at	an	exponential	rate.	Different	types	of	brain	scanning	methods	being
pursued	range	from	completely	noninvasive	methods	that	can	be	used	with
humans	to	more	invasive	or	destructive	methods	on	animals.
MRI	(magnetic	resonance	imaging),	a	noninvasive	imaging	technique	with
relatively	high	temporal	resolution,	has	steadily	improved	at	an	exponential
pace,	to	the	point	that	spatial	resolutions	are	now	close	to	100	microns
(millionths	of	a	meter).
A	Venn	diagram	of	brain	imaging	methods.
19

Tools	for	imaging	the	brain.
20
MRI	spatial	resolution	in	microns.
21

Spatial	resolution	of	destructive	imaging	techniques.
22
Spatial	resolution	of	nondestructive	imaging	techniques	in	animals.
23
Destructive	imaging,	which	is	performed	to	collect	the	connectome	(map	of
all	interneuronal	connections)	in	animal	brains,	has	also	improved	at	an
exponential	pace.	Current	maximum	resolution	is	around	four	nanometers,
which	is	sufficient	to	see	individual	connections.
Artificial	intelligence	technologies	such	as	natural-language-understanding

systems	are	not	necessarily	designed	to	emulate	theorized	principles	of	brain
function,	but	rather	for	maximum	effectiveness.	Given	this,	it	is	notable	that	the
techniques	that	have	won	out	are	consistent	with	the	principles	I	have	outlined	in
this	book:	self-organizing,	hierarchical	recognizers	of	invariant	self-associative
patterns	with	redundancy	and	up-and-down	predictions.	These	systems	are	also
scaling	up	exponentially,	as	Watson	has	demonstrated.
A	primary	purpose	of	understanding	the	brain	is	to	expand	our	toolkit	of
techniques	to	create	intelligent	systems.	Although	many	AI	researchers	may	not
fully	appreciate	this,	they	have	already	been	deeply	influenced	by	our	knowledge
of	the	principles	of	the	operation	of	the	brain.	Understanding	the	brain	also	helps
us	to	reverse	brain	dysfunctions	of	various	kinds.	There	is,	of	course,	another
key	goal	of	the	project	to	reverse-engineer	the	brain:	understanding	who	we	are.

OBJECTIONS
	
If	a	machine	can	prove	indistinguishable	from	a	human,	we	should	award	it
the	respect	we	would	to	a	human—we	should	accept	that	it	has	a	mind.
—Stevan	Harnad
	
T
	he	most	significant	source	of	objection	to	my	thesis	on	the	law	of	accelerating
returns	and	its	application	to	the	amplification	of	human	intelligence	stems	from
the	linear	nature	of	human	intuition.	As	I	described	earlier,	each	of	the	several
hundred	million	pattern	recognizers	in	the	neocortex	processes	information
sequentially.	One	of	the	implications	of	this	organization	is	that	we	have	linear
expectations	about	the	future,	so	critics	apply	their	linear	intuition	to	information
phenomena	that	are	fundamentally	exponential.
I	call	objections	along	these	lines	“criticism	from	incredulity,”	in	that
exponential	projections	seem	incredible	given	our	linear	predilection,	and	they
take	a	variety	of	forms.	Microsoft	cofounder	Paul	Allen	(born	in	1953)	and	his
colleague	Mark	Greaves	recently	articulated	several	of	them	in	an	essay	titled

“The	Singularity	Isn’t	Near”	published	in	
	magazine.
1
	While
my	response	here	is	to	Allen’s	particular	critiques,	they	
represent	a	typical	range
of	objections	to	the	arguments	I’ve	made,	especially	with	regard	to	the	brain.
Although	Allen	references	
	in	the	title	of	his	essay,	his
only	citation	in	the	piece	is	to	an	essay	I	wrote	in	2001	(“The	Law	of
Accelerating	Returns”).	Moreover,	his	article	does	not	acknowledge	or	respond
to	arguments	I	actually	make	in	the	book.	Unfortunately,	I	find	this	often	to	be

magazine.
1
	While
the	case	with	critics	of	my	work.
When	
	was	published	in	1999,	augmented
later	by	the	2001	essay,	it	generated	several	lines	of	criticism,	such	as:	
Moore’s
law	will	come	to	an	end;	hardware	capability	may	be	expanding	exponentially
but	software	is	stuck	in	the	mud;	the	brain	is	too	complicated;	there	are
capabilities	in	the	brain	that	inherently	cannot	be	replicated	in	software;
	and

several	others.	One	of	the	reasons	I	wrote	
	was	to
respond	to	those	critiques.
I	cannot	say	that	Allen	and	similar	critics	would	necessarily	have	been
convinced	by	the	arguments	I	made	in	that	book,	but	at	least	he	and	others	could
have	responded	to	what	I	actually	wrote.	Allen	argues	that	“the	Law	of
Accelerating	Returns	(LOAR)…is	not	a	physical	law.”	I	would	point	out	that
most	scientific	laws	are	not	physical	laws,	but	result	from	the	emergent
properties	of	a	large	number	of	events	at	a	lower	level.	A	classic	example	is	the
laws	of	thermodynamics	(LOT).	If	you	look	at	the	mathematics	underlying	the
LOT,	it	models	each	particle	as	following	a	random	walk,	so	by	definition	we
cannot	predict	where	any	particular	particle	will	be	at	any	future	time.	Yet	the
overall	properties	of	the	gas	are	quite	predictable	to	a	high	degree	of	precision,
according	to	the	
laws
	of	thermodynamics.	So	it	is	with	the	law	of	accelerating

laws
returns:	Each	technology	project	and	contributor	is	unpredictable,	yet	the	overall
trajectory,	as	quantified	by	basic	measures	of	price/performance	and	capacity,
nonetheless	follows	a	remarkably	predictable	path.
If	computer	technology	were	being	pursued	by	only	a	handful	of
researchers,	it	would	indeed	be	unpredictable.	But	it’s	the	product	of	a
sufficiently	dynamic	system	of	competitive	projects	that	a	basic	measure	of	its
price/performance,	such	as	calculations	per	second	per	constant	dollar,	
follows	a

follows	a
very	smooth	exponential	path,	dating	back	to	the	1890	American	census	as	I
noted	in	the	
.	While	the	theoretical	basis	for	the	LOAR	is
presented	extensively	in	
,	the	strongest	case	for	it	is	made
by	the	extensive	empirical	evidence	that	I	and	others	present.
Allen	writes	that	“these	‘laws’	work	until	they	don’t.”	Here	he	is	confusing
paradigms	with	the	ongoing	trajectory	of	a	basic	area	of	information	technology.

follows	a
If	we	were	examining,	for	example,	the	trend	of	creating	ever	smaller	vacuum
tubes—the	paradigm	for	improving	computation	in	the	1950s—it’s	true	that	it
continued	until	it	didn’t.	But	as	the	end	of	this	particular	paradigm	became	clear,
research	pressure	grew	for	the	next	paradigm.	The	technology	of	transistors	kept
the	underlying	trend	of	the	exponential	growth	of	price/performance	of
computation	going,	and	that	led	to	the	fifth	paradigm	(Moore’s	law)	and	the
continual	compression	of	features	on	integrated	circuits.	There	have	been	regular

predictions	that	Moore’s	law	will	come	to	an	end.	The	semiconductor	industry’s
“International	Technology	Roadmap	for	Semiconductors”	projects	seven-
nanometer	features	by	the	early	2020s.
2
	At	that	point	key	features	will	be	the
width	of	thirty-five	carbon	atoms,	and	it	will	be	difficult	to	continue	shrinking
them	any	farther.	However,	Intel	and	other	chip	makers	are	already	taking	the
first	steps	toward	the	sixth	paradigm,	computing	in	three	dimensions,	to	continue

exponential	improvement	in	price/performance.	Intel	projects	that	three-
dimensional	chips	will	be	mainstream	by	the	teen	years;	three-dimensional
transistors	and	3-D	memory	chips	have	already	been	introduced.	This	sixth
paradigm	will	keep	the	LOAR	going	with	regard	to	computer	price/performance
to	a	time	later	in	this	century	when	a	thousand	dollars’	worth	of	computation	will
be	trillions	of	times	more	powerful	than	the	human	brain.
3
	(It	appears	that	Allen
and	I	are	at	least	in	agreement	on	what	level	of	computation	is	required	to
functionally	simulate	the	human	brain.)
4
Allen	then	goes	on	to	give	the	standard	argument	that	software	is	not
progressing	in	the	same	exponential	manner	as	hardware.	In	
Near
	I	addressed	this	issue	at	length,	citing	different	methods	of	
measuring
complexity	and	capability	in	software	that	do	demonstrate	a	similar	exponential
growth.
5
	One	recent	study	(“Report	to	the	President	and	Congress,	Designing	a

4
Near
measuring
growth.
5
Digital	Future:	Federally	Funded	Research	and	Development	in	Networking	and
Information	Technology,”	by	the	President’s	Council	of	Advisors	on	Science	and
Technology)	states	the	following:
Even	more	remarkable—and	even	less	widely	understood—is	that	in
many	areas,	
performance	gains	due	to	improvements	in	algorithms	have
vastly	exceeded	even	the	dramatic	performance	gains	due	to	increased
processor	speed
.	The	algorithms	that	we	use	today	for	speech	recognition,

for	natural	language	translation,	for	chess	playing,	for	logistics	planning,
have	evolved	remarkably	in	the	past	decade….	Here	is	just	one	example,
provided	by	Professor	Martin	Grötschel	of	Konrad-Zuse-Zentrum	für
Informationstechnik	Berlin.	Grötschel,	an	expert	in	optimization,	observes
that	a	benchmark	production	planning	model	solved	using	linear
programming	would	have	taken	82	years	to	solve	in	1988,	using	the
computers	and	the	linear	programming	algorithms	of	the	day.	Fifteen	years

later—in	2003—this	same	model	could	be	solved	in	roughly	1	minute,	an
improvement	by	a	factor	of	roughly	43	million.	Of	this,	a	factor	of	roughly
1,000	was	due	to	increased	processor	speed,	whereas	a	factor	of	roughly
43,000	was	due	to	improvements	in	algorithms!	Grötschel	also	cites	an
algorithmic	improvement	of	roughly	30,000	for	mixed	integer
programming	between	1991	and	2008.	The	design	and	analysis	of
algorithms,	and	the	study	of	the	inherent	computational	complexity	of

problems,	are	fundamental	subfields	of	computer	science.
	
Note	that	the	linear	programming	that	Grötschel	cites	above	as	having
benefited	from	an	improvement	in	performance	of	43	million	to	1	is	the

mathematical	technique	that	is	used	to	optimally	assign	resources	in	a
hierarchical	memory	system	such	as	HHMM	that	I	discussed	earlier.	I	cite	many
other	similar	examples	like	this	in	
.
6
Regarding	AI,	Allen	is	quick	to	dismiss	IBM’s	Watson,	an	opinion	shared
by	many	other	critics.	Many	of	these	detractors	don’t	know	anything	about
Watson	other	than	the	fact	that	it	is	software	running	on	a	computer	(albeit	a
parallel	one	with	720	processor	cores).	Allen	writes	that	systems	such	as	Watson
“remain	brittle,	their	performance	boundaries	are	rigidly	set	by	their	internal
assumptions	and	defining	algorithms,	they	cannot	generalize,	and	they
frequently	give	nonsensical	answers	outside	of	their	specific	areas.”
First	of	all,	we	could	make	a	similar	observation	about	humans.	I	would
also	point	out	that	Watson’s	“specific	areas”	include	
all
	of	Wikipedia	plus	many
other	knowledge	bases,	which	hardly	constitute	a	narrow	focus.	Watson	deals

all
with	a	vast	range	of	human	knowledge	and	is	capable	of	dealing	with	subtle
forms	of	language,	including	puns,	similes,	and	metaphors	in	virtually	all	fields
of	human	endeavor.	It’s	not	perfect,	but	neither	are	humans,	and	it	was	good
enough	to	be	victorious	on	
Jeopardy!
	over	the	best	human	players.
Allen	argues	that	Watson	was	assembled	by	the	scientists	themselves,
building	each	link	of	narrow	knowledge	in	specific	areas.	This	is	simply	not	true.
Although	a	few	areas	of	Watson’s	data	were	programmed	directly,	Watson

Jeopardy!
acquired	the	significant	majority	of	its	knowledge	on	its	own	by	reading	natural-
language	documents	such	as	Wikipedia.	That	represents	its	key	strength,	as	does
its	ability	to	understand	the	convoluted	language	in	
Jeopardy!
	queries	(answers
in	search	of	a	question).
As	I	mentioned	earlier,	much	of	the	criticism	of	Watson	is	that	it	works
through	statistical	probabilities	rather	than	“true”	understanding.	Many	readers
interpret	this	to	mean	that	Watson	is	merely	gathering	statistics	on	word

Jeopardy!
sequences.	The	term	“statistical	information”	in	the	case	of	Watson	actually
refers	to	distributed	coefficients	and	symbolic	connections	in	self-organizing
methods	such	as	hierarchical	hidden	Markov	models.	One	could	just	as	easily
dismiss	the	distributed	neurotransmitter	concentrations	and	redundant
connection	patterns	in	the	human	cortex	as	“statistical	information.”	Indeed	we
resolve	ambiguities	in	much	the	same	way	
that	Watson	does—by	considering	the
likelihood	of	different	interpretations	of	a	phrase.

Allen	continues,	“Every	structure	[in	the	brain]	has	been	precisely	shaped
by	millions	of	years	of	evolution	to	do	a	particular	thing,	whatever	it	might	be.	It
is	not	like	a	computer,	with	billions	of	identical	transistors	in	regular	memory
arrays	that	are	controlled	by	a	CPU	with	a	few	different	elements.	In	the	brain
every	individual	structure	and	neural	circuit	has	been	individually	refined	by

evolution	and	environmental	factors.”
This	contention	that	every	structure	and	neural	circuit	in	the	brain	is	unique
and	there	by	design	is	simply	impossible,	for	it	would	mean	that	the	blueprint	of
the	brain	would	require	hundreds	of	trillions	of	bytes	of	information.	The	brain’s
structural	plan	(like	that	of	the	rest	of	the	body)	is	contained	in	the	genome,	and
the	brain	itself	cannot	contain	more	design	information	than	the	genome.	Note
that	epigenetic	information	(such	as	the	peptides	controlling	gene	expression)
does	not	appreciably	add	to	the	amount	of	information	in	the	genome.
Experience	and	learning	do	add	significantly	to	the	amount	of	information
contained	in	the	brain,	but	the	same	can	be	said	of	AI	systems	like	Watson.	I
show	in	
	that,	after	lossless	compression	(due	to	massive
redundancy	in	the	genome),	the	amount	of	design	information	in	the	genome	is
about	50	million	bytes,	roughly	half	of	which	(that	is,	about	25	million	bytes)
pertains	to	the	brain.
7

show	in	
7
	That’s	not	simple,	but	it	is	a	level	of	complexity	we	can
deal	with	and	represents	less	complexity	than	many	software	systems	in	the
modern	world.	Moreover	much	of	the	brain’s	25	million	bytes	of	genetic	design
information	pertain	to	the	biological	requirements	of	neurons,	not	to	their
information-processing	algorithms.
How	do	we	arrive	at	on	the	order	of	100	to	1,000	trillion	connections	in	the
brain	from	only	tens	of	millions	of	bytes	of	design	information?	Obviously,	the

7
answer	is	through	massive	redundancy.	Dharmendra	Modha,	manager	of
Cognitive	Computing	for	IBM	Research,	writes	that	“neuroanatomists	have	not
found	a	hopelessly	tangled,	arbitrarily	connected	network,	completely
idiosyncratic	to	the	brain	of	each	individual,	
but	instead	a	great	deal	of	repeating
structure	within	an	individual	brain	and	a	great	deal	of	homology	across
species….	The	astonishing	natural	reconfigurability	gives	hope	that	the	core

algorithms	of	neurocomputation	are	independent	of	the	specific	sensory	or	motor
modalities	and	that	much	of	the	observed	variation	in	cortical	structure	across
areas	represents	a	refinement	of	a	canonical	circuit;	it	is	indeed	this	canonical
circuit	we	wish	to	reverse	engineer.”
8
Allen	argues	in	favor	of	an	inherent	“complexity	brake	that	would
necessarily	limit	progress	in	understanding	the	human	brain	and	replicating	its
capabilities,”	based	on	his	notion	that	each	of	the	approximately	100	to	1,000

8
trillion	connections	in	the	human	brain	is	there	by	explicit	design.	His
“complexity	brake”	confuses	the	forest	with	the	trees.	If	you	want	to	understand,
model,	simulate,	and	re-create	a	pancreas,	you	don’t	need	to	re-create	or	simulate
every	organelle	in	every	pancreatic	islet	cell.	You	would	want	instead	to
understand	one	islet	cell,	then	abstract	its	basic	functionality	as	it	pertains	to

insulin	control,	and	then	extend	that	to	a	large	group	of	such	cells.	This
algorithm	is	well	understood	with	regard	to	islet	cells.	There	are	now	artificial
pancreases	that	utilize	this	functional	model	being	tested.	Although	there	is
certainly	far	more	intricacy	and	variation	in	the	brain	than	in	the	massively
repeated	islet	cells	of	the	pancreas,	there	is	nonetheless	massive	repetition	of
functions,	as	I	have	described	repeatedly	in	this	book.
Critiques	along	the	lines	of	Allen’s	also	articulate	what	I	call	the	“scientist’s
pessimism.”	Researchers	working	on	the	next	generation	of	a	technology	or	of
modeling	a	scientific	area	are	invariably	struggling	with	that	immediate	set	of
challenges,	so	if	someone	describes	what	the	technology	will	look	like	in	ten
generations,	their	eyes	glaze	over.	One	of	the	pioneers	of	integrated	circuits	was
recalling	for	me	recently	the	struggles	to	go	from	10-micron	(10,000
nanometers)	feature	sizes	to	5-micron	(5,000	nanometers)	features	over	thirty

years	ago.	The	scientists	were	cautiously	confident	of	reaching	this	goal,	but
when	people	predicted	that	someday	we	would	actually	have	circuitry	with
feature	sizes	under	1	micron	
(1,000	nanometers),	most	of	them,	focused	on	their
own	goal,	thought	that	was	too	wild	to	contemplate.	Objections	were	made
regarding	the	fragility	of	circuitry	at	that	level	of	precision,	thermal	effects,	and
so	on.	Today	Intel	is	starting	to	use	chips	with	22-nanometer	gate	lengths.
We	witnessed	the	same	sort	of	pessimism	with	respect	to	the	Human

Genome	Project.	Halfway	through	the	fifteen-year	effort,	only	1	percent	of	the
genome	had	been	collected,	and	critics	were	proposing	basic	limits	on	how
quickly	it	could	be	sequenced	without	destroying	the	delicate	genetic	structures.
But	thanks	to	the	exponential	growth	in	both	capacity	and	price/performance,	the
project	was	finished	seven	years	later.	The	project	to	reverse-engineer	the	human
brain	is	making	similar	progress.	It	is	only	recently,	for	example,	that	we	have
reached	a	threshold	with	noninvasive	scanning	techniques	so	that	we	can	see

individual	interneuronal	connections	forming	and	firing	in	real	time.	Much	of	the
evidence	I	have	presented	in	this	book	was	dependent	on	such	developments	and
has	only	recently	been	available.
Allen	describes	my	proposal	about	reverse-engineering	the	human	brain	as
simply	scanning	the	brain	to	understand	its	fine	structure	and	then	simulating	an
entire	brain	“bottom	up”	without	comprehending	its	information-processing
methods.	This	is	not	my	proposition.	We	do	need	to	understand	in	detail	how

individual	types	of	neurons	work,	and	then	gather	information	about	how
functional	modules	are	connected.	The	functional	methods	that	are	derived	from
this	type	of	analysis	can	then	guide	the	development	of	intelligent	systems.
Basically,	we	are	looking	for	biologically	inspired	methods	that	can	accelerate
work	in	AI,	much	of	which	has	progressed	without	significant	insight	as	to	how

the	brain	performs	similar	functions.	From	my	own	work	in	speech	recognition,	I
know	that	our	work	was	greatly	accelerated	when	we	gained	insights	as	to	how
the	brain	prepares	and	transforms	auditory	information.
The	way	that	the	massively	redundant	structures	in	the	brain	differentiate	is
through	learning	and	experience.	The	current	state	of	the	art	in	AI	does	in	fact
enable	systems	to	also	learn	from	their	own	experience.	The	
Google	self-driving
cars	learn	from	their	own	driving	experience	as	well	as	from	data	from	Google
cars	driven	by	human	drivers;	Watson	learned	most	of	its	knowledge	by	reading
on	its	own.	It	is	interesting	to	note	that	the	methods	deployed	today	in	AI	have
evolved	to	be	mathematically	very	similar	to	the	mechanisms	in	the	neocortex.
Another	objection	to	the	feasibility	of	“strong	AI”	(artificial	intelligence	at
human	levels	and	beyond)	that	is	often	raised	is	that	the	human	brain	makes
extensive	use	of	analog	computing,	whereas	digital	methods	inherently	cannot

replicate	the	gradations	of	value	that	analog	representations	can	embody.	It	is
true	that	one	bit	is	either	on	or	off,	but	multiple-bit	words	easily	represent
multiple	gradations	and	can	do	so	to	any	desired	degree	of	accuracy.	This	is,	of
course,	done	all	the	time	in	digital	computers.	As	it	is,	the	accuracy	of	analog
information	in	the	brain	(synaptic	strength,	for	example)	is	only	about	one	level
within	256	levels	that	can	be	represented	by	eight	bits.
In	
chapter	9
	I	cited	Roger	Penrose	and	Stuart	Hameroff’s	objection,	which

In	
chapter	9
concerned	microtubules	and	quantum	computing.	Recall	that	they	claim	that	the
microtubule	structures	in	neurons	are	doing	quantum	computing,	and	since	it	is
not	possible	to	achieve	that	in	computers,	the	human	brain	is	fundamentally
different	and	presumably	better.	As	I	argued	earlier,	there	is	no	evidence	that
neuronal	microtubules	are	carrying	out	quantum	computation.	Humans	in	fact	do
a	very	poor	job	of	solving	the	kinds	of	problems	that	a	quantum	computer	would
excel	at	(such	as	factoring	large	numbers).	And	if	any	of	this	proved	to	be	true,

there	would	be	nothing	barring	quantum	computing	from	also	being	used	in	our
John	Searle	is	famous	for	introducing	a	thought	experiment	he	calls	“the
Chinese	room,”	an	argument	I	discuss	in	detail	in	
.
9
	In
short,	it	involves	a	man	who	takes	in	written	questions	in	Chinese	and	then
answers	them.	In	order	to	do	this,	he	uses	an	elaborate	rulebook.	Searle	claims
that	the	man	has	no	true	understanding	of	Chinese	and	is	not	“conscious”	of	the
language	(as	he	does	not	understand	the	questions	or

.
9
	In
language	(as	he	does	not	understand	the	questions	or	
the	answers)	despite	his
apparent	ability	to	answer	questions	in	Chinese.	Searle	compares	this	to	a
computer	and	concludes	that	a	computer	that	could	answer	questions	in	Chinese
(essentially	passing	a	Chinese	Turing	test)	would,	like	the	man	in	the	Chinese
room,	have	no	real	understanding	of	the	language	and	no	consciousness	of	what

it	was	doing.
There	are	a	few	philosophical	sleights	of	hand	in	Searle’s	argument.	For	one
thing,	the	man	in	this	thought	experiment	is	comparable	only	to	the	central
processing	unit	(CPU)	of	a	computer.	One	could	say	that	a	CPU	has	no	true
understanding	of	what	it	is	doing,	but	the	CPU	is	only	part	of	the	structure.	In
Searle’s	Chinese	room,	it	is	the	man	
with
	his	rulebook	that	constitutes	the	whole
system.	That	system	does	have	an	understanding	of	Chinese;	otherwise	it	would
not	be	capable	of	convincingly	answering	questions	in	Chinese,	which	would
violate	Searle’s	assumption	for	this	thought	experiment.
The	attractiveness	of	Searle’s	argument	stems	from	the	fact	that	it	is
difficult	today	to	infer	true	understanding	and	consciousness	in	a	computer
program.	The	problem	with	his	argument,	however,	is	that	you	can	apply	his
own	line	of	reasoning	to	the	human	brain	itself.	Each	neocortical	pattern
recognizer—indeed,	each	neuron	and	each	neuronal	component—is	following	an

algorithm.	(After	all,	these	are	molecular	mechanisms	that	follow	natural	law.)	If
we	conclude	that	following	an	algorithm	is	inconsistent	with	true	understanding
and	consciousness,	then	we	would	have	to	also	conclude	that	the	human	brain
does	not	exhibit	these	qualities	either.	You	can	take	John	Searle’s	Chinese	room
argument	and	simply	substitute	“manipulating	interneuronal	connections	and
synaptic	strengths”	for	his	words	“manipulating	symbols”	and	you	will	have	a

convincing	argument	to	the	effect	that	human	brains	cannot	truly	understand
anything.
Another	line	of	argument	comes	from	the	nature	of	nature,	which	has
become	a	new	sacred	ground	for	many	observers.	For	example,	New	Zealand
biologist	Michael	Denton	(born	in	1943)	sees	a	profound	difference	between	the
design	principles	of	machines	and	those	of	biology.	Denton	writes	that	natural
entities	are	“self-organizing,…self-referential,…
self-replicating,…reciprocal,…
self-formative,	and…holistic.”
10
	He	claims	that	such	biological	forms	can	only

anything.
10
be	created	through	biological	processes	and	that	these	forms	are	thereby
“immutable,…impenetrable,	and…fundamental”	realities	of	existence,	and	are
therefore	basically	a	different	philosophical	category	from	machines.
The	reality,	as	we	have	seen,	is	that	machines	can	be	designed	using	these
same	principles.	Learning	the	specific	design	paradigms	of	nature’s	most
intelligent	entity—the	human	brain—is	precisely	the	purpose	of	the	brain
reverse-engineering	project.	It	is	also	not	true	that	biological	systems	are

completely	“holistic,”	as	Denton	puts	it,	nor,	conversely,	do	machines	need	to	be
completely	modular.	We	have	clearly	identified	hierarchies	of	units	of
functionality	in	natural	systems,	especially	the	brain,	and	AI	systems	are	using
comparable	methods.

It	appears	to	me	that	many	critics	will	not	be	satisfied	until	computers
routinely	pass	the	Turing	test,	but	even	that	threshold	will	not	be	clear-cut.
Undoubtedly,	there	will	be	controversy	as	to	whether	claimed	Turing	tests	that
have	been	administered	are	valid.	Indeed,	I	will	probably	be	among	those	critics
disparaging	early	claims	along	these	lines.	By	the	time	the	arguments	about	the
validity	of	a	computer	passing	the	Turing	test	do	settle	down,	computers	will
have	long	since	surpassed	unenhanced	human	intelligence.
My	emphasis	here	is	on	the	word	“unenhanced,”	because	enhancement	is
precisely	the	reason	that	we	are	creating	these	“mind	children,”	as	Hans
Moravec	calls	them.
11
	Combining	human-level	pattern	recognition	with	the
inherent	speed	and	accuracy	of	computers	will	result	in	very	powerful	abilities.
But	this	is	not	an	alien	invasion	of	intelligent	machines	from	Mars—we	are
creating	these	tools	to	make	ourselves	smarter.	I	believe	that	most	observers	will

11
agree	with	me	that	this	is	what	is	unique	about	the	human	species:	We	build
these	tools	to	extend	our	own	reach.

EPILOGUE
	
The	picture’s	pretty	bleak,	gentlemen…The	world’s	climates	are	changing,
the	mammals	are	taking	over,	and	we	all	have	a	brain	about	the	size	of	a
walnut.
—Dinosaurs	talking,	in	
	by	Gary	Larson
	
I
ntelligence	may	be	defined	as	the	ability	to	solve	problems	with	limited
resources,	in	which	a	key	such	resource	is	time.	Thus	the	ability	to	more	quickly
solve	a	problem	like	finding	food	or	avoiding	a	predator	reflects	greater	power
of	intellect.	Intelligence	evolved	because	it	was	useful	for	survival—a	fact	that
may	seem	obvious,	but	one	with	which	not	everyone	agrees.	As	practiced	by	our
species,	it	has	enabled	us	not	only	to	dominate	the	planet	but	to	steadily	improve
the	quality	of	our	lives.	This	latter	point,	too,	is	not	apparent	to	everyone,	given
that	there	is	a	widespread	perception	today	that	life	is	only	getting	worse.	For
example,	a	Gallup	poll	released	on	May	4,	2011,	revealed	that	only	“44	percent
of	Americans	believed	that	today’s	youth	will	have	a	better	life	than	their

parents.”
1
If	we	look	at	the	broad	trends,	not	only	has	human	life	expectancy
quadrupled	over	the	last	millennium	(and	more	than	doubled	in	the	last	two
centuries),
2
	but	per	capita	GDP	(in	constant	current	dollars)	has	gone	
from
hundreds	of	dollars	in	1800	to	thousands	of	dollars	today,	with	even	more
pronounced	trends	in	the	developed	world.
3
	Only	a	handful	of	democracies
existed	a	century	ago,	whereas	they	are	the	norm	today.	For	a	historical
perspective	on	how	far	we	have	advanced,	I	suggest	people	read	Thomas
Hobbes’s

2
from
3
Hobbes’s	
Leviathan
	(1651),	in	which	he	describes	the	“life	of	man”	as	“solitary,
poor,	nasty,	brutish,	and	short.”	For	a	modern	perspective,	the	recent	book
Abundance
	(2012),	by	X-Prize	Foundation	founder	(and	cofounder	with	me	of
Singularity	University)	Peter	Diamandis	and	science	writer	Steven	Kotler,

documents	the	extraordinary	ways	in	which	life	today	has	steadily	improved	in
every	dimension.	Steven	Pinker’s	recent	
The	Better	Angels	of	Our	Nature:	Why
Violence	Has	Declined
	(2011)	painstakingly	documents	the	steady	rise	of
peaceful	relations	between	people	and	peoples.	American	lawyer,	entrepreneur,
and	author	Martine	Rothblatt	(born	in	1954)	documents	the	steady	improvement
in	civil	rights,	noting,	for	example,	how	in	a	couple	of	decades	same-sex
marriage	went	from	being	legally	recognized	nowhere	in	the	world	to	being
legally	accepted	in	a	rapidly	growing	number	of	jurisdictions.
4
A	primary	reason	that	people	believe	that	life	is	getting	worse	is	because
our	information	about	the	problems	of	the	world	has	steadily	improved.	If	there
is	a	battle	today	somewhere	on	the	planet,	we	experience	it	almost	as	if	we	were
there.	During	World	War	II,	tens	of	thousands	of	people	might	perish	in	a	battle,
and	if	the	public	could	see	it	at	all	it	was	in	a	grainy	newsreel	in	a	movie	theater

4
weeks	later.	During	World	War	I	a	small	elite	could	read	about	the	progress	of
the	conflict	in	the	newspaper	(without	pictures).	During	the	nineteenth	century
there	was	almost	no	access	to	news	in	a	timely	fashion	for	anyone.
The	advancement	we	have	made	as	a	species	due	to	our	intelligence	is
reflected	in	the	evolution	of	our	knowledge,	which	includes	our	technology	and
our	culture.	Our	various	technologies	are	increasingly	becoming	information
technologies,	which	inherently	continue	to	progress	in	an	exponential	manner.	It

is	through	such	technologies	that	we	are	able	to	address	the	grand	challenges	of
humanity,	such	as	maintaining	a	healthy	
environment,	providing	the	resources
for	a	growing	population	(including	energy,	food,	and	water),	overcoming
disease,	vastly	extending	human	longevity,	and	eliminating	poverty.	It	is	only	by
extending	ourselves	with	intelligent	technology	that	we	can	deal	with	the	scale
of	complexity	needed	to	address	these	challenges.
These	technologies	are	not	the	vanguard	of	an	intelligent	invasion	that	will

compete	with	and	ultimately	displace	us.	Ever	since	we	picked	up	a	stick	to
reach	a	higher	branch,	we	have	used	our	tools	to	extend	our	reach,	both
physically	and	mentally.	That	we	can	take	a	device	out	of	our	pocket	today	and
access	much	of	human	knowledge	with	a	few	keystrokes	extends	us	beyond
anything	imaginable	by	most	observers	only	a	few	decades	ago.	The	“cell
phone”	(the	term	is	placed	in	quotes	because	it	is	vastly	more	than	a	phone)	in
my	pocket	is	a	million	times	less	expensive	yet	thousands	of	times	more

powerful	than	the	computer	all	the	students	and	professors	at	MIT	shared	when	I
was	an	undergraduate	there.	That’s	a	several	billion-fold	increase	in
price/performance	over	the	last	forty	years,	an	escalation	we	will	see	again	in	the
next	twenty-five	years,	when	what	used	to	fit	in	a	building,	and	now	fits	in	your
pocket,	will	fit	inside	a	blood	cell.

In	this	way	we	will	merge	with	the	intelligent	technology	we	are	creating.
Intelligent	nanobots	in	our	bloodstream	will	keep	our	biological	bodies	healthy
at	the	cellular	and	molecular	levels.	They	will	go	into	our	brains	noninvasively
through	the	capillaries	and	interact	with	our	biological	neurons,	directly
extending	our	intelligence.	This	is	not	as	futuristic	as	it	may	sound.	There	are
already	blood	cell–sized	devices	that	can	cure	type	I	diabetes	in	animals	or
detect	and	destroy	cancer	cells	in	the	bloodstream.	Based	on	the	law	of
accelerating	returns,	these	technologies	will	be	a	billion	times	more	powerful
within	three	decades	than	they	are	today.
I	already	consider	the	devices	I	use	and	the	cloud	of	computing	resources	to
which	they	are	virtually	connected	as	extensions	of	myself,	and	feel	less	than
complete	if	I	am	cut	off	from	these	brain	extenders.	That	is	why	the	one-day
strike	by	Google,	Wikipedia,	and	thousands	of	other	Web	sites	against	the	SOPA
(Stop	Online	Piracy	Act)	on	January	18,	2012,

(Stop	Online	Piracy	Act)	on	January	18,	2012,	
was	so	remarkable:	I	felt	as	if	part
of	my	brain	were	going	on	strike	(although	I	and	others	did	find	ways	to	access
these	online	resources).	It	was	also	an	impressive	demonstration	of	the	political
power	of	these	sites	as	the	bill—which	looked	as	if	it	was	headed	for	ratification
—was	instantly	killed.	But	more	important,	it	showed	how	thoroughly	we	have
already	outsourced	parts	of	our	thinking	to	the	cloud	of	computing.	It	is	already
part	of	who	we	are.	Once	we	routinely	have	intelligent	nonbiological	intelligence

in	our	brains,	this	augmentation—and	the	cloud	it	is	connected	to—will	continue
to	grow	in	capability	exponentially.
The	intelligence	we	will	create	from	the	reverse-engineering	of	the	brain
will	have	access	to	its	own	source	code	and	will	be	able	to	rapidly	improve	itself
in	an	accelerating	iterative	design	cycle.	Although	there	is	considerable	plasticity
in	the	biological	human	brain,	as	we	have	seen,	it	does	have	a	relatively	fixed
architecture,	which	cannot	be	significantly	modified,	as	well	as	a	limited

capacity.	We	are	unable	to	increase	its	300	million	pattern	recognizers	to,	say,
400	million	unless	we	do	so	nonbiologically.	Once	we	can	achieve	that,	there
will	be	no	reason	to	stop	at	a	particular	level	of	capability.	We	can	go	on	to	make
it	a	billion	pattern	recognizers,	or	a	trillion.
From	quantitative	improvement	comes	qualitative	advance.	The	most
important	evolutionary	advance	in	
	was	quantitative:	the
development	of	a	larger	forehead	to	accommodate	more	neocortex.	Greater
neocortical	capacity	enabled	this	new	species	to	create	and	contemplate	thoughts

at	higher	conceptual	levels,	resulting	in	the	establishment	of	all	the	varied	fields
of	art	and	science.	As	we	add	more	neocortex	in	a	nonbiological	form,	we	can
expect	ever	higher	qualitative	levels	of	abstraction.
British	mathematician	Irvin	J.	Good,	a	colleague	of	Alan	Turing’s,	wrote	in

1965	that	“the	first	ultraintelligent	machine	is	the	last	invention	that	man	need
ever	make.”	He	defined	such	a	machine	as	one	that	could	surpass	the
“intellectual	activities	of	any	man	however	clever”	and	concluded	that	“since	the
design	of	machines	is	one	of	these	intellectual	activities,	an	
ultraintelligent
machine	could	design	even	better	machines;	there	would	then	unquestionably	be
an	‘intelligence	explosion.’”
The	last	invention	that	biological	evolution	needed	to	make—the	neocortex
—is	inevitably	leading	to	the	last	invention	that	humanity	needs	to	make—truly
intelligent	machines—and	the	design	of	one	is	inspiring	the	other.	Biological
evolution	is	continuing	but	technological	evolution	is	moving	a	million	times
faster	than	the	former.	According	to	the	law	of	accelerating	returns,	by	the	end	of
this	century	we	will	be	able	to	create	computation	at	the	limits	of	what	is
possible,	based	on	the	laws	of	physics	as	applied	to	computation.
5
	We	call	matter

5
and	energy	organized	in	this	way	“computronium,”	which	is	vastly	more
powerful	pound	per	pound	than	the	human	brain.	It	will	not	just	be	raw
computation	but	will	be	infused	with	intelligent	algorithms	constituting	all	of
human-machine	knowledge.	Over	time	we	will	convert	much	of	the	mass	and
energy	in	our	tiny	corner	of	the	galaxy	that	is	suitable	for	this	purpose	to
computronium.	Then,	to	keep	the	law	of	accelerating	returns	going,	we	will	need
to	spread	out	to	the	rest	of	the	galaxy	and	universe.

If	the	speed	of	light	indeed	remains	an	inexorable	limit,	then	colonizing	the
universe	will	take	a	long	time,	given	that	the	nearest	star	system	to	Earth	is	four
light-years	away.	If	there	are	even	subtle	means	to	circumvent	this	limit,	our
intelligence	and	technology	will	be	sufficiently	powerful	to	exploit	them.	This	is
one	reason	why	the	recent	suggestion	that	the	muons	that	traversed	the	730
kilometers	from	the	CERN	accelerator	on	the	Swiss-French	border	to	the	Gran

Sasso	Laboratory	in	central	Italy	appeared	to	be	moving	faster	than	the	speed	of
light	was	such	potentially	significant	news.	This	particular	observation	appears
to	be	a	false	alarm,	but	there	are	other	possibilities	to	get	around	this	limit.	We
do	not	even	need	to	exceed	the	speed	of	light	if	we	can	find	shortcuts	to	other
apparently	faraway	places	through	spatial	dimensions	beyond	the	three	with
which	we	are	familiar.	Whether	we	are	able	to	surpass	or	otherwise	get	around

the	speed	of	light	as	a	limit	will	be	the	key	strategic	issue	for	the	human-machine
civilization	at	the	beginning	of	the	twenty-second	century.
Cosmologists	argue	about	whether	the	world	will	end	in	fire	(a	big	crunch
to	match	the	big	bang)	or	ice	(the	death	of	the	stars	as	they	spread	out	into	an
eternal	expansion),	but	this	does	not	take	into	account	the	power	of	intelligence,
as	if	its	emergence	were	just	an	entertaining	sideshow	to	the	grand	celestial
mechanics	that	now	rule	the	universe.	How	long	will	it	take	for	us	to	spread	our

intelligence	in	its	nonbiological	form	throughout	the	universe?	If	we	can
transcend	the	speed	of	light—admittedly	a	big	if—for	example,	by	using
wormholes	through	space	(which	are	consistent	with	our	current	understanding
of	physics),	it	could	be	achieved	within	a	few	centuries.	Otherwise,	it	will	likely
take	much	longer.	In	either	scenario,	waking	up	the	universe,	and	then
intelligently	deciding	its	fate	by	infusing	it	with	our	human	intelligence	in	its
nonbiological	form,	is	our	destiny.

1.
	Here	is	one	sentence	from	
One	Hundred	Years	of	Solitude
	by
Gabriel	García	Márquez:
Aureliano	Segundo	was	not	aware	of	the	singsong	until	the
following	day	after	breakfast	when	he	felt	himself	being	bothered	by	a
buzzing	that	was	by	then	more	fluid	and	louder	than	the	sound	of	the
rain,	and	it	was	Fernanda,	who	was	walking	throughout	the	house
complaining	that	they	had	raised	her	to	be	a	queen	only	to	have	her
end	up	as	a	servant	in	a	madhouse,	with	a	lazy,	idolatrous,	libertine
husband	who	lay	on	his	back	waiting	for	bread	to	rain	down	from
heaven	while	she	was	straining	her	kidneys	trying	to	keep	afloat	a
home	held	together	with	pins	where	there	was	so	much	to	do,	so	much
to	bear	up	under	and	repair	from	the	time	God	gave	his	morning
sunlight	until	it	was	time	to	go	to	bed	that	when	she	got	there	her	eyes
were	full	of	ground	glass,	and	yet	no	one	ever	said	to	her,	“Good
morning,	Fernanda,	did	you	sleep	well?,”	nor	had	they	asked	her,	even

out	of	courtesy,	why	she	was	so	pale	or	why	she	awoke	with	purple
rings	under	her	eyes	in	spite	of	the	fact	that	she	expected	it,	of	course,
from	a	family	that	had	always	considered	her	a	nuisance,	an	old	rag,	a
booby	painted	on	the	wall,	and	who	were	always	going	around	saying
things	against	her	behind	her	back,	calling	her	churchmouse,	calling
her	Pharisee,	calling	her	crafty,	and	even	Amaranta,	may	she	rest	in
peace,	had	said	aloud	that	she	was	one	of	those	people	who	could	not

tell	their	rectums	from	their	ashes,	God	have	mercy,	such	words,	and
she	had	tolerated	everything	with	resignation	because	of	the	Holy
Father,	but	she	had	not	been	able	to	tolerate	it	any	more	when	that	evil
José	Arcadio	Segundo	
said	that	the	damnation	of	the	family	had	come
when	it	opened	its	doors	to	a	stuck-up	highlander,	just	imagine,	a
bossy	highlander,	Lord	save	us,	a	highlander	daughter	of	evil	spit	of
the	same	stripe	as	the	highlanders	the	government	sent	to	kill	workers,

you	tell	me,	and	he	was	referring	to	no	one	but	her,	the	godchild	of	the
Duke	of	Alba,	a	lady	of	such	lineage	that	she	made	the	liver	of
presidents’	wives	quiver,	a	noble	dame	of	fine	blood	like	her,	who	had
the	right	to	sign	eleven	peninsular	names	and	who	was	the	only	mortal
creature	in	that	town	full	of	bastards	who	did	not	feel	all	confused	at

the	sight	of	sixteen	pieces	of	silverware,	so	that	her	adulterous	husband
could	die	of	laughter	afterward	and	say	that	so	many	knives	and	forks
and	spoons	were	not	meant	for	a	human	being	but	for	a	centipede,	and
the	only	one	who	could	tell	with	her	eyes	closed	when	the	white	wine
was	served	and	on	what	side	and	in	which	glass	and	when	the	red	wine
and	on	what	side	and	in	which	glass	and	not	like	that	peasant	of	an
Amaranta,	may	she	rest	in	peace,	who	thought	that	white	wine	was
served	in	the	daytime	and	red	wine	at	night,	and	the	only	one	on	the
whole	coast	who	could	take	pride	in	the	fact	that	she	took	care	of	her
bodily	needs	only	in	golden	chamberpots,	so	that	Colonel	Aureliano
Buendía,	may	he	rest	in	peace,	could	have	the	effrontery	to	ask	her
with	his	Masonic	ill	humor	where	she	had	received	that	privilege	and
whether	she	did	not	shit	shit	but	shat	sweet	basil,	just	imagine,	with
those	very	words,	and	so	that	Renata,	her	own	daughter,	who	through

an	oversight	had	seen	her	stool	in	the	bedroom,	had	answered	that
even	if	the	pot	was	all	gold	and	with	a	coat	of	arms,	what	was	inside
was	pure	shit,	physical	shit,	and	worse	even	than	any	other	kind
because	it	was	stuck-up	highland	shit,	just	imagine,	her	own	daughter,
so	that	she	never	had	any	illusions	about	the	rest	of	the	family,	but	in
any	case	she	had	the	right	to	expect	a	little	more	consideration	from
her	husband	because,	for	better	or	for	worse,	he	was	her	consecrated

spouse,	her	helpmate,	her	legal	despoiler,	who	took	upon	himself	of
his	own	free	and	sovereign	will	the	grave	responsibility	of	taking	her
away	from	her	paternal	home,	where	she	never	wanted	for	or	suffered
from	anything,	where	she	wove	funeral	wreaths	as	a	pastime,	since	her
godfather	had	sent	a	letter	with	his	signature	and	the	stamp	of	his	ring
on	the	sealing	wax	simply	to	say	that	the	hands	of	his	goddaughter
were	not	meant	for	tasks	of	this	world	except	to	play	the	clavichord,

and,	nevertheless,	her	insane	husband	had	taken	her	from	her	home
with	all	manner	of	admonitions	and	warnings	and	had	brought	her	to
that	frying	pan	of	hell	where	a	person	could	not	breathe	because	of	the
heat,	and	before	she	had	completed	her	Pentecostal	fast	he	had	gone
off	with	his	wandering	trunks	and	his	wastrel’s	accordion	to	loaf	in
adultery	with	a	wretch	of	whom	it	was	only	enough	to	see	her	behind,
well,	that’s	been	said,	to	see	her	wiggle	her	mare’s	behind	in	order	to
guess	that	she

guess	that	she	
was	a,	that	she	was	a,	just	the	opposite	of	her,	who	was	a
lady	in	a	palace	or	a	pigsty,	at	the	table	or	in	bed,	a	lady	of	breeding,
God-fearing,	obeying	His	laws	and	submissive	to	His	wishes,	and	with
whom	he	could	not	perform,	naturally,	the	acrobatics	and	trampish

antics	that	he	did	with	the	other	one,	who,	of	course,	was	ready	for
anything,	like	the	French	matrons,	and	even	worse,	if	one	considers
well,	because	they	at	least	had	the	honesty	to	put	a	red	light	at	their
door,	swinishness	like	that,	just	imagine,	and	that	was	all	that	was
needed	by	the	only	and	beloved	daughter	of	Doña	Renata	Argote	and
Don	Fernando	del	Carpio,	and	especially	the	latter,	an	upright	man,	a
fine	Christian,	a	Knight	of	the	Order	of	the	Holy	Sepulcher,	those	who
receive	direct	from	God	the	privilege	of	remaining	intact	in	their
graves	with	their	skin	smooth	like	the	cheeks	of	a	bride	and	their	eyes
alive	and	clear	like	emeralds.
2.
	See	the	graph	“Growth	in	Genbank	DNA	Sequence	Data”	in	
chapter
10
.
3.
	Cheng	Zhang	and	Jianpeng	Ma,	“Enhanced	Sampling	and
Applications	in	Protein	Folding	in	Explicit	Solvent,”	
Journal	of
Chemical	Physics
	132,	no.	24	(2010):	244101.	See	also
http://folding.stanford.edu/English/About	about	the	Folding@home

2.
chapter
10
.
3.
Journal	of
project,	which	has	harnessed	over	five	million	computers	around	the
world	to	simulate	protein	folding.
4.
	For	a	more	complete	description	of	this	argument,	see	the	section
“[The	Impact…]	on	the	Intelligent	Destiny	of	the	Cosmos:	Why	We
Are	Probably	Alone	in	the	Universe”	in	chapter	6	of	
The	Singularity
Is	Near
	by	Ray	Kurzweil	(New	York:	Viking,	2005).
5.
	James	D.	Watson,	
Discovering	the	Brain
	(Washington,	DC:	National
Academies	Press,	1992).
6.
	Sebastian	Seung,	
Connectome:	How	the	Brain’s	Wiring	Makes	Us
Who	We	Are

4.
Is	Near
5.
6.
Who	We	Are
	(New	York:	Houghton	Mifflin	Harcourt,	2012).
7.
	“Mandelbrot	Zoom,”	http://www.youtube.com/watch?
v=gEw8xpb1aRA;	“Fractal	Zoom	Mandelbrot	Corner,”
http://www.youtube.com/watch?v=G_GBwuYuOOs.

Chapter	1:	Thought	Experiments	on	the	World
1.
	Charles	Darwin,	
The	Origin	of	Species
	(P.	F.	Collier	&	Son,	1909),
185/95–96.
2.
	Darwin,	
,	751	(206.1.1-6),	Peckham’s
Variorum	edition,	edited	by	Morse	Peckham,	
by	Charles	Darwin:	A	Variorum	Text
	(Philadelphia:	University	of
Pennsylvania	Press,	1959).
3.
	R.	Dahm,	“Discovering	DNA:	Friedrich	Miescher	and	the	Early
Years	of	Nucleic	Acid	Research,”	
Human	Genetics
	122,	no.	6
(2008):	565–81,	doi:10.1007/s00439-007-0433-0;	PMID	17901982.
4.
	Valery	N.	Soyfer,	“The	Consequences	of	Political	Dictatorship	for
Russian	Science,”	
	2,	no.	9	(2001):	723–29,
doi:10.1038/35088598;	PMID	11533721.
5.
	J.	D.	Watson	and	F.	H.	C.	Crick,	“A	Structure	for	Deoxyribose
Nucleic	Acid,”	
Nature
	171	(1953):	737–38,
http://www.nature.com/nature/dna50/watsoncrick.pdf	and	“Double
Helix:	50	Years	of	DNA,”	
Nature
	archive,
http://www.nature.com/nature/dna50/archive.xhtml.
6.

4.
5.
Nature
Nature
	archive,
6.
	Franklin	died	in	1958	and	the	Nobel	Prize	for	the	discovery	of	DNA
was	awarded	in	1962.	There	is	controversy	as	to	whether	or	not	she
would	have	shared	in	that	prize	had	she	been	alive	in	1962.
7.
	Albert	Einstein,	“On	the	Electrodynamics	of	Moving	Bodies”
(1905).	This	paper	established	the	special	theory	of	relativity.	See
Robert	Bruce	Lindsay	and	Henry	Margenau,	
(Woodbridge,	CT:	Ox	Bow	Press,	1981),	330.
8.
	 “Crookes	 radiometer,”	 Wikipedia,

6.
7.
8.
http://en.wikipedia.org/wiki/Crookes_radiometer.
9.
	Note	that	some	of	the	momentum	of	the	photons	is	transferred	to	the
air	molecules	in	the	bulb	(since	it	is	not	a	perfect	vacuum)	and	then
transferred	from	the	heated	air	molecules	to	the	vane.
10.
	Albert	Einstein,	“Does	the	Inertia	of	a	Body	Depend	Upon	Its
Energy	Content?”	(1905).	This	paper	established	Einstein’s
famous	formula	
E
	=	
mc
2
.
11.
	“Albert	Einstein’s	Letters	to	President	Franklin	Delano	Roosevelt,”
http://hypertextbook.com/eworld/einstein.shtml.

Chapter	3:	A	Model	of	the	Neocortex:	The
Pattern	Recognition	Theory	of	Mind
1.
	Some	nonmammals,	such	as	crows,	parrots,	and	octopi,	are	reported
to	be	capable	of	some	level	of	reasoning;	however,	this	is	limited
and	has	not	been	sufficient	to	create	tools	that	have	their	own
evolutionary	course	of	development.	These	animals	may	have
adapted	other	brain	regions	to	perform	a	small	number	of	levels	of
hierarchical	thinking,	but	a	neocortex	is	required	for	the	relatively
unrestricted	hierarchical	thinking	that	humans	can	perform.
2.
	V.	B.	Mountcastle,	“An	Organizing	Principle	for	Cerebral	Function:
The	Unit	Model	and	the	Distributed	System”	(1978),	in	Gerald	M.
Edelman	and	Vernon	B.	Mountcastle,	
The	Mindful	Brain:	Cortical
Organization	and	the	Group-Selective	Theory	of	Higher	Brain
Function
	(Cambridge,	MA:	MIT	Press,	1982).
3.
	Herbert	A.	Simon,	“The	Organization	of	Complex	Systems,”	in
Howard	H.	Pattee,	ed.,	
Hierarchy	Theory:	The	Challenge	of
Complex	Systems
	(New	York:	George	Braziller,	Inc.,	1973),

2.
Function
3.
http://blog.santafe.edu/wp-content/uploads/2009/03/simon	1973.pdf.
4.
	Marc	D.	Hauser,	Noam	Chomsky,	and	W.	Tecumseh	Fitch,	“The
Faculty	of	Language:	What	Is	It,	Who	Has	It,	and	How	Did	It
Evolve?”	
Science
	298	(November	2002):	1569–79,
http://www.sciencemag.org/content/298/5598/1569.short.
5.
	The	following	passage	from	the	book	
Transcend:	Nine	Steps	to
Living	Well	Forever
,	by	Ray	Kurzweil	and	Terry	Grossman	(New
York:	Rodale,	2009),	describes	this	lucid	dreaming	technique	in
more	detail:

4.
Evolve?”	
Science
5.
I’ve	developed	a	method	of	solving	problems	while	I	sleep.	I’ve
perfected	it	for	myself	over	several	decades	and	have	learned	the
subtle	means	by	which	this	is	likely	to	work	better.
I	start	out	by	assigning	myself	a	problem	when	I	get	into	bed.
This	can	be	any	kind	of	problem.	It	could	be	a	math	problem,	an	issue
with	one	of	my	inventions,	a	business	strategy	question,	or	even	an
interpersonal	problem.
I’ll	think	about	the	problem	for	a	few	minutes,	but	I	try	not	to

solve	it.	That	would	just	cut	off	the	creative	problem	solving	to	come.	I
do	try	to	think	about	it.	What	do	I	know	about	this?	What	form	could	a
solution	take?	And	then	I	go	to	sleep.	Doing	this	primes	my
subconscious	mind	to	work	on	the	problem.
Terry:
	Sigmund	Freud	pointed	out	that	when	we	dream,	many	of
the	censors	in	our	brain	are	relaxed,	so	that	we	might	dream	about
things	that	are	socially,	culturally,	or	even	sexually	taboo.	We	can
dream	about	weird	things	that	we	wouldn’t	allow	ourselves	to	think
about	during	the	day.	That’s	at	least	one	reason	why	dreams	are
strange.
Ray:
	There	are	also	professional	blinders	that	prevent	people	from
thinking	creatively,	many	of	which	come	from	our	professional
training,	mental	blocks	such	as	“you	can’t	solve	a	signal	processing
problem	that	way”	or	“linguistics	is	not	supposed	to	use	those	rules.”
These	mental	assumptions	are	also	relaxed	in	our	dream	state,	so	I’ll
dream	about	new	ways	of	solving	problems	without	being	burdened	by
these	daytime	constraints.

strange.
Ray:
Terry:
	There’s	another	part	of	our	brain	also	not	working	when	we
dream,	our	rational	faculties	to	evaluate	whether	an	idea	is	reasonable.
So	that’s	another	reason	that	weird	or	fantastic	things	happen	in	our
dreams.	When	the	elephant	walks	through	the	wall,	we	aren’t	shocked
as	to	how	the	elephant	could	do	this.	We	just	say	to	our	dream	selves,
“Okay,	an	elephant	walked	through	the	wall,	no	big	deal.”	Indeed,	if	I
wake	up	in	the	middle	of	the	night,	I	often	find	that	I’ve	been

Terry:
dreaming	in	strange	and	oblique	ways	about	the	problem	that	I
assigned	myself.
Ray:
	The	next	step	occurs	in	the	morning	in	the	halfway	state
between	dreaming	and	being	awake,	which	is	often	called	
lucid
dreaming
.	In	this	state,	I	still	have	the	feelings	and	imagery	from	my
dreams,	but	now	I	do	have	my	rational	faculties.	I	realize,	for	example,
that	I	am	in	a	bed.	And	I	could	formulate	the	rational	thought	that	I
have	a	lot	to	do	so	I	had	better	get	out	of	bed.	But	that	would	be	a

Ray:
lucid
dreaming
mistake.	Whenever	I	can,	I	will	stay	in	bed	and	continue	in	this	lucid
dream	state	because	that	is	key	to	this	creative	problem-solving
method.	By	the	way,	this	doesn’t	work	if	the	alarm	rings.

Reader:
	Sounds	like	the	best	of	both	worlds.
Ray:
	Exactly.	I	still	have	access	to	the	dream	thoughts	about	the
problem	I	assigned	myself	the	night	before.	But	now	I’m	sufficiently
conscious	and	rational	to	evaluate	the	new	creative	ideas	that	came	to
me	during	the	night.	I	can	determine	which	ones	make	sense.	After
perhaps	20	minutes	of	this,	I	invariably	will	have	keen	new	insights
into	the	problem.
I’ve	come	up	with	inventions	this	way	(and	spent	the	rest	of	the
day	writing	a	patent	application),	figured	out	how	to	organize	material
for	a	book	such	as	this,	and	come	up	with	useful	ideas	for	a	diverse	set
of	problems.	If	I	have	a	key	decision	to	make,	I	will	always	go	through
this	process,	after	which	I	am	likely	to	have	real	confidence	in	my
decision.
The	key	to	the	process	is	to	let	your	mind	go,	to	be	nonjudgmental,
and	not	to	worry	about	how	well	the	method	is	working.	It	is	the
opposite	of	a	mental	discipline.	Think	about	the	problem,	but	then	let

decision.
ideas	wash	over	you	as	you	fall	asleep.	Then	in	the	morning,	let	your
mind	go	again	as	you	review	the	strange	ideas	that	your	dreams
generated.	I	have	found	this	to	be	an	invaluable	method	for	harnessing
the	natural	creativity	of	my	dreams.
Reader:
	Well,	for	the	workaholics	among	us,	we	can	now	work	in
our	dreams.	Not	sure	my	spouse	is	going	to	appreciate	this.
Ray:
	Actually,	you	can	think	of	it	as	getting	your	dreams	to	do
your	work	for	you.

Chapter	4:	The	Biological	Neocortex
1.
	Steven	Pinker,	
How	the	Mind	Works
	(New	York:	Norton,	1997),
152–53.
2.
	D.	O.	Hebb,	
The	Organization	of	Behavior
	(New	York:	John	Wiley
&	Sons,	1949).
3.
	Henry	Markram	and	Rodrigo	Perrin,	“Innate	Neural	Assemblies	for
Lego	Memory,”	
Frontiers	in	Neural	Circuits
	5,	no.	6	(2011).
4.
	E-mail	communication	from	Henry	Markram,	February	19,	2012.
5.
	Van	J.	Wedeen	et	al.,	“The	Geometric	Structure	of	the	Brain	Fiber
Pathways,”	
Science
	335,	no.	6076	(March	30,	2012).
6.
	Tai	Sing	Lee,	“Computations	in	the	Early	Visual	Cortex,”	
Journal	of
Physiology—Paris
	97	(2003):	121–39.
7.
	A	list	of	papers	can	be	found	at
http://cbcl.mit.edu/people/poggio/tpcv_short_pubs.pdf.
8.
	Daniel	J.	Felleman	and	David	C.	Van	Essen,	“Distributed
Hierarchical	Processing	in	the	Primate	Cerebral	Cortex,”	
Cerebral
Cortex
	1,	no.	1	(January/February	1991):	1–47.	A	compelling
analysis	of	the	Bayesian	mathematics	of	the	top-down	and	bottom-

Science
6.
Journal	of
7.
8.
Cerebral
Cortex
up	communication	in	the	neocortex	is	provided	by	Tai	Sing	Lee	in
“Hierarchical	Bayesian	Inference	in	the	Visual	Cortex,”	
Journal	of
the	Optical	Society	of	America
	20,	no.	7	(July	2003):	1434–48.
9.
	Uri	Hasson	et	al.,	“A	Hierarchy	of	Temporal	Receptive	Windows	in
Human	Cortex,”	
	28,	no.	10	(March	5,
2008):	2539–50.
10.
	Marina	Bedny	et	al.,	“Language	Processing	in	the	Occipital	Cortex
of	Congenitally	Blind	Adults,”	
Academy	of	Sciences

Journal	of
9.
10.
	108,	no.	11	(March	15,	2011):	4429–34.
11.
	Daniel	E.	Feldman,	“Synaptic	Mechanisms	for	Plasticity	in
Neocortex,”	
Annual	Review	of	Neuroscience
	32	(2009):	33–55.
12.
	Aaron	C.	Koralek	et	al.,	“Corticostriatal	Plasticity	Is	Necessary	for
Learning	Intentional	Neuroprosthetic	Skills,”	
Nature
	483	(March
15,	2012):	331–35.
13.
	E-mail	communication	from	Randal	Koene,	January	2012.
14.
	Min	Fu,	Xinzhu	Yu,	Ju	Lu,	and	Yi	Zuo,	“Repetitive	Motor
Learning	Induces	Coordinated	Formation	of	Clustered	Dendritic

Spines	
in	Vivo
,”	
Nature
	483	(March	1,	2012):	92–95.
15.
	Dario	Bonanomi	et	al.,	“Ret	Is	a	Multifunctional	Coreceptor	That
Integrates	Diffusible-and	Contact-Axon	Guidance	Signals,”	
Cell
148,	no.	3	(February	2012):	568–82.
16.
	See	
endnote	7	in	chapter	11
.

Chapter	5:	The	Old	Brain
1.
	Vernon	B.	Mountcastle,	“The	View	from	Within:	Pathways	to	the
Study	of	Perception,”	
Johns	Hopkins	Medical	Journal
	136	(1975):
109–31.
2.
	B.	Roska	and	F.	Werblin,	“Vertical	Interactions	Across	Ten	Parallel,
Stacked	Representations	in	the	Mammalian	Retina,”	
Nature
	410,	no.
6828	(March	29,	2001):	583–87;	“Eye	Strips	Images	of	All	but	Bare
Essentials	Before	Sending	Visual	Information	to	Brain,	UC
Berkeley	Research	Shows,”	University	of	California	at	Berkeley
news	release,	March	28,	2001,
www.berkeley.edu/news/media/releases/2001/03/28_wers1.xhtml.
3.
	Lloyd	Watts,	“Reverse-Engineering	the	Human	Auditory	Pathway,”
in	J.	Liu	et	al.,	eds.,	
WCCI	2012
	(Berlin:	Springer-Verlag,	2012),
47–59.	Lloyd	Watts,	“Real-Time,	High-Resolution	Simulation	of	the
Auditory	Pathway,	with	Application	to	Cell-Phone	Noise
Reduction,”	
ISCAS
	(June	2,	2010):	3821–24.	For	other	papers	see
http://www.lloydwatts.com/publications.xhtml.
4.
	See	Sandra	Blakeslee,	“Humanity?	Maybe	It’s	All	in	the	Wiring,”

3.
WCCI	2012
ISCAS
4.
New	York
	
Times
,	December	11,	2003,
http://www.nytimes.com/2003/12/09/science/09BRAI.xhtml.
5.
	T.	E.	J.	Behrens	et	al.,	“Non-Invasive	Mapping	of	Connections
between	Human	Thalamus	and	Cortex	Using	Diffusion	Imaging,”
	6,	no.	7	(July	2003):	750–57.
6.
	Timothy	J.	Buschman	et	al.,	“Neural	Substrates	of	Cognitive
Capacity	Limitations,”	
Sciences
	108,	no.	27	(July	5,	2011):	11252–55,
http://www.pnas.org/content/108/27/11252.long.
7.

New	York
	
Times
5.
6.
Sciences
7.
	Theodore	W.	Berger	et	al.,	“A	Cortical	Neural	Prosthesis	for
Restoring	and	Enhancing	Memory,”	
Journal	of	Neural	Engineering
8,	no.	4	(August	2011).
8.
	Basis	functions	are	nonlinear	functions	that	can	be	combined
linearly	(by	adding	together	multiple	weighted-basis	functions)	to
approximate	any	nonlinear	function.	A.	Pouget	and	L.	H.	Snyder,
“Computational	Approaches	to	Sensorimotor	Transformations,”
	3,	no.	11	Supplement	(November	2000):

1192–98.
9.
	J.	R.	Bloedel,	“Functional	Heterogeneity	with	Structural
Homogeneity:	How	Does	the	Cerebellum	Operate?”	
Behavioral	and
Brain	Sciences
	15,	no.	4	(1992):	666–78.
10.
	S.	Grossberg	and	R.	W.	Paine,	“A	Neural	Model	of	Cortico-
Cerebellar	Interactions	during	Attentive	Imitation	and	Predictive
Learning	of	Sequential	Handwriting	Movements,”	
Neural
Networks
	13,	no.	8–9	(October–November	2000):	999–1046.
11.
	Javier	F.	Medina	and	Michael	D.	Mauk,	“Computer	Simulation	of
Cerebellar	Information	Processing,”	
	3
(November	2000):	1205–11.
12.
	James	Olds,	“Pleasure	Centers	in	the	Brain,”	
(October	1956):	105–16.	Aryeh	Routtenberg,	“The	Reward
System	of	the	Brain,”	
	239	(November	1978):
154–64.	K.	C.	Berridge	and	M.	L.	Kringelbach,	“Affective
Neuroscience	of	Pleasure:	Reward	in	Humans	and	Other
Animals,”	
Psychopharmacology
	199	(2008):	457–80.	Morten	L.
Kringelbach,	
The	Pleasure	Center:	Trust	Your	Animal	Instincts

	3
12.
Animals,”	
(New	York:	Oxford	University	Press,	2009).	Michael	R.
Liebowitz,	
The	Chemistry	of	Love
	(Boston:	Little,	Brown,	1983).
W.	L.	Witters	and	P.	Jones-Witters,	
Human	Sexuality:	A
Biological	Perspective
	(New	York:	Van	Nostrand,	1980).

Chapter	6:	Transcendent	Abilities
1.
	Michael	Nielsen,	
Reinventing	Discovery:	The	New	Era	of
Networked	Science
	(Princeton,	NJ:	Princeton	University	Press,
2012),	1–3.	T.	Gowers	and	M.	Nielsen,	“Massively	Collaborative
Mathematics,”	
Nature
	461,	no.	7266	(2009):	879–81.	“A
Combinatorial	Approach	to	Density	Hales-Jewett,”	
Gowers’s
Weblog
,	http://gowers.wordpress.com/2009/02/01/a-combinatorial-
approach-to-density-hales-jewett/.	Michael	Nielsen,	“The	Polymath
Project:	Scope	of	Participation,”	March	20,	2009,
http://michaelnielsen.org/blog/?p=584.	Julie	Rehmeyer,	“SIAM:
Massively	Collaborative	Mathematics,”	Society	for	Industrial	and
Applied	Mathematics,	April	1,	2010,
http://www.siam.org/news/news.php?id=1731.
2.
	P.	Dayan	and	Q.	J.	M.	Huys,	“Serotonin,	Inhibition,	and	Negative
Mood,”	
PLoS
	
Computational	Biology
	4,	no.	1	(2008),
http://compbiol.plosjournals.org/perlserv/?request=get-
document&doi=10.1371/journal.pcbi.0040004.

Chapter	7:	The	Biologically	Inspired	Digital
Neocortex
1.
	Gary	Cziko,	
Without	Miracles:	Universal	Selection	Theory	and	the
Second	Darwinian	Revolution
	(Cambridge,	MA:	MIT	Press,	1955).
2.
	David	Dalrymple	has	been	a	mentee	of	mine	since	he	was	eight
years	old	in	1999.	You	can	read	his	background	here:
http://esp.mit.edu/learn/teachers/davidad/bio.xhtml,	 and
http://www.brainsciences.org/Research-Team/mr-david-
dalrymple.xhtml.
3.
	Jonathan	Fildes,	“Artificial	Brain	‘10	Years	Away,’”	BBC	News,
July	22,	2009,	http://news.bbc.co.uk/2/hi/8164060.stm.	See	also	the
video	“Henry	Markram	on	Simulating	the	Brain:	The	Next	Decisive
Years,”	http://www.kurzweilai.net/henry-markram-simulating-the-
brain-next-decisive-years.
4.
	M.	Mitchell	Waldrop,	“Computer	Modelling:	Brain	in	a	Box,”
Nature	News
,	February	22,	2012,
http://www.nature.com/news/computer-modelling-brain-in-a-box-
1.10066.
5.
	Jonah	Lehrer,	“Can	a	Thinking,	Remembering,	Decision-Making
Biologically	Accurate	Brain	Be	Built	from	a	Supercomputer?”	
Seed
,

4.
1.10066.
5.
Seed
,
http://seedmagazine.com/content/article/out_of_the_blue/.
6.
	Fildes,	“Artificial	Brain	‘10	Years	Away.’”
7.
	See	http://www.humanconnectomeproject.org/.
8.
	Anders	Sandberg	and	Nick	Bostrom,	
Whole	Brain	Emulation:	A
Roadmap
,	Technical	Report	#2008–3	(2008),	Future	of	Humanity
Institute,	Oxford	University,	www.fhi.ox.ac.uk/reports/2008-3.pdf.
9.
	Here	is	the	basic	schema	for	a	neural	net	algorithm.	Many	variations
are	possible,	and	the	designer	of	the	system	needs	to	provide	certain
critical	parameters	and	methods,	detailed	on	the	following	pages.

6.
7.
8.
Roadmap
9.
Creating	a	neural	net	solution	to	a	problem	involves	the	following
steps:
Define	the	input.
	
Define	the	topology	of	the	neural	net	(i.e.,	the	layers	of	neurons

and	the	connections	between	the	neurons).
	
Train	the	neural	net	on	examples	of	the	problem.
	
Run	the	trained	neural	net	to	solve	new	examples	of	the	problem.
	
Take	your	neural	net	company	public.
	
These	steps	(except	for	the	last	one)	are	detailed	below:
The	Problem	Input
The	problem	input	to	the	neural	net	consists	of	a	series	of	numbers.
This	input	can	be:
In	a	visual	pattern	recognition	system,	a	two-dimensional	array	of
numbers	representing	the	pixels	of	an	image;	or
	
In	an	auditory	(e.g.,	speech)	recognition	system,	a	two-dimensional
array	of	numbers	representing	a	sound,	in	which	the	first	dimension
represents	parameters	of	the	sound	(e.g.,	frequency	components)	and
the	second	dimension	represents	different	points	in	time;	or
	
In	an	arbitrary	pattern	recognition	system,	an	
n
-dimensional	array
of	numbers	representing	the	input	pattern.
	
Defining	the	Topology
To	set	up	the	neural	net,	the	architecture	of	each	neuron	consists	of:
Multiple	inputs	in	which	each	input	is	“connected”	to	either	the

n
	
output	of	another	neuron	or	one	of	the	input	numbers.
	
Generally,	a	single	output,	which	is	connected	to	either	the	input	of
another	neuron	(which	is	usually	in	a	higher	layer)	or	the	final	output.
	
Set	Up	the	First	Layer	of	Neurons
Create	
N
0
	neurons	in	the	first	layer.	For	each	of	these	neurons,
“connect”	each	of	the	multiple	inputs	of	the	neuron	to	“points”	(i.e.,
numbers)	in	the	problem	input.	These	connections	can	be	determined
randomly	or	using	an	evolutionary	algorithm	(see	below).

Assign	an	initial	“synaptic	strength”	to	each	connection	created.
These	weights	can	start	out	all	the	same,	can	be	assigned	randomly,	or
can	be	determined	in	another	way	(see	below).
	
Set	Up	the	Additional	Layers	of	Neurons
Set	up	a	total	of	
M
	layers	of	neurons.	For	each	layer,	set	up	the	neurons
in	that	layer.
For	layer
i
:
Create	
N
i
	neurons	in	layer
i
.	For	each	of	these	neurons,	“connect”
each	of	the	multiple	inputs	of	the	neuron	to	the	outputs	of	the	neurons
in	layer
i–1
	(see	variations	below).
	
The	outputs	of	the	neurons	in	layer
M
	are	the	outputs	of	the	neural
net	(see	variations	below).
	
The	Recognition	Trials
How	Each	Neuron	Works
Once	the	neuron	is	set	up,	it	does	the	following	for	each
recognition	trial:
	
Each	weighted	input	to	the	neuron	is	computed	by	multiplying	the

i–1
	
M
	
output	of	the	other	neuron	(or	initial	input)	that	the	input	to	this	neuron
is	connected	to	by	the	synaptic	strength	of	that	connection.
	
All	of	these	weighted	inputs	to	the	neuron	are	summed.
	
If	this	sum	is	greater	than	the	firing	threshold	of	this	neuron,	then
this	neuron	is	considered	to	fire	and	its	output	is	1.	Otherwise,	its
output	is	0	(see	variations	below).
	
Do	the	Following	for	Each	Recognition	Trial

For	each	layer,	from	layer
0
	to	layer
M
:
For	each	neuron	in	the	layer:
Sum	its	weighted	inputs	(each	weighted	input	=	the	output	of	the
other	neuron	[or	initial	input]	that	the	input	to	this	neuron	is	connected
to,	multiplied	by	the	synaptic	strength	of	that	connection).
	
If	this	sum	of	weighted	inputs	is	greater	than	the	firing	threshold
for	this	neuron,	set	the	output	of	this	neuron	=	1,	otherwise	set	it	to	0.

To	Train	the	Neural	Net
Run	repeated	recognition	trials	on	sample	problems.
	
After	each	trial,	adjust	the	synaptic	strengths	of	all	the	interneuronal
connections	to	improve	the	performance	of	the	neural	net	on	this	trial	(see
the	discussion	below	on	how	to	do	this).
	
Continue	this	training	until	the	accuracy	rate	of	the	neural	net	is	no
longer	improving	(i.e.,	reaches	an	asymptote).
	
Key	Design	Decisions
In	the	simple	schema	above,	the	designer	of	this	neural	net	algorithm	needs
to	determine	at	the	outset:
What	the	input	numbers	represent.
	
The	number	of	layers	of	neurons.
	
The	number	of	neurons	in	each	layer.	(Each	layer	does	not	necessarily
need	to	have	the	same	number	of	neurons.)
	
The	number	of	inputs	to	each	neuron	in	each	layer.	The	number	of
inputs	(i.e.,	interneuronal	connections)	can	also	vary	from	neuron	to	neuron
and	from	layer	to	layer.
	
The	actual	“wiring”	(i.e.,	the	connections).	For	each	neuron	in	each
layer,	this	consists	of	a	list	of	other	neurons,	the	outputs	of	which	constitute

the	inputs	to	this	neuron.	This	represents	a	key	design	area.	There	are	a
number	of	possible	ways	to	do	this:
	
(1)	Wire	the	neural	net	randomly;	or
	
(2)	Use	an	evolutionary	algorithm	(see	below)	to	determine	an	optimal
wiring;	or

(3)	Use	the	system	designer’s	best	judgment	in	determining	the	wiring.
	
The	initial	synaptic	strengths	(i.e.,	weights)	of	each	connection.	There
are	a	number	of	possible	ways	to	do	this:
	
(1)	Set	the	synaptic	strengths	to	the	same	value;	or
	
(2)	Set	the	synaptic	strengths	to	different	random	values;	or
	
(3)	Use	an	evolutionary	algorithm	to	determine	an	optimal	set	of	initial
values;	or
	
(4)	Use	the	system	designer’s	best	judgment	in	determining	the	initial
values.
	
The	firing	threshold	of	each	neuron.
	
Determine	the	output.	The	output	can	be:
	
(1)	the	outputs	of	layer
M
	of	neurons;	or
	
(2)	the	output	of	a	single	output	neuron,	the	inputs	of	which	are	the	outputs
of	the	neurons	in	layer
M
;	or
	
(3)	a	function	of	(e.g.,	a	sum	of)	the	outputs	of	the	neurons	in	layer
M
;	or
	
(4)	another	function	of	neuron	outputs	in	multiple	layers.
	
Determine	how	the	synaptic	strengths	of	all	the	connections	are	adjusted
during	the	training	of	this	neural	net.	This	is	a	key	design	decision	and	is

M
	
M
;	or
	
M
;	or
	
the	subject	of	a	great	deal	of	research	and	discussion.	There	are	a	number	of
possible	ways	to	do	this:
	
(1)	For	each	recognition	trial,	increment	or	decrement	each	synaptic
strength	by	a	(generally	small)	fixed	amount	so	that	the	neural	net’s
output	more	closely	matches	the	correct	answer.	One	way	to	do	this	is
to	try	both	incrementing	and	decrementing	and	see	which	has	the	more
desirable	effect.	This	can	be	time-consuming,	so	other	methods	exist
for	making	local	decisions	on	whether	to	increment	or	decrement	each

synaptic	strength.

(2)	Other	statistical	methods	exist	for	modifying	the	synaptic	strengths	after
each	recognition	trial	so	that	the	performance	of	the	neural	net	on	that
trial	more	closely	matches	the	correct	answer.
	
Note	that	neural	net	training	will	work	even	if	the	answers	to	the
training	trials	are	not	all	correct.	This	allows	using	real-world	training	data
that	may	have	an	inherent	error	rate.	One	key	to	the	success	of	a	neural	net–
based	recognition	system	is	the	amount	of	data	used	for	training.	Usually	a
very	substantial	amount	is	needed	to	obtain	satisfactory	results.	As	with
human	students,	the	amount	of	time	that	a	neural	net	spends	learning	its
lessons	is	a	key	factor	in	its	performance.
	
Many	variations	of	the	above	are	feasible.	For	example:
There	are	different	ways	of	determining	the	topology.	In	particular,	the
interneuronal	wiring	can	be	set	either	randomly	or	using	an	evolutionary
algorithm.
	
There	are	different	ways	of	setting	the	initial	synaptic	strengths.

The	inputs	to	the	neurons	in	layer
i
	do	not	necessarily	need	to	come	from
the	outputs	of	the	neurons	in	layer
i–1
.	Alternatively,	the	inputs	to	the
neurons	in	each	layer	can	come	from	any	lower	layer	or	any	layer.
	
There	are	different	ways	to	determine	the	final	output.
	
The	method	described	above	results	in	an	“all	or	nothing”	(1	or	0)	firing
called	a	nonlinearity.	There	are	other	nonlinear	functions	that	can	be	used.
Commonly	a	function	is	used	that	goes	from	0	to	1	in	a	rapid	but	more

i
i–1
	
gradual	fashion.	Also,	the	outputs	can	be	numbers	other	than	0	and	1.
	
The	different	methods	for	adjusting	the	synaptic	strengths	during
training	represent	key	design	decisions.
	
The	above	schema	describes	a	“synchronous”	neural	net,	in	which	each
recognition	trial	proceeds	by	computing	the	outputs	of	each	layer,	starting
with	layer
0
	through	layer
M
.	In	a	true	parallel	system,	in	which	each	neuron

is	operating	independently	of	the	others,	the	neurons	can	operate
“asynchronously”	(i.e.,	independently).	In	an	asynchronous	approach,	each
neuron	is	constantly	scanning	its	inputs	and	fires	whenever	the	sum	of	its
weighted	inputs	exceeds	its	threshold	(or	whatever	its	output	function
specifies).
	
10.
	Robert	Mannell,	“Acoustic	Representations	of	Speech,”	2008,
http://clas.mq.edu.au/acoustics/frequency/acoustic_speech.xhtml.
11.
	Here	is	the	basic	schema	for	a	genetic	(evolutionary)	algorithm.	Many
variations	are	possible,	and	the	designer	of	the	system	needs	to	provide
certain	critical	parameters	and	methods,	detailed	below.

The	Evolutionary	Algorithm
Create	
N
	solution	“creatures.”	Each	one	has:
A	genetic	code:	a	sequence	of	numbers	that	characterize	a	possible
solution	to	the	problem.	The	numbers	can	represent	critical	parameters,
steps	to	a	solution,	rules,	
etc.
	
For	each	generation	of	evolution,	do	the	following:
Do	the	following	for	each	of	the	
N
	solution	creatures:
	
Apply	this	solution	creature’s	solution	(as	represented	by	its	genetic
code)	to	the	problem,	or	simulated	environment.	Rate	the	solution.
	
Pick	the	
L
	solution	creatures	with	the	highest	ratings	to	survive	into	the
next	generation.
	
Eliminate	the	(
N
	–	
L
)	nonsurviving	solution	creatures.
	
Create	(
N
	–	
L
)	new	solution	creatures	from	the	
L
	surviving	solution
creatures	by:
	
(1)	Making	copies	of	the	
L
	surviving	creatures.	Introduce	small	random
variations	into	each	copy;	or
	
(2)	Create	additional	solution	creatures	by	combining	parts	of	the	genetic
code	(using	“sexual”	reproduction,	or	otherwise	combining	portions	of
the	chromosomes)	from	the

solution	creatures	with	the	highest	ratings	to	survive	into	the
	
N
	–	
L
	
Create	(
N
	–	
L
L
	
L
	
the	chromosomes)	from	the	
L
	surviving	creatures;	or
	
(3)	Do	a	combination	of	(1)	and	(2).
	
Determine	whether	or	not	to	continue	evolving:
Improvement	=	(highest	rating	in	this	generation)	–	(highest	rating	in
the	previous	generation).
	
If	Improvement	<	Improvement	Threshold	then	we’re	done.
	
The	solution	creature	with	the	highest	rating	from	the	last	generation	of

evolution	has	the	best	solution.	Apply	the	solution	defined	by	its	genetic
code	to	the	problem.
	
In	the	simple	schema	above,	the	designer	needs	to	determine	at	the	outset:
Key	parameters:
	
N
	
L
	
Improvement	threshold.
	
What	the	numbers	in	the	genetic	code	represent	and	how	the	solution	is
computed	from	the	genetic	code.
	
A	method	for	determining	the	
N
	solution	creatures	in	the	first
generation.	In	general,	these	need	only	be	“reasonable”	attempts	at	a
solution.	If	these	first-generation	solutions	are	too	far	afield,	the
evolutionary	algorithm	may	have	difficulty	converging	on	a	good	solution.
It	is	often	worthwhile	to	create	the	initial	solution	creatures	in	such	a	way
that	they	are	reasonably	diverse.	This	will	help	prevent	the	evolutionary
process	from	just	finding	a	“locally”	optimal	solution.
	
How	the	solutions	are	rated.
	
How	the	surviving	solution	creatures	reproduce.
	
There	does	not	need	to	be	a	fixed	number	of	surviving	solution
creatures	(
L
)	from	each	generation.	The	survival	rule(s)	can	allow	for	a
variable	number	of	survivors.
	
There	does	not	need	to	be	a	fixed	number	of	new	solution	creatures
created	in	each	generation	(
N
	–	
L
).	The	procreation	rules	can	be
independent	of	the	size	of	the	population.	Procreation	can	be	related	to
survival,	thereby	allowing	the	fittest	solution	creatures	to	procreate	the
most.

The	decision	as	to	whether	or	not	to	continue	evolving	can	be	varied.	It
can	consider	more	than	just	the	highest-rated	solution	creature	from	the
most	recent	generation(s).	It	can	also	consider	a	trend	that	goes	beyond	just
the	last	two	generations.
	
12.
	Dileep	George,	“How	the	Brain	Might	Work:	A	Hierarchical	and
Temporal	Model	for	Learning	and	Recognition”	(PhD	dissertation,
Stanford	University,	June	2008).
13.
	A.	M.	Turing,	“Computing	Machinery	and	Intelligence,”	
Mind
,	October
1950.
14.
	Hugh	Loebner	has	a	“Loebner	Prize”	competition	that	is	run	each	year.
The	Loebner	silver	medal	will	go	to	a	computer	that	passes	Turing’s
original	text-only	test.	The	gold	medal	will	go	to	a	computer	that	can
pass	a	version	of	the	test	that	includes	audio	and	video	input	and
output.	In	my	view,	the	inclusion	of	audio	and	video	does	not	actually
make	the	test	more	challenging.
15.
	“Cognitive	Assistant	That	Learns	and	Organizes,”	Artificial	Intelligence
Center,	SRI	International,	http://www.ai.sri.com/project/CALO.

15.
16.
	Dragon	Go!	Nuance	Communications,	Inc.,
http://www.nuance.com/products/dragon-go-in-action/index.htm.
17.
	“Overcoming	Artificial	Stupidity,”	
WolframAlpha	Blog
,	April	17,	2012,
http://blog.wolframalpha.com/author/stephenwolfram/.

Chapter	8:	The	Mind	as	Computer
1.
	Salomon	Bochner,	
A	Biographical	Memoir	of	John	von	Neumann
(Washington,	DC:	National	Academy	of	Sciences,	1958).
2.
	A.	M.	Turing,	“On	Computable	Numbers,	with	an	Application	to	the
Entscheidungsproblem,”	
Proceedings	of	the	London	Mathematical
Society
	Series	2,	vol.	42	(1936–37):	230–65,
http://www.comlab.ox.ac.uk/activities/ieg/e-library/sources/tp2-
ie.pdf.	A.	M.	Turing,	“On	Computable	Numbers,	with	an
Application	to	the	Entscheidungsproblem:	A	Correction,”
Proceedings	of	the	London	Mathematical	Society
	43	(1938):	544–
46.
3.
	John	von	Neumann,	“First	Draft	of	a	Report	on	the	EDVAC,”
Moore	School	of	Electrical	Engineering,	University	of
Pennsylvania,	June	30,	1945.	John	von	Neumann,	“A	Mathematical
Theory	of	Communication,”	
Bell	System	Technical	Journal
,	July
and	October	1948.
4.
	Jeremy	Bernstein,	
The	Analytical	Engine:	Computers—Past,
Present,	and	Future
,	rev.	ed.	(New	York:	William	Morrow	&	Co.,
1981).
5.

46.
3.
,	July
4.
1981).
5.
	“Japan’s	K	Computer	Tops	10	Petaflop/s	to	Stay	Atop	TOP500
List,”	
Top	500
,	November	11,	2011,
http://top500.org/lists/2011/11/pressrelease.
6.
	Carver	Mead,	
Analog	VLSI	and	Neural	Systems
	(Reading,	MA:
Addison-Wesley,	1986).
7.
	“IBM	Unveils	Cognitive	Computing	Chips,”	IBM	news	release,
August	 18,	 2011,	 http://www-
03.ibm.com/press/us/en/pressrelease/35251.wss.
8.
List.”

Chapter	9:	Thought	Experiments	on	the	Mind
1.
	John	R.	Searle,	“I	Married	a	Computer,”	in	Jay	W.	Richards,	ed.,
Are	We	Spiritual	Machines?	Ray	Kurzweil	vs.	the	Critics	of	Strong
AI
	(Seattle:	Discovery	Institute,	2002).
2.
	Stuart	Hameroff,	
Ultimate	Computing:	Biomolecular	Consciousness
and	Nanotechnology
	(Amsterdam:	Elsevier	Science,	1987).
3.
	P.	S.	Sebel	et	al.,	“The	Incidence	of	Awareness	during	Anesthesia:	A
Multicenter	United	States	Study,”	
Anesthesia	and	Analgesia
	99
(2004):	833–39.
4.
	Stuart	Sutherland,	
The	International	Dictionary	of	Psychology
	(New
York:	Macmillan,	1990).
5.
	David	Cockburn,	“Human	Beings	and	Giant	Squids,”	
69,	no.	268	(April	1994):	135–50.
6.
	Ivan	Petrovich	Pavlov,	from	a	lecture	given	in	1913,	published	in
Lectures	on	Conditioned	Reflexes:	Twenty-Five	Years	of	Objective
Study	of	the	Higher	Nervous	Activity	[Behavior]	of	Animals
(London:	Martin	Lawrence,	1928),	222.
7.
	Roger	W.	Sperry,	from	James	Arthur	Lecture	on	the	Evolution	of	the
Human	Brain,	1964,	p.	2.

	(New
5.
6.
7.
8.
	Henry	Maudsley,	“The	Double	Brain,”	
Mind
	14,	no.	54	(1889):
161–87.
9.
	Susan	Curtiss	and	Stella	de	Bode,	“Language	after
Hemispherectomy,”	
Brain	and	Cogn
ition	43,	nos.	1–3	(June–August
2000):	135–38.
10.
	E.	P.	Vining	et	al.,	“Why	Would	You	Remove	Half	a	Brain?	The
Outcome	of	58	Children	after	Hemispherectomy—the	Johns
Hopkins	Experience:	1968	to	1996,”	Pediatrics	100	(August
1997):	163–71.	M.	B.	Pulsifer	et	al.,	“The	Cognitive	Outcome	of
Hemispherectomy	in	71	Children,”	
Epilepsia
	45,	no.	3	(March

Henry	Maudsley,	“The	Double	Brain,”	
Mind
161–87.
9.
10.
Epilepsia
2004):	243–54.
11.
	S.	McClelland	III	and	R.	E.	Maxwell,	“Hemispherectomy	for
Intractable	Epilepsy	in	Adults:	The	First	Reported	Series,”	
Annals
of	Neurology
	61,	no.	4	(April	2007):	372–76.
12.
	Lars	Muckli,	Marcus	J.	Naumerd,	and	Wolf	Singer,	“Bilateral
Visual	Field	Maps	in	a	Patient	with	Only	One	Hemisphere,”

	106,	no.	31
(August	4,	2009),	http://dx.doi.org/10.1073/pnas.0809688106.
13.
	Marvin	Minsky,	
	(New	York:	Simon	and
Schuster,	1988).
14.
	F.	Fay	Evans-Martin,	
The	Nervous	System
	(New	York:	Chelsea
House,	2005),	http://www.scribd.com/doc/5012597/The-Nervous-
System.
15.
	Benjamin	Libet,	
Mind	Time:	The	Temporal	Factor	in
	(Cambridge,	MA:	Harvard	University	Press,
2005).
16.
	Daniel	C.	Dennett,	
Freedom	Evolves
	(New	York:	Viking,	2003).
17.
	Michael	S.	Gazzaniga,	
Who’s	in	Charge?	Free	Will	and	the	Science
of	the	Brain
	(New	York:	Ecco/HarperCollins,	2011).
18.
	David	Hume,	
An	Enquiry	Concerning	Human	Understanding
(1765),	2nd	ed.,	edited	by	Eric	Steinberg	(Indianapolis:	Hackett,
1993).
19.
	Arthur	Schopenhauer,	
The	Wisdom	of	Life
.
20.
On	the	Freedom	of	the	Will
	(1839).
21.
	From	Raymond	Smullyan,	
5000	B.C.	and	Other	Philosophical
Fantasies
	(New	York:	St.	Martin’s	Press,	1983).
22.

(New	York:	Viking,	2003).
17.
18.
1993).
19.
.
20.
	(1839).
21.
Fantasies
22.
	For	an	insightful	and	entertaining	examination	of	similar	issues	of
identity	and	consciousness,	see	Martine	Rothblatt,	“The	Terasem
Mind	Uploading	Experiment,”	
International	Journal	of	Machine
	4,	no.	1	(2012):	141–58.	In	this	paper,	Rothblatt
examines	the	issue	of	identity	with	regard	to	software	that
emulates	a	person	based	on	“a	database	of	video	interviews	and
associated	information	about	a	predecessor	person.”	In	this
proposed	future	experiment,	the	software	is	successfully

22.
emulating	the	person	it	is	based	on.
23.
	“How	Do	You	Persist	When	Your	Molecules	Don’t?”	
Science	and
Consciousness	Review
	1,	no.	1	(June	2004),	http://www.sci-
con.org/articles/20040601.xhtml.

Chapter	10:	The	Law	of	Accelerating	Returns
Applied	to	the	Brain
1.
	“DNA	Sequencing	Costs,”	National	Human	Genome	Research
Institute,	NIH,	http://www.genome.gov/sequencingcosts/.
2.
	“Genetic	Sequence	Data	Bank,	Distribution	Release	Notes,”
December	15,	2009,	National	Center	for	Biotechnology
Information,	National	Library	of	Medicine,
ftp://ftp.ncbi.nih.gov/genbank/gbrel.txt.
3.
	“DNA	Sequencing—The	History	of	DNA	Sequencing,”	January	2,
2012,	http://www.dnasequencing.org/history-of-dna.
4.
	 “Cooper’s	 Law,”	 ArrayComm,
http://www.arraycomm.com/technology/coopers-law.
5.
	 “The	 Zettabyte	 Era,”	 Cisco,
http://www.cisco.com/en/US/solutions/collateral/ns341/ns525/ns537/ns705/ns827/VNI_Hyperconnectivity_WP.xhtml,
and	“Number	of	Internet	Hosts,”	Internet	Systems	Consortium,
http://www.isc.org/solutions/survey/history.
6.
	TeleGeography	©	PriMetrica,	Inc.,	2012.
7.
	Dave	Kristula,	“The	History	of	the	Internet”	(March	1997,	update
August	2001),	http://www.davesite.com/webstation/net-

4.
5.
6.
7.
history.shtml;	Robert	Zakon,	“Hobbes’	Internet	Timeline	v8.0,”
http://www.zakon.org/robert/internet/timeline;	 Quest
Communications,	8-K	for	9/13/1998	EX-99.1;	
Converge!	Network
Digest
,	 December	 5,	 2002,
http://www.convergedigest.com/Daily/daily.asp?
vn=v9n229&fecha=December%2005,%202002;	Jim	Duffy,	“AT&T
Plans	Backbone	Upgrade	to	40G,”	
Computerworld
,	June	7,	2006,
http://www.computerworld.com/action/article.do?
command=viewArticleBasic&articleId=9001032;	
“40G:	The	Fastest

Digest
Connection	You	Can	Get?”	InternetNews.com,	November	2,	2007,
http://www.internetnews.com/infra/article.php/3708936;	“Verizon
First	Global	Service	Provider	to	Deploy	100G	on	U.S.	Long-Haul
Network,”	 news	 release,	 Verizon,
http://newscenter.verizon.com/press-releases/verizon/2011/verizon-
first-global-service.xhtml.

8.
	 Facebook,	 “Key	 Facts,”
http://newsroom.fb.com/content/default.aspx?NewsAreaId=22.
9.
	http://www.kurzweilai.net/how-my-predictions-are-faring.
10.
	
Calculations	per	Second	per	$1,000

11.
	Top	500	Supercomputer	Sites,	http://top500.org/.
12.
	“Microprocessor	Quick	Reference	Guide,”	Intel	Research,
http://www.intel.com/pressroom/kits/quickreffam.htm.
13.
	1971–2000:	VLSI	Research	Inc.
2001–2006:	
The	International	Technology	Roadmap	for
Semiconductors
,	2002	Update	and	2004	Update,	Table	7a,	“Cost—
Near-term	Years,”	“DRAM	cost/bit	at	(packaged	microcents)	at
production.”
2007–2008:	
,	2007,	Tables	7a	and	7b,	“Cost—Near-term	Years,”
“Cost—Long-term	 Years,”
http://www.itrs.net/Links/2007ITRS/ExecSum2007.pdf.
2009–2022:	
,	2009,	Tables	7a	and	7b,	“Cost—Near-term	Years,”
http://www.itrs.net/Links/2009ITRS/Home2009.htm.
14.
	To	make	all	dollar	values	comparable,	computer	prices	for	all	years
were	converted	to	their	year	2000	dollar	equivalent	using	the
Federal	Reserve	Board’s	CPI	data	at
http://minneapolisfed.org/research/data/us/calc/.	For	example,	$1

million	in	1960	is	equivalent	to	$5.8	million	in	2000,	and	$1
million	in	2004	is	equivalent	to	$0.91	million	in	2000.
1949:
http://www.cl.cam.ac.uk/UoCCL/misc/EDSAC99/statistics.xhtml,
http://www.davros.org/misc/chronology.xhtml.
1951:	Richard	E.	Matick,	
Computer	Storage	Systems	and
	(New	York:	John	Wiley	&	Sons,	1977);
http://inventors.about.com/library/weekly/aa062398.htm.
1955:	Matick,	
Computer	Storage	Systems	and	Technology;
OECD,	1968,	http://members.iinet.net.au/~dgreen/timeline.xhtml.
1960:	 ftp://rtfm.mit.edu/pub/usenet/alt.sys.pdp8/PDP-
8_Frequently_Asked_Questions_%28posted_every_other_month%29;
http://www.dbit.com/~greeng3/pdp1/pdp1.xhtml#INTRODUCTION.
1962:	 ftp://rtfm.mit.edu/pub/usenet/alt.sys.pdp8/PDP-
8_Frequently_Asked_Questions_%28posted_every_other_month%29.
1964:	Matick,	
http://www.research.microsoft.com/users/gbell/craytalk;
http://www.ddj.com/documents/s=1493/ddj	0005hc/.
1965:	Matick,	

http://www.fourmilab.ch/documents/univac/config1108.xhtml;
http://www.frobenius.com/univac.htm.
1968:	Data	General.
1969,	 1970:
http://www.eetimes.com/special/special_issues/millennium/mile
stones/whittier.xhtml.
1974:	Scientific	Electronic	Biological	Computer	Consulting
(SCELBI).
1975–1996:	
Byte
	magazine	advertisements.
1997–2000:	
PC	Computing
2001:	www.pricewatch.com	(http://www.jc-news.com/parse.cgi?
news/pricewatch/raw/pw-010702).

(SCELBI).
Byte
2002:	www.pricewatch.com	(http://www.jc-news.com/parse.cgi?
news/price	watch/raw/pw-020624).
2003:
http://sharkyextreme.com/guides/WMPG/article.php/10706_2227191_2.
2004:	http://www.pricewatch.com	(11/17/04).
2008:	http://www.pricewatch.com	(10/02/08)	($16.61).
15.
	Dataquest/Intel	and	Pathfinder	Research:
Year
$
Log	($)

2001
0.00000035
−14.8653
2002
0.00000026
−15.1626
2003
0.00000017
−15.5875
2004
0.00000012
−15.9358
2005
0.000000081
−16.3288
2006
0.000000063
−16.5801
2007
0.000000024
−17.5452
2008
0.000000016
−17.9507
	
16.
	Steve	Cullen,	In-Stat,	September	2008,	www.instat.com.
Year
Mbits
Bits
1971
921.6
9.216E+08
1972
3788.8
3.789E+09
1973
8294.4
8.294E+09
1974
19865.6
1.987E+10
1975
42700.8
4.270E+10
1976
130662.4
1.307E+11
1977
276070.4
2.761E+11
1978
663859.2
6.639E+11
1979
1438720.0
1.439E+12
1980
3172761.6
3.173E+12
1981
4512665.6
4.513E+12
1982
11520409.6
1.152E+13
1983
29648486.4
2.965E+13
1984
68418764.8
6.842E+13
1985
87518412.8
8.752E+13
1986
192407142.4
1.924E+14
1987
255608422.4
2.556E+14
1988
429404979.2
4.294E+14
1989
631957094.4
6.320E+14
1990
950593126.4
9.506E+14
1991
1546590618
1.547E+15
1992
2845638656
2.846E+15

1993
4177959322
4.178E+15
1994
7510805709
7.511E+15
1995
13010599936
1.301E+16
1996
23359078007
2.336E+16
1997
45653879161
4.565E+16
1998
85176878105
8.518E+16
1999
1.47327E+11
1.473E+17
2000
2.63636E+11
2.636E+17
2001
4.19672E+11
4.197E+17
2002
5.90009E+11
5.900E+17
2003
8.23015E+11
8.230E+17
2004
1.32133E+12
1.321E+18
2005
1.9946E+12
1.995E+18
2006
2.94507E+12
2.945E+18
2007
5.62814E+12
5.628E+18
	
17.
	“Historical	Notes	about	the	Cost	of	Hard	Drive	Storage	Space,”
http://www.littletechshoppe.com/ns1625/winchest.xhtml;	
Byte
magazine	advertisements,	1977–1998;	
	magazine
advertisements,	3/1999;	
Understanding
	
Computers:	Memory	and
Storage
	(New	York:	Time	Life,	1990);
http://www.cedmagic.com/history/ibm-305-ramac.xhtml;	John	C.
McCallum,	“Disk	Drive	Prices	(1955–2012),”
http://www.jcmit.com/diskprice.htm;	IBM,	“Frequently	Asked
Questions,”	 http://www-
03.ibm.com/ibm/history/documents/pdf/faq.pdf;	IBM,	“IBM	355
Disk	 Storage	 Unit,”	 http://www-

Byte
	magazine
	
Storage
03.ibm.com/ibm/history/exhibits/storage/storage_355.xhtml;
IBM,	“IBM	3380	Direct	Access	Storage	Device,”	http://www.03-
ibm.com/ibm/history/exhibits/storage/storage_3380.xhtml.
18.
	“Without	Driver	or	Map,	Vans	Go	from	Italy	to	China,”	
Sydney
Morning	Herald
,	October	29,	2010,
http://www.smh.com.au/technology/technology-news/without-
driver-or-map-vans-go-from-italy-to-china-20101029-
176ja.xhtml.

19.
	KurzweilAI.net.
20.
	Adapted	with	permission	from	Amiram	Grinvald	and	Rina
Hildesheim,	“VSDI:	A	New	Era	in	Functional	Imaging	of
Cortical	Dynamics,”	
	5	(November
2004):	874–85.
The	main	tools	for	imaging	the	brain	are	shown	in	this	diagram.
Their	capabilities	are	depicted	by	the	shaded	rectangles.
Spatial	resolution	refers	to	the	smallest	dimension	that	can	be
measured	with	a	technique.	Temporal	resolution	is	imaging	time	or
duration.	There	are	tradeoffs	with	each	technique.	For	example,	EEG
(electroencephalography),	which	measures	“brain	waves”	(electrical
signals	from	neurons),	can	measure	very	rapid	brain	waves	(occurring
in	short	time	intervals),	but	can	only	sense	signals	near	the	surface	of
the	brain.
In	contrast,	fMRI	(functional	magnetic	resonance	imaging),
which	uses	a	special	MRI	machine	to	measure	blood	flow	to	neurons
(indicating	neuron	activity),	can	sense	a	lot	deeper	in	the	brain	(and
spinal	cord)	and	with	higher	resolution,	down	to	tens	of	microns

the	brain.
(millionths	of	a	meter).	However,	fMRI	operates	very	slowly
compared	with	EEG.
These	are	noninvasive	techniques	(no	surgery	or	drugs	are
required).	MEG	(magnetoencephalography)	is	another	noninvasive
technique.	It	detects	magnetic	fields	generated	by	neurons.	MEG	and
EEG	can	resolve	events	with	a	temporal	resolution	of	down	to	1
millisecond,	but	better	than	fMRI,	which	can	at	best	resolve	events
with	a	resolution	of	several	hundred	milliseconds.	MEG	also
accurately	pinpoints	sources	in	primary	auditory,	somatosensory,	and
motor	areas.

Optical	imaging	covers	almost	the	entire	range	of	spatial	and
temporal	resolutions,	but	is	invasive.	VSDI	(voltage-sensitive	dyes)	is
the	most	sensitive	method	of	measuring	brain	activity,	but	is	limited	to
measurements	near	the	surface	of	the	cortex	of	animals.
The	exposed	cortex	is	covered	with	a	transparent	sealed	chamber;
after	the	cortex	is	stained	with	a	suitable	voltage-sensitive	dye,	it	is
illuminated	with	light	and	a	sequence	of	images	is	taken	with	a	high-
speed	camera.	Other	optical	techniques	used	in	the	lab	include	ion

imaging	(typically	calcium	or	sodium	ions)	and	fluorescence	imaging
systems	(confocal	imaging	and	multiphoton	imaging).
Other	lab	techniques	include	PET	(positron	emission	tomography,

a	nuclear	medicine	imaging	technique	that	produces	a	3-D	image),
2DG	(2-deoxyglucose	postmortem	histology,	or	tissue	analysis),
lesions	(involves	damaging	neurons	in	an	animal	and	observing	the
effects),	patch	clamping	(to	measure	ion	currents	across	biological
membranes),	and	electron	microscopy	(using	an	electron	beam	to
examine	tissues	or	cells	at	a	very	fine	scale).	These	techniques	can	also
be	integrated	with	optical	imaging.
21.
	MRI	spatial	resolution	in	microns	(μm),	1980–2012:
22.
	Spatial	resolution	in	nanometers	(nm)	of	destructive	imaging
techniques,	1983–2011:

23.
	Spatial	resolution	in	microns	(μm)	of	nondestructive	imaging
techniques	in	animals,	1985–2012:
Year
Finding
	
2012
Resolution
0.07
	
Citation
Sebastian	Berning	et	al.,	“Nanoscopy	in	a	Living	Mouse
Brain,”	
Science
	335,	no.	6068	(February	3,	2012):	551.
	
URL
http://dx.doi.org/10.1126/science.1215369
	
Technique
Stimulated	emission	depletion	(STED)	fluorescence
nanoscopy

Notes
Highest	resolution	achieved	in	vivo	so	far
2012
0.25
	
Citation
Brain,”	
Science
	
URL
	
Technique
Confocal	and	multiphoton	microscopy
2004
50
	
Citation
Amiram	Grinvald	and	Rina	Hildesheim,	“VSDI:	A	New
Era	in	Functional	Imaging	of	Cortical	Dynamics,”	
Nature
Reviews	Neuroscience
	5	(November	2004):	874–85.
	
URL
http://dx.doi.org/10.1038/nrn1536
	
Technique
Imaging	based	on	voltage-sensitive	dyes	(VSDI)
	
Notes
“VSDI	has	provided	high-resolution	maps,	which
correspond	to	cortical	columns	in	which	spiking	occurs,
and	offer	a	spatial	resolution	better	than	50	μm.”
1996
50
	
Citation
Dov	Malonek	and	Amiram	Grinvald,	“Interactions
between	Electrical	Activity	and	Cortical	Microcirculation
Revealed	by	Imaging	Spectroscopy:	Implications	for
Functional	Brain	Mapping,”	
Science
	272,	no.	5261	(April
26,	1996):	551–54.
	
URL

Technique
	
Notes
1996
50
	
Citation
Science
	
URL
http://dx.doi.org/10.1126/science.272.5261.551
	
Technique
Imaging	spectroscopy
	
Notes
“The	study	of	spatial	relationships	between	individual
cortical	columns	within	a	given	brain	area	has	become
feasible	with	optical	imaging	based	on	intrinsic	signals,	at
a	spatial	resolution	of	about	50	μm.”
1995
50
	
Citation
D.	H.	Turnbull	et	al.,	“Ultrasound	Backscatter	Microscope
Analysis	of	Early	Mouse	Embryonic	Brain
Development,”	
Sciences
	92,	no.	6	(March	14,	1995):	2239–43.
	
URL

Technique
	
Notes
1995
50
	
Citation
Sciences
	
URL
http://www.pnas.org/content/92/6/2239.short
	
Technique
Ultrasound	backscatter	microscopy
“We	demonstrate	application	of	a	real-time	imaging

Notes
method	called	ultrasound	backscatter	microscopy	for
visualizing	mouse	early	embryonic	neural	tubes	and
hearts.	This	method	was	used	to	study	live	embryos	in
utero	between	9.5	and	11.5	days	of	embryogenesis,	with	a
spatial	resolution	close	to	50	μm.”
1985
500
	
Citation
H.	S.	Orbach,	L.	B.	Cohen,	and	A.	Grinvald,	“Optical
Mapping	of	Electrical	Activity	in	Rat	Somatosensory	and
Visual	Cortex,”	
	5,	no.	7	(July	1,
1985):	1886–95.
	
URL
http://www.jneurosci.org/content/5/7/1886.short
	
Technique
Optical	methods

Chapter	11:	Objections
1.
	Paul	G.	Allen	and	Mark	Greaves,	“Paul	Allen:	The	Singularity	Isn’t
Near,”	
,	October	12,	2011,
http://www.technologyreview.com/blog/guest/27206/.
2.
	ITRS,	“International	Technology	Roadmap	for	Semiconductors,”
http://www.itrs.net/Links/2011ITRS/Home2011.htm.
3.
	Ray	Kurzweil,	
	(New	York:	Viking,	2005),
chapter	2.
4.
	Endnote	2	in	Allen	and	Greaves,	“The	Singularity	Isn’t	Near,”	reads
as	follows:	“We	are	beginning	to	get	within	range	of	the	computer
power	we	might	need	to	support	this	kind	of	massive	brain
simulation.	Petaflop-class	computers	(such	as	IBM’s	BlueGene/P
that	was	used	in	the	Watson	system)	are	now	available
commercially.	Exaflop-class	computers	are	currently	on	the	drawing
boards.	These	systems	could	probably	deploy	the	raw	computational
capability	needed	to	simulate	the	firing	patterns	for	all	of	a	brain’s
neurons,	though	currently	it	happens	many	times	more	slowly	than
would	happen	in	an	actual	brain.”
5.
	Kurzweil,

5.
	Kurzweil,	
,	chapter	9,	section	titled	“The
Criticism	from	Software”	(pp.	435–42).
6.
	Ibid.,	chapter	9.
7.
	Although	it	is	not	possible	to	precisely	determine	the	information
content	in	the	genome,	because	of	the	repeated	base	pairs	it	is
clearly	much	less	than	the	total	uncompressed	data.	Here	are	two
approaches	to	estimating	the	compressed	information	content	of	the
genome,	both	of	which	demonstrate	that	a	range	of	30	to	100
million	bytes	is	conservatively	high.

5.
	Kurzweil,	
6.
7.
1.	In	terms	of	the	uncompressed	data,	there	are	3	billion	DNA
rungs	in	the	human	genetic	code,	each	coding	2	bits	(since	there	are
four	possibilities	for	each	DNA	base	pair).	Thus	the	human	genome	is
about	800	million	bytes	uncompressed.	The	noncoding	DNA	used	to
be	called	“junk	DNA,”	but	it	is	now	clear	that	it	plays	an	important
role	in	gene	expression.	However,	it	is	very	inefficiently	coded.	For
one	thing,	there	are	massive	redundancies	(for	example,	the	sequence

called	“ALU”	is	repeated	hundreds	of	thousands	of	times),	which

compression	algorithms	can	take	advantage	of.
With	the	recent	explosion	of	genetic	data	banks,	there	is	a	great
deal	of	interest	in	compressing	genetic	data.	Recent	work	on	applying
standard	data	compression	algorithms	to	genetic	data	indicates	that
reducing	the	data	by	90	percent	(for	bit	perfect	compression)	is
feasible:	Hisahiko	Sato	et	al.,	“DNA	Data	Compression	in	the	Post
Genome	Era,”	
Genome	Informatics
	12	(2001):	512–14,
http://www.jsbi.org/journal/GIW01/GIW01P130.pdf.
Thus	we	can	compress	the	genome	to	about	80	million	bytes
without	loss	of	information	(meaning	we	can	perfectly	reconstruct	the
full	800-million-byte	uncompressed	genome).
Now	consider	that	more	than	98	percent	of	the	genome	does	not
code	for	proteins.	Even	after	standard	data	compression	(which
eliminates	redundancies	and	uses	a	dictionary	lookup	for	common
sequences),	the	algorithmic	content	of	the	noncoding	regions	appears
to	be	rather	low,	meaning	that	it	is	likely	that	we	could	code	an

algorithm	that	would	perform	the	same	function	with	fewer	bits.
However,	since	we	are	still	early	in	the	process	of	reverse-engineering
the	genome,	we	cannot	make	a	reliable	estimate	of	this	further
decrease	based	on	a	functionally	equivalent	algorithm.	I	am	using,
therefore,	a	range	of	30	to	100	million	bytes	of	compressed
information	in	the	genome.	The	top	part	of	this	range	assumes	only
data	compression	and	no	algorithmic	simplification.
Only	a	portion	(although	the	majority)	of	this	information

characterizes	the	design	of	the	brain.
2.	Another	line	of	reasoning	is	as	follows.	Though	the	human
genome	contains	around	3	billion	bases,	only	a	small	percentage,	as
mentioned	above,	codes	for	proteins.	By	current	estimates,	there	are
26,000	genes	that	code	for	proteins.	If	we	assume	those	genes	average
3,000	bases	of	useful	data,	those	equal	only	approximately	78	million
bases.	A	base	of	DNA	requires	only	2	bits,	which	translate	to	about	20
million	bytes	(78	million	bases	divided	by	four).	In	the	protein-coding

sequence	of	a	gene,	each	“word”	(codon)	of	three	DNA	bases
translates	into	one	amino	acid.	There	are,	therefore,	4
3
	(64)	possible
codon	codes,	each	consisting	of	three	DNA	bases.	There	are,	however,
only	20	amino	acids	used	plus	a	stop	codon	(null	amino	acid)	out	of
the	64.	The	rest	of	the	43	codes	are	used	as	synonyms	of	the	21	useful
ones.	Whereas	6	bits	are	required	to	code	for	64	possible

combinations,	only	about	4.4	(log
2
	21)	bits	are	required	to	code	for	21
possibilities,	a	savings	of	1.6	out	of	6	bits	(about	27	percent),	bringing
us	down	to	about	15	million	bytes.	In	addition,	some	standard
compression	based	on	repeating	sequences	is	feasible	here,	although
much	less	compression	is	possible	on	this	protein-coding	portion	of	the
DNA	than	in	the	so-called	junk	DNA,	which	has	massive
redundancies.	So	this	will	bring	the	figure	probably	below	12	million
bytes.	However,	now	we	have	to	add	information	for	the	noncoding
portion	of	the	DNA	that	controls	gene	expression.	Although	this
portion	of	the	DNA	constitutes	the	bulk	of	the	genome,	it	appears	to
have	a	low	level	of	information	content	and	is	replete	with	massive
redundancies.	Estimating	that	it	matches	the	approximately	12	million
bytes	of	protein-coding	DNA,	we	again	come	to	approximately	24
million	bytes.	From	this	perspective,	an	estimate	of	30	to	100	million
bytes	is	conservatively	high.
8.

8.
	Dharmendra	S.	Modha	et	al.,	“Cognitive	Computing,”
	54,	no.	8	(2011):	62–71,
http://cacm.acm.org/magazines/2011/8/114944-cognitive-
computing/fulltext.
9.
	Kurzweil,	
Criticism	from	Ontology:	Can	a	Computer	Be	Conscious?”	(pp.
458–69).
10.
	Michael	Denton,	“Organism	and	Machine:	The	Flawed	Analogy,”
in	
Are	We	Spiritual	Machines?	Ray	Kurzweil	vs.	the	Critics	of
Strong	AI
11.
	Hans	Moravec,

9.
	Kurzweil,	
458–69).
10.
in	
Strong	AI
11.
	Hans	Moravec,	
University	Press,	1988).

Epilogue
1.
	“In	U.S.,	Optimism	about	Future	for	Youth	Reaches	All-Time	Low,”
Gallup	Politics,	May	2,	2011,
http://www.gallup.com/poll/147350/optimism-future-youth-reaches-
time-low.aspx.
2.
	James	C.	Riley,	
Rising	Life	Expectancy:	A	Global	History
(Cambridge:	Cambridge	University	Press,	2001).
3.
	J.	Bradford	DeLong,	“Estimating	World	GDP,	One	Million	B.C.—
Present,”	 May	 24,	 1998,
http://econ161.berkeley.edu/TCEH/1998_Draft/World_GDP/Estimating_World_GDP.xhtml,
and	http://futurist.typepad.com/my_weblog/2007/07/economic-
growth.xhtml.	See	also	Peter	H.	Diamandis	and	Steven	Kotler,
Abundance:	The	Future	Is	Better	Than	You	Think
	(New	York:	Free
Press,	2012).
4.
	Martine	Rothblatt,	
Transgender	to	Transhuman
	(privately	printed,
2011).	She	explains	how	a	similarly	rapid	trajectory	of	acceptance	is
most	likely	to	occur	for	“transhumans,”	for	example,	nonbiological
but	convincingly	conscious	minds	as	discussed	in	chapter	9.
5.
	The	following	excerpt	from	
,	chapter	3	(pp.

4.
5.
133–35),	by	Ray	Kurzweil	(New	York:	Viking,	2005),	discusses	the
limits	of	computation	based	on	the	laws	of	physics:
The	ultimate	limits	of	computers	are	profoundly	high.	Building
on	work	by	University	of	California	at	Berkeley	Professor	Hans
Bremermann	and	nanotechnology	theorist	Robert	Freitas,	MIT
Professor	Seth	Lloyd	has	estimated	the	maximum	computational
capacity,	according	to	the	known	laws	of	physics,	of	a	computer
weighing	one	kilogram	and	occupying	one	liter	of	volume—about	the

size	and	weight	of	a	small	laptop	computer—what	he	calls	the
“ultimate	laptop.”
[Note:	Seth	Lloyd,	“Ultimate	Physical	Limits	to	Computation,”
Nature
	406	(2000):	1047–54.
[Early	work	on	the	limits	of	computation	were	done	by	Hans	J.
Bremermann	in	1962:	Hans	J.	Bremermann,	“Optimization	Through
Evolution	and	Recombination,”	in	M.	C.	Yovits,	C.	T.	Jacobi,	C.	D.

Goldstein,	eds.,	
Self-Organizing	Systems
	(Washington,	D.C.:	Spartan
Books,	1962),	pp.	93–106.
[In	1984	Robert	A.	Freitas	Jr.	built	on	Bremermann’s	work	in
Robert	A.	Freitas	Jr.,	“Xenopsychology,”	
Analog
	104	(April	1984):
41–53,
http://www.rfreitas.com/Astro/Xenopsychology.htm#SentienceQuotient.]
The	potential	amount	of	computation	rises	with	the	available
energy.	We	can	understand	the	link	between	energy	and	computational
capacity	as	follows.	The	energy	in	a	quantity	of	matter	is	the	energy
associated	with	each	atom	(and	subatomic	particle).	So	the	more
atoms,	the	more	energy.	As	discussed	above,	each	atom	can	potentially
be	used	for	computation.	So	the	
more	atoms,	the	more	computation.
The	energy	of	each	atom	or	particle	grows	with	the	frequency	of	its
movement:	the	more	movement,	the	more	energy.	The	same
relationship	exists	for	potential	computation:	the	higher	the	frequency
of	movement,	the	more	computation	each	component	(which	can	be	an

atom)	can	perform.	(We	see	this	in	contemporary	chips:	the	higher	the
frequency	of	the	chip,	the	greater	its	computational	speed.)
So	there	is	a	direct	proportional	relationship	between	the	energy
of	an	object	and	its	potential	to	perform	computation.	The	potential
energy	in	a	kilogram	of	matter	is	very	large,	as	we	know	from
Einstein’s	equation	
E
	=	
mc
2
.	The	speed	of	light	squared	is	a	very	large
number:	approximately	10
17
	meter
2
/second
2
.	The	potential	of	matter	to
compute	is	also	governed	by	a	very	small	number,	Planck’s	constant:
6.6	×	10
−

E
	=	
mc
2
17
	meter
2
/second
2
6.6	×	10
−
34
	joule-seconds	(a	joule	is	a	measure	of	energy).	This	is	the
smallest	scale	at	which	we	can	apply	energy	for	computation.	We
obtain	the	theoretical	limit	of	an	object	to	perform	computation	by
dividing	the	total	energy	(the	average	energy	of	each	atom	or	particle
times	the	number	of	such	particles)	by	Planck’s	constant.
Lloyd	shows	how	the	potential	computing	capacity	of	a	kilogram
of	matter	equals	pi	times	energy	divided	by	Planck’s	constant.	Since
the	energy	is	such	a	large	number	and	Planck’s	constant	is	so	small,

this	equation	generates	an	extremely	large	number:	about	5	×	10
50
operations	per	second.
[Note:	π	×	maximum	energy	(10
17
	kg	×	meter
2
/second
2
)	/	(6.6	×
10
–34
)	joule-seconds)	=	~	5	×	10
50
	operations/second.]
If	we	relate	that	figure	to	the	most	conservative	estimate	of
human	brain	capacity	(10
19
	cps	and	10
10
	humans),	it	represents	the

equivalent	of	about	5	billion	trillion	human	civilizations.
[Note:	5	×	10
50
	cps	is	equivalent	to	5	×	10
21
	(5	billion	trillion)
human	civilizations	(each	requiring	10
29
	cps).]
If	we	use	the	figure	of	10
16
	cps	that	I	believe	will	be	sufficient	for
functional	emulation	of	human	intelligence,	the	ultimate	laptop	would
function	at	the	equivalent	brain	power	of	5	trillion	trillion	human
civilizations.
[Note:	Ten	billion	(10
10
)	humans	at	10
16
	cps	each	is	10
26
	cps	for
human	civilization.	So	5	×	10
50
24
	(5	trillion
trillion)	human	civilizations.]
Such	a	laptop	could	perform	the	equivalent	of	all	human	thought
over	the	last	ten	thousand	years	(that	is,	ten	billion	human	brains
operating	for	ten	thousand	years)	in	one	ten-thousandth	of	a
nanosecond.
[Note:	This	estimate	makes	the	conservative	assumption	that
we’ve	had	ten	billion	humans	for	the	past	ten	thousand	years,	which	is
obviously	not	the	case.	The	actual	number	of	humans	has	been

50
24
increasing	gradually	over	the	past	to	reach	about	6.1	billion	in	2000.
There	are	3	×	10
7
	seconds	in	a	year,	
and	3	×	10
11
	seconds	in	ten
thousand	years.	So,	using	the	estimate	of	10
26
	cps	for	human
civilization,	human	thought	over	ten	thousand	years	is	equivalent	to
certainly	no	more	than	3	×	10
37
	calculations.	The	ultimate	laptop
performs	5	×	10
50
	calculations	in	one	second.	So	simulating	ten
thousand	years	of	ten	billion	humans’	thoughts	would	take	it	about	10
–
13

7
and	3	×	10
11
26
37
50
–
13
	seconds,	which	is	one	ten-thousandth	of	a	nanosecond.]
Again,	a	few	caveats	are	in	order.	Converting	all	of	the	mass	of
our	2.2-pound	laptop	into	energy	is	essentially	what	happens	in	a
thermonuclear	explosion.	Of	course,	we	don’t	want	the	laptop	to
explode	but	to	stay	within	its	one-liter	dimension.	So	this	will	require
some	careful	packaging,	to	say	the	least.	By	analyzing	the	maximum
entropy	(degrees	of	freedom	represented	by	the	state	of	all	the
particles)	in	such	a	device,	Lloyd	shows	that	such	a	computer	would

have	a	theoretical	memory	capacity	of	10
31
	bits.	It’s	difficult	to
imagine	technologies	that	would	go	all	the	way	in	achieving	these
limits.	But	we	can	readily	envision	technologies	that	come	reasonably
close	to	doing	so.	As	the	University	of	Oklahoma	project	shows,	we
already	demonstrated	the	ability	to	store	at	least	fifty	bits	of
information	per	atom	(although	only	on	a	small	number	of	atoms,	so

far).	Storing	10
27
	bits	of	memory	in	the	10
25
	atoms	in	a	kilogram	of
matter	should	therefore	be	eventually	achievable.
But	because	many	properties	of	each	atom	could	be	exploited	to
store	information—such	as	the	precise	position,	spin,	and	quantum
state	of	all	of	its	particles—we	can	probably	do	somewhat	better	than
10
27
	bits.	Neuroscientist	Anders	Sandberg	estimates	the	potential
storage	capacity	of	a	hydrogen	atom	at	about	four	million	bits.	These
densities	have	not	yet	been	demonstrated,	however,	so	we’ll	use	the
more	conservative	estimate.
[Note:	Anders	Sandberg,	“The	Physics	of	the	Information
Processing	Superobjects:	Daily	Life	Among	the	Jupiter	Brains,”
	
	5	(December	22,	1999),
http://www.transhumanist.com/volume5/Brains2.pdf.]
As	discussed	above,	10
42
	calculations	per	second	could	be
achieved	without	producing	significant	heat.	By	fully	deploying
reversible	computing	techniques,	using	designs	that	generate	low

42
levels	of	errors,	and	allowing	for	reasonable	amounts	of	energy
dissipation,	we	should	end	up	somewhere	between	10
42
	and	10
50
calculations	per	second.
The	design	terrain	between	these	two	limits	is	complex.
Examining	the	technical	issues	that	arise	as	we	advance	from	10
42
	to
10
50
	is	beyond	the	scope	of	this	chapter.	We	should	keep	in	mind,
however,	that	the	way	this	will	play	out	is	not	by	starting	with	the
ultimate	limit	of	10
50
	and	working	backward	based	on	various
practical	considerations.	Rather,	technology	will	continue	
to	ramp	up,

42
	and	10
50
42
	to
10
50
50
to	ramp	up,
always	using	its	latest	prowess	to	progress	to	the	next	level.	So	once
we	get	to	a	civilization	with	10
42
	cps	(for	every	2.2	pounds),	the
scientists	and	engineers	of	that	day	will	use	their	essentially	vast
nonbiological	intelligence	to	figure	out	how	to	get	10
43
,	then	10
44
,	and
so	on.	My	expectation	is	that	we	will	get	very	close	to	the	ultimate
limits.
Even	at	10
42
	cps,	a	2.2-pound	“ultimate	portable	computer”
would	be	able	to	perform	the	equivalent	of	all	human	thought	over	the

to	ramp	up,
42
43
,	then	10
44
,	and
limits.
Even	at	10
42
last	ten	thousand	years	(assumed	at	ten	billion	human	brains	for	ten
thousand	years)	in	ten	microseconds.
[Note:	See	note	above.	10
42
	cps	is	a	factor	of	10
–8
	less	than	10
50
cps,	so	one	ten-thousandth	of	a	nanosecond	becomes	10
microseconds.]

If	we	examine	the	Exponential	Growth	of	Computing	chart
(
chapter	2
),	we	see	that	this	amount	of	computing	is	estimated	to	be
available	for	one	thousand	dollars	by	2080.

INDEX
	
Page	numbers	in	
italics
	refer	to	graphs	and	illustrations.
abortion,	212–13
Abundance
	(Diamandis	and	Kotler),	278
Ackerman,	Diane,	179
ACTH	(adrenocorticotropin),	107
addictive	behaviors,	105–6,	118
adrenal	glands,	107
adrenaline,	107
Age	of	Intelligent	Machines,	The
	(Kurzweil),	4,	165–66,	256–57
Age	of	Spiritual	Machines,	The
	(Kurzweil),	4,	257,	267
A.I.
	(film),	210
Aiken,	Howard,	189
Alexander,	Richard	D.,	224
algorithms,	intelligent,	6–7
Allen,	Paul,	266–72
Allman,	John	M.,	179
Alzheimer’s	disease,	102
amygdala,	71,	
77,
	106–8,	109
analog	processing,	digital	emulation	of,	194–95,	274
Analytical	Engine,	189–90
anesthesia	awareness,	206
animal	behavior,	evolution	of,	122
apical	dendrites,	110
aptitude,	111–12
artificial	intelligence	(AI),	7,	37–38,	50,	265,	280
Allen	on,	270–71
biological	models	for,	273
chess	playing	and,	6,	38–39,	165–66,	257
conversation	and,	168–69

as	extension	of	neocortex,	172,	276
knowledge	bases	and,	4,	6–7,	170–71,	246,	247
language	and	speech	processing	in,	72–73,	92,	115–16,	122–23,	128,	135–
41,	142–46,	
145,
	149–50,	152–53,	156,	157–72
medicine	and,	6–7,	39,	108,	156,	160–61,	168
omnipresence	of,	158
optimization	of	pattern	recognition	in,	112
sparse	coding	in,	95–96
see	also
	neocortex,	digital
Audience,	Inc.,	96–97,	
98
auditory	association,	
77
auditory	cortex,	7,	
77,
	
97,
	128
auditory	information	processing,	96–97,	
97
auditory	nerve,	
97,
	128
data	reduction	in,	138
auditory	pathway,	
97
autoassociation,	59–61,	133,	173
automobiles,	self-driving,	7,	159,	261,	274
axons,	36,	
42,
	43,	66,	67,	90,	100,	113,	150,	173
as	digital	processors,	191
Babbage,	Charles,	189–90
Bainbridge,	David,	179
bandwidth,	of	Internet,	
254
basis	functions,	103–4
Bedny,	Marina,	87
Bell	System	Technical	Journal,
	184
Berger,	Theodore,	102
Berners-Lee,	Tim,	172
Bernoulli’s	principle,	5,	8
Better	Angels	of	Our	Nature:	Why	Violence	Has	Declined
	(Pinker),	27

97,
	128
97
axons,	36,	
42,
254
	184
Bierce,	Ambrose,	66
BINAC,	189
Bing,	171
biology,	37
DNA	as	unifying	theory	of,	17
reverse-engineering	of,	4–5
biomedicine,	LOAR	and,	251,	
252,
	253
Blackmore,	Susan,	211

Blakeslee,	Sandra,	73,	156
Blue	Brain	Project,	63,	80,	124–28,	
125
Bombe,	187
Bostrom,	Nick,	129–30,	222
Boyden,	Ed,	126
brain,	evolution	of,	2
brain,	human:
analog	computing	in,	274
complexity	of,	8–9,	181,	272
digital	implants	in,	243–44
digital	neocortex	as	extension	of,	172,	276
hemispheres	of,	
77,
	224–49
LOAR	as	applied	to,	261–63,	
263,
	
264,
	265
prediction	by,	250
redundancy	of,	9
reverse-engineering	of,	
see
	brain,	human,	computer	emulation	of;
neocortex,	digital
structure	of,	
77
brain,	human,	computer	emulation	of,	5,	7,	179–98,	273,	280
invariance	and,	197
memory	requirements	of,	196–97
parallel	processing	in,	197
processing	speed	in,	195–96
redundancy	in,	197
singularity	and,	194
Turing	test	and,	159–60,	169,	170,	178,	191,	213,	214,	233,	276,	298
n
von	Neumann	on,	191–95
see	also
brain,	mammalian:
hierarchical	thinking	as	unique	to,	2–3,	35
neocortex	in,	
78,
	93,	286
n
brain	plasticity,	79,	87–89,	91,	182,	193,	197,	225,	280

77
n
see	also
78,
	93,	286
n
as	evidence	of	universal	neocortical	processing,	86,	88,	152
limitations	on,	88–89
brain	scanning,	7,	
263,
	308
n
destructive,	
264,
	265,	309
n
–11
n
LOAR	and,	262–63,	
263,
	
264,
	265
nondestructive,	127,	129,	
264,
	312
n
–13
n

noninvasive,	273
Venn	diagram	of,	
262
brain	simulations,	124–31,	262
brain	stem,	36,	
99
Bremermann,	Hans,	316
n
Britain,	Battle	of,	187
Brodsky,	Joseph,	199
Burns,	Eric	A.,	113
busy	beaver	problem,	207
Butler,	Samuel,	62,	199–200,	224,	248–49
Byron,	Ada,	Countess	of	Lovelace,	190,	191
California,	University	of,	at	Berkeley,	88
“CALO”	project,	162
carbon	atoms,	information	structures	based	on,	2
Carroll,	Lewis,	109
cells,	replacement	of,	245,	246
cellular	automata,	236–39
cerebellum,	7,	
77,
	103–4
uniform	structure	of,	103
cerebral	cortex,	7–8
see	also
	neocortex
Chalmers,	David,	201–2,	218,	241
“chatbots,”	161
chemistry,	37
chess,	AI	systems	and,	6,	38–39,	165–66,	257
chimpanzees:
language	and,	3,	41
tool	use	by,	41
“Chinese	room”	thought	experiment,	170,	274–75
Chomsky,	Noam,	56,	158
Church,	Alonzo,	186
Church-Turing	thesis,	186
civil	rights,	278
cloud	computing,	116–17,	123,	246,	279–80
cochlea,	96,	
97,
	135,	138
cochlear	implants,	243
Cockburn,	David,	214
Cold	Spring	Harbor	Laboratory,	129

Colossus,	187,	188
“common	sense,”	40
communication,	reliability	of,	182–85,	190
communication	technology,	LOAR	and,	253,	
254
compatibilism,	234
complexity,	198,	233
of	human	brain,	8–9,	181,	272
modeling	and,	37–38
true	vs.	apparent,	10–11
computation:
price/performance	of,	4–5,	250–51,	257,	
257,
	267–68,	301
n
–3
n
thinking	compared	with	to,	26–27
universality	of,	26,	181–82,	185,	188,	192,	207
Computer	and	the	Brain,	The
	(von	Neumann),	191
computers:
brain	emulated	by,	
see
	brain,	human,	computer	emulation	of
consciousness	and,	209–11,	213–15,	223
intelligent	algorithms	employed	by,	6–7
knowledge	base	expanded	by,	4,	246,	247
logic	gates	in,	185
memory	in,	185,	
259,
	
260,
	268,	301
n
–3
n,
	306
n
–7
n
reliability	of	communication	by,	182–85,	190
see	also
“Computing	Machinery	and	Intelligence”	(Turing),	191
conditionals,	65,	69,	153,	189,	190
confabulation,	70,	217,	227,	228,	229
connectionism,	133,	191
“connectome,”	262
consciousness,	11,	199–209
cerebral	hemispheres	and,	226–29

259,
	
260,
	268,	301
n
–3
n,
	306
n
–7
n
see	also
computers	and,	209–11,	213–15,	223,	233
Descartes	on,	221–22
dualist	views	of,	202–3
Eastern	vs.	Western	views	of,	218–24
free	will	and,	233–34
Kurzweil’s	thought	experiment	on,	210
leap-of-faith	view	of,	209–10,	233
as	meme,	211,	235
memory	and,	28–29,	206–7,	217

moral	and	legal	systems	as	based	on,	212–13
of	nonhuman	life-forms,	213–14
panprotopsychist	view	of,	203,	213
as	philosophical	construct,	201–9
qualia	and,	203–5,	211
as	scientifically	unverifiable,	205,	211,	228
as	spiritual	construct,	222–23
as	subjective	experience,	211
Wittgenstein	on,	220–21
zombie	thought	experiment	and,	202
conversation,	AI	and,	168–69
convictions,	courage	of,	11,	23–24,	112,	117
Cooper’s	law,	253
corpus	callosum,	70,	
77,
	226
cortical	association	areas,	58
cortisol,	107
Craig,	Arthur,	100
creativity,	113–17
and	expansion	of	neocortex,	116–17
Cretaceous-Paleogene	extinction	event,	79
Crick,	Francis,	16–17
critical	thinking,	6,	176,	197
“criticism	from	incredulity,”	266–72
Crookes	radiometer,	20–21,	
21
crossbar	switching,	
85
Curtiss,	Susan,	225
	(Wiener),	115
Cyc	project,	162,	164
D2	gene,	106
Dalai	Lama,	109
Dalrymple,	David,	124,	291
n
DARPA,	162,	163
Darwin,	Charles,	
15
Lyell’s	influence	on,	14–15,	114,	177
thought	experiments	of,	14–16,	23

21
85
n
15
data,	determined	vs.	predictable,	26,	239
data	traffic,	on	Internet,	
254
de	Bode,	Stella,	225

Deep	Blue,	39,	166
DeMille,	Cecil	B.,	113
dendrites,	
42,
	43,	66,	67,	90,	150
as	analog	processors,	191–92
apical,	110
Dennett,	Daniel,	205–6,	230,	234
Denton,	Michael,	275–76
Descartes,	René,	221–22,	240
destructive	imaging	techniques,	
264,
	265,	309
n
–11
n
determined	outcomes,	predictable	outcomes	vs.,	26,	239
determinism,	232–33
free	will	and,	232–33,	234
randomness	and,	236
Devil’s	Dictionary,	The
	(Bierce),	66
Diamandis,	Peter,	278
Diamond,	Marian,	25
Dickinson,	Emily,	1
diffusion	tractography,	129
digital	processors:
emulation	of	analog	processing	in,	194–95,	274
see	also
	computers;	neocortex,	digital
Diogenes	Laertius,	246
DNA,	9–10
animal	behavior	encoded	in,	122
discovery	and	description	of,	16–17
encoding	of	information	in,	2,	17
as	unifying	theory	of	biology,	17
see	also
	genome,	human
dopamine,	105–6,	107,	118
Dostoevsky,	Fyodor,	199
Dragon	Dictation,	152–53
Dragon	Go!,	162,	164
Dragon	Naturally	Speaking,	152
Drave,	Scott,	149
dreams:
conscious	thinking	vs.,	71–72
taboos	and,	71–72

see	also
DNA,	9–10
see	also
dreams:
as	undirected	thoughts,	70–72
dynamic	RAM	memory,	growth	in,	
259,
	301
n
–3
n

E	=	mc
2,	22
Eckert,	J.	Presper,	189
EDSAC,	189
EDVAC,	189
Einstein,	Albert,	11,	25,	35,	71
thought	experiments	of,	18–23,	114,	117
Electric	Sheep,	149
Emerson,	Ralph	Waldo,	13
emotional	intelligence,	110,	194,	201,	213
emotions:
high-level,	109–11
as	products	of	both	old	brain	and	neocortex,	107–8
energy,	mass	equivalent	of,	22–23
ENIAC,	180,	189
Enigma	coding	machine,	187
EPAM	(elementary	perceiver	and	memorizer),	37–38
estrogen,	118
ether	theory,	Michelson-Morley	disproof	of,	18,	19,	36,	114
evolution,	76–79
Darwin’s	theory	of,	14–16
encoding	of	information	and,	2
intelligence	as	goal	of,	76–78,	277,	278
LOAR	and,	4
of	neocortex,	35–36
of	simulated	organisms,	147–53
as	spiritual	process,	223
survival	as	goal	of,	79,	104,	242
see	also
	natural	selection
evolutionary	(genetic)	algorithms	(GAs),	147–53,	173
existentialism,	221
expectation	(excitatory)	signals,	
42,
	52,	54,	60,	67,	73,	85,	91,	100,	112,
173,	175,	196–97
expert	managers,	166–67,	168
experts,	core	knowledge	of,	39–40
see

LOAR	and,	4
see	also
42,
see
	law	of	accelerating	returns
eye	movement,	pattern	recognition	and,	73
Far	Side,
	277
fear,	in	old	and	new	brains,	104–8

feature	invariance,	
see
	invariance
feedforward	neural	net,	
134,
	135
Feldman,	Daniel	E.,	88
Felleman,	Daniel	J.,	86
fetus,	brain	of,	62
field	programmable	gate	array	(FPGA),	83
“fight	or	flight”	mechanism,	107,	118
“First	Draft	of	a	Report	on	EDVAC”	(von	Neumann),	188
Forest,	Craig,	126
formants,	135,	
137
fractals,	9,	10–11,	
10
Franklin,	Rosalind,	16–17,	
17
free	will,	11,	224–40
consciousness	and,	233–34
definition	of,	231–32
determinism	and,	232–33,	234
as	meme,	235
responsibility	and,	235
Freitas,	Robert,	316
n
Freud,	Sigmund,	66,	71,	72
Friston,	K.	J.,	75
frontal	lobe,	36,	41,	
77
fusiform	gyrus,	89,	
95,
	111
gambling,	106
ganglion	cells,	95
García	Márquez,	Gabriel,	3–4,	283
n
–85
n
Gazzaniga,	Michael,	226–29,	234
General	Electric,	149
genetic	(evolutionary)	algorithms,	
see
	evolutionary	(genetic)	algorithms
genome,	human,	4,	103,	251
design	information	encoded	in,	90,	147,	155,	271,	314
n
–15
n
redundancy	in,	271,	314
n,
	315
n
sequencing	of,	LOAR	and,	252,	
252,
	253
see	also
	DNA

n
77
95,
	111
n
–85
n
see
n
–15
n
n,
	315
n
252,
	253
see	also
	DNA
George,	Dileep,	41,	73,	156
Ginet,	Carl,	234
God,	concept	of,	223
Gödel,	Kurt,	187

incompleteness	theorem	of,	187,	207–8
“God	parameters,”	147
Good,	Irvin	J.,	280–81
Google,	279
self-driving	cars	of,	7,	159,	261,	274
Google	Translate,	163
Google	Voice	Search,	72,	161
Greaves,	Mark,	266–72
Grossman,	Terry,	287
n
–88
n
Grötschel,	Martin,	269
Hameroff,	Stuart,	206,	208,	274
Hameroff-Penrose	thesis,	208–9
Hamlet
	(Shakespeare),	209
Harnad,	Stevan,	266
Harry	Potter	and	the	Half-Blood	Prince
	(Rowling),	117
Harry	Potter	and	the	Prisoner	of	Azkaban
	(Rowling),	121
Hasson,	Uri,	86
Havemann,	Joel,	25
Hawkins,	Jeff,	41,	73,	156
Hebb,	Donald	O.,	79–80
Hebbian	learning,	80
hemispherectomy,	225
hidden	Markov	models	(HMMs),	68,	141–44,	
143,
	
145,
	147,	162
hierarchical	hidden	Markov	models	(HHMMs),	51,	68,	72,	74,	144–46,
149–50,	152–53,	155,	156,	162,	164,	167–68,	195,	269,	270
pruning	of	unused	connections	by,	144,	147,	155
hierarchical	learning,	164,	195,	197
hierarchical	memory:
digital,	156–57
temporal,	73
hierarchical	systems,	4,	35
hierarchical	thinking,	8,	69,	105,	117,	153–54,	177,	233,	286
n

143,
	
145,
	147,	162
n
bidirectional	flow	of	information	in,	52
language	and,	56,	159,	162,	163
in	mammalian	brain,	2–3
pattern	recognition	as,	33,	41–53
recursion	in,	3,	7–8,	56,	65,	91,	109,	156,	177
routine	tasks	and,	32–33

as	survival	mechanism,	79
as	unique	to	mammalian	brain,	35
hippocampus,	63,	
77,
	101–2
Hitchhiker’s	Guide	to	the	Galaxy,	The
	(Adams),	161
Hobbes,	Thomas,	278
Hock,	Dee,	113
Horwitz,	B.,	75
Hubel,	David	H.,	34
Human	Connectome	Project,	129
human	genome,	
see
Human	Genome	Project,	251,	273
humans:
merger	of	intelligent	technology	with,	266–72,	276,	279–82
tool-making	ability	of,	3,	27,	276,	279
Hume,	David,	234–35
IBM,	6–7,	108,	128,	165–66
Cognitive	Computing	Group	of,	195
ideas,	recursive	linking	of,	3
identity,	10,	11,	240–47
as	pattern	continuity,	246,	247
thought	experiments	on,	242–47
importance	parameters,	
42,
	60,	66,	67
incompatibilism,	234,	236
incompleteness,	Gödel’s	theorem	of,	187,	207–8
inference	engines,	162–63
information,	encoding	of:
in	DNA,	2,	17,	122
evolution	and,	2
in	human	genome,	90,	147,	155,	271,	314
n
–15
n
information	structures,	carbon-based,	2
information	technologies:
exponential	growth	of,	278–79
LOAR	and,	4,	249–57,	
252,
	
257,
	
258,
	
259,
	
260,
	261,	
261

42,
	60,	66,	67
n
–15
n
252,
	
257,
	
258,
	
259,
	
260,
	261,	
261
inhibitory	signals,	
42,
	52–53,	67,	85,	91,	100,	173
insula,	99–100,	
99,
	110
integrated	circuits,	
85
Intel,	268
intelligence,	1–2

emotional,	110,	194,	201,	213
as	evolutionary	goal,	76–78,	277,	278
evolution	of,	177
as	problem-solving	ability,	277
International	Dictionary	of	Psychology
	(Sutherland),	211
“International	Technology	Roadmap	for	Semiconductors,”	268
Internet,	exponential	growth	of,	
254
Interpretation	of	Dreams,	The
	(Freud),	66
intuition,	linear	nature	of,	266
invariance,	in	pattern	recognition,	30,	59–61,	133,	135,	137,	175
and	computer	emulation	of	brain,	197
one-dimensional	representations	of	data	and,	141–42
vector	quantization	and,	141
inventors,	timing	and,	253,	255
I,	Robot
Jacquard	loom,	189,	190
James,	William,	75–76,	98–99
Jeffers,	Susan,	104
Jennings,	Ken,	157–58,	165
Jeopardy!
	(TV	show),	6–7,	108,	157–58,	160,	165,	166,	167,	168,	169,	172,
178,	232–33,	270
Joyce,	James,	55
Kasparov,	Garry,	39,	166
K	Computer,	196
knowledge	bases:
AI	systems	and,	4,	6–7,	170–71,	246,	247
of	digital	neocortex,	177
exponential	growth	of,	3
as	inherently	hierarchical,	220
language	and,	3
professional,	39–40

I,	Robot
Jeopardy!
as	recursively	linked	ideas,	3
Kodandaramaiah,	Suhasa,	126
Koene,	Randal,	89
Koltsov,	Nikolai,	16
Kotler,	Steven,	278
KurzweilAI.net,	161

Kurzweil	Applied	Intelligence,	144
Kurzweil	Computer	Products,	122
Kurzweil	Voice,	160
lamina	1	neurons,	97
language:
chimpanzees	and,	3,	41
and	growth	of	knowledge	base,	3
hierarchical	nature	of,	56,	159,	162,	163
as	metaphor,	115
as	translation	of	thinking,	56,	68
language	software,	51,	72–73,	92,	115–16,	122–23,	144–45,	
145,
	156,	157–
72,	174,	270
expert	managers	in,	166–67
hand-coded	rules	in,	164–65,	166,	168
HHMMs	in,	167–68
hierarchical	systems	in,	162–65
Larson,	Gary,	277
“Last	Voyage	of	the	Ghost,	The”	(García	Márquez),	3–4
lateral	geniculate	nucleus,	
95,
	100
law	of	accelerating	returns	(LOAR),	4,	6,	7,	41,	123
as	applied	to	human	brain,	261–63,	
263,
	
264,
	265
biomedicine	and,	251,	
252,
	253
communication	technology	and,	253,	
254
computation	capacity	and,	281,	316
n
–19
n
information	technology	and,	4,	249–57,	
252,
	
257,
	
258,
	
259,
	
260,
	261,	
261
objections	to,	266–82
predictions	based	on,	256–57,	
257,
	
258,
	
259,
	
260,
	
261
and	unlikelihood	of	other	intelligent	species,	5

95,
	100
263,
	
264,
	265
252,
	253
254
n
–19
n
252,
	
257,
	
258,
	
259,
	
260,
	261,	
261
257,
	
258,
	
259,
	
260,
	
261
“Law	of	Accelerating	Returns,	The”	(Kurzweil),	267
laws	of	thermodynamics,	37,	267
learning,	61–65,	122,	155,	273–74
conditionals	in,	65
and	difficulty	of	grasping	more	than	one	conceptual	level	at	a	time,	65
in	digital	neocortex,	127–28,	175–76
environment	and,	119
Hebbian,	80
hierarchical,	164,	195,	197
in	neural	nets,	132–33

neurological	basis	of,	79–80
pattern	recognition	as	basic	unit	of,	80–81
of	patterns,	63–64,	90
recognition	as	simultaneous	with,	63
simultaneous	processing	in,	63,	146
legal	systems,	consciousness	as	basis	of,	212–13
Leibniz,	Gottfried	Wilhelm,	34,	223
Lenat,	Douglas,	162
Leviathan
	(Hobbes),	278
Lewis,	Al,	93
Libet,	Benjamin,	229–30,	231,	234
light,	speed	of,	281
Einstein’s	thought	experiments	on,	18–23
linear	programming,	64
LISP	(LISt	Processor),	153–55,	163
pattern	recognition	modules	compared	with,	154,	155
Lloyd,	Seth,	316
n,
	317
n
Loebner,	Hugh,	298
n
Loebner	Prize,	298
n
logic,	38–39
logical	positivism,	220
logic	gates,	185
Lois,	George,	113
love,	117–20
biochemical	changes	associated	with,	118–19
evolutionary	goals	and,	119
pattern	recognition	modules	and,	119–20
“Love	Is	the	Drug,”	118
Lovelace,	Ada	Byron,	Countess	of,	190,	191
lucid	dreaming,	72,	287
n
–88
n
Lyell,	Charles,	14–15,	114,	177
McCarthy,	John,	153
McClelland,	Shearwood,	225
McGinn,	Colin,	200
magnetic	data	storage,	growth	in,	
261,

n,
	317
n
n
n
n
–88
n
261,
	301
n
–3
n
magnetoencephalography,	129
Manchester	Small-Scale	Experimental	Machine,	189
Mandelbrot	set,	10–11,	
10

Marconi,	Guglielmo,	253
Mark	1	Perceptron,	131–32,	
134,
	135,	189
Markov,	Andrei	Andreyevich,	143
Markram,	Henry,	80–82,	124–27,	129
mass	equivalent,	of	energy,	22–23
Mathematica,	171
“Mathematical	Theory	of	Communication,	A”	(Shannon),	184
Mauchly,	John,	189
Maudsley,	Henry,	224
Maxwell,	James	Clerk,	20
Maxwell,	Robert,	225
Mead,	Carver,	194–95
medial	geniculate	nucleus,	
97,
	100
medicine,	AI	and,	6–7,	39,	108,	156,	160–61,	168
memes:
consciousness	as,	211,	235
free	will	as,	235
memory,	in	computers,	185,	
259,
	
260,
	268,	301
n
–3
n,
	306
n
–7
n
memory,	memories,	human:
abstract	concepts	in,	58–59
capacity	of,	192–93
computers	as	extensions	of,	169
consciousness	vs.,	28–29,	206–7,	217
dimming	of,	29,	59
hippocampus	and,	101–2
as	ordered	sequences	of	patterns,	27–29,	54
redundancy	of,	59
unexpected	recall	of,	31–32,	54,	68–69
working,	101
Menabrea,	Luigi,	190
metacognition,	200,	201
metaphors,	14–15,	113–17,	176–77
Michelson,	Albert,	18,	19,	36,	114
Michelson-Morley	experiment,	19,	36,	114

259,
	
260,
	268,	301
n
–3
n,
	306
n
–7
n
microtubules,	206,	207,	208,	274
Miescher,	Friedrich,	16
mind,	11
pattern	recognition	theory	of	(PRTM),	5–6,	8,	11,	34–74,	79,	80,	86,	92,
111,	172,	217

thought	experiments	on,	199–247
mind-body	problem,	221
Minsky,	Marvin,	62,	133–35,	
134,
	199,	228
MIT	Artificial	Intelligence	Laboratory,	134
MIT	Picower	Institute	for	Learning	and	Memory,	101
MobilEye,	159
modeling,	complexity	and,	37–38
Modha,	Dharmendra,	128,	195,	271–72
momentum,	20–21
conservation	of,	21–22
Money,	John	William,	118,	119
montane	vole,	119
mood,	regulation	of,	106
Moore,	Gordon,	251
Moore’s	law,	251,	255,	268
moral	intelligence,	201
moral	systems,	consciousness	as	basis	of,	212–13
Moravec,	Hans,	196
Morley,	Edward,	18,	19,	36,	114
Moskovitz,	Dustin,	156
motor	cortex,	36,	
99
motor	nerves,	
99
Mountcastle,	Vernon,	36,	37,	94
Mozart,	Leopold,	111
Mozart,	Wolfgang	Amadeus,	111,	112
MRI	(magnetic	resonance	imaging),	129
spatial	resolution	of,	262–65,	
263,
	309
n
MT	(V5)	visual	cortex	region,	83,	95
Muckli,	Lars,	225
music,	as	universal	to	human	culture,	62
mutations,	simulated,	148
names,	recalling,	32
National	Institutes	of	Health,	129
natural	selection,	76

99
99
263,
	309
n
geologic	process	as	metaphor	for,	14–15,	114,	177
see	also
	evolution
Nature,
	94
nematode	nervous	system,	simulation	of,	124

neocortex,	3,	7,	
77,
	
78
AI	reverse-engineering	of,	
see
bidirectional	flow	of	information	in,	85–86,	91
evolution	of,	35–36
expansion	of,	through	AI,	172,	266–72,	276
expansion	of,	through	collaboration,	116
hierarchical	order	of,	41–53
learning	process	of,	
see
	learning
linear	organization	of,	250
as	metaphor	machine,	113
neural	leakage	in,	150–51
old	brain	as	modulated	by,	93–94,	105,	108
one-dimensional	representations	of	multidimensional	data	in,	53,	66,	91,
141–42
pattern	recognition	in,	
see
	pattern	recognition
pattern	recognizers	in,	
see
	pattern	recognition	modules
plasticity	of,	
see
	brain	plasticity
prediction	by,	50–51,	52,	58,	60,	66–67,	250
PRTM	as	basic	algorithm	of,	6
pruning	of	unused	connections	in,	83,	90,	143,	174
redundancy	in,	9,	224
regular	grid	structure	of,	82–83,	
84,
	
85,
	129,	262
sensory	input	in,	58,	60
simultaneous	processing	of	information	in,	193
specific	types	of	patterns	associated	with	regions	of,	86–87,	89–90,	91,	111,
152

141–42
see
see
see
84,
	
85,
	129,	262
152
structural	simplicity	of,	11
structural	uniformity	of,	36–37
structure	of,	35–37,	38,	75–92
as	survival	mechanism,	79,	250
thalamus	as	gateway	to,	100–101
total	capacity	of,	40,	280
total	number	of	neurons	in,	230
unconscious	activity	in,	228,	231,	233
unified	model	of,	24,	34–74
as	unique	to	mammalian	brain,	93,	286
n
universal	processing	algorithm	of,	86,	88,	90–91,	152,	272
see	also
	cerebral	cortex
neocortex,	digital,	6–8,	41,	116–17,	121–78,	195

benefits	of,	123–24,	247
bidirectional	flow	of	information	in,	173
as	capable	of	being	copied,	247
critical	thinking	module	for,	176,	197
as	extension	of	human	brain,	172,	276
HHMMs	in,	174–75
hierarchical	structure	of,	173
knowledge	bases	of,	177
learning	in,	127–28,	175–76
metaphor	search	module	in,	176–77
moral	education	of,	177–78
pattern	redundancy	in,	175
simultaneous	searching	in,	177
structure	of,	172–78
virtual	neural	connections	in,	173–74
neocortical	columns,	36–37,	38,	90,	124–25
nervous	systems,	2
neural	circuits,	unreliability	of,	185
neural	implants,	243,	245
neural	nets,	131–35,	144,	155
algorithm	for,	291
n
–97
n
feedforward,	
134,
	135
learning	in,	132–33
neural	processing:
digital	emulation	of,	195–97
massive	parallelism	of,	192,	193,	195
speed	of,	192,	195
neuromorphic	chips,	194–95,	196
neuromuscular	junction,	
99
neurons,	2,	36,	38,	43,	80,	172
neurotransmitters,	105–7
new	brain,	
see
	neocortex
Newell,	Allen,	181
New	Kind	of	Science,	A
	(Wolfram),	236,	239
Newton,	Isaac,	94

n
–97
n
134,
	135
99
new	brain,	
see
	neocortex
Nietzsche,	Friedrich,	117
nonbiological	systems,	as	capable	of	being	copied,	247
nondestructive	imaging	techniques,	127,	129,	
264,
	312
n
–13
n
nonmammals,	reasoning	by,	286
n

noradrenaline,	107
norepinephrine,	118
	(Dostoevsky),	199
Nuance	Speech	Technologies,	6–7,	108,	122,	152,	161,	162,	168
nucleus	accumbens,	
77,
	105
Numenta,	156
NuPIC,	156
obsessive-compulsive	disorder,	118
occipital	lobe,	36
old	brain,	63,	71,	90,	93–108
neocortex	as	modulator	of,	93–94,	105,	108
sensory	pathway	in,	94–98
olfactory	system,	100
Oluseun,	Oluseyi,	204
OmniPage,	122
	(García	Márquez),	283
n
–85
n
	(Hawkins	and	Blakeslee),	73,	156
	(Darwin),	15–16
optical	character	recognition	(OCR),	122
optic	nerve,	
95,
	100
channels	of,	94–95,	
96
organisms,	simulated,	evolution	of,	147–53
overfitting	problem,	150
oxytocin,	119
pancreas,	37
panprotopsychism,	203,	213
Papert,	Seymour,	134–35,	
134
parameters,	in	pattern	recognition:
“God,”	147
importance,	
42,
	48–49,	60,	66,	67
size,	
42,
	49–50,	60,	61,	66,	67,	73–74,	91–92,	173
size	variability,	
42,
	49–50,	67,	73–74,	91–92
Parker,	Sean,	156

95,
	100
96
134
“God,”	147
42,
size,	
42,
42,
Parkinson’s	disease,	243,	245
particle	physics,	
see
	quantum	mechanics
Pascal,	Blaise,	117
patch-clamp	robotics,	125–26,	
126

pattern	recognition,	195
of	abstract	concepts,	58–59
as	based	on	experience,	50,	90,	273–74
as	basic	unit	of	learning,	80–81
bidirectional	flow	of	information	in,	52,	58,	68
distortions	and,	30
eye	movement	and,	73
as	hierarchical,	33,	90,	138,	142
of	images,	48
invariance	and,	
see
	invariance,	in	pattern	recognition
learning	as	simultaneous	with,	63
list	combining	in,	60–61
in	neocortex,	
see
redundancy	in,	39–40,	57,	60,	64,	185
pattern	recognition	modules,	35–41,	
42,
	90,	198
autoassociation	in,	60–61
axons	of,	
42,
	43,	66,	67,	113,	173
bidirectional	flow	of	information	to	and	from	thalamus,	100–101
dendrites	of,	
42,
	43,	66,	67
digital,	172–73,	175,	195
expectation	(excitatory)	signals	in,	
42,
genetically	determined	structure	of,	80
“God	parameter”	in,	147
importance	parameters	in,	
42,
inhibitory	signals	in,	
42,
input	in,	41–42,	
42,
	53–59
love	and,	119–20

42,
	90,	198
axons	of,	
42,
42,
	43,	66,	67
42,
42,
42,
42,
	53–59
neural	connections	between,	90
as	neuronal	assemblies,	80–81
one-dimensional	representation	of	multidimensional	data	in,	53,	66,	91,
141–42
prediction	by,	50–51,	52,	58,	60,	66–67
redundancy	of,	42,	
43,
	48,	91
sequential	processing	of	information	by,	266
simultaneous	firings	of,	57–58,	
57,
	146
size	parameters	in,	
42,
size	variability	parameters	in,	
42,
	67,	73–74,	91–92,	173
of	sounds,	48

thresholds	of,	48,	52–53,	60,	66,	67,	111–12,	173
total	number	of,	38,	40,	41,	113,	123,	280
universal	algorithm	of,	111,	275
pattern	recognition	theory	of	mind	(PRTM),	5–6,	8,	11,	34–74,	79,	80,	86,
92,	111,	172,	217
patterns:
hierarchical	ordering	of,	41–53
higher-level	patterns	attached	to,	43,	45,	66,	67
input	in,	41,	
42,
	44,	66,	67
learning	of,	63–64,	90
name	of,	42–43
output	of,	
42,
	44,	66,	67
redundancy	and,	64
specific	areas	of	neocortex	associated	with,	86–87,	89–90,	91,	111,	152
storing	of,	64–65
structure	of,	41–53
Patterns,	Inc.,	156
Pavlov,	Ivan	Petrovich,	216
Penrose,	Roger,	207–8,	274
perceptions,	as	influenced	by	expectations	and	interpretations,	31
perceptrons,	131–35
	(Minsky	and	Papert),	134–35,	
134
phenylethylamine,	118
	(Wittgenstein),	221
phonemes,	61,	135,	137,	146,	152
photons,	20–21
physics,	37
computational	capacity	and,	281,	316
n
–19
n
laws	of,	37,	267
standard	model	of,	2
see	also
Pinker,	Steven,	76–77,	278

134
n
–19
n
see	also
pituitary	gland,	
77
Plato,	212,	221,	231
pleasure,	in	old	and	new	brains,	104–8
Poggio,	Tomaso,	85,	159
posterior	ventromedial	nucleus	(VMpo),	99–100,	
99
prairie	vole,	119
predictable	outcomes,	determined	outcomes	vs.,	26,	239

President’s	Council	of	Advisors	on	Science	and	Technology,	269
price/performance,	of	computation,	4–5,	250–51,	257,	
257,
n
–
3
n
	(Russell	and	Whitehead),	181
probability	fields,	218–19,	235–36
professional	knowledge,	39–40
proteins,	reverse-engineering	of,	4–5
qualia,	203–5,	210,	211
quality	of	life,	perception	of,	277–78
quantum	computing,	207–9,	274
quantum	mechanics,	218–19
observation	in,	218–19,	235–36
randomness	vs.	determinism	in,	236
Quinlan,	Karen	Ann,	101
Ramachandran,	Vilayanur	Subramanian	“Rama,”	230
random	access	memory:
growth	in,	
259,
	
260,
	301
n
–3
n,
	306
n
–7
n
three-dimensional,	268
randomness,	determinism	and,	236
rationalization,	
see
	confabulation
reality,	hierarchical	nature	of,	4,	56,	90,	94,	172
recursion,	3,	7–8,	56,	65,	91,	153,	156,	177,	188
“Red”	(Oluseum),	204
redundancy,	9,	39–40,	64,	184,	185,	197,	224
in	genome,	271,	314
n,
	315
n
of	memories,	59
of	pattern	recognition	modules,	42,	
43,
	48,	91
thinking	and,	57
religious	ecstacy,	118

growth	in,	
259,
	
260,
	301
n
–3
n,
	306
n
–7
n
see
n,
	315
n
43,
	48,	91
“Report	to	the	President	and	Congress,	Designing	a	Digital	Future”
(President’s	Council	of	Advisors	on	Science	and	Technology),	269
retina,	
95
reverse-engineering:
of	biological	systems,	4–5
of	human	brain,	
see
	brain,	human,	computer	emulation	of;	neocortex,
digital
Rosenblatt,	Frank,	131,	133,	
134,
	135,	191

Roska,	Boton,	94
Rothblatt,	Martine,	278
routine	tasks,	as	series	of	hierarchical	steps,	32–33
Rowling,	J.	K.,	117,	121
Roxy	Music,	118
Russell,	Bertrand,	104,	181,	220
Rutter,	Brad,	165
saccades,	73
Salk	Institute,	89
same-sex	marriage,	278
Sandberg,	Anders,	129–30,	318
n
Schopenhauer,	Arthur,	235,	240
science:
as	based	on	objective	measurement,	211
specialization	in,	115
Science,
	82–83
“scientist’s	pessimism,”	272–73
Searle,	John,	170,	201,	205,	206,	222
“Chinese	room”	thought	experiment	of,	170,	274–75
Seinfeld
	(TV	show),	75
selective	serotonin	reuptake	inhibitors,	106
self-organizing	systems,	144,	147,	149,	150,	154–55,	162,	168,	171–72,
175,	197,	270
sensorimotor	area,	
77
sensory	cortex,	
99
sensory	nerves,	
99
sensory	organs,	58
sensory	receptors,	
99
sensory-touch	pathway,	58,	60,	94–98,	
95,
	97–100,	
97,
	
99
serotonin,	105,	106,	107,	118
Seung,	Sebastian,	10
	(TV	show),	117
sexual	reproduction,	118
simulated,	148
Shakespeare,	William,	39,	114–15,	209
Shannon,	Claude,	183–84,	190

Seinfeld
77
99
99
99
95,
	97–100,	
97,
	
99
Shashua,	Amnon,	159
Shaw,	J.	C.,	181

Simon,	Herbert	A.,	37–38,	181
singularity,	194
Singularity	Is	Near,	The
	(Kurzweil),	4,	5,	196,	251,	253,	256,	257,	267,
268–69,	271,	274,	316
n
–19
n
“Singularity	Isn’t	Near,	The”	(Allen	and	Greaves),	266–72
Siri,	7,	72,	116,	123,	153,	161–62,	164,	168,	171
size	parameters,	
42,
	60,	66,	67,	73–74,	91–92
size	variability	parameters,	
42,
Skinner,	B.	F.,	13
Smullyan,	Raymond,	241
Society	of	Mind,	The
	(Minsky),	62,	199
Sonnet	73	(Shakespeare),	114–15
SOPA	(Stop	Online	Piracy	Act),	279
sparse	coding,	95–96,	135–41
specialization,	increasing,	115
spectrograms,	135,	
136,
	
137
speech	recognition	software,	49–50,	51,	53,	61,	72–74,	92,	115–16,	122–23,
128,	135–46,	
145,
	198,	273
GAs	in,	149–50,	152
HHMM	in,	149–50,	152–53
speed	of	light,	281
Sperry,	Roger	W.,	218
spinal	cord,	36,	
99
spindle	neurons,	109–11
split-brain	patients,	70,	226–27
Stanford	Encyclopedia	of	Philosophy,	The,
	232

136,
	
137
145,
	198,	273
99
	232
	(TV	show),	210
Star	Wars
	films,	210
stochastic	variation,	9
supercomputer	power,	growth	in,	
258,
	301
n
–3
n
survival:
as	evolutionary	goal,	79,	104,	242
as	individual	goal,	242
Sutherland,	Stuart,	211
SyNAPSE	chips,	195,	196
Szent-Györgyi,	Albert,	93

taboos,	dreams	and,	71–72
Taylor,	J.	G.,	75
technology,	as	compensating	for	human	limitation,	3,	27,	276,	279
Technology	Review,
	266
Tegmark,	Max,	208
	films,	210
testosterone,	118
thalamus,	36,	
77,
	
95,
	97,	
97,
	98–101
as	gateway	to	neocortex,	100–101
thermodynamics,	177
Thiel,	Peter,	156
thinking:
computing	compared	with,	26–27
disorderliness	of,	55,	69
language	as	translation	of,	56,	68
limitations	to,	23–24,	27
redundancy	and,	57
as	statistical	analysis,	170
statistical	probability	and,	270–71
thought	experiments	on,	24,	25–33
undirected	vs.	directed,	54–55,	68–69
see	also
	hierarchical	thinking
thought	experiments,	114
“Chinese	room,”	170,	274–75
on	computer	consciousness,	202,	210
of	Darwin,	14–16,	23
of	Einstein,	18–23,	114,	117
on	identity,	242–47
on	the	mind,	199–247
on	thinking,	24,	25–33
of	Turing,	185–87,	188
Thrun,	Sebastian,	158
time,	Einstein’s	thought	experiments	on,	19–20
tool	making,	by	humans,	3,	27,	276,	279

see	also
	(Wittgenstein),	219–21
	(Kurzweil	and	Grossman),
287
n
–88
n
	films,	210

transistors:
per	chip,	growth	in,	
258,
	301
n
–3
n
price	decrease	in,	
260,
	304
n
–6
n
Turing,	Alan,	121,	159–60,	185,	191
thought	experiments	of,	185–87,	188
unsolvable	problem	theorem	of,	187,	207–8
Turing	machine,	185–87,	
186,
	188,	192,	207–8
Turing	test,	159–60,	169,	170,	178,	191,	213,	214,	233,	276,	298
n
UIMA	(Unstructured	Information	Management	Architecture),	167–68
Ulam,	Stan,	194
Unitarianism,	222
universality	of	computation,	26,	181–82,	185,	188,	192,	207
universe,	as	capable	of	encoding	information,	2
University	College	(London),	118
unsolvable	problems,	Turing’s	theorem	of,	187,	207–8
vasopressin,	119
vector	quantization,	135,	138–39,	145
invariance	and,	141
ventral	pallidum,	105
Vicarious	Systems,	156
visual	association,	
77
visual	cortex,	7,	
77,
	83,	
95,
	193
of	congenitally	blind	people,	87
digital	simulation	of,	128
hierarchical	structure	of,	85–86
V1	region,	83,	85,	87,	95,	100
V2	region	of,	83,	85,	87,	95
V5	(MT)	region	of,	83,	95

77
77,
	83,	
95,
	193
visual	information	processing,	94–96,	
95,
	
96
visual	pathway,	
95
visual	recognition	systems,	53
von	Neumann,	John,	179,	186–89,	190,	195
brain/computer	comparison	of,	191–95
stored	program	concept	of,	186–87,	188
von	Neumann	machine,	187–89,	190,	193
	(Darwin),	14

Wall-E
Watson	(IBM	computer),	6–7,	108,	157–58,	159,	160,	165,	166,	167–68,
171,	172,	178,	200,	232–33,	239,	247,	265,	270–71,	274
Watson,	James	D.,	8–9,	16–17
Watts,	Lloyd,	96
wave	function,	collapse	of,	218–19,	235–36
Wedeen,	Van	J.,	82–83,	90,	129,	262
Werblin,	Frank	S.,	94–95
Whitehead,	Alfred	North,	181
	(Sandberg	and	Bostrom),	129–30,
130,
	
131
Wiener,	Norbert,	115,	143
Wikipedia,	6,	156,	166,	170,	176,	232,	270,	279
Wittgenstein,	Ludwig,	219–21
Wolfram,	Stephen,	170–71,	177,	236–39
Wolfram	Alpha,	161,	170–72,	177
Wolfram	Research,	170–71
working	memory,	101
World	War	I,	278
World	War	II,	187,	278
writing,	as	backup	system,	123–24
Young,	Thomas,	18
Z-3	computer,	189
Zuo,	Yi,	89
Zuse,	Konrad,	189

DRAFTAAIIPRENTICE HALLSERIES
IN ARTIFICIAL INTELLIGENCE
StuartRussell andPeterNorvig,Editors
FORSYTH& PONCE ComputerVision: AModernApproach
GRAHAM ANSICommonLisp
JURAFSKY & MARTIN SpeechandLanguageProcessing
NEAPOLITAN LearningBayesianNetworks
RUSSELL& NORVIG ArtiﬁcialIntelligence: A ModernApproach

DRAFTSpeechand Language Processing
An Introduction toNatural Language Processing,
Computational Linguistics, and Speech Recognition
DanielJurafskyandJamesH.Martin
Upper SaddleRiver,New Jersey07458

DRAFTLibrary ofCongress Cataloging-in-Publication Data
Jurafsky, Daniel S.(Daniel Saul)
Speech and Language Processing /Daniel Jurafsky, James H.M artin.
p. cm.
Includes bibliographical references and index.
ISBN 0-13-095069-6 FIXTHIS
Editor-in-Chief: FIXTHISSTUFF
Publisher: Tracy Dunkelberger
Editorial/production supervision: Scott Disanno
Editorial assistant:
Executive managing editor:
Cover design director:
Cover design execution:
Manufacturing manager:
Manufacturing buyer:
Assistant vice-president of production and manufacturing :
Cover design: Daniel Jurafsky, James H. Martin, and Linda Martin. FIX THIS The front cover drawing
is the action for the Jacquard Loom (Usher, 1954). The back co ver drawing is Alexander Graham Bell’s
Gallows telephone (Rhodes, 1929).
This book was set in Times-Roman and TIPA(IPA) by theauthors using LATEX2ε.
c⃝2008 by Prentice-Hall, Inc.
Pearson Higher Education
Upper Saddle River, New Jersey 07458

Theauthor and publisher ofthis book have used their besteff orts in preparing this book. Theseefforts
include thedevelopment, research, and testing of the theor ies and programs to determine their effectiveness.
Theauthor and publisher shall notbeliable in any event for i ncidental or consequential damages in
connection with, or arising outof, thefurnishing, perform ance, oruse of these programs.
Allrightsreserved. Nopartofthisbookmaybereproduced,i nanyformorbyanymeans,withoutpermission

in writing fromthe publisher.
Printed in the United States of America
10 9 8 7 6 5 4 3 2 1
ISBN 0-13-095069-6 FIXTHISTOO
Prentice-Hall International (UK)Limited, London
Prentice-Hall of Australia Pty.Limited, Sydney
Prentice-Hall Canada, Inc., Toronto
Prentice-Hall Hispanoamericana, S.A., Mexico
Prentice-Hall of India Private Limited, New Delhi
Prentice-Hall of Japan, Inc., Tokyo
Prentice-Hall AsiaPte. Ltd., Singapore
Editora Prentice-Hall do Brasil, Ltda., Rio deJaneiro

DRAFTSummary of Contents
Preface ............................................ .................xxiii
1 Introduction...................................... ............... 1
I Words
2 RegularExpressionsand Automata..................... .......... 17
3 Words& Transducers................................. ........... 45
4 N-grams........................................... .............. 83
5 Part-of-SpeechTagging ............................. ............. 123
6 Hidden Markovand Maximum EntropyModels............... ....173
II Speech
7 Phonetics......................................... ...............215
8 Speech Synthesis................................... ..............249
9 AutomaticSpeech Recognition........................ ............287
10 Speech Recognition: AdvancedTopics ................. ........... 337
11 ComputationalPhonology........................... .............365
III Syntax
12 FormalGrammarsofEnglish.......................... ...........389

II Speech
III Syntax
13 Parsingwith Context-FreeGrammars .................. .......... 431
14 StatisticalParsing............................... .................465
15 Featuresand Uniﬁcation ............................ ............. 495
16 Languageand Complexity............................ ............537
IV Semantics andPragmatics
17 RepresentingMeaning .............................. ............. 553
18 ComputationalSemantics........................... ............. 593

19 LexicalSemantics................................. ...............627
20 ComputationalLexicalSemantics.................... .............653
21 ComputationalDiscourse........................... ..............697
V Applications
22 InformationExtraction............................ ...............741
23 QuestionAnswering andSummarization................ ..........783
24 DialogueandConversationalAgents.................. ............ 829
25 Machine Translation............................... .............. 879
Bibliography 929
Index 981
vii

DRAFTContents
Preface xxiii
1 Introduction 1
1.1 Knowledgein SpeechandLanguageProcessing . . . . . . . . . . . 2
1.2 Ambiguity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 ModelsandAlgorithms . . . . . . . . . . . . . . . . . . . . . . . . 5
1.4 Language,Thought,andUnderstanding . . . . . . . . . . . . . . . . 6
1.5 TheState oftheArt . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.6 SomeBriefHistory . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.6.1 FoundationalInsights: 1940sand1950s . . . . . . . . . . . 9
1.6.2 TheTwo Camps: 1957–1970. . . . . . . . . . . . . . . . . 10
1.6.3 FourParadigms: 1970–1983 . . . . . . . . . . . . . . . . . 11
1.6.4 EmpiricismandFinite State ModelsRedux: 1983–1993 . . 12
1.6.5 TheField ComesTogether: 1994–1999 . . . . . . . . . . . 12
1.6.6 TheRise ofMachineLearning: 2000–2007 . . . . . . . . . 13
1.6.7 OnMultipleDiscoveries . . . . . . . . . . . . . . . . . . . 13
1.6.8 AFinal BriefNoteonPsychology . . . . . . . . . . . . . . 14

1.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
BibliographicalandHistorical Notes . . . . . . . . . . . . . . . . . . . . . 15
I Words
2 RegularExpressionsandAutomata 17
2.1 RegularExpressions . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.1.1 Basic RegularExpressionPatterns . . . . . . . . . . . . . . 18
2.1.2 Disjunction,Grouping,andPrecedence . . . . . . . . . . . 2 1
2.1.3 ASimpleExample . . . . . . . . . . . . . . . . . . . . . . 22
2.1.4 AMoreComplexExample . . . . . . . . . . . . . . . . . . 23

I Words
2.1.5 AdvancedOperators . . . . . . . . . . . . . . . . . . . . . 24
2.1.6 RegularExpressionSubstitution,Memory,andELIZA . . . 25
2.2 Finite-StateAutomata . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.2.1 UsinganFSA toRecognizeSheeptalk . . . . . . . . . . . . 26
2.2.2 FormalLanguages . . . . . . . . . . . . . . . . . . . . . . 30
2.2.3 AnotherExample . . . . . . . . . . . . . . . . . . . . . . . 31
2.2.4 Non-DeterministicFSAs . . . . . . . . . . . . . . . . . . . 32
2.2.5 UsinganNFSA toAcceptStrings . . . . . . . . . . . . . . 33

2.2.6 RecognitionasSearch . . . . . . . . . . . . . . . . . . . . 35
2.2.7 RelatingDeterministicandNon-DeterministicAutom ata . . 38
2.3 RegularLanguagesandFSAs . . . . . . . . . . . . . . . . . . . . . 38
2.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
BibliographicalandHistorical Notes . . . . . . . . . . . . . . . . . . . . . 42
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
ix

DRAFTx Contents
3 Words& Transducers 45
3.1 Surveyof(Mostly)EnglishMorphology . . . . . . . . . . . . . . . 47
3.1.1 InﬂectionalMorphology . . . . . . . . . . . . . . . . . . . 48
3.1.2 DerivationalMorphology. . . . . . . . . . . . . . . . . . . 50
3.1.3 Cliticization. . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.1.4 Non-concatenativeMorphology . . . . . . . . . . . . . . . 52
3.1.5 Agreement . . . . . . . . . . . . . . . . . . . . . . . . . . 52
3.2 Finite-StateMorphologicalParsing . . . . . . . . . . . . . . . . . . 53
3.3 Buildinga Finite-StateLexicon . . . . . . . . . . . . . . . . . . . . 54
3.4 Finite-StateTransducers . . . . . . . . . . . . . . . . . . . . . . . . 5 7
3.4.1 SequentialTransducersandDeterminism . . . . . . . . . . 5 9
3.5 FSTsforMorphologicalParsing . . . . . . . . . . . . . . . . . . . 60
3.6 TransducersandOrthographicRules . . . . . . . . . . . . . . . . . 63
3.7 CombiningFST LexiconandRules . . . . . . . . . . . . . . . . . . 65

3.8 Lexicon-FreeFSTs: ThePorterStemmer . . . . . . . . . . . . . . . 68
3.9 WordandSentenceTokenization . . . . . . . . . . . . . . . . . . . 69
3.9.1 SegmentationinChinese . . . . . . . . . . . . . . . . . . . 70
3.10 DetectingandCorrectingSpellingErrors . . . . . . . . . . . . . . . 72
3.11 MinimumEditDistance . . . . . . . . . . . . . . . . . . . . . . . . 74
3.12 HumanMorphologicalProcessing . . . . . . . . . . . . . . . . . . 7 7
3.13 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

BibliographicalandHistorical Notes . . . . . . . . . . . . . . . . . . . . . 80
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
4 N-grams 83
4.1 CountingWordsin Corpora . . . . . . . . . . . . . . . . . . . . . . 84
4.2 Simple(Unsmoothed) N-grams . . . . . . . . . . . . . . . . . . . . 86
4.3 TrainingandTest Sets . . . . . . . . . . . . . . . . . . . . . . . . . 91
4.3.1 N-gramSensitivityto theTrainingCorpus . . . . . . . . . . 92
4.3.2 UnknownWords: Openversusclosedvocabularytasks . . . 94

4.4 Evaluating N-grams: Perplexity . . . . . . . . . . . . . . . . . . . . 95
4.5 Smoothing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
4.5.1 LaplaceSmoothing . . . . . . . . . . . . . . . . . . . . . . 98
4.5.2 Good-TuringDiscounting . . . . . . . . . . . . . . . . . . 101
4.5.3 SomeadvancedissuesinGood-Turingestimation . . . . . . 102
4.6 Interpolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
4.7 Backoff. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

4.7.1 Advanced: DetailsofcomputingKatzbackoff αandP∗. . 106
4.8 PracticalIssues: ToolkitsandDataFormats . . . . . . . . . . . . . . 107
4.9 AdvancedIssuesinLanguageModeling . . . . . . . . . . . . . . . 1 09
4.9.1 AdvancedSmoothingMethods: Kneser-NeySmoothing . . 109
4.9.2 Class-basedN-grams . . . . . . . . . . . . . . . . . . . . . 111
4.9.3 LanguageModelAdaptationandUsingtheWeb . . . . . . 111
4.9.4 UsingLongerDistanceInformation: A BriefSummary . . . 112
4.10 Advanced: InformationTheoryBackground . . . . . . . . . . . . . 113

4.10.1 Cross-EntropyforComparingModels . . . . . . . . . . . . 11 6

DRAFTContents xi
4.11 Advanced: TheEntropyofEnglishandEntropyRate Const ancy . . 117
BibliographicalandHistorical Notes . . . . . . . . . . . . . . . . . . . . . 119
4.12 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
5 Part-of-SpeechTagging 123
5.1 (Mostly)EnglishWordClasses . . . . . . . . . . . . . . . . . . . . 12 4
5.2 TagsetsforEnglish. . . . . . . . . . . . . . . . . . . . . . . . . . . 130
5.3 Part-of-SpeechTagging . . . . . . . . . . . . . . . . . . . . . . . . 133
5.4 Rule-BasedPart-of-SpeechTagging . . . . . . . . . . . . . . . . . . 137
5.5 HMMPart-of-SpeechTagging . . . . . . . . . . . . . . . . . . . . 139
5.5.1 Computingthemost-likelytag sequence: Anexample . . . 142
5.5.2 FormalizingHiddenMarkovModeltaggers . . . . . . . . . 14 4
5.5.3 TheViterbiAlgorithmforHMMTagging . . . . . . . . . . 145
5.5.4 ExtendingtheHMMalgorithmtotrigrams . . . . . . . . . 149

5.6 Transformation-BasedTagging . . . . . . . . . . . . . . . . . . . . 1 51
5.6.1 HowTBLRulesAreApplied . . . . . . . . . . . . . . . . 152
5.6.2 HowTBLRulesAreLearned . . . . . . . . . . . . . . . . 152
5.7 EvaluationandErrorAnalysis . . . . . . . . . . . . . . . . . . . . . 1 53
5.7.1 ErrorAnalysis . . . . . . . . . . . . . . . . . . . . . . . . 156
5.8 AdvancedIssuesinPart-of-SpeechTagging . . . . . . . . . . . . . 157
5.8.1 PracticalIssues: TagIndeterminacyandTokenizatio n . . . . 157

5.8.2 UnknownWords . . . . . . . . . . . . . . . . . . . . . . . 158
5.8.3 Part-of-SpeechTaggingforOtherLanguages . . . . . . . . 160
5.8.4 CombiningTaggers . . . . . . . . . . . . . . . . . . . . . . 163
5.9 Advanced: TheNoisyChannelModelforSpelling . . . . . . . . . . 163
5.9.1 ContextualSpellingErrorCorrection . . . . . . . . . . . . 1 67
5.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
BibliographicalandHistorical Notes . . . . . . . . . . . . . . . . . . . . . 169

Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
6 HiddenMarkovandMaximum EntropyModels 173
6.1 MarkovChains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
6.2 TheHiddenMarkovModel . . . . . . . . . . . . . . . . . . . . . . 177
6.3 ComputingLikelihood: TheForwardAlgorithm . . . . . . . . . . . 179
6.4 Decoding: TheViterbiAlgorithm . . . . . . . . . . . . . . . . . . . 1 84
6.5 TrainingHMMs: TheForward-BackwardAlgorithm . . . . . . . . . 187

6.6 MaximumEntropyModels: Background . . . . . . . . . . . . . . . 19 3
6.6.1 LinearRegression . . . . . . . . . . . . . . . . . . . . . . 194
6.6.2 Logisticregression . . . . . . . . . . . . . . . . . . . . . . 197
6.6.3 Logisticregression: Classiﬁcation . . . . . . . . . . . . . . 199
6.6.4 Advanced: Learningin logisticregression . . . . . . . . . . 200
6.7 MaximumEntropyModeling . . . . . . . . . . . . . . . . . . . . . 201
6.7.1 Whydowe callit MaximumEntropy?. . . . . . . . . . . . 205

6.8 MaximumEntropyMarkovModels . . . . . . . . . . . . . . . . . . 207
6.8.1 DecodingandLearningin MEMMs . . . . . . . . . . . . . 210

DRAFTxii Contents
6.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
BibliographicalandHistorical Notes . . . . . . . . . . . . . . . . . . . . . 212
II Speech
7 Phonetics 215
7.1 SpeechSoundsandPhoneticTranscription . . . . . . . . . . . . . . 216
7.2 ArticulatoryPhonetics . . . . . . . . . . . . . . . . . . . . . . . . . 21 8
7.2.1 TheVocal Organs. . . . . . . . . . . . . . . . . . . . . . . 218
7.2.2 Consonants: PlaceofArticulation . . . . . . . . . . . . . . 22 0
7.2.3 Consonants: MannerofArticulation . . . . . . . . . . . . . 22 1
7.2.4 Vowels . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
7.3 PhonologicalCategoriesandPronunciationVariation . . . . . . . . 225
7.3.1 PhoneticFeatures . . . . . . . . . . . . . . . . . . . . . . . 227
7.3.2 PredictingPhoneticVariation . . . . . . . . . . . . . . . . 228
7.3.3 FactorsInﬂuencingPhoneticVariation . . . . . . . . . . . . 229
7.4 AcousticPhoneticsandSignals . . . . . . . . . . . . . . . . . . . . 2 30

7.4.1 Waves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
7.4.2 SpeechSoundWaves . . . . . . . . . . . . . . . . . . . . . 231
7.4.3 FrequencyandAmplitude;Pitch andLoudness . . . . . . . 2 33
7.4.4 InterpretingPhonesfroma Waveform . . . . . . . . . . . . 236
7.4.5 SpectraandtheFrequencyDomain . . . . . . . . . . . . . 236
7.4.6 TheSource-FilterModel . . . . . . . . . . . . . . . . . . . 241
7.5 PhoneticResources . . . . . . . . . . . . . . . . . . . . . . . . . . 241

7.6 Advanced: ArticulatoryandGesturalPhonology . . . . . . . . . . . 244
7.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
BibliographicalandHistorical Notes . . . . . . . . . . . . . . . . . . . . . 246
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
8 SpeechSynthesis 249
8.1 TextNormalization . . . . . . . . . . . . . . . . . . . . . . . . . . 250
8.1.1 SentenceTokenization . . . . . . . . . . . . . . . . . . . . 251

8.1.2 Non-StandardWords . . . . . . . . . . . . . . . . . . . . . 253
8.1.3 HomographDisambiguation . . . . . . . . . . . . . . . . . 256
8.2 PhoneticAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
8.2.1 DictionaryLookup . . . . . . . . . . . . . . . . . . . . . . 258
8.2.2 Names . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
8.2.3 Grapheme-to-Phoneme . . . . . . . . . . . . . . . . . . . . 259
8.3 ProsodicAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 263

8.3.1 ProsodicStructure . . . . . . . . . . . . . . . . . . . . . . 263
8.3.2 Prosodicprominence . . . . . . . . . . . . . . . . . . . . . 264
8.3.3 Tune . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
8.3.4 Moresophisticatedmodels: ToBI . . . . . . . . . . . . . . 267
8.3.5 Computingdurationfromprosodiclabels . . . . . . . . . . 2 70
8.3.6 ComputingF0fromprosodiclabels . . . . . . . . . . . . . 271
8.3.7 Finalresultoftextanalysis: InternalRepresentati on . . . . 272

DRAFTContents xiii
8.4 DiphoneWaveformsynthesis . . . . . . . . . . . . . . . . . . . . . 273
8.4.1 Buildinga diphonedatabase . . . . . . . . . . . . . . . . . 274
8.4.2 DiphoneconcatenationandTD-PSOLAforprosody . . . . 2 75
8.5 UnitSelection(Waveform)Synthesis . . . . . . . . . . . . . . . . . 278
8.6 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
BibliographicalandHistorical Notes . . . . . . . . . . . . . . . . . . . . . 283
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
9 AutomaticSpeechRecognition 287
9.1 SpeechRecognitionArchitecture . . . . . . . . . . . . . . . . . . . 289
9.2 Applyingthe HiddenMarkovModeltoSpeech . . . . . . . . . . . . 293
9.3 FeatureExtraction: MFCC vectors . . . . . . . . . . . . . . . . . . 2 97
9.3.1 Preemphasis . . . . . . . . . . . . . . . . . . . . . . . . . 298
9.3.2 Windowing . . . . . . . . . . . . . . . . . . . . . . . . . . 298
9.3.3 DiscreteFourierTransform. . . . . . . . . . . . . . . . . . 300

9.3.4 Melﬁlter bankandlog . . . . . . . . . . . . . . . . . . . . 301
9.3.5 TheCepstrum: InverseDiscreteFourierTransform . . . . . 302
9.3.6 DeltasandEnergy . . . . . . . . . . . . . . . . . . . . . . 304
9.3.7 Summary: MFCC . . . . . . . . . . . . . . . . . . . . . . 304
9.4 ComputingAcousticLikelihoods . . . . . . . . . . . . . . . . . . . 3 05
9.4.1 VectorQuantization . . . . . . . . . . . . . . . . . . . . . 305
9.4.2 GaussianPDFs . . . . . . . . . . . . . . . . . . . . . . . . 308

9.4.3 Probabilities,logprobabilitiesanddistancefunct ions . . . . 315
9.5 TheLexiconandLanguageModel . . . . . . . . . . . . . . . . . . 316
9.6 SearchandDecoding . . . . . . . . . . . . . . . . . . . . . . . . . 316
9.7 EmbeddedTraining . . . . . . . . . . . . . . . . . . . . . . . . . . 326
9.8 Evaluation: Word ErrorRate . . . . . . . . . . . . . . . . . . . . . 330
9.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332
BibliographicalandHistorical Notes . . . . . . . . . . . . . . . . . . . . . 333

Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336
10 SpeechRecognition: AdvancedTopics 337
10.1 MultipassDecoding: N-bestlists andlattices . . . . . . . . . . . . . 338
10.2 A∗(‘Stack’)Decoding . . . . . . . . . . . . . . . . . . . . . . . . . 343
10.3 Context-DependentAcousticModels: Triphones . . . . . . . . . . . 347
10.4 DiscriminativeTraining . . . . . . . . . . . . . . . . . . . . . . . . 3 51
10.4.1 MaximumMutualInformationEstimation . . . . . . . . . . 3 52

10.4.2 AcousticModelsbasedonPosteriorClassiﬁers . . . . . . . 354
10.5 ModelingVariation . . . . . . . . . . . . . . . . . . . . . . . . . . 355
10.5.1 EnvironmentalVariationandNoise . . . . . . . . . . . . . 35 5
10.5.2 Speaker and Dialect Adaptation: Variation due to spe aker
differences . . . . . . . . . . . . . . . . . . . . . . . . . . 356
10.5.3 PronunciationModeling: VariationduetoGenre . . . . . . 357
10.6 Metadata: Boundaries,Punctuation,andDisﬂuencies . . . . . . . . 359

10.7 SpeechRecognitionbyHumans . . . . . . . . . . . . . . . . . . . . 36 1
10.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363

DRAFTxiv Contents
BibliographicalandHistorical Notes . . . . . . . . . . . . . . . . . . . . . 363
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364
11 ComputationalPhonology 365
11.1 Finite-StatePhonology . . . . . . . . . . . . . . . . . . . . . . . . 36 5
11.2 AdvancedFinite-StatePhonology . . . . . . . . . . . . . . . . . . . 369
11.2.1 Harmony . . . . . . . . . . . . . . . . . . . . . . . . . . . 369
11.2.2 TemplaticMorphology . . . . . . . . . . . . . . . . . . . . 370
11.3 ComputationalOptimalityTheory . . . . . . . . . . . . . . . . . . . 371
11.3.1 Finite-StateTransducerModelsofOptimalityTheor y. . . . 373
11.3.2 StochasticModelsofOptimalityTheory . . . . . . . . . . . 375
11.4 Syllabiﬁcation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376
11.5 LearningPhonology&Morphology . . . . . . . . . . . . . . . . . 38 0
11.5.1 LearningPhonologicalRules . . . . . . . . . . . . . . . . . 380
11.5.2 LearningMorphology . . . . . . . . . . . . . . . . . . . . 381

11.5.3 LearninginOptimalityTheory . . . . . . . . . . . . . . . . 385
11.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386
BibliographicalandHistorical Notes . . . . . . . . . . . . . . . . . . . . . 386
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 388
III Syntax
12 FormalGrammarsofEnglish 389
12.1 Constituency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 390

III Syntax
12.2 Context-FreeGrammars . . . . . . . . . . . . . . . . . . . . . . . . 391
12.2.1 Formaldeﬁnitionofcontext-freegrammar . . . . . . . . . 395
12.3 SomeGrammarRulesforEnglish . . . . . . . . . . . . . . . . . . . 39 6
12.3.1 Sentence-LevelConstructions . . . . . . . . . . . . . . . . 39 6
12.3.2 ClausesandSentences . . . . . . . . . . . . . . . . . . . . 398
12.3.3 TheNounPhrase . . . . . . . . . . . . . . . . . . . . . . . 398
12.3.4 Agreement . . . . . . . . . . . . . . . . . . . . . . . . . . 403

12.3.5 TheVerbPhraseandSubcategorization . . . . . . . . . . . 4 04
12.3.6 Auxiliaries . . . . . . . . . . . . . . . . . . . . . . . . . . 406
12.3.7 Coordination . . . . . . . . . . . . . . . . . . . . . . . . . 407
12.4 Treebanks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 408
12.4.1 Example: ThePennTreebankProject . . . . . . . . . . . . 408
12.4.2 UsingaTreebankasaGrammar . . . . . . . . . . . . . . . 410
12.4.3 SearchingTreebanks . . . . . . . . . . . . . . . . . . . . . 412

12.4.4 HeadsandHeadFinding . . . . . . . . . . . . . . . . . . . 413
12.5 GrammarEquivalenceandNormalForm . . . . . . . . . . . . . . . 4 16
12.6 Finite-StateandContext-FreeGrammars . . . . . . . . . . . . . . . 417
12.7 DependencyGrammars . . . . . . . . . . . . . . . . . . . . . . . . 418
12.7.1 TheRelationshipBetweenDependenciesandHeads . . . . 419
12.7.2 CategorialGrammar . . . . . . . . . . . . . . . . . . . . . 420
12.8 SpokenLanguageSyntax . . . . . . . . . . . . . . . . . . . . . . . 421

12.8.1 DisﬂuenciesandRepair . . . . . . . . . . . . . . . . . . . 422

DRAFTContents xv
12.8.2 TreebanksforSpokenLanguage . . . . . . . . . . . . . . . 423
12.9 GrammarsandHumanProcessing. . . . . . . . . . . . . . . . . . . 42 3
12.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425
BibliographicalandHistorical Notes . . . . . . . . . . . . . . . . . . . . . 426
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 428
13 ParsingwithContext-FreeGrammars 431
13.1 ParsingasSearch . . . . . . . . . . . . . . . . . . . . . . . . . . . 432
13.1.1 Top-DownParsing . . . . . . . . . . . . . . . . . . . . . . 433
13.1.2 Bottom-UpParsing . . . . . . . . . . . . . . . . . . . . . . 434
13.1.3 ComparingTop-DownandBottom-UpParsing . . . . . . . 43 5
13.2 Ambiguity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 436
13.3 SearchintheFaceofAmbiguity . . . . . . . . . . . . . . . . . . . 43 8
13.4 DynamicProgrammingParsingMethods . . . . . . . . . . . . . . . 439

13.4.1 CKY Parsing . . . . . . . . . . . . . . . . . . . . . . . . . 440
13.4.2 TheEarleyAlgorithm . . . . . . . . . . . . . . . . . . . . 447
13.4.3 ChartParsing . . . . . . . . . . . . . . . . . . . . . . . . . 452
13.5 PartialParsing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 454
13.5.1 Finite-StateRule-BasedChunking . . . . . . . . . . . . . . 4 55
13.5.2 MachineLearning-BasedApproachestoChunking . . . . . 456
13.5.3 EvaluatingChunkingSystems . . . . . . . . . . . . . . . . 459
13.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 460

BibliographicalandHistorical Notes . . . . . . . . . . . . . . . . . . . . . 461
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 462
14 StatisticalParsing 465
14.1 ProbabilisticContext-FreeGrammars . . . . . . . . . . . . . . . . . 466
14.1.1 PCFGsforDisambiguation. . . . . . . . . . . . . . . . . . 467
14.1.2 PCFGsforLanguageModeling . . . . . . . . . . . . . . . 469
14.2 ProbabilisticCKY ParsingofPCFGs . . . . . . . . . . . . . . . . . 470

14.3 LearningPCFG RuleProbabilities . . . . . . . . . . . . . . . . . . 473
14.4 Problemswith PCFGs . . . . . . . . . . . . . . . . . . . . . . . . . 474
14.4.1 Independenceassumptions miss structural dependen ciesbe-
tweenrules . . . . . . . . . . . . . . . . . . . . . . . . . . 474
14.4.2 Lackofsensitivityto lexicaldependencies . . . . . . . . . 475
14.5 ImprovingPCFGs bySplittingandMergingNonterminals . . . . . . 477
14.6 ProbabilisticLexicalizedCFGs . . . . . . . . . . . . . . . . . . . . 479

14.6.1 TheCollinsParser . . . . . . . . . . . . . . . . . . . . . . 481
14.6.2 Advanced: FurtherDetailsoftheCollinsParser . . . . . . . 483
14.7 EvaluatingParsers . . . . . . . . . . . . . . . . . . . . . . . . . . . 485
14.8 Advanced: DiscriminativeReranking . . . . . . . . . . . . . . . . . 486
14.9 Advanced: Parser-BasedLanguageModeling. . . . . . . . . . . . . 488
14.10 HumanParsing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 489

14.11 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 491
BibliographicalandHistorical Notes . . . . . . . . . . . . . . . . . . . . . 492
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 494

DRAFTxvi Contents
15 FeaturesandUniﬁcation 495
15.1 FeatureStructures . . . . . . . . . . . . . . . . . . . . . . . . . . . 496
15.2 UniﬁcationofFeatureStructures . . . . . . . . . . . . . . . . . . . 499
15.3 FeatureStructuresinthe Grammar . . . . . . . . . . . . . . . . . . 503
15.3.1 Agreement . . . . . . . . . . . . . . . . . . . . . . . . . . 504
15.3.2 HeadFeatures . . . . . . . . . . . . . . . . . . . . . . . . 507
15.3.3 Subcategorization . . . . . . . . . . . . . . . . . . . . . . 508
15.3.4 Long-DistanceDependencies . . . . . . . . . . . . . . . . 513
15.4 ImplementingUniﬁcation . . . . . . . . . . . . . . . . . . . . . . . 51 3
15.4.1 UniﬁcationDataStructures. . . . . . . . . . . . . . . . . . 514
15.4.2 TheUniﬁcationAlgorithm . . . . . . . . . . . . . . . . . . 515
15.5 ParsingwithUniﬁcationConstraints . . . . . . . . . . . . . . . . . 519
15.5.1 IntegratingUniﬁcationintoanEarleyParser . . . . . . . . 520
15.5.2 Uniﬁcation-BasedParsing . . . . . . . . . . . . . . . . . . 526

15.6 TypesandInheritance . . . . . . . . . . . . . . . . . . . . . . . . . 528
15.6.1 Advanced: ExtensionstoTyping . . . . . . . . . . . . . . 531
15.6.2 OtherExtensionstoUniﬁcation . . . . . . . . . . . . . . . 532
15.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 532
BibliographicalandHistorical Notes . . . . . . . . . . . . . . . . . . . . . 533
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 534
16 LanguageandComplexity 537

16.1 TheChomskyHierarchy . . . . . . . . . . . . . . . . . . . . . . . . 538
16.2 HowtoTell if aLanguageIsn’tRegular . . . . . . . . . . . . . . . 540
16.2.1 ThePumpingLemma . . . . . . . . . . . . . . . . . . . . 541
16.2.2 AreEnglishandOtherNaturalLanguagesRegularLang uages?543
16.3 IsNaturalLanguageContext-Free? . . . . . . . . . . . . . . . . . . 546
16.4 ComplexityandHumanProcessing . . . . . . . . . . . . . . . . . . 5 48
16.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 550

BibliographicalandHistorical Notes . . . . . . . . . . . . . . . . . . . . . 551
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 552
17 RepresentingMeaning 553
17.1 ComputationalDesiderataforRepresentations . . . . . . . . . . . . 555
17.1.1 Veriﬁability . . . . . . . . . . . . . . . . . . . . . . . . . . 555
17.1.2 UnambiguousRepresentations . . . . . . . . . . . . . . . . 556
17.1.3 CanonicalForm . . . . . . . . . . . . . . . . . . . . . . . 557
17.1.4 InferenceandVariables. . . . . . . . . . . . . . . . . . . . 559

17.1.5 Expressiveness . . . . . . . . . . . . . . . . . . . . . . . . 559
17.2 Model-TheoreticSemantics . . . . . . . . . . . . . . . . . . . . . . 5 60
17.3 First-OrderLogic . . . . . . . . . . . . . . . . . . . . . . . . . . . 563
17.3.1 Basic ElementsofFirst OrderLogic . . . . . . . . . . . . . 56 3
17.3.2 VariablesandQuantiﬁers . . . . . . . . . . . . . . . . . . . 566
17.3.3 LambdaNotation . . . . . . . . . . . . . . . . . . . . . . . 568

DRAFTContents xvii
17.3.4 TheSemanticsofFirst-OrderLogic . . . . . . . . . . . . . 56 9
17.3.5 Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . 570
17.4 RepresentingEventsandStates . . . . . . . . . . . . . . . . . . . . 572
17.4.1 RepresentingTime . . . . . . . . . . . . . . . . . . . . . . 575
17.4.2 Aspect . . . . . . . . . . . . . . . . . . . . . . . . . . . . 578
17.5 RelatedRepresentationalApproaches . . . . . . . . . . . . . . . . . 581
17.5.1 DescriptionLogics . . . . . . . . . . . . . . . . . . . . . . 582
17.6 AlternativeApproachesto Meaning . . . . . . . . . . . . . . . . . . 587
17.6.1 MeaningasAction . . . . . . . . . . . . . . . . . . . . . . 588
17.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 588
BibliographicalandHistorical Notes . . . . . . . . . . . . . . . . . . . . . 589
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 590
18 ComputationalSemantics 593

18.1 Syntax-DrivenSemanticAnalysis . . . . . . . . . . . . . . . . . . . 593
18.2 SemanticAugmentationstoContext-FreeGrammarRules . . . . . . 595
18.3 QuantiﬁerScopeAmbiguityandUnderspeciﬁcation . . . . . . . . . 602
18.3.1 StoreandRetrieveApproaches. . . . . . . . . . . . . . . . 602
18.4 Uniﬁcation-BasedApproachesto SemanticAnalysis . . . . . . . . . 604
18.5 SemanticAttachmentsforaFragmentofEnglish . . . . . . . . . . . 610
18.5.1 Sentences . . . . . . . . . . . . . . . . . . . . . . . . . . . 610

18.5.2 NounPhrases . . . . . . . . . . . . . . . . . . . . . . . . . 612
18.5.3 VerbPhrases . . . . . . . . . . . . . . . . . . . . . . . . . 615
18.5.4 PrepositionalPhrases . . . . . . . . . . . . . . . . . . . . . 617
18.6 IntegratingSemanticsintotheEarleyParser . . . . . . . . . . . . . 619
18.7 IdiomsandCompositionality . . . . . . . . . . . . . . . . . . . . . 6 21
18.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 622

BibliographicalandHistorical Notes . . . . . . . . . . . . . . . . . . . . . 623
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 624
19 LexicalSemantics 627
19.1 WordSenses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 628
19.2 RelationsbetweenSenses . . . . . . . . . . . . . . . . . . . . . . . 63 1
19.2.1 SynonymyandAntonymy . . . . . . . . . . . . . . . . . . 631
19.2.2 Hyponymy . . . . . . . . . . . . . . . . . . . . . . . . . . 632
19.2.3 SemanticFields . . . . . . . . . . . . . . . . . . . . . . . . 633

19.3 WordNet: A DatabaseofLexicalRelations . . . . . . . . . . . . . . 633
19.4 EventParticipants: SemanticRolesandSelectionalRe strictions . . . 635
19.4.1 ThematicRoles . . . . . . . . . . . . . . . . . . . . . . . . 636
19.4.2 DiathesisAlternations . . . . . . . . . . . . . . . . . . . . 637
19.4.3 ProblemswithThematicRoles . . . . . . . . . . . . . . . . 639
19.4.4 ThePropositionBank . . . . . . . . . . . . . . . . . . . . 640
19.4.5 FrameNet . . . . . . . . . . . . . . . . . . . . . . . . . . . 641

19.4.6 SelectionalRestrictions . . . . . . . . . . . . . . . . . . . 643
19.5 PrimitiveDecomposition . . . . . . . . . . . . . . . . . . . . . . . 64 5
19.6 Advancedconcepts: Metaphor . . . . . . . . . . . . . . . . . . . . 64 7

DRAFTxviii Contents
19.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 648
BibliographicalandHistorical Notes . . . . . . . . . . . . . . . . . . . . . 649
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 650
20 ComputationalLexicalSemantics 653
20.1 WordSense Disambiguation: Overview . . . . . . . . . . . . . . . . 654
20.2 SupervisedWordSenseDisambiguation . . . . . . . . . . . . . . . 655
20.2.1 ExtractingFeatureVectorsforSupervisedLearning . . . . . 656
20.2.2 NaiveBayesandDecisionList Classiﬁers . . . . . . . . . . 657
20.3 WSD Evaluation,Baselines, andCeilings . . . . . . . . . . . . . . . 660
20.4 WSD: DictionaryandThesaurusMethods . . . . . . . . . . . . . . 662
20.4.1 TheLesk Algorithm . . . . . . . . . . . . . . . . . . . . . 662
20.4.2 SelectionalRestrictionsandSelectionalPreferen ces . . . . 664
20.5 MinimallySupervisedWSD: Bootstrapping . . . . . . . . . . . . . 666
20.6 WordSimilarity: ThesaurusMethods . . . . . . . . . . . . . . . . . 668

20.7 WordSimilarity: DistributionalMethods . . . . . . . . . . . . . . . 674
20.7.1 Deﬁninga Word’sCo-occurrenceVectors . . . . . . . . . . 6 75
20.7.2 MeasuresofAssociationwith Context . . . . . . . . . . . . 6 76
20.7.3 Deﬁningsimilaritybetweentwo vectors . . . . . . . . . . . 679
20.7.4 EvaluatingDistributionalWordSimilarity . . . . . . . . . . 683
20.8 Hyponymyandotherwordrelations . . . . . . . . . . . . . . . . . 6 84
20.9 SemanticRoleLabeling . . . . . . . . . . . . . . . . . . . . . . . . 687

20.10 Advanced: UnsupervisedSenseDisambiguation . . . . . . . . . . . 690
BibliographicalandHistorical Notes . . . . . . . . . . . . . . . . . . . . . 691
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 695
21 ComputationalDiscourse 697
21.1 DiscourseSegmentation . . . . . . . . . . . . . . . . . . . . . . . . 70 0
21.1.1 UnsupervisedDiscourseSegmentation . . . . . . . . . . . 7 00
21.1.2 SupervisedDiscourseSegmentation . . . . . . . . . . . . . 7 02

21.1.3 EvaluatingDiscourseSegmentation . . . . . . . . . . . . . 7 04
21.2 TextCoherence . . . . . . . . . . . . . . . . . . . . . . . . . . . . 705
21.2.1 RhetoricalStructureTheory . . . . . . . . . . . . . . . . . 706
21.2.2 AutomaticCoherenceAssignment . . . . . . . . . . . . . . 708
21.3 ReferenceResolution . . . . . . . . . . . . . . . . . . . . . . . . . 711
21.4 ReferencePhenomena . . . . . . . . . . . . . . . . . . . . . . . . . 714
21.4.1 FiveTypesofReferringExpressions . . . . . . . . . . . . . 7 14

21.4.2 InformationStatus . . . . . . . . . . . . . . . . . . . . . . 716
21.5 FeaturesforPronominalAnaphoraResolution . . . . . . . . . . . . 717
21.6 Threealgorithmsforpronominalanaphoraresolution . . . . . . . . 720
21.6.1 PronominalAnaphoraBaseline: TheHobbsAlgorithm . . 720
21.6.2 ACenteringAlgorithmforAnaphoraResolution . . . . . . 722
21.6.3 ALog-LinearmodelforPronominalAnaphoraResoluto n . 724
21.6.4 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . 725
21.7 CoreferenceResolution . . . . . . . . . . . . . . . . . . . . . . . . 72 6

21.8 EvaluatingCoreferenceResolution . . . . . . . . . . . . . . . . . . 728

DRAFTContents xix
21.9 Advanced: Inference-BasedCoherenceResolution . . . . . . . . . . 728
21.10 PsycholinguisticStudiesofReferenceandCoherence . . . . . . . . 734
21.11 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 735
BibliographicalandHistorical Notes . . . . . . . . . . . . . . . . . . . . . 736
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 738
22 InformationExtraction 741
22.1 NamedEntityRecognition . . . . . . . . . . . . . . . . . . . . . . 743
22.1.1 AmbiguityinNamedEntityRecognition . . . . . . . . . . 74 5
22.1.2 NERasSequenceLabeling . . . . . . . . . . . . . . . . . 745
22.1.3 EvaluatingNamedEntityRecognition . . . . . . . . . . . . 7 49
22.1.4 PracticalNERArchitectures . . . . . . . . . . . . . . . . . 750
22.2 RelationDetectionandClassiﬁcation . . . . . . . . . . . . . . . . . 751
22.2.1 SupervisedLearningApproachestoRelationAnalysi s . . . 752
22.2.2 LightlySupervisedApproachestoRelationAnalysis . . . . 755

22.2.3 EvaluatingRelationAnalysisSystems . . . . . . . . . . . . 758
22.3 TemporalandEventProcessing . . . . . . . . . . . . . . . . . . . . 7 59
22.3.1 TemporalExpressionRecognition . . . . . . . . . . . . . . 76 0
22.3.2 TemporalNormalization . . . . . . . . . . . . . . . . . . . 762
22.3.3 EventDetectionandAnalysis . . . . . . . . . . . . . . . . 765
22.3.4 TimeBank. . . . . . . . . . . . . . . . . . . . . . . . . . . 766
22.4 Template-Filling . . . . . . . . . . . . . . . . . . . . . . . . . . . . 768

22.4.1 Statistical ApproachestoTemplate-Filling . . . . . . . . . 769
22.4.2 Finite-StateTemplate-FillingSystems . . . . . . . . . . . . 770
22.5 Advanced: BiomedicalInformationExtraction∗. . . . . . . . . . . 773
22.5.1 BiologicalNamedEntityRecognition . . . . . . . . . . . . 7 74
22.5.2 GeneNormalization . . . . . . . . . . . . . . . . . . . . . 775
22.5.3 BiologicalRolesandRelations . . . . . . . . . . . . . . . . 77 6
22.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 778

BibliographicalandHistorical Notes . . . . . . . . . . . . . . . . . . . . . 778
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 779
23 QuestionAnsweringand Summarization 783
23.1 InformationRetrieval . . . . . . . . . . . . . . . . . . . . . . . . . 78 5
23.1.1 TheVectorSpaceModel . . . . . . . . . . . . . . . . . . . 786
23.1.2 TermWeighting . . . . . . . . . . . . . . . . . . . . . . . 789
23.1.3 TermSelectionandCreation . . . . . . . . . . . . . . . . . 790

23.1.4 EvaluatingInformationRetrievalSystems . . . . . . . . . . 790
23.1.5 Homonymy,Polysemy,andSynonymy . . . . . . . . . . . 794
23.1.6 ImprovingUserQueries . . . . . . . . . . . . . . . . . . . 795
23.2 FactoidQuestionAnswering . . . . . . . . . . . . . . . . . . . . . 79 6
23.2.1 QuestionProcessing . . . . . . . . . . . . . . . . . . . . . 798
23.2.2 PassageRetrieval . . . . . . . . . . . . . . . . . . . . . . . 801
23.2.3 AnswerProcessing . . . . . . . . . . . . . . . . . . . . . . 802

DRAFTxx Contents
23.2.4 EvaluationofFactoidAnswers . . . . . . . . . . . . . . . . 805
23.3 Summarization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 805
23.3.1 SummarizingSingleDocuments . . . . . . . . . . . . . . . 808
23.4 Multi-DocumentSummarization . . . . . . . . . . . . . . . . . . . 8 14
23.4.1 ContentSelectioninMulti-DocumentSummarization . . . 815
23.4.2 InformationOrderingin Multi-DocumentSummarizat ion . 816
23.5 Between Question Answering and Summarization: Query- Focused
Summarization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 819
23.6 SummarizationEvaluation . . . . . . . . . . . . . . . . . . . . . . . 8 23
23.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 825
BibliographicalandHistorical Notes . . . . . . . . . . . . . . . . . . . . . 826
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 828
24 Dialogueand ConversationalAgents 829
24.1 PropertiesofHumanConversations . . . . . . . . . . . . . . . . . . 831

24.1.1 TurnsandTurn-Taking . . . . . . . . . . . . . . . . . . . . 832
24.1.2 LanguageasAction: SpeechActs . . . . . . . . . . . . . . 833
24.1.3 LanguageasJointAction: Grounding . . . . . . . . . . . . 83 4
24.1.4 ConversationalStructure . . . . . . . . . . . . . . . . . . . 836
24.1.5 ConversationalImplicature . . . . . . . . . . . . . . . . . . 83 7
24.2 Basic DialogueSystems . . . . . . . . . . . . . . . . . . . . . . . . 839
24.2.1 ASRcomponent . . . . . . . . . . . . . . . . . . . . . . . 839
24.2.2 NLUcomponent . . . . . . . . . . . . . . . . . . . . . . . 841

24.2.3 GenerationandTTScomponents . . . . . . . . . . . . . . 844
24.2.4 DialogueManager . . . . . . . . . . . . . . . . . . . . . . 845
24.2.5 DialogueManagerErrorHandling: Conﬁrmation/Reje ction 849
24.3 VoiceXML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 851
24.4 DialogueSystemDesignandEvaluation . . . . . . . . . . . . . . . 854
24.4.1 DesigningDialogueSystems . . . . . . . . . . . . . . . . . 854
24.4.2 DialogueSystemEvaluation . . . . . . . . . . . . . . . . . 855

24.5 Information-state& DialogueActs . . . . . . . . . . . . . . . . . . 857
24.5.1 DialogueActs . . . . . . . . . . . . . . . . . . . . . . . . 859
24.5.2 InterpretingDialogueActs . . . . . . . . . . . . . . . . . . 860
24.5.3 DetectingCorrectionActs . . . . . . . . . . . . . . . . . . 862
24.5.4 GeneratingDialogueActs: ConﬁrmationandRejectio n . . . 863
24.6 MarkovDecisionProcessArchitecture . . . . . . . . . . . . . . . . 865
24.7 Advanced: Plan-basedDialogueAgents . . . . . . . . . . . . . . . 869

24.7.1 Plan-InferentialInterpretationandProduction . . . . . . . . 869
24.7.2 TheIntentionalStructureofDialogue . . . . . . . . . . . . 872
24.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 874
BibliographicalandHistorical Notes . . . . . . . . . . . . . . . . . . . . . 875
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 876
25 MachineTranslation 879
25.1 WhyisMachineTranslationSo Hard? . . . . . . . . . . . . . . . . 8 82

25.1.1 Typology . . . . . . . . . . . . . . . . . . . . . . . . . . . 882

DRAFTContents xxi
25.1.2 OtherStructuralDivergences. . . . . . . . . . . . . . . . . 88 5
25.1.3 LexicalDivergences . . . . . . . . . . . . . . . . . . . . . 885
25.2 Classical MT&theVauquoisTriangle . . . . . . . . . . . . . . . . 887
25.2.1 DirectTranslation . . . . . . . . . . . . . . . . . . . . . . 887
25.2.2 Transfer . . . . . . . . . . . . . . . . . . . . . . . . . . . . 890
25.2.3 Combiningdirectandtranferapproachesinclassic M T . . . 892
25.2.4 TheInterlinguaIdea: UsingMeaning . . . . . . . . . . . . 89 3
25.3 Statistical MT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 895
25.4P(F|E): the Phrase-BasedTranslationModel . . . . . . . . . . . . . 897
25.5 AlignmentinMT . . . . . . . . . . . . . . . . . . . . . . . . . . . 900
25.5.1 IBMModel1 . . . . . . . . . . . . . . . . . . . . . . . . . 901
25.5.2 HMMAlignment . . . . . . . . . . . . . . . . . . . . . . . 903
25.6 TrainingAlignmentModels . . . . . . . . . . . . . . . . . . . . . . 90 5

25.6.1 EMforTrainingAlignmentModels . . . . . . . . . . . . . 906
25.7 SymmetrizingAlignmentsforPhrase-basedMT . . . . . . . . . . . 908
25.8 DecodingforPhrase-BasedStatistical MT . . . . . . . . . . . . . . 910
25.9 MTEvaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 914
25.9.1 UsingHumanRaters . . . . . . . . . . . . . . . . . . . . . 915
25.9.2 AutomaticEvaluation: Bleu . . . . . . . . . . . . . . . . . 915
25.10 Advanced: SyntacticModelsforMT . . . . . . . . . . . . . . . . . 918

25.11 Advanced: IBM Model3 forfertility-basedalignment . . . . . . . . 920
25.11.1 TrainingforModel3 . . . . . . . . . . . . . . . . . . . . . 923
25.12 Advanced: Log-linearModelsforMT . . . . . . . . . . . . . . . . 924
BibliographicalandHistorical Notes . . . . . . . . . . . . . . . . . . . . . 925
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 927
Index 981

DRAFTForeword
TOBE ADDED
PeterNorvig&StuartRussell, Editors
PrenticeHall Seriesin ArtiﬁcialIntelligence
xxii

DRAFTPreface
This is an exciting time to be working in speech and language p rocessing. Histori-
cally distinct ﬁelds (natural language processing, speech recognition, computational
linguistics, computational psycholinguistics) have begu n to merge. The explosion of
Web-based language techniques, and the commercial availab ility of telephone-based
dialogue systems, and speech synthesis and speech recognit ion have provided an im-
portant impetus for the development of real systems. The ava ilability of very large
on-linecorporahasenabledstatistical modelsoflanguage at everylevel,fromphonet-
ics to discourse. We have tried to draw on this emerging state of the art in the design
ofthispedagogicalandreferencework:
1.Coverage
In attempting to describe a uniﬁed vision of speech and langu age processing,
we cover areas that traditionally are taught in different co urses in different de-
partments: speech recognition in electrical engineering; parsing, semantic in-

terpretation, and machine translation in natural language processing courses in
computer science departments; and computational morpholo gy, phonology and
pragmatics in computational linguistics courses in lingui stics departments. The
bookintroducesthefundamentalalgorithmsofeachofthese ﬁelds,whetherorig-
inally proposed for spoken or written language, whether log ical or statistical in
origin,and attempts to tie togetherthe descriptionsof alg orithmsfromdifferent
domains. We havealso includedcoverageofapplicationslik espelling-checking

andinformationretrievalandextractionaswellasareasli kecognitivemodeling.
A potential problem with this broad-coverage approach is th at it required us to
include introductorymaterial for each ﬁeld; thus linguist s may want to skip our
descriptionof articulatoryphonetics, computerscientis ts may want to skip such
sectionsasregularexpressions,andelectricalengineers skipthesectionsonsig-
nal processing. Of course, even in a book this long, we didn’t have room for
everything. Thus this book should not be considered a substi tute for important

relevant courses in linguistics, automata and formal langu age theory, artiﬁcial
intelligence,machinelearning,statistics, orinformati ontheory.
2.Emphasisonpracticalapplications
It is important to show how language-related algorithms and techniques (from
HMMs to uniﬁcation, from the lambda calculus to log-linear m odels) can be
appliedtoimportantreal-worldproblems: spellchecking, textdocumentsearch,
machine translation, speech recognition, information ext raction from the web,

and spoken-languagedialogue. We have attempted to do this b y integrating the
descriptionoflanguageprocessingapplicationsintoeach chapter. Theadvantage
of this approach is that as the relevant linguistic knowledg e is introduced, the
studenthasthebackgroundto understandandmodelaparticu lardomain.
3.Emphasisonscientiﬁcevaluation
The recent prevalence of statistical algorithms in languag e processing and the
growth of organized evaluations of speech and language proc essing systems

has led to a new emphasis on evaluation. Most chapters theref ore include an
evaluationsection describing modern empirical methods fo r evaluatingsystems
xxiii

DRAFTxxiv Preface
and performingerror analysis, includingsuch conceptsas t rainingand test sets,
cross-validation,andinformation-theoreticevaluation metricslikeperplexity.
4.Descriptionof widelyavailablelanguageprocessingresou rces
Modernspeechandlanguageprocessingisheavilybasedonco mmonresources:
raw speech and text corpora, annotated corpora and treebank s, standard tagsets
for labeling pronunciation, part-of-speech, parses, word -sense, and dialogue-
levelphenomena. We havetried to introducemanyofthese imp ortantresources
throughout the book (e.g., the Brown, Switchboard, Fisher, CALLHOME , ATIS,
TREC, MUC, and BNC corpora) and provide complete listings of many useful
tagsets and coding schemes (such as the Penn Treebank, CLAWS C5 and C7,
and the ARPAbet) but some inevitably got left out. Furthermo re,rather than in-
clude references to URLs for many resources directly in the t extbook, we have
placedthemonthebook’sWeb site, wheretheycanmorereadil yupdated.

The book is primarily intended for use in a graduate or advanc ed undergraduate
course or sequence. Because of its comprehensive coverage a nd the large number of
algorithms,the bookis also useful as a referenceforstuden tsand professionalsin any
oftheareasofspeechandlanguageprocessing.
Overviewofthe Book
The book is divided into ﬁve parts in addition to an introduct ion and end matter.
Part I, “Words”, introduces concepts related to the process ing of words and simple
word sequences: word segmentation, word morphology, word e dit distance, parts-of-

speech, and the algorithmsused to processthem: regularexp ressions,ﬁnite automata,
ﬁnite transducers, N-grams, Hidden Markov Models, and log-linear models. Part I I,
“Speech”, introduces linguistic phonetics, and then cover s speech synthesis, speech
recognition,andlinguistictopicsincomputationalphono logy.PartIII,“Syntax”,intro-
ducesphrasestructuregrammarsforEnglishandgivesessen tialalgorithmsforprocess-
ingstructuredsyntacticrelationshipsamongwords: theCK YandEarleyalgorithmsfor

parsing,statisticalparsing,uniﬁcationandtypedfeatur estructures,andanalyticaltools
liketheChomskyhierarchyandthepumpinglemma. PartIV,“S emanticsandPragmat-
ics”, introducesﬁrst orderpredicatecalculusand other wa ysof representingmeaning,
the lambda calculus, lexical semantics, lexical semantic r esources such as WordNet,
PropBank, and FrameNet,, and computational models of lexic al semantics for word
similarity, word sense disambiguation, and discourse topi cs like coreference and co-

herence. Part V, “Applications”, covers information extra ction, machine translation
anddialogandconversationalagents.
Using this Book
The book providesenough material to be used for a full-year s equence in speech and
languageprocessing. Itisalsodesignedsothatitcanbeuse dforanumberofdifferent
usefulone-termcourses:

DRAFTPreface xxv
NLP NLP Speech+NLP Comp. Ling.
1quarter 1semester 1semester 1quarter
1.Intro 1.Intro 1.Intro 1.Intro
2.Regex, FSA 2.Regex, FSA 2.Regex, FSA 2.Regex, FSA
4.N-grams 4.N-grams 4.N-grams 3.Morph., FST
5.POStagging 5.POStagging 5.POStagging 4.N-grams
12.CFGs 6.HMMS 6.HMMs 5.POStagging
13.Parsing 12.CFGs 8.TTS 13.Parsing
14.Stat.Parsing 13.Parsing 9.ASR 14.Stat.Parsing
19.Lex. Semantics 14.Stat.Parsing 12.CFGs 15.Complexity
20.Comp. Lex. Sem. 17.Semantics 13.Parsing 16.Uniﬁcation
23.QA & Summar. 18.Comp. Semantics 14.Stat.Parsing 20.Comp. Lex.Sem.
25.MT 19.Lex. Semantics 17.Semantics 21.Discourse
20.Comp. Lex. Sem. 19.Lex.Sem.
21.Discourse 20.Comp. Lex.Sm.
22.IE 22.IE
23.QA & Summar. 24.Dialog
25.MT 25.MT.
Selected chapters from the book could also be used to augment courses in Artiﬁ-
cial Intelligence, Cognitive Science, InformationRetrie val, or Electrical Engineering-
orientedcoursesinSpeechProcessing.
Resourcesassociated withthe booksuch asonlineversionso fﬁguresandpointers

towebcontentcanbefoundat thebookhome-page:
http://www.cs.colorado.edu/˜martin/slp.html .
AndyKehlerwrote the Discoursechapterforthe ﬁrst edition ;we tookAndy’schapter
as a starting point for this second edition chapter. Similar ly, Nigel Ward wrote most
of the MT chapter for the ﬁrst edition, and we used this as the s tarting point for the
MT chapter of this second edition. Kevin Bretonnel Cohen wro te section 22.5 on

biomedicalinformationextraction. Keith VanderLindenwr otethe Generationchapter
inthe ﬁrstedition.
Danwouldliketothankhisparentsforencouraginghimtodot herightthing,doit
in a timely fashion, and make time for going to the gym. He woul d also like to thank
NelsonMorgan,forintroducinghimtospeechrecognitionan dteachinghimtoask“but
doesitwork?”;JerryFeldman,forsharinghiscommitmentto ﬁndingtherightanswers
andteachinghim toask “but isit reallyimportant?”;ChuckF illmore,hisﬁrst advisor,

forsharinghisloveforlanguageandteachinghimtoalwaysg olookatthedata,Robert
Wilensky,hisdissertationadvisor,forteachinghimtheim portanceofcollaborationand
groupspiritinresearch,ChrisManningforbeingasuperbco llaboratoratStanford,and
ofcourseall hisformercolleaguesatBoulder.
Jim would like to thank his parents for encouraging him and al lowing him to fol-
low what must have seemed like an odd path at the time. He would also like to thank
his dissertation advisor, Robert Wilensky,for giving him h is start in NLP at Berkeley;

PeterNorvig,forprovidingmanypositiveexamplesalongth eway;RickAlterman,for
encouragementand inspiration at a critical time; and Chuck Fillmore, George Lakoff,

DRAFTxxvi Preface
Paul Kay, andSusanna Cummingforteaching himwhat little he knowsabout linguis-
tics; MarthaPalmer, TammySumnerand Wayne Ward forbeing wo nderfulcollabora-
torsat Boulder. Finally,Jim wouldlike to thankhiswife Lin daforall her supportand
patience through the years, and his daughter Katie who has wa ited her entire life for
thecompletionofthisedition.
BoulderandStanfordhavebeenveryrewardingplacesto work onspeechandlan-
guageprocessing. We’dliketothankourdepartments,ourco lleagues,andourstudents
at both places for their collaborations,which have greatly inﬂuencedour research and
teaching.
We are grateful to the many people who helped enormously on th e ﬁrst edition
of the book. This second edition has also beneﬁted from our ma ny readers and from
their course-testing. Special thanksfor extraordinarily helpfulcommentsand ideason
wide swaths of the book to Regina Barzilay, Philip Resnik, Em ily Bender, and Adam
Przepi´ orkowski. Our editor Tracy Dunkelberger, our serie s editors Peter Norvig and

teaching.
Stuart Russell, and our production editor Scott DiSanno mad e many helpful sugges-
tionsondesignandcontent. We arealso indebtedto manyfrie ndsandcolleagueswho
read individual sections of the book or answered our many que stions for their com-
ments and advice, including the students in our classes at th e University of Colorado,
Boulder,StanfordUniversity,and the LSA Summer Institute sat the University of Illi-
noisat Urbana-Champaign(1999),MIT(2005),andStanford( 2007),aswell as:

Rieks op den Akker, Kayra Akman, Angelos Alexopoulos, Robin Aly, S. M. Niaz Ar-
iﬁn, Nimar S. Arora, Tsz-Chiu Au, G. W. Blackwood, Bai Xiaoji ng, Ellie Baker, Jason
Baldridge, Clay Beckner, Jonathan Boiser, Marion Bond, Mar co Aldo Piccolino Boni-
forti, Onn Brandman, Chris Brew, Tore Bruland, Denis Bueno, Sean M. Burke, Bill
Byrne, Kai-Uwe Carstensen, Alejandro Cdebaca, PichuanCha ng, Grace Chung, Andrew
Clausen,KevinB.Cohen, FrederikCoppens, StephenCox,Her ibertoCuay´ ahuitl, Martin

Davidsson, Paul Davis, Jon Dehdari, Franz Deuzer, Mike Dill inger, Bonnie Dorr, Jason
Eisner, John Eng, Ersin Er, Hakan Erdogan, G¨ ulsen Eryiˇ git , Barbara Di Eugenio, Eric
Fosler-Lussier,OlacFuentes,DaleGerdemann,DanGildea, FilipGinter,CynthiaGirand,
AnthonyGitter,JohnA.Goldsmith,MichelleGregory,Rocio Guillen,JeffreyS.Haemer,
Adam Hahn, Patrick Hall, Harald Hammarstr¨ om, Mike Hammond , Eric Hansen, Marti
Hearst, Paul Hirschb¨ uhler, Julia Hirschberg, Julia Hocke nmaier, Jeremy Hoffman, Greg

Hullender, Gaja Jarosz, Eric W. Johnson, Chris Jones, Edwin de Jong, Bernadette Joret,
Fred Karlsson, Graham Katz, Stefan Kaufmann, Andy Kehler, M anuel Kirschner, Shel-
donKlein,KevinKnight,Jean-PierreKoenig,KimmoKoskenn iemi,AlexanderKostyrkin,
ValerieKrugler,MikkoKurimo,MikeLeBeau,Chia-YingLee, JaeyongLee,ScottLeish-
man, Szymon Letowski, Liuyang Li, Marc Light, Greger Lind’e n, Pierre Lison, Di-
ane Litman, Chao-Lin Liu, Feng Liu, Roussanka Louka, Artyom Lukanin, Jean Ma,

Maxim Makatchev, Inderjeet Mani, Steve Marmon, Marie-Cath erine de Marneffe, Hen-
drik Maryns, Jon May, Dan Melamed, Johanna Moore, Nelson Mor gan, Emad Naw-
fal, Mark-Jan Nederhof, Hwee Tou Ng, John Niekrasz, Rodney N ielsen, Yuri Niya-
zov, Kris Nuttycombe, Mike O’Connell, Robert Oberbrecklin g, Scott Olsson, Woodley
Packard, Gabor Palagyi, Gerald Penn, Rani Pinchuk, Sameer P radhan, Kathryn Pruitt,
Drago Radev, William J. Rapaport, Dan Ramage, Ron Regan, Ehu d Reiter, Steve Re-

nals, Chang-han Rhee, Dan Rose, Mike Rosner, Deb Roy, Teodor Rus, William Gregory
Sakas,MuratSaraclar,StefanSchaden, AnnaSchapiro,Zhan gSen,MattShannon, Stuart
C.Shapiro,IlyaSherman,LokeshShrestha,NathanSilberma n,OtakarSmrz,RionSnow,
Niyue Tan, Frank Yung-Fong Tang, Ahmet C¨ uneyd Tantuˇ g, Pau l Taylor, Lorne Temes,

DRAFTPreface xxvii
RichThomason, AlmerS.Tigelaar,RichardTrahan,AntoineT rux,ClementWang,Nigel
Ward, Rachel Weston, Janyce Wiebe, Lauren Wilcox, Ben Wing, Dean Earl Wright III,
DekaiWu,LeiWu,EricYeh,AlanC.Yeung,MargalitZabludows ki, MennovanZaanen,
SamShaojun Zhao, andXingtao Zhao.
DanielJurafsky
Stanford,California
JamesH.Martin
Boulder,Colorado

DRAFTPREL IMIN AR Y PR O OFS .
Unpublished Work c⃝2008 by Pearson Education, Inc. To be published by Pearson Pr entice Hall,
Pearson Education, Inc., Upper Saddle River, New Jersey. Al l rights reserved. Permission to use
this unpublished Work is granted to individuals registerin g through Melinda_Haggerty@prenhall.com
for the instructional purposes not exceeding one academic t erm or semester.
Chapter1
Dave Bowman: Openthe pod bay doors, HAL.
HAL:I’msorry Dave, I’mafraidI can’t dothat.
StanleyKubrick andArthur C.Clarke,
screenplay of 2001: A Space Odyssey
Theideaofgivingcomputerstheabilitytoprocesshumanlan guageisasoldastheidea
of computers themselves. This book is about the implementat ion and implications of
thatexcitingidea. We introduceavibrantinterdisciplina ryﬁeldwith manynamescor-
responding to its many facets, names like speech and language processing ,human
language technology ,natural language processing ,computational linguistics , and

speech recognition and synthesis . The goal of this new ﬁeld is to get computers
toperformusefultasksinvolvinghumanlanguage,taskslik eenablinghuman-machine
communication,improvinghuman-humancommunication,ors implydoingusefulpro-
cessingoftextorspeech.
Oneexampleofausefulsuchtaskisa conversationalagent . TheHAL9000com-Conversationalagent
puterinStanleyKubrick’sﬁlm 2001: ASpaceOdyssey isoneofthemostrecognizable
charactersin twentieth-centurycinema. HAL is an artiﬁcia l agent capableof such ad-

vancedlanguage-processingbehaviorasspeakingandunder standingEnglish,andat a
crucialmomentintheplot,evenreadinglips. Itisnowclear thatHAL’screatorArthur
C. Clarke was a little optimistic in predicting when an artiﬁ cial agent such as HAL
would be available. But just how far off was he? What would it t ake to create at least
thelanguage-relatedpartsofHAL?We callprogramslikeHAL thatconversewithhu-
mansvianaturallanguage conversationalagents ordialoguesystems . Inthistextwe Dialogue system

study the various components that make up modern conversati onal agents, including
language input ( automatic speech recognition andnatural language understand-
ing)andlanguageoutput( naturallanguagegeneration andspeech synthesis ).
Let’sturn to anotherusefullanguage-relatedtask, that of makingavailable to non-
English-speakingreaders the vast amount of scientiﬁc info rmation on the Web in En-
glish. OrtranslatingforEnglishspeakersthehundredsofm illionsofWebpageswritten

in other languageslike Chinese. The goal of machine translation is to automaticallyMachinetranslation
translate a document from one language to another. We will in troduce the algorithms
and mathematical toolsneeded to understandhow modernmach ine translationworks.
Machine translation is far from a solved problem; we will cov er the algorithms cur-
rentlyusedin theﬁeld,aswell asimportantcomponenttasks .
Many other language processing tasks are also related to the Web. Another such

taskisWeb-basedquestionanswering . Thisisageneralizationofsimplewebsearch,Questionanswering
where instead of just typing keywords a user might ask comple te questions, ranging
fromeasyto hard,likethefollowing:
•Whatdoes“divergent”mean?
•WhatyearwasAbrahamLincolnborn?
•HowmanystateswereintheUnitedStatesthat year?

DRAFT2 Chapter 1. Introduction
•HowmuchChinesesilkwasexportedtoEnglandbytheendofthe 18thcentury?
•Whatdoscientiststhinkabouttheethicsofhumancloning?
Someofthese, suchas deﬁnition questions,or simple factoidquestionslike dates
and locations, can already be answered by search engines. Bu t answering more com-
plicated questions might require extracting informationt hat is embeddedin other text
on a Web page, or doing inference (drawing conclusions based on known facts), or
synthesizingandsummarizinginformationfrommultipleso urcesorwebpages. Inthis
text we study the variouscomponentsthat make up modern unde rstandingsystems of
thiskind,including informationextraction ,word sense disambiguation ,andso on.
Although the subﬁelds and problems we’ve described above ar e all very far from
completely solved, these are all very active research areas and many technologies are
already available commercially. In the rest of this chapter we brieﬂy summarize the

kinds of knowledgethat is necessary for these tasks (and oth ers likespell correction ,
grammarchecking ,andsoon),aswellasthemathematicalmodelsthatwillbein tro-
ducedthroughoutthebook.
1.1 Knowledgein SpeechandLanguageProcessing
What distinguishes language processing applications from other data processing sys-
tems is their use of knowledge of language . Consider the Unix wcprogram, which is
used to count the total number of bytes, words, and lines in a t ext ﬁle. When used to

countbytesandlines, wcisanordinarydataprocessingapplication. However,wheni t
is used to count the words in a ﬁle it requires knowledge about what it means to be a
word,andthusbecomesa languageprocessingsystem.
Of course, wcis an extremely simple system with an extremely limited and i m-
poverished knowledge of language. Sophisticated conversa tional agents like HAL,
or machine translation systems, or robust question-answer ing systems, require much
broaderand deeperknowledgeof language. To get a feeling fo r the scope and kind of

requiredknowledge,considersomeofwhatHALwouldneedtok nowtoengageinthe
dialoguethat beginsthis chapter,or for a questionansweri ng system to answer one of
thequestionsabove.
HAL must be able to recognize words from an audio signal and to generate an
audiosignalfromasequenceofwords. Thesetasksof speechrecognition andspeech
synthesis tasks require knowledge about phonetics and phonology ; how words are
pronouncedin termsof sequencesof sounds,andhoweachof th esesoundsis realized
acoustically.

Note also that unlike Star Trek’s Commander Data, HAL is capa ble of producing
contractionslike I’mandcan’t. Producing and recognizing these and other variations
of individual words (e.g., recognizing that doorsis plural) requires knowledge about
morphology ,thewaywordsbreakdownintocomponentpartsthatcarrymea ningslike
singularversusplural.
Movingbeyondindividualwords,HAL must use structuralkno wledgeto properly
string together the words that constitute its response. For example, HAL must know

DRAFTSection1.1. KnowledgeinSpeechandLanguageProcessing 3
thatthefollowingsequenceofwordswill notmakesensetoDa ve,despitethefactthat
it containspreciselythesameset ofwordsastheoriginal.
(1.1)I’mI do,sorrythat afraidDaveI’mcan’t.
Theknowledgeneededto orderandgroupwordstogethercomes underthe headingof
syntax.
Nowconsideraquestionansweringsystem dealingwiththefo llowingquestion:
(1.2)HowmuchChinesesilk wasexportedtoWestern Europebytheen dofthe18th
century?
In orderto answer this questionwe needto knowsomethingabo utlexical seman-
tics,themeaningofallthewords( export,orsilk) aswellas compositionalsemantics
(what exactly constitutes Western Europe as opposed to Eastern or Southern Europe,
what does endmean when combined with the 18th century . We also need to know
something about the relationship of the words to the syntact ic structure. For example
we need to knowthat by the endof the 18thcentury is a temporalend-point,and nota
descriptionoftheagent,astheby-phraseis inthefollowin gsentence:

(1.3)HowmuchChinesesilk wasexportedtoWestern Europebysouth ern
merchants?
WealsoneedthekindofknowledgethatletsHALdeterminetha tDave’sutterance
is a request for action, as opposedto a simple statement abou t the world or a question
aboutthedoor,asinthefollowingvariationsofhisorigina lstatement.
REQUEST : HAL,openthepodbaydoor.
STATEMENT : HAL,thepodbaydoorisopen.
INFORMATION QUESTION :HAL,isthepodbaydooropen?
Next, despite its bad behavior, HAL knows enough to be polite to Dave. It could,

for example, have simply replied NoorNo, I won’t open the door . Instead, it ﬁrst
embellishesitsresponsewiththe phrases I’msorry andI’mafraid ,andthenonlyindi-
rectly signals its refusal by saying I can’t, rather than the more direct (and truthful) I
won’t.1This knowledge about the kind of actions that speakers inten d by their use of
sentencesis pragmatic ordialogue knowledge.
Anotherkindofpragmaticor discourse knowledgeisrequiredtoanswertheques-
tion
(1.4)HowmanystateswereintheUnitedStates thatyear?

tion
What year is that year? In order to interpret words like that year a question an-
sweringsystemneedstoexaminetheearlierquestionsthatw ereasked;inthiscasethe
previousquestiontalkedabouttheyearthatLincolnwasbor n. Thusthistaskof coref-
erenceresolution makesuseofknowledgeabouthowwordslike thatorpronounslike
itorsherefertopreviouspartsofthe discourse .
To summarize, engaging in complex language behavior requir es various kinds of
knowledgeoflanguage:

tion
1For those unfamiliar with HAL, it is neither sorry nor afraid , nor is it incapable of opening the door. It
has simply decided in aﬁtof paranoia to kill its crew.

DRAFT4 Chapter 1. Introduction
•PhoneticsandPhonology— knowledgeaboutlinguisticsound s
•Morphology— knowledgeofthemeaningfulcomponentsofword s
•Syntax— knowledgeofthestructuralrelationshipsbetween words
•Semantics—knowledgeofmeaning
•Pragmatics— knowledgeof the relationshipof meaningto the goals and inten-
tionsofthespeaker.
•Discourse— knowledgeaboutlinguisticunitslargerthana s ingleutterance
1.2 Ambiguity
A perhaps surprising fact about these categories of linguis tic knowledge is that most
tasks in speech and languageprocessingcan be viewedas reso lvingambiguity at one Ambiguity
of these levels. We say some input is ambiguous if there are multiple alternative lin- Ambiguous
guisticstructuresthatcanbebuiltforit. Considerthespo kensentence Imadeherduck.
Hereareﬁvedifferentmeaningsthissentencecouldhave(se eifyoucanthinkofsome
more),eachofwhichexempliﬁesanambiguityat somelevel:
(1.5)Icookedwaterfowlforher.
(1.6)Icookedwaterfowlbelongingtoher.
(1.7)Icreatedthe (plaster?) ducksheowns.

(1.8)Icausedherto quicklylowerherheadorbody.
(1.9)Iwavedmymagicwandandturnedherintoundifferentiatedwa terfowl.
Thesedifferentmeaningsarecausedbyanumberofambiguiti es. First,thewords duck
andherare morphologicallyor syntactically ambiguousin their pa rt-of-speech. Duck
can be a verb or a noun, while hercan be a dative pronoun or a possessive pronoun.
Second,theword makeissemanticallyambiguous;itcanmean createorcook. Finally,
the verbmakeis syntactically ambiguous in a different way. Makecan be transitive,

that is, taking a single direct object (1.6), or it can be ditr ansitive, that is, taking two
objects(1.9),meaningthattheﬁrstobject( her)gotmadeintothesecondobject( duck).
Finally,makecantakeadirectobjectandaverb(1.8),meaningthattheobj ect(her)got
caused to perform the verbal action ( duck). Furthermore, in a spoken sentence, there
is an even deeper kind of ambiguity; the ﬁrst word could have b eeneyeor the second
wordmaid.
Wewilloftenintroducethemodelsandalgorithmswepresent throughoutthebook

wordmaid.
as waysto resolveordisambiguate these ambiguities. For exampledecidingwhether
duckis a verb or a noun can be solved by part-of-speech tagging . Deciding whether
makemeans“create” or “cook” can be solved by word sense disambiguation . Reso-
lutionofpart-of-speechandwordsenseambiguitiesaretwo importantkindsof lexical
disambiguation . A wide variety of tasks can be framed as lexical disambiguat ion
problems. Forexample,atext-to-speechsynthesissystemr eadingtheword leadneeds

to decide whether it should be pronounced as in lead pipe or as inlead me on . By
contrast,decidingwhether herandduckarepartofthesameentity(asin(1.5)or(1.8))
oraredifferententity(asin(1.6))isanexampleof syntacticdisambiguation andcan

DRAFTSection1.3. ModelsandAlgorithms 5
be addressed by probabilistic parsing . We will also consider ambiguities that don’t
arise in thisparticularexample,such as determiningwheth era sentence is a statement
ora question(whichcanberesolvedby speech actinterpretation ).
1.3 ModelsandAlgorithms
One of the key insights of the last 50 years of research in lang uage processing is that
the various kinds of knowledgedescribed in the last section s can be captured through
theuseofasmallnumberofformalmodels,ortheories. Fortu nately,thesemodelsand
theoriesarealldrawnfromthestandardtoolkitsofcompute rscience,mathematics,and
linguisticsandshouldbegenerallyfamiliartothosetrain edinthoseﬁelds. Amongthe
mostimportantmodelsare statemachines ,rulesystems ,logic,probabilisticmodels ,
andvector-space models . These models, in turn, lend themselves to a small number
of algorithms, among the most important of which are state space search algorithms
such asdynamic programming , and machine learning algorithms such as classiﬁers

andExpectation-Maximization( EM) andotherlearningalgorithms.
In their simplest formulation, state machines are formal mo dels that consist of
states, transitions among states, and an input representat ion. Some of the variations
of this basic model that we will consider are deterministic andnon-deterministic
ﬁnite-stateautomata andﬁnite-statetransducers .
Closely related to these modelsare their declarativecount erparts: formal rule sys-
tems. Amongthemoreimportantoneswewillconsider(inboth probabilisticandnon-

probabilisticformulations)are regulargrammars andregularrelations ,context-free
grammars , andfeature-augmentedgrammars . State machinesand formal rule sys-
temsarethemaintoolsusedwhendealingwithknowledgeofph onology,morphology,
andsyntax.
A third class of models that plays a critical role in capturin g knowledge of lan-
guageare modelsbased on logic. We will discuss ﬁrst order logic , also knownas the
predicate calculus , as well as such related formalisms as lambda-calculus, fea ture-

structures, and semantic primitives. These logical repres entations have traditionally
been used for modeling semantics and pragmatics, although m ore recent work has
tended to focus on potentially more robust techniques drawn from non-logical lexical
semantics.
Probabilistic models are crucial for capturing every kind o f linguistic knowledge.
Each of the other models(state machines, formal rule system s, and logic) can be aug-
mented with probabilities. For example the state machine ca n be augmented with

probabilities to become the weighted automaton orMarkov model . We will spend
a signiﬁcant amount of time on hidden Markov models orHMMs, which are used
everywhereintheﬁeld,inpart-of-speechtagging,speechr ecognition,dialogueunder-
standing, text-to-speech, and machine translation. The ke y advantage of probabilistic
modelsistheirabilitytosolvethemanykindsofambiguityp roblemsthatwediscussed
earlier;almostanyspeechandlanguageprocessingproblem canberecastas: “given N
choicesforsomeambiguousinput,choosethemostprobableo ne”.

Finally,vector-spacemodels,basedonlinearalgebra,und erlieinformationretrieval

DRAFT6 Chapter 1. Introduction
andmanytreatmentsofwordmeanings.
Processinglanguageusinganyofthesemodelstypicallyinv olvesa searchthrough
a space of states representing hypotheses about an input. In speech recognition, we
searchthrougha spaceofphonesequencesforthe correctwor d. Inparsing,we search
througha space oftreesforthe syntacticparse ofan inputse ntence. In machinetrans-
lation,wesearchthroughaspaceoftranslationhypotheses forthecorrecttranslationof
a sentence into another language. For non-probabilisticta sks, such as tasks involving
state machines, we use well-known graph algorithms such as depth-ﬁrst search . For
probabilistictasks,weuseheuristicvariantssuchas best-ﬁrst andA*search ,andrely
ondynamicprogrammingalgorithmsforcomputationaltract ability.
Machinelearningtoolssuch as classiﬁers andsequence models play a signiﬁcant
role in many language processing tasks. Based on attributes describing each object,
a classifer attempts to assign a single object to a single cla ss while a sequence model

attemptstojointlyclassifya sequenceofobjectsintoa seq uenceofclasses.
Forexample,inthetaskofdecidingwhetherawordisspelled correctlyornot,clas-
siﬁers such as decision trees ,support vector machines ,Gaussian Mixture Models ,
andlogistic regression could be used to make a binary decision (correct or incorrect )
for one word at a time. Sequence models such as hidden Markov models ,maxi-
mumentropyMarkovmodels ,andconditionalrandomﬁeldscouldbeusedtoassign
correct/incorrectlabelstoall thewordsina sentenceaton ce.

Finally, researchers in language processing use many of the same methodologi-
cal tools that are used in machine learning research—the use of distinct training and
test sets, statistical techniques like cross-validation , and careful evaluation of trained
systems.
1.4 Language,Thought,andUnderstanding
To many, the ability of computers to process language as skil lfully as we humans do
will signal the arrival of truly intelligent machines. The b asis of this belief is the fact
that the effective use of language is intertwined with our ge neral cognitive abilities.

systems.
Amongtheﬁrstto considerthecomputationalimplicationso fthisintimateconnection
was Alan Turing (1950). In this famous paper, Turing introdu ced what has come to
be knownas the Turing test . Turingbeganwith the thesis that the questionof what it Turingtest
would mean for a machine to think was essentially unanswerab le due to the inherent
imprecision in the terms machineandthink. Instead, he suggested an empirical test, a
game, in which a computer’s use of language would form the bas is for determining if

it couldthink. Ifthe machinecouldwinthegameitwouldbeju dgedintelligent.
In Turing’s game, there are three participants: two people a nd a computer. One of
thepeopleisacontestantandplaystheroleofaninterrogat or. Towin,theinterrogator
must determine which of the other two participants is the mac hine by asking a series
of questions via a teletype. The task of the machine is to fool the interrogator into
believingitisapersonbyrespondingasapersonwouldtothe interrogator’squestions.

The task of the second humanparticipant is to convincethe in terrogatorthat the other
participantisthemachine,andthatsheishuman.

DRAFTSection1.4. Language,Thought,andUnderstanding 7
The following interaction from Turing’s paper illustrates the kind of interactions
he had in mind. Clearly a convincingsimulationof humanityd oesn’trequirebeingan
expertinall domains:
Q:Please write measonnetonthetopicoftheForthBridge.
A:Countmeoutonthisone. Inevercouldwrite poetry.
Q:Add34957to 70764.
A:(Pauseabout30secondsandthengiveansweras) 105621.
Turing predicted that roughly by the end of the twentieth cen tury a machine with 10
gigabytesofmemorywouldhavearounda30%chanceoffooling ahumaninterrogator
after 5 minutesof questioning. Giventhe fact that you can fo olsome of the peopleall
the time, it is not clear how rigorous this particular standa rd is. Nevertheless, the
critical issue for Turing was that using language as humans d o is sufﬁcient, by itself,
asanoperationaltest forintelligence.
Surprisingly,manyofthe issuesraisedbyTuringbecamerel evantasearlyas1966
in the context of the ELIZAprogram(Weizenbaum, 1966). ELIZA was an early nat- ELIZA

urallanguageprocessingsystemcapableofcarryingona lim itedformofconversation
with a user. Consider the following session with a version of ELIZA that imitated the
responsesofa Rogerianpsychotherapist.
User1:Youare like myfather insome ways.
ELIZA1:WHATRESEMBLANCEDO YOUSEE
User2:Youare not very aggressive butI thinkyou don’t want me tonot ice that.
ELIZA2:WHATMAKESYOUTHINKI AMNOTAGGRESSIVE
User3:Youdon’t argue withme.
ELIZA3:WHYDO YOUTHINKI DON’TARGUEWITHYOU
User4:Youare afraidof me.
ELIZA4:DOESITPLEASEYOUTOBELIEVEIAM AFRAIDOFYOU

As we will discuss in more detail in Ch. 2, ELIZA is a remarkabl y simple program
that makes use of pattern-matching to process the input and t ranslate it into suitable
outputs. The success of this simple technique in this domain is due to the fact that
ELIZA doesn’t actually need to knowanything to mimic a Rogerian psychotherapist.
AsWeizenbaumnotes,thisisoneofthefewdialoguegenreswh erethelistenercanact
asiftheyknownothingofthe world.
ELIZA’sdeep relevanceto Turing’sideas is that many people who interacted with

ELIZA came to believe that it really understood them and their problems. Indeed,
Weizenbaum (1976) notes that many of these people continued to believe in ELIZA’s
abilities even after the program’s operation was explained to them. In more recent
years,Weizenbaum’sinformaltestshavebeenrepeatedinas omewhatmorecontrolled
setting. Since1991,aneventknownastheLoebnerPrizecomp etitionhasattemptedto
putvariouscomputerprogramstotheTuringtest. Althought hesecontestsseemtohave
little scientiﬁcinterest,a consistentresultovertheyea rshasbeenthateventhecrudest

programscan fool some of the judgessome of the time (Shieber ,1994a). Not surpris-
ingly,theseresultshavedonenothingtoquelltheongoingd ebateoverthesuitabilityof
theTuringtestasatestforintelligenceamongphilosopher sandAIresearchers(Searle,
1980).

DRAFT8 Chapter 1. Introduction
Fortunately, for the purposes of this book, the relevance of these results does not
hinge on whether or not computers will ever be intelligent, o r understand natural lan-
guage. Far more important is recent related research in the s ocial sciences that has
conﬁrmedanotherofTuring’spredictionsfromthesame pape r.
Nevertheless I believe that at the end of the century the use o f words and
educated opinion will have altered so much that we will be abl e to speak
ofmachinesthinkingwithoutexpectingtobe contradicted.
Itisnowclearthatregardlessofwhatpeoplebelieveorknow abouttheinnerworkings
ofcomputers,theytalkaboutthemandinteractwiththemass ocialentities. Peopleact
toward computers as if they were people; they are polite to th em, treat them as team
members, and expectamong otherthings that computersshoul d be able to understand
their needs, and be capable of interacting with them natural ly. For example, Reeves

and Nass (1996)foundthat when a computerasked a humanto eva luate how well the
computerhadbeendoing,thehumangivesmorepositiverespo nsesthanwhenadiffer-
entcomputerasksthesamequestions. Peopleseemedtobeafr aidofbeingimpolite. In
a differentexperiment,Reeves andNass foundthat peopleal so givecomputershigher
performance ratings if the computer has recently said somet hing ﬂattering to the hu-
man. Given these predispositions, speech and language-bas ed systems may provide
many users with the most natural interface for many applicat ions. This fact has led to

along-termfocusintheﬁeldonthedesignof conversationalagents ,artiﬁcialentities
thatcommunicateconversationally.
1.5 TheStateoftheArt
We canonly see ashort distance ahead, but wecan see plenty th ere that needs tobe done.
AlanTuring.
This is an exciting time for the ﬁeld of speech and language pr ocessing. The
startling increase in computing resources available to the average computer user, the
rise of the Web as a massive source of information and the incr easing availability of

wireless mobile access have all placed speech and language p rocessing applications
in the technology spotlight. The following are examples of s ome currently deployed
systemsthatreﬂectthistrend:
•Travelers calling Amtrak, United Airlines and other travel -providers interact
withconversationalagentsthatguidethemthroughthe proc essofmakingreser-
vationsandgettingarrivalanddepartureinformation.
•Luxury car makers such as Mercedes-Benz models provide auto matic speech

recognition and text-to-speech systems that allow drivers to control their envi-
ronmental, entertainment and navigational systems by voic e. A similar spoken
dialoguesystemhasbeendeployedbyastronautsontheInter nationalSpaceSta-
tion.
•Blinkx and other video search companiesprovidesearch serv ices for million of
hoursofvideoontheWebbyusingspeechrecognitiontechnol ogytocapturethe
wordsin thesoundtrack.

DRAFTSection1.6. SomeBriefHistory 9
•Google provides cross-language information retrieval and translation services
wherea user can supplyqueriesin their native languageto se arch collectionsin
anotherlanguage. Googletranslatesthequery,ﬁndsthemos trelevantpagesand
thenautomaticallytranslatesthembacktotheuser’snativ elanguage.
•Large educational publishers such as Pearson, as well as tes ting services like
ETS,useautomatedsystemstoanalyzethousandsofstudente ssays,gradingand
assessingthemina mannerthatisindistinguishablefromhu mangraders.
•Interactivetutors, based on lifelike animated characters , serve as tutorsfor chil-
drenlearningtoread(Wiseet al.,2007).
•Text analysis companies such as Nielsen Buzzmetrics, Umbri a, and Collective
Intellect provide marketing intelligence based on automat ed measurements of
useropinions,preferences,attitudesasexpressedin webl ogs,discussionforums
andusergroups.
1.6 SomeBriefHistory
Historically,speechandlanguageprocessinghasbeentrea tedverydifferentlyin com-

puter science, electrical engineering, linguistics, and p sychology/cognitive science.
Because of this diversity, speech and language processing e ncompasses a number of
differentbutoverlappingﬁeldsinthesedifferentdepartm ents:computationallinguis-
ticsinlinguistics, naturallanguageprocessing incomputerscience, speechrecogni-
tionin electrical engineering, computational psycholinguistics in psychology. This
sectionsummarizesthedifferenthistoricalthreadswhich havegivenrisetotheﬁeldof

speech and language processing. This section will provideo nly a sketch, but many of
thetopicslisted herewill becoveredinmoredetail insubse quentchapters.
1.6.1 Foundational Insights: 1940sand 1950s
The earliest roots of the ﬁeld date to the intellectually fer tile period just after World
WarIIthatgaverisetothecomputeritself. Thisperiodfrom the1940sthroughtheend
of the 1950s saw intense work on two foundational paradigms: theautomaton and
probabilistic orinformation-theoreticmodels .

The automaton arose in the 1950s out of Turing’s (1936) model of algorithmic
computation, considered by many to be the foundation of mode rn computer science.
Turing’sworkledﬁrst tothe McCulloch-Pittsneuron (McCullochandPitts, 1943),a
simpliﬁedmodeloftheneuronasakindofcomputingelementt hatcouldbedescribed
in terms of propositional logic, and then to the work of Kleen e (1951) and (1956) on
ﬁnite automata and regular expressions. Shannon(1948) app lied probabilistic models

of discrete Markov processes to automata for language. Draw ing on the idea of a
ﬁnite-state Markov process from Shannon’s work, Chomsky (1 956) ﬁrst considered
ﬁnite-state machines as a way to characterize a grammar, and deﬁned a ﬁnite-state
language as a language generated by a ﬁnite-state grammar. T hese early models led
to the ﬁeld of formal language theory , which used algebra and set theory to deﬁne
formal languages as sequences of symbols. This includes the context-free grammar,

DRAFT10 Chapter 1. Introduction
ﬁrstdeﬁnedbyChomsky(1956)fornaturallanguagesbutinde pendentlydiscoveredby
Backus(1959)andNauretal.(1960)intheirdescriptionsof theALGOLprogramming
language.
Thesecondfoundationalinsightofthisperiodwasthedevel opmentofprobabilistic
algorithms for speech and language processing, which dates to Shannon’s other con-
tribution: the metaphor of the noisy channel anddecoding for the transmission of
languagethroughmedia like communicationchannelsand spe ech acoustics. Shannon
also borrowed the concept of entropyfrom thermodynamics as a way of measuring
the information capacity of a channel, or the information co ntent of a language, and
performedtheﬁrst measureoftheentropyofEnglishusingpr obabilistictechniques.
It was also during this early period that the sound spectrogr aph was developed
(Koenig et al., 1946), and foundational research was done in instrumental phonetics
that laid the groundwork for later work in speech recognitio n. This led to the ﬁrst

machinespeechrecognizersin the early1950s. In 1952,rese archersat Bell Labsbuilt
a statistical system that could recognize any of the 10 digit s from a single speaker
(Davis et al., 1952). The system had 10 speaker-dependent st ored patterns roughly
representingtheﬁrsttwovowelformantsinthedigits. They achieved97–99%accuracy
by choosing the pattern that had the highest relative correl ation coefﬁcient with the
input.
1.6.2 The Two Camps: 1957–1970
Bytheendofthe1950sandtheearly1960s,speechandlanguag eprocessinghadsplit

input.
verycleanlyintotwo paradigms: symbolicandstochastic.
Thesymbolicparadigmtookofffromtwolinesofresearch. Th eﬁrstwasthework
ofChomskyandothersonformallanguagetheoryandgenerati vesyntaxthroughoutthe
late1950sandearlyto mid1960s,andtheworkofmanylinguis ticsandcomputersci-
entists on parsing algorithms, initially top-downand bott om-up and then via dynamic
programming. One of the earliest completeparsing systems w as Zelig Harris’sTrans-
formationsandDiscourse AnalysisProject (TDAP), whichwa s implementedbetween

June 1958 and July 1959 at the University of Pennsylvania (Ha rris, 1962).2The sec-
ondline of researchwasthe new ﬁeld of artiﬁcial intelligen ce. In the summerof1956
John McCarthy, Marvin Minsky, Claude Shannon, and Nathanie l Rochester brought
together a group of researchers for a two-month workshop on w hat they decided to
call artiﬁcial intelligence(AI).AlthoughAI alwaysinclu deda minorityof researchers
focusing on stochastic and statistical algorithms (includ ing probabilistic models and

neural nets), the major focus of the new ﬁeld was the work on re asoning and logic
typiﬁed by Newell and Simon’s work on the Logic Theorist and t he General Problem
Solver. At this point early natural language understanding systems were built. These
simplesystemsworkedinsingledomainsmainlybyacombinat ionofpatternmatching
and keyword search with simple heuristics for reasoning and question-answering. By
thelate 1960smoreformallogicalsystemsweredeveloped.

The stochastic paradigmtook holdmainly in departmentsof s tatistics and of elec-
2This system was reimplemented recently and is described by J oshi and Hopely (1999) and Karttunen
(1999), who note that the parser was essentially implemente d as acascade of ﬁnite-state transducers.

DRAFTSection1.6. SomeBrief History 11
tricalengineering. Bythelate1950stheBayesianmethodwa sbeginningtobeapplied
to the problem of optical character recognition. Bledsoe an d Browning (1959) built
a Bayesian system for text-recognition that used a large dic tionary and computed the
likelihoodof each observedletter sequence giveneach word in the dictionaryby mul-
tiplyingthelikelihoodsforeachletter. MostellerandWal lace (1964)appliedBayesian
methodstothe problemofauthorshipattributionon TheFederalist papers.
The 1960s also saw the rise of the ﬁrst serious testable psych ological models of
human language processing based on transformational gramm ar, as well as the ﬁrst
on-linecorpora: theBrowncorpusofAmericanEnglish,a1mi llionwordcollectionof
samples from 500 written texts from different genres (newsp aper, novels, non-ﬁction,
academic, etc.), which was assembled at Brown University in 1963–64 (Kuˇ cera and
Francis, 1967; Francis, 1979; Francis and Kuˇ cera, 1982), a nd William S. Y. Wang’s

1967DOC(DictionaryonComputer),anon-lineChinese diale ctdictionary.
1.6.3 FourParadigms: 1970–1983
The next period saw an explosion in research in speech and lan guage processing and
thedevelopmentofa numberofresearchparadigmsthat still dominatetheﬁeld.
Thestochastic paradigm played a huge role in the development of speech reco g-
nition algorithmsin this period,particularlythe use of th e Hidden MarkovModel and
themetaphorsofthenoisychannelanddecoding,developedi ndependentlybyJelinek,

Bahl, Mercer, and colleagues at IBM’s Thomas J. Watson Resea rch Center, and by
Baker at Carnegie Mellon University, who was inﬂuenced by th e work of Baum and
colleagues at the Institute for Defense Analyses in Princet on. AT&T’s Bell Laborato-
ries was also a center for work on speech recognition and synt hesis; see Rabiner and
Juang(1993)fordescriptionsofthewiderangeofthiswork.
Thelogic-based paradigm was begun by the work of Colmerauer and his col-

leagues on Q-systems and metamorphosis grammars (Colmerau er, 1970, 1975), the
forerunnersof Prolog,and Deﬁnite Clause Grammars(Pereir a andWarren, 1980). In-
dependently,Kay’s(1979)workonfunctionalgrammar,ands hortlylater,Bresnanand
Kaplan’s(1982)workonLFG,establishedtheimportanceoff eaturestructureuniﬁca-
tion.
Thenatural languageunderstanding ﬁeld took off duringthis period,beginning
withTerryWinograd’sSHRDLUsystem,whichsimulatedarobo tembeddedinaworld
oftoyblocks(Winograd,1972a). Theprogramwasabletoacce ptnaturallanguagetext

tion.
commands (Move the red block on top of the smaller green one) of a hitherto unseen
complexity and sophistication. His system was also the ﬁrst to attempt to build an
extensive (for the time) grammar of English, based on Hallid ay’s systemic grammar.
Winograd’s model made it clear that the problem of parsing wa s well enough under-
stood to begin to focus on semantics and discourse models. Ro ger Schank and his
colleagues and students (in what was often referred to as the Yale School ) built a se-

riesoflanguageunderstandingprogramsthatfocusedonhum anconceptualknowledge
such as scripts, plans and goals, and human memory organizat ion(Schankand Albel-
son, 1977; Schank and Riesbeck, 1981; Cullingford, 1981; Wi lensky, 1983; Lehnert,
1977). This work often used network-based semantics (Quill ian, 1968; Norman and
Rumelhart, 1975; Schank, 1972; Wilks, 1975c, 1975b; Kintsc h, 1974) and began to

DRAFT12 Chapter 1. Introduction
incorporate Fillmore’s notion of case roles (Fillmore, 196 8) into their representations
(Simmons,1973).
The logic-based and natural-language understanding parad igms were uniﬁed in
systems that used predicate logic as a semantic representat ion, such as the LUNAR
question-answeringsystem(Woods,1967,1973).
Thediscourse modeling paradigm focused on four key areas in discourse. Grosz
and her colleaguesintroducedthe study of substructure in d iscourse, and of discourse
focus (Grosz, 1977a; Sidner, 1983), a number of researchers began to work on au-
tomatic reference resolution (Hobbs, 1978), and the BDI(Belief-Desire-Intention)
framework for logic-based work on speech acts was developed (Perrault and Allen,
1980;CohenandPerrault,1979).
1.6.4 Empiricism andFiniteState Models Redux: 1983–1993
This next decade saw the return of two classes of modelswhich had lost popularityin
thelate1950sandearly1960s,partiallyduetotheoretical argumentsagainstthemsuch

asChomsky’sinﬂuentialreviewofSkinner’s VerbalBehavior (Chomsky,1959b). The
ﬁrst class was ﬁnite-state models, which began to receive at tention again after work
on ﬁnite-state phonology and morphology by Kaplan and Kay (1 981) and ﬁnite-state
models of syntax by Church (1980). A large body of work on ﬁnit e-state models will
bedescribedthroughoutthebook.
The second trend in this period was what has been called the “r eturn of empiri-
cism”; most notably here was the rise of probabilistic model s throughout speech and

language processing, inﬂuenced strongly by the work at the I BM Thomas J. Watson
Research Center on probabilistic models of speech recognit ion. These probabilistic
methodsandothersuchdata-drivenapproachesspreadfroms peechintopart-of-speech
tagging, parsing and attachment ambiguities, and semantic s. This empirical direction
was also accompanied by a new focus on model evaluation, base d on using held-out
data, developing quantitative metrics for evaluation, and emphasizing the comparison
ofperformanceonthesemetricswith previouspublishedres earch.

Thisperiodalsosaw considerableworkonnaturallanguageg eneration.
1.6.5 The FieldComesTogether: 1994–1999
By the last ﬁve years of the millennium it was clear that the ﬁe ld was undergoing
major changes. First, probabilistic and data-driven model s had become quite stan-
dard throughout natural language processing. Algorithms f or parsing, part-of-speech
tagging,referenceresolution,anddiscourseprocessinga ll begantoincorporateproba-
bilities, and employ evaluation methodologiesborrowed fr om speech recognition and

informationretrieval. Second,theincreasesinthespeeda ndmemoryofcomputershad
allowedcommercialexploitationofa numberofsubareasofs peechandlanguagepro-
cessing, in particular speech recognition and spelling and grammar checking. Speech
and language processing algorithms began to be applied to Au gmentative and Alter-
native Communication (AAC). Finally, the rise of the Web emp hasized the need for
language-basedinformationretrievalandinformationext raction.

DRAFTSection1.6. SomeBrief History 13
1.6.6 The RiseofMachine Learning: 2000–2007
The empiricist trends begun in the latter part of the 1990s ac celerated at an astound-
ing pace in the new century. This acceleration was largely dr iven by three synergistic
trends. First, large amounts of spoken and written material became widely available
through the auspices of the Linguistic Data Consortium (LDC ), and other similar or-
ganizations. Importantly, included among these materials were annotated collections
suchasthePennTreebank(Marcusetal.,1993),PragueDepen dencyTreebank(Hajiˇ c,
1998), PropBank (Palmer et al., 2005), Penn Discourse Treeb ank (Miltsakaki et al.,
2004b),RSTBank(Carlsonetal.,2001)andTimeBank(Pustej ovskyet al.,2003b),all
of which layered standard text sources with various forms of syntactic, semantic and
pragmaticannotations. The existenceof these resourcespr omotedthe trendof casting
more complex traditional problems, such as parsing and sema ntic analysis, as prob-

lemsinsupervisedmachinelearning. Theseresourcesalsop romotedtheestablishment
ofadditionalcompetitiveevaluationsforparsing(Dejean andTjongKimSang,2001),
informationextraction(NIST, 2007a; Sang, 2002; Sang and D e Meulder,2003),word
sense disambiguation(Palmeret al., 2001a;Kilgarriffand Palmer,2000),questionan-
swering(VoorheesandTice,1999),andsummarizationDang( 2006).
Second, this increased focus on learning led to a more seriou s interplay with the
statistical machine learning community. Techniques such a s support vector machines

(Boser et al., 1992; Vapnik, 1995), maximum entropy techniq ues and their equiva-
lent formulationasmultinomiallogisticregression(Berg eret al., 1996),andgraphical
Bayesian models (Pearl, 1988) became standard practice in c omputationallinguistics.
Third, the widespread availability of high-performanceco mputing systems facilitated
the training and deployment of systems that could not have be en imagined a decade
earlier.
Finally,near the end of this period,largelyunsuperviseds tatistical approachesbe-

earlier.
gan to receive renewed attention. Progress on statistical a pproachesto machine trans-
lation (Brown et al., 1990; Och and Ney, 2003) and topic model ing (Blei et al., 2003)
demonstratedthat effective applicationscould be constru ctedfromsystems trained on
unannotateddataalone. Inaddition,thewidespreadcostan ddifﬁcultyofproducingre-
liably annotated corpora became a limiting factor in the use of supervised approaches
for many problems. This trend towards the use of unsupervise dtechniqueswill likely
increase.

increase.
1.6.7 OnMultipleDiscoveries
Eveninthisbriefhistoricaloverview,wehavementionedan umberofcasesofmultiple
independentdiscoveriesofthesameidea. Justafewofthe“m ultiples”tobediscussed
in thisbookincludethe applicationofdynamicprogramming to sequencecomparison
by Viterbi, Vintsyuk, Needleman and Wunsch, Sakoe and Chiba , Sankoff, Reichert
et al., and Wagner and Fischer (Chapters 3, 5 and 6) the HMM/noisy ch annel model
of speech recognition by Baker and by Jelinek, Bahl, and Merc er (Chapters 6, 9, and
10); the developmentof context-freegrammarsby Chomsky an d by Backus and Naur

(Chapter12);theproofthatSwiss-Germanhasanon-context -freesyntaxbyHuybregts
and by Shieber (Chapter 15); the application of uniﬁcation t o language processing by

DRAFT14 Chapter 1. Introduction
Colmerauer etal.andbyKayin(Chapter16).
Are these multiplesto be consideredastonishingcoinciden ces? A well-knownhy-
pothesisby sociologist of science Robert K. Merton (1961)a rgues,quite the contrary,
that
allscientiﬁc discoveriesareinprinciplemultiples,incl udingthosethat on
thesurfaceappearto besingletons.
Of course there are many well-known cases of multiple discov ery or invention; just a
fewexamplesfromanextensivelistinOgburnandThomas(192 2)includethemultiple
inventionof the calculus by Leibnitz and by Newton, the mult iple developmentof the
theory of natural selection by Wallace and by Darwin, and the multiple invention of
the telephoneby Gray and Bell.3But Merton givesa furtherarray of evidencefor the
hypothesisthatmultiplediscoveryistheruleratherthant heexception,includingmany
casesofputativesingletonsthatturnoutbearediscoveryo fpreviouslyunpublishedor
perhapsinaccessiblework. Anevenstrongerpieceofeviden ceishisethnomethodolog-

icalpointthatscientiststhemselvesactundertheassumpt ionthatmultipleinventionis
the norm. Thusmanyaspectsof scientiﬁc lifeare designedto helpscientists avoidbe-
ing “scooped”; submission dates on journal articles; caref ul dates in research records;
circulationofpreliminaryortechnicalreports.
1.6.8 A FinalBrief NoteonPsychology
Many of the chapters in this book include short summaries of p sychological research
on human processing. Of course, understandinghumanlangua geprocessing is an im-

portant scientiﬁc goal in its own right and is part of the gene ral ﬁeld of cognitive sci-
ence. However, an understanding of human language processi ng can often be helpful
in building better machine models of language. This seems co ntrary to the popular
wisdom, which holdsthat direct mimicry of nature’salgorit hmsis rarely useful in en-
gineering applications. For example, the argument is often made that if we copied
nature exactly, airplanes would ﬂap their wings; yet airpla nes with ﬁxed wings are a

more successful engineeringsolution. But languageis not a eronautics. Cribbingfrom
natureissometimesusefulforaeronautics(afterall, airp lanesdohavewings),but it is
particularly useful when we are trying to solve human-cente redtasks. Airplane ﬂight
has differentgoalsthan birdﬂight; but the goal of speech re cognitionsystems, for ex-
ample, is to perform exactly the task that human court report ers perform every day:
transcribespokendialog. Sincepeoplealreadydothiswell ,wecanlearnfromnature’s

previous solution. Since an important application of speec h and language processing
systems is for human-computerinteraction, it makes sense t o copy a solution that be-
havesthewaypeopleare accustomedto.
3Ogburn and Thomas are generally credited with noticing that the prevalence of multiple inventions sug-
geststhatthecultural milieu andnotindividual genius ist hedeciding causalfactor inscientiﬁc discovery. In
an amusing bit of recursion, however, Merton notes that even this idea has been multiply discovered, citing

sources from the19th century and earlier!

DRAFTSection1.7. Summary 15
1.7 Summary
Thischapterintroducestheﬁeldofspeechandlanguageproc essing. Thefollowingare
someofthehighlightsofthischapter.
•A good way to understand the concerns of speech and language p rocessing re-
search is to consider what it would take to create an intellig ent agent like HAL
from2001: A Space Odyssey,or builda web-basedquestionans werer,or a ma-
chinetranslationengine.
•Speechand languagetechnologyrelies on formal models, or r epresentations,of
knowledge of language at the levels of phonology and phoneti cs, morphology,
syntax, semantics, pragmatics and discourse. A number of fo rmal models in-
cludingstate machines, formal rule systems, logic, and pro babilistic modelsare
usedto capturethisknowledge.
•Thefoundationsofspeechandlanguagetechnologylieincom puterscience,lin-
guistics,mathematics,electricalengineeringandpsycho logy. Asmallnumberof
algorithmsfrom standard frameworksare used throughoutsp eech and language
processing.

•The critical connection between language and thought has pl aced speech and
languageprocessingtechnologyatthecenterofdebateover intelligentmachines.
Furthermore,researchonhowpeopleinteractwithcomplexm ediaindicatesthat
speech and language processing technology will be critical in the development
offuturetechnologies.
•Revolutionary applications of speech and language process ing are currently in
use around the world. The creation of the web, as well as signi ﬁcant recent

improvementsin speech recognition and synthesis, will lea d to many more ap-
plications.
Bibliographicaland HistoricalNotes
Research in the various subareas of speech and language proc essing is spread across
a wide number of conferenceproceedingsand journals. The co nferencesand journals
most centrallyconcernedwith naturallanguageprocessing andcomputationallinguis-
tics are associated with the Association for Computational Linguistics (ACL), its Eu-

ropean counterpart (EACL), and the International Conferen ce on Computational Lin-
guistics (COLING). The annual proceedings of ACL, NAACL, an d EACL, and the
biennial COLING conference are the primary forums for work i n this area. Related
conferencesinclude various proceedingsof ACL Special Int erest Groups (SIGs) such
as the Conferenceon Natural LanguageLearning(CoNLL), as w ell as the conference
onEmpiricalMethodsin NaturalLanguageProcessing(EMNLP ).
Research on speech recognition, understanding, and synthe sis is presented at the

annual INTERSPEECH conference, which is called the Interna tional Conference on

DRAFT16 Chapter 1. Introduction
SpokenLanguageProcessing(ICSLP) andthe EuropeanConfer enceonSpeechCom-
municationandTechnology(EUROSPEECH)inalternatingyea rs,ortheannualIEEE
InternationalConferenceonAcoustics,Speech,andSignal Processing(IEEEICASSP).
Spokenlanguagedialogueresearchispresentedattheseora tworkshopslikeSIGDial.
Journalsinclude ComputationalLinguistics ,NaturalLanguageEngineering ,Speech
Communication ,Computer Speech and Language , theIEEE Transactions on Audio,
Speech & Language Processing and theACM Transactions on Speech and Language
Processing .
Work on language processing from an Artiﬁcial Intelligence perspective can be
found in the annual meetings of the American Association for Artiﬁcial Intelligence
(AAAI), as well as the biennial International Joint Confere nce on Artiﬁcial Intelli-
gence (IJCAI) meetings. Artiﬁcial intelligence journalst hat periodicallyfeature work
on speech and language processing include Machine Learning ,Journal of Machine

LearningResearch ,andtheJournalofArtiﬁcialIntelligenceResearch .
There are a fair number of textbooksavailable covering vari ous aspects of speech
andlanguageprocessing. ManningandSch¨ utze(1999)( FoundationsofStatisticalLan-
guage Processing ) focuses on statistical models of tagging, parsing, disamb iguation,
collocations, and other areas. Charniak (1993) ( Statistical Language Learning ) is an
accessible, thougholder and less-extensive, introductio nto similar material. Manning

etal.(2008)focusesoninformationretrieval,textclassi ﬁcation,andclustering. NLTK,
the Natural Language Toolkit (Bird and Loper, 2004), is a sui te of Python modules
and data for natural language processing, together with a Na tural Language Process-
ing book based on the NLTK suite. Allen (1995) ( Natural Language Understanding )
provides extensive coverage of language processing from th e AI perspective. Gazdar
and Mellish (1989) ( Natural Language Processing in Lisp/Prolog ) covers especially

automata, parsing, features, and uniﬁcation and is availab le free online. Pereira and
Shieber(1987)givesaProlog-basedintroductiontoparsin gandinterpretation. Russell
andNorvig(2002)is an introductionto artiﬁcial intellige ncethat includeschapterson
natural language processing. Partee et al. (1990) has a very broad coverageof mathe-
matical linguistics. A historically signiﬁcant collectio n of foundational papers can be
foundin Groszet al.(1986)( Readingsin NaturalLanguageProcessing ).

Chapter2
Regular Expressions andAutomata
User1: Men are all alike.
ELIZA1: IN WHATWAY
User2: They’re always bugging us about something or other.
ELIZA2: CANYOUTHINKOF ASPECIFICEXAMPLE
User3: Well, my boyfriend made me come here.
ELIZA3: YOURBOYFRIENDMADEYOUCOMEHERE
User4: He says I’mdepressed muchof the time.
ELIZA4: I AM SORRYTOHEARYOUAREDEPRESSED.
Weizenbaum(1966)
Imagine that you have become a passionate fan of woodchucks a nd have recently
learned that groundhogs and woodchucks are different names for the very same an-
imal. Since you are writing a term paper on woodchucks, you no w need to search

throughyourpaperforeverytimetheterm woodchuck occursandreplace woodchucks
withwoodchucks(groundhogs) . Butyoualsoneedtoreplacesingular woodchuck with
woodchuck(groundhog) . Insteadofhavingtodothissearchtwice,youwouldprefert o
performa single commandfor something like woodchuck with an optionalﬁnal s . Or
perhapsyoumightwanttosearchforallthepricesinsomedoc ument;youmightwant
to see all strings that looklike $199or$25or$24.99in order to automatically extract
atableofprices. Inthischapterweintroducethe regularexpression ,thestandardno-

tation for characterizingtext sequences. The regular expr ession is used for specifying
textstringsin allsortsoftextprocessingandinformation extractionapplications.
Afterwe havedeﬁnedregularexpressions,we showhowtheyca nbeimplemented
via theﬁnite-state automaton . The ﬁnite-state automaton is not only the mathemati-
cal device used to implement regular expressions, but also o ne of the most signiﬁcant
tools of computational linguistics. Variations of automat a such as ﬁnite-state trans-

ducers, Hidden Markov Models, and N-gram grammars are important components of
applicationsthat we will introduce in later chapters, incl uding speech recognitionand
synthesis,machinetranslation,spell-checking,andinfo rmation-extraction.
2.1 RegularExpressions
SIR ANDREW: Her C’s,her U’s and her T’s: why that?
Shakespeare, Twelfth Night
Oneoftheunsungsuccessesinstandardizationincomputers ciencehasbeenthe regu-
larexpression (RE),alanguageforspecifyingtextsearchstrings. Theregula rexpres-Regularexpression

sionlanguagesusedforsearchingtextsinUNIX(vi,Perl,Em acs,grep)andMicrosoft

DRAFT18 Chapter 2. RegularExpressionsandAutomata
Word are almost identical, and many RE features exist in the v arious Web search en-
gines. Besidesthispracticaluse,theregularexpressioni sanimportanttheoreticaltool
throughoutcomputerscience andlinguistics.
A regularexpression(ﬁrst developedby Kleene (1956)but se e the History section
for more details) is a formula in a special language that is us ed for specifying simple
classes of strings. A string is a sequence of symbols; for the purpose of most tex t- Strings
based search techniques, a string is any sequence of alphanu meric characters (letters,
numbers,spaces, tabs, andpunctuation). For these purpose sa space is just a character
likeanyother,andwe representit withthesymbol .
Formally, a regular expression is an algebraic notation for characterizing a set of
strings. Thustheycanbeusedtospecifysearchstringsaswe llastodeﬁnealanguagein
aformalway. Wewillbeginbytalkingaboutregularexpressi onsasawayofspecifying

searches in texts, and proceed to other uses. Section 2.3 sho ws that the use of just
three regular expression operators is sufﬁcient to charact erize strings, but we use the
more convenient and commonly-used regular expression synt ax of the Perl language
throughoutthissection. Sincecommontext-processingpro gramsagreeonmost ofthe
syntax of regular expressions, most of what we say extends to all UNIX, Microsoft
Word, and WordPerfect regular expressions. Appendix A show s the few areas where
theseprogramsdifferfromthePerl syntax.

Regularexpressionsearchrequiresa patternthatwewanttosearchfor,anda cor-
pusoftextstosearchthrough. Aregularexpressionsearchfunc tionwillsearchthrough Corpus
the corpus returning all texts that contain the pattern. In a n information retrieval (IR)
systemsuchasaWebsearchengine,thetextsmightbeentired ocumentsorWebpages.
Inaword-processor,thetextsmightbeindividualwords,or linesofadocument. Inthe
restofthischapter,wewillusethislastparadigm. Thuswhe nwegiveasearchpattern,
wewillassumethatthesearchenginereturnsthe lineofthedocument returned. Thisis

what the UNIX grepcommand does. We will underline the exact part of the pattern
that matchesthe regularexpression. A search can be designe dto returnall matchesto
a regularexpressionoronlytheﬁrst match. We will showonly theﬁrst match.
2.1.1 BasicRegularExpressionPatterns
The simplest kind of regular expression is a sequence of simp le characters. For ex-
ample, to search for woodchuck , we type /woodchuck/ . So the regular expression
/Buttercup/ matches any string containing the substring Buttercup , for example

the lineI’m called little Buttercup ) (recall that we are assuming a search application
thatreturnsentirelines). Fromhereonwewillputslashesa roundeachregularexpres-
sion to make it clear what is a regular expression and what is a pattern. We use the
slash since this is the notation used by Perl, but the slashes arenotpart of the regular
expressions.
The search string can consist of a single character (like /!/) or a sequence of
characters(like /urgl/);Theﬁrstinstanceofeachmatchtotheregularexpressionis

underlined below (although a given application might choos e to return more than just
theﬁrst instance):

DRAFTSection2.1. RegularExpressions 19
RE Example PatternsMatched
/woodchucks/ “interestinglinkstowoodchucks andlemurs”
/a/ “MaryAnnstoppedbyMona’s”
/Claire says,/ “ “Dagmar,mygiftplease,”Claire says, ”
/DOROTHY/ “SURRENDERDOROTHY ”
/!/ “You’veleft theburglarbehindagain! ” saidNori
Regular expressions are case sensitive ; lowercase /s/is distinct from uppercase
/S/(/s/matchesa lower case sbut notan uppercase S).Thismeansthat the pattern
/woodchucks/ will not match the string Woodchucks . We can solve this problem
with the use of the square braces [and]. The string of characters inside the braces
specify a disjunction of characters to match. For example Fig. 2.1 shows that the
pattern /[wW]/ matchespatternscontainingeither worW.
RE Match Example Patterns
/[wW]oodchuck/ Woodchuckorwoodchuck “Woodchuck ”
/[abc]/ ‘a’,‘b’,or‘c’ “Inuomini,insolda ti”
/[1234567890]/ anydigit “plentyof7 to5”
Figure2.1 The use of the brackets []tospecifya disjunction of characters.

Theregularexpression /[1234567890]/ speciﬁedanysingledigit. Whileclasses
ofcharacterslikedigitsorlettersareimportantbuilding blocksinexpressions,theycan
getawkward(e.g.,it’sinconvenienttospecify
/[ABCDEFGHIJKLMNOPQRSTUVWXYZ]/
to mean“anycapital letter”). Inthese casesthe bracketsca nbeused withthe dash( -)
tospecifyanyonecharacterina range. Thepattern /[2-5]/ speciﬁesanyoneofthe Range
characters 2,3,4,or5. Thepattern /[b-g]/ speciﬁesoneofthecharacters b,c,d,e,
f, org. Someotherexamples:
RE Match Example PatternsMatched

/[A-Z]/ anuppercaseletter “we shouldcallit ‘D renchedBlossoms’”
/[a-z]/ a lowercaseletter “mybeanswereimpatienttobe hoed!”
/[0-9]/ a singledigit “Chapter1 : DowntheRabbitHole”
Figure2.2 The use of the brackets []plus the dash -tospecify a range.
The square braces can also be used to specify what a single cha ractercannotbe,
by use of the caret ˆ. If the caret ˆis the ﬁrst symbol after the open square brace [,
the resultingpattern is negated. For example,the pattern /[ˆa]/ matchesany single

character(includingspecialcharacters)except a. Thisisonlytruewhenthecaretisthe
ﬁrst symbol after the open square brace. If it occurs anywher e else, it usually stands
foracaret;Fig. 2.3showssomeexamples.
The use of square braces solves our capitalization problem f orwoodchucks . But
we still haven’t answered our original question; how do we sp ecify both woodchuck
andwoodchucks ? We can’tusethesquarebrackets,becausewhiletheyallowu stosay

“s or S”, they don’t allow us to say “s or nothing”. For this we u se the question-mark
/?/,whichmeans“the precedingcharacterornothing”,asshown inFig. 2.4.

DRAFT20 Chapter 2. RegularExpressionsandAutomata
RE Match(single characters) ExamplePatternsMatched
[ˆA-Z] notanuppercaseletter “Oyfnpripetchik”
[ˆSs] neither‘S’ nor‘s’ “Ihavenoexquisitereasonfor’t”
[ˆ\.] nota period “ourresidentDjinn”
[eˆ] either‘e’or‘ ˆ’ “lookupˆ now”
aˆb thepattern‘ aˆb’ “lookupaˆb now”
Figure2.3 Usesofthecaret ˆfornegationorjusttomean ˆ. We’lldiscussbelow theneed
toescape the periodby abackslash.
RE Match ExamplePatternsMatched
woodchucks? woodchuckorwoodchucks “woodchuck ”
colou?r colororcolour “colour”
Figure2.4 The question-mark ?marks optionality of the previous expression.
We canthinkof the question-markasmeaning“zeroor oneinst ancesofthe previ-
ouscharacter”. Thatis, it’s a way ofspecifyinghow manyof s omethingthat we want.
So far we haven’t needed to specify that we want more than one o f something. But
sometimes we need regular expressions that allow repetitio ns. For example, consider
thelanguageof(certain)sheep,whichconsistsofstringst hatlooklike thefollowing:
baa!
baaa!
baaaa!

baa!
baaa!
baaaa!
baaaaa!
...
Thislanguageconsistsofstringswitha b,followedbyatleast two as, followedby
anexclamationpoint. Theset ofoperatorsthat allowusto sa ythingslike “somenum-
ber ofas” are based on the asterisk or *, commonly called the Kleene * (pronounced Kleene*
“cleany star”). The Kleene star means “zero or more occurren ces of the immediately
previouscharacterorregularexpression”. So /a*/means“anystringofzeroormore
as”. This will match aoraaaaaabut it will also match Off Minor , since the string Off

baaa!
baaaa!
baaaaa!
...
Minorhas zeroas. So the regular expression for matching one or more ais/aa*/,
meaning one afollowed by zero or more as. More complex patterns can also be re-
peated. So /[ab] */means “zero or more as orbs” (not “zero or more right square
braces”). Thiswill matchstringslike aaaaorabababorbbbb.
Wenowknowenoughtospecifypartofourregularexpressionf orprices: multiple
digits. Recall that theregularexpressionforanindividua ldigitwas /[0-9]/ . So the

regularexpression for an integer (a string of digits) is /[0-9][0-9] */. (Why isn’t
it just/[0-9] */?)
Sometimesit’sannoyingtohavetowritetheregularexpress ionfordigitstwice,so
thereisashorterwaytospecify“atleastone”ofsomecharac ter. Thisisthe Kleene+, Kleene+
whichmeans“oneormoreofthepreviouscharacter”. Thusthe expression /[0-9]+/
isthenormalwaytospecify“asequenceofdigits”. Thereare thustwowaystospecify
thesheeplanguage: /baaa *!/or/baa+!/ .
One very important special character is the period ( /./), awildcard expression

thatmatchesanysingle character( excepta carriagereturn):

DRAFTSection2.1. RegularExpressions 21
RE Match ExamplePatterns
/beg.n/ anycharacterbetween begandnbegin, beg’n, begun
Figure2.5 The use of the period .tospecify anycharacter.
The wildcard is often used together with the Kleene star to me an “any string of
characters”. Forexamplesupposewe wantto ﬁndanyline inwh ich aparticularword,
for example aardvark, appearstwice. We can specify this with the regular express ion
/aardvark. *aardvark/ .
Anchorare special characters that anchor regular expressions to p articular places Anchor
in a string. The most commonanchorsare the caret ˆand the dollar-sign $. The caret
ˆmatches the start of a line. The pattern /ˆThe/ matches the word Theonly at the
start of a line. Thus there are three uses of the caret ˆ: to match the start of a line, as
a negation inside of square brackets, and just to mean a caret . (What are the contexts
that allow Perl to know which functiona given caret is suppos edto have?) The dollar

sign$matches the end of a line. So the pattern $is a useful pattern for matching
a space at the end of a line, and /ˆThe dog\.$/ matches a line that contains only
the phrase The dog. (We have to use the backslash here since we want the .to mean
“period”andnotthewildcard.)
Therearealsotwootheranchors: \bmatchesawordboundary,while \Bmatches
a non-boundary. Thus /\bthe\b/ matches the word thebut not the word other.
Moretechnically,Perl deﬁnesa wordasanysequenceofdigit s,underscoresorletters;
thisisbasedonthedeﬁnitionof“words”inprogramminglang uageslikePerlorC.For

example, /\b99\b/ will match the string 99inThere are 99 bottles of beer on the
wall(because 99 follows a space) but not 99inThere are 299 bottles of beer on the
wall(since99followsanumber). Butitwillmatch 99in$99(since99followsadollar
sign($),whichisnota digit,underscore,orletter).
2.1.2 Disjunction, Grouping, andPrecedence
Suppose we need to search for texts about pets; perhapswe are particularlyinterested
in cats and dogs. In such a case we might want to search for eith er the string cator

the string dog. Since we can’t use the square-bracketsto search for “cat or dog” (why
not?) weneedanewoperator,the disjunction operator,alsocalledthe pipesymbol |. Disjunction
Thepattern /cat|dog/ matcheseitherthestring catorthestring dog.
Sometimes we need to use this disjunction operator in the mid st of a larger se-
quence. For example, suppose I want to search for informatio n about pet ﬁsh for my
cousin David. How can I specify both guppyandguppies? We cannot simply say

/guppy|ies/ , because that would match only the strings guppyandies. This is
because sequences like guppytakeprecedence over the disjunction operator |. In Precedence
orderto make the disjunction operatorapply onlyto a speciﬁ c pattern,we need to use
the parenthesisoperators (and). Enclosinga patternin parenthesesmakesit act like
a single character for the purposes of neighboring operator s like the pipe |and the
Kleene*. So the pattern /gupp(y|ies)/ wouldspecify that we meant the disjunc-
tiononlyto applytothesufﬁxes yandies.

The parenthesis operator (is also useful when we are using counters like the

DRAFT22 Chapter 2. RegularExpressionsandAutomata
Kleene*. Unlike the |operator, the Kleene *operator applies by default only to a
single character, not a whole sequence. Suppose we want to ma tch repeated instances
of a string. Perhaps we have a line that has column labels of th e formColumn 1 Col-
umn2 Column3 . Theexpression /Column [0-9]+ */willnotmatchanynumber
of columns; instead, it will match a single columnfollowedb y any numberof spaces!
The star here applies only to the space that precedes it, not the whole sequence.
With the parentheses, we could write the expression /(Column [0-9]+ *)*/to
matchthe word Column,followedbya numberandoptionalspaces, the wholepattern
repeatedanynumberoftimes.
Thisideathatoneoperatormaytakeprecedenceoveranother ,requiringustosome-
times use parenthesesto specify what we mean, is formalized by theoperator prece-
dence hierarchy for regular expressions. The following table gives the orde r of REOperator
precedence

operatorprecedence,fromhighestprecedencetolowest pre cedence:
Parenthesis ()
Counters *+ ? {}
Sequencesandanchors the ˆmy end$
Disjunction |
Thus, because counters have a higher precedence than sequen ces,
/the*/matchestheeeeebutnotthethe. Becausesequenceshaveahigherprecedence
thandisjunction, /the|any/ matchestheoranybutnottheny.
Patterns can be ambiguous in another way. Consider the expre ssion/[a-z] */
when matching against the text once upon a time . Since /[a-z] */matches zero or
moreletters,thisexpressioncouldmatchnothing,orjustt heﬁrstletter o,oron,oronc,

oronce. In these cases regular expressions always match the largeststring they can;
we saythatpatternsare greedy,expandingto coverasmuchofa stringastheycan. Greedy
2.1.3 A SimpleExample
SupposewewantedtowriteaREtoﬁndcasesoftheEnglisharti clethe. Asimple(but
incorrect)patternmightbe:
/the/
One problem is that this pattern will miss the word when it beg ins a sentence and
henceiscapitalized(i.e., The). Thismightleadustothefollowingpattern:
/[tT]he/

/the/
/[tT]he/
But we will still incorrectly return texts with theembedded in other words (e.g.,
otherortheology). Soweneedtospecifythatwewantinstanceswithawordboun dary
onbothsides:
/\b[tT]he\b/
Supposewe wanted to do this without the use of /\b/. We might want this since
/\b/won’ttreat underscoresandnumbersas wordboundaries;but we mightwant to
ﬁndthein some context where it might also have underlines or number snearby ( the

DRAFTSection2.1. RegularExpressions 23
orthe25). We need to specify that we want instances in which there are no alphabetic
lettersoneitherside ofthe the:
/[ˆa-zA-Z][tT]he[ˆa-zA-Z]/
But there is still one more problem with this pattern: it won’ t ﬁnd the word the
when it begins a line. This is because the regular expression [ˆa-zA-Z] , which we
used to avoid embedded thes, implies that there must be some single (although non-
alphabetic) character before the the. We can avoid this by specifying that before the
thewerequire eitherthebeginning-of-lineoranon-alphabeticcharacter,andt hesame
at theendofthe line:
/(ˆ|[ˆa-zA-Z])[tT]he([ˆa-zA-Z]|$)/
The process we just went through was based on ﬁxing two kinds o f errors:false
positives, strings that we incorrectly matched like otherorthere, andfalse negatives , False positive
False negative strings that we incorrectly missed, like The. Addressing these two kinds of errors
comes up again and again in implementing speech and language processing systems.

Reducingtheerrorrateforanapplicationthusinvolvestwo antagonisticefforts:
•Increasing accuracy (minimizingfalse positives)
•Increasing coverage (minimizingfalse negatives).
2.1.4 A MoreComplex Example
Let’stryoutamoresigniﬁcantexampleofthepowerofREs. Su pposewewanttobuild
anapplicationtohelpauserbuyacomputerontheWeb. Theuse rmightwant“anyPC
with morethan500MHz and32 Gbof disk spaceforless than $100 0”. In orderto do
thiskindofretrievalwe will ﬁrst needto beableto lookfore xpressionslike 500MHz

or32GborCompaqorMacor$999.99. Intherestofthissectionwe’llworkoutsome
simpleregularexpressionsforthistask.
First, let’s complete our regular expression for prices. He re’s a regular expression
foradollarsignfollowedbyastringofdigits. NotethatPer lissmartenoughtorealize
that$heredoesn’tmeanend-of-line;howmightit knowthat?
/$[0-9]+/
Now we just need to deal with fractions of dollars. We’ll add a decimal point and
twodigitsafterwards:
/$[0-9]+\.[0-9][0-9]/

/$[0-9]+/
Thispatternonlyallows $199.99butnot$199. Weneedtomakethecentsoptional,
andmakesurewe’reat awordboundary:
/\b$[0-9]+(\.[0-9][0-9])?\b/
How aboutspeciﬁcationsforprocessorspeed (in megahertz= MHz or gigahertz=
GHz)? Here’sa patternforthat:
/\b[0-9]+ *(MHz|[Mm]egahertz|GHz|[Gg]igahertz)\b/

DRAFT24 Chapter 2. RegularExpressionsandAutomata
Notethatweuse /*/tomean“zeroormorespaces”,sincetheremightalwaysbe
extraspaceslyingaround. Dealingwithdiskspaceormemory size(inGB=gigabytes),
we need to allow for optional fractions again ( 5.5 GB). Note the use of ?for making
theﬁnal soptional:
/\b[0-9]+(\.[0-9]+)? *(Gb|[Gg]igabytes?)\b/
Finally,wemightwantsomesimple patternstospecifyopera tingsystems:
/\b(Windows *(Vista|XP)?)\b/
/\b(Mac|Macintosh|Apple|OS X)\b/
2.1.5 Advanced Operators
REExpansion Match Examples
\d [0-9] anydigit Partyof5
\D [ˆ0-9] anynon-digit Bluemoon
\w [a-zA-Z0-9_] anyalphanumeric/underscore Daiyu
\W [ˆ\w ] a non-alphanumeric !!!!
\s [\r\t\n\f]whitespace(space,tab)
\S [ˆ\s ] Non-whitespace inConcord
Figure2.6 Aliasesfor common sets of characters.
There are also some useful advancedregular expression oper ators. Fig. 2.6 shows
some aliases for common ranges, which can be used mainly to sa ve typing. Besides
theKleene*andKleene+,wecanalsouseexplicitnumbersasc ounters,byenclosing

them in curly brackets. The regular expression /{3}/means “exactly 3 occurrences
of the previouscharacter or expression”. So /a\.{24}z/ will match afollowed by
24dotsfollowedby z(butnotafollowedby23or25dotsfollowedbya z).
Arangeofnumberscanalsobespeciﬁed;so /{n,m}/ speciﬁesfromntomoccur-
rencesofthepreviouscharorexpression,while /{n,}/ meansatleast noccurrences
ofthepreviousexpression. REsforcountingaresummarized inFig.2.7.
RE Match
* zeroormoreoccurrencesofthepreviouscharorexpression

RE Match
+ oneormoreoccurrencesofthepreviouscharorexpression
? exactlyzerooroneoccurrenceofthepreviouscharorexpres sion
{n} noccurrencesofthepreviouscharorexpression
{n,m} fromntomoccurrencesofthepreviouscharorexpression
{n,} atleastnoccurrencesofthepreviouscharorexpression
Figure2.7 Regular expression operators for counting.
Finally, certain special characters are referred to by spec ial notation based on the
backslash ( \). The most common of these are the newlinecharacter \nand thetab Newline

character \t. To refer to characters that are special themselves (like .,*,[, and\),
precedethemwitha backslash,(i.e., /\./,/\*/,/\[/,and/\\/).

DRAFTSection2.1. RegularExpressions 25
\* anasterisk “*” “K*A*P*L*A*N”
\. a period“.” “Dr.Livingston,I presume”
\? a questionmark “Whydon’ttheycomeandlendahand? ”
\n a newline
\t a tab
Figure2.8 Some characters that need tobe backslashed.
2.1.6 RegularExpression Substitution, Memory,and ELIZA
Animportantuseofregularexpressionsisin substitutions . Forexample,thePerlsub- Substitution
stitutionoperator s/regexp1/pattern/ allowsastringcharacterizedbyaregular
expressiontobereplacedbyanotherstring:
s/colour/color/
Itisoftenusefultobeabletorefertoaparticularsubparto fthestringmatchingthe
ﬁrstpattern. Forexample,supposewewantedtoputanglebra cketsaroundallintegers
inatext,changinge.g., the35boxes tothe<35>boxes. We’dlikeawaytoreferback
to the integer we’ve found so that we can easily add the bracke ts. To do this, we put
parentheses (and)around the ﬁrst pattern, and use the numberoperator \1in the
secondpatterntoreferback. Here’showit looks:
s/([0-9]+)/<\1>/

The parenthesis and number operators can also be used to spec ify that a certain
stringorexpressionmustoccurtwiceinthetext. Forexampl e,supposewearelooking
forthepattern“theXertheywere,theXertheywillbe”,wher ewewanttoconstrainthe
twoX’stobethesamestring. Wedothisbysurroundingtheﬁrs tXwiththeparenthesis
operator,andreplacingthesecondXwith thenumberoperato r\1,asfollows:
/the (. *)er they were, the \1er they will be/
Here the \1will be replacedby whateverstring matchedthe ﬁrst item in p arentheses.

Sothiswillmatch Thebiggertheywere,thebiggertheywillbe butnotThebiggerthey
were, thefastertheywill be .
The number operator can be used with other numbers: if you mat ch two different
setsofparenthesis, \2meanswhatevermatchedthe secondset. Forexample
/the (. *)er they (. *), the \1er they \2/
will match The bigger they were, the bigger they were but notThe bigger they were,
thebiggertheywill be . Thesenumberedmemoriesarecalled registers (e.g. register1, Register

register 2, register 3, etc). This memoryfeature is not part of everyregularexpression
languageandisoftenconsideredan“extended”featureofre gularexpressions.
Substitutions using memory are useful in implementing simp le natural-language
understanding programs like ELIZA (Weizenbaum, 1966). Rec all that ELIZA simu-
latesa Rogerianpsychologistbycarryingonconversations likethefollowing:

DRAFT26 Chapter 2. RegularExpressionsandAutomata
User1:Menareall alike.
ELIZA 1:INWHAT WAY
User2:They’realwaysbuggingusaboutsomethingorother.
ELIZA 2:CAN YOUTHINKOF ASPECIFICEXAMPLE
User3:Well, myboyfriendmademecomehere.
ELIZA 3:YOURBOYFRIEND MADEYOUCOME HERE
User4:HesaysI’mdepressedmuchofthetime.
ELIZA 4:IAM SORRY TOHEARYOUAREDEPRESSED.
Elizaworksbyhavingacascadeofregularexpressionsubsti tutionsthateachmatched
some part of the input lines and changed them. The ﬁrst substi tutions changed all in-
stancesof mytoYOUR,andI’mtoYOU ARE ,andsoon. Thenextsetofsubstitutions,
matchedandreplacedotherpatternsintheinput. Hereare so meexamples:
s/.*YOU ARE (depressed|sad) . */I AM SORRY TO HEAR YOU ARE \1/
s/.*YOU ARE (depressed|sad) . */WHY DO YOU THINK YOU ARE \1/
s/.*all .*/IN WHAT WAY/
s/.*always . */CAN YOU THINK OF A SPECIFIC EXAMPLE/
Sincemultiplesubstitutionscanapplytoagiveninput,sub stitutionsareassigneda
rankandappliedinorder. CreatingpatternsisthetopicofE xercise2.
2.2 Finite-StateAutomata

Theregularexpressionismorethanjustaconvenientmetala nguagefortextsearching.
First, a regular expression is one way of describing a ﬁnite-state automaton (FSA).Finite-stateautomaton
FSAFinite-stateautomataarethetheoreticalfoundationofag ooddealofthecomputational
work we will describe in this book. Any regular expression ca n be implemented as a
ﬁnite-state automaton (except regular expressions that us e the memory feature; more
onthislater). Symmetrically,anyﬁnite-stateautomatonc anbedescribedwitharegular

expression. Second,aregularexpressionisonewayofchara cterizingaparticularkind
of formal language called a regular language . Both regular expressions and ﬁnite- Regularlanguage
state automata can be used to describe regular languages. A t hird equivalent method
of characterizing the regular languages, the regular grammar , will be introduced in
Ch.15. Therelationamongthesetheoreticalconstructions issketchedin Fig.2.9.
This section will begin by introducing ﬁnite-state automat a for some of the regu-

lar expressions from the last section, and then suggest how t he mapping from regular
expressions to automata proceeds in general. Although we be gin with their use for
implementingregularexpressions,FSAshaveawidevariety ofotherusesthatwewill
explorein thischapterandthenext.
2.2.1 UsinganFSA toRecognize Sheeptalk
After a while, withthe parrot’s help, the Doctor got to learn the language of the animals
sowellthat he could talk tothem himself and understand ever ything they said.

DRAFTSection2.2. Finite-StateAutomata 27
regular 
grammarsfinite
automataregular
expressions regular
languages
Figure2.9 Finiteautomata,regularexpressions,andregulargrammar sareallequivalentways
of describing regular languages.
Hugh Lofting, TheStory of Doctor Dolittle
Let’s begin with the “sheep language” we discussed previous ly. Recall that we
deﬁnedthesheeplanguageasanystringfromthefollowing(i nﬁnite)set:
baa!
baaa!
baaaa!
baaaaa!
...
q3a
q0q4 q1q2b a a !
Figure2.10 A ﬁnite-stateautomaton for talkingsheep.
The regular expression for this kind of “sheeptalk” is /baa+!/ . Fig. 2.10 shows
anautomaton for modeling this regular expression. The automaton (i.e., machine, Automaton
also called ﬁnite automaton ,ﬁnite-state automaton , orFSA) recognizes a set of
strings,inthiscasethestringscharacterizingsheeptalk ,inthesamewaythataregular
expressiondoes. Werepresenttheautomatonasadirectedgr aph: aﬁnitesetofvertices
(alsocallednodes),togetherwithasetofdirectedlinksbe tweenpairsofverticescalled

arcs. We’llrepresentverticeswithcirclesandarcswithar rows. Theautomatonhasﬁve
statess, which are represented by nodesin the graph. State 0 is the start state . In our State
Start state examplesstate0willgenerallybethestartstate;tomarkan otherstate asthestartstate
we can add an incomingarrowto the start state. State 4 is the ﬁnal state oraccepting
state, which we represent by the double circle. It also has four transitions , which we
representbyarcsin thegraph.
TheFSAcanbeusedforrecognizing(wealsosay accepting )stringsinthefollow-

ing way. First, think of the input as being written on a long ta pe broken up into cells,
withonesymbolwrittenin eachcell ofthetape,asinFig. 2.1 1.
Themachinestartsin thestart state( q0), anditeratesthefollowingprocess: Check
the next letter of the input. If it matches the symbol on an arc leaving the current
state, then cross that arc, move to the next state, and also ad vance one symbol in the

DRAFT28 Chapter 2. RegularExpressionsandAutomata
aba!bq0
Figure2.11 A tape withcells.
input. If we are in the accepting state ( q4) when we run out of input, the machine has
successfullyrecognizedaninstanceofsheeptalk. Ifthema chinenevergetsto theﬁnal
state,eitherbecauseitrunsoutofinput,oritgetssomeinp utthatdoesn’tmatchanarc
(as in Fig. 2.11), or if it just happens to get stuck in some non -ﬁnal state, we say the
machinerejectsorfailstoacceptaninput. Rejecting
We can also representan automatonwitha state-transitiontable . As in thegraphState-transitiontable
Input
Stateba!
01/ 0/ 0
1/ 02/ 0
2/ 03/ 0
3/ 034
4:/ 0/ 0/ 0notation, the state-transitiontable representsthe start state, the ac-
cepting states, and what transitions leave each state with w hich
symbols. On the right is the state-transition table for the F SA of
Fig. 2.10. We’ve markedstate 4 with a colon to indicate that i t’s a
ﬁnal state (youcan haveas manyﬁnal statesas youwant), andt he

Input
Stateba!
01/ 0/ 0
1/ 02/ 0
2/ 03/ 0
3/ 034
/ 0 indicates an illegal or missing transition. We can read th e ﬁrst
row as “if we’re in state 0 and we see the input bwe must go to
state 1. Ifwe’reinstate 0andwesee theinput aor!,we fail”.
Moreformally,a ﬁniteautomatonis deﬁnedbythefollowingﬁ veparameters:
Q=q0q1q2...qN−1aﬁnite set of Nstates
Σ aﬁniteinput alphabet ofsymbols
q0 thestartstate
F theset of ﬁnalstates ,F⊆Q
δ(q,i) thetransition function or transition matrix be-
tween states. Given a state q∈Qand an input
symboli∈Σ,δ(q,i)returns a new state q′∈Q.

δis thusarelationfrom Q×ΣtoQ;
For the sheeptalk automaton in Fig. 2.10, Q={q0,q1,q2,q3,q4},Σ={a,b,!},
F={q4},andδ(q,i)isdeﬁnedbythetransitiontableabove.
Fig. 2.12 presents an algorithm for recognizing a string usi ng a state-transition
table. The algorithmis called D-RECOGNIZE for“deterministic recognizer”. A deter-
ministicalgorithmis one that has no choice points; the algorithmalw ays knowswhat Deterministic
to do for any input. The next section will introduce non-dete rministic automata that
mustmakedecisionsaboutwhichstatestomoveto.

D-RECOGNIZE takesasinputatapeandanautomaton. Itreturns acceptifthestring
it is pointing to on the tape is accepted by the automaton, and rejectotherwise. Note
that since D-RECOGNIZE assumes it is alreadypointingat the string to be checked,it s
task isonlya subpartofthe generalproblemthat we oftenuse regularexpressionsfor,
ﬁndinga stringinacorpus. (Thegeneralproblemis leftasEx ercise9forthereader.)
D-RECOGNIZE beginsbysettingthevariable indextothebeginningofthetape,and
current-state tothemachine’sinitialstate. D-RECOGNIZE thenentersaloopthatdrives

DRAFTSection2.2. Finite-StateAutomata 29
function D-RECOGNIZE (tape,machine )returnsaccept or reject
index←Beginning oftape
current-state←Initial stateof machine
loop
ifEndof input has been reached then
ifcurrent-state is anaccept state then
returnaccept
else
returnreject
elsiftransition-table[current-state,tape[index]] isemptythen
else
current-state←transition-table[current-state,tape[index]]
index←index+ 1
end
Figure2.12 An algorithm for deterministic recognition of FSAs. This al gorithm returns ac-
ceptif the entire string it is pointing at is in the language deﬁne d by the FSA, and rejectif the
stringisnot inthe language.
the algorithm. It ﬁrst checkswhetherit has reachedthe end o f its input. If so, it either
acceptstheinput(ifthecurrentstate isanacceptstate) or rejectstheinput(ifnot).
Ifthereisinputleftonthetape, D-RECOGNIZE looksatthetransitiontabletodecide
which state to move to. The variable current-state indicates which row of the table to

consult, while the current symbol on the tape indicates whic h column of the table to
consult. The resulting transition-table cell is used to upd ate the variable current-state
andindexis incremented to move forward on the tape. If the transition -table cell is
emptythenthemachinehasnowhereto goandmustrejectthe in put.
Fig. 2.13 traces the execution of this algorithm on the sheep language FSA given
thesampleinputstring baaa!.
baaa!q0q1q2q3q3q4
Figure2.13 Tracing the executionof FSA#1 onsome sheeptalk.

Beforeexaminingthebeginningofthetape,themachineisin stateq0. Findinga b
oninputtape,itchangestostate q1asindicatedbythecontentsof transition-table [q0,b]
on page 28. It then ﬁnds an aand switches to state q2, another aputs it in state q3, a
thirdaleavesitinstate q3,whereitreadsthe“!”,andswitchestostate q4. Sincethereis
nomoreinput,theEndof input conditionatthebeginningoftheloopissatisﬁed
for the ﬁrst time and the machinehalts in q4. Stateq4is an accepting state, and so the

machinehasacceptedthestring baaa!asa sentencein thesheeplanguage.
Thealgorithmwillfailwheneverthereisnolegaltransitio nforagivencombination

DRAFT30 Chapter 2. RegularExpressionsandAutomata
of state and input. The input abcwill fail to be recognized since there is no legal
transition out of state q0on the input a, (i.e., this entry of the transition table on page
28 has a / 0). Even if the automaton had allowed an initial ait would have certainly
failedonc,sincecisn’teveninthesheeptalkalphabet! We canthinkofthese“e mpty”
elements in the table as if they all pointed at one “empty” sta te, which we might call
thefail state orsink state . In a sense then, we could view any machine with empty Fail state
transitions as ifwe had augmented it with a fail state, and drawn in all the extr a arcs,
so we always had somewhere to go from any state on any possible input. Just for
completeness,Fig.2.14showsthe FSA fromFig.2.10withthe failstateqFﬁlled in.
q3a
qfailb!!! !b
baab
Figure2.14 Adding a failstate toFig.2.10.
2.2.2 FormalLanguages
We can use the same graph in Fig. 2.10 as an automaton for GENERATING sheeptalk.

q3a
baab
If we do, we would say that the automaton starts at state q0, and crosses arcs to new
states,printingoutthesymbolsthatlabeleacharcitfollo ws. Whentheautomatongets
to the ﬁnal state it stops. Notice that at state 3, the automat on has to chose between
printingouta !andgoingto state 4, orprintingout an aandreturningtostate 3. Let’s
say for now that we don’t care how the machine makes this decis ion; maybe it ﬂips a
coin. For now, we don’t care which exact string of sheeptalk w e generate, as long as
it’sa stringcapturedbytheregularexpressionforsheepta lkabove.

Formal Language: A model which can both generate and recognize all and
onlythestringsofaformallanguageactsasa deﬁnition oftheformallanguage.
Aformal language is a set of strings, each string composed of symbols from a Formal language
ﬁnite symbol-set called an alphabet (the same alphabet used above for deﬁning an Alphabet
automaton!). The alphabet for the sheep language is the set Σ={a,b,!}. Given a
modelm(such as a particular FSA), we can use L(m)to mean “the formal language

characterizedby m”. So theformallanguagedeﬁnedbyoursheeptalkautomaton min
Fig.2.10(andthetransitiontableonpage28)istheinﬁnite set:
L(m) ={baa!,baaa!,baaaa!,baaaaa!,baaaaaa!,...} (2.1)
The usefulness of an automaton for deﬁning a language is that it can express an
inﬁnite set (such as this one above) in a closed form. Formal l anguages are not the

DRAFTSection2.2. Finite-StateAutomata 31
same asnatural languages , which are the kind of languages that real people speak. Natural language
In fact, a formal language may bear no resemblance at all to a r eal language (e.g., a
formal language can be used to model the differentstates of a soda machine). But we
often use a formal language to model part of a natural languag e, such as parts of the
phonology,morphology,or syntax. The term generativegrammar is sometimesused
inlinguisticstomeanagrammarofaformallanguage;theori ginofthetermisthisuse
ofanautomatonto deﬁnea languagebygeneratingall possibl estrings.
2.2.3 Another Example
In the previous examples our formal alphabet consisted of le tters; but we can also
have a higher level alphabet consisting of words. In this way we can write ﬁnite-state
automatathatmodelfactsaboutwordcombinations. Forexam ple,supposewewanted
to build an FSA that modeled the subpart of English dealing wi th amounts of money.

Such a formal language would model the subset of English cons isting of phrases like
tencents,three dollars ,onedollarthirty-ﬁvecents andsoon.
We might break this down by ﬁrst building just the automaton t o account for the
numbersfrom1to99,sincewe’ll needthemto dealwithcents. Fig.2.15showsthis.
q q1q2 twenty
thirty
forty
fiftysixty
seventy
eighty
ninetyone
two
three
four
fivesix
seven
eight
nineone
two
three
four
fivesix
seven
eight
nine
teneleven
twelve
thirteen
fourteenfifteen
sixteen
seventeen
eighteen
nineteen

thirty
forty
seventy
eighty
ninetyone
two
three
four
fivesix
seven
eight
nineone
two
three
four
fivesix
seven
eight
nine
teneleven
twelve
thirteen
sixteen
seventeen
eighteen
nineteen
Figure2.15 AnFSAfor the words for Englishnumbers 1–99.
We could now add centsanddollarsto our automaton. Fig. 2.16 shows a simple
version of this, where we just made two copies of the automato n in Fig. 2.15 and
appendedthewords centsanddollars.
q0q1q2 twenty
thirty
forty
seventy
eighty
ninetyone
two
three
four
fivesix
seven
eight
nineq

q
q5q

twenty
thirty
forty
seventy
eighty
ninetyone
two
three
four
fivesix
seven
eight
nineone
two
three
four
fivesix
seven
eight
ninesixteen
seventeen
eighteen
nineteenten
twenty
thirty
forty
seventy
eighty

thirty
forty
seventy
eighty
ninetyone
two
three
four
fivesix
seven
eight
nineq

q
q5q

twenty
thirty
forty
seventy
eighty
ninetyone
two
three
four
fivesix
seven
eight
nineone
two
three
four
fivesix
seven
eight
seventeen
eighteen
twenty
thirty
forty
seventy
eighty
ninety
q
cents
cents dollarsone
two
three
four
fivesix
seven
eight
nineeleven
twelve
thirteen
fourteen
fifteensixteen
seventeen
eighteen
twenty
thirty
forty
seventy
eighty
ninetyeleven
twelve
thirteen
fourteen
fifteen
Figure2.16 FSAfor the simple dollars andcents.
We would now need to add in the grammar for different amounts o f dollars; in-
cluding highernumberslike hundred,thousand. We’d also need to make sure that the

DRAFT32 Chapter 2. RegularExpressionsandAutomata
nounslike centsanddollarsare singular when appropriate( one cent,one dollar ), and
plural when appropriate ( ten cents,two dollars ). This is left as an exercise for the
reader (Exercise 3). We can think of the FSAs in Fig. 2.15 and F ig. 2.16 as simple
grammars of parts of English. We will return to grammar-buil ding in Part II of this
book,particularlyinCh. 12.
2.2.4 Non-Deterministic FSAs
Let’s extend our discussion now to another class of FSAs: non-deterministic FSA s
(orNFSAs). Consider the sheeptalk automaton in Fig. 2.17, which is m uch like our
ﬁrst automatonin Fig.2.10:

3a
0
 q1q2b a a !
Figure2.17 A non-deterministic ﬁnite-state automaton for talking she ep (NFSA #1). Com-
pare withthe deterministic automaton inFig.2.10.
The only difference between this automaton and the previous one is that here in
Fig. 2.17 the self-loop is on state 2 instead of state 3. Consi der using this network as

3a
0
 q1q2b a a !
an automaton for recognizing sheeptalk. When we get to state 2, if we see an awe
don’t know whether to remain in state 2 or go on to state 3. Auto mata with decision
points like this are called non-deterministic FSA s (orNFSAs). Recall by contrast Non-deterministic
NFSAthat Fig. 2.10 speciﬁed a deterministic automaton, i.e., one whose behavior during
recognition is fully determined by the state it is in and the symbol it is looking at. A
deterministicautomatoncanbereferredtoasa DFSA.Thatisnottrueforthemachine DFSA
inFig. 2.17(NFSA#1).

There is another common type of non-determinism, caused by a rcs that have no
symbols on them (called ǫ-transitions ). The automaton in Fig. 2.18 deﬁnes the exact ǫ-transition
samelanguageasthelast one,orourﬁrst one,butit doesit wi thanǫ-transition.

3

0q4 q1q2b a a !∋
Figure2.18 Another NFSA for the sheep language (NFSA #2). It differs fro m NFSA #1 in
Fig.2.17inhaving an ǫ-transition.
We interpret this new arc as follows: If we are in state 3, we ar e allowed to move
to state 2 withoutlooking at the input, or advancing our input pointer. So this intro-

3

duces another kind of non-determinism — we might not know whe ther to follow the
ǫ-transitionorthe !arc.

DRAFTSection2.2. Finite-StateAutomata 33
2.2.5 UsinganNFSA toAccept Strings
If we want to knowwhethera stringis an instance of sheeptalk or not, andif we use a
non-deterministicmachinetorecognizeit, we mightfollow the wrongarcandrejectit
whenwe shouldhave acceptedit. That is, since thereis moret hanone choiceat some
point, we might take the wrong choice. This problem of choice in non-deterministic
models will come up again and again as we build computational models, particularly
forparsing. Therearethreestandard solutionstothe problemofnon-determinism :
•Backup: Wheneverwe cometo a choice point,we couldput a markerto mark Backup
wherewewereintheinput,andwhatstatetheautomatonwasin . Thenifitturns
outthat wetookthewrongchoice,wecouldbackupandtryanot herpath.
•Look-ahead: We couldlookaheadin theinputtohelpusdecidewhichpathto Look-ahead
take.
•Parallelism: Whenever we come to a choice point, we could look at every Parallelism
alternativepathinparallel.

take.
We will focushereonthebackupapproachanddeferdiscussio nof thelook-ahead
andparallelismapproachestolaterchapters.
The backup approach suggests that we should blithely make ch oices that might
lead to deadends, knowing that we can always return to the une xplored alternatives.
Therearetwo keystothisapproach: weneedtorememberallth ealternativesforeach
choicepoint,andwe needtostore sufﬁcientinformationabo uteachalternativeso that
we can return to it when necessary. When a backup algorithm re aches a point in its

processing where no progress can be made (because it runs out of input, or has no
legal transitions), it returns to a previous choice point, s elects one of the unexplored
alternatives, and continues from there. Applying this noti on to our non-deterministic
recognizer, we need only remember two things for each choice point: the state, or
node,ofthemachinethatwecangotoandthecorrespondingpo sitiononthetape. We
will call the combination of the node and position the search-state of the recognition Search-state

algorithm. To avoid confusion,we will refer to the state of t he automaton(as opposed
tothe stateofthe search)asa nodeoramachine-state .
Input
Stateba!ǫ
01/ 0/ 0/ 0
1/ 02/ 0/ 0
2/ 02,3/ 0/ 0
3/ 0/ 04/ 0
4:/ 0/ 0/ 0/ 0Beforegoingontodescribethemainpartofthisalgorithm,
we should note two changes to the transition table that drive s
it. First, to represent nodes that have outgoing ǫ-transitions,
we add a new ǫ-column to the transition table. If a node has
anǫ-transition, we list the destinationnodein the ǫ-columnfor
that node’s row. The second addition is needed to account for

Input
Stateba!ǫ
01/ 0/ 0/ 0
1/ 02/ 0/ 0
3/ 0/ 04/ 0
multipletransitionstodifferentnodesfromthesameinput sym-
bol. We let each cell entryconsist ofa list of destinationno des
rather than a single node. On the rightwe show the transition table for the machine in
Fig. 2.17 (NFSA #1). While it has no ǫ-transitions, it doesshow that in machine-state
q2theinput acanleadbackto q2orontoq3.
Fig. 2.19 shows the algorithm for using a non-deterministic FSA to recognize an
input string. The function ND-RECOGNIZE uses the variable agendato keep track of

all the currently unexplored choices generated during the c ourse of processing. Each

DRAFT34 Chapter 2. RegularExpressionsandAutomata
choice (search state) is a tuple consisting of a node (state) of the machine and a posi-
tion on the tape. The variable current-search-state represents the branch choice being
currentlyexplored.
ND-RECOGNIZE begins by creating an initial search-state and placing it on the
agenda. For now we don’t specify what order the search-state s are placed on the
agenda. This search-state consists of the initial machine- state of the machine and a
pointer to the beginning of the tape. The function NEXTis then called to retrieve an
itemfromtheagendaandassign itto thevariable current-search-state .
As with D-RECOGNIZE , the ﬁrst task of the main loop is to determine if the en-
tire contents of the tape have been successfully recognized . This is done via a call
toACCEPT-STATE?, which returns acceptif the current search-state contains both an
accepting machine-state and a pointer to the end of the tape. If we’re not done, the

machine generates a set of possible next steps by calling GENERATE -NEW-STATES,
which creates search-states for any ǫ-transitions and any normal input-symbol transi-
tions from the transition table. All of these search-state t uples are then added to the
currentagenda.
Finally, we attempt to get a new search-state to process from the agenda. If the
agenda is empty we’ve run out of options and have to reject the input. Otherwise, an
unexploredoptionisselectedandtheloopcontinues.

It is important to understand why ND-RECOGNIZE returns a value of reject only
whenthe agendais foundto be empty. Unlike D-RECOGNIZE , it doesnotreturnreject
when it reaches the end of the tape in a non-accept machine-st ate or when it ﬁnds
itselfunabletoadvancethetapefromsomemachine-state. T hisisbecause,inthenon-
deterministiccase,suchroadblocksonlyindicatefailure downagivenpath,notoverall
failure. Wecanonlybesurewecanrejectastringwhenallpos siblechoiceshavebeen
examinedandfoundlacking.

Fig. 2.20 illustrates the progress of ND-RECOGNIZE as it attempts to handle the
inputbaaa!. Each strip illustrates the state of the algorithm at a given point in its
processing. The current-search-state variable is captured by the solid bubbles repre-
sentingthemachine-statealongwiththearrowrepresentin gprogressonthetape. Each
strip lowerdownin theﬁgurerepresentsprogressfromone current-search-state to the
next.
Littleofinteresthappensuntilthealgorithmﬁndsitselfi nstateq2whilelookingat

next.
the seconda onthe tape. An examinationofthe entryfortrans ition-table[ q2,a] returns
bothq2andq3. Search states are created for each of these choices and plac ed on the
agenda. Unfortunately,our algorithm chooses to move to sta teq3, a move that results
in neither an accept state nor any new states since the entry f or transition-table[ q3, a]
isempty. At thispoint,thealgorithmsimplyaskstheagenda fora newstate topursue.
Sincethechoiceofreturningto q2fromq2istheonlyunexaminedchoiceontheagenda

it isreturnedwith thetapepointeradvancedtothenexta. So mewhatdiabolically, ND-
RECOGNIZE ﬁndsitselffacedwiththesamechoice. Theentryfortransit ion-table[ q2,a]
still indicates that looping back to q2or advancing to q3are valid choices. As before,
statesrepresentingbothareplacedontheagenda. Thesesea rchstatesarenotthesame
asthepreviousonessincetheirtapeindexvalueshaveadvan ced. Thistimetheagenda
providesthemoveto q3asthenextmove. Themoveto q4,andsuccess,isthenuniquely
determinedbythetapeandthetransition-table.

DRAFTSection2.2. Finite-StateAutomata 35
function ND-RECOGNIZE (tape,machine )returnsaccept orreject
agenda←{(Initial stateof machine, beginning of tape) }
current-search-state ←NEXT(agenda)
loop
ifACCEPT-STATE?(current-search-state ) returns true then
else
agenda←agenda∪GENERATE -NEW-STATES(current-search-state )
ifagendaisemptythen
else
end
function GENERATE -NEW-STATES(current-state )returnsa set of search-states
current-node←the node the current search-state isin
index←the point on the tape the current search-state is looking at
returna listof searchstates from transitiontable as follows:
(transition-table[current-node, ǫ], index)
∪
(transition-table[current-node, tape[index]], index +1 )
function ACCEPT-STATE?(search-state )returnstrue or false
current-node←the node search-state is in
index←the point on the tape search-state is looking at
ifindexisat the endof the tape andcurrent-node isanaccept stateof machine
then
returntrue
else
returnfalse

∪
then
else
Figure2.19 An algorithm for NFSA recognition. The word nodemeans a state of the FSA,
whilestateorsearch-state means “the state of the search process”, i.e., a combination ofnode
andtape-position .
2.2.6 RecognitionasSearch
ND-RECOGNIZE accomplishesthetask ofrecognizingstringsin aregularla nguageby
providinga way to systematically exploreall the possible p athsthrougha machine. If
this explorationyields a path endingin an accept state, it a ccepts the string, otherwise

then
else
it rejects it. This systematic exploration is made possible by the agenda mechanism,
whichoneachiterationselectsapartialpathtoexploreand keepstrackofanyremain-
ing,asyetunexplored,partialpaths.
Algorithms such as ND-RECOGNIZE , which operate by systematically searching
for solutions, are known as state-space search algorithms. In such algorithms, the State-space search
problem deﬁnition creates a space of possible solutions; th e goal is to explore this

space, returning an answer when one is found or rejecting the input when the space

DRAFT36 Chapter 2. RegularExpressionsandAutomata
baaa!q0
baaa!q0q1
baaa!q1q2
baaa!q4baaa!q3baaa!q2q3
baaa!q2
baaa!q3   
   
   1
2
3
4
56
7
8
Figure2.20 Tracing the executionof NFSA#1(Fig.2.17) onsome sheeptal k.
has been exhaustively explored. In ND-RECOGNIZE , search states consist of pairings
of machine-states with positions on the input tape. The stat e-space consists of all the
pairings of machine-state and tape positions that are possi ble given the machine in
question. The goal of the search is to navigate through this s pace from one state to
anotherlookingforapairingofanacceptstate withan endof tapeposition.
Thekeytotheeffectivenessofsuchprogramsisoftenthe orderinwhichthestates
in the space are considered. A poor ordering of states may lea d to the examinationof
a largenumberofunfruitfulstatesbeforea successfulsolu tionis discovered. Unfortu-
nately, it is typically not possible to tell a good choice fro m a bad one, and often the

best wecandoistoinsurethateachpossiblesolutioniseven tuallyconsidered.
Carefulreadersmayhavenoticedthattheorderingofstates inND-RECOGNIZE has
been left unspeciﬁed. We know only that unexplored states ar e added to the agenda
astheyarecreatedandthatthe(undeﬁned)functionN EXTreturnsanunexploredstate
from the agenda when asked. How should the function N EXTbe deﬁned? Consider
an ordering strategy where the states that are considered ne xt are the most recently
createdones. Sucha policycan beimplementedbyplacingnew lycreatedstates at the

front of the agenda and having N EXTreturn the state at the front of the agenda when
called. Thusthe agendais implementedbya stack. Thisis commonlyreferredto asa

DRAFTSection2.2. Finite-StateAutomata 37
depth-ﬁrstsearch orLast In FirstOut (LIFO)strategy. Depth-ﬁrst
Such a strategy dives into the search space following newly d eveloped leads as
they are generated. It will only return to consider earlier o ptionswhen progressalong
a current lead has been blocked. The trace of the executionof ND-RECOGNIZE on the
stringbaaa!as shownin Fig. 2.20illustratesa depth-ﬁrstsearch. The al gorithmhits
the ﬁrst choice point after seeing bawhen it has to decide whether to stay in q2or
advance to state q3. At this point, it chooses one alternative and follows it unt il it is
sureit’swrong. Thealgorithmthenbacksupandtriesanothe rolderalternative.
Depth ﬁrst strategies have one major pitfall: under certain circumstances they can
enter an inﬁnite loop. This is possible either if the search s pace happens to be set
up in such a way that a search-state can be accidentally re-vi sited, or if there are an
inﬁnite number of search states. We will revisit this questi on when we turn to more

complicatedsearchproblemsinparsinginCh.13.
The second way to order the states in the search space is to con sider states in the
order in which they are created. Such a policy can be implemen ted by placing newly
created states at the back of the agenda and still have N EXTreturn the state at the
front of the agenda. Thus the agenda is implemented via a queue. This is commonly
referredto as a breadth-ﬁrst search orFirst In First Out (FIFO) strategy. Consider Breadth-ﬁrst

a differenttraceof theexecutionof ND-RECOGNIZE on thestring baaa!as shownin
Fig.2.21. Again,thealgorithmhitsitsﬁrstchoicepointaf terseeing bawhenit hadto
decide whether to stay in q2or advance to state q3. But now rather than picking one
choiceandfollowingit up,we imagineexaminingall possibl e choices,expandingone
plyofthesearchtreeat atime.
baaa!q0
baaa!q0q1
baaa!q1q2
baaa!q2
baaa!q3   
   
   1
2
3
4
54
5
6baaa!q2
5
Figure2.21 A breadth-ﬁrst traceof FSA#1 onsome sheeptalk.

DRAFT38 Chapter 2. RegularExpressionsandAutomata
Like depth-ﬁrst search, breadth-ﬁrst search has its pitfal ls. As with depth-ﬁrst if
the state-space is inﬁnite, the search may never terminate. More importantly, due to
growthin the size of the agendaif the state-space is evenmod eratelylarge, the search
may require an impractically large amount of memory. For sma ll problems, either
depth-ﬁrst or breadth-ﬁrst search strategies may be adequa te, although depth-ﬁrst is
normally preferred for its more efﬁcient use of memory. For l arger problems, more
complexsearchtechniquessuchas dynamicprogramming orA∗mustbeused,aswe
will see inChapters7and10.
2.2.7 Relating DeterministicandNon-Deterministic Autom ata
It may seem that allowingNFSAs to have non-deterministicfe atureslike ǫ-transitions
would make them more powerful than DFSAs. In fact this is not t he case; for any
NFSA, there is an exactly equivalent DFSA. In fact there is a s imple algorithm for

converting an NFSA to an equivalent DFSA, although the numbe r of states in this
equivalentdeterministicautomatonmaybe muchlarger. See LewisandPapadimitriou
(1988)or Hopcroftand Ullman (1979)forthe proofof the corr espondence. The basic
intuitionoftheproofisworthmentioning,however,andbui ldsonthewayNFSAsparse
theirinput. RecallthatthedifferencebetweenNFSAsandDF SAsisthatinanNFSAa
stateqimayhavemorethanonepossiblenextstategivenaninput i(forexample qaand
qb). ThealgorithminFig.2.19dealtwiththisproblembychoos ingeither qaorqband

thenbacktracking if the choice turned out to be wrong. We mentioned that a paral lel
versionofthealgorithmwouldfollowbothpaths(toward qaandqb) simultaneously.
ThealgorithmforconvertingaNFSA toaDFSA islikethispara llelalgorithm;we
build an automatonthat hasa deterministic path for everypa th our parallel recognizer
might havefollowedin the search space. We imaginefollowin gboth pathssimultane-
ously,andgrouptogetherintoan equivalenceclass all the s tates we reachon the same
input symbol (i.e., qaandqb). We now give a new state label to this new equivalence

classstate(forexample qab). We continuedoingthisforeverypossibleinputforevery
possiblegroupof states. The resultingDFSA can haveasmany statesas thereare dis-
tinct sets of states in the original NFSA. The numberof diffe rent subsets of a set with
Nelementsis2N, hencethe newDFSAcanhaveasmanyas2Nstates.
2.3 RegularLanguagesandFSAs
Aswesuggestedabove,theclassoflanguagesthataredeﬁnab lebyregularexpressions
is exactly the same as the class of languages that are charact erizable by ﬁnite-state

automata (whether deterministic or non-deterministic). B ecause of this, we call these
languages the regular languages . In order to give a formal deﬁnition of the class of Regularlanguage
regularlanguages,weneedtoreferbacktotwoearlierconce pts: thealphabet Σ,which
isthesetofallsymbolsinthelanguage,andthe emptystring ǫ,whichisconventionally
not includedin Σ. In addition, we make referenceto the empty set / 0 (whichis distinct
fromǫ). Theclassofregularlanguages(or regularsets )overΣisthenformallydeﬁned

DRAFTSection2.3. RegularLanguagesandFSAs 39
asfollows:1
1. / 0is aregularlanguage
2.∀a∈Σ∪ǫ,{a}isa regularlanguage
3. IfL1andL2are regularlanguages,thenso are:
(a)L1·L2={xy|x∈L1,y∈L2},theconcatenation ofL1andL2
(b)L1∪L2,theunionordisjunction ofL1andL2
(c)L∗
1,theKleeneclosure ofL1
Only languages which meet the above propertiesare regular l anguages. Since the
regularlanguagesarethelanguagescharacterizablebyreg ularexpressions,allthereg-
ular expression operators introduced in this chapter (exce pt memory) can be imple-
mented by the three operations which deﬁne regular language s: concatenation, dis-
junction/union(alsocalled“ |”),andKleeneclosure. Forexampleallthecounters( *,+,
{n,m}) are just a special case of repetition plus Kleene *. All the a nchors can be
thought of as individualspecial symbols. The square braces []are a kind of disjunc-
tion (i.e., [ab]means“aorb”, or the disjunctionof aandb). Thusit is true that any
regular expression can be turned into a (perhapslarger) exp ression which only makes

useofthe threeprimitiveoperations.
Regular languages are also closed under the following opera tions (Σ∗means the
inﬁniteset ofall possiblestringsformedfromthe alphabet Σ):
intersection ifL1andL2are regular languages, then so is L1∩L2, the
languageconsistingofthesetofstringsthatareinboth L1
andL2.
difference ifL1andL2are regular languages, then so is L1−L2, the
language consisting of the set of strings that are in L1but
notL2.
complementation IfL1isaregularlanguage,thensois Σ∗−L1,thesetofall

andL2.
notL2.
possiblestringsthataren’tin L1.
reversal IfL1is a regular language, then so is LR
1, the language
consistingoftheset ofreversalsofall thestringsin L1.
The proof that regular expressions are equivalent to ﬁnite- state automata can be
found in Hopcroft and Ullman (1979), and has two parts: showi ng that an automaton
can be built for each regular language, and conversely that a regular language can be
builtforeachautomaton.
We won’t give the proof, but we give the intuition by showing h ow to do the ﬁrst

part: take any regular expression and build an automaton fro m it. The intuition is
inductive on the number of operators: for the base case we bui ld an automaton to
correspondto the regularexpressionswith no operators,i. e. the regularexpressions / 0,
ǫ,oranysinglesymbol a∈Σ. Fig.2.22showstheautomataforthesethreebasecases.
Now for the inductive step, we show that each of the primitive operations of a
regularexpression(concatenation,union,closure)canbe imitatedbyanautomaton:

1Following van Santen and Sproat (1998), Kaplan and Kay (1994 ), and Lewis and Papadimitriou (1988).

DRAFT40 Chapter 2. RegularExpressionsandAutomata
(a) r=εq0qfq0qfq0qf
(b) r= /uni2205.15477 (c) r=aa
Figure2.22 Automata for the base case (no operators) for the induction s howing that any
regular expression can be turnedintoan equivalent automat on.
•concatenation : WejuststringtwoFSAsnexttoeachotherbyconnectingallt he
ﬁnalstatesofFSA 1tothe initialstate ofFSA 2byanǫ-transition.
q0qfq0 qf
ε
FSA1FSA2
Figure2.23 The concatenation of twoFSAs.
•closure: We create a new ﬁnal and initial state, connect the original ﬁnal states
of the FSA back to the initial states by ǫ-transitions (this implements the rep-
etition part of the Kleene *), and then put direct links betwe en the new initial
and ﬁnal states by ǫ-transitions (this implements the possibility of having zero
occurrences). We’dleave outthislast parttoimplementKle ene-plusinstead.
q0qfq0
qfε
FSA1εε
ε
Figure2.24 The closure (Kleene *) ofan FSA.
•union: We adda single new initial state q′
0, and add new ǫ-transitionsfromit to

q0qfq0
qfε
FSA1εε
ε
theformerinitial statesofthetwo machinestobe joined.
Wewillreturntoregularlanguagesandtheirrelationshipt oregulargrammarsinCh.15.
2.4 Summary
Thischapterintroducedthe most importantfundamentalcon ceptin languageprocess-
ing, theﬁnite automaton , and the practical tool based on automaton, the regular ex-

DRAFTSection2.4. Summary 41
q0qf
q0 qfε
FSA1
FSA2q0
qfε
ε ε
Figure2.25 The union (|)of twoFSAs.
pression. Here’sasummaryofthemainpointswecoveredabouttheseid eas:
•Theregularexpression languageisa powerfultoolforpattern-matching.
•Basic operationsin regular expressionsinclude concatenation of symbols, dis-
junction of symbols ( [],|, and.),counters (*,+, and{n,m}),anchors(ˆ,
$)andprecedenceoperators( (,)).
•Anyregularexpressioncanberealizedasa ﬁnite-stateautomaton (FSA).
•Memory( \1togetherwith ())isanadvancedoperationthatisoftenconsidered
partofregularexpressions,butwhichcannotberealizedas a ﬁniteautomaton.
•An automaton implicitly deﬁnes a formal language as the set of strings the
automaton acceptsoveranyvocabulary(set ofsymbols).
•The behavior of a deterministic automaton ( DFSA) is fully determined by the
stateit is in.
•Anon-deterministic automaton ( NFSA) sometimes has to make a choice be-
tweenmultiplepathstotake giventhesamecurrentstate and nextinput.
•AnyNFSAcanbeconvertedto a DFSA.

•The order in which a NFSAchoosesthe next state to exploreon the agendade-
ﬁnesitssearch strategy . Thedepth-ﬁrst search orLIFOstrategycorresponds
to the agenda-as-stack; the breadth-ﬁrst search orFIFOstrategy corresponds
totheagenda-as-queue.
•Any regular expression can be automatically compiled into a NFSAand hence
intoaFSA.

DRAFT42 Chapter 2. RegularExpressionsandAutomata
Finite automata arose in the 1950s out of Turing’s (1936) mod el of algorithmic com-
putation, considered by many to be the foundation of modern c omputer science. The
Turingmachinewasanabstractmachinewithaﬁnitecontrola ndaninput/outputtape.
In one move, the Turing machine could read a symbol on the tape , write a different
symbol on the tape, change state, and move left or right. Thus the Turing machine
differs from a ﬁnite-state automaton mainly in its ability t o change the symbols on its
tape.
Inspired by Turing’s work, McCulloch and Pitts built an auto mata-like model of
the neuron (see von Neumann, 1963, p. 319). Their model, whic h is now usually
called the McCulloch-Pitts neuron (McCulloch and Pitts, 1943), was a simpliﬁedMcCulloch-Pittsneuron
modeloftheneuronasakindof“computingelement”thatcoul dbedescribedinterms
of propositional logic. The model was a binary device, at any point either active or

tape.
not, which took excitatory and inhibitatory input from othe r neurons and ﬁred if its
activationpassed some ﬁxedthreshold. Based on the McCullo ch-Pittsneuron,Kleene
(1951) and (1956) deﬁned the ﬁnite automaton and regular exp ressions, and proved
their equivalence. Non-deterministic automata were intro duced by Rabin and Scott
(1959),whoalsoprovedthemequivalentto deterministicon es.
KenThompsonwasoneoftheﬁrsttobuildregularexpressions compilersintoedi-

torsfortextsearching(Thompson,1968). Hiseditor edincludedacommand“g/regular
expression/p”, or Global Regular Expression Print, which l ater became the UNIX
greputility.
There are many general-purpose introductions to the mathem atics underlying au-
tomata theory, such as Hopcroft and Ullman (1979) and Lewis a nd Papadimitriou
(1988). Thesecoverthemathematicalfoundationsofthesim pleautomataofthischap-
ter,aswellastheﬁnite-statetransducersofCh.3,thecont ext-freegrammarsofCh.12,
and the Chomsky hierarchy of Ch. 15. Friedl (1997) is a very us eful comprehensive

guideto theadvanceduseofregularexpressions.
The metaphor of problem-solvingas search is basic to Artiﬁc ial Intelligence(AI);
more details on search can be found in any AI textbook such as R ussell and Norvig
(2002).
Exercises
2.1Write regular expressionsfor the following languages: You may use either Perl
notation or the minimal “algebraic” notation of Sec. 2.3, bu t make sure to say
which one you are using. By “word”, we mean an alphabetic stri ng separated

(2002).
Exercises
from other words by white space, any relevant punctuation, l ine breaks, and so
forth.

DRAFTSection2.4. Summary 43
a. theset ofall alphabeticstrings.
b. theset ofall lowercasealphabeticstringsendingina b.
c. the set of all strings with two consecutive repeated words ( e.g., “Humbert
Humbert”and“thethe”butnot“thebug”or“thebigbug”).
d. the set of all stringsfromthe alphabet a,bsuch that each ais immediately
precededandimmediatelyfollowedbya b.
e. allstringsthatstartatthebeginningofthelinewithanin tegerandthatend
at theendofthelinewith aword.
f. allstringswhichhaveboththeword grottoandtheword raveninthem. (but
not,forexample,wordslike grottosthatmerely containtheword grotto).
g. write a patternwhich placesthe ﬁrst word of an Englishsent ence in a reg-
ister. Deal withpunctuation.
2.2Implement an ELIZA-like program, using substitutions such as those described
on page 26. You may choose a different domain than a Rogerian p sychologist,
ifyouwish,althoughkeepinmindthatyouwouldneedadomain inwhichyour
programcanlegitimatelydoalot ofsimple repeating-back.

2.3CompletetheFSAforEnglishmoneyexpressionsinFig.2.15a ssuggestedinthe
textfollowingthe ﬁgure. Youshouldhandleamountsupto $10 0,000,andmake
surethat“cent”and“dollar”havetheproperpluralendings whenappropriate.
2.4DesignanFSA thatrecognizessimpledateexpressionslike March15 ,the22nd
of November ,Christmas . You should try to include all such “absolute” dates,
(e.g. not“deictic”onesrelativetothecurrentdaylike thedaybeforeyesterday ).
Each edge of the graph should have a word or a set of words on it. You should

use some sort of shorthandfor classes of wordsto avoid drawi ngtoo many arcs
(e.g.,furniture→desk,chair,table).
2.5Now extend your date FSA to handle deictic expressionslike yesterday ,tomor-
row,a week from tomorrow ,the day before yesterday ,Sunday,next Monday ,
threeweeksfrom Saturday .
2.6WriteanFSAfortime-of-dayexpressionslike eleveno’clock ,twelve-thirty ,mid-
night,oraquartertoten andothers.
2.7(DuetoPaulineWelby;thisproblemprobablyrequirestheab ilitytoknit.) Write

a regular expression (or draw an FSA) which matches all knitt ing patterns for
scarveswiththefollowingspeciﬁcation: 32stitcheswide,K1P1ribbingonboth
ends, stockinette stitch body, exactly two raised stripes . All knitting patterns
must include a cast-on row (to put the correct numberof stitc hes on the needle)
and a bind-offrow (to end the pattern and preventunraveling ). Here’s a sample
patternforonepossiblescarfmatchingtheabovedescripti on:2

2Knitandpurlare two different types of stitches. The notation K nmeans do nknit stitches. Similarly for
purl stitches. Ribbing has a striped texture—most sweaters have ribbing at the sleeves, bottom, and neck.
Stockinette stitch is aseries of knit and purlrows that prod uces aplain pattern— socks or stockings are knit
with this basic pattern, hence thename.

DRAFT44 Chapter 2. RegularExpressionsandAutomata
1. Caston 32stitches. cast on; puts stitcheson needle
2. K1P1across row (i.e. do(K1P1) 16times). K1P1ribbing
3. Repeat instruction2seven more times. adds length
4. K32, P32. stockinette stitch
5. Repeat instruction4anadditional 13times. adds length
6. P32,P32. raised stripestitch
7. K32, P32. stockinette stitch
8. Repeat instruction7anadditional 251 times. adds length
9. P32,P32. raised stripestitch
10. K32, P32. stockinette stitch
11. Repeat instruction10 anadditional 13 times. adds length
12. K1P1across row. K1P1ribbing
13. Repeat instruction12 anadditional 7times. adds length
14. Bindoff 32stitches. binds offrow: ends pattern
2.8Writea regularexpressionforthelanguageacceptedbythe N FSA inFig.2.26.
q q0q1q
Figure2.26 A mysterylanguage
2.9Currently the function D-RECOGNIZE in Fig. 2.12 only solves a subpart of the
importantproblemofﬁndingastringinsometext. Extendthe algorithmtosolve
the following two deﬁciencies: (1) D-RECOGNIZE currently assumes that it is

q q0q1q
already pointing at the string to be checked, and (2) D-RECOGNIZE fails if the
stringitispointingincludesasapropersubstringalegals tringfortheFSA.That
is,D-RECOGNIZE failsif thereisan extracharacterat theendofthe string.
2.10Give an algorithm for negating a deterministic FSA. The nega tion of an FSA
accepts exactly the set of strings that the original FSA reje cts (over the same
alphabet),andrejectsall thestringsthat theoriginalFSA accepts.
2.11Why doesn’t your previous algorithm work with NFSAs? Now ext end your al-

gorithmto negateanNFSA.

Chapter3
Words &Transducers
Howcan there be any sininsincere?
Where is the good ingoodbye?
MeredithWillson, The Music Man
Chapter2introducedtheregularexpression,showingforex amplehowa singlesearch
string could help us ﬁnd both woodchuck andwoodchucks . Hunting for singular or
plural woodchuckswas easy; the plural just tacks an son to the end. But suppose we
were looking for another fascinating woodland creatures; l et’s say a fox, and aﬁsh,
thatsurly peccaryandperhapsaCanadian wildgoose . Huntingforthepluralsofthese
animals takes more than just tacking on an s. The plural of foxisfoxes; ofpeccary,

peccaries ; and ofgoose,geese. To confuse matters further, ﬁsh don’t usually change
theirformwhentheyareplural1.
It takes two kinds of knowledge to correctly search for singu lars and plurals of
theseforms. Orthographicrules tellusthatEnglishwordsendingin -yarepluralized
bychangingthe -yto-i-andaddingan -es.Morphologicalrules tell usthat ﬁshhasa
nullplural,andthatthepluralof gooseisformedbychangingthevowel.
The problem of recognizing that a word (like foxes) breaks down into component

morphemes( foxand-es) and building a structured representation of this fact is ca lled
morphologicalparsing .Morphologicalparsing
Parsingmeanstakinganinputandproducingsomesortoflinguistics tructureforit. Parsing
We will use thetermparsingverybroadlythroughoutthisboo k,includingmanykinds
ofstructuresthatmightbeproduced;morphological,synta ctic,semantic,discourse;in
theformofastring,oratree,oranetwork. Morphologicalpa rsingorstemmingapplies
tomanyafﬁxesotherthanplurals;forexamplewemightneedt otakeanyEnglishverb

form ending in -ing(going,talking,congratulating ) and parse it into its verbal stem
plusthe-ingmorpheme. So giventhe surfaceorinput form going,we might want to Surface form
producetheparsedform VERB-go + GERUND-ing .
Morphologicalparsingisimportantthroughoutspeechandl anguageprocessing. It
plays a crucial role in Web search for morphologically compl ex languages like Rus-
sian or German; in Russian the word Moscowhas different endings in the phrases
Moscow,of Moscow ,from Moscow , and so on. We want to be able to automatically

searchforthe inﬂectedformsof the wordevenif theuser only typedin thebase form.
Morphologicalparsingalsoplaysacrucialroleinpart-of- speechtaggingforthesemor-
phologicallycomplexlanguages,as we will see inCh. 5. It is importantforproducing
the large dictionaries that are necessary for robust spell- checking. We will need it in
machine translation to realize for example that the French w ordsvaandallershould
bothtranslateto formsoftheEnglishverb go.
To solve the morphological parsing problem, why couldn’t we just store all the

plural forms of English nouns and -ingforms of English verbs in a dictionary and do
parsing by lookup? Sometimes we can do this, and for example f or English speech
1Seee.g.,Seuss (1960)

DRAFT46 Chapter 3. Words& Transducers
recognition this is exactly what we do. But for many NLP appli cations this isn’t pos-
sible because -ingisaproductive sufﬁx; bythiswe meanthatit appliesto everyverb. Productive
Similarly -sappliestoalmosteverynoun. Productivesufﬁxesevenapply tonewwords;
thus the new word faxcan automatically be used in the -ingform:faxing. Since new
words (particularly acronyms and proper nouns) are created every day, the class of
nouns in English increases constantly, and we need to be able to add the plural mor-
pheme-sto each of these. Additionally, the plural form of these new n ouns depends
on the spelling/pronunciationof the singular form; for exa mple if the noun ends in -z
thenthepluralformis -esratherthan -s. We’ll needtoencodetheserulessomewhere.
Finally, we certainly cannot list all the morphological var iants of every word in
morphologicallycomplexlanguageslikeTurkish,whichhas wordslike:
(3.1)uygarlas ¸tıramadıklarımızdanmıs ¸sınızcasına
uygar
civilized+las ¸
+BEC+tır

uygar
+BEC+tır
+CAUS+ama
+NABL+dık
+PART+lar
+PL+ımız
+P1PL+dan
+ABL+mıs ¸
+PAST+sınız
+2PL+casına
+AsIf
“(behaving) as ifyou are among those whom we could not civili ze”
Thevariouspiecesofthisword(the morphemes ) havethese meanings:
+BEC“become”
+CAUSthe causative verbmarker (‘cause toX’)
+NABL“not able”
+PARTpast participle form
+P1PL1st person pl possessive agreement
+2PL2nd person pl
+ABLablative (from/among) case marker
+AsIf derivationally forms an adverb froma ﬁnite verb
NotallTurkishwordslooklikethis;theaverageTurkishwor dhasaboutthreemor-

+PL+ımız
+P1PL+dan
+ABL+mıs ¸
+AsIf
phemes. But such long words do exist; indeed Kemal Oﬂazer, wh o came up with this
example, notes (p.c.) that verbs in Turkish have 40,000 poss ible forms not counting
derivational sufﬁxes. Adding derivational sufﬁxes, such a s causatives, allows a the-
oretically inﬁnite number of words, since causativization can be repeated in a single
word (You cause X to cause Y to ...do W ). Thus we cannot store all possible Turkish
wordsinadvance,andmustdomorphologicalparsingdynamic ally.

InthenextsectionwesurveymorphologicalknowledgeforEn glishandsomeother
languages. We thenintroducethe key algorithmfor morpholo gicalparsing,the ﬁnite-
state transducer . Finite-state transducersare a crucial technologythroug houtspeech
andlanguageprocessing,sowe will returntothemagaininla ter chapters.
Afterdescribingmorphologicalparsing,wewillintroduce somerelatedalgorithms
in this chapter. In some applicationswe don’tneed to parse a word,but we doneed to

map from the word to its root or stem. For example in informati on retrieval and web
search (IR), we might want to map from foxestofox; but mightnot need to also know
thatfoxesis plural. Just strippingoffsuch wordendingsiscalled stemming in IR. We Stemming
will describea simplestemmingalgorithmcalledthe Porterstemmer .
For other speech and language processing tasks, we need to kn ow that two words
haveasimilarroot,despitetheirsurfacedifferences. For examplethewords sang,sung,
andsingsareallformsoftheverb sing. Theword singissometimescalledthecommon

lemmaofthesewords,andmappingfromalloftheseto singiscalledlemmatization .2Lemmatization

DRAFTSection3.1. Surveyof(Mostly)EnglishMorphology 47
Next, we will introduce another task related to morphologic al parsing. Tokeniza-
tionorword segmentation is the task of separating out (tokenizing)wordsfrom run- Tokenization
ning text. In English, words are often separated from each ot her by blanks (whites-
pace), but whitespace is not always sufﬁcient; we’ll need to notice that New York and
rock’n’roll areindividualwordsdespitethefactthattheycontainspac es,butformany
applicationswe’ll needtoseparate I’mintothetwo words Iandam.
Finally, for many applications we need to know how similar tw o words are ortho-
graphically. Morphological parsing is one method for compu ting this similarity, but
anotheristojust comparethestringsofletterstosee howsi milartheyare. A common
way of doing this is with the minimum edit distance algorithm, which is important
throughout NLP. We’ll introduce this algorithm and also sho w how it can be used in
spell-checking.
3.1 Surveyof(Mostly)EnglishMorphology

Morphologyis the study of the way words are built up from smal ler meaning-bearing
units,morphemes . Amorphemeisoftendeﬁnedastheminimalmeaning-bearingu nit Morpheme
in a language. So for example the word foxconsists of a single morpheme (the mor-
phemefox) while the word catsconsistsof two: the morpheme catand the morpheme
-s.
As this example suggests, it is often useful to distinguish t wo broad classes of
morphemes: stemsandafﬁxes. Theexactdetailsofthedistinctionvaryfromlanguage Stem

-s.
Afﬁxto language, but intuitively, the stem is the “main” morphem e of the word, supplying
themainmeaning,while theafﬁxesadd“additional”meaning sofvariouskinds.
Afﬁxes are further divided into preﬁxes,sufﬁxes,inﬁxes, andcircumﬁxes . Pre-
ﬁxes precede the stem, sufﬁxes follow the stem, circumﬁxes d o both, and inﬁxes are
inserted inside the stem. For example, the word eatsis composed of a stem eatand
the sufﬁx -s. The word unbuckle is composedof a stem buckleand the preﬁx un-. En-

glish doesn’t have any good examples of circumﬁxes, but many other languages do.
In German, for example, the past participle of some verbs is f ormed by adding ge-to
the beginningof the stem and -tto the end; so the past participle of the verb sagen(to
say)isgesagt(said). Inﬁxes,inwhichamorphemeisinsertedinthemiddle ofaword,
occurverycommonlyforexampleinthePhilipinelanguageTa galog. Forexamplethe
afﬁxum,whichmarkstheagentofanaction,isinﬁxedtotheTagalogs temhingi“bor-

row” to produce humingi. There is oneinﬁx that occursin some dialectsof English in
which the taboo morphemes“f**king”or “bl**dy”or othersli ke them are inserted in
themiddleofotherwords(“Man-f**king-hattan”,“abso-bl **dy-lutely”3)(McCawley,
1978).
Awordcanhavemorethanoneafﬁx. Forexample,theword rewriteshasthepreﬁx
2Lemmatization is actually morecomplex, since itsometimes involves deciding onwhich sense ofaword
is present. Wereturn to this issue in Ch. 20.

1978).
3Alan JayLerner, thelyricist ofMyFair Lady,bowdlerized th elatter to abso-bloomin’lutely in thelyric to
“Wouldn’t ItBe Loverly?” (Lerner, 1978, p. 60).

DRAFT48 Chapter 3. Words& Transducers
re-, the stem write, and the sufﬁx -s. The word unbelievably has a stem ( believe) plus
three afﬁxes ( un-,-able, and-ly). While English doesn’t tend to stack more than four
or ﬁve afﬁxes, languages like Turkish can have words with nin e or ten afﬁxes, as we
saw above. Languagesthat tend to string afﬁxes together lik e Turkish does are called
agglutinative languages.
Thereare manywaystocombinemorphemestocreate words. Fou rofthese meth-
ods are common and play important roles in speech and languag e processing: inﬂec-
tion,derivation ,compounding ,andcliticization . Inﬂection
Derivation
Compounding
CliticizationInﬂection is the combinationof a word stem with a grammatical morpheme ,usu-
allyresultingin awordofthe sameclass astheoriginalstem , andusuallyﬁllingsome
syntacticfunctionlikeagreement. Forexample,Englishha stheinﬂectionalmorpheme
-sformarkingthe pluralonnouns,andtheinﬂectionalmorpheme -edformarkingthe

past tenseonverbs. Derivation isthecombinationofa wordstem with agrammatical
morpheme, usually resulting in a word of a differentclass, often with a meaning hard
to predict exactly. For example the verb computerize can take the derivational sufﬁx
-ationtoproducethenoun computerization .Compounding isthecombinationofmul-
tiple word stems together. For example the noun doghouse is the concatenationof the
morpheme dogwith the morpheme house. Finally, cliticization is the combinationof

a word stem with a clitic. A clitic is a morpheme that acts syntactically like a word, Clitic
but is reduced in form and attached (phonologicallyand some times orthographically)
to anotherword. Forexamplethe Englishmorpheme ’vein the word I’veisa clitic, as
is the French deﬁnite article l’in the word l’opera. In the following sections we give
moredetailsontheseprocesses.
3.1.1 Inﬂectional Morphology
English has a relatively simple inﬂectional system; only no uns, verbs, and sometimes

adjectives can be inﬂected, and the number of possible inﬂec tional afﬁxes is quite
small.
Englishnounshave onlytwo kindsof inﬂection: anafﬁx that m arkspluralandan Plural
afﬁx that marks possessive . For example, many (but not all) English nounscan either
appearinthe barestem or singular form,or takea pluralsufﬁx. Here are examplesof Singular
theregularpluralsufﬁx -s(alsospelled -es),andirregularplurals:
RegularNouns IrregularNouns
Singular catthrush mouse ox
Plural catsthrushes mice oxen

small.
While the regularplural is spelled -safter most nouns, it is spelled -esafter words
ending in -s(ibis/ibises ),-z(waltz/waltzes ),-sh(thrush/thrushes ),-ch(ﬁnch/ﬁnches ),
andsometimes -x(box/boxes ). Nounsendingin -yprecededbyaconsonantchangethe
-yto-i(butterﬂy/butterﬂies ).
The possessive sufﬁx is realized by apostrophe + -sfor regular singular nouns
(llama’s) and plural nouns not ending in -s(children’s ) and often by a lone apostro-
pheafterregularpluralnouns( llamas’)andsomenamesendingin -sor-z(Euripides’
comedies).

DRAFTSection3.1. Surveyof(Mostly)EnglishMorphology 49
English verbal inﬂection is more complicated than nominal i nﬂection. First, En-
glish has three kinds of verbs; main verbs , (eat, sleep, impeach ),modal verbs (can,
will,should ),andprimaryverbs (be,have,do )(usingthetermsofQuirketal.,1985).
In thischapterwe will mostly beconcernedwith the mainandp rimaryverbs,because
itisthesethathaveinﬂectionalendings. Oftheseverbsala rgeclassare regular,thatis Regularverb
tosay allverbsofthisclasshavethesame endingsmarkingth esame functions. These
regularverbs(e.g. walk, orinspect)havefourmorphologicalforms,asfollow:
MorphologicalClass RegularlyInﬂectedVerbs
stem walk merge trymap
-sform walks merges triesmaps
-ingparticiple walking merging tryingmapping
Past formor -edparticiple walked merged triedmapped
These verbs are called regular because just by knowing the st em we can predict
the other forms by adding one of three predictable endings an d making some regular

spelling changes (and as we will see in Ch. 7, regular pronunc iationchanges). These
regularverbsandformsaresigniﬁcantinthemorphologyofE nglishﬁrst becausethey
cover a majority of the verbs, and second because the regular class isproductive . As
discussed earlier, a productiveclass is one that automatic ally includes any new words
that enter the language. For example the recently-createdv erbfax(My mom faxed me
the note from cousin Everett ) takes the regularendings -ed,-ing,-es. (Note that the -s

formisspelled faxesratherthan faxs; we will discussspellingrulesbelow).
Theirregular verbs are those that have some more or less idiosyncratic forms of Irregularverb
inﬂection. Irregular verbs in English often have ﬁve differ ent forms, but can have as
manyaseight (e.g.,the verb be)or as fewas three (e.g. cutorhit). While constituting
a much smaller class of verbs (Quirk et al. (1985) estimate th ere are only about 250
irregular verbs, not counting auxiliaries), this class inc ludes most of the very frequent

verbsofthelanguage.4Thetablebelowshowssomesampleirregularforms. Notethat
anirregularverbcaninﬂectinthepast form(alsocalledthe preterite)bychangingits Preterite
vowel(eat/ate),oritsvowelandsomeconsonants( catch/caught ),orwithnochangeat
all (cut/cut).
MorphologicalClass IrregularlyInﬂected Verbs
stem eatcatch cut
-sform eatscatches cuts
-ingparticiple eatingcatching cutting
preterite atecaught cut
past participle eatencaught cut
Theway these formsare usedin a sentencewill bediscussed in the syntaxandse-

manticschaptersbutisworthabriefmentionhere. The -sformisusedinthe“habitual
present”formto distinguishthethird-personsingularend ing(ShejogseveryTuesday )
4Ingeneral, themorefrequentawordform,themorelikely iti stohaveidiosyncratic properties; thisisdue
to afact about language change; very frequent words tend to p reserve their form even if other words around
them are changing so as to become moreregular.

DRAFT50 Chapter 3. Words& Transducers
from the other choices of person and number ( I/you/we/they jog every Tuesday ). The
stem form is used in the inﬁnitive form, and also after certai n other verbs ( I’d rather
walk home ,I want to walk home ). The-ingparticiple is used in the progressive con- Progressive
structiontomarkpresentorongoingactivity( Itisraining ),orwhentheverbistreated
asanoun;thisparticularkindofnominaluseofaverbiscall edagerunduse:Fishing Gerund
is ﬁne if you live nearwater. The-ed/-enparticiple is used in the perfectconstruction Perfect
(He’s eaten lunch already ) or the passive construction ( The verdict was overturned
yesterday ).
In addition to noting which sufﬁxes can be attached to which s tems, we need to
capture the fact that a number of regular spelling changes oc cur at these morpheme
boundaries. For example, a single consonant letter is doubl ed before adding the -ing
and-edsufﬁxes (beg/begging/begged ). If the ﬁnal letter is “c”, the doublingis spelled

“ck” (picnic/picnicking/picnicked ). If the base ends in a silent -e, it is deleted before
adding-ingand-ed(merge/merging/merged ). Justasfornouns,the -sendingisspelled
-esafter verb stems ending in -s(toss/tosses ) ,-z, (waltz/waltzes )-sh, (wash/washes )
-ch, (catch/catches )and sometimes -x(tax/taxes). Also like nouns, verbs ending in -y
precededbya consonantchangethe -yto-i(try/tries).
TheEnglishverbalsystemismuchsimplerthanforexampleth eEuropeanSpanish
system, which has as many as ﬁfty distinct verb forms for each regular verb. Fig. 3.1

shows just a few of the examples for the verb amar, ‘to love’. Other languages can
haveevenmoreformsthanthisSpanishexample.
Present Imperfect Future Preterite Present Conditional Imperfect Future
Indicative Indicative Subjunctive Subjunctive Subjunctive
1SG amo amaba amar´ e am´ e ame amar´ ıa amara amare
2SG amas amabas amar´ as amaste ames amar´ ıas amaras amares
3SG ama amaba amar´ a am´ o ame amar´ ıa amara am´ areme
1PL amamos am´ abamos amaremos amamos amemos amar´ ıamos am´ aramos am´ aremos

2PL am´ ais amabais amar´ eis amasteis am´ eis amar´ ıais amarais amareis
3PL aman amaban amar´ an amaron amen amar´ ıan amaran amaren
Figure3.1 To love in Spanish. Some of the inﬂected forms of the verb amarin European
Spanish.1SGstands for “ﬁrstperson singular”, 3PL for“thirdperson plu ral”, and soon.
3.1.2 DerivationalMorphology
While English inﬂection is relatively simple compared to ot her languages, derivation
in English is quite complex. Recall that derivation is the co mbination of a word stem

with a grammatical morpheme, usually resulting in a word of a differentclass, often
witha meaninghardtopredictexactly.
A very common kind of derivation in English is the formation o f new nouns, of-
ten from verbs or adjectives. This process is called nominalization . For example, nominalization
the sufﬁx -ationproducesnounsfrom verbs ending often in the sufﬁx -ize(computer-
ize→computerization ). Here are examples of some particularly productive Englis h
nominalizingsufﬁxes.

DRAFTSection3.1. Surveyof(Mostly)EnglishMorphology 51
Sufﬁx Base Verb/Adjective Derived Noun
-ation computerize(V) computerization
-ee appoint(V) appointee
-er kill (V) killer
-ness fuzzy(A) fuzziness
Adjectivescan also be derived from nouns and verbs. Here are examples of a few
sufﬁxesderivingadjectivesfromnounsorverbs.
Sufﬁx Base Noun/Verb DerivedAdjective
-al computation(N) computational
-able embrace(V) embraceable
-less clue(N) clueless
Derivation in English is more complex than inﬂection for a nu mber of reasons.
Oneisthatitisgenerallylessproductive;evenanominaliz ingsufﬁxlike -ation,which
can be added to almost any verb ending in -ize, cannot be added to absolutely ev-
ery verb. Thus we can’t say * eatationor *spellation (we use an asterisk (*) to mark
“non-examples” of English). Another is that there are subtl e and complex meaning
differencesamongnominalizingsufﬁxes. Forexample sincerityhasasubtledifference
inmeaningfrom sincereness .
3.1.3 Cliticization

Recallthatacliticisaunitwhosestatusliesinbetweentha tofanafﬁxandaword. The
phonologicalbehaviorofcliticsislikeafﬁxes;theytendt obeshortandunaccented(we
will talkmoreaboutphonologyinCh. 8). Theirsyntacticbeh aviorismorelikewords,
oftenacting aspronouns,articles, conjunctions,or verbs . Clitics precedinga wordare
calledproclitics ,while thosefollowingare enclitics. Proclitic
Enclitic Englishcliticsincludethese auxiliaryverbalforms:
Full Form Clitic Full Form Clitic
am ’m have ’ve
are ’re has ’s
is ’s had ’d
will ’ll would ’d

is ’s had ’d
Note that the clitics in English are ambiguous; Thus she’scan mean she isorshe
has. Except for a few such ambiguities, however, correctly segm enting off clitics in
English is simpliﬁed by the presence of the apostrophe. Clit ics can be harder to parse
in other languages. In Arabic and Hebrew, for example, the de ﬁnite article ( the;Alin
Arabic,hain Hebrew) is cliticized on to the front of nouns. It must be se gmented off
in order to do part-of-speech tagging, parsing, or other tas ks. Other Arabic proclitics

include prepositions like b‘by/with’, and conjunctions like w‘and’. Arabic also has
encliticsmarking certain pronouns. For example the word and by their virtues has
clitics meaning and,by, andtheir, a stemvirtue, and a plural afﬁx. Note that since

DRAFT52 Chapter 3. Words& Transducers
Arabicisreadrightto left,these wouldactuallyappearord eredfromrightto leftin an
Arabicword.
proclitic proclitic stemafﬁxenclitic
Arabic w b HsnAthm
Gloss and by virtuestheir
3.1.4 Non-concatenative Morphology
The kind of morphologywe have discussed so far, in which a wor d is composed of a
stringofmorphemesconcatenatedtogetherisoftencalled concatenativemorphology .Concatenativemorphology
Anumberoflanguageshaveextensive non-concatenativemorphology ,inwhichmor-
phemesarecombinedinmorecomplexways. TheTagaloginﬁxat ionexampleaboveis
one example of non-concatenativemorphology,since two mor phemes(hingiandum)
areintermingled.
Another kind of non-concatenative morphology is called templatic morphology
orroot-and-pattern morphology. This is very common in Arabic, Hebrew, and other
Semitic languages. In Hebrew, for example, a verb (as well as other parts-of-speech)
is constructed using two components: a root, consisting usu ally of three consonants

(CCC) and carrying the basic meaning, and a template, which g ives the ordering of
consonants and vowels and speciﬁes more semantic informati on about the resulting
verb, such as the semantic voice (e.g., active, passive, mid dle). For example the He-
brew tri-consonantal root lmd, meaning ‘learn’ or ‘study’, can be combined with the
active voice CaCaC template to produce the word lamad, ‘he studied’, or the inten-
sive CiCeC template to produce the word limed, ‘he taught’, or the intensive passive

templateCuCaCtoproducetheword lumad,‘hewastaught’. ArabicandHebrewcom-
bine this templatic morphology with concatenative morphol ogy (like the cliticization
exampleshownin theprevioussection).
3.1.5 Agreement
We introduced the plural morpheme above, and noted that plur al is marked on both
nounsandverbsinEnglish. Wesaythatthesubjectnounandth emainverbinEnglish
have toagreein number, meaning that the two must either be both singular o r both Agreement

plural. There are other kinds of agreement processes. For ex ample nouns, adjectives,
and sometimes verbs in many languages are marked for gender. A gender is a kind Gender
of equivalence class that is used by the language to categori ze the nouns; each noun
falls into one class. Many languages (for example Romance la nguages like French,
Spanish, or Italian) have 2 genders, which are referred to as masculine and feminine.
Other languages (like most Germanic and Slavic languages) h ave three (masculine,

feminine, neuter). Some languages, for example the Bantu la nguages of Africa, have
as many as 20 genders. When the number of classes is very large , we often refer to
themasnounclasses insteadofgenders. Noun class
Genderis sometimesmarkedexplicitlyon a noun;forexample Spanish masculine
words often end in -oand feminine words in -a. But in many cases the gender is not
marked in the letters or phonesof the noun itself. Instead, i t is a propertyof the word

DRAFTSection3.2. Finite-State MorphologicalParsing 53
thatmust bestoredin alexicon. We will see anexampleofthis inFig.3.2.
3.2 Finite-StateMorphologicalParsing
Let’s now proceed to the problem of parsing morphology. Our g oal will be to take
input forms like those in the ﬁrst and third columns of Fig. 3. 2, produce output forms
likethosein thesecondandfourthcolumn.
English Spanish
Input Morphological Parse InputMorphological Parse Gloss
cats cat +N +PL pavospavo+N +Masc +Pl ‘ducks’
cat cat +N +SG pavopavo+N +Masc +Sg ‘duck’
cities city+N +Pl bebobeber +V +PInd+1P +Sg ‘I drink’
geese goose +N +Pl cantocantar +V +PInd+1P+Sg ‘I sing’
goose goose +N +Sg cantocanto+N +Masc +Sg ‘song’
goose goose +V puseponer +V+Perf +1P+Sg ‘I was able’
gooses goose +V +1P +Sg vinovenir +V +Perf+3P +Sg ‘he/she came’
merging merge +V +PresPart vinovino+N +Masc +Sg ‘wine’
caught catch +V +PastPart lugarlugar +N+Masc +Sg ‘place’
caught catch +V +Past
Figure3.2 Output of a morphological parse for some English and Spanish words. Spanish

output modiﬁedfrom the XeroxXRCEﬁnite-statelanguage too ls.
The second column contains the stem of each word as well as ass orted morpho-
logicalfeatures. These features specify additional information about the s tem. For Feature
example the feature +Nmeans that the word is a noun; +Sgmeans it is singular, +Pl
that it is plural. Morphologicalfeatures will be referred t o again in Ch. 5 and in more
detail in Ch. 16; for now, consider +Sgto be a primitive unit that means “singular”.

Spanishhassomefeaturesthatdon’toccurinEnglish;forex amplethenouns lugarand
pavoaremarked +Masc(masculine). BecauseSpanishnounsagreeingenderwithad-
jectives,knowingthegenderofanounwill beimportantfort aggingandparsing.
Note that some of the input forms (like caught,goose,canto, orvino) will be am-
biguousbetweendifferentmorphologicalparses. Fornow,w ewillconsiderthegoalof
morphologicalparsing merely to list all possible parses. W e will return to the task of
disambiguatingamongmorphologicalparsesinCh.5.

Inorderto buildamorphologicalparser,we’ll needat least thefollowing:
1.lexicon:thelistofstemsandafﬁxes,togetherwithbasicinformatio naboutthem Lexicon
(whethera stem isaNounstemora Verbstem,etc.).
2.morphotactics: themodelofmorphemeorderingthatexplainswhichclasseso f morphotactics
morphemescan followotherclasses ofmorphemesinside a wor d. Forexample,
thefactthattheEnglishpluralmorphemefollowsthenounra therthanpreceding
itis amorphotacticfact.

3.orthographic rules: thesespelling rules are used to model the changes that
occurinaword,usuallywhentwomorphemescombine(e.g.,th ey→iespelling

DRAFT54 Chapter 3. Words& Transducers
rulediscussedabovethatchanges city + -s tocitiesratherthan citys).
The next section will discuss how to represent a simple versi on of the lexicon just
forthesub-problemofmorphologicalrecognition,includi nghowtouseFSAstomodel
morphotacticknowledge.
In following sections we will then introduce the ﬁnite-stat e transducer (FST) as a
wayofmodelingmorphologicalfeaturesin thelexicon,anda ddressingmorphological
parsing. Finally,we showhowto useFSTsto modelorthograph icrules.
3.3 BuildingaFinite-StateLexicon
A lexicon is a repository for words. The simplest possible le xicon would consist of
anexplicitlist ofeverywordofthe language( everyword,i.e.,includingabbreviations
(“AAA”)andpropernames(“Jane”or“Beijing”))asfollows:
a,AAA,AA, Aachen, aardvark, aardwolf, aba, abaca, aback, . ..
Since it will often be inconvenient or impossible, for the va rious reasons we dis-
cussed above, to list every word in the language, computatio nal lexicons are usually

structured with a list of each of the stems and afﬁxes of the la nguage together with a
representation of the morphotactics that tells us how they c an ﬁt together. There are
many ways to model morphotactics; one of the most common is th e ﬁnite-state au-
tomaton. A very simple ﬁnite-state model for English nomina l inﬂection might look
likeFig. 3.3.
q	q1q2reg-noun plural -s
irreg-sg-nounirreg-pl-noun
Figure3.3 Aﬁnite-state automaton forEnglishnominal inﬂection.

The FSA in Fig. 3.3 assumes that the lexicon includes regular nouns (reg-noun )
thattaketheregular -splural(e.g., cat,dog,fox,aardvark). Thesearethevastmajority
ofEnglishnounssincefornowwe will ignorethefact thatthe pluralofwordslike fox
have an inserted e:foxes. The lexicon also includes irregular noun forms that don’t
take-s, both singular irreg-sg-noun (goose, mouse ) and plural irreg-pl-noun (geese,
mice).
reg-noun irreg-pl-noun irreg-sg-noun plural
fox geese goose -s
cat sheep sheep
aardvark mice mouse

mice).
Asimilar modelforEnglishverbalinﬂectionmightlooklike Fig.3.4.

DRAFTSection3.3. Buildinga Finite-StateLexicon 55
q0qreg-verb-stempast (-ed)
irreg-verb-stemreg-verb-stem
q2irreg-past-verb-form
past participle (-ed)
present participle (-ing)3sg (-s)
Figure3.4 Aﬁnite-state automaton forEnglishverbal inﬂection
Thislexiconhasthreestemclasses(reg-verb-stem,irreg- verb-stem,andirreg-past-
verb-form), plus four more afﬁx classes ( -edpast,-edparticiple, -ingparticiple, and
thirdsingular -s):
reg-verb-stem irreg-verb-stem irreg-past-stem pastpast-part pres-part 3sg
walk cut caught -ed-ed -ing -s
fry speak ate
talk sing eaten
impeach sang
Englishderivationalmorphologyissigniﬁcantlymorecomp lexthanEnglishinﬂec-
tional morphology, and so automata for modeling English der ivation tend to be quite
complex. Some models of English derivation, in fact, are bas ed on the more complex
context-freegrammarsofCh. 12(Sproat,1993).
Consider a relatively simpler case of derivation: the morph otactics of English ad-
jectives. HerearesomeexamplesfromAntworth(1990):

big, bigger, biggest, cool, cooler, coolest, coolly
happy, happier, happiest, happily red,redder, reddest
unhappy, unhappier, unhappiest, unhappily real,unreal, r eally
clear, clearer, clearest,clearly, unclear, unclearly
An initial hypothesismight be that adjectivescan have an op tionalpreﬁx ( un-),an
obligatory root ( big, cool, etc.) and an optional sufﬁx ( -er, -est, or-ly). This might
suggestthetheFSA in Fig.3.5.
q
qq2un- adj-root∋q3-er  -est  -ly

q
Figure3.5 AnFSAforafragmentofEnglishadjectivemorphology: Antwo rth’sProposal#1.
Alas,whilethisFSAwillrecognizealltheadjectivesinthe tableabove,itwillalso
recognizeungrammaticalformslike unbig,unfast,oranger, orsmally. We need to set
up classes of roots and specify their possible sufﬁxes. Thus adj-root 1would include
adjectivesthatcanoccurwith un-and-ly(clear,happy,andreal)whileadj-root 2will
includeadjectivesthatcan’t( big,small),andso on.

DRAFT56 Chapter 3. Words& Transducers
This givesan idea of the complexityto be expectedfrom Engli sh derivation. As a
furtherexample,wegiveinFigure3.6anotherfragmentofan FSAforEnglishnominal
andverbalderivationalmorphology,basedonSproat(1993) ,Bauer(1983),andPorter
(1980). This FSA models a number of derivational facts, such as the well known
generalizationthat any verb ending in -izecan be followed by the nominalizingsufﬁx
-ation(Bauer,1983;Sproat,1993). Thussincethereisaword fossilize, wecanpredict
the word fossilization by following states q0,q1, andq2. Similarly, adjectives ending
in-alor-ableatq5(equal,formal,realizable ) can take the sufﬁx -ity, or sometimes
the sufﬁx -nessto stateq6(naturalness ,casualness ). We leave it as an exercise for
the reader(Exercise1) to discoversome of the individualex ceptionsto manyof these
constraints,andalsoto giveexamplesofsomeofthevarious nounandverbclasses.
q0q1q2nouni-ize/Vq3-ation/Nq4adj-al
q5q6-er/N -able/A
-ness/N-ity/N
adj-al
q7q8q9verbj-ive/Aadj-ous

adj-al
-ly/Adv-ness/N
q10q11-ly/Adv-ful/A -ative/Averbk
nounl
Figure3.6 AnFSAfor another fragment of Englishderivational morphol ogy.
We can now use these FSAs to solve the problem of morphological recognition ;
that is, ofdeterminingwhetheraninput stringof lettersma kesupa legitimate English
word or not. We do this by taking the morphotacticFSAs, and pl uggingin each “sub-
lexicon”intothe FSA. That is, we expandeach arc (e.g.,the reg-noun-stem arc) with

adj-al
nounl
allthemorphemesthatmakeupthesetof reg-noun-stem . TheresultingFSAcanthen
bedeﬁnedat theleveloftheindividualletter.
f
∋
g

o

ta
o s e
e e s e
Figure3.7 ExpandedFSAforafewEnglishnounswiththeirinﬂection. No tethatthisautoma-
tonwillincorrectlyaccepttheinput foxs. Wewillseebeginningonpage62howtocorrectlydeal
withthe inserted einfoxes.
Fig. 3.7 shows the noun-recognitionFSA producedby expandi ngthe Nominal In-
ﬂectionFSAofFig.3.3withsampleregularandirregularnou nsforeachclass. Wecan

DRAFTSection3.4. Finite-StateTransducers 57
use Fig. 3.7 to recognize strings like aardvarks by simply starting at the initial state,
and comparing the input letter by letter with each word on eac h outgoing arc, and so
on,justaswe sawin Ch.2.
3.4 Finite-StateTransducers
We’ve now seen that FSAs can represent the morphotactic stru cture of a lexicon, and
can be used for word recognition. In this section we introduc e the ﬁnite-state trans-
ducer. The next section will show how transducers can be appl ied to morphological
parsing.
A transducer maps between one representation and another; a ﬁnite-state trans-
ducerorFSTis a type of ﬁnite automaton which maps between two sets of sym bols. FST
We canvisualizeanFSTasatwo-tapeautomatonwhichrecogni zesorgenerates pairs
of strings. Intuitively, we can do this by labeling each arc i n the ﬁnite-state machine
with two symbol strings, one from each tape. Fig. 3.8 shows an example of an FST
whereeacharc islabeledbyaninputandoutputstring,separ atedbyacolon.
q0aa:b
q1b:

parsing.
q0aa:b
q1b:
b:a
b:b
a:ba
∋
Figure3.8 Aﬁnite-state transducer, modiﬁedfrom Mohri (1997).
The FST thus has a more general function than an FSA; where an F SA deﬁnes a
formallanguageby deﬁninga set of strings, anFST deﬁnesa relationbetweensets of
strings. Another way of looking at an FST is as a machine that r eads one string and
generatesanother. Here’sa summaryof thisfour-foldway of thinkingabouttransduc-
ers:
•FSTasrecognizer: atransducerthattakesa pairofstringsasinputandoutputs

q0aa:b
q1b:
b:a
b:b
a:ba
∋
ers:
acceptifthe string-pairisin thestring-pairlanguage,and rejectif it isnot.
•FSTasgenerator: amachinethatoutputspairsofstringsofthelanguage. Thus
theoutputisayesorno,anda pairofoutputstrings.
•FSTastranslator: amachinethat readsa stringandoutputsanotherstring
•FSTasset relater: a machinethatcomputesrelationsbetweensets.
Allofthesehaveapplicationsinspeechandlanguageproces sing. Formorphologi-
calparsing(andformanyotherNLPapplications),wewillap plytheFSTastranslator

metaphor, taking as input a string of letters and producing a s output a string of mor-
phemes.
Let’s begin with a formal deﬁnition. An FST can be formally de ﬁned with 7 pa-
rameters:

DRAFT58 Chapter 3. Words& Transducers
Q a ﬁniteset of Nstatesq0,q1,...,qN−1
Σ a ﬁniteset correspondingtothe inputalphabet
Δ a ﬁniteset correspondingtothe outputalphabet
q0∈Qthe startstate
F⊆Qthe set ofﬁnal states
δ(q,w)the transition function or transition matrix between state s; Given a
stateq∈Qand a string w∈Σ∗,δ(q,w)returnsa set of new states
Q′∈Q.δis thus a function from Q×Σ∗to 2Q(because there are
2Qpossible subsets of Q).δreturns a set of states rather than a
singlestatebecauseagiveninputmaybeambiguousinwhichs tate
it mapsto.
σ(q,w)theoutputfunctiongivingthesetofpossibleoutputstring sforeach
state and input. Given a state q∈Qand a string w∈Σ∗,σ(q,w)
givesa set ofoutputstrings, each a string o∈Δ∗.σis thus a func-
tionfrom Q×Σ∗to2Δ∗
Where FSAs are isomorphic to regular languages, FSTs are iso morphic to regu-
lar relations . Regular relations are sets of pairs of strings, a natural ex tension of the Regularrelation
regular languages, which are sets of strings. Like FSAs and r egular languages, FSTs

it mapsto.
and regular relations are closed under union, although in ge neral they are not closed
under difference, complementation and intersection (alth ough some useful subclasses
ofFSTsareclosedundertheseoperations;ingeneralFSTsthatarenota ugmentedwith
theǫare more likely to have such closure properties). Besides un ion, FSTs have two
additionalclosurepropertiesthatturnout tobeextremely useful:
inversion : The inversion of a transducer T(T−1) simply switches the Inversion
input andoutputlabels. Thusif Tmapsfromthe input alphabet Ito

theoutputalphabet O,T−1mapsfrom OtoI.
composition : IfT1isatransducerfrom I1toO1andT2atransducerfrom Composition
O1toO2,thenT1◦T2mapsfrom I1toO2.
Inversionisusefulbecauseit makesiteasytoconvertaFST- as-parserintoanFST-
as-generator.
Compositionisusefulbecauseitallowsustotaketwotransd ucersthatruninseries
andreplacethemwithonemorecomplextransducer. Composit ionworksasinalgebra;
applying T1◦T2to an input sequence Sis identical to applying T1toSand thenT2to
theresult; thus T1◦T2(S) =T2(T1(S)).
Fig.3.9,forexample,showsthecompositionof [a:b]+ with[b:c]+ toproduce
[a:c]+.

[a:c]+.
q0q1a:c
a:c
q0q1b:c
b:c
q0q1a:b
a:b
=
Figure3.9 The composition of [a:b]+ with[b:c]+ toproduce [a:c]+.

DRAFTSection3.4. Finite-StateTransducers 59
Theprojection of an FST is the FSA that is producedby extracting only one sid e Projection
oftherelation. We canrefertotheprojectiontotheleftoru ppersideoftherelationas
theupperorﬁrstprojectionandtheprojectiontothelowerorrightsideofth erelation
asthelowerorsecondprojection.
3.4.1 Sequential Transducers andDeterminism
Transducersas we have describedthem maybe nondeterminist ic,in that a giveninput
may translate to manypossible outputsymbols. Thus using ge neralFSTs requiresthe
kinds of search algorithmsdiscussed in Ch. 2, making FSTs qu ite slow in the general
case. Thissuggeststhatitwouldnicetohaveanalgorithmto convertanondeterministic
FST to a deterministic one. But while every non-determinist ic FSA is equivalent to
somedeterministicFSA, notall ﬁnite-statetransducersca nbedeterminized.
Sequential transducers , by contrast, are a subtype of transducers that are deter-Sequential
transducers

ministic on their input. At any state of a sequential transdu cer, each given symbol of
the input alphabet Σcan label at most one transition out of that state. Fig. 3.10 g ives
an example of a sequential transducer from Mohri (1997); not e that here, unlike the
transducer in Fig. 3.8, the transitions out of each state are deterministic based on the
state and the input symbol. Sequential transducers can have epsilon symbols in the
outputstring,butnotonthe input.
q0q1b:a:b
b:b
a:ba
∋

q0q1b:a:b
b:b
a:ba
∋
Figure3.10 A sequential ﬁnite-state transducer, from Mohri (1997).
Sequentialtransducersarenotnecessarilysequentialont heiroutput. Mohri’strans-
ducerinFig.3.10isnot,forexample,sincetwodistincttra nsitionsleavingstate0have
the same output ( b). Since the inverse of a sequential transducer may thus not b e se-
quential, we always need to specify the direction of the tran sduction when discussing
sequentiality. Formally, the deﬁnition of sequential tran sducers modiﬁes the δandσ

q0q1b:a:b
b:b
a:ba
∋
functions slightly; δbecomes a function from Q×Σ∗toQ(rather than to 2Q), andσ
becomesa functionfrom Q×Σ∗toΔ∗(ratherthanto 2Δ∗).
A generalization of sequential transducers, the subsequential transducer , gener-Subsequential
transducer
ates an additionaloutputstring at the ﬁnal states, concate natingit ontothe outputpro-
duced so far (Sch¨ utzenberger,1977). What makes sequentia l and subsequential trans-
ducers important is their efﬁciency; because they are deter ministic on input, they can

beprocessedintimeproportionaltothenumberofsymbolsin theinput(theyarelinear
in their input length) rather than proportionalto some much larger number which is a
functionofthenumberofstates. Anotheradvantageofsubse quentialtransducersisthat
there exist efﬁcient algorithms for their determinization (Mohri, 1997) and minimiza-
tion (Mohri, 2000), extendingthe algorithmsfor determini zationand minimization of
ﬁnite-stateautomatathat wesaw inCh. 2. alsoanequivalenc ealgorithm.

Whilebothsequentialandsubsequentialtransducersarede terministicandefﬁcient,

DRAFT60 Chapter 3. Words& Transducers
neither of them is able to handle ambiguity, since they trans duce each input string
to exactly one possible output string. Since ambiguity is a c rucial property of natu-
ral language, it will be useful to have an extension of subseq uential transducers that
can deal with ambiguity, but still retain the efﬁciency and o ther useful properties of
sequential transducers. One such generalization of subseq uential transducers is the
p-subsequential transducer. A p-subsequential transducer allows for p(p≥1)ﬁnal
outputstringsto be associated with eachﬁnal state (Mohri, 1996). Theycan thushan-
dlea ﬁnite amountofambiguity,whichisusefulformanyNLP t asks. Fig. 3.11shows
anexampleofa2-subsequentialFST.
q0q1a:a
b:aq2q3a:a
b:ba
b
Figure3.11 A 2-subsequential ﬁnite-state transducer, from Mohri (199 7).
Mohri(1996,1997)showanumberoftaskswhoseambiguitycan belimitedinthis
way, including the representation of dictionaries, the com pilation of morphological

q0q1a:a
b:ba
b
and phonological rules, and local syntactic constraints. F or each of these kinds of
problems,heandothershaveshownthattheyare p-subsequentializable ,andthuscan
bedeterminizedandminimized. Thisclassoftransducersin cludesmany,althoughnot
necessarilyall,morphologicalrules.
3.5 FSTsforMorphologicalParsing
Let’snowturntothetaskofmorphologicalparsing. Giventh einputcats,forinstance,
we’d like to output cat +N +Pl , telling us that catis a plural noun. Given the Spanish
inputbebo(‘I drink’),we’d like beber+V +PInd +1P +Sg , telling us that bebois the

presentindicativeﬁrst personsingularformoftheSpanish verbbeber,‘todrink’.
In theﬁnite-state morphology paradigm that we will use, we represent a word as
a correspondence between a lexical level , which represents a concatenation of mor-
phemes making up a word, and the surfacelevel, which represents the concatenation Surface
of letters which make up the actual spelling of the word. Fig. 3.12 shows these two
levelsfor(English) cats.
ca+N+Pl
cats Lexical
Surface
Figure3.12 Schematic examples of the lexical and surface tapes; the act ual transducers will

ca+N+Pl
Surface
involve intermediate tapes as well.
For ﬁnite-state morphology it’s convenient to view an FST as having two tapes.
Theupperorlexical tape , is composed from characters from one alphabet Σ. The Lexicaltape

DRAFTSection3.5. FSTsforMorphologicalParsing 61
lowerorsurfacetape,iscomposedofcharactersfromanotheralphabet Δ. Inthetwo-
level morphology of Koskenniemi (1983), we allow each arc only to have a single
symbol from each alphabet. We can then combine the two symbol alphabets ΣandΔ
to create a new alphabet, Σ′, which makes the relationshipto FSAs quite clear. Σ′is a
ﬁnite alphabet of complex symbols. Each complex symbol is co mposed of an input-
output pair i:o; one symbol ifrom the input alphabet Σ, and one symbol ofrom an
outputalphabet Δ,thusΣ′⊆Σ×Δ.ΣandΔmayeachalsoincludetheepsilonsymbol ǫ.
Thuswherean FSA acceptsa languagestated overa ﬁnite alpha betofsinglesymbols,
suchasthealphabetofoursheeplanguage:
Σ={b,a,!} (3.2)
anFST deﬁnedthiswayacceptsa languagestated over pairsofsymbols,asin:
Σ′={a:a,b:b,!:!,a:!,a:ǫ, ǫ:!} (3.3)
Intwo-levelmorphology,thepairsofsymbolsin Σ′arealsocalled feasiblepairs . Thus Feasible pair
eachfeasiblepairsymbol a:binthe transduceralphabet Σ′expresseshowthesymbol

afromone tape is mappedto the symbol bonthe othertape. Forexample a:ǫmeans
that anaon the upper tape will correspond to nothingon the lower tape. Just as for
an FSA, we can write regular expressions in the complex alpha betΣ′. Since it’s most
commonforsymbolstomaptothemselves,intwo-levelmorpho logywecallpairslike
a:adefaultpairs ,andjustrefertothembythesingleletter a. Default pair
WearenowreadytobuildanFSTmorphologicalparseroutofou rearliermorpho-
tactic FSAs and lexica by adding an extra “lexical” tape and t he appropriate morpho-

logical features. Fig. 3.13 shows an augmentation of Fig. 3. 3 with the nominal mor-
phological features ( +Sgand+Pl) that correspond to each morpheme. The symbol
ˆ indicates a morpheme boundary , while the symbol #indicates a word boundary .Morpheme
boundary
#
wordboundaryThe morphologicalfeatures map to the empty string ǫor the boundary symbols since
thereisnosegmentcorrespondingtothemonthe outputtape.
q0q1
q
reg-noun
irreg-pl-nounirreg-sg-nounq2
q3q4
q5
q6+N
+N
+N+Pl
+Pl
###^s#+Sg
+Sg
∋∋∋

boundary
#
q0q1
q
reg-noun
q3q4
q5
q6+N
+N
+N+Pl
+Pl
###^s#+Sg
+Sg
∋∋∋
Figure3.13 A schematic transducer for English nominal number inﬂectio nTnum. The sym-
bolsaboveeacharcrepresentelementsofthemorphological parseinthelexicaltape;thesymbols
below each arc represent the surface tape (or the intermedia te tape, to be described later), using
the morpheme-boundary symbol ˆ and word-boundary marker #. The labels on the arcs leaving
q0are schematic, and need tobe expanded byindividual words in the lexicon.
In order to use Fig. 3.13 as a morphological noun parser, it ne eds to be expanded

with all theindividualregularandirregularnounstems, re placingthe labels reg-noun

DRAFT62 Chapter 3. Words& Transducers
etc. In orderto do this we need to update the lexiconfor this t ransducer,so that irreg-
ular plurals like geesewill parse into the correct stem goose +N +Pl . We do this
by allowing the lexicon to also have two levels. Since surfac egeesemaps to lexical
goose, the new lexical entry will be “ g:g o:e o:e s:s e:e ”. Regular forms
are simpler; the two-level entry for foxwill now be “ f:f o:o x:x ”, but by relying
ontheorthographicconventionthat fstandsfor f:fandsoon,wecansimplyreferto
it asfoxand the form for geeseas “g o:e o:e s e ”. Thus the lexicon will look
onlyslightlymorecomplex:
reg-noun irreg-pl-noun irreg-sg-noun
fox g o:eo:ese goose
aardvark m o:iu: ǫs:c emouse
0f
3412
6
7g^s#
ox
ta
o s e
o o s e5+N∋
+N∋
+N∋gcf
ao x
e e s eesoot+Pl
+Sg
+Sg
+Pl
###
Figure3.14 A ﬂeshed-out English nominal inﬂection FST Tlex, expanded from Tnumby re-
placing the three arcs withindividual word stems(only a few sample wordstems are shown).

0f
3412
6
7g^s#
ox
ta
o s e
o o s e5+N∋
+N∋
+N∋gcf
ao x
+Sg
+Sg
+Pl
###
The resulting transducer, shown in Fig. 3.14, will map plura l nouns into the stem
plus the morphological marker +Pl, and singular nouns into the stem plus the mor-
phological marker +Sg. Thus a surface catswill map to cat +N +Pl . This can be
viewedin feasible-pairformatasfollows:
c:c a:a t:t +N: ǫ+Pl:ˆs#
Sincetheoutputsymbolsincludethemorphemeandwordbound arymarkersˆand
#, the lower labels Fig. 3.14 do not correspondexactly to the surface level. Hence we
refer to tapes with these morpheme boundary markers in Fig. 3 .15 asintermediate

tapes;the nextsectionwill showhowtheboundarymarkerisr emoved.

DRAFTSection3.6. TransducersandOrthographicRules 63
f+N+Pl
f #Lexical
Intermediate
Figure3.15 A schematic view of the lexical and intermediate tapes.
3.6 TransducersandOrthographicRules
The method described in the previous section will successfu lly recognize words like
aardvarks andmice. But just concatenating the morphemes won’t work for cases
where there is a spelling change; it would incorrectly rejec t an input like foxesand
accept an input like foxs. We need to deal with the fact that English often requires
spelling changes at morpheme boundaries by introducing spelling rules (orortho- Spellingrule
graphic rules ) This section introduces a number of notations for writing s uch rules
and shows how to implement the rules as transducers. In gener al, the ability to im-
plement rules as a transducer turns out to be useful througho ut speech and language
processing. Here’ssomespellingrules:
Name DescriptionofRule Example
Consonant 1-letterconsonantdoubledbefore -ing/-edbeg/begging
doubling

doubling
Edeletion Silente droppedbefore -ingand-ed make/making
Einsertion e addedafter -s,-z,-x,-ch,-shbefore-s watch/watches
Yreplacement -ychangesto -iebefore-s,-ibefore-edtry/tries
Kinsertion verbsendingwith vowel + -c add-k panic/panicked
Wecanthinkofthesespellingchangesastakingasinputasim pleconcatenationof
morphemes(the “intermediate output” of the lexical transd ucer in Fig. 3.14) and pro-
ducing as output a slightly-modiﬁed (correctly-spelled) c oncatenation of morphemes.

doubling
Fig. 3.16shows in schematic formthe three levelswe are talk ingabout: lexical, inter-
mediate,andsurface. So forexamplewe couldwrite an E-inse rtionrule that performs
the mapping from the intermediate to surface levels shown in Fig. 3.16. Such a rule
mightsaysomethinglike“insertan eonthesurfacetapejust whenthelexicaltapehas
amorphemeendingin x(orz,etc)andthenextmorphemeis -s”. Here’saformalization
oftherule:
ǫ→e/

x
s
z

ˆs# (3.4)
This is the rule notation of Chomsky and Halle (1968); a rule o f the form a→

ǫ→e/

x
s
z

ˆs# (3.4)
b/cdmeans “rewrite aasbwhen it occurs between candd”. Since the symbol
ǫmeans an empty transition, replacing it means inserting som ething. Recall that the
symbol ˆ indicates a morpheme boundary. These boundariesar e deleted by including
thesymbolˆ: ǫinthedefaultpairsforthetransducer;thusmorphemebound arymarkers

DRAFT64 Chapter 3. Words& Transducers
f+N+Pl
f # Intermediate
e SurfaceLexical
Figure3.16 An example of the lexical, intermediate, and surface tapes. Between each pair
of tapes is a two-level transducer; the lexical transducer o f Fig. 3.14 between the lexical and
intermediatelevels,andtheE-insertionspellingrulebet weentheintermediateandsurfacelevels.
The E-insertion spelling rule inserts an eon the surface tape when the intermediate tape has a
morpheme boundary ˆ followedby the morpheme -s.
aredeletedonthesurfacelevelbydefault. The#symbolisas pecialsymbolthatmarks
a word boundary. Thus (3.4) means “insert an eafter a morpheme-ﬁnal x,s, orz, and
beforethemorpheme s”. Fig. 3.17showsanautomatonthat correspondstothisrule .
q q q4 q1q2qother
^:   
other
#
s :e^:s
^:
z, x
##, other#,otherz,s,xz,s,xz,s,x∋∋
∋
∋
Figure3.17 The transducer for the E-insertion rule of (3.4), extended f rom a similar trans-
ducer in Antworth (1990). We additionally need to delete the # symbol from the surface string;

^:   
other
#
s :e^:s
^:
z, x
∋
∋
this can be done either by interpreting the symbol # as the pai r #:ǫ, or by postprocessing the
output toremove wordboundaries.
The idea in building a transducer for a particular rule is to e xpress only the con-
straints necessary for that rule, allowing any other string of symbols to pass through
unchanged. Thisruleisusedtoensurethatwecanonlyseethe ǫ:epairifweareinthe
proper context. So state q0, which models having seen only default pairs unrelated to

therule,isanacceptingstate,asis q1,whichmodelshavingseena z,s,orx.q2models
havingseenthemorphemeboundaryafterthe z,s,orx,andagainisanacceptingstate.
Stateq3modelshaving just seen the E-insertion; it is not an accepti ng state, since the
insertionisonlyallowedif it isfollowedbythe smorphemeandthentheend-of-word
symbol#.
TheothersymbolisusedinFig.3.17tosafelypassthroughanypartsof wordsthat
don’t play a role in the E-insertion rule. othermeans “any feasible pair that is not in

DRAFTSection3.7. CombiningFST LexiconandRules 65
this transducer”. So for example when leaving state q0, we go to q1on thez,s, orx
symbols,rather thanfollowingthe otherarc and stayingin q0. Thesemanticsof other
dependsonwhatsymbolsareonotherarcs;since #ismentionedonsomearcs,itis(by
deﬁnition)not includedin other, andthus, for example,is explicitlymentionedon the
arcfromq2toq0.
Atransducerneedstocorrectlyrejectastringthatapplies therulewhenitshouldn’t.
One possible bad string would have the correct environment f or the E-insertion, but
have noinsertion. State q5is used to ensure that the eis alwaysinserted wheneverthe
environmentis appropriate; the transducer reaches q5only when it has seen an safter
an appropriatemorphemeboundary. If the machine is in state q5and the next symbol
is#, the machine rejects the string (because there is no legal tr ansition on #fromq5).
Fig. 3.18 shows the transition table for the rule which makes the illegal transitions
explicitwiththe“–”symbol.

State\Input s:s x:x z:z ˆ:ǫ ǫ:e#other
q0: 1 1 1 0- 00
q1: 1 1 1 2- 00
q2: 5 1 1 03 00
q3 4 - - -- --
q4 - - - -- 0-
q5 1 1 1 2- -0
Figure3.18 The state-transition table for the E-insertion rule of Fig. 3.17, extended from a
similartransducer inAntworth(1990).
Thenextsectionwill showa traceofthisE-insertiontransd ucerrunningona sam-
pleinputstring.
3.7 CombiningFSTLexiconandRules
Wearenowreadytocombineourlexiconandruletransducersf orparsingandgenerat-
ing. Fig. 3.19shows the architectureof a two-level morphol ogysystem, whetherused

q3 4 - - -- --
q4 - - - -- 0-
q5 1 1 1 2- -0
forparsingor generating. The lexicontransducermapsbetw een the lexicallevel, with
itsstemsandmorphologicalfeatures,andanintermediatel evelthatrepresentsasimple
concatenation of morphemes. Then a host of transducers, eac h representing a single
spellingruleconstraint,all runin parallelso asto mapbet weenthisintermediatelevel
and the surface level. Putting all the spelling rules in para llel is a design choice; we
could also have chosen to run all the spelling rules in series (as a long cascade), if we

slightlychangedeachrule.
Thearchitecturein Fig. 3.19is a two-level cascadeof transducers. Cascading two Cascade
automatameansrunningthem in series with the outputofthe ﬁ rst feedingthe input to
the second. Cascades can be of arbitrary depth, and each leve l might be built out of
many individual transducers. The cascade in Fig. 3.19 has tw o transducers in series:
the transducer mapping from the lexical to the intermediate levels, and the collection

ofparalleltransducersmappingfromtheintermediatetoth esurfacelevel. Thecascade

DRAFT66 Chapter 3. Words& Transducers
fox+N+PL
fox^s#
foxesLEXICON-FST
FST1FSTnorthographic rules
Figure3.19 Generating or parsingwithFSTlexicon andrules
can be run top-down to generate a string, or bottom-up to pars e it; Fig. 3.20 shows a
traceofthesystem accepting themappingfrom fox +N +PL tofoxes.
f+N+Pl
012567 Tlex
Te-insert00012403
Figure3.20 Accepting foxes: The lexicon transducer Tlexfrom Fig. 3.14 cascaded with the
E-insertiontransducer inFig.3.17.
The power of ﬁnite-state transducers is that the exact same c ascade with the same
statesequencesisusedwhenthemachineisgeneratingthesu rfacetapefromthelexical
tape, or when it is parsing the lexical tape from the surface t ape. For example, for
generation, imagine leaving the Intermediate and Surface t apes blank. Now if we run
thelexicontransducer,given fox +N +PL ,itwillproduce foxˆs#ontheIntermediate
tapeviathesamestatesthatitacceptedtheLexicalandInte rmediatetapesinourearlier

example. If we then allow all possible orthographictransdu cers to run in parallel, we
will producethesame surfacetape.
Parsing can be slightly more complicated than generation, b ecause of the problem

DRAFTSection3.7. CombiningFST LexiconandRules 67
ofambiguity . For example, foxescan also be a verb (albeit a rare one, meaning “to Ambiguity
bafﬂe or confuse”), and hence the lexical parse for foxescould be fox +V +3Sg as
wellas fox +N +PL .Howarewetoknowwhichoneistheproperparse? Infact,for
ambiguouscases of this sort, the transduceris not capable o f deciding. Disambiguat-
ingwill require some external evidence such as the surrounding words. Thus foxesis Disambiguating
likelytobeanouninthesequence Isawtwofoxesyesterday ,butaverbinthesequence
Thattricksterfoxesmeeverytime! . Wewilldiscusssuchdisambiguationalgorithmsin
Ch.5andCh.20. Barringsuchexternalevidence,thebestour transducercandoisjust
enumeratethe possible choices; so we can transduce foxˆs#into both fox +V +3SG
andfox +N +PL .
There is a kind of ambiguity that we need to handle: local ambi guity that occurs
during the process of parsing. For example, imagine parsing the input verb assess.

After seeing ass, our E-insertion transducer may propose that the ethat follows is
inserted by the spelling rule (for example, as far as the tran sducer is concerned, we
might have been parsing the word asses). It is not until we don’tsee the #afterasses,
butratherrunintoanother s,that werealizewe havegonedownanincorrectpath.
Becauseofthisnon-determinism,FST-parsingalgorithmsn eedtoincorporatesome
sort of search algorithm. Exercise 7 asks the reader to modif y the algorithm for non-

deterministicFSA recognitionin Fig.2.19inCh. 2todoFST p arsing.
Notethatmanypossiblespurioussegmentationsoftheinput ,suchasparsing assess
asˆaˆsˆsesˆswill beruledoutsince noentryinthelexiconw ill matchthisstring.
Running a cascade, particularly one with many levels, can be unwieldy. Luckily,
we’ve already seen how to compose a cascade of transducers in series into a single
more complex transducer. Transducers in parallel can be com bined by automaton

intersection . TheautomatonintersectionalgorithmjusttakestheCarte sianproductof intersection
thestates,i.e.,foreachstate qiinmachine1andstate qjinmachine2,wecreateanew
stateqij. Then for any input symbol a, if machine 1 would transition to state qnand
machine 2 would transition to state qm, we transition to state qnm. Fig. 3.21 sketches
howthisintersection( ∧) andcomposition( ◦)processmightbecarriedout.
LEXICON-FST
FST1 FSTnLEXICON-FST
FSTA   (=FST1 ^ FST2 ^ ... ^ FSTN)LEXICON-FST
o
FSTA
}intersect/gid99compose

o
FSTA
Figure3.21 Intersection andcomposition of transducers.
Since there are a number of rule →FST compilers, it is almost never necessary in
practice to write an FST by hand. Kaplan and Kay (1994) give th e mathematics that
deﬁnethemappingfromrulestotwo-levelrelations,andAnt worth(1990)givesdetails
of the algorithms for rule compilation. Mohri (1997) gives a lgorithms for transducer

DRAFT68 Chapter 3. Words& Transducers
minimizationanddeterminization.
3.8 Lexicon-FreeFSTs: ThePorterStemmer
While building a transducer from a lexicon plus rules is the s tandard algorithm for
morphologicalparsing,therearesimpleralgorithmsthatd on’trequirethelargeon-line
lexicondemandedbythisalgorithm. Theseareusedespecial lyinInformationRetrieval
(IR) tasks like web search (Ch. 23), in which a query such as a B oolean combination
of relevant keywords or phrases, e.g., ( marsupial OR kangaroo OR koala ) returns Keyword
documentsthathavethesewordsinthem. Sinceadocumentwit htheword marsupials
might not match the keyword marsupial , some IR systems ﬁrst run a stemmer on the
query and document words. Morphological information in IR i s thus only used to
determinethattwo wordshavethesamestem; thesufﬁxesaret hrownaway.
Oneofthemostwidelyusedsuch stemming algorithmsisthesimple andefﬁcient Stemming
Porter (1980) algorithm, which is based on a series of simple cascaded rewrite rules. Porterstemmer

Sincecascadedrewriterulesarejust thesortofthingthatc ouldbeeasilyimplemented
asanFST,thePorteralgorithmalsocanbeviewedasalexicon -freeFSTstemmer(this
idea will be developed further in the exercises (Exercise 6) . The algorithm contains a
seriesofruleslikethese:
ATIONAL→ATE (e.g.,relational →relate)
ING→ǫif stemcontainsvowel(e.g.,motoring →motor)
SSES→SS (e.g.,grasses →grass)
DetailedrulelistsforthePorterstemmer,aswellascode(i nJava,Python,etc.) canbe
foundonMartinPorter’sofﬁcialhomepage;see alsotheorig inalpaper(Porter,1980).

Stemming tends to improve the performance of information re trieval, especially
with smaller documents (the larger the document, the higher the chance the keyword
will occur in the exact form used in the query). But lexicon-f ree stemmers like the
Porteralgorithm,whilesimplerthanfulllexicon-basedmo rphologicalparsers,commit
errorslike thefollowing(Krovetz,1993):
ErrorsofCommission ErrorsofOmission
organization organ European Europe
doing doe analysis analyzes
numerical numerous noise noisy
policy police sparse sparsity

Modernstemmerstendtobemorecomplicated,becauseforexa mplewedon’twant
to stem, e.g., the word Illustrator toillustrate, since the capitalized form Illustrator
tendstorefertothesoftwarepackage. We returntothisissu e inCh. 23.

DRAFTSection3.9. WordandSentenceTokenization 69
3.9 WordandSentenceTokenization
We have focused so far in this chapter on a problem of segmenta tion: how words
can be segmented into morphemes. We turn now to a brief discus sion of the very
related problem of segmenting running text into words and se ntences. This task is
calledtokenization . Tokenization
Word tokenizationmayseem verysimple in a languagelike Eng lishthat separates
words via a special ‘space’ character. As we will see below, n ot every language does
this (Chinese, Japanese, and Thai, for example, do not). But a closer examination
will make it clear that whitespace is not sufﬁcient by itself . Consider the following
sentencesfroma Wall Street JournalandNew YorkTimesartic le,respectively:
Mr. Sherwood said reaction to Sea Containers’ proposal
has been "very positive." In New York Stock Exchange
composite trading yesterday, Sea Containers closed at
$62.625, up 62.5 cents.
‘‘I said, ‘what’re you? Crazy?’ ’’ said Sadowsky. ‘‘I

can’t afford to do that.’’
Segmentingpurelyonwhite-spacewouldproducewordsliket hese:
cents. said, positive." Crazy?
We could address these errors by treating punctuation, in ad dition to whitespace, as a
wordboundary. But punctuationoftenoccurswordinternall y,inexampleslike m.p.h,,
Ph.D.,AT&T,cap’n,01/02/06,andgoogle.com . Similarly,assumingthatwewant 62.5
tobeaword,we’llneedtoavoidsegmentingeveryperiod,sin cethatwillsegmentthis
into 62 and 5. Numberexpressionsintroduceother complicat ionsas well; while com-

masnormallyappearatwordboundaries,commasareusedinsi denumbersinEnglish,
every three digits: 555,500.50 . Languages differ on punctuation styles for numbers;
many continental Europeanlanguageslike Spanish, French, and German, by contrast,
uses a comma to mark the decimal point, and spaces (or sometim es periods) where
Englishputscommas: 555500,50 .
Another useful task a tokenizer can do for us is to expand clit ic contractions that
are marked by apostrophes, for example converting what’reabove to the two tokens

what are, andwe’retowe are. This task is complicated by the fact that apostrophes
arequiteambiguous,sincetheyarealsousedasgenitivemar kers(asin thebook’sover
orinContainers’ above)orasquotativemarkers(asin ‘what’reyou? Crazy?’ above).
Suchcontractionsoccurinotheralphabeticlanguages,inc ludingarticlesandpronouns
inFrench( j’ai,l’homme). While thesecontractionstendtobeclitics, notall cliti csare
marked this way with contraction. In general, then, segment ing and expandingclitics
can be done as part of the process of morphological parsing pr esented earlier in the
chapter.

chapter.
Depending on the application, tokenization algorithms may also tokenize multi-
wordexpressionslike NewYork orrock’n’roll ,whichrequiresamultiwordexpression

DRAFT70 Chapter 3. Words& Transducers
dictionary of some sort. This makes tokenization intimatel y tied up with the task of
detecting names, dates, and organizations,which is called named entity detection and
will bediscussedinCh. 22.
In addition to word segmentation, sentence segmentation is a crucial ﬁrst step inSentencesegmentation
text processing. Segmenting a text into sentences is genera lly based on punctuation.
This is because certain kinds of punctuation (periods, ques tion marks, exclamation
points) tend to mark sentence boundaries. Question marks an d exclamationpointsare
relativelyunambiguousmarkersofsentenceboundaries. Pe riods,ontheotherhand,are
more ambiguous. The period character‘.’ is ambiguousbetwe en a sentence boundary
marker and a marker of abbreviationslike Mr.orInc.The previoussentence that you
justreadshowedanevenmorecomplexcaseofthisambiguity, inwhichtheﬁnalperiod
ofInc.markedbothanabbreviationandthesentenceboundarymarke r. Forthisreason,

sentencetokenizationandwordtokenizationtendto beaddr essedjointly.
In general, sentence tokenization methods work by building a binary classiﬁer
(based on a sequence of rules, or on machine learning) which d ecides if a period is
part of the word or is a sentence boundarymarker. In makingth is decision, it helps to
know if the period is attached to a commonly used abbreviatio n; thus an abbreviation
dictionaryis useful.
State-of-the-art methods for sentence tokenization are ba sed on machine learning

and will be introducedin later chapters. But a useful ﬁrst st ep can still be taken via a
sequence of regular expressions. We introducehere the ﬁrst part; a word tokenization
algorithm. Fig.3.22givesasimplePerlwordtokenizationa lgorithmbasedonGrefen-
stette (1999). The algorithm is quite minimal, designed mai nly to clarify many of the
segmentationissueswe discussedinpreviousparagraphs.
The algorithm consists of a sequence of regular expressions ubstitution rules. The

ﬁrstruleseparatesunambiguouspunctuationlikequestion marksandparentheses. The
next rule segments commas unless they are inside numbers. We then disambiguate
apostrophesandpull offword-ﬁnalclitics. Finally,we dea l with periods,usinga (toy)
abbreviationdictionaryandsomeheuristicsfordetecting otherabbreviations.
The fact that a simple tokenizer can be build with such simple regular expression
patternssuggest that tokenizerslike the one in Fig. 3.22 ca n be easily implementedin

FSTs. Thisisindeedthecase,and(Karttunenetal.,1996)an d(BeesleyandKarttunen,
2003)givedescriptionsofsuchFST-basedtokenizers.
3.9.1 Segmentation inChinese
We mentioned above that some languages, including Chinese, Japanese, and Thai, do
not use spaces to mark potential word-boundaries. Alternat ive segmentation methods
areusedfortheselanguages.
In Chinese, for example,words are composedof characterskn ownashanzi. Each
character generally represents a single morpheme and is pro nounceable as a single

syllable. Wordsonaverageareabout2.4characterslong. As implealgorithmthatdoes
remarkablywellforsegmentingChinese,andisoftenusedas abaselinecomparisonfor
more advancedmethods, is a version of greedy search called maximum matching orMaximummatching
sometimes maxmatch . Thealgorithmrequiresadictionary(wordlist)ofthelang uage.
Themaximummatchingalgorithmstartsbypointingatthebeg inningofastring. It

DRAFTSection3.9. WordandSentenceTokenization 71
#!/usr/bin/perl
$letternumber = "[A-Za-z0-9]";
$notletter = "[ˆA-Za-z0-9]";
$alwayssep = "[\\?!()\";/\\|‘]";
$clitic = "(’|:|-|’S|’D|’M|’LL|’RE|’VE|N’T|’s|’d|’m|’ ll|’re|’ve|n’t)";
$abbr{"Co."} = 1; $abbr{"Dr."} = 1; $abbr{"Jan."} = 1; $abbr {"Feb."} = 1;
while ($line = <>){ # read the next line from standard input
# put whitespace around unambiguous separators
$line =˜ s/$alwayssep/ $& /g;
# put whitespace around commas that aren’t inside numbers
$line =˜ s/([ˆ0-9]),/$1 , /g;
$line =˜ s/,([ˆ0-9])/ , $1/g;
# distinguish singlequotes from apostrophes by
# segmenting off single quotes not preceded by letter
$line =˜ s/ˆ’/$& /g;
$line =˜ s/($notletter)’/$1 ’/g;
# segment off unambiguous word-final clitics and punctuati on
$line =˜ s/$clitic$/ $&/g;
$line =˜ s/$clitic($notletter)/ $1 $2/g;
# now deal with periods. For each possible word
@possiblewords=split(/\s+/,$line);
foreach $word (@possiblewords) {
# if it ends in a period,
if (($word =˜ /$letternumber\./)

&& !($abbr{$word}) # and isn’t on the abbreviation list
# and isn’t a sequence of letters and periods (U.S.)
# and doesn’t resemble an abbreviation (no vowels: Inc.)
&& !($word =˜
/ˆ([A-Za-z]\.([A-Za-z]\.)+|[A-Z][bcdfghj-nptvxz]+\. )$/)) {
# then segment off the period
$word =˜ s/\.$/ \./;
}
# expand clitics
$word =˜s/’ve/have/;
$word =˜s/’m/am/;
print $word," ";
}
print "\n";
}
Figure3.22 A sample English tokenization script, adapted from Grefens tette (1999) and

}
}
}
Palmer(2000). A real scriptwould have a longer abbreviatio n dictionary.
choosesthelongestwordinthedictionarythatmatchesthei nputatthecurrentposition.
Thepointeristhenadvancedpast eachcharacterinthat word . If nowordmatches,the
pointer is instead advanced one character (creating a one-c haracter word). The algo-
rithm is then iteratively applied again starting from the ne w pointer position. To help
visualize this algorithm,Palmer (2000)givesan Englishan alogy,which approximates

theChinesesituationbyremovingthespacesfromtheEnglis hsentence thetabledown
theretoproduce thetabledownthere . Themaximummatchalgorithm(givenalongEn-
glishdictionary)wouldﬁrst matchtheword thetainthe input,sincethat isthelongest
sequence of letters that matches a dictionaryword. Startin g from the end of theta, the
longest matching dictionary word is bled, followed by ownand thenthere, producing
theincorrectsequence thetabledownthere .
The algorithm seems to work better in Chinese (with such shor t words) than in

languageslikeEnglishwithlongwords,asourfailedexampl eshows. EveninChinese,

DRAFT72 Chapter 3. Words& Transducers
however, maxmatch has a number of weakness, particularly wi thunknown words
(words not in the dictionary) or unknown genres (genres which differ a lot from the
assumptionsmadebythedictionarybuilder).
Thereisanannualcompetition(technicallycalleda bakeoff)forChinesesegmen-
tation algorithms. These most successful modern algorithm s for Chinese word seg-
mentation are based on machine learning from hand-segmente dtraining sets. We will
returntothese algorithmsafterwe introduceprobabilisti cmethodsin Ch.5.
3.10 DetectingandCorrectingSpellingErrors
ALGERNON : Butmy ownsweet Cecily,I have never writtenyou any letters .
CECILY: You need hardly remind me of that, Ernest. I remember only to o well
thatIwasforcedtowriteyour lettersforyou. Iwrotealways three timesaweek,
andsometimes oftener.
ALGERNON : Oh,do letme read them, Cecily?
CECILY: Oh, I couldn’t possibly. They would make you far too conceit ed. The

three you wrote me after I had broken off the engagement are so beautiful, and
sobadly spelled, thateven now I canhardly read them without crying alittle.
Oscar Wilde, TheImportance of BeingEarnest
LikeOscarWilde’sfabulousCecily,alotofpeoplewerethin kingaboutspellingduring
the last turn of the century. Gilbert and Sullivan providema ny examples. The Gondo-
liers’Giuseppe,forexample,worriesthathisprivatesecretary is“shakyinhisspelling”
whileIolanthe’s Phyllis can “spell every word that she uses”. Thorstein Ve blen’s ex-

planation(inhis1899classic TheTheoryoftheLeisureClass )wasthatamainpurpose
of the “archaic, cumbrous, and ineffective” English spelli ng system was to be difﬁ-
cult enough to provide a test of membership in the leisure cla ss. Whatever the social
role of spelling, we can certainly agree that many more of us a re like Cecily than like
Phyllis. Estimates for the frequency of spelling errors in h uman typed text vary from
0.05% of the words in carefully edited newswire text to 38% in difﬁcult applications
liketelephonedirectorylookup(Kukich,1992).

In this section we introduce the problem of detecting and cor recting spelling er-
rors. Sincethestandardalgorithmforspellingerrorcorre ctionisprobabilistic,wewill
continue our spell-checking discussion later in Ch. 5 after we deﬁne the probabilistic
noisychannelmodel. Thedetectionandcorrectionofspelli ngerrorsisanintegralpart
of modernword-processorsandsearch engines, and is also im portantin correctinger-
rors inoptical character recognition (OCR), the automatic recognition of machine OCR

or hand-printed characters, and on-line handwriting recognition , the recognition of
humanprintedorcursivehandwritingastheuseriswriting. FollowingKukich(1992),
we candistinguishthreeincreasinglybroaderproblems:
1.non-word error detection: detecting spelling errors that result in non-words
(likegraffeforgiraffe).

DRAFTSection3.10. DetectingandCorrectingSpellingErrors 73
2.isolated-word error correction: correcting spelling errors that result in non-
words, for example correcting graffetogiraffe, but looking only at the word in
isolation.
3.context-dependent error detection and correction: using the context to help
detect and correct spelling errors even if they accidentall y result in an actual
word of English ( real-word errors ). This can happen from typographical er- Real-worderrors
rors (insertion, deletion, transposition) which accident ally produce a real word
(e.g.,thereforthree), or because the writer substituted the wrong spelling of a
homophoneornear-homophone(e.g., dessertfordesert, orpieceforpeace).
Detectingnon-worderrorsisgenerallydonebymarkinganyw ordthatisnotfound
in a dictionary. Forexample,the misspelling graffeabovewouldnot occurin a dictio-
nary. Someearlyresearch(Peterson,1986)hadsuggestedth atsuchspellingdictionar-
ieswouldneedtobekeptsmall,becauselargedictionariesc ontainveryrarewordsthat

resemble misspellings of other words. For example the rare w ordswontorveeryare
also common misspelling of won’tandvery. In practice, Damerau and Mays (1989)
found that while some misspellings were hidden by real words in a larger dictionary,
the larger dictionary proved more help than harm by avoiding marking rare words as
errors. This is especially true with probabilistic spell-c orrection algorithms that can
use word frequencyas a factor. Thusmodernspell-checkings ystems tend to be based
onlargedictionaries.

The ﬁnite-state morphologicalparsersdescribed througho utthis chapter providea
technologyforimplementingsuchlargedictionaries. Bygi vingamorphologicalparser
for a word, an FST parser is inherently a word recognizer. Ind eed, an FST morpho-
logicalparsercanbeturnedintoan evenmoreefﬁcientFSA wo rdrecognizerbyusing
theprojection operationtoextractthelower-sidelanguagegraph. SuchFS Tdictionar-
ies also have the advantageof representingproductivemorp hologylike the English -s
and-edinﬂections. Thisis importantfor dealingwith new legitima te combinationsof

stems and inﬂection . For example, a new stem can be easily add ed to the dictionary,
andthenalltheinﬂectedformsareeasilyrecognized. Thism akesFST dictionarieses-
peciallypowerfulforspell-checkinginmorphologicallyr ichlanguageswherea single
stem canhavetensorhundredsofpossiblesurfaceforms.5
FSTdictionariescanthushelpwithnon-worderrordetectio n. But howabouterror
correction? Algorithms for isolated-word error correctio n operate by ﬁnding words
which are the likely source of the errorful form. For example , correcting the spelling

errorgrafferequiressearchingthroughallpossiblewordslike giraffe,graf,craft,grail,
etc,topickthemostlikelysource. Tochooseamongthesepot entialsourcesweneeda
distancemetric betweenthesourceandthesurfaceerror. Intuitively, giraffeisa more
likely source than grailforgraffe, because giraffeis closer in spelling to graffethan
grailis tograffe. The most powerful way to capture this similarity intuition requires
the use ofprobabilitytheoryand will be discussedin Ch. 4. T healgorithmunderlying
this solution,however,is the non-probabilistic minimum edit distance algorithmthat

we introduceinthenextsection.
5Early spell-checkers, by contrast, allowed any word to have any sufﬁx –thus Unix SPELLaccepts bizarre
preﬁxed words like misclamandantiundoggingly and sufﬁxed words from thelikethehoodandtheness.

DRAFT74 Chapter 3. Words& Transducers
3.11 MinimumEditDistance
Decidingwhichoftwowordsisclosertosomethirdwordinspe llingisaspecialcaseof
thegeneralproblemof stringdistance . Thedistancebetweentwostringsisameasure Stringdistance
ofhowaliketwostringsaretoeachother.
Many important algorithmsfor ﬁnding string distance rely o n some version of the
minimumeditdistance algorithm,namedbyWagnerandFischer(1974)butindepen-Minimum editdistance
dently discoveredby manypeople(summarizedlater, in the H istory section of Ch. 6).
Theminimumeditdistancebetweentwostringsistheminimum numberofeditingop-
erations (insertion, deletion, substitution) needed to tr ansform one string into another.
For example the gap between the words intention andexecution is ﬁve operations,
shownin Fig.3.23asan alignment betweenthe twostrings. Giventwo sequences,an Alignment
alignment isacorrespondancebetweensubstringsofthetwosequences . ThusIaligns
with the empty string, N with E, T with X, and so on. Beneath the aligned strings is

anotherrepresentation;aseriesofsymbolsexpressingan operationlist forconverting
thetopstringintothebottomstring;dfordeletion,sforsu bstitution,iforinsertion.
I N T E *N T I O N
| | | | | | | | | |
*E X E C U T I O N
d s s i s
Figure3.23 Representing the minimum edit distance between two strings as analignment .
The ﬁnal row gives the operation list for converting the top s tring into the bottom string; d for
deletion, s for substitution, ifor insertion.

d s s i s
We can also assign a particular cost or weight to each of these operations. The
Levenshtein distancebetweentwosequencesisthesimplestweightingfa ctorinwhich
eachofthethreeoperationshasacostof1(Levenshtein,196 6).6ThustheLevenshtein
distance between intention andexecution is 5. Levenshteinalso proposedan alternate
versionofhismetricinwhicheachinsertionordeletionhas acostofone,andsubstitu-
tionsarenotallowed(equivalenttoallowingsubstitution ,butgivingeachsubstitutiona
cost of2, since anysubstitutioncan berepresentedbyonein sertionandonedeletion).

Usingthisversion,the Levenshteindistancebetween intention andexecution is8.
The minimum edit distance is computed by dynamic programming . Dynamicdynamicprogramming
programmingisthenameforaclassofalgorithms,ﬁrstintro ducedbyBellman(1957),
thatapplyatable-drivenmethodtosolveproblemsbycombin ingsolutionstosubprob-
lems. Thisclassofalgorithmsincludesthemostcommonly-u sedalgorithmsinspeech
andlanguageprocessing;besidesminimumeditdistance,th eseincludethe Viterbiand
forwardalgorithms(Ch.6),andthe CYKandEarleyalgorithm(Ch.13).

The intuition of a dynamic programming problem is that a larg e problem can be
solved by properly combining the solutions to various subpr oblems. For example,
6Weassumethat thesubstitution of aletter for itself, e.g. s ubstitution tfort, has zero cost.

DRAFTSection3.11. MinimumEditDistance 75
considerthesequenceor“path”oftransformedwordsthatco mprisetheminimumedit
distancebetweenthestrings intention andexecution showninFig.3.24.
Figure3.24 Operation listtransforming intention toexecution (afterKruskal 1983)
Imagine some string (perhaps it is exention) that is in this optimal path (what-
ever it is). The intuition of dynamic programmingis that if exentionis in the optimal
operation-list, then the optimal sequence must also includ e the optimal path from in-
tentiontoexention. Why? If there were a shorter path from intention toexentionthen
we could use it instead, resulting in a shorter overall path, and the optimal sequence
wouldn’tbeoptimal,thusleadingtoa contradiction.
Dynamic programming algorithms for sequence comparison wo rk by creating a
distance matrix with one column for each symbol in the target sequence and one row
foreach symbolin the sourcesequence(i.e., targetalongth e bottom,sourcealongthe

side). For minimum edit distance, this matrix is the edit-distance matrix. Each cell
edit-distance [i,j] contains the distance between the ﬁrst icharacters of the target and
the ﬁrstjcharactersof the source. Each cell can be computedas a simpl e functionof
thesurroundingcells;thusstartingfromthebeginningoft hematrixitispossibletoﬁll
in everyentry. The valuein each cell is computedby takingth e minimumof the three
possiblepathsthroughthematrixwhicharrivethere:
distance [i,j] =min

distance [i−1,j]+ins-cost (targeti−1)


distance [i−1,j−1]+sub-cost (sourcej−1,targeti−1)
distance [i,j−1]+del-cost (sourcej−1))(3.5)
The algorithm itself is summarized in Fig. 3.25, while Fig. 3 .26 shows the results
ofapplyingthealgorithmtothedistancebetween intention andexecution assumingthe
versionof Levenshteindistance in which the insertionsand deletionseach have a cost
of1(ins-cost(·)=del-cost(·)=1),andsubstitutionshaveacostof2(exceptsubstitutio n
ofidenticallettershaszerocost).
Knowingthe minimum edit distance is useful for algorithmsl ike ﬁnding potential

spellingerrorcorrections. But the editdistancealgorith misimportantin anotherway;
with a small change, it can also provide the minimum cost alignment between two
strings. Aligning two strings is useful throughoutspeech a nd languageprocessing. In
speech recognition, minimum edit distance alignment is use d to compute word error
rate in speech recognition (Ch. 9). Alignment plays a role in machine translation, in

DRAFT76 Chapter 3. Words& Transducers
function MIN-EDIT-DISTANCE(target,source)returnsmin-distance
n←LENGTH(target)
m←LENGTH(source)
Createa distance matrix distance[n+1,m+1]
Initializethe zerothrow and column tobe the distance from t he empty string
distance[0,0] = 0
foreachcolumn ifrom1tondo
distance[i,0]←distance[i-1,0] +ins-cost(target[i])
foreachrow jfrom1tomdo
distance[0,j]←distance[0,j-1] +del-cost(source[j])
distance[i,j]←MIN(distance[i−1,j] +ins-cost(targeti−1),
distance[i−1,j−1] +sub-cost(sourcej−1,targeti−1),
distance[i,j−1] +del-cost(sourcej−1))
returndistance[n,m]
Figure3.25 The minimum edit distance algorithm, an example of the class of dynamic pro-
gramming algorithms. The various costs can either be ﬁxed (e .g.∀x,ins-cost (x) =1), or can
be speciﬁc to the letter (to model the fact that some letters a re more likely to be inserted than
others). Weassumethatthereisnocostforsubstitutingale tterforitself(i.e. sub-cost (x,x) =0).
n9891011121110 98

o8789101110 989
i7678910 98910
t656789891011
n5456789101110
e4345678910 9
t3456787898
n2345678787
i1234567678
#0123456789
#execution
Figure3.26 Computation of minimum edit distance between intention andexecution via al-
gorithm of Fig. 3.25, using Levenshtein distance with cost o f 1 for insertions or deletions, 2 for
substitutions. Initalicsare the initialvalues represent ing thedistance from the emptystring.
whichsentencesin aparallelcorpus(acorpuswitha textin t wolanguages)needtobe
matcheduptoeachother.

Inordertoextendtheeditdistancealgorithmtoproduceana lignment,wecanstart
byvisualizinganalignmentasapaththroughtheeditdistan cematrix. Fig.3.27shows
thispathwiththeboldfacedcell. Eachboldfacedcellrepre sentsanalignmentofapair
of letters in the two strings. If two boldfaced cells occur in the same row, there will
be an insertion in going from the source to the target; two bol dfaced cells in the same
columnindicatesadeletion.

DRAFTSection3.12. HumanMorphologicalProcessing 77
Fig.3.27alsoshowstheintuitionofhowtocomputethisalig nmentpath. Thecom-
putationproceedsintwosteps. Intheﬁrststep,weaugmentt heminimumeditdistance
algorithm to store backpointersin each cell. The backpoint erfrom a cell points to the
previous cell (or cells) that were extended from in entering the current cell. We’ve
shown a schematic of these backpointers in Fig. 3.27, after a similar diagram in Gus-
ﬁeld (1997). Some cells have multiple backpointers, becaus e the minimum extension
couldhavecomefrommultiplepreviouscells. Intheseconds tep,we performa back-
trace. In a backtrace, we start from the last cell (at the ﬁnal row an d column), and Backtrace
follow the pointers back through the dynamic programming ma trix. Each complete
path between the ﬁnal cell and the initial cell is a minimumdi stance alignment. Exer-
cise 12 asks you to modify the minimum edit distance algorith m to store the pointers
andcomputethe backtraceto outputanalignment.

n9↓8ւ←↓9ւ←↓10ւ←↓11ւ←↓12↓11↓10↓9ւ8
o8↓7ւ←↓8ւ←↓9ւ←↓10ւ←↓11↓10↓9ւ8←9
i7↓6ւ←↓7ւ←↓8ւ←↓9ւ←↓10↓9ւ8←9←10
t6↓5ւ←↓6ւ←↓7ւ←↓8ւ←↓9ւ8←9←10←↓11
n5↓4ւ←↓5ւ←↓6ւ←↓7ւ←↓8ւ←↓9ւ←↓10ւ←↓11ւ↓10
e4ւ3←4ւ←5←6←7←↓8ւ←↓9ւ←↓10↓9
t3ւ←↓4ւ←↓5ւ←↓6ւ←↓7ւ←↓8ւ7←↓8ւ←↓9↓8
n2ւ←↓3ւ←↓4ւ←↓5ւ←↓6ւ←↓7ւ←↓8↓7ւ←↓8ւ7
i1ւ←↓2ւ←↓3ւ←↓4ւ←↓5ւ←↓6ւ←↓7ւ6←7←8
Figure3.27 Whenenteringa value ineachcell,wemarkwhichof the 3neigh boring cellswe
came from with up to three arrows. After the table is full we co mpute an alignment (minimum

edit path) via a backtrace , starting at the 8in the upper right corner and following the arrows.
Thesequenceofdarkgreycellrepresentsonepossibleminim umcostalignmentbetweenthetwo
strings.
There are various publicly available packages to compute ed it distance, including
UNIXdiff, and the NIST sclite program (NIST, 2005); Minimum edit distance
can also be augmented in various ways. The Viterbi algorithm , for example, is an
extension of minimum edit distance which uses probabilisti c deﬁnitions of the oper-

strings.
ations. In this case instead of computing the “minimum edit d istance” between two
strings, we are interested in the “maximum probability alig nment” of one string with
another. The Viterbi algorithm is crucial in probabilistic tasks like speech recognition
andpart-of-speechtagging.
3.12 HumanMorphologicalProcessing
In this section we brieﬂy survey psycholinguistic studies o n how multi-morphemic
wordsarerepresentedinthemindsofspeakersofEnglish. Co nsidertheword walkand

its inﬂected forms walks, andwalked. Are all three in the human lexicon? Or merely

DRAFT78 Chapter 3. Words& Transducers
walkalong with -edand-s? How about the word happyand its derivedforms happily
andhappiness ? We can imagine two ends of a spectrum of possible representa tions.
Thefull listing hypothesisproposesthatall wordsofa languageare listed i nthe men- Fulllisting
tal lexicon without any internal morphologicalstructure. On this view, morphological
structure is an epiphenomenon, and walk,walks,walked,happy, andhappilyare all
separatelylistedinthelexicon. Thishypothesisisuntena bleformorphologicallycom-
plexlanguageslikeTurkish. The minimumredundancy hypothesissuggeststhatonlyMinimumredundancy
the constituentmorphemesarerepresentedin thelexicon,a ndwhenprocessing walks,
(whetherforreading,listening, or talking)we must access bothmorphemes( walkand
-s)andcombinethem. Thisviewisprobablytoostrictaswell.
Someoftheearliest evidencethatthe humanlexiconreprese ntsat least somemor-
phological structure comes from speech errors , also called slips of the tongue . In

conversationalspeech,speakersoftenmixuptheorderofth ewordsorsounds:
ifyoubreak it it’ll drop
In slips of the tonguecollected by Fromkin and Ratner (1998) and Garrett (1975),
inﬂectionalandderivationalafﬁxescanappearseparately fromtheirstems. Theability
of these afﬁxes to be produced separately from their stem sug gests that the mental
lexiconcontainssomerepresentationofmorphologicalstr ucture.
it’snotonlyuswhohavescrewlooses (for“screwsloose”)
wordsofruleformation(for“rulesofwordformation”)
easyenoughly (for“easilyenough”)

More recent experimental evidence suggests that neither th e full listing nor the
minimum redundancy hypotheses may be completely true. Inst ead, it’s possible that
some but not all morphologicalrelationships are mentally r epresented. Stanners et al.
(1979), for example, found that some derived forms ( happiness ,happily) seem to be
stored separately from their stem ( happy), but that regularlyinﬂected forms( pouring)
are not distinct in the lexicon from their stems ( pour). They did this by using a repe-

tition primingexperiment. In short, repetitionprimingta kes advantageof the fact that
a word is recognized faster if it has been seen before (if it is primed). They found Priming
thatliftingprimedlift, andburnedprimedburn,but forexample selectivedidn’tprime
select. Marslen-Wilson et al. (1994) found that spokenderived words can prime their
stems, but only if the meaning of the derived form is closely r elated to the stem. For
examplegovernment primesgovern, butdepartment does not prime depart. Marslen-

Wilsonet al.(1994)representamodelcompatiblewith their ownﬁndingsasfollows:
department depart govern-al -ure -s
-ing
Figure3.28 Marslen-Wilsonetal.(1994) result: Derivedwords arelink ed totheirstemsonly
ifsemantically related.
In summary, these early results suggest that (at least) prod uctive morphology like
inﬂection does play an online role in the human lexicon. More recent studies have
showneffectsofnon-inﬂectionalmorphologicalstructure onwordreadingtimeaswell,

DRAFTSection3.13. Summary 79
suchasthe morphologicalfamilysize . ThemorphologicalfamilysizeofawordistheMorphological
family size
numberofothermultimorphemicwordsandcompoundsinwhich itappears;thefamily
forfear, for example, includes fearful, fearfully, fearfulness, fearless, fearlessly, f ear-
lessness,fearsome, andgodfearing (accordingtotheCELEXdatabase),foratotalsize
of 9. Baayen and colleagues (Baayen et al., 1997; De Jong et al ., 2002; Moscoso del
PradoMart´ ın et al., 2004a)haveshownthat wordswith a larg ermorphologicalfamily
size arerecognizedfaster. Recentworkhasfurthershownth atwordrecognitionspeed
is effectedby the total amountof information (orentropy)containedbythe morpho-
logicalparadigm(MoscosodelPradoMart´ ınetal.,2004a); entropywillbeintroduced
inthe nextchapter.
3.13 Summary
This chapter introduced morphology , the arena of language processing dealing with
thesubpartsofwords,andthe ﬁnite-statetransducer ,thecomputationaldevicethatis

importantformorphologybutwillalsoplayaroleinmanyoth ertasksinlaterchapters.
We also introduced stemming ,word and sentence tokenization , andspelling error
detection . Here’sasummaryofthemainpointswecoveredabouttheseid eas:
•Morphologicalparsing is the processof ﬁndingthe constituent morphemes in
aword(e.g., cat +N +PL forcats).
•English mainly uses preﬁxesandsufﬁxesto express inﬂectional andderiva-
tionalmorphology.
•Englishinﬂectional morphology is relatively simple and includes person and

numberagreement( -s) and tense markings( -edand-ing). English derivational
morphologyismorecomplexandincludessufﬁxeslike -ationand-ness,andpre-
ﬁxeslike co-andre-. ManyconstraintsontheEnglish morphotactics (allowable
morphemesequences)canberepresentedbyﬁniteautomata.
•Finite-statetransducers areanextensionofﬁnite-stateautomatathatcangener-
ateoutputsymbols. ImportantFSToperationsinclude composition ,projection ,
andintersection .
•Finite-state morphology andtwo-level morphology are applications of ﬁnite-

statetransducersto morphologicalrepresentationandpar sing.
•Automatic transducer-compilers can produce a transducer f or any rewrite rule.
Thelexiconandspellingrulescanbecombinedby composing andintersecting
transducers.
•ThePorter algorithm is a simple and efﬁcient way to do stemming , stripping
off afﬁxes. It is not as accurate as a lexicon-basedtransduc er model but is rele-
vantfortaskslike informationretrieval inwhichexactmorphologicalstructure
isnotneeded.

•Word tokenization can be done by simple regular expressions substitutions or
bytransducers.
•Spelling error detection is normally done by ﬁnding words which are not in a
dictionary;anFST dictionarycanbeusefulforthis.

DRAFT80 Chapter 3. Words& Transducers
•Theminimum edit distance between two strings is the minimum number of
operations it takes to edit one into the other. Minimum edit d istance can be
computedby dynamicprogramming ,whichalsoresultsinan alignment ofthe
twostrings.
Despite the close mathematical similarity of ﬁnite-state t ransducers to ﬁnite-state au-
tomata,thetwomodelsgrewoutofsomewhatdifferenttradit ions. Ch.2describedhow
the ﬁnite automaton grew out of Turing’s (1936) model of algo rithmic computation,
and McCulloch and Pitts ﬁnite-state-like models of the neur on. The inﬂuence of the
Turing machine on the transducer was somewhat more indirect . Huffman(1954) pro-
posed what was essentially a state-transition table to mode l the behaviorof sequential
circuits, basedon the workof Shannon(1938)onan algebraic modelof relaycircuits.
BasedonTuringandShannon’swork,andunawareofHuffman’s work,Moore(1956)

introduced the term ﬁnite automaton for a machine with a ﬁnite number of states
with an alphabet of input symbols and an alphabet of output sy mbols. Mealy (1955)
extendedandsynthesizedtheworkofMooreandHuffman.
TheﬁniteautomatainMoore’soriginalpaper,andtheextens ionbyMealydiffered
in an important way. In a Mealy machine, the input/output sym bols are associated
with the transitions between states. In a Moore machine, the input/output symbols
are associated with the state. The two types of transducers a re equivalent; any Moore

machine can be converted into an equivalent Mealy machine an d vice versa. Further
earlyworkonﬁnite-statetransducers,sequentialtransdu cers,andsoon,wasconducted
bySalomaa(1973),Sch¨ utzenberger(1977).
Earlyalgorithmsformorphologicalparsingusedeitherthe bottom-up ortop-down
methods that we will discuss when we turn to parsing in Ch. 13. An early bottom-up
afﬁx-stripping approach as Packard’s (1973) parser for ancient Greek which itera-
tively strippedpreﬁxesand sufﬁxesoff the inputword, maki ngnote of them, andthen

looked up the remainder in a lexicon. It returned any root tha t was compatible with
the stripped-offafﬁxes. AMPLE (A MorphologicalParser for Linguistic Exploration)
(Weber and Mann, 1981; Weber et al., 1988; Hankamer and Black , 1991) is another
earlybottom-upmorphologicalparser. Hankamer’s(1986)k eCiisaanearlytop-down
generate-and-test oranalysis-by-synthesis morphological parser for Turkish which is
guided by a ﬁnite-state representation of Turkish morpheme s. The program begins
with a morphemethat might match the left edge of the word, and appliesevery possi-

blephonologicalruletoit, checkingeachresultagainstth einput. Ifoneoftheoutputs
succeeds,theprogramthenfollowstheﬁnite-statemorphot acticstothenextmorpheme
andtriesto continuematchingtheinput.
The idea of modeling spelling rules as ﬁnite-state transduc ers is really based on
Johnson’s (1972) early idea that phonological rules (to be d iscussed in Ch. 7) have
ﬁnite-state properties. Johnson’s insight unfortunately did not attract the attention of
thecommunity,andwasindependentlydiscoveredbyRonaldK aplanandMartinKay,

DRAFTSection3.13. Summary 81
ﬁrst in an unpublished talk (Kaplan and Kay, 1981) and then ﬁn ally in print (Kaplan
and Kay, 1994) (see page 13 for a discussion of multiple indep endent discoveries).
Kaplan and Kay’s work was followed up and most fully worked ou t by Koskenniemi
(1983), who described ﬁnite-state morphological rules for Finnish. Karttunen (1983)
built a program called KIMMO based on Koskenniemi’s models. Antworth (1990)
gives many details of two-level morphology and its applicat ion to English. Besides
Koskenniemi’s work on Finnish and that of Antworth (1990) on English, two-level or
other ﬁnite-state models of morphology have been worked out for many languages,
such as Turkish (Oﬂazer, 1993) and Arabic (Beesley, 1996). B arton et al. (1987)
bring up some computational complexity problems with two-l evel models, which are
responded to by Koskenniemi and Church (1988). Readers with further interest in
ﬁnite-state morphology should turn to Beesley and Karttune n (2003). Readers with

furtherinterestincomputationalmodelsofArabicandSemi ticmorphologyshouldsee
Smrˇ z(1998),Kiraz(2001),Habashetal. (2005).
Anumberofpracticalimplementationsofsentencesegmenta tionwereavailableby
the1990s. Summariesofsentencesegmentationhistoryandv ariousalgorithmscanbe
foundinPalmer(2000),Grefenstette(1999),andMikheev(2 003). Wordsegmentation
has been studied especially in Japanese and Chinese. While t he max-matchalgorithm
we describe is very commonly used as a baseline, or when a simp le but accurate al-

gorithm is required, more recent algorithms rely on stochas tic and machine learning
algorithms; see for example such algorithms as Sproat et al. (1996), Xue and Shen
(2003),andTsenget al.(2005a).
Gusﬁeld (1997) is an excellent book coveringeverything you could want to know
aboutstringdistance,minimumedit distance,andrelateda reas.
Students interested in automata theory should see Hopcroft and Ullman (1979) or
Lewis and Papadimitriou(1988). Roche and Schabes (1997b)i s the deﬁnitive mathe-

matical introductionto ﬁnite-state transducersfor langu age applications, and together
with Mohri (1997) and Mohri (2000) give many useful algorith ms such as those for
transducerminimizationanddeterminization.
TheCELEXdictionaryisanextremelyusefuldatabaseformor phologicalanalysis,
containingfullmorphologicalparsesofalargelexiconofE nglish,German,andDutch
(Baayen et al., 1995). Roark and Sproat (2007) is a general in troduction to computa-
tionalissuesin morphologyandsyntax. Sproat(1993)isan o ldergeneralintroduction

tocomputationalmorphology.
Exercises
3.1Give examples of each of the noun and verb classes in Fig. 3.6, and ﬁnd some
exceptionstothe rules.
3.2Extendthetransducerin Fig.3.17todealwith shandch.
3.3Writea transducer(s)fortheK insertionspellingrulein En glish.

DRAFT82 Chapter 3. Words& Transducers
3.4Writea transducer(s)fortheconsonantdoublingspellingr uleinEnglish.
3.5The Soundex algorithm (Odell and Russell, 1922; Knuth, 1973 ) is a method
commonly used in libraries and older Census records for repr esenting people’s
names. Ithastheadvantagethatversionsofthenamesthatar eslightlymisspelled
or otherwise modiﬁed (common, for example, in hand-written census records)
willstillhavethesamerepresentationascorrectly-spell ednames. (e.g.,Jurafsky,
Jarofsky,Jarovsky,andJarovskiall maptoJ612).
a. Keep the ﬁrst letter of the name, and drop all occurrencesof non-initial a,
e, h,i,o,u,w,y
b. Replacethe remainingletterswiththefollowingnumbers:
b,f, p,v→1
c, g,j,k,q,s, x,z→2
d,t→3
l→4
m, n→5
r→6
c. Replace any sequencesof identicalnumbers, onlyif they de rivefrom two
or morelettersthatwere adjacentin the originalname,with a single num-
ber(i.e.,666→6).
d. Converttotheform Letter Digit Digit Digit bydroppingdigits

b,f, p,v→1
d,t→3
l→4
m, n→5
r→6
past thethird(if necessary)orpaddingwithtrailingzeros (ifnecessary).
Theexercise: writea FSTto implementtheSoundexalgorithm .
3.6Read Porter(1980)or see Martin Porter’sofﬁcial homepageo n the Porterstem-
mer. ImplementoneofthestepsofthePorterStemmerasatran sducer.
3.7Write the algorithm for parsing a ﬁnite-state transducer, u sing the pseudo-code
introduced in Chapter 2. You should do this by modifying the a lgorithm ND-
RECOGNIZE inFig.2.19in Chapter2.
3.8Write a program that takes a word and, using an on-line dictio nary, computes

possibleanagramsoftheword,eachofwhichisa legalword.
3.9InFig. 3.17,whyistherea z, s, xarcfromq5toq1?
3.10Computing minimum edit distances by hand, ﬁgure out whether driveis closer
tobriefor todivers, and what the edit distance is. You may use any version of
distancethatyoulike.
3.11Nowimplementaminimumeditdistancealgorithmanduseyour hand-computed
resultstocheckyourcode.
3.12Augmentthe minimumedit distancealgorithmto outputan ali gnment; youwill
needtostore pointersandadda stagetocomputethebacktrac e.

Chapter4
N-grams
Butitmustberecognizedthatthenotion“probabilityofase ntence”isanentirelyuseless
one, under any knowninterpretation of this term.Noam Chomsky (1969, p. 57)
Anytimealinguist leaves the group the recognition rate goe s up.
FredJelinek(thenof the IBMspeech group) (1988)1
Being able to predict the future is not always a good thing. Ca ssandra of Troy had
the gift of foreseeing, but was cursed by Apollo that her pred ictions would never be
believed. Her warnings of the destruction of Troy were ignor ed and to simplify, let’s
just saythatthingsjust didn’tgowell forherlater.

Predictingwordsseems somewhat less fraught,and in this ch apter we take up this
ideaofwordprediction. Whatword,forexample,islikelyto follow:
Pleaseturnyour homework ...
Hopefully most of you concluded that a very likely word is in, or possibly over,
but probably not the. We formalize this idea of word prediction with probabilistic
models called N-gram models , which predict the next word from the previous N−1 N-gram
words. Such statistical modelsof word sequencesare also ca lledlanguage models or Language model

LMs. Computing the probability of the next word will turn out to b e closely related LM
to computing the probability of a sequence of words. The foll owing sequence, for
example,hasa non-zeroprobabilityofappearingina text:
...allof asudden Inotice three guys standing on the sidewal k...
whilethissameset ofwordsina differentorderhasa muchmuc hlowerprobability:
onguys allI of notice sidewalk three a sudden standing the
As we will see, estimators like N-grams that assign a conditional probability to
possible next words can be used to assign a joint probability to an entire sentence.

Whether estimating probabilities of next words or of whole s equences, the N-gram
modelisoneofthemostimportanttoolsin speechandlanguag eprocessing.
N-grams are essential in any task in which we have to identify w ords in noisy,
ambiguousinput. In speechrecognition ,forexample,theinputspeechsoundsarevery
1This wording from his address is as recalled by Jelinek himse lf; the quote didn’t appear in the proceed-
ings (Palmer and Finin, 1990). Some remember a more snappy ve rsion:Every time I ﬁre a linguist the

performance ofthe recognizer improves .

DRAFT84 Chapter 4. N-grams
confusable and many words sound extremely similar. Russell and Norvig (2002) give
an intuition from handwriting recognition for how probabilities of word sequences
canhelp. Inthe movie Take the MoneyandRun ,WoodyAllentries toroba bankwith
a sloppily written hold-upnote that the teller incorrectly reads as “I have a gub”. Any
speech and language processing system could avoid making th is mistake by using the
knowledgethat the sequence“Ihavea gun”isfar moreprobabl ethanthe non-word“I
haveagub”oreven“I havea gull”.
N-gram models are also essential in statistical machine translation . Suppose we
aretranslatinga Chinesesourcesentence
 andas
partoftheprocesswehaveaset ofpotentialroughEnglishtr anslations:
hebriefedtoreportersonthechiefcontentsofthestatemen t
hebriefedreportersonthechiefcontentsofthestatement
hebriefedtoreportersonthemaincontentsofthestatement
he briefedreporterson themain contentsofthestatement
AnN-gram grammar might tell us that, even after controlling for length,briefed

 andas
reporters ismorelikelythan briefedtoreporters ,andmaincontents ismorelikelythan
chief contents . This lets us select the bold-faced sentence above as the mos t ﬂuent
translationsentence,i.e. theonethathasthe highestprob ability.
Inspellingcorrection ,weneedtoﬁndandcorrectspellingerrorslikethefollowin g
(fromKukich(1992))that accidentallyresultinrealEngli shwords:
Theyareleavinginaboutﬁfteen minuetsto goto herhouse.
Thedesign anconstructionofthesystem will takemorethana year.

Since these errors have real words, we can’t ﬁnd them by just ﬂ agging words that
aren’t in the dictionary. But note that in about ﬁfteen minuets is a much less probable
sequencethan in aboutﬁfteen minutes . A spellcheckercan use a probabilityestimator
bothto detecttheseerrorsandtosuggesthigher-probabili tycorrections.
Wordpredictionisalsoimportantfor augmentativecommunication (Newelletal.,augmentativecommunication
1998) systems that help the disabled. People who are unable t o use speech or sign-

language to communicate, like the physicist Steven Hawking , can communicate by
using simple body movements to select words from a menu that a re spoken by the
system. Wordpredictioncanbeusedtosuggestlikelywordsf orthe menu.
Besides these sample areas, N-grams are also crucial in NLP tasks like part-of-
speech tagging ,natural language generation , andword similarity , as well as in
applications from authorship identiﬁcation andsentiment extraction topredictive
textinput systemsforcellphones.

4.1 CountingWordsinCorpora
[upon being asked if there weren’t enough words in theEnglis h language for him]:
“Yes,there are enough, butthey aren’t the right ones.”
James Joyce, reported inBates (1997)

DRAFTSection4.1. CountingWordsin Corpora 85
Probabilitiesarebasedoncountingthings. Beforewetalka boutprobabilities,weneed
to decide what we are goingto count. Counting of thingsin nat ural languageis based
on acorpus(pluralcorpora), an on-line collection of text or speech. Let’s look at Corpus
Corpora two popular corpora, Brown and Switchboard. The Brown corpu sis a 1 million word
collectionofsamplesfrom500writtentextsfromdifferent genres(newspaper,novels,
non-ﬁction, academic, etc.), assembled at Brown Universit y in 1963-64 (Kuˇ cera and
Francis, 1967; Francis, 1979; Francis and Kuˇ cera, 1982). H ow many words are in the
followingBrownsentence?
(4.1)Hesteppedoutintothe hall,wasdelightedto encounterawat er brother.
Example (4.1) has 13 words if we don’t count punctuation mark s as words, 15 if
wecountpunctuation. Whetherwetreatperiod(“ .”),comma(“ ,”),andso onaswords
depends on the task. Punctuation is critical for ﬁnding boun daries of things (com-

mas, periods, colons), and for identifying some aspects of m eaning (question marks,
exclamation marks, quotation marks). For some tasks, like p art-of-speech tagging or
parsing or speech synthesis, we sometimes treat punctuatio n marks as if they were
separatewords.
The Switchboard corpus of telephone conversations between strangers was col-
lected in the early 1990s and contains 2430 conversations av eraging 6 minutes each,
totaling 240 hours of speech and about 3 million words (Godfr ey et al., 1992). Such

corpora of spoken language don’t have punctuation, but do in troduce other complica-
tions with regard to deﬁning words. Let’s look at one utteran ce from Switchboard; an
utterance isthe spokencorrelateofasentence: Utterance
(4.2)Idouhmain-mainlybusinessdataprocessing
This utterance has two kinds of disﬂuencies . The broken-offword main-is called Disﬂuency
afragment . Words like uhandumare called ﬁllersorﬁlled pauses . Should we Fragment
Filledpause considerthesetobewords? Again,itdependsontheapplicat ion. Ifwearebuildingan

automatic dictation system based on automatic speech recog nition, we might want to
eventuallystrip outthedisﬂuencies.
But we also sometimes keep disﬂuencies around. How disﬂuent a person is can
be used to identifythem, or to detect whetherthey are stress ed or confused. Disﬂuen-
cies also often occur with particular syntactic structures , so they may help in parsing
and word prediction. Stolcke and Shriberg (1996) found for e xample that treating uh
as a word improves next-word prediction (why might this be?) , and so most speech
recognitionsystemstreat uhandumaswords.2

Arecapitalizedtokenslike Theyanduncapitalizedtokenslike theythesameword?
Thesearelumpedtogetherinspeechrecognition,whileforp art-of-speech-taggingcap-
italization is retainedas a separate feature. For the rest o f this chapterwe will assume
ourmodelsarenotcase-sensitive.
How about inﬂected forms like catsversuscat? These two words have the same
lemmacatbut are different wordforms. Recall from Ch. 3 that a lemma is a set of
lexical forms having the same stem, the same major part-of-s peech, and the same

word-sense. The wordform is the full inﬂected or derived form of the word. For Wordform
morphologically complex languages like Arabic we often nee d to deal with lemmati-
2Clark and Fox Tree(2002) showed that uhandumhave different meanings. What do you think they are?

DRAFT86 Chapter 4. N-grams
zation.N-grams for speech recognition in English, however, and all t he examples in
thischapter,arebasedonwordforms.
Aswecansee, N-grammodels,andcountingwordsingeneral,requiresthatw edo
thekindoftokenizationortextnormalizationthatwe intro ducedin thepreviouschap-
ter: separating out punctuation, dealing with abbreviatio ns likem.p.h., normalizing
spelling,andso on.
How many words are there in English? To answer this question w e need to dis-
tinguishtypes, the number of distinct words in a corpus or vocabulary size V, from Word type
tokens, the total number Nof running words. The following Brown sentence has 16 Word token
tokensand14types(notcountingpunctuation):
(4.3)Theypicnicked by thepool, thenlayback onthe grass and look ed at the stars.
The Switchboard corpus has about 20,000 wordform types (fro m about 3 million
wordformtokens). Shakespeare’scomplete works have 29,06 6wordformtypes (from
884,647 wordform tokens) (Kuˇ cera, 1992). The Brown corpus has 61,805 wordform

types from 37,851lemma types (from 1 million wordformtoken s). Lookingat a very
largecorpusof583millionwordformtokens,Brownetal.(19 92)foundthatitincluded
293,181differentwordformtypes. Dictionariescan help in givinglemma counts; dic-
tionary entries, or boldface forms are a very rough upper bound on the number of
lemmas (since some lemmas have multiple boldface forms). Th e American Heritage
Dictionary lists 200,000 boldface forms. It seems like the l arger corpora we look at,
the more word types we ﬁnd. In general Gale and Church (1990) s uggest that the vo-

cabulary size (the number of types) grows with at least the sq uare root of the number
oftokens(i.e. V>O(√
N).
Intherestofthischapterwewillcontinuetodistinguishbe tweentypesandtokens,
using“types”to meanwordformtypes.
4.2 Simple(Unsmoothed) N-grams
Let’sstartwithsomeintuitivemotivationsfor N-grams. We assumethatthereaderhas
acquiredsomeverybasicbackgroundinprobabilitytheory. Ourgoalistocomputethe
probabilityof a word wgiven some history h, orP(w|h). Suppose the history his “its

N).
waterissotransparentthat ”andwewanttoknowtheprobabilitythatthenextwordis
the:
P(the|itswaterisso transparentthat ). (4.4)
Howcanwecomputethisprobability? Onewayistoestimateit fromrelativefrequency
counts. Forexample,we couldtakea verylargecorpus,count thenumberoftimeswe
seethe wateris so transparentthat , and countthe numberof timesthis is followedby
the. Thiswouldbeansweringthequestion“Outofthetimeswesaw thehistory h,how
manytimeswasit followedbytheword w”, asfollows:
P(the|itswaterisso transparentthat ) =

DRAFTSection4.2. Simple(Unsmoothed) N-grams 87
C(itswaterisso transparentthatthe )
C(itswaterisso transparentthat )(4.5)
Withalargeenoughcorpus,suchastheweb,wecancomputethe secounts,andes-
timatetheprobabilityfromEq.4.5. Youshouldpausenow,go tothewebandcompute
thisestimateforyourself.
While this method of estimating probabilities directly fro m counts works ﬁne in
many cases, it turns out that even the web isn’t big enough to g ive us good estimates
in most cases. This is because language is creative; new sent ences are created all the
time, and we won’t always be able to count entire sentences. E ven simple extensions
of the examplesentencemay havecountsof zeroon the web (suc has “WaldenPond’s
waterisso transparentthatthe ”).
Similarly,ifwewantedtoknowthejointprobabilityofanen tiresequenceofwords
likeitswaterissotransparent ,wecoulddoitbyasking“outofallpossiblesequences
of 5 words, how many of them are its water is so transparent ?” We would have to

get the count of its water is so transparent , and divide by the sum of the counts of all
possible5wordsequences. Thatseemsratheralot toestimat e!
Forthisreason,we’ll needto introduceclevererwaysofest imatingtheprobability
of a word wgiven a history h, or the probability of an entire word sequence W. Let’s
start with a little formalizing of notation. In order to repr esent the probability of a
particular random variable Xitaking on the value “the”, or P(Xi=“the”), we will use

the simpliﬁcation P(the). We’ll represent a sequence of Nwords either as w1...wn
orwn
1. For the joint probability of each word in a sequence having a particular value
P(X=w1,Y=w2,Z=w3,...,W=wn)we’ll use P(w1,w2,...,wn).
Now how can we computeprobabilitiesof entire sequenceslik eP(w1,w2,...,wn)?
One thing we can do is to decompose this probability using the chain rule of proba-
bility:
P(X1...Xn) =P(X1)P(X2|X1)P(X3|X2
1)...P(Xn|Xn−1
1)
=n
∏
k=1P(Xk|Xk−1
1) (4.6)
Applyingthechainruletowords,weget:
P(wn
1) =P(w1)P(w2|w1)P(w3|w2

orwn
bility:
1)
=n
∏
1) (4.6)
P(wn
1)...P(wn|wn−1
1)
=n
∏
k=1P(wk|wk−1
1) (4.7)
Thechainruleshowsthelinkbetweencomputingthejointpro babilityofasequence
and computing the conditional probability of a word given pr evious words. Eq. 4.7
suggests that we could estimate the joint probability of an e ntire sequence of words
by multiplying together a number of conditional probabilit ies. But using the chain
rule doesn’t really seem to help us! We don’t know any way to co mpute the exact
probability of a word given a long sequence of preceding word s,P(wn|wn−1
1). As we

DRAFT88 Chapter 4. N-grams
said above,we can’t just estimate by countingthe numberof t imes every word occurs
following every long string, because language is creative a nd any particular context
mighthaveneveroccurredbefore!
The intuition of the N-gram model is that instead of computing the probability of
a word given its entire history, we will approximate the history by just the last few
words.
Thebigrammodel, for example, approximates the probability of a word g iven Bigram
all the previous words P(wn|wn−1
1)by using only the conditional probability of the
precedingword P(wn|wn−1). Inotherwords,insteadofcomputingtheprobability
P(the|WaldenPond’swaterisso transparentthat ) (4.8)
we approximateit with theprobability
P(the|that) (4.9)
Whenweuseabigrammodeltopredicttheconditionalprobabi lityofthenextword
we arethusmakingthefollowingapproximation:
P(wn|wn−1
1)≈P(wn|wn−1) (4.10)
Thisassumption that the probabilityof a word dependsonly o n the previousword

P(wn|wn−1
is called a Markovassumption. Markov models are the class of probabilistic mo dels Markov
thatassumethatwecanpredicttheprobabilityofsomefutur eunitwithoutlookingtoo
farintothepast. We cangeneralizethebigram(whichlookso newordintothepast)to
thetrigram(whichlookstwowordsintothepast)andthustot heN-gram(whichlooks N-gram
N−1wordsintothepast).
Thusthegeneralequationforthis N-gramapproximationtotheconditionalproba-
bilityofthenextwordin asequenceis:
P(wn|wn−1
1)≈P(wn|wn−1
n−N+1) (4.11)

P(wn|wn−1
Given the bigram assumption for the probability of an indivi dual word, we can
compute the probability of a complete word sequence by subst ituting Eq. 4.10 into
Eq.4.7:
P(wn
1)≈n
∏
k=1P(wk|wk−1) (4.12)
Howdoweestimatethesebigramor N-gramprobabilities? Thesimplestandmost
intuitiveway to estimate probabilitiesis called Maximum Likelihood Estimation , orMaximumLikelihoodEstimationMLE. We get the MLE estimate for the parameters of an N-gram model by taking MLE
countsfromacorpus,and normalizing themso theylie between0 and1.3Normalize

Eq.4.7:
P(wn
1)≈n
∏
Forexample,to computeaparticularbigramprobabilityofa wordygivena previ-
ousword x, we’llcomputethecountofthe bigram C(xy)andnormalizebythe sumof
all thebigramsthat sharethe sameﬁrst word x:
3For probabilistic models, normalizing means dividing by so metotal count so that the resulting probabili-
ties fall legally between 0 and 1.

DRAFTSection4.2. Simple(Unsmoothed) N-grams 89
P(wn|wn−1) =C(wn−1wn)
∑wC(wn−1w)(4.13)
We can simplify this equation, since the sum of all bigram cou nts that start with a
given word wn−1must be equal to the unigram count for that word wn−1(the reader
shouldtakea momenttobeconvincedofthis):
C(wn−1)(4.14)
Let’s work throughan example using a mini-corpusof three se ntences. We’ll ﬁrst
need to augment each sentence with a special symbol <s>at the beginning of the
sentence, to give us the bigram context of the ﬁrst word. We’l l also need a special
end-symbol </s>.4
<s> I am Sam </s>
<s> Sam I am </s>
<s> I do not like green eggs and ham </s>
Herearethecalculationsforsomeofthebigramprobabiliti esfromthiscorpus
P(I|<s> ) =2
3=.67 P(Sam|<s> ) =1
3=.33P(am|I) =2
3=.67
P(</s>|Sam ) =1
2=0.5P(Sam|am ) =1
2=.5P(do|I) =1
3=.33
Forthegeneralcase ofMLE N-gramparameterestimation:
P(wn|wn−1
n−N+1) =C(wn−1
n−N+1wn)
C(wn−1
n−N+1)(4.15)

3=.67
3=.33
P(wn|wn−1
n−N+1wn)
C(wn−1
Eq.4.15(likeEq. 4.14)estimatesthe N-gramprobabilitybydividingthe observed
frequencyof a particularsequenceby the observedfrequenc yof a preﬁx. This ratio is
calledarelativefrequency . Wesaidabovethatthisuseofrelativefrequenciesasaway Relativefrequency
toestimateprobabilitiesisanexampleofMaximumLikeliho odEstimationorMLE.In
MaximumLikelihoodEstimation,theresultingparameterse tmaximizesthelikelihood
ofthetrainingset Tgiventhemodel M(i.e.,P(T|M)). Forexample,supposetheword
Chineseoccurs400timesinacorpusofamillionwordsliketheBrownc orpus. Whatis

theprobabilitythatarandomwordselectedfromsomeothert extofsayamillionwords
will be the word Chinese? The MLE estimate of its probability is400
1000000or.0004.
Now.0004 is not the best possible estimate of the probabilityof Chineseoccurringin
allsituations;itmightturnoutthatinsomeOTHERcorpusor contextChineseisavery
unlikelyword. Butitistheprobabilitythatmakesit mostlikely thatChinesewilloccur
400 times in a million-word corpus. We will see ways to modify the MLE estimates
slightlytogetbetterprobabilityestimatesinSec. 4.5.

Let’s move on to some examples from a slightly larger corpus t han our 14-word
example above. We’ll use data from the now-defunct Berkeley Restaurant Project,
a dialogue system from the last century that answered questi ons about a database of
4As Chen and Goodman (1998) point out, we need the end-symbol t o make the bigram grammar a true
probability distribution. Withoutanend-symbol,thesent enceprobabilities forallsentences ofagivenlength
would sumto one, and the probability of the whole language wo uld beinﬁnite.

DRAFT90 Chapter 4. N-grams
restaurants in Berkeley, California (Jurafsky et al., 1994 ). Here are some sample user
queries,lowercasedandwithnopunctuation(arepresentat ivecorpusof9332sentences
isonthewebsite):
canyou tellme about anygood cantonese restaurants close by
midpricedthai foodis what i’mlooking for
tellme about chez panisse
canyou give me a listingof the kinds of food thatare availabl e
i’mlooking for a good place toeat breakfast
whenis caffe venezia open during the day
Figure 4.1 shows the bigram counts from a piece of a bigram gra mmar from the
Berkeley Restaurant Project. Note that the majority of the v alues are zero. In fact,
we have chosen the sample words to cohere with each other; a ma trix selected from a
randomset ofsevenwordswouldbeevenmoresparse.
i want to eat chinese food lunch spend
i 5 827 090 0 0 2
want 20 608 1 6 6 5 1
to 20 4 686 2 06 211
eat 0 0 2016 2 42 0
chinese 10 0 0 0 82 1 0
food 150 1501 4 0 0
lunch 20 0 0 0 10 0
spend 10 10 0 0 0 0

Figure4.1 Bigramcounts foreightofthewords(outof V=1446)intheBerkeleyRestaurant
Projectcorpus of 9332 sentences.
Fig. 4.2 shows the bigram probabilities after normalizatio n (dividing each row by
thefollowingunigramcounts):
iwanttoeatchinesefoodlunchspend
253392724177461581093341278
i 0.002 0.33 00.00360 0 0 0.00079
want 0.0022 00.66 0.0011 0.0065 0.0065 0.0054 0.0011
to 0.00083 00.0017 0.28 0.00083 00.0025 0.087
eat 0 0 0.002700.021 0.0027 0.056 0

chinese 0.0063 0 0 0 0 0.52 0.0063 0
food 0.014 00.01400.00092 0.0037 0 0
lunch 0.0059 0 0 0 0 0.00290 0
spend 0.0036 00.00360 0 0 0 0
Figure4.2 Bigram probabilities for eight words in the Berkeley Restau rant Project corpus of
9332 sentences.
Herearea fewotherusefulprobabilities:

DRAFTSection4.3. TrainingandTest Sets 91
P(i|<s> ) =0.25 P(english|want ) =0.0011
P(food|english ) =0.5P(</s>|food ) =0.68
Nowwecancomputetheprobabilityofsentenceslike IwantEnglishfood orIwant
Chinese food by simply multiplying the appropriate bigram probabilitie s together, as
follows:
P(<s> i want english food </s> )
=P(i|<s> )P(want|i )P(english|want )
P(food|english )P(</s>|food )
=.25×.33×.0011×0.5×0.68
= =.000031
Weleaveitasanexerciseforthereadertocomputetheprobab ilityofiwantchinese
food. But that exercise does suggest that we’ll want to think a bit about what kindsof
linguisticphenomenaarecapturedinbigrams. Someofthebi gramprobabilitiesabove
encodesomefactsthatwethinkofasstrictlysyntacticinna ture,likethefactthatwhat
comesafter eatisusuallyanounoranadjective,orthatwhatcomesafter toisusuallya
verb. Othersmightbe more culturalthan linguistic, like th e low probabilityof anyone
askingforadviceonﬁndingEnglishfood.
Although we will generally show bigram models in this chapte r for pedagogical

purposes, note that when there is sufﬁcient training data we are more likely to use
trigrammodels, which condition on the previous two words rather tha n the previous Trigram
word. To computetrigram probabilitiesat the very beginnin gof sentence, we can use
twopseudo-wordsfortheﬁrst trigram(i.e., P(I|<s><s> ).
4.3 TrainingandTestSets
TheN-gram model is a good example of the kind of statistical model s that we will
beseeing throughoutspeechandlanguageprocessing. Thepr obabilitiesofan N-gram
model comefrom the corpusit is trainedon. In general,the pa rametersof a statistical

modelaretrainedonsomesetofdata,andthenweapplythemod elstosomenewdata
in some task (such as speech recognition)and see how well the y work. Of course this
newdataortaskwon’tbetheexactsamedatawe trainedon.
We can formalize this idea of training on some data, and testi ng on some other
data bytalkingaboutthese two datasets as a trainingset andatest set(oratraining Trainingset
Test set corpusand atest corpus ). Thus when using a statistical model of language given

some corpus of relevant data, we start by dividing the data in to training and test sets.
We train the statistical parameters of the model on the train ing set, and then use this
trainedmodeltocomputeprobabilitiesonthetest set.
This training-and-testingparadigm can also be used to evaluate differentN-gram Evaluation
architectures. Suppose we want to compare different langua ge models (such as those
based on N-grams of different orders N, or using the different smoothing algorithms

to be introduced in Sec. 4.5). We can do this by taking a corpus and dividing it into

DRAFT92 Chapter 4. N-grams
a training set and a test set. Then we train the two different N-gram models on the
training set and see which one better models the test set. But what does it mean to
“model the test set”? There is a useful metric for how well a gi ven statistical model
matchesatestcorpus,called perplexity ,introducedonpage95. Perplexityisbasedon
computingtheprobabilityofeachsentenceinthetestset;i ntuitively,whichevermodel
assigns a higherprobabilityto the test set (hencemore accu ratelypredictsthe test set)
isa bettermodel.
Sinceourevaluationmetric isbased ontest set probability ,it’s importantnotto let
thetestsentencesintothetrainingset. Supposewearetryi ngtocomputetheprobability
ofaparticular“test”sentence. Ifourtestsentenceispart ofthetrainingcorpus,wewill
mistakenly assign it an artiﬁcially high probability when i t occurs in the test set. We
callthissituation trainingonthetestset . Trainingonthetestsetintroducesabiasthat

makestheprobabilitiesall looktoohighandcauseshugeina ccuraciesinperplexity.
Inadditionto trainingand test sets, other divisionsofdat a are oftenuseful. Some-
times we need an extra source of data to augment the training s et. Such extra data is
called aheld-out set, because we hold it out from our training set when we train our Held-out set
N-gramcounts. Theheld-outcorpusis thenusedtoset someoth erparameters;forex-
amplewe will see theuse ofheld-outdata toset interpolatio nweightsin interpolated
N-grammodelsinSec.4.6. Finally,sometimesweneedtohavem ultipletestsets. This

happens because we might use a particular test set so often th at we implicitly tune to
itscharacteristics. Thenwewoulddeﬁnitelyneedafreshte stsetwhichistrulyunseen.
In such cases, we call the initial test set the development test set or, devset. We will Developmenttest
discussdevelopmenttest setsagainin Ch.5.
Howdowedivideourdataintotraining,dev,andtestsets? Th ereisatradeoff,since
we want our test set to be as large as possible and a small test s et may be accidentally

unrepresentative. Ontheotherhand,wewantasmuchtrainin gdataaspossible. Atthe
minimum, we would want to pick the smallest test set that give s us enough statistical
power to measure a statistically signiﬁcant difference bet ween two potential models.
In practice, we often just divide our data into 80% training, 10% development, and
10% test. Given a large corpus that we want to divide into trai ning and test, test data
can either be taken from some continuous sequence of text ins ide the corpus, or we

can remove smaller “stripes” of text from randomly selected parts of our corpus and
combinethemintoa test set.
4.3.1 N-gram Sensitivityto the Training Corpus
TheN-gram model, like many statistical models, is very dependen t on the training
corpus. Oneimplicationofthisisthattheprobabilitiesof tenencodeveryspeciﬁcfacts
about a given training corpus. Another implication is that N-grams do a better and
betterjobofmodelingthetrainingcorpusaswe increasethe valueofN.

Wecanvisualizebothofthesefactsbyborrowingthetechniq ueofShannon(1951)
and Miller and Selfridge (1950), of generating random sente nces from different N-
gram models. It’s simplest to visualize how this works for th e unigramcase. Imagine
all the words of English covering the probability space betw een 0 and 1, each word
coveringan interval equal to its frequency. We choose a rand omvalue between 0 and
1, and print out the word whose interval includes the real val ue we have chosen. We

DRAFTSection4.3. TrainingandTest Sets 93
continuechoosingrandomnumbersandgeneratingwordsunti l we randomlygenerate
the sentence-ﬁnal token </s>. The same technique can be used to generate bigrams
byﬁrstgeneratingarandombigramthatstartswith <s>(accordingtoitsbigramprob-
ability),thenchoosingarandombigramtofollowit(again, accordingtoitsconditional
probability),andso on.
To give an intuition for the increasing power of higher-orde rN-grams, Fig. 4.3
shows random sentences generated from unigram, bigram, tri gram, and quadrigram
modelstrainedonShakespeare’sworks.
Unigram
Tohim swallowed confess hear both. Which. Ofsave ontrailfo r are aydevice and rote lifehave
Everyenter now severallyso, let
Hillhe latespeaks; or! a more tolegless ﬁrstyouenter
Arewhere exeunt and sighs have rise excellency tookof.. Sle epknave we. near; vile like
Bigram
Whatmeans, sir. Iconfess she? thenall sorts,he istrim,cap tain.
Whydost stand forththycanopy, forsooth; he isthis palpabl e hitthe KingHenry. Live king. Follow.

Unigram
Bigram
Whatwe, hathgot soshe that I restand sent toscoldand nature bankrupt, nor the ﬁrstgentleman?
Trigram
Sweetprince, Falstaffshall die. Harryof Monmouth’s grave .
Thisshall forbiditshould be branded, ifrenown made it empt y.
Indeed the duke; and had avery good friend.
Fly,andwillridme these news ofprice. Therefore the sadnes s of parting, as theysay, ’tisdone.
Quadrigram
KingHenry.What! I willgoseekthe traitor Gloucester. Exeu nt some of the watch. A great banquet serv’d in;
Willyou not tellme who Iam?
Itcannot be but so.

Trigram
Indeed the short andthe long. Marry, ’tisa noble Lepidus.
Figure4.3 Sentences randomly generated from four N-gram models computed from Shakespeare’s works. All
characters were mapped tolower case and punctuation marks w ere treatedas words. Output was hand-corrected for
capitalizationtoimprove readability.
The longer the context on which we train the model, the more co herent the sen-
tences. In theunigramsentences,thereisnocoherentrelat ionbetweenwords,norany
sentence-ﬁnal punctuation. The bigram sentences have some very local word-to-word

coherence (especially if we consider that punctuation coun ts as a word). The trigram
andquadrigramsentencesarebeginningto looka lot like Sha kespeare. Indeeda care-
ful investigation of the quadrigram sentences shows that th ey look a little too much
like Shakespeare. The words It cannot be but so are directly from King John . This is
because, not to put the knock on Shakespeare, his oeuvre is no t very large as corpora
go (N=884,647,V=29,066), and our N-gram probability matrices are ridiculously

sparse. Thereare V2=844,000,000possiblebigramsalone,andthenumberofpossi-
blequadrigramsis V4=7×1017. Thusoncethegeneratorhaschosentheﬁrstquadri-
gram(Itcannotbebut ),thereareonlyﬁvepossiblecontinuations( that,I,he,thou,and
so);indeedformanyquadrigramsthereis onlyonecontinuatio n.
To get an idea of the dependence of a grammar on its training se t, let’s look at
anN-gram grammar trained on a completely different corpus: the Wall Street Jour-

DRAFT94 Chapter 4. N-grams
nal(WSJ) newspaper. Shakespeare and the Wall Street Journal are both English, so
we might expect some overlap between our N-grams for the two genres. In order to
checkwhetherthisistrue,Fig.4.4showssentencesgenerat edbyunigram,bigram,and
trigramgrammarstrainedon40millionwordsfromWSJ.
Unigram
Months the my and issue of year foreign new exchange’s septem ber were recession ex-
change new endorsed a acquire tosixexecutives
Bigram
Last December through the way to preserve the Hudson corpora tion N. B. E. C. Taylor
would seem to complete the major central planners one point ﬁ ve percent of U. S. E. has
already old M. X. corporation of living on information such a s more frequently ﬁshing to
keep her
Trigram
They alsopoint toninety nine point six billiondollars from two hundred four ohsix three
percent of therates of interest stores as Mexico andBrazilo n market conditions
Figure4.4 Sentences randomly generated from three N-gram models computed from 40 mil-

keep her
Trigram
lion words of the Wall Street Journal , lower-casing all characters and treating punctuation as
words. Output was thenhand corrected for capitalizationto improve readability.
Comparetheseexamplestothepseudo-ShakespeareinFig.4. 3. Whilesuperﬁcially
they both seem to model “English-like sentences” there is ob viouslyno overlapwhat-
soeverin possible sentences, and little if anyoverlapeven in small phrases. Thisstark
difference tells us that statistical models are likely to be pretty useless as predictors if

thetrainingsets andthetest sets areasdifferentasShakes peareandWSJ.
Howshouldwe deal with this problemwhenwe build N-grammodels? In general
weneedtobesuretouseatrainingcorpusthatlookslikeourt estcorpus. Weespecially
wouldn’t choose training and tests from different genresof text like newspaper text,
early English ﬁction, telephone conversations,and web pag es. Sometimes ﬁnding ap-
propriatetrainingtextforaspeciﬁcnewtaskcanbedifﬁcul t;tobuild N-gramsfortext

prediction in SMS (Short Message Service), we need a trainin g corpus of SMS data.
TobuildN-gramsonbusinessmeetings,wewouldneedtohavecorporaof transcribed
businessmeetings.
For general research where we know we want written English bu t don’t have a
domain in mind, we can use a balanced training corpus that inc ludes cross sections
fromdifferentgenres,suchasthe1-million-wordBrowncor pusofEnglish(Francisand
Kuˇ cera,1982)orthe 100-million-wordBritish NationalCo rpus(Leechet al.,1994).

Recent research has also studied ways to dynamically adaptlanguage models to
differentgenres;see Sec.4.9.4.
4.3.2 Unknown Words: Open versus closedvocabulary tasks
Sometimes we have a language task in which we know all the word s that can occur,
and hence we know the vocabulary size Vin advance. The closed vocabulary as- Closedvocabulary
sumption is the assumption that we have such a lexicon, and th at the test set can only
containwordsfromthislexicon. Theclosedvocabularytask thusassumesthereareno
unknownwords.

DRAFTSection4.4. Evaluating N-grams: Perplexity 95
Butofcoursethisisasimpliﬁcation;aswesuggestedearlie r,thenumberofunseen
wordsgrowsconstantly,sowecan’tpossiblyknowinadvance exactlyhowmanythere
are, and we’d like our model to do something reasonable with t hem. We call these
unseen events unknown words, or out of vocabulary (OOV) words. The percentage OOV
ofOOVwordsthatappearin thetest set iscalledthe OOVrate .
Anopenvocabulary systemisonewherewemodelthesepotentialunknownwords OpenVocabulary
in the test set by adding a pseudo-word called <UNK>. We can train the probabilities
oftheunknownwordmodel <UNK>asfollows:
1.Chooseavocabulary (wordlist) whichisﬁxedin advance.
2.Convertinthetrainingsetanywordthatisnotinthisset(anyOOVwor d)tothe
unknownwordtoken <UNK>ina textnormalizationstep.
3.Estimate theprobabilitiesfor <UNK>fromitscountsjust like anyotherregular
wordinthe trainingset.
4.4 Evaluating N-grams: Perplexity
The best way to evaluate the performance of a language model i s to embed it in an

application and measure the total performance of the applic ation. Such end-to-end
evaluationiscalled extrinsicevaluation ,andalsosometimescalled invivoevaluationExtrinsicevaluation
(Sparck Jones and Galliers, 1996). Extrinisic evaluation i s the only way to know if a
particular improvement in a component is really going to hel p the task at hand. Thus
for speech recognition, we can compare the performance of tw o language models by
runningthespeechrecognizertwice,oncewitheachlanguag emodel,andseeingwhich

givesthemoreaccuratetranscription.
Unfortunately, end-to-end evaluation is often very expens ive; evaluating a large
speech recognition test set, for example, takes hours or eve n days. Thus we would
likeametricthatcanbeusedtoquicklyevaluatepotentiali mprovementsinalanguage
model. An intrinsitic evaluation metricis onewhichmeasuresthequalityofa modelIntrinsicevaluation
independent of any application. Perplexity is the most common intrinsic evaluation

metric for N-gram language models. While an (intrinsic) improvement in perplexity
does not guarantee an (extrinsic) improvementin speech rec ognition performance(or
any other end-to-end metric), it often correlates with such improvements. Thus it is
commonly used as a quick check on an algorithm and an improvem ent in perplexity
canthenbeconﬁrmedbyanend-to-endevaluation.
The intuitionof perplexityis that given two probabilistic models, the better model
is the one that has a tighter ﬁt to the test data, or predicts th e details of the test data

better. Wecanmeasurebetterpredictionbylookingatthepr obabilitythemodelassigns
tothe test data;thebettermodelwill assigna higherprobab ilitytothe test data.
Moreformally,the perplexity (sometimescalled PPforshort)ofalanguagemodel Perplexity
ona test set isa functionoftheprobabilitythat thelanguag emodelassignsto thattest
set. For a test set W=w1w2...wN, the perplexity is the probability of the test set,
normalizedbythenumberofwords:

DRAFT96 Chapter 4. N-grams
PP(W) =P(w1w2...wN)−1
N (4.16)
=N√
1
P(w1w2...wN)
We canusethe chainruletoexpandtheprobabilityof W:
PP(W) =N√
N
∏
i=11
P(wi|w1...wi−1)(4.17)
Thus if we are computing the perplexity of Wwith a bigram language model, we
get:
PP(W) =N√
N
∏
i=11
P(wi|wi−1)(4.18)
Note that because of the inversein Eq. 4.17, the higherthe co nditionalprobability
of the word sequence,the lower the perplexity. Thusminimiz ingperplexityis equiva-
lent to maximizing the test set probability accordingto the languagemodel. What we
generallyuseforwordsequenceinEq.4.17orEq.4.18isthee ntiresequenceofwords
insometestset. Sinceofcoursethissequencewillcrossman ysentenceboundaries,we
needtoincludethebegin-andend-sentencemarkers <s>and</s>intheprobability
computation. We also need to include the end-of-sentence ma rker</s>(but not the
beginning-of-sentencemarker <s>)inthetotal countofwordtokens N.
Thereisanotherwaytothinkaboutperplexity:asthe weightedaveragebranching

factorofalanguage. Thebranchingfactorofalanguageisthenumbe rofpossiblenext
wordsthatcan followanyword. Considerthe taskofrecogniz ingthe digitsin English
(zero, one, two,..., nine), given that each of the 10 digits o ccur with equal probability
P=1
10. Theperplexityofthismini-languageisinfact10. Toseeth at,imagineastring
ofdigitsoflength N. By Eq.4.17,the perplexitywill be:
N
= (1
10N
)−1
N
=1
10−1
=10 (4.19)
But nowsupposethatthe numberzerois reallyfrequentandoc curs10times more
oftenthanothernumbers. Nowweshouldexpecttheperplexit ytobelower,sincemost

P=1
N
= (1
10N
)−1
N
=1
10−1
=10 (4.19)
ofthetimethenextnumberwillbezero. Thusalthoughthebra nchingfactorisstill10,

DRAFTSection4.5. Smoothing 97
the perplexityor weighted branchingfactor is smaller. We l eave this calculationas an
exerciseto thereader.
We’ll see in Sec. 4.10 that perplexity is also closely relate d to the information-
theoreticnotionofentropy.
Finally, let’s see an example of how perplexity can be used to compare three N-
grammodels. Wetrainedunigram,bigram,andtrigramgramma rson38millionwords
(includingstart-of-sentencetokens)fromthe Wall Street Journal,using a 19,979word
vocabulary.5We then computedthe perplexityof each of these modelson a te st set of
1.5 million words via Eq. 4.18. The table below shows the perp lexity of a 1.5 million
wordWSJtest set accordingto eachofthese grammars.
Unigram Bigram Trigram
Perplexity 962 170 109
As we see above, the more information the N-gram gives us about the word se-
quence, the lower the perplexity (since as Eq. 4.17 showed, p erplexity is related in-
verselyto thelikelihoodofthe test sequenceaccordingtot hemodel).

Notethatincomputingperplexitiesthe N-grammodel Pmustbeconstructedwith-
out any knowledge of the test set. Any kind of knowledge of the test set can cause
the perplexityto be artiﬁcially low. For example,we deﬁned abovethe closed vocab-
ularytask, in which the vocabulary for the test set is speciﬁed in a dvance. This can
greatlyreducetheperplexity. Aslongasthisknowledgeisp rovidedequallytoeachof
the models we are comparing, the closed vocabulary perplexi ty can still be useful for

comparing models, but care must be taken in interpreting the results. In general, the
perplexityoftwolanguagemodelsisonlycomparableifthey usethesamevocabulary.
4.5 Smoothing
NeverdoI everwant
tohearanotherword!
Thereisn’tone,
Ihaven’theard!
Eliza Doolittle in Alan
Jay Lerner’s My Fair
Lady
Thereisamajorproblemwiththemaximumlikelihoodestimat ionprocesswehave
seen for training the parameters of an N-gram model. This is the problem of sparse
datacausedbythefactthatourmaximumlikelihoodestimatewasb asedonaparticular Sparse data

Lady
set of training data. For any N-gram that occurred a sufﬁcient number of times, we
mighthavea goodestimateof itsprobability. But becausean ycorpusis limited,some
perfectly acceptable English word sequences are bound to be missing from it. This
5More speciﬁcally, Katz-style backoff grammars with Good-T uring discounting trained on 38 million
wordsfromtheWSJ0corpus (LDC,1993),open-vocabulary, us ingthe <UNK>token; seelater sections for
deﬁnitions.

DRAFT98 Chapter 4. N-grams
missing data means that the N-gram matrix for any given training corpus is bound to
have a very large number of cases of putative “zero probabili tyN-grams” that should
really have some non-zero probability. Furthermore, the ML E method also produces
poorestimateswhenthecountsarenon-zerobutstill small.
We need a method which can help get better estimates for these zero or low-
frequencycounts. Zerocountsturnouttocauseanotherhuge problem. The perplexity
metric deﬁned above requires that we compute the probabilit y of each test sentence.
But if a test sentence has an N-gram that neverappearedin the trainingset, the Maxi-
mum Likelihood estimate of the probability for this N-gram, and hence for the whole
test sentence, will be zero! This means that in order to evalu ate our language mod-
els, we need to modify the MLE method to assign some non-zero p robability to any
N-gram,evenonethatwasneverobservedin training.

For these reasons, we’ll want to modify the maximum likeliho od estimates for
computing N-gram probabilities, focusing on the N-gram events that we incorrectly
assumed hadzero probability. We use the term smoothing for such modiﬁcationsthat Smoothing
address the poor estimates that are due to variability in sma ll data sets. The name
comes from the fact that (lookingahead a bit) we will be shavi ng a little bit of proba-
bilitymassfromthehighercounts,andpilingitinsteadont hezerocounts,makingthe
distributiona little lessjagged.

In the next few sections we will introduce some smoothing alg orithms and show
howtheymodifytheBerkeleyRestaurantbigramprobabiliti esinFig. 4.2.
4.5.1 LaplaceSmoothing
One simple way to do smoothing might be just to take our matrix of bigram counts,
before we normalize them into probabilities, and add one to a ll the counts. This algo-
rithmiscalled Laplacesmoothing ,orLaplace’sLaw(Lidstone,1920;Johnson,1932;Laplace
smoothing
Jeffreys,1948). Laplacesmoothingdoesnotperformwellen oughtobeusedinmodern

smoothing
N-gram models, but we begin with it because it introduces many of the concepts that
we willsee in othersmoothingalgorithmsandalsogivesusau sefulbaseline.
Let’sstart withthe applicationofLaplacesmoothingtouni gramprobabilities. Re-
call that the unsmoothed maximum likelihood estimate of the unigram probability of
theword wiisitscount cinormalizedbythetotalnumberofwordtokens N:
P(wi) =ci
N
Laplace smoothing merely adds one to each count (hence its al ternate name add-

P(wi) =ci
N
onesmoothing). Since there are Vwords in the vocabulary, and each one got in- Add-one
cremented, we also need to adjust the denominator to take int o account the extra V
observations. (Whathappenstoour Pvaluesif wedon’tincreasethe denominator?)
PLaplace (wi) =ci+1
N+V(4.20)
Insteadofchangingboththenumeratoranddenominatoritis convenienttodescribe
how a smoothing algorithm affects the numerator, by deﬁning anadjusted count c∗.

DRAFTSection4.5. Smoothing 99
This adjusted count is easier to compare directly with the ML E counts, and can be
turnedintoaprobabilitylikeanMLEcountbynormalizingby N. Todeﬁnethiscount,
sincewe areonlychangingthenumerator,in additionto addi ngonewe’ll alsoneedto
multiplybya normalizationfactorN
N+V:
c∗
i= (ci+1)N
N+V(4.21)
We cannowturn c∗
iintoa probability p∗
ibynormalizingby N.
A related way to view smoothing is as discounting (lowering) some non-zero Discounting
counts in order to get the probability mass that will be assig ned to the zero counts.
Thus instead of referring to the discounted counts c∗, we might describe a smoothing
algorithm in terms of a relative discount dc, the ratio of the discounted counts to the Discount
originalcounts:
dc=c∗
c
Now that we have the intuition for the unigram case, let’s smo oth our Berkeley
Restaurant Project bigrams. Fig. 4.5 shows the add-one smoo thed counts for the bi-
gramsinFig.4.1.
i 6 828 1101 1 1 3
want 31 609 2 7 7 6 2

dc=c∗
c
to 31 5 687 3 17 212
eat 1 1 3117 3 43 1
chinese 21 1 1 1 83 2 1
food 161 1612 5 1 1
lunch 31 1 1 1 21 1
spend 21 21 1 1 1 1
Figure4.5 Add-one smoothed bigram counts for eight of the words (out of V=1446) in the
Berkeley Restaurant Projectcorpus of 9332 sentences.
Fig. 4.6showsthe add-onesmoothedprobabilitiesforthe bi gramsin Fig. 4.2. Re-
call that normalbigramprobabilitiesarecomputedby norma lizingeach rowofcounts
bytheunigramcount:
C(wn−1)(4.22)
For add-one smoothed bigram counts we need to augment the uni gram count by

thenumberoftotalwordtypesin thevocabulary V:
P∗
Laplace(wn|wn−1) =C(wn−1wn)+1
C(wn−1)+V(4.23)
Thus each of the unigram counts given in the previous section will need to be
augmentedby V=1446. TheresultisthesmoothedbigramprobabilitiesinFig .4.6.

DRAFT100 Chapter 4. N-grams
i 0.0015 0.21 0.00025 0.0025 0.00025 0.00025 0.00025 0.00075
want 0.0013 0.00042 0.26 0.00084 0.0029 0.0029 0.0025 0.00084
to 0.00078 0.00026 0.0013 0.18 0.00078 0.00026 0.0018 0.055
eat 0.00046 0.00046 0.0014 0.00046 0.0078 0.0014 0.02 0.00046
chinese 0.0012 0.00062 0.00062 0.00062 0.00062 0.052 0.0012 0.00062
food 0.0063 0.00039 0.0063 0.00039 0.00079 0.002 0.00039 0.00039
lunch 0.0017 0.00056 0.00056 0.00056 0.00056 0.0011 0.00056 0.00056
spend 0.0012 0.00058 0.0012 0.00058 0.00058 0.00058 0.00058 0.00058
Figure4.6 Add-one smoothed bigram probabilities for eight of the word s (out ofV=1446) in the BeRP corpus
of 9332 sentences.
It is often convenient to reconstruct the count matrix so we c an see how much a
smoothing algorithm has changed the original counts. These adjusted counts can be
computedbyEq.4.24. Fig.4.7showsthereconstructedcount s.
c∗(wn−1wn) =[C(wn−1wn)+1]×C(wn−1)
C(wn−1)+V(4.24)

i 3.8 527 0.64 6.4 0.64 0.64 0.64 1.9
want 1.2 0.39 238 0.78 2.7 2.7 2.3 0.78
to 1.9 0.63 3.1 430 1.9 0.634.4 133
eat 0.34 0.34 1 0.34 5.8 1 15 0.34
chinese 0.2 0.098 0.098 0.098 0.098 8.2 0.2 0.098
food 6.9 0.43 6.9 0.43 0.86 2.2 0.43 0.43
lunch 0.57 0.19 0.19 0.19 0.19 0.380.19 0.19
spend 0.32 0.16 0.32 0.16 0.16 0.16 0.16 0.16
Figure4.7 Add-onereconstitutedcountsforeightwords(of V=1446)intheBeRPcorpusof
Notethatadd-onesmoothinghasmadeaverybigchangetothec ounts.C(wantto )

changed from 608 to 238! We can see this in probability space a s well:P(to|want)
decreasesfrom.66intheunsmoothedcaseto.26inthesmooth edcase. Lookingatthe
discountd(the ratio between new and old counts) shows us how strikingl ythe counts
for each preﬁx word have been reduced; the discount for the bi gramwant tois .39,
whilethediscountfor Chinesefood is.10,a factorof10!
The sharp changein countsand probabilitiesoccursbecause too much probability

mass is moved to all the zeros. We could move a bit less mass by a dding a frac-
tionalcountratherthan1(add- δsmoothing;(Lidstone,1920;Johnson,1932;Jeffreys,
1948)),butthismethodrequiresamethodforchoosing δdynamically,resultsinanin-
appropriatediscountformanycounts,andturnsouttogivec ountswithpoorvariances.
For these and other reasons (Gale and Church, 1994), we’ll ne ed better smoothing
methodsfor N-gramslike theoneswe’llsee in thenextsection.

DRAFTSection4.5. Smoothing 101
4.5.2 Good-Turing Discounting
There are a number of much better discountingalgorithms tha t are only slightly more
complex than add-one smoothing. In this section we introduc e one of them, known
asGood-Turing smoothing. The Good-Turingalgorithmwas ﬁrst describedby Good Good-Turing
(1953),whocreditsTuringwiththeoriginalidea.
The intuition of a number of discounting algorithms (Good-T uring,Witten-Bell
discounting , andKneyser-Ney smoothing ) is to use the count of things you’ve seen
onceto help estimate the count of things you’ve never seen . A word or N-gram (or
any event) that occurs once is called a singleton , or ahapax legomenon . The Good- Singleton
Turing intuition is to use the frequencyof singletonsas a re -estimate of the frequency
ofzero-countbigrams.
Let’s formalize the algorithm. The Good-Turingalgorithm i s based on computing
Nc,thenumberof N-gramsthatoccur ctimes. We refertothenumberof N-gramsthat

occurctimesasthe frequencyoffrequency c. So applyingthe ideato smoothingthe
jointprobabilityofbigrams, N0is thenumberofbigramswith count0, N1the number
ofbigramswithcount1(singletons),andsoon. Wecanthinko feachofthe Ncasabin
which stores the number of different N-grams that occur in the training set with that
frequency c. Moreformally:
Nc=∑
x:count(x)=c1 (4.25)
TheMLEcountfor Ncisc. TheGood-Turingintuitionistoestimatetheprobability
ofthingsthatoccur ctimesinthetrainingcorpusbytheMLEprobabilityofthings that

Nc=∑
occurc+1 times in the corpus. So the Good-Turingestimate replacest he MLE count
cforNcwitha smoothedcount c∗thatisa functionof Nc+1:
c∗= (c+1)Nc+1
Nc(4.26)
We can use Eq. 4.26 to replace the MLE counts for all the bins N1,N2, and so on.
Insteadof usingthisequationdirectlyto re-estimatethe s moothedcount c∗forN0, we
usethefollowingequationfortheprobability P∗
GTforthingsthathadzerocount N0,or
whatwe mightcall the missing mass :
P∗
GT(thingswith frequencyzerointraining ) =N1
N(4.27)

Nc(4.26)
P∗
N(4.27)
HereN1isthecountofitemsin bin1,i.e. thatwereseenonceintrain ing,andNis
the total numberof itemswe have seen in training. Eq. 4.27th us givesthe probability
thattheN+1st bigramwesee will beonethatweneversaw intraining. Sho wingthat
Eq.4.27followsfromEq.4.26is leftasExercise8 fortherea der.
TheGood-Turingmethodwas ﬁrst proposedfor estimatingthe populationsofani-
malspecies. Let’sconsideranillustrativeexamplefromth isdomaincreatedbyJoshua
Goodman and Stanley Chen. Suppose we are ﬁshing in a lake with 8 species (bass,

carp, catﬁsh, eel, perch, salmon, trout, whiteﬁsh) and we ha ve seen 6 species with the
following counts: 10 carp, 3 perch, 2 whiteﬁsh, 1 trout, 1 sal mon, and 1 eel (so we
haven’tyetseenthecatﬁshorbass). Whatistheprobability thatthenextﬁshwecatch

DRAFT102 Chapter 4. N-grams
will bea newspecies,i.e.,onethat hada zerofrequencyinou rtrainingset, i.e.,inthis
case eithera catﬁsh ora bass?
The MLE count cof a hitherto-unseen species (bass or catﬁsh) is 0. But Eq. 4. 27
tellsusthat theprobabilityofa newﬁsh beingoneofthese un seenspeciesis3
18, since
N1is3andNis18:
P∗
GT(thingswithfrequencyzeroin training ) =N1
N=3
18(4.28)
What is the probability that the next ﬁsh will be another trou t? The MLE count
for trout is 1, so the MLE estimated probability is1
18. But the Good-Turing estimate
must be lower, since we just stole3
18of our probabilitymass to use on unseen events!
We’ll need to discount the MLE probabilities for trout, perc h, carp, etc. In summary,
the revised counts c∗and Good-Turing smoothed probabilities p∗
GTfor species with
count0(like bassorcatﬁsh)orcount1(liketrout,salmon,o reel)areasfollows:
unseen (bassorcatﬁsh) trout
c 0 1
MLE p p=0
18=01
18
c∗c∗(trout) =2×N2
N1=2×1
3=.67
GTp∗
GTp∗
GT(unseen) =N1
N=3
18=.17p∗
GT(trout) =.67
18=1
27=.037

c 0 1
MLE p p=0
18=01
18
N1=2×1
3=.67
GTp∗
GTp∗
N=3
18=.17p∗
18=1
27=.037
Notethattherevisedcount c∗foreelwasdiscountedfrom c=1.0toc∗=.67,(thus
leaving some probabilitymass p∗
GT(unseen) =3
18=.17 for the catﬁsh and bass). And
since we know there were 2 unknown species, the probability o f the next ﬁsh being
speciﬁcallya catﬁshis p∗
GT(catﬁsh) =1
2×3
18=.085.
Fig. 4.8 gives two examples of the application of Good-Turin g discounting to bi-
gramgrammars,oneontheBeRPcorpusof9332sentences,anda largerexamplecom-
putedfrom22millionwordsfromtheAssociatedPress(AP)ne wswirebyChurchand

2×3
18=.085.
Gale(1991). Forbothexamplestheﬁrstcolumnshowsthecoun tc,i.e.,thenumberof
observedinstancesofabigram. Thesecondcolumnshowsthen umberofbigramsthat
had this count. Thus 449,721of the AP bigrams have a count of 2 . The third column
showsc∗,the Good-Turingre-estimationofthecount.
4.5.3 Someadvanced issues inGood-Turing estimation
Good-Turingestimationassumesthatthedistributionofea chbigramisbinomial(Church
et al., 1991) and assumes we know N0, the number of bigrams we haven’t seen. We

know this because given a vocabulary size of V, the total number of bigrams is V2,
henceN0isV2minusall thebigramswe haveseen.
There are a number of additional complexities in the use of Go od-Turing. For
example, we don’t just use the raw Ncvalues in Eq. 4.26. This is because the re-
estimatec∗forNcdependson Nc+1,henceEq.4.26isundeﬁnedwhen Nc+1=0. Such
zerosoccurquiteoften. Inoursampleproblemabove,forexa mple,since N4=0,how
can we compute N3? One solution to this is called Simple Good-Turing (Gale andSimple
Good-Turing

DRAFTSection4.6. Interpolation 103
AP Newswire Berkeley Restaurant
c (MLE) Ncc∗(GT) c (MLE) Ncc∗(GT)
0 74,671,100,000 0.0000270 0 2,081,496 0.002553
1 2,018,046 0.446 1 5315 0.533960
2 449,721 1.26 2 1419 1.357294
3 188,933 2.24 3 642 2.373832
4 105,668 3.24 4 381 4.081365
5 68,379 4.22 5 311 3.781350
6 48,190 5.19 6 196 4.500000
Figure4.8 Bigram “frequencies of frequencies” and Good-Turing re-es timations for the 22
million AP bigrams from Church and Gale (1991) and from the Be rkeley Restaurant corpus of
Sampson, 1995). In Simple Good-Turing, after we compute the binsNc, but before
we compute Eq. 4.26 from them, we smooth the Nccounts to replace any zeros in the
sequence. The simplest thing is just to replace the value Ncwith a value computed
fromalinearregressionwhichisﬁttomap Nctocinlogspace(seeGaleandSampson
(1995)fordetails):
log(Nc) =a+blog(c) (4.29)
In addition, in practice, the discounted estimate c∗is not used for all counts c.

Large counts (where c>kfor some threshold k) are assumed to be reliable. Katz
(1987)suggestssetting kat 5. Thuswedeﬁne
c∗=cforc>k (4.30)
Thecorrectequationfor c∗whensome kisintroduced(fromKatz(1987))is:
c∗=(c+1)Nc+1
Nc−c(k+1)Nk+1
N1
1−(k+1)Nk+1
N1,for1≤c≤k. (4.31)
Second, with Good-Turing discounting as with any other, it i s usual to treat N-
gramswithlowrawcounts(especiallycountsof1)asiftheco untwere0,i.e.,toapply
Good-Turingdiscountingto theseasiftheywereunseen.
It turns out that Good-Turing discounting is not used by itse lf in discounting N-

N1
grams; it is only used in combination with the backoff and int erpolation algorithms
describedinthenextsections.
4.6 Interpolation
The discounting we have been discussing so far can help solve the problem of zero
frequency N-grams. But there is an additional source of knowledge we can draw on.
If we are tryingto compute P(wn|wn−2wn−1), but we havenoexamplesof a particular

DRAFT104 Chapter 4. N-grams
trigramwn−2wn−1wn,wecaninsteadestimateitsprobabilitybyusingthebigram prob-
abilityP(wn|wn−1). Similarly,ifwedon’thavecountstocompute P(wn|wn−1),wecan
lookto theunigram P(wn).
Therearetwowaystousethis N-gram“hierarchy”, backoffandinterpolation . In Backoff
Interpolation backoff,if we have non-zerotrigramcounts, we rely solely o n the trigram counts. We
only “back off” to a lower order N-gram if we have zero evidence for a higher-order
N-gram. By contrast, in interpolation, we always mix the prob ability estimates from
all theN-gramestimators, i.e., we do a weightedinterpolationof tr igram,bigram,and
unigramcounts.
In simple linear interpolation, we combine differentorder N-grams by linearly in-
terpolating all the models. Thus we estimate the trigram pro babilityP(wn|wn−2wn−1)
bymixingtogethertheunigram,bigram,andtrigramprobabi lities,eachweightedbya
λ:
ˆP(wn|wn−2wn−1) =λ1P(wn|wn−2wn−1)
+λ2P(wn|wn−1)
+λ3P(wn) (4.32)
suchthatthe λssumto 1:
∑
iλi=1 (4.33)

λ:
∑
In a slightly more sophisticated version of linear interpol ation, each λweight is
computedina moresophisticatedway,byconditioningonthe context. Thiswayifwe
have particularly accurate counts for a particular bigram, we assume that the counts
of the trigramsbased on this bigram will be more trustworthy ,so we can make the λs
for those trigrams higher and thus give that trigram more wei ght in the interpolation.
Eq.4.34showstheequationforinterpolationwithcontext- conditionedweights:
ˆP(wn|wn−2wn−1) =λ1(wn−1
n−2)P(wn|wn−2wn−1)
+λ2(wn−1
n−2)P(wn|wn−1)
+λ3(wn−1
n−2)P(wn) (4.34)

+λ2(wn−1
+λ3(wn−1
How are these λvalues set? Both the simple interpolation and conditional i nter-
polationλs are learned from a held-out corpus. Recall from Sec. 4.3 that a held-out Held-out
corpus is an additional training corpus that we use not to set theN-gram counts, but
to set other parameters. In this case we can use such data to se t theλvalues. We
can do this by choosing the λvalues which maximize the likelihood of the held-out
corpus. That is, we ﬁx the N-gram probabilities and then search for the λvalues that

when plugged into Eq. 4.32 give us the highest probability of the held-out set, There
arevariouswaystoﬁndthisoptimalset of λs. Oneway isto use the EMalgorithmto
be deﬁned in Ch. 6, which is an iterative learning algorithm t hat convergeson locally
optimalλs(Baum,1972;Dempsteret al.,1977;JelinekandMercer,198 0).

DRAFTSection4.7. Backoff 105
4.7 Backoff
While simple interpolationis indeed simple to understanda nd implement, it turns out
thatthereareanumberofbetteralgorithms. Oneoftheseisb ackoffN-grammodeling.
The version of backoffthat we describe uses Good-Turingdis countingas well. It was
introduced by Katz (1987), hence this kind of backoff with di scounting is also called
Katzbackoff . InaKatzbackoff N-grammodel,ifthe N-gramweneedhaszerocounts, Katz backoff
weapproximateit bybackingofftothe( N-1)-gram. We continuebackingoffuntilwe
reacha historythathassomecounts:
Pkatz(wn|wn−1
n−N+1)=

P∗(wn|wn−1
n−N+1), ifC(wn
n−N+1)>0
α(wn−1
n−N+1)Pkatz(wn|wn−1
n−N+2),otherwise.
(4.35)
Equation4.35showsthatthe Katz backoffprobabilityforan N-gramjust relieson
the (discounted)probability P∗if we’ve seen this N-gram before (i.e. if we have non-
zerocounts). Otherwise,werecursivelybackofftotheKatz probabilityfortheshorter-
history(N-1)-gram. We’ll deﬁne thediscountedprobability P∗, the normalizingfactor

n−N+1)=

n−N+1)>0
α(wn−1
(4.35)
α,andotherdetailsaboutdealingwithzerocountsinSec.4.7 .1. Basedonthesedetails,
the trigramversionof backoffmightberepresentedas follo ws(whereforpedagogical
clarity, since it’s easy to confuse the indices wi,wi−1and so on, we refer to the three
wordsina sequenceas x,y, zinthat order):
Pkatz(z|x,y) =

P∗(z|x,y), ifC(x,y,z)>0
α(x,y)Pkatz(z|y),else ifC(x,y)>0
P∗(z), otherwise.(4.36)
Pkatz(z|y) ={
P∗(z|y),ifC(y,z)>0
α(y)P∗(z),otherwise.(4.37)
Katz backoff incorporates discounting as an integral part o f the algorithm. Our


previousdiscussions of discountingshowed how a methodlik e Good-Turingcould be
usedtoassignprobabilitymasstounseenevents. Forsimpli city,weassumedthatthese
unseen events were all equally probable, and so the probabil ity mass got distributed
evenly among all unseen events. Katz backoff gives us a bette r way to distribute the
probability mass among unseen trigram events, by relying on information from uni-
grams and bigrams. We use discounting to tell us how much tota l probabilitymass to

set aside for all the eventswe haven’tseen andbackoffto tel l us howto distributethis
probability.
Discounting is implemented by using discounted probabilit iesP∗(·)rather than
MLEprobabilities P(·)inEq.4.35andEq.4.37.
Whydoweneeddiscountsand αvaluesinEq.4.35andEq.4.37? Whycouldn’twe
just have threesets of MLE probabilitieswithoutweights? B ecause without discounts
andαweights, the result of the equation would not be a true probab ility! The MLE

DRAFT106 Chapter 4. N-grams
estimates of P(wn|wn−1
n−N+1)are true probabilities; if we sum the probability of all wi
overagiven N-gramcontext,we shouldget1:
∑
iP(wi|wjwk) =1 (4.38)
But if that is the case, if we use MLE probabilities but back of f to a lower order
model when the MLE probability is zero, we would be adding ext ra probability mass
intotheequation,andthetotal probabilityofawordwouldb e greaterthan1!
Thus any backoff language model must also be discounted. The P∗is used to
discount the MLE probabilities to save some probability mas s for the lower order N-
grams. The αis used to ensure that the probability mass from all the lower order
N-gramssumsuptoexactlytheamountthat we savedbydiscount ingthehigher-order
N-grams. We deﬁne P∗as the discounted ( c∗) estimate of the conditional probability
ofanN-gram,(andsave PforMLEprobabilities):
P∗(wn|wn−1
n−N+1) =c∗(wn
n−N+1)
c(wn−1
n−N+1)(4.39)
Because on averagethe (discounted) c∗will be less than c, this probability P∗will

n−N+1)
c(wn−1
beslightlylessthantheMLEestimate,whichis
c(wn
n−N+1)
c(wn−1
n−N+1)
This will leave some probability mass for the lower order N-grams which is then
distributed by the αweights; details of computing αare in Sec. 4.7.1. Fig. 4.7 shows
theKatzbackoffbigramprobabilitiesforour8samplewords ,computedfromtheBeRP
corpususingtheSRILMtoolkit.
i 0.0014 0.326 0.00248 0.00355 0.000205 0.0017 0.00073 0.000489
want 0.00134 0.00152 0.656 0.000483 0.00455 0.00455 0.00384 0.000483

c(wn
n−N+1)
c(wn−1
n−N+1)
to 0.000512 0.00152 0.00165 0.284 0.000512 0.0017 0.00175 0.0873
eat 0.00101 0.00152 0.00166 0.00189 0.0214 0.00166 0.0563 0.000585
chinese 0.00283 0.00152 0.00248 0.00189 0.000205 0.519 0.00283 0.000585
food 0.0137 0.00152 0.0137 0.00189 0.000409 0.00366 0.00073 0.000585
lunch 0.00363 0.00152 0.00248 0.00189 0.000205 0.00131 0.00073 0.000585
spend 0.00161 0.00152 0.00161 0.00189 0.000205 0.0017 0.00073 0.000585
Figure4.9 Good-Turingsmoothedbigramprobabilitiesforeightwords (ofV=1446)intheBeRPcorpusof9332

sentences, computing byusing SRILM,with k=5and counts of 1replaced by0.
4.7.1 Advanced: Detailsofcomputing Katzbackoff αandP∗
Inthissectionwegivetheremainingdetailsofthecomputat ionofthediscountedprob-
abilityP∗andthebackoffweights α(w).
We begin with α, which passes the left-over probability mass to the lower or der
N-grams. Let’srepresentthetotalamountofleft-overproba bilitymassbythefunction

DRAFTSection4.8. PracticalIssues: ToolkitsandDataFormats 10 7
β, a function of the ( N-1)-gram context. For a given ( N-1)-gram context, the total
left-overprobabilitymass can be computedby subtractingf rom 1 the total discounted
probabilitymassforall N-gramsstartingwiththatcontext:
β(wn−1
n−N+1) =1−∑
wn:c(wn
n−N+1)>0P∗(wn|wn−1
n−N+1) (4.40)
This gives us the total probability mass that we are ready to d istribute to all ( N-
1)-gram (e.g., bigrams if our original model was a trigram). Each individual ( N-1)-
gram (bigram)will only get a fraction of this mass, so we need to normalize βby the
total probability of all the ( N-1)-grams (bigrams) that begin some N-gram (trigram)
whichhaszerocount. Theﬁnalequationforcomputinghowmuc hprobabilitymassto
distributefroman N-gramto an( N-1)-gramisrepresentedbythefunction α:
α(wn−1
n−N+1) =β(wn−1
n−N+1)
∑wn:c(wn
n−N+1)=0Pkatz(wn|wn−1
n−N+2)
=1−∑wn:c(wn
n−N+1)
1−∑wn:c(wn
n−N+2)(4.41)

α(wn−1
n−N+1)
∑wn:c(wn
n−N+2)
n−N+1)
Note that αis a function of the preceding word string, that is, of wn−1
n−N+1; thus
the amount by which we discount each trigram ( d), and the mass that gets reassigned
to lower order N-grams (α) are recomputed for every ( N-1)-gram that occurs in any
N-gram.
We only need to specify what to do when the countsof an ( N-1)-gramcontext are
0,(i.e.,when c(wn−1
n−N+1) =0)andourdeﬁnitioniscomplete:
n−N+1) =Pkatz(wn|wn−1
n−N+2) ifc(wn−1
n−N+1) =0 (4.42)
and
n−N+1) =0 if c(wn−1
n−N+1) =0 (4.43)
and
β(wn−1
n−N+1) =1 if c(wn−1
n−N+1) =0 (4.44)

N-gram.
and
and
β(wn−1
4.8 PracticalIssues: ToolkitsandDataFormats
Let’s now examine how N-gram language models are represented. We represent and
compute language model probabilities in log format, in orde r to avoid underﬂow and
also to speed up computation. Since probabilities are (by de ﬁnition) less than 1, the
moreprobabilitieswe multiplytogetherthesmaller the pro ductbecomes. Multiplying

DRAFT108 Chapter 4. N-grams
enoughN-grams together would result in numerical underﬂow. By usin g log prob-
abilities instead of raw probabilities, the numbers are not as small. Since adding in
log spaceis equivalentto multiplyingin linear space, we co mbinelog probabilitiesby
adding them. Besides avoiding underﬂow,addition is faster to compute than multipli-
cation. Since wedoallcomputationandstoragein logspace, if weeverneedtoreport
probabilitieswejust takethe expofthelogprob:
p1×p2×p3×p4=exp(logp1+logp2+logp3+logp4) (4.45)
BackoffN-gram language models are generally stored in ARPA format . AnN-
gram in ARPA format is an ASCII ﬁle with a small header followe d by a list of all
thenon-zero N-gramprobabilities(alltheunigrams,followedbybigrams ,followedby
trigrams, and so on). Each N-gram entry is stored with its discounted log probability
(in log10format) and its backoff weight α. Backoff weights are only necessary for
N-gramswhich forma preﬁxof a longer N-gram,so no αis computedforthe highest

orderN-gram(inthiscasethetrigram)or N-gramsendingintheend-of-sequencetoken
<s>. Thusfora trigramgrammar,theformatofeach N-gramis:
unigram: log p∗(wi) wi logα(wi)
bigram: log p∗(wi|wi−1)wi−1wilogα(wi−1wi)
trigram: log p∗(wi|wi−2,wi−1)wi−2wi−1wi
\data\
ngram 1=1447
ngram 2=9420
ngram 3=5201
\1-grams:\
-0.8679678 </s>
-99 <s> -1.068532
-4.743076 chow-fun -0.1943932
-4.266155 fries -0.5432462
-3.175167 thursday -0.7510199
-1.776296 want -1.04292
...
\2-grams:\
-0.6077676 <s> i -0.6257131
-0.4861297 i want 0.0425899

\data\
...
-2.832415 to drink -0.06423882
-0.5469525 to eat -0.008193135
-0.09403705 today </s>
...
\3-grams:\
-2.579416 <s> i prefer
-1.148009 <s> about fifteen
-0.4120701 to go to
-0.3735807 me a list
-0.260361 at jupiter </s>
-0.260361 a malaysian restaurant
...
\end\
Figure4.10 ARPA format for N-grams, showing some sample N-grams. Each is represented
byalogprob, thewordsequence, w1...wn,followedbythelogbackoff weight α. Notethatno α
is computed for the highest-order N-gram or for N-grams ending in <s>.
Fig. 4.8 showsan ARPA formattedLM ﬁle with selected N-gramsfrom the BeRP

...
...
\end\
corpus. Given one of these trigrams, the probability P(z|x,y)for the word sequence
x,y,zcanbecomputedasfollows(repeatedfrom(4.37)):

DRAFTSection4.9. AdvancedIssuesinLanguageModeling 109

P∗(z), otherwise.(4.46)
α(y)P∗(z),otherwise.(4.47)
Toolkits: There are two commonly used available toolkits for building language
models, the SRILM toolkit (Stolcke, 2002) and the Cambridge -CMU toolkit (Clark-
son and Rosenfeld, 1997). Both are publicly available, and h ave similar functionality.
In training mode, each toolkit takes a raw text ﬁle, one sente nce per line with words
separatedby white-space, andvariousparameterssuch as th e orderN, the type of dis-
counting(Good-TuringorKneser-Ney,discussedinSec.4.9 .1),andvariousthresholds.
The output is a languagemodel in ARPA format. In perplexityo r decodingmode, the
toolkits take a language model in ARPA format, and a sentence or corpus, and pro-
duce the probability and perplexity of the sentence or corpu s. Both also implement

manyadvancedfeaturestobediscussedlaterinthischapter andin followingchapters,
includingskip N-grams,wordlattices,confusionnetworks,and N-grampruning.
4.9 AdvancedIssuesin LanguageModeling
4.9.1 Advanced Smoothing Methods: Kneser-Ney Smoothing
Inthissectionwegiveabriefintroductiontothemostcommo nlyusedmodern N-gram
smoothingmethod,theinterpolated Kneser-Ney algorithm. Kneser-Ney
Kneser-Neyhasitsrootsinadiscountingmethodcalled absolutediscounting . Ab-

solute discounting is a much better method of computing a rev ised count c∗than the
Good-TuringdiscountformulawesawinEq.4.26,basedonfre quencies-of-frequencies.
To get the intuition, let’s revisit the Good-Turingestimat es of the bigram c∗extended
fromFig.4.8andreformattedbelow:
c (MLE) 0 123456789
c∗(GT)0.0000270 0.4461.262.243.244.225.196.217.248.25
The astute reader may have noticed that except for the re-est imated counts for 0
and 1, all the other re-estimated counts c∗could be estimated pretty well by just sub-

tracting 0.75 from the MLE count c!Absolute discounting formalizes this intuition,Absolutediscounting
by subtracting a ﬁxed (absolute) discount dfrom each count. The intuition is that we
have good estimates already for the high counts, and a small d iscountdwon’t affect
them much. It will mainly modify the smaller counts, for whic h we don’t necessarily
trust the estimate anyway. The equation for absolute discou nting applied to bigrams
(assuminga propercoefﬁcient αonthebackofftomakeeverythingsumto one)is:

DRAFT110 Chapter 4. N-grams
Pabsolute(wi|wi−1) ={C(wi−1wi)−D
C(wi−1),ifC(wi−1wi)>0
α(wi)P(wi),otherwise.(4.48)
In practice, we might also want to keep distinct discount val uesDfor the 0 and 1
counts.
Kneser-Ney discounting (Kneser and Ney, 1995) augments absolute discounting
with a more sophisticated way to handle the backoff distribu tion. Consider the job of
predicting the next word in this sentence, assuming we are ba cking off to a unigram
model:
Ican’tsee withoutmyreading .
Theword glassesseemsmuchmorelikelytofollowherethantheword Francisco .
ButFrancisco is in fact more common, so a unigram model will prefer it to glasses.
We would like to capture the intuition that although Francisco is frequent, it is only
frequent after the word San, i.e. in the phrase San Francisco . The word glasseshas a
muchwiderdistribution.
Thus instead of backing off to the unigram MLE count (the numb er of times the
wordwhasbeenseen),wewanttouseacompletelydifferentbackoff distribution! We

wantaheuristicthatmoreaccuratelyestimatesthenumbero ftimeswemightexpectto
see word win a new unseencontext. The Kneser-Neyintuitionis to base o ur estimate
onthenumberofdifferentcontextswordwhasappearedin . Wordsthathaveappeared
inmorecontextsaremorelikelytoappearinsomenewcontext aswell. Wecanexpress
thisnewbackoffprobability,the“continuationprobabili ty”,asfollows:
PCONTINUATION (wi) =|{wi−1:C(wi−1wi)>0}|
∑wi|{wi−1:C(wi−1wi)>0}|(4.49)
The Kneser-Ney backoff intuition can be formalizedas follo ws (again assuming a

propercoefﬁcient αonthe backofftomakeeverythingsumtoone):
PKN(wi|wi−1) =

C(wi−1wi)−D
C(wi−1), ifC(wi−1wi)>0
α(wi)|{wi−1:C(wi−1wi)>0}|
∑wi|{wi−1:C(wi−1wi)>0}|otherwise.(4.50)
Finally, it turns out to be better to use an interpolated rather than backoffform
of Kneser-Ney. While simple linearinterpolation is generally not as successful as
Katz backoff, it turns out that more powerful interpolated m odels, such as interpo-
lated Kneser-Ney, work better than their backoff version. Interpolated Kneser-NeyInterpolated
Kneser-Ney


discountingcanbecomputedwithanequationlikethefollow ing(omittingthecompu-
tationofβ):
PKN(wi|wi−1) =C(wi−1wi)−D
C(wi−1)+β(wi)|{wi−1:C(wi−1wi)>0}|
∑wi|{wi−1:C(wi−1wi)>0}|(4.51)
Aﬁnalpracticalnote: itturnsoutthatanyinterpolationmo delcanberepresentedas
abackoffmodel,hencestoredinARPAbackoffformat. Wesimp lydotheinterpolation

DRAFTSection4.9. AdvancedIssuesinLanguageModeling 111
when we build the model, so the ‘bigram’ probability stored i n the backoff format is
really‘bigramalreadyinterpolatedwithunigram’.
4.9.2 Class-basedN-grams
Theclass-basedN-gram orclusterN-gram isavariantofthe N-gramthatusesinfor-Class-basedN-gram
Cluster N-gram mation about word classes or clusters. Class-based N-gramscan be useful for dealing
with sparsity in the training data. Suppose for a ﬂight reser vation system we want to
computetheprobabilityofthebigram toShanghai ,butthisbigramneveroccursinthe
training set. Instead, our training data has to London ,to Beijing , andto Denver . If we
knew that these were all cities, and assuming Shanghai does appear in the training set
inothercontexts,wecouldpredictthelikelihoodofacityf ollowing from.
Therearemanyvariantsofcluster N-grams. Thesimplestoneissometimesknown
asIBM clustering , after its originators(Brown et al., 1992). IBM clustering is a kind IBMclustering

ofhard clustering , in whicheach wordcan belongto onlyone class. Themodelest i-
matestheconditionalprobabilityofaword wibymultiplyingtwofactors: theprobabil-
ity of the word’s class cigiven the precedingclasses (based on an N-gram of classes),
andtheprobabilityof wigivenci. HereistheIBM modelinbigramform:
P(wi|wi−1)≈P(ci|ci−1)×P(wi|ci)
If we had a training corpus in which we knew the class for each w ord, the maxi-
mum likelihoodestimate (MLE)of the probabilityof the word given the class and the

probabilityoftheclassgiventhe previousclasscouldbeco mputedasfollows:
P(w|c) =C(w)
C(c)
P(ci|ci−1) =C(ci−1ci)
∑cC(ci−1c)
ClusterN-gramsaregenerallyusedintwoways. Indialogsystems(Ch. 24),weof-
tenhand-designdomain-speciﬁcwordclasses. Thusforanai rlineinformationsystem,
we might use classes like CITYNAME ,AIRLINE,DAYOFWEEK , orMONTH. In other
cases,wecanautomaticallyinducetheclassesbyclusterin gwordsinacorpus(Brown
et al., 1992). Syntactic categories like part-of-speechta gs don’t seem to work well as

C(c)
classes(Niesleret al.,1998).
Whether automatically induced or hand-designed, cluster N-grams are generally
mixedwithregularword-based N-grams.
4.9.3 Language Model Adaptationand Usingthe Web
Oneofthemostexcitingrecentdevelopmentsinlanguagemod elingislanguagemodel
adaptation . Thisisrelevantwhenwe haveonlya small amountofin-domai ntraining Adaptation
data, but a large amount of data from some other domain. We can train on the larger
out-of-domain dataset and adapt our models to the small in-d omain set. (Iyer and

Ostendorf,1997,1999a,1999b;BacchianiandRoark,2003;B acchianiet al.,2004).

DRAFT112 Chapter 4. N-grams
An obviouslarge data source for this type of adaptationis th e web. Indeed,use of
the web doesseem to be helpfulin languagemodeling. The simp lest way to applythe
webtoimprove,say,trigramlanguagemodelsistousesearch enginestogetcountsfor
w1w2w3andw1w2w3,andthencompute:
ˆpweb=cweb(w1w2w3)
cweb(w1w2)(4.52)
We can then mix ˆ pwebwith a conventional N-gram (Berger and Miller, 1998; Zhu
and Rosenfeld, 2001). We can also use more sophisticated com bination methods that
make use of topic or class dependencies,to ﬁnd domain-relev antdata on the web data
(Bulykoet al.,2003).
In practice it is difﬁcult or impossible to download every pa ge from the web in
orderto compute N-grams. For this reason most uses of web data rely on pagecoun ts
fromsearchengines. Pagecountsareonlyanapproximationt oactualcountsformany
reasons: a pagemaycontainan N-grammultipletimes, most searchenginesroundoff
their counts,punctuationis deleted,andthe countsthemse lvesmay beadjusteddue to

link and other information. It seems that this kind of noise d oes not hugely affect the
resultsofusingthewebasacorpus(KellerandLapata,2003; NakovandHearst,2005),
although it is possible to perform speciﬁc adjustments, suc h as ﬁtting a regression to
predictactualwordcountsfrompagecounts(ZhuandRosenfe ld,2001).
4.9.4 UsingLonger DistanceInformation: A Brief Summary
There are many methods for incorporating longer-distance c ontext into N-gram mod-

eling. While we have limited our discussion mainly to bigram sand trigrams, state-of-
the-artspeechrecognitionsystems,forexample,arebased onlonger-distance N-grams,
especially 4-grams, but also 5-grams. Goodman (2006) showe d that with 284 million
words of training data, 5-grams do improve perplexity score s over 4-grams, but not
by much. Goodman checked contexts up to 20-grams, and found t hat after 6-grams,
longercontextsweren’tuseful,at least notwith284millio nwordsoftrainingdata.
Manymodelsfocusonmoresophisticatedwaystogetlonger-d istanceinformation.

Forexamplepeopletendtorepeatwordstheyhaveusedbefore . Thusifawordisused
once in a text, it will probably be used again. We can capture t his fact by a cache Cache
languagemodel(KuhnandDeMori,1990). Forexampletouseau nigramcachemodel
topredictword iofatestcorpus,wecreateaunigramgrammarfromtheprecedi ngpart
of the test corpus (words 1 to i−1) and mix this with our conventional N-gram. We
might use only a shorter window from the previous words, rath er than the entire set.

Cache language models are very powerful in any applications where we have perfect
knowledgeof the words. Cache modelsworkless well in domain swherethe previous
wordsarenotknownexactly. Inspeechapplications,forexa mple,unlessthereissome
way for users to correct errors, cache models tend to “lock in ” errors they made on
earlierwords.
The fact that words are often repeated in a text is a symptom of a more general
fact about texts; texts tend to be aboutthings. Documents which are about particular

topics tend to use similar words. This suggests that we could train separate language
modelsfordifferenttopics. In topic-based languagemodels(Chenetal.,1998;Gildea Topic-based

DRAFTSection4.10. Advanced: InformationTheoryBackground 113
andHofmann,1999),wetrytotakeadvantageofthefactthatd ifferenttopicswillhave
differentkindsofwords. Forexamplewe cantraindifferent languagemodelsforeach
topict,andthenmixthem,weightedbyhowlikelyeachtopicisgiven thehistory h:
p(w|h) =∑
tP(w|t)P(t|h) (4.53)
A very similar class of models relies on the intuition that up coming words are se-
mantically similar to preceding words in the text. These mod els use a measure of
semantic word association such as the latent semantic indexing described in Ch. 20Latent semanticindexing
(Coccaro and Jurafsky, 1998; Bellegarda, 1999, 2000), or on -line dictionaries or the-
sauri (Demetriouet al.,1997)to computea probabilitybase d ona word’ssimilarity to
precedingwords,andthenmixit witha conventional N-gram.
There are also various ways to extend the N-gram model by having the previous
(conditioning) word be something other than a ﬁxed window of previous words. For

example we can choose as a predictor a word called a triggerwhich is not adjacent Trigger
butwhichisveryrelated(hashighmutualinformationwith) thewordwearetryingto
predict (Rosenfeld, 1996; Niesler and Woodland, 1999; Zhou and Lua, 1998). Or we
can create skip N-grams ,where the precedingcontext‘skipsover’some intermediat e SkipN-gram
words, for example computing a probability such as P(wi|wi−1,wi−3). We can also
use extra previous context just in cases where a longer phras e is particularly frequent

or predictive, producing a variable-length N-gram(Ney et al., 1994; Kneser, 1996;Variable-length
N-gram
Niesler andWoodland,1996).
Ingeneral,usingverylargeandrichcontextscanresultinv erylargelanguagemod-
els. Thus these modelsare often prunedby removinglow-prob abilityevents. Pruning
isalsoessentialforusinglanguagemodelsonsmallplatfor mssuchascellphones(Stol-
cke,1998;Churchet al.,2007).
Finally, there is a wide body of research on integrating soph isticated linguistic

N-gram
structuresintolanguagemodeling. Languagemodelsbasedo nsyntacticstructurefrom
probabilistic parsers are described in Ch. 14. Language mod els based on the current
speechactin dialoguearedescribedin Ch.24.
4.10 Advanced: InformationTheoryBackground
Igotthehorseright here
FrankLoesser, GuysandDolls
We introduced perplexity in Sec. 4.4 as a way to evaluate N-gram models on a
test set. A better N-gram model is one which assigns a higher probability to the t est

data, and perplexity is a normalized version of the probabil ity of the test set. Another
way to think about perplexity is based on the information-th eoreticconcept of cross-
entropy. Inordertogiveanotherintuitionintoperplexity asametric,thissectiongives
a quick review of fundamental facts from information theory including the concept
of cross-entropythat underliesperplexity. Theintereste dreadershouldconsulta good
informationtheorytextbooklikeCoverandThomas(1991).

DRAFT114 Chapter 4. N-grams
Perplexityisbasedontheinformation-theoreticnotionof cross-entropy ,whichwe
willnowworktowarddeﬁning. Entropy isameasureofinformation,andisinvaluable Entropy
throughoutspeech and language processing. It can be used as a metric for how much
information there is in a particular grammar, for how well a g iven grammar matches
a given language, for how predictive a given N-gram grammar is about what the next
wordcouldbe. Giventwo grammarsanda corpus,wecanuseentr opytotell uswhich
grammarbetter matchesthe corpus. We can also use entropyto comparehowdifﬁcult
two speech recognition tasks are, and also to measure how wel l a given probabilistic
grammarmatcheshumangrammars.
Computingentropyrequiresthatweestablisharandomvaria bleXthatrangesover
whatever we are predicting(words, letters, parts of speech , the set of which we’ll call
χ), and that has a particular probability function, call it p(x). The entropy of this
randomvariable Xisthen
H(X) =−∑
x∈χp(x)log2p(x) (4.54)

H(X) =−∑
Thelogcaninprinciplebecomputedinanybase. Ifweuselogb ase2,theresulting
valueofentropywill bemeasuredin bits.
The most intuitive way to deﬁne entropy for computer scienti sts is to think of the
entropy as a lower bound on the number of bits it would take to e ncode a certain
decisionorpieceofinformationinthe optimalcodingschem e.
Cover and Thomas (1991) suggest the following example. Imag ine that we want
to place a bet on a horse race but it is too far to go all the way to Yonkers Racetrack,

and we’d like to send a short message to the bookie to tell him w hich horse to bet on.
Supposethereareeighthorsesinthisparticularrace.
One way to encode this message is just to use the binary repres entation of the
horse’s number as the code; thus horse 1 would be 001, horse 2 010, horse 3 011,
and so on, with horse 8 coded as 000. If we spend the whole day betting, and each
horseiscodedwith3bits, onaveragewewouldbesending3bit sperrace.
Can we do better? Suppose that the spread is the actual distri bution of the bets

placed,andthatwe representit asthepriorprobabilityofe achhorseasfollows:
Horse11
2Horse51
64
Horse21
4Horse61
64
Horse31
8Horse71
64
Horse41
16Horse81
64
The entropy of the random variable Xthat ranges over horses gives us a lower
boundonthenumberofbits,andis:
H(X) =−i=8
∑
i=1p(i)logp(i)
=−1
2log1
2−1
4log1
4−1
8log1
8−1
16log1
16−4(1
64log1
64)
=2bits (4.55)

DRAFTSection4.10. Advanced: InformationTheoryBackground 115
Acodethataverages2bitsperracecanbebuiltbyusingshort encodingsformore
probablehorses,andlongerencodingsforlessprobablehor ses. Forexample,wecould
encode the most likely horse with the code 0, and the remaining horses as 10, then
110,1110,111100,111101,111110,and111111.
Whatifthehorsesareequallylikely? Wesawabovethatifweu sedanequal-length
binary code for the horse numbers, each horse took 3 bits to co de, so the average was
3. Isthe entropythesame? Inthis case each horsewouldhavea probabilityof1
8. The
entropyofthe choiceofhorsesisthen:
H(X) =−i=8
∑
i=11
8log1
8=−log1
8=3bits (4.56)
Until now we have been computing the entropy of a single varia ble. But most of
whatwewill useentropyforinvolves sequences . Foragrammar,forexample,wewill
be computing the entropy of some sequence of words W={w0,w1,w2,...,wn}. One
way to do this is to have a variable that rangesover sequences of words. For example

8. The
H(X) =−i=8
∑
i=11
8log1
8=−log1
we can compute the entropyof a randomvariable that rangesov er all ﬁnite sequences
ofwordsoflength nin somelanguage Lasfollows:
H(w1,w2,...,wn) =−∑
Wn
1∈Lp(Wn
1)logp(Wn
1) (4.57)
We could deﬁne the entropy rate (we could also think of this as the per-word Entropy rate
entropy)astheentropyofthissequencedividedbythe numberofword s:
1
nH(Wn
1) =−1
n∑
Wn
1∈Lp(Wn
1)logp(Wn
1) (4.58)
But to measure the true entropy of a language, we need to consi der sequences of
inﬁnite length. If we think of a language as a stochastic proc essLthat produces a

Wn
1∈Lp(Wn
1)logp(Wn
1) (4.57)
1
nH(Wn
1) =−1
n∑
Wn
1∈Lp(Wn
1)logp(Wn
1) (4.58)
sequenceofwords,itsentropyrate H(L)isdeﬁnedas:
H(L) =−lim
n→∞1
nH(w1,w2,...,wn)
=−lim
n→∞1
n∑
W∈Lp(w1,...,wn)logp(w1,...,wn) (4.59)
The Shannon-McMillan-Breiman theorem (Algoet and Cover, 1 988; Cover and
Thomas,1991)statesthatif thelanguageisregularincerta inways(to beexact,if it is
bothstationaryandergodic),
H(L) =lim
n→∞−1
nlogp(w1w2...wn) (4.60)
Thatis,wecantakeasinglesequencethatislongenoughinst eadofsummingover
all possible sequences. The intuition of the Shannon-McMil lan-Breiman theorem is

H(L) =−lim
n→∞1
=−lim
n→∞1
n∑
H(L) =lim
n→∞−1
that a longenoughsequenceof words will containin it many ot hershorter sequences,

DRAFT116 Chapter 4. N-grams
andthateach oftheseshortersequenceswill reoccurin the l ongersequenceaccording
totheirprobabilities.
A stochastic process is said to be stationary if the probabilities it assigns to a Stationary
sequence are invariant with respect to shifts in the time ind ex. In other words, the
probability distribution for words at time tis the same as the probability distribution
at timet+1. Markov models, and hence N-grams, are stationary. For example, in
a bigram, Piis dependent only on Pi−1. So if we shift our time index by x,Pi+xis
still dependent on Pi+x−1. But natural language is not stationary, since as we will
see in Ch. 12, the probability of upcoming words can be depend ent on events that
were arbitrarily distant and time dependent. Thus our stati stical models only give an
approximationtothe correctdistributionsandentropieso fnaturallanguage.
Tosummarize,bymakingsomeincorrectbut convenientsimpl ifyingassumptions,

we can compute the entropy of some stochastic process by taki ng a very long sample
of the output, and computing its average log probability. In the next section we talk
aboutthewhyandhow: whywewouldwanttodothis(i.e.,forwhatkindsofproblems
would the entropy tell us something useful), and howto compute the probability of a
verylongsequence.
4.10.1 Cross-Entropy forComparing Models
In this section we introduce cross-entropy , and discuss its usefulness in comparing Cross-entropy

different probabilistic models. The cross-entropy is usef ul when we don’t know the
actual probabilitydistribution pthat generatedsome data. It allows us to use some m,
which is a model of p(i.e., an approximation to p). The cross-entropy of monpis
deﬁnedby:
H(p,m) =lim
n→∞−1
n∑
W∈Lp(w1,...,wn)logm(w1,...,wn) (4.61)
Thatis,wedrawsequencesaccordingtotheprobabilitydist ributionp,butsumthe
logoftheirprobabilitiesaccordingto m.
Again,followingtheShannon-McMillan-Breimantheorem,f orastationaryergodic
process:
n→∞−1

deﬁnedby:
n→∞−1
n∑
process:
n→∞−1
nlogm(w1w2...wn) (4.62)
Thismeansthat,asforentropy,wecanestimatethecross-en tropyofamodel mon
somedistribution pbytakingasinglesequencethatislongenoughinsteadofsum ming
overall possiblesequences.
What makes the cross entropy useful is that the cross entropy H(p,m)is an upper
boundontheentropy H(p). Foranymodel m:
H(p)≤H(p,m) (4.63)
This means that we can use some simpliﬁed model mto help estimate the true
entropyofasequenceofsymbolsdrawnaccordingtoprobabil ityp. Themoreaccurate
mis, the closer the cross entropy H(p,m)will be to the true entropy H(p). Thus

DRAFTSection4.11. Advanced: TheEntropyofEnglishandEntropyR ate Constancy 117
the difference between H(p,m)andH(p)is a measure of how accurate a model is.
Between two models m1andm2, the more accurate model will be the one with the
lower cross-entropy. (Thecross-entropycan neverbe lower than the true entropy,so a
modelcannoterrbyunderestimatingthetrueentropy).
We are ﬁnally ready to see the relation between perplexity an d cross-entropy as
we saw it in Eq. 4.62. Cross-entropy is deﬁned in the limit, as the length of the ob-
servedwordsequencegoestoinﬁnity. Wewillneedanapproxi mationtocross-entropy,
relying on a (sufﬁciently long) sequence of ﬁxed length. Thi s approximation to the
cross-entropyofamodel M=P(wi|wi−N+1...wi−1)ona sequenceofwords Wis:
H(W) =−1
NlogP(w1w2...wN) (4.64)
Theperplexity ofamodel Ponasequenceofwords Wisnowformallydeﬁnedasthe Perplexity
expofthiscross-entropy:
Perplexity (W) =2H(W)
=P(w1w2...wN)−1
N
=N√
1
=N√
N
∏
i=11
P(wi|w1...wi−1)(4.65)

H(W) =−1
N
=N√
1
=N√
N
∏
i=11
4.11 Advanced: The Entropy of English and Entropy Rate
Constancy
As we suggested in the previous section, the cross-entropy o f some model mcan be
usedasanupperboundonthetrueentropyofsomeprocess. Wec anusethismethodto
get an estimate of the true entropyof English. Why should we c are about the entropy
ofEnglish?
One reason is that the true entropy of English would give us a s olid lower bound
forallofourfutureexperimentsonprobabilisticgrammars . Anotheristhatwecanuse
the entropyvaluesforEnglishto helpunderstandwhat parts of a languageprovidethe

Constancy
most information (for example, is the predictability of Eng lish mainly based on word
order,onsemantics, onmorphology,onconstituency,oronp ragmaticcues?) Thiscan
helpusimmenselyin knowingwheretofocusourlanguage-mod elingefforts.
There are two common methods for computing the entropy of Eng lish. The ﬁrst
was employedby Shannon(1951), as part of his groundbreakin gwork in deﬁning the
ﬁeldofinformationtheory. Hisideawastousehumansubject s,andtoconstructapsy-

chologicalexperimentthat requiresthemto guessstringso fletters. Bylookingathow
manyguessesittakesthemtoguessletterscorrectlywecane stimatetheprobabilityof
theletters, andhencetheentropyofthe sequence.

DRAFT118 Chapter 4. N-grams
The actual experimentis designed as follows: we present a su bject with some En-
glishtextandaskthesubjecttoguessthenextletter. Thesu bjectswillusetheirknowl-
edgeofthelanguagetoguessthemostprobableletterﬁrst,t henextmostprobablenext,
and so on. We recordthe numberof guessesit takesfor the subj ect to guess correctly.
Shannon’sinsightwasthat the entropyof thenumber-of-gue ssessequenceisthe same
astheentropyofEnglish. (Theintuitionisthatgiventhenu mber-of-guessessequence,
wecouldreconstructtheoriginaltextbychoosingthe“ nthmostprobable”letterwhen-
ever the subject took nguesses). This methodology requires the use of letter guess es
rather than word guesses (since the subject sometimes has to do an exhaustive search
of all the possible letters!), so Shannon computed the per-letter entropy of English
rather thanthe per-wordentropy. He reportedan entropyof 1 .3bits (for 27 characters
(26lettersplusspace)). Shannon’sestimateislikelytobe toolow,sinceitisbasedona

singletext( JeffersontheVirginian byDumasMalone). Shannonnotesthathissubjects
had worse guesses (hence higher entropies) on other texts (n ewspaper writing, scien-
tiﬁcwork,andpoetry). MorerecentvariationsontheShanno nexperimentsincludethe
use ofa gamblingparadigmwherethesubjectsgetto betonthe nextletter(Coverand
King,1978;CoverandThomas,1991).
The second method for computing the entropy of English helps avoid the single-
text problem that confounds Shannon’s results. This method is to take a very good

stochastic model, train it on a very large corpus, and use it t o assign a log-probability
toa verylongsequenceofEnglish,usingtheShannon-McMill an-Breimantheorem:
H(English )≤lim
n→∞−1
nlogm(w1w2...wn) (4.66)
Forexample,Brownet al. (1992)trainedatrigramlanguagem odelon583million
words of English (293,181 different types) and used it to com pute the probability of
the entire Brown corpus (1,014,312 tokens). The training da ta include newspapers,

n→∞−1
encyclopedias,novels,ofﬁcecorrespondence,proceeding softheCanadianparliament,
andothermiscellaneoussources.
TheythencomputedthecharacterentropyoftheBrowncorpus byusingtheirword-
trigramgrammartoassignprobabilitiestotheBrowncorpus ,consideredasasequence
ofindividualletters. Theyobtainedanentropyof1 .75bitspercharacter(wheretheset
ofcharactersincludedall the95printableASCII character s).
The average length of English written words (including spac e) has been reported
at 5.5 letters (N´ adas, 1984). If this is correct, it means th at the Shannon estimate of

1.3bitsperlettercorrespondstoaper-wordperplexityof1 42forgeneralEnglish. The
numbers we report earlier for the WSJ experiments are signiﬁ cantly lower than this,
sincethetrainingandtestsetcamefromthesamesubsampleo fEnglish. Thatis,those
experiments underestimate the complexity of English (sinc e the Wall Street Journal
looksverylittle likeShakespeare,forexample)
A number of scholars have independently made the intriguing suggestion that en-
tropyrate playsa role in humancommunicationin general (Li ndblom,1990;Van Son

et al., 1998; Aylett, 1999; Genzel and Charniak, 2002; Van So n and Pols, 2003). The
idea is that people speak so as to keep the rate of information being transmitted per
second roughly constant, i.e., transmitting a constant num ber of bits per second, or
maintaining a constant entropy rate. Since the most efﬁcien t way of transmitting in-

DRAFTSection4.11. Advanced: TheEntropyofEnglishandEntropyR ate Constancy 119
formation through a channel is at a constant rate, language m ay even have evolved
for such communicative efﬁciency (Plotkin and Nowak, 2000) . There is a wide vari-
ety of evidence for the constant entropy rate hypothesis. On e class of evidence, for
speech, shows that speakers shorten predictable words (i.e ., they take less time to say
predictable words) and lengthen unpredictable words (Ayle tt, 1999; Jurafsky et al.,
2001;Aylett andTurk,2004). Inanotherline ofresearch,Ge nzelandCharniak(2002,
2003)showthatentropyrateconstancymakespredictionsab outtheentropyofindivid-
ual sentences from a text. In particular, they show that it pr edicts that local measures
of sentenceentropywhich ignorepreviousdiscoursecontex t(forexamplethe N-gram
probabilityofsentence),shouldincreasewiththesentenc enumber,andtheydocument
thisincrease in corpora. Keller (2004)providesevidencet hat entropyrate playsa role

for the addressee as well, showing a correlation between the entropy of a sentence
and the processingeffort it causes in comprehension,as mea suredby readingtimes in
eye-trackingdata.
Theunderlyingmathematicsofthe N-gramwasﬁrstproposedbyMarkov(1913),who
usedwhatarenowcalled Markovchains (bigramsandtrigrams)topredictwhetheran
upcomingletterinPushkin’s EugeneOnegin wouldbeavoweloraconsonant. Markov
classiﬁed20,000lettersasVorCandcomputedthebigramand trigramprobabilitythat

a givenletter would be a vowel giventhe previousone or two le tters. Shannon(1948)
appliedN-grams to compute approximations to English word sequences . Based on
Shannon’swork, Markov models were commonlyused in enginee ring,linguistic, and
psychologicalworkonmodelingwordsequencesbythe 1950s.
In a series of extremely inﬂuential papers starting with Cho msky (1956) and in-
cludingChomsky(1957)andMillerandChomsky(1963),NoamC homskyarguedthat

“ﬁnite-stateMarkovprocesses”,whileapossiblyusefulen gineeringheuristic,werein-
capableofbeingacompletecognitivemodelofhumangrammat icalknowledge. These
argumentsled many linguists and computationallinguists t o ignore work in statistical
modelingfordecades.
Theresurgenceof N-grammodelscamefromJelinek,Mercer,Bahl,andcolleague s
attheIBMThomasJ.WatsonResearchCenter,whowereinﬂuenc edbyShannon, and
Baker at CMU, who was inﬂuenced by the work of Baum and colleag ues. Indepen-
dently these two labs successfully used N-grams in their speech recognition systems

(Baker, 1990; Jelinek, 1976; Baker, 1975; Bahl et al., 1983; Jelinek, 1990). A trigram
model was used in the IBM TANGORA speech recognition system i n the 1970s, but
theideawasnotwrittenupuntillater.
Add-one smoothing derives from Laplace’s 1812 law of succes sion, and was ﬁrst
applied as an engineering solution to the zero-frequency pr oblem by Jeffreys (1948)
based onan earlier Add-KsuggestionbyJohnson(1932). Prob lemswith the Add-one
algorithmaresummarizedinGaleandChurch(1994). TheGood -Turingalgorithmwas

ﬁrst applied to the smoothingof N-gram grammarsat IBM by Katz, as cited in N´ adas

DRAFT120 Chapter 4. N-grams
(1984). Church and Gale (1991) give a good description of the Good-Turingmethod,
as well as the proof. Sampson (1996) also has a useful discuss ion of Good-Turing.
Jelinek(1990)summarizesthisandmanyotherearlylanguag emodelinnovationsused
inthe IBMlanguagemodels.
A wide variety of different language modeling and smoothing techniques were
tested through the 1980’s and 1990’s, including Witten-Bel l discounting (Witten and
Bell, 1991), varieties of class-based models (Jelinek, 199 0; Kneser and Ney, 1993;
Heeman, 1999; Samuelsson and Reichl, 1999), and others (Gup ta et al., 1992). In
the late 1990’s, Chen and Goodman produced a very inﬂuential series of papers with
a comparison of different language models (Chen and Goodman , 1996, 1998, 1999;
Goodman,2006). Theyperformeda numberofcarefullycontro lledexperimentscom-
paring different discounting algorithms, cache models, cl ass-based (cluster) models,
and other language model parameters. They showed the advant ages of Interpolated

Kneser-Ney, which has since become one of the most popular cu rrent methods for
language modeling. These papers inﬂuenced our discussion i n this chapter, and are
recommendedreadingifyouhavefurtherinterestinlanguag emodeling.
Aswesuggestedearlierinthechapter,recentresearchinla nguagemodelinghasfo-
cusedonadaptation,ontheuseofsophisticatedlinguistic structuresbasedonsyntactic
anddialoguestructure,andonveryverylarge N-grams. For examplein 2006,Google
publiclyreleasedaverylargesetof N-gramsthatisausefulresearchresource,consist-

ingofalltheﬁve-wordsequencesthatappearatleast40time sfrom1,024,908,267,229
wordsofrunningtext;thereare1,176,470,663ﬁve-wordseq uencesusingover13mil-
lion unique words types (Franz and Brants, 2006). Large lang uage models generally
needtobe prunedto bepractical,usingtechniquessuchasSt olcke (1998)andChurch
et al.(2007).
4.12 Summary
Thischapterintroducedthe N-gram,oneoftheoldestandmostbroadlyusefulpractical
toolsin languageprocessing.
•AnN-gramprobabilityistheconditionalprobabilityofawordg iventheprevious

N−1 words. N-gram probabilities can be computed by simply counting in a
corpus and normalizing (the Maximum Likelihood Estimate ) or they can be
computed by more sophisticated algorithms. The advantage o fN-grams is that
they take advantage of lots of rich lexical knowledge. A disa dvantage for some
purposesis thattheyareverydependentonthecorpustheywe retrainedon.
•Smoothing algorithms provide a better way of estimating the probabili ty ofN-
gramsthanMaximumLikelihoodEstimation. Commonlyused N-gramsmooth-

ingalgorithmsrelyonlower-order N-gramcountsvia backofforinterpolation .
•Bothbackoffandinterpolationrequirediscountingsuchas Kneser-Ney ,Witten-
BellorGood-Turing discounting.
•N-gramlanguagemodels areevaluatedbyseparatingthecorpusintoa training
setandatestset,trainingthemodelonthetrainingset,andevaluatingonth etest

DRAFTSection4.12. Summary 121
set. Theperplexity 2Hof the language model on a test set is used to compare
languagemodels.
Exercises
4.1Writeouttheequationfortrigramprobabilityestimation( modifyingEq.4.14).
4.2Writea programtocomputeunsmoothedunigramsandbigrams.
4.3Run your N-gram program on two different small corpora of your choice ( you
might use email text or newsgroups). Now compare the statist ics of the two
corpora. What are the differences in the most common unigram s between the
two? Howaboutinterestingdifferencesinbigrams?
4.4Addanoptionto yourprogramtogeneraterandomsentences.
4.5Addanoptionto yourprogramtodoGood-Turingdiscounting.
4.6Addanoptionto yourprogramtoimplementKatzbackoff.
4.7Addanoptionto yourprogramtocomputetheperplexityofa te st set.
4.8(Adapted from Michael Collins). Prove Eq. 4.27 given Eq. 4.2 6 and any nec-
essary assumptions. That is, show that given a probability d istribution deﬁned
by the GT formula in Eq. 4.26 for the Nitems seen in training, that the proba-

bility of the next, (i.e. N+1st) item being unseen in training can be estimated
by Eq. 4.27. You may make any necessary assumptionsfor the pr oof, including
assumingthatall Ncarenon-zero.
4.9(Advanced) Suppose someone took all the words in a sentence a nd reordered
them randomly. Write a program which take as input such a bag of words and bag of words
produces as output a guess at the original order. You will nee d to anN-gram
grammar produced by your N-gram program (on some corpus), and you will

need to use the Viterbi algorithm introduced in the next chap ter. This task is
sometimescalled baggeneration . Bag generation
4.10Theﬁeldof authorshipattribution isconcernedwithdiscoveringtheauthorofAuthorship
attribution
a particular text. Authorship attribution is important in m any ﬁelds, including
history, literature, and forensic linguistics. For exampl e Mosteller and Wallace
(1964) applied authorship identiﬁcation techniques to dis cover who wrote The

Federalist papers. The Federalist papers were written in 1787-1788by A lexan-
der Hamilton, John Jay and James Madison to persuade New York to ratify the
United States Constitution. They were published anonymous ly, and as a result,
althoughsomeofthe85essayswereclearlyattributabletoo neauthororanother,
the authorship of 12 were in dispute between Hamilton and Mad ison. Foster

DRAFT122 Chapter 4. N-grams
(1989) applied authorship identiﬁcation techniques to sug gest that W.S.’s Fu-
neral Elegy for William Peter might have been written by William Shakesp eare
(he turned out to be wrong on this one), and that the anonymous author of Pri-
mary Colors , the roman ` a clef about the Clinton campaign for the America n
presidency,wasjournalistJoeKlein(Foster,1996).
Astandardtechniqueforauthorshipattribution,ﬁrstused byMostellerandWal-
lace, is a Bayesian approach. For example, they trained a pro babilistic model
of the writing of Hamilton and another model on the writings o f Madison, then
computedthemaximum-likelihoodauthorforeachofthedisp utedessays. There
are many complex factors that go into these models, includin g vocabulary use,
word length, syllable structure, rhyme,grammar; see Holme s (1994)for a sum-
mary. This approach can also be used for identifying which ge nre a text comes
from.
One factor in manymodelsis the use of rarewords. As a simple a pproximation

from.
to this one factor, applythe Bayesian methodto the attribut ionof any particular
text. Youwillneedthreethings: atexttotestandtwopotent ialauthorsorgenres,
with a large on-line text sample of each. One of them should be the correct
author. Train a unigram language model on each of the candida te authors. You
are only going to use the singleton unigramsin each language model. You will
compute P(T|A1), the probability of the text given author or genre A1, by (1)
takingthelanguagemodelfrom A1,(2)bymultiplyingtogethertheprobabilities

ofalltheunigramsthatonlyoccuronceinthe“unknown”text and(3)takingthe
geometricmeanofthese(i.e.,the nthroot,where nisthenumberofprobabilities
youmultiplied). Dothesamefor A2. Choosewhicheverishigher. Diditproduce
thecorrectcandidate?

Chapter5
Part-of-Speech Tagging
Conjunction Junction, what’s your function?
Bob Dorough, Schoolhouse Rock,1973
A gnostic was seated before a grammarian. The grammarian sai d, ‘A word must be
oneofthreethings: eitheritisanoun, averb,oraparticle. ’ Thegnostictorehisrobeand
cried, “Alas! Twenty years of my life and striving and seekin g have gone to the winds,
for I laboured greatly in the hope that there was another word outside of this. Now you
have destroyed my hope.’ Though the gnostic had already atta ined the word which was
hispurpose, he spoke thus inorder toarouse the grammarian.

Rumi (1207–1273), TheDiscourses of Rumi ,Translated byA.J. Arberry
Dionysius Thrax of Alexandria ( c.100B.C.), or perhaps someone else (exact au-
thorship being understandablydifﬁcult to be sure of with te xts of this vintage), wrote
a grammatical sketch of Greek (a “ techn¯e”) which summarized the linguistic knowl-
edge of his day. This work is the direct source of an astonishi ng proportion of our
modern linguistic vocabulary, including among many other w ords,syntax,diphthong ,

clitic, andanalogy. Also included are a description of eight parts-of-speech : noun, parts-of-speech
verb, pronoun,preposition, adverb, conjunction, partici ple, and article. Although ear-
lier scholars (including Aristotle as well as the Stoics) ha d their own lists of parts-of-
speech,itwasThrax’ssetofeightwhichbecamethebasisfor practicallyallsubsequent
part-of-speechdescriptionsofGreek,Latin,andmostEuro peanlanguagesforthenext
2000years.
Schoolhouse Rock was a popular series of 3-minute musical an imated clips ﬁrst

aired on television in 1973. The series was designed to inspi re kids to learn multipli-
cation tables, grammar, and basic science and history. The G rammar Rock sequence,
for example,includedsongsaboutparts-of-speech,thusbr ingingthese categoriesinto
the realm of popular culture. As it happens, Grammar Rock was remarkably tradi-
tionalinitsgrammaticalnotation,includingexactlyeigh tsongsaboutparts-of-speech.
Although the list was slightly modiﬁed from Thrax’s origina l, substituting adjective
and interjection for the original participle and article, t he astonishing durability of the

parts-of-speech through two millenia is an indicator of bot h the importance and the
transparencyoftheirrolein humanlanguage.
Morerecent lists of parts-of-speech(or tagsets) have manymoreword classes; 45 tagsets
for the Penn Treebank (Marcus et al., 1993), 87 for the Brown c orpus(Francis, 1979;
FrancisandKuˇ cera,1982),and146forthe C7tagset (Garsid eet al.,1997).
The signiﬁcance of parts-of-speech (also known as POS,word classes ,morpho- POS

logicalclasses ,orlexicaltags )forlanguageprocessingisthelargeamountofinforma-
tiontheygiveaboutawordanditsneighbors. Thisisclearly trueformajorcategories,

DRAFT124 Chapter 5. Part-of-SpeechTagging
(verbversusnoun), but is also true forthe manyﬁner distinctions. For exampl ethese
tagsets distinguish between possessive pronouns( my,your,his,her,its) and personal
pronouns( I,you,he,me). Knowing whether a word is a possessive pronounor a per-
sonal pronoun can tell us what words are likely to occur in its vicinity (possessive
pronounsare likely to be followed by a noun, personal pronou nsby a verb). This can
beusefulina languagemodelforspeechrecognition.
A word’s part-of-speechcan tell us somethingabout how the w ord is pronounced.
As Ch. 8 will discuss, the word content, for example, can be a noun or an adjective.
They are pronounced differently (the noun is pronounced CONtent and the adjective
conTENT ).Thusknowingthepart-of-speechcanproducemorenatural pronunciations
inaspeechsynthesissystemandmoreaccuracyinaspeechrec ognitionsystem. (Other
pairs like this include OBject(noun) and obJECT(verb),DIScount (noun) and dis-
COUNT(verb);see Cutler(1986)).

Parts-of-speechcanalsobeusedinstemmingforinformatio nalretrieval(IR),since
knowing a word’s part-of-speech can help tell us which morph ological afﬁxes it can
take, as we saw in Ch. 3. They can also enhance an IR applicatio n by selecting out
nouns or other important words from a document. Automatic as signment of part-of-
speechplaysaroleinparsing,inword-sensedisambiguatio nalgorithms,andinshallow
parsing of texts to quickly ﬁnd names, times, dates, or other named entities for the

information extraction applications discussed in Ch. 22. F inally, corpora that have
been marked for parts-of-speech are very useful for linguis tic research. For example,
theycanbe usedtohelpﬁndinstancesorfrequenciesofparti cularconstructions.
This chapter focuses on computational methods for assignin g parts-of-speech to
words(part-of-speechtagging ). Manyalgorithmshavebeen appliedto this problem,
includinghand-writtenrules( rule-basedtagging ),probabilisticmethods( HMMtag-

gingandmaximumentropytagging ),aswellasothermethodssuchas transformation-
based tagging andmemory-based tagging . We will introduce three of these algo-
rithms in this chapter: rule-based tagging, HMM tagging, an d transformation-based
tagging. But before turning to the algorithms themselves, l et’s begin with a summary
ofEnglishwordclasses, andofvarioustagsetsforformally codingthese classes.
5.1 (Mostly)EnglishWordClasses
Until now we have been using part-of-speech terms like nounandverbrather freely.

In this section we give a more complete deﬁnition of these and other classes. Tradi-
tionally the deﬁnitionof parts-of-speechhasbeen based on syntactic and morphologi-
calfunction;wordsthatfunctionsimilarlywithrespectto whatcanoccurnearby(their
“syntacticdistributionalproperties”),orwithrespectt otheafﬁxestheytake(theirmor-
phologicalproperties)aregroupedintoclasses. Whilewor dclassesdohavetendencies
towardsemanticcoherence(nounsdoinfactoftendescribe“ people,placesorthings”,

andadjectivesoftendescribeproperties),thisisnotnece ssarilythecase,andingeneral
we don’tuse semanticcoherenceasadeﬁnitionalcriterionf orparts-of-speech.
Parts-of-speechcan be divided into two broad supercategor ies:closed class types closedclass
andopen class types. Closed classes are those that have relatively ﬁxed me mbership. openclass

DRAFTSection5.1. (Mostly)EnglishWordClasses 125
Forexample,prepositionsare aclosedclass becausetherei sa ﬁxedset oftheminEn-
glish;newprepositionsarerarelycoined. Bycontrastnoun sandverbsareopenclasses
becausenewnounsandverbsarecontinuallycoinedorborrow edfromotherlanguages
(e.g.,thenewverb tofaxortheborrowednoun futon). Itislikelythatanygivenspeaker
orcorpuswill havedifferentopenclasswords,butallspeak ersofa language,andcor-
porathat arelargeenough,will likelysharethe set ofclose dclass words. Closed class
wordsare also generally function words likeof,it,and,oryou,whichtendto be very functionwords
short,occurfrequently,andoftenhavestructuringusesin grammar.
Thereare fourmajoropenclasses thatoccurin the languages oftheworld; nouns,
verbs,adjectives ,andadverbs. ItturnsoutthatEnglishhasallfourofthese,although
noteverylanguagedoes.
Nounis the name given to the syntactic class in which the wordsfor most people, Noun
places, or things occur. But since syntactic classes like nounare deﬁned syntacti-

cally and morphologically rather than semantically, some w ords for people, places,
andthingsmaynotbenouns,andconverselysomenounsmaynot bewordsforpeople,
places, or things. Thus nouns include concrete terms like shipandchair, abstractions
likebandwidth andrelationship , and verb-like terms like pacingas inHis pacing to
and fro became quite annoying . What deﬁnes a noun in English, then, are things like
its ability to occur with determiners ( a goat, its bandwidth, Plato’s Republic ), to take

possessives ( IBM’s annual revenue ), and for most but not all nouns, to occur in the
pluralform( goats,abaci ).
Nouns are traditionally grouped into proper nouns andcommon nouns . Proper proper nouns
common nouns nouns, like Regina,Colorado , andIBM, are names of speciﬁc persons or entities. In
English,theygenerallyaren’tprecededbyarticles(e.g., thebookisupstairs ,butRegina
isupstairs ). InwrittenEnglish,propernounsareusuallycapitalized .
Inmanylanguages,includingEnglish,commonnounsaredivi dedintocountnouns count nouns

andmass nouns . Count nounsare those that allow grammaticalenumeration; that is, mass nouns
they can occur in both the singular and plural ( goat/goats, relationship/relationships )
and they can be counted ( one goat, two goats ). Mass nounsare used when something
is conceptualizedas a homogeneousgroup. So wordslike snow, salt , andcommunism
are not counted (i.e., *two snows or*two communisms ). Mass nouns can also appear
without articles where singular count nouns cannot ( Snow is white but not*Goat is
white).

white).
Theverbclass includes most of the words referring to actions and pro cesses, in- verb
cluding main verbs like draw,provide,differ, andgo. As we saw in Ch. 3, English
verbs have a number of morphologicalforms (non-3rd-person -sg(eat), 3rd-person-sg
(eats),progressive( eating),pastparticiple( eaten)). AsubclassofEnglishverbscalled
auxiliaries will bediscussedwhenwe turnto closedclassforms. auxiliaries
While many researchers believe that all human languages hav e the categories of
noun and verb, others have argued that some languages, such a s Riau Indonesian and

Tongan,don’tevenmakethisdistinction(Broschart,1997; Evans,2000;Gil,2000).
The third open class English form is adjectives; semantical ly this class includes
many terms that describe properties or qualities. Most lang uages have adjectives for
the conceptsofcolor( white,black), age(old,young),and value( good,bad),but there
are languages without adjectives. In Korean, for example, t he words corresponding
to English adjectives act as a subclass of verbs, so what is in English an adjective

DRAFT126 Chapter 5. Part-of-SpeechTagging
‘beautiful’actsinKoreanlikea verbmeaning‘tobebeautif ul’(Evans,2000).
Theﬁnalopenclassform, adverbs,isratherahodge-podge,bothsemanticallyand adverbs
formally. ForexampleSchachter(1985)pointsoutthatinas entencelikethefollowing,
all theitalicizedwordsareadverbs:
Unfortunately ,Johnwalked homeextremely slowly yesterday
What coherencethe class has semantically may be solely that each of these words
can be viewed as modifying something (often verbs, hence the name “adverb”, but
also other adverbsand entire verb phrases). Directionaladverbs orlocative adverbs locative
(home,here,downhill)specifythedirectionorlocationofsomeaction; degreeadverbs degree
(extremely ,very,somewhat ) specify the extent of some action, process, or property;
manner adverbs (slowly,slinkily,delicately ) describe the manner of some action or manner
process; and temporal adverb describe the time that some action or event took place temporal

(yesterday ,Monday). Because of the heterogeneousnatureofthis class, somead verbs
(for example temporal adverbs like Monday) are tagged in some tagging schemes as
nouns.
Theclosedclassesdiffermorefromlanguagetolanguagetha ndotheopenclasses.
Here’saquickoverviewofsomeofthemoreimportantclosedc lassesinEnglish,with
a fewexamplesofeach:
•prepositions: on,under,over,near,by,at, from,to,with
•determiners: a,an,the
•pronouns: she,who,I,others
•conjunctions: and,but,or,as,if, when
•auxiliaryverbs: can,may,should,are
•particles: up,down,on,off,in,out,at, by,

nouns.
•numerals: one,two, three,ﬁrst, second,third
Prepositions occur before noun phrases; semantically they are relationa l, often prepositions
indicatingspatialortemporalrelations,whetherliteral (onit,beforethen ,bythehouse )
or metaphorical ( on time,with gusto ,beside herself ). But they often indicate other
relationsas well ( Hamlet was written by Shakespeare ,and [fromShakespeare]“ And I
did laugh sans intermission an hour by his dial”). Fig. 5.1 shows the prepositions of

English according to the CELEX on-line dictionary (Baayen e t al., 1995), sorted by
their frequency in the COBUILD 16 million word corpus of Engl ish. Fig. 5.1 should
not be considered a deﬁnitive list, since different diction aries and tagsets label word
classesdifferently. Furthermore,thislist combinesprep ositionsandparticles.
Aparticleis a word that resembles a preposition or an adverb, and is use d in particle
combinationwithaverb. Whenaverbandaparticlebehaveasa singlesyntacticand/or

semantic unit, we call the combinationa phrasal verb . Phrasal verbscan behave as a phrasal verb
semantic unit; thustheyoftenhave a meaningthat is notpred ictablefromthe separate
meaningsof the verb and the particle. Thus turn down means something like ‘reject’,
ruleoutmeans‘eliminate’, ﬁndoutis‘discover’,and goonis‘continue’;thesearenot
meaningsthatcouldhavebeenpredictedfromthemeaningsof theverbandtheparticle
independently. Here aresomeexamplesofphrasalverbsfrom Thoreau:

DRAFTSection5.1. (Mostly)EnglishWordClasses 127
of 540,085 through 14,964 worth 1,563 pace 12
in 331,235 after 13,670 toward 1,390 nigh 9
for 142,421 between 13,275 plus 750 re 4
to 125,691 under 9,525 till 686 mid 3
with 124,965 per 6,515 amongst 525 o’er 2
on 109,129 among 5,090 via 351 but 0
at 100,169 within 5,030 amid 222 ere 0
by 77,794 towards 4,700 underneath 164 less 0
from 74,843 above 3,056 versus 113 midst 0
about 38,428 near 2,026 amidst 67 o’ 0
than 20,210 off 1,695 sans 20 thru 0
over 18,071 past 1,575 circa 14 vice 0
Figure5.1 Prepositions (and particles) of English from the CELEX on-l ine dictionary. Fre-
quency counts are from the COBUILD16millionwordcorpus.
SoIwent onforsomedayscuttingandhewingtimber...
Moralreformistheeffortto throwoff sleep...
Particles don’t always occur with idiomatic phrasal verb se mantics; here are more
examplesofparticlesfromtheBrowncorpus:
...she hadturnedthepaper over.
Hearoseslowlyandbrushedhimself off.
Hepacked uphisclothes.

WeshowinFig.5.1alistofsingle-wordparticlesfromQuirk etal.(1985). Sinceit
isextremelyhardtoautomaticallydistinguishparticlesf romprepositions,sometagsets
(liketheoneusedforCELEX)donotdistinguishthem,andeve nincorporathatdo(like
the Penn Treebank) the distinction is very difﬁcult to make r eliably in an automatic
process,so we donotgivecounts.
aboard aside besides forward(s) opposite through
about astray between home out throughout
above away beyond in outside together
across back by inside over under

ahead before close instead overhead underneath
alongside behind down near past up
apart below east, etc. off round within
around beneath eastward(s),etc. on since without
Figure5.2 Englishsingle-word particlesfrom Quirket al.(1985).
A closed class that occurs with nouns, often marking the begi nning of a noun
phrase, is the determiners . One small subtype of determinersis the articles: English determiners
articles hasthreearticles: a,an,andthe. Otherdeterminersinclude this(asinthischapter )and

that(asinthatpage).Aandanmarkanounphraseasindeﬁnite,while thecanmarkit
as deﬁnite; deﬁniteness is a discourse and semantic propert y that will be discussed in
Ch. 21. Articlesare quitefrequentin English;indeed theisthe most frequentlyoccur-
ringwordinmost corporaof writtenEnglish. Hereare COBUIL D statistics, againout
of16millionwords:
the: 1,071,676 a: 413,887 an: 59,359

DRAFT128 Chapter 5. Part-of-SpeechTagging
Conjunctions are used to join two phrases, clauses, or sentences. Coordin ating conjunctions
conjunctions like and,or, andbut, join two elements of equal status. Subordinating
conjunctions are used when one of the elements is of some sort of embedded status.
For example thatin“I thoughtthat you might like some milk” is a subordinatingcon-
junctionthatlinksthemainclause Ithought withthesubordinateclause youmightlike
some milk . Thisclauseis calledsubordinatebecausethisentire clau seis the“content”
ofthe mainverb thought. Subordinatingconjunctionslike thatwhichlinka verbto its
argumentinthiswayarealsocalled complementizers . Ch.12andCh.16willdiscuss complementizers
complementationin moredetail. Table5.3listsEnglishcon junctions.
and 514,946 yet 5,040 considering 174 forasmuch as 0
that 134,773 since 4,843 lest 131 however 0
but 96,889 where 3,952 albeit 104 immediately 0
or 76,563 nor 3,078 providing 96 inas far as 0
as 54,608 once 2,826 whereupon 85 insofar as 0

if 53,917 unless 2,205 seeing 63 inasmuch as 0
when 37,975 why 1,333 directly 26 insomuch as 0
because 23,626 now 1,290 ere 12 insomuch that 0
so 12,933 neither 1,120 notwithstanding 3 like 0
before 10,720 whenever 913 according as 0 neither nor 0
though 10,329 whereas 867 as if 0 now that 0
than 9,511 except 864 as long as 0 only 0
while 8,144 till 686 as though 0 provided that 0
after 7,042 provided 594 both and 0 providing that 0
whether 5,978 whilst 351 but that 0 seeing as 0
for 5,935 suppose 281 but then 0 seeing as how 0

although 5,424 cos 188 but then again 0 seeing that 0
until 5,072 supposing 185 either or 0 without 0
Figure5.3 Coordinatingandsubordinating conjunctions ofEnglishfr omCELEX.Frequency
counts are from COBUILD(16millionwords).
Pronouns are forms that often act as a kind of shorthand for referring t o some Pronouns
noun phrase or entity or event. Personal pronouns refer to persons or entities ( you, Personal
she,I,it,me, etc.).Possessive pronouns are formsof personal pronounsthat indicate Possessive

either actual possession ormore oftenjust an abstract rela tionbetweenthe personand
some object ( my, your, his, her, its, one’s, our, their ).Wh-pronouns (what, who, Wh
whom,whoever )areusedincertainquestionforms,ormayalsoactascomple mentizers
(Frieda, who I met ﬁve years ago ... ). Table 5.1 shows English pronouns,again from
CELEX.
AclosedclasssubtypeofEnglishverbsarethe auxiliary verbs. Crosslinguistically, auxiliary
auxiliaries are words (usually verbs) that mark certain sem antic features of a main

CELEX.
verb, including whether an action takes place in the present , past or future (tense),
whetheritiscompleted(aspect),whetheritisnegated(pol arity),andwhetheranaction
isnecessary,possible,suggested,desired,etc. (mood).
English auxiliaries include the copulaverbbe, the two verbs doandhave, along copula
with their inﬂected forms, as well as a class of modal verbs .Beis called a copula modal
becauseitconnectssubjectswithcertainkindsofpredicat enominalsandadjectives( He

isa duck). The verb haveis used for exampleto mark the perfect tenses ( I havegone,
I hadgone), whilebeis used as part of the passive ( We wererobbed), or progressive
(Weareleaving)constructions. Themodalsareusedtomarkthemoodassocia tedwith
the event or action depicted by the main verb. So canindicates ability or possibility,

DRAFTSection5.1. (Mostly)EnglishWordClasses 129
it 199,920 how 13,137 yourself 2,437 no one 106
I 198,139 another 12,551 why 2,220 wherein 58
he 158,366 where 11,857 little 2,089 double 39
you 128,688 same 11,841 none 1,992 thine 30
his 99,820 something 11,754 nobody 1,684 summat 22
they 88,416 each 11,320 further 1,666 suchlike 18
this 84,927 both 10,930 everybody 1,474 fewest 15
that 82,603 last 10,816 ourselves 1,428 thyself 14
she 73,966 every 9,788 mine 1,426 whomever 11
her 69,004 himself 9,113 somebody 1,322 whosoever 10
we 64,846 nothing 9,026 former 1,177 whomsoever 8
all 61,767 when 8,336 past 984 wherefore 6
which 61,399 one 7,423 plenty 940 whereat 5
their 51,922 much 7,237 either 848 whatsoever 4
what 50,116 anything 6,937 yours 826 whereon 2
my 46,791 next 6,047 neither 618 whoso 2
him 45,024 themselves 5,990 fewer 536 aught 1
me 43,071 most 5,115 hers 482 howsoever 1
who 42,881 itself 5,032 ours 458 thrice 1
them 42,099 myself 4,819 whoever 391 wheresoever 1

no 33,458 everything 4,662 least 386 you-all 1
some 32,863 several 4,306 twice 382 additional 0
other 29,391 less 4,278 theirs 303 anybody 0
your 28,923 herself 4,016 wherever 289 each other 0
its 27,783 whose 4,005 oneself 239 once 0
our 23,029 someone 3,755 thou 229 one another 0
these 22,697 certain 3,345 ’un 227 overmuch 0
any 22,666 anyone 3,318 ye 192 such andsuch 0
more 21,873 whom 3,229 thy 191 whate’er 0
many 17,343 enough 3,197 whereby 176 whenever 0
such 16,880 half 3,065 thee 166 whereof 0

those 15,819 few 2,933 yourselves 148 whereto 0
own 15,741 everyone 2,812 latter 142 whereunto 0
us 15,724 whatever 2,571 whichever 121 whichsoever 0
Figure5.4 Pronouns of English from the CELEX on-line dictionary. Freq uency counts are
from the COBUILD16millionwordcorpus.
mayindicates permission or possibility, mustindicates necessity, and so on. Fig. 5.1
gives counts for the frequencies of the modals in English. In addition to the perfect

havementioned above, there is a modal verb have(e.g.,I haveto go), which is very
common in spoken English. Neither it nor the modal verb dare, which is very rare,
have frequency counts because the CELEX dictionary does not distinguish the main
verb sense ( I havethree oranges ,He dared me to eat them ), from the modal sense
(There has to be some mistake ,DareI confront him? ), from the non-modal auxiliary
verbsense ( Ihaveneverseenthat ).
English also has many words of more or less unique function, i ncluding interjec-

tions(oh,ah,hey,man,alas,uh,um ),negatives (no,not),politenessmarkers (please, interjections
negatives
politenessmarkersthank you ),greetings (hello, goodbye ),and the existential there(thereare two on the
table) among others. Whether these classes are assigned particul ar names or lumped
together(asinterjectionsorevenadverbs)dependsonthep urposeofthe labeling.

DRAFT130 Chapter 5. Part-of-SpeechTagging
can 70,930 might 5,580 shouldn’t 858
will 69,206 couldn’t 4,265 mustn’t 332
may 25,802 shall 4,118 ’ll 175
would 18,448 wouldn’t 3,548 needn’t 148
should 17,760 won’t 3,100 mightn’t 68
must 16,520 ’d 2,299 oughtn’t 44
need 9,955 ought 1,845 mayn’t 3
can’t 6,375 will 862 dare, have ???
Figure5.5 English modal verbs from the CELEX on-line dictionary. Freq uency counts are
5.2 Tagsetsfor English
The previoussection gave broaddescriptionsof the kindsof syntactic classes that En-
glishwordsfallinto. Thissectionﬂeshesoutthatsketchby describingtheactualtagsets
used in part-of-speechtagging, in preparationfor the vari oustagging algorithmsto be
describedinthefollowingsections.
There are a small number of popular tagsets for English, many of which evolved
from the 87-tag tagset used for the Brown corpus(Francis, 19 79; Francis and Kuˇ cera,
1982). The Brown corpus is a 1 million word collection of samp les from 500 writ-

ten texts from different genres (newspaper, novels, non-ﬁc tion, academic, etc.) which
wasassembledatBrownUniversityin1963–1964(Kuˇ ceraand Francis,1967;Francis,
1979;FrancisandKuˇ cera,1982). Thiscorpuswastaggedwit hparts-of-speechbyﬁrst
applyingthe T AGGITprogramandthenhand-correctingthe tags.
Besides this original Brown tagset, two of the most commonly used tagsets are
the small 45-tag Penn Treebank tagset (Marcus et al., 1993), and the medium-sized
61 tag C5 tagset used by the Lancaster UCREL project’s CLAWS ( the Constituent

LikelihoodAutomaticWord-taggingSystem)taggertotagth eBritishNationalCorpus
(BNC) (Garside et al., 1997). We give all three of these tagse ts here, focusing on the
smallest, the Penn Treebank set, and discuss difﬁcult taggi ng decisions in that tag set
andsomeusefuldistinctionsmadeinthelargertagsets.
ThePennTreebanktagset,showninFig.5.2,hasbeenapplied totheBrowncorpus,
the Wall Street Journal corpus, and the Switchboard corpus a mong others; indeed,
perhapspartlybecauseofitssmallsize,it isoneofthemost widelyusedtagsets. Here

are some examplesof tagged sentencesfrom the Penn Treebank versionof the Brown
corpus(we will represent a tagged word by placing the tag aft er each word, delimited
byaslash):
(5.1)The/DTgrand/JJjury/NNcommented/VBDon/INa/DTnumber/N Nof/IN
other/JJtopics/NNS./.
(5.2)There/EX are/VBP 70/CDchildren/NNS there/RB
(5.3)Although/INpreliminary/JJﬁndings/NNSwere/VBD reported/VBN
more/RBRthan/INa/DTyear/NNago/IN,/,the/DTlatest/JJS results/NNS
appear/VBPin/INtoday/NN ’s/POSNew/NNPEngland/NNPJournal/NNP
of/INMedicine/NNP,/,

Example(5.1)showsphenomenathatwediscussedintheprevi oussection;thede-

DRAFTSection5.2. TagsetsforEnglish 131
Tag Description Example Tag Description Example
CC Coordin. Conjunction and, but, or SYM Symbol +,%,&
CD Cardinal number one, two, three TO “to” to
DT Determiner a, the UH Interjection ah, oops
EX Existential‘there’ there VB Verb,base form eat
FW Foreignword meaculpa VBD Verb,past tense ate
IN Preposition/sub-conj of, in,by VBG Verb,gerund eating
JJ Adjective yellow VBN Verb,past participle eaten
JJR Adj.,comparative bigger VBP Verb,non-3sg pres eat
JJS Adj.,superlative wildest VBZ Verb,3sg pres eats
LS Listitemmarker 1, 2,One WDT Wh-determiner which, that
MD Modal can, should WP Wh-pronoun what, who
NN Noun, sing. ormass llama WP$ Possessive wh- whose
NNS Noun, plural llamas WRB Wh-adverb how, where
NNP Proper noun, singular IBM $ Dollarsign $
NNPS Proper noun, plural Carolinas # Poundsign #
PDT Predeterminer all,both “ Leftquote ‘ or “
POS Possessive ending ’s ” Rightquote ’ or ”
PRP Personal pronoun I,you, he ( Leftparenthesis [,(, {,<

PRP$ Possessive pronoun your, one’s ) Rightparenthesis ],), },>
RB Adverb quickly,never , Comma ,
RBR Adverb, comparative faster . Sentence-ﬁnal punc . ! ?
RBS Adverb, superlative fastest : Mid-sentence punc : ; ... –-
RP Particle up, off
Figure5.6 PennTreebank part-of-speech tags (including punctuation ).
terminers theanda, the adjectives grandandother, the common nouns jury,number,
andtopics, the past tense verb commented . Example (5.2) shows the use of the EX

tag to mark the existential thereconstructionin English, and, for comparison,another
use oftherewhich is tagged as an adverb (RB). Example (5.3) shows the seg menta-
tion of the possessive morpheme ’s, and shows an example of a passive construction,
‘werereported’,inwhichtheverb reportedismarkedasapastparticiple(VBN),rather
thanasimplepast(VBD).Notealsothatthepropernoun NewEngland istaggedNNP.
Finally, note that since New England Journal of Medicine is a proper noun, the Tree-
banktaggingchoosestomarkeachnouninitseparatelyasNNP ,including journaland

medicine,whichmightotherwisebelabeledascommonnouns(NN).
Some tagging distinctions are quite hard for both humans and machines to make.
Forexampleprepositions(IN),particles(RP),andadverbs (RB)canhavealargeover-
lap. Wordslike aroundcanbeall three:
(5.4)Mrs./NNPShaefer/NNPnever/RB got/VBD around/RP to/TOjoining/VBG
(5.5)All/DTwe/PRPgotta/VBNdo/VB is/VBZgo/VB around/IN the/DTcorner/NN
(5.6)Chateau/NNPPetrus/NNPcosts/VBZ around/RB 250/CD
Making these decisions requires sophisticated knowledge o f syntax; tagging man-

uals (Santorini, 1990) give various heuristics that can hel p human coders make these
decisions,andthatcanalsoprovideusefulfeaturesforaut omatictaggers. Forexample
twoheuristicsfromSantorini(1990)arethatprepositions generallyareassociatedwith
afollowingnounphrase(althoughtheyalsomaybefollowedb yprepositionalphrases),

DRAFT132 Chapter 5. Part-of-SpeechTagging
and that the word aroundis taggedas an adverbwhen it means“approximately”. Fur-
thermore, particles often can either precede or follow a nou n phrase object, as in the
followingexamples:
(5.7)Shetoldoff/RPherfriends
(5.8)Shetoldherfriendsoff/RP.
Prepositions,ontheotherhand,cannotfollowtheirnounph rase(*isusedheretomark
anungrammaticalsentence,a conceptwhichwewill returnto in Ch.12):
(5.9)Shesteppedoff/INthe train
(5.10)*Shesteppedthetrainoff/IN.
Another difﬁculty is labeling the words that can modify noun s. Sometimes the
modiﬁersprecedingnounsare commonnounslike cottonbelow,othertimesthe Tree-
bank tagging manual speciﬁes that modiﬁers be tagged as adje ctives (for example if
the modiﬁerisa hyphenatedcommonnounlike income-tax )andothertimesas proper
nouns(formodiﬁerswhicharehyphenatedpropernounslike Gramm-Rudman ):
(5.11)cotton/NNsweater/NN
(5.12)income-tax/JJreturn/NN
(5.13)the/DTGramm-Rudman/NPAct/NP

Somewordsthat canbe adjectives,commonnouns,orproperno uns,are taggedin
theTreebankascommonnounswhenactingasmodiﬁers:
(5.14)Chinese/NNcooking/NN
(5.15)Paciﬁc/NN waters/NNS
A third known difﬁculty in tagging is distinguishing past pa rticiples (VBN) from
adjectives (JJ). A word like marriedis a past participle when it is being used in an
eventive, verbal way, as in (5.16) below, and is an adjective when it is being used to
expressa property,asin (5.17):
(5.16)Theyweremarried/VBNbythe JusticeofthePeace yesterdaya t5:00.

(5.17)At thetime,she wasalreadymarried/JJ.
Tagging manuals like Santorini (1990) give various helpful criteria for deciding
how‘verb-like’or‘eventive’a particularwordisina speci ﬁccontext.
ThePennTreebanktagsetwasculledfromtheoriginal87-tag tagsetfortheBrown
corpus. Thisreducedsetleavesoutinformationthatcanber ecoveredfromtheidentity
of the lexical item. For example the original Brown and C5 tag sets include a separate
tag for each of the different forms of the verbs do(e.g. C5 tag “VDD” for didand
“VDG”for doing),be,andhave. ThesewereomittedfromtheTreebankset.

CertainsyntacticdistinctionswerenotmarkedinthePennT reebanktagsetbecause
Treebanksentenceswereparsed,notmerelytagged,andsoso mesyntacticinformation
is represented in the phrase structure. For example, the sin gle tag IN is used for both
prepositions and subordinating conjunctions since the tre e-structure of the sentence
disambiguatesthem (subordinatingconjunctionsalways pr ecede clauses, prepositions
precede noun phrases or prepositional phrases). Most taggi ng situations, however, do
notinvolveparsedcorpora;forthisreasonthePennTreeban ksetisnotspeciﬁcenough

DRAFTSection5.3. Part-of-SpeechTagging 133
for many uses. The original Brown and C5 tagsets, for example , distinguish preposi-
tions(IN)fromsubordinatingconjunctions(CS), asinthef ollowingexamples:
(5.18)after/CS spending/VBGa/AT day/NNat/IN the/ATBrown/NPPalace/NN
(5.19)after/INa/ATwedding/NNtrip/NN to/INCorpus/NPChristi/NP ./.
The original Brown and C5 tagsets also have two tags for the wo rdto; in Brown
theinﬁnitiveuseistaggedTO,whiletheprepositionalusea sIN:
(5.20)to/TOgive/VBpriority/NN to/INteacher/NNpay/NNraises/NNS
Brown also has the tag NR for adverbial nouns like home,west,Monday, andto-
morrow. Because the Treebank lacks this tag, it has a much less consi stent policy for
adverbialnouns; Monday,Tuesday,andotherdaysoftheweekaremarkedNNP, tomor-
row,west, andhomeare marked sometimes as NN, sometimes as RB. This makes the
Treebanktagsetlessusefulforhigh-levelNLP tasksliketh edetectionoftimephrases.
Nonetheless,the Treebanktagset hasbeen the most widely us ed in evaluatingtag-

gingalgorithms,andsomanyofthealgorithmswedescribebe lowhavebeenevaluated
mainly on this tagset. Of course whether a tagset is useful fo r a particular application
dependsonhowmuchinformationtheapplicationneeds.
5.3 Part-of-SpeechTagging
Part-of-speech tagging (or just taggingfor short) is the process of assigning a part- tagging
of-speech or other syntactic class marker to each word in a co rpus. Because tags are
generallyalsoappliedto punctuation,taggingrequiresth atthepunctuationmarks(pe-

riod, comma, etc) be separated off of the words. Thus tokenization of the sort de-
scribedinCh.3isusuallyperformedbefore,oraspartof,th etaggingprocess,separat-
ing commas, quotation marks, etc., from words, and disambig uating end-of-sentence
punctuation (period, question mark, etc) from part-of-wor d punctuation (such as in
abbreviationslike e.g.andetc.)
The input to a tagging algorithm is a string of words and a spec iﬁed tagset of the
kinddescribedintheprevioussection. Theoutputisasingl ebesttagforeachword. For

example,herearesomesamplesentencesfromtheATIScorpus ofdialoguesaboutair-
travel reservationsthat we will discuss in Ch. 12. For each w e have shown a potential
taggedoutputusingthePennTreebanktagset deﬁnedinFig.5 .2onpage131:
(5.21)Book/VBthat/DTﬂight/NN./.
(5.22)Does/VBZ that/DTﬂight/NNserve/VBdinner/NN?/.
The previous section discussed some tagging decisions that are difﬁcult to make
for humans. Even in these simple examples, automatically as signing a tag to each
word is not trivial. For example, bookisambiguous . That is, it has more than one ambiguous

possible usage and part-of-speech. It can be a verb (as in bookthat ﬂight orto book
the suspect ) or a noun(as in handme that book , ora bookof matches ). Similarly that
can be a determiner (as in Does that ﬂight serve dinner ), or a complementizer(as in I
thought that your ﬂight was earlier ). The problem of POS-tagging is to resolvethese resolve

DRAFT134 Chapter 5. Part-of-SpeechTagging
Tag Description Example
( opening parenthesis (,[
) closing parenthesis ),]
* negator not n’t
, comma ,
– dash –
. sentence terminator . ;? !
: colon :
ABL pre-qualiﬁer quite, rather, such
ABN pre-quantiﬁer half, all,
ABX pre-quantiﬁer, double conjunction both
AP post-determiner many, next,several, last
AT article a the anno aevery
BE/BED/BEDZ/BEG/BEM/BEN/BER/BEZ be/were/was/being/am/been/are/is
CC coordinating conjunction and or buteither neither
CD cardinal numeral two, 2, 1962, million
CS subordinating conjunction that as afterwhether before
DO/DOD/DOZ do, did, does
DT singular determiner, this,that
DTI singular or plural determiner some, any
DTS plural determiner these those them
DTX determiner, double conjunction either, neither
EX existential there there
HV/HVD/HVG/HVN/HVZ have, had, having, had, has
IN preposition of infor bytoonat
JJ adjective
JJR comparative adjective better, greater,higher, larger ,lower

JJS semantically superlative adj. main, top, principal, ch ief, key, foremost
JJT morphologically superlative adj. best, greatest, high est, largest,latest,worst
MD modal auxiliary would, will,can, could, may, must, shoul d
NN (common) singular or mass noun time,world, work, school, family, door
NN$ possessive singular common noun father’s, year’s, city ’s,earth’s
NNS plural common noun years, people, things, children, pro blems
NNS$ possessive plural noun children’s, artist’sparent’s years’

NP singular proper noun Kennedy, England, Rachel, Congress
NP$ possessive singular proper noun Plato’sFaulkner’s Vio la’s
NPS plural proper noun Americans, Democrats, Chinese,
NPS$ possessive plural proper noun Yankees’, Gershwins’ Ea rthmen’s
NR adverbial noun home, west,tomorrow, Friday, North,
NR$ possessive adverbial noun today’s, yesterday’s, Sunda y’s, South’s
NRS plural adverbial noun Sundays Fridays
OD ordinal numeral second, 2nd, twenty-ﬁrst,mid-twentiet h

PN nominal pronoun one, something, nothing, anyone, none,
PN$ possessive nominal pronoun one’s someone’s anyone’s
PP$ possessive personal pronoun his their her itsmyour your
PP$$ second possessive personal pronoun mine, his, ours, yo urs, theirs
PPL singular reﬂexive personal pronoun myself, herself
PPLS plural reﬂexive pronoun ourselves, themselves
PPO objective personal pronoun me, us, him
PPS 3rd. sg. nominative pronoun he, she, it
PPSS other nominative pronoun I,we, they
QL qualiﬁer very, too, most, quite, almost,extremely

QLP post-qualiﬁer enough, indeed
RB adverb
RBR comparative adverb later,more, better, longer, furthe r
RBT superlative adverb best, most, highest, nearest
RN nominal adverb here, then
Figure5.7 Firstpart of original 87-tag Browncorpus tagset (Francis a ndKuˇ cera, 1982).

DRAFTSection5.3. Part-of-SpeechTagging 135
RP adverbor particle across, off, up
TO inﬁnitivemarker to
UH interjection,exclamation well,oh, say, please, okay, u h, goodbye
VB verb, base form make, understand, try, determine, drop
VBD verb, past tense said, went, looked, brought, reached ke pt
VBG verb, present participle, gerund getting, writing, inc reasing
VBN verb, past participle made, given, found, called,requi red
VBZ verb, 3rdsingular present says, follows, requires, tra nscends
WDT wh-determiner what, which
WP$ possessive wh- pronoun whose
WPO objective wh- pronoun whom, which, that
WPS nominative wh- pronoun who, which, that
WQL how
WRB wh-adverb how, when
Figure5.8 Restof 87-tag Browncorpus tagset (Francisand Kuˇ cera, 198 2).
ambiguities,choosingthepropertagforthecontext. Part- of-speechtaggingisthusone
ofthemany disambiguation taskswewill see inthisbook. disambiguation
How hard is the tagging problem? The previous section descri bed some difﬁcult

WQL how
tagging decisions; how commonis tag ambiguity? It turns out that most wordsin En-
glishareunambiguous;i.e.,theyhaveonlyasingletag. But manyofthemostcommon
wordsof Englishare ambiguous(for example cancan be an auxiliary(‘to be able’),a
noun(‘ametalcontainer’),oraverb(‘toputsomethinginsu chametalcontainer’)). In
fact,DeRose(1988)reportsthatwhileonly11.5%ofEnglish wordtypesintheBrown
corpus are ambiguous, over 40% of Brown tokens are ambiguous . Fig. 5.10 shows

the number of word types with different levels of part-of-sp eech ambiguity from the
Brown corpus. We show these computations from two versions o f the tagged Brown
corpus, the original tagging done at Brown by Francis and Kuˇ cera (1982), and the
Treebank-3 tagging done at the University of Pennsylvania. Note that despite having
morecoarse-grainedtags,the45-tagcorpusunexpectedlyh asmoreambiguitythanthe
87-tagcorpus.
Luckily, it turns out that many of the 40% ambiguous tokens ar e easy to disam-

biguate. Thisisbecausethe varioustagsassociatedwith a w ordarenot equallylikely.
Forexample, acanbeadeterminer,ortheletter a(perhapsaspartofanacronymoran
initial). But thedeterminersense of aismuchmorelikely.
Mosttaggingalgorithmsfallintooneoftwoclasses: rule-based taggersand stochas- rule-based
tic taggers taggers. Rule-based taggers generally involve a large data base of hand- stochastic taggers
written disambiguation rules which specify, for example, t hat an ambiguous word is

a noun rather than a verb if it follows a determiner. The next s ection will describe a
sample rule-based tagger, EngCG, based on the Constraint Grammar architecture of
Karlssonet al.(1995b).
Stochastic taggers generally resolve tagging ambiguities by using a training cor-
pus to compute the probability of a given word having a given t ag in a given context.
Sec.5.5describestheHiddenMarkovModelor HMMtagger . HMM tagger
Finally, Sec. 5.6 will describe an approach to tagging calle d thetransformation-

based tagger or theBrill tagger , after Brill (1995). The Brill tagger shares features Brill tagger
of both tagging architectures. Like the rule-based tagger, it is based on rules which
determine when an ambiguousword should have a given tag. Lik e the stochastic tag-

DRAFT136 Chapter 5. Part-of-SpeechTagging
AJ0 adjective (unmarked) good,old
AJC comparative adjective better,older
AJS superlative adjective best,oldest
AT0 article the,a,an
AV0 adverb (unmarked) often,well,longer, furthest
AVP adverb particle up, off,out
AVQ wh-adverb when, how, why
CJC coordinating conjunction and, or
CJS subordinating conjunction although, when
CJT the conjunction that
CRD cardinal numeral (except one) 3,twenty-ﬁve ,734
DPS possessive determiner your, their
DT0 general determiner these, some
DTQ wh-determiner whose, which
EX0 existential there
ITJ interjectionor other isolate oh, yes,mhm
NN0 noun (neutral for number) aircraft,data
NN1 singular noun pencil, goose
NN2 plural noun pencils, geese
NP0 proper noun London, Michael, Mars
ORD ordinal sixth,77th, last
PNI indeﬁnite pronoun none, everything
PNP personal pronoun you, them,ours
PNQ wh-pronoun who, whoever
PNX reﬂexive pronoun itself,ourselves
POS possessive ’sor’
PRF the preposition of

PRP preposition (except of) for,above, to
PUL punctuation –left bracket (or [
PUN punctuation –general mark . ! ,: ;- ? ...
PUQ punctuation –quotation mark ‘’ ”
PUR punctuation –right bracket )or ]
TO0 inﬁnitivemarker to
UNC unclassiﬁeditems (not English)
VBB base forms of be(except inﬁnitive) am,are
VBD past form of be was,were
VBG -ingform of be being
VBI inﬁnitiveof be
VBN past participle of be been
VBZ -sform of be is,’s
VDB/D/G/I/N/Z form of do do, does, did, doing, todo, etc.

VHB/D/G/I/N/Z form of have have, had, having, tohave, etc.
VM0 modal auxiliary verb can, could, will,’ll
VVB base form of lexical verb(except inﬁn.) take, live
VVD past tense form of lexical verb took, lived
VVG -ingform of lexicalverb taking, living
VVI inﬁnitiveof lexical verb take, live
VVN past participle form of lex. verb taken, lived
VVZ -sform of lexical verb takes, lives
XX0 the negative notorn’t
ZZ0 alphabetical symbol A,B,c,d

Figure5.9 UCREL’sC5tagset for the BritishNational Corpus (Garside e tal.,1997).
gers, it hasa machine-learningcomponent: the rulesareaut omaticallyinducedfroma
previouslytaggedtrainingcorpus.

DRAFTSection5.4. Rule-Based Part-of-SpeechTagging 137
87-tagOriginalBrown 45-tagTreebankBrown
Unambiguous(1tag) 44,019 38,857
Ambiguous(2–7tags) 5,490 8844
Details: 2tags 4,967 6,731
3tags 411 1621
4tags 91 357
5tags 17 90
6tags 2 ( well,beat) 32
7tags 2 ( still,down) 6 ( well,set,round,
open,ﬁt,down)
8tags 4 ( ’s,half,back,a)
9tags 3 ( that,more,in)
Figure5.10 The amount of tag ambiguity for word types in the Brown corpus , from the
ICAME release of the original (87-tag) tagging and the Treeb ank-3 (45-tag) tagging. Numbers
are not strictly comparable because only the Treebank segme nts’s. An earlier estimate of some
of these numbers is reportedinDeRose (1988).
5.4 Rule-BasedPart-of-SpeechTagging
Theearliestalgorithmsforautomaticallyassigningpart- of-speechwerebasedonatwo-
stage architecture(Harris,1962;KleinandSimmons,1963; GreeneandRubin,1971).
Theﬁrst stageusedadictionarytoassigneachwordalist ofp otentialparts-of-speech.
Thesecondstageusedlargelistsofhand-writtendisambigu ationrulestowinnowdown

thislist toa singlepart-of-speechforeachword.
Modern rule-based approaches to part-of-speech tagging ha ve a similar architec-
ture, although the dictionaries and the rule sets are vastly larger than in the 1960’s.
One of the most comprehensiverule-basedapproachesis the C onstraint Grammarap-
proach (Karlsson et al., 1995a). In this section we describe a tagger based on this
approach,the EngCGtagger(Voutilainen,1995,1999). EngCG
TheEngCGENGTWOLlexiconisbasedonthetwo-levelmorpholo gydescribedin

Ch.3, andhasabout56,000entriesforEnglishwordstems(He ikkil¨ a,1995),counting
awordwithmultipleparts-of-speech(e.g.,nominalandver balsensesof hit)asseparate
entries, and not counting inﬂected and many derived forms. E ach entry is annotated
withasetofmorphologicalandsyntacticfeatures. Fig.5.4 showssomeselectedwords,
togetherwitha slightlysimpliﬁedlistingoftheirfeature s,asusedinrulewriting.
Most of the features in Fig. 5.4 are relatively self-explana tory; SG for singular,

-SG3 for other than third-person-singular. NOMINATIVE mea ns non-genitive and
PCP2 meanspast participle. PRE, CENTRAL, and POST are order ingslots for deter-
miners (predeterminers ( all) come before determiners ( the):all the president’s men ).
NOINDEFDETERMINER means that words like furniture do not appear with the in-
deﬁnite determiner a. SV, SVO, and SVOO specify the subcategorization orcom- subcategorization
plementation pattern for the verb. Subcategorization will be discussed i n Ch. 12 and complementation

Ch.16,butbrieﬂySVmeanstheverbappearssolelywithasubj ect(nothingoccurred );
SVO with a subject and an object ( I showed the ﬁlm ); SVOO with a subject and two
complements: Sheshowedherthe ball .

DRAFT138 Chapter 5. Part-of-SpeechTagging
Word POS AdditionalPOSfeatures
smaller ADJ COMPARATIVE
fast ADV SUPERLATIVE
that DET CENTRALDEMONSTRATIVESG
all DET PREDETERMINERSG/PLQUANTIFIER
dog’s N GENITIVESG
furniture N NOMINATIVESGNOINDEFDETERMINER
one-third NUM SG
she PRON PERSONALFEMININENOMINATIVESG3
show V PRESENT-SG3VFIN
show N NOMINATIVESG
shown PCP2 SVOOSVOSV
occurred PCP2 SV
occurred V PASTVFINSV
Figure5.11 Lexical entries inthe ENGTWOLlexicon(Voutilainen, 1995; Heikkil¨ a,1995).
Intheﬁrststageofthetagger,eachwordisrunthroughthetw o-levellexicontrans-
ducer and the entries for all possible parts-of-speech are r eturned. For example the
phrasePavlov had shown that salivation ... would return the following list (one line
perpossibletag,withthecorrecttag showninboldface):
Pavlov PAVLOV N NOMSGPROPER
had HAVE VPASTVFIN SVO
HAVEPCP2SVO
shown SHOWPCP2SVOOSVOSV
that ADV
PRONDEMSG
DETCENTRALDEMSG
CS
salivation NNOM SG
...
EngCG then applies a large set of constraints (as many as 3,74 4 constraints in

that ADV
PRONDEMSG
CS
...
the EngCG-2 system) to the input sentence to rule out incorre ct parts-of-speech. The
boldfaced entries in the table above show the desired result , in which the simple past
tensetag(ratherthanthepastparticipletag)isappliedto had,andthecomplementizer
(CS) tag is applied to that. The constraints are used in a negative way, to eliminate
tags that are inconsistent with the context. For example one constraint eliminates all
readings of thatexcept the ADV (adverbial intensiﬁer) sense (this is the sen se in the

sentenceit isn’tthat odd). Here’sa simpliﬁedversionoftheconstraint:
ADVERBIAL -THAT RULE
Given input :“that”
if
(+1A/ADV/QUANT); /*ifnext wordis adj, adverb, or quantiﬁer */
(+2SENT-LIM); /*and followingwhichis asentence boundary, */
(NOT-1SVOC/A); /*and the previous wordis not averblike */
/*‘consider’ whichallows adjs as object complements */
theneliminate non-ADV tags
elseeliminate ADV tag

DRAFTSection5.5. HMM Part-of-SpeechTagging 139
The ﬁrst two clauses of this rule check to see that the thatdirectly precedes a
sentence-ﬁnal adjective, adverb, or quantiﬁer. In all othe r cases the adverb reading
is eliminated. The last clause eliminates cases preceded by verbs like considerorbe-
lievewhich can take a noun and an adjective; this is to avoid taggin g the following
instanceof thatasanadverb:
Iconsiderthatodd.
Anotherruleisusedtoexpresstheconstraintthatthecompl ementizersenseof that
is most likely to be used if the previous word is a verb which ex pects a complement
(likebelieve,think,orshow),andifthatisfollowedbythebeginningofanounphrase,
andaﬁnite verb.
This description oversimpliﬁes the EngCG architecture; th e system also includes
probabilisticconstraints,andalsomakesuseofothersynt acticinformationwe haven’t
discussed. TheinterestedreadershouldconsultKarlssone tal.(1995b)andVoutilainen
(1999).
5.5 HMMPart-of-SpeechTagging

(1999).
The use of probabilitiesin tags is quite old; probabilities in taggingwere ﬁrst used by
Stolzet al.(1965),acompleteprobabilistictaggerwithVi terbidecodingwassketched
by Bahl and Mercer (1976), and various stochastic taggers we re built in the 1980s
(Marshall,1983;Garside, 1987;Church,1988;DeRose,1988 ). Thissectiondescribes
aparticularstochastictaggingalgorithmgenerallyknown astheHiddenMarkovModel
orHMMtagger. HiddenMarkovModelsthemselveswillbemoref ullyintroducedand
deﬁnedinCh.6. Inthissection,wepreﬁgureCh.6abitbygivi nganinitialintroduction

tothe HiddenMarkovModelasappliedtopart-of-speechtagg ing.
UseofaHiddenMarkovModeltodopart-of-speech-tagging,a swewilldeﬁneit,is
aspecialcaseof Bayesianinference ,aparadigmthathasbeenknownsincetheworkofBayesian
inference
Bayes (1763). Bayesian inference or Bayesian classiﬁcatio n was applied successfully
to language problems as early as the late 1950s, including th e OCR work of Bledsoe
in 1959, and the seminal work of Mosteller and Wallace (1964) on applyingBayesian
inferencetodeterminetheauthorshipoftheFederalistpap ers.

inference
Inaclassiﬁcationtask,wearegivensomeobservation(s)an dourjobistodetermine
which of a set of classes it belongsto. Part-of-speechtaggi ngis generallytreated as a
sequence classiﬁcation task. So here the observationis a se quence of words (let’s say
a sentence),andit isourjobtoassignthema sequenceofpart -of-speechtags.
Forexample,saywe aregivena sentencelike
(5.23)Secretariatisexpectedto racetomorrow.
What is the best sequence of tags which corresponds to this se quence of words?

The Bayesian interpretation of this task starts by consider ing all possible sequences
of classes—in this case, all possible sequences of tags. Out of this universe of tag
sequences, we want to choose the tag sequence which is most pr obable given the ob-
servation sequence of nwordswn
1. In other words, we want, out of all sequences of n

DRAFT140 Chapter 5. Part-of-SpeechTagging
tagstn
1thesingletag sequencesuchthat P(tn
1|wn
1)ishighest. We usethe hatnotation ˆ ˆ
tomean“ourestimate ofthecorrecttagsequence”.
ˆtn
1=argmax
tn
1P(tn
1|wn
1) (5.24)
The function argmaxxf(x)means “the xsuch that f(x)is maximized”. Eq. 5.24
thus means, out of all tag sequences of length n, we want the particular tag sequence
tn
1which maximizes the right-hand side. While (5.24) is guaran teed to give us the
optimaltagsequence,itisnotclearhowtomaketheequation operational;thatis,fora
giventag sequence tn
1and word sequence wn
1, we don’t knowhow to directly compute
P(tn
1|wn
1).
The intuition of Bayesian classiﬁcation is to use Bayes’ rul e to transform (5.24)
into a set of other probabilities which turn out to be easier t o compute. Bayes’ rule is
presentedin(5.25);it givesusa wayto breakdownanycondit ionalprobability P(x|y)
intothreeotherprobabilities:
P(x|y) =P(y|x)P(x)
P(y)(5.25)
We canthensubstitute(5.25)into(5.24)toget(5.26):
ˆtn
1=argmax
tn
1P(wn
1|tn
1)P(tn
1)

P(tn
1|wn
1).
ˆtn
1=argmax
tn
1P(wn
1|tn
1)P(tn
1)
P(wn
1)(5.26)
We can conveniently simplify 5.26 by dropping the denominat orP(wn
1). Why is
that? Sincewearechoosingatagsequenceoutofalltagseque nces,wewillbecomput-
ingP(wn
1|tn
1)P(tn
1)
P(wn
1)foreachtagsequence. But P(wn
1)doesn’tchangeforeachtagsequence;
we are always asking aboutthe most likely tag sequencefor th e same observation wn
1,
which must have the same probability P(wn
1). Thus we can choose the tag sequence
whichmaximizesthissimplerformula:
ˆtn
1=argmax
tn
1P(wn
1|tn
1)P(tn
1) (5.27)

1)
P(wn
1)(5.26)
1). Why is
ingP(wn
1|tn
1)P(tn
1)
P(wn
1,
ˆtn
1=argmax
tn
1P(wn
1|tn
1)P(tn
1) (5.27)
To summarize, the most probable tag sequence ˆtn
1given some word string wn
1can
be computed by taking the product of two probabilities for ea ch tag sequence, and
choosing the tag sequence for which this product is greatest . The two terms are the
prior probability of the tag sequence P(tn
1)), and the likelihood of the word string prior probability
likelihood P(wn
1|tn
1):
ˆtn
1=argmax
tn
1likelihood
P(wn
1|tn
1)prior
P(tn
1) (5.28)

ˆtn
1=argmax
tn
1P(wn
1|tn
1)P(tn
1) (5.27)
1can
1|tn
1):
ˆtn
1=argmax
tn
P(wn
1|tn
P(tn
1) (5.28)
Unfortunately,(5.28)is still too hard to compute directly . HMM taggers therefore
make two simplifying assumptions. The ﬁrst assumption is th at the probability of a
word appearingis dependentonly on its own part-of-speecht ag; that it is independent
ofotherwordsaroundit, andoftheothertagsaroundit:

DRAFTSection5.5. HMM Part-of-SpeechTagging 141
P(wn
1|tn
1)≈n
∏
i=1P(wi|ti) (5.29)
Thesecondassumptionis that the probabilityof a tag appear ingis dependentonly
ontheprevioustag,the bigramassumptionwesaw inCh. 4:
P(tn
1)≈n
∏
i=1P(ti|ti−1) (5.30)
Plugging the simplifying assumptions (5.29) and (5.30) int o (5.28) results in the
followingequationbywhichabigramtaggerestimatesthemo stprobabletagsequence:
ˆtn
1=argmax
tn
1P(tn
1|wn
1)≈argmax
tn
1n
∏
i=1P(wi|ti)P(ti|ti−1) (5.31)
Eq. 5.31 contains two kinds of probabilities, tag transitio n probabilities and word
likelihoods. Let’s take a moment to see what these probabili ties represent. The tag
transition probabilities, P(ti|ti−1), represent the probabilityof a tag giventhe previous
tag. For example, determiners are very likely to precede adj ectives and nouns, as in
sequenceslike that/DT ﬂight/NN andthe/DT yellow/JJhat/NN . Thuswe wouldexpect
the probabilities P(NN|DT)andP(JJ|DT)to be high. But in English, adjectivesdon’t

tendtoprecededeterminers,so theprobability P(DT|JJ)oughtto below.
We can compute the maximum likelihood estimate of a tag trans ition probability
P(NN|DT)by taking a corpus in which parts-of-speech are labeled and c ounting, out
of the timeswe see DT, how manyofthose times we see NN after th e DT. That is, we
computethefollowingratioofcounts:
P(ti|ti−1) =C(ti−1,ti)
C(ti−1)(5.32)
Let’s choose a speciﬁc corpus to examine. For the examples in this chapter we’ll

usetheBrowncorpus,the1millionwordcorpusofAmericanEn glishdescribedearlier.
TheBrowncorpushasbeentaggedtwice,onceinthe1960’swit hthe87-tagtagset,and
againinthe1990’swiththe45-tagTreebanktagset. Thismak esitusefulforcomparing
tagsets,andisalso widelyavailable.
In the 45-tag TreebankBrown corpus, the tag DT occurs116,45 4times. Of these,
DT is followed by NN 56,509 times (if we ignore the few cases of ambiguous tags).
ThustheMLEestimateofthe transitionprobabilityiscalcu latedasfollows:
P(NN|DT) =C(DT,NN)
C(DT)=56,509
116,454=.49 (5.33)

The probability of getting a common noun after a determiner, .49, is indeed quite
high,aswe suspected.
Thewordlikelihoodprobabilities, P(wi|ti), representtheprobability,giventhatwe
see agiventag,that itwill beassociatedwithagivenword. F orexampleifwe wereto
seethetagVBZ(thirdpersonsingularpresentverb)andgues stheverbthatislikelyto

DRAFT142 Chapter 5. Part-of-SpeechTagging
have that tag, we might likely guess the verb is, since the verb to beis so common in
English.
WecancomputetheMLEestimateofawordlikelihoodprobabil itylikeP(is|VBZ)
againby counting,out of the times we see VBZ in a corpus,how m anyof those times
theVBZ islabelingthe word is. Thatis, wecomputethefollowingratioofcounts:
P(wi|ti) =C(ti,wi)
C(ti)(5.34)
In Treebank Brown corpus, the tag VBZ occurs 21,627 times, an d VBZ is the tag
foris10,073times. Thus:
P(is|VBZ) =C(VBZ,is)
C(VBZ)=10,073
21,627=.47 (5.35)
Forthosereaderswhoarenew toBayesianmodelingnotethatt hislikelihoodterm
is not asking “which is the most likely tag for the word is”. That is, the term is not
P(VBZ|is). Instead we are computing P(is|VBZ). The probability,slightly counterin-
tuitively,answersthequestion“Ifwewereexpectingathir dpersonsingularverb,how
likelyisit thatthisverbwouldbe is?”.
We have now deﬁnedHMM taggingas a task of choosinga tag-sequ encewith the

maximumprobability,derivedtheequationsbywhichwewill computethisprobability,
andshownhowtocomputethecomponentprobabilities. Infac twehavesimpliﬁedthe
presentationoftheprobabilitiesin manyways; inlater sec tionswe will returnto these
equationsandintroducethedeletedinterpolationalgorit hmforsmoothingthesecounts,
thetrigrammodeloftaghistory,anda modelforunknownword s.
Butbeforeturningtotheseaugmentations,weneedtointrod ucethedecodingalgo-
rithmbywhichtheseprobabilitiesarecombinedtochooseth emostlikelytagsequence.

5.5.1 Computing the most-likelytag sequence: Anexample
The previous section showed that the HMM tagging algorithm c hooses as the most
likelytagsequencetheonethatmaximizestheproductoftwo terms;theprobabilityof
the sequenceof tags,andthe probabilityofeachtag generat inga word. Inthissection
wegroundtheseequationsinaspeciﬁcexample,showingfora particularsentencehow
the correct tag sequence achieves a higher probability than one of the many possible
wrongsequences.
We will focus on resolving the part-of-speech ambiguity of t he wordrace, which

canbeanounorverbinEnglish,asweshowintwoexamplesmodi ﬁedfromtheBrown
andSwitchboardcorpus. Forthisexample,wewillusethe87- tagBrowncorpustagset,
becauseithasaspeciﬁctagfor to,TO,usedonlywhen toisaninﬁnitive;prepositional
usesoftoaretaggedasIN.Thiswill comeinhandyinourexample.1
In(5.36)raceisa verb(VB)whilein (5.37) raceisa commonnoun(NN):
(5.36)Secretariat/NNPis/BEZ expected/VBNto/TO race/VBtomorrow/NR
1The 45-tag Treebank-3 tagset does make this distinction in t he Switchboard corpus but not, alas, in the

Brown corpus. Recall that in the 45-tag tagset time adverbs l iketomorrow are tagged as NN; in the 87-tag
tagset they appear as NR.

DRAFTSection5.5. HMM Part-of-SpeechTagging 143
(5.37)People/NNScontinue/VBto/TOinquire/VBthe/ATreason/NN for/INthe/AT
race/NNfor/INouter/JJspace/NN
Let’slookat how racecanbe correctlytaggedas a VBinsteadof anNN in (5.36).
HMM part-of-speechtaggersresolve this ambiguitygloball yrather than locally, pick-
ing the best tag sequence for the whole sentence. There are ma ny hypotheticallypos-
sible tag sequences for (5.36), since there are other ambigu ities in the sentence (for
exampleexpectedcanbeanadjective(JJ),apasttense/preterite(VBD)orapa stpartici-
ple (VBN)). But let’s just considertwo of the potential sequ ences,shown in Fig. 5.12.
Note that these sequences differ only in one place; whether t he tag chosen for raceis
VBorNN.
isNNP VBZ VBN TO VB NR
Secretariat expected to race tomorrow(a)
isNNP VBZ VBN TO NN NR
Secretariat expected to race tomorrow(b)
Figure5.12 Two of the possible sequences of tags corresponding to the Se cretariat sentence,

VBorNN.
one of them corresponding to the correct sequence, in which raceis a VB. Each arc in these
graphs would be associated with a probability. Note that the two graphs differ only in 3 arcs,
hence in3probabilities.
Almost all the probabilities in these two sequences are iden tical; in Fig. 5.12 we
have highlighted in boldface the three probabilities that d iffer. Let’s consider two of
these, correspondingto P(ti|ti−1)andP(wi|ti). The probability P(ti|ti−1)in Fig. 5.12a
isP(VB|TO),while inFig.5.12bthetransitionprobabilityis P(NN|TO).

Thetagtransitionprobabilities P(NN|TO)andP(VB|TO)giveustheanswertothe
question “How likely are we to expect a verb (noun) given the p revious tag?” As we
saw in the previous section, the maximum likelihood estimat e for these probabilities
canbederivedfromcorpuscounts.
Sincethe(87-tagBrowntagset)tagTOisusedonlyfortheinﬁ nitivemarker to,we
expect that only a very small numberof nouns can follow this m arker (as an exercise,
try to think of a sentence where a noun can follow the inﬁnitiv e marker use of to).

Sure enough,a look at the (87-tag)Brown corpusgivesusthe f ollowingprobabilities,
showingthatverbsareabout500timesaslikelyasnounstooc curafterTO:
P(NN|TO) =.00047
P(VB|TO) =.83

DRAFT144 Chapter 5. Part-of-SpeechTagging
Let’s now turn to P(wi|ti), the lexical likelihood of the word racegiven a part-of-
speechtag. ForthetwopossibletagsVBandNN,thesecorresp ondtotheprobabilities
P(race|VB)andP(race|NN). HerearethelexicallikelihoodsfromBrown:
P(race|NN) =.00057
P(race|VB) =.00012
Finally,weneedtorepresentthetagsequenceprobabilityf orthefollowingtag(inthis
case thetagNRfor tomorrow ):
P(NR|VB) =.0027
P(NR|NN) =.0012
If we multiply the lexical likelihoods with the tag sequence probabilities, we see
that the probability of the sequence with the VB tag is higher and the HMM tagger
correctlytags raceasaVBinFig.5.12despitethefactthatitisthelesslikelys enseof
race:
P(VB|TO)P(NR|VB)P(race|VB) =.00000027
P(NN|TO)P(NR|NN)P(race|NN) =.00000000032
5.5.2 FormalizingHiddenMarkov Model taggers
Nowthatwehaveseentheequationsandsomeexamplesofchoos ingthemostprobable
tagsequence,weshowabriefformalizationofthisproblema saHiddenMarkovModel
(seeCh. 6forthemorecompleteformalization).

race:
The HMM is an extension of the ﬁnite automata of Ch. 3. Recall t hat a ﬁnite
automaton is deﬁned by a set of states, and a set of transition s between states that are
takenbasedonthe inputobservations. A weightedﬁnite-stateautomaton isa simple weighted
augmentationoftheﬁniteautomatoninwhicheacharcisasso ciatedwithaprobability,
indicating how likely that path is to be taken. The probabili ty on all the arcs leaving
a node must sum to 1. A Markov chain is a special case of a weighted automaton Markov chain

in which the input sequence uniquely determines which state s the automaton will go
through. Becausetheycan’trepresentinherentlyambiguou sproblems,aMarkovchain
isonlyusefulforassigningprobabilitiestounambiguouss equences.
While the Markov chain is appropriate for situations where w e can see the actual
conditioningevents, it is not appropriatein part-of-spee chtagging. This is because in
part-of-speech tagging, while we observe the words in the in put, we do notobserve

the part-of-speechtags. Thus we can’t condition any probab ilities on, say, a previous
part-of-speechtag,becausewecannotbecompletelycertai nexactlywhichtagapplied
to the previousword. A Hidden Markov Model (HMM) allowsusto talk aboutbothHiddenMarkovModel
observed events (like words that we see in the input) and hiddenevents (like part-of-
speechtags)thatwe thinkofascausalfactorsinourprobabi listicmodel.
AnHMMisspeciﬁedbythefollowingcomponents: HMM

theBobservation likelihoods, correspondingrespectively to t hepriorandlikelihood
probabilitiesthatwesawinequation(5.31). Fig.5.13illu stratesthepriorprobabilities
inanHMMpart-of-speechtagger,showing3samplestatesand someofthe Atransition
probabilitiesbetween them. Fig. 5.14 shows another view of an HMM part-of-speech
tagger, focusing on the word likelihoods B. Each hidden state is associated with a
vectoroflikelihoodsforeachobservationword.
5.5.3 The Viterbi Algorithm forHMMTagging
For anymodel,such asan HMM, that containshiddenvariables ,the task ofdetermin-
ingwhichsequenceofvariablesistheunderlyingsourceofs omesequenceofobserva-
tions is called the decoding task. The Viterbialgorithm is perhapsthe most common decoding
Viterbi decodingalgorithmused forHMMs, whether for part-of-spee chtaggingor forspeech
recognition. The term Viterbiis commonin speech and languageprocessing,but this

DRAFT146 Chapter 5. Part-of-SpeechTagging
Figure5.14 TheBobservation likelihoods for the HMM in the previous ﬁgure. E ach state
(except the non-emitting Start and End states) is associate d with a vector of probabilities, one
likelihood for eachpossible observation word.
is really a standard application of the classic dynamic programming algorithm, and
looksa lot like the minimum edit distance algorithmof Ch. 3. The Viterbialgorithm
was ﬁrst applied to speech and language processing in the con text of speech recogni-
tion by Vintsyuk (1968), but has what Kruskal (1983) calls a ‘ remarkable history of
multiple independentdiscoveryand publication’; see the H istory section at the end of
Ch.6 formoredetails.
The slightly simpliﬁed version of the Viterbi algorithm tha t we will present takes
asinputasingleHMMandasetofobservedwords O= (o1o2o3...oT)andreturnsthe
mostprobablestate/tag sequence Q= (q1q2q3...qT), togetherwithitsprobability.
Let the HMM be deﬁned by the two tables in Fig. 5.15 and Fig. 5.1 6. Fig. 5.15

expresses the aijprobabilities, the transition probabilities between hidden states (i.e.
part-of-speechtags). Fig. 5.16 expressesthe bi(ot)probabilities,the observation like-
lihoodsofwordsgiventags.
VB TO NN PPSS
<s> .019 .0043 .041 .067
VB .0038 .035 .047 .0070
TO .83 0 .00047 0
NN .0040 .016 .087 .0045
PPSS .23 .00079 .0012 .00014
Figure5.15 Tag transition probabilities (the aarray,p(ti|ti−1)) computed from the 87-tag
Brown corpus without smoothing. The rows are labeled with th e conditioning event; thus

P(PPSS|VB)is .0070. The symbol <s>is the start-of-sentence symbol.
Fig. 5.17 shows pseudocode for the Viterbi algorithm. The Vi terbi algorithm sets
up a probability matrix, with one column for each observatio ntand one row for each
state in the state graph. Each column thus has a cell for each s tateqiin the single
combinedautomatonforthefourwords.

DRAFTSection5.5. HMM Part-of-SpeechTagging 147
I want to race
VB 0 .0093 0 .00012
TO 0 0 .99 0
NN 0 .000054 0 .00057
PPSS .37 0 0 0
Figure5.16 Observation likelihoods (the barray) computed from the 87-tag Brown corpus
without smoothing.
function VITERBI(observations of lenT,state-graph of lenN)returnsbest-path
create a pathprobability matrix viterbi[N+2,T]
foreachstate sfrom1toNdo ;initializationstep
viterbi[s,1]←a0,s∗bs(o1)
backpointer [s,1]←0
foreachtime step tfrom2toTdo ;recursionstep
foreachstate sfrom1toNdo
viterbi[s,t]←Nmax
s′=1viterbi [s′,t−1]∗as′,s∗bs(ot)
backpointer [s,t]←Nargmax
s′=1viterbi [s′,t−1]∗as′,s
viterbi[qF,T]←Nmax
s=1viterbi [s,T]∗as,qF ;termination step
backpointer [qF,T]←Nargmax
s=1viterbi [s,T]∗as,qF ; terminationstep
returnthe backtrace path by following backpointers to states back in time from
backpointer [qF,T]
Figure5.17 Viterbi algorithm for ﬁnding optimal sequence of tags. Give n an observation
sequence andanHMM λ= (A,B),thealgorithmreturnsthestate-paththroughtheHMMwhich

assigns maximum likelihood to the observation sequence. No te that states 0 and qFare non-
emitting.
The algorithm ﬁrst creates Nor four state columns. The ﬁrst column corresponds
to the observation of the ﬁrst word i, the second to the second word want, the third to
the third word to, and the fourthto the fourthword race. We begin in the ﬁrst column
bysettingtheviterbivalueineachcelltotheproductofthe transitionprobability(into
it from the state state) and the observation probability (of the ﬁrst word); the reader

emitting.
shouldﬁndthisinFig.5.18.
Thenwemoveon,columnbycolumn;foreverystateincolumn1, wecomputethe
probabilityof movinginto each state in column2, andso on. F or each state qjat time
t, the value viterbi [s,t]is computed by taking the maximum over the extensionsof all
thepathsthat leadtothecurrentcell,followingthefollow ingequation:
vt(j) =Nmax
i=1vt−1(i)aijbj(ot) (5.38)
ThethreefactorsthataremultipliedinEq.5.38forextendi ngthepreviouspathsto
computetheViterbiprobabilityat time tare:

DRAFT148 Chapter 5. Part-of-SpeechTagging
vt−1(i)thepreviousViterbipathprobability fromtheprevioustime step
aijthetransitionprobability frompreviousstate qito currentstate qj
bj(ot)thestate observation likelihood of the observationsymbol otgiven
thecurrentstate j
startVB
PP
SSVB
PP
SSVB
PP
SSend
P(PPSS|start) * P(start).067 x 1.0 = .067v1(2) x P(VB|VB)
0 x .0038 = 0
v1(1) x P(VB|PPSS)
.025 x .23  = .0055 
P(VB|start)xP(start).019 x 1.0 = .019v1(2)=.019 x 0 = 0
v1(1) = .067 x .37 = .025v2(2)= max(0,0,0,.0055) x .0093 = .000051 
start start start
tPPS
SVBend end end qend
q2
q1
q0
o1i race want
o2o3VB
PP
SS
startend
toTO TO TO TO TONN NN NN NN NN
v1(3)=.0043 x 0 = 0v1(4)=.041 x 0=0
o4v1(3) * P(VB|TO)
0 x .83 = 0v1(4) * P(VB|NN)
0 x .0040 = 0
v0(0) = 1.0P(TO|start)xP(start).0043 x 1.0 = .0043P(NN|start)xP(start).041 x 1.0 = .041
backtrace
backtraceq3q4
Figure5.18 The entries in the individual state columns for the Viterbi a lgorithm. Each cell keeps the probability

tPPS
q2
q1
q0
o2o3VB
PP
SS
startend
0 x .0040 = 0
backtrace
ofthe best pathsofarand apointer tothe previous cell along thatpath. Wehave onlyﬁlledout columns 0and1and
one cell of column 2; the rest is left as an exercise for the rea der. After the cells are ﬁlled in, backtracing from the
endstate,we should be able toreconstruct the correct stateseq uence PPSS VB TO VB.
In Fig. 5.18, each cell of the trellis in the column for the wor dIis computed by
multiplying the previous probability at the start state (1. 0), the transition probability

from the start state to the tag for that cell, and the observat ion likelihood of the word
Igiven the tag for that cell. As it turns out, three of the cells are zero (since the word
Ican be neither NN, TO nor VB). Next, each cell in the wantcolumn gets updated
with the maximum probability path from the previous column. We have shown only

DRAFTSection5.5. HMM Part-of-SpeechTagging 149
the value for the VB cell. That cell gets the max of four values ; as it happens in this
case,threeofthemarezero(sincetherewerezerovaluesint hepreviouscolumn). The
remaining value is multiplied by the relevant transition pr obability, and the (trivial)
max is taken. In this case the ﬁnal value, .000051, comes from the PPSS state at the
previouscolumn.
The reader should ﬁll in the rest of the trellis in Fig. 5.18, a nd backtrace to recon-
structthecorrectstate sequence PPSS VB TO VB.
5.5.4 Extending the HMMalgorithm to trigrams
WementionedearlierthatHMMtaggersinactualusehaveanum berofsophistications
not present in the simpliﬁed tagger as we have described it so far. One important
missing feature has to do with the tag context. In the tagger d escribed above, we
assumethatthe probabilityofatag appearingisdependento nlyontheprevioustag:
P(tn
1)≈n
∏
i=1P(ti|ti−1) (5.39)
Most modern HMM taggers actually use a little more of the hist ory, letting the

P(tn
1)≈n
∏
probabilityofa tagdependonthetwo previoustags:
P(tn
1)≈n
∏
i=1P(ti|ti−1,ti−2) (5.40)
In addition to increasing the window before a tagging decisi on, state-of-the-art
HMM taggers like Brants (2000) let the tagger know the locati on of the end of the
sentencebyaddingdependenceonanend-of-sequencemarker fortn+1. Thisgivesthe
followingequationforpart-of-speechtagging:
ˆtn
1=argmax
tn
1P(tn
1|wn
1)≈argmax
tn
1[
n
∏
i=1P(wi|ti)P(ti|ti−1,ti−2)]
P(tn+1|tn) (5.41)
Intagginganysentencewith(5.41),threeofthetagsusedin thecontextwillfalloff

P(tn
1)≈n
∏
ˆtn
1=argmax
tn
1P(tn
1|wn
1)≈argmax
tn
1[
n
∏
the edge of the sentence, and hence will not match regular wor ds. These tags, t−1,t0,
andtn+1,canallbesettobeasinglespecial‘sentenceboundary’tag whichisaddedto
the tagset. This requires that sentences passed to the tagge r have sentence boundaries
demarcated,asdiscussedinCh. 3.
There is one large problem with (5.41); data sparsity. Any pa rticular sequence of
tagsti−2,ti−1,tithatoccursinthetestsetmaysimplyneverhaveoccurredint hetraining
set. That means we cannot compute the tag trigram probabilit y just by the maximum

likelihoodestimatefromcounts,followingEq.5.42:
P(ti|ti−1,ti−2) =C(ti−2,ti−1,ti)
C(ti−2,ti−1): (5.42)

DRAFT150 Chapter 5. Part-of-SpeechTagging
Why not? Because many of these counts will be zero in any train ing set, and we will
incorrectlypredict that a given tag sequence will neverocc ur! What we need is a way
to estimate P(ti|ti−1,ti−2)even if the sequence ti−2,ti−1,tinever occurs in the training
data.
Thestandardapproachto solvethisproblemisto estimate th eprobabilitybycom-
bining more robust, but weaker estimators. For example, if w e’ve never seen the tag
sequence PRP VB TO, so we can’t compute P(TO|PRP,VB) from this frequency, we
still could rely on the bigram probability P(TO|VB), or even the unigram probabil-
ityP(TO). The maximum likelihood estimation of each of these pro babilities can be
computedfroma corpusviathefollowingcounts:
Trigrams ˆP(ti|ti−1,ti−2) =C(ti−2,ti−1,ti)
C(ti−2,ti−1)(5.43)
Bigrams ˆP(ti|ti−1) =C(ti−1,ti)
C(ti−1)(5.44)
Unigrams ˆP(ti) =C(ti)
N(5.45)
How should these three estimators be combined in order to est imate the trigram

N(5.45)
probability P(ti|ti−1,ti−2)? Thesimplestmethodofcombinationislinearinterpolatio n.
In linear interpolation,we estimate the probability P(ti|ti−1ti−2)by a weightedsum of
theunigram,bigram,andtrigramprobabilities:
P(ti|ti−1ti−2) =λ1ˆP(ti|ti−1ti−2)+λ2ˆP(ti|ti−1)+λ3ˆP(ti) (5.46)
We require λ1+λ2+λ3=1, insuringthat the resultingP is a probabilitydistribu-
tion. How shouldthese λs be set? One goodway is deleted interpolation ,developeddeletedinterpolation

by Jelinek and Mercer (1980). In deleted interpolation, we s uccessively delete each
trigram from the training corpus, and choose the λs so as to maximize the likelihood
of the rest of the corpus. The idea of the deletion is to set the λs in such a way as to
generalizetounseendataandnotoverﬁtthetrainingcorpus . Fig.5.19givestheBrants
(2000)versionofthedeletedinterpolationalgorithmfort agtrigrams.
Brants(2000)achievesan accuracyof 96.7%onthe PennTreeb ankwith a trigram
HMM tagger. Weischedel et al. (1993) and DeRose (1988) have a lso reported accu-

racies of above 96% for HMM tagging. (Thede and Harper, 1999) offer a number of
augmentations of the trigram HMM model, including the idea o f conditioning word
likelihoodsonneighboringwordsandtags.
The HMM taggers we have seen so far are trained on hand-tagged data. Kupiec
(1992),Cuttingetal.(1992a),andothersshowthatitisals opossibletotrainanHMM
taggeronunlabeleddata,usingtheEMalgorithmthatwewill introduceinCh.6. These
taggers still start with a dictionary which lists which tags can be assigned to which

words; the EM algorithm then learns the word likelihood func tion for each tag, and
thetag transitionprobabilities. AnexperimentbyMeriald o(1994),however,indicates
that with even a small amount of training data, a tagger train ed on hand-tagged data
worked better than one trained via EM. Thus the EM-trained “p ure HMM” tagger is

DRAFTSection5.6. Transformation-BasedTagging 151
function DELETED-INTERPOLATION (corpus)returnsλ1,λ2,λ3
λ1←0
λ2←0
λ3←0
foreachtrigramt1,t2,t3withf(t1,t2,t3)>0
depending on the maximum of the following three values
caseC(t1,t2,t3)−1
C(t1,t2)−1:increment λ3byC(t1,t2,t3)
caseC(t2,t3)−1
C(t2)−1: increment λ2byC(t1,t2,t3)
caseC(t3)−1
N−1:increment λ1byC(t1,t2,t3)
end
end
normalize λ1,λ2,λ3
returnλ1,λ2,λ3
Figure5.19 The deleted interpolation algorithm for setting the weight s for combining uni-
gram, bigram, and trigram tag probabilities. If the denomin ator is 0 for any case, we deﬁne the
result of thatcase tobe 0. N isthe total number of tokens inth e corpus. After Brants(2000).
probably best suited to cases where no training data is avail able, for example when
tagginglanguagesforwhichthereisnopreviouslyhand-tag geddata.
5.6 Transformation-BasedTagging
Transformation-Based Tagging, sometimes called Brill tag ging, is an instance of the

Transformation-Based Learning (TBL) approach to machine learning (Brill, 1995),Transformation-
BasedLearning
and draws inspiration from both the rule-based and stochast ic taggers. Like the rule-
based taggers, TBL is based on rules that specify what tags sh ould be assigned to
what words. But like the stochastic taggers, TBL is a machine learning technique,
in which rules are automatically induced from the data. Like some but not all of the
HMMtaggers,TBLisasupervisedlearningtechnique;itassu mesapre-taggedtraining
corpus.

corpus.
Samuel et al. (1998) offer a useful analogy for understandin g the TBL para ¯digm,
whichtheycredittoTerryHarvey. Imagineanartistpaintin gapictureofawhitehouse
with green trim against a blue sky. Suppose most of the pictur e was sky, and hence
most of the picture was blue. The artist might begin by using a very broad brush and
painting the entire canvas blue. Next she might switch to a so mewhat smaller white
brush, and paint the entire house white. She would just color in the whole house, not
worrying about the brown roof, or the blue windows or the gree n gables. Next she

takes a smaller brownbrushand colorsoverthe roof. Now she t akesup the blue paint
onasmall brushandpaintsinthebluewindowsonthehouse. Fi nallyshetakesa very
ﬁnegreenbrushanddoesthetrim onthe gables.
Thepainterstartswith a broadbrushthat coversa lot ofthe c anvasbutcolorsa lot

DRAFT152 Chapter 5. Part-of-SpeechTagging
of areas that will have to be repainted. The next layer colors less of the canvas, but
also makes fewer “mistakes”. Each new layer uses a ﬁner brush that corrects less of
the picture, but makes fewer mistakes. TBL uses somewhat the same method as this
painter. The TBL algorithm has a set of tagging rules. A corpu s is ﬁrst tagged using
the broadest rule, that is, the one that applies to the most ca ses. Then a slightly more
speciﬁcruleischosen,whichchangessomeoftheoriginalta gs. Nextanevennarrower
rule, which changes a smaller number of tags (some of which mi ght be previously
changedtags).
5.6.1 HowTBL Rules Are Applied
Let’s look at one of the rules used by Brill’s (1995)tagger. B efore the rules apply,the
taggerlabelseverywordwith itsmost-likelytag. We getthe semost-likelytagsfroma
taggedcorpus. Forexample,in theBrowncorpus, raceis mostlikelytobea noun:
P(NN|race) =.98
P(VB|race) =.02
Thismeansthat thetwo examplesof racethat wesaw abovewill bothbecodedas

NN. Intheﬁrst case,thisisa mistake,as NN is theincorrectt ag:
(5.47)is/VBZ expected/VBNto/TOrace/ NNtomorrow/NN
Inthe secondcase this raceiscorrectlytaggedasan NN:
(5.48)the/DTrace/ NNfor/INouter/JJspace/NN
After selecting the most-likely tag, Brill’s tagger applie s its transformation rules.
As it happens, Brill’s tagger learned a rule that applies exa ctly to this mistagging of
race:
ChangeNNtoVBwhenthe previoustagis TO
Thisrulewouldchange race/NNtorace/VBinexactlythefollowingsituation,since
it isprecededby to/TO:

race:
(5.49)expected/VBNto/TOrace/NN →expected/VBNto/TOrace/VB
5.6.2 HowTBL Rules Are Learned
Brill’s TBL algorithm has three major stages. It ﬁrst labels every word with its most-
likely tag. It then examines every possible transformation , and selects the one that
results in the most improvedtagging. Finally, it then re-ta gsthe data accordingto this
rule. The last two stages are repeateduntil some stoppingcr iterionis reached,such as
insufﬁcientimprovementoverthepreviouspass. Notethats tagetworequiresthatTBL

knowsthe correcttagofeachword;that is, TBLisa supervise dlearningalgorithm.
The output of the TBL process is an ordered list of transforma tions; these then
constitute a “tagging procedure”that can be applied to a new corpus. In principle the
setofpossibletransformationsisinﬁnite,sincewecouldi maginetransformationssuch
as “transform NN to VB if the previous word was “IBM” and the wo rd “the” occurs
between 17 and 158 words before that”. But TBL needs to consid er every possible

DRAFTSection5.7. EvaluationandErrorAnalysis 153
transformation,in orderto pick thebest oneoneachpass thr oughthe algorithm. Thus
thealgorithmneedsawaytolimitthesetoftransformations . Thisisdonebydesigning
a small set of templates (abstracted transformations). Everyallowabletransform ation templates
isaninstantiationofoneofthetemplates. Brill’sset ofte mplatesislisted inFig.5.20.
Fig.5.21givesthedetailsofthisalgorithmforlearningtr ansformations.
Thepreceding(following)wordis tagged z.
Thewordtwo before(after)istagged z.
Oneofthetwo preceding(following)wordsis tagged z.
Oneofthethreepreceding(following)wordsistagged z.
Theprecedingwordistagged zandthefollowingwordistagged w.
Thepreceding(following)wordis tagged zandtheword
two before(after)is tagged w.
Figure5.20 Brill’s (1995) templates. Each begins with “Change tag ato tagbwhen: ...” .
The variables a,b,z, andwrange over parts-of-speech.
At the heart of Fig. 5.21 are the two functions G ETBESTTRANSFORMATION

and GETBESTINSTANCE . GETBESTTRANSFORMATION is called with a list of
potential templates; for each template, it calls G ETBESTINSTANCE . GETBEST-
INSTANCE iteratively tests every possible instantiation of each tem plate by ﬁlling in
speciﬁcvaluesforthetagvariables a,b,z, andw.
In practice, there are a number of ways to make the algorithm m ore efﬁcient. For
example, templatesand instantiated transformationscan b e suggested in a data-driven
manner; a transformation-instance might only be suggested if it would improve the

tagging of some speciﬁc word. The search can also be made more efﬁcient by pre-
indexingthe wordsin thetrainingcorpusbypotentialtrans formation. RocheandSch-
abes(1997a)showhowthe taggercanalso bespeededupbyconv ertingeachruleinto
a ﬁnite-statetransducerandcomposingall thetransducers .
Fig.5.6.2showsafew ofthe ruleslearnedbyBrill’sorigina ltagger.
5.7 EvaluationandErrorAnalysis
TheprobabilitiesinastatisticalmodellikeanHMMPOS-tag gercomefromthecorpus
it is trained on. We saw in Sec. 4.3 that in order to train stati stical modelslike taggers

orN-grams, we need to set aside a training set . The design of the training set or
trainingcorpus needsto be carefullyconsidered. Ifthe trainingcorpusist oo speciﬁc
to the task or domain, the probabilities may be too narrow and not generalize well to
tagging sentences in very different domains. But if the trai ning corpus is too general,
theprobabilitiesmaynotdoa sufﬁcientjobofreﬂectingthe task ordomain.
For evaluating N-grams models, we said in Sec. 4.3 that we need to divide our

corpus into a distinct training set, test set, and a second te st set called a development
test set. We train our tagger on the training set. Then we use t hedevelopment test
set(also called a devtestset) to perhaps tune some parameters, and in general decidedevelopmenttestset
devtest what the best model is. Then once we come up with what we think i s the best model,

DRAFT154 Chapter 5. Part-of-SpeechTagging
function TBL(corpus)returnstransforms-queue
INTIALIZE -WITH-MOST-LIKELY-TAGS(corpus)
untilendcondition is met do
templates←GENERATE -POTENTIAL -RELEVANT -TEMPLATES
best-transform←GET-BEST-TRANSFORM (corpus,templates )
APPLY-TRANSFORM (best-transform ,corpus)
ENQUEUE(best-transform-rule ,transforms-queue )
end
return(transforms-queue )
function GET-BEST-TRANSFORM (corpus,templates )returnstransform
for eachtemplateintemplates
(instance,score)←GET-BEST-INSTANCE (corpus,template)
if(score>best-transform.score )thenbest-transform←(instance,score)
return(best-transform )
function GET-BEST-INSTANCE (corpus,template) returnstransform
forfrom-tag←fromtag1totagndo
forto-tag←fromtag1totagndo
forpos←from1tocorpus-size do
if(correct-tag (pos) ==to-tag&&current-tag (pos) ==from-tag)
num-good-transforms (current-tag (pos−1))++
elseif(correct-tag (pos)==from-tag&&current-tag (pos)==from-tag)
num-bad-transforms (current-tag (pos−1))++
end

end
best-Z←ARGMAX t(num-good-transforms (t) -num-bad-transforms (t))
if(num-good-transforms (best-Z) -num-bad-transforms (best-Z)
>best-instance.score )then
best.rule←“Change tagfrom from-tagtoto-tagif prevtagis best-Z”
best.score←num-good-transforms (best-Z) -num-bad-transforms (best-Z)
return(best)
procedure APPLY-TRANSFORM (transform ,corpus)
if(current-tag (pos)==best-rule-from )
&& (current-tag (pos−1)==best-rule-prev ))
current-tag (pos)←best-rule-to

end
Figure5.21 TheBrill(1995)TBLalgorithmforlearningtotag. G ETBESTINSTANCE would
have to change for transformation templates other than “Change tag from X to Y if previous tag
is Z”.
we run it on the (hitherto unseen) test set to see its performa nce. We might use 80%
of our data for training, and save 10% each for devtest and tes t. Why do we need a
development test set distinct from the ﬁnal test set? Becaus e if we used the ﬁnal test
set tocomputeperformanceforall ourexperimentsduringou rdevelopmentphase,we

is Z”.
wouldbe tuningthe variouschangesand parametersto thisse t. Ourﬁnal errorrate on

DRAFTSection5.7. EvaluationandErrorAnalysis 155
Change tags
#From To Condition Example
1 NN VB Previous tagis TO to/TOrace/NN →VB
2 VBP VB One of the previous 3tags isMD might/MD vanish/VBP →VB
3 NN VB One of the previous 2tags isMD might/MD not reply/NN →VB
4 VB NN One of the previous 2tags is DT
5 VBD VBN One of the previous 3tags is VBZ
Figure5.22 The ﬁrst20nonlexicalized transformations from Brill(199 5).
thetest set wouldthenbe optimistic: itwouldunderestimat ethe trueerrorrate.
Theproblemwith havinga ﬁxed trainingset, devset, and test set is that in orderto
savelotsofdatafortraining,thetestsetmightnotbelarge enoughtoberepresentative.
Thus a better approach would be to somehow use allour data both for training and
test. How is this possible? The idea is to use crossvalidation . In crossvalidation, we crossvalidation
randomlychoose a training and test set division of our data, train our tagger, and then
computetheerrorrateonthetestset. Thenwerepeatwithadi fferentrandomlyselected

trainingset and test set. We do thissamplingprocess10time s, and thenaveragethese
10runsto getanaverageerrorrate. Thisiscalled 10-foldcrossvalidation .10-fold
crossvalidation
Theonlyproblemwith cross-validationis that becauseall t he datais usedfortest-
ing,weneedthewholecorpustobeblind;wecan’texamineany ofthedatatosuggest
possible features, and in general see what’s going on. But lo okingat the corpusis of-
ten important for designing the system. For this reason it is common to create a ﬁxed

training set and test set, and then to do 10-fold crossvalida tion inside the training set,
butcomputeerrorratethenormalway inthetest set.
Oncewehaveatestset,taggersareevaluatedbycomparingth eirlabelingofthetest
set with a human-labeled Gold Standard test set, based on accuracy : the percentage
of all tags in the test set where the tagger and the Gold standa rd agree. Most current
taggingalgorithmshaveanaccuracyofaround96–97%forsim pletagsetslikethePenn

Treebankset. These accuraciesare for wordsand punctuatio n;the accuracyforwords
onlywouldbelower.
How good is 97%? Since tagsets and tasks differ, the performa nce of tags can be
comparedagainstalower-bound baselineandanupper-bound ceiling. Oneway toset baseline
ceiling a ceiling is to see how well humansdo on the task. Marcuset al. (1993),forexample,
foundthathumanannotatorsagreedonabout96–97%ofthetag sinthePennTreebank
version of the Brown corpus. This suggests that the Gold Stan dard may have a 3-4%

margin of error, and that it is meaningless to get 100% accura cy, (modeling the last
3%wouldjustbemodelingnoise). IndeedRatnaparkhi(1996) showedthatthetagging
ambiguitiesthatcausedproblemsforhistaggerwereexactl ytheonesthathumanshad
labeled inconsistently in the training set. Two experiment s by Voutilainen (1995, p.
174), however, found that when humans were allowed to discus s tags, they reached
consensuson100%ofthetags.
HumanCeiling: WhenusingahumanGoldStandardtoevaluateaclassiﬁca-

tionalgorithm,checktheagreementrateofhumansonthesta ndard.
The standard baseline, suggested by Gale et al. (1992b) (in the slightly different
context of word-sense disambiguation), is to choose the unigram most-likely tag for

DRAFT156 Chapter 5. Part-of-SpeechTagging
each ambiguous word. The most-likely tag for each word can be computed from a
hand-taggedcorpus(whichmaybethesameasthetrainingcor pusforthetaggerbeing
evaluated).
Most Frequent Class Baseline: Always compare a classiﬁer against a base-
line at least as good as the most frequentclass baseline (ass igning each token
totheclassit occurredin mostoftenin thetrainingset).
Tagging algorithms since Harris (1962) incorporate this ta g frequency intuition.
Charniaketal.(1993)showedthatthisbaselinealgorithma chievesanaccuracyof90–
91% onthe 87-tagBrown tagset; Toutanovaet al. (2003)showe dthat a morecomplex
version, augmented with an unknown word model, achieved 93. 69% on the 45-tag
Treebanktagset.
When comparing models it is important to use statistical tes ts (introduced in any
statistics class or textbook for the social sciences) to det ermine if the difference be-
tween two models is signiﬁcant. Cohen (1995) is a useful refe rence which focuses

on statistical research methods for artiﬁcial intelligenc e. Dietterich (1998) focuses on
statistical tests forcomparingclassiﬁers. Whenstatisti cally comparingsequencemod-
els like part-of-speech taggers, it is important to use paired tests . Commonly used pairedtests
paired tests for evaluating part-of-speech taggers includ e theWilcoxon signed-rank
test,paired t-tests , versions of matched t-tests such as the Matched-Pair Sente nceWilcoxonsigned-rank test

pairedt-tests SegmentWord Error( MAPSSWE ) test originallyappliedto speech recognitionword
MAPSSWE errorrate,andthe McNemartest .
McNemartest
5.7.1 Error Analysis
Inordertoimproveanymodelweneedtounderstandwhereitwe ntwrong. Analyzing
the error in a classiﬁer like a part-of-speech tagger is done via aconfusion matrix ,
orcontingency table . A confusion matrix for an N-way classiﬁcation task is an N-
by-Nmatrix where the cell (x,y)contains the number of times an item with correct
classiﬁcation xwasclassiﬁedbythemodelas y. Forexample,thefollowingtableshows

aportionoftheconfusionmatrixfromtheHMMtaggingexperi mentsofFranz(1996).
The row labels indicate correct tags, column labels indicat e the tagger’s hypothesized
tags, and each cell indicates percentage of the overall tagg ing error. Thus 4.4% of
the total errors were caused by mistagging a VBD as a VBN. Comm on errors are
highlighted.
INJJNNNNPRBVBDVBN
IN-.2 .7
JJ.2-3.32.11.7.22.7
NN 8.7- .2
NNP.23.34.1-.2
RB2.22.0.5 -
VBD .3.5 -4.4
VBN 2.8 2.6-
The confusion matrix above, and related error analyses in Fr anz (1996), Kupiec

IN-.2 .7
NN 8.7- .2
(1992),andRatnaparkhi(1996),suggestthat somemajorpro blemsfacingcurrenttag-

DRAFTSection5.8. AdvancedIssuesin Part-of-SpeechTagging 157
gersare:
1.NN versus NNP versus JJ: These are hard to distinguish prenominally. Dis-
tinguishing proper nouns is especially important for infor mation extraction and
machinetranslation.
2.RP versus RB versus IN: All of these can appear in sequences of satellites
immediatelyfollowingtheverb.
3.VBD versusVBN versusJJ: Distinguishingthese isimportantforpartialpars-
ing(participlesareusedtoﬁndpassives),andforcorrectl ylabelingtheedgesof
noun-phrases.
Erroranalysislikethisisacrucialpartofanycomputation allinguisticapplication. Er-
roranalysiscanhelpﬁndbugs,ﬁndproblemsinthetrainingd ata,and,mostimportant,
helpindevelopingnewkindsofknowledgeoralgorithmstous e insolvingproblems.
5.8 AdvancedIssuesin Part-of-SpeechTagging
5.8.1 Practical Issues: Tag Indeterminacy and Tokenizatio n
Tag indeterminacy arises when a word is ambiguous between mu ltiple tags and it is
impossible or very difﬁcult to disambiguate. In this case, s ome taggers allow the use

ofmultipletags. ThisisthecaseinboththePennTreebankan dintheBritishNational
Corpus. Common tag indeterminacies include adjective vers us preterite versus past
participle (JJ/VBD/VBN), and adjective versus noun as pren ominalmodiﬁer (JJ/NN).
Givenacorpuswiththeseindeterminatetags,thereare3way stodealwithtagindeter-
minacywhentrainingandscoringpart-of-speechtaggers:
1. Somehowreplacetheindeterminatetagswithonlyonetag.
2. In testing, count a tagger as having correctly taggedan in determinatetoken if it

giveseitherofthecorrecttags. Intraining,somehowchoos eonlyoneofthetags
fortheword.
3. Treattheindeterminatetagasa singlecomplextag.
The second approach is perhaps the most sensible, although m ost previous published
results seem to have used the third approach. This third appr oach applied to the Penn
TreebankBrowncorpus,forexample,resultsinamuchlarger tagsetof85tagsinstead
of 45, but the additional 40 complextags covera total of only 121 word instances out
ofthemillionwordcorpus.

Most tagging algorithmsassume a process of tokenization ha s been applied to the
tags. Ch. 3 discussed the issue of tokenization of periods fo r distinguishing sentence-
ﬁnal periods from word-internal period in words like etc.. An additional role for tok-
enizationisinwordsplitting. ThePennTreebankandtheBri tishNationalCorpussplit
contractionsandthe ’s-genitivefromtheirstems:
would/MDn’t/RB
children/NNS’s/POS

DRAFT158 Chapter 5. Part-of-SpeechTagging
Indeed,thespecialTreebanktagPOSisusedonlyforthemorp heme’swhichmust
besegmentedoffduringtokenization.
Another tokenization issue concerns multi-part words. The Treebank tagset as-
sumes that tokenization of words like New York is done at whitespace. The phrase
a New York City ﬁrm is tagged in Treebank notation as ﬁve separate words: a/DT
New/NNPYork/NNPCity/NNPﬁrm/NN .TheC5tagset,bycontrast,allowprepositions
like “in terms of ” to be treated as a single word by adding numbers to each tag, a s in
in/II31terms/II32of/II33 .
5.8.2 Unknown Words
words people
never use —
could be
only I
know them
Ishikawa Takuboku 1885–1912
Allthetaggingalgorithmswehavediscussedrequireadicti onarythatliststhepossible
parts-of-speech of every word. But the largest dictionary w ill still not contain every
possibleword,aswesawinCh.7. Propernamesandacronymsar ecreatedveryoften,
andevennew commonnounsandverbsenter thelanguageat a sur prisingrate. There-

never use —
could be
only I
know them
fore in order to build a complete tagger we cannot always use a dictionary to give us
p(wi|ti). We needsomemethodforguessingthetagofan unknownword.
The simplest possible unknown-word algorithm is to pretend that each unknown
word is ambiguous among all possible tags, with equal probab ility. Then the tagger
must rely solely on the contextual POS-trigrams to suggest t he proper tag. A slightly
more complex algorithm is based on the idea that the probabil ity distribution of tags
over unknown words is very similar to the distribution of tag s over words that oc-

curred only once in a training set, an idea that was suggested by both Baayen and
Sproat(1996)andDermatasandKokkinakis(1995). Thesewor dsthatonlyoccuronce
are known as hapax legomena (singular hapax legomenon ). For example, unknown hapax legomena
words and hapax legomena are similar in that they are both most likely to be nouns,
followed by verbs, but are very unlikely to be determiners or interjections. Thus the
likelihood P(wi|ti)for an unknownword is determined by the average of the distri bu-

tion overall singleton words in the training set. This idea o f using “thingswe’ve seen
once” as an estimator for “things we’ve never seen” will prov e useful in the Good-
TuringalgorithmofCh.4.
Most unknown-word algorithms, however, make use of a much mo re powerful
source of information: the morphology of the words. For exam ple, words that end
in-sare likely to be plural nouns (NNS), words ending with -edtend to be past par-
ticiples (VBN), words ending with abletend to be adjectives (JJ), and so on. Even if

we’ve never seen a word, we can use facts about its morphologi cal form to guess its
part-of-speech. Besides morphological knowledge, orthog raphic information can be
very helpful. For example words starting with capital lette rs are likely to be proper
nouns(NP).Thepresenceofahyphenisalsoausefulfeature; hyphenatedwordsinthe
TreebankversionofBrownaremostlikelytobeadjectives(J J).ThisprevalenceofJJs

DRAFTSection5.8. AdvancedIssuesin Part-of-SpeechTagging 159
iscausedbythelabelinginstructionsfortheTreebank,whi chspeciﬁedthatprenominal
modiﬁersshouldbelabeledasJJif theycontaineda hyphen.
Howarethese featurescombinedandusedin part-of-speecht aggers? Onemethod
is to train separate probability estimators for each featur e, assume independence, and
multiply the probabilities. Weischedel et al. (1993) built such a model, based on four
speciﬁc kinds of morphological and orthographic features. They used 3 inﬂectional
endings(-ed,-s,-ing),32derivationalendings(suchas -ion,-al,-ive,and-ly),4values
ofcapitalizationdependingonwhetherawordissentence-i nitial(+/-capitalization,+/-
initial)andwhetherthewordwashyphenated. Foreachfeatu re,theytrainedmaximum
likelihoodestimatesoftheprobabilityofthefeaturegive natagfromalabeledtraining
set. They then combined the features to estimate the probabi lity of an unknownword
byassumingindependenceandmultiplying:
P(wi|ti) =p(unknown-word|ti)∗p(capital|ti)∗p(endings/hyph|ti) (5.50)

AnotherHMM-basedapproach,duetoSamuelsson(1993)andBr ants(2000),gen-
eralizes this use of morphology in a data-driven way. In this approach, rather than
pre-selecting certain sufﬁxes by hand, all ﬁnal letter sequ ences of all words are con-
sidered. They consider such sufﬁxes of up to ten letters, com puting for each sufﬁx of
lengthitheprobabilityofthe tag tigiventhesufﬁx:
P(ti|ln−i+1...ln) (5.51)
These probabilities are smoothed using successively short er and shorter sufﬁxes.

Separatesufﬁxtriesarekeptforcapitalizedanduncapital izedwords.
Ingeneral,mostunknownwordmodelstrytocapturethefact t hatunknownwords
areunlikelytobeclosed-classwordslikeprepositions. Br antsmodelsthisfactbyonly
computing sufﬁx probabilities from the training set for wor ds whose frequencyin the
trainingset is≤10. Inthe HMMtaggingmodelofThedeandHarper(1999),thisf act
ismodeledinsteadbyonlytrainingonopen-classwords.
Notethat(5.51)givesanestimateof p(ti|wi);sincefortheHMMtaggingapproach

we need the likelihood p(wi|ti), this can be derived from (5.51) using Bayesian inver-
sion(i.e. usingBayesruleandcomputationofthetwopriors P(ti)andP(ti|ln−i+1...ln)).
In addition to using capitalization information for unknow n words, Brants (2000)
also uses capitalization information for tagging known wor ds, by adding a capitaliza-
tionfeaturetoeachtag. Thusinsteadofcomputing P(ti|ti−1,ti−2)asin(5.44),heactu-
ally computesthe probability P(ti,ci|ti−1,ci−1,ti−2,ci−2). This is equivalent to having

a capitalizedanduncapitalizedversionofeachtag,essent iallydoublingthe size ofthe
tagset.
A non-HMM based approach to unknown word detection was that o f Brill (1995)
usingtheTBLalgorithm,wheretheallowabletemplateswere deﬁnedorthographically
(theﬁrstNlettersofthe words,thelast Nlettersoftheword,etc.).
Most recent approaches to unknown word handling, however, c ombine these fea-
turesin a third way: by using maximumentropy( MaxEnt)modelssuch as the Maxi-

tagset.
mum EntropyMarkov Model (MEMM) ﬁrst introducedby Ratnaparkhi(1996)and
McCallum et al. (2000),andwhich we will studyin Ch. 6. The ma ximumentropyap-
proachisoneafamilyoflog-linearapproachestoclassiﬁca tioninwhichmanyfeatures

DRAFT160 Chapter 5. Part-of-SpeechTagging
are computed for the word to be tagged, and all the features ar e combined in a model
based on multinomial logistic regression. The unknown word model in the tagger of
Toutanovaet al. (2003)usesa featureset extendedfromRatn aparkhi(1996),in which
eachfeaturerepresentsa propertyofa word,includingfeat ureslike:
word contains a number
word contains anupper-case letter
word contains a hyphen
word isallupper-case
word contains a particular preﬁx(from the set of allpreﬁxes of length≤4)
word contains a particular sufﬁx(from the set of allpreﬁxes of length≤4)
word isupper-case andhas a digitand a dash(like CFC-12)
word isupper-case andfollowed within3words byCo., Inc.,e tc
Toutanova et al. (2003) found this last feature, implementi ng a simple company
name detector, to be particularlyuseful. 3 words by a word li ke Co. or Inc. Note that
theRatnaparkhi(1996)modelignoredallfeatureswithcoun tslessthan10.
LoglinearmodelshavealsobeenappliedtoChinesetaggingb yTsengetal.(2005b).

Chinesewordsareveryshort(around2.4charactersperunkn ownwordcomparedwith
7.7 for English), but Tseng et al. (2005b) found that morphol ogical features nonethe-
less gave a huge increase in tagging performance for unknown words. For example
for each character in an unknown word and each POS tag, they ad ded a binary fea-
ture indicating whether that character ever occurred with t hat tag in any training set
word. Thereisalso aninterestingdistributionaldifferen cein unknownwordsbetween

Chinese andEnglish. While Englishunknownwordstendto be p ropernouns(41%of
unknownwords in WSJ are NP), in Chinese the majority of unkno wnwordsare com-
mon nouns and verbs (61% in the Chinese TreeBank 5.0). These r atios are similar to
German,andseemtobecausedbytheprevalenceofcompoundin gasamorphological
devicein ChineseandGerman.
5.8.3 Part-of-Speech Tagging forOther Languages
As the previous paragraph suggests, part-of-speech taggin g algorithms have all been

appliedtomanyotherlanguagesaswell. Insomecases,theme thodsworkwellwithout
largemodiﬁcations; Brants(2000)showedthe exact same per formancefortaggingon
the German NEGRA corpus(96.7%)as on the English Penn Treeba nk. But a number
ofaugmentationsandchangesbecomenecessarywhendealing withhighlyinﬂectedor
agglutinativelanguages.
One problem with these languages is simply the large number o f words, when
comparedtoEnglish. RecallfromCh.3thatagglutinativela nguageslikeTurkish(and
to someextentmixedagglutinative-inﬂectionallanguages likeHungarian)arethose in

whichwordscontainlongstringsof morphemes,whereeach mo rphemehasrelatively
fewsurfaceforms,andsoitisoftenpossibletoclearlyseet hemorphemesinthesurface
text. ForexampleMegyesi(1999)givesthe followingtypica l exampleof a Hungarian
wordmeaning“oftheirhits”:
(5.52)tal´ alataiknak

DRAFTSection5.8. AdvancedIssuesin Part-of-SpeechTagging 161
tal´ al
hit/ﬁnd-at
nominalizer-a
his-i
poss.plur-k
their-nak
dat/gen
“of theirhits”
Similarly,thefollowinglist,excerptedfromHakkani-T¨ u retal.(2002),showsafew
ofthewordsproducibleinTurkishfromtheroot uyu-,’sleep’:
uyuyorum ‘Iam sleeping’ uyuyorsun ‘you are sleeping’
uyuduk ‘we slept’ uyumadan ‘without sleeping’
uyuman ‘your sleeping’ uyurken ‘while (somebody) is sleepi ng’
uyutmak ‘tocause someone tosleep’ uyutturmak ‘to cause som eone to cause another
person tosleep’
These productive word-formation processes result in a larg e vocabulary for these
languages. OraveczandDienes(2002),forexample,show tha t a quarter-millionword
corpus of English has about 19,000 different words (i.e. wor d types); the same size
corpusof Hungarianhas almost 50,000different words. This problem continueseven
with much larger corpora; note in the table below on Turkish f rom Hakkani-T¨ uret al.

(2002) that the vocabulary size of Turkish is far bigger than that of English and is
growingfaster thanEnglishevenat 10millionwords.
CorpusSize VocabularySize
Turkish English
1Mwords 106,547 33,398
10Mwords 417,775 97,734
The large vocabulary size seems to cause a signiﬁcant degrad ationin tagging per-
formancewhentheHMMalgorithmisapplieddirectlytoagglu tinativelanguages. For
example Oravecz and Dienes (2002) applied the exact same HMM software (called

‘TnT’) that Brants (2000) used to achieve 96.7% on both Engli sh and German, and
achievedonly92.88%onHungarian. Theperformanceonknown words(98.32%)was
comparable to English results; the problem was the performa nce on unknown words:
67.07%onHungarian,comparedtoaround84-85%forunknownw ordswithacompa-
rable amount of English training data. Hajiˇ c (2000)notes t he same problemin a wide
variety of other languages (including Czech, Slovene, Esto nian, and Romanian); the
performanceof these taggers is hugely improvedby adding a d ictionary which essen-

tially gives a better model of unknown words. In summary, one difﬁculty in tagging
highlyinﬂectedandagglutinativelanguagesistaggingofu nknownwords.
A second,related issue with such languagesis the vast amoun tof informationthat
iscodedinthemorphologyoftheword. InEnglish,lotsofinf ormationaboutsyntactic
function of a word is represented by word order, or neighbori ng function words. In
highly inﬂectional languages, information such as the case (nominative, accusative,
genitive)orgender(masculine,feminine)ismarkedonthew ordsthemselves,andword

order plays less of a role in marking syntactic function. Sin ce tagging is often used a
preprocessingstepforotherNLPalgorithmssuchasparsing orinformationextraction,
this morphologicalinformation is crucial to extract. This means that a part-of-speech
tagging output for Turkish or Czech needs to include informa tion about the case and
genderof eachwordin orderto beas usefulas parts-of-speec hwithoutcase orgender
areinEnglish.

DRAFT162 Chapter 5. Part-of-SpeechTagging
Forthisreason,tagsetsforagglutinativeandhighlyinﬂec tionallanguagesareusu-
ally muchlargerthanthe50-100tagswehaveseen forEnglish . Tagsin suchenriched
tagsetsare sequencesof morphologicaltagsratherthana si ngle primitivetag. Assign-
ingtagsfromsucha tagset towordsmeansthatwe are jointlys olvingthe problemsof
part-of-speech tagging and morphological disambiguation . Hakkani-T¨ ur et al. (2002)
give the following example of tags from Turkish, in which the wordizinhas three
possiblemorphological/part-of-speechtags(andmeaning s):
1. Yerdeki izintemizlenmesi gerek. iz+ Noun+A3sg+Pnon+Gen
Thetrace on the ﬂoorshould be cleaned.
2.¨Uzerinde parmak izinkalmis ¸ iz+ Noun+A3sg+P2sg+Nom
Yourﬁngerprintislefton (it).
3. Ic ¸erigirmekic ¸in izinalmangerekiyor. izin+ Noun+A3sg+Pnon+Nom
Youneeda permission toenter.
Usinga morphologicalparse sequencelike Noun+A3sg+Pnon+Gen as the part-
of-speech tag greatly increases the number of parts-of-spe ech, of course. We can see

thisclearlyinthemorphologicallytaggedMULTEXT-Eastco rpora,inEnglish,Czech,
Estonian, Hungarian,Romanian, and Slovene (Dimitrova et a l., 1998; Erjavec, 2004).
Hajiˇ c (2000)givesthe followingtagset sizesforthese cor pora:
Language Tagset Size
English 139
Czech 970
Estonian 476
Hungarian 401
Romanian 486
Slovene 1033
Withsuchlargetagsets,itisgenerallynecessarytoperfor mmorphologicalanalysis
on each word to generate the list of possible morphologicalt ag sequences(i.e. the list

Czech 970
of possible part-of-speechtags) for the word. The role of th e tagger is then to disam-
biguate among these tags. The morphological analysis can be done in various ways.
TheHakkani-T¨ uretal.(2002)modelofTurkishmorphologic alanalysisisbasedonthe
two-levelmorphologyweintroducedinCh.3. ForCzechandth eMULTEXT-Eastlan-
guages, Hajiˇ c (2000) and Hajiˇ c and Hladk´ a (1998) use a ﬁxe d external dictionary for
eachlanguagewhichcompilesoutall thepossibleformsofea chword,andlists possi-

ble tags for each wordform. The morphologicalparse also cru cially helps address the
problem of unknown words, since morphological parsers can a ccept unknown stems
andstill segmenttheafﬁxesproperly.
Given such a morphological parse, various methods for the ta gging itself can be
used. The Hakkani-T¨ ur et al. (2002) model for Turkish uses a Markov model of tag
sequences. Themodelassignsaprobabilitytosequencesoft agslike
izin+Noun+A3sg+Pnon+Nom
bycomputingtagtransitionprobabilitiesfroma trainings et. Othermodelsuse similar

techniques to those for English. Hajiˇ c (2000) and Hajiˇ c an d Hladk´ a (1998), for ex-
ample,usealog-linearexponentialtaggerfortheMULTEXT- Eastlanguages,Oravecz
andDienes(2002)andDˇ zeroskietal.(2000)usetheTnTHMMt agger(Brants,2000),
andsoon.

DRAFTSection5.9. Advanced: TheNoisyChannelModelforSpelling 163
5.8.4 Combining Taggers
The various part-of-speech tagging algorithms we have desc ribed can also be com-
bined. The most common approach to tagger combination is to r un multiple taggers
in parallel on the same sentence, and then combine their outp ut, either by voting or
by training another classiﬁer to choose which tagger to trus t in a given context. Brill
and Wu (1998),for example,combinedunigram,HMM, TBL, and m aximum-entropy
taggers by voting via a higher-order classiﬁer, and showed a small gain over the best
ofthe fourclassiﬁers. Ingeneral,thiskindofcombination isonlyusefulif thetaggers
have complementaryerrors, and so researchon combinationo ften beginsby checking
to see if the errors are indeed different from different tagg ers. Another option is to
combine taggers in series. Hajiˇ c et al. (2001) apply this op tion for Czech, using the
rule-basedapproachto removesome ofthe impossibletag pos sibilities foreach word,

andthenanHMM taggertochoosethebestsequencefromtherem ainingtags.
5.9 Advanced: TheNoisyChannelModelforSpelling
The Bayesian inference model introduced in Sec. 5.5 for tagg ing has another inter-
pretation: as an implementation of the noisy channel model, a crucial tool in speech
recognitionandmachinetranslation.
In this section we introducethis noisy channel model and sho w how to apply it to
the task of correcting spelling errors. The noisy channel mo del is used in Microsoft

Wordandinmanysearchengines,andingeneralisthemostwid elyusedalgorithmfor
correctinganykindofsingle-wordspellingerror,includi ngnon-wordspelling errors
andforreal-wordspelling errors .
Recall that non-word spelling errors are those which are not English words (like
recieveforreceive), and we can detectthese by simply looking for any word not in
a dictionary. We saw in Sec. 3.10 that candidate corrections for some spelling errors
could be found by looking for words that had a small edit distance to the misspelled
word.

word.
The Bayesian models we have seen in this chapter, and the nois y channel model,
will give us a better way to ﬁnd these corrections. Furthermo re, we’ll be able to use
thenoisychannelmodelfor contextualspellchecking ,whichisthetaskofcorrecting
real-wordspelling errors like thefollowing:
Thestudywasconductedmainly beJohnBlack.
Sincetheseerrorshaverealwords,wecan’tﬁndthembyjustﬂ aggingwordsnotin
the dictionary, and we can’t correct them just using edit dis tance alone. But note that

words around the candidate correction in about ﬁfteen minutes make it a much more
probable word sequence than the original in about ﬁfteen minuets . The noisy channel
modelwill implementthisideavia N-grammodels.
The intuition of the noisy channel model (see Fig. 5.23) is to treat the misspelled noisy channel
wordasifacorrectly-spelledwordhadbeen‘distorted’byb eingpassedthroughanoisy

DRAFT164 Chapter 5. Part-of-SpeechTagging
noisy wordoriginal wordnoisy channel
decoder
word hyp1
word hyp2
...
word hypN guessed wordnoisy 1
noisy 2
noisy N
Figure5.23 In the noisy channel model, we imagine that the surface form w e see is actually
a ‘distorted’ form of an original word passed through a noisy channel. The decoder passes each
hypothesis through a model of this channel and picks the word that best matches the surface
noisy word.
communication channel. This channel introduces “noise” in the form of substitutions
or other changes to the letters which makes it hard to recogni ze the “true” word. Our
goal is then to build a model of the channel. Given this model, we then ﬁnd the true
word by taking every word of the language, passing each word t hrough our model of
thenoisychannel,andseeingwhichonecomestheclosest to t hemisspelledword.
This noisy channel model, like the HMM tagging architecture we saw earlier, is a

special case of Bayesian inference . We see an observation O(amisspelled word)and Bayesian
ourjobistoﬁndtheword wwhichgeneratedthismisspelledword. Outofallpossible
wordsinthevocabulary Vwewantto ﬁndtheword wsuchthat P(w|O)ishighest,or: V
ˆw=argmax
w∈VP(w|O) (5.53)
As we saw for part-of-speechtagging, we will use Bayes rule t o turn the problem
around(andnotethat,asfortagging,we canignorethedenom inator):
ˆw=argmax
w∈VP(O|w)P(w)
P(O)=argmax
w∈VP(O|w)P(w) (5.54)

ˆw=argmax
ˆw=argmax
To summarize, the noisy channel model says that we have some t rue underlying
wordw,andwehaveanoisychannelwhichmodiﬁesthewordintosomep ossiblemis-
spelled surface form. The probability of the noisy channel p roducing any particular
observationsequence Ois modeledby P(O|w). The probabilitydistribution overpos-
sible hidden words is modeled by P(w). The most probable word ˆ wgiven that we’ve
seensome observedmisspelling Ocanbecomputedbytakingtheproductoftheword
priorP(w)and the observation likelihood P(O|w)and choosing the word for which

thisproductisgreatest.
Let’sapplythenoisychannelapproachtocorrectingnon-wo rdspellingerrors. This
approach was ﬁrst suggested by Kernighan et al. (1990); thei r program, correct ,
takes wordsrejected by the Unix spellprogram,generatesa list of potential correct

DRAFTSection5.9. Advanced: TheNoisyChannelModelforSpelling 165
words, ranks them according to Eq. 5.54, and picks the highes t-ranked one. We’ll
apply the algorithm to the example misspelling acress. The algorithm has two stages:
proposingcandidatecorrections andscoringthecandidates .
In order to propose candidate corrections Kernighan et al. m ake the reasonable
(Damerau,1964)simplifyingassumptionthatthecorrectwo rdwilldifferfromthemis-
spelling by a single insertion, deletion, substitution, or transposition. The list of can-
didate words is generated from the typo by applying any singl e transformation which
resultsinawordinalargeon-linedictionary. Applyingall possibletransformationsto
acressyieldsthelist ofcandidatewordsinFig. 5.24.
Transformation
Correct Error Position
Error Correction Letter Letter (Letter#) Type
acress actress t – 2 deletion
acress cress – a 0 insertion
acress caress ca ac 0 transposition
acress access c r 2 substitution
acress across o e 3 substitution
acress acres – 2 5 insertion

acress acres – 2 4 insertion
Figure5.24 Candidate corrections for the misspelling acressand the transformations that
would have produced the error(after Kernighan et al.(1990) ). “–” represents a null letter.
The second stage of the algorithmscores each correctionby E q. 5.54. Let trepre-
sent the typo(themisspelled word),andlet crangeovertheset Cof candidatecorrec-
tions. Themostlikelycorrectionis then:
ˆc=argmax
c∈Clikelihood
P(t|c)prior
P(c) (5.55)

ˆc=argmax
The prior probability of each correction P(c)is the language model probability
of the word cin context; for in this section for pedagogical reasons we’l l make the
simplifying assumption that this is the unigram probabilit yP(c), but in practice in
spelling correction this is extended to trigram or 4-gram pr obabilities. Let’s use the
corpusofKernighanetal.(1990),whichisthe1988APnewswi recorpusof44million
words. Since in this corpus the word actressoccurs 1343 times out of 44 million,

the word acres2879 times, and so on, the resulting unigram prior probabili ties are as
follows:
cfreq(c)p(c)
actress1343.0000315
cress0.000000014
caress4.0000001
access2280.000058
across8436.00019
acres2879.000065

DRAFT166 Chapter 5. Part-of-SpeechTagging
How can we estimate P(t|c)? It is very difﬁcult to model the actual channel per-
fectly (i.e. computing the exact probability that a word wil l be mistyped) because it
would require knowing who the typist was, whether they were l eft-handed or right-
handed, and many other factors. Luckily, it turns out we can g et a pretty reasonable
estimate of p(t|c)just by looking at simple local context factors, because the most
importantfactorspredictingan insertion, deletion, tran spositionare the identity of the
correct letter itself, how the letter was misspelled, and th e surrounding context. For
example, the letters mandnare often substituted for each other; this is partly a fact
abouttheiridentity(thesetwolettersarepronouncedsimi larlyandtheyarenexttoeach
other on the keyboard), and partly a fact about context (beca use they are pronounced
similarly,theyoccurin similar contexts). Kernighanet al .(1990)useda simple model

of this sort. They estimated e.g. p(acress|across )just using the number of times that
theletter ewassubstitutedfortheletter oinsomelargecorpusoferrors. Thisisrepre-
sentedbya confusionmatrix ,asquare26×26matrixwhichrepresentsthenumberof confusionmatrix
times one letter was incorrectly used instead of another. Fo r example, the cell labeled
[o,e]inasubstitutionconfusionmatrixwouldgivethecountofti mesthatewassubsti-
tutedforo. Thecelllabeled [t,s]inaninsertionconfusionmatrixwouldgivethecount

of times that twas inserted after s. A confusion matrix can be computed by coding
a collection of spelling errors with the correct spelling an d then counting the number
of times different errors occurred (Grudin, 1983). Kernigh an et al. (1990) used four
confusionmatrices,oneforeachtypeofsingleerror:
•del[x,y]containsthe numberof times in the trainingset that the char actersxyin
thecorrectwordweretypedas x.
•ins[x,y]containsthenumberoftimesinthetrainingsetthatthechar acterxinthe
correctwordwastypedas xy.

•sub[x,y]thenumberoftimesthat xwastypedas y.
•trans[x,y]thenumberoftimesthat xywastypedas yx.
Note that they chose to condition their insertion and deleti on probabilities on the
previouscharacter;theycouldalsohavechosentoconditio nonthefollowingcharacter.
Usingthesematrices,theyestimated p(t|c)asfollows(where cpisthepthcharacterof
theword c):
P(t|c) =

del[cp−1,cp]
count [cp−1cp],if deletion
ins[cp−1,tp]
count [cp−1],if insertion
sub[tp,cp]
count [cp],if substitution
trans [cp,cp+1]

P(t|c) =

count [cpcp+1],if transposition(5.56)
Fig.5.25showstheﬁnalprobabilitiesforeachofthepotent ialcorrections;theuni-
grampriorismultipliedbythelikelihood(computedusingE q.5.56andthe confusion
matrices). Theﬁnal columnshowsthe“normalizedpercentag e”.
This implementation of the Bayesian algorithm predicts acresas the correct word
(at a totalnormalizedpercentageof45%),and actressasthe secondmostlikely word.

DRAFTSection5.9. Advanced: TheNoisyChannelModelforSpelling 167
c freq(c) p(c) p(t |c) p(t|c)p(c) %
actress 1343 .0000315 .000117 3 .69×10−937%
cress 0 .000000014 .00000144 2 .02×10−140%
caress 4 .0000001 .00000164 1 .64×10−130%
access 2280 .000058 .000000209 1 .21×10−110%
across 8436 .00019 .0000093 1 .77×10−918%
acres 2879 .000065 .0000321 2 .09×10−921%
acres 2879 .000065 .0000342 2 .22×10−923%
Figure5.25 Computation of the ranking for each candidate correction. N ote that the highest
ranked wordisnot actressbutacres(the twolines atthebottom of the table),since acrescanbe
generated intwoways. The del[],ins[],sub[],andtrans[] confusion matricesaregiven infullin
Kernighan et al.(1990).
Unfortunately, the algorithm was wrong here: The writer’s i ntention becomes clear
from the context: ...was called a “stellar and versatile acresswhose combination of
sassandglamourhasdeﬁnedher... ”. Thesurroundingwordsmakeitclearthat actress
andnotacreswastheintendedword. Thisisthereasonthatinpracticeweu setrigram

(orlarger)languagemodelsinthenoisychannelmodel,rath erthaanunigrams. Seeing
whether a bigrammodel of P(c)correctly solves this problem is left as Exercise 10
forthereader.
The algorithm as we have described it requires hand-annotat ed data to train the
confusion matrices. An alternative approach used by Kernig han et al. (1990) is to
compute the matrices by iteratively using this very spellin g error correctionalgorithm
itself. The iterative algorithm ﬁrst initializes the matri ces with equal values; thus any

character is equally likely to be deleted, equally likely to be substituted for any other
character, etc. Next the spelling error correction algorit hm is run on a set of spelling
errors. Giventhe set oftypospairedwith theircorrections ,theconfusionmatricescan
now be recomputed, the spelling algorithm run again, and so o n. This clever method
turnsout tobe an instanceofthe important EMalgorithm(Dempsteret al., 1977)that
we willdiscussinCh. 6.
5.9.1 Contextual Spelling Error Correction
As we mentionedabove,the noisy channel approachcan also be appliedto detect and

correctreal-wordspellingerrors, errorsthatresultinanactualwordofEnglish. Thisreal-word errordetection
can happen from typographicalerrors (insertion, deletion , transposition) that acciden-
tally produce a real word (e.g., thereforthree), or because the writer substituted the
wrong spelling of a homophoneor near-homophone(e.g., dessertfordesert, orpiece
forpeace). The task of correcting these errors is also called context-sensitive spell

correction . A number of studies suggest that between of 25% and 40% of spe llingcontext-sensitivespellcorrection
errorsarevalidEnglishwords(Kukich,1992);someofKukic h’sexamplesinclude:
Canthey lavehimmymessages?

DRAFT168 Chapter 5. Part-of-SpeechTagging
We can extend the noisy channel model to deal with real-word s pelling errors by
generating a candidate spelling set for every word in a sentence (Mays et al., 1991).
Thecandidateset includestheworditself,pluseveryEngli shwordthatwouldbegen-
erated from the word by either typographical modiﬁcations ( letter insertion, deletion,
substitution), or from a homophone list. The algorithm then chooses the spelling for
each word that gives the whole sentence the highest probabil ity. That is, given a sen-
tenceW={w1,w2,...,wk,...,wn}, wherewkhas alternativespelling w′
k,w′′
k, etc., we
choose the spelling among these possible spellings that max imizesP(W), using the
N-gramgrammartocompute P(W).
More recent research has focused on improving the channel mo delP(t|c), such
as by incorporatingphonetic information, or allowing more complex errors (Brill and
Moore, 2000; Toutanova and Moore, 2002). The most important improvement to the

k,w′′
k, etc., we
languagemodel P(c)is touse verylargecontexts,forexamplebyusingtheveryla rge
set of 5-grams publicly released by Google in 2006 (Franz and Brants, 2006). See
Norvig(2007)fora nice explanationand Pythonimplementat ionof the noisy channel
model;the endofthechapterhasfurtherpointers.
5.10 Summary
Thischapterintroducedthe idea of parts-of-speech andpart-of-speechtagging . The
mainideas:
•Languages generally have a relatively small set of closed class words, which
areoftenhighlyfrequent,generallyact as function words , andcan beveryam-

biguousintheirpart-of-speechtags. Openclasswordsgene rallyincludevarious
kindsofnouns,verbs,adjectives . Thereareanumberofpart-of-speechcoding
schemes,basedon tagsetsofbetween40and200tags.
•Part-of-speechtagging istheprocessofassigningapart-of-speechlabeltoeach
ofasequenceofwords. Rule-basedtaggersusehand-written rulestodistinguish
tag ambiguity. HMM taggers choose the tag sequence which max imizes the
productofwordlikelihoodandtagsequenceprobability. Ot hermachinelearning
modelsusedfortaggingincludemaximumentropyandotherlo g-linearmodels,

decisiontrees,memory-basedlearning,andtrans ¯for¯ ma¯tion-basedlearning.
•The probabilitiesin HMM taggersare trained on hand-labele dtraining corpora,
combiningdifferent N-gramlevelsusingdeletedinterpolation,andusingsophis -
ticatedunknownwordmodels.
•Given an HMM and an input string, the Viterbi algorithm is use d to decode the
optimaltagsequence.
•Taggersare evaluatedby comparingtheir outputfrom a test s et to human labels
for that test set. Error analysis can help pinpoint areas whe re a tagger doesn’t
performwell.

DRAFTSection5.10. Summary 169
The earliest implemented part-of-speechassignment algor ithm may have been part of
the parser in Zellig Harris’s Transformationsand Discours e AnalysisProject (TDAP),
which was implemented between June 1958 and July 1959 at the U niversity of Penn-
sylvania (Harris, 1962). Previous natural language proces sing systems had used dic-
tionaries with part-of-speech information for words, but h ave not been described as
performing part-of-speech disambiguation. As part of its p arsing, TDAP did part-of-
speech disambiguation via 14 hand-written rules, whose use of part-of-speech tag se-
quences preﬁgures all the modern algorithms, and which were run in an order based
on the relative frequencyof tagsfor a word. The parser/tagg erwas reimplementedre-
cently and is described by Joshi and Hopely (1999) and Karttu nen (1999), who note
that the parser was essentially implemented (in a very moder n way) as a cascade of
ﬁnite-statetransducers.

Soon after the TDAP parser was the Computational Grammar Cod er (CGC) of
Klein andSimmons(1963). The CGChadthreecomponents: a lex icon,a morpholog-
ical analyzer,anda contextdisambiguator. The small 1500- wordlexiconincludedex-
ceptionalwordsthat couldnot be accountedforin the simple morphologicalanalyzer,
including function words as well as irregular nouns, verbs, and adjectives. The mor-
phologicalanalyzerused inﬂectionalandderivationalsuf ﬁxestoassign part-of-speech
classes. A wordwas run throughthe lexiconand morphologica lanalyzerto producea

candidate set of parts-of-speech. A set of 500 context rules were then used to disam-
biguate this candidate set, by relying on surrounding islan ds of unambiguous words.
Forexample,onerulesaidthatbetweenanARTICLEandaVERB, theonlyallowable
sequences were ADJ-NOUN, NOUN-ADVERB, or NOUN-NOUN. The CG C algo-
rithmreported90%accuracyonapplyinga30-tagtagsettoar ticlesfromtheScientiﬁc
Americanandachildren’sencyclopedia.
TheTAGGITtagger(GreeneandRubin,1971)wasbasedontheKleinandSim mons

(1963)system,usingthesamearchitecturebutincreasingt hesizeofthedictionaryand
the size ofthe tagset (to87tags). For examplethefollowing sample rule,whichstates
that a word xis unlikelyto be a plural noun(NNS) beforea third personsin gularverb
(VBZ):
xVBZ→notNNS
TAGGITwas applied to the Brown corpus and, according to Francis and Kuˇ cera
(1982,p. 9),“resultedin theaccuratetaggingof77%of thec orpus”(theremainderof
theBrowncorpuswastaggedbyhand).
In the 1970s the Lancaster-Oslo/Bergen (LOB) corpus was com piled as a British

(VBZ):
English equivalentof the Brown corpus. It was tagged with th e CLAWS tagger (Mar-
shall,1983,1987;Garside,1987),aprobabilisticalgorit hmwhichcanbeviewedasan
approximation to the HMM tagging approach. The algorithm us ed tag bigram prob-
abilities, but instead of storing the word-likelihood of ea ch tag, tags were marked ei-
ther asrare(P(tag|word)< .01)infrequent (P(tag|word)< .10), ornormally frequent
(P(tag|word)> .10),

DRAFT170 Chapter 5. Part-of-SpeechTagging
The probabilistic PARTStagger of Church (1988) was very close to a full HMM
tagger. ItextendedtheCLAWSideatoassignfulllexicalpro babilitiestoeachword/tag
combination, and used Viterbi decoding to ﬁnd a tag sequence . Like the CLAWS
tagger,however,it storedtheprobabilityofthetag givent he word:
P(tag|word)∗P(tag|previousntags) (5.57)
ratherthanusingtheprobabilityofthewordgiventhetag,a sanHMMtaggerdoes:
P(word|tag)∗P(tag|previousntags) (5.58)
LatertaggersexplicitlyintroducedtheuseoftheHiddenMa rkovModel,oftenwith
the EM training algorithm (Kupiec, 1992; Merialdo, 1994; We ischedel et al., 1993),
includingthe useofvariable-lengthMarkovmodels(Sch¨ ut zeandSinger,1994).
Most recent tagging algorithms, like the HMM and TBL approac heswe have dis-
cussed, are machine-learning classiﬁers which estimate th e best tag-sequence for a
sentence given variousfeatures such as the current word, ne ighboringparts-of-speech

or words, and unknown word features such as orthographic and morphological fea-
tures. Many kinds of classiﬁers have been used to combine the se features, includ-
ing decision trees (Jelinek et al., 1994; Magerman,1995a), maximumentropymodels
(Ratnaparkhi, 1996), other log-linear models (Franz, 1996 ), memory-based learning
(Daelemans et al., 1996), and networks of linear separators (SNOW) (Roth and Ze-
lenko, 1998). Most machine learning models seem to achieve r elatively similar per-

formancegivensimilar features,roughly96-97%ontheTree bank45-tagtagsetonthe
Wall Street Journal corpus. As of the writing of this chapter , the highest performing
publishedmodelonthisWSJTreebanktaskisalog-lineartag gerthatusesinformation
about neighboring words as well as tags, and a sophisticated unknown-word model,
achieving 97.24% accuracy (Toutanova et al., 2003). Most su ch models are super-
vised,althoughthereis beginningtobeworkonunsupervise dmodels(Sch¨ utze,1995;

Brill, 1997;Clark,2000;BankoandMoore,2004;Goldwatera ndGrifﬁths,2007).
Readersinterestedin the historyof parts-of-speechshoul dconsult a historyof lin-
guistics such as Robins (1967) or Koerner and Asher (1995), p articularly the article
by Householder(1995) in the latter. Sampson (1987) and Gars ide et al. (1997) give a
detailedsummaryoftheprovenanceandmakeupoftheBrownan dothertagsets. More
informationonpart-of-speechtaggingcanbefoundinvanHa lteren(1999).
Algorithms for spelling error detection and correction hav e existed since at least

Blair (1960). Most early algorithm were based on similarity keys like the Soundex
algorithm discussed in the exercises on page 82 (Odell and Ru ssell, 1922; Knuth,
1973). Damerau (1964) gave a dictionary-based algorithm fo r error detection; most
error-detection algorithms since then have been based on di ctionaries. Damerau also
gave a correction algorithm that worked for single errors. M ost algorithms since then
havereliedondynamicprogramming,beginningwithWagnera ndFischer(1974). Ku-

kich (1992) is the deﬁnitive survey article on spelling erro r detection and correction.
Modern algorithms are based on statistical or machine learn ing algorithm, following
e.g., Kashyap and Oommen (1983) and Kernighan et al. (1990). Recent approaches
to spelling include extensions to the noisy channel model (B rill and Moore, 2000;
Toutanova and Moore, 2002) as well as many other machine lear ning architectures

DRAFTSection5.10. Summary 171
such as Bayesian classiﬁers, (Gale et al., 1993; Golding, 19 97; Golding and Sch-
abes, 1996), decision lists (Yarowsky, 1994), transformat ion based learning (Mangu
andBrill,1997)latentsemanticanalysis(JonesandMartin ,1997)andWinnow(Gold-
ingandRoth,1999). HirstandBudanitsky(2005)explorethe useofwordrelatedness;
see Ch. 20. Noisy channel spelling correction is used in a num ber of commercial ap-
plications,includingtheMicrosoftWordcontextualspell checker.
Exercises
5.1Find one tagging error in each of the following sentences tha t are tagged with
thePennTreebanktagset:
a. I/PRP need/VBPa/DT ﬂight/NNfrom/INAtlanta/NN
b. Does/VBZ this/DTﬂight/NNserve/VBdinner/NNS
c. I/PRP have/VBa/DT friend/NNliving/VBGin/IN Denver/NNP
d. Can/VBP you/PRPlist/VB the/DTnonstop/JJafternoon/NNﬂ ights/NNS
5.2Use the Penn Treebank tagset to tag each word in the following sentences from
DamonRunyon’sshort stories. You may ignore punctuation. S ome of these are
quitedifﬁcult;doyourbest.

a. It isanice night.
b. Thiscrapgameis overa garageinFifty-secondStreet...
c. ...Nobodyevertakesthenewspapersshe sells...
d. Heisatall,skinnyguywithalong,sad,mean-lookingkisse r,andamourn-
fulvoice.
e. ...I am sitting in Mindy’srestaurantputtingonthe geﬁllt e ﬁsh, whichis a
dishI amveryfondof,...
f. Whenaguyandadollgettotakingpeeksbackandforthateach other,why
thereyouareindeed.
5.3Now compare your tags from the previous exercise with one or t wo friend’s
answers. On whichwordsdidyoudisagreethemost? Why?

fulvoice.
5.4Now tag the sentences in Exercise 2 using the more detailed Br own tagset in
Fig.5.2.
5.5Implementthe TBL algorithm in Fig. 5.21. Create a small numb erof templates
andtrainthetaggeronanyPOS-taggedtrainingset youcanﬁn d.
5.6Implement the “most-likely tag” baseline. Find a POS-tagge d training set, and
use it to computeforeach wordthe tag whichmaximizes p(t|w). Youwill need
to implement a simple tokenizer to deal with sentence bounda ries. Start by as-

Fig.5.2.
suming all unknown words are NN and compute your error rate on known and
unknownwords. Nowwriteatleast5rulestodoabetterjoboft aggingunknown
words,andshowthedifferenceinerrorrates.

DRAFT172 Chapter 5. Part-of-SpeechTagging
5.7RecallthattheChurch(1988)taggerisnotanHMMtaggersinc eit incorporates
theprobabilityofthetaggiventheword:
P(tag|word)∗P(tag|previousntags) (5.59)
rather than using the likelihood of the word given the tag, as an HMM tagger
does:
P(word|tag)∗P(tag|previousntags) (5.60)
Asagedanken-experiment,constructasentence,aset oftag transitionprobabil-
ities, and a set of lexical tag probabilities that demonstra te a way in which the
HMM tagger can produce a better answer than the Church tagger , and another
examplein whichthe Churchtaggerisbetter.
5.8Build an HMM tagger. This requires (1) that you have implemen ted the Viterbi
algorithm from this chapter and Ch. 6, (2) that you have a dict ionary with part-
of-speech information and (3) that you have either (a) a part -of-speech-tagged
corpus or (b) an implementation of the Forward Backward algo rithm. If you
have a labeled corpus, train the transition and observation probabilities of an

HMMtaggerdirectlyonthe hand-taggeddata. If youhaveanun labeledcorpus,
trainusingForwardBackward.
5.9Nowrunyouralgorithmonasmalltestsetthatyouhavehand-l abeled. Findﬁve
errorsandanalyzethem.
5.10Computeabigramgrammaronalargecorpusandreestimatethe spellingcorrec-
tion probabilitiesshown in Fig. 5.25 giventhe correctsequ ence...was called a
“stellarandversatile acresswhosecombinationofsassandglamourhasdeﬁned
her...”. Doesabigramgrammarpreferthecorrectword actress?
5.11Read Norvig (2007) and implement one of the extensions he sug gests to his
Pythonnoisychannelspell checker.

Chapter6
Hidden Markov and Maximum En-tropy Models
Numquamponendaest pluralitassine necessitat
‘Pluralityshouldneverbeproposedunlessneeded’
WilliamofOccam
HersisterwascalledTatiana.
Fortheﬁrst time withsuchaname
thetenderpagesofa novel,
we’ll whimsicallygrace.
Pushkin,EugeneOnegin ,in theNabokovtranslation
Alexander Pushkin’s novel in verse, Eugene Onegin , serialized in the early 19th cen-
tury, tells of the young dandy Onegin, his rejection of the lo ve of young Tatiana, his
duel with his friend Lenski, and his later regret for both mis takes. But the novel is

mainly belovedfor its style and structurerather than its pl ot. Amongother interesting
structuralinnovations,the noveliswritten in a formnowkn ownas the Oneginstanza ,
iambictetrameterwithanunusualrhymescheme. Theseeleme ntshavecausedcompli-
cationsandcontroversyinitstranslationintootherlangu ages. Manyofthetranslations
have been in verse, but Nabokov famously translated it stric tly literally into English
prose. The issue of its translation, and the tension between literal and verse transla-

tionshaveinspiredmuchcommentary(see forexampleHofsta dter(1997)).
In 1913 A. A. Markov asked a less controversial question abou t Pushkin’s text:
could we use frequency counts from the text to help compute th e probability that the
next letter in sequence would be a vowel. In this chapter we in troduce two impor-
tant classes of statistical models for processing text and s peech, both descendants of
Markov’s models. One of them is the Hidden Markov Model (HMM). The other,

is theMaximum Entropy model (MaxEnt), and particularly a Markov-related vari-
antofMaxEntcalledthe Maximum EntropyMarkovModel (MEMM). Allofthese
aremachine learning models. We have already touched on some aspects of machine
learning;indeedwebrieﬂyintroducedtheHiddenMarkovMod elinthepreviouschap-
ter,andwehaveintroducedthe N-grammodelinthechapterbefore. Inthischapterwe
givea morecompleteandformalintroductiontothese twoimp ortantmodels.
HMMs and MEMMs are both sequence classiﬁers . A sequence classiﬁer or se-Sequence
classiﬁer

classiﬁer
quence labeler is a model whose job is to assign some label or class to each uni t in a
sequence. Theﬁnite-state transducerwe studiedinCh. 3is a kindofnon-probabilistic
sequenceclassiﬁer, for example transducingfrom sequence sof wordsto sequencesof
morphemes. TheHMMandMEMMextendthisnotionbybeingproba bilisticsequence
classiﬁers; givena sequenceofunits(words,letters,morp hemes,sentences,whatever)
their job is to compute a probability distribution over poss ible labels and choose the
best labelsequence.

DRAFT174 Chapter 6. HiddenMarkovandMaximumEntropyModels
We have already seen one important sequence classiﬁcation t ask: part-of-speech
tagging, where each word in a sequence has to be assigned a par t-of-speech tag. Se-
quencelabelingtasks come up throughoutspeechand languag eprocessing,a fact that
isn’ttoosurprisingifweconsiderthatlanguageconsistso fsequencesatmanyrepresen-
tationallevels. Besidespart-of-speechtagging,inthisb ookwewillseetheapplication
of these sequencemodelsto tasks like speech recognition(C h. 9), sentencesegmenta-
tion and grapheme-to-phonemeconversion(Ch. 8), partial p arsing/chunking(Ch. 13),
andnamedentityrecognitionandinformationextraction(C h.22).
Thischapterisroughlydividedintotwosections: HiddenMa rkovModelsfollowed
by Maximum Entropy Markov Models. Our discussion of the Hidd en Markov Model
extends what we said about HMM part-of-speech tagging. We be gin in the next sec-
tion by introducing the Markov Chain, then give a detailed ov erview of HMMs and

the forward and Viterbi algorithmswith more formalization , and ﬁnally introduce the
important EM algorithm for unsupervised (or semi-supervis ed) learning of a Hidden
MarkovModel.
Inthesecondhalfofthechapter,weintroduceMaximumEntro pyMarkovModels
gradually, beginning with techniques that may already be fa miliar to you from statis-
tics: linear regressionand logistic regression. We next in troduceMaxEnt. MaxEnt by
itself is not a sequence classiﬁer; it is used to assign a clas s to a single element. The

name Maximum Entropy comes from the idea that the classiﬁer ﬁ nds the probabilis-
tic model which follows Occam’s Razor in being the simplest ( least constrained; has
the maximum entropy) yet still consistent with some speciﬁc constraints. The Maxi-
mumEntropyMarkovModelistheextensionofMaxEnttotheseq uencelabelingtask,
addingcomponentssuchastheViterbialgorithm.
Although this chapter introduces MaxEnt, which is a classiﬁ er, we will not focus

in general on non-sequential classiﬁcation. Non-sequenti al classiﬁcation will be ad-
dressedinlaterchapterswiththeintroductionofclassiﬁe rslikethe GaussianMixture
Modelin(Ch.9)andthe NaiveBayes anddecision list classiﬁersin(Ch.20).
6.1 MarkovChains
The Hidden Markov Model is one of the most important machine l earning models in
speech and language processing. In order to deﬁne it properl y, we need to ﬁrst in-
troduce the Markov chain , sometimes called the observed Markov model . Markov
chainsandHiddenMarkovModelsarebothextensionsoftheﬁn iteautomataofCh.2.

Recall that a ﬁnite automaton is deﬁned by a set of states, and a set of transitions be-
tween states that are taken based on the input observations. Aweighted ﬁnite-state WeightedFSA
automaton isasimpleaugmentationoftheﬁniteautomatoninwhicheach arcisasso-
ciatedwithaprobability,indicatinghowlikelythatpathi stobetaken. Theprobability
onall thearcsleavinganodemust sumto1.
AMarkov chain is a special case of a weighted automaton in which the input Markov chain
sequence uniquely determines which states the automaton wi ll go through. Because

it can’t represent inherently ambiguous problems, a Markov chain is only useful for
assigningprobabilitiesto unambiguoussequences.

of words w1...wn. ThisMarkovchain shouldbe familiar; in fact it represents a bigram
language model. Given the two models in Fig. 6.1 we can assign a probability to any
sequencefromourvocabulary. We’ll gooverhowtodothissho rtly.
First, let’s be more formal. We’ll view a Markov chain as a kin d of probabilis-
ticgraphical model ; a way of representing probabilistic assumptions in a graph . A
Markovchainisspeciﬁedby thefollowingcomponents:
Q=q1q2...qN aset ofNstates
A=a01a02...an1...annatransition probability matrix A, eachaijrep-
resenting the probability of moving from state i
tostatej, s.t.∑n
j=1aij=1∀i
q0,qF a special start state andend (ﬁnal) state which
arenotassociatedwithobservations.
Fig.6.1showsthatwe representthestates(includingstart andendstates) asnodes
inthe graph,andthetransitionsasedgesbetweennodes.
A Markovchain embodiesan importantassumptionaboutthese probabilities. Ina
ﬁrst-order Markovchain,theprobabilityofaparticularstate isdepen dentonlyontheFirst-orderMarkov chain
previousstate:

MarkovAssumption: P(qi|q1...qi−1) =P(qi|qi−1) (6.1)
Notethatbecauseeach aijexpressestheprobability p(qj|qi),the lawsofprobabil-
ityrequirethat thevaluesoftheoutgoingarcsfroma givens tate mustsumto1:
n
∑
j=1aij=1∀i (6.2)
An alternate representationthat is sometimes used for Mark ov chains doesn’t rely

DRAFT176 Chapter 6. HiddenMarkovandMaximumEntropyModels
on a start or end state, instead representing the distributi on over initial states and ac-
ceptingstatesexplicitly:
π=π1,π2,...,πNaninitial probability distribution over states. πiis the
probabilitythat the Markovchain will start in state i. Some
statesjmayhave πj=0,meaningthattheycannotbeinitial
states. Also, ∑n
i=1πi=1
QA={qx,qy...}a setQA⊂Qoflegalacceptingstates
Thusthe probabilityof state 1 being the ﬁrst state can be rep resentedeither as a01
or asπ1. Note that because each πiexpresses the probability p(qi|START ), all theπ
probabilitiesmustsumto 1:
n
∑
i=1πi=1 (6.3)
(a) (b)
Figure6.2 Another representation of the same Markov chain for weather shown in Fig. 6.1. Instead of using a
specialstartstatewith a01transitionprobabilities,weusethe πvector,whichrepresentsthedistributionoverstarting
state probabilities. The ﬁgure in(b) shows sample probabil ities.
Before you go on, use the sample probabilities in Fig. 6.2b to compute the proba-

n
∑
(a) (b)
bilityofeachofthefollowingsequences:
(6.4)hothothothot
(6.5)coldhotcoldhot
What doesthe differencein these probabilitiestell you abo uta real-worldweather
factencodedinFig.6.2b?

DRAFTSection6.2. TheHiddenMarkovModel 177
6.2 TheHiddenMarkovModel
A Markov chain is useful when we need to compute a probability for a sequence of
events that we can observe in the world. In many cases, howeve r, the events we are
interested in may not be directly observable in the world. Fo r example, in part-of-
speech tagging (Ch. 5) we didn’t observe part of speech tags i n the world; we saw
words, and had to infer the correct tags from the word sequenc e. We call the part-of-
speech tags hiddenbecause they are not observed. The same architecture will co me
up in speech recognition; in that case we’ll see acoustic eve nts in the world, and have
to infer the presence of ‘hidden’ words that are the underlyi ng causal source of the
acoustics. A Hidden Markov Model (HMM) allows us to talk about both observedHiddenMarkovModel
events(likewordsthatweseeintheinput)and hiddenevents(likepart-of-speechtags)
thatwe thinkofascausalfactorsin ourprobabilisticmodel .

Toexemplifythese models,we’ll use a task conceivedof byJa son Eisner(2002a).
Imagine that you are a climatologist in the year 2799 studyin g the history of global
warming. You cannot ﬁnd any records of the weather in Baltimo re, Maryland, for the
summerof2007,butyoudoﬁndJasonEisner’sdiary,whichlis tshowmanyicecreams
Jason ate everyday that summer. Our goal is to use these obser vationsto estimate the
temperatureeveryday. We’llsimplifythisweathertaskbya ssumingthereareonlytwo

kindsofdays: cold(C) andhot(H).Sothe Eisnertask isasfol lows:
Given a sequence of observations O, each observation an integer corre-
spondingto the numberof ice creamseaten on a givenday, ﬁgur e out the
correct‘hidden’sequence Qofweatherstates(HorC)whichcausedJason
toeatthe icecream.
Let’s begin with a formal deﬁnition of a Hidden Markov Model, focusing on how
it differsfroma Markovchain. An HMMisspeciﬁedbythefollowingcomponents: HMM
A=a11a12...an1...annatransition probabilitymatrix A, eachaijrep-

O=o1o2...oT a sequence of Tobservations , each one drawn
froma vocabulary V=v1,v2,...,vV.
B=bi(ot) a sequence of observation likelihoods: , also
calledemission probabilities , each expressing
the probability of an observation otbeing gen-
eratedfromastate i.
are not associated with observations, together
withtransitionprobabilities a01a02..a0noutofthe
start stateand a1Fa2F...anFintotheendstate.

Aswe notedforMarkovchains, analternaterepresentationt hatis sometimesused
forHMMsdoesn’trelyonastartorendstate,insteadreprese ntingthedistributionover

DRAFT178 Chapter 6. HiddenMarkovandMaximumEntropyModels
initialandacceptingstatesexplicitly. Wewon’tbeusingt heπnotationinthistextbook,
butyoumaysee it intheliterature:
i=1πi=1
Aﬁrst-orderHiddenMarkovModelinstantiatestwosimplify ingassumptions. First,
aswithaﬁrst-orderMarkovchain,theprobabilityofaparti cularstateisdependentonly
onthepreviousstate:
MarkovAssumption: P(qi|q1...qi−1) =P(qi|qi−1) (6.6)
Second, the probability of an output observation oiis dependent only on the state
thatproducedtheobservation qi,andnotonanyotherstatesoranyotherobservations:
OutputIndependence: P(oi|q1...qi,...,qT,o1,...,oi,...,oT) =P(oi|qi) (6.7)
Fig. 6.3 shows a sample HMM for the ice cream task. The two hidd en states (H

and C) correspond to hot and cold weather, while the observat ions (drawn from the
alphabetO={1,2,3}) correspond to the number of ice creams eaten by Jason on a
givenday.
Figure6.3 A Hidden Markov Model for relating numbers of ice creams eate n by Jason (the
observations) to the weather (H or C, the hidden variables). For this example we are not using
anend-state, instead allowingboth states1and 2tobe a ﬁnal (accepting) state.
NoticethatintheHMMinFig.6.3,thereisa(non-zero)proba bilityoftransitioning

givenday.
betweenanytwo states. Suchan HMM iscalleda fully-connected orergodicHMM . ErgodicHMM
Sometimes,however,we haveHMMs in whichmanyofthe transit ionsbetweenstates
have zero probability. For example, in left-to-right (also called Bakis) HMMs, the Bakisnetwork
state transitions proceed from left to right, as shown in Fig . 6.4. In a Bakis HMM,
there are notransitionsgoingfroma higher-numberedstate to a lower-numberedstate

DRAFTSection6.3. ComputingLikelihood: TheForwardAlgorithm 1 79
(or,moreaccurately,anytransitionsfromahigher-number edstatetoalower-numbered
state have zero probability). Bakis HMMs are generally used to model temporal pro-
cesseslike speech;we will see moreoftheminCh. 9.
Figure6.4 Two 4-state Hidden Markov Models; a left-to-right (Bakis) H MM on the left, and
a fully-connected (ergodic) HMM on the right. In the Bakis mo del, all transitions not shown
have zeroprobability.
Now that we have seen the structure of an HMM, we turn to algori thms for com-
puting things with them. An inﬂuential tutorial by Rabiner ( 1989), based on tutorials
byJackFergusoninthe1960s,introducedtheideathatHidde nMarkovModelsshould
becharacterizedby threefundamentalproblems :
Problem1(Likelihood): Given an HMM λ= (A,B)and an observation se-
quenceO,determinethelikelihood P(O|λ).
Problem2(Decoding): Givenanobservationsequence Oandan HMM λ=
(A,B), discoverthebest hiddenstate sequence Q.

Problem3(Learning): Givenanobservationsequence Oandthesetofstates
inthe HMM,learnthe HMMparameters AandB.
We already saw an example of problem(2) in Ch. 5. In the next th ree sections we
introduceall threeproblemsmoreformally.
6.3 ComputingLikelihood: TheForwardAlgorithm
Ourﬁrstproblemistocomputethelikelihoodofaparticular observationsequence. For
example, given the HMM in Fig. 6.2b, what is the probability o f the sequence 3 1 3?
Moreformally:
Computing Likelihood: Given an HMM λ= (A,B)and an observation

sequence O,determinethelikelihood P(O|λ).
For a Markov chain, where the surface observations are the sa me as the hidden
events,wecouldcomputetheprobabilityof 313justbyfollowingthestateslabeled 31
3andmultiplyingtheprobabilitiesalongthearcs. ForaHidd enMarkovModel,things
are not so simple. We want to determine the probability of an i ce-cream observation
sequencelike 3 13,butwe don’tknowwhat thehiddenstatesequenceis!

DRAFT180 Chapter 6. HiddenMarkovandMaximumEntropyModels
Let’sstart with a slightlysimplersituation. Supposewe al readyknewthe weather,
and wanted to predict how much ice cream Jason would eat. This is a useful part of
manyHMM tasks. Fora givenhiddenstate sequence(e.g. hot hotcold )we can easily
computetheoutputlikelihoodof 3 13.
Let’ssee how. First, recall that forHiddenMarkovModels, e achhiddenstate pro-
duces only a single observation. Thus the sequence of hidden states and the sequence
ofobservationshavethesamelength.1
Giventhisone-to-onemapping,andtheMarkovassumptionse xpressedinEq.6.6,
foraparticularhiddenstatesequence Q=q0,q1,q2,...,qTandanobservationsequence
O=o1,o2,...,oT, thelikelihoodoftheobservationsequenceis:
P(O|Q) =T
∏
i=1P(oi|qi) (6.8)
The computation of the forward probability for our ice-crea m observation 3 1 3
from one possible hidden state sequence hot hot cold is as follows (Fig. 6.5 shows a
graphicrepresentationofthis):
P(313|hothotcold ) =P(3|hot)×P(1|hot)×P(3|cold) (6.9)

P(O|Q) =T
∏
Figure6.5 Thecomputationoftheobservationlikelihoodfortheice-c reamevents 313given
the hidden state sequence hot hot cold .
But of course, we don’t actually know what the hidden state (w eather) sequence
was. We’ll need to computethe probabilityof ice-creameven ts3 1 3instead by sum-
ming over all possible weather sequences, weighted by their probability. First, let’s
computethejointprobabilityofbeingin aparticularweath ersequence Qandgenerat-
ingaparticularsequence Oofice-creamevents. Ingeneral,thisis:
P(O,Q) =P(O|Q)×P(Q) =n
∏

∏
i=1P(oi|qi)×n
∏
i=1P(qi|qi−1) (6.10)
The computation of the joint probability of our ice-cream ob servation 3 1 3and
onepossiblehiddenstate sequence hothotcold isasfollows(Fig.6.6showsagraphic
representationofthis):
1Therearevariants ofHMMscalled segmentalHMMs (inspeechrecognition) or semi-HMMs (innatural
language processing) in which this one-to-one mapping betw een thelength of thehidden state sequence and
the length of the observation sequence does not hold.

DRAFTSection6.3. ComputingLikelihood: TheForwardAlgorithm 1 81
P(31 3,hothotcold ) =P(hot|start)×P(hot|hot)×P(cold|hot)
×P(3|hot)×P(1|hot)×P(3|cold) (6.11)
Figure6.6 The computation of the joint probability of the ice-cream ev ents3 1 3and the
hidden state sequence hot hot cold .
Nowthatwe knowhowtocomputethe jointprobabilityof theob servationswitha
particular hidden state sequence, we can compute the total p robability of the observa-
tionsjust bysummingoverall possiblehiddenstate sequenc es:
P(O) =∑
QP(O,Q) =∑
QP(O|Q)P(Q) (6.12)
For our particular case, we would sum over the 8 three-event s equences cold cold
cold,coldcoldhot ,i.e.:
P(313)=P(313,coldcoldcold )+P(313,coldcoldhot )+P(313,hothotcold )+...
(6.13)
ForanHMM with Nhiddenstatesandanobservationsequenceof Tobservations,
there areNTpossible hiddensequences. For real tasks, where NandTare bothlarge,
NTis a very large number,and so we cannot computethe total obse rvationlikelihood

P(O) =∑
QP(O,Q) =∑
(6.13)
bycomputingaseparateobservationlikelihoodforeachhid denstatesequenceandthen
summingthemup.
Instead of using such an extremely exponential algorithm, w e use an efﬁcient
(O(N2T)) algorithm called the forward algorithm . The forward algorithm is a kindForwardalgorithm
ofdynamicprogramming algorithm,i.e.,analgorithmthatusesatabletostoreinte r-
mediatevaluesasitbuildsuptheprobabilityoftheobserva tionsequence. Theforward
algorithm computes the observation probability by summing over the probabilities of

allpossiblehiddenstatepathsthatcouldgeneratetheobse rvationsequence,butitdoes
so efﬁcientlybyimplicitlyfoldingeachofthesepathsinto a singleforwardtrellis .
Fig. 6.7 shows an example of the forwardtrellis for computin gthe likelihood of 3
13giventhe hiddenstate sequence hothotcold .
Each cell of the forward algorithmtrellis αt(j)representsthe probability of being
instatejafterseeingtheﬁrst tobservations,giventheautomaton λ. Thevalueofeach
cellαt(j)iscomputedbysummingovertheprobabilitiesofeverypatht hatcouldlead

ustothiscell. Formally,eachcell expressesthe following probability:
αt(j) =P(o1,o2...ot,qt=j|λ) (6.14)

DRAFT182 Chapter 6. HiddenMarkovandMaximumEntropyModels
startH!H!H!end
P(C|start) * P(3|C).2 * .1P(H|H) * P(1|H)
.7 * .2
P(C|C) * P(1|C)
.6 * .5P(C|H) * P(1|C)
.3 * .5
P(H|C) * P(1|H).4 * .2
P(H|start)*P(3|H).8 * .4α1(2)=.32
α1(1) = .02α2(2)= .32*.014 + .02*.08 = .00608
α2(1) = .32*.15 + .02*.30 = .054
tCHend end end qF
q2
q1
q0
o13 3 1
o2 o3
Figure6.7 Theforwardtrellisforcomputingthetotalobservationlik elihoodfortheice-creamevents 313. Hidden
states are in circles, observations in squares. White (unﬁl led) circles indicate illegal transitions. The ﬁgure shows
the computation of αt(j)for two states at two time steps. The computation in each cell follows Eq. 6.15: αt(j) =
∑N
i=1αt−1(i)aijbj(ot). Theresultingprobabilityexpressed ineachcellisEq.6.1 4:αt(j) =P(o1,o2...ot,qt=j|λ).
Hereqt=jmeans“theprobabilitythatthe tthstateinthesequenceofstatesisstate
j”. We compute this probability by summing over the extension s of all the paths that

∑N
leadtothe currentcell. Foragivenstate qjat timet, thevalue αt(j)is computedas:
αt(j) =N
∑
i=1αt−1(i)aijbj(ot) (6.15)
The three factors that are multipliedin Eq. 6.15in extendin gthe previouspaths to
computetheforwardprobabilityattime tare:
αt−1(i)thepreviousforwardpath probability fromtheprevioustimestep
aij thetransitionprobability frompreviousstate qitocurrentstate qj
the currentstate j

αt(j) =N
∑
Consider the computationin Fig. 6.7 of α2(1), the forwardprobability of being at
timestep2instate1havinggeneratedthepartialobservati on31. Thisiscomputedby

DRAFTSection6.3. ComputingLikelihood: TheForwardAlgorithm 1 83
extendingthe αprobabilitiesfromtimestep1,viatwopaths,eachextensio nconsisting
ofthethreefactorsabove: α1(1)×P(H|H)×P(1|H)andα1(2)×P(H|C)×P(1|H).
Fig. 6.8showsanothervisualizationof thisinductionstep forcomputingthe value
inonenewcell ofthetrellis.
"
t-1ota1ja2jaNj
a3j
bj(ot)
#$
%
& ' (
i αt31(i) aij bj(ot) 
q1q2q3qN
q1qj
q2
q1q2
ot+1ot-2q1q2q3q3qNqNαt31(N)
αt31(3)
αt31(2)
αt31(1)αt32(N)
αt32(3)
αt32(2)
αt32(1)
Figure6.8 Visualizing the computation of a single element αt(i)in the trellis by summing
all the previous values αt−1weighted by their transition probabilities aand multiplying by the
observation probability bi(ot+1). For many applications of HMMs, many of the transitionprob-
abilities are 0, so not all previous states will contribute t o the forward probability of the current
state. Hidden states are in circles, observations in square s. Shaded nodes are included in the
probability computation for αt(i). Startand endstates are not shown.

function FORWARD(observations of lenT,state-graph of lenN)returnsforward-prob
create a probabilitymatrix forward[N+2,T]
forward[s,1]←a0,s∗bs(o1)
forward[s,t]←N
∑
s′=1forward [s′,t−1]∗as′,s∗bs(ot)
forward[qF,T]←N
∑
s=1forward [s,T]∗as,qF ; terminationstep
returnforward[qF,T]
Figure6.9 The forwardalgorithm. We’ve used the notation forward [s,t]torepresent αt(s).
Wegivetwoformaldeﬁnitionsoftheforwardalgorithm;thep seudocodeinFig.6.9

∑
∑
andastatementofthedeﬁnitionalrecursionhere:
1. Initialization:
α1(j) =a0jbj(o1)1≤j≤N (6.16)

DRAFT184 Chapter 6. HiddenMarkovandMaximumEntropyModels
2. Recursion(sincestates0andF arenon-emitting):
αt(j) =N
∑
i=1αt−1(i)aijbj(ot); 1≤j≤N,1<t≤T (6.17)
3. Termination:
P(O|λ) =αT(qF) =N
∑
i=1αT(i)aiF (6.18)
6.4 Decoding: TheViterbiAlgorithm
Foranymodel,suchasanHMM,thatcontainshiddenvariables ,thetaskofdetermining
whichsequenceofvariablesistheunderlyingsourceofsome sequenceofobservations
is called the decoding task. In the ice cream domain, given a sequence of ice cream Decoding
observations 3 1 3and an HMM, the task of the decoderis to ﬁnd the best hidden Decoder
weathersequence( HH H).Moreformally,
Decoding : Given as input an HMM λ= (A,B)and a sequence of ob-
servations O=o1,o2,...,oT, ﬁnd the most probable sequence of states
Q=q1q2q3...qT.
We might propose to ﬁnd the best sequence as follows: for each possible hidden
statesequence( HHH,HHC,HCH,etc.),wecouldruntheforwardalgorithmandcom-
putethelikelihoodoftheobservationsequencegiventhath iddenstatesequence. Then

we could choose the hidden state sequence with the max observ ation likelihood. It
should be clear from the previous section that we cannot do th is because there are an
exponentiallylargenumberofstate sequences!
Instead, the most common decoding algorithms for HMMs is the Viterbi algo-
rithm. Like the forwardalgorithm, Viterbiis a kind of dynamic programming , and Viterbialgorithm
makesuses ofa dynamicprogrammingtrellis. Viterbi also st ronglyresemblesanother

dynamicprogrammingvariant,the minimum edit distance algorithmofCh.3.
Fig.6.10showsanexampleoftheViterbitrellisforcomputi ngthebesthiddenstate
sequencefortheobservationsequence 313. Theideaistoprocesstheobservationse-
quencelefttoright,ﬁllingoutthetrellis. Eachcellofthe Viterbitrellis, vt(j)represents
theprobabilitythattheHMMisinstate jafterseeingtheﬁrst tobservationsandpass-
ing through the most probable state sequence q0,q1,...,qt−1, given the automaton λ.
The value of each cell vt(j)is computedby recursivelytaking the most probablepath

thatcouldleadustothiscell. Formally,eachcellexpresse sthefollowingprobability:
vt(j) =max
q0,q1,...,qt−1P(q0,q1...qt−1,o1,o2...ot,qt=j|λ) (6.19)
Note that we represent the most probable path by taking the ma ximum over all
possible previous state sequences max
q0,q1,...,qt−1. Like other dynamic programming algo-
rithms, Viterbi ﬁlls each cell recursively. Given that we ha d already computed the
probabilityofbeingin everystateat time t−1,We computetheViterbiprobabilityby

DRAFTSection6.4. Decoding: TheViterbiAlgorithm 185
startH
CH
CH
Cend
.7 * .2
.3 * .5
P(H|start)*P(3|H).8 * .4v1(2)=.32
v1(1) = .02v2(2)= max(.32*.014, .02*.08) = .0448
v2(1) = max(.32*.15, .02*.30) = .048
q2
q1
q0
o13 3 1
o2 o3
Figure6.10 TheViterbitrellisfor computingthebest paththrough theh iddenstatespace fortheice-cream eating
events313. Hiddenstatesareincircles,observations insquares. Whi te(unﬁlled)circlesindicateillegaltransitions.
The ﬁgure shows the computation of vt(j)for two states at two time steps. The computation in each cell follows
Eq.6.20: vt(j) =max1≤i≤N−1vt−1(i)aijbj(ot)Theresultingprobabilityexpressed ineachcellisEq.6.19 :vt(j) =
P(q0,q1,...,qt−1,o1,o2,...,ot,qt=j|λ).
takingthemostprobableoftheextensionsofthepathsthatl eadtothecurrentcell. For
a givenstate qjattimet,thevalue vt(j)iscomputedas:
i=1vt−1(i)aijbj(ot) (6.20)

ThethreefactorsthataremultipliedinEq.6.20forextendi ngthepreviouspathsto
vt−1(i)thepreviousViterbipathprobability fromthe previoustime step
Fig. 6.11 shows pseudocode for the Viterbi algorithm. Note t hat the Viterbi algo-
rithmisidenticaltotheforwardalgorithmexceptthatitta kesthemaxovertheprevious

DRAFT186 Chapter 6. HiddenMarkovandMaximumEntropyModels
Figure6.11 Viterbi algorithm for ﬁnding optimal sequence of hidden sta tes. Given an obser-
vationsequence andanHMM λ= (A,B),thealgorithmreturnsthestate-paththroughtheHMM
which assigns maximum likelihood to the observation sequen ce. Note that states 0 and qFare
non-emitting.
pathprobabilitieswheretheforwardalgorithmtakesthe sum. NotealsothattheViterbi

algorithmhas one componentthat the forwardalgorithmdoes n’t have: backpointers .
This is because while the forward algorithm needs to produce an observation likeli-
hood, the Viterbi algorithm must produce a probability and a lso the most likely state
sequence. We computethis best state sequenceby keepingtra ck of the path of hidden
states that led to each state, as suggested in Fig. 6.12,and t hen at the end tracing back
thebest pathto thebeginning(theViterbi backtrace ). Viterbibacktrace

Finally,wecangiveaformaldeﬁnitionoftheViterbirecurs ionasfollows:
1.Initialization:
v1(j) =a0jbj(o1)1≤j≤N (6.21)
bt1(j) =0 (6.22)
2.Recursion (recallstates0and qFarenon-emitting):
i=1vt−1(i)aijbj(ot); 1≤j≤N,1<t≤T (6.23)
btt(j) =Nargmax
i=1vt−1(i)aijbj(ot); 1≤j≤N,1<t≤T (6.24)
3.Termination:
Thebestscore: P∗=vt(qF) =Nmax
i=1vT(i)∗ai,F (6.25)
Thestart ofbacktrace: qT∗=btT(qF) =Nargmax
i=1vT(i)∗ai,F (6.26)

DRAFTSection6.5. TrainingHMMs: TheForward-BackwardAlgorith m 187
startH
CH
CH
Cend
.7 * .2
.3 * .5
q2
q1
q0
o13 3 1
o2 o3
Figure6.12 The Viterbi backtrace. As we extend each path to a new state ac count for the next observation, we
keep a backpointer (shown withbroken blue lines) tothe best paththat ledus tothis state.
6.5 TrainingHMMs: TheForward-BackwardAlgorithm
We turnto the thirdproblemforHMMs: learningthe parameter sof an HMM,i.e., the
AandBmatrices. Formally,
Learning: Givenanobservationsequence Oandthesetofpossiblestates
intheHMM,learntheHMMparameters AandB.
The input to such a learning algorithm would be an unlabeled s equence of obser-
vationsOand a vocabulary of potential hidden states Q. Thus for the ice cream task,

wewouldstartwithasequenceofobservations O={1,3,2,...,},andthesetofhidden
statesHandC. For the part-of-speechtaggingtask we would start with a se quenceof
observations O={w1,w2,w3...}and a set of hiddenstates NN, NNS, VBD, IN,... and
so on.
The standard algorithm for HMM training is the forward-backward orBaum-Forward-backward
Welchalgorithm (Baum, 1972), a special case of the Expectation-Maximization or Baum-Welch
EMalgorithm(Dempster et al., 1977). The algorithm will let us train both the transi- EM

so on.
tionprobabilities Aandtheemissionprobabilities Bofthe HMM.

DRAFT188 Chapter 6. HiddenMarkovandMaximumEntropyModels
Let us begin by considering the much simpler case of training a Markov chain
rather than a Hidden MarkovModel. Since the states in a Marko vchain are observed,
wecanrunthemodelontheobservationsequenceanddirectly seewhichpathwetook
through the model, and which state generated each observati on symbol. A Markov
chainofcoursehasnoemissionprobabilities B(alternativelywecouldviewaMarkov
chain as a degenerate Hidden Markov Model where all the bprobabilities are 1.0 for
theobservedsymboland0forallothersymbols.). Thustheon lyprobabilitiesweneed
totrainarethe transitionprobabilitymatrix A.
We getthe maximumlikelihoodestimate ofthe probability aijof aparticulartran-
sition betweenstates iandjbycountingthe numberof timesthetransitionwas taken,
which we couldcall C(i→j), and then normalizingby the total countof all timeswe
tookanytransitionfromstate i:
aij=C(i→j)
∑q∈QC(i→q)(6.27)
WecandirectlycomputethisprobabilityinaMarkovchainbe causeweknowwhich

states we were in. For an HMM we cannot compute these counts di rectly from an
observationsequencesince we don’tknow whichpath of state s was taken throughthe
machineforagiveninput. TheBaum-Welchalgorithmusestwo neatintuitionstosolve
this problem. The ﬁrst idea is to iteratively estimate the counts. We will start with an
estimate for the transition and observation probabilities , and then use these estimated
probabilities to derive better and better probabilities. T he second idea is that we get

our estimated probabilities by computing the forward proba bility for an observation
and then dividing that probability mass among all the differ ent paths that contributed
tothisforwardprobability.
Inorderto understandthealgorithm,weneedtodeﬁneausefu lprobabilityrelated
tothe forwardprobability,calledthe backwardprobability .Backwardprobability
Thebackwardprobability βistheprobabilityofseeingtheobservationsfromtime
t+1totheend,giventhatweareinstate iattimet(andofcoursegiventheautomaton
λ):

λ):
βt(i) =P(ot+1,ot+2...oT|qt=i,λ) (6.28)
Itiscomputedinductivelyin asimilar mannerto theforward algorithm.
βT(i) =ai,F,1≤i≤N (6.29)
2.Recursion (againsincestates0 and qFare non-emitting):
βt(i) =N
∑
j=1aijbj(ot+1)βt+1(j),1≤i≤N,1≤t<T (6.30)
P(O|λ) =αT(qF) =β1(0) =N
∑
j=1a0jbj(o1)β1(j) (6.31)

DRAFTSection6.5. TrainingHMMs: TheForward-BackwardAlgorith m 189
Fig.6.13illustratesthe backwardinductionstep.
)
t+1otai1ai2aiN
ai3
b1(ot+1)
*
+(i)= 
,*t+1(j) aij  bj(ot+1) 
q1q2q3qN
q1qi
q2
q1q2
ot-1q3qNβt+1(N)
βt+1(3)
βt+1(2)
βt+1(1)b2(ot+1)b2(ot+1)b2(ot+1)
Figure6.13 Thecomputationof βt(i)bysummingallthesuccessive values βt+1(j)weighted
by their transition probabilities aijand their observation probabilities bj(ot+1). Start and end
states not shown.
We are now ready to understand how the forward and backward pr obabilities can
help us compute the transition probability aijand observation probability bi(ot)from
an observation sequence, even though the actual path taken t hrough the machine is
hidden.
Let’sbeginbyshowinghowtoestimate ˆ aijbyavariantof(6.27):
ˆaij=expectednumberoftransitionsfromstate itostatej
expectednumberoftransitionsfromstate i(6.32)
How do we compute the numerator? Here’s the intuition. Assum e we had some

hidden.
estimate of the probabilitythat a given transition i→jwas taken at a particular point
in timetin the observation sequence. If we knew this probability for each particular
timet,wecouldsumoveralltimes ttoestimatethetotalcountforthetransition i→j.
Moreformally,let’sdeﬁnetheprobability ξtastheprobabilityofbeinginstate iat
timetandstate jattimet+1,giventheobservationsequenceandofcoursethemodel:
ξt(i,j) =P(qt=i,qt+1=j|O,λ) (6.33)
In order to compute ξt, we ﬁrst compute a probability which is similar to ξt, but

differs in including the probability of the observation; no te the different conditioning
ofOfromEq.6.33:
not-quite- ξt(i,j) =P(qt=i,qt+1=j,O|λ) (6.34)
Fig. 6.14 shows the various probabilities that go into compu ting not-quite- ξt: the
transition probability for the arc in question, the αprobability before the arc, the β
probability after the arc, and the observation probability for the symbol just after the
arc. Thesefouraremultipliedtogethertoproduce not-quite- ξtasfollows:

DRAFT190 Chapter 6. HiddenMarkovandMaximumEntropyModels
ot+2ot+1αt(i)
ot-1otaijbj(ot+1) 
-
i
-./t+1(
Figure6.14 Computationofthejointprobabilityofbeinginstate iattimetandstate jattime
t+1. The ﬁgure shows the various probabilities that need to be c ombined to produce P(qt=
i,qt+1=j,O|λ): theαandβprobabilities, the transition probability aijand the observation
probability bj(ot+1). AfterRabiner (1989).
not-quite- ξt(i,j) =αt(i)aijbj(ot+1)βt+1(j) (6.35)
Inordertocompute ξtfromnot-quite- ξt,thelawsofprobabilityinstructustodivide
byP(O|λ),since:
P(X|Y,Z) =P(X,Y|Z)
P(Y|Z)(6.36)
The probability of the observation given the model is simply the forward proba-
bility of the whole utterance, (or alternatively the backwa rd probability of the whole
utterance!),whichcanthusbecomputedin anumberofways:
P(O|λ) =αT(N) =βT(1) =N
∑
j=1αt(j)βt(j) (6.37)
So,theﬁnalequationfor ξtis:
ξt(i,j) =αt(i)aijbj(ot+1)βt+1(j)
αT(N)(6.38)
Theexpectednumberof transitionsfromstate ito statejis thenthe sum overall t

∑
ofξ. For ourestimate of aijin (6.32),we just needonemorething: thetotal expected
numberoftransitionsfromstate i. We can getthisbysummingoverall transitionsout
ofstatei. Here’stheﬁnalformulafor ˆ aij:
ˆaij=∑T−1
t=1ξt(i,j)
∑T−1
t=1∑N
j=1ξt(i,j)(6.39)

DRAFTSection6.5. TrainingHMMs: TheForward-BackwardAlgorith m 191
We also need a formula for recomputing the observation proba bility. This is the
probability of a given symbol vkfrom the observation vocabulary V, given a state j:
ˆbj(vk). We will dothisbytryingto compute:
ˆbj(vk) =expectednumberoftimesin state jandobservingsymbol vk
expectednumberoftimesin state j(6.40)
Forthiswewillneedtoknowtheprobabilityofbeinginstate jattimet,whichwe
will callγt(j):
γt(j) =P(qt=j|O,λ) (6.41)
Once again, we will compute this by including the observatio n sequence in the
probability:
γt(j) =P(qt=j,O|λ)
P(O|λ)(6.42)
ot+1αt(
ot-1ot
0
12
3(
Figure6.15 The computation of γt(j), the probability of being in state jat timet. Note that
γis really a degenerate case of ξand hence this ﬁgure is like a version of Fig. 6.14 with state i
collapsed withstate j. AfterRabiner (1989).
AsFig.6.15shows,thenumeratorof(6.42)isjusttheproduc toftheforwardprob-
abilityandthebackwardprobability:
γt(j) =αt(j)βt(j)
P(O|λ)(6.43)

ot+1αt(
ot-1ot
0
12
3(
We are readyto compute b. For the numerator,we sum γt(j)for all time steps tin
whichtheobservation otisthesymbol vkthatweareinterestedin. Forthedenominator,
we sumγt(j)over all time steps t. The result will be the percentage of the times that
we were in state jand we saw symbol vk(the notation ∑T
t=1s.t.Ot=vkmeans “sum over
alltforwhichthe observationattime twasvk”):

DRAFT192 Chapter 6. HiddenMarkovandMaximumEntropyModels
ˆbj(vk) =∑T
t=1s.t.Ot=vkγt(j)
∑T
t=1γt(j)(6.44)
Wenowhavewaysin(6.39)and(6.44)to re-estimate thetransition Aandobserva-
tionBprobabilities from an observation sequence Oassuming that we already have a
previousestimateof AandB.
Thesere-estimationsformthecoreoftheiterativeforward -backwardalgorithm.
The forward-backward algorithm starts with some initial es timate of the HMM
parameters λ= (A,B). We then iteratively run two steps. Like other cases of the EM
(expectation-maximization)algorithm,theforward-back wardalgorithmhastwosteps:
theexpectation step,orE-step,andthemaximization step, orM-step. E-step
M-step In the E-step, we compute the expected state occupancy count γand the expected
state transitioncount ξ,fromtheearlier AandBprobabilities. IntheM-step,weuse γ
andξtorecomputenew AandBprobabilities.
function FORWARD-BACKWARD (observations of lenT,output vocabulary V ,hidden
state setQ )returnsHMM=(A,B)
initialize AandB
iterateuntil convergence
E-step

E-step
P(O|λ)∀tandj
αT(N)∀t,i,andj
M-step
ˆaij=T−1
∑
T−1
∑
t=1N
∑
j=1ξt(i,j)
ˆbj(vk) =T
∑
T
∑
t=1γt(j)
returnA,B
Figure6.16 The forward-backward algorithm.
Althoughinprincipletheforward-backwardalgorithmcand ocompletelyunsuper-
vised learning of the AandBparameters, in practice the initial conditions are very
important. For this reason the algorithm is often given extr a information. For exam-

E-step
M-step
ˆaij=T−1
∑
T−1
∑
t=1N
∑
ˆbj(vk) =T
∑
T
∑
t=1γt(j)
returnA,B
ple, for speech recognition, in practice the HMM structure i s very often set by hand,

DRAFTSection6.6. MaximumEntropyModels: Background 193
and only the emission ( B) and (non-zero) Atransition probabilities are trained from
a set of observation sequences O. Sec. 9.7 in Ch. 9 will also discuss how initial A
andBestimates are derived in speech recognition. We will also se e that for speech
thatthe forward-backwardalgorithmcanbeextendedtoinpu tswhicharenon-discrete
(“continuousobservationdensities”).
6.6 MaximumEntropyModels: Background
We turn now to a second probabilistic machine learning frame workcalled Maximum
Entropy modeling, MaxEntforshort. MaxEntismorewidelyknownas multinomial
logisticregression .
Our goal in this chapter is to introduce the use of MaxEnt for s equence classiﬁca-
tion. Recall that the task of sequence classiﬁcation or sequ ence labelling is to assign
a label to each element in some sequence, such as assigning a p art-of-speech tag to
a word. The most common MaxEnt sequence classiﬁer is the Maximum Entropy

Markov Model orMEMM, to be introduced in Sec. 6.8. But before we see this use
ofMaxEntasa sequenceclassiﬁer,we needtointroducenon-s equentialclassiﬁcation.
Thetaskofclassiﬁcationistotakeasingleobservation,ex tractsomeusefulfeatures
describingtheobservation,andthenbasedonthesefeature s,toclassifytheobservation
into one of a set of discrete classes. A probabilistic classiﬁer does slightly more than
this;inadditiontoassigningalabelorclass,itgivesthe probability oftheobservation

being in that class; indeed, for a given observation a probab ilistic classiﬁer gives a
probabilitydistributionoverall classes.
Suchnon-sequentialclassiﬁcationtasksoccurthroughout speechandlanguagepro-
cessing. For example, in text classiﬁcation we might need to decide whether a par-
ticular email should be classiﬁed as spam or not. In sentiment analysis we have to
determine whether a particular sentence or document expres ses a positive or negative
opinion. In many tasks, we’ll need to know where the sentence boundar ies are, and

so we’ll need to classify a period character (‘.’) as either a sentence boundary or not.
We’ll see moreexamplesofthe needforclassiﬁcationthroug houtthisbook.
MaxEntbelongstothefamilyofclassiﬁersknownasthe exponential orlog-linearLog-linear
classiﬁer
classiﬁers. MaxEntworksbyextractingsomesetoffeatures fromtheinput,combining
themlinearly(meaningthatwe multiplyeachbya weightandthenaddthemup ),and
then,forreasonswewill see below,usingthissum asan expon ent.
Let’s ﬂesh out this intuition just a bit more. Assume that we h ave some input x

classiﬁer
(perhapsit isawordthatneedstobetagged,oradocumenttha tneedstobeclassiﬁed)
fromwhichweextractsomefeatures. Afeaturefortaggingmi ghtbethiswordendsin
-ingorthepreviouswordwas‘the’ . Foreachsuchfeature fi,wehavesomeweight wi.
Given the features and weights, our goal is to choose a class ( for example a part-
of-speechtag) for the word. MaxEnt does this by choosingthe most probabletag; the
probabilityofa particularclass cgiventheobservation xis:

DRAFT194 Chapter 6. HiddenMarkovandMaximumEntropyModels
p(c|x) =1
Zexp(∑
iwifi) (6.45)
HereZis a normalizing factor, used to make the probabilities corr ectly sum to 1;
and as usual exp (x) =ex. As we’ll see later, this is a simpliﬁed equation in various
ways; for example in the actual MaxEnt model the features fand weights ware both
dependent on the class c(i.e., we’ll have different features and weights for differ ent
classes).
In order to explain the details of the MaxEnt classiﬁer, incl uding the deﬁnition
of the normalizing term Zand the intuition of the exponential function, we’ll need
to understand ﬁrst linear regression , which lays the groundworkfor prediction using
features,and logisticregression ,whichisourintroductionto exponentialmodels. We
cover these areas in the next two sections. Readers who have h ad a grounding in
these kinds of regression may want to skip the next two sectio ns. Then in Sec. 6.7
we introduce the details of the MaxEnt classiﬁer. Finally in Sec. 6.8 we show how

the MaxEnt classiﬁer is used for sequence classiﬁcation in t heMaximum Entropy
MarkovModel orMEMM.
6.6.1 Linear Regression
In statistics we use two different names for tasks that map so me input features into
some output value: we use the word regression when the output is real-valued, and
classiﬁcation whenthe outputisoneofa discreteset ofclasses.
You may already be familiar with linear regression from a sta tistics class. The
idea is that we are given a set of observations, each observat ionassociated with some

features,andwe wanttopredictsomereal-valuedoutcomefo reachobservation. Let’s
seeanexamplefromthedomainofpredictinghousingprices. LevittandDubner(2005)
showedthatthewordsusedinarealestateadcanbeusedasago odpredictorofwhether
a housewill sell formoreorlessthanitsaskingprice. Theys howed,forexample,that
houseswhoserealestateadshadwordslike fantastic,cute,orcharming ,tendedtosell
for lower prices, while houses whose ads had words like mapleandgranitetended to
sell for higher prices. Their hypothesiswas that real estat e agentsused vague positive

wordslike fantastictomaskthelackofanyspeciﬁcpositivequalitiesinthehous e. Just
forpedagogicalpurposes,we createdthefakedatain Fig.6. 17.
# ofvagueadjectives Amounthouse sold overaskingprice
4 0
3 $1000
2 $1500
2 $6000
1 $14000
0 $18000
Figure6.17 Somemade-updataonthenumberofvagueadjectives( fantastic,cute,charming )
ina realestate ad, and the amount the house sold forover the a skingprice.

DRAFTSection6.6. MaximumEntropyModels: Background 195
y = -4900x + 16550
-500005000100001500020000
0 1 2 3 4 5
Number of AdjectivesIncrease in House Sale Price
Figure6.18 A plot of the (made-up) points in Fig. 6.17 and the regression line that best ﬁts
them, withthe equation y=−4900x+16550.
Fig. 6.18 shows a graph of these points, with the feature (# of adjectives) on the
x-axis, and the price on the y-axis. We have also plotted a regression line , which is Regressionline
the line that best ﬁts the observed data. The equation of any l ine isy=mx+b; as we
show on the graph, the slope of this line is m=−4900, while the intercept is 16550.
We can think of these two parametersof this line (slope mand intercept b) as a set of
weightsthat we use to mapfromourfeatures(in thiscase x, numbersof adjectives)to
ouroutputvalue y(inthiscase price). We can representthislinearfunctionu singwto
referto weightsasfollows:
price=w0+w1∗NumAdjectives (6.46)
ThusEq.6.46givesusa linearfunctionthat letsusestimate thesalespriceforany

number of these adjectives. For example, how much would we ex pect a house whose
adhas5adjectivesto sell for?
The true power of linear models comes when we use more than one feature (tech-
nically we call this multiple linear regression ). For example, the ﬁnal house price
probably depends on many factors such as the average mortgag e rate that month, the
numberofunsoldhousesonthemarket,andmanyothersuchfac tors. Wecouldencode
each of these as a variable, and the importance of each factor would be the weight on
thatvariable,asfollows:

price=w0+w1∗NumAdjectives +w2∗MortgageRate +w3∗NumUnsoldHouses (6.47)
In speech and language processing, we often call each of thes e predictive factors
like the number of adjectives or the mortgage rate a feature. We represent each ob- Feature
servation (each house for sale) by a vector of these features . Suppose a house has 1
adjectivein itsad,andthe mortgageratewas6.5andtherewe re10,000unsoldhouses
inthecity. Thefeaturevectorforthe housewouldbe ⃗f= (1,6.5,10000 ). Supposethe

weight vector that we had previouslylearned for this task wa s⃗w= (w0,w1,w2,w3) =
(18000 ,−5000,−3000,−1.8). Thenthe predictedvalueforthishousewouldbe com-
putedbymultiplyingeachfeaturebyitsweight:

DRAFT196 Chapter 6. HiddenMarkovandMaximumEntropyModels
price=w0+N
∑
i=1wi×fi (6.48)
Ingeneralwewillpretendthatthereisanextrafeature f0whichhasthevalue1,an
interceptfeature ,whichmaketheequationssimplerwithregardtothatpesky w0,and
so ingeneralwe canrepresentalinearregressionforestima tingthevalueof yas:
linearregression: y=N
∑
i=0wi×fi (6.49)
Takingtwovectorsandcreatingascalarbymultiplyingeach elementinapairwise
fashionandsummingtheresultsis called the dot product . Recall thatthe dot product Dot product
a·bbetweentwo vectors aandbisdeﬁnedas:
dotproduct: a·b=N
∑
i=1aibi=a1b1+a2b2+···+anbn (6.50)
Thus Eq. 6.49 is equivalent to the dot product between the wei ghts vector and the
featurevector:
y=w·f (6.51)
Vector dot products occur very frequently in speech and lang uage processing; we
will oftenrelyonthedotproductnotationto avoidthemessy summationsigns.
Learningin linearregression
How do we learn the weights for linear regression? Intuitive ly we’d like to choose

∑
weights that make the estimated values yas close as possible to the actual values that
we sawin thetrainingset.
Consider a particular instance x(j)from the training set (we’ll use superscripts in
parenthesestorepresenttraininginstances),whichhasan observedlabelinthetraining
sety(j)
obs. Ourlinearregressionmodelpredictsavaluefor y(j)asfollows:
y(j)
pred=N
∑
i=0wi×f(j)
i(6.52)
We’d like to choose the whole set of weights Wso as to minimize the difference
between the predicted value y(j)
predand the observed value y(j)
obs, and we want this dif-

sety(j)
y(j)
pred=N
∑
i(6.52)
ference minimized over all the Mexamples in our training set. Actually we want to
minimizetheabsolutevalueof thedifference(sincewe don’ twant a negativedistance
inoneexampletocanceloutapositivedifferenceinanother example),soforsimplicity
(and differentiability) we minimize the square of the diffe rence. Thus the total value
we want to minimize, which we call the sum-squared error , is this cost function ofSum-squarederror
thecurrentset ofweights W:
cost(W) =M
∑
j=0(
y(j)
pred−y(j)
obs)2
(6.53)

DRAFTSection6.6. MaximumEntropyModels: Background 197
We won’t give here the details of choosingthe optimal set of w eights to minimize
the sum-squared error. But, brieﬂy, it turns out that if we pu t the entire training set
into a single matrix Xwith each row in the matrix consisting of the vector of featur es
associatedwitheachobservation x(i),andputalltheobserved yvaluesinavector ⃗y,that
there is a closed-form formula for the optimal weight values Wwhich will minimize
cost(W):
W= (XTX)−1XT⃗y (6.54)
Implementations of this equation are widely available in st atistical packages like
SPSS orR.
6.6.2 Logisticregression
Linearregressionis what we want whenwe are predictinga rea l-valuedoutcome. But
somewhat more commonly in speech and language processing we are doing classiﬁ-
cation, in which the output ywe are trying to predict takes on one from a small set of
discretevalues.
Consider the simplest case of binary classiﬁcation, where w e want to classify

cost(W):
SPSS orR.
whether some observation xis in the class (true) or not in the class (false). In other
wordsycan only take on the values1 (true) or 0 (false), and we’d like a classiﬁer that
can take features of xand return true or false. Furthermore, instead of just retur ning
the 0 or 1 value, we’d like a model that can give us the probability that a particular
observationisinclass0or1. Thisisimportantbecauseinmo streal-worldtaskswe’re
passing the results of this classiﬁer onto some further clas siﬁer to accomplish some

task. Since we are rarely completelycertain about which cla ss an observationfalls in,
we’d prefer not to make a hard decision at this stage, ruling o ut all other classes. In-
stead, we’d like to pass on to the later classiﬁer as much info rmation as possible: the
entireset ofclasses, withtheprobabilityvaluethatwe ass igntoeachclass.
Couldwemodifyourlinearregressionmodeltouseit forthis kindofprobabilistic
classiﬁcation? Suppose we just tried to train a linear model to predict a probabilityas
follows:

follows:
P(y=true|x) =N
∑
i=0wi×fi (6.55)
=w·f (6.56)
Wecouldtrainsuchamodelbyassigningeachtrainingobserv ationthetargetvalue
y=1 if it was in the class (true) and the target value y=0 if it was not (false). Each
observation xwouldhaveafeaturevector f,andwewouldtraintheweightvector wto
minimize the predictive error from 1 (for observationsin th e class) or 0 (for observa-
tionsnotintheclass). Aftertraining,wewouldcomputethe probabilityofaclassgiven

follows:
∑
an observationbyjust takingthe dotproductofthe weightve ctorwith the featuresfor
thatobservation.
The problem with this model is that there is nothing to force t he output to be a
legal probability,i.e. to lie between zero and 1. The expres sion∑N
i=0wi×fiproduces

DRAFT198 Chapter 6. HiddenMarkovandMaximumEntropyModels
valuesfrom−∞to∞. How can we ﬁx this problem? Suppose that we keep our linear
predictor w·f,butinsteadofhavingitpredictaprobability,wehaveitpr edictaratioof
two probabilities. Speciﬁcally,supposewe predictthe rat ioofthe probabilityof being
in the class to the probabilityof not beingin the class. This ratio is called the odds. If Odds
an event has probability .75 of occurringand probability .2 5 of not occurring, we say
theoddsofoccurringis .75/.25=3. Wecouldusethelinearmodeltopredicttheodds
ofybeingtrue:
p(y=true)|x
1−p(y=true|x)=w·f (6.57)
This last model is close: a ratio of probabilities can lie bet ween 0 and ∞. But we
need the left-handside of the equation to lie between −∞and∞. We can achieve this
bytakingthenaturallogofthisprobability:
ln(p(y=true|x)
1−p(y=true|x))
=w·f (6.58)
Now both the left and right hand lie between −∞and∞. This function on the left
(thelogoftheodds)isknownasthe logitfunction : Logit function
logit(p(x)) =ln(p(x)
1−p(x))

1−p(x))
(6.59)
The model of regression in which we use a linear function to es timate, not the
probability, but the logit of the probability, is known as logistic regression . If theLogisticregression
linear functionis estimating the logit, what is the actual f ormulain logistic regression
for the probability P(y=true)? You should stop here and take Eq. 6.58 and apply
somesimplealgebrato solvefortheprobability P(y=true).
Hopefully when you solved for P(y=true)you came up with a derivation some-
thinglike thefollowing:
=w·f

=w·f
p(y=true|x)
1−p(y=true|x)=ew·f(6.60)
p(y=true|x) = (1−p(y=true|x))ew·f
p(y=true|x) =ew·f−p(y=true|x)ew·f
p(y=true|x)+p(y=true|x)ew·f=ew·f
p(y=true|x)(1+ew·f) =ew·f
p(y=true|x) =ew·f
1+ew·f(6.61)
Oncewehavethisprobability,wecaneasilystatetheprobab ilityoftheobservation
notbelongingtotheclass, p(y=false|x), asthetwomust sumto1:

DRAFTSection6.6. MaximumEntropyModels: Background 199
p(y=false|x) =1
1+ew·f(6.62)
Herearetheequationsagainusingexplicitsummationnotat ion:
p(y=true|x) =exp(∑N
i=0wifi)
1+exp(∑N
i=0wifi)(6.63)
1+exp(∑N
i=0wifi)(6.64)
Wecanexpresstheprobability P(y=true|x)inaslightlydifferentway,bydividing
thenumeratoranddenominatorin (6.61)by e−w·f:
1+ew·f(6.65)
=1
1+e−w·f(6.66)
These last equationis now in the form of what is called the logistic function , (the Logistic function
functionthat giveslogistic regressionits name). Thegene ralformofthelogisticfunc-
tionis:
1
1+e−x(6.67)
Thelogistic functionmapsvaluesfrom −∞and∞tolie between0and1
Again,wecanexpress P(y=false|x)soasto makethe probabilitiessum toone:
p(y=false|x) =e−w·f
1+e−w·f(6.68)
6.6.3 Logisticregression: Classiﬁcation
Given a particular observation, how do we decide which of the two classes (‘true’ or
‘false’) it belongsto? This is the task of classiﬁcation , also called inference . Clearly Classiﬁcation

tionis:
1
Inference thecorrectclassistheonewiththehigherprobability. Thu swecansafelysaythatour
observationshouldbelabeled‘true’if:
p(y=true|x)>p(y=false|x)
p(y=false|x)>1
1−p(y=true|x)>1
andsubstitutingfromEq.6.60forthe oddsratio:
ew·f>1
w·f>0 (6.69)

DRAFT200 Chapter 6. HiddenMarkovandMaximumEntropyModels
orwiththeexplicitsumnotation:
N
∑
i=0wifi>0 (6.70)
Thus in order to decide if an observation is a member of the cla ss we just need to
compute the linear function, and see if its value is positive ; if so, the observationis in
theclass.
Amoreadvancedpoint: theequation ∑N
i=0wifi=0istheequationofa hyperplane
(ageneralizationofalineto Ndimensions). Theequation ∑N
i=0wifi>0isthusthepart
ofN-dimensionalspaceabovethishyperplane. Thuswecanseeth elogisticregression
functionaslearningahyperplanewhichseparatespointsin spacewhichareintheclass
(’true’)frompointswhichare notin theclass.
6.6.4 Advanced: Learning inlogisticregression
Inlinearregression,learningconsistedofchoosingthewe ightswwhichminimizedthe
sum-squarederroron the training set. In logistic regressi on, by contrast, we generally

useconditionalmaximum likelihood estimation . What thismeansisthat we chooseConditionalmaximumlikelihoodestimationtheparameters wthatmaketheprobabilityoftheobserved yvaluesinthetrainingdata
to be the highest, given the observations x. In other words, for an individual training
observation x, wewant tochoosetheweightsasfollows:
ˆw=argmax
wP(y(i)|x(i)) (6.71)
Andwe’dliketo choosetheoptimalweightsfortheentiretra iningset:
ˆw=argmax
w∏
iP(y(i)|x(i)) (6.72)
We generallyworkwiththe loglikelihood:
ˆw=argmax
w∑
ilogP(y(i)|x(i)) (6.73)

ˆw=argmax
ˆw=argmax
w∏
ˆw=argmax
w∑
So,moreexplicitly:
ˆw=argmax
w∑
ilog{P(y(i)=1|x(i)))fory(i)=1
P(y(i)=0|x(i)))fory(i)=0(6.74)
This equation is unwieldy, and so we usually apply a convenie nt representational
trick. Note that if y=0 the ﬁrst term goes away, while if y=1 the second term goes
away:
ˆw=argmax
w∑
iy(i)logP(y(i)=1|x(i)))+(1−y(i))logP(y(i)=0|x(i)) (6.75)
Nowifwe substitutein(6.66)and(6.68),we get:

DRAFTSection6.7. MaximumEntropyModeling 201
ˆw=argmax
w∑
iy(i)log1
1+e−w·f+(1−y(i))loge−w·f
1+e−w·f(6.76)
Findingtheweightswhichresultinthemaximumlog-likelih oodaccordingto(6.76)
is a problem in the ﬁeld known as convex optimization . Among the most com-Convexoptimization
monly used algorithms are quasi-Newton methods like L-BFGS, as well as gradient
ascent, conjugate gradient, and various iterative scaling algorithms (Darroch and Rat-
cliff, 1972; Della Pietra et al., 1997; Malouf, 2002). These learning algorithms are
available in the various MaxEnt modeling toolkits but are to o complex to deﬁne here;
interested readers should see the machine learning textboo ks suggested at the end of
thechapter.
6.7 MaximumEntropyModeling
We showed above how logistic regression can be used to classi fy an observation into
one of two classes. But most of the time the kinds of classiﬁca tion problems that
come up in language processing involve larger numbers of cla sses (such as the set

of part-of-speech classes). Logistic regression can also b e deﬁned for such functions
with many discrete values. In such cases it is called multinomial logistic regression .Multinomiallogistic regression
As we mentioned above, multinomial logistic regression is c alledMaxEnt in speech MaxEnt
and language processing (see Sec. 6.7.1 on the intuition beh ind the name ‘maximum
entropy’).
The equations for computing the class probabilities for a Ma xEnt classiﬁer are a
generalizationofEqs.6.63-6.64above. Let’sassumethatt hetargetvalue yisarandom

variablewhichcantakeon Cdifferentvaluescorrespondingtotheclasses c1,c2,...,cC.
We said earlier in this chapter that in a MaxEnt model we estim ate the probability
thatyisa particularclass cas:
p(c|x) =1
Zexp∑
iwifi (6.77)
Let’s now add some details to this schematic equation. First we’ll ﬂesh out the
normalizationfactorZ,specifythenumberoffeaturesas N, andmakethevalueofthe
weightdependentonthe class c. Theﬁnalequationis:
p(c|x) =exp(
N
∑
i=0wcifi)
∑
c′∈Cexp(
N
∑
i=0wc′ifi) (6.78)
Notethatthenormalizationfactor Zisjustusedtomaketheexponentialintoatrue

p(c|x) =1
Zexp∑
N
∑
i=0wcifi)
∑
c′∈Cexp(
N
∑
probability;

DRAFT202 Chapter 6. HiddenMarkovandMaximumEntropyModels
Z=∑
Cp(c|x) =∑
c′∈Cexp(
N
∑
i=0wc′ifi)
(6.79)
We need to make one morechangeto see the ﬁnal MaxEntequation . So far we’ve
been assuming that the features fiare real-valued. It is more common in speech and
languageprocessing,however,to usebinary-valuedfeatur es. A featurethat onlytakes
on the values 0 and 1 is also called an indicator function . In general,the featureswe Indicatorfunction
use are indicator functions of some property of the observat ion and the class we are
considering assigning. Thus in MaxEnt, instead of the notat ionfi, we will often use
thenotation fi(c,x), meaninga feature ifora particularclass cfora givenobservation
x.
The ﬁnal equation for computing the probability of ybeing of class cgivenxin
MaxEntis:
N
∑
i=0wcifi(c,x))
∑
c′∈Cexp(
N
∑
i=0wc′ifi(c′,x)) (6.80)
To get a clearer intuition of this use of binary features, let ’s look at some sample

x.
MaxEntis:
N
∑
∑
c′∈Cexp(
N
∑
features for the task of part-of-speech tagging. Suppose we are assigning a part-of-
speechtagtotheword racein(6.81),repeatedfrom(5.36):
(6.81)Secretariat/NNPis/BEZ expected/VBNto/TO race/??tomorrow/
Again, for now we’re just doing classiﬁcation, not sequence classiﬁcation, so let’s
consider just this single word. We’ll discuss in Sec. 6.8 how to perform tagging for a
wholesequenceofwords.
We would like to know whether to assign the class VBtorace(or instead assign

someotherclasslike NN).Oneusefulfeature,we’llcallit f1,wouldbethefactthatthe
currentwordis race. We canthusadda binaryfeaturewhichistrueifthisistheca se:
f1(c,x) ={
1 ifwordi=“race” & c=NN
0 otherwise
Anotherfeaturewouldbewhetherthe previouswordhastheta gTO:
f2(c,x) ={
1 ifti−1=TO &c=VB
Twomorepart-of-speechtaggingfeaturesmightfocusonasp ectsofaword’sspelling
andcase:
f3(c,x) ={1 if sufﬁx (wordi) =“ing” & c=VBG

DRAFTSection6.7. MaximumEntropyModeling 203
f4(c,x) ={1 if islowercase(wordi)&c=VB
Sinceeachfeatureisdependentonbotha propertyoftheobse rvationandtheclass
being labeled, we would need to have separate feature for, e. g, the link between race
andVB,orthe linkbetweena previousTOandNN:
f5(c,x) ={1 ifwordi=”race” & c=VB
f6(c,x) ={1 ifti−1=TO &c=NN
Eachofthesefeatureshasacorrespondingweight. Thusthew eightw1(c,x)would
indicate how strong a cue the word raceis for the tag VB, the weight w2(c,x)would
indicate how stronga cue the previoustag TOis for the currentword beinga VB, and
so on.
f1 f2 f3 f4 f5 f6
VB f 0 1 0 1 1 0
VB w .8 .01 .1
NN f 1 0 0 0 0 1
NN w .8 -1.3
Figure6.19 Some sample feature values and weights for tagging theword racein(6.81).
Let’sassume that the featureweightsfor the two classes VBa nd VN are as shown
inFig.6.19. Let’scallthecurrentinputobservation(wher ethecurrentwordis race)x.
We cannowcompute P(NN|x)andP(VB|x), usingEq.6.80:
P(NN|x) =e.8e−1.3

so on.
VB f 0 1 0 1 1 0
NN f 1 0 0 0 0 1
NN w .8 -1.3
e.8e−1.3+e.8e.01e.1=.20 (6.82)
P(VB|x) =e.8e.01e.1
e.8e−1.3+e.8e.01e.1=.80 (6.83)
NoticethatwhenweuseMaxEnttoperform classiﬁcation ,MaxEntnaturallygives
usaprobabilitydistributionovertheclasses. Ifwewantto doahard-classiﬁcationand
choose the single-best class, we can choose the class that ha s the highest probability,
i.e.:
ˆc=argmax
c∈CP(c|x) (6.84)
Classiﬁcation in MaxEnt is thus a generalization of classiﬁ cation in (boolean) lo-
gistic regression. In boolean logistic regression, classi ﬁcation involves building one

DRAFT204 Chapter 6. HiddenMarkovandMaximumEntropyModels
linear expression which separates the observations in the c lass from the observations
not in the class. Classiﬁcation in MaxEnt, by contrast, invo lves building a separate
linearexpressionforeachof Cclasses.
But as we’ll see later in Sec. 6.8, we generally don’t use MaxE nt for hard classi-
ﬁcation. Usually we want to use MaxEnt as part of sequence cla ssiﬁcation, where we
want not the best single class for one unit, but the best total sequence. For this task,
it’s useful to exploitthe entire probabilitydistribution for each individualunit, to help
ﬁnd the best sequence. Indeed even in many non-sequence appl ications a probability
distributionovertheclassesismoreusefulthana hardchoi ce.
Thefeatureswehavedescribedsofarexpressa singlebinary propertyofanobser-
vation. Butitisoftenusefultocreatemorecomplexfeature sthatexpresscombinations
of propertiesof a word. Some kinds of machine learning model s, like Support Vector

Machines(SVMs),canautomaticallymodeltheinteractions betweenprimitiveproper-
ties,butinMaxEntanykindofcomplexfeaturehastobedeﬁne dbyhand. Forexample
a word starting with a capital letter (like the word Day) is more likely to be a proper
noun(NNP)thanacommonnoun(forexampleintheexpression UnitedNationsDay ).
But a wordwhichis capitalizedbutwhichoccursat the beginn ingofthe sentence(the
previous word is <s>), as inDay after day.... , is not more likely to be a proper noun.
Even if each of these properties were already a primitive fea ture, MaxEnt would not

model their combination, so this boolean combination of pro perties would need to be
encodedasa featurebyhand:
f125(c,x) ={1 ifwordi−1=<s>& isupperﬁrst (wordi)&c=NNP
A key to successful use of MaxEnt is thus the design of appropr iate features and
featurecombinations.
LearningMaximum EntropyModels
Learning a MaxEnt model can be done via a generalization of th e logistic regression
learning algorithms described in Sec. 6.6.4; as we saw in (6. 73), we want to ﬁnd the

parameters wwhichmaximizetheloglikelihoodofthe Mtrainingsamples:
ˆw=argmax
w∑
ilogP(y(i)|x(i)) (6.85)
As with binary logistic regression, we use some convexoptim ization algorithm to
ﬁndtheweightswhichmaximizethisfunction.
Abriefnote: oneimportantaspectofMaxEnttrainingisakin dofsmoothingofthe
weightscalled regularization . The goalof regularizationis to penalizelargeweights; Regularization
it turns out that otherwise a MaxEnt model will learn very hig h weights which overﬁt

ˆw=argmax
w∑
the training data. Regularization is implemented in traini ng by changing the likeli-
hoodfunctionthat is optimized. Instead ofthe optimizatio nin (6.85),we optimizethe
following:
ˆw=argmax
w∑
ilogP(y(i)|x(i))−αR(w) (6.86)

DRAFTSection6.7. MaximumEntropyModeling 205
whereR(w)is aregularization term used to penalize large weights. It is common to
maketheregularizationterm R(w)bea quadraticfunctionoftheweightvalues:
R(W) =N
∑
j=1w2
j (6.87)
Subtractingsquaresoftheweightswill thusresultin prefe rringsmallerweights:
ˆw=argmax
w∑
ilogP(y(i)|x(i))−αN
∑
j=1w2
j (6.88)
It turns that this kind of regularization corresponds to ass uming that weights are
distributed according to a Gaussian distribution with mean µ=0. In a Gaussian or
normaldistribution,thefurtherawayavalueisfromthemea n,theloweritsprobability
(scaled by the variance σ). By using a Gaussian prior on the weights, we are saying
thatweightspreferto havethevaluezero. A Gaussianforawe ightwjis:
1√
2πσ2
jexp(
−(wj−µj)2
2σ2
j)
(6.89)
If we multiply each weight by a Gaussian prior on the weight, w e are thus maxi-
mizingthe followingconstraint:
ˆw=argmax
wM
∏
iP(y(i)|x(i))×N
∏
j=11√
2πσ2
jexp(
−(wj−µj)2
2σ2
j)
(6.90)
whichinlogspace,with µ=0,correspondsto
ˆw=argmax
w∑

1√
2πσ2
jexp(
−(wj−µj)2
2σ2
j)
(6.89)
ˆw=argmax
wM
∏
∏
j=11√
2πσ2
jexp(
−(wj−µj)2
2σ2
j)
(6.90)
ˆw=argmax
w∑
ilogP(y(i)|x(i))−N
∑
j=1w2
j
2σ2
j(6.91)
whichisinthesame formasEq.6.88.
There is a vast literature on the details of learning in MaxEn t; see the end of the
chapterforpointerstofurtherdetails.
6.7.1 Whydo wecall itMaximum Entropy?
Why do we refer to multinomial logistic regression models as MaxEnt or Maximum
Entropy models? Let’s give the intuition of this interpreta tion in the context of part-
of-speech tagging. Suppose we want to assign a tag to the word zzﬁsh(a word we

ˆw=argmax
w∑
∑
j=1w2
j
2σ2
j(6.91)
made up for this example). What is the probabilistic tagging model (the distribution
of part-of-speechtags across words) that makes the fewest a ssumptions, imposing no
constraintsat all? Intuitivelyit wouldbetheequiprobabl edistribution:
NNJJNNSVBNNPINMDUHSYMVBGPOSPRPCCCD...
1
451
451
451
451
451
451
451
451
451
451
451
451
451
45...

DRAFT206 Chapter 6. HiddenMarkovandMaximumEntropyModels
Nowsupposewehadsometrainingdatalabeledwithpart-of-s peechtags,andfrom
this data we learned only one fact: the set of possible tags fo rzzﬁshare NN, JJ, NNS,
and VB (so zzﬁshis a word something like ﬁsh, but which can also be an adjective).
What is the tagging model which relies on this constraint, bu t makes no further as-
sumptionsat all? Since oneofthese mustbethecorrecttag,w e knowthat
P(NN)+P(JJ)+P(NNS)+P(VB) =1 (6.92)
Sincewehavenofurtherinformation,amodelwhichmakesnof urtherassumptions
beyondwhatwe knowwouldsimplyassignequalprobabilityto eachofthese words:
1
41
41
41
40000000000...
Intheﬁrstexample,wherewewantedanuninformeddistribut ionover45parts-of-
speech, and in this case, where we wanted an uninformeddistr ibutionover 4 parts-of-
speech, it turns out that of all possible distributions, the equiprobabledistribution has
themaximum entropy . Recall fromSec. 4.10that the entropyofthe distributiono fa

1
41
41
41
randomvariable xiscomputedas:
H(x) =−∑
xP(x)log2P(x) (6.93)
An equiprobable distribution in which all values of the rand om variable have the
same probability has a higher entropy than one in which there is more information.
Thus of all distributions over four variables the distribut ion{1
4,1
4,1
4,1
4}has the maxi-
mumentropy. (To havean intuition forthis, use Eq. 6.93to co mputethe entropyfora
few other distributionssuch as the distribution {1
4,1
2,1
8,1
8}, and make sure they are all
lowerthanthe equiprobabledistribution.)

H(x) =−∑
4,1
4,1
4,1
4,1
2,1
8,1
The intuition of MaxEnt modeling is that the probabilistic m odel we are building
should follow whatever constraints we impose on it, but beyo nd these constraints it
shouldfollowOccam’sRazor,i.e.,makethefewest possible assumptions.
Let’s add some more constraints into our tagging example. Su ppose we looked at
our tagged training data and noticed that 8 times out of 10, zzﬁshwas tagged as some
sortofcommonnoun,eitherNNorNNS.Wecanthinkofthisassp ecifyingthefeature

’wordiszzﬁshandti=NNorti=NNS’.Wemightnowwanttomodifyourdistribution
so thatwegive8
10ofourprobabilitymassto nouns,i.e. nowwe have2 constrain ts
P(NN)+P(JJ)+P(NNS)+P(VB) =1
P(wordiszzﬁshandti=NN orti=NNS) =8
10
but make no further assumptions (keep JJ and VB equiprobable , and NN and NNS
equiprobable).
NNJJNNSVBNNP...
4
101
104
101
100...

DRAFTSection6.8. MaximumEntropyMarkovModels 207
Now suppose we don’t have any more information about zzﬁsh. But we notice in
thetrainingdatathatforallEnglishwords(notjust zzﬁsh)verbs(VB)occuras1word
in20. We cannowaddthisconstraint(correspondingtothefe atureti=VB):
P(wordiszzﬁshandti=NNorti=NNS) =8
10
P(VB) =1
20
Theresultingmaximumentropydistributionisnowasfollow s:
NNJJNNSVB
4
103
204
101
20
Insummary,theintuitionofmaximumentropyistobuildadis tributionbycontinu-
ouslyaddingfeatures. Eachfeatureisanindicatorfunctio n,whichpicksoutasubsetof
thetrainingobservations. Foreachfeatureweaddaconstra intonourtotaldistribution,
specifying that our distribution for this subset should mat ch the empirical distribution
we saw in ourtrainingdata. We then choosethe maximumentrop ydistributionwhich
otherwise accords with these constraints. Berger et al. (19 96) pose the optimization
problemofﬁndingthisdistributionasfollows:
“Toselectamodelfromaset Cofallowedprobabilitydistributions,choose

themodel p∗∈Cwith maximumentropyH (p)”:
p∗=argmax
p∈CH(p) (6.94)
Now we come to the important conclusion. Berger et al. (1996) show that the
solutiontothisoptimizationproblemturnsouttobeexactl ytheprobabilitydistribution
of a multinomial logistic regression model whose weights Wmaximize the likelihood
of the training data! Thus the exponential model for multino mial logistic regression,
when trained accordingto the maximum likelihood criterion , also ﬁnds the maximum
entropydistributionsubjectto theconstraintsfromthefe aturefunctions.
6.8 MaximumEntropyMarkovModels

We began our discussion of MaxEnt by pointing out that the bas ic MaxEnt model is
notinitselfaclassiﬁerforsequences. Instead,itisusedt oclassifyasingleobservation
intooneofasetofdiscreteclasses,asintextclassiﬁcatio n(choosingbetweenpossible
authorsof an anonymoustext, or classifying an email as spam ), or tasks like deciding
whethera periodmarksthe endofa sentence.
We turn in this section to the Maximum Entropy Markov Model orMEMM,

which is an augmentation of the basic MaxEnt classiﬁer so tha t it can be applied to

DRAFT208 Chapter 6. HiddenMarkovandMaximumEntropyModels
assign a class to each element in a sequence, just as we do with HMMs. Why would
we want a sequence classiﬁer built on MaxEnt? How might such a classiﬁer be better
thananHMM?
ConsidertheHMM approachto part-of-speechtagging. TheHM M taggingmodel
is based on probabilities of the form P(tag|tag)andP(word|tag). That means that if
we want to include some source of knowledge into the tagging p rocess, we must ﬁnd
a way to encode the knowledge into one of these two probabilit ies. But many knowl-
edge sources are hard to ﬁt into these models. For example, we saw in Sec. 5.8.2
that for tagging unknown words, useful features include cap italization, the presence
of hyphens, word endings, and so on. There is no easy way to ﬁt p robabilities like
P(capitalization|tag),P(hyphen|tag),P(sufﬁx|tag),andsoonintoanHMM-stylemodel.
Wegavetheinitialpartofthisintuitionintheprevioussec tion,whenwediscussed

applying MaxEnt to part-of-speech tagging. Part-of-speec h tagging is deﬁnitely a se-
quence labeling task, but we only discussed assigning a part -of-speechtag to a single
word.
How can we take this single local classiﬁer and turn it into a g eneral sequence
classiﬁer? Whenclassifyingeachwordwecanrelyonfeature sfromthe currentword,
features from surroundingwords, as well as the output of the classiﬁer from previous
words. Forexamplethe simplest methodis to runourlocal cla ssiﬁer left-to-right,ﬁrst

word.
making a hard classiﬁcation of the ﬁrst word in the sentence, then the second word,
andsoon. Whenclassifyingeachword,wecanrelyontheoutpu toftheclassiﬁerfrom
the previous word as a feature. For example, we saw in tagging the word racethat a
usefulfeaturewasthetagofthepreviousword;apreviousTO isagoodindicationthat
raceis a VB, whereas a previous DT is a good indication that raceis a NN. Such a
strict left-to-rightsliding windowapproachhas beenshow n to yield surprisinglygood
resultsacrossawide rangeofapplications.

While it is possible to performpart-of-speechtaggingin th is way, this simple left-
to-rightclassiﬁer has an importantﬂaw: it makesa harddeci sion oneach wordbefore
movingontothenextword. Thismeansthattheclassiﬁerisun abletouseinformation
fromlaterwordstoinformitsdecisionearlyon. Recallthat inHiddenMarkovModels,
by contrast, we didn’t have to make a hard decision at each wor d; we used Viterbi
decodingto ﬁnd the sequence of part-of-speechtags which wa s optimal for the whole
sentence.
TheMaximumEntropyMarkovModel(orMEMM)allowsustoachie vethissame

sentence.
advantage, by mating the Viterbi algorithm with MaxEnt. Let ’s see how it works,
again looking at part-of-speech tagging. It is easiest to un derstand an MEMM when
comparingittoanHMM.RememberthatinusinganHMMtomodelt hemostprobable
part-of-speech tag sequence we rely on Bayes rule, computin gP(W|T)P(W)instead
ofdirectlycomputing P(T|W):
ˆT=argmax
TP(T|W)
=argmax
TP(W|T)P(T)
=argmax
T∏
iP(wordi|tagi)∏
iP(tagi|tagi−1) (6.95)

DRAFTSection6.8. MaximumEntropyMarkovModels 209
That is, an HMM as we’ve described it is a generative model tha t optimizes the
likelihood P(W|T), andwe estimate theposteriorbycombiningthelikelihooda ndthe
priorP(T).
InanMEMM,bycontrast,wecomputetheposterior P(T|W)directly. Becausewe
train the model directly to discriminate among the possible tag sequences, we call an
MEMM a discriminative model rather than a generative model. In an MEMM, weDistriminativemodel
breakdownthe probabilitiesasfollows:
ˆT=argmax
TP(T|W)
=argmax
T∏
iP(tagi|wordi,tagi−1) (6.96)
Thus in an MEMM instead of having a separate model for likelih oods and priors,
we train a single probabilistic model to estimate P(tagi|wordi,tagi−1). We will use
MaxEntforthislast piece,estimatingtheprobabilityof ea chlocaltaggiventheprevi-
oustag,theobservedword,and,aswe will see,anyotherfeat ureswe wantto include.
WecanseetheHMMversusMEMMintuitionsofthePOStaggingta skinFig.6.20,
whichrepeatstheHMMmodelofFig.5.12afromCh.5,andaddsa newmodelforthe

ˆT=argmax
TP(T|W)
=argmax
T∏
MEMM. Note that the HMM model includes distinct probability estimates for each
transitionandobservation,whiletheMEMMgivesoneprobab ilityestimateperhidden
state,whichistheprobabilityofthenexttaggiventheprev ioustagandtheobservation.
Secretariat expected to race tomorrow
4
Figure6.20 The HMM (top) and MEMM (bottom) representation of the probab ility compu-

4
tationforthecorrect sequence of tagsfortheSecretariats entence. Eacharcwouldbeassociated
witha probability; the HMM computes two separate probabili ties for the observation likelihood
andtheprior,whiletheMEMMcomputesasingleprobabilityf unctionateachstate,conditioned
onthe previous stateand current observation.
Fig. 6.21 emphasizes another advantage of MEMMs over HMMs no t shown in
Fig.6.20: unliketheHMM,theMEMMcanconditiononanyusefu lfeatureoftheinput

observation. In the HMM this wasn’t possible because the HMM is likelihood-based,
hencewouldhaveneededtocomputethelikelihoodofeachfea tureoftheobservation.

DRAFT210 Chapter 6. HiddenMarkovandMaximumEntropyModels
expectedisNNP VBZ VBN TO VB NR
Secretariat to race tomorrow#
Figure6.21 An MEMM for part-of-speech tagging, augmenting the descrip tion in Fig. 6.20
by showing that an MEMM can condition on many features of the i nput, such as capitalization,
morphology(endingin -sor-ed),aswellasearlierwordsortags. Wehaveshownsomepotenti al
additional features for the ﬁrstthree decisions, using dif ferent linestyles for each class.
Moreformally,intheHMMwecomputetheprobabilityofthest atesequencegiven
theobservationsas:
P(Q|O) =n
∏
∏
i=1P(qi|qi−1) (6.97)
In the MEMM, we compute the probability of the state sequence given the obser-
vationsas:
P(Q|O) =n
∏
i=1P(qi|qi−1,oi) (6.98)
In practice, however, an MEMM can also condition on many more features than
theHMM,so ingeneralwe conditiontheright-handside onman ymorefactors.
To estimate the individual probability of a transition from a stateq′to a state q
producingan observation o,webuilda MaxEntmodelasfollows:

P(Q|O) =n
∏
∏
P(Q|O) =n
∏
P(q|q′,o) =1
Z(o,q′)exp(
∑
iwifi(o,q))
(6.99)
6.8.1 Decoding and Learning inMEMMs
Like HMMs, the MEMM uses the Viterbi algorithm to perform the task of decoding
(inference). Concretely,thisinvolvesﬁllingan N×Tarraywiththeappropriatevalues
forP(ti|ti−1,wordi), maintaining backpointers as we proceed. As with HMM Viterb i,
when the table is ﬁlled we simply follow pointers back from th e maximum value in
the ﬁnal column to retrieve the desired set of labels. The req uisite changes from the

∑
(6.99)
HMM-style application of Viterbi only have to do with how we ﬁ ll each cell. Recall
fromEq.6.23thattherecursivestepoftheViterbiequation computestheViterbivalue
oftimetforstatejas:
i=1vt−1(i)aijbj(ot); 1≤j≤N,1<t≤T (6.100)
whichistheHMMimplementationof

DRAFTSection6.8. MaximumEntropyMarkovModels 211
i=1vt−1(i)P(sj|si)P(ot|sj)1≤j≤N,1<t≤T (6.101)
TheMEMMrequiresonlyaslightchangetothislatterformula ,replacingthe aand
bpriorandlikelihoodprobabilitieswiththedirectposteri or:
i=1vt−1(i)P(sj|si,ot)1≤j≤N,1<t≤T (6.102)
Fig. 6.22showsan exampleof the Viterbi trellis for an MEMM a ppliedto the ice-
cream task from Sec. 6.4. Recall that the task is ﬁguring out t he hidden weather (Hot
orCold)fromobservednumbersofice-creamseatenin JasonE isner’sdiary. Fig.6.22
shows the abstract Viterbi probability calculation assumi ng that we have a MaxEnt
modelwhichcomputes P(si|si−1,oi)forus.
startH
CH
CH
Cend
P(C|start,3)P(H|H,1)
P(C|C,1)P(C|H,1)
P(H|C,1)
P(H|start,3)v1(2)=P(H|start,3)
v1(1) = P(C|start,3)v2(2)= max(  P(H|H,1)*P(H|start,3),
                       P(H|C,1)*P(C|start,3)  )
v2(1) = max(  P(C|H,1)*P(H|start,3),
                    P(C|C,1)*P(C|start,3)    )
tCHend end end end
H
C
startqend
q2
q1
q0
o13 3 1
o2 o3

startH
CH
CH
Cend
P(H|C,1)
H
C
startqend
q2
q1
q0
o13 3 1
o2 o3
Figure6.22 Inference from ice-cream eating computed by an MEMM instead of an HMM. The Viterbitrellis for
computing the best path through the hidden state space for th e ice-cream eating events 3 1 3, modiﬁed from the
HMMﬁgure inFig.6.10.
Learning in MEMMs relies on the same supervised learning alg orithms we pre-
sented for logistic regression and MaxEnt. Given a sequence of observations, fea-
ture functions, and corresponding hidden states, we train t he weights so as maximize

q2
q1
q0
o13 3 1
o2 o3
the log-likelihood of the training corpus. As with HMMs, it i s also possible to train
MEMMs in semi-supervised modes, for example when the sequen ce of labels for the
trainingdataismissingorincompleteinsomeway: aversion oftheEMalgorithmcan
beusedforthispurpose.

DRAFT212 Chapter 6. HiddenMarkovandMaximumEntropyModels
6.9 Summary
Thischapterdescribedtwoimportantmodelsforprobabilis ticsequenceclassiﬁcation :
theHidden MarkovModel andtheMaximum EntropyMarkovModel . Both mod-
elsarewidelyusedthroughoutspeechandlanguageprocessi ng.
•Hidden Markov Models ( HMMs) are a way of relating a sequence of obser-
vationsto a sequence of hidden classes orhidden states which explain the
observations.
•The process of discovering the sequence of hidden states giv en the sequence
of observations is known as decoding orinference . TheViterbialgorithm is
commonlyusedfordecoding.
•The parameters of an HMM are the Atransition probability matrix and the B
observation likelihood matrix. Both can be trained using th eBaum-Welch or
forward-backward algorithm.
•AMaxEntmodelisaclassiﬁerwhichassignsa classtoanobservation bycom-
putinga probabilityfrom an exponentialfunction of a weighted set offeatures
ofthe observation.
•MaxEntmodelscanbetrainedusingmethodsfromtheﬁeldof convexoptimiza-

tionalthoughwe don’tgivethedetailsin thistextbook.
•AMaximum Entropy Markov Model orMEMM is a sequence model aug-
mentationofMaxEntwhichmakesuse oftheViterbidecodinga lgorithm.
•MEMMscanbetrainedbyaugmentingMaxEnttrainingwitha ver sionofEM.
As we discussed at the end of Ch. 4, Markovchainswere ﬁrst use d by Markov(1913,
2006), to predict whether an upcoming letter in Pushkin’s Eugene Onegin would be a
voweloraconsonant.

TheHiddenMarkovModelwasdevelopedbyBaumandcolleagues attheInstitute
forDefenseAnalysesin Princeton(BaumandPetrie,1966;Ba umandEagon,1967).
TheViterbialgorithm was ﬁrst applied to speech and language processin g in the
contextofspeechrecognitionbyVintsyuk(1968),but haswh at Kruskal(1983)callsa
‘remarkablehistoryof multipleindependentdiscoveryand publication’.2Kruskaland
others give at least the following independently-discover ed variants of the algorithm
publishedinfourseparateﬁelds:

2Seven is pretty remarkable, but seepage 13 for adiscussion o f the prevalence of multiple discovery.

DRAFTSection6.9. Summary 213
Citation Field
Viterbi(1967) informationtheory
Vintsyuk(1968) speechprocessing
NeedlemanandWunsch(1970) molecularbiology
SakoeandChiba(1971) speechprocessing
Sankoff(1972) molecularbiology
Reichertet al.(1973) molecularbiology
WagnerandFischer(1974) computerscience
The use of the term Viterbiis now standard for the application of dynamic pro-
gramming to any kind of probabilistic maximization problem in speech and language
processing. For non-probabilistic problems (such as for mi nimum edit distance) the
plain term dynamic programming is often used. ForneyJr. (1973)is an early survey
paper which exploresthe origin of the Viterbi algorithm in t he context of information
andcommunicationstheory.
Our presentation of the idea that Hidden Markov Models shoul d be characterized
by three fundamental problems was modeled after an inﬂuenti al tutorial by Rabiner
(1989), which was itself based on tutorials by Jack Ferguson of IDA in the 1960s.

Jelinek (1997) and Rabiner and Juang (1993) give very comple te descriptions of the
forward-backward algorithm, as applied to the speech recog nition problem. Jelinek
(1997) also shows the relationship between forward-backwa rd and EM. See also the
description of HMMs in other textbooks such as Manning and Sc h¨ utze (1999). See
Durbinetal.(1998)fortheapplicationofprobabilisticmo delslikeHMMstobiological
sequencesofproteinsandnucleicacids. Bilmes(1997)isa t utorialonEM.
Whilelogisticregressionandotherlog-linearmodelshave beenusedinmanyﬁelds

sincethemiddleofthe20thcentury,theuseofMaximumEntro py/multinomiallogistic
regression in natural language processing dates from work i n the early 1990s at IBM
(Bergeretal.,1996;DellaPietraetal.,1997). Thisearlyw orkintroducedthemaximum
entropy formalism, proposed a learning algorithm (improve d iterative scaling), and
proposedtheuseofregularization. Anumberofapplication sofMaxEntfollowed. For
further discussion of regularization and smoothing for max imum entropy models see

(inter alia ) Chen and Rosenfeld (2000), Goodman (2004), and Dud´ ık and S chapire
(2006).
Although the second part of this chapter focused on MaxEnt-s tyle classiﬁcation,
numerousother approachesto classiﬁcation are used throug houtspeech and language
processing. Naive Bayes (Duda et al., 2000) is often employe d as a good baseline
method(often yieldingresults that are sufﬁciently goodfo r practical use); we’ll cover
naiveBayesinCh.20. SupportVectorMachines(Vapnik,1995 )havebeensuccessfully

(2006).
used in text classiﬁcation and in a wide variety of sequence p rocessing applications.
Decision lists have been widely used in word sense discrimin ation, and decision trees
(Breiman et al., 1984; Quinlan, 1986) have been used in many a pplications in speech
processing. Good referencesto supervised machine learnin g approachesto classiﬁca-
tionincludeDudaetal. (2000),Hastie etal. (2001),andWit ten andFrank(2005).
Maximum Entropy Markov Models (MEMMs) were introduced by Ra tnaparkhi
(1996)andMcCallumetal. (2000).

There are many sequence models that augment the MEMM, such as theCondi-

DRAFT214 Chapter 6. HiddenMarkovandMaximumEntropyModels
tional Random Field (CRF) (Lafferty et al., 2001; Sutton and McCallum, 2006). InConditionalRandom Field
CRFaddition,therearevariousgeneralizationsof maximummargin methods(theinsights
thatunderlieSVM classiﬁers) tosequencetasks.

Chapter7
Phonetics
(Uponbeing asked by DirectorGeorge Cukor toteach RexHarri son, the star of the 1964
ﬁlm”MyFairLady”, how tobehave like a phonetician:)
“My immediate answer was, ‘I don’t have a singing butler and t hree maids
whosing,butIwill tell youwhatI canasanassistantprofess or.’”
PeterLadefoged, quoted inhisobituary, LATimes, 2004
The debate between the “whole language” and “phonics” metho ds of teaching read-
ing to children seems at very glance like a purely modern educ ational debate. Like
many modern debates, however, this one recapitulates an imp ortant historical dialec-

tic, in this case in writing systems. The earliest independe ntly-invented writing sys-
tems (Sumerian, Chinese, Mayan) were mainly logographic: o ne symbol represented
a whole word. But from the earliest stages we can ﬁnd, most suc h systems contain
elements of syllabic or phonemic writing systems, in which s ymbols are used to rep-
resent the sounds that make up the words. Thus the Sumerian sy mbol pronounced ba
and meaning “ration”could also functionpurely as the sound /ba/. Even modernChi-

nese, which remains primarily logographic, uses sound-bas ed characters to spell out
foreign words. Purely sound-based writing systems, whethe r syllabic (like Japanese
hiragana orkatakana),alphabetic(liketheRomanalphabetusedinthisbook),or con-
sonantal (like Semitic writing systems), can generally be t raced back to these early
logo-syllabicsystems,oftenastwo culturescametogether . ThustheArabic,Aramaic,
Hebrew, Greek, and Roman systems all derive from a West Semit ic script that is pre-

sumed to have been modiﬁed by Western Semitic mercenariesfr om a cursive form of
Egyptianhieroglyphs. The Japanese syllabarieswere modiﬁ ed froma cursive formof
a set of Chinese characterswhich were used to representsoun ds. These Chinese char-
acters themselves were used in Chinese to phonetically repr esent the Sanskrit in the
Buddhistscripturesthatwere broughttoChinainthe Tangdy nasty.
Whatever its origins, the idea implicit in a sound-based wri ting system, that the

spoken word is composed of smaller units of speech, is the Ur- theory that underlies
all our modern theories of phonology . This idea of decomposing speech and words
into smaller units also underlies the modern algorithms for speech recognition (tran-
scrbiningacousticwaveformsintostringsoftextwords)an dspeechsynthesis ortext-
to-speech (convertingstringsoftextwordsintoacousticwaveforms) .
Inthischapterweintroduce phonetics fromacomputationalperspective. Phonetics
isthestudyoflinguisticsounds,howtheyareproducedbyth earticulatorsofthehuman

vocaltract,howtheyarerealizedacoustically,andhowthi sacousticrealizationcanbe
digitizedandprocessed.
We begin with a key element of both speech recognition and tex t-to-speech sys-
tems: howwordsarepronouncedintermsofindividualspeech unitscalled phones. A

DRAFT216 Chapter 7. Phonetics
speech recognition system needs to have a pronunciationfor every word it can recog-
nize, and a text-to-speech system needs to have a pronunciat ionfor every word it can
say. The ﬁrst section of this chapter will introduce phonetic alphabets for describing
thesepronunciations. Wethenintroducethetwomainareaso fphonetics, articulatory
phonetics , the study of how speech soundsare producedby articulators in the mouth,
andacousticphonetics , thestudyofthe acousticanalysisofspeechsounds.
We also brieﬂy touch on phonology , the area of linguistics that describes the sys-
tematic way that sounds are differently realized in differe nt environments, and how
this system of sounds is related to the rest of the grammar. In doing so we focus on
the crucialfactof variation inmodelingspeech;phonesarepronounceddifferentlyin
differentcontexts.
7.1 SpeechSoundsandPhoneticTranscription
The study of the pronunciationof words is part of the ﬁeld of phonetics , the study of Phonetics

thespeechsoundsusedinthelanguagesoftheworld. Wemodel thepronunciationofa
wordasastringofsymbolswhichrepresent phonesorsegments . Aphoneisaspeech Phone
sound;phonesare representedwithphoneticsymbolsthat be arsome resemblancetoa
letterin analphabeticlanguagelike English.
This section surveys the different phones of English, parti cularly American En-
glish, showing howthey are producedand how theyare represe ntedsymbolically. We
will be using two different alphabets for describing phones . TheInternational Pho-

neticAlphabet (IPA)isanevolvingstandardoriginallydevelopedbytheIntern ational IPA
Phonetic Association in 1888 with the goal of transcribing t he sounds of all human
languages. TheIPA isnotjustanalphabetbutalsoa set ofpri nciplesfortranscription,
which differ according to the needs of the transcription, so the same utterance can be
transcribed in different ways all according to the principl es of the IPA. The ARPAbet
(Shoup, 1980) is another phonetic alphabet, but one that is s peciﬁcally designed for
AmericanEnglishandwhichusesASCIIsymbols;itcanbethou ghtofasaconvenient

ASCIIrepresentationofanAmerican-EnglishsubsetoftheI PA.ARPAbetsymbolsare
often used in applicationswhere non-ASCIIfonts are inconv enient,such as in on-line
pronunciation dictionaries. Because the ARPAbet is very co mmon for computational
representationsof pronunciations,we will rely on it rathe rthan the IPA in the remain-
der of this book. Fig. 7.1 and Fig. 7.2 show the ARPAbet symbol s for transcribing
consonantsandvowels,respectively,togetherwiththeirI PA equivalents.

Many of the IPA and ARPAbet symbols are equivalent to the Roma n letters used
in the orthographyof English and many other languages. So fo r example the ARPA-
bet phone [p]represents the consonant sound at the beginning of platypus,puma, and
pachyderm , the middle of leopard, or the end of antelope. In general, however, the
1The phone [ux] is rare in general American English and not gen erally used in speech recogni-
tion/synthesis. It is used to represent the fronted [uw] whi ch appeared in (at least) Western and Northern

Cities dialects of American English starting in the late 197 0s (Labov, 1994). This fronting was ﬁrst called
to public by imitations and recordings of ‘Valley Girls’ spe ech by Moon Zappa (Zappa and Zappa, 1982).
Nevertheless, for mostspeakers [uw] is still much more comm on than [ux] in words like dude.

DRAFTSection7.1. SpeechSoundsandPhoneticTranscription 217
ARPAbet IPA ARPAbet
Symbol Symbol Word Transcription
[p] [p] p arsley [paars liy]
[t] [t] t ea [tiy]
[k] [k] c ook [kuhk]
[b] [b] b ay [bey]
[d] [d] d ill [dihl]
[g] [g] g arlic [gaarl ix k]
[m] [m] m int [mihnt]
[n] [n] n utmeg [naht mehg]
[ng] [ N] baking [beyk ixng]
[f] [f] f lour [fl awaxr]
[v] [v] clov e [klow v]
[th] [ T] th ick [thihk]
[dh] [ D] th ose [dhowz]
[s] [s] s oup [suw p]
[z] [z] eggs [ehg z]
[sh] [ S] squash [sk waa sh]
[zh] [ Z] ambros ia [aembr owzhax]
[ch] [t S] ch erry [cheh riy]
[jh] [d Z] j ar [jhaar]
[l] [l] l icorice [lih kaxrixsh]
[w] [w] kiw i [kiyw iy]
[r] [r] r ice [rays]
[y] [j] y ellow [yehl ow]
[h] [h] h oney [hahniy]
Lesscommonlyusedphonesandallophones
[q] [ P] uh -oh [qahqow]
[dx] [ R] butt er [bahdxaxr]
[nx] [˜ R] winn er [wihnxaxr]
[el] [l"] table [teybel]
Figure7.1 ARPAbet symbols for transcription of English consonants, w ith IPA equivalents.

Note that some rarer symbols like the ﬂap [dx], nasal ﬂap [nx] , glottal stop [q] and the syllabic
consonants, are used mainlyfor narrow transcriptions.
mapping between the letters of English orthographyand phon es is relatively opaque;
a single letter can represent very different sounds in diffe rent contexts. The English
letterccorresponds to phone [k] in cougar[k uw g axr], but phone [s] in cell[s eh
l]. Besides appearingas candk, the phone [k] can appear as part of x(fox[f aa k s]),

asck(jackal[jh ae k el] and as cc(raccoon[r ae k uw n]). Many other languages,
forexampleSpanish,aremuchmore transparent intheirsound-orthographymapping
thanEnglish.

DRAFT218 Chapter 7. Phonetics
[iy] [i] lily [lih l iy]
[ih] [I] lily [lih l iy]
[ey] [eI] daisy [deyziy]
[eh] [E] pen [pehn]
[ae] [æ] aster [aest axr]
[aa] [A] poppy [paap iy]
[ao] [O] orchid [aorkix d]
[uh] [U] wood [wuhd]
[ow] [oU] lotus [low dxaxs]
[uw] [u] tulip [tuw lix p]
[ah] [2] buttercup [bahdxaxrkahp]
[er] [Ç] bird [berd]
[ay] [aI] iris [ayrixs]
[aw] [aU] sunﬂower [sahnfl aw axr]
[oy] [oI] soil [soyl]
Reducedanduncommonphones
[ax] [@] lotus [low dxaxs]
[axr] [Ä] heather [hehdhaxr]
[ix] [1] tulip [tuw lix p]
[ux] [0] dude1[duxd]
Figure7.2 ARPAbetsymbols for transcriptionofEnglishvowels, withI PAequivalents. Note
againthelistofrarerphonesandreducedvowels(seeSec.7. 2.4);forexample[ax]isthereduced
vowel schwa, [ix] is the reduced vowel corresponding to [ih] , and [axr] is the reduced vowel
corresponding to[er].
7.2 ArticulatoryPhonetics
The list of ARPAbet phones is useless without an understandi ng of how each phone

is produced. We thus turn to articulatory phonetics , the study of how phones arearticulatory
phonetics
produced,asthevariousorgansinthemouth,throat,andnos emodifytheairﬂowfrom
thelungs.
7.2.1 The VocalOrgans
Sound is produced by the rapid movement of air. Most sounds in human spoken lan-
guages are produced by expelling air from the lungs through t he windpipe (techni-
cally the trachea) and then out the mouth or nose. As it passes through the trach ea,

phonetics
thelungs.
the air passes through the larynx, commonly known as the Adam’s apple or voice-
box. The larynx contains two small folds of muscle, the vocal folds (often referred
to non-technically as the vocal cords ) which can be moved together or apart. The
space between these two foldsis called the glottis. If the folds are close together (but Glottis
not tightly closed), they will vibrate as air passes through them; if they are far apart,
theywon’tvibrate. Soundsmadewiththevocalfoldstogethe randvibratingarecalled

DRAFTSection7.2. ArticulatoryPhonetics 219
Figure7.3 Thevocalorgans, showninside view. DrawingbyLaszloKubin yi fromSundberg
(1977),c⃝ScientiﬁcAmerican, used bypermission.
voiced; sounds made without this vocal cord vibration are called unvoiced orvoice- Voicedsound
Unvoicedsound less. Voiced sounds include [b], [d], [g], [v], [z], and all the En glish vowels, among
voiceless others. Unvoicedsoundsinclude[p],[t],[k],[f],[s], and others.
The area abovethe trachea is called the vocal tract , and consists of the oral tract
and thenasal tract . After the air leaves the trachea, it can exit the body throug h the
mouth or the nose. Most sounds are made by air passing through the mouth. Sounds
made by air passing through the nose are called nasal sounds ; nasal sounds use both Nasal
the oral and nasal tracts as resonatingcavities; English na sal soundsinclude m, andn,
andng.
Phones are divided into two main classes: consonants andvowels. Both kinds of Consonant

andng.
Vowelsoundsareformedbythemotionofair throughthemouth,thro atornose. Consonants
are made by restricting or blocking the airﬂow in some way, an d may be voiced or

DRAFT220 Chapter 7. Phonetics
unvoiced. Vowels have less obstruction, are usually voiced , and are generally louder
and longer-lasting than consonants. The technical use of th ese terms is much like the
common usage; [p], [b], [t], [d], [k], [g], [f], [v], [s], [z] , [r], [l], etc., are consonants;
[aa], [ae], [ao], [ih], [aw], [ow], [uw], etc., are vowels. Semivowels (such as [y] and
[w])havesomeofthepropertiesofboth;theyarevoicedlike vowels,buttheyareshort
andlesssyllabiclikeconsonants.
7.2.2 Consonants: PlaceofArticulation
Because consonants are made by restricting the airﬂow in som e way, consonants can
be distinguished by where this restriction is made: the poin t of maximum restriction
is called the place of articulation of a consonant. Places of articulation, shown inPlaceof
articulation
Fig. 7.4, are often used in automatic speech recognition as a useful way of grouping
phonestogetherintoequivalenceclasses:
dental
palatal alveolar
bilabial velar
glottal(nasal  tract)

dental
Figure7.4 Major Englishplaces of articulation.
labial:Consonantswhosemainrestrictionisformedbythetwolipsc omingtogether Labial
have abilabialplace of articulation. In English these include [p] as in possum,
[b] as inbear, and [m] as in marmot. The English labiodental consonants [v]
and [f] are made by pressing the bottom lip against the upper r ow of teeth and
lettingtheairﬂowthroughthe spaceintheupperteeth.
dental:Soundsthataremadebyplacingthetongueagainsttheteetha redentals. The Dental

main dentals in English are the [th] of thingor the [dh] of though, which are
made by placing the tongue behind the teeth with the tip sligh tly between the
teeth.
alveolar: The alveolar ridge is the portion of the roof of the mouth just behind the Alveolar
upper teeth. Most speakers of American English make the phon es [s], [z], [t],
and [d] by placing the tip of the tongue against the alveolar r idge. The word
coronalisoftenusedtorefertobothdentalandalveolar. Coronal

teeth.
palatal:Theroofofthemouth(the palate)risessharplyfromthebackofthealveolar Palatal
Palate ridge. The palato-alveolar sounds[sh] ( shrimp), [ch](china),[zh](Asian),and
[jh] (jar) are made with the blade of the tongue against this rising bac k of the

DRAFTSection7.2. ArticulatoryPhonetics 221
alveolar ridge. The palatal sound [y] of yakis made by placing the front of the
tongueupcloseto thepalate.
velar:Thevelumor soft palate is a movable muscular ﬂap at the very back of the Velar
Velum roofof the mouth. The sounds[k] ( cuckoo),[g] (goose), and [N](kingﬁsher) are
madebypressingthe backofthetongueupagainstthevelum.
glottal:The glottal stop [q] (IPA [P]) is made by closing the glottis (by bringing the Glottal
vocalfoldstogether).
7.2.3 Consonants: Manner ofArticulation
Consonantsarealsodistinguishedby howtherestrictioninairﬂowismade,forexample
whetherthereisacompletestoppageofair,oronlyapartial blockage,etc. Thisfeature
is called the manner of articulation of a consonant. The combination of place andMannerof
mannerofarticulationisusuallysufﬁcienttouniquelyide ntifyaconsonant. Following
arethemajormannersofarticulationforEnglishconsonant s:
Astopis a consonant in which airﬂow is completely blocked for a sho rt time. stop

This blockage is followed by an explosive sound as the air is r eleased. The period
of blockage is called the closureand the explosion is called the release. English has
voiced stops like [b], [d], and [g] as well as unvoicedstops l ike [p], [t], and [k]. Stops
are also called plosives. Some computational systems use a more narrow (detailed)
transcriptionstylethathasseparatelabelsfortheclosur eandreleasepartsofastop. In
oneversionoftheARPAbet,forexample,theclosureofa [p], [t],or[k]isrepresented

as [pcl], [tcl], or [kcl] (respectively), while the symbols [p], [t], and [k] are used to
mean only the release portion of the stop. In another version the symbols [pd], [td],
[kd], [bd], [dd], [gd] are used to mean unreleased stops (sto ps at the end of words or
phrasesoftenaremissingtheexplosiverelease),while[p] ,[t],[k],etcareusedtomean
normal stops with a closure and a release. The IPA uses a speci al symbol to mark
unreleased stops: [p^],[t^], or[k^]. We will not be using these narrow transcription

stylesinthischapter;wewillalwaysuse[p]tomeanafullst opwithbothaclosureand
a release.
Thenasalsounds[n], [m], and[ng]are made by loweringthe velumand al lowing Nasal
airto passintothe nasalcavity.
Infricatives , airﬂow is constricted but not cut off completely. The turbu lent air- Fricative
ﬂow that results from the constriction produces a character istic “hissing” sound. The
Englishlabiodentalfricatives[f]and[v]areproducedbyp ressingthelowerlipagainst

a release.
theupperteeth,allowingarestrictedairﬂowbetweentheup perteeth. Thedentalfrica-
tives [th] and [dh]allow air to ﬂow aroundthe tonguebetween the teeth. The alveolar
fricatives[s]and[z]areproducedwiththetongueagainstt healveolarridge,forcingair
overtheedgeoftheteeth. Inthepalato-alveolarfricative s[sh]and[zh]thetongueisat
the back of the alveolar ridge forcing air through a groove fo rmed in the tongue. The
higher-pitched fricatives (in English [s], [z], [sh] and [z h] are called sibilants. Stops Sibilant

thatarefollowedimmediatelybyfricativesarecalled affricates ;theseincludeEnglish
[ch](chicken)and[jh]( giraffe).
Inapproximants , the two articulators are close together but not close enoug h to Approximant
cause turbulent airﬂow. In English [y] ( yellow), the tongue moves close to the roof
of the mouth but not close enough to cause the turbulence that would characterize

DRAFT222 Chapter 7. Phonetics
a fricative. In English [w] ( wood), the back of the tongue comes close to the velum.
American[r]canbeformedinatleasttwoways;withjustthet ipofthetongueextended
and close to the palate or with the whole tongue bunched up nea r the palate. [l] is
formed with the tip of the tongue up against the alveolar ridg e or the teeth, with one
or both sides of the tongue lowered to allow air to ﬂow over it. [l] is called a lateral
soundbecauseofthedropinthesidesofthetongue.
Ataporﬂap[dx] (or IPA [R]) is a quick motion of the tongueagainst the alveolar Tap
ﬂapridge. The consonant in the middle of the word lotus([l ow dx ax s]) is a tap in most
dialectsofAmericanEnglish;speakersofmanyUK dialectsw oulduse a[t]insteadof
a tapinthisword.
7.2.4 Vowels
Likeconsonants,vowelscanbecharacterizedbythepositio nofthearticulatorsasthey
are made. The three most relevant parameters for vowels are w hat is called vowel
height, which correlates roughly with the height of the highest par t of the tongue,

vowelfrontness orbackness , which indicates whether this high point is toward the
front or back of the oral tract, and the shape of the lips ( rounded or not). Fig. 7.5
showsthepositionofthetonguefordifferentvowels.
heed [iy] had [ae] who’d [uw]nasal  tract
palatetongueclosed
velum
Figure7.5 Positions of the tongue for three Englishvowels, high front [iy],low front [ae]and
high back [uw]; tongue positions modeled afterLadefoged (1996).
In the vowel [iy], for example, the highest point of the tongu e is toward the front

velum
of the mouth. In the vowel [uw], by contrast, the high-point o f the tongue is located
toward the back of the mouth. Vowels in which the tongue is rai sed toward the front
are called frontvowels ; those in whichthe tongueis raisedtowardthe backare calle d Front vowel
back vowels . Note that while both [ih]and [eh] are front vowels, the tong ueis higher Back vowel
for[ih]thanfor[eh]. Vowelsinwhichthehighestpointofth etongueiscomparatively
higharecalled highvowels ;vowelswithmidorlowvaluesofmaximumtongueheight High vowel
arecalled mid vowels orlow vowels ,respectively.

Fig.7.6showsaschematiccharacterizationofthevowelhei ghtofdifferentvowels.
Itisschematicbecausetheabstractproperty heightonlycorrelatesroughlywithactual
tongue positions; it is in fact a more accurate reﬂection of a coustic facts. Note that
the chart has two kinds of vowels: those in which tongue heigh t is represented as a
point and those in which it is represented as a vector. A vowel in which the tongue
positionchangesmarkedlyduringtheproductionofthevowe lisadiphthong . English Diphthong
isparticularlyrichin diphthongs.

DRAFTSection7.2. ArticulatoryPhonetics 223
high
front back
ae
lowiy
ihy uw uw
uh
aw
aaey
axoy
ayehow
ao
uh
Figure7.6 Qualitiesof Englishvowels (after Ladefoged (1993)).
The second important articulatory dimension for vowels is t he shape of the lips.
Certain vowels are pronounced with the lips rounded (the sam e lip shape used for
whistling). These rounded vowelsinclude[uw],[ao],and[ow]. Roundedvowel
Syllables
Consonants and vowels combine to make a syllable. There is no completely agreed- Syllable
upondeﬁnitionofasyllable;roughlyspeakingasyllableis avowel-like(or sonorant )
sound together with some of the surroundingconsonantsthat are most closely associ-
ated with it. The word doghas one syllable, [d aa g], while the word catniphas two
syllables, [k ae t] and [n ih p], We call the vowel at the core of a syllable the nucleus. Nucleus
The optionalinitial consonantor set of consonantsis calle dtheonset. If the onset has Onset
more than one consonant (as in the word strike[s t r ay k]), we say it has a complex

onset. Thecoda. is the optional consonant or sequence of consonants follow ing the Coda
nucleus. Thus[d]istheonsetof dog,while[g]isthecoda. The rime. orrhyme. isthe Rime
nucleuspluscoda. Fig.7.7showssomesamplesyllablestruc tures.
σ
Onset
hRime
Nucleus
aeCoda
mσ
Onset
g rRime
Nucleus
iyCoda
nσ
Rime
Nucleus
ehCoda
g z
Figure7.7 Syllablestructure of ham,green,eggs.σ=syllable.
The task of automatically breaking up a word into syllables i s calledsyllabiﬁca-
tion,andwillbe discussedinSec.11.4. Syllabiﬁcation

σ
Onset
hRime
Nucleus
aeCoda
mσ
Onset
g rRime
Nucleus
iyCoda
nσ
Rime
Nucleus
ehCoda
g z
Syllable structure is also closely related to the phonotactics of a language. The
termphonotactics means the constraints on which phones can follow each other i n a Phonotactics

DRAFT224 Chapter 7. Phonetics
language. For example, English has strong constraints on wh at kinds of consonants
can appear together in an onset; the sequence [zdr], for exam ple, cannot be a legal
Englishsyllableonset. Phonotacticscanberepresentedby listingconstraintsonﬁllers
of syllable positions, or by creating a ﬁnite-state model of possible phone sequences.
It is also possible to create a probabilisticphonotactics, by training N-gramgrammars
onphonesequences.
LexicalStressand Schwa
In a naturalsentence of AmericanEnglish, certain syllable sare more prominent than
others. These are called accented syllables, and the linguistic marker associated with
this prominence is called a pitch accent . Words or syllables which are prominent Pitchaccent
are said to bear(be associated with) a pitch accent. Pitch accent is also som etimes
referred to as sentence stress , although sentence stress can instead refer to only the
mostprominentaccentina sentence.

Accented syllables may be prominent by being louder, longer , by being associ-
ated with a pitch movement, or by any combination of the above . Since accent plays
important roles in meaning, understanding exactly why a spe aker chooses to accent a
particularsyllableisverycomplex,andwewillreturntoth isindetailinSec.8.3.2. But
one important factor in accent is often representedin pronu nciationdictionaries. This
factoriscalled lexicalstress . Thesyllablethat haslexicalstressis theonethatwill be Lexicalstress

louder or longer if the word is accented. For example the word parsleyis stressed in
its ﬁrst syllable, not its second. Thus if the word parsleyreceives a pitch accent in a
sentence,it isthe ﬁrst syllablethatwill bestronger.
In IPA we write the symbol ["]beforea syllable to indicate that it has lexical stress
(e.g. ["par.sli]). This differencein lexical stress can affect the meaning o f a word. For
exampletheword contentcanbeanounoranadjective. Whenpronouncedinisolation

the two senses are pronounced differently since they have di fferent stressed syllables
(thenounispronounced ["kAn.tEnt] andtheadjective [k@n."tEnt] ).
Vowelswhichareunstressedcanbeweakenedevenfurtherto reducedvowels . The Reducedvowel
most common reduced vowel is schwa([ax]). Reduced vowels in English don’t have Schwa
theirfullform;thearticulatorygestureisn’tascomplete asforafullvowel. Asaresult
theshapeofthe mouthissomewhatneutral;the tongueisneit herparticularlyhighnor

particularlylow. For examplethe secondvowel in parakeetis a schwa: [p ae r ax k iy
t].
Whileschwaisthemostcommonreducedvowel,it isnottheonl yone,atleastnot
in some dialects. Bolinger (1981) proposed that American En glish had three reduced
vowels: a reduced mid vowel [@], a reduced front vowel [1], and a reduced rounded
vowel [8]. The full ARPAbet includes two of these, the schwa [ax] and [i x] ([1]), as
wellas[axr]whichisanr-coloredschwa(oftencalled schwar),although[ix]isgener-

t].
allydroppedincomputationalapplications(Miller,1998) ,and[ax]and[ix]arefalling
togetherinmanydialectsofEnglishWells (1982,p. 167–168 ).
Notallunstressedvowelsarereduced;anyvowel,anddiphth ongsinparticularcan
retain their full quality even in unstressed position. For e xample the vowel [iy] can
appearinstressedpositionasintheword eat[iyt]orinunstressedpositionintheword
carry[kae riy].

DRAFTSection7.3. PhonologicalCategoriesandPronunciationVa riation 225
Some computational ARPAbet lexicons mark reduced vowels li ke schwa explic-
itly. But in general predictingreduction requires knowled geof things outside the lex-
icon (the prosodic context, rate of speech, etc, as we will se e the next section). Thus
other ARPAbet versions mark stress but don’t mark how stress affects reduction. The
CMUdictionary(CMU,1993),forexample,markseachvowelwi ththenumber0(un-
stressed) 1 (stressed), or 2 (secondary stress). Thus the wo rdcounteris listed as [K
AW1N TER0],andtheword tableas[TEY1BAH0L]. Secondarystress isdeﬁned Secondary stress
asalevelofstresslowerthanprimarystress,buthighertha nanunstressedvowel,asin
theword dictionary [DIH1K SH AH0NEH2RIY0]
Wehavementionedanumberofpotentiallevelsof prominence : accented,stressed, Prominence
secondary stress, full vowel, and reduced vowel. It is still an open research question
exactlyhowmanylevelsareappropriate. Veryfewcomputati onalsystemsmakeuseof

all ﬁve of these levels,most usingbetweenone andthree. We r eturnto thisdiscussion
whenweintroduceprosodyinmoredetail inSec.8.3.1.
7.3 PhonologicalCategoriesandPronunciationVariation
’Scuse me, whileI kiss the sky
JimiHendrix, PurpleHaze
’Scuse me, whileI kiss thisguy
Common mis-hearing of samelyrics
If each word was pronounced with a ﬁxed string of phones, each of which was pro-
nouncedthesameinallcontextsandbyallspeakers,thespee chrecognitionandspeech
synthesis tasks would be really easy. Alas, the realization of words and phonesvaries

massivelydependingonmanyfactors. Fig.7.8showsasample ofthewidevariationin
pronunciationin the words becauseandaboutfrom the hand-transcribedSwitchboard
corpusofAmerican Englishtelephoneconversations(Green berget al.,1996).
because about
ARPAbet % ARPAbet % ARPAbet % ARPAbet %
biy kahz 27% ks 2% axbaw 32% bae 3%
bix kahz 14% kixz 2% axbaw t 16% baw t 3%
kahz 7% kihz 2% baw 9% axb awdx 3%
kaxz 5% biyk ahzh 2% ixb aw 8% axb ae 3%
bix kaxz 4% biyk ahs 2% ixb awt 5% baa 3%
bih kahz 3% biyk ah 2% ixb ae 4% bae dx 3%

baxk ahz 3% biyk aaz 2% axbae dx 3% ixb awdx 2%
kuhz 2% axz 2% baw dx 3% ixb aat 2%
Figure7.8 The 16 most common pronunciations of becauseandaboutfrom the hand-
transcribedSwitchboardcorpusofAmericanEnglishconver sational telephone speech (Godfrey
et al.,1992; Greenberget al.,1996).
How can we model and predict this extensive variation? One us eful tool is the
assumption that what is mentally represented in the speaker ’s mind are abstract cate-

DRAFT226 Chapter 7. Phonetics
gories rather than phones in all their gory phonetic detail. For example consider the
different pronunciations of [t] in the words tunaﬁshandstarﬁsh. The [t] of tunaﬁsh
isaspirated . Aspiration is a period of voicelessness after a stop closur e and before
the onset of voicing of the following vowel. Since the vocal c ords are not vibrating,
aspiration sounds like a puff of air after the [t] and before t he vowel. By contrast, a
[t] following an initial [s] is unaspirated ; thus the [t] in starﬁsh([s t aa r f ih sh]) Unaspirated
has no period of voicelessness after the [t] closure. This va riation in the realization of
[t] is predictable: whenever a [t] begins a word or unreduced syllable in English, it is
aspirated. The same variationoccursfor [k]; the [k] of skyis oftenmis-heardas [g] in
Jimi Hendrix’slyricsbecause[k]and[g]are bothunaspirat ed.2
There are other contextual variants of [t]. For example, whe n [t] occurs between

two vowels, particularly when the ﬁrst is stressed, it is oft en pronounced as a tap.
Recall that a tap is a voiced sound in which the top of the tongu e is curled up and
back and struck quicklyagainst the alveolarridge. Thus the wordbuttercup is usually
pronounced [b ah dx axr k uh p] rather than [b ah t axr k uh p]. Ano ther variant of
[t] occurs before the dental consonant [th]. Here the [t] bec omes dentalized (IPA [t ”]).

That is, instead of the tongue forming a closure against the a lveolar ridge, the tongue
touchesthe backoftheteeth.
In both linguisticsand in speech processing, we use abstrac t classes to capture the
similarity amongall these [t]s. The simplest abstract clas s is called the phoneme , and Phoneme
its different surface realizations in different contexts a re calledallophones . We tradi- Allophone
tionally write phonemes inside slashes. So in the above exam ples, /t/is a phoneme
whose allophones include (in IPA) [th],[R], and [t ”]. Fig. 7.3 summarizes a number

of allophones of /t/. In speech synthesis and recognition, w e use phonesets like the
ARPAbettoapproximatethisideaofabstractphonemeunits, andrepresentpronuncia-
tion lexiconsusing ARPAbet phones. For this reason, the all ophoneslisted in Fig. 7.1
tend to be used for narrow transcriptions for analysis purpo ses, and less often used in
speechrecognitionorsynthesissystems.
IPA ARPABet Description Environment Example
th[t] aspirated ininitialposition toucan
t unaspirated after [s]or inreduced syllables starﬁsh

P[q] glottalstop word-ﬁnallyor after vowel before [n] kitten
Pt[qt] glottalstopt sometimes word-ﬁnally cat
R[dx] tap between vowels butter
t^[tcl] unreleased t before consonants or word-ﬁnally fruitcake
t ” dental t before dental consonants ( [T]) eighth
deletedt sometimes word-ﬁnally past
Figure7.9 Some allophones of /t/inGeneral AmericanEnglish.
VariationisevenmorecommonthanFig.7.3suggests. Onefac torinﬂuencingvari-
ationisthatthemorenaturalandcolloquialspeechbecomes ,andthefasterthespeaker

talks,themorethesoundsareshortened,reducedandgenera llyruntogether. Thisphe-
nomena is knownas reduction orhypoarticulation . For example assimilation is the Reduction
Hypoarticulation
Assimilation2TheARPAbetdoesnothaveaway ofmarking aspiration; in theI PAaspiration ismarked as [h],soinIPA
the word tunaﬁshwould betranscribed [thun@fIS].

DRAFTSection7.3. PhonologicalCategoriesandPronunciationVa riation 227
change in a segment to make it more like a neighboring segment . The dentalization
of[t]to ([t ”]) before the dental consonant [T]is an example of assimilation. A com-
mon type of assimilation cross-linguistically is palatalization , when the constriction palatalization
forasegmentmovesclosertothepalatethanitnormallywoul d,becausethefollowing
segment is palatal or alveolo-palatal. In the most commonca ses, /s/ becomes [sh], /z/
becomes[zh],/t/becomes[ch]and/d/becomes[jh],Wesawon ecaseofpalatalization
in Fig. 7.8 in the pronunciation of becauseas [b iy k ah zh], because the following
wordwas you’ve. Thelemma you(you,your,you’ve,andyou’d)isextremelylikelyto
causepalatalizationin theSwitchboardcorpus.
Deletion isquitecommoninEnglishspeech. We sawexamplesofdeletio nofﬁnal Deletion
/t/ above, in the words aboutandit. Deletion of ﬁnal /t/ and /d/ has been extensively
studied. /d/ismorelikelytobedeletedthan /t/,andbotharemorelikelytobedeleted

beforea consonant(Labov,1972). Fig. 7.3 shows examplesof palatalization and ﬁnal
t/ddeletionfromtheSwitchboardcorpus.
Palatalization Finalt/dDeletion
Phrase Lexical Reduced Phrase Lexical Reduced
setyour seh tyow r s ehcher ﬁndhim fay ndhihm f aynixm
not yet naa tyeht naa ch eht and we ae ndw iy ehnw iy
didyou dihd yuw dihjhyah draft the drae ft dhiy dr ae fdh iy
Figure7.10 Examples of palatalization and ﬁnal t/d/ deletion from the S witchboard corpus.
Some of the t/dexamples may have glottalizationinsteadof b eing completelydeleted.
7.3.1 Phonetic Features

The phoneme gives us only a very gross way to model contextual effects. Many of
the phonetic processes like assimilation and deletion are b est modeled by more ﬁne-
grainedarticulatoryfactsabouttheneighboringcontext. Fig.7.3showedthat/t/and/d/
weredeletedbefore[h],[dh],and[w];ratherthanlistallt hepossiblefollowingphones
which could inﬂuence deletion, we’d like to generalize that /t/ often deletes “before
consonants”. Similarly, ﬂapping can be viewed as a kind of vo icing assimilation, in

which unvoiced /t/ becomes a voiced tap [dx] in between voice d vowels or glides.
Rather than list every possible vowel or glide, we’d like to s ay that ﬂapping happens
‘near vowels or voiced segments’. Finally, vowels that prec ede nasal sounds [n], [m],
and [ng], often acquire some of the nasal quality of the follo wing vowel. In each of
thesecases, aphoneisinﬂuencedbythearticulationofthen eighboringphones(nasal,
consonantal, voiced). The reason these changes happen is th at the movement of the

speecharticulators(tongue,lips,velum)duringspeechpr oductioniscontinuousandis
subject to physical constraints like momentum. Thus an arti culator may start moving
during one phone to get into place in time for the next phone. W hen the realization
of a phone is inﬂuenced by the articulatory movementof neigh boringphones, we say
it is inﬂuenced by coarticulation .Coarticulation is the movement of articulators to Coarticulation
anticipatethenextsound,orperseveratingmovementfromt helast sound.

Wecancapturegeneralizationsaboutthedifferentphonest hatcausecoarticulation
by usingdistinctive features . Featuresare (generally)binaryvariableswhich express Distinctivefeature
somegeneralizationsaboutgroupsofphonemes. Forexample thefeature[voice]istrue

DRAFT228 Chapter 7. Phonetics
ofthevoicedsounds(vowels,[n],[v],[b],etc);wesaythey are[+voice]whileunvoiced
sounds are [-voice]. These articulatory features can draw o n the articulatory ideas of
placeandmannerthatwedescribedearlier. Common placefeaturesinclude[+labial]
([p,b,m]),[+coronal]([chd dhjh l nr ssh t th z zh]),and[+do rsal]. Mannerfeatures
include [+consonantal] (or alternatively [+vocalic]), [+ continuant], [+sonorant]. For
vowels,featuresinclude[+high],[+low],[+back],[+roun d]andsoon. Distinctivefea-
turesareusedtorepresenteachphonemeasamatrixoffeatur evalues. Manydifferent
sets of distinctive featuresexist; probablyanyof these ar e perfectlyadequatefor most
computationalpurposes. Fig. 7.11shows the values for some phonesfrom one partial
set offeatures.
syl son cons strident nasal high back round tense voice labia l coronal dorsal
b - - + - - - - + + + + - -
p - - + - - - - - + - + - -
iy + + - - - + - - - + - - -

Figure7.11 Some partial feature matrices for phones, values simpliﬁed from Chomsky and Halle (1968). Sylis
short for syllabic; sonfor sonorant, and consfor consonantal.
Onemainuseofthesedistinctivefeaturesisincapturingna turalarticulatoryclasses
of phones. In both synthesis and recognition, as we will see, we often need to build
models of how a phone behaves in a certain context. But we rare ly have enough data
to model the interactionof everypossible left and rightcon textphoneon the behavior

of a phone. For this reason we can use the relevant feature ([v oice], [nasal], etc) as
a useful model of the context; the feature functions as a kind of backoff model of the
phone. Anotheruse in speechrecognitionis to buildarticul atoryfeaturedetectorsand
use them to help in the task of phone detection; for example Ki rchhoff et al. (2002)
builtneural-netdetectorsforthefollowingsetofmulti-v aluedarticulatoryfeaturesand
usedthemtoimprovethedetectionofphonesin Germanspeech recognition:
Feature Values Feature Value

voicing +voice, -voice, silence manner stop, vowel, lateral, nasal, fricative, silence
cplace labial, coronal, palatal, velar vplace glottal, high, mid, low, silence
front-back front, back, nil, silence rounding +round, -round, nil, silence
7.3.2 Predicting PhoneticVariation
For speech synthesis as well as recognition, we need to be abl e to represent the rela-
tion between the abstract category and its surface appearan ce, and predict the surface
appearance from the abstract category and the context of the utterance. In early work

in phonology,the relationshipbetween a phonemeandits all ophoneswas capturedby
writing a phonological rule . Here is the phonological rule for ﬂapping in the tradi-
tionalnotationofChomskyandHalle (1968):
/{
t
d}/
→[dx] /´VV (7.1)
Inthisnotation,thesurfaceallophoneappearstotheright ofthearrow,andthepho-
neticenvironmentisindicatedbythe symbolssurroundingt heunderbar( ). Simple

DRAFTSection7.3. PhonologicalCategoriesandPronunciationVa riation 229
rulesliketheseareusedinbothspeechrecognitionandsynt hesiswhenwewanttogen-
eratemanypronunciationsforaword;inspeechrecognition thisisoftenusedasaﬁrst
step towardpickingthemostlikelysinglepronunciationfo raword(seeSec. 10.5.3).
Ingeneral,however,therearetworeasonswhythesesimple‘ Chomsky-Halle’-type
rulesdon’tdowellat tellingus whena givensurfacevariantislikelytobeused. First,
variation is a stochastic process; ﬂapping sometimes occur s, and sometimes doesn’t,
eveninthesameenvironment. Second,manyfactorsthataren otrelatedtothephonetic
environmentare importantto this predictiontask. Thus lin guistic researchand speech
recognition/synthesisbothrelyonstatisticaltoolstopr edictthesurfaceformofaword
byshowingwhichfactorscause,e.g.,aparticular/t/ toﬂap ina particularcontext.
7.3.3 FactorsInﬂuencing PhoneticVariation
One important factor that inﬂuences phonetic variation is t herate of speech , gener- Rateof speech

ally measured in syllables per second. Rate of speech varies both across and within
speakers. Manykindsof phoneticreductionprocessesare mu chmorecommonin fast
speech, including ﬂapping, vowel reduction, and ﬁnal /t/ an d /d/ deletion (Wolfram,
1969). Measuringsyllablesper second(orwordsper second) can be donewith a tran-
scription(bycountingthenumberofwordsorsyllablesinth etranscriptionofaregion
anddividingbythenumberofseconds),orbyusingsignal-pr ocessingmetrics(Morgan
andFosler-Lussier,1989). Anotherfactoraffectingvaria tioniswordfrequencyorpre-

dictability. Final/t/ and/d/deletionisparticularlylik elyto happeninwordswhichare
veryfrequentlike andandjust(Labov,1975;Neu,1980). Deletionisalso morelikely
whenthetwo wordssurroundingthesegmentarea collocation (Bybee,2000;Gregory
et al., 1999; Zwicky, 1972). The phone [t] is more likely to be palatalized in frequent
wordsandphrases. Wordswithhigherconditionalprobabili tygiventhepreviousword
are more likely to have reduced vowels, deleted consonants, and ﬂapping (Bell et al.,
2003;Gregoryet al.,1999).

Other phonetic, phonological, and morphological factors a ffect variation as well.
For example /t/ is much more likely to ﬂap than /d/; and there a re complicated inter-
actions with syllable, foot, and word boundaries (Rhodes, 1 992). As we will discuss
in Ch. 8, speech is broken up into units called intonation phrases orbreath groups .
Words at the beginning or end of intonation phrases are longe r and less likely to be
reduced. As for morphology, it turns out that deletion is les s likely if the word-ﬁnal

/t/ or /d/ is the English past tense ending (Guy, 1980). For ex ample in Switchboard,
deletion is more likely in the word around(73% /d/-deletion) than in the word turned
(30%/d/-deletion)eventhoughthe twowordshavesimilar fr equencies.
Variationis also affectedbythe speaker’sstate ofmind. Fo r examplethe word the
canbepronouncedwithafullvowel[dhiy]orreducedvowel[d hax]. Itismorelikely
to be pronounced with the full vowel [iy] when the speaker is d isﬂuent and having
“planning problems”; in general speakers are more likely to use a full vowel than a

reduced one if they don’t know what they are going to say next ( Fox Tree and Clark,
1997;Bell et al.,2003;Keatinget al.,1994).
Sociolinguistic factors like gender, class, and dialectalso affect pronunciation sociolinguistic
Dialect variation. North American English is often divided into eig ht dialect regions (North-
ern, Southern, New England, New York/Mid-Atlantic, North M idlands, South Mid-

DRAFT230 Chapter 7. Phonetics
lands, Western, Canadian). Southern dialect speakers use a monophthong or near-
monophthong [aa] or [ae] instead of a diphthong in some words with the vowel [ay].
In these dialects riceis pronounced[r aa s]. African-American Vernacular EnglishAfrican-American
VernacularEnglish(AAVE)sharesmanyvowelswithSouthernAmericanEnglish,a ndalsohasindividual
words with speciﬁc pronunciationssuch as [b ih d n ih s] for businessand [ae k s] for
ask. For older speakers or those not from the American West or Mid west, the words
caughtandcothavedifferentvowels([kaot]and[kaat]respectively). Yo ungAmeri-
canspeakersorthosefromtheWestpronouncethetwowords cotandcaughtthesame;
the vowels [ao] and [aa] are usually not distinguished in the se dialects except before
[r]. For speakers of most non-American and some American dia lects of English (for
example Australian English), the words Mary([m ey r iy]), marry([m ae r iy]) and
merry([mehriy])areallpronounceddifferently. MostAmericans peakerspronounce

all threeofthese wordsidenticallyas([meh riy]).
Other sociolinguisticdifferencesare due to registerorstyle; a speaker might pro- Register
Stylenounce the same word differently dependingon who they were t alking to or what the
social situationis. Oneof themost well-studiedexampleso fstyle-variationisthe suf-
ﬁx-ing(as insomething ), which can be pronounced [ih ng] or [ih n] (this is often
writtensomethin’ ). Mostspeakersusebothforms;asLabov(1966)shows,theyu se[ih
ng]when theyare beingmoreformal,and [ih n] whenmorecasua l. Wald andShopen

(1981)foundthatmenaremorelikelytousethenon-standard form[ihn]thanwomen,
that both men and women are more likely to use more of the stand ard form [ih ng]
whentheaddresseeisa women,andthat men(butnotwomen)ten dto switchto [ihn]
whentheyaretalkingwithfriends.
Manyoftheseresultsonpredictingvariationrelyonlogist icregressiononphonetically-
transcribedcorpora,atechniquewithalonghistoryinthea nalysisofphoneticvariation
(Cedergren and Sankoff, 1974), particularly using the VARB RUL and GOLDVARB
software(RandandSankoff,1990).

Finally, the detailed acoustic realization of a particular phone is very strongly in-
ﬂuenced by coarticulation with its neighboringphones. We will return to these ﬁne-
grained phonetic details in the following chapters (Sec. 8. 4 and Sec. 10.3) after we
introduceacousticphonetics.
7.4 AcousticPhoneticsandSignals
Wewillbeginwithabriefintroductiontotheacousticwavef ormandhowitisdigitized,
summarizethe idea of frequencyanalysisand spectra. This w ill be an extremelybrief
overview;theinterestedreaderisencouragedtoconsultth ereferencesattheendofthe
chapter.

DRAFTSection7.4. AcousticPhoneticsandSignals 231
7.4.1 Waves
Acousticanalysisisbasedonthesineandcosinefunctions. Fig.7.12showsaplotofa
sine wave,in particularthefunction:
y=A∗sin(2πft) (7.2)
wherewehaveset theamplitudeAto 1andthefrequency fto10cyclespersecond.
Time (s)0 0.5–0.990.99
0
0 0.1 0.2 0.3 0.4 0.5
Figure7.12 A sine wave witha frequency of 10 Hz andanamplitude of 1.
Recall from basic mathematicsthat two importantcharacter istics of a wave are its
frequency andamplitude . Thefrequencyisthenumberoftimesa secondthatawave Frequency
Amplitude repeats itself, i.e. the number of cycles. We usually measure frequency in cycles per
second. The signal in Fig. 7.12 repeats itself 5 times in .5 seconds, hence 10 cycles Cyclespersecond
per second. Cycles per second are usually called Hertz(shortened to Hz), so the Hertz
frequencyin Fig. 7.12 would be described as 10 Hz. The amplitude Aof a sine wave
isthemaximumvalueontheY axis.
TheperiodTofthe wave is deﬁnedasthe time it takes foronecycle to compl ete, Period

deﬁnedas
T=1
f(7.3)
In Fig. 7.12 we can see that each cycle lasts a tenth of a second , henceT=.1
seconds.
7.4.2 Speech Sound Waves
Let’s turn from hypotheticalwaves to sound waves. The input to a speech recognizer,
like the input to the human ear, is a complex series of changes in air pressure. These
changesinairpressureobviouslyoriginatewiththespeake r,andarecausedbythespe-
ciﬁcwaythatairpassesthroughtheglottisandouttheoralo rnasalcavities. Werepre-
sentsoundwavesbyplottingthechangeinairpressureovert ime. Onemetaphorwhich

seconds.
sometimes helps in understandingthese graphs is to imagine a vertical plate which is
blockingtheairpressurewaves(perhapsinamicrophoneinf rontofaspeaker’smouth,
or the eardrum in a hearer’s ear). The graph measures the amou nt ofcompression or
rarefaction (uncompression)oftheairmoleculesatthisplate. Fig.7.1 3showsashort
segmentofa waveformtakenfromthe Switchboardcorpusof te lephonespeechof the
vowel[iy]fromsomeonesaying“shejust hadababy”.

DRAFT232 Chapter 7. Phonetics
Time (s)0 0.03875–0.016970.02283
0
Figure7.13 A waveform of the vowel [iy] from an utterance tobe shown inFi g. 7.17. The y-axis shows the level
of air pressure above and below normal atmospheric pressure . Thex-axis shows time. Notice that the wave repeats
regularly.
Let’s explore how the digital representation of the sound wa ve shown in Fig. 7.13
would be constructed. The ﬁrst step in processingspeech is t o convertthe analog rep-
resentations(ﬁrstairpressure,andthenanalogelectrics ignalsinamicrophone),intoa
digital signal. This process of analog-to-digitalconversion has two steps: sampling Sampling
andquantization . Asignalissampledbymeasuringitsamplitudeataparticul artime;
thesampling rate is the number of samples taken per second. In order to accurat ely Sampling rate
measure a wave, it is necessary to have at least two samples in each cycle: one mea-
suring the positive part of the wave and one measuring the neg ative part. More than

two samples per cycle increases the amplitude accuracy, but less than two samples
will cause the frequency of the wave to be completely missed. Thus the maximum
frequency wave that can be measured is one whose frequency is half the sample rate
(sinceeverycycleneedstwosamples). Thismaximumfrequen cyforagivensampling
rateiscalledthe Nyquistfrequency . Mostinformationinhumanspeechisinfrequen- Nyquist frequency
ciesbelow10,000Hz;thusa20,000Hzsamplingratewouldben ecessaryforcomplete
accuracy. Buttelephonespeechisﬁlteredbytheswitchingn etwork,andonlyfrequen-

cieslessthan4,000Hzaretransmittedbytelephones. Thusa n8,000Hz samplingrate
is sufﬁcient for telephone-bandwidth speech like the Switchboard corpus. A 16,000Telephone
bandwidth
Hz samplingrate(sometimescalled wideband )isoftenusedformicrophonespeech. Wideband
Even an 8,000 Hz sampling rate requires 8000 amplitude measu rements for each
secondofspeech,andsoitisimportanttostoretheamplitud emeasurementefﬁciently.
Theyareusuallystoredasintegers,either8-bit(valuesfr om-128–127)or16bit(values

bandwidth
from -32768–32767). This process of representing real-val ued numbers as integers is
calledquantization becausethereisaminimumgranularity(thequantumsize)an dall quantization
valueswhichare closertogetherthanthisquantumsize arer epresentedidentically.
Once data is quantized, it is stored in various formats. One p arameter of these
formatsis the sample rate and sample size discussed above; t elephone speech is often
sampledat 8kHzandstoredas8-bitsamples,whilemicrophon edataisoftensampled

at 16 kHz and stored as 16-bit samples. Another parameter of t hese formats is the
numberof channels. Forstereodata,orfortwo-partyconversations,wecansto reboth Channel
channels in the same ﬁle, or we can store them in separate ﬁles . A ﬁnal parameter
is whether each sample is stored linearly or whether it is com pressed. One common
compression format used for telephone speech is µ-law (often written u-law but still
pronounced mu-law). The intuition of log compression algor ithms like µ-law is that

humanhearingismoresensitiveat small intensitiesthanla rgeones;the logrepresents
small values with more faithfulnessat the expense of more er ror on large values. The

DRAFTSection7.4. AcousticPhoneticsandSignals 233
linear (unlogged)values are generally referred to as linear PCM values (PCM stands PCM
forPulseCodeModulation,but nevermindthat). Here’sthee quationforcompressing
a linearPCM samplevalue xto8-bitµ-law,(where µ=255for8bits):
F(x) =sgn(s)log(1+µ|s|)
log(1+µ)(7.4)
Thereareanumberofstandardﬁleformatsforstoringtheres ultingdigitizedwave-
ﬁle, such as Microsoft’s WAV, Apple AIFF and Sun AU, all of whi ch have special
headers; simple headerless‘raw’ ﬁles are also used. For exa mple the .wav format is a
subset of Microsoft’s RIFF format for multimedia ﬁles; RIFF is a general format that
canrepresentaseriesofnestedchunksofdataandcontrolin formation. Fig.7.14shows
a simple.wavﬁle witha singledatachunktogetherwithitsfo rmatchunk:
Figure7.14 Microsoft waveﬁle header format, assuming simple ﬁle witho ne chunk. Follow-
ingthis 44-byte header wouldbe the data chunk.
7.4.3 Frequency andAmplitude; Pitch andLoudness

Sound waves, like all waves, can be described in terms of freq uency, amplitude and
theothercharacteristicsthatweintroducedearlierforpu resinewaves. Insoundwaves
these are not quite as simple to measure as they were for sine w aves. Let’s consider
frequency. Notein Fig.7.13thatalthoughnotexactlya sine , thatthe waveis nonethe-
less periodic, and that there are 10 repetitions of the wave i n the 38.75 milliseconds
(.03875 seconds) we have captured in the ﬁgure. Thus the freq uency of this segment

ofthewaveis10/.03875or258Hz.
Where does this periodic 258Hz wave come from? It comes from t he speed of
vibration of the vocal folds; since the waveform in Fig. 7.13 is from the vowel [iy], it
is voiced. Recall that voicing is caused by regular openings and closing of the vocal
folds. When the vocal folds are open, air is pushing up throug h the lungs, creating a
regionofhighpressure. Whenthefoldsareclosed,thereisn opressurefromthelongs.
Thuswhenthevocalfoldsarevibrating,weexpecttoseeregu larpeaksinamplitudeof

thekindweseeinFig.7.13,eachmajorpeakcorrespondingto anopeningofthevocal
folds. Thefrequencyofthevocalfoldvibration,orthefreq uencyofthecomplexwave,
is called the fundamental frequency of the waveform, often abbreviated F0. We canFundamentalfrequency
F0plotF0overtime in a pitch track . Fig. 7.15shows the pitch track of a short question,
Pitchtrack “Threeo’clock?”representedbelowthewaveform. Notether iseinF0attheendofthe
question.
The vertical axis in Fig. 7.13 measures the amount of air pres sure variation; pres-

question.
sure is force per unit area, measuredin Pascals (Pa). A high v alue on the vertical axis
(a high amplitude)indicatesthat there is more air pressure at that pointin time, a zero

DRAFT234 Chapter 7. Phonetics
three o’clock
Time (s)0 0.5443750 Hz500 Hz
Figure7.15 Pitchtrackofthequestion“Threeo’clock?”, shownbelowth ewaveﬁle. Notethe
rise in F0 at the end of the question. Note the lack of pitch tra ce during the very quiet part (the
“o’”of“o’clock”;automaticpitchtrackingisbasedoncoun tingthepulsesinthevoicedregions,
and doesn’t workif there isno voicing(or insufﬁcient sound at all).
value means there is normal (atmospheric)air pressure, whi le a negative value means
thereislowerthannormalair pressure(rarefaction).
In addition to this value of the amplitude at any point in time , we also often need
to know the average amplitude over some time range, to give us some idea of how
great the average displacement of air pressure is. But we can ’t just take the average
of the amplitude values over a range; the positive and negati ve values would (mostly)
cancelout,leavinguswitha numbercloseto zero. Instead,w egenerallyuse theRMS

(root-mean-square)amplitude, which squares each number b efore averaging (making
it positive),andthentakesthesquarerootatthe end.
RMS amplitudeN
i=1=√
N
∑
i=1x2
i
N(7.5)
Thepowerofthe signal is relatedto the squareof the amplitude. If the numberof Power
samplesofa soundis N, thepoweris
Power =1
Nn
∑
i=1x[i]2(7.6)
Rather than power, we more often refer to the intensity of the sound, which nor- Intensity
malizesthe powerto thehumanauditorythreshold,andis mea suredin dB. If P0is the
auditorythresholdpressure=2 ×10−5Pathenintensityisdeﬁnedasfollows:

i=1=√
N
∑
i=1x2
i
N(7.5)
Power =1
Nn
∑
Intensity =10log101
NP0n
∑
i=1x2
i (7.7)
Fig. 7.16 shows an intensity plot for the sentence “Is it a lon g movie?” from the
CallHomecorpus,againshownbelowthewaveformplot.

DRAFTSection7.4. AcousticPhoneticsandSignals 235
is it a long movie?
Time (s)0 1.1675
Figure7.16 Intensity plot for the sentence “Is it a long movie?”. Note th e intensity peaks at
each vowel,and the especiallyhigh peak forthe word long.
Two importantperceptualproperties, pitchandloudness, are related to frequency
and intensity. The pitchof a sound is the mental sensation or perceptual correlate of Pitch
fundamental frequency; in general if a sound has a higher fun damental frequency we
perceiveitashavingahigherpitch. Wesay“ingeneral”beca usetherelationshipisnot
linear, since human hearing has different acuities for diff erent frequencies. Roughly
speaking,humanpitchperceptionismostaccuratebetween1 00Hzand1000Hz,andin
thisrangepitchcorrelateslinearlywithfrequency. Human hearingrepresentsfrequen-
ciesabove1000Hzlessaccuratelyandabovethisrangepitch correlateslogarithmically
with frequency. Logarithmic representation means that the differences between high

frequenciesare compressed, and hence not as accurately per ceived. There are various
psychoacousticmodelsofpitchperceptionscales. Onecomm onmodelisthe melscale Mel
(Stevens et al., 1937; Stevens and Volkmann, 1940). A mel is a unit of pitch deﬁned
so that pairs of soundswhich are perceptuallyequidistant i n pitch are separated by an
equal number of mels. The mel frequency mcan be computed from the raw acoustic
frequencyasfollows:
m=1127ln (1+f
700) (7.8)
We will returnto the mel scale in Ch. 9 when we introducethe MF CC representa-

700) (7.8)
tionofspeechusedin speechrecognition.
Theloudness of a sound is the perceptual correlate of the power. So sounds with
higheramplitudesareperceivedaslouder,butagaintherel ationshipisnotlinear. First
of all, as we mentioned above when we deﬁned µ-law compression, humans have
greater resolution in the low power range; the ear is more sen sitive to small power
differences. Second, it turns out that there is a complex rel ationship between power,
frequency,andperceivedloudness;soundsincertainfrequ encyrangesareperceivedas
beinglouderthanthoseinotherfrequencyranges.

Various algorithms exist for automatically extracting F 0. In a slight abuse of ter-
minologythesearecalled pitchextraction algorithms. Theautocorrelationmethodof Pitchextraction

DRAFT236 Chapter 7. Phonetics
pitch extraction, for example, correlates the signal with i tself, at various offsets. The
offset that gives the highest correlation gives the period o f the signal. Other methods
forpitch extractionarebased onthe cepstral featureswe wi ll returnto in Ch. 9. There
are variouspublicly available pitch extraction toolkits; for example an augmentedau-
tocorrelationpitchtrackerisprovidedwith Praat(BoersmaandWeenink,2005).
7.4.4 Interpreting Phones from a Waveform
Much can be learned from a visual inspection of a waveform. Fo r example, vowels
are pretty easy to spot. Recall that vowels are voiced; anoth er property of vowels is
that they tend to be long, and are relatively loud (as we can se e in the intensity plot
in Fig. 7.16). Length in time manifests itself directly on th e x-axis, while loudness
is related to (the square of) amplitude on the y-axis. We saw i n the previous section
that voicing is realized by regularpeaks in amplitude of the kind we saw in Fig. 7.13,

each major peak corresponding to an opening of the vocal fold s. Fig. 7.17 shows the
waveform of the short phrase ‘she just had a baby’. We have lab eled this waveform
with word and phonelabels. Notice that each ofthe six vowels in Fig. 7.17,[iy], [ax],
[ae],[ax],[ey],[iy],all haveregularamplitudepeaksind icatingvoicing.
she just had a baby
sh iy j ax sh ae dxaxb ey b iy
Time (s)0 1.059
Figure7.17 Awaveformofthesentence “Shejusthadababy” fromtheSwitc hboardcorpus (conversation4325).

The speaker is female, was 20 years old in1991, which is appro ximately when the recording was made, and speaks
the SouthMidlands dialect of AmericanEnglish.
For a stop consonant, which consists of a closure followed by a release, we can
oftenseeaperiodofsilenceornearsilencefollowedbyasli ghtburstofamplitude. We
cansee thisforbothofthe [b]’sin babyinFig. 7.17.
Anotherphonethat is oftenquite recognizablein a waveform is a fricative. Recall
that fricatives, especially very strident fricatives like [sh], are made when a narrow

channel for airﬂow causes noisy, turbulent air. The resulti ng hissy soundshave a very
noisy,irregularwaveform. Thiscanbeseen somewhatin Fig. 7.17;it’s evenclearerin
Fig.7.18,wherewe’vemagniﬁedjustthe ﬁrst word she.
7.4.5 Spectra and theFrequency Domain
While some broad phonetic features (such as energy, pitch, a nd the presence of voic-
ing, stop closures, or fricatives) can be interpreted direc tly from the waveform, most
computationalapplicationssuchasspeechrecognition(as wellashumanauditorypro-

DRAFTSection7.4. AcousticPhoneticsandSignals 237
she
sh iy
Time (s)0 0.257
Figure7.18 A more detailed view of the ﬁrst word “she” extracted from the waveﬁle in Fig. 7.17. Notice the
difference betweenthe random noise of the fricative[sh] an d the regular voicingof the vowel [iy].
cessing)arebasedona differentrepresentationofthe soun dintermsofitscomponent
frequencies. Theinsightof Fourieranalysis is thateverycomplexwave canberepre-
sented as a sum of many sine waves of different frequencies. C onsider the waveform
inFig.7.19. Thiswaveformwascreated(in Praat)bysummingtwosinewaveforms,
oneoffrequency10Hzandoneoffrequency100Hz.
Time (s)0 0.5–11
0
Figure7.19 A waveform created by summing two sine waveforms, one of freq uency 10 Hz
(note the 5 repetitions in the half-second window) and one of frequency 100 Hz, both with
amplitude 1.
Wecanrepresentthesetwocomponentfrequencieswitha spectrum . Thespectrum Spectrum
ofasignalisarepresentationofeachofitsfrequencycompo nentsandtheiramplitudes.

0
Fig.7.20showsthespectrumofFig.7.19. FrequencyinHziso nthex-axisandampli-
tude on the y-axis. Note that there are two spikes in the ﬁgure , one at 10 Hz and one
at100Hz. Thusthespectrumisanalternativerepresentatio noftheoriginalwaveform,
andwe use the spectrumasa toolto study the componentfreque nciesof a soundwave
at aparticulartime point.
Let’slooknowatthefrequencycomponentsofaspeechwavefo rm. Fig.7.21shows
part of the waveform for the vowel [ae] of the word had, cut out from the sentence
shownin Fig.7.17.

Note that there is a complex wave which repeats about ten time s in the ﬁgure; but
there is also a smaller repeated wave which repeats four time s for every larger pattern
(notice the four small peaks inside each repeated wave). The complex wave has a

DRAFT238 Chapter 7. Phonetics
Frequency (Hz)1 10 100 2 20 200 5 50Sound pressure level (dB /Hz)
406080
Figure7.20 The spectrum ofthe waveform inFig.7.19.
Time (s)0 0.04275–0.055540.04968
0
Figure7.21 The waveform of part of the vowel [ae] from the word hadcut out from the
waveform shown inFig.7.17.
frequencyof about 234 Hz (we can ﬁgure this out since it repea ts roughly10 times in
.0427seconds,and10cycles/.0427seconds=234Hz).
Thesmallerwavethenshouldhaveafrequencyofroughlyfour timesthefrequency
ofthelargerwave,orroughly936Hz. Thenifyoulookcareful lyyoucanseetwolittle
waves on the peak of many of the 936 Hz waves. The frequency of t his tiniest wave
mustberoughlytwice thatofthe936Hzwave,hence1872Hz.
Fig.7.22showsa smoothedspectrumforthewaveforminFig. 7 .21,computedvia
a DiscreteFourierTransform(DFT).
Frequency (Hz)0 4000Sound pressure level (dB /Hz)
–20020
0 2000 4000 0 1000 2000 3000 4000
Figure7.22 Aspectrumforthevowel[ae]fromtheword hadinthewaveformof Shejusthad
ababyinFig.7.17.

–20020
Thex-axisofaspectrumshowsfrequencywhilethe y-axisshowssomemeasureof
the magnitude of each frequencycomponent (in decibels (dB) , a logarithmic measure

DRAFTSection7.4. AcousticPhoneticsandSignals 239
of amplitude that we saw earlier). Thus Fig. 7.22 shows that t here are signiﬁcant fre-
quencycomponentsat around930 Hz, 1860 Hz, and 3020 Hz, alon g with many other
lower-magnitudefrequencycomponents. Theseﬁrst two comp onentsarejust whatwe
noticedin thetimedomainbylookingat thewavein Fig.7.21!
Whyisaspectrumuseful? Itturnsoutthatthesespectralpea ksthatareeasilyvisi-
bleinaspectrumareverycharacteristicofdifferentphone s;phoneshavecharacteristic
spectral“signatures”. Justaschemicalelementsgiveoffd ifferentwavelengthsoflight
when they burn, allowing us to detect elements in stars looki ngat the spectrum of the
light,wecandetectthecharacteristicsignatureofthedif ferentphonesbylookingatthe
spectrum of a waveform. This use of spectral information is e ssential to both human
and machine speech recognition. In human audition, the func tion of the cochleaor Cochlea
inner ear is to computea spectrum of the incomingwaveform. Similarly , the various

kinds of acoustic features used in speech recognition as the HMM observation are all
differentrepresentationsofspectralinformation.
Let’s look at the spectrum of different vowels. Since some vo wels change over
time,we’ll use adifferentkindofplot calleda spectrogram . While a spectrumshows
the frequency componentsof a wave at one point in time, a spectrogram is a way of spectrogram
envisioninghow the differentfrequenciesthat make up a wav eformchange overtime.

Thex-axisshowstime,asitdidforthewaveform,butthe y-axisnowshowsfrequencies
in Hertz. The darknessof a point on a spectrogramcorrespond ingto the amplitude of
thefrequencycomponent. Verydarkpointshavehighamplitu de,lightpointshavelow
amplitude. Thus the spectrogram is a useful way of visualizi ng the three dimensions
(timexfrequencyxamplitude).
Fig. 7.23 shows spectrograms of 3 American English vowels, [ ih], [ae], and [ah].
Notethateachvowelhasasetofdarkbarsatvariousfrequenc ybands,slightlydifferent

bandsforeach vowel. Each of these representsthe same kind o f spectral peakthat we
saw inFig.7.21.
Time (s)0 2.8139705000Frequency (Hz)
Figure7.23 Spectrograms for three American English vowels, [ih], [ae] , and [uh], spoken by
the ﬁrstauthor.
Each dark bar (or spectral peak) is called a formant. As we will discuss below, a Formant
formantis a frequencybandthat is particularlyampliﬁedby the vocal tract. Sincedif-
ferentvowelsareproducedwiththevocaltractindifferent positions,theywillproduce

different kinds of ampliﬁcations or resonances. Let’s look at the ﬁrst two formants,
called F1 and F2. Note that F1, the dark bar closest to the bott om is in different posi-
tion for the 3 vowels; it’s low for [ih] (centeredat about 470 Hz)and somewhathigher
for[ae]and[ah](somewherearound800Hz)BycontrastF2,th eseconddarkbarfrom

DRAFT240 Chapter 7. Phonetics
thebottom,ishighestfor[ih],inthemiddlefor[ae],andlo west for[ah].
We cansee the same formantsin runningspeech,althoughthe r eductionandcoar-
ticulation processes make them somewhat harder to see. Fig. 7.24 shows the spectro-
gramof‘shejusthadababy’whosewaveformwasshowninFig.7 .17. F1andF2(and
also F3)areprettyclearforthe[ax]of just, the[ae]of had,andthe [ey]of baby.
sh iy j ax shaedxaxb ey biy
Figure7.24 A spectrogram of the sentence “She just had a baby” whose wave form was shown in Fig. 7.17. We
canthink of aspectrogram is as a collectionof spectra (time -slices) like Fig.7.22placed endtoend. Note
Whatspeciﬁccluescanspectralrepresentationsgiveforph oneidentiﬁcation? First,
since different vowels have their formants at characterist ic places, the spectrum can
be used to distinguish vowels from each other. We’ve seen tha t [ae] in the sample
waveform had formants at 930 Hz, 1860 Hz, and 3020 Hz. Conside r the vowel [iy],

at the beginning of the utterance in Fig. 7.17. The spectrum f or this vowel is shown
in Fig. 7.25. The ﬁrst formantof [iy] is 540 Hz; much lower tha n the ﬁrst formantfor
[ae], while the second formant (2581 Hz) is much higher than t he second formant for
[ae]. If you look carefully you can see these formants as dark bars in Fig. 7.24 just
around0.5seconds.
−1001020304050607080
0 1000 2000 3000
Figure7.25 A smoothed (LPC) spectrum for the vowel [iy] at the start of She just had a
baby. Note thatthe ﬁrstformant (540 Hz)is muchlower thanthe ﬁrs tformant for [ae]shown in

Fig.7.22, while the second formant (2581 Hz) ismuch higher t hanthe second formant for[ae].

DRAFTSection7.5. PhoneticResources 241
The location of the ﬁrst two formants (called F1 and F2) plays a large role in de-
termining vowel identity, although the formants still diff er from speaker to speaker.
Higherformantstendtobecausedmorebygeneralcharacteri sticofthespeakersvocal
tract rather than by individualvowels. Formantsalso can be used to identify the nasal
phones[n],[m],and[ng],andtheliquids[l]and[r].
7.4.6 The Source-Filter Model
Why do different vowels have different spectral signatures ? As we brieﬂy mentioned
above,theformantsarecausedbytheresonantcavitiesofth emouth. The source-ﬁlterSource-ﬁlter
model
model is a way of explaining the acoustics of a sound by modeli ng how the pulses
producedbythe glottis(the source)areshapedbythevocaltract(the ﬁlter).
Let’s see how this works. Whenever we have a wave such as the vi bration in air
causedbytheglottalpulse,thewavealsohas harmonics . Aharmonicisanotherwave Harmonic
whose frequency is a multiple of the fundamental wave. Thus f or example a 115 Hz

model
glottalfoldvibrationleadstoharmonics(otherwaves)of2 30Hz,345Hz,460Hz,and
soonon. Ingeneraleachofthesewaveswillbeweaker,i.e. ha vemuchlessamplitude
thanthewaveat thefundamentalfrequency.
Itturnsout,however,thatthevocaltractactsasakindofﬁl terorampliﬁer;indeed
anycavitysuchasatubecauseswavesofcertainfrequencies tobeampliﬁed,andothers
to be damped. This ampliﬁcationprocessis causedby the shap eof the cavity; a given
shapewillcausesoundsofacertainfrequencytoresonatean dhencebeampliﬁed. Thus

bychangingtheshapeofthecavitywecancausedifferentfre quenciestobeampliﬁed.
Now when we produce particular vowels, we are essentially ch anging the shape
of the vocal tract cavity by placing the tongue and the other a rticulators in particular
positions. Theresultisthatdifferentvowelscausediffer entharmonicstobeampliﬁed.
So a wave of the same fundamentalfrequencypassed throughdi fferentvocal tract
positionswill resultindifferentharmonicsbeingampliﬁe d.
Wecanseetheresultofthisampliﬁcationbylookingatthere lationshipbetweenthe

shapeoftheoraltract andthe correspondingspectrum. Fig. 7.26showsthe vocaltract
position for three vowels and a typical resulting spectrum. The formantsare places in
thespectrumwherethevocaltracthappenstoamplifypartic ularharmonicfrequencies.
7.5 PhoneticResources
A wide variety of phonetic resources can be drawn on for compu tational work. One
key set of resources are pronunciation dictionaries . Such on-line phonetic dictio-Pronunciationdictionary
naries give phonetic transcriptions for each word. Three co mmonly used on-line dic-

tionaries for English are the CELEX, CMUdict, and PRONLEX le xicons; for other
languages,the LDC has releasedpronunciationdictionarie sforEgyptianArabic, Ger-
man, Japanese,Korean, Mandarin,and Spanish. All these dic tionariescan be usedfor
bothspeechrecognitionandsynthesiswork.
TheCELEXdictionary(Baayenetal.,1995)isthemostrichly annotatedofthedic-
tionaries. ItincludesallthewordsintheOxfordAdvancedL earner’sDictionary(1974)

DRAFT242 Chapter 7. Phonetics
[iy]  (tea) [ae] (cat) [uw]  (moo)Frequency (Hz)0 4000Sound pressure level (dB /Hz)
020
268 2416F1
F2
020
903 1695F1 F2
–200
295817F1 F2
Figure7.26 Visualizing the vocal tract position as a ﬁlter: the tongue p ositions for three English vowels and the
resultingsmoothed spectra showing F1and F2. Tongue positi ons modeled after Ladefoged (1996).
(41,000lemmata)andtheLongmanDictionaryofContemporar yEnglish(1978)(53,000
lemmata),intotalithaspronunciationsfor160,595wordfo rms. Its(Britishratherthan
American) pronunciations are transcribed using an ASCII ve rsion of the IPA called
SAM. Inadditionto basicphoneticinformationlike phonest rings,syllabiﬁcation,and
stress levelforeachsyllable,eachwordis also annotatedw ithmorphological,part-of-
speech, syntactic, and frequency information. CELEX (as we ll as CMU and PRON-

LEX) represent three levels of stress: primary stress, seco ndary stress, and no stress.
Forexample,some ofthe CELEX informationforthe word dictionary includesmulti-
ple pronunciations (’dIk-S@n-rI and ’dIk-S@-n@-rI, corre sponding to ARPABET [d
ih k sh ax n r ih] and [d ih k sh ax n ax r ih] respectively), togeth er with the CV-
skelata for each one ([CVC][CVC][CV] and [CVC][CV][CV][CV ]), the frequencyof
theword,thefactthatit isa noun,anditsmorphologicalstr ucture(diction+ary).

ThefreeCMUPronouncingDictionary(CMU,1993)haspronunc iationsforabout
125,000 wordforms. It uses an 39-phone ARPAbet-derived pho neme set. Transcrip-
tions are phonemic, and thus instead of marking any kind of su rface reduction like
ﬂapping or reduced vowels, it marks each vowel with the numbe r 0 (unstressed) 1
(stressed), or2 (secondarystress). Thusthe word tigeris listed as [T AY1 G ER0] the
wordtableas[TEY1BAH0L],andtheword dictionary as[DIH1KSHAH0NEH2

R IY0]. The dictionaryisnot syllabiﬁed,althoughthe nucle usis implicitlymarkedby
the(numbered)vowel.
The PRONLEX dictionary (LDC, 1995) was designed for speech r ecognition and
contains pronunciations for 90,694 wordforms. It covers al l the words used in many
yearsoftheWallStreetJournal,aswellastheSwitchboardC orpus. PRONLEXhasthe
advantage that it includes many proper names (20,000, where CELEX only has about
1000). Names are important for practical applications, and they are both frequentand

difﬁcult;wereturnto adiscussionofderivingnamepronunc iationsinCh. 8.

DRAFTSection7.5. PhoneticResources 243
Anotherusefulresourceisa phoneticallyannotatedcorpus ,inwhichacollection
ofwaveformsishand-labeledwiththe correspondingstring ofphones. Two important
phonetic corpora in English are the TIMIT corpus and the Swit chboard Transcription
Projectcorpus.
The TIMIT corpus (NIST, 1990) was collected as a joint projec t between Texas
Instruments (TI), MIT, and SRI. It is a corpus of 6300 read sen tences, where 10 sen-
tenceseachfrom630speakers. The6300sentencesweredrawn fromasetof2342pre-
designedsentences,someselectedtohaveparticulardiale ctshibboleths,otherstomax-
imize phoneticdiphonecoverage. Eachsentencein the corpu swasphoneticallyhand-
labeled,the sequenceofphoneswas automaticallyalignedw ith the sentencewaveﬁle,
and then the automatic phone boundaries were manually hand- corrected (Seneff and
Zue, 1988). The result is a time-aligned transcription ; a transcription in which eachTime-alignedtranscription

phone in the transcript is associated with a start and end tim e in the waveform; we
showeda graphicalexampleofa time-alignedtranscription inFig.7.17.
ThephonesetforTIMIT,andfortheSwitchboardTranscripti onProjectcorpusbe-
low,isamoredetailedonethantheminimalphonemicversion oftheARPAbet. Inpar-
ticular, these phonetic transcriptionsmake use of the vari ousreduced and rare phones
mentionedin Fig. 7.1 and Fig. 7.2; the ﬂap [dx],glottal stop [q], reducedvowels [ax],

[ix], [axr], voiced allophone of [h] ([hv]), and separate ph ones for stop closure ([dcl],
[tcl],etc)andrelease([d],[t],etc). Anexampletranscri ptionisshowninFig.7.5.
shehad yourdark suitingreasy wash water allyear
shiyhvae dcl jhaxrdcl d aa rkcl s uxqengcl gr iys ix w aa sh qw aa dxaxr q aa lyixaxr
Figure7.27 Phonetic transcription from the TIMIT corpus. Note palatal ization of [d] in had, unreleased ﬁnal
stop indark, glottalization of ﬁnal [t] in suitto [q], and ﬂap of [t] in water. The TIMIT corpus also includes time-

alignments foreach phone (not shown).
Where TIMIT is based on read speech, the more recent Switchbo ard Transcrip-
tion Project corpusis based on the Switchboardcorpusof con versationalspeech. This
phonetically-annotated portion consists of approximatel y 3.5 hours of sentences ex-
tracted from various conversations (Greenberg et al., 1996 ). As with TIMIT, each
annotatedutterancecontainsatime-alignedtranscriptio n. TheSwitchboardtranscripts
are time-aligned at the syllable level rather than at the pho ne level; thus a transcript

consists of a sequence of syllables with the start and end tim e of each syllables in the
correspondingwaveﬁle. Fig. 7.28shows an examplefromthe S witchboardTranscrip-
tionProject,forthephrase they’rekindofinbetweenright now :
0.470 0.640 0.720 0.900 0.953 1.279 1.410 1.630
dher k aa nax vih m bix t w iyn ray naw
Figure7.28 Phonetic transcriptionof the Switchboard phrase they’re kindof inbetween right
now. Notevowelreductionin they’reandof,codadeletionin kindandright,andresyllabiﬁcation

(the [v] of ofattaches as the onset of in). Time is given innumber of seconds from beginning of
sentence tostartof each syllable.
Phonetically transcribed corpora are also available for ot her languages; the Kiel
corpus of German is commonly used, as are various Mandarin co rpora transcribed by
theChineseAcademyofSocial Sciences(Liet al.,2000).

DRAFT244 Chapter 7. Phonetics
In addition to resources like dictionaries and corpora, the re are many useful pho-
netic software tools. One of the most versatile is the free Pr aat package(Boersmaand
Weenink, 2005), which includes spectrum and spectrogram an alysis, pitch extraction
and formantanalysis, and an embeddedscripting languagefo r automation. It is avail-
ableonMicrosoft,Macintosh,andUNIX environments.
7.6 Advanced: ArticulatoryandGesturalPhonology
We saw in Sec. 7.3.1that we coulduse distinctive features to capturegeneralizations
across phone class. These generalizations were mainly arti culatory (although some,
like[strident]andthevowelheightfeatures,areprimaril yacoustic).
This idea that articulation underlies phonetic production is used in a more sophis-
ticated way in articulatory phonology , in which the articulatory gesture is the un-Articulatory
phonology
Gesture derlyingphonologicalabstraction(BrowmanandGoldstein ,1986,1992). Articulatory

phonology
gestures are deﬁned as parameterized dynamical systems . Since speech production
requires the coordinated actions of tongue, lips, glottis, etc, articulatory phonology
represents a speech utterances as a sequence of potentially overlapping articulatory
gestures. Fig. 7.29 shows the sequent of gestures (or gestural score ) required for the Gestural score
productionof the word pawn[p aa n]. The lips ﬁrst close, then the glottis opens, then

the tongue body moves back toward the pharynx wall for the vow el [aa], the velum
dropsfor the nasal sounds, and ﬁnally the tonguetip closes a gainst the alveolar ridge.
Thelinesinthediagramindicategestureswhicharephasedw ithrespecttoeachother.
With such a gestural representation, the nasality in the [aa ] vowel is explained by the
timingofthegestures;thevelumdropsbeforethetonguetip hasquiteclosed.
Figure7.29 Thegesturalscorefortheword pawnaspronounced [paan],afterBrowmanand
Goldstein(1989) and Browmanand Goldstein(1995).

Theintuitionbehindarticulatoryphonologyisthattheges turalscoreislikelytobe
muchbetter as a set of hiddenstates at capturingthe continu ousnature of speechthan
a discrete sequence of phones. In addition, using articulat ory gestures as a basic unit
can helpin modelingthe ﬁne-grainedeffectsofcoarticulat ionofneighboringgestures

DRAFTSection7.7. Summary 245
that we will explore further when we introduce diphones (Sec. 8.4) and triphones
(Sec.10.3).
Computational implementationsof articulatory phonology have recently appeared
in speech recognition,using articulatorygesturesrather than phonesasthe underlying
representation or hidden variable. Since multiple articul ators (tongue, lips, etc) can
movesimultaneously,usinggesturesasthe hiddenvariable impliesa multi-tierhidden
representation. Fig. 7.6 shows the articulatory feature se t used in the work of Livescu
and Glass (2004b) and Livescu (2005); Fig. 7.6 shows example s of how phones are
mappedontothisfeatureset.
Feature Description Value =meaning
LIP-LOC position of lips LAB =labial (neutral position); PR O =protruded (rounded); DEN= dental
LIP-OPEN degreeofopeningof
lipsCL =closed; CR =critical (labial/labio-dental fricative) ; NA =narrow (e.g.,[w],[uw]);
WI =wide(all other sounds)
TT-LOC location of tongue

tipDEN=inter-dental ([th], [dh]); ALV=alveolar ([t], [n]); P -A=palato-alveolar ([sh]);
RET=retroﬂex ([r])
TT-OPEN degreeofopeningof
tongue tipCL =closed (stop); CR =critical (fricative); NA =narrow ([r ],alveolar glide);
M-N =medium-narrow;MID =medium;WI=wide
TB-LOC location of tongue
bodyPAL=palatal (e.g. [sh],[y]); VEL=velar (e.g.,[k], [ng]);
UVU =uvular (neutral position); PHA=pharyngeal (e.g. [aa] )
TB-OPEN degreeofopeningof

tongue bodyCL =closed (stop); CR =critical (e.g. fricated [g] in ”legal ”); NA =narrow (e.g. [y]);
M-N =medium-narrow; MID =medium; WI=wide
VEL state of the velum CL =closed (non-nasal); OP=open (nasa l)
GLOT state of the glottis CL =closed (glottal stop); CR =crit ical (voiced); OP=open (voiceless)
Figure7.30 Articulatory-phonology-based feature setfrom Livescu (2 005).
phone LIP-LOC LIP-OPEN TT-LOC TT-OPEN TB-LOC TB-OPEN VEL GL OT
aa LAB W ALV W PHA M-N CL(.9),OP(.1) CR
ae LAB W ALV W VEL W CL(.9),OP(.1) CR

b LAB CR ALV M UVU W CL CR
f DEN CR ALV M VEL M CL OP
n LAB W ALV CL UVU M OP CR
s LAB W ALV CR UVU M CL OP
uw PRO N P-A W VEL N CL(.9),OP(.1) CR
Figure7.31 Livescu(2005): sampleofmappingfromphonestounderyling targetarticulatoryfeaturevalues. Note
that some values are probabilistic.
7.7 Summary
This chapter has introduced many of the important concepts o f phonetics and compu-
tationalphonetics.
•We can represent the pronunciation of words in terms of units calledphones.
The standard system for representing phonesis the International Phonetic Al-

phabetorIPA. The most common computational system for transcription of
Englishisthe ARPAbet ,whichconvenientlyusesASCII symbols.

DRAFT246 Chapter 7. Phonetics
•Phones can be described by how they are produced articulatorily by the vocal
organs;consonantsaredeﬁnedintermsoftheir placeandmannerofarticulation
andvoicing,vowelsbytheir height,backness , androundness .
•Aphoneme isageneralizationorabstractionoverdifferentphonetic realizations.
Allophonicrules expresshowa phonemeisrealizedin agivencontext.
•Speech sounds can also be described acoustically . Sound waves can be de-
scribed in terms of frequency ,amplitude , or their perceptual correlates, pitch
andloudness.
•Thespectrum of a sound describes its different frequency components. Wh ile
somephoneticpropertiesarerecognizablefromthewavefor m,bothhumansand
machinesrelyonspectralanalysisforphonedetection.
•Aspectrogram isa plotofa spectrumovertime. Vowelsaredescribedbychar -
acteristicharmonicscalled formants .
•Pronunciationdictionaries arewidelyavailable,andusedforbothspeechrecog-
nitionandspeechsynthesis,includingtheCMUdictionaryf orEnglishandCELEX

dictionaries for English, German, and Dutch. Other diction aries are available
fromtheLDC.
•Phoneticallytranscribedcorporaareausefulresourcefor buildingcomputational
modelsofphonevariationandreductioninnaturalspeech.
The major insights of articulatory phonetics date to the lin guists of 800–150 B.C. In-
dia. They invented the concepts of place and manner of articu lation, worked out the
glottal mechanism of voicing, and understood the concept of assimilation. European

sciencedidnotcatchupwiththeIndianphoneticiansuntilo ver2000yearslater,inthe
late 19thcentury. The Greeksdid have some rudimentaryphon eticknowledge;by the
time of Plato’s Theaetetus andCratylus, for example, they distinguished vowels from
consonants, and stop consonants from continuants. The Stoi cs developed the idea of
thesyllableandwereawareofphonotacticconstraintsonpo ssiblewords. Anunknown
Icelandicscholarofthetwelfthcenturyexploitedtheconc eptofthephoneme,proposed

a phonemic writing system for Icelandic, including diacrit ics for length and nasality.
But histext remainedunpublisheduntil 1818andeventhenwa s largelyunknownout-
side Scandinavia (Robins, 1967). The modern era of phonetic sis usually said to have
begun with Sweet, who proposed what is essentially the phone me in his Handbook
of Phonetics (1877). He also devised an alphabet for transcription and di stinguished
betweenbroadandnarrowtranscription, proposing many ideas that were eventually

incorporatedintothe IPA. Sweet wasconsideredthe best pra cticingphoneticianofhis
time; he made the ﬁrst scientiﬁc recordings of languages for phonetic purposes, and
advancedthe state of the art of articulatory description. H e was also infamouslydifﬁ-
culttogetalongwith,atraitthatiswellcapturedinHenryH iggins,thestagecharacter

DRAFTSection7.7. Summary 247
that George Bernard Shaw modeled after him. The phoneme was ﬁ rst named by the
PolishscholarBaudouindeCourtenay,whopublishedhisthe oriesin 1894.
Studentswithfurtherinterestintranscriptionandarticu latoryphoneticsshouldcon-
sult an introductoryphoneticstextbooksuch as Ladefoged( 1993)or Clark and Yallop
(1995). Pullum and Ladusaw (1996)is a comprehensiveguide t o each of the symbols
and diacritics of the IPA. A good resource for details about r eduction and other pho-
netic processes in spoken English is Shockey (2003). Wells ( 1982) is the deﬁnitive
3-volumesourceondialectsofEnglish.
Many of the classic insights in acoustic phonetics had been d eveloped by the late
1950’sorearly1960’s;justafewhighlightsincludetechni queslikethesoundspectro-
graph(Koenigetal.,1946),theoreticalinsightslikethew orkingoutofthesource-ﬁlter
theoryandotherissuesinthemappingbetweenarticulation andacoustics(Fant,1960;
Stevens et al., 1953; Stevens and House, 1955; Heinz and Stev ens, 1961; Stevens and

House, 1961), the F1xF2 space of vowel formants Peterson and Barney (1952), the
understandingof the phonetic nature of stress and the use of duration and intensity as
cues (Fry, 1955), and a basic understanding of issues in phon e perception Miller and
Nicely (1955),Libermanet al. (1952). Lehiste (1967)is a co llection of classic papers
onacousticphonetics. ManyoftheseminalpapersofGunnarF anthavebeencollected
inFant(2004).
ExcellenttextbooksonacousticphoneticsincludeJohnson (2003)and(Ladefoged,
1996). (Coleman,2005)includesanintroductiontocomputa tionalprocessingofacous-

tics as well as other speech processing issues, from a lingui stic perspective. (Stevens,
1998) lays out an inﬂuential theory of speech sound producti on. There are a wide va-
riety of booksthat address speech from a signal processing a nd electrical engineering
perspective. Theoneswiththegreatestcoverageofcomputa tionalphoneticsissuesin-
clude(Huangetal.,2001),(O’Shaughnessy,2000),and(Gol dandMorgan,1999). An
excellenttextbookondigitalsignalprocessingisLyons(2 004).

Thereare a numberof software packagesfor acoustic phoneti canalysis. Probably
themostwidely-usedoneisPraat(BoersmaandWeenink,2005 ).
Many phoneticspapers of computationalinterest are to be fo undin the Journal of
theAcousticalSocietyofAmerica(JASA) ,ComputerSpeechandLanguage ,andSpeech
Communication .
Exercises
7.1Findthemistakesin theARPAbet transcriptionsofthe follo wingwords:
a.“three”[dhr i] d.“study”[st uhd i] g.“slight”[sliy t]
b.“sing”[sih ng] e.“though”[thow]
c.“eyes”[ays] f.“planning”[pplaa nihng]

DRAFT248 Chapter 7. Phonetics
7.2Translate the pronunciationsof the following color words f rom the IPA into the
ARPAbet (and make a note if you think you pronounce them diffe rently than
this!):
a.[rEd] e.[blæk] i.[pjus]
b.[blu] f.[waIt] j.[toUp]
c.[grin] g.["OrIndZ]
d.["jEloU] h.["pÇpl"]
7.3Ira Gershwin’s lyric for Let’s Call the Whole Thing Off talks about two pronun-
ciationsoftheword“either”(inadditiontothetomatoandp otatoexamplegiven
atthebeginningofthechapter. TranscribeIraGershwin’st wopronunciationsof
“either”inthe ARPAbet.
7.4Transcribethe followingwordsinthe ARPAbet:
a. dark
b. suit
c. greasy
d. wash
e. water
7.5Take a waveﬁle of your choice. Some examples are on the textbo ok website.
Downloadthe PRAATsoftware,anduseittotranscribethewaveﬁlesattheword
level, and into ARPAbet phones, using Praat to help you play p ieces of each
wavﬁle,andtolookat thewaveﬁle andthespectrogram.
7.6Record yourself saying ﬁve of the English vowels: [aa], [eh] , [ae], [iy], [uw].
FindF1andF2foreachofyourvowels.

Chapter8
Speech Synthesis
And computers are getting smarter all the time: Scientists t ell us that soon they will be
able to talk to us. (By ‘they’ I mean ‘computers’: I doubt scie ntists will ever be able to
talktous.)
Dave Barry
In Vienna in 1769, Wolfgang von Kempelen built for the Empres s Maria Theresa the
famousMechanicalTurk,achess-playingautomatonconsist ingofawoodenboxﬁlled
withgears,andarobotmannequinsittingbehindtheboxwhop layedchessbymoving
pieceswithhismechanicalarm. TheTurktouredEuropeandth eAmericasfordecades,
defeating Napolean Bonaparte and even playing Charles Babb age. The Mechanical

Dave Barry
Turk mighthave been one of the early successes of artiﬁcial i ntelligenceif it were not
forthefactthatitwas,alas,ahoax,poweredbyahumanchess playerhiddeninsidethe
box.
What is perhapsless well-knownis that von Kempelen, an extr aordinarilyproliﬁc
inventor, also built between 1769 and 1790 what is deﬁnitely not a hoax: the ﬁrst
full-sentence speech synthesizer. His device consisted of a bellows to simulate the
lungs, a rubber mouthpiece and a nose aperature, a reed to sim ulate the vocal folds,

box.
various whistles for each of the fricatives. and a small auxi liary bellows to provide
the puff of air for plosives. By moving levers with both hands , opening and closing
various openings, and adjusting the ﬂexible leather ‘vocal tract’, different consonants
andvowelscouldbeproduced.
More than two centuries later, we no longer build our speech s ynthesizers out of
wood, leather, and rubber, nor do we need trained human opera tors. The modern task

ofspeechsynthesis ,alsocalled text-to-speech orTTS, istoproducespeech(acoustic Speechsynthesis
Text-to-speech
TTSwaveforms)fromtextinput.
Modernspeechsynthesishasawidevarietyofapplications. Synthesizersareused,
together with speech recognizers, in telephone-based conv ersational agents that con-
duct dialogues with people (see Ch. 23). Synthesizer are als o important in non-
conversationalapplications that speak topeople, such as in devices that read out loud
fortheblind,orinvideogamesorchildren’stoys. Finally, speechsynthesiscanbeused

tospeakforsufferersofneurologicaldisorders,suchasastrophysici stStevenHawking
who,havinglost theuseofhisvoiceduetoALS, speaksbytypi ngtoaspeechsynthe-
sizerandhavingthesynthesizerspeakoutthewords. Stateo ftheartsystemsinspeech
synthesiscanachieveremarkablynaturalspeechfora veryw idevarietyofinputsitua-
tions, althougheventhe best systemsstill tendto soundwoo denand are limited in the
voicestheyuse.
Thetask ofspeechsynthesisistomapatext likethefollowin g:
(8.1)PG&E will file schedules on April 20.
toa waveformlikethefollowing:

DRAFT250 Chapter 8. SpeechSynthesis
Speech synthesis systems perform this mapping in two steps, ﬁrst converting the
input text into a phonemic internal representation and then converting this internal
representationintoawaveform. We willcalltheﬁrst step textanalysis andthesecond Text analysis
stepwaveformsynthesis (althoughothernamesarealsousedforthesesteps).Waveform
synthesis
Asampleoftheinternalrepresentationforthissentenceis showninFig. 8.1. Note
that the acronym PG&Eis expanded into the words P G AND E, the number 20is
expandedinto twentieth, a phonesequenceis givenfor each of the words, and there is
also prosodicandphrasinginformation(the*’s)whichwe wi ll deﬁnelater.
* * * L-L%
PGANDEWILL FILE SCHEDULES ON APRIL TWENTIETH
piyjhiyaendiywihlfaylskehjhaxlzaaneyprihltwehntiyaxth
Figure8.1 Intermediate output for a unit selection synthesizer for th e sentence PG&E willﬁle schedules on April
20.. The numbers andacronyms have beenexpanded, words havebee n converted intophones, andprosodic features

* * * L-L%
have been assigned.
Whiletextanalysisalgorithmsarerelativelystandard,th erearethreewidelydiffer-
ent paradigms for waveform synthesis: concatenative synthesis ,formant synthesis ,
andarticulatory synthesis . The architecture of most modern commercial TTS sys-
tems is based on concatenativesynthesis, in which sampleso f speech are choppedup,
storedinadatabase,andcombinedandreconﬁguredtocreate newsentences. Thuswe
will focusonconcatenativesynthesisformostofthischapt er,althoughwewill brieﬂy

introduceformantandarticulatorysynthesisat theendoft hechapter.
Fig.8.2showstheTTSarchitectureforconcatenativeunits electionsynthesis,using
the two-step hourglass metaphor of Taylor (2008). In the following sections, we’llHourglass
metaphor
examineeachofthecomponentsinthisarchitecture.
8.1 TextNormalization
In order to generate a phonemic internal representation,ra w text ﬁrst needs to be pre-
processed or normalized in a variety of ways. We’ll need to break the input text into textnormalization

metaphor
sentences,anddealwiththeidiosyncraciesofabbreviatio ns,numbers,andsoon. Con-
sider the difﬁculties in the following text drawn from the En ron corpus (Klimt and
Yang,2004):
He said the increase incredit limits helped B.C. Hydro achie ve record net income
of about $1 billion during the year ending March 31. This ﬁgur e does not include
any write-downs that may occur if Powerex determines that an y of its customer
accounts are not collectible. Cousins, however, was insist ent that all debts will

be collected: “We continue to pursue monies owing and we expe ct to be paid for
electricitywe have sold.”

DRAFTSection8.1. TextNormalization 251
e
 Analysis
Waveform 
SynthesisPG&E will file schedules on April 20. 
p iy jh iy ae n d ...Text Normalization
Phonetic Analysis
Prosodic Analysis
Unit
Database    
  
Unit 
Selection
    Phonemic
Internal
 Represenation
Figure8.2 Architecture for the unit selection (concatenative) archi tecture for speech synthe-
sis.
The ﬁrst task in text normalization is sentence tokenization . In order to segmentSentencetokenization
this paragraph into separate utterances for synthesis, we n eed to know that the ﬁrst
sentence ends at the period after March 31 , not at the period of B.C.. We also need
to know that there is a sentence ending at the word collected, despite the punctuation
beingacolonratherthanaperiod. Thesecondnormalization taskisdealingwith non-
standard words . Non-standard words include number, acronyms, abbreviati ons, and
so on. For example, March 31 needs to be pronounced March thirty-ﬁrst , notMarch

threeone;$1billionneedstobepronounced onebilliondollars ,withtheword dollars
appearingaftertheword billion.
8.1.1 Sentence Tokenization
We saw two examplesabovewheresentencetokenizationisdif ﬁcultbecausesentence
boundaries are not always indicated by periods, and can some times be indicated by
punctuation like colons. An additional problem occurs when an abbreviation ends a
sentence,in whichcase theabbreviation-ﬁnalperiodispla yinga dualrole:

(8.2)Hesaidthe increase increditlimitshelped B.C.Hydro achie ve recordnet income of
about $1 billionduringthe year ending March31.
(8.3)Cousins, however, was insistentthat all debts willbe colle cted: “Wecontinue topursue
monies owing andwe expect tobe paidfor electricitywe have s old.”
(8.4)Thegroup included Dr. J.M. Freemanand T.Boone Pickens Jr.
A key part of sentence tokenization is thus period disambigu ation; we’ve seen a
simple perl script for period disambiguation in Ch. 3. Most s entence tokenization al-

gorithmsareslightlymorecomplexthanthisdeterministic algorithm,andinparticular

DRAFT252 Chapter 8. SpeechSynthesis
are trained by machine learning methods rather than being ha nd-built. We do this by
hand-labeling a training set with sentence boundaries, and then using any supervised
machinelearningmethod(decisiontrees,logisticregress ion,SVM,etc)totrainaclas-
siﬁer tomarkthesentenceboundarydecisions.
Morespeciﬁcally,we couldstart bytokenizingthe inputtex tinto tokensseparated
bywhitespace,andthenselectanytokencontainingoneofth ethreecharacters !,.or
?(or possibly also :). After hand-labeling a corpus of such tokens, then we train a
classiﬁer to make a binary decision (EOS (end-of-sentence) versusnot-EOS) on these
potentialsentenceboundarycharactersinside thesetoken s.
The success of such a classiﬁer depends on the features that a re extracted for the
classiﬁcation. Let’s consider some feature templates we mi ght use to disambiguate
thesecandidate sentence boundary characters, assuming we have a small amou nt of
trainingdata,labeledforsentenceboundaries:

•thepreﬁx(theportionofthecandidatetokenprecedingthec andidate)
•thesufﬁx(theportionofthecandidatetokenfollowingthec andidate)
•whetherthepreﬁxorsufﬁxisanabbreviation(fromalist)
•thewordprecedingthe candidate
•thewordfollowingthecandidate
•whetherthewordprecedingthecandidateisanabbreviation
•whetherthewordfollowingthecandidateisanabbreviation
Considerthefollowingexample:
(8.5)ANLPCorp. chairmanDr. Smithresigned.
Given these feature templates, the feature values for the pe riod.in the word Corp.
in(8.5)wouldbe:
PreviousWord = ANLP NextWord = chairman
Prefix = Corp Suffix = NULL

PreviousWordAbbreviation = 1 NextWordAbbreviation = 0
If our training set is large enough, we can also look for lexic al cues about sen-
tence boundaries. For example, certain words may tend to occ ur sentence-initially, or
sentence-ﬁnally. We canthusaddthefollowingfeatures:
•Probability[candidateoccursatendofsentence]
•Probability[wordfollowingcandidateoccursatbeginning ofsentence]
Finally, while most of the above features are relatively lan guage-independent,we
can use language-speciﬁc features. For example, in English , sentences usually begin

withcapitalletters, suggestingfeatureslike thefollowi ng:
•caseofcandidate: Upper,Lower,AllCap,Numbers
•caseofwordfollowingcandidate: Upper,Lower,AllCap,Num bers
Similary, we can have speciﬁc subclasses of abbreviations, such as honoriﬁcs or
titles (e.g., Dr., Mr., Gen.), corporate designators (e.g. , Corp., Inc.), or month-names
(e.g.,Jan.,Feb.).
Any machine learning method can be applied to train EOS class iﬁers. Logistic
regression and decision trees are two very common methods; l ogistic regression may

DRAFTSection8.1. TextNormalization 253
have somewhat higher accuracy, although we have instead sho wn an example of a
decision tree in Fig. 8.3 because it is easier for the reader t o see how the features are
used.
Figure8.3 A decision tree for predicting whether a period ’.’ is an end o f sentence (YES) or
not anend-of-sentence (NO),usingfeatureslike theloglik elihood ofthecurrent wordbeingthe
beginningofasentence( bprob),thepreviouswordbeinganendofsentence( eprob),thecap-
italizationofthenextword,andtheabbreviationsubclass (company,state,unitofmeasurement).
Afterslides by RichardSproat.
8.1.2 Non-Standard Words
The second step in text normalization is normalizing non-standard words . Non-Non-standardwords
standard words are tokens like numbers or abbreviations, wh ich need to be expanded
intosequencesofEnglishwordsbeforetheycanbepronounce d.
What is difﬁcult about these non-standard words is that they are often very am-
biguous. Forexample,the number 1750can bespokenin at least threedifferentways,

dependingonthe context:
seventeen fifty: (in‘TheEuropeaneconomyin1750’ )
one seven five zero: (in‘Thepasswordis1750’ )
seventeen hundred and fifty: (in‘1750dollars’ )
one thousand, seven hundred, and fifty: (in‘1750dollars’ )
Similar ambiguitiesoccurfor Roman numeralslike IV, (whichcan be pronounced
four,fourth, or as the letters I V(meaning ‘intravenous’)),or 2/3, which can be
two thirds orFebruary third ortwo slash three .
Inadditiontonumbers,variousnon-standardwordsarecomp osedofletters. Three

types non-standard words include abbreviations ,letter sequences , andacronyms .
Abbreviationsaregenerallypronouncedby expanding them;thus Jan1ispronounced
January first ,andWedispronounced Wednesday .Lettersequences likeUN,
DVD, PC, andIBMare pronouncedby pronouncingeach letter in a sequence (IBM is
thus pronounced ay b iy eh m ).Acronyms likeIKEA, MoMA, NASA , andUNICEF
are pronounced as if they were words; MoMAis pronounced m ow m ax . Ambiguity

DRAFT254 Chapter 8. SpeechSynthesis
occurs here as well; should Janbe read as a word (the name Jan) or expandedas the
monthJanuary ?
These different types of numeric and alphabetic non-standa rd words can be sum-
marized in Fig. 8.1.2. Each of the types has a particular real ization (or realizations).
For example, a year NYERis generally read in the pairedmethod, in which each pair Paireddigits
of digits is pronounced as an integer (e.g., seventeen fifty for 1750), while a
U.S.zipcode NZIPisgenerallyreadinthe serialmethod,asasequenceofsingledigits Serial digits
(e.g.,nine four one one zero for94110). The type BMONEY deals with the
idiosyncracies of expressions like $3.2 billion , which must be read out with the word
dollars at theend,as three point two billion dollars .
For the alphabetic NSWs, we have the class EXPN for abbreviat ions like N.Y.
whichareexpanded,LSEQ foracronymspronouncedasletters equences,andASWD
foracronymspronouncedasif theywerewords.
ALPHAEXPN abbreviation adv, N.Y.,mph, gov’t

LSEQ lettersequence DVD,D.C.,PC,UN,IBM,
ASWD readas word IKEA,unknown words/namesNUMBERSNUM number (cardinal) 12, 45, 1/2, 0.6
NORD number (ordinal) May 7, 3rd, BillGates III
NTEL telephone (or part of) 212-555-4523
NDIG number as digits Room 101
NIDE identiﬁer 747, 386, I5,pc110, 3A
NADDR number as streetaddress 747, 386, I5,pc110, 3A
NZIP zipcode or POBox 91020
NTIME a (compound) time 3.20, 11:45
NDATE a (compound) date 2/28/05, 28/02/05
NYER year(s) 1998, 80s, 1900s, 2008
MONEY money (USor other) $3.45, HK$300, Y20,200, $200K
BMONEY money tr/m/billions $3.45billion

PRCT percentage 75% 3.4%
Figure8.4 Some types of non-standard words in text normalization, sel ected from Table 1 of
Sproatetal.(2001);notlistedaretypesforURLs,emails,a ndsomecomplexusesofpunctuation.
Dealingwithnon-standardwordsrequiresatleastthreeste ps:tokenization tosep-
arate out and identify potential non-standard words, classiﬁcation to label them with
a type from Fig. 8.1.2, and expansion to convert each type into a string of standard
words.
Inthetokenizationstep,wecantokenizetheinputbywhites pace,andthenassume

words.
that any word which is not in the pronunciation dictionary is a non-standard word.
More sophisticated tokenization algorithms would also dea l with the fact that some
dictionaries already contain some abbreviations. The CMU d ictionary, for example,
contains abbreviated (and hence incorrect) pronunciation s forst, mr, mrs , as well as
day and month abbreviationslike mon, tues, nov, dec , etc. Thus in addition to unseen
words, we also need to label any of these acronymsand also sin gle-charactertoken as

potential non-standardwords. Tokenizationalgorithmsal so needto split wordswhich
are combinations of two tokens, like 2-carorRVing. Words can be split by simple
heuristics,suchassplittingatdashes,orat changesfroml ower-casetoupper-case.

DRAFTSection8.1. TextNormalization 255
The next step is assigning a NSW type; many types can be detect ed with simple
regular expressions. For example, NYERcould be detected by the following regular
expression:
/(1[89][0-9][0-9])|(20[0-9][0-9]/
Other classes might be harder to write rules for, and so a more powerful option is
touse amachinelearningclassiﬁer withmanyfeatures.
Todistinguishbetweenthealphabetic ASWD,LSEQandEXPNclasses, forexample
wemightwantfeaturesoverthecomponentletters. Thusshor t,all-capitalwords( IBM,
US) might be LSEQ, longer all-lowercase words with a single-qu ote (gov’t, cap’n )
might be EXPN, and all-capital words with multiple vowels ( NASA, IKEA ) might be
morelikelyto be ASWD.
Another very useful features is the identity of neighboring words. Consider am-
biguousstringslike 3/4,whichcanbean NDATEmarch third oranumthree-fourths .
NDATEmight be preceded by the word on, followed by the word of, or have the word
Mondaysomewhere in the surrounding words. By contrast, NUMexamples might be

preceded by another number, or followed by words like mileandinch. Similarly, Ro-
man numerals like VIItend to be NORD(seven) when preceded by Chapter,part, or
Act,butNUM(seventh)whenthewords kingorPopeoccurintheneighborhood.These
contextwordscanbechosenasfeaturesbyhand,orcanbelear nedbymachinelearning
techniqueslikethe decisionlist algorithmofCh.8.
We can achieve the most power by building a single machine lea rning classiﬁer
which combines all of the above ideas. For example, the NSW cl assiﬁer of (Sproat

et al., 2001) uses 136 features, including letter-based fea tures like ‘all-upper-case’ ,
‘has-two-vowels’ ,‘contains-slash’ , and‘token-length’ , as well as binary features for
the presence of certain words like Chapter,on, orkingin the surrounding context.
Sproatet al. (2001)alsoincludedarough-draftrule-based classiﬁer, whichusedhand-
written regular expression to classify many of the number NS Ws. The output of this
rough-draftclassiﬁer wasusedasjustanotherfeaturein th emainclassiﬁer.
In order to build such a main classiﬁer, we need a hand-labele d training set, in

which each token has been labeled with its NSW category; one s uch hand-labeled
data-base was producedby Sproat et al. (2001). Given such a l abeled training set, we
canuseanysupervisedmachinelearningalgorithmtobuildt heclassiﬁer.
Formally,wecanmodelthistaskasthegoalofproducingthet agsequence Twhich
ismostprobablegiventheobservationsequence:
T∗=argmax
TP(T|O) (8.6)
One way to estimate this probability is via decision trees. F or example, for each
observed token oi, and for each possible NSW tag tj, the decision tree produces the

T∗=argmax
posterior probability P(tj|oi). If we make the incorrect but simplifying assumption
that each tagging decision is independentof its neighbors, we can predict the best tag
sequence ˆT=argmax TP(T|O)usingthetree:
ˆT=argmax
TP(T|O)

DRAFT256 Chapter 8. SpeechSynthesis
≈m
∏
i=1argmax
tP(t|oi) (8.7)
The third step in dealing with NSWs is expansioninto ordinar ywords. One NSW
type,EXPN,isquitedifﬁcultto expand. Thesearetheabbreviationsan dacronymslike
NY. Generally these must be expanded by using an abbreviation d ictionary, with any
ambiguities dealt with by the homonym disambiguation algor ithms discussed in the
nextsection.
Expansion of the other NSW types is generally deterministic . Many expansions
are trivial; for example, LSEQexpands to a sequence of words, one for each letter,
ASWDexpandstoitself, NUMexpandstoasequenceofwordsrepresentingthecardinal
number, NORDexpands to a sequence of words representing the ordinal numb er, and
NDIGandNZIPbothexpandto asequenceofwords,oneforeachdigit.
Othertypesareslightlymorecomplex; NYERexpandstotwopairsofdigits,unless
the year endsin 00, in which case the fouryearsare pronouncedas a cardinal num ber
(2000astwo thousand ) or in the hundreds method (e.g., 1800 as eighteen Hundreds digits

hundred ).NTELcan be expandedjust as a sequence of digits; alternatively, the last
four digits can be read as paired digits , in which each pair is read as an integer. It is
alsopossibletoreadtheminaformknownas trailingunit ,inwhichthedigitsareread Trailingunitdigits
serially until the last nonzero digit, which is pronounced f ollowed by the appropriate
unit(e.g., 876-5000 aseight seven six five thousand ). Theexpansionof
NDATE,MONEY,andNTIMEisleftasexercises(1)-(4)forthereader.
Of course many of these expansions are dialect-speciﬁc. In A ustralian English,

the sequence 33in a telephone number is generally read double three . Other
languages also present additional difﬁculties in non-stan dard word normalization. In
French or German, for example, in addition to the above issue s, normalization may
depend on morphological properties. In French, the phrase 1 ﬁlle(‘one girl’) is nor-
malized to une fille , but1 garc ¸on (‘one boy’) is normalized to un garcc ¸on .
Similarly,inGerman HeinrichIV (‘HenryIV’)canbenormalizedto Heinrich der

Vierte,Heinrich des Vierten ,Heinrich dem Vierten ,orHeinrich
den Vierten dependingonthegrammaticalcase ofthenoun(Demberg,2006 ).
8.1.3 HomographDisambiguation
The goal of our NSW algorithms in the previous section was to d etermine which se-
quence of standard words to pronounce for each NSW. But somet imes determining
how to pronounceevenstandard wordsis difﬁcult. This is par ticularlytrue for homo-
graphs,whicharewordswiththesamespellingbutdifferentpronun ciations. Hereare Homograph

someexamplesoftheEnglishhomographs use,live,andbass:
(8.8)It’snouse (/yuws/) toaskto use (/yuwz/) thetelephone.
(8.9)Doyoulive (/l ihv/)nearazoowithlive (/l ayv/)animals?
(8.10)Ipreferbass (/bae s/)ﬁshingto playingthe bass (/beys/) guitar.
French homographs include ﬁls(which has two pronunciations [ﬁs] ‘son’ versus
[ﬁl]‘thread]),orthemultiplepronunciationsfor ﬁer(‘proud’or‘totrust’),and est(‘is’
or‘East’)(DivayandVitale, 1997).

DRAFTSection8.2. PhoneticAnalysis 257
Luckilyforthetaskofhomographdisambiguation,thetwofo rmsofhomographsin
English(aswellasinsimilarlanguageslikeFrenchandGerm an)tendtohavedifferent
partsofspeech.Forexample,the two formsof useaboveare (respectively)a nounand
averb,whilethetwoformsof liveare(respectively)averbandanoun. Fig.8.5shows
some interesting systematic relations between the pronunc iation of some noun-verb
andadj-verbhomographs.
Finalvoicing Stress shift -ate ﬁnalvowel
N(/s/)V(/z/) N(init. stress) V(ﬁn. stress) N/A(ﬁnal/ax/) V(ﬁnal/ey/)
useyuw s yuw z recordr eh1kaxr0 d rix0 kao1r d estimate eh st ihm axt ehs tihm eyt
closekl ow s klow z insultih1ns ax0 lt ix0ns ah1l t separate s ehpax raxt s ehpaxr eyt
househaw s hawz objectaa1 bjeh0 kt ax0b jeh1 kt moderate m aa dax raxt m aa daxr eyt
Figure8.5 Some systematic relationships between homographs: ﬁnal co nsonant (noun /s/ versus verb /z/), stress
shift (noun initialversus verbﬁnal stress),and ﬁnalvowel weakening in -atenoun/adjs.

Indeed, Liberman and Church (1992) showed that many of the mo st frequent ho-
mographsin 44millionwordsofAP newswirearedisambiguata blejust byusingpart-
of-speech(themost frequent15homographsin orderare: use,increase,close,record,
house,contract,lead,live,lives,protest,survey,project,separate,present,read).
Thus because knowledge of part-of-speechis sufﬁcient to di sambiguate many ho-
mographs, in practice we perform homographdisambiguation by storing distinct pro-
nunciations for these homographslabeled by part-of-speec h, and then running a part-

of-speechtaggertochoosethe pronunciationfora givenhom ographin context.
There are a number of homographs,however, where both pronun ciationshave the
same part-of-speech. We saw two pronunciations for bass(ﬁsh versus instrument)
above. Other examples of these include lead(because there are two noun pronuncia-
tions, /l iy d/ (a leash orrestraint) and/l ehd/ (a metal)). W e can also thinkof the task
of disambiguatingcertain abbreviations(mentionedearly as NSW disambiguation)as

homograph disambiguation. For example, Dr.is ambiguous between doctor and
drive, andSt.between Saintorstreet. Finally, there are some words that dif-
fer in capitalizations like polish/Polish, which are homographs only in situations like
sentencebeginningsorall-capitalizedtext.
In practice, these latter classes of homographsthat cannot be resolved using part-
of-speech are often ignored in TTS systems. Alternatively, we can attempt to resolve
themusingthewordsensedisambiguationalgorithmsthatwe willintroduceinCh.20,

likethedecision-list algorithmofYarowsky(1997).
8.2 PhoneticAnalysis
The next stage in synthesis is to take the normalized word str ings from text analysis
and produce a pronunciationfor each word. The most importan t component here is a
large pronunciation dictionary. Dictionaries alone turn o ut to be insufﬁcient, because
running text always contains words that don’t appear in the d ictionary. For example

DRAFT258 Chapter 8. SpeechSynthesis
Black et al. (1998) used a British English dictionary, the OA LD lexicon on the ﬁrst
section of the Penn Wall Street JournalTreebank. Of the 3992 3words(tokens)in this
section, 1775 word tokens (4.6%) were not in the dictionary, of which 943 are unique
(i.e. 943types). Thedistributionsoftheseunseenwordtok enswasasfollows:
namesunknown typosandother
1360351 64
76.6%19.8% 3.6%
Thusthetwomainareaswheredictionariesneedtobeaugment edisindealingwith
names and with other unknown words. We’ll discuss dictionar ies in the next section,
followedbynames,andthenturntographeme-to-phonemerul esfordealingwithother
8.2.1 Dictionary Lookup
PhoneticdictionarieswereintroducedinSec.7.5ofCh.8. O neofthemostwidely-used
forTTSisthefreelyavailableCMUPronouncingDictionary( CMU,1993),whichhas
pronunciations for about 120,000 words. The pronunciation s are roughly phonemic,
from a 39-phoneARPAbet-derived phoneme set. Phonemic tran scriptions means that

instead of marking surface reductions like the reduced vowe ls [ax] or [ix], CMUdict
markseachvowelwithastresstag,0(unstressed),1(stress ed),or2(secondarystress).
Thus (non-diphthong)vowels with 0 stress generally corres pondto [ax] or [ix]. Most
wordshaveonlyasinglepronunciation,butabout8,000ofth ewordshavetwooreven
three pronunciations, and so some kinds of phonetic reducti ons are marked in these
pronunciations. The dictionary is not syllabiﬁed, althoug h the nucleus is implicitly

markedbythe(numbered)vowel. Fig.8.2.1showssomesample pronunciations.
ANTECEDENTS AE2N TIH0SIY1D AH0NT S PAKISTANI PAE2K IH0S TAE1NIY0
CHANG CHAE1NG TABLE TEY1B AH0L
DICTIONARY DIH1K SHAH0N EH2R IY0 TROTSKY TR AA1TSK IY2
DINNER DIH1N ER0 WALTER WAO1L TER0
LUNCH LAH1N CH WALTZING WAO1L TSIH0NG
MCFARLAND MAH0KF AA1R LAH0N D WALTZING(2) WAO1L SIH0NG
Figure8.6 Some sample pronunciations from the CMU Pronouncing Dictio nary.
The CMU dictionary was designed for speech recognition rath er than synthesis

uses;thusitdoesnotspecifywhichofthemultiplepronunci ationstouseforsynthesis,
doesnotmarksyllableboundaries,andbecauseitcapitaliz esthedictionaryheadwords,
doesnot distinguish betweene.g., USandus(the form UShas the two pronunciations
[AH1S]and[YUW1 EH1S].
The 110,000word UNISYN dictionary, freely available for re search purposes, re-
solves many of these issues as it was designed speciﬁcally fo r synthesis (Fitt, 2002).
UNISYN gives syllabiﬁcations, stress, and some morphologi cal boundaries. Further-

more, pronunciationsin UNISYN can also be read off in any of d ozens of dialects of
English, includingGeneralAmerican, RP British, Australi a, and so on. The UNISYN
usesaslightlydifferentphoneset; herearesomeexamples:

DRAFTSection8.2. PhoneticAnalysis 259
going: { g*ou }.> i ng >
antecedents: {*a n . tˆ i . s ˜ ii . d n! t }> s >
dictionary: { d*i k . sh @ . n ˜ e . r ii }
8.2.2 Names
As the error analysis above indicated, names are an importan t issue in speech synthe-
sis. Themanytypescanbecategorizedintopersonalnames(ﬁ rstnamesandsurnames),
geographicalnames(city,street,andotherplacenames),a ndcommercialnames(com-
panyandproductnames). For personalnamesalone, Spiegel( 2003)givesan estimate
from Donnelly and other household lists of about two million different surnames and
100,000ﬁrst names just for the United States. Two million is a very large number; an
order of magnitude more than the entire size of the CMU dictio nary. For this reason,
most large-scale TTS systems include a large name pronuncia tion dictionary. As we
sawinFig.8.2.1theCMUdictionaryitselfcontainsawideva rietyofnames;inpartic-
ular it includes the pronunciationsof the most frequent 50, 000 surnames from an old

Bell LabestimateofUS personalnamefrequency,aswellas6, 000ﬁrst names.
How many names are sufﬁcient? Liberman and Church (1992) fou nd that a dic-
tionary of 50,000 names covered 70% of the name tokens in 44 mi llion words of AP
newswire. Interestingly,many of the remaining names (up to 97.43%of the tokens in
their corpus) could be accounted for by simple modiﬁcations of these 50,000 names.
Forexample,somenamepronunciationscanbecreatedbyaddi ngsimplestress-neutral

sufﬁxeslike sorvilleto namesinthe50,000,producingnewnamesasfollows:
walters = walter+s lucasville = lucas+ville abelson = abel+ son
Otherpronunciationsmightbecreatedbyrhymeanalogy. Ifw ehavethepronunci-
ationforthename Trotsky,butnotthename Plotsky,wecanreplacetheinitial/tr/from
Trotskywithinitial /pl/toderivea pronunciationfor Plotsky.
Techniques such as this, including morphological decompos ition, analogical for-
mation,andmappingunseennamestospellingvariantsalrea dyinthedictionary(Fack-

rell and Skut, 2004), have achieved some success in name pron unciation. In general,
however, name pronunciation is still difﬁcult. Many modern systems deal with un-
knownnamesviathegrapheme-to-phonememethodsdescribed inthenextsection,of-
tenbybuildingtwopredictivesystems,onefornamesandone fornon-names. Spiegel
(2003,2002)summarizesmanymoreissuesin propernamepron unciation.
8.2.3 Grapheme-to-Phoneme
Once we have expanded non-standard words and looked them all up in a pronuncia-

tion dictionary, we need to pronounce the remaining, unknow n words. The process
of converting a sequence of letters into a sequence of phones is calledgrapheme-to-
phoneme conversion, sometimes shortened g2p. The job of a grapheme-to-phonemeGrapheme-to-
phoneme
algorithmisthustoconvertaletterstringlike cakeintoaphonestringlike [K EY K] .

DRAFT260 Chapter 8. SpeechSynthesis
Theearliestalgorithmsforgrapheme-to-phonemeconversi onwereruleswrittenby
hand using the Chomsky-Halle phonological rewrite rule for mat of Eq. 7.1 in Ch. 7.
These are often called letter-to-sound or LTS rules, and they are still used in some Letter-to-sound
systems. LTS rules are applied in order, with later (default ) rules only applying if the
context for earlier rules are not applicable. A simple pair o f rulesfor pronouncingthe
lettercmightbeasfollows:
c→[k] /{a,o}V ; context-dependent (8.11)
c→[s] ; context-independent (8.12)
Actual rules must be much more complicated (for example ccan also be pro-
nounced [ch] in celloorconcerto). Even more complex are rules for assigning stress,
which are famously difﬁcult for English. Consider just one o f the many stress rules
fromAllenet al. (1987),wherethesymbol Xrepresentsall possiblesyllableonsets:
(8.13)V→[+stress] / XC*{VshortCC?|V}{VshortC*|V}
Thisrulerepresentsthefollowingtwo situations:

1. Assign1-stresstothevowelinasyllableprecedingaweak syllablefollowedbyamorpheme-
ﬁnalsyllable containing a short vowel and 0or more consonan ts (e.g.difﬁcult)
2. Assign1-stresstothevowelinasyllableprecedingaweak syllablefollowedbyamorpheme-
ﬁnalvowel (e.g. oregano)
While some modern systems still use such complex hand-writt en rules, most sys-
temsachievehigheraccuracybyrelyinginsteadonautomati corsemi-automaticmeth-
odsbasedonmachinelearning. Thismodernprobabilisticgr apheme-to-phonemeprob-

lem was ﬁrst formalized by Lucassen and Mercer (1984). Given a letter sequence L,
we aresearchingforthemost probablephonesequence P:
ˆP=argmax
PP(P|L) (8.14)
The probabilistic method assumes a training set and a test se t; both sets are lists of
words from a dictionary, with a spelling and a pronunciation for each word. The next
subsections show how the popular decision tree model for estimating this probability
P(P|L)canbetrainedandappliedto producethepronunciationfora n unseenword.

ˆP=argmax
Finding aletter-to-phonealignmentforthe trainingset
Most letter-to-phone algorithms assume that we have an alignment , which tells us
which phones align with each letter. We’ll need this alignme nt for each word in the
training set. Some letters might align to multiple phones(e .g.,xoften aligns to k s),
while other letters might align with no phonesat all, like th e ﬁnal letter of cakein the
followingalignment:
L:cake
||||
P:KEYKǫ

DRAFTSection8.2. PhoneticAnalysis 261
One method for ﬁnding such a letter-to-phone alignment is th e semi-automatic
method of (Black et al., 1998). Their algorithm is semi-auto matic because it relies
on a hand-written list of the allowable phones that can realize each letter. Here are
allowableslists fortheletters cande:
c:k ch s sh t-s ǫ
e:ih iy er ax ah eh ey uw ay ow y-uw oy aa ǫ
In order to produce an alignment for each word in the training set, we take this
allowables list for all the letters, and for each word in the t raining set, we ﬁnd all
alignmentsbetween the pronunciationand the spelling that conformto the allowables
list. From this large list of alignments, we compute, by summ ing over all alignments
for all words, the total count for each letter being aligned t o each phone (or multi-
phoneor ǫ). Fromthese countswe cannormalizeto get foreachphone piandletter lj
a probability P(pi|lj):
P(pi|lj) =count(pi,lj)
count(lj)(8.15)

We can now take these probabilities and realign the letters t o the phones, using
the Viterbi algorithm to produce the best (Viterbi) alignme nt for each word, where
the probability of each alignment is just the product of all t he individual phone/letter
alignments.
Inthiswaywecanproduceasinglegoodalignment Aforeachparticularpair (P,L)
inourtrainingset.
Choosingthebest phone stringforthetestset
Given a new word w, we now need to map its letters into a phone string. To do this,

we’llﬁrsttrainamachinelearningclassiﬁer,likeadecisi ontree,onthealignedtraining
set. Thejoboftheclassiﬁerwillbetolookataletterofthew ordandgeneratethemost
probablephone.
Whatfeaturesshouldweuseinthisdecisiontreebesidesthe alignedletter liitself?
Obviously we can do a better job of predicting the phone if we l ook at a window
of surrounding letters; for example consider the letter a. In the word cat, theais
pronounce AE. But in our word cake,ais pronounced EY, because cakehas a ﬁnal e;

thus knowing whether there is a ﬁnal eis a useful feature. Typically we look at the k
previouslettersandthe kfollowingletters.
Anotherusefulfeaturewouldbethecorrectidentityof thep reviousphone. Know-
ingthiswouldallowustogetsomephonotacticinformationi ntoourprobabilitymodel.
Of course, we can’t know the true identity of the previouspho ne, but we can approxi-
matethisbylookingatthepreviousphonethatwaspredicted byourmodel. Inorderto
dothis,we’ll needto runourdecisiontreeleft toright,gen eratingphonesonebyone.

Insummary,inthemostcommondecisiontreemodel,theproba bilityofeachphone
piisestimatedfromawindowof kpreviousand kfollowingletters,aswellasthemost
recentkphonesthatwerepreviouslyproduced.
Fig. 8.7 shows a sketch of this left-to-right process, indic ating the features that a
decision tree would use to decide the letter corresponding t o the letter sin the word
Jurafsky. As this ﬁgure indicates, we can integrate stress predictio n into phone pre-
diction by augmenting our set of phones with stress informat ion. We can do this by

DRAFT262 Chapter 8. SpeechSynthesis
having two copies of each vowel (e.g., AEandAE1), or possibly even the three levels
ofstress AE0,AE1,andAE2,thatwe sawintheCMU lexicon. We’ll also wanttoadd
otherfeaturesintothedecisiontree,includingthepart-o f-speechtagoftheword(most
part-of-speechtaggersprovideanestimate ofthepart-of- speechtagevenforunknown
words)andfactssuchaswhetherthepreviousvowelwasstres sed.
In addition, grapheme-to-phonemedecision trees can also i nclude other more so-
phisticatedfeatures. Forexample,wecanuseclassesoflet ters(correspondingroughly
toconsonants,vowels,liquids,andsoon). Inaddition,for somelanguages,weneedto
knowfeaturesaboutthefollowingword. ForexampleFrenchh asaphenomenoncalled
liaison, in whichthe realizationof the ﬁnal phoneof some wordsdepe ndson whether Liaison
there is a next word, and whether it starts with a consonant or a vowel. For example
the French word sixcan be pronounced [sis] (in j’en veux six ‘I want six’), [siz] ( six

enfants‘sixchildren’),[si]( six ﬁlles‘six girls’).
Finally,mostsynthesissystemsbuildtwoseparategraphem e-to-phonemedecision
trees, one for unknown personal names and one for other unkno wn words. For pro-
nouncing personal names it turns out to be helpful to use addi tional features that in-
dicate which foreign language the names originally come fro m. Such features could
be the output of a foreign-languageclassiﬁer based on lette r sequences (different lan-
guageshavecharacteristicletter N-gramsequences).
##Jurafsky##5

6/uni005F.64AXRAE1F?g2p 
Classifier
ali-3li-2li-1
pi-3pi-2pi-1
LANG=Russian
POS=NNPlili+1li+2li+3
Figure8.7 The process of converting graphemes to phonemes, showing th e left-to-right pro-
cess making a decision for the letter s. The features used by the decision tree are shown inblue.
Wehave shown thecontext window k=3; inreal TTSsystems the window sizeis likelytobe 5
or even larger.
The decision tree is a conditional classiﬁer, computing the phoneme string that

has the highest conditional probability given the grapheme sequence. More recent
grapheme-to-phonemeconversionmakes use of a joint classi ﬁer, in which the hidden
state is a combination of phone and grapheme called a graphone ; see the end of the
chapterforreferences.

DRAFTSection8.3. ProsodicAnalysis 263
8.3 ProsodicAnalysis
The ﬁnal stage of linguistic analysisis prosodicanalysis. In poetry,the word prosody Prosody
refers to the study of the metrical structure of verse. In lin guistics and language pro-
cessing, however, we use the term prosodyto mean the study of the intonational and
rhythmic aspects of language. More technically, prosody ha s been deﬁned by Ladd
(1996)asthe‘useofsuprasegmentalfeaturestoconveysent ence-levelpragmaticmean-
ings’. The term suprasegmental meansaboveand beyondthe level of the segmentor Suprasegmental
phone, and refers especially to the uses of acoustic feature s likeF0 duration , and
energyindependentlyofthephonestring.
Bysentence-level pragmatic meaning , Ladd is referring to a number of kinds
of meaning that have to do with the relation between a sentenc e and its discourse
or external context. For example, prosody can be used to mark discourse structure

or function , like the difference between statements and questions, or t he way that a
conversation is structured into segments or subdialogs. Pr osody is also used to mark
saliency,suchasindicatingthata particularwordorphraseisimpor tantorsalient. Fi-
nally,prosodyis heavilyusedforaffectiveandemotionalm eaning,suchasexpressing
happiness,surprise,oranger.
Inthenextsectionswewillintroducethethreeaspectsofpr osody,eachofwhichis
important for speech synthesis: prosodic prominence ,prosodic structure andtune.

Prosodicanalysisgenerallyproceedsintwoparts. First,w ecomputeanabstractrepre-
sentationof the prosodicprominence,structureandtune of the text. Forunit selection
synthesis, this is all we need to do in the text analysis compo nent. For diphone and
HMM synthesis,we haveonefurtherstep, whichisto predict duration andF0values
fromthese prosodicstructures.
8.3.1 ProsodicStructure
Spoken sentenceshave prosodic structure in the sense that s ome words seem to group

naturally together and some words seem to have a noticeable b reak or disjuncture be-
tween them. Often prosodic structure is described in terms o fprosodic phrasing , Prosodic Phrasing
meaningthat an utterancehasa prosodicphrasestructurein a similar way to it having
asyntacticphrasestructure. Forexample,inthesentence IwantedtogotoLondon,but
couldonlygetticketsforFrance thereseemstobetwomain intonationphrases ,their Intonationphrase
boundary occurring at the comma. Furthermore, in the ﬁrst ph rase, there seems to be

another set of lesser prosodic phrase boundaries (often cal ledintermediate phrase s)intermediatephrase
thatsplit upthewordsasfollows Iwanted|to go|to London .
Prosodicphrasinghasmanyimplicationsforspeechsynthes is; the ﬁnal vowelofa
phraseislongerthan usual,we ofteninsert a pauseafteran i ntonationphrases,and,as
wewilldiscussinSec.8.3.6,thereisoftenaslightdropinF 0fromthebeginningofan
intonationphrasetoitsend,whichresetsat thebeginningo fanewintonationphrase.
Practical phrase boundaryprediction is generally treated as a binary classiﬁcation

task,wherewearegivenawordandwehavetodecidewhetheror nottoputaprosodic
boundaryafterit. Asimplemodelforboundarypredictionca nbebasedondeterminis-
ticrules. Averyhigh-precisionruleistheonewesawforsen tencesegmentation: insert

DRAFT264 Chapter 8. SpeechSynthesis
a boundaryafter punctuation. Another commonlyused rule in serts a phrase boundary
beforeafunctionwordfollowinga contentword.
More sophisticated models are based on machine learning cla ssiﬁers. To create
a training set for classiﬁers, we ﬁrst choose a corpus, and th en mark every prosodic
boundaries in the corpus. One way to do this prosodic boundar y labeling is to use
an intonational model like ToBI or Tilt (see Sec. 8.3.4), hav e human labelers listen to
speechandlabelthetranscriptwiththeboundaryeventsdeﬁ nedbythetheory. Because
prosodic labeling is extremely time-consuming,however,a text-only alternative is of-
ten used. In this method,a humanlabelerlooksonly at the tex t of the trainingcorpus,
ignoringthe speech. The labeler marksany juncture between words where they feel a
prosodicboundarymightlegitimatelyoccurifthe utteranc ewerespoken.
Given a labeled training corpus, we can train a decision tree or other classiﬁer to

make a binary (boundary vs. no boundary) decision at every ju ncture between words
(WangandHirschberg,1992;OstendorfandVeilleux,1994;T aylorandBlack,1998).
Featuresthatarecommonlyusedinclassiﬁcationinclude:
•Length features : phrases tend to be of roughly equal length, and so we can
use various feature that hint at phrase length (Bachenko and Fitzpatrick, 1990;
Grosjeanet al.,1979;GeeandGrosjean,1983).
–Thetotalnumberofwordsandsyllablesinutterance
–Thedistanceofthejuncturefromthebeginningandendofthe sentence(in
wordsorsyllables)

–Thedistanceinwordsfromthelast punctuationmark
•Neighboringpart-of-speechandpunctuation :
–The part-of-speechtags for a window of words around the junc ture. Gen-
erallythetwo wordsbeforeandafterthejunctureareused.
–Thetypeoffollowingpunctuation
There is also a correlationbetween prosodicstructure and t hesyntactic structure
that will be introducedin Ch. 12, Ch. 13, and Ch. 14 (Price et a l., 1991). Thus robust
parsers like Collins (1997) can be used to label the sentence with rough syntactic in-

formation,fromwhichwe canextractsyntacticfeaturessuc hasthe size ofthebiggest
syntactic phrase that endswith this word (Ostendorfand Vei lleux, 1994; Koehnet al.,
2000).
8.3.2 Prosodicprominence
In any spoken utterance, some words sound more prominent than others. Prominent Prominence
wordsareperceptuallymoresalient tothelistener; speake rsmakeawordmoresalient
in English by saying it louder, saying it slower (so it has a lo nger duration), or by
varyingF0 duringtheword,makingithigherormorevariable .

2000).
We generally capture the core notion of prominence by associ ating a linguistic
marker with prominentwords, a marker called pitch accent . Words which are promi- Pitchaccent
nentaresaidto bear(beassociatedwith)apitchaccent. Pitchaccentisthuspar tofthe
phonologicaldescriptionofa wordincontextin aspokenutt erance.
Pitchaccentisrelatedto stress,whichwediscussedinCh.7. Thestressedsyllable
of a word is where pitch accent is realized. In other words, if a speaker decides to

DRAFTSection8.3. ProsodicAnalysis 265
highlight a word by giving it a pitch accent, the accent will a ppear on the stressed
syllableoftheword.
The following example shows accented words in capital lette rs, with the stressed
syllablebearingtheaccent(thelouder,longer,syllable) inboldface:
(8.16)I’malittle SUR PRISED tohearit CHARACTERIZEDasUP BEAT.
Notethatthefunctionwordstendnottobearpitchaccent,wh ilemostofthecontent
wordsareaccented. Thisisaspecialcaseofthemoregeneral factthatveryinformative
words (content words, and especially those that are new or un expected) tend to bear
accent(Ladd,1996;Bolinger,1972).
We’ve talked so far as if we only need to make a binary distinct ion between ac-
cented and unaccented words. In fact we generally need to mak e more ﬁne-grained
distinctions. For example the last accent in a phrase genera lly is perceived as being
more prominent than the other accents. This prominent last a ccent is called the nu-

clear accent . Emphatic accents like nuclear accent are generally used fo r semantic Nuclearaccent
purposes, for example to indicate that a word is the semantic focus of the sentence
(see Ch. 21) or that a word is contrastive or otherwise import ant in some way. Such
emphatic words are the kind that are often written IN CAPITAL LETTERS or with
**STARS**aroundthem in SMS or email or Alice in Wonderland ;here’san example
fromthelatter:
(8.17)‘IknowSOMETHINGinterestingissureto happen,’shesaid to herself,

Anotherway that accent can be more complexthan just binaryi s that some words
can belessprominentthan usual. We introducedin Ch. 7 the idea that fun ctionwords
areoftenphoneticallyvery reduced.
Aﬁnalcomplicationisthataccentscandifferaccordingtot hetuneassociatedwith
them; for example accents with particularly high pitch have different functions than
those with particularly low pitch; we’ll see how this is mode led in the ToBI model in
Sec.8.3.4.

Ignoringtune for the moment, we can summarize by saying that speech synthesis
systemscanuseasmanyasfourlevelsofprominence: emphaticaccent ,pitchaccent ,
unaccented , andreduced. In practice, however,manyimplementedsystems makedo
witha subsetofonlytwoorthreeoftheselevels.
Let’s see how a 2-level system would work. With two-levels, p itch accent predic-
tion is a binary classiﬁcation task, where we are given a word and we have to decide
whetherit isaccentedornot.
Since content words are very often accented, and function wo rds are very rarely

accented, the simplest accent prediction system is just to a ccent all content wordsand
nofunctionwords. Inmostcasesbettermodelsarenecessary .
In principle accent prediction requires sophisticated sem antic knowledge, for ex-
ample to understand if a word is new or old in the discourse, wh ether it is being used
contrastively,andhowmuchnewinformationawordcontains . Earlymodelsmadeuse
of sophisticated linguistic models of all of this informati on (Hirschberg, 1993). But

Hirschberg and others showed better prediction by using sim ple, robust features that
correlatewith thesesophisticatedsemantics.
For example, the fact that new or unpredictable information tends to be accented
canbemodeledbyusingrobustfeatureslike N-gramsorTF*IDF(PanandHirschberg,

DRAFT266 Chapter 8. SpeechSynthesis
2000; Pan and McKeown, 1999). The unigram probability of a wo rdP(wi)and its
bigram probability P(wi|wi−1), both correlate with accent; the more probable a word,
thelesslikelyitistobeaccented. Similarly,aninformati on-retrievalmeasureknownas
TF*IDF(Term-Frequency/Inverse-DocumentFrequency;seeCh.23) isausefulaccent TF*IDF
predictor. TF*IDFcapturesthesemanticimportanceofawor dinaparticulardocument
d, by downgrading words that tend to appear in lots of differen t documents in some
large background corpus with Ndocuments. There are various versions of TF*IDF;
one versioncan be expressed formallyas follows, assuming Nwis the frequencyof w
inthedocument d,andkisthetotalnumberofdocumentsinthecorpusthatcontain w:
TF*IDF (w) =Nw×log(N
k) (8.18)
For words which have been seen enough times in a training set, theaccent ratio Accentratio
feature can be used, which models a word’s individual probab ility of being accented.

k) (8.18)
The accent ratio of a word is equal to the estimated probabili ty of the word being ac-
centedifthisprobabilityissigniﬁcantlydifferentfrom0 .5,andequalto0.5otherwise.
Moreformally,
AccentRatio (w) ={
k
NifB(k,N,0.5)≤0.05
0.5 otherwise
whereNis the total number of times the word woccurred in the training set, kis the
number of times it was accented, and B(k,n,0.5)is the probability (under a binomial
distribution)thatthereare ksuccessesin ntrialsiftheprobabilityofsuccessandfailure
isequal(Nenkovaetal., 2007;Yuanet al.,2005).

k
Featureslike part-of-speech, N-grams, TF*IDF, and accent ratio can then be com-
bined in a decision tree to predict accents. While these robu st featureswork relatively
well,a numberofproblemsinaccentpredictionstill remain the subjectofresearch.
For example, it is difﬁcult to predict which of the two words s hould be accented
in adjective-noun or noun-noun compounds. Some regulariti es do exist; for example
adjective-nouncombinationslike newtruck arelikelytohaveaccentontherightword
(new TRUCK ), while noun-nouncompoundslike TREE surgeon are likely to have ac-

centontheleft. Butthemanyexceptionstotheserulesmakea ccentpredictioninnoun
compounds quite complex. For example the noun-noun compoun dAPPLE cake has
the accent on the ﬁrst word while the noun-noun compound apple PIE orcity HALL
both have the accent on the second word (Liberman and Sproat, 1992; Sproat, 1994,
1998a).
Another complication has to do with rhythm; in general speak ers avoid putting
accents too close together (a phenomenon known as clash) or too far apart ( lapse). Clash

1998a).
LapseThuscityHALL andPARKINGlot combineas CITYhallPARKINGlot withtheaccent
onHALLshifting forward to CITYto avoid the clash with the accent on PARKING
(LibermanandPrince,1977),
Some of these rhythmic constraints can be modeled by using ma chine learning
techniques that are more appropriate for sequence modeling . This can be done by
runninga decision tree classiﬁer left to right through a sen tence, and using the output
of the previous word as a feature, or by using more sophistica ted machine learning

modelslikeConditionalRandomFields(CRFs) (GregoryandA ltun,2004).

DRAFTSection8.3. ProsodicAnalysis 267
8.3.3 Tune
Twoutteranceswiththesameprominenceandphrasingpatter nscanstilldifferprosod-
ically by having different tunes. Thetuneof an utterance is the rise and fall of its Tune
F0 over time. A very obvious example of tune is the difference between statements
and yes-noquestionsin English. The same sentencecan be sai d with a ﬁnal rise in F0
to indicate a yes-no-question, or a ﬁnal fall in F0 to indicat e a declarative intonation.
Fig. 8.8 shows the F0 track of the same words spoken as a questi on or a statement.
Note that the questionrises at the end; this is often called a question rise . The falling Question rise
intonationofthe statementiscalleda ﬁnal fall. Final fall
Time (s)0 0.922Pitch (Hz)
50250
youknowwhatimean
Time (s)0 0.912Pitch (Hz)
50250
Figure8.8 The same text read as the statement You know what I mean . (on the left) and as a question You know

50250
50250
what I mean? (onthe right). Notice that yes-no-question intonationinE nglishhas asharp ﬁnalrise inF0.
It turnsout that English makesvery wide use of tune to expres smeaning. Besides
thiswell knownrise for yes-noquestions,an Englishphrase containinga list of nouns
separatedbycommasoftenhasashortrisecalleda continuationrise aftereachnoun. Continuationrise
English also has characteristic contours to express contra diction, to express surprise,
andmanymore.
The mapping between meaning and tune in English is extremely complex, and

linguistic theories of intonation like ToBI have only begun to develop sophisticated
modelsofthismapping. Inpractice,therefore,mostsynthe sissystemsjust distinguish
two or three tunes, such as the continuation rise (at commas), the question rise (at
questionmarkif thequestionisa yes-noquestion),anda ﬁnal fall otherwise.
8.3.4 Moresophisticated models: ToBI
While currentsynthesissystems generallyuse simple model sof prosodylike the ones
discussed above, recent research focuses on the developmen t of much more sophisti-
catedmodels. We’ll verybrieﬂydiscussthe ToBI, andTiltmodelshere.
ToBI

ToBI
Oneofthemostwidelyusedlinguisticmodelsofprosodyisth eToBI(ToneandBreak ToBI
Indices)model(Silvermanetal.,1992;BeckmanandHirschb erg,1994;Pierrehumbert,
1980; Pitrelli et al., 1994). ToBI is a phonologicaltheory o f intonation which models
prominence,tune,and boundaries. ToBI’s modelof prominen ceandtunesis basedon
the5pitch accents and4boundarytones shownin Fig.8.3.4.
Anutterancein ToBIconsistsofa sequenceofintonationalp hrases,eachofwhich

ToBI
endsin one ofthe four boundary tones . The boundarytonesare usedto representthe Boundary tone

DRAFT268 Chapter 8. SpeechSynthesis
PitchAccents BoundaryTones
H* peakaccent L-L% “ﬁnalfall”: “declarativecontour”ofAmerican
English”
L* low accent L-H% continuation rise
L*+H scooped accent H-H% “question rise”: cantonical yes-no question
contour
L+H* risingpeakaccent H-L% ﬁnal level plateau (plateau because H- causes
“upstep” of following)
H+!H* stepdown
Figure8.9 The accent and boundary tones labels from the ToBI transcrip tion system for
AmericanEnglishintonation (Beckman andAyers, 1997; Beck man and Hirschberg, 1994).
utteranceﬁnal aspectsoftunediscussedinSec. 8.3.3. Each wordinthe utterancescan
optionallybe associatedwithoneoftheﬁvetypesofpitchac cents.
Each intonational phrase consists of one or more intermediate phrase . These
phrases can also be marked with kinds of boundary tone, inclu ding the%Hhigh ini-
tial boundary tone, which is used to mark a phrase which is par ticularly high in the
speakers’pitchrange,aswell asﬁnalphraseaccents H-andL-.

Inadditionto accentsandboundarytones, ToBIdistinguish esfourlevelsofphras-
ing, which are labeled on a separate break index tier. The largest levels of phrasing Break index
are the intonational phrase (break index 4) and the intermediate phrase (break index
3), and were discussed above. Break index 2is used to mark a disjuncture or pause
between words that is smaller than an intermediate phrase, w hile1is used for normal
phrase-medialwordboundaries.
Fig. 8.10 shows the tone, orthographic,and phrasing tiersof a ToBI transcription, Tier

usingthe praatprogram. Weseethesamesentencereadwithtwodifferentint onation
patterns. In (a), the word Marianna is spokenwith a highH* accent, andthe sentence
has the declarative boundary tone L-L%. In (b), the word Marianna is spoken with
a low L* accent and the yes-no question boundary tone H-H%. On e goal of ToBI is
to express different meanings to the different type of accen ts. Thus, for example, the
L*accentaddsameaningof surprisetothesentence(i.e.,withaconnotationlike‘Are

youreallysayingitwasMarianna?’). (HirschbergandPierr ehumbert,1986;Steedman,
2003).
ToBI models have been proposed for many languages, such as th e JTOBI system
forJapanese(Venditti,2005);see Jun(2005).
OtherIntonationmodels
TheTiltmodel (Taylor, 2000) resembles ToBI in using sequences of in tonational Tilt
events like accents and boundary tones. But Tilt does not use ToBI-style discrete
phonemic classes for accents. Instead, each event is modele d by continuous param-
etersthatrepresenttheF0shapeoftheaccent.

2003).
Insteadofgivingeacheventacategorylabel,asinToBI,eac hTiltprosodiceventis
characterizedbyasetofthreeacousticparameters: thedur ation,theamplitude,andthe
tiltparameter. Theseacousticparametersaretrainedonacorpu swhichhasbeenhand-
labeled for pitch accents ( a) and boundary tones ( b). The human labeling speciﬁes
the syllable which bears the accent or tone; the acoustic par ameters are then trained

DRAFTSection8.3. ProsodicAnalysis 269
H* L–L
<SIL> marianna madethe marmalade <SIL>
1 11 4
Time (s)0 1.3L* H–H
marianna madethe marmalade
1 11 4
Time (s)0 1.49
Figure8.10 The same sentence read by Mary Beckman with two different int onation patterns and transcribed in
ToBI.(a)shows anH*accentandthetypicalAmericanEnglish declarative ﬁnalfallL-L%. (b)showstheL*accent,
withthe typical AmericanEnglishyes-no question riseH-H% .
a a a b7 7 7 7 7 7 7 7 7 7 7 7
Figure8.11 Schematic view of events in the Tilt model (Taylor, 2000). Ea ch pitch accent ( a)
and boundary tone ( b) isaligned witha syllable nucleus s.
automatically from the waveﬁle. Fig. 8.11 shows a sample of a Tilt representation.
Each accent in Tilt is viewed as having a (possibly zero) rise component up to peak,
followed by a (possible zero) fall component . An automatic accent detector ﬁnds
the start, peak, and end point of each accent in the waveﬁle, w hich determines the
durationandamplitudeoftheriseandfallcomponents. Thet iltparameterisanabstract

description of the F0 slope of an event, calculated by compar ing the relative sizes of
the rise and fall for an event. A tilt value of 1.0 indicates a r ise, tilt of -1.0 a fall, 0
equalriseandfall,-0.5is anaccentwith arise andalargerf all, andso on:
tilt=tiltamp+tiltdur
2
=|Arise|−|Afall|
|Arise|+|Afall|+Drise−Dfall
Drise+Dfall(8.19)

DRAFT270 Chapter 8. SpeechSynthesis
Seetheendofthechapterforpointerstootherintonational models.
8.3.5 Computing duration from prosodiclabels
The results of the text analysis processes described so far i s a string of phonemes,
annotated with words, with pitch accent marked on relevant w ords, and appropriate
boundary tones marked. For the unit selection synthesis approaches that we will de-
scribeinSec. 8.5,thisisa sufﬁcientoutputfromthe textan alysiscomponent.
Fordiphonesynthesis,aswell asotherapproacheslikeformantsynthes is,we also
needtospecifythe duration andtheF0valuesofeachsegment.
Phonesvaryquitea bit induration. Someofthedurationisin herenttothe identity
of the phone itself. Vowels, for example, are generally much longer than consonants;
in the Switchboard corpus of telephone speech, the phone [aa ] averages 118 millisec-
onds, while [d] averages 68 milliseconds. But phone duratio n is also affected by a
wide variety of contextual factors, which can be modeled by r ule-based or statistical
methods.

methods.
The most well-known of the rule-based methods is the method o f Klatt (1979),
which uses rules to model how the average or ‘context-neutra l’duration of a phone ¯d
islengthenedorshortenedbycontext,while stayingabovea minimumduration dmin.
EachKlatt ruleisassociatedwith adurationmultiplicativ efactor;someexamples:
PrepasualLengthening: The vowel or syllabic consonant in the syllable before a
pause islengthened by1.4.
Non-phrase-ﬁnalShortening: Segments which are not phrase-ﬁnal are shortened by 0.6.

Phrase-ﬁnalpostvocalicliquidsandnasalsarelengthened by
1.4.
UnstressedShortening: Unstressed segments are more compressible, so their mini-
mum duration dminis halved, and are shortened by .7 for
most phone types.
Lengtheningfor Accent: A vowel whichbears accent islengthened by1.4
ShorteninginClusters: A consonant followed bya consonant isshortened by 0.5.
Pre-voiceless shortening: Vowels areshortened before a voiceless plosive by0.7
Giventhe Nfactorweights f, theKlatt formulaforthedurationofaphoneis:
d=dmin+N
∏

1.4.
d=dmin+N
∏
i=1fi×(¯d−dmin) (8.20)
Morerecentmachine-learningsystemsusetheKlatthand-wr ittenrulesasthebasis
fordeﬁningfeatures,forexampleusingfeaturessuchasthe following:
•identityoftheleft andrightcontextphone
•lexicalstress andaccentvaluesofcurrentphone
•positioninsyllable,word,phrase
•followingpause
We can then train machine learning classiﬁers like decision trees or the sum-of-
products model(vanSanten,1994,1997,1998),tocombinethefeature stopredictthe Sum-of-products
ﬁnaldurationofthesegment.

DRAFTSection8.3. ProsodicAnalysis 271
8.3.6 Computing F0 from prosodic labels
Fordiphone,articulatory,HMM,andformantsynthesisweal so needtospecifytheF0
valuesofeachsegment. Forthetonesequencemodelslike ToB IorTilt, thisF0 gener-
ation can be done by specifying F0 target points for each pitch accent and boundary Target point
tone; the F0 contour for the whole sentence can be created by i nterpolating among
thesetargets(Andersonet al.,1984).
In order to specify a target point we need to describe what it i s (the F0 value)
and when it occurs (the exact time at which this peak or trough occurs in the sylla-
ble). The F0 values of the target points are generally not spe ciﬁed in absolute terms
of Hertz. Instead, they are deﬁned relativeto pitch range . A speaker’s pitch range is Pitchrange
therangebetweenthelowestfrequencytheyuseinaparticul arutterance(the baseline
frequency ) and the highest frequencyin the utterance (the topline). In some models,Baselinefrequency

Topline targetpointsarespeciﬁedrelativeto aline inbetweencall edthereferenceline .
Referenceline For example, we might write a rule specifying that the very be ginningof an utter-
ance have a target point of 50% (halfway between the baseline and topline). In the
rule-basedsystemofJilkaetal.(1999)thetargetpointfor anH*accentisat100%(the
topline) and for an L* accent at 0% (at the baseline). L+H* acc ents have two target
points, at 20% and 100%. Final boundary tones H-H% and L-L% ar e extra-high and
extra-lowat120%and-20%respectively.

Second, we must also specify exactly where in the accented sy llable the targets
apply; this is known as accent alignment . In the rule-based system of Jilka et al. Alignment
(1999), again, H* accents are aligned 60% of the way through t he voiced part of the
accent syllable (although IP-initial accents are aligned s omewhat later in the syllable,
whileIP-ﬁnalaccentsarealignedsomewhatearlier).
Instead of writing these rules by hand, the mapping from pitc h accent sequence
to F0 value may be learned automatically. For example Black a nd Hunt (1996) used

linearregressionto assigntargetvaluestoeachsyllable. Foreachsyllablewith apitch
accent or boundary tone, they predicted three target values , at the beginning, middle,
and end of the syllable. They trained three separate linear r egression models, one for
eachofthethreepositionsin thesyllable. Featuresinclud ed:
•accent type on the current syllable as well as two previous an d two following
syllables
•lexicalstress ofthissyllableandsurroundingsyllables
•numberofsyllablesto startofphraseandto endofphrase

syllables
•numberofaccentedsyllablesto endofphrase
Such machine learning models require a training set that is l abeled for accent; a
number of such prosodically-labeled corpora exist, althou gh it is not clear how well
thesemodelsgeneralizetounseencorpora.
Finally, F0 computation models must model the fact that pitc h tends to decline
througha sentence; this subtle drop in pitch across an utter ance is called declination ; Declination
anexampleisshownin Fig.8.12.

The exact nature of declination is a subject of much research ; in some models, it
is treated by allowing the baseline (or both baseline and top -line) to decrease slowly
overthe utterance. In ToBI-like models, this downdriftin F 0 is modeled by two sepa-

DRAFT272 Chapter 8. SpeechSynthesis
Time (s)0 1.81392Pitch (Hz)100400
Figure8.12 F0 declination in the sentence ‘I was pretty goofy for about t wenty-four hours
afterwards’.
rate components; in addition to declination, certain high t ones are marked as carrying
downstep . Each downstepped high accent causes the pitch range to be co mpressed, Downstep
resultingin aloweredtoplineforeachsuchaccent.
8.3.7 Finalresult oftextanalysis: Internal Representati on
The ﬁnal output of text analysis is what we called the internal representation of the
input text sentence. For unit selection synthesis, the inte rnal representation can be as
simple as a phone string together with indications of prosod ic boundaries and promi-
nentsyllables,asshowninFig.8.1. Fordiphonesynthesisa swellasnon-concatenative
synthesisalgorithmstheinternalrepresentationmustals oincludeadurationandanF0
valueforeachphone.
Fig.8.13showssomesampleTTSoutputfromtheFESTIVAL(Bla cket al.,1999)

diphone speech synthesis system for the sentence Do you really want to see all of
it?. This output, together with the F0 values shown in Fig. 8.14 w ould be the input
to thewaveform synthesis component described in Sec. 8.4. The durations here are
computedbya CART-style decisiontree(Riley,1992).
H* L* L-H%
do you really want tosee allof it
duwyuwrihliywaanttaxsiyaolahviht
110110505075645782575072414347541307690446246220

H* L* L-H%
Figure8.13 Output oftheFESTIVAL(Blacketal.,1999) generator forthe sentenceDoyoureallywant toseeall
of it?,together withthe F0contour shown inFig.8.14. Figure than ks toPaul Taylor.
As was suggested above,determiningthe properprosodicpat ternfor a sentence is
difﬁcult, as real-world knowledge and semantic informatio n is needed to know which
syllablesto accent,and whichtuneto apply. Thissort ofinf ormationisdifﬁcult toex-
tractfromthetextandhenceprosodymodulesoftenaimtopro ducea“neutraldeclara-

tive”versionoftheinputtext,whichassumethesentencesh ouldbespokeninadefault
waywithnoreferencetodiscoursehistoryorreal-worldeve nts. Thisisoneofthemain
reasonswhyintonationinTTSoftensounds“wooden”.

DRAFTSection8.4. DiphoneWaveformsynthesis 273
do you really want to see all of itH*H%
L-L*
Figure8.14 The F0 contour for the sample sentence generated by the
FESTIVALsynthesis system inFig.8.13, thanks toPaulTaylo r.
8.4 DiphoneWaveformsynthesis
We are now ready to see how the internal representation can be turned into a wave-
form. We will present two kinds of concatentative synthesis: diphone synthesis in
thissection,and unit selectionsynthesis inthe nextsection.
Recallthatfordiphonesynthesis,ourinternalrepresenta tionisasshowninFig.8.13
andFig.8.14,consistingofalist ofphones,eachphoneasso ciatedwithadurationand
a set ofF0targets.
Thediphoneconcatenativesynthesismodelgeneratesawave formfromasequence
of phones by selecting and concatenating units from a prerec orded database of di-
phones. Adiphoneisaphone-likeunitgoingfromroughlythemiddle ofonephoneto Diphone
the middle of the followingphone. Diphoneconcatenativesy nthesiscan be character-
izedbythefollowingsteps:
Training:

Training:
1. Recordasinglespeakersayingan exampleofeachdiphone.
2. Cut each diphone out from the speech and store all diphones in a diphone
database.
Synthesis:
1. Take from the database a sequence of diphones that corresp onds to the
desiredphonesequence.
2. Concatenatethediphones,doingsomeslightsignalproce ssingatthebound-
aries
3. Use signal processing to change the prosody (f0, duration ) of the diphone
sequenceto thedesiredprosody.
We tend touse diphonesratherthanphonesforconcatenative synthesisbecauseof

database.
aries
the phenomenonof coarticulation . In Ch. 7 we deﬁned coarticulation as the move- Coarticulation
ment of articulators to anticipate the next sound, or persev erating movementfrom the
last sound. Because of coarticulation, each phone differs s lightly depending on the
previousandfollowingphone. Thisifwejustconcatenatedp honestogether,wewould
haveverylargediscontinuitiesat theboundaries.

DRAFT274 Chapter 8. SpeechSynthesis
In a diphone, we model this coarticulation by including the t ransition to the next
phone inside the unit. The diphone [w-eh], for example, incl udes the transition from
the [w] phoneto the [eh] phone. Because a diphoneis deﬁnedfr omthe middleof one
phone to the middle of the next, when we concatenate the dipho nes, we are concate-
nating the middle of phones, and the middle of phones tend to b e less inﬂuenced by
thecontext. Fig.10.11showstheintuitionthatthebeginni ngandendofthevowel[eh]
havemuchmoremovementthanthecenter.
w eh db eh n
Time (s)0 0.63
Figure8.15 The vowel [eh] in different surrounding contexts, in the wor dswedandBen.
Notice the differences in the second formants (F2) at the beg inning and end of the [eh], but
the relativelysteadystate portion inthe middle atthe blue line.
8.4.1 Building a diphone database
Therearesix stepsinbuildinga diphonedatabase:
1. Createa diphoneinventory
2. Recruitaspeaker
3. Createa textforthespeakertoreadforeachdiphone

w eh db eh n
4. Recordthespeakerreadingeachdiphone
5. Segment,label,andpitch-markthediphones
6. Excisethediphones
Whatistheinventoryofdiphonesthatweneedforasystem? If wehave43phones
(like the AT&T system of Olive et al. (1998)), there are 432=1849 hypothetically
possible diphone combinations. Not all of these diphones ca n actually occur. For
example, English phonotactic constraints rule out some combinations; phones like
[h], [y], and [w] can only occur before vowels. In addition, s ome diphone systems

don’tbotherstoringdiphonesifthereisnopossiblecoarti culationbetweenthephones,
such as across the silence between successive voiceless sto ps. The 43-phone system
of Olive et al. (1998)thus has only 1162diphonesrather than the 1849 hypothetically
possibleset.
Next we recruit our speaker,often called a voice talent . The databaseof diphones Voice talent

DRAFTSection8.4. DiphoneWaveformsynthesis 275
forthisspeakeriscalleda voice;commercialsystemsoftenhavemultiplevoices,such Voice
asonemaleandonefemalevoice.
We’ll now create a text for the voice talent to say, and record each diphone. The
most important thing in recording diphones is to keep them as consistent as possible;
if possible, they should have constant pitch, energy, and du ration, so they are easy to
paste together without noticeable breaks. We do this by encl osing each diphone to be
recorded in a carrier phrase . By putting the diphone in the middle of other phones, Carrier phrase
we keep utterance-ﬁnal lengtheningor initial phone effect s from making any diphone
louder or quieter than the others. We’ll need different carr ier phrases for consonant-
vowel, vowel-consonant,phone-silence,and silence-phon esequences. For example,a
consonant vowel sequence like [b aa] or [b ae] could be embedd edbetween the sylla-
bles[taa]and[maa]:
pauset aa baam aapause
pauset aa baem aapause
pauset aa behmaa pause

...
If we have an earlier synthesizer voice lying around, we usua lly use that voice to
read the prompts out loud, and have our voice talent repeat af ter the prompts. This
is another way to keep the pronunciation of each diphone cons istent. It is also very
importantto use a highqualitymicrophoneand a quiet roomor ,better, a studio sound
booth.
Once we have recorded the speech, we need to label and segment the two phones
that make up each diphone. This is usually done by running a sp eech recognizer in
forced alignment mode . In forced alignment mode, a speech recognition is told ex-

booth.
actly what the phone sequence is; its job is just to ﬁnd the exa ct phone boundaries
in the waveform. Speech recognizers are not completely accu rate at ﬁnding phone
boundaries,andso usuallytheautomaticphonesegmentatio nishand-corrected.
Wenowhavethetwophones(forexample[baa])withhand-corr ectedboundaries.
Therearetwowayswecancreatethe/b-aa/diphoneforthedat abase. Onemethodisto
userulestodecidehowfarintothephonetoplacethediphone boundary. Forexample,

forstops,weputplacethediphoneboundary30%ofthewayint othephone. Formost
otherphones,we placethediphoneboundary50%intothephon e.
A more sophisticated way to ﬁnd diphone boundaries is to stor e the entire two
phones, and wait to excise the diphones until we are know what phone we are about
to concatenate with. In this method, known as optimal coupling , we take the two Optimal coupling
(complete, uncut) diphoneswe need to concatenate, and we ch eck every possible cut-
tingpointforeachdiphones,choosingthetwocuttingpoint sthatwouldmaketheﬁnal

frameoftheﬁrstdiphoneacousticallymostsimilartotheen dframeofthenextdiphone
(Taylorand Isard,1991;Conkie and Isard,1996). Acoustica l similar can be measured
byusingcepstralsimilarity ,to bedeﬁnedinSec. 9.3.
8.4.2 Diphone concatenation andTD-PSOLA forprosody
We are now ready to see the remaining steps for synthesizing a n individual utterance.
Assume that we have completedtext analysis for the utteranc e, and hence arrived at a

DRAFT276 Chapter 8. SpeechSynthesis
sequence of diphones and prosodic targets, and that we have a lso grabbed the appro-
priate sequence of diphones from the diphone database. Next we need to concatenate
the diphones together and then adjust the prosody (pitch, en ergy, and duration) of the
diphonesequenceto matchthe prosodicrequirementsfromth e intermediaterepresen-
tation.
Given two diphones, what do we need to do to concatenate them s uccessfully?
If the waveforms of the two diphones edges across the junctur e are very different,
a perceptible clickwill result. Thus we need to apply a windowing function to the Click
edge of both diphonesso that the samples at the juncture have low or zero amplitude.
Furthermore,if both diphonesare voiced, we need to insure t hat the two diphonesare
joinedpitch-synchronously . This means that the pitch periods at the end of the ﬁrst Pitch-synchronous
diphone must line up with the pitch periods at the beginning o f the second diphone;

otherwise the resulting single irregular pitch period at th e juncture is perceptible as
well.
Now given our sequence of concatenated diphones, how do we mo dify the pitch
and duration to meet our prosodic requirements? It turns out there is a very sim-
ple algorithm for doing this called TD-PSOLA (Time-Domain Pitch-Synchronous TD-PSOLA
OverLap-and-Add ).
As we just said, a pitch-synchronous algorithmis one in which we do something
ateachpitchperiodor epoch. Forsuchalgorithmsitisimportanttohaveveryaccurate

well.
pitch markings: measurementsofexactly whereeach pitchpu lse orepochoccurs. An
epoch can be deﬁned by the instant of maximum glottal pressur e, or alternatively by
the instant of glottal closure. Note the distinction betwee npitch marking orepoch Pitchmarking
detection andpitch tracking . Pitch trackinggivesthe valueofF0 (theaveragecycles Pitchtracking
persecondoftheglottis)ateachparticularpointintime,a veragedoveraneighborhood.

Pitch marking ﬁnds the exact point in time at each vibratory c ycle at which the vocal
foldsreachsomespeciﬁc point(epoch).
Epoch-labeling can be done in two ways. The traditional way, and still the most
accurate, is to use an electroglottograph orEGG(often also called a laryngographElectroglotto-
graph
EGG
LaryngographorLx). An EGG is a devicewhichstraps ontothe (outsideof the)spe aker’snecknear
Lxthe larynx and sends a small current through the Adam’s apple . A transducer detects

graph
EGG
whether the glottis is open or closed by measuring the impeda nce across the vocal
folds. Some modernsynthesis databasesare still recordedw ith an EGG. The problem
with using an EGG is that it must be attached to the speaker whi le they are recording
the database. Although an EGG isn’t particularly invasive, this is still annoying, and
the EGG must be used during recording; it can’t be used to pitc h-mark speech that
has already been collected. Modern epoch detectors are now a pproaching a level of

accuracythat EGGs are no longer used in most commercialTTS e ngines. Algorithms
forepochdetectionincludeBrookesandLoke(1999),Veldhu is(2000).
Given an epoch-labeled corpus, the intuition of TD-PSOLA is that we can mod-
ify the pitch and duration of a waveform by extracting a frame for each pitch period
(windowed so that the frame doesn’t have sharp edges) and the n recombining these
frames in various ways by simply overlappingand adding the w indowed pitch period

frames(wewill introducetheideaofwindowsinSec.9.3.2). Theideathatwemodify
a signal by extracting frames, manipulating them in some way and then recombin-
ing them by adding up the overlappedsignals is called the overlap-and-add orOLA Overlap-and-add
OLA

DRAFTSection8.5. Unit Selection(Waveform)Synthesis 277
algorithm; TD-PSOLA is a special case of overlap-and-add in which the frames are
pitch-synchronous,andthewholeprocesstakesplaceinthe timedomain.
For example, in order to assign a speciﬁc duration to a diphon e, we might want to
lengthentherecordedmasterdiphone. To lengthenasignal w ithTD-PSOLA,we sim-
plyinsertextracopiesofsomeofthepitch-synchronousfra mes,essentiallyduplicating
a pieceofthe signal. Fig. 8.16showstheintuition.89B : ; <89B: ; < =
Figure8.16 TD-PSOLA for duration modiﬁcation. Individual pitch-sync hronous frames can
be duplicated tolengthen the signal (as shown here), or dele tedtoshorten the signal.
TD-PSOLAcan also be usedto changethe F0 valueof a recordedd iphoneto give
a higher or lower value. To increase the F0, we extract each pi tch-synchronousframe
from the original recorded diphone signal, place the frames closer together (overlap-
ping them), with the amount of overlap determined by the desi red period and hence

frequency, and then add up the overlapping signals to produc e the ﬁnal signal. But
notethat bymovingall theframescloser together,we make th esignal shorterin time!
Thus in order to change the pitch while holding the durationc onstant, we need to add
duplicateframes.
Fig. 8.17 shows the intuition; in this ﬁgure we have explicit lyshown the extracted
pitch-synchronousframeswhichareoverlappedandadded;n otethattheframesmoved
closer together (increasing the pitch) while extra frames h ave been added to hold the

durationconstant.
8.5 UnitSelection(Waveform)Synthesis
Diphone waveform synthesis suffers from two main problems. First, the stored di-
phone database must be modiﬁed by signal process methods lik e PSOLA to produce

DRAFT278 Chapter 8. SpeechSynthesis> ?3 @ A B7 C
> ? 3 @ A
Figure8.17 TD-PSOLA for pitch (F0) modiﬁcation. In order to increase th e pitch, the indi-
vidual pitch-synchronous frames are extracted, Hanning wi ndowed, moved closer together and
then added up. To decrease the pitch, we move the frames furth er apart. Increasing the pitch
willresult ina shorter signal (since the frames are closer t ogether), so we also need toduplicate
frames ifwe want tochange the pitch whileholding the durati on constant.
thedesiredprosody. Anykindofsignalprocessingofthesto redspeechleavesartifacts
in the speech which can make the speech sound unnatural. Seco nd, diphonesynthesis
onlycapturesthecoarticulationduetoasingleneighborin gphone. Buttherearemany
more global effects on phonetic realization, including mor e distant phones, syllable
structure,thestresspatternsofnearbyphones,andevenwo rd-leveleffects.
For this reason, modern commercial synthesizers are based o n a generalization of

diphone synthesis called unit selection synthesis . Like diphone synthesis, unit selec-Unit selectionsynthesis
tion synthesis is a kind of concatenative synthesis algorit hm. It differs from classic
diphonesynthesisin twoways:
1. Indiphonesynthesisthedatabasestoresexactlyonecopy ofeachdiphone,while
in unit selection, the unit database is many hours long, cont aining many copies
ofeachdiphone.

DRAFTSection8.5. Unit Selection(Waveform)Synthesis 279
2. Indiphonesynthesis,theprosodyoftheconcatenateduni tsismodiﬁedbyPSOLA
or similar algorithms, while in unit selection no (or minima l) signal processing
isappliedtotheconcatenatedunits.
The strengths of unit selection are due to the large unit data base. In a sufﬁciently
large database, entire words or phrases of the utterance we w ant to synthesize may be
already present in the database, resulting in an extremely n atural waveform for these
words or phrases. In addition, in cases where we can’t ﬁnd a la rge chunk and have to
backoffto individualdiphones,the fact that thereareso ma nycopiesofeach diphone
makesit morelikelythat wewill ﬁndonethatwill ﬁt inveryna turally.
The architecture of unit selection can be summarized as foll ows. We are given a
largedatabaseofunits;let’sassumethesearediphones(al thoughit’salsopossibletodo
unit selection with other kinds of units such half-phones, s yllables, or half-syllables).

We arealso givena characterizationofthetarget‘internal representation’,i.e. aphone
string together with features such as stress values, word id entity, F0 information, as
describedinFig. 8.1.
The goal of the synthesizer is to select from the database the best sequence of
diphone units that corresponds to the target representatio n. What do we mean by the
‘best’sequence? Intuitively,the bestsequencewouldbeon ein which:
•eachdiphoneunitweselectexactlymeetsthespeciﬁcations ofthetargetdiphone

(intermsofF0,stresslevel,phoneticneighbors,etc)
•each diphone unit concatenates smoothly with its neighbori ng units, with no
perceptiblebreak.
Of course, in practice, we can’t guarantee that there wil be a unit which exactly
meetsourspeciﬁcations,andweareunlikelytoﬁndasequenc eofunitsinwhichevery
single join is imperceptible. Thus in practice unit selecti on algorithms implement a
gradientversionofthese constraints,and attemptto ﬁnd th esequenceof unitwhichat
least minimizesthe targetcost andthejoincost: Target cost

Joincost
Targetcost T(ut,st):how well the target speciﬁcation stmatches the potential
unitut
Joincost J(ut,ut+1):howwell (perceptually)the potentialunit utjoinswith its
potentialneighbor ut+1
TheTandJvalues are expressed as costsmeaning that high values indicate bad
matchesandbadjoins(HuntandBlack, 1996a).
Formally, then, the task of unit selection synthesis, given a sequence SofTtarget
speciﬁcations,istoﬁndthesequence ˆUofTunitsfromthedatabasewhichminimizes
thesumofthese costs:
ˆU=argmin
UT
∑
t=1T(st,ut)+T−1
∑
t=1J(ut,ut+1) (8.21)

unitut
ˆU=argmin
UT
∑
∑
Let’s now deﬁne the target cost and the join cost in more detai l before we turn to
thedecodingandtrainingtasks.
The target cost measures how well the unit matches the target diphone speciﬁca-
tion. We canthinkofthespeciﬁcationforeachdiphonetarge tasa featurevector;here

DRAFT280 Chapter 8. SpeechSynthesis
arethreesamplevectorsforthreetargetdiphonespeciﬁcat ions,usingdimensions(fea-
tures)like shouldthesyllablebestressed , andwherein theintonationalphraseshould
thediphonecomefrom :
/ih-t/, +stress, phrase internal, high F0, content word
/n-t/, -stress, phrase final, high F0, function word
/dh-ax/, -stress, phrase initial, low F0, word ‘the’
We’d like the distance between the target speciﬁcation sand the unit to be some
function of the how different the unit is on each of these dime nsionsfrom the speciﬁ-
cation. Let’s assume that for each dimension p, we can come up with some subcost
Tp(st[p],uj[p]). The subcost for a binary feature like stressmight be 1 or 0. The sub-
costforacontinuousfeaturelikeF0mightbethedifference (orlogdifference)between
thespeciﬁcationF0andunitF0. Sincesomedimensionsaremo reimportanttospeech
perceptions than others, we’ll also want to weight each dime nsion. The simplest way

to combine all these subcosts is just to assume that they are i ndependentand additive.
Using this model, the total target cost for a given target/un it pair is the weighted sum
overall thesesubcostsforeachfeature/dimension:
T(st,uj) =P
∑
p=1wpTp(st[p],uj[p]) (8.22)
The target cost is a function of the desired diphone speciﬁca tion and a unit from
the database. The join cost, by contrast, is a function of two units from the database.
The goalof the joincost is to be low (0)whenthe join is comple telynatural,and high

∑
when the join would be perceptible or jarring. We do this by me asuring the acoustic
similarity of the edges of the two units that we will be joinin g. If the two units have
very similar energy, F0, and spectral features, they will pr obably join well. Thus as
withthe targetcost,we computea joincostbysummingweight edsubcosts:
J(ut,ut+1) =P
∑
p=1wpJp(ut[p],ut+1[p]) (8.23)
The three subcosts used in the classic Hunt and Black (1996b) algorithm are the
cepstral distance at the point of concatenation, and the absolute differences in log

∑
powerandF0. We will introducethecepstruminSec. 9.3.
In addition, if the two units utandut+1to be concatenated were consecutive di-
phones in the unit database (i.e. they followed each other in the original utterance),
then we set the join cost to 0: J(ut,ut+1) =0. This is an important feature of unit
selection synthesis, since it encourageslarge natural seq uences of units to be selected
fromthedatabase.
How do we ﬁnd the best sequence of units which minimizes the su m of the target
andjoincostsasexpressedinEq.8.21? Thestandardmethodi stothinkoftheunitse-

lectionproblemasaHiddenMarkovModel. Thetargetunitsar etheobservedoutputs,
and the units in the database are the hidden states. Our job is to ﬁnd the best hidden
state sequence. We will usetheViterbialgorithmtosolveth isproblem,justaswesaw

DRAFTSection8.5. Unit Selection(Waveform)Synthesis 281
it in Ch. 5 and Ch. 6, and will see it again in Ch. 9. Fig. 8.18 sho ws a sketch of the
searchspaceaswell asthe best(Viterbi)paththatdetermin esthebestunitsequence.
Ds-ih1
Es-ihih-k k-ss-# #
s-ih2
s-ih3ih-k1
ih-k2
ih-k3k-s1
k-s2s-#1
s-#2#TARGETS
UNITS
Join CostTarget Costs
Figure8.18 Theprocessofdecodinginunitselection. Theﬁgureshowsth esequenceoftarget
(speciﬁcation) diphones for the word six, and the set of possible database diphone units that we
must search through. The best (Viterbi) path that minimizes the sum of the target and join costs
is showninbold.
The weights for join and target costs are often set by hand, si nce the number of
weights is small (on the order of 20) and machine learning alg orithms don’t always
achievehumanperformance. Thesystem designerlistenstoe ntiresentencesproduced
bythesystem,andchoosesvaluesforweightsthat resultinr easonablesoundingutter-

ances. Various automatic weight-setting algorithms do exi st, however. Many of these
assumewehavesomesortofdistancefunctionbetweentheaco usticsoftwosentences,
perhapsbasedoncepstraldistance. ThemethodofHuntandBl ack(1996b),forexam-
ple,holdsoutatestsetofsentencesfromtheunitselection database. Foreachofthese
test sentences, we take the word sequence and synthesize a se ntence waveform (using
unitsfromtheothersentencesinthetrainingdatabase). No wwecomparetheacoustics
of the synthesized sentence with the acoustics of the true hu man sentence. Now we

haveasequenceofsynthesizedsentences,eachoneassociat edwithadistancefunction
toits humancounterpart. Nowwe use linearregressionbased onthesedistancestoset
thetargetcostweightsso astominimizethedistance.
Thereare also moreadvancedmethodsofassigningbothtarge tandjoin costs. For
example,abovewecomputedtargetcostsbetweentwounitsby lookingat thefeatures
of the two units, doing a weighted sum of feature costs, and ch oosing the lowest-
cost unit. An alternative approach (which the new reader mig ht need to come back to

after learning the speech recognition techniques introduc ed in the next chapters) is to
map the target unit into some acoustic space, and then ﬁnd a un it which is near the
targetinthatacousticspace. InthemethodofDonovanandEi de(1998),Donovanand
Woodland (1995), for example, all the training units are clu stered using the decision
tree algorithmof speechrecognitiondescribedin Sec. 10.3 . Thedecisiontree isbased
onthesamefeaturesdescribedabove,buthereforeachsetof features,wefollowapath
down the decision tree to a leaf node which contains a cluster of units that have those

features. This cluster of units can be parameterized by a Gau ssian model, just as for
speech recognition,so that we can map a set of features into a probabilitydistribution

DRAFT282 Chapter 8. SpeechSynthesis
over cepstral values, and hence easily compute a distance be tween the target and a
unit in the database. As for join costs, more sophisticated m etrics make use of how
perceivablea particular join might be (Wouters and Macon, 1 998; Syrdal and Conkie,
2004;BulykoandOstendorf,2001).
8.6 Evaluation
Speech synthesis systems are evaluated by human listeners. The development of a
good automatic metric for synthesis evaluation, that would eliminate the need for ex-
pensiveandtime-consuminghumanlisteningexperiments,r emainsanopenandexiting
researchtopic.
The minimal evaluation metric for speech synthesis systems isintelligibility : the Intelligibility
ability of a humanlistener to correctlyinterpret the words and meaningof the synthe-
sized utterance. A further metric is quality; an abstract measure of the naturalness, Quality
ﬂuency,orclarityofthespeech.
Themostlocalmeasuresofintelligibilitytesttheability ofalistenertodiscriminate

between two phones. The Diagnostic Rhyme Test (DRT) (Voiers et al., 1975) testsDiagnostic Rhyme
Test
DRTthe intelligibility of initial consonants. It is based on 96 pairs of confusable rhyming
wordswhichdifferonlyinasinglephoneticfeature,suchas (dense/tense )orbond/pond
(differingin voicing)or mean/beat orneck/deck (differingin nasality), and so on. For
each pair, listeners hear one member of the pair, and indicat e which they think it is.
Thepercentageofrightanswersisthenusedasanintelligib ilitymetric. The Modiﬁed

Test
Rhyme Test (MRT) (House et al., 1965) is a similar test based on a different se t ofModiﬁedRhyme
Test
MRT300 words, consisting of 50 sets of 6 words. Each 6-word set di ffers in either initial
or ﬁnal consonants(e.g., went,sent,bent,dent,tent,rentorbat,bad,back,bass,ban,
bath). Listeners are again given a single word and must identify f rom a closed list of
six words; the percentage of correct identiﬁcations is agai n used as an intelligibility
metric.
Sincecontexteffectsareveryimportant,bothDRT andMRT wo rdsareembedded
incarrierphrases likethe following: Carrier phrase

Test
metric.
Now we will say <word> again.
In order to test larger units than single phones, we can use semantically unpre-
dictable sentences (SUS) (Benoˆ ıt et al., 1996). These are sentences constructed by SUS
taking a simple POS template like DET ADJ NOUN VERB DET NOUN and inserting
randomEnglishwordsintheslots, toproducesentenceslike
The unsure steaks closed the fish.
Measures of intelligibility like DRT/MRT and SUS are design ed to factor out the
role of context in measuring intelligibility. While this al lows us to get a carefully

controlled measure of a system’s intelligibility, such aco ntextual or semantically un-
predictable sentences aren’t a good ﬁt to how TTS is used in mo st commercial appli-
cations. Thus in commercial applications instead of DRT or S US, we generally test
intelligibility using situations that mimic the desired ap plications; reading addresses
outloud,readinglinesofnewstext,andsoon.

DRAFTSection8.6. Evaluation 283
Tofurtherevaluatethe qualityofthesynthesizedutterances,wecanplayasentence
foralistenerandaskthemtogivea meanopinionscore (MOS),aratingofhowgood MOS
the synthesized utterances are, usually on a scale from 1-5. We can then compare
systems by comparing their MOS scores on the same sentences ( using, e.g., t-tests to
test forsigniﬁcantdifferences).
If we are comparing exactly two systems (perhaps to see if a pa rticular change
actually improved the system), we can use AB tests In AB tests, we play the same AB tests
sentence synthesized by two different systems (an A and a B sy stem). The human
listener chooses which of the two utterances they like bette r. We can do this for 50
sentences and compare the number of sentences preferred for each systems. In order
to avoid ordering preferences, for each sentence we must pre sent the two synthesized
waveformsinrandomorder.
Aswenotedatthebeginningofthechapter,speechsynthesis isoneoftheearliestﬁelds

ofspeechandlanguageprocessing. The18thcenturysawanum berofphysicalmodels
of the articulation process, including the von Kempelen mod el mentioned above, as
well asthe1773vowelmodelofKratzensteininCopenhagenus ingorganpipes.
But the modern era of speech synthesis can clearly be said to h ave arrived by the
early 1950’s, when all three of the major paradigms of wavefo rm synthesis had been
proposed(formantsynthesis,articulatorysynthesis,and concatenativesynthesis).
Concatenativesynthesisseemstohavebeenﬁrstproposedby Harris(1953)atBell

Laboratories, who literally spliced together pieces of mag netic tape corresponding to
phones. Harris’sproposalwasactuallymorelike unitselec tionsynthesisthandiphone
synthesis, in that he proposed storing multiple copies of ea ch phone, and proposed
the use of a join cost (choosing the unit with the smoothest fo rmant transitions with
the neighboring unit). Harris’s model was based on the phone , rather than diphone,
resulting in problems due to coarticulation. Peterson et al . (1958) added many of the

basic ideas of unit selection synthesis, including the use o f diphones, a database with
multiple copiesof each diphonewith differingprosody,and each unit labeledwith in-
tonational features including F0, stress, and duration, an d the use of join costs based
on F0 and formant distant between neighboringunits. They al so proposed microcon-
catenationtechniqueslikewindowingthewaveforms. ThePe tersonetal.(1958)model
waspurelytheoretical,however,andconcatenativesynthe siswasnotimplementedun-

til the 1960’s and 1970’s, when diphone synthesis was ﬁrst im plemented (Dixon and
Maxey, 1968; Olive, 1977). Later diphone systems included l arger units such as con-
sonantclusters (Oliveand Liberman,1979). Modernunit sel ection,includingthe idea
oflargeunitsofnon-uniformlength,andtheuseofatargetc ost,wasinventedbySag-
isaka (1988), Sagisaka et al. (1992). Hunt and Black (1996b) formalized the model,
andputitintheforminwhichwehavepresenteditinthischap terinthecontextofthe

ATR CHATR system (Black and Taylor, 1994). The idea of automa tically generating
synthesis units by clustering was ﬁrst invented by Nakajima and Hamada (1988), but

DRAFT284 Chapter 8. SpeechSynthesis
wasdevelopedmainlyby(Donovan,1996)byincorporatingde cisiontreeclusteringal-
gorithmsfrom speech recognition. Many unit selection inno vationstook place as part
oftheATTNextGensynthesizer(Syrdaletal., 2000;Syrdala ndConkie,2004).
Wehavefocusedinthischapteronconcatenativesynthesis, buttherearetwoother
paradigmsforsynthesis: formantsynthesis , inwhichwe attempttobuildruleswhich
generate artiﬁcial spectra, including especially formant s, andarticulatory synthesis ,
in which we attempt to directly model the physics of the vocal tract and articulatory
process.
Formantsynthesizers originallywereinspiredbyattemptstomimichumanspeech
bygeneratingartiﬁcial spectrograms. TheHaskinsLaborat oriesPatternPlaybackMa-
chine generated a sound wave by painting spectrogram patter ns on a moving trans-
parent belt, and using reﬂectanceto ﬁlter the harmonicsof a waveform(Cooper et al.,
1951);otherveryearlyformantsynthesizersincludeLawre nce(1953)andFant(3951).

process.
Perhapsthemostwell-knownoftheformantsynthesizerswer etheKlattformantsyn-
thesizerand its successor systems, including the MITalk system (All en et al., 1987),
and the Klattalk software used in Digital Equipment Corpora tion’s DECtalk (Klatt,
1982). See Klatt (1975)fordetails.
Articulatory synthesizers attempt to synthesize speech by modeling the physics
of the vocal tract as an open tube. Representative models, bo th early and somewhat
morerecentincludeStevensetal.(1953),Flanaganetal.(1 975),Fant(1986)SeeKlatt

(1975)andFlanagan(1972)formoredetails.
DevelopmentofthetextanalysiscomponentsofTTScamesome whatlater,astech-
niques were borrowed from other areas of natural language pr ocessing. The input to
earlysynthesissystemswasnottext,butratherphonemes(t ypedinonpunchedcards).
The ﬁrst text-to-speech system to take text as input seems to have been the system of
UmedaandTeranishi(Umedaetal.,1968;TeranishiandUmeda ,1968;Umeda,1976).
The system included a lexicalized parser which was used to as sign prosodic bound-

aries, as well as accent and stress; the extensions in Coker e t al. (1973) added addi-
tional rules, for example for deaccenting light verbs and ex plored articulatory models
aswell. TheseearlyTTSsystemsusedapronunciationdictio naryforwordpronuncia-
tions. Inordertoexpandtolargervocabularies,earlyform ant-basedTTSsystemssuch
as MITlak (Allen et al., 1987) used letter-to-sound rules in stead of a dictionary, since
computermemorywasfartooexpensivetostorelargediction aries.

Moderngrapheme-to-phonememodelsderivefromtheinﬂuent ialearlyprobabilis-
ticgrapheme-to-phonememodelofLucassenandMercer(1984 ),whichwasoriginally
proposed in the context of speech recognition. The widespre ad use of such machine
learning models was delayed, however, because early anecdo tal evidence suggested
that hand-written rules worked better than e.g., the neural networks of Sejnowski and
Rosenberg (1987). The careful comparisonsof Damper et al. ( 1999)showed that ma-

chinelearningmethodswereingenerallysuperior. Anumber ofsuchmodelsmakeuse
of pronunciation by analogy (Byrd and Chodorow, 1985; ?; Dae lemans and van den
Bosch, 1997; Marchand and Damper, 2000) or latent analogy (B ellegarda, 2005);
HMMs (Taylor, 2005) have also been proposed. The most recent work makes use
ofjointgraphone models,inwhichthe hiddenvariablesare phoneme-grapheme pairs Graphone
andtheprobabilisticmodelisbasedonjointratherthancon ditionallikelihood(Deligne

et al., 1995; Luk and Damper, 1996; Galescu and Allen, 2001; B isani and Ney, 2002;

DRAFTSection8.6. Evaluation 285
Chen,2003).
Thereisa vast literatureonprosody. Besidesthe ToBIandTI LTmodelsdescribed
above, other important computational models include the Fujisaki model (Fujisaki Fujisaki
andOhno,1997). IViE(Grabe,2001)isanextensionofToBIth atfocusesonlabelling
different varieties of English (Grabe et al., 2000). There i s also much debate on the
units of intonational structure ( intonational phrases (Beckman and Pierrehumbert,
1986),intonationunits (DuBoisetal.,1983)or toneunits (Crystal,1969)),andtheir Intonationunit
Tone unit relation to clauses and other syntactic units (Chomsky and H alle, 1968; Langendoen,
1975; Streeter, 1978; Hirschbergand Pierrehumbert,1986; Selkirk, 1986; Nespor and
Vogel,1986;Croft,1995;Ladd,1996;FordandThompson,199 6;Fordet al.,1996).
One of the most exciting new paradigmsfor speech synthesis i sHMM synthesis , HMM synthesis
ﬁrstproposedbyTokudaetal.(1995b)andelaboratedinToku daetal.(1995a),Tokuda

et al. (2000), and Tokuda et al. (2003). See also the textbook summary of HMM syn-
thesisinTaylor(2008).
More details on TTS evaluation can be found in Huang et al. (20 01) and Gibbon
et al. (2000). Other descriptions of evaluation can be found in the annual speech syn-
thesis competition called the Blizzard Challenge (Black and Tokuda, 2005; Bennett,BlizzardChallenge
2005).
Muchrecentworkonspeechsynthesishasfocusedongenerati ngemotionalspeech

2005).
(Cahn,1990;Bulut1etal.,2002;Hamzaetal.,2004;Eideeta l.,2004;Leeetal.,2006;
Schroder,2006,interalia)
Two classic text-to-speech synthesis systems are describe d in Allen et al. (1987)
(theMITalksystem) and Sproat (1998b) (the Bell Labs system). Recent te xtbooks
includeDutoit(1997),Huangetal.(2001),Taylor(2008),a ndAlanBlack’sonlinelec-
turenotesat http://festvox.org/festtut/notes/festtut_toc.html .
InﬂuentialcollectionsofpapersincludevanSantenetal.( 1997),Sagisakaetal.(1997),
NarayananandAlwan(2004). Conferencepublicationsappea rinthemainspeechengi-

neeringconferences(INTERSPEECH, IEEEICASSP ),andthe SpeechSynthesisWork-
shops. Journalsinclude SpeechCommunication ,ComputerSpeechandLanguage ,the
IEEE Transactionson Audio, Speech, and LanguageProcessin g, and theACM Trans-
actionsonSpeechandLanguageProcessing .
Exercises
8.1Implement the text normalization routine that deals with MONEY, i.e. mapping
stringsofdollaramountslike $45,$320,and$4100towords(eitherwritingcode
directlyordesigninganFST).Iftherearemultiplewaystop ronouncea number
youmaypickyourfavoriteway.

Exercises
8.2Implement the text normalization routine that deals with NTEL, i.e. seven-digit
phonenumberslike 555-1212 ,555-1300 ,andsoon. Youshouldusea combina-

DRAFT286 Chapter 8. SpeechSynthesis
tion of the pairedandtrailing unit methods of pronunciation for the last four
digits. (Againyoumayeitherwrite codeordesignanFST).
8.3Implementthetextnormalizationroutinethatdealswithty peDATEinFig.8.1.2
8.4Implementthetextnormalizationroutinethatdealswithty peNTIMEinFig.8.1.2.
8.5(SuggestedbyAlanBlack). DownloadthefreeFestivalspeec hsynthesizer. Aug-
mentthe lexiconto correctlypronouncethenamesofeveryon einyourclass.
8.6DownloadtheFestivalsynthesizer. Recordandtrainadipho nesynthesizerusing
yourownvoice.

Chapter9
Automatic SpeechRecognition
WhenFredericwas alittleladhe proved sobrave and daring,
Hisfather thought he’d ’prentice him tosome career seafari ng.
Iwas, alas! his nurs’rymaid, and soitfelltomylot
Totake andbind the promising boy apprentice toa pilot—
Alife not badfor a hardy lad,though surelynot a high lot,
Though I’ma nurse, you might do worse thanmake your boy a pilo t.
Iwas a stupid nurs’rymaid, on breakers always steering,
AndI didnot catchthe word aright,through beinghard of hear ing;
Mistakingmy instructions, whichwithinmybraindidgyrate ,
Itook andbound thispromising boy apprentice toa pirate.

The Piratesof Penzance , Gilbertand Sullivan,1877
Alas,thismistakebynurserymaidRuthledtoFrederic’slon gindentureasapirateand,
due to a slight complication involving 21st birthdays and le ap years, nearly led to 63
extrayearsofapprenticeship. Themistakewasquitenatura l,in aGilbert-and-Sullivan
sort of way; as Ruth later noted, “The two words were so much al ike!” True, true;
spoken languageunderstandingis a difﬁcult task, and it is r emarkablethat humansdo
aswellatit aswedo. Thegoalof automaticspeechrecognition (ASR)researchisto ASR

address this problem computationally by building systems t hat map from an acoustic
signaltoastringofwords. Automaticspeechunderstanding (ASU)extendsthisgoal
toproducingsomesortofunderstandingofthesentence,rat herthanjustthe words.
Thegeneralproblemofautomatictranscriptionofspeechby anyspeakerinanyen-
vironmentis still far fromsolved. But recent yearshave see n ASR technologymature
to the point where it is viable in certain limited domains. On e major application area

is in human-computer interaction. While many tasks are bett er solved with visual or
pointing interfaces, speech has the potential to be a better interface than the keyboard
fortaskswherefullnaturallanguagecommunicationisusef ul,orforwhichkeyboards
arenotappropriate. Thisincludeshands-busyoreyes-busy applications,suchaswhere
the user has objects to manipulate or equipment to control. A nother important ap-
plication area is telephony, where speech recognition is al ready used for example in

spoken dialogue systems for entering digits, recognizing “ yes” to accept collect calls,
ﬁndingoutairplaneortraininformation,andcall-routing (“Accounting,please”,“Prof.
Regier, please”). In some applications, a multimodal inter face combining speech and
pointing can be more efﬁcient than a graphical user interfac e without speech (Cohen
et al., 1998). Finally, ASR is applied to dictation, that is, transcription of extended
monologuebya singlespeciﬁcspeaker. Dictationiscommoni nﬁeldssuchaslawand

isalsoimportantaspartofaugmentativecommunication(in teractionbetweencomput-
ersandhumanswithsomedisabilityresultingintheinabili tytotype,ortheinabilityto
speak). TheblindMilton famouslydictated ParadiseLost to his daughters,andHenry
Jamesdictatedhislaternovelsaftera repetitivestressin jury.
Before turning to architectural details, let’s discuss som e of the parameters of the
speech recognition task. One dimension of variation in spee ch recognition tasks is

DRAFT288 Chapter 9. AutomaticSpeechRecognition
the vocabulary size. Speech recognition is easier if the num ber of distinct words we
need to recognize is smaller. So tasks with a two word vocabul ary, likeyesversusno
detection, or an eleven word vocabulary,like recognizings equencesof digits, in what
is called the digits task task, are relatively easy. On the other end, tasks with large Digit recognition
vocabularies, like transcribing human-humantelephone co nversations, or transcribing
broadcastnews,taskswithvocabulariesof64,000wordsorm ore,aremuchharder.
Aseconddimensionofvariationishowﬂuent,natural,orcon versationalthespeech
is.Isolatedword recognition,inwhicheachwordissurroundedbysomesortof pause, Isolatedword
ismucheasierthanrecognizing continuousspeech ,inwhichwordsrunintoeachotherContinuousspeech
and have to be segmented. Continuous speech tasks themselve s vary greatly in difﬁ-
culty. For example, human-to-machine speech turns out to be far easier to recognize

than human-to-human speech. That is, recognizing speech of humans talking to ma-
chines, either reading out loud in read speech (which simulates the dictation task), or Readspeech
conversing with speech dialogue systems, is relatively eas y. Recognizing the speech
of two humanstalkingto each other,in conversationalspeech recognition,for exam-Conversationalspeech
ple for transcribing a business meeting or a telephone conve rsation, is much harder.

It seems that when humans talk to machines, they simplify the ir speech quite a bit,
talkingmoreslowlyandmoreclearly.
Athirddimensionofvariationischannelandnoise. The dictation task(andmuch
laboratory research in speech recognition) is done with hig h quality, head mounted
microphones. Head mounted microphones eliminate the disto rtion that occurs in a
table microphone as the speaker’s head moves around. Noise o f any kind also makes
recognitionharder. Thusrecognizingaspeakerdictatingi naquietofﬁceismucheasier

thanrecognizinga speakerinanoisycaronthehighwaywitht he windowopen.
A ﬁnal dimension of variation is accent or speaker-class cha racteristics. Speech is
easier to recognize if the speaker is speaking a standard dia lect, or in general one that
matches the data the system was trained on. Recognition is th us harder on foreign-
accented speech, or speech of children (unless the system wa s speciﬁcally trained on
exactlythese kindsofspeech).
Table 9.1 shows the roughpercentageof incorrect words(the word error rate , or
WER, deﬁnedonpage330)fromstate-of-the-artsystemsondi fferentASRtasks.

Task Vocabulary ErrorRate %
TI Digits 11(zero-nine,oh) .5
Wall StreetJournalreadspeech 5,000 3
Wall StreetJournalreadspeech 20,000 3
BroadcastNews 64,000+ 10
ConversationalTelephoneSpeech(CTS) 64,000+ 20
Figure9.1 Roughworderrorrates(%ofwordsmisrecognized)reporteda round2006forASR
onvarioustasks;theerrorratesforBroadcastNewsandCTSa rebasedonparticulartrainingand
test scenarios and should be taken as ballpark numbers; erro r rates for differently deﬁned tasks
mayrange uptoa factor of two.
Variationduetonoiseandaccentincreasestheerrorratesq uiteabit. Theworderror

rateonstronglyJapanese-accentedorSpanishaccentedEng lishhasbeenreportedtobe
about3 to 4 timeshigherthan fornativespeakerson the same t ask (Tomokiyo,2001).

DRAFTSection9.1. SpeechRecognitionArchitecture 289
And addingautomobilenoise with a 10dBSNR (signal-to-nois eratio)can cause error
ratestogoupby2to 4times.
Ingeneral,theseerrorratesgodowneveryyear,asspeechre cognitionperformance
has improved quite steadily. One estimate is that performan ce has improved roughly
10 percent a year over the last decade (Deng and Huang, 2004), due to a combination
ofalgorithmicimprovementsandMoore’slaw.
While the algorithms we describe in this chapter are applica ble across a wide va-
riety of these speech tasks, we chose to focus this chapter on the fundamentalsof one
crucial area: Large-Vocabulary Continuous Speech Recognition (LVCSR). Large- LVCSR
vocabulary generally means that the systems have a vocabula ry of roughly 20,000
to 60,000 words. We saw above that continuous means that the words are run to-
gether naturally. Furthermore, the algorithms we will disc uss are generally speaker-
independent ; that is, theyare able to recognizespeechfrom peoplewhose speech theSpeaker

independent
systemhasneverbeenexposedtobefore.
The dominant paradigm for LVCSR is the HMM, and we will focus o n this ap-
proach in this chapter. Previous chapters have introducedm ost of the core algorithms
usedinHMM-basedspeechrecognition. Ch.7introducedthek eyphoneticandphono-
logical notionsof phone,syllable, and intonation. Ch. 5 and Ch. 6 introducedthe use
of theBayes rule , theHidden Markov Model (HMM), theViterbialgorithm, and
the Baum-Welch training algorithm. Ch. 4 introduced the N-gramlanguage model

and theperplexity metric. In this chapter we begin with an overview of the archi tec-
ture for HMM speech recognition,offeran all-too-briefove rviewof signal processing
for feature extractionand the extractionof the importantM FCC features, and then in-
troduceGaussianacousticmodels. Wethencontinuewithhow Viterbidecodingworks
in the ASR context, and give a complete summary of the trainin g procedurefor ASR,
calledembedded training . Finally,weintroduceworderrorrate,thestandardevalua -

tionmetric. Thenextchapterwill continuewithsomeadvanc edASRtopics.
9.1 SpeechRecognitionArchitecture
The task of speech recognition is to take as input an acoustic waveform and produce
as output a string of words. HMM-based speech recognition sy stems view this task
using the metaphor of the noisy channel. The intuition of the noisy channel model Noisy channel
(see Fig. 9.2) is to treat the acoustic waveform as an “noisy” version of the string of

words, i.e.. a version that has been passed through a noisy co mmunications channel.
This channel introduces“noise” which makes it hard to recog nize the “true” string of
words. Our goal is then to build a model of the channelso that w e can ﬁgure out how
it modiﬁedthis“true”sentenceandhencerecoverit.
Theinsightof thenoisychannelmodelis thatif weknowhowth e channeldistorts
the source, we could ﬁnd the correct source sentence for a wav eform by taking every
possible sentence in the language, running each sentence th rough our noisy channel

model, and seeing if it matches the output. We then select the best matching source
sentenceasourdesiredsourcesentence.
Implementingthenoisy-channelmodelaswehaveexpressedi tinFig.9.2requires

DRAFT290 Chapter 9. AutomaticSpeechRecognition
noisy sentencesource sentencenoisy channel
decoder
Every happy family
In a hole in the ground
...
If music be the food of loveguess at source:noisy 1
noisy 2
noisy NIf music be 
the food of love...
If music be 
Figure9.2 The noisy channel model. We search through a huge space of pot ential “source”
sentences and choose the one which has the highest probabili ty of generating the “noisy” sen-
tence. Weneedmodelsofthepriorprobabilityofasourcesen tence(N-grams),theprobabilityof
wordsbeingrealizedascertainstringsofphones(HMMlexic ons),andtheprobabilityofphones
being realizedas acoustic or spectral features (Gaussian M ixture Models).
solutions to two problems. First, in order to pick the senten ce that best matches the
noisy input we will need a complete metric for a “best match”. Because speech is so
variable, an acoustic input sentence will never exactly mat ch any model we have for

thissentence. Aswehavesuggestedinpreviouschapters,we willuseprobabilityasour
metric. This makes the speech recognition problem a special case ofBayesian infer- Bayesian
ence,amethodknownsincetheworkofBayes(1763). Bayesianinfe renceorBayesian
classiﬁcation was appliedsuccessfully by the 1950sto lang uageproblemslike optical
characterrecognition(BledsoeandBrowning,1959)andtoa uthorshipattributiontasks
liketheseminalworkofMostellerandWallace(1964)ondete rminingtheauthorshipof

theFederalistpapers. Ourgoalwillbetocombinevariouspr obabilisticmodelstogeta
completeestimatefortheprobabilityofanoisyacousticob servation-sequencegivena
candidatesourcesentence. We canthen search throughthe sp ace of all sentences,and
choosethe sourcesentencewiththehighestprobability.
Second,sincethesetofallEnglishsentencesishuge,wenee danefﬁcientalgorithm
that will not search through all possible sentences, but onl y ones that have a good
chanceofmatchingtheinput. Thisisthe decoding orsearchproblem,whichwehave

already explored with the Viterbi decoding algorithm for HM Ms in Ch. 5 and Ch. 6.
Sincethesearchspaceissolargeinspeechrecognition,efﬁ cientsearchisanimportant
partofthetask,andwewill focusonanumberofareasinsearc h.
In the rest of this introduction we will review the probabili stic or Bayesian model
forspeechrecognitionthatweintroducedforpart-of-spee chtagginginCh.5. Wethen
introducethe variouscomponentsofamodernHMM-basedASRs ystem.
Recallthatthegoaloftheprobabilisticnoisychannelarch itectureforspeechrecog-
nitioncanbesummarizedasfollows:

“What is the most likely sentence out of all sentences in the l anguage L
givensome acousticinputO?”

DRAFTSection9.1. SpeechRecognitionArchitecture 291
We cantreat theacousticinput Oasasequenceofindividual“symbols”or“obser-
vations” (for example by slicing up the input every 10 millis econds, and representing
each slice by ﬂoating-point values of the energy or frequenc ies of that slice). Each
index then represents some time interval, and successive oiindicate temporally con-
secutiveslicesoftheinput(notethatcapitalletterswill standforsequencesofsymbols
andlower-caselettersforindividualsymbols):
O=o1,o2,o3,...,ot (9.1)
Similarly,wetreat asentenceasifit werecomposedofa stri ngofwords:
W=w1,w2,w3,...,wn (9.2)
Both of these are simplifying assumptions; for example divi ding sentences into
wordsissometimestooﬁneadivision(we’dliketomodelfact saboutgroupsofwords
ratherthanindividualwords)andsometimestoogrossadivi sion(weneedtodealwith
morphology). Usually in speech recognition a word is deﬁned by orthography (after
mapping every word to lower-case): oakis treated as a different word than oaks, but

theauxiliary can(“canyoutellme...”) istreatedasthesamewordasthenoun can(“i
needacanof...” ).
Theprobabilisticimplementationofourintuitionabove,t hen,canbeexpressedas:
ˆW=argmax
W∈LP(W|O) (9.3)
Recallthatthefunctionargmaxxf(x)means“thexsuchthatf(x)islargest”. Eq.9.3
is guaranteed to give us the optimal sentence W; we now need to make the equation
operational. That is, for a given sentence Wand acoustic sequence Owe need to
computeP(W|O). Recall that given any probability P(x|y), we can use Bayes’ rule to

ˆW=argmax
breakit downasfollows:
P(y)(9.4)
We sawin Ch.5 thatwecansubstitute(9.4)into(9.3)asfollo ws:
ˆW=argmax
W∈LP(O|W)P(W)
P(O)(9.5)
The probabilities on the right-hand side of (9.5) are for the most part easier to
compute than P(W|O). For example, P(W), the prior probability of the word string
itself is what is estimated by the N-gram language models of Ch. 4. And we will see
belowthat P(O|W)turnsouttobeeasytoestimateaswell. But P(O),theprobabilityof

P(y)(9.4)
ˆW=argmax
P(O)(9.5)
the acoustic observationsequence, is harderto estimate. L uckily, we can ignore P(O)
just as we saw in Ch. 5. Why? Since we are maximizing over all po ssible sentences,
we will becomputingP(O|W)P(W)
P(O)foreachsentencein the language. But P(O)doesn’t
changeforeachsentence! Foreachpotentialsentencewe are still examiningthe same
observations O, whichmust havethe sameprobability P(O). Thus:
ˆW=argmax
W∈LP(O|W)P(W) (9.6)

DRAFT292 Chapter 9. AutomaticSpeechRecognition
To summarize, the most probable sentence Wgiven some observation sequence
Ocan be computed by taking the product of two probabilities fo r each sentence, and
choosing the sentence for which this product is greatest. Th e general components of
the speech recognizer which compute these two terms have nam es;P(W), theprior
probability , is computed by the language model . whileP(O|W), theobservation Language model
likelihood ,iscomputedbythe acousticmodel . Acousticmodel
ˆW=argmax
W∈Llikelihood
P(O|W)prior
P(W) (9.7)
Thelanguagemodel(LM)prior P(W)expresseshowlikelyagivenstringofwords
is to be a source sentence of English. We have already seen in C h. 4 how to compute
suchalanguagemodelprior P(W)byusingN-gramgrammars. Recallthatan N-gram
grammarlets usassigna probabilityto asentencebycomputi ng:
P(wn
1)≈n
∏
k−N+1) (9.8)
This chapter will show how the HMM we covered in Ch. 6 can be use d to build

ˆW=argmax
P(W) (9.7)
P(wn
1)≈n
∏
an Acoustic Model (AM) which computesthe likelihood P(O|W). Given the AM and
LM probabilities,the probabilisticmodelcan be operation alizedin a searchalgorithm
soasto computethemaximumprobabilitywordstringforagiv enacousticwaveform.
Fig. 9.3 shows the componentsof an HMM speech recognizeras i t processes a single
utterance, indicating the computation of the prior and like lihood. The ﬁgure shows
therecognitionprocessinthreestages. Inthe featureextraction orsignalprocessing

stage, the acoustic waveform is sampled into frames(usually of 10, 15, or 20 mil-
liseconds) which are transformed into spectral features . Each time window is thus
representedby a vectorofaround39featuresrepresentingt his spectralinformationas
wellasinformationaboutenergyandspectralchange. Sec.9 .3givesan(unfortunately
brief)overviewofthefeatureextractionprocess.
In theacoustic modeling orphone recognition stage, we compute the likelihood
ofthe observedspectralfeaturevectorsgivenlinguisticu nits(words,phones,subparts

of phones). For example, we use Gaussian Mixture Model (GMM) classiﬁers to com-
pute for each HMM state q, corresponding to a phone or subphone, the likelihood of
a given feature vector given this phone p(o|q). A (simpliﬁed) way of thinking of the
output of this stage is as a sequence of probability vectors, one for each time frame,
eachvectorat eachtimeframecontainingthelikelihoodsth ateachphoneorsubphone
unitgeneratedthe acousticfeaturevectorobservationatt hat time.
Finally,inthe decoding phase,wetaketheacousticmodel(AM),whichconsistsof

thissequenceofacousticlikelihoods,plusanHMMdictiona ryofwordpronunciations,
combinedwith the languagemodel (LM) (generallyan N-gram grammar),and output
themostlikelysequenceofwords. AnHMMdictionary,aswewi llseeinSec.9.2,isa
listofwordpronunciations,eachpronunciationrepresent edbyastringofphones. Each
wordcanthenbethoughtofasanHMM,wherethephones(orsome timessubphones)
arestatesintheHMM,andtheGaussianlikelihoodestimator ssupplytheHMMoutput
likelihood function for each state. Most ASR systems use the Viterbi algorithm for

DRAFTSection9.2. ApplyingtheHiddenMarkovModeltoSpeech 293
decoding, speeding up the decoding with wide variety of soph isticated augmentations
suchaspruning,fast-match,andtree-structuredlexicons .
cepstral 
feature 
extraction
Gaussian 
Acoustic ModelMFCC features
phone 
likelihoodsHMM lexiconN-gram 
language 
model
Viterbi Decoder
if music be the food of love...
F
WP(W)
P(O|W)
Figure9.3 Schematic architecture for a (simpliﬁed)speech recognize r decoding a single sen-
tence. A real recognizer is more complex since various kinds of pruning and fast matches are
neededforefﬁciency. Thisarchitectureisonlyfordecodin g;wealsoneedaseparatearchitecture
for trainingparameters.
9.2 ApplyingtheHiddenMarkovModeltoSpeech
Let’s turn now to how the HMM model is applied to speech recogn ition. We saw in
Ch.6 thata HiddenMarkovModelischaracterizedbythefollo wingcomponents:
Q=q1q2...qN aset ofstates
A=a01a02...an1...annatransition probabilitymatrix A, eachaijrep-

O=o1o2...oN asetofobservations ,eachonedrawnfromavo-
cabularyV=v1,v2,...,vV.
B=bi(ot) A set of observation likelihoods: , also called
emission probabilities , each expressing the
probability of an observation otbeing generated
froma state i.
q0,qend aspecialstartandendstate whicharenotasso-
ciatedwithobservations.

DRAFT294 Chapter 9. AutomaticSpeechRecognition
Furthermore, the chapter introduced the Viterbialgorithm for decoding HMMs,
andtheBaum-Welch orForward-Backward algorithmfortrainingHMMs.
All of these facets of the HMM paradigm play a crucial role in A SR. We begin
here by discussing how the states, transitions, and observa tions map into the speech
recognitiontask. WewillreturntotheASRapplicationsofV iterbidecodinginSec.9.6.
TheextensionstotheBaum-Welchalgorithmsneededtodealw ithspokenlanguageare
coveredinSec. 9.4andSec.9.7.
Recall the examples of HMMs we saw earlier in the book. In Ch. 5 , the hid-
den states of the HMM were parts-of-speech, the observation s were words, and the
HMMdecodingtaskmappedasequenceofwordstoasequenceofp arts-of-speech. In
Ch. 6, the hidden states of the HMM were weather, the observat ions were ‘ice-cream
consumptions’, and the decoding task was to determine the we ather sequence from a
sequence of ice-cream consumption. For speech, the hidden s tates are phones, parts

of phones, or words, each observation is information about t he spectrum and energy
of the waveform at a point in time, and the decoding process ma ps this sequence of
acousticinformationtophonesandwords.
Theobservationsequenceforspeechrecognitionisasequen ceofacousticfeature
vectors. Eachacousticfeaturevectorrepresentsinformationsuch astheamountofen-
ergyindifferentfrequencybandsataparticularpointinti me. WewillreturninSec.9.3
tothenatureoftheseobservations,butfornowwe’llsimply notethateachobservation

consists of a vector of 39 real-valued features indicating s pectral information. Obser-
vationsaregenerallydrawnevery10milliseconds,so1 seco ndof speechrequires100
spectralfeaturevectors,eachvectoroflength39.
The hidden states of Hidden Markov Models can be used to model speech in a
number of different ways. For small tasks, like digit recognition , (the recognition of
the10digitwords zerothroughnine),orforyes-norecognition(recognitionofthetwo

wordsyesandno), we could build an HMM whose states correspondto entire wor ds.
Formostlargertasks,however,thehiddenstatesoftheHMMc orrespondtophone-like
units,andwordsare sequencesofthesephone-likeunits.
Let’s begin by describing an HMM model in which each state of a n HMM corre-
spondsto a single phone (if you’veforgottenwhat a phoneis, go back and look again
at the deﬁnition in Ch. 7). In such a model, a word HMM thus cons ists of a sequence
ofHMM states concatenatedtogether. Fig.9.4 showsa schema ticofthe structureofa
basicphone-stateHMMfortheword six.
3a

3a
G GH IStart0End5s4a22a33a44
a01a12a23a34a45
Figure9.4 An HMM for the word six, consisting of four emitting states, two non-emitting
states,and the transitionprobabilities A.The the observa tion probabilities Bare not shown.
Notethat onlycertain connectionsbetweenphonesexist in F ig. 9.4. In theHMMs
described in Ch. 6, there were arbitrary transitions betwee n states; any state could
transitionto anyother. Thiswasalso inprincipletrueofth e HMMsforpart-of-speech

3a
tagging in Ch. 5; although the probability of some tag transi tions was low, any tag

DRAFTSection9.2. ApplyingtheHiddenMarkovModeltoSpeech 295
couldinprinciplefollowanyothertag. UnlikeotherHMMapp lications,HMMmodels
for speech recognition do not allow arbitrary transitions. Instead, they place strong
constraints on transitions based on the sequential nature o f speech. Except in unusual
cases,HMMsforspeechdon’tallowtransitionsfromstatest ogotoearlierstatesinthe
word;in otherwords,statescantransitiontothemselvesor tosuccessivestates. Aswe
saw inCh.6,thiskindof left-to-right HMMstructureiscalleda Bakis network . Bakisnetwork
Themostcommonmodelusedforspeech,illustratedinasimpl iﬁedforminFig.9.4
is evenmore constrained,allowinga state to transition onl yto itself (self-loop)or to a
single succeeding state. The use of self-loops allows a sing le phone to repeat so as to
coveravariableamountoftheacousticinput. Phoneduratio nsvaryhugely,dependent
on the phone identify, the speaker’s rate of speech, the phon etic context, and the level

of prosodic prominence of the word. Looking at the Switchboa rd corpus, the phone
[aa] varies in length from 7 to 387 milliseconds (1 to 40 frame s), while the phone [z]
varies in durationfrom 7 milliseconds to more than 1.3 secon ds(130 frames) in some
utterances! Self-loopsthusallowa singlestate toberepea tedmanytimes.
For very simple speech tasks (recognizing small numbers of w ords such as the
10 digits), using an HMM state to represent a phone is sufﬁcie nt. In general LVCSR

tasks,however,amoreﬁne-grainedrepresentationisneces sary. Thisisbecausephones
can last over 1 second, i.e., over 100 frames, but the 100 fram es are not acoustically
identical. The spectral characteristicsof a phone, and the amount of energy,vary dra-
matically across a phone. For example, recall from Ch. 7 that stop consonants have
a closure portion, which has very little acoustic energy, fo llowed by a release burst.
Similarly,diphthongsarevowelswhoseF1andF2changesign iﬁcantly. Fig.9.5shows

these large changes in spectral characteristics over time f or each of the two phones in
theword“Ike”,ARPAbet [ayk].
Time (s)0.48152 0.93720305000Frequency (Hz)
ay k
Figure9.5 The two phones of the word ”Ike”, pronounced [ay k]. Note the c ontinuous
changes inthe [ay] vowel onthe left, as F2rises and F1falls, and the sharpdifferences between
the silence and release parts of the [k]stop.
To capture this fact about the non-homogeneous nature of pho nes over time, in

DRAFT296 Chapter 9. AutomaticSpeechRecognition
LVCSR we generally model a phonewith more than one HMM state. The most com-
monconﬁgurationistousethreeHMMstates,abeginning,mid dle,andendstate. Each
phone thus consists of 3 emitting HMM states instead of one (p lus two non-emitting
states at either end), as shown in Fig. 9.6. It is common to res erve the word modelor
phone model to referto theentire5-statephoneHMM,andusetheword HMMstate Phone model
HMMstate (orjuststateforshort)torefertoeachofthe3individualsubphoneHMMst ates.
fi311
beg1mid2Start0End4a22a33
a01a12a23a34
Figure9.6 A standard 5-state HMM model for a phone, consisting of three emitting states
(correspondingtothetransition-in,steadystate,andtra nsition-outregionsofthephone) andtwo
non-emitting states.
To build a HMM for an entire word using these more complex phon e models, we
can simply replace each phone of the word model in Fig. 9.4 wit h a 3-state phone
HMM. We replace the non-emitting start and end states for eac h phone model with

fi311
transitionsdirectly to the emitting state of the preceding and followingphone,leaving
onlytwo non-emittingstatesforthe entireword. Fig.9.7sh owstheexpandedword.
ih2ih0ih1Start End s2s0s1k2k0k1s2s0s1
Figure9.7 Acomposite wordmodel for “six”,[s ihks],formedbyconcate nating four phone
models, each withthree emittingstates.
Insummary,anHMMmodelofspeechrecognitionisparameteri zedby:
Q=q1q2...qN aset ofstatescorrespondingto subphones

resenting the probability for each subphone of
takingaself-loop orgoingtothe nextsubphone.
probability of a cepstral feature vector (observa-
tionot) beinggeneratedfromsubphonestate i.
Anotherway oflookingat the Aprobabilitiesandthe states Qis that togetherthey
represent a lexicon: a set of pronunciations for words, each pronunciation cons isting
ofaset ofsubphones,withtheorderofthesubphonesspeciﬁe dbythetransitionprob-

abilitiesA.
We have now covered the basic structure of HMM states for repr esenting phones
and words in speech recognition. Later in this chapter we wil l see further augmenta-
tions of the HMM word model shown in Fig. 9.7, such as the use of triphone models
whichmakeuseofphonecontext,andtheuseofspecialphones tomodelsilence. First,

DRAFTSection9.3. FeatureExtraction: MFCC vectors 297
though, we need to turn to the next component of HMMs for speec h recognition: the
observationlikelihoods. Andin orderto discussobservati onlikelihoods,we ﬁrst need
tointroducetheactualacousticobservations: featurevec tors. Afterdiscussingthesein
Sec. 9.3, we turn in Sec. 9.4 the acoustic model and details of observation likelihood
computation. Wethenre-introduceViterbidecodingandsho whowtheacousticmodel
andlanguagemodelare combinedtochoosethe bestsentence.
9.3 FeatureExtraction: MFCCvectors
Our goal in this sectionis to describehow we transformthe in putwaveforminto a se-
quenceofacoustic featurevectors ,eachvectorrepresentingtheinformationinasmall Feature vector
timewindowofthesignal. Whiletherearemanypossiblesuch featurerepresentations,
by far the most commonin speech recognitionis the MFCC, themel frequency cep- MFCC
stral coefﬁcients . These are based on the important idea of the cepstrum . We will Cepstrum

give a relatively high-level description of the process of e xtraction of MFCCs from a
waveform; we strongly encouragestudents interested in mor e detail to follow up with
a speechsignalprocessingcourse.
We begin by repeating from Sec. 7.4.2 the process of digitizi ng and quantizing an
analog speech waveform. Recall that the ﬁrst step in process ing speech is to convert
the analog representations (ﬁrst air pressure, and then ana log electric signals in a mi-
crophone),into a digitalsignal. Thisprocessof analog-to-digitalconversion has two

steps:sampling andquantization . A signal is sampled by measuring its amplitude Sampling
at a particular time; the sampling rate is the numberof samples taken per second. In Sampling rate
ordertoaccuratelymeasureawave,itisnecessarytohaveat leasttwosamplesineach
cycle: onemeasuringthepositivepartofthewaveandonemea suringthenegativepart.
More than two samples per cycle increases the amplitude accu racy, but less than two
sampleswill causethefrequencyofthewavetobecompletely missed. Thusthemaxi-

mum frequencywave that can be measured is one whose frequenc yis half the sample
rate(sinceeverycycleneedstwosamples). Thismaximumfre quencyforagivensam-
pling rate is called the Nyquist frequency . Most information in human speech is in Nyquist frequency
frequenciesbelow 10,000Hz; thus a 20,000 Hz sampling rate w ould be necessary for
completeaccuracy. Buttelephonespeechisﬁlteredbythesw itchingnetwork,andonly
frequenciesless than 4,000 Hz are transmitted by telephone s. Thus an 8,000 Hz sam-

pling rate is sufﬁcient for telephone-bandwidth speech like the Switchboard corpus.Telephone-
bandwidth
A16,000Hzsamplingrate(sometimescalled wideband )isoftenusedformicrophone Wideband
speech.

bandwidth
speech.
calledquantization becausethereisaminimumgranularity(thequantumsize)an dall Quantization
We refer to each sample in the digitized quantized waveform a sx[n], wherenis
an index over time. Now that we have a digitized, quantized re presentation of the

DRAFT298 Chapter 9. AutomaticSpeechRecognition
speech
signalMFCC 12
coefﬁcientspreC
emphasisDFT windowMel ﬁlterC
banklog IDFT deltas
energy 1 energy feature12 MFCC 
12  ∆ MFCC
12 ∆∆ MFCC
1 energy
1 ∆ energy
 1 ∆∆ energy
Figure9.8 Extractinga sequence of 39-dimensional MFCCfeature vecto rs from a quantized digitized waveform
waveform,we are readyto extractMFCC features. The sevenst epsofthis processare
shownin Fig.9.8andindividuallydescribedineachofthefo llowingsections.
9.3.1 Preemphasis
The ﬁrst stage in MFCC feature extraction is to boost the amou nt of energy in the
high frequencies. It turns out that if we look at the spectrum for voiced segments like
vowels,thereismoreenergyatthelowerfrequenciesthanth ehigherfrequencies. This
dropinenergyacrossfrequencies(whichiscalled spectraltilt )iscausedbythenature Spectral tilt
oftheglottalpulse. Boostingthehighfrequencyenergymak esinformationfromthese
higher formants more available to the acoustic model and imp roves phone detection
accuracy.

accuracy.
Thispreemphasisisdonebyusingaﬁlter1Fig.9.9showsanexampleofaspectral
slice from the ﬁrst author’s pronunciation of the single vow el [aa] before and after
preemphasis.
Frequency (Hz)0 22050Sound pressure level (dB /Hz)
–40–20020
–40–20020
(a) (b)
Figure9.9 Aspectral slice from the vowel [aa] before (a) andafter (b) p reemphasis.
9.3.2 Windowing
Recall that the goal of feature extraction is to provide spec tral features that can help

accuracy.
–40–20020
–40–20020
(a) (b)
usbuildphoneorsubphoneclassiﬁers. We thereforedon’twa ntto extractourspectral
features from an entire utterance or conversation, because the spectrum changes very
quickly. Technically, we say that speech is a non-stationary signal, meaning that its Non-stationary
statistical properties are not constant across time. Inste ad, we want to extract spectral
1Forstudents whohavehadsignalprocessing: thispreemphas is ﬁlter isaﬁrst-order high-pass ﬁlter. Inthe

time domain, with input x[n]and 0.9≤α≤1.0, the ﬁlter equation is y[n] =x[n]−αx[n−1].

DRAFTSection9.3. FeatureExtraction: MFCC vectors 299
features from a small windowof speech that characterizes a particular subphone and
for which we can make the (rough) assumption that the signal i sstationary (i.e. its Stationary
statistical propertiesareconstantwithinthisregion).
We’ll do this by using a window which is non-zero inside some r egion and zero
elsewhere,runningthiswindowacrossthespeechsignal,an dextractingthewaveform
insidethiswindow.
We can characterize such a windowing process by three parame ters: how wideis
thewindow(inmilliseconds),whatisthe offsetbetweensuccessivewindows,andwhat
is theshapeof the window. We call the speech extracted from each window a frame, Frame
andwe call thenumberofmillisecondsin theframethe framesize andthenumberof Frame size
millisecondsbetweentheleft edgesofsuccessivewindowst heframeshift . Frame shift
FRAME SIZE
25 msFRAME
 SHIFT
10 ms
Figure9.10 The windowing process, showing the frame shift and frame siz e, assuming a

FRAME SIZE
25 msFRAME
 SHIFT
10 ms
frame shift of 10ms, a frame size of 25 ms, and a rectangular wi ndow. After a ﬁgure by Bryan
Pellom.
The extraction of the signal takes place by multiplying the v alue of the signal at
timen,s[n], withthevalueofthewindowat time n,w[n]:
y[n] =w[n]s[n] (9.9)
Fig. 9.10 suggests that these window shapes are rectangular , since the extracted
windowedsignal looksjust like the originalsignal. Indeed the simplest windowis the
rectangular window. The rectangularwindow can cause problems, however ,because Rectangular

Pellom.
it abruptly cuts of the signal at its boundaries. These disco ntinuities create problems
whenwedoFourieranalysis. Forthisreason,amorecommonwi ndowusedinMFCC
extraction is the Hamming window, which shrinks the values of the signal toward Hamming
zeroatthewindowboundaries,avoidingdiscontinuities. F ig.9.11showsbothofthese
windows;theequationsareasfollows(assumingawindowtha t isLframeslong):

DRAFT300 Chapter 9. AutomaticSpeechRecognition
rectangular w [n] ={1 0≤n≤L−1
0 otherwise(9.10)
hamming w [n] ={
0.54−0.46cos(2πn
L)0≤n≤L−1
0 otherwise(9.11)
Time (s)0 0.0475896–0.50.4999
0
Rectangular windowHamming window
Time (s)0.00455938 0.0256563–0.48260.4999
0
Time (s)0.00455938 0.0256563–0.50.4999
0
Figure9.11 Windowing a portion of a pure sine wave with the rectangular a nd Hamming
windows.
9.3.3 DiscreteFourierTransform
The next step is to extract spectral information for our wind owed signal; we need to
know how much energy the signal contains at different freque ncybands. The tool for
extracting spectral information for discrete frequency ba nds for a discrete-time (sam-
pled)signalis the DiscreteFourierTransform orDFT.Discrete FourierTransform
DFT The input to the DFT is a windowed signal x[n]...x[m], and the output, for each of
Ndiscretefrequencybands,isacomplexnumber X[k]representingthemagnitudeand
phase of that frequency component in the original signal. If we plot the magnitude

againstthe frequency,we can visualizethe spectrum that we introducedin Ch. 7. For
example, Fig. 9.12 shows a 25 ms Hamming-windowed portion of a signal and its
spectrumascomputedbya DFT(withsomeadditionalsmoothin g).
WewillnotintroducethemathematicaldetailsoftheDFTher e,excepttonotethat
Fourieranalysisin generalrelieson Euler’sformula : Euler’sformula
ejθ=cosθ+jsinθ (9.12)

DRAFTSection9.3. FeatureExtraction: MFCC vectors 301
Time (s)0.0141752 0.039295–0.041210.04414
0
Frequency (Hz)0 8000Sound pressure level (dB /Hz)
–20020
(a) (b)
Figure9.12 (a) A 25 ms Hamming-windowed portion of a signal from the vowe l [iy] and (b)
itsspectrum computed by a DFT.
Asabriefreminderforthosestudentswhohavealreadyhadsi gnalprocessing,theDFT
isdeﬁnedasfollows:
X[k] =N−1
∑
n=0x[n]e−j2π
Nkn(9.13)
A commonly used algorithm for computing the DFT is the Fast Fourier Trans-
formorFFT. This implementation of the DFT is very efﬁcient, but only wo rks forFast FourierTransform
FFTvaluesofNwhicharepowersoftwo.
9.3.4 Mel ﬁlterbank and log
The results of the FFT will be information about the amount of energy at each fre-
quencyband. Humanhearing,however,isnotequallysensiti veatall frequencybands.
It is less sensitive at higher frequencies, roughly above 10 00 Hertz. It turns out that
modeling this property of human hearing during feature extr action improves speech

recognition performance. The form of the model used in MFCCs is to warp the fre-
quencies output by the DFT onto the melscale mentioned in Ch. 7. A mel(Stevens Mel
et al., 1937; Stevens and Volkmann, 1940) is a unit of pitch de ﬁned so that pairs of
soundswhichareperceptuallyequidistantinpitcharesepa ratedbyanequalnumberof
mels. ThemappingbetweenfrequencyinHertzandthemelscal eislinearbelow1000
Hz and the logarithmic above 1000 Hz. The mel frequency mcan be computed from
therawacousticfrequencyasfollows:
mel(f) =1127ln (1+f

700) (9.14)
DuringMFCCcomputation,thisintuitionisimplementedbyc reatingabankofﬁl-
terswhichcollectenergyfromeachfrequencyband,with10ﬁ ltersspacedlinearlybe-
low1000Hz,andtheremainingﬁltersspreadlogarithmicall yabove1000Hz. Fig.9.13
showsthebankoftriangularﬁltersthatimplementthisidea .
Finally, we take the log of each of the mel spectrum values. In general the human
response to signal level is logarithmic; humans are less sen sitive to slight differences
inamplitudeathighamplitudesthanatlowamplitudes. Inad dition,usingalogmakes

DRAFT302 Chapter 9. AutomaticSpeechRecognitionJ1
JK
JL ... Mel Spectrum 0
M N N N
N
OAmplitude
Frequency (Hz)0 1000 2000 3000 4000
Figure9.13 The Mel ﬁlter bank, after Davis and Mermelstein (1980). Each triangular ﬁlter
collects energy from a given frequency range. Filters are sp aced linearly below 1000 Hz, and
logarithmicallyabove 1000 Hz.
thefeatureestimateslesssensitiveto variationsin input (forexamplepowervariations
duetothespeaker’smouthmovingcloserorfurtherfromthem icrophone).
9.3.5 The Cepstrum: InverseDiscrete FourierTransform
Whileitwouldbepossibletousethemelspectrumbyitselfas afeaturerepresentation
forphonedetection,thespectrumalsohassomeproblems,as wewillsee. Forthisrea-
son,thenextstepinMFCCfeatureextractionisthecomputat ionofthe cepstrum . The Cepstrum
cepstrumhasanumberofusefulprocessingadvantagesandal sosigniﬁcantlyimproves
phonerecognitionperformance.
One way to think about the cepstrum is as a useful way of separa ting thesource

andﬁlter. Recall from Sec. 7.4.6 that the speech waveform is created w hen a glottal
source waveform of a particular fundamental frequency is pa ssed through the vocal
tract, which because of its shape has a particular ﬁltering c haracteristic. But many
characteristicsoftheglottal source(itsfundamentalfrequency,thedetailsoftheglottal
pulse, etc) are not important for distinguishing different phones. Instead, the most
useful informationfor phone detectionis the ﬁlter, i.e. the exact position of the vocal

tract. If we knew the shape ofthe vocaltract, we would knowwh ichphonewas being
produced. This suggests that useful features for phone dete ction would ﬁnd a way to
deconvolve (separate) the source and ﬁlter and show us only t he vocal tract ﬁlter. It
turnsoutthatthecepstrumisonewayto dothis.
(a) (b) (c)
Figure9.14 PLACEHOLDER FIGURE. The magnitude spectrum (a), the log mag nitude spectrum (b), and the

(a) (b) (c)
cepstrum (c). From Taylor (2008). The two spectra have a smoo thed spectral enveloped laid on top of them to help
visualize the spectrum.

DRAFTSection9.3. FeatureExtraction: MFCC vectors 303
For simplicity, let’s ignore the pre-emphasis and mel-warp ing that are part of the
deﬁnition of MFCCs, and look just at the basic deﬁnition of th e cepstrum. The cep-
strum can be thought of as the spectrum of the log of the spectrum . This may sound
confusing. But let’s beginwith theeasy part: the log of thespectrum . Thatis, thecep-
strum beginswith a standard magnitude spectrum, such as the one for a vowel shown
in Fig. 9.14(a) from Taylor (2008). We then take the log, i.e. replace each amplitude
valueinthemagnitudespectrumwith itslog,asshowninFig. 9.14(b).
Thenextstep istovisualizethe logspectrum asif itself were awaveform . Inother
words, consider the log spectrum in Fig. 9.14(b). Let’s imag ine removing the axis
labelsthat tell usthat thisis a spectrum(frequencyonthe x -axis)and imaginethat we
aredealingwithjustanormalspeechsignalwithtimeonthex -axis. Nowwhatcanwe
say about the spectrum of this ‘pseudo-signal’? Notice that there is a high-frequency

repetitivecomponentin this wave: small wavesthat repeata bout8 times in each1000
along the x-axis, for a frequency of about 120 Hz. This high-f requencycomponent is
causedbythefundamentalfrequencyofthesignal,andrepre sentsthelittlepeaksinthe
spectrum at each harmonic of the signal. In addition, there a re some lower frequency
componentsin this‘pseudo-signal’;forexamplethe envelo peorformantstructurehas
aboutfourlargepeaksin thewindow,fora muchlowerfrequen cy.
Fig. 9.14(c) shows the cepstrum : the spectrum that we have been describing of

the log spectrum. This cepstrum (the word cepstrum is formed by reversing the ﬁrst
lettersofspectrum )isshownwith samplesalongthex-axis. Thisisbecausebytaking
the spectrum of the log spectrum, we have left the frequencyd omain of the spectrum,
andgonebacktothetimedomain. Itturnsoutthatthecorrect unitofacepstrumisthe
sample.
Examining this cepstrum, we see that there is indeed a large p eak around 120,
corresponding to the F0 and representing the glottal pulse. There are other various

sample.
components at lower values on the x-axis. These represent th e vocal tract ﬁlter (the
positionofthetongueandtheotherarticulators). Thusifw eareinterestedindetecting
phones, we can make use of just the lower cepstral values. If w e are interested in
detectingpitch,wecanuse thehighercepstralvalues.
For the purposes of MFCC extraction, we generally just take t he ﬁrst 12 cepstral
values. These 12 coefﬁcients will represent information so lely about the vocal tract
ﬁlter,cleanlyseparatedfrominformationabouttheglotta lsource.

It turns out that cepstral coefﬁcients have the extremely us eful property that the
variance of the different coefﬁcients tends to be uncorrela ted. This is not true for the
spectrum, where spectral coefﬁcients at different frequen cybands are correlated. The
factthatcepstralfeaturesareuncorrelatedmeans,aswewi llseeinthenextsection,that
the Gaussian acoustic model (the Gaussian Mixture Model, or GMM) doesn’t have to
represent the covariance between all the MFCC features, whi ch hugely reduces the
numberofparameters.

Forthosewhohavehadsignalprocessing,thecepstrumismor eformallydeﬁnedas
theinverseDFTofthelogmagnitudeoftheDFTofasignal ,henceforawindowed
frameofspeech x[n]:

DRAFT304 Chapter 9. AutomaticSpeechRecognition
c[n] =N−1
∑
n=0log(⏐⏐⏐⏐⏐N−1
∑
Nkn⏐⏐⏐⏐⏐)
ej2π
Nkn(9.15)
9.3.6 Deltasand Energy
The extraction of the cepstrum via the Inverse DFT from the pr evious section results
in 12cepstral coefﬁcientsforeachframe. We nextadda thirt eenthfeature: the energy
fromthe frame. Energycorrelateswithphoneidentityandso isa usefulcueforphone
detection (vowels and sibilants have more energy than stops , etc). The energyin a Energy
frameisthesumovertimeofthepowerofthesamplesinthefra me;thusforasignal x
ina windowfromtime sample t1to timesample t2,theenergyis:
Energy =t2
∑
t=t1x2[t] (9.16)
Anotherimportantfact aboutthespeechsignalisthat it isn otconstantfromframe
to frame. This change, such as the slope of a formant at its tra nsitions, or the nature
of the change from a stop closure to stop burst, can provide a u seful cue for phone
identity. For this reasonwe also add featuresrelated to the changein cepstral features
overtime.

Energy =t2
∑
overtime.
We do this by adding for each of the 13 features (12 cepstral fe atures plus en-
ergy) adeltaorvelocityfeature, and a double delta oracceleration feature. Each of Deltafeature
Double delta the 13 delta features represents the change between frames i n the correspondingcep-
stral/energy feature, while each of the 13 double delta feat ures represents the change
betweenframesinthe correspondingdeltafeatures.
A simple way to compute deltas would be just to compute the dif ference between

overtime.
frames; thus the delta value d(t)for a particular cepstral value c(t)at timetcan be
estimatedas:
d(t) =c(t+1)−c(t−1)
2(9.17)
Insteadofthissimple estimate, however,it ismorecommont o makemoresophis-
ticatedestimatesoftheslope,usinga widercontextoffram es.
9.3.7 Summary: MFCC
Afteraddingenergy,andthendeltaanddouble-deltafeatur estothe12cepstralfeatures,
we endupwith39MFCC features:
12cepstralcoefﬁcients
12deltacepstralcoefﬁcients
12doubledeltacepstralcoefﬁcients
1energycoefﬁcient
1deltaenergycoefﬁcient

2(9.17)
1doubledeltaenergycoefﬁcient
39MFCC features

DRAFTSection9.4. ComputingAcousticLikelihoods 305
Again, one of the most useful facts about MFCC features is tha t the cepstral coef-
ﬁcients tend to be uncorrelated,which will turn out to make o ur acoustic model much
simpler.
9.4 ComputingAcousticLikelihoods
ThelastsectionshowedhowwecanextractMFCCfeaturesrepr esentingspectralinfor-
mation from a waveﬁle, and produce a 39-dimensional vector e very 10 milliseconds.
We are now ready to see how to computethe likelihood of these f eature vectorsgiven
an HMM state. Recall from Ch. 6 that this output likelihood is computed by the B
probability function of the HMM. Given an individual state qiand an observation ot,
theobservationlikelihoodsin Bmatrixgaveus p(ot|qi), whichwecalled bt(i).
For part-of-speech tagging in Ch. 5, each observation otis a discrete symbol (a
word)andwe cancomputethe likelihoodof an observationgiv ena part-of-speechtag
just by counting the number of times a given tag generates a gi ven observation in the

training set. But for speech recognition, MFCC vectors are r eal-valued numbers; we
can’t compute the likelihood of a given state (phone) genera ting an MFCC vector by
counting the number of times each such vector occurs (since e ach one is likely to be
unique).
Inbothdecodingandtraining,weneedanobservationlikeli hoodfunctionthatcan
computep(ot|qi)onreal-valuedobservations. Indecoding,wearegivenanob servation
otandweneedtoproducetheprobability p(ot|qi)foreachpossibleHMMstate,sowe

unique).
canchoosethemostlikelysequenceofstates. Oncewehaveth isobservationlikelihood
Bfunction, we need to ﬁgure out how to modify the Baum-Welch al gorithm of Ch. 6
totrainit aspartoftrainingHMMs.
9.4.1 VectorQuantization
One way to make MFCC vectors look like symbols that we could co unt is to build a
mappingfunctionthat mapseach input vector into one of a sma ll numberof symbols.
Then we could just compute probabilities on these symbols by counting, just as we
didforwordsinpart-of-speechtagging. Thisideaofmappin ginputvectorstodiscrete

quantizedsymbolsiscalled vectorquantization orVQ(Gray,1984). AlthoughvectorVectorquantization
VQquantizationistoosimpletoactastheacousticmodelinmod ernLVCSRsystems,itis
a useful pedagogicalstep, and plays an importantrole in var iousareas of ASR, so we
useit to beginourdiscussionofacousticmodeling.
In vector quantization, we create the small symbol set by map ping each training
feature vector into a small number of classes, and then we rep resent each class by a
discrete symbol. More formally, a vector quantization syst em is characterized by a

codebook ,aclusteringalgorithm ,andadistance metric .
Acodebook isalist ofpossibleclasses, aset ofsymbolsconstitutinga vocabulary Codebook
V={v1,v2,...,vn}. For each symbol vkin the codebook we list a prototype vector , Prototype vector
alsoknownasa codeword ,whichisaspeciﬁcfeaturevector. Forexampleifwechoose Codeword
to use 256 codewords we could represent each vector by a value from 0 to 255; (this

DRAFT306 Chapter 9. AutomaticSpeechRecognition
is referred to as 8-bit VQ, since we can represent each vector by a single 8-bit value).
Eachofthese256valueswouldbeassociatedwitha prototype featurevector.
The codebook is created by using a clustering algorithm to cluster all the feature Clustering
vectorsin the trainingset into the 256 classes. Thenwe chos e a representativefeature
vector from the cluster, and make it the prototype vector or c odeword for that cluster.
K-meansclustering isoftenused,butwewon’tdeﬁneclusteringhere;seeHuange tal.K-meansclustering
(2001)orDudaetal. (2000)fordetaileddescriptions.
Oncewe’vebuiltthecodebook,foreachincomingfeaturevec tor,wecompareit to
each of the 256 prototype vectors, select the one which is clo sest (by some distance
metric),andreplacetheinputvectorbytheindexofthisprototype vector. Aschematic
ofthisprocessis showninFig. 9.15.
TheadvantageofVQisthatsincethereareaﬁnitenumberofcl asses,foreachclass

vk,wecancomputetheprobabilitythatitisgeneratedbyagive nHMMstate/sub-phone
bysimplycountingthenumberoftimesitoccursinsometrain ingset whenlabeledby
thatstate, andnormalizing.
Figure9.15 Schematic architecture of the (trained) vector quantizati on (VQ) process for
choosing a symbol vqforeachinput feature vector. The vector iscompared toeach codeword in
thecodebook, theclosestentry(bysomedistancemetric)is selected,andtheindexoftheclosest
codeword isoutput.

Both the clustering process and the decoding process requir e adistance metric Distance metric
ordistortion metric, that speciﬁes how similar two acoustic feature vect ors are. The
distancemetricisusedtobuildclusters,toﬁndaprototype vectorforeachcluster,and
tocompareincomingvectorstotheprototypes.
The simplest distance metric for acoustic feature vectors i sEuclidean distance .Euclideandistance
Euclidean distance is the distance in N-dimensionalspace b etween the two points de-
ﬁnedbythetwovectors. Inpracticeweusethephrase‘Euclid eandistance’eventhough

we actually often use the square of the Euclidean distance. T hus given a vector xand
a vectoryof length D, the (square of the) Euclideandistance between t hem is deﬁned
as:
deuclidean(x,y) =D
∑
i=1(xi−yi)2(9.18)

DRAFTSection9.4. ComputingAcousticLikelihoods 307
The (squared) Euclidean distance described in (9.18) (and s hown for two dimen-
sionsinFig.9.16)isalsoreferredtoasthesum-squarederr or,andcanalsobeexpressed
usingthevectortransposeoperatoras:
deuclidean(x,y) = (x−y)T(x−y) (9.19)
Figure9.16 Euclidean distance in two dimensions; by the Pythagorean th eorem, the distance
between twopoints ina plane x= (x1,y1)andy= (x2,y2)d(x,y) =√
(x1−x2)2+(y1−y2)2.
The Euclidean distance metric assumes that each of the dimen sions of a feature
vector are equally important. But actually each of the dimen sions have very different
variances. If a dimension tends to have a lot of variance, the n we’d like it to count
less in the distance metric; a large differencein a dimensio nwith low variance should
countmorethan a largedifferencein a dimensionwith highva riance. A slightly more
complex distance metric, the Mahalanobis distance , takes into account the differentMahalanobisdistance
variancesofeachofthedimensions.

If we assume that each dimension iof the acoustic feature vectors has a variance
σ2
i,thentheMahalanobisdistanceis:
dmahalanobis(x,y) =D
∑
i=1(xi−yi)2
σ2
i(9.20)
For those readers with more backgroundin linear algebra her e’s the general form
ofMahalanobisdistance,whichincludesafullcovariancem atrix(covariancematrices
will bedeﬁnedbelow):
dmahalanobis(x,y) = (x−y)TΣ−1(x−y) (9.21)
In summary, when decodinga speech signal, to computean acou stic likelihood of
a feature vector otgiven an HMM state qjusing VQ, we compute the Euclidean or

σ2
∑
σ2
i(9.20)
Mahalanobisdistancebetweenthefeaturevectorandeachof theNcodewords,choose
theclosestcodeword,gettingthecodewordindex vk. Wethenlookupthelikelihoodof
thecodewordindex vkgiventheHMMstate jinthepre-computed Blikelihoodmatrix
deﬁnedbytheHMM:

DRAFT308 Chapter 9. AutomaticSpeechRecognition
ˆbj(ot) =bj(vk)s.t.vkiscodewordofclosest vectorto ot (9.22)
Since VQ is so rarely used, we don’t use up space here giving th e equations for
modifying the EM algorithm to deal with VQ data; instead, we d efer discussion of
EM training of continuous input parameters to the next secti on, when we introduce
Gaussians.
9.4.2 GaussianPDFs
Vectorquantizationhastheadvantageofbeingextremelyea sytocomputeandrequires
very little storage. Despite these advantages, vector quan tization turns out not to be a
goodmodelofspeech. Asmallnumberofcodewordsisinsufﬁci enttocapturethewide
variabilityinthe speechsignal. Speechissimplynota cate gorical,symbolicprocess.
Modern speech recognition algorithmstherefore do not use v ector quantization to
computeacoustic likelihoods. Instead,theyare based onco mputingobservationprob-
abilities directly on the real-valued, continuous input fe ature vector. These acoustic

models are based on computing a probability density function orpdfover a contin-Probability
density function
uous space. By far the most common method for computing acous tic likelihoods is
theGaussianMixtureModel (GMM)pdfs,althoughneuralnetworks,supportvectorGaussian MixtureModel
GMMmachines(SVMs) andconditionalrandomﬁelds(CRFs) areals oused.
Let’sbeginwith the simplest use of Gaussian probabilityes timators, slowly build-
ingupthemoresophisticatedmodelsthatare used.
UnivariateGaussians

TheGaussian distribution, also known as the normal distribution , is the bell-curve Gaussian
Normaldistribution function familiar from basic statistics. A Gaussian distri bution is a function parame-
terized by a mean, or average value, and a variance, which characterizes the average Mean
Variance spread or dispersal from the mean. We will use µto indicate the mean, and σ2to
indicatethevariance,givingthe followingformulafora Ga ussianfunction:
f(x|µ,σ) =1√
2πσ2exp(−(x−µ)2
2σ2) (9.23)

Recall from basic statistics that the mean of a random variab leXis the expected
valueofX. Foradiscretevariable X,thisistheweightedsumoverthevaluesof X(for
a continuousvariable,it istheintegral):
µ=E(X) =N
∑
i=1p(Xi)Xi (9.24)
The variance of a random variable Xis the weigthed squared average deviation
fromthemean:
σ2=E(Xi−E(X))2=N
∑
i=1p(Xi)(Xi−E(X))2(9.25)
Whena Gaussianfunctionis usedas a probabilitydensityfun ction,the areaunder
thecurveisconstrainedtobeequaltoone. Thentheprobabil itythatarandomvariable

DRAFTSection9.4. ComputingAcousticLikelihoods 309
−4 −3 −2 −1 0 1 2 3 400.20.40.60.811.21.41.6
m=0,s=.5
m=1,s=1
m=−1,s=0.2
m=0,s=0.3
Figure9.17 Gaussian functions withdifferent means andvariances.
takes on any particular range of values can be computed by sum ming the area under
thecurveforthatrangeofvalues. Fig.9.18showstheprobab ilityexpressedbythearea
underanintervalofa Gaussian.
−4 −3 −2 −1 0 1 2 3 400.050.10.150.20.250.30.350.4Probability Density← P(shaded region) = .341
Figure9.18 AGaussianprobabilitydensityfunction,showingaregionf rom0to1withatotal
probabilityof.341. ThusforthissampleGaussian, theprob abilitythatavalue ontheXaxislies
between 0and1is .341.
We can use a univariate Gaussian pdf to estimate the probabil ity that a particular
HMMstate jgeneratesthevalueofasingledimensionofafeaturevector byassuming
thatthepossiblevaluesof(thisonedimensionofthe)obser vationfeaturevector otare

DRAFT310 Chapter 9. AutomaticSpeechRecognition
normally distributed. In other words we represent the obser vation likelihood function
bj(ot)foronedimensionoftheacousticvectorasaGaussian. Takin g,forthemoment,
ourobservationasasinglerealvaluednumber(asingleceps tralfeature),andassuming
that each HMM state jhas associated with it a mean value µjand variance σ2
j, we
computethelikelihood bj(ot)viatheequationfora Gaussianpdf:
bj(ot) =1√
2πσ2
jexp(
−(ot−µj)2
2σ2
j)
(9.26)
Eq. 9.26 shows us how to compute bj(ot), the likelihood of an individualacoustic
observationgivenasingleunivariateGaussianfromstate jwithitsmeanandvariance.
We cannowusethisprobabilityin HMMdecoding.
Butﬁrst we needtosolvethetrainingproblem;howdowecompu tethismeanand
variance of the Gaussian for each HMM state qi? Let’s start by imaginingthe simpler
situation of a completely labeled training set, in which eac h acoustic observation was
labeledwiththe HMMstate thatproducedit. Insucha trainin gset, we couldcompute

the mean of each state by just taking the average of the values for eachotthat corre-
sponded to state i, as show in (9.27). The variance could just be computed from t he
sum-squarederrorbetweeneachobservationandthemean,as shownin(9.28).
ˆµi=1
TT
∑
t=1ots.t.qtisstatei (9.27)
ˆσ2
j=1
TT
∑
t=1(ot−µi)2s.t.qtisstatei (9.28)
But since states are hiddenin an HMM, we don’tknowexactly wh ich observation
vectorotwas produced by which state. What we would like to do is assign each ob-
servation vector otto every possible state i, prorated by the probabilitythat the HMM

ˆµi=1
TT
∑
ˆσ2
j=1
TT
∑
was in state iat timet. Luckily, we already know how to do this prorating; the prob-
ability of being in state iat timetwas deﬁned in Ch. 6 as ξt(i), and we saw how to
compute ξt(i)as part of the Baum-Welch algorithm using the forward and bac kward
probabilities. Baum-Welch is an iterative algorithm, and w e will need to do the prob-
ability computation of ξt(i)iteratively since getting a better observation probabilit yb
will also help us be moresure of the probability ξof beingin a state at a certain time.

Thuswegiveequationsforcomputinganupdatedmeanandvari anceˆµandˆσ2:
ˆµi=∑T
t=1ξt(i)ot
∑T
t=1ξt(i)(9.29)
ˆσ2
i=∑T
t=1ξt(i)(ot−µi)2
∑T
t=1ξt(i)(9.30)
Eq.9.29andEq.9.30arethenusedintheforward-backward(B aum-Welch)training
oftheHMM.Aswewillsee,thevaluesof µiandσiareﬁrstsettosomeinitialestimate,
whichisthenre-estimateduntilthenumbersconverge.

DRAFTSection9.4. ComputingAcousticLikelihoods 311
MultivariateGaussians
Eq. 9.26 shows how to use a Gaussian to compute an acoustic lik elihood for a single
cepstral feature. Since an acoustic observation is a vector of 39 features, we’ll need
to use a multivariate Gaussian, which allows us to assign a pr obability to a 39-valued
vector. Where a univariateGaussian is deﬁnedbya mean µand a variance σ2, a mul-
tivariateGaussianis deﬁnedbya meanvector ⃗µofdimensionalityD anda covariance
matrixΣ,deﬁnedbelow. Aswediscussedintheprevioussection,fora typicalcepstral
featurevectorin LVCSR,D is39:
f(⃗x|⃗µ,Σ) =1
(2π)D
2|Σ|1
2exp(
−1
2(x−µ)TΣ−1(x−µ))
(9.31)
The covariance matrix Σcaptures the variance of each dimension as well as the
covariancebetweenanytwodimensions.
Recall again from basic statistics that the covariance of tw o random variables X
andYistheexpectedvalueoftheproductoftheiraveragedeviati onsfromthemean:
Σ=E[(X−E(X))(Y−E(Y)]) =N
∑
i=1p(XiYi)(Xi−E(X))(Yi−E(Y)) (9.32)

(2π)D
2|Σ|1
2exp(
−1
(9.31)
∑
Thusfor a givenHMM state with meanvector µjand covariancematrix Σj, and a
givenobservationvector ot, themultivariateGaussianprobabilityestimate is:
bj(ot) =1
(2π)D
2|Σ|1
2exp(
−1
2(ot−µj)TΣ−1
j(ot−µj))
(9.33)
The covariance matrix Σjexpresses the variance between each pair of feature di-
mensions. Suppose we made the simplifying assumption that f eatures in different di-
mensions did not covary, i.e., that there was no correlation between the variances of

∑
bj(ot) =1
(2π)D
2|Σ|1
2exp(
−1
j(ot−µj))
(9.33)
different dimensions of the feature vector. In this case, we could simply keep a dis-
tinct variancefor each featuredimension. It turnsout that keepinga separate variance
for each dimension is equivalent to having a covariance matr ix that is diagonal, i.e. Diagonal
non-zero elements only appear along the main diagonal of the matrix. The main di-
agonalofsuchadiagonalcovariancematrixcontainsthevar iancesofeachdimension,
σ2
1,σ2
2,...σ2
D;

σ2
1,σ2
2,...σ2
D;
Let’s look at some illustrations of multivariate Gaussians , focusing on the role of
the full versus diagonal covariance matrix. We’ll explore a simple multivariate Gaus-
sianwithonly2dimensions,ratherthanthe39thataretypic alinASR.Fig.9.19shows
three different multivariate Gaussians in two dimensions. The leftmost ﬁgure shows
a Gaussian with a diagonal covariance matrix, in which the va riances of the two di-
mensionsareequal. Fig.9.20shows3contourplotscorrespo ndingtotheGaussiansin
Fig. 9.19; each is a slice throughthe Gaussian. The leftmost graphin Fig. 9.20 shows

a slice through the diagonal equal-variance Gaussian. The s lice is circular, since the
variancesareequalinboththeXandY directions.
ThemiddleﬁgureinFig.9.19showsaGaussianwithadiagonal covariancematrix,
but where the variancesare not equal. It is clear from this ﬁg ure, and especially from

DRAFT312 Chapter 9. AutomaticSpeechRecognition
−4−2024
−4−202400.050.10.150.20.250.30.35
−4−2024
−4−2024
−4−202400.20.40.60.81
(a) (b) (c)
Figure9.19 Three different multivariate Gaussians in two dimensions. The ﬁrst two have
diagonalcovariancematrices,onewithequalvarianceinth etwodimensions[1 0
0 1]
,thesecond
with different variances in the two dimensions,[.6 0
0 2]
, and the third with non-zero elements
inthe off-diagonal of the covariance matrix:[
1.8
.8 1]
.
−3 −2 −1 0 1 2 3−3−2−10123
(a) (b) (c)
Figure9.20 The same three multivariate Gaussians as in the previous ﬁgu re. From left to
right, a diagonal covariance matrix with equal variance, di agonal with unequal variance, and
nondiagonal covariance. Withnon-diagonal covariance, kn owingthevalue ondimensionXtells
you something about thevalue on dimension Y.
thecontoursliceshowinFig.9.20,thatthevarianceismore than3timesgreaterinone
dimensionthanthe other.

.
(a) (b) (c)
The rightmost graph in Fig. 9.19 and Fig. 9.20 shows a Gaussia n with a non-
diagonal covariance matrix. Notice in the contour plot in Fi g. 9.20 that the contour
isnotlinedupwith thetwo axes,asit is inthe othertwo plots . Becauseofthis, know-
ingthevalueinonedimensioncanhelpinpredictingthevalu eintheotherdimension.
Thushavinganon-diagonalcovariancematrixallowsustomo delcorrelationsbetween
thevaluesofthefeaturesinmultipledimensions.
AGaussianwithafullcovariancematrixisthusamorepowerf ulmodelofacoustic

likelihoodthanonewithadiagonalcovariancematrix. Andi ndeed,speechrecognition
performanceis better using full-covarianceGaussianstha ndiagonal-covarianceGaus-
sians. But there are two problems with full-covariance Gaus sians that makes them
difﬁcult to use in practice. First, they are slow to compute. A full covariance matrix
hasD2parameters, where a diagonal covariance matrix has only D. This turns out to
makealargedifferenceinspeedinrealASRsystems. Second, afullcovariancematrix

hasmanymoreparametersandhencerequiresmuchmoredatato trainthanadiagonal
covariance matrix. Using a diagonal covariance model means we can save room for

DRAFTSection9.4. ComputingAcousticLikelihoods 313
using our parameters for other things like triphones (conte xt-dependentphones) to be
introducedinSec. 10.3.
For this reason, in practice most ASR systems use diagonal co variance. We will
assumediagonalcovarianceforthe remainderofthissectio n.
Eq.9.33canthusbesimpliﬁedtotheversionin(9.34)inwhic hinsteadofacovari-
ance matrix, we simply keep a mean and variance for each dimen sion. Eq. 9.34 thus
describes how to estimate the likelihood bj(ot)of aD-dimensional feature vector ot
givenHMMstate j, usinga diagonal-covariancemultivariateGaussian.
bj(ot) =D
∏
d=11√
2πσ2
jdexp(
−1
2[(otd−µjd)2
σjd2])
(9.34)
Traininga diagonal-covariancemultivariateGaussian is a simple generalizationof
training univariateGaussians. We’ll do the same Baum-Welc h training, where we use
the value of ξt(i)to tell us the likelihood of being in state iat timet. Indeed, we’ll
useexactlythesameequationasin(9.30),exceptthatnowwe aredealingwithvectors

bj(ot) =D
∏
d=11√
2πσ2
jdexp(
−1
σjd2])
(9.34)
instead of scalars; the observation otis a vector of cepstral features, the mean vector
⃗µis a vector of cepstral means, and the variance vector⃗σ2
iis a vector of cepstral
variances.
ˆµi=∑T
∑T
t=1ξt(i)(9.35)
ˆσ2
i=∑T
t=1ξt(i)(ot−µi)(ot−µi)T
∑T
t=1ξt(i)(9.36)
GaussianMixtureModels
The previoussubsection showed that we can use a multivariat e Gaussian model to as-
sign a likelihood score to an acoustic feature vector observ ation. This models each
dimension of the feature vector as a normal distribution. Bu t a particular cepstral fea-

ˆµi=∑T
∑T
ˆσ2
i=∑T
∑T
ture might have a very non-normal distribution; the assumpt ion of a normal distribu-
tionmaybetoostronganassumption. Forthisreason,weofte nmodeltheobservation
likelihoodnotwithasinglemultivariateGaussian,butwit haweightedmixtureofmul-
tivariate Gaussians. Such a model is called a Gaussian Mixture Model orGMM.Gaussian MixtureModel
GMMEq. 9.37 shows the equation for the GMM function; the resulti ng function is the sum
ofMGaussians. Fig.9.21showsanintuitionofhowamixtureofGa ussianscanmodel
arbitraryfunctions.
f(x|µ,Σ) =M
∑
k=1ck1√

∑
k=1ck1√
2π|Σk|exp[(x−µk)TΣ−1(x−µk)] (9.37)
Eq.9.38showsthedeﬁnitionoftheoutputlikelihoodfuncti onbj(ot)
bj(ot) =M
∑
m=1cjm1√
2π|Σjm|exp[(x−µjm)TΣ−1
jm(ot−µjm)] (9.38)

DRAFT314 Chapter 9. AutomaticSpeechRecognition
−4 −3 −2 −1 0 1 2 3 400.10.20.30.40.50.60.70.8
Figure9.21 Anarbitraryfunction approximated by a mixture of 3gaussia ns.
Let’sturntotrainingtheGMMlikelihoodfunction. Thismay seemhardtodo;how
canwetrainaGMMmodelifwedon’tknowinadvancewhichmixtu reissupposedto
accountforwhichpart of eachdistribution? Recall that a si ngle multivariateGaussian
couldbetrainedevenif wedidn’tknowwhichstate accounted foreachoutput,simply
byusingthe Baum-Welchalgorithmto tell usthe likelihoodo fbeingin eachstate jat
timet. ItturnsoutthesametrickwillworkforGMMs;wecanuseBaum -Welchtotell
us the probability of a certain mixture accounting for the ob servation, and iteratively
updatethisprobability.
We usedthe ξfunctionabovetohelpuscomputethestate probability. Bya nalogy
withthisfunction,let’sdeﬁne ξtm(j)tomeantheprobabilityofbeinginstate jattime
twith themth mixture component accounting for the output observation ot. We can
computeξtm(j)asfollows:
ξtm(j) =∑i=1Nαt−1(j)aijcjmbjm(ot)βt(j)

αT(F)(9.39)
Now if we had the values of ξfrom a previous iteration of Baum-Welch, we can
useξtm(j)torecomputethemean,mixtureweight,andcovarianceusing thefollowing
equations:
ˆµim=∑T
t=1ξtm(i)ot
∑T
t=1∑M
m=1ξtm(i)(9.40)
ˆcim=∑T
t=1ξtm(i)
∑T
t=1∑M
k=1ξtk(i)(9.41)
ˆΣim=∑T
t=1ξt(i)(ot−µim)(ot−µim)T
∑T
t=1∑M
k=1ξtm(i)(9.42)

DRAFTSection9.4. ComputingAcousticLikelihoods 315
9.4.3 Probabilities,log probabilities anddistance funct ions
Up to now, all the equationswe have givenfor acoustic modeli nghave used probabil-
ities. It turnsout, however,that a logprobability (orlogprob)is mucheasier to work Logprob
with than a probability. Thus in practice throughout speech recognition (and related
ﬁelds)we computelog-probabilitiesratherthanprobabili ties.
One major reason that we can’t use probabilities is numeric u nderﬂow. To com-
pute a likelihood for a whole sentence, say, we are multiplyi ng many small prob-
ability values, one for each 10ms frame. Multiplying many pr obabilities results in
smaller and smaller numbers, leading to underﬂow. The log of a small number like
.00000001 =10−8, on the other hand,is a nice easy-to-work-with-numberlike −8. A
second reason to use log probabilities is computational spe ed. Instead of multiplying
probabilities, we add log-probabilities, and adding is fas ter than multiplying. Log-

probabilities are particularly efﬁcient when we are using G aussian models, since we
canavoidexponentiating.
Thus for example for a single multivariate diagonal-covari ance Gaussian model,
insteadofcomputing:
bj(ot) =D
∏
d=11√
2πσ2
jdexp(
−1
2(otd−µjd)2
σ2
jd)
(9.43)
we wouldcompute
logbj(ot) =−1
2D
∑
d=1[
log(2π)+σ2
jd+(otd−µjd)2
σ2
jd]
(9.44)
With some rearrangementof terms, we can rewrite this equati onto pull out a constant
C:
logbj(ot) =C−1
2D
∑
d=1(otd−µjd)2
σ2
jd(9.45)
whereCcanbeprecomputed:
C=−1
2D
∑
d=1(

bj(ot) =D
∏
d=11√
2πσ2
jdexp(
−1
σ2
jd)
(9.43)
2D
∑
d=1[
σ2
jd]
(9.44)
C:
2D
∑
σ2
jd(9.45)
C=−1
2D
∑
d=1(
jd)
(9.46)
Insummary,computingacousticmodelsinlogdomainmeansam uchsimplercom-
putation,muchofwhichcanbe precomputedforspeed.
The perceptive reader may have noticed that equation (9.45) looks very much like
theequationforMahalanobisdistance(9.20). Indeed,onew aytothinkaboutGaussian
logprobsisasjust a weighteddistancemetric.
A furtherpointaboutGaussian pdfs,forthose readerswith c alculus. Althoughthe
equations for observation likelihood such as (9.26) are mot ivated by the use of Gaus-
sianprobabilitydensityfunctions,thevaluestheyreturn fortheobservationlikelihood,

bj(ot), are not technically probabilities; they may in fact be grea ter than one. This is
because we are computingthe value of bj(ot)at a single point, rather than integrating
overaregion. WhilethetotalareaundertheGaussianPDFcur veisconstrainedtoone,

DRAFT316 Chapter 9. AutomaticSpeechRecognition
the actual value at any point could be greater than one. (Imag ine a very tall skinny
Gaussian;thevaluecouldbegreaterthanoneatthecenter,a lthoughtheareaunderthe
curveis still 1.0). If we were integratingovera region,we w ould be multiplyingeach
point by its width dx, which would bring the value down below one. The fact that the
Gaussian estimate is not a true probabilitydoesn’tmatter f or choosingthe most likely
HMMstate,sincewearecomparingdifferentGaussians,each ofwhichismissingthis
dxfactor.
Insummary,thelastfewsubsectionsintroducedGaussianmo delsforacoustictrain-
ing in speech recognition. Beginning with simple univariat e Gaussian, we extended
ﬁrst to multivariate Gaussians to deal with the multidimens ionality acoustic feature
vectors. We then introduced the diagonal covariance simpli ﬁcation of Gaussians, and
thenintroducedGaussiansmixtures(GMMs).
9.5 TheLexiconandLanguageModel
Sincepreviouschaptershadextensivediscussionsofthe N-gramlanguagemodel(Ch.4)

dxfactor.
and the pronunciationlexicon (Ch. 7), in this section we jus t brieﬂy recall them to the
reader.
LanguagemodelsforLVCSRtendto betrigramsorevenfourgra ms;goodtoolkits
areavailabletobuildandmanipulatethem(Stolcke,2002;Y oungetal.,2005). Bigrams
and unigram grammars are rarely used for large-vocabulary a pplications. Since tri-
gramsrequirehugeamountsofspace,however,languagemode lsformemory-constrained
applications like cell phones tend to use smaller contexts ( or use compression tech-

reader.
niques). As we will discuss in Ch. 24, some simple dialogue ap plications take ad-
vantage of their limited domain to use very simple ﬁnite-sta te or weighted ﬁnite-state
grammars.
Lexicons are simply lists of words, with a pronunciationfor each word expressed
as a phone sequence. Publicly available lexicons like the CM U dictionary (CMU,
1993)canbeusedtoextractthe64,000wordvocabulariescom monlyusedforLVCSR.
Mostwordshavea singlepronunciation,althoughsomewords suchashomonymsand

grammars.
frequent function words may have more; the average number of pronunciations per
word in most LVCSR systems seems to range from 1 to 2.5. Sec. 10 .5.3 in Ch. 10
discussestheissue ofpronunciationmodeling.
9.6 SearchandDecoding
We are now very close to having described all the parts of a com plete speech recog-
nizer. Wehaveshownhowtoextractcepstralfeaturesforafr ame,andhowtocompute
the acoustic likelihood bj(ot)for that frame. We also know how to represent lexical

knowledge, that each word HMM is composed of a sequence of pho ne models, and
each phonemodelof a set ofsubphonestates. Finally, in Ch. 4 we showedhow to use
N-gramstobuildamodelofwordpredictability.

DRAFTSection9.6. SearchandDecoding 317
Inthissectionweshowhowtocombineallofthisknowledgeto solvetheproblem
ofdecoding : combiningall these probabilityestimators to producethe most probable Decoding
string of words. We can phrase the decoding question as: ‘Giv en a string of acoustic
observations,howshouldwechoosethestringofwordswhich hasthehighestposterior
probability?’
Recallfromthebeginningofthechapterthenoisychannelmo delforspeechrecog-
nition. Inthismodel,weuseBayesrule,withtheresultthat thebestsequenceofwords
is the one that maximizes the product of two factors, a langua ge model prior and an
acousticlikelihood:
ˆW=argmax
P(W) (9.47)
Now that we have deﬁned both the acoustic model and language m odel we are
ready to see how to ﬁnd this maximum probability sequence of w ords. First, though,
it turns out that we’ll need to make a modiﬁcation to Eq. 9.47, because it relies on
someincorrectindependenceassumptions. Recall thatwetr ainedamultivariateGaus-

ˆW=argmax
sian mixture classiﬁer to compute the likelihood of a partic ular acoustic observation
(a frame) given a particular state (subphone). By computing separate classiﬁers for
each acoustic frame and multiplying these probabilities to get the probability of the
whole word, we are severely underestimating the probabilit y of each subphone. This
is because there is a lot of continuity across frames; if we we re to take into account
the acousticcontext,we wouldhavea greaterexpectationfo ra givenframeand hence

could assign it a higher probability. We must thereforerewe ight the two probabilities.
We do this by adding in a language model scaling factor orLMSF, also called the LMSF
languageweight . Thisfactorisanexponentonthelanguagemodelprobabilit yP(W).
BecauseP(W)islessthanoneandtheLMSFisgreaterthanone(between5and 15,in
manysystems),thishastheeffectofdecreasingthevalueof the LMprobability:
ˆW=argmax
W∈LP(O|W)P(W)LMSF(9.48)
Reweightingthelanguagemodelprobability P(W)inthiswayrequiresustomake

ˆW=argmax
one more change. This is because P(W)has a side-effect as a penalty for inserting
words. It’s simplest to see this in the case of a uniform langu age model, where every
word in a vocabulary of size |V|has an equal probability1
|V|. In this case, a sentence
withNwordswillhavealanguagemodelprobabilityof1
|V|foreachofthe Nwords,for
a totalpenaltyofN
|V|. Thelarger Nis(themorewordsinthe sentence),themoretimes
this1
Vpenalty multiplier is taken, and the less probable the sente nce will be. Thus if

this1
(on average) the language model probability decreases (cau sing a larger penalty), the
decoder will prefer fewer, longer words. If the language mod el probability increases
(largerpenalty),thedecoderwillprefermoreshorterword s. ThusouruseofaLMSFto
balancetheacousticmodelhastheside-effectofdecreasin gthewordinsertionpenalty.
Tooffsetthis, weneedtoaddbackina separate wordinsertion penalty :Word insertionpenalty

DRAFT318 Chapter 9. AutomaticSpeechRecognition
ˆW=argmax
W∈LP(O|W)P(W)LMSFWIPN(9.49)
Sinceinpracticeweuse logprobs,thegoalofourdecoderis:
ˆW=argmax
W∈LlogP(O|W)+LMSF×logP(W)+N×logWIP (9.50)
Nowthatwehaveanequationtomaximize,let’slookathowtod ecode. It’sthejob
of a decoder to simultaneously segment the utterance into wo rds and identify each of
these words. This task is made difﬁcult by variation, both in terms of how words are
pronounced in terms of phones, and how phones are articulate d in acoustic features.
Just togiveanintuitionofthedifﬁcultyoftheproblemimag ineamassivelysimpliﬁed
versionofthespeechrecognitiontask,inwhichthedecoder isgivenaseriesofdiscrete
phones. In such a case, we would know what each phone was with p erfect accuracy,
and yet decoding is still difﬁcult. For example, try to decod e the following sentence
from the (hand-labeled)sequence of phonesfrom the Switchb oardcorpus(don’tpeek
ahead!):
[ayd ihshherdsahm thih ngaxb awmuhvih ngrihsenl ih]

ahead!):
The answer is in the footnote.2The task is hard partly because of coarticulation
andfastspeech(e.g.,[d]fortheﬁrstphoneof just!). Butit’salsohardbecausespeech,
unlike English writing, has no spaces indicating word bound aries. The true decoding
task, in which we have to identify the phones at the same time a s we identify and
segmentthewords,isofcoursemuchharder.
For decoding,we will start with the Viterbi algorithmthat w e introducedin Ch. 6,
in the domain of digit recognition , a simple task with a vocabulary size of 11 (the

numbersonethroughninepluszeroandoh).
Recallthe basiccomponentsofanHMMmodelforspeechrecogn ition:
takingaself-loop or goingto the nextsubphone.
Together, QandAimplement a pronunciation
lexicon, an HMM state graph structure for each
wordthat thesystemiscapableofrecognizing.

The HMM structure for each word comes from a lexicon of word pr onunciations.
Generally we use an off-the-shelf pronunciation dictionar y such as the free CMUdict
dictionarydescribedinCh.7. Recall frompage295thattheH MMstructureforwords
2Ijust heard something about moving recently.

DRAFTSection9.6. SearchandDecoding 319
in speech recognitionis a simple concatenationof phone HMM s, each phone consist-
ingof3subphonestates,whereeverystate hasexactlytwotr ansitions: a self-loopand
alooptothenextphones. ThustheHMMstructureforeachdigi twordinourdigitrec-
ognizeris computedsimply by taking the phonestring fromth e dictionary,expanding
each phone into 3 subphones, and concatenating together. In addition, we generally
addanoptionalsilencephoneattheendofeachword,allowin gthepossibilityofpaus-
ing between words. We usually deﬁne the set of states Qfrom some version of the
ARPAbet,augmentedwithsilencephones,andexpandedtocre atethreesubphonesfor
eachphone.
TheAandBmatrices for the HMM are trained by the Baum-Welch algorithm in
theembedded training procedure that we will describe in Sec. 9.7. For now we’ll
assumethatthese probabilitieshavebeentrained.
Fig. 9.22 shows the resulting HMM for digit recognition. Not e that we’ve added

non-emittingstartandendstates,withtransitionsfromth eendofeachwordtotheend
state, and a transition from the end state back to the start st ate to allow for sequences
ofdigits. Notealso theoptionalsilencephonesat theendof eachword.
Digit recognizersoftendon’tuse word probabilities,sinc e in manydigit situations
(phone numbers or credit card numbers) each digit may have an equal probability of
appearing. But we’ve included transition probabilities in to each word in Fig. 9.22,
mainlytoshowwheresuchprobabilitieswouldbeforotherki ndsofrecognitiontasks.

As it happens,there are cases wheredigit probabilitiesdo m atter,such as in addresses
(whichareoftenlikelyto endin0 or00)orin cultureswheres ome numbersarelucky
andhencemorefrequent,suchasthe luckynumber‘8’in Chine se.
Now that we have an HMM, we can use the same forward and Viterbi algorithms
that we introduced in Ch. 6. Let’s see how to use the forward al gorithm to generate
P(O|W), the likelihood of an observation sequence Ogiven a sequence of words W;

we’ll use the single word “ﬁve”. In order to compute this like lihood, we need to sum
over all possible sequences of states; assuming ﬁvehas the states [f], [ay], and [v], a
10-observationsequenceincludesmanysequencessuchasth e following:
f ay ay ay ay v v v v v
f f ay ay ay ay v v v v
f f f f ay ay ay ay v v
f f ay ay ay ay ay ay v v
f f ay ay ay ay ay ay ay v
f f ay ay ay ay ay v v v
...
The forward algorithm efﬁciently sums over this large numbe r of sequences in
O(N2T)time.
Let’s quickly review the forward algorithm. It is a dynamic p rogramming algo-

...
rithm,i.e. analgorithmthat usesa tableto store intermedi atevaluesasit buildsupthe
probability of the observation sequence. The forward algor ithm computes the obser-
vation probability by summing over the probabilities of all possible paths that could
generatetheobservationsequence.
Eachcelloftheforwardalgorithmtrellis αt(j)orforward [t,j]representstheproba-
bilityofbeinginstate jafterseeingtheﬁrst tobservations,giventheautomaton λ. The

DRAFT320 Chapter 9. AutomaticSpeechRecognition
Figure9.22 An HMM for the digit recognition task. A lexicon speciﬁes the phone sequence,
andeachphoneHMMiscomposedofthreesubphones eachwithaG aussianemissionlikelihood
model. Combining these and adding an optional silence at the end of each word, results in a
single HMMfor thewhole task. Notethe transitionfromthe En dstatetothe Startstate toallow
digit sequences of arbitrarylength.
value of each cell αt(j)is computed by summing over the probabilities of every path
αt(j) =P(o1,o2...ot,qt=j|λ) (9.51)
αt(j) =N
∑
i=1αt−1(i)aijbj(ot) (9.52)
The threefactorsthat are multipliedin Eq˙ 9.52in extendin gthe previouspathsto

αt(j) =N
∑

DRAFTSection9.6. SearchandDecoding 321
αt−1(i)thepreviousforwardpathprobability fromtheprevioustimestep
Thealgorithmis describedinFig.9.23.
∑
∑
Figure9.23 The forwardalgorithmfor computing likelihoodof observat ion sequence givena
word model. a[s,s′]is the transition probability from current state sto next state s′, andb[s′,ot]
is the observation likelihood of s’givenot. The observation likelihood b[s′,ot]is computed by
theacoustic model .

∑
∑
Let’s see a trace of the forward algorithm running on a simpli ﬁed HMM for the
single word ﬁvegiven 10 observations; assuming a frame shift of 10ms, this c omes to
100ms. TheHMMstructureisshownverticallyalongthelefto fFig.9.24,followedby
the ﬁrst 3 time-steps of the forward trellis. The complete tr ellis is shown in Fig. 9.6,
togetherwith Bvaluesgivingavectorofobservationlikelihoodsforeachf rame. These
likelihoodscouldbe computedbyanyacoustic model(GMMsor other);in thisexam-

plewe’vehand-createdsimplevaluesforpedagogicalpurpo ses.
Let’s now turn to the question of decoding. Recall the Viterb i decodingalgorithm
fromourdescriptionof HMMsin Ch. 6. TheViterbi algorithmr eturnsthe most likely
state sequence (which is not the same as the most likely word s equence, but is often a
goodenoughapproximation)intime O(N2T).
Each cell of the Viterbi trellis, vt(j)represents the probabilitythat the HMM is in
statejafter seeing the ﬁrst tobservations and passing through the most likely state

sequence q1...qt−1, given the automaton λ. The value of each cell vt(j)is computed
by recursively taking the most probable path that could lead us to this cell. Formally,
eachcellexpressesthefollowingprobability:
vt(j) =P(q0,q1...qt−1,o1,o2...ot,qt=j|λ) (9.53)
Like other dynamic programming algorithms, Viterbi ﬁlls ea ch cell recursively.
Given that we had already computed the probability of being i n every state at time
t−1,WecomputetheViterbiprobabilitybytakingthemostprob ableoftheextensions

DRAFT322 Chapter 9. AutomaticSpeechRecognition
Figure9.24 The ﬁrst 3 time-steps of the forward trellis computation for the word ﬁve. TheA
transitionprobabilitiesareshownalongtheleftedge; the Bobservationlikelihoods areshownin
Fig.9.6.
V 0 0 0.008 0.0093 0.0114 0.00703 0.00345 0.00306 0.00206 0.00 117
AY 0 0.04 0.054 0.0664 0.0355 0.016 0.00676 0.00208 0.000532 0. 000109
F 0.8 0.32 0.112 0.0224 0.00448 0.000896 0.000179 4.48e-05 1. 12e-05 2.8e-06
Time 1 2 3 4 5 6 7 8 9 10
f 0.8 f 0.8 f 0.7 f 0.4 f 0.4 f 0.4 f 0.4 f 0.5 f 0.5 f 0.5
ay 0.1 ay 0.1 ay 0.3 ay 0.8 ay 0.8 ay 0.8 ay 0.8 ay 0.6 ay 0.5 ay 0.4
B v 0.6 v 0.6 v 0.4 v 0.3 v 0.3 v 0.3 v 0.3 v 0.6 v 0.8 v 0.9
p 0.4 p 0.4 p 0.2 p 0.1 p 0.1 p 0.1 p 0.1 p 0.1 p 0.3 p 0.3
iy 0.1 iy 0.1 iy 0.3 iy 0.6 iy 0.6 iy 0.6 iy 0.6 iy 0.5 iy 0.5 iy 0.4
Figure9.25 The forward trellis for 10 frames of the word ﬁve, consisting of 3 emitting states ( f,ay,v), plus non-

emitting start and end states (not shown). The bottom half of the table gives part of the Bobservation likelihood
vector for the observation oat each frame, p(o|q)for each phone q.Bvalues are created by hand for pedagogical
purposes. ThistableassumestheHMMstructurefor ﬁveshowninFig.9.24,eachemittingstatehavinga.5loopback
ofthepathsthatleadtothecurrentcell. Foragivenstate qjattimet,thevalue vt(j)is
computedas:
i=1vt−1(i)aijbj(ot) (9.54)
ThethreefactorsthataremultipliedinEq.9.54forextendi ngthepreviouspathsto

vt−1(i)thepreviousViterbi pathprobability fromtheprevioustimestep
aijthetransitionprobability frompreviousstate qitocurrentstate qj
Fig.9.26showstheViterbialgorithm,repeatedfromCh.6.
Recall that the goal of the Viterbi algorithm is to ﬁnd the bes t state sequence q=
(q1q2q3...qT)given the set of observations o= (o1o2o3...oT). It needs to also ﬁnd

DRAFTSection9.6. SearchandDecoding 323
Figure9.26 Viterbi algorithm for ﬁnding optimal sequence of hidden sta tes. Given an ob-
servation sequence of words and an HMM (as deﬁned by the AandBmatrices), the algorithm
returns the state-path through the HMM which assigns maximu m likelihood to the observation
sequence. a[s′,s]is the transition probability from previous state s′to current state s, andbs(ot)

istheobservationlikelihoodof sgivenot. Notethatstates0andFarenon-emittingstartandend
states.
theprobabilityofthisstatesequence. NotethattheViterb ialgorithmisidenticaltothe
forward algorithm except that it takes the MAX over the previ ous path probabilities
whereforwardtakestheSUM.
Fig. 9.27 shows the computation of the ﬁrst three time-steps in the Viterbi trellis
corresponding to the forward trellis in Fig. 9.24. We have ag ain used the made-up

states.
probabilitiesfor the cepstral observations; here we also f ollowcommonconventionin
notshowingthezerocellsintheupperleftcorner. Notethat onlythemiddlecellinthe
thirdcolumndiffersfromViterbitoforward. Fig.9.6shows thecompletetrellis.
Note the difference between the ﬁnal values from the Viterbi and forward algo-
rithms for this (made-up) example. The forward algorithm gi ves the probability of
the observationsequence as .00128, which we get by summing t he ﬁnal column. The
Viterbialgorithmgivestheprobabilityoftheobservation sequencegiventhebestpath,

which we get from the Viterbi matrix as .000493. The Viterbi p robability is much
smaller than the forward probability, as we should expect si nce Viterbi comes from a
singlepath,wheretheforwardprobabilityisthe sumoveral l paths.
The real usefulness of the Viterbi decoder, of course, lies i n its ability to decode
a string of words. In order to do cross-word decoding, we need to augment the A
matrix, which only has intra-word state transitions, with t he inter-word probability of

transitioning from the end of one word to the beginning of ano ther word. The digit
HMM model in Fig. 9.22 showed that we could just treat each wor d as independent,
anduse onlythe unigramprobability. Higher-order N-gramsare muchmorecommon.
Fig. 9.29, for example, shows an augmentationof the digit HM M with bigram proba-

DRAFT324 Chapter 9. AutomaticSpeechRecognition
Figure9.27 The ﬁrst 3 time-steps of the viterbi trellis computation for the word ﬁve. TheA
Fig.9.6. Inthiscomputationwemakethesimplifyingassump tionthattheprobabilityofstarting
instate1(phone [f])is1.0
V 0 0 0.008 0.0072 0.00672 0.00403 0.00188 0.00161 0.000667 0. 000493
AY 0 0.04 0.048 0.0448 0.0269 0.0125 0.00538 0.00167 0.000428 8 .78e-05
Bv 0.6 v 0.6 v 0.4 v 0.3 v 0.3 v 0.3 v 0.3 v 0.6 v 0.8 v 0.9
Figure9.28 The Viterbi trellis for 10 frames of the word ﬁve, consisting of 3 emitting states ( f,ay,v), plus non-

bilities.
A schematic of the HMM trellis for such a multi-word decoding task is shown
in Fig. 9.30. The intraword transitions are exactly as shown in Fig. 9.27. But now

bilities.
between words we’ve added a transition. The transition prob abilityon this arc, rather
than coming from the Amatrix inside each word, comes from the language model
P(W).
Once the entire Viterbi trellis has been computed for the utt erance, we can start
from the most-probable state at the ﬁnal time step and follow the backtrace pointers
backwardstogetthemostprobablestringofstates,andhenc ethemostprobablestring
of words. Fig. 9.31 shows the backtrace pointers being follo wed back from the best

P(W).
state, which happens to be at w2, eventually through wNandw1, resulting in the ﬁnal
wordstring w1wN···w2.
The Viterbi algorithm is much more efﬁcient than exponentia lly running the for-
ward algorithm for each possible word string. Nonetheless, it is still slow, and much
modern research in speech recognition has focused on speedi ng up the decoding pro-

DRAFTSection9.6. SearchandDecoding 325
iyiyiyzzz rrr owowowahahahwww nnn
uwuwuwttt
...p( one | two ) p( two | one )p( one | one )
p( two | two )p( one | zero )
p( zero | zero )p( zero | one )p( two | zero )
p( zero | two )
Figure9.29 A bigram grammar network for the digit recognition task. The bigrams give the
probability of transitioningfrom the endof one wordtothe b eginning of the next.
P
QRSTU
1
V
2
VW
VX
VYw1w
Z
w2P( wN | w1 )
P( w2 | w1 )
P( w1 | w1 )
Figure9.30 A schematic of the HMM Viterbi trellis for a bigram language m odel. The in-
traword transitions are the same as in Fig. 9.27. Between wor ds, a potential transition is added
(shownjustfrom w1asadarkdashedline)fromtheendstateofeachwordtothebeg inningstate
of every word, labeledwiththe bigram probability of the wor dpair.
cess. For example in practice in large-vocabulary recognit ion we do not consider all
possible words when the algorithm is extending paths from on e state-column to the

next. Instead, low-probabilitypaths are prunedat each time step and not extended to Pruning
thenextstate column.
This pruning is usually implemented via beam search (Lowerre, 1968). In beam Beamsearch
search, at each time t, we ﬁrst compute the probability of the best (most-probable )
state/path D. We then prune away any state which is worse than Dby some ﬁxed

DRAFT326 Chapter 9. AutomaticSpeechRecognition
[
\
]^_`
1
a
2
ab
ac
d d dd d daew1wf
w2ac g h
Figure9.31 Viterbi backtrace in the HMM trellis. The backtrace starts i n the ﬁnal state, and
results inabest phone stringfrom whicha wordstringis deri ved.
threshold ( beam width )θ. We can talk about beam-search in both the probability Beamwidth
and negative log probability domain. In the probability dom ain any path/state whose
probability is less than θ∗Dis pruned away; in the negative log domain, any path
whosecostisgreaterthen θ+Dispruned. Beamsearchisimplementedbykeepingfor
each time step an active list of states. Only transitionsfrom these wordsare extended Activelist
whenmovingtothenexttimestep.
Making this beam search approximation allows a signiﬁcant s peed-up at the cost
of a degradation to the decoding performance. Huang et al. (2 001) suggest that em-
pirically a beam size of 5-10% of the search space is sufﬁcien t; 90-95% of the states

arethusnotconsidered. Becauseinpracticemostimplement ationsofViterbiusebeam
search, some of the literature uses the term beam search ortime-synchronous beam
searchinsteadofViterbi.
9.7 EmbeddedTraining
We turn now to see how an HMM-based speech recognition system is trained. We’ve
already seen some aspects of training. In Ch. 4 we showed how t o train a language
model. In Sec. 9.4, we saw how GMM acousticmodelsare trained byaugmentingthe
EM algorithm to deal with training the means, variances, and weights. We also saw

how posterior AM classiﬁers like SVMs or neural nets could be trained, although for
neuralnetswehaven’tyetseenhowwegettrainingdatainwhi cheachframeislabeled
witha phoneidentity.
InthissectionwecompletethepictureofHMMtrainingbysho winghowthisaug-
mented EM training algorithmﬁts into the whole process of tr aining acoustic models.
Forreview,herearethe threecomponentsofthe acousticmodel :

DRAFTSection9.7. EmbeddedTraining 327
Q=q1q2...qN thesubphones representedasa set of states
A=a01a02...an1...annasubphone transition probability matrix A,
eachaijrepresenting the probability for each
subphone of taking a self-loop or going to the
next subphone. Together, QandAimplement
apronunciation lexicon , an HMM state graph
structure for each word that the system is capa-
bleofrecognizing.
Wewillassumethatthepronunciationlexicon,andthustheb asicHMMstategraph
structure for each word, is pre-speciﬁed as the simple linea r HMM structures with
loopbacks on each state that we saw in Fig. 9.7 and Fig. 9.22. I n general, speech
recognitionsystemsdonotattempttolearnthestructureof theindividualwordHMMs.
Thus we only need to train the Bmatrix, and we need to train the probabilities of

the non-zero (self-loop and next-subphone) transitions in theAmatrix. All the other
probabilitiesinthe Amatrixare set tozeroandneverchange.
The simplest possible training method, is hand-labeled isolated word training,
in which we train separate the BandAmatrices for the HMMs for each word based
on hand-aligned training data. We are given a training corpu s of digits, where each
instanceofaspokendigitisstoredinawaveﬁle,andwiththe startandendofeachword
andphonehand-segmented.Givensuchahand-labeleddataba se,wecancomputethe B

Gaussiansobservationlikelihoodsandthe Atransitionprobabilitiesbymerelycounting
in the training data! The Atransition probability are speciﬁc to each word, but the B
Gaussianswouldbesharedacrosswordsifthesamephoneoccu rredinmultiplewords.
Unfortunately,hand-segmentedtraining data is rarely use d in training systems for
continuousspeech. Onereasonisthatit isveryexpensiveto use humanstohand-label
phonetic boundaries; it can take up to 400 times real time (i. e. 400 labeling hours
to label each 1 hour of speech). Another reason is that humans don’t do phonetic

labeling very well for units smaller than the phone; people a re bad at consistently
ﬁndingtheboundariesofsubphones. ASRsystemsaren’tbett erthanhumansatﬁnding
boundaries,buttheirerrorsareat least consistentbetwee nthetrainingandtest sets.
Forthisreason,speechrecognitionsystemstraineachphon eHMMembeddedinan
entire sentence, and the segmentation and phone alignment a re done automatically as
part of the training procedure. This entire acoustic model t raining processis therefore

calledembeddedtraining . Handphonesegmentationdostillplaysomerole,however,Embeddedtraining
forexampleforbootstrappinginitial systemsfor discrimi native(SVM; non-Gaussian)
likelihoodestimators,orfortaskslikephonerecognition .
Inordertotrainasimpledigitssystem,we’llneedatrainin gcorpusofspokendigit
sequences. For simplicity assume that the training corpus i s separated into separate
waveﬁles, each containing a sequence of spoken digits. For e ach waveﬁle, we’ll need

to knowthe correctsequenceof digitwords. We’ll thusassoc iatewith each waveﬁlea

DRAFT328 Chapter 9. AutomaticSpeechRecognition
transcription(astringofwords). We’ll alsoneedapronunc iationlexiconanda phone-
set, deﬁning a set of (untrained) phone HMMs. From the transc ription, lexicon, and
phoneHMMs, we can build a “whole sentence”HMM for each sente nce, as shown in
Fig.9.32.
Figure9.32 The input tothe embedded trainingalgorithm; a waveﬁle of sp oken digits witha corresponding tran-
scription. ThetranscriptionisconvertedintoarawHMM,re adytobealignedandtrainedagainstthecepstralfeatures
extractedfrom the waveﬁle.
WearenowreadytotrainthetransitionmatrixAandoutputli kelihoodestimatorB
fortheHMMs. ThebeautyoftheBaum-Welch-basedparadigmfo rembeddedtraining
of HMMs is that this is all the training data we need. In partic ular, we don’t need
phoneticallytranscribeddata. We don’tevenneedtoknowwh ereeachwordstartsand
ends. The Baum-Welch algorithm will sum over all possible se gmentations of words
andphones,using ξj(t), the probabilityofbeingin state jat timetandgeneratingthe
observationsequenceO.

We will, however, need an initial estimate for the transitio n and observationprob-
abilitiesaijandbj(ot). The simplest way to do this is with a ﬂat start. In ﬂat start, Flat start
weﬁrstset tozeroanyHMMtransitionsthatwewanttobe‘stru cturallyzero’,suchas
transitionsfromlater phonesback to earlier phones. The γprobabilitycomputationin
Baum-Welchincludesthepreviousvalueof aij,sothosezerovalueswillneverchange.
Then we make all the rest of the (non-zero)HMM transitions eq uiprobable. Thus the

two transitions out of each state (the self-loop and the tran sition to the following sub-
phone) each would have a probability of 0.5. For the Gaussian s, a ﬂat start initializes
the mean and variance for each Gaussian identically, to the g lobal mean and variance
fortheentiretrainingdata.
Now we have initial estimates for the AandBprobabilities. For a standard Gaus-
sian HMM system, we now run multiple iterations of the Baum-W elch algorithm on

DRAFTSection9.7. EmbeddedTraining 329
theentiretrainingset. EachiterationmodiﬁestheHMMpara meters,andwestopwhen
the system converges. During each iteration, as discussed i n Ch. 6, we compute the
forward and backward probabilitiesfor each sentence given the initial AandBproba-
bilities,andusethemtore-estimatethe AandBprobabilities. Wealsoapplythevarious
modiﬁcationstoEMdiscussedintheprevioussectiontocorr ectlyupdatetheGaussian
meansandvariancesformultivariateGaussians. Wewilldis cussinSec.10.3inCh.10
howtomodifytheembeddedtrainingalgorithmto handlemixt ureGaussians.
Insummary,thebasic embedded trainingprocedure isasfollows:
Given: phoneset,pronunciationlexicon,andthetranscrib edwaveﬁles
1. Builda“wholesentence”HMMforeachsentence,asshownin Fig.9.32.
2. Initialize Aprobabilitiesto0.5(forloop-backsorforthecorrectnext
subphone)ortozero(forall othertransitions).
3. Initialize Bprobabilities by setting the mean and variance for each
Gaussianto theglobalmeanandvariancefortheentiretrain ingset.

4. RunmultipleiterationsoftheBaum-Welchalgorithm.
The Baum-Welch algorithm is used repeatedly as a component o f the embedded
training process. Baum-Welch computes ξt(i), the probability of being in state iat
timet, by using forward-backward to sum over all possible paths th at were in state
iemitting symbol otat timet. This lets us accumulate counts for re-estimating the
emission probability bj(ot)from all the paths that pass through state jat timet. But
Baum-Welchitself canbetime-consuming.

There is an efﬁcient approximationto Baum-Welch training t hat makes use of the
Viterbi algorithm. In Viterbi training , instead of accumulatingcounts by a sum over Viterbitraining
all paths that pass through a state jat timet, we approximate this by only choosing
the Viterbi (most-probable) path. Thus instead of running E M at every step of the
embeddedtraining,werepeatedlyrunViterbi.
Running the Viterbi algorithm over the training data in this way is called forced

Viterbi alignment or justforced alignment . In Viterbi training (unlike in Viterbi Forcedalignment
decoding on the test set) we know which word string to assign t o each observation
sequence, So we can ‘force’ the Viterbi algorithm to pass thr ough certain words, by
setting the aijs appropriately. A forced Viterbi is thus a simpliﬁcation of the regular
Viterbi decodingalgorithm,since it only has to ﬁgure out th e correctstate (subphone)

sequence,butdoesn’thavetodiscoverthewordsequence. Th eresultisa forcedalign-
ment: thesinglebeststatepathcorrespondingtothetrainingob servationsequence. We
cannowusethisalignmentofHMMstatestoobservationstoac cumulatecountsforre-
estimatingtheHMMparameters. We sawearlierthatforcdali gnmentcanalsobeused
inotherspeechapplicationsliketext-to-speech,whenver wehaveawordtranscriptand
a waveﬁleinwhichwewant toﬁndboundaries.
Theequationsforretraininga(non-mixture)Gaussianfrom aViterbialignmentare
asfollows:
ˆµi=1
TT
∑
t=1ots.t.qtisstatei (9.55)

DRAFT330 Chapter 9. AutomaticSpeechRecognition
ˆσ2
j=1
TT
∑
t=1(ot−µi)2s.t.qtisstatei (9.56)
We saw these equations already, as (9.27) and (9.28) on page 3 10, when we were
‘imaginingthesimplersituationofacompletelylabeledtr ainingset’.
It turnsout that thisforcedViterbialgorithmis also usedi n the embeddedtraining
ofhybridmodelslikeHMM/MLPorHMM/SVMsystems. Webeginwi thanuntrained
MLP,andusingitsnoisyoutputsasthe BvaluesfortheHMM,performaforcedViterbi
alignment of the training data. This alignment will be quite errorful, since the MLP
was random. Now this (quite errorful) Viterbi alignment giv e us a labeling of feature
vectors with phone labels. We use this labeling to retrain th e MLP. The counts of the
transitionswhichare takenin theforcedalignmentscan beu sedto estimate the HMM
transition probabilities. We continue this hill-climbing process of neural-net training
andViterbialignmentuntilthe HMMparametersbegintoconv erge.
9.8 Evaluation: WordErrorRate

The standardevaluationmetric for speechrecognitionsyst emsis the word error rate. Word error
The word error rate is based on how much the word string return ed by the recognizer
(often called the hypothesized word string) differs from a correct or reference tran-
scription. Given such a correct transcription, the ﬁrst ste p in computing word error is
to compute the minimum edit distance in words between the hypothesized and cor-
rectstrings,asdescribedinCh. 3. Theresult ofthiscomput ationwill betheminimum
numberofword substitutions , wordinsertions , andword deletions necessarytomap

between the correct and hypothesized strings. The word erro r rate (WER) is then de-
ﬁned as follows (note that because the equation includes ins ertions, the error rate can
begreaterthan100%):
WordErrorRate =100×Insertions +Substitutions +Deletions
Total WordsinCorrectTranscript
We sometimes also talk about the SER (Sentence Error Rate), w hich tells us how
manysentenceshadatleast oneerror:
SentenceErrorRate =100×# ofsentenceswith atleast oneworderror
total#ofsentences

Here is an example of the alignments between a reference and a hypothesized Alignment
utterancefromtheCALLHOMEcorpus,showingthecountsused tocomputetheword
errorrate:
REF: i *** ** UM the PHONE IS i LEFT THE portable **** PHONE UPST AIRS last night
HYP: i GOT IT TO the ***** FULLEST i LOVE TO portable FORM OF STO RES last night
Eval: I I S D S S S I S S
Thisutterancehassixsubstitutions,threeinsertions,an donedeletion:
WordErrorRate =1006+3+1
13=76.9%

DRAFTSection9.8. Evaluation: WordErrorRate 331
Thestandardmethodforimplementingminimumeditdistance andcomputingword
error rates is a free script called sclite, available from the National Institute of
Standards and Technologies (NIST) (NIST, 2005). sclite is given a series of ref-
erence (hand-transcribed, gold-standard) sentences and a matching set of hypothesis
sentences. Besides performingalignments, and computing w ord error rate, sclite per-
forms a number of other useful tasks. For example, it gives us eful information for
error analysis , such as confusionmatrices showing which words are often mi srecog-
nized for others, and gives summary statistics of words whic h are often inserted or
deleted. sclite alsogiveserrorratesbyspeaker(ifsentencesarelabeledf orspeaker
id),aswellasusefulstatisticslikethe sentenceerrorrate ,thepercentageofsentencesSentenceerrorrate
withat least oneworderror.
Finally, sclite can be used to compute signiﬁcance tests. Suppose we make

some changes to our ASR system and ﬁnd that our word error rate has decreased by
1%. In order to know if our changes really improved things, we need a statistical test
to make sure that the 1% difference is not just due to chance. T he standard statistical
test for determiningif two word error rates are different is the Matched-Pair Sentence
Segment Word Error (MAPSSWE) test, which is also available i nsclite (although
theMcNemar test issometimesusedaswell). McNemartest
The MAPSSWE test is a parametric test that looks at the differ ence between the

numberofworderrorsthetwosystemsproduce,averagedacro ssanumberofsegments.
Thesegmentsmaybequiteshortoraslongasanentireutteran ce;ingeneralwewantto
havethelargestnumberof(short)segmentsinordertojusti fythenormalityassumption
andformaximumpower. Thetestrequiresthattheerrorsinon esegmentbestatistically
independentof the errors in another segment. Since ASR syst ems tend to use trigram
LMs, this can be approximated by deﬁning a segment as a region bounded on both
sidesbywordsthat bothrecognizersget correct(orturn/ut teranceboundaries).

Here’sanexamplefromNIST(2007b)withfourregions:
I II III IV
REF: |it was|the best|of|times it|was the worst|of times| | it was
| | | | | | | |
SYS A:|ITS |the best|of|times it|IS the worst |of times|OR| it was
| | | | | | | |
SYS B:|it was|the best| |times it|WON the TEST |of times| |it was
In region I, system A has 2 errors (a deletion and an insertion ) and system B has
0; in region III system A has 1 (substitution) error and syste m B has 2. Let’s deﬁne
a sequence of variables Zrepresenting the difference between the errors in the two

I II III IV
| | | | | | | |
| | | | | | | |
systemsasfollows:
Ni
Athenumberoferrorsmadeonsegment ibysystem A
Ni
Bthenumberoferrorsmadeonsegment ibysystem B
Z Ni
A−Ni
B,i=1,2,···,nwherenisthenumberofsegments
For example in the example above the sequence of Zvalues is{2,−1,−1,1}. In-
tuitively, if the two systems are identical, we would expect the averagedifference, i.e.
the average of the Zvalues, to be zero. If we call the true average of the differen ces
muz,wewouldthusliketoknowwhether muz=0. Followingcloselytheoriginalpro-

DRAFT332 Chapter 9. AutomaticSpeechRecognition
posal and notation of Gillick and Cox (1989), we can estimate the true average from
ourlimitedsampleas ˆµz=∑n
i=1Zi/n.
Theestimate ofthevarianceofthe Zi’sis:
σ2
z=1
n−1n
∑
i=1(Zi−µz)2(9.57)
Let
W=ˆµz
σz/√n(9.58)
Foralargeenough n(>50)Wwillapproximatelyhaveanormaldistributionwithuni t
variance. The null hypothesis is H0:µz=0, and it can thus be rejected if 2 ∗P(Z≥
|w|)≤0.05(two-tailed)or P(Z≥|w|)≤0.05(one-tailed). where Zisstandardnormal
andwis the realized value W; these probabilities can be looked up in the standard
tablesofthe normaldistribution.
Couldweimproveonworderrorrateasametric? Itwouldbenic e,forexample,to
havesomethingwhichdidn’tgiveequalweighttoeveryword, perhapsvaluingcontent
wordslike Tuesdaymorethanfunctionwordslike aorof. Whileresearchersgenerally
agree that this would be a good idea, it has proved difﬁcult to agree on a metric that
works in everyapplication of ASR. For dialogue systems, how ever,where the desired

semanticoutputis moreclear,a metriccalled concepterrorrate hasprovedextremely
useful,andwill bediscussedinCh. 24onpage851.
9.9 Summary
Togetherwith Ch. 4 andCh. 6, thischapterintroducedthe fun damentalalgorithmsfor
addressingtheproblemof LargeVocabularyContinuousSpeech Recognition .
•The input to a speech recognizer is a series of acoustic waves . Thewaveform ,
spectrogram andspectrum areamongthevisualizationtoolsusedtounderstand
theinformationinthesignal.

•In the ﬁrst step in speech recognition, sound waves are sampled,quantized ,
andconvertedtosomesortof spectralrepresentation ;Acommonlyusedspec-
tral representation is the mel cepstrum orMFCCwhich provides a vector of
featuresforeachframeoftheinput.
•GMMacousticmodelsareusedtoestimatethe phoneticlikelihoods (alsocalled
observationlikelihoods )ofthese featurevectors foreachframe.
•Decoding orsearchorinference is the processof ﬁndingthe optimal sequence
of model states which matches a sequence of input observatio ns. (The fact that

therearethreetermsforthisprocessisahintthatspeechre cognitionisinherently
inter-disciplinary, and draws its metaphors from more than one ﬁeld; decoding
comesfrominformationtheory,and searchandinference fromartiﬁcial intelli-
gence).

DRAFTSection9.9. Summary 333
•We introduced two decoding algorithms: time-synchronous Viterbidecoding
(whichisusuallyimplementedwithpruningandcanthenbeca lledbeamsearch )
andstackorA∗decoding. Both algorithmstake as input a sequenceof cepstr al
feature vectors, a GMM acoustic model, and an N-gram language model, and
produceastringofwords.
•Theembeddedtraining paradigmisthenormalmethodfortrainingspeechrec-
ognizers. Givenaninitiallexiconwithhand-builtpronunc iationstructures,itwill
trainthe HMMtransitionprobabilitiesandtheHMMobservat ionprobabilities.
The ﬁrst machine which recognized speech was probably a comm ercial toy named
“Radio Rex” which was sold in the 1920s. Rex was a celluloid do g that moved (via
a spring) when the spring was released by 500 Hz acoustic ener gy. Since 500 Hz is
roughlytheﬁrst formantofthe vowel[eh]in “Rex”,the dogse emedto comewhenhe
wascalled(DavidandSelfridge,1962).
Bythelate1940sandearly1950s,anumberofmachinespeechr ecognitionsystems

had been built. An early Bell Labs system could recognize any of the 10 digits from
a single speaker (Davis et al., 1952). This system had 10 spea ker-dependent stored
patterns, one for each digit, each of which roughly represen ted the ﬁrst two vowel
formantsin the digit. They achieved97–99%accuracy by choo singthe pattern which
had the highest relative correlation coefﬁcient with the in put. Fry (1959) and Denes
(1959) built a phoneme recognizer at University College, Lo ndon, which recognized

four vowels and nine consonantsbased on a similar pattern-r ecognitionprinciple. Fry
and Denes’s system was the ﬁrst to use phoneme transition pro babilities to constrain
therecognizer.
The late 1960s and early 1970s produced a number of important paradigm shifts.
First werea numberoffeature-extractionalgorithms,incl udetheefﬁcientFast Fourier
Transform (FFT) (Cooley and Tukey, 1965), the application o f cepstral processing to
speech(Oppenheimetal.,1968),andthedevelopmentofLPCf orspeechcoding(Atal

and Hanauer, 1971). Second were a number of ways of handling warping; stretching Warping
orshrinkingtheinputsignal tohandledifferencesinspeak ingrate andsegmentlength
whenmatchingagainststoredpatterns. Thenaturalalgorit hmforsolvingthisproblem
was dynamic programming, and, as we saw in Ch. 6, the algorith m was reinvented
multiple times to addressthis problem. The ﬁrst applicatio n to speech processingwas
by Vintsyuk (1968), although his result was not picked up by o ther researchers, and

was reinvented by Velichko and Zagoruyko (1970) and Sakoe an d Chiba (1971) (and
(1984)). Soon afterward, Itakura (1975) combined this dyna mic programming idea
with the LPC coefﬁcients that had previously been used only f or speech coding. The
resulting system extracted LPC features for incoming words and used dynamic pro-
gramming to match them against stored LPC templates. The non -probabistic use of
dynamicprogrammingtomatchatemplateagainstincomingsp eechiscalled dynamic
time warping .Dynamic timewarping

DRAFT334 Chapter 9. AutomaticSpeechRecognition
ThethirdinnovationofthisperiodwastheriseoftheHMM.Hi ddenMarkovMod-
elsseemtohavebeenappliedtospeechindependentlyattwol aboratoriesaround1972.
Oneapplicationarosefromtheworkofstatisticians,inpar ticularBaumandcolleagues
at the Institute for Defense Analyses in Princeton on HMMs an d their application to
variouspredictionproblems(Baum and Petrie, 1966; Baum an d Eagon, 1967). James
Baker learned of this work and applied the algorithm to speec h processing (Baker,
1975) during his graduate work at CMU. Independently, Frede rick Jelinek, Robert
Mercer, and Lalit Bahl (drawing from their research in infor mation-theoretical mod-
els inﬂuenced by the work of Shannon (1948)) applied HMMs to s peech at the IBM
Thomas J. Watson Research Center (Jelinek et al., 1975). IBM ’s and Baker’s sys-
tems were very similar, particularly in their use of the Baye sian framework described
in this chapter. One early difference was the decoding algor ithm; Baker’s DRAGON

system used Viterbi (dynamicprogramming)decoding,while the IBM system applied
Jelinek’s stack decoding algorithm (Jelinek, 1969). Baker then joined the IBM group
forabrieftimebeforefoundingthespeech-recognitioncom panyDragonSystems. The
HMMapproachtospeechrecognitionwouldturnouttocomplet elydominatetheﬁeld
by the end of the century; indeed the IBM lab was the driving fo rce in extending sta-
tistical models to natural language processing as well, inc luding the development of

class-based N-grams, HMM-basedpart-of-speechtagging, statistical ma chinetransla-
tion,andthe useofentropy/perplexityasanevaluationmet ric.
The use of the HMM slowly spread through the speech community . One cause
was a numberof research and developmentprogramssponsored by the AdvancedRe-
search Projects Agency of the U.S. Department of Defense (AR PA). The ﬁrst ﬁve-
year program starting in 1971, and is reviewed in Klatt (1977 ). The goal of this ﬁrst
program was to build speech understanding systems based on a few speakers, a con-

strained grammar and lexicon (1000 words), and less than 10% semantic error rate.
Four systems were funded and compared against each other: th e System Develop-
ment Corporation (SDC) system, Bolt, Beranek & Newman (BBN) ’s HWIM system,
Carnegie-Mellon University’s Hearsay-II system, and Carn egie-Mellon’s Harpy sys-
tem (Lowerre, 1968). The Harpy system used a simpliﬁed versi on of Baker’s HMM-
basedDRAGONsystemandwasthebestofthetestedsystems,an daccordingtoKlatt
the onlyonetomeet the originalgoalsofthe ARPA project(wi tha semantic accuracy

rateof94%onasimple task).
Beginningin the mid-1980s,ARPA fundeda number of new speec h research pro-
grams. Theﬁrstwasthe“ResourceManagement”(RM)task(Pri ceetal.,1988),which
liketheearlierARPAtaskinvolvedtranscription(recogni tion)ofread-speech(speakers
readingsentencesconstructedfroma1000-wordvocabulary )butwhichnowincludeda
componentthatinvolvedspeaker-independentrecognition . Latertasksincludedrecog-
nition of sentences read from the Wall Street Journal (WSJ) b eginning with limited

systems of 5,000 words, and ﬁnally with systems of unlimited vocabulary (in prac-
tice most systems use approximately 60,000 words). Later sp eech-recognition tasks
moved away from read-speech to more natural domains; the Bro adcast News domain
(LDC, 1998; Graff, 1997) (transcription of actual news broa dcasts, including quite
difﬁcult passages such as on-the-street interviews) and th e Switchboard, CALLHOME ,
CALLFRIEND , and Fisher domains (Godfrey et al., 1992; Cieri et al., 2004 ) (natural

telephone conversations between friends or strangers) . Th e Air Trafﬁc Information

DRAFTSection9.9. Summary 335
System (ATIS) task (Hemphill et al., 1990) was an earlier spe ech understanding task
whosegoalwasto simulatehelpinga userbookaﬂight,byansw eringquestionsabout
potentialairlines,times, dates,andso forth.
Each of the ARPA tasks involved an approximately annual bake-off at which all Bake-off
ARPA-fundedsystems,andmanyother‘volunteer’systemsfr omNorthAmericanand
Europe,wereevaluatedagainsteachotherintermsofworder rorrateorsemanticerror
rate. In the early evaluations, for-proﬁt corporations did not generally compete, but
eventually many (especially IBM and ATT) competed regularl y. The ARPA competi-
tions resulted in widescale borrowing of techniques among l abs, since it was easy to
see which ideas had provided an error-reduction the previou s year, and were proba-
bly an important factor in the eventual spread of the HMM para digm to virtual every
major speech recognition lab. The ARPA program also resulte d in a number of use-

ful databases, originally designed for training and testin g systems for each evaluation
(TIMIT, RM, WSJ, ATIS, BN, CALLHOME, Switchboard, Fisher) b ut then made
availableforgeneralresearchuse.
Speechresearchincludesanumberofareasbesidesspeechre cognition;wealready
saw computationalphonologyin Ch. 7, speechsynthesisin Ch . 8, andwe will discuss
spoken dialogue systems in Ch. 24. Another important area is speaker identiﬁcationSpeaker
identiﬁcation

andspeaker veriﬁcation , in which we identify a speaker (for example for securitySpeaker
veriﬁcation
when accessing personal information over the telephone) (R eynolds and Rose, 1995;
Shriberg et al., 2005; Doddington, 2001). This task is relat ed tolanguage identiﬁca-
tion, in which we are given a waveﬁle and have to identify which lan guage is beingLanguage
idenntiﬁcation
spoken;thisis usefulforautomaticallydirectingcallers to humanoperatorsthat speak
appropriatelanguages.
Thereareanumberoftextbooksandreferencebooksonspeech recognitionthatare

goodchoicesforreaderswhoseekamorein-depthunderstand ingofthematerialinthis
chapter: Huangetal.(2001)isbyfarthemostcomprehensive andup-to-datereference
volumeandishighlyrecommended. Jelinek(1997),GoldandM organ(1999),andRa-
binerandJuang(1993)aregoodcomprehensivetextbooks. Th elasttwotextbooksalso
havediscussionsofthehistoryoftheﬁeld,andtogetherwit hthesurveypaperofLevin-
son(1995)haveinﬂuencedourshorthistorydiscussioninth ischapter. Ourdescription
of the forward-backward algorithm was modeled after Rabine r (1989), and we were

also inﬂuenced by another useful tutorial paper, Knill and Y oung (1997). Research in
the speech recognition ﬁeld often appears in the proceeding s of the annual INTER-
SPEECH conference, (which is called ICSLP and EUROSPEECH in alternate years)
aswell astheannualIEEEInternationalConferenceonAcous tics,Speech,andSignal
Processing(ICASSP).Journalsinclude SpeechCommunication ,ComputerSpeechand
Language , theIEEE Transactions on Audio, Speech, and Language Processin g, and
theACM TransactionsonSpeechandLanguageProcessing .

DRAFT336 Chapter 9. AutomaticSpeechRecognition
Exercises
9.1Analyzeeachoftheerrorsintheincorrectlyrecognizedtra nscriptionof“umthe
phone is I left the...” on page 330. For each one, give your bes t guess as to
whetheryou think it is caused by a problemin signal processi ng, pronunciation
modeling,lexiconsize,languagemodel,orpruninginthede codingsearch.
9.2In practice, speech recognizers do all their probability co mputation using the
log probability (orlogprob) rather than actual probabilities. This helps avoid Logprob
underﬂowforverysmallprobabilities,butalsomakestheVi terbialgorithmvery
efﬁcient,sinceall probabilitymultiplicationscanbeimp lementedbyaddinglog
probabilities. Rewrite the pseudocode for the Viterbi algo rithm in Fig. 9.26 on
page323to makeuse oflogprobsinsteadofprobabilities.
9.3Now modify the Viterbi algorithm in Fig. 9.26 on page 323 to im plement the
beamsearchdescribedonpage325. Hint: Youwillprobablyne edtoaddincode
tocheckwhethera givenstate isat theendofa wordornot.

9.4Finally,modifytheViterbialgorithminFig.9.26onpage32 3withmoredetailed
pseudocodeimplementingthearrayofbacktracepointers.
9.5Using the tutorialsavailable as part of a publicly availabl e recognizerlike HTK
orSonic,builda digitrecognizer.
9.6Take the digit recognizerabove and dump the phone likelihoo dsfor a sentence.
Now take your implementation of the Viterbi algorithm and sh ow that you can
successfullydecodetheselikelihoods.

Chapter10
SpeechRecognition: AdvancedTop-ics
True, their voice-print machine was unfortunately a crude o ne. It could discriminate
among only a few frequencies, and it indicated amplitude by i ndecipherable blots. But it
hadnever been intended forsuch vitallyimportant work.
Aleksandr I. Solzhenitsyn, The FirstCircle ,p. 505
Thekejucivil service examinations of Imperial China lasted almost 1300 years,
from the year 606 until it was abolished in 1905. In its peak, m illions of would-be
ofﬁcials from all over China competed for high-rankinggove rnmentpositions by par-

ticipating in a uniform examination. For the ﬁnal ‘metropol itan’ part of this exam in
the capital city, the candidates would be locked into an exam ination compound for a
gruelingninedaysandnightsansweringquestionsabouthis tory,poetry,theConfucian
classics, andpolicy.
Naturallyall thesemillionsofcandidatesdidn’tall showu pinthe capital. Instead,
the exam had progressivelevels; candidateswho passed a one -daylocal exam in their
localprefecturecouldthensitforthebiannualprovincial exam,andonlyuponpassing

that exam in the provincial capital was a candidate eligible for the metropolitan and
palaceexaminations.
This algorithm for selecting capable ofﬁcials is an instanc e of multi-stage search.
The ﬁnal 9-day process requires far too many resources (in bo th space and time) to
examine every candidate. Instead, the algorithm uses an eas ier, less intensive 1-day
processtocomeupwithapreliminarylistofpotentialcandi dates,andappliestheﬁnal
test onlyto thislist.

Thekejualgorithmcan also be applied to speechrecognition. We’d li ke to be able
to applyveryexpensivealgorithmsin the speech recognitio nprocess, such as 4-gram,
5-gram, or even parser-based language models, or context-d ependent phone models
that can see two or three phones into the future or past. But th ere are a huge number
of potential transcriptions sentences for any given wavefo rm, and it’s too expensive
(in time, space, or both) to apply these powerful algorithms to every single candidate.

Instead,we’llintroduce multipassdecoding algorithmsinwhichefﬁcientbut dumber
decoding algorithms produce shortlists of potential candi dates to be rescored by slow
but smarter algorithms. We’ll also introduce the context-dependent acoustic model ,
whichisoneofthesesmarterknowledgesourcesthatturnsou ttobeessentialinlarge-
vocabulary speech recognition. We’ll also brieﬂy introduc e the important topics of
discriminativetrainingandthemodelingofvariation.

DRAFT338 Chapter 10. SpeechRecognition: AdvancedTopics
10.1 MultipassDecoding: N-best listsandlattices
Thepreviouschapterappliedthe ViterbialgorithmforHMM d ecoding. Thereare two
main limitations of the Viterbi decoder, however. First, th e Viterbi decoder does not
actuallycomputethesequenceofwordswhichismostprobabl egiventheinputacous-
tics. Instead,it computesanapproximationto this: the seq uenceofstates(i.e.,phones
orsubphones ) which is most probable given the input. More formally, reca ll that the
truelikelihoodofanobservationsequence Oiscomputedbytheforwardalgorithmby
summingoverallpossiblepaths:
P(O|W) =∑
S∈ST
1P(O,S|W) (10.1)
TheViterbialgorithmapproximatesthissumbyusingthepro babilityofthebest path:
P(O|W)≈max
S∈ST
1P(O,S|W) (10.2)
Itturnsoutthatthis Viterbiapproximation isnottoobad,sincethemostprobableViterbiapproximation
sequence of phones usually turns out to correspond to the mos t probable sequence
of words. But not always. Consider a speech recognition syst em whose lexicon has

P(O|W) =∑
S∈ST
S∈ST
multiple pronunciationsfor each word. Suppose the correct word sequenceincludesa
word with very many pronunciations. Since the probabilitie s leaving the start arc of
each word must sum to 1.0, each of these pronunciation-paths through this multiple-
pronunciationHMMwordmodelwill haveasmallerprobabilit ythanthepaththrough
a word with only a single pronunciation path. Thus because th e Viterbi decoder can
only follow one of these pronunciation paths, it may ignore t his many-pronunciation

word in favor of an incorrect word with only one pronunciatio n path. In essence, the
Viterbiapproximationpenalizeswordswith manypronuncia tions.
A secondproblemwith theViterbi decoderisthat it is imposs ibleorexpensivefor
it to take advantage of many useful knowledge sources. For ex ample the Viterbi al-
gorithm as we have deﬁned it cannot take complete advantageo f any language model
more complex than a bigram grammar. This is because of the fac t mentioned earlier

that a trigram grammar, for example, violates the dynamic programming invariant .
Recall that this invariant is the simplifying (but incorrec t) assumption that if the ulti-
matebestpathfortheentireobservationsequencehappenst ogothroughastate qi,that
this best path must include the best path up to and including s tateqi. Since a trigram
grammarallows the probabilityof a word to be based on the two previouswords, it is
possible thatthe best trigram-probabilitypath forthe sen tencemaygo througha word

but not include the best path to that word. Such a situation co uld occur if a particular
wordwxhas a high trigram probability given wy,wz, but that conversely the best path
towydidn’t include wz(i.e.,P(wy|wq,wz)was low for all q). Advanced probabilistic
LMslikeSCFGs alsoviolatethesame dynamicprogrammingass umptions.
There are two solutions to these problems with Viterbi decod ing. The most com-
mon is to modify the Viterbi decoder to return multiple poten tial utterances, instead

of just the singlebest, and thenuse otherhigh-levellangua gemodelorpronunciation-

DRAFTSection10.1. MultipassDecoding: N-best listsandlattices 339
modeling algorithms to re-rank these multiple outputs (Sch wartz and Austin, 1991;
SoongandHuang,1990;Murveitet al.,1993).
The second solution is to employ a completely different deco ding algorithm, such
as thestack decoder , orA∗decoder(Jelinek, 1969;Jelinek et al., 1975). We begin in stackdecoder
A∗thissectionwithmultiple-passdecoding,andthenreturnt o stackdecoding.
Inmultiple-pass decoding we break up the decoding process into two stages. In
theﬁrst stagewe usefast,efﬁcientknowledgesourcesoralg orithmsto performanon-
optimal search. So for example we might use an unsophisticat ed but time-and-space
efﬁcientlanguagemodellikeabigram,orusesimpliﬁedacou sticmodels. Inthesecond
decoding pass we can apply more sophisticated but slower dec oding algorithms on a
reduced search space. The interface between these passes is anN-best list orword
lattice.
The simplest algorithm for multipass decoding is to modify t he Viterbi algorithm

lattice.
to return the N-bestsentences (word sequences) for a given speech input. Suppos e N-best
forexampleabigramgrammarisusedwithsuchan N-best-Viterbialgorithmtoreturn
the1000mosthighly-probablesentences,eachwiththeirAM likelihoodandLMprior
score. This 1000-best list can now be passed to a more sophist icated language model
like a trigram grammar. This new LM is used to replace the bigr am LM score of
each hypothesized sentence with a new trigram LM probabilit y. These priors can be

combined with the acoustic likelihood of each sentence to ge nerate a new posterior
probability for each sentence. Sentences are thus rescored and re-ranked using this Rescoring
moresophisticatedprobability. Fig.10.1showsanintuiti onforthisalgorithm.
RescoringN-Best
DecoderSpeech InputSimple
Knowledge
SourceSmarter
Knowledge
Source
N-Best List 1-Best Utterance
? Alice was beginning to get...
? Every happy family
? In a hole in the ground...
? If music be the food of love...
? If music be the foot of dove...If music be the
food of love

Knowledge
Knowledge
Source
Figure10.1 The use of N-best decoding as part of a two-stage decoding model. Efﬁcie nt but
unsophisticated knowledge sources are used to return the N-best utterances. This signiﬁcantly
reduces the searchspace for second pass models, whichcanth us be sophisticated butslow.
ThereareanumberofalgorithmsforaugmentingtheViterbia lgorithmtogenerate
N-best hypotheses. It turns out that there is no polynomial-t imeadmissible algorithm
forﬁndingthe Nmost likely hypotheses(Young,1984). Thereare however,a n umber

of approximate (non-admissible) algorithms; we will intro duce just one of them, the
“ExactN-best” algorithm of Schwartz and Chow (1990). In Exact N-best, instead of
eachstate maintaininga singlepath/backtrace,wemaintai nuptoNdifferentpathsfor
each state. But we’dlike to insure that these pathscorrespo ndto differentword paths;
we don’t want to waste our Npaths on different state sequences that map to the same
words. To do this, we keep for each path the word history , the entire sequence of

words up to the current word/state. If two paths with the same word history come to
a state at the same time, we merge the paths and sum the path pro babilities. To keep
theNbest word sequences, the resulting algorithm requires O(N)times the normal

DRAFT340 Chapter 10. SpeechRecognition: AdvancedTopics
Viterbi time. We’ll see this mergingof paths againwhen we in troducingdecodingfor
statistical machinetranslation,whereit iscalled hypothesisrecombination .
Rank Path AMlogprob LMlogprob
1. it’san area that’snaturally sortof mysterious -7193.53 -20.25
2. that’s anarea that’s naturallysort of mysterious -7192. 28 -21.11
3. it’san area that’snot reallysort of mysterious -7221.68 -18.91
4. that scenario that’snaturally sortof mysterious -7189. 19 -22.08
5. there’s anarea that’snaturallysort of mysterious -7198 .35 -21.34
6. that’s anarea that’s not reallysort of mysterious -7220. 44 -19.77
7. the scenario that’s naturallysort of mysterious -7205.4 2 -21.50
8. soit’s anarea that’snaturally sortof mysterious -7195. 92 -21.71
9. that scenario that’snot reallysort of mysterious -7217. 34 -20.70
10. there’s anarea that’snot reallysort of mysterious -722 6.51 -20.01
Figure10.2 An example 10-Best list from the Broadcast News corpus, prod uced by the CU-

HTK BN system (thanks to Phil Woodland). Logprobs use log 10; the language model scale
factor (LMSF)is15.
Theresultofanyofthesealgorithmsisan N-bestlistliketheoneshowninFig.10.1.
InFig. 10.1the correcthypothesishappensto bethe ﬁrst one ,butofcoursethe reason
to useN-best lists is that isn’t always the case. Each sentence in an N-best list is also
annotated with an acoustic model probability and a language model probability. This
allowsasecond-stageknowledgesourceto replaceoneoftho setwo probabilitieswith

animprovedestimate.
One problem with an N-best list is that when Nis large, listing all the sentences
is extremely inefﬁcient. Another problemis that N-best lists don’t give quite as much
informationaswe mightwantfora second-passdecoder. Fore xample,wemightwant
distinct acousticmodelinformationforeach wordhypothes isso that we can reapplya
new acoustic model for the word. Or we might want to have avail able different start
andendtimesofeachwordso thatwe canapplyanewdurationmo del.

For this reason, the output of a ﬁrst-pass decoder is usually a more sophisticated
representation called a word lattice (Murveit et al., 1993; Aubert and Ney, 1995). A word lattice
wordlatticeisadirectedgraphthatefﬁcientlyrepresents muchmoreinformationabout
possiblewordsequences.1Insomesystems,nodesinthegrapharewordsandarcsare
transitions between words. In others, arcs represent word h ypotheses and nodes are
pointsintime. Let’susethislattermodel,andsoeacharcre presentslotsofinformation

about the word hypothesis, including the start and end time, the acoustic model and
languagemodelprobabilities,the sequenceof phones(thep ronunciationof theword),
or eventhe phonedurations. Fig. 10.3shows a sample lattice correspondingto the N-
bestlist inFig.10.1. Notethatthelatticecontainsmanydi stinctlinks(records)forthe
same word, each with a slightly differentstarting or ending time. Such lattices are not
producedfrom N-best lists; instead,a latticeis producedduringﬁrst-pas sdecodingby

includingsome of the word hypotheseswhich were active (in t he beam) at each time-
1Actually an ASRlattice is notthe kind oflattice that maybef amiliar to youfrommathematics, sinceitis
not required to have the properties of a true lattice (i.e., b e a partially ordered set with particular properties,
such as a unique join for each pair of elements). Really it’s j ust a graph, but it is conventional to call it a
lattice.

DRAFTSection10.1. MultipassDecoding: N-best listsandlattices 341
step. Since the acoustic and language models are context-de pendent, distinct links
need to be created for each relevant context, resulting in a l arge number of links with
the same word but different times and contexts. N-best lists like Fig. 10.1 can also be
producedby ﬁrst building a lattice like Fig. 10.3and then tr acing throughthe paths to
produceNwordstrings.
Figure10.3 Word lattice corresponding to the N-best list in Fig. 10.1. The arcs beneath each
word show the different start and end times for each word hypo thesis in the lattice; for some of
these we’ve shown schematically how each word hypothesis mu st start at the end of a previous
hypothesis. Not shown in this ﬁgure are the acoustic and lang uage model probabilities that
decorate eacharc.
The fact that each word hypothesis in a lattice is augmented s eparately with its
acoustic model likelihood and language model probability a llows us to rescore any

path through the lattice, using either a more sophisticated language model or a more
sophisticated acoustic model. As with N-best lists, the goal of this rescoring is to
replace the 1-best utterance with a different utterance that perhapshad a lower score
on the ﬁrst decodingpass. For this second-passknowledgeso urce to get perfect word
error rate, the actual correct sentence would have to be in th e lattice or N-best list. If
the correct sentence isn’t there, the rescoring knowledge s ource can’t ﬁnd it. Thus it

is importantwhen working with a lattice or N-best list to consider the baseline lattice
errorrate (Woodlandetal.,1995;Ortmannsetal.,1997): thelowerbou ndworderror Lattice error rate
rate from the lattice. The lattice error rate is the word erro r rate we get if we chose
the lattice path (the sentence) that has the lowest word erro r rate. Because it relies on
perfect knowledge of which path to pick, we call this an oracleerror rate, since we Oracle error rate
needsomeoracletotell uswhichsentence/pathtopick.

Another important lattice concept is the lattice density , which is the number of Latticedensity
edgesinalatticedividedbythenumberofwordsintherefere ncetranscript. Aswesaw
schematicallyinFig.10.3,reallatticesareoftenextreme lydense,withmanycopiesof
individual word hypotheses at slightly different start and end times. Because of this
density,latticesareoftenpruned.
Besides pruning,lattices are oftensimpliﬁed into a differ ent,more schematic kind

DRAFT342 Chapter 10. SpeechRecognition: AdvancedTopics
of lattice that is sometimes called a word graph orﬁnite-state machine , although Word graph
often it’s still just referred to as a word lattice. In these w ord graphs, the timing in-
formation is removed and multiple overlapping copies of the same word are merged.
The timing of the words is left implicit in the structure of th e graph. In addition, the
acoustic model likelihood information is removed, leaving only the language model
probabilities. The resulting graph is a weighted FSA, which is a natural extension of
anN-gram language model; the word graph corresponding to Fig. 1 0.3 is shown in
Fig. 10.4. This word graph can in fact be used as the language m odel for another de-
codingpass. Since sucha wordgraphlanguagemodelvastlyre strictsthesearchspace,
it canmakeit possibletouse acomplicatedacousticmodelwh ichistooslowtouse in
ﬁrst-passdecoding.
Figure10.4 Word graph corresponding to the N-best list in Fig. 10.1. Each word hypothesis

inthe latticealsohas language model probabilities (not sh own inthis ﬁgure).
Aﬁnaltypeoflatticeisusedwhenweneedtorepresentthepos teriorprobabilityof
individualwordsinalattice. Itturnsoutthatinspeechrec ognition,wealmostneversee
thetrueposteriorprobabilityofanything,despitethefac tthatthegoalofspeechrecog-
nition is to compute the sentence with the maximum a posterio ri probability. This is
becauseinthe fundamentalequationofspeechrecognitionw e ignorethe denominator
inourmaximization:
ˆW=argmax
W∈LP(O|W)P(W) (10.3)

ˆW=argmax
The product of the likelihood and the prior is notthe posterior probability of the
utterance. It is not even a probability, since it doesn’t nec essarily lie between 0 and
1. It’s just a score. Why does it matter that we don’t have a tru e probability? The
reason is that without having true probability, we can choos e the best hypothesis, but
we can’t know how good it is. Perhaps the best hypothesis is st ill really bad, and we
needtoasktheusertorepeatthemselves. Ifwehadtheposter iorprobabilityofaword

it could be used as a conﬁdence metric, since the posterior is an absolute rather than
relative measure. A conﬁdence metric is a metric that the spe ech recognizer can give
toahigher-levelprocess(likedialogue)toindicatehowco nﬁdenttherecognizeristhat
the word string that it returns is a good one. We’ll return to t he use of conﬁdence in
Ch.24.
In order to compute the posterior probability of a word, we’l l need to normalize

Ch.24.
over all the different word hypotheses available at a partic ular point in the utterances.

DRAFTSection10.2. A∗(‘Stack’)Decoding 343
At each point we’ll need to know which words are competing or c onfusable. The
lattices that show these sequencesof wordconfusionsare ca lledconfusion networks ,Confusion
network
meshes,sausages,orpinched lattices . Aconfusionnetworkconsistsofasequenceof Mesh
Sausage
Pinchedlatticeword positions. At each position is a set of mutually exclusi ve word hypotheses. The
networkrepresentstheset ofsentencesthatcanbecreatedb ychoosingonewordfrom
eachposition.
Figure10.5 Confusion network corresponding to the word lattice in Fig. 10.3. Each word is
associated with a posterior probability. Note that some of t he words from the lattice have been
pruned away. (Probabilitiescomputed bythe SRI-LMtoolkit ).
Note that unlike lattices or word graphs, the process of cons tructing a confusion
network actually adds paths that were not in the original lat tice. Confusion networks
have other uses besides computing conﬁdence. They were orig inally proposed for

use in minimizing word error rate, by focusing on maximizing improving the word
posteriorprobabilityratherthanthe sentencelikelihood . Recently confusionnetworks
havebeenusedtotraindiscriminativeclassiﬁersthatdist inguishbetweenwords.
Roughlyspeaking, confusionnetworksare built by taking th e different hypothesis
paths in the lattice and aligning them with each other. The po sterior probability for
each word is computing by ﬁrst summing over all paths passing through a word, and

then normalizing by the sum of the probabilities of all compe ting words. For further
details see Mangu et al. (2000), Evermann and Woodland (2000 ), Kumar and Byrne
(2002),Doumpiotisetal. (2003b).
Standardpubliclyavailablelanguagemodelingtoolkitsli keSRI-LM(Stolcke,2002)
(http://www.speech.sri.com/projects/srilm/ )andtheHTKlanguage
modeling toolkit (Young et al., 2005) ( http://htk.eng.cam.ac.uk/ ) can be
usedtogenerateandmanipulatelattices, N-best lists, andconfusionnetworks.

Therearemanyotherkindsofmultiple-stagesearch,suchas theforward-backward forward-backward
searchalgorithm(nottobeconfusedwiththe forward-backward algorithmforHMM
parameter setting) (Austin et al., 1991) which performs a si mple forward search fol-
lowedbya detailedbackward(i.e.,time-reversed)search.
10.2 A∗(‘Stack’)Decoding
Recall that the Viterbi algorithm approximated the forward computation, computing
the likelihoodof the single best (MAX) path throughthe HMM, while the forwardal-

gorithmcomputesthelikelihoodofthetotal(SUM)ofallthe pathsthroughtheHMM.

DRAFT344 Chapter 10. SpeechRecognition: AdvancedTopics
TheA∗decodingalgorithmallowsustousethecompleteforwardpro bability,avoiding
the Viterbi approximation. A∗decoding also allows us to use any arbitrary language
model. ThusA∗is aone-passalternativeto multi-passdecoding.
The A∗decoding algorithm is a best-ﬁrst search of the tree that imp licitly deﬁnes
thesequenceofallowablewordsinalanguage. Considerthet reeinFig.10.6,rootedin
theSTARTnodeontheleft. Eachleafofthistreedeﬁnesonese ntenceofthelanguage;
the oneformedby concatenatingall thewordsalongthe pathf romSTART to the leaf.
We don’t represent this tree explicitly, but the stack decod ing algorithm uses the tree
implicitlyasawayto structurethedecodingsearch.
Figure10.6 A visual representation of the implicit lattice of allowabl e word sequences that
deﬁnes a language. The setof sentences of a language is far to olargetorepresent explicitly,but
the latticegives a metaphor for exploring preﬁxes.

The algorithm performsa search from the root of the tree towa rd the leaves, look-
ingforthehighestprobabilitypath,andhencethehighestp robabilitysentence. Aswe
proceedfromroottowardtheleaves,eachbranchleavingagi venwordnoderepresents
a word which may follow the current word. Each of these branch es has a probabil-
ity, which expresses the conditionalprobabilityof this ne xt word given the part of the
sentence we’ve seen so far. In addition, we will use the forwa rd algorithm to assign

each word a likelihood of producing some part of the observed acoustic data. The
A∗decoder must thus ﬁnd the path (word sequence) from the root t o a leaf which
has the highest probability, where a path probability is deﬁ ned as the product of its
language model probability (prior) and its acoustic match t o the data (likelihood). It
doesthisbykeepinga priorityqueue ofpartialpaths(i.e.,preﬁxesofsentences,each Priorityqueue
annotatedwithascore). Inapriorityqueueeachelementhas ascore,andthe popoper-

ationreturnstheelementwiththehighestscore. TheA∗decodingalgorithmiteratively
choosesthebestpreﬁx-so-far,computesallthepossiblene xtwordsforthatpreﬁx,and
addstheseextendedsentencesto thequeue. Fig.10.7showst hecompletealgorithm.
Let’s consider a stylized example of an A∗decoder working on a waveform for
which the correct transcription is If music be the food of love . Fig. 10.8 shows the
search space after the decoderhas examinedpaths of length o ne from the root. A fast

matchisusedtoselectthelikelynextwords. Afastmatchisoneofa classofheuristics Fast match
designed to efﬁciently winnow down the number of possible fo llowing words, often
by computing some approximation to the forward probability (see below for further
discussionoffastmatching).
At this point in our example, we’ve done the fast match, selec ted a subset of the

DRAFTSection10.2. A∗(‘Stack’)Decoding 345
function STACK-DECODING ()returnsmin-distance
Initialize the priorityqueue witha null sentence.
Popthe best (highest score) sentence soff the queue.
If (sis marked end-of-sentence (EOS)) output sand terminate.
Get listof candidate next words bydoing fast matches.
For eachcandidate next word w:
Create anew candidate sentence s+w.
Use forwardalgorithm tocompute acoustic likelihood Lofs+w
Compute language model probability Pof extended sentence s+w
Compute “score” for s+w(a function of L,P,and???)
if(end-of-sentence) setEOSﬂagfor s+w.
Inserts+wintothe queue together withitsscore andEOSﬂag
Figure10.7 TheA∗decoding algorithm(modiﬁedfromPaul(1991)andJelinek(1 997)). The
evaluation function that is used to compute the score for a se ntence is not completely deﬁned
here; possible evaluation functions are discussed below.
possiblenextwords,andassignedeachofthemascore. Thewo rdAlicehasthehighest
score. We haven’tyet saidexactlyhowthescoringworks.
[START]If
Alice
In
Every130
40

[START]If
Alice
In
Every130
40
4
25P("If" | START)
P("Every" | START)
Figure10.8 The beginning of the search for the sentence If music be the food of love . At this
earlystage Aliceis the most likelyhypothesis. (Ithas a higher score thanthe other hypotheses.)
Fig. 10.9a show the next stage in the search. We have expanded theAlicenode.
Thismeansthatthe Alicenodeisnolongeronthequeue,butitschildrenare. Notethat
now the node labeled ifactually has a higher score than any of the children of Alice.

[START]If
Alice
In
Every130
40
4
Fig. 10.9b shows the state of the search after expanding the ifnode, removing it, and
addingif music,if muscle,andif messyonto thequeue.
Weclearlywantthescoringcriterionforahypothesistober elatedtoitsprobability.
Indeedit mightseemthatthescorefora stringofwords wi
1givenanacousticstring yj
1
shouldbethe productofthepriorandthelikelihood:
P(yj
1|wi
1)P(wi
1)
Alas, the score cannot be this probability because the proba bility will be much

1
P(yj
1|wi
1)P(wi
1)
smaller for a longer path than a shorter one. This is due to a si mple fact about prob-

DRAFT346 Chapter 10. SpeechRecognition: AdvancedTopics
[START]If
Alice
In
Every130
40
4
i
j ki
j l
mki
j nnk2
2
2P(O " if") = 
    forward probability
[START]If
Alice
In
Every130
40
4
25P("If" | START)i
j ki
j l
mki
j nnk2
2
o p k qro p k r nso s k kt32
31
25P( "music" | "if" )
(a) (b)
Figure10.9 The next steps of the search for the sentence If music be the food of love . In (a)
we’ve now expanded the Alicenode and added three extensions which have a relatively high
score; the highest-scoring node is START if, which is not along the START Alice path at all. In
(b) we’ve expanded the ifnode. The hypothesis STARTifmusic thenhas the highest score.
abilities and substrings; any preﬁx of a string must have a hi gher probability than the
string itself (e.g., P(START the ...) will be greater than P( START the book)). Thus
if we used probabilityas the score, the A∗decoding algorithmwould get stuck on the
single-wordhypotheses.

Instead, we use the A∗evaluation function (Nilsson, 1980; Pearl, 1984) f∗(p),
givena partialpath p:
f∗(p) =g(p)+h∗(p)
f∗(p)is theestimated score of the best complete path (complete sentence) which
starts with the partial path p. In other words, it is an estimate of how well this path
would do if we let it continue through the sentence. The A∗algorithm builds this
estimatefromtwocomponents:
•g(p)is the score from the beginning of utterance to the end of the p artial path
p. Thisgfunction can be nicely estimated by the probability of pgiven the

acousticssofar(i.e.,as P(O|W)P(W)forthewordstring Wconstituting p).
•h∗(p)isanestimateofthebestscoringextensionofthepartialpa thtotheendof
theutterance.
Coming up with a good estimate of h∗is an unsolved and interesting problem. A
very simple approach is to chose an h∗estimate which correlates with the number of
words remaining in the sentence (Paul, 1991). Slightly smar ter is to estimate the ex-
pectedlikelihoodperframefortheremainingframes,andmu ltiplethisbytheestimate

of the remaining time. This expected likelihood can be compu ted by averaging the
likelihoodperframeinthetrainingset. See Jelinek(1997) forfurtherdiscussion.

DRAFTSection10.3. Context-DependentAcousticModels: Triphon es 347
TreeStructuredLexicons
WementionedabovethatboththeA∗andvariousothertwo-stagedecodingalgorithms
require the use of a fast match for quickly ﬁnding which words in the lexicon are
likely candidates for matching some portion of the acoustic input. Many fast match
algorithmsarebasedontheuseofa tree-structuredlexicon ,whichstoresthepronun-Tree-structuredlexicon
ciationsofall the wordsin such a way that the computationof the forwardprobability
can be shared for words which start with the same sequence of p hones. The tree-
structured lexicon was ﬁrst suggested by Klovstad and Monds hein (1975); fast match
algorithms which make use of it include Gupta et al. (1988), B ahl et al. (1992) in the
contextofA∗decoding,andNeyet al. (1992)andNguyenandSchwartz(1999 )in the
context of Viterbi decoding. Fig. 10.10 shows an example of a tree-structuredlexicon
from the Sphinx-IIrecognizer (Ravishankar, 1996). Each tr ee root represents the ﬁrst

phone of all words beginning with that context dependent pho ne (phone context may
or may not be preserved across word boundaries), and each lea f is associated with a
word.
AX(#,B)N(AW,DD)
B(AX,AW)
B(AX,AH)AW(B,N)
AW(B,TD)
AH(B,V)DD(N,#)
B(#,EY)TD(KD,#)EY(B,KD)
EY(B,K)KD(EY,#)
KD(EY,TD)
K(EY,IX)
K(EY,AXR)AXR(K,#)
AXR(K,IY) IY(AXR,#)IX(K,NG) NG(IX,#)ABOUND
ABOUT TD(AW,#)
V(AH,#)ABOVE
BAKE
BAKED
BAKING
BAKER
BAKERY
Figure10.10 A tree-structured lexicon from the Sphinx-II recognizer (a fter Ravishankar

word.
B(AX,AW)
AW(B,TD)
KD(EY,TD)
K(EY,IX)
BAKE
BAKED
BAKING
BAKER
BAKERY
(1996)). Each node corresponds to a particular triphone in t he slightly modiﬁed version of the
ARPAbetusedbySphinx-II.ThusEY(B,KD)meansthephoneEYp recededbyaBandfollowed
bythe closure of a K.
10.3 Context-DependentAcousticModels: Triphones
In our discussion in Sec. 9.4 of how the HMM architecture is ap plied to ASR, we
showed how an HMM could be created for each phone, with its thr ee emitting states
correspondingto subphones at the beginning, middle, and en d of the phone. We thus

represent each subphone (“beginning of [eh]”, “beginning o f [t]”, “middle of [ae]”)
withits ownGMM.
There is a problem with using a ﬁxed GMM for a subphone like ”be ginning of
[eh]”. Theproblemisthatphonesvaryenormouslybasedonth ephonesoneitherside.
This is because the movement of the articulators (tongue, li ps, velum) during speech
production is continuous and is subject to physical constra ints like momentum. Thus

DRAFT348 Chapter 10. SpeechRecognition: AdvancedTopics
an articulatormaystart movingduringonephoneto get intop lace in time for the next
phone. InCh.7wedeﬁnedtheword coarticulation asthemovementofarticulatorsto Coarticulation
anticipate the next sound, or perseverating movement from t he last sound. Fig. 10.11
showscoarticulationdueto neighboringphonecontextsfor thevowel[eh].
Time (s)0 1.1917505000Frequency (Hz)
WED YELL BEN
Figure10.11 The vowel [eh] in three different triphone contexts, in the w ordswed,yell, and
Ben. Notice the marked differences in the second formant (F2) at the beginning and end of the
[eh] inallthree cases.
In order to model the marked variation that a phone exhibits i n different contexts,
most LVCSR systems replace the idea of a context-independen t (CI phone ) HMM CIphone
withacontext-dependentor CDphones . Themostcommonkindofcontext-dependent CDphones
modelisa triphone HMM(Schwartzetal.,1985;Dengetal.,1990). Atriphonemod el Triphone

represents a phone in a particular left and right context. Fo r example the triphone [y-
eh+l]means“[eh]precededby[y]andfollowedby[l]”. Ingeneral, [a-b+c]willmean
“[b] preceded by [a] and followed by [c]”. In situations wher e we don’t have a full
triphonecontext,we’ll use [a-b]tomean“[b]precededby[a ]”and[b+c]to mean“[b]
followedby[c]”.
Context-dependentphonescapture an important source of va riation, and are a key
part of modern ASR systems. But unbridled context-dependen cyalso introduces the

sameproblemwesawinlanguagemodeling: trainingdataspar sity. Themorecomplex
the model we try to train, the less likely we are to have seen en ough observations of
eachphone-typetotrainon. Foraphonesetwith50phones,in principlewewouldneed
503or 125,000 triphones. In practice not every sequence of thre e phones is possible
(Englishdoesn’tseemtoallowtriphonesequenceslike[ae- eh+ow]or[m-j+t]). Young
et al. (1994) found that 55,000 triphones are needed in the 20 K Wall Street Journal

task. But they found that only 18,500 of these triphones, i.e . less than half, actually
occurredin theSI84sectionoftheWSJ trainingdata.
Because of the problem of data sparsity, we must reduce the nu mber of triphone
parameters that we need to train. The most common way to do thi s is by clustering
some of the contexts together and tyingsubphones whose contexts fall into the same Tiedstates

DRAFTSection10.3. Context-DependentAcousticModels: Triphon es 349
cluster (Young and Woodland, 1994). For example, the beginn ing of a phone with an
[n]onits left maylookmuchlike the beginningofa phonewith an[m]on itsleft. We
can therefore tie together the ﬁrst (beginning)subphone of , say, the [m-eh+d] and [n-
eh+d] triphones. Tying two states together means that they s hare the same Gaussians.
SoweonlytrainasingleGaussianmodelfortheﬁrstsubphone ofthe[m-eh+d]and[n-
eh+d]triphones. Likewise, it turnsoutthatthe left contex tphones[r]and[w]produce
a similareffectonthe initialsubphoneoffollowingphones .
Fig. 10.12 shows, for example the vowel [iy] preceded by the c onsonants [w], [r],
[m],and[n]. Noticethat thebeginningof[iy]hasa similarr ise inF2after[w]and[r].
And notice the similarity of the beginningof [m] and [n]; as C h. 7 noted, the position
ofnasalformantsvariesstronglyacrossspeakers,butthis speaker(theﬁrst author)has
a nasalformant(N2)around1000Hz.
Time (s)0 3.1207905000Frequency (Hz)

[wiy] [r iy] [miy] [niy]
Figure10.12 The words we,re,me, andknee. The glides [w] and [r] have similar effects on
the beginning of the vowel [iy], asdo the twonasals [n] and [m ].
Fig.10.13showsanexampleofthekindoftriphonetyinglear nedbytheclustering
algorithm. Each mixture Gaussian model is shared by the subp hone states of various
triphoneHMMs.
t-iy+n t-iy+ng f-iy+l s-iy+l
... etc.
Figure10.13 Four triphones showing the result of clustering. Notice tha t the initialsubphone

... etc.
of [t-iy+n] and [t-iy+ng] is tied together, i.e. shares the s ame Gaussian mixture acoustic model.
AfterYoung et al.(1994).
How do we decide what contexts to cluster together? The most c ommon method
is to use a decision tree. For each state (subphone) of each ph one, a separate tree is
built. Fig. 10.14shows a sample tree fromthe ﬁrst (beginnin g)state of the phone/ih/,
modiﬁed from Odell (1995). We begin at the root node of the tre e with a single large

cluster containing(the beginningstate of) all triphonesc enteredon /ih/. At each node

DRAFT350 Chapter 10. SpeechRecognition: AdvancedTopics
in the tree, we split the current cluster into two smaller clu sters by asking questions
about the context. For example the tree in Fig. 10.14 ﬁrst spl its the initial cluster into
twoclusters,onewithnasalphoneontheleft,andonewithou t. Aswedescendthetree
fromtheroot,eachoftheseclustersisprogressivelysplit . ThetreeinFig.10.14would
split all beginning-state/ih/ triphonesinto5clusters, l abeledA-Ein theﬁgure.
Figure10.14 Decision tree for choosing which triphone states (subphone s) to tie together.
This particular tree will cluster state 0 (the beginning sta te) of the triphones /n-ih+l/, /ng-ih+l/,
/m-ih+l/, into cluster class A, and various other triphones into classes B-E. Adapted from Odell
(1995).
Thequestionsusedinthedecisiontreeaskwhetherthephone totheleftorrighthas
a certainphonetic feature ,of the type introducedin Ch. 7. Fig. 10.3 shows a few de-
cisiontreequestions;notethatthereareseparatequestio nsforvowelsandconsonants.

(1995).
Real treeswouldhavemanymorequestions.
How are decision trees like the one in Fig. 10.14trained? The trees are growntop
downfromtheroot. Ateachiteration,thealgorithmconside rseachpossiblequestion q
andeachnode ninthetree. Foreachquestion,itconsidershowthenewsplit wouldim-
pacttheacousticlikelihoodofthetrainingdata. Thealgor ithmcomputesthedifference
between the current acoustic likelihood of the training dat a, and the new likelihood if
themodelsweretiedbasedonsplittingviaquestion q. Thealgorithmpicksthe node n

andquestion qthatgivethemaximumlikelihood. Theproceduretheniterat es,stopping
wheneachleaf nodehasa minimumthresholdnumberofexample s.
Wealsoneedtomodifytheembeddedtrainingalgorithmwesaw inSec.9.7todeal
withcontext-dependentphonesandalsotohandlemixtureGa ussians. Inbothcaseswe
use a morecomplexprocessthat involves cloningandusingextraiterationsof EM,as Cloning
describedinYounget al.(1994).

DRAFTSection10.4. DiscriminativeTraining 351
Feature Phones
Stop bdgkpt
Nasal m nng
Fricative chdh fjhs shthvz zh
Liquid l rw y
Vowel aa ae ahao awax axr ayeher eyihixiyow oy uh uw
FrontVowel ae ehihixiy
CentralVowel aa ahao axr er
BackVowel axow uh uw
HighVowel ihixiyuhuw
Rounded aoow oy uh uw w
Reduced axaxr ix
Unvoiced chf hhkps sht th
Coronal chddh jhlnr ssh tthz zh
Figure10.15 Sample decision tree questions onphonetic features used by Odell (1995).
To train context-dependent models, for example, we ﬁrst use the standard em-
bedded training procedureto train context-independentmo dels, using multiple passes
of EM and resulting in separate single-Gaussians models for each subphone of each
monophone/aa/, /ae/, etc. We then cloneeach monophonemodel, i.e. make identical
copies of the model with its 3 substates of Gaussians, one clo ne for each potential tri-
phone. The Atransition matrices are not cloned, but tied together for al l the triphone

clonesofa monophone. We then runan iterationofEM againand retrainthe triphone
Gaussians. Now for each monophone we cluster all the context -dependent triphones
using the clustering algorithmdescribedon page 350 to get a set of tied state clusters.
Onetypicalstate ischosenastheexemplarforthisclustera ndthe restaretied toit.
We use this same cloning procedure to learn Gaussian mixture s. We ﬁrst use em-
beddedtrainingwithmultipleiterationsofEMtolearnsing le-mixtureGaussianmodels

for each tied triphone state as described above. We then clon e (split) each state into 2
identical Gaussians, perturb the values of each by some epsi lon, and run EM again to
retrain these values. We then split each of the two mixtures, resulting in four, perturb
them, retrain. We continue until we have an appropriate numb er of mixtures for the
amountofobservationsineachstate.
A full context-dependingGMM triphone model is thus created by applying these
twocloning-and-retrainingproceduresin series,asshown schematicallyinFig.10.16.

10.4 DiscriminativeTraining
The Baum-Welch and embedded training models we have present ed for training the
HMM parameters (the AandBmatrices) are based on maximizing the likelihood of
the trainingdata. An alternativeto this maximum likelihood estimation (MLE) is to MLE
focus not on ﬁtting the best model to the data, but rather on discriminating the bestDiscriminativetraining
model from all the other models. Such training procedures in clude Maximum Mu-
tual Information Estimation (MMIE) (Woodland and Povey, 20 02) the use of neural

net/SVMclassiﬁers(BourlardandMorgan,1994)aswellasot hertechniqueslikeMin-
imumClassiﬁcationErrortraining(Chouetal.,1993;McDer mottandHazen,2004)or
Minimum Bayes Risk estimation (Doumpiotis et al., 2003a). W e summarize the ﬁrst

DRAFT352 Chapter 10. SpeechRecognition: AdvancedTopics
iy
t-iy+n t-iy+ng f-iy+l s-iy+l(1) Train monophone 
single Gaussian
models
(2) Clone monophones
 to triphones
... etc.
t-iy+n t-iy+ng f-iy+l s-iy+l (3) Cluster and tie 
triphones
... etc.
... etc.(4) Expand to 
GMMs
Figure10.16 The four stages intraininga tied-mixture triphone acousti c model. AfterYoung etal. (1994).
twoofthese inthenexttwosubsections.
10.4.1 MaximumMutual InformationEstimation
Recall that in Maximum Likelihood Estimation (MLE), we trai n our acoustic model
parameters( AandB)so as to maximizethelikelihoodofthe trainingdata. Consi dera
particular observationsequence O, and a particularHMM model Mkcorrespondingto
word sequence Wk, out of all the possible sentences W′∈L. The MLE criterion thus
maximizes
FMLE(λ) =Pλ(O|Mk) (10.4)
Since our goal in speech recognition is to have the correct tr anscription for the
largest number of sentences, we’d like on average for the pro bability of the correct

maximizes
word string Wkto be high; certainly higher than the probabilityof all the wrongword
stringsWjs.t.j̸=k. But the MLE criterion above does not guarantee this. Thus we ’d
liketopicksomeothercriterionwhichwillletuschosethem odelλwhichassignsthe
highest probability to the correct model, i.e. maximizes Pλ(Mk|O). Maximizing the
probabilityofthewordstringratherthantheprobabilityo ftheobservationsequenceis
calledconditionalmaximumlikelihood estimation orCMLE: CMLE

DRAFTSection10.4. DiscriminativeTraining 353
FCMLE(λ) =Pλ(Mk|O) (10.5)
UsingBayesLaw,wecanexpressthisas
FCMLE(λ) =Pλ(Mk|O) =Pλ(O|Mk)P(Mk)
Pλ(O)(10.6)
Let’s now expand Pλ(O)by marginalizing (summing over all sequences which
could have produced it). The total probability of the observ ation sequence is the
weightedsumoverallwordstringsoftheobservationlikeli hoodgiventhatwordstring:
P(O) =∑
W∈LP(O|W)P(W) (10.7)
Soa completeexpansionofEq.10.6is:
∑M∈LPλ(O|M)P(M)(10.8)
Inaslightlyconfusingbitofstandardnomenclature,CMLEi sgenerallyreferredto
instead as MaximumMutual InformationEstimation(MMIE).T hisis becauseit turns
out that maximizing the posterior P(W|O)and maximizing the mutual information
I(W,O)are equivalent if we assume that the language model probabil ity of each sen-
tenceWis constant (ﬁxed) during acoustic training, an assumption we usually make.
Thusfromhereon we will refer to thiscriterionas the MMIE cr iterionratherthan the
CMLEcriterion,andsohereisEq.10.8restated:

FMMIE(λ) =Pλ(Mk|O) =Pλ(O|Mk)P(Mk)
∑M∈LPλ(O|M)P(M)(10.9)
In a nutshell, then, the goal of MMIE estimation is to maximiz e (10.9)rather than
(10.4). Now if our goal is to maximize Pλ(Mk|O), we not only need to maximize the
numeratorof(10.9),butalsominimizethedenominator. Not icethatwecanrewritethe
denominatortomakeitclearthatitincludesatermequaltot hemodelwearetryingto
maximizeandatermforall othermodels:
Pλ(Mk|O) =Pλ(O|Mk)P(Mk)
Pλ(O|Mk)P(Mk)+∑i̸=kPλ(O|Mi)P(Mi)(10.10)
Thus in order to maximize Pλ(Mk|O), we will need to incrementally change λso

that it increases the probabilityof the correct model, whil e simultaneouslydecreasing
the probability of each of the incorrect models. Thus traini ng with MMIE clearly
fulﬁllstheimportantgoalof discriminating betweenthecorrectsequenceandallother
sequences.
TheimplementationofMMIEisquitecomplex,andwedon’tdis cussithereexcept
to mention that it relies on a variant of Baum-Welch training called Extended Baum-
Welch that maximizes(10.9)instead of (10.4). Brieﬂy, we ca n view this as a two step

algorithm; we ﬁrst use standard MLE Baum-Welch to compute th e forward-backward
counts for the training utterances. Then we compute another forward-backward pass

DRAFT354 Chapter 10. SpeechRecognition: AdvancedTopics
using all other possible utterances and subtract these from the counts. Of course it
turnsoutthatcomputingthisfulldenominatoriscomputati onallyextremelyexpensive,
because it requires runninga full recognitionpass on all th e training data. Recall that
in normal EM, we don’t need to run decoding on the training dat a, since we are only
trying to maximize the likelihood of the correctword sequence; in MMIE, we need
to compute the probabilities of allpossible word sequences. Decoding is very time-
consuming because of complex language models. Thus in pract ice MMIE algorithms
estimate the denominatorby summingoveronlythe pathsthat occurin a word lattice,
asanapproximationtothe fullset ofpossiblepaths.
CMLE was ﬁrst proposed by N´ adas (1983) and MMIE by Bahl et al. (1986), but
practical implementations that actually reduced word erro r rate came much later; see
WoodlandandPovey(2002)orNormandin(1996)fordetails.

10.4.2 AcousticModels basedonPosteriorClassiﬁers
Anotherwaytothinkaboutdiscriminativetrainingistocho oseaclassiﬁerattheframe
levelwhichisdiscriminant. ThuswhiletheGaussian classi ﬁer isbyfarthemost com-
monly used acoustic likelihood classiﬁer, it is possible to instead use classiﬁers that
are naturally discriminative or posterior estimators, suc h as neural networks or SVMs
(supportvectormachines).
The posterior classiﬁer (neural net or SVM) is generally int egrated with an HMM

architecture,isoftencalleda HMM-SVM orHMM-MLPhybrid approach(Bourlard
andMorgan,1994).
The SVM or MLP approaches, like the Gaussian model, estimate the probability
with respect to a cepstral featurevectorat a single time t. Unlike the Gaussian model,
the posterior approaches often uses a larger window of acous tic information, relying
on cepstral feature vectorsfrom neighboringtime periods a s well. Thus the input to a
typical acoustic MLP or SVM might be feature vectors for the c urrent frame plus the

fourpreviousandfourfollowingframes,i.e. atotalof9cep stralfeaturevectorsinstead
ofthesingleonethattheGaussianmodeluses. Becausetheyh avesuchawidecontext,
SVM or MLP models generally use phones rather than subphones or triphones, and
computeaposteriorforeachphone.
TheSVMorMLPclassiﬁersarethuscomputingtheposteriorpr obabilityofastate
jgiventheobservationvectors,i.e. P(qj|ot). (alsoconditionedonthecontext,butlet’s
ignore that for the moment). But the observation likelihood we need for the HMM,

bj(ot),isP(ot|qj). TheBayesrulecanhelpusseehowtocomputeonefromtheothe r.
Thenetiscomputing:
p(qj|ot) =P(ot|qj)p(qj)
p(ot)(10.11)
We canrearrangethetermsasfollows:
p(ot|qj)
p(ot)=P(qj|ot)
p(qj)(10.12)
Thetwo termsontheright-handside of(10.12)canbedirectl ycomputedfromthe
posterior classiﬁer; the numerator is the output of the SVM o r MLP, and the denomi-

DRAFTSection10.5. ModelingVariation 355
nator is the total probability of a given state, summing over all observations (i.e., the
sum over all tofξj(t)). Thus although we cannot directly compute P(ot|qj), wecan
use (10.12)to computep(ot|qj)
p(ot), which is knownas a scaled likelihood (the likelihood Scaledlikelihood
divided by the probability of the observation). In fact, the scaled likelihood is just
as good as the regular likelihood, since the probability of t he observation p(ot)is a
constantduringrecognitionanddoesn’thurtusto havein th eequation.
The supervised training algorithms for training a SVM or MLP posterior phone
classiﬁers require that we know the correct phone label qjfor each observation ot.
We can use the same embedded training algorithm that we saw for Gaussians; we
start with some initial version of our classiﬁer and a word tr anscript for the training
sentences. We run a forced alignment of the training data, pr oducing a phone string,
andnowwe retraintheclassiﬁer, anditerate.

10.5 ModelingVariation
Aswenotedatthebeginningofthischapter,variationisone ofthelargestobstaclesto
successfulspeechrecognition. Wementionedvariationdue tospeakerdifferencesfrom
vocalcharacteristicsordialect,duetogenre(suchasspon taneousversusreadspeech),
and due to the environment (such as noisy versus quiet enviro nments). Handling this
kindofvariationisa majorsubjectofmodernresearch.
10.5.1 Environmental Variationand Noise
Environmentalvariationhasreceivedthemostattentionfr omthespeechliterature,and

a number of techniques have been suggested for dealing with e nvironmental noise.
Spectral subtraction , for example, is used to combat additive noise . Additive noiseSpectral
subtraction
Additivenoise is noise from external sound sources like engines or wind or f ridges that is relatively
constant and can be modeled as a noise signal that is just adde d in the time domain to
the speech waveform to produce the observed signal. In spect ral subtraction, we esti-

mate theaveragenoiseduringnon-speechregionsandthensu btractthisaveragevalue
from the speech signal. Interestingly,speakers often comp ensatefor high background
noise levels by increasing their amplitude, F0, and formant frequencies. This change
in speech production due to noise is called the Lombard effect , named for Etienne Lombard effect
Lombardwhoﬁrst describedit in1911(Junqua,1993).
Other noise robustness techniques like cepstral mean normalization are used toCepstral mean
normalization

deal with convolutionalnoise , noise introducedby channel characteristicslike differ-Convolutionalnoise
entmicrophones. Herewecomputetheaverageofthecepstrum overtimeandsubtract
it from each frame; the average cepstrum models the ﬁxed spec tral characteristics of
themicrophoneandtheroomacoustics(Atal,1974).
Finally, some kinds of short non-verbal sounds like coughs, loud breathing, and
throat clearing, or environmental sounds like beeps, telep hone rings, and door slams,

can be modeled explicitly. For each of these non-verbal soun ds, we create a special
phone and add to the lexicon a word consisting only of that pho ne. We can then use

DRAFT356 Chapter 10. SpeechRecognition: AdvancedTopics
normalBaum-Welch training to train these phonesjust by mod ifyingthe trainingdata
transcripts to include labels for these new non-verbal ‘wor ds’ (Ward, 1989). These
words also need to be added to the language model; often by jus t allowing them to
appearinbetweenanyword.
differences
Speech recognition systems are generally designed to be spe aker-independent, since
it’s rarely practical to collect sufﬁcient training data to build a system for a single
user. But in cases where we have enough data to build speaker- dependent systems,
theyfunctionbetterthanspeaker-independentsystems. Th isonlymakessense;wecan
reducethevariabilityandincreasetheprecisionofourmod elsifweareguaranteedthat
thetest datawill lookmorelikethe trainingdata.
While it is rare to have enough data to train on an individual s peaker, we do have

enough data to train separate models for two important group s of speakers: men ver-
sus women. Since women and men have different vocal tracts an d other acoustic and
phonetic characteristics, we can split the training data by gender, and train separate
acoustic modelsfor men and forwomen. Then when a test senten ce comesin, we use
agenderdetectortodecideifitismaleorfemale,andswitch tothoseacousticmodels.
GenderdetectorscanbebuiltoutofbinaryGMMclassiﬁersba sedoncepstralfeatures.
Suchgender-dependentacousticmodeling isusedinmostLVCSRsystems.

Althoughwe rarelyhave enoughdata to train on a speciﬁc spea ker,there are tech-
niques that work quite well at adapting the acoustic models t o a new speaker very
quickly. For example the MLLR(Maximum Likelihood Linear Regression ) tech- MLLR
nique (Leggetterand Woodland, 1995)is used to adapt Gaussi an acoustic modelsto a
small amount of data from a new speaker. The idea is to use the s mall amountof data
to train a linear transformto warp the means of the Gaussians . MLLR and other such

techniques for speaker adaptation have been one of the largest sources of improve-Speaker
adaptation
mentinASRperformanceinrecentyears.
The MLLRalgorithmbeginswith a trained acoustic model and a small adaptation
dataset from a new speaker. The adaptation set can be as small as 3 sentences or 10
secondsofspeech. Theideaisto learna lineartransformmat rix(W) andabiasvector
(ω) to transform the means of the acoustic model Gaussians. If t he old mean of a
Gaussianis µ, theequationforthenewmean ˆµisthus:
ˆµ=Wµ+ω (10.13)

In the simplest case, we canlearn a single globaltransforma nd applyit to each Gaus-
sianmodels. Theresultingequationfortheacousticlikeli hoodisthusonlyveryslightly
modiﬁed:
bj(ot) =1√
2π|Σj|exp(
−1
2(ot−(Wµj+ω))TΣ−1
j(ot−(Wµj+ω)))
(10.14)
The transform is learned by using linear regression to maxim ize the likelihood of
the adaptationdataset. We ﬁrst run forward-backwardalign menton the adaptationset

DRAFTSection10.5. ModelingVariation 357
to compute the state occupation probabilities ξj(t). We then compute Wby solving a
system of simultaneousequationsinvolving ξj(t). Ifenoughdata is available,it’s also
possibletolearna largernumberoftransforms.
MLLRisan exampleofthe linear transform approachtospeakeradaptation,one
ofthethreemajorclassesofspeakeradaptationmethods;th eothertwoare MAPadap-
tationandSpeakerClustering/SpeakerSpace approaches. SeeWoodland(2001)for
a comprehensivesurveyofspeakeradaptationwhichcoversa ll threefamilies.
MLLRandotherspeakeradaptationalgorithmscanalsobeuse dtoaddressanother
large source of error in LVCSR, the problem of foreign or dial ect accented speakers.
Word error rates go up when the test set speaker speaks a diale ct or accent (such as
Spanish-accentedEnglishorsouthernaccentedMandarinCh inese)thatdiffersfromthe
(usually standard) training set, Here we can take an adaptat ion set of a few sentences
fromsay 10 speakers, andadapt to them as a group,creatingan MLLRtransformthat

addresses whatever characteristics are present in the dial ect or accent (Huang et al.,
2000;TomokiyoandWaibel,2001;Wanget al.,2003;Zhenget a l.,2005).
Another useful speaker adaptation technique is to control f or the differing vocal
tract lengths of speakers. Cues to the speaker’s vocal tract length are present in the
signal; for example speakers with longer vocal tracts tend t o have lower formants.
Vocaltractlengthcanthereforebedetectedandnormalized ,inaprocesscalled VTLN VTLN

(VocalTractLengthNormalization );see theendnotesfordetails.
10.5.3 Pronunciation Modeling: Variationdue to Genre
Wesaidatthebeginningofthechapterthatrecognizingconv ersationalspeechisharder
forASRsystemsthanrecognizingreadspeech. Whataretheca usesofthisdifference?
Isitthedifferenceinvocabulary? Grammar? Somethingabou tthespeakerthemselves?
Perhapsit’sa factaboutthemicrophonesortelephoneusedi n theexperiment.
Noneoftheseseemstobethecause. Inawell-knownexperimen t,Weintraubet al.

(1996) compared ASR performance on natural conversational speech versus perfor-
manceonreadspeech,controllingfortheinﬂuenceofpossib lecausal factors. Pairsof
subjects in the lab had spontaneous conversations on the tel ephone. Weintraub et al.
(1996) then hand-transcribed the conversations, and invit ed the participants back into
the lab to read their own transcripts to each other overthe sa me phone lines as if they
weredictating. Boththenaturalandreadconversationswer erecorded. NowWeintraub

et al. (1996) had two speech corpora from identical transcri pts; one original natural
conversation, and one read speech. In both cases the speaker , the actual words, and
the microphone were identical; the only difference was the n aturalness or ﬂuency of
the speech. They found that read speech was much easier (WER= 29%) than conver-
sational speech (WER=53%). Since the speakers, words, and c hannelwere controlled
for,this differencemust be modelablesomewherein the acou stic modelor pronuncia-
tionlexicon.

Saraclar et al. (2000) tested the hypothesis that this difﬁc ulty with conversational
speech was due to changed pronunciations, i.e., to a mismatc h between the phone
strings in the lexicon and what people actually said. Recall from Ch. 7 that conver-
sational corpora like Switchboard contain many different p ronunciations for words,
(such as 12 differentpronunciationsfor becauseand hundredsfor the). Saraclar et al.

DRAFT358 Chapter 10. SpeechRecognition: AdvancedTopics
(2000) showed in an oracle experiment that if a Switchboard r ecognizer is told which
pronunciationsto useforeachword,theworderrorratedrop sfrom47%to27%.
Ifknowingwhichpronunciationtouseimprovesaccuracy,co uldweimproverecog-
nitionbysimplyaddingmorepronunciationsforeachwordto thelexicon?
Alas,itturnsoutthataddingmultiplepronunciationsdoes n’tworkwell,evenifthe
list ofpronunciationisrepresentedasanefﬁcientpronunc iationHMM(Cohen,1989).
Adding extra pronunciations adds more confusability; if a c ommon pronunciation of
the word “of” is the single vowel [ax], it is now very confusab le with the word “a”.
Another problem with multiple pronunciations is the use of V iterbi decoding. As we
said on page 338, the Viterbi decoder ﬁnds the best phone stri ng, rather than the best
word string, hence biasing against words with many pronunci ations. Finally, using
multiple pronunciations to model coarticulatory effects m ay be unnecessary because

CD phones (triphones) are already quite good at modeling the contextual effects in
phones due to neighboring phones like ﬂapping and vowel-red uction (Jurafsky et al.,
2001).
Instead, most current LVCSR systems use a very small number o f pronunciations
per word. What is commonly done is to start with a multiple pro nunciation lexicon,
where the pronunciations are found in dictionaries or are ge nerated via phonological
rulesofthetypedescribedinCh.7. AforcedViterbiphoneal ignmentisthenrunofthe

2001).
trainingset,usingthisdictionary. Theresultofthealign mentisaphonetictranscription
of the training corpus, showing which pronunciation was use d, and the frequency of
each pronunciation. We can then collapse similar pronuncia tions(for example if two
pronunciations differ only in a single phone substitution w e chose the more frequent
pronunciation). We then chose the maximum likelihood pronu nciationfor each word.
Forfrequentwordswhichhavemultiplehigh-frequencypron unciations,somesystems

chosemultiplepronunciations,andannotatethedictionar ywiththeprobabilityofthese
pronunciations;theprobabilitiesareusedincomputingth eacousticlikelihood(Cohen,
1989;Hainet al.,2001;Hain,2002).
Finding a better method to deal with pronunciationvariatio n remains an unsolved
researchproblem. Onepromisingavenueistofocusonnon-ph oneticfactorsthataffect
pronunciation. Forexamplewordswhicharehighlypredicta ble,oratthebeginningor
end of intonation phrases, or are followed by disﬂuencies, a re pronounced very dif-

ferently (Jurafsky et al., 1998; Fosler-Lussier and Morgan , 1999; Bell et al., 2003).
Fosler-Lussier (1999) shows reductions in word errors by us ing these sorts of factors
to predict which pronunciation to use. Another exciting lin e of research in pronun-
ciation modeling uses a dynamic Bayesian network to model th e complex overlap in
articulatorsthat producesphoneticreduction(Livescuan dGlass, 2004b,2004a).
Another important issue in pronunciationmodeling is deali ng with unseen words.

In web-based applications such as telephone-based interfa ces to the Web, the recog-
nizer lexicon must be automatically augmented with pronunc iations for the millions
of unseen words, particularly names, that occur on the Web. G rapheme-to-phoneme
techniqueslikethosedescribedin Sec.8.2.3areusedtosol vethisproblem.

DRAFTSection10.6. Metadata: Boundaries,Punctuation,andDisﬂ uencies 359
10.6 Metadata: Boundaries,Punctuation,andDisﬂuencies
The output of the speech recognition process as we have descr ibed it so far is just
a string of raw words. Consider the following sample gold-st andard transcript (i.e.,
assumingperfectwordrecognition)ofpartofa dialogue(Jo nesetal.,2003):
yeah actually um i belong to a gym down here a gold’s gym uh-huh and uh
exercisei try to exerciseﬁve daysa week um andi usually doth at uhwhat type
ofexercisingdoyoudointhegym
Comparethedifﬁculttranscriptabovewiththefollowingmu chclearerversion:
A: Yeah I belong to a gym down here. Gold’s Gym. And I try to exer cise ﬁve
daysa week. AndI usuallydothat.
B: What typeofexercisingdoyoudointhegym?
The raw transcript is not divided up among speakers, there is no punctuation or
capitalization, and disﬂuencies are scattered among the wo rds. A number of studies
have shown that such raw transcripts are harder for people to read Jones et al. (2003,

2005) and that adding, for example, commas back into the tran script improve sthe
accuracy of informationextraction algorithms on the trans cribed text (Makhoulet al.,
2005; Hillard et al., 2006). Post-processing ASR output inv olves tasks including the
diarization: Many speech tasks have multiple speakers, such as telephone conver- diarization
sations, business meetings, and news reports (with multipl e broadcasters). Di-
arizationisthetaskofbreakingupaspeechﬁlebyspeakeras signingpartsofthe

transcripttotherelevantspeakers,likethe A:andB:labelsabove.
sentence boundary detection: We discussed the task of breaking speech into sen-sentencesegmentation
tences(sentencesegmentation)inCh.3andCh.8. Butfortho setaskswealready
addpunctuationlikeperiodstohelpus;fromspeechwedon’t alreadyhavepunc-
tuation,just words. Sentencesegmentationfromspeechhas the addeddifﬁculty
that the transcribed words will be errorful, but has the adva ntage that prosodic

featureslike pausesandsentence-ﬁnalintontationcanbeu sedascues.
truecasing: Words in a clean transcript need to have sentence-initial wo rds start- Truecasing
ing with an upper-case letter, acronyms all in capitals, and so on. Truecasing
is the task of assigning the correct case for a word, and is oft en addressed as
a HMM classiﬁcation task like part-of-speech tagging, with hidden states like
ALL-LOWER CASE ,UPPER-CASE-INITIAL,all-caps,andsoon.
punctuation detection: In addition to segmenting sentences, we need to choosePunctuationdetection

sentence-ﬁnal punctuation (period, question mark, exclam ation mark), and in-
sertcommasandquotationmarksandso on.
disﬂuency detection: Disﬂuencies can be removed from a transcript for readabilit y,Disﬂuency
detection
or at least marked off with commas or font changes. Since stan dard recogniz-
ers don’tactually includedisﬂuencies (like word fragment s)in their transcripts,
disﬂuency detection algorithms can also play an important r ole in avoiding the
misrecognizedwordsthat mayresult.

DRAFT360 Chapter 10. SpeechRecognition: AdvancedTopics
Marking these features (punctuation, boundaries, diariza tion) in the text output is
oftencalled metadata orsometimes richtranscription . Let’slookatacoupleofthese Metadata
Richtranscription tasksinslightlymoredetail.
Sentence segmentation can be modeled as a binary classiﬁcation task, in which
each boundary between two words is judged as a sentence bound ary or as sentence-
internal. Such classiﬁers can use similar features to the se ntence segmentation dis-
cussed in Sec. 8.3.1, such as words and part-of-speech tags a round each candidate
boundary,or length featuressuch as the distance from the pr eviouslyfoundboundary.
We can also make use of prosodic features, especially pause d uration, word duration
(recallthatsentence-ﬁnalwordsarelengthened),andpitc hmovements.
without drinking water due to the flood   (pause)  many communities are still cut off...
200ms200ms 200ms 200msSENTENCE
BOUNDARY

BOUNDARY
Figure10.17 Candidate sentence boundaries computed at each inter-word boundary showing
prosodic feature extractionregions from the Shribergetal . (2000) algorithm.
Fig. 10.17 shows the candidate boundary locations in a sampl e sentence. Com-
monlyextractedfeaturesinclude:
pausefeatures: durationofthe interwordpauseat thecandidateboundary.
duration features: durations of the phone and rime (nucleus plus coda) precedin g
the candidate boundary. Since some phones are inherently lo nger than others,
eachphoneisnormalizedto themeandurationforthatphone.

F0 features: thechange in pitch across the boundary; sentence boundaries often
havepitch reset (an abrupt change in pitch), while non-boundaries are more
likely to have continuous pitch across the boundary. Anothe r useful F0 feature
isthepitchrange ofthepreboundaryword;sentencesoftenendwitha ﬁnalfall
(Sec.8.3.3)whichiscloseto thespeaker’sF0baseline.
Forpunctuationdetection ,similar featuresare used asforsentenceboundaryde-
tection,butwithmultiplehiddenclasses(comma,sentence -ﬁnalquestionmark,quota-
tionmark,nopunctuation). insteadofjust two.

For both of these tasks, instead of a simple binary classiﬁer , sequence informa-
tioncan beincorporatedbymodelingsentencesegmentation asan HMMin whichthe
hidden states correspond to sentence boundary or non-bound ary decisions. We will
describe methodsfor combiningprosodicand lexical featur esin more detail when we
introducedialogueact detectioninSec.24.5.2.
Recall from Sec. 12.8.1 that disﬂuencies orrepairin conversation include phe- Disﬂuencies
Repair nomenalikethefollowing:

DRAFTSection10.7. SpeechRecognitionbyHumans 361
Disﬂuencytype Example
ﬁllers(orﬁlled pauses ):But,uh,that wasabsurd
wordfragments A guywentto a d-,a landﬁll
repetitions: it wasjust a changeof ,changeof location
restarts it’s –Iﬁndit verystrange
The ATIS sentence in Fig. 10.18 shows examples of a restart an d the ﬁller uh,
showingthe interruptionpoint thatstartsthe editingphase . Interruptionpoint
Figure10.18 Repeated from Fig. 12.19An example of a disﬂuency (after Shr iberg (1994); terminology is from
Levelt (1983)).
Detection methods for disﬂuencies are very similar to detec ting sentence bound-
aries; a classiﬁer is trained to make a decision at each word b oundary,using both text
and prosodic features. HMM and CRF classiﬁers are commonly u sed, and features
are quite similar to the features for boundary detection, in cluding neighoring words
and part-of-speech tags, the duration of pauses at the word b oundary, the duration of

the word and phonesprecedingthe boundary,the differencei n pitch values across the
boundary,andso on.
For detecting fragments, features for detecting voice qual ity are used (Liu, 2004),
suchasjitter,ameasureofperturbationinthepitchperiod(Rosenberg,1 971),spectral Jitter
tilt, the slope of the spectrum, (see Sec. 9.3.1), and open quotient , the percentage of Spectral tilt
Open quotient theglottalcycleinwhichthevocalfoldsareopen(Fant,199 7).
10.7 SpeechRecognitionbyHumans
Humans are of course much better at speech recognition than m achines; current ma-

chines are roughly about ﬁve times worse than humans on clean speech, and the gap
seemstoincreasewithnoisyspeech.
SpeechrecognitioninhumanssharessomefeatureswithASRa lgorithms. Wemen-
tioned above that signal processing algorithms like PLP ana lysis (Hermansky, 1990)
were in fact inspired by properties of the human auditory sys tem. In addition, three
properties of human lexical access (the process of retrieving a word from the mental Lexicalaccess
lexicon) are also true of ASR models: frequency ,parallelism , andcue-based pro-

cessing. For example, as in ASR with its N-gram language models, human lexical
access is sensitive to word frequency . High-frequency spoken words are accessed
faster or with less information than low-frequency words. T hey are successfully rec-

DRAFT362 Chapter 10. SpeechRecognition: AdvancedTopics
ognized in noisier environmentsthan low frequency words, o r when only parts of the
wordsarepresented(Howes,1957;Grosjean,1980;Tyler,19 84,interalia). LikeASR
models, human lexical access is parallel: multiple words are active at the same time
(Marslen-WilsonandWelsh, 1978;SalasooandPisoni,1985, interalia).
Finally, human speech perception is cue based : speech input is interpreted by in-
tegrating cues at many different levels. Human phone percep tion combines acous-
tic cues, such as formant structure or the exact timing of voi cing, (Oden and Mas-
saro, 1978; Miller, 1994) visual cues, such as lip movement ( McGurk and Macdon-
ald, 1976; Massaro and Cohen, 1983; Massaro, 1998) and lexic al cues such as the
identity of the word in which the phone is placed (Warren, 197 0; Samuel, 1981; Con-
nine and Clifton, 1987; Connine, 1990). For example, in what is often called the

phoneme restoration effect , Warren (1970) took a speech sample and replaced onePhonemerestoration effect
phone (e.g. the [s] in legislature ) with a cough. Warren found that subjects listening
to the resulting tape typically heard the entire word legislature including the [s], and
perceived the cough as background. In the McGurk effect , (McGurk and Macdon- McGurk effect
ald, 1976) showed that visual input can interfere with phone perception, causing us

to perceive a completely different phone. They showed subje cts a video of someone
saying the syllable gain which the audio signal was dubbed instead with someone
saying the syllable ba. Subjects reported hearing something like dainstead. It is def-
initely worth trying this out yourself from video demos on th e web; see for example
http://www.haskins.yale.edu/featured/heads/mcgurk.h tml.
Othercuesinhumanspeechperceptionincludesemantic wordassociation (words Word association
are accessed more quicklyif a semanticallyrelated wordhas beenheard recently)and

repetitionpriming (wordsareaccessedmorequicklyiftheythemselveshavejus tbeenRepetitionprimining
heard). Theintuitionsofboththese resultsareincorporat edinto recentlanguagemod-
els discussed in Ch. 4, such as the cache model of Kuhn and De Mo ri (1990), which
modelsrepetitionpriming,orthetriggermodelofRosenfel d(1996)andtheLSAmod-
elsofCoccaroandJurafsky(1998)andBellegarda(1999),wh ichmodelwordassocia-
tion. Inafascinatingreminderthatgoodideasareneverdis coveredonlyonce,Coleand

Rudnicky (1983) point out that many of these insights about c ontext effects on word
and phone processing were actually discovered by William Ba gley (1901). Bagley
achieved his results, including an early version of the phon eme restoration effect, by
recording speech on Edison phonograph cylinders, modifyin g it, and presenting it to
subjects. Bagley’sresultswereforgottenandonlyredisco veredmuchlater.2
One differencebetween current ASR modelsand humanspeech r ecognitionis the

time-course of the model. It is important for the performanc e of the ASR algorithm
that the decoding search optimizes over the entire utteranc e. This means that the best
sentence hypothesisreturnedby a decoder at the end of the se ntence may be very dif-
ferent than the current-best hypothesis, halfway into the s entence. By contrast, there
is extensiveevidencethat humanprocessingis on-line: peopleincrementallysegmentOn-lineprocessing
and utterance into words and assign it an interpretation as t hey hear it. For example,

Marslen-Wilson (1973) studied close shadowers : people who are able to shadow (re-
peat back) a passage as they hear it with lags as short as 250 ms . Marslen-Wilson
found that when these shadowers made errors, they were synta ctically and semanti-
2Recall the discussion on page 13 of multiple independent dis covery in science.

DRAFTSection10.8. Summary 363
cally appropriatewith the context,indicatingthat word se gmentation,parsing, and in-
terpretationtookplacewithinthese250ms. Cole(1973)and Cole andJakimik(1980)
foundsimilareffectsintheirworkonthedetectionofmispr onunciations. Theseresults
haveled psychologicalmodelsofhumanspeechperception(s uchastheCohortmodel
(Marslen-WilsonandWelsh,1978)andthecomputationalTRA CEmodel(McClelland
and Elman, 1986)) to focus on the time-course of word selecti on and segmentation.
TheTRACEmodel,forexample,isaconnectionistinteractiv e-activationmodel,based
onindependentcomputationalunitsorganizedintothreele vels: feature,phoneme,and
word. Eachunit representsa hypothesisaboutitspresencei n the input. Unitsare acti-
vatedinparallelbytheinput,andactivationﬂowsbetweenu nits;connectionsbetween
unitsondifferentlevelsareexcitatory,whileconnection sbetweenunitsonsinglelevel
areinhibitatory. Thusthe activationofawordslightlyinh ibitsallotherwords.

We have focused on the similarities between human and machin e speech recogni-
tion; there are also many differences. In particular, many o ther cues have been shown
to play a role in human speech recognition but have yet to be su ccessfully integrated
into ASR. The most important class of these missing cues is pr osody. To give only
one example, Cutler and Norris (1988), Cutler and Carter (19 87) note that most mul-
tisyllabic English word tokens have stress on the initial sy llable, suggesting in their

metrical segmentation strategy (MSS) that stress should be used as a cue for word
segmentation. Anotherdifferenceis that humanlexicalacc ess exhibits neighborhood
effects(the neighborhood of a word is the set of words which closely r esemble it).
Words with large frequency-weightedneighborhoodsare acc essed slower than words
with less neighbors(Luce et al., 1990). Current models of AS R don’t generallyfocus
onthisword-levelcompetition.
10.8 Summary

•We introduced two advanced decoding algorithms: The multip ass (N-best or
lattice)decodingalgorithm,and stackorA∗decoding.
•Advancedacousticmodelsarebasedoncontext-dependent triphones ratherthan
phones. Because the complete set of triphones would be too la rge, we use a
smallernumberofautomaticallyclusteredtriphonesinste ad.
•Acousticmodelscanbe adaptedtonewspeakers.
•Pronunciation variation is a source of errors in human-huma n speech recogni-
tion,butonethatis notsuccessfullyhandledbycurrenttec hnology.

See the previouschapterformost of the relevantspeechreco gnitionhistory. Note that
although stack decoding is equivalent to the A∗searchdeveloped in artiﬁcial intelli- A∗search
gence, the stack decoding algorithm was developed independ ently in the information

DRAFT364 Chapter 10. SpeechRecognition: AdvancedTopics
theory literature and the link with AI best-ﬁrst search was n oticed only later (Jelinek,
1976). Useful references on vocal tract length normalizati on include (Cohen et al.,
1995;Wegmannetal., 1996;EideandGish,1996;LeeandRose, 1996;Wellinget al.,
2002;Kimet al.,2004).
Therearemanynewdirectionsincurrentspeechrecognition researchinvolvingal-
ternativesto the HMM model. Such new architecturesinclude newgraphical models
(dynamicbayesnets,factorialHMMs,etc)(Zweig,1998;Bil mes,2003;Livescuetal.,
2003; Bilmes and Bartels, 2005; Frankel et al., 2007), as wel l as attempts to replace
theframe-based HMM acoustic model (that make a decision about each frame) wi th Frame-based
segment-based recognizers that attempt to detect variable-length segments (phones) Segment-based
(Digilakis, 1992; Ostendorf et al., 1996; Glass, 2003). New landmark-based recog-
nizersandarticulatoryphonology-basedrecognizersfocu sontheuseofdistinctivefea-

tures,deﬁnedacousticallyorarticulatorily(respective ly)(Niyogietal.,1998;Livescu,
2005;Hasegawa-Johnsonandet al, 2005;JunejaandEspy-Wil son,2003).
See Shriberg (2005) for an overview of metadata research. Sh riberg (2002) and
Nakatani and Hirschberg (1994) are computationally-focus ed corpus studies of the
acoustic and lexical properties of disﬂuencies. Early pape rs on sentence segmenta-
tionfromspeechincludeWangandHirschberg(1992),Ostend orfandRoss(1997)See

Shriberg et al. (2000), Liu et al. (2006a) for recent work on s entence segmentation,
Kim and Woodland (2001), Hillard et al. (2006) on punctuatio n detection, Nakatani
and Hirschberg (1994), Honal and Schultz (2003, 2005), Leas e et al. (2006), and a
numberof papersthat jointly addressmultiplemetadata ext ractiontasks (Heemanand
Allen,1999;Liuet al.,2005,2006b).
Exercises
10.1Implement the Stack decoding algorithm of Fig. 10.7 on page 3 45. Pick a very
simpleh∗function like an estimate of the number of words remaining in the
sentence.

Exercises
sentence.
10.2Modifythe forward algorithmof Fig. 9.23 from Ch. 9 to use the tree-structured
lexiconofFig. 10.10onpage347.
10.3Many ASR systems, including the Sonic and HTK systems, use a d ifferent al-
gorithm for Viterbi called the token-passing Viterbi algorithm (Young et al.,
1989). Readthispaperandimplementthisalgorithm.

Chapter11
Computational Phonology
bidakupadotigolabubidakutupiropadotigolabutupirobid aku...
Wordsegmentation stimulus (Saffranet al.,1996a)
Recall from Ch. 7 that phonology is the area of linguistics that describes the sys-
this system of sounds is related to the rest of the grammar. Th is chapter introduces
computationalphonology ,the useofcomputationalmodelsin phonologicaltheory.computational
phonology
Onefocusofcomputationalphonologyisoncomputationalmo delsofphonological

phonology
representation,andonhowtousephonologicalmodelstomap fromsurfacephonolog-
ical forms to underlying phonologicalrepresentation. Mod els in (non-computational)
phonologicaltheoryaregenerative;thegoalofthemodelis torepresenthowanunder-
lyingformcangenerateasurfacephonologicalform. Incomp utation,wearegenerally
moreinterestedinthealternativeproblemof phonologicalparsing ;goingfromsurface
formto underlyingstructure. Onemajortoolforthistask is the ﬁnite-stateautomaton,
which is employed in two families of models: ﬁnite-state phonology andoptimality

theory.
Arelatedkindofphonologicalparsingtaskis syllabiﬁcation : thetaskofassigning
syllable structure to sequences of phones. Besides its theo retical interest, syllabiﬁ-
cation turns out to be a useful practical tool in aspects of sp eech synthesis such as
pronunciation dictionary design. We therefore summarize a few practical algorithms
forsyllabiﬁcation.
Finally, we spend the remainder of the chapter on the key prob lemof how phono-
logicalandmorphologicalrepresentationscanbelearned.
11.1 Finite-StatePhonology

Ch. 3 showed that spelling rules can be implemented by transd ucers. Phonological
rules can be implemented as transducers in the same way; inde ed the original work
by Johnson (1972) and Kaplan and Kay (1981) on ﬁnite-state mo dels was based on
phonological rules rather than spelling rules. There are a n umber of different models
ofcomputationalphonology thatuseﬁniteautomatainvariouswaystorealizephono-
logicalrules. We will describethe two-levelmorphology ofKoskenniemi(1983)ﬁrst

mentionedinCh.3. Let’sbeginwiththeintuition,byseeing thetransducerinFig.11.1
whichmodelsthesimpliﬁedﬂappingrulein(11.1):
/t/→[dx] /´VV (11.1)
The transducer in Fig. 11.1 accepts any string in which ﬂaps o ccur in the correct

DRAFT366 Chapter 11. ComputationalPhonology
0 1
3other
V:@
tV:@
t:dx
t
t   otherV: @
otherV:@/
V: @/
V: @/
Figure11.1 Transducer forEnglishFlapping: ARPAbet“dx” indicates aﬂ ap,andthe“other”
symbolmeans“anyfeasiblepairnotusedelsewhereinthetra nsducer”. “@”means“anysymbol
not used elsewhere on anyarc”.
places(afterastressedvowel,beforeanunstressedvowel) ,andrejectsstringsinwhich
ﬂappingdoesn’toccur,orinwhichﬂappingoccursin thewron genvironment.1
We’ve seen both transducers and rules before; the intuition of two-level morphol-
ogy is to augment the rule notation to correspond more natura lly to transducers. We
motivatehisideabybeginningwiththenotionof ruleordering . Inatraditionalphono-
logical system, many different phonologicalrules apply be tween the lexical form and
the surface form. Sometimes these rules interact; the outpu t from one rule affects the
input to another rule. One way to implement rule-interactio n in a transducer system

is to run transducersin a cascade. Consider, for example, the rules that are needed to
deal with the phonologicalbehaviorof the English nounplur alsufﬁx-s. Thissufﬁx is
pronounced[ix z] after the phones [s], [sh], [z], [zh], [ch] , or [jh] (so peachesis pro-
nounced[piy ch ix z], and faxesis pronounced[f ae k s ix z]),[z] aftervoiced sounds
(pigsis pronounced [p ih g z]), and [s] after unvoiced sounds ( catsis pronounced [k
ae t s]). We model this variation by writing phonological rul es for the realization of

the morpheme in different contexts. We ﬁrst need to choose on e of these three forms
([s], [z], [ix z]) as the “lexical” pronunciationof the sufﬁ x; we chose [z] only because
it turns out to simplify rule writing. Next we write two phono logicalrules. One, sim-
ilar to the E-insertion spelling rule of page 63, inserts an [ ix] after a morpheme-ﬁnal
sibilant and beforethe plural morpheme[z]. The other makes sure that the -ssufﬁx is
properlyrealizedas[s] afterunvoicedconsonants.
ǫ→ix /[+sibilant] ˆ z # (11.2)
z→s/[-voice] ˆ # (11.3)

These two rules must be ordered; rule (11.2) must apply before (11.3). This is
because the environmentof (11.2) includes z, and the rule (11.3)changes z. Consider
runningbothrulesonthe lexicalform foxconcatenatedwiththeplural -s:
Lexicalform: faa kˆz
(11.2)applies: faa ksˆix z
(11.3)doesn’tapply: faa ksˆ ixz
1For pedagogical purposes, this example assumes (incorrect ly) that the factors that inﬂuence ﬂapping are
purely phonetic and are non-stochastic.

DRAFTSection11.1. Finite-StatePhonology 367
If the devoicingrule (11.3)was orderedﬁrst, we would get th e wrong result. This
situation,inwhichoneruledestroystheenvironmentforan other,iscalled bleeding:2bleeding
Lexical form: f aak sˆz
(11.3)applies: f aak sˆs
(11.2)doesn’tapply: f aak sˆs
As was suggested in Ch. 3, each of these rulescan be represent edby a transducer.
Sincetherulesareordered,thetransducerswouldalsoneed tobeordered. Forexample
if they are placed in a cascade, the output of the ﬁrst transducer would feed the input
ofthesecondtransducer.
Many rules can be cascaded togetherthis way. As Ch. 3 discuss ed, runninga cas-
cade, particularly one with many levels, can be unwieldy, an d so transducer cascades
areusuallyreplacedwithasinglemorecomplextransducerb ycomposing theindivid-
ualtransducers.
Koskenniemi’s method of two-level morphology that was sketchily introduced in
Ch. 3 is another way to solve the problem of rule ordering. Kos kenniemi (1983) ob-

servedthatmost phonologicalrulesina grammarareindepen dentof oneanother;that
feedingand bleedingrelationsbetween rules are not the nor m.3Since this is the case,
Koskenniemi proposed that phonologicalrules be run in para llel rather than in series.
Thecaseswherethereisruleinteraction(feedingorbleedi ng)wedealwithbyslightly
modifying some rules. Koskenniemi’s two-level rules can be thought of as a way of
expressing declarativeconstraints onthewell-formednessofthelexical-surfacemap-
ping.
Two-levelrulesalso differfromtraditionalphonological rulesby explicitlycoding

ping.
whentheyareobligatoryoroptional,byusingfourdifferin gruleoperators ;the⇔rule
correspondstotraditional obligatory phonologicalrules,whilethe ⇒ruleimplements
optionalrules :
Rule type Interpretation
a:b⇐cdaisalwaysrealizedas bin thecontext cd
a:b⇒cdamayberealizedas bonlyinthecontext cd
a:b⇔cdamustberealizedas bincontext cdandnowhereelse
a:b/⇐cdaisneverrealizedas binthecontext cd
The most important intuition of the two-level rules, and the mechanism that lets

them avoidfeedingand bleeding,is their ability to represe ntconstraintson two levels .
Thisisbasedontheuseofthecolon(“:”),whichwastouchedo nverybrieﬂyinCh.3.
Thesymbol a:bmeansa lexical athat mapsto a surface b. Thusa:b⇔:cmeans
ais realized as bafter asurfacec. By contrast a:b⇔c:means that ais realized
asbafter alexicalc. As discussed in Ch. 3, the symbol cwith no colon is equivalent
toc:cthat meansalexical cwhichmapsto asurface c.
2If wehad chosen to represent thelexical pronunciation of -sas [s] rather than [z], wewould have written

theruleinversely tovoicethe -safter voicedsounds,buttheruleswouldstillneedtobeorde red; theordering
would simply ﬂip.
3Feeding is a situation in which one rule creates the environm ent for another rule and so must be run
beforehand.

DRAFT368 Chapter 11. ComputationalPhonology
Fig.11.2showsanintuitionforhowthetwo-levelapproacha voidsorderingforthe
ix-insertionand z-devoicingrules. The idea is that the z-d evoicingrule maps a lexical
z-insertiontoa surfacesandtheixrulereferstothe lexicalz.
lexical level
surface levelix-insertion
[+sib]
[-voice]^z
ixs
z devoicing
Figure11.2 The constraints for the 1-insertion and z-devoicing rules both refer to a lexicalz,
not asurfacez.
Thetwo-levelrulesthatmodelthisconstraintareshownin( 11.4)and(11.5):
ǫ:ix⇔[+sibilant]: ˆ z: # (11.4)
z:s⇔[-voice]: ˆ # (11.5)
As Ch. 3 discussed, there are compilation algorithms for cre ating automata from
rules. Kaplan and Kay (1994) give the general derivation of t hese algorithms, and
Antworth (1990) gives one that is speciﬁc to two-level rules . The automata corre-
sponding to the two rules are shown in Fig. 11.3 and Fig. 11.4. Fig. 11.3 is based on
Fig. 3.17 of Ch. 3; see page 64 for a reminder of how this automa ton works. Note in

Fig.11.3thatthepluralmorphemeisrepresentedbyz:,indi catingthattheconstraintis
expressedabouta lexicalratherthansurfacez.
0 4 12other
^:   
other
#
z: :ix^:z:
^:
s, sh
##, other#,other[+sib][+sib][+sib]∋∋
∋
∋
Figure11.3 The transducer for the ix-insertion rule 11.2. The rule can b e readwhenever a
morpheme ends inasibilant,and the followingmorpheme is wo rd-ﬁnalz,insert [ix].
Fig.11.5showsthetwoautomataruninparallelontheinput[ faaksˆz]. Notethat
boththeautomataassumesthedefaultmappingˆ: ǫtoremovethemorphemeboundary,

0 4 12other
^:   
other
#
z: :ix^:z:
^:
s, sh
∋
∋
andthatbothautomataendinan acceptingstate.

DRAFTSection11.2. AdvancedFinite-StatePhonology 369
1
u^
other
z:s ^:
z, sh,
# #, other#,other:[-voice]:[-voice]∋
∋
#
s, zh
Figure11.4 The transducer for the z-devoicing rule 11.3. This rule migh t be summarized
Devoice the morpheme z ifit followsamorpheme-ﬁnal voicele ss consonant.
aaks^ Intermediate
aaksixz Surfaceix-insertion 0
100
14
0
v
000
1 z-devoicing0
03
0
v1
Figure11.5 The transducer for the ix-insertion rule 11.2 and the z-devo icing rule 11.3 run in
parallel.
11.2 AdvancedFinite-StatePhonology
11.2.1 Harmony
Finite-state modelsofphonologyhavealso beenappliedtom oresophisticatedphono-
logical and morphological phenomena. Let’s consider a ﬁnit e-state model of a well-
known complex interaction of three phonological rules in th e Yawelmani dialect of
Yokuts,a NativeAmericanlanguagespokenin California.4
First, Yokuts (like many other languagesincluding for exam ple Turkish and Hun-
garian)has vowelharmony . Vowelharmonyisaprocessinwhichavowelchangesits vowelharmony

form to look like a neighboring vowel. In Yokuts, a sufﬁx vowe l changes its form to
agreeinbacknessandroundnesswiththeprecedingstemvowe l. Thatis,afrontvowel
like/i/will appear as a back vowel [u]if the stem vowel is /u/. ThisHarmony rule
appliesifthesufﬁxandstemvowelsareofthesameheight(e. g.,/u/and/i/bothhigh,
/o/and/a/bothlow):5
4Theserules wereﬁrstdrawn upinthetraditional Chomskyand Halle (1968)formatbyKisseberth (1969)
following the ﬁeld work of Newman (1944).

5Examples fromColeand Kisseberth (1995). Someparts of syst emsuch asvowelunderspeciﬁcation have
been removed for pedagogical simpliﬁcation (Archangeli, 1 984).

DRAFT370 Chapter 11. ComputationalPhonology
High Stem Low Stem
Lexical Surface Gloss Lexical Surface Gloss
Harmony dub+hin→dubhun “tangles” bok’+al→bok’ol “might eat”
NoHarmony xil+hin→xilhin “leads bythe hand” xat’+al→xat’al “might ﬁnd”
Thesecondrelevantrule, Lowering ,causeslonghighvowelstobecomelow; /u:/
becomes [o:]and/i:/becomes [e:], while the third rule, Shortening , shortens long
vowelsin closedsyllables:
Lowering Shortening
Pu:t’+it→Po:t’ut “steal, passive aorist” s:ap+hin→saphin
mi:k’+it→me:k’+it “swallow, passive aorist” sudu:k+hin→sudokhun
The three Yokuts rules must be ordered, just as the ix-insert ion and z-devoicing
rules had to be ordered. Harmony must be ordered before Lower ing because the /u:/
in the lexical form /Pu:t’+it/ causes the /i/to become [u]before it lowers in the
surface form [Po:t’ut]. Lowering must be ordered before Shortening because the /u:/
in/sudu:k+hin/ lowersto [o];ifitwasorderedaftershorteningitwouldappearonthe
surfaceas [u].

The Yokuts data can be modeled either as a cascade of three rul es in series, or in
thetwo-levelformalismasthreerulesinparallel;Fig.11. 6showsthetwoarchitectures
(Goldsmith, 1993; Lakoff, 1993; Karttunen, 1998). Just as i n the two-level examples
presented earlier, the rules work by referring sometimes to the lexical context, some-
timestothesurfacecontext;writingthe rulesisleftasExe rcise4forthe reader.
?+i
Roundingw
Loweringw
Shorteningrounding shortening?+i
lowering

?+i
Roundingw
Loweringw
lowering
Figure11.6 Combining the rounding, lowering, and shortening rules for YawelmaniYokuts.
11.2.2 TemplaticMorphology
Finite-state models of phonology and morphology have also b een proposed for the
templatic(non-concatenative)morphology(discussedonp age52)commoninSemitic
languageslikeArabic,Hebrew,andSyriac.
Thereareanumberofcomputationalﬁnite-stateimplementa tionsofnon-concatenative
morphology. ManyofthemdrawontheCV approachofMcCarthy( 1981),inwhicha
word like /katab/ is representedby three separatemorpheme s;arootmorpheme con-

sisting of consonants ( ktb), avocalic morpheme consisting of vowels ( a), and a CV
patternmorpheme(sometimescalleda binyanoraCVskeleton )(CVCVC).McCarthy
representedthismorphemesonthreeseparatemorphologica ltiers(Goldsmith,1976). tiers

DRAFTSection11.3. ComputationalOptimalityTheory 371
An inﬂuential model by Kay (1987), for example, uses separat e tapes for each of
McCarthy’s tiers. A high-level intuition of Kay’s model is s hown in Fig. 11.7, which
showshisspecialtransducerthatreadsfourtapesinsteado ftwo.
ax yaz { z
x y z
| } } | } | }
~ {
lexical tapeconsonantal root tapevocalic morpheme tape
binyan tape
Figure11.7 A ﬁnite-state model of templatic (“non-concatenative”) mo rphology. Adapted
from Kay(1987) and Sproat(1993).
Thecomplicationwithsuchamulti-tapemodelisdesigninga machinewhichaligns
the various strings on the tapes in the correct way; Kay propo sedthat the binyan tape
couldact asa sort ofguideforalignment. Kay’sintuitionha sledto a numberof more
fullyworkedoutﬁnite-state modelsofSemitic morphology; see theendofthechapter
fordetailsofthesemodels,aswellasalternativesbasedon newﬁnite-stateoperations.
11.3 ComputationalOptimalityTheory
In a traditional phonological derivation, we are given an un derlying lexical form and

a surface form. The phonological system then consists of a se quence of rules which
map the underlying form to the surface form. Optimality Theory (OT) (Prince and OptimalityTheory
OTSmolensky,1993)offersan alternativeway of viewing phono logicalderivation,based
onthemetaphorofﬁlteringratherthantransforming. AnOTm odelincludestwofunc-
tions (GEN and EVAL) and a set of ranked violable constraints (CON). Given an un-
derlying form, the GEN function produces all imaginable sur face forms, even those
whichcouldn’tpossiblybealegalsurfaceformfortheinput . TheEVALfunctionthen

applieseachconstraintin CON to these surfaceformsin orde rof constraintrank. The
surfaceformwhichbest meetsthe constraintsischosen.
Let’s brieﬂy introduceOT, using some Yawlemani data, and th en turn to the com-
putational ramiﬁcations.6In addition to the interesting vowel harmony phenomena
discussed above, Yawelmani has phonotactic constraints th at rule out sequences of
consonants; three consonants in a row (CCC) are not allowed t o occur in a surface
word. Sometimes,however,awordcontainstwoconsecutivem orphemessuchthatthe

6The following explication of OT via the Yawelmani example dr aws heavily from Archangeli (1997) and
alecture by Jennifer Cole at the1999 LSALinguistic Institu te.

DRAFT372 Chapter 11. ComputationalPhonology
ﬁrst one endsin two consonantsandthe secondonestarts with oneconsonant(orvice
versa). What does the languagedo to solve this problem? It tu rnsout that Yawelmani
eitherdeletesoneoftheconsonantsorinsertsa vowelinbet ween.
IfastemendsinaC,anditssufﬁxstartswithCC,theﬁrstCoft hesufﬁxisdeleted
(“+”heremeansa morphemeboundary):
C-deletion: C→ǫ/C+ C (11.6)
Forexample,simplifyingsomewhat,the CCVC “passive conse quentadjunctive”mor-
pheme hne:ldrops the initial C if the previous morpheme ends in a consona nt. Thus
after diyel“guard”, we would get the form diyel-ne:l-aw , “guard - passive consequent
adjunctive- locative”.
If a stem ends in CC and the sufﬁx starts with C, the language in stead inserts a
voweltobreakuptheﬁrst two consonants:
V-insertion: ǫ→V/C C+C (11.7)
Forexamplein iisinsertedintotheroot Pilk-“sing”whenitisfollowedbytheC-initial
sufﬁx -hin, “past”, producing Pilik-hin, “sang”, but not when followed by a V-initial
sufﬁxlike -en, “future”in Pilken“willsing”.

Kisseberth (1970) proposed that these two rules have the sam e function: avoiding
three consonants in a row. Let’s restate this in terms of syll able structure. It happens
thatYawelmanisyllablescanonlybeoftheformCVCorCV; com plexonsetsorcom-
plexcodasi.e.,withmultipleconsonants,aren’tallowed. SinceCVCCsyllablesaren’t
allowed on the surface, CVCC roots must be resyllabiﬁed when they appear on the resyllabiﬁed
surface. From the point of view of syllabiﬁcation, then, the se insertions and deletions

allhappensoastoallowYawelmaniwordstobeproperlysylla biﬁed. Here’sexamples
ofresyllabiﬁcationswithnochange,withaninsertion,and witha deletion:
Underlying Surface
Morphemes Syllabiﬁcation Gloss
Pilk-en Pil.ken “willsing”
Pilk-hin Pi.lik.hin “sang”
diyel-hnil-aw di.yel.ne:.law “guard–passivecons. adjunctive– locative”
TheintuitionofOptimalityTheoryistotryto directlyrepr esentthesekindofcon-
straintsonsyllablestructuredirectly,ratherthanusing idiosyncraticinsertionanddele-

tion rules. One such constraint, *C OMPLEX, says “No complex onsets or codas”.
Anotherclassofconstraintsrequiresthesurfaceformtobe identicalto(faithfulto)the
underlyingform. ThusF AITHV says“Don’tdeleteorinsertvowels”and F AITHCsays
“Don’tdeleteorinsertconsonants”. Givenanunderlyingfo rm,theGENfunctionpro-
ducesallpossiblesurfaceforms(i.e.,everypossibleinse rtionanddeletionofsegments
with every possible syllabiﬁcation) and they are ranked by t he EVAL function using

these(violable)constraints. Theideaisthatwhileingene ralinsertionanddeletionare
dispreferred, in some languages and situations they are pre ferred over violating other
constraints,suchasthoseofsyllablestructure. Fig.11.8 showsthearchitecture.
The EVAL function works by applying each constraint in ranke d order to each
candidate. Starting with the highest-ranked constraints, if one candidate either does

DRAFTSection11.3. ComputationalOptimalityTheory 373
/?ilk−hin/
[?i.lik.hin]?ilk.hin ?i.lik.hin?il.khin ?il.hin ?ak.pidGEN
EVAL(*COMPLEX, FAITHC, FAITHV)
Figure11.8 The architecture of a derivation inOptimalityTheory (afte rArchangeli (1997)).
not violate no constraints or violates less of them than all t he other candidates, that
candidateisdeclaredoptimal. Iftwocandidatestie(havet hesamehighestrankedvio-
lation),thenthenext-highestrankedviolationisconside red. Thisevaluationisusually
shownona tableau(pluraltableaux). Thetopleft-handcellshowstheinput,thecon- tableau
straints are listed in order of rank across the top row, and th e possible outputs along
the left-most column.7If a form violates a constraint, the relevant cell contains *; a *
*!indicates the fatal violation which causes a candidate to be eliminated. Cells for *!
constraintswhichareirrelevant(sincea higher-levelcon straintisalreadyviolated)are
shaded.
/Pilk-hin/*COMPLEX FAITHCFAITHV
Pilk.hin *!
Pil.khin *!
Pil.hin *!
☞ Pi.lik.hin *
Pak.pid *!

shaded.
Pil.hin *!
Pak.pid *!
OneappealofOptimalityTheoreticderivationsisthatthec onstraintsarepresumed
tobecross-linguisticgeneralizations. Thatisalllangua gesarepresumedtohavesome
version of faithfulness, some preference for simple syllab les, and so on. Languages
differinhowtheyranktheconstraints;thusEnglish,presu mably,ranksF AITHChigher
than *C OMPLEX. (Howdoweknowthis?)
11.3.1 Finite-State Transducer Models ofOptimalityTheor y
Nowthatwe’vesketchedthelinguisticmotivationsforOpti malityTheory,let’sturnto
the computational implications. We’ll explore two: implem entation of OT via ﬁnite-

state models,andstochasticversionsofOT.
7Although there are an inﬁnite number of candidates, it is tra ditional to show only the ones which are
‘close’; in the tableau below we have shown the output Pak.pidjust to make it clear that even very different
surface formsare to be included.

DRAFT374 Chapter 11. ComputationalPhonology
Cana derivationin OptimalityTheorybe implementedby ﬁnit e-statetransducers?
Frank and Satta (1998), following the foundational work of E llison (1994), showed
that (1) if GEN is a regular relation (for example assuming th e input doesn’t contain
context-free trees of some sort), and (2) if the number of all owed violations of any
constraint has some ﬁnite bound, then an OT derivation can be computed by ﬁnite-
state means. This second constraint is relevant because of a property of OT that we
haven’t mentioned: if two candidates violate exactly the sa me number of constraints,
the winning candidate is the one which has the smallest numbe r of violations of the
relevantconstraint.
One way to implement OT as a ﬁnite-state system was worked out by Karttunen
(1998), following the above-mentioned work and that of Hamm ond (1997). In Kart-
tunen’s model, GEN is implemented as a ﬁnite-state transduc er which is given an un-

derlying form and producesa set of candidate forms. For exam ple for the syllabiﬁca-
tion exampleabove,GEN wouldgenerateall stringsthat are v ariantsof the inputwith
consonantdeletionsorvowelinsertions,andtheirsyllabi ﬁcations.
Each constraint is implemented as a ﬁlter transducer that le ts pass only strings
which meet the constraint. For legal strings, the transduce r thus acts as the iden-
tity mapping. For example, *C OMPLEX would be implemented via a transducer that
mapped any input string to itself, unless the input string ha d two consonants in the

onsetorcoda,in whichcase it wouldbemappedtonull.
Theconstraintscanthenbeplacedinacascade,inwhichhigh er-rankedconstraints
aresimplyrunﬁrst, assuggestedin Fig.11.9.
GEN
o
*COMPLEX
o
FAITHC
o
FAITHV
Figure11.9 Version#1(“mercilesscascade”) ofKarttunen’s ﬁnite-sta tecascade implementa-
tionof OT.
There is one crucial ﬂaw with the cascade model in Fig. 11.9. R ecall that the
constraints-transducersﬁlteroutanycandidatewhichvio latesaconstraint. Butinmany
derivations,includingtheproperderivationof Pi.lik.hin,eventheoptimalformstillvi-

GEN
o
*COMPLEX
o
FAITHC
o
FAITHV
tionof OT.
olates a constraint. The cascade in Fig. 11.8 would incorrec tly ﬁlter it out, leaving
no surface form at all! Frank and Satta (1998) and Hammond (19 97) both point out
that it is essential to only enforce a constraint if it does no t reduce the candidate set
to zero. Karttunen (1998) formalizes this intuition with th elenient composition op-lenientcomposition
erator. Lenient composition is a combinationof regularcom positionand an operation
calledpriorityunion . Thebasicideaisthatifanycandidatesmeettheconstraint these

candidateswill be passed throughthe ﬁlter as usual. If no ou tputmeetsthe constraint,
lenientcompositionretains allofthecandidates. Fig.11.10showsthegeneralidea;the

DRAFTSection11.3. ComputationalOptimalityTheory 375
interestedreadershouldsee Karttunen(1998)forthedetai ls.
GEN
o

*COMPLEX
oL
FAITHC
oL
FAITHV/?ilk-hin/
GEN
?ilk.hin ?il.khin ?il.hin ?ak.pid ?i.lik.hin
*COMPLEX
?il.hin  ?ak.pid  ?i.lik.hin
FAITHC
?i.lik.hin
FAITHV
[?i.lik.hin]
Figure11.10 Version #2 (“lenient cascade”) of Karttunen’s ﬁnite-state cascade implementa-
tion of OT, showing a visualization of the candidate populat ions that would be passed through
each FSTconstraint.
11.3.2 StochasticModels ofOptimalityTheory
Classic OT was not designed to handle variation of the kind we saw in Sec. 7.3, since
it assigns a single most-harmonic output for each input. Dea ling with variation re-
quiresamoredynamicconceptofconstraintranking. Wement ionedinthatsectionthe
variationist model in sociolinguistics, in which logistic regression is used to combine
phonetic,contextual,andsocialfactorstopredictaproba bilityofa particularphonetic

variant. Part of this variationist intuition can be absorbe d into an Optimality Theory
frameworkthroughprobabilisticaugmentations.
Onesuchaugmentationis StochasticOT (BoersmaandHayes,2001). InStochas- StochasticOT
ticOT,insteadoftheconstraintsbeingrank-ordered,each constraintisassociatedwith
a valueona continuousscale. The continuousscale offerson ethinga rankingcannot:
therelativeimportanceorweightoftwoconstraintscanbep roportionaltothedistance
betweenthem. Fig.11.11showsa sketchofsucha continuouss cale.

Figure11.11 The Boersma andHayes (2001) continuous scale inStochastic OT.
Howcanthedistancebetweenconstraintsplaya roleinevalu ation? StochasticOT
makes a furtherassumption about the valuesof constraints. Instead of each constraint
having a ﬁxed value as in Fig. 11.11, it has a Gaussian distrib ution of values centered
onaﬁxedvalue,asinFig.11.12. Atevaluationtime,avaluef ortheconstraintisdrawn
(aselectionpoint )withaprobabilitydeﬁnedbythemeanandvarianceoftheGau ssian
associatedwitheachconstraint.

DRAFT376 Chapter 11. ComputationalPhonology
Figure11.12 ThreeconstraintsinStochasticOTwhicharestrictlyranke d;thusnon-stochastic
OTis a special case of Stochastic OT.
If the distribution for two constraints is far enough apart, as shown in Fig. 11.12
there will be little or no probability of the lower ranked con straint outranking the
higher-rankedone. ThusStochasticOTincludesnon-stocha sticOTasa specialcase.
Figure11.13 Three constraints inStochasticOT inwhich C3willsometimes outrank C2.
The interesting cases arise when two constraints in Stochas tic OT overlap in their
distribution,whenthereissomeprobabilitythatalower-r ankedconstraintwilloverride
a higher-ranked constraint. In Fig. 11.13, for example, con straintC2will generally
outrankC3butoccasionallyoutrank C2. ThisallowsStochasticOTtomodelvariation,
sinceforthesameunderlyingformdifferingselectionpoin tscancausedifferentsurface
variantstobemosthighlyranked.
In addition to the advantage of modeling variation, Stochas tic OT differs from

non-stochastic OT in having a stochastic learning theory, w hich we will return to in
Sec.11.5.3.
We can see stochastic OT itself as a special case of the genera l linear models of
Ch.6.
11.4 Syllabiﬁcation
Syllabiﬁcation , the task of segmenting a sequence of phones into syllables, is impor- Syllabiﬁcation
tant in a varietyof speech applications. In speech synthesi s, syllables are importantin
predictingprosodicfactorslike accent; the realizationo f a phoneis also dependenton
itspositioninthesyllable(onset[l]ispronounceddiffer entlythancoda[l]). Inspeech

Ch.6.
recognitionsyllabiﬁcationhasbeenusedto buildrecogniz erswhichrepresentpronun-
ciationsintermsofsyllablesratherthanphones. Syllabiﬁ cationcanhelpﬁnderrorsin

DRAFTSection11.4. Syllabiﬁcation 377
pronunciationdictionaries,by ﬁndingwordsthatcan’tbe s yllabiﬁed,andcan helpan-
notatecorporawith syllableboundariesforcorpuslinguis ticsresearch. Syllabiﬁcation
also playsanimportantroleintheoreticalgenerativephon ology.
One reason syllabiﬁcation is a difﬁcult computationaltask is that there is no com-
pletelyagreed-upondeﬁnitionofsyllableboundaries. Dif ferenton-linesyllabiﬁeddic-
tionaries(suchasthe CMU andtheCELEX lexicons)sometimes choosedifferentsyl-
labiﬁcations. Indeed, as Ladefoged (1993) points out, some times it isn’t even clear
how many syllables a word has; some words ( meal, teal, seal, hire, ﬁre, hour ) can be
viewedeitherashavingonesyllableortwo.
Like much work in speech and language processing, syllabiﬁe rs can be based on
hand-writtenrules,oronmachinelearningfromhand-label edtrainingsets. Whatkinds
of knowledge can we use in designing either kind of syllabiﬁe r? One possible con-

straint is the Maximum Onset principle,which says that when a series of consonants Maximum Onset
occur word-medially before a vowel (VCCV), as many as possib le (given the other
constraintsof the language)should be syllabiﬁed into the o nset of the second syllable
ratherthanthecodaoftheﬁrstsyllable. ThustheMaximumOn setprinciplefavorsthe
syllabiﬁcationV.CCV overthe syllabiﬁcationsVC.CV orVCC .V.
Another principle is to use the sonority of a sound, which is a measure of how sonority
perceptually salient, loud or vowel-like it is. There are va rious attempts to deﬁne a

sonorityhierarchy ; in general,all thingsbeingequal, vowelsare moresonorou sthan sonority hierarchy
glides (w, y), which are more sonorous than liquids (l, r), fo llowed by nasals (n, m,
ng), fricatives (z, s, sh, zh, v, f th, dh), and stops. The sono rity constraint on syllable
structure says that the nucleus of the syllable must be the mo st sonorous phone in a
sequence(the sonoritypeak ), andthat sonoritydecreasesmonotonicallyout fromthe
nucleus (toward the coda and toward the onset). Thus in a syll able C1C2VC3C4, the

nucleusVwillbethemostsonorouselement,consonant C2willbemoresonorousthan
C1andconsonant C3will bemoresonorantthanconsonant C4.
GoldwaterandJohnson(2005)implementasimplerule-based language-independent
classiﬁer based only on maximum onset and sonority sequenci ng. Given a cluster of
consonants between two syllable nuclei, sonority constrai ns the syllable boundary to
be either just before or just after the consonant with the low est sonority. Combining
sonoritywith maximumonset,their parserpredictsa syllab leboundaryjust beforethe

consonant with the lowest sonority. They show that this simp le syllabiﬁer correctly
syllabiﬁes86-87%ofmultisyllabicwordsin EnglishandGer man.
While this error rate is not unreasonable, and there is furth er linguistic and some
psychological evidence that these principles play a role in syllable structure, both
Maximum Onset and sonority sequencing seem to have exceptio ns. For example in
the English syllable-initial clusters /sp st sk/ in words li kespell, the less sonorous

/p/ occurs between the more sonorous /s/ and the vowel, viola ting sonority sequenc-
ing (Blevins, 1995). Without some way to rule out onset clust ers that are disallowed
language-speciﬁcallylike/kn/inEnglish,thecombinatio nofsonoritysequencingplus
maximum onset incorrectly predicts the syllabiﬁcation of w ords like weakness to be
wea.kness ratherthan weak.ness . Furthermore,otherconstraintsseem tobe important,
includingwhether a syllable is stressed (stressed syllabl es tend to have more complex

codas),the presenceorabsence ofmorphologicalboundarie s,andeventhe spellingof
theword(TitoneandConnine,1997;Treimanetal.,2002).

DRAFT378 Chapter 11. ComputationalPhonology
Achieving higher performance thus requires the use of these sorts of language-
speciﬁc knowledge. The most commonly used rule-based sylla biﬁer is based on the
dissertation of Kahn (1976), available in an implementatio n by Fisher (1996). The
Kahn algorithm makes use of language-speciﬁc information i n the form of lists of al-
lowable English initial clusters, allowable English ﬁnal c lusters, and ’universallybad’
clusters. The algorithm takes strings of phones, together w ith other information like
word boundaries and stress if they are available, and assign s syllable boundaries be-
tweenthephones. Syllablesarebuiltupincrementallybase donthreerules,assketched
outinFig.11.14. Rule 1formsnucleiat eachsyllabicsegmen t,Rule 2aattachesonset
consonants to the nucleus, and Rule 2b attaches coda consona nts.8Rule 2a and 2b
make use of lists of legal onset consonantsequences(includ inge.g. [b], [b l], [b r], [b

y], [ch], [d], [d r], [d w], [d y], [dh], [f], [f l], [f r], [f y], [g], [g l], [g r], [g w], etc).
and legal coda clusters. There are a very large number of coda consonant clusters in
English;someofthe longer(4-consonant)clustersinclude :
ks ts l fths m f s t ndths nks t rkt s r pts
ks ths l kts m pf t nts t nkts rl dz r s ts
l ts t m ps t ntths nkths rm pth r ts t
The algorithm also takes a parameter indicating how fast or c asual the speech is;

the faster or more informal the speech, the more resyllabiﬁc ation happens, based on
furtherruleswehaven’tshown.
m i s i s i p i
S S S SC1...CnV−→C1...CiCi+1...CnV
S SVC1...Cn−→VC1...CjCj+1...Cn
S S
Rule 1: Form Nuclei:
link S with each [+syl-
labic] segmentRule 2a: Add Onsets: where
Ci+1...Cnis a permissible initial
cluster but C iCi+1...Cnis notRule 2b: Add Codas: where
C1...Cjisa permissible coda cluster
but C1...CjCj+1is not
Figure11.14 Firstthree syllabiﬁcationrules of Kahn (1976). Rule 2bmay not apply across wordboundaries.

m i s i s i p i
S S
Instead of hand-written rules, we can apply a machine learni ng approach, using
a hand-syllabiﬁed dictionary as a supervised training set. For example the CELEX
syllabiﬁed lexicon discussed in Sec. 7.5 is often used this w ay, selecting some words
as a training set, and reserving others as a dev-test and test set. Statistical classiﬁers
canbeusedtopredictsyllabiﬁcations,includingdecision trees(vandenBosch,1997),
weighted ﬁnite-state transducers(Kiraz and M¨ obius, 1998 ),and probabilisticcontext-

freegrammars(Seneffetal.,1996;M¨ uller,2002,2001;Gol dwaterandJohnson,2005).
ForexampletheKirazandM¨ obius(1998)algorithmisaweigh tedﬁnite-statetrans-
ducerwhichinsertsasyllableboundaryinasequenceofphon es(akintothemorpheme-
boundarieswe saw in Ch. 3). A weighted FST (Pereira et al., 1994) is a simple aug- weightedFST
mentationof the ﬁnite transducerin which each arc is associ ated with a probabilityas
well asa pairofsymbols. Theprobabilityindicateshowlike lythatpathistobetaken;

theprobabilityonall thearcsleavinga nodemustsumto 1.
The syllabiﬁcation automaton of Kiraz and M¨ obius (1998) is composed of three
separate weighted transducers, one for onsets, one for nucl ei, and one for codas, con-
8Notethat thefact that Rule 2a precedes Rule 2b can be seen as a n implementation of Maximum Onset.

DRAFTSection11.4. Syllabiﬁcation 379
catenated togetherinto an FST that inserts a syllable marke r after the end of the coda.
Kiraz and M¨ obius (1998) compute path weights from frequenc ies in the training set;
each path (for example the nucleus [iy]) of frequency fis assigned a weight of 1 /f.
Another way to convert frequencies to costs is to use log prob abilities. Fig. 11.15
showsasampleautomaton,simpliﬁedfromKirazandM¨ obius( 1998). We haveshown
theweightsonlyforsomeofthenuclei. Thearcsforeachposs ibleonset,nucleus,and
coda,aredrawnfromalanguage-dependentlistliketheoneu sedintheKahnalgorithm
above.
Figure11.15 Syllabiﬁer automaton, showing onset (o), coda (c), and nucl eus arcs. Costs on
eacharc shown only forsome sample nucleus arcs. The syllabl e boundary marker ‘-’is inserted
after everynon-ﬁnal syllable. eps stands for ǫ. Simpliﬁedfrom Kirazand M¨ obius (1998).
TheautomatonshowninFig.11.15canbeusedtomapfromaninp utsequencelike

above.
the phonetic representation of weakness [w iy k n eh s] into an output sequence that
includesthesyllabiﬁcationmarkerlike “-”: [wiy k -n ehs]. Iftherearemultiplepos-
sible legal syllabiﬁcations of a word, the Viterbi algorith m is used to choose the most
likely path through the FST, and hence the most probable segm entation. For exam-
ple,theGermanword Fenster,“window”,hasthreepossiblesyllabiﬁcations: [ fEns-t5]
<74>, [fEn-st5]<75>, and [ fEnst-5]<87>(with costs shown in angle brackets).

Their syllabiﬁer correctlychooses the lowest cost syllabi ﬁcation fEns-t5, based on the
frequencies of onsets and codas from the training set. Note t hat since morphological
boundaries also are important for syllabiﬁcation, the Kira z and M¨ obius (1998) syl-
labiﬁcation transducer can be placed after a morphological parsing transducer, so that
syllabiﬁcationcanbeinﬂuencedbymorphologicalstructur e.
Morerecentsyllabiﬁersbasedonprobabilisticcontext-fr eegrammars(PCFGs)can

modelmorecomplexhierarchicalprobabilisticdependenci esbetweensyllables(Seneff
et al., 1996; M¨ uller, 2002, 2001; Goldwater and Johnson, 20 05). Together with other
machine learning approaches like van den Bosch (1997), mode rn statistical syllabiﬁ-
cation approaches have a word accuracy of around 97–98% corr ect, and probabilistic
model of syllable structure have also been shown to predict h uman judgments of the
acceptabilityofnonsensewords(ColemanandPierrehumber t,1997).

Thereareanumberofotherdirectionsinsyllabiﬁcation. On eistheuseofunsuper-
vised machine learning algorithms(Ellison, 1992; M¨ uller et al., 2000; Goldwater and
Johnson, 2005) Another is the use of other cues for syllabiﬁc ation such as allophonic

DRAFT380 Chapter 11. ComputationalPhonology
detailsfromanarrowphonetictranscription(Church,1983 ).
11.5 LearningPhonology&Morphology
Machinelearningofphonologicalstructuresisanactivere searchareaincomputational
phonology above and beyond the induction of syllable struct ure discussed in the pre-
vious section. Supervised learning work is based on a traini ng set that is explicitly
labeledforthephonological(ormorphological)structure to beinduced. Unsupervised
workattemptstoinducephonologicalormorphologicalstru cturewithoutlabeledtrain-
ing data. Let’s look at three representativeareas of learni ng: learning of phonological
rules,learningofmorphologicalrules,andlearningofOTc onstraintrankings
11.5.1 Learning PhonologicalRules
In this section we brieﬂy summarize some early literature in learning phonological
rules, generally couched either in terms of ﬁnite-state mod els of two-level phonology
orclassic Chomsky-Hallerules.
Johnson (1984) gives one of the ﬁrst computational algorith ms for phonological

ruleinduction. Hisalgorithmworksforrulesoftheform
(11.8)a→b/C
whereCis the feature matrix of the segments around a. Johnson’s algorithm sets
up a system of constraint equations which Cmust satisfy, by considering both the
positive contexts, i.e., all the contexts Ciin which a boccurs on the surface, as well
as all the negative contexts Cjin which an aoccurs on the surface. Touretzky et al.
(1990)extendedJohnsonsworkinvariousways,includingde alingwithepenthesisand
deletionrules.

The algorithm of Gildea and Jurafsky (1996) was designed to i nduce transducers
representing two-level rules of the type we have discussed e arlier. Gildea and Juraf-
sky’s supervised algorithm was trained on pairs of underlyi ngand surface forms. For
example, they attempted to learn the rule of English ﬂapping , (focusing only on the
phoneticcontextandignoringsocialandotherfactors). Th etrainingset thusconsisted
of underlying/surface pairs, either with an underlying /t/ and surface ﬂap [dx], or an

underlying/t/ andsurface[t],asfollows:
ﬂapping non-ﬂapping
butter/b aht axr/→[bah dxaxr] stop/st aap/→[st aap]
meter/m iyt axr/→[miydxaxr] cat/kae t/→[kae t]
The algorithm was based on OSTIA (Oncina et al., 1993), a gene ral learning al-
gorithm for the subsequential transducers deﬁned on page 59. Gildea and Jurafsky
showedthatbyitself,theOSTIAalgorithmwastoogeneralto learnphonologicaltrans-
ducers,evengivenalargecorpusofunderlying-form/surfa ce-formpairs. Forexample,

given25,000underlying/surfacepairsliketheexamplesab ove,thealgorithmendedup
with thehugeandincorrectautomatonin Fig.11.16(a). Gild eaandJurafskythenaug-
mentedthedomain-independentOSTIAsystemwithlearningb iaseswhicharespeciﬁc

DRAFTSection11.5. LearningPhonology&Morphology 381
to natural language phonology. For example they added a Faithfulness bias that un-
derlyingsegmentstendtoberealizedsimilarlyonthesurfa ce(i.e. thatall thingsbeing
equal, an underlying/p/ was likely to emergeas a surface [p] ). They did this by start-
ingOSTIAwiththeunderlyingandsurfacestringsalignedus ingLevenshteindistance.
Theyalso addedknowledgeaboutphoneticfeatures(vowelve rsusconsonant,reduced
versusnon-reducedvowel,etc). Together,addingthesebia sesenabledOSTIAtolearn
the automaton in Fig. 11.16(b), as well as correct automaton s for other phonological
ruleslike Germanconsonantdevoicing.
0 1 2 3 4 5 6 7 8 9 10 11
12 13 14 15 16 17 18 19 20 21 22 23
24 25 26 27 28 29 30 31 32 33 34 35
36 37 38 39 40 41 42 43 44 45 46 47
48 49 50 51 52 53 54 55 56 57 58 59
60 61 62 63 64 65 66 67 68 69 70 71
72 73 74 75 76 77 78 79 80 81 82 83
84 85 86 87 88 89 90 91 92 93 94 95
96 97 98 99 100 101 102 103 104 105 106 107
108 109 110 111 112 113 114 115 116 117 118 119

120 121 122 123 124 125 126 127 128 129 130 131
132 133 134 135 136 137 138 139 140
0 1
3tr
 ∋V V/
/
V : t V
C: t C
r : t r
V : dx V
# : t/rV
C
(a) (b)
Figure11.16 Induction of a ﬂapping rule transducer (after Gildea and Jur afsky (1996)). The
transducer in (a) is the initial attempt at learning. The tra nsducer in(b) is the correct transducer
induced after a faithfulness bias.
This phonologicallearning experimentillustrates that su ccessful learning requires

0 1
3tr
 ∋V V/
/
V : t V
C: t C
r : t r
V : dx V
# : t/rV
C
(a) (b)
twocomponents: amodelwhichﬁtssomeempiricaldataandsom epriorknowledgeor
biasesaboutthestructureofthemodel.
Recentworkonlearninghasfocusedeitheronmorphological learning,oronrank-
ing of OT constraintsratherthan on the inductionof rules an dconstraints, and will be
discussedinthe nexttwosections.
11.5.2 Learning Morphology
We discussed in Ch. 3 the use of ﬁnite-state transducers for m orphological parsing.
In general, these morphologicalparsers are built by hand an d have relatively high ac-

curacy, although there has also been some work on supervised machine learning of
morphologicalparsers (van den Bosch, 1997). Recent work, h owever,has focused on
unsupervised ways to automatically bootstrap morphologic al structure. The unsuper-
vised (or weakly supervised) learning problem has practica l applications, since there
are many languages for which a hand-built morphological par ser, or a morphological
segmentedtrainingcorpus,doesnotyetexist. Inaddition, thelearnabilityoflinguistic

structure is a much-discussedscientiﬁc topic in linguisti cs; unsupervisedmorphologi-
cal learningmayhelpusunderstandwhat makeslanguagelear ningpossible.

DRAFT382 Chapter 11. ComputationalPhonology
Approachesto unsupervised morphologyinduction have empl oyed a wide variety
of heuristics or cues to a proper morphological parse. Early approaches were all es-
sentiallysegmentation-based;givena corpusofwordsthey attemptedto segmenteach
word into a stem and an afﬁx using various unsupervised heuri stics. For example the
earliest work hypothesized morpheme boundaries at the poin t in a word where there
is large uncertainty about the following letters (Harris, 1 954, 1988; Hafer and Weiss,
1974). For example, Fig. 11.17 shows a trie9which stores the words car,care,cars, trie
cares,cared, etc. Note that there are certainnodesin the tree in Fig. 11. 17that havea
wide branching factor (after carand after care). If we think of the task of predicting
thenextlettergivingthepathinthetriesofar,wecansayth atthesepointshaveahigh
conditional entropy; there are many possible continuation s.10While this is a useful

heuristic,it is notsufﬁcient;in thisexamplewe wouldneed a way torule outthe mor-
phemecaraswellas carebeingpartoftheword careful;thisrequiresacomplexsetof
thresholds.
Figure11.17 Exampleofalettertrie. AHarrisstylealgorithmwouldinse rtmorphemebound-
aries after carandcare. AfterSchone and Jurafsky(2000).
Anotherclass of segmentation-basedapproachesto morphol ogyinductionfocuses
on globally optimizinga single criterion for the whole gram mar,the criterion of min-

imum description length , orMDL. The MDL principle is widely used in languageminimumdescriptionlength
MDLlearning,and we will see it againin grammarinductionin Ch. 14. The idea is that we
are trying to learn the optimal probabilistic model of some d ata. Given any proposed
model, we can assign a likelihoodto the entire data set. We ca n also use the proposed
modeltoassignacompressedlengthtothisdata(withprobab ilisticmodelswecanuse
the intuitionthat the compressedlength of the data is relat ed to the entropy,which we

can estimate from the log probability). We can also assign a l ength to the proposed
model itself. The MDL principle says to choose the model for w hich the sum of the
data length and the model length is the smallest. The princip leis often viewed froma
Bayesianperspective;Ifwe areattemptingtolearnthe best modelˆMoutofallmodels
Mfor some data Dwhich has the maximum a posteriori probability P(M|D), we can
9Atrieis a tree structure used for storing strings, in which a strin g is represented as a path from the root

to a leaf. Each non-terminal node in the tree thus stores a pre ﬁx of a string; every common preﬁx is thus
represented by anode. Theword triecomes from retrievaland is pronounced either [tr iy] or[t ray].
10Interestingly, this idea of placing boundaries at regions o f low predictability has been shown to be used
by infants for word segmentation (Saffran etal., 1996b).

DRAFTSection11.5. LearningPhonology&Morphology 383
useBayesRule toexpressthebestmodel ˆMas:
ˆM=argmax MP(M|D) =argmax MP(D|M)P(M)
P(D)=argmax MP(D|M)P(M) (11.9)
Thusthe best modelis the one which maximizestwo terms: the l ikelihoodof the data
P(D|M)andthepriorofthemodel P(M). TheMDLprinciplecanbeviewedassaying
thatthepriortermonthe modelshouldberelatedtothelengt hofthemodel.
MDL approaches to segmentation induction were ﬁrst propose d by de Marcken
(1996) and Brent (1999), as well as Kazakov (1997); let’s sum marize from a more
recent instantiation by Goldsmith (2001). The MDL intuitio n can be seen from the
schematicexampleinFig.11.18inspiredbyGoldsmith.


cookedcookscooking
playedplaysplaying
boiledboilsboiling



cook
play
boil



ed
s
ing


(a)Wordlistwithnostructure (b)Wordlistwithmorphologi cal structure
Totalletter count: 54 Totallettercount: 18 letters
Figure11.18 Naive version of MDL, showing the reduction in the descripti on length of a






cook
play
boil



ed
s
ing


lexicon withmorphological structure;adapted fromGoldsm ith (2001).
As we see in Fig. 11.18, using morphological structure makes it possible to rep-
resent a lexicon with far fewer letters. Of course this examp le doesn’t represent the
true complexity of morphological representations, since i n reality not every word is
combinablewith every afﬁx. One way to represent slightly mo re complexityis to use
signatures . Asignatureisalistofsufﬁxesthatcanappearwithapartic ularstem. Here signatures
aresomesamplesignaturesfromGoldsmith(2001):
Signature Example

NULL.ed.ing.s remainremained remainingremains
NULL.s cow cows
e.ed.es.ing notice noticed notices noticing
The Goldsmith (2001) version of MDL considers all possible s egmentations of
every word into a stem and a sufﬁx. It then chooses the set of se gmentations for the
wholecorpusthatjointlyminimizethecompressedlengthof thecorpusandthelength
ofthemodel. Thelengthofthemodelisthesumofthelengthso ftheafﬁxes,thestems,
andthe signatures. The lengthof the corpusis computedby us ingthe model to assign

aprobabilitytothecorpusandusingthisprobablytocomput ethecross-entropyofthe
corpusgiventhemodel.
While approaches based solely on stem and afﬁx statistics li ke MDL have been
quite successful in morphological learning, they do have a n umber of limitations.
For example Schone and Jurafsky (2000, 2001) noted in an erro r analysis that MDL
sometimes segments valid afﬁxes inappropriately (such as s egmenting the word ally
toall+y), or fails to segment valid but non-productive afﬁxes (miss ing the relation-

ship between dirtanddirty). They argued that such problems stemmed from a lack

DRAFT384 Chapter 11. ComputationalPhonology
ofsemanticorsyntacticknowledge,andshowedhowtouserel ativelysimplesemantic
featurestoaddressthem. TheSchoneandJurafsky(2000)alg orithmusesatrietocome
upwith“pairsofpotentialmorphologicalvariants”,(PPMV s)wordswhichdifferonly
in potential afﬁxes. For each pair, they compute the semanti c similarity between the
words, using the Latent Semantic Analysis (LSA) algorithmo f Ch. 23. LSA is an un-
supervisedmodelofwordsimilaritywhichisinduceddirect lyfromthedistributionsof
wordincontext. SchoneandJurafsky(2000)showedthatusin gthesemanticsimilarity
alone was at least as good a predictor of morphological struc ture as MDL. The table
belowshowstheLSA-basedsimilaritybetweenPPMVs; inthis examplethesimilarity
ishighonlyforwordsthat aremorphologicallyrelated.
PPMV Score PPMV Score PPMV Score PPMV Score
ally/allies 6.5 dirty/dirt 2.4 car/cares -0.14 car/cared -.096
car/cars 5.6 rating/rate 0.97 car/caring -0.71 ally/all -1.3

Schone and Jurafsky (2001) extended the algorithm to learn p reﬁxes and circum-
ﬁxes, and incorporated other useful features, including sy ntactic and other effects of
neighboring word context (Jacquemin, 1997), and the Levens htein distance between
thePPMVs(Gaussier,1999).
Thealgorithmswe havementionedso farhavefocusedonthe pr oblemoflearning
regularmorphology. YarowskyandWicentowski(2000)focus edonthemorecomplex
problemof learningirregularmorphology. Their idea was to probabilisticallyalign an

inﬂectedform(suchasEnglish tookorSpanish juegan)witheachpotentialstem(such
as English takeor Spanish jugar). The result of their alignment-based algorithm was
a inﬂection-rootmapping,with bothanoptionalstem change anda sufﬁx,asshownin
thefollowingtable:
rootinﬂection stem change sufﬁx rootinﬂection stemchange sufﬁx
taketook ake→ook+ǫ jugarjuega gar→eg+a
taketaking e→ǫ +ing jugarjugamos ar→ǫ +amos
skipskipped ǫ→p +ed tenertienen ener→ien+en

The Yarowsky and Wicentowski (2000) algorithm requires som ewhat more infor-
mation than the algorithms for inducing regular morphology . In particular it assumes
knowledge of the regular inﬂectional afﬁxes of the language and a list of open class
stems; both are things that might be induced by the MDL or othe r algorithms men-
tioned above. Given an inﬂected form, the Yarowsky and Wicen towski (2000) algo-
rithm usesvariousknowledgesourcesto weight the potentia lstem, includingthe rela-

tivefrequencyoftheinﬂectedformandpotentialstem,thes imilarityinlexicalcontext,
andtheLevenshteindistancebetweenthem.
11.5.3 Learning inOptimalityTheory
Let’s conclude with a brief sketch of work which addresses th e learning problem in
Optimality Theory. Most work on OT learning has assumed that the constraints are
alreadygiven,andthetaskistolearntheranking. Twoalgor ithmsforlearningrankings

DRAFTSection11.5. LearningPhonology&Morphology 385
have been worked out in some detail; the constraint demotion algorithm of Tesar
and Smolensky (2000)and the Gradual Learning Algorithm of Boersma and Hayes
(2001).
TheConstraintDemotion algorithmmakestwoassumptions: thatweknowalltheConstraintDemotion
possible OT constraints of the language, and that each surfa ce form is annotated with
its complete parse and underlyingform. The intuition of the algorithm is that each of
thesesurfaceobservationsgivesusimplicitevidenceabou ttheconstraintranking.
Giventheunderlyingform,wecanusetheGENalgorithmtoimp licitlyformtheset
of competitors. Now we can constructa set of pairsconsistin g ofthe correctobserved
grammatical form and each competitor. The learner must ﬁnd a constraint ranking
that prefers the observed learning winnerover each (non-observed) competitor loser.
Becausethesetofconstraintsisgiven,wecanusethestanda rdOTparsingarchitecture
todetermineforeachwinnerorloserexactlywhichconstrai ntstheyviolate.

For example, consider the learning algorithm that has obser ved Candidate 1, but
whosecurrentconstraintrankingprefersCandidate2,asfo llows(thisexampleandthe
followingtablesaremodiﬁedfromBoersmaandHayes(2001)) :
/underlyingform/ C1C2C3C4C5C6C7C8
Candidate1 (learningobservation) *!**** *
☞Candidate2 (learner’soutput) *****
Givenasetofsuch winner/loser pairs,theConstraintDemotionalgorithmneedsto
demote each constraint that is violated by the winner Candid ate 2, until the observed
form(Candidate1)ispreferred. Thealgorithmﬁrstcancels anymarksduetoviolations

thatareidenticalbetweenthe twocandidates:
Candidate1 (learningobservation) ∗!∗∗∗∗∗
☞Candidate2 (learner’soutput) ∗∗∗∗∗
Theseconstraintsarepusheddowninthehierarchyuntilthe yaredominatedbythe
constraints violated by the loser. The algorithm divides co nstraints into strata, and
tries to ﬁnd a lower strata to move the constraints into. Here ’s shows a simpliﬁcation
ofthisintuition,as C1andC2getmovedbelow C8.
/underlyingform/ C3C4C5C6C7C8C1C2
☞Candidate1(learningobservation) * **

Candidate2(learner’soutput) *!*
TheGradualLearningAlgorithm (GLA)of(BoersmaandHayes,2001)isagen-Gradual Learning
Algorithm
eralization of Constraint Demotion that learns constraint rankings in Stochastic Opti-
mality Theory. Since OT is a special case of Stochastic OT, th e algorithmalso implic-
itlylearnsOTrankings. ItgeneralizesConstraintDemotio nbybeingabletolearnfrom
cases of free variation. Recall from Sec. 11.3 that in Stocha stic OT each constraint is
associated with a ranking value on a continuous scale. The ranking value is deﬁned

DRAFT386 Chapter 11. ComputationalPhonology
asthemeanoftheGaussiandistributionthatconstitutesth econstraint. Thegoalofthe
GLA is to assign a rankingvalueforeach constraint. The algo rithmis a simple exten-
siontotheConstraintDemotionalgorithm,andfollowsexac tlythesamestepsuntilthe
ﬁnal step. Inside of demoting constraints to a lower strata, the ranking value of each
constraintviolatedbythelearningobservation(Candidat e1)isdecreasedslightly,and
the ranking value of each constraint violated by the learner ’s output (Candidate 2) is
increasedslightly,asshownbelow:
Candidate1 (learningobservation) ∗!→∗→∗→
☞Candidate2 (learner’soutput) ←∗←∗
11.6 Summary
tationalphonology.
•Transducers can be used to model phonologicalrules just as they were used in
Ch. 3 to model spelling rules. Two-level morphology is a theory of morphol-
ogy/phonologywhichmodelsphonologicalrulesasﬁnite-st atewell-formedness

constraints onthemappingbetweenlexicalandsurfaceform.
•Optimalitytheory is a theoryof phonologicalwell-formedness;there are com-
putationalimplementations,andrelationshipstotransdu cers.
•Computationalmodels exist for syllabiﬁcation , insertingsyllable boundariesin
phonestrings.
•There are numerous algorithms for learning phonological an d morphological
rules,bothsupervisedandunsupervised.
Computational phonology is a fairly recent ﬁeld. The idea th at phonological rules

could be modeled as regular relations dates to Johnson (1972 ), who showed that any
phonologicalsystem that didn’t allow rules to apply to thei r own output (i.e., systems
thatdidnothaverecursiverules)couldbemodeledwithregu larrelations(orﬁnite-state
transducers). Virtually all phonologicalrulesthat had be enformulatedat the time had
thisproperty(exceptsomeruleswithintegral-valuedfeat ures,likeearlystressandtone
rules). Johnson’s insight unfortunatelydid not attract th e attention of the community,

andwasindependentlydiscoveredbyRonaldKaplanandMarti nKay;seeCh.3forthe
restofthehistoryoftwo-levelmorphology. Karttunen(199 3)givesatutorialintroduc-
tion to two-level morphologythat includesmore of the advan ceddetails thanwe were

DRAFTSection11.6. Summary 387
able to present here, and the deﬁnitive text on ﬁnite-state m orphology is Beesley and
Karttunen(2003). OtherFSA modelsofphonologyincludeBir dandEllison(1994).
Earlier computational ﬁnite-state models that deal with te mplatic morphology in
languageslikeArabicincludeKatajaandKoskenniemi(1988 ),Kornai(1991),Birdand
Ellison(1994),andBeesley(1996). ExtensionsoftheKay(1 987)modelincludeKiraz
(1997, 2000, 2001). Recent models based on extensions to the ﬁnite-state calculus
includeBeesley andKarttunen(2000).
OptimalitytheorywasdevelopedbyPrinceandSmolenskyand circulatedasatech-
nicalreport(PrinceandSmolensky,1993)untilitspublica tionmorethanadecadelater
(Prince and Smolensky, 2004). A selection from the extensiv e ﬁnite-state literature in
OTincludesEisner(1997,2000b,2002b),GerdemannandvanN oord(2000),andRig-
gle(2005).
Recent work on phonological learning has focused on some new areas. One is
learningphonotacticconstraints ontheallowableword-internalsequencesinthelan-

guage, includingprobabilistic(Coleman and Pierrehumber t,1997; Frisch et al., 2000;
Bailey and Hahn, 2001; Hayes and Wilson, 2007; Albright, 200 7) as well as non-
probabilistic phonotactic constraints (Hayes, 2004; Prin ce and Tesar, 2004; Tesar and
Prince, 2007). A related task is the learning of underlying forms and phonological
alternations given the observed surface forms and the set of constraints. Many of the
unsupervisedalgorithmsforlearningunderlyingformsare basedon a constraintsatis-

faction approach,in which sets of possible underlyingform sare proposedbyexamin-
ingalternatingsurfaceforms,andtheniterativelyruling outpossibleunderlyingforms
(Tesar and Prince, 2007; Alderete et al., 2005; Tesar, 2006a , 2006b). The recent un-
supervised Maximum Likelihood Learning of Lexicons and Gra mmars (MLG) model
of Jarosz (2006, 2008) learns underlying forms and constrai nt rankings given surface
formsinaprobabilisticversionofOTusingtheExpectation -Maximization(EM)algo-
rithmdescribedinCh.6.

Indeed, in addition to this probabilistic model of Jarosz (2 008), as well as the
Stochastic OT described earlier in the chapter, much recent work in computational
phonology has focused on models with weighted constraints, including Harmonic
Grammar andMaximumEntropyModels . Forexample HarmonicGrammar isanHarmonicGrammar
extensiontoOptimalityTheory(ormoreproperlyisthetheo rythatOptimalityTheory
originallygrewoutof)inwhichoptimalityforaformisdeﬁn edasmaximal harmony . harmony

Harmony is deﬁned by the sum of weighted constraints (Smolen sky and Legendre,
2006). InusingsumsofweightratherthanOT-stylerankings ,HarmonyTheoryresem-
blesthelog-linearmodelsofCh.6. Recentcomputationalwo rkincludetheapplication
toOTofMaximumEntropyModels(GoldwaterandJohnson,2003 )andtheHarmonic
GrammarrelatedmodelsofPateret al.(2007)andPater(2007 ).
Word segmentation is one of the earliest problems in computa tional linguistics,
andmodelsdatebacktoHarris(1954). Amongthemanymodernm odelsareBayesian

ones like Brent (1999) and Goldwater et al. (2006). The word s egmentation problem
isimportantalso in computationaldevelopmentalpsycholi nguistics;forrepresentative
recent work see Christiansen et al. (1998), Kuhl et al. (2003 ), Thiessen and Saffran
(2004) and Thiessen et al. (2005). Recent work on morphology induction includes
Baronietal. (2002),Clark(2002),andAlbrightandHayes(2 003).
Readerswithfurtherinterestinphonologyshouldconsultp honologytextbookslike

DRAFT388 Chapter 11. ComputationalPhonology
Odden(2005)andKager(2000).
Exercises
11.1Buildanautomatonforrule(11.3).
11.2One difference between one dialect of Canadian English and m ost dialects of
American English is called Canadian raising . Bromberger and Halle (1989) Canadianraising
notethat someCanadiandialectsofEnglishraise /aI/to[2I]and/aU/to[2U]in
stressed position before a voiceless consonant. A simpliﬁe d version of the rule
dealingonlywith /aI/canbestated as:
/aI/→[2I]/[
C
−voice]
(11.10)
This rule has an interesting interaction with the ﬂapping ru le. In some Cana-
dian dialects the word riderandwriterare pronounceddifferently: rideris pro-
nounced [raIRÄ]whilewriterispronounced [r2IRÄ].Writeatwo-levelruleandan
automatonforboththeraisingruleandtheﬂappingrulewhic hcorrectlymodels
thisdistinction. Youmaymakesimplifyingassumptionsasn eeded.
11.3Write the lexicalentryforthe pronunciationof theEnglish past tense (preterite)
sufﬁx-d, and the two level-rulesthat express the differencein its p ronunciation

dependingon the previouscontext. Don’t worryabout the spe lling rules. (Hint:
makesureyoucorrectlyhandlethepronunciationofthepast tensesofthewords
add,pat,bake,andbag.)
11.4Writetwo-levelrulesfortheYawelmaniYokutsphenomenaof Harmony,Short-
ening,and Loweringintroducedon page 369. Make sure yourru les are capable
ofrunninginparallel.

DRAFTSpeech and Language Processing: An Introduction to Speech R ecognition, Computational Linguistics
and Natural Language Processing: Second Edition, Daniel Ju rafsky & James H. Martin. Copyright c⃝
2007, All rights reserved. Draft of January 9, 2008. Do not ci te.
Chapter12
Formal Grammars ofEnglish
Sentence
NP VP
the man Verb NP
took the book
The ﬁrst context-free grammar parse tree
(Chomsky,1956)
If onawinter’s night atraveler byItaloCalvino
Nuclear andRadiochemistry byGerhart Friedlander et al.
The FireNextTime by James Baldwin
A TadOverweight, butViolet Eyes toDieFor byG. B.Trudeau
Sometimes a Great Notion byKen Kesey
Dancer fromthe Dance by Andrew Holleran
Six books in English whose titles are not con-
stituents,fromPullum(1991,p. 195)
Thestudyofgrammarhasanancientpedigree;Panini’sgramm arofSanskritwaswrit-
ten over two thousand years ago, and is still referenced toda y in teaching Sanskrit.
By contrast, Geoff Pullum noted in a recent talk that “almost everything most edu-

cated AmericansbelieveaboutEnglishgrammaris wrong”. In thischapterwe makea
preliminary stab at addressing some of these gaps in our know ledge of grammar and
syntax, as well as introducing some of the formal mechanisms that are available for
capturingthisknowledge.
The word syntaxcomes from the Greek s´yntaxis, meaning “setting out together Syntax
or arrangement”, and refers to the way words are arranged tog ether. We have seen
various syntactic notions in previous chapters. The regula r languages introduced in

Ch. 2 offered a simple way to represent the ordering of string s of words, and Ch. 4
showedhowtocomputeprobabilitiesforthesewordsequence s. Ch.5showedthatpart-
of-speech categories could act as a kind of equivalence clas s for words. This chapter
and the followingones introducesophisticated notionsof s yntax and grammarthat go
wellbeyondthesesimplernotions. Inthischapter,weintro ducethreemainnewideas:
constituency ,grammaticalrelations ,andsubcategorizationanddependency .
The fundamental idea of constituency is that groups of words may behave as a

single unit or phrase, called a constituent. For example we w ill see that a group of
words called a noun phrase often acts as a unit; noun phrases include single words
likesheorMichaeland phrases like the house ,Russian Hill , anda well-weathered

DRAFT390 Chapter 12. FormalGrammarsofEnglish
three-storystructure . Thischapterwillintroducetheuseof context-freegrammars ,a
formalismthatwill allowustomodeltheseconstituencyfac ts.
Grammaticalrelations areaformalizationofideasfromtraditionalgrammarsuch
asSUBJECTS andOBJECTS, and other related notions. In the following sentence the
nounphrase SheistheSUBJECT andamammothbreakfast istheOBJECT:
(12.1)Sheate a mammothbreakfast.
Subcategorization anddependency relations refer to certain kinds of relations
between words and phrases. For example the verb wantcan be followed by an inﬁni-
tive,asin I wantto ﬂy toDetroit , or a nounphrase,as in I want aﬂightto Detroit . But
the verbﬁndcannot be followed by an inﬁnitive ( *I found to ﬂy to Dallas ). These are
calledfactsaboutthe subcategorization ofthe verb.
As we’ll see, none of the syntactic mechanismsthat we’ve dis cussed up until now
can easily capture such phenomena. They can be modeled much m ore naturally by

grammars that are based on context-free grammars. Context- free grammars are thus
the backbone of many formal models of the syntax of natural la nguage (and, for that
matter,ofcomputerlanguages). Assuchtheyareintegralto manycomputationalappli-
cations including grammar checking, semantic interpretat ion, dialogue understanding
and machine translation. They are powerful enough to expres s sophisticated relations
among the words in a sentence, yet computationallytractabl e enoughthat efﬁcient al-

gorithms exist for parsing sentences with them (as we will se e in Ch. 13). Later in
Ch. 14 we’ll show that adding probability to context-free gr ammars gives us a model
ofdisambiguation,andalso helpsmodelcertainaspectsofh umanparsing.
Inadditiontoanintroductiontothegrammarformalism,thi schapteralsoprovides
abriefoverviewofthegrammarofEnglish. Wehavechosenado mainwhichhasrela-
tively simple sentences, the Air Trafﬁc InformationSystem (ATIS) domain(Hemphill

etal.,1990). ATISsystemsareanearlyexampleofspokenlan guagesystemsforhelp-
ing book airline reservations. Users try to bookﬂights by co nversingwith the system,
specifyingconstraintslike I’dlike to ﬂyfrom Atlantato Denver . TheU.S. government
funded a number of different research sites to collect data a nd build ATIS systems in
theearly1990s. Thesentenceswewill bemodelinginthischa pteraredrawnfromthe
corpusofuserqueriestothesystem.
12.1 Constituency
How do words group together in English? Consider the noun phrase , a sequence of Nounphrase

wordssurroundingatleastonenoun. Herearesomeexampleso fnounphrases(thanks
toDamonRunyon):
HarrytheHorse a high-classspot suchasMindy’s
the Broadwaycoppers the reasonhecomesintotheHot Box
they threepartiesfromBrooklyn
How do we know that these words group together (or “form const ituents”)? One
piece of evidence is that they can all appear in similar synta ctic environments, for
examplebeforeaverb.

DRAFTSection12.2. Context-FreeGrammars 391
threepartiesfromBrooklyn arrive...
ahigh-classspotsuchasMindy’s attracts...
theBroadwaycoppers love...
theysit
Butwhilethewholenounphrasecanoccurbeforeaverb,thisi snottrueofeachof
the individual words that make up a noun phrase. The followin g are not grammatical
sentences of English (recall that we use an asterisk (*) to ma rk fragmentsthat are not
grammaticalEnglishsentences):
*fromarrive...*asattracts...
*theis... *spotis...
Thusto correctlydescribefactsaboutthe orderingofthese wordsin English,we must
beableto saythingslike“ NounPhrasescanoccurbeforeverbs ”.
Other kinds of evidence for constituency come from what are c alledpreposed or Preposed
postposed constructions. For example, the prepositional phrase on September sev- Postposed
enteenthcan be placed in a number of different locations in the follow ing examples,
includingpreposedat thebeginning,andpostposedatthe en d:
OnSeptemberseventeenth ,I’dliketo ﬂyfromAtlantato Denver

I’dliketo ﬂy onSeptemberseventeenth fromAtlantato Denver
I’dliketo ﬂyfromAtlanta toDenver onSeptemberseventeenth
But again, while the entire phrase can be placed differently , the individual words
makingupthephrasecannotbe:
*OnSeptember , I’dlike toﬂy seventeenth fromAtlantato Denver
*OnI’dliketoﬂy Septemberseventeenth fromAtlantato Denver
*I’dlike toﬂy onSeptember fromAtlanta toDenverseventeenth
Section 12.6 will give other motivations for context-free g rammars based on their
ability to model recursive structures. See Radford (1988) f or further examples of

groupsofwordsbehavingasasingle constituent.
12.2 Context-FreeGrammars
The most commonly used mathematical system for modeling con stituent structure in
Englishandothernaturallanguagesisthe Context-FreeGrammar ,orCFGContext- CFG
free grammars are also called Phrase-Structure Grammars , and the formalism is
equivalent to what is also called Backus-Naur Form orBNF. The idea of basing
a grammar on constituent structure dates back to the psychol ogist Wilhelm Wundt

(1900), but was not formalized until Chomsky (1956) and, ind ependently, Backus
(1959).
A context-free grammar consists of a set of rulesorproductions , each of which Rules
expressesthe waysthat symbolsofthe languagecanbe groupe dandorderedtogether,
and alexiconof words and symbols. For example, the following production sexpress Lexicon

DRAFT392 Chapter 12. FormalGrammarsofEnglish
that aNP(ornoun phrase ), can be composedof either a ProperNoun or a determiner NP
(Det)followedbya Nominal;aNominalcanbeoneormore Nouns.
NP→DetNominal
NP→ProperNoun
Nominal→Noun|NominalNoun
Context-freerulescanbehierarchicallyembedded,soweca ncombinetheprevious
ruleswith otherslike thefollowingwhichexpressfactsabo utthe lexicon:
Det→a
Det→the
Noun→ﬂight
ThesymbolsthatareusedinaCFG aredividedintotwoclasses . Thesymbolsthat
correspondtowordsinthelanguage(“the”,“nightclub”)ar ecalledterminal symbols; Terminal
the lexicon is the set of rules that introduce these terminal symbols. The symbols that
expressclustersor generalizationsof these are called non-terminals . Ineachcontext- Non-terminal
free rule, the item to the right of the arrow ( →) is an ordered list of one or more
terminals and non-terminals, while to the left of the arrow i s a single non-terminal
symbol expressingsome cluster or generalization. Notice t hat in the lexicon, the non-

terminalassociatedwith eachwordisitslexicalcategory, orpart-of-speech,whichwe
deﬁnedin Ch.5.
A CFG can be thought of in two ways: as a device for generating s entences, and
as a device for assigning a structureto a givensentence. We s aw this same dualism in
our discussion of ﬁnite-state transducers in Ch. 3. As a gene rator, we can read the →
arrowas“rewritethesymbolontheleft withthestringofsym bolsonthe right”.
Sostartingfromthe symbol: NP,
wecanuse rule12.2torewrite NPas: DetNominal
andthenrule12.2: DetNoun
andﬁnallyviarules12.2and12.2as: aﬂight

We say the string a ﬂightcan be derived from the non-terminal NP. Thus a CFG
can be used to generate a set of strings. This sequence of rule expansions is called a
derivation of the string of words. It is common to represent a derivation by aparse Derivation
tree(commonly shown inverted with the root at the top). Fig. 12.1 shows the tree Parse tree
representationofthisderivation.
NP
Det
aNom
Noun
ﬂight
Figure12.1 A parse tree for “a ﬂight”.
IntheparsetreeshowninFig.12.1,wecansaythatthenode NPdominates allthe Dominates

DRAFTSection12.2. Context-FreeGrammars 393
nodes in the tree ( Det,Nom,Noun,a,ﬂight). We can say further that it immediately
dominatesthenodes DetandNom.
The formallanguagedeﬁned by a CFG is the set of strings that a re derivablefrom
the designated start symbol . Each grammar must have one designated start symbol, Start symbol
whichisoftencalled S.Sincecontext-freegrammarsareoftenusedtodeﬁnesenten ces,
Sis usuallyinterpretedasthe “sentence”node,andthe set of stringsthat arederivable
fromSis theset ofsentencesinsomesimpliﬁedversionofEnglish.
Let’saddto ourlist ofrulesa fewhigher-levelrulesthatex pandS,anda coupleof
others. One will expressthe fact that a sentencecanconsist of a nounphrasefollowed
byaverb phrase : Verbphrase
S→NP VP I preferamorningﬂight
A verb phrase in English consists of a verb followed by assort ed other things; for
example,onekindofverbphraseconsistsofaverbfollowedb yanounphrase:
VP→Verb NP preferamorningﬂight
Or the verb phrase may have a verb followed by a noun phrase and a prepositional

phrase:
VP→Verb NP PP leaveBostoninthe morning
Orthe verbmaybefollowedbya prepositionalphrasealone:
VP→Verb PP leavingonThursday
A prepositionalphrasegenerallyhasa prepositionfollowe dby a nounphrase. For
example, a very common type of prepositional phrase in the AT IS corpus is used to
indicatelocationordirection:
PP→Preposition NP fromLosAngeles
TheNPinsideaPPneednotbealocation; PPsareoftenusedwithtimesanddates,
and with other nouns as well; they can be arbitrarily complex . Here are ten examples
fromtheATIScorpus:

to Seattle onthese ﬂights
in Minneapolis aboutthegroundtransportationinChicago
onWednesday oftheroundtripﬂightonUnitedAirlines
in theevening oftheAP ﬁftysevenﬂight
ontheninthofJuly witha stopoverinNashville
Fig.12.2givesasamplelexiconandFig.12.3summarizesthe grammarruleswe’ve
seen so far, which we’ll call L0. Note that we can use the or-symbol |to indicate that
a non-terminalhasalternatepossibleexpansions.
We can use this grammar to generate sentences of this “ATIS-l anguage”. We start

withS, expand it to NP VP, then choose a random expansion of NP(let’s say to I),
and a randomexpansion of VP(let’s say to Verb NP), and so on until we generate the

DRAFT394 Chapter 12. FormalGrammarsofEnglish
Noun→ﬂights|breeze|trip|morning
Verb→is|prefer|like|need|want|ﬂy
Adjective→cheapest|non−stop|ﬁrst|latest
|other|direct
Pronoun→me|I|you|it
Proper-Noun→Alaska|Baltimore|LosAngeles
|Chicago|United|American
Determiner→the|a|an|this|these|that
Preposition→from|to|on|near
Conjunction→and|or|but
Figure12.2 The lexicon for L0.
GrammarRules Examples
S→NPVP I +want amorningﬂight
NP→Pronoun I
|Proper-Noun LosAngeles
|DetNominal a + ﬂight
Nominal→NominalNoun morning+ ﬂight
|Noun ﬂights
VP→Verb do
|VerbNP want +a ﬂight
|VerbNPPP leave+ Boston+ inthemorning
|VerbPP leaving+onThursday
PP→PrepositionNP from+ LosAngeles
Figure12.3 The grammar for L0,withexample phrases for eachrule.
S
NP
Pro
IVP
Verb
preferNP
Det
aNom
Nom
Noun
morningNoun
ﬂight
Figure12.4 The parse tree for “I prefer amorning ﬂight” according togra mmar L0.
stringIpreferamorningﬂight . Fig.12.4showsaparsetreethatrepresentsacomplete
derivationof I preferamorningﬂight .

DRAFTSection12.2. Context-FreeGrammars 395
It is sometimes convenient to represent a parse tree in a more compact format
calledbracketed notation , essentially the same as LISP tree representations; here isBracketednotation
thebracketedrepresentationoftheparsetreeofFig.12.4:
(12.2)[S[NP[ProI]][VP[Vprefer][ NP[Deta] [Nom[Nmorning][ Nom[Nﬂight]]]]]]
A CFG like that of L0deﬁnes a formal language. We saw in Ch. 2 that a formal
languageisasetofstrings. Sentences(stringsofwords)th atcanbederivedbyagram-
mar are in the formal language deﬁned by that grammar, and are calledgrammatical Grammatical
sentences. Sentencesthat cannot be derivedby a given forma l grammarare not in the
language deﬁned by that grammar, and are referred to as ungrammatical . This hard Ungrammatical
linebetween“in”and“out”characterizesallformallangua gesbutisonlyaverysimpli-
ﬁedmodelofhownaturallanguagesreallywork. Thisisbecau sedeterminingwhether
a givensentenceispartofa givennaturallanguage(sayEngl ish)oftendependsonthe

context. Inlinguistics,theuseofformallanguagestomode lnaturallanguagesiscalled
generative grammar , since the language is deﬁned by the set of possible sentence sGenerativegrammar
“generated”bythegrammar.
12.2.1 Formaldeﬁnition ofcontext-free grammar
We conclude this section by way of summary with a quick formal description of a
context-free grammar and the language it generates. A conte xt-free grammar Gis
deﬁnedbyfourparameters N,Σ,R,S( technically“isa 4-tuple”):
Naset ofnon-terminalsymbols (orvariables )

Σaset ofterminalsymbols (disjointfrom N)
Raset ofrulesorproductions,eachofthe form A→β,
whereAisanon-terminal,
βisa stringofsymbolsfromthe inﬁniteset ofstrings (Σ∪N)∗
Sadesignated startsymbol
For the remainder of the book we’ll adhere to the following co nventions when dis-
cussing the formal properties (as opposed to explaining par ticular facts about English
orotherlanguages)ofcontext-freegrammars.
Capital letterslike A,B, andS Non-terminals
S Thestart symbol
Lower-caseGreekletterslike α,β,andγStringsdrawnfrom (Σ∪N)∗
Lower-caseRomanletterslike u,v, andwStringsofterminals

A language is deﬁned via the concept of derivation. One strin g derives another
one if it can be rewritten as the second one via some series of r ule applications. More
formally,followingHopcroftandUllman(1979),
ifA→βis a production of Pandαandγare any strings in the set (Σ∪
N)∗,thenwesay that αAγdirectlyderives αβγ, orαAγ⇒αβγ. Directlyderives

DRAFT396 Chapter 12. FormalGrammarsofEnglish
Derivationisthenageneralizationofdirectderivation:
Letα1,α2, ...,αmbestringsin (Σ∪N)∗,m≥1,suchthat
α1⇒α2,α2⇒α3,...,αm−1⇒αm (12.3)
We saythat α1derivesαm, orα1∗⇒αm. Derives
We can then formally deﬁne the language LGgenerated by a grammar Gas the
setofstringscomposedofterminalsymbolswhichcanbederi vedfromthedesignated
start symbol S.
LG={w|wisinΣ∗andS∗⇒w} (12.4)
Theproblemofmappingfroma stringofwordstoitsparsetree iscalledsyntactic
parsing;we will deﬁnealgorithmsforparsingin Ch.13. Syntacticparsing
12.3 SomeGrammarRulesfor English
In this section we introduce a few more aspects of the phrase s tructure of English; for
consistencywewillcontinuetofocusonsentencesfromtheA TISdomain. Becauseof
space limitations, ourdiscussion will necessarily be limi ted to highlights. Readersare
strongly advised to consult a good reference grammar of Engl ish, such as Huddleston
andPullum(2002).
12.3.1 Sentence-Level Constructions

Inthesmallgrammar L0,weprovidedonlyonesentence-levelconstructionfordecl ar-
ativesentenceslike Ipreferamorningﬂight . Therearealargenumberofconstructions
for English sentences, but four are particularly common and important: declarative
structure,imperativestructure,yes-no-questionstruct ure,andwh-questionstructure.
Sentences with declarative structure have a subject noun phrase followed by a Declarative
verbphrase,like “Iprefera morningﬂight”. Sentenceswith thisstructurehaveagreat

number of different uses that we will follow up on in Ch. 24. He re are a number of
examplesfromtheATISdomain:
Theﬂight shouldbeelevena.m. tomorrow
Thereturnﬂight shouldleaveataroundsevenp.m.
I’dliketo ﬂythecoachdiscountclass
Iwanta ﬂightfromOntariotoChicago
IplantoleaveonJulyﬁrst aroundsix thirtyintheevening
Sentences with imperative structure often begin with a verb phrase, and have no Imperative
subject. Theyarecalledimperativebecausetheyarealmost alwaysusedforcommands

andsuggestions;intheATIS domaintheyarecommandsto thes ystem.
Showthe lowestfare
Showme thecheapestfarethat haslunch
Giveme Sunday’sﬂightsarrivinginLasVegasfromNew YorkCi ty

DRAFTSection12.3. SomeGrammarRulesforEnglish 397
Listall ﬂightsbetweenﬁveandsevenp.m.
Showme allﬂightsthatdepartbeforetena.m. andhaveﬁrst cl assfares
Pleaselist theﬂightsfromCharlotteto LongBeacharriving afterlunchtime
Showme thelast ﬂighttoleave
We canmodelthissentencestructurewith anotherruleforth eexpansionof S:
S→VP
Sentenceswith yes-noquestion structureareoften(thoughnotalways)usedtoask Yes-no question
questions (hence the name), and begin with an auxiliary verb , followed by a subject
NP, followed by a VP. Here are some examples (note that the third example is not
really a question but a command or suggestion; Ch. 24 will dis cuss the uses of these
questionformstoperformdifferent pragmatic functionssuchasasking,requesting,or
suggesting.)
Doanyoftheseﬂightshavestops?
DoesAmerican’sﬂight eighteentwentyﬁveservedinner?
Canyougivemethesame informationforUnited?
Here’sthe rule:
S→Aux NP VP
Themost complexofthesentence-levelstructureswewill ex aminearethevarious

S→Aux NP VP
wh-structures. These are so named because one of their constitu entsis awh-phrase , Wh-phrase
that is, one that includes a wh-word (who, whose, when, where, what, which, how, Wh-word
why). Thesemaybebroadlygroupedintotwoclassesofsentence- levelstructures. The
wh-subject-question structure is identical to the declarative structure, excep t that the
ﬁrst nounphrasecontainssomewh-word.
Whatairlinesﬂy fromBurbanktoDenver?
WhichﬂightsdepartBurbankafternoonandarrivein Denverb ysix p.m?
Whoseﬂightsservebreakfast?

WhichoftheseﬂightshavethelongestlayoverinNashville?
Hereisarule. Exercise10discussesrulesfortheconstitue ntsthatmakeupthe Wh-NP.
S→Wh-NP VP
In thewh-non-subject question structure, the wh-phrase is not the subject of theWh-non-subjectquestion
sentence, andso the sentence includesanothersubject. In t hese typesof sentencesthe
auxiliaryappearsbeforethesubject NP,just asintheyes-no-questionstructures. Here
isanexamplefollowedbyasamplerule:
WhatﬂightsdoyouhavefromBurbanktoTacomaWashington?
S→Wh-NPAuxNPVP

S→Wh-NP VP
Constructions like the wh-non-subject-question contain what are called long-
distance dependencies because the Wh-NP what ﬂights is far away from the predi-Long-distance
dependencies

DRAFT398 Chapter 12. FormalGrammarsofEnglish
cate that it is semantically related to, the main verb havein theVP. In some models
of parsing and understanding compatible with the grammar ru le above, long-distance
dependencies like the relation between ﬂightsandhaveare thought of as a semantic
relation. In such models, the job of ﬁguring out that ﬂightsis the argument of have
is done during semantic interpretation. In other models of p arsing, the relationship
betweenﬂightsandhaveis considered to be a syntactic relation, and the grammar is
modiﬁed to insert a small marker called a traceorempty category after the verb.
We’ll returnto suchempty-categorymodelswhenwe introduc ethe PennTreebankon
page408.
There are other sentence-levelstructures we won’t try to mo del here, like topical-
izationorotherfrontingconstructions. Intopicalization(alsot reatedasalong-distance
dependencyin the Penn Treebank),a phrase is placedat the be ginningof the sentence
fordiscoursepurposes.

page408.
OnTuesday,I’dliketo ﬂyfromDetroitto SaintPetersburg
12.3.2 Clausesand Sentences
Before we move on, we should clarify the status of the Srules in the grammars we
just described. Srules are intended to account for entire sentences that stan d alone
as fundamental units of discourse. However, as we’ll see, Scan also occur on the
right-handside of grammarrulesand hence can be embeddedwi thin largersentences.
Clearlythenthere’smoreto beingan Sthenjust standingaloneasa unitofdiscourse.
What differentiates sentence constructions (i.e., the Srules) from the rest of the

grammaristhenotionthattheyareinsomesense complete. Inthiswaytheycorrespond
tothenotionofa clauseintraditionalgrammars,whichareoftendescribedasformi ng Clause
acompletethought. Onewayofmakingthisnotionof‘complet ethought’moreprecise
is to say an Sis a node of the parse tree below which the main verb of the Shas all
of itsarguments . We’ll deﬁne verbal arguments later, but for now let’s just s ee an
illustrationfromthe tree for I prefera morningﬂight in Fig. 12.4. Theverb preferhas

two arguments: the subject Iand the object a morning ﬂight . One of the arguments
appearsbelow the VPnode, but the other one, the subject NP, appearsonly below the
Snode.
12.3.3 The NounPhrase
OurL0grammar introduced three of the most frequent types of noun p hrases that
occur in English: pronouns, proper-nouns and the NP→Det Nominal construction.
Whilepronounsandproper-nounscanbecomplexintheirownw ays,thecentralfocus
ofthissectionisonthelasttypesincethatiswherethebulk ofthesyntacticcomplexity

Snode.
resides. We can view these noun phrases consisting of a head, the central noun in the
nounphrase,alongwithvariousmodiﬁersthatcanoccurbefo reoraftertheheadnoun.
Let’stakea closelookat thevariousparts.
The Determiner
Nounphrasescanbeginwithsimplelexicaldeterminers,asi nthefollowingexamples:

DRAFTSection12.3. SomeGrammarRulesforEnglish 399
a stop theﬂights thisﬂight
thoseﬂights anyﬂights someﬂights
TheroleofthedeterminerinEnglishnounphrasescanalsobe ﬁlledbymorecom-
plexexpressions,asfollows:
United’sﬂight
United’spilot’sunion
Denver’smayor’smother’scanceledﬂight
In these examples, the role of the determiner is ﬁlled by a pos sessive expression con-
sisting of a noun phrase followed by an ’sas a possessive marker, as in the following
rule.
Det→NP′s
Thefactthatthisruleisrecursive(sincean NPcanstartwitha Det),willhelpusmodel
the latter two examplesabove,wherea sequenceofpossessiv e expressionsservesasa
determiner.
Therearealso circumstancesunderwhichdeterminersareop tionalinEnglish. For
example,determinersmaybeomittedifthenountheymodifyi splural:
(12.5)Showme ﬂightsfromSanFranciscotoDenveronweekdays
As we saw in Ch. 5, mass nouns also don’t require determination. Recall that mass
nouns often (not always) involve something that is treated l ike a substance (including

e.g.,waterandsnow), don’ttake the indeﬁnite article “ a”, and don’ttend to pluralize.
Many abstract nouns are mass nouns ( music,homework ). Mass nouns in the ATIS
domaininclude breakfast,lunch,anddinner:
(12.6)Doesthisﬂightservedinner?
Exercise4 asksthe readertorepresentthisfactinthe CFG fo rmalism.
The Nominal
The nominal construction follows the determiner and contai nsany pre- and post-head
nounmodiﬁers. Asindicatedingrammar L0,initssimplestformanominalcanconsist
ofa singlenoun.
Nominal→Noun

As we’ll see, this rule also providesthe basis for the bottom of variousrecursiverules
usedtocapturemorecomplexnominalconstructions.
Beforethe HeadNoun
A number of different kinds of word classes can appear before the head noun (the
“postdeterminers”)inanominal. Theseinclude cardinalnumbers ,ordinalnumbers , Cardinal numbers
Ordinal numbers andquantiﬁers . Examplesofcardinalnumbers:
Quantiﬁerstwo friends onestop
Ordinal numbers include ﬁrst,second,third, and so on, but also words like next,

last,past,other,andanother:

DRAFT400 Chapter 12. FormalGrammarsofEnglish
the ﬁrst one thenextday thesecondleg
the last ﬂight theotherAmericanﬂight
Somequantiﬁers( many,(a)few,several)occuronlywithpluralcountnouns:
manyfares
Thequantiﬁers muchandalittleoccuronlywithnoncountnouns.
Adjectivesoccurafterquantiﬁersbutbeforenouns.
aﬁrst-class fare a nonstopﬂight
thelongestlayover the earliestlunchﬂight
Adjectives can also be grouped into a phrase called an adjective phrase or AP. Adjectivephrase
APscanhaveanadverbbeforetheadjective(seeCh.5fordeﬁn itionsofadjectivesand
adverbs):
theleastexpensive fare
We cancombineall theoptionsforprenominalmodiﬁerswith o neruleasfollows:
NP→(Det) (Card) (Ord) (Quant) (AP) Nominal
This simpliﬁed noun phrase rule has a ﬂatter structure and he nce is simpler than
would be assumed by most modern generative theories of gramm ar; as we will see
in Sec. 12.4, ﬂat structuresare often used for simplicity in computationalapplications
(and indeed, there is no universally agreed-upon internal c onstituency for the noun

adverbs):
phrase).
Note the use of parentheses “( )” to mark optional constituents . A rule with one
set of parentheses is really a shorthand for two rules, one wi th the parentheses, one
without.
AftertheHeadNoun
Aheadnouncanbefollowedby postmodiﬁers . Threekindsofnominalpostmodiﬁers
areverycommonin English:
prepositionalphrases all ﬂights from Cleveland
non-ﬁniteclauses anyﬂights arrivingafterelevena.m.
relativeclauses a ﬂight thatservesbreakfast
Prepositional phrase postmodiﬁers are particularly commo n in the ATIS corpus,

without.
since theyare used to mark the originand destinationof ﬂigh ts. Here are some exam-
ples,withbracketsinsertedtoshowtheboundariesofeachP P;notethatmorethanone
PP canbestrungtogether:
anystopovers [forDelta sevenﬁftyone]
allﬂights [from Cleveland][to Newark]
arrival[inSanJose][beforesevenp.m.]
areservation [onﬂightsix ohsix][from Tampa][to Montreal]
Here’sa newnominalruletoaccountforpostnominal PPs:
Nominal→Nominal PP

DRAFTSection12.3. SomeGrammarRulesforEnglish 401
Thethreemostcommonkindsof non-ﬁnite postmodiﬁersarethegerundive( -ing), Non-ﬁnite
-ed,andinﬁnitiveforms.
Gerundive postmodiﬁers are so-called because they consist of a verb ph rase that Gerundive
beginswith the gerundive( -ing)formof the verb. In the followingexamples, the verb
phrases happen to all have only prepositional phrases after the verb, but in general
this verb phrase can have anything in it (anything, that is, w hich is semantically and
syntacticallycompatiblewiththe gerundverb).
anyofthose [leavingonThursday]
anyﬂights [arriving afterelevena.m.]
ﬂights[arrivingwithin thirtyminutesof eachother]
We candeﬁnethe Nominals withgerundivemodiﬁersasfollows,makinguse ofanew
non-terminal GerundVP :
Nominal→NominalGerundVP
Wecanmakerulesfor GerundVP constituentsbyduplicatingallofourVPproductions,
substituting GerundV forV.
GerundVP→GerundVNP
|GerundVPP|GerundV|GerundVNPPP
GerundV canthenbedeﬁnedas:
GerundV→being|arriving|leaving|...

Thephrasesinitalicsbelowareexamplesofthetwoothercom monkindsofnon-ﬁnite
clauses,inﬁnitivesand -edforms:
thelast ﬂight toarrive inBoston
Ineedto havedinner served
Whichistheaircraft usedbythisﬂight ?
A postnominal relative clause (more correctly a restrictive relative clause ), is a
clausethatoftenbeginswitha relativepronoun (thatandwhoarethemostcommon). Relativepronoun
The relativepronounfunctionsasthe subject of the embedde dverb(isa subject rela-
tive)inthefollowingexamples:
aﬂightthatservesbreakfast

ﬂightsthatleaveinthemorning
theUnitedﬂight thatarrivesin SanJosearoundtenp.m.
theonethatleavesattenthirty ﬁve
We mightaddruleslikethefollowingtodealwith these:
Nominal→NominalRelClause
RelClause→(who|that)VP
The relative pronoun may also function as the object of the em bedded verb, as in
the following example; we leave as an exercise for the reader writing grammar rules
formorecomplexrelativeclausesofthiskind.

DRAFT402 Chapter 12. FormalGrammarsofEnglish
NP
PreDet
allNP
Det
theNom
Nom
Nom
Nom
Nom
Noun
ﬂightsPP
from DenverPP
to TampaGerundiveVP
leavingbefore10
Figure12.5 A parse tree for “allthe morning ﬂights fromDenver toTampa l eavingbefore 10”.
theearliestAmericanAirlinesﬂightthatIcanget
Variouspostnominalmodiﬁerscanbecombined,asthefollow ingexamplesshow:
aﬂight[fromPhoenix toDetroit][leaving Monday evening]
Ineedaﬂight [toSeattle][leavingfromBaltimore][makingastopinMinn eapolis]
evening ﬂights [from Nashville toHouston] [that serve dinner]
afriend[livinginDenver] [that wouldlike tovisitme here inWashin gton DC]
Beforethe NounPhrase
Word classes that modifyandappearbefore NPs are called predeterminers . Manyof Predeterminers
thesehavetodowithnumberoramount;acommonpredetermine risall:
all theﬂights allﬂights allnon-stopﬂights
TheexamplenounphrasegiveninFig.12.5illustratessomeo fthecomplexitythat
ariseswhenthese rulesarecombined.
12.3.4 Agreement

In Ch. 3 we discussed English inﬂectional morphology. Recal l that most verbs in En-
glish can appear in two forms in the present tense: the form us ed for third-person,
singularsubjects( theﬂightdoes ),andtheformusedforall otherkindsofsubjects( all

DRAFTSection12.3. SomeGrammarRulesforEnglish 403
the ﬂights do ,I do). The third-person-singular( 3sg) form usually has a ﬁnal -swhere
the non-3sg form does not. Here are some examples, again usin g the verb do, with
varioussubjects:
Do[NPall oftheseﬂights]offerﬁrst classservice?
Do[NPI]getdinneronthisﬂight?
Do[NPyou]haveaﬂightfromBostontoForthWorth?
Does[NPthisﬂight]stopinDallas?
Herearemoreexampleswiththeverb leave:
Whatﬂights leavein themorning?
Whatﬂight leavesfromPittsburgh?
This agreement phenomenon occurs whenever there is a verb th at has some noun
acting as its subject. Note that sentences in which the subje ct does not agree with the
verbareungrammatical:
*[Whatﬂight] leaveinthe morning?
*Does[ NPyou]havea ﬂightfromBostonto ForthWorth?
*Do[NPthisﬂight]stopin Dallas?
Howcanwemodifyourgrammartohandletheseagreementpheno mena? Oneway
isto expandourgrammarwith multiplesets ofrules,onerule set for3sgsubjects,and
one for non- 3sgsubjects. For example, the rule that handled these yes-no-q uestions

usedtolooklikethis:
S→Aux NP VP
We couldreplacethiswithtwo rulesofthefollowingform:
S→3sgAux 3sgNP VP
S→Non3sgAux Non3sgNP VP
We couldthenaddrulesforthelexiconlikethese:
3sgAux→does|has|can|...
Non3sgAux→do|have|can|...
But we would also need to add rules for 3sgNPandNon3sgNP , again by making
twocopiesofeachrulefor NP.Whilepronounscanbeﬁrst,second,orthirdperson,full
lexical noun phrases can only be third person, so for them we j ust need to distinguish
between singular and plural (dealing with the ﬁrst and secon d person pronounsis left
asanexercise):

3SgNP→DetSgNominal
Non3SgNP→DetPlNominal
SgNominal→SgNoun
PlNominal→PlNoun
SgNoun→ﬂight|fare|dollar|reservation|...
PlNoun→ﬂights|fares|dollars|reservations|...

DRAFT404 Chapter 12. FormalGrammarsofEnglish
Theproblemwith thismethodofdealingwith numberagreemen tisthat it doubles
the size of the grammar. Every rule that refers to a noun or a ve rb needs to have a
“singular” version and a “plural” version. Unfortunately, subject-verb agreement is
only the tip of the iceberg. We’ll also have to introduce copi es of rules to capture the
factthat headnounsandtheirdeterminershavetoagreeinnu mberaswell:
thisﬂight *thisﬂights
thoseﬂights *thoseﬂight
Rule proliferation will also have to happen for the noun’s case; for example English Case
pronouns have nominative (I, she, he, they ) andaccusative (me, her, him, them ) ver- Nominative
Accusative sions. We willneednewversionsofevery NPandNruleforeachofthese.
These problems are compounded in languages like German or Fr ench, which not
only have number-agreement as in English, but also have gender agreement . We Gender agreement
mentionedbrieﬂy in Ch. 3 that the genderof a nounmust agree w ith the gender of its

modifyingadjectiveanddeterminer. Thisaddsanothermult ipliertotherulesetsofthe
language.
Ch. 16 will introduce a way to deal with these agreement probl ems without ex-
ploding the size of the grammar, by effectively parameterizing each non-terminal of
the grammarwith featurestructures anduniﬁcation . But for manypractical compu-
tational grammars, we simply rely on CFGs and make do with the large numbers of
rules.
12.3.5 The VerbPhrase andSubcategorization
The verbphraseconsists of the verband a numberofother cons tituents. In the simple

language.
rules.
rules we have built so far, these otherconstituentsinclude NPs andPPs and combina-
tionsofthetwo:
VP→Verbdisappear
VP→VerbNP prefera morningﬂight
Verb phrases can be signiﬁcantly more complicated than this . Many other kinds
of constituents can follow the verb, such as an entire embedd ed sentence. These are
calledsententialcomplements :Sententialcomplements
You[VP[Vsaid[Sthereweretwoﬂightsthatwerethecheapest]]]
You[VP[Vsaid[Syouhadatwo hundredsixtysix dollarfare]]

[VP[VTell][NPme][ShowtogetfromtheairportinPhiladelphiatodowntown]]
I[VP[Vthink[SIwouldliketotake theninethirtyﬂight]]
Here’sa ruleforthese:
VP→Verb S
Another potential constituent of the VP is another VP. This i s often the case for
verbslike want,wouldlike ,try,intend,need:

DRAFTSection12.3. SomeGrammarRulesforEnglish 405
Iwant[ VPtoﬂy fromMilwaukeetoOrlando]
Hi,Iwant [ VPtoarrangethreeﬂights]
Hello, I’m trying [ VPto ﬁnd a ﬂight that goes from Pittsburgh to Denver after
twop.m.]
RecallfromCh.5thatverbscanalsobefollowedby particles,wordsthatresemble
aprepositionbutthatcombinewiththeverbtoforma phrasalverb liketakeoff. These
particles are generallyconsideredto be an integral part of the verbin a way that other
post-verbalelementsarenot;phrasalverbsaretreatedasi ndividualverbscomposedof
twowords.
While a verb phrase can have many possible kinds of constitue nts, not every verb
is compatible with every verb phrase. For example, the verb wantcan either be used
with an NP complement ( I want a ﬂight ... ), or with an inﬁnitive VP complement ( I
wanttoﬂyto... ). Bycontrast,averblike ﬁndcannottakethissortofVPcomplement.
(*Ifoundto ﬂytoDallas ).
This idea that verbs are compatible with different kinds of c omplements is a very

twowords.
old one; traditional grammar distinguishes between transitive verbs like ﬁnd, which Transitive
take a direct objectNP ( I founda ﬂight ),andintransitive verbslike disappear ,which Intransitive
donot(*Idisappeareda ﬂight ).
Wheretraditionalgrammars subcategorize verbsintothesetwocategories(transi- Subcategorize
tiveand intransitive),moderngrammarsdistinguishasman yas100subcategories. (In
fact, tagsets for many such subcategorization frames exist ; see Macleod et al. (1998)
fortheCOMLEXtagset,Sanﬁlippo(1993)fortheACQUILEXtag set,andfurtherdis-

cussion in Ch. 16). We say that a verb like ﬁndsubcategorizes for anNP, while a Subcategorizesfor
verb like wantsubcategorizes for either an NPor a non-ﬁnite VP. We also call these
constituents the complements of the verb (hence our use of the term sentential com- Complements
plementabove). So we say that wantcan take a VPcomplement. These possible sets
of complementsare called the subcategorizationframe for the verb. Anotherway ofSubcategorization
frame
talking about the relation between the verb and these other c onstituents is to think of

frame
the verb as a logical predicate and the constituents as logic al arguments of the predi-
cate. So we can think of such predicate-argumentrelations a sFIND(I,A FLIGHT ), or
WANT(I,TO FLY). WewilltalkmoreaboutthisviewofverbsandargumentsinC h.17
whenwetalkaboutpredicatecalculusrepresentationsofve rbsemantics.
Frame Verb Example
/ 0 eat, sleep Iate
NP prefer,ﬁnd,leave Find[NPthe ﬂightfromPittsburghtoBoston]
NP NP show,give Show[NPme][NPairlineswithﬂightsfromPittsburgh]
PPfromPPtoﬂy,travel Iwouldliketo ﬂy[ PPfromBoston][ PPto Philadelphia]

NP PPwith help,load Canyouhelp[ NPme][PPwith aﬂight]
VPto prefer,want,need Iwouldprefer[ VPtotogobyUnitedairlines]
VPbrst can,would,might Ican[VPbrstgofromBoston]
S mean Doesthismean[ SAA hasa hubin Boston]?
Figure12.6 Subcategorization frames for a set ofexample verbs.
Subcategorization frames for a set of example verbs are give n in Fig. 12.6. Note
thataverbcansubcategorizeforaparticulartypeofverbph rase,suchasaverbphrase

DRAFT406 Chapter 12. FormalGrammarsofEnglish
whose verb is an inﬁnitive ( VPto), or a verb phrase whose verb is a bare stem (un-
inﬂected: VPbrst). Note also that a single verb can take different subcategor ization
frames. Theverb ﬁnd,forexample,cantake an NP NPframe(ﬁndmea ﬂight )aswell
asanNPframe.
How can we represent the relation between verbs and their com plements in a
context-free grammar? One thing we could do is to do what we di d with agreement
features: make separate subtypes of the class Verb ( Verb-with-NP-complement ,Verb-
with-Inf-VP-complement ,Verb-with-S-complement ,andso on):
Verb-with-NP-complement →ﬁnd|leave|repeat|...
Verb-with-S-complement →think|believe|say|...
Verb-with-Inf-VP-complement →want|try|need|...
Theneach VPrulecouldbemodiﬁedtorequiretheappropriateverbsubtyp e:
VP→Verb-with-no-complement disappear
VP→Verb-with-NP-compNP preferamorningﬂight
VP→Verb-with-S-compS said thereweretwo ﬂights
Theproblemwiththisapproach,aswiththesamesolutiontot heagreementfeature

problem, is a vast explosion in the number of rules. The stand ard solution to both of
these problemsis the featurestructure ,whichwill be introducedin Ch. 16wherewe
will also discussthe factthat nouns,adjectives,andprepo sitionscansubcategorizefor
complementsjustasverbscan.
12.3.6 Auxiliaries
The subclass of verbs called auxiliaries orhelping verbs have particular syntactic Auxiliaries
constraintswhichcanbeviewedasakindofsubcategorizati on. Auxiliariesincludethe
modalverbscan, could, may, might, must, will, would, shall , andshould, theperfect Modal verb

Perfect auxiliary have, theprogressive auxiliary be, and the passiveauxiliary be. Each of
Progressive
Passivethese verbs places a constraint on the form of the following v erb, and each of these
mustalso combinein aparticularorder.
Modal verbssubcategorize for a VPwhose head verb is a bare stem; for example,
cangointhemorning ,willtrytoﬁndaﬂight . Theperfectverb havesubcategorizesfor
aVPwhoseheadverbisthepastparticipleform: havebooked3ﬂights . Theprogressive
verbbesubcategorizesfora VPwhoseheadverbisthegerundiveparticiple: amgoing

from Atlanta . The passiveverb besubcategorizesfora VPwhoseheadverbis the past
participle: wasdelayedbyinclementweather .
A sentence can have multiple auxiliary verbs, but they must o ccur in a particular
order:modal <perfect <progressive <passive. Here are some examplesofmultiple
auxiliaries:
modalperfect couldhavebeen a contender
modalpassive will bemarried
perfectprogressive havebeen feasting
modalperfectpassive mighthavebeen prevented

DRAFTSection12.3. SomeGrammarRulesforEnglish 407
Auxiliaries are often treated just like verbs such as want,seem, orintend, which
subcategorize for particular kinds of VPcomplements. Thus canwould be listed in
the lexicon as a verb-with-bare-stem-VP-complement . One way of capturing the or-
dering constraints among auxiliaries, commonly used in the systemic grammar of Systemic grammar
Halliday (1985), is to introduce a special constituent call ed theverb group , whose Verbgroup
subconstituentsincludeall the auxiliariesas well asthe m ainverb. Some ofthe order-
ing constraintscan also be capturedin a differentway. Sinc e modals, for example,do
not have a progressiveor participle form, they simply will n ever be allowed to follow
progressiveorpassive beorperfect have. Exercise8 asks the readerto write grammar
rulesforauxiliaries.
The passive construction has a number of properties that mak e it different than
otherauxiliaries. Oneimportantdifferenceisasemantico ne;whilethesubjectofnon-

passive(active)sentenceisoftenthesemanticagentoftheeventdescribed bytheverb Active
(Iprevented a catastrophe ) the subject of the passive is often the undergoeror patient
oftheevent( acatastrophe wasprevented ). Thiswill bediscussedfurtherinCh.18.
12.3.7 Coordination
The major phrase types discussed here can be conjoined with conjunctions likeand, Conjunctions
or, andbutto form larger constructions of the same type. For example a coordinate Coordinate
nounphrasecanconsist oftwoothernounphrasesseparatedb ya conjunction:
Pleaserepeat[ NP[NPtheﬂights] and[NPthecosts]]

Ineedto know[ NP[NPtheaircraft] and[NPtheﬂightnumber]]
Here’sa rulethatallowsthesestructures:
NP→NPandNP
Note that the ability to form coordinate phrases via conjunc tions is often used as
a test for constituency. Consider the following examples wh ich differ from the ones
givenabovein thattheylacktheseconddeterminer.
Pleaserepeatthe[ Nom[Nomﬂights]and[Nomcosts]]
Ineedto knowthe[ Nom[Nomaircraft]and[Nomﬂightnumber]]
The fact that these phrasescan be conjoinedisevidencefort he presenceof the under-

lyingNominalconstituentwe havebeenmakinguseof. Here’sa newrulefort his:
Nominal→NominalandNominal
Thefollowingexamplesillustrateconjunctionsinvolving VPsandSs.
What ﬂightsdoyouhave[ VP[VPleavingDenver] and[VParrivingin San
Francisco]]
[S[SI’minterested in a ﬂight fromDallas to Washington] and[SI’m also
interestedingoingto Baltimore]]
Therulesfor VPandSconjunctionsmirrorthe NPonegivenabove.
VP→VPandVP
S→SandS

DRAFT408 Chapter 12. FormalGrammarsofEnglish
Since all the major phrasetypes can be conjoinedin this fash ionit is also possible
to represent this conjunction fact more generally; a number of grammar formalisms
suchas(Gazdaret al.,1985)dothisvia metarules suchasthefollowing: Metarules
X→XandX
Thismetarulesimplystatesthatanynon-terminalcanbecon joinedwiththesamenon-
terminal to yield a constituent of the same type. Of course, t he variable Xmust be
designated as a variable that stands for any non-terminal ra ther than a non-terminal
itself.
12.4 Treebanks
Context-free grammar rules of the type that we have explored so far in this chapter
can be used, in principle, to assign a parse tree to any senten ce. This means that it
is possible to build a corpus in which every sentence is synta ctically annotated with
a parse tree. Such a syntactically annotated corpus is calle d atreebank . Treebanks Treebank
play an important roles in parsing, as we will see in Ch. 13, an d in various empirical

itself.
investigationsofsyntacticphenomena.
A wide variety of treebanks have been created, generally by u sing parsers (of the
sort described in the next two chapters) to automatically pa rse each sentence, and
then using humans(linguists) to hand-correctthe parses. T hePenn Treebank project PennTreebank
(whose POS tagset we introduced in Ch. 5) has produced treeba nks from the Brown,
Switchboard, ATIS, and Wall Street Journal corpora of Engli sh, as well as treebanks
in Arabic and Chinese. Other treebanksinclude the Prague De pendencyTreebankfor

Czech,theNegratreebankforGerman,andtheSusannetreeba nkforEnglish.
12.4.1 Example: The Penn Treebank Project
Fig. 12.7 shows sentences from the Brown and ATIS portions of the Penn Treebank.1
Note the formattingdifferencesfor the part-of-speechtag s; such small differencesare
common and must be dealt with in processing treebanks. The Pe nn Treebank part-
of-speech tagset was deﬁned in Ch. 5. The use of LISP-style pa renthesized notation

fortrees is extremelycommon,and resemblesthe bracketedn otationwe saw abovein
(12.2). For those who are not familiar with it we show a standa rd node-and-line tree
representationinFig. 12.8.
Fig.12.9showsatreefromtheWallStreetJournal. Thistree showsanotherfeature
of the Penn Treebanks: the use of traces(-NONE- nodes) to mark long-distance de- Traces
pendenciesor syntactic movement . For example, quotationsoften follow a quotativeSyntacticmovement
verblike say. But in thisexamplethe quotation“We wouldhaveto wait unti lwe have

collected on those assets” precedes the words he said. An empty S containing only
1The Penn Treebank project released treebanks in multiple la nguages and in various stages; for example
there were Treebank I (Marcus et al., 1993), Treebank II (Mar cus et al., 1994), and Treebank III releases of
English treebanks. Wewill useTreebank III for our examples .

DRAFTSection12.4. Treebanks 409
((S
(NP-SBJ (DT That)
(JJ cold) (, ,)
(JJ empty) (NN sky) )
(VP (VBD was)
(ADJP-PRD (JJ full)
(PP (IN of)
(NP (NN fire)
(CC and)
(NN light) ))))
(. .) ))((S
(NP-SBJ The/DT flight/NN )
(VP should/MD
(VP arrive/VB
(PP-TMP at/IN
(NP eleven/CD a.m/RB ))
(NP-TMP tomorrow/NN )))))
(a) (b)
Figure12.7 ParsedsentencesfromtheLDCTreebank3versionoftheBrown (a)andATIS(b)
corpora.
S
NP-SBJ
DT
ThatJJ
cold,
,JJ
emptyNN
skyVP
VBD
wasADJP-PRD
JJ
fullPP
IN
ofNP
NN
ﬁreCC
andNN
light.
.
Figure12.8 The tree corresponding tothe Browncorpus sentence inthe pr evious ﬁgure.
the node -NONE- is usedto markthepositionafter saidwherethe quotationsentence
often occurs. This empty node is marked (in Treebanks II and I II) with the index 2,
as is the quotation S at the beginning of the sentence. Such co indexing may make it
easier for some parsers to recover the fact that this fronted or topicalized quotation is
the complement of the verb said. A similar -NONE- node is used mark the fact that

.
there is no syntactic subject right before the verb to wait; instead, the subject is the
earlierNP We. Again,theyarebothcoindexedwiththe index1.
ThePennTreebankIIandTreebankIIIreleasesaddedfurther informationtomake
iteasiertorecovertherelationshipsbetweenpredicatesa ndarguments. Certainphrases
were marked with tags indicating the grammatical function o f the phrase (as surface
subject, logical topic, cleft, non-VP predicates) whether it appeared in particular text

categories (headlines, titles), and its semantic function (temporal phrases, locations)

DRAFT410 Chapter 12. FormalGrammarsofEnglish
( (S (‘‘ ‘‘)
(S-TPC-2
(NP-SBJ-1 (PRP We) )
(VP (MD would)
(VP (VB have)
(S
(NP-SBJ (-NONE- *-1) )
(VP (TO to)
(VP (VB wait)
(SBAR-TMP (IN until)
(S
(NP-SBJ (PRP we) )
(VP (VBP have)
(VP (VBN collected)
(PP-CLR (IN on)
(NP (DT those)(NNS assets)))))))))))))
(, ,) (’’ ’’)
(NP-SBJ (PRP he) )
(VP (VBD said)
(S (-NONE- *T*-2) ))
(. .) ))
Figure12.9 AsentencefromtheWallStreetJournalportionoftheLDCPen nTreebank. Note
the use of the empty -NONE- nodes.
(Marcusetal.,1994;Biesetal.,1995). Fig.12.9showsexam plesofthe -SBJ(surface
subject) and -TMP(temporal phrase) tags. Fig. 12.8 shows in addition the -PRDtag,
which is used for predicates which are not VPs (the one in Fig. 12.8 is an ADJP).
Fig.12.20showsthetag -UNFinNP-UNF meaning‘unﬁnishedorincompletephrase’.
12.4.2 Usinga Treebank asa Grammar
The sentences in a treebank implicitly constitute a grammar of the language. For ex-
ample, we can take the three parsed sentences in Fig. 12.7 and Fig. 12.9 and extract

each of the CFG rules in them. For simplicity, let’s strip off the rule sufﬁxes ( -SBJ
andsoon). TheresultinggrammarisshowninFig.12.10.
The grammar used to parse the Penn Treebank is relatively ﬂat , resulting in very
manyandverylongrules. Forexampleamongtheapproximatel y4,500differentrules
forexpandingVPareseparaterulesforPPsequencesofanyle ngth,andeverypossible
arrangementofverbarguments:
VP→VBD PP
VP→VBD PP PP
VP→VBD PP PP PP
VP→VBD PP PP PP PP
VP→VB ADVP PP
VP→VB PP ADVP

DRAFTSection12.4. Treebanks 411
Grammar Lexicon
S→NPVP. PRP→we|he
S→NPVP DT→the|that|those
S→”S”,NPVP. JJ→cold|empty|full
S→-NONE- NN→sky|ﬁre|light|ﬂight|tomorrow
NP→DTNN NNS→assets
NP→DTNNS CC→and
NP→NNCCNN IN→of|at|until|on
NP→CDRB CD→eleven
NP→DTJJ,JJNN RB→a.m.
NP→PRP VB→arrive|have|wait
NP→-NONE- VBD→was|said
VP→MDVP VBP→have
VP→VBDADJP VBN→collected
VP→VBDS MD→should|would
VP→VBNPP TO→to
VP→VBS
VP→VBSBAR
VP→VBPVP
VP→VBNPP
VP→TOVP
SBAR→INS
ADJP→JJPP
PP→INNP
Figure12.10 A sample of the CFG grammar rules and lexical entries that wou ld be extracted
from the three treebank sentences inFig.12.7and Fig.12.9.
VP→ADVP VB PP
aswell asevenlongerrules,sucha:
VP→VBP PP PP PP PP PP ADVP PP
whichcomesfromtheVP markedinitalics:
(12.7)Thismostlyhappensbecausewe gofrom footballinthefall toliftingin the
wintertofootballagaininthe spring .
SomeofthemanythousandsofNP rulesinclude:
NP→DT JJ NN
NP→DT JJ NNS
NP→DT JJ NN NN
NP→DT JJ JJ NN
NP→DT JJ CD NNS
NP→RB DT JJ NN NN
NP→RB DT JJ JJ NNS
NP→DT JJ JJ NNP NNS

NP→DT JJ NN
NP→DT NNP NNP NNP NNP JJ NN
NP→DT JJ NNP CC JJ JJ NN NNS
NP→RB DT JJS NN NN SBAR
NP→DT VBG JJ NNP NNP CC NNP
NP→DT JJ NNS , NNS CC NN NNS NN

DRAFT412 Chapter 12. FormalGrammarsofEnglish
NP→DT JJ JJ VBG NN NNP NNP FW NNP
NP→NP JJ , JJ ‘‘ SBAR ’’ NNS
Thelast twoofthoserules,forexample,comefromthefollow ingtwoNPs:
(12.8) [DTThe] [JJstate-owned ] [JJindustrial ] [VBGholding ] [NNcompany ] [NNP
Instituto ] [NNPNacional ] [FWde] [NNPIndustria ]
(12.9) [NPShearson’s ] [JJeasy-to-ﬁlm ],[JJblack-and-white ]“[SBARWhereWe Stand ]”
[NNScommercials ]
Viewed as a large grammar in this way, the Penn Treebank III Wa ll Street Journal
corpus,whichcontainsabout1millionwords,alsohasabout 1millionnon-lexicalrule
tokens,consistingofabout17,500distinctruletypes.
Variousfactsaboutthetreebankgrammars,suchastheirlar genumbersofﬂatrules,
pose problems for probabilistic parsing algorithms. For th is reason, it is common to
make various modiﬁcations to a grammar extracted from a tree bank. We will discuss
thesefurtherinCh. 14.
12.4.3 Searching Treebanks
It is often important to search througha treebankto ﬁnd exam plesof particular gram-

matical phenomena, either for linguistic research or for an swering analytic questions
about a computational application. But neither the regular expressions used for text
search nor the boolean expressions over words used for web se arch are a sufﬁcient
searchtool. What is neededis a languagethatcan specifycon straintsaboutnodesand
linksin aparsetree,so astosearchforspeciﬁc patterns.
Various such tree-searching languages exist in different t ools.Tgrep(Pito, 1993)

andTGrep2(Rohde, 2005) are publicly-available tools for searching t reebanks that
use a similar language for expressing tree constraints. We’ ll describe the more recent
languageusedby TGrep2,drawingfromtheonlinemanual(Rohde,2005).
A pattern in tgreporTGrep2consists of a speciﬁcation of a node, possibly fol-
lowed by links to other nodes. A node speciﬁcation can then be used to return the
subtreerootedat thatnode. Forexample,thepattern
NP
returns all subtrees in a corpus whose root is NP. Nodes can be speciﬁed by a name,

NP
a regular expression inside slashes, or a disjunction of the se. For example, we can
specify a singular or plural noun ( NNorNNS) using Penn Treebank notation as either
ofthefollowing:
/NNS?/ NN|NNS
A node which either is the word bushor else ends in the string treecan be expressed
as:
/tree$/|bush
The power of tgrep/TGrep2 patterns is the ability to specify information about
links. The operator <meansimmediately dominates ; the following pattern thus
matchesanNP immediatelydominatingaPP
NP < PP

DRAFTSection12.4. Treebanks 413
The relation <<is used to specify dominance; this pattern matches an NP domi nating
a PP:
NP << PP
Thispreviouspatternwouldthusmatcheitherofthefollowi ngtrees:
(12.10) (NP (NP (NN reinvestment))
(PP (IN of)
(NP (NNS dividends))))
(12.11) (NP (NP (DT the) (JJ austere) (NN company) (NN dormitory))
(VP (VBN run)
(PP (IN by) (NP (DT a) (JJ prying) (NN caretaker)))))
Therelation .isusedtomarklinearprecedence. Thefollowingpatternmat chesan
NP thatimmediatelydominatesa JJ andisimmediatelyfollow edbya PP, forexample
matchingtheNP dominating theausterecompanydormitory in (12.11)above:2
NP < JJ . VP
Each of the relations in a tgrep/TGrep2expression is interpreted as referring to
the ﬁrst or root node. Thus for example the followingexpress ion means an NP which
bothprecedesaPP anddominatesanS:
NP . PP < S
Ifwe wantedinstead to specifythat the PP dominatedthe S, we coulduse parentheses
NP . (PP < S)
Fig.12.11givesthemajorlinkoperationsfor TGrep2.
12.4.4 Headsand HeadFinding

NP < JJ . VP
NP . PP < S
NP . (PP < S)
We suggested informally earlier that syntactic constituen ts could be associated with a
lexicalhead;Nis the head of an NP,Vis the head of a VP. This idea of a head for
each constituent dates back to Bloomﬁeld (1914). It is centr al to such linguistic for-
malismssuchasHead-DrivenPhraseStructureGrammar(Poll ardandSag,1994),and
hasbecomeextremelypopularin computationallinguistics with therise of lexicalized
grammars(Ch.14).
In one simple model of lexical heads, each context-free rule is associated with

a head (Charniak, 1997; Collins, 1999). The head is the word i n the phrase which
is grammatically the most important. Heads are passed up the parse tree; thus each
non-terminal in a parse-tree is annotated with a single word which is its lexical head.
Fig. 12.12 shows an example of such a tree from Collins (1999) , in which each non-
terminal is annotatedwith its head. “Workersdumpedsacks i nto a bin” is a shortened
formofa WSJsentence.

Inorderto generatesuch a tree, each CFG rule must be augment edto identifyone
right-hand-side constituent to be the head daughter. The he adword for a node is then
2The deﬁnition of linear precedence differs slightly betwee ntgrepandTGrep2. See Rohde (2005) for
moredetails.

DRAFT414 Chapter 12. FormalGrammarsofEnglish
Link Explanation
A < B Ais the parent of (immediatelydominates) B.
A > B Ais the childof B.
A <N B Bis the Nthchildof A(the ﬁrstchildis <1).
A >N B Ais the Nthchildof B(the ﬁrstchildis >1).
A <, B Synonymous with A <1 B.
A >, B Synonymous with A >1 B.
A <-N B Bis the Nth-to-lastchildof A (the lastchildis <-1).
A >-N B Ais the Nth-to-lastchild ofB (the lastchildis >-1).
A <- B Bis the lastchildof A (synonymous with A <-1 B ).
A >- B Ais the lastchild ofB (synonymous with A >-1 B ).
A <‘ B Bis the lastchildof A (alsosynonymous with A <-1 B ).
A >‘ B Ais the lastchild ofB (alsosynonymous with A >-1 B ).
A <: B Bis the onlychild ofA
A >: B Ais the only childof B
A << B Adominates B (Ais anancestor of B).
A >> B Ais dominated by B(A isa descendant of B).
A <<, B Bis a left-mostdescendant of A.
A >>, B Ais a left-mostdescendant of B.
A <<‘ B Bis a right-most descendant of A.
A >>‘ B Ais a right-most descendant ofB.

A <<: B There isa single path of descent from A andB is onit.
A >>: B There isa single path of descent from B andA is onit.
A . B Aimmediately precedes B.
A , B Aimmediately follows B.
A .. B Aprecedes B.
A ,, B Afollows B.
A $ B Ais a sisterof B (andA ̸=B).
A $. B Ais a sisterof and immediatelyprecedes B.
A $, B Ais a sisterof and immediatelyfollows B.
A $.. B Ais a sisterof and precedes B.
A $,, B Ais a sisterof and follows B.
Figure12.11 Links inTGrep2,summarized from Rohde (2005).

set to the headword of its head daughter. Choosing these head daughtersis simple for
textbookexamples( NNisthe headof NP) butiscomplicatedandindeedcontroversial
for most phrases. (Shouldthe complementizer toor the verbbe the head of an inﬁnite
verb-phrase?) Modernlinguistictheoriesofsyntaxgenera llyincludeacomponentthat
deﬁnesheads(seee.g.,PollardandSag,1994).
An alternative approach to head-ﬁnding is used in most pract ical computational
systems. Instead of specifying head rules in the grammar its elf, heads are identiﬁed

dynamically in the context of trees for speciﬁc sentences. I n other words, once a sen-
tence isparsed,the resultingtree is walkedto decorateeac hnodewith the appropriate
head. Mostcurrentsystemsrelyonasimplesetofhand-writt enrules,suchasapracti-
cal one for Penn Treebankgrammarsgiven in Collins (1999)bu t developedoriginally
by Magerman (1995b). For example their rule for ﬁnding the he ad of an NP is as
followsCollins(1999,238):
•Ifthe lastword istagged POS,returnlast-word.

DRAFTSection12.5. GrammarEquivalenceandNormalForm 415
S(dumped)
NP(workers) VP(dumped)
NNS(workers) VBD(dumped) NP(sacks) PP(into)
NNS(sacks) P(into) NP(bin)
DT(a) NN(bin)
workers dumped sacks into a bin
Figure12.12 A lexicalizedtree from Collins (1999).
•Else search from right to left for the ﬁrst child which is an NN , NNP, NNPS, NX, POS,
orJJR.
•Elsesearchfrom left toright for the ﬁrstchildwhichis anNP .
•Elsesearchfrom right toleftfor the ﬁrstchildwhichis a $,A DJP,or PRN.
•Elsesearchfrom right toleftfor the ﬁrstchildwhichis a CD.
•Elsesearchfrom right toleftfor the ﬁrstchildwhichis a JJ, JJS,RB or QP.
•Elsereturnthe lastword
Selected other rules from their set are shown in Fig. 12.13. F or example, for VP
rules of the form VP→Y1···Yn, the algorithm would start from the left of Y1···Yn
looking for the ﬁrst Yiof type TO; if no TOs are found it would search for the ﬁrst Yi
oftype VBD; if noVBDs are foundit wouldsearch fora VBP, ands o on. See Collins
(1999)formoredetails.
Parent Direction PriorityList

ADJP Left NNS QP NN $ADVPJJ VBN VBG ADJP JJRNP JJS DTFW RBRRBS
SBAR RB
ADVP Right RB RBR RBS FW ADVPTOCD JJR JJINNP JJS NN
PRN Left
PRT Right RP
QP Left $ INNNS NNJJ RBDT CD NCD QP JJRJJS
S Left TOIN VPS SBAR ADJP UCP NP
SBAR Left WHNP WHPP WHADVP WHADJP INDT S SQ SINVSBAR FRAG
VP Left TOVBD VBN MD VBZ VBVBG VBP VP ADJP NNNNS NP
Figure12.13 Selected head rules from Collins (1999). The set of head rule s is often called a head percolation
table.
12.5 GrammarEquivalenceandNormalForm

SBAR RB
PRN Left
table.
A formal language is deﬁned as a (possibly inﬁnite) set of str ings of words. This
suggests that we could ask if two grammars are equivalent by a sking if they generate

DRAFT416 Chapter 12. FormalGrammarsofEnglish
thesamesetofstrings. Infactitispossibletohavetwodist inctcontext-freegrammars
generatethesame language.
We usuallydistinguishtwo kindsofgrammarequivalence: weak equivalence and
strong equivalence . Two grammarsare strongly equivalent if they generate the s ame
set of strings andif they assign the same phrase structure to each sentence (al lowing
merelyforrenamingofthenon-terminalsymbols). Twogramm arsare weaklyequiva-
lentif theygeneratethesameset ofstringsbutdonotassign thesamephrasestructure
toeachsentence.
It is sometimes useful to have a normal form for grammars, in which each of Normal form
the productions takes a particular form. For example a conte xt-free grammar is in
Chomsky Normal Form (CNF) (Chomsky, 1963) if it is ǫ-free and if in additionChomsky Normal
Form
each production is either of the form A→B CorA→a. That is, the right-hand side
of each rule either has two non-terminal symbols or one termi nal symbol. Chomsky

Form
normal form grammars are binary branching , i.e. have binary trees (down to the Binary branching
prelexical nodes). We will make use of this binary branching property in the CKY
parsingalgorithminCh. 13.
Any grammar can be converted into a weakly-equivalent Choms ky normal form
grammar. Forexample,a ruleofthe form
A→B C D
can be converted into the following two CNF rules (Exercise 1 1 asks the reader to
formulatethecompletealgorithm):
A→B X
X→C D
Sometimes using binary branching can actually produce smal ler grammars. For

A→B C D
A→B X
X→C D
examplethesentencesthatmightbecharacterizedasfollow s:
VP -> VBD NP PP *
arerepresentedinthePennTreebankbythisseriesofrules:
VP→VBD NP PP
VP→VBD NP PP PP
VP→VBD NP PP PP PP
VP→VBD NP PP PP PP PP
...
butcouldalso begeneratedbythe followingtwo-rulegramma r:
VP→VP PP
To generate a symbol A with a potentially inﬁnite sequence of symbols B by using a
ruleoftheform A→A Bisknownas Chomsky-adjunction .Chomsky-
adjunction

DRAFTSection12.6. Finite-StateandContext-FreeGrammars 417
12.6 Finite-StateandContext-FreeGrammars
We arguedin Sec. 12.1 that adequate models of grammar need to be able to represent
complex interrelated facts about constituency, subcatego rization, and dependency re-
lations, and we implied that at the least the power of context -freegrammarsis needed
to accomplish this. But why is it that we can’t just use ﬁnite- state methods to cap-
ture these syntactic facts? The answer to this question is cr itical since, as we’ll see in
Ch. 13, thereis a considerableprice to be paid in terms of pro cessingspeed whenone
switchesfromregularlanguagesto context-freeones.
There are two answers to this question. The ﬁrst is mathemati cal; we’ll show in
Ch. 15 that given certain assumptions, that certain syntact ic structures present in En-
glish (and other natural languages) make them not regular la nguages. The second an-
swerismoresubjectiveandhastodowithnotionsofexpressi veness;evenwhenﬁnite-

state methods are capable of dealing with the syntactic fact s in question, they often
don’texpress them in ways that make generalizationsobviou s, lead to understandable
formalisms,orproducestructuresofimmediateuseinsubse quentsemanticprocessing.
ThemathematicalobjectionwillbediscussedmorefullyinC h.15,butwe’llbrieﬂy
review it here. We mentionedin passing in Ch. 2 that there is a completely equivalent
alternative to ﬁnite-state machines and regular expressio ns for describing regular lan-

guages,called regulargrammars . Therulesinaregulargrammararearestrictedform
of the rules in a context-free grammar because they are in rig ht-linear or left-linear
form. In a right-linear grammar, for example, the rules are a ll of the form A→w∗or
A→w∗B,thatisthenon-terminalseitherexpandtoastringoftermi nalsortoastring
of terminals followed by a non-terminal. These rules look an awful lot like the rules
we’vebeenusingthroughoutthischapter,sowhatcan’tthey do? Whattheycan’tdois

expressrecursive center-embedding rules like the following, where a non-terminalis
rewrittenasitself, surroundedby(non-empty)strings:
A∗⇒αAβ (12.12)
In other words, a language can be generated by a ﬁnite-state m achine if and only
if the grammar that generates Lthat does not have any center-embedded recursions
of this form (Chomsky, 1959a; Bar-Hillel et al., 1961; Neder hof, 2000). Intuitively,
this is because grammarrules in which the non-terminalsymb olsare always on either

therightorleftedgeofarulecanbeprocessediterativelyr atherthanrecursively. Such
center-embeddingrulesareneededtodealwithartiﬁcialpr oblemssuchasthelanguage
anbn, or for practical problems such as checking for correctly ma tching delimiters in
programmingandmarkuplanguages. Itturnsoutthattherear enoslam-dunkexamples
ofthisforEnglish,but exampleslikethefollowinggivea ﬂa voroftheproblem.
(12.13)Theluggagearrived.
(12.14)Theluggagethatthepassengerscheckedarrived.
(12.15)Theluggagethatthepassengersthatthestormdelayedcheck edarrived.

Atleastintheory,thiskindofembeddingcouldgoon,althou ghitgetsincreasingly
difﬁcult to process such examples and they are luckily fairl y rare outside textbooks

DRAFT418 Chapter 12. FormalGrammarsofEnglish
like this one. Ch. 15 will discuss this and related issues as t o whether or not even
context-freegrammarsareupto thetask.
So is there no role for ﬁnite-state methods in syntactic anal ysis? A quick review
of the rules used for noun-phrases in this chapter, as well as those used in the Penn
treebankgrammar,revealsthataconsiderableportionofth emcanbehandledbyﬁnite-
state methods. Consider the following rule for a noun group , the pre-nominal and Noun group
nominalportionsofanounphrase:
Nominal→(Det) (Card) (Ord) (Quant ) (AP)Nominal
Assuming we convert the pre-nominal elements of this rule in to terminals, this rule
is effectively right-linear and can be captured by a ﬁnite-s tate machine. Indeed, it
is possible to automatically build a regular grammar which i s an approximation of a
given context-free grammar; see the references at the end of the chapter. Thus for
manypracticalpurposeswherematchingsyntacticandseman ticrulesaren’tnecessary,

ﬁnite-staterulesare quitesufﬁcient.
12.7 DependencyGrammars
We have focused in this chapter on context-free grammars bec ause many available
treebanksandparsersproducethesekindsofsyntacticrepr esentation. Butinaclassof
grammarformalismscalled dependencygrammars thatarebecomingquiteimportantDependencygrammar
in speechandlanguageprocessing,constituentsandphrase -structurerulesdonotplay
anyfundamentalrole. Instead,thesyntacticstructureofa sentenceisdescribedpurely

in terms of words and binary semantic or syntactic relations between these words.
Dependency grammars often draw heavily from the work of Tesn i` ere (1959), and the
namedependency might have been used ﬁrst by early computational linguist Da vid Dependency
Hays. Butthislexicaldependencynotionofgrammarisinfac tolderthantherelatively
recentphrase-structureorconstituencygrammars,andhas itsrootsintheancientGreek
andIndianlinguistictraditions. Indeedthe notionintrad itionalgrammarof“parsinga

sentenceintosubjectandpredicate”isbasedonlexicalrel ationsratherthanconstituent
relations.
Figure12.14 Asampledependencygrammarparse,usingthedependencyfor malismofKarls-
sonet al.(1995b), after J¨ arvinen andTapanainen (1997).
Fig. 12.14 shows an example parse of the sentence I gave him my address , using
the dependency grammar formalism of J¨ arvinen and Tapanain en (1997) and Karls-
son et al. (1995b). Note that there are no non-terminal or phr asal nodes; each link in

DRAFTSection12.7. DependencyGrammars 419
Dependency Description
subj syntacticsubject
obj directobject(incl. sententialcomplements)
dat indirectobject
pcomp complementofa preposition
comp predicatenominals(complementsofcopulas)
tmp temporaladverbials
loc locationadverbials
attr premodifying(attributive)nominals(genitives,etc.)
mod nominalpostmodiﬁers(prepositionalphrases,etc.)
Figure12.15 A representative set of grammatical relations from (J¨ arvi nen and Tapanainen,
1997).
the parse tree holds between two lexical nodes (augmented wi th the special <ROOT >
node). The links are drawn from a ﬁxed inventory of around 35 r elations, most of
which roughly represent grammatical functions or very gene ral semantic relations.
Other dependency-based computational grammars, such as Link Grammar (Sleator Link Grammar
and Temperley,1993),use differentbut roughlyoverlappin glinks. Table 12.15shows
a fewoftherelationsusedin J¨ arvinenandTapanainen(1997 ):

1997).
As we will see in Ch. 14, one advantage of dependency formalis ms is the strong
predictiveparsingpowerthatwordshavefortheirdependen ts. Knowingtheidentityof
the verb is often a very useful cue for deciding which noun is l ikely to be the subject
ortheobject. Dependencygrammarresearchersarguethaton eofthemainadvantages
of pure dependencygrammars is their ability to handle langu ages with relatively free
wordorder . ForexamplethewordorderinlanguageslikeCzechismuchmo reﬂexible Freewordorder

than in English; an objectmight occur beforeor after a location adverbial or acomp.
A phrase-structure grammar would need a separate rule for ea ch possible place in the
parsetreethatsuchanadverbialphrasecouldoccur. Adepen dencygrammarwouldjust
have one link-type representing this particular adverbial relation. Thus a dependency
grammarabstracts away from word-ordervariation, represe ntingonly the information
thatisnecessaryforthe parse.
There are a number of computational implementations of depe ndency grammars;

LinkGrammar(SleatorandTemperley,1993)andConstraintG rammar(Karlssonetal.,
1995b)are easily-available broad-coveragedependencygr ammarsand parsersforEn-
glish. Dependencygrammarsarealsooftenusedforotherlan guages. Hajiˇ c(1998),for
example, describes the 500,000 word Prague Dependency Tree bank for Czech which
hasbeenusedtotrainprobabilisticdependencyparsers(Co llinset al.,1999).
12.7.1 The RelationshipBetween Dependencies and Heads
Thereadermayhavenoticedthesimilaritybetweendependen cygraphslikeFig.12.14

and head structures like Fig. 12.12. In fact an (unlabeled) d ependency graph can be
automatically derived from a context-free parse by using th e head rules; here’s an al-
gorithmfromXiaandPalmer(2001):
1. Marktheheadchildofeachnodeinaphrasestructure,usin gtheheadpercolation
table.

DRAFT420 Chapter 12. FormalGrammarsofEnglish
2. In the dependency structure, make the head of each non- hea d-child depend on
theheadofthehead-child.
This algorithm applied to the parse tree in Fig. 12.16 would p roduce the dependency
structurein Fig.12.17.
S
NP-SBJ
NNP
VinkenVP
MD
willVP
VB
joinNP
DT
theNN
boardPP-CLR
IN
asNP
DT
aJJ
nonexecutiveNN
directorNP-TMP
NNP
NovCD
29.
.
Figure12.16 A phrase structure tree from the Wall Street Journal compone nt of the Penn
Treebank 3
join
Vinken will board
theas
director
a nonexecutive29
Nov
Figure12.17 The dependency treeproduced from Fig.12.16 bythe algorith m givenabove.
We will return to the discussion of heads and dependencies wh en we discuss lex-
icalized parsing in Ch. 14 and again when we introduce head fe atures and subcatego-
rizationin Ch.16.
12.7.2 CategorialGrammar
Categorialgrammar isanearlylexicalizedgrammarmodel(Adjukiewicz,1935;B ar-Categorialgrammar

DRAFTSection12.8. SpokenLanguageSyntax 421
Hillel,1953). Inthissectionwewillgiveasimpliﬁedoverv iewofoneimportantexten-
sion to categorial grammar, combinatory categorial grammar , or CCG (Steedman,Combinatory
categorialgrammar1996, 1989, 2000). A categorial grammar has two components. Thecategorial lex-
iconassociates each word with a syntactic and semantic category . Thecombinatory
rulesallow functions and arguments to be combined. There are two t ypes of cate-
gories: functorsandarguments. Arguments,likenouns,hav esimplecategorieslike N.
Verbs or determiners act as functors. For example, a determi ner can be thought of as
a functionthat appliesto an N on its right to producean NP. Su ch complexcategories
are built using the X/Y and X \Y operators. X/Y means a function from Y to X, that
is, somethingwhichcombineswith a Y on itsrightto producea nX. Determinersthus
receivethe categoryNP/N: somethingthat combineswith an N on its rightto produce

anNP.TransitiveverbsmighthavethecategoryVP/NP;somet hingthatcombineswith
anNPontherighttoproduceaVP.Ditransitiveverbslike givemighthavethecategory
(VP/NP)/NP; something which combines with an NP on its right to yield a transitive
verb. The simplest combination rules just combine an X/Y with a Y on its right to
produceanX ora X \Ywitha Yonitsleft toproduceanX.
Consider the simple sentence Harry eats apples from Steedman (1989). Instead
of using a primitive VP category, let’s assume that a ﬁnite ve rb phrase like eat apples

hasthecategory(S \NP);somethingwhichcombineswithanNPonthelefttoproduc e
a sentence. Harryandapplesare both NPs. Eatsis a ﬁnite transitive verb which
combines with an NP on the right to produce a ﬁnite VP: (S \NP)/NP. The derivation
ofS proceedsasfollows:
(12.16)Harryeats apples
NP (S\NP)/NP NP
S\NP
S
Modern categorial grammars include more complex combinato ry rules which are
needed for coordinationand other complexphenomena,and al so include composition
ofsemanticcategoriesaswellassyntacticones. Seetheend ofthechapterforapointer
tousefulreferences.

S\NP
S
12.8 SpokenLanguageSyntax
The grammar of written English and the grammar of conversati onal spoken English
sharemanyfeatures,butalsodifferinanumberofrespects. Thissectiongivesa quick
sketchofa numberofthecharacteristicsofthesyntaxofspo kenEnglish.
We usually use the term utterance rather than sentence for the units of spoken Utterance
language. Fig. 12.18 shows some sample spoken ATIS utteranc es that exhibit many
aspectsofspokenlanguagegrammar.
This is a standard style of transcription used in transcribi ng speech corpora for

speech recognition. The comma “,” marks a short pause, and ea ch period “.” marksa
longpause. Fragments (incompletewordslike wha-forincomplete what)aremarked
with a dash, and the square brackets “[smack]” mark non-verb al events ( lipsmacks ,

DRAFT422 Chapter 12. FormalGrammarsofEnglish
the. [exhale]. . . [inhale]. . uhdoesAmericanairlines. off erany. onewayﬂights. uhonewayfares,
foronehundredandsixtyonedollars
[mm]i’dliketoleaveiguessbetweenum. [smack]. ﬁveo’cloc kno,ﬁveo’clockanduh,seveno’clock
. P M
all right, [throat clear] . . i’d like to know the . give me the ﬂight . times . in the morning . for
Septembertwentieth. nineteenninetyone
uhoneway
. w-wha-whatisthelowest,cost, fare
[click]. i needto ﬂy,betwee-. leaving. Philadelphia. to,A tlanta[exhale]
on United airlines. . give me, the . . time . . fromNew York . [sm ack]. to Boise-, to . I’m sorry . on
Unitedairlines. [uh]givemethe ﬂight,numbers,theﬂightt imesfrom. [uh]Boston . toDallas
Figure12.18 Sample spoken utterances fromusers interactingwithan ATI Ssystem.
breaths,etc.).
Thereareanumberofwaystheseutterancesdifferfromwritt enEnglishsentences.
Oneisinthelexicalstatistics; forexamplespokenEnglish ismuchhigherinpronouns

than written English; the subject of a spoken sentence is alm ost invariably a pronoun.
Spokensentencesoften consist of short fragmentsor phrase s(one wayoraroundfour
p.m.,whicharelesscommonin writtenEnglish. Spokensentences havephonological,
prosodic, and acoustic characteristics that of course writ ten utterances don’t have; we
will return to these in Ch. 8. Finally, spoken sentenceshave variouskindsof disﬂuen-
cies(hesitations,repairs,restarts,etc)tobe discussed below.
12.8.1 Disﬂuencies and Repair

Perhaps the most salient syntactic feature that distinguis hes spoken and written lan-
guageistheclassofphenomenaknownindividuallyasdisﬂue nciesandcollectivelyas
thephenomenonofrepair.
Disﬂuencies include the use of the words uhandum, word repetitions, restarts, Restarts
andwordfragments. TheATISsentencein Fig.12.19showsexa mplesofa restartand
the use of uh. The restart here occurs when the speaker starts by asking fo rone-way
ﬂights. andthenstopsandcorrectsherself,restartingandasking aboutone-wayfares .

Figure12.19 Anexample of a disﬂuency (afterShriberg(1994)); terminol ogy is from Levelt(1983)).

DRAFTSection12.8. SpokenLanguageSyntax 423
The segment one-way ﬂights is referred to as the reparandum , and the replacing Reparandum
sequence one-wayfares isreferredtoasthe repair. Theinterruptionpoint ,wherethe Repair
Interruptionpoint speakerbreaksoffthe originalword sequence,hereoccursr ightafter the word ﬂights.
Intheeditingphaseweseewhatareoftencalled editterms ,suchasyouknow ,Imean, Editterms
uh,andum.
The words uhandum(sometimes called ﬁlled pauses orﬁllers) are generally Filledpauses
treatedlikeregularwordsinspeechrecognitionlexiconsa ndgrammars.
Incomplete words like wha-andbetwee-in Fig. 12.18 are known as fragments . Fragments
Fragments are extremely problematic for speech recognitio n systems, since they are
often incorrectly attached to previous or following words, resulting in word misseg-
mentation.
Disﬂuencies are very common. One count in the Switchboard Tr eebank corpus
foundthat37%ofthesentenceswithmorethantwowordswered isﬂuentinsomeway.

Indeed,the word uhisoneofthemostfrequentwordsin Switchboard.
For applications like speech understanding, where our goal is to build a meaning
fortheinputsentence,itmaybeusefultodetecttheseresta rtsinordertoeditoutwhat
the speaker probably considered the “corrected” words. For example in the sentence
above,ifwe coulddetectthattherewas a restart,we couldju st delete thereparandum,
andparsetheremainingpartsofthesentence:
DoesAmericanairlinesofferanyone-wayﬂightsuh one-wayfaresfor160
dollars?

dollars?
Howdodisﬂuenciesinteractwiththeconstituentstructure ofthesentence? Hindle
(1983)showedthattherepairoftenhasthesamestructureas theconstituentjustbefore
theinterruptionpoint. Thusintheexampleabove,therepai risanNP,asisthereparan-
dum. This means that if it is possible to automatically ﬁnd th e interruptionpoint, it is
also oftenpossibletoautomaticallydetecttheboundaries ofthereparandum.
There are other interactions between disﬂuencies and synta ctic structure. For ex-

dollars?
ample when there is a disﬂuency immediately after a subject N P, the repair always
repeats the subject but not the preceding discourse marker. If the repair happensafter
an auxiliary or main verb, the verb and subject are (almost) a lways recycled together
(FoxandJasperson,1995).
12.8.2 Treebanks forSpoken Language
Treebanks for spoken corpora like Switchboard use an augmen ted notation to deal
with spoken language phenomena like disﬂuencies. Fig. 12.2 0 shows the parse tree

for Switchboard sentence (12.17). This sentence shows how t he Treebank marks dis-
ﬂuencies; square brackets are used to separate out the entir e repair area, including
the reparandum, editing phase, and the repair. The plus symb ol marks the end of the
reparandum.
(12.17)ButI don’thave[any,+ {Fuh,}any] realidea

DRAFT424 Chapter 12. FormalGrammarsofEnglish
S
CC
ButNP-SBJ
PRP
IVP
VBP
doRB
n’tVP
VB
haveEDITED
RM
-DFL-
[NP-UNF
DT
any,
,IP
-DFL-
+INTJ
UH
uh,
,NP
DT
anyRS
-DFL-
]JJ
realNN
idea.
.
Figure12.20 PennTreebankIIIparsetreeforaSwitchboardsentence, sho winghow thedisﬂuencyinformationis
represented in the parse tree. Note the .EDITEDnode, with th e .RM and .RS nodes marking the beginning and end
of the repair portion, and the use of the ﬁlledpause uh.
12.9 GrammarsandHumanProcessing
Do people use context-free grammars in their mental process ing of language? It has
proved very difﬁcult to ﬁnd clear-cut evidence that they do. For example, some early
experimentsaskedsubjectstojudgewhichwordsinasentenc eweremorecloselycon-
nected (Levelt, 1970), ﬁnding that their intuitive groupin gs corresponded to syntactic
constituents. Other experimenters examined the role of con stituents in auditory com-
prehensionby havingsubjectslisten to sentenceswhile als o listening to short “clicks”

at different times. Fodor and Bever (1965) found that subjec ts often mis-heard the
clicks as if they occurred at constituent boundaries. They a rgued that the constituent
was thus a “perceptual unit” which resisted interruption. U nfortunately there were
severe methodological problems with the click paradigm (se e e.g., Clark and Clark
(1977)foradiscussion).
A broader problem with all these early studies is that they do not control for the

fact that constituents are often semantic units as well as sy ntactic units. Thus, as will
be discussed further in Ch. 18, a single odd block is a constituent (an NP) but also a
semantic unit (an object of type BLOCKwhich has certain properties). Thus experi-
ments which show that people notice the boundaries of consti tuents could simply be
measuringasemanticratherthana syntacticfact.
Thusit is necessaryto ﬁndevidencefora constituentwhichi snota semanticunit.
Furthermore, since there are many non-constituent-based t heories of grammar based

onlexicaldependencies,it isimportanttoﬁndevidencetha tcannotbeinterpretedasa
lexicalfact; thatis, evidenceforconstituencythatisnotbasedon particularwords.
One series of experiments arguing for constituency has come from Kathryn Bock
and her colleagues. Bock and Loebell (1990), for example, av oided all these earlier
pitfallsby studyingwhethera subject whouses a particular syntacticconstituent(e.g.,

DRAFTSection12.10. Summary 425
a verb-phraseof a particular type, like V NP PP), is more likely to use the constituent
infollowingsentences. Inotherwords,theyaskedwhetheru seofaconstituent primes
itsusein subsequentsentences. Aswesaw inpreviouschapte rs,primingisa common
way to test for the existence of a mental structure. Bock and L oebell relied on the
Englishditransitive alternation . A ditransitive verb is one like givewhich can take
twoarguments:
(12.18)Thewealthywidowgave[ NPthechurch][ NPherMercedes].
The verb giveallows another possible subcategorization frame, called a preposi-
tionaldative inwhichtheindirectobjectisexpressedasaprepositional phrase:
(12.19)Thewealthywidowgave[ NPherMercedes][ PPtothechurch].
As we discussed on page 406, many verbs other than givehave such alternations Alternations
(send,sell,etc.;seeLevin(1993)forasummaryofmanydifferentalter nationpatterns).
Bock and Loebell relied on these alternationsby giving subj ects a picture, and asking

them to describe it in one sentence. The picture was designed to elicit verbs like give
orsellby showing an event such as a boy handing an apple to a teacher. Since these
verbsalternate,subjectsmight,forexample,say The boygavetheappleto theteacher
orTheboygavethe teacheranapple .
Before describing the picture, subjects were asked to read a n unrelated “priming”
sentence out loud; the priming sentences either had V NP NP orV NP PP structure.
Crucially, while these priming sentences had the same constituent structure as the da-

tive alternation sentences, they did not have the same semantics . For example, the
primingsentencesmightbeprepositional locatives,ratherthan datives:
(12.20)IBMmoved[ NPabiggercomputer][ PPtothe Searsstore].
Bock and Loebell foundthat subjects who had just read a V NP PP sentence were
more likely to use a V NP PP structure in describing the picture. This suggested that
the use of a particular constituent primedthe later use of that constituent, and hence
thattheconstituentmustbementallyrepresentedinordert oprimeandbe primed.

In more recent work, Bock and her colleagueshave continuedt o ﬁnd evidencefor
thiskindofconstituencystructure.
12.10 Summary
Thischapterhasintroducedanumberoffundamentalconcept sinsyntaxviathe context-
freegrammar .
•Inmanylanguages,groupsofconsecutivewordsactasagroup oraconstituent ,
whichcanbemodeledby context-freegrammars (alsoknownas phrase-structure
grammars ).
•A context-free grammar consists of a set of rulesorproductions , expressed
overa set of non-terminal symbolsand a set of terminal symbols. Formally,a

particular context-freelanguage isthesetofstringswhichcanbe derivedfrom
aparticular context-freegrammar .

DRAFT426 Chapter 12. FormalGrammarsofEnglish
•Agenerativegrammar isatraditionalnameinlinguisticsforaformallanguage
whichisusedto modelthegrammarofanaturallanguage.
•There are many sentence-level grammatical constructions i n English; declar-
ative,imperative ,yes-no-question , andwh-question are four very common
types,whichcanbemodeledwithcontext-freerules.
•An English noun phrase can have determiners ,numbers ,quantiﬁers , and
adjectivephrases precedingthe headnoun ,whichcanbefollowedbyanumber
ofpostmodiﬁers ;gerundive VPs,inﬁnitives VPs,and pastparticipial VPsare
commonpossibilities.
•Subjects inEnglish agreewiththemainverbin personandnumber.
•Verbs can be subcategorized by the types of complements they expect. Sim-
ple subcategoriesare transitive andintransitive ; most grammarsinclude many
morecategoriesthanthese.
•The correlate of sentences in spoken language are generally called utterances .
Utterances may be disﬂuent, containing ﬁlled pauses likeumanduh,restarts,
andrepairs.

•Treebanks of parsed sentences exist for many genres of English and for m any
languages. Treebankscanbesearchedusingtree-searchtoo ls.
•Anycontext-freegrammarcanbeconvertedto Chomskynormalform ,inwhich
theright-hand-sideofeachrulehaseithertwonon-termina lsorasingleterminal.
•Context-free grammars are more powerful than ﬁnite-state a utomata, but it is
nonethelesspossibleto approximate a context-freegrammarwitha FSA.
•There is some evidence that constituency plays a role in the h uman processing
oflanguage.

“den sprachlichen Ausdruck f¨ ur die willk¨ urliche Glieder ung einer Gesammt-
vorstellungin ihrein logischeBeziehungzueinandergeset ztenBestandteile”
“the linguistic expression for the arbitrary division of a t otal idea into its con-
stituentpartsplacedin logicalrelationsto oneanother”
Wundt’s(1900:240)deﬁnitionofthesentence;theoriginof
the ideaofphrasalconstituency,citedinPercival(1976).
According to Percival (1976), the idea of breaking up a sente nce into a hierarchy

of constituentsappearedin the V¨olkerpsychologie of the groundbreakingpsychologist
Wilhelm Wundt (Wundt, 1900). Wundt’s idea of constituency w as taken up into lin-
guistics by LeonardBloomﬁeld in his early book An Introductionto the Study of Lan-
guage(Bloomﬁeld,1914). Bythetimeofhislaterbook Language (Bloomﬁeld,1933),
what was then called “immediate-constituentanalysis” was a well-establishedmethod
ofsyntacticstudyintheUnitedStates. Bycontrast,tradit ionalEuropeangrammar,dat-

DRAFTSection12.10. Summary 427
ingfromtheClassicalperiod,deﬁnedrelationsbetween wordsratherthanconstituents,
andEuropeansyntacticiansretainedthisemphasisonsuch dependency grammars.
AmericanStructuralismsawanumberofspeciﬁcdeﬁnitionso ftheimmediatecon-
stituent,couchedintermsoftheirsearchfora“discoveryp rocedure”;amethodological
algorithmfordescribingthe syntax of a language. In genera l,these attempt to capture
the intuition that “The primary criterion of the immediate c onstituent is the degree in
which combinations behave as simple units” (Bazell, 1966, p . 284). The most well-
knownofthespeciﬁcdeﬁnitionsisHarris’ideaofdistribut ionalsimilaritytoindividual
units,withthe substitutability test. Essentially,themethodproceededbybreakingupa
constructionintoconstituentsbyattemptingtosubstitut esimplestructuresforpossible
constituents—if a substitution of a simple form, say man, was substitutable in a con-
structionforamorecomplexset(like intenseyoungman ),thentheform intenseyoung

manwas probablya constituent. Harris’s test was the beginning of the intuition that a
constituentisa kindofequivalenceclass.
The ﬁrst formalization of this idea of hierarchical constit uency was the phrase-
structure grammar deﬁned in Chomsky (1956), and further expanded upon (and
argued against) in Chomsky (1957) and Chomsky (1975). From t his time on, most
generative linguistic theories were based at least in part o n context-free grammars
or generalizations of them (such as Head-Driven Phrase Stru cture Grammar (Pollard

andSag,1994),Lexical-FunctionalGrammar(Bresnan,1982 ),GovernmentandBind-
ing (Chomsky, 1981), and Construction Grammar (Kay and Fill more, 1999), inter
alia); many of these theories used schematic context-free t emplates known as X-bar
schemata whichalso reliedonthe notionofsyntactichead. X-bar schemata
Shortly after Chomsky’s initial work, the context-free gra mmar was rediscovered
by Backus (1959) and independently by Naur et al. (1960) in th eir descriptions of
the ALGOL programming language; Backus (1996) noted that he was inﬂuenced by

the productions of Emil Post and that Naur’s work was indepen dent of his (Backus’)
own. (Recall the discussion on page 13 of multiple invention in science.) After this
early work, a great number of computational models of natura l language processing
were based on context-free grammars because of the early dev elopment of efﬁcient
algorithmstoparsethese grammars(see Ch.13).
Aswehavealreadynoted,grammarsbasedoncontext-freerul esarenotubiquitous.
VariousclassesofextensionstoCFGsaredesignedspeciﬁca llytohandlelong-distance

dependencies. We noted earlier that some grammars treat lon g-distance-dependent
items as being related semantically but not syntactically; the surface syntax does not
represent the long-distance link (Kay and Fillmore, 1999; C ulicover and Jackendoff,
2005). But there are alternatives. One extendedformalism i sTree Adjoining Gram-
mar(TAG) (Joshi, 1985). The primary data structure in Tree Adjo ining Grammar
is the tree, rather than the rule. Trees come in two kinds; initial trees andauxiliary

trees. Initial trees might, for example, represent simple senten tial structures, while
auxiliary trees are used to add recursion into a tree. Trees a re combined by two oper-
ationscalled substitution andadjunction . Theadjunctionoperationisusedto handle
long-distance dependencies. See Joshi (1985) for more deta ils. An extension of Tree
Adjoining Grammar called Lexicalized Tree Adjoining Gramm ars will be discussed
in Ch. 14. Tree Adjoining Grammar is a member of the family of mildly context-

sensitive languages to beintroducedinCh.15.

DRAFT428 Chapter 12. FormalGrammarsofEnglish
We mentioned on page 408 another way of handling long-distan ce dependencies,
based on the use of empty categories and co-indexing. The Pen n Treebank uses this
model,whichdraws(invariousTreebankcorpora)fromtheEx tendedStandardTheory
andMinimalism(Radford,1997).
Representativeexamplesof grammarsthat are based on word r elationsrather than
constituencyincludethedependencygrammarof Mel’ˇ cuk(1 979),the WordGrammar
ofHudson(1984),andthe ConstraintGrammarofKarlssonet a l.(1995b).
There are a variety of algorithms for building a regular gram mar which approxi-
mates a CFG (Pereira and Wright, 1997; Johnson,1998a; Lange ndoenand Langsam,
1987;Nederhof,2000;MohriandNederhof,2001).
Readers interested in the grammar of English should get one o f the three large
reference grammars of English: Huddleston and Pullum (2002 ), Biber et al. (1999),
andQuirket al.(1985),AnotherusefulreferenceisMcCawle y(1998).

Therearemanygoodintroductorytextbooksonsyntaxfromdi fferentperspectives.
Sagetal.(2003)isanintroductiontosyntaxfroma generative perspective,focusingon generative
the use of phrase-structure,uniﬁcation, and the type-hier archyin Head-DrivenPhrase
Structure Grammar. Van Valin and La Polla (1997) is an introd uction from a func-
tionalperspective, focusing on cross-linguistic data and on the f unctional motivation functional
forsyntacticstructures.
See Bach (1988) for an introduction to basic categorial gram mar. Various exten-

sionstocategorialgrammarsarepresentedinLambek(1958) ,Dowty(1979),andAdes
andSteedman(1982)interalia;theotherpapersinOehrleet al.(1988)giveasurveyof
extensions. Combinatory categorial grammar is presented i n Steedman (1989, 2000);
seeSteedmanandBaldridge(2007)foratutorialintroducti on. SeeCh.18foradiscus-
sionofsemanticcomposition.
Exercises
12.1Drawtree structuresforthefollowingATISphrases:
a. Dallas
b. fromDenver
c. afterﬁvep.m.
d. arrivinginWashington
e. earlyﬂights
f. all redeyeﬂights
g. onThursday
h. a one-wayfare
i. anydelaysin Denver

Exercises
a. Dallas
12.2Drawtree structuresforthefollowingATISsentences:
a. DoesAmericanairlineshavea ﬂightbetweenﬁvea.m. andsix a.m.

DRAFTSection12.10. Summary 429
b. I wouldlike toﬂyonAmericanairlines.
c. Please repeatthat.
d. DoesAmerican487haveaﬁrst classsection?
e. I needtoﬂy betweenPhiladelphiaandAtlanta.
f. What isthefarefromAtlantato Denver?
g. IsthereanAmericanairlinesﬂightfromPhiladelphiatoDa llas?
12.3Augmentthegrammarrulesonpage403tohandlepronouns. Dea lproperlywith
personandcase.
12.4Modify the noun phrase grammar of Sections 12.3.3–12.3.4to correctly model
massnounsandtheiragreementproperties
12.5Howmanytypesof NPswouldtheruleonpage400expandtoifwedidn’tallow
parenthesesinourgrammarformalism?
12.6Assume a grammar that has many VPrules for different subcategorizations, as
expressed in Sec. 12.3.5, and differently subcategorized v erb rules like Verb-
with-NP-complement . How would the rule for post-nominal relative clauses
(12.7) need to be modiﬁed if we wanted to deal properly with ex amples like
the earliest ﬂight that you have ? Recall that in such examples the pronoun that

istheobjectoftheverb get. Yourrulesshouldallowthisnounphrasebutshould
correctlyruleouttheungrammaticalS *Iget.
12.7Does your solution to the previousproblem correctly model t he NPthe earliest
ﬂight that I can get ? How about the earliest ﬂight that I think my mother wants
metobookforher ? Hint: thisphenomenoniscalled long-distancedependency .
12.8WriterulesexpressingtheverbalsubcategoryofEnglishau xiliaries;forexample
youmighthavea rule verb-with-bare-stem-VP-complement →can.

12.9NPs likeFortune’s ofﬁce ormy uncle’s marks are called possessive orgenitive possessive
genitive nounphrases. A possessive nounphrase canbe modeledbytrea tingthe sub-NP
likeFortune’s ormy uncle’s as a determiner of the following head noun. Write
grammarrules for English possessives. You may treat ’sas if it were a separate
word(i.e.,asiftherewerealwaysaspacebefore ’s).
12.10Page 397discussed the needfora Wh-NPconstituent. The simplest Wh-NPis
one of the Wh-pronouns (who, whom, whose, which ). The Wh-words whatand

whichcan be determiners: which four will you have? ,what credit do you have
withthe Duke? Write rulesforthedifferenttypesof Wh-NPs.
12.11Writeanalgorithmforconvertinganarbitrarycontext-fre egrammarintoChom-
skynormalform.

Chapter13
ParsingwithContext-FreeGrammars
There are and can exist but two ways of investigating and disc overing truth. The one
hurries on rapidly from the senses and particulars to the mos t general axioms, and from
them...derives and discovers the intermediate axioms. The other constructs its axioms
from the senses and particulars, by ascending continually a nd gradually, till it ﬁnally
arrives at the most general axioms.
FrancisBacon, Novum Organum BookI.19(1620)
We deﬁned parsing in Ch. 3 as a combination of recognizing an i nput string and as-

signing a structure to it. Syntactic parsing, then, is the ta sk of recognizing a sentence
and assigning a syntactic structure to it. This chapter focu ses on the kind of struc-
tures assigned by context-free grammars of the kind describ ed in Ch. 12. However,
since they are based on a purely declarative formalism, cont ext-free grammars don’t
specifyhowthe parse tree for a given sentence should be computed. We’ll therefore
need to specify algorithmsthat employthese grammarsto pro ducetrees. This chapter
presents three of the most widely used parsing algorithms fo r automatically assigning

a completecontext-free(phrasestructure)treeto aninput sentence.
These kinds of parse trees are directly useful in applicatio ns such as grammar
checking in word-processing systems: a sentence which cannot be pars ed may have
grammatical errors (or at least be hard to read). More typica lly, however, parse trees
serve as an important intermediate stage of representation forsemantic analysis (as
we will see in Ch. 18), and thus play an important role in appli cations like question

answering andinformationextraction . Forexample,to answerthequestion
Whatbookswere written byBritish womenauthorsbefore1800 ?
we’ll need to know that the subject of the sentence was what books and that the by-
adjunct was British women authors to help us ﬁgure out that the user wants a list of
books(andnota list ofauthors).
Beforepresentinganyparsingalgorithms,webeginbydescr ibingsomeofthefac-
tors that motivate the standard algorithms. First, we revis it thesearch metaphor for

parsing and recognition, which we introduced for ﬁnite-sta te automata in Ch. 2, and
talk about the top-down andbottom-up search strategies. We then discuss how the
ambiguity problem rears its head again in syntactic process ing, and how it ultimately
makessimplistic approachesbasedonbacktrackinginfeasi ble.
ThesectionsthatfollowthenpresenttheCocke-Kasami-You nger(CKY)algorithm
(Kasami, 1965; Younger, 1967), the Earley algorithm (Earle y, 1970), and the chart

parsing approach (Kay, 1982; Kaplan, 1973). These approach es all combine insights
frombottom-upandtop-downparsingwithdynamicprogrammi ngtoefﬁcientlyhandle

DRAFT432 Chapter 13. Parsingwith Context-FreeGrammars
S→NPVP Det→that|this|a
S→AuxNPVP Noun→book|ﬂight|meal|money
S→VP Verb→book|include|prefer
NP→Pronoun Pronoun→I|she|me
NP→Proper-Noun Proper-Noun→Houston|NWA
NP→DetNominal Aux→does
Nominal→Noun Preposition→from|to|on|near|through
Nominal→NominalNoun
Nominal→NominalPP
VP→Verb
VP→VerbNP
VP→VerbNPPP
VP→VerbPP
VP→VPPP
PP→PrepositionNP
Figure13.1 TheL1miniature Englishgrammar andlexicon.
complex inputs. Recall that we’ve already seen several appl ications of dynamic pro-
grammingalgorithmsinearlierchapters—Minimum-Edit-Di stance,Viterbi,Forward.
Finally, we discuss partial parsing methods , for use in situations where a superﬁcial
syntacticanalysisofaninputmaybesufﬁcient.
13.1 ParsingasSearch
Chapters 2 and 3 showed that ﬁnding the right path through a ﬁn ite-state automaton,
or ﬁnding the right transduction for an input, can be viewed a s a search problem. For
ﬁnite-state automata, the search is through the space of all possible paths through a

machine. Insyntacticparsing,theparsercanbeviewedasse archingthroughthespace
of possible parse trees to ﬁnd the correct parse tree for a giv en sentence. Just as the
search space of possible paths was deﬁned by the structure of an automaton, so the
search space of possible parse trees is deﬁned by a grammar. C onsider the following
ATISsentence:
(13.1)Bookthatﬂight.
Fig. 13.1introducesthe L1grammar,which consistsof the L0grammarfromthe
last chapter with a few additional rules. Giventhis grammar ,the correct parse tree for

thisexamplewouldbetheoneshownin Fig.13.2.
How can we use L1to assign the parse tree in Fig. 13.2 to this example? The
goal of a parsing search is to ﬁnd all the trees whose root is th e start symbol Sand
which cover exactly the words in the input. Regardless of the search algorithm we
choose, there are two kinds of constraints that should help g uide the search. One set
of constraints comes from the data, that is, the input senten ce itself. Whatever else
is true of the ﬁnal parse tree, we know that there must be three leaves, and that they

DRAFTSection13.1. ParsingasSearch 433
S
VP
Verb
BookNP
Det
thatNominal
Noun
ﬂight
Figure13.2 The parse tree for the sentence Book that ﬂight according togrammar L1.
mustbethewords book,that,andﬂight. Thesecondkindofconstraintcomesfromthe
grammar. We know that whatever else is true of the ﬁnal parse t ree, it must have one
root,whichmustbethestart symbol S.
These two constraints, invoked by Bacon at the start of this c hapter, give rise to
the twosearchstrategiesunderlyingmostparsers: top-down orgoal-directedsearch ,
andbottom-up ordata-directed search . These constraints are more than just search
strategies. They reﬂect two important insights in the weste rn philosophical tradition:
therationalist tradition, which emphasizes the use of prior knowledge, and theem- Rationalist
piricisttradition,whichemphasizesthedatainfrontofus. Empiricist
13.1.1 Top-DownParsing
Atop-down parser searches for a parse tree by trying to build from the ro ot nodeS Top-down

down to the leaves. Let’s consider the search space that a top -down parser explores,
assuming for the moment that it builds all possible trees in p arallel. The algorithm
starts by assuming that the input can be derived by the design ated start symbol S. The
next step is to ﬁnd the tops of all trees which can start with S, by looking for all the
grammarruleswith Sontheleft-handside. InthegrammarinFig.13.1,therearet hree
rules that expand S, so the second ply, or level, of the search space in Fig. 13.1.1 has Ply
threepartialtrees.

We next expand the constituents in these three new trees, jus t as we originally
expanded S.Theﬁrsttreetellsustoexpectan NPfollowedbya VP,thesecondexpects
anAuxfollowedbyan NPandaVP,andthethirda VPbyitself. Toﬁtthesearchspace
on the page, we have shown in the third ply of Fig. 13.1.1 only a subset of the trees
that result from the expansion of the left-most leaves of eac h tree. At each ply of the
searchspaceweusetheright-handsidesoftherulestoprovi denewsetsofexpectations
for the parser, which are then used to recursively generate t he rest of the trees. Trees

are grown downward until they eventually reach the part-of- speech categories at the
bottom of the tree. At this point, trees whose leaves fail to m atch all the words in the
input can be rejected, leaving behind those trees that repre sent successful parses. In
Fig. 13.1.1, only the ﬁfth parse tree in the third ply (the one which has expanded the
ruleVP→VerbNP)will eventuallymatchtheinputsentence Bookthatﬂight .

DRAFT434 Chapter 13. Parsingwith Context-FreeGrammars
S
S S S
NP VP Aux NP VP VP
S S S S S S
NP VP NP VP Aux NP VP Aux NP VP VP VP
Det Nom PropN Det Nom PropN V NP V
Figure13.3 An expanding top-down search space. Each ply is created by ta king each tree from the previous ply,
replacingtheleftmostnon-terminalwitheachofitspossib leexpansions,andcollectingeachofthesetreesintoanew
ply.
13.1.2 Bottom-Up Parsing
Bottom-up parsing is the earliest known parsing algorithm (it was ﬁrst suggested by Bottom-up
Yngve(1955)),andisusedintheshift-reduceparserscommo nforcomputerlanguages
(Ahoand Ullman,1972). In bottom-upparsing,the parser sta rtswith the wordsof the
input, and tries to build trees from the words up, again by app lying rules from the
grammaroneatatime. Theparseissuccessfuliftheparsersu cceedsinbuildingatree
rootedin the start symbol Sthat coversall of the input. Fig. 13.1.2shows the bottom-
up search space, beginning with the sentence Book that ﬂight . The parser begins by

looking up each input word in the lexicon and building three p artial trees with the
part-of-speech for each word. But the word bookis ambiguous; it can be a noun or a
verb. Thus the parser must consider two possible sets of tree s. The ﬁrst two plies in
Fig.13.1.2showthisinitial bifurcationofthesearchspac e.
Eachofthetreesinthesecondplyisthenexpanded. Inthepar seontheleft(theone
in which bookis incorrectly considered a noun), the Nominal→Nounrule is applied

to both of the nouns ( bookandﬂight). This same rule is also applied to the sole noun
(ﬂight)ontheright,producingthe treesonthethirdply.
Ingeneral,theparserextendsoneplytothenextbylookingf orplacesintheparse-
in-progress where the right-hand side of some rule might ﬁt. This contrasts with the
earlier top-down parser, which expanded trees by applying r ules when their left-hand
side matchedanunexpandednon-terminal.
Thus in the fourth ply, in the ﬁrst and third parse, the sequen ceDetNominal is
recognizedastheright-handsideofthe NP→DetNominal rule.

Intheﬁfthply,theinterpretationof bookasanounhasbeenprunedfromthesearch
space. This is because this parse cannotbe continued: there is no rule in the grammar
with the right-hand side NominalNP . The ﬁnal ply of the search space (not shown in
Fig.13.1.2)containsthecorrectparse(seeFig. 13.2).

DRAFTSection13.1. ParsingasSearch 435
Book that ﬂight
Noun Det Noun Verb Det Noun
Book that ﬂight Book that ﬂight
Nominal Nominal Nominal
NP NP
Nominal Nominal VP Nominal Nominal
Noun Det Noun Verb Det Noun Verb Det Noun
Book that ﬂight Book that ﬂight Book that ﬂight
VP
VP NP NP
Nominal Nominal
Verb Det Noun Verb Det Noun
Figure13.4 Anexpanding bottom-upsearchspaceforthesentence Bookthatﬂight . Thisﬁguredoesnotshowthe
ﬁnal phase of the search with the correct parse tree (see Fig. 13.2). Make sure you understand how that ﬁnal parse
tree follows from the searchspace inthisﬁgure.
13.1.3 Comparing Top-Downand Bottom-Up Parsing
Each of these two architectures has its own advantages and di sadvantages. The top-
down strategy never wastes time exploring trees that cannot result in an S, since it
begins by generating just those trees. This means it also nev er explores subtrees that

cannotﬁnd a place in some S-rootedtree. In the bottom-upstrategy,by contrast, trees
that have no hope of leading to an S, or ﬁtting in with any of their neighbors, are
generatedwith wildabandon.
The top-down approach has its own inefﬁciencies. While it do es not waste time
with trees that do not lead to an S, it doesspend considerableeffort on Strees that are
not consistent with the input. Note that the ﬁrst four of the s ix trees in the third ply in
Fig.13.1.1allhaveleftbranchesthatcannotmatchtheword book. Noneofthesetrees

DRAFT436 Chapter 13. Parsingwith Context-FreeGrammars
S
NP
Pronoun
IVP
Verb
shotNP
Det
anNominal
Nominal
Noun
elephantPP
inmypajamasS
NP
Pronoun
IVP
VP
Verb
shotNP
Det
anNominal
Noun
inmypajamas
Figure13.5 Two parse trees for an ambiguous sentence. Parse (a) corresp onds to the humorous reading in which
the elephant isinthe pajamas, parse (b)tothe reading inwhi chCaptainSpaulding didthe shooting inhis pajamas.
could possibly be used in parsing this sentence. This weakne ss in top-down parsers
arises from the fact that they generate trees before ever exa mining the input. Bottom-
upparsers,onthe otherhand,neversuggesttreesthatare no tat least locallygrounded
inthe input.
13.2 Ambiguity
OnemorningIshotanelephantinmypajamas. Howhegotintomy paja-
masIdon’tknow.
Groucho Marx, AnimalCrackers , 1930
Ambiguityis perhapsthe most seriousproblemfacedbyparse rs. Ch. 5introduced
thenotionsof part-of-speechambiguity andpart-of-speechdisambiguation . Inthis

section we introduce a new kind of ambiguity, which arises in the syntactic structures
used in parsing, called structural ambiguity . Structural ambiguity occurs when theStructuralambiguity
grammar assigns more than one possible parse to a sentence. G roucho Marx’s well-
knownline as Captain Spauldingis ambiguousbecause the phr asein my pajamas can
bepartofthe NPheadedby elephantoroftheverb-phraseheadedby shot.
Structural ambiguity, appropriately enough, comes in many forms. Two common
kindsofambiguityare attachmentambiguity andcoordinationambiguity .

Asentencehasan attachmentambiguity ifaparticularconstituentcanbeattachedAttachmentambiguity
totheparsetreeatmorethanoneplace. TheGrouchoMarxsent enceaboveisanexam-
pleofPP-attachmentambiguity. Variouskindsofadverbialphrases arealso subjectto
this kindof ambiguity. For examplein the followingexample the gerundive- VPﬂying
to Pariscan bepart of a gerundivesentence whosesubject is the Eiffel Tower orit can
beanadjunctmodifyingthe VPheadedby saw:
(13.2)We saw theEiffelTowerﬂyingto Paris.

Incoordinationambiguity therearedifferentsetsofphrasesthatcanbeconjoinedCoordinationambiguity

DRAFTSection13.2. Ambiguity 437
by a conjunctionlike and. For example,the phrase old men and women can be brack-
eted as[old [men and women]] , referring to old menandold women , or as[old men]
and[women] , inwhichcaseit isonlythemenwhoareold.
These ambiguities combine in complex ways in real sentences . A program that
summarized the news, for example, would need to be able to par se sentences like the
followingfromtheBrowncorpus:
(13.3)PresidentKennedytodaypushedasideotherWhiteHousebusi nesstodevote
allhistimeandattentiontoworkingontheBerlincrisisadd resshewilldeliver
tomorrownightto theAmericanpeopleovernationwidetelev isionandradio.
This sentence has a number of ambiguities, although since th ey are semantically
unreasonable, it requires a careful reading to see them. The last noun phrase could
be parsed [nationwide [television and radio]] or[[nationwide television] and radio] .
Thedirectobjectof pushedaside shouldbe otherWhiteHousebusiness butcouldalso

be the bizarre phrase [other White House business to devote all his time and attent ion
to working] (i.e., a structure like Kennedy afﬁrmed [his intention to propose a new
budget to address the deﬁcit] ). Then the phrase on the Berlin crisis address he will
delivertomorrownighttotheAmericanpeople couldbeanadjunctmodifyingtheverb
pushed. APPlikeovernationwidetelevisionandradio couldbeattachedtoanyofthe
higherVPsorNPs(e.g.,it couldmodify peopleornight).
Thefact that thereare manyunreasonableparsesfor natural lyoccurringsentences

is an extremely irksome problemthat affects all parsers. Ul timately, most natural lan-
guage processing systems need to be able to choose the correc t parse from the multi-
tude of possible parses via a process known as syntactic disambiguation . Unfortu-Syntactic
disambiguation
nately, effectivedisambiguationalgorithmsgenerallyre quire statistical, semantic, and
pragmaticknowledgenotreadilyavailableduringsyntacti cprocessing(techniquesfor
makinguseofsuchknowledgewill beintroducedlater,inCh. 14andCh.18).

Lacking such knowledge, we are left with the choice of simply returning all the
possibleparsetreesfora giveninput. Unfortunately,gene ratingall the possibleparses
from robust, highly ambiguous, wide-coverage grammars suc h as the Penn Treebank
grammar described in Ch. 12 is problematic. The reason for th is lies in the poten-
tially exponential number of parses that are possible for ce rtain inputs. Consider the
followingATISexample:
(13.4)ShowmethemealonFlightUA 386fromSanFranciscotoDenver.

Therecursive VP→VPPPandNominal→NominalPP rulesconspirewiththethree
prepositional phrases at the end of this sentence to yield a t otal of 14 parse trees for
thissentence. Forexample fromSanFrancisco couldbepartofthe VPheadedby show
(whichwouldhavethebizarreinterpretationthattheshowi ngwashappeningfromSan
Francisco). Church and Patil (1982) showed that the number o f parses for sentences
ofthistypegrowsexponentiallyat thesamerateasthenumbe rofparenthesizationsof
arithmeticexpressions.

Evenif a sentence isn’t ambiguous(i.e. it doesn’thave more than one parse in the
end), it can be inefﬁcient to parse due to local ambiguity . Local ambiguity occurs Local ambiguity
when some part of a sentence is ambiguous, that is, has more th an one parse, even
if the whole sentence is not ambiguous. For example the sente nceBook that ﬂight is

DRAFT438 Chapter 13. Parsingwith Context-FreeGrammars
S
VP
Verb
showNP
Pronoun
meNP
Det
theNominal
Nominal
Noun
mealPP
P
onNP
Nominal
Nominal
Nominal
FlightUA 386PP
from San FranciscoPP
toDenver
Figure13.6 A reasonable parse for Ex. 13.4.
unambiguous, but when the parser sees the ﬁrst word Book, it cannot know if it is a
verbora noununtillater. Thusit mustconsiderbothpossibl eparses.
13.3 SearchintheFaceofAmbiguity
Tofullyunderstandtheproblemthatlocalandglobalambigu ityposeforsyntacticpars-
ing, let’s return to our earlier description of top-down and bottom-up parsing. There
we made the simplifying assumption that we could explore all possible parse trees in
parallel. Thus each ply of the search in Fig. 13.1.1 and Fig. 1 3.1.2 showed parallel
expansionsof theparse treeson thepreviousplies. Althoug hit is certainlypossible to
implement this method directly, it typically requiresan un realistic amount of memory
tostorethespaceoftreesastheyareconstructed. Thisises peciallytruesincerealistic

grammarshavemuchmoreambiguitythantheminiaturegramma rwe’vebeenusing.
A common alternative approach to exploring complex search- spaces is to use an
agenda-basedbacktrackingstrategysuchasthoseusedtoim plementthevariousﬁnite-
state machines in Chs. 2 and 3. A backtracking approach expan ds the search space
incrementally by systematically exploring one state at a ti me. The state chosen for

DRAFTSection13.4. DynamicProgrammingParsingMethods 439
expansioncan be based on simple systematic strategies such as depth-ﬁrst or breadth-
ﬁrstmethods,oronmorecomplexmethodsthatmakeuseofprob abilisticandsemantic
considerations. When the given strategy arrives at a tree th at is inconsistent with the
input,thesearchcontinuesbyreturningtoanunexploredop tionalreadyontheagenda.
The net effect of this strategy is a parser that single-minde dly pursues trees until they
eithersucceedorfailbeforereturningtoworkontreesgene ratedearlierintheprocess.
Unfortunately, the pervasive ambiguity in typical grammar s leads to intolerable
inefﬁcienciesinanybacktrackingapproach. Backtracking parserswilloftenbuildvalid
treesforportionsof theinput,and thendiscardthem during backtracking,onlyto ﬁnd
thattheyhavetoberebuiltagain. Considerthetop-downbac ktrackingprocessinvolved
inﬁndinga parseforthe NPin(13.5):
(13.5)a ﬂightfromIndianapolisto HoustononNWA

The preferred complete parse shown as the bottom tree in Fig. 13.7. While there are
numerous parses of this phrase, we will focus here on the amou nt of repeated work
expendedonthe pathto retrievingthissinglepreferredpar se.
A typical top-down, depth-ﬁrst, left-to-right backtracki ng strategy leads to small
parsetreesthatfailbecausetheydonotcoveralloftheinpu t. Thesesuccessivefailures
trigger backtracking events which lead to parses that incre mentally cover more and

more of the input. The sequenceof trees attempted on the way t o the correct parse by
thistop-downapproachisshownin Fig.13.7.
This ﬁgure clearly illustrates the kind of reduplicationof work that arises in back-
tracking approaches. Except for its topmost component, eve ry part of the ﬁnal tree
is derived more than once. The work done on this simple exampl e would, of course,
be magniﬁed by any ambiguity introducedat the verb phrase or sentential level. Note
that althoughthisexampleis speciﬁcto top-downparsing,s imilar examplesof wasted

effortexistforbottom-upparsingaswell.
13.4 DynamicProgrammingParsingMethods
Theprevioussectionpresentedsomeoftheproblemsthatafﬂ ictstandardbottom-upor
top-downparsersduetoambiguity. Luckily,thereisasingl eclassofalgorithmswhich
can solve these problems. Dynamic programming once again provides a framework
forsolvingthisproblem,justas it helpeduswiththe Minimu mEditDistance,Viterbi,
andForwardalgorithms. Recallthatdynamicprogrammingap proachessystematically
ﬁll in tablesofsolutionsto sub-problems. When complete,t hetablescontainthesolu-

tiontoall thesub-problemsneededtosolvetheproblemasaw hole.
Inthecaseofparsing,suchtablesareusedtostoresubtrees foreachconstituentsin
theinputasitisdiscovered. Theefﬁciencygainarisesfrom thefactthatthesesubtrees
are discovered once, stored, and then used in all parses call ing for that constituent.
Thissolvesthere-parsingproblem(subtreesarelookedup, notre-parsed)andpartially
solves the ambiguity problem (the dynamic programming tabl e implicitly stores all

possible parses by storing all the constituents with links t hat enable the parses to be
reconstructed). As we mentioned earlier, the three most wid ely used methods are the

DRAFT440 Chapter 13. Parsingwith Context-FreeGrammars
NP
Det
aNominal
Noun
ﬂight...NP
Det
aNominal
Nominal
Noun
ﬂightPP
from Indianapolis...NP
Det
aNominal
Nominal
Nominal
Noun
ﬂightPP
fromIndianapolisPP
toHouston...
NP
Det
aNominal
Nominal
Nominal
Nominal
Noun
ﬂightPP
toHoustonPP
onTWA
Figure13.7 Reduplicated effort caused bybacktracking intop-down par sing.
Cocke-Kasami-Younger(CKY) algorithm,theEarleyalgorit hm,andchartparsing.
13.4.1 CKYParsing
Let’sbeginourinvestigationofCKYalgorithmbyexamining oneofitsmajorrequire-
ments: the grammars used with it must be in Chomsky Normal For m (CNF). Recall
from Ch. 12 that grammars in CNF are restricted to rules of the formA→BC, or
A→w. That is, the right-hand side of each rule must expand to eith er two non-
terminals or to a single terminal. Recall also that restrict ing a grammar to CNF does
notleadtoanylossinexpressivenesssinceanycontext-fre egrammarcanbeconverted

DRAFTSection13.4. DynamicProgrammingParsingMethods 441
into a correspondingCNF grammar that accepts exactly the sa me set of strings as the
originalgrammar. Thissingle restrictiongivesrise to an e xtremelysimple andelegant
table-basedparsingmethod.
ConversiontoCNF
Let’s start with the process of convertinga generic CFG into one represented in CNF.
Assuming we’re dealing with an ǫ-free grammar, there are three situations we need to
address in any generic grammar: rules that mix terminals wit h non-terminals on the
right-handside, rules that have a single non-terminalon th e right, and rules where the
right-handside’slengthisgreaterthantwo.
The remediation for rules that mix terminals and non-termin alsis to simply intro-
duce a new dummy non-terminalthat covers only the original t erminal. For example,
aruleforaninﬁnitiveverbphrasesuchas INF-VP→toVPwouldbereplacedbythe
tworules INF-VP→TOVPandTO→to.
Rules with a single non-terminal on the right are called unit productions . Unit Unit productions

productions are eliminated by rewriting the right-hand sid e of the original rules with
the right-hand side of all the non-unit production rules tha t they ultimately lead to.
More formally, if A∗⇒Bby a chain of one or more unit productions, and B→γis
a non-unit production in our grammar, then we add A→γfor each such rule in the
grammar, and discard all the intervening unit productions. As we’ll see with our toy
grammar, this can lead to a substantial ﬂattening of the grammar, and a consequent

promotionofterminalsto fairlyhighlevelsinthe resultin gtrees.
Rules with right-handsides longer than 2 are normalizedthr ough the introduction
of new non-terminals that spread the longer sequences over s everal new rules. For-
mally,if wehavearulelike
A→BCγ
we replacetheleftmost pairofnon-terminalswitha new non- terminalandintroducea
newproductionresult inthefollowingnewrules.
X1→BC
A→X1γ
Inthecaseoflongerright-handsides,wesimplyiteratethi sprocessuntiltheoffending

A→BCγ
X1→BC
A→X1γ
rule has been replaced by rules of length 2. The choice of repl acing the leftmost pair
of non-terminalsis purelyarbitrary;any systematic schem ethat resultsin binaryrules
wouldsufﬁce.
In our current grammar, the rule S→AuxNPVP would be replaced by the two
rulesS→X1VPandX1→AuxNP.
Theentireconversionprocesscanbe summarizedasfollows:
1. Copyall conformingrulesto thenewgrammarunchanged,
2. Convertterminalswithinrulestodummynon-terminals,
3. Convertunit-productions,
4. Binarizeallrulesandaddtonewgrammar.

DRAFT442 Chapter 13. Parsingwith Context-FreeGrammars
L1Grammar L1in CNF
S→NPVP S→NPVP
S→AuxNPVP S→X1VP
X1→AuxNP
S→VP S→book|include|prefer
S→VerbNP
S→X2PP
S→VerbPP
S→VPPP
NP→Pronoun NP→I|she|me
NP→Proper-Noun NP→TWA|Houston
NP→DetNominal NP→DetNominal
Nominal→Noun Nominal→book|ﬂight|meal|money
Nominal→NominalNoun Nominal→NominalNoun
Nominal→NominalPP Nominal→NominalPP
VP→Verb VP→book|include|prefer
VP→VerbNP VP→VerbNP
VP→VerbNPPP VP→X2PP
X2→VerbNP
VP→VerbPP VP→VerbPP
VP→VPPP VP→VPPP
PP→PrepositionNP PP→PrepositionNP
Figure13.8 L1Grammar and its conversion to CNF. Note that although they ar en’t shown
here all the original lexical entriesfrom L1carry over unchanged as well.
Figure 13.8 shows the results of applying this entire conver sion procedure to the
L1grammar introduced earlier on page 432. Note that this ﬁgure doesn’t show the
original lexical rules; since these original lexical rules are already in CNF, they all
carry over unchangedto the new grammar. Fig. 13.8 does, howe ver,show the various

places where the process of eliminating unit productions ha s, in effect, created new
lexical rules. For example, all the original verbs have been promoted to both VPs and
toSsintheconvertedgrammar.
CKYRecognition
WithourgrammarnowinCNF,eachnon-terminalnodeabovethe part-of-speechlevel
in a parse tree will have exactly two daughters. A simple two- dimensionalmatrix can
be used to encode the structure of an entire tree. More speciﬁ cally, for a sentence of

lengthn, we will be working with the upper-triangularportion of an (n+1)×(n+1)
matrix. Each cell [i,j]in this matrix contains a set of non-terminals that represen t all
theconstituentsthatspanpositions ithroughjoftheinput. Sinceourindexingscheme
begins with 0, it’s natural to think of the indexes as pointin g at the gaps between the
input words (as in 0Book1that2ﬂight3). It follows then that the cell that represents
theentireinputresidesinposition [0,n]in thematrix.
SinceourgrammarisinCNF,thenon-terminalentriesinthet ablehaveexactlytwo

daughtersin the parse. Therefore,for each constituent rep resentedby an entry [i,j]in

DRAFTSection13.4. DynamicProgrammingParsingMethods 443
the table there must be a position in the input, k, where it can be split into two parts
such that i<k<j. Given such a position k, the ﬁrst constituent [i,k]must lie to the
left of entry [i,j]somewhere along row i, and the second entry [k,j]must lie beneath
it, alongcolumn j.
To make this more concrete, consider the following example w ith its completed
parsematrix,showninFig.13.9.
(13.6)BooktheﬂightthroughHouston.
The superdiagonal row in the matrix contains the parts of spe ech for each input word
in the input. The subsequent diagonals above that superdiag onal contain constituents
thatcoverall thespansofincreasinglengthin theinput.
Book the flight through Houston
S, VP, Verb, 
Nominal, 
NounS,VP,X2 S, VP
Det NP NP
Nominal,
NounNominal
Prep PP
NP,
Proper-
Noun[0,1] [0,2] [0,3] [0,4] [0,5]
[1,2] [1,3]
[2,3][1,4]
[2,5][2,4]
[3,4]
[4,5][3,5][1,5]
Figure13.9 Completed parse table for Book the ﬂight through Houston.

Nominal, 
Det NP NP
Nominal,
Prep PP
NP,
Proper-
[3,4]
Given all this, CKY recognition is simply a matter of ﬁlling t he parse table in the
right way. To do this, we’ll proceed in a bottom-up fashion so that at the point where
we are ﬁlling any cell [i,j], the cells containing the parts that could contribute to thi s
entry, (i.e. the cells to the left and the cells below) have al ready been ﬁlled. There are
several ways to do this; as the right side of Fig. 13.9 illustr ates, the algorithm given

in Fig. 13.10 ﬁlls the upper-triangularmatrix a column at a t ime working from left to
right. Each column is then ﬁlled from bottom to top. This sche me guarantees that at
eachpointintimewehavealltheinformationweneed(tothel eft,sinceallthecolumns
to the left have already been ﬁlled, and below since we’re ﬁll ing bottom to top). It
also mirrors on-line parsing since ﬁlling the columnsfrom l eft to right correspondsto
processingeachwordoneat atime.
The outermostloopof the algorithmgivenin Fig. 13.10itera tes overthe columns,

and the second loop iterates over the rows, from the bottom up . The purpose of the
inner-most loop is to rangeover all the places where a substr ing spanning itojin the
input might be split in two. As kranges over the places where the string can be split,
thepairsofcellsweconsidermove,inlockstep,totheright alongrow ianddownalong

DRAFT444 Chapter 13. Parsingwith Context-FreeGrammars
function CKY-P ARSE(words,grammar )returnstable
forj←from1toLENGTH(words)do
table[j−1,j]←{A|A→words[j]∈grammar }
fori←fromj−2downto0do
fork←i+1toj−1do
table[i,j]←table[i,j]∪
{A|A→BC∈grammar ,
B∈table[i,k],
C∈table[k,j]}
Figure13.10 The CKY algorithm
columnj. Fig.13.11illustratesthe generalcase of ﬁllingcell [i,j]. At eachsuchsplit,
thealgorithmconsiderswhetherthecontentsofthetwocell scanbecombinedinaway
that is sanctioned by a rule in the grammar. If such a rule exis ts, the non-terminal on
itsleft-handsideisenteredintothe table.
Fig.13.12showshowtheﬁvecellsofcolumn5ofthetableareﬁ lledaftertheword
Houstonisread. Thearrowspointoutthetwospansthatarebeingused toaddanentry
tothetable. Notethattheactionincell [0,5]indicatesthepresenceofthreealternative
parses for this input, one where the PPmodiﬁes the ﬂight, one where it modiﬁes the
booking,and one that capturesthe secondargumentin the ori ginalVP→VerbNPPP
rule,nowcapturedindirectlywiththe VP→X2PPrule.

Infact,sinceourcurrentalgorithmmanipulates setsofnon-terminalsascellentries,
itwon’tincludemultiplecopiesofthesamenon-terminalin thetable;thesecond Sand
VPdiscoveredwhileprocessing [0,5]wouldhavenoeffect. We’ll revisitthisbehavior
inthe nextsection.
CKYParsing
The algorithm given in Fig. 13.10 is a recognizer, not a parse r; for it to succeed it
simply has to ﬁnd an Sin cell [0,N]. To turn it into a parser capable of returning all
possibleparsesfora giveninput,we’ll maketwo simple chan gesto the algorithm: the

ﬁrst change is to augment the entries in the table so that each non-terminal is paired
with pointers to the table entries from which it was derived ( more or less as shown in
Fig.13.12),the secondchangeisto permitmultipleversion softhesamenon-terminal
to be entered into the table (again as shown in Fig. 13.12.) Wi th these changes, the
completedtablecontainsallthepossibleparsesforagiven input. Returninganarbitrary
singleparseconsistsofchoosingan Sfromcell [0,n]andthenrecursivelyretrievingits
componentconstituentsfromthetable.

Ofcourse,returningalltheparsesforagiveninputmayincu rconsiderablecost. As
we saw earlier, there may be an exponentialnumberof parses a ssociated with a given
input. Insuchcases,returningalltheparseswillhaveanun avoidableexponentialcost.
Looking forward to Ch. 14, we can also think about retrieving the best parse for a
given input by further augmenting the table to contain the pr obabilities of each entry.
Retrieving the most probable parse consists of running a sui tably modiﬁed version of
theViterbialgorithmfromCh.5 overthe completedparsetab le.

DRAFTSection13.4. DynamicProgrammingParsingMethods 445
...
...[0,0] [0,0]
[n,n][i,i] [i,i+1] [i,i+2] [i,j-2][i,j-1]
[i+1,j]
[i+2,j]
[j,j][j-1,j][j-2,j][i,j]...
Figure13.11 All the ways toﬁllthe [i,j]thcell inthe CKYtable.
CKYin Practice
Finally,weshouldnotethatwhiletherestrictiontoCNFdoe snotposeaproblemtheo-
retically,itdoesposesomenon-trivialproblemsinpracti ce. Obviously,asthingsstand
now,ourparserisn’treturningtreesthatareconsistentwi ththegrammargiventousby
ourfriendlysyntacticians. Inadditiontomakingourgramm ardevelopersunhappy,the
conversionto CNF will complicateanysyntax-drivenapproa chto semanticanalysis.
One approach to getting around these problems is to keep enou gh information
around to transform our trees back to the original grammar as a post-processing step
of the parse. This is trivial in the case of the transformatio nused for rules with length
greater than 2. Simply deleting the new dummy non-terminals and promoting their
daughtersrestoresthe originaltree.

In the case of unit productions, it turns out to be more conven ient to alter the ba-
sic CKY algorithm to handle them directly than it is to store t he information needed
to recover the correct trees. Exercise 3 asks you to make this change. Many of the

DRAFT446 Chapter 13. Parsingwith Context-FreeGrammars
Nominal, 
NounS,VP,X2
Det NP
Nominal,
Prep
NP,
Proper-
[3,4]
[4,5][3,5][1,5]Book the flight through Houston
Nominal, 
Det NP NP
Nominal,
Noun
Prep PP
NP,
Proper-
[3,4]
Nominal, 
Det NP NP
Nominal,
Prep PP
NP,
Proper-
[3,4]
Nominal, 
Det NP NP
Nominal,
Prep PP
NP,
Proper-
[3,4]
Nominal, 
NounS,
VP,
X2S1, VP
Det NP NP
Nominal,
Prep PP
NP,
Proper-

Nominal, 
Det NP NP
Nominal,
Prep PP
NP,
Proper-
[3,4]
Nominal, 
Det NP NP
Nominal,
Prep PP
NP,
Proper-
[3,4]
Nominal, 
NounS,
VP,
X2S1, VP
Det NP NP
Nominal,
Prep PP
NP,
Proper-
Noun[0,1] [0,2] [0,3] [0,4]
[3,4]
[4,5][3,5][1,5]S2, VP
SS3
Figure13.12 Fillingthe last column afterreading the word Houston.

DRAFTSection13.4. DynamicProgrammingParsingMethods 447
probabilistic parsers presented in Ch. 14 use the CKY algori thm altered in just this
manner. Another solution is to adopt a more complex dynamicp rogrammingsolution
thatsimplyacceptsarbitraryCFGs. Thenextsectionpresen tssuchanapproach.
13.4.2 The Earley Algorithm
In contrast with the bottom-upsearch implementedby the CKY algorithm, the Earley
algorithm(Earley,1970)usesdynamicprogrammingtoimple mentatop-downsearch
ofthekinddiscussedearlierinSec.13.1.1. ThecoreoftheE arleyalgorithmisasingle
left-to-right pass that ﬁlls an array we’ll call a chartthat hasN+1 entries. For each Chart
wordpositionin thesentence,the chartcontainsa list ofst atesrepresentingthepartial
parse trees that have been generated so far. As with the CKY al gorithm, the indexes
represent the locations between the words in an input (as in 0Book1that2ﬂight3).
By the end of the sentence, the chart compactly encodes all th e possible parses of the

input. Eachpossiblesubtreeis representedonlyonceandca nthusbesharedbyall the
parsesthat needit.
The individual states contained within each chart entry con tain three kinds of in-
formation: a subtree corresponding to a single grammar rule , information about the
progressmade in completing this subtree, and the position o f the subtree with respect
to the input. We’ll use a •within the right-hand side of a state’s grammar rule to in-

dicate the progress made in recognizing it. The resulting st ructure is called a dotted
rule. A state’s position with respect to the input will be represe nted by two numbers Dottedrule
indicatingwherethe state beginsandwhereitsdotlies.
Considerthefollowingexamplestates,whichwouldbeamong thosecreatedbythe
Earleyalgorithminthecourseofparsing(13.7):
(13.7)Bookthatﬂight.
S→•VP,[0,0]
NP→Det•Nominal ,[1,2]
VP→VNP•,[0,3]
Theﬁrststate,withitsdottotheleftofitsconstituent,re presentsatop-downprediction

for this particular kind of S. The ﬁrst 0 indicates that the constituent predicted by this
stateshouldbeginatthestartoftheinput;thesecond0reﬂe ctsthefactthatthedotlies
at the beginningas well. The secondstate, createdat a later stage in the processingof
thissentence,indicatesthatan NPbeginsatposition1,thata Dethasbeensuccessfully
parsed and that a Nominalis expectednext. The third state, with its dot to the right of
allitstwoconstituents,representsthesuccessfuldiscov eryofatreecorrespondingtoa

VPthatspanstheentireinput.
ThebasicoperationofanEarleyparseristomarchthroughth eN+1setsofstates
in the chart in a left-to-rightfashion, processing the stat es within each set in order. At
eachstep,oneofthethreeoperatorsdescribedbelowisappl iedtoeachstatedepending
on its status. In each case, this results in the addition of ne w states to the end of
either the current or the next set of states in the chart. The a lgorithm always moves
forward through the chart making additions as it goes; state s are never removed and

DRAFT448 Chapter 13. Parsingwith Context-FreeGrammars
the algorithm never backtracks to a previous chart entry onc e it has moved on. The
presenceofastate S→α•,[0,N]inthelist ofstatesinthelast chartentryindicatesa
successfulparse. Fig.13.13givesthecompletealgorithm.
function EARLEY-PARSE(words,grammar )returnschart
ENQUEUE((γ→•S,[0,0]),chart[0])
fori←from0toLENGTH(words)do
for eachstateinchart[i]do
ifINCOMPLETE ?(state)and
NEXT-CAT(state) isnot apart of speech then
PREDICTOR (state)
elseifINCOMPLETE ?(state)and
NEXT-CAT(state) isa part of speech then
SCANNER(state)
else
COMPLETER (state)
end
end
return(chart)
procedure PREDICTOR ((A→α•Bβ,[i,j]))
for each (B→γ)inGRAMMAR -RULES-FOR(B,grammar)do
ENQUEUE((B→•γ,[j,j]),chart[j])
end
procedure SCANNER((A→α•Bβ,[i,j]))
ifB⊂PARTS-OF-SPEECH(word[j])then
ENQUEUE((B→word[j],[j,j+1]),chart[j+1] )
procedure COMPLETER ((B→γ•,[j,k]))
for each(A→α•Bβ,[i,j])inchart[j]do
ENQUEUE((A→αB•β,[i,k]),chart[k])
end
procedure ENQUEUE(state,chart-entry )

else
end
end
end
end
ifstateisnot alreadyin chart-entry then
PUSH(state,chart-entry )
end
Figure13.13 The Earleyalgorithm
The following three sections describe in detail the three op erators used to process
states in the chart. Each takes a single state as input and der ives new states from it.
These new states are then added to the chart as long as they are not already present.
The PREDICTOR and the C OMPLETER add states to the chart entry being processed,
whilethe S CANNER addsa state tothenextchartentry.

DRAFTSection13.4. DynamicProgrammingParsingMethods 449
Predictor
As might be guessed from its name, the job of P REDICTOR is to create new states
representingtop-downexpectationsgeneratedduringthep arsingprocess. P REDICTOR
isappliedtoanystate that hasa non-terminalimmediatelyt o therightofitsdotthat is
not a part-of-speechcategory. Thisapplication results in the creation of one new state
for each alternative expansion of that non-terminal provid ed by the grammar. These
newstates areplacedintothe samechartentryasthe generat ingstate. Theybeginand
endatthe pointinthe inputwherethegeneratingstate ends.
Forexample,applyingP REDICTOR tothestate S→•VP,[0,0]resultsintheaddi-
tionofthefollowingﬁvestatestotheﬁrst chartentry.
VP→•Verb,[0,0]
VP→•VerbNP ,[0,0]
VP→•VerbNPPP ,[0,0]
VP→•VerbPP ,[0,0]
VP→•VPPP ,[0,0]
Scanner
Whenastatehasapart-of-speechcategorytotherightofthe dot,SCANNER iscalledto
examinetheinputandincorporateastatecorrespondingtot hepredictionofawordwith

Scanner
a particularpart-of-speechintothe chart. Thisisaccompl ishedbycreatinga newstate
fromthe inputstate withthe dotadvancedoverthepredicted inputcategory. Note that
unlike CKY, Earley uses top-downinput to help deal with part -of-speechambiguities;
onlythoseparts-of-speechofa wordthatarepredictedbyso meexistingstatewill ﬁnd
theirwayintothechart.
Returning to our example, when the state VP→•VerbNP ,[0,0]is processed,
SCANNER consultsthecurrentwordintheinputsincethecategoryfol lowingthedotis

a part-of-speech. Itthennotesthat bookcan beaverb,matchingtheexpectationin the
current state. This results in the creation of the new state Verb→book•,[0,1]. This
newstateisthenaddedtothechartentrythat followstheonecurrentlybeingprocessed.
The noun sense of booknever enters the chart since it is not predicted by any rule at
thispositioninthe input.
We should note that our version of the S CANNER and PREDICTOR differ slightly
fromthecorrespondingoperationsinthe originalformulat ionofthe algorithm(Earley,

1970). Thereterminalsweretreateduniformlyasordinaryp artsofthegrammarbyboth
thePREDICTOR andtheS CANNER. Inthisapproach,astatelike VP→•VerbNP ,[0,0]
would trigger predicted states correspondingto any rule th at hadVerbas its left-hand
side. In our current example, the state Verb→•bookwould be predicted. The origi-
nalSCANNER wouldsubsequentlyencounterthispredictedstateandmatc hthecurrent
input token against the predicted token, with the result bei ng a new state with the dot
advanced, Verb→book•.

Unfortunately, this approach is not practical for applicat ions with large lexicons,
since states representing everyword in a given word class would be entered into the
chart as soon as that class was predicted. In our current exam ple, states representing

DRAFT450 Chapter 13. Parsingwith Context-FreeGrammars
every known verb would be added in addition to the one for book. For this reason,
our version of the P REDICTOR does not create states representing predictions for in-
dividual lexical items. The S CANNER makes up for this by explicitly inserting states
representingcompletedlexicalitemsdespitethefactthat therearenostatesinthechart
predictingthem.
Completer
COMPLETER is applied to a state when its dot has reached the right end of t he rule.
Thepresenceofsuchastate representsthefactthatthepars erhassuccessfullydiscov-
ered a particular grammatical category over some span of the input. The purpose of
COMPLETER istoﬁnd,andadvance,allpreviouslycreatedstatesthatwe relookingfor
this grammatical category at this position in the input. New states are then created by
copyingthe older state, advancing the dot over the expected categor y, and installing
thenewstate inthe currentchartentry.
In the current example, when the state NP→DetNominal•,[1,3]is processed,

COMPLETER looks for incompletestates ending at position 1 and expecti nganNP. It
ﬁndsthestates VP→Verb•NP,[0,1]andVP→Verb•NPPP ,[0,1]. Thisresultsinthe
additionofthenewcompletestate VP→VerbNP•,[0,3],andthenewincompletestate
VP→VerbNP•PP,[0,3]to thechart.
A CompleteExample
Fig. 13.14showsthe sequenceof states createdduringthe co mpleteprocessingof Ex.
13.7; each row indicates the state number for reference, the dotted rule, the start and
end points, and ﬁnally the function that added this state to t he chart. The algorithm

beginsbyseedingthechartwithatop-downexpectationfora nS.Thisisaccomplished
by adding a dummy state γ→•S,[0,0]to Chart[0]. When this state is processed, it
ispassedto P REDICTOR leadingtothecreationofthethreestatesrepresentingpre dic-
tions for each possible type of S, and transitively to states for all of the left-cornersof
those trees. When the state VP→•Verb,[0,0]is reached, S CANNER is calledand the
ﬁrst word is read. A state representing the verb sense of Bookis added to the entry
for Chart[1]. Note that when the subsequent sentence initia lVPstates are processed,

SCANNER willbecalledagain. However,newstatesarenotaddedsince theywouldbe
identicaltothe Verbstate alreadyin thechart.
When all the states of Chart[0] have been processed, the algo rithm moves on to
Chart[1] where it ﬁnds the state representing the verb sense ofbook. This is a com-
plete state with its dot to the right of its constituent and is therefore passed to C OM-
PLETER. COMPLETER then ﬁnds the four previously existing VPstates expecting a

Verb at this point in the input. These states are copied with th eir dots advanced and
addedto Chart[1]. The completedstate correspondingto an i ntransitive VPthen leads
tothecreationofan Srepresentinganimperativesentence. Alternatively,thed otinthe
transitiveverbphraseleadstothecreationofthethreesta tespredictingdifferentforms
ofNPs. The state NP→•DetNominal ,[1,1]causes S CANNER to read the word that
andadda correspondingstate toChart[2].
Moving on to Chart[2], the algorithm ﬁnds the state represen ting the determiner

sense ofthat. This complete state leads to the advancement of the dot in th eNPstate

DRAFTSection13.4. DynamicProgrammingParsingMethods 451
Chart[0] S0 γ→•S [0,0] Dummy startstate
S1 S→•NPVP [0,0] Predictor
S2 S→•AuxNPVP [0,0] Predictor
S3 S→•VP [0,0] Predictor
S4 NP→•Pronoun [0,0] Predictor
S5 NP→•Proper-Noun [0,0] Predictor
S6 NP→•DetNominal [0,0] Predictor
S7 VP→•Verb [0,0] Predictor
S8 VP→•VerbNP [0,0] Predictor
S9 VP→•VerbNPPP [0,0] Predictor
S10 VP→•VerbPP [0,0] Predictor
S11 VP→•VPPP [0,0] Predictor
Chart[1] S12 Verb→book• [0,1] Scanner
S13 VP→Verb• [0,1] Completer
S14 VP→Verb•NP [0,1] Completer
S15 VP→Verb•NPPP [0,1] Completer
S16 VP→Verb•PP [0,1] Completer
S17 S→VP• [0,1] Completer
S18 VP→VP•PP [0,1] Completer
S19 NP→•Pronoun [1,1] Predictor
S20 NP→•Proper-Noun [1,1] Predictor
S21 NP→•DetNominal [1,1] Predictor
S22 PP→•PrepNP [1,1] Predictor
Chart[2] S23 Det→that• [1,2] Scanner
S24 NP→Det•Nominal [1,2] Completer
S25 Nominal→•Noun [2,2] Predictor
S26 Nominal→•NominalNoun [2,2] Predictor
S27 Nominal→•NominalPP [2,2] Predictor
Chart[3] S28 Noun→ﬂight• [2,3] Scanner

S29 Nominal→Noun• [2,3] Completer
S30 NP→DetNominal• [1,3] Completer
S31 Nominal→Nominal•Noun[2,3] Completer
S32 Nominal→Nominal•PP[2,3] Completer
S33 VP→VerbNP• [0,3] Completer
S34 VP→VerbNP•PP [0,3] Completer
S35 PP→•PrepNP [3,3] Predictor
S36 S→VP• [0,3] Completer
S37 VP→VP•PP [0,3] Completer
Figure13.14 Chart entries created during an Earley parse of Book that ﬂight. Each entry
shows the state,itsstart andend points, andthe function th at placedit inthe chart.

predictedinChart[1],andalsotothepredictionsfortheva riouskindsof Nominal. The
ﬁrst ofthesecauses S CANNER to becalledforthe last timetoprocesstheword ﬂight.
Finally moving on to Chart[3], the presence of the state repr esentingﬂightleads
in succession to the completionof an NP, transitive VP, andanS. The presence of the
stateS→VP•,[0,3]inthelast chartentrysignalsthediscoveryofa successful parse.
ItisusefultocontrastthisexamplewiththeCKY examplegiv enearlier. Although

DRAFT452 Chapter 13. Parsingwith Context-FreeGrammars
Chart[1] S12 Verb→book•[0,1] Scanner
Chart[3] S28 Noun→ﬂight•[2,3] Scanner
S29 Nominal→Noun•[2,3] (S28)
S30 NP→DetNominal•[1,3] (S23,S29)
S33 VP→VerbNP•[0,3] (S12,S30)
S36 S→VP• [0,3] (S33)
Figure13.15 States that participate in the ﬁnal parse of Book that ﬂight , including structural
parse information.
Earleymanagedtoavoidaddinganentryforthenounsenseof book,itsoverallbehavior
is clearly much morepromiscuousthan CKY. This promiscuity arises fromthe purely
top-downnature of the predictionsthat Earley makes. Exerc ise 6 asks you to improve
thealgorithmbyeliminatingsomeoftheseunnecessarypred ictions.
RetrievingParseTreesfromaChart
As with the CKY algorithm, this version of the Earley algorit hm is a recognizer not
a parser. Valid sentences will simply leave the state S→α•,[0,N]in the chart. To
retrieveparsesfromthe chart the representationof each st ate must be augmentedwith

an additional ﬁeld to store information about the completed states that generated its
The information needed to ﬁll these ﬁelds can be gathered by m aking a simple
change to the C OMPLETER function. Recall that C OMPLETER creates new states by
advancing existing incomplete states when the constituent following the dot has been
discoveredintherightplace. Theonlychangenecessaryist ohaveC OMPLETER adda
pointertotheolderstate ontoa list ofconstituent-states forthe newstate. Retrievinga

parsetreefromthechartisthenmerelyamatteroffollowing pointersstartingwiththe
state (or states) representinga complete Sin the ﬁnal chart entry. Figure 13.15 shows
thechartentriesproducedbyanappropriatelyupdatedC OMPLETER thatparticipatein
theﬁnal parseforthisexample.
13.4.3 ChartParsing
InboththeCKYandEarleyalgorithms,theorderinwhicheven tsoccur(addingentries
to the table, reading words, making predictions, etc.) is st atically determined by the

proceduresthatmakeupthesealgorithms. Unfortunately,d ynamicallydeterminingthe
order in which events occur based on the current information is often necessary for a
variety of reasons. Fortunately, chart parsing , an approach advanced by Martin Kay Chart parsing
andhiscolleagues(Kaplan,1973;Kay,1982),permitsamore ﬂexibledeterminationof
theorderinwhichchartentriesareprocessed. Thisisaccom plishedthroughtheuseof
anexplicit agenda. Inthisscheme,asstates(called edgesinthisapproach)arecreated,

they are added to an agendathat is kept orderedaccordingto a policythat is speciﬁed
separately fromthemainparsingalgorithm. Thiscanbeviewedasanothe rinstanceof
state-spacesearchthatwe’veseenseveraltimesbefore. Th eFSA andFSTrecognition

DRAFTSection13.4. DynamicProgrammingParsingMethods 453
and parsing algorithmsin Chs. 2 and 3 employed agendas with s imple static policies,
while the A∗decoding algorithm described in Ch. 9 is driven by an agenda t hat is
orderedprobabilistically.
Fig.13.16presentsagenericversionofaparserbasedonsuc hascheme. Themain
part of the algorithm consists of a single loop that removes a n edge from the front of
an agenda, processes it, and then moves on to the next entry in the agenda. When the
agenda is empty, the parser stops and returns the chart. The p olicy used to order the
elements in the agenda thus determines the order in which fur ther edges are created
andpredictionsaremade.
function CHART-PARSE(words,grammar,agenda-strategy )returnschart
INITIALIZE (chart,agenda,words )
whileagenda
current-edge←POP(agenda)
PROCESS-EDGE(current-edge )
procedure PROCESS-EDGE(edge)
ADD-TO-CHART(edge)
ifINCOMPLETE ?(edge)
FORWARD-FUNDAMENTAL -RULE(edge)
else
BACKWARD -FUNDAMENTAL -RULE(edge)

else
MAKE-PREDICTIONS (edge)
procedure FORWARD-FUNDAMENTAL ((A→α•Bβ,[i,j]))
for each(B→γ•,[j,k])inchart
ADD-TO-AGENDA(A→αB•β,[i,k])
procedure BACKWARD -FUNDAMENTAL ((B→γ•,[j,k]))
for each(A→α•Bβ,[i,j])inchart
procedure ADD-TO-CHART(edge)
ifedgeisnot already in chartthen
Addedgetochart
procedure ADD-TO-AGENDA(edge)
ifedgeis not already in agendathen
APPLY(agenda-strategy ,edge,agenda)
Figure13.16 A Chart ParsingAlgorithm

else
The key principle in processing edges in this approach is wha t Kay termed the
fundamental rule of chart parsing. The fundamental rule states that when the c hart Fundamental rule
contains two contiguous edges where one of the edges provide s the constituent that
the other one needs, a new edge should be created that spans th e original edges and
incorporates the provided material. More formally, the fun damental rule states the
following: if the chart containstwo edges A→α•Bβ,[i,j]andB→γ•,[j,k]then

we should add the new edge A→αB•β[i,k]to the chart. It should be clear that the

DRAFT454 Chapter 13. Parsingwith Context-FreeGrammars
fundamentalrule is a generalizationof the basic table-ﬁll ing operationsfound in both
theCKY andEarleyalgorithms.
The fundamentalrule is triggeredin Fig. 13.16when an edge i s removedfrom the
agenda and passed to the P ROCESS-EDGEprocedure. Note that the fundamental rule
itself does not specify which of the two edges involved has tr iggered the processing.
PROCESS-EDGEhandles both cases by checking to see whether or not the edge i n
question is complete. If it is complete then the algorithm lo oks earlier in the chart to
see if any existing edge can be advanced; if it is incomplete t hen it looks later in the
charttosee ifit canbeadvancedbyanypre-existingedgelat er inthechart.
The next piece of the algorithm to specify is the method for ma king predictions
based on the edge being processed. There are two key componen tsto making predic-
tions in chart parsing: the events that trigger predictions , and the nature of the predic-

tion. These components vary depending on whether we are purs uing a top-down or
bottom-up strategy. As in Earley, top-down predictions are triggered by expectations
that arise fromincompleteedgesthat have been enteredinto the chart; bottom-uppre-
dictionsaretriggeredbythediscoveryofcompletedconsti tuents. Fig.13.17illustrates
howthesetwostrategiescanbeintegratedintothechartpar singalgorithm.
procedure MAKE-PREDICTIONS (edge)
ifTop-Down andINCOMPLETE ?(edge)
TD-PREDICT(edge)
elsifBottom-Up andCOMPLETE ?(edge)

BU-PREDICT(edge)
procedure TD-PREDICT((A→α•Bβ,[i,j]))
for each(B→γ)ingrammar do
ADD-TO-AGENDA(B→•γ,[j,j])
procedure BU-PREDICT((B→γ•,[i,j]))
for each(A→Bβ) ingrammar
ADD-TO-AGENDA(A→B•β,[i,j])
Figure13.17 A Chart ParsingAlgorithm
Obviously, we’ve left out many of the bookkeeping details th at would have to be
speciﬁed to turn this approach into a real parser. Among the d etails that have to be
workedoutarehowtheI NITIALIZE proceduregetsthingsstarted,howandwhenwords
are read, the organization of the chart, and specifying an ag enda strategy. Indeed, in

describing the approach here, ? (?) refers to it as an algorithm schema rather than Algorithmschema
an algorithm, since it more accurately speciﬁes an entire fa mily of parsers rather than
any particular parser. Exercise 7 asks you to exploresome of the available choices by
implementingvariouschartparsers.

DRAFTSection13.5. PartialParsing 455
13.5 PartialParsing
Many language-processingtasks do not require complex, com plete parse trees for all
inputs. For these tasks, a partial parse , orshallow parse , of input sentences may be Partial parse
Shallowparse sufﬁcient. For example, information extraction systems ge nerally do not extract all
the possible informationfroma text; theysimply identifya ndclassify the segmentsin
a text that are likely to contain valuable information. Simi larly, information retrieval
systemsmaychoosetoindexa textbasedona subsetofthecons tituentsfoundin it.
There are many different approaches to partial parsing. Som e make use of cas-
cades of FSTs, of the kind discussed in Ch. 3, to produce tree- like representations.
Theseapproachestypicallyproduceﬂattertreesthantheon eswe’vebeendiscussingin
this chapter and the previous one. This ﬂatness arises from t he fact that FST cascade
approaches generally defer decisions that may require sema ntic or contextual factors,

suchasprepositionalphraseattachments,coordinationam biguities,andnominalcom-
poundanalyses. Neverthelesstheintentistoproduceparse treesthatlinkallthemajor
constituentsin aninput.
Analternativestyle of partialparsingis knownas chunking . Chunkingis the pro- Chunking
cessofidentifyingandclassifyingtheﬂat,non-overlappi ngsegmentsofasentencethat
constitute the basic non-recursivephrases corresponding to the major parts-of-speech
foundinmostwide-coveragegrammars. Thissettypicallyin cludesnounphrases,verb

phrases, adjective phrases, and prepositional phrases; in other words, the phrases that
correspond to the content-bearing parts-of-speech. Of cou rse, not all applications re-
quire the identiﬁcation of all of these categories; indeed t he most common chunking
task istosimplyﬁndall thebasenounphrasesina text.
Since chunked texts lack a hierarchical structure, a simple bracketing notation is
sufﬁcient to denote the location and the type of the chunks in a given example. The

followingexampleillustratesa typicalbracketednotatio n.
(13.8)[NPThemorningﬂight][ PPfrom][NPDenver][ VPhasarrived.]
This bracketing notation makes clear the two fundamental ta sks that are involved in
chunking: ﬁndingthenon-overlappingextentsofthechunks ,andassigningthecorrect
labeltothe discoveredchunks.
Note that in this example all the words are contained in some c hunk. This will
not be the case in all chunking applications. Often many word s in any input will fall

outsideofanychunk,forexampleinsystemssearchingforba seNPsintheirinputs,as
inthe following:
(13.9)[NPThemorningﬂight]from[ NPDenver]hasarrived.
The details of what constitutesa syntactic base phrase for a ny givensystem varies
according to the syntactic theories underlyingthe system a nd whether the phrases are
being derived from a treebank. Nevertheless, some standard guidelines are followed
in most systems. First and foremost, base phrases of a given t ype do not recursively

containanyconstituentsofthe same type. Eliminatingthis kindofrecursionleavesus
with the problemof determiningthe boundariesof the non-re cursivephrases. In most
approaches,base phrasesincludethe headwordofthe phrase ,alongwithanypre-head

DRAFT456 Chapter 13. Parsingwith Context-FreeGrammars
materialwithintheconstituent,whilecruciallyexcludin ganypost-headmaterial. Elim-
inatingpost-headmodiﬁersfromthe majorcategoriesautom aticallyremovesthe need
to resolve attachment ambiguities. Note that exclusion doe s lead to certain oddities
such as the fact that PPs andVPs often consist solely of their heads. Thus our earlier
exampleaﬂightfromIndianapolistoHoustononNWA isreducedtothefollowing:
(13.10)[NPaﬂight][ PPfrom][NPIndianapolis][ PPto][NPHouston][ PPon][NPNWA]
13.5.1 Finite-State Rule-Based Chunking
Syntacticbasephrasesofthekindwe’reconsideringcanbec haracterizedbyﬁnite-state
automata (or ﬁnite-state rules, or regular expressions) of the kind discussed in Chs. 2
and 3. In ﬁnite-state rule-based chunking, a set of rules is h and-craftedto capture the
phrasesofinterestforanyparticularapplication. Inmost rule-basedsystems,chunking
proceedsfrom left-to-right,ﬁndingthe longest matchingc hunk from the beginningof

the sentence, continuingwith the ﬁrst word after the end of t he previouslyrecognized
chunk. The process continues until the end of the sentence. T his is a greedy process
andisnotguaranteedtoﬁndthebest globalanalysisforanyg iveninput.
Theprimarylimitationplacedonthesechunkrulesisthatth eycannotcontainany
recursion; the right-hand side of the rule can not reference directly, or indirectly, the
category that the rule is designed to capture. In other words , rules of the form NP→
DetNominal areﬁne,butrulessuchas Nominal→NominalPP arenot. Considerthe

followingexamplechunkrulesadaptedfromAbney(1996).
NP→(Det)Noun* Noun
NP→Proper-Noun
VP→Verb
VP→AuxVerb
The process of turning these rules into a single ﬁnite-state transducer is the same
we introduced in Ch. 3 to capture spelling and phonological r ules for English. Finite
state transducersare created correspondingto each rule an d are then unionedtogether
toforma singlemachinethat canthenbedeterminizedandmin imized.
Aswe saw inCh.3,a majorbeneﬁtoftheﬁnite-stateapproachi stheabilityto use

VP→Verb
the outputof earliertransducersasinputsto subsequenttr ansducersto form cascades.
Inpartialparsing , thistechniquecanbe used to moreclosely approximatethe o utput
oftruecontext-freeparsers. Inthisapproach,aninitials etoftransducersisused,inthe
way just described, to ﬁnd a subset of syntactic base phrases . These base phrases are
thenpassedasinputtofurthertransducersthatdetectlarg erandlargerconstituentssuch
as prepositionalphrases, verb phrases, clauses, and sente nces. Consider the following

rules,againadaptedfromAbney(1996).
FST2PP→INNP
FST3S→PP*NPPP*VPPP*
Combiningthesetwo machineswiththe earlierrule-setresu ltsina threemachinecas-
cade. TheapplicationofthiscascadetoEx. 13.8isshownin F ig.13.18.

DRAFTSection13.5. PartialParsing 457
FST1S
NP PP VP
NP IN NP VP
DT  NN  NN IN PRP Aux VB 
The morning flight from Denver has arrivedFST2FST3
Figure13.18 Chunk-based partial parsing via a set of ﬁnite-state cascad es. FST 1transduces
from part-of-speech tags to base noun phrases and verb phras es. FST 2ﬁnds prepositional
phrases. Finally,FST 3detects sentences.
13.5.2 MachineLearning-Based Approaches to Chunking
Aswithpart-of-speechtagging,analternativeto rule-bas edprocessingistousesuper-
visedmachinelearningtechniquesto trainachunkerusingannotateddataasatraining
set. AsdescribedearlierinCh.6,wecanviewthetaskasoneo fsequentialclassiﬁca-
tion,whereaclassiﬁeristrainedtolabeleachelementoftheinp utinsequence. Anyof
the standard approaches to training classiﬁers apply to thi s problem. In the work that
pioneered this approach, Ramshaw and Marcus (1995) used the transformation-based
learningmethoddescribedinCh.5.
The critical ﬁrst step in such an approach is to ﬁnd a way to vie w the chunking

process that is amenable to sequential classiﬁcation. A par ticularly fruitful approach
is to treat chunkingas a tagging task similar to part-of-spe echtagging (Ramshaw and
Marcus, 1995). In this approach, a small tagset simultaneou sly encodes both the seg-
mentationandthe labelingof the chunksin the input. The sta ndardway to do this has
come to be called IOB tagging and is accomplished by introducing tags to represent IOB tagging

the beginning ( B) and internal ( I) parts of each chunk, as well as those elements of
the input that are outside ( O) any chunk. Under this scheme, the size of the tagset is
(2n+1)wherenis the number of categories to be classiﬁed. The following ex ample
showsthebracketingnotationofEx. 13.8onpg.455reframed asa taggingtask:
(13.11)The
BNPmorning
INPﬂight
INPfrom
BPPDenver
BNPhas
BVParrived
IVP
Thesamesentencewithonlythebase-NPstaggedillustrates theroleofthe Otags.
(13.12)The
INPﬂight
INPfrom
ODenver
BNPhas
Oarrived.
O

INPﬂight
INPfrom
BPPDenver
BNPhas
IVP
INPﬂight
INPfrom
ODenver
BNPhas
Oarrived.
O
Notice that there is no explicit encoding of the end of a chunk in this scheme; the

DRAFT458 Chapter 13. Parsingwith Context-FreeGrammars
B_NPI_NP?     
The flightfromDenverhasarrived Classifier
DT NN NN IN NNP
Corresponding feature representation
The, DT, B_NP, morning, NN, I_NP, flight, NN, from, IN, Denver, NNP, I_NPLabel
I_NPmorning
Figure13.19 A sequential classiﬁer-based approach to chunking. The chu nker slides a context window over the
sentence, classifying words as it proceeds. At this point, t he classiﬁer is attempting to label ﬂight. Features derived
from the context typically include the words, part-of-spee ch tags as wellas the previously assigned chunk-tags.
end of any chunk is implicit in any transition from an I or B, to a B or O tag. This
encodingreﬂectsthenotionthatwhensequentiallylabelin gwords,itisgenerallyeasier
(at least in English) to detect the beginning of a new chunk th an it is to know when a
chunk has ended. Not surprisingly, there are a variety of oth er tagging schemes that
representchunksin subtly differentways, includingsome t hat explicitlymark the end

of constituents. TjongKim Sang and Veenstra(1999)describ e threevariationsonthis
basictaggingschemeandinvestigatetheirperformanceona varietyofchunkingtasks.
Given such a tagging scheme, building a chunker consists of t raining a classiﬁer
to label each word of an input sentence with one of the IOB tags from the tagset.
Of course, training requires training data consisting of th e phrases of interest delim-
ited and marked with the appropriate category. The direct ap proach is to annotate
a representative corpus. Unfortunately, annotation effor ts can be both expensive and

time-consuming. It turnsout that the best place to ﬁnd such d ata forchunkingis in an
existingtreebanksuchasthePennTreebankdescribedinCh. 12.
Such treebanks provide a complete syntactic parse for each c orpus sentence, al-
lowing base syntactic phrases to be extracted from the parse constituents. To ﬁnd the
phrases we’re interested in we just need to know the appropri ate non-terminal names
in the corpus. Finding chunk boundariesrequires ﬁnding the head and then including

the material to the left of the head, ignoring the text to the r ight. This is somewhat
error-pronesinceitreliesontheaccuracyofthehead-ﬁndi ngrulesdescribedinCh.12.
Havingextracteda trainingcorpusfroma treebank,we must n ow cast the training
data into a form that’s useful for training classiﬁers. In th is case, each input can be
represented as a set of features extracted from a context win dow that surrounds the

DRAFTSection13.5. PartialParsing 459
word to be classiﬁed. Using a window that extends two words be fore, and two words
after the word being classiﬁed seems to provide reasonable p erformance. Features
extracted from this window include: the words themselves, t heir parts-of-speech, as
well asthechunktagsoftheprecedinginputsinthe window.
Fig. 13.19 illustrates this scheme with the example given ea rlier. During training,
the classiﬁer would be provided with a training vector consi sting of the values of 12
features;the2wordstotherightofthedecisionpoint,thei rparts-of-speechandchunk
tags, the word to be tagged along with its part-of-speech, th e two words that follow
along with their parts-of speech, and ﬁnally the correct chu nk tag, in this case INP.
During classiﬁcation, the classiﬁer is given the same vecto r without the answer and
assignsthemostappropriatetagfromitstagset.
13.5.3 Evaluating Chunking Systems
As with the evaluation of part-of-speechtaggers, the evalu ation of chunkers proceeds

bycomparingchunkeroutputagainstgold-standardanswers providedbyhumananno-
tators. However, unlike part-of-speech tagging, word-by- wordaccuracy measures are
notadequate. Instead,chunkersareevaluatedusingthenot ionsofprecision,recalland
theF measureborrowedfromthe ﬁeldofinformationretrieva l.
Precision measures the percentage of chunks that were provided by a sys tem that Precision
werecorrect. Correctheremeansthatboththeboundariesof thechunkandthechunk’s
labelarecorrect. Precisionisthereforedeﬁnedas:

Precision: =Numberofcorrectchunksgivenbysystem
Totalnumberofchunksgivenbysystem
Recallmeasures the percentage of chunks actually present in the in put that were Recall
correctlyidentiﬁedbythesystem. Recall isdeﬁnedas:
Recall: =Numberofcorrectchunksgivenbysystem
Totalnumberofactualchunksin thetext
TheF-measure (vanRijsbergen,1975)providesawayto combinethesetwome a- F-measure
suresintoa singlemetric. TheF-measureisdeﬁnedas:
Fβ=(β2+1)PR
β2P+R

β2P+R
Theβparameterisusedtodifferentiallyweighttheimportanceo frecallandprecision,
basedperhapsontheneedsofanapplication. Valuesof β>1favorrecall,whilevalues
ofβ<1 favor precision. When β=1, precision and recall are equally balanced; this
issometimescalled Fβ=1orjustF1:
F1=2PR
P+R(13.13)
F-measure comes from a weighted harmonic mean of precision a nd recall. The har-
monicmeanofasetofnumbersisthereciprocalofthearithme ticmeanofreciprocals:
HarmonicMean (a1,a2,a3,a4,...,an) =n
1
a11
a21
a3...1
an(13.14)

DRAFT460 Chapter 13. Parsingwith Context-FreeGrammars
andhenceF-measureis
F=1
1
αP×1
(1−α)Ror(
withβ2=1−α
α)
F=(β2+1)PR
β2P+R(13.15)
The best current systems achieve an F-measure of around .96 o n the task of base-
NP chunking. Learning-based systems designed to ﬁnd a more c omplete set of base-
phrases, such as the ones given in Fig. 13.20, achieve F-meas ures in the .92 to .94
range. The exact choice of learning approach seems to have li ttle impact on these re-
sults;awide-rangeofmachinelearningapproachesachieve essentiallythesameresults
(Cardieet al.,2000). FST-based systemsof thekinddiscuss ed in Sec. 13.5.1achieved
F-measuresrangingfrom.85to.92onthistask.
Factors limiting the performance of current systems includ e the accuracy of the
part-of-speechtaggersusedtoprovidefeaturesforthesys temduringtesting,inconsis-
tencies in the training data introduced by the process of ext racting chunks from parse
trees,anddifﬁcultyresolvingambiguitiesinvolvingconj unctions. Considerthefollow-

ingexamplesthatinvolvepre-nominalmodiﬁersandconjunc tions.
(13.16)[NPLatearrivalsanddepartures]are commonplaceduringwinte r.
(13.17)[NPLatearrivals]and[ NPcancellations]arecommonplaceduringwinter.
Intheﬁrstexample, lateissharedbyboth arrivalsanddepartures yieldingasingle
long base-NP. In the second example, lateis not shared and modiﬁes arrivalsalone,
thusyieldingtwo base-NPs. Distinguishingthese two situa tions,andotherslike them,
requiresaccesstosemanticandcontextinformationunavai lableto currentchunkers.
Label Category Proportion(%) Example

NPNounPhrase 51 Themost frequentlycancelledﬂight
VPVerbPhrase 20 maynotarrive
PPPrepositionalPhrase 20 toHouston
ADVPAdverbialPhrase 4 earlier
SBARSubordinateClause 2 that
ADJPAdjectivePhrase 2 late
Figure13.20 Mostfrequentbase-phrasesusedinthe2000CONLLsharedtas k. Thesechunks
correspond tothe major categories contained inthe PennTre ebank.
13.6 Summary
The two major ideasintroducedin this chapterare those of parsingandpartial pars-
ing. Here’sasummaryofthemainpointswecoveredabouttheseid eas:

•Parsingcanbeviewedasa searchproblem.
•Twocommonarchitecturalmetaphorsforthissearchare top-down (startingwith
the rootSand growing trees down to the input words) and bottom-up (starting
withthewordsandgrowingtreesuptowardtheroot S).

DRAFTSection13.6. Summary 461
•Ambiguity combinedwiththe repeatedparsingofsub-trees poseproblemsfor
simplebacktrackingalgorithms.
•A sentence is structurally ambiguous if the grammarassigns it more than one
possibleparse.
•Common kinds of structural ambiguity include PP-attachment ,coordination
ambiguity andnoun-phrasebracketingambiguity .
•Thedynamic programming parsing algorithms use a table of partial-parses to
efﬁciently parse ambiguous sentences. The CKY,Earley, andChart parsing
algorithms all use dynamic-programmingto solve the repeat ed parsing of sub-
treesproblem.
•TheCKYalgorithmrestrictstheformofitsgrammartoChomsk yNormalForm;
theEarleyandchartparsersacceptunrestrictedcontext-f reegrammars.
•Many practical problems including information extraction problems can be
solvedwithoutfullparsing.
•Partial parsing andchunking are methods for identifying shallow syntactic
constituentsina text.
•High accuracy partial parsing can be achieved either throug h rule-based or ma-
chinelearning-basedmethods.

Writingaboutthehistoryofcompilers,Knuthnotes:
In thisﬁeld there hasbeen an unusualamountof parallel disc overyof the
sametechniquebypeopleworkingindependently.
Well, perhaps not unusual, if multiple discovery is the norm (see page 13). But there
has certainly been enough parallel publication that this hi story will err on the side
of succinctness in giving only a characteristic early menti on of each algorithm; the
interestedreadershouldsee AhoandUllman(1972).
Bottom-upparsingseemstohavebeenﬁrstdescribedbyYngve (1955),whogavea

breadth-ﬁrstbottom-upparsingalgorithmas part of an illu stration of a machinetrans-
lationprocedure. Top-downapproachestoparsingandtrans lationweredescribed(pre-
sumably independently) by at least Glennie (1960), Irons (1 961), and Kuno and Oet-
tinger(1963). Dynamicprogrammingparsing,onceagain,ha sahistoryofindependent
discovery. According to Martin Kay (personal communicatio n), a dynamic program-
mingparsercontainingthe rootsoftheCKY algorithmwas ﬁrs t implementedbyJohn

Cocke in 1960. Later work extended and formalized the algori thm, as well as prov-
ing its time complexity(Kay, 1967; Younger,1967; Kasami, 1 965). The related well-
formedsubstringtable (WFST)seemstohavebeenindependentlyproposedbyKuno WFST
(1965), as a data structure which stores the results of all pr eviouscomputationsin the
courseoftheparse. BasedonageneralizationofCocke’swor k,asimilardata-structure
hadbeenindependentlydescribedbyKay(1967)andKay(1973 ). Thetop-downappli-

cationofdynamicprogrammingtoparsingwasdescribedinEa rley’sPh.D.dissertation

DRAFT462 Chapter 13. Parsingwith Context-FreeGrammars
(Earley,1968,1970). Sheil(1976)showedtheequivalenceo ftheWFSTandtheEarley
algorithm. Norvig (1991) shows that the efﬁciency offered b y dynamic programming
canbecapturedinanylanguagewitha memoization function(suchasLISP)simplyby
wrappingthe memoization operationarounda simpletop-downparser.
While parsing via cascades of ﬁnite-state automata had been common in the early
history of parsing (Harris, 1962), the focus shifted to full CFG parsing quite soon af-
terward. Church (1980) argued for a return to ﬁnite-state gr ammars as a processing
model for natural language understanding; other early ﬁnit e-state parsing models in-
cludeEjerhed(1988). Abney(1991)arguedfortheimportant practicalroleofshallow
parsing. Much recent workon shallow parsingapplies machin elearningto the task of
learning the patterns; see for example Ramshaw and Marcus (1 995), Argamon et al.
(1998),Munozet al.(1999).

The classic reference for parsing algorithms is Aho and Ullm an (1972); although
the focus of that book is on computer languages, most of the al gorithms have been
applied to natural language. A good programming languages t extbook such as Aho
et al.(1986)isalso useful.
Exercises
13.1Implementthealgorithmtoconvertarbitrarycontext-free grammarstoCNF.Ap-
plyyourprogramto the L1grammar.
13.2ImplementtheCKY algorithmandtest it usingyourconverted L1grammar.
13.3Rewrite the CKY algorithmgivenon page13.10so that it can ac cept grammars
thatcontainunitproductions.

Exercises
13.4AugmenttheEarleyalgorithmofFig. 13.13toenableparsetr eestobe retrieved
from the chart by modifying the pseudocode for the C OMPLETER as described
onpage452.
13.5ImplementtheEarleyalgorithmasaugmentedintheprevious exercise. Checkit
ona test sentenceusingthe L1grammar.
13.6Alter the Earley algorithm so that it makes better use of bott om-upinformation
toreducethenumberofuselesspredictions.
13.7Attemptto recasttheCKY andEarleyalgorithmsinthechartp arsingparadigm.

13.8Discusstherelativeadvantagesanddisadvantagesofparti alversusfullparsing.
13.9Implementa more extensive ﬁnite-state grammar for noun-gr oupsusing the ex-
amples given in Sec. 13.5 and test it on some NPs. Use an on-lin e dictionary
withparts-of-speechif available;ifnot,buildamorerest rictedsystem byhand.
13.10Discusshowtoaugmentaparsertodealwith inputthat maybe i ncorrect,such
asspellingerrorsorspeechrecognitionmisrecognitions.

Chapter14
StatisticalParsing
Tworoads diverged inawood, and I–
Itook the one less traveledby...
RobertFrost, TheRoadNot Taken
Thecharactersin DamonRunyon’sshortstoriesarewillingt o bet“onanyproposition
whatever”, as Runyon says about Sky Masterson in The Idyll of Miss Sarah Brown ;
fromthe probabilityofgettingaces back-to-backto the odd sagainsta manbeingable
to throwa peanut fromsecond base to homeplate. There is a mor al here for language
processing: with enough knowledge we can ﬁgure the probabil ity of just about any-
thing. Thelasttwochaptershaveintroducedsophisticated modelsofsyntacticstructure

and its parsing. In this chapter we show that it is possible to build probabilistic mod-
els of syntactic knowledge and use some of this probabilisti c knowledge in efﬁcient
probabilisticparsers.
Onecrucialuseofprobabilisticparsingistosolvetheprob lemofdisambiguation .
Recall fromCh. 13thatsentencesonaveragetendto beverysy ntacticallyambiguous,
due to problemslike coordinationambiguity andattachmentambiguity . The CKY
and Earley parsing algorithms could represent these ambigu ities in an efﬁcient way,

but were not equipped to resolve them. A probabilistic parse r offers a solution to the
problem: computetheprobabilityofeachinterpretation,a ndchoosethemost-probable
interpretation. Thus,duetotheprevalenceofambiguity,m ostmodernparsersusedfor
naturallanguageunderstandingtasks(thematicrolelabel ing,summarization,question-
answering,machinetranslation)areofnecessityprobabil istic.
Anotherimportantuse of probabilisticgrammarsand parser sis inlanguagemod-

elingfor speech recognition. We saw that N-gram grammars are used in speech rec-
ognizers to predict upcoming words, helping constrain the a coustic model search for
words. Probabilistic versions of more sophisticated gramm ars can provide additional
predictivepowertoa speechrecognizer. Of coursehumansha veto dealwiththe same
problems of ambiguity as do speech recognizers, and it is int eresting that psycholog-
ical experiments suggest that people use something like the se probabilistic grammars

inhumanlanguage-processingtasks(e.g.,humanreadingor speechunderstanding).
The most commonlyused probabilistic grammaris the probabilistic context-free
grammar (PCFG), a probabilistic augmentation of context-free gram mars in which
each rule is associated with a probability. We introduce PCF Gs in the next section,
showing how they can be trained on a hand-labeledTreebankgr ammar, and how they
can be parsed. We present the most basic parsing algorithm fo r PCFGs, which is the

probabilisticversionofthe CKYalgorithm that wesaw inCh. 13.
We then show a number of ways that we can improve on this basic p robability
model (PCFGs trained on Treebank grammars). One method of im proving a trained

DRAFT464 Chapter 14. Statistical Parsing
Treebank grammar is to change the names of the non-terminals . By making the non-
terminalssometimesmorespeciﬁcandsometimesmoregenera l,wecancomeupwith
a grammarwith a better probabilitymodel that leads to impro vedparsing scores. An-
other augmentation of the PCFG works by adding more sophisti cated conditioning
factors, extending PCFGs to handle probabilistic subcategorization information and
probabilistic lexicaldependencies .
Finally, we describe the standard PARSEVAL metrics for eval uating parsers, and
discusssomepsychologicalresultsonhumanparsing.
14.1 ProbabilisticContext-FreeGrammars
The simplest augmentationof the context-freegrammaris th eProbabilisticContext-
Free Grammar (PCFG), also known as the Stochastic Context-Free Grammar PCFG
(SCFG), ﬁrst proposed by Booth (1969). Recall that a context-free grammar Gis SCFG
deﬁned by four parameters ( N,Σ,R,S); a probabilistic context-free grammar is also

deﬁnedbyfourparameters,witha slightaugmentationtoeac hoftherulesin R:
Raset ofrulesorproductions,eachofthe form A→β[p],
βisa stringofsymbolsfromthe inﬁniteset ofstrings (Σ∪N)∗,
andpisanumberbetween0 and1expressing P(β|A)
That is, a PCFG differsfrom a standardCFG by augmentingeach rule inRwith a
conditionalprobability:
A→β[p] (14.1)

Herepexpressesthe probabilitythatthe givennon-terminal Awill beexpandedto
the sequence β. That is, pis the conditional probabilityof a given expansion βgiven
theleft-hand-side(LHS)non-terminal A. We canrepresentthisprobabilityas
P(A→β)
oras
P(A→β|A)
oras
P(RHS|LHS)
Thus if we consider all the possible expansions of a non-term inal, the sum of their
probabilitiesmustbe1:
∑
βP(A→β) =1

DRAFTSection14.1. ProbabilisticContext-FreeGrammars 465
S→NPVP [.80]Det→that[.10]|a[.30]|the[.60]
S→AuxNPVP [.15]Noun→book[.10]|ﬂight[.30]
S→VP [.05]|meal[.15]|money [.05]
NP→Pronoun [.35]|ﬂights [.40]|dinner [.10]
NP→Proper-Noun [.30]Verb→book[.30]|include [.30]
NP→DetNominal [.20]|prefer;[.40]
NP→Nominal [.15]Pronoun→I[.40]|she[.05]
Nominal→Noun [.75]|me[.15]|you[.40]
Nominal→NominalNoun [.20]Proper-Noun→Houston [.60]
Nominal→NominalPP [.05]|NWA[.40]
VP→Verb [.35]Aux→does[.60]|can[40]
VP→VerbNP [.20]Preposition→from[.30]|to[.30]
VP→VerbNPPP [.10]|on[.20]|near[.15]
VP→VerbPP [.15]|through [.05]
VP→VerbNPNP [.05]
VP→VPPP [.15]
PP→PrepositionNP [1.0]
Figure14.1 APCFGwhichisaprobabilisticaugmentation ofthe L1miniature EnglishCFG
grammar and lexicon of Fig. 13.1 in Ch. 13. These probabiliti es were made up for pedagogical
purposes and arenot based ona corpus (since anyreal corpus w ouldhave many more rules,and
sothe true probabilities of each rule wouldbe muchsmaller) .

Fig.14.1showsaPCFG: aprobabilisticaugmentationofthe L1miniatureEnglish
CFGgrammarandlexicon. Notethattheprobabilitiesofallo ftheexpansionsofeach
non-terminalsumto1. Alsonotethattheseprobabilitieswe remadeupforpedagogical
purposes. Inanyrealgrammarthereareagreatmanymorerule sforeachnon-terminal
andhencetheprobabilitiesofanyparticularrulewouldten dto bemuchsmaller.
A PCFG is said to be consistent if the sum of the probabilitiesof all sentences in Consistent

thelanguageequals1. Certainkindsofrecursiverulescaus eagrammartobeinconsis-
tent by causing inﬁnitely looping derivations for some sent ences. For example a rule
S→Swith probability 1 would lead to lost probability mass due to derivations that
never terminate. See Booth and Thompson (1973) for more deta ils on consistent and
inconsistentgrammars.
How are PCFGs used? A PCFG can be used to estimate a number of us eful prob-
abilities concerninga sentenceand its parse tree(s), incl udingthe probabilityof a par-

ticularparsetree(usefulindisambiguation)andtheproba bilityofasentenceorapiece
ofa sentence(usefulin languagemodeling). Let’ssee howth isworks.
14.1.1 PCFGsforDisambiguation
APCFGassignsaprobabilitytoeachparsetree T(i.e.,each derivation )ofasentence
S. This attribute is useful in disambiguation . For example, consider the two parses
of the sentence “Book the dinner ﬂights” shown in Fig. 14.2. T he sensible parse on
the left means “Book ﬂights that serve dinner”. The nonsensi cal parse on the right,
however,wouldhavetomeansomethinglike“Bookﬂightsonbe halfof‘thedinner’?”,

DRAFT466 Chapter 14. Statistical Parsing
the way that a structurally similar sentence like “Can you bo ok John ﬂights?” means
somethinglike“Can youbookﬂightsonbehalfofJohn?”.
Theprobabilityofaparticularparse Tisdeﬁnedastheproductoftheprobabilities
of all the nrules used to expand each of the nnon-terminalnodes in the parse tree T,
(whereeachrule icanbeexpressedas LHSi→RHSi):
P(T,S) =n
∏
i=1P(RHSi|LHSi) (14.2)
The resulting probability P(T,S)is both the joint probability of the parse and the
sentence, and also the probability of the parse P(T). How can this be true? First, by
thedeﬁnitionofjointprobability:
P(T,S) =P(T)P(S|T) (14.3)
Butsince aparsetreeincludesall thewordsofthesentence, P(S|T)is1. Thus:
P(T,S) =P(T)P(S|T) =P(T) (14.4)
S
VP
Verb
BookNP
Det
Nominal
Noun
dinnerNoun
ﬂightS
VP
Verb
BookNP
Det
Noun
dinnerNP
Nominal
Noun
ﬂight
Rules P Rules P
S→VP .05 S →VP .05
VP→VerbNP .20 VP →VerbNPNP .10
NP→Det Nominal .20 NP →DetNominal .20
Nominal→Nominal Noun .20 NP →Nominal .15

S
VP
Verb
BookNP
Det
Nominal
Noun
ﬂightS
VP
Verb
BookNP
Det
Noun
dinnerNP
Nominal
Noun
ﬂight
Nominal→Noun .75 Nominal →Noun .75
Nominal→Noun .75
Verb→book .30 Verb →book .30
Det→the .60 Det →the .60
Noun→dinner .10 Noun →dinner .10
Noun→ﬂights .40 Noun →ﬂights .40
Figure14.2 Two parse trees for an ambiguous sentence, The transitive pa rse (a) corresponds
to the sensible meaning “Book ﬂights that serve dinner”, whi le the ditransitive parse (b) to the
nonsensical meaning “Book ﬂights onbehalf of ‘the dinner”’ .
The probability of each of the trees in Fig. 14.2 can be comput ed by multiplying

together the probabilitiesof each of the rules used in the de rivation. For example, the

DRAFTSection14.1. ProbabilisticContext-FreeGrammars 467
probability of the left tree in Fig. 14.2a (call it Tleft) and the right tree (Fig. 14.2b or
Tright)canbecomputedasfollows:
P(Tleft) =.05∗.20∗.20∗.20∗.75∗.30∗.60∗.10∗.40=2.2×10−6
P(Tright) =.05∗.10∗.20∗.15∗.75∗.75∗.30∗.60∗.10∗.40=6.1×10−7
Wecanseethattheleft(transitive)treeinFig.14.2(a)has amuchhigherprobability
than the ditransitive tree on the right. Thus this parse woul d correctly be chosen by a
disambiguationalgorithmwhichselectstheparsewith theh ighestPCFG probability.
Let’s formalize this intuition that picking the parse with t he highest probability is
the correctway to do disambiguation. Consider all the possi ble parsetrees for a given
sentenceS. Thestringofwords Siscalledthe yieldofanyparsetreeover S. Thusout Yield
of all parse trees with a yield of S, the disambiguation algorithm picks the parse tree
whichismostprobablegiven S:
ˆT(S) =argmax
Ts.t.S=yield (T)P(T|S) (14.5)

Bydeﬁnition,theprobability P(T|S)canberewrittenas P(T,S)/P(S),thusleadingto:
Ts.t.S=yield (T)P(T,S)
P(S)(14.6)
Since we are maximizingoverall parse treesforthe same sent ence,P(S)will be a
constantforeachtree,sowe caneliminateit:
Ts.t.S=yield (T)P(T,S) (14.7)
Furthermore, since we showed above that P(T,S) =P(T), the ﬁnal equation for
choosingthe most likely parse neatlysimpliﬁes to choosing the parse with the highest
Ts.t.S=yield (T)P(T) (14.8)

14.1.2 PCFGsforLanguage Modeling
AsecondattributeofaPCFG isthatitassignsa probabilityt othestringofwordscon-
stitutingasentence. Thisisimportantin languagemodeling ,whetherforuseinspeech
recognition, machine translation, spell-correction, aug mentative communication, or
other applications. The probability of an unambiguous sent ence isP(T,S) =P(T)
or just the probability of the single parse tree for that sent ence. The probability of an

ambiguous sentence is the sum of the probabilities of all the parse trees for the sen-
tence:

DRAFT468 Chapter 14. Statistical Parsing
P(S) =∑
Ts.t.S=yield (T)P(T,S) (14.9)
=∑
Ts.t.S=yield (T)P(T) (14.10)
AnadditionalfeatureofPCFGsthatisusefulforlanguagemo delingistheirability
to assign a probability to substrings of a sentence. For exam ple, suppose we want to
know the probabilityof the next word wiin a sentence givenall the wordswe’ve seen
so farw1,...,wi−1. Thegeneralformulaforthisis:
P(wi|w1,w2,...,wi−1) =P(w1,w2,...,wi−1,wi,...)
P(w1,w2,...,wi−1,...)(14.11)
We saw in Ch. 4 a simple approximation of this probability usi ngN-grams, con-
ditioning on only the last word or two instead of the entire co ntext; thus the bigram
approximation wouldgiveus:
P(wi|w1,w2,...,wi−1)≈P(wi−1,wi)
P(wi−1)(14.12)
Butthefactthatthe N-grammodelcanonlymakeuseofacouplewordsofcontext
means it is ignoring potentially useful prediction cues. Co nsider predicting the word
afterinthefollowingsentencefromChelbaandJelinek(2000):
(14.13)thecontractendedwitha lossof7centsaftertradingaslowa s9 cents

A trigram grammar must predict afterfrom the words 7 cents, while it seems clear
that the verb endedand the subject contractwould be useful predictors that a PCFG-
based parser could help us make use of. Indeed, it turns out th at a PCFGs allow us to
conditiononthe entirepreviouscontext w1,w2,...,wi−1shownin Eq. 14.11. We’ll see
the details of waysto use PCFGs and augmentationsof PCFGs as languagemodelsin
Sec.14.9.
In summary, this section and the previousone have shown that PCFGs can be ap-
plied both to disambiguation in syntactic parsing and to wor d prediction in language

Sec.14.9.
modeling. Bothoftheseapplicationsrequirethatwebeable tocomputetheprobability
ofparsetree Tforagivensentence S. Thenextfewsectionsintroducesomealgorithms
forcomputingthisprobability.
14.2 ProbabilisticCKYParsingofPCFGs
The parsing problem for PCFGs is to produce the most-likely p arseˆTfor a given
sentenceS,i.e.,
Ts.t.S=yield (T)P(T) (14.14)

DRAFTSection14.2. ProbabilisticCKY ParsingofPCFGs 469
The algorithms for computing the most-likely parse are simp le extensions of the
standard algorithms for parsing; there are probabilistic v ersions of both the CKY and
EarleyalgorithmsofCh.13. Mostmodernprobabilisticpars ersarebasedonthe prob-
abilistic CKY algorithm,ﬁrst describedbyNey(1991). ProbabilisticCKY
As with the CKY algorithm, we will assume for the probabilist ic CKY algorithm
that the PCFG is in Chomsky normal form. Recall from page 416 t hat grammars in
CNF are restricted to rules of the form A→BC, orA→w. That is, the right-hand
side ofeachrulemustexpandtoeithertwonon-terminalsort oa singleterminal.
For the CKY algorithm, we represented each sentence as havin g indices between
thewords. Thusanexamplesentencelike
(14.15)Booktheﬂight throughHouston.
wouldassume thefollowingindicesbetweeneachword:
(14.16) 0⃝Book①the②ﬂight③through④Houston⑤
Using these indices, each constituent in the CKY parse tree i s encoded in a two-

dimensionalmatrix. Speciﬁcally,forasentenceoflength nandagrammarthatcontains
Vnon-terminals,weusetheupper-triangularportionofan (n+1)×(n+1)matrix. For
CKY, each cell table[i,j]contained a list of constituents that could span the sequenc e
of words from itoj. For probabilistic CKY, it’s slightly simpler to think of th e con-
stituents in each cell as constituting a third dimension of m aximum length V. This
thirddimensioncorrespondstoeachnonterminalthatcanbe placedinthiscell,andthe

value of the cell is then a probabilityfor that nonterminal/ constituentratherthan a list
ofconstituents. Insummary,eachcell [i,j,A]inthis (n+1)×(n+1)×Vmatrixisthe
probabilityofa constituent Athat spanspositions ithroughjoftheinput.
Fig. 14.3 gives pseudocode for this probabilistic CKY algor ithm, extending the
basicCKY algorithmfromFig.13.10.
function PROBABILISTIC -CKY(words,grammar )returnsmost probable parse
and itsprobability
for all{A|A→words[j]∈grammar}
table[j−1,j,A]←P(A→words[j])

for all{A|A→BC∈grammar ,
andtable[i,k,B]>0andtable[k,j,C]>0}
if(table[i,j,A]<P(A→BC)×table[i,k,B]×table[k,j,C])then
table[i,j,A]←P(A→BC)×table[i,k,B]×table[k,j,C]
back[i,j,A]←{k,B,C}
return BUILD TREE(back[1, LENGTH(words),S]),table[1, LENGTH(words),S]
Figure14.3 The probabilistic CKY algorithm for ﬁnding the maximum prob ability parse of
a stringof numwordswords given a PCFGgrammar with numrulesrules in Chomsky Normal
Form.backis an array of back-pointers used to recover the best parse. T hebuildtreefunction
is leftas anexercise tothe reader.

DRAFT470 Chapter 14. Statistical Parsing
Like the CKY algorithm, the probabilistic CKY algorithm as s hown in Fig. 14.3
requires a grammar in Chomsky Normal Form. Converting a prob abilistic grammar
to CNF requires that we also modify the probabilities so that the probability of each
parseremainsthesameunderthenewCNFgrammar. Exercise2a sksyoutomodifythe
algorithmforconversiontoCNFinCh.13sothatitcorrectly handlesruleprobabilities.
In practice, we more often use a generalized CKY algorithm wh ich handles unit
productionsdirectly rather than convertingthem to CNF. Re call that Exercise 3 asked
you to make this change in CKY; Exercise 3 asks you to extend th is change to proba-
bilistic CKY.
Let’s see an example of the probabilistic CKY chart, using th e following mini-
grammarwhichisalreadyin CNF:
S→NPVP .80 Det→the.50
NP→Det N .30 Det→a.40
VP→V NP .20 N→meal.01
V→includes .05 N→flight.02
Given this grammar,Fig. 14.4 shows the ﬁrst steps in the prob abilisticCKY parse
ofthissentence:

(14.17)Theﬂight includesa meal
The flight[0,1] [0,2] [0,3]
[2,3]Det:   .40
includes a meal[3,4]
[4,5]N:   .02
V:   .05NP: .30 *.40 *.02
= .0024
[0,4]
[1,4]
[2,4]
[3,5][3,5][1,5][0,5]
Figure14.4 Thebeginning oftheprobabilisticCKYmatrix. Fillingoutt he restofthechartis
leftas Exercise 4for the reader.

DRAFTSection14.3. LearningPCFG RuleProbabilities 471
14.3 LearningPCFGRuleProbabilities
Where do PCFG rule probabilities come from? There are two way s to learn proba-
bilities for the rules of a grammar. The simplest way is to use a treebank, a corpus of
already-parsedsentences. RecallthatweintroducedinCh. 12theideaoftreebanksand
thecommonly-used PennTreebank (Marcusetal.,1993),acollectionofparsetreesin
English, Chinese, and other languagesdistributed by the Li nguistic Data Consortium.
Givenatreebank,theprobabilityofeachexpansionofanon- terminalcanbecomputed
bycountingthenumberoftimesthat expansionoccursandthe nnormalizing.
P(α→β|α) =Count(α→β)
∑γCount(α→γ)=Count(α→β)
Count(α)(14.18)
If we don’t have a treebank, but we do have a (non-probabilist ic) parser, we can
generate the counts we need for computing PCFG rule probabil ities by ﬁrst parsing a
corpus of sentences with the parser. If sentences were unamb iguous, it would be as

simple as this: parse the corpus, increment a counter for eve ry rule in the parse, and
thennormalizeto getprobabilities.
But wait! Since most sentencesare ambiguous,i.e. have mult ipleparses,we don’t
know which parse to count the rules in. Instead, we need to kee p a separate count for
each parse of a sentence and weight each of these partial coun ts by the probability of
the parseit appearsin. But to get these parse probabilities to weight the ruleswe need
toalreadyhavea probabilisticparser.

Theintuitionforsolvingthischicken-and-eggproblemist oincrementallyimprove
ourestimatesbybeginningwithaparserwith equalruleprob abilities,parsingthesen-
tence,computeaprobabilityforeachparse,usetheseproba bilitiestoweightthecounts,
then reestimate the rule probabilities, and so on, until our probabilitiesconverge. The
standard algorithm for computingthis is called the inside-outside algorithm, and was Inside-outside
proposed by Baker (1979) as a generalization of the forward- backward algorithm of

Ch.6. Likeforward-backward,inside-outsideisaspecialc aseoftheEM(expectation-
maximization)algorithm,andhencehastwosteps: the expectationstep ,andthemax- Expectationstep
imization step . See Lari and Young (1990) or Manning and Sch¨ utze (1999) for a Maximizationstep
completedescriptionofthealgorithm.
This use of the inside-outside algorithm to estimate the rul e probabilities for a
grammar is actually a kind of limited use of inside-outside. The inside-outside al-

gorithm can actually be used not only to set the rule probabil ities, but even to induce
the grammarrulesthemselves. It turnsout, however,that gr ammarinductionis so dif-
ﬁcultthatinside-outsidebyitselfisnotaverysuccessful grammarinducer;seetheend
notesforpointerstoothergrammarinductionalgorithms.
14.4 ProblemswithPCFGs
Whileprobabilisticcontext-freegrammarsareanaturalex tensiontocontext-freegram-
mars,theyhavetwomainproblemsasprobabilityestimators :

DRAFT472 Chapter 14. Statistical Parsing
poor independence assumptions: CFG rules impose an independence assumption
on probabilities, resulting in poor modeling of structural dependencies across
theparsetree.
lack of lexical conditioning: CFG rules don’t model syntactic facts about speciﬁc
words, leading to problems with subcategorization ambigui ties, preposition at-
tachment,andcoordinatestructureambiguities.
Because of these problems, most current probabilistic pars ing models use some
augmented version of PCFGs, or modify the Treebank-based gr ammar in some way.
Inthe nextfew sectionsafterdiscussingthe problemsin mor edetail wewill introduce
someoftheseaugmentations.
14.4.1 Independenceassumptionsmissstructuraldependen ciesbe-
tween rules
Let’s look at these problems in more detail. Recall that in a C FG the expansion of a
non-terminalisindependentofthecontext,i.e.,oftheoth ernearbynon-terminalsinthe
parse tree. Similarly, in a PCFG, the probability of a partic ular rule like NP→DetN

is also independentof the rest of the tree. By deﬁnition, the probability of a group of
independent events is the product of their probabilities. T hese two facts explain why
inaPCFG wecomputetheprobabilityofatreebyjustmultiply ingtheprobabilitiesof
eachnon-terminalexpansion.
Unfortunatelythis CFG independence assumption results in poor probability esti-
mates. This is because in English the choice of how a node expa nds can after all be

dependentonthelocationofthenodeintheparsetree. Forex ample,inEnglishitturns
out that NPs that are syntactic subjectsare far more likely to be pronouns,while NPs
thataresyntactic objectsarefarmorelikelytobenon-pronominal(e.g.,apropernoun
oradeterminernounsequence),asshownbythesestatistics forNPsintheSwitchboard
corpus(Franciset al.,1999):1
Pronoun Non-Pronoun
Subject91% 9%
Object34% 66%
Unfortunatelythere is no way to represent this contextual d ifference in the proba-
bilities in a PCFG. Consider two expansions of the non-termi nalNPas a pronoun or

as a determiner+noun. How shall we set the probabilitiesof t hese two rules? If we set
their probabilitiesto their overall probabilityin the Swi tchboard corpus, the two rules
haveaboutequalprobability.
NP→DT NN .28
NP→PRP .25
1Distribution of subjects from 31,021 declarative sentence s; distribution of objects from 7,489 sentences.
This tendency is caused by the use of subject position to real ize thetopicor old information in a sentence

NP→PRP .25
(Giv´ on, 1990). Pronouns are a way to talk about old informat ion, while non-pronominal (“lexical”) noun-
phrases areoften used to introduce new referents. We’ll tal k moreabout new and old information in Ch. 21.

DRAFTSection14.4. ProblemswithPCFGs 473
Because PCFGs don’t allow a rule probability to be condition ed on surrounding
context, this equal probability is all we get; there is no way to capture the fact that in
subject position, the probability for NP→PRPshould go up to .91, while in object
position,theprobabilityfor NP→DT NNshouldgoupto .66.
These dependencies could be captured if the probability of e xpanding an NP as a
pronoun(e.g., NP→PRP)versusa lexicalNP (e.g., NP→DTNN) wereconditioned
onwhethertheNPwasasubjectoranobject. Sec.14.5willint roducethetechniqueof
parentannotation foraddingthiskindofconditioning.
14.4.2 Lackofsensitivityto lexicaldependencies
A second class of problemswith PCFGs is their lack of sensiti vity to the words in the
parse tree. Words do play a role in PCFGs, since the parse prob ability includes the
probability of a word given a part-of-speech (i.e., from rul es likeV→sleep,NN→
book,etc).
But it turns out that lexical information is useful in other p laces in the grammar,

such as in resolving prepositional phrase attachment ( PP) ambiguities. Since prepo-
sitional phrases in English can modify a noun phrase or a verb phrase, when a parser
ﬁnds a prepositional phrase, it must decide where to attach i t into the tree. Consider
thefollowingexample:
(14.19)Workersdumpedsacksintoa bin.
Fig. 14.5 shows two possible parse trees for this sentence; t he one on the left is
the correct parse; Fig. 14.6 shows another perspective on th e preposition attachment
problem, demonstrating that resolving the ambiguity in Fig . 14.5 is equivalent to de-

ciding whether to attach the prepositional phrase into the r est of the tree at the NP or
VP nodes; we say that the correct parse requires VP attachment while the incorrect VPattachment
parseimplies NP attachment . NP attachment
S
NP
NNS
workersVP
VBD
dumpedNP
NNS
sacksPP
P
intoNP
DT
aNN
binS
NP
NNS
workersVP
VBD
dumpedNP
NP
NNS
sacksPP
P
intoNP
DT
aNN
bin
Figure14.5 Two possible parse trees for a prepositional phrase attachment ambiguity . The left parse is the

S
NP
NNS
workersVP
VBD
dumpedNP
NNS
sacksPP
P
intoNP
DT
aNN
binS
NP
NNS
workersVP
VBD
dumpedNP
NP
NNS
sacksPP
P
intoNP
DT
aNN
bin
sensible one, inwhich‘intoabin’describes the resultingl ocationofthe sacks. Inthe rightincorrect parse,thesacks
tobe dumped are the ones whichare already ‘intoabin’, whate ver thatcould mean.

DRAFT474 Chapter 14. Statistical Parsing
S
NP VP
NNS VBD NP PP
workers NNS P NP
dumped into DT NN
sacks a bin
Figure14.6 Another view of the preposition attachment problem; should the PP on the right attach to the VP or
NPnodes of the partial parse tree onthe left?
Why doesn’t a PCFG already deal with PP attachment ambiguiti es? Note that the
two parse trees in Fig. 14.5 have almost the exact same rules; they differ only in that
theleft-handparsehasthisrule:
VP→VBDNPPP
whiletheright-handparsehasthese:
VP→VBDNP
NP→NPPP
Depending on how these probabilities are set, a PCFG will alwayseither prefer NP
attachmentor VP attachment. As it happens,NP attachmentis slightly more common
in English, and so if we trained these rule probabilities on a corpus, we might always
preferNP attachment,causingustomisparsethissentence.
But supposewe set the probabilitiesto preferthe VP attachm entfor this sentence.
Nowwe wouldmisparsethefollowingsentencewhichrequires NP attachment:
(14.20)ﬁshermencaughttonsofherring

VP→VBDNP
NP→NPPP
What is the information in the input sentence which lets us kn ow that (14.20) re-
quiresNP attachmentwhile(14.19)requiresVP attachment?
Itshouldbeclearthatthesepreferencescomefromtheident itiesoftheverbs,nouns
andprepositions. Itseemsthattheafﬁnitybetweentheverb dumpedandthepreposition
intois greater than the afﬁnity between the noun sacksand the preposition into, thus
leadingto VP attachment. On the otherhandin (14.20), the af ﬁnity between tonsand
ofisgreaterthanthat between caughtandof, leadingtoNP attachment.

Thusinordertogetthecorrectparseforthesekindsofexamp les,weneedamodel
which somehow augments the PCFG probabilities to deal with t heselexical depen-
dencystatistics fordifferentverbsandprepositions.Lexicaldependency
Coordinationambiguitiesare another case where lexical de pendenciesare the key
to choosing the proper parse. Fig. 14.7 shows an example from Collins (1999), with
two parses for the phrase dogs in houses and cats . Because dogsis semantically a
better conjunct for catsthanhouses(and because dogs can’t ﬁt inside cats) the parse

DRAFTSection14.5. ImprovingPCFGs bySplittingandMergingNont erminals 475
[dogsin[ NPhousesandcats]] isintuitivelyunnaturalandshouldbedispreferred. The
two parses in Fig. 14.7, however,have exactly the same PCFG r ules and thusa PCFG
will assignthemthesame probability.
(a) NP (b) NP
NP Conj NP NP PP
NP PP and Noun Noun Prep NP
Noun Prep NP cats dogs in NP Conj NP
dogs in Noun Noun and Noun
houses houses cats
Figure14.7 An instance of coordination ambiguity. Although the left st ructure is intuitively
thecorrectone,aPCFGwillassignthemidenticalprobabili tiessincebothstructureusetheexact
same rules. After Collins(1999).
In summary, we have shown in this section and the previousone that probabilistic
context-freegrammarsareincapableofmodelingimportant structural andlexicalde-
pendencies. Inthenexttwosectionswesketchcurrentmetho dsforaugmentingPCFGs
todealwith boththese issues.
14.5 Improving PCFGs by Splitting and Merging Nonter-
minals

minals
Let’s start with the ﬁrst of the two problems with PCFGs menti oned above: their in-
abilityto modelstructuraldependencies,like the fact tha t NPsin subjectpositiontend
tobepronouns,whereNPsinobjectpositiontendtohavefull lexical(non-pronominal)
form. How could we augment a PCFG to correctly model this fact ? One idea would
be tosplitthe NP non-terminal into two versions: one for subjects, one for objects. Split
Having two nodes (e.g., NPsubjectandNPobject) would allow us to correctly model
theirdifferentdistributionalproperties,since we would havedifferentprobabilitiesfor

theruleNPsubject→PRPandtherule NPobject→PRP.
One way to implement this intuition of splits is to do parent annotation (John- Parent annotation
son, 1998b), in which we annotate each node with its parent in the parse tree. Thus a
nodeNP which is the subject of the sentence, and hencehas par entS, would be anno-
tated NPˆS, while a direct object NP, whose parent is VP, woul d be annotated NPˆVP.
Fig. 14.8showsanexampleofa tree producedbya grammarthat parentannotatesthe
phrasalnon-terminals(like NPandVP).

In addition to splitting these phrasal nodes, we can also imp rovea PCFG by split-
ting the preterminal part-of-speech nodes (Klein and Manni ng, 2003b). For example,
differentkindsof adverbs(RB) tend to occurin differentsy ntactic positions: the most

DRAFT476 Chapter 14. Statistical Parsing
a) S
NP
PRP
IVP
VBD
needNP
DT
aNN
ﬂightb) S
NPˆS
PRP
IVPˆS
VBD
needNPˆVP
DT
aNN
ﬂight
Figure14.8 A standard PCFG parse tree (a) and one which has parent annotation on the
nodes which aren’t preterminal (b). All the non-terminal no des (except the preterminal part-of-
speech nodes) inparse (b) have been annotated withthe ident ityof their parent.
common adverbs with ADVP parents are alsoandnow, with VP parents are n’tand
not, and with NP parents onlyandjust. Thusadding tags like RBˆADVP, RBˆVP, and
RBˆNP canbeusefulinimprovingPCFG modeling.
Similarly, the Penn Treebank tag IN is used to mark a wide vari ety of parts-of-
speech, including subordinating conjunctions ( while,as,if), complementizers ( that,
for), and prepositions ( of,in,from). Some of these differences can be captured by
parent annotation (subordinating conjunctions occur unde r S, prepositions under PP),
while others require speciﬁcally splitting the pre-termin al nodes. Fig. 14.9 shows an

example from Klein and Manning (2003b), where even a parent a nnotated grammar
incorrectlyparses worksasanounin toseeifadvertisingworks . Splittingpreterminals
toallowifto preferasententialcomplementresultsinthe correctver balparse.
VPˆS
TO
toVPˆVP
VB
seePPˆVP
IN
ifNPˆPP
NN
advertisingNNS
worksVPˆS
TOˆVP
toVPˆVP
VBˆVP
seeSBARˆVP
INˆSBAR
ifSˆSBAR
NPˆS
NNˆNP
advertisingVPˆS
VBZˆVP
works
Figure14.9 An incorrect parse even with a parent annotated parse (left) . The correct parse (right), was produced

VPˆS
TO
toVPˆVP
VB
seePPˆVP
IN
ifNPˆPP
NN
worksVPˆS
TOˆVP
toVPˆVP
VBˆVP
INˆSBAR
ifSˆSBAR
NPˆS
NNˆNP
VBZˆVP
works
byagrammarinwhichthepre-terminalnodes havebeensplit, allowingtheprobabilisticgrammartocapture thefact
thatifprefers sentential complements; adapted from KleinandMan ning (2003b).
In order to deal with cases where parent annotation is insufﬁ cient, we can also
hand-writerulesthatspecify a particularnodesplit based onotherfeaturesofthe tree.
For example to distinguish between complementizer IN and su bordinating conjunc-

DRAFTSection14.6. ProbabilisticLexicalizedCFGs 477
tion IN, both of which can have the same parent, we could write rules conditionedon
other aspects of the tree such as the lexical identity (the le xemethatis likely to be a
complementizer, asasubordinatingconjunction).
Node-splitting is not without problems; it increases the si ze of the grammar, and
hence reduces the amount of training data available for each grammarrule, leading to
overﬁtting. Thus it is important to split to just the correct level of granularity for a
particulartrainingset. Whileearlymodelsinvolvedhand- writtenrulestotrytoﬁndan
optimal number of rules (Klein and Manning, 2003b), modern m odels automatically
search for the optimal splits. The split and merge algorithm of Petrov et al. (2006), Splitandmerge
for example starts with a simple X-bar grammar, and then alte rnately splits the non-
terminals,andmergestogethernon-terminals,ﬁndingthes etofannotatednodeswhich

maximizes the likelihood of the training set treebank. As of the time of this writing,
theperformanceofthe Petrovet al.(2006)algorithmastheb est ofanyknownparsing
algorithmonthe PennTreebank.
14.6 ProbabilisticLexicalizedCFGs
The previous section showed that a simple probabilistic CKY algorithm for parsing
raw PCFGs can achieveextremelyhigh parsingaccuracyif the grammarrulesymbols
areredesignedviaautomaticsplitsandmerges.
Inthissection,wediscussanalternativefamilyofmodelsi nwhichinsteadofmod-

ifyingthe grammarrules, we modifythe probabilisticmodel of the parser to allow for
lexicalized rules. The resulting family of lexicalized parsers include s the well-known
Collinsparser (Collins,1999)and Charniakparser (Charniak,1997),bothofwhich Collinsparser
Charniak parser arepubliclyavailableandwidelyusedthroughoutnaturall anguageprocessing.
WesawinSec.12.4.4inCh.12thatsyntacticconstituentsco uldbeassociatedwith
a lexicalhead, and we deﬁned a lexicalized grammar in which each non-terminalinLexicalizedgrammar

the tree is annotatedwith its lexical head, where a rule like VP→VBDNPPP would
beextendedas:
VP(dumped)→VBD(dumped) NP(sacks) PP(into) (14.21)
In the standard type of lexicalized grammar we actually make a further extension,
which is to associate the head tag , the part-of-speech tags of the headwords, with Head tag
the nonterminal symbols as well. Each rule is thus lexicaliz ed by both the headword
andtheheadtag ofeachconstituentresultingina formatfor lexicalizedruleslike:
VP(dumped,VBD) →VBD(dumped,VBD) NP(sacks,NNS) PP(into,IN) (14.22)

WeshowalexicalizedparsetreewithheadtagsinFig.14.10, extendedfromFig.12.12.
Inorderto generatesuch a lexicalizedtree,each PCFG rulem ust be augmentedto
identify one right-hand side constituent to be the head daug hter. The headword for a
node is then set to the headword of its head daughter, and the h ead tag to the part-of-
speech tag of the headword. Recall that we gave in Fig. 12.13 a set of hand-written
rulesforidentifyingtheheadsofparticularconstituents .

DRAFT478 Chapter 14. Statistical Parsing
TOP
S(dumped,VBD)
NP(workers,NNS)
NNS(workers,NNS)
workersVP(dumped,VBD)
VBD(dumped,VBD)
dumpedNP(sacks,NNS)
NNS(sacks,NNS)
sacksPP(into,P)
P(into,P)
intoNP(bin,NN)
DT(a,DT)
aNN(bin,NN)
bin
InternalRules Lexical Rules
TOP →S(dumped,VBD) NNS(workers,NNS) →workers
S(dumped,VBD) →NP(workers,NNS) VP(dumped,VBD) VBD(dumped,VBD) →dumped
NP(workers,NNS) →NNS(workers,NNS) NNS(sacks,NNS) →sacks
VP(dumped,VBD) →VBD(dumped, VBD) NP(sacks,NNS) PP(into,P) P(into,P) →into
PP(into,P) →P(into,P) NP(bin,NN) DT(a,DT) →a
NP(bin,NN) →DT(a,DT) NN(bin,NN) NN(bin,NN) →bin
Figure14.10 A lexicalizedtree,including headtags,fora WSJsentence, adaptedfrom Collins(1999). Belowwe
showthePCFGrulesthatwouldbeneededforthisparsetree,i nternalrulesontheleft,andlexicalrulesontheright.
A natural way to think of a lexicalized grammaris like parent annotation,i.e. as a
simplecontext-freegrammarwithmanycopiesofeachrule,o necopyforeachpossible

headword/headtagforeachconstituent. Thinkingofaproba bilisticlexicalizedCFGin
thiswaywouldleadtothesetofsimplePCFGrulesshownbelow thetreeinFig.14.10.
Note that Fig. 14.10 shows two kinds of rules: lexical rules , which express the Lexicalrules
expansion of a preterminal to a word, and internal rules , which express the other Internal rule
rule expansions. We need to distinguish these kinds of rules in a lexicalized gram-
mar because they are associated with verydifferent kindsof probabilities. The lexical

rules are deterministic, i.e., have probability 1.0, since a lexicalized preterminal like
NN(bin,NN)can onlyexpandto the word bin. But for the internal ruleswe will need
toestimate probabilities.
SupposeweweretotreataprobabilisticlexicalizedCFGlik eareallybigCFGthat
justhappenedtohavelotsofverycomplexnon-terminalsand estimatetheprobabilities
for each rule from maximum likelihood estimates. Thus, usin g Eq. 14.18, the MLE
estimate for the probability for the rule P(VP(dumped,VBD) →VBD(dumped, VBD)

NP(sacks,NNS)PP(into,P)) wouldbe:

DRAFTSection14.6. ProbabilisticLexicalizedCFGs 479
Count(VP(dumped ,VBD)→VBD(dumped ,VBD)NP(sacks,NNS)PP(into,P))
Count(VP(dumped ,VBD))(14.23)
But there’s no way we can get good estimates of counts like tho se in (14.23), be-
cause they are so speciﬁc: we’re very unlikely to see many (or even any) instances of
a sentence with a verbphraseheadedby dumpedthat hasone NP argumentheadedby
sacksand a PP argument headed by into. In other words, counts of fully lexicalized
PCFG rules like this will be far too sparse and most rule proba bilities will come out
zero.
Theideaof lexicalizedparsingis tomake somefurtherindep endenceassumptions
tobreakdowneachrule,so thatwe wouldestimatethe probabi lity
P(VP(dumped ,VBD)→VBD(dumped ,VBD)NP(sacks,NNS)PP(into,P))
(14.24)
astheproductofsmallerindependentprobabilityestimate sforwhichwecouldacquire
reasonablecounts. The nextsection summarizesonesuchmet hod,the Collinsparsing
method.
14.6.1 The CollinsParser

zero.
(14.24)
method.
Modernstatisticalparsersdifferinexactlywhichindepen denceassumptionstheymake.
In this section we describe a simpliﬁed version of Collins’s (1999)Model 1, but there
are a number of other parsers that are worth knowing about; se e the summary at the
endofthechapter.
TheﬁrstintuitionoftheCollinsparseristothinkoftherig ht-handsideofevery(in-
ternal)CFG ruleas consistingofa headnon-terminal,toget herwith the non-terminals
to the left of the head, and the non-terminals to the right of t he head. In the abstract,

we thinkaboutthese rulesasfollows:
LHS→LnLn−1...L1HR1...Rn−1Rn (14.25)
Sincethisisa lexicalizedgrammar,eachofthesymbolslike L1orR3orHorLHS
is actually a complexsymbolrepresentingthe categoryand i ts head and headtag, like
VP(dumped,VP) orNP(sacks,NNS) .
Now instead of computing a single MLE probability for this ru le, we are going to
breakdownthisruleviaaneatgenerativestory,aslightsim pliﬁcationofwhatiscalled
Collins Model 1. This new generative story is that given the l eft-hand side, we ﬁrst

generatetheheadoftherule,andthengeneratethedependen tsofthehead,onebyone,
fromtheinsideout. Eachofthesegenerationstepswill have itsownprobability.
We are also going to add a special STOPnon-terminal at the left and right edges
of the rule; this non-terminal will allow the model to know wh en to stop generating
dependentsonagivenside. We’llgeneratedependentsonthe leftsideoftheheaduntil
we’vegenerated STOPontheleft side ofthehead,at whichpointwe movetotheright

side of the head and start generating dependentsthere until we generate STOP. So it’s

DRAFT480 Chapter 14. Statistical Parsing
asifwe aregeneratinga ruleaugmentedasfollows:
P(VP(dumped ,VBD)→ (14.26)
STOPVBD(dumped ,VBD)NP(sacks,NNS)PP(into,P)STOP
Let’s see the generativestory for this augmentedrule. We’r e goingto make use of
three kindsof probabilities: PHfor generatingheads, PLfor generatingdependentson
theleft,and PRforgeneratingdependentsontheright.
1) Firstgenerate the head VBD(dumped,VBD) with probabilit y
P(H|LHS) =P(VBD(dumped,VBD) |VP(dumped,VBD))VP(dumped,VBD)
2) Then generate the left dependent (which is STOP, since the re
isn’t one) with probability
P(STOP|VP(dumped,VBD) VBD(dumped,VBD)VP(dumped,VBD)
STOP VBD(dumped,VBD)
3) Then generate right dependent NP(sacks,NNS) with probab ility
Pr(NP(sacks,NNS|VP(dumped,VBD), VBD(dumped,VBD))VP(dumped,VBD)
STOP VBD(dumped,VBD) NP(sacks,NNS)
4) Then generate theright dependent PP(into,P) with probab ility
Pr(PP(into,P)|VP(dumped,VBD), VBD(dumped,VBD))VP(dumped,VBD)
STOP VBD(dumped,VBD) NP(sacks,NNS) PP(into,P)

5) Finally generate the right dependent STOPwith probabili ty
Pr(STOP|VP(dumped,VBD), VBD(dumped,VBD))VP(dumped,VBD)
STOP VBD(dumped,VBD) NP(sacks,NNS) PP(into,P) STOP
Insummary,theprobabilityofthisrule:
P(VP(dumped ,VBD)→ (14.27)
VBD(dumped ,VBD)NP(sacks,NNS)PP(into,P) )
isestimatedas:
PH(VBD|VP,dumped )×PL(STOP|VP,VBD,dumped ) (14.28)
×PR(NP(sacks,NNS)|VP,VBD,dumped )
×PR(PP(into,P)|VP,VBD,dumped )
×PR(STOP|VP,VBD,dumped )
Each of these probabilities can be estimated from much small er amounts of data

thanthefullprobabilityin(14.28). Forexample,themaxim umlikelihoodestimatefor
thecomponentprobability PR(NP(sacks,NNS)|VP,VBD,dumped )is:
Count(VP(dumped ,VBD)withNNS(sacks)as a daughter somewhere onthe right )
Count(VP(dumped ,VBD) )
(14.29)

DRAFTSection14.6. ProbabilisticLexicalizedCFGs 481
These counts are much less subject to sparsity problems than complex counts like
thosein(14.28).
More generally, if we use hto mean a headword together with its tag, lto mean a
word+tagontheleftand rtomeanaword+tagontheright,theprobabilityofanentire
rulecanbeexpressedas:
1. Generatetheheadofthephrase H(hw,ht)withprobability PH(H(hw,ht)|P,hw,ht)
2. Generatemodiﬁersto theleftoftheheadwithtotal probab ility:
n+1
∏
i=1PL(Li(lwi,lti)|P,H,hw,ht)
suchthatLn+1(lwn+1,ltn+1) =STOP,andwestopgeneratingoncewe’vegen-
erateda STOPtoken.
3. Generatemodiﬁersto therightofthe headwithtotal proba bility:
n+1
∏
i=1PP(Ri(rwi,rti)|P,H,hw,ht)
such that Rn+1(rwn+1,rtn+1) =STOP, and we stop generating once we’ve
generateda STOPtoken.
14.6.2 Advanced: Further Details ofthe CollinsParser
The actual Collins parser models are more complex (in a coupl e of ways) than the
simple model presented in the previous section. Collins Mod el 1 includes a distance Distance

n+1
∏
feature. Thusinsteadofcomputing PLandPRasfollows:
PL(Li(lwi,lti)|P,H,hw,ht) (14.30)
PR(Ri(rwi,rti)|P,H,hw,ht) (14.31)
CollinsModel1conditionsalso ona distancefeature:
PL(Li(lwi,lti)|P,H,hw,ht,distance L(i−1)) (14.32)
PR(Ri(rwi,rti)|P,H,hw,ht,distance R(i−1)) (14.33)
Thedistancemeasureisafunctionofthesequenceofwords belowthepreviousmodi-
ﬁers(i.e. thewordswhicharetheyieldofeachmodiﬁernon-t erminalwehavealready
generatedontheleft). Fig. 14.11,adaptedfromCollins(20 03)showsthecomputation

oftheprobability P(R2(rh2,rt2)|P,H,hw,ht,distance R(1)):
The simplest versionof this distance measure is just a tuple of two binary features
based on the surface string below these previous dependenci es: (1) is the string of
length zero? (i.e. were no previous words generated?) (2) do es the string contain a
verb?
Collins Model 2 adds more sophisticated features, conditio ningon subcategoriza-
tionframesforeachverb,anddistinguishingargumentsfro madjuncts.

DRAFT482 Chapter 14. Statistical Parsing
P(hw,ht)
H(hw,ht)
...h...R1(rw1,rt1)
|←distance→|R2(rw2,rt2)
Figure14.11 The next child R2is generated with probability
P(R2(rh2,rt2)|P,H,hw,ht,distance R(1)). The distance is the yield of the previous depen-
dent nonterminal R1. Had there been another intervening dependent, its yield wo uld have been
included as well. Adapted from Collins (2003).
Finally,smoothingisasimportantforstatistical parsers asit wasfor N-grammod-
els. This is particularly true for lexicalized parsers, sin ce (even using the Collins or
other methods of independenceassumptions) the lexicalize d rules will otherwise con-
ditiononmanylexicalitemsthatmayneveroccurintraining .
Consider the probability PR(Ri(rwi,rti)|P,hw,ht). What do we do if a particular
right-handside constituent never occurs with this head? Th e Collins model addresses
this problemby interpolatingthree backed-offmodels: ful ly lexicalized (conditioning
ontheheadword),backingofftojust theheadtag,andaltoge therunlexicalized:

BackoffLevel PR(Ri(rwi,rti|...) Example
1 PR(Ri(rwi,rti)|P,hw,ht)PR(NP(sacks,NNS)|VP,VBD, dumped)
2 PR(Ri(rwi,rti)|P,ht) PR(NP(sacks,NNS)|VP,VBD)
3 PR(Ri(rwi,rti)|P) PR(NP(sacks,NNS)|VP)
Similar backoff models are built also for PLandPH. Although we’ve used the
word ‘backoff’, in fact these are not backoff models but inte rpolated models. The
three models above are linearly interpolated, where e1,e2, ande3are the maximum
likelihoodestimatesofthe threebackoffmodelsabove:
PR(...) =λ1e1+(1−λ1)(λ2e2+(1−λ2)e3) (14.34)

The values of λ1andλ2are set to implement Witten-Bell discounting (Witten and
Bell, 1991)followingBikel etal. (1997).
Unknown words are dealt with in the Collins model by replacin g any unknown
wordinthetest set,andanywordoccurringlessthan6timesi nthetrainingset, witha
special UNKNOWN word token. Unknownwords in the test set are assigned a part- of-
speech tag in a preprocessing step by the Ratnaparkhi (1996) tagger; all other words
aretaggedaspartoftheparsingprocess.

The parsing algorithm for the Collins model is an extension o f probabilistic CKY;
seeCollins(2003). ExtendingtheCKYalgorithmtohandleba siclexicalizedprobabil-
itiesisleft asanexerciseforthereader.

DRAFTSection14.7. EvaluatingParsers 483
14.7 EvaluatingParsers
The standard techniques for evaluating parsers and grammar s are called the PARSE-
VALmeasures,andwereproposedbyBlacketal.(1991)basedo nthesameideasfrom
signal-detection theory that we saw in earlier chapters. Th e intuition of the PARSE-
VALmetricistomeasurehowmuchthe constituents inthehypothesisparsetreelook
like theconstituentsin a hand-labeledgoldreferencepars e. PARSEVAL thusassumes
we have a human-labeled “gold standard” parse tree for each s entence in the test set;
we generallydrawthesegoldstandardparsesfromatreebank like thePennTreebank.
Given these gold standard reference parses for a test set, a g iven constituent in a
hypothesis parse Chof a sentence sis labeled “correct” if there is a constituent in the
referenceparse Crwiththesamestartingpoint,endingpoint,andnon-termina lsymbol.
We can then measure the precision and recall just as we did for chunking in the
previouschapter.
labeledrecall: =# ofcorrectconstituentsinhypothesisparseof s

# ofcorrectconstituentsinreferenceparseof s
labeledprecision: =#ofcorrectconstituentsinhypothesisparseof s
# oftotal constituentsin hypothesisparseof s
Aswithotherusesofprecisionandrecall,insteadofreport ingthemseparately,we
oftenreportasinglenumber,the F-measure (vanRijsbergen,1975): TheF-measureis F-measure
deﬁnedas:
β2P+R

deﬁnedas:
β2P+R
F1=2PR
P+R(14.35)
The F-measure derives from a weighted harmonic mean of preci sion and recall.
Recall that the harmonic mean of a set of numbers is the recipr ocal of the arithmetic
meanofthe reciprocals:
1
a11
a21
a3...1
an(14.36)
F=1
1
αP×1
(1−α)Ror(
α)
β2P+R(14.37)
We additionallyuse anewmetric,crossingbrackets,foreac hsentence s:

DRAFT484 Chapter 14. Statistical Parsing
cross-brackets: the numberofconstituentsforwhich thereferenceparsehas a brack-
eting such as ((A B) C) but the hypothesisparse has a bracketi ngsuch as (A (B
C)).
As of the time of this writing, the performance of modern pars ers that are trained
and tested on the Wall Street Journal treebank is somewhat hi gher than 90% recall,
90%precision,andabout1%cross-bracketedconstituentsp ersentence.
For comparing parsers which use different grammars, the PAR SEVAL metric in-
cludes a canonicalization algorithm for removing informat ion likely to be grammar-
speciﬁc(auxiliaries,pre-inﬁnitival“to”,etc.) andforc omputingasimpliﬁedscore. The
interested reader should see Black et al. (1991). The canoni cal publicly-available im-
plementationofthePARSEVAL metricsis called evalb(SekineandCollins, 1997). evalb
You might wonder why we don’t evaluate parsers by measuring h ow many sen-
tencesare parsed correctly, instead of measuring constituent accuracy. The reason we

use constituents is that measuring constituents gives us a m ore ﬁne-grained metric.
Thisisespeciallytrueforlongsentences,wheremostparse rsdon’tgetaperfectparse.
If we just measured sentence accuracy, we wouldn’t be able to distinguish between a
parse that got most of the constituents wrong, and one that ju st got one constituent
wrong.
Nonetheless, constituents are not always an optimal domain for parser evaluation.
For example, using the PARSEVAL metrics requires that our pa rser produce trees in

wrong.
the exact same format as the gold standard. That means that if we want to evaluate a
parser which produces different styles of parses (dependen cyparses, or LFG feature-
structures, etc.) against say the Penn Treebank (or against another parser which pro-
duces Treebank format), we need to map the output parses into Treebank format. A
related problem is that constituency may not be the level we c are the most about. We
mightbemoreinterestedinhowwelltheparserdoesatrecove ringgrammaticaldepen-

dencies (subject, object, etc), which could give us a better metric for how useful the
parseswould be to semantic understanding. For these purpos eswe can use alternative
evaluation metrics based on measuring the precision and rec all of labeled dependen-
cies, where the labels indicate the grammatical relations ( Lin, 1995; Carroll et al.,
1998; Collins et al., 1999). Kaplan et al. (2004), for exampl e, compared the Collins
(1999)parser with the Xerox XLE parser (Riezler et al., 2002 ),which producesmuch

richer semantic representations,by convertingboth parse trees to a dependencyrepre-
sentation.
14.8 Advanced: DiscriminativeReranking
The models we have seen of parsing so far, the PCFG parser and t he Collins lexical-
ized parser, are generative parsers. By this we mean that the probabilistic model im-
plemented in these parsers gives us the probability of gener atinga particular sentence
byassigninga probabilityto eachchoicethe parsercouldma kein thisgenerationpro-
cedure.

DRAFTSection14.8. Advanced: DiscriminativeReranking 485
Generative models have some signiﬁcant advantages; they ar e easy to train using
maximum likelihood and they give us an explicit model of how d ifferent sources of
evidencearecombined. Butgenerativeparsingmodelsalsom akeithardtoincorporate
arbitrarykindsofinformationintotheprobabilitymodel. Thisisbecausetheprobabil-
ityisbasedonthegenerativederivationofasentence;itis difﬁculttoaddfeaturesthat
arenotlocaltoa particularPCFG rule.
Consider for example how to represent global facts about tre e structure. Parse
trees in English tend to be right-branching; we’d therefore like our model to assign
a higher probability to a tree which is more right-branching , all else being equal. It
is also the case that heavy constituents (those with a large n umber of words) tend to
appear later in the sentence. Or we might want to conditionou r parse probabilitieson
global facts like the identity of the speaker (perhapssome s peakers are more likely to

use complex relative clauses, or use the passive). Or we migh t want to condition on
complex discourse factors across sentences. None of these k inds of global factors is
trivialtoincorporateintothegenerativemodelswehavebe enconsidering. Asimplistic
modelthatforexamplemakeseachnon-terminaldependenton howright-branchingthe
tree is in the parse so far, or makes each NP non-terminal sens itive to the number of
relativeclausesthespeakerorwriterusedinprevioussent ences,wouldresultincounts
thatarefartoosparse.
We discussed this problem in Ch. 6, where the need for these ki nds of global

features motivated the use of log-linear (MEMM) models for P OS tagging instead
of HMMs. For parsing, there are two broad classes of discrimi native models: dy-
namic programmingapproachesand two-stage modelsof parsi ng that use discrimina-
tivereranking. We’lldiscussdiscriminativererankingin therestofthissection;seethe
endofthechapterforpointerstodiscriminativedynamicpr ogrammingapproaches.
In the ﬁrst stage of a discriminative rerankingsystem, we ca n run a normal statis-

tical parser ofthe typewe’vedescribedso far. But instead o f just producingthe single
best parse, we modify the parser to producea ranked list of pa rses together with their
probabilities. We call this ranked list of Nparses the N-best list (theN-best list was N-best list
ﬁrst introduced in Ch. 9 when discussing multiple-pass deco ding models for speech
recognition). Therearevariouswaystomodifystatistical parserstoproducean N-best
listofparses;seetheendofthechapterforpointerstothel iterature. Foreachsentence

in the training set and the test set, we run this N-best parser and produce a set of N
parse/probabilitypairs.
The second stage of a discriminative reranking model is a cla ssiﬁer which takes
each of these sentences with their Nparse/probability pairs as input, extracts some
largeset offeaturesandchoosesthesingle best parsefromt heN-best list. We can use
any type of classiﬁer for the reranking, such as the log-line ar classiﬁers introduced in
Ch.6.
A wide variety of features can be used for reranking. One impo rtant feature to

Ch.6.
include is the parse probabilityassigned by the ﬁrst-stage statistical parser. Other fea-
turesmightincludeeachoftheCFG rulesinthetree,thenumb erofparallelconjuncts,
howheavy eachconstituentis, measuresofhow right-branch ingthe parsetree is, how
manytimesvarioustreefragmentsoccur,bigramsofadjacen tnon-terminalsinthetree,
andsoon.
The two-stage architecture has a weakness: the accuracy rat e of the complete ar-

DRAFT486 Chapter 14. Statistical Parsing
chitecture can neverbe better than the accuracyrate of the b est parse in the ﬁrst-stage
N-best list. This is because the reranking approach is merely choosing one of the N-
bestparses;evenifwepickedtheverybestparseinthelist, wecan’tget100%accuracy
ifthe correctparseisn’tinthelist! Thereforeit isimport anttoconsiderthe ceiling or-
acle accuracy (often measured in F-measure) of the N-best list. The oracle accuracy Oracle accuracy
(F-measure) of a particular N-best list is the accuracy(F-measure) we get if we chose
theparsethathadthehighestaccuracy. Wecallthisan oracleaccuracybecauseitrelies
on perfect knowledge(as if from an oracle) of which parse to p ick.2Of course it only
makes sense to implement discriminative reranking if the N-best F-measure is higher
than the 1-best F-measure. Luckily this is often the case; fo r example the Charniak
(2000) parser has an F-measure of 0.897 on section 23 of the Pe nn Treebank, but the

Charniak and Johnson (2005) algorithm for producing the 50- best parses has a much
higheroracleF-measureof0.968.
14.9 Advanced: Parser-BasedLanguageModeling
We said earlier that statistical parsers can take advantage of longer-distance informa-
tion than N-grams, which suggests that they might do a better job at lang uage model-
ing/wordprediction. It turnsoutthatif wehavea verylarge amountoftrainingdata,a
4-gramor 5-gram grammaris nonethelessstill the best way to do languagemodeling.

But in situations where there is not enough data for such huge models, parser-based
language models are beginning to be developed which have hig her accuracy N-gram
models.
Two common applications for language modeling are speech re cognition and ma-
chine translation. The simplest way to use a statistical par ser for language modeling
foreitheroftheseapplicationsisviaatwo-stagealgorith mofthetypediscussedinthe
previoussectionandinSec.10.1. Intheﬁrststage,werunan ormalspeechrecognition

models.
decoder,ormachinetranslationdecoder,usinganormal N-gramgrammar. Butinstead
of just producing the single best transcription or translat ion sentence, we modify the
decoder to produce a ranked N-best list of transcriptions/translations sentences, eac h
onetogetherwith itsprobability(or,alternatively,a lat tice).
Theninthesecondstage,werunourstatisticalparserandas signaparseprobability
to each sentence in the N-best list or lattice. We then rerank the sentences based on

this parse probability and choose the single best sentence. This algorithm can work
better than using a simple trigram grammar. For example, on t he task of recognizing
spoken sentences from the Wall Street Journal using this two -stage architecture, the
probabilities assigned by the Charniak (2001) parser impro ved the word error rate by
about2percentabsolute,overasimpletrigramgrammarcomp utedon40millionwords
(Hall and Johnson, 2003). We can either use the parse probabi lities assigned by the

parseras-is,orwe canlinearlycombineit withtheoriginal N-gramprobability.
An alternative to the two-pass architecture, at least for sp eech recognition, is to
modifytheparsertorunstrictlyleft-to-right,sothatitc anincrementallygivetheproba-
2Weintroduced this same oracle idea in Ch. 9when wetalked abo ut thelattice error rate .

DRAFTSection14.10. HumanParsing 487
bilityofthenextwordinthesentence. Thiswouldallowthep arsertobeﬁtdirectlyinto
the ﬁrst-pass decoding pass and obviate the second-pass alt ogether. While a number
of such left-to-right parser-based language modeling algo rithms exist (Stolcke, 1995;
Jurafsky et al., 1995; Roark, 2001; Xu et al., 2002), it is fai r to say that it is still early
daysforthe ﬁeldofparser-basedstatistical languagemode ls.
14.10 HumanParsing
Are the kinds of probabilistic parsing models we have been di scussing also used by
humans when they are parsing? This question lies in a ﬁeld cal ledhuman sentence
processing ? Recent studies suggest that there are at least two ways in wh ich humansHuman sentenceprocessing
apply probabilistic parsing algorithms, although there is still disagreement on the de-
tails.
One family of studies has shown that when humans read, the pre dictability of a
word seems to inﬂuence the reading time ; more predictable words are read more Readingtime

tails.
quickly. One way of deﬁning predictability is from simple bi gram measures. For
example,Scott andShillcock (2003)hadparticipantsreads entenceswhile monitoring
their gaze with an eye-tracker . They constructed the sentences so that some would
have a verb-noun pair with a high bigram probability (such as (14.38a)) and others a
verb-nounpairwitha lowbigramprobability(suchas(14.38 b)).
(14.38) a)HIGHPROB: Onewayto avoidconfusion istomakethechanges
duringvacation;
b)LOW PROB: Oneway to avoiddiscovery isto makethechanges
duringvacation

Theyfoundthatthehigherthebigrampredictabilityofawor d,theshorterthetime
thatparticipantslookedat theword(the initial-ﬁxationduration ).
While this result onlyprovidesevidencefor N-gramprobabilities,morerecentex-
perimentshavesuggestedthattheprobabilityofanupcomin gwordgiventhesyntactic
parse of the preceding sentence preﬁx also predicts word rea ding time Hale (2006),
Levy(2007).
Interestingly, this effect of probability on reading time h as also been shown for
morphologicalstructure; the time to recognize a word is inﬂ uenced by entropy of the

wordandtheentropyoftheword’smorphologicalparadigmMo scosodelPradoMart´ ın
et al.(2004b).
The second family of studies has examined how humans disambi guate sentences
which have multiple possible parses, suggesting that human s prefer whichever parse
is more probable. These studies often rely on a speciﬁc class of temporarily ambigu-
oussentencescalled garden-path sentences. Thesesentences,ﬁrstdescribedbyBever Garden-path
(1970),aresentenceswhicharecleverlyconstructedtohav ethreepropertiesthatcom-

binetomakethemverydifﬁcultforpeopletoparse:
1. They are temporarily ambiguous : The sentence is unambiguous,but its initial
portionisambiguous.

DRAFT488 Chapter 14. Statistical Parsing
2. Oneofthetwoormoreparsesintheinitialportionissomeh owpreferabletothe
humanparsingmechanism.
3. Butthedispreferredparseisthe correctoneforthesente nce.
The result of these three properties is that people are “led d own the garden path”
towardtheincorrectparse,andthenareconfusedwhentheyr ealizeit’sthewrongone.
Sometimesthisconfusionisquiteconscious,asinBever’se xample(14.39);infactthis
sentenceissohardtoparsethatreadersoftenneedtobeshow nthecorrectstructure. In
thecorrectstructure racedispartofareducedrelativeclausemodifying Thehorse ,and
means“Thehorse[whichwasracedpastthebarn]fell”;thiss tructureisalsopresentin
thesentence“StudentstaughtbytheBerlitzmethoddoworse whentheygettoFrance”.
(14.39)Thehorseracedpastthebarnfell.
(a) S (b) S
NP VP NP VP
NP VP
PP PP
NP ? NP
Det N V P Det N V Det N V P Det N V
The horse raced past the barn fell The horse raced past the bar n fell
In Marti Hearst’s example (14.40), subjects often misparse the verbhousesas a

(a) S (b) S
NP VP NP VP
NP VP
PP PP
NP ? NP
noun (analyzing the complex houses as a noun phrase, rather than a noun phrase and
a verb). Other times the confusion caused by a garden-pathse ntence is so subtle that
it can only be measured by a slight increase in readingtime. T husin example(14.41)
readers often mis-parse the solution as the direct object of forgotrather than as the
subject of an embedded sentence. This mis-parse is subtle, a nd is only noticeable
because experimental participants take longer to read the w ordwasthan in control

sentences. This “mini-garden-path”effect at the word wassuggests that subjects had
chosen the direct object parse and had to re-analyze or rearr ange their parse now that
theyrealizetheyareina sententialcomplement.
(14.40)Thecomplexhousesmarriedandsinglestudentsandtheirfam ilies.
(a) S (b) S
NP NP VP
Det Adj N Det N V
The complex houses The complex houses
(14.41)Thestudentforgotthesolutionwasinthebackofthebook.

DRAFTSection14.11. Summary 489
(a) S (b) S
NP VP NP VP
S
NP ? NP VP
Det N V Det N V Det N V Det N V
The students forgot the solution was The students forgot the solution was
Whilemanyfactorsseemto playa roleinthese preferencesfo raparticular(incor-
rect)parse,atleastonefactorseemstobesyntacticprobab ilities,especiallylexicalized
(subcategorization)probabilities. For example, the prob ability of the verb forgottak-
ing a direct object ( VP→VNP) is higher than the probability of it taking a sentential
complement ( VP→VS); this difference causes readers to expect a direct object a fter
forgetand be surprised (longer reading times) when they encounter a sentential com-
plement. By contrast, a verb which prefersa sentential comp lement(like hope)didn’t
causeextrareadingtime at was.
Similarly,thegardenpathin(14.40)maybecausedbythefac tthatP(houses|Noun)>
P(houses|Verb)andP(complex|Adjective )>P(complex|Noun), and the garden path

in (14.39) at least partially by the low probability of the re duced relative clause con-
struction.
Besides grammatical knowledge,human parsing is affected b y many other factors
which we will describe later, including resource constrain ts (such as memory limita-
tions,tobediscussedinCh.15),thematicstructure(sucha swhetheraverbexpectsse-
manticagentsorpatients,tobediscussedinCh.19)anddiscourseconstraints(Ch.21 ).
14.11 Summary
This chapter has sketched the basics of probabilistic parsing, concentratingon prob-

abilistic context-free grammars andprobabilistic lexicalized context-free gram-
mars.
•Probabilistic grammars assign a probability to a sentence o r string of words,
while attempting to capture more sophisticated syntactic i nformation than the
N-gramgrammarsofCh.4.
•Aprobabilistic context-free grammar (PCFG) is a context-free
grammar in which every rule is annotated with the probabilit y of choosing that
rule. EachPCFGruleistreatedasifitwere conditionallyindependent ;thusthe
probability of a sentence is computed by multiplying the probabilities of each
rulein theparseofthesentence.

mars.
•The probabilistic CKY ( Cocke-Kasami-Younger ) algorithm is a probabilistic
version of the CKY parsing algorithm. There are also probabi listic versions of
otherparsersliketheEarleyalgorithm.

DRAFT490 Chapter 14. Statistical Parsing
•PCFG probabilitiescanbe learningbycountingin a parsed corpus , orbypars-
ingacorpus. The inside-outside algorithmisawayofdealingwiththefactthat
thesentencesbeingparsedareambiguous.
•RawPCFGssufferfrompoorindependenceassumptionsbetwee nrulesandlack
ofsensitivity tolexicaldependencies.
•Onewaytodealwiththisproblemistosplitandmergenon-ter minals(automat-
icallyorbyhand).
•Probabilistic lexicalized CFG s are another solution to this problem in which
the basic PCFG model is augmented with a lexical head for each rule. The
probabilityofarulecanthenbeconditionedonthelexicalh eadornearbyheads.
•Parsers for lexicalized PCFGs (like the Charniak and Collin s parsers) are based
onextensionsto probabilisticCKY parsing.
•Parsersareevaluatedusingthreemetrics: labeledrecall ,labeledprecision ,and
cross-brackets .
•There is evidence based on garden-path sentences and other on-line sentence-
processing experiments that the human parser uses some kind s of probabilistic

informationaboutgrammar.
Manyoftheformalpropertiesofprobabilisticcontext-fre egrammarswereﬁrstworked
out by Booth (1969) and Salomaa (1969). Baker (1979) propose d the inside-outside
algorithmforunsupervisedtrainingofPCFGprobabilities ,andusedaCKY-stylepars-
ingalgorithmtocomputeinsideprobabilities. Jelinekand Lafferty(1991)extendedthe
CKY algorithm to compute probabilities for preﬁxes. Stolck e (1995) drew on both of
thesealgorithmsin adaptingthe Earleyalgorithmto usewit hPCFGs.

A number of researchers starting in the early 1990s worked on adding lexical de-
pendencies to PCFGs, and on making PCFG rule probabilities m ore sensitive to sur-
rounding syntactic structure. For example Schabes et al. (1 988) and Schabes (1990)
presented early work on the use of heads. Many papers on the us e of lexical depen-
dencieswereﬁrstpresentedattheDARPASpeechandNaturalL anguageWorkshopin
June, 1990. A paper by Hindle and Rooth (1990) applied lexica l dependencies to the

problemofattachingprepositionalphrases;inthequestio nsessiontoalaterpaperKen
Churchsuggestedapplyingthismethodto full parsing(Marc us,1990). Earlyworkon
such probabilisticCFG parsing augmentedwith probabilist icdependencyinformation
includes Magerman and Marcus (1991), Black et al. (1992), Bo d (1993), and Jelinek
et al. (1994), in addition to Collins (1996), Charniak (1997 ), and Collins (1999) dis-
cussedabove. OtherrecentPCFG parsingmodelsincludeKlei nandManning(2003a)
andPetrovet al.(2006). .

This early lexical probabilistic work led initially to work focused on solving spe-
ciﬁc parsing problems like preposition-phrase attachment , using methods including
Transformation Based Learning (TBL) (Brill and Resnik, 199 4), Maximum Entropy

DRAFTSection14.11. Summary 491
(Ratnaparkhi et al., 1994), Memory-Based Learning (Zavrel and Daelemans, 1997),
log-linearmodels(Franz,1997),decisiontreesusingsema nticdistancebetweenheads
(computed from WordNet) (Stetina and Nagao, 1997), and Boos ting (Abney et al.,
1999b).
Another direction extended the lexical probabilistic pars ing work to build proba-
bilisticformulationsofgrammarotherthanPCFGs,suchasp robabilisticTAGgrammar
(Resnik,1992;Schabes,1992),basedontheTAGgrammarsdis cussedinCh.12,prob-
abilistic LR parsing (Briscoe and Carroll, 1993),and proba bilisticlink grammar(Laf-
fertyetal.,1992). Anapproachtoprobabilisticparsingca lled[Supertagging]supertagging [
extendsthepart-of-speechtaggingmetaphortoparsingbyu singverycomplextagsthat
are in fact fragments of lexicalized parse trees (Bangalore and Joshi, 1999; Joshi and
Srinivas, 1994),based on the lexicalized TAG grammarsof Sc habeset al. (1988). For
examplethenoun purchase wouldhaveadifferenttagastheﬁrst nounina nouncom-

pound (where it might be on the left of a small tree dominated b y Nominal) than as
the second noun (where it might be on the right). Supertaggin ghas also been applied
to CCG parsing and HPSG parsing (Clark and Curran, 2004a; Mat suzaki et al., 2007;
Blunsom and Baldwin, 2006). Non-supertagging statistical parsers for CCG include
HockenmaierandSteedman(2002).
Goodman(1997), Abney (1997), and Johnson et al. (1999)gave early discussions
of probabilistic treatments of feature-based grammars. Ot her recent work on building

statistical models of feature-based grammarformalisms li ke HPSG and LFG includes
Riezler etal. (2002),Kaplanet al.(2004),andToutanovaet al. (2005).
We mentioned earlier that discriminative approaches to par sing fall into the two
broadcategoriesofdynamicprogrammingmethodsanddiscri minativererankingmeth-
ods. Recall that discriminative reranking approaches requ ireN-best parses. Parsers
based on A* search can easily be modiﬁed to generate N-best lists just by continuing
the search past the ﬁrst-best parse (Roark, 2001). Dynamic p rogramming algorithms

like the ones described in this chapter can be modiﬁed by elim inating the dynamic
programmingand using heavy pruning (Collins, 2000; Collin s and Koo, 2005; Bikel,
2004), or via new algorithms (Jim´ enez and Marzal, 2000; Gil dea and Jurafsky, 2002;
Charniak and Johnson, 2005; Huang and Chiang, 2005), some ad apted from speech
recognitionalgorithmssuchasSchwartzandChow(1990)(se e Sec.10.1).
By contrast, in dynamic programming methods, instead of out putting and then

rerankingan N-bestlist,theparsesarerepresentedcompactlyinachart, andlog-linear
and other methods are applied for decoding directly from the chart. Such modern
methods include Johnson (2001),Clark and Curran (2004b), a nd Taskar et al. (2004).
Other reranking developments include changing the optimiz ation criterion (Titov and
Henderson,2006).
Another important recent area of research is dependency par sing; algorithms in-
clude Eisner’s bilexical algorithm (Eisner, 1996b, 1996a, 2000a), maximum spanning

treeapproaches(usingon-linelearning)(McDonaldetal., 2005,2005),andapproaches
based on building classiﬁers for parser actions (Kudo and Ma tsumoto, 2002; Yamada
and Matsumoto, 2003; Nivre et al., 2006; Titov and Henderson , 2007). A distinction
isusuallymadebetweenprojectiveand non-projectivedependencies . Non-projectiveNon-projective
dependenciesare those in which the dependencylines cross; this is not very common
in English, but is very common in many languages with more fre e word order. Non-

DRAFT492 Chapter 14. Statistical Parsing
projective dependency algorithms include McDonald et al. ( 2005) and Nivre (2007).
TheKlein-Manningparsercombinesdependencyandconstitu encyinformation(Klein
andManning,2003c).
Manning and Sch¨ utze (1999) has an extensive coverage of pro babilistic parsing.
Collins’ (1999) dissertation includes a very readable surv ey of the ﬁeld and introduc-
tiontohisparser.
Theﬁeldofgrammarinductioniscloselyrelatedtostatisti cal parsing,andaparser
is often used as part of a grammar induction algorithm. One of the earliest statistical
works in grammar induction was Horning (1969), who showed th at PCFGs could be
inducedwithoutnegativeevidence. Earlymodernprobabili sticgrammarworkshowed
that simply using EM was insufﬁcient (Lari and Young, 1990; C arroll and Charniak,
1992). Recent probabilisticwork such as Yuret (1998),Clar k (2001),Klein and Man-
ning(2002),andKleinandManning(2004),aresummarizedin Klein(2005)andAdri-

aans and van Zaanen (2004). Work since that summary includes Smith and Eisner
(2005),HaghighiandKlein(2006),andSmithandEisner(200 7).
Exercises
14.1ImplementtheCKY algorithm.
14.2ModifythealgorithmforconversiontoCNFfromCh.13tocorr ectlyhandlerule
probabilities. MakesurethattheresultingCNFassignsthe sametotalprobability
toeachparsetree.
14.3Recall that Exercise 3 asked you to update the CKY algorithm t o handles unit
productionsdirectly rather than converting them to CNF. Ex tend this change to
probabilisticCKY.

Exercises
14.4Fill outtherestofthe probabilisticCKY chartinFig. 14.4.
14.5Sketchout howthe CKY algorithmwouldhaveto be augmentedto handlelexi-
calizedprobabilities.
14.6ImplementyourlexicalizedextensionoftheCKY algorithm.
14.7Implement the PARSEVAL metrics described in Sec. 14.7. Next either use a
treebank or create your own hand-checked parsed testset. No w use your CFG
(or other) parser and grammar and parse the testset and compu te labeled recall,
labeledprecision,andcross-brackets.

Chapter15
Features and Uniﬁcation
FRIARFRANCIS:If either of you know any inward impediment why you should
not be conjoined, charge you, onyour souls, toutter it.
William Shakespeare, MuchAdoAboutNothing
From a reductionist perspective, the history of the natural sciences over the last few
hundred years can be seen as an attempt to explain the behavio r of larger structures
by the combinedaction of smaller primitives. In biology,th e propertiesof inheritance
have been explained by the action of genes, and then again the properties of genes

have been explained by the action of DNA. In physics, matter w as reduced to atoms
and then again to subatomic particles. The appeal of reducti onism has not escaped
computational linguistics. In this chapter we introduce th e idea that grammatical cat-
egories like VPto,Sthat,Non3sgAux , or3sgNP, as well as the grammatical rules like
S→NPVPthatmakeuse ofthem,shouldbe thoughtof as objectsthat canhavecom-
plex sets of properties associated with them. The information in these properties i s

represented by constraints , and so these kindsof models are often called constraint-
based formalisms .Constraint-basedformalisms
Whydoweneedamoreﬁne-grainedwayofrepresentingandplac ingconstraintson
grammaticalcategories? OneproblemaroseinCh.12,wherew esawthatnaivemodels
of grammaticalphenomenasuch as agreementand subcategori zationcanlead to over-
generationproblems. Forexample,inordertoavoidungramm aticalnounphrasessuch
asthisﬂights andverbphraseslike disappearedaﬂight ,wewereforcedtocreateahuge

proliferationofprimitivegrammaticalcategoriessuchas Non3sgVPto ,NPmass,3sgNP
andNon3sgAux . These new categories led, in turn, to an explosion in the num ber of
grammar rules and a corresponding loss of generality in the g rammar. A constraint-
based representationscheme will allow us to represent ﬁne- grainedinformationabout
number and person, agreement, subcategorization, as well a s semantic categories like
mass/count.
Constraint-based formalisms have other advantages that we will not cover in this

chapter,suchastheabilitytomodelmorecomplexphenomena thancontext-freegram-
mars, and the ability to efﬁciently and conveniently comput e semantics for syntactic
representations.
Considerbrieﬂyhowthisapproachmightworkinthecaseofgr ammaticalnumber.
AswesawinCh.12,nounphraseslike thisﬂight andthoseﬂights canbedistinguished
based on whether they are singular or plural. This distincti on can be captured if we
associate a property called NUMBER that can have the value singular or plural, with

appropriatemembersof the NPcategory. Given this ability, we can say that this ﬂight
isamemberofthe NPcategoryand,inaddition,hasthevaluesingularforits NUMBER
property. This same property can be used in the same way to dis tinguish singular and

DRAFT494 Chapter 15. FeaturesandUniﬁcation
pluralmembersofthe VPcategorysuchas serveslunch andserve lunch .
Ofcourse,simplyassociatingthesepropertieswithvariou swordsandphrasesdoes
not solve any of our overgeneration problems. To make these p roperties useful, we
need the ability to perform simple operations, such as equal ity tests, on them. By
pairingsuch tests with ourcore grammarrules, we can addvar iousconstraintsto help
ensure that only grammatical strings are generated by the gr ammar. For example, we
might want to ask whether or not a given noun phrase and verb ph rase have the same
valuesfortheirrespectivenumberproperties. Suchatesti sillustratedbythefollowing
kindofrule.
S→NPVP
OnlyifthenumberoftheNP isequalto thenumberoftheVP.
Theremainderofthischapterprovidesthedetailsofonecom putationalimplemen-
tationofaconstraint-basedformalism,basedonfeaturest ructuresanduniﬁcation. The
nextsectiondescribesfeaturestructures,the representa tionused tocapturethekindof

S→NPVP
grammatical propertieswe have in mind. Section 15.2 then in troducesthe uniﬁcation
operatorthatisusedtoimplementbasicoperationsoverfea turestructures. Section15.3
then covers the integration of these structures into a gramm atical formalism. Section
15.4 then introduces the uniﬁcation algorithm and its requi red data structures. Next,
Section 15.5 describes how feature structures and the uniﬁc ation operator can be in-
tegrated into a parser. Finally, Section 15.6 discusses the most signiﬁcant extension

to this constraint-based formalism, the use of types and inh eritance, as well as other
extensions.
15.1 FeatureStructures
One of the simplest ways to encode the kind of properties that we have in mind is
through the use of feature structures . These are simply sets of feature-value pairs, Featurestructures
wherefeaturesareunanalyzableatomicsymbolsdrawnfroms omeﬁniteset,andvalues
are eitheratomicsymbolsor featurestructuresthemselves . Suchfeaturestructuresare

illustrated with the following kind of diagram, called an attribute-value matrix orAttribute-valuematrix
AVM:

FEATURE 1value1
FEATURE 2value2
...
FEATURE nvaluen

To make this concrete, consider the number property discuss ed above. To capture
thisproperty,wewill usethesymbol NUMBER todesignatethisgrammaticalattribute,
and the symbols sgandpl(introducedin Ch. 3) to designate the possible valuesit can
take on in English. A simple feature structure consisting of this single feature would
thenbeillustratedasfollows:

DRAFTSection15.1. FeatureStructures 495
[
NUMBER sg]
Adding an additional feature-value pair to capture the gram matical notion of person
leadsto thefollowingfeaturestructure:
[
NUMBER sg
PERSON 3rd]
Next we can encode the grammatical category of the constitue nt that this structure
corresponds to through the use of the CATfeature. For example, we can indicate that
thesefeaturesareassociatedwitha nounphrasebyusingthe followingstructure:

CAT NP
NUMBER sg
PERSON 3rd

This structure can be used to represent the 3sgNPcategory introduced in Ch. 12 to
capture a restricted subcategoryof noun phrases. The corre spondingplural version of
thisstructurewouldbe capturedasfollows:

CAT NP
NUMBER pl

Note that the values of the CATandPERSONfeatures remain the same for these last
two structures. This illustrates how the use of feature stru ctures allows us to both
preservethecoreset ofgrammaticalcategoriesanddrawdis tinctionsamongmembers
ofa singlecategory.



CAT NP
NUMBER pl

As mentioned earlier, features are not limited to atomic sym bols as their values;
they can also have other feature structures as their values. This is particularly useful
when we wish to bundle a set of feature-valuepairs togetherf or similar treatment. As
an example of this, consider that the NUMBER andPERSONfeatures are often lumped
togethersincegrammaticalsubjectsmustagreewiththeirp redicatesinboththeirnum-
ber and person properties. This lumping together can be capt ured by introducing an

AGREEMENT featurethattakesafeaturestructureconsistingofthe NUMBER andPER-
SONfeature-value pairs as its value. Introducing this feature into our third person
singularnounphraseyieldsthefollowingkindofstructure .

CAT NP
AGREEMENT[
NUMBER sg
PERSON 3rd]

Given this kind of arrangement, we can test for the equality o f the values for both the
NUMBER andPERSONfeatures of two constituents by testing for the equality of t heir
AGREEMENT features.


CAT NP
NUMBER sg

This ability to use feature structures as values leads fairl y directly to the notion of
afeature path . A feature path is nothing more than a sequence of features th rough a Feature path
feature structure leading to a particular value. For exampl e, in the last feature struc-
ture,wecansaythat the ⟨AGREEMENT NUMBER ⟩pathleadsto thevalue sg, whilethe

DRAFT496 Chapter 15. FeaturesandUniﬁcation
CAT
AGREEMENTNP
sg
3rdPERSONNUMBER
Figure15.1 A feature structure with shared values. The location (value ) found by following
the⟨HEAD SUBJECT AGREEMENT ⟩path isthe same as that found via the ⟨HEAD AGREEMENT ⟩
path.
⟨AGREEMENT PERSON ⟩path leads to the value 3rd. This notion of a path leads natu-
rally to an alternative graphical way of illustrating featu re structures, shown in Figure
15.1, which as we will see in Section 15.4 is suggestive of how they will be imple-
mented. In these diagrams, feature structures are depicted as directed graphs where
featuresappearaslabelededgesandvaluesasnodes.
Whilethisnotionofpathswillproveusefulinanumberofset tings,weintroduceit
heretohelpexplainanadditionalimportantkindoffeature structure: thosethatcontain
features that actually share some feature structure as a val ue. Such feature structures
will be referred to as reentrant structures structures. What we have in mind hereReentrantstructures

is not the simple idea that two features might have equal valu es, but rather that they
share precisely the same feature structure(or node in the gr aph). These two cases can
be distinguished clearly if we think in terms of paths throug h a graph. In the case of
simple equality,two pathslead to distinct nodesin the grap hthat anchoridentical, but
distinct structures. In the case of a reentrant structure, t wo feature paths actually lead
tothe samenodeinthe structure.
Figure15.2illustratesasimpleexampleofreentrancy. Int hisstructure,the⟨HEAD

SUBJECT AGREEMENT ⟩path and the⟨HEAD AGREEMENT ⟩path lead to the same lo-
cation. Shared structures like this will be denoted in our AV M diagrams by adding
numerical indexes that signal the values to be shared. The AV M version of the fea-
ture structure from Figure 15.2would be denotedas follows, using the notationof the
PATR-IIsystem(Shieber,1986),basedonKay(1979):
CATS
HEAD
AGREEMENT 1[
NUMBER sg
SUBJECT[
AGREEMENT 1]


As we will see, these simple structures give us the ability to express linguistic

HEAD
NUMBER sg
SUBJECT[


generalizationsin surprisinglycompactandelegantways.

DRAFTSection15.2. UniﬁcationofFeatureStructures 497
S
sg
3rd PERSONNUMBERAGREEMENT
SUBJECT
AGREEMENTCAT
Figure15.2 A feature structure with shared values. The location (value ) found by following
path.
15.2 UniﬁcationofFeatureStructures
As noted earlier, feature structures would be of little use w ithout our being able to
perform reasonably efﬁcient and powerful operations on the m. As we will show, the
twoprincipaloperationsweneedtoperformaremergingthei nformationcontentoftwo
structures and rejecting the merger of structures that are i ncompatible. Fortunately, a
single computational technique, called , sufﬁces for both o f these purposes. The bulk
of this section will illustrate through a series of examples how uniﬁcation instantiates
these notionsofmergerandcompatibility. Discussionofth e uniﬁcationalgorithmand
itsimplementationwill bedeferredtoSection15.4.
We beginwiththefollowingsimpleapplicationoftheuniﬁca tionoperator.[

NUMBER sg]
⊔[
NUMBER sg]
=[
NUMBER sg]
As this equation illustrates, uniﬁcation is a binary operat ion (represented here as ⊔)
that accepts two feature structures as arguments and return s a feature structure when
it succeeds. In this example, uniﬁcation is being used to per form a simple equality
check. The uniﬁcation succeeds because the corresponding NUMBER features in each
structureagreeastotheirvalues. Inthiscase,sincetheor iginalstructuresareidentical,
the outputis the same as the input. The followingsimilar kin dof checkfails since the

NUMBER featuresinthe twostructureshaveincompatiblevalues.[
NUMBER sg]
⊔[
NUMBER pl]
Fails!
This next uniﬁcation illustrates an important aspect of the notion of compatibility
inuniﬁcation.[
NUMBER sg]
⊔[
NUMBER []]
=[
NUMBER sg]
In this situation, these features structures are taken to be compatible, and are hence
capableofbeingmerged,despitethefactthatthegivenvalu esfortherespective NUM-

DRAFT498 Chapter 15. FeaturesandUniﬁcation
BERfeaturesaredifferent. The [] valuein thesecondstructure indicatesthatthe value
has been left unspeciﬁed. A feature with such a [] value can be successfully matched
to any value in a corresponding feature in another structure . Therefore, in this case,
the value sgfrom the ﬁrst structure can match the [] value from the second , and as is
indicated by the output shown, the result of this type of uniﬁ cation is a structure with
thevalueprovidedbythemorespeciﬁc,non-null,value.
Thenextexampleillustratesanotherofthemergeraspectso funiﬁcation.
[
NUMBER sg]
⊔[
=[
NUMBER sg
Here the result of the uniﬁcation is a merger of the original t wo structures into one
larger structure. This larger structure contains the union of all the information stored
in each of the original structures. Althoughthis is a simple example,it is importantto
understandwhy these structuresare judged to be compatible : they are compatible be-

[
NUMBER sg]
⊔[
=[
NUMBER sg
cause theycontainnofeaturesthat are explicitlyincompat ible. Thefact that theyeach
containa feature-valuepairthattheotherdoesnotisnotar easonfortheuniﬁcationto
fail.
We willnowconsideraseriesofcasesinvolvingtheuniﬁcati onofsomewhatmore
complexreentrantstructures. Thefollowingexampleillus tratesanequalitycheckcom-
plicatedbythe presenceofareentrantstructurein theﬁrst argument.

NUMBER sg
SUBJECT[

⊔
SUBJECT
AGREEMENT[
PERSON 3rd
NUMBER sg]


=
NUMBER sg
SUBJECT[

fail.

NUMBER sg
SUBJECT[

⊔
SUBJECT
PERSON 3rd


=
NUMBER sg
SUBJECT[

The important elementsin this exampleare the SUBJECT featuresin the two input
structures. The uniﬁcation of these features succeeds beca use the values found in the
ﬁrstargumentbyfollowingthe 1numericalindex,matchthosethataredirectlypresent
in the second argument. Note that, by itself, the value of the AGREEMENT feature in
the ﬁrst argument would have no bearing on the success of uniﬁ cation since the sec-
ond argumentlacks an AGREEMENT feature at the top level. It only becomesrelevant


becausethevalueofthe AGREEMENT featureissharedwiththe SUBJECT feature.
Thefollowingexampleillustratesthe copyingcapabilitie sofuniﬁcation.
(15.1)
AGREEMENT 1
SUBJECT[


DRAFTSection15.2. UniﬁcationofFeatureStructures 499
⊔
SUBJECT
PERSON 3rd


=
AGREEMENT 1
SUBJECT
AGREEMENT 1[
PERSON 3rd


Herethevaluefoundviathesecondargument’s ⟨SUBJECT AGREEMENT ⟩pathiscopied
over to the corresponding place in the ﬁrst argument. In addi tion, the AGREEMENT
feature of the ﬁrst argument receives a value as a side-effec t of the index linking it to
thevalueatthe endofthe ⟨SUBJECT AGREEMENT ⟩path.
The next exampledemonstratesthe important differencebet ween features that ac-
tuallysharevaluesversusthosethat merelyhaveidentical lookingvalues.
(15.2)
AGREEMENT[
NUMBER sg]
SUBJECT[
NUMBER sg]]

⊔
SUBJECT
PERSON 3


=
AGREEMENT[
NUMBER sg]
SUBJECT
NUMBER sg
PERSON 3]


Thevaluesattheendofthe ⟨SUBJECT AGREEMENT ⟩pathandthe⟨AGREEMENT⟩path
are the same, but not shared, in the ﬁrst argument. The uniﬁca tion of the SUBJECT

(15.2)
NUMBER sg]
SUBJECT[

⊔
SUBJECT
PERSON 3


=
NUMBER sg]
SUBJECT
NUMBER sg
PERSON 3]


featuresofthetwoargumentsaddsthe PERSONinformationfromthesecondargument
to the result. However, since there is no index linking the AGREEMENT feature to
the⟨SUBJECT AGREEMENT ⟩path, this information is not added to the value of the
AGREEMENT feature.
Finally,considerthefollowingexampleofa failuretounif y.
NUMBER sg
PERSON 3]
SUBJECT[

⊔
AGREEMENT[
NUMBER sg
PERSON 3]
SUBJECT
NUMBER PL
PERSON 3]


Fails!

DRAFT500 Chapter 15. FeaturesandUniﬁcation
Proceedingthroughthefeaturesinorder,weﬁrst ﬁndthatth eAGREEMENT featuresin
these examplessuccessfully match. However, when we move on to theSUBJECT fea-
tures,weﬁndthatthevaluesfoundattherespective ⟨SUBJECT AGREEMENT NUMBER
⟩pathsdiffer,causingauniﬁcationfailure.
Featurestructuresareawayofrepresentingpartialinform ationaboutsomelinguis-
tic object or placing informational constraints on what the object can be. Uniﬁcation
canbeseenasawayofmergingtheinformationineachfeature structure,ordescribing
objects which satisfy both sets of constraints. Intuitivel y, unifying two feature struc-
tures produces a new feature structure which is more speciﬁc (has more information)
than,orisidenticalto, eitheroftheinputfeaturestructu res. We say thata less speciﬁc
(more abstract) feature structure subsumes an equally or more speciﬁc one. Sub- Subsumes
sumption is represented by the operator ⊑. A feature structure Fsubsumes a feature
structureG(F⊑G)if andonlyif:

1. For every feature xinF,F(x)⊑G(x)(whereF(x)means “the value of the
featurexoffeaturestructure F”).
2. Forall paths pandqinFsuchthat F(p) =F(q), it is alsothe case that G(p) =
G(q).
Forexample,considerthesefeaturestructures:
(15.3)[
NUMBER sg]
(15.4)[
PERSON 3]
(15.5)[
NUMBER sg
PERSON 3]
(15.6)
CAT VP
AGREEMENT 1
SUBJECT[

(15.7)
CAT VP
SUBJECT
PERSON 3


Thefollowingsubsumptionrelationsholdamongthem:
15.3⊑15.5
15.4⊑15.5
15.6⊑15.7

G(q).
(15.3)[
NUMBER sg]
(15.4)[
PERSON 3]
(15.5)[
NUMBER sg
PERSON 3]
(15.6)
CAT VP
SUBJECT[

(15.7)
SUBJECT
PERSON 3


15.3⊑15.5
15.4⊑15.5
15.6⊑15.7
Subsumption is a partial ordering; there are pairs of featur e structures that neither
subsumenoraresubsumedbyeachother:
15.3̸⊑15.4

DRAFTSection15.3. FeatureStructuresintheGrammar 501
15.4̸⊑15.3
Since every feature structure is subsumed by the empty struc ture [], the relation
among feature structures can be deﬁned as a semilattice . Uniﬁcation can be deﬁned Semilattice
in terms of the subsumption semilattice. Given two feature s tructures FandG,F⊔G
is deﬁned as the most general feature structure Hsuch that F⊑HandG⊑H. Since
the information ordering deﬁned by subsumption is a semilat tice, the uniﬁcation op-
eration is monotonic (Pereira and Shieber, 1984; Roundsand Kasper, 1986; Moshie r, Monotonic
1988). This means that if some description is true of a featur e structure, unifying it
with another feature structure results in a feature structu re that still satisﬁes the origi-
nal description. The uniﬁcation operationis thereforeass ociative; given a ﬁnite set of
featurestructurestounify,we cancheckthemin anyorderan dget thesameresult.
To summarize, uniﬁcation is a way of implementing the integr ation of knowledge

fromdifferentconstraints. Giventwocompatiblefeatures tructuresasinput,itproduces
themostgeneralfeaturestructurewhichnonethelessconta insalltheinformationinthe
inputs. Giventwoincompatiblefeaturestructures,it fail s.
15.3 FeatureStructuresin theGrammar
Our primarypurposein introducingfeaturestructuresand u niﬁcationhas beento pro-
vide a way to elegantly expresssyntactic constraintsthat w ould be difﬁcult to express
using the mechanisms of context-free grammars alone. Our ne xt step, therefore, is to
specify a way to integrate feature structures and uniﬁcatio n operations into the speci-

ﬁcation of a grammar. This can be accomplishedby augmenting the rules of ordinary
context-free grammars with attachments that specify featu re structures for the con-
stituents of the rules, along with appropriate uniﬁcation o perations that express con-
straints on those constituents. From a grammatical point of view, these attachments
will beusedtoaccomplishthefollowinggoals:
•to associate complex feature structures with both lexical i tems and instances of
grammaticalcategories
•toguidethecompositionoffeaturestructuresforlargergr ammaticalconstituents

basedonthefeaturestructuresoftheircomponentparts
•toenforcecompatibilityconstraintsbetweenspeciﬁedpar tsofgrammaticalcon-
structions
We will use the followingnotation to denote the grammaraugm entationsthat will
allow us to accomplish all of these goals, based on the PATR-I I system described in
Shieber(1986):
β0→β1···βn
{set of constraints}
Thespeciﬁedconstraintshaveoneofthe followingforms.
⟨βifeature path⟩=Atomicvalue
⟨βifeature path⟩=⟨βjfeature path⟩

DRAFT502 Chapter 15. FeaturesandUniﬁcation
The notation⟨βifeature path⟩denotes a feature path through the feature structure as-
sociated with the βicomponent of the context-free part of the rule. The ﬁrst styl e of
constraint speciﬁes that the value found at the end of the giv en path must unify with
the speciﬁed atomic value. The second form speciﬁes that the valuesfound at the end
ofthetwo givenpathsmustbeuniﬁable.
Toillustratetheuseoftheseconstraints,letusreturntot heinformalsolutiontothe
numberagreementproblemproposedatthebeginningofthisc hapter.
S→NPVP
Usingthenewnotation,thisrulecannowbeexpressedasfoll ows.
S→NPVP
⟨NPNUMBER⟩=⟨VPNUMBER⟩
Note that in cases where there are two or more constituents of the same syntactic
category in a rule, we will subscript the constituents to kee p them straight, as in
VP→VNP1NP2.
Taking a step back from the notation, it is important to note t hat in this approach

S→NPVP
S→NPVP
the simple generative nature of context-freerules has been fundamentallychanged by
this augmentation. Ordinary context-freerules are based o n the simple notion of con-
catenation; an NPfollowed by a VPis anS, or generatively, to produce an Sall we
need to do is concatenate an NPto aVP. In the new scheme, this concatenation must
beaccompaniedbyasuccessfuluniﬁcationoperation. Thisl eadsnaturallytoquestions
about the computational complexity of the uniﬁcation opera tion and its effect on the

generativepowerofthisnewgrammar. Theseissueswill bedi scussedinCh. 15.
Toreview,thereare twofundamentalcomponentstothisappr oach.
•The elements of context-freegrammar rules will have featur e-based constraints
associated with them. This reﬂects a shift from atomic gramm atical categories
tomorecomplexcategorieswithproperties.
•Theconstraintsassociatedwithindividualrulescanmaker eferencetothefeature
structuresassociatedwiththe partsoftheruletowhichthe yareattached.

The following sections present applications of uniﬁcation constraints to four in-
terestinglinguisticphenomena: agreement,grammaticalh eads,subcategorization,and
long-distancedependencies.
15.3.1 Agreement
AsdiscussedinCh.12,agreementphenomenashowupinanumbe rofdifferentplaces
in English. This section illustrates how uniﬁcation can be u sed to capture the two
maintypesofEnglishagreementphenomena: subject-verbag reementanddeterminer-
nominalagreement. WewillusethefollowingATISsentences asexamplesthroughout
thisdiscussionto illustratethesephenomena.

(15.8)Thisﬂight servesbreakfast.

DRAFTSection15.3. FeatureStructuresintheGrammar 503
(15.9)Doesthisﬂightservebreakfast?
(15.10)Dotheseﬂightsservebreakfast?
Noticethattheconstraintusedtoenforce SUBJECT-VERBagreementgivenaboveis
deﬁcient in that it ignoresthe PERSONfeature. The followingconstraint which makes
useofthe AGREEMENT featuretakescareofthisproblem.
S→NPVP
⟨NPAGREEMENT⟩=⟨VPAGREEMENT⟩
Examples15.9and15.10illustrateaminorvariationon SUBJECT-VERBagreement.
In these yes-no-questions, the subject NPmust agree with the auxiliary verb, rather
thanthemainverbofthesentence,whichappearsinanon-ﬁni teform. Thisagreement
constraintcanbehandledbythefollowingrule.
S→AuxNPVP
⟨AuxAGREEMENT⟩=⟨NPAGREEMENT⟩
Agreementbetweendeterminersandnominalsinnounphrases ishandledinasim-
ilar fashion. The basic task is to permit the expressions giv en above, but block the
unwanted *this ﬂights and*those ﬂight expressions where the determiners and nomi-
nals clash in their NUMBER feature. Again, the logical place to enforcethis constrain t

S→AuxNPVP
isinthe grammarrulethatbringsthepartstogether.
⟨DetAGREEMENT⟩=⟨Nominal AGREEMENT⟩
⟨NPAGREEMENT⟩=⟨Nominal AGREEMENT⟩
Thisrulestatesthatthe AGREEMENT featureofthe Detmustunifywiththe AGREE-
MENTfeature of the Nominal, and moreover, that the AGREEMENT feature of the NP
mustalso unifywiththe Nominal.
Having expressed the constraints needed to enforce subject -verb and determiner-
nominalagreement,wemustnowﬁllintherestofthemachiner yneededtomakethese
constraintswork. Speciﬁcally,wemustconsiderhowthevar iousconstituentsthattake

part in these constraints (the Aux,VP,NP,Det, andNominal) acquire values for their
variousagreementfeatures.
We can begin by noting that our constraints involve both lexi cal and non-lexical
constituents. The simpler lexical constituents, AuxandDet, receive values for their
respectiveagreementfeaturesdirectlyfromthelexiconas inthefollowingrules.
Aux→do
⟨AuxAGREEMENT NUMBER ⟩=pl
⟨AuxAGREEMENT PERSON ⟩=3rd
Aux→does
⟨AuxAGREEMENT NUMBER ⟩=sg

DRAFT504 Chapter 15. FeaturesandUniﬁcation
Det→this
⟨DetAGREEMENT NUMBER ⟩=sg
Det→these
⟨DetAGREEMENT NUMBER ⟩=pl
Returning to our ﬁrst Srule, let us ﬁrst consider the AGREEMENT feature for the
VPconstituent. Theconstituentstructureforthis VPisspeciﬁedbythefollowingrule.
VP→VerbNP
It seems clear that the agreement constraint for this consti tuent must be based on
its constituent verb. This verb, as with the previous lexica l entries, can acquire its
agreementfeaturevaluesdirectlyfromlexiconasinthefol lowingrules.
Verb→serve
⟨VerbAGREEMENT NUMBER ⟩=pl
Verb→serves
⟨VerbAGREEMENT NUMBER ⟩=sg
⟨VerbAGREEMENT PERSON ⟩=3rd
Allthatremainsistostipulatethattheagreementfeatureo ftheparent VPisconstrained
tobethe sameasitsverbconstituent.
VP→VerbNP
⟨VPAGREEMENT⟩=⟨VerbAGREEMENT⟩
In other words, non-lexical grammatical constituents can a cquire values for at least
someoftheirfeaturesfromtheircomponentconstituents.
The same technique works for the remaining NPandNominal categories. The

VP→VerbNP
valuesfortheagreementfeaturesforthesecategoriesared erivedfromthenouns ﬂight
andﬂights.
⟨NounAGREEMENT NUMBER ⟩=sg
Noun→ﬂights
⟨NounAGREEMENT NUMBER ⟩=pl
Nominalfeaturescanbeconstrainedtohavethesamevaluesastheirc onstituentnouns.
⟨Nominal AGREEMENT⟩=⟨NounAGREEMENT⟩
Note that this section hasonlyscratched the surfaceof the E nglishagreementsys-
tem,andthattheagreementsystemofotherlanguagescanbec onsiderablymorecom-
plexthanEnglish.

DRAFTSection15.3. FeatureStructuresintheGrammar 505
15.3.2 HeadFeatures
To account for the way that compositional grammatical const ituents such as noun
phrases, nominals, and verb phrases come to have agreement f eatures, the preceding
section introducedthe notion of copyingfeature structure sfrom phrase structure chil-
dren to their parents. This turns out to be a speciﬁc instance of a much more general
phenomenon in constraint-based grammars. Speciﬁcally, th e features for most gram-
matical categories are copied from oneof the children to the parent. The child that
provides the features is called the head of the phrase and the features copied are re-
ferredtoas head features . Headfeatures
This notion of heads, ﬁrst introduced in Sec. 12.4.4, plays a n important role in
constraint-basedgrammars. Considerthefollowingthreer ulesfromthelast section.
VP→VerbNP

VP→VerbNP
In each of these rules, the constituent providingthe agreem ent feature structure to
its parent is the head of the phrase. More speciﬁcally, the ve rb is the head of the verb
phrase, the nominal is the head of the noun phrase, and the nou n is the head of the
nominal. As a result, we can say that the agreement feature st ructure is a head fea-
ture. We canrewriteourrulestoreﬂectthesegeneralizatio nsbyplacingtheagreement

feature structure under a HEADfeature and then copyingthat feature upward as in the
followingconstraints.
VP→VerbNP (15.11)
⟨VPHEAD⟩=⟨VerbHEAD⟩
NP→DetNominal (15.12)
⟨NPHEAD⟩=⟨Nominal HEAD⟩
⟨DetHEAD AGREEMENT ⟩=⟨Nominal HEAD AGREEMENT ⟩
Nominal→Noun (15.13)
⟨Nominal HEAD⟩=⟨NounHEAD⟩

DRAFT506 Chapter 15. FeaturesandUniﬁcation
Similarly, the lexical entries that introduce these featur es must now reﬂect this
HEADnotion,asin thefollowing.
⟨NounHEAD AGREEMENT NUMBER ⟩=pl
⟨VerbHEAD AGREEMENT NUMBER ⟩=sg
⟨VerbHEAD AGREEMENT PERSON ⟩=3rd
15.3.3 Subcategorization
Recall that subcategorization is the notion that verbs can b e picky about the patterns
of arguments they will allow themselves to appear with. In Ch . 12, to prevent the
generationofungrammaticalsentenceswithverbsandverbp hrasesthat donotmatch,
we were forced to split the category of verb into multiple sub -categories. These more
speciﬁc verb categories were then used in the deﬁnition of th e speciﬁc verb phrases
thattheywereallowedto occurwith,asinthefollowing.
Verb-with-S-comp→think
VP→Verb-with-S-compS
Clearly,thisapproachintroducesexactlythesame undesir ableproliferationofcat-
egories that we saw with the similar approach to solving the n umber problem. The

proper way to avoid this proliferation is to introduce featu re structures to distinguish
among the various members of the verb category. This goal can be accomplished by
associating an atomic feature called SUBCAT, with an appropriate value, with each of
theverbsinthelexicon. Forexample,thetransitiveversio nofservescouldbeassigned
thefollowingfeaturestructurein thelexicon.
⟨VerbHEAD SUBCAT⟩=trans

TheSUBCATfeature is a signal to the rest of the grammar that this verb sh ould only
appearinverbphraseswithasinglenounphraseargument. Th isconstraintisenforced
by adding correspondingconstraints to all the verb phrase r ules in the grammar, as in
thefollowing.
VP→Verb
⟨VPHEAD SUBCAT⟩=intrans
VP→VerbNP
⟨VPHEAD SUBCAT⟩=trans

DRAFTSection15.3. FeatureStructuresintheGrammar 507
VP→VerbNPNP
⟨VPHEAD SUBCAT⟩=ditrans
Theﬁrst uniﬁcationconstraintin these rulesstates that th e verbphrasereceivesits
HEADfeaturesfromitsverbconstituent,whilethesecondconstr aintspeciﬁeswhatthe
valueofthat SUBCATfeaturemustbe. Anyattempttouseaverbwithaninappropria te
verb phrase will fail since the value of the SUBCATfeature of the VPwill fail to unify
withtheatomicsymbolgiveninthesecondconstraint. Notet hatthisapproachrequires
uniquesymbolsforeachofthe50–100verbphraseframesinEn glish.
Thisisasomewhatclumsyapproachsincetheseunanalyzable SUBCATsymbolsdo
notdirectlyencodeeitherthenumberortypeofthe argument sthatthe verbexpectsto
take. To see this, note that one can not simply examine a verb’ s entry in the lexicon
and know what its subcategorization frame is. Rather, you mu st use the value of the
SUBCATfeature indirectly as a pointer to those verb phrase rules in the grammar that
canaccepttheverbinquestion.

Amoreelegantsolution,whichmakesbetteruseoftheexpres sivepoweroffeature
structures, allows the verb entries to directly specify the order and category type of
the argumentsthey require. The following entry for servesis an example of one such
approach, in which the verb’s subcategory feature expresse s alistof its objects and
complements.
⟨VerbHEAD SUBCAT FIRST CAT ⟩=NP
⟨VerbHEAD SUBCAT SECOND ⟩=end
This entry uses the FIRSTfeature to state that the ﬁrst post-verbal argument must

be anNP; the value of the SECOND feature indicates that this verb expects only one
argument. A verb like leave Boston in the morning , with two arguments, would have
thefollowingkindofentry.
Verb→leaves
⟨VerbHEAD SUBCAT SECOND CAT ⟩=PP
⟨VerbHEAD SUBCAT THIRD ⟩=end
Thisschemeis,ofcourse,aratherbaroquewayofencodingal ist;itisalsopossible
touse theideaof typesdeﬁnedin Sec.15.6todeﬁnea list typemorecleanly.
The individual verb phrase rules must now check for the prese nce of exactly the

elementsspeciﬁedbytheirverb,asin thefollowingtransit iverule.
VP→VerbNP (15.14)

DRAFT508 Chapter 15. FeaturesandUniﬁcation
⟨VPHEAD SUBCAT FIRST CAT ⟩=⟨NPCAT⟩
⟨VPHEAD SUBCAT SECOND ⟩=end
The second constraint in this rule’s constraints states tha t the category of the ﬁrst
element ofthe verb’s SUBCATlist must match the categoryof the constituentimmedi-
atelyfollowingtheverb. Thethirdconstraintgoesontosta te thatthisverbphraserule
expectsonlyasingle argument.
Our previous examples have shown rather simple subcategori zation structures for
verbs. In fact, verbscan subcategorize for quite complex subcategorizationframes ,Subcategorization
frames
(e.g.,NP PP,NP NP, orNP S) and these frames can be composed of many different
phrasal types. In orderto come up with a list of possible subc ategorizationframesfor
English verbs, we ﬁrst need to have a list of possible phrase t ypes that can make up
these frames. Fig. 15.3.3 shows one short list of possible ph rase types for making up
subcategorization frames for verbs; this list is modiﬁed fr om one used to create verb

frames
subcategorizationframesin the FrameNet project(Johnson ,1999; Baker et al., 1998),
and includes phrase types for special subjects of verbs like thereandit, as well as for
objectsandcomplements.
To use the phrase types in Fig. 15.3.3 in a uniﬁcation grammar , each phrase type
couldbedescribedusingfeatures. Forexampletheform VPto,whichissubcategorized
forbywantmightbeexpressedas:
Verb→want
⟨VerbHEAD SUBCAT FIRST CAT ⟩=VP
⟨VerbHEAD SUBCAT FIRST FORM ⟩=inﬁnitive
Each of the 50 to 100 possible verb subcategorizationframes in English would be

Verb→want
described as a list drawn from these phrase types. For exampl e, here is an example
of the two-complement want. We can use this example to demonstrate two different
notational possibilities. First, lists can be represented via an angle brackets notation ⟨
and⟩. Second, instead of using a rewrite-rule annotated with pat h equations, we can
representthelexicalentryasa singlefeaturestructure:

ORTHwant
CATVerb
HEAD
SUBCAT⟨[
CATNP]
,
CATVP
HEAD[
VFORMinﬁnitival]
⟩




CATVerb
HEAD
SUBCAT⟨[
CATNP]
,
CATVP
HEAD[
⟩


Combiningevena limited set of phrasetypesresults in a very largeset of possible
subcategorization frames. Furthermore, each verb allows m any different subcatego-
rizationframes. Fig. 15.4providesa set ofsubcategorizat ionpatternsfortheverb ask,
withexamplesfromtheBNC:
A number of comprehensive subcategorization-frame tagset s exist, such as the
COMLEX set (Macleod et al., 1998), which includes subcatego rization frames for
verbs, adjectives, and nouns, and the ACQUILEX tagset of ver b subcategorization

DRAFTSection15.3. FeatureStructuresintheGrammar 509
NounPhrase Types
There nonreferential there Thereis stillmuchtolearn
It nonreferential it Itwas evident that my ideas
NP noun phrase Ashe was relating hisstory
PrepositionPhraseTypes
PP preposition phrase couch their message interms
PPing gerundive PP censured him for nothavingintervened
PPpart particle turnitoff
VerbPhrase Types
VPbrst bare stem VP she could discussit
VPto to-marked inﬁn. VP Why doyou want toknow?
VPwh wh-VP itis worthconsidering how towrite
VPing gerundive VP I wouldconsider usingit
Complement Clausetypes
Sﬁn ﬁnite clause maintain that thesituationwas unsatisfactory
Swh wh-clause ittells us wherewe are
Sif whether/if clause askwhetherAristophanes isdepictinga
Sing gerundive clause seesome attentionbeinggiven
Sto to-marked clause knowthemselves toberelatively unhealthy
Sforto for-toclause She was waiting for himtomake some reply
Sbrst bare stem clause commanded thathissermons bepublished
Other Types
AjP adjective phrase thought it possible

Quo quotes asked“What was itlike?”
Figure15.3 A small set of potential phrase types which can be combined to create a set of
potentialsubcategorizationframesforverbs. Modiﬁedfro mtheFrameNettagset(Johnson,1999;
Baker et al.,1998). The sample sentence fragments are from t he BritishNational Corpus.
Subcat Example
Quo asked[Quo“What wasit like?”]
NP asking[ NPa question]
Swh asked[Swhwhattradesyou’reinterestedin]
Sto ask [Stohimto tellyou]
PP that meansasking[ PPat home]
Vto asked[Vtoto seea girlcalledEvelyn]

NP Sif asked[NPhim][Sifwhetherhecouldmake]
NP NP asked[NPmyself][ NPaquestion]
NP Swh asked[NPhim][Swhwhyhetooktimeoff]
Figure15.4 A set of sample subcategorization patterns for the verb askwith examples from
the BNC.
frames (Sanﬁlippo, 1993). Many subcategorization-frame t agsets add other informa-
tion about the complements, such as specifying the identity of the implicit subject in
a lower verb phrase that has no overt subject; this is called controlinformation. For Control

the BNC.
exampleTemmy promised Ruth to go (at least in some dialects) implies that Temmy
willdothegoing,while TemmypersuadedRuthtogo impliesthatRuthwilldothego-

DRAFT510 Chapter 15. FeaturesandUniﬁcation
ing. Someofthemultiplepossiblesubcategorizationframe sforaverbcanbepartially
predicted by the semantics of the verb; for example many verb s of transfer (like give,
send,carry)predictablytakethe twosubcategorizationframes NP NPandNP PP:
NPNP sent FAAAdministratorJamesBuseyaletter
NPPP sent aletter tothechairmanoftheArmedServicesCommi ttee
These relationships between subcategorization frames acr oss classes of verbs are
called argument-structure alternations , and will be discussed in Ch. 19 when we dis- Alternations
cuss the semantics of verbal argument structure. Ch. 14 will introduce probabilities
for modeling the fact that verbs generally have preferences even among the different
subcategorizationframestheyallow.
Subcategorizationin OtherPartsofSpeech
Althoughthenotionofsubcategorization,or valenceasitisoftencalled,wasoriginally Valence
conceivedforverbs,morerecentworkhasfocusedonthefact thatmanyotherkindsof

wordsexhibit formsof valence-likebehavior. Consider the followingcontrastinguses
oftheprepositions whileandduring.
(15.15)Keepyourseatbelt fastenedwhile we aretakingoff .
(15.16)*Keepyourseatbeltfastenedwhile takeoff.
(15.17)Keepyourseatbelt fastenedduring takeoff.
(15.18)*Keepyourseatbeltfastenedduring we are takingoff .
Despite the apparent similarities between these words, the y make quite different de-
mands on their arguments. Representing these differencesi s left as Exercise 5 for the
reader.

reader.
Many adjectives and nouns also have subcategorization fram es. Here are some
examples using the adjectives apparent,aware, andunimportant and the nouns as-
sumption andquestion:
It wasapparent [Sﬁnthat thekitchenwastheonlyroom...]
It wasapparent [PPfromthewayshe restedherhandoverhis]
aware[Sﬁnhemayhavecausedoffense]
it isunimportant [Swhethwhetheronlya little bitisaccepted]
theassumption [Sﬁnthatwastefulmethodshavebeenemployed]
thequestion [Swhethwhethertheauthoritiesmighthavedecided]

reader.
SeeMacleodetal.(1998)andJohnson(1999)fordescription sofsubcategorization
framesfornounsandadjectives.
Verbsexpress subcategorizationconstraints on their subj ects as well as their com-
plements. For example, we need to represent the lexical fact that the verb seemcan
take anSﬁnas its subject ( That she was affectedseems obvious ), while the verb paint
cannot. The SUBJECT featurecanbeusedtoexpresstheseconstraints.
15.3.4 Long-DistanceDependencies

The model of subcategorizationwe have developed so far has t wo components. Each
head word has a SUBCATfeature which contains a list of the complements it expects.

DRAFTSection15.4. ImplementingUniﬁcation 511
Then phrasal rules like the VPrule in (15.15) match up each expected complement
in theSUBCATlist with an actual constituent. This mechanism works ﬁne wh en the
complementsofa verbarein facttobefoundintheverbphrase .
Sometimes, however, a constituent subcategorized for by th e verb is not locally
instantiated, but stands in a long-distance relationship with its predicate. Here are
someexamplesofsuch long-distancedependencies :Long-distance
WhatcitiesdoesContinentalservice?
WhatﬂightsdoyouhavefromBostonto Baltimore?
Whattime doesthat ﬂightleaveAtlanta?
In the ﬁrst example, the constituent what cities is subcategorized for by the verb
service, but because the sentence is an example of a wh-non-subject-question , the
objectislocatedatthefrontofthesentence. RecallfromCh .12thata(simple)phrase-
structurerulefora wh-non-subject-question is somethinglike thefollowing:

Now that we have features, we can augment this phrase-struct ure rule to require
theAuxand theNPto agree (sincethe NPis the subject). But we also need some way
to augment the rule to tell it that the Wh-NPshould ﬁll some subcategorization slot
in theVP. The representation of such long-distance dependencies is a quite difﬁcult
problem, because the verb whose subcategorizationrequire mentis being ﬁlled can be
quite distant from the ﬁller. In the following (made-up) sen tence, for example, the

wh-phrasewhich ﬂight must ﬁll the subcategorization requirements of the verb book,
despitethefact thattherearetwootherverbs( wantandhave)in between:
Whichﬂightdoyouwantmeto havethetravelagentbook?
Many solutions to representing long-distance dependencie s in uniﬁcation gram-
mars involve keeping a list, often called a gap list, implemented as a feature GAP, Gaplist
which is passed up from phrase to phrase in the parse tree. The ﬁller(for example Filler

which ﬂight above)is put on the gap list, and must eventuallybe uniﬁed wi th the sub-
categorization frame of some verb. See Sag and Wasow (1999) f or an explanation of
such a strategy, together with a discussion of the many other complications that must
bemodeledinlong-distancedependencies.
15.4 ImplementingUniﬁcation
As discussed, the uniﬁcationoperatortakes two feature str ucturesas input and returns
a single merged feature structure if successful, or a failur e signal if the two inputs are

notcompatible. The inputfeaturestructuresare represent edasdirectedacyclicgraphs
(DAGs),wherefeaturesaredepictedaslabelsondirecteded ges,andfeaturevaluesare
either atomic symbols or DAGs. As we will see, the implementa tion of the operator
is a relativelystraightforwardrecursive graphmatching a lgorithm, suitably tailored to
accommodatethevariousrequirementsofuniﬁcation. Rough lyspeaking,thealgorithm
loopsthroughthe featuresin one inputand attemptsto ﬁnd a c orrespondingfeature in

DRAFT512 Chapter 15. FeaturesandUniﬁcation
theother. Ifalloftherespectivefeaturevaluesmatch,the ntheuniﬁcationissuccessful.
If there is a mismatch then the uniﬁcation fails. Not surpris ingly, the recursion is
motivated by the need to correctlymatch those featuresthat have feature structuresas
theirvalues.
A notable aspect of this algorithm is that rather than constr ucting a new feature
structure with the uniﬁed information from the two argument s, it destructively alters
the arguments so that in the end they point to exactly the same information. Thus,
the result of a successful call to the uniﬁcation operator co nsists of suitably altered
versions of the arguments. As is discussed in the next sectio n, the destructive nature
of this algorithm necessitates certain minor extensions to the simple graph version of
featurestructuresasDAGswehavebeenassuming.
15.4.1 UniﬁcationData Structures
To facilitate the destructive merger aspect of the algorith m, we add a small compli-

cation to the DAGs used to represent the input feature struct ures; feature structures
are representedusing DAGswith additionaledges, or ﬁelds. Speciﬁcally,each feature
structure consists of two ﬁelds: a content ﬁeld and a pointer ﬁeld. The content ﬁeld
may be null or contain an ordinary feature structure. Simila rly, the pointer ﬁeld may
benullorcontainapointertoanotherfeaturestructure. If thepointerﬁeld oftheDAG
is null, then the content ﬁeld of the DAG contains the actual f eature structure to be

processed. If, on the other hand, the pointer ﬁeld is non-nul l, then the destination of
the pointerrepresentsthe actual feature structure to be pr ocessed. The mergeraspects
ofuniﬁcationwillbeachievedbyalteringthepointerﬁeldo fDAGsduringprocessing.
To see howthis works, let’s considerthe extendedDAG repres entationforthe fol-
lowingfamiliarfeaturestructure.
(15.19)[
NUMBER sg
Figure15.5showsthisextendedrepresentationin itsgraph icalform. Note that the

(15.19)[
NUMBER sg
extended representationcontains content and pointer link s both for the top-level layer
of features,as well as foreach of the embeddedfeaturestruc turesall the way downto
theatomicvalues.
Before presentingthe details of the uniﬁcation algorithm, let’s illustrate the use of
this extended DAG representation with the following simple example. The original
extendedrepresentationoftheargumentstothisuniﬁcatio nareshowninFigure15.6.
(15.20)[
NUMBER sg]
⊔[
=[
NUMBER sg

(15.20)[
NUMBER sg]
⊔[
=[
NUMBER sg
Atabasiclevel,thisuniﬁcationresultsinthecreationofa newstructurecontaining
theunionoftheinformationfromthetwooriginalarguments . Withtheextendednota-
tion, we can see howthe uniﬁcationis accomplishedby making some additionsto the
originalargumentsandchangingsomeofthepointersfromon estructuretotheotherso
that in theend theycontainthe same content. Inthisexample ,this isaccomplishedby
ﬁrst adding a PERSONfeature to the ﬁrst argument, and assigning it a value by ﬁlli ng

DRAFTSection15.4. ImplementingUniﬁcation 513
itsPOINTER ﬁeld with a pointerto the appropriatelocationin the second argument,as
shownin Figure15.7.
The process is, however, not yet complete. While it is clear f rom Figure 15.7
that the ﬁrst argument now contains all the correct informat ion, the second one does
not; it lacks a NUMBER feature. We could, of course, add a NUMBER feature to this
argument with a pointer to the appropriate place in the ﬁrst o ne. This change would
result in the two arguments having all the correct informati on from this uniﬁcation.
Unfortunately, this solution is inadequate since it does no t meet our requirement that
thetwoargumentsbetrulyuniﬁed. Sincethetwoargumentsar enotcompletelyuniﬁed
at the top level, futureuniﬁcationsinvolvingone of the arg umentswould not show up
in the other. The solution to this problem is to simply set the POINTER ﬁeld of the
secondargumenttopointattheﬁrstone. Whenthisisdoneany futurechangetoeither

argumentwillbeimmediatelyreﬂectedinboth. Theresultof thisﬁnalchangeisshown
inFigure15.8.
15.4.2 The UniﬁcationAlgorithm
Theuniﬁcationalgorithmthatwehavebeenleadinguptoissh owninFigure15.9. This
algorithmacceptstwofeaturestructuresrepresentedusin gtheextendedDAGrepresen-
tation and returns as its value a modiﬁed version of one of the arguments, or a failure
signalinthe eventthatthefeaturestructuresareincompat ible.
The ﬁrst step in this algorithm is to acquire the true content s of both of the argu-

ments. Recall that if the pointerﬁeld ofan extendedfeature structureis non-null,then
therealcontentofthatstructureisfoundbyfollowingthep ointerfoundinpointerﬁeld.
Thevariables f1andf2aretheresultofthispointerfollowingprocess,oftenrefe rredto
asdereferencing . Dereferencing
CONTENTCONTENT
CONTENT POINTERPOINTER
POINTERNUMBER
PERSONsg
NULLNULL
NULL3rd
Figure15.5 Anextended DAG notation forExample 15.19.

DRAFT514 Chapter 15. FeaturesandUniﬁcation
CONTENT
POINTERPOINTERPERSON3rd
NULLNULLCONTENTCONTENT
POINTERPOINTERNUMBERsg
NULLNULL
CONTENT
Figure15.6 The original arguments toExample 15.20.
As with all recursive algorithms, the next step is to test for the various base cases
of the recursion before proceeding on to a recursive call inv olving some part of the
originalarguments. Inthiscase,therearethreepossibleb ase cases:
•Theargumentsareidentical
•Oneorbothoftheargumentshasanullvalue
•Theargumentsarenon-nullandnon-identical
If the structures are identical, then the pointer of the ﬁrst is set to the second and
the second is returned. It is important to understand why thi s pointer change is done
in this case. After all, since the arguments are identical, r eturning either one would
appear to sufﬁce. This might be true for a single uniﬁcation b ut recall that we want
the two arguments to the uniﬁcation operator to be truly uniﬁ ed. The pointer change

is necessary since we want the argumentsto be truly identica l, so that any subsequent
uniﬁcationthataddsinformationtoonewill addit to both.
Inthecase whereeitheroftheargumentsisnull,thepointer ﬁeld forthenullargu-
ment is changed to point to the other argument, which is then r eturned. The result is
thatbothstructuresnowpointat thesame value.
If neither of the preceding tests is true then there are two po ssibilities: they are

non-identicalatomic values, or they are non-identicalcom plex structures. The former

DRAFTSection15.4. ImplementingUniﬁcation 515
CONTENT
NULLNULLCONTENT
CONTENT
NULLNULL
CONTENTPERSON
NULLCONTENT
POINTER
Figure15.7 The arguments after assigning the ﬁrst argument’s new PERSONfeature to the
appropriate value inthe second argument.
case signals an incompatibility in the arguments that leads the algorithm to return a
failuresignal. Inthelattercase,arecursivecallisneede dtoensurethatthecomponent
partsofthesecomplexstructuresarecompatible. Inthisim plementation,thekeytothe
recursionis a loopoverall the featuresof the secondargument, f2. Thisloopattempts
to unify the value of each feature in f2with the corresponding feature in f1. In this
loop, if a feature is encountered in f2that is missing from f1, a feature is added to f1
and given the value NULL. Processing then continues as if the feature had been there
tobeginwith. If everyoneofthese uniﬁcationssucceeds,thenthepointerﬁeldof f2is

set tof1completing the uniﬁcation of the structures and f1is returned as the value of
theuniﬁcation.
An Example
Toillustrate thisalgorithm,let’swalkthroughthefollow ingexample.

DRAFT516 Chapter 15. FeaturesandUniﬁcation
CONTENT
CONTENT
NULLNULL
POINTER
POINTER
Figure15.8 The ﬁnal resultof unifying F1andF2.
(15.21)
AGREEMENT 1[
NUMBER sg]
SUBJECT[

⊔[
SUBJECT[
PERSON 3rd]]]
Figure 15.10shows the extendedrepresentationsfor the arg umentsto this uniﬁca-
tion. These original argumentsare neither identical, nor n ull, nor atomic, so the main
loop is entered. Looping over the features of f2, the algorithm is led to a recursive
attempttounifythevaluesofthecorresponding SUBJECT featuresof f1andf2.[
AGREEMENT 1]
⊔[
PERSON 3rd]]
These arguments are also non-identical, non-null, and non- atomic so the loop is
enteredagainleadingto arecursivecheckofthe valuesofth eAGREEMENT features.[
NUMBER sg]
⊔[
In looping over the features of the second argument, the fact that the ﬁrst argu-
mentlacksa PERSONfeatureisdiscovered. A PERSONfeatureinitializedwitha NULL

⊔[
NUMBER sg]
⊔[
value is, therefore, added to the ﬁrst argument. This, in eff ect, changes the previous
uniﬁcationtothefollowing.[
NUMBER sg
PERSON null]
⊔[

DRAFTSection15.5. Parsingwith UniﬁcationConstraints 517
function UNIFY(f1-orig,f2-orig )returnsf-structure or failure
f1←Dereferenced contents of f1-orig
f2←Dereferenced contents of f2-orig
iff1andf2are identical then
f1.pointer←f2
returnf2
else iff1is nullthen
returnf2
else iff2is nullthen
f2.pointer←f1
returnf1
else ifbothf1andf2are complex feature structures then
for eachf2-feature inf2do
f1-feature←Findor create a corresponding feature in f1
ifUNIFY(f1-feature.value ,f2-feature.value )returnsfailurethen
returnfailure
returnf1
else return failure
Figure15.9 The uniﬁcationalgorithm.
Aftercreatingthisnew PERSONfeature,thenextrecursivecallleadstotheuniﬁca-
tionofthe NULLvalueofthenewfeatureintheﬁrstargumentwiththe 3rdvalueofthe
secondargument. Sincetherearenofurtherfeaturestochec kinthef2argumentatany
level of recursion,each of the recursivecalls to U NIFYreturns. The result is shown in
Figure15.11.
15.5 ParsingwithUniﬁcationConstraints

returnf1
Wenowhaveallthepiecesnecessarytointegratefeaturestr ucturesanduniﬁcationinto
a parser. Fortunately, the order-independentnature of uni ﬁcation allows us to largely
ignoretheactualsearchstrategyusedintheparser. Oncewe haveassociateduniﬁcation
constraints with the context-freerules of the grammar, and feature structures with the
states of the search, any of the standard search algorithms d escribed in Ch. 13 can be
used.
Ofcourse,thisleavesafairlylargerangeofpossibleimple mentationstrategies. We

used.
could, for example, simply parse as we did before using the co ntext-free components
of the rules, and then build the feature structures for the re sulting trees after the fact,
ﬁltering out those parses that contain uniﬁcation failures . Although such an approach
would result in only well-formed structures in the end, it fa ils to use the power of
uniﬁcationtoreducethesize oftheparser’ssearchspacedu ringparsing.

DRAFT518 Chapter 15. FeaturesandUniﬁcation
POINTERPOINTERPOINTERPOINTERPOINTER
POINTERPOINTER
CONTENTCONTENTCONTENTCONTENTCONTENT
SUBJSUBJ
AGRAGR
AGR
PERSONCONTENT
3rdsg
NUMBER
NULLNULL
NULL POINTER
NULL
NULL
NULL
NULLNULL
Figure15.10 The initialarguments f1andf2toExample 15.21.
The next section describes an approach that makes better use of the power of uni-
ﬁcation by integrating uniﬁcation constraints directly in to the Earley parsing process,
allowing ill-formed structures to be eliminated as soon as t hey are proposed. As we
will see, this approach requires only minimal changes to the basic Earley algorithm.
We then move on to brieﬂy consider an approach to uniﬁcation- based parsing that
movesevenfurtherawayfromstandardcontext-freemethods .
15.5.1 Integrating Uniﬁcation into anEarley Parser
We have two goals in integrating feature structures and uniﬁ cation into the Earley al-
gorithm: tousefeaturestructurestoprovidearicherrepre sentationfortheconstituents

oftheparse,andtoblockentryintothechartofill-formedc onstituentsthatviolateuni-
ﬁcationconstraints. Aswewillsee,thesegoalscanbeaccom plishedbyfairlyminimal
changesto theoriginalEarleyschemegivenonpage448.
Theﬁrst changeinvolvesthevariousrepresentationsusedi n theoriginalcode. Re-
call that the Earley algorithmoperatesby using a set of unad ornedcontext-freegram-
mar rules to ﬁll in a data-structure called a chart with a set o f states. At the end of
the parse, the states that make up this chart represent all po ssible parses of the input.

Therefore,webeginourchangesbyalteringtherepresentat ionsofboththecontext-free
grammarrules,andthestatesin thechart.
The rules are altered so that in addition to their current com ponents, they also in-

DRAFTSection15.5. Parsingwith UniﬁcationConstraints 519
POINTERPOINTERPOINTERPOINTER
CONTENTCONTENTCONTENTCONTENTCONTENTCONTENT
SUBJSUBJ
AGRAGR
AGRPERSON
3rd
 
NUMBER
NULLNULL
NULL
NULLNULL
Figure15.11 The ﬁnal structures of f1andf2at the end.
clude a feature structure derived from their uniﬁcation con straints. More speciﬁcally,
wewill usetheconstraintslistedwith aruletobuilda featu restructure,representedas
a DAG,forusewith thatruleduringparsing.
Considerthefollowingcontext-freerulewith uniﬁcationc onstraints.
S→NPVP
⟨NPHEAD AGREEMENT ⟩=⟨VPHEAD AGREEMENT ⟩
⟨SHEAD⟩=⟨VPHEAD⟩
Convertingtheseconstraintsintoafeaturestructureresu ltsin thefollowingstructure:

S[
HEAD 1]
NP[
HEAD[
AGREEMENT 2]]
VP[
HEAD 1[
AGREEMENT 2]]

In this derivation, we combined the various constraints int o a single structure by ﬁrst
creating top-level features for each of the parts of the cont ext-free rule, S,NP, and

S→NPVP

S[
HEAD 1]
NP[
HEAD[
VP[
HEAD 1[

VPin this case. We then add further components to this structur e by following the

DRAFT520 Chapter 15. FeaturesandUniﬁcation
path equations in the constraints. Note that this is a purely notational conversion; the
DAGs and the constraint equations contain the same informat ion. However, tying the
constraints together in a single feature structure puts it i n a form that can be passed
directlyto ouruniﬁcationalgorithm.
Thesecondchangeinvolvesthestatesusedtorepresentpart ialparsesintheEarley
chart. Theoriginalstatescontainﬁeldsforthecontext-fr eerulebeingused,theposition
of the dot representinghow much of the rule has been complete d, the positions of the
beginning and end of the state, and a list of other states that represent the completed
sub-partsofthe state. Tothisset ofﬁelds,we simplyaddana dditionalﬁeldto contain
the DAG representingthe feature structure correspondingt o the state. Note that when
a rule is ﬁrst used by P REDICTOR to create a state, the DAG associated with the state
willsimplyconsistoftheDAGretrievedfromtherule. Forex ample,whenP REDICTOR

uses the above Srule to enter a state into the chart, the DAG given above will b e its
initial DAG. We’ll denote states like this as follows, where Dagdenotes the feature
structuregivenabove.
S→•NPVP ,[0,0],[],Dag
Given these representational additions, we can move on to al tering the algorithm
itself. The most important change concerns the actions that take place when a new
state is created by the extension of an existing state, which takes place in the C OM-

PLETERroutine. Recall that C OMPLETER is called when a completed constituent has
been addedto the chart. Its task is to attempt to ﬁnd, and exte nd,existing states in the
chart that are looking for constituents that are compatible with the newly completed
constituent. C OMPLETER is, therefore, a function that creates new states by combin-
ingthe information from two other states, and as such is a likely place to apply the
uniﬁcationoperation.
Tobemorespeciﬁc, C OMPLETER addsa newstate intothe chartbyﬁndinganex-

istingstatewhose •canbeadvancedbythenewlycompletedstate. A •canbeadvanced
whenthe categoryof theconstituentimmediatelyfollowing it matchesthecategoryof
the newly completed constituent. To accommodate the use of f eature structures, we
can alter this scheme by unifyingthe feature structureasso ciated with the newly com-
pleted state with the appropriate part of the feature struct ure being advanced. If this
uniﬁcation succeeds, then the DAG of the new state receives t he uniﬁed structure and
is entered into the chart. If it fails, then no new state is ent ered into the chart. The

appropriatealterationsto C OMPLETER are showninFigure15.12.
Consider this process in the context of parsing the phrase That ﬂight , where the
Thathasalreadybeenseen,asiscapturedbythe followingstate.
NP→Det•Nominal [0,1],[SDet],Dag1
Dag1
NP[
HEAD 1]
DET[
HEAD[
AGREEMENT 2[
NUMBER SG]]]
NOMINAL[
HEAD 1[


DRAFTSection15.5. Parsingwith UniﬁcationConstraints 521
Now consider the later situation where the parser has proces sedﬂightand has subse-
quentlyproducedthefollowingstate.
Nominal→Noun•,[1,2],[SNoun],Dag2
Dag2
NOMINAL[
HEAD 1]
NOUN[
HEAD 1[
NUMBER SG]]]

To advance the NPrule, the parser uniﬁes the feature structure foundunder th eNOM-
INALfeature of Dag2, with the feature structure found under the NOMINAL feature of
theNP’sDag1. Asintheoriginalalgorithm,anewstateiscreatedtorepre sentthefact
that an existing state has been advanced. This new state’s DA G is given the DAG that
resultedfromthisuniﬁcation.
The ﬁnal change to the original algorithm concerns the check for states already
contained in the chart. In the original algorithm, the E NQUEUE function refused to
enter into the chart any state that was identical to one already present in the chart.
“Identical”meantthe same rule,withthe same start andﬁnis hpositions,andthe same

positionofthe•. Itisthischeckthatallowsthealgorithmto,amongotherth ings,avoid
theinﬁniterecursionproblemsassociatedwithleft-recur siverules.
The problem, of course, is that our states are now more comple x since they have
complex feature structures associated with them. States th at appeared identical under
the original criteria might in fact now be different since th eir associated DAGs may
differ. Onesolutiontothisproblemistoextendtheidentit ychecktoincludetheDAGs
associatedwiththe states, butit turnsoutthat wecanimpro veonthissolution.

The motivation for the improvement lies in the motivation fo r the identity check.
Its purposeis to prevent the wasteful additionof a state int o the chart whose effect on
the parse would be accomplished by an already existing state . Put another way, we
want to preventthe entryinto the chart of any state that woul d duplicatethe work that
will eventually be done by other states. Of course, this will clearly be the case with
identical states, but it turns out it is also the case for stat es in the chart that are more

generalthannewstatesbeingconsidered.
Considerthe situation wherethe chart containsthe followi ngstate, where the Dag
placesnoconstraintsonthe Det.
NP→•DetNP ,[i,i],[],Dag
Such a state simply says that it is expecting a Detat position i, and that any Detwill
do.
Now consider the situation where the parser wants to insert a new state into the
chartthatisidenticaltothisone,withtheexceptionthati tsDAGrestrictsthe Dettobe
singular. In this case, although the states in question are n ot identical, the addition of

do.
thenewstatetothechartwouldaccomplishnothingandshoul dthereforebeprevented.
To see this let’s considerall the cases. If the new state is ad ded,then a subsequent
singularDetwillmatchbothrulesandadvanceboth. Duetotheuniﬁcation offeatures,
both will have DAGs indicating that their Dets are singular, with the net result being

DRAFT522 Chapter 15. FeaturesandUniﬁcation
ADDTOCHART((γ→•S,[0,0],dagγ),chart[0])
else
end
end
procedure PREDICTOR ((A→α•Bβ,[i,j],dagA))
ADDTOCHART((B→•γ,[j,j],dagB),chart[j])
end
procedure SCANNER((A→α•Bβ,[i,j],dagA))
ifB∈PARTS-OF-SPEECH(word[j])then
ADDTOCHART((B→word[j]•,[j,j+1],dagB),chart[j+1] )
procedure COMPLETER ((B→γ•,[j,k],dagB))
for each(A→α•Bβ,[i,j],dagA)inchart[j]do
ifnew-dag←UNIFY-STATES(dagB,dagA,B)̸=Fails!
ADDTOCHART((A→αB•β,[i,k],new-dag),chart[k])
end
procedure UNIFY-STATES(dag1,dag2,cat)
dag1-cp←COPYDAG(dag1)
dag2-cp←COPYDAG(dag2)
UNIFY(FOLLOW-PATH(cat,dag1-cp),FOLLOW-PATH(cat,dag2-cp))
procedure ADDTOCHART(state,chart-entry )

end
end
ifstateisnot subsumed by astate in chart-entry then
PUSH-ON-END(state,chart-entry )
end
Figure15.12 Modiﬁcations tothe Earleyalgorithm toinclude uniﬁcation .
duplicatestates in the chart. If on the otherhand, a plural Detis encountered,the new
state will reject it and not advance, while the old rule will a dvance, entering a single
new state into the chart. On the other hand, if the new state is not placed in the chart,
a subsequent plural or singular Detwill match the more general state and advance it,

end
leading to the addition of one new state into the chart. Note t hat this leaves us in
exactly the same situation as if the new state had been entere d into the chart, with the
exceptionthattheduplicationisavoided. Insum,nothingw orthwhileis accomplished

DRAFTSection15.5. Parsingwith UniﬁcationConstraints 523
byenteringintothecharta state thatismorespeciﬁcthana s tatealreadyinthechart.
Fortunately,thenotionof subsumption introducedearliergivesusaformalwayto
talkaboutthegeneralizationandspecializationrelation samongfeaturestructures. This
suggests that the proper way to alter E NQUEUE is to check whether a newly created
state issubsumed byanyexistingstatesin the chart. If it is, thenit will notb eallowed
into the chart. More speciﬁcally, a new state that is identic al in terms of its rule, start
andﬁnishpositions,subparts,and •position,toanexistingstate,willbenotbeentered
into the chart if its DAG is subsumedby the DAG of an existing s tate (ie. if Dagold⊑
Dagnew). The necessary change to the original Earley E NQUEUE procedure is shown
inFigure15.12.
The Need forCopying
The calls to C OPYDAGwithin the U NIFY-STATEprocedurerequiresome elaboration.
Recall that one of the strengthsof the Earley algorithm(and of the dynamicprogram-

mingapproachingeneral)isthatoncestateshavebeenenter edintothecharttheymay
be used again and again as part of differentderivations, inc luding ones that in the end
do not lead to successful parses. This ability is the motivat ion for the fact that states
already in the chart are not updated to reﬂect the progress of their•, but instead are
copied and then updated, leaving the original states intact so that they can be used
againin furtherderivations.
The call to C OPYDAGin UNIFY-STATEis required to preserve this behavior be-

cause of the destructive nature of our uniﬁcation algorithm . If we simply uniﬁed the
DAGs associated with the existing states, those states woul d be altered by the uniﬁca-
tion, and hence would not be available in the same form for sub sequent uses by the
COMPLETER function. Notethatthishasnegativeconsequencesregardl essofwhether
theuniﬁcationsucceedsorfails,since ineithercase theor iginalstatesarealtered.
Let’s consider what would happen if the call to C OPYDAGwas absent in the fol-
lowingexamplewhereanearlyuniﬁcationattemptfails.

(15.22)Showme morningﬂights.
Let’s assume that our parser has the following entry for the d itransitive version of the
verbshow,aswell asthefollowingtransitiveandditransitiveverbp hraserules.
Verb→show
⟨VerbHEAD SUBCAT SECOND CAT ⟩=NP
⟨VerbHEAD SUBCAT THIRD ⟩=END
VP→VerbNP
⟨VPHEAD SUBCAT SECOND ⟩=END

DRAFT524 Chapter 15. FeaturesandUniﬁcation
⟨VPHEAD SUBCAT FIRST CAT ⟩=⟨NP1CAT⟩
⟨VPHEAD SUBCAT SECOND CAT ⟩=⟨NP2CAT⟩
⟨VPHEAD SUBCAT THIRD ⟩=END
When the word meis read, the state representing the transitive verb phrase w ill
be completed since its dot has moved to the end. C OMPLETER will, therefore, call
UNIFY-STATESbeforeattemptingto enter this complete state into the char t. This will
failsincethe SUBCATstructuresofthesetworulescannotbeuniﬁed. Thisis,ofco urse,
exactlywhatwewantsincethisversionof showisditransitive. Unfortunately,because
ofthedestructivenatureofouruniﬁcationalgorithmwe hav ealreadyalteredtheDAG
attached to the state representing show, as well as the one attached to the VPthereby
ruiningthemforusewiththecorrectverbphraserulelatero n. Thus,tomakesurethat
states can be used again and again with multiple derivations , copies are made of the
dagsassociatedwithstatesbeforeattemptinganyuniﬁcati onsinvolvingthem.

All of this copying can be quite expensive. As a result, a numb er of alternative
techniqueshavebeendevelopedthatattempttominimizethi scost(Pereira,1985;Kart-
tunenandKay,1985;Tomabechi,1991;Kogure,1990). Kiefer etal.(1999b)andPenn
and Munteanu (2003) describe a set of related techniques use d to speed up a large
uniﬁcation-basedparsingsystem.
15.5.2 Uniﬁcation-Based Parsing
Amoreradicalapproachtousinguniﬁcationinparsingcanbe motivatedbylookingat
an alternative way of denoting our augmented grammar rules. Consider the following

Srulethatwe havebeenusingthroughoutthischapter.
S→NPVP
An interesting way to alter the context-free part of this rul e is to change the way its
grammaticalcategoriesare speciﬁed. In particular,we can place the categoricalinfor-
mation about the parts of the rule inside the feature structu re, rather than inside the
context-freepartof the rule. A typicalinstantiationofth isapproachwouldgiveusthe
followingrule(Shieber,1986).
X0→X1X2
⟨X0CAT⟩=S
⟨X1CAT⟩=NP
⟨X2CAT⟩=VP
⟨X1HEAD AGREEMENT ⟩=⟨X2HEAD AGREEMENT ⟩

X0→X1X2
⟨X0CAT⟩=S
⟨X0HEAD⟩=⟨X2HEAD⟩
Focusing solely on the context-free component of the rule, t his rule now simply
states that the X0constituent consists of two components, and that the X1constituent

DRAFTSection15.5. Parsingwith UniﬁcationConstraints 525
is immediately to the left of the X2constituent. The information about the actual cat-
egories of these components is placed inside the rule’s feat ure structure; in this case,
indicating that X0is anS,X1is anNP, andX2is aVP. Altering the Earley algorithm
to deal with this notational change is trivial. Instead of se eking the categories of con-
stituents in the context-freecomponentsof the rule, it sim ply needsto look at the CAT
featurein theDAGassociatedwitha rule.
Ofcourse,sinceit is thecase thatthese two rulescontainpr eciselythesame infor-
mation,itisn’tclearthatthereisanybeneﬁttothischange . Toseethepotentialbeneﬁt
ofthischange,considerthe followingrules.
X0→X1X2
⟨X0CAT⟩=⟨X1CAT⟩
⟨X2CAT⟩=PP
X0→X1andX2
⟨X1CAT⟩=⟨X2CAT⟩
The ﬁrst rule is an attempt to generalize over various rules t hat we have already
seen, such as NP→NPPPandVP→VPPP. It simply states that any category can

X0→X1X2
be followed by a prepositional phrase, and that the resultin g constituent has the same
category as the original. Similarly, the second rule is an at tempt to generalize over
rulessuchas S→SandS,NP→NPand NP ,andso on.1It statesthatanyconstituent
canbeconjoinedwithaconstituentofthesamecategorytoyi eldanewcategoryofthe
same kind. What these ruleshavein commonis theiruse of phra sestructurerulesthat
contain constituents with constrained, but unspeciﬁed, ca tegories, something that can
notbeaccomplishedwith ouroldruleformat.

Of course, since these rules rely on the use the CATfeature, their effect could be
approximatedin theoldformatbysimplyenumeratingall the variousinstantiationsof
therule. Amorecompellingcaseforthenewapproachismotiv atedbytheexistenceof
grammatical rules, or constructions, that contain constit uents that are not easily char-
acterizedusinganyexistingsyntacticcategory.
Consider the following examples of the English H OW-MANYconstruction from
theWSJ (Jurafsky,1992).
(15.23)Howearly doesit open?
(15.24)Howdeep is herGreenness?
(15.25)Howpapery are yourproﬁts?

(15.26)Howquickly we forget.
(15.27)Howmanyofyou cannamethreefamoussportingBlanchards?
Asisillustratedintheseexamples,theH OW-MANYconstructionhastwocomponents:
the lexical item how, and a lexical item or phrase that is rather hard to character ize
syntactically. It is this secondelement that is of interest to us here. As these examples
1Theserules should not bemistaken for correct, orcomplete, accounts of the phenomena in question.

DRAFT526 Chapter 15. FeaturesandUniﬁcation
show, it can be an adjective, adverb, or some kind of quantiﬁe d phrase (although not
all members of these categories yield grammatical results) . Clearly, a better way to
describethis secondelementis as a scalarconcept,a constraintcan be capturedusing
featurestructures,asin thefollowingrule.
X0→X1X2
⟨X1ORTH⟩=⟨how⟩
⟨X2SEM⟩=⟨SCALAR⟩
Acompleteaccountofruleslikethisinvolvessemanticsand willthereforehavetowait
for Ch. 17. The key point here is that by using feature structu res a grammatical rule
can placeconstraintsonits constituentsin a mannerthat do esnotmake anyuse of the
notionofasyntacticcategory.
Of course, dealing this kind of rule requires some changes to our parsing scheme.
All of the parsing approaches we have considered thus far are driven by the syntac-
tic category of the various constituents in the input. More s peciﬁcally, they are based
on simple atomic matches between the categories that have be en predicted, and cate-

goriesthat have beenfound. Consider,for example,the oper ationof the C OMPLETER
functionshown in Figure 15.12. Thisfunctionsearchesthe c hart forstates that can be
advanced by a newly completed state. It accomplishes this by matching the category
of the newly completedstate against the category of the cons tituent followingthe •in
the existing state. Clearly this approach will run into trou ble when there are no such
categoriestoconsult.
The remedy for this problem with C OMPLETER is to search the chart for states

whose DAGs unifywith the DAG of the newly completed state. This eliminates an y
requirementthat states or rules have a category. The P REDICTOR can be changedin a
similarfashionbyhavingitaddstatestothechartstateswh oseX0DAGcomponentcan
unify with the constituent following the •of the predicting state. Exercise 6 asks you
tomakethenecessarychangestothepseudo-codeinFigure15 .12toeffectthisstyleof
parsing. Exercise 7 asks you to considersome of the implicat ionsof these alterations,
particularlywith respecttoprediction.
15.6 TypesandInheritance

I am surprised that ancient and modern writers have not attri buted greater im-
portanceto thelawsofinheritance...
Alexis de Tocqueville, Democracy inAmerica , 1840
The basic featurestructureswe have presentedso far have tw o problemsthat have
led to extensions to the formalism. The ﬁrst problem is that t here is no way to place
a constraint on what can be the value of a feature. For example , we have implicitly
assumed that the NUMBER attribute can take only sgandplas values. But in our

currentsystem,thereisnothing,forexample,tostop NUMBER fromhavethevalue 3rd
orfeminineasvalues:

DRAFTSection15.6. TypesandInheritance 527
[
NUMBER FEMININE]
This problem has caused many uniﬁcation-basedgrammatical theories to add var-
ious mechanisms to try to constrain the possible values of a f eature. Formalisms like
Functional Uniﬁcation Grammar (FUG) (Kay, 1979, 1984, 1985 ) and Lexical Func-
tional Grammar(LFG) (Bresnan,1982),forexample,focused on waysto keep intran-
sitive verb like sneezefrom unifying with a direct object ( Marin sneezed Toby ). This
was addressed in FUG by adding a special atom nonewhich is not allowed to unify None
withanything,andinLFGbyadding coherence conditionswhichspeciﬁedwhenafea-
tureshouldnotbeﬁlled. TheGeneralizedPhraseStructureG rammar(GPSG)(Gazdar
etal.,1985,1988)addedaclassof featureco-occurrencerestrictions ,toprevent,for
example,nounsfromhavingsomeverbalproperties.
Thesecondproblemwithsimplefeaturestructuresisthatth ereisnowaytocapture
generalizations across them. For example, the many types of English verb phrases

describedin the Subcategorizationsection on page 506shar e many features, as do the
manykindsofsubcategorizationframesforverbs. Syntacti cianswerelookingforways
toexpressthesegeneralities.
A generalsolution to both of these problemsis the use of types. Type systems for Types
uniﬁcationgrammarshavethefollowingcharacteristics:
1. Eachfeaturestructureislabeledbya type.
2. Conversely, each type has appropriateness conditions expressing which fea- Appropriateness

turesareappropriateforit andwhattypesofvaluesthencan take.
3. The types are organized into a type hierarchy , in which more speciﬁc types Type hierarchy
inheritpropertiesofmoreabstractones.
4. The uniﬁcation operation is modiﬁed to unify the types of f eature structures in
additionto unifyingtheattributesandvalues.
Insuchtypedfeaturestructure systems,typesareanewclassofobjects,justlikeTypedfeaturestructures
attributes and values were for standard feature structures . Types come in two kinds:

simple types (also called atomic types ), andcomplex types . Let’s begin with simple Simple types
Complex types types. A simple type is an atomic symbol like sgorpl(we will use boldface for all
types), and replaces the simple atomic values used in standa rd feature structures. All
typesareorganizedintoamultiple-inheritance typehierarchy (akindof partialorder
called alattice). Fig. 15.13 shows the type hierarchy for the new type agreement ,

which will be the type of the kind of atomic object that can be t he value of an AGREE
feature.
In the hierarchy in Fig. 15.13, 3rdis asubtypeofagr, and3-sgis asubtypeof Subtype
both3rdandsg. Typescan beuniﬁedin thetypehierarchy;theuniﬁcationof anytwo
typesisthemost-generaltypethat ismorespeciﬁcthanthet wo inputtypes. Thus:
3rd⊔sg=3sg
1st⊔pl=1pl
1st⊔agr=1st
3rd⊔1st=undeﬁned
The uniﬁcation of two types which do not have a deﬁned uniﬁer i s undeﬁned, al-
thoughit isalsopossibletoexplicitlyrepresentthis failtypeusingthesymbol⊥(A¨ ıt- Fail type

DRAFT528 Chapter 15. FeaturesandUniﬁcation
agr
1st 3rd sg pl
1st-sg 3rd-sg 1st-pl 3rd-pl
3sg-masc 3sg-fem 3sg-neut
Figure15.13 A simple type hierarchy for the subtypes of type agrwhich can be the value of
theAGREEattribute. After Carpenter (1992).
Kaci,1984).
Thesecondkindoftypesarecomplextypes,whichspecify:
•aset offeaturesthatareappropriateforthat type
•restrictionsonthevaluesofthosefeatures(expressedint ermsoftypes)
•equalityconstraintsbetweenthevalues
Consider a simpliﬁed representation of the complex type verb, which just repre-
sentsagreementandverbalmorphologyinformation. Adeﬁni tionofverbwoulddeﬁne
thetwoappropriatefeatures, AGREEandVFORM,andwouldalsodeﬁnethetypeofthe
values of the two features. Let’s suppose that the AGREEfeature takes values of type
agrdeﬁned in Fig. 15.13 above, and the VFORMfeature takes values of type vform
wherevformsubsumes the seven subtypes ﬁnite,inﬁnitive ,gerund,base,present-
participle ,past-participle , andpassive-participle . Thusverbwould be deﬁned as

follows (where the convention is to indicate the type either at the top of the AVM or
just tothelowerleftofthe leftbracket):

verb
AGREE agr
VFORM vform

By contrast, the type nounmight be deﬁned with the AGREEfeature, but without
theVFORMfeature:[
noun
AGREE agr]
Theuniﬁcationoperationisaugmentedfortypedfeaturestr ucturesjustbyrequiring
that the types of the two structures unify in addition to the v alues of the component
featuresunifying.

DRAFTSection15.6. TypesandInheritance 529

verb
AGREE 1st
VFORM gerund
⊔
verb
AGREE sg
=
verb
AGREE 1-sg

Complex types are also part of the type hierarchy. Subtypes o f complex types in-
heritallthefeaturesoftheirparents,togetherwiththeco nstraintsontheirvalues. San-
ﬁlippo (1993), for example, uses a type hierarchy to encode t he hierarchical structure
ofthe lexicon. Fig. 15.14showsa small partof thishierarch y,the partthat modelsthe
various subcategories of verbs which take sentential compl ements; these are divided
into the transitive ones (which take direct objects: ( ask yourself whether you have be-
come better informed ) and the intransitive ones ( Monsieur asked whether I wanted to
ride). The type trans-comp-cat would introducethe requireddirect object, constrain-
ingittobeoftype noun-phrase ,whiletypeslike sbase-comp-cat wouldintroducethe
baseform(barestem)complementandconstrainitsvformtob ethe baseform.
comp-cat

comp-cat
trans-comp-cat sfin-comp-cat swh-comp-cat sbase-comp-cat intrans-comp-cat sinf-comp-cat
intr-swh-comp-cat intr-sinf-comp-cat
tr-swh-comp-cat intr-sfin-comp-cat intr-sbase-comp-cattr-sbase-comp-cat tr-sfin-comp-cat
Figure15.14 Part of the type hierarchy for the verb type verb-cat, showing the subtypes of the comp-cat type.
These are allsubcategories of verbs whichtake sentential c omplements. AfterSanﬁlippo(1993).
15.6.1 Advanced: Extensions to Typing

comp-cat
Typed feature structures can be extended by allowing for inh eritance with defaults. Defaults
Defaultsystemshavemainlybeenusedinlexicaltypehierar chiesofthesortdescribed
intheprevioussection,inordertoencodegeneralizations andsubregularexceptionsto
them. Inearlyversionsofdefaultuniﬁcationtheoperation wasorder-dependent,based
on thepriority union operation (Kaplan, 1987). More recent architectures are or der- Priority union
independent (Lascarides and Copestake, 1997; Young and Rou nds, 1993), related to
Reiter’sdefaultlogic(Reiter,1980).

Many uniﬁcation-based theories of grammar, including HPSG (Pollard and Sag,
1987,1994)andLFG(Bresnan,1982)useanadditionalmechan ismbesidesinheritance
forcapturinglexicalgeneralizations: the lexicalrule . Lexicalrules(Jackendoff,1975) Lexicalrule
express lexical generalizations by allowing a reduced, hen ce more redundancy-free
lexicon to be automatically expanded by the rules. See Polla rd and Sag (1994) for
examples,Carpenter(1991)oncomplexityissues, andMeure rsandMinnen(1997)on
efﬁcient implementation. Conversely, see Krieger and Nerb onne (1993) on using the

typehierarchytoreplacelexicalrules.

DRAFT530 Chapter 15. FeaturesandUniﬁcation
Types can also be used to represent constituency. Rules like (15.12) on page 505
usedanormalphrasestructureruletemplateandaddedthefe aturesviapathequations.
Instead, it’s possible to represent the whole phrase struct ure rule as a type. In order
to do this, we need a way to represent constituents as feature s. One way to do this,
following Sag and Wasow (1999), is to take a type phrasewhich has a feature called
DTRS(“daughters”), whose value is a list of phrases. For example the phrase I love
NewYork couldhavethefollowingrepresentation,(showingonlythe DTRSfeature):

phrase
DTRS⟨[
CATPRO
ORTHI]
,
CATVP
DTRS⟨[
CATVerb
ORTHlove]
,[
CATNP
ORTHNewYork]
⟩
⟩

15.6.2 OtherExtensions toUniﬁcation
Therearemanyotherextensionstouniﬁcationbesidestypin g,including pathinequal-
ities(Moshier,1988;Carpenter,1992;CarpenterandPenn,1994) ,negation (Johnson, Pathinequalities
Negation 1988,1990), set-valued features (Pollard and Moshier, 1990),and disjunction (Kay,


DTRS⟨[
CATPRO
ORTHI]
,
CATVP
DTRS⟨[
CATVerb
ORTHlove]
,[
CATNP
⟩
⟩

Set-valuedfeatures
Disjunction1979;Kasper andRounds,1986). Insome uniﬁcationsystemst hese operationsarein-
corporatedinto feature structures. Kasper and Rounds (198 6) and others, by contrast,
implement them in a separate metalanguage which is used to describefeature struc-
tures. This idea derivesfrom the work of Pereira and Shieber (1984),and even earlier
work by Kaplan and Bresnan (1982), all of whom distinguished between a metalan-
guage for describing feature structures and the actual feat ure structures themselves.

The descriptions may thus use negation and disjunction to de scribe a set of feature
structures(i.e.,acertainfeaturemustnotcontainacerta invalue,ormaycontainanyof
a set of values) but an actual instance of a feature structure that meets the description
wouldnothavenegatedordisjointvalues.
The uniﬁcation grammarsas described so far have no mechanis mfor disambigua-
tion. Much recent work in uniﬁcation grammars has focused on this disambiguation

problem,particularvia the use ofprobabilisticaugmentat ions. See the Historysection
forimportantreferences.
15.7 Summary
This chapter introduced feature structures and the uniﬁcat ion operation which is used
tocombinethem.
•A feature structure is a set of features-valuepairs, where f eaturesare unanalyz-
able atomic symbols drawn from some ﬁnite set, and values are either atomic
symbols or feature structures. They are represented either asattribute-value
matrices (AVMs) or as directed acyclic graphs ( DAGs), where features are di-

rectedlabelededgesandfeaturevaluesarenodesinthegrap h.

DRAFTSection15.7. Summary 531
•Uniﬁcation isthe operationforbothcombininginformation(mergingth e infor-
mation content of two feature structures) and comparing inf ormation (rejecting
themergerofincompatiblefeatures).
•A phrase-structurerule can be augmented with feature struc tures, and with fea-
ture constraints expressing relations among the feature st ructures of the con-
stituentsoftherule. Subcategorization constraintscanberepresentedasfeature
structures on head verbs (or other predicates). The element s which are subcat-
egorized for by a verb may appear in the verb phrase or may be re alized apart
fromtheverb,asa long-distancedependency .
•Feature structures can be typed. The resulting typed feature structures place
constraints on which type of values a given feature can take, and can also be
organizedintoa typehierarchy tocapturegeneralizationsacrosstypes.
The use of features in linguistic theory comes originally fr om phonology. Anderson

(1985) credits Jakobson (1939) with being the ﬁrst to use fea tures (called distinctive
features) as an ontological type in a theory, drawing on previous uses of features by
Trubetskoi (1939) and others. The semantic use of features f ollowed soon after; see
Ch. 19 for the history of componentialanalysis in semantics . Features in syntax were
well establishedbythe1950sandwerepopularizedbyChomsk y(1965).
Theuniﬁcationoperationinlinguisticswasdevelopedinde pendentlybyKay(1979)

(feature structure uniﬁcation) and Colmerauer (1970, 1975 ) (term uniﬁcation) (see
page 13). Both were working in machine translation and looki ng for a formalism for
combining linguistic information which would be reversibl e. Colmerauer’s original
Q-system was a bottom-up parser based on a series of rewrite r ules which contained
logical variables, designed for a English to French machine translation system. The
rewrite rules were reversible to allow them to work for both p arsing and generation.

Colmerauer, Fernand Didier, Robert Pasero, Philippe Rouss el, and Jean Trudel de-
signed the Prolog language based on extended Q-systems to fu ll uniﬁcation based on
theresolutionprincipleofRobinson(1965),andimplement edaFrenchanalyzerbased
on it (Colmerauer and Roussel, 1996). The modern use of Prolo g and term uniﬁca-
tion for natural languagevia Deﬁnite Clause Grammars was based on Colmerauer’sDeﬁnite Clause
Grammars
(1975)metamorphosisgrammars,andwasdevelopedandnamed byPereiraandWarren

Grammars
(1980). Meanwhile Martin Kay and Ron Kaplan had been working with Augmented
Transition Network ( ATN) grammars. An ATN is a Recursive Transition Network ATN
(RTN) in which the nodes are augmented with feature register s. In an ATN analysis
of a passive, the ﬁrst NP would be assigned to the subject regi ster, then when the pas-
sive verbwas encountered,the valuewouldbe movedinto theo bject register. Inorder
to make this process reversible, they restricted assignmen ts to registers so that certain

registers could only be ﬁlled once, that is, couldn’tbe over writtenonce written. They
thus moved toward the concepts of logical variables without realizing it. Kay’s orig-
inal uniﬁcation algorithm was designed for feature structu res rather than terms (Kay,

DRAFT532 Chapter 15. FeaturesandUniﬁcation
1979). The integration of uniﬁcation into an Earley-style a pproach given in Section
15.5isbasedonShieber(1985b).
See Shieber (1986)for a clear introductionto uniﬁcation, a nd Knight (1989) for a
multidisciplinarysurveyofuniﬁcation.
Inheritanceandappropriatenessconditionswereﬁrstprop osedforlinguisticknowl-
edge by Bobrow and Webber (1980) in the context of an extensio n of the KL-ONE
knowledgerepresentationsystem(BrachmanandSchmolze,1 985). Simpleinheritance
withoutappropriatenessconditionswastakenupbynumbero fresearchers;earlyusers
include Jacobs (1985, 1987). A¨ ıt-Kaci (1984) borrowed the notion of inheritance in
uniﬁcation from the logic programming community. Typing of feature structures, in-
cluding both inheritance and appropriatenessconditions, was independentlyproposed
by Calder (1987), Pollard and Sag (1987), and Elhadad (1990) . Typed feature struc-
tures were formalized by King (1989) and Carpenter (1992). T here is an extensive

literatureontheuse oftype hierarchiesinlinguistics, pa rticularlyforcapturinglexical
generalizations; besides the papers previously discussed , the interested reader should
consult Evans and Gazdar (1996) for a description of the DATR language, designed
for deﬁning inheritance networks for linguistic knowledge representation, Fraser and
Hudson (1992) for the use of inheritance in a dependency gram mar, and Daelemans
et al. (1992) for a general overview. Formalisms and systems for the implementation

ofconstraint-basedgrammarsviatypedfeaturestructures includethePAGEsystemus-
ing the TDL language(Krieger and Sch¨ afer, 1994), ALE (Carp enterand Penn, 1994),
ConTroll(G¨ otzet al.,1997)andLKB(Copestake,2002).
EfﬁciencyissuesinuniﬁcationparsingarediscussedbyKie feret al.(1999a),Mal-
oufet al. (2000),andMunteanuandPenn(2004).
Grammatical theories based on uniﬁcation include Lexical F unctional Grammar
(LFG)(Bresnan,1982),Head-DrivenPhraseStructureGramm ar(HPSG)(Pollardand

Sag, 1987, 1994), Construction Grammar (Kay and Fillmore, 1 999), and Uniﬁcation
CategorialGrammar(Uszkoreit,1986).
Much recent computational work on uniﬁcation grammars has f ocused on proba-
bilistic augmentationsfordisambiguation. Key relevantp apersincludeAbney(1997),
Goodman (1997), Johnson et al. (1999), Riezler et al. (2000) , Geman and Johnson
(2002), Riezler et al. (2002, 2003), Kaplan et al. (2004), Mi yao and Tsujii (2005),
Toutanovaet al.(2005),Ninomiyaetal. (2006)andBlunsoma ndBaldwin(2006).
Exercises

Exercises
15.1Drawthe DAGscorrespondingtotheAVMsgiveninExamples15. 1–15.2.
15.2ConsiderthefollowingBERP examples,focusingontheiruse ofpronouns.
I wanttospendlotsofmoney.
Tell meaboutChez-Panisse.

DRAFTSection15.7. Summary 533
I’dliketotake herto dinner.
Shedoesn’tlike Italian.
Assuming that these pronouns all belong to the category Pro, write lexical and
grammatical entries with uniﬁcation constraints that bloc k the following exam-
ples.
*Mewantto spendlotsofmoney.
*TellI aboutChez-Panisse.
*Iwouldliketotake shetodinner.
*Herdoesn’tlikeItalian.
15.3Drawapictureofthesubsumptionsemilatticecorrespondin gtothefeaturestruc-
turesinExamples15.3to15.7. Besuretoincludethemostgen eralfeaturestruc-
ture[].
15.4Considerthefollowingexamples.
Thesheeparebaaaaing.
Thesheepisbaaaaing.
Create appropriatelexical entriesforthe words the,sheep,andbaaaaing . Show
thatyourentriespermitthecorrectassignmentofavalueto theNUMBER feature
forthesubjectsofthese examples,aswellastheirvariousp arts.
15.5Create feature structures expressing the different SUBCATframes for whileand
duringshownonpage510.
15.6AlterthepseudocodeshowninFigure15.12sothatit perform sthemoreradical
kindofuniﬁcation-basedparsingdescribedonpage524.

ture[].
15.7ConsiderthefollowingproblematicgrammarsuggestedbySh ieber(1985b).
S→T
⟨TF⟩= a
T1→T2A
⟨T1F⟩=⟨T2F F⟩
S→A
A→a
Show the ﬁrst Sstate entered into the chart using your modiﬁed PREDICTOR
fromthepreviousexercise,thendescribeanyproblematicb ehaviordisplayedby
PREDICTOR onsubsequentiterations. Discussthecauseoftheproblema ndhow
inmightbe remedied.
15.8Using the list approach to representing a verb’s subcategor ization frame, show
howa grammarcouldhandleanynumberofverbsubcategorizat ionframeswith

DRAFT534 Chapter 15. FeaturesandUniﬁcation
only the following two VPrules. More speciﬁcally, show the constraints that
wouldhavetobeaddedtothese rulestomakethiswork.
VP→Verb
VP→VPX
The solution to this problem involves thinking about a recur sive walk down
a verb’s subcategorization frame. This is a hard problem; yo u might consult
Shieber(1986)if yougetstuck.
15.9Page 530 showed how to use typed feature structures to repres ent constituency.
Usethat notationtorepresentrules15.12,15.13,and15.14 shownonpage505.

Chapter16
Language andComplexity
This is the dog, that worried the cat, that killed the rat, tha t ate the malt, that lay in the
house that Jack built.
MotherGoose, The HousethatJackBuilt
Thisisthe malt that the rat that the cat that the dog worriedk illedate.
VictorH.Yngve(1960)
Much of the humor in musical comedy and comic operetta comes f rom entwining
the main characters in fabulously complicated plot twists. Casilda, the daughter of
the Duke of Plaza-Toro in Gilbert and Sullivan’s The Gondoliers , is in love with her
father’sattendantLuiz. Unfortunately,Casilda discover sshe hasalreadybeenmarried

(by proxy) as a babe of six months to “the infant son and heir of His Majesty the
immeasurablywealthyKingofBarataria”. Itisrevealedtha tthisinfantsonwasspirited
awaybytheGrandInquisitorandraisedbya “highlyrespecta blegondolier”inVenice
as a gondolier. The gondolier had a baby of the same age and cou ld never remember
which child was which, and so Casilda was in the unenviable po sition, as she puts it,
of“beingmarriedtooneoftwo gondoliers,butit isimpossib letosaywhich”. Byway

of consolation, the Grand Inquisitor informs her that “such complications frequently
occur”.
Luckily,suchcomplicationsdon’tfrequentlyoccurinnatu rallanguage. Ordothey?
Infacttherearesentencesthataresocomplexthattheyareh ardtounderstand,suchas
Yngve’ssentenceabove,orthesentence:
“TheRepublicanswhothesenatorwhoshevotedforchastisedw eretrying
tocutall beneﬁtsforveterans ”.
Studying such sentences, and more generally understanding what level of complexity
tendsto occur in natural language,is an importantarea of la nguageprocessing. Com-

occur”.
plexity plays an important role, for example, in deciding wh en we need to use a par-
ticular formal mechanism. Formal mechanisms like ﬁnite aut omata, Markov models,
transducers, phonological rewrite rules, and context-fre e grammars, can be described
in terms of their power, or equivalentlyin terms of the complexity of the phenomena power
complexity that they can describe. This chapter introduces the Chomsky hierarchy, a theoretical
tool that allows us to compare the expressive power or comple xity of these different

formalmechanisms. Withthistoolinhand,wesummarizeargu mentsaboutthecorrect
formalpower of the syntax of natural languages,in particul arEnglish but also includ-
inga famousSwissdialectofGermanthathastheinteresting syntacticpropertycalled
cross-serial dependencies . This property has been used to argue that context-free

DRAFT536 Chapter 16. LanguageandComplexity
grammars are insufﬁciently powerful to model the morpholog y and syntax of natural
language.
In additionto using complexityas a metric for understandin gthe relation between
natural language and formal models, the ﬁeld of complexity i s also concerned with
what makesindividualconstructionsorsentenceshardto un derstand. Forexamplewe
saw above that certain nestedorcenter-embedded sentences are difﬁcult for people
to process. Understanding what makes some sentences difﬁcu lt for people to process
isanimportantpartofunderstandinghumanparsing.
16.1 TheChomskyHierarchy
How are automata, context-free grammars, and phonological rewrite rules related?
What they have in common is that each describes a formal language , which we have
seen is a set of strings over a ﬁnite alphabet. But the kind of g rammars we can write
with each of these formalism are of different generative power . One grammar is of generative power

greatergenerativepoweror complexity thananotherifitcandeﬁnealanguagethatthe
other cannot deﬁne. We will show, for example, that a context -free grammar can be
used to describe formal languagesthat cannot be describedw ith a ﬁnite-state automa-
ton.
It is possible to construct a hierarchyof grammars, where th e set of languagesde-
scribable by grammarsof greater power subsumesthe set of la nguagesdescribableby
grammars of lesser power. There are many possible such hiera rchies; the one that is

ton.
most commonlyused in computationallinguisticsis the Chomsky hierarchy (Chom-Chomsky
hierarchy
sky, 1959a), which includes four kinds of grammars: Fig. 16. 1 shows the four gram-
marsintheChomskyhierarchyaswellasausefulﬁfthtype,th emildlycontext-sensitive
languages.
This decrease in the generative power of languages from the m ost powerful to the
weakestcaningeneralbeaccomplishedbyplacingconstrain tsonthewaythegrammar
rules are allowed to be written. Fig. 16.2 shows the ﬁve types of grammars in the

hierarchy
extended Chomsky hierarchy, deﬁned by the constraints on th e form that rules must
take. Intheseexamples, Aisasinglenon-terminal,and α,β,andγarearbitrarystrings
of terminal and non-terminal symbols. They may be empty unle ss this is speciﬁcally
disallowedbelow. xisanarbitrarystringofterminalsymbols.
Turing-equivalent ,Type 0orunrestricted grammars have no restrictions on the
form of their rules, except that the left-hand side cannot be the empty string ǫ. Any

(non-null)string can be written as anyother string(or as ǫ). Type 0 grammarscharac-
terizethe recursivelyenumerable languages,thatis,thosewhosestringscanbelistedrecursively
enumerable
(enumerated)bya TuringMachine.
Context-sensitive grammars have rules that rewrite a non-terminal symbol A in Context-sensitive
thecontext αAβasanynon-emptystringofsymbols. Theycanbeeitherwritte ninthe
formαAβ→αγβor in the form A→γ/αβ. We have seen this latter version in
the Chomsky-Halle representation of phonological rules (C homsky and Halle, 1968)
likethisﬂappingrule:

DRAFTSection16.1. TheChomskyHierarchy 537
Regular (or Right Linear) LanguagesContext-Free Languages (with no epsilon productions)Mildly Context-Sensitive LanguagesContext-Sensitive LanguagesRecursively Enumerable Languages
Figure16.1 A Venn diagram of the four languages on the Chomsky Hierarchy , augmented
withaﬁfthclass,the mildlycontext-sensitive languages.
TypeCommon Name RuleSkeleton Linguistic Example
0TuringEquivalent α→β,s.t.α̸=ǫ HPSG,LFG,Minimalism
1ContextSensitive αAβ→αγβ, s.t.γ̸=ǫ
-MildlyContextSensitive TAG,CCG
2ContextFree A→γ PhraseStructure Grammars
3Regular A→xBorA→x Finite State Automata
Figure16.2 The Chomsky Hierarchy, augumented bythe mildlycontext-se nsitive grammars.
/t/→[dx]/´VV
While the form of these rules seems context-sensitive, Ch. 7 showed that phono-
logical rule systems that do not have recursion are actually equivalent in power to the
regulargrammars.
Anotherway of conceptualizinga rule in a context-sensitiv egrammaris as rewrit-

ing a string of symbols δas another string of symbols φin a “non-decreasing” way;
suchthat φhasat least asmanysymbolsas δ.
We studied context-free grammarsin Ch. 12. Context-free rules allow any single context-free
non-terminal to be rewritten as any string of terminals and n on-terminals. A non-
terminal may also be rewritten as ǫ, although we didn’t make use of this option in
Ch.12.
Regular grammars are equivalent to regular expressions. Th at is, a given regular

Ch.12.
language can be characterized either by a regular expressio n of the type we discussed
in Chapter 2, or by a regular grammar. Regular grammars can ei ther beright-linear right-linear
orleft-linear . A rule in a right-linear grammar has a single non-terminal o n the left, left-linear
and at most one non-terminal on the right-hand side. If there is a non-terminal on
the right-hand side, it must be the last symbol in the string. The right-hand-side of
left-linear grammarsis reversed(the right-hand-sidemus t start with (at most) a single

non-terminal). Allregularlanguageshavebothaleft-line arandaright-lineargrammar.
Fortherest ofourdiscussion,we will consideronlythe righ t-lineargrammars.

DRAFT538 Chapter 16. LanguageandComplexity
Forexample,considerthefollowingregular(right-linear )grammar:
S→aA
S→bB
A→aS
B→bbS
S→ǫ
It is regular,since the left-hand-sideof each rule is a sing le non-terminaland each
right-handside has at most one (rightmost) non-terminal. H ere is a sample derivation
inthe language:
S⇒aA⇒aaS⇒aabB⇒aabbbS⇒aabbbaA
⇒aabbbaaS⇒aabbbaa
WecanseethateachtimeSexpands,itproduceseither aaSorbbbS;thusthereader
shouldconvincethemselfthatthislanguagecorrespondsto theregularexpression (aa∪
bbb)∗.
Wewillnotpresenttheproofthatalanguageisregularifand onlyifitisgenerated
by a regular grammar; it was ﬁrst proved by Chomsky and Miller (1958) and can be
found in textbooks like Hopcroft and Ullman (1979) and Lewis and Papadimitriou
(1988). Theintuitionisthatsince thenon-terminalsareal waysattherightorleft edge
ofa rule,theycanbeprocessediterativelyratherthanrecu rsively.
The ﬁfth class of languages and grammars that is useful to con sider is the mildly

bbb)∗.
context-sensitive grammars and themildly context-sensitive languages . Mildlymildlycontext-sensitive
context-sensitivelanguagesare apropersubset ofthecont ext-sensitivelanguages,and
a propersuperset of the context-freelanguages. The rules f or mildly context-sensitive
languagescanbedescribedinanumberofways;indeeditturn soutthatvariousgram-
mar formalisms, including Tree-Adjoining Grammars (Joshi , 1985), Head Grammars
Pollard (1984), Combinatory Categorial Grammars (CCG), (S teedman, 1996, 2000)

and also a speciﬁc version of Minimalist Grammars (Stabler, 1997), are all weakly
equivalent(Joshi etal.,1991).
16.2 HowtoTellif aLanguageIsn’tRegular
How do we know which type of rules to use for a given problem? Co uld we use
regularexpressionstowriteagrammarforEnglish? Ordowen eedtousecontext-free
rules or even context-sensitive rules? It turns out that for formal languages there are
methods for deciding this. That is, we can say for a given form al languagewhether it

is representable by a regular expression, or whether it inst ead requires a context-free
grammar,andsoon.

DRAFTSection16.2. HowtoTell ifa LanguageIsn’tRegular 539
Soif we wantto knowif somepartofnaturallanguage(the phon ologyofEnglish,
let’s say, or perhaps the morphologyof Turkish) is represen table by a certain class of
grammars,we needtoﬁnd a formallanguagethat modelstherel evantphenomenaand
ﬁgureoutwhichclassofgrammarsisappropriateforthisfor mallanguage.
Why should we care whether (say) the syntax of English is repr esentable by a
regular language? One main reason is that we’d like to know wh ich type of rule to
use in writing computational grammars for English. If Engli sh is regular, we would
write regular expressions, and use efﬁcient automata to pro cess the rules. If English
is context-free,we would write context-freerulesanduse t he CKY algorithmto parse
sentences,andso on.
Another reason to care is that it tells us something about the formal properties
of different aspects of natural language; it would be nice to know where a language

“keeps”itscomplexity;whetherthephonologicalsystemof alanguageissimplerthan
the syntactic system, or whether a certain kind of morpholog ical system is inherently
simpler than another kind. It would be a strong and exciting c laim, for example, if
we couldshow that the phonologyof Englishwas capturableby a ﬁnite-state machine
rather than the context-sensitive rules that are tradition ally used; it would mean that
Englishphonologyhas quitesimple formalproperties. Inde ed,this fact was shownby
Johnson (1972), and helped lead to the modern work in ﬁnite-s tate methods shown in

Chapters3and4.
16.2.1 The Pumping Lemma
Themostcommonwaytoprovethatalanguageisregularistoac tuallybuildaregular
expression for the language. In doing this we can rely on the f act that the regular
languages are closed under union, concatenation, Kleene st ar, complementation, and
intersection. We saw examples of union, concatenation, and Kleene star in Ch. 2. So
ifwecanindependentlybuildaregularexpressionfortwodi stinctpartsofalanguage,

we can use the union operator to build a regular expression fo r the whole language,
provingthatthelanguageisregular.
Sometimes we want to prove that a given language is notregular. An extremely
usefultoolfordoingthisisthe PumpingLemma . Therearetwointuitionsbehindthis Pumping Lemma
lemma. (Our descriptionof the pumpinglemma drawsfrom Lewi s and Papadimitriou
(1988)andHopcroftandUllman(1979).) First,ifalanguage canbemodeledbyaﬁnite
automaton with a ﬁnite number of states, we must be able to dec ide with a bounded

amount of memory whether any string was in the language or not . This amount of
memory can be different for different automata, but for a giv en automaton it can’t
growlargerfordifferentstrings(sinceagivenautomatonh asaﬁxednumberofstates).
Thusthememoryneedsmustnotbeproportionaltothelengtho ftheinput. Thismeans
for examplethat languageslike anbnare not likely to be regular,since we would need
some way to remember what nwas in order to make sure that there were an equal
numberof a’sandb’s. Thesecondintuitionreliesonthefactthatifaregularl anguage

hasanylongstrings(longerthanthenumberofstatesinthea utomaton),theremustbe
some sort of loop in the automatonfor the language. We can use this fact by showing
thatif alanguage doesn’thavesucha loop,thenitcan’tberegular.
Let’sconsideralanguage LandthecorrespondingdeterministicFSA M,whichhas

DRAFT540 Chapter 16. LanguageandComplexity
Nstates. Consideran input string also of length N. The machinestarts out in state q0;
after seeing 1 symbol it will be in state q1; afterNsymbols it will be in state qn. In
otherwords,astringoflength Nwillgothrough N+1states(from q0toqN). Butthere
are onlyNstates in the machine. This means that at least two of the stat es along the
accepting path (call them qiandqj) must be the same. In other words, somewhere on
an accepting path from the initial to ﬁnal state, there must b e a loop. Fig. 16.3 shows
an illustration of this point. Let xbe the string of symbols that the machine reads on
goingfromtheinitialstate q0tothebeginningoftheloop qi.yisthestringofsymbols
that the machine reads in going through the loop. zis the string of symbols from the
endoftheloop( qj) totheﬁnal acceptingstate ( qN).
qi=jy
q0qN
x z
Figure16.3 A machine with Nstates accepting a string xyzofNsymbols
The machine accepts the concatenation of these three string s of symbols, that is,

qi=jy
q0qN
x z
xyz. But if the machine accepts xyzit must accept xz! This is because the machine
could just skip the loop in processing xz. Furthermore, the machine could also go
aroundthe loop anynumberof times; thusit must also accept xyyz,xyyyz,xyyyyz, and
so on. Infact,it mustacceptanystringoftheform xynzforn≥0.
The version of the pumping lemma we give is a simpliﬁed one for inﬁnite regular
languages; stronger versions can be stated that also apply t o ﬁnite languages, but this
onegivesthe ﬂavorofthisclassoflemmas:

Pumping Lemma . LetLbe an inﬁnite regular language. Then there are
stringsx,y, andz, suchthat y̸=ǫandxynz∈Lforn≥0.
Thepumpinglemmastates that if a languageis regular,thent hereis some string y
that can be “pumped” appropriately. But this doesn’t mean th at if we can pump some
stringy, the language must be regular. Non-regular languages may al so have strings
thatcanbepumped. Thusthelemmaisnotusedforshowingthat alanguage isregular.

Rather it is used for showing that a language isn’tregular, by showing that in some
languagethereisnopossiblestringthatcanbepumpedinthe appropriateway.
Let’s use the pumping lemma to show that the language anbn(i.e., the language
consistingof stringsof as followedby an equal numberof bs) is not regular. We must
show that any possible string sthat we pick cannot be divided up into three parts x,y,
andzsuchthatycanbepumped. Givenarandomstring sfromanbn,wecandistinguish

three ways of breaking sup, and show that no matter which way we pick, we cannot
ﬁndsome ythatcanbepumped:
1.yis composedonlyof as. (Thisimpliesthat xisallastoo,and zcontainsall the

DRAFTSection16.2. HowtoTell ifa LanguageIsn’tRegular 541
bs,perhapsprecededbysome as.) Butif yisallas,thatmeans xynzhasmore as
thanxyz. But this means it has more as thanbs, and so cannot be a member of
thelanguage anbn!
2.yis composed only of bs. The problem here is similar to case 1; If yis allbs,
thatmeans xynzhasmore bsthanxyz,andhencehasmore bsthanas.
3.yis composed of both as andbs (this implies that xis onlyas, whilezis only
bs). This means that xynzmust have some bs beforeas, and again cannot be a
memberofthelanguage anbn!
Thus there is no string in anbnthat can be divided into x,y,zin such a way that y
canbepumped,andhence anbnisnotaregularlanguage.
But while anbnis not a regular language, it is a context-freelanguage. In f act, the
context-freegrammarthatmodels anbnonlytakestwo rules! Heretheyare:
S→a S b
S→ǫ
Here’sa sampleparsetree usingthisgrammartoderivethese ntenceaabb:
S
S
S
a a ǫb b
Figure16.4 Context-free parse treefor aabb.

S→a S b
S→ǫ
S
S
S
a a ǫb b
Thereisalsoapumpinglemmaforcontext-freelanguages,th atcanbeusedwhether
or not a language is context-free; complete discussions can be found in Hopcroft and
Ullman(1979)andParteeetal. (1990).
16.2.2 Are English and Other Natural Languages Regular Lan-
guages?
“How’sbusiness?”Iasked.
“Lousy and terrible.” Fritz grinned richly. “Or I pull off a n ew deal in
the nextmonthorIgoasagigolo,”
“Either...or...,” Icorrected,from forceofprofessionalhabit.
“I’m speaking a lousy English just now,” drawled Fritz, with great self-
satisfaction. “Sallysaysmaybeshe’llgiveme afewlessons .”

guages?
Christopher Isherwood, “Sally Bowles”, from
Goodbye toBerlin . 1935

DRAFT542 Chapter 16. LanguageandComplexity
Consider a formal version of the English language modeled as a set of strings of
words. Isthislanguagearegularlanguage? Itisgenerallya greedthatnaturallanguages
likeEnglish,viewedinthisway,arenotregular,althoughm ostattemptedproofsofthis
arewell-knownto beincorrect.
One kind of argumentthat is often made informallyis that Eng lish number agree-
ment cannot be captured by a regular grammar, because of the p otentially unbounded
distancebetweenthesubjectandtheverbin sentenceslike t hese:
(16.1)Whichproblemdidyourprofessorsay shethought wasunsolvable?
(16.2)Whichproblems didyourprofessorsayshe thought wereunsolvable?
In fact, a simple regular grammar canmodel number agreement, as Pullum and
Gazdar (1982) show. Here’s their regular (right-linear) gr ammar that models these
sentences:
S→Which problemdidyourprofessorsay T
S→Which problemsdidyourprofessorsayU
T→she thoughtT|youthoughtT|wasunsolvable
U→she thoughtU|youthoughtU|wereunsolvable

SoaregulargrammarcouldmodelEnglishagreement. Thisgra mmarisn’telegant,
andwouldhaveahugeexplosioninthenumberofgrammarrules ,butthat’snotrelevant
tothe questionoftheregularityornon-regularityofEngli sh.
Anothercommonﬂawwithpreviouslyattemptedproofs,point edoutbyMohriand
Sproat (1998), is that the fact that a language Lcontains a subset L′at position P′in
the Chomsky hierarchy does not imply that the language Lis also at position P′. For
example, a regular language can contain as a proper subset a c ontext-free language.

Thusthefollowingtwolanguagesarecontext-free
L1={anbn:n∈N} (16.3)
L2={wwR:w∈Σ∗} (16.4)
andyetboth L1andL2are containedinthe regularlanguage L:
L={apbq:p,q∈N} (16.5)
Thus, the fact that a language Lcontains a sublanguage that is very complex says
nothingabouttheoverallcomplexityoflanguage L.
TherearecorrectproofsthatEnglish(orrather“theset ofs tringsofEnglishwords
considered as a formal language”) is not a regular language, based on the pumping
lemma. A proof by Partee et al. (1990), for example, is based o n a famous class of

sentences with center-embedded structures (Yngve, 1960); here is a variant of these center-embedded
Thecat likestunaﬁsh.
Thecat thedogchasedlikestunaﬁsh.
Thecat thedogthe ratbitchasedlikestunaﬁsh.

DRAFTSection16.2. HowtoTell ifa LanguageIsn’tRegular 543
Thecat thedogthe rattheelephantadmiredbitchasedlikest unaﬁsh.
These sentences get harder to understand as they get more com plex. For now,
let’s assume that the grammar of English allows an indeﬁnite number of embeddings.
TheninordertoshowthatEnglishisnotregular,we needtosh owthatlanguageswith
sentencesliketheseareisomorphictosomenon-regularlan guage. Sinceeveryfronted
NPmusthaveitsassociatedverb,thesesentencesareofthefor m:
(the+ noun)n(transitiveverb)n−1likestunaﬁsh.
Theideaof theproofwill beto show thatsentencesofthese st ructurescanbe pro-
ducedbyintersectingEnglishwitharegularexpression. We willthenusethepumping
lemmatoprovethat theresultinglanguageisn’tregular.
In order to build a simple regular expression that we can inte rsect with English to
producethesesentences,wedeﬁneregularexpressionsfort henoungroups( A)andthe
verbs(B):
A={thecat,the dog,therat,theelephant,thekangaroo,... }
B={chased,bit,admired,ate,befriended,... }

verbs(B):
Nowifwetaketheregularexpression /A*B*likes tuna fish/ andinter-
sect it withEnglish(consideredasaset ofstrings),theres ultinglanguageis:
L=xnyn−1likestunaﬁsh ,x∈A,y∈B
This language L can be shown to be non-regularvia the pumping lemma (see Ex-
ercise 2). Since the intersection of English with a regular l anguage is not a regular
language,Englishcannot be a regularlanguageeither (sinc e the regularlanguagesare
closedunderintersection).
Thereisawell-knownﬂaw,oratleastanoverlystrongassump tionwiththisproof,

which is the assumption that these structures can be nested i ndeﬁnitely. Sentences of
English are clearly bounded by some ﬁnite length; perhaps we can safely say that all
sentencesof Englishare lessthan a billionwordslong. If th e set of sentencesis ﬁnite,
then all natural languages are clearly ﬁnite-state. This is a ﬂaw with all such proofs
about the formal complexity of natural language. We will ign ore this objection for
now, since conveniently imagining that English has an inﬁni te number of sentences

canproveenlighteningin understandingthepropertiesofﬁ niteEnglish.
A more worrisome potential ﬂaw with this proof is that it depe ndson the assump-
tionthatthesedoublerelativizationsofobjectsarestric tlygrammatical(evenifhardto
process). The research of Karlsson (2007) suggests that, wh ile some kinds of center-
embeddings are grammatical, these double relativizations of objects are in fact un-
grammatical. Inanycase,sentenceslikethisgethardmuchf asterthanabillionwords,

and are difﬁcult to understand after a couple nestings. We wi ll return to this issue in
Sec.16.4.

DRAFT544 Chapter 16. LanguageandComplexity
16.3 IsNaturalLanguageContext-Free?
The previoussection argued that English (considered as a se t of strings) doesn’t seem
likearegularlanguage. Thenaturalnextquestiontoaskisw hetherEnglishisacontext-
freelanguage. ThisquestionwasﬁrstaskedbyChomsky(1956 ),andhasaninteresting
history; a number of well-known attempts to prove English an d other languages non-
context-freehave been published, and all except two have be en disprovedafter publi-
cation. Oneofthesetwocorrect(oratleast not-yetdisprov ed)argumentsderivesfrom
thesyntaxofadialectofSwissGerman;theotherfromthemor phologyofBambara,a
NorthwesternMandelanguagespokeninMaliandneighboring countries(Culy,1985).
The interested reader should see Pullum (1991, pp. 131–146) for an extremely witty
historyofboththeincorrectandcorrectproofs;thissecti onwillmerelysummarizeone
ofthecorrectproofs,theonebasedonSwiss German.
Bothofthecorrectarguments,andmostoftheincorrectones ,makeuseofthefact

thatthefollowinglanguages,andonesthathavesimilarpro perties,arenotcontext-free:
{xx|x∈{a,b}∗} (16.6)
Thislanguageconsistsofsentencescontainingtwoidentic alstringsconcatenated. The
followingrelatedlanguageisalsonotcontext-free:
anbmcndm(16.7)
Thenon-context-freenatureofsuchlanguagescanbeshownu singthepumpinglemma
forcontext-freelanguages.
The attempts to prove that the natural languages are not a sub set of the context-
free languages do this by showing that natural languages hav e a property of these xx

languages called cross-serial dependencies . In a cross-serial dependency, words orcross-serialdependencies
largerstructuresarerelatedinleft-to-rightorderassho wninFig.16.5. Alanguagethat
hasarbitrarilylongcross-serialdependenciescanbemapp edtothexxlanguages.
x1...x2xn......1y2yny
Figure16.5 A schematic of a cross-serial dependency.
The successful proof, independently proposed by Huybregts (1984) and Shieber
(1985a) (as we might expect from the prevalence of multiple d iscovery in science;

see page 13) shows that a dialect of Swiss German spoken in Z¨ u rich has cross-serial
constraints which make certain parts of that language equiv alent to the non-context-
freelanguage anbmcndm. Theintuitionisthat SwissGermanallowsasentencetohave
a string of dative nounsfollowed by a string of accusative no uns, followedby a string
ofdative-takingverbs,followedbya stringofaccusative- takingverbs.

DRAFTSection16.3. IsNaturalLanguageContext-Free? 545
We will follow the version of the proof presented in Shieber ( 1985a). First, he
notesthatSwissGermanallowsverbsandtheirargumentstob eorderedcross-serially.
Assumethat all theexampleclauseswe presentbelowareprec ededbythestring “Jan
s¨ait das”(“Jansaysthat”):
(16.8)...mer
...weemHans
Hans/DATes
thehuus
house/ACCh¨alfed
helpedaastriiche.
paint.
“...we helpedHanspaintthehouse.”
Notice the cross-serial nature of the semantic dependency: both nouns precede
both verbs, and em Hans (Hans) is the argumentof h¨alfed(helped)while es huus(the
house) is the argument of aastriiche (paint). Furthermore, there is a cross-serial case
dependencybetween the nouns and verbs; h¨alfed(helped) requiresthe dative, and em
Hansisdative,while aastriiche (paint)takestheaccusative,and eshuus(thehouse)is
accusative.
Shieber points out that this case marking can occur even acro ss triply embedded
cross-serialclauseslike thefollowing:
(16.9)...mer
...wed’chind
thechildren/ACCem Hans

thehuus
house/ACChaend
havewele
wantedtolaa
let
h¨alfe
helpaastriiche.
paint.
“...we havewantedtolet thechildrenhelpHanspaintthehou se.”
Shiebernotesthatamongsuchsentences,thosewithalldati veNPsprecedingallac-
cusativeNPs,andalldative-subcategorizingV’sprecedin gallaccusative-subcategorizing
V’sareacceptable.
(16.10)Jans¨ ait dasmer(d’chind)∗(emHans)∗eshuushaendwele laa∗h¨ alfe∗
aastriche.
Let’s call the regular expression above R. Since it’s a regul ar expression (you see

thehuus
havewele
let
h¨alfe
paint.
it only has concatenation and Kleene stars) it must deﬁne a re gular language, and so
we can intersect R with Swiss German, and if the result is cont ext-free, so is Swiss
German.
But it turns out that Swiss German requires that the number of verbs requiring
dative objects ( h¨alfe) must equal the number of dative NPs ( em Hans) and similarly
for accusatives. Furthermore,an arbitrarynumber of verbs can occur in a subordinate

German.
clause of this type (subject to performance constraints). T his means that the result of
intersectingthisregularlanguagewithSwiss Germanisthe followinglanguage:
(16.11)L=Jans¨ aitdasmer(d’chind)n(emHans)meshuushaendwele(laa)n(h¨ alfe)m
aastriiche.
Butthislanguageis ofthe form wanbmxcndmy,whichisnotcontext-free!
Sowe canconcludethatSwiss Germanisnotcontext-free.

DRAFT546 Chapter 16. LanguageandComplexity
16.4 ComplexityandHumanProcessing
We noted in passing earlier that many of the sentences that we re used to argue for
the non-ﬁnite-statenature of English (like the “center-em bedded”sentences) are quite
difﬁcult to understand. If you are a speaker of Swiss German ( or if you have a friend
who is), you will notice that the long cross-serial sentence s in Swiss German are also
ratherdifﬁculttofollow. Indeed,asPullumandGazdar(198 2)pointout,
precisely those construction-types that ﬁgure in the vario us proofs that
Englishisnotcontext-freeappeartocausemassivedifﬁcul tyinthehuman
processingsystem...
This bringsus to a second use of the term complexity . In the previoussection we
talked about the complexityof a language. Here we turn to a qu estion that is as much
psychological as computational: the complexity of an indiv idual sentence. Why are
certain sentences hard to comprehend? Can this tell us anyth ing about computational
processes?

Many things can make a sentence hard to understand. For examp le we saw in
Ch. 14 that a word is read more slowly if it is unpredictable; i .e., has a low N-gram
probability or a low parse probability. We also saw in Ch. 14 garden-path sentences
where ambiguity can cause difﬁculty; if there are multiple p ossible parses, a human
reader (or listener) sometimes chooses the incorrect parse , leading to a double-take
when switching back to the other parse. Other factors that af fect sentence difﬁculty

includeimplausiblemeaningsandbadhandwriting.
Anotherkindofdifﬁcultyseemsto berelatedtohumanmemory limitations,andit
isthisparticularkindofcomplexity(oftencalled “lingui sticcomplexity”or“syntactic
complexity”)thatbearsaninterestingrelationtotheform al-languagecomplexityfrom
theprevioussection.
Consider these sentences from Gibson (1998) that cause difﬁ culties when people
try to read them (we will use the # to mean that a sentence cause s extreme processing

difﬁculty). In each case the (ii) example is signiﬁcantly mo re complex than the (i)
example:
(16.12) (i) Thecat likestunaﬁsh.
(ii) #Thecatthedogtheratthegoatlickedbitchasedlikest unaﬁsh.
(16.13) (i) Thechilddamagedthe pictureswhichweretakenbythe pho tographer
whotheprofessormetat theparty.
(ii) #Thepictureswhichthephotographerwhotheprofessor metat theparty
tookweredamagedbythechild.
(16.14) (i) Thefactthat theemployeewhothe managerhiredstoleofﬁ cesupplies
worriedthe executive.
(ii) #Theexecutivewhothe factthattheemployeestole ofﬁc esupplies

example:
worriedhiredthemanager.
The earliest work on sentences of this type noticed that they all exhibit nestingor
center-embedding (Chomsky, 1957; Yngve, 1960; Chomsky and Miller, 1963; Mill er
and Chomsky, 1963). That is, they all contain exampleswhere a syntactic categoryA
isnestedwithinanothercategoryB, andsurroundedbyother words(XandY):

DRAFTSection16.4. ComplexityandHumanProcessing 547
[BX[A]Y]
In each of the examples above, part (i) has zero or one embeddi ng, while part (ii)
has two or more embeddings. For example in (16.12ii)above, t here are three reduced
relativeclausesembeddedinsideeachother:
(16.15)#[SThecat [ S′thedog[ S′therat[ S′theelephantadmired]bit]chased]likes
tunaﬁsh].
In (16.13ii), the relative clause who the professor met at the party is nested in be-
tweenthe photographer andtook. The relativeclause which the photographer...took
isthennestedbetween Thepictures andwere damagedbythechild.
(16.16)#Thepictures[whichthephotographer[ whothe professorme t attheparty]
took]weredamagedbythechild.
Thedifﬁcultywiththesenestedstructuresisnotcausedbyu ngrammaticality,since
thestructuresthatareusedinthecomplexsentencesin(16. 12ii)–(16.14ii)arethesame
ones used in the easier sentences (16.12i)–(16.14i). The di fference between the easy
and complex sentences seems to relate to the number of embedd ings. But there is no

natural way to write a grammar that allows Nembeddingsbut not N+1 embeddings.
Rather,thecomplexityofthesesentencesseemstobeaproce ssingphenomenon;some
factaboutthehumanparsingmechanismisunabletodealwith thesekindsofmultiple
nestings, in Englishand in other languages(Cowper,1976;B abyonyshevand Gibson,
1999).
Thedifﬁcultyofthesesentencesseemstohavesomethingtod owithmemorylimi-
tations. Earlyformalgrammarianssuggestedthatthismighthaveso methingtodowith
how the parser processed embeddings. For example Yngve (196 0) suggested that the

1999).
human parser is based on a limited-size stack, and that the mo re incomplete phrase-
structure rules the parser needs to store on the stack, the mo re complex the sentence.
Miller and Chomsky (1963) hypothesized that self-embedded structures are particu- self-embedded
larlydifﬁcult. Aself-embeddedstructurecontainsasynta cticcategory Anestedwithin
anotherexampleof A, andsurroundedbyotherwords( xandybelow);suchstructures
mightbedifﬁcultbecauseastack-basedparsermightconfus etwocopiesoftheruleon
thestack.
A
xAy

thestack.
A
xAy
The intuitions of these early models are important, althoug h we no longer believe
that the complexity problems have to do with an actual stack. For example, we now
knowthattherearecomplexitydifferencesbetweensentenc esthathavethesamenum-
ber of embeddings,such as the well-knowndifferencebetwee nsubject-extractedrela-
tiveclauses(16.17ii)andobject-extractedrelativeclau ses(16.17i):
(16.17) (i) [SThereporter[ S′who[Sthe senatorattacked]]admittedtheerror].
(ii) [SThereporter[ S′who[Sattackedthesenator]]admittedtheerror].

Theobject-extractedrelativeclauses are more difﬁcultto process, asmeasuredfor
exampleby the amountof time it takes to read them, and other f actors (MacWhinney,
1977, 1982; MacWhinney and Csaba Pl´ eh, 1988; Ford, 1983; Wa nner and Maratsos,
1978; King and Just, 1991; Gibson, 1998). Indeed, Karlsson ( 2007) has shown in a

DRAFT548 Chapter 16. LanguageandComplexity
study of seven languages that the grammaticality of center e mbeddings depends a lot
ontheparticularsyntacticstructure(e.g.,relativeclau sesversusdoublerelativizationof
objects) being embedded. Another problem for the old-fashi onedstack-based models
isthefactthatdiscoursefactorscanmakesomedoublyneste drelativeclauseseasierto
process,suchasthe followingdoublenestedexample:
(16.18)Thepictures[that thephotographer[ whoImetat theparty] t ook] turned
outverywell.
Whatseemstomakethisstructurelesscomplexisthatoneoft heembeddedNPsis
the word I; pronounslike Iandyouseem tobe easier toprocess,perhapsbecausethey
donotintroducea newentityto thediscourse.
One human parsing model that accounts for all of this data is t he Dependency
LocalityTheory(Gibson,1998,2003). TheintuitionoftheD LTisthatobjectrelatives
aredifﬁcultbecausetheyhavetwonounsthatappearbeforea nyverb. Thereadermust
holdontothese twonounswithoutknowinghowtheywill ﬁt int othesentences.

More speciﬁcally, the DLT proposes that the processing cost of integrating a new
wordwis proportional to the distance between wand the syntactic item with which
wis being integrated. Distance is measured not just in words, but in how many new
phrases or discourse referents have to be held in memory at th e same time. Thus the
memory load for a word is higher if there have been many interv eningnew discourse
referentssince the word has been predicted. Thus the DLT predictsthat a sequenceof

NPs can bemade easier to processif oneof themis a pronountha t isalreadyactive in
thediscourse,explaining(16.18).
In summary, the complexity of these ‘center-embedded’ and o ther examples does
seem to be related to memory, although not in as direct a link t o parsing stack size as
was ﬁrst thought40 yearsago. Understandingthe relationsh ipbetweenthese memory
factors and the statistical parsing factors mentioned in Ch . 14 is an exciting research
areathatisjust beginningtobeinvestigated.
16.5 Summary

Thischapterintroducedtwo differentideasof complexity : thecomplexityofa formal
language,andthecomplexityofa humansentence.
•Grammarscan be characterized by their generative power . One grammar is of
greater generative power or complexity than another if it can deﬁne a language
that the other cannot deﬁne. The Chomsky hierarchy is a hierarchy of gram-
mars based on their generative power. It includes Turing equivalent ,context-
sensitive,context-free ,andregulargrammars.

•Thepumping lemma canbeusedtoprovethatagivenlanguageis notregular .
English is not a regular language, although the kinds of sent ences that make
English non-regularare exactly those that are hard for peop le to parse. Despite
manydecadesof attemptsto provethe contrary,Englishdoes ,however,seem to
beacontext-freelanguage. ThesyntaxofSwiss-Germanandt hemorphologyof

DRAFTSection16.5. Summary 549
Bambara, by contrast, are not context-free and seem to requi re mildly context-
sensitivegrammars.
•Certaincenter-embedded sentences are hard for people to parse. Many theo-
ries agree that this difﬁculty is somehow caused by memory limitations of the
humanparser.
Chomsky (1956) ﬁrst asked whether ﬁnite-state automata or c ontext-free grammars
were sufﬁcient to capture the syntax of English. His suggest ion in that paper that
English syntax contained “examples that are not easily expl ained in terms of phrase
structure”wasa motivationforhisdevelopmentofsyntacti ctransformations.
Choksky’s proof was based on the language {xxR:x∈{a,b}∗}.xRmeans “the
reverseof x”,soeachsentenceofthislanguageconsistsofastringof asandbsfollowed
bythereverseor“mirrorimage”ofthestring. Thislanguage isnotregular;Parteeetal.
(1990)showsthisbyintersectingit with theregularlangua geaa∗bbaa∗. The resulting

languageis anb2an; it isleft asan exerciseforthe reader(Exercise3)toshow t hatthis
isnotregularbythe pumpinglemma.
Chomsky proof shows that English had mirror-like propertie s, relying on multi-
ple embeddings of the following English syntactic structur es, where S1,S2,...,Snare
declarativesentencesinEnglish,
•IfS1,thenS2
•EitherS3,orS4
•Themanwhosaid S5is arrivingtoday
SeeChomsky(1956)fordetails.
Pullum(1991,pp. 131–146)isthedeﬁnitivehistoricalstud yofresearchonthenon-
context-free-ness of natural language. The early history o f attempts to prove natural

languagesnon-context-freeissummarizedinPullumandGaz dar(1982). Thepumping
lemma was originally presented by Bar-Hillel et al. (1961), who also offer a number
of important proofs about the closure and decidability prop erties of ﬁnite-state and
context-freelanguages. Furtherdetails, includingthe pu mpinglemmaforcontext-free
languages(alsodueto Bar-Hillel et al.(1961))can befound ina textbookin automata
theorysuchasHopcroftandUllman(1979).
Yngve’s idea that the difﬁculty of center-embedded sentenc es could be explained

if the human parser was ﬁnite-state was taken up by Church (19 80) in his master’s
thesis. Heshowedthataﬁnite-stateparserthatimplements thisideacouldalsoexplain
a number of other grammatical and psycholinguistic phenome na. While the cognitive
modeling ﬁeld has turned toward more sophisticated models o f complexity, Church’s
work can be seen as the beginning of the return to ﬁnite-state models in speech and
languageprocessingthat characterizedthe1980sand1990s .
There are a number of other ways of looking at complexity that we didn’t have

space to go into here. One is whether language processing is N P-complete. NP-

DRAFT550 Chapter 16. LanguageandComplexity
complete is the name of a class of problems which are suspected to be par ticularly NP-complete
difﬁcult to process. Barton et al. (1987) prove a number of co mplexity results about
theNP-completenessofnaturallanguagerecognitionandpa rsing. Amongotherthings,
theyshowedthat
1. Maintaininglexicalandagreementfeatureambiguitieso verapotentiallyinﬁnite-
lengthsentencecausestheproblemofrecognizingsentence sinsomeuniﬁcation-
basedformalismslikeLexical-FunctionalGrammarto beNP- complete.
2. Two-levelmorphologicalparsing(orevenjustmappingbe tweenlexicalandsur-
faceform)isalsoNP-complete.
Recentworkhasalsobeguntolinkprocessingcomplexitywit hinformation-theoretic
measureslikeKolmogorovcomplexity(Juola,1999).
Finally,recentworkhaslookedattheexpressivepowerofdi fferentkindsofproba-
bilisticgrammars,showingforexamplethatweightedconte xt-freegrammars(inwhich
each rule hasa weight) andprobabilisticcontext-freegram mars(in which the weights

oftherulesforanon-terminalmust sumto1) areequallyexpr essive(SmithandJohn-
son,2007;Abneyetal.,1999a;Chi, 1999).
Exercises
16.1Isthelanguage anb2ancontext-free?
16.2Usethe pumpinglemmato showthislanguageisnotregular:
16.3Parteeetal.(1990)showedthatthelanguage xxR,x∈a,b∗isnotregular,byinter-
sectingitwiththeregularlanguage aa∗bbaa∗. Theresultinglanguageis anb2an.
Usethepumpinglemmatoshowthatthislanguageisnotregula r,completingthe
proofthat xxR,x∈a,b∗isnotregular.
16.4Buildacontext-freegrammarforthelanguage
L={xxR|x∈a,b∗}

DRAFTSpeech and Language Processing: An Introduction to Natural Language Processing, Computational
Linguistics and Speech Recognition: Second Edition, Danie l Jurafsky & James H. Martin. Copyright
c⃝2007, All rights reserved. Draft of January 9, 2008. Do not ci te.
Chapter17
Representing Meaning
ISHMAEL:Surely all this isnot without meaning.
HermanMelville, MobyDick
The approach to semantics that is introduced here, and elabo rated on in the next four
chapters, is based on the notion that the meaning of linguist ic expressions can be
captured in formal structures, which we will call meaning representations . Corre-Meaningrepresentations
spondingly,the frameworksthat are used to specify the synt ax and semantics of these
representations will be called meaning representation languages . These meaningMeaningrepresentation
languagesrepresentations play a role analogous to that of the phonolo gical, morphological, and
syntacticrepresentationsintroducedinearlierchapters .

Theneedformeaningrepresentationsariseswhenneitherth erawlinguisticinputs,
noranyofthestructuresderivablefromthembyanyofthetra nsducerswehavestudied
thus far, facilitate the kind of semantic processing that is required. More speciﬁcally,
what weneedare representationsthatbridgethegapfromlin guistic inputstothe non-
linguistic knowledge of the world needed to perform tasks in volving the meaning of
linguistic inputs. To illustrate this notion, consider the following everyday language
tasksthatrequiresomeformofsemanticprocessingofnatur allanguage:

•Answeringessayquestionsonan exam
•Decidingwhattoorderat arestaurantbyreadinga menu
•Learningtouse anewpieceofsoftwarebyreadingthemanual
•Realizingthatyou’vebeeninsulted
•Followingrecipes
Simplyhavingaccesstothephonological,morphological,a ndsyntacticrepresentations
that we have discussed thus far will not get us very far on acco mplishingany of these
tasks. Rather, they require access to representations that link the linguistic elements
involvedinthetasktothenon-linguistic knowledgeoftheworld neededtosuccessfully

accomplish them. For example, some of the world knowledge ne eded to perform the
abovetaskswouldincludethefollowing:
•Answering and grading essay questions requires background knowledge about
the topic of the question, the desired knowledge level of the students, and how
suchquestionsare normally answered.
•Reading a menu and deciding what to order, giving advice abou t where to go
to dinner, following a recipe, and generating new recipes al l require knowledge

aboutfood,itspreparation,whatpeoplelike toeatandwhat restaurantsarelike.
•Learningto use a piece of softwareby readinga manual,or giv ingadviceabout
how to do the same, requires knowledge about current compute rs, the speciﬁc

DRAFT552 Chapter 17. RepresentingMeaning
∃e,yHaving (e)∧Haver (e,Speaker )∧HadThing (e,y)∧Car(y)
Having
Haver Had-Thing
Speaker Car
Car Having
⇑POSS-BY Haver: Speaker
Speaker HadThing: Car
Figure17.1 Alistofsymbols,twodirectedgraphs,andarecordstructur e: asamplerofmean-
ingrepresentations for I have acar.
software in question, similar software applications, and k nowledge about users
ingeneral.
In the representationalapproach presented here and elabor ated on in the next four
chapters, we assume that linguistic expressions have meani ng representations that are
made up of the same kind of stuff that is used to represent this kind of everyday com-
monsenseknowledgeof the world. Theprocesswherebysuch re presentationsarecre-
atedandassignedto linguisticinputsiscalled semantic analysis . Semantic analysis
Tomakethesenotionsabitmoreconcrete,considerFig.17.1 ,whichshowssample
meaningrepresentationsfor the sentence I have a car using fourrepresentativemean-

ingrepresentationlanguages. Theﬁrstrowillustratesase ntencein First-OrderLogic ,
whichwillbecoveredindetailinSec.17.3;thegraphinthec enterisanexampleof Se-
mantic Network , which will be discussed furtherin Sec. 17.5; the third row c ontains
aConceptual Dependency diagram, discussed in more detail in Ch. 19, and ﬁnally a
Frame-Based representation,discussedinSec.17.5andagaininCh. 22.
Whiletherearenon-trivialdifferencesamongtheseapproa ches,atanabstractlevel
theyallshareasacommonfoundationthenotionthatameanin grepresentationconsists

of structures composed from a set of symbols, or representat ional vocabulary. When
appropriatelyarranged,thesesymbolstructuresaretaken tocorrespond to theobjects,
properties of objects and relations among objects in some st ate of affairs being repre-
sented. Inthiscase, all fourrepresentationsmakeuseof sy mbolscorrespondingto the
speaker,a car,andrelationsdenotingthepossessionofone bytheother.
It is important to note that these representations can be vie wed from at least two

distinct perspectivesin all fourof these approaches: as re presentationsof themeaning
of the particular linguistic input I have a car , and as representations of the state of

DRAFTSection17.1. ComputationalDesiderataforRepresentatio ns 553
affairsin some world. It is thisdualperspectivethat allow stheserepresentationsto be
usedtolinklinguisticinputstotheworldandtoourknowled geofit.
The structure of this part of the book parallels that of the pr evious parts. We will
alternate discussions of the nature of meaning representat ions with discussions of the
computational processes that can produce them. More speciﬁ cally, this chapter intro-
ducesthebasicsofwhatisneededinameaningrepresentatio n,whileCh.18introduces
a numberof techniquesfor assigning meaningsto linguistic inputs. Ch. 19 exploresa
rangeofcomplexrepresentationalissuesrelatedtothemea ningsofwords. Ch.20then
explores some robust computational methods designed to exp loit these lexical repre-
sentations.
Since the focus of this chapter is on some of the basic require ments for meaning
representations,wewilldeferanumberofextremelyimport antissuestolaterchapters.

In particular, the focus of this chapter is on representing w hat is sometimes called
theliteral meaning of sentences. By this, we have in mind representations that a re Literal meaning
closely tied to the conventional meanings of the words that a re used to create them,
and that do not reﬂect much of the context in which they occur. The shortcomings
of such representations with respect to phenomena such as id ioms and metaphor will
be discussed in Ch. 19; the task of producing representation s for larger stretches of

discoursewill bediscussedin Ch.21.
There are ﬁve major parts to this chapter. Section 17.1 explo res some of the key
computational requirements for what we need in a meaning rep resentation language.
Section17.2discusseshowwecanprovidesomeguaranteesth atthese representations
will actually do what we need them to do — provide a correspond anceto the state of
affairs being represented. Section 17.3 then provides an in troduction to First Order
Logic, which has historically been the primary technique us ed to investigate issues

in natural language semantics. Section 17.4 then describes howFOLcan be used to
capturethesemanticsofeventsandstatesinEnglish.
17.1 ComputationalDesiderataforRepresentations
Webeginbyconsideringtheissueofwhymeaningrepresentat ionsareneededandwhat
theyshoulddoforus. Tofocusthisdiscussion,wewillconsi derinmoredetailthetask
of giving advice about restaurants to tourists. In this disc ussion, we will assume that
we have a computer system that accepts spoken language queri es from tourists and

constructappropriateresponsesby usinga knowledgebase o frelevantdomainknowl-
edge. A seriesof exampleswill serveto introducesome ofthe basicrequirementsthat
a meaning representation must fulﬁll, and some of the compli cations that inevitably
arise in the process of designing such meaning representati ons. In each of these ex-
amples, we will examinethe role that the representationof t he meaningof the request
mustplayinthe processofsatisfyingit.
17.1.1 Veriﬁability
Letusbeginbyconsideringthefollowingsimplequestion:

DRAFT554 Chapter 17. RepresentingMeaning
(17.1)DoesMaharaniservevegetarianfood?
This example illustrates the most basic requirement for a me aning representation: it
must be possible to use the representation to determine the r elationship between the
meaning of a sentence and the world as we know it. In other word s, we need to be
able to determine the truth of our representations. Section 17.2 explores this topic in
some detail. For now let’s assume that we can give computatio nalsystems the ability
to compare, or match, the representation of the meaning of a expressions against the
representationsin a knowledgebase , itsstoreofinformationaboutitsworld. Knowledge base
In this example, let us assume that the meaning of this questi on contains, as a
component,themeaningunderlyingthe proposition Maharaniserves vegetarianfood .
Fornow,wewill simplyglossthisrepresentationas:
Serves (Maharani ,VegetarianFood ) (17.2)
Itisthisrepresentationoftheinputthatwillbematchedag ainsttheknowledgebaseof

factsabouta set ofrestaurants. If the system ﬁndsa represe ntationmatchingthe input
proposition in its knowledge base, it can return an afﬁrmati ve answer. Otherwise, it
musteithersay No,ifitsknowledgeoflocalrestaurantsiscomplete,orsayth atitdoes
notknowifthereisreasontobelievethat itsknowledgeisin complete.
This notion is known as veriﬁability , and concerns a system’s ability to compare Veriﬁability
thestateofaffairsdescribedbyarepresentationtothesta teofaffairsinsomeworldas
modeledina knowledgebase.

17.1.2 Unambiguous Representations
Thedomainofsemantics,like allthe otherdomainswehavest udied,is subjecttoam-
biguity. Speciﬁcally, individual linguistic expressions can legitimately have different
meaning representations assigned to them based on the circu mstances in which they
occur. Considerthefollowingexamplefromthe BERPcorpus:
(17.3)Iwannaeat someplacethat’scloseto ICSI.
Giventheallowableargumentstructuresfortheverb eat,thissentencecaneithermean

that the speaker wants to eat atsome nearby location, or under a Godzilla as speaker
interpretation, the speaker may want to devour some nearby l ocation. The answer
generatedby the system forthis requestwill dependon which interpretationis chosen
asthecorrectone.
Sinceambiguitiessuchasthisaboundinallgenresofalllan guages,somemeansof
determiningthat certain interpretationsare preferable( or alternativelyless preferable)
thanothersisneeded. Thevariouslinguisticphenomenatha tgiverisetosuchambigu-

ities, and the techniques that can be employed to deal with th em, will be discussed in
detailin thenextfourchapters.
Our concern in this chapter, however, is with the status of ou r meaning represen-
tationswith respect to ambiguity,and not with the meansby w hich we might arriveat
correct interpretations. Since we reason about, and act upo n, the semantic content of
linguisticinputs,theﬁnalrepresentationofaninput’sme aningshouldbefreefromany

ambiguity. Therefore, regardlessof any ambiguity in the ra w input, it is critical that a

DRAFTSection17.1. ComputationalDesiderataforRepresentatio ns 555
meaningrepresentationlanguagesupportrepresentations thathaveasingleunambigu-
ousinterpretation1.
A concept closely related to ambiguity is vagueness . Like ambiguity, vagueness Vagueness
canmakeit difﬁculttodeterminewhattodowitha particular inputbasedonitsmean-
ingrepresentation. Vagueness,however,doesnotgiverise tomultiplerepresentations.
Considerthefollowingrequestasanexample:
(17.4)Iwant toeat Italianfood.
While the use ofthe phrase Italianfood mayprovideenoughinformationfora restau-
rantadvisortoprovidereasonablerecommendations,itisn everthelessquite vagueasto
what the user really wants to eat. Therefore,a vague represe ntationof the meaningof
thisphrasemaybeappropriateforsomepurposes,whileamor especiﬁcrepresentation
may be needed for other purposes. It will, therefore, be adva ntageous for a mean-
ing representation language to support representations th at maintain a certain level of

vagueness. Note that it is not always easy to distinguish amb iguity from vagueness.
ZwickyandSadock(1975)provideausefulsetofteststhatca nbeusedasdiagnostics.
17.1.3 CanonicalForm
Thenotionthatsinglesentencescanbeassignedmultipleme aningsleadstotherelated
phenomenon of distinct inputs that should be assigned the sa me meaning representa-
tion. Considerthefollowingalternativewaysofexpressin g(17.1):
(17.5)DoesMaharanihavevegetariandishes?
(17.6)Dotheyhavevegetarianfoodat Maharani?
(17.7)Arevegetariandishesservedat Maharani?

(17.8)DoesMaharaniservevegetarianfare?
Given that these alternatives use different words and have w idely varying syntac-
tic analyses, it would not be unreasonable to expect them to h ave substantially dif-
ferent meaning representations. Such a situation would, ho wever, have undesirable
consequences for how we determine the truth of our represent ations. If the system’s
knowledge base contains only a single representation of the fact in question, then the
representationsunderlyingall but one of our alternatives will fail to producea match.

We could, of course, store all possible alternative represe ntations of the same fact in
theknowledgebase,butthiswouldleadtoanenormousnumber ofproblemsrelatedto
keepingsuchaknowledgebase consistent.
The way out of this dilemma is motivated by the fact that since the answers given
foreachofthesealternativesshouldbethesameinallsitua tions,wemightsaythatthey
allmeanthesamething,atleastforthepurposesofgivingre staurantrecommendations.
Inotherwords,atleastinthisdomain,wecanlegitimatelyc onsiderassigningthesame

meaning representation to the propositions underlying eac h of these requests. Tak-
ing such an approach would guarantee that our simple scheme f or answering Yes/No
questionswill still work.
1This does not preclude the use of intermediate semantic repr esentations that maintain some level of am-
biguity on the way to a single unambiguous form. Examples of s uch representations will be discussed in
Ch. 18.

DRAFT556 Chapter 17. RepresentingMeaning
The notion that inputs that mean the same thing should have th e same meaning
representation is known as the doctrine of canonical form . This approach greatly Canonical form
simpliﬁes variousreasoningtasks since systems need only d eal with a single meaning
representationforapotentiallywiderangeofexpressions .
Canonical form does, of course, complicate the task of seman tic analysis. To see
this,notethatthealternativesgivenaboveusecompletely differentwordsandsyntaxto
refer to vegetarianfare and to what restaurantsdo with it. M ore speciﬁcally, to assign
the same representation to all of these requests our system w ill have to conclude that
vegetarian fare ,vegetarian dishes andvegetarian food refer to the same thing in this
context, that the use here of havingandservingare similarly equivalent, and that the
different syntactic parses underlying these requests are a ll compatible with the same
meaningrepresentation.

Being able to assign the same representation to such diverse inputs is a tall order.
Fortunately there are some systematic meaning relationshi ps among word senses and
among grammatical constructions that can be exploited to ma ke this task tractable.
Considerthe issueofthemeaningsofthewords food,dishandfareintheseexamples.
A little introspection, or a glance at a dictionary, reveals that these words have a fair
number of distinct uses. However, it also reveals that there is at least one sense that

is shared among them all. If a system has the ability to choose that shared sense, then
an identical meaning representation can be assigned to the p hrases containing these
words.
In general,we say that these wordsall have various word senses and that some of Word senses
the senses are synonymouswith one another. The process of ch oosing the right sense
in context is called word sense disambiguation , or word sense tagging by analogyWord sensedisambiguation

words.
to part-of-speechtagging. The topics of synonymy,sense ta gging, and a host of other
topicsrelatedtowordmeaningswillbecoveredinChapters1 9and20. Sufﬁceittosay
herethatthefactthatinputsmayusedifferentwordsdoesno tprecludetheassignment
ofidenticalmeaningstothem.
Just as there are systematic relationships among the meanin gs of different words,
there are similar relationships related to the role that syn tactic analyses play in as-
signing meanings to sentences. Speciﬁcally, alternative s yntactic analyses often have

meaningsthat are, if not identical, at least systematicall y related to one another. Con-
siderthefollowingpairofexamples:
(17.9)Maharaniservesvegetariandishes.
(17.10)Vegetariandishesare servedbyMaharani.
Despite the different placement of the arguments to servein these examples, we can
still assign Maharani andvegetarian dishes to the same roles in both of these exam-
plesbecauseofourknowledgeoftherelationshipbetweenac tiveandpassivesentence
constructions. Inparticular,we canuse knowledgeofwhere grammaticalsubjectsand

directobjectsappearintheseconstructionstoassign Maharani ,totheroleoftheserver,
andvegetarian dishes to the role of thing being served in both of these examples, de -
spite the fact that they appear in different surface locatio ns. The precise role of the
grammarin theconstructionofmeaningrepresentationswil l becoveredin Ch.18.

DRAFTSection17.1. ComputationalDesiderataforRepresentatio ns 557
17.1.4 Inference andVariables
Continuingwith the topic of the computationalpurposestha t meaningrepresentations
shouldserve,weshouldconsidermorecomplexrequestssuch asthefollowing:
(17.11)Canvegetarianseat at Maharani?
Here, it would be a mistake to invoke canonical form to force o ur system to assign
the same representation to this request as for the previous e xamples. The fact that
this request results in the same answer as the others arises n ot because they mean the
samething,butbecausethereisa commonsenseconnectionbe tweenwhatvegetarians
eat and what vegetarian restaurants serve. This is a fact abo ut the world and not a
fact about any particular kind of linguistic regularity. Th is implies that no approach
based on canonical form and simple matching will give us an ap propriate answer to
thisrequest. Whatisneededisasystematicwaytoconnectth emeaningrepresentation

of this request with the facts about the world as they are repr esented in a knowledge
base.
We will use the term inference to refer generically to a system’s ability to draw Inference
valid conclusionsbased on the meaning representationof in puts and its store of back-
ground knowledge. It must be possible for the system to draw c onclusions about the
truth of propositionsthat are not explicitly represented i n the knowledgebase, but are
neverthelesslogicallyderivablefromthepropositionsth at arepresent.

base.
Nowconsiderthefollowingsomewhatmorecomplexrequest:
(17.12)I’dliketo ﬁndarestaurantwhereIcanget vegetarianfood.
Unlike our previous examples, this request does not make ref erence to any particular
restaurant. The user is stating that they would like informa tionaboutan unknownand
unnamedentitythat is a restaurantthatservesvegetarianf ood. Since thisrequestdoes
notmentionanyparticularrestaurant,the kindofsimple ma tching-basedapproachwe

have been advocating is not going to work. Rather, answering this request requires
a more complex kind of matching that involves the use of varia bles. We can gloss a
representationcontainingsuchvariablesasfollows:
Serves (x,VegetarianFood ) (17.13)
Matching such a proposition succeeds only if the variable xcan be replaced by some
knownobjectintheknowledgebaseinsuchawaythattheentir epropositionwillthen
match. The concept that is substituted for the variable can t hen be used to fulﬁll the

user’s request. Of course, this simple example only hints at the issues involved in the
use of such variables. Sufﬁce it to say that linguistic input scontain manyinstances of
allkindsofindeﬁnitereferencesanditisthereforecritic alforanymeaningrepresenta-
tionlanguageto beabletohandlethiskindofexpression.
17.1.5 Expressiveness
Finally, to be useful a meaning representation scheme must b e expressive enough to
handleanextremelywiderangeofsubjectmatter. Theideals ituation,ofcourse,would

be to have a single meaning representation language that cou ld adequately represent
the meaningof anysensible natural languageutterance. Alt houghthis is probablytoo

DRAFT558 Chapter 17. RepresentingMeaning
muchtoexpectfromanysinglerepresentationalsystem,Sec .17.3willshowthatFirst-
OrderLogicisexpressiveenoughtohandlequitealotofwhat needstoberepresented.
17.2 Model-TheoreticSemantics
Thelasttwosectionsfocusedonvariousdesiderataformean ingrepresentationsandon
some of the ways in which natural languagesconveymeaning. W e haven’tsaid much
formally about what it is about meaning representation lang uages that allows them to
do all the things we want them to. In particular, we might like to have some kind of
guarantee that these representations can do the work that we require of them: bridge
the gap from merely formal representations to representati ons that tell us something
aboutsomestate ofaffairsinthe world.
To see how we might provide such a guarantee, let’s start with the basic notions
shared by most meaning representation schemes. What they al l have in common is
theabilityto representobjects,propertiesofobjectsand relationsamongobjects. This

point of view can be formalized via the notion of a model. The basic idea is that a Model
modelisaformalconstructthatstandsfortheparticularst ateofaffairsintheworldthat
we’re trying to represent. Expressions in a meaning represe ntationlanguagewill then
be mapped in a systematic way to the elements of the model. If t he model accurately
captures the facts we’re interested in concerning some stat e of affairs in the world,
then a systematic mapping between the meaning representati on and model provides

the necessarybridgebetweenthe meaningrepresentationan dworld beingconsidered.
As we’ll see, models provide a surprisingly simple and power ful way to ground the
expressionsinmeaningrepresentationlanguages.
Before we start let’s introduce some terminology. The vocab ulary of a meaning
representation consists of two parts: the non-logical voca bulary and the logical vo-
cabulary. The non-logical vocabulary consists of the open-ended set of names forNon-logical
vocabulary

the objects, properties and relations that make up the world we’re trying to represent.
Theseappearinvariousschemesaspredicates,nodes,label sonlinks,orlabelsinslots
in frames, The logical vocabulary consists of the closed set of symbols, operators,Logical
quantiﬁers, links, etc. that provide the formal means for co mposing expressions in a
givenmeaningrepresentationlanguage.
We’llstartbyrequiringthateachelementofthenon-logica lvocabularyofamean-

ing representation have a denotation in the model. By denotation, we simply mean Denotation
that every element of the non-logical vocabulary correspon ds to a ﬁxed well-deﬁned
part of the model. Let’s start with objects, the most basic no tion in most represen-
tational schemes. The domainof a model is simply the set of objects that are part of Domain
theapplication,orstateofaffairs,beingrepresented. Ea chdistinctconcept,categoryor
individualinanapplicationdenotesauniqueelementinthe domain. Adomainisthere-

fore formally a set. Note that it isn’t the case that every ele ment of the domain have a
correspondingconceptin ourmeaningrepresentation;it’s perfectlyacceptabletohave
domainelementsthat aren’tmentionedor conceivedof in the meaningrepresentation.
Nor do we require that elements of the domain have a single den oting concept in the
meaningrepresentation;agivenelementinthedomainmight haveseveraldistinctrep-

DRAFTSection17.2. Model-TheoreticSemantics 559
resentationsdenotingit,suchas Mary,WifeOf(Abe) ,orMotherOf(Robert) .
Wecancapturepropertiesofobjectsinamodelbydenotingth osedomainelements
that have the property in question; that is, properties deno te sets. Similarly, relations
among objects denote sets of ordered lists, or tuples, of dom ain elements that take
part in the corresponding relations. This approach to prope rties and relations is thus
anextensional one; the denotation of properties like redis the set of things we think
are red, the denotation of a relation like Marriedis simply the set of pairs of domain
elementsthataremarried. Tosummarize:
•Objectsdenote elementsofthedomain
•Propertiesdenote setsof elements ofthedomain
•Relationsdenote setsof tuplesofelements ofthedomain
Thereisoneadditionalelementthatweneedtomakethissche mework. Weneeda
mappingthatsystematicallygetsusfromourmeaningrepres entationtothecorrespond-
ingdenotations. Moreformally,weneedafunctionthatmaps fromthenon-logicalvo-

cabulary of our meaning representation to the proper denota tions in the model. We’ll
call suchamappingan interpretation . Interpretation
To make these notions more concrete, let’s return to the real m of restaurants we
introducedinCh.4. Assumethatourapplicationconcernsap articularsetofrestaurant
patrons and restaurants, various facts about the likes and d islikes of the patrons, and
factsabouttherestaurantssuchastheircuisine,typicalc ost,andnoiselevel.
To begin populating our domain, D, let’s assume that in the current state of af-

fairs we’re dealing with four patrons designated by the non- logical symbols Matthew,
Franco, Katie andCaroline. These four symbols will denote 4 unique domain ele-
ments. We’ll use the constants a,b,cand,dto stand for these domain elements. Note
that we’re deliberately using meaningless, non-mnemonic n ames for our domain ele-
mentstoemphasizethefactthatwhateverit isthatweknowab outtheseentitieshasto
comefromthe formalpropertiesof themodeland notfromthe n amesof the symbols.
Continuing, let’s assume that our application includes thr ee restaurants, designated as

Frasca,MedandRioin our meaning representation, that denote the domain eleme nts
e,fandg. Finally,let’sassumethatwe’redealingwiththethreecui sinesItalian,Mex-
ican,andEclectic,denoting i,j, andkin ourmodel.
Having populated the domain, let’s move on to the properties and relations we
believetobe truein thisparticularstate of affairs. Let’s assumethat in ourapplication
weneedtorepresentsomepropertiesofrestaurantssuchast hefactthatsomearenoisy
or expensive. Properties like Noisydenote the subset of restaurants from our domain

that are known to be noisy. Two-place relational notions, su ch as which restaurants
individualpatrons Like,denoteorderedpairs,ortuples,oftheobjectsfromthedom ain.
Similarly, since we decided to represent cuisines as object s in our model, we can also
capture which restaurants Servewhich cuisines as a set of tuples. One particular state
ofaffairsusingthisschemeisgiveninFig. 17.2.
Given this simple scheme, we can ground the meaning of pretty much any of the
representations shown earlier in Fig. 17.1 by simply consul ting the appropriate de-

notations in the corresponding model. A representation cla iming, for example, that
MatthewlikestheRio ,orthatthe TheMedservesItalian canbeevaluatedbymapping
theobjectsinthemeaningrepresentationstotheircorresp ondingdomainelements,and

DRAFT560 Chapter 17. RepresentingMeaning
Domain D={a,b,c,d,e,f,g,h,i,j}
Matthew,Franco,KatieandCaroline a,b,c,d
Frasca, Med,Rio e,f,g
Italian,Mexican,Eclectic h,i,j
Properties
Noisy Noisy ={e,f,g}
Frasca,MedandRio arenoisy
Relations
Likes Likes ={⟨a,f⟩,⟨c,f⟩,⟨c,g⟩,⟨b,e⟩,⟨d,f⟩,⟨d,g⟩}
MatthewlikestheMed
Katie likestheMedandRio
FrancolikesFrasca
CarolinelikestheMedandRio
Serves Serves ={⟨e,j⟩,⟨f,i⟩,⟨e,h⟩}
Medserveseclectic
Rio servesMexican
Frasca servesItalian
Figure17.2 A model of the restaurant world.
anylinks,predicates,orslotsinthemeaningrepresentati ontotheappropriaterelations
in the model. More concretely, a representation asserting t hatMatthew likes Frasca
canbeveriﬁedbyﬁrstusingourinterpretationfunctiontom apthesymbol Matthew to
itsdenotation a,Frascatoe,andtheLikesrelationtotheappropriateset oftuples. We
thensimplycheckthatsetoftuplesforthepresenceofthetu ple⟨a,e⟩. If,asitisinthis
case, thetupleis presentin themodelthenwe canconcludeth atMatthewlikesFrasca
istrue,andif it isn’twecan’t.

This is all pretty much straightforward, we’re simply using sets and operationson
sets togroundthe expressionsinourmeaningrepresentatio ns. Ofcourse,themorein-
terestingpartcomeswhenweconsidermorecomplexexamples suchasthefollowing:
(17.14)KatielikestheRio andMatthewlikestheMed.
(17.15)KatieandCarolinelikethesame restaurants.
(17.16)Francolikesnoisy,expensiverestaurants.
(17.17)NoteverybodylikesFrasca.
Clearly, our simple scheme for grounding the meaning of repr esentations is not

adequate for examples such as these. Plausible meaning repr esentations for these ex-
ampleswillnotmapdirectlytoindividualentities,proper tiesorrelations. Instead,they
involve complications such as conjunctions, equality, qua ntiﬁed variables and nega-
tions. To assess whether or not these statements are consist ent with our model we’ll
have to tear them apart, assess the parts and then determine t he meaning of the whole
fromthemeaningofthepartsaccordingto thedetailsofhowt hewholeisassembled.

Considertheﬁrstexamplegivenabove. Atypicalmeaningrep resentationforexam-
ples like this will include two distinct propositions expre ssing the individual patron’s
preferences,conjoinedwithsomekindofimplicitorexplic itconjunctionoperator. Ob-
viously,ourmodeldoesn’thavearelationthatencodesthep airwisepreferencesforall

DRAFTSection17.3. First-OrderLogic 561
of the patrons and restaurants in our model, nor does it need t o. We know from our
model that Matthew likes the Med and separately that Katie likes the Rio (that is, we
knowthat thetuples ⟨a,f⟩and⟨c,g⟩aremembersofthe set denotedbythe Likesrela-
tion.) All we really needto knowis how to deal with the semant icsof the conjunction
operator. If we assume the simplest possible semantics for t he English word and, the
wholestatementistrueif it isthecase eachofthecomponent sis trueinourmodel. In
this case, both componentsare true since the appropriatetu ples are present and there-
forethe sentenceasa wholeis true.
What we’ve done implicitly in this example is to provide what is called a truth-
conditional semantics for the assumed conjunction operator in some meaning repre-Truth-conditionalsemantics
sentation. That is, we’ve provided a method for determining the truth of a complex
expression from the meanings of the parts (by consulting a mo del) and the meaning

of an operatorby essentially consulting a truth-table. The variousrepresentationsthat
populate Fig. 17.1 are truth-conditional to the extent that they give a formal speciﬁ-
cation as to how we can assess the meaning of complex sentence s from the meaning
of their parts. In particular, we’ll need to know the semanti cs of the entire logical
vocabularyofthemeaningrepresentationschemebeingused .
Note that although the details of how this happens is depende nt on details of the

particularmeaningrepresentationbeingused,itshouldbe clearthatassessingthetruth
conditions of examples like these involves nothing beyond t he simple set operations
we’vebeendiscussing. We’llreturntotheseissuesinthene xtsectionwherewediscuss
theminthecontextofthesemanticsofFirst OrderLogic.
17.3 First-OrderLogic
First-Order Logic ( FOL) is a ﬂexible, well-understood and computationally tracta ble
approachtotherepresentationofknowledgethatsatisﬁesm anyofthedesideratagiven

in Sec. 17.1 for a meaning representation language. Speciﬁc ally, it provides a sound
computational basis for the veriﬁability, inference, and e xpressiveness requirements,
aswell asasoundmodel-theoreticsemantics.
In addition, an attractive feature of FOLis that it makes very few speciﬁc commit-
ments as to how things ought to be represented. As we will see, the speciﬁc commit-
mentsitdoesmakeareonesthatarefairlyeasytolivewith,a ndaresharedbymanyof
theschemesmentionedearlier;therepresentedworldconsi stsofobjects,propertiesof

objects,andrelationsamongobjects.
Theremainderof thissection ﬁrst providesan introduction to the basic syntaxand
semantics of FOL, describes the application of FOLto the representation of events,
Sec.17.6thendiscussestheconnectionsbetween FOLandsomeoftheotherrepresen-
tationalapproaches.
17.3.1 BasicElements ofFirstOrderLogic
We will explore FOLin a bottom-upfashion by ﬁrst examiningits variousatomic e le-
mentsandthenshowinghowtheycanbecomposedtocreatelarg ermeaningrepresen-

DRAFT562 Chapter 17. RepresentingMeaning
Formula→AtomicFormula
|FormulaConnectiveFormula
|QuantiﬁerVariable ,...Formula
| ¬Formula
|(Formula )
AtomicFormula→Predicate (Term,...)
Term→Function (Term,...)
|Constant
|Variable
Connective→ ∧|∨| ⇒
Quantiﬁer→ ∀|∃
Constant→A|VegetarianFood|Maharani···
Variable→x|y|···
Predicate→Serves|Near|···
Function→LocationOf|CuisineOf|···
Figure17.3 Acontext-freegrammarspeciﬁcationofthesyntaxofFirstO rderLogicrepresen-
tations. Adapted from Russell andNorvig (1995)
tations. Fig. 17.3, which provides a complete context-free grammar for the particular
syntaxof FOLthatwe will beusing,will beourroadmapforthissection.
Let’s begin by examining the notion of a Term, theFOLdevice for representing Term
objects. As can be seen from Fig. 17.3, FOLprovides three ways to represent these
basic building blocks: constants, functions, and variable s. Each of these devices can
be thoughtof as a way of naming, or pointingto, an object in th e world underconsid-
eration.

eration.
Constants inFOLrefer to speciﬁc objects in the world being described. Such Constants
constantsareconventionallydepictedaseithersinglecap italizedletterssuchas AandB
orsinglecapitalizedwordsthatareoftenreminiscentofpr opernounssuchas Maharani
andHarry. Like programminglanguageconstants, FOLconstantsrefer to exactlyone
object. Objectscan,however,havemultipleconstantsthat refertothem.
Functions inFOLcorrespond to concepts that are often expressed in English a s Functions

eration.
genitives such as Frasca’s location . AFOLtranslation of such an expression might
looklike thefollowing.
LocationOf (Frasca ) (17.18)
FOLfunctionsaresyntacticallythe sameas singleargumentpre dicates. It isimportant
to remember, however, that while they have the appearance of predicates they are in
factTermsin that they refer to uniqueobjects. Functionsprovidea con venientway to
refer to speciﬁc objectswithout having to associate a named constantwith them. This

is particularly convenient in cases where many named object s, like restaurants, will
haveauniqueconceptsuchasalocationassociatedwiththem .
Thenotionofa variableisourﬁnal FOLmechanismforreferringto objects. Vari- Variable
ables, which are normally depicted as single lower-case let ters, give us the ability to
makeassertionsanddrawinferencesaboutobjectswithouth avingtomakereferenceto
anyparticularnamedobject. Thisabilitytomakestatement saboutanonymousobjects
comesin two ﬂavors: makingstatementsabouta particularun knownobject,andmak-

DRAFTSection17.3. First-OrderLogic 563
ing statements about all the objects in some arbitraryworld of objects. We will return
to the topic of variables after we have presented quantiﬁers , the elements of FOLthat
will makethemuseful.
Nowthat we havethe meansto referto objects,we can moveonto theFOLmech-
anismsthatare usedto state relationsthatholdamongobjec ts. Predicatesaresymbols
that refer to, or name, the relations that hold among some ﬁxe d number of objects in
a given domain. Returning to the example introduced informa lly in Sec. 17.1, a rea-
sonable FOLrepresentation for Maharani serves vegetarian food might look like the
followingformula:
Serves (Maharani ,VegetarianFood ) (17.19)
ThisFOLsentenceassertsthat Serves,atwo-placepredicate,holdsbetweentheobjects
denotedbythe constants Maharani andVegetarianFood .
Asomewhatdifferentuseofpredicatesisillustratedbythe followingfairlytypical
representationforasentencelike Maharaniisarestaurant :
Restaurant (Maharani ) (17.20)

This is an exampleof a one-placepredicatethat is used, not t o relate multipleobjects,
but rather to assert a property of a single object. In this cas e, it encodes the category
membershipof Maharani .
With the ability to refer to objects, to assert facts about ob jects, and to relate ob-
jects to one another, we have the ability to create rudimenta ry composite representa-
tions. These representationscorrespondto the atomic form ula level in Fig. 17.3. This
ability to compose complex representations is, however, no t limited to the use of sin-

gle predicates. Larger composite representations can also be put together through the
use oflogical connectives As can be seen from Fig. 17.3, logical connectivesgive usLogicalconnectives
the ability to create largerrepresentationsby conjoining logical formulasusing one of
three operators. Consider, for example,the following BERPsentence and one possible
representationforit:
(17.21)IonlyhaveﬁvedollarsandI don’thavealot oftime.
Have(Speaker ,FiveDollars )∧¬Have(Speaker ,LotOfTime ) (17.22)

Thesemanticrepresentationforthisexampleis builtupin a straightforwardway from
semanticsoftheindividualclausesthroughtheuseofthe ∧and¬operators. Notethat
the recursive nature of the grammar in Fig. 17.3 allows an inﬁ nite number of logical
formulas to be created through the use of these connectives. Thus as with syntax, we
havetheabilitytocreateaninﬁnitenumberofrepresentati onsusingaﬁnite device.
17.3.2 VariablesandQuantiﬁers
We now have all the machinery necessary to return to our earli er discussion of vari-

ables. As noted above, variables are used in two ways in FOL: to refer to particular
anonymous objects and to refer generically to all objects in a collection. These two
uses are made possible through the use of operators known as quantiﬁers . The two Quantiﬁers
operatorsthatare basicto FOLaretheexistentialquantiﬁer,whichisdenoted ∃, andis

DRAFT564 Chapter 17. RepresentingMeaning
pronounced as “there exists”, and the universal quantiﬁer, which is denoted∀, and is
pronouncedas“forall”.
The need for an existentially quantiﬁed variable is often si gnaled by the presence
ofanindeﬁnitenounphraseinEnglish. Considerthefollowi ngexample:
(17.23)arestaurantthat servesMexicanfoodnearICSI.
Here,referenceisbeingmadetoananonymousobjectofaspec iﬁedcategorywithpar-
ticular properties. The followingwould be a reasonablerep resentationof the meaning
ofsuchaphrase:
∃xRestaurant (x)∧Serves (x,MexicanFood ) (17.24)
∧Near((LocationOf (x),LocationOf (ICSI))
Theexistentialquantiﬁerattheheadofthissentenceinstr uctsusonhowtointerpret
the variable xin the context of this sentence. Informally, it says that for this sentence
to be true there must be at least one object such that if we were to substitute it for the
variablex, theresultingsentencewouldbetrue. Forexample,if AyCaramba isa Mex-
ican restaurant near ICSI, then substituting AyCaramba forxresults in the following

logicalformula:
Restaurant (AyCaramba )∧Serves (AyCaramba ,MexicanFood ) (17.25)
∧Near((LocationOf (AyCaramba ),LocationOf (ICSI))
Basedonthesemanticsofthe ∧operator,thissentencewillbetrueifallofitsthree
componentatomicformulasaretrue. Theseinturnwillbetru eiftheyareeitherpresent
in the system’s knowledge base or can be inferred from other f acts in the knowledge
base.
The use of the universal quantiﬁer also has an interpretatio n based on substitution

base.
of known objects for variables. The substitution semantics for the universal quantiﬁer
takes the expression for allquite literally; the∀operatorstates that forthe logicalfor-
mulainquestiontobetruethesubstitutionof anyobjectintheknowledgebaseforthe
universally quantiﬁed variable should result in a true form ula. This is in marked con-
trast to the∃operatorwhichonlyinsists ona single valid substitutionf or thesentence
tobetrue.
(17.26)Allvegetarianrestaurantsservevegetarianfood.

tobetrue.
A reasonablerepresentationforthissentencewouldbesome thinglikethefollowing:
∀xVegetarianRestaurant (x)⇒Serves (x,VegetarianFood ) (17.27)
Forthissentencetobetrue,itmustbethecasethateverysub stitutionofaknownobject
forxmust result in a sentence that is true. We can divide up the set of all possible
substitutions into the set of objects consisting of vegetar ian restaurants and the set
consistingofeverythingelse. Letusﬁrstconsiderthecase wherethesubstitutedobject

actually is a vegetarian restaurant; one such substitution would result in the following
sentence:
VegetarianRestaurant (Maharani )⇒Serves (Maharani ,VegetarianFood ) (17.28)

DRAFTSection17.3. First-OrderLogic 565
Ifweassumethat weknowthat theconsequentclause,
Serves (Maharani ,VegetarianFood ) (17.29)
is true then this sentence as a whole must be true. Both the ant ecedent and the conse-
quent have the value Trueand, therefore, according to the ﬁrst two rows of Fig. 17.4
thesentenceitself can havethe value True. Thisresultwill, of course,bethe samefor
all possiblesubstitutionsof Termsrepresentingvegetarianrestaurantsfor x.
Remember, however, that for this sentence to be true it must b e true for all possi-
ble substitutions. What happens when we consider a substitu tion from the set of ob-
jects that are not vegetarianrestaurants? Consider the sub stitution of a non-vegetarian
restaurantsuchas Ay Caramba’s forthevariable x:
VegetarianRestaurant (AyCaramba )⇒Serves (AyCaramba ,VegetarianFood ) (17.30)
Since the antecedent of the implication is False, we can determine from Fig. 17.4
thatthesentenceisalways True,againsatisfyingthe ∀constraint.

Note, that it may still be the case that Ay Caramba serves vegetarianfood without
actually being a vegetarianrestaurant. Note also, that des pite our choice of examples,
thereare noimpliedcategoricalrestrictionsonthe object sthat can besubstitutedfor x
by this kind of reasoning. In other words, there is no restric tion ofxto restaurants or
conceptsrelatedtothem. Considerthe followingsubstitut ion:
VegetarianRestaurant (Carburetor )⇒Serves (Carburetor ,VegetarianFood ) (17.31)
Here the antecedent is still false and hence the rule remains true under this kind of

irrelevantsubstitution.
To review, variables in logical formulas must be either exis tentially (∃) or univer-
sally(∀)quantiﬁed. Tosatisfyanexistentiallyquantiﬁedvariabl e,theremustbeatleast
one substitution that results in a true sentence. Sentences with universally quantiﬁed
variablesmust betrueunderall possiblesubstitutions.
17.3.3 Lambda Notation
The ﬁnal element we need to complete our discussion of FOLis called the lambda
notation (Church, 1940). This notation provides a way to abstract awa y from fully Lambda notation

speciﬁed FOLformula in a way that will be particularly useful for semanti c analysis.
Thelambdanotationextendsthesyntaxof FOLtoincludeexpressionsofthefollowing
form:
λx.P(x) (17.32)
Such expressions consist of the Greek symbol λ, followed by one or more variables,
followedbya FOLformulathat makesuseofthosevariables.
Theusefulnessofthese λ-expressionsisbasedontheabilitytoapplythemtologi-
caltermstoyieldnew FOLexpressionswheretheformalparametervariablesarebound

form:
to the speciﬁed terms. This processis knownas λ-reduction and consists of a simple λ-reduction
textual replacement of the λvariables with the speciﬁed FOLterms, accompanied by
the subsequent removal of the λ. The following expressions illustrate the application

DRAFT566 Chapter 17. RepresentingMeaning
ofaλ-expressiontotheconstant A,followedbytheresultofperforminga λ-reduction
onthisexpression:
λx.P(x)(A) (17.33)
P(A)
An importantand useful variation of this techniqueis the us e of oneλ-expression
asthebodyofanotherasin thefollowingexpression:
λx.λy.Near(x,y) (17.34)
This fairly abstract expressioncan be glossed as the state o f somethingbeing near
somethingelse. Thefollowingexpressionsillustrate a sin gleλ-applicationandsubse-
quentreductionwiththiskindofembedded λ-expression:
λx.λy.Near(x,y)(Bacaro ) (17.35)
λy.Near(Bacaro ,y)
Theimportantpointhereisthatthe resultingexpressionis still aλ-expression;the
ﬁrst reduction bound the variable xand removed the outer λ, thus revealing the inner
expression. As might be expected, this resulting λ-expressioncan, in turn, be applied
toanothertermtoarriveat a fullyspeciﬁedlogicalformula ,asin thefollowing:
λy.Near(Bacaro ,y)(Centro ) (17.36)
Near(Bacaro ,Centro )

Thisgeneraltechnique,called currying2(Sch¨ onkﬁnkel,1924)isawayofconvert- Currying
ingapredicatewithmultipleargumentsintoasequenceofsi ngleargumentpredicates.
AswewillseeinCh.18,the λ-notationprovideswhentheargumentstoapredicate
donotallappeartogetherasdaughtersofthepredicateina p arsetree.
17.3.4 The Semantics ofFirst-Order Logic
The various objects, properties, and relations represente d in aFOLknowledge base
acquire their meanings by virtue of their correspondence to objects, properties, and

relationsoutintheexternalworldbeingmodeled. We canacc omplishthisbyemploy-
ing the model-theoretic approach introduced in Sec. 17.2. R ecall that this approach
employs simple set-theoretic notions to provide a truth-co nditionalmapping from the
expressionsin a meaning representationto the state of affa irs being modeled. We can
applythisappoachto FOL, by goingthroughall the elementsin Fig. 17.3andspecify-
inghoweachshouldbeaccountedfor.
Wecanstartbyassertingthattheobjectsinourworld, FOLterms,denoteelements

in a domain, and that atomic formulas are captured either as s ets of domain elements
for properties, or as sets of tuples of elements for relation s. As an example consider
thefollowing:
2Currying is the standard term, although Heim and Kratzer (1998) prese nt an interesting argument for the
termSch¨ onkﬁnkelization over currying, since Curry laterbuilt on Sch¨ onﬁnkel’s work.

DRAFTSection17.3. First-OrderLogic 567
(17.37)CentroisnearBacaro.
Capturing the meaning of this example in FOLinvolves identifying the Termsand
Predicates that correspond to the various grammatical elements in the s entence, and
creatinglogicalformulasthatcapturetherelationsimpli edbythewordsandsyntaxof
thesentence. Forthisexample,suchaneffortmightyieldso methinglikethefollowing:
Near(Centro ,Bacaro ) (17.38)
Themeaningofthislogicalformulaisbasedonwhetherthedo mainelementsdenoted
bytheterms CentroandBacaroarecontainedamongthetuplesdenotedbytherelation
denotedbythe predicate Nearin thecurrentmodel.
Theinterpretationsofformulasinvolvinglogicalconnect ivesisbasedonthemean-
ingofthecomponentsin theformulascombinedwith themeani ngsofthe connectives
they contain. Fig. 17.4 givesinterpretationsfor each of th e logical operatorsshown in
Fig.17.3.
P Q ¬P P∧Q P∨Q P⇒Q
False False True False False True
False True True False True True
True False False False True False
True True False True True True

Fig.17.3.
Figure17.4 Truthtable giving the semantics of the various logical conn ectives.
Thesemantics of the ∧(and),and¬(not)operatorsare fairly straightforward,and
are correlated with at least some of the senses of the corresp onding English terms.
However,it isworthpointingoutthatthe ∨(or)operatorisnotdisjunctivein thesame
waythatthecorrespondingEnglishwordis,andthatthe ⇒(implies)operatorisonly
looselybasedonanycommonsensenotionsofimplicationorc ausation.
Theﬁnalbitweneedtoaddressinvolvesvariablesandquanti ﬁers. Recallthatthere

arenovariablesinourset-basedmodels,onlyelementsofth edomainandrelationsthat
hold amongthem. We can providea model-basedaccountfor for mulaswith variables
byemployingthenotionofasubstitutionintroducedearlie ron563. Formulasinvolving
∃aretrueifthereisasubstitutionoftermsforvariablestha tresultsinaformulathatis
trueinthemodel. Formulasinvolving ∀mustbetrueunderallpossiblesubstitutions.
17.3.5 Inference
One of the most important desiderata given in Sec. 17.1 for a m eaning representa-
tion language is that it should support inference—the abili ty to add valid new propo-

sitions to a knowledge base, or to determine the truth of prop ositions not explicitly
containedwithin a knowledgebase. This section brieﬂydisc ussesmodus ponens , the
most widely implementedinferencemethod providedby FOL. Applicationsof modus
ponenstoinferenceindiscoursewill bediscussedinCh.21.
Modus ponens is a familiar form of inference that corresponds to what is in for- Modusponens
mally knownas if-thenreasoning. We can abstractly deﬁne modusponensas follows,

DRAFT568 Chapter 17. RepresentingMeaning
whereαandβshouldbetakenas FOLformulas:
α
α⇒β
β(17.39)
A schema like this indicates that the formula below the line c an be inferred from the
formulas above the line by some form of inference. Modus pone ns simply states that
if the left-hand side of an implication rule is true, then the right-hand side of the rule
can be inferred. In the following discussions, we will refer to the left-hand side of an
implicationastheantecedent,andtheright-handside asth econsequent.
As an exampleof a typical use of modusponens, consider the fo llowingexample,
whichusesa rulefromthelast section:
VegetarianRestaurant (Leaf)
∀xVegetarianRestaurant (x)⇒Serves (x,VegetarianFood )
Serves (Leaf,VegetarianFood )(17.40)
Here,theformula VegetarianRestaurant (Leaf)matchestheantecedentoftherule,thus
allowingustouse modusponenstoconclude Serves (Leaf,VegetarianFood ).
Modusponenscanbeputtopracticaluseinoneoftwoways: for wardchainingand

backward chaining. In forward chaining systems, modus ponensis used in precisely Forwardchaining
themannerjustdescribed. Asindividualfactsareaddedtot heknowledgebase,modus
ponens is used to ﬁre all applicable implication rules. In th is kind of arrangement,
as soon as a new fact is added to the knowledge base, all applic able implication rules
are foundand applied, each resulting in the additionnew fac ts to the knowledgebase.
Thesenewpropositionsinturncanbeusedtoﬁreimplication rulesapplicabletothem.
Theprocesscontinuesuntilnofurtherfactscanbededuced.

The forward chaining approach has the advantage that facts w ill be present in the
knowledge base when needed, since in a sense all inference is performed in advance.
Thiscansubstantiallyreducethe time neededto answersubs equentqueriessince they
should all amount to simple lookups. The disadvantage of thi s approach is that facts
may be inferred and stored that will never be needed. Production systems , whichProductionsystems
are used extensively in cognitive modeling research, are fo rward chaining inference

systems augmentedwith additionalcontrolknowledgethat g overnswhich rules are to
beﬁred.
Inbackward chaining , modus ponens is run in reverse to prove speciﬁc proposi-Backwardchaining
tions,calledqueries. Theﬁrst stepistoseeifthequeryfor mulaistruebydetermining
if it is present in the knowledge base. If it is not, then the ne xt step is to search for
applicable implication rules present in the knowledge base . An applicable rule is one
where the consequent of the rule matches the query formula. I f there are any such

beﬁred.
rules, then the query can be proved if the antecedent of any on e them can be shown
to be true. Not surprisingly, this can be performed recursiv ely by backward chaining
on the antecedent as a new query. The Prologprogramming language is a backward
chainingsystemthatimplementsthisstrategy.
To see how this works, let’s assume that we have been asked to v erify the truth
of the proposition Serves (Leaf,VegetarianFood ), assuming the facts given above the

DRAFTSection17.4. RepresentingEventsandStates 569
line in (17.40). Since it is not present in the knowledge base , a search for an ap-
plicable rule is initiated that results in the rule given abo ve. After substituting, the
constantLeaffor the variable x, our next task is to prove the antecedent of the rule,
VegetarianRestaurant (Leaf),whichofcourseisoneofthefactsweare given.
Notethatitiscriticaltodistinguishbetweenreasoningvi abackwardchainingfrom
queriestoknownfacts,andreasoningbackwardsfromknownc onsequentstounknown
antecedents. Tobespeciﬁc,byreasoningbackwardswemeant hatiftheconsequentof
aruleisknowntobetrue,weassumethattheantecedentwillb easwell. Forexample,
let’s assume that we know that Serves (Leaf,VegetarianFood )is true. Since this fact
matchesthe consequentofourrule,we mightreasonbackward sto the conclusionthat
VegetarianRestaurant (Leaf).
While backwardchainingis a sound methodof reasoning, reas oningbackwardsis
an invalid,thoughfrequentlyuseful, form of plausiblereasoning . Plausible reasoning

from consequentsto antecedentsis knownas abduction , and as we will see in Ch. 21 Abduction
is often useful in accounting for many of the inferences peop le make while analyzing
extendeddiscourses.
Whileforwardandbackwardreasoningaresound,neitheris complete . Thismeans Complete
that there are valid inferences that can not be found by syste ms using these methods
alone. Fortunately, there is an alternative inference tech nique called resolution that Resolution

is sound and complete. Unfortunately, inference systems ba sed on resolution are far
morecomputationallyexpensivethan forwardor backwardch ainingsystems. In prac-
tice,therefore,mostsystemsusesomeformofchaining,and placeaburdenonknowl-
edge base developersto encode the knowledgein a fashion tha t permits the necessary
inferencestobedrawn.
17.4 RepresentingEventsandStates
Muchofthesemanticsthatweneedtocaptureinlanguageisco mprisedofrepresenta-
tions of states and events. Roughly speaking, states are con ditions, or properties, that

remain unchangedover some period of time while events denot e changes in the some
state of affairs. The representation of both can involve a ho st of participants, props,
timesandlocations.
The representations for events and states that we have emplo yed thus far have
consisted of single predicates with as many arguments as are needed to incorporate
all the roles associated with a given example. For example, t he representation for
an expression such as Leaf serves vegetarian fare consists of a single predicate with

argumentsfortheentitydoingtheservingandthethingserv ed.
Serves (Leaf,VegetarianFare ) (17.41)
Suchanapproachsimplyassumesthatthepredicatedenoting themeaningofaverbhas
the same numberof argumentsas are present in the verb’ssynt actic subcategorization
frame. Unfortunately,therearefourproblemswiththisapp roachthatmakeitawkward
toapplyinpractice:

DRAFT570 Chapter 17. RepresentingMeaning
•Determiningthecorrectnumberofrolesforanygivenevent
•Representingfactsaboutthe rolesassociatedwithanevent
•Ensuring that all the correct inferences can be derived dire ctly from the repre-
sentationofanevent
•Ensuring that no incorrect inferences can be derived from th e representation of
anevent
We will explore these, and other related issues, by consider ing a series of repre-
sentationsforevents. Thisdiscussionwillfocusonthefol lowingexamplesoftheverb
eat:
(17.42)Iate.
(17.43)Iate aturkeysandwich.
(17.44)Iate aturkeysandwichat mydesk.
(17.45)Iate atmydesk.
(17.46)Iate lunch.
(17.47)Iate aturkeysandwichforlunch.
(17.48)Iate aturkeysandwichforlunchat mydesk.
Clearly, the variable number of arguments for a predicate-b earing verb like eat
posesatrickyproblem. Whilewewouldliketothinkthatall o ftheseexamplesdenote
the same kind of event, predicates in FOLhave ﬁxed arity—they take a ﬁxed number Arity
ofarguments.

eat:
Onepossiblesolutionissuggestedbythewaythatexamplesl ikethesearehandled
syntactically. For example, the solution given in Ch. 16 was to create one subcate-
gorization frame for each of the conﬁgurations of arguments that a verb allows. The
semanticanalogtothisapproachistocreateasmanydiffere nteatingpredicatesasare
needed to handle all of the ways that eatbehaves. Such an approach would yield the
followingkindsofrepresentationsfor(17.42)through(17 .48).
Eating 1(Speaker )
Eating 2(Speaker ,TurkeySandwich )
Eating 3(Speaker ,TurkeySandwich ,Desk)

Eating 4(Speaker ,Desk)
Eating 5(Speaker ,Lunch )
Eating 6(Speaker ,TurkeySandwich ,Lunch)
Eating 7(Speaker ,TurkeySandwich ,Lunch,Desk)
Thisapproachsimplysidestepstheissueofhowmanyargumen tstheEatingpredicate
should have by creating distinct predicates for each of the s ubcategorization frames.
Unfortunately, this approach comes at a rather high cost. Ot her than the suggestive
namesofthepredicates,thereisnothingtotie theseevents tooneanothereventhough
there are obviouslogical relationsamongthem. Speciﬁcall y, if (17.48)is true then all

oftheotherexamplesaretrueaswell. Similarly,if(17.47) istruethen(17.42),(17.43),
and (17.46) must also be true. Such logical connectionscan n ot be made on the basis
ofthesepredicatesalone. Moreover,wewouldexpectacommo nsenseknowledgebase

DRAFTSection17.4. RepresentingEventsandStates 571
to contain logical connectionsbetween concepts like Eatingand related concepts like
HungerandFood.
One method to solve these problems involves the use of what ar e calledmeaning
postulates . Considerthefollowingexamplepostulate:Meaning
postulates
∀w,x,y,zEating 7(w,x,y,z)⇒Eating 6(w,x,y) (17.49)
This postulate explicitly ties together the semantics of tw o of our predicates. Other
postulatescouldbecreatedtohandletherestofthelogical relationsamongthevarious
Eatingsandtheconnectionsfromthemto otherrelatedconcepts.
Althoughsuchanapproachmightbemadetoworkinsmalldomai ns,itclearlyhas
scalabilityproblems. Asomewhatmoresensibleapproachis tosaythat(17.42)through
(17.48)allreferencethesamepredicatewithsomeofthearg umentsmissingfromsome
of the surface forms. Under this approach, as many arguments are included in the
deﬁnitionof the predicateas everappearwith it in an input. Adoptingthe structureof

a predicate like Eating 7as an examplewould give us a predicate with four arguments
denoting the eater, thing eaten, meal being eaten and the loc ation of the eating. The
followingformulaswouldthencapturethesemanticsofoure xamples:
∃w,x,yEating (Speaker ,w,x,y)
∃w,xEating (Speaker ,TurkeySandwich ,w,x)
∃wEating (Speaker ,TurkeySandwich ,w,Desk)
∃w,xEating (Speaker ,w,x,Desk)
∃w,xEating (Speaker ,w,Lunch ,x)
∃wEating (Speaker ,TurkeySandwich ,Lunch,w)
Eating (Speaker ,TurkeySandwich ,Lunch,Desk)
Thisapproachdirectlyyieldstheobviouslogicalconnecti onsamongtheseformulas

without the use of meaning postulates. Speciﬁcally, all of t he sentences with ground
terms as arguments logically imply the truth of the formulas with existentially bound
variablesasarguments.
Unfortunately, this approach has at least two glaring deﬁci encies: it makes too
manycommitments,andit doesnotlet usindividuateevents. Asanexampleofhowit
makestoomanycommitments,considerhowweaccommodatedth eforlunch comple-
ment in (17.46)through(17.48);a thirdargument,the meal b eingeaten, was addedto

theEatingpredicate. The presence of this argument implicitly makes i t the case that
all eating events are associated with a meal (i.e., breakfas t, lunch, or dinner). More
speciﬁcally,theexistentiallyquantiﬁedvariableforthe mealargumentintheaboveex-
amplesstatesthatthereissomeformalmealassociatedwith eachoftheseeatings. This
isclearlysillysinceonecancertainlyeatsomethingindep endentofitbeingassociated
witha meal.
Toseehowthisapproachfailstoproperlyindividuateevent s,considerthefollowing
formulas.

formulas.
∃w,xEating (Speaker ,w,Lunch ,Desk)

DRAFT572 Chapter 17. RepresentingMeaning
If we knew that the ﬁrst two formulaswere referringto the sam e event,they couldbe
combinedtocreatethethirdrepresentation. Unfortunatel y,withthecurrentrepresenta-
tionwehavenowayoftellingifthisispossible. Theindepen dentfactsthat Iateatmy
deskandI ate lunch do not permit us to conclude that I ate lunch at my desk . Clearly
what is lacking in this approach is some way of referring to th e particular events in
question.
We can solve these problems if we employ reiﬁcation to elevat e events to entities
that can be quantiﬁed over. To accomplish, we can simply add a nevent variable as Eventvariable
the ﬁrst argument to the representation of any event. Consid er the representation of
(17.48)underthiskindofapproach.
∃eEating (e,Speaker ,TurkeySandwich ,Lunch,Desk) (17.50)
The variable enow gives us a handle on the event in question. If we need to mak e
additionalassertionsaboutthiseventwecandosoviathisv ariable. Forexample,ifwe

subsequently determine that this Eatingevent happened on a Tuesday we can assert
thatasfollows:
∃eEating (e,Speaker ,TurkeySandwich ,Lunch,Desk)∧Time(e,Tuesday ) (17.51)
Eventsrepresentedinthisfashionareoftenreferredtoas Davidsonian eventrepre- Davidsonian
sentationsafterthephilosopherDonaldDavidsonwhointro ducedthetechnique(David-
son,1967).
This approach still leaves us with the problem of deciding a ﬁ xed set of semantic
roles for each predicate and then capturing other ancillary facts with additional pred-

ications. For example, in (17.51) we captured the location o f the event as the fourth
argument to the Eatingpredicate, while we captured the time with the Timerelation.
We can eliminate this dichotomyby capturing all the event ar gumentswith additional
∃eEating (e)∧Eater(e,Speaker )∧Eaten(e,TurkeySandwich ) (17.52)
∧Meal(e,Lunch )∧Location (e,Desk)∧Time(e,Tuesday )
This style of representation distills the representation o f events down to a single
argument that stands for the event itself. Everything else i s captured via additional

predications. Representationsofthissortaretypicallyr eferredtoas Neo-Davidsonian Neo-Davidsonian
eventrepresentations(Parsons,1990). Tosummarize,inth eneo-Davidsonianapproach
toeventrepresentations:
•There is no need to specify a ﬁxed number of arguments for a giv en surface
predicate;ratherasmanyrolesandﬁllerscanbegluedonasa ppearintheinput.
•Nomorerolesarepostulatedthanarementionedintheinput.
•Thelogicalconnectionsamongcloselyrelatedexamplesare satisﬁedwithoutthe
needformeaningpostulates.
17.4.1 Representing Time

Inourdiscussionofevents,wedidnotseriouslyaddressthe issueofcapturingthetime
whentherepresentedeventsaresupposedtohaveoccurred. T herepresentationofsuch

DRAFTSection17.4. RepresentingEventsandStates 573
information in a useful form is the domain of temporal logic . This discussion will Temporal logic
serve to introduce the most basic concerns of temporal logic along with a brief dis-
cussion of the meansby which humanlanguagesconveytempora linformation,which
amongotherthingsincludes tenselogic ,thewaysthatverbtensesconveytemporalin- Tense logic
formation. A more detailed discussion of robust approaches to the representation and
analysisoftemporalexpressionsispresentedinCh. 22.
Themost straightforwardtheoryoftime holdthat it ﬂowsine xorablyforward,and
thateventsareassociatedwitheitherpointsorintervalsi ntime,asonatimeline. Given
these notions, an ordering can be imposed on distinct events by situating them on the
timeline. More speciﬁcally, we can say that one event precedes another, if the ﬂow
of time leads from the ﬁrst event to the second. Accompanying these notions in most
theories is the idea of the current moment in time. Combining this notion with the

ideaofatemporalorderingrelationshipyieldsthefamilia rnotionsofpast,presentand
future.
Not surprisingly,there are a largenumberof schemesfor rep resentingthis kind of
temporalinformation. Theonepresentedhereisafairlysim pleonethatstayswithinthe
FOLframework of reiﬁed events that we have been pursuing. Consi der the following
examples:
(17.53)IarrivedinNewYork.
(17.54)IamarrivinginNew York.
(17.55)Iwill arriveinNew York.
These sentencesall referto the same kind of eventand differ solely in the tense of the

future.
examples:
verb. Inourcurrentschemeforrepresentingevents,allthr eewouldsharethefollowing
kindofrepresentation,whichlacksanytemporalinformati on:
∃eArriving (e)∧Arriver (e,Speaker )∧Destination (e,NewYork ) (17.56)
The temporal information provided by the tense of the verbs c an be exploited by
predicatingadditionalinformationaboutthe eventvariab lee. Speciﬁcally,we canadd
temporalvariablesrepresentingtheintervalcorrespondi ngtotheevent,theendpointof

theevent,andtemporalpredicatesrelatingthisendpointt othecurrenttimeasindicated
bythetenseoftheverb. Suchanapproachyieldsthefollowin grepresentationsforour
arrivingexamples:
∃e,i,n,t Arriving (e)∧Arriver (e,Speaker )∧Destination (e,NewYork ) (17.57)
∧IntervalOf (e,i)∧EndPoint (i,e)∧Precedes (e,Now)
∃e,i,n,t Arriving (e)∧Arriver (e,Speaker )∧Destination (e,NewYork ) (17.58)
∧IntervalOf (e,i)∧MemberOf (i,Now)
∃e,i,n,t Arriving (e)∧Arriver (e,Speaker )∧Destination (e,NewYork ) (17.59)
∧IntervalOf (e,i)∧EndPoint (e,n)∧Precedes (Now,e)

This representation introduces a variable to stand for the i nterval of time associated
with the event, and a variable that stands for the end of that i nterval. The two-place

DRAFT574 Chapter 17. RepresentingMeaning
predicate Precedes represents the notion that the ﬁrst time point argument prec edes
the second in time; the constant Nowrefers to the current time. For past events, the
endpointofthe intervalmustprecedethe currenttime. Simi larly,forfutureeventsthe
current time must precede the end of the event. For events hap pening in the present,
thecurrenttimeis containedwithintheeventinterval.
Unfortunately,the relation between simple verb tenses and points in time is by no
meansstraightforward. Considerthefollowingexamples:
(17.60)Ok,weﬂy fromSanFranciscotoBostonat 10.
(17.61)Flight1390will beatthe gateanhournow.
In the ﬁrst example, the present tense of the verb ﬂyis used to refer to a future event,
whilein thesecondthefuturetense isusedtoreferto apast e vent.
More complications occur when we consider some of the other v erb tenses. Con-
siderthefollowingexamples:
(17.62)Flight1902arrivedlate.
(17.63)Flight1902hadarrivedlate.

Although both refer to events in the past, representing them in the same way seems
wrong. Thesecondexampleseemstohaveanotherunnamedeven tlurkingintheback-
ground(e.g., Flight 1902had already arrivedlate whensomething else happened). To
accountforthisphenomena,Reichenbach(1947)introduced the notionofa reference
point. In our simple temporal scheme, the current moment in time is equated with Referencepoint
the time of the utterance, and is used as a reference point for when the event occurred

(before, at, or after). In Reichenbach’s approach, the noti on of the reference point is
separated out from the utterance time and the event time. The following examples
illustratethe basicsofthisapproach:
(17.64)WhenMary’sﬂightdeparted,Iate lunch.
(17.65)WhenMary’sﬂightdeparted,Ihadeatenlunch.
Inbothoftheseexamples,theeatingeventhashappenedinth epast,i.e. priortothe
utterance. However, the verb tense in the ﬁrst example indic ates that the eating event

beganwhentheﬂightdeparted,whilethesecondexampleindi catesthattheeatingwas
accomplished prior to the ﬂight’s departure. Therefore, in Reichenbach’s terms the
departure event speciﬁes the reference point. These facts can be accom modated by
asserting additional constraints relating the eatinganddeparture events. In the ﬁrst
example,thereferencepointprecedesthe eatingevent,andinthesecondexample,the
eating precedes the reference point. Figure 17.5 illustrat es Reichenbach’s approach

with the primary English tenses. Exercise 6 asks you to repre sent these examples in
FOL.
This discussion has focused narrowly on the broad notions of past, present, and
futureandhowtheyaresignaledbyvariousEnglishverbtens es. Of course,languages
alsohavemanyothermoredirectandmorespeciﬁcwaystoconv eytemporalinforma-
tion, including the use of a wide variety of temporal express ions as in the following
ATISexamples:
(17.66)I’dliketo goat6:45,in themorning.
(17.67)Somewherearoundnoon,please.

DRAFTSection17.4. RepresentingEventsandStates 575
Past Perfect Simple Past Present Perfect
Present Simple Future Future PerfectI had eaten I ate I have eaten
I eat I will eat I will have eatenU U
URE
E RE R,UR,E
U,R E U,R,E
Figure17.5 Reichenbach’s approach applied to various English tenses. In these diagrams,
timeﬂowsfromlefttoright,an Edenotesthetimeoftheevent,an Rdenotes thereference time,
and anUdenotes the time of the utterance.
(17.68)Laterintheafternoon,near6PM.
As we will see in Ch. 22,grammarsforsuch temporalexpressio nsare of considerable
practicalimportancetoinformationextractionandquesti on-answeringapplications.
Finally,we shouldnotethat there is a systematic conceptua lorganizationreﬂected
in examples like these. In particular, temporal expression s in English are frequently
expressedinspatialterms,asisillustratedbythevarious usesofat,in,somewhere and
nearin these examples(Lakoff and Johnson, 1980; Jackendoff, 19 83). Metaphorical

organizationssuch as these, whereone domainis systematic ally expressedin termsof
another,will bediscussedin moredetailinCh. 19.
17.4.2 Aspect
In the last section, we discussed waysto representthe time o f an eventwith respect to
the time of an utterancedescribingit. In thissection, we ad dressthe notionof aspect, Aspect
which concerns a cluster of related topics, including wheth er an event has ended or
is ongoing, whether it is conceptualized as happening at a po int in time or over some

interval,andwhetherornotanyparticularstateintheworl dcomesaboutbecauseofit.
Based on these and related notions, event expressions have t raditionally been divided
intofourgeneralclassesillustratedin thefollowingexam ples:
Stative: Iknowmydeparturegate.
Activity: Johnisﬂying.
Accomplishment: Sally bookedherﬂight.
Achievement: Shefoundhergate.
Althoughtheearliestversionsofthisclassiﬁcationwered iscussedbyAristotle,theone
presented here is due to Vendler (1967). In the following dis cussion, we’ll present a

briefcharacterizationofeachofthefourclasses,alongwi thsomediagnostictechniques
suggestedin Dowty(1979)foridentifyingexamplesofeachk ind.
Stativeexpressions representthenotionofaneventparticipanthavingapartic ularStativeexpressions

DRAFT576 Chapter 17. RepresentingMeaning
property,orbeinginastate,atagivenpointintime. Assuch ,theycanbethoughtofas
capturing an aspect of a world at a single point in time. Consi der the following ATIS
examples.
(17.69)IlikeFlight 840arrivingat 10:06.
(17.70)Ineedthecheapestfare.
(17.71)Ihavearoundtripticketfor$662.
(17.72)Iwantto goﬁrst class.
In examples like these, the event participant denoted by the subject can be seen as
experiencingsomethingataspeciﬁcpointintime. Whethero rnottheexperiencerwas
inthe samestate earlier,orwill beinthe futureisleftunsp eciﬁed.
There are a number of diagnostic tests for identifying stati ves. As an example,
stativeverbsaredistinctlyoddwhenusedin theprogressiv eform.
(17.73)*Iamneedingthe cheapestfareonthisday.
(17.74)*Iamwantingtogoﬁrst class.
We should note that in these and subsequent examples, we are u sing an * to indicate
a broadened notion of ill-formedness that may include both s emantic and syntactic
factors.
Stativesarealso oddwhenusedasimperatives.

factors.
(17.75)*Needthecheapestfare!
Finally,stativesare noteasily modiﬁedbyadverbslike deliberately andcarefully.
(17.76)*Ideliberatelylike Flight840arrivingat10:06.
(17.77)*Icarefullylike Flight840arrivingat10:06.
Activityexpressions describeeventsundertakenbya participantthat havenopar -Activityexpressions
ticular end point. Unlike statives, activities are seen as o ccurring over some span of
time, and are thereforenot associated with single points in time. Consider the follow-
ingexamples:
(17.78)Shedrovea Mazda.

(17.79)IliveinBrooklyn.
These examples both specify that the subject is engaged in, o r has engaged in, the
activityspeciﬁedbythe verbforsomeperiodoftime.
Unlikestatives,activityexpressionsareﬁneinboththepr ogressiveandimperative
forms.
(17.80)Sheislivingin Brooklyn.
(17.81)Drivea Mazda!
However,likestatives,activityexpressionsareoddwhent emporallymodiﬁedwith
temporalexpressionsusing in.
(17.82)*Ilivein Brooklynin amonth.
(17.83)*ShedroveaMazdainanhour.
Theycan,however,successfullybeusedwith fortemporaladverbials,asinthefollow-

forms.

DRAFTSection17.4. RepresentingEventsandStates 577
(17.84)IliveinBrooklynfora month.
(17.85)Shedrovea Mazdaforanhour.
Unlikeactivities, accomplishmentexpressions describeeventsthathavea naturalAccomplishmentexpressions
endpointandresultina particularstate. Considerthefoll owingexamples:
(17.86)Hebookedmeareservation.
(17.87)Unitedﬂew meto NewYork.
In these examples,there is an event that is seen as occurring oversome periodof time
thatendswhentheintendedstate isaccomplished.
A number of diagnostics can be used to distinguish accomplis hment events from
activities. Considerthefollowingexamples,whichmakeus eoftheword stopasatest.
(17.88)Istoppedlivingin Brooklyn.
(17.89)Shestoppedbookingmyﬂight.
In the ﬁrst example, which is an activity, one can safely conc lude that the statement I
lived in Brooklyn even thoughthis activity came to an end. However,fromthe se cond
example, one can not conclude the statement She booked her ﬂight , since the activity

was stopped before the intended state was accomplished. The refore, although stop-
ping an activity entails that the activity took place, stopp ing an accomplishmentevent
indicatesthattheeventdidnotsucceed.
Activitiesandaccomplishmentscanalsobedistinguishedb yhowtheycanbemod-
iﬁedbyvarioustemporaladverbials. Considerthefollowin gexamples:
(17.90)*IlivedinBrooklynina year.
(17.91)Shebookeda ﬂightina minute.
Ingeneral,accomplishmentscanbemodiﬁedby intemporalexpressions,whilesimple
activitiescannot.

The ﬁnal aspectual class, achievement expressions , are similar to accomplish-Achievementexpressions
mentsinthat theyresultin astate. Considerthefollowinge xamples:
(17.92)Shefoundhergate.
(17.93)IreachedNewYork.
Unlikeaccomplishments,achievementeventsarethoughtof ashappeninginaninstant,
andarenotequatedwithanyparticularactivityleadingupt othestate. Tobemorespe-
ciﬁc, the events in these examples may have been preceded by e xtendedsearching or

traveling events,buttheeventscorrespondingdirectlyto foundandreachareconceived
ofaspointsnotintervals.
The point-like nature of these events has implicationsfor h ow they can be tempo-
rallymodiﬁed. Inparticular,considerthefollowingexamp les:
(17.94)Ilivedin NewYorkfora year.
(17.95)*IreachedNew Yorkforafewminutes.
Unlikeactivityandaccomplishmentexpressions,achievem entscannotbemodiﬁedby
foradverbials.
Achievements can also be distinguished from accomplishmen ts by employing the
wordstop,aswe didearlier. Considerthefollowingexamples:

DRAFT578 Chapter 17. RepresentingMeaning
(17.96)Istoppedbookingmyﬂight.
(17.97)*IstoppedreachingNew York.
Aswesawearlier,using stopwithanaccomplishmentexpressionresultsinafailureto
reachtheintendedstate. Note,however,thattheresulting expressionisperfectlywell-
formed. Onthe otherhand,using stopwithanachievementexampleisunacceptable.
We should note that since both accomplishmentsand achievem entsare eventsthat
result in a state, they are sometimes characterized as sub-t ypes of a single aspectual
class. Membersofthiscombinedclassareknownas teliceventualities . Telic eventualities
Before moving on, we should make two points about this classi ﬁcation scheme.
Theﬁrstpointisthateventexpressionscaneasilybeshifte dfromoneclasstoanother.
Considerthefollowingexamples:
(17.98)Iﬂew.
(17.99)Iﬂew toNewYork.
The ﬁrst example is a simple activity; it has no natural end po int and can not be tem-
porallymodiﬁedby intemporalexpressions. Ontheotherhand,thesecondexample is

clearly an accomplishmentevent since it has an end point, re sults in a particularstate,
and can be temporallymodiﬁed in all the ways that accomplish mentscan. Clearly the
classiﬁcationofaneventisnotsolelygovernedbytheverb, butbythesemanticsofthe
entireexpressionin context.
Thesecondpointisthatwhileclassiﬁcationssuchasthison eareoftenuseful,they
donotexplainwhyitisthateventsexpressedinnaturallanguagesfallint othesepartic-
ularclasses. WewillrevisitthisissueinCh.19wherewewil lsketcharepresentational

approachduetoDowty(1979)thataccountsfortheseclasses .
17.5 RelatedRepresentationalApproaches
Overtheyears,afairnumberofrepresentationalschemesha vebeeninventedtocapture
the meaning of linguistic utterances for use in natural lang uage processing systems.
Other than FOL, the most widely used schemes have been semantic networks andSemanticnetworks
frames,whicharealsosometimescalled slot-ﬁller representations. Frames
In semantic networks, objects are represented as nodes in a g raph, with relations

between objects being represented by named links. In frame- based systems, objects
are represented as feature-structures similar to those dis cussed in Ch. 16, which can,
ofcourse,also benaturallyrepresentedasgraphs. Inthisa pproach,featuresarecalled
slots and the values, or ﬁllers, of these slots can either be a tomic values or other em-
beddedframes. Thefollowingdiagramillustrateshowthefo llowingexamplemightbe
capturedina frame-basedapproach.
(17.100) I believeMaryate British food.

DRAFTSection17.5. Related RepresentationalApproaches 579

BELIEVING
BELIEVER SPEAKER
BELIEVED
EATING
EATER MARY
EATEN BRITISHFOOD


It is now widely accepted that meanings represented in these approaches can in
principle be translated into equivalent statements in FOLwith relative ease. The dif-
ﬁculty is that in many of these approaches the semantics of a s tatement is deﬁned
procedurally. That is, the meaning arises from whatever the system that interprets it
does with it. The following section describes a more princip led approach that brings
togethertheformalpowerof FOLwiththeintuitiveappealofsemanticnetworks.
17.5.1 DescriptionLogics
Description Logics can be viewed as an effort to better under stand and specify the
semantics of these earlier structured network representat ions, and to provide a con-
ceptual framework that is especially well-suited to certai n kinds of domain modeling.
Formally, the term Description Logics refers to a family of l ogical approaches that

correspond to varying subsets of FOL. The various restrictions placed on the expres-
siveness of Description Logics serve to guarantee the tract ability of various critical
kinds of inference. Our focus here, however, will be on the mo deling aspects of DLs
ratherthancomputationalcomplexityissues.
When using Description Logics to model an application domai n, the emphasis is
on the representation of knowledge about categories, indiv iduals that belong to those
categories, and the relationships that can hold among these individuals. The set of

categories, or concepts, that make up the particular applic ation domain is called its
Terminology . Theportionofa knowledge-basethatcontainsthe terminol ogyistradi- Terminology
tionally called the TBox; this is in contrastto the ABoxthat containsfacts aboutindi- TBox
ABoxviduals. The terminology is typically arranged into a hiera rchical organization called
anOntology that capturesthesubset/supersetrelationsamongthecate gories. Ontology
To illustrate this approach, let’s return to our earlier cul inary domain, which in-

cluded notions like restaurants, cuisines, and patrons, am ong others. We represented
concepts like these in FOLby using unary predicates such as Restaurant (x); the DL
equivalent simply omits the variable, so the category corre sponding to the notion of
a restaurant is simply written as Restaurant .3To capture the notion that a particular
domainelement,suchas Frasca,isarestaurantwesimplyassert Restaurant(Frasca)
in much the same way we would in FOL. The semantics of these categories is speci-

ﬁed in preciselythe same way that was introducedearlier in S ec. 17.2: a categorylike
Restaurant simplydenotestheset ofdomainelementsthatarerestauran ts.
Having speciﬁed the categories of interest in a state of affa irs, the next step is to
arrangethesecategoriesintoahierarchicalstructure. Th erearetwowaystocapturethe
hierarchicalrelationshipspresentinaterminology: weca ndirectlyassert relationsbe-
3DL statements are conventionally typeset with a sans serif font. We’ll follow that convention here,

reverting back to our standard mathematical notation when g ivingFOLequivalents of DLstatements.

DRAFT580 Chapter 17. RepresentingMeaning
tweencategoriesthatarerelatedhierarchically,orwecan providecompletedeﬁnitions
for our concepts and then rely on these deﬁnitions to infer hi erarchical relationships.
The choicebetween these methodshingeson the use to whichth e resultingcategories
will beputandthefeasibilityofformulatingprecisedeﬁni tionsformanynaturallyoc-
curring categories. We’ll discuss the ﬁrst option here, and the return to the notion of
deﬁnitionslater inthissection.
To directly specify a hierarchical structure, we can assert subsumption relations Subsumption
between the appropriate concepts in a terminology. The subs umption relation is con-
ventionallywritten as C⊑D, and is read as Cis subsumed by D; that is, all members
of the category Care also members of the category D. Not surprisingly, the formal
semanticsofthisrelationisprovidedbyasimplesetrelati on;anydomainelementthat
isinthe set denotedby Cisalsoin theset denotedby D.

Continuingwithourrestauranttheme,addingthefollowing statementstotheTBox
asserts that all restaurantsare commercialestablishment s, and moreoverthat there are
varioussub-typesofrestaurants.
Restaurant⊑CommercialEstablishment (17.101)
ItalianRestaurant ⊑Restaurant (17.102)
ChineseRestaurant ⊑Restaurant (17.103)
MexicanRestaurant ⊑Restaurant (17.104)
Ontologies such as this are conventionally illustrated usi ng diagrams such as the one
showninFig.17.6wheresubsumptionrelationsaredenotedb ylinksbetweenthenodes

representingthecategories.
Commercial
Establishment
Restaurant
Chinese
Restaurant Mexican
RestaurantItalian
Figure17.6 Agraphicalnetworkrepresentationofasetofsubsumptionr elationsintherestau-
rant domain.
Note however that it was precisely the vague nature of networ k diagrams like this

DRAFTSection17.5. Related RepresentationalApproaches 581
thatmotivatedthedevelopmentofDescriptionLogics. Fore xample,fromthisdiagram
we can’t tell whether or not the given set of categories is exh austive or disjoint. That
is,we can’ttell iftheseareallthekindsofrestaurantstha twe’ll bedealingwithinour
domain, or whether there might be others. We also can’t tell i f an individual restau-
rant must fall into only oneof these categories, or if it is possible, for example, for a
restaurant to be bothItalian and Chinese. The DL statements given above are more
transparentintheirmeaning;theysimplyassertasetofsub sumptionrelationsbetween
categoriesandmakenoclaimsaboutcoverageormutualexclu sion.
Ifanapplicationrequirescoverageanddisjointnessinfor mationthenit needstobe
made explicitly. The simplest ways to capture this kind of in formation is through the
use of negationand disjunctionoperators. Forexample,the followingassertion would
tell usthat Chineserestaurantscan’talsobeItalianresta urants.

ChineseRestaurant ⊑notItalianRestaurant (17.105)
Specifying that a set of sub-conceptscovers a category can b e achieved with disjunc-
tion,asinthefollowing:
Restaurant⊑ (17.106)
(orItalianRestaurant ChineseRestaurant MexicanRestaurant )
Of course, having a hierarchy such as the one given in Fig. 17. 6 tells us next to
nothingabout the conceptsin it. We certainly don’tknow any thingabout what makes
a restaurant a restaurant, much less Italian, Chinese or exp ensive. What is needed are
additional assertions about what it means to be a member of an y of these categories.

In DescriptionLogics such statements come in the form of rel ationsbetween the con-
cepts being describedand other conceptsin the domain. In ke eping with its originsin
structurednetworkrepresentations,relationsinDescrip tionLogicsaretypicallybinary
andareoftenreferredtoasroles,orrole-relations.
To see how such relations work, let’s consider some of the fac ts about restaurants
discussed earlier in the chapter. We’ll use the hasCuisine relation to capture infor-
mationasto whatkindsof foodrestaurantsserve,andthe hasPriceRange relationto

capture how pricey particularrestaurantstend to be. We can use these relations to say
something more concrete about our variousclasses of restau rants. Let’s start with our
ItalianRestaurant concept. Asa ﬁrst approximation,we mightsay somethingunc on-
troversial like Italian restaurants serve Italian cuisine . To capture these notions, let’s
ﬁrst add some new concepts to our terminology to represent va rious kinds of cuisine.
MexicanCuisine⊑Cuisine
ItalianCuisine⊑Cuisine
ChineseCuisine⊑Cuisine
VegetarianCuisine ⊑CuisineExpensiveRestaurant ⊑Restaurant

ModerateRestaurant ⊑Restaurant
CheapRestaurant ⊑Restaurant
Next let’s revise our earlier version of ItalianRestaurant to capture cuisine infor-
mation.
ItalianRestaurant ⊑Restaurant⊓∃hasCuisine .ItalianCuisine (17.107)

DRAFT582 Chapter 17. RepresentingMeaning
The correct way to read this expression is that individuals i n the category Italian-
Restaurant aresubsumedbothbythecategory Restaurant ,andbyanunnamedclass
deﬁned by the existential clause — the set of entities that se rve Italian cuisine. An
equivalentstatementin FOLwouldbe:
∀xItalianRestaurant (x)→Restaurant (x) (17.108)
∧(∃yServes (x,y)∧ItalianCuisine (y))
ThisFOLtranslation should make it clear what the DL assertions give n abovedo, and
do not entail. In particular, they don’t say that domain enti ties classiﬁed as Italian
restaurantscan’tengageinotherrelationslikebeingexpe nsive,orevenservingChinese
cuisine. And critically, they don’t say much about domain en tities that we know do
serveItaliancuisine. Infact,inspectionofthe FOLtranslationthemakesitclearthatwe
can’tinferthat any new entities belong to this category based on their c haracteristics.
The best we can do is infer new facts about restaurants that we ’re explicitly told are
membersofthiscategory

Of course, inferring the category membership of individual s given certain charac-
teristicsisacommonandcriticalreasoningtaskthatwenee dtosupport. Thisbringsus
backtothealternativeapproachtocreatinghierarchicals tructuresinaterminology: ac-
tually providinga deﬁnition of the categorieswe’re creating in the form of necessary
and sufﬁcient conditions for category membership. In this c ase, we might explicitly
providea deﬁnitionfor ItalianRestaurant as beingthoserestaurantsthat serveItalian
cuisine,and ModerateRestaurant asbeingthosewhosepricerangeismoderate.

ItalianRestaurant ≡Restaurant⊓∃hasCuisine .ItalianCuisine (17.109)
ModerateRestaurant ≡Restaurant⊓hasPriceRange .ModeratePrices (17.110)
While our earlier statements provided necessary condition s for membership in these
categories,thesestatementsprovidebothnecessaryandsu fﬁcientconditions.
Finally, let’s now consider the superﬁcially similar case o f vegetarian restaurants.
Clearly, vegetarian restaurants are those that serve veget arian cuisine. But they don’t

merely serve vegetarian fare, that’s all they serve. We can a ccommodate this kind of
constraintbyaddinganadditionalrestrictionintheformo fauniversalquantiﬁertoour
earlierdescriptionof VegetarianRestaurants , asfollows:
VegetarianRestaurant ≡Restaurant (17.111)
⊓∃hasCuisine .VegetarianCuisine
⊓∀hasCuisine .VegetarianCuisine
Inference
Paralleling the focus of Description Logics on categories, relations and individuals, is
a processing focus on a restricted subset of logical inferen ce. Rather than employ the

Inference
fullrangeofreasoningpermittedby FOL,DLreasoningsystemsemphasizetheclosely
coupledproblemsofsubsumptionandinstancechecking.
Subsumption , as a form of inference, is the task of determining whether a s uper- Subsumption

DRAFTSection17.5. Related RepresentationalApproaches 583
set/subsetrelationshipexistsbetweentwoconceptsbased onthefactsassertedinater-
minology. Correspondingly, instance checking asks if an individualcan be a member Instance checking
of a particularcategorygiventhe facts we knowabout both th e individualand the ter-
minology. The inference mechanisms underlying subsumptio n and instance checking
go beyond simply checking for explicitly stated subsumptio n relations in a terminol-
ogy. They must explicitly reason using the relational infor mation asserted about the
terminologytoinferappropriatesubsumptionandmembersh iprelations.
Returning to our restaurant domain, let’s add a new kind of re staurant using the
followingstatement:
OliveGarden⊑ModerateRestaurant ⊓∃hasCuisine .ItalianCuisine (17.112)
Giventhisassertion,wemightaskwhetherthe OliveGarden chainofrestaurantsmight
beclassiﬁedasanItalianrestaurant,oravegetarianresta urant. Moreprecisely,wecan
posethefollowingquestionstoourreasoningsystem:

OliveGarden⊑ItalianRestaurant (17.113)
OliveGarden⊑VegetarianRestaurant (17.114)
The answer to the ﬁrst question is positive since OliveGarden meets the criteria
wespeciﬁedforthecategory ItalianRestaurant : it’saRestaurant sinceweexplicitly
classiﬁeditasa ModerateRestaurant ,whichisasubtypeof Restaurant ,anditmeets
thehas.Cuisine class restrictionsincewe’veassertedthatdirectly.
Theanswertothesecondquestionis negative. Recall, thato urcriteriaforvegetar-
ianrestaurantscontainstwo requirements: it hastoservev egetarianfare,andthat’sall

itcanserve. Ourcurrentdeﬁnitionfor OliveGarden failsonbothcountssincewehave
not asserted any relations that state that OliveGarden serves vegetarian fare, and the
relationwe haveasserted, hasCuisine.ItalianCuisine , contradictsthesecondcriteria.
A related reasoning task, based on the basic subsumption inf erence, is to derive
theimplied hierarchy for a terminologygivenfacts about the categoriesin the ter mi-
nology. This task roughly corresponds to a repeated applica tion of the subsumption

operatorto pairs of conceptsin the terminology. Givenour c urrentcollectionof state-
ments,theexpandedhierarchyshowninFig.17.7canbeinfer red. Youshouldconvince
yourself that this diagram contains all and only the subsump tion links that should be
presentgivenourcurrentknowledge.
Notethatwhereassubsumptionisallaboutconceptsandcate gories,instancecheck-
ingis the task of determining whether a particular individual c an be classiﬁed as a
memberof a particularcategory. Thisprocesstakeswhat isk nownabouta givenindi-

vidual, in the form of relationsand explicit categoricalst atements, and then compares
thatinformationagainstwhatisknownaboutthecurrentter minology. Itthenreturnsa
list ofthemost speciﬁc categoriestowhichtheindividualcanbelong.
As an example of a categorization problem consider an establ ishment that we’re
toldisa restaurantandservesItaliancuisine.
Restaurant (Gondolier ) (17.115)
hasCuisine (Gondolier ,ItalianCuisine ) (17.116)

DRAFT584 Chapter 17. RepresentingMeaning
Chinese
RestaurantExpensive
RestaurantCheap
RestaurantModerate
OliveGardenVegetarian
Figure17.7 A graphical network representation of the complete set of su bsumption relations
inthe restaurant domain given the current set of assertions inthe TBox.
Here,we’rebeingtoldthattheentitydenotedbytheterm Gondolier isarestaurantand
serves Italian food. Given this new informationand the cont ents of our currentTBox,
we might reasonably like to ask if this is an Italian restaura nt, a vegetarian restaurant
orifit hasmoderateprices.
Assuming the deﬁnitional statements given earlier, we can i ndeed categorize the
Gondolier as an Italian restaurant. That is, the informatio n we’ve been given about it
meetsthenecessaryandsufﬁcientconditionsrequiredform embershipinthiscategory.
Andaswiththe OliveGarden category,thisindividualfailstomatchthestatedcriteri a

for theVegetarianRestaurant . Finally, the Gondolier might also turn out to be an
moderately priced restaurant, but we can’t tell at this poin t since we don’t know any-
thingaboutits prices. What thismeansisthatgivenourcurr entknowledgetheanswer
tothequery ModerateRestaurant (Gondolier )wouldbefalsesinceitlackstherequired
hasPriceRange relation.
The implementation of subsumption, instance checking, as w ell as other kinds of
inferences needed for practical applications, varies depe nding on the expressivity of

the Description Logic being used. However, for Description Logics of even modest
power,the primaryimplementationtechniquesare based ons atisﬁability methodsthat
inturnrelyontheunderlyingmodel-basedsemanticsintrod ucedearlierinthischapter.
OWLandthe SemanticWeb
The highest-proﬁle role for Description Logics has been as a part of the development
of the Semantic Web. The Semantic Web is an ongoing effort to p rovide a way to
formally speciﬁc the semantics of the contents of the Web (Fe nsel et al., 2003). A

key component of this effort involves the creation and deplo yment of ontologies for
various application areas of interest. The meaning represe ntation language used to
represent this knowledge is the Web Ontology Language (OWL) (McGuiness andWeb Ontology
Language
vanHarmelen,2004). OWLembodiesaDescriptionLogicthatc orrespondsroughlyto

DRAFTSection17.6. AlternativeApproachestoMeaning 585
theonewe’vebeendescribinghere.
17.6 AlternativeApproachesto Meaning
The idea that the translation of linguistic inputs into a for mal representation made up
of discrete symbols adequately captures the notion of meani ng is, not surprisingly,
subject to a considerable amount of debate. The following se ction give brief, wholly
inadequate,overviewsofsomeofthe majorconcernsinthese debates.
17.6.1 Meaning asAction
Anapproachthatholdsconsiderableappealwhenweconsider thesemanticsofimper-
ative sentences is the notion of meaning as action . Under this view, utterances are Meaningas action
viewed as actions, and the meanings of these utterances resi de inprocedures that are
activatedinthehearerasaresultofhearingtheutterance. Thisapproachwasfollowed
in the creation of the historically important SHRDLU system, and is summed up well
byitscreatorTerryWinograd(1972b).
One of the basic viewpointsunderlyingthe model is that all l anguageuse

canbe thoughtofas a way of activatingprocedureswithinthe hearer. We
canthinkofanutteranceasaprogram—onethatindirectlyca usesaset of
operationsto becarriedoutwithinthehearer’scognitives ystem.
Amorerecentproceduralmodelofsemanticsisthe executingschema orx-schema X-schema
modelofBaileyetal.(1997),Narayanan(1997a,1997b),and Changetal.(1998). The
intuition of this model is that various parts of the semantic s of events, including the
aspectual factors discussed on page 575, are based on schematized desc riptions of

sensory-motorprocesses like inception, iteration, enabl ing, completion, force, and ef-
fort. Themodelrepresentstheaspectualsemanticsofevent sviaakindofprobabilistic
automaton called a Petri net (Murata, 1989). The nets used in the model have states
likeready,process,ﬁnish,suspend,andresult.
Themeaningrepresentationofanexamplelike Jackiswalkingtothestore activates
theprocessstate of the walking event. An accomplishment event like Jack walked to
the store activates the resultstate. An iterative activity like Jack walked to the store

every week is simulated in the model by an iterative activation of the processandre-
sultnodes. This idea of using sensory-motor primitives as a foun dation for semantic
descriptionis also based on the work of Regier (1996)on the r ole of visual primitives
ina computationalmodeloflearningthesemanticsofspatia l prepositions.
17.7 Summary
This chapter has introduced the representational approach to meaning. The following
aresomeofthehighlightsofthischapter:

DRAFT586 Chapter 17. RepresentingMeaning
•A major approachto meaning in computationallinguistics in volvesthe creation
offormal meaning representations that capture the meaning-related content
of linguistic inputs. These representations are intended t o bridge the gap from
languagetocommonsenseknowledgeofthe world.
•The frameworks that specify the syntax and semantics of thes e representations
are called meaning representation languages . A wide variety of such lan-
guagesareusedinnaturallanguageprocessingandartiﬁcia lintelligence.
•Such representations need to be able to support the practica l computational re-
quirementsofsemanticprocessing. Amongthesearetheneed todetermine the
truth of propositions , to support unambiguous representations , to represent
variables ,tosupport inference ,andto besufﬁciently expressive .
•Humanlanguageshavea wide varietyof featuresthat are used to conveymean-
ing. Among the most important of these is the ability to conve y apredicate-
argumentstructure .

•First-OrderLogic isawell-understoodcomputationallytractablemeaningre p-
resentation languagethat offersmuch of what is needed in a m eaningrepresen-
tationlanguage.
•Important elements of semantic representation including statesandeventscan
becapturedin FOL.
•Semanticnetworks andframescanbecapturedwithinthe FOLframework.
•Modern bf Description Logics consist of useful and computat ionally tractable
subsetsoffullﬁrst-orderlogic. Themostprominentuse ofa descriptionlogicis
theWeb OntologyLanguage (OWL), usedin the speciﬁcation ofthe Semantic
Web.

Web.
The earliest computational use of declarative meaning repr esentations in natural lan-
guageprocessingwasinthecontextofquestion-answerings ystems(Greenetal.,1961;
Raphael, 1968; Lindsey, 1963). These systems employed ad-h oc representations for
the facts needed to answer questions. Questions were then tr anslated into a form that
could be matched against facts in the knowledge base. Simmon s (1965) provides an
overviewoftheseearlyefforts.

Web.
Woods(1967)investigatedthe use of FOL-like representationsinquestionanswer-
ing as a replacement for the ad-hoc representations in use at the time. Woods (1973)
further developed and extended these ideas in the landmark L unar system. Interest-
ingly, the representationsused in Lunar had both a truth-co nditionaland a procedural
semantics. Winograd (1972b) employed a similar representa tion based on the Micro-
Plannerlanguageinhis SHRDLUsystem.
During this same period, researchers interested in the cogn itive modeling of lan-

guage and memory had been working with various forms of assoc iative network rep-
resentations. Masterman (1957) was the ﬁrst to make computa tional use of a seman-

DRAFTSection17.7. Summary 587
tic network-like knowledge representation, although sema ntic networks are generally
credited to Quillian (1968). A considerable amount of work i n the semantic network
framework was carried out during this era (Norman and Rumelh art, 1975; Schank,
1972;Wilks, 1975c,1975b;Kintsch,1974). It wasduringthi speriodthat anumberof
researchers began to incorporate Fillmore’s notion of case roles (Fillmore, 1968) into
their representations. Simmons(1973)was the earliest ado pterof case roles as part of
representationsfornaturallanguageprocessing.
Detailed analyses by Woods (1975) and Brachman (1979) aimed at ﬁguring out
whatsemanticnetworksactuallymeanledtothedevelopment ofanumberofmoreso-
phisticated network-like languages including KRL(Bobrow and Winograd, 1977) and
KL-ONE(BrachmanandSchmolze,1985). Astheseframeworksbecamem oresophisti-
catedandwell-deﬁneditbecameclearthattheywererestric tedvariantsof FOLcoupled

withspecializedindexinginferenceprocedures. Ausefulc ollectionofpaperscovering
muchofthisworkcanbefoundinBrachmanandLevesque(1985) . RussellandNorvig
(1995)describea modernperspectiveontheserepresentati onalefforts.
Linguistic efforts to assign semantic structures to natura l language sentences in
the generative era began with the work of Katz and Fodor (1963 ). The limitations
of their simple feature-based representations and the natu ral ﬁt of logic to many of

linguistic problems of the day quickly led to the adoption of a variety of predicate-
argument structures as preferred semantic representation s (Lakoff, 1972; McCawley,
1968). The subsequent introduction by Montague (1973) of tr uth-conditional model-
theoretic framework into linguistic theory led to a much tig hter integration between
theories of formal syntax and a wide range of formal semantic frameworks. Good
introductions to Montague semantics and its role in linguis tic theory can be found in
Dowtyetal. (1981),Partee(1976).

The representation of events as reiﬁed objects is due to Davi dson (1967) The ap-
proach presented here, which explicitly reiﬁes event parti cipants, is due to Parsons
(1990). The use of modal operatorsand in the representation of knowledgeand belief
is due to Hintikka (1969). Moore (1977) was the ﬁrst to make co mputational use of
this approach. Fauconnier (1985) deals with a wide range of i ssues relating to beliefs
and belief spaces from a cognitive science perspective. Mos t current computational

approaches to temporal reasoning are based on Allen’s notio n of temporal intervals
(Allen, 1984). ter Meulen (1995) provides a modern treatmen t of tense and aspect.
Davis (1990) describes the use of FOLto represent knowledge across a wide range of
commonsense domainsincludingquantities,space,time,an dbeliefs.
A recentcomprehensivetreatmentoflogic andlanguagecan b e foundin vanBen-
them and ter Meulen (1997). The classic semantics text is Lyo ns (1977). McCawley

(1993) is an indispensable textbook covering a wide range of topics concerning logic
and language. Chierchia and McConnell-Ginet (1991) also pr ovides broad coverage
of semantic issues from a linguistic perspective. Heim and K ratzer (1998) is a more
recenttextwrittenfromtheperspectiveofcurrentgenerat ivetheory.

DRAFT588 Chapter 17. RepresentingMeaning
Exercises
17.1Peruseyourdailynewspaperforthreeexamplesofambiguous sentencesorhead-
lines. Describethevarioussourcesoftheambiguities.
17.2Considera domainwhere the word coffeecan referto the followingconceptsin
aknowledge-based: acaffeinatedordecaffeinatedbeverag e,groundcoffeeused
tomakeeitherkindofbeverage,andthebeansthemselves. Gi veargumentsasto
whichofthefollowingusesofcoffeeareambiguousandwhich arevague.
a. I’vehadmycoffeefortoday.
b. Buysomecoffeeonyourwayhome.
c. Please grindsomemorecoffee.
17.3The following rule, which we gave as a translation for Exampl e 17.26, is not a
reasonabledeﬁnitionofwhatit meansto beavegetarianrest aurant.
Give aFOLrule that better deﬁnes vegetarian restaurants in terms of w hat they
serve.
17.4GiveaFOLtranslationsforthefollowingsentences:
a. Vegetariansdonoteatmeat.
b. Notall vegetarianseateggs.
17.5Givea set offactsandinferencesnecessarytoprovethefoll owingassertions:

serve.
a. McDonaldsisnota vegetarianrestaurant.
b. Somevegetarianscaneat at McDonalds.
Don’t just place these facts in your knowledge base. Show tha t they can be
inferredfromsomemoregeneralfactsaboutvegetariansand McDonalds.
17.6GiveFOLtranslationsforthefollowingsentencesthatcapturethet emporalrela-
tionshipsbetweentheevents.
a. WhenMary’sﬂightdeparted,Iate lunch.
b. WhenMary’sﬂightdeparted,Ihadeatenlunch.
17.7Givea reasonable FOLtranslationofthe followingexample.
Ifyou’reinterestedinbaseball,theRockiesare playingto night.

17.8On Page 567, we gave the following representation as a transl ation for the sen-
tenceCentro isnearBacaro .
Near(Centro ,Bacaro )
In our truth-conditional semantics, this formula is either true or false given the
contentsof someknowledge-base. Critique thistruth-cond itionalapproachwith
respectto themeaningofwordslike near.

Chapter18
Computational Semantics
“Thenyou should say what you mean,”the March Hare went on.
“I do,” Alice hastily replied; “at least–at least I mean what I say–that’s the same thing,
youknow.”
“Notthe samethingabit!” saidthe Hatter. “Youmightjustas wellsaythat ‘Isee whatI
eat’is the same thingas ‘Ieat what I see’!”
LewisCarroll, AliceinWonderland
This chapter presents a principled computational approach to the problem of seman-
tic analysis the process wherebymeaningrepresentationsof the kind dis cussed in the Semantic analysis
last chapter are composed for linguistic expressions. The a utomated creation of ac-

youknow.”
curate and expressive meaning representations necessaril y involves a wide range of
knowledge-sources and inference techniques. Among the sou rces of knowledge that
are typically involved are the meanings of words, the conven tional meanings associ-
ated with grammatical constructions, knowledge about the s tructure of the discourse,
common-sense knowledge about the topic at hand and knowledg e about the state of
affairsinwhichthediscourseis occurring.
The focus of this chapter is a kind of syntax-driven semantic analysis that isSyntax-driven
semantic analysis

fairly modest in its scope. In this approach, meaning repres entations are assigned to
sentencesbasedsolelyonknowledgegleanedfromthelexico nandthegrammar. When
we refer to an expression’s meaning, or meaning representat ion, we have in mind a
representationthat is both context independentand free of inference. Representations
of this type correspond to the traditional notion of literal meaning discussed in the
There are two motivations for proceeding along these lines: there are application

domains, including question answering, where such primiti ve representationsare suf-
ﬁcient to produce useful results, and these impoverished re presentations can serve as
usefulinputstosubsequentprocessesthatcanproducerich er,morecomplete,meaning
representations. Chs.21and24willdiscusshowthesemeani ngrepresentationscanbe
usedinprocessingextendeddiscoursesanddialogs.
18.1 Syntax-DrivenSemanticAnalysis
The approach detailed in this section is based on the principle of compositionality .Principleof
compositionality
Thekeyideabehindthisapproachisthatthemeaningofasent encecanbeconstructed

from the meanings of its parts. When interpreted superﬁcial ly this principle is some-
what lessthanuseful. We knowthat sentencesarecomposedof words, andthatwords

DRAFT590 Chapter 18. ComputationalSemantics
Syntactic Analysis Semantic Analysis Inputs
Syntactic Structures
Meaning
Representations
Figure18.1 A simple pipeline approach tosemantic analysis.
are the primary carriersof meaning in language. It would see m then that all this prin-
cipletellsusisthatweshouldcomposethemeaningrepresen tationforsentencesfrom
themeaningsofthewordsthatmakethemup.
Fortunately, the Mad Hatter has provided us with a hint as to h ow to make this
principleuseful. Themeaningofasentenceisnotbasedsole lyonthewordsthatmake
it up, but also on the ordering and grouping of words, and on th e relations among the
wordsinthesentence. Thisisjustanotherwayofsayingthat themeaningofasentence
ispartiallybasedonitssyntacticstructure. Therefore,i nsyntax-drivensemanticanaly-
sis, thecompositionofmeaningrepresentationsisguidedb ythesyntactic components
andrelations providedbythekindofgrammarsdiscussedin Ch.12.
Let’s begin by assuming that the syntactic analysis of an inp ut sentence serves as

the input to a semantic analyzer. Figure 18.1 illustrates an obvious pipeline-oriented
approach that follows directly from this assumption. An inp ut is ﬁrst passed through
a parser to derive its syntactic analysis. This analysis is t hen passed as input to a se-
manticanalyzer toproduceameaningrepresentation. Notethatalthoughthi sdiagram Semantic analyzer
showsa parse tree asinput, othersyntacticrepresentation ssuch as ﬂat chunks,feature
structures,ordependencystructurescanalsobeused. Fort heremainderofthischapter
we’ll assumetree-likeinputs.

Beforemovingon,weshouldtouchontheroleofambiguityint hisstory. Aswe’ve
seen, ambiguous representations can arise from numerous so urces including compet-
ing syntactic analyses, ambiguous lexical items, competin g anaphoric references and
as we’ll see later in this chapterambiguousquantiﬁerscope s. In the syntax-drivenap-
proachpresentedhere,weassumethatsyntactic,lexicalan danaphoricambiguitiesare
not a problem. That is, we’ll assume that some larger system i s capable of iterating

through the possible ambiguous interpretations and passin g them individually to the
kindofsemanticanalyzerdescribedhere.
Let’sconsiderhowsuchananalysismightproceedwith thefo llowingexample:
(18.1)FrancolikesFrasca.
Fig. 18.2showsa simpliﬁedparse tree (lackinganyfeaturea ttachments),alongwith a
plausiblemeaningrepresentationforthisexample. Assugg estedbythedashedarrows,
asemanticanalyzergiventhistreeasinputmightfruitfull yproceedbyﬁrstretrievinga

DRAFTSection18.2. SemanticAugmentationsto CFG Rules 591
S∃eLiking (e)∧Liker(e,Franco )∧Liked(e,Frasca )
NP VP
NP
ProperNoun Verb ProperNoun
Franco likes Frasca
Figure18.2 Parse treefor the sentence Francolikes Frasca.
skeletal meaning representationfrom the subtree correspo ndingto the verb likes. The
analyzerwouldthenretrieveorcomposemeaningrepresenta tionscorrespondingtothe
two noun phrases in the sentence. Then using the representat ion acquired from the
verb as a kind of template, the noun phrase meaningrepresent ationswould be used to
bind the appropriate variables in the verb representation, thus producing the meaning
representationforthesentenceasa whole.
Unfortunately,there are a number of serious difﬁculties wi th this simpliﬁed story.
As described, the function used to interpret the tree in Fig. 18.2 must know, among
other things, that it is the verbthat carries the template up onwhich the ﬁnal represen-
tationisbased,whereitscorrespondingargumentsareandw hichargumentﬁllswhich

role in the verb’s meaning representation. In other words, i t requires a good deal of
speciﬁc knowledge about this particular example and its parse tree to create the re-
quired meaning representation. Given that there are an inﬁn ite number of such trees
for any reasonable grammar, any approach based on one semant ic function for every
possibletreeisin serioustrouble.
Fortunately,wehavefacedthisproblembefore. Languagesa renotdeﬁnedbyenu-
merating the strings or trees that are permitted, but rather by specifying ﬁnite devices

that arecapableof generatingthe desiredset of outputs. It wouldseem, therefore,that
the right place for semantic knowledgein a syntax-directed approachis with the ﬁnite
set of devices that are used to generate trees in the ﬁrst plac e: the grammar rules and
thelexicalentries. Thisisknownasthe rule-to-rulehypothesis (Bach,1976).Rule-to-rulehypothesis
Designingananalyzerbasedonthisapproachbringsusbackt o thenotionofparts
and what it means for them to have meanings. The following sec tion is an attempt to
answerthefollowingtwoquestions:

•Whatdoesit meanfora syntacticconstituenttohaveameanin g?
•What characteristics do these meanings have to have so that t hey can be com-
posedintolargermeanings?
18.2 SemanticAugmentationstoCFGRules
InkeepingwiththeapproachusedinCh.16,wewillbeginbyau gmentingourcontext-
free grammar rules with semantic attachments . These attachments are instructionsSemanticattachments
that specify how to compute the meaning representation of a c onstruction from the
meanings of its constituent parts. Abstractly, our augment ed rules have the following

DRAFT592 Chapter 18. ComputationalSemantics
structure:
A→α1...αn{f(αj.sem,...,αk.sem)}
The semantic attachment to the basic context-free rule is sh own in the{...}to
the right of the rule’s syntactic constituents. This notati on states that the meaning
representation assigned to the construction A, which we will denote as A.sem, can be
computedbyrunningthefunction fonsomesubsetofthesemanticattachmentsof A’s
Therearemyriadwaystoinstantiatethisstyleofrule-to-r uleapproach. Ourseman-
tic attachments could, for example, take the form of arbitra ry programming language
fragments. A meaning representationfor a given derivation could then be constructed
bypassing the appropriatefragmentsto aninterpreterin a b ottom-upfashionandthen
storing the resulting representations as the value for the a ssociated non-terminals.1
Such anapproachwould allowus to create anymeaningreprese ntationwe might like.
Unfortunately, the unrestricted power of this approach wou ld also allow us to create

representations that have no correspondenceat all with the kind of formal logical ex-
pressionsdescribedinthelast chapter. Moreover,thisapp roachwouldprovideuswith
very little guidance as to how to go about designing the seman tic attachments to our
grammarrules.
For these reasons, more principled approachesare typicall y used to instantiate the
rule-to-ruleapproach. We’llintroducetwosuchconstrain edapproachesinthischapter.
Theﬁrstmakesdirectuseof FOLandtheλ-calculusnotationintroducedinCh.17. This

approachessentially uses a logical notation to guide the cr eation of logical forms in a
principled fashion. The second approach, described later i n Sec. 18.4 is based on the
feature-structureanduniﬁcationformalismsintroducedi nCh. 16.
To get started, let’s take a look at a very basic example along with a simpliﬁed
targetsemanticrepresentation.
(18.2)Maharaniclosed.
Closed (Maharani )
Let’sworkourway bottom-upthroughthe rulesinvolvedin th isexample’sderiva-

tion. Startingwiththepropernoun,thesimplestpossiblea pproachistoassignaunique
FOLconstanttoit, asinthefollowing.
ProperNoun→Maharani{Maharani}
Thenon-branching NPrulethatdominatesthisonedoesn’taddanythingsemantica lly,
so we’lljust copythesemanticsofthe ProperNoun upunchangedtothe NP.
NP→ProperNoun{ProperNoun .sem}
Moving on to the VP, the semantic attachment for the verb needs to provide the
name of the predicate, specify its arity and provide the mean s to incorporate an ar-

gument once it’s discovered. We’ll make use of a λ-expression to accomplish these
tasks.
1Thosefamiliar with compiler tools such as YACC and Bison wil l recognize this approach.

DRAFTSection18.2. SemanticAugmentationsto CFG Rules 593
VP→Verb{Verb.sem}
Verb→closed{λx.Closed (x)}
Thisattachmentstipulatesthatthe verb closedhasa unarypredicate Closedasits rep-
resentation. The λ-notationgives us the means to leave unspeciﬁed, as the xvariable,
the entity that is closing. As with our earlier NPrule, the intransitive VPrule that
dominatestheverbsimplycopiesupwardthesemanticsofthe verbbelowit.
Proceeding upward, it remains for the semantic attachment f or theSrule to bring
things together by inserting the semantic representation o f the subject NPas the ﬁrst
argumenttothepredicate.
S→NPVP{VP.sem(NP.sem)}
Sincethevalueof VP.semisaλ-expressionandthevalueof NP.semisasimplya FOL
constant, we can create our desiredﬁnal meaningrepresenta tionby using λ-reduction
toapplythe VP.semto theNP.sem.
λx.Closed (x)(Maharani )
This example illustrates a general pattern which will repea t itself throughout this

chapter. The semantic attachments to our grammar rules will consist primarily of λ-
reductions, where one element of an attachment serves as a fu nctor and the rest serve
as argumentsto it. As we’ll see, the real workresides in the l exiconwherethe bulk of
themeaningrepresentationsareintroduced.
Although this example illustrates the basic approach, the f ull story is a bit more
complex. Let’sbeginbyreplacingourearliertargetrepres entationwithonethatismore
inkeepingwiththeneo-Davidsonianrepresentationsintro ducedinthelastchapter,and

byconsideringanexamplewitha morecomplexnounphraseasi tssubject.
(18.3)Everyrestaurantclosed.
Thetargetrepresentationforthisexampleshouldbe thefol lowing.
∀xRestaurant (x)⇒(∃eClosed (e)∧ClosedThing (e,x)
Clearly, the semantic contribution of the subject noun phra se in this example is
muchmoreextensivethaninourpreviousone. Inourearliere xample,the FOLconstant
representingthe subjectwas simplypluggedintothecorrec tplacein Closedpredicate
via a single λ-reduction. Here the ﬁnal result involves a complex intertw ining of the

contentprovidedbythe NPandthecontentprovidedbythe VP.We’llhavetodosome
workifwe wantrelyon λ-reductionto producewhatwe wanthere.
The ﬁrst step is to determine exactly what we’d like the meani ng representation
ofEvery restaurant to be. Let’s start by assuming that Everyinvokes the∀quantiﬁer

DRAFT594 Chapter 18. ComputationalSemantics
andthatrestaurant speciﬁesthecategoryofconceptthatwe’requantifyingove r,which
we’ll call the restriction of the noun phrase. Putting these together we might expect Restriction
the meaning representation to be something like ∀xRestaurant (x). Although this is a
validFOLformulaitsnota terriblyusefulone,since it saysthatever ythingisa restau-
rant. What’s missing from it is the notion that noun phrases l ikeevery restaurant are
normallyembeddedinexpressionsthatstipulatesomething abouttheuniversallyquan-
tiﬁed variable. That is, we’re probably trying to say something about all restaurants.
This notion is traditionally referred to as the NP’snuclear scope . In this case, the Nuclearscope
nuclearscopeofthisnounphraseis closed.
Wecancapturethesenotionsinourtargetrepresentationby addingadummypred-
icate,Q,representingthescopeandattachingthatpredicatetothe restrictionpredicate
withan⇒logicalconnective,leavinguswith thefollowingexpressi on:
∀xRestaurant (x)⇒Q(x)

Ultimately,whatweneedtodotomakethisexpressionmeanin gfulistoreplace Qwith
the logical expression correspondingto the nuclear scope. Fortunately,the λ-calculus
cancometo ourrescueagain. All we needtodois topermit λ-variablesto rangeover
FOLpredicates as well as terms. The following expression captu res exactly what we
need.
λQ.∀xRestaurant (x)⇒Q(x)
The following series of grammar rules with their semantic at tachments serve to
producethisdesiredmeaningrepresentationforthiskindo fNP.
NP→DetNominal{Det.Sem(Nominal .Sem)}
Det→every{λP.λQ.∀xP(x)⇒Q(x)}

need.
Nominal→Noun{Noun.sem}
Noun→restaurant{λx.Restaurant (x)}
Thecriticalstepinthissequenceinvolvesthe λ-reductioninthe NPrule. Thisrule
appliesthe λ-expressionattachedtothe Dettothesemanticattachmentofthe Nominal,
whichisitselfa λ-expression. Thefollowingaretheintermediatestepsinth isprocess.
λP.λQ.∀xP(x)⇒Q(x)(λx.Restaurant (x))
λQ.∀xλx.Restaurant (x)(x)⇒Q(x)

DRAFTSection18.2. SemanticAugmentationsto CFG Rules 595
The ﬁrst expression is the expansion of the Det.Sem(Nominal .Sem)semantic attach-
ment to the NPrule. The second formula is the result of this λ-reduction. Note that
this second formula has a λ-application embedded in it. Reducing this expression in
placegivesustheﬁnal form.
Havingrevisedoursemanticattachmentforthesubjectnoun phraseportionof our
example, let’s move to the SandVPandVerbrules to see how they need to change
to accommodate these revisions. Let’s start with the Srule and work our way down.
Sincethemeaningofthesubject NPisnowaλ-expression,it makessensetoconsider
it as a functor to be called with the meaning of the VPas its argument. The following
attachmentaccomplishesthis.
S→NPVP{NP.sem(VP.sem)}
Notethatwe’veﬂippedtheroleoffunctorandargumentfromo uroriginalproposalfor
thisSrule.
The last attachment to revisit is the one for the verb close. We need to update it

to provide a proper event-orientedrepresentationand to ma ke sure that it is interfaces
well withthenew SandNPrules. Thefollowingattachmentaccomplishesbothgoals.
Verb→close{λx.∃eClosed (e)∧ClosedThing (e,x)}
Thisattachmentispassedunchangedtothe VPconstituentviatheintransitive VPrule.
Itisthencombinedwiththemeaningrepresentationof Everyrestaurant asdictatedby
thesemanticattachmentforthe Sgivenearlier. Thefollowingexpressionsillustratethe
intermediatestepsinthisprocess.
λQ.∀xRestaurant (x)⇒Q(x)(λy.∃eClosed (e)∧ClosedThing (e,y))

∀xRestaurant (x)⇒λy.∃eClosed (e)∧ClosedThing (e,y)(x)
∀xRestaurant (x)⇒∃eClosed (e)∧ClosedThing (e,x)
These steps achieve our goal of getting the VP’s meaning representation spliced in as
thenuclearscopeinthe NP’srepresentation.
Asisalwaysthecasewithanykindofgrammarengineeringeff ortwenowneedto
makesurethatourearliersimplerexamplesstillwork. Onea reathatweneedtorevisit
is ourrepresentationof propernouns. Let’s considerthemi n the contextof ourearlier
example.
(18.4)Maharaniclosed.
TheSrulenowexpectsthesubject NP’ssemanticattachmenttobeafunctorapplied

example.
to thesemanticsofthe VP,thereforeourearlierrepresentationofpropernounsas FOL
constants won’t do. Fortunately, we can once again exploit t he ﬂexibility of the λ-
calculustoaccomplishwhatwe needwith thefollowingexpre ssion.
λx.x(Maharani )

DRAFT596 Chapter 18. ComputationalSemantics
Thistrickturnsasimple FOLconstantintoa λ-expression,whichwhenreducedserves
to inject the constant into a larger expression. You should w ork through our original
examplewithallofthenewsemanticrulestomakesurethatyo ucancomeupwiththe
followingintendedrepresentation:
∃eClosed (e)∧ClosedThing (Maharani )
Asoneﬁnalexercise,let’sseehowthisapproachextendstoa nexpressioninvolving
a transitiveverbphrase,asin thefollowing.
(18.5)Matthewopeneda restaurant.
Ifwe’vedonethingscorrectlyweoughttobeabletospecifyt hesemanticattachments
for transitive verb phrases, for the verb openand for the determiner a, while leaving
therest ofourrulesalone.
Let’sstartbymodelingthesemanticsforthedeterminer aonourearlierattachment
forevery.
Det→a{λP.λQ.∃xP(x)∧Q(x)}
This rule differs from the attachment for everyin two ways. First we’re using the
existential quantiﬁer ∃to capture the semantics of a. And second we’ve replaced the

forevery.
⇒operator with a logical ∧. The overall framework remains the same with the λ-
variables PandQstandinginfortherestrictionandnuclearscopestobeﬁlle din later.
With this additionour existing NPrule will create the appropriaterepresentationfor a
restaurant :
λQ.∃xRestaurant (x)∧Q(x)
Nextlet’s moveonto the VerbandVPrules. Therearetwo argumentsthat needto
beincorporatedintotheunderlyingmeaningrepresentatio n. Oneargumentisavailable
at the level of the transitive VPrule, and the second at the Srule. Let’s assume the

followingformforthe VPsemanticattachment.
VP→VerbNP{Verb.Sem(NP.Sem)}
Thisattachmentassumesthattheverb’ssemanticattachmen twillbeappliedasafunc-
tor to the semantics of its noun phrase argument. And let’s as sume for now that the
representationswedevelopedearlierforquantiﬁednounph rasesandpropernounswill
remain unchanged. With these assumptions in mind, the follo wing attachment for the
verbopenedwill dowhatwe want.
Verb→opened
{λw.λz.w(λx.∃eOpened (e)∧Opener (e,z)∧OpenedThing (e,x))}
With this attachment in place, the transitive VPrule will incorporate the variable

standing for a restaurant as the second argument to opened, incorporate the entire
expression representing the openingevent as the nuclear scope of a restaurant and

DRAFTSection18.2. SemanticAugmentationsto CFG Rules 597
GrammarRule Semantic Attachment
S→NPVP {NP.sem(VP.sem)}
NP→DetNominal {Det.sem(Nominal .sem)}
NP→ProperNoun {ProperNoun .sem}
VP→Verb {Verb.sem}
VP→VerbNP {Verb.sem(NP.sem)}
Det→every {λP.λQ.∀xP(x)⇒Q(x)}
Det→a {λP.λQ.∃xP(x)∧Q(x)}
Noun→restaurant {λr.Restaurant (r)}
ProperNoun→Matthew{λm.m(Matthew )}
ProperNoun→Franco{λf.f(Franco )}
ProperNoun→Frasca{λf.f(Frasca )}
Verb→closed {λx.∃eClosed (e)∧ClosedThing (e,x)}
Verb→opened {λw.λz.w(λx.∃eOpened (e)∧Opener (e,z)
∧Opened (e,x))}
Figure18.3 Semantic attachments for a fragment ofour Englishgrammar a nd lexicon.
ﬁnally produce a λ-expression suitable for use with our Srule. As with the previous
example you should walk through this example step by step to m ake sure that you
arriveat ourintendedmeaningrepresentation.
∃xRestaurant (x)∧∃eOpened (e)∧Opener (e,Matthew )∧OpenedThing (e,x)
The list of semantic attachments which we’ve developed for t his small grammar

fragment is shown in Fig. 18.2. Sec. 18.5 expands the coverag e of this fragment to
someofthemoreimportantconstructionsinEnglish.
In walking through these examples, we have introduced three techniques that in-
stantiate the rule-to-rule approach to semantic analysis i ntroduced at the beginningof
thissection:
1. Associatingcomplex,function-like, λ-expressionswith lexicalitems
2. Copyingofsemanticvaluesfromchildrentoparentsinnon -branchingrules

3. Function-likeapplicationof the semantics of one of the c hildrenof a rule to the
semanticsoftheotherchildrenoftherulevia λ-reduction.
These techniques serve to illustrate a general division of l abor that guides the de-
sign of semantic attachments in this compositional framewo rk. In general, it is the
lexical rules that introduce quantiﬁers, predicates and te rms into our meaning repre-
sentations. Thesemanticattachmentsforgrammarrulesput theseelementstogetherin
the right ways, but do not in general introduce new elements i nto the representations
beingcreated.

DRAFT598 Chapter 18. ComputationalSemantics
18.3 QuantiﬁerScopeAmbiguityandUnderspeciﬁcation
The grammarfragmentdevelopedin the last section appearst o be sufﬁcient to handle
exampleslike thefollowingthat containtwoormorequantiﬁ ednounphrases.
(18.6)Everyrestauranthasa menu.
Systematically applying the rules given in Fig. 18.2 to this example produces the
followingperfectlyreasonablemeaningrepresentation.
∀xRestaurant (x)⇒
∃yMenu (y)∧∃eHaving (e)∧Haver (e,x)∧Had(e,y)
Thisformulamoreorlesscorrespondstothecommonsensenot ionthatallrestaurants
havemenus.
Unfortunately,this isn’t the only possible interpretatio nfor this example. The fol-
lowingisalso possible.
∃yMenu (y)∧∀xRestaurant (x)⇒
∃eHaving (e)∧Haver (e,x)∧Had(e,y)
This formula asserts that there is one menu out there in the wo rld and all restaurants
share it. Now from a common sense point of view this seems pret ty unlikely, but
remember that our semantic analyzer only has access to the se mantic attachments in

the grammar and the lexicon in producing meaning representa tions. Of course, world
knowledge and contextual information can be used to select b etween these two read-
ings,butonlyif we areabletoproduceboth.
This example illustrates that expressions containing quan tiﬁed terms can give rise
to ambiguous representations even in the absence of syntact ic, lexical or anaphoric
ambiguities. This is known as the problem of quantiﬁer scoping . The difference Quantiferscoping

between the two interpretations given above arises from whi ch of the two quantiﬁed
variableshastheouterscope.
Theapproachoutlinedinthelastsectioncannothandlethis phenomena. Toﬁxthis
we’ll needthefollowingcapabilities.
•The ability to efﬁciently create underspeciﬁed representations that embody all
possiblereadingswithoutexplicitlyenumeratingthem
•A means to generate, or extract, all of the possible readings from this represen-
tation
•Andtheabilityto chooseamongthepossiblereadings

tation
Thefollowingsectionswilloutlineapproachestotheﬁrstt woproblems. Thesolu-
tion to the last, most importantproblem,requiresthe use of context and world knowl-
edgeandunfortunatelyremainsalargelyunsolvedproblem.
18.3.1 Store andRetrieveApproaches
Onewaytoaddressthequantiﬁerscopeproblemistoaddanewn otationtoourexisting
semantic attachments to facilitate the compositional crea tion of the desired meaning

DRAFTSection18.3. QuantiﬁerScopeAmbiguityandUnderspeciﬁca tion 599
representations. Inthiscase,we’llintroducethenotiono facomplex-term thatpermits Complex-term
FOLexpressions like∀xRestaurant (x)to appear in places where we would normally
only allow FOLterms to appear. Formally, a complex-termwill be an express ion with
thefollowingthree-partstructure:
⟨Quantiﬁervariableformula ⟩
Applying this notation to our current example, we would arri ve at the following
representation:
∃eHaving (e)
∧Haver (e,⟨∀xRestaurant (x)⟩)
∧Had(e,⟨∃yMenu (y)⟩)
The intent of the this approach is to capture the basic predic ate argument structure of
an expression, while remaining agnostic about where the var ious quantiﬁers will end
upintheﬁnal representation.
Aswas the case with λ-expressions,thisnotationaldeviceis onlyusefulif we ca n
provide an algorithm to convert it back into an ordinary FOLexpression. This can be
accomplished by rewriting any predicate containing a compl ex-term according to the
followingschema:

P(⟨Quantiﬁervariableformula ⟩)
=⇒
QuantiﬁervariableformulaConnectiveP (variable )
Inotherwords,thecomplex-term:
1. isextractedfromthepredicateinwhichit appears,
2. isreplacedbythe speciﬁedvariable,
3. and has its variable, quantiﬁer, and formula prepended to the new expression
throughtheuseofanappropriateconnective.
The connective that is used to attach the extracted formula t o the front of the new
expressiondependsonthetypeofthequantiﬁerbeingused: ∧isusedwith∃,and⇒
isusedwith∀.

=⇒
Howdoesthisschemehelpwith ourambiguityproblem? Noteth atournewrepre-
sentationcontainstwocomplexterms. Theorderinwhichwep rocessthemdetermines
which of the two readings we end up with. Let’s consider the ca se where we proceed
left-to-right through the expression transforming the com plex terms as we ﬁnd them.
In this case, we encounter Every restaurant ﬁrst; transforming it yields the following
expression.
∀xRestaurant (x)⇒∃eHaving (e)∧Haver (e,x)∧Had(e,⟨∃yMenu (y)⟩)

Proceedingonwardwenextencounter amenu. Transformingthiscomplextermyields
thefollowingﬁnalformwhichcorrespondstothenon-intuit ivereadingthatwecouldn’t
getwith ourearliermethod.
∃yMenu (y)∧∀xRestaurant (x)⇒∃eHaving (e)∧Haver (e,x)∧Had(e,y)

DRAFT600 Chapter 18. ComputationalSemantics
To get the more common-sense reading that we had earlier all w e have to is pull
outthecomplex-termsintheotherorder;ﬁrst a menuandtheneveryrestaurant .
This approach to quantiﬁer scope provides solutions to the t wo of the desiderata
given earlier: complex terms provide a compact underspeciﬁ ed representation of all
the possible quantiﬁer-based ambiguous readings, and the m ethod for transforming
them providesa deterministic method for eliminating compl exterms and thus retriev-
ing valid FOLformulas. And by altering the ordering by which complex term s are
eliminatedwecanrecoverallthepossiblereadings. Ofcour se,sentenceswith Nquan-
tiﬁerswill have O(N!)differentquantiﬁer-basedreadings.
Inpractice,mostsystemsemployanadhocsetofheuristicpr eferencerulesthatcan
beusedto generatepreferredformsin orderoftheiroverall likelihood. Incaseswhere
no preference rules apply, a left-to-right quantiﬁer order ing that mirrors the surface

orderof the quantiﬁersis used. Domain speciﬁc knowledgeca n thenbe used to either
accept a quantiﬁed formula, or reject it and request another formula. Alshawi (1992)
presentsacomprehensiveapproachtogeneratingplausible quantiﬁerscopings.
18.4 Uniﬁcation-BasedApproachesto SemanticAnalysis
As mentioned in Sec. 18.2, feature structures and the uniﬁca tion operator provide an
effective way to implement syntax-drivensemantic analysi s. Recall that in Ch. 16 we
pairedcomplexfeaturestructureswithindividualcontext -freegrammarrulestoencode

syntacticconstraintssuchasnumberagreementandsubcate gorization;constraintsthat
wereawkwardorinsomecasesimpossibletoconveydirectlyu singcontext-freegram-
mars. For example, the following rule was used to capture agr eement constraints on
Englishnounphrases.
Rulessuchasthisoneservetwo functionsat thesametime: th eyinsurethatthegram-
marrejectsexpressionsthatviolatethisconstraint,andm oreimportantlyforourcurrent

topic, they create complexstructures that can be associate d with parts of grammatical
derivations. The following structure, for example, result s from the application of the
aboveruleto asingularnounphrase.
[
NUMBER sg]]
We’ll usethislattercapabilitytocomposemeaningreprese ntationsandassociatethem
withconstituentsinparse.
In this uniﬁcation-based approach, our FOLrepresentationsand λ-based semantic
attachments are replaced by complex feature structures and uniﬁcation equations. To

[
see how this works, let’s walk througha series of examplessi milar to those discussed

DRAFTSection18.4. Uniﬁcation-BasedApproachesto SemanticAna lysis 601
earlier in Sec. 18.2. Let’s start with a simple intransitive sentence with a proper noun
asit’ssubject.
(18.7)Rhumbaclosed
Usinganevent-orientedapproach,themeaningrepresentat ionforthissentenceshould
besomethinglikethefollowing.
∃eClosing (e)∧Closed (e,Rhumba )
Our ﬁrst task will be to show that we can encode representatio ns like this within the
feature structure framework. The most straightforwardway to approachthis task is to
simply follow the BNF-style deﬁnition of FOLstatements given in Ch. 17. The rel-
evant elements of this deﬁnition stipulate that FOLformulas come in three varieties:
atomic formulas consisting of predicates with the appropri ate number of term argu-
ments, formulas conjoined with other formulas via the ∧,∨and⇒operators, and
ﬁnally quantiﬁed formulas which consist of a quantiﬁer, var iables and a formula. Us-
ing this deﬁnition as a guide, we can capture this FOLexpression with the following
featurestructure.

QUANT∃
VAR 1
FORMULA
OP AND
FORMULA 1[
PRED CLOSING
ARG01]
FORMULA 2
PRED CLOSED
ARG01
ARG1 RHUMBA



Fig.18.4showsthisexpressionusingtheDAG-stylenotatio nintroducedinCh.16.
This ﬁgure reveals the way that variables are handled. Inste ad of introducing explicit
FOLvariables,we’ll use the path-basedfeature-sharingcapab ilityof featurestructures
to accomplish the same goal. In this example, the event varia bleeis captured by the
threepathsleadingtothesame sharednode.

VAR 1
FORMULA
FORMULA 1[
ARG01]
FORMULA 2
ARG01

Ournextstepistoassociate uniﬁcationequationswiththe g rammarrulesinvolved
inthisexample’sderivation. Let’sstart atthe topwith the Srule.
S→NPVP
⟨SSEM⟩=⟨NPSEM⟩
⟨VPARG0⟩=⟨NPINDEXVAR⟩
⟨NPSCOPE⟩=⟨VPSEM⟩
Theﬁrst line simplyequatesthe meaningrepresentationof t heNP(encodedunderthe
SEMfeature) with our top-level S. The purpose of the second equation is to assign
the subject NPto the appropriate role inside the VP’s meaning representation. More

S→NPVP
concretely,it ﬁlls the appropriaterole in the VP’s semantic representationby unifying
theARG0 featurewith a path that leadsto a representationof the sem anticsofthe NP.

DRAFT602 Chapter 18. ComputationalSemantics
RhumbaClosedClosing/uni2203.15485
/uni2227.15480Quant
Var
Formula
Formula2Op
Formula1
Pred
Arg1Arg0Pred
Arg0
Figure18.4 A directedgraph notation forsemantic feature structures.
Finally, it uniﬁesthe SCOPEfeaturein the NP’s meaningrepresentationwith a pointer
to the VP’s meaning representation. As we’ll see, this is a so mewhat convoluted way
tobringtherepresentationofaneventuptowhereitbelongs intherepresentation. The
motivationforthis apparatusshould becomeclear in the ens uingdiscussion wherewe
considerquantiﬁednounphrases.
Carryingon,let’sconsidertheattachmentsforthe NPandProperNoun partsofthis
derivation.
⟨NPSEM⟩=⟨ProperNoun SEM⟩
⟨NPSCOPE⟩=⟨ProperNoun SCOPE⟩
⟨NPINDEXVAR⟩=⟨ProperNoun INDEXVAR⟩
ProperNoun→Rhumba
⟨ProperNoun SEM PRED⟩=RHUMBA
⟨ProperNoun INDEXVAR⟩=⟨ProperNoun SEM PRED⟩
As we saw earlier, there isn’t much to the semanticsof proper nounsin this approach.
Here we’re just introducinga constant and providingan inde x variableto point at that

constant.

DRAFTSection18.4. Uniﬁcation-BasedApproachesto SemanticAna lysis 603
Next,let’smoveontothesemanticattachmentsforthe VPandVerbrules.
VP→Verb
⟨VPSEM⟩=⟨VerbSEM⟩
⟨VPARG0⟩=⟨VerbARG0⟩
Verb→closed
⟨VerbSEM QUANT⟩=∃
⟨VerbSEM FORMULA OP ⟩=∧
⟨VerbSEM FORMULA FORMULA 1PRED⟩=CLOSING
⟨VerbSEM FORMULA FORMULA 1ARG0⟩=⟨VerbSEM VAR⟩
⟨VerbSEM FORMULA FORMULA 2PRED⟩=CLOSED
⟨VerbSEM FORMULA FORMULA 2ARG0⟩=⟨VerbSEM VAR⟩
⟨VerbSEM FORMULA FORMULA 2ARG1⟩=⟨VerbARG0⟩
The attachments for the VPrule parallel our earlier treatment of non-branching
grammatical rules. These uniﬁcation equationsare simply m aking the appropriatese-
mantic fragments of the Verbavailable at the VPlevel. In contrast, the uniﬁcation
equationsfor the Verbintroducethe bulk of the event representationthat is at the core
of this example. Speciﬁcally, it introduces the quantiﬁer, event variable and predica-
tionsthatmakeupthebodyoftheﬁnalexpression. Whatwould beaneventvariablein
FOLiscapturedbytheequationsunifyingthe VerbSEM VAR pathwiththeappropriate

arguments to the predicates in the body of the formula. Final ly, it exposes the single
missingargument(theentitybeingclosed)throughthe ⟨VerbARG0⟩equation.
Taking a step back we can see that these equationsserve the sa me basic functions
as theλ-expressions in Sec. 18.2; they provide the content of the FOLformula being
created,andtheyservetoexposeandnametheexternalargum entsthatwillbeﬁlledin
laterat higherlevelsinthe grammar.
Theselastfewrulesalsodisplaythedivisionoflaborthatw e’veseenseveraltimes

now;lexicalrulesintroducethe bulkofthe semanticconten t,while higherlevelgram-
maticalrulesassemblethe piecesintherightway,ratherth anintroducingcontent.
Of course, as was the case with the λ-based approach things get quite a bit more
complex when we look at expressions containing quantiﬁers. To see this, let’s work
throughthefollowingexample.
(18.8)Everyrestaurantclosed
Again,themeaningrepresentationforthisexpressionshou ldbethefollowing
∀xRestaurant (x)⇒(∃eClosing (e)∧Closed (e,x))

whichiscapturedbythefollowingfeaturestructure.

DRAFT604 Chapter 18. ComputationalSemantics

QUANT∀
VAR 1
FORMULA
OP⇒
FORMULA 1[
PRED RESTAURANT
ARG01]
FORMULA 2
QUANT EXISTS
VAR 2
FORMULA
OP∧
FORMULA 1[
ARG02]
FORMULA 2
ARG02
ARG11




As we saw earlier with the λ-based approach, the outer structure for expressions
like this comes largely from the subject noun phrase. Recall that schematically this
semanticstructurehastheform ∀xP(x)⇒Q(x)wherethe Pexpressionistraditionally
referredtoasthe restrictor andisprovidedbytheheadnounand Qisreferredtoasthe
nuclearscope andcomesfromtheverbphrase.
This structure gives rise to two distinct tasks for our seman tic attachments: the
semantics of the VPsemantics must be uniﬁed with the nuclear scope of the subjec t
noun phrase, and the variable representing that noun phrase must be assigned to the

ARG1roleofthe CLOSEDpredicateintheeventstructure. Thefollowingrulesinvol ved
inthe derivationof Everyrestaurant addressthese twotasks
⟨NPSEM⟩=⟨DetSEM⟩
⟨NPSEM VAR⟩=⟨NPINDEXVAR⟩
⟨NPSEM FORMULA FORMULA 1⟩=⟨Nominal SEM⟩
⟨NPSEM FORMULA FORMULA 2⟩=⟨NPSCOPE⟩
⟨Nominal SEM⟩=⟨NounSEM⟩
⟨Nominal INDEXVAR⟩=⟨NounINDEXVAR⟩
Noun→restaurant
⟨NounSEM PRED⟩=⟨RESTAURANT⟩
⟨NounINDEXVAR⟩=⟨NounSEM PRED⟩
Det→every

DRAFTSection18.4. Uniﬁcation-BasedApproachesto SemanticAna lysis 605
⟨DetSEM QUANT⟩=∀
⟨DetSEM FORMULA OP ⟩=⇒
Asoneﬁnal exercise,let’swalkthroughanexamplewitha tra nsitiveverbphrase.
(18.9)Francoopeneda restaurant
Thisexamplehasthefollowingmeaningrepresentation.
∃xRestaurant (x)∧∃eOpening (e)∧Opener (e,Franco )∧Opened (e,x)

QUANT EXISTS
VAR 1
FORMULA
OP∧
FORMULA 1[
ARG11]
FORMULA 2
QUANT∃
VAR 2
FORMULA
OP∧
FORMULA 1[
PRED OPENING
ARG02]
FORMULA 2
PRED OPENER
ARG0 2
ARG1 FRANCO

FORMULA 3
PRED OPENED
ARG02
ARG11





Theonlyreallynewelementthatweneedtoaddressinthisexa mpleisthefollowing
transitive VPrule.
VP→VerbNP
⟨VerbARG1⟩=⟨NPINDEXVAR⟩

FORMULA 1[
ARG11]
FORMULA 2
VAR 2
FORMULA
FORMULA 1[
ARG02]
FORMULA 2
ARG0 2

FORMULA 3
ARG02
ARG11

VP→VerbNP
This rule has the two primary tasks that parallel those in our Srule: it has to ﬁll the
nuclear scope of the object NPwith the semantics of the VP, and it has to insert the
variable representing the object into to the right role in th eVP’s meaning representa-
tion.
Oneobviousproblemwiththeapproachwejustdescribedisth atitfailstogenerate
all the possible ambiguous representations arising from qu antiﬁer scope ambiguities.
Fortunately,the approachesto underspeciﬁcationdescrib edearlier in Sec. 18.3can be

tion.
adaptedtothe uniﬁcation-basedapproach.

DRAFT606 Chapter 18. ComputationalSemantics
18.5 SemanticAttachmentsfor aFragmentofEnglish
Thissectiondescribesasetofsemanticattachmentsforasm allfragmentofEnglish,the
bulk of which are based on those used in the Core LanguageEngi ne (Alshawi, 1992).
As in the rest of this chapter, to keep the presentation simpl e, we omit the feature
structuresassociated with these ruleswhen they are not nee ded. Remember that these
featuresareneededtoensurethatthecorrectrulesareappl iedinthecorrectsituations.
Most importantly for this discussion, they are needed to ens ure that the correct verb
entriesarebeingemployedbasedontheirsubcategorizatio nfeaturestructures.
18.5.1 Sentences
Tothispoint,we’veonlydealtwithsimpledeclarativesent ences. Thissectionexpands
ourcoveragetoincludetheothersentencetypesﬁrstintrod ucedinCh.12: imperatives,
yes-no-questions,andwh-questions. Let’sstartbyconsid eringthefollowingexamples:
(18.10)Flight487serveslunch.
(18.11)Servelunch.
(18.12)DoesFlight207servelunch?
(18.13)Whichﬂightsservelunch?

The meaning representations of these examples all contain p ropositions concern-
ing the serving of lunch on ﬂights. However, they differ with respect to the role that
thesepropositionsareintendedtoserveinthesettingsinw hichtheyareuttered. More
speciﬁcally, the ﬁrst example is intended to convey factual information to a listener,
the second is a request for an action, and the last two are requ ests for information. To
capture these differences, we will introduce a set of operat ors that can be applied to

FOLsentencesinthe same way that beliefoperatorswereused in C h. 17. Speciﬁcally,
theoperators DCL,IMP,YNQ, andWHQwill beappliedtothe FOLrepresentationsof
declaratives,imperatives,yes-no-questions,andwh-que stions,respectively.
Producing meaning representations that make appropriate u se of these operators
requires the right set of semantic attachments for each of th e possible sentence types.
For declarative sentences, we can simply alter the basic sen tence rule we have been
usingasfollows:

S→NPVP{DCL(NP.sem (VP.sem ))}
The normal interpretation for a representation headed by th eDCLoperator would be
asa factualstatementto beaddedtothecurrentknowledge-b ase.
Imperative sentencesbeginwith a verbphrase andlack anovertsubject. Because
of the missing subject, the meaning representation for the m ain verb phrase will con-
sist of aλ-expression with an unbound λ-variable representing this missing subject.
To deal with this, we can simply supplya subject to the λ-expression by applying a

ﬁnalλ-reduction to a dummy constant. The IMPoperator can then be applied to this
representationasinthe followingsemanticattachment:
S→VP{IMP(VP.sem (DummyYou ))}

DRAFTSection18.5. SemanticAttachmentsfora FragmentofEnglis h 607
Applyingthisruleto(18.11),resultsinthe followingrepr esentation:
IMP(∃eServing (e)∧Server (e,DummyYou )∧Served (e,Lunch)
Aswill bediscussedinCh. 24,imperativescanbeviewedasak indofspeech act .
As discussed in Ch. 12, yes-no-questions consist of a sentence-initial auxiliary
verb,followedbyasubjectnounphraseandthenaverbphrase . Thefollowingseman-
ticattachmentsimplyignorestheauxiliary,andwiththeex ceptionofthe YNQoperator,
constructsthe same representationthat wouldbecreatedfo r thecorrespondingdeclar-
ativesentence:
S→AuxNPVP{YNQ(VP.sem (NP.sem ))}
Theuseofthisrulewithforexample(18.12)producesthefol lowingrepresentation:
YNQ(∃eServing (e)∧Server (e,Flt207)∧Served (e,Lunch ))
Yes-no-questionsshould be thoughtas asking whetherthe pr opositionalpart of its
meaning is true or false given the knowledge currently conta ined in the knowledge-
base. Adopting the kind of semantics described in Ch. 17, yes -no-questions can be

answeredbydeterminingifthepropositionisintheknowled ge-base,orcanbeinferred
fromit.
Unlikeyes-no-questions, wh-subject-questions ask forspeciﬁc informationabout
the subject of the sentence rather than the sentence as a whol e. The following at-
tachment produces a representation that consists of the ope ratorWHQ, the variable
correspondingtothesubjectofthesentence,andthe bodyof theproposition:
S→WhWordNPVP {WHQ(NP.sem.var ,VP.sem (NP.sem ))}

fromit.
Thefollowingrepresentationistheresult ofapplyingthis ruleto example(18.13):
WHQ(x,∃e,xServing (e)∧Server (e,x)
∧Served (e,Lunch )∧Flight (x))
Suchquestionscan beansweredbyreturninga set ofassignme ntsforthe subject vari-
able that make the resulting proposition true with respect t o the current knowledge-
base.
Finally,considerthefollowing wh-non-subject-question :
(18.14)HowcanIgofromMinneapolistoLongBeach?
In examples like this, the question is not about the subject o f the sentence but rather

base.
some other argument, or some aspect of the proposition as a wh ole. In this case, the
representation needs to provide an indication as to what the question is about. The
followingattachmentprovidesthisinformationbyprovidi ngthe semanticsofthe aux-
iliaryasanargumenttothe WHQoperator:

DRAFT608 Chapter 18. ComputationalSemantics
S→WhWordAuxNPVP {WHQ(WhWord.semVP.sem (NP.sem ))}
Thefollowingrepresentationwouldresultfromanapplicat ionofthisruletoexam-
ple(18.14):
WHQ(How,∃eGoing (e)∧Goer(e,User)
∧Origin (e,Minn)∧Destination (e,LongBeach ))
Aswe’llseeinCh.24,correctlyansweringthiskindofquest ioninvolvesafairamount
ofdomainspeciﬁcreasoning. Forexample,thecorrectwayto answerexample(18.14)
istosearchforﬂightswiththespeciﬁeddepartureandarriv alcities. Note,however,that
there is no mention of ﬂights or ﬂying in the actual question. The question-answerer,
therefore, has to apply knowledge speciﬁc to this domain to t he effect that questions
aboutgoingplacesarereallyquestionsaboutﬂightsto thos eplaces.
Finally, we should make it clear that this particular attach ment is only useful for
rather simple wh-questions without missing arguments or em bedded clauses. As dis-
cussed in Ch. 16, the presence of long-distance dependencie s in these questions re-

quires additional mechanismsto determine exactly what is b eing asked about. Woods
(1977) and Alshawi (1992) provide extensive discussions of general mechanisms for
handlingwh-non-subjectquestions.
18.5.2 NounPhrases
As we have already seen, the meaning representations for nou n phrases can be either
normal FOLterms or complex-terms. The following sections detail the s emantic at-
tachments needed to produce meaning representations for so me of the most frequent

kinds of English noun phrases. Unfortunately, as we will see , the syntax of English
nounphrasesprovidessurprisinglylittle insight intothe irmeaning. It isoftenthecase
that the best we can do is provide a rather vague intermediate level of meaning repre-
sentationthatcanserveasinputtofurtherinterpretation processes.
Compound Nominals
Compoundnominals,alsoknownasnoun-nounsequences,cons istofsimplesequences
ofnouns,asinthefollowingexamples:
(18.15) Flight schedule
(18.16) Summerﬂightschedule

As noted in Ch. 12, the syntactic structure of this construct ion can be captured by the
regularexpression Noun∗,orbythefollowingcontext-freegrammarrules:
In these constructions,the ﬁnal nounin the sequence is the h ead of the phraseand
denotes an object that is semantically related in some unspe ciﬁed way to the other

DRAFTSection18.5. SemanticAttachmentsfora FragmentofEnglis h 609
nounsthatprecedeitinthesequence. Ingeneral,anextreme lywiderangeofcommon-
sense relations can be denoted by this construction. Discer ning the exact nature of
these relationshipsis well beyondthe scope of the kind of su perﬁcial semantic analy-
sis presented in this chapter. The attachment in the followi ng rule builds up a vague
representationthat simply notes the existence of a semanti c relation between the head
noun and the modifying nouns, by incrementally noting such a relation between the
headnounandeachnountoitsleft:
{λxNominal.sem(x)∧NN(Noun.sem,x)}
The relation NNis used to specify that a relation holds between the modifyin g
elements of a compound nominal and the head Noun. In the examples given above,
thisleadstothe followingmeaningrepresentations:
λx.Schedule (x)∧NN(x,Flight )
λx.Schedule (x)∧NN(x,Flight )∧NN(x,Summer )
Note that this representation correctly instantiates a ter m representinga Schedule,

whileavoidingthecreationoftermsrepresentingeithera FlightorSummer.
GenitiveNounPhrases
Recall from Ch. 12 that genitive noun phrases make use of comp lex determiners that
consist of noun phrases with possessive markers, as in Atlanta’s airport andMaha-
rani’s menu . It is quite tempting to represent the relation between thes e words as an
abstract kind of possession. A little introspection, howev er, reveals that the relation
between a city and its airport has little in common with a rest aurant and its menu.

Therefore, as with compound nominals, it’s best to simply st ate an abstract semantic
relationbetweenthevariousconstituents.
NP→ComplexDetNominal
{⟨∃xNominal .sem(x)∧GN(x,ComplexDet .sem)⟩}
ComplexDet→NP’s{NP.sem}
Applyingtheserulesto Atlanta’sairport resultsin thefollowingcomplex-term:
⟨∃xIsa(x,Airport )∧GN(x,Atlanta )⟩
Subsequent semantic interpretation would have to determin e that the relation denoted
bytherelation GNis actuallya location.

DRAFT610 Chapter 18. ComputationalSemantics
AdjectivePhrases
Englishadjectivescanbesplitintotwomajorcategories: p re-nominalandpredicative.
ThesecategoriesareexempliﬁedbythefollowingBERP examp les:
(18.17)Idon’tminda cheaprestaurant.
(18.18)Thisrestaurantischeap.
Forthepre-nominalcase,anobvious andoftenincorrect proposalforthesemantic
attachmentisillustratedinthe followingrules:
Nominal→AdjNominal
{λxNominal .sem(x)∧Isa(x,Adj.sem)}
Adj→cheap{Cheap}
Thissolutionmodiﬁesthesemanticsofthenominalbyapplyi ngthepredicateprovided
by the adjective to the variable representing the nominal. F or our cheap restaurant
example,thisyieldsthe followingnotunreasonablerepres entation:
λxIsa(x,Restaurant )∧Isa(x,Cheap )
This is an example of what is knownas intersective semantics since the meaningIntersectivesemantics
of the phrase can be thought of as the intersection of the cate gory stipulated by the
nominal and the category stipulated by the adjective. In thi s case, this amounts to the

intersectionofthecategoryofcheapthingswiththecatego ryofrestaurants.
Unfortunately,thissolutionoften doesthe wrongthing. Fo r example,considerthe
following meaning representations for the phrases small elephant ,former friend , and
fakegun:
λxIsa(x,Elephant )∧Isa(x,Small)
λxIsa(x,Friend )∧Isa(x,Former )
λxIsa(x,Gun)∧Isa(x,Fake)
Each of these representations is peculiar in some way. The ﬁr st one states that this
particularelephantisa memberofthegeneralcategoryofsm allthings,whichisprob-

fakegun:
ably not true. The second exampleis strange in two ways: it as serts that the person in
question is a friend, which is false, and it makes use of a fair ly unreasonablecategory
offormer things . Similarly, the third example asserts that the object in que stion is a
gundespitethefact that fakemeansit isnotone.
As with compound nominals, there is no clever solution to the se problems within
theboundsofourcurrentcompositionalframework. Therefo re,thebestapproachisto

simplynotethestatusofaspeciﬁc kindofmodiﬁcationrelat ionandassumethatsome

DRAFTSection18.5. SemanticAttachmentsfora FragmentofEnglis h 611
S
NP VP
NP VPto
VP
PP
NP
Pro Verb Prop-Noun Inf-To Verb Prep PropNoun
I told Harry to go to Maharani
Figure18.5 Parse treefor I toldHarry togotoMaharani.
further procedurewith access to additional relevant knowl edgecan replace this vague
relationwith anappropriaterepresentation(Alshawi,199 2).
{λxNominal .sem(x)∧AM(x,Adj.sem)}
Applyingthisruleto acheaprestaurant resultsinthe followingformula:
∃xIsa(x,Restaurant )∧AM(x,Cheap )
Note that even this watered-down proposal produces represe ntationsthat are logi-
callyincorrectforthe fakeandformerexamples. Inbothcases,itassertsthattheobjects
in question are in fact members of their stated categories. I n general, the solution to
this problem has to be based on the speciﬁc semantics of the ad jectives and nouns in
question. For example, the semantics of formerhas to involvesome form of temporal
reasoning, while fakerequires the ability to reason about the nature of concepts a nd

categories.
18.5.3 VerbPhrases
The general schema for computing the semantics of verb phras es relies on the notion
of functionapplication. In most cases, the λ-expressionattached to the verbis simply
applied to the semantic attachments of the verb’s arguments . There are, however, a
numberofsituationsthat forceusto departsomewhatfromth isgeneralpattern.
Inﬁnitive VerbPhrases
A fair number of English verbs take some form of verb phrase as one of their argu-

ments. This complicates the normal verb phrase semantic sch ema since these argu-
ment verb phrases interact with the other argumentsof the he ad verb in ways that are
notcompletelyobvious.

DRAFT612 Chapter 18. ComputationalSemantics
(18.19)ItoldHarryto goto Maharani.
Themeaningrepresentationforthisexampleshouldbesomet hinglikethefollowing:
∃e,f,xIsa(e,Telling )∧Isa(f,Going )
∧Teller (e,Speaker )∧Tellee (e,Harry )∧ToldThing (e,f)
∧Goer(f,Harry )∧Destination (f,x)
Therearetwointerestingthingstonoteaboutthismeaningr epresentation: theﬁrst
is that it consists of two events, and the second is that one of the participants, Harry,
playsaroleinbothofthetwoevents. Thedifﬁcultyincreati ngthiscomplexrepresen-
tation falls to the verb phrase dominatingthe verb tellwhich will need something like
thefollowingasitssemanticattachment:
λx,yλz∃eIsa(e,Telling )
∧Teller (e,z)∧Tellee (e,x)∧ToldThing (e,y)
Semantically,we caninterpretthissubcategorizationfra meforTellasprovidingthree
semanticroles: apersondoingthetelling,arecipientofth etelling,andtheproposition
beingconveyed.
The difﬁcult part of this example involves getting the meani ng representation for

the main verb phrase correct. As shown in Figure 18.5, Harryplays the role of both
theTelleeof theTellingevent and the Goerof theGoingevent. However, Harryis
notavailablewhenthe Goingeventiscreatedwithinthe inﬁnitiveverbphrase.
Although there are several possible solutions to this probl em, it is usually best to
stick with a uniform approach to these problems. Therefore, we will start by simply
applyingthe semantics of the verbto the semantics of the oth er argumentsof the verb
VP→VerbNPVPto {Verb.sem(NP.sem,VPto.sem) }

Since the toin the inﬁnitive verb phrase construction does not contribu te to its
meaning,wesimplycopythemeaningofthechildverbphraseu ptotheinﬁnitiveverb
phrase. Recall, that we are relyingon the unseenfeaturestr ucturesto ensurethat only
thecorrectverbphrasescanbeusedwith thisconstruction.
VPto→toVP{VP.sem}
In this solution, the verb’s semantic attachment has two tas ks: incorporating the
NP.sem,theGoer,intotheVPto.sem ,andincorporatingthe Goingeventasthe ToldThing
oftheTelling. Thefollowingattachmentperformsbothtasks:
Verb→tell
{λx,y
λz

Verb→tell
{λx,y
λz
∃e,y.variableIsa (e,Telling )
∧Teller (e,z)∧Tellee (e,x)
∧ToldThing (e,y.variable )∧y(x)

DRAFTSection18.5. SemanticAttachmentsfora FragmentofEnglis h 613
In this approach, the λ-variable xplays the role of the Telleeof the telling and the
argument to the semantics of the inﬁnitive, which is now cont ained as a λ-expression
in the variable y. The expression y(x)representsa λ-reductionthat inserts Harryinto
theGoingeventasthe Goer. Thenotation y.variable,isanalogoustothenotationused
for complex-termvariables, and gives us access to the event variable representing the
Goingeventwithintheinﬁnitive’smeaningrepresentation.
Note that this approach plays fast and loose with the deﬁniti on ofλ-reduction, in
that it allows λ-expressions to be passed as arguments to other λ-expressions, when
technicallyonly FOLtermscanservethatrole. Thistechniqueisaconveniencesi milar
totheuseofcomplex-termsinthatitallowsustotemporaril ytreatcomplexexpressions
astermsduringthecreationofmeaningrepresentations.
18.5.4 PrepositionalPhrases
Atafairlyabstractlevel,prepositionalphrasesservetwo distinctfunctions: theyassert

binary relations between their heads and the constituents t o which they are attached,
and they signal arguments to constituents that have an argum entstructure. These two
functionsarguefortwodistincttypesofprepositionalphr asesthatdifferbasedontheir
semantic attachments. We will consider three places in the g rammar where preposi-
tional phrasesserve these roles: modiﬁersof nounphrases, modiﬁers of verb phrases,
andargumentstoverbphrases.
NominalModiﬁer PrepositionalPhrases

Modiﬁer prepositional phrases denote a binary relation bet ween the concept being
modiﬁed, which is external to the prepositional phrase, and the head of the preposi-
tional phrase. Consider the following example and its assoc iated meaning representa-
tion:
(18.20)ArestaurantonBroadway.
∃xIsa(x,Restaurant )∧On(x,Pearl)
Therelevantgrammarrulesthat governthisexamplearethef ollowing:
PP→PNP
Proceeding in a bottom-up fashion, the semantic attachment for this kind of rela-

tion:
PP→PNP
tional preposition should provide a two-place predicate wi th its argumentsdistributed
overtwoλ-expressions,asin thefollowing:
P→on{λyλxOn(x,y)}
With this kind of arrangement, the ﬁrst argument to the predi cate is provided by the
head of prepositional phrase and the second is provided by th e constituent that the

DRAFT614 Chapter 18. ComputationalSemantics
prepositionalphraseisultimatelyattachedto. Thefollow ingsemanticattachmentpro-
videstheﬁrst part:
PP→PNP{P.sem(NP.sem)}
Thisλ-applicationresultsin a new λ-expressionwhere the remainingargumentis the
innerλ-variable.
This remaining argument can be incorporated using the follo wing nominal con-
struction:
Nominal→NominalPP{λzNominal.sem(z)∧PP.sem(z)}
VerbPhraseModiﬁer PrepositionalPhrases
The generalapproachto modifyingverbphrases is similar to that of modifyingnomi-
nals. The differenceslie in the details of the modiﬁcation i n the verb phrase rule; the
attachments for the preposition and prepositional phrase r ules are unchanged. Let’s
consider the phrase ate dinner in a hurry which is governed by the following verb
phraserule:
VP→VPPP
Themeaningrepresentationof the verbphrase constituenti n this construction, ate
dinner,isaλ-expressionwherethe λ-variablerepresentstheasyetunseensubject.
λx∃eIsa(e,Eating )∧Eater(e,x)∧Eaten(e,Dinner )

VP→VPPP
The representation of the prepositional phrase is also a λ-expression where the
λ-variableisthesecondargumentinthe PPsemantics.
λxIn(x,<∃hHurry (h)>)
Thecorrectrepresentationforthemodiﬁedverbphraseshou ldcontaintheconjunc-
tion of these two representations with the Eatingevent variable ﬁlling the ﬁrst argu-
ment slot of the Inexpression. In addition, this modiﬁed representation must remain
aλ-expressionwith the unbound Eatervariableasthe new λ-variable. Thefollowing
attachmentexpressionfulﬁllsall oftheserequirements:

VP→VPPP{λyVP.sem(y)∧PP.sem(VP.sem.variable) }
There are two aspects of this attachment that require some el aboration. The ﬁrst
involvestheapplicationoftheconstituentverbphrases’ λ-expressiontothevariable y.
Binding the lower λ-expression’svariable to a new variable allows us to liftthe lower
variable to the level of the newly created λ-expression. The result of this techniqueis
a newλ-expression with a variable that, in effect, plays the same r ole as the original
variableinthelowerexpression. Inthiscase,thisallowsa λ-expressiontobemodiﬁed

duringtheanalysisprocessbeforetheargumenttotheexpre ssionisactuallyavailable.

DRAFTSection18.6. IntegratingSemanticsintothe EarleyParser 615
The second notable aspect of this attachment involves the VP.sem.variablenota-
tion. This notation is used to access the event-variable rep resenting the underlying
meaning of the verb phrase, in this case, e. This is analogous to the notation used to
provideaccesstothevariouspartsofcomplex-termsintrod ucedearlier.
Applying this attachment to the current example yields the f ollowing representa-
tion,whichissuitable forcombinationwitha subsequentsu bjectnounphrase:
λy∃eIsa(e,Eating )∧Eater(e,y)∧Eaten(e,Dinner )
∧In(e,<∃hHurry (h)>)
VerbArgument PrepositionalPhrases
Theprepositionalphrasesinthiscategoryservetosignalt heroleanargumentplaysin
some larger event structure. As such, the preposition itsel f does not actually modify
the meaning of the noun phrase. Consider the following examp le of role signaling
prepositionalphrases:
(18.21)IneedtogofromBostontoDallas.
Inexampleslikethis,theargumentsof goareexpressedasprepositionalphrases. How-

ever,themeaningrepresentationsofthesephrasesshouldc onsistsolelyoftheunaltered
representationof their head nouns. To handlethis, argumen tprepositionalphrasesare
treatedinthesamewaythatnon-branchinggrammaticalrule sare;thesemanticattach-
mentofthenounphraseiscopiedunchangedtothesemanticso fthe largerphrase.
PP→PNP{NP.sem}
The verb phrase can then assign this meaning representation to the appropriate event
role. A more complete account of how these argument bearing p repositional phrases

maptounderlyingeventroleswill bepresentedinCh. 19.
18.6 IntegratingSemanticsintothe EarleyParser
In Section 18.1, we suggested a simple pipeline architectur e for a semantic analyzer
where the results of a complete syntactic parse are passed to a semantic analyzer. The
motivationforthisnotionstemsfromthefactthatthecompo sitionalapproachrequires
the syntactic parse before it can proceed. It is, however, al so possible to perform se-
mantic analysis in parallel with syntactic processing. Thi s is possible because in our

compositionalframework,the meaningrepresentationfor a constituent can be created
as soon as all of its constituent parts are present. This sect ion describes just such an
approachtointegratingsemanticanalysisintotheEarleyp arserfromCh.13.
The integration of semantic analysis into an Earley parser i s straightforward and
follows precisely the same lines as the integration of uniﬁc ation into the algorithm
giveninCh. 16. Threemodiﬁcationsarerequiredtotheorigi nalalgorithm:

1. The rulesof the grammarare givena new ﬁeld to contain thei r semantic attach-
ments.

DRAFT616 Chapter 18. ComputationalSemantics
2. The states in the chart are given a new ﬁeld to hold the meani ng representation
ofthe constituent.
3. The E NQUEUE function is altered so that when a complete state is entered i nto
thechartits semanticsarecomputedandstoredin thestate’ ssemanticﬁeld.
ifINCOMPLETE ?(state)then
else ifUNIFY-STATE(state) succeeds then
ifAPPLY-SEMANTICS (state) succeeds then
procedure APPLY-SEMANTICS (state)
meaning-rep←APPLY(state.semantic-attachment ,state)
ifmeaning-rep does not equal failurethen
state.meaning-rep←meaning-rep
Figure18.6 The ENQUEUE function modiﬁed to handle semantics. If the state is comple te
and uniﬁcation succeeds then E NQUEUE calls A PPLY-SEMANTICS to compute and store the
meaning representation of completed states.
Figure 18.6 shows E NQUEUE modiﬁed to create meaning representations. When

ENQUEUE is passed a complete state that can successfully unify its un iﬁcation con-
straints it calls A PPLY-SEMANTICS to compute and store the meaning representation
for this state. Note the importance of performingfeature-s tructureuniﬁcation prior to
semanticanalysis. Thisensuresthatsemanticanalysiswil lbeperformedonlyonvalid
treesandthatfeaturesneededforsemanticanalysiswillbe present.
Theprimaryadvantageofthisintegratedapproachoverthep ipelineapproachliesin
thefactthatA PPLY-SEMANTICS canfailinamannersimilartothewaythatuniﬁcation

can fail. If a semantic ill-formedness is found in the meanin g representation being
created, the corresponding state can be blocked from enteri ng the chart. In this way,
semantic considerations can be brought to bear during synta ctic processing. Ch. 19
describes in some detail the various ways that this notion of ill-formedness can be
realized.
Unfortunately,this also illustrates one of the primary dis advantagesof integrating
semantics directly into the parser—considerable effort ma y be spent on the semantic

realized.
analysis of orphanconstituentsthat do not in the end contribute to a successfu l parse.
The question of whether the gains made by bringing semantics to bear early in the
processoutweighthecostsinvolvedinperformingextraneo ussemanticprocessingcan
onlybeansweredonacase-by-casebasis.

DRAFTSection18.7. IdiomsandCompositionality 617
18.7 IdiomsandCompositionality
Ce corps qui s’appelait et qui s’appelle encore le saint empi re romain
n’´ etaitenaucunemani` ereni saint,niromain,niempire.
This body, which called itself and still calls itself the Hol y Roman Em-
pire,wasneitherHoly,norRoman,noranEmpire.
Voltaire2,1756
As innocuous as it seems, the principle of compositionality runs into trouble fairly
quickly when real languageis examined. There are many cases where the meaningof
aconstituentisnotbasedonthemeaningofitsparts,atleas tnotinthestraightforward
compositionalsense. ConsiderthefollowingWSJexamples:
(18.22) Couponsare justthe tipoftheiceberg.
(18.23) TheSEC’s allegationsareonlythetip oftheiceberg.
(18.24) Coronarybypasssurgery,hipreplacementandintensive-ca reunitsare but
the tipoftheiceberg.
Thephrase thetipoftheiceberg ineachoftheseexamplesclearlydoesn’thavemuchto
do with tips or icebergs. Instead, it roughly means somethin g likethe beginning . The

most straightforward way to handle idiomatic construction s like these is to introduce
new grammar rules speciﬁcally designed to handle them. Thes e idiomatic rules mix
lexical items with grammatical constituents, and introduc e semantic content that is
not derived from any of its parts. Consider the following rul e as an example of this
approach:
NP→thetipof theiceberg
{Beginning}
The lower case items on the right-hand side of this rule are in tended to represent

approach:
precisely words in the input. Although, the constant Beginning should not be taken
too seriously as a meaningrepresentationfor this idiom, it does illustrate the idea that
the meaning of this idiom is not based on the meaning of any of i ts parts. Note that
an Earley-style analyzer with this rule will now produce two parses when this phrase
is encountered: one representing the idiom and one represen ting the compositional
meaning.
As with the rest of the grammar, it may take a few tries to get th ese rules right.

meaning.
Considerthefollowing icebergexamplesfromtheWSJ corpus:
(18.25)Andthat’sbutthetip ofMrs. Ford’siceberg.
(18.26)Thesecommentsdescribeonlythe tipofa 1,000-pageiceberg .
(18.27)The10employeesrepresentthemerest tipoftheiceberg.
The rule given above is clearly not general enough to handle t hese cases. These ex-
amples indicate that there is a vestigial syntactic structu re to this idiom that permits
2Essaisurlesmoeursetlesespritdesnations. TranslationbyY.Sills,asquotedinSillsandMerton(1991) .

DRAFT618 Chapter 18. ComputationalSemantics
some variationin the determinersused, and also permitssom e adjectival modiﬁcation
of both the icebergand thetip. A more promising rule would be something like the
NP→TipNPof IcebergNP
Here the categories TipNPandIcebergNP can be given an internal nominal-like
structurethatpermitssomeadjectivalmodiﬁcationandsom evariationinthedetermin-
ers, while still restricting the heads of these noun phrases to the lexical items tipand
iceberg. Note that this syntactic solution ignores the thorny issue that the modiﬁers
mereand1000-page seem to indicate that both the tipandicebergmay in fact play
some compositional role in the meaning of the idiom. We will r eturn to this topic in
Ch.19,whenwetakeuptheissue ofmetaphor.
To summarize,handlingidioms requiresat least the followi ngchangesto the gen-
eralcompositionalframework:
•Allowthemixingoflexicalitemswith traditionalgrammati calconstituents.

•Allowthecreationofadditionalidiom-speciﬁcconstituen tsneededtohandlethe
correctrangeofproductivityofthe idiom.
•Permitsemantic attachmentsthat introducelogicaltermsa ndpredicatesthat are
notrelatedtoanyoftheconstituentsoftherule.
This discussion is obviously only the tip of an enormous iceb erg. Idioms are far
more frequent and far more productive than is generally reco gnized and pose serious
difﬁcultiesformanyapplications,including,aswewillse einCh.25,machinetransla-
tion.
18.8 Summary

tion.
Thischapter exploresthe notionof syntax-drivensemantic analysis. Amongthe high-
lightsofthischapterarethefollowingtopics:
•Semantic analysis is the process whereby meaning representations are created
andassignedtolinguisticinputs.
•Semantic analyzers that make use of static knowledge from the lexicon and
grammarcancreatecontext-independentliteral, orconven tional,meanings.
•ThePrincipleofCompositionality statesthatthemeaningofasentencecanbe
composedfromthemeaningsofitsparts.

tion.
•InSyntax-driven semantic analysis , the parts are the syntactic constituents of
aninput.
•Compositionalcreationof FOLformulasispossiblewitha fewnotationalexten-
sionsincluding λ-expressions andcomplex-terms .
•Compositional creation of FOLformulas is also possible using the mechanisms
providedbyfeaturestructuresanduniﬁcation.

DRAFTSection18.8. Summary 619
•Natural language quantiﬁers introduce a kind of ambiguity that is difﬁcult to
handle compositionally. Complex-terms can be used to compa ctly encode this
ambiguity.
•Idiomatic language deﬁes the principle of compositionality but can easily be
handled by adapting the techniques used to design grammar ru les and their se-
manticattachments.
As noted earlier, the principle of compositionality is trad itionally attributed to Frege;
Janssen (1997) discusses this attribution. Using the categ orial grammar framework
described in Ch. 12, Montague (1973) demonstrated that a com positional approach
could be systematically applied to an interesting fragment of natural language. The
rule-to-rulehypothesiswasﬁrstarticulatedbyBach(1976 ). Onthecomputationalside
of things, Woods’s L UNARsystem (Woods, 1977) was based on a pipelined syntax-
ﬁrst compositional analysis. Schubert and Pelletier (1982 ) developed an incremental

rule-to-rule system based on Gazdar’s GPSG approach (Gazda r, 1981, 1982; Gazdar
et al., 1985). Main and Benson (1983) extended Montague’s ap proach to the domain
ofquestion-answering.
In one of the all-too-frequent cases of parallel developmen t, researchers in pro-
gramminglanguagesdevelopedessentiallyidenticalcompo sitionaltechniquestoaidin
the design of compilers. Speciﬁcally, Knuth (1968) introdu ced the notion of attribute
grammars that associate semantic structures with syntacti c structures in a one-to-one

correspondence. As a consequence, the style of semantic att achments used in this
chapterwillbefamiliartousersoftheYACC-style(Johnson andLesk,1978)compiler
tools.
SemanticGrammarsare duetoBurton(BrownandBurton,1975) . Similar notions
developed around the same time included Pragmatic Grammars (Woods, 1977) and
PerformanceGrammars(Robinson,1975). Allcenteredaroun dthenotionofreshaping
syntactic grammars to serve the needs of semantic processin g. It is safe to say that
most modernsystemsdevelopedforuse inlimited domainsmak euse ofsome formof
semanticgrammar.

tools.
Most of the techniques used in the fragment of English presen ted in Section 18.5
are adapted from SRI’s Core Language Engine (Alshawi, 1992) . Additional bits and
pieces were adapted from Woods (1977), Schubert and Pelleti er (1982), and Gazdar
et al.(1985). Ofnecessity,a largenumberofimportanttopi cswerenotcoveredinthis
chapter. See Alshawi (1992) for the standard gap-threading approach to semantic in-
terpretationinthepresenceoflong-distancedependencie s. terMeulen(1995)presents

an modern treatment of tense, aspect, and the representatio n of temporal information.
Extensive coverage of approaches to quantiﬁer scoping can b e found in Hobbs and
Shieber (1987) and Alshawi (1992). van Lehn (1978) presents a set of human prefer-
ences for quantiﬁer scoping. Over the years, a considerable amountof efforthas been
directed toward the interpretation of compound nominals. L inguistic research on this

DRAFT620 Chapter 18. ComputationalSemantics
topic can be found in Lees (1970), Downing (1977), Levi (1978 ), and Ryder (1994),
more computationalapproachesare describedin Gershman(1 977),Finin (1980),Mc-
Donald (1982), Pierre (1984), Arens et al. (1987), Wu (1992) , Vanderwende (1994),
andLauer(1995).
Thereisalongandextensiveliteratureonidioms. Fillmore etal.(1988)describea
generalgrammaticalframeworkcalledConstructionGramma rthatplacesidiomsatthe
centerofitsunderlyingtheory. Makkai(1972)presentsane xtensivelinguisticanalysis
of manyEnglishidioms. Hundredsof idiomdictionariesfors econd-languagelearners
are also available. On the computational side, Becker (1975 ) was among the ﬁrst to
suggest the use of phrasal rules in parsers. Wilensky and Are ns (1980) were among
the ﬁrst to successfully make use of this notionin their PHRANsystem. Zernik(1987)
demonstratedasystemthatcouldlearnsuchphrasalidiomsi ncontext. Acollectionof
papersoncomputationalapproachesto idiomsappearedin(F asset al.,1992).

Finally, we have skipped an entire branch of semantic analys is in which expecta-
tions driven from deep meaning representations drive the an alysis process. Such sys-
tems avoid the direct representation and use of syntax, rare ly making use of anything
resembling a parse tree. Some of the earliest and most succes sful efforts along these
lines were developed by Simmons (1973, 1978, 1983) and (Wilk s, 1975a, 1975c). A
series of similar approacheswere developedby Roger Schank and his students (Ries-

beck, 1975; Birnbaum and Selfridge, 1981; Riesbeck, 1986). In these approaches,the
semantic analysis process is guided by detailed procedures associated with individual
lexical items. The CIRCUSinformationextraction system (Lehnertet al., 1991) trace s
itsrootstothese systems.
Exercises
18.1The attachment given on page 609 for handling noun phrases wi th complex de-
terminersisnotgeneralenoughtohandlemostpossessiveno unphrases. Specif-
ically,it doesn’tworkforphraseslikethefollowing:
a. Mysister’s ﬂight

Exercises
b. Myﬁance’smother’sﬂight
Createa newset ofsemanticattachmentstohandlecaseslike these.
18.2Develop a set of grammar rules and semantic attachments to ha ndle predicate
adjectivessuchastheonefollowing:
a. Flight308fromNewYorkis expensive.
b. Murphy’srestaurantischeap.
18.3None of the attachments given in this chapter provide tempor al information.
Augment a small number of the most basic rules to add temporal information
alongthelinessketchedinCh. 17. Use yourrulestocreate me aningrepresenta-

tionsforthe followingexamples:
a. Flight299departedat 9o’clock.

DRAFTSection18.8. Summary 621
b. Flight208will arriveat3 o’clock.
c. Flight1405will arrivelate.
18.4AsnotedinCh.17,thepresenttenseinEnglishcanbeusedtor efertoeitherthe
present or the future. However,it can also be used to express habitual behavior,
asin thefollowing:
a. Flight208leavesat 3o’clock.
This could be a simple statement about today’s Flight 208, or alternatively it
might state that this ﬂight leaves at 3 o’clock every day. Cre ate aFOLmean-
ingrepresentationalongwith appropriatesemantic attach mentsforthis habitual
sense.
18.5Implement an Earley-style semantic analyzer based on the di scussion on page
615.
18.6It has been claimed that it is not necessary to explicitly lis t the semantic attach-
mentformostgrammarrules. Instead,thesemanticattachme ntforaruleshould
beinferablefromthesemantictypesoftherule’sconstitue nts. Forexample,ifa
rule has two constituents, where one is a single argument λ-expression and the
other is a constant, then the semantic attachment should obv iouslyapply the λ-

sense.
615.
expressiontotheconstant. Giventheattachmentspresente dinthischapter,does
thistype-drivensemantics seem likea reasonableidea?
18.7Add a simple type-driven semantics mechanism to the Earley a nalyzer you im-
plementedforExercise5.
18.8UsingaphrasalsearchonyourfavoriteWebsearchengine,co llectasmallcorpus
ofthe tip of the iceberg examples. Be certain that you search for an appropriate
range of examples (i.e., don’t just search for “the tip of the iceberg”.) Analyze
theseexamplesandcomeupwitha set ofgrammarrulesthat cor rectlyaccounts

forthem.
18.9Collect a similar corpusof examplesfor the idiom miss the boat . Analyze these
examples and come up with a set of grammar rules that correctl y accounts for
them.
18.10TherearenowafairnumberofWeb-basednaturallanguageque stionanswering
services that purport to provide answers to questions on a wi de range of topics
(see the book’sWeb page for pointersto current services). D evelop a corpusof
questionsforsomegeneraldomainofinterest anduse it toev aluateoneormore
of these services. Report your results. What difﬁculties di d you encounter in

them.
applyingthestandardevaluationtechniquesto thistask?

Chapter19
Lexical Semantics
“When Iuse aword”,Humpty Dumpty saidinrather ascornful
tone, “itmeans just what Ichoose ittomean–neither more nor less.”
Howmany legs does adog have if youcall itstail aleg?
Four.
Callinga tailalegdoesn’t make itone.
AttributedtoAbraham Lincoln
Theprevioustwochaptersfocusedontherepresentationofm eaningrepresentationsfor
entiresentences. Inthosediscussions,wemadeasimplifyi ngassumptionbyrepresent-
ingwordmeanings asunanalyzedsymbolslike EATorJOHNorRED. Butrepresenting
themeaningofawordbycapitalizingitisaprettyunsatisfa ctorymodel. Inthischapter

Four.
weintroducearichermodelofthesemanticsofwords,drawin gonthelinguisticstudy
ofwordmeaning,a ﬁeldcalled lexicalsemantics . Lexicalsemantics
Beforewe try to deﬁne word meaning in the next section, we ﬁrst need to be clear
on what we mean by word, since we have used the word wordin manydifferentways
inthisbook.
We can use the word lexemeto mean a pairing of a particular form (orthographic Lexeme
orphonological)withitsmeaning,anda lexiconisaﬁnitelist oflexemes. Forthepur- Lexicon
posesoflexicalsemantics,particularlyfordictionaries andthesauruses,werepresenta

lexeme by a lemma. Alemmaorcitation form is the grammatical form that is used Lemma
Citationform to represent a lexeme. This is often the base form; thus carpetis the lemma for car-
pets. The lemma or citation form for sing,sang,sungissing. In many languages the
inﬁnitive form is used as the lemma for the verb; thus in Spani shdormir‘to sleep’ is
thelemmaforverbformslike duermes‘yousleep’. Thespeciﬁcforms sungorcarpets
orsingorduermesarecalled wordforms . Wordform

The process of mapping from a wordform to a lemma is called lemmatization . Lemmatization
Lemmatization is not always deterministic, since it may dep end on the context. For
example, the wordform foundcan map to the lemma ﬁnd(meaning ‘to locate’) or the
lemmafound(‘tocreateaninstitution’),asillustratedinthefollowi ngWSJexamples:
(19.1)He haslookedat 14baseballandfootballstadiumsand foundthatonlyone–
privateDodgerStadium–broughtmoremoneyintoa citythani t tookout.
(19.2)Culturallyspeaking,thiscityhasincreasinglydisplayed itsdeterminationto

foundthesortofinstitutionsthatattract theesteemofEasternu rbanites.
Inaddition,lemmasarepart-of-speechspeciﬁc;thusthewo rdformtableshastwopos-
sible lemmas,the noun tableandtheverb table.
Onewaytodolemmatizationisviathemorphologicalparsing algorithmsofCh.3.
Recallthatmorphologicalparsingtakesasurfaceformlike catsandproduces cat+PL.
Butalemmaisnotnecessarilythesameasthestemfromthemor phologicalparse. For

DRAFT624 Chapter 19. LexicalSemantics
example, the morphological parse of the word celebrations might produce the stem
celebrate with the afﬁxes -ionand-s, while the lemma for celebrations is the longer
formcelebration . In general lemmas may be larger than morphological stems (e .g.,
New York orthrow up). The intuitionis that we want to havea differentlemma when -
ever we need to have a completely different dictionary entry with its own meaning
representation;weexpecttohave celebrations andcelebration shareanentry,sincethe
difference in their meanings is mainly just grammatical, bu t not necessarily to share
onewithcelebrate.
In the remainder of this chapter, when we refer to the meaning (or meanings) of a
‘word’,wewill generallybereferringtoa lemmaratherthan awordform.
Nowthat we havedeﬁnedthe locusofword meaning,we will proc eedto different
ways to represent this meaning. In the next section we introd uce the idea of word
senseas the part of a lexeme that represents word meaning. In follo wing sections we

thendescribewaysofdeﬁningandrepresentingthesesenses ,aswellasintroducingthe
lexicalsemanticaspectsoftheeventsdeﬁnedinCh. 17.
19.1 WordSenses
The meaning of a lemma can vary enormously given the context. Consider these two
uses of the lemma bank, meaning something like ‘ﬁnancial institution’ and ‘slopi ng
mound’,respectively:
(19.3)Instead,a bankcanholdthe investmentsina custodialaccountinthe client ’s
name.
(19.4)But asagricultureburgeonsontheeast bank,theriverwill shrinkevenmore.

name.
We represent some of this contextualvariation by saying tha t the lemma bankhas
twosenses. Asense(orword sense ) is a discrete representation of one aspect of the Word sense
meaning of a word. Loosely followinglexicographictraditi on, we will represent each
sense by placinga superscripton the orthographicformofth e lemma asin bank1and
bank2.1
The senses of a word might not have any particular relation be tween them; it may
bealmostcoincidentalthattheyshareanorthographicform . Forexample,the ﬁnancial

bank2.1
institution andsloping mound senses of bank seem relatively unrelated. In such cases
we say that the two senses are homonyms , and the relation between the senses is one Homonym
ofhomonymy . Thusbank1(‘ﬁnancial institution’) and bank2(‘sloping mound’)are Homonymy
homonyms.
Sometimes, however, there is some semantic connection betw een the senses of a
word. ConsiderthefollowingWSJ’bank’example:
(19.5)While some banksfurnishspermonlyto marriedwomen,othersaremuchless
restrictive.

homonyms.
1Confusingly, theword“lemma”isitself veryambiguous;iti salsosometimesusedtomeantheseseparate
senses, rather than the citation form of theword. You should beprepared to see both uses in theliterature.

DRAFTSection19.1. WordSenses 625
Althoughthisisclearlynotauseofthe‘slopingmound’mean ingofbank,itjustas
clearly is not a reference to a promotional giveaway at a ﬁnan cial institution. Rather,
bankhasawholerangeofusesrelatedtorepositoriesforvarious biologicalentities,as
inbloodbank ,eggbank,andspermbank . Sowe couldcallthis‘biologicalrepository’
sensebank3. Now this new sense bank3has some sort of relation to bank1; both
bank1andbank3are repositories for entities that can be deposited and take n out; in
bank1theentityismoney,wherein bank3theentityisbiological.
When two senses are related semantically, we call the relati onship between them
polysemy rather than homonymy. In many cases of polysemy the semantic relation Polysemy
between the senses is systematic and structured. For exampl e consider yet another
sense ofbank,exempliﬁedinthefollowingsentence:
(19.6)ThebankisonthecornerofNassauandWitherspoon.
Thissense,whichwecancall bank4,meanssomethinglike‘thebuildingbelonging

to a ﬁnancial institution’. It turns out that these two kinds of senses (an organization,
andthebuildingassociatedwithanorganization)occurtog etherformanyotherwords
as well (school,university ,hospital, etc). Thus there is a systematic relationship be-
tweensensesthatwe mightrepresentas
BUILDING↔ORGANIZATION
Thisparticularsubtypeofpolysemyrelationisoftencalle dmetonymy . Metonymy Metonymy
is the use of one aspect of a concept or entity to refer to other aspects of the entity, or

to the entity itself. Thus we are performing metonymy when we use the phrase the
White House to refer to the administration whose ofﬁce is in the White Hou se. Other
commonexamplesofmetonymyincludetherelationbetweenth efollowingpairingsof
senses:
Author( Jane AustenwroteEmma )↔WorksofAuthor( Ireally love Jane Austen )
Animal( The chickenwas domesticated inAsia )↔Meat(The chickenwas overcooked )
Tree(Plums have beautiful blossoms )↔Fruit(Iate apreserved plum yesterday )

senses:
While it can be useful to distinguish polysemy from homonymy , there is no hard
threshold for ‘how related’ two senses have to be to be consid ered polysemous. Thus
the difference is really one of degree. This fact can make it v ery difﬁcult to decide
how many senses a word has, i.e., whether to make separate sen ses for closely related
usages. Thereare variouscriteria fordecidingthat the dif feringuses of a word should
be represented as distinct discrete senses. We might consid er two senses discrete if

theyhaveindependenttruthconditions,differentsyntact icbehavior,independentsense
relations,orexhibitantagonisticmeanings.
Considerthefollowingusesofthe verb servefromtheWSJ corpus:
(19.7)Theyrarely serveredmeat,preferringtoprepareseafood,poultryorgame
birds.
(19.8)HeservedasU.S.ambassadortoNorwayin1976and1977.
(19.9)He mighthave servedhistime,comeoutandledanupstandinglife.
Theserveofserving red meat and that of serving time clearly have different truth

birds.
conditionsandpresuppositions;the serveofserve asambassador hasthe distinctsub-
categorization structure serve as NP . These heuristic suggests that these are probably

DRAFT626 Chapter 19. LexicalSemantics
threedistinctsensesof serve. Onepracticaltechniquefordeterminingiftwosensesare
distinct is to conjoin two uses of a word in a single sentence; this kind of conjunction
ofantagonisticreadingsiscalled zeugma. ConsiderthefollowingATISexamples: Zeugma
(19.10)Whichofthoseﬂightsservebreakfast?
(19.11)DoesMidwestExpressservePhiladelphia?
(19.12)?DoesMidwestExpressservebreakfastandPhiladelphia?
We use(?) tomarkexamplethosethataresemanticallyill-fo rmed. Theoddnessofthe
invented third example (a case of zeugma) indicates there is no sensible way to make
a single sense of servework for both breakfast and Philadelphia. We can use this as
evidencethat servehastwo differentsensesin thiscase.
Dictionaries tend to use many ﬁne-grainedsenses so as to cap ture subtle meaning
differences, a reasonable approach given that traditional role of dictionaries in aiding
wordlearners. Forcomputationalpurposes,weoftendon’tn eedtheseﬁnedistinctions

andsowemaywanttogrouporclusterthesenses;wehavealrea dydonethisforsome
oftheexamplesin thischapter.
We generally reserve the word homonym for two senses which share both a pro-
nunciation and an orthography. A special case of multiple se nses that causes prob-
lemsforspeechrecognitionandspellingcorrectionis homophones .Homophones are Homophone
senses that are linked to lemmas with the same pronunciation but different spellings,
such aswood/wouldorto/two/too. A related problem for speech synthesis are ho-

mographs (Ch. 8).Homographs are distinct senses linked to lemmas with the same Homograph
orthographicformbutdifferentpronunciations,suchasth esehomographsof bass:
(19.13)TheexpertanglerfromDora,Mo.,wasﬂy-castingfor bassratherthanthe
traditionaltrout.
(19.14)Thecurtainrisestothesoundofangrydogsbayingandominou sbasschords
sounding.
Howcanwedeﬁnethemeaningofawordsense? Canwejustlookin adictionary?
Considerthefollowingfragmentsfromthedeﬁnitionsof right,left,red,andbloodfrom

sounding.
theAmericanHeritageDictionary (Morris,1985).
rightadj.located nearer the right hand esp. being on the right when
facingthesame directionasthe observer.
leftadj.locatednearertothissideofthe bodythantheright.
redn.thecolorofbloodoraruby.
bloodn.theredliquidthatcirculatesintheheart,arteriesandvei nsof
animals.
Note the amount of circularity in these deﬁnitions. The deﬁn ition ofrightmakes
twodirectreferencestoitself,whiletheentryfor leftcontainsanimplicitself-reference

animals.
inthephrase thissideofthebody ,whichpresumablymeansthe leftside. Theentriesfor
redandbloodavoidthiskindofdirectself-referencebyinsteadreferen cingeachother
intheir deﬁnitions. Suchcircularityis, ofcourse,inhere ntinall dictionarydeﬁnitions;
these examples are just extreme cases. For humans, such entr ies are still useful since
theuserofthedictionaryhassufﬁcientgraspofthese other terms.

DRAFTSection19.2. RelationsbetweenSenses 627
For computational purposes, one approach to deﬁning a sense is to make use of
a similar approach to these dictionary deﬁnitions; deﬁning a sense via its relationship
with other senses. For example, the above deﬁnitions make it clear that rightandleft
aresimilarkindsoflemmasthatstandinsomekindofalterna tion,oropposition,toone
another. Similarly,we cangleanthat redis a color,it can beappliedtoboth bloodand
rubies, and that bloodis aliquid.Sense relations of this sort are embodiedin on-line
databases like WordNet . Given a sufﬁciently large database of such relations, many
applicationsare quitecapableofperformingsophisticate dsemantictasks(evenif they
donotreallyknowtheirrightfromtheirleft).
A second computational approach to meaning representation is to create a small
ﬁnite set of semantic primitives, atomic units of meaning, a nd then create each sense
deﬁnition out of these primitives. This approach is especia lly common when deﬁning

aspectsofthe meaningof eventssuchassemanticroles .
We will explore both of these approaches to meaning in this ch apter. In the next
section we introduce various relations between senses, fol lowed by a discussion of
WordNet,a senserelationresource. We thenintroducea numb erofmeaningrepresen-
tationapproachesbasedonsemanticprimitivessuchassema nticroles.
19.2 RelationsbetweenSenses
Thissectionexploressomeoftherelationsthatholdamongw ordsenses,focusingona
fewthathavereceivedsigniﬁcantcomputationalinvestiga tion:synonymy ,antonymy ,

andhypernymy ,aswell asa briefmentionofotherrelationslike meronymy .
19.2.1 Synonymy and Antonymy
When the meaning of two senses of two different words (lemmas ) are identical or
nearlyidenticalwesaythetwosensesare synonyms . Synonymsincludesuchpairsas: Synonym
couch/sofa vomit/throwup ﬁlbert/hazelnut car/automobil e
A more formal deﬁnition of synonymy (between words rather th an senses) is that
two words are synonymousif they are substitutable one for th e other in any sentence

without changing the truth conditions of the sentence. We of ten say in this case that
thetwo wordshavethesame propositionalmeaning .Propositionalmeaning
Whilesubstitutionsbetweensomepairsofwordslike car/automobile orwater/H2O
are truth-preserving,the words are still not identical in m eaning. Indeed, probablyno
twowordsareabsolutelyidenticalinmeaning,andifwe deﬁn esynonymyasidentical
meanings and connotations in all contexts, there are probab ly no absolute synonyms.

Manyotherfacetsofmeaningthatdistinguishthesewordsar eimportantbesidespropo-
sitional meaning. For example H2Ois used in scientiﬁc contexts, and would be inap-
propriatein ahikingguide;thisdifferencein genreis part ofthe meaningofthe word.
Inpracticetheword synonymisthereforecommonlyusedto describearelationshipof
approximateorroughsynonymy.

DRAFT628 Chapter 19. LexicalSemantics
Instead of talking about two wordsbeing synonyms, in this chapter we will deﬁne
synonymy (and other relations like hyponymy and meronymy) a s a relation between
sensesratherthanbetweenwords. Wecanseetheusefulnesso fthisbyconsideringthe
wordsbigandlarge. ThesemayseemtobesynonymsinthefollowingATISsentence s,
sincewe couldswap bigandlargein eithersentenceandretainthe samemeaning:
(19.15)Howbigisthatplane?
(19.16)WouldIbeﬂyingonalargeorsmall plane?
But notethe followingWSJ sentencewherewe cannotsubstitu telargeforbig:
(19.17)MissNelson,forinstance,becamea kindofbigsister toBenj amin.
(19.18)?MissNelson,forinstance,becameakindoflargesister toB enjamin.
That is because the word bighas a sense that means being older, or grown up, while
largelacks this sense. Thus it will be convenient to say that some s enses ofbigand
largeare(nearly)synonymouswhileotheronesarenot.
Synonyms are words with identical or similar meanings. Antonyms , by contrast, Antonym

arewordswithoppositemeaningsuchasthefollowing:
long/short big/little fast/slow cold/hot dark/light
rise/fall up/down in/out
It isdifﬁcultto givea formaldeﬁnitionofantonymy. Two sen sescan be antonyms
iftheydeﬁneabinaryopposition,orareatoppositeendsofs omescale. Thisisthecase
forlong/short,fast/slow, orbig/little, which are at opposite ends of the lengthorsize
scale. Anothergroupsof antonymsis reversives , which describe some sort of change
ormovementinoppositedirections,suchas rise/fallorup/down.

From one perspective, antonyms have very different meaning s, since they are op-
posite. From another perspective, they have very similar me anings, since they share
almost all aspects of their meaning except their position on a scale, or their direction.
Thusautomaticallydistinguishingsynonymsfromantonyms canbedifﬁcult.
19.2.2 Hyponymy
One sense is a hyponym of another sense if the ﬁrst sense is more speciﬁc, denoting Hyponym
a subclass of the other. For example, caris a hyponymof vehicle;dogis a hyponym

ofanimal, andmangois a hyponym of fruit. Conversely, we say that vehicleis a
hypernym ofcar, andanimalis a hypernym of dog. It is unfortunate that the two Hypernym
words (hypernym and hyponym) are very similar and hence easi ly confused; for this
reasonthe word superordinate isoftenusedinsteadof hypernym . Superordinate
Superordinate vehiclefruitfurniture mammal
Hyponym carmangochair dog
We can deﬁne hypernymy more formally by saying that the class denoted by the

superordinateextensionallyincludestheclassdenotedby thehyponym. Thustheclass
of animals includes as members all dogs, and the class of movi ng actions includes
all walking actions. Hypernymy can also be deﬁned in terms of entailment. Under
this deﬁnition, a sense Ais a hyponym of a sense Bif everything that is Ais alsoB

DRAFTSection19.3. WordNet: A DatabaseofLexicalRelations 629
and hence being an Aentails being a B, or∀x A(x)⇒B(x). Hyponymy is usually
a transitive relation; if A is a hyponym of B and B is a hyponym o f C, then A is a
hyponymofC.
The conceptof hyponymyis closely related to a numberof othe rnotionsthat play
central roles in computer science, biology, and anthropolo gy and computer science.
Theterm ontology usuallyreferstoasetofdistinctobjectsresultingfroman analysisof Ontology
a domain,or microworld . Ataxonomy is a particulararrangementof the elementsof Taxonomy
an ontologyintoa tree-likeclass inclusionstructure. Nor mally,thereare a set of well-
formednessconstraintson taxonomiesthat go beyondtheir c omponentclass inclusion
relations. For example, the lexemes hound,mutt, andpuppyare all hyponymsof dog,
as aregolden retriever andpoodle, but it would be odd to construct a taxonomy from
all those pairs since the concepts motivating the relations is different in each case.

Instead, we normally use the word taxonomy to talk about the hypernymy relation
betweenpoodleanddog;bythisdeﬁnition taxonomy isa subtypeofhypernymy.
19.2.3 Semantic Fields
So far we’ve seen the relations of synonymy, antonymy, hyper nomy, and hyponymy.
Anotherverycommonrelationis meronymy ,thepart-whole relation. A legispartof Meronymy
Part-whole achair; awheelis part of a car. We say that wheelis ameronym ofcar, andcaris a
Meronym holoynm ofwheel.

Holonym But there is a more general way to think about sense relations and word meaning.
Wheretherelationswe’vedeﬁnedsofarhavebeenbinaryrela tionsbetweentwosenses,
asemanticﬁeld is amodelofamoreintegrated,orholistic,relationshipam ongentire Semanticﬁeld
setsofwordsfroma singledomain. Considerthefollowingse t ofwords:
reservation,ﬂight,travel,buy,price, cost,fare,rates, meal,plane
We could assert individual lexical relations of hyponymy, s ynonymy, and so on
betweenmanyofthewordsinthislist. Theresultingsetofre lationsdoesnot,however,

add up to a complete account of how these words are related. Th ey are clearly all
deﬁned with respect to a coherent chunk of common sense backg round information
concerning air travel. Background knowledge of this kind ha s been studied under a
variety of frameworks and is known variously as a frame(Fillmore, 1985), model
(Johnson-Laird,1983),or script(SchankandAlbelson,1977),andplaysacentralrole
ina numberofcomputationalframeworks.
We will discussin Sec. 19.4.5the FrameNet project(Baker et al., 1998),whichis

anattempttoprovidearobustcomputationalresourceforth iskindofframeknowledge.
In the FrameNet representation,eachof the wordsin the fram e is deﬁnedwith respect
tothe frame,andsharesaspectsofmeaningwithotherframew ords.
19.3 WordNet: A Databaseof LexicalRelations
The most commonlyused resourcefor English sense relations is theWordNet lexical WordNet
database (Fellbaum, 1998). WordNet consists of three separ ate databases, one each

DRAFT630 Chapter 19. LexicalSemantics
The noun “bass” has 8senses inWordNet.
1. bass1- (thelowest part of the musical range)
2. bass2, bass part1- (thelowest part inpolyphonic music)
3. bass3, basso1-(anadult male singer withthe lowest voice)
4. sea bass1, bass4- (the leanﬂeshof asaltwater ﬁshof the familySerranidae)
5. freshwater bass1,bass5- (anyof various NorthAmericanfreshwater ﬁshwith
leanﬂesh(especiallyof the genus Micropterus))
6. bass6, bass voice1,basso2-(the lowest adult male singing voice)
7. bass7- (themember withthe lowest range of a familyof musical inst ruments)
8. bass8- (nontechnical name for any of numerous edible marine and
freshwater spiny-ﬁnned ﬁshes)
The adjective “bass” has 1sense inWordNet.
1. bass1, deep6-(having or denoting a lowvocal or instrumental range)
”adeepvoice”; ”abass voice islower than abaritone voice”;
”abass clarinet”
Figure19.1 A portionof the WordNet 3.0entryfor the noun bass.
for nounsandverbs, and a third foradjectivesand adverbs;c losed class wordsare not

included. Each database consists of a set of lemmas, each one annotated with a set of
senses. The WordNet 3.0 release has 117,097nouns, 11,488ve rbs, 22,141adjectives,
and 4,601 adverbs. The average noun has 1.23 senses, and the a verage verb has 2.16
senses. WordNetcanbeaccessedviathewebordownloadedand accessed locally.
Partsofatypicallemmaentryforthenounandadjective bassareshowninFig.19.1.
Note that there are eight senses for the noun and one for the ad jective, each of which

has agloss(a dictionary-style deﬁnition), a list of synonyms for the s ense (called a Gloss
synset), and sometimes also usage examples (shown for the adjectiv e sense). Unlike
dictionaries, WordNet doesn’trepresent pronunciation,s o doesn’t distinguish the pro-
nunciation[b ae s] in bass4,bass5, andbass8from the other senses pronounced[b ey
s].
Thesetofnear-synonymsforaWordNetsenseiscalleda synset(forsynonymset ); Synset
synsetsareanimportantprimitiveinWordNet. Theentryfor bassincludessynsetslike

s].
bass1, deep6, orbass6, bass voice1, basso2. We can think of a synset as representing
aconceptof the type we discussed in Ch. 17. Thus instead of representi ng concepts
using logical terms, WordNet representsthem as a lists of th e word-sensesthat can be
usedtoexpresstheconcept. Here’sanothersynsetexample:
{chump1, fool2, gull1, mark9, patsy1, fall guy1, sucker1,
soft touch1, mug2}
The gloss of this synset describes it as a person who is gullible and easy to take ad-

vantage of . Each of the lexical entries included in the synset can, ther efore, be used
to express this concept. Synsets like this one actually cons titute the senses associated
with WordNet entries, and hence it is synsets, not wordforms , lemmas or individual
senses, thatparticipateinmost ofthelexicalsenserelati onsinWordNet.
Let’s turn now to these lexical sense relations, some of whic h are illustrated in
Fig. 19.2 and Fig. 19.3. WordNet hyponymy relations corresp ond to the notion of

DRAFTSection19.4. EventParticipants: SemanticRolesandSelec tionalRestrictions 631
Relation Alsocalled Deﬁnition Example
Hypernym Superordinate Fromconceptstosuperordinates breakfast1→meal1
Hyponym Subordinate Fromconceptstosubtypes meal1→lunch1
MemberMeronym Has-Member Fromgroupsto theirmembers faculty2→professor1
Has-Instance Fromconceptstoinstancesoftheconcept composer1→Bach1
Instance Frominstancestotheirconcepts Austen1→author1
MemberHolonym Member-Of Frommemberstotheirgroups copilot1→crew1
Part Meronym Has-Part Fromwholesto parts table2→leg3
Part Holonym Part-Of Frompartstowholes course7→meal1
Antonym Opposites leader1→follower1
Figure19.2 Noun relations inWordNet.
Relation Deﬁnition Example
Hypernym Fromeventstosuperordinateevents ﬂy9→travel5
Troponym Fromaverb(event)toa speciﬁcmannerelaboration ofthatverb walk1→stroll1
Entails Fromverbs(events)totheverbs(events)theyentai l snore1→sleep1
Antonym Opposites increase1⇐⇒decrease1
Figure19.3 Verbrelations inWordNet.

immediatehyponymydiscussedonpage628. Eachsynsetisrel atedtoitsimmediately
more general and more speciﬁc synsets via direct hypernym an d hyponym relations.
These relations can be followed to produce longer chains of m ore general or more
speciﬁcsynsets. Fig.19.4showshypernymchainsfor bass3andbass7.
In this depiction of hyponymy, successively more general sy nsets are shown on
successive indented lines. The ﬁrst chain starts from the co ncept of a human bass

singer. Its immediate superordinate is a synset correspond ing to the generic concept
of a singer. Following this chain leads eventually to concep ts such as entertainer and
person. The second chain, which starts from musical instrument, ha s a completely
different path leading eventually to such concepts as music al instrument, device and
physical object. Both paths do eventually join at the very ab stract synset whole, unit ,
and then proceed together to entitywhich is the top (root) of the noun hierarchy (in

WordNetthisrootisgenerallycalledthe unique beginner ). Unique beginner
19.4 Event Participants: Semantic Roles and Selectional
Restrictions
An important aspect of lexical meaning has to do with the sema ntics of events. When
we discussed events in Ch. 17, we introduced the importance o f predicate-argument
structure for representingan event, and the use of Davidson ian reiﬁcation of events to
represent each participant distinctly from the event itsel f. We turn in this section to
representing the meaning of these event participants orarguments . We introduce two

kinds of semantic constraints on the arguments of event pred icates:semantic roles
andselectionalrestrictions . Webeginwithaparticularmodelofsemanticrolescalled

DRAFT632 Chapter 19. LexicalSemantics
Sense 3
bass, basso --
(an adult male singer with the lowest voice)
=> singer, vocalist, vocalizer, vocaliser
=> musician, instrumentalist, player
=> performer, performing artist
=> entertainer
=> person, individual, someone...
=> organism, being
=> living thing, animate thing,
=> whole, unit
=> object, physical object
=> physical entity
=> entity
=> causal agent, cause, causal agency
=> entity
Sense 7
bass --
(the member with the lowest range of a family of
musical instruments)
=> musical instrument, instrument
=> device
=> instrumentality, instrumentation
=> artifact, artefact
=> entity
Figure19.4 Hyponymychainsfortwoseparatesensesofthelemma bass. Notethatthechains
are completely distinct,only converging at the veryabstra ct levelwhole, unit .
thematicroles .
19.4.1 Thematic Roles
Consider how we represented the meaning of arguments in Ch. 1 7 for sentences like
these:

=> device
=> entity
these:
(19.19)Sashabrokethewindow.
(19.20)Patopenedthedoor.
Aneo-Davidsonianeventrepresentationofthesetwo senten ceswouldbe:
∃e,x,yIsa(e,Breaking )∧Breaker (e,Sasha)
∧BrokenThing (e,y)∧Isa(y,Window )
∃e,x,yIsa(e,Opening )∧Opener (e,Pat)
∧OpenedThing (e,y)∧Isa(y,Door)

DRAFTSection19.4. EventParticipants: SemanticRolesandSelec tionalRestrictions 633
Thematic Role Deﬁnition
AGENT Thevolitionalcauserofan event
EXPERIENCER Theexperiencerofanevent
FORCE Thenon-volitionalcauseroftheevent
THEME Theparticipantmostdirectlyaffectedbyanevent
RESULT Theendproductofanevent
CONTENT Thepropositionorcontentofa propositionalevent
INSTRUMENT Aninstrumentusedinanevent
BENEFICIARY Thebeneﬁciaryofanevent
SOURCE Theoriginofthe objectofa transferevent
GOAL Thedestinationofan objectofa transferevent
Figure19.5 Some commonly-used thematic roles withtheirdeﬁnitions.
In this representation, the roles of the subjects of the verb sbreakandopenare
BreakerandOpenerrespectively. These deep roles are speciﬁc to each possible kind Deeproles
of event; Breaking eventshave Breakers,Openingeventshave Openers,Eatingevents
haveEaters,andsoon.
Ifwearegoingtobeabletoanswerquestions,performinfere nces,ordoanyfurther
kinds of natural language understanding of these events, we ’ll need to know a little

more about the semantics of these arguments. Breakers andOpenershave something
incommon. Theyarebothvolitionalactors,oftenanimate,a ndtheyhavedirectcausal
responsibilityfortheirevents.
Thematic roles are one attempt to capture this semantic commonality betwee n Thematic role
Breakers andEaters. We say that the subjects of both these verbs are agents. Thus Agent
AGENTis the thematic role which representsan abstract idea such a s volitionalcausa-
tion. Similarly,thedirectobjectsofboththeseverbs,the BrokenThing andOpenedThing ,

arebothprototypicallyinanimateobjectswhichareaffect edinsomewaybytheaction.
Thethematicrolefortheseparticipantsis theme. Theme
Thematicrolesareoneof theoldest linguisticmodels,prop osedﬁrst by theIndian
grammarian Panini sometime between the 7th and 4th centurie s BCE. Their modern
formulationisduetoFillmore(1968)andGruber(1965). Alt houghthereisnouniver-
sally agreed-upon set of thematic roles, Fig. 19.5 and Fig. 1 9.6 present a list of some
thematic roles which have been used in various computationa l papers, together with
roughdeﬁnitionsandexamples.

19.4.2 DiathesisAlternations
Themainreasoncomputationalsystemsusethematicroles,a ndsemanticrolesingen-
eral, is to act as a shallow semantic language that can let us m ake simple inferences
that aren’t possible from the pure surface string of words, o r even the parse tree. For
example,if a documentsaysthat CompanyA acquiredCompanyB , we’dlike to know
that this answers the query Was Company B acquired? despite the fact that the two
sentences have very different surface syntax. Similarly, t his shallow semantics might

act asa usefulintermediatelanguageinmachinetranslatio n.
Thus thematic roles are used in helping us generalize over di fferent surface real-

DRAFT634 Chapter 19. LexicalSemantics
Thematic Role Example
AGENT Thewaiter spilledthesoup.
EXPERIENCER Johnhasa headache.
FORCE Thewind blowsdebrisfromthemallintoouryards.
THEME OnlyafterBenjaminFranklinbroke theice...
RESULT The French governmenthas built a regulation-size baseball di-
amond...
CONTENT Monaasked “Youmet Mary Annata supermarket”?
INSTRUMENT He turned to poaching catﬁsh, stunning them with a shocking
device...
BENEFICIARY WheneverAnnCallahanmakeshotelreservations forherboss ...
SOURCE Iﬂew infrom Boston .
GOAL Idroveto Portland .
Figure19.6 Some prototypical examples ofvarious thematic roles.
izationsofpredicatearguments. For examplewhile the AGENTis oftenrealizedas the
subject of the sentence, in other cases the THEMEcan be the subject. Consider these
possiblerealizationsofthe thematicargumentsoftheverb break:
(19.21)John
AGENTbrokethe window.
THEME
(19.22)John
AGENTbrokethe window
THEMEwith arock.
INSTRUMENT
(19.23)Therock
INSTRUMENTbrokethedoor.
THEME
(19.24)Thewindow
THEMEbroke.

THEME
THEME
(19.25)Thewindow
THEMEwasbrokenbyJohn.
AGENT
Theexamplesabovesuggestthat breakhas(atleast)thepossiblearguments AGENT,
THEME,andINSTRUMENT . Thesetofthematicroleargumentstakenbyaverbisoften
called the thematic grid ,θ-grid, orcase frame . We can see that there are (among Thematic grid
Case frame others)the followingpossibilitiesfortherealizationof these argumentsof break:
•AGENT:Subject, THEME:Object
•AGENT:Subject, THEME:Object, INSTRUMENT :PPwith
•INSTRUMENT :Subject, THEME:Object
•THEME:Subject

AGENT
It turns out that many verbs allow their thematic roles to be r ealized in various
syntactic positions. For example, verbs like givecan realize the THEMEandGOAL
argumentsintwo differentways:
(19.26) a.Doris
AGENTgavethebook
THEMEto Cary.
GOAL
b.Doris
AGENTgaveCary
GOALthe book.
THEME

DRAFTSection19.4. EventParticipants: SemanticRolesandSelec tionalRestrictions 635
Thesemultipleargumentstructurerealizations(thefactt hatbreakcantake AGENT,
INSTRUMENT , orTHEMEas subject, and givecan realize its THEMEandGOALin
either order) are called verb alternations ordiathesis alternations . The alternation Verbalternation
Diathesisalternation weshowedabove give,thedativealternation ,seemstooccurwithparticularsemantic
Dativealternation classes of verbs, including “verbs of future having” ( advance,allocate,offer,owe),
“send verbs” ( forward,hand,mail), “verbs of throwing” ( kick,pass,throw), and so
on. Levin (1993) is a reference book which lists for a large se t of English verbs the
semantic classes they belong to and the various alternation s that they participate in.
These lists of verb classes have been incorporated into the o nline resource VerbNet
(Kipperet al.,2000).
19.4.3 Problems withThematic Roles
Representingmeaningatthethematicrolelevelseemslikei tshouldbeusefulindealing

with complicationslike diathesis alternations. But despi te this potential beneﬁt, it has
proved very difﬁcult to come up with a standard set of roles, a nd equally difﬁcult to
producea formaldeﬁnitionofroleslike AGENT,THEME,orINSTRUMENT .
Forexample,researchersattemptingtodeﬁnerolesetsofte nﬁndtheyneedtofrag-
ment a rolelike AGENTorTHEMEinto manyspeciﬁc roles. Levin andRappaportHo-
vav(2005)summarizesanumberofsuchcases,suchasthefact thereseemtobeatleast
two kindsof INSTRUMENTS ,intermediary instrumentsthat canappearassubjectsand

enablinginstrumentsthatcannot:
(19.27) Thecookopenedthejarwith thenewgadget.
Thenewgadgetopenedthejar.
(19.28) Shellyate theslicedbananawith afork.
*Theforkatethesliced banana.
Inadditiontothefragmentationproblem,therearecaseswh erewe’dliketoreason
aboutandgeneralizeacrosssemanticroles,buttheﬁnitedi scretelistsofrolesdon’tlet
usdothis.
Finally,it hasprovedverydifﬁcult to formallydeﬁne the se manticroles. Consider
theAGENTrole; most cases of AGENTSare animate, volitional, sentient, causal, but
anyindividualnounphrasemightnotexhibitallofthese pro perties.

usdothis.
Theseproblemshaveledmostresearchtoalternativemodels ofsemanticroles. One
suchmodelisbasedondeﬁning generalizedsemanticroles thatabstractoverthespe-Generalizedsemantic role
ciﬁc thematicroles. Forexample PROTO-AGENTandPROTO-PATIENTare generalized Proto-agent
Proto-patient roles that express roughly agent-like and roughly patient- like meanings. These roles
are deﬁned, not by necessary and sufﬁcient conditions, but r ather by a set of heuristic
featuresthataccompanymoreagent-likeormorepatient-li kemeanings. Thusthemore

an argument displays agent-like properties (intentionali ty, volitionality, causality, etc)
thegreaterlikelihoodtheargumentcanbelabeleda PROTO-AGENT. Themorepatient-
like properties (undergoing change of state, causally affe cted by another participant,
stationaryrelativetootherparticipants,etc),thegreat erlikelihoodtheargumentcanbe
labeleda PROTO-PATIENT.
In addition to using proto-roles, many computational model s avoid the problems
with thematic roles by deﬁning semantic roles that are speci ﬁc to a particular verb, or

DRAFT636 Chapter 19. LexicalSemantics
speciﬁcto aparticularset ofverbsornouns.
In the next two sections we will describe two commonly used le xical resources
which make use of some of these alternative versions of seman tic roles. PropBank
uses both proto-rolesand verb-speciﬁcsemantic roles. FrameNet uses frame-speciﬁc
semanticroles.
19.4.4 The PropositionBank
ThePropositionBank , generallyreferredto as PropBank , is a resource of sentences PropBank
annotated with semantic roles. The English PropBank labels all the sentences in the
Penn TreeBank; there is also a Chinese PropBank which labels sentences in the Penn
Chinese TreeBank. Because of the difﬁculty of deﬁning a univ ersal set of thematic
roles, the semantic roles in PropBank are deﬁned with respec t to an individual verb
sense. Each sense of each verb thus has a speciﬁc set of roles, which are given only
numbers rather than names: Arg0,Arg1 Arg2 , and so on. In general, Arg0is used

to represent the PROTO-AGENT, andArg1thePROTO-PATIENT; the semantics of the
otherrolesare speciﬁc to eachverbsense. Thusthe Arg2ofoneverbis likelyto have
nothingin commonwith the Arg2ofanotherverb.
HerearesomeslightlysimpliﬁedPropBankentriesforonese nse eachoftheverbs
agreeandfall; the deﬁnitionsfor each role (“Other entity agreeing”,“am ount fallen”)
areinformalglossesintendedtobe readbyhumans,ratherth anformaldeﬁnitions.
(19.29)Frameset agree.01
Arg0: Agreer
Arg1: Proposition
Arg2: Otherentityagreeing

Ex1: [Arg0The group] agreed[Arg1it wouldn’t make an offer unless it had
GeorgiaGulf’sconsent].
Ex2: [ArgM-TMP Usually] [Arg0John]agrees[Arg2with Mary] [Arg1on ev-
erything].
(19.30)fall.01“movedownward”
Arg1: Logicalsubject,patient,thingfalling
Arg2: Extent,amountfallen
Arg3: startpoint
Arg4: endpoint,endstate ofarg1
ArgM-LOC: medium
Ex1: [ Arg1Sales]fell[Arg4to$251.2million][ Arg3from$278.7million].
Ex1: [Arg1Theaveragejunkbond] fell[Arg2by4.2%].

Note that there is no Arg0 role for fall, because the normal subject of fallis a
PROTO-PATIENT.
The PropBank semantic roles can be useful in recovering shal low semantic infor-
mationaboutverbalarguments. Considertheverb increase:
(19.31)increase.01 “goupincrementally”

DRAFTSection19.4. EventParticipants: SemanticRolesandSelec tionalRestrictions 637
Arg0: causerofincrease
Arg1: thingincreasing
Arg2: amountincreasedby,EXT,orMNR
Arg3: start point
Arg4: endpoint
APropBanksemanticrolelabelingwouldallowustoinferthe commonalityinthe
eventstructuresofthefollowingthreeexamples,i.e.,tha tineachcase BigFruit Co. is
theAGENT,andthepriceofbananas istheTHEME,despitethedifferingsurfaceforms.
(19.32)[Arg0Big FruitCo. ] increased[Arg1thepriceofbananas].
(19.33)[Arg1Thepriceofbananas]wasincreasedagain[ Arg0byBig FruitCo. ]
(19.34)[Arg1Thepriceofbananas]increased[ Arg25%].
19.4.5 FrameNet
While makinginferencesaboutthe semanticcommonalitiesa crossdifferentsentences
withincreaseisuseful,itwouldbeevenmoreusefulifwecouldmakesuchin ferences
inmanymoresituations,acrossdifferentverbs,andalsobe tweenverbsandnouns.
Forexample,we’dlike toextractthesimilaritybetweenthe se threesentences:
(19.35)[Arg1Thepriceofbananas]increased[Arg25%].
(19.36)[Arg1Thepriceofbananas]rose[ Arg25%].

(19.37)Therehasbeena [ Arg25%]rise [ Arg1in thepriceofbananas].
Note that the second example uses the different verb rise, and the third example
uses the noun rather than the verb rise. We’d like a system to recognize that the price
of bananas is what went up, and that 5%is the amount it went up, no matter whether
the5%appearsastheobjectoftheverb increased orasanominalmodiﬁerofthenoun
rise.
TheFrameNet projectisanothersemanticrolelabelingprojectthatatte mptstoad- FrameNet
dressjust these kindsof problems(Bakeret al., 1998;Lowee t al., 1997;Ruppenhofer

rise.
et al., 2006). Where roles in the PropBank project are speciﬁ c to an individual verb,
roles in the FrameNet project are speciﬁc to a frame. Aframeis a script-like struc- Frame
ture, which instantiates a set of frame-speciﬁc semantic ro les called frame elements . Frame element
Eachword evokesa frameand proﬁlessome aspect ofthe framea nd its elements. For
example,the changepositiononascaleframeisdeﬁnedasfollows:
ThisframeconsistsofwordsthatindicatethechangeofanIt em’sposition

on a scale (the Attribute) from a starting point (Initial value) to an end
point(Final value).
Some of the semantic roles (frame elements) in the frame, sep arated into coreroles
andnon-core roles , are deﬁned as follows (deﬁnitions are taken from the FrameN et
labelersguide(Ruppenhoferet al.,2006)).

DRAFT638 Chapter 19. LexicalSemantics
CoreRoles
ATTRIBUTE The ATTRIBUTE isascalar propertythatthe I TEMpossesses.
DIFFERENCE The distance by which an I TEMchanges its position on the
scale.
FINALSTATE A description that presents the I TEM’s state after the change in
the ATTRIBUTE ’svalueasanindependentpredication.
FINALVALUE Thepositiononthescale wheretheItemendsup.
INITIAL STATEA description that presents the I TEM’s state before the change
inthe A TTRIBUTE ’svalueasanindependentpredication.
INITIAL VALUEThe initial position on the scale from which the I TEMmoves
away.
ITEM Theentitythathasapositiononthescale.
VALUERANGE A portion of the scale, typically identiﬁed by its end points ,
alongwhichthevaluesofthe A TTRIBUTE ﬂuctuate.
Some Non-CoreRoles
DURATION Thelengthoftime overwhichthechangetakesplace.
SPEED Therateofchangeofthe V ALUE.
GROUP The GROUPin which an I TEMchanges the value of an A T-
TRIBUTE in aspeciﬁedway.
Herearesomeexamplesentences:

away.
(19.38)[ITEMOil]rose[ATTRIBUTE inprice]inprice[ DIFFERENCE by2%].
(19.39)[ITEMIt]hasincreased [FINALSTATEto havingthem1 daya month].
(19.40)[ITEMMicrosoftshares] fell[FINALVALUEto75/8].
(19.41)[ITEMColoncancerincidence] fell[DIFFERENCE by50%][ GROUPamong
men].
(19.42)asteadyincrease [INITIAL VALUEfrom9.5][ FINALVALUEto 14.3][ ITEMin
dividends]
(19.43)a[DIFFERENCE 5%][ITEMdividend] increase...
Note from these example sentences that the frame includes ta rget words like rise,
fall,andincrease. Infact,thecompleteframeconsistsofthefollowingwords :

men].
VERBS: dwindle move soar escalation shift
advance edge mushroom swell explosion tumble
climb explode plummet swing fall
decline fall reach triple ﬂuctuation ADVERBS:
decrease ﬂuctuate rise tumble gain increasingly
diminish gain rocket growth
dip grow shift NOUNS: hike
double increase skyrocket decline increase
drop jump slide decrease rise
FrameNetalsocodesrelationshipsbetweenframesandframe elements. Framescan
inheritfromeachother,andgeneralizationsamongframeel ementsin differentframes

can be captured by inheritance as well. Other relations betw een frames like causa-
tionarealsorepresented. Thusthereisa Causechangeofpositiononascaleframe
whichislinkedtothe Changeofpositiononascaleframebythe causerelation,but
addsan A GENTroleandisusedforcausativeexamplessuchasthefollowing :

DRAFTSection19.4. EventParticipants: SemanticRolesandSelec tionalRestrictions 639
(19.44)[AGENTThey]raised[ITEMthepriceoftheirsoda][ DIFFERENCE by2%].
Together, these two frames would allow an understanding sys tem to extract the
common event semantics of all the verbal and nominal causati ve and non-causative
usages.
Ch. 20 will discuss automatic methods for extracting variou s kinds of semantic
roles; indeed one main goal of PropBank and FrameNet is to pro videtrainingdata for
suchsemanticrolelabelingalgorithms.
19.4.6 Selectional Restrictions
Semantic roles gave us a way to express some of the semantics o f an argument in its
relation to the predicate. In this section we turn to another way to express semantic
constraints on arguments. A selectional restriction is a kind of semantic type con-
straint that a verbimposeson the kind of conceptsthat are al lowed to ﬁll its argument
roles. Considerthe twomeaningsassociatedwiththe follow ingexample:
(19.45)Iwantto eatsomeplacethat’scloseto ICSI.

There are two possible parses and semantic interpretations for this sentence. In the
sensible interpretation eatis intransitive and the phrase someplace that’sclose to ICSI
is an adjunctthat givesthe locationof theeating event. In t he nonsensical speaker-as-
Godzillainterpretation, eatis transitive and the phrase someplace that’s close to ICSI
is the direct object and the THEMEof the eating, like the NP Malaysian food in the
followingsentences:
(19.46) I wantto eatMalaysianfood.

How do we know that someplace that’s close to ICSIisn’t the direct object in this
sentence? Oneuseful cueis the semantic fact that the THEMEof EATINGeventstends
to be something that is edible. This restriction placed by the verb eaton the ﬁller of
itsTHEMEargument, is called a selectional restriction . A selectional restriction is aSelectionalrestriction
constraintonthesemantictypeofsomeargument.
Selectional restrictionsare associated with senses, not e ntire lexemes. We can see
thisinthefollowingexamplesofthelexeme serve:

(19.47) Well, therewasthe timetheyservedgreen-lippedmusselsfr omNew
Zealand.
(19.48) Which airlinesserveDenver?
Example (19.47) illustrates the cooking sense of serve, which ordinarily restricts its
THEMEto be some kind foodstuff. Example (19.48) illustrates the provides a com-
mercial service to sense ofserve, which constrains its THEMEto be some type of ap-
propriate location. We will see in Ch. 20 that the fact that se lectional restrictions are
associatedwithsensescanbeusedasa cuetohelpinwordsens edisambiguation.

Zealand.
Selectionalrestrictionsvarywidelyintheirspeciﬁcity. Theverbimagine,forexam-
ple, imposes strict requirements on its AGENTrole (restricting it to humans and other
animateentities)butplacesveryfewsemanticrequirement sonits THEMErole. Averb
likediagonalize ,on the other hand, places a veryspeciﬁc constraint on the ﬁl ler of its
THEMErole: it has to be a matrix, while the argumentsof the adjecti vesodorlessare
restrictedtoconceptsthat couldpossessanodor:

DRAFT640 Chapter 19. LexicalSemantics
(19.49)Inrehearsal,Ioftenask themusiciansto imagineatennisgame.
(19.50)Radonisan odorlessgasthatcan’tbedetectedbyhumansenses.
(19.51)Todiagonalizea matrixistoﬁnditseigenvalues.
These examples illustrate that the set of concepts we need to represent selectional
restrictions (being a matrix, being able to possess an oder, etc) is quite open-ended.
This distinguishes selectional restrictions from other fe atures for representing lexical
knowledge,likeparts-of-speech,whicharequitelimitedi nnumber.
RepresentingSelectionalRestrictions
One way to capture the semantics of selectional restriction s is to use and extend the
event representation of Ch. 17. Recall that the neo-Davidso nian representation of an
event consists of a single variable that stands for the event , a predicate denoting the
kind of event, and variables and relations for the event role s. Ignoring the issue of
theλ-structures, and using thematic roles rather than deep even t roles, the semantic

contributionofa verblike eatmightlooklike thefollowing:
∃e,x,yEating (e)∧Agent(e,x)∧Theme (e,y)
With this representation, all we know about y, the ﬁller of the THEMErole, is that it
is associated with an Eatingevent via the Themerelation. To stipulate the selectional
restrictionthat ymust besomethingedible,wesimplyaddanewtermto thateffe ct:
∃e,x,yEating (e)∧Agent(e,x)∧Theme (e,y)∧Isa(y,EdibleThing )
When a phrase like ate a hamburger is encountered,a semantic analyzercan form the
followingkindofrepresentation:

∃e,x,yEating (e)∧Eater(e,x)∧Theme (e,y)∧Isa(y,EdibleThing )
∧Isa(y,Hamburger )
This representation is perfectly reasonable since the memb ership of yin the category
Hamburger isconsistentwithitsmembershipinthecategory EdibleThing ,assuminga
reasonableset of facts in the knowledgebase. Correspondin gly,the representationfor
a phrase such as ate a takeoff would be ill-formed because membership in an event-
like category such as Takeoffwould be inconsistent with membership in the category
EdibleThing .

While this approach adequately captures the semantics of se lectional restrictions,
there are two practical problems with its direct use. First, usingFOPCto perform
the simple task of enforcing selectional restrictions is ov erkill. There are far simpler
formalisms that can do the job with far less computational co st. The second problem
is that this approach presupposes a large logical knowledge -base of facts about the
concepts that make up selectional restrictions. Unfortuna tely, although such common

sense knowledge-bases are being developed, none currently have the kind of scope
necessaryto thetask.
A more practical approach is to state selectional restricti ons in terms of WordNet
synsets,ratherthanlogicalconcepts. Eachpredicatesimp lyspeciﬁesaWordNetsynset

DRAFTSection19.5. PrimitiveDecomposition 641
Sense 1
hamburger, beefburger --
(a fried cake of minced beef served on a bun)
=> sandwich
=> snack food
=> dish
=> nutriment, nourishment, nutrition...
=> food, nutrient
=> substance
=> matter
=> entity
Figure19.7 Evidence from WordNetthat hamburgers are edible.
as the selectional restriction on each of its arguments. A me aning representation is
well-formedif theroleﬁllerwordisa hyponym(subordinate )ofthissynset.
Forourateahamburger example,forexample,wecouldsettheselectionalrestric-
tion on the THEMErole of the verb eatto the synset{food, nutrient}, glossed as any
substancethatcanbemetabolizedbyananimaltogiveenergy andbuildtissue : Luck-
ily, the chain ofhypernymsfor hamburger shownin Fig. 19.7revealsthat hamburgers
areindeedfood. Again,theﬁllerofaroleneednotmatchther estrictionsynsetexactly,
it just needsto havethe synsetasoneofits superordinates.
We can apply this approach to the THEMEroles of the verbs imagine,liftanddi-

agonalize , discussed earlier. Let us restrict imagine’sTHEMEto the synset{entity},
lift’sTHEMEto{physical entity}anddiagonalize to{matrix}. This arrangementcor-
rectly permits imagine a hamburger andlift a hamburger , while also correctly ruling
outdiagonalizea hamburger .
OfcourseWordNetisunlikelytohavetheexactlyrelevantsy nsetstospecifyselec-
tionalrestrictionsforallpossiblewordsofEnglish;othe rtaxonomiesmayalsobeused.
Inaddition,itis possibletolearnselectionalrestrictio nsautomaticallyfromcorpora.

WewillreturntoselectionalrestrictionsinCh.20wherewe introducetheextension
toselectionalpreferences ,whereapredicatecanplaceprobabilisticpreferencesrat her
thanstrict deterministicconstraintsonitsarguments.
19.5 PrimitiveDecomposition
Back at the beginning of the chapter, we said that one way of de ﬁning a word is to
decompose its meaning into a set of primitive semantics elem ents or features. We
saw one aspect of this methodin ourdiscussion of ﬁnite lists of thematic roles(agent,

patient, instrument,etc). We turn nowto a brief discussion of how this kindof model,
calledprimitive decomposition , orcomponential analysis , could be applied to the
meanings of all words. Wierzbicka (1992, 1996) shows that th is approach dates back
at least tocontinentalphilosopherslikeDescartesandLei bniz.
Considertryingtodeﬁnewordslike hen,rooster,orchick. Thesewordshavesome-

DRAFT642 Chapter 19. LexicalSemantics
thing in common (they all describe chickens) and something d ifferent (their age and
sex). This can be represented by using semantic features , symbols which represent Semantic feature
somesortofprimitivemeaning:
hen +female, +chicken, +adult
rooster -female, +chicken, +adult
chick +chicken, -adult
Anumberofstudiesofdecompositionalsemantics,especial lyinthecomputational
literature,havefocusedonthemeaningofverbs. Considert heseexamplesfortheverb
kill:
(19.52)Jimkilledhisphilodendron.
(19.53)Jimdidsomethingtocausehisphilodendrontobecomenotali ve.
There is a truth-conditional (‘propositional semantics’) perspective from which these
two sentenceshavethe samemeaning. Assumingthisequivale nce,we couldrepresent
themeaningof killas:
(19.54) KILL(x,y)⇔CAUSE(x,BECOME(NOT(ALIVE(y))))
thususingsemanticprimitiveslike do,cause,becomenot ,andalive.
Indeed, one such set of potential semantic primitiveshas be en used to account for

kill:
some of the verbal alternations discussed in Sec. 19.4.2(La koff, 1965; Dowty, 1979).
Considerthefollowingexamples.
(19.55)Johnopenedthedoor. ⇒(CAUSE(John(BECOME(OPEN(door)))))
(19.56)Thedooropened.⇒(BECOME(OPEN(door)))
(19.57)Thedooris open.⇒(OPEN(door))
Thedecompositionalapproachassertsthata singlestate-l ikepredicateassociatedwith
openunderliesalloftheseexamples. Thedifferencesamongthem eaningsoftheseex-
amplesarisesfromthe combinationofthissingle predicate with theprimitives CAUSE
andBECOME.

Whilethisapproachtoprimitivedecompositioncanexplain thesimilaritybetween
states and actions, or causative and non-causativepredica tes, it still relies on having a
very large number of predicates like open. More radical approaches choose to break
down these predicates as well. One such approach to verbal pr edicate decomposition
isConceptualDependency (CD),asetoftenprimitivepredicates,showninFig.19.8.Conceptual
Dependency
Below is an example sentence along with its CDrepresentation. The verb brought

is translated into the two primitives ATRANSandPTRANSto indicate the fact that the
waiterbothphysicallyconveyedthechecktoMaryandpassed controlofittoher. Note
thatCDalsoassociatesaﬁxedsetofthematicroleswitheachprimit ivetorepresentthe
variousparticipantsin theaction.
(19.58)Thewaiter broughtMarythe check.
∃x,yAtrans (x)∧Actor(x,Waiter )∧Object (x,Check )∧To(x,Mary)
∧Ptrans (y)∧Actor(y,Waiter )∧Object (y,Check )∧To(y,Mary)

DRAFTSection19.6. Advancedconcepts: Metaphor 643
Primitive Deﬁnition
ATRANS Theabstracttransferofpossessionorcontrolfromoneenti tyto
another.
PTRANS Thephysicaltransferofanobjectfromonelocationtoanoth er
MTRANS The transfer of mental concepts between entities or within a n
entity.
MBUILD Thecreationofnewinformationwithinanentity.
PROPEL Theapplicationofphysicalforcetomoveanobject.
MOVE Theintegralmovementofabodypartbyananimal.
INGEST Thetakingin ofasubstancebyananimal.
EXPEL Theexpulsionofsomethingfromananimal.
SPEAK Theactionofproducinga sound.
ATTEND Theactionoffocusinga senseorgan.
Figure19.8 A setof conceptual dependency primitives.
There are also sets of semantic primitives that cover more th an just simple nouns
andverbs. Thefollowinglist comesfromWierzbicka(1996):
substantives: I,YOU,SOMEONE ,SOMETHING ,PEOPLE
mentalpredicates: THINK,KNOW,WANT,FEEL,SEE,HEAR
speech: SAY
determinersandquantiﬁers: THIS,THE SAME ,OTHER,ONE,TWO,
MANY(MUCH),ALL,SOME,MORE
actionsandevents: DO,HAPPEN
evaluators: GOOD,BAD

descriptors: BIG,SMALL
time: WHEN,BEFORE,AFTER
space: WHERE,UNDER,ABOVE,
partonomyandtaxonomy: PART(OF),KIND(OF)
movement,existence,life: MOVE,THERE IS ,LIVE
metapredicates: NOT,CAN,VERY
interclausallinkers: IF,BECAUSE ,LIKE
space: FAR,NEAR,SIDE,INSIDE,HERE
time: A LONG TIME ,A SHORT TIME ,NOW
imaginationandpossibility: IF...WOULD,CAN,MAYBE
Because of the difﬁculty of coming up with a set of primitives that can represent
all possible kinds of meanings, most current computational linguistic work does not

use semantic primitives. Instead, most computational work tends to use the lexical
relationsofSec.19.2todeﬁnewords.
19.6 Advancedconcepts: Metaphor
We use a metaphor when we refer to and reason about a concept or domain us- Metaphor
ing words and phrases whose meanings come from a completely d ifferent domain.

DRAFT644 Chapter 19. LexicalSemantics
Metaphor is similar to metonymy , which we introduced as the use of one aspect of
a concept or entity to refer to other aspects of the entity. In Sec. 19.1 we introduced
metonymieslike thefollowing,
(19.59)Author( Jane Austenwrote Emma )↔WorksofAuthor( Ireally love Jane Austen ).
in whichtwo sensesof a polysemouswordaresystematically r elated. In metaphor,by
contrast, there is a systematic relation between two comple tely different domains of
meaning.
Metaphorispervasive. ConsiderthefollowingWSJsentence :
(19.60) Thatdoesn’tscareDigital, whichhasgrownto betheworld’s second-largest
computermakerbypoachingcustomersofIBM’smid-rangemac hines.
The verb scaremeans ‘to cause fear in’, or ‘to cause to lose courage’. For th is
sentencetomakesense,ithastobethecasethatcorporation scanexperienceemotions
likefearorcourageaspeopledo. Ofcoursetheydon’t,butwe certainlyspeakofthem
andreasonaboutthemasiftheydo. Wecanthereforesaythatt hisuseof scareisbased

meaning.
on a metaphorthat allows us to view a corporationas a person, which we will refer to
theCORPORATION AS PERSON metaphor.
This metaphor is neither novel nor speciﬁc to this use of scare. Instead, it is a
fairlyconventionalway tothinkaboutcompaniesandmotiva testheuse of resuscitate ,
hemorrhage andmindin thefollowingWSJ examples:
(19.61) FuquaIndustriesInc. said TritonGroupLtd.,acompanyit he lped
resuscitate ,hasbegunacquiringFuquashares.
(19.62) AndFordwas hemorrhaging ;itslosseswouldhit$1.54billionin1980.

(19.63) But if it changedits mind, however,it woulddoso forinvestmentreasons,
the ﬁlingsaid.
Each of these examples reﬂects an elaborated use of the basic CORPORATION AS
PERSONmetaphor. The ﬁrst two examples extend it to use the notion of health to
express a corporation’sﬁnancial status, while the third ex ample attributes a mind to a
corporationto capturethenotionofcorporatestrategy.
Metaphorical constructs such as CORPORATION AS PERSON are known as con-

ventional metaphors . Lakoff and Johnson (1980) argue that many if not most of theConventionalmetaphor
metaphorical expressions that we encounter every day are mo tivated by a relatively
small numberofthesesimpleconventionalschemas.
19.7 Summary
This chapter has covered a wide range of issues concerning th e meanings associated
withlexicalitems. Thefollowingareamongthehighlights:
•Lexical semantics is the study of the meaning of words, and the systematic
meaning-relatedconnectionsbetweenwords.

•Aword sense is the locus of word meaning; deﬁnitions and meaning relatio ns
aredeﬁnedat thelevelofthe wordsense ratherthanwordform sasa whole.

DRAFTSection19.7. Summary 645
•Homonymy is the relation between unrelated senses that share a form, w hile
polysemy is therelationbetweenrelatedsensesthatsharea form.
•Synonymy holdsbetweendifferentwordswith thesamemeaning.
•Hyponymy relations hold between words that are in a class-inclusion r elation-
ship.
•Semantic ﬁelds are used to capture semantic connectionsamonggroupsof lex -
emesdrawnfroma singledomain.
•WordNet isa largedatabaseoflexicalrelationsforEnglishwords.
•Semantic roles abstract away from the speciﬁcs of deep semantic rolesby gen -
eralizingoversimilarrolesacrossclasses ofverbs.
•Thematic roles are a model of semantic roles based on a single ﬁnite list of
roles. Other semantic role models include per-verb semanti c roles lists and
proto-agent /proto-patient both of which are implemented in PropBank , and
per-framerolelists, implementedin FrameNet .
•Semantic selectional restrictions allow words (particularly predicates) to post
constraintsonthesemanticpropertiesoftheirargumentwo rds.

•Primitive decomposition is another way to represent the meaning of word, in
termsofﬁnitesetsofsub-lexicalprimitives.
Cruse (2004) is a useful introductory linguistic text on lex ical semantics. Levin and
Rappaport Hovav (2005) is a research survey covering argume nt realization and se-
mantic roles. Lyons (1977) is another classic reference. Co llections describing com-
putational work on lexical semantics can be found in Pustejo vsky and Bergler (1992),
Saint-DizierandViegas(1995)andKlavans(1995).

The most comprehensivecollection of work concerningWordN et can be found in
Fellbaum (1998). There have been many efforts to use existin g dictionariesas lexical
resources. One of the earliest was Amsler’s (1980, 1981)use of the Merriam Webster
dictionary. The machine readable version of Longman’s Dict ionary of Contemporary
English has also been used (Boguraev and Briscoe, 1989). See Pustejovsky (1995),
Pustejovsky and Boguraev (1996), Martin (1986) and Copesta ke and Briscoe (1995),

inter alia, for computational approaches to the representa tion of polysemy. Puste-
jovsky’s theory of the Generative Lexicon , and in particular his theory of the qualiaGenerativeLexicon
structure ofwords,isanotherwayofaccountingforthedynamicsystem aticpolysemy Qualiastructure
ofwordsincontext.
As we mentioned earlier, thematic roles are one of the oldest linguistic models,
proposed ﬁrst by the Indian grammarian Panini sometimes bet ween the 7th and 4th
centuriesBCE.TheirmodernformulationisduetoFillmore( 1968)andGruber(1965).

Fillmore’s work had a large and immediate impact on work in na tural language pro-
cessing,asmuchearlyworkinlanguageunderstandinguseds omeversionofFillmore’s
caseroles(e.g.,Simmons(1973,1978,1983)). Fillmore’se xtensionofthisworktothe

DRAFT646 Chapter 19. LexicalSemantics
FrameNet project is described in Baker et al. (1998), Naraya nan et al. (1999), and
Bakeret al.(2003).
Workonselectionalrestrictionsasawayofcharacterizing semanticwell-formedness
beganwithKatzandFodor(1963). McCawley(1968)wastheﬁrs ttopointoutthatse-
lectionalrestrictionscouldnotberestrictedtoaﬁnite li stofsemanticfeatures,buthad
tobedrawnfromalargerbaseofunrestrictedworldknowledg e.
Lehrer (1974) is a classic text on semantic ﬁelds. More recen t papers addressing
thistopiccanbefoundinLehrerandKittay(1992).
Theuseofsemanticprimitivestodeﬁnewordmeaningdatesba cktoLeibniz;inlin-
guistics,thefocusoncomponentialanalysisinsemanticsw asduetoHjelmslev(1969).
See Nida (1975) for a comprehensive overview of work on compo nential analysis.
Wierzbicka (1996) has long been a major advocate of the use of primitives in linguis-
tic semantics; Wilks (1975a)has madesimilar argumentsfor the computationaluse of
primitivesin machinetranslationandnaturallanguageund erstanding. Anotherpromi-

nentefforthasbeenJackendoff’sConceptualSemanticswor k(1983,1990),whichhas
also beenappliedin machinetranslation(Dorr,1993,1992) .
Computational approaches to the interpretation of metapho r include convention-
based and reasoning-basedapproaches. Convention-baseda pproachesencodespeciﬁc
knowledgeaboutarelativelysmallcoresetofconventional metaphors. Theserepresen-
tationsarethenusedduringunderstandingtoreplaceoneme aningwithanappropriate
metaphorical one (Norvig, 1987; Martin, 1990; Hayes and Bay er, 1991; Veale and

Keane, 1992; Jones and McCoy, 1992). Reasoning-based appro aches eschew repre-
senting metaphoric conventions, instead modeling ﬁgurati ve language processing via
general reasoning ability, such as analogical reasoning, r ather than as a speciﬁcally
language-relatedphenomenon. (Russell, 1976; Carbonell, 1982; Gentner, 1983; Fass,
1988,1991,1997).
An inﬂuential collection of papers on metaphor can be found i n Ortony (1993).
LakoffandJohnson(1980)istheclassicworkonconceptualm etaphorandmetonymy.
Russell(1976)presentsoneoftheearliestcomputationala pproachestometaphor. Ad-

ditionalearlyworkcanbefoundinDeJongandWaltz(1983),W ilks(1978)andHobbs
(1979b). More recent computationaleffortsto analyze meta phorcan be foundin Fass
(1988, 1991, 1997), Martin (1990), Veale and Keane (1992), I verson and Helmreich
(1992),Chandler (1991),and Martin (2006). Martin (1996)p resentsa surveyof com-
putational approaches to metaphor and other types of ﬁgurat ive language. Gries and
Stefanowitsch (2006) is a recent collection of papers on cor pus-based approaches to
metaphor.
Exercises

metaphor.
Exercises
19.1Collect three deﬁnitionsof ordinarynon-technicalEnglis h words from a dictio-
nary of your choice that you feel are ﬂawed in some way. Explai n the nature of
theﬂawandhowitmightberemedied.
19.2Give a detailed account of similarities and differences amo ng the following set

DRAFTSection19.7. Summary 647
oflexemes: imitation,synthetic,artiﬁcial,fake,andsimulated .
19.3Examine the entries for these lexemes in WordNet (or some dic tionary of your
choice). Howwell doesit reﬂectyouranalysis?
19.4The WordNet entry for the noun batlists 6 distinct senses. Cluster these senses
using the deﬁnitionsof homonymyand polysemygiven in this c hapter. For any
sensesthat arepolysemous,givean argumentasto howthesen sesarerelated.
19.5Assign the various verb arguments in the following WSJ examp les to their ap-
propriatethematicrolesusingtheset ofrolesshownin Fig. 19.6.
a. Theintenseheatbuckledthehighwayaboutthreefeet.
b. He meltedherreservewith ahusky-voicedpaeantohereyes.
c. ButMingo,amajorUnionPaciﬁcshippingcenterinthe1890s ,hasmelted
awaytolittle morethanthegrainelevatornow.
19.6Using WordNet, describe appropriateselectional restrict ionson the verbs drink,
kiss, andwrite.
19.7Collect a small corpus of examples of the verbs drink,kiss, andwrite, and ana-

lyzehowwell yourselectionalrestrictionsworked.
19.8Considerthefollowingexamplesfrom(McCawley,1968):
Myneighborisa fatherofthree.
?Mybuxomneighborisa fatherofthree.
What does the ill-formedness of the second example imply abo ut how con-
stituentssatisfy,orviolate,selectionalrestrictions?
19.9Findsomearticlesaboutbusiness,sports,orpoliticsfrom yourdailynewspaper.
Identify as many uses of conventional metaphors as you can in these articles.
How many of the words used to express these metaphors have ent ries in either

WordNetoryourfavoritedictionarythatdirectlyreﬂect th emetaphor.
19.10Considerthefollowingexample:
The stock exchange wouldn’t talk publicly, but a spokesman s aid a news
conferenceisset fortodaytointroduceanewtechnologypro duct.
Assumingthat stockexchangesarenotthe kindsofthingstha t canliterallytalk,
givea sensibleaccountforthisphrasein termsofa metaphor ormetonymy.
19.11ChooseanEnglishverbthatoccursinbothFrameNetandPropB ank. Compare
andcontrasttheFrameNetandPropBankrepresentationsoft heargumentsofthe
verb.

Chapter20
Computational LexicalSemantics
Toget asingle right meaning is better thanaship-load of pea rls,
Toresolve asingle doubt is like the bottom fallingoffthe bu cket.
YuenMei (1785) (translationby Arthur Waley)
The asphalt that Los Angeles is famous for occurs mainly on it s freeways. But in the
middleofthecityisanotherpatchofasphalt,theLaBreatar pits,andthisasphaltpre-
serves millions of fossil bones from the last of the Ice Ages o f the Pleistocene Epoch.
One of these fossils is the Smilodon , or sabre-toothed tiger, instantly recognizable by

its long canines. Five million years ago or so, a completely d ifferentsabre-tooth tiger
calledThylacosmilus lived in Argentina and other parts of South America. Thylaco s-
milus was a marsupial where Smilodon was a placental mammal, but had the same
long upper caninesand, like Smilodon,had a protectivebone ﬂange on the lower jaw.
Thesimilarityofthesetwomammalsisoneofmanyexampleofp arallelorconvergent
evolution, in which particular contexts or environments le ad to the evolution of very
similarstructuresindifferentspecies(Gould,1980).

The role of context is also important in the similarity of a le ss biological kind of
organism: the word. Suppose we wanted to decide if two words h ave similar mean-
ings. Not surprisingly, words with similar meanings often o ccur in similar contexts,
whether in terms of corpora (having similar neighboringwor ds or syntactic structures
in sentences)orin termsofdictionariesandthesauruses(h avingsimilar deﬁnitions,or
being nearbyin the thesaurushierarchy). Thussimilarity o f context turnsout to be an

important way to detect semantic similarity. Semantic simi larity turns out to play an
importantrolesinadiversesetofapplicationsincludingi nformationretrieval,question
answering, summarizationand generation,text classiﬁcat ion, automatic essay grading
andthedetectionofplagiarism.
Inthischapterweintroduceaseriesoftopicsrelatedtocom putingwithwordmean-
ings, orcomputational lexical semantics . Roughly in parallel with the sequence of

topics in Ch. 19, we’ll introduce computationaltasks assoc iated with word senses, re-
lationsamongwords,andthethematicstructureofpredicat e-bearingwords. We’ll see
theroleofimportantroleofcontextandsimilarityofsense in eachofthese.
We beginwith word sense disambiguation ,the task of examiningwordtokensin
context and determiningwhich sense of each word is being use d. WSD is a task with
a long history in computational linguistics, and as we will s ee, is a non-trivial under-
taking given the somewhat elusive nature of many word senses . Nevertheless, there

arerobustalgorithmsthatcanachievehighlevelsofaccura cygivencertainreasonable
assumptions. Manyofthesealgorithmsrelyoncontextualsi milaritytohelpchoosethe
propersense.
This will lead us natural to a considerationof the computati onofword similarity
andotherrelationsbetweenwords,includingthe hypernym ,hyponym ,andmeronym
WordNet relations introduced in Ch. 19. We’ll introduce met hods based purely on
corpussimilarity,andothersbasedonstructuredresource ssuchasWordNet.

DRAFT650 Chapter 20. ComputationalLexicalSemantics
Finally,wedescribealgorithmsfor semanticrolelabeling ,alsoknownas caserole
orthematic role assignment . These algorithmsgenerallyuse featuresextracted from
syntacticparsestoassignsemanticrolessuchas AGENT,THEMEandINSTRUMENT to
thephrasesina sentencewithrespecttoparticularpredica tes.
20.1 WordSenseDisambiguation: Overview
Our discussion of compositional semantic analyzers in Ch. 1 8 pretty much ignored
the issue of lexical ambiguity. It should be clear by now that this is an unreasonable
approach. Without some means of selecting correct senses fo r the words in an input,
the enormousamountof homonymyand polysemyin the lexiconw ouldquicklyover-
whelmanyapproachin anavalancheofcompetinginterpretat ions.
Thetaskofselectingthecorrectsenseforawordiscalled wordsensedisambigua-
tion, orWSD. Disambiguatingwordsenses hasthe potentialto improvema nynaturalword sensedisambiguation
WSDlanguage processing tasks. As we’ll see in Ch. 25, machine translation is one area

where word sense ambiguities can cause severe problems; oth ers include question-
answering ,information retrieval , andtext classiﬁcation . The way that WSD is
exploited in these and other applications varies widely bas ed on the particular needs
of the application. The discussion presented here ignores t hese application-speciﬁc
differences and focuses on the implementation and evaluati on of WSD systems as a
stand-alonetask.
In their most basic form, WSD algorithms take as input a word i n context along

with a ﬁxed inventory of potential word senses, and return th e correct word sense for
that use. Both the nature of the input and the inventoryof sen ses dependson the task.
For machine translation from English to Spanish, the sense t ag inventory for an En-
glishwordmightbe theset ofdifferentSpanishtranslation s. If speechsynthesisis our
task, the inventory might be restricted to homographs with d iffering pronunciations
such asbassandbow. If our task is automatic indexing of medical articles, the s ense

taginventorymightbethe set ofMeSH (MedicalSubjectHeadi ngs)thesaurusentries.
When we are evaluatingWSD in isolation, we can use the set of s enses froma dictio-
nary/thesaurusresource like WordNet or LDOCE. Fig. 20.1 sh ows an example for the
wordbass,whichcanrefertoa musicalinstrumentorakindofﬁsh.1
WordNet Spanish Roget
Sense Translation Category Target Word inContext
bass4lubina FISH/INSECT ...ﬁshas Paciﬁcsalmon andstriped bassand...
bass4lubina FISH/INSECT ...produce ﬁletsof smoked bassor sturgeon...

bass7bajo MUSIC ...excitingjazz bassplayer since RayBrown...
bass7bajo MUSIC ...playbassbecause he doesn’t have tosolo...
Figure20.1 Possible deﬁnitions for the inventory of sense tags for bass.
1The WordNet database includes 8 senses; we have arbitrarily selected two for this example; we have
also arbitrarily selected one of the many possible Spanish n ames for ﬁshes which could be used to translate
Englishsea-bass.

DRAFTSection20.2. SupervisedWordSense Disambiguation 651
It is useful to distinguish two variants of the generic WSD ta sk. In the lexical
sampletask,asmallpre-selectedsetoftargetwordsischosen,alo ngwithaninventory lexicalsample
of senses for each word from some lexicon. Since the set of wor ds and the set of
senses is small, supervised machine learning approaches are often used to handle
lexical sampletasks. Foreach word,a numberof corpusinsta nces(contextsentences)
can be selected and hand-labeled with the correct sense of th e target word in each.
Classiﬁer systems can then be trained using these labeled ex amples. Unlabeled target
words in context can then be labeled using such a trained clas siﬁer. Early work in
wordsensedisambiguationfocusedsolelyonlexicalsample tasksofthissort,building
word-speciﬁcalgorithmsfordisambiguatingsinglewordsl ikeline,interest, orplant.
In contrast, in the all-words task systems are given entire texts and a lexicon with all-words

an inventory of senses for each entry, and are required to dis ambiguate every content
word in the text. The all-words task is very similar to part-o f-speech tagging, except
withamuchlargersetoftags,sinceeachlemmahasitsownset . Aconsequenceofthis
largerset of tags is a serious data sparsenessproblem; ther e is unlikelyto be adequate
trainingdataforeverywordinthetestset. Moreover,given thenumberofpolysemous
words in reasonably-sized lexicons, approaches based on tr aining one classiﬁer per

termareunlikelytobe practical.
In the following sections we explore the application of vari ous machine learning
paradigmstowordsensedisambiguation. Webeginwithsuper visedlearning,followed
by a section on how systems are standardly evaluated. We then turn to a variety of
methods for dealing with the lack of sufﬁcient day for fully- supervised training, in-
cludingdictionary-basedapproachesandbootstrappingte chniques.
Finally,afterwe haveintroducedthe necessarynotionsofd istributionalwordsim-

ilarityinSec.20.7,wereturninSec.20.10totheproblemof unsupervisedapproaches
tosense disambiguation.
20.2 SupervisedWordSenseDisambiguation
If we have data which has been hand-labeled with correct word senses, we can use
asupervised learning approach to the problem of sense disambiguation. extractin g
featuresfromthe text that are helpfulin predictingpartic ularsenses, andthen training
a classiﬁer to assign the correct sense given these features . The output of training is

thusaclassiﬁersystemcapableofassigningsenselabelsto unlabeledwordsincontext.
Forlexical sample tasks, there are various labeled corpora for individual wor ds,
consistingofcontextsentenceslabeledwiththecorrectse nseforthetargetword. These
include the line-hard-serve corpus containing 4,000 sense-tagged examples of lineas
a noun,hardas an adjective and serveas a verb (Leacocket al., 1993), and the inter-
estcorpus with 2,369 sense-tagged examples of interestas a noun (Bruce and Wiebe,

1994). The SENSEVAL projecthas also produceda numberof such sense-labeledlex i-
calsamplecorpora( SENSEVAL -1with34wordsfromthe HECTORlexiconandcorpus
(KilgarriffandRosenzweig,2000;Atkins, 1993), SENSEVAL -2 and -3 with 73and57
targetwords,respectively(Palmeret al.,2001b;Kilgarri ff,2001)).
For training all-word disambiguation tasks we use a semantic concordance , asemanticconcordance

DRAFT652 Chapter 20. ComputationalLexicalSemantics
corpus in which each open-class word in each sentence is labe led with its word sense
fromaspeciﬁcdictionaryorthesaurus. Onecommonlyusedco rpusisSemCor,asubset
of the Brown Corpus consisting of over 234,000 words which we re manually tagged
with WordNet senses (Miller et al., 1993; Landes et al., 1998 ). In addition, sense-
tagged corpora have been built for the SENSEVAL all-word tasks. The SENSEVAL -3
English all-words test data consisted of 2081 tagged conten t word tokens, from 5,000
totalrunningwordsofEnglishfromtheWSJandBrowncorpora (Palmeretal.,2001b).
20.2.1 Extracting Feature Vectors forSupervised Learning
Theﬁrst stepinsupervisedtrainingistoextractausefulse toffeaturesthatarepredic-
tive of word senses. As Ide and V´ eronis (1998b) point out, th e insight that underlies
all modern algorithms for word sense disambiguation was ﬁrs t articulated by Weaver
(1955)inthe contextofmachinetranslation:

If one examines the words in a book, one at a time as through an o paque mask
with a hole in it one word wide, then it is obviously impossibl e to determine, one
at a time, the meaning of the words. [...] But if one lengthens the slit in the
opaque mask, until one can see not only the central word in que stion but also say
N words on either side, then if N is large enough one can unambi guously decide
themeaningof thecentral word. [...] The practicalquestio nis: “Whatminimum

value of N will, at least in a tolerable fraction of cases, lea d to the correct choice
ofmeaning for the central word?”
To extract useful featuresfromsuch a window,a minimal amou ntof processingis
ﬁrst performed on the sentence containing the window. This p rocessing varies from
approach to approach but typically includes part-of-speec h tagging, lemmatization or
stemming, and in some cases syntactic parsing to reveal info rmation such as head
wordsand dependencyrelations. Context featuresrelevant to the targetword can then

beextractedfromthisenrichedinput. A featurevector consistingofnumericornom- feature vector
inal values is used to encode this linguistic information as an input to most machine
learningalgorithms.
Two classes of features are generally extracted from these n eighboring contexts:
collocationalfeaturesand bag-of-wordsfeatures. A collocation is a wordor phrase in collocation
a position-speciﬁc relationship to a target word (i.e., exa ctly one word to the right, or
exactly4wordstotheleft,andsoon). Thus collocationalfeatures encodeinformationcollocationalfeatures

aboutspeciﬁcpositions located to the left or right of the target word. Typ ical features
extracted for these context words include the word itself, t he root form of the word,
andtheword’spart-of-speech. Suchfeaturesareeffective atencodinglocallexicaland
grammaticalinformationthatcanoftenaccuratelyisolate agivensense.
As an example of this type of feature-encoding, consider the situation where we
needtodisambiguatethe word bassin thefollowingWSJsentence:
(20.1)Anelectricguitarand bassplayerstandoffto oneside,notreallypartofthe

scene,just asa sortofnodto gringoexpectationsperhaps.
A collocationalfeature-vector,extractedfrom a windowof two wordsto the right and
left of the targetword, made up of the wordsthemselvesand th eir respectiveparts-of-

DRAFTSection20.2. SupervisedWordSense Disambiguation 653
speech,i.e.,
[wi−2,POSi−2,wi−1,POSi−1,wi+1,POSi+1,wi+2,POSi+2] (20.2)
wouldyieldthefollowingvector:
[guitar, NN, and, CC, player, NN, stand, VB]
Thesecondtypeoffeatureconsistsof bag-of-words informationaboutneighboring
words. A bag-of-words meansanunorderedsetofwords,ignoringtheirexactpositi on. bag-of-words
Thesimplestbag-of-wordsapproachrepresentsthecontext ofatargetwordbyavector
offeatures,eachbinaryfeatureindicatingwhetheravocab ularyword wdoesordoesn’t
occurin the context. Thisvocabularyis typically preselec tedas some useful subset of
wordsin a trainingcorpus. In most WSD applications,the con textregionsurrounding
the target word is generallya small symmetricﬁxed size wind ow with the targetword
at the center. Bag-of-word features are effective at captur ing the general topic of the
discoursein whichthe targetwordhasoccurred. This, in tur n,tendsto identifysenses
of a word that are speciﬁc to certain domains. We generally do n’t use stop-words

as features, and may also limit the bag-of-words to only cons ider a small number of
frequentlyusedcontentwords.
For example a bag-of-words vector consisting of the 12 most f requent content
words from a collection of basssentences drawn from the WSJ corpus would have
thefollowingorderedwordfeatureset:
[ﬁshing, big, sound, player, ﬂy, rod, pound, double, runs, pl aying, guitar, band ]
Using these word features with a window size of 10, example (2 0.1) would be
representedbythefollowingbinaryvector:

[0,0,0,1,0,0,0,0,0,0,1,0]
We’llrevisitthebag-of-wordstechniqueinCh.23wherewe’ llseethatitformsthe
basisforthe vectorspacemodel ofsearchin modernsearchengines.
Mostapproachestosense disambiguationuse bothcollocati onalandbag-of-words
features,eitherbyjoiningthemintoonelongvector,orbyb uildingadistinctclassiﬁer
foreachfeaturetype,andcombiningthemin somemanner.
20.2.2 NaiveBayes and DecisionListClassiﬁers
Giventrainingdatatogetherwiththeextractedfeatures,a nysupervisedmachinelearn-

ing paradigm can be used to train a sense classiﬁer. We will re strict our discussion
heretothenaiveBayesanddecisionlist approaches,sincet heyhavebeenthe focusof
considerable work in word sense disambiguation and have not yet been introduced in
previouschapters.
ThenaiveBayesclassiﬁer approachtoWSDisbasedonthepremisethatchoosingnaiveBayes
classiﬁer
the best sense ˆ sout of the set of possible senses Sfor a feature vector ⃗famounts to
choosingthemostprobablesense giventhatvector. Inother words:
ˆs=argmax
s∈SP(s|⃗f) (20.3)

classiﬁer
ˆs=argmax
Asisalmostalwaysthecase,itwouldbedifﬁculttocollectr easonablestatisticsforthis
equationdirectly. Toseethis,considerthatasimplebinar ybagofwordsvectordeﬁned

DRAFT654 Chapter 20. ComputationalLexicalSemantics
over a vocabulary of 20 words would have 220possible feature vectors. It’s unlikely
that any corpus we have access to will provide coverage to ade quately train this kind
of feature vector. To get around this problem we ﬁrst reformu late our problem in the
usualBayesianmannerasfollows:
ˆs=argmax
s∈SP(⃗f|s)P(s)
P(⃗f)(20.4)
Eventhisequationisn’thelpfulenough,sincethedataavai lablethatassociatesspe-
ciﬁcvectors ⃗fwitheachsense sisalsotoosparse. However,whatisavailableingreater
abundance in a tagged training set is information about indi vidual feature-value pairs
inthecontextofspeciﬁcsenses. Therefore,wecanmakethei ndependenceassumption
that gives this methodits name, and that has served us well in part-of-speechtagging,
speech recognition, and probabilistic parsing — naivelyassume that the features are
independentofoneanother. Makingthisassumptionthatthe featuresare conditionally
independent giventhe wordsense yieldsthe followingapproximationfor P(⃗f|s):

P(⃗f|s)≈n
∏
j=1P(fj|s) (20.5)
Inotherwords,wecanestimatetheprobabilityofanentirev ectorgivenasensebythe
productoftheprobabilitiesofitsindividualfeaturesgiv enthatsense. Since P(⃗f)isthe
same for all possible senses, it does not effect the ﬁnal rank ing of senses, leaving us
withthe followingformulationofa naive Bayesclassiﬁer forWSD :
ˆs=argmax
s∈SP(s)n
∏
j=1P(fj|s) (20.6)
Given this equation, training a naive Bayes classiﬁer consists of estimating each

P(⃗f|s)≈n
∏
ˆs=argmax
s∈SP(s)n
∏
of these probabilities. (20.6)ﬁrst requiresan estimate fo rthe priorprobabilityof each
senseP(s). Wegetthemaximumlikelihoodestimateofthisprobability fromthesense-
taggedtrainingcorpusbycountingthenumberoftimesthese nsesioccursanddividing
by the total count of the target word wj(i.e. the sum of the instances of each sense of
theword). Thatis:
P(si) =count(si,wj)
count(wj)(20.7)
We also need to know each of the individual feature probabili tiesP(fj|s). The
maximumlikelihoodestimate forthese wouldbe:
P(fj|s) =count(fj,s)
count(s)(20.8)

Thus, if a collocational feature such as [ wi−2=guitar] occurred 3 times for sense
bass1,andsensebass1itselfoccurred60timesintraining,theMLEestimateis P(fj|s)=
0.05. Binarybag-of-wordfeaturesaretreatedinasimilarman ner;wesimplycountthe
number of times a given vocabulary item is present with each o f the possible senses
anddividebythecountforeachsense.

DRAFTSection20.2. SupervisedWordSense Disambiguation 655
Rule Sense
ﬁshwithinwindow ⇒ bass1
stripedbass ⇒ bass1
guitarwithinwindow ⇒ bass2
bassplayer ⇒ bass2
pianowithinwindow ⇒ bass2
tenorwithinwindow ⇒ bass2
seabass ⇒ bass1
play/Vbass ⇒ bass2
riverwithinwindow ⇒ bass1
violinwithinwindow ⇒ bass2
salmonwithinwindow ⇒ bass1
onbass ⇒ bass2
bassare ⇒ bass1
Figure20.2 An abbreviated decision list for disambiguating the ﬁsh sen se of bass from the
music sense (Yarowsky, 1997).
With the necessary estimates in place, we can assign senses t o words in context
by applying Eq. 20.6. More speciﬁcally, we take the target wo rd in context, extract
the speciﬁed features, compute P(s)∏n
j=1P(fj|s)for each sense, and return the sense
associated with the highest score. Note that in practice, th e probabilitiesproducedfor
even the highest scoring senses will be dangerously low due t o the various multipli-
cationsinvolved;mappingeverythingto log-spaceandinst eadperformingadditionsis
theusualsolution.

The use of a simple maximum likelihood estimator means that i n testing, when a
target word cooccurs with a word that it did not cooccur with i n training, all of its
senseswill receiveaprobabilityofzero. Smoothingis ther eforeessential to thewhole
enterprise. Naive Bayes approachesto sense disambiguatio ngenerally use the simple
Laplace(add-oneoradd-k)smoothingdiscussedinCh. 4.
One problem with naive Bayes and some other classiﬁers is tha t it’s hard for hu-
mans to examine their workings and understand their decisio ns. Decision lists and

decision trees are somewhat more transparent approaches th at lend themselves to in-
spection. Decision list classiﬁers are equivalent to simple case statements in mostDecisionlistclassiﬁers
programming languages. In a decision list classiﬁer, a sequ ence of tests is applied to
each target word feature vector. Each test is indicative of a particular sense. If a test
succeeds, then the sense associated with that test is return ed. If the test fails, then the
next test in the sequence is applied. This continues until th e end of the list, where a

defaulttest simplyreturnsthemajoritysense.
Fig. 20.2 shows a portion of a decision list for the task of dis criminating the ﬁsh
sense ofbassfrom the music sense. The ﬁrst test says that if the word ﬁshoccurs
anywhere within the input context then bass1is the correct answer. If it doesn’t then
each of the subsequent tests is consulted in turn until one re turns true; as with case
statementsa defaulttest that returnstrueisincludedatth e endofthelist.
Learning a decision list classiﬁer consists of generating a nd ordering individual

tests based on the characteristics of the training data. The re are a wide number of

DRAFT656 Chapter 20. ComputationalLexicalSemantics
methodsthatcanbeusedtocreatesuchlists. Intheapproach usedbyYarowsky(1994)
for binary homonym discrimination, each individual featur e-value pair constitutes a
test. Wecanmeasurehowmuchafeatureindicatesaparticula rsensebycomputingthe
log-likelihoodof the sense given the feature. The ratio bet ween the log-likelihoodsof
thetwo sensestellsushowdiscriminativea featureisbetwe ensenses:
⏐⏐⏐⏐Log(P(Sense1|fi)
P(Sense2|fi))⏐⏐⏐⏐(20.9)
The decision list is then created from these tests by simply o rdering the tests in the
list accordingto the log-likelihoodratio. Each test is che cked in order and returnsthe
appropriate sense. This training method differs quite a bit from standard decision list
learningalgorithms. Forthedetailsandtheoreticalmotiv ationfortheseapproachessee
Rivest (1987)orRussell andNorvig(1995).
20.3 WSDEvaluation,Baselines,andCeilings
Evaluating component technologies like WSD is always a comp licated affair. In the

longterm,we’re primarilyinterestedin the extentto which theyimproveperformance
in some end-to-end application such as information retriev al, question answering or
machinetranslation. EvaluatingcomponentNLP tasks embed dedin end-to-endappli-
cations is called extrinsic evaluation ,task-based evaluation, end-to-end evaluation,extrinsicevaluation
orinvivoevaluation. Itisonlywithextrinsicevaluationthatwecan tellifatechnology invivo
such as WSD is workingin the sense ofactually improvingperf ormanceon some real

task.
Extrinsic evaluations are much more difﬁcult and time-cons uming to implement,
however, since they require integration into complete work ing systems. Furthermore,
an extrinsic evaluation may only tell us something about WSD in the context of the
application,andmaynotgeneralizetootherapplications.
Forthesereasons,WSDsystemsaretypicallydevelopedande valuatedintrinsically.
Inintrinsic orin vitrowe treat a WSD component as if it were a stand-alone system intrinsic
invitro operatingindependentlyof any given application. In this s tyle of evaluation, systems

areevaluatedeitherusingexactmatch senseaccuracy : thepercentageofwordsthatare sense accuracy
tagged identically with the hand-labeled sense tags in a tes t set; or with standard pre-
cisionandrecallmeasuresifsystemsarepermittedtopasso nlabelingsomeinstances.
Ingeneral,weevaluateusingheldoutdatafromthesamesens e-taggedcorporathatwe
used for training, such as the SemCor corpus discussed above , or the various corpora
producedbythe SENSEVAL effort.
Manyaspectsofsenseevaluationhavebeenstandardizedbyt heSENSEVAL /SEMEVAL

efforts (Palmer et al., 2006; Kilgarriff and Palmer, 2000). This framework providesa
shared task with training and testing materials along with s ense inventories for all-
wordsandlexicalsampletasksina varietyoflanguages.
WhicheverWSD taskweareperforming,weideallyneedtwoadd itionalmeasures
toassesshowwellwe’redoing: abaselinemeasuretotelluse howwellwe’redoingas

DRAFTSection20.3. WSD Evaluation,Baselines, andCeilings 657
comparedto relatively simple approaches, and a ceiling to t ell us how close we are to
optimalperformance.
The simplest baseline is to choose the most frequent sense for each word (Galemost frequentsense
et al., 1992b) from the senses in a labeled corpus. For WordNe t, this corresponds to
thetake the ﬁrst sense heuristic, since senses in WordNet are generallyorderedfr om take the ﬁrst sense
most-frequent to least-frequent. WordNet sense frequenci es come from the SemCor
sense-taggedcorpusdescribedabove.
Unfortunately,manyWordNetsensesdonotoccurinSemCor;t heseunseensenses
are thus ordered arbitrarily after those that do. The four Wo rdNet senses of the noun
plant,forexample,areasfollows:
FreqSynset Gloss
338plant1,works, industrial plant buildings for carrying on industrial labor
207plant2,ﬂora,plant life aliving organism lacking the power of locomotion
2plant3something planted secretly for discovery by another

0plant4an actor situated in theaudience whoseacting is rehearsed
butseems spontaneous to the audience
Themostfrequentsensebaselinecanbequiteaccurate,andi sthereforeoftenused
asadefault,tosupplyawordsensewhenasupervisedalgorit hmhasinsufﬁcienttrain-
ing data. A second commonly used baseline is the Lesk algorithm , discussed in the
Human inter-annotator agreement is generally considered a s a ceiling, or upper
bound,for sense disambiguationevaluations. Human agreem entis measured by com-

paring the annotations of two human annotatorson the same da ta given the same tag-
ging guidelines. The ceiling (inter-annotator agreement) for many all-words corpora
usingWordNet-stylesenseinventoriesseemstorangefroma bout75%to80%(Palmer
et al., 2006). Agreement on more coarse grained, often binar y, sense inventories is
closerto90%(Galeet al.,1992b).
While usinghand-labeledtest sets is the best currentmetho dfor evaluation,label-

ing large amountsof data is still quite expensive. For super visedapproaches,we need
thisdataanyhowfortrainingsotheefforttolabellargeamo untsofdataseemsjustiﬁed.
But for unsupervised algorithms like those we will discuss i n Sec. 20.10, it would be
nicetohaveanevaluationmethodthatavoidedhandlabeling . Theuseof pseudowords pseudowords
isonesuchsimpliﬁedevaluationmethod(Galeetal.,1992a; Sch¨ utze,1992a). Apseu-
doword is an artiﬁcial word created by concatenatingtwo ran domly-chosenwords to-
gether (e.g., bananaanddoorto create banana-door .) Each occurrence of the two

wordsinthetestsetisreplacedbythenewconcatenation,cr eatinganew‘word’which
isnowambiguousbetweenthesenses bananaanddoor. The‘correctsense’isdeﬁned
by the originalword, and so we can apply our disambiguationa lgorithmand compute
accuracyasusual. Ingeneral,pseudowordsgiveanoverlyop timisticmeasureofperfor-
mance,sincetheyareabiteasiertodisambiguatethanavera geambiguouswords. This
is becausethedifferentsenses ofreal wordstend to besimil ar, whilepseudowordsare

generallynotsemanticallysimilar,actinglikehomonymou sbutnotpolysemouswords
(Gaustad, 2001). Nakov and Hearst (2003) shows that it is pos sible to improve the
accuracyofpseudowordevaluationbymorecarefullychoosi ngthepseudowords.

DRAFT658 Chapter 20. ComputationalLexicalSemantics
20.4 WSD:Dictionaryand ThesaurusMethods
Supervised algorithms based on sense-labeled corpora are t he best performing algo-
rithmsforsense disambiguation. However,suchlabeledtra iningdataisexpensiveand
limitedandsupervisedapproachesfailonwordsnotinthetr ainingdata. Thusthissec-
tionandthenextdescribedifferentwaystogetindirectsup ervisionfromothersources.
In this section, we describe methods for using a dictionary o r thesaurus as an indirect
kindofsupervision;thenextsectiondescribesbootstrapp ingapproaches.
20.4.1 The LeskAlgorithm
By far the most well-studied dictionary-based algorithm fo r sense disambiguation is
theLesk algorithm ,really a family of algorithmsthat choosethe sense whose di ctio- Lesk algorithm
nary gloss or deﬁnition shares the most words with the target word’s neighborhood.
Fig.20.3showsthesimplestversionofthealgorithm,often calledthe SimpliﬁedLesk SimpliﬁedLesk
algorithm(KilgarriffandRosenzweig,2000).

function SIMPLIFIED LESK(word,sentence )returnsbest sense of word
best-sense←most frequent sense for word
max-overlap←0
context←set of words in sentence
for eachsenseinsenses of worddo
signature←setof words inthegloss andexamples of sense
overlap←COMPUTEOVERLAP(signature,context)
ifoverlap >max-overlap then
max-overlap←overlap
best-sense←sense
end
return(best-sense )
Figure20.3 The Simpliﬁed Lesk Algorithm. The C OMPUTEOVERLAPfunction returns the

end
number of words in common between two sets, ignoring functio n words or other words on a
stop list. The original Lesk algorithm deﬁnes the contextin a more complex way. The Corpus
Leskalgorithmweightseachoverlappingword wbyits−logP(w),andincludeslabeledtraining
corpus data inthe signature.
As an example of the Lesk algorithm at work, consider disambi guating the word
bankinthe followingcontext:
(20.10)Thebankcanguaranteedepositswilleventuallycoverfuturetuitio ncosts
becauseit investsinadjustable-ratemortgagesecurities .

giventhefollowingtwoWordNetsenses:

DRAFTSection20.4. WSD: DictionaryandThesaurusMethods 659
bank1Gloss: a ﬁnancial institution that accepts deposits and cha nnels the
money intolending activities
Examples: “he cashed a check at the bank”, “that bank holds th e mortgage
on myhome”
bank2Gloss: sloping land(especially the slope beside a body of wa ter)
Examples: “they pulled the canoe up on the bank”, “he sat on th e bank of
the river and watched the currents”
Sensebank1has two (non-stop) words overlapping with the context in (20 .10):
depositsandmortgage ,whilesensebank2haszero,so sense bank1ischosen.
TherearemanyobviousextensionstoSimpliﬁedLesk. Theori ginalLeskalgorithm
(Lesk, 1986) is slightly more indirect. Instead of comparin g a target word’s signature
with thecontextwords,the targetsignatureis comparedwit h the signaturesof eachof
the context words. For example, consider Lesk’s example of s electing the appropriate
senseofconeinthephrase pinecone giventhefollowingdeﬁnitionsfor pineandcone.

pine 1 kindsofevergreentreewith needle-shapedleaves
2 waste awaythroughsorroworillness
cone 1 solid bodywhichnarrowstoa point
2 somethingofthisshapewhethersolidorhollow
3 fruitofcertainevergreentrees
Inthisexample,Lesk’smethodwouldselect cone3asthecorrectsensesincetwoofthe
wordsinitsentry, evergreen andtree,overlapwithwordsintheentryfor pine,whereas
neither of the other entries have any overlap with words in th e deﬁnition of pine. In
generalSimpliﬁedLeskseemstoworkbetterthanoriginalLe sk.

Theprimaryproblemwitheithertheoriginalorsimpliﬁedap proaches,however,is
that the dictionary entries for the target words are short, a nd may not provide enough
chanceofoverlapwiththecontext.2Oneremedyisto expandthelist ofwordsusedin
theclassiﬁertoincludewordsrelatedto,butnotcontained intheirindividualsensedef-
initions. Butthebestsolution,ifanysense-taggedcorpus datalikeSemCorisavailable,
is to add all the wordsin the labeled corpussentencesfor a wo rd sense into the signa-

ture for that sense. This version of the algorithm, the Corpus Lesk algorithm is the Corpus Lesk
best-performing of all the Lesk variants (Kilgarriff and Ro senzweig, 2000; Vasilescu
et al., 2004) and is used as a baseline in the SENSEVAL competitions. Instead of just
counting up the overlappingwords, the Corpus Lesk algorithm also applies a weight
to each overlapping word. The weight is the inverse document frequency orIDF,inverse documentfrequency
IDFa standard information-retrieval measure to be introduced in Ch. 23. IDF measures

how many different ’documents’(in this case glosses and exa mples) a word occurs in
(Ch. 23) and is thus a way of discounting function words. Sinc e function words like
the,of, etc, occur in many documents, their IDF is very low, while th e IDF of content
wordsishigh. CorpusLeskthususesIDFinsteadofa stoplist .
Formallythe IDFforaword icanbedeﬁnedas
idfi=log(Ndoc
ndi)
(20.11)
2Indeed, Lesk (1986) notes that the performance of his system seems to roughly correlate with the length
of the dictionary entries.

DRAFT660 Chapter 20. ComputationalLexicalSemantics
whereNdocis the total number of ‘documents’(glosses and examples) an dndiis the
numberofthese documentscontainingword i.
Finally, it is possible to combine the Lesk and supervised ap proaches, by adding
newLesk-likebag-of-wordsfeatures. Forexample,the glos sesandexamplesentences
forthetargetsenseinWordNetcouldbeusedtocomputethesu pervisedbag-of-words
featuresinsteadof(orinadditionto)thewordsintheSemCo rcontextsentenceforthe
sense (Yuret,2004).
20.4.2 Selectional Restrictions andSelectional Preferen ces
One ofthe earliest knowledge-sourcesfor sense disambigua tionis the notionof selec-
tionalrestrictions deﬁnedinCh.19. Forexampletheverb eatmighthavearestriction
that its THEMEargumentbe [+FOOD] . In early systems, selectional restrictionswere
used to rule out senses that violate the selectional restric tions of neighboring words
(Katz andFodor,1963; Hirst, 1987). Consider the following pair of WSJ examplesof
theword dish:

(20.12)“Inourhouse,everybodyhasa careerandnoneoftheminclude swashing
dishes,” hesays.
(20.13)Inhertinykitchenat home,Ms. Chenworksefﬁciently,stir- fryingseveral
simpledishes, includingbraisedpig’searsandchickenliverswith green
peppers.
These correspondto WordNet dish1(a piece of dishware normally used as a con-
tainerforholdingorservingfood),withhypernymslike artifact,anddish2(aparticular
itemofpreparedfood)withhypernymslike food.
The fact that we perceive no ambiguity in these examples can b e attributed to the

peppers.
selectional restrictions imposed by washandstir-fryon their THEMEsemantic roles.
The restrictions imposed by wash(perhaps [+ WASHABLE ]) conﬂict with dish2. The
restrictionson stir-fry([+EDIBLE])conﬂict with dish1. In earlysystems, the predicate
strictly selected the correct sense of an ambiguous argumen t by eliminating the sense
that fails to match one of its selectional restrictions. But such hard constraints have
a number of problems. The main problem is that selectional re striction violations of-

ten occur in well-formed sentences, either because they are negated as in (20.14), or
becauseselectionalrestrictionsareoverstatedasin(20. 15):
(20.14)Butit fell apartin 1931,perhapsbecausepeoplerealizedyo ucan’teatgold
forlunchif you’rehungry.
(20.15)Inhistwo championshiptrials, Mr. Kulkarni ateglassonanemptystomach,
accompaniedonlybywater andtea.
As Hirst (1987)observes, exampleslike these often result i n the elimination of all
senses,bringingsemanticanalysistoahalt. Modernmodels thusadopttheviewofse-

lectional restrictions as preferences, rather than rigid r equirements. Although there
have been many instantiations of this approach over the year s (e.g., Wilks, 1975c,
1975b, 1978), we’ll discuss a member of the popular probabil istic or information-
theoreticfamilyofapproaches: Resnik’s(1997)modelof selectionalassociation .
Resnik ﬁrst deﬁnes the selectional preference strength as the general amount ofselectionalpreference
strength

DRAFTSection20.4. WSD: DictionaryandThesaurusMethods 661
information that a predicate tells us about the semantic cla ss of its arguments. For
example, the verb eattells us a lot about the semantic class of its direct object, s ince
they tend to be edible. The verb be, by contrast, tells us less about its direct objects.
The selectional preference strength can be deﬁned by the dif ference in information
between two distributions: the distribution of expected se mantic classes P(c)(how
likely is it that a direct object will fall into class c) and the distribution of expected
semanticclasses forthe particularverb P(c|v)(howlikelyis it that the directobjectof
speciﬁcverb vwillfallintosemanticclass c). Thegreaterthedifferencebetweenthese
distributions, the more information the verb is giving us ab out possible objects. This
differencecanbequantiﬁedbythe relativeentropy betweenthesetwodistributions,or relativeentropy

Kullback-Leibler divergence (Kullbackand Leibler,1951). The Kullback-LeiblerorKullback-Leiblerdivergence
KL divergence D(P||Q)canbe usedto expressthe differencebetweentwo probabilit y
distributions PandQ, and will be discussed further when we discuss word similari ty
inEq.20.50.
D(P||Q) =∑
xP(x)logP(x)
Q(x)(20.16)
The selectional preference SR(v)uses the KL divergenceto express how much in-
formation, in bits, the verb vexpresses about the possible semantic class of its argu-
ment.
SR(v) =D(P(c|v)||P(c))
=∑
cP(c|v)logP(c|v)
P(c)(20.17)

D(P||Q) =∑
ment.
=∑
Resnikthendeﬁnesthe selectionalassociation ofaparticularclassandverbastheselectionalassociation
relativecontributionofthat classtothegeneralselectio nalpreferenceoftheverb:
AR(v,c) =1
SR(p)P(c|v)logP(c|v)
P(c)(20.18)
Theselectionalassociationisthusaprobabilisticmeasur eofthestrengthofassoci-
ationbetweenapredicateandaclassdominatingtheargumen ttothepredicate. Resnik
estimates the probabilities for these associations by pars ing a corpus, counting all the
timeseachpredicateoccurswith eachargumentword,andass umingthat eachwordis

AR(v,c) =1
a partial observation of all the WordNet concepts containin g the word. The following
tablefromResnik(1996)showssomesamplehighandlowselec tionalassociationsfor
verbsandsomeWordNetsemanticclassesoftheirdirectobje cts.
DirectObject DirectObject
Verb SemanticClass Assoc SemanticClass Assoc
read WRITING 6.80 ACTIVITY -.20
write WRITING 7.26 COMMERCE 0
see ENTITY 5.79 METHOD -0.01
Resnik (1998) shows that these selectional associations ca n be used to perform a
limitedformofwordsensedisambiguation. Roughlyspeakin gthealgorithmselectsas

DRAFT662 Chapter 20. ComputationalLexicalSemantics
the correct sense for an argument the one that has the highest selectional association
betweenoneofitsancestorhypernymsandthepredicate.
While we have presented only the Resnik model of selectional preferences, there
are other more recent models, using probabilistic methods a nd using other relations
than just direct object; see the end of the chapter for a brief summary. In general,
selectionalrestrictionapproachesperformaswell asothe runsupervisedapproachesat
sense disambiguation,butnotaswell asLeskorassupervise dapproaches.
20.5 MinimallySupervisedWSD:Bootstrapping
Boththesupervisedapproachandthedictionary-basedappr oachtoWSDrequirelarge
hand-builtresources;supervisedtrainingsetsinonecase ,largedictionariesintheother.
We caninstead use bootstrapping algorithms,oftencalled semi-supervised learning bootstrapping
orminimallysupervisedlearning ,whichneedonlyaverysmallhand-labeledtraining

set. The most widely emulated bootstrapping algorithm for W SD is the Yarowsky
algorithm (Yarowsky,1995).Yarowsky
algorithm
The goal of the Yarowsky algorithm is to learn a classiﬁer for a target word (in a
lexical-sample task). The algorithm is given a small seed-s etΛ0of labeled instances
of each sense, and a much larger unlabeled corpus V0. The algorithm ﬁrst trains an
initial decision-list classiﬁer on the seed-set Λ0. It then uses this classiﬁer to label
the unlabeled corpus V0. The algorithm then selects the examples in V0that it is most

algorithm
conﬁdentabout,removesthem,andaddsthemto the trainings et (call it now Λ1). The
algorithmthentrainsanewdecisionlistclassiﬁer(anewse tofrules)on Λ1,anditerates
byapplyingtheclassiﬁertothenow-smallerunlabeledset V1,extractinganewtraining
setΛ2andsoon. Witheachiterationofthisprocess,thetrainingc orpusgrowsandthe
untaggedcorpusshrinks. Theprocessisrepeateduntilsome sufﬁcientlylowerror-rate
on the training set is reached, or until no further examples f rom the untagged corpus
areabovethreshold.

The key to any bootstrappingapproach lies in its ability to c reate a larger training
set from a small set of seeds. This requires an accurate initi al set of seeds and a good
conﬁdencemetricforpickinggoodnewexamplestoaddtothet rainingset. Theconﬁ-
dencemetricusedby Yarowsky(1995)isthe measuredescribe dearlierin Sec. 20.2.2,
thelog-likelihoodratioofthe decision-listrulethatcla ssiﬁed theexample.
One way to generate the initial seeds is to hand-label a small set of examples

(Hearst, 1991). Instead of hand-labeling, it is also possib le to use a heuristic to auto-
maticallyselectaccurateseeds. Yarowsky(1995)usedthe OneSenseperCollocationOne Sense per
Collocation
heuristic,whichreliesontheintuitionthatcertainwords orphrasesstronglyassociated
withthetargetsensestendnottooccurwiththeothersense. Yarowskydeﬁneshisseed
setbychoosingasinglecollocationforeachsense. Asanill ustrationofthistechnique,
considergeneratingseedsentencesfortheﬁshandmusicals ensesofbass. Withouttoo

muchthought,wemightcomeupwith ﬁshasareasonableindicatorof bass1,andplay
as a reasonableindicatorof bass2. Fig. 20.5showsa partial result ofsuch a searchfor
thestrings“ﬁsh” and“play”ina corpusof bassexamplesdrawnfromthe WSJ.
We can also suggest collocates automatically, for example e xtracting words from

?
???
?
?
??
??
?
?
???
??
?
??
???
??
??
???
?
??
A?A??
??
??
??
????
???
A
BAA
?
?
???
??
?
?B
???
B
BB
??
B
?BB?? ??
????
??
??
???
??AA
?
????
?????
??
???A
BB
??B
??????
????B
B??
A? AA
?
A
???A
AAA
A
AALIFE
B
B
MANUFACTURING EQUIPMENTEMPLOYEE
?
??B
B?
???
???ANIMALMICROSCOPICV0 V1
Λ0 Λ1
(a) (b)
Figure20.4 TheYarowskyalgorithmdisambiguating’plant’attwostage s;’?’ indicatesanunlabeledobservation,
A and B are observations labeled as SENSE-A or SENSE-B. ‘LIFE ’ indicates observations occur with collocate
“life”. The initial stage (a) shows only seed sentences Λ0labeled by collocates (‘life’ and ’manufacturing’). An
intermediatestageisshownin(b)wheremorecollocateshav ebeendiscovered(‘equipment’,‘microscopic’,etc)and
more instances in V0have been moved into Λ1, leaving a smaller unlabeled set V1. Figure adapted from Yarowsky
(1995).
We need more good teachers – right now, there are only a half a d ozen who can playthe
freebasswithease.

(1995).
Anelectricguitarand bassplay erstandofftoone side,notreallypartofthescene, justas
a sort of nod togringo expectations perhaps.
When the New Jersey Jazz Society, in a fund-raiser for the Ame rican Jazz Hall of Fame,
honors this historic night next Saturday, Harry Goodman, Mr . Goodman’s brother and
bass play er atthe original concert, willbe inthe audience withother familymembers.
Theresearchers saidtheworms spendpartoftheirlifecycle insuchﬁshasPaciﬁcsalmon
and striped bassandPaciﬁcrockﬁsh orsnapper.
And itallstartedwhen ﬁshermendecided the striped bassinLake Mead were tooskinny.

Though still a far cry from the lake’s record 52-pound bassof a decade ago, “you could
ﬁlletthese ﬁshagain, andthat made people very, veryhappy,” Mr. Paulson sa ys.
Figure20.5 Samples of basssentences extracted from the WSJ using the simple correlate s
playandﬁsh.
machine readable dictionary entries, and selecting seeds u sing collocational statistics
suchasthosedescribedinSec.20.7(Yarowsky,1995).
TheoriginalYarowskyalgorithmalso makesuse ofa secondhe uristic,called One

Sense Per Discourse , based on the work of Gale et al. (1992c), who noticed that aOne Sense PerDiscourse
particularwordappearingmultipletimesinatextordiscou rseoftenappearedwiththe
same sense. Yarowsky (1995), for example, showed in a corpus of 37,232 examples
that everytime the word bassoccurredmore than once in a discourse, that it occurred
in onlythe ﬁshoronlythe musiccoarse-grainsensethroughoutthediscourse. Theva-
lidityofthisheuristicdependsonthegranularityofthese nseinventoryandisnotvalid

DRAFT664 Chapter 20. ComputationalLexicalSemantics
in every discourse situation; it seems to be true mostly for c oarse-grain senses, and
particularly for cases of homonymy rather than polysemy (Kr ovetz, 1998). Nonethe-
less, it has still been useful in a number of unsupervised and semi-supervised sense
disambiguationsituations.
20.6 WordSimilarity: ThesaurusMethods
Weturnnowtothecomputationofvarioussemanticrelations thatholdbetweenwords.
We saw in Ch. 19 that such relationsincludesynonymy,antony my,hyponymy,hyper-
nymy,andmeronymy. Ofthese,theonethathasbeenmostcompu tationallydeveloped
andhasthegreatestnumberofapplicationsistheideaofwor dsynonymy andsimilar-
ity.
Synonymy is a binary relation between words; two words are ei ther synonyms or
not. Formostcomputationalpurposesweuseinsteadalooser metricof wordsimilar-
ityorsemantic distance . Two words are more similar if they share more features of wordsimilarity

ity.
semantic distance meaning, or are near-synonyms. Two words are less similar, o r have greater semantic
distance, if they have fewer commonmeaning elements. Altho ughwe have described
them as relationsbetweenwords, synonymy,similarity,and distance are actually rela-
tions between word senses. For exampleof the two senses of bank, we might say that
the ﬁnancial sense is similar to one of the senses of fundwhile the riparian sense is
more similar to one of the senses of slope. In the next few sectionsof thischapter,we

will needto computethese relationsoverbothwordsandsens es.
The ability to compute word similarity is a useful part of man y language under-
standingapplications. In informationretrieval orquestionanswering wemightwant
toretrievedocumentswhosewordshavesimilarmeaningstot hequerywords. In sum-
marization ,generation , andmachine translation , we need to know whether two
wordsare similar to know if we can substitute one for the othe r in particularcontexts.
Inlanguagemodeling ,wecanusesemanticsimilaritytoclusterwordsforclass-b ased

models. One interesting class of applicationsfor word simi larity is automatic grading
of student responses. For example algorithmsfor automatic essay grading use word
similarity to determine if an essay is similar in meaning to a correct answer. We can
also use word-similarity as part of an algorithm to takean exam, such as a multiple-
choicevocabularytest. Automaticallytakingexamsisusef ulintestdesignsinorderto
see howeasy orharda particularmultiple-choicequestiono rexamis.

There are two classes of algorithms for measuring word simil arity. This section
focuses on thesaurus-based algorithms, in which we measure the distance between
twosensesinanon-linethesauruslikeWordNetorMeSH.Then extsectionfocuseson
distributional algorithms,inwhichwe estimate wordsimilaritybyﬁndingw ordsthat
havesimilardistributionsin acorpus.
The thesaurus-based algorithms use the structure of the the saurus to deﬁne word
similarity. In principle we could measure similarity using any information available

in a thesaurus (meronymy, glosses, etc). In practice, howev er, thesaurus-based word
similarityalgorithmsgenerallyuseonlythehypernym/hyp onym(is-aorsubsumption)
hierarchy. In WordNet, verbs and nouns are in separate hyper nym hierarchies, so a

DRAFTSection20.6. Word Similarity: ThesaurusMethods 665
thesaurus-basedalgorithmforWordNetcanthusonlycomput enoun-nounsimilarity,or
verb-verbsimilarity; we can’t compare nounsto verbs, or do anythingwith adjectives
orotherpartsofspeech.
Resnik (1995)andBudanitskyandHirst (2001)drawthe impor tantdistinctionbe-
tweenword similarity andwordrelatedness . Two wordsaresimilar if theyarenear- word relatedness
synonyms,orroughlysubstitutableincontext. Wordrelate dnesscharacterizesa larger
set of potential relationships between words; antonyms, fo r example, have high relat-
edness,butlowsimilarity. Thewords carandgasolineareveryrelated,butnotsimilar,
whilecarandbicyclearesimilar. Wordsimilarityisthusasubcaseofwordrelate dness.
Ingeneral,theﬁvealgorithmswe describein thissectiondo notattemptto distinguish
between similarity and semantic relatedness; for convenie nce we will call them simi-
laritymeasures, although some would be more appropriatelydescri bed as relatedness
measures;wereturnto thisquestioninSec.20.8.

Figure20.6 A fragment of the WordNet hypernym hierarchy, showing path l engths from
nickeltocoin(1),dime(2),money(5), andRichter scale (7).
The oldest and simplest thesaurus-basedalgorithmsare bas ed on the intuition that
theshorterthe pathbetweentwowordsorsensesinthegraphdeﬁnedbythethesaur us
hierarchy,themoresimilartheyare. Thusaword/senseisve rysimilartoitsparentsor
itssiblings, andlesssimilar towordsthat arefar awayinth e network. Thisnotioncan
be operationalizedby measuring the number of edges between the two concept nodes

inthethesaurusgraph. Fig.20.6showsanintuition;thecon ceptdimeismostsimilarto
nickelandcoin,lesssimilarto money,andevenlesssimilarto Richterscale . Formally,
we specifypathlengthasfollows:
pathlen (c1,c2) =thenumberofedgesintheshortestpathinthethesaurus
graphbetweenthesense nodes c1andc2
Path-basedsimilarity can bedeﬁnedjust as the pathlength, oftenwith a logtransform
(LeacockandChodorow,1998),resultinginthefollowingco mmondeﬁnitionof path-
lengthbased similarity :path-lengthbased
similarity
simpath(c1,c2) =−logpathlen (c1,c2) (20.19)

Formostapplications,wedon’thavesense-taggeddata,and thusweneedouralgo-
rithm to give us the similarity between words rather than bet ween senses or concepts.

DRAFT666 Chapter 20. ComputationalLexicalSemantics
For any of the thesaurus-based algorithms, following Resni k (1995), we can approxi-
mate the correct similarity (which would require sense disa mbiguation) by just using
the pair of senses for the two words that results in maximum se nse similarity. Thus
basedonsensesimilaritywe candeﬁne word similarity asfollows: wordsimilarity
wordsim (w1,w2) =max
c1∈senses (w1)
c2∈senses (w2)sim(c1,c2) (20.20)
Thebasicpath-lengthalgorithmmakestheimplicitassumpt ionthateachlinkinthe
networkrepresentsa uniformdistance. In practice,this as sumptionis not appropriate.
Somelinks(forexamplethosethatareverydeepintheWordNe thierarchy)oftenseem
to represent an intuitively narrow distance, while other li nks (e.g., higher up in the
WordNet hierarchy)representan intuitivelywiderdistanc e. Forexample,in Fig. 20.6,
the distance from nickeltomoney(5) seems intuitivelymuchshorter than the distance
fromnickelto an abstract word standard; the link between medium of exchange and

standard seemswiderthanthatbetween,say, coinandcoinage.
Itispossibletoreﬁnepath-basedalgorithmswithnormaliz ationsbasedondepthin
the hierarchy(Wu and Palmer, 1994), but in general we’d like an approachwhich lets
usrepresentthedistanceassociatedwitheachedgeindepen dently.
A secondclass of thesaurus-basedsimilarity algorithmsat temptsto offerjust such
aﬁne-grainedmetric. These informationcontentwordsimilarity algorithmsstillrelyinformationcontent
onthe structureof thethesaurus,but also addprobabilisti cinformationderivedfroma

corpus.
Using similar notions to those we introduced earlier to deﬁn e soft selectional re-
strictions, let’s ﬁrst deﬁne P(c), following Resnik (1995), as the probability that a
randomly selected word in a corpus is an instance of concept c(i.e., a separate ran-
dom variable, ranging over words, associated with each conc ept). This implies that
P(root) =1, since any word is subsumed by the root concept. Intuitivel y, the lower
a concept in the hierarchy, the lower its probability. We tra in these probabilities by

counting in a corpus; each word in the corpus counts as an occu rrence of each con-
cept that containsit. For example,in Fig. 20.6above,an occ urrenceof the word dime
would count toward the frequency of coin,currency,standard, etc. More formally,
Resnikcomputes P(c)asfollows:
P(c) =∑w∈words (c)count(w)
N(20.21)
where words (c)is the set of words subsumed by concept c, andNis the total number
ofwordsinthecorpusthat arealsopresentinthethesaurus.

N(20.21)
Fig. 20.7, from Lin (1998b), shows a fragment of the WordNet c oncept hierarchy
augmentedwiththe probabilities P(c).
We nowneedtwo additionaldeﬁnitions. First, followingbas ic informationtheory,
we deﬁnetheinformationcontent(IC)ofa concept cas:
IC(c) =−logP(c) (20.22)
Second,wedeﬁnethe lowest commonsubsumer orLCSoftwo concepts:lowest commonsubsumer
LCS

DRAFTSection20.6. Word Similarity: ThesaurusMethods 667
entity0.395
inanimate-object 0.167
natural-object 0.0163
geological-formation 0.00176
0.000113 natural-elevation
0.0000189 hillshore0.0000836
coast0.0000216
Figure20.7 A fragment of the WordNet hierarchy, showing the probabilit yP(c)attached to
each content, adapted from a ﬁgure from Lin(1998b)
LCS(c1,c2) =the lowest common subsumer, i.e., the lowest node in the
hierarchythat subsumes(isa hypernymof)both c1andc2
Therearenowanumberofwaystousetheinformationcontento fanodeinaword
similarity metric. The simplest way was ﬁrst proposed by Res nik (1995). We think
of the similarity between two wordsas relatedto their commo ninformation;the more
twowordshaveincommon,themoresimilartheyare. Resnikpr oposestoestimatethe
common amount of information by the information content of the lowest common
subsumer ofthetwonodes . Moreformally,the Resnik similarity measureis: Resniksimilarity
simresnik(c1,c2) =−logP(LCS(c1,c2)) (20.23)

Lin (1998b) extended the Resnik intuition by pointing out th at a similarity metric
between objects A and B needs to do more than measure the amoun t of information
in common between A and B. For example, he pointed out that in a ddition, the more
differences betweenA andB, theless similartheyare. Insummary:
•commonality : themoreinformationAandBhaveincommon,themoresimilar
theyare.
•difference : the more differences between the information in A and B, the less
similartheyare

theyare.
Lin measuresthe commonalitybetweenA and B as the informati oncontent of the
propositionthatstates thecommonalitybetweenA andB:
IC(Common(A,B)) (20.24)
He measuresthedifferencebetweenA andBas
IC(description(A,B)) −IC(common(A,B)) (20.25)
where description(A,B)describesA and B. Given a few additi onalassumptions about
similarity,Linprovesthefollowingtheorem:

DRAFT668 Chapter 20. ComputationalLexicalSemantics
Similarity Theorem: The similarity between A and B is measur ed by the ratio
between the amount of information needed to state the common ality of A and B
andthe information needed tofullydescribe what A andB are:
simLin(A,B) =logP(common(A,B) )
logP(description(A,B) )(20.26)
Applyingthis idea to the thesaurusdomain, Lin shows (in a sl ight modiﬁcationof
Resnik’s assumption) that the information in common betwee n two concepts is twice
the information in the lowest common subsumer LCS (c1,c2). Adding in the above
deﬁnitions of the information content of thesaurus concept s, the ﬁnal Lin similarity Linsimilarity
functionis:
simLin(c1,c2) =2×logP(LCS(c1,c2))
logP(c1)+logP(c2)(20.27)
Forexample,usingsimlin, Lin (1998b)showsthat the similarity betweenthe con-
ceptsofhillandcoastfromFig.20.7is:
simLin(hill,coast) =2×logP(geological-formation )
logP(hill)+logP(coast))=0.59 (20.28)

A very similar formula, Jiang-Conrath distance (Jiang and Conrath, 1997) (al-Jiang-Conrath
distance
though derived in a completely different way from Lin, and ex pressed as a distance
rather than similarity function) has been shown to work as we ll or better than all the
otherthesaurus-basedmethods:
distJC(c1,c2) =2×logP(LCS(c1,c2))−(logP(c1)+logP(c2)) (20.29)
distjccanbetransformedintoasimilaritybytakingthereciproca l.
Finally, we describe a dictionary-based method, an extension of the Lesk algo-

distance
rithm for word-sense disambiguation described in Sec. 20.4 .1. We call this a dictio-
nary rather than a thesaurus method because it makes use of gl osses, which are in
general a property of dictionaries rather than thesauri (al though WordNet does have
glosses). Like the Lesk algorithm, the intuition of this Extended Gloss Overlap , orExtendedGlossOverlap
Extended Lesk measure (Banerjee and Pedersen, 2003) is that two concepts/ senses ExtendedLesk

in a thesaurus are similar if their glosses contain overlapp ing words. We’ll begin by
sketchinganoverlapfunctionfortwo glosses. Considerthe se two concepts,with their
glosses:
•drawingpaper: paperthatis speciallypreparedforuse indrafting
•decal:theartoftransferringdesignsfrom speciallypreparedpapertoawoodor
glassormetalsurface.
Foreachn-wordphrasethat occursin bothglosses, ExtendedLesk adds in a score
ofn2(the relation is non-linearbecause of the Zipﬁan relations hipbetween lengthsof

glosses:
phrases and their corpus frequencies; longer overlaps are r are so should be weighted
more heavily). Here the overlapping phrases are paperandspecially prepared , for a
totalsimilarityscoreof12+22=5.

DRAFTSection20.6. Word Similarity: ThesaurusMethods 669
Givensuchanoverlapfunction,whencomparingtwoconcepts (synsets),Extended
Lesk not only looksfor overlapbetweentheir glosses, but al so betweenthe glosses of
thesenses whicharehypernyms,hyponyms,meronyms,andoth errelationsofthe two
concepts. Forexampleifwejustconsideredhyponyms,andde ﬁnedgloss(hypo(A))as
theconcatenationofalltheglossesofallthehyponymsense sofA,thetotalrelatedness
betweentwo conceptsA andB mightbe:
similarity(A,B) =overlap(gloss(A),gloss(B))
+overlap(gloss(hypo(A)),gloss(hypo(B)))
+overlap(gloss(A),gloss(hypo(B)))
+overlap(gloss(hypo(A)),gloss(B))
Let RELS be the set of possible WordNet relations whose gloss es we compare;
assumingabasicoverlapmeasureassketchedabove,wecanth endeﬁnethe Extended
Leskoverlapmeasureas:
simeLesk(c1,c2) =∑
r,q∈RELSoverlap (gloss(r(c1)),gloss(q(c2))) (20.30)
simpath(c1,c2) =−logpathlen (c1,c2)
simResnik(c1,c2) =−logP(LCS(c1,c2))
logP(c1)+logP(c2)
simjc(c1,c2) =1

2×logP(LCS(c1,c2))−(logP(c1)+logP(c2))
r,q∈RELSoverlap (gloss(r(c1)),gloss(q(c2)))
Figure20.8 Five thesaurus-based (anddictionary-based) similaritym easures.
Fig.20.8summarizestheﬁvesimilaritymeasureswehavedes cribedinthissection
The publicly available Wordnet::Similarity package implementing all these
and other thesaurus-based word similarity measures is desc ribed in Pedersen et al.
(2004).
EvaluatingThesaurus-basedSimilarity Whichofthesesimilaritymeasuresisbest?

(2004).
Wordsimilaritymeasureshavebeenevaluatedintwo ways. On eintrinsicmethodisto
compute the correlation coefﬁcient between word similarit y scores from an algorithm
and word similarity ratings assigned by humans; such human r atings have been ob-
tainedfor65wordpairsbyRubensteinandGoodenough(1965) ,and30wordpairsby
Miller and Charles (1991). Another more extrinsic evaluati onmethod is to embed the
similaritymeasureinsomeendapplicationlikedetectiono fmalapropisms (real-word

spelling errors) (Budanitsky and Hirst, 2006; Hirst and Bud anitsky, 2005), or other

DRAFT670 Chapter 20. ComputationalLexicalSemantics
NLP applicationslike word-sensedisambiguation(Patward hanet al., 2003;McCarthy
et al., 2004) and evaluate its impact on end-to-end performa nce. All of these evalu-
ations suggest that all the above measures perform relative ly well, and that of these,
Jiang-Conrathsimilarity and ExtendedLesk similarity are two ofthe best approaches,
dependingonthe application.
20.7 WordSimilarity: DistributionalMethods
The previous section showed how to compute similarity betwe en any two senses in a
thesaurus,and by extensionbetweenany two wordsin the thes aurushierarchy. But of
course we don’t have such thesauri for every language. Even f or languageswhere we
do have such resources, thesaurus-based methods have a numb er of limitations. The
obviouslimitation is that thesaurioftenlack words, espec iallynew or domain-speciﬁc
words. In addition, thesaurus-based methods only work if ri ch hyponymyknowledge

is present in the thesaurus. While we have this for nouns, hyp onym information for
verbs tends to be much sparser, and doesn’t exist at all for ad jectives and adverbs.
Finally,itismoredifﬁcultwiththesaurus-basedmethodst ocomparewordsindifferent
hierarchies,suchasnounswithverbs.
For these reasons, methods which can automatically extract synonyms and other
word relations from corpora have been developed. In this sec tion we introduce such
distributional methods, which can be applied directly to supply a word relat edness

measure for NLP tasks. Distributional methods can also be us ed forautomatic the-
saurusgeneration forautomaticallypopulatingoraugmentingon-linethesau ruseslike
WordNetwithnewsynonymsand,aswewillseeinSec.20.8,wit hotherrelationslike
hyponymyandmeronymy.
The intuition of distributional methods is that the meaning of a word is related to
the distribution of words around it; in the famous dictum of F irth (1957), “You shall
knowawordbythecompanyit keeps!”. Considerthefollowing example,modiﬁedby
Lin(1998a)from(?):
(20.31)Abottleof tezg¨uinoisonthetable.

Everybodylikes tezg¨uino.
Tezg¨uinomakesyoudrunk.
We make tezg¨uinooutofcorn.
The contexts in which tezg¨uinooccurs suggest that it might be some kind of fer-
mentedalcoholicdrinkmadefromcorn. Thedistributionalm ethodtriestocapturethis
intuitionbyrepresentingfeaturesofthecontextof tezg¨uinothatmightoverlapwithfea-
tures of similar words like beer,liquor,tequila, and so on. For example such features
mightbe occursbeforedrunkoroccursafterbottleoristhedirectobjectoflikes.

We can then represent a word was afeature vector just as we saw with the bag- feature vector
of-words features in Sec. 20.2. For example, suppose we had o ne binary feature fi
representingeach of the Nwordsin the lexicon vi. The feature means woccursinthe
neighborhoodofwordvi,andhencetakesthevalue1if wandvioccurinsomecontext
window, and 0 otherwise. We could represent the meaning of wo rdwas the feature

DRAFTSection20.7. WordSimilarity: DistributionalMethods 671
vector
⃗w= (f1,f2,f3,···,fN)
Ifw=tezg¨uino,v1=bottle,v2=drunk,andv3=matrix,theco-occurrencevectorfor w
fromthecorpusabovewouldbe:
⃗w= (1,1,0,···)
Giventwowordsrepresentedbysuchsparsefeaturevectors, wecanapplyavector
distance measure and say that the words are similar if the two vectors are close by
this measure. Fig. 20.7 shows an intuition about vector simi larity for the four words
apricot,pineapple ,digital, andinformation . Based on the meanings of these four
words, we would like a metric that shows apricotandpineapple to be similar, digital
andinformation , to be similar, and the other four pairings to produce low sim ilarity.
For each word, Fig. 20.7 shows a short piece (8 dimensions) of the (binary) word co-
occurrence vectors, computed from words that occur within a two-line context in the
Brown corpus. The reader should convince themselves that th e vectors for apricot

andpineapple are indeed more similar than those of, say, apricotandinformation .
For pedagogical purposes we’ve shown the context words that are particularly good
at discrimination. Notethat since vocabulariesarequite l arge(10,000-100,000words)
andmostwordsdon’toccurneareachotherinanycorpus,real vectorsarequitesparse.
arts boil data function large sugar summarized water
apricot 01 0 0 11 0 1
pineapple 01 0 0 11 0 1
digital 0 0 1 1 1 0 1 0
information 0 0 1 1 1 0 1 0

Figure20.9 Co-occurrence vectors for four words, computed from the Bro wn corpus, show-
ing only 8 of the (binary) dimensions (hand-picked for pedag ogical purposes to show discrimi-
nation). Notethat largeoccurs inallthecontexts and artsoccurs innone; arealvectorwouldbe
extremelysparse.
Now that we have some intuitions, let’s move on to examine the details of these
measures. Specifyingadistributionalsimilaritymeasure requiresthatwespecifythree
parameters: (1) how the co-occurrenceterms are deﬁned (i.e . what counts as a neigh-

bor),(2)howthese termsare weighted(binary? frequency? m utualinformation?) and
(3) what vector distance metric we use (cosine? Euclidean di stance?). Let’s look at
eachoftheserequirementsin thenextthreesubsections.
20.7.1 Deﬁning a Word’s Co-occurrence Vectors
In our example feature vector, we used the feature woccursintheneighborhoodof
wordvj. That is, for a vocabulary size N, each word whadNfeatures, specifying
whether vocabulary element vjoccurred in the neighborhood. Neighborhoods range
from a small window of words (as few as one or two words on eithe r side) to very

largewindowsof±500words. Ina minimalwindow,forexample,wemighthavetwo

DRAFT672 Chapter 20. ComputationalLexicalSemantics
features for each word vjin the vocabulary, wordvkoccursimmediatelybeforeword
wandwordvkoccursimmediatelyafterwordw.
To keep these contexts efﬁcient, we often ignore very freque nt words which tend
not to be very discriminative, e.g., function words such as a,am,the,of,1,2, and so
on. Theseremovedwordsarecalled stopwords orthestoplist. stopwords
stoplist Evenwiththeremovalofthestopwords,whenusedonverylarg ecorporatheseco-
occurrencevectorstend to be very large. Insteadof using ev eryword in the neighbor-
hood,Hindle(1990)suggestedchoosingwordsthatoccurins omesortof grammatical
relationordependency to the target words. Hindle suggested that nouns which bear
the same grammatical relation to the same verb might be simil ar. For example, the
wordstea,water, andbeerare all frequentdirect objectsofthe verb drink. The words
senate,congress,panel, andlegislature all tend to be subjects of the verbs consider,
vote,andapprove.

Hindle’s intuition follows from the early work of Harris (19 68), who suggested
that:
Themeaningofentities,andthemeaningofgrammaticalrela tionsamong
them,is relatedto the restrictionof combinationsof these entitiesrelative
tootherentities.
There have been a wide variety of realizationsof Hindle’s id ea since then. In general,
in these methods each sentence in a large corpus is parsed and a dependency parse is
extracted. We saw in Ch. 12 lists of grammatical relations pr oduced by dependency

that:
parsers, including noun-verb relations like subject, obje ct, indirect object, and noun-
noun relations like genitive, ncomp, and so on. A sentence li ke the following would
resultin theset ofdependenciesshownhere:
(20.32)Idiscovereddriedtangerines:
discover (subject I) I (subj-of discover)
tangerine (obj-of discover) tangerine (adj-mod dried)
dried(adj-mod-of tangerine)
Since each word can be in a variety of different dependency re lations with other

words, we’ll need to augment the feature space. Each feature is now a pairing of a
word and a relation, so instead of a vector of Nfeatures, we have a vector of N×R
features, where Ris the number of possible relations. Fig. 20.10 shows a schem atic
example of such a vector, taken from Lin (1998a), for the word cell. As the value of
each attribute we have shown the frequency of the feature co- occurring with cell; the
nextsectionwill discusstheuse ofwhatvaluesandweightst ouse foreachattribute.

Since full parsing is very expensive, it is common to use a chu nker or shallow
parserofthetypedeﬁnedinSec.13.5,withthegoalofextrac tingonlyasmallerset of
relationslike subject,directobject,andprepositionalo bjectof aparticularpreposition
(Curran,2003).
20.7.2 Measures ofAssociationwithContext
Nowthatwehaveadeﬁnitionforthefeaturesordimensionsof aword’scontextvector,
we arereadytodiscussthevaluesthatshouldbeassociatedw ith thosefeatures. These

values are typically thought of as weightsor measures of association between each association

DRAFTSection20.7. WordSimilarity: DistributionalMethods 673subj-of,absorb
subj-of,adapt
subj-of,behave
...
pobj-of,inside
pobj-of,into
...
nmod-of,abnormality
nmod-of,anemia
nmod-of,architecture
...
obj-of,attack
obj-of,call
obj-of,comefrom
obj-of,decorate
...
nmod,bacteria
nmod,body
nmod,bonemarrow
cell111163038161132322
Figure20.10 Co-occurrence vectorfortheword cell,fromLin(1998a),showinggrammatical
function(dependency) features. Valuesforeachattribute arefrequencycountsfroma64-million
wordcorpus, parsed byanearlyversion of MINIPAR.
target word wand a given feature f. In the example in Fig. 20.7, our association
measurewasa binaryvalueforeachfeature,1 iftherelevant wordhadoccurredinthe
context, 0 if not. In the example in Fig. 20.10, we used a riche r association measure,
the relative frequency with which the particular context fe ature had co-occurred with
thetargetword.
Frequency, or probability, is certainly a better measure of association than just a

binary value; features that occur often with a target word ar e more likely to be good
indicators of the word’s meaning. Let’s deﬁne some terminol ogy for implementing
a probabilistic measure of association. For a target word w, each element of its co-
occurrence vector is a feature f, consisting of a relation rand a related word w′; we
can sayf= (r,w′). For example, one of the features of the word cellin Fig. 20.10 is
f= (r,w′) =(obj-of,attack). The probability of a feature fgiven a target word wis

P(f|w), forwhichthemaximumlikelihoodestimate is:
P(f|w) =count(f,w)
count(w)(20.33)
Similarly,themaximumlikelihoodestimate forthejointpr obability P(f,w)is:
P(f,w) =count(f,w)
∑w′count(w′))(20.34)
P(w)andP(f)are computedsimilarly.
Thus if we were to deﬁne simple probability as a measure of ass ociation it would
lookasfollows:
assocprob(w,f) =P(f|w) (20.35)
It turnsout, however,that simple probabilitydoesn’twork as well as moresophis-
ticatedassociationschemesforwordsimilarity.

Why isn’t frequencyor probabilitya good measure of associa tion between a word
and a context feature? Intuitively, if we want to know what ki nds of contexts are
sharedby apricotandpineapple butnotby digitalandinformation ,we’renotgoingto
get good discrimination from words like the,it, orthey, which occur frequently with
all sorts ofwords,and aren’tinformativeaboutanyparticu larword. We’d like context
words which are particularly informative about the target w ord. We, therefore, need

DRAFT674 Chapter 20. ComputationalLexicalSemantics
a weighting or measure of association which asks how much mor e often than chance
that the feature co-occurs with the target word. As Curran (2 003) points out, such a
weighting is what we also want for ﬁnding good collocations , and so the measuresof collocations
association used for weighting context words for semantic s imilarity are exactly the
samemeasureusedforﬁndinga word’scollocations.
One of the most important measures of association was ﬁrst pr oposed by Church
and Hanks (1989,1990) and is based on the notion of mutual information . Themu-
tualinformation betweentworandomvariables XandYismutualinformation
I(X,Y) =∑
x∑
yP(x,y)log2P(x,y)
P(x)P(y)(20.36)
Thepointwise mutual information (Fano, 1961)3is a measure of how often twopointwise mutual
eventsxandyoccur,comparedwith whatwewouldexpectif theywereindepe ndent:
I(x,y) =log2P(x,y)
P(x)P(y)(20.37)
We can apply this intuition to co-occurrence vectors, by deﬁ ning the pointwise

I(X,Y) =∑
x∑
mutualinformationassociationbetweena targetword wanda feature fas:
assocPMI(w,f) =log2P(w,f)
P(w)P(f)(20.38)
The intuition of the PMI measure is that the numerator tells u s how often we ob-
servedthetwowordstogether(assumingwecomputeprobabil ityusingMLEasabove).
Thedenominatortellsushowoftenwewould expectthetwowordstoco-occurassum-
ing they each occurred independently, so their probabiliti es could just be multiplied.
Thus the ratio gives us an estimate of how much more the target and feature co-occur
thanweexpectbychance.

Sincefis itself composed of two variables randw′, there is a slight variant on
this model, due to Lin (1998a), that breaks down the expected value for P(f)slightly
differently;we’llcallitthe Linassociationmeasure assocLin,nottobeconfusedwithLin associationmeasure
theWordNetmeasuresimLinthatwe discussedintheprevioussection:
assocLin(w,f) =log2P(w,f)
P(w)P(r|w)P(w′|w)(20.39)
For both assocPMIand assocLin, we generally only use the feature ffor a word
wif the assoc value is positive, since negative PMI values (im plying things are co-

occurring less often than we would expect by chance) tend to be unreliable unless t he
training corpora are enormous (Dagan et al., 1993; Lin, 1998 a). In addition, when
we are using the assoc-weighted features to compare two targ et words, we only use
featuresthatco-occurwithbothtargetwords.
Fig.20.11fromHindle(1990)showsthedifferencebetweenr awfrequencycounts
andPMI-styleassociation,forsome directobjectsoftheve rbdrink.
3Fano actually used the phrase mutual information to refer to what we now call pointwise mutual infor-

mation, and the phrase expectation of the mutual information for what we now call mutual information ; the
termmutual information is still often used to mean pointwise mutual information .

DRAFTSection20.7. WordSimilarity: DistributionalMethods 675
Object Count PMIassoc Object Count PMIassoc
bunchbeer 2 12.34 wine 2 9.34
tea 2 11.75 water 7 7.65
Pepsi 2 11.75 anything 3 5.15
champagne 4 11.75 much 3 5.15
liquid 2 10.53 it 3 1.25
beer 5 10.20 <SOMEAMOUNT >2 1.22
Figure20.11 Objects of the verb drink, sortedbyPMI, from Hindle (1990).
One of the most successful association measures for word sim ilarity attempts to
capturethesameintuitionasmutualinformation,butusest het-teststatistictomeasure t-test
how much more frequent the association is than chance. This m easure was proposed
for collocation-detectionby Manningand Sch¨ utze (1999,C hapter 5) and then applied
towordsimilaritybyCurranandMoens(2002),Curran(2003) .
Thet-test statistic computesthe differencebetweenobser vedandexpectedmeans,
normalizedbythevariance. Thehigherthevalueof t,themorelikelywecanrejectthe
nullhypothesisthat theobservedandexpectedmeansarethe same.
t=¯x−µ√
s2
N(20.40)

t=¯x−µ√
s2
N(20.40)
When applied to association between words, the null hypothe sis is that the two
words are independent, and hence P(f,w) =P(f)P(w)correctly models the relation-
ship between the two words. We want to know how different the a ctual MLE proba-
bilityP(f,w)is from this null hypothesisvalue, normalizedby the varian ce. Note the
similarity to the comparison with the product model in the PM I measure above. The
variances2can be approximatedby the expected probability P(f)P(w)(see Manning

t=¯x−µ√
s2
N(20.40)
and Sch¨ utze (1999)). Ignoring N(since it is constant), the resulting t-test association
measurefromCurran(2003)isthus:
assoct-test(w,f) =P(w,f)−P(w)P(f)√
P(f)P(w)(20.41)
See the history section for a summary of variousother weight ing factorsthat have
beentestedonwordsimilarity.
20.7.3 Deﬁning similaritybetween two vectors
From the previous sections we can now compute a co-occurrenc e vector for a target
word, with each co-occurrencefeature weighted by an associ ation measure, giving us

a distributionaldeﬁnitionofthemeaningofa targetword.
To deﬁne similarity between two target words vandw, we need a measure for
takingtwosuchvectorsandgivingameasureofvectorsimila rity. Perhapsthesimplest
two measuresof vectordistanceare the Manhattanand Euclid eandistance. Fig.20.12
shows a graphical intuition for Euclidean and Manhattan dis tance between two two-
dimensional vectors ⃗aand⃗b. TheManhattan distance , also known as LevenshteinManhattandistance
distanceorL1 norm ,isLevenshteindistance
L1 norm

DRAFT676 Chapter 20. ComputationalLexicalSemantics
distancemanhattan(⃗x,⃗y) =N
∑
i=1|xi−yi| (20.42)
TheEuclidean distance ,also calledthe L2 norm ,wasintroducedinCh. 9: L2 norm
distanceeuclidean(⃗x,⃗y) =√
N
∑
i=1(xi−yi)2(20.43)
Figure20.12 The Euclidean and Manhattan distance metrics for vectors a= (a1,a2), and
b= (b1,b2), just to give the reader a graphical intuition about the idea of distance between
vectors; these particular metrics are generally not used fo r word similarity. See Ch. 9 for more
ondistance metrics.
Although the Euclidean and Manhattan distance metrics prov ide a nice geometric
intuition for vector similarity and distance, these measur es are rarely used for word
similarity. Thisisbecausebothmeasuresturnouttobevery sensitivetoextremevalues.
Instead of these simple distance metrics, word similarity i s based on closely related
metrics from information retrieval and from information theory . The information
retrieval methodsseem to workbetter for wordsimilarity, s o we’ll deﬁne a numberof

thesein thissection.
Let’s begin with the intuition for a similarity metric in Fig . 20.7, in which the
similarity between two binary vectors was just the number of features the two words
had in common. If we assume a feature vectoris a binary vector , we can deﬁne such binary vector
a similarity metric as follows, using the dot product orinner product operator from dot product
innerproduct linearalgebra:
simdot-product(⃗v,⃗w) =⃗v·⃗w=N
∑
i=1vi×wi (20.44)

∑
Inmostcases,though,aswesawintheprevioussection,thev aluesofourvectorare
notbinary. Let’sassumefortherestofthissectionthatthe entriesintheco-occurrence
vectorarethe association valuesbetweenthetargetwordsandeachofthefeatures. In
otherwords,let’sdeﬁnethevectorforatargetword ⃗wwithNfeaturesf1..fNas:
⃗w= (assoc(w,f1),assoc(w,f2),assoc(w,f3),...,assoc(w,fN)) (20.45)

DRAFTSection20.7. WordSimilarity: DistributionalMethods 677
Nowwecanapplysimdot-producttovectorswithvaluesdeﬁnedasassociations,to
getthedot-productsimilaritybetweenweightedvalues. Th israwdot-product,however,
has a problem as a similarity metric: it favors longvectors. The vector length is vectorlength
deﬁnedas:
|⃗v|=√
N
∑
i=1v2
i(20.46)
A vector can be longer because it has more non-zero values, or because each dimen-
sion has a higher value. Both of these facts will increase the dot product. It turns
out that both of these can occur as a by-product of word freque ncy. A vector from
a very frequent word will have more non-zero co-occurrence a ssociation values, and
will probably have higher values in each (even using associa tion weights that control
somewhatforfrequency). Therawdotproductthusfavorsfre quentwords.
Weneedtomodifythedotproducttonormalizeforthevectorl ength. Thesimplest
way is just to divide the dot product by the lengths of each of t he two vectors. This

normalized dot product turns out to be the same as the cosine of the angle betweennormalized dotproduct
thetwo vectors. The cosineornormalizeddotproductsimilarity metricis thus: cosine
simcosine(⃗v,⃗w) =⃗v·⃗w
|⃗v||⃗w|=∑N
i=1vi×wi√
∑N
i=1v2
i√
∑N
i=1w2
i(20.47)
Because we have transformed the vectors to unit length, the c osine metric, unlike
Euclidean or Manhattan distance, is no longer sensitive to l ong vectors from high-
frequency words. The cosine value ranges from 1 for vectors p ointing in the same

i=1vi×wi√
∑N
i=1v2
i√
∑N
i=1w2
i(20.47)
direction, through 0 for vectors which are orthogonal (shar e no common terms), to
-1 for vectors pointing in opposite directions, although in practice values tend to be
positive.
Let’sdiscusstwomoresimilaritymeasuresderivedfrominf ormationretrieval. The
Jaccard (Jaccard, 1908, 1912) (also called Tanimoto ormin/max (Dagan, 2000)) Jaccard
Tanimoto
min/maxmeasure was originally designed for binary vectors. It was e xtended by Grefenstette
(1994)tovectorsofweightedassociationsasfollows:
simJaccard(⃗v,⃗w) =∑N

positive.
Tanimoto
i=1min(vi,wi)
∑N
i=1max(vi,wi)(20.48)
The numerator of the Grefenstette/Jaccard function uses th e min function, essen-
tially computingthe (weighted) number of overlappingfeat ures (since if either vector
has a zero association value for an attribute, the result wil l be zero). The denominator
canbeviewedasa normalizingfactor.
A very similar measure, the Dicemeasure, was similarly extended from binary Dice
vectorstovectorsofweightedassociations;oneextension fromCurran(2003)usesthe

∑N
Jaccardnumerator,butusesasthedenominatornormalizati onfactorthetotalweighted
valueofnon-zeroentriesin thetwovectors.
simDice(⃗v,⃗w) =2×∑N
∑N
i=1(vi+wi)(20.49)

DRAFT678 Chapter 20. ComputationalLexicalSemantics
assocprob(w,f) =P(f|w) ( 20.35)
i=1vi×wi√
∑N
i=1v2
i√
∑N
i=1w2
i(20.47)
∑N
∑N
simJS(⃗v||⃗w) = D(⃗v|⃗v+⃗w
2)+D(⃗w|⃗v+⃗w
2) ( 20.52)
Figure20.13 Deﬁning word similarity: measures of association between a target word wand
a feature f= (r,w′)to another word w′, and measures of vector similarity between word co-
occurrence vectors ⃗vand⃗w.
Finally,thereisafamilyofinformation-theoreticdistri butionalsimilaritymeasures,
(Pereiraetal.,1993;Daganetal.,1994,1999;Lee,1999),a lsobasedontheconditional
probabilityassociationmeasure P(f|w). Theintuitionofthesemodelsisthat two vec-
tors⃗vand⃗ware similar to the extent that their probability distributi onsP(f|w)and

P(f|v)aresimilar. Thebasisofcomparingtwoprobabilitydistrib utionsPandQisthe
Kullback-Leibler divergence orKL divergence orrelative entropy (Kullback and KL divergence
Leibler,1951):
D(P||Q) =∑
Q(x)(20.50)
Unfortunately,theKL-divergenceisundeﬁnedwhen Q(x) =0andP(x)̸=0,which
is a problem since these word distribution vectors are gener ally quite sparse. One
alternative (Lee, 1999) is to use the Jenson-Shannon divergence , which representsJenson-Shannondivergence
the divergence of each distribution from the mean of the two, and doesn’t have this
problemwith zeros:

D(P||Q) =∑
JS(P||Q) =D(P|P+Q
2)+D(Q|P+Q
2) (20.51)
Rephrasedintermsofvectors ⃗vand⃗w,
simJS(⃗v||⃗w) =D(⃗v|⃗v+⃗w
2) (20.52)
Fig. 20.13 summarizes the measures of association and of vec tor similarity that
we have designed. See the history section for a summary of oth er vector similarity
measures.

DRAFTSection20.7. WordSimilarity: DistributionalMethods 679
Finally, let’s look at some of the results of distributional word similarity. The fol-
lowingarethetenmostsimilarwordstothedifferentpartso fspeechof hopeandbrief,
derivedusingtheonlinedependency-basedsimilaritytool (Lin,2007);thistooldeﬁnes
the co-occurrence vector using all minipar grammatical rel ations, uses the assocLin
measureofassociation,anda vectorsimilaritymetricfrom Lin(1998a).
•hope (N): optimism 0.141, chance 0.137, expectation 0.137, prospect 0.126,
dream 0.119, desire 0.118, fear 0.116, effort 0.111, conﬁde nce 0.109, promise
0.108
•hope (V): would like 0.158, wish 0.140, plan 0.139, say 0.137, believe 0.135,
think0.133,agree0.130,wonder0.130,try0.127,decide0. 125,
•brief (N): legal brief 0.139, afﬁdavit 0.103, ﬁling 0.0983, petition 0 .0865, doc-
ument 0.0835, argument 0.0832, letter 0.0786, rebuttal 0.0 778, memo 0.0768,
article0.0758
•brief(A): lengthy0.256,hour-long0.191,short0.174,extended0.16 3,frequent

0.108
0.163,recent0.158,short-lived0.155,Prolonged0.149,w eek-long0.149,occa-
sional0.146
20.7.4 Evaluating DistributionalWordSimilarity
Distributional similarity can be evaluated in the same ways as thesaurus-based simi-
larity; we can compare intrinsically to human similarity sc ores, or we can evaluate it
extrinsically as part of end-to-end applications. Besides word sense disambiguation
and malapropism detection, similarity measures have been u sed as a part of systems
forthegradingofexamsandessays(Landaueretal.,1997),o rtakingTOEFLmultiple-

choiceexams(LandauerandDumais,1997;Turneyet al.,2003 ).
Distributionalalgorithmsarealso oftenevaluatedina thi rdintrinsicway: by com-
parison with a gold-standard thesaurus. This comparison ca n be direct with a single
thesaurus (Grefenstette, 1994; Lin, 1998a) or by using prec ision and recall measure
against an ensemble of thesauri (Curran and Moens, 2002). Le tSbe the set of words
that are deﬁned as similar in the thesaurus, by being in the sa me synset, or perhaps
sharingthesamehypernym,orbeinginthehypernym-hyponym relation. Let S′bethe

set of words that are classiﬁed as similar by some algorithm. We can deﬁne precision
andrecallas:
precision =|S∩S′|
|S′|recall=|S∩S′|
|S|(20.53)
Curran (2003) evaluated a number of distributional algorit hms using comparison
withthesauriandfoundthattheDiceandJaccardmethodsper formedbestasmeasures
of vector similarity, while t-test performedbest as a measu re of association. Thus the
best metric weighted the associations with t-test, and then used either Dice or Jaccard
tomeasurevectorsimilarity.

DRAFT680 Chapter 20. ComputationalLexicalSemantics
20.8 Hyponymyandotherwordrelations
Similarity is only one kind of semantic relation between wor ds. As we discussed in
Ch. 19, WordNet and MeSH both include hyponymy/hypernymy , as do many the-
sauruses for other languages, such as CiLin for Chinese (?). WordNet also includes
antonymy ,meronymy ,andotherrelations. Thusifwewanttoknowiftwosensesare
related by one of these relations, and the senses occur in Wor dNet or MeSH, we can
just look them up. But since many words are not in these resour ces, it is important to
beableto learnnewhypernymandmeronymrelationsautomati cally.
Much work on automatic learning of word relations is based on a key insight ﬁrst
articulated by Hearst (1992), that the presence of certain l exico-syntactic patterns can
indicatea particularsemantic relationshipbetweentwo no uns. Considerthefollowing
sentenceextractedbyHearst fromtheGroliersencyclopedi a:
(20.54)Agarisa substancepreparedfromamixtureofredalgae,such asGelidium,

forlaboratoryorindustrialuse.
Hearstpointsoutthatmosthumanreaderswillnotknowwhat Gelidium is,butthat
they can readily infer that it is a kind of (a hyponym of)red algae , whatever that is.
Shesuggeststhat thefollowing lexico-syntacticpattern
NP0suchasNP 1{,NP2...,(and|or)NPi},i≥1 (20.55)
impliesthefollowingsemantics
∀NPi,i≥1,hyponym (NPi,NP0) (20.56)
allowingustoinfer
hyponym (Gelidium ,redalgae ) (20.57)
NP{,NP}∗{,}(and|or) otherNPH ...temples,treasuries, and other important civic buildin gs.
NPHsuch as {NP,}*(or|and)NP red algae such as Gelidium

suchNPHas{NP,}*(or|and)NP works by such authors as Herrick, Goldsmith, and Shakespear e
NPH{,}including {NP,}*(or|and)NP Allcommon-law countries, including Canada and England
NPH{,}especially {NP,}*(or|and)NP ...mostEuropean countries, especially France, England, a nd Spain
Figure20.14 Hand-built lexico-syntactic patterns for ﬁnding hypernym s (Hearst,1992, 1998)
Fig. 20.14 shows ﬁve patterns Hearst (1992, 1998) suggested for inferring the hy-
ponymrelation;we’veshown NPHastheparent/hyponym.Thereareanumberofother

attemptstoextractdifferentWordNetrelationsusingsuch patterns;seethehistorysec-
tionformoredetails.
Ofcourse,thecoverageofsuchpattern-basedmethodsislim itedbythenumberand
accuracyoftheavailablepatterns. Unfortunately,onceth eobviousexampleshavebeen
found, the process of creating patterns by hand becomes a dif ﬁcult and slow process.
Fortunately, we’ve already seen the solution to this kind of problem. We can ﬁnd
newpatternsusing bootstrapping methodsthatarecommonininformationextraction
(Riloff, 1996; Brin, 1998), and are also key to the Yarowskym ethod described earlier
inSec. 20.5.

DRAFTSection20.8. Hyponymyandotherwordrelations 681
The key insight for the use of bootstrapping in relational pa ttern discovery is that
with a large corpus we can expect that words involved in a rela tion to show up with
many different patterns that express that same relation. Th erefore, in theory at least,
we need only start with a small number of precise patterns to a cquire a set of seed
words involved in a given relation. These words can then be us ed to query a large
corpus for sentences containing both terms in some kind of de pendencyrelation; new
patternscan then be extractedfrom these new sentences. The processcan be repeated
untilthepatternset islargeenough.
As an example of this process, consider the terms “red algae” and “Gelidium”
discovered earlier using Hearst’s simple pattern set. Amon g the results of a simple
Googlesearchusingtheseasquerytermsisthefollowingexa mple:
(20.58)Oneexampleofa redalgaeisGelidium.
Removing the seed words from such a sentence and replacing th em with simple

wildcardsis the crudestkind ofpatterngeneration. In this case, submittingthe pattern
“Oneexampleofa*is*”toGooglecurrentlyyieldsnearly500 ,000hits,includingthe
followingexample:
(20.59)Oneexampleofa bosonisa photon.
We can also extract slightly more sophisticated patterns by parsing the extracted
sentencesandputtingwildcardsintotheparsetree.
The key to the success of bootstrapping approaches is to avoi d thesemantic drift
that tends to occur as part of repeated applications of boots trapping. The further we

get fromtheoriginalset of seed wordsor patternsthe moreli kelyit isthat we’ll come
across patterns with meanings quite different from what we s et out to discover. We’ll
see methodsfor dealingwith this drift whenwe discuss boots trappingforinformation
extractioninCh.22.
An alternative to bootstrappingis to use large lexical reso urces like WordNet as a
source of training information, in which each WordNet hyper nym/hyponympair tells

ussomethingaboutkindsof wordsare inthis relation,andwe traina classiﬁer to help
ﬁndnewwordsthatexhibitthisrelation.
This hyponym learning algorithm of Snow et al. (2005), for ex ample, relies on
WordNet to help learn large numbers of weak hyponym patterns , and then combine
themina supervisedclassiﬁerin 4steps:
1. Collect all pairs of WordNet noun concepts ci,cjthat are in the hypernym /
hyponymrelation.
2. For each noun pair, collect all sentences (in a 6 million wo rd corpus) in which
bothnounsoccur.
3. Parsethesentencesandautomaticallyextracteveryposs ibleHearst-stylelexico-

syntacticpatternfromtheparsetree
4. Usethe largeset ofpatternsasfeaturesinanlogisticreg ressionclassiﬁer
5. Given a pair of nouns in the test set, extract features and u se the classiﬁer to
determineif thenounpairisrelatedbythehypernym/hypony mrelationornot.
Fourofthenewpatternsautomaticallylearnedbythisalgor ithminclude:

DRAFT682 Chapter 20. ComputationalLexicalSemantics
NPHlike NP NP HcalledNP
NP isa NP H NP,a NP H(appositive):
Snow et al. (2005) then showed good hypernym detection perfo rmance by using
eachofthesepatternsasaweakfeaturecombinedbya logisti cregressionclassiﬁer.
AnotherwaytouseWordNettohelpaddressthehypernymprobl emistomodelthe
task aschoosingtheplaceto insertunknownwordsintoan oth erwisecompletehierar-
chy. It is possible to do this without using lexico-syntacti c patterns. For example, we
canuseasimilarityclassiﬁer(usingdistributionalinfor mation,ormorphologicalinfor-
mation) to ﬁnd the words in the hierarchy that are most simila r to an unknown word,
using an approach like K-Nearest-Neighbors, and insert the new word there (Tseng,
2003). Or we can treat the task of hypernym labeling as a label ing task like named-
entity tagging. Ciaramita and Johnson (2003) take this appr oach, using as tags 26
supersenses , from the 26 broad-category ‘lexicographer class’ labels f rom WordNet supersenses

(person,location,event,quantity,etc). Theyuse featuressuch as surroundingpart-of-
speech tags, word bigram and trigram features, spelling and morphological features,
andapplyamulticlassperceptronclassiﬁer.
Findingmeronyms seems to be harder than hyponyms; here are some examples
fromGirjuet al.(2003):
(20.60)The car’s mail messenger isbusyat work inthe <PART>mailcar </PART> as the
<WHOLE> train</WHOLE> moves along.
(20.61)Through the open <PART>sidedoor </PART> of the<WHOLE> car</WHOLE> ,

movingscenery can be seen.
Meronyms are hard to ﬁnd because the lexico-syntactic patte rns that characterize
them are very ambiguous. For example the two most common patt erns indicating
meronymyaretheEnglishgenitiveconstructions[NP 1ofNP2]and[NP 1’sNP2],which
alsoexpressmanyothermeaningssuchas possession ;seeGirjuetal.(2003,2006)for
discussionandpossiblealgorithms.
Learningindividualrelationsbetweenwordsisanimportan tcomponentofthegen-

eral task of thesaurus induction . In thesaurus induction, we combine our estimatesthesaurusinduction
of word similarity with our hypernymor other relations to bu ild an entire ontologyor
thesaurus. Forexamplethetwo-stepthesaurusinductional gorithmofCaraballo(1999,
2001) ﬁrst applies a bottom-up clustering algorithm to group together semantically
similar words into an unlabeled word hierarchy. Recall from Sec. 20.10 that in ag-
glomerative clustering, we start by assigning each word its own cluster. New clusters

are then formed in a bottom-up fashion by successively mergi ng the two clusters that
are most similar; we can use any metric for semantic similari ty, such as one of the
distributional metrics described in the previous section. In the second step, given the
unlabeledhierarchy,the algorithmuses a pattern-basedhy ponymclassiﬁer to assign a
hypernymlabel to each cluster of words. See the historysect ion for morerecentwork
onthesaurusinduction.

DRAFTSection20.9. SemanticRole Labeling 683
20.9 SemanticRoleLabeling
The ﬁnal task we’ll discuss in this chapter links word meanin gs with sentence mean-
ings. Thisisthetaskof semanticrolelabeling ,sometimescalled thematicrolelabel-semantic rolelabeling
ing,case role assignment or evenshallow semantic parsing . Semantic role labeling
isthetaskofautomaticallyﬁndingthe semanticroles foreachpredicateinasentence.
Morespeciﬁcally,thatmeansdeterminingwhichconstituen tsinasentenceareseman-
tic argumentsfora givenpredicate,andthen determiningth e appropriateroleforeach
ofthosearguments. Semanticrolelabelinghasthepotentia ltoimproveperformancein
any language understanding task, although to date its prima ry applications have been
inquestionansweringandinformationextraction.
Current approaches to semantic role labeling are based on su pervised machine
learning and hence require access to adequate amounts of tra ining and testing mate-
rials. Over the last few years, both the FrameNet and PropBan k resources discussed

in Ch. 19 have played this role. That is, they have been used to specify what counts
as a predicate, to deﬁne the set of roles used in the task and to provide training and
test data. The SENSEVAL -3 evaluation used Framenet, while the CONLL evaluations
in2004and2005werebasedonPropBank.
The following examples show the different representations from the two efforts.
RecallthatFrameNet(20.62)employsalargenumberofframe -speciﬁcframeelements
asroles,whilePropBank(20.63)makesuseofasmallernumbe rofnumberedargument
labelswhichcanbeinterpretedasverb-speciﬁclabels.

(20.62)[You] can’t [blame] [the program] [for being unable toident ifyit]
COGNIZER TARGET EVALUEE REASON
(20.63)[The SanFranciscoExaminer] issued [a special edition] [ye sterday]
ARG0 TARGET ARG 1 ARGM-TMP
A simpliﬁed semantic role labelingalgorithmis sketchedin Fig. 20.15. Following
the very earliest work on semantic role analysis (Simmons, 1 973), most work on se-
manticrolelabelingbeginsbyparsingthesentence. Public lyavailablebroad-coverage
parsers(suchasCollins(1996)orCharniak(1997))aretypi callyusedtoassignaparse

to the input string. Fig. 20.16 shows a parse of (20.63) above . The resulting parse
is then traversed to ﬁnd all predicate-bearing words. For ea ch of these predicates the
tree is again traversed to determine which role, if any, each constituent in the parse
plays with respect to that predicate. This judgment is made b y ﬁrst characterizing the
constituent as a set of features with respect to the predicat e. A classiﬁer trained on
an appropriate training set is then passed this feature set a nd makes the appropriate
assignment.

Let’slookinmoredetailatthesimplesetoffeaturessugges tedbyGildeaandJuraf-
sky(2000,2002),whichhavebeenincorporatedintomostrol e-labelingsystems. We’ll
extract them for the ﬁrst NPin Fig. 20.16, the NP-SBJconstituent the San Francisco
Examiner .
•The governing predicate , in this case the verb issued. For PropBank, the pred-
icates are always verbs; FrameNet also has noun and adjectiv e predicates. The

DRAFT684 Chapter 20. ComputationalLexicalSemantics
function SEMANTIC ROLELABEL(words)returnslabeled tree
parse←PARSE(words)
for eachpredicate inparsedo
for eachnodeinparsedo
featurevector←EXTRACTFEATURES (node,predicate,parse)
CLASSIFYNODE(node,featurevector ,parse)
Figure20.15 A generic semantic role labeling algorithm. The C LASSIFYNODEcomponent
can be a simple 1-of-N classiﬁer which assigns a semantic rol e (or NONE for non-role con-
stituents). C LASSIFYNODEcanbe trainedonlabeled data suchas FrameNet or PropBank.
S
NP-SBJ=ARG0 VP
DT NNP NNP NNP
The San FranciscoExaminer
VBD=TARGET NP=ARG1 PP-TMP =ARGM-TMP
issued DT JJ NN IN NP
a special edition around NN NP-TMP
noon yesterday
Figure20.16 Parse tree for a PropBank sentence, showing the PropBank arg ument labels. The dotted line shows
thepathfeature NP↑S↓VP↓VBDfor ARG0,the NP-SBJconstituent the San Francisco Examiner .
predicate is a crucial feature, since both PropBank and Fram eNet labels are de-
ﬁnedonlywithrespecttoa particularpredicate.

S
•Thephrase type of the constituent,in this case NP(orNP-SBJ).This is simply
the name of the parse node which dominates this constituent i n the parse tree.
Somesemanticrolestendto appearas NPs, othersas SorPP,andsoon.
•Thehead word of the constituent, Examiner . The head word of a constituent
can be computed using standard head rules, such as those give n in Ch. 12 in
Fig. 12.13. Certain head words (e.g. pronouns) place strong constraints on the
possiblesemanticrolestheyarelikelytoﬁll.

•Thehead wordpart-of-speech oftheconstituent, NNP.
•Thepathin the parse tree from the constituent to the predicate. This path is
markedby the dotted line in Fig. 20.16. Following(Gildea an d Jurafsky,2000),
we can use a simple linear representationof the path, NP ↑S↓VP↓VBD.↑and↓
representupward and downwardmovementin the tree respecti vely. The path is

DRAFTSection20.9. SemanticRole Labeling 685
very useful as a compact representationof many kinds of gram matical function
relationshipsbetweentheconstituentandthepredicate.
•Thevoiceof the clause in which the constituent appears, in this case active
(as contrasted with passive). Passive sentences tend to have strongly different
linkingsofsemanticrolestosurfaceformthanactiveones.
•Thebinary linearposition oftheconstituentwithrespecttothepredicate,either
beforeorafter.
•Thesub-categorization of the predicate. Recall from Ch. 12 that the subcat-
egorization of a verb is the set of expected arguments that ap pear in the verb
phrase. We can extract this information by using the phrase s tructure rule that
expandstheimmediateparentofthepredicate;VP →NPPPforthepredicatein
Fig.20.16.
Manyotherfeaturesaregenerallyextractedbysemanticrol elabelingsystems,such
as named entity tags (it is useful to know if a constituent is a LOCATION orPERSON,
forexample),ormorecomplexversionsofthepathfeatures( theupwardordownward

halves,whetherparticularnodesoccurin the path),the rig htmostor leftmost wordsof
theconstituent,andsoon.
We now have a set of observationslike the following example, each with a vector
of features; we have shown the features in the order describe d above (recall that most
observationswillhavethevalueNONEratherthane.g., ARG0,sincemostconstituents
inthe parsetree will notbeara semanticrole):
ARG0: [issued, NP,Examiner, NNP,NP ↑S↓VP↓VBD,active,before, VP →NPPP]
Just as we saw for word sense disambiguation, we can divide th ese observations

into a training and a test set, use the training examples in an y supervised machine
learningalgorithm,andbuildaclassiﬁer. SVMandMaximumE ntropyclassiﬁershave
yielded good results on this task on standard evaluations. O nce trained, the classi-
ﬁer can be used on unlabeled sentences to propose a role for ea ch constituent in the
sentence. More precisely, an input sentence is parsed and a p rocedure similar to that
describedearlierfortrainingisemployed.
Insteadoftrainingasinglestageclassiﬁer,somerolelabe lingalgorithmsdoclassi-

ﬁcationinmultiplestagesforefﬁciency:
•Pruning: to speed up execution,some constituentsare eliminated fro m consid-
erationaspossibleroles,basedonsimplerules
•Identiﬁcation: a binaryclassiﬁcationof eachnodeasan ARGto belabeledora
NONE.
•Classiﬁcation: aone-of-Nclassiﬁcationofalltheconstituentsthatwerel abeled
asARGbythepreviousstage.
Thereareanumberofcomplicationsthatallsemanticrolela belingsystemsneedto
dealwith. ConstituentsinFrameNetandPropBankarerequir edtobenon-overlapping.
Thusifasystemincorrectlylabelstwooverlappingconstit uentsasarguments,itneeds

NONE.
to decide which of the two is correct. Additionally, the sema ntic roles of constituents
are not independent; since PropBank does not allow multiple identical arguments, la-
beling one constituent as an ARG0 would greatly increase the probability of another

DRAFT686 Chapter 20. ComputationalLexicalSemantics
constituent being labeled ARG1. Both these problems can be addressed by the two-
stage approaches based on lattice or N-best rescoring discussed in Ch. 9: having the
classiﬁer assign multiple labels to each constituent, each with a probability,and using
a secondglobaloptimizationpassto pickthebestlabelsequ ence.
Instead of using parses as input, it is also possible to do sem antic role labeling
directlyfromraw (or part-of-speechtagged)text by applyi ngthe chunkingtechniques
used for named entity extraction or partial parsing. Such te chniques are particularly
useful in domains such as bioinformatics where it is unlikel y that syntactic parsers
trainedontypicalnewswiretextwill performwell.
Finally,semanticrole labelingsystems havebeengenerall yevaluatedbyrequiring
that each argument label must be assigned to the exactly corr ect word sequence or
parse constituent. Precision, recall, and F-measure can th en be computed. A simple

rule-based system can be used as a baseline, for example tagg ing the ﬁrst NP before
thepredicateas ARG0andtheﬁrstNPafterthepredicateas ARG1,andswitchingthese
ifthe verbphraseispassive.
20.10 Advanced: UnsupervisedSenseDisambiguation
Let’s brieﬂy return to the WSD task. It is expensive and difﬁc ult to build large cor-
pora in which each word is labeled for its word sense. For this reason, unsupervised
approachesto sensedisambiguationareanexcitingandimpo rtantresearcharea.
Inunsupervisedapproaches,wedon’tusehuman-deﬁnedword senses. Instead,the

set of‘senses’ ofeachwordare createdautomaticallyfromt heinstancesofeachword
in the training set. Let’s introduce a simpliﬁed version of t he methods of Sch¨ utze’s
(Sch¨ utze, 1992b, 1998) on unsupervised sense disambiguat ion. In Sch¨ utze’s method,
we ﬁrst represent each instance of a word in the training set b y distributional con-
text feature-vectorsthat are a slight generalizationof th e featurevectorswe deﬁnedin
Sec. 20.7. (It is for this reason that we turned to unsupervis ed sense disambiguation
onlyafterintroducingwordsimilarity.)

As in Sec. 20.7 we will represent a word was a vector based on frequenciesof its
neighboringwords. Forexampleforagiventargetword(type )w,wemightselect1000
words that occur most frequently within 25 words of any insta nce ofw. These 1000
wordsbecomethe dimensionof the vector. Let’s deﬁne fito mean the frequencywith
whichword ioccursinthecontextofword w. Wedeﬁnethewordvector ⃗w(foragiven
token(observation)of w)as:
⃗w= (f1,f2,f3,···,f1000)
So far this is just a version of the distributional context we saw in Sec. 20.7. We

canalsouseaslightlymorecomplexversionofthedistribut ionalcontext. Forexample
Schuetze deﬁnes the context vector of a word wnot as this ﬁrst-order vector, but
insteadbyits second orderco-occurrence . Thatis, thecontextvectorfora word wis
builtbytakingeachword xinthecontextof w,foreach xcomputingitswordvector ⃗x,
andthentakingthecentroid(average)ofthevectors ⃗x.

DRAFTSection20.10. Advanced: UnsupervisedSense Disambiguati on 687
Let’sseehowweusethesecontextvectors(whetherﬁrst-ord erorsecond-order)in
unsupervisedsensedisambiguationofa word w. Intraining,we’llneedonly3steps:
1. Foreachtoken wiofwordwina corpus,computea contextvector ⃗c.
2. Use a clustering algorithm toclusterthese wordtokencontextvectors ⃗cintoa
predeﬁnednumberofgroupsorclusters. Eachclusterdeﬁnes asense of w.
3. Computethe vector centroid of each cluster. Each vectorcentroid ⃗sjis asense
vectorrepresentingthatsense of w.
Since this is an unsupervised algorithm we won’t have names f or each of these
‘senses’of w; we just refertothe jthsense of w.
Nowhowdowedisambiguateaparticulartoken tofw? Againwehavethreesteps:
1. Computeacontextvector ⃗cfortasdiscussedabove.
2. Retrieveall sense vectors sjforw.
3. Assign tto thesenserepresentedbythesensevector sjthat isclosest to t.
Allweneedisaclusteringalgorithm,andadistancemetrics betweenvectors. For-

tunately, clustering is a well-studied problem with a wide n umber of standard algo-
rithms that can be applied to inputs structured as vectors of numerical values (Duda
and Hart, 1973). A frequently used technique in language app lications is known as
agglomerative clustering . In this technique, each of the Ntraining instances is ini-agglomerative
clustering
tially assignedto itsown cluster. New clustersare thenfor medina bottom-upfashion
by successivelymergingthe two clusters that are most simil ar. Thisprocesscontinues

untileitheraspeciﬁednumberofclustersisreached,orsom eglobalgoodnessmeasure
amongtheclustersisachieved. Incaseswherethenumberoft raininginstancesmakes
this method too expensive, random sampling can be used on the original training set
(Cuttingetal.,1992b)to achievesimilarresults.
How can we evaluate unsupervised sense disambiguation appr oaches? As usual,
the best way is to do extrinsic or in vivo evaluation, in which the WSD algorithm is
embeddedin some end-to-endsystem. Intrinsic evaluationc an also be useful, though,

if we have some way to map the automatically derived sense cla sses into some hand-
labeled gold standard set, so that we can compare a hand-labe led test set with a set
labeled by our unsupervisedclassiﬁer. One way of doing this mappingis to map each
sense cluster to a pre-deﬁned sense by choosing the sense tha t (in some training set)
hasthe most wordtokensoverlappingwith the cluster. Anoth eris to considerall pairs
ofwordsinthetest set,testingforeachwhetherboththesys temandthehand-labeling
putbothmembersofthepairin thesameclusterornot.

Word sense disambiguationtraces its roots to some of the ear liest applicationsof dig-
ital computers. We saw above Warren Weaver’s (1955) suggest ion to disambiguate
a word by looking at a small window around it, in the context of machine transla-
tion. Othernotionsﬁrst proposedinthisearlyperiodinclu detheuse ofathesaurusfor

DRAFT688 Chapter 20. ComputationalLexicalSemantics
disambiguation(Masterman,1957),supervisedtrainingof Bayesianmodelsfordisam-
biguation (Madhu and Lytel, 1965), and the use of clustering in word sense analysis
(SparckJones,1986).
An enormous amount of work on disambiguation has been conduc ted within the
context of early AI-orientednatural languageprocessing s ystems. While most natural
language analysis systems of this type exhibited some form o f lexical disambiguation
capability, a number of these efforts made word sense disamb iguation a larger focus
of their work. Among the most inﬂuential efforts were the eff orts of Quillian (1968)
and Simmons (1973) with semantic networks, the work of Wilks withPreference Se-
manticsWilks (1975c, 1975b, 1975a), and the work of Small and Rieger (1982) and
Riesbeck (1975)on word-basedunderstandingsystems. Hirs t’sABSITYsystem (Hirst
and Charniak, 1982; Hirst, 1987, 1988), which used a techniq ue based on semantic

networkscalled markerpassing, representsthe most advanc edsystem of this type. As
with these largely symbolic approaches,most connectionis t approachesto word sense
disambiguation have relied on small lexicons with hand-cod ed representations (Cot-
trell,1985;Kawamoto,1988).
Considerable work on sense disambiguation has been conduct ed in the areas of
Cognitive Science and psycholinguistics. Appropriately e nough, it is generally de-
scribedusingadifferentname: lexicalambiguityresoluti on. Smalletal.(1988)present

a varietyofpapersfromthisperspective.
The earliest implementation of a robust empirical approach to sense disambigua-
tion is due to Kelly and Stone (1975) who directed a team that h and-crafted a set of
disambiguationrules for 1790 ambiguousEnglish words. Les k (1986)was the ﬁrst to
use a machine readable dictionaryfor word sense disambigua tion. Wilks et al. (1996)
describeextensiveexplorationsofthe use ofmachinereada bledictionaries. Theprob-
lem of dictionarysenses being too ﬁne-grainedor lacking an appropriateorganization

has been addressed with models of clustering word senses Dol an (1994), Peters et al.
(1998), Chen and Chang (1998), Mihalcea and Moldovan (2001) , Agirre and de La-
calle (2003), Chklovski and Mihalcea (2003),Palmer et al. ( 2004),McCarthy (2006),
Navigli (2006), Snow et al. (2007); corpora with clustered w ord senses for training
clusteringalgorithmsincludePalmeret al.(2006)and OntoNotes (Hovyet al.,2006). OntoNotes
Moderninterest in supervised machine learningapproaches to disambiguationbe-

gan with Black (1988), who applied decision tree learning to the task. The need for
large amounts of annotated text in these methods led to inves tigations into the use of
bootstrappingmethods(Hearst,1991;Yarowsky,1995). The problemofhowtoweigh
and combine disparate sources of evidence is explored in Ng a nd Lee (1996), McRoy
(1992),andStevensonandWilks(2001).
Among the semi-supervised methods, more recent models of se lectional prefer-
enceincludeLiandAbe(1998),CiaramitaandJohnson(2000) ,McCarthyandCarroll

(2003), Light and Greiff (2002). Diab and Resnik (2002) give a semi-supervised al-
gorithm for sense disambiguationbased on aligned parallel corporain two languages.
For example,the fact that the French word catastrophe might be translated as English
disasterin one instance and tragedyin another instance can be used to disambiguate
the senses of the two English words (i.e. to choose senses of disasterandtragedy
that are similar). Abney (2002, 2004) explores the mathemat ical foundations of the
Yarowsky algorithm and its relation to co-training. The mos t-frequent-senseheuristic

DRAFTSection20.10. Advanced: UnsupervisedSense Disambiguati on 689
is an extremely powerful one, but requires large amounts of s upervised training data.
McCarthy et al. (2004) propose an unsupervised way to automa tically estimate the
mostfrequentsense,basedonthethesaurussimilaritymetr icsdeﬁnedin Sec.20.6.
Theearliestattempttouseclusteringinthestudyofwordse nsesisduetoSparckJones
(1986). Zernik (1991)successfully applied a standard info rmationretrieval clustering
algorithm to the problem, and provided an evaluation based o n improvements in re-
trieval performance. More extensiverecent work on cluster ing can be foundin Peder-
senandBruce(1997)andSch¨ utze(1997,1998).
A few algorithmshave attemptedto exploitthe power of mutua llydisambiguating
all the words in a sentence, either by multiple passes (Kelly and Stone, 1975) to take
advantage of easily disambiguated words, or by parallel sea rch (Cowie et al., 1992;
VeronisandIde,1990).

Recent work has focused on ways to use the web for training dat a for word sense
disambiguation,either unsupervised (Mihalcea and Moldov an,1999) or by using vol-
unteerstolabeldata(ChklovskiandMihalcea,2002).
Resnik(2006)describespotentialapplicationsofWSD.One recentapplicationhas
beentoimprovemachinetranslationChanet al.(2007),Carp uatandWu (2007).
AgirreandEdmonds(2006)isacomprehensiveeditedvolumet hatsummarizesthe
state of the art in WSD. Ide and Veronis (1998a) provide a comp rehensive review of

the history of word sense disambiguation up to 1998. Ng and Ze lle (1997) provide a
morefocusedreviewfromamachinelearningperspective. Wi lksetal.(1996)describe
dictionaryandcorpusexperiments,alongwithdetaileddes criptionsofveryearlywork.
The models of distributional word similarity we discussed a rose out of research
in linguistics and psychology of the 1950’s. The idea that me aning was related to
distributionofwordsincontextwaswidespreadinlinguist ictheoryofthe1950’s;even

before the well-known Firth (1957) and Harris (1968) dictum s discussed earlier, Joos
(1950)stated that
the linguist’s ‘meaning’ of a morpheme...is by deﬁnition th e set of conditional
probabilities of itsoccurrence incontext withall other mo rphemes’
The related idea that the meaning of a word could be modeled as a point in a Eu-
clideanspace,andthatthesimilarityofmeaningbetweentw owordscouldbemodeled
as the distance between these points, was proposed in psycho logy by Osgood et al.

(1957). The application of these ideas in a computational fr amework was ﬁrst made
by Sparck Jones (1986), and became a core principle of inform ation retrieval, from
whenceit cameintobroaderuse inspeechandlanguageproces sing.
Thereare a wide varietyofotherweightingsandmethodsforw ordsimilarity. The
largest class of methods not discussed in this chapter are th e variants to and details of
theinformation-theoretic methods like Jensen-Shannon divergence, KL-divergence

andα-skew divergence that we brieﬂy introduced (Pereira et al., 1993; Dagan et al.,
1994,1999;Lee,1999,2001);therearealsoothermetricsfr omHindle(1990)andLin
(1998a). Alternative paradigms include the co-occurrence retrieval model (Weeds,
2003;WeedsandWeir,2005). ManningandSch¨ utze(1999,Cha pter5and8)givecol-
locationmeasuresandother relatedsimilarity measures. A commonlyused weighting
isweighted mutual information (Fung and McKeown,1997)in which the pointwiseweightedmutual

mutual information is weighted by the joint probability. In information retrieval the
TF/IDFweight is widely used, as we will see in Ch. 23. See Dagan (2000 ), Mo-

DRAFT690 Chapter 20. ComputationalLexicalSemantics
hammad and Hirst (2005), Curran (2003) and Weeds (2003) for g ood summaries of
distributionalsimilarity.
An alternative vector space model of semantic similarity, Latent Semantic In-
dexing(LSI) orLatent Semantic Analysis (LSA), usessingular value decomposi-Latent SemanticIndexing
LSAtionto reduce the dimensionality of the vector space with the int ent of discovering
higher-orderregularities(Deerwesteret al.,1990). We ha vealreadydiscussedSch¨ utze
(1992b),anothersemanticsimilaritymodelbasedonsingul arvaluedecomposition.
There is a wide variety of recent literature on other lexical relations and thesaurus
induction. The use of distributional word similarity for th esaurus induction was ex-
plored systematically by Grefenstette (1994). A wide varie ty of distributional cluster-
ing algorithmshave been applied to the task of discoveringg roupingsof semantically
similar words, including hard clustering (Brown et al., 199 2), soft clustering (Pereira

etal.,1993),aswellasnewalgorithmslike ClusteringByCommittee (CBC)(Linand
Pantel, 2002). For particular relations, Lin et al. (2003) a pplied hand-craftedpatterns
to ﬁndantonyms , with the goal of improving synonym-detection. The distrib utional
word similarity algorithms from Sec. 20.7 often incorrectl y assign high similarity to
antonyms. Linetal.(2003)showedthatwordsappearinginth epatterns fromXtoY or
either X or Y tended to be antonyms. Girju et al. (2003,2006) show improve mentsin

meronym extractionbylearninggeneralizationsaboutthesemantic superclassesofthe
twonouns. ChklovskiandPantel(2004)usedhand-builtpatt ernstoextractﬁne-grained
relationsbetweenverbssuchas strength. Muchrecentworkhasfocusedonthesaurus
inductionby combiningdifferentrelationextractors. Pan tel andRavichandran(2004),
forexample,extendCaraballo’salgorithmforcombiningsi milarity andhyponymyin-
formation, while Snow et al. (2006) integrate multiple rela tion extractors to compute

themostprobablethesaurusstructure. Recentworkonsimil arityfocusesontheuseof
the Web, for example relying on Wikipedia Strube and Ponzett o (2006), Gabrilovich
and Markovitch (2007); this Web-based work is also closely r elated to unsupervised
informationextraction;see Ch. 22andreferenceslikeEtzi oniet al.(2005).
While not as old a ﬁeld as word similarity or sense disambigua tion, semantic role
labelinghasa longhistoryin computationallinguistics. T heearliestworkonsemantic

role labeling(Simmons, 1973)ﬁrst parsed a sentence usinga n ATN parser. Each verb
then had a set of rules specifying how the parse should be mapp ed to semantic roles.
These rulesmainlymade referenceto grammaticalfunctions (subject, object,comple-
ment of speciﬁc prepositions), but also checked constituen t-internal features such as
theanimacyofheadnouns.
Statistical work in the area revived in 2000 after the FrameN et and PropBank
projecthadcreateddatabaseslargeenoughandconsistente noughtomaketrainingand

testingpossible. Manypopularfeaturesusedforrolelabel ingaredeﬁnedinGildeaand
Jurafsky (2002), Chen and Rambow (2003), Surdeanu et al. (20 03), Xue and Palmer
(2004),Pradhanetal. (2003,2005).
To avoid the needfor hugelabeledtrainingsets, recentwork hasfocusedon unsu-
pervisedapproachesforsemanticrolelabeling(SwierandS tevenson,2004).
The semantic labeling work described above focuses on label ing each sentence
tokeninacorpuswithsemanticroles. Analternativeapproa chtosemanticrolelabeling

focusesonlexiconlearning,usingunsupervisedlearningo nacorpustolearnthekinds
of semantic classes a verb can belong to in terms of its possib le semantic roles or

DRAFTSection20.10. Advanced: UnsupervisedSense Disambiguati on 691
argument alternation patterns (Stevenson and Merlo, 1999; Schulte im Walde, 2000;
MerloandStevenson,2001;Merloet al.,2001;GrenagerandM anning,2006).
Exercises
20.1Collect a small corpusof example sentences of varying lengt hsfrom any news-
paper or magazine. Using WordNet, or any standard dictionar y, determine how
many senses there are for each of the open-class words in each sentence. How
manydistinctcombinationsofsensesarethereforeachsent ence? Howdoesthis
numberseemto varywithsentencelength?
20.2Using WordNet, or a standard reference dictionary, tag each open-class word
in your corpus with its correct tag. Was choosing the correct sense always a
straightforwardtask. Reportonanydifﬁcultiesyouencoun tered.
20.3Using the same corpus, isolate the words taking part in all th e verb-subjectand
verb-object relations. How often does it appear to be the cas e that the words

taking part in these relations could be disambiguated using only information
aboutthewordsintherelation?
20.4Between the words eatandﬁndwhichwould youexpectto be moreeffectivein
selectionalrestriction-basedsense disambiguation? Why ?
20.5Using your favorite dictionary, simulate the Original Lesk word overlap disam-
biguationalgorithmdescribedonpage659onthephrase Timeﬂieslikeanarrow .
Assume that the wordsare to be disambiguatedone at a time, fr om left to right,
andthatthe resultsfromearlierdecisionsareusedlaterin theprocess.

20.6Buildanimplementationofyoursolutiontothe previousexe rcise. UsingWord-
Net, implement the Original Lesk word overlap disambiguati on algorithm de-
scribedonpage659onthe phrase Time ﬂieslike anarrow .
20.7Implementandexperimentwithadecision-listsensedisamb iguationsystem. As
a model, use the kinds of features shown in Fig. 20.2. Use one o f the publicly
available decision-list packages like WEKA (or see Russell and Norvig (1995)
for more details on implementing decision-list learning yo urself). To facilitate

evaluation of your system, you should obtain one of the freel y available sense-
taggedcorpora.
20.8Evaluatetwoorthreeofthesimilaritymethodsfromthepubl iclyavailable Wordnet::Similarity
package(Pedersenet al., 2004). Youmight dothis byhand-la belingsomeword
pairswithsimilarityscoresandseeinghowwellthealgorit hmsapproximateyour
handlabels.
20.9Implementadistributionalwordsimilarityalgorithmthat cantakedifferentmea-
sures of association and different measures of vector simil arity. Now evalu-
ate two measures of association and two measures of vector si milarity from

Fig. 20.13. Again, you might do this by hand-labeling some wo rd pairs with

DRAFT692 Chapter 20. ComputationalLexicalSemantics
similarity scoresand seeing how well the algorithmsapprox imateyourhand la-
bels.

Chapter21
Computational Discourse
Gracie: Oh yeah...and then Mr. and Mrs. Joneswere having mat rimonial
trouble,andmy brotherwashiredtowatchMrs. Jones.
George: Well, I imagineshe wasaveryattractivewoman.
Gracie: Shewas,andmybrotherwatchedherdayandnightfors ixmonths.
George: Well, what happened?
Gracie: Sheﬁnallygotadivorce.
George: Mrs. Jones?
Gracie: No, mybrother’swife.
GeorgeBurnsandGracieAllenin TheSalesgirl
Orson Welles’ movie Citizen Kane was groundbreakingin many ways, perhaps most
notablyin itsstructure. Thestoryofthelife ofﬁctionalme diamagnateCharlesFoster

Kane,the moviedoesnot proceedin chronologicalorderthro ughKane’slife. Instead,
the ﬁlm begins with Kane’s death, (famously murmuring “Rosebud” ), and is struc-
turedaroundﬂashbackstohislifeinsertedamongscenesofa reporterinvestigatinghis
death. The novel idea that the structure of a movie does not ha ve to linearly follow
the structure of the real timeline made apparent for 20th cen tury cinematography the
inﬁnitepossibilitiesandimpactofdifferentkindsofcohe rentnarrativestructures.

But coherent structure is not just a fact about movies, or wor ks of art. Up to this
point of the book, we have focused primarily on language phen omena that operate at
thewordorsentencelevel. Butjustlikemovies,languagedo esnotnormallyconsistof
isolated,unrelatedsentences,but insteadofcollocated, structured, coherent groupsof
sentences. We referto suchacoherentstructuredgroupofse ntencesasa discourse . Discourse
The chapter you are now reading is an example of a discourse. I t is in fact a dis-

course of a particular sort: a monologue . Monologuesare characterized by a speaker Monologue
(a term which will be used to include writers, as it is here), a nd ahearer(which,
analogously, includes readers). The communication ﬂows in only one direction in a
monologue,thatis, fromthe speakertothe hearer.
After reading this chapter, you may have a conversation with a friend about it,
whichwouldconsistofamuchfreerinterchange. Suchadisco urseiscalleda dialogue, Dialogue

speciﬁcallya human-humandialogue . Inthiscase,eachparticipantperiodicallytakes
turnsbeingaspeakerandhearer. Unlikeatypicalmonologue ,dialoguesgenerallycon-
sist of many different types of communicativeacts: asking q uestions, giving answers,
makingcorrections,andsoforth.
You may also, for some purposes, such as booking an airline or train trip, have a
conversation with a computer conversational agent . This use of human-computer
dialogue forhuman-computer interaction , orHCIhas properties that distinguish it HCI

from normal human-humandialogue, in part due to the present -daylimitations on the
abilityofcomputersystemstoparticipatein free,unconst rainedconversation.
While many discourse processing problems are common to thes e three forms of

DRAFT694 Chapter 21. ComputationalDiscourse
discourse,theydifferinenoughrespectsthatdifferentte chniqueshaveoftenbeenused
toprocessthem. Thischapterfocusesontechniquescommonl yappliedtotheinterpre-
tationofmonologues;techniquesforconversationalagent sandotherdialogueswillbe
describedinCh. 24.
Languageis rife with phenomenathat operate at the discours e level. Consider the
discourseshowninexample(21.1).
(21.1)TheTinWoodmanwenttotheEmeraldCity to seethe WizardofOz andask
fora heart. Afterheaskedforit, theWoodmanwaitedfortheW izard’s
response.
What do pronouns such as heanditdenote? No doubt the reader had little trouble
ﬁguring out that hedenotes the Tin Woodman and not the Wizard of Oz, and that it
denotes the heart and not the Emerald City. Furthermore, it i s clear to the reader that
the Wizard is the same entity as the Wizard of Oz , andthe Woodman isthe same as the
Tin Woodman .
But doingthis disambiguationautomaticallyis a difﬁcult t ask. This goal of decid-

response.
ing what pronouns and other noun phrases refer to is called coreference resolution .
Coreferenceresolutionis importantfor informationextraction ,summarization ,and
forconversationalagents . Infact,itturnsoutthatjustaboutanyconceivablelangua ge
processing application requires methods for determining t he denotations of pronouns
andrelatedexpressions.
Thereareotherimportantdiscoursestructuresbesidether elationshipsbetweenpro-
nounsandothernouns. Considerthetaskof summarizing thefollowingnewspassage:

(21.2)First UnionCorpiscontinuingtowrestle withsevereproble ms. Accordingto
industryinsidersatPaineWebber,theirpresident,JohnR. Georgius,is
planningtoannouncehisretirementtomorrow.
We mightwantto extracta summarylike thefollowing:
(21.3)First UnionPresidentJohnR. Georgiusisplanningto announ cehisretirement
tomorrow.
Inorderto build such a summary,we need to knowthat the secon dsentence is the
more important of the two, and that the ﬁrst sentence is subor dinate to it, just giving

tomorrow.
background information. Relationships of this sort betwee n sentences in a discourse
arecalled coherencerelations ,anddeterminingthecoherencestructuresbetweendis-
coursesentencesisan importantdiscoursetask.
Sincecoherence is also a property of a good text, automatically detecting co her-
ence relations is also useful for tasks that measure text qua lity, likeautomatic essay
grading. Inautomaticessaygrading,shortstudentessaysareassig nedagradebymea-

suring the internal coherence of the essay as well as compari ng its content to source
material and hand-labeledhigh-quality essays. Coherence is also used to evaluate the
outputqualityofnaturallanguagegenerationsystems.
Discourse structure and coreferenceare related in deep way s. Notice that in order
to perform the summary above, a system must correctly identi fyFirst Union Corp as
the denotation of their(as opposed to Paine Webber , for instance). Similarly, it turns
outthatdeterminingthediscoursestructurecanhelpindet erminingcoreference.

DRAFT695
Coherence
Let’sconcludethisintroductionbydiscussingwhat it mean sfora text to be coherent.
Assume that you have collected an arbitrary set of well-form ed and independentlyin-
terpretable utterances, for instance, by randomly selecti ng one sentence from each of
thepreviouschaptersofthisbook. Doyouhaveadiscourse? A lmostcertainlynot. The
reasonisthattheseutterances,whenjuxtaposed,willnote xhibitcoherence . Consider, Coherence
forexample,thedifferencebetweenpassages(21.4)and(21 .5).
(21.4)JohnhidBill’scarkeys. He wasdrunk.
(21.5)?? JohnhidBill’s carkeys. He likesspinach.
While most people ﬁnd passage (21.4) to be rather unremarkab le, they ﬁnd passage
(21.5) to be odd. Why is this so? Like passage (21.4), the sent ences that make up
passage (21.5) are well formed and readily interpretable. S omething instead seems to
be wrong with the fact that the sentences are juxtaposed. The hearer might ask, for
instance,whathidingsomeone’scarkeyshastodowithlikin gspinach. Byaskingthis,

thehearerisquestioningthecoherenceofthepassage.
Alternatively, the hearer might try to construct an explana tion that makes it co-
herent, for instance, by conjecturing that perhaps someone offered John spinach in
exchange for hiding Bill’s car keys. In fact, if we consider a context in which we had
knownthis already,the passage now soundsa lot better! Why i s this? Thisconjecture
allows the hearer to identify John’s liking spinach as the ca use of his hiding Bill’s car

keys, which would explain how the two sentences are connecte d. The very fact that
hearerstrytoidentifysuchconnectionsisindicativeofth eneedtoestablishcoherence
aspartofdiscoursecomprehension.
Inpassage(21.4),orinournewmodelofpassage(21.5),thes econdsentenceoffers
the reader an EXPLANATION orCAUSEfor the ﬁrst sentence. These examples show
that a coherent discourse must have meaningful connections between its utterances,
connectionslike EXPLANATION that are often called coherence relations and will beCoherencerelation
introducedinSec. 21.2.

Let’sintroduceasecondaspectofcoherencebyconsidering thefollowingtwotexts
fromGroszet al.(1995a):
(21.6)a. Johnwentto hisfavoritemusicstoreto buya piano.
b. Hehadfrequentedthestoreformanyyears.
c. Hewasexcitedthat hecouldﬁnallybuyapiano.
d. He arrivedjust asthestorewasclosingforthe day.
(21.7)a. Johnwentto hisfavoritemusicstoreto buya piano.
b. Itwasa storeJohnhadfrequentedformanyyears.
d. It wasclosingjust asJohnarrived.
While these two texts differ only in how the two entities (Joh n and the store) are

realized in the sentences, the discourse in (21.6) is intuit ively more coherent than the
one in (21.7). AsGrosz et al. (1995a)pointout, this is becau sethe discoursein (21.6)
isclearlyaboutoneindividual,John,describinghisactio nsandfeelings. Thediscourse

DRAFT696 Chapter 21. ComputationalDiscourse
in (21.7), by contrast, focuses ﬁrst on John, then the store, then back to John, then to
thestoreagain. It lacksthe‘aboutness’oftheﬁrst discour se.
These examples show that for a discourse to be coherent it mus t exhibit certain
kindsofrelationshipswiththeentitiesitisabout,introd ucingthemandfollowingthem
in a focused way. This kind of coherence can be called entity-based coherence , We
will introducethe Centering modelofentity-basedcoherencein Sec.21.6.2.
In the rest of the chapter we’ll study aspects of both discour se structure and dis-
course entities. We begin in Sec. 21.1 with the simplest kind of discourse structure:
simplediscourse segmentation of a document into a linear sequence of multipara-
graphpassages. In Section 21.2, we then introducemore ﬁne- graineddiscourse struc-
ture,thecoherencerelation ,andgivesomealgorithmsforinterpretingtheserelations .
Finally, in Section 21.3, we turn to entities, describing me thods for interpreting refer-

ringexpressions suchaspronouns.
21.1 DiscourseSegmentation
The ﬁrst kind of discoursetask we examineis an approximatio nto the global or high-
levelstructureofatextordiscourse. Manygenresoftextar eassociatedwithparticular
conventional structures. Academic articles might be divid ed into sections like Ab-
stract, Introduction, Methodology, Results, Conclusion. A newspaper story is often
described as having an inverted pyramid structure, in which the opening paragraphs

(thelede)containsthemostimportantinformation. Spokenpatientr eportsaredictated Lede
bydoctorsinfoursectionsfollowingthestandardSOAPform at(Subjective,Objective,
Assessment, Plan).
Automatically determining all of these types of structures for a large discourse
is a difﬁcult and unsolved problem. But some kinds of discour se structure detec-
tion algorithms exist. This section introduces one such alg orithm, for the simpler
problem of discourse segmentation ; separating a document into a linear sequence ofDiscoursesegmentation

subtopics. Such segmentation algorithms are unable to ﬁnd s ophisticated hierarchical
structure. Nonetheless, linear discourse segmentation ca n be important for informa-
tion retrieval , for example, for automatically segmenting a TV news broadc ast or a
long news story into a sequence of stories so as to ﬁnd a releva nt story, or for text
summarization algorithms which need to make sure that different segments o f the
document are summarized correctly, or for information extraction algorithmswhich
tendtoextractinformationfrominsidea singlediscourses egment.

Inthe nexttwo sectionswe introducebothan unsupervisedan d a supervisedalgo-
rithmfordiscoursesegmentation.
21.1.1 Unsupervised DiscourseSegmentation
Let’s consider the task of segmenting a text into multi-para graph units that represent
subtopics or passages of the original text. As we suggested a bove, this task is often
calledlinear segmentation , to distinguish it from the task of deriving more sophisti-Linearsegmentation
cated hierarchical discourse structure. The goal of a segme nter, given raw text, might

DRAFTSection21.1. DiscourseSegmentation 697
be to assign subtopic groupingssuch as the ones deﬁned by Hea rst (1997) for the fol-
lowing 21-paragraphscience newsarticle called Stargazers on the existence of life on
earthandotherplanets(numbersindicateparagraphs):
l-3 Intro- the searchfor lifeinspace
4–5 The moon’s chemical composition
6-8 How earlyearth-moon proximity shaped the moon
9–12 How the moon helped lifeevolve onearth
13 Improbability of the earth-moon system
14–16 Binary/trinarystar systems make lifeunlikely
17–18 The low probabilityof nonbinary/trinary systems
19–20 Properties of earth’s sunthat facilitatelife
21 Summary
An important class of unsupervised algorithms for the linea r discourse segmenta-
tiontaskrelyontheconceptof cohesion (HallidayandHasan,1976). Cohesion isthe Cohesion
use of certain linguistic devices to link or tie together tex tual units. Lexical cohesion Lexicalcohesion
is cohesion indicated by relations between words in the two u nits, such as use of an

21 Summary
identicalword,a synonym,or a hypernym. Forexamplethe fac t that the words house,
shingled,andIoccurin bothofthe two sentencesin (21.8ab),is a cuethat th e two are
tiedtogetherasa discourse:
(21.8)Beforewinter Ibuilta chimney,and shingled thesidesofmy house...
Ihavethusa tight shingled andplastered house
InEx.(21.9),lexicalcohesionbetweenthetwosentencesis indicatedbythehyper-
nymrelationbetween fruitandthe words pearsandapples.
(21.9)Peel, coreandslice thepearsandthe apples . Addthefruit to theskillet.

Therearealso non-lexicalcohesionrelations,such asthe u se ofanaphora ,shown
here between Woodhouses andthem(we will deﬁne and discuss anaphora in detail in
Sec.21.6):
(21.10)The Woodhouses wereﬁrst inconsequencethere. All lookedupto them.
In addition to single examples of lexical cohesion between t wo words, we can have a
cohesionchain ,in whichcohesionis indicatedbya wholesequenceofrelate dwords: Cohesionchain
(21.11)Peel,coreandslice the pearsand theapples . Addthe fruit tothe skillet.
Whentheyaresoft...

Coherence andcohesion areoftenconfused;let’sreviewthedifference. Cohesion
refers to the way textual units are tied or linked together. A cohesive relation is like
a kind of glue grouping together two units into a single unit. Coherence refers to
themeaning relation between the two units. A coherence relation explai ns how the
meaningofdifferenttextualunitscancombinetojointlybu ildadiscoursemeaningfor
thelargerunit.
The intuition of the cohesion-based approach to segmentati on is that sentences or

paragraphs in a subtopic are cohesive with each other, but no t with paragraphs in a
neighboring subtopic. Thus if we measured the cohesion betw een every neighboring
sentence,we mightexpecta‘dip’in cohesionatsubtopicbou ndaries.
Let’slookatonesuchcohesion-basedapproach,the TextTiling algorithm(Hearst, TextTiling

DRAFT698 Chapter 21. ComputationalDiscourse
1997). Thealgorithmhasthreesteps: tokenization ,lexicalscoredetermination ,and
boundary identiﬁcation . In the tokenization stage, each space-delimited word in th e
input is converted to lower-case, words in a stop list of func tion words are thrown
out, and the remaining words are morphologicallystemmed. T he stemmed words are
grouped into pseudo-sentences of length w=20 (equal-length pseudo-sentences are
usedratherthanrealsentences).
Now we look at each gap between pseudo-sentences,and comput e alexical cohe-
sion score across that gap. The cohesion score is deﬁned as the average s imilarity of
thewordsinthepseudo-sentencesbeforegaptothepseudo-s entencesafterthegap. We
generallyuse ablockof k=10pseudo-sentencesoneachside ofthegap. Tocompute
similarity,wecreateawordvector bfromtheblockbeforethegap,andavector afrom
theblockafterthegap,wherethevectorsareoflength N(thetotalnumberofnon-stop

words in the document)and the ith element of the word vector is the frequencyof the
wordwi. Now we can compute similarity by the cosine (= normalized do t product)
measuredeﬁnedinEq.20.47fromCh. 20,rewrittenhere:
simcosine(⃗b,⃗a) =⃗b·⃗a
|⃗b||⃗a|=∑N
i=1bi×ai√
∑N
i=1b2
i√
∑N
i=1a2
i(21.12)
This similarity score (measuring how similar pseudo-sente ncesi−ktoiare to
sentences i+1 toi+k+1) is computed for each gap ibetween pseudo-sentences.
Let’slookattheexampleinFig.21.1,where k=2. Fig.21.1ashowsaschematicview

i=1bi×ai√
∑N
i=1b2
i√
∑N
i=1a2
i(21.12)
of four pseudo-sentences. Each 20-word pseudo-sentence mi ght have multiple true
sentencesinit;we’veshowneachwithtwotruesentences. Th eﬁgurealsoindicatesthe
computationofthedot-productbetweensuccessivepseudos entences. Thusforexample
in the ﬁrst pseudo-sentence,consisting of sentences1 and2 , the word A occurstwice,
B once, C twice, and so on. The dot productbetween the ﬁrst two pseudosentencesis
2×1+1×1+2×1+1×1+2×1=8. What is the cosine between these ﬁrst two,
assumingall wordsnotshownhavezerocount?

Finally, we computea depth score for each gap, measuringthe depth of the ‘sim-
ilarity valley’ at the gap. The depth score is the distance fr om the peakson both sides
ofthevalleyto thevalley;InFig. 21.1(b),thiswouldbe (ya1−ya2)+(ya3−ya2).
Boundariesareassignedatanyvalleywhichisdeeperthanac utoffthreshold(such
as ¯s−σ,i.e. onestandarddeviationdeeperthanthemeanvalleydep th).
Instead of using these depth score thresholds, more recent c ohesion-based seg-

menters use divisive clustering (Choi, 2000; Choi et al., 2001); see the end of the
chapterformoreinformation.
21.1.2 Supervised DiscourseSegmentation
We’ve now seen a method for segmenting discourses when no han d-labeled segment
boundariesexist. For some kinds of discourse segmentation tasks, however, it is rela-
tivelyeasytoacquireboundary-labeledtrainingdata.
Consider the spoken discourse task of segmentation of broad cast news. In order
to do summarization of radio or TV broadcasts, we ﬁrst need to assign boundaries

between news stories. This is a simple discourse segmentati ontask, and training sets

DRAFTSection21.1. DiscourseSegmentation 699
A   A
B         
C   C
D   
    E    
     
         A
B         
C       
    D
E   E 
             
    B     
 
E   
F   F
G 
H   H       
    B     
 
F    
G   G
H   H
         I36  

a1ya3
ya2
a2
(a) (b)
Figure21.1 The TextTiling algorithm, showing (a) the dot-product comp utation of similarity
between two sentences (1 and 2) and 2 following sentences (3 a nd 4); capital letters (A, B, C,
etc) indicate occurrences of words. (b) shows the computati on of the depth score of a valley.
AfterHearst (1997).
with hand-labeled news story boundaries exist. Similarly, for speech recognition of
monologueslikelecturesorspeeches,weoftenwanttoautom aticallybreakthetextup
into paragraphs. For the task of paragraph segmentation , it is trivial to ﬁnd labeledParagraphsegmentation
trainingdatafromtheweb (markedwith <p>)orothersources.
Every kind of classiﬁer has been used for this kind of supervi sed discourse seg-

mentation. Forexample,wecanusea binaryclassiﬁer (SVM,d ecisiontree)andmake
a yes-no boundary decision between any two sentences. We can also use a sequence
classiﬁer (HMM,CRF), makingit easier toincorporateseque ntialconstraints.
The features in supervised segmentation are generally a sup erset of those used in
unsupervisedclassiﬁcation. We cancertainlyusecohesion featuressuchaswordover-
lap,wordcosine,LSA,lexicalchains,coreference,andso o n.
A keyadditionalfeaturethat is oftenused for supervisedse gmentationis the pres-

enceofdiscoursemarkers orcuewords . Adiscoursemarkerisawordorphrasethat Discourse marker
Cue word functionsto signal discourse structure. Discourse marker s will play an important role
throughout this chapter. For the purpose of broadcast news s egmentation, important
discourse markers might include a phrase like good evening, I’m ⟨PERSON⟩, which
tendstooccuratthebeginningofbroadcasts,ortheword joining,whichtendstooccur
inthephrase joiningusnowis⟨PERSON⟩,whichoftenoccursatbeginningsofspeciﬁc

segments. Similarly, the cue phrase coming up often appears at the end of segments
(Reynar,1999;Beefermanet al.,1999).
Discourse markers tend to be very domain-speciﬁc. For the ta sk of segmenting
newspaper articles from the Wall Street Journal, for exampl e, the word incorporated
is a useful feature, since Wall Street Journal articles ofte n start by introducinga com-
pany with the full name XYZ Incorporated , but later using just XYZ. For the task of

segmentingoutrealestateads,Manning(1998)useddiscour secuefeatureslike ‘isthe
followingwordaneighborhoodname?’ ,‘ispreviouswordaphonenumber?’ andeven

DRAFT700 Chapter 21. ComputationalDiscourse
punctuationcueslike ‘isthe followingwordcapitalized?’ .
Itispossibletowritehand-writtenrulesorregularexpres sionstoidentifydiscourse
markersforagivendomain. Suchrulesoftenreferto nameden tities(like the PERSON
examples above), and so a named entity tagger must be run as a p reprocessor. Auto-
matic methods for ﬁnding discourse markers for segmentatio n also exist. They ﬁrst
encode all possible words or phrases as features to a classiﬁ er, and then doing some
sort offeature selection on the training set to ﬁnd only the words that are the best
indicatorsofa boundary(Beefermanetal., 1999;Kawaharae tal., 2004).
21.1.3 Evaluating DiscourseSegmentation
Discourse segmentation is generally evaluated by running t he algorithm on a test set
in which boundarieshave been labeled by humans. The perform anceof the algorithm
is computed by comparing the automatic and human boundary la bels using the Win-
dowDiff(PevznerandHearst,2002)or Pk(Beefermanet al.,1999)metrics.

We generally don’t use precision, recall and F-measure for e valuating segmenta-
tion because they are not sensitive to near misses. Using sta ndard F-measure, if our
algorithm was off by one sentence in assigning each boundary , it would get as bad a
score as an algorithm which assigned boundaries nowhere nea r the correct locations.
BothWindowDiff andPkassign partial credit. We will present WindowDiff, since it is
a morerecentimprovementto Pk.

WindowDiffcomparesareference(humanlabeled)segmentat ionwithahypothesis
segmentation by sliding a probe, a moving window of length k, across the hypothesis
segmentation. At each position in the hypothesis string, we compare the number of
reference boundaries that fall within the probe ( ri) to the number of hypothesized
boundariesthat fall within the probe ( hi). The algorithm penalizes any hypothesisfor
whichri̸=hi, i.e. for which|ri−hi|̸=0. The windowsize kis set as half the average
segmentin thereferencestring. Fig.21.2showsa schematic ofthecomputation.
Ref
Hyp

Ref
Hyp
0110
Figure21.2 The WindowDiff algorithm, showing the moving window slidin g over the hy-
pothesis string, and the computation of |ri−hi|at four positions. After Pevzner and Hearst
(2002).
Moreformally,if b(i,j)isthenumberofboundariesbetweenpositions iandjina
text,andNisthenumberofsentencesinthe text:

DRAFTSection21.2. TextCoherence 701
WindowDiff (ref,hyp) =1
N−kN−k
∑
i=1(|b(refi,refi+k)−b(hypi,hypi+k)|̸=0) (21.13)
WindowDiffreturnsavaluebetween0and1,where0indicates thatall boundaries
areassignedcorrectly.
21.2 TextCoherence
Theprevioussectionshowedthat cohesivedevices,like lex icalrepetition,canbe used
to ﬁnd structure in a discourse. The existence of such device s alone, however, does
not satisfy a stronger requirement that a discourse must mee t, that of being coherent.
We brieﬂy introduced coherence in the introduction. In this section we offer more
details on what it means for a text to be coherent, and computa tional mechanisms for
determining coherence. We will focus on coherence relations and reserve entity-
based coherence fordiscussionin Sec.21.6.2.
Recallfromtheintroductionthe differencebetweenpassag es(21.14)and(21.15).
(21.14)JohnhidBill’s carkeys. Hewasdrunk.
(21.15)?? JohnhidBill’scarkeys. He likesspinach.
The reason (21.14) is more coherent is that the reader can for m a connection be-

tween the two utterances, in which the second utterance prov ides a potential CAUSE
orEXPLANATION for the ﬁrst utterance. This link is harder to form for (21.15 ). The
possible connectionsbetween utterancesin a discourse can be speciﬁed as a set of co-
herencerelations . Afewsuchrelations,proposedbyHobbs(1979a),aregivenb elow.Coherencerelation
Theterms S0andS1representthemeaningsofthetwo sentencesbeingrelated.
Result:Infer that the state or event asserted by S0causes or could cause the state or
eventassertedby S1.

(21.16)TheTin Woodmanwascaughtintherain. Hisjointsrusted.
Explanation: Inferthatthestateoreventassertedby S1causesorcouldcausethestate
oreventassertedby S0.
(21.17)JohnhidBill’s carkeys. Hewasdrunk.
Parallel: Inferp(a1,a2,...)fromtheassertionof S0andp(b1,b2,...)fromtheassertion
ofS1, whereaiandbiaresimilar,forall i.
(21.18)TheScarecrowwantedsomebrains. TheTin Woodmanwanteda he art.
Elaboration: Inferthesameproposition Pfromtheassertionsof S0andS1.
(21.19)DorothywasfromKansas. ShelivedinthemidstofthegreatKa nsasprairies.

DRAFT702 Chapter 21. ComputationalDiscourse
Occasion: Achangeofstatecanbeinferredfromtheassertionof S0,whoseﬁnalstate
can be inferred from S1, or a change of state can be inferred from the assertion of S1,
whoseinitialstate canbeinferredfrom S0.
(21.20)Dorothypickeduptheoil-can. SheoiledtheTinWoodman’sjo ints.
We can also talk about the coherence of an entire discourse, b y considering the
hierarchicalstructurebetweencoherencerelations. Cons iderpassage(21.21).
(21.21)Johnwent tothebankto deposithispaycheck. (S1)
HethentookatraintoBill’s cardealership. (S2)
Heneededto buya car. (S3)
Thecompanyheworksfornowisn’tnearanypublictransporta tion. (S4)
Healso wantedtotalkto Bill abouttheirsoftballleague. (S 5)
Intuitively, the structure of passage (21.21) is not linear . The discourse seems to be
primarily about the sequence of events described in sentenc es S1 and S2, whereas
sentences S3 and S5 are related most directly to S2, and S4 is r elated most directly

to S3. The coherence relationships between these sentences result in the discourse
structureshowninFig.21.3.
Occasion ( e1;e2)
S1(e1) Explanation ( e2)
S2(e2) Parallel( e3;e5)
Explanation ( e3) S5( e5)
S3(e3) S4( e4)
Figure21.3 The discourse structure of passage (21.21).
Each node in the tree represents a group of locally coherent c lauses or sentences,
called adiscourse segment . Roughly speaking, one can think of discourse segmentsDiscoursesegment
asbeinganalogoustoconstituentsin sentencesyntax.

Nowthatwe’veseenexamplesofcoherence,wecanseemorecle arlyhowacoher-
encerelationcanplayaroleinsummarizationorinformatio nextraction. Forexample,
discourses that are coherent by virtue of the Elaboration re lation are often character-
ized by a summary sentence followed by one or more sentences a dding detail to it, as
inpassage(21.19). Althoughtherearetwosentencesdescri bingeventsinthispassage,
the Elaboration relation tells us that the same event is bein g described in each. Au-

tomatic labeling of the Elaboration relation could thus tel l an information extraction
or summarization system to merge the information from the se ntences and produce a
singleeventdescriptioninsteadoftwo.
21.2.1 Rhetorical Structure Theory
Another theory of coherence relations that has received bro ad usage is Rhetorical
Structure Theory (RST), a model of text organization that was originally proposedRhetoricalStructureTheory
RST

DRAFTSection21.2. TextCoherence 703
forthestudyoftextgeneration(MannandThompson,1987).
RST is based on a set of 23 rhetorical relations that can hold between spans of
text within a discourse. Most relations hold between two tex t spans (often clauses or
sentences), a nucleusand asatellite. The nucleus is the unit that is more central to Nucleus
Satellite thewriter’spurpose,andthatisinterpretableindependen tly;thesatelliteislesscentral,
andgenerallyisonlyinterpretablewith respectto thenucl eus.
Considerthe Evidence relation,inwhichasatellitepresentsevidenceforthepro po- Evidence
sitionorsituationexpressedinthenucleus:
(21.22)Kevinmustbehere. Hiscarisparkedoutside.
RST relations are traditionally represented graphically; the asymmetric Nucleus-
Satellite relationisrepresentedwithanarrowfromthesat ellite tothe nucleus:
Kevin must be here. His car is parked outside
In the original (Mann and Thompson, 1987) formulation, an RS T relation is for-

mallydeﬁnedbyasetof constraints onthenucleusandsatellite,havingtodowiththe
goalsandbeliefsof the writer(W) and reader(R), andby the effecton the reader(R).
TheEvidencerelation,forexample,isdeﬁnedasfollows:
RelationName: Evidence
Constraintson N: RmightnotbelieveN toa degreesatisfactorytoW
Constraintson S: RbelievesS orwill ﬁndit credible
ConstraintsonN+S: R’s comprehendingS increasesR’sbeliefofN
Effects: R’s beliefofN isincreased
TherearemanydifferentsetsofrhetoricalrelationsinRST andrelatedtheoriesand

implementations. The RST TreeBank (Carlson et al., 2001), f or example, deﬁnes 78
distinctrelations,groupedinto16classes. Herearesomec ommonRSTrelations,with
deﬁnitionsadaptedfromCarlsonandMarcu(2001).
Elaboration: Therearevariouskindsofelaborationrelations; in eachon e, thesatel-
litegivesfurtherinformationaboutthecontentofthenucl eus:
[NThe company wouldn’t elaborate,] [ Scitingcompetitive reasons]
Attribution: The satellite gives the source of attribution for an instanc e of reported
speechin thenucleus.

[SAnalysts estimated,][ Nthat sales at U.S.stores declined inthe quarter, too]
Contrast: Thisisamultinuclearrelation,inwhichtwoormorenucleic ontrastalong
someimportantdimension:
[NThe priest was ina verybadtemper,] [ Nbut the lama was quite happy.]
List:In this multinuclear relation, a series of nuclei is given, w ithout contrast or
explicitcomparison:
[NBillyBones was the mate; ][ NLong John, he was quartermaster]

DRAFT704 Chapter 21. ComputationalDiscourse
Background: Thesatellite givescontextforinterpretingthenucleus:
[ST isthe pointer tothe root of a binary tree.] [ NInitializeT.]
Just as we saw for the Hobbs coherence relations, RST relatio ns can be hierarchi-
cally organizedintoan entirediscoursetree. Fig. 21.4sho wsonefromMarcu(2000a)
forthetextin(21.23)fromtheScientiﬁc Americanmagazine .
(21.23)Withitsdistantorbit–50percentfartherfromthesunthanE arth–andslim
atmosphericblanket,Marsexperiencesfrigidweathercond itions. Surface
temperaturestypicallyaverageabout-60degreesCelsius( -76degrees
Fahrenheit)atthe equatorandcandipto-123degreesCneart hepoles. Only
themiddaysunat tropicallatitudesiswarmenoughtothawic eonoccasion,but
anyliquidwater formedin thisway wouldevaporatealmost in stantlybecause
ofthe lowatmosphericpressure.itle
(1)
Mars2-9
evidence
2-3
background
  (2)
WIth its 
distant orbit  
<p> -- 50
 percent 
farther from 
the sun than 
Earth -- </p> 
and slim 
atmospheric 
blanket,(3)
Mars

(1)
Mars2-9
evidence
2-3
  (2)
WIth its 
<p> -- 50
 percent 
and slim 
Mars
frigid weather
conditions.4-9
elaboration-additional
(4)
Surface 
temperatures
 typically average 
about -60 
degrees Celsius
 <p> (-76 degrees
Fahrenheit)</p>
 at the equator4-5
List
(5)
and can dip
to -123
degrees C
near the
poles.6-9
Contrast
6-7
(6)
Only the
midday sun at
tropical latitudes
is warm enough(7)
to thaw ice
on occasion,purpose8-9
explanation-argumentative
(8)
but any liquid water
formed in this way 
would evaporate 
almost instantly(9)
because of
the low
atmospheric
pressure.

(4)
Surface 
about -60 
List
(5)
and can dip
to -123
degrees C
near the
poles.6-9
Contrast
6-7
(6)
Only the
to thaw ice
(8)
because of
the low
pressure.
Figure21.4 AdiscoursetreefortheScientiﬁcAmericantextin(21.23), fromMarcu(2000a). Notethatasymmet-
ric relations arerepresented witha curved arrow from the sa tellitetothe nucleus.
See the end of the chapterfor pointersto othertheoriesof co herencerelationsand
related corpora,andCh. 23for the applicationofRST and sim ilar coherencerelations
tosummarization.
21.2.2 AutomaticCoherence Assignment
Given a sequence of sentences, how can we automatically dete rmine the coherence

the low
pressure.
relationsbetweenthem? WhetherweuseRST,Hobbs,oroneoft hemanyothersetsof
relations(seetheendofthechapter),wecallthistask coherencerelationassignment .
If we extend this task from assigning a relation between two s entences to the larger
goal of extracting a tree or graph representingan entire dis course, the term discourse
parsingisoftenused. Discourse parsing
Bothofthesetasksarequitedifﬁcult,andremainunsolvedo penresearchproblems.
Nonetheless,avarietyofmethodshavebeenproposed,andin thissectionwe describe

shallow algorithms based on cue phrases . In the following section we sketch a more
sophisticatedbutless robustalgorithmbasedon abduction .

DRAFTSection21.2. TextCoherence 705
Ashallowcue-phrase-basedalgorithmforcoherenceextrac tionhasthreestages:
1. Identifythecuephrasesina text
2. Segmentthetextintodiscoursesegments,usingcuephras es
3. Classifytherelationshipbetweeneachconsecutivedisc oursesegment,usingcue
phrases.
We said earlier that a cue phrase (ordiscourse marker orcue word ) is a word Cue phrase
Discourse marker or phrase that functions to signal discourse structure, esp ecially by linking together
discoursesegments. In Sec. 21.1we mentionedcue phrasesor featureslike joiningus
nowis⟨PERSON⟩(forbroadcastnewssegmentation)or followingwordisthenameof
a neighborhood (for real estate ad segmentation). For extracting coherenc e relations,
we rely on cue phrases called connectives , which are often conjunctions or adverbs, Connective
and which give us a ‘cue’ to the coherence relations that hold between segments. For
example,the connective becausestrongly suggeststhe EXPLANATION relation in pas-
sage(21.24).

(21.24)JohnhidBill’s carkeysbecause hewasdrunk.
Other such cue phrases include although,but,for example ,yet,with, andand.
Discourse markers can be quite ambiguous between these discourse uses and non-
discourse related sentential uses. For example, the word withcan be used as a cue Sentential
phraseasin (21.25),orin asententialuse asin (21.26)1:
(21.25)Withitsdistantorbit,Marsexhibitsfrigidweatherconditions
(21.26)We cansee Mars withanordinarytelescope.

Somesimple disambiguationofthe discourseversussentent ial useof a cue phrase
can be done with simple regular expressions, once we have sen tence boundaries. For
example, if the words WithorYetare capitalized and sentence-initial, they tend to
be discourse markers. The words becauseorwheretend to be discourse markers if
preceded by a comma. More complete disambiguation requires the WSD techniques
of Ch. 20 using many other features. If speech is available, f or example, discourse

markersoftenbeardifferentkindsofpitchaccentthan sent entialuses(Hirschbergand
Litman,1993).
Thesecondstepindeterminingthecorrectcoherencerelati onistosegmentthetext
intodiscourse segments . Discourse segmentsgenerallycorrespondto clauses or sen -
tences, although sometimes they are smaller than clauses. M any algorithms approx-
imate segmentation by using entire sentences, employing th e sentence segmentation
algorithmofFig.3.22(page71),orthealgorithmofSec.8.1 .1.

Often, however, a clause or clause-like unit is a more approp riate size for a dis-
coursesegment,asweseeinthefollowingexamplesfromSpor lederandLapata(2004):
(21.27) [We can’twin] [but wemust keep trying] ( CONTRAST )
(21.28) [The abilitytooperate at these temperature isadvantageou s], [because the devices
needless thermal insulation] ( EXPLANATION )
One way to segment these clause-like units is to use hand-wri tten segmentation
rules based on individual cue phrases. For example, if the cu e-phraseBecauseoccurs

1Whereperhaps it will beacue instead for the semantic role INSTRUMENT

DRAFT706 Chapter 21. ComputationalDiscourse
sentence-initiallyand is eventuallyfollowedby a comma (a s in (21.29)),it may begin
a segment (terminated by the comma) that relates to the claus e after the comma. If
becauseoccurs sentence-medially, it may divide the sentence into a previousand fol-
lowing discourse segment (as in (21.30)). These cases can be distinguished by hand-
writtenrulesbasedonpunctuationandsentenceboundaries .
(21.29)[Because ofthe lowatmosphericpressure,][anyliquidwaterwould
evaporateinstantly]
(21.30)[Anyliquidwaterwouldevaporateinstantly][because ofthelow
atmosphericpressure.]
If a syntactic parser is available, we can write more complex segmentation rules
makinguseofsyntacticphrases.
The third step in coherence extraction is to automatically c lassify the relation be-
tween each pairof neighboringsegments. We can againwrite r ulesfor each discourse
marker, just as we did for determining discourse segment bou ndaries. Thus a rule

could specify that a segmenting beginning with sentence-in itialBecauseis a satellite
inaCAUSErelationshipwitha nucleussegmentthatfollowsthecomma.
In general, the rule-based approach to coherence extractio n does not achieve ex-
tremely high accuracy. Partly this is because cue phrasesar e ambiguous; because, for
example, can indicate both CAUSEandEVIDENCE ,butcan indicate CONTRAST ,AN-
TITHESIS , andCONCESSION , andso on. We needadditionalfeaturesthan just the cue
phrases themselves. But a deeper problem with the rule-base d method is that many

coherence relations are not signaled by cue phrases at all. I n the RST corpus of Carl-
son et al. (2001), for example, Marcu and Echihabi (2002) fou nd that only 61 of the
238CONTRAST relations,andonly79ofthe307 EXPLANATION -EVIDENCE relations,
wereindicatedbyexplicitcuephrases. Instead,manycoher encerelationsaresignalled
bymoreimplicitcues. Forexample,thefollowingtwosenten cesareinthe CONTRAST
relation, but there is no explicit in contrast orbutconnective beginning the second
sentence:

sentence:
(21.31)The$6billionthatsome40companiesarelookingto raisein t heyearending
March31compareswithonly$2.7billionraisedonthecapita lmarketinthe
previousﬁscal year
(21.32)Inﬁscal 1984beforeMr. Gandhicametopower,only$810milli onwas
raised.
How can we extract coherence relations between discourse se gments if no cue
phrases exist? There are certainly many implicit cues that w e could use. Consider
thefollowingtwodiscoursesegments:
(21.33)[Idon’twanta truck;][I’dprefera convertible.]

sentence:
raised.
TheCONTRAST relationbetweenthesesegmentsissignalledbytheirsynta cticpar-
allelism, by the use of negation in the ﬁrst segment, and by th e lexical coordinate
relation between convertible andtruck. But many of these features are quite lexical,
requiringa largenumberofparameterswhichcouldn’tbetra inedonthesmall amount
oflabeledcoherencerelationdatathatcurrentlyexists.
Thissuggeststhe use of bootstrapping to automaticallylabela largercorpuswith
coherencerelationsthat couldthenbe usedto train these mo reexpensivefeatures. We

DRAFTSection21.3. ReferenceResolution 707
can do this by relying on discourse markers that are very stro ng unambiguous cues
for particular relations. For example consequently is an unambiguous signal for RE-
SULT,inotherwords forSUMMARY ,forexample forELABORATION ,andsecondlyfor
CONTINUATION . We write regular expressions to extract pairs of discourse segments
surrounding these cue phrases, and then remove the cue phras es themselves. The re-
sultingsentencepairs,withoutthecuephrases,areusedas asupervisedtrainingsetfor
thesecoherencerelations.
Given this labeled training set, any supervised machine lea rning method may be
used. MarcuandEchihabi(2002),forexample,useanaiveBay esclassiﬁerbasedonly
on word-pair features (w1,w2), where the ﬁrst word w1occurs in the ﬁrst discourse
segment, and the second w2occurs in the following segment. This feature captures
lexicalrelationslike convertible /truckabove. SporlederandLascarides(2005)include

other features, including individual words, parts of speec h, or stemmed words in the
left and rightdiscourse segment. They found,forexample,t hat wordslike other,still,
andnotwere chosen by feature selection as goodcues for CONTRAST . Words like so,
indeed,andundoubtedly werechosenascuesfor RESULT.
21.3 ReferenceResolution
andeven Stigand, the patriotic archbishop ofCanterbury, f ound itadvisable–”’
‘FoundWHAT?’ saidthe Duck.
‘FoundIT,’the Mouse replied rather crossly: ‘of course you know what ”it”means.’

‘I know what “it”means well enough, when I ﬁnd a thing,’ said t he Duck: ‘it’s generally
afrogor aworm. The question is,what didthe archbishop ﬁnd?
Lewis Carroll,AliceinWonderland
In order to interpret the sentences of any discourse, we need to know who or what
entityisbeingtalkedabout. Considerthefollowingpassag e:
(21.34)VictoriaChen,ChiefFinancialOfﬁcerofMegabucks BankingCorpsince
2004,saw herpayjump20%,to $1.3million,as the37-year-oldalsobecame

theDenver-basedﬁnancial-servicescompany’spresident.It hasbeenten
yearssince shecameto MegabucksfromrivalLotsabucks.
Inthispassage,eachoftheunderlinedphrasesisusedbythe speakertodenoteone
person named Victoria Chen. We refer to this use of linguisti c expressionslike heror
Victoria Chen to denote an entity or individual as reference . In the next few sections Reference
of this chapter we study the problemof reference resolution . Reference resolution isReference
resolution
thetask ofdeterminingwhatentitiesarereferredtobywhic hlinguisticexpressions.

We ﬁrst deﬁne some terminology. A natural language expressi on used to perform
reference is called a referring expression , and the entity that is referred to is calledReferringexpression
thereferent. Thus,VictoriaChen andsheinpassage(21.34)arereferringexpressions, Referent
and Victoria Chen is their referent. (To distinguish betwee n referring expressionsand
theirreferents,weitalicizetheformer.) Asa convenients horthand,wewill sometimes
speakofareferringexpressionreferringtoareferent,e.g .,wemightsaythat sherefers

DRAFT708 Chapter 21. ComputationalDiscourse
to Victoria Chen. However, the reader should keep in mind tha t what we really mean
is that the speaker is performingthe act of referring to Vict oria Chen by uttering she.
Two referringexpressionsthat are used to refer to the same e ntity are said to corefer; Corefer
thusVictoria Chen andshecorefer in passage (21.34). There is also a term for a
referring expression that licenses the use of another, in th e way that the mention of
JohnallowsJohntobesubsequentlyreferredtousing he. Wecall Johntheantecedent Antecedent
ofhe. Reference to an entity that has been previouslyintroduced into the discourse is
calledanaphora ,andthereferringexpressionusedissaidtobe anaphoric . Inpassage Anaphora
Anaphoric (21.34), the pronouns sheandher, and the deﬁnite NP the 37-year-old are therefore
anaphoric.
Natural languagesprovidespeakerswith a variety of ways to refer to entities. Say
that your friend has a 1961 Ford Falcon automobile and you wan t to refer to it. De-

pending on the operative discourse context , you might say it, this, that, this car, that Discorse context
car, the car ,the Ford,the Falcon , ormy friend’s car , among many other possibilities.
However, you are not free to choose between any of these alter natives in any con-
text. For instance, you cannot simply say itorthe Falcon if the hearer has no prior
knowledge of your friend’s car, it has not been mentioned bef ore, and it is not in the
immediate surroundings of the discourse participants (i.e ., thesituational context of Situationalcontext
thediscourse).

The reason for this is that each type of referring expression encodes different sig-
nals about the place that the speaker believesthe referent o ccupieswithin the hearer’s
set of beliefs. A subset of these beliefs that has a special st atus form the hearer’s
mental model of the ongoing discourse, which we call a discourse model (Webber, Discourse model
1978). The discourse model containsrepresentationsof the entities that have been re-
ferred to in the discourse and the relationships in which the y participate. Thus, there

are two componentsrequired by a system to successfully inte rpret (or produce) refer-
ring expressions: a method for constructing a discourse mod el that evolves with the
dynamically-changingdiscourse it represents, and a metho d for mapping between the
signals that various referring expressions encode and the h earer’s set of beliefs, the
latterofwhichincludesthisdiscoursemodel.
We will speak in terms of two fundamental operations to the di scourse model.
When a referent is ﬁrst mentioned in a discourse, we say that a representation for it

isevokedinto the model. Upon subsequent mention, this representati on isaccessed Evoke
Access from the model. The operations and relationships are illust rated in Fig. 21.5. As we
will see in Sec. 21.8, the discourse model plays an important role in how coreference
algorithmsareevaluated.
We are now ready to introduce two reference resolution tasks :coreference reso-
lutionandpronominal anaphora resolution . Coreference resolution is the task ofCoreference
ﬁnding referringexpressionsin a text that refer to the same entity, i.e. ﬁnding expres-

sionsthat corefer. We calltheset ofcoreferringexpressionsa coreferencechain . For Coreferencechain
example,in processing(21.34),a coreferenceresolutiona lgorithmwould need to ﬁnd
fourcoreferencechains:
1.{VictoriaChen ,ChiefFinancialOfﬁcerofMegabucks BankingCorpsince1994 ,her,the
37-year-old ,the Denver-based ﬁnancial-services company’s president ,She}
2.{Megabucks BankingCorp ,the Denver-based ﬁnancial-services company ,Megabucks}

DRAFTSection21.3. ReferenceResolution 709
Discourse Model
"John" " he" coreferrefer (evoke)refer (access)
Figure21.5 Reference operations andrelationships withrespect tothe discourse model.
3.{her pay}
4.{Lotsabucks}
Coreferenceresolutionthusrequiresﬁndingallreferring expressionsinadiscourse,
and grouping them into coreference chains. By contrast, pronominal anaphora res-
olutionis the task of ﬁnding the antecedent for a single pronoun; for example, givenanaphora
thepronoun her,ourtaskistodecidethattheantecedentof herisVictoriaChen . Thus
pronominalanaphoraresolutioncanbeviewedasasubtaskof coreferenceresolution.2
In the next section we introducedifferent kinds of referenc e phenomena. We then
give various algorithms for reference resolution. Pronomi nal anaphora has received
a lot of attention in speech and language processing, and so w e will introduce three
algorithmsfor pronounprocessing: the Hobbsalgorithm,a Centering algorithm,and

alog-linear (MaxEnt) algorithm. We then give an algorithm for the more ge neral
coreferenceresolutiontask.
We will see that each of these algorithms focuses on resolvin g reference to enti-
ties or individuals. It is important to note, however, that d iscourses do include ref-
erence to many other types of referents than entities. Consi der the possibilities in
example(21.35),adaptedfromWebber(1991).
(21.35)AccordingtoDoug,Suejust boughta1961FordFalcon.
a. Butthatturnedouttobea lie.
b. Butthatwasfalse.

c.Thatstruckmeasa funnywaytodescribethesituation.
d.ThatcausedaﬁnancialproblemforSue.
The referentof thatis a speech act (see Ch. 24) in (21.35a),a propositionin (21. 35b),
a manner of description in (21.35c), and an event in (21.35d) . The ﬁeld awaits the
developmentofrobustmethodsforinterpretingthesetypes ofreference.
2Although technically there are cases of anaphora that are no t cases of coreference; see van Deemter and
Kibble (2000) for morediscussion.

DRAFT710 Chapter 21. ComputationalDiscourse
21.4 ReferencePhenomena
The set of referential phenomena that natural languages pro vide is quite rich indeed.
In this section, we provide a brief description of several ba sic reference phenomena,
surveying ﬁve types of referring expression: indeﬁnite noun phrases ,deﬁnite noun
phrases,pronouns ,demonstratives , andnames. We then summarize the way these
referring expressions are used to encode givenandnewinformation, along the way
introducing two types of referents that complicate the refe rence resolution problem:
inferrables andgenerics.
21.4.1 FiveTypes ofReferring Expressions
Indeﬁnite Noun Phrases Indeﬁnite referenceintroducesentities that are new to the
hearer into the discourse context. The most common form of in deﬁnite reference is
markedwiththedeterminer a(oran),butit canalsobemarkedbyaquantiﬁersuchas
someoreventhe determiner this:
(21.36) (a) Mrs. Martinwasso verykindastosendMrs. Goddard abeautiful
goose.

goose.
(b) He hadgoneroundonedayto bringher somewalnuts .
(c) I saw thisbeautifulFordFalcon today.
Such noun phrases evoke a representation for a new entity tha t satisﬁes the given de-
scriptionintothediscoursemodel.
The indeﬁnite determiner adoes not indicate whether the entity is identiﬁable to
the speaker, which in some cases leads to a speciﬁc/non-speciﬁc ambiguity. Example
(21.36a)onlyhasthespeciﬁcreading,sincethespeakerhas aparticulargooseinmind,

goose.
particularly the one Mrs. Martin sent. In sentence (21.37), on the other hand, both
readingsarepossible.
(21.37)Iamgoingtothe butcherstobuya goose.
That is, the speaker may already have the goose picked out (sp eciﬁc), or may just be
planningto pickoneoutthatisto herliking(nonspeciﬁc).
DeﬁniteNounPhrases Deﬁnitereferenceisusedtorefertoanentitythatisidenti ﬁ-
able to the hearer. An entity can be identiﬁableto the hearer becauseit has been men-
tionedpreviouslyinthetext,andthusisalreadyrepresent edin thediscoursemodel:

(21.38)Itconcernsa whitestallionwhichIhavesoldtoanofﬁcer. Bu t thepedigree
ofthewhitestallion wasnotfullyestablished.
Alternatively,anentitycanbeidentiﬁablebecauseitisco ntainedinthehearer’sset
ofbeliefsabouttheworld,ortheuniquenessoftheobjectis impliedbythedescription
itself,inwhichcaseit evokesarepresentationoftherefer entintothediscoursemodel,
asin(21.39):
(21.39)Ireadaboutit in TheNew YorkTimes .

DRAFTSection21.4. ReferencePhenomena 711
Pronouns Anotherformofdeﬁnite referenceis pronominalization,il lustratedin ex-
ample(21.40).
(21.40)Emmasmiledandchattedascheerfullyas shecould,
The constraints on using pronominalreference are stronger than for full deﬁnite noun
phrases, requiring that the referent have a high degree of ac tivation or saliencein the Salience
discourse model. Pronouns usually (but not always) refer to entities that were intro-
duced no further than one or two sentences back in the ongoing discourse, whereas
deﬁnite nounphrasescan often refer further back. This is il lustrated by the difference
betweensentences(21.41d)and(21.41d’).
(21.41) a. JohnwenttoBob’sparty,andparkednexttoaclassic FordF alcon.
b. He wentinsideandtalkedtoBobformorethananhour.
c. Bobtoldhimthat herecentlygotengaged.
d. ?? Healso saidthat hebought ityesterday.
d.’ He alsosaid thathebought theFalcon yesterday.
Bythetimethelastsentenceisreached,theFalconnolonger hasthedegreeofsalience

requiredto allowforpronominalreferencetoit.
Pronouns can also participate in cataphora , in which they are mentioned before Cataphora
theirreferentsare,asinexample(21.42).
(21.42)Evenbefore shesawit,DorothyhadbeenthinkingabouttheEmeraldCity
everyday.
Here,thepronouns sheanditbothoccur beforetheirreferentsareintroduced.
Pronouns also appear in quantiﬁed contexts in which they are considered to be
bound, asinexample(21.43). Bound
(21.43)Everydancerbrought herleftarmforward.
Under the relevant reading, herdoes not refer to some woman in context, but instead

everyday.
behaves like a variable bound to the quantiﬁed expression every dancer . We will not
beconcernedwiththe boundinterpretationofpronounsinth ischapter.
Demonstratives Demonstrative pronouns, like thisandthat, behave somewhat dif-
ferentlythansimpledeﬁnitepronounslike it. Theycanappeareitheraloneorasdeter-
miners,forinstance, thisingredient ,thatspice .Thisandthatdifferinlexicalmeaning;
(this, theproximal demonstrative ,indicating literal or metaphoricalcloseness, while Proximal
that,thedistaldemonstrative indicatingliteralormetaphoricaldistance(furtheraway Distal

intime,asin thefollowingexample)):
(21.44)Ijust boughta copyofThoreau’s Walden. I hadboughtoneﬁveyearsago.
Thatone hadbeenverytattered; thisonewasinmuchbettercondition.
Note that this NPis ambiguous; in colloquial spoken English, it can be indeﬁn ite,
asin(21.36),ordeﬁnite,asin (21.44).
Names Names are a very commonformof referringexpression,includ ingnamesof
people,organizations,andlocations,as wewill see in thed iscussionofnamedentities
inSec. 22.1. Namescanbeusedtorefertobothnewandoldenti tiesin thediscourse:

(21.45) a.Miss Woodhouse certainlyhadnotdonehimjustice.

DRAFT712 Chapter 21. ComputationalDiscourse
b.InternationalBusiness Machines soughtpatentcompensationfrom
Amazon; I.B.M.hadpreviouslysuedothercompanies.
21.4.2 InformationStatus
We notedabovethat the samereferringexpressions(suchas m anyindeﬁniteNPs) can
beusedtointroducenewreferents,whileotherexpressions (suchasmanydeﬁniteNPs,
or pronouns)can be used to referanaphoricallyto old refere nts. This idea of studying
thewaydifferentreferentialformsareusedtoprovidenewo roldinformationiscalled
informationstatus orinformationstructure . Information status
Informationstructure There are a variety of theories that express the relation bet ween different types of
referential form and the informativity or saliency of the re ferent in the discourse. For
example, the givenness hierarchy (Gundel et al., 1993) is a scale representing sixGivennesshierarchy
kindsofinformationstatusthat differentreferringexpre ssionareusedtosignal:
Thegivenness hierarchy:
uniquely type

infocus >activated >familiar >identiﬁable >referential >identiﬁable
{it}

that
this
thisN

{that N }{the N} {indef.thisN}{aN}
The related accessibility scale of Ariel (2001) is based on the idea that referents Accessibilityscale
that are more salient will be easier for the hearer to call to m ind, and hence can be
referred to with less linguistic material. By contrast, les s salient entities will need a
longer and more explicit referring expression to help the he arer recover the referent.

{it}

that
this
thisN

Thefollowingshowsasamplescale goingfromlowto highacce ssibility:
Full name >long deﬁnite description >short deﬁnite description >last name
>ﬁrst name >distal demonstrative >proximate demonstrative >NP>stressed
pronoun >unstressed pronoun
Note that accessibility correlates with length, with less a ccessible NPs tending to
be longer. Indeed, if we follow a coreference chain in a disco urse, we will often ﬁnd
longerNPs(forexamplelongdeﬁnitiondescriptionswithre lativeclauses) earlyin the

discourse,andmuchshorterones(forexamplepronouns)lat erinthediscourse.
Anotherperspective,basedontheworkof(Prince,1992),is toanalyzeinformation
status in terms of two crosscutting dichotomies: hearer status anddiscourse status .
Thehearer status of a referentexpresseswhether it is previouslyknownto the hearer,
or whether it is new. The discourse status expresses whether the referent has been
previouslymentionedin thediscourse.
The relationship between referring expression form and inf ormation status can be

complicated; we summarize below three such complicating fa ctors (the use of in-
ferrables ,generics,andnon-referentialforms ):
Inferrables: In some cases, a referring expression does not refer to an ent ity that
hasbeen explicitlyevokedin the text, but instead onethat i s inferentiallyrelated to an
evokedentity. Suchreferentsarecalled inferrables ,bridginginferences ,ormediated Inferrbale
Bridginginference
Mediated

DRAFTSection21.5. FeaturesforPronominalAnaphoraResolution 713
(HavilandandClark,1974;Prince,1981;Nissimetal.,2004 )Considertheexpressions
adoorandthe engine insentence(21.46).
(21.46)Ialmostboughta1961FordFalcontoday,but adoorhada dentand the
engineseemednoisy.
The indeﬁnite noun phrase a doorwould normally introduce a new door into the dis-
coursecontext,but in thiscase the heareris to infer someth ingmore: that it is notjust
any door, but one of the doors of the Falcon. Similarly, the us e of the deﬁnite noun
phrasethe engine normally presumes that an engine has been previously evoked or is
otherwise uniquely identiﬁable. Here, no engine has been ex plicitly mentioned, but
the hearer makes a bridging inference to infer that the referent is the engine of the
previouslymentionedFalcon.
Generics: Anotherkindofexpressionthat doesnotreferback to anenti ty explicitly
evokedinthe textis genericreference. Considerexample(21.47).
(21.47)I’minterestedinbuyingaMac laptop. Theyare verystylish.

Here,theyrefers, not to a particular laptop (or even a particular set o f laptops), but
instead to the class of Mac laptopsin general. Similarly, th e pronoun youcan be used
genericallyinthefollowingexample:
(21.48)InMarchinBoulder youhavetoweara jacket.
Non-referential uses: Finally, some non-referential forms bear a confusing super -
ﬁcial resemblance to referring expressions. For example in addition to its referring
usages, the word itcan be used in pleonastic cases like it is raining , in idiomslike hit Pleonastic

itoff,orinparticularsyntacticsituationslike clefts(21.49a)or extraposition (21.49b): Cleft
(21.49) (a)ItwasFrodowhocarriedthering.
(b)Itwasgoodthat Frodocarriedthering.
21.5 Featuresfor PronominalAnaphoraResolution
We nowturnto the task ofresolvingpronominalreference. In general,thisproblemis
formulatedas follows. We are given a single pronoun( he, him, she, her, it , and some-
timesthey/them ),togetherwiththe previouscontext. Ourtask isto ﬁndthea ntecedent
of the pronoun in this context. We present three systems for t his task; but ﬁrst we

summarizeusefulconstraintsonpossiblereferents.
We begin with ﬁve relatively hard-and-fast morphosyntacti c features that can be
used to ﬁlter the set of possible referents: number,person,gender, andbinding
theoryconstraints.
Number Agreement: Referring expressions and their referents must agree in num -
ber; for English, this means distinguishing between singularandpluralreferences.
Englishshe/her/he/him/his/it are singular, we/us/they/them are plural, and youis un-
speciﬁedfornumber. Someillustrationsoftheconstraints onnumberagreement:

DRAFT714 Chapter 21. ComputationalDiscourse
John has a FordFalcon. Itis red. *John has a FordFalcon. They are red.
John has three FordFalcons. Theyare red. *John has three For dFalcons. Itis red.
We cannot always enforce a very strict grammatical notion of number agreement,
sincesometimessemanticallypluralentitiescanbereferr edto byeither itorthey:
(21.50)IBMannouncedanewmachinetranslationproductyesterday. Theyhave
beenworkingonit for20years.
Person Agreement: English distinguishesbetweenthreeformsofperson: ﬁrst, sec-
ond, and third. The antecedent of a pronoun must agree with th e pronoun in number.
A ﬁrst person pronoun ( I,me,my) must have a ﬁrst person antecedent ( I,me, ormy).
A secondpersonpronoun( youoryour) must havea secondpersonantecedent( youor
your). A third person pronoun( he, she, they, him, her, them, his, her, their ) must have
a thirdpersonantecedent(oneoftheaboveoranyothernounp hrase).
Gender Agreement: Referents also must agree with the gender speciﬁed by the re-

ferringexpression. English third personpronounsdisting uish between male, (he, him,
his),female, (she, her) andnonpersonal (it) genders. Unlike in some languages, En-
glish male andfemale pronoungendersonlyapplyto animatee ntities; inanimateenti-
tiesarealwaysnonpersonal/neuter. Someexamples:
(21.51)JohnhasaFord. He isattractive. (he=John,nottheFord)
(21.52)JohnhasaFord. It isattractive. (it=theFord,notJohn)
Binding Theory Constraints: Reference relations may also be constrained by the

syntacticrelationshipsbetweenareferentialexpression andapossibleantecedentnoun
phrase when both occur in the same sentence. For instance, th e pronounsin all of the
followingsentencesare subjecttotheconstraintsindicat edinbrackets.
(21.53)Johnboughthimselfa newFord. [himself =John]
(21.54)JohnboughthimanewFord. [him ̸=John]
(21.55)Johnsaid thatBill boughthimanewFord. [him ̸=Bill]
(21.56)Johnsaid thatBill boughthimselfanewFord. [himself =Bill]
(21.57)Hesaid thatheboughtJohna newFord. [He̸ =John;he̸=John]

English pronouns such as himself,herself, andthemselves are called reﬂexives . Reﬂexive
Oversimplifyingthe situation, a reﬂexive corefers with th e subject of the most imme-
diateclausethatcontainsit(ex.21.53),whereasanonreﬂe xivecannotcoreferwiththis
subject (ex. 21.54). That this rule applies only for the subj ect of the most immediate
clause is shownbyexamples(21.55)and (21.56),in whichthe oppositereferencepat-
tern is manifest between the pronoun and the subject of the hi gher sentence. On the

other hand, a full noun phrase like Johncannot corefer with the subject of the most
immediateclausenorwitha higher-levelsubject(ex.21.57 ).
These constraintsare often called the binding theory (Chomsky,1981),and quite Bindingtheory
complicated versions of these constraints have been propos ed. A complete statement
oftheconstraintsrequiresreferencetosemanticandother factors,andcannotbestated
purely in terms of syntactic conﬁguration. Nonetheless, fo r the algorithms discussed

later in this chapter we will assume a simple syntactic accou nt of restrictions on in-
trasententialcoreference.

DRAFTSection21.5. FeaturesforPronominalAnaphoraResolution 715
Selectional Restrictions: The selectional restrictions that a verb places on its argu-
ments(seeCh.19)mayberesponsibleforeliminatingrefere nts,asinexample(21.58).
(21.58)Johnparkedhiscarinthegarageafterdrivingit aroundforh ours.
Therearetwopossiblereferentsfor it,thecarandthegarage. Theverb drive,however,
requiresthatits directobjectdenotesomethingthat canbe driven,suchas a car,truck,
or bus, but not a garage. Thus, the fact that the pronoun appea rs as the object of
driverestricts the set of possible referents to the car. Selectio nal restrictions can be
implemented by storing a dictionary of probabilistic depen dencies between the verb
associatedwiththe pronounandthepotentialreferent.
Recency: We next turn to features for predicting the referent of a pron oun that are
lesshard-and-fast. Entitiesintroducedinrecentutteran cestendtobemoresalientthan
those introducedfrom utterancesfurther back. Thus, in exa mple (21.59),the pronoun

itismorelikelyto referto Jim’smapthanthedoctor’smap.
(21.59)Thedoctorfoundanoldmapin thecaptain’schest. Jim founda nevenolder
maphiddenontheshelf. Itdescribedan island.
Grammatical Role: Many theories specify a salience hierarchy of entities that is
ordered by the grammatical position of the expressions whic h denote them. These
typicallytreatentitiesmentionedinsubjectpositionasm oresalientthanthoseinobject
position,whichareinturnmoresalient thanthosementione dinsubsequentpositions.

Passages such as (21.60) and (21.61) lend support for such a h ierarchy. Although
the ﬁrst sentence in each case expresses roughly the same pro positional content, the
preferred referent for the pronoun hevaries with the subject in each case – John in
(21.60)andBill in(21.61).
(21.60)BillyBoneswenttothebarwith JimHawkins. Hecalledforagl assofrum.
[he= Billy ]
(21.61)JimHawkinswenttothe barwith Billy Bones. Hecalledforagl assofrum.
[he= Jim ]
Repeated Mention: Some theories incorporate the idea that entities that have b een

[he= Jim ]
focused on in the prior discourse are more likely to continue to be focused on in sub-
sequentdiscourse,andhencereferencestothemaremorelik elyto bepronominalized.
Forinstance,whereasthepronouninexample(21.61)hasJim asitspreferredinterpre-
tation,thepronounintheﬁnalsentenceofexample(21.62)m aybemorelikelytorefer
toBilly Bones.
(21.62)BillyBoneshadbeenthinkingaboutaglassofrumeversincet hepirateship
docked. He hobbledovertotheOld Parrotbar. Jim Hawkinswen twith him.
Hecalledforaglassofrum. [ he=Billy ]

Parallelism: There are also strong preferences that appear to be induced b y paral-
lelism effects,asin example(21.63).
(21.63)LongJohnSilverwentwith Jimto theOldParrot. Billy Bonesw ent withhim
totheOld AnchorInn. [ him=Jim ]

DRAFT716 Chapter 21. ComputationalDiscourse
ThegrammaticalrolehierarchydescribedaboveranksLongJ ohnSilverasmoresalient
than Jim, and thus should be the preferred referent of him. Furthermore, there is no
semantic reason that Long John Silver cannot be the referent . Nonetheless, himis
insteadunderstoodtoreferto Jim.
Verb Semantics Certain verbs appear to place a semantically-oriented emph asis on
one of their argument positions, which can have the effect of biasing the manner in
whichsubsequentpronounsareinterpreted. Comparesenten ces(21.64)and(21.65).
(21.64) Johntelephoned Bill. He lostthe laptop.
(21.65) JohncriticizedBill. He lost the laptop.
Theseexamplesdifferonlyintheverbusedintheﬁrstsenten ce,yetthesubjectpronoun
inpassage(21.64)istypicallyresolvedtoJohn,whereasth epronouninpassage(21.65)
is resolved to Bill. It has been argued that this effect resul ts from what the “implicit
causality” of a verb: the implicit cause of a “criticizing” e vent is considered to be

its object, whereas the implicit cause of a “telephoning” ev ent is considered to be its
subject. This emphasis results in a higher degree of salienc e for the entity in this
argumentposition.
21.6 Threealgorithmsforpronominalanaphoraresolution
21.6.1 Pronominal Anaphora Baseline: The HobbsAlgorithm
The ﬁrst of the three algorithms we present for pronominal an aphoraresolution is the
Hobbsalgorithm . TheHobbsalgorithm(thesimpleroftwoalgorithmspresent edorig- Hobbs algorithm

inallyinHobbs(1978))dependsonlyonasyntacticparserpl usamorphologicalgender
andnumberchecker. Forthisreasonitisoftenusedasabasel inewhenevaluatingnew
pronominalanaphoraresolutionalgorithms.
The input to the Hobbs algorithm is a pronoun to be resolved, t ogether with a
syntacticparseofthesentencesuptoandincludingthecurr entsentence. Thealgorithm
searchesforanantecedentnounphraseinthese trees. Thein tuitionofthealgorithmis
to start with the target pronounand walk up the parse tree to t he rootS. For each NP

orSnode that it ﬁnds, it does a breadth-ﬁrst left-to-rightsear ch of the node’schildren
to the left of the target. As each candidate noun phrase is pro posed, it is checked for
gender, number, and person agreement with the pronoun. If no referent is found, the
algorithmperformsthesame breadth-ﬁrstsearchonprecedi ngsentences.
TheHobbsalgorithmdoesnotcapturealltheconstraintsand preferencesonpronom-
inalizationdescribedabove. Itdoes,however,approximat ethebindingtheory ,recency,

andgrammatical role preferences by the order in which the search is performed, an d
thegender,person,andnumberconstraintsbya ﬁnalcheck.
An algorithm that searches parse trees must also specify a gr ammar, since the as-
sumptionsregardingthe structure of syntactic trees will a ffect the results. A fragment
forEnglish that the algorithmuses is givenin Fig. 21.6. The steps ofthe Hobbsalgo-
rithmareasfollows:
1. Beginat thenounphrase(NP)nodeimmediatelydominating thepronoun.

DRAFTSection21.6. Threealgorithmsforpronominalanaphorares olution 717
S→NPVP
NP→

(Det) Nominal({PP
Rel})∗
pronoun


Det→{
determiner
NP ’s}
PP→prepositionNP
Nominal→noun(PP)∗
Rel→wh-wordS
VP→verbNP (PP)∗
Figure21.6 A grammar fragment forthe Tree Searchalgorithm.
2. Go up the tree to the ﬁrst NP or sentence (S) node encountere d. Call this node
X,andcallthepathusedtoreachit p.
3. TraverseallbranchesbelownodeXtotheleftofpath pinaleft-to-right,breadth-
ﬁrst fashion. Propose as the antecedent any NP node that is en counteredwhich
hasanNP orS nodebetweenit andX.
4. If node X is the highest S node in the sentence, traverse the surface parse trees
of previous sentences in the text in order of recency, the mos t recent ﬁrst; each
tree is traversed in a left-to-right,breadth-ﬁrst manner, and when an NP node is
encountered, it is proposed as antecedent. If X is not the hig hest S node in the
sentence,continuetostep5.
5. FromnodeX,goupthetreetotheﬁrstNPorSnodeencountere d. Callthisnew

nodeX,andcallthe pathtraversedto reachit p.
6. IfXisanNPnodeandifthepath ptoXdidnotpassthroughtheNominalnode
thatX immediatelydominates,proposeX asthe antecedent.
7. TraverseallbranchesbelownodeXtothe leftofpathpinaleft-to-right,breadth-
ﬁrstmanner. ProposeanyNP nodeencounteredasthe antecede nt.
8. IfXisanSnode,traverseallbranchesofnodeXtothe rightofpathpinaleft-to-
right, breadth-ﬁrst manner, but do not go below any NP or S nod e encountered.
ProposeanyNP nodeencounteredastheantecedent.
9. GotoStep4.

Demonstrating that this algorithm yields the correct coref erence assignments for an
examplesentenceis leftasExercise2.
Most parsers return number information (singular or plural ), and person informa-
tioniseasily encodedbyrulefortheﬁrst andsecondpersonp ronouns. But parsersfor
English rarely return gender informationfor common or prop er nouns. Thus the only
additional requirement to implementing the Hobbs algorith m, besides a parser, is an
algorithmfordetermininggenderforeachantecedentnounp hrase.

One common way to assign gender to a noun phrase is to extract t he head noun,
and then use WordNet (Ch. 19) to look at the hypernynsof the he ad noun. Ancestors
likepersonorliving thing indicate an animate noun. Ancestors like femaleindicate a
female noun. A list of personal names associated with gender s, or patterns like Mr.
canalsobe used(CardieandWagstaff,1999).

DRAFT718 Chapter 21. ComputationalDiscourse
Morecomplexalgorithmsexist, such as that of Bergsmaand Li n (2006);Bergsma
and Lin also make freely available a large list of nouns and th eir (automatically ex-
tracted)genders.
21.6.2 A Centering AlgorithmforAnaphora Resolution
The Hobbs algorithm does not use an explicit representation of a discourse model.
By contrast Centering theory , (Grosz et al., 1995b, henceforth GJW) is a family of Centeringtheory
modelswhich hasan explicitrepresentationofa discoursem odel,andincorporatesan
additional claim: that there is a single entity being “cente red” on at any given point
in the discourse which is to be distinguished from all other e ntities that have been
evoked. Centering theory has been applied to many problems i n discourse, such as
the computation of entity-based coherence ; in this section we see its application to
anaphoraresolution.
TherearetwomainrepresentationstrackedintheCentering theorydiscoursemodel.

Inwhatfollows,take UnandUn+1tobetwoadjacentutterances. The backwardlook-
ing center ofUn, denoted as Cb(Un), represents the entity currently being focused onBackwardlookingcenter
in the discourse after Unis interpreted. The forward looking centers ofUn, denotedForwardlookingcenter
asCf(Un), form an ordered list containing the entities mentioned in Un, all of which
could serve as the Cbof the followingutterance. In fact, Cb(Un+1)is by deﬁnition the
most highly ranked element of Cf(Un)mentioned in Un+1. (The C bof the ﬁrst utter-

anceinadiscourseisundeﬁned.) Asforhowtheentitiesinth eCf(Un)areordered,for
simplicity’ssake wecanuse thegrammaticalrolehierarchy below.3
subject >existentialpredicatenominal >object >indirectobjectoroblique
>demarcatedadverbialPP
As a shorthand, we will call the highest-ranked forward-loo king center Cp(for “pre-
ferredcenter”).
Wedescribeacentering-basedalgorithmforpronouninterp retationduetoBrennan
etal.(1987,henceforthBFP).(SeealsoWalkeretal.(1994) andtheendofthechapter

for other centering algorithms). In this algorithm, prefer red referents of pronounsare
computedfrom relationsthat hold between the forwardand ba ckwardlookingcenters
in adjacent sentences. Four intersentential relationship s between a pair of utterances
UnandUn+1aredeﬁnedwhichdependontherelationshipbetween Cb(Un+1),Cb(Un),
andCp(Un+1); theseareshownin Fig.21.7.
Cb(Un+1) =Cb(Un) Cb(Un+1)̸=Cb(Un)
orundeﬁned Cb(Un)
Cb(Un+1) =Cp(Un+1) Continue Smooth-Shift
Cb(Un+1)̸=Cp(Un+1) Retain Rough-Shift
Figure21.7 Transitions inthe BFPalgorithm.

Thefollowingrulesareusedbythe algorithm:
3Thisis an extended form of thehierarchy used in Brennan etal . (1987), described below.

DRAFTSection21.6. Threealgorithmsforpronominalanaphorares olution 719
Rule 1:If any element of Cf(Un)is realized by a pronoun in utterance
Un+1, thenCb(Un+1)mustberealizedasa pronounalso.
Rule 2:Transitionstates are ordered. Continueis preferredto Ret ain is
preferredtoSmooth-ShiftispreferredtoRough-Shift.
Havingdeﬁnedthese conceptsandrules,the algorithmisdeﬁ nedasfollows.
1. Generatepossible Cb-Cfcombinationsforeachpossible set ofreferenceassign-
ments.
2. Filter by constraints, e.g., syntactic coreference cons traints, selectional restric-
tions,centeringrulesandconstraints.
3. Rankbytransitionorderings.
The pronominal referents that get assigned are those which y ield the most preferred
relation in Rule 2, assuming that Rule 1 and other coreferenc e constraints (gender,
number,syntactic,selectionalrestrictions)arenotviol ated.
Letusstepthroughpassage(21.66)toillustrate thealgori thm.
(21.66)Johnsaw abeautiful1961FordFalconat theusedcardealersh ip. (U1)
Heshowedit to Bob. ( U2)
Heboughtit. ( U3)

UsingthegrammaticalrolehierarchytoordertheC f, forsentence U1we get:
Cf(U1):{John,Ford,dealership }
Cp(U1): John
Cb(U1): undeﬁned
Sentence U2containstwo pronouns: he, which is compatible with John, and it, which
iscompatiblewiththeFordorthedealership. Johnisbydeﬁn itionCb(U2), becausehe
isthehighestrankedmemberof Cf(U1)mentionedin U2(sinceheistheonlypossible
referent for he). We compare the resulting transitions for each possible re ferent ofit.
Ifwe assume itrefersto theFalcon,theassignmentswouldbe:
Cf(U2):{John,Ford,Bob}
Cp(U2): John
Cb(U2): John

Result: Continue ( Cp(U2)=Cb(U2);Cb(U1)undeﬁned)
Ifwe assume itrefersto thedealership,theassignmentswouldbe:
Cf(U2):{John,dealership,Bob }
SincebothpossibilitiesresultinaContinuetransition,t healgorithmdoesnotsaywhich
toaccept. Forthesakeofillustration,wewillassumethatt iesarebrokenintermsofthe
orderingon the previous Cflist. Thus, we will take itto refer to the Falcon instead of
thedealership,leavingthecurrentdiscoursemodelasrepr esentedintheﬁrstpossibility

above.

DRAFT720 Chapter 21. ComputationalDiscourse
In sentence U3,heis compatiblewith either John or Bob, whereas itis compatible
withtheFord. Ifweassume hereferstoJohn,thenJohnis Cb(U3)andtheassignments
wouldbe:
Cf(U3):{John,Ford}
Cp(U3): John
Cb(U3): John
Result: Continue ( Cp(U3)=Cb(U3)=Cb(U2))
Ifwe assume hereferstoBob,thenBobis Cb(U3)andtheassignmentswouldbe:
Cf(U3):{Bob,Ford}
Cp(U3): Bob
Cb(U3): Bob
Result: Smooth-Shift ( Cp(U3)=Cb(U3);Cb(U3)̸=Cb(U2))
Since a Continue is preferred to a Smooth-Shift per Rule 2, Jo hn is correctly taken to
bethereferent.
The main salience factors that the centering algorithm impl icitly incorporates in-
cludethegrammaticalrole,recency,andrepeatedmentionp references. Themannerin
whichthegrammaticalrolehierarchyaffectssalienceisin direct,sinceitistheresulting
transitiontypethat determinestheﬁnal referenceassignm ents. Inparticular,a referent
ina low-rankedgrammaticalrolewillbepreferredtooneina morehighlyrankedrole

if the former leads to a more highly ranked transition. Thus, the centering algorithm
may incorrectlyresolvea pronounto a low salience referent . Forinstance, in example
(21.67),
(21.67)Bobopenedupa newdealershiplast week. Johntooka lookat th eFordsin
hislot. He endedupbuyingone.
the centering algorithm will assign Bob as the referent of th e subject pronoun hein
thethirdsentence–sinceBobis Cb(U2), thisassignmentresultsina Continuerelation

(21.67),
whereas assigning John results in a Smooth-Shift relation. On the other hand, the
Hobbsalgorithmwill correctlyassignJohnasthereferent.
LiketheHobbsalgorithm,thecenteringalgorithmrequires afullsyntacticparseas
well asmorphologicaldetectorsforgender.
Centeringtheoryisalsoamodelofentitycoherence,andhen cehasimplicationsfor
otherdiscourseapplicationslikesummarization;seethee ndofthechapterforpointers.
21.6.3 A Log-Linear model forPronominal Anaphora Resoluto n
Asourﬁnalmodelofpronominalanaphoraresolution,wepres entasimplesupervised

machine learning approach, in which we train a log-linear cl assiﬁer on a corpus in
which the antecedents are marked for each pronoun. Any super vised classiﬁer can
be used for this purpose; log-linear models are popular, but Naive Bayes and other
classiﬁershavebeenusedaswell.
Fortraining,thesystemreliesonahand-labeledcorpusinw hicheachpronounhas
been linked by hand with the correct antecedent. The system n eeds to extract positive
and negative examples of anaphoric relations. Positive exa mples occur directly in the

DRAFTSection21.6. Threealgorithmsforpronominalanaphorares olution 721
trainingset. Negativeexamplescanbefoundbypairingeach pronounwithsomeother
noun phrase. Features (discussed in the next section) are ex tracted for each training
observation, and a classiﬁer is trained to predict 1for the true pronoun-antecedent
pairs,and 0forthe incorrectpronoun-antecedentpairs.
Fortesting,justaswesawwithaswiththeHobbsandCenterin gclassiﬁers,thelog-
linear classiﬁer takes as input a pronoun ( he, him, his, she, her, it, they, them, their ),
togetherwiththecurrentandprecedingsentences.
In order to deal with non-referential pronouns, we ﬁrst ﬁlte r out pleonastic pro-
nouns (like the pleonastic it is raining ), using hand-written rules based on frequent
lexicalpatterns.
Theclassiﬁer thenextractsall potentialantecedentsbydo ingaparseofthecurrent
and previous sentences, either using a full parser or a simpl e chunker. Next, each NP
in the parse is considered a potential antecedent for each fo llowing pronoun. Each

pronoun-potentialantecedentpairisthenpresentedtothe classiﬁer.
21.6.4 Features
Somecommonlyusedfeaturesforpronominalanaphoraresolu tionbetweenapronoun
Proianda potentialreferent NPjinclude:
1.strict gender [true orfalse]. True if there is a strict match in gender(e.g. male
pronounProiwithmaleantecedent NPj).
2.compatiblegender [true orfalse]. TrueifProiandNPjare merelycompatible
(e.g. malepronoun Proiwithantecedent NPjofunknowngender).
3.strict number [true orfalse]True if there is a strict match in number (e.g.

singularpronounwith singularantecedent)
4.compatiblenumber[true orfalse]. TrueifProiandNPjaremerelycompatible
(e.g. singularpronoun Proiwith antecedent NPjofunknownnumber).
5.sentencedistance[0,1,2,3,...] . Thenumberofsentencesbetweenpronounand
potentialantecedent.
6.Hobbsdistance[0,1,2,3,...] . ThenumberofnoungroupsthattheHobbsalgo-
rithmhastoskip,startingbackwardsfromthepronoun Proi,beforethepotential
antecedent NPjisfound.
7.grammatical role [subject, object, PP] . Whether the potential antecedent is a
syntacticsubject,directobject,orisembeddedin aPP.

8.linguistic form [proper, deﬁnite, indeﬁnite, pronoun] . Whether the potential
antecedent NPjis a proper name, deﬁnite description, indeﬁnite NP, or a pro -
noun.
Fig. 21.8 shows some sample feature values for potential ant ecedents for the ﬁnal
HeinU3:
(21.68)Johnsaw abeautiful1961FordFalconat theusedcardealersh ip. (U1)
The classiﬁer will learn weightsindicatingwhich of these f eaturesare more likely
to be good predictors of a successful antecedent (e.g. being nearby the pronoun, in

DRAFT722 Chapter 21. ComputationalDiscourse
He(U2) it ( U2) Bob( U2) John( U1)
strictnumber 1 1 1 1
compatiblenumber 1 1 1 1
strictgender 1 0 1 1
compatiblegender 1 0 1 1
sentence distance 1 1 1 2
Hobbsdistance 2 1 0 3
grammaticalrole subject object PP subject
linguistic form pronoun pronoun proper proper
Figure21.8 Feature values inlog-linear classiﬁer,forvarious pronou ns from (21.68).
subjectposition,agreeingingenderandnumber). Thuswher etheHobbsandCentering
algorithmsrely onhand-builtheuristicsfor antecedentse lection, the machinelearning
classiﬁerslearntheimportanceofthesedifferentfeature sbasedontheirco-occurrence
inthe trainingset.
21.7 CoreferenceResolution
In the previous few sections, we concentrated on interpreti ng a particular subclass of
the reference phenomena that we outlined in Sec. 21.4: the pe rsonal pronouns such
ashe,she, andit. But for the general coreference task we’ll need to decide wh ether
any pair of noun phrases corefer. This means we’ll need to dea l with the other types

of referring expressions from Sec. 21.4, the most common of w hich aredeﬁnite noun
phrasesandnames. Let’sreturntoourcoreferenceexample,repeatedbelow:
(21.69)VictoriaChen,ChiefFinancialOfﬁcerofMegabucks BankingCorpsince
theDenver-basedﬁnancial-servicescompany’spresident.It hasbeentenyears
sinceshecameto MegabucksfromrivalLotsabucks.
Recallthat weneedtoextractfourcoreferencechainsfromt hisdata:

As before, we have to deal with pronominalanaphora (ﬁguring out thatherrefers
toVictoria Chen ). And we still need to ﬁlter out non-referential pronouns li ke the
pleonastic ItinIt hasbeentenyears ), aswedidforpronominalanaphora.
But for full NP coreference we’ll also need to deal with deﬁni te noun phrases, to

ﬁgure out that the 37-year-old is coreferentwith Victoria Chen , andthe Denver-based
ﬁnancial-services company is the same as Megabucks . And we’ll need to deal with
names,torealizethat Megabucks is thesameas MegabucksBankingCorp .

DRAFTSection21.7. CoreferenceResolution 723
An algorithm for coreference resolution can use the same log -linear classiﬁer ar-
chitecturewe saw forpronominalanaphora. Thuswe’ll build abinaryclassiﬁer which
is givenan anaphorand a potentialantecedentand returnstr ue (the two are coreferen-
tial) or false (the two are not coreferential). We’ll use thi s classiﬁer in the resolution
algorithm as follows. We process a documentfrom left to righ t. For each NPjwe en-
counter, we’ll search backwards through the document exami ning each previous NP.
For each such potential antecedent NPi, we’ll run our classiﬁer, and if it returns true,
we successfully coindex NPiandNPj. The process for each NPjterminates when we
either ﬁnd a successful antecedent NPior reach the beginning of the document. We
thenmoveonto thenextanaphor NPj.
In order to train our binary coreference classiﬁer, just as f or pronoun resolution,
we’ll need a labeled training set in which each anaphor NPihas been linked by hand

with the correct antecedent. In order to build a classiﬁer, w e’ll need both positive and
negativetrainingexamplesofcoreferencerelations. Apos itiveexamplesfor NPiisthe
nounphrase NPjwhich is markedas coindexed. We get negativeexamplesby pai ring
the anaphor NPjwith the interveningNPs NPi+1,NPi+2which occurbetweenthe true
antecedent NPiandtheanaphor NPj.
Nextfeaturesare extractedforeach trainingobservation, and a classiﬁer is trained
to predict whether an ( NPj,NPi) pair corefer or not. Which features should we use in

the binary coreference classiﬁer? We can use all the feature s we used for anaphora
resolution; number, gender, syntactic position, and so on. But we will also need to
add new features to deal with phenomenathat are speciﬁc to na mes and deﬁnite noun
phrases. For example, we’ll want a feature representing the fact thatMegabucks and
MegabucksBankingCorp sharetheword Megabucks ,orthatMegabucksBankingCorp
andthe Denver-based ﬁnancial-services company both end in words ( Corpandcom-
pany)indicatingacorporateorganization.

Here are some commonly used features for coreference betwee n an anaphor NPi
and a potential antecedent NPj(in addition to the features for pronominal anaphora
resolutionlisted onpage721):
1.anaphor edit distance [0,1,2,...,] . The character minimum edit distance from
the potential antecedent to the anaphor. Recall from Ch. 3 th at the character
minimum edit distance is the minimum number of character edi ting operations
(insertions, substitutions, deletions) necessary to turn one string into another.
100×m−(s+i+d)
m

m
given the antecedent length m, and the number of substitutions s, insertions i,
anddeletions d.
2.antecedent edit distance [0,1,2,...,] . Theminimum edit distance from the
anaphortotheantecedent. Giventheanaphorlength n:
100×n−(s+i+d)
n
3.alias[true orfalse]: A multi-partfeature proposedby Soon et al. (2001)which
requiresa namedentitytagger . Returnstrueif NPiandNPjarebothnameden-
titiesofthesametype,and NPiisanaliasofNPj. Themeaningof aliasdepends

DRAFT724 Chapter 21. ComputationalDiscourse
onthetypes;twodatesarealiasesofeachotheriftheyrefer tothesamedate. For
typePERSON,preﬁxeslike Dr.orChairman arestrippedoffandthentheNPsare
checkedto see iftheyare identical. Fortype ORGANIZATION , thealias function
checksforacronyms(e.g., IBMforInternationalBusinessMachinesCorp ).
4.appositive[true orfalse]: Trueiftheanaphorisinthesyntacticappositionrela-
tiontotheantecedent. ForexampletheNP ChiefFinancialOfﬁcerofMegabucks
BankingCorp isinappositiontotheNP VictoriaChen . Thesecanbedetectedus-
ingaparser,ormoreshallowlybylookingforcommasandrequ iringthatneither
NPhavea verbandoneofthembeaname.
5.linguistic form [proper, deﬁnite, indeﬁnite, pronoun] . Whether the potential
anaphorNPjisa propername,deﬁnitedescription,indeﬁniteNP ora pron oun.
21.8 EvaluatingCoreferenceResolution
OnestandardwayofevaluatingcoreferenceistheModel-The oreticcoreferencescoring
scheme(Vilain et al., 1995),originallyproposedforthe MU C-6 andMUC-7 informa-

tionextractionevaluation(Sundheim,1995a).
Theevaluationisbasedonahuman-labeledgoldstandardfor coreferencebetween
referringexpressions. We can representthis goldinformat ionas a set of identity links
between referring expressions. For example, the fact that r eferring expression A and
referring expression B are coreferent could be represented as a link A-B. If A, B, and
C are coreferent, this can be represented as the two links A-B , B-C (or alternatively
as A-C, B-C). We can call this set of correct links the reference orkeyset of links.

Similarly, the hypothesis orresponse from a coreference algorithm can be viewed as
a set oflinks.
What we’d like to do is compute the precision and recall of the response links
againstthe keylinks. ButrecallthatifentitiesA,B,andCarecoreferenti nthekey,this
canberepresentedeithervia(A-B,B-C)orvia(A-C,B-C).As longasourcoreference
system correctlyﬁguresout that A, B, andC arecoreferent,w e don’twant to penalize
it forrepresentingthisfact ina differentset oflinksthan happentobeinthe key.

Forexample,supposethatA,B,C,andDarecoreferent,andth ishappenstoberep-
resented in the key by links (A-B, B-C, C-D). Suppose further that a particular coref-
erence algorithm returns (A-B, C-D). What score should be gi ven to this response?
Intuitively the precision should be 1 (since both links corr ectly join referring expres-
sionsthatindeedcorefer). Therecallshouldbe2/3,sincei ntuitivelyittakesthreelinks
to correctly indicate that 4 expressions are coreferent, an d the algorithm returned two

of these three links. The details of this intuition are ﬂeshe d out in the Vilain et al.
(1995) algorithm, which is based on computing the number of e quivalence classes of
expressionsgeneratedbythekey.

DRAFTSection21.9. Advanced: Inference-BasedCoherenceResolu tion 725
21.9 Advanced: Inference-BasedCoherenceResolution
Thealgorithmswe haveseen in thischapterforthe resolutio nofcoherenceandcoref-
erencehavereliedsolelyonshallowinformationlikecueph rasesandotherlexicaland
simple syntactic cues. But many problems in resolution seem to require much more
sophisticated kinds of knowledge. Consider the following e xample of coreference,
adaptedfromWinograd(1972b):
(21.70)Thecity councildeniedthedemonstratorsa permitbecause
a. theyfearedviolence.
b. theyadvocatedviolence.
Determining the correct antecedent for the pronoun theyrequires understanding
ﬁrst that the second clause is intended as an Explanation of the ﬁrst clause, and also
that city councils are perhaps more likely than demonstrato rs to fear violence, and
demonstrators might be more likely to advocate violence. A m ore advanced method
forcoherenceresolutionmightassignthisExplanationrel ationandindoingsohelpus
ﬁgureoutthe referentsofbothpronouns.

We might performthis kind of more sophisticatedcoherencer esolutionby relying
onthe semanticconstraintsthat are associatedwith eachco herencerelation,assuming
a parserthatcouldassigna reasonablesemanticstoeachcla use.
Applying these constraints requires a method for performin g inference. Perhaps
the most familiartype of inferenceis deduction ; recall fromSec. 17.3that the central Deduction
ruleofdeductionismodusponens:
α⇒β
α
β
Anexampleofmodusponensisthefollowing:
All Falconsarefast.
John’scar isanFalcon.
John’scarisfast.

α⇒β
α
β
Deduction is a form of sound inference : if the premises are true, then the conclusion Soundinference
mustbetrue.
However, much of language understanding is based on inferen ces that are not
sound. While the ability to draw unsound inferences allows f or a greater range of
inferences to be made, it can also lead to false interpretati onsand misunderstandings.
A method for such inference is logical abduction (Peirce, 1955). The central rule of Abduction
abductiveinferenceis:
α⇒β
β
α

DRAFT726 Chapter 21. ComputationalDiscourse
Whereas deduction runs an implication relation forward, ab duction runs it backward,
reasoningfromaneffecttoapotentialcause. Anexampleofa bductionisthefollowing:
Obviously, this may be an incorrect inference: John’s car ma y be made by another
manufactureryetstill befast.
In general,a giveneffect βmay have manypotential causes αi. We generallywill
not want to merely reason froma fact to a possibleexplanationof it, we want to iden-
tify thebestexplanationof it. To do this, we need a methodfor comparingt he quality
of alternative abductiveproofs. This can be done with proba bilistic models(Charniak
and Goldman,1988;Charniak and Shimony,1990),or with heur istic strategies(Char-
niak and McDermott, 1985, Chapter 10), such as preferring th e explanation with the
smallest number of assumptions, or the most speciﬁc explana tion. We will illustrate

a third approach to abductive interpretation, due to Hobbs e t al. (1993), which ap-
pliesamoregeneralcost-basedstrategythatcombinesfeat uresoftheprobabilisticand
heuristic approaches. To simplify the discussion, however , we will largely ignore the
cost componentofthesystem,keepingin mindthatoneisnone thelessnecessary.
Hobbs et al. (1993) apply their method to a broad range of prob lems in language
interpretation; here we focus on its use in establishing dis course coherence, in which

worldanddomainknowledgeareusedtodeterminethemostpla usiblecoherencerela-
tionholdingbetweenutterances. Letusstepthroughtheana lysisthatleadstoestablish-
ing the coherence of passage (21.4). First, we need axioms ab out coherence relations
themselves. Axiom(21.71)statesthat a possiblecoherence relationis theExplanation
relation;otherrelationswouldhaveanalogousaxioms.
∀ei,ejExplanation (ei,ej)⇒CoherenceRel (ei,ej) (21.71)
The variables eiandejrepresent the events (or states) denoted by the two utteranc es

being related. In this axiom and those given below, quantiﬁe rs always scope over
everything to their right. This axiom tells us that, given th at we need to establish a
coherence relation between two events, one possibility is t o abductively assume that
therelationisExplanation.
TheExplanationrelationrequiresthatthesecondutteranc eexpressthecauseofthe
effectthattheﬁrst sentenceexpresses. We canstate thisas axiom(21.72).
∀ei,ejcause(ej,ei)⇒Explanation (ei,ej) (21.72)

Inadditionto axiomsaboutcoherencerelations,wealso nee daxiomsrepresenting
general knowledge about the world. The ﬁrst axiom we use says that if someone is
drunk, then others will not want that person to drive, and tha t the former causes the
latter(forconvenience,the state ofnotwantingisdenoted bythediswantpredicate).
∀x,y,eidrunk(ei,x)⇒
∃ej,ekdiswant (ej,y,ek)∧drive(ek,x)∧cause(ei,ej)(21.73)

DRAFTSection21.9. Advanced: Inference-BasedCoherenceResolu tion 727
Before we moveon,a few notesarein orderconcerningthisaxi omandthe otherswe
will present. First, axiom (21.73) is stated using universa l quantiﬁers to bind several
ofthe variables,whichessentially saysthatin all casesin whichsomeoneis drunk,all
peopledonotwant that persontodrive. Althoughwemighthop ethat thisisgenerally
thecase,suchastatementisnonethelesstoostrong. Theway inwhichthisishandledin
theHobbsetal.systemisbyincludinganadditionalrelatio n,calledan etcpredicate,in
theantecedentofsuchaxioms. An etcpredicaterepresentsallthe otherpropertiesthat
must be true for the axiom to apply, but which are too vague to s tate explicitly. These
predicates therefore cannot be proven, they can only be assu med at a corresponding
cost. Because rules with high assumption costs will be dispr eferred to ones with low
costs, the likelihood that the rule applies can be encoded in terms of this cost. Since

we have chosen to simplify our discussion by ignoringcosts, we will similarly ignore
theuse of etcpredicates.
Second, each predicate has what may look like an “extra” vari able in the ﬁrst ar-
gument position; for instance, the drivepredicate has two arguments instead of one.
Thisvariableis usedto reifythe relationshipdenotedbyth e predicateso that it can be
referred to from argument places in other predicates. For in stance, reifying the drive

predicate with the variable ekallows us to expressthe idea of not wanting someone to
drivebyreferringtoit in theﬁnalargumentofthe diswantpredicate.
Pickingup where we left off, the second world knowledgeaxio mwe use says that
if someone does not want someone else to drive, then they do no t want this person to
havehiscarkeys,sincecarkeysenablesomeonetodrive.
∀x,y,ej,ekdiswant (ej,y,ek)∧drive(ek,x)⇒
∃z,el,emdiswant (el,y,em)∧have(em,x,z)
∧carkeys (z,x)∧cause(ej,el)(21.74)
The third axiom says that if someone doesn’t want someone els e to have something,

hemighthideit fromhim.
∀x,y,z,el,emdiswant (el,y,em)∧have(em,x,z)⇒
∃enhide(en,y,x,z)∧cause(el,en)(21.75)
The ﬁnal axiom says simply that causality is transitive, tha t is, ifeicausesejandej
causesek,theneicausesek.
∀ei,ej,ekcause(ei,ej)∧cause(ej,ek)⇒cause(ei,ek) (21.76)
Finally, we have the content of the utterances themselves, t hat is, that John hid Bill’s
carkeys(fromBill),
(21.77)hide(e1,John,Bill,ck)∧carkeys (ck,Bill)
and that someone described using the pronoun “he” was drunk; we will represent the
pronounwith thefreevariable he.

(21.78)drunk(e2,he)
We can now see how reasoning with the content of the utterance s along with the
aforementionedaxiomsallowsthecoherenceofpassage(21. 4)tobeestablishedunder

DRAFT728 Chapter 21. ComputationalDiscourse
theExplanationrelation. ThederivationissummarizedinF ig.21.9;thesentenceinter-
pretationsare shownin boxes. We start byassumingthere isa coherencerelation,and
usingaxiom(21.71)hypothesizethatthisrelationisExpla nation,
(21.79)Explanation (e1,e2)
which,byaxiom(21.72),meanswe hypothesizethat
(21.80)cause(e2,e1)
holds. By axiom(21.76),we canhypothesizethat thereis ani ntermediatecause e3,
(21.81)cause(e2,e3)∧cause(e3,e1)
and we can repeat this again by expanding the ﬁrst conjunct of (21.81) to have an
intermediatecause e4.
(21.82)cause(e2,e4)∧cause(e4,e3)
We can take the hidepredicate from the interpretation of the ﬁrst sentence in (2 1.77)
and the second causepredicate in (21.81),and, using axiom (21.75),hypothesiz e that
Johndidnotwant Bill tohavehiscarkeys:
(21.83)diswant (e3,John,e5)∧have(e5,Bill,ck)
From this, the carkeyspredicate from (21.77), and the second causepredicate from
(21.82),wecanuseaxiom(21.74)tohypothesizethatJohndo esnotwantBilltodrive:

(21.84)diswant (e4,John,e6)∧drive(e6,Bill)
Fromthis,axiom(21.73),andthesecond causepredicatefrom(21.82),wecanhypoth-
esize thatBill wasdrunk:
(21.85)drunk(e2,Bill)
But now we ﬁnd that we can “prove” this fact from the interpret ation of the second
sentence if we simply assume that the free variable heis bound to Bill. Thus, the
establishmentofcoherencehasgonethrough,aswehaveiden tiﬁedachainofreasoning
betweenthesentenceinterpretations–onethatincludesun provableassumptionsabout

axiom choice and pronoun assignment – that results in cause(e2,e1), as required for
establishingthe Explanationrelationship.
Thisderivationillustratesa powerfulpropertyof coheren ceestablishment,namely
its ability to cause the hearer to infer information about th e situation described by the
discourse that the speaker has left unsaid. In this case, the derivation required the
assumptionthatJohnhidBill’skeysbecausehedidnotwanth imtodrive(presumably
out of fear of him having an accident, or getting stopped by th e police), as opposed

to some other explanation, such as playing a practical joke o n him. This cause is not
stated anywhere in passage (21.4); it arises only from the in ference process triggered
bythe need to establish coherence. Inthis sense, the meanin gof a discourseis greater
than the sum of the meaningsof its parts. That is, a discourse typically communicates
farmoreinformationthaniscontainedintheinterpretatio nsoftheindividualsentences
thatcompriseit.
We now return to passage (21.5),repeated below as (21.87),w hich was notable in

thatit lacksthecoherencedisplayedbypassage(21.4),rep eatedbelowas(21.86).
(21.86)JohnhidBill’s carkeys. Hewasdrunk.

DRAFTSection21.9. Advanced: Inference-BasedCoherenceResolu tion 729
CoherenceRel(e 1,e2)
Explanation(e 1,e2)
cause(e 2,e1)
cause(e 2,e3) cause(e 3,e1) hide(e 1,john,bill,ck)
cause(e 4,e3) diswant(e 3,j,e5)∧have(e5,bill,ck) carkeys(ck,bill)
cause(e 2,e4) diswant(e 4,y,e6)∧drive(e6,he)
drunk(e 2,bill) (he=bill)
Figure21.9 Establishingthe coherence of passage (21.4).
(21.87)?? JohnhidBill’scarkeys. He likesspinach.
Wecannowseewhythisis: thereisnoanalogouschainofinfer encecapableoflinking
the two utterance representations, in particular, there is no causal axiom analogous to
(21.73) that says that liking spinach might cause someone to not want you to drive.
Without additionalinformationthat can supportsuch a chai n of inference(such as the
aforementioned scenario in which someone promised John spi nach in exchange for
hidingBill’s carkeys),thecoherenceofthe passagecannot beestablished.
Because abduction is a form of unsound inference, it must be p ossible to subse-

quently retract the assumptions made during abductive reas oning, that is, abductive
inferences are defeasible . For instance, if passage (21.86) was followed by sentence Defeasible
(21.88),
(21.88)Bill’scarisn’thereanyway;Johnwasjustplayinga practic aljokeonhim.
the system would have to retract the original chain of infere nce connecting the two
clauses in (21.86), and replace it with one utilizing the fac t that the hiding event was
partofapracticaljoke.
Inamoregeneralknowledgebasedesignedtosupportabroadr angeofinferences,

(21.88),
one would want axioms that are more general than those we used to establish the co-
herence of passage (21.86). For instance, consider axiom (2 1.74), which says that if
youdonotwantsomeonetodrive,thenyoudonotwantthemtoha vetheircarkeys. A
moregeneralformoftheaxiomwouldsaythatifyoudonotwant someonetoperform
an action, and an object enables them to perform that action, then you do not want
them to have the object. The fact that car keysenable someone to drivewouldthen be
encodedseparately,alongwithmanyothersimilarfacts. Li kewise,axiom(21.73)says

that if someoneis drunk,you don’twant them to drive. We migh t replacethis with an
axiom that says that if someone does not want something to hap pen, then they don’t
want something that will likely cause it to happen. Again, th e facts that people typi-
cally don’t want other people to get into car accidents, and t hat drunk driving causes
accidents,wouldbeencodedseparately.
Whileitisimportanttohavecomputationalmodelsthatshed lightonthecoherence
establishment problem, large barriers remain for employin g this and similar methods

on a wide-coverage basis. In particular, the large number of axioms that would be

DRAFT730 Chapter 21. ComputationalDiscourse
required to encode all of the necessary facts about the world , and the lack of a robust
mechanism for constraining inference with such a large set o f axioms, makes these
methodslargelyimpracticalinpractice. Nonetheless,app roximationstothesekindsof
knowledgeandinferentialrulescanalreadyplayan importa ntrolein naturallanguage
understandingsystems.
21.10 PsycholinguisticStudiesofReferenceandCoherence
To what extent do the techniques described in this chapter mo del human discourse
comprehension? We summarize here a few selected results fro m the substantial body
of psycholinguistic research; for reasons of space we focus here solely on anaphora
resolution.
A signiﬁcant amount of work has been concernedwith the exten t to which people
usethepreferencesdescribedinSection21.5tointerpretp ronouns,theresultsofwhich
are often contradictory. Clark and Sengal (1979) studied th e effects that sentence re-

cencyplays in pronouninterpretationusing a set of reading time experiments . Afterreadingtimeexperiments
receiving and acknowledging a three sentence context to rea d, human subjects were
givenatargetsentencecontaininga pronoun. Thesubjectsp resseda buttonwhenthey
felt that they understood the target sentence. Clark and Sen gal found that the reading
time was signiﬁcantly faster when the referent for the prono un was evoked from the

most recent clause in the context than when it was evoked from two or three clauses
back. On the otherhand, there was no signiﬁcant differenceb etween referentsevoked
from two clauses and three clauses back, leading them to clai m that “the last clause
processedgrantstheentitiesitmentionsaprivilegedplac ein workingmemory”.
Crawley et al. (1990) compared the grammatical role paralle lism preference with
a grammatical role preference, in particular, a preference for referents evoked from
the subject position of the previous sentence over those evo ked from object position.

Unlikepreviousstudieswhichconﬂatedthesepreferencesb yconsideringonlysubject-
to-subjectreferenceeffects,Crawleyetal.studiedprono unsinobjectpositiontosee if
theytendedtobeassignedtothesubjectorobjectofthelast sentence. Theyfoundthat
intwotaskenvironments–a questionansweringtask whichrevealedhowthehumanquestionanswering
subjects interpreted the pronoun, and a referent naming task in which the subjectsReferentnaming
task
identiﬁed the referentof the pronoundirectly – the humansu bjectsresolvedpronouns

task
tothe subjectofthe previoussentencemoreoftenthantheob ject.
However, Smyth (1994) criticized the adequacy of Crawley et al.’s data for eval-
uating the role of parallelism. Using data that met more stri ngent requirements for
assessing parallelism, Smyth found that subjects overwhel mingly followed the paral-
lelism preference in a referent naming task. The experiment supplied weaker support
for the preference for subject referents over object refere nts, which he posited as a

defaultstrategywhenthe sentencesinquestionarenotsufﬁ cientlyparallel.
Caramazza et al. (1977) studied the effect of the “implicit c ausality” of verbs on
pronoun resolution. Verbs were categorized in terms of havi ng subject bias or object
bias using a sentence completion task . Subjects were given sentence fragmentssuchSentencecompletiontask
as(21.89).

DRAFTSection21.11. Summary 731
(21.89)JohntelephonedBill becausehe
The subjects provided completions to the sentences, which i dentiﬁed to the experi-
menterswhatreferentforthepronountheyfavored. Verbsfo rwhichalargepercentage
of human subjects indicated a grammatical subject or object preference were catego-
rized as having that bias. A sentence pair was then construct ed for each biased verb:
a“congruent”sentenceinwhichthesemanticssupportedthe pronounassignmentsug-
gested by the verb’s bias, and an “incongruent”sentence in w hich the semantics sup-
ported the opposite prediction. For example, sentence (21. 90) is congruent for the
subject-bias verb “telephoned”, since the semantics of the second clause supports as-
signingthesubject Johnastheantecedentof he,whereassentence(21.91)isincongru-
entsince thesemanticssupportsassigningthe object Bill.
(21.90)JohntelephonedBill becausehewantedsomeinformation.
(21.91)JohntelephonedBill becausehewithheldsomeinformation.

Inareferentnamingtask,Caramazzaetal.foundthatnaming timeswerefasterforthe
congruentsentencesthanfortheincongruentones. Perhaps surprisingly,thiswaseven
true for cases in which the two people mentioned in the ﬁrst cl ause were of different
genders,thusrenderingthereferenceunambiguous.
Matthewsand Chodorow(1988)analyzedthe problemof intras entential reference
and the predictions of syntactically-based search strateg ies. In a question answering

task, they found that subjects exhibited slower comprehens ion times for sentences in
which a pronoun antecedent occupied an early, syntacticall y deep position than for
sentencesinwhichtheantecedentoccupieda late,syntacti callyshallowposition. This
resultisconsistentwith thesearchprocessusedin Hobbs’s tree searchalgorithm.
Therehasalsobeenpsycholinguisticworkconcernedwithte stingtheprinciplesof
centeringtheory. In a set of readingtime experiments,Gord onet al. (1993)foundthat

reading times were slower when the current backward-lookin gcenter was referred to
usingafullnounphraseinsteadofapronoun,eventhoughthe pronounswereambigu-
ous and the propernames were not. This effect – which they cal led arepeated name
penalty– was found only for referents in subject position, suggesti ng that the Cbisrepeatedname
penalty
preferentially realized as a subject. Brennan (1995) analy zed how choice of linguis-
tic form correlates with centering principles. She ran a set of experiments in which

penalty
a human subject watched a basketball game and had to describe it to a second per-
son. She found that the human subjects tended to refer to an en tity using a full noun
phrase in subject position before subsequentlypronominal izingit, even if the referent
hadalreadybeenintroducedinobjectposition.
21.11 Summary
Inthischapter,wesawthatmanyoftheproblemsthatnatural languageprocessingsys-
temsfaceoperatebetweensentences,thatis, atthe discourse level. Hereisasummary
ofsomeofthemainpointswediscussed:
•Discourses, like sentences, have hierarchical structure. In the simplest kind of

structure detection, we assume a simpler linear structure, and segment a dis-

DRAFT732 Chapter 21. ComputationalDiscourse
courseontopicorotherboundaries. Themaincuesforthisar elexicalcohesion
aswell asdiscoursemarkers/cuephrases.
•Discourses are not arbitrary collections of sentences; the y must be coherent.
Among the factors that make a discourse coherent are coheren ce relations be-
tweenthesentencesandentity-basedcoherence.
•Varioussetsof coherencerelations andrhetoricalrelationshavebeenproposed.
Algorithms for detecting these coherence relations can use surface-based cues
(cuephrases,syntacticinformation).
•Discourse interpretation requires that one build an evolvi ng representation of
discourse state, called a discourse model , that contains representations of the
entitiesthathavebeenreferredtoandtherelationshipsin whichtheyparticipate.
•Natural languages offer many ways to refer to entities. Each form of reference
sendsitsownsignalstothehearerabouthowitshouldbeproc essedwithrespect
toherdiscoursemodelandset ofbeliefsabouttheworld.

•Pronominal reference can be used for referents that have an a dequate degree
ofsaliencein the discourse model. There are a variety of lexical, synta ctic,
semantic,anddiscoursefactorsthatappearto affectsalie nce.
•The Hobbs, Centering, and Log-linear models for pronominal anaphora offer
differentwaysofdrawingonandcombiningvariousofthese c onstraints.
•ThefullNPcoreferencetaskalsohastodealwithnamesandde ﬁniteNPs. String
editdistanceisausefulfeaturesforthese.
•Advanced algorithms for establishing coherence apply cons traints imposed by

oneormorecoherencerelations,oftenleadstothe inferenc eofadditionalinfor-
mationleft unsaidbythe speaker. Theunsoundrule oflogica labduction can be
usedforperformingsuchinference.
Building on the foundations set by early systems for natural language understanding
(Woodsetal.,1972a;Winograd,1972b;Woods,1978),muchof thefundamentalwork
in computational approaches to discourse was performed in t he late 70’s. Webber’s
(1978, 1983) work provided fundamental insights into how en tities are represented

in the discourse model and the ways in which they can license s ubsequent reference.
Many of the examplesshe providedcontinueto challenge theo riesof referenceto this
day. Grosz (1977a) addressed the focus of attention that con versational participants
maintain as the discourseunfolds. She deﬁned two levelsof f ocus; entities relevant to
the entirediscoursewere said tobe in globalfocus,whereasentitiesthatare locallyin
focus (i.e., most central to a particular utterance) were sa id to be in immediate focus.

Sidner (1979, 1983) described a method for tracking (immedi ate) discourse foci and
theiruseinresolvingpronounsanddemonstrativenounphra ses. Shemadeadistinction
between the current discourse focus and potential foci, whi ch are the predecessors to
thebackwardandforwardlookingcentersofcenteringtheor yrespectively.

DRAFTSection21.11. Summary 733
Therootsofthecenteringapproachoriginatefrompapersby JoshiandKuhn(1979)
and Joshi and Weinstein (1981), who addressed the relations hip between immediate
focus and the inferences required to integrate the current u tterance into the discourse
model. Grosz et al. (1983) integrated this work with the prio r work of Sidner and
Grosz. This led to a manuscript on centering which, while wid ely circulated since
1986,remainedunpublisheduntilGroszetal.(1995b). Aser iesofpapersoncentering
basedonthismanuscript/paperweresubsequentlypublishe d(Kameyama,1986;Bren-
nan et al., 1987; Di Eugenio,1990;Walker et al., 1994; Di Eug enio,1996; Strubeand
Hahn, 1996; Kehler, 1997a, inter alia). A collection of late r centering papers appears
in Walker et al. (1998), and see Poesio et al. (2004) for more r ecent work. We have
focused in this chapter on Centering and anaphora resolutio n; Sse Karamanis (2003,
2007), Barzilay and Lapata (2007) and related papers discus sed in Ch. 23 for the ap-

plicationofCenteringto entity-basedcoherence.
Thereisa longhistoryin linguisticsofstudiesof informationstatus (Chafe,1976;
Prince, 1981; Ariel, 1990; Prince, 1992; Gundel et al., 1993 ; Lambrecht, 1994, in-
teralia).
Beginning with Hobbs’s (1978) tree-search algorithm, rese archers have pursued
syntax-basedmethodsforidentifyingreferencerobustlyi nnaturallyoccurringtext. An
early system for a weighted combination of different syntac tic and other features was

teralia).
LappinandLeass(1994),whichwedescribedindetailinourﬁ rstedition. Kennedyand
Boguraev(1996)describeasimilarsystemthatdoesnotrely onafullsyntacticparser,
but merely a mechanism for identifying noun phrases and labe ling their grammatical
roles. Both approaches use Alshawi’s (1987) framework for i ntegrating salience fac-
tors. An algorithm that uses this framework for resolving re ferences in a multimodal
(i.e., speech and gesture)human-computerinterfaceis des cribedin Huls et al. (1995).

A discussion of a variety of approaches to reference in opera tional systems can be
foundin MitkovandBoguraev(1997).
Methodsforreferenceresolutionbasedonsupervisedlearn ingwereproposedquite
early (Connolly et al., 1994; Aone and Bennett, 1995; McCart hy and Lehnert, 1995;
Kehler, 1997b; Ge et al., 1998, inter alia). More recently bo th supervised and unsu-
pervisedapproacheshavereceivedalotofresearchattenti on,focusedbothonanaphora
resolutionKehleretal.(2004),BergsmaandLin(2006)andf ullNPcoreference(Cardie

andWagstaff,1999;NgandCardie,2002b;Ng,2005). Fordeﬁn iteNPreference,there
are general algorithms (Poesio and Vieira, 1998; Vieira and Poesio, 2000), as well as
speciﬁcalgorithmsthatfocusondecidingifaparticularde ﬁniteNPisanaphoricornot
(BeanandRiloff,1999,2004;NgandCardie,2002a;Ng,2004) .
SeeMitkov(2002)foranexcellentcomprehensiveoverviewo fanaphoraresolution
andBrancoet al.(2002)fora collectionofpapers.
The idea of using cohesion for linear discourse segmentatio n was implicit in the

groundbreaking work of (Halliday and Hasan, 1976), but was ﬁ rst explicitly imple-
mentedbyMorrisandHirst (1991),and quicklypickedup byma nyotherresearchers,
including(Kozima,1993;Reynar,1994;Hearst, 1994,1997; Reynar,1999;Kanet al.,
1998; Choi, 2000; Choi et al., 2001; Brants et al., 2002; Best gen, 2006). Power et al.
(2003) studies discourse structure, while Filippova and St rube (2006), Sporleder and
Lapata(2004,2006)focusonparagraphsegmentation.
Theuseofcuephrasesinsegmentationhasbeenwidelystudie d,includingworkon

DRAFT734 Chapter 21. ComputationalDiscourse
manytextualgenresas well asspeech (PassonneauandLitman , 1993;Hirschbergand
Litman,1993;Manning,1998;Kawaharaetal.,2004)
Many researchers have posited sets of coherence relations t hat can hold between
utterances in a discourse (Halliday and Hasan, 1976; Hobbs, 1979a; Longacre, 1983;
MannandThompson,1987;Polanyi,1988;Hobbs,1990;Sander setal.,1992;Carlson
et al., 2001, 2002; Asher and Lascarides, 2003; Baldridge et al., 2007, inter alia). A
compendium of over 350 relations that have been proposed in t he literature can be
foundin Hovy(1990).
There are a wide variety of approaches to coherence extracti on. The cue-phrase
based model described in Sec. 21.2.2 is due to Daniel Marcu an d colleagues (Marcu,
2000b, 2000a; Carlson et al., 2001, 2002). The Linguistic Di scourse Model (Polanyi,
1988; Scha and Polanyi, 1988; Polanyi et al., 2004a, 2004b) i s a framework in which
discourse syntax is more heavily emphasized; in this approa ch, a discourse parse tree

is built on a clause-by-clausebasis in direct analogywith h ow a sentence parse tree is
built on a constituent-by-constituent basis. Corston-Oli ver (1998) also explores syn-
tactic and parser-based features. A more recent line of work has applied a version of
thetree-adjoininggrammarformalismtodiscourseparsing (Webberetal.,1999;Web-
ber, 2004). This model has also been used to annotate the Penn Discourse Treebank
(Miltsakakietal.,2004b,2004a). SeeAsherandLascarides (2003)andBaldridgeetal.
(2007) on Segmented Discourse Representation Structure ( SDRT). Wolf and Gibson SDRT

(2005) argue that coherence structure includes crossed bra cketings which make it im-
possibletorepresentasa tree,andproposea graphrepresen tationinstead.
In addition to determining discourse structure and meaning , theories of discourse
coherencehavebeenusedinalgorithmsforinterpretingdis course-levellinguisticphe-
nomena, including pronoun resolution (Hobbs, 1979a; Kehle r, 2000), verb phrase el-
lipsisandgapping(Pr¨ ust,1992;Asher,1993;Kehler,1993 ,1994a),andtenseinterpre-

tation (Lascarides and Asher, 1993; Kehler, 1994b, 2000). A n extensive investigation
into the relationship between coherence relations and disc ourse connectives can be
foundin KnottandDale (1994).
Exercises
21.1Early work in syntactic theory attempted to characterize ru les for pronominal-
izationthroughpurelysyntacticmeans. Arulewasproposed inwhichapronoun
was interpreted by deleting it from the syntactic structure of the sentence that
contains it, and replacing it with the syntactic representa tion of the antecedent
nounphrase.

Exercises
Explainwhythefollowingsentences(called“Bach-Peters” sentences)areprob-
lematicforsuchananalysis:
(21.92)Themanwhodeservesit getstheprizehewants.
(21.93)Thepilotwhoshotat it hittheMIGthat chasedhim.
What other types of reference discussed on pages 710–713 are problematic for
thistypeofanalysis?

DRAFTSection21.11. Summary 735
21.2Draw syntactic trees for example (21.66) on page 719 and appl y Hobbs’s tree
searchalgorithmtoit, showingeachstepin thesearch.
21.3Hobbs(1977)citesthefollowingexamplesfromhiscorpusas beingproblematic
forhistree-searchalgorithm:
(21.94)Thepositionsofpillarsinonehallweremarkedbyriverboul dersanda
shapedconvexcushionofbronzethathadservedastheir footings.
(21.95)Theywereat onceassignedanimportantplaceamongthescant y
remainswhichrecordthephysicaldevelopmentsofthehuman race
fromthetime ofits ﬁrst appearanceinAsia.
(21.96)Sitesat whichthecoarsegreypotteryoftheShangperiodhas been
discovereddonotextendfarbeyondthesouthernmostreacho fthe
Yellowriver,orwestward beyondits junctionwiththeWei.
(21.97)Thethin,hard,black-burnishedpottery,madeinshapesofa ngular
proﬁle,whicharchaeologistsconsiderastheclearesthall markofthe
LungShanculture,developedintheeast. Thesite fromwhich ittakes
itsnameisinShantung. It istracedtothenorth-eastasfarasLiao-ning
province.

province.
(21.98)He hadthe dutyofperformingthenationalsacriﬁcesto heave nand
earth: hisroleassourceofhonoursandmaterialrewardsfor services
renderedbyfeudallordsandministersiscommemoratedin th ousands
ofinscriptionsmadebytherecipientsonbronzevesselswhi chwere
eventuallydepositedin their graves.
In each case, identify the correct referent of the underline d pronoun and the
one that the algorithm will identify incorrectly. Discuss a ny factors that come
into play in determining the correct referent in each case, a nd what types of

informationmightbenecessaryto accountforthem.
21.4ImplementtheHobbsalgorithm. Test it ona sampleofthePenn TreeBank. You
will need to modify the algorithm to deal with differences be tween the Hobbs
andTreeBankgrammars.
21.5Considerthefollowingpassage,fromBrennanetal. (1987):
(21.99)BrennandrivesanAlfaRomeo.
Shedrivestoofast.
Friedmanracesheronweekends.
Shegoesto LagunaSeca.
Identify the referent that the BFP algorithm ﬁnds for the pro noun in the ﬁnal
sentence. Doyouagreewiththischoice,ordoyouﬁndtheexam pleambiguous?

Discusswhyintroducinga new nounphrasein subject positio n,with a pronom-
inalized reference in object position, might lead to an ambi guity for a subject
pronouninthe nextsentence. What preferencesarecompetin ghere?
21.6Considerpassages(21.100a-b),adaptedfromWinograd(197 2b).
(21.100) Thecitycouncildeniedthedemonstratorsa permitbecause

DRAFT736 Chapter 21. ComputationalDiscourse
What are the correct interpretations for the pronouns in eac h case? Sketch out
an analysisofeach in the interpretationas abductionframe work,in which these
referenceassignmentsare made as a by-productof establish ing the Explanation
relation.
21.7Select aneditorialcolumnfromyourfavoritenewspaper,an ddeterminethe dis-
coursestructurefora10-20sentenceportion. Whatproblem sdidyouencounter?
Were you helped by superﬁcial cues the speaker included (e.g ., discourse con-
nectives)inanyplaces?

Chapter22
Information Extraction
Iam the very model of amodern Major-General,
I’ve informationvegetable, animal,and mineral,
I know the kings of England, and I quote the ﬁghts historical
FromMarathon toWaterloo, inorder categorical...
Gilbertand Sullivan, Piratesof Penzance
Imagine that you are an analyst with an investment ﬁrm that tr acks airline stocks.
You’regiventhetaskofdeterminingtherelationship(ifan y)betweenairlineannounce-
mentsoffareincreasesandthebehavioroftheirstocksonth efollowingday. Historical
dataaboutstockpricesiseasytocomeby,butwhataboutthei nformationaboutairline
announcements? To do a reasonablejob on this task, you would need to knowat least
thenameoftheairline,thenatureoftheproposedfarehike, thedatesoftheannounce-

ment and possibly the response of other airlines. Fortunate ly,this informationresides
in archives of news articles reporting on airline’s actions , as in the following recent
example.
Citinghighfuelprices,UnitedAirlinessaidFridayithasi ncreasedfaresby$6per
round trip on ﬂights to some cities also served by lower-cost carriers. American
Airlines, a unit of AMR Corp., immediately matched the move, spokesman Tim
Wagner said. United, a unit of UAL Corp., said the increase to ok effect Thursday

example.
and applies to most routes where it competes against discoun t carriers, such as
ChicagotoDallas andDenver toSanFrancisco.
Of course,distillinginformationlike names, datesandamo untsfromnaturallyoc-
curring text is a non-trivial task. This chapter presents a s eries of techniques that can
be used to extract limited kinds of semantic content from tex t. This process of in-
formation extraction (IE) turns the unstructured information embedded in texts i ntoInformationextraction

structureddata. Moreconcretely,informationextraction isaneffectivewaytopopulate
thecontentsofarelationaldatabase. Oncetheinformation isencodedformally,wecan
apply all the capabilities provided by database systems, st atistical analysis packages
and other forms of decision support systems to address the pr oblems we’re trying to
solve.
As we proceedthroughthis chapter,we’ll see that robust sol utionsto IE problems
are actually clever combinationsof techniques we’ve seen e arlier in the book. In par-

solve.
ticular, the ﬁnite-state methods described in Chs. 2 and 3, t he probabilistic models
introducedin Chs. 4 through 6 and the syntactic chunkingmet hods from Ch. 13 form
the core of most current approaches to information extracti on. Before diving into the
detailsofhowthesetechniquesareapplied,let’squicklyi ntroducethemajorproblems
inIEandhowtheycanbeapproached.
The ﬁrst step in most IE tasks is to detect and classify all the proper names men-

tionedinatext—ataskgenerallyreferredtoas namedentityrecognition (NER).NotNamedentityrecognition

DRAFT738 Chapter 22. InformationExtraction
surprisingly,whatconstitutesapropernameandthepartic ularschemeusedtoclassify
themis application-speciﬁc. GenericNERsystems tendto fo cusonﬁndingthe names
of people, places and organizationsthat are mentioned in or dinary news texts; practi-
cal applicationshave also been built to detect everythingf romthe namesof genesand
proteins(Settles, 2005)to thenamesofcollegecourses(Mc Callum,2005).
Ourintroductoryexamplecontains13instancesofproperna mes,whichwe’llrefer
to asnamed entity mentions , which can be classiﬁed as either organizations,people,Namedentitymentions
places,timesoramounts.
Having located all of the mentions of named entities in a text , it is useful to link,
orcluster,these mentionsintosets that correspondto thee ntitiesbehindthe mentions.
Thisisthetask of referenceresolution ,whichweintroducedinCh.21,andisalsoan
importantcomponentin IE. Inour sampletext, we wouldlike t o knowthat the United

Airlinesmentionintheﬁrstsentenceandthe Unitedmentioninthethirdsentencerefer
to the same real world entity. This general reference resolu tion problem also includes
anaphoraresolutionas a sub-problem. In this case, determi ningthat the two uses of it
refertoUnitedAirlines andUnitedrespectively.
The task of relation detection and classiﬁcation is to ﬁnd and classify semanticRelationdetectionandclassiﬁcation
relationsamongtheentitiesdiscoveredinagiventext. Inm ostpracticalsettings,thefo-
cusofrelationdetectionisonsmallﬁxedsetsofbinaryrela tions. Genericrelationsthat

appear in standard system evaluationsinclude family, empl oyment, part-whole, mem-
bership, and geospatial relations. The relation detection and classiﬁcation task is the
one that most closely corresponds to the problem of populati ng a relational database.
Relation detectionamongentities is also closely related t o the problemof discovering
semanticrelationsamongwordsintroducedinCh. 20.
Oursampletextcontains3explicitmentionsofgenericrela tions:Unitedisapartof

UAL,AmericanAirlines isapartof AMRandTimWagner isanemployeeof American
Airlines. Domain-speciﬁcrelationsfromtheairlineindustrywould includethefactthat
UnitedservesChicago,Dallas,Denver andSanFrancisco .
Inadditionto knowingaboutthe entitiesin a text andtheirr elationto oneanother,
wemightliketoﬁndandclassifytheeventsinwhichtheentit iesareparticipating;this
istheproblemof eventdetectionandclassiﬁcation . Inoursampletext,thekeyeventsEventdetectionandclassiﬁcation
arethefareincreaseby Unitedandtheensuingincreaseby American . Inaddition,there

are severaleventsreportingthese maineventsas indicated bythe two usesof saidand
the use of cite. As with entity recognition, event detection brings with it the problem
of reference resolution; we need to ﬁgure out which of the man y event mentions in a
textrefertothesameevent. Inourrunningexample,theeven tsreferredtoas themove
andthe increase in the second and third sentences are the same as the increasein the
ﬁrst sentence.
Theproblemofﬁguringoutwhentheeventsinatexthappeneda ndhowtheyrelate

to each other in time raises the twin problems of temporal expression recognitionTemporalexpressionrecognitionandtemporal analysis . Temporal expression detection tells us that our sample tex t Temporal analysis
containsthetemporalexpressions FridayandThursday . Temporalexpressionsinclude
date expressions such as days of the week, months, holidays, etc., as well as relative
expressionsincluding phraseslike two days from now ornext year. They also include
expressionsforclocktimessuchas noonor3:30PM.

The overall problem of temporal analysis is to map temporal expressions onto

DRAFTSection22.1. NamedEntityRecognition 739
speciﬁc calendar dates or times of day and then to use those ti mes to situate events in
time. It includesthefollowingsubtasks.
•Fixing the temporal expressionswith respect to an anchorin gdate or time, typi-
callythedatelineofthe storyin thecase ofnewsstories;
•Associatingtemporalexpressionswiththe eventsin thetex t;
•Arrangingtheeventsintoa completeandcoherenttimeline.
In our sample text, the temporal expressions FridayandThursday should be an-
chored with respect to the dateline associated with the arti cle itself. We also know
thatFridayrefers to the time of United’s announcement, and Thursday refers to the
time that the fare increase went into effect (i.e. the Thursd ay immediately preceding
the Friday). Finally, we can use this informationto produce a timeline whereUnited’s
announcement follows the fare increase and American’s anno uncement follows both
ofthoseevents. Temporalanalysisofthiskindisusefulinn earlyanyNLPapplication

that deals with meaning, including question answering, sum marization and dialogue
systems.
Finally,manytextsdescribestereotypicalsituationstha trecurwithsomefrequency
in the domainof interest. The task of template-ﬁlling is to ﬁnd documentsthat evoke Template-ﬁlling
suchsituationsandthenﬁlltheslotsintemplateswithappr opriatematerial. Theseslot-
ﬁllersmayconsistoftextsegmentsextracteddirectlyfrom thetext,ortheymayconsist
of conceptsthat have beeninferredfromtext elementsvia so me additionalprocessing

systems.
(times,amounts,entitiesfromanontology,etc.).
Our airline text is an example of this kind of stereotypical s ituation since airlines
are oftenattempting to raise faresand then waiting to see if competitorsfollow along.
In this situation, we can identify Unitedas a lead airline that initially raised its fares,
$6astheamountbywhichfaresarebeingraised, Thursday astheeffectivedateforthe
fare increase, and American as an airline that followed along. A ﬁlled template from

ouroriginalairlinestorymightlooklikethe following.
FARE-RAISEATTEMPT:
LEADAIRLINE: U NITEDAIRLINES
AMOUNT: $6
EFFECTIVE DATE: 2006-10-26
FOLLOWER : A MERICAN AIRLINES

The following sections will review current approaches to ea ch of these problems
in the context of generic news text. Section 22.5 then descri bes how many of these
problemsarisein thecontextofprocecessingbiologytexts .
22.1 NamedEntityRecognition
Thestartingpointformostinformationextractionapplica tionsisthedetectionandclas-

AMOUNT: $6

siﬁcation of the named entities in a text. By named entity , we simply mean anything Namedentity
that can be referredto with a propername. Thisprocessof named entity recognition

DRAFT740 Chapter 22. InformationExtraction
Type Tag Sample Categories
People PERIndividuals,ﬁctionalcharacters,smallgroups
Organization ORGCompanies,agencies,politicalparties,religiousgroups ,sportsteams
Location LOCPhysicalextents,mountains,lakes, seas
Geo-PoliticalEntity GPECountries,states, provinces,counties
Facility FACBridges,buildings,airports
Vehicles VEHPlanes,trainsandautomobiles
Figure22.1 A listof generic named entitytypes withthe kinds of entitie stheyrefer to.
Type Example
People Turingisoftenconsideredtobethefatherofmoderncomputerscien ce.
Organization The IPCCsaid it islikelythatfuturetropicalcycloneswill becomem oreintense.
Location The Mt. Sanitas loophikebeginsat thebaseof SunshineCanyon .
Geo-PoliticalEntity PaloAlto islookingatraisingthefeesforparkingintheUniversityA venuedistrict
Facility Drivers were advised to consider either the Tappan Zee Bridge or theLincoln
Tunnel.
Vehicles Theupdated MiniCooper retainsits charmandagility.
Figure22.2 Named entitytypes withexamples.

Tunnel.
refers to the combined task of ﬁnding spans of text that const itute proper names and
thenclassifyingtheentitiesbeingreferredtoaccordingt otheirtype.
Generic news-oriented NER systems focus on the detection of things like people,
places, and organizations. Figures 22.1 and 22.2 provide li sts of typical named entity
types with examples of each. Specialized applications may b e concerned with many
other types of entities, including commercial products, we apons, works of art, or as
we’ll see in Sec. 22.5, proteins, genes and other biological entities. What these ap-

plications all share is a concern with proper names, the char acteristic ways that such
namesaresignaledinagivenlanguageorgenre,andaﬁxedset ofcategoriesofentities
froma domainofinterest.
By the way that names are signaled, we simply mean that names a re denoted in a
waythatsetsthemapartfromordinarytext. Forexample,ifw e’redealingwithstandard
English text, then two adjacent capitalized words in the mid dle of a text are likely to
constitutea name. Further,iftheyareprecededbya Dr.orfollowedbyan MD, thenit

islikelythatwe’redealingwithaperson. Incontrast,ifth eyareprecededby arrivedin
orfollowedby NYthenwe’reprobablydealingwithalocation. Notethatthese signals
includefactsaboutthepropernamesaswellastheirsurroun dingcontexts.
The notion of a named entity is commonly extended to include t hings that aren’t
entities per se, but nevertheless have practical importanc e and do have characteristic
signaturesthat signal their presence;examplesincludeda tes, times, namedeventsand

other kinds of temporal expressions , as well as measurements, counts, prices andTemporalexpressions
otherkindsof numericalexpressions . We’ll considersomeoftheselaterinSec.22.3.Numericalexpressions
Let’srevisitthesampletextintroducedearlierwiththena medentitiesmarked(with
TIMEandMONEYusedtoto markthetemporalandmonetaryexpressions).
Citinghigh fuel prices, [ ORGUnited Airlines] said [ TIMEFriday] it has increased
fares by [ MONEY$6] per round tripon ﬂights tosome cities alsoserved by lowe r-

cost carriers. [ ORGAmerican Airlines],aunit of [ ORGAMR Corp.],immediately

DRAFTSection22.1. NamedEntityRecognition 741
Name Possible Categories
Washington Person,Location,PoliticalEntity,Organization,Facili ty
DowningSt. Location,Organization
IRA Person,Organization,MonetaryInstrument
LouisVuitton Person,Organization,CommercialProduct
Figure22.3 Common categorical ambiguities associatedwithvarious pr oper names.
[PERSWashington]wasbornintoslaveryonthe farmofJamesBurrou ghs.
[ORGWashington]wentup2gamesto 1inthefour-gameseries.
Blairarrivedin[ LOCWashington]forwhatmaywell behislast state visit.
InJune,[ GPEWashington]passeda primaryseatbelt law.
The[FACWashington]hadprovedto bealeakyship,everypassageI mad e...
Figure22.4 Examples of type ambiguities inthe use of the name Washington .
matched the move, spokesman [ PERSTim Wagner] said. [ ORGUnited], a unit of
[ORGUAL Corp.], said the increase took effect [ TIMEThursday] and applies to
mostroutes where itcompetes againstdiscount carriers,su chas[LOCChicago] to
[LOCDallas] and[ LOCDenver] to[ LOCSanFrancisco].

As shown, this text contains 13 mentionsof named entities in cluding 5 organizations,
4locations,2times,1person,and1mentionofmoney. The5or ganizationalmentions
correspond to 4 unique organizations, since UnitedandUnited Airlines are distinct
mentionsthatrefertothesame entity.
22.1.1 AmbiguityinNamed Entity Recognition
Named entity recognition systems face two types of ambiguit y. The ﬁrst arises from
the fact the same name can refer to different entities of the s ame type. For example,

JFKcanrefertotheformerpresidentorhisson. Thisisbasicall yareferenceresolution
problemandapproachesto resolvingthiskindofambiguitya rediscussedinCh. 21.
The second source of ambiguity arises from the fact that iden tical named entity
mentions can refer to entities of completely different type s. For example, in addition
to people, JFKmight refer to the airport in New York, or to any number of scho ols,
bridgesandstreetsaroundtheUnitedStates. Someexamples ofthiskindofcross-type
confusionaregivenin Figures22.3and22.4.

NoticethatsomeoftheambiguitiesshowninFig.22.3arecom pletelycoincidental.
Thereisnorelationshipbetweentheﬁnancialandorganizat ionalusesofthename IRA
— they simply arose coincidentally as acronyms from differe nt sources ( Individual
Retirement Account andInternational Reading Association ). On the other hand, the
organizational uses of Washington andDowning St. are examples of a LOCATION -
FOR-ORGANIZATION metonymy ,asdiscussedinCh. 19.
22.1.2 NER asSequence Labeling
Thestandardwaytoapproachtheproblemofnamedentityreco gnitionisasaword-by-

word sequence labeling task, where the assigned tags captur e both the boundary and
thetypeofanydetectednamedentities. Viewedinthislight ,namedentityrecognition

DRAFT742 Chapter 22. InformationExtraction
looksverymuchlike theproblemofsyntacticbase-phrasech unking. Infact,thedom-
inant approach to NER is based on the same statistical sequen ce labeling techniques
introducedinCh. 5forpartofspeechtaggingandCh. 13forsy ntacticchunking.
InthesequencelabelingapproachtoNER,classiﬁersaretra inedtolabelthetokens
in a text with tags that indicate the presence of particular k inds of named entities.
This approach makes use of the same style of IOBencoding employed for syntactic
chunking. Recall that in this scheme an Iis used to label tokens insideof a chunk, B
is used to mark the beginning of a chunk, and Olabels tokens outside any chunk of
interest. Considerthe followingsentencefromourrunning example.
(22.1)[ORGAmericanAirlines],a unitof[ ORGAMRCorp.],immediatelymatched
themove,spokesman[ PERSTimWagner]said.
This bracketing notation provides us with the extent and the type of the named
entities in this text. Fig. 22.5 shows a standard word-by-wo rd IOB-style tagging that

captures the same information. As with syntactic chunking, the tagset for such an
encodingconsists of 2 tags for each entity type being recogn ized, plus 1 for the O tag
outsideanyentity,or (2×N)+1tags.
Words Label
American BORG
Airlines IORG
, O
a O
unit O
of O
AMR BORG
Corp. IORG
, O
immediately O
matched O
the O
move O
, O
spokesman O
Tim BPERS
Wagner IPERS
said O
. O
Figure 22.5 IOBencoding.Having encoded our training data with IOBtags, the
next step is to select a set of features to associate with

, O
a O
unit O
of O
AMR BORG
Corp. IORG
, O
matched O
the O
move O
, O
Tim BPERS
said O
. O
each input example(i.e. each of the tokensto be labeled
in Fig. 22.5). These features should be plausible predic-
tors of the class label and should be easily and reliably
extractablefromthesourcetext. Recallthatsuchfeatures
canbebasednotonlyoncharacteristicsofthetokentobe
classiﬁed, but also on the text in a surrounding window
aswell.
Fig.22.6givesalistofstandardfeaturesemployedin
state-of-the-artnamedentityrecognitionsystems. We’ve
seenmanyofthesefeaturesbeforeinthecontextofpart-

aswell.
of-speech tagging and syntactic base-phrase chunking.
Several, however, are particularly important in the con-
text of NER. The shape feature feature includes the Shape feature
usual upper case, lower case and capitalized forms, as
well as more elaborate patterns designed to capture ex-
pressions that make use of numbers ( A9), punctuation
(Yahoo!) and atypical case alternations ( eBay). It turns
out that this feature by itself accounts for a considerable
partofthesuccessofNERsystemsforEnglishnewstext.

And as we’ll see in Sec. 22.5, shape features are also
particularly important in recognizing names of proteins an d genes in biological texts.
Fig.22.7describessomecommonlyemployedshapefeatureva lues.
Thepresenceinanamedentitylist featurecanbeverypredictive. Extensivelists
of names for all mannerof things are available from both publ icly available and com-
mercial sources. Lists of place names, called gazetteers , contain millions of entries Gazetteers

forall manneroflocationsalongwith detailedgeographica l,geologicandpoliticalin-

DRAFTSection22.1. NamedEntityRecognition 743
Feature Explanation
Lexicalitems Thetokentobelabeled
Stemmedlexicalitems Stemmedversionofthetargettoken
Shape Theorthographicpatternofthetargetword
Characterafﬁxes Characterlevelafﬁxesofthe targetandsu rroundingwords
Partofspeech Partofspeechoftheword
Syntacticchunklabels Base phrasechunklabel
Gazetteerornamelist Presenceofthewordinoneormorename dentitylists
Predictivetoken(s) Presenceofpredictivewordsinsurrou ndingtext
Bag ofwords/BagofN-grams Wordsand/orN-gramsoccurringi nthe surroundingcontext.
Figure22.6 Features commonly used intrainingnamed entityrecognitio n systems.
Shape Example
Lower cummings
Capitalized Washington
All caps IRA
Mixedcase eBay
Capitalizedcharacterwithperiod H.
Endsindigit A9
Containshyphen H-P
Figure22.7 Selectedshape features.
formation.1The United States Census Bureau provides extensive lists of ﬁrst names
andsurnamesderivedfromitsdecadalcensusintheU.S.2Similarlistsofcorporations,

commercialproducts,andallmannerofthingsbiologicalan dmineralarealsoavailable
froma varietyofsources.
Thisfeatureistypicallyimplementedasabinaryvectorwit habitforeachavailable
kindof namelist. Unfortunately,such lists can be difﬁcult to create andmaintain,and
their usefulness varies considerably based on the named ent ity class. It appears that
gazetteerscanbe quiteeffective,while extensivelistsof personsandorganizationsare
notnearlyasbeneﬁcial(Mikheevet al.,1999).

Finally, features based on the presence of predictive words and N-grams in the
context window can also be very informative. When they are pr esent, preceding and
followingtitles,honoriﬁcs,andothermarkerssuchas Rev.,MDandInc.canaccurately
indicatetheclassofanentity. Unlikenamelistsandgazett eers,theselistsarerelatively
shortandstable overtimeandarethereforeeasyto developa ndmaintain.
Therelativeusefulnessofanyofthesefeatures,orcombina tionoffeatures,depends
to a great extent on the application, genre, media, language and text encoding. For

example,shape features,which arecritical for Englishnew swiretexts, are of little use
withmaterialstranscribedfromspokentextviaautomatics peechrecognition,materials
gleaned from informally edited sources such as blogs and dis cussion forums, and for
character-basedlanguageslikeChinesewherecaseinforma tionisn’tavailable. Theset
offeaturesgiveninFig.22.6shouldthereforebethoughtof asonlyastartingpointfor
1www.geonames.org
2www.census.gov

DRAFT744 Chapter 22. InformationExtraction
Features Label
American NNP B NPcap B ORG
Airlines NNPS I NPcap I ORG
, PUNC O punc O
a DT B NPlower O
unit NN I NPlower O
of IN B PPlower O
AMR NNP B NPupper B ORG
Corp. NNP I NPcappunc I ORG
immediately RB B ADVPlower O
matched VBD B VPlower O
the DT B NPlower O
move NN I NPlower O
spokesman NN B NPlower O
Tim NNP I NPcap B PER
Wagner NNP I NPcap I PER
said VBD B VPlower O
. PUNC O punc O
Figure22.8 Simple word-by-word feature encoding for NER.
anygivenapplication.
Onceanadequatesetoffeatureshasbeendeveloped,theyare extractedfromarep-
resentativetraining set and encodedin a form appropriatet o train a machine learning-
based sequence classiﬁer. A standard way of encoding these f eatures is to simply
augment our earlier IOBscheme with more columns. Fig. 22.8 illustrates the result o f
addingpart-of-speechtags,syntacticbase-phrasechunkt ags,andshapeinformationto
ourearlierexample.

Given such a training set, a sequential classiﬁer can be trai ned to label new sen-
tences. As with part-of-speech tagging and syntactic chunk ing, this problem can be
cast either as Markov-style optimization using HMMs or MEMM s as described in
Ch. 6, or as a multi-way classiﬁcation task deployed as a slid ing-window labeler as
described in Ch. 13. Figure Fig. 22.9 illustrates the operat ion of such a sequence la-
beler at the point where the token Corp.is next to be labeled. If we assume a context

windowthat includesthe 2 precedingand followingwords, th en the featuresavailable
to the classiﬁer are those shown in the boxed area. Fig. 22.10 summarizes the overall
sequencelabelingapproachto creatingaNERsystem.
22.1.3 Evaluating NamedEntity Recognition
The familiar metrics of recall,precision andF1measure introduced in Ch. 13 are
usedtoevaluateNERsystems. Recallthatrecallistheratio ofthenumberofcorrectly
labeled responses to the total that should have been labeled ; precision is the ratio of

the number of correctly labeled responses to the total label ed. The F-measure (van
Rijsbergen,1975)providesa way to combinethese two measur esinto a single metric.

DRAFTSection22.1. NamedEntityRecognition 745
IN NNPNNP RB VBD
unitof a...lowerB_PP
... AMR Corp. immediately matchedB_NP
upperI_NP
cap_puncB_ADVP
lowerB_VP
lowerOB_ORG ? ... ...
,PUNC
O
punc
Figure22.9 Named entity recognition as sequence labeling. The feature s available to the
classiﬁerduring trainingand classiﬁcationare those inth e boxed area.
Document
Document
DocumentRepresentative 
Document 
CollectionHuman 
Annotation
Feature 
Extraction and 
IOB Encoding
Train Classifiers to Perform 
Multiway Sequence 
Labeling (MEMMs, CRFs, 
SVMs, HMMs, etc.)Annotated 
Documents
Training 
DataNER System
Figure22.10 Basic steps in the statistical sequence labeling approach t o creating a named
entityrecognition system.
TheF-measureisdeﬁnedas:
β2P+R(22.2)

Documents
Training 
F1=2PR
P+R(22.3)
As with syntactic chunking,it is important to distinguish t he metrics used to mea-
sureperformanceattheapplicationlevelfromthoseuseddu ringtraining. Attheappli-

DRAFT746 Chapter 22. InformationExtraction
cationlevel,recallandprecisionaremeasuredwithrespec ttotheactualnamedentities
detected. Onthe otherhand,withan IOBencodingschemethelearningalgorithmsare
attempting to optimize performance at the tag level. Perfor mance at these two levels
can be quite different; since the vast majority of tags in any giventext are outside any
entity,simplyemittingan Otagforeverytokengivesfairlyhightag-levelperformance .
High-performing systems at recent standardized evaluatio ns have entity level F-
measures around .92 for PERSONS andLOCATIONS , and around .84 for ORGANIZA -
TIONS(SangandDeMeulder,2003).
22.1.4 PracticalNER Architectures
Commercial approaches to NER are often based on pragmatic co mbinations of lists,
rules and supervised machine learning(Jackson and Moulini er, 2002). One common
approach is to make repeated passes over a text allowing the r esults of one pass to
inﬂuencethenext. Thestagestypicallyﬁrstinvolvetheuse ofrulesthathaveextremely

high precision but low recall. Subsequent stages employ mor e error-prone statistical
methodsthattake theoutputoftheﬁrst passintoaccount.
1. First usehigh-precisionrulestotagunambiguousentity mentions;
2. Thensearchforsub-stringmatchesofthepreviouslydete ctednamesusingprob-
abilisticstringmatchingmetrics(asdescribedin Ch.19).
3. Consult application-speciﬁc name lists to identify like ly name entity mentions
fromthegivendomain.
4. Finally, apply probabilistic sequence labeling techniq ues that make use of the
tagsfrompreviousstagesasadditionalfeatures.

The intuition behind this staged approach is two-fold. Firs t, some of the entity
mentions in a text will be more clearly indicative of a given e ntity’s class than others.
Second, once an unambiguousentity mention is introducedin to a text, it is likely that
subsequentshortened versionswill refer to the same entity (and thus the same type of
entity).
22.2 RelationDetectionandClassiﬁcation
Next on our list of tasks is the ability to discern the relatio nshipsthat exist among the

entity).
entitiesdetectedinatext. Toseewhatthismeans,let’sret urntooursampleairlinetext
withall theentitiesmarked.

This text stipulates a set of relations among the named entit ies mentioned within
it. We know, for example, that Tim Wagner is a spokesman for American Airlines ,

DRAFTSection22.2. RelationDetectionandClassiﬁcation 747
Relations Examples Types
Afﬁliations
Personal marriedto ,motherof PER→PER
Organizational spokesmanfor ,presidentof PER→ORG
Artifactual owns,invented,produces (PER|ORG)→ART
Geospatial
Proximity near,onoutskirts LOC→LOC
Directional southeastof LOC→LOC
Part-Of
Organizational aunitof,parentof ORG→ORG
Political annexed,acquired GPE→GPE
Figure22.11 Semantic relations withexamples andthe named entitytypes they involve.
thatUnitedis a unit of UAL Corp. , and that American is a unit of AMR. These are all
binaryrelationsthatcanbeseenasinstancesofmoregeneri crelationssuchas part-of
oremploysthatoccurwithfairlyhighfrequencyin news-styletexts. F ig.22.11shows
a list of generic relations of the kind used in recent standar dized evaluations.3More
domain-speciﬁcrelationsthatmightbeextractedincludet henotionofanairlineroute.
Forexample,fromthistextwecanconcludethatUnitedhasro utestoChicago,Dallas,
DenverandSanFrancisco.

These relations correspondnicely to the model-theoreticn otions we introduced in
Ch. 17 to ground the meanings of the logical forms. That is, a r elation consists of set
of ordered tuples over elements of a domain. In most standard information extraction
applications,thedomainelementscorrespondeithertothe namedentitiesthatoccurin
thetext,totheunderlyingentitiesthatresultfromco-ref erenceresolution,ortoentities
selected from a domain ontology. Fig. 22.12 shows a model-ba sed view of the set

of entities and relations that can be extracted from our runn ing example. Notice how
this model-theoretic view subsumes the NER task as well; nam ed entity recognition
correspondstotheidentiﬁcationofa classofunaryrelatio ns.
22.2.1 Supervised Learning Approaches to RelationAnalysi s
Supervisedmachinelearningapproachestorelationdetect ionandclassiﬁcationfollow
a scheme that should be familiar by now. Texts are annotated w ith relations chosen

from a small ﬁxed set by humananalysts. These annotatedtext s are then used to train
systems to reproduce similar annotations on unseen texts. S uch annotations indicate
thetextspansofthetwo arguments,the rolesplayedbyeacha rgumentandthetypeof
therelationinvolved.
The most straightforward approach breaks the problem down i nto two sub-tasks:
detecting when a relation is present between two entities an d then classifying any de-
tected relations. In the ﬁrst stage, a classiﬁer is trained t o make a binary decision as

to whether or not a given pair of named entities participate i n a relation. Positive ex-
amples are extracted directly from the annotated corpus, wh ile negative examples are
generatedfromwithin-sentenceentitypairsthatarenotan notatedwitha relation.
3http://www.nist.gov/speech/tests/ace/

DRAFT748 Chapter 22. InformationExtraction
Domain D={a,b,c,d,e,f,g,h,i}
United,UAL,AmericanAirlines,AMR a,b,c,d
TimWagner e
Chicago,Dallas, Denver,andSanFrancisco f,g,h,i
Classes
United,UAL,AmericanandAMRareorganizations Org={a,b,c,d}
TimWagnerisa person Pers={e}
Chicago,Dallas, DenverandSanFranciscoareplaces Loc={f,g,h,i}
Relations
Unitedisa unitofUAL PartOf ={⟨a,b⟩,⟨c,d⟩}
Americanisa unitofAMR
TimWagnerworksforAmericanAirlines OrgAff ={⟨c,e⟩}
UnitedservesChicago,Dallas, DenverandSanFrancisco Serves ={⟨a,f⟩,⟨a,g⟩,⟨a,h⟩,⟨a,i⟩}
Figure22.12 A model-based view of the relations andentities inour sampl e text.
function FINDRELATIONS (words)returnsrelations
relations←nil
entities←FINDENTITIES(words)
forall entitypairs ⟨e1,e2⟩inentitiesdo
ifRELATED?(e1,e2)
relations←relations+CLASSIFYRELATION(e1,e2)
Figure22.13 Finding andclassifying the relations among entitiesina te xt.
In the second phase, a classiﬁer is trained to label the relat ions that exist between

candidate entity pairs. As discussed in Ch. 6, techniques su ch as decision trees, naive
BayesorMaxEnthandlemulticlasslabelingdirectly. Binar yapproachesbasedondis-
coveringseparating hyperplanessuch as SVMs solve multicl ass problemsby employ-
ingaone-versus-alltrainingparadigm. Inthisapproach,a setsofclassiﬁersaretrained
whereeachclassiﬁeristrainedononelabelasthepositivec lassandalltheotherlabels
as the negative class. Final classiﬁcation is performed by p assing each instance to be
labeled to all of the classiﬁers and then choosing the label f rom the classiﬁer with the

mostconﬁdence,orreturningarankorderingoverthepositi velyrespondingclassiﬁers.
Fig.22.13illustratesthebasicapproachforﬁndingandcla ssifyingrelationsamongthe
namedentitieswithinadiscourseunit.
Aswithnamedentityrecognition,themostimportantstepin thisprocessistoiden-
tify surface features that will be useful for relation class iﬁcation (Zhou et al., 2005).
The ﬁrst source of information to consider are features of the named entities them-
selves.
•Namedentitytypesofthetwo candidatearguments
•Concatenationofthetwo entitytypes

selves.
•Headwordsofthearguments
•Bagofwordsfromeachofthearguments

DRAFTSection22.2. RelationDetectionandClassiﬁcation 749
NP
NP
NNP
AmericanNNPS
AirlinesPUNC
,NP
NP
DT
aNN
unitPP
IN
ofNP
NNP
AMRNNP
Inc.
Figure22.14 Anappositive construction expressing an a-part-of relation.
The next set of features are derived from the words in the text being examined.
It is useful to think of these features as being extracted fro m three locations: the text
between the two candidate arguments, a ﬁxed window before th e ﬁrst argument, and
a ﬁxedwindow after the secondargument. Given these locatio ns, the followingword-
basedfeatureshaveproventobeuseful.
•Thebagofwordsandbagofbigramsbetweentheentities
•Stemmedversionsofthesame
•Wordsandstemsimmediatelyprecedingandfollowingtheent ities
•Distanceinwordsbetweenthe arguments
•Numberofentitiesbetweenthearguments
Finally,the syntacticstructure of a sentencecan signalmanyofthe relationships
amonganyentitiescontainedwithinit. Thefollowingfeatu rescanbederivedfromvar-

ious levels of syntactic analysis including base-phrase ch unking, dependency parsing
andfullconstituentparsing.
•Presenceofparticularconstructionsina constituentstru cture
•Chunkbase-phrasepaths
•Bagsofchunkheads
•Dependency-treepaths
•Constituent-treepaths
•Treedistancebetweenthearguments
One methodof exploitingparse treesis to create detectorst hat signal the presence
of particular syntactic constructions and then associate b inary features with those de-
tectors. Asanexampleofthis,considerthesub-treeshowni nFig.22.14thatdominates

the named entities American andAMR Inc. TheNPconstructionthat dominatesthese
twoentitiesiscalledanappositiveconstructionandisoft enassociatedwithboth part-
ofanda-kind-of relations in English. A binary feature indicating the prese nce of this
constructioncanbeusefulindetectingthese relations.
This method of feature extraction relies on a certain amount of a priori linguistic
analysis to identify those syntactic constructionsthat ma y be useful predictorsof cer-

DRAFT750 Chapter 22. InformationExtraction
Entity-basedfeatures
Entity1type ORG
Entity1head airlines
Entity2type PERS
Entity2head Wagner
Concatenatedtypes ORGPERS
Word-basedfeatures
Between-entitybagofwords {a, unit, of, AMR, Inc., immediately, matched, the, move,
spokesman}
Word(s)beforeEntity 1 NONE
Word(s)afterEntity 2 said
Syntacticfeatures
Constituentpath NP↑NP↑S↑S↓NP
Base syntacticchunkpath NP→NP→PP→NP→VP→NP→NP
Typed-dependencypath Airlines←subjmatched←compsaid→subjWagner
Figure22.15 Sample of features extracted whileclassifying the <AmericanAirlines,Tim Wagner >tuple.
tain classes. An alternative method is to automatically enc ode certain aspects of tree
structures as feature values and allow the machine learning algorithms to determine
which values are informative for which classes. One simple a nd effective way to do
this this involves the use of syntactic paths through trees. Consider again the tree
discussed earlier that dominates American Airlines andAMR Inc. The syntactic rela-

tionship between these arguments can be characterized by th e path traversed through
thetreein gettingfromonetotheother:
NP↑NP↓NP↓PP↓NP
Similarpathfeaturesdeﬁnedoversyntacticdependencytre esaswellasﬂatbase-phrase
chunkstructureshavebeen shownto be useful forrelationde tectionand classiﬁcation
(Culotta and Sorensen, 2004; Bunescu and Mooney, 2005). Rec all that syntactic path
featuresfeaturedprominentlyinCh. 20in thecontextofsem anticrolelabeling.
Fig. 22.15 illustrates some of the features that would be ext racted while trying to

classifytherelationshipbetween AmericanAirlines andTimWagner fromourexample
text.
22.2.2 Lightly Supervised Approaches to RelationAnalysis
The supervisedmachine learning approachjust described as sumes that we have ready
accesstoalargecollectionofpreviouslyannotatedmateri alwithwhichtotrainclassi-
ﬁers. Unfortunately,thisassumptionisimpracticalinman yreal-worldsettings. Asim-
ple approach to extracting relational information without large amounts of annotated
material is to use regular expression patterns to match text segments that are likely to

text.
containexpressionsoftherelationsinwhichweare interes ted.
Consider the problem of building a table containing all the h ub cities that various
airlines utilize. Assuming we have access a search engine th at permits some form of
phrasalsearchwith wildcards,we mighttrysomethingliket hefollowingasaquery:

DRAFTSection22.2. RelationDetectionandClassiﬁcation 751
/*has a hub at */
Given access to a reasonable amount of material of the right k ind, such a search will
yieldafairnumberofcorrectanswers. ArecentGooglesearc husingthispatternyields
thefollowingrelevantsentencesamongthe returnset.
(22.4)Milwaukee-basedMidwesthasa hubat KCI.
(22.5)Delta hasa hubat LaGuardia.
(22.6)BulgariaAir hasa hubat SoﬁaAirport,asdoesHemusAir.
(22.7)AmericanAirlineshasahubatthe SanJuanairport.
Of course, patterns such as this can fail in the two ways we dis cussed all the way
backinCh.2: byﬁndingsomethingstheyshouldn’t,andbyfai lingto ﬁndthingsthey
should. As an example of the ﬁrst kind of error, consider the f ollowing sentences that
werealso includedin theearlierreturnset.
(22.8)airlinej hasa hubat airportk
(22.9)Thecatheterhasa hubatthe proximalend
(22.10)Astar topologyoftenhasahubatits center.
We can addressthese errorsby makingour proposedpattern mo respeciﬁc. In this

case, replacingthe unrestrictedwildcardoperatorwith a n amedentity class restriction
wouldrulethese examplesout:
/[ORG] has a hub at [LOC]/
The second problem is that we can’t know if we’ve found all the hubs for all air-
lines, since we’ve limited ourselves to this one rather spec iﬁc pattern. Consider the
followingclosecallsmissedbyourﬁrst pattern.
(22.11)NofrillsrivaleasyJet,whichhasestablisheda hubat Liver pool...
(22.12)Ryanairalso hasa continentalhubatCharleroiairport(Bel gium).

Theseexamplesare missedbecausetheycontainminorvariat ionsthat causethe origi-
nalpatterntofail. Therearetwowaystoaddressthisproble m. Theﬁrstistogeneralize
our pattern to capture expressionslike these that contain t he informationwe are seek-
ing. Thiscan be accomplishedby relaxingthe patternto allo w matchesthat skip parts
of the candidate text. Of course, this approach is likely int roduce more of the false
positives that we tried to eliminate by making our pattern mo re speciﬁc in the ﬁrst
place.

place.
Thesecond,morepromisingsolution,istoexpandoursetofs peciﬁchigh-precision
patterns. Given a large and diverse document collection, an expanded set of patterns
should be able to capture more of the information we’re looki ng for. One way to
acquire these additional patterns is to simply have human an alysts familiar with the
domaincomeupwithmorepatternsandhopetogetbettercover age. Amoreinteresting
automaticalternativeistoinducenewpatternsby bootstrapping fromtheinitialsearch Bootstrapping

place.
resultsfromasmall set of seed patterns . Seedpatterns
To see how this works, let’s assume that we’ve discovered tha t Ryanair has a hub
at Charleroi. We can use this fact to discover new patterns by ﬁnding other mentions
of this relation in our corpus. The simplest way to do this is t o search for the terms
Ryanair,Charleroi andhubin some proximity. The following are among the results
froma recentsearchinGoogleNews.

DRAFT752 Chapter 22. InformationExtraction
Pattern-Based Relation ExtractionSeed 
Patterns
Seed 
TuplesTuple 
ExtractionPattern 
Search Pattern 
ExtractionTuple 
Search
Relational
TableTuple
Set
Pattern
Set
Figure22.16 Patternandbootstrapping-based relationextraction.
(22.13)BudgetairlineRyanair,whichusesCharleroiasahub,scrap pedall weekend
ﬂightsoutoftheairport.
(22.14)AllﬂightsinandoutofRyanair’sBelgianhubat Charleroiai rportwere
groundedonFriday...
(22.15)Aspokesmanat Charleroi,a mainhubforRyanair,estimatedt hat 8000
passengershadalreadybeenaffected.
From these results, patterns such as the following can be ext racted that look for
relevantnamedentitiesofvarioustypesintherightplaces .
/ [ORG], which uses [LOC] as a hub /
/ [ORG]’s hub at [LOC] /
/ [LOC] a main hub for [ORG] /
Thesenewpatternscanthenbeusedto searchforadditionalt uples.
Fig.22.16illustratestheoverallbootstrappingapproach . Thisﬁgureshowsthatthe
dual nature of patterns and seeds permits the processto star t with either a small set of

seed tuples or a set of seed patterns . This style of bootstrapping and pattern-based
relationextractioniscloselyrelatedtothetechniquesdi scussedinCh.20forextracting
hyponymandmeronym-basedlexicalrelations.
Thereare,ofcourse,afairnumberoftechnicaldetailstobe workedouttoactually
implementsuchanapproach. Thefollowingareamongsomeoft hekeyproblems.
•Representingthe searchpatterns

DRAFTSection22.2. RelationDetectionandClassiﬁcation 753
•Assessingthe accuracyandcoverageofdiscoveredpatterns
•Andassessing thereliabilityofthediscoveredtuples
Patternsaretypicallyrepresentedin away thatcapturesth efollowingfourfactors.
•Contextpriortothe ﬁrst entitymention
•Contextbetweenthe entitymentions
•Contextfollowingthesecondmention
•Theorderoftheargumentsinthepattern
Contexts are either captured as regular expression pattern s or as vectors of features
similar to those described earlier for machine learning-ba sed approaches. In either
case, they can be deﬁned over character strings, word-level tokens, or syntactic and
semanticstructures. Ingeneral,regularexpressionappro achestendtobeveryspeciﬁc,
yielding high precision results; feature-basedapproache s,on the other hand, are more
capableofignoringpotentiallyinconsequentialelements ofcontexts.
Our next problemis how to assess the reliability of newly dis coveredpatternsand

tuples. Recall that we don’t, in general, have access to anno tated materials giving us
the right answers. We therefore have to rely on the accuracy o f the initial seed sets
of patterns and/or tuples for gold-standard evaluation, an d we have to ensure that we
don’tpermitanysigniﬁcant semanticdrift tooccuraswe’relearningnewpatternsand Semanticdrift
tuples. Semantic drift occurs when an erroneous pattern lea ds to the introduction of
erroneoustuples,whichcanthen,turn,leadto thecreation ofproblematicpatterns.

Tosee thisconsiderthe followingexample.
(22.16)SydneyhasaferryhubatCircularQuay.
If accepted as a positive example, this expression could lea d to the introduction of
the tuple⟨Sydney ,CircularQuay⟩. Patternsbasedonthistuplecouldpropagatefurther
errorsintothe database.
Therearetwofactorsthatneedtobebalancedinassessingap roposednewpattern:
the pattern’s performance with respect to the current set of tuples, and the pattern’s
productivityintermsofthenumberofmatchesitproducesin thedocumentcollection.
Moreformally,givenadocumentcollection D,acurrentsetoftuples T,andaproposed

patternp, therearethreefactorsthatwe needto track.
•hits: theset oftuplesin Tthatpmatcheswhilelookingin D;
•misses: Theset oftuplesin Tthatpmisseswhilelookingat D;
•finds: Thetotalset oftuplesthat pﬁndsin D.
Thefollowingequationbalancesthese considerations(Ril offandJones,1999).
ConfRlogF(p) =hitsp
hitsp+missesp×log(ﬁndsp) (22.17)
It isusefulto be ableto treatthismetric asa probability,s o we’ll needtonormalizeit.
A simple way to do this is to track the range of conﬁdences in a d evelopmentset and

divide by some previously observed maximum conﬁdence (Agic htein and Gravano,
2000).
We can assess the conﬁdence in a proposed new tuple by combini ng the evidence
supportingitfromallthepatterns P′thatmatchthattuplein D(AgichteinandGravano,

DRAFT754 Chapter 22. InformationExtraction
2000). One way to combine such evidence is the noisy-or technique. Assume that a Noisy-or
given tuple is supportedby a subset of the patternsin P, each with its own conﬁdence
assessed as above. In the noisy-or model, we make two basic as sumptions. First,
that for a proposed tuple to be false, allof its supporting patterns must have been in
error, and second that the sources of their individual failu res are all independent. If
we loosely treat our conﬁdence measures as probabilities, t hen the probability of any
individual pattern pfailing is 1−Conf(p); the probability of all of the supporting
patternsfor a tuple beingwrong is the productof their indiv idualfailure probabilities,
leavinguswiththe followingequationforourconﬁdencein a newtuple.
Conf(t) =1−∏
p∈P′1−Conf(p) (22.18)
The independence assumptions underlying the noisy-or mode l are very strong in-
deed. If the failure mode of the patterns are not independent , then the method will

overestimate the conﬁdence for the tuple. This overestimat e is typically compensated
forbysetting averyhighthresholdfortheacceptanceofnew tuples.
Giventhesemeasures,wecandynamicallyassessourconﬁden ceinbothnewtuples
and patterns as the bootstrapping process iterates. Settin g conservative thresholds for
theacceptanceofnewpatternsandtuplesshouldhelppreven tthesystemfromdrifting
fromthetargetedrelation.
Althoughthere have been no standardizedevaluationsfor th is style of relation ex-
traction on publicly available sources, the technique has g ained wide acceptance as a

practical way to quickly populate relational tables from op en source materials (most
commonlyfromtheWeb) (Etzioniet al.,2005).
22.2.3 Evaluating RelationAnalysisSystems
There are two separate methods for evaluating relation dete ction systems. In the ﬁrst
approach, the focus is on how well systems can ﬁnd and classif y all the relation men-
tions in a given text. In this approach, labeled and unlabele d recall, precision and F-
measures are used to evaluate systems against a test collect ion with human annotated

gold-standardrelations. Labeledprecisionandrecallreq uiresthesystemtoclassifythe
relationcorrectly,whileunlabeledmethodssimplymeasur easystem’sabilitytodetect
entitiesthatarerelated.
The second approach focuses on the tuples to be extracted fro m a body of text,
rather than on the relation mentions. In this method, system s need not detect every
mention of a relation to be scored correctly. Instead, the ﬁn al evaluation is based on
the set of tuplesoccupyingthe database when the system is ﬁn ished. That is, we want

toknowifthesystemcandiscoverthatRyanAirhasahubatCha rleroi;wedon’treally
carehowmanytimesit discoversit.
This method has typically used to evaluate unsupervised met hods of the kind dis-
cussed in the last section. In these evaluationshuman analy stssimply examinethe set
oftuplesproducedbythesystem. Precisionissimplythefra ctionofcorrecttuplesout
ofall thetuplesproducedasjudgedbythehumanexperts.
Recall remains a problem in this approach. It is obviously to o costly to search
by hand for all the relations that could have been extracted f rom a potentially large

DRAFTSection22.3. TemporalandEventProcessing 755
collection such as the Web. One solution is to compute recall at various levels of
precisionas describedin Ch. 23 (Etzioni et al., 2005). Of co urse, this isn’t true recall,
since we’re measuring against the number of correct tuples d iscovered rather than the
numberoftuplesthat aretheoreticallyextractablefromth e text.
Another possibility is to evaluate recall on problems where large resources con-
taining comprehensive lists of correct answers are availab le. Examples of include
gazetteers for facts about locations, the Internet Movie Da tabase (IMDB) for facts
about movies or Amazon for facts about books. The problem wit h this approach is
thatit measuresrecallagainstadatabasethatmaybefar mor ecomprehensivethanthe
textcollectionsusedbyrelationextractionsystem.
22.3 TemporalandEventProcessing
Ourfocusthusfarhasbeenonextractinginformationaboute ntitiesandtheirrelations
to one another. However, in most texts, entities are introdu ced in the course of de-

scribingthe eventsin which theytake part. Findingand anal yzingthe eventsin a text,
andhowtheyrelate to eachotherin time,is crucialto extrac tinga morecompletepic-
ture of the contents of a text. Such temporal information is p articularly important in
applicationssuchasquestionansweringandsummarization .
In question answering, whether or not a system detects a corr ect answer may de-
pend on temporal relations extracted from both the question and the potential answer

text. As an example of this, consider the following sample qu estion and potential an-
swer text.
Whendidairlinesasagrouplast raisefares?
Last week, Delta boosted thousands of fares by $10 per round t rip, and
mostbignetworkrivalsimmediatelymatchedtheincrease. ( Dateline7/2/2007).
Thissnippetdoesprovideananswertothequestion,butextr actingitrequirestemporal
reasoning to anchor the phrase last week , to link that time to the boosting event, and
ﬁnallytolinkthetime ofthe matching eventtothat.

swer text.
The followingsectionsintroduceapproachesto recognizin gtemporalexpressions,
ﬁguring out the times that those expressions refer to, detec ting events and associating
timeswiththoseevents.
22.3.1 Temporal Expression Recognition
Temporal expressions are those that refer to absolute point s in time, relative times,
durations and sets of these. Absolute temporal expressions are those that can beAbsolute temporalexpressions
mapped directly to calendar dates, times of day, or both. Relative temporal expres-

sionsmap to particular times via some other reference point (as in a week from lastRelativetemporalexpressions
Tuesday.) Finally, durations denotespansoftimeatvaryinglevelsofgranularity(sec- Durations
onds, minutes,days, weeks, centuriesetc.) Fig. 22.17prov idessome sample temporal
expressionsineachofthesecategories.
Syntactically, temporal expressions are syntactic constr uctions that have temporal

DRAFT756 Chapter 22. InformationExtraction
Absolute Relative Durations
April24,1916 yesterday fourhours
Thesummerof’77 nextsemester threeweeks
10:15AM twoweeksfromyesterday six days
The3rdquarterof2006 last quarter thelast threequarters
Figure22.17 Examples of absolute, relationand durational temporal exp ressions.
Category Examples
Noun morning,noon,night,winter,dusk,dawn
ProperNoun January,Monday,Ides,Easter,RoshHashana,Ramadan,Tet
Adjective recent,past,annual,former
Adverb hourly,daily,monthly,yearly
Figure22.18 Examples of temporal lexical triggers.
lexicaltriggers astheirheads. Intheannotationschemeinwidest use,lexic altriggers Lexicaltriggers
can be nouns, proper nouns, adjectives, and adverbs; full te mporal expression consist
of their phrasal projections: noun phrases, adjective phra ses and adverbial phrases.
Figure22.18providesexamplesoflexicaltriggersfromthe secategories.
The annotation scheme in widest use is derived from the TIDES standard(Ferro

etal.,2005). TheapproachpresentedhereisbasedontheTim eMLeffort(Pustejovsky
et al., 2005). TimeML provides an XML tag, TIMEX3, along with various attributes
to that tag, for annotating temporal expressions. The follo wing example illustrates
the basic use of this scheme (ignoringthe additional attrib utes, which we’ll discuss as
neededlaterin Sec.22.3.2).
A fare increase initiated <TIMEX3 >last week </TIMEX3 >by UAL
Corp’sUnitedAirlineswas matchedbycompetitorsover <TIMEX3 >the

weekend </TIMEX3 >, marking the second successful fare increase in
<TIMEX3 >twoweeks </TIMEX2 >.
The temporal expression recognition task consists of ﬁndin g the start and end of
all of the text spans that correspond to such temporal expres sions. Although there
are myriad ways to compose time expressions in English, the s et of temporal trigger
terms is, for all practical purposes, static and the set of co nstructionsused to generate
temporal phrases is quite conventionalized. These facts su ggest that any of the major

approaches to ﬁnding and classifying text spans that we’ve a lready studied should be
successful. The following three approaches have all been su ccessfully employed in
recentevaluations.
•Rule-basedsystemsbasedonpartialparsingorchunking
•Statistical sequenceclassiﬁersbasedonstandardtoken-b y-token IOBencoding
•Constituent-basedclassiﬁcationasusedinsemanticrolel abeling
Rule-based approaches to temporal expression recognition use cascades of au-
tomatatorecognizepatternsatincreasinglevelsofcomple xity. Sincetemporalexpres-

sions are limited to a ﬁxed set of standard syntactic categor ies, most of these systems
make use of pattern-based methods for recognizing syntacti c chunks. That is, tokens
are ﬁrst part-of-speechtagged and then larger and larger ch unks are recognized using

DRAFTSection22.3. TemporalandEventProcessing 757
the results from previous stages. The only difference from t he usual partial parsing
approachesisthefactthattemporalexpressionsmustconta intemporallexicaltriggers.
Patterns must, therefore, contain either speciﬁc trigger w ords (e.g. February), or pat-
terns representing classes (e.g. MONTH). Fig. 22.19 illustrates this approach with a
small representativefragmentfroma rule-basedsystemwri tteninPerl.
Sequence labeling approaches follow exactly the same scheme introduced in
Ch. 13 for syntactic chunking. The three tags I,OandBare used to mark tokens
thatareeitherinside,outsideorbeginatemporalexpressi on,asdelimitedbyTIMEX3
tags. Example22.3.1wouldbelabeledasfollowsin thissche me.
A
Ofare
Oincrease
Oinitiated
Olast
Bweek
Iby
OUAL
OCorp’s...
O
As expected, features are extracted from the context surrou nding a token to be
tagged and a statistical sequence labeler is trained using t hose features. As with syn-

A
Ofare
Oincrease
Olast
Bweek
Iby
OUAL
O
tactic chunking and named entity recognition, any of the usu al statistical sequence
methods can be applied. Fig. 22.20 lists the standard featur es used in the machine
learning-basedapproachtotemporaltagging.
Constituent-basedmethods combineaspectsofbothchunkingandtoken-by-token
labeling. In this approach, a complete constituent parse is produced by automatic
means. The nodes in the resulting tree are then classiﬁed, on e by one, as to whether
they contain a temporal expression or not. This task is accom plished by training a bi-

naryclassiﬁerwithannotatedtrainingdata,usingmanyoft hesamefeaturesemployed
inIOB-style training. Thisapproachseparatestheclassiﬁcatio n problemfromthe seg-
mentationproblembyassigningthesegmentationproblemto thesyntacticparser. The
motivation for this choice was mentioned earlier; in curren tly available training mate-
rials, temporalexpressionsare limited to syntactic const ituentsfromoneof a ﬁxedset
of syntactic categories. Therefore, it makes sense to allow a syntactic parser to solve
thesegmentationpartoftheproblem.

In standard evaluations, temporal expression recognizers are evaluated using the
usual recall, precision and F-measures. In recent evaluati ons, both rule-basedand sta-
tistical systems achieve about the same level of performanc e, with the best systems
reachinganF-measureofaround.87onastrict exactmatchcr iteria. Onaloosercrite-
rionbasedon overlapwith goldstandardtemporalexpressio ns,thebest systems reach
anF-measureof.94.4
Themajordifﬁcultiesforalloftheseapproachesareachiev ingreasonablecoverage,

correctly identifying the extent of temporal expressions a nd dealing with expressions
thattriggerfalsepositives. Theproblemoffalsepositive sarisesfromtheuseoftempo-
ral triggerwordsaspartsof propernames. For example,all o fthe followingexamples
arelikelyto causefalsepositivesforeitherrule-basedor statistical taggers.
(22.19)1984tells thestoryofWinstonSmithandhisdegradationbytheto talitarian
statein whichhe lives.
(22.20)Edgeisset tojoinBonoonstageto performU2’sclassic SundayBloody
Sunday.

Sunday.
4http://www.nist.gov/speech/tests/ace/

DRAFT758 Chapter 22. InformationExtraction
# yesterday/today/tomorrow
$string =˜ s/(($OT+(early|earlier|later?)$CT+\s+)?(($ OT+the$CT+\s+)?$OT+day$CT+\s+
$OT+(before|after)$CT+\s+)?$OT+$TERelDayExpr$CT+(\s +$OT+(morning|afternoon|
evening|night)$CT+)?)/<TIMEX2 TYPE=\"DATE\">$1<\/TIM EX2>/gio;
$string =˜ s/($OT+\w+$CT+\s+)
<TIMEX2 TYPE=\"DATE\"[ˆ>] *>($OT+(Today|Tonight)$CT+)<\/TIMEX2>/$1$2/gso;
# this/that (morning/afternoon/evening/night)
$string =˜ s/(($OT+(early|earlier|later?)$CT+\s+)?$OT +(this|that|every|the$CT+\s+
$OT+(next|previous|following))$CT+\s *$OT+(morning|afternoon|evening|night)
$CT+(\s+$OT+thereafter$CT+)?)/<TIMEX2 TYPE=\"DATE\"> $1<\/TIMEX2>/gosi;
Figure22.19 Fragment ofPerl code from MITRE’sTempExtemporal tagging s ystem.
Token Thetargettokentobelabeled
Tokensinwindow Bag oftokensin thewindowarounda target
Shape Charactershapefeatures
POS Partsofspeechoftargetandwindowwords
Chunktags Base-phrasechunktag fortargetandwordsina win dow

Lexicaltriggers Presenceina list oftemporalterms
Figure22.20 Typical features usedtotrain IOBstyle temporal expression taggers.
<TIMEX3 id=t1 type="DATE" value="2007-07-02" functionIn Document="CREATION_TIME">
July 2, 2007 </TIMEX3> A fare increase initiated <TIMEX3 id= "t2" type="DATE"
value="2007-W26" anchorTimeID="t1">last week</TIMEX3> by UAL Corp’s United Airlines
was matched by competitors over <TIMEX3 id="t3" type="DURA TION" value="P1WE"
anchorTimeID="t1"> the weekend </TIMEX3>, marking the sec ond successful fare increase

in <TIMEX3 id="t4" type="DURATION" value="P2W" anchorTim eID="t1"> two weeks </TIMEX3>.
Figure22.21 TimeML markup including normalized values for temporal exp ressions.
(22.21)BlackSeptember triedto detonatethreecarbombsinNewYorkCityin
March1973.
22.3.2 Temporal Normalization
The task of recognizingtemporal expressionsis typically f ollowed by the task of nor-
malization. Temporal normalization refers to the process of mapping a temporalTemporal

expressiontoeitheraspeciﬁcpointintime,ortoaduration . Pointsintimecorrespond
either to calendar dates or to times of day (or both). Duratio ns primarily consist of
lengths of time, but may also include information concernin g the start and end points
ofa durationwhenthatinformationisavailable.
Normalizedrepresentationsoftemporalexpressionsareca pturedusingthe VALUE
attributefromtheISO8601standardforencodingtemporalv alues(ISO8601,2004). To
illustrate some aspects of this scheme, let’s return to our e arlier example, reproduced

inFig. 22.21with thevalueattributesaddedin.
The dateline, or document date, for this text was July 2, 2007 . The ISO represen-
tation for this kind of expressionis YYYY-MM-DD, or in this c ase, 2007-07-02. The
encodingsforthetemporalexpressionsinoursampletextal lfollowfromthisdate,and
areshownhereasvaluesforthe VALUEattribute. Let’sconsidereachofthesetemporal

DRAFTSection22.3. TemporalandEventProcessing 759
Unit Pattern Sample Value
FullySpeciﬁedDates YYYY-MM-DD 1991-09-28
Weeks YYYY-nnW 2007-27W
Weekends PnWE P1WE
24hourclocktimes HH:MM:SS 11:13:45
DatesandTimes YYYY-MM-DDTHH:MM:SS 1991-09-28T11:00:00
Financialquarters Qn 1999-3Q
Figure22.22 Sample ISOpatterns forrepresenting various times anddura tions.
expressionsinturn.
The ﬁrst temporal expression in the text proper refers to a pa rticular week of the
year. IntheISOstandard,weeksarenumberedfrom01to53,wi ththeﬁrstweekofthe
yearbeingtheonethathastheﬁrst Thursdayoftheyear. Thes eweeksare represented
usingthetemplateYYYY-Wnn. TheISOweekforourdocumentda teisweek27,thus
thevaluefor last week isrepresentedas“2007-W26”.
The next temporal expression is the weekend . ISO weeks begin on Monday, thus,
weekends occur at the end of a week and are fully contained wit hin a single week.
Weekends are treated as durations, so the value of the VALUEattribute has to be a

length. DurationsarerepresentedusingthepatternP nx,wherenisanintegerdenoting
the length and xrepresentsthe unit, as in P3Y for three years or P2D for two days. In
this example, one weekend is captured as P1WE. In this case, t here is also sufﬁcient
informationto anchorthisparticularweekendaspartofa pa rticularweek. Suchinfor-
mationisencodedinthe ANCHORTIMEID attribute. Finally,thephrase twoweeks also
denotesa durationcapturedasP2W.

There is a lot more to both the ISO 8601 standard and the variou s temporal anno-
tation standards— far too much to cover here. Fig. 22.22 desc ribessome of the basic
ways that other times and durations are represented. Consul t (ISO8601, 2004; Ferro
et al.,2005;Pustejovskyet al.,2005)formoredetails.
Most current approaches to temporal normalization employ r ule-based methods
thatassociatesemanticanalysisprocedureswithpatterns matchingparticulartemporal
expressions. This is a domain-speciﬁc instantiation of the compositional rule-to-rule

approachintroducedin Ch. 18. In this approach,the meaning of a constituent is com-
puted fromthe meaningof its parts, and the methodused to per formthis computation
is speciﬁc to the constituent being created. The only differ ence here is that the se-
mantic composition rules involve simple temporal arithmet ic rather than λ-calculus
attachments.
Tonormalizetemporalexpressions,we’ll needrulesforfou rkindsofexpressions.
•Fullyqualiﬁedtemporalexpressions
•Absolutetemporalexpressions
•Relativetemporalexpressions
•Durations

Fully qualiﬁed date expressions contain a year, month and day in some conven-Fullyqualiﬁed
date expressions
tionalform. Theunitsintheexpressionmustbedetectedand thenplacedinthecorrect
place in the corresponding ISO pattern. The following patte rn normalizes the fully
qualiﬁedtemporalexpressionusedinexpressionslike April 24,1916 .

DRAFT760 Chapter 22. InformationExtraction
FQTE→MonthDate ,Year{Year.val−Month .val−Date.val}
In this rule, the non-terminals Month,Date, andYearrepresent constituents that have
alreadybeenrecognizedandassignedsemanticvalues,acce ssedviathe *.valnotation.
Thevalueofthis FQEconstituentcan,inturn,beaccessedas FQTE.val duringfurther
Fully qualiﬁed temporal expressions are fairly rare in real texts. Most temporal
expressionsinnewsarticlesareincompleteandareonlyimp licitlyanchored,oftenwith
respect to the dateline of the article, which we’ll refer to a s the document’s temporal
anchor. Thevaluesofrelativelysimpletemporalexpressionssuch astoday,yesterday , Temporal anchor
ortomorrow can all be computed with respect to this temporal anchor. The semantic
procedurefor todaysimplyassignstheanchor,whiletheattachmentsfor tomorrow and
yesterday adda day andsubtracta dayfromthe anchor,respectively. Of course,given
the circular nature of our representations for months, week s, days and times of day,

ourtemporalarithmeticproceduresmustusemoduloarithme ticappropriatetothetime
unitbeingused.
Unfortunately, even simple expressions such as the weekend orWednesday intro-
duceafairamountofcomplexity. Inourcurrentexample, theweekend clearlyrefersto
theweekendoftheweekthatimmediatelyprecedesthedocume ntdate. Butthiswon’t
alwaysbethecase,asisillustratedin thefollowingexampl e.
(22.22)RandomsecuritychecksthatbeganyesterdayatSkyHarborwi ll continueat
leastthroughtheweekend.

In this case, the expression the weekend refers to the weekend of the week that the
anchoringdate is part of (i.e. the coming weekend). The info rmation that signals this
comesfromthetense of continue,theverbgoverning theweekend .
Relative temporalexpressionsare handledwith temporalar ithmetic similar to that
usedfortodayandyesterday . Toillustratethis, considertheexpression lastweek from
our example. From the document date, we can determine that th e ISO week for the
articleisweek27,so last week issimply1minusthecurrentweek.

Again, even simple constructions such as this can be ambiguo us in English. The
resolution of expressions involving nextandlastmust take into account the distance
from the anchoringdate to the nearest unit in question. For e xample,a phrase such as
next Friday can refer to either the immediatelynext Friday, or to the Fri day following
that. The determining factor has to do with the proximity to t he reference time. The
closerthedocumentdateistoaFriday,themorelikelyit ist hatthephrase nextFriday

will skip the nearest one. Such ambiguities are handled by en coding language and
domainspeciﬁcheuristicsintothetemporalattachments.
Theneedtoassociatehighlyidiosyncratictemporalproced ureswithparticulartem-
poralconstructionsaccountsforthewidespreaduseofofru le-basedmethodsintempo-
ral expression recognition. Even when high performance sta tistical methods are used
for temporal recognition, rule-based patterns are still re quired for normalization. Al-

though the construction of these patterns can be tedious and ﬁlled with exceptions, it
appears that sets of patterns that provide good coverage in n ewswire domains can be
createdfairlyquickly(Ahnet al.,2005).

DRAFTSection22.3. TemporalandEventProcessing 761
Characterafﬁxes Character-levelpreﬁxesandsufﬁxesofta rgetword
Nominalizationsufﬁx Characterlevelsufﬁxesfornominali zations(eg. -tion)
Partofspeech Part ofspeechofthetargetword
Lightverb Binaryfeatureindicatingthat thetargetisgove rnedbya lightverb
Subjectsyntacticcategory Syntacticcategoryofthe subje ctofthesentence
Morphologicalstem Stemmedversionofthetargetword
Verbroot Root formoftheverbbasisfora nominalization
Wordnethypernyms Hypernymset forthetarget
Figure22.23 Features commonly usedinboth rule-basedand statisticala pproaches toevent detection.
Finally,manytemporalexpressionsareanchoredtoeventsm entionedina textand
notdirectlyto othertemporalexpressions. Considerthefo llowingexample.
(22.23)Oneweekafterthe storm,JetBlue issuedits customerbillof rights.
To determine when JetBlue issued its customer bill of rights we need to determine
the time of the storm event, and then that time needs to be modiﬁed by the temporal

expression one week after . We’ll return to this issue when we take up event detection
22.3.3 Event Detectionand Analysis
Thetaskof eventdetectionandclassiﬁcation istoidentifymentionsofeventsintextsEventdetectionandclassiﬁcation
and then assign those events to a variety of classes. For the p urposes of this task, an
event mention is any expression denoting an event or state th at can be assigned to a
particular point, or interval, in time. The following marku p of Example 22.3.1 shows
all theeventsinthistext.

[EVENTCiting]highfuelprices,UnitedAirlines[ EVENTsaid]Fridayithas[ EVENT
increased]faresby$6perroundtriponﬂightstosomecities alsoservedbylower-
cost carriers. American Airlines, a unit of AMR Corp., immed iately [EVENT
matched] [ EVENTthe move], spokesman Tim Wagner [ EVENTsaid]. United, a
unit of UAL Corp., [ EVENTsaid] [EVENTthe increase] took effect Thursday and
[EVENTapplies] tomost routes where it[ EVENTcompetes] against discount carri-
ers,such as ChicagotoDallas andDenver toSanFrancisco.

In English, most event mentions correspond to verbs, and mos t verbs introduce
events. However, as we can see from our example this is not alw ays the case. Events
canbeintroducedbynounphrases,asin themoveandtheincrease ,andsomeverbsfail
to introduce events, as in the phrasal verb took effect , which refers to when the event
began rather than to the event itself. Similarly, light verb s such as make,take, and
haveoftenfailtodenoteevents. Inthesecases,theverbissimpl yprovidingasyntactic

structurefortheargumentstoaneventexpressedbythedire ctobjectasin tookaﬂight .
Both rule-based and statistical machine learning approach es have been applied to
theproblemofeventdetection. Bothapproachesmakeuseofs urfaceinformationsuch
as parts of speech information, presence of particular lexi cal items, and verb tense
information. Fig. 22.23illustrates the key featuresused i n currentevent detectionand
classiﬁcationsystems.

DRAFT762 Chapter 22. InformationExtraction
Having detected both the events and the temporal expression s in a text, the next
logical task is to use this informationto ﬁt the eventsintoa completetimeline. Sucha
timeline would be useful for applicationssuch as question a nswering and summariza-
tion. Thisambitioustaskisisthesubjectofconsiderablec urrentresearchbutisbeyond
thecapabilitiesofcurrentsystems.
A somewhat simpler, but still useful, task is to impose a part ial ordering on the
events and temporal expressions mentioned in a text. Such an ordering can provide
many of the same beneﬁts as a true timeline. An example of such a partial ordering
would be to determine that the fare increase by American Airlines cameafterthe fare
increaseby Unitedin oursampletext. Determiningsuchanorderingcanbe viewe das
a binary relation detection and classiﬁcation task similar to those described earlier in
Sec.22.2.
Current approachesto this problemattempt to identify a sub set of Allen’s 13 tem-

Sec.22.2.
poral relationsdiscussed earlier in Ch. 17, and shown here i n Fig. 22.24. Recent eval-
uation efforts have focused on detecting the before,afterandduringrelations among
thetemporalexpressions,documentdateandeventmentions ina text(Verhagenet al.,
2007). Most of the top-performing systems employ statistic al classiﬁers, of the kind
discussed earlier in Sec. 22.2, trained on the TimeBank corp us (Pustejovsky et al.,
2003c).
22.3.4 TimeBank

2003c).
Aswe’veseenwithothertasks,it’stremendouslyusefultoh aveaccesstotextannotated
with the types and relations in which we’re interested. Such resources facilitate both
corpus-basedlinguisticresearchaswellasthetrainingof systemstoperformautomatic
tagging. The TimeBank corpusconsistsoftextannotatedwithmuchoftheinformati on TimeBank
we’vebeendiscussingthroughoutthissection (Pustejovsk yet al., 2003c). Thecurrent
release (TimeBank 1.2) of the corpus consists of 183 news art icles selected from a
varietyofsources,includingthePennTreeBankandPropBan kcollections.

Each article in the TimeBank corpus has had the temporal expr essions and event
mentions in them explicitly annotated in the TimeML annotat ion (Pustejovsky et al.,
2003a). In addition to temporal expressions and events, the TimeML annotation pro-
vides temporal links between events and temporal expressio ns that specify the nature
of the relation between them. Consider the following sample sentence and its corre-
spondingmarkupshowninFig. 22.25selectedfromoneoftheT imeBankdocuments.

(22.24)DeltaAir Linessoared33%to arecordinthe ﬁscal ﬁrstquarte r,buckingthe
industrytrendtowarddecliningproﬁts.
As annotated, this text includes three events and two tempor al expressions. The
eventsare all in the occurrenceclass andare givenuniqueid entiﬁersforuse in further
annotations. The temporal expressions include the creatio n time of the article, which
servesasthedocumenttime,anda singletemporalexpressio nwithinthetext.
In addition to these annotations, TimeBank provides 4 links that capture the tem-

poral relations between the events and times in the text. The following are the within
sentencetemporalrelationsannotatedforthisexample.
•Soaring e1isincluded inthe ﬁscal ﬁrst quarter t58

DRAFT764 Chapter 22. InformationExtraction
•Soaring e1isbefore1989-10-26 t57
•Soaring e1issimultaneous withthebucking e3
•Declining e4includessoaring e1
The set of 13 temporalrelationsused in TimeBankare based on Allen’s (Allen, 1984)
relationsintroducedearlierin Fig.22.24.
22.4 Template-Filling
Manytextscontainreportsofevents,andpossiblysequence sofevents,thatoftencorre-
spondtofairlycommon,stereotypicalsituationsinthewor ld. Theseabstractsituations
can be characterized as scripts, in that they consist of prototypical sequences of sub- Scripts
events, participants, roles and props (Schank and Abelson, 1977). The use of explicit
representationsofsuchscriptsinlanguageprocessingcan assistinmanyoftheIEtasks
we’ve beendiscussing. In particular,the strong expectati onsprovidedby these scripts
can facilitate the proper classiﬁcation of entities, the as signment of entities into roles
andrelations, andmost critically,the drawingof inferenc esthat ﬁll in thingsthat have
beenleftunsaid.

In their simplest form, such scripts can be represented as templates consisting of Templates
ﬁxedsets of slotswhichtake asvalues slot-ﬁllers belongingto particularclasses. The
task oftemplate-ﬁlling is to ﬁnd documents that invoke particular scripts and then
ﬁll the slotsin theassociated templateswith ﬁllersextrac tedfromthe text. These slot-
(times,amounts,entitiesfromanontology,etc.)

Aﬁlled templatefromouroriginalairlinestorymightlookl ikethefollowing.
AMOUNT: $6

Note that as is often the case, the slot-ﬁllers in this exampl e all correspond to
detectable named entities of various kinds (organizations , amounts and times). This
suggests that template-ﬁlling applications should rely on tags provided by named en-
tityrecognition,temporalexpressionandco-referenceal gorithmstoidentifycandidate
slot-ﬁllers.

AMOUNT: $6

The next section describes a straightforward approach to ﬁl ling slots using se-
quence labeling techniques. Sec. 22.4.2 then describes a sy stem designed to address
a considerably more complex template-ﬁlling task, based on the use of cascades of

DRAFTSection22.4. Template-Filling 765
22.4.1 StatisticalApproaches to Template-Filling
A surprisingly effective approach to template-ﬁlling simp ly casts it as a statistical se-
quence labeling problem. In this approach, systems are trai ned to label sequences of
tokens as potential ﬁllers for particular slots. There are t wo basic ways to instantiate
thisapproach: the ﬁrst isto trainseparate sequenceclassi ﬁers for eachslot to be ﬁlled
andthensend the entiretext througheach labeler,the other is to trainonelargeclassi-
ﬁer (usually an HMM) that assigns labelsfor eachof the slots to be recognized. We’ll
focusontheformerapproachhere;we’lltakeupthesinglela rgeclassiﬁerapproachin
Ch.24.
Under the one classiﬁer per slot approach, slots are ﬁlled wi th the text segments
identiﬁed by each slot’s correspondingclassiﬁer. As with t he other IE tasks described
earlier in this chapter, all manner of statistical sequence classiﬁers have been applied

Ch.24.
to this problem, all using the usual set of features: tokens, shapes of tokens, part-of-
speechtags,syntacticchunktags, andnamedentitytags.
There is the possibility in this approach that multiple non- identical text segments
will be labeled with the same slot label. This situation can a rise in two ways: from
competingsegmentsthat referto the same entityusing diffe rentreferringexpressions,
or from competing segments that represent truly distinct hy potheses. In our sample

text, we might expectthe segments United,United Airlines to be labeled as the L EAD
AIRLINE. These are not incompatiblechoices and the referenceresol utiontechniques
introducedinCh. 21canprovidea pathtoa solution.
Truly competinghypothesesarise when a text containsmulti ple entities of the ex-
pectedtypeforagivenslot. Inourexample, UnitedAirlines andAmericanAirlines are
both airlines and it is possible for both to be tagged as L EADAIRLINEbased on their
similarity to exemplars in the training data. In general, mo st systems simply choose

the hypothesis with the highest conﬁdence. Of course, the im plementation of this
conﬁdence heuristic is dependent on the style of sequence cl assiﬁer being employed.
Markov-basedapproachessimplyselectthesegmentwiththe highestprobabilitylabel-
ing(FreitagandMcCallum,1999).
Avarietyofannotatedcollectionshavebeenusedtoevaluat ethisstyleofapproach
to template-ﬁlling, including sets of job announcements, c onference calls for papers,
restaurant guides and biological texts. A frequently emplo yed collection is the CMU

SeminarAnnouncementCorpus5,acollectionof485seminarannouncementsretrieved
fromtheWebwithslotsannotatedforthe SPEAKER ,LOCATION ,START TIME andEND
TIME. State-of-the-art F-measures on this dataset range from ar ound .98 for the start
and end time slots, to as high as .77 for the speaker slot (Roth and tau Yih, 2001;
PeshkinandPfefer,2003).
As impressive as these results are, they are due as much to the constrained nature
ofthetaskastothetechniquestheyhavebeenemployed. Thre estrongtaskconstraints
have contributed to this success. First, in most evaluation s all the documents in the

collection are all relevant and homogeneous, that is they ar e known to contain the
slots of interest. Second, the documents are all relatively small, providing little room
for distractor segments that might incorrectly ﬁll slots. A nd ﬁnally, the target output
5http://www.isi.edu/info-agents/RISE/

DRAFT766 Chapter 22. InformationExtraction
TIE-UP-1:
RELATIONSHIP : TIE-UP
ENTITIES: “BridgestoneSportsCo.”
“alocalconcern”
“aJapanesetradinghouse”
JOINTVENTURECOMPANY “BridgestoneSportsTaiwanCo.”
ACTIVITY ACTIVITY-1
AMOUNT NT$20000000
ACTIVITY-1:
COMPANY “BridgestoneSportsTaiwanCo.”
PRODUCT “ironand“metalwood”clubs”
STARTDATE DURING: January1990
Figure22.26 Thetemplatesproducedbythe FASTUS(Hobbsetal.,1997) informationextrac-
tionengine given the input texton page 766.
consists solely of a small set of slots which are to be ﬁlled wi th snippetsfrom the text
itself.
22.4.2 Finite-State Template-FillingSystems
The tasks introduced in the Message Understanding Conferences (MUC) (Sundheim,
1993),a seriesofU.S. Government-organizedinformatione xtractionevaluations,rep-
resent a considerably more complextemplate-ﬁlling proble m. Consider the following
sentencesselectedfromtheMUC-5materialsfromGrishmana ndSundheim(1995).
Bridgestone Sports Co. said Friday it has set up a joint ventu re in Taiwan with a

itself.
local concern and a Japanese trading house to produce golf cl ubs to be shipped to
Japan.
The joint venture, Bridgestone Sports Taiwan Co., capitali zed at 20 million new
Taiwan dollars, will start production in January 1990 with p roduction of 20,000
ironand“metal wood” clubs amonth.
TheMUC-5evaluationtaskrequiredsystemstoproducehiera rchicallylinkedtem-
plates describing the participants in the joint venture, th e resulting company, and its
intended activity, ownership and capitalization. Fig. 22. 26 shows the resulting struc-

Japan.
ture produced by the FASTUSsystem (Hobbs et al., 1997). Note how the ﬁller of the
ACTIVITY slot ofthe TIE-UPtemplateisitself a templatewith slotstobeﬁlled.
TheFASTUSsystem produces the template given above, based on a cascade of
transducersinwhicheachleveloflinguisticprocessingex tractssomeinformationfrom
thetext,whichispassedontothenexthigherlevel,asshown inFigure22.27
Most systems base most of these levels on ﬁnite-automata, al though in practice
most complete systems are not technically ﬁnite-state, eit her because the individual

automataareaugmentedwithfeatureregisters(asin FASTUS),orbecausetheyareused
only as preprocessingsteps for full parsers (e.g., Gaizaus kas et al., 1995; Weischedel,
1995) , or are combined with other components based on statis tical methods (Fisher
et al.,1995).

DRAFTSection22.4. Template-Filling 767
No. Step Description
1 Tokens: Transfer an input stream of characters into a
tokensequence.
2 ComplexWords: Recognizemulti-wordphrases,numbers,an d
propernames.
3 Basic phrases: Segment sentences into noun groups, verb
groups,andparticles.
4 Complexphrases: Identify complex noun groups and complex
verbgroups.
5 SemanticPatterns: Identify semantic entities and events and in-
sertintotemplates.
6 Merging: Merge references to the same entity or event
fromdifferentpartsofthetext.
Figure22.27 Levels of processing in FASTUS(Hobbs et al., 1997). Each level extracts a
speciﬁc type of information whichis thenpassed ontothe nex t higher level.
Let’s sketch the FASTUSimplementation of each of these levels, following Hobbs
et al. (1997) and Appelt et al. (1995). After tokenization, t he second level recognizes
multiwordslike set up, andjoint venture , and nameslike Bridgestone SportsCo. . The
named entity recognizer is a transducer, composed of a large set of speciﬁc mappings

designedtohandlethe usualset ofnamedentities.
The following are typical rules for modeling names of perfor ming organizations
likeSan Francisco Symphony Orchestra andCanadian Opera Company . While the
rules are written using a context-free syntax, there is no re cursion and therefore they
canbeautomaticallycompiledintoﬁnite-statetransducer s.
Performer-Org→(pre-location)Performer-Noun+Perf-Org-Sufﬁx
pre-location→locname|nationality
locname→city|region
Perf-Org-Sufﬁx→orchestra,company
Performer-Noun→symphony,opera

nationality→Canadian,American,Mexican
city→SanFrancisco,London
Thesecondstagealsomighttransducesequenceslike fortytwo intotheappropriate
numericvalue(recallthediscussionofthisprobleminCh. 8 ).
The third FASTUSstage implements chunking and produces a sequence of basic
syntactic chunks, such as noungroups, verb groups, and so on , using ﬁnite-state rules
ofthesortdiscussedin Ch.13.
Theoutputofthe FASTUSbasicphraseidentiﬁerisshowninFigure22.28;notethe
useofsomedomain-speciﬁcbasic phraseslike Company andLocation.

Recall that Ch. 13 described how these basic phrases can be co mbined into more
complexnoungroupsandverbgroups. Thisisaccomplishedin Stage4of FASTUS,by
dealingwithconjunctionandwith the attachmentofmeasure phrasesas inthe follow-
ing.
20,000ironand“metalwood”clubsa month,
andprepositionalphrases:

DRAFT768 Chapter 22. InformationExtraction
PhraseType Phrase
Company BridgestoneSportsCo.
VerbGroup said
NounGroup Friday
NounGroup it
VerbGroup hadset up
NounGroup a jointventure
Preposition in
Location Taiwan
Preposition with
NounGroup a localconcern
Conjunction and
NounGroup a Japanesetradinghouse
VerbGroup to produce
NounGroup golfclubs
VerbGroup to beshipped
Preposition to
Location Japan
Figure22.28 The output of Stage 2of the FASTUSbasic-phrase extractor, w hich uses ﬁnite-
state rules ofthe sort described by Appelt andIsrael (1997) .
Template/Slot Value
1 RELATIONSHIP : TIE-UP
“a localconcern”
“a Japanesetradinghouse”
2 ACTIVITY: PRODUCTION
PRODUCT: “golfclubs”
3 RELATIONSHIP : TIE-UP
JOINTVENTURECOMPANY: “BridgestoneSportsTaiwanCo.”
AMOUNT: NT$20000000
4 ACTIVITY: PRODUCTION
COMPANY: “BridgestoneSportsTaiwanCo.”
STARTDATE: DURING: January1990
5 ACTIVITY: PRODUCTION
PRODUCT: “ironand“metalwood”clubs”

Figure22.29 The ﬁve partial templates produced by Stage 5 of the FASTUSsystem. These
templates willbe merged bythe Stage 6merging algorithm top roduce the ﬁnal template shown
inFig.22.26 onpage 766.
productionof20,000ironand“metalwood”clubsamonth,
The output of Stage 4 is a list of complex noun groups and verb g roups. Stage
5 takes this list, ignoring all input that has not been chunke d into a complex group,
recognizesentitiesandeventsinthecomplexgroups,andin sertstherecognizedobjects

into the appropriate slots in templates. The recognition of entities and events is done
byhand-codedﬁnite-stateautomatawhosetransitionsareb asedonparticularcomplex-

DRAFTSection22.5. Advanced: BiomedicalInformationExtractio n∗769
phrase types annotated by particular head words or particul ar features like company,
currency,ordate.
As an example, the ﬁrst sentence of the news story above reali zes the semantic
patterns based on the following two regular expressions (wh ere NG indicates Noun-
GroupandVGVerb-Group).
•NG(Company/ies) VG(Set-up)NG(Joint-Venture) withNG(Co mpany/ies)
•VG(Produce) NG(Product)
The second sentence realizes the second pattern above as wel l as the following two
patterns:
•NG(Company) VG-Passive(Capitalized) atNG(Currency)
•NG(Company) VG(Start)NG(Activity)in/onNG(Date)
Theresultofprocessingthesetwosentencesisthesetofﬁve drafttemplatesshown
in Fig. 22.29. These ﬁve templates must then be merged into th e single hierarchical
structure shownin Fig. 22.26. The mergingalgorithmdecide swhethertwo activity or
relationshipstructuresaresufﬁcientlyconsistentthatt heymightbedescribingthesame

patterns:
events, and merges them if so. The merging algorithm must als o perform reference
resolutionasdescribedinCh.21.
22.5 Advanced: BiomedicalInformationExtraction∗
Information extraction from biomedical journal articles h as become an important ap-
plication area in recent years. The motivation for this work comes primarily from
biologists, who ﬁnd themselves faced with an enormous incre ase in the number of
publicationsin their ﬁeld since the advent of moderngenomi cs— so many that keep-

ing up with the relevant literature is nearly impossible for many scientists. Fig. 22.30
amply demonstratesthe severity of the problemfaced by thes e scientists. Clearly, ap-
plicationsthatcanautomatethe extractionandaggregatio nofusefulinformationfrom
suchsourceswouldbea boontoresearchers.
A growingapplicationarea for informationextractionin th e biomedicaldomainis
as an aid to the construction of large databases of genomic an d related information.

Withouttheavailabilityofinformationextraction-based curatorassistancetools,many
manual database construction efforts will not be complete f or decades — a time-span
muchtoolongto beuseful(Baumgartneret al.,2007).
A good example of this kind of application is the MuteXt syste m. This system
targetstwo namedentity types— mutationsin proteinsandtw o veryspeciﬁc typesof
proteins called G-coupled protein receptors and nuclear hormone receptors . MuteXt
was used to build a database that drew informationfrom 2,008 documents; building it

byhandwouldhavetakenanenormouslytime-consumingandex pensiveundertaking.
Mutations in G-protein coupled receptors are associated wi th a range of diseases that
includes diabetes, ocular albinism, and retinitis pigment osa, so even this simple text
miningsystemhasaclear applicationto thereliefofhumans uffering.
∗Thissection was written by Kevin Bretonnel Cohen

DRAFT770 Chapter 22. InformationExtraction
Medline Growth Rate
338,819356,763375,151391,119398,391400,592403,471411,454421,648432,492442,671440,445457,072475,789515,304523,415536,096562,134621,342
6,907,2127,263,9757,639,1268,030,2458,428,6368,829,2289,232,6999,644,15310,065,80110,498,29310,940,96411,381,40911,838,48112,314,27012,829,57413,352,98913,889,08514,451,21915,072,561
00E+02E+64E+66E+68E+610E+612E+614E+616E+618E+61986198719881989199019911992199319941995199619971998199920002001200220032004
YearMedline Entriesnew
total
Figure22.30 ExponentialgrowthinnumberofarticlesavailableinthePu bMeddatabasefrom
1986 to2004 (Data from (Cohenand Hunter, 2004)).
Biologists and bioinformaticians have recently come up wit h even more innova-
tive uses for text mining systems, in which the output is neve r intended for viewing
by humans, but rather is used as part of the analysis of high-t hroughput assays—
experimental methods which produce masses of data points th at would have been

total
unimaginable just twenty years ago—and as part of technique s for using data in ge-
nomic data repositories. Ng (2006) provides a review and an i nsightful analysis of
workinthisvein.
22.5.1 BiologicalNamed Entity Recognition
Informationextractiontasksinthebiologicalrealmarech aracterizedbyamuchwider
range of relevant types of entities than the PERSON,ORGANIZATION , andLOCATION
semantic classes that characterizework that is focused on n ews-style texts. Fig. 22.31

and the following example illustrate just a small subset of t he variety of semantic
classes of named entities that have been the target of NER sys tems in the biomedi-
cal domain.
[TISSUEPlasma] [ GPBNP] concentrations were higher inboth the [ POPULATION
judo] and [ POPULATION marathon groups] than in [ POPULATION controls], and
positivelycorrelatedwith[ ANATLV]mass as wellas withdeceleration time.
NearlyallofthetechniquesdescribedinSec.22.1havebeen appliedtothebiomed-
ical NERproblem,with a particularfocuson the problemof re cognizinggene/protein

DRAFTSection22.5. Advanced: BiomedicalInformationExtractio n∗771
Semanticclass Examples
Cell lines T98G,HeLacell,Chinese hamsterovarycells, CHO cells
Cell types primaryT lymphocytes,naturalkillercells, NKcells
Chemicals citric acid,1,2-diiodopentane,C
Drugs cyclosporinA, CDDP
Genes/proteins white, HSP60,proteinkinaseC, L23A
Malignancies carcinoma,breastneoplasms
Medical/clinicalconcepts amyotrophiclateralsclerosis
Mousestrains LAFT,AKR
Mutations C10T,Ala64→Gly
Populations judogroup
Figure22.31 A sample of the semantic classes of named entities that have b een recognized in biomedical NLP.
Note the surface similaritiesbetween many of the examples.
names. Thistaskisparticularlydifﬁcultduetothewideran geofformsthatgenenames
can take: white, insulin, BRCA1, ether a go-go, andbreast cancerassociated 1 are all
the names of genes. The choice of algorithm for gene name reco gnition seems to be
less importantthan the choice of features; typical feature sets include word-shapeand

contextual features, as discussed earlier; additionally, knowledge-basedfeatures, such
as using the count of Google hits for a sequence like BRCA1 gene to decide whether
or not a token of the string BRCA1is a reference to a gene or not, are sometimes
incorporatedintostatistical systems.
Surprisingly,the use of huge publicly available lists of ge ne names has not gener-
ally contributed to the performance of a gene/protein NER sy stem (Yeh et al., 2005),

and in fact may actually degrade it (Baumgartner et al., 2006 ). It is not uncommon
for gene names to be many tokens long (e.g. breast cancer associated 1 ). Gene name
length has a demonstrableeffect on NER system performance( Kinoshita et al., 2005;
Yehetal.,2005),andanytechniqueforcorrectlyﬁndingthe boundariesofmulti-token
namesseemsto increaseperformance. Use ofthe abbreviatio n-deﬁnition-detectional-
gorithm (Schwartz and Hearst, 2003) is common for this purpo se, since many such

names appear as abbreviation or symbol deﬁnitions at some po int in a publication.
Basenoungroupchunkerscanalsobeusefulinthisregard,as canasurprisinglysmall
numberofheuristicrules(Kinoshitaet al.,2005).
22.5.2 GeneNormalization
Having identiﬁed all the mentions of biological entities in a text, the next step is to
map them to unique identiﬁers in databases or ontologies. Th is task has been most
heavily studied for genes, where it is known as gene normalization . Some of theGenenormalization

complexitiesoftheproblemcomefromhighdegreesofvariab ilityintherealizationof
thenamesofspeciﬁcentitiesinnaturally-occurringtext; thenatureoftheproblemwas
ﬁrst delineated by Cohen et al. (2002). In that work a standar d discovery procedure
from descriptive linguistics was used to determine what sor ts of variability in gene
namescanbeignored,andwhatsortsmustnotbeignored. More recently,Morganetal.
(2007) have shown how linguistic characteristics of commun ity-speciﬁcgene-naming

conventions affect the complexity of this task when the norm alization of genes from

DRAFT772 Chapter 22. InformationExtraction
varying species is attempted. Gene normalization can be con sidered a type of word
sense disambiguation task, midway between a targetted WSD t ask and an all-words
WSD task.
An important thread of work on this problem involves mapping named entities to
biomedicalontologies,especiallytheGeneOntology(Ashb urneretal.,2000). Thishas
provenconsiderablymorechallenging;termsinthe GeneOnt ologytendto belong,to
have many possible lexical and syntactic forms, and to somet imes require signiﬁcant
amountsofinference.
22.5.3 BiologicalRoles andRelations
Findingandnormalizingallthementionsofbiologicalenti tiesinatextisapreliminary
step to determining the roles played by entities in the text. Two ways to do this that
havebeenthefocusofrecentresearcharetodiscoverandcla ssifytheexpressedbinary
relations between the entities in a text, and to identify and classify the roles played
by entities with respect to the central events in the text. Th ese two tasks correspond

roughlytothetasksofclassifyingtherelationshipbetwee npairsofentitiesasdescribed
inSec. 22.2,andto thesemanticrolelabelingtaskintroduc edinCh. 20.
Considerthefollowingexampletextsthatexpressbinaryre lationsbetweenentities.
(22.25)TheseresultssuggestthatconA-induced[ DISEASEhepatitis]wasameliorated
bypretreatmentwith[ TREATMENT TJ-135].
(22.26)[DISEASEMalignantmesodermalmixedtumoroftheuterus]following
[TREATMENT irradiation]
Each of these examples asserts a relationship between a diseaseand atreatment . In

the ﬁrst example, the relationship can be classiﬁed as that o fcuring. In the second
example,thedisease isa resultofthementionedtreatment. RosarioandHearst(2004)
present a system for the classiﬁcation of 7 kinds disease-tr eatment relations. In this
work, a series of HMM-based generative models as well as a dis criminative neural
networkmodelweresuccessfullyapplied.
More generally, a wide-range of rule-based and statistical approaches have been

appliedtobinaryrelationrecognitionproblemssuchasthi s. Examplesofotherwidely
studied biomedical relation recognition problems include genes and their biological
functions(Blaschke et al., 2005),genesand drugs(Rindﬂes ch et al., 2000),genesand
mutations(Rebholz-Schuhmannetal.,2004),andprotein-p roteininteractions(Rosario
andHearst,2005).
Now consider the following example that corresponds to a sem antic role labeling
style ofproblem.
(22.27)[THEMEFull-lengthcPLA2]was[ TARGETphosphorylated]stoichiometrically

by[AGENTp42mitogen-activatedprotein(MAP)kinase]invitro... an dthe
majorsite ofphosphorylationwasidentiﬁedbyaminoacidse quencingas[ SITE
Ser505]
Thephosphorylation event that lies at the core of this text has three semantic rol es
associatedwithit: thecausal AGENToftheevent,the THEMEorentitybeingphospho-
rylated and ﬁnally the location, or SITEof the event. The problem is to identify the

DRAFTSection22.5. Advanced: BiomedicalInformationExtractio n∗773
constituents in the input that play these roles and assign th em the correct role labels.
Note that this example, contains a further complication in t hat the second event men-
tionphosphorylation must be identiﬁedas coreferringwith the ﬁrst phosphorylated in
ordertocapturethe SITErolecorrectly.
Muchof the difﬁcultywith semantic role labelingin the biom edicaldomainstems
from the preponderanceof nominalizationsin these texts. N ominalizations like phos-
phorylation typically offer fewer syntactic cues to signal their argume nts than their
verbal equivalents, making the identiﬁcation task more dif ﬁcult. A further compli-
cation is that different semantic roles arguments often occ ur as parts of the same, or
dominatingnominalconstituents. Tosee thisconsiderthef ollowingexamples.
(22.28)Serumstimulationofﬁbroblastsin ﬂoatingmatricesdoesno tresultin
[TARGET[ARG1ERK] translocation]tothe [ ARG3nucleus]andtherewas

decreasedserumactivationofupstreammembersofthe ERKsi gnaling
pathway,MEKandRaf,
(22.29)ThetranslocationofRelA/p65wasinvestigatedusingWeste rn blottingand
immunocytochemistry.theCOX-2inhibitorSC236workeddir ectlythrough
suppressing[ TARGET[ARG3nuclear]translocation]of[ ARG1RelA/p65].
(22.30)FollowingUV treatment,Mcl-1proteinsynthesisisblocked ,theexisting
poolofMcl-1proteinis rapidlydegradedbytheproteasome, and[ARG1[ARG2
cytosolic]Bcl-xL][ TARGETtranslocates]to the[ ARG3mitochondria]

Each these examplescontains argumentsthat are bundled int o constituents with other
arguments or with the target predicate itself. For example, in the second example the
constituent nucleartranslocation signalsboththe TARGETandthe ARG3 role.
Bothrule-basedandstatisticalapproacheshavebeenappli edtothesesemanticrole-
like problems. As with relation-ﬁnding and NER, the choice o f algorithm is less im-
portantthanthe choiceof features,manyof whichare derive dfromaccuratesyntactic

analyses. However,sincetherearenolargetreebanksavail ableforbiologicaltexts,we
are left with the option using off-the-shelf parsers traine d on generic newswire texts.
Ofcourse,theerrorsintroducedinthisprocessmaynegatew hateverpowerwecande-
rive from syntactic features. Therefore,an important area of research revolvesaround
theadaptationofgenericsyntactictoolstothisdomain(Bl itzeret al.,2006).
Relationalandeventextractionapplicationsinthisdomai noftenhaveanextremely

limited foci. The motivationfor this is that even systems wi th narrowscope can make
a contribution to the productivity of working bioscientist s. An extreme example of
this is the RLIMS-P system discussed earlier. It tackles onl y the verb phosphorylate
and the associated nominalization phosphorylization. Nevertheless, this system was
successfully used to produce a large online database that is in widespread use by the
researchcommunity.
Asthetargetsofbiomedicalinformationextractionapplic ationshavebecomemore

ambitious, the range of BioNLP application types has become correspondingly more
broad. Computational lexical semantics and semantic role l abelling (Verspoor et al.,
2003; Wattarujeekrit et al., 2004; Ogren et al., 2004; Kogan et al., 2005; Cohen and
Hunter, 2006), summarization (Lu et al., 2006), and questio n-answeringare all active
researchtopicsinthebiomedicaldomain. Sharedtaskslike BioCreativecontinuetobe
a source of large data sets for named entity recognition, que stion-answering, relation

DRAFT774 Chapter 22. InformationExtraction
extraction, and document classiﬁcation (Hirschman and Bla schke, 2006), as well as a
venueforhead-to-headassessmentofthebeneﬁtsofvarious approachestoinformation
extractiontasks.
22.6 Summary
Thischapterhasexploredaseriesoftechniquesforextract inglimitedformsofseman-
tic content from texts. Most techniquescan be characterize d as problemsin detection
followedbyclassiﬁcation.
•Named entities can be recognizedand classiﬁed by statistical sequence label-
ingtechniques.
•Relationsamongentities canbedetectedandclassiﬁedusingsupervisedlearn-
ing methods when annotated training data is available; ligh tly supervised boot-
strapping methods can be used when small numbers of seed tuples orseed
patterns areavailable.
•Reasoningabouttime can be facilitated by detectingandnor malizingtemporal
expressions through a combination of statistical learning and rule-bas ed meth-
ods.
•Rule-based and statistical methods can be used to detect, cl assify and order

ods.
eventsin time. The TimeBank corpus can facilitate the training and evalua-
tionoftemporalanalysissystems.
•Template-ﬁlling applicationscan recognizestereotypicalsituationsin te xtsand
assignelementsfromthetextto rolesrepresentedas ﬁxedsetsofslots .
•Informationextractiontechniqueshaveproventobepartic ularlyeffectiveinpro-
cessingtextsfromthe biologicaldomain .
•Scripts,plansandgoals...
Theearliestworkoninformationextractionaddressedthet emplate-ﬁllingtaskandwas

performedin the contextof the Frumpsystem (DeJong,1982a) . Later work was stim-
ulated by the U.S. government sponsored MUC conferences (Su ndheim, 1991, 1992,
1993, 1995b). Chinchor et al. (1993) describes the evaluati on techniques used in the
MUC-3 and MUC-4 conferences. Hobbs (1997) partially credit s the inspiration for
FASTUSto the success of the University of Massachusetts CIRCUSsystem (Lehnert
et al., 1991) in MUC-3. The SCISORsystem is another system based loosely on cas-

cadesandsemanticexpectationsthatdidwell inMUC-3(Jaco bsandRau,1990).
Due to the difﬁculty of reusing or porting systems from one do main to another,
attention shifted to the problem of automatic knowledge acq uisition for these sys-
tems. TheearliestsupervisedlearningapproachestoIEare describedinCardie(1993),

DRAFTSection22.6. Summary 775
Cardie (1994), Riloff (1993), Soderland et al. (1995), Huff man (1996), and Freitag
(1998).
Theseearlylearningeffortsfocusedonautomatingtheknow ledgeacquisitionpro-
cessformostlyﬁnite-staterule-basedsystems. Theirsucc ess,andtheearliersuccessof
HMM-basedmethodsforautomaticspeechrecognition,ledto thedevelopmentofsta-
tistical systems based onsequencelabeling. Early efforts applyingHMMs to IE prob-
lems include the work of Bikel et al. (1997,1999) and Freitag and McCallum (1999).
Subsequenteffortsdemonstratedthe effectivenessof a ran ge of statistical methodsin-
cluding MEMMs (McCallum et al., 2000), CRFs (Lafferty et al. , 2001) and SVMs
(SassanoandUtsuro,2000;McNameeandMayﬁeld,2002).
Progress in this area continues to be stimulated by formal ev aluations with shared
benchmark datasets. The MUC evaluations of the mid-1990s we re succeeded by the
AutomaticContentExtraction(ACE)programevaluationshe ldperiodicallyfrom2000

to 2007.6These evaluations focused on the named entity recognition, relation detec-
tion, and temporalexpression detectionand normalization tasks. Other IE evaluations
includethe2002and2003CoNLLsharedtasksonlanguage-ind ependentnamedentity
recognition(Sang,2002;SangandDeMeulder,2003),andthe 2007SemEvaltaskson
temporalanalysis(Verhagenet al.,2007)andpeoplesearch (Artileset al.,2007).
Thescopeofinformationextractioncontinuestoexpandtom eettheever-increasing
needs of applications for novel kinds of information. Some o f the emerging IE tasks

that we haven’t discussed include the classiﬁcation of gend er (Koppel et al., 2002),
moods (Mishne and de Rijke, 2006), sentiment, affect and opi nions (Qu et al., 2004).
Muchofthisworkinvolves usergeneratedcontent inthecontextof socialmedia suchUser generatedcontent
Social media as blogs, discussion forums, newsgroupsand the like. Resea rch results in this domain
have been the focusof a numberof recent workshopsand confer ences(Nicolovet al.,
2006;NicolovandGlance,2007).
Exercises

Exercises
22.1Develop a set of regular expressions to recognize the charac ter shape features
describedin Fig.22.7.
22.2Usingastatisticalsequencemodelingtoolkitofyourchoos ing,developandeval-
uateanNERsystem.
22.3TheIOBlabeling scheme given in this chapter isn’t the only possibl e one. For
example, an Etag might be added to mark the end of entities, or the Btag can
bereservedonlyforthosesituationswhereanambiguityexi stsbetweenadjacent
entities. Propose a new set of IOBtags for use with your NER system. Perform

experiments and compare its performance against the scheme presented in this
chapter.
22.4Names of works of art (books, movies, video games, etc.) are q uite different
fromthekindsofnamedentitieswe’vediscussedinthischap ter. Collectalistof
6www.nist.gov/speech/tests/ace/

DRAFT776 Chapter 22. InformationExtraction
names of works of art from a particular category from a web-ba sed source (eg.
gutenberg.org,amazon.com, imdb.com,etc.). Analyze your list and give exam-
ples of ways that the names in it are likely to be problematic f or the techniques
describedin thischapter.
22.5Develop an NER system speciﬁc to the category of names that yo u collected in
the last exercise. Evaluate your system on a collection of te xt likely to contain
instancesofthesenamedentities.
22.6Acronymexpansion,theprocessofassociatingaphrasewith aparticularacronym,
can be accomplished by a simple form of relational analysis. Develop a sys-
tem based on the relation analysis approaches described in t his chapter to pop-
ulate a database of acronym expansions. If you focus on Engli shThree Letter
Acronyms (TLAs)youcanevaluateyoursystem’sperformancebycompar ingit
toWikipedia’sTLApage( en.wikipedia.org/wiki/Category:Lists_of_TLAs).

22.7Collect a corpus of biographical Wikipedia entries of promi nent people from
some coherent area of interest (sports, business, computer science, linguistics,
etc.). Developasystemthatcanextractanoccupationaltim elineforthesubjects
ofthesearticles. Forexample,theWikipediaentryforPete rNorvigmightresult
in the ordering: Sun, Harlequin, Junglee, NASA, Google; the entry for David
Beckhamwouldbe: ManchesterUnited,Real Madrid,LosAngel esGalaxy.
22.8A useful functionality in newer email and calendar applicat ions is the ability

to associate temporal expressions associated with eventsi n emails (doctor’sap-
pointments, meeting planning, party invitations, etc.) wi th speciﬁc calendar en-
tries. Collectacorpusofemailscontainingtemporalexpre ssionsrelatedtoevent
planning. How do these expressions compare to the kind of exp ressions com-
monlyfoundinnewstextthatwe’vebeendiscussingin thisch apter?
22.9Developand evaluate a recognitionsystem capable of recogn izingtemporalex-
pressionsofthekindappearinginyouremailcorpus.

22.10Design a system capable of normalizing these expressions to the degree re-
quiredto insertthemintoastandardcalendaringapplicati on.
22.11AcquiretheCMUseminarannouncementcorpusanddevelopate mplate-ﬁlling
system using any of the techniques mentioned in Sec. 22.4. An alyze how well
yoursystem performsascomparedto state-of-the-artresul tsonthiscorpus.
22.12Developa newtemplate that coversa situationcommonlyrepo rtedonby stan-
dard news sources. Carefully characterize your slots in ter ms of the kinds of

entities that appear as slot-ﬁllers. Your ﬁrst step in this e xercise should be to
acquireareasonablysizedcorpusofstoriesthatinstantia teyourtemplate.
22.13Givenyourcorpus,developanapproachtoannotatingtherel evantslotsinyour
corpus so that it can serve as a training corpus. Your approac h should involve
somehand-annotation,butshouldnotbebasedsolelyonit.
22.14Retrainyoursystem andanalyzehowwell it functionsonyour newdomain.
22.15Species identiﬁcation is a critical issue for biomedical in formation extraction

applications such as document routing and classiﬁcation. B ut it is especially
crucialforrealisticversionsofthegenenormalizationpr oblem.

DRAFTSection22.6. Summary 777
Buildaspeciesidentiﬁcationsystemthatworksonthedocum entlevel,usingthe
machine learning or rule-based method of your choice. As gol d standard data,
usetheBioCreativegenenormalizationdata( biocreative.sourceforge.net ).
22.16Build, or borrow, a named entity recognition system that tar gets mentions of
genes and gene products in texts. As development data, use th e BioCreative
genementioncorpus( biocreative.sourceforge.net ).
22.17Build a gene normalizationsystem that maps the output of you r gene mention
recognitionsystem to the appropriate database entry. Use t he BioCreative gene
normalization data as your development and test data, be sure you don’t give
yoursystem accesstothe speciesidentiﬁcationinthe metad ata.

Chapter23
QuestionAnsweringandSummariza-
tion
‘Alright’,saidDeep Thought. ‘TheAnswer tothe Great Quest ion...’
‘Yes!’
‘OfLifeThe Universe and Everything...’ saidDeep Thought.
‘Yes!’
‘Is...’
‘Yes...!!!...?’
‘Forty-two’,said DeepThought, withinﬁnite majesty and ca lm...
Douglas Adams, The Hitchhiker’s Guide tothe Galaxy
IreadWarandPeace ...It’saboutRussia...
WoodyAllen, WithoutFeathers
Because so muchtext informationis available generallyon t he Web, or in special-
ized collections such as PubMed, or even on the hard drives of our laptops, the single

‘Yes!’
‘Yes!’
‘Is...’
most important use of language processing these days is to he lp us query and extract
meaningfromtheselargerepositories. Ifwehaveaverystru cturedideaofwhatweare
looking for, we can use the informationextraction algorith ms of the previouschapter.
But many times we have an information need that is best expres sed more informally
in words or sentences, and we want to ﬁnd either a speciﬁc answ er fact, or a speciﬁc
document,orsomethinginbetween.

In this chapter we introduce the tasks of question answering (QA) andsumma-
rization, tasks which produce speciﬁc phrases, sentences, or short p assages, often in
response to a user’s need for information expressed in a natu ral language query. In
studying these topics, we will also cover highlights from th e ﬁeld of information re-
trieval(IR), the task ofreturningdocumentswhichare relevantto a par ticularnatural
language query. IR is a complete ﬁeld in its own right, and we w ill only be giving a

brief introduction to it here, but one that is essential for u nderstand QA and summa-
rization.
In this chapter we focus on a central idea behind all of these s ubﬁelds, the idea of
meetingauser’sinformationneedsby extracting passagesdirectlyfromdocumentsor
fromdocumentcollectionsliketheWeb.
Information retrieval (IR) is an extremely broad ﬁeld, encompassing a wide-
rangeoftopicspertainingtothestorage,analysis,andret rievalofallmannerofmedia,

rization.
including text, photographs, audio, and video (Baeza-Yate s and Ribeiro-Neto, 1999).
Ourconcerninthischapterissolelywiththestorageandret rievaloftextdocumentsin
response to users’ word-basedqueries for information. In s ection 23.1 we present the
vectorspace model ,somevariantofwhichisusedin mostcurrentsystems, inclu ding
mostwebsearchengines.

DRAFT780 Chapter 23. QuestionAnsweringandSummarization
Rather than make the user read through an entire document, we ’d often prefer to
give a single concise short answer. Researchers have been tr ying to automate this
process of question answering since the earliest days of computational linguistics
(Simmons,1965).
The simplest form of question answering is dealing with factoid questions . As
thenameimplies,theanswerstofactoidquestionsaresimpl efactsthatcanbefoundin
shorttextstrings. Thefollowingarecanonicalexamplesof thiskindofquestion.
(23.1)WhofoundedVirginAirlines?
(23.2)What istheaverageageoftheonsetofautism?
(23.3)WhereisAppleComputerbased?
Eachofthese questionscan beanswereddirectlywitha texts tringthat containthe
name of person, a temporal expression, or a location, respec tively. Factoid questions,
therefore, are questions whose answers can be found in short spans of text and corre-
spondto a speciﬁc, easily characterized,category,oftena namedentity of thekindwe

discussed in Ch. 22. These answers may be found on the Web, or a lternatively within
some smaller text collection. For example a system might ans wer questions about a
company’sproduct line by searching for answers in document s on a particular corpo-
rate website or internal set of documents. Effective techni ques for answering these
kindsofquestionsaredescribedinSec. 23.2.
Sometimes we are seeking information whose scope is greater than a single fac-
toid, but less than an entire document. In such cases we might need asummary of

a document or set of documents. The goal of text summarization is to produce an
abridged version of a text which contains the important or re levant information. For
example we might want to generate an abstract of a scientiﬁc article, a summary of
email threads, a headline for a news article, or generate the short snippets that web Snippet
searchengineslike Googlereturntothe user todescribeeac hretrieveddocument. For
example, Fig. 23.1 shows some sample snippets from Google su mmarizing the ﬁrst
fourdocumentsreturnedfromthequery GermanExpressionismBr ¨ucke.

To producethese variouskindsof summaries, we’ll introduc ealgorithmsfor sum-
marizingsingledocuments,andthoseforproducingsummari esofmultipledocuments
bycombininginformationfromdifferenttextualsources.
Finally, we turn to a ﬁeld that tries to go beyond factoid ques tion answering by
borrowing techniques from summarization to try to answer mo re complex questions
likethe following:
(23.4)WhoisCelia Cruz?
(23.5)Whatisa Hajj?
(23.6)Inchildrenwith anacutefebrileillness, whatistheefﬁcac yof

single-medicationtherapywithacetaminophenoribuprofe ninreducingfever?
Answers to questions such as these do not consist of simple na med entity strings.
Rather they involve potentially lengthy coherent texts tha t knit together an array of
associated facts to produce a biography, a complete deﬁniti on, a summary of current
events,oracomparisonofclinicresultsonparticularmedi calinterventions. Inaddition
to the complexity and style differences in these answers, th e facts that go into such

answersmaybecontext,user,andtimedependent.

DRAFTSection23.1. InformationRetrieval 781
Figure23.1 The ﬁrst4snippets fromGoogle for GermanExpressionism Br¨ ucke .
Current methods answer these kinds of complex questions by piecing together Complex question
relevant text segments that come from summarizing longer do cuments. For example
we might construct an answer from text segments extracted fr om a corporate report,
a set of medical research journal articles, or a set of releva nt news articles or web
pages. Thisideaofsummarizingtextinresponsetoauserque ryiscalled query-based
summarization orfocusedsummarization ,andwill beexploredinSec. 23.5.Query-basedsummarization
Finally, we reserve for Ch. 24 all discussion of the role that questions play in ex-
tendeddialogues;thischapterfocusesonlyonrespondingt oa singlequery.
23.1 InformationRetrieval
Informationretrieval (IR)isagrowingﬁeldthatencompassesawiderangeoftopicsInformation
retrieval
IRrelated to the storageand retrievalof all mannerof media. T he focusof this section is

retrieval
with the storage of text documentsand their subsequentretr ieval in response to users’
requests for information. In this section our goal is just to give a sufﬁcient overview
of information retrieval techniques to lay a foundation for the following sections on
question answering and summarization. Readers with more in terest speciﬁcally in in-
formationretrievalshouldseethe referencesat theendoft hechapter.
Most currentinformationretrieval systems are based on a ki nd of extreme version

of compositional semantics in which the meaning of a documen t resides solely in the
setofwordsitcontains. TorevisittheMadHatter’squotefr omthebeginningofCh.19,
in these systems I see what I eat andI eat what I see mean precisely the same thing.
The ordering and constituency of the words that make up the se ntences that make up
documents play no role in determining their meaning. Becaus e they ignore syntactic
information,theseapproachesareoftenreferredtoas bag-of-words models. Bag-of-words

Before moving on, we need to introduce some new terminology. In information

DRAFT782 Chapter 23. QuestionAnsweringandSummarization
retrieval, a document refers generically to the unit of text indexed in the system a nd Document
availableforretrieval. Dependingontheapplication,a do cumentcanrefertoanything
fromintuitivenotionslikenewspaperarticles, orencyclo pediaentries,to smallerunits
suchasparagraphsandsentences. Inweb-basedapplication s,itcanrefertoawebpage,
apartofapage,ortoanentirewebsite. A collection referstoasetofdocumentsbeing Collection
usedtosatisfyuserrequests. A termreferstoalexicalitemthatoccursinacollection, Term
but it may also include phrases. Finally, a queryrepresentsa user’s information need Query
expressedasaset ofterms.
Thespeciﬁcinformationretrievaltaskthatwewillconside rindetailisknownas ad
hocretrieval . Inthistask,itisassumedthatanunaideduserposesaquery toaretrieval Adhoc retrieval
system,whichthenreturnsapossiblyorderedsetofpotenti allyusefuldocuments. The
highlevelarchitectureisshownin Fig.23.2.
Document
DocumentDocument
DocumentDocumentDocument
Query

Document
Query 
ProcessingIndexing
Search
(vector space or
probabilistic)Document
Document
Document
Document
DocumentRanked 
DocumentsDocument
Query
Figure23.2 The architecture of anad hoc IR system.
23.1.1 The VectorSpace Model
In thevector space model of information retrieval, documents and queries are rep-Vectorspace
model
resented as vectors of features representing the terms (wor ds) that occur within the
collection(Salton,1971).
Thevalueofeachfeatureiscalledthe termweight andisusuallyafunctionofthe Term weight
term’sfrequencyinthedocument,alongwith otherfactors.

Document
Document
Document
Query
model
Forexample,inafriedchickenrecipewefoundontheWebthef ourterms chicken,
fried,oil,andpepperoccurwith termfrequencies8, 2,7, and4, respectively. So i fwe
justusedsimpletermfrequencyasourweights,andassuming wepretendedonlythese
4 words occurred in the collection and we put the features are in the above order, the
vectorforthisdocument(call it j) wouldbe:
⃗dj= (8,2,7,4)
Moregenerally,we representavectorfora document djas
⃗dj= (w1,j,w2,j,w3,j,···,wn,j)

DRAFTSection23.1. InformationRetrieval 783
where ⃗djdenotes a particular document, and the vector contains a wei ght feature for
eachofthe Ntermsthatoccurinthecollectionasawhole; w2,jthusreferstotheweight
thatterm2hasindocument j.
We can also represent a query in the same way. For example, a qu eryqforfried
chickenwouldhavetherepresentation:
⃗q= (1,1,0,0)
Moregenerally,
⃗q= (w1,q,w2,q,w3,q,···,wn,q)
Note that N, the number of dimensions in the vector, is the total number o f terms
in the whole collection. This can be hundreds of thousands of words, even if (as is
oftendone)wedon’tconsidersomefunctionwordsintheseto fpossibleterms. Butof
course a query or even a long documentcan’t contain verymany of these hundredsof
thousands of terms. Thus most of the values of the query and do cument vectors will
bezero. Thusinpracticewedon’tactuallystoreallthezero s(weusehashesandother
sparserepresentations).
Now consider a differentdocument, a recipe for poachedchic ken; here the counts
are:
⃗dk= (6,0,0,0)

are:
Intuitively we’d like the query q fried chicken to match document dj(the fried
chicken recipe) rather than document dk(the poached chicken recipe). A brief glance
at thefeaturesuggeststhatthismightbe thecase; boththeq ueryandthefriedchicken
recipe have the words friedandchicken, while the poached chicken recipe is missing
theword fried.
It is useful to view the features used to represent documents and queries in this
model as dimensions in a multi-dimensional space, where the feature weights serve

to locate documents in that space. When a user’s query is tran slated into a vector it
denotes a point in that space. Documents that are located clo se to the query can then
bejudgedasbeingmorerelevantthandocumentsthatarefart heraway.
Fig. 23.3 shows a graphical illustration, plotting the ﬁrst two dimensions ( chicken
andfried) forall threevectors. Note that if we measurethe similarit ybetween vectors
by the angle between the vectors, that qis more similar to djthan todk, because the
anglebetween qanddjissmaller.

In vector-based information retrieval we standardly use th ecosinemetric that we Cosine
introducedinCh.20ratherthantheactualangle. Wemeasure thedistancebetweentwo
documentsbythe cosineofthe anglebetweentheirvectors. Whentwo documentsare
identicaltheywillreceiveacosineofone;whentheyareort hogonal(sharenocommon
terms)theywill receivea cosineofzero. Theequationforco sineis:
sim(⃗q,⃗dj) =∑N
i=1wi,q×wi,j√
∑N
i=1w2
i,q×√
∑N
i=1w2
i,j(23.7)
Recall from Ch. 20 that another way to think of the cosine ias a s thenormalized

∑N
i=1w2
i,q×√
∑N
i=1w2
i,j(23.7)
dotproduct . Thatis, the cosineisthe dotproductbetweenthetwo vector sdividedby

thedotproduct : Dot product
dot-product (⃗x,⃗y) =⃗x·⃗y=N
∑
i=1xi×ui (23.8)
while the denominatorof the cosine containsterms for the le ngths of the two vectors;
recallthat vectorlength isdeﬁnedas: Vectorlength
|⃗x|=√
N
∑
i=1x2
i(23.9)
This characterization of documents and queries as vectors p rovides all the basic
parts for an ad hoc retrieval system. A document retrieval sy stem can simply accept
a user’s query,create a vector representationfor it, compa reit against the vectorsrep-
resenting all known documents, and sort the results. The res ult is a list of documents
rankorderedbytheirsimilarityto thequery.
A further note on representation; the characterization of d ocuments as vectors of
termweightsallowsustoviewthedocumentcollectionasawh oleasa(sparse)matrix
of weights, where wi,jrepresents the weight of term iin document j. This weight
matrix is typically called a term-by-documentmatrix . Under this view, the columnsTerm-by-

documentmatrixof the matrix represent the documents in the collection, and the rows represent the
terms. Theterm-by-documentmatrixforthetworecipedocum entsabove(againusing
onlythe rawtermfrequencycountsasthe termweights)would be:
A=
8 6
2 0
7 0
4 0


DRAFTSection23.1. InformationRetrieval 785
23.1.2 Term Weighting
In the examples above, we assumed that the term weights were s et as the simple fre-
quencycountsofthe termsinthedocuments. Thisisasimpliﬁ cationofwhatwe doin
practice. The method used to assign terms weightsin the docu mentand queryvectors
has an enormous impact on the effectiveness of a retrieval sy stem. Two factors have
provento becritical in derivingeffectivetermweights. We havealreadyseen theﬁrst,
thetermfrequency,initssimplestformtherawfrequencyof atermwithinadocument
(Luhn,1957). This reﬂects the intuition that termsthat occ ur frequentlywithin a doc-
umentmayreﬂect its meaningmorestronglythan termsthat oc curless frequentlyand
shouldthushavehigherweights.
The second factor is used to give a higher weight to words that only occur in a
few documents. Terms that are limited to a few documents are u seful for discrim-
inating those documents from the rest of the collection, whi le terms that occur fre-

quently across the entire collection aren’t as helpful. doc uments. The inverse doc-
ument frequency orIDFterm weight (Sparck Jones, 1972) is one way of assigningInverse documentfrequency
IDFhigher weights to these more discriminative words. IDF is de ﬁned via the fraction
N/ni, whereNis the total number of documents in the collection, and niis the num-
ber of documents in which term ioccurs, The fewer documents a term occurs in, the
higher this weight. The lowest weight of 1 is assigned to term s that occur in all the

documents. Due to the large number of documents in many colle ctions, this measure
is usually squashed with a log function. The resulting deﬁni tion for inverse document
frequency(IDF)isthus:
idfi=log(N
ni)
(23.10)
CombiningtermfrequencywithIDFresultsinaschemeknowna stf-idfweighting: tf-idf
wi,j=tfi,j×idfi (23.11)
Intf-idfweighting,theweightofterm iinthevectorfordocument jistheproductofits
overallfrequencyin jwith the log of its inverse documentfrequencyin the collect ion
(sometimes the term frequency is logged as well). Tf-idf thu s prefers words which

ni)
(23.10)
are frequent in the current document jbut rare overall in the collection. Let’s repeat
the cosine formula for query-document comparison with tf-i df weights added. We’ll
modify the formula slightly, since as we noted earlier, most values for any query or
documentvectorwillbezero. Thismeansthatinpracticewed on’tcomputethecosine
by iterating over all the (mostly zero) dimensions. Instead we only compute over the
wordsthat are present, as suggested by the followingequati on for the tf-idf weighted
cosinebetweena query qanda document d:

sim(⃗q,⃗d) =∑
w∈q,dtfw,qtfw,d(idfw)2
√
∑
qi∈q(tfqi,qidfqi)2×√
∑
di∈d(tfdi,didfdi)2(23.12)
With some minor variations, this tf-idf weighting scheme is used to assign term
weights to documents in nearly all vector space retrieval mo dels. The tf-idf scheme

DRAFT786 Chapter 23. QuestionAnsweringandSummarization
is also used in many other aspects of language processing; we ’ll see it again when we
introduce summarization onpage809.
23.1.3 Term Selection and Creation
Thusfar,wehavebeenassumingthatitispreciselythewords thatoccurinacollection
that areusedto indexthedocumentsin thecollection. Twoco mmonvariationsonthis
assumptioninvolvetheuseof stemming , andastoplist.
Stemming , as we discussed in Ch. 3, is the process of collapsing the mor pholog- Stemming
ical variants of a word together. For example, without stemm ing, the terms process,
processing andprocessed will be treated as distinct items with separate term frequen -
cies in a term-by-documentmatrix; with stemmingtheywill b e conﬂatedto the single
termprocesswith a single summed frequency count. The major advantage to using
stemming is that it allows a particular query term to match do cumentscontaining any
ofthemorphologicalvariantsoftheterm. ThePorterstemme r(Porter,1980)described

inCh. 3isfrequentlyusedforretrievalfromcollectionsof Englishdocuments.
A problem with this approach is that it throws away useful dis tinctions. For ex-
ample, consider the use of the Porter stemmer on documents an d queries containing
the words stocksandstockings. In this case, the Porter stemmer reducesthese surface
forms to the single term stock. Of course, the result of this is that queries concerning
stock prices will return documents about stockings, and queries about stockings will

ﬁnd documents about stocks. Additionally we probably don’t want to stem, e.g., the
wordIllustrator toillustrate, since the capitalized form Illustrator tends to refer to
the software package. Most modern web search engines theref ore need to use more
sophisticatedmethodsforstemming.
Asecondcommontechniqueinvolvestheuseofstoplists, whi chaddresstheissue
of what words should be allowed into the index. A stop listis simply a list of high Stoplist

frequency words that are eliminated from the representatio n of both documents and
queries. Two motivationsare normallygivenforthis strate gy: highfrequency,closed-
classtermsareseenascarryinglittlesemanticweightanda rethusunlikelytohelpwith
retrieval, and eliminating them can save considerable spac e in the inverted index ﬁles
used to map from terms to the documents that contain them. The downside of using
a stop list is that it makes it difﬁcult to search for phrases t hat contain words in the

stoplist. Forexample,acommonstoplistpresentedinFrake sandBaeza-Yates(1992),
wouldreducethe phrase tobe ornottobe tothephrase not.
23.1.4 Evaluating InformationRetrieval Systems
The basic tools used to measure the performance of ranked ret rieval system are the
precision andrecallmeasures we employed in earlier settings. Here we assume tha t
the returned items can be divided into two categories: those relevant to our purposes
and those that are not. Therefore, precision is the fraction of the returned documents

that arerelevant,whilerecall is thefractionofall possib le relevantdocumentsthat are
containedinthereturnset. Moreformally,let’sassumetha twehavebeengivenatotal
ofTranked documents in response to a given information request , a subset of these
documents, R, consists of relevantdocuments,and a disjoint subset, N, consists of the

DRAFTSection23.1. InformationRetrieval 787
remainingirrelevantdocuments,andﬁnallylet’sassumeth atthereare Udocumentsin
the collection as a whole that are relevant to this particula r request. Given all this we
candeﬁneourprecisionandrecallmeasurestobe:
Precision =|R|
|T|(23.13)
Recall =|R|
|U|(23.14)
Unfortunately, these metrics are not quite sufﬁcient to mea sure the performance of a
systemthat ranksthedocumentsitreturn. Thatis,ifwearecomparingtheperf ormance
of two rankedretrieval systems, we requirea metric that wil l preferthe one that ranks
the relevant documents higher. Simple precision and recall as deﬁned above are not
dependent on rank in any way; we need to adapt them to capture h ow well a system
does at putting relevant documents higher in the ranking. Th e two standard methods
in information retrieval for accomplishing this are based o n plotting precision/recall
curvesandonaveragingprecisionmeasuresinvariousways.
Rank Judgment Precision RankRecallRank
1 R 1.0 .11
2 N .50 .11
3 R .66 .22

1 R 1.0 .11
2 N .50 .11
3 R .66 .22
4 N .50 .22
5 R .60 .33
6 R .66 .44
7 N .57 .44
8 R .63 .55
9 N .55 .55
10 N .50 .55
11 R .55 .66
12 N .50 .66
13 N .46 .66
14 N .43 .66
15 R .47 .77
16 N .44 .77
17 N .44 .77
18 R .44 .88
19 N .42 .88
20 N .40 .88
21 N .38 .88
22 N .36 .88
23 N .35 .88
24 N .33 .88
25 R .36 1.0
Figure23.4 Rank-speciﬁc precisionandrecallvalues calculatedaswep roceed downthrough
a set ofranked documents.

DRAFT788 Chapter 23. QuestionAnsweringandSummarization
Let’s consider each of these methods in turn using the data gi ven in the table in
Fig. 23.4. This table provides rank-speciﬁc precision and r ecall values calculated as
we proceed down through a set of ranked items. That is, the pre cision numbers are
the fraction of relevant documents seen at a given rank, and r ecall is the fraction of
relevant documents found at the same rank. The recall measur es in this example are
based on this query having 9 relevant documents in the collec tion as a whole. Note
thatrecallisnon-decreasingasweproceed,whenrelevanti temsareencounteredrecall
increasesandwhennon-relevantdocumentsarefounditrema insunchanged. Precision
on the other hand hops up and down, increasing when relevant d ocuments are found
anddecreasingotherwise.
One common way to get a handle on this kind of data is to plot pre cision against
recall on a single graph using data gathered from across a set of queries. To do this

we’llneedawaytoaveragetherecallandprecisionvaluesac rossaset ofqueries. The
standard way to do this is to plot averaged precision values a t 11 ﬁxed levels of recall
(0to100,instepsof10). Ofcourse,asisillustratedbyoure arliertablewe’renotlikely
to have datapointsat these exact levels for all (or any) of th e queriesin our evaluation
set. We’ll therefore use interpolated precision values for the 11 recall values fromInterpolatedprecision

the datapointswe dohave. Thisisaccomplishedbychoosingt he maximumprecision
value achieved at any level of recall at or above the one we’re calculating. In other
words,
IntPrecision (r) =max
i>=rPrecision (i) (23.15)
Note that this interpolation scheme not only provides us wit h the means to average
performance over a set of queries, but it also provides a sens ible way to smooth over
the irregularprecisionvaluesin theoriginaldata. Thispa rticularsmoothingmethodis
designedto givesystemsthe beneﬁtofthe doubtbyassigning the maximumprecision

words,
value achieved at higher levels of recall from the one being m easured. The interpo-
lateddata pointsforourearlierexamplearegivenin the fol lowingtableandplottedin
Fig.23.5.
InterpolatedPrecision Recall
1.0 0.0
1.0 .10
.66 .20
.66 .30
.66 .40
.63 .50
.55 .60
.47 .70
.44 .80
.36 .90
.36 1.0
Figure23.5 Interpolated data points from Fig.23.4.
Givencurvessuchasthiswecancomparetwosystemsorapproa chesbycomparing

DRAFTSection23.1. InformationRetrieval 789
Interpolated Precision Recall Curve
00.10.20.30.40.50.60.70.80.91
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
RecallPrecision
Figure23.6 An 11 point interpolated precision-recall curve. Precisio n at each of the 11 stan-
dard recall levels is interpolated for each query from the ma ximum at any higher level of recall.
The original measured precision recallpoints are alsoshow n.
their curves. Clearly curves that are higher in precision ac ross all recall values are
preferred. However,thesecurvescanalsoprovideinsighti ntotheoverallbehaviorofa
system. Systems that are higherin precision towardsthe lef t may favor precisionover
recall,whilesystemsthataremoregearedtowardsrecallwi llbehigherathigherlevels
ofrecall(totheright).
A second popular way to evaluate ranked retrieval systems is known as mean av-
erage precision (MAP). In this approach, we again descend throughthe ranked list ofMeanaverageprecision

items and note the precision only at those points where a rele vant item has been en-
countered. For a single query, we average these individual p recision measurements
overthe return set up to some ﬁxed cutoff. More formally,if w e assume that Rris the
set of relevant documents at or above r, then the average precision for a single query
is:
1
|Rr|∑
d∈RrPrecision r(d) (23.16)
wherePrecision r(d)is the precision measured at the rank where document dwas
found. For an ensemble of queries, we then average over these averages, giving us

is:
1
|Rr|∑
our mean average precision measure. Applying this techniqu e to the data in Fig. 23.5
yieldsa MAPmeasureof0.6forthissingle retrieval.
MAPhastheadvantageofprovidinga singlecrispmetricthat canbeusedtocom-
parecompetingsystemsorapproaches. Note,thatMAPwillte ndtofavorsystemsthat
provide relevant documentsat high ranks. Of course, this is n’t really a problem since

DRAFT790 Chapter 23. QuestionAnsweringandSummarization
thatisabigpartofwhatwe’relookingforinaretrievalsyst em. But sincethemeasure
essentially ignoresrecall, it can favor those systems that are tuned to returnsmall sets
ofdocumentsinwhichtheyarehighlyconﬁdent,attheexpens eofsystemsthatattempt
tobemorecomprehensivebytryingtoattainhigherlevelsof recall.
The U.S. government-sponsored TREC (Text REtrieval Confer ence) evaluations,
run annually since 1992, provide a rigorous testbed for the e valuation of a variety of
information retrieval tasks and techniques. TREC provides large document sets for
bothtrainingandtesting,alongwithauniformscoringsyst em. Trainingmaterialscon-
sist of sets of documentsaccompanied by sets of queries (cal led topics in TREC) and
relevancejudgments. TRECsubtasksovertheyearshaveincl udedquestionanswering,
IRin ChineseandSpanish,interactiveIR,retrievalfromsp eechandvideo,andothers.
See Voorhees and Harman (2005). Details of all of the meeting s can be found at the

TRECpageontheNationalInstituteofStandardsandTechnol ogywebsite.
23.1.5 Homonymy,Polysemy,and Synonymy
Since the vector space model is based solely on the use of simp le terms, it is use-
ful to consider the effect that various lexical semantic phe nomena may have on the
model. Consider a querycontaining the word canine, a word that has senses meaning
something like toothanddog. A query containing caninewill be judged similar to
documents making use of either of these senses. However, giv en that users are prob-

ably only interested in one of these senses, the documents co ntaining the other sense
will be judged non-relevant. Homonymyand polysemy, theref ore, can have the effect
ofreducing precision by leading a system to return documentsirrelevant to the use r’s
informationneed.
Nowconsideraqueryconsistingofthelexeme dog. Thisquerywillbejudgedclose
todocumentsthatmakefrequentuseoftheterm dog,butmayfailtomatchdocuments
that use close synonymslike canine, as well as documentsthat use hyponymssuch as
Malamute . Synonymyandhyponymy,therefore,canhavetheeffectof reducingrecall

bycausingtheretrievalsystemtomissrelevantdocuments.
Note that it is inaccurate to state ﬂatly that polysemy reduc es precision, and syn-
onymyreducesrecallsince,aswe discussonpage789,bothme asuresarerelativetoa
ﬁxedcutoff. Asaresult,everynon-relevantdocumentthatr isesabovethecutoffdueto
polysemytakesupaslotintheﬁxedsizereturnset,andmayth uspusharelevantdocu-
mentbelowthreshold,thusreducingrecall. Similarly,whe na documentis misseddue
tosynonymy,aslotisopenedinthereturnsetforanon-relev antdocument,potentially
reducingprecisionaswell.

These issues lead naturally to the question of whether or not word sense disam-
biguationcanhelpininformationretrieval. Thecurrentev idenceonthispointismixed,
withsomeexperimentsreportingagainusingdisambiguatio n-liketechniques(Sch¨ utze
and Pedersen, 1995), and others reporting either no gain, or a degradation in perfor-
mance(KrovetzandCroft,1992;Sanderson,1994;Voorhees, 1998).

DRAFTSection23.1. InformationRetrieval 791
23.1.6 Improving UserQueries
One of the most effective ways to improve retrieval performa nce is to ﬁnd a way to
improve user queries. The techniques presented in this sect ion have been shown to
varyingdegreesto beeffectiveatthistask.
Thesinglemosteffectivewaytoimproveretrievalperforma nceinthevectorspace
model is the use of relevance feedback (Rocchio, 1971). In this method, a userRelevancefeedback
presents a query to the system and is presented with a small se t of retrieved docu-
ments. The useris thenaskedto specifywhichof thesedocume ntsappearsrelevantto
their need. The user’s original query is then reformulatedb ased on the distribution of
terms in the relevant and non-relevant documents that the us er examined. This refor-
mulated query is then passed to the system as a newquery with the new results being
shown to the user. Typically an enormousimprovementis seen after a single iteration
ofthistechnique.

The formal basis for the implementation of this technique fa lls out directly from
some of the basic geometric intuitions of the vector model. I n particular, we would
liketopushthevectorrepresentingtheuser’soriginalquerytowardth edocumentsthat
havebeenfoundtoberelevant,andawayfromthedocumentsju dgednotrelevant. This
canbeaccomplishedbyaddinganaveragedvectorrepresenti ngtherelevantdocuments
to the originalquery,and subtractingan averagedvectorre presentingthe non-relevant
documents.
More formally, let’s assume that ⃗qirepresents the user’s original query, Ris the

numberofrelevantdocumentsreturnedfromtheoriginalque ry,Sisthenumberofnon-
relevant documents, and documents in the relevant and non-r elevant sets are denoted
as⃗rand⃗s, respectively. In addition, assume that βandγrange from 0 to 1 and that
β+γ=1. Given these assumptions, the following represents a stan dard relevance
feedbackupdateformula:
⃗qi+1=⃗qi+β
RR
∑
j=1⃗rj−γ
SS
∑
k=1⃗sk
Thefactors βandγinthisformularepresentparametersthatcanbeadjustedex per-

RR
∑
j=1⃗rj−γ
SS
∑
k=1⃗sk
imentally. Intuitively, βrepresents how far the new vector should be pushed towards
the relevant documents, and γrepresents how far it should be pushed away from the
non-relevant ones. Salton and Buckley (1990) report good re sults with β=.75 and
γ=.25.
Weshouldnotethatevaluatingsystemsthatuserelevancefe edbackisrathertricky.
In particular, an enormous improvement is often seen in the d ocuments retrieved by
the ﬁrst reformulated query. This should not be too surprisi ng since it includes the

γ=.25.
documents that the user told the system were relevant on the ﬁ rst round. The pre-
ferred way to avoid this inﬂation is to only compute recall an d precision measuresfor
what is called the residual collection , the original collectionwithout any of the docu-Residualcollection
ments shown to the user on any previousround. This usually ha s the effect of driving
the system’s raw performancebelow that achieved with the ﬁr st query, since the most
highlyrelevantdocumentshavenowbeeneliminated. Nevert heless,thisisaneffective

techniquetousewhencomparingdistinctrelevancefeedbac kmechanisms.

DRAFT792 Chapter 23. QuestionAnsweringandSummarization
An alternative approachto query improvementfocuseson ter ms that comprise the
query vector. In query expansion , the user’s original query is expanded by adding Query expansion
terms that are synonymous with or related to the original ter ms. Query expansion is
thusatechniqueforimprovingrecall,perhapsattheexpens eofprecision. Forexample
the query Steve Jobs could be expanded by adding terms like Apple,Macintosh , and
personalcomputer .
The terms to be added to the query are taken from a thesaurus . It is possible to Thesaurus
useahand-builtresourcelikeWordNetorUMLSasthethesaur usforqueryexpansion,
when the domain is appropriate. But often these thesauruses are not suitable for the
collection,and instead, we do thesaurus generation ,generatinga thesaurusautomat-Thesaurusgeneration
ically from documentsin the collection. We can do this by clu stering the wordsin the

collection, a method known as term clustering . Recall from our characterization of Term clustering
the term-by-documentmatrix that the columns in the matrix r epresent the documents
and the rows representthe terms. Thus, in thesaurusgenerat ion,the rows can be clus-
teredtoformsetsofsynonyms,whichcanthenbeaddedtotheu ser’soriginalqueryto
improve its recall. The distance metric for clustering can b e simple cosine, or any of
theotherdistributionalmethodsforwordrelatednessdisc ussedin Ch.20.

The thesaurus can be generated once from the document collec tion as a whole
(CrouchandYang,1992),orsetsofsynonym-liketermscanbe generateddynamically
from the returned set for the original query (Attar and Fraen kel, 1977). Note that this
second approach entails far more effort, since in effect a sm all thesaurus is generated
forthedocumentsreturnedforeveryquery,ratherthanonce forthe entirecollection.
23.2 FactoidQuestionAnswering
Therearemanysituationswheretheuserwantsaparticularp ieceofinformationrather

thananentiredocumentordocumentset. Weusetheterm questionanswering forthe
taskofreturningaparticularpieceofinformationtotheus erinresponsetoaquestion.
We call the task factoid question answering if the information is a simple fact, and
particularly if this fact has to do with a named entity like a person, organization, or
location.
The task of a factoid question answering system is thus to ans wer questions by
ﬁnding,eitherfromtheWeborsomeothercollectionofdocum ents,shorttextsegments

location.
thatarelikelytocontainanswerstoquestions,reformatti ngthem,andpresentingthem
totheuser. Fig.23.7showssomesamplefactoidquestionsto getherwiththeiranswers.
Since factoid question answering is based on information re trieval techniques to
ﬁnd these segments, it is subject to the same difﬁcultiesas i nformationretrieval. That
is, the fundamentalproblemin factoid questionansweringi s the gap betweenthe way
thatquestionsareposedandthewaythatanswersareexpress edinatext. Considerthe

followingquestion/answerpairfromtheTRECquestionansw eringtask:
User Question: What companysellsthemostgreetingcards?
PotentialDocumentAnswer: Hallmarkremainsthelargestmakerofgreetingcards.
Here the user uses the verbal phrase sells the most while the document segment

DRAFTSection23.2. FactoidQuestionAnswering 793
Question Answer
WhereistheLouvreMuseumlocated? inParis, France
What’stheabbreviationforlimitedpartnership? L.P.
What arethenamesofOdin’sravens? HuginnandMuninn
What currencyisusedinChina? theyuan
What kindofnutsareusedinmarzipan? almonds
What instrumentdoesMaxRoachplay? drums
What’stheofﬁciallanguageofAlgeria? Arabic
What is the telephone number for the University of
Colorado,Boulder?(303)492-1411
Howmanypoundsarethereina stone? 14
Figure23.7 Some sample factoid questions andtheir answers.
uses a nominal the largest maker . The solution to the possible mismatches between
questionandanswerformliesintheabilitytorobustlyproc essbothquestionsandcan-
didate answer texts in such a way that a measure of similarity between the question
andputativeanswerscanbeperformed. Aswe’llsee,thispro cessinvolvesmanyofthe
techniquesthat we haveintroducedin earlierchaptersincl udinglimitedformsof mor-

phologicalanalysis, part-of-speechtagging,syntactic p arsing, semantic role labelling,
named-entityrecognition,andinformationretrieval.
Becauseit is impracticalto employthese relativelyexpens iveNLPtechniqueslike
parsing or role labeling on vast amounts of textual data, que stion answering systems
generally use information retrieval methods to ﬁrst retrie ve a smallish number of po-
tential documents. Themostexpensivetechniquesthenused in a secondpassonthese
smallernumbersofcandidaterelevanttexts.

Fig. 23.8 shows the three phases of a modern factoid question answering system:
questionprocessing,passageretrievalandranking,andan swerprocessing.
Passage
RetrievalDocument
Question 
ProcessingIndexingDocument
QuestionQuery 
Formulation
Question 
ClassiﬁcationPassage 
Retrieval passagesDocume
ntDocume
ntDocume
ntDocume
ntDocume
ntRelevant
DocsAnswer 
ProcessingAnswer
Document 
Retrieval

Passage
Question 
Question 
ntDocume
ntDocume
ntDocume
ntDocume
Document 
Retrieval
Figure23.8 The 3 stages of a generic question answering system: questio n processing, passage retrieval, and
answer processing..

DRAFT794 Chapter 23. QuestionAnsweringandSummarization
23.2.1 QuestionProcessing
Thegoalofthe questionprocessingphaseis toextracttwo th ingsfromthe question: a
keywordquerysuitable as input to an IR system and an answer type , a speciﬁcation
ofthekindofentitythat wouldconstituteareasonableansw ertothequestion.
QueryFormulation
The process of query formulation is very similar to the processing done on other IR
queries. Our goal is to create from the question a list of keyw ords that forms an IR
query.
Exactly what query to form depends on the question answering application. If
question answering is applied to the Web, we might simply cre ate a keyword from
every word in the question, letting the web search engine aut omatically remove any
stopwords. Often we leave out the question word ( where,when, etc). Alternatively,
keywords can be formed from only the terms found in the noun ph rases in the ques-
tion,applyingstopwordliststoignorefunctionwordsandh igh-frequency,low-content
verbs.

query.
verbs.
When question answering is applied to smaller sets of docume nts, for example to
answerquestionsaboutcorporateinformationpages,westi lluseanIRenginetosearch
ourdocumentsforus. Butforthissmallersetofdocumentswe generallyneedtoapply
queryexpansion. Onthe Web theanswer toa questionmightapp earinmanydifferent
forms,andsoifwesearchwithwordsfromthequestionwe’llp robablyﬁndananswer
written in the same form. In smaller sets of corporate pages, by contrast, an answer

verbs.
might appear only once, and the exact wording might look noth ing like the question.
Thus query expansion methods can add query terms hoping to ma tch the particular
formoftheanswerasit appears.
Thuswe mightaddto the queryall morphologicalvariantsof t he contentwordsin
the question, as well as applying the thesaurus-based or oth er query expansion algo-
rithms discussed in the previous section to get a larger set o f keywords for the query.
Many systems use WordNet as a thesaurus, while others rely on special-purpose the-

saurusesthatarespeciﬁcallyhand-builtforquestion-ans wering.
Anotherquery formulationapproachthat is sometimesused w hen questioningthe
Webistoapplyasetof queryreformulation rulestothequery. TherulesrephrasetheQuery
reformulation
question to make it look like a substring of possible declara tiveanswers. For example
the question “when was the laser invented?” would be reformulated as the laser was
invented; the question “where is the Valley of the Kings?” might be reformulated as
“the Valley of the Kings is locatedin” . We can applymultiple such rulesto the query,

andpassalltheresultingreformulatedqueriestothewebse archengine. Herearesome
samplehand-writtenreformulationrulesfromLin(2007):
(23.17)wh-word didAverbB→...Averb+edB
(23.18)WhereisA→A islocatedin
QuestionClassiﬁcation
The second task in question processing is to classify the que stion by its expected an-
swer type . For example a question like “Who founded Virgin Airlines” expects an Answer type

DRAFTSection23.2. FactoidQuestionAnswering 795
answer of type PERSON. A question like “What Canadian city has the largest popu-
lation?” expects an answer of type CITY. This task is called question classiﬁcationQuestion
classiﬁcation
oranswer type recognition . Ifwe knowthe answertype fora question,we canavoidAnswer typerecognition
looking at every sentence or noun phrase in the entire suite o f documents for the an-
swer, instead focusing on, e.g., just people or cities. Know ing an answer type is also
important for presenting the answer. A DEFINITION question like “What is a prism”
might use a simple answer template like “A prism is...” while an answer to a BIOG-
RAPHYquestion like “Who is Zhou Enlai?” might use a biography-speciﬁctemplate,
perhaps beginning with the persons nationality and proceed ing to their dates of birth
andotherbiographicalinformation.
Assome ofthe aboveexamplessuggest,we mightdrawthe set of possibleanswer
typesforaquestionclassiﬁerfromasetofnamedentitiesli kethePERSON,LOCATION ,

andORGANIZATION described in Ch. 22. Usually, however, a somewhat richer set of
answer types is used. These richer tagsets are often hierarc hical, and so we usually
call them an answer type taxonomy or aquestion ontology . Such taxonomies canAnswer typetaxonomy
Questionontology be built semi-automatically and dynamically, for example f rom WordNet (Harabagiu
et al.,2000;Pasca,2003),ortheycanbedesignedbyhand.
Fig.23.2.1showsonesuchhand-builtontology,thehierarc hicalLiandRoth(2005)

tagset. In this tagset, each question can be labeled with a co arse-grained tag like HU-
MAN,oraﬁne-grainedtaglike HUMAN:DESCRIPTION ,HUMAN:GROUP,HUMAN:IND,
and so on. Similar tags are used in other systems; the type HUMAN:DESCRIPTION is
often called a BIOGRAPHY question, because the answer requires giving a brief biog-
raphyofthe person,ratherthanjust a name.
Question classiﬁers can be built by hand-writing rules, via supervised machine
learning,orvia somecombination. TheWebclopediaQATypol ogy,forexample,con-

tains276hand-writtenrulesassociatedwiththeapproxima tely180answertypesinthe
typology (Hovy et al., 2002). A regular expression rule for d etecting an answer type
likeBIOGRAPHY (which assumes the question has been named-entity tagged) m ight
be:
(23.19)who{is|was|are|were}PERSON
Mostmodernquestionclassiﬁers,however,arebasedonsupe rvisedmachinelearn-
ing techniques. These classiﬁers are trained on databases o f questions that have been
hand-labeled with an answer type such as the corpus of Li and R oth (2002). Typical

be:
features used for classiﬁcation include the words in the que stions, the part-of-speech
ofeachword,andnamedentitiesin thequestions.
Oftena single wordin the questiongivesextra informationa boutthe answer type,
and its identity is used as a feature. This word is sometimesc alled the question head-
wordor theanswer type word , and may be deﬁned as the headword of the ﬁrst NP
after the question’s wh-word; headwords are indicated in boldface in the following
examples:
(23.20)WhichcityinChinahasthe largestnumberofforeignﬁnancialcompanie s?

examples:
(23.21)Whatisthe state ﬂowerofCalifornia?
Finally,itoftenhelpstousesemanticinformationaboutth ewordsinthequestions.
The WordNet synset id of the word can be used as a feature, as ca n the ids of the
hypernymandhyponymsofeachwordinthequestion.

DRAFT796 Chapter 23. QuestionAnsweringandSummarization
Tag Example
ABBREVIATION
abb What’s the abbreviation for limited partnership?
exp What does the “c” stand for in the equation E=mc2?
DESCRIPTION
deﬁnition What aretannins ?
description What arethe words to theCanadian National anth em?
manner How can you get ruststains out of clothing?
reason What caused the Titanic to sink ?
ENTITY
animal What arethe names of Odin’s ravens?
body What partof your body contains the corpus callosum ?
color What colors makeup arainbow ?
creative In what book can I ﬁnd thestory of Aladdin?
currency What currency isused in China?
disease/medicine What does Salk vaccine prevent ?
event What war involved the battle of Chapultepec?
food What kind of nuts areused in marzipan?
instrument What instrument does Max Roach play?
lang What’s the ofﬁcial language of Algeria?
letter What letter appears on the cold-water tap in Spain?
other What is thename of King Arthur’s sword?
plant What aresomefragrant white climbing roses?

product What is thefastest computer ?
religion What religion has the mostmembers ?
sport What was the nameof the ball gameplayed by the Mayans?
substance What fueldo airplanes use?
symbol What is thechemical symbolfor nitrogen ?
technique What is thebest way to remove wallpaper?
term How do you say “Grandma” in Irish ?
vehicle What was the nameof Captain Bligh’s ship ?
word What’s the singular of dice?
HUMAN
description Who was Confucius?
group What arethe major companies that arepart of Dow Jones ?

HUMAN
ind Who was the ﬁrstRussian astronaut to do aspacewalk?
title What was Queen Victoria’s title regarding India?
LOCATION
city What’s the oldest capital city in the Americas ?
country What country borders the mostothers?
mountain What is thehighest peak in Africa?
other What river runs through Liverpool?
state What states do nothave state income tax?
NUMERIC
code What is thetelephone number for theUniversity of Color ado?
count About how many soldiers died in World WarII?
date What is thedate ofBoxing Day?

LOCATION
NUMERIC
distance How long was Mao’s 1930s Long March?
money How much did aMcDonald’s hamburger costin 1963?
order Where does Shanghai rank among world cities in populat ion?
other What is thepopulation of Mexico?
period What was the average life expectancy during the Stone Age?
percent
speed What is thespeed of the Mississippi River?
temp How fast mustaspacecraft travel to escape Earth’s grav ity?
size What is thesize of Argentina?
weight How many pounds are there in astone?

percent
Figure23.9 Question typology from Li and Roth (2002, 2005). Example sen tences are from
theircorpusof5500labeledquestions. Aquestioncanbelab eledeitherwithacoarse-grainedtag
likeHUMANorNUMERIC , or a ﬁne-grained tag like HUMAN:DESCRIPTION ,HUMAN:GROUP,
HUMAN:IND,and soon.

DRAFTSection23.2. FactoidQuestionAnswering 797
In general question classiﬁcation accuracies are relative ly high on easy question
typeslike PERSON,LOCATION ,andTIMEquestions;detecting REASONandDESCRIP-
TIONquestionscanbemuchharder.
23.2.2 PassageRetrieval
The query that was created in the question processing phase i s next used to query
an information retrieval system, either a general IR engine over a proprietary set of
indexeddocumentsorawebsearchengine. Theresultofthisd ocumentretrievalstage
isa set ofdocuments.
Although the set of documents is generally ranked by relevan ce, the top-ranked
document is probably not the answer to the question. This is b ecause documents are
notanappropriateunittorankwithrespecttothegoalsofaq uestionansweringsystem.
A highly relevant and large document that does not prominent ly answer a question is
notanidealcandidateforfurtherprocessing.
Therefore, the next stage is to extract a set of potential ans wer passages from the

retrievedsetofdocuments. Thedeﬁnitionofapassageisnec essarilysystemdependent,
butthetypicalunitsincludesections,paragraphsandsent ences. Forexample,wemight
run a paragraph segmentation algorithm of the type discusse d in Ch. 21 on all the
returneddocumentsandtreateachparagraphasa segment.
We next perform passage retrieval . In this stage we ﬁrst ﬁlter out passages in the Passage retrieval
returneddocumentsthatdon’tcontainpotentialanswers,a ndthenranktherestaccord-

ing to how likely they are to contain an answer to the question . The ﬁrst step in this
processistorunanamedentityoranswer-typeclassiﬁcatio nontheretrievedpassages.
The answer type that we determined from the question tells us the possible answer
types(extendednamedentities)weexpecttosee intheanswe r. We canthereforeﬁlter
outdocumentsthat don’tcontainanyentitiesoftherightty pe.
Theremainingpassagesarethenranked;eitherviahand-cra ftedrulesorsupervised
training with machine learning techniques. In either case, the ranking is based on

a relatively small set of features that can be easily and efﬁc iently extracted from a
potentiallylargenumberofanswerpassages. Amongthemore commonfeaturesare:
•Thenumberof namedentities oftherighttypeinthe passage
•Thenumberof questionkeywords inthepassage
•Thelongestexactsequenceofquestionkeywordsthat occurs inthe passage
•Therankofthedocumentfromwhichthepassagewasextracted
•Theproximity ofthekeywordsfromtheoriginalquerytoeachother:
For each passage identify the shortest span that covers the k eywords contained

in that passage. Prefer smaller spans that include more keyw ords (Pasca, 2003;
Monz,2004).
•TheN-gramoverlap betweenthepassageandthequestion:
Count the N-grams in the question and the N-grams in the answer passages.
Prefer the passages with higher N-gram overlap with the question (Brill et al.,
2002).
For question answering from the Web, instead of extracting p assages from all the
returneddocuments,wecanrelyonthewebsearchtodopassag eextractionforus. We

DRAFT798 Chapter 23. QuestionAnsweringandSummarization
dothisbyusing snippetsproducedbythewebsearchengineasthereturnedpassages.
For example, Fig. 23.10 shows some snippets for the ﬁrst 5 doc ument returned from
theGooglesearchengineforthequery Whenwasmovabletypemetalprintinginvented
inKorea?
Figure23.10 Five snippets from Google in response to the query When was movable type
metal printinginvented inKorea?
23.2.3 AnswerProcessing
The ﬁnal stage of question answeringis to extract a speciﬁc a nswer fromthe passage,
so as to be able to present the user with an answer like 300 million to the question
“Whatisthe currentpopulationof theUnitedStates” .
Two classes of algorithms have been applied to the answer ext raction task, one
basedon answer-typepatternextraction andonebasedon N-gramtiling .
Inthepatternextraction methodsforanswerprocessing,weuseinformationabout
the expected answer type together with regular expression p atterns. For example, for

questionswitha HUMANanswertypeweruntheanswertypeornamedentitytaggeron

DRAFTSection23.2. FactoidQuestionAnswering 799
the candidatepassage orsentence, andreturnwhateverenti tyis labeled with type HU-
MAN. Thusinthefollowingexamples,theunderlinednamedentit iesareextractedfrom
thecandidateanswerpassagesastheanswertothe HUMANandDISTANCE -QUANTITY
questions:
“Whoistheprime ministerofIndia”
ManmohanSingh , Prime Minister of India, had told left leaders that the
dealwouldnotberenegotiated.
“HowtallisMt. Everest?
TheofﬁcialheightofMountEverest is29035feet
Unfortunately,theanswerstosomequestions,suchas DEFINITION questions,don’t
tendtobeofaparticularnamedentitytype. Forsomequestio ns,then,insteadofusing
answertypes,weusehandwrittenregularexpressionpatter nstohelpextracttheanswer.
These patternsare also useful in cases where a passage conta insmultiple examplesof
thesamenamedentitytype. Fig.23.2.3showssomepatternsf romPasca(2003)forthe
questionphrase(QP)andanswerphrase(AP)ofdeﬁnitionque stions.
Pattern Question Answer

<AP>such as <QP>Whatis autism? ”,developmental disorders such as autism”
<QP>(an<AP>) Whatis a caldera? ”the Long Valley caldera, a volcanic crat er19
mileslong”
Figure23.11 Some answer extractionpatterns for deﬁnitionquestions (P asca,2003).
The patterns are speciﬁc to each question type, and can eithe r be written by hand
orlearnedautomatically.
TheautomaticpatternlearningmethodofRavichandranandH ovy(2002),Echihabi
et al. (2005),for example,makesuse of the pattern-basedme thodsfor relationextrac-

tion we introduced in Ch. 20 and Ch. 22 (Brin, 1998; Agichtein and Gravano, 2000).
The goal of the pattern learning method is to learn a relation between a particular an-
swertypesuchas YEAR-OF-BIRTH,andaparticularaspectofthequestion,inthiscase
the nameof the person whose birth year we want. We are thustry ing to learn patterns
which are good cues for a relation between two phrases ( PERSON-NAME/YEAR-OF-
BIRTH, orTERM-TO-BE-DEFINED/DEFINITION , etc). This task is thus very similar to

thetaskoflearninghyponym/hyponymrelationsbetweenWor dNetsynsetsintroduced
in Ch. 20 , or learningACE relations between words from Ch. 22 . Here is a sketch of
thealgorithmasappliedtoquestion-answerrelationextra ction:
1. For a given relation between two terms (i.e. person-name →year-of-birth), we
start with a hand-built list of correct pairs (e.g., “gandhi :1869”,“mozart:1756”,
etc).
2. NowquerytheWebwithinstancesofthesepairs(e.g.,”gan dhi”and”1869”,etc)
andexaminethetopX returneddocuments.

etc).
3. Break each document into sentences, and keep only sentenc es containing both
terms(e.g., PERSON-NAMEandBIRTH-YEAR).
4. Extracta regularexpressionpatternrepresentingthewo rdsandpunctuationthat
occurbetweenandaroundthetwo terms.

DRAFT800 Chapter 23. QuestionAnsweringandSummarization
5. Keepall patternsthataresufﬁcientlyhigh-precision.
In Ch. 20 and Ch. 22 we discussed various ways to measure accur acy of the pat-
terns. Amethodusedinquestion-answerpatternmatchingis tokeeppatternswhichare
high-precision . Precision is measured by performing a query with only the qu estion
terms, but not the answer terms (i.e. query with just “gandhi ” or “mozart”). We then
runtheresultingpatternsonthesentencesfromthedocumen t,andextractabirth-date.
Since we know the correct birth-date, we can compute the perc entage of times this
patternproduceda correctbirthdate. Thispercentageisth eprecisionofthepattern.
For the YEAR-OF-BIRTHanswer type, this method learns patterns like the follow-
ing:
<NAME> (<BD>-<DD>)
<NAME> (<BD>-<DD>),
<NAME> was born on <BD>
These two methods, named entity detection and question-ans wer pattern extrac-
tion, are still not sufﬁcient for answer extraction. Not eve ry relation is signaled by

ing:
unambiguous surrounding words or punctuation, and often mu ltiple instances of the
same named-entity type occur in the answer passages. The mos t successful answer-
extractionmethodisthustocombineallthesemethods,usin gthemtogetherwithother
informationas featuresin a classiﬁer that ranks candidate answers. We extract poten-
tial answers using named entities or patterns or even just lo oking at every sentence
returnedfrom passage retrieval, and rank them using a class iﬁer with features like the

Answer typematch: Trueifthe candidateanswer containsa phrasewith the corre ct
answertype.
Patternmatch: Theidentityofapatternthat matchesthecandidateanswer.
Number of matched question keywords: How many question keywords are con-
tainedin thecandidateanswer.
Keyworddistance: The distancebetweenthe candidateanswer andquerykeyword s
(measuredinaveragenumberofwords,orasthenumberofkeyw ordsthatoccur
inthesamesyntacticphraseasthecandidateanswer.

Noveltyfactor: True if at least one wordin the candidateansweris novel,i.e . not in
thequery.
Apposition features: True if the candidate answer is an appositive to a phrase con-
taining many question terms. Can be approximated by the numb er of question
termsseparatedfromthecandidateanswerthroughat most th reewordsandone
commaPasca(2003).
Punctuation location: True if the candidate answer is immediately followed by a
comma,period,quotationmarks,semicolon,orexclamation mark.
Sequences of question terms: The lengthof the longest sequence of question terms

thequery.
thatoccursin thecandidateanswer.
Analternativeapproachtoanswerextraction,usedsolelyi nwebsearch,isbasedon
N-gramtiling ,sometimescalledthe redundancy-basedapproach (Brill et al.,2002; N-gram tiling
Lin, 2007). This simpliﬁed method begins with the snippets r eturned from the web

DRAFTSection23.3. Summarization 801
search engine, produced by a reformulated query. In the ﬁrst step of the method, N-
grammining ,everyunigram,bigram,andtrigramoccurringinthesnippe tisextracted N-gram mining
andweighted. Theweightisafunctionofthenumberofsnippe tstheN-gramoccurred
in, and the weight of the query reformulation pattern that re turned it. In the N-gram
ﬁlteringstep,N-grams are scored by how well they match the predictedanswer type. N-gram ﬁltering
These scores are computedby hand-writtenﬁlters built for e ach answer type. Finally,
anN-gram tiling algorithm concatenates overlapping N-gram fragments into longer
answers. A standard greedy method is to start with the highes t-scoringcandidate and
try to tile each other candidate with this candidate. The bes t scoring concatenation is
addedtothesetofcandidates,thelowerscoringcandidatei sremoved,andtheprocess
continuesuntila singleanswerisbuilt.
For any of these answer extraction methods, the exact answer phrase can just be

presented to the user by itself. In practice, however, users are rarely satisﬁed with an
unadorned number or noun as an answer; they prefer to see the a nswer accompanied
byenoughpassageinformationtosubstantiatetheanswer. T husweoftengivetheuser
anentirepassagewiththeexactanswerinsideit highlighte dorboldfaced.
23.2.4 EvaluationofFactoidAnswers
A wide varietyof techniqueshave beenemployedto evaluateq uestionansweringsys-
tems. Byfarthemostinﬂuentialevaluationframeworkhasbe enprovidedbytheTREC
Q/A trackﬁrst introducedin 1999.

The primary measure used in TREC is an intrinsic orin vitroevaluation metric
known as mean reciprocal rank , orMRR. As with the ad hoc information retrievalMeanreciprocal
rank
MRRtaskdescribedinSec.23.1,MRRassumesatestsetofquestio nsthathavebeenhuman-
labeled with correct answers. MRR also assumes that systems are returning a short
rankedlist of answers, or passages containinganswers. Each quest ion is then scored
based on the reciprocalof the rankof the ﬁrst correctanswer. For exampleif the sys-

rank
tem returned 5 answers but the ﬁrst 3 are wrong and hence the hi ghest-rankedcorrect
answer is ranked 4, the reciprocal rank score for that questi on would be1
4. Questions
with returnsets that donotcontainanycorrectanswersare a ssigneda zero. Thescore
ofasystemisthentheaverageofthescoreforeachquestioni ntheset. Moreformally,
for an evaluation of a system returning Mranked answers for test set consisting of N
questions,theMRR isdeﬁnedas:
MRR=∑N
i=11
ranki
N(23.22)
23.3 Summarization

MRR=∑N
i=11
ranki
N(23.22)
The algorithmswe havedescribedso far in thischapterprese ntthe user an entire doc-
ument (information retrieval), or a short factoid answer ph rase (factoid question an-
swering). Butsometimestheuserwantssomethingthatliesi nbetweentheseextremes:
somethinglikea summary ofa documentorset ofdocuments.

DRAFT802 Chapter 23. QuestionAnsweringandSummarization
Text summarization isthe process of distilling the most important informationTextsummarization
from a text to produce an abridged version for a particular ta sk and user (deﬁnition
adapted from Mani and Maybury (1999)). Important kinds of su mmaries that are the
focusofcurrentresearchinclude:
•outlinesofanydocument
•abstracts ofa scientiﬁcarticle
•headlines ofa newsarticle
•snippetssummarizinga webpageona searchengineresultspage
•actionitemsorothersummaries ofa(spoken)businessmeeting
•summaries ofemailthreads
•compressedsentences forproducingsimpliﬁedorcompressedtext
•answerstocomplexquestions,constructedbysummarizingmultiple documents
These kinds of summarization goals are often characterized by their position on
twodimensions:
•singledocument versusmultiple document summarization
•genericsummarizationversus query-focused summarization
Insingle document summarization we are given a single documentand produceSingle documentsummarization

a summary. Single document summarization is thus used in sit uations like producing
a headline or an outline, where the ﬁnal goal is to characteri ze the content of a single
document.
Inmultipledocumentsummarization ,theinputisagroupofdocuments,andourMultiple
documentsummarizationgoal is to produce a condensation of the content of the entire group. We might use
multiple document summarization when we are summarizing a s eries of news stories
on the same event, or wheneverwe have web content on the same t opic that we’d like
tosynthesizeandcondense.

document.
Ageneric summary is one in which we don’t consider a particular user or a Generic summary
particular information need; the summary simply gives the i mportant information in
the document(s). By contrast, in query-focused summarization , also called focusedQuery-focusedsummarization
summarization ,topic-based summarization anduser-focused summarization , the
summaryisproducedinresponsetoauserquery. We canthinko fquery-focusedsum-
marizationasakindoflonger,non-factoidanswerto auser q uestion.

In the remainder of this section we give a brief overview of th e architecture of
automatictextsummarizationsystems; thefollowingsecti onsthengivedetails.
One crucial architectural dimension for text summarizers i s whether they are pro-
ducingan abstractoranextract. Thesimplestkindofsummary,an extract,isformed Extract
by selecting ( extracting ) phrases or sentences from the document to be summarized
and pasting them together. By contrast, an abstract uses different words to describe Abstract

the contentsofthe document. We’ll illustratethe differen cebetweenan extractandan
abstract using the well-knownGettysburg address, a famous speech by Abraham Lin-
coln, shown in Fig. 23.12.1Fig. 23.13 shows an extractive summary from the speech
followedbyanabstractofthespeech.
1In general one probably wouldn’t need asummary of such ashor t speech, buta shorttext makes iteasier
to see how the extract maps to the original for pedagogical pu rposes. For an amusing alternative application

of modern technology to the Gettysburg Address, see Norvig ( 2005).

DRAFTSection23.3. Summarization 803
Fourscore and seven years ago our fathers brought forth on th is continent a new
nation,conceivedinliberty,anddedicatedtotheproposit ionthatallmenarecreated
equal. Now we are engaged in a great civil war, testing whethe r that nation, or
any nation so conceived and so dedicated, can long endure. We are met on a great
battle- ﬁeld of that war. We have come to dedicate a portion of that ﬁeld as a ﬁnal
resting-place for those who here gave their lives that this n ation might live. It is
altogetherﬁttingandproperthatweshoulddothis. But,ina largersense,wecannot
dedicate...we cannot consecrate...we cannot hallow... th is ground. The brave men,
living and dead, who struggled here, have consecrated it far above our poor power
to add or detract. The world will little note nor long remembe r what we say here,
but it can never forget what they did here. It is for us, the liv ing, rather, to be
dedicated here to the unﬁnished work which they who fought he re have thus far so

nobly advanced. It is rather for us to be here dedicated to the great task remaining
beforeus...thatfromthesehonoreddeadwetakeincreasedd evotiontothatcausefor
which they gave the last full measure of devotion; that we her e highly resolve that
these deadshall not have diedin vain; that this nation,unde rGod, shall havea new
birth of freedom; and that governmentof the people, by the pe ople, for the people,
shall notperishfromtheearth.
Figure23.12 The GettysburgAddress. Abraham Lincoln,1863.
ExtractfromtheGettysburgAddress:

Four score and seven years ago our fathers broughtforth upon this continent a new
equal. Nowweareengagedinagreatcivilwar,testingwhethe rthatnationcanlong
endure. We are met on a great battleﬁeld of that war. We have co me to dedicate a
portion of that ﬁeld. But the brave men, living and dead, who s truggled here, have
consecratedit faraboveourpoorpowertoaddor detract. Fro mthese honoreddead
we take increased devotion to that cause for which they gave t he last full measure

of devotion— thatgovernmentof thepeople,bythe peoplefor the peopleshall not
perishfromtheearth.
AbstractoftheGettysburgAddress:
ThisspeechbyAbrahamLincolncommemoratessoldierswhola iddowntheir lives
in the Battle of Gettysburg. It reminds the troops that it is t he future of freedom in
Americathattheyareﬁghtingfor.
Figure23.13 An extract versus an abstract from the Gettysburg Address (a bstract from Mani
(2001)).
Most current text summarizers are extractive, since extrac tion is much easier than

(2001)).
abstracting;thetransitiontomoresophisticatedabstrac tivesummarizationisakeygoal
ofrecentresearch.
Textsummarizationsystemsand,asitturnsout, naturallanguagegeneration sys-
temsaswell,aregenerallydescribedbytheirsolutionstot hefollowingthreeproblems:
1.Content Selection: What information to select from the document(s) we are
summarizing. We usually make the simplifying assumption th at the granularity
ofextractionis thesentenceorclause. Contentselectiont husmainlyconsistsof

DRAFT804 Chapter 23. QuestionAnsweringandSummarization
choosingwhichsentencesorclausestoextractintothesumm ary.
2.InformationOrdering: Howto orderandstructurethe extractedunits.
3.Sentence Realization: What kind of clean up to performon the extracted units
sotheyareﬂuentintheirnewcontext.
Inthenextsectionswe’llshowthesecomponentsinthreesum marizationtasks: sin-
gledocument summarization, multipledocument summarization,and query-focused
summarization .
23.3.1 Summarizing SingleDocuments
Let’s ﬁrst consider the task of building an extractive summa ry for a single document.
Assumingthat the unitsbeingextractedare at the levelof th e sentence,the threesum-
marizationstagesforthistaskare:
1.ContentSelection: Choosesentencestoextractfromthe document
2.Information Ordering: Choose an order to place these sentences in the sum-
mary
3.Sentence Realization: Clean up the sentences, for example by removing non-
essential phrases from each sentence, or fusing multiple se ntences into a single

mary
sentence,orbyﬁxingproblemsincoherence.
Content SelectionSentence 
SegmentationInformation 
OrderingDocument
Sentence
ExtractionAll sentences
from documentsExtracted 
sentences
Summary Sentence 
Realization
Sentence 
Simpliﬁcation
Figure23.14 The basic architecture of a generic single document summari zer.
We’ll ﬁrstdescribebasicsummarizationtechniqueswithon lyoneofthesecompo-
nents:content selection . Indeed, many single document summarizers have no infor-

Sentence
sentences
Sentence 
mation orderingcomponent,simply orderingthe extracted s entencesin the order they
appeared in the original document. In addition, we’ll assum e for now that sentences
arenotcombinedorcleanedupaftertheyareextracted,alth oughwe’llbrieﬂymention
laterhowthisisdone.
Unsupervised ContentSelection
Thecontent selection task of extracting sentences is often treated as a classiﬁca tion Content selection
task. The goal of the classiﬁer is to label each sentence in a d ocument with a binary

label:important versusunimportant (orextract-worthy versusnot extractworthy ). We
begin with some unsupervised algorithms for sentence class iﬁcation and then turn to
supervisedalgorithmsinthenextsection.
The simplest unsupervised algorithm, based on an intuition that dates back to the
early summarizer of (Luhn, 1958), is to select sentences tha t have more salientor
informative words. Sentences that contain more informative words tend t o be more

DRAFTSection23.3. Summarization 805
extract-worthy. Saliency is usually deﬁned by computing th etopic signature , a set Topic signature
ofsalientorsignature terms , each of whose saliency scores is greater than some Signature terms
threshold θ.
Saliency could be measured in terms of simple word frequency ,but frequencyhas
the problemthat a wordmighthave a highprobabilityin Engli shin generalbutnot be
particularly topical to a particular document. Thereforew eighting schemes like tf-idf
orlog-likelihoodratio aremoreoftenused.
Recall from page 786 that the tf-idf scheme gives a high weigh t to words that ap-
pear frequentlyin the currentdocument,but rarely in the ov eralldocumentcollection,
suggestingthat the wordis particularlyrelevantto this do cument. For each term ithat
occursinthesentencetobeevaluated,wecomputeitscounti nthecurrentdocument j
tfi,j,andmultiplybytheinversedocumentfrequencyoverthewho lecollectionidf i:
weight (wi) =tfi,j×idfi (23.23)

A better performing method for ﬁnding informative words is log likelihood ratioLog likelihoodratio
(LLR).Theloglikelihoodratiofora word,generallycalled λ(w), istheratio between
theprobabilityofobserving wbothintheinputandinthebackgroundcorpusassuming
equalprobabilitiesinbothcorpora,andtheprobabilityof observing winbothassuming
different probabilities for win the input and the background corpus. See Dunning
(1993), Moore (2004)and Manning and Sch¨ utze (1999) for det ails on log likelihood
andhowit iscalculated.

It turns out for log likelihood ratio that the quantity −2log(λ)is asymptotically
wellapproximatedbythe χ2distribution,whichmeansthatawordappearsintheinput
signiﬁcantly more often than in the backgroundcorpus(at α=0.001) if−2log(λ)>
10.8. Lin and Hovy (2000) ﬁrst suggested that this made log likel ihood ratio partic-
ularly appropriate for selecting a topic signature for summ arization. Thus the word
weightwithloglikelihoodratioisgenerallydeﬁnedasfoll ows:
weight (wi) ={1 if−2log(λ(wi))>10
0 otherwise.(23.24)
Eq.23.24isusedtoset a weightof1or0foreachwordinthesen tence. Thescore

forasentence siisthentheaverageweightofitsnon-stopwords:
weight (si) =∑
w∈siweight (w)
|{w|w∈si}|(23.25)
The summarization algorithms computes this weight for ever y sentence, and then
ranks all sentences by their score. The extracted summary co nsists of the top ranked
sentences.
The family of algorithms that this thresholdedLLR algorith m belongs to is called
centroid-based summarization because we can view the set of signature terms as a

pseudo-sentence which is the ‘centroid’ of all the sentence s in the document and we
arelookingforsentenceswhichareascloseaspossibletoth iscentroidsentence.
A common alternative to the log likelihood ratio/centroid m ethod is to use a dif-
ferent model of sentence centrality . These other centrality based methods resemble Centrality
the centroid method described above, in that their goal is to rank the input sentences

DRAFT806 Chapter 23. QuestionAnsweringandSummarization
in terms of how central they are in representing the informat ion present in the docu-
ment. But rather than just ranking sentences by whether they contain salient words,
centralitybasedmethodscomputedistancesbetweeneachca ndidatesentenceandeach
other sentence and choose sentences that are on average clos er to other sentences. To
computecentrality,we can representeachsentence as a bag- of-wordsvectorof length
Nas described in Ch. 20. For each pair of sentences xandy, we compute the tf-idf
weightedcosineasdescribedinEq.23.12above.
Each of the ksentences in the input is then assigned a centrality score wh ich is its
averagecosinewithall othersentences:
centrality (x) =1
K∑
ytf-idf-cosine (x,y) (23.26)
Sentencesarerankedbythiscentralityscore,andthesente ncewhichhasthehighest
average cosine across all pairs, i.e. is most like other sent ences, is chosen as the most
‘representative’or‘topical’ofall thesentencesinthe in put.

K∑
It is also possible to extend this centrality score to use mor e complex graph-based
measuresofcentralitylikePageRank(ErkanandRadev,2004 ).
Unsupervised Summarizationbased onRhetoricalParsing
The sentence extraction algorithm we introduced above for c ontent extraction relied
solely on a single shallow feature, word saliency, ignoring possible higher-level cues
suchas discourseinformation. Inthissection we brieﬂysum marizea way to get more
sophisticateddiscourseknowledgeintothesummarization task.
The summarization algorithm we’ll describe makes use of coherence relations

such as the RST (rhetorical structure theory) relations des cribed in Ch. 21. Recall
that RST relations are often expressed in terms of a satelliteand anucleus; nucleus
sentence are more likely to be appropriate for a summary. For example, consider the
following two paragraphs taken from the Scientiﬁc American magazine text that we
introducedinSec. 21.2.1:
With its distant orbit – 50 percent farther from the sun than E arth – and
slimatmosphericblanket,Marsexperiencesfrigidweather conditions. Sur-

face temperatures typically average about -70 degrees Fahr enheit at the
equator,andcandipto -123degreesCnearthe poles.
Only the midday sun at tropical latitudes is warm enough to th aw ice
on occasion, but any liquid water formed in this way would eva porate al-
most instantly because of the low atmospheric pressure. Alt hough the at-
mosphereholdsa small amountof water,andwater-ice clouds sometimes
develop,mostMartianweatherinvolvesblowingdustorcarb ondioxide.

Theﬁrsttwodiscourseunitsinthispassagearerelatedbyth eRSTJUSTIFICATION
relation,withtheﬁrstdiscourseunitjustifyingthesecon dunit,asshowninFig.23.15.
The second unit ( “Mars experiences frigid weather conditions” ) is thus the nucleus,
andcapturesbetterwhatthispartofthe documentisabout.
We can use this intuitionfor summarizationby ﬁrst applying a discourse parser of
thetypediscussedinCh.21tocomputethecoherencerelatio nsbetweeneachdiscourse

DRAFTSection23.3. Summarization 807
With its distant orbit - 50 percent farther from 
the sun than Earth - and slim atmospheric blanket,Mars experiences frigid weather conditionsJUSTIFICATION
Figure23.15 The justiﬁcation relation between two discourse units, a sa tellite (on the left)
and anucleus (on the right).
unit. Once a sentence has been parsed into a coherence relati on graph or parse tree,
we can use the intuition that the nuclear units are important for summarization by
recursivelyextractingthesalient unitsofatext.
Consider the coherence parse tree in Fig. 23.16. The salienc e of each node in the
treecanbedeﬁnedrecursivelyasfollows:
•Base case: Thesalient unitofaleafnodeistheleafnodeitse lf
•Recursive case: The salient units of an intermediate node ar e the union of the
salientunitsofitsimmediate nuclearchildren
Bythisdeﬁnition,discourseunit(2)isthemostsalientuni toftheentiretext(since
the root node spanning units 1-8 has the node spanning units 1 -6 as its nucleus, and

unit2isthe nucleusofthenodespanningunits1-6.)
If we rank each discourse unit by the height of the nodes that i t is the nucleus of,
we can assign a partial ordering of salience to units; the alg orithm of Marcu (1995)
assignsthefollowingpartialorderingto thisdiscourse:
2>8>3>1,4,5,7>6 (23.27)
See Marcu(1995,2000b)forthe detailsof exactlyhow thispa rtial orderis computed,
andTeufelandMoens(2002)foranothermethodforusingrhet oricalstructureinsum-
marization.
Supervised ContentSelection

While the use of topic signatures for unsupervised content s election is an extremely
effective method, topic signatures is only a single cue for ﬁ nding extractworthy sen-
tences. Many other cues exist, including the alternative sa liency methods discussed
abovelike centrality and PageRank mathods, as well as other cueslike the position of
the sentence in the document(sentencesat the very beginnin gor end of the document
tendtobemoreimportant),thelengthofeachsentence,ands oon. We’dlikeamethod

thatcanweighandcombineall ofthesecues.
The best principled method for weighing and combining evide nce is supervised
machine learning. For supervised machine learning, we’ll n eed a training set of doc-
uments paired with human-created summary extracts, such as the Ziff-Davis corpus
(Marcu, 1999). Since these are extracts, each sentence in the summary is, by deﬁni-
tion, taken from the document. That meanswe can assign a labe l to everysentence in

the document; 1if it appearsin theextract, 0if it doesn’t. To buildourclassiﬁer, then,
wejustneedtochoosefeaturestoextractwhicharepredicti veofbeingagoodsentence

DRAFT808 Chapter 23. QuestionAnsweringandSummarization
2
Elaboration
2
3
Elaboration8
Example
8
Concession
most Martian weather 
involves blowing dust 
or carbon dioxide
(8)Although the atmosphere
holds a small amount 
of water, and water-ice 
clouds sometimes develop
(7)Surface temperatures 
typically average about -60 
degrees Celsius {(-76 
degrees Fahrenheit) } at the 
equator and can dip to -123 
degrees C near the poles.
(3)4,5
Contrast
Only the midday sun at 
tropical latitudes is 
warm enough to thaw 
ice on occasion
(4)5
Evidence 
Cause
but any liquid water 
formed in this way would 
evaporate almost instantly
(5)With its distant orbit 
{ -50 percent farther 
from the sun than 
Earth- } and slim 
atomspheric blanket
(1)2
Background
Justiﬁcation
Mars experiences 
frigid weather 
conditions.
(2)
because of the low 
atmospheric pressure.
(6)
Figure23.16 The discourse tree for the text on page 806. Boldface links co nnect nodes to their nuclei children;

(4)5
Evidence 
Cause
(1)2
(2)
(6)
dottedlines tothe satellitechildren. AfterMarcu (2000a) .
toappearinasummary. Someofthefeaturescommonlyusedins entenceclassiﬁcation
areshowninFig.23.17.
Eachsentence in ourtrainingdocumentthushas a label (0 if t he sentence is not in
the training summary for that document, 1 if it is) and set of e xtracted feature values
like those in Fig. 23.17. We can then train our classiﬁer to es timate these labels for
unseen data; for example a probabilistic classiﬁer like nai ve Bayes or MaxEnt would

be computing the probability that a particular sentence sis extractworthy given a set
of features f1...fn; then we can just extract any sentences for which this probab ilityis
greaterthan0.5:
P(extractworthy (s)|f1,f2,f3,...,fn) (23.28)
There is one problem with the algorithm as we’ve described it : it requires that we
have a training summary for each document which consists sol ely of extracted sen-
tences. If we could weaken this restriction, we could apply t he algorithm to a much

wider variety of summary-document pairs, such as conferenc e papers or journal arti-
cles and their abstracts. Luckily it turns out that when huma nswrite summaries, even
with the goal of writing abstractive summaries, they very of ten use phrases and sen-
tencesfromthe documentto composethe summary. But they don ’tuseonlyextracted
sentences; theyoftencombinetwo sentencesinto one,orcha ngesome ofthewordsin
thesentences,orwritecompletelynewabstractivesentenc es. Hereisanexampleofan

extracted sentence from a human summary that, althoughmodi ﬁed in the ﬁnal human
summary,wasclearlyadocumentsentencethatshouldbelabe ledasextractworthy:

DRAFTSection23.3. Summarization 809
position The position of the sentence in the document. For example Hov y and
Lin (1999) found that the single most extract-worthy senten ce in most
newspaper articles is the title sentence. In the Ziff-Davis corpus they
examined, the next most informative was the ﬁrst sentence of paragraph
2 (P1S1), followed by the ﬁrst sentence of paragraph 3 (P3S1) ; thus
the list of ordinal sentence positions starting from the mos t informative
was: T1,P2S1,P3S1,P4S1,P1S1,P2S2,...
Position, like almost all summarization features, is heavi ly
genre-dependent. In Wall Street Journal articles, they fou nd
the most important information appeared in the following se n-
tences: T1,P1S1,P1S2,...
cuephrases Sentences containing phrases like in summary ,in conclusion , orthis paper are
more likely to be extract-worthy. These cue phrases are very dependent on the
genre. For example in British House of Lords legal summaries , the phrase it
seemsto methat isa usefulcuephrase. (HacheyandGrover,2005).

word
informativenessSentencesthatcontainmoretermsfromthe topicsignature ,asdescribedinthe
previoussection,aremoreextractworthy.
sentence
lengthVery short sentences are rarely appropriate for extracting . We usually capture
this fact by using a binary feature based on a cutoff (true if t he sentence has
morethan,say,5words).
cohesion Recall from Ch. 21 that a lexical chain is a series of related words that occurs
throughout a discourse. Sentences which contain more terms from a lexical

word
sentence
chain are often extractworthybecause they are indicative o f a continuingtopic.
(Barzilay and Elhadad, 1997). This kind of cohesion can also be computed
by graph-based methods (Mani and Bloedorn, 1999). The PageR ank graph-
based measures of sentence centrality discussed above can a lso be viewed as a
coherencemetric(ErkanandRadev,2004).
Figure23.17 Some features commonly used in supervised classiﬁers for de termining whether a document sen-
tence should be extractedintoa summary;

(23.29)Humansummary : Thispaperidentiﬁesthedesirablefeaturesofanideal
multisensorgasmonitorandlists thedifferentmodelscurr entlyavailable.
(23.30)Originaldocument sentence : Thepresentpartlists thedesirablefeatures
andthedifferentmodelsofportable,multisensorgasmonit orscurrently
available.
Thus an important preliminary stage is to aligneach training document with its
summary,with the goal of ﬁnding which sentences in the docum entwere (completely

or mostly) included in the summary. A simple algorithm for alignment is to ﬁnd the Alignment
source document and abstract sentences with the longest com mon subsequences of
non-stopwords;alternativelyminimumeditdistancecanbe computed,ormoresophis-
ticatedknowledgesourcescanbeused,suchasWordNet. Rece ntworkhasfocusedon
morecomplexalignmentalgorithmssuchas the use of HMMs(Ji ng, 2002;Daum´ eIII
andMarcu,2005,interalia).
Given such alignment algorithms, supervised methods for co ntent selection can

DRAFT810 Chapter 23. QuestionAnsweringandSummarization
make use of parallel corporaof documentsand humanabstract ive summaries, such as
academicpaperswiththeirabstracts(TeufelandMoens,200 2).
SentenceSimpliﬁcation
Onceasetofsentenceshasbeenextractedandordered,theﬁn alstepinsingle-document
summarizationis sentencerealization . Onecomponentofsentencerealizationis sen-
tencecompression orsentencesimpliﬁcation . Thefollowingexamples,takenbyJingSentencecompression
Sentencesimpliﬁcation (2000) from a human summary, show that the human summarizer c hose to eliminate
someoftheadjectivemodiﬁersandsubordinateclauseswhen expressingtheextracted
sentenceinthesummary:
(23.31)Originalsentence: WhenitarrivessometimenewyearinnewTVsets,the
V-chipwillgiveparentsa newandpotentiallyrevolutionarydeviceto blockout
programstheydon’twanttheirchildrentosee.
(23.32)Simpliﬁed sentence byhumans: TheV-chipwill giveparentsadeviceto
blockoutprogramstheydon’twanttheirchildrentosee.

The simplest algorithms for sentence simpliﬁcation use rul es to select parts of the
sentence to prune or keep, often by running a parser or partia l parser over the sen-
tences. Some representative rules from Zajic et al. (2007), Conroy et al. (2006), and
Vanderwendeetal. (2007a)removethefollowing:
appositives Rajam,28,anartistwhowaslivingatthetimeinPhiladelphia,
found the inspirationinthe back of citymagazines.
attributionclauses Rebels agreed to talks with government ofﬁcials, international
observerssaidTuesday.

PPs without
named entitiesThe commercial ﬁshing restrictions in Washington will not b e
lifted [SBAR unless the salmon population 329 increases [PP to
asustainablenumber]
initialadverbials “For example”, “On the other hand”, “As a matter of fact”, ”At
this point”
More sophisticated models of sentence compression are base d on supervised ma-
chine learning, in which a parallel corpus of documents toge ther with their human
summariesisusedto computetheprobabilitythat particula rwordsorparsenodeswill
bepruned. See theendofthechapterforpointerstothisexte nsiverecentliterature.

23.4 Multi-DocumentSummarization
Whenweapplysummarizationtechniquestogroupsofdocumen tsratherthanasingle
documentwecallthegoal multi-documentsummarization . Multi-documentsumma-Multi-documentsummarization
rizationisparticularlyappropriateforweb-basedapplic ations,forexampleforbuilding
summaries of a particular event in the news by combining info rmation from different
news stories, or ﬁnding answers to complex questions by incl uding componentsfrom
extractedfrommultipledocuments.

While multi-documentsummarization is far from a solved pro blem, even the cur-
rent technology can be useful for information-ﬁnding tasks . McKeown et al. (2005),

DRAFTSection23.4. Multi-DocumentSummarization 811
for example, gave humanexperimentalparticipantsdocumen tstogether with a human
summary, an automatically generated summary, or no summary , and had the partic-
ipants perform time-restricted fact-gathering tasks. The participants had to answer
three related questions about an event in the news; subjects who read the automatic
summariesgavehigher-qualityanswersto thequestions.
Multi-documentsummarizationalgorithmsarebasedonthes amethreestepswe’ve
seen before. In many cases we assume that we start with a clust er of documents that
we’dliketosummarize,andwemustthenperform contentselection ,informationor-
dering, andsentence realization ,asdescribedin the nextthreesectionsand sketched
inFig. 23.18
OrderingSummaryDocument
Document
Document
Document
Document
Input Docs
Sentence 
Sentence
from documentsAll sentences
plus simpliﬁed 
versionsExtracted 
sentences
Sentence

Document
Document
Document
Document
Input Docs
Sentence 
Sentence
sentences
Sentence 
Figure23.18 The basic architecture of a multi-document summarizer.
23.4.1 Content SelectioninMulti-Document Summarization
Insingledocumentsummarizationweusedbothsupervisedan dunsupervisedmethods
for content selection. For multiple document summarizatio n supervised training sets
arelessavailable,andwefocusmoreonunsupervisedmethod s.
Themajordifferencebetweenthetasksofsingledocumentan dmultipledocument
summarizationis thegreateramountof redundancy whenwe start withmultipledoc-
uments. A group of documents can have signiﬁcant overlap in w ords, phrases, and

concepts, in addition to information that might be unique to each article. While we
want each sentence in the summaryto be about the topic, we don ’twant the summary
toconsist ofaset ofidenticalsentences.
For this reason, algorithms for multi-document summarizat ion focus on ways to
avoid redundancywhen selected sentencesfor the summary. W hen adding a new sen-
tence to a list of extracted sentences we need some way to make sure the sentence
doesn’toverlaptoomuchwiththe already-extractedsenten ces.

Asimplemethodofavoidingredundancyistoexplicitlyincl udearedundancyfac-
torinthescoringforchoosingasentencetoextract. Thered undancyfactorisbasedon
the similarity between a candidate sentence and the sentenc es that have already been
extractedintothe summary;a sentence is penalizedif it is t oo similar to the summary.
For example the MMRorMaximal Marginal Relevance scoring system Carbonell MMRMaximalMarginal
RelevanceandGoldstein (1998),Goldsteinet al. (2000)includesthe f ollowingpenalizationterm

for representing the similarity between a sentence sand the set of sentences already
extractedforthesummary Summary ,whereλisa weightthat canbetunedandSim is
somesimilarityfunction:
MMRpenalizationfactor (s) =λmaxsi∈SummarySim(s,si) (23.33)

DRAFT812 Chapter 23. QuestionAnsweringandSummarization
An alternative to MMR-based methodis to instead apply a clus tering algorithmto
allthesentencesinthedocumentstobesummarizedtoproduc eanumberofclustersof
related sentences and then to select a single (centroid)sen tence from each cluster into
thesummary.
By adding MMR or clustering methods for avoiding redundancy , we can also do
sentence simpliﬁcationor compressionat the content selec tion stage rather than at the
sentence realization stage. A commonway to ﬁt simpliﬁcatio n into the architecture is
to run varioussentencesimpliﬁcationrules(Sec. 23.3.1)o n each sentencein the input
corpus. The result will be multiple versions of the input sen tence, each version with
differentamountsofsimpliﬁcation. Forexample,thefollo wingsentence:
FormerDemocraticNationalCommitteeﬁnancedirectorRich ardSullivan
faced more pointed questioning from Republicans during his second day
onthe witnessstandin theSenate’sfund-raisinginvestiga tion.

mightproducedifferentshortenedversions:
•RichardSullivanfaced pointedquestioning.
•RichardSullivanfaced pointedquestioning from Republica ns
•RichardSullivanfacedpointedquestioningfromRepublica nsduringdayonstandinSen-
atefundraising investigation
•Richard Sullivan faced pointed questioning from Republica ns in Senate fundraising in-
vestigation
This expanded corpusis now used as the input to content extra ction. Redundancy
methods such as clustering or MMR will choose only the (optim ally long) single ver-
sionofeachoriginalsentence.

23.4.2 InformationOrdering inMulti-Document Summarizat ion
Thesecondstageofanextractivesummarizeristheordering orstructuringofinforma-
tion,wherewemustdecidehowtoconcatenatetheextracteds entencesintoacoherent
order. Recall that in single document summarization, we can just use the original ar-
ticle ordering for these sentences. This isn’t appropriate for most multiple document
applications,althoughwecancertainlyapplyitifmanyora lloftheextractedsentences
happentocomefromasinglearticle.

For sentences extracted from news stories, one technique is to use the dates asso-
ciated with the story, a strategy known as chronological ordering . It turns out thatChronological
ordering
pure chronological ordering can produce summaries which la ck cohesion; this prob-
lemcanbeaddressedbyorderingslightlylargerchunksofse ntencesratherthansingle
sentences;see Barzilayetal. (2002).
Perhapsthemostimportantfactorforinformationordering ,however,is coherence .

ordering
Recall fromCh. 21the variousdevicesthat contributeto the coherenceof a discourse.
One is having sensible coherence relations between the sent ences; thus we could pre-
fer orderings in summaries that resulting in sensible coher ence relations between the
sentences. Anotheraspectofcoherencehastodowithcohesi onandlexicalchains;we
could for example prefer orderingswhich have more local coh esion. A ﬁnal aspect of
coherenceiscoreference;acoherencediscourseisoneinwh ichentitiesarementioned

incoherentpatterns. We couldpreferorderingswithcohere ntentitymentionpatterns.

DRAFTSection23.4. Multi-DocumentSummarization 813
All of these kinds of coherence have been used for informatio n ordering. For ex-
ample we can use lexical cohesion as an ordering heuristic by ordering each sentence
next to sentences containing similar words. This can done by deﬁning the standard
tf-idf cosine distance between each pair of sentences and ch oosing the overall order-
ing that minimizes the average distance between neighborin g sentences Conroy et al.
(2006),orbybuildingmodelsofpredictablewordsequences acrosssentences(Soricut
andMarcu,2006).
Coreference-based coherence algorithms have also made use of the intuitions of
Centering . Recall that the Centering algorithm was based on the idea th at each dis-
course segment has a salient entity, the focus. Centering theory proposed that certain
syntactic realizationsof the focus(i.e. as subject or obje ct) and certain transitionsbe-
tween these realizations (e.g., if the same entity is the sub ject of adjacent sentences)

createdamorecoherentdiscourse. Thuswecanpreferorderi ngsinwhichthetransition
betweenentitymentionsis apreferredone.
Forexampleintheentity-basedinformationapproachofBar zilayandLapata(2005,
2007),a trainingset ofsummariesis parsedandlabeledforc oreference. Theresulting
sequence of entity realizations can be automatically extra cted and represented into an
entity grid . Fig. 23.19 shows a simpliﬁed version of a parsed summary and the ex- Entitygrid
tractedgrid. Aprobabilisticmodelofparticularentitytr ansitions(i.e.{S,O,X,−}can

then be trained from the entity grid. For example the transit ions{X,O,S,S}for the
headword Microsoft exemplifythefactthatnewentitiesina discourseareoften intro-
duced ﬁrst in obliqueor object position and then only later a ppear in subject position.
See BarzilayandLapata(2007)fordetails.
A general way to view all of these methods is as assigning a coh erence score to
a sequence of sentences via a local coherence score between p airs or sequences of
sentences;asinglegeneraltransitionscorebetweensente ncescouldthencombinelex-

ical coherence and entity-based coherence. Once we have suc h a scoring function,
choosing an ordering which optimizes all these local pairwi se distances is known to
be quite difﬁcult. The task of ﬁnding the optimal ordering of a set of sentences given
a set of pairwise distances between the sentences is equival ent to very hard problems
like Cyclic Ordering and the TravelingSalesman Problem.2Sentence orderingis thus
equivalent to the difﬁcult class of problems known as NP-complete . While difﬁcult

to solve exactly, there are a number of good approximation me thods for solving NP-
completeproblemsthathavebeenappliedtotheinformation orderingtask. SeeAlthaus
et al. (2004),Knight(1999a),Cohen et al. (1999),Brew (199 2)for therelevantproofs
andapproximationtechniques.
In the models described above, the information ordering tas k is completely sepa-
rate from content extraction. An alternative approach is to learn the two tasks jointly,

resulting in a model that both selects sentences and orders t hem. For example in the
HMM model of Barzilay and Lee (2004), the hidden states corre spond to document
content topics and the observations to sentences. For examp le for newspaper articles
on earthquakes, the hidden states (topics) might be strength of earthquake ,location,
rescue efforts , andcasualties . They apply clustering and HMM induction to induce
2TheTraveling Salesman Problem: given a set of cities and the pairwise distances between them, ﬁnd the
shortest path that visits each city exactly once.

DRAFT814 Chapter 23. QuestionAnsweringandSummarization
[The Justice Department]S is conducting an [anti-trust trial]O against [Microsoft Corp.]X 
[The case]S resolves around [evidence]O of  [Microsoft]S aggressively 
pressuring  [Netscape]O  into merging [browser software]O[Microsoft]O is accused of trying to forcefully buy into [markets]X where
[its own products]S are not competitive enough to unseat [established brands]O
[Microsoft]S claims [its tactics]S are commonplace and good economically.1
2
3
4
DepartmentTrialMicrosoftMarketsProductsBrandsCaseNetscapeSoftwareTactics1  S  O  X  -  -    -   -   -   -   -
2   -   -  O  X  S  O   -   -   -   -
3   -   -  S  O  -   -  S  O  O   -
4   -   -  S   -   -   -   -   -   -   O
Figure23.19 A summary (showing entities in subject (S), object (O) or obl ique (X) position), and the entity grid
that isextracted from it. Adapted from BarzilayandLapata ( 2005).
these hiddenstates and the transitionsbetweenthem. For ex ample,hereare three sen-

2
3
4
tencesfromthe locationcluster theyinduce:
(23.34)The Athens seismological institute saidthe temblor’s epic enter was located 380
kilometers (238miles) southof the capital.
(23.35)Seismologists inPakistan’s Northwest FrontierProvince s aidthe temblor’s epicenter
was about 250 kilometers (155 miles)north of the provincial capital Peshawar.
(23.36)The temblor was centered60 kilometers (35miles) northwest of the provincial capital
ofKunming, about 2,200 kilometers (1,300miles) southwest of Beijing, a bureau
seismologist said.

The learned structure of the HMM then implicitly represent i nformation ordering
facts like mention ‘casualties’ prior to ‘rescue efforts’ via the HMM transition proba-
bilities.
Insummary,we’veseeninformationorderingbasedon chronologicalorder ,based
oncoherence , and an orderingthat is learnedautomaticallyfromthe data . In the next
sectiononquery-focusedsummarizationwe’llintroduceaﬁ nalmethodinwhichinfor-
mationorderingcanbespeciﬁedaccordingtoanorderingtem platewhichispredeﬁned
advancefordifferentquerytypes.
SentenceRealization

bilities.
While discourse coherence can be factored in during sentenc e ordering, the resulting
sentencesmaystillhavecoherenceproblems. Forexample,a swesawinCh.21,when
a referent appears multiple times in a coreference chain in a discourse, the longer or
more descriptive noun phrases occur before shorter, reduce d, or pronominal forms.
But the orderingwe chooseforthe extractedsentencesmayno t respectthiscoherence
preference.
For example the boldfaced names in the original summary in Fi g. 23.20 appear in

an incoherent order; the full name U.S. President George W. Bush occurs only after
theshortenedform Bushhasbeenintroduced.
One possible way to address this problem in the sentence real ization stage is to
apply a coreferenceresolution algorithmto the output, ext racting namesand applying
somesimplecleanuprewriteruleslike thefollowing:
(23.37)Usethefullname at theﬁrst mention,andjust the lastname atsubsequent
mentions.
(23.38)Useamodiﬁed formfortheﬁrst mention,but removeappositivesor

mentions.
premodiﬁersfromanysubsequentmentions.

DRAFTSection23.5. QuestionAnsweringMeetsSummarization: Foc usedSummarization 815
The rewritten summary in Fig. 23.20 shows how such rules woul d apply; in general
suchmethodswoulddependonhigh-accuracycoreferenceres olution.
Originalsummary:
Presidential advisers do not blame O’Neill, but they’ve long recognized that a
shakeup of the economic team would help indicate Bushwas doing everything he
could to improve matters. U.S. President George W. Bush pushed out Treasury
Secretary Paul O’Neill and top economic adviser Lawrence Lindsey on Friday,
launching the ﬁrst shake - up of his administration to tackle the ailing economy
beforethe2004electioncampaign.
Rewrittensummary:
Presidential advisers do not blame Treasury Secretary Paul O’Neill , but they’ve
longrecognizedthatashakeupoftheeconomicteamwouldhel pindicate U.S.Pres-
ident George W. Bush was doing everything he could to improve matters. Bush
pushed out O’Neilland White House economic adviser Lawrence Lindsey on Fri-

day, launching the ﬁrst shake-up of his administration to ta ckle the ailing economy
Figure23.20 Rewritingreferences, from Nenkova and McKeown (2003)
Recent research has also focused on a ﬁner granularityfor re alization than the ex-
tracted sentence, by using sentence fusion algorithms to combine phrases or clauses Sentencefusion
from different sentences into one new sentence. The sentenc e fusion algorithm of

Barzilay and McKeown (2005) parses each sentence, uses mult iple-sequence align-
ment of the parses to ﬁnd areas of common information, builds a fusion lattice with
overlappinginformation,and creates a fused sentence by li nearizing a string of words
fromthelattice.
23.5 Question Answering Meets Summarization: Focused
Summarization
As noted in at the beginningof this chapter,most interestin g questionsare not factoid
questions. User needs require longer, more informative ans wers than a single phrase

canprovide. Forexample,whilea DEFINITION questionmightbeansweredbyashort
phraselike“ Autismis adevelopmentaldisorder ”or“Acalderais avolcaniccrater ”,
a usermightwant moreinformation,asin thefollowingdeﬁni tionofwaterspinach :
Waterspinach (ipomoeaaquatica)isasemi-aquaticleafygreenplantchar -
acterized by long hollow stems and spear-shaped or heart-sh aped leaves
which is widely grown throughout Asia as a leaf vegetable. Th e leaves
andstemsareofteneatenstir-friedasgreenswithsaltorsa ltysauces,orin

soups. Other commonnames include morning glory vegetable ,kangkong
(Malay),raumuong (Vietnamese), ongchoi(Cantonese),and kongxincai

DRAFT816 Chapter 23. QuestionAnsweringandSummarization
(Mandarin). It is not related to spinach, but is closely rela ted to sweet
potatoandconvolvulus.
Complexquestionscan also be asked in domainslike medicine , such as this ques-
tionabouta particulardrugintervention:
(23.39)Inchildrenwith anacutefebrileillness, whatistheefﬁcac yof
single-medicationtherapywith acetaminophenoribuprofe ninreducingfever?
Forthismedicalquestion,we’dliketobeabletoextractana nswerofthefollowing
type,perhapsgivingthedocumentid(s)that theextractcam efrom,andsome estimate
ofourconﬁdenceintheresult:
Ibuprofenprovidedgreater temperaturedecrementand long erduration of
antipyresisthan acetaminophenwhen the two drugswere admi nisteredin
approximatelyequaldoses. (PubMedID:1621668,EvidenceS trength: A)
Questionscanbeevenmorecomplex,suchasthisonefromtheD ocumentUnderstand-
ingConferenceannualsummarizationcompetition:
(23.40)Wherehavepoachersendangeredwildlife,whatwildlifehas been
endangeredandwhatstepshavebeentakento preventpoachin g?

Where a factoid answer might be foundin a single phrase in a si ngle documentor
web page,thesekindsofcomplexquestionsare likelyto requ iremuchlongeranswers
whicharesynthesizedfrommanydocumentsorpages.
Forthis reason,summarizationtechniquesare oftenused to buildanswersto these
kinds of complex questions. But unlike the summarization al gorithms introduced
above, the summaries produced for complex question answeri ng must be relevant to
some user question. When a document is summarized for the pur pose of answering

someuserqueryorinformationneed,wecall thegoal query-focusedsummarizationQuery-focusedsummarization
or sometimes just focused summarization . (The terms topic-based summarization
anduser-focused summarization are also used.) A query-focused summary is thus
reallyakindoflonger,non-factoidanswertoauser questio norinformationneed.
Onekindofquery-focusedsummaryisa snippet,thekindthatwebsearchengines Snippet
likeGooglereturntotheusertodescribeeachretrieveddoc ument. Snippetsarequery-
focusedsummariesof a single document. But since for comple xquerieswe will want

to aggregate informationfrom multiple documents, we’ll ne ed to summarize multiple
Indeed, the simplest way to do query-focusedsummarization is to slightly modify
thealgorithmsformultipledocumentsummarizationthatwe introducedintheprevious
section to make use of the query. For example, when ranking se ntences from all the
returned documents in the content selection phase, we can re quire that any extracted
sentence must contain at least one word overlapping with the query. Or we can just

add the cosine distance from the query as one of the relevance features in sentence
extraction. We can characterize such a method of query-focu sed summarization as a
bottom-up,domain-independentmethod.
An alternative way to do query-focused summarization is to m ake additional use
of top-down or information-extraction techniques, buildi ng speciﬁc content selection
algorithmsfordifferenttypesofcomplexquestions. Thusw ecouldspeciﬁcallybuilda

DRAFTSection23.5. QuestionAnsweringMeetsSummarization: Foc usedSummarization 817
query-focusedsummarizer for the kinds of advancedquestio ns introducedabove, like
deﬁnition questions, biographyquestions, certain medica l questions. In each case, we
useourtop-downexpectationsforwhatmakesagooddeﬁnitio n,biography,ormedical
answertoguidewhat kindsofsentenceswe extract.
For example, a deﬁnition of a term often includes information about the term’s
genusandspecies. The genus is the hypernym or superordinate of the word; thus a Genus
Species sentencelike The Hajjis atype ofritual isa genussentence. Thespeciesgivesimpor-
tant additional properties of the term that differentiate t he term from other hyponyms
ofthegenus;anexampleis “TheannualhajjbeginsinthetwelfthmonthoftheIslamic
year”. Other kinds of information that can occur in a deﬁnition inc ludesynonyms ,
etymology ,subtypes,andso on.
Inordertobuildextractiveanswersfordeﬁnitionquestion s,we’llneedtomakesure

we extract sentences with the genus information, the specie s information, and other
generally informative sentences. Similarly, a good biography of a person contains
information such as the person’s birth/death ,fame factor ,education ,nationality
and so on; we’ll need to extract sentences with each of these k inds of information. A
medicalanswerthatsummarizestheresultsofa studyonappl yinga drugto amedical
problemwould need to contain informationlike the problem (the medical condition),
theintervention (thedrugorprocedure),andthe outcome (theresultofthestudy).

Fig. 23.21 shows some example predicates for deﬁnition, bio graphy, and medical
interventionquestions.
Deﬁnition
genus The Hajj is a type of ritual
species the annual hajj begins in the twelfth month of
the Islamic year
synonym The Hajj, or Pilgrimage to Mecca, is the central
duty of Islam
subtype Qiran, Tamattu’, and Ifrad are three different
types of Hajj
Biography
dates was assassinated on April 4, 1968
nationality was born in Atlanta, Georgia
education entered Boston University as a doctoral student
Drugefﬁcacy

Deﬁnition
Biography
population 37 otherwise healthy children aged 2 to 12 years
problem acute, intercurrent, febrile illness
intervention acetaminophen (10 mg/kg)
outcome ibuprofen provided greater temperature
decrement and longer duration of antipyresis
than acetaminophen when the two drugs were
administered in approximately equal doses
Figure23.21 Examples ofsome different typesof informationthatmustbe extractedinorder
toproduce answer tocertainkinds of complex questions.

DRAFT818 Chapter 23. QuestionAnsweringandSummarization
Ineachcaseweusethe informationextraction methodsofCh. 22toﬁndspeciﬁc
sentences for genus and species (for deﬁnitions), or dates, nationality, and education
(forbiographies),orproblems,interventionsandoutcome s(formedicalquestions). We
can then use standard domain-independent content selectio n algorithms to ﬁnd other
goodsentencestoaddonto these.
A typical architecture consists of the four steps shown in Fi g. 23.22 from the def-
inition extraction system of Blair-Goldensohn et al. (2004 ). The input is a deﬁnition
questionT, the number Nof documentsto retrieve,and the length Lof the answer (in
sentences).
Document 
Retrieval11 Web documents
1127 total 
sentencesPredicate 
IdentificationData-Driven
 Analysis383 Non-Specific Definitional sentences
Sentence clusters, 
Importance ordering
Creation9 Genus-Species Sentences
The Hajj, or pilgrimage to Makkah (Mecca), is the central duty of Islam.
The Hajj is a milestone event in a Muslim's life.

Document 
1127 total 
The hajj is one of five pillars that make up the foundation of Islam.
...
The Hajj, or pilgrimage to Makkah [Mecca], is the central duty of Islam. More than two million Muslims are expected to take 
the Hajj this year. Muslims must perform the hajj at least once in their lifetime if physically and financially able. The Hajj is a 
milestone event in a Muslim's life. The annual hajj begins in the twelfth month of the Islamic year (which is lunar, not solar,

...
milestone event in a Muslim's life. The annual hajj begins in the twelfth month of the Islamic year (which is lunar, not solar, 
so that hajj and Ramadan fall sometimes in summer, sometimes in winter). The Hajj is a week-long pilgrimage that begins 
in the 12th month of the Islamic lunar calendar. Another ceremony, which was not connected with the rites of the Ka'ba 
before the rise of Islam, is the Hajj, the annual pilgrimage to 'Arafat, about two miles east of Mecca, toward Mina…"What is the Hajj?"
N = 20   L = 8
Figure23.22 Architectureofaquery-focusedsummarizerfordeﬁnitionq uestions(Blair-Goldensohnetal.,2004).

N = 20   L = 8
The ﬁrst step in any IE-based complex question answering sys tem is information
retrieval. In this case a handwritten set of patterns is used to extract the term to be
deﬁned from the query T(Hajj) and generate a series of queries that are sent to an IR
engine. Similarly,in a biographysystem it wouldbe the name that wouldbeextracted
andpassedtotheIRengine. Thereturneddocumentsarebroke nupintosentences.
Inthesecondstage,we applyclassiﬁersto labeleachsenten cewithanappropriate

set of classes for the domain. For deﬁnition questions, Blai r-Goldensohnet al. (2004)
used of four classes: genus,species,other deﬁnitional , orother. The third class,
other deﬁnitional , is used to select other sentencesthat might be addedinto th e sum-
mary. These classiﬁers can be based on any of the information extraction techniques
introducedinCh. 22,includinghand-writtenrules,orsupe rvisedmachinelearning.
Inthethirdstage,wecanusethemethodsdescribedinthesec tionongeneric(non-

query-focused)multipledomainsummarizationcontentsel ectiontoaddadditionalsen-
tencestoouranswerthatmightnotfallintoaspeciﬁcinform ationextractiontype. For
example for deﬁnition questions, all the sentences that are classiﬁed as other deﬁni-
tionalare examined,anda set of relevantsentencesis selected fro mthem. This selec-
tion can be done by the centroid method, in which we form a TF-I DF vector for each
sentence, ﬁnd the centroid of all the vectors, and then choos e theKsentences clos-

est to the centroid. Alternatively we can use a method for avo iding redundancy, like
clusteringthevectorsandchoosingthebest sentencefrome achcluster.
Because query-focused summarizers of this type or domain-s peciﬁc, we can use
domain-speciﬁcmethodsforinformationorderingas well, s uch asusing a ﬁxedhand-
builttemplate. Forbiographyquestionswe mightuse atempl atelikethefollowing:

DRAFTSection23.6. SummarizationEvaluation 819
(23.41) <NAME>is<WHYFAMOUS >. Shewasbornon <BIRTHDATE >in
<BIRTHLOCATION >. She<EDUCATION >.<DESCRIPTIVE
SENTENCE >.<DESCRIPTIVESENTENCE >.
The various sentences or phrases selected in the content sel ection phase can then
be ﬁt into this template. These templates can also be somewha t more abstract. For
example, for deﬁnitions, we could place a genus-species sen tence ﬁrst, followed by
remainingsentencesorderedbytheirsaliencyscores.
23.6 SummarizationEvaluation
As is true for other speech and language processing areas lik e machine translation,
there are a wide variety of evaluation metrics for summariza tion, metrics requiring
humanannotation,aswellascompletelyautomaticmetrics.3
Aswehaveseenforothertasks,wecanevaluateasystemvia extrinsic (task-based)
orintrinsic (task-independent)methods. Wedescribedakindofextrins icevaluationof
multi-documentsummarization in Sec. 23.4, in which subjec ts were asked to perform

time-restrictedfact-gatheringtasks,andweregivenfull documentstogetherwitheither
no summaries, humansummaries, or automaticallygenerated summariesto read. The
subjects had to answer three related questions about an even t in the news. For query-
focusedsingle-documentsummarization(likethetaskofge neratingweb snippets),we
canmeasurehowdifferentsummarizationalgorithmsaffect humanperformanceat the
task of deciding if a document is relevant/not-relevantto a query by looking solely at

Themostcommonintrinsicsummarizationevaluationmetric isanautomaticmethod
calledROUGE,Recall-OrientedUnderstudyforGistingEvaluation (LinandHovy, ROUGE
2003;Lin,2004). ROUGEisinspiredbytheBLEUmetricusedfo revaluatingmachine
translation output, and like BLEU, automatically scores a m achine-generated candi-
datesummarybymeasuringtheamountof N-gramoverlapbetweenthecandidateand
human-generatedsummaries(thereferences).
Recall that BLEU is computed by averaging the number of overl appingN-grams

of different length between the hypothesis and reference tr anslations. In ROUGE, by
contrast, the length of the N-gram is ﬁxed; ROUGE-1 uses unigram overlap, while ROUGE-1
ROUGE-2 uses bigram overlap. We’ll choose to deﬁne ROUGE-2; the deﬁn itions of ROUGE-2
all the other ROUGE-N metrics follows. ROUGE-2 is a measure o f the bigram recall
betweenthe candidatesummaryandtheset ofhumanreference summaries:
ROUGE2=∑
S∈{ReferenceSummaries }∑
bigram∈SCountmatch(bigram )
∑

ROUGE2=∑
∑
bigram∈SCount(bigram )(23.42)
3Wefocus hereon evaluation ofentire summarization algorit hms and ignore evaluation ofsubcomponents
such asinformation ordering, although seeforexample (Lap ata, 2006) ontheuseofKendall’s τ,ametric of
rank correlation, for information ordering.

DRAFT820 Chapter 23. QuestionAnsweringandSummarization
The function Countmatch(bigram )returns the maximum number of bigrams that
co-occur in the candidate summary and the set of reference su mmaries. ROUGE-1 is
thesame butcountingunigramsinsteadofbigrams.
NotethatROUGEisarecall-orientedmeasure,whereBLEUisa precision-oriented
measure. Thisis because the denominatorof (23.42)is the to tal sum of the numberof
bigrams in the reference summaries. By contrast, in BLEU the denominator is the
total sum of the number of N-grams in the candidates. Thus ROUGE is measuring
something like how many of the human reference summary bigra ms are covered by
the candidate summary, where BLEU is measuring something li ke how many of the
candidatetranslationbigramsoccurredinthehumanrefere ncetranslations.
VariantsofROUGEinclude ROUGE-L ,whichmeasurethe longestcommonsub- ROUGE-L
sequence betweenthereferenceandcandidatesummaries,and ROUGE-S andROUGE- ROUGE-S

SUwhich measure the number of skip bigrams between the reference and candidate ROUGE-SU
Skipbigram summaries. A skip bigram is a pair of words in their sentence o rder, but allowing for
anynumberofotherwordstoappearbetweenthepair.
While ROUGE is the most commonly applied automatic baseline , it is not as ap-
plicable to summarization as similar metrics like BLEU are t o machine translation.
This is because human summarizers seem to disagree strongly about which sentences
toincludeinasummary,makingeventheoverlapofhumanswit heachotherverylow.

Thisdifferencein whichsentenceshumanschoosetoextract hasmotivatedhuman
evaluation methods which attempt to focus more on meaning. O ne metric, the Pyra-
mid Method , is a way of measuring how many units of meaning are shared bet ween PyramidMethod
the candidate and reference summaries, and also weights the units of meaning by im-
portance;unitsofmeaningwhichoccurinmoreofthehumansu mmariesareweighted
morehighly. Theunitsofmeaningarecalled SummaryContentUnits (SCU),whichSummary Content
Units

Units
aresub-sententialsemanticunitswhichroughlycorrespon dtopropositionsorcoherent
piecesofpropositions.
InthePyramidMethod,humanslabeltheSummaryContentUnit sineachreference
andcandidatesummary,andthenanoverlapmeasureiscomput ed.
Let’s see an example from Nenkova et al. (2007) of how two SCUs are labeled
in sentences from six human abstracts. We’ll ﬁrst show sente nces from the human
summariesindexedbyaletter(correspondingtooneofthe6h umansummaries)anda
number(thepositionofthe sentencein thehumansummary):

Units
A1. The industrial espionage case involving GM and VW began w iththehiringof
JoseIgnacioLopez,anemployeeofGM subsidiary Adam Opel, byVW as a
productiondirector.
B3. However, heleftGMforVW under circumstances, which along with ensuing
events, were described by a German judge as “potentially the biggest-ever case
ofindustrialespionage”.
C6.HeleftGMforVWinMarch1993 .
D6. The issue stems from the alleged recruitmentofGM’s eccentric and visionary

Basque-born procurement chief JoseIgnacioLopez de Arriortura and seven of
Lopez’sbusinesscolleagues.
E1.On March 16, 1993 , with Japanese car import quotas to Europe expiring in two
years, renowned cost-cutter, AgnacioLopezDeArriortura,lefthisjobas head

DRAFTSection23.7. Summary 821
ofpurchasing atGeneralMotor’sOpel,Germany, tobecomeVolkswagen’sPur-
chasingandProductiondirector.
F3.InMarch1993 ,Lopezandsevenother GMexecutives movedtoVWovernight.
The annotators ﬁrst identify similar sentences, like those above, and then label
SCUs. The underlined and italicized spans of words in the abo ve sentences result
in the following two SCUs, each one with a weight correspondi ng to the number of
summariesit appearsin(6fortheﬁrst SCU, and3 forthesecon d):
SCU1(w=6):Lopezleft GM forVW
A1.thehiringofJoseIgnacioLopez,anemployeeofGM...by VW
B3.heleft GMforVW
C6.Heleft GMforVW
D6.recruitmentofGMs. . . Jose IgnacioLopez
E1.AgnacioLopezDeArriortura,lefthisjob. . . atGeneralMoto rsOpel
. . . tobecomeVolkswagens. . . director
F3.Lopez. . . GM. . . movedto VW
SCU2(w=3)Lopezchangesemployersin March1993
C6.inMarch,1993
E1.OnMarch16,1993
F3.InMarch1993
Once the annotationis done, the informativenessof a given s ummarycan be mea-

sured as the ratio of the sum of the weights of its SCUs to the we ight of an optimal
summary with the same number of SCUs. See the end of the chapte r for more details
andpointerstotheliterature.
The standard baselines for evaluating summaries are the random sentences base-Random sentencebaseline
line and the leading sentences baseline. Assuming we are evaluating summaries ofLeading sentence
baseline
lengthNsentences, the random baseline just chooses Nrandom sentences, while the

baseline
leading baseline chooses the ﬁrst Nsentences. The leading sentences method is quite
a strongbaselineandmanyproposedsummarizationalgorith msfailto beatit.
23.7 Summary
•The dominant models of information retrieval represent the meanings of docu-
mentsandqueriesasbagsofwords.
•Thevectorspacemodel viewsdocumentsandqueriesasvectorsinalargemulti-
dimensionalspace. Inthismodel,thesimilaritybetweendo cumentsandqueries,
or other documents, can be measured by the cosine of the angle between the
vectors.

vectors.
•The main componentsof a factoid question answering system a re thequestion
classiﬁcation module to determine the named-entity type of the answer, a pas-
sage retrieval module to identify relevant passages, and an answer process ing
moduleto extractandformattheﬁnalanswer.

DRAFT822 Chapter 23. QuestionAnsweringandSummarization
•Factoidquestionanswerscanbe evaluatedvia mean reciprocalrank (MRR).
•Summarization can be abstractive orextractive ; most current algorithms are
extractive.
•Threecomponentsof summarizationalgorithms includecontentselection ,in-
formationordering ,andsentence realization .
•Current single document summarization algorithms focus ma inly onsentence
extraction , relying on features like positionin the discourse, word informa-
tiveness,cue phrases ,andsentence length .
•Multiple document summarization algorithms often perform sentence simpliﬁ-
cationondocumentsentences.
•Redundancyavoidance is importantin multipledocumentsummarization;it is
often implemented by adding a redundancy penalization term likeMMRinto
sentenceextraction.
•Information ordering algorithms in multi-document summarization are often
basedonmaintaining coherence .
•Query-focusedsummarization canbedoneusingslightmodiﬁcationsto generic

summarization algorithms,orbyusinginformation-extractionmethods.
Luhn (1957) is generally credited with ﬁrst advancing the no tion of fully automatic
indexingofdocumentsbasedontheircontents. Overtheyear sSalton’sSMARTproject
(Salton, 1971) at Cornell developed or evaluated many of the most important notions
ininformationretrievalincludingthevectormodel,termw eightingschemes,relevance
feedback, and the use of cosine as a similarity metric. The no tion of using inverse

document frequency in term weighting is due to Sparck Jones ( 1972). The original
notionofrelevancefeedbackisduetoRocchio(1971).
An alternative to the vector model that we have not covered is theprobabilistic
modeloriginallyshowneffective by Robinsonand SparckJones (19 76). See Crestani Probabilistic IR
et al. (1998)andChapter11ofManninget al. (2008)onprobab ilisticmodelsin infor-
mationretrieval.
Manning et al. (2008) is a comprehensive modern text on infor mation retrieval.

GoodbutslightlyoldertextsincludeBaeza-YatesandRibei ro-Neto(1999)andFrakes
and Baeza-Yates(1992);older classic textsinclude Salton and McGill (1983)and van
Rijsbergen(1975). Manyoftheclassicpapersintheﬁeldcan befoundinSparckJones
and Willett (1997). Current work is published in the annual p roceedingsof the ACM
SpecialInterestGrouponInformationRetrieval(SIGIR).T heUSNationalInstituteof
Standards and Technology (NIST) has run an annual evaluatio n project for text infor-

mationretrievalandextractioncalledthe TextREtrievalC onference(TREC)since the
early 1990s; the conference proceedings from TREC contain r esults from these stan-
dardizedevaluations. Theprimaryjournalsintheﬁeldaret heJournaloftheAmerican

DRAFTSection23.7. Summary 823
Society of InformationSciences ,ACM Transactionson InformationSystems ,Informa-
tionProcessingandManagement ,andInformationRetrieval .
QuestionansweringwasoneoftheearliesttasksforNLPsyst emsinthe1960’sand
1970’s (Green et al., 1961; Simmons, 1965; Woods et al., 1972 b; Lehnert, 1977), but
theﬁeldlaydormantforafewdecadesuntiltheneedforquery ingtheWebbroughtthe
task back into focus. The U.S. government-sponsoredTREC (T ext REtrieval Confer-
ence) QA track began in 1999 and a wide variety of factoid and n on-factoid systems
havebeencompetinginannualevaluationssincethen. Seeth ereferencesinthechapter
andStrzalkowskiandHarabagiu(2006)foracollectionofre centresearchpapers.
ResearchontextsummarizationbeganwiththeworkofLuhn(1 958)onextractive
methods for the automatic generation of abstracts, focusin g on surface features like
term frequency, and the later work of Edmunson (1969) incorp orating positional fea-
turesaswell. Term-basedfeatureswerealsousedintheearl yapplicationofautomatic

summarizationatChemicalAbstractsService(PollockandZ amora,1975). The1970s
and 1980s saw a number of approaches grounded in AI methodolo gy such as scripts
DeJong (1982b), semantic networks Reimer and Hahn (1988), o r combinationsof AI
andstatistical methodsRau etal. (1989).
The work of Kupiec et al. (1995) on training a sentence classi ﬁer with supervised
machine learning led to many statistical methods for senten ce extraction. Around the
turn of the century, the growth of the Web led naturally to int erest in multi-document

summarizationandquery-focusedsummarization.
There have naturally been a wide variety of algorithms for th e main components
of summarizers. The simple unsupervised log-linear conten t selection algorithm we
describe is simpliﬁed from the SumBasic algorithm of Nenkova and Vanderwende SumBasic
(2005),Vanderwendeet al. (2007b)and the centroid algorithmof Radev et al. (2000) Centroid
and Radev et al. (2001). A number of algorithms for informati on ordering have used
entity coherence, including Kibble and Power (2000), Lapat a (2003), Karamanis and

Manurung (2002), Karamanis (2003), Barzilay and Lapata (20 05, 2007). Algorithms
for combining multiple cues for coherence and searching for the optimal ordering in-
clude Althaus et al. (2004), based on linear programming, th e genetic algorithms of
Mellishetal.(1998)andKaramanisandManurung(2002),and theSoricutandMarcu
(2006)algorithm,whichuses A∗searchbased onIDL-expressions. Karamanis(2007)
showed that adding coherence based on rhetorical relations to entity coherence didn’t

improve sentence ordering. See Lapata (2006, 2003), Karama nis et al. (2004), Kara-
manis(2006)onmethodsforevaluatinginformationorderin g.
Sentence compression is a very popular area of research. Ear ly algorithms fo-
cusedontheuseofsyntacticknowledgeforeliminatingless importantwordsorphrases
Grefenstette (1998), Mani et al. (1999), Jing (2000). Recen t research has focused on
using supervised machine learning, in which a parallel corp us of documents together

with their human summaries is used to compute the probabilit y that particular words
or parse nodes will be pruned. Methods include the use of maxi mum entropy Rie-
zler et al. (2003), the noisy channel model and synchronous c ontext-free grammars
(Galley and McKeown, 2007; Knight and Marcu, 2000; Turner an d Charniak, 2005;
Daum´ e III and Marcu, 2002), Integer Linear ProgrammingCla rke and Lapata (2007),
and large-marginlearning McDonald(2006). These methodsr ely on variousfeatures,

especially includingsyntacticor parse knowledgeJing (20 00),Dorret al. (2003),Sid-

DRAFT824 Chapter 23. QuestionAnsweringandSummarization
dharthanet al. (2004),Galley and McKeown (2007),Zajic et a l. (2007),Conroy et al.
(2006), Vanderwende et al. (2007a), but also including cohe rence information Clarke
andLapata(2007). Alternativerecentmethodsareabletofu nctionwithoutthesekinds
of parallel document/summary corpora (Hori and Furui, 2004 ; Turner and Charniak,
2005;ClarkeandLapata,2006).
See Daum´ e III and Marcu (2006) for a recent Bayesian model of query-focused
summarization.
Formoreinformationonsummarizationevaluation,seeNenk ovaetal.(2007),Pas-
sonneau et al. (2005), and Passonneau (2006) for details on t he Pyramid method, van
Halteren and Teufel (2003) and Teufel and van Halteren (2004 ) on related semantic-
coverage evaluation methods, and Lin and Demner-Fushman (2 005) on the link be-
tween evaluations for summarizationand question answerin g. A NIST program start-
ing in 2001, the Document Understanding Conference (DUC), h as sponsored an an-

nual evaluation of summarization algorithms. These have in cluded single document,
multiple document, and query-focused summarization; proc eedings from the annual
workshopareavailableonline.
Mani and Maybury(1999)is the deﬁnitivecollection of class ic paperson summa-
rization. SparckJones(2007)isa goodrecentsurvey,and Ma ni(2001)isthe standard
textbook.
The task of paraphrase detection is an importanttask related to improvingrecallParaphrase
detection

textbook.
detection
in question answering and avoiding redundancy in summariza tion, and also very rel-
evant for tasks like textual entailment. See Lin and Pantel ( 2001), Barzilay and Lee
(2003), Pang et al. (2003), Dolan et al. (2004), Quirk et al. ( 2004) for representative
papersontechniquesfordetectingparaphrases.
Another task related to information retrieval and summariz ation is the text cate-
gorization task, which is to assign a new document to one of a pre-existin g set ofTextcategorization

detection
document classes. The standard approach is to use supervise d machine learning to
train classiﬁers on a set of documentsthat have been labeled with the correct class. A
veryimportantapplicationoftextcategorizationisfor spam detection . Spamdetection
Exercises
23.1Do some erroranalysison web-basedquestionanswering. Cho ose 10 questions
and type them all into two different search engines. Analyze the errors (e.g.,
what kinds of questions could neither system answer; which k inds of questions

Exercises
didoneworkbetter on;was therea typeofquestionthat could beansweredjust
fromthesnippets,etc).
23.2Read Brill et al. (2002) and reimplement a simple version of t he AskMSR sys-
tem.

Chapter24
DialogueandConversationalAgents
C: I want youtotellme the names of the fellowsonthe St. Louis team.
A: I’m tellingyou. Who’s onﬁrst, What’s on second, IDon’t Kn ow is onthird.
C: Youknow the fellows’names?
A: Yes.
C: Well, then, who’s playing ﬁrst?
A: Yes.
C: I meanthe fellow’s name on ﬁrst.
A: Who.
C: The guy onﬁrst base.
A: Who is onﬁrst.
C: Well what are youaskin’ mefor?
A: I’m not asking you–I’m tellingyou. Who ison ﬁrst.
Who’s on First –Bud Abbott and Lou Costello’s version of an
oldburlesque standard.
The literature of the fantastic abounds in inanimate object s magically endowed with

A: Yes.
A: Yes.
A: Who.
sentience and the gift of speech. From Ovid’s statue of Pygma lion to Mary Shelley’s
Frankenstein, Cao Xue Qin’s Divine Luminescent Stone-in-W aiting to Snow White’s
mirror, there is something deeply touching about creating s omething and then having
a chat with it. Legend has it that after ﬁnishing his sculptur e ofMoses, Michelangelo
thoughtit so lifelikethathetappedit onthekneeandcomman dedit tospeak. Perhaps
thisshouldn’tbesurprising. Languagehasalwaysbeenthem arkofhumanityandsen-

tience,and conversation ordialogue isthemostfundamentalandspeciallyprivileged conversation
Dialogue arenaof language. It is theﬁrst kindoflanguagewe learnasc hildren,andformost of
us, it is the kind of language we most commonly indulge in, whe ther we are ordering
curry for lunch or buying spinach, participating in busines s meetings or talking with
ourfamilies, bookingairlineﬂightsorcomplainingaboutt heweather.
This chapter introduces the fundamental structures and alg orithms in conversa-

tional agents . Conversational agents most often communicate via speech r ather than
text, and so they are also known as spoken dialogue systems , orspoken language
systems. Theseprogramscommunicatewithusersinspokennaturalla nguageinorder
tomaketravelarrangements,answerquestionsaboutweathe rorsports,routetelephone
calls, actasa generaltelephoneassistant, orperformeven moresophisticatedtasks.
Many of these tasks involve telephony,and also involve mobi le situations where a

large screen and keyboard may not be available (Cohen and Ovi att, 1994). Applica-
tionsrelatingtotravelarethusverycommon. Indeed,trave lplanningandmanagement
hasbeena keyconcernofdialoguesystemssince theearlyinﬂ uentialGUS systemfor
planning airline travel (Bobrow et al., 1977). More recentl y, a number of spoken lan-
guage systems for booking airline ﬂights were built under th e Air Trafﬁc Information
System (ATIS) (Hemphill et al., 1990) and Communicator (Wal ker, 2001) programs.

DRAFT826 Chapter 24. DialogueandConversationalAgents
We will use this travel domain as an example throughoutthe ch apter (as we used it to
discuss grammarrules for parsing ATIS sentences in Ch. 12). Fig. 24 shows a sample
dialoguefromaCommunicatorsystem,takenfromXu andRudni cky(2000).
S1: Hello. You’ve reached the [Communicator]. Tellme your f ull name
U2:hi i’dlike toﬂy toseattletuesday morning
S3: TravelingtoSeattleon Tuesday, August eleventh inthe m orning. Your full name?
U4:[John Doe]
S5: Hello, [John], I’m gladtohear from you again. Youcan now askme for help.
S6: OK.Let’sseeIhaveaUnitedﬂightdepartingPittsburgha ttenfortyﬁvea.m. arriving
intoSeattleat one ﬁftyseven p.m. Isthat OK?
U7:Yes
S8: Willyou returntoPittsburghfrom Seattle?
U9:no iwillgotoLos Angeles
S10: TravelingtoLos Angeles. onwhat date
U11:Sorry can wego back totalk about theprevious legagain
S12: For the ﬁrst leg, you are on a United ﬂight 1446 departing Pittsburghat ten forty ﬁve

U7:Yes
A.M. tomorrow, arriving inSeattleat one ﬁftyseven P.M.Doy ou want totake that?
U13:actually iwant toleave on wednesday
S14: a ﬂight onwednesday.
S15: I have a ﬂight departing Pittsburgh at ten forty ﬁve a.m. , arrives Seattle at one ﬁfty
seven p.m. Is that OK?
U16:Yes
Figure24.1 Thetraveldomain: afragmentfromasuccessfulconversatio nbetweenauser(U)
and the Communicator system (S)of XuandRudnicky (2000).
Anotherpromisingdomainis automaticcall routing. A call r outingsystem directs

U16:Yes
incomingcallsinatelephonecallcenter,transferringthe calltotheappropriatehuman.
Callroutingdialoguesystemsgenerallybeginbyaskingthe useropen-endedquestions
likeHow may I help you? (Gorin et al., 1997), or How may I direct your call? (Chu-
Carroll and Carpenter, 1999). Fig. 24 shows sample conversa tions from the AT&T
HowMayIHelp You(HMIHY)system (Gorinet al.,1997).
S: How may Ihelp you?
U:can youtell mehow muchitis totokyo?
S: Youwant toknow the cost of a call?
U:yes that’s right
S: Please holdon for rateinformation.

U:what’s the area code for chicago?
S: Youwant anarea code.
U:(silence)
S: The area code for Chicagois 312.
Figure24.2 The call routing domain: two conversations between a user (U ) and the AT&T
How May IHelpYou(HMIHY) system (S)(Gorinet al.,1997).
A somewhat different domain, which is very far from the telep hony area, is auto-
matic tutoring systems. Fig. 24 shows part of a dialoguefrom the ITSPOKE dialogue

DRAFTSection24.1. PropertiesofHumanConversations 827
system (Litman and Silliman, 2004), which uses the Why2-Atl as text-based tutoring
systemasits tutoringbackend(VanLehnet al.,2002).
S: Ididn’t understand. What force acts onallobjects within close proximitytoearth?
U:uhthe force of gravity
S: Fine. Besides the Earth’s gravitational force, are there any other forces acting on the
person, keys, and elevator after the release?
U:no
S: Yep. Thus, keep inmindthat after the release,the onlyfor ce on the person, keys, and
elevator is the force of gravity. If gravity is the only force acting on an object, will it
be moving or staying still?
U:itwillbe moving
S: Fine. Notonlyaretheperson,keys, andelevator moving,t heyhaveonlygravitational
forces on them. When an object is falling and has only gravita tional force on it, it is
saidtobe inwhat?
U:freefall
Figure24.3 The tutoring domain: part of an conversation between a stude nt user (U) and the
ITSPOKEsystem (S)of LitmanandSilliman(2004).

Sec. 24.1 starts out with a summary of facts about human conve rsation, includ-
ing the idea of turns and utterances, speech acts, grounding , dialogue structure, and
conversational implicature. The next few sections introdu ce the components of spo-
ken language systems and some evaluation metrics. We then tu rn in Sec. 24.5 and
Sec. 24.6 to the more sophisticated information-state and M arkov decision processes
modelsof conversationalagents, and we concludewith some a dvancedtopics like the
BDI (belief-desire-intention)paradigm.

24.1 PropertiesofHumanConversations
Conversationbetweenhumansisanintricateandcomplexjoi ntactivity. Becauseofthe
limitations of our current technologies, conversations be tween humans and machines
are vastly simpler and more constrained than these human con versations. Nonethe-
less, beforewe attemptto designa conversationalagentto c onversewith humans,it is
crucialtounderstandsomethingabouthowhumansconversew itheachother.
In this section we discuss some propertiesof human-humanco nversationthat dis-

tinguish it from the kinds of (text-based) discourses we hav e seen so far. The main
difference is that conversation is a kind of joint activity between two (or more) in-
terlocutors. This basic fact has a number of ramiﬁcations; c onversations are built up
out of consecutive turns, each turn consists of joint action of the speaker and hearer,
and the hearer make special inferences called conversational implicatures about the
speaker’sintendedmeaning.
24.1.1 Turns and Turn-Taking

Dialogueis characterizedby turn-taking ;SpeakerA says something,then speakerB, Turn-taking

DRAFT828 Chapter 24. DialogueandConversationalAgents
then speaker A, and so on. If having a turn (or “taking the ﬂoor ”) is a resource to be
allocated, what is the process by which turns are allocated? How do speakers know
whenitis thepropertime tocontributetheirturn?
It turns out that conversation and language itself are struc tured in such a way as
to deal efﬁciently with this resource allocation problem. O ne source of evidence for
this is the timing of the utterances in normal human conversa tions. While speakers
can overlap each other while talking, it turns out that on ave rage the total amount of
overlapisremarkablysmall; perhapslessthan5% (Levinson ,1983). Furthermore,the
amountoftimebetweenturnsisgenerallylessthanafewhund redmilliseconds,which
isquiteshortgiventhatittakesaspeakerhundredsofmilli secondsforaspeakertoplan
the motor routines for an utterance. Thus speakers must begi n planning exactly what
moment to start their next utterance before the previous spe aker has ﬁnished talking.

For this to be possible, natural conversation must be set up i n such a way that (most
of the time) people can quickly ﬁgure out whoshould talk next, and exactly when
they should talk. This kind of turn-taking behavior is gener ally studied in the ﬁeld of
ConversationAnalysis (CA).Inakeyconversation-analyticpaper,Sacksetal.(1974)ConversationAnalysis
arguedthat turn-takingbehavior,at least in American Engl ish, is governedby a set of
turn-taking rules. These rules apply at a transition-relevance place , orTRP; places

where the structure of the language allows speaker shift to o ccur. Here is a version of
theturn-takingrulessimpliﬁedfromSacksetal. (1974):
(24.1)Turn-takingRule. At eachTRP ofeachturn:
a. IfduringthisturnthecurrentspeakerhasselectedA asth enextspeaker
thenA mustspeaknext.
b. Ifthecurrentspeakerdoesnotselectthe nextspeaker,an yotherspeaker
maytakethenextturn.
c. Ifnooneelse takesthenextturn,currentspeakermaytake the nextturn.
There are a number of important implications of rule (24.1) f or dialogue model-

ing. First, subrule(24.1a)impliesthat there are some utte rancesby which the speaker
speciﬁcally selects who the next speaker will be. The most ob viousof these are ques-
tions, in which the speaker selects another speaker to answe r the question. Two-part
structures like QUESTION -ANSWER are called adjacency pairs (Schegloff, 1968) or Adjacencypair
dialogic pair (Harris, 2005). Other adjacency pairs include GREETING followed by Dialogic pair

GREETING ,COMPLIMENT followedby DOWNPLAYER ,REQUEST followedby GRANT.
We will see that these pairs and the dialogueexpectationsth ey set up will play an im-
portantroleindialoguemodeling.
Subrule (24.1a) also has an implication for the interpretat ion of silence. While
silence can occur after any turn, silence in between the two p arts of an adjacencypair
issigniﬁcantsilence . ForexampleLevinson(1983)notesthisexamplefromAtkins on Signiﬁcant silence
andDrew(1979);pauselengthsaremarkedin parentheses(in seconds):
(24.2)A: Istheresomethingbotheringyouornot?
(1.0)

(1.0)
A: Yesorno?
(1.5)
A: Eh?
B: No.

DRAFTSection24.1. PropertiesofHumanConversations 829
SinceAhasjustaskedBaquestion,thesilenceisinterprete dasarefusaltorespond,
orperhapsa dispreferred response(aresponse,likesaying“no”toarequest,whichis dispreferred
stigmatized). By contrast, silence in other places, for exa mple a lapse after a speaker
ﬁnishes a turn, is not generally interpretable in this way. T hese facts are relevant for
user interface design in spoken dialoguesystems; users are disturbedby the pauses in
dialoguesystemscausedbyslowspeechrecognizers(Yankel ovichetal., 1995).
Another implication of (24.1) is that transitions between s peakers don’t occur just
anywhere; the transition-relevance places where they tend to occur are generally at
utterance boundaries. Recall from Ch. 12 that spoken utterances diffe r from written Utterance
sentences in a number of ways. They tend to be shorter, are mor e likely to be single
clauses or even just single words, the subjects are usually p ronouns rather than full

lexicalnounphrases,andtheyincludeﬁlledpausesandrepa irs. A hearermusttakeall
this(andothercueslikeprosody)intoaccounttoknowwhere tobegintalking.
24.1.2 Languageas Action: Speech Acts
The previous section showed that conversation consists of a sequence of turns, each
of which consists of one or more utterance. A key insight into conversation due to
Wittgenstein(1953)butworkedoutmorefullybyAustin(196 2)isthatanutterancein
a dialogueisa kindof actionbeingperformedbythe speaker.

The idea that an utterance is a kind of action is particularly clear inperformative performative
sentenceslikethe following:
(24.3)Inamethisshipthe Titanic.
(24.4)Isecondthatmotion.
(24.5)Ibet youﬁvedollarsit will snowtomorrow.
When uttered by the proper authority, for example, (24.3) ha s the effect of changing
thestateoftheworld(causingtheshiptohavethename Titanic)justasanyactioncan
change the state of the world. Verbs like nameorsecondwhich perform this kind of

action are called performative verbs, and Austin called the se kinds of actions speech
acts. What makes Austin’s work so far-reaching is that speech act s are not conﬁned Speechact
to this small class of performative verbs. Austin’s claim is that the utterance of any
sentenceina realspeechsituationconstitutesthreekinds ofacts:
locutionaryact: theutteranceofa sentencewith aparticularmeaning.
illocutionaryact: the act of asking, answering,promising,etc., in utteringa
sentence.

sentence.
perlocutionaryact: the (often intentional) production of certain effects upon
thefeelings,thoughts,oractionsoftheaddresseeinutter -
inga sentence.
Forexample,Austinexplainsthattheutteranceofexample( 24.6)mighthavethe il-
locutionaryforce ofprotestingandtheperlocutionaryeffectofstoppingthe addressee Illocutionaryforce
fromdoingsomething,orannoyingthe addressee.
(24.6)Youcan’tdothat.

DRAFT830 Chapter 24. DialogueandConversationalAgents
The term speech act is generally used to describe illocutionary acts rather tha n
either of the other two types of acts. Searle (1975b), in modi fying a taxonomy of
Austin’s,suggeststhat allspeechactscanbeclassiﬁed int ooneofﬁvemajorclasses:
Assertives: committing the speaker to something’s being the case ( suggesting ,
puttingforward ,swearing,boasting,concluding ).
Directives: attemptsbythespeakertogettheaddresseetodosomething( asking,
ordering,requesting ,inviting,advising,begging).
Commissives: committing the speaker to some future course of action ( promising,
planning,vowing,betting,opposing ).
Expressives: expressing the psychologicalstate of the speaker about a st ate of af-
fairsthanking,apologizing ,welcoming ,deploring .
Declarations: bringingaboutadifferentstateoftheworldviatheutteran ce(includ-
ingmanyoftheperformativeexamplesabove; Iresign,You’reﬁred .)
24.1.3 Languageas JointAction: Grounding

Theprevioussectionsuggestedthateachturnorutterancec ouldbeviewedasanaction
by a speaker. But dialogue is not a series of unrelated indepe ndent acts. Instead,
dialogue is a collective act performed by the speaker and the hearer. One implication
of joint action is that, unlike in monologue, the speaker and hearer must constantly
establish [Common ground]common ground (Stalnaker, 1978), the set of t hings that [
are mutually believed by both speakers. The need to achieve c ommon ground means

that the hearer must groundthe speaker’s utterances, making it clear that the hearer Grounding
hasunderstoodthespeaker’smeaningandintention.
As Clark (1996) points out, people need closure or grounding for non-linguistic
actionsas well. For example,why doesa well-designedeleva torbutton light up when
it’s pressed? Because this indicates to the elevator travel er that she has successfully
calledtheelevator. Clarkphrasesthisneedforclosureasf ollows,afterNorman(1988):

Principle of closure. Agents performing an action require evidence, sufﬁcient
forcurrentpurposes,thattheyhavesucceededinperformin git.
Groundingis also importantwhenthe hearerneedsto indicat ethatthe speakerhas
notsucceeded in performing an action. If the hearer has problem s in understanding,
she must indicate these problems to the speaker, again so tha t mutual understanding
caneventuallybeachieved.
How is closure achieved? Clark and Schaefer (1989) introduc e the idea that each

joint linguistic act or contribution has two phases, called presentation andaccep- Contribution
tance. In the ﬁrst phase, a speaker presents the hearer with an utte rance, performing
a sort of speech act. In the acceptance phase, the hearer has t o ground the utterance,
indicatingtothespeakerwhetherunderstandingwasachiev ed.
What methodscan the hearer(call her B) use to groundthe spea kerA’s utterance?
Clark and Schaefer (1989) discuss ﬁve main types of methods, ordered from weakest
tostrongest:

DRAFTSection24.1. PropertiesofHumanConversations 831
Continuedattention: B shows she is continuingto attend and thereforeremains sat isﬁed with
A’spresentation.
NextContribution: Bstartsin onthe nextrelevantcontribution.
Acknowledgement: B nods or says a continuer like uh-huh,yeah, or the like, or an assess-
mentlikethat’sgreat .
Demonstration: B demonstrates all or part of what she has understood A to mean , for
example by reformulating (paraphrasing)A’s utterance, or by collabo-
rativecompletion ofA’sutterance.
Display: Bdisplaysverbatimall orpartofA’spresentation.
Let’s look for examples of grounding in a conversation betwe en a human travel
agent and a human client in Fig. 24.4. We’ll return to this dia logue throughout the
chaptertoinformourdesignofa machinetraveldialogueage nt.
C1: ...I needtotravelin May.
A1: And,whatdayinMaydidyouwant totravel?
C2: OKuhIneedtobethereforameetingthat’sfromthe12thtoth e15th.
A2: Andyou’reﬂyingintowhatcity?
C3: Seattle.
A3: Andwhattime wouldyouliketoleavePittsburgh?

C4: UhhmmI don’tthinkthere’smanyoptionsfornon-stop.
A4: Right. There’sthreenon-stopstoday.
C5: Whatarethey?
A5: TheﬁrstonedepartsPGHat10:00amarrivesSeattleat12:05 theirtime. The
second ﬂight departs PGH at 5:55pm, arrives Seattle at 8pm. A nd the last
ﬂightdepartsPGH at 8:15pmarrivesSeattle at 10:28pm.
C6: OKI’ll takethe5ishﬂightonthenightbeforeonthe11th.
A6: Onthe11th? OK.Departingat5:55pmarrivesSeattleat8pm, U.S.Airﬂight
115.
C7: OK.
Figure24.4 Partof a conversation between a travelagent (A) andclient ( C).

115.
C7: OK.
UtteranceA 1,inwhichtheagentrepeats inMay,repeatedbelowinboldface,shows
the strongest form of grounding, in which the hearer display s their understanding by
repeatingverbatimpartofthespeakerswords:
C1:...I needtotravel in May.
A1:And,whatday in Maydidyouwantto travel?
This particular fragment doesn’t have an example of an acknowledgement , but
there’sanexampleinanotherfragment:
C:Hewantstoﬂy fromBoston
A:Mm hmm
C:toBaltimoreWashingtonInternational
The word mm-hmm here is a continuer , also often called a backchannel or an Continuer
backchannel

DRAFT832 Chapter 24. DialogueandConversationalAgents
acknowledgement token . A continuer is a (short) optional utterance which acknowl-
edgesthe contentof the utteranceof the other, and which doe sn’trequirean acknowl-
edgement by the other (Yngve, 1970; Jefferson, 1984; Schegl off, 1982; Ward and
Tsukahara,2000).
In Clark and Schaefer’s third method, the speaker starts in o n their relevant next
contribution. We see a number of examples of this in the dialo gue in Fig. 24.4, for
example where the speaker asks a question and the hearer answ ers it. We mentioned
theseadjacencypairs above;otherexamplesinclude PROPOSAL followedby ACCEP-
TANCEorREJECTION ,APOLOGY followedby ACCEPTANCE /REJECTION ,SUMMONS
followedby ANSWER,andsoon.
Inamoresubtlebutveryimportantkindofgroundingact,the speakercancombine
thismethodwiththepreviousone. Forexamplenoticethatwh enevertheclientanswers
aquestion,theagentbeginsthenextquestionwith And. TheAndindicatestotheclient

thattheagenthassuccessfullyunderstoodtheanswertothe last question:
And,whatdayinMaydidyouwant totravel?
...
Andyou’reﬂyingintowhatcity?
...
Andwhat timewouldyouliketoleave Pittsburgh?
As we will see in Sec. 24.5, the notions of grounding and contr ibutions can be
combined with speech acts to give a more sophisticated model of joint action in con-
versation;thesemoresophisticatedmodelsarecalled dialogueacts .
Groundingis just ascrucial in human-machineconversation as it is in humancon-

...
...
versation. The examples below, from Cohen et al. (2004), sug gest how unnatural it
sounds when a machine doesn’t ground properly. The use of Okaymakes (24.7) a
muchmorenaturalresponsethan(24.8)to groundauser’srej ection:
(24.7)System: Didyouwanttoreviewsomemoreofyourpersonalproﬁ le?
Caller: No.
System:Okay,what’snext?
(24.8)System: Didyouwanttoreviewsomemoreofyourpersonalproﬁ le?
System: What’snext?
Indeed, this kind of lack of grounding can cause errors. Stif elman et al. (1993)

and Yankelovich et al. (1995) found that humans get confused when a conversational
systemdoesn’tgiveexplicitacknowledgements.
24.1.4 ConversationalStructure
We havealready seen how conversationis structuredby adjac encypairs and contribu-
tions. Herewe’ll brieﬂydiscussone aspectof the overallorganization ofa conversa-
tion: conversationalopenings. The openingsof telephonec onversations,for example,
tendtohavea4-partstructure(Clark,1994;Schegloff,196 8,1979):
Stage1:Entera conversation,with summons-responseadjacencypai r

Stage2:Identiﬁcation

DRAFTSection24.1. PropertiesofHumanConversations 833
Stage3:Establishjointwillingnesstoconverse
Stage4:Theﬁrst topicis raised,usuallybythecaller.
These four stages appear in the opening of this short task-or iented conversation
fromClark(1994).
Stage Speaker& Utterance
1A1:(ringsB’s telephone)
1,2B1:BenjaminHolloway
2A1:thisisProfessorDwight’ssecretary,fromPolymaniaColle ge
2,3B1:oohyes–
4A1:uh:m. aboutthe: lexicology*seminar*
4B1:*yes*
Itiscommonforthepersonwhoanswersthephonetospeakﬁrst (sincethecaller’s
ring functions as the ﬁrst part of the adjacency pair) but for the caller to bring up the
ﬁrst topic, as the caller did above concerning the “lexicolo gyseminar”. This fact that
the caller usually brings up the ﬁrst topic causes confusion when the answerer brings
up the ﬁrst topic instead; here’s an example of this from the B ritish directory enquiry
servicefromClark(1994):
Customer: (rings)
Operator: DirectoryEnquiries,forwhichtownplease?
Customer: Couldyougivemethe phonenumberofum: Mrs. um: Sm ithson?

Operator: Yes,whichtownis thisat please?
Customer: Huddleston.
Operator: Yes. Andthenameagain?
Customer: Mrs. Smithson.
Intheconversationabove,theoperatorbringsupthetopic( forwhichtownplease? )
inherﬁrstsentence,confusingthecaller,whoignoresthis topicandbringsupherown.
This fact that callers expect to bring up the topic explains w hy conversational agents
for call routing or directory information often use very ope n prompts like How may

I help you? orHow may I direct your call? rather than a directive prompt like For
which town please? . Open prompts allow the caller to state their own topic, redu cing
recognitionerrorscausedbycustomerconfusion.
Conversation has many other kinds of structure, including t he intricate nature of
conversational closings and the wide use of presequences. W e will discuss structure
basedon coherence inSec. 24.7.
24.1.5 ConversationalImplicature

We have seen that conversation is a kind of joint activity, in which speakers produce
turns according to a systematic framework, and that the cont ributions made by these
turns include a presentation phase of performing a kind of ac tion, and an acceptance
phaseofgroundingthepreviousactionsoftheinterlocutor . Sofarwehaveonlytalked
aboutwhatmightbecalledthe‘infrastructure’ofconversa tion. Butwehavesofarsaid
nothingabouttheactualinformationthatgetscommunicate dfromspeakertohearerin
dialogue.

DRAFT834 Chapter 24. DialogueandConversationalAgents
While Ch. 17 showed how we can compute meanings from sentence s, it turns out
thatinconversation,themeaningofacontributionisoften quiteabitextendedfromthe
compositional meaning that might be assigned from the words alone. This is because
inferenceplays a crucial role in conversation. The interpr etationof an utterancerelies
on more than just the literal meaning of the sentences. Consi der the client’s response
C2fromthesampleconversationinFig.24.4,repeatedhere:
C2: OKuhIneedtobethereforameetingthat’sfromthe 12thtoth e15th.
Noticethattheclientdoesnotinfactanswerthequestion. T heclient merelystates
that he has a meeting at a certain time. The semantics for this sentence produced by
a semantic interpreter will simply mention this meeting. Wh at is it that licenses the
agenttoinferthattheclientismentioningthismeetingsoa stoinformtheagentofthe
traveldates?

Now consider another utterance from the sample conversatio n, this one by the
agent:
A4: ...There’sthreenon-stopstoday.
Now this statement would still be true if there were seven non -stops today, since
if there are seven of something, there are by deﬁnition also t hree. But what the agent
means here is that there are three and not more than three non-stops today. How is
theclientto inferthat theagentmeans onlythree non-stops?
Thesetwocaseshavesomethingincommon;inbothcasesthesp eakerseemstoex-

agent:
pectthehearertodrawcertaininferences;inotherwords,t hespeakeriscommunicating
more informationthan seems to be present in the uttered word s. These kind of exam-
ples were pointed out by Grice (1975, 1978) as part of his theo ry ofconversational
implicature .Implicature means a particular class of licensed inferences. Grice pro- Implicature
posedthatwhat enableshearerstodrawthese inferencesist hat conversationisguided

by a set of maxims, general heuristics which play a guiding role in the interpr etation Maxim
ofconversationalutterances. He proposedthefollowingfo urmaxims:
•MaximofQuantity: Be exactlyasinformativeasis required: quantity
1. Make your contribution as informative as is required (for the current pur-
posesofthe exchange).
2. Donotmakeyourcontributionmoreinformativethanisreq uired.
•MaximofQuality: Trytomakeyourcontributiononethatis true: quality
1. Donotsay whatyoubelieveto befalse.

2. Donotsay thatforwhichyoulack adequateevidence.
•MaximofRelevance: Be relevant. relevance
•MaximofManner: Be perspicuous: manner
1. Avoidobscurityofexpression.
2. Avoidambiguity.
3. Be brief(avoidunnecessaryprolixity).
4. Be orderly.

DRAFTSection24.2. Basic DialogueSystems 835
ItistheMaximofQuantity(speciﬁcallyQuantity1)thatall owsthehearertoknow
thatthreenon-stops didnotmean sevennon-stops . Thisis becausethehearerassumes
the speaker is following the maxims, and thus if the speaker m eant seven non-stops
she would have said seven non-stops (“as informative as is re quired”). The Maxim
of Relevance is what allows the agent to know that the client w ants to travel by the
12th. The agent assumes the client is following the maxims, a nd hence would only
have mentioned the meeting if it was relevant at this point in the dialogue. The most
natural inference that would make the meeting relevant is th e inference that the client
meanttheagentto understandthathisdeparturetimewasbef orethemeetingtime.
24.2 BasicDialogueSystems
We’venowseen a bitabouthowhumandialogueworks,although aswe’ll see, notev-
ery aspect of human-humanconversationis modeled in human- machineconversation.

Let’sthereforeturntothespokendialoguesystemsusedinc ommercialapplications.
Fig.24.5showsatypicalarchitectureforadialoguesystem . Ithassixcomponents.
Thespeechrecognitionandunderstandingcomponentsextra ctmeaningfromtheinput,
whilethegenerationandTTScomponentsmapfrommeaningtos peech. Thedialogue
manager controlsthe whole process, along with a task manage rwhich has knowledge
about the task domain (such as air travel). We’ll go through e ach of these compo-

nents in the next sections. Then we’ll explore more sophisti cated research systems in
followingsections.
S
RecognitionNatural Language
Dialogue
ManagerTask
Manager
Natural Language
GenerationText-to-Speech 
Synthesis
Figure24.5 Simpliﬁedarchitecture ofthe components of a conversation al agent.
24.2.1 ASR component
TheASR(automaticspeechrecognition)componenttakesaud ioinput,generallyfrom
a telephone,orfroma PDA ordesktopmicrophone,andreturns a transcribedstringof
words,asdiscussedin Ch.9.

S
Dialogue
Manager
Synthesis
Various aspects of the ASR system may be optimized speciﬁcal ly for use in con-
versationalagents. Forexample,thelargevocabularyspee chrecognizerswediscussed
inCh.9fordictationortranscriptionfocusedontranscrib inganysentenceonanytopic
using any English word. But for domain-dependentdialogue s ystems it is of little use

DRAFT836 Chapter 24. DialogueandConversationalAgents
tobeabletotranscribesuchawidevarietyofsentences. The sentencesthatthespeech
recognizerneedsto be able to transcribeneedare just those that can be understoodby
the natural language understanding component. For this rea son commercial dialogue
systems generally use non-probabilistic language models b ased on ﬁnite-state gram-
mars. These grammars are generally hand-written, and speci fy all possible responses
that the system understands. We’ll see an example of such a ha nd-written grammar
for a VoiceXML system in Sec. 24.3. Such grammar-basedlangu age models can also
be compiled automatically from, e.g., uniﬁcation grammars used for natural language
understanding(Rayneret al.,2006).
Becausewhattheusersaystothesystemisrelatedtowhatthe systemhasjustsaid,
languagemodelsinconversationalagentareusually dialogue-statedependent . Forex-
ample, if the system has just asked the user “What city are you departing from?”, the

ASRlanguagemodelcanbeconstrainedtoonlyconsistofcity names,orperhapssen-
tencesoftheform‘Iwantto(leave |depart)from[CITYNAME]’.Thesedialogue-state-
speciﬁc language models often consist of hand-written ﬁnit e-state (or even context-
free)grammarsasdiscussedabove,oneforeachdialoguesta te.
In some systems, the understanding component is more powerf ul, and the set of
sentences the system can understand is larger. In such cases , instead of a ﬁnite-state
grammar, we can use an N-gram language model whose probabilities are similarly
conditionedonthedialoguestate.

Whether we use a ﬁnite-state, context-free,or an N-gram languagemodel,we call
such a dialogue-state dependent language model a restrictive grammar . When theRestrictivegrammar
system wants to constrain the user to respond to the system’s last utterance, it can use
a restrictivegrammar. Whenthe systemwantstoallowthe use rmoreoptions,it might
mix this state-speciﬁc language model with a more general la nguage model. As we
willsee,thechoicebetweenthesestrategiescanbetunedba sedonhowmuch initiative
theuserisallowed.

Speechrecognitionindialogue,aswellasinmanyotherappl icationslikedictation,
has the property that the identity of the speaker remains con stant across many utter-
ances. This means that speaker adaptation techniques like M LLR and VTLN (Ch. 9)
canbeappliedto improverecognitionasthe systemgetsmore speechfromtheuser.
Embedding an ASR engine in a dialogue system also requires th at an ASR en-
gine to have realtime response, since users are unwilling to accept long pauses before
responses. Dialoguesystemsalsogenerallyrequirethatan ASRsystemreturna conﬁ-

dencevalue for a sentence, which can then be used for example for de ciding whether
toask theusertoconﬁrma response.
24.2.2 NLU component
The NLU (naturallanguageunderstanding)componentof dial oguesystems must pro-
duceasemanticrepresentationwhichisappropriateforthe dialoguetask. Manyspeech-
baseddialoguesystems,sinceasfarbackastheGUSsystem(B obrowetal.,1977),are
based on the frame-and-slot semantics discussed in Chapter 15. A travel system, for
example,which has the goal of helpinga user ﬁnd an appropria teﬂight, wouldhave a

frame with slots for information about the ﬂight; thus a sent ence like Show me morn-
ingﬂightsfromBostontoSanFranciscoonTuesday mightcorrespondtothefollowing

DRAFTSection24.2. Basic DialogueSystems 837
ﬁlled-outframe(fromMiller etal. (1994)):
SHOW:
FLIGHTS:
ORIGIN:
CITY: Boston
DATE:
DAY-OF-WEEK: Tuesday
TIME:
PART-OF-DAY: morning
DEST:
CITY: San Francisco
How does the NLU component generate this semantic represent ation? Some dia-
logue systems use general-purpose uniﬁcation grammars wit h semantic attachments,
suchastheCoreLanguageEngineintroducedinCh.18. A parse rproducesasentence
meaning,fromwhichtheslot-ﬁllersareextracted(Lewinet al.,1999).
Other dialogue systems rely on simpler domain-speciﬁc sema ntic analyzers, such
assemanticgrammars . AsemanticgrammarisaCFGinwhichtherulelefthandsides
correspondtothesemanticentitiesbeingexpressed,asin t hefollowingfragment:
SHOW →show me|iwant|cani see|...
DEPART TIMERANGE→(after|around|before) HOUR|
morning|afternoon|evening
HOUR →one|two|three|four...|twelve(AMPM)
FLIGHTS →(a)ﬂight|ﬂights
AMPM →am|pm
ORIGIN →from CITY
DESTINATION →toCITY
CITY →Boston|SanFrancisco|Denver|Washington

These grammars take the form of context-free grammars or rec ursive transition
networks (Issar and Ward, 1993; Ward and Issar, 1994), and he nce can be parsed by
anystandardCFGparsingalgorithm,suchastheCKYorEarley algorithmsintroduced
in Ch. 13. The result of the CFG or RTN parse is a hierarchical l abeling of the input
stringwithsemanticnodelabels:
SHOW FLIGHTS ORIGIN DESTINATION DEPART_DATE DEPART_TIME
to CITY
Show me flights from boston to san francisco on tuesday morni ng

to CITY
Since semantic grammar nodes like ORIGIN correspond to the s lots in the frame,
the slot-ﬁllers can be read almost directly off the resultin g parse above. It remains
only to put the ﬁllers into some sort of canonical form (for ex ample as discussed in
Chapter15datescanbenormalizedintoaDD:MM:YYform,time sinto24-hourtime,
etc.).
Thesemanticgrammarapproachisverywidelyused,butisuna bletodealwitham-
biguity,andrequireshand-writtengrammarsthatcanbeexp ensiveandslow tocreate.

etc.).
Ambiguity can be addressed by adding probabilities to the gr ammar; one such
probabilistic semantic grammar system is the TINA system (S eneff, 1995) shown in
Fig. 24.6; note the mix of syntactic and semantic node names. The grammar rules in
TINA are written by hand, but parse tree node probabilities a re trained by a modiﬁed
versionoftheSCFG methoddescribedinCh. 14.
An alternative to semantic grammars that is probabilistic a nd also avoids hand-
coding of grammars is the semantic HMM model of Pieraccini et al. (1991). The

DRAFT838 Chapter 24. DialogueandConversationalAgents
S
Q-SUBJECT
WHAT
WhatSTREET
streetBE-QUESTION
LINK
isSUBJECT
ARTICLE
theA-PLACE
A-HOTEL
HOTEL-NAME
HyattPRED-ADJUNCT
ON-STREET
ON
onA-STREET
Q-SUBJECT
Figure24.6 A parse of a sentence inthe TINAsemantic grammar, after Sene ff (1995).
hidden states of this HMM are semantic slot labels, while the observed words are the
ﬁllers of the slots. Fig. 24.7 shows how a sequence of hidden s tates, correspondingto
slot names, couldbe decodedfrom(or could generate)a seque nceof observedwords.
Note thatthe modelincludesa hiddenstate calledDUMMY whic hisusedto generate
wordswhichdonotﬁll anyslotsintheframe.
S flights that go from Boston to SanFranciscoORIGIN SHOW DUMMY FLIGHTS DEST
Figure24.7 The Pieraccini et al. (1991) HMM model of semantics for ﬁllin g slots in frame-
based dialogue systems. Each hidden state can generate a seq uence of words; such a model,
in which a single hidden state can correspond to multiple obs ervations, is technically called a
semi-HMM .

semi-HMM .
The goal of the HMM model is to compute the labeling of semanti c rolesC=
c1,c2,...,ci(Cfor‘cases’ or‘concepts’)that hasthe highestprobabili tyP(C|W)given
somewords W=w1,w2,...,wn. Asusual,we useBayesRuleasfollows:
argmax
CP(C|W) =argmax
CP(W|C)P(C)
P(W)
=argmax
=N
∏
i=2P(wi|wi−1...w1,C)P(w1|C)M
∏
i=2P(ci|ci−1...c1) (24.9)
The Pieraccini et al. (1991) model makes a simpliﬁcation tha t the concepts (the
hiddenstates) are generatedbya Markovprocess(a concept M-grammodel),and that

semi-HMM .
argmax
P(W)
=argmax
=N
∏
∏
theobservationprobabilitiesforeachstatearegenerated byastate-dependent(concept-
dependent)word N-gramwordmodel:

DRAFTSection24.2. Basic DialogueSystems 839
P(wi|wi−1,...,w1,C) =P(wi|wi−1,...,wi−N+1,ci) (24.10)
P(ci|ci−1,...,c1) =P(ci|ci−1,...,ci−M+1) (24.11)
Withthissimplifyingassumption,theﬁnal HMMmodelequati onsare:
argmax
CP(C|W) =N
∏
i=2P(wi|wi−1...wi−N+1,ci)M
∏
i=2P(ci|ci−1...ci−M+1) (24.12)
These probabilities can be trained on a labeled training cor pus, in which each
sentence is hand-labeled with the concepts/slot-names ass ociated with each string of
words. Thebestsequenceofconceptsfora sentence,andthea lignmentofconceptsto
wordsequences,canbecomputedbythestandardViterbideco dingalgorithm.
Insummary,theresultingHMMmodelisagenerativemodelwit htwocomponents.
TheP(C)component represents the choice of what meaning to express; it assigns a
prior over sequences of semantic slots, computed by a concep tN-gram.P(W|C)rep-
resents the choice of what words to use to express that meanin g; the likelihood of a
particularstring of wordsbeinggeneratedfroma givenslot . It is computedby a word

N-gram conditioned on the semantic slot. This model is very si milar to the HMM
model for named entity detection we saw in Ch. 22. Technically, HMM models like
this, in which each hiddenstate correspondto multiple outp utobservations,are called
semi-HMMs . InaclassicHMM,bycontrast,eachhiddenstatecorrespond stoasingle Semi-HMM
outputobservation.
Many other kinds of statistical models have been proposed fo r the semantic un-
derstandingcomponentofdialoguesystems. Theseincludet heHiddenUnderstanding

Model (HUM), which adds hierarchical structure to the HMM to combine the advan-
tages of the semantic grammar and semantic HMM approaches (M iller et al., 1994,
1996,2000),orthedecision-listmethodofRaynerandHocke y(2003).
24.2.3 Generationand TTS components
Thegenerationcomponentofaconversationalagentchooses theconceptstoexpressto
the user, plans out how to expressthese conceptsin words, an d assigns any necessary
prosody to the words. The TTS component then takes these word s and their prosodic

annotationsandsynthesizesa waveform,asdescribedinCh. 8.
Thegenerationtaskcanbeseparatedintotwotasks: whattosay ,andhowtosayit .
Thecontentplanner moduleaddressestheﬁrsttask,decideswhatcontenttoexpr essto
theuser,whethertoaskaquestion,presentananswer,andso on. Thecontentplanning
component of dialogue systems is generally merged with the d ialogue manager, and
we willreturntoit below.
Thelanguagegeneration moduleaddressesthe second task, choosingthe syntac-

ticstructuresandwordsneededtoexpressthemeaning. Lang uagegenerationmodules
are implemented in one of two ways. In the simplest and most co mmon method, all
or most of the words in the sentence to be uttered to the user ar e prespeciﬁed by the

DRAFT840 Chapter 24. DialogueandConversationalAgents
dialogue designer. This method is known as template-based g eneration, and the sen-
tences created by these templates are often called prompts. While most of the words Prompt
in the template are ﬁxed, templates can include some variabl es which are ﬁlled in by
thegenerator,asinthefollowing:
Whattime doyouwanttoleaveCITY-ORIG?
Will youreturntoCITY-ORIGfromCITY-DEST?
A second method for language generation relies on technique sfrom the ﬁeld nat-
ural language generation . Here the dialogue manager builds a representation of the
meaning of the utterance to be expressed, and passes this mea ning representation to
a full generator. Such generators generally have three comp onents, a sentence plan-
ner, surface realizer, and prosody assigner. A sketch of thi s architecture is shown in
Fig.24.8. SeeReiterandDale (2000)forfurtherinformatio nonnaturallanguagegen-
erationsystemsandtheiruseindialogue.
Content
PlannerSentence
PlannerSurface
Realizer
Prosody
Assigner

Content
Realizer
Prosody
Assigner
What to say How to Say itto Speech
 Synthesizer
Figure24.8 Architecture of anatural language generationsystem fora d ialogue system,after
Walker andRambow (2002).
Inthehand-designedpromptsthatarecommonincurrentsyst ems,thereareanum-
ber of important conversational and discourse constraints that must be implemented.
Like any discourse, a conversation needs to be coherent. For example, as Cohen
et al. (2004) show, the use of discourse markers and pronouns in the hand-built sys-
tempromptsmakesthedialoguein(24.14)morenaturalthant hedialoguein(24.13):
(24.13) Please saythe data.
...

...
Please saythe start time.
...
Please saythe duration.
...
Please saythe subject.
(24.14) First,tellme the date.
...
Next, I’llneed the timeit starts.
...
Thanks. <pause>Now, how longis itsupposed tolast?
...
Last of all,Ijust need a briefdescription...
Another important case of discourse coherence occurs when p articular prompts
mayneedtobesaidtotheuserrepeatedly. Inthesecases,iti sstandardindialoguesys-
temstouse taperedprompts ,promptswhichgetincrementallyshorter. Thefollowing Tapered prompt
examplefromCohenet al. (2004)showsa seriesof(hand-desi gned)taperedprompts:

DRAFTSection24.2. Basic DialogueSystems 841
(24.15)System: Now,what’stheﬁrst companytoaddtoyourwatchlist ?
Caller: Cisco
System: What’sthenextcompanyname? (Or,youcansay,“Fini shed.”)
Caller: IBM
System: Tell methenextcompanyname,orsay,“Finished.”
Caller: Intel
System: Nextone?
Caller: AmericaOnline.
System: Next?
Caller: ...
Other constraints on generation are more speciﬁc to spoken d ialogue, and refer
to facts about human memory and attentional processes. For e xample, when humans
are promptedto give a particular response, it taxes their me moryless if the suggested
response is the last thing they hear. Thus as Cohen et al. (200 4)point out, the prompt
“To hear the list again, say ‘Repeat list’” is easier for user s than “Say ‘Repeat list’ to
hearthelist again.”
Similarly,presentationoflonglistsofqueryresults(e.g .,potentialﬂights,ormovies)
cantaxusers. Thusmostdialoguesystemshavecontentplann ingrulestodealwiththis.
Inthe Mercurysystem fortravel planning(Seneff,2002),fo rexample,a rulespeciﬁes

that if there are more than three ﬂights to describe to the use r, the system will just list
theavailableairlinesanddescribeexplicitlyonlytheear liestﬂight.
24.2.4 DialogueManager
Theﬁnal componentofa dialoguesystem is the dialoguemanag er,whichcontrolsthe
architecture and structure of the dialogue. The dialogue ma nagertakes input from the
ASR/NLUcomponents,maintainssomesortofstate,interfac eswiththetaskmanager,
andpassesoutputto theNLG/TTSmodules.
WesawaverysimpledialoguemanagerinChapter2’sELIZA,wh osearchitecture

wasasimpleread-substitute-printloop. Thesystemreadin asentence,appliedaseries
of text transformationsto the sentence, and then printed it out. No state was kept; the
transformation rules were only aware of the current input se ntence. In addition to its
ability to interact with a task manager, a modern dialogue ma nager is very different
than ELIZA’s manager in both the amount of state that the mana ger keeps about the
conversation,and the ability of the managerto modelstruct uresof dialogueabovethe
levelofasingleresponse.

Fourkindsofdialoguemanagementarchitecturesare most co mmon. Thesimplest
and most commercially developed architectures, ﬁnite-sta te and frame-based, are dis-
cussed in this section. Later sections discuss the more powe rfulinformation-statedia-
loguemanagers,includinga probabilisticversionofinfor mation-statemanagersbased
onMarkovDecisionProcesses,andﬁnallythemoreclassic pl an-basedarchitectures.
Thesimplestdialoguemanagerarchitectureisaﬁnite-stat emanager. Forexample,
imagineatrivialairlinetravelsystemwhosejobwastoaskt heuserforadeparturecity,

a destination city, a time, and whether the trip was round-tr ipor not. Fig. 24.9 shows
a sample dialogue manager for such a system. The states of the FSA correspond to
questionsthatthedialoguemanageraskstheuser,andthear cscorrespondtoactionsto

DRAFT842 Chapter 24. DialogueandConversationalAgents
What city are you leaving from?
Do you want to go from
 <FROM> to <TO> on <DATE>?YesWhere are you going?
What date do you want to leave?
Is it a one-way trip?
What date do you want to return?
Do you want to go from <FROM> to <TO> 
on <DATE> returning on <RETURN>?No
No Yes
YesNo
Book the flight
Figure24.9 A simple ﬁnite-stateautomaton architecture for a dialogue manager.
takedependingonwhattheuserresponds. Thissystemcomple telycontrolstheconver-
sationwiththeuser. Itaskstheuseraseriesofquestions,i gnoring(ormisinterpreting)
anything the user says that is not a direct answer to the syste m’s question, and then
goingontothenextquestion.
Systems that control the conversation in this way are called system initiative or system initiative
single initiative systems. We say thatthespeakerthatis incontrolofthe conv ersation single initiative
has theinitiative ; in normal human-human dialogue, initiative shifts back an d forth Initiative

between the participants (Walker and Whittaker, 1990).1The limited single-initiative
ﬁnite-state dialogue manager architecture has the advanta ge that the system always
knows what question the user is answering. This means the sys tem can prepare the
speech recognition engine with a speciﬁc language model tun ed to answers for this
question. Knowingwhattheuserisgoingtobetalkingabouta lsomakesthetaskofthe
naturallanguageunderstandingengineeasier. Mostﬁnite- statesystemsalsoallow uni-

versalcommands. Universalsarecommandsthatcanbesaidanywhere inthedialogue; Universal
every dialogue state recognizes the universal commands in a ddition to the answer to
the questionthat the system just asked. Commonuniversalsi ncludehelp, whichgives
the user a (possibly state-speciﬁc) help message, start over (ormain menu ), which
returns the user to some speciﬁed main start state, and some s ort of command to cor-
rect the system’sunderstandingof the userslast statement (San-Segundoet al., 2001).

System-initiative ﬁnite-state dialogue managers with uni versals may be sufﬁcient for
very simple tasks such as entering a credit card number, or a n ame and password, on
thephone.
Pure system-initiative ﬁnite-state dialogue manager arch itectures are probably too
1Single initiative systems can also be controlled by the user , in which case they are called user initiative
systems. Pure user initiative systems are generally used fo r stateless database querying systems, where the

thephone.
user asks single questions of the system, which the system co nverts into SQL database queries, and returns
the results from somedatabase.

DRAFTSection24.2. Basic DialogueSystems 843
restricted, however, even for the relatively uncomplicate d task of a spoken dialogue
travelagentsystem. Theproblemisthatpuresystem-initia tivesystemsrequirethatthe
user answer exactly the question that the system asked. But t his can make a dialogue
awkwardandannoying. Usersoftenneedtobeabletosaysomet hingthatisnotexactly
the answer to a single question from the system. For example, in a travel planning
situation, users often want to express their travel goals wi th complex sentences that
may answer more than one question at a time, as in Communicato r example (24.16)
repeatedfromFig.24,orATISexample(24.17).
(24.16)HiI’dliketo ﬂytoSeattle Tuesdaymorning
(24.17)Iwanta ﬂightfromMilwaukeetoOrlandoonewayleavingafter ﬁvep.m. on
Wednesday.
Aﬁnitestatedialoguesystem,astypicallyimplemented,ca n’thandlethesekindsof
utterancessince it requiresthat the user answereach quest ionas it is asked. Of course

it istheoreticallypossibletocreatea ﬁnitestatearchite cturewhichhasa separatestate
for each possible subset of questionsthat the user’s statem ent couldbe answering,but
this would require a vast explosion in the number of states, m aking this a difﬁcult
architecturetoconceptualize.
Therefore, most systems avoid the pure system-initiative ﬁ nite-state approachand
use an architecturethat allows mixed initiative , in which conversationalinitiative can mixedinitiative

shiftbetweenthesystemanduserat variouspointsinthedia logue.
One common mixed initiative dialogue architecture relies o n the structure of the
frame itself to guide the dialogue. These frame-based orform-based dialogue man- Frame-based
Form-based agers asks the user questions to ﬁll slots in the frame, but al low the user to guide the
dialogue by giving information that ﬁlls other slots in the f rame. Each slot may be
associatedwitha questiontoask theuser,ofthe followingt ype:
Slot Question
ORIGINCITY “Fromwhatcityare youleaving?”

DESTINATIONCITY “Whereareyougoing?”
DEPARTURE TIME “Whenwouldyoulike toleave?”
ARRIVALTIME “Whendoyouwant toarrive?”
A frame-based dialogue manager thus needs to ask questions o f the user, ﬁlling
any slot that the user speciﬁes, until it has enough informat ionto perform a data base
query, and then return the result to the user. If the user happ ens to answer two or
three questions at a time, the system has to ﬁll in these slots and then remember not
to ask the user the associated questions for the slots. Not ev ery slot need have an

associated question, since the dialogue designer may not wa nt the user deluged with
questions. Nonetheless, the system must be able to ﬁll these slots if the user happens
to specify them. This kind of form-ﬁlling dialogue manager t hus does away with the
strict constraints that the ﬁnite-state manager imposes on the order that the user can
specifyinformation.
While some domains may be representable with a single frame, others, like the
traveldomain,seemtorequiretheabilitytodealwithmulti pleframes. Inordertohan-

dle possible user questions, we might need frameswith gener alroute information(for
questions like Which airlines ﬂy from Boston to San Francisco? ), information about

DRAFT844 Chapter 24. DialogueandConversationalAgents
airfarepractices(forquestionslike DoIhavetostayaspeciﬁcnumberofdaystogeta
decentairfare? ) oraboutcarorhotelreservations. Sinceusersmayswitchf romframe
to frame, the system must be able to disambiguate which slot o f which frame a given
inputis supposedto ﬁll, andthenswitchdialoguecontrolto that frame.
Becauseofthisneedtodynamicallyswitch control,frame-b asedsystemsareoften
implemented as production rule systems. Different types of inputs cause different
productionsto ﬁre, each of which can ﬂexibly ﬁll in differen t frames. The production
rulescanthenswitchcontrolbasedonfactorssuchastheuse r’sinputandsomesimple
dialoguehistorylike the last questionthat the system aske d. TheMercuryﬂight reser-
vationsystem(SeneffandPolifroni,2000;Seneff,2002)us esalarge‘dialoguecontrol
table’tostore200-350rules,coveringrequestforhelp,ru lestodetermineiftheuseris
referringtoa ﬂightinalist (”I’lltakethatninea.m. ﬂight ”),andrulestodecidewhich
ﬂightsto describetothe userﬁrst.

Nowthat we’veseen the frame-basedarchitecture,let’s ret urnto our discussionof
conversationalinitiative. It’spossibleinthesameagent toallowsystem-initiative,user-
initiative,andmixed-initiativeinteractions. We saidea rlierthatinitiativereferstowho
has control of the conversation at any point. The phrase mixed initiative is generally
usedintwo ways. It canmeanthatthesystem ortheusercoulda rbitrarilytake orgive
uptheinitiativeinvariousways(WalkerandWhittaker,199 0;Chu-CarrollandBrown,

1997). Thiskindof mixedinitiativeis difﬁcultto achievei n currentdialoguesystems.
Inform-baseddialoguesystem,thetermmixedinitiativeis usedforamorelimitedkind
ofshift,operationalizedbasedonacombinationofpromptt ype(openversusdirective)
andthetypeofgrammarusedintheASR.An openprompt isoneinwhichthesystem Open prompt
gives the user very few constraints, allowing the user to res pondhoweverthey please,
asin:
HowmayI helpyou?
Adirectiveprompt isonewhichexplicitlyinstructstheuserhowtorespond: Directiveprompt

asin:
Sayyesifyouacceptthecall; otherwise,say no.
Arestrictive grammar(Sec.24.2.1)isalanguagemodelwhichstronglycon strainsthe
ASRsystem,onlyrecognizingproperresponsesto agivenpro mpt.
We can combine these as in Fig. 24.10 to deﬁne initiative as us ed in form-based
dialoguesystems,followingSinghet al.(2002)andothers.
PromptType
Grammar Open Directive
Restrictive Doesn’tmakesense SystemInitiative
Non-Restrictive User Initiative MixedInitiative
Figure24.10 Operational deﬁnitionof initiative,followingSingh etal .(2002).

Here a system initiative interaction uses a directive promp tand a restrictive gram-
mar; the user is told how to respond, and the ASR system is cons trained to only rec-
ognizetheresponsesthat are promptedfor. Inuser initiati ve,the useris givenan open
prompt,andthegrammarmustrecognizeanykindofresponse, sincetheusercouldsay
anything. Finally, in a mixed initiative interaction, the s ystem gives the user a direc-

DRAFTSection24.2. Basic DialogueSystems 845
tive prompt with particular suggestions for response, but t he non-restrictive grammar
allowsthe usertorespondoutsidethescopeoftheprompt.
Deﬁninginitiativeasapropertyofthepromptandgrammarty peinthiswayallows
systemstodynamicallychangetheirinitiativetypefordif ferentusersandinteractions.
Novice users, or users with high speech recognition error, m ight be better served by
moresysteminitiative. Expertusers,orthosewhohappento speakmorerecognizably,
might do well with mixed or user initiative interactions. We will see in Sec. 24.6 how
machinelearningtechniquescanbeusedtochooseinitiativ e.
24.2.5 DialogueManagerErrorHandling: Conﬁrmation/Reje ction
In a dialogue system, mishearings are a particularly import ant class of problems, be-
cause speech recognition has such a high error rate. It is the refore important for di-
alogue systems to make sure that they have achieved the corre ct interpretation of the

user’s input. This is generallydone by two methods: conﬁrming understandingswith
theuser,and rejecting utterancesthat thesystemislikelyto havemisunderstood.
Various strategies can be employed for conﬁrmation with the user. A system us-
ing theexplicit conﬁrmation strategy asks the user a direct question to conﬁrm theirExplicit
conﬁrmation
understanding. Here are two examples of explicit conﬁrmati ons from travel planning
systems. The(boldface)conﬁrmationquestionsarebothyes -noquestions,oneusinga

single sentence, the other presentinga declarativesenten cefollowedby a tag question
(ashortquestionphraselike“right?” or“isn’tit?”):
S:Whichcitydoyouwantto leavefrom?
U:Baltimore.
S:DoyouwanttoleavefromBaltimore?
U:Yes.
U:I’d like to ﬂy from Denver Colorado to New York City on Septemb er
twentyﬁrst in themorningonUnitedAirlines
S:Let’ssee then. I haveyou goingfromDenverColoradotoNew Yo rk
onSeptembertwentyﬁrst. Is thatcorrect?
U:Yes
Asystemusingthe implicitconﬁrmation strategy,ratherthanaskingadirectques-Implicit

U:Yes.
U:Yes
tion,usesthe demonstration ordisplaygroundingstrategiesdescribedabove,conﬁrm-
ingtotheuserbyrepeatingbackwhat thesystemunderstoodt heuserto havesaid:
U:I wanttotravelto Berlin
S:When do youwanttotraveltoBerlin?
U2:Hi I’dlike toﬂy toSeattle TuesdayMorning
A3:Traveling toSeattleon Tuesday, August eleventh in the morn ing.
Yourfullname?
Explicit and implicit conﬁrmation have complementarystre ngths. Explicit conﬁr-
mation makes it easier for users to correct the system’s misr ecognitionssince the user

canjustanswer‘no’totheconﬁrmationquestion. Butexplic itconﬁrmationisawkward
andincreasesthe lengthoftheconversation(DanieliandGe rbino,1995;Walkeret al.,

DRAFT846 Chapter 24. DialogueandConversationalAgents
1998). Theexplicitconﬁrmationdialoguefragmentsaboves oundnon-naturalanddef-
initelynon-human;implicitconﬁrmationismuchmoreconve rsationallynatural.
Conﬁrmationis just one kind of conversationalaction that a system has to express
lack of understanding. Another option is rejection . An ASR system rejects an utter- Rejection
ance by giving the user a prompt like I’m sorry, I didn’t understand that . Sometimes
utterancesarerejectedmultipletimes. Thismightmeantha ttheuserisusinglanguage
that the system is unable to follow. Thus when an utterance is rejected, systems often
follow a strategy of progressive prompting orescalating detail (Yankelovich et al.,Progressiveprompting
1995;WeinschenkandBarker,2000)asinthisexamplefromCo henet al.(2004):
System: Whenwouldyouliketoleave?
Caller: Well,um,IneedtobeinNewYorkintimefortheﬁrstWorldSeri esgame.
System: <reject>. Sorry,Ididn’tgetthat. Pleasesaythemonthanddayyou’dl ike
toleave.
Caller: IwannagoonOctoberﬁfteenth.

toleave.
In this example, instead of just repeating ‘When would you li ke to leave?’, the re-
jectionpromptgivesthecaller moreguidanceabouthowtofo rmulatean utterancethe
systemwillunderstand. These you-can-say helpmessagesareimportantinhelpingim-
provesystemsunderstandingperformance(BohusandRudnic ky,2005). If thecaller’s
utterance gets rejected yet again, the prompt can reﬂect thi s (‘Istilldidn’t get that’),
andgivethecallerevenmoreguidance.
An alternativestrategyfor errorhandlingis rapid reprompting ,in whichthe sys-Rapidreprompting

tem rejects an utterance just by saying “I’m sorry?” or “What was that?”. Only if the
caller’s utterance is rejected a second time does the system start applying progressive
prompting. Cohen et al. (2004) summarizes experiments show ing that users greatly
preferrapidrepromptingasa ﬁrst-levelerrorprompt.
24.3 VoiceXML
VoiceXML istheVoiceExtensibleMarkupLanguage,anXML-baseddialo guedesign VoiceXML
language released by the W3C, and the most commonly used of th e various speech
markuplanguages(suchasSALT).ThegoalofVoiceXML(or vxml)istocreatesimple vxml

audio dialogues of the type we have been describing, making u se of ASR and TTS,
and dealing with very simple mixed-initiative in a frame-ba sed architecture. While
VoiceXML is more common in the commercial rather than academ ic setting, it is a
goodwayforthestudentto geta hands-ongraspofdialoguesy stemdesignissues.
AVoiceXMLdocumentcontainsasetofdialogues,eachofwhic hcanbea formor
amenu. Wewilllimitourselvestointroducingforms;see http://www.voicexml.
org/for more information on VoiceXML in general. The VoiceXML do cument in

Fig. 24.11 deﬁnes a form with a single ﬁeld named ‘transportt ype’. The ﬁeld has an
attached prompt, Please choose airline, hotel, or rental car , which can be passed to
theTTSsystem. Italsohasagrammar(languagemodel)whichi spassedtothespeech
recognition engine to specify which words the recognizer is allowed to recognize. In

DRAFTSection24.3. VoiceXML 847
<form>
<field name="transporttype">
<prompt>
Please choose airline, hotel, or rental car.
</prompt>
<grammar type="application/x=nuance-gsl">
[airline hotel "rental car"]
</grammar>
</field>
<block>
<prompt>
You have chosen <value expr="transporttype">.
</prompt>
</block>
</form>
Figure24.11 A minimal VoiceXML script for a form with a single ﬁeld. User i s prompted,
and the response isthen repeatedback.
the example in Fig. 24.11, the grammar consists of a disjunct ion of the three words
airline,hotel,andrentalcar .
A<form> generally consists of a sequence of <field> s, together with a few
other commands. Each ﬁeld has a name ( transporttype is the name of the ﬁeld
in Fig. 24.11)which is also the name of the variablewhere the user’s responsewill be
stored. Thepromptassociatedwiththeﬁeldisspeciﬁedviat he<prompt> command.
The grammar associated with the ﬁeld is speciﬁed via the <grammar> command.
VoiceXML supports various ways of specifying a grammar, inc luding XML Speech

Grammar, ABNF, and commercial standards, like Nuance GSL. W e will be using the
NuanceGSL formatinthe followingexamples.
The VoiceXML interpreter walks through a form in document or der, repeatedly
selecting each item in the form. If there are multiple ﬁelds, the interpreter will visit
eachonein order. Theinterpretationordercanbechangedin variousways, aswe will
see later. The examplein Fig. 24.12showsa form with three ﬁe lds, for specifyingthe
origin,destination,andﬂightdate ofan airlineﬂight.

The prologue of the example shows two global defaults for err or handling. If
the user doesn’t answer after a prompt (i.e., silence exceed s a timeout threshold), the
VoiceXML interpreterwill play the <noinput> prompt. If the user says something,
but it doesn’t match the grammar for that ﬁeld, the VoiceXML i nterpreter will play
the<nomatch> prompt. After any failure of this type, it is normalto re-ask the user
the question that failed to get a response. Since these routi nes can be called from any

ﬁeld, and hence the exact prompt will be different every time , VoiceXML provides a
<reprompt\> command,whichwillrepeatthepromptforwhateverﬁeldcaus edthe
error.
The three ﬁelds of this form show another feature of VoiceXML , the<filled>
tag. The <filled> tag for a ﬁeld is executed by the interpreter as soon as the ﬁel d
hasbeenﬁlled bythe user. Here, thisfeatureis used to givet he user a conﬁrmationof
theirinput.
The last ﬁeld, departdate , shows another feature of VoiceXML, the typeat-

error.
tribute. VoiceXML2.0speciﬁessevenbuilt-ingrammartype s,boolean ,currency ,
date,digits,number,phone, andtime. Since the type of this ﬁeld is date,
a data-speciﬁc language model (grammar) will be automatica lly passed to the speech

DRAFT848 Chapter 24. DialogueandConversationalAgents
<noinput>
I’m sorry, I didn’t hear you. <reprompt/>
</noinput>
<nomatch>
I’m sorry, I didn’t understand that. <reprompt/>
</nomatch>
<form>
<block> Welcome to the air travel consultant. </block>
<field name="origin">
<prompt> Which city do you want to leave from? </prompt>
[(san francisco) denver (new york) barcelona]
<filled>
<prompt> OK, from <value expr="origin"> </prompt>
</filled>
</field>
<field name="destination">
<prompt> And which city do you want to go to? </prompt>
<filled>
<prompt> OK, to <value expr="destination"> </prompt>
</filled>
</field>
<field name="departdate" type="date">
<prompt> And what date do you want to leave? </prompt>
<filled>
<prompt> OK, on <value expr="departdate"> </prompt>
</filled>
</field>
<block>
<prompt> OK, I have you are departing from <value expr="orig in">

<filled>
</filled>
</field>
<filled>
</filled>
</field>
<block>
to <value expr="destination"> on <value expr="departdate ">
</prompt>
send the info to book a flight...
</block>
</form>
Figure24.12 A VoiceXML script for a form with 3 ﬁelds, which conﬁrms each ﬁ eld and
handles the noinput andnomatch situations.
recognizer,sowe don’tneedtospecifythegrammarhereexpl icitly.
Fig.24.13givesaﬁnalexamplewhichshowsmixedinitiative . Inamixedinitiative
dialogue, users can choose not to answer the question that wa s asked by the system.

</prompt>
</block>
</form>
For example, they might answer a different question, or use a long sentence to ﬁll in
multiple slots at once. This means that the VoiceXML interpr eter can no longer just
evaluate each ﬁeld of the form in order; it needs to skip ﬁelds whose values are set.
This is done by a guard condition , a test that keeps a ﬁeld from being visited. The
default guard condition for a ﬁeld tests to see if the ﬁeld’s f orm item variable has a
value,andifso theﬁeld isnotinterpreted.
Fig. 24.13also showsa muchmorecomplexuse of a grammar. Thi s grammaris a

CFG grammar with two rewrite rules, named Flight andCity. The Nuance GSL
grammar formalism uses parentheses () to mean concatenatio n and square brackets []
to mean disjunction. Thusa rule like (24.18)means that Wantsentence can be ex-
pandedas i want to fly ori want to go ,andAirports can beexpanded
assan francisco ordenver.
(24.18) Wantsentence (i want to [fly go])

DRAFTSection24.3. VoiceXML 849
<noinput> I’m sorry, I didn’t hear you. <reprompt/> </noinp ut>
<nomatch> I’m sorry, I didn’t understand that. <reprompt/> </nomatch>
<form>
<![ CDATA[
Flight ( ?[
(i [wanna (want to)] [fly go])
(i’d like to [fly go])
([(i wanna)(i’d like a)] flight)
]
[
( [from leaving departing] City:x) {<origin $x>}
( [(?going to)(arriving in)] City:x) {<destination $x>}
( [from leaving departing] City:x
[(?going to)(arriving in)] City:y) {<origin $x> <destinat ion $y>}
]
?please
)
City [ [(san francisco) (s f o)] {return( "san francisco, cal ifornia")}
[(denver) (d e n)] {return( "denver, colorado")}
[(seattle) (s t x)] {return( "seattle, washington")}
]
]]> </grammar>
<initial name="init">
<prompt> Welcome to the consultant. What are your travel pla ns? </prompt>
</initial>
<filled>
</filled>
</field>

]
?please
)
]
<filled>
</filled>
</field>
<filled>
</filled>
</field>
<block>
to <value expr="destination">. </prompt>
</block>
</form>
Figure24.13 A mixed initiative VoiceXML dialogue. The grammar allows se ntences which
specify the origin or destination cities or both. User can re spond to the initial prompt by speci-

<filled>
</filled>
</field>
<block>
</block>
</form>
fyingorigin city,destination city,or both.
Airports [(san francisco) denver]
Grammarrulescanrefertoothergrammarrulesrecursively, andsointhegrammar
inFig. 24.13we seethe grammarfor Flight referringtotherulefor City.
VoiceXML grammars take the form of CFG grammars with optiona l semantic
attachments. The semantic attachments are generally eithe r a text string (such as
"denver, colorado" ) or a slot and a ﬁller. We can see an example of the for-
merinthe semanticattachmentsforthe Cityrule(the return statementsat theend

ofeach line),whichpassupthe city andstate name. The seman ticattachmentsforthe
Flight ruleshowsthelattercase, wheretheslot ( <origin> or<destination>
orboth)isﬁlled withthevaluepassedupin thevariable xfromthe Cityrule.
BecauseFig.24.13isamixedinitiativegrammar,thegramma rhastobeapplicable

DRAFT850 Chapter 24. DialogueandConversationalAgents
to any of the ﬁelds. This is done by making the expansionfor Flight a disjunction;
notethatit allowsthe usertospecifyonlytheorigincity,t hedestinationcity,orboth.
24.4 DialogueSystemDesignandEvaluation
24.4.1 DesigningDialogueSystems
Howdoesadialoguesystemdeveloperchoosedialoguestrate gies,architectures,prompts,
error messages, and so on? This process is often called VUI(Voice User Interface ) VUI
design. The user-centereddesign principlesofGouldandLewis(1985)are:
1. Study the user and task: Understand the potential users and the nature of the
task,viainterviewswithusersandinvestigationofsimila rsystems,andstudyofrelated
human-humandialogues.
2. Build simulations and prototypes: InWizard-of-Oz systems (WOZ) or Wizard-of-Oz
PNAMBIC(PayNoAttentiontotheManBehIndtheCurtain)syst ems,theusersinter-
actwithwhattheythinkisasoftwaresystem,butisinfactah umanoperator(“wizard”)
behind some disguising interface software (e.g. Gould et al ., 1983; Good et al., 1984;

Fraser and Gilbert, 1991).2A WOZ system can be used to test out an architecturebe-
foreimplementation;onlytheinterfacesoftwareanddatab asesneedtobeinplace. The
wizard’slinguisticoutputcanbedisguisedbyatext-to-sp eechsystem,orviatext-only
interactions. It is difﬁcult for the wizard to exactly simul ate the errors, limitations, or
time constraintsof areal system; resultsof WOZ studiesare thussomewhatidealized,
butstill canprovideausefulﬁrst ideaofthedomainissues.

3. Iteratively test the design on users: An iterative design cycle with embed-
ded user testing is essential in system design (Nielsen, 199 2; Cole et al., 1994, 1997;
Yankelovich et al., 1995; Landauer, 1995). For example Stif elman et al. (1993) built
a system that originally required the user to press a key to in terrupt the system. They
found in user testing that users instead tried to interrupt t he system ( barge-in), sug- Barge-in
gestinga redesignofthe systemto recognizeoverlappedspe ech. Theiterativemethod

is also important for designing prompts which cause the user to respond in normative
ways, such as the use in particularsituationsof constraine dforms(Oviatt et al., 1993)
ordirectiveprompts ratherthanopenprompts(Kamm,1994;Coleetal.,1993). Sim -
ulations can also be used at this stage; user simulations tha t interact with a dialogue
systemcanhelptest theinterfaceforbrittlenessorerrors (Chung,2004).
SeeCohenetal.(2004),Harris(2005)formoreonconversati onalinterfacedesign.
24.4.2 DialogueSystem Evaluation

We said above that user testing and evaluation is crucial in d ialogue system design.
Computing a user satisfaction rating can be done by having users interact with a
dialogue system to perform a task, and then having them compl ete a questionnaire
(Shriberget al., 1992; Polifroniet al., 1992;Stifelman et al., 1993;Yankelovichet al.,
2The name comes from the children’s book The Wizard of Oz (Baum, 1900), in which the Wizard turned
outto bejust asimulation controlled by aman behind acurtai n.

DRAFTSection24.4. DialogueSystemDesignandEvaluation 851
TTSPerformance Wasthe system easytounderstand ?
ASR Performance Didthe system understand what you said?
Task Ease Wasit easytoﬁndthe message/ﬂight/trainyou wanted?
Interaction Pace Wasthe pace of interactionwiththe system appropriate?
User Expertise Didyouknow what you could sayat eachpoint?
System Response How oftenwas the system sluggishand slow toreplytoyou?
Expected Behavior Didthe system workthe way you expected itto?
FutureUse Doyou thinkyou’d use the system inthefuture?
Figure24.14 User satisfactionsurvey, adapted from Walker etal. (2001) .
1995; M¨ oller, 2002). For example Fig. 24.4.2 shows multipl e-choicequestions of the
sort used by Walker et al. (2001); responses are mapped into t he range of 1 to 5, and
thenaveragedoverall questionsto getatotal usersatisfac tionrating.
It is often economically infeasible to run complete user sat isfaction studies after
everychangeinasystem. Forthisreasonitisoftenusefulto haveperformanceevalua-

tionheuristicswhichcorrelatewellwithhumansatisfacti on. Anumberofsuchfactors
andheuristicshavebeenstudied. One methodthat hasbeenus ed to classify these fac-
torsis based onthe idea that an optimal dialoguesystem is on e whichallowsa user to
accomplishtheirgoals(maximizingtasksuccess)withthel eastproblems(minimizing
costs). Thenwecanstudymetricswhichcorrelatewiththese two criteria.
Task CompletionSuccess: Tasksuccesscanbemeasuredbyevaluatingthecorrect-
ness ofthe total solution. For a frame-basedarchitecture, thismight be the percentage

ofslotsthatwereﬁlledwiththecorrectvalues,ortheperce ntageofsubtasksthatwere
completed (Polifroni et al., 1992). Since different dialog ue systems may be applied
to different tasks, it is hard to compare them on this metric, so Walker et al. (1997)
suggestedusingtheKappacoefﬁcient, κ,tocomputeacompletionscorewhichisnor-
malizedforchanceagreementandbetterenablescross-syst emcomparison.
Efﬁciency Cost: Efﬁciency costs are measures of the system’s efﬁciency at he lping
users. This can be measured via the total elapsed time for the dialogue in seconds,

the number of total turns or of system turns, or the total numb er of queries (Polifroni
etal.,1992). Othermetricsincludethenumberofsystemnon -responses,andthe“turn
correction ratio”: the number of system or user turns that we re used solely to correct
errors, divided by the total number of turns (Danieli and Ger bino, 1995; Hirschman
andPao,1993).
QualityCost: Qualitycostmeasuresotheraspectsoftheinteractionthat affectusers’
perception of the system. One such measure is the number of ti mes the ASR system

failed to return any sentence, or the number of ASR rejection prompts. Similar met-
rics include the number of times the user had to barge-in (interrupt the system), or Barge-in
the numberof time-out promptsplayed when the user didn’t re spondquickly enough.
Other quality metrics focus on how well the system understoo d and responded to the
user. This can include the inappropriateness (verbose or am biguous) of the system’s

questions, answers, and error messages (Zue et al., 1989), o r the correctness of each
question, answer, or error message (Zue et al., 1989; Polifr oni et al., 1992). A very
importantqualitycostis conceptaccuracy orconcepterrorrate ,whichmeasuresthe Conceptaccuracy
percentageofsemanticconceptsthattheNLUcomponentretu rnscorrectly. Forframe-

DRAFT852 Chapter 24. DialogueandConversationalAgents
based architectures this can be measured by counting the per centage of slots that are
ﬁlled withthecorrectmeaning. Forexampleif thesentence‘ Iwant toarrivein Austin
at5:00’ismisrecognizedtohavethesemantics”DEST-CITY: Boston,Time: 5:00”the
conceptaccuracywouldbe50%(oneoftwo slotsarewrong).
MAXIMIZE USER SATISFACTION
MAXIMIZE TASK SUCCESS MINIMIZE COSTS
EFFICIENCY MEASURES
 QUALITY MEASURES
Figure24.15 PARADISE’s structure of objectives for spoken dialogue per formance. After
Walker etal. (1997).
How should these success and cost metrics be combined and wei ghted? One ap-
proach is the PARADISE algorithm (PARAdigm for DIalogue Sys tem Evaluation),
which applies multiple regression to this problem. The algo rithm ﬁrst assigns each
dialogue a user satisfaction rating using questionnaires l ike the one in Fig. 24.4.2. A
set of cost and success factors like those above is then treat ed as a set of independent

factors; multiple regression is used to train a weight for ea ch factor, measuringits im-
portance in accounting for user satisfaction. Fig. 24.15 sh ows the particular model of
performance that the PARADISE experiments have assumed. Ea ch box is related to
a set of factors that we summarized on the previous page. The r esulting metric can
be used to compare quite differentdialogue strategies; eva luationsusing methods like
PARADISEhavesuggestedthattaskcompletionandconceptac curacymaybethemost
importantpredictorsofusersatisfaction; seeWalker et al .(1997,2001,2002).

Awidevarietyofotherevaluationmetricsandtaxonomiesha vebeenproposedfor
describing the quality of spoken dialogue systems (Fraser, 1992; M¨ oller, 2002, 2004;
DelgadoandAraki,2005,interalia).
24.5 Information-state& DialogueActs
The basic frame-based dialogue systems we have introduced s o far are only capable
of limited domain-speciﬁc conversations. This is because t he semantic interpretation
and generation processes in frame-based dialogue systems a re based only on what is

needed to ﬁll slots. In order to be usable for more than just fo rm-ﬁlling applications,
a conversational agent needs to be able to do things like deci de when the user has
asked a question, made a proposal, or rejected a suggestion, and needs to be able to
grounda users utterance, ask clariﬁcation questions, and s uggest plans. This suggests
thataconversationalagentneedssophisticatedmodelsofi nterpretationandgeneration
in terms of speech acts and grounding,and a more sophisticat ed representation of the

dialoguecontextthanjust a list ofslots.

DRAFTSection24.5. Information-state& DialogueActs 853
In this section we sketch a more advanced architecture for di alogue management
whichallowsforthese moresophisticatedcomponents. This modelisgenerallycalled
theinformation-state architecture(TraumandLarsson,2003,2000),althoughwew ill Information-state
use the term loosely to includearchitecturessuch as Allen e t al. (2001). A probabilis-
tic architecture which can be seen as an extension of the info rmation-state approach,
theMarkov decision process model, will be described in the next section. The term
information-state architecture is really a cover term for a number of quite different
effortstowardmoresophisticatedagents;we’ll assumeher ea structureconsistingof5
components:
•theinformationstate(the‘discoursecontext’or‘mentalm odel’)
•adialogueactinterpreter(or“interpretationengine”)
•adialogueactgenerator(or“generationengine”)
•a set of update rules, which update the information state as d ialogue acts are

interpreted,andwhichincluderulesto generatedialoguea cts.
•acontrolstructureto select whichupdaterulesto apply
The term information state is intended to be very abstract, and might include
things like the discourse context and the common groundof th e two speakers, the be-
liefsor intentionsofthe speakers,user models,andso on. C rucially,informationstate
is intendedto be a more complexnotion than the static states in a ﬁnite-state dialogue
manager;thecurrentstateincludesthevaluesofmanyvaria bles,thediscoursecontext,

andotherelementsthatarenoteasily modeledbya state-num berina ﬁnitenetwork.
Dialogueactsareanextensionofspeechactswhichintegrat eideasfromgrounding
theory,andwillbedeﬁnedmorefullyinthenextsubsection. Theinterpretationengine
takes speech as input and ﬁgures out sentential semantics an d an appropriatedialogue
act. The dialogue act generator takes dialogue acts and sent ential semantics as input
andproducestext/speechasoutput.
Finally,theupdaterulesmodifytheinformationstatewith theinformationfromthe

dialogue acts. These update rules are a generalization of th e production rules used in
frame-baseddialoguesystemsdescribedabove(SeneffandP olifroni,2000, interalia).
A subset of update rules, called selection rules , are used to generate dialogue acts.
For example, an update rule might say that when the interpret ation engine recognizes
an assertion, that the information state should be updated w ith the information in the
assertion, andan obligationto performa groundingact need sto be addedto the infor-

mation state. When a question is recognized,an update rule m ight specify the need to
answer the question. We can refer to the combination of the up date rules and control
structureasthe BehavioralAgent (Allenet al.,2001),assuggestedinFig.24.16.
While the intuition of the information-state model is quite simple, the details can
be quite complex. The information state might involve rich d iscourse models such as
Discourse Representation Theory or sophisticated models o f the user’s belief, desire,

and intention (which we will return to in Sec. 24.7). Instead of describinga particular
implementation here, we will focus in the next few sections o n the dialogue act inter-
pretationandgenerationengines,anda probabilisticinfo rmation-statearchitecturevia
Markovdecisionprocesses.

DRAFT854 Chapter 24. DialogueandConversationalAgents
Natural Language Understanding
Dialogue Act InterpreterNatural Language Generation
Behavioral Agent
   -update rules
   -controlDialogue Act GeneratorSpeech
Information State
   -discourse context
   -beliefs
   -goals
   -user model
   -task contextSpeech
Figure24.16 A version of the information-state approach todialogue arc hitecture.
24.5.1 DialogueActs
As we implied above,the speech acts as originallydeﬁned by A ustin don’tmodelkey
features of conversation such as grounding, contributions , adjacency pairs and so on.
In order to capture these conversational phenomena, we use a n extension of speech
acts called dialogue acts (Bunt, 1994) (or dialogue moves orconversational moves dialogue act
Conversationalmove(Power,1979;Carletta et al., 1997b). A dialogueact extend sspeech actswith internal
structure related speciﬁcally to these other conversation al functions (Allen and Core,
1997;Bunt,2000).

A wide variety of dialogue act tagsets have been proposed. Fi g. 24.17 shows a
very domain-speciﬁctagset for the Verbmobil two-partysch edulingdomain, in which
speakerswereaskedtoplanameetingatsomefuturedate. Not icethatithasmanyvery
domain-speciﬁctags,suchas S UGGEST,usedforwhensomeoneproposesaparticular
date to meet, and A CCEPTand REJECT, used to accept or reject a proposal for a date.
Thus it has elements both from the presentation and acceptan ce phases of the Clark
contributionsdiscussedonpage830.

Thereare a numberofmore generalanddomain-independentdi alogueact tagsets.
In the DAMSL (Dialogue Act Markup in Several Layers) archite cture inspired by the
work of Clark and Schaefer (1989), Allwood et al. (1992), and (Allwood, 1995) each
utteranceis taggedfortwo typesoffunctions, forwardlooking functions like speech
act functions,and backward looking functions,like groundingand answering,which
‘look back’ to the interlocutor’s previous utterance (Alle n and Core, 1997; Walker

et al.,1996;Carlettaet al.,1997a;Coreet al.,1999).
Traum and Hinkelman (1992) proposed that the core speech act s and grounding
acts that constitute dialogue acts could ﬁt into an even rich er hierarchy of conversa-
tionacts. Fig.24.5.1showsthefourlevelsofacttypestheypropose, withthetwomid- Conversationact
dle levels corresponding to DAMSL dialogue acts (grounding and core speech acts).
The two new levels include turn-taking acts and a type of cohe rence relations called
argumentation relations.

Theactsformahierarchy,inthatperformanceofanactatahi gherlevel(forexam-
ple a core speechact) entailsperformanceof a lower level ac t (takinga turn). We will
see the use of conversationalacts in generationlater on in t his section, and will return

DRAFTSection24.5. Information-state& DialogueActs 855
THANK Thanks
GREET HelloDan
INTRODUCE It’s me again
BYE Allright bye
REQUEST-COMMENT How does that look?
SUGGEST from thirteenththrough seventeenth June
REJECT NoFriday I’m booked all day
ACCEPT Saturday sounds ﬁne,
REQUEST-SUGGEST What isagood day of the week for you?
INIT I wanted tomake anappointment withyou
GIVEREASON Because I have meetings allafternoon
FEEDBACK Okay
DELIBERATE Let mecheck mycalendar here
CONFIRM Okay, that would be wonderful
CLARIFY Okay, do youmean Tuesday the23rd?
DIGRESS [we could meet for lunch] and eat lots ofice cream
MOTIVATE We should gotovisitour subsidiary inMunich
GARBAGE Oops, I-
Figure24.17 The 18 high-level dialogue acts used in Verbmobil-1, abstra cted over a total of
43more speciﬁc dialogue acts. Examples are from Jekat et al. (1995).
Acttype SampleActs
turn-taking take-turn, keep-turn, release-turn, assign- turn
grounding acknowledge, repair,continue

core speech acts inform, wh-question, accept, request, off er
argumentation elaborate, summarize, question-answer, cl arify
Figure24.18 Conversation acttypes, from Traum andHinkelman (1992).
tothe questionofcoherenceanddialoguestructurein Sec.2 4.7.
24.5.2 Interpreting DialogueActs
How can we do dialogueact interpretation,decidingwhether a giveninput is a QUES-
TION, aSTATEMENT , aSUGGEST (directive), or an ACKNOWLEDGEMENT ? Perhaps
we canjust relyonsurfacesyntax? We saw inCh. 12thatyes-no -questionsin English

haveaux-inversion (the auxiliary verb precedes the subject) statements have d eclara-
tivesyntax(noaux-inversion),andcommandshavenosyntac ticsubject:
(24.19) YES-NO-QUESTION Will breakfastbeservedonUSAir 1557?
STATEMENT Idon’tcareaboutlunch
COMMAND ShowmeﬂightsfromMilwaukeeto Orlando.
Alas,asisclearfromAbbottandCostello’sfamous Who’sonFirst routineatthebe-
ginningof thechapter,the mappingfromsurfaceformto illo cutionaryact is complex.
For example, the following ATIS utterance looks like a YES-NO-QUESTION meaning

somethinglike Are youcapableofgivingmealist of...? :
(24.20)Canyougivemea list oftheﬂightsfromAtlantatoBoston?
Infact,however,this personwasnot interestedin whethert hesystem was capable
of giving a list; this utterance was a polite form of a REQUEST , meaning something

DRAFT856 Chapter 24. DialogueandConversationalAgents
morelike Pleasegivemealist of... . Thuswhatlooksonthesurfacelikea QUESTION
canreallybea REQUEST .
Similarly,whatlooksonthesurfacelikea STATEMENT canreallybea QUESTION .
The verycommon CHECKquestion(Carletta et al., 1997b; Labovand Fanshel, 1977),
is used to ask an interlocutor to conﬁrm something that she ha s privileged knowledge
about.CHECKShavedeclarativesurfaceform:
AOPEN-OPTION I was wanting to make some arrangements for a trip that I’m goi ng
tobe taking uhtoLAuh beginning of the weekafter next.
BHOLD OK uh let me pull up your proﬁle and I’ll be right with you here.
[pause]
BCHECK Andyou saidyou wantedtotravel next week?
AACCEPT Uhyes.
Utterances that use a surface statement to ask a question, or a surface question to
issue arequest,arecalled indirectspeech acts .Indirectspeechact
Inordertoresolvethese dialogueactambiguitieswecanmod eldialogueact inter-
pretation as a supervisedclassiﬁcation task, with dialogu eact labels as hidden classes

[pause]
tobedetected. We trainclassiﬁersonacorpusinwhicheachu tteranceishand-labeled
for dialogue acts. The features used for dialogue act interp retation derive from the
conversationalcontext and from the act’s microgrammar (Goodwin, 1996) (its char- microgrammar
acteristiclexical,grammatical,andprosodicproperties ):
1.WordsandCollocations: Pleaseorwouldyou isagoodcuefora REQUEST ,are
youforYES-NO-QUESTIONS ,detectedvia dialogue-speciﬁc N-gramgrammars.

2.Prosody: Risingpitchisagoodcuefora YES-NO-QUESTION ,whiledeclarative
utterances (like STATEMENTS ) haveﬁnal lowering : a drop in F0 at the end of Final lowering
theutterance. Loudnessorstresscanhelpdistinguishthe yeahthatisan AGREE-
MENTfromtheyeahthatisa BACKCHANNEL . Wecanextractacousticcorrelates
ofprosodicfeatureslikeF0,duration,andenergy.
3.ConversationalStructure: Ayeahfollowingaproposalisprobablyan AGREE-
MENT;ayeahafteran INFORMislikelya BACKCHANNEL . Drawingontheidea
ofadjacencypairs(Schegloff,1968;Sacksetal.,1974),we canmodelconversa-

tionalstructureasabigramofdialogueacts.
Formallyourgoalisto ﬁndthe dialogueact d∗that hasthehighestposteriorprob-
abilityP(d|o)giventheobservationofa sentence,
d∗=argmax
dP(d|o)
=argmax
dP(d)P(o|d)
P(o)
=argmax
dP(d)P(o|d) (24.21)
Makingsome simplifying assumptions(that the prosodyof th e sentence fand the
wordsequence Wareindependent,andthatthepriorofa dialogueactcanbemo deled
by the conditional given the previous dialogue act) we can es timate the observation
likelihoodfora dialogueact dasin(24.22):

DRAFTSection24.5. Information-state& DialogueActs 857
P(o|d) =P(f|d)P(W|d) (24.22)
d∗=argmax
dP(d|dt−1)P(f|d)P(W|d) (24.23)
where
P(W|d) =N
∏
i=2P(wi|wi−1...wi−N+1,d) (24.24)
Trainingtheprosodicpredictortocompute P(f|d)hasoftenbeendonewithadeci-
siontree. Shriberget al.(1998),forexample,builta CART t ree todistinguishthefour
dialogue acts STATEMENT (S),YES-NO QUESTION (QY),DECLARATIVE -QUESTION
likeCHECK(QD) and WH-QUESTION (QW) based on acoustic features such as the
slope of F0 at the end of the utterance, the average energy at d ifferent places in the
utterance, and various normalized duration measures. Fig. 24.19 shows the decision
treewhichgivestheposteriorprobability P(d|f)ofadialogueact dtypegivenasetof
acousticfeatures f. Notethatthe differencebetweenS andQY towardthe rightof the
tree is based on the feature normf0diff(normalizeddifferencebetween mean F0
of endand penultimateregions),while the differencebetwe enQW and QD at the bot-

tom left is based on uttgrad, which measuresF0 slope across the whole utterance.
Since decision trees produce a posterior probability P(d|f), and equation (24.23)
requires a likelihood P(f|d), we need to massage the output of the decision tree by
Bayesianinversion(dividingbytheprior P(di)toturnitintoalikelihood);wesawthis
sameprocesswiththeuseofSVMsandMLPsinsteadofGaussian classiﬁersinspeech
recognitioninSec.10.4.2. Afteralloursimplifyingassum ptionstheresultingequation
forchoosinga dialogueact tagwouldbe:
d∗=argmax
dP(d)P(f|d)P(W|d)

d∗=argmax
=argmax
dP(d|dt−1)P(d|f)
P(d)N
∏
i=2P(wi|wi−1...wi−N+1,d) (24.25)
24.5.3 Detecting Correction Acts
Inadditiontogeneral-purposedialogueactinterpretatio n,wemaywanttobuildspecial-
purpose detectors for particular acts. Let’s consider one s uch detector, for the recog-
nition of user correction of system errors. If a dialogue system misrecognizes an Correction
utterance(usuallyasaresultofASRerrors)theuserwillge nerallycorrecttheerrorby
repeatingthemselves,orrephrasingtheutterance. Dialog uesystemsneedtorecognize

P(d)N
∏
that users are doinga correction,andthen ﬁgure out what the user is trying to correct,
perhapsbyinteractingwiththeuserfurther.
Unfortunately,correctionsare actually harderto recognizethan normalsentences.
Swerts et al. (2000)foundthat correctionsin the TOOTdialo guesystem were misrec-
ognizedabouttwiceasoften(intermsofWER)asnon-correct ions. Onereasonforthis

DRAFT858 Chapter 24. DialogueandConversationalAgents
QD S QY QW 
  0.25 0.25 0.25 0.25
QW 
 0.2561 0.1642 0.2732 0.3065cont_speech_frames < 196.5
S 
 0.2357 0.4508 0.1957 0.1178cont_speech_frames >= 196.5
QW 
 0.2327 0.2018 0.1919 0.3735end_grad < 32.345
QY 
 0.2978 0.09721 0.4181 0.1869end_grad >= 32.345
S 
 0.276 0.2811 0.1747 0.2683f0_mean_zcv < 0.76806
QW 
 0.1859 0.116 0.2106 0.4875f0_mean_zcv >= 0.76806
QW 
 0.2935 0.1768 0.2017 0.328cont_speech_frames_n < 98.388
S 
 0.2438 0.4729 0.125 0.1583cont_speech_frames_n >= 98.388
QW 
 0.2044 0.1135 0.1362 0.5459utt_grad < -36.113
QD 
 0.3316 0.2038 0.2297 0.2349utt_grad >= -36.113
QW 
 0.3069 0.08995 0.1799 0.4233stdev_enr_utt < 0.02903
S 
 0.2283 0.5668 0.1115 0.09339stdev_enr_utt >= 0.02903S 
 0.2581 0.2984 0.2796 0.164cont_speech_frames_n < 98.334
S 
 0.2191 0.5637 0.1335 0.08367cont_speech_frames_n >= 98.334
S 
 0.3089 0.3387 0.1419 0.2105norm_f0_diff < 0.064562
QY 
 0.1857 0.241 0.4756 0.09772norm_f0_diff >= 0.064562
S

0.2438 0.4729 0.125 0.1583cont_speech_frames_n >= 98.388
QW 
QD 
QW 
S 
S 
S 
QY 
S 
 0.3253 0.4315 0.1062 0.137f0_mean_zcv < 0.76197
QW 
 0.2759 0.1517 0.2138 0.3586f0_mean_zcv >= 0.76197
Figure24.19 Decision tree for classifying DECLARATIVE QUESTIONS (QD),STATEMENT (S),YES-NO QUES -
TIONS(QY),and WH-QUESTIONS (QW),afterShribergetal.(1998). Eachnodeinthetreeshow sfourprobabilities,
oneforeachofthefourdialogueactsintheorderQD,S,QY,QW ;themostlikelyofthefourisshownasthelabelfor
the node. Note that questions are shorter than statements (c ontspeechframes), thatQY’s riseat the end(end grad),

0.3253 0.4315 0.1062 0.137f0_mean_zcv < 0.76197
QW 
and thatQD’s risethroughout the utterance (utt grad).
isthatspeakersuseaverydifferentprosodicstylecalled hyperarticulation forcorrec- Hyperarticulation
tions. In hyperarticulated speech, some part of the utteran ce has exaggerated energy,
duration,orF0contours,suchas IsaidBAL-TI-MORE,notBoston (Wade etal.,1992;
Oviattet al.,1998;Levow,1998;Hirschberget al.,2001).
Speakers use various strategies to make corrections; Swert s et al. (2000) catego-

rize these as repetition(REP), paraphrase(PAR), addingco ntent(ADD),and omitting
content(OMIT),asshowninFig. 24.5.3. Theyfoundthatmost correctionswererepe-
titionsandomissionsratherthanparaphrases.
Variousmachinelearningclassiﬁers havebeenappliedto th eproblemof detecting
correctionacts. Useful featuresinclude lexical informat ion(words like “no”, “correc-
tion”, “I don’t”, swear words), prosodic and hyperarticula tion features (increases in
F0 range, pause duration, and word duration, generally norm alized by the values for

previous sentences), features indicating utterance lengt h, ASR features (conﬁdence,
language model probability), and various dialogue feature s(Levow, 1998; Hirschberg
et al.,2001;Bulykoet al.,2005).
In addition to correction detection, a conversational agen t also needs appropriate
controlorupdaterulesinthe dialoguemanager(Bulykoet al .,2005).
24.5.4 Generating DialogueActs: Conﬁrmationand Rejectio n
Deciding which dialogue acts to generate has received much l ess attention than dia-

logue act interpretation. Stent (2002) is one recent model o f dialogue act generation

DRAFTSection24.5. Information-state& DialogueActs 859
Turn Corr
Type
S Onwhichdayof the week doyou want toleave?
S Sorry,I didn’t hear anything from you. Pleasetellme your d eparture date
UUmtomorrow NA
S Youcansaythedeparture date. Forexample, youcansay’tod ay’or’tomorrow’.
Oryou cansay ’Monday’, ’Tuesday’, ’Wednesday’...
UTomorrow REP
USaturday PAR
S Doyouwant toleave onSaturday?
U Yes
S Whattime doyou want toleave?
U Eighto’clock A M
S Doyouwant toleave around tenoclock p-m
UEighto’clock OMIT
UNo OMIT
UEighto’clock A M ADD
Figure24.20 Dialogue fragment from the TOOT system labeled for correcti on type, from
Swertset al.(2000).
Cue Turn-takingacts signaled
um KEEP-TURN,TAKE-TURN,RELEASE-TURN
<lipsmack >,<click>, so,uh KEEP-TURN,TAKE-TURN
youknow, isn’tthat so ASSIGN-TURN

U Yes
UNo OMIT
Figure24.21 Language usedtoperform turn-takingacts, from Stent(2002 ).
in the TRIPS system (Allen et al., 2001), based on Conversati on Acts (page 855) and
theBDImodeltobedescribedinSec.24.7. Stentusesaset ofu pdaterulesforcontent
planning. One such rule says that if a user has just released t he turn, the system can
performaTAKE-TURNact. Anotherrulesaysthatifthesystem hasaproblem-solving
needtosummarizesomeinformationfortheuser,thenitshou ldusetheASSERT con-

versationactwiththatinformationasthesemanticcontent . Thecontentisthenmapped
into words using the standard techniques of natural languag e generation systems (see
e.g., Reiter and Dale (2000)). After an utterance is generat ed, the information state
(discourse context) is updated with its words, syntactic st ructure, semantic form, and
semanticandconversationactstructure. WewillsketchinS ec.24.7someoftheissues
inmodelingandplanningthatmakegenerationa toughongoin gresearcheffort.

Stentshowedthatacrucialissueindialoguegenerationtha tdoesn’toccurinmono-
logue text generationis turn-takingacts. Fig. 24.5.4show s some exampleof the turn-
taking function of various linguistic forms, from her label ing of conversation acts in
theMonroecorpus.
A focus of much work on dialogue act generation is the task of g enerating the
conﬁrmation andrejection acts discussed in Sec. 24.2.5. Because this task is often
solvedbyprobabilisticmethods,we’llbeginthisdiscussi onhere,butcontinueitinthe

DRAFT860 Chapter 24. DialogueandConversationalAgents
followingsection.
Forexample,while earlydialoguesystemstendedto ﬁx the ch oiceofexplicitver-
susimplicitconﬁrmation,recentsystemstreatthequestionofhowtocon ﬁrmmorelike
adialogueactgenerationtask,inwhichtheconﬁrmationstr ategyisadaptive,changing
fromsentencetosentence.
Variousfactorscanbeincludedintheinformation-statean dthenusedasfeaturesto
a classiﬁer in making this decision. For example the conﬁdence that the ASR system
assigns to an utterance can be used by explicitly conﬁrmingl ow-conﬁdencesentences
(Bouwmanetal.,1999;San-Segundoetal.,2001;Litmanetal .,1999;LitmanandPan,
2002). Recallfrompage10.1thatconﬁdenceisametricthatt hespeechrecognizercan
assign toits transcriptionofa sentencetoindicatehowcon ﬁdentit is in thattranscrip-
tion. Conﬁdence is often computed from the acoustic log-lik elihood of the utterance
(greater probability means higher conﬁdence), but prosodi c features can also be used

in conﬁdence prediction. For example utterances preceded b y longer pauses, or with
large F0 excursions, or longer durations are likely to be mis recognized(Litman et al.,
2000).
Another common feature in conﬁrmation is the costof making an error. Thus for
example explicit conﬁrmation is common before actually boo king a ﬂight or moving
moneyin anaccount(Kamm,1994;Cohenet al.,2004).
A system can also choose to rejectan utterance when the ASR conﬁdence is so

2000).
low, or the best interpretation is so semantically ill-form ed, that the system can be
relativelysure thatthe user’sinputwas notrecognizedat a ll. Systemsthusmighthave
a three-tiered level of conﬁdence; below a certain conﬁdenc e threshold, an utterance
is rejected. Above the threshold, it is explicitly conﬁrmed . If the conﬁdence is even
higher,the utteranceisimplicitlyconﬁrmed.
Instead of rejecting or conﬁrming entire utterances, it wou ld be nice to be able to
clarifyonlythepartsoftheutterancethatthesystemdidn’ tunderstand. Ifasystemcan

assign conﬁdence at a more ﬁne-grained level than the uttera nce, it can clarify such
individualelementsvia clariﬁcationsubdialogues . Clariﬁcation
Much of the recent work on generating dialogue acts has been w ithin the Markov
DecisionProcessframework,whichwethereforeturnto next .
24.6 MarkovDecisionProcessArchitecture
Oneofthefundamentalinsightsoftheinformation-stateap proachtodialoguearchitec-
tureisthatthechoiceofconversationalactionsisdynamic allydependentonthecurrent
informationstate. The previoussection discussed how dial oguesystems couldchange

conﬁrmationandrejectionstrategiesbasedoncontext. For exampleiftheASRorNLU
conﬁdence is low, we might choose to do explicit conﬁrmation . If conﬁdenceis high,
we might chose implicit conﬁrmation, or even decide not to co nﬁrm at all. Using a
dynamic strategy lets us choose the action which maximizes d ialogue success, while
minimizing costs. This idea of changing the actions of a dial ogue system based on
optimizingsome kinds of rewards or costs is the fundamental intuition behindmodel-

ingdialogueasa Markovdecisionprocess . Thismodelextendstheinformation-stateMarkov decisionprocess

DRAFTSection24.6. MarkovDecisionProcessArchitecture 861
modelbyaddingaprobabilisticwayofdecidingontheproper actionsgiventhecurrent
state.
A Markov decision process or MDPis characterized by a set of statesSan agent MDP
can be in, a set of actionsAthe agent can take, and a rewardr(a,s) that the agent
receives for taking an action in a state. Given these factors , we can compute a policy
πwhichspeciﬁeswhichaction atheagentshouldtakewheninagivenstate s, soasto
receive the best reward. To understand each of these compone nts, we’ll need to look
at a tutorial example in which the state space is extremely re duced. Thus we’ll return
to the simple frame-and-slot world, looking at a pedagogica l MDP implementation
takenfromLevinet al. (2000). Their tutorialexampleis a “D ay-and-Month”dialogue
system, whose goal is to get correct values of day and month fo r a two-slot frame via
theshortestpossibleinteractionwiththe user.
In principle, a state of an MDP could include any possible inf ormation about the

dialogue, such as the complete dialogue history so far. Usin g such a rich model of
state would make the number of possible states extraordinar ily large. So a model of
state isusuallychosenwhichencodesa muchmorelimitedset ofinformation,suchas
the valuesof the slots in the currentframe, the most recent q uestionasked to the user,
the usersmost recentanswer,the ASRconﬁdence,and so on. Fo r theDay-and-Month
example let’s represent the state of the system as the values of the two slots dayand

month. Thereare a total of 411states(366states with a dayand mont h(countingleap
year), 12 states with a month but no day (d=0, m= 1,2,...12), 3 1 states with a day but
nomonth(m=0,d=1,2,...31),andaspecial initialstate siandﬁnalstate sf.
ActionsofaMDPdialoguesystemmightincludegeneratingpa rticularspeechacts,
orperformingadatabasequerytoﬁndoutinformation. Forth eDay-and-Monthexam-
ple,Levinet al.(2000)proposethefollowingactions:
•ad: a questionaskingfortheday
•am: a questionaskingforthemonth

•adm: a questionaskingforboththedayandthemonth
•af: a ﬁnalactionsubmittingtheformandterminatingthedialo gue
Sincethegoalofthesystemistogetthecorrectanswerwitht heshortestinteraction,
onepossiblerewardfunctionforthesystemwouldintegrate threeterms:
R=−(wini+wene+wfnf) (24.26)
The term niis the numberof interactionswith the user, neis the numberof errors,
nfisthenumberofslotswhichareﬁlled (0,1,or2),andthe wsareweights.
Finally, a dialogue policy πspeciﬁes which actions to apply in which state. Con-

sidertwopossiblepolicies: (1)askingfordayandmonthsep arately,and(2)askingfor
themtogether. Thesemightgeneratethetwo dialoguesshown in Fig.24.22.
In policy 1, the action speciﬁed for the no-date/no-monthst ate is to ask for a day,
whiletheactionspeciﬁedforanyofthe31stateswherewehav eadaybutnotamonth
is to ask for a month. In policy 2, the action speciﬁed for the n o-date/no-monthstate
isto ask anopen-endedquestion( Whichdate )toget botha dayanda month. The two

policies have different advantages; an open prompt can lead to shorter dialogues but
is likely to cause more errors, while a directive prompt is sl ower but less error-prone.
Thus the optimal policy dependson the values of the weights w, and also on the error

DRAFT862 Chapter 24. DialogueandConversationalAgents

md=D
m=0d=D
m=Md=-1
m=-1
d=D
m=Md=-1
m=-1d=0
m=0Which day? Which month?
What date? Goodbye.Goodbye.Policy 1 (directive)
Policy 2 (open)c1 = -3wi + 2pdwe
c2 = -2wi + 2powe
Figure24.22 Twopolicies forgetting amonth and a day. After Levinetal. ( 2000).
rates of the ASR component. Let’s call pdthe probability of the recognizer making
an errorinterpretinga monthor a dayvalueaftera directive prompt. The(presumably
higher) probability of error interpreting a month or day val ue after an open prompt
we’llcall po. TherewardfortheﬁrstdialogueinFig.24.22isthus −3×wi+2×pd×
we. The reward for the second dialogue in Fig. 24.22 is −2×wi+2×po×we. The
directive promptpolicy,policy 1, is thus better than polic y 2 when the improvederror
ratejustiﬁesthe longerinteraction,i.e.,when pd−po>wi
2we.
In the example we’ve seen so far, there were only two possible actions, and hence
only a tiny number of possible policies. In general, the numb er of possible actions,

2we.
states, and policies is quite large, and so the problem of ﬁnd ing the optimal policy π∗
ismuchharder.
Markov decision theory together with classical reinforcem ent learning gives us a
way to think about this problem. First, generalizing from Fi g. 24.22, we can think of
anyparticulardialogueasatrajectoryin state space:
s1→a1,r1s2→a2,r2s3→a3,r3··· (24.27)
Thebestpolicy π∗istheonewiththegreatestexpectedrewardoveralltraject ories.
What istheexpectedrewardfora givenstate sequence? Themo stcommonwaytoas-

signutilities orrewardstosequencesisto use discounted rewards . Herewe computeDiscountedreward
theexpectedcumulativereward Qofasequenceasa discountedsumoftheutilitiesof
theindividualstates:
Q([s0,a0,s1,a1,s2,a2···]) =R(s0,a0)+γR(s1,a1)+γ2R(s2,a2)+···, (24.28)
The discount factor γis a number between 0 and 1. This makes the agent care
more about current rewards than future rewards; the more fut ure a reward, the more
discountedits value.
Giventhismodel,itispossibletoshowthattheexpectedcum ulativereward Q(s,a)

fortakingaparticularactionfromaparticularstate isthe followingrecursiveequation
calledthe Bellman equation : Bellmanequation
Q(s,a) =R(s,a)+γ∑
s′P(s′|s,a)max
a′Q(s′,a′) (24.29)

DRAFTSection24.6. MarkovDecisionProcessArchitecture 863
WhattheBellmanequationsaysisthattheexpectedcumulati verewardforagiven
state/action pair is the immediate reward for the current st ate plus the expected dis-
counted utility of all possible next states s′, weighted by the probability of moving to
thatstate s′,andassumingoncetherewe taketheoptimalaction a′.
Eq. 24.29 makes use of two parameters. We need a model of P(s′|s,a), i.e. how
likely a given state/action pair (s,a)is to lead to a new state s′. And we also need
a good estimate of R(s,a). If we had lots of labeled training data, we could simply
compute both of these from labeled counts. For example, with labeled dialogues, we
could simply count how many times we were in a given state s, and out of that how
many times we took action ato get to state s′, to estimate P(s′|s,a). Similarly, if we
hadahand-labeledrewardforeachdialogue,we couldbuilda modelof R(s,a).
Given these parameters, it turns out that there is an iterati ve algorithm for solving

the Bellman equation and determiningproper Q values, the value iteration algorithm Value iteration
(Suttonand Barto, 1998; Bellman, 1957). We won’tpresent th is here,but see Chapter
17 of Russell and Norvig (2002) for the details of the algorit hm as well as further
informationonMarkovDecisionProcesses.
How dowe get enoughlabeled trainingdata to set these parame ters? Thisis espe-
cially worrisomein anyreal problem,wherethe numberof sta tessis extremelylarge.

Twomethodshavebeenappliedinthepast. Theﬁrstistocaref ullyhand-tunethestates
and policies so that there are a very small number of states an d policies that need to
be set automatically. In this case we can build a dialogue sys tem which explore the
statespacebygeneratingrandomconversations. Probabili tiescanthenbesetfromthis
corpus of conversations. The second is to build a simulated u ser. The user interacts
withthesystemmillionsoftimes,andthesystemlearnsthes tatetransitionandreward
probabilitiesfromthiscorpus.

The ﬁrst approach, using real users to set parameters in a sma ll state space, was
taken by Singh et al. (2002). They used reinforcementlearni ngto make a small set of
optimal policy decisions. Their NJFun system learned to cho ose actions which varied
the initiative (system, user, or mixed)and the conﬁrmation strategy (explicit or none).
The state of the system was speciﬁed by values of 7 features in cluding which slot in
the frame is being worked on (1-4), the ASR conﬁdence value (0 -5), how many times

a current slot question had been asked, whether a restrictiv e or non-restrictive gram-
mar was used, and so on. The result of using only 7 features wit h a small number of
attributes resulted in a small state space (62 states). Each state had only 2 possible
actions (system versus user initiative when asking questio ns, explicit versus no con-
ﬁrmation when receiving answers). They ran the system with r eal users, creating 311

conversations. Each conversation had a very simple binary r eward function; 1 if the
user completed the task (ﬁnding speciﬁed museums, theater, winetasting in the New
Jersey area), 0 if the user did not. The system successful lea rned a good dialoguepol-
icy (roughly,start with user initiative,then backofto eit her mixedorsystem initiative
whenreaskingforanattribute;conﬁrmonlyatlowerconﬁden cevalues;bothinitiative
andconﬁrmationpolicies,however,aredifferentfordiffe rentattributes). Theyshowed
thattheirpolicyactuallywasmoresuccessfulbasedonvari ousobjectivemeasuresthan

manyhand-designedpoliciesreportedin theliterature.
The simulated user strategy was taken by Levin et al. (2000), in their MDP model
with reinforcement learning in the ATIS task. Their simulat ed user was a generative

DRAFT864 Chapter 24. DialogueandConversationalAgents
stochasticmodelthatgiventhesystem’scurrentstateanda ctions,producesaframe-slot
representationofauserresponse. Theparametersofthesim ulateduserwereestimated
fromacorpusofATISdialogues. Thesimulateduserwasthenu sedtointeractwiththe
systemfortensofthousandsofconversations,leadingtoan optimaldialoguepolicy.
While the MDP architecture offers a powerful new way of model ing dialogue be-
havior, it relies on the problematic assumption that the sys tem actually knows what
state it is in. This is of course not true in a number of ways; th e system never knows
the true internal state of the user, and even the state in the d ialogue may be obscured
by speech recognition errors. Recent attempts to relax this assumption have relied on
PartiallyObservableMarkovDecisionProcesses,orPOMDPs (sometimespronounced
‘pom-deepeez’). In a POMDP, we model the user output as an obs erved signal gen-
erated from yet another hidden variable. There are also prob lems with MDPs and

POMDPs related to computationalcomplexityand simulation swhich aren’t reﬂective
oftrueuserbehavior;Seetheendnotesforreferences.
24.7 Advanced: Plan-basedDialogueAgents
One of the earliest models of conversational agent behavior , and also one of the most
sophisticated,isbasedontheuseofAIplanningtechniques . Forexample,theRochester
TRIPSagent(Allenetal.,2001)simulateshelpingwithemer gencymanagement,plan-
ning where and how to supply ambulancesor personnelin a simu lated emergencysit-

uation. Thesameplanningalgorithmsthatreasonhowto geta nambulancefrompoint
AtopointBcanbeappliedtoconversationaswell. Sincecomm unicationandconver-
sationarejustspecialcasesofrationalactionintheworld ,theseactionscanbeplanned
like anyother. Soan agentseekingto ﬁndoutsomeinformatio ncancomeupwith the
plan of asking the interlocutorfor the information. An agen t hearing an utterance can
interpreta speechactbyrunningtheplanner‘inreverse’,u singinferencerulestoinfer
whatplantheinterlocutormighthavehadtocausethemto say whattheysaid.

Usingplanstogenerateandinterpretsentencesinthiswayr equirethattheplanner
have goodmodelsof its beliefs,desires, andintentions (BDI), as well as those of the
interlocutor. Plan-based models of dialogue are thus often referred to as BDImodels. BDI
BDI modelsofdialoguewereﬁrst introducedbyAllen,Cohen, Perrault,andtheircol-
leaguesand studentsin a numberof inﬂuential papersshowin ghow speech acts could
be generated (Cohen and Perrault, 1979), and interpreted (P errault and Allen, 1980;

Allen and Perrault, 1980). At the same time, Wilensky (1983) introduced plan-based
models of understanding as part of the task of interpreting s tories. In another related
line of research, Grosz and her colleaguesand students show ed how using similar no-
tions of intention and plans allowed ideas of discourse stru cture and coherence to be
appliedto dialogue.
24.7.1 Plan-Inferential Interpretation and Production
Let’s ﬁrst sketch out the ideas of plan-based comprehension and production. How

might a plan-based agent act as the human travel agent to unde rstand sentence C 2in

DRAFTSection24.7. Advanced: Plan-basedDialogueAgents 865
thedialoguerepeatedbelow?
C1: Ineedto travelinMay.
TheGriceanprincipleofRelevancecanbeusedtoinferthatt heclient’smeetingis
relevantto the ﬂight booking. The system may knowthat one pr econditionfor having
ameeting(atleast beforewebconferencing)isbeingat thep lacewherethemeetingis
in. Onewayofbeingataplaceisﬂyingthere,andbookingaﬂig htisapreconditionfor
ﬂying there. The system can follow this chain of inference, a bducing that user wants
toﬂy onadatebeforethe12th.
Next, consider how our plan-based agent could act as the huma n travel agent to
produce sentence A 1in the dialogue above. The planning agent would reason that i n
ordertohelpaclientbookaﬂightit mustknowenoughinforma tionabouttheﬂightto
bookit. It reasonsthat knowingthemonth(May)isinsufﬁcie ntinformationtospecify
a departureorreturndate. Thesimplest way toﬁndout thenee deddate informationis

toask theclient.
In the rest of this section, we’ll ﬂesh out the sketchy outlin es of planning for un-
derstandingand generation using Perrault and Allen’s form al deﬁnitions of belief and
desireinthepredicatecalculus. Reasoningaboutbeliefis donewithanumberofaxiom
schemas inspired by Hintikka (1969). We’ll represent “ Sbelieves the proposition P”
as the two-place predicate B(S,P), with axiom schemas such as B(A,P)∧B(A,Q)⇒
B(A,P∧Q). Knowledgeisdeﬁnedas“truebelief”; SknowsthatP willberepresented

asKNOW (S,P),deﬁnedasKNOW (S,P)≡P∧B(S,P).
ThetheoryofdesirereliesonthepredicateWANT.Ifanagent SwantsPtobetrue,
we sayWANT (S,P), orW(S,P)for short. Pcan be a state or the execution of some
action. Thusif ACT isthe nameofanaction, W(S,ACT(H))meansthat SwantsHto
doACT.Thelogic ofWANT reliesonitsownset ofaxiomschemas just like thelogic
ofbelief.
The BDI models also require an axiomatization of actions and planning; the sim-
plest of these is based on a set of action schema s based on the simple AI planning Actionschema

ofbelief.
modelSTRIPS (FikesandNilsson, 1971). Eachactionschemah asa set ofparameters
withconstraints aboutthetypeofeachvariable,andthreeparts:
•Preconditions: Conditionsthatmustalreadybetruetoperformtheaction.
•Effects:Conditionsthatbecometrueasaresultofperformingtheact ion.
•Body:A set of partiallyorderedgoal states that must beachievedi n performing
theaction.
In the travel domain,for example, the action of agent Abookingﬂight F1 for client C
mighthavethefollowingsimpliﬁeddeﬁnition:

DRAFT866 Chapter 24. DialogueandConversationalAgents
BOOK-FLIGHT(A,C,F) :
Constraints: Agent(A) ∧Flight(F)∧Client(C)
Precondition: Know(A,depart-date(F)) ∧Know(A,depart-time(F))
∧Know(A,origin(F)) ∧Know(A,ﬂight-type(F))
∧Know(A,destination(F)) ∧Has-Seats(F)∧
W(C,(BOOK(A,C,F))) ∧...
Effect: Flight-Booked(A,C,F)
Body: Make-Reservation(A,F,C)
This same kind of STRIPS action speciﬁcation can be used for s peech acts. IN-
FORM isthespeechactofinformingthehearerofsomeproposi tion,basedonGrice’s
(1957)ideathataspeakerinformsthehearerofsomethingme relybycausingthehearer
tobelievethatthespeakerwantsthemtoknowsomething:
INFORM(S,H,P):
Constraints: Speaker(S) ∧Hearer(H)∧Proposition(P)
Precondition: Know(S,P) ∧W(S, INFORM(S,H, P))
Effect: Know(H,P)
Body: B(H,W(S,Know(H,P)))
REQUEST is the directive speech act for requesting the heare r to perform some
action:
REQUEST(S,H,ACT):
Constraints: Speaker(S) ∧Hearer(H)∧ACT(A)∧H isagentofACT
Precondition: W(S,ACT(H))
Effect: W(H,ACT(H))
Body: B(H,W(S,ACT(H)))

action:
Let’snowsee howa plan-baseddialoguesystemmightinterpr etthesentence:
C2: Ineedtobetherefora meetingthat’sfromthe12thtothe15t h.
We’ll assume the system has the BOOK-FLIGHT plan mentioned a bove. In ad-
dition, we’ll need knowledge about meetings and getting to t hem, in the form of the
MEETING,FLY-TO,andTAKE-FLIGHTplans,sketchedbroadlyb elow:
MEETING(P,L,T1,T2):
Constraints: Person(P) ∧Location(L)∧Time (T1)∧Time(T2)∧Time(TA)
Precondition: At (P,L, TA)
Before(TA,T1)
Body: ...
FLY-TO(P,L, T):

Body: ...
Constraints: Person(P) ∧Location(L)∧Time(T)
Effect: At (P,L,T)
Body: TAKE-FLIGHT(P,L,T)

DRAFTSection24.7. Advanced: Plan-basedDialogueAgents 867
TAKE-FLIGHT(P,L, T):
Constraints: Person(P) ∧Location(L)∧Time (T)∧Flight (F)∧Agent(A)
Precondition: BOOK-FLIGHT(A,P, F)
Destination-Time(F)= T
Destination-Location(F)= L
Body: ...
Nowlet’sassumethatanNLUmodulereturnsasemanticsforth eclient’sutterance
which(amongotherthings)includesthefollowingsemantic content:
MEETING (P,?L, T1,T2)
Constraints: P = Client ∧T1=May12∧T2= May15
Our plan-based system now has two plans established, one MEE TING plan from
this utterance,and oneBOOK-FLIGHT plan fromthe previousu tterance. The system
implicitly uses the Gricean Relevance intuition to try to co nnect them. Since BOOK-
FLIGHTisapreconditionforTAKE-FLIGHT,thesystemmayhyp othesize(infer)that
the user is planning a TAKE-FLIGHT. Since TAKE-FLIGHT is in t he body of FLY-
TO, the system further infers a FLY-TO plan. Finally, since t he effect of FLY-TO is
a precondition of the MEETING, the system can unify each of th e people, locations,

andtimes of all of these plans. Theresult will be that the sys tem knowsthat the client
wantstoarriveat thedestinationbeforeMay12th.
Let’sturntothedetailsofoursecondexample:
How does a plan-based agent know to ask question A 1? This knowledge comes
from the BOOK-FLIGHT plan, whose preconditionswere that th e agent know a vari-
ety of ﬂight parameters including the departure date and tim e, origin and destination

cities,andsoforth. UtteranceC 1containstheorigincityandpartialinformationabout
the departure date; the agent has to request the rest. A plan- based agent would use an
action schema like REQUEST-INFO to represent a plan for aski ng information ques-
tions(simpliﬁedfromCohenandPerrault(1979)):
REQUEST-INFO(A,C,I) :
Constraints: Agent(A) ∧Client(C)
Precondition: Know(C,I)
Effect: Know(A,I)
Body: B(C,W(A,Know(A,I)))
BecausetheeffectsofREQUEST-INFOmatcheachpreconditio nofBOOK-FLIGHT,

theagentcanuseREQUEST-INFOto achievethemissinginform ation.
24.7.2 The Intentional Structure ofDialogue
In Sec. 21.2 we introduced the idea that the segments of a disc ourse are related by
coherence relations likeExplanation orElaboration which describe the informa-
tionalrelationbetweendiscoursesegments. TheBDIapproachtout teranceinterpreta-
tiongivesrisetoanotherviewofcoherencewhichisparticu larlyrelevantfordialogue,

DRAFT868 Chapter 24. DialogueandConversationalAgents
theintentional approach(Grosz and Sidner, 1986). According to this approa ch,what
makesadialoguecoherentisits intentionalstructure ,theplan-basedintentionsoftheIntentionalstructure
speakerunderlyingeachutterance.
Theseintentionsare instantiated in the modelby assumingt hat eachdiscourse has
an underlying purpose held by the person who initiates it, ca lled thediscourse pur-
pose(DP). Each discoursesegmentwithinthe discourse hasa corr espondingpurpose,Discoursepurpose
adiscourse segment purpose (DSP), which has a role in achieving the overall DP.Discoursesegment purpose
PossibleDPs/DSPsincludeintendingthatsomeagentintend toperformsomephysical
task,orthatsomeagentbelievesomefact.
Asopposedtothelargersetsofcoherencerelationsusedini nformationalaccounts
of coherence, Grosz and Sidner propose only two such relatio ns:dominance and
satisfaction-precedence . DSP1dominatesDSP 2ifsatisfyingDSP 2isintendedtopro-

vide part of the satisfaction of DSP 1. DSP1satisfaction-precedesDSP 2if DSP1must
besatisﬁed beforeDSP 2.
A5: The ﬁrst one departsPGH at 10:00amarrivesSeattle at 12:05 their time. The

A6: On the 11th? OK. Departingat 5:55pmarrivesSeattle at 8pm, U.S. Air ﬂight
115.
C7: OK.
Figure24.23 A fragment from a telephone conversation between a client (C ) and a travel
agent (A)(repeated fromFig. 24.4).
Consider the dialogue between a client (C) and a travel agent (A) that we saw

115.
C7: OK.
earlier, repeated here in Fig. 24.23. Collaboratively, the caller and agent successfully
identify a ﬂight that suits the caller’s needs. Achieving th is joint goal requires that
a top-level discourse intention be satisﬁed, listed as I1 be low, in addition to several
intermediateintentionsthatcontributedtothesatisfact ionofI1,listedasI2-I5:
I1: (IntendC(IntendA(A ﬁndaﬂightforC)))
I2: (IntendA (IntendC(Tell CA departuredate)))
I3: (IntendA (IntendC(Tell CA destinationcity)))
I4: (IntendA (IntendC(Tell CA departuretime)))

I5: (IntendC(IntendA(A ﬁndanonstopﬂightforC)))

DRAFTSection24.8. Summary 869
IntentionsI2–I5areallsubordinatetointentionI1,asthe ywerealladoptedtomeetpre-
conditions for achieving intention I1. This is reﬂected in t he dominance relationships
below:
I1dominatesI2∧I1dominatesI3∧I1dominatesI4∧I1dominatesI5
Furthermore, intentions I2 and I3 needed to be satisﬁed befo re intention I5, since the
agentneededtoknowthedeparturedateanddestinationinor dertostartlistingnonstop
ﬂights. Thisis reﬂectedin thesatisfaction-precedencere lationshipsbelow:
I2satisfaction-precedesI5 ∧I3satisfaction-precedesI5
Thedominancerelationsgiverise tothediscoursestructur edepictedinFig.24.24.
Eachdiscoursesegmentisnumberedincorrespondencewitht heintentionnumberthat
servesasitsDP/DSP.
DS1
C1DS2DS3DS4DS5
A1–C2A2–C3A3C4–C7
Figure24.24 Discourse Structure ofthe Flight ReservationDialogue
Intentions and their relationships give rise to a coherent d iscourse based on their
role in the overall planthat the caller is inferred to have. We assume that the caller

DS1
and agent have the plan BOOK-FLIGHT described on page 866. Th is plan requires
thattheagentknowthedeparturetimeanddateandso on. Aswe discussedabove,the
agent can use the REQUEST-INFO action scheme from page 867 to ask the user for
thisinformation.
Subsidiarydiscourse segmentsare also called subdialogues ; DS2 and DS3 in par- Subdialogue
ticularare information-sharing (Chu-CarrollandCarberry,1998) knowledgeprecon-
ditionsubdialogues(Lochbaumetal.,1990;Lochbaum,1998),sinc etheyareinitiated

bytheagenttohelpsatisfypreconditionsofahigher-level goal.
Algorithms for inferring intentional structure in dialogu e work similarly to algo-
rithmsforinferringdialogueacts,eitheremployingtheBD Imodel(e.g.,Litman,1985;
GroszandSidner,1986;LitmanandAllen,1987;Carberry,19 90;PassonneauandLit-
man, 1993; Chu-Carroll and Carberry, 1998), or machine lear ningarchitecturesbased
on cue phrases (Reichman, 1985; Grosz and Sidner, 1986; Hirs chberg and Litman,

1993), prosody (Hirschberg and Pierrehumbert, 1986; Grosz and Hirschberg, 1992;
PierrehumbertandHirschberg,1990;HirschbergandNakata ni,1996),andothercues.
24.8 Summary
Conversational agents are a crucial speech and language processing application th at
are alreadywidely used commercially. Research on these age ntsreliescruciallyon an
understandingofhumandialogueorconversationalpractic es.

DRAFT870 Chapter 24. DialogueandConversationalAgents
•Dialoguesystemsgenerallyhave5components: speechrecog nition,naturallan-
guage understanding, dialogue management, natural langua ge generation, and
speech synthesis. They may also have a task manager speciﬁc t o the task do-
main.
•Dialoguearchitecturesforconversationalagentsinclude ﬁnite-statesystems, frame-
basedproductionsystems,andadvancedsystemssuchasinformati on-state,Markov
DecisionProcesses, and BDI(belief-desire-intention )models.
•Turn-taking, grounding, conversational structure, impli cature, and initiative are
crucialhumandialoguephenomenathatmustalsobedealtwit hinconversational
agents.
•Speakingindialogueisa kindofaction;these actsarerefer redtoasspeechacts
ordialogueacts . Modelsexistforgeneratingandinterpretingtheseacts.
Early work on speech and language processing had very little emphasis on the study
of dialogue. The dialogue manager for the simulation of the p aranoid agent PARRY

agents.
(Colby et al., 1971), was a little more complex. Like ELIZA, i t was based on a pro-
duction system, but where ELIZA’s rules were based only on th e words in the user’s
previoussentence,PARRY’srulesalsorelyonglobalvariab lesindicatingitsemotional
state. Furthermore,PARRY’s output sometimes makes use of s cript-like sequencesof
statements when the conversation turns to its delusions. Fo r example, if PARRY’s
angervariableishigh,hewillchoosefromasetof“hostile”outpu ts. Iftheinputmen-

tions his delusion topic, he will increase the value of his fearvariable and then begin
toexpressthesequenceofstatementsrelatedtohisdelusio n.
The appearance of more sophisticated dialogue managers awa ited the better un-
derstanding of human-human dialogue. Studies of the proper ties of human-human
dialogue began to accumulate in the 1970’s and 1980’s. The Co nversation Analy-
sis community (Sacks et al., 1974; Jefferson, 1984; Scheglo ff, 1982) began to study

the interactional propertiesof conversation. Grosz’s (19 77b) dissertation signiﬁcantly
inﬂuenced the computational study of dialogue with its intr oduction of the study of
dialogue structure, with its ﬁnding that “task-oriented di alogues have a structure that
closely parallels the structure of the task being performed ” (p. 27), which led to her
work on intentional and attentional structure with Sidner. Lochbaumet al. (2000)is a
good recent summary of the role of intentional structure in d ialogue. The BDI model

integratingearlier AI planningwork (Fikes and Nilsson, 19 71)with speech act theory
(Austin, 1962; Gordon and Lakoff, 1971; Searle, 1975a) was ﬁ rst worked out by Co-
henandPerrault(1979),showinghowspeechactscouldbegen erated,andPerraultand
Allen(1980)andAllenandPerrault(1980),applyingtheapp roachtospeech-actinter-
pretation. Simultaneouswork on a plan-basedmodel of under standingwas developed
byWilensky(1983)in theSchankiantradition.
Probabilistic models of dialogue act interpretation were i nformed by linguistic

work which focused on the discourse meaning of prosody (Sag a nd Liberman, 1975;

DRAFTSection24.8. Summary 871
Pierrehumbert, 1980), by Conversation Analysis work on mic rogrammar (e.g. Good-
win,1996),byworksuchasHinkelmanandAllen(1989),whosh owedhowlexicaland
phrasalcuescouldbeintegratedintotheBDI model,andthen workedoutat a number
of speech and dialoguelabs in the 1990’s(Waibel, 1988; Daly and Zue, 1992; Kompe
et al., 1993; Nagata and Morimoto, 1994; Woszczyna and Waibe l, 1994; Reithinger
et al., 1996; Kita et al., 1996; Warnke et al., 1997; Chu-Carr oll, 1998; Stolcke et al.,
1998;Tayloret al.,1998;Stolckeet al.,2000).
Modern dialogue systems drew on research at many different l abs in the 1980’s
and 1990’s. Models of dialogue as collaborative behavior we re introduced in the late
1980’sand1990’s,includingtheideasofcommonground(Cla rkandMarshall,1981),
reference as a collaborative process (Clark and Wilkes-Gib bs, 1986), and models of
joint intentions (Levesque et al., 1990), and shared plans (Grosz and Sidner, 1980).

Related to this area is the study of initiative in dialogue, studying how the dialogue
control shifts between participants (Walker and Whittaker , 1990; Smith and Gordon,
1997;Chu-CarrollandBrown,1997).
AwidebodyofdialogueresearchcameoutofAT&TandBellLabo ratoriesaround
the turn of the century, including much of the early work on MD P dialogue systems
as well as fundamentalwork on cue-phrases,prosody,and rej ectionand conﬁrmation.
Work on dialogueacts and dialoguemovesdrew from a numberof sources, including

HCRC’s Map Task (Carletta et al., 1997b), and the work of Jame s Allen and his col-
leagues and students, for example Hinkelman and Allen (1989 ), showing how lexical
and phrasal cues could be integrated into the BDI model of spe ech acts, and Traum
(2000),TraumandHinkelman(1992),andfromSadek(1991).
Muchrecentacademicworkindialoguefocusesonmultimodal applications(John-
stonetal.,2007;NiekraszandPurver,2006,interalia),on theinformation-statemodel

(TraumandLarsson,2003,2000)oronreinforcementlearnin garchitecturesincluding
POMDPs (Roy et al., 2000; Young, 2002; Lemon et al., 2006; Wil liams and Young,
2005,2000). WorkinprogressonMDPsandPOMDPsfocusesonco mputationalcom-
plexity(theycurrentlycanonlyberunonquitesmalldomain swithlimitednumbersof
slots), and on improvingsimulationsto make them more reﬂec tiveof true user behav-
ior. Alternative algorithmsinclude SMDPs (Cuay´ ahuitl et al., 2007). See Russell and
Norvig(2002)andSuttonandBarto(1998)forageneralintro ductiontoreinforcement
learning.

learning.
Recent yearshave seen the widespreadcommercialuse of dial oguesystems, often
basedonVoiceXML.Somemoresophisticatedsystemshaveals oseendeployment. For
exampleClarissa, the ﬁrst spokendialoguesystem used in space, is a speech-e nabled Clarissa
procedure navigator that was used by astronauts on the Inter national Space Station
(RaynerandHockey,2004;Aistetal.,2002). Muchresearchf ocusesonmoremundane
in-vehicle applications in cars Weng et al. (2006, inter alia ). Among the important

learning.
technicalchallengesinembeddingthesedialoguesystemsi nrealapplicationsaregood
techniquesforendpointing(decidingifthespeakerisdone talking)(Ferreretal.,2003)
andfornoiserobustness.
Good surveys on dialogue systems include Harris (2005), Coh en et al. (2004),
McTear (2002, 2004), Sadek and De Mori (1998), Delgado and Ar aki (2005), and
thedialoguechapterinAllen (1995).

DRAFT872 Chapter 24. DialogueandConversationalAgents
Exercises
24.1List the dialogue act misinterpretationsin the Who’s On First routine at the be-
ginningofthechapter.
24.2Write a ﬁnite-state automaton for a dialogue manager for che cking your bank
balanceandwithdrawingmoneyat anautomatedtellermachin e.
24.3Dispreferredresponses(forexampleturningdownarequest )areusuallysignaled
by surface cues, such as signiﬁcant silence. Try to notice th e next time you or
someoneelseuttersadispreferredresponse,andwritedown theutterance. What
are some other cues in the response that a system might use to d etect a dispre-
ferredresponse? Considernon-verbalcueslikeeye-gazean dbodygestures.
24.4When asked a question to which they aren’t sure they know the a nswer, peo-
ple display their lack of conﬁdence via cues that resemble ot her dispreferred
responses. Try to notice some unsure answers to questions. W hat are some of
thecues? If youhavetroubledoingthis, read Smithand Clark (1993)andlisten

speciﬁcallyforthecuestheymention.
24.5BuildaVoiceXMLdialoguesystemforgivingthecurrenttime aroundtheworld.
The system should ask the user for a city and a time format (24 h our, etc) and
shouldreturnthecurrenttime,properlydealingwith timez ones.
24.6Implementasmallair-travelhelpsystembasedontextinput . Yoursystemshould
get constraints from the user about a particular ﬂight that t hey want to take,
expressed in natural language, and display possible ﬂights on a screen. Make
simplifyingassumptions. You may build in a simple ﬂight dat abase or you may

usea ﬂightinformationsystem onthewebasyourbackend.
24.7Augment your previous system to work with speech input via Vo iceXML. (or
alternatively, describe the user interface changes you wou ld have to make for it
toworkvia speechoverthephone). What werethemajordiffer ences?
24.8Design a simple dialogue system for checking your email over the telephone.
ImplementinVoiceXML.
24.9Test your email-reading system on some potential users. Cho ose some of the
metricsdescribedinSec.24.4.2andevaluateyoursystem.

Chapter25
Machine Translation
Theprocessoftranslatingcomprisesinitsessencethewhol e
secret of human understanding and social communication...
AttributedtoHans-GeorgGadamer
What is translation? Onaplatter
Apoet’s pale and glaringhead,
A parrot’s screech, amonkey’s chatter,
Andprofanation ofthe dead.
Nabokov, OnTranslating Eugene Onegin
Proper words inproper places
Jonathan Swift
Thischapterintroducestechniquesfor machinetranslation (MT),theuseofcomput-Machinetranslation
MTersto automatesome orall ofthe processof translatingfrom onelanguageto another.

Translation, in its full generality, is a difﬁcult, fascina ting, and intensely human en-
deavor, as rich as any other area of human creativity. Consid er the following passage
fromtheendofChapter45ofthe18th-centurynovel TheStoryoftheStone ,alsocalled
DreamoftheRedChamber ,byCaoXueQin(Cao,1792),transcribedintheMandarin
dialect:
dai yu zi zai chuang shang gan nian bao chai...you ting jian ch uang wai zhu shao xiang
ye zhe shang, yusheng xi li,qinghan toumu, bujue you di xia le ilai.

dialect:
Fig. 25.1 shows the English translation of this passage by Da vid Hawkes, in sen-
tenceslabeledE 1-E4. Foreaseofreading,insteadofgivingtheChinese,wehaves hown
theEnglishglossesofeachChineseword IN SMALL CAPS . WordsinblueareChinese
wordsnottranslatedintoEnglish,orEnglishwordsnotinth eChinese. Wehaveshown
alignment linesbetweenwordsthatroughlycorrespondinthetwo langu ages.
Consider some of the issues involved in this translation. Fi rst, the English and

Chinese texts are very different structurally and lexicall y. The four English sentences
(notice the periodsin blue) correspondto one long Chinese s entence. The word order
ofthetwotextsisverydifferent,aswe canseebythemanycro ssedalignmentlinesin
Fig. 25.1. The English has many more words than the Chinese, a s we can see by the
large number of English words marked in blue. Many of these di fferencesare caused
by structural differences between the two languages. For ex ample, because Chinese

rarely marks verbal aspect or tense, the English translatio n has additional words like
as,turned to, andhad begun , and Hawkes had to decide to translate Chinese touas
penetrated , rather than say was penetrating orhad penetrated . Chinese has fewer
articles than English, explaining the large number of blue thes. Chinese also uses far

DRAFT874 Chapter 25. MachineTranslation
Figure25.1 A Chinese passage from Dream of the RedChamber , withthe Chinese words represented byEnglish
glosses IN SMALL CAPS . Alignment linesare drawnbetween‘Chinese’ words and thei rEnglishtranslations. Words
initalicsare Chinese words not translatedintoEnglish, or Englishwords not inthe original Chinese.
fewerpronounsthanEnglish,so Hawkeshadto insert sheandherin manyplacesinto
theEnglishtranslation.
Stylistic and cultural differences are another source of di fﬁculty for the transla-
tor. Unlike English names, Chinese names are made up of regul ar content words with
meanings. Hawkes chose to use transliterations ( Daiyu) for the names of the main
characters but to translate names of servants by their meani ngs (Aroma, Skybright).
To make the image clear for English readers unfamiliar with C hinese bed-curtains,
Hawkes translated ma(‘curtain’)as curtains of her bed . The phrase bambootip plan-

tainleaf,althoughelegantinChinese,wheresuchfour-characterph rasesareahallmark
of literate prose, would be awkward if translated word-for- word into English, and so
Hawkesusedsimply bamboosandplantains .
Translationofthissortclearlyrequiresadeepandrichund erstandingofthesource
language and the input text, and a sophisticated, poetic, an d creative command of the
target language. The problem of automatically performingh igh-qualityliterary trans-

lation between languages as different as Chinese to English is thus far too hard to
automatecompletely.
However,evennon-literarytranslationsbetweensuchsimi larlanguagesasEnglish
and French can be difﬁcult. Here is an English sentence from t he Hansards corpus of
Canadianparliamentaryproceedings,withitsFrenchtrans lation:
English: Followingatwo-yeartransitionalperiod,thenewFoodstu ffsOrdinanceforMin-
eral Water came into effect on April 1, 1988. Speciﬁcally, it contains more stringent re-
quirements regarding qualityconsistency and purityguara ntees.

French: La nouvelle ordonnance f` ed` erale sur les denr´ ees aliment aires concernant entre
autresleseauxmin´ erales, entr´ eeenvigueur leleravril1 988apr´ es une p´ eriode transitoire

DRAFT875
de deux ans. exige surtout une plus grande constance dans la q ualit´ e et une garantie de la
puret´ e.
French gloss: THE NEW ORDINANCE FEDERAL ON THE STUFF FOOD CONCERNING
AMONG OTHERS THE WATERS MINERAL CAME INTO EFFECT THE 1STAPRIL1988
AFTER A PERIOD TRANSITORY OF TWO YEARS REQUIRES ABOVE ALL A LA RGER
CONSISTENCY IN THE QUALITY AND A GUARANTEE OF THE PURITY .
Despitethestrongstructuralandvocabularyoverlapsbetw eenEnglishandFrench,
suchtranslation,likeliterarytranslation,stillhastod ealwithdifferencesinwordorder
(e.g.,the locationof the followinga two-yeartransitionalperiod phrase)and in struc-
ture (e.g., English uses the noun requirements while the French uses the verb exige
‘REQUIRE ’).
Nonetheless,suchtranslationsaremucheasier,andanumbe rofnon-literarytrans-
lation tasks can be addressed with current computational mo dels of machine transla-
tion, including: (1) tasks for which a rough translation is adequate, (2) tasks where

a human post-editor is used, and (3) tasks limited to small sublanguage domains in
whichfullyautomatichighqualitytranslation (FAHQT)isstill achievable. FAHQT
Information acquisition on the web is the kind of task where a rough translation
maystill beuseful. Supposeyouwereatthemarketthismorni ngandsawsomelovely
pl´atanos(plantains, a kind of banana) at the local Caribbean grocery store and you
wantto knowhowto cookthem. Yougototheweb,andﬁndthefoll owingrecipe:
PlatanoenNaranja Para6personas

3Pl´ atanos maduros 2cucharadas de mantequilla derretida
1taza de jugo (zumo) de naranja 5cucharadas de az´ ucar moren a oblanc
1/8cucharadita de nuez moscada en polvo 1cucharada de ralla dura de naranja
1cucharada de canela enpolvo (opcional)
Pelar los pl´ atanos, cortarlos por la mitad y, luego, a lo lar go. Engrasar una fuente o pirex con
margarina. Colocar los pl´ atanos y ba˜ narlos con la mantequ illa derretida. En un recipiente

hondo, mezclar el jugo (zumo) de naranja con el az´ ucar, jeng ibre, nuez moscada y ralladura
de naranja. Verter sobre los pl´ atanos y hornear a 325◦F. Los primeros 15 minutos, dejar los
p´ atanos cubiertos, hornear 10 o15minutos m´ as destapando los pl´ atanos
AnMTengineproducesthefollowingtranslation:
PlatanoinOrange For6people
3Bananas mature 2tablespoon meltedbutter
1cup juice (juice)orange 5tablespoons brownsugar or white
1/8teaspoon nutmeg powder 1tablespoon ralladura orange
1tablespoon cinnamon powder (optional)

Peelbananas, cutinhalfandthenalong. Grease asource orpi rexwithmargarine. Putbananas
andshowering themwiththe meltedbutter. Inadeepbowl, mix thejuice (juice)orange withthe
sugar, ginger, nutmegand ralladuraorange. Pourover banan as andbake to350◦F.The ﬁrst 15
minutes, leave covered bananas, bake 10 to15minutes more un covering bananas.
While there are still lots of confusionsin thistranslation (is it forbananasor plan-
tains? What exactly is the pot we should use? What is ralladura ?) it’s probably
enough,perhapsafterlookinguponeortwowords,togetabas icideaofsomethingto

tryinthe kitchenwithyournewpurchase!
AnMTsystemcanalsobeusedtospeed-upthehumantranslatio nprocess,bypro-

DRAFT876 Chapter 25. MachineTranslation
ducingadrafttranslationthatisﬁxedupina post-editing phasebyahumantranslator. Post-editing
Strictly speaking, systems used in this way are doing computer-aided human trans-
lation(CAHT or CAT) rather than (fully automatic) machine transla tion. This modelcomputer-aided
human translation
of MT usage is effective especially for high volume jobs and t hose requiring quick
turn-around,such as the translationof software manualsfo rlocalization to reach new localization
markets.
Weather forecasting is an example of a sublanguage domain that can be modeled sublanguage
completely enough to use raw MT output even without post-edi ting. Weather fore-
casts consist of phrases like Cloudy with a chance of showers today and Thursday ,
orOutlook for Friday: Sunny . This domain has a limited vocabulary and only a few
basic phrase types. Ambiguity is rare, and the senses of ambi guous words are easily
disambiguated based on local context, using word classes an d semantic features such

asWEEKDAY ,PLACE, orTIME POINT . Other domains that are sublanguage-like in-
cludeequipmentmaintenancemanuals,airtravelqueries,a ppointmentscheduling,and
restaurantrecommendations.
Applicationsfor machine translation can also be character ized by the number and
direction of the translations. Localization tasks like tra nslations of computer manuals
require one-to-many translation (from English into many la nguages). One-to-many
translation is also needed for non-English speakers around the world to access web

informationin English. Conversely,many-to-onetranslat ion (into English) is relevant
for anglophone readers who need the gist of web content writt en in other languages.
Many-to-manytranslationisrelevantforenvironmentslik etheEuropeanUnion,where
23ofﬁciallanguages(at thetimeofthiswriting)needtobei ntertranslated.
BeforeweturntoMTsystems,webegininsection25.1bysumma rizingkeydiffer-
ences among languages. The three classic models for doing MT are then presented in
Sec.25.2: the direct,transfer,andinterlingua approaches. Wetheninvestigateinde-

tail modern statisticalMT in Secs. 25.3-25.8,ﬁnishingin Sec. 25.9with a discussion
ofevaluation .
25.1 Why isMachineTranslationSo Hard?
We beganthis chapterwith some of the issues that made it hard to translate The Story
of the Stone from Chinese to English. In this section we look in more detai l about
what makes translation difﬁcult. We’ll discuss what makes l anguages similar or dif-
ferent, including systematic differences that we can model in a general way, as well
asidiosyncratic and lexical differences that must be dealt with one by one. Th ese

differences between languages are referred to as translation divergences and an un-Translationdivergence
derstanding of what causes them will help us in building mode ls that overcome the
differences(Dorr,1994).
25.1.1 Typology
Whenyouaccidentallypickuparadioprograminsomeforeign languageitseemslike
chaos, completely unlike the familiar languages of your eve ryday life. But there are

DRAFTSection25.1. Whyis MachineTranslationSoHard? 877
patternsinthischaos,andindeed,someaspectsofhumanlan guageseemtobe univer-
sal, holdingtrue foreverylanguage. Manyuniversalsarise fro mthefunctionalroleof Universal
languageasa communicativesystem byhumans. Everylanguag e,forexample,seems
tohavewordsforreferringtopeople,fortalkingaboutwome n,men,andchildren,eat-
inganddrinking,forbeingpoliteornot. Otheruniversalsa remoresubtle;forexample
Ch.5 mentionedthateverylanguageseemsto havenounsandve rbs.
Evenwhenlanguagesdiffer,thesedifferencesoftenhavesy stematicstructure. The
studyofsystematiccross-linguisticsimilaritiesanddif ferencesiscalled typology (Croft Typology
(1990), Comrie (1989)). This section sketches some typolog ical facts about crosslin-
guisticsimilarityanddifference.
Morphologically ,languagesareoftencharacterizedalongtwodimensionsof vari-
ation. The ﬁrst is the number of morphemes per word, ranging f romisolating lan- Isolating

guages like Vietnamese and Cantonese, in which each word gen erally has one mor-
pheme, to polysynthetic languageslike Siberian Yupik (“Eskimo”), in which a single Polysynthetic
wordmayhave verymanymorphemes,correspondingto a wholes entencein English.
The second dimension is the degree to which morphemes are seg mentable, ranging
fromagglutinative languages like Turkish (discussed in Ch. 3), in which morphe mes Agglutinative

have relatively clean boundaries, to fusionlanguages like Russian, in which a single Fusion
afﬁxmayconﬂatemultiplemorphemes,like -omintheword stolom,(table- SG-INSTR-
DECL1) which fuses the distinct morphologicalcategories instr umental, singular, and
ﬁrst declension.
Syntactically ,languagesare perhapsmost saliently differentin the basi c word or-
der of verbs, subjects, and objects in simple declarative cl auses. German, French,
English, and Mandarin, for example, are all SVO(Subject-Verb-Object ) languages, SVO

meaningthattheverbtendstocomebetweenthesubjectandob ject. HindiandJapanese,
bycontrast,are SOVlanguages,meaningthattheverbtendstocomeattheendofba sic SOV
clauses, while Irish, Arabic, and Biblical Hebrew are VSOlanguages. Two languages VSO
that share their basic word-ordertype often have other simi larities. For example SVO
languagesgenerallyhave prepositions whileSOVlanguagesgenerallyhave postposi-
tions.
For example in the following SVO English sentence, the verb adoresis followed
by its argument VP listening to music , the verb listening is followed by its argument

tions.
PPto music, and the preposition tois followed by its argument music. By contrast, in
the Japanese example which follows, each of these orderings is reversed; both verbs
arepreceded bytheirarguments,andthe postpositionfollowsitsargume nt.
(25.1)English: Headoreslisteningtomusic
Japanese: kare
hehaongaku
musicwo
tokiku
listeningnogadaisuki
adoresdesu
Another important dimension of typological variation has t o do with argument
structure andlinkingof predicates with their arguments, such as the difference b e-

musicwo
tokiku
tweenhead-marking anddependent-marking languages (Nichols, 1986). Head- Head-marking
marking languages tend to mark the relation between the head and its dependents on
the head. Dependent-marking languages tend to mark the rela tion on the non-head.
Hungarian, for example, marks the possessive relation with an afﬁx (A) on the head
noun(H),whereEnglishmarksit onthe(non-head)possessor :

DRAFT878 Chapter 25. MachineTranslation
(25.2)English:
Hungarian:the
az
theman-A’s
ember
manHhouse
Hh´az-Aa
house-his
Typologicalvariation in linking can also relate to how the c onceptual propertiesof an
event are mapped onto speciﬁc words. Talmy (1985)and (1991) noted that languages
canbecharacterizedbywhetherdirectionofmotionandmann erofmotionaremarked
ontheverboronthe“satellites”: particles,prepositiona lphrases,oradverbialphrases.
For example a bottle ﬂoating out of a cave would be described i n English with the
direction marked on the particle out, while in Spanish the direction would be marked
ontheverb:
(25.3)English: The bottleﬂoatedout.
Spanish: La
Thebotella
bottlesali´ o
exitedﬂotando.
ﬂoating.
Verb-framed languages mark the direction of motion on the verb (leaving t he Verb-framed
satellites to mark the manner of motion), like Spanish acercarse ‘approach’, alcan-
zar‘reach’,entrar‘enter’,salir‘exit’.Satellite-framed languagesmarkthe direction Satellite-framed

ﬂoating.
of motion on the satellite (leaving the verb to mark the manne r of motion), like En-
glishcrawlout ,ﬂoatoff,jumpdown ,walkoverto ,runafter. LanguageslikeJapanese,
Tamil, and the many languagesin the Romance, Semitic, and Ma yan languagesfami-
lies, are verb-framed;Chinese as well as non-RomanceIndo- Europeanlanguageslike
English,Swedish,Russian,Hindi,andFarsi,aresatellite -framed(Talmy,1991;Slobin,
1996).
Finally,languagesvaryalongatypologicaldimensionrela tedtothethingstheycan

1996).
omit. Many languages require that we use an explicit pronoun when talking about a
referentthat is givenin the discourse. In otherlanguages, however,we can sometimes
omit pronouns altogether as the following example from Span ish shows, using the / 0-
notationintroducedinCh.21:
(25.4)[Eljefe] idioconun libro. / 0 iMostr´ o aun descifrador ambulante.
[The boss] came upon a book. [He] showed ittoa wandering deco der.
Languageswhichcanomitpronounsinthesewaysarecalled pro-drop languages. Pro-drop

Even among the pro-drop languages, there are marked differe nces in frequencies of
omission. Japanese andChinese,forexample,tendto omitfa rmorethanSpanish. We
refer to this dimension as referentialdensity ; languageswhich tend to use more pro- Referentialdensity
nounsaremorereferentiallydensethanthosethatusemorez eros. Referentiallysparse
languages,likeChineseorJapanese,thatrequiretheheare rtodomoreinferentialwork
torecoverantecedentsarecalled coldlanguages. Languagesthataremoreexplicitand Coldlanguage

make it easier for the hearerare called hotlanguages. The terms hotandcoldare bor- Hot language
rowed from Marshall McLuhan’s (1964) distinction between h ot media like movies,
whichﬁll inmanydetailsfortheviewer,versuscoldmediali ke comics,whichrequire
thereaderto domoreinferentialworkto ﬁll outtherepresen tation(Bickel,2003).
Each typological dimension can cause problems when transla ting between lan-
guagesthatdifferalongthem. ObviouslytranslatingfromS VOlanguageslikeEnglish

toSOVlanguageslikeJapaneserequireshugestructuralreo rderings,sinceallthecon-
stituents are at different places in the sentence. Translat ing from a satellite-framed to

DRAFTSection25.1. Whyis MachineTranslationSoHard? 879
a verb-framed language, or from a head-marking to a dependen t-marking language,
requires changes to sentence structure and constraints on w ord choice. Languages
withextensivepro-drop,likeChineseorJapanese,causehu geproblemsfortranslation
into non-pro-droplanguages like English, since each zero h as to be identiﬁed and the
anaphorrecovered.
25.1.2 OtherStructural Divergences
Many structural divergencesbetween languages are based on typological differences.
Others, however, are simply idiosyncratic differences tha t are characteristic of partic-
ular languages or language pairs. For example in English the unmarked order in a
noun-phrasehasadjectivesprecedenouns,butinFrenchand Spanishadjectivesgener-
allyfollownouns.1
(25.5)Spanish bruja verde Frenchmaison bleue
witch green house blue
English “greenwitch” “bluehouse”
Chineserelativeclausesarestructuredverydifferentlyt hanEnglishrelativeclauses,
makingtranslationoflongChinese sentencesverycomplex.

Language-speciﬁc constructions abound. English, for exam ple, has an idiosyn-
cratic syntactic construction involving the word therethat is often used to introduce a
new scene in a story, as in there burst into the room three men with guns . To give an
idea ofhowtrivial, yetcrucial,these differencescan be,t hinkofdates. Datesnot only
appear in various formats — typically DD/MM/YY in British En glish, MM/DD/YY
in AmericanEnglish, and YYMMDD in Japanese—butthe calenda rsthemselvesmay

also differ. Dates in Japanese, for example,are often relat iveto the start of the current
Emperor’sreignratherthantothe startoftheChristian Era .
25.1.3 Lexical Divergences
Lexical divergencesalso cause huge difﬁculties in transla tion. We saw in Ch. 20, for
example, that the English source language word basscould appear in Spanish as the
ﬁshlubinaor the instrument bajo. Thus translation often requires solving the exact
sameproblemsaswordsensedisambiguation,andthetwo ﬁeld sarecloselylinked.

InEnglishtheword bassishomonymous;thetwosensesofthewordarenotclosely
related semantically, and so it is natural that we would have to disambiguate in order
to translate. Even in cases of polysemy, however, we often ha ve to disambiguate if
the target language doesn’t have the exact same kind of polys emy. The English word
know, for example, is polysemous; it can refer to knowing of a fact or proposition ( I
knowthat snowis white )orfamiliaritywith a personorlocation( IknowJonStewart ).

It turns out that translating these different senses requir esusing distinct French verbs,
including the verbs connaˆıtre, andsavoir.Savoiris generally used with sentential
complements to indicate knowledge or mental representatio n of a fact or proposition,
1As always, there are exceptions to this generalization, suc h asgalorein English and grosin French;
furthermore in French someadjectives can appear before the noun with adifferent meaning; routemauvaise

‘bad road, badly-paved road’ versus mauvaise route ‘wrong road’ (Waugh, 1976).

DRAFT880 Chapter 25. MachineTranslation
or verbal complements to indicate knowledge of how to do some thing (e.g., WordNet
3.0 senses #1, #2, #3). Connaˆıtreis generally used with NP complements to indicate
familiarityoracquaintancewithpeople,entities,orloca tions(e.g.,WordNet3.0senses
#4,#7). SimilardistinctionsoccurinGerman,Chinese,and manyotherlanguages:
(25.6)English: I knowhejust boughta book.
(25.7)French:Je saisqu’ilvientd’acheterunlivre.
(25.8)English: I knowJohn.
(25.9)French:Je connaisJean.
Thesavoir/conna ˆıtredistinctioncorrespondstodifferentgroupsofWordNetsen ses.
Sometimes, however, a target language will make a distincti on that is not even recog-
nized in ﬁne-grained dictionaries. German, for example, us es two distinct words for
whatinEnglishwouldbecalleda wall:Wandforwallsinsideabuilding,and Mauerfor
walls outside a building. Similarly, where English uses the wordbrotherfor any male
sibling, both Japanese and Chinese have distinct words for older brother andyounger

brother(Chinese gegeanddidi,respectively).
Inadditiontothesedistinctions,lexicaldivergencescan begrammatical. Forexam-
ple,awordmaytranslatebesttoadifferentpart-of-speech inthetargetlanguage. Many
Englishsentencesinvolvingtheverb likemustbetranslatedintoGermanusingthead-
verbialgern; thusshe likes to sing maps tosie singt gerne (SHE SINGS LIKINGLY ).
Intranslation,we can thinkof sense disambiguationasa kin dofspeciﬁcation ;we
havetomakeavaguewordlike knoworbassmorespeciﬁcinthetargetlanguage. This

kind of speciﬁcation is also quite common with grammatical d ifferences. Sometimes
onelanguageplacesmoregrammaticalconstraintsonwordch oicethananother. French
and Spanish, for example, marks gender on adjectives, so an E nglish translation into
Frenchrequiresspecifyingadjectivegender. Englishdist inguishesgenderinpronouns
where Mandarin does not; thus translating a third-person si ngular pronoun t¯afrom
MandarintoEnglish( he,she,orit) requiresdecidingwhotheoriginalreferentwas. In

Japanese,becausethereisnosinglewordfor is,thetranslatormustchoosebetween iru
oraru,basedonwhetherthe subjectisanimateornot.
The way that languages differ in lexically dividing up conce ptual space may be
more complex than this one-to-many translation problem, le ading to many-to-many
mappings. For example Fig. 25.2 summarizes some of the compl exities discussed by
HutchinsandSomers(1992)inrelatingEnglish leg,foot,andpaw,totheFrench jambe,
pied,patte ,etc.

Further, one language may have a lexical gap , where no word or phrase, short of Lexicalgap
anexplanatoryfootnote,canexpressthemeaningof awordin theotherlanguage. For
example,Japanesedoesnothaveawordfor privacy,andEnglishdoesnothaveaword
forJapanese oyakokoorChinese xi´ao(wemakedowiththeawkwardphrase ﬁlialpiety
forboth).

DRAFTSection25.2. Classical MT& theVauquoisTriangle 881
Figure25.2 The complex overlap between English leg,foot, etc, and various French transla-
tions like pattediscussed by Hutchins and Somers (1992) .
25.2 ClassicalMT& theVauquoisTriangle
The next few sections introduce the classical pre-statisti cal architectures for machine
translation. Real systems tend to involve combinations of e lements from these three
architectures; thus each is best thought of as a point in an al gorithmic design space
ratherthanasanactualalgorithm.
Indirecttranslation, we proceed word-by-wordthrough the source la nguage text,
translating each word as we go. Direct translation uses a lar ge bilingual dictionary,
each of whose entries is a small program with the job of transl ating one word. In
transferapproaches,weﬁrstparsetheinputtext,andthenapplyrule stotransformthe
source language parse structure into a target language pars e structure. We then gener-

ate the target language sentence from the parse structure. I ninterlingua approaches,
we analyzethe sourcelanguagetext intosome abstractmeani ngrepresentation,called
aninterlingua . We thengenerateintothe targetlanguagefromthisinterli ngualrepre-
A common way to visualize these three approaches is with Vauquois triangle Vauquoistriangle
shown in Fig. 25.3. The triangle shows the increasing depth o f analysis required (on
both the analysis and generation end) as we move from the dire ct approach through

transfer approaches, to interlingual approaches. In addit ion, it shows the decreasing
amountof transferknowledgeneededas we moveup the triangl e,from hugeamounts
of transfer at the direct level (almost all knowledge is tran sfer knowledge for each
word) through transfer (transfer rules only for parse trees or thematic roles) through
interlingua(nospeciﬁctransferknowledge).
Inthenextsectionswe’ll see howthesealgorithmsaddresss omeofthe fourtrans-
lationexamplesshownin Fig.25.2
25.2.1 DirectTranslation

Indirect translation , we proceed word-by-word through the source language text, Direct translation
translatingeachwordaswe go. We makeuse ofnointermediate structures,exceptfor
shallowmorphologicalanalysis;eachsourcewordisdirect lymappedontosometarget
word. Direct translationisthusbasedon a largebilinguald ictionary;eachentryin the

DRAFT882 Chapter 25. MachineTranslation
Figure25.3 The Vauquois (1968) triangle.
English Marydidn’t slapthe green witch
⇒Spanish Maria
Maryno
notdi´ o
gaveuna
abofetada
slapa
tola
thebruja
witchverde
green
English The greenwitchis athome this week
⇒German Diese
thisWoche
weekist
isdie
thegr¨ une
greenHexe
witchzu
atHause.
house
English He adores listeningtomusic
⇒Japanese kare
musicwo
tokiku
Chinese cheng long
Jackie Chandao
toxiang gang
Hong Kongqu
go
⇒English Jackie Chan went toHong Kong
Figure25.4 Example sentences used throughout the chapter.
dictionarycanbe viewedasa small programwhosejobis to tra nslateoneword. After
the words are translated, simple reordering rules can apply , for example for moving
adjectivesafternounswhentranslatingfromEnglishtoFre nch.
The guiding intuition of the direct approach is that we trans late by incrementally
transforming the source language text into a target language text. While t he pure

go
direct approach is no longer used, this transformationalin tuition underliesall modern
systems,bothstatistical andnon-statistical.
Figure25.5 Direct machine translation. The major component, indicate d by size here, is the
bilingual dictionary.
Let’s look at a simpliﬁed direct system on our ﬁrst example, t ranslating from En-

DRAFTSection25.2. Classical MT& theVauquoisTriangle 883
Input: Marydidn’tslap thegreenwitch
After1: Morphology MaryDO-PASTnotslap thegreenwitch
After2: LexicalTransfer MariaPAST nodarunabofetadaa lav erdebruja
After3: Localreordering MarianodarPAST unabofetadaa lab rujaverde
After4: Morphology Marianodi´ ounabofetadaa labrujaverd e
Figure25.6 Anexample of processing ina directsystem
glishintoSpanish:
(25.10) Marydidn’tslapthegreenwitch
Maria
Maryno
notdi´o
gaveuna
abofetada
slapa
tola
thebruja
green
Thefourstepsoutlinedin Fig.25.5wouldproceedasshownin Fig.25.6.
Step 2 presumes that the bilingual dictionary has the phrase dar una bofetada a
as the Spanish translation of English slap. The local reordering step 3 would need
to switch the adjective-noun ordering from green witch tobruja verde . And some
combination of ordering rules and the dictionary would deal with the negation and
past tense in English didn’t. These dictionary entries can be quite complex; a sample

slapa
tola
thebruja
green
dictionaryentryfromanearlydirectEnglish-Russiansyst emisshowninFig.25.7.
function DIRECTTRANSLATE MUCH/MANY(word)returnsRussiantranslation
ifpreceding wordis howreturnskol’ko
else ifpreceding word is asreturnstol’kozhe
else ifword ismuch
ifpreceding word is veryreturnnil
else iffollowingword isa noun returnmnogo
else/*word ismany */
ifpreceding word isa preposition and followingword isa noun returnmnogii
else return mnogo
Figure25.7 Aprocedurefortranslating muchandmanyintoRussian,adaptedfromHutchins’

(1986, pg. 133) discussion of Panov 1960. Note the similarit y to decision list algorithms for
wordsense disambiguation.
WhilethedirectapproachcandealwithoursimpleSpanishex ample,andcanhan-
dle single-word reorderings, it has no parsing component or indeed any knowledge
aboutphrasingorgrammaticalstructureinthesourceortar getlanguage. Itthuscannot
reliably handle longer-distancereorderings,or those inv olvingphrasesor largerstruc-
tures. Thiscan happeneven in languagesvery similar to Engl ish, like German,where

adverbs like heute(‘today’) occur in different places, and the subject (e.g., die gr¨une
Hexe)canoccurafterthemainverb,asshownin Fig.25.8.
Similar kinds of reorderings happen between Chinese (where goal PPs often oc-
cur preverbally) and English (where goal PPs must occur post verbally), as shown in
Fig.25.9.
Finally,evenmorecomplexreorderingsoccurwhenwetransl atefromSVOtoSOV
languages,asweseeintheEnglish-JapaneseexamplefromYa madaandKnight(2002):

DRAFT884 Chapter 25. MachineTranslation
Figure25.8 Complex reorderings necessary when translating from Engli sh to German. Ger-
man often puts adverbs in initial position that English woul d more naturally put later. German
tensed verbs often occur in second position in the sentence, causing the subject and verb to be
inverted.
Figure25.9 Chinese goal PPsoftenoccur preverbally, unlike inEnglish
.
(25.11)Headoreslisteningto music
kare
musicwo
tokiku
Thesethree examplessuggest that the directapproachis too focusedon individual
words, and that in order to deal with real examples we’ll need to add phrasal and
structural knowledge into our MT models. We’ll ﬂesh out this intuition in the next
section.
25.2.2 Transfer
As Sec. 25.1 illustrated, languagesdiffer systematically in structural ways. One strat-
egyfordoingMTistotranslatebya processofovercomingthe sedifferences,altering
the structure of the input to make it conform to the rules of th e target language. This

section.
can be done by applying contrastive knowledge , that is, knowledge about the differ-Contrastiveknowledge
ences between the two languages. Systems that use this strat egy are said to be based
onthetransfermodel . transfer model
The transfer model presupposes a parse of the source languag e, and is followed
by a generation phase to actually create the output sentence . Thus, on this model,
MT involvesthree phases: analysis,transfer, andgeneration ,wheretransfer bridges
the gap between the output of the source language parser and t he input to the target
languagegenerator.

ItisworthnotingthataparseforMTmaydifferfromparsesre quiredforotherpur-
poses. Forexample,supposeweneedtotranslate Johnsawthegirlwiththebinoculars
into French. The parser does not need to bother to ﬁgure out wh ere the prepositional
phraseattaches,becausebothpossibilitiesleadtothesam e Frenchsentence.
Once we have parsed the source language, we’ll need rules for syntactic transfer
andlexicaltransfer . Thesyntactictransferruleswill tell ushowtomodifythe s ource
parsetreetoresemblethetargetparsetree.

DRAFTSection25.2. Classical MT& theVauquoisTriangle 885
Nominal
Adj Noun⇒Nominal
Noun Adj
Figure25.10 A simple transformation thatreorders adjectives and nouns
Fig. 25.10 gives an intuition for simple cases like adjectiv e-noun reordering; we
transform one parse tree, suitable for describing an Englis h phrase, into another parse
tree, suitable for describinga Spanish sentence. These syntactic transformations are syntactictransformations
operationsthatmapfromonetree structuretoanother.
The transfer approach and this rule can be applied to our exam pleMary did not
slap the green witch . Besides this transformation rule, we’ll need to assume tha t the
morphologicalprocessingﬁguresoutthat didn’tiscomposedof do-PAST plusnot,and
that the parser attaches the PAST feature onto the VP. Lexica l transfer, via lookup in
the bilingual dictionary, will then remove do, changenottono, and turn slapinto the
phrasedar una bofetadaa , with a slight rearrangementof the parse tree, as suggested
inFig. 25.11.
VP[+PAST]
Neg
notVP

VP[+PAST]
Neg
notVP
V
slapNP
DT
Adj
greenNoun
witch⇒ VP[+PAST]
Neg
notVP
V
slapNP
DT
Noun
witchAdj
green⇒ VP[+PAST]
Neg
noVP
V
darNP
DT
unaNN
bofetadaPP
IN
aNP
DT
laNominal
Noun
brujaAdj
verde
Figure25.11 A further sketch of the transfer approach.
For translatingfrom SVO languageslike English to SOV langu ageslike Japanese,
we’llneedevenmorecomplextransformations,formovingth everbtotheend,chang-
ingprepositionsintopostpositions,andsoon. Anexampleo ftheresultofsuchrulesis
shownin Fig.25.12. Aninformalsketchofsometransferrule sisshownin Fig.25.13.

Neg
notVP
V
slapNP
DT
Noun
witchAdj
Neg
noVP
V
darNP
DT
unaNN
IN
aNP
DT
laNominal
Noun
brujaAdj
verde
Transfer systems can be based on richer structures than just pure syntactic parses.
ForexampleatransferbasedsystemfortranslatingChinese toEnglishmighthaverules
to deal with the fact shown in Fig. 25.9 that in Chinese PPs tha t ﬁll the semantic role
GOAL(liketo the store inI went to the store ) tend to appear before the verb, while in
English these goal PPs must appear after the verb. In order to build a transformation
to deal with this and related PP ordering differences, the pa rse of the Chinese must

including thematic structure, so as to distinguish BENEFACTIVE PPs (which must oc-
cur before the verb) from DIRECTION andLOCATIVE PPs (which preferentiallyoccur
before the verb) from RECIPIENT PPs (which occur after) (Li and Thompson, 1981).
We discussed how to do this kindof semantic role labeling in C h. 20. Using semantic
roles in this way is generally called semantic transfer ; a simple such transformation Semantictransfer

DRAFT886 Chapter 25. MachineTranslation
VB
PRP
HeVB1
adoresVB2
VB
listeningTO
TO
toNN
music⇒ VB
PRP
HeVB2
TO
NN
musicTO
toVB
listeningVB1
adores
Figure25.12 The result of Yamada and Knight’s (2001) syntactic transfor mations from En-
glish order (SVO) to Japanese order (SOV) for the sentence He adores listening to music (kare
ha ongaku wo kiku no ga daisuki desu ). This transform would require rules for moving verbs
after theirNPand VPcomplements, and changing preposition s topostpositions.
English toSpanish:
1. NP→Adjective 1Noun2⇒ NP→Noun2Adjective 1
Chinese toEnglish:
2. VP→PP[+Goal]V ⇒ VP→VPP[+Goal]
English toJapanese:
3. VP→V NP ⇒ VP→NP V
4. PP→P NP ⇒ PP→NP P
5. NP→NP1Rel. Clause 2⇒ NP→Rel. Clause 2NP1
Figure25.13 Aninformal description of some transformations.
isshowninFig.25.13.
In addition to syntactic transformations, transfer-based systems need to have lex-
ical transfer rules. Lexical transfer is generally based on a bilingual dictionary, just

as for direct MT. The dictionary itself can also be used to dea l with problems of lex-
ical ambiguity. For example the English word homehas many possible translations
in German, including nach Hause (in the sense of going home )Heim(in the sense of
ahome game ),Heimat(in the sense of homeland ,home country , orspiritual home ),
andzu Hause (in the sense of being at home). In this case, the phrase at homeis very
likelytobetranslated zuHause,andso thebilingualdictionarycanlist thistranslation
idiomatically.

Manycasesoflexicaltransferaretoocomplextodealwithvi aaphrasaldictionary.
Inthesecasestransfersystemscandodisambiguationdurin gthesourcelanguageanal-
ysis, byapplyingthesense disambiguationtechniquesofCh .20.
25.2.3 Combining direct andtranfer approaches inclassicM T
Although the transfer metaphor offers the ability to deal wi th more complex source
language phenomena than the direct approach, it turns out th e simple SVO→SOV
ruleswe’vedescribedabovearenotsufﬁcient. Inpractice, we needmessy ruleswhich

combinerichlexicalknowledgeofbothlanguageswithsynta cticandsemanticfeatures.
We brieﬂysawan exampleofsucha ruleforchanging slaptodarunabofetadaa .

DRAFTSection25.2. Classical MT& theVauquoisTriangle 887
For this reason, commercial MT systems tend to be combinatio ns of the direct
and transfer approaches, using rich bilingual dictionarie s, but also using taggers and
parsers. TheSystransystem,forexample,asdescribedinHu tchinsandSomers(1992),
Senellartet al. (2001),hasthreecomponents. First isa sha llowanalysisstage,includ-
ing:
•morphologicalanalysisandpart-of-speechtagging
•chunkingofNPs, PPs, andlargerphrases
•shallowdependencyparsing(subjects,passives,head-mod iﬁers)
Nextisa transferphase,including:
•translationofidioms,
•wordsense disambiguation
•assigningprepositionsbasedongoverningverbs
Finally,in the synthesis stage,thesystem:
•appliesarichbilingualdictionaryto dolexicaltranslati on
•dealswithreorderings
•performsmorphologicalgeneration
Thuslikethedirectsystem,theSystransystemreliesformu chofitsprocessingon
the bilingual dictionary, which has lexical, syntactic, an d semantic knowledge. Also

like a direct system, Systran does reordering in a post-proc essing step. But like a
transfer system, many of the steps are informed by syntactic and shallow semantic
processingofthesourcelanguage.
25.2.4 The Interlingua Idea: Using Meaning
One problem with the transfer model is that it requires a dist inct set of transfer rules
foreachpairoflanguages. Thisisclearlysuboptimalfortr anslationsystemsemployed
inmany-to-manymultilingualenvironmentsliketheEurope anUnion.

Thissuggestsadifferentperspectiveonthenatureoftrans lation. Insteadofdirectly
transforming the words of the source language sentence into the target language, the
interlingua intuition is to treat translation as a process o f extracting the meaning of
the input and then expressing that meaning in the target lang uage. If this could be
done, an MT system could do without contrastive knowledge, m erely relying on the
same syntactic and semantic rules used by a standardinterpr eterand generatorfor the

language. Theamountofknowledgeneededwouldthenbepropo rtionaltothenumber
oflanguagesthesystemhandles,ratherthanto thesquare.
Thisschemepresupposestheexistenceofameaningrepresen tation,orinterlingua , Interlingua
inalanguage-independentcanonicalform,likethesemanti crepresentationswesawin
Ch. 17. The idea is for the interlinguato represent all sente ncesthat mean the “same”
thing in the same way, regardlessof the language they happen to be in. Translation in

thismodelproceedsbyperformingadeepsemanticanalysiso ntheinputfromlanguage
X into the interlingual representation and generating from the interlingua to language
Y.
What kind of representation scheme can we use as an interling ua? The predicate
calculus,oravariantsuchasminimalrecursionsemantics, isonepossibility. Semantic

DRAFT888 Chapter 25. MachineTranslation

EVENT SLAPPING
AGENT MARY
TENSE PAST
POLARITY NEGATIVE
THEME
WITCH
DEFINITENESS DEF
ATTRIBUTES[
HAS-COLOR GREEN]


Figure25.14 Interlingual representation of Mary didnot slapthe green witch .
decompositioninto some kind of atomic semantic primitives is another. We will illus-
trate a third common approach, a simple event-based represe ntation, in which events
are linked to their arguments via a small ﬁxed set of thematic roles. Whether we use
logics or other representationsof events, we’ll need to spe cify temporal and aspectual
properties of the events, and we’ll also need to represent no n-eventive relationships
between entities, such as the has-color relation between greenandwitch. Fig. 25.2.4
showsa possibleinterlingualrepresentationfor Mary didnot slapthe greenwitch asa
uniﬁcation-stylefeaturestructure.
Wecancreatetheseinterlingualrepresentationfromtheso urcelanguagetextusing

thesemantic analyzer techniquesof Ch. 18 and Ch. 20; using a semantic role labeler
todiscoverthe AGENTrelationbetween Maryandtheslapevent,orthe THEMErelation
betweenthe witchandtheslapevent. We wouldalsoneedtododisambiguationofthe
noun-modiﬁer relation to recognize that the relationship b etweengreenandwitchis
thehas-color relation, and we’ll need to discover that this event has nega tive polarity
(from the word didn’t). The interlingua thus requires more analysis work than the
transfermodel,whichonlyrequiredsyntacticparsing(ora tmostshallowthematicrole

labeling). But generation can now proceed directly from the interlingua with no need
forsyntactictransformations.
Inadditiontodoingwithoutsyntactictransformations,th einterlingualsystemdoes
without lexical transfer rules. Recall our earlier problem of whether to translate know
into French as savoirorconnaˆıtre. Most of the processing involved in making this
decision is not speciﬁc to the goal of translating into Frenc h; German, Spanish, and
Chineseallmakesimilardistinctions,andfurthermorethe disambiguationof knowinto

concepts such as HAVE-A-PROPOSITION -IN-MEMORY andBE-ACQUAINTED -WITH-
ENTITYis also important for other NLU applications that require wo rd-senses. Thus
by using such concepts in an interlingua, a larger part of the translation process can
bedonewithgenerallanguageprocessingtechniquesandmod ules,andtheprocessing
speciﬁctotheEnglish-to-Frenchtranslationtask canbeel iminatedorat least reduced,
assuggestedinFig. 25.3.
The interlingual model has its own problems. For example, in order to trans-
late from Japanese to Chinese the universal interlingua mus t include concepts such

asELDER-BROTHER andYOUNGER -BROTHER . Using these same concepts translat-
ing fromGerman-to-Englishwould then require large amount sof unnecessarydisam-
biguation. Furthermore,doingthe extraworkinvolvedbyth e interlinguacommitment
requires exhaustive analysis of the semantics of the domain and formalization into

DRAFTSection25.3. Statistical MT 889
an ontology. Generally this is only possible in relatively s imple domains based on a
databasemodel,asintheairtravel,hotelreservation,orr estaurantrecommendationdo-
mains,wherethedatabasedeﬁnitiondeterminesthepossibl eentitiesandrelations. For
thesereasons,interlingualsystemsaregenerallyonlyuse din sublanguagedomains.
25.3 StatisticalMT
The three classic architectures for MT (direct, transfer, a nd interlingua) all provide
answers to the questions of what representations to use and w hat steps to perform to
translate. But there is anotherway to approachthe problemo f translation: to focuson
the result, not the process. Taking this perspective, let’s consider what it means for a
sentencetobea translationofsomeothersentence.
This is an issue to which philosophers of translation have gi ven a lot of thought.
Theconsensusseemstobe,sadly,thatitisimpossibleforas entenceinonelanguageto
beatranslationofasentenceinother,strictlyspeaking. F orexample,onecannotreally

translateHebrew adonairoi (‘theLordis myshepherd’)into thelanguageof a culture
that has no sheep. On the one hand, we can write something that is clear in the target
language,atsomecostinﬁdelitytotheoriginal,something liketheLordwilllookafter
me. On the other hand, we can be faithful to the original, at the c ost of producing
something obscure to the target language readers, perhaps l ikethe Lord is for me like
somebody who looks after animals with cotton-like hair . As another example, if we

translatethe Japanesephrase fukakuhanseishite orimasu ,aswe apologize ,we arenot
beingfaithfultothemeaningoftheoriginal,butifweprodu cewearedeeplyreﬂecting
(on our past behavior, and what we did wrong, and how to avoid t he problem next
time),thenouroutputisunclearorawkward. Problemssuchasthes earisenotonlyfor
culture-speciﬁcconcepts,but wheneveronelanguageuses a metaphor,a construction,
a word,oratense withoutanexactparallelintheotherlangu age.

So, true translation, which is both faithful to the source la nguage and natural as
an utterance in the target language, is sometimes impossibl e. If you are going to go
ahead and produce a translation anyway, you have to compromi se. This is exactly
whattranslatorsdoinpractice: theyproducetranslations thatdotolerablywellonboth
criteria.
ThisprovidesuswithahintforhowtodoMT.Wecanmodelthego aloftranslation
as the productionof an output that maximizes some value func tion that represents the

criteria.
importance of both faithfulness and ﬂuency. Statistical MT is the name for a class
of approaches that do just this, by building probabilistic m odels of faithfulness and
ﬂuency, and then combining these models to choose the most pr obable translation. If
wechosetheproductoffaithfulnessandﬂuencyasourqualit ymetric,wecouldmodel
thetranslationfromasourcelanguagesentence Stoa targetlanguagesentence ˆTas:
best-translation ˆT=argmaxTfaithfulness(T,S)ﬂuency(T)

ThisintuitiveequationclearlyresemblestheBayesian noisychannelmodel we’ve
seen in Ch. 5 for spelling and Ch. 9 for speech. Let’s make the a nalogy perfect and
formalizethenoisychannelmodelforstatistical machinet ranslation.

DRAFT890 Chapter 25. MachineTranslation
First of all, for the rest of this chapter, we’ll assume we are translating from a
foreign language sentence F=f1,f2,...,fmto English. For some examples we’ll use
Frenchas the foreignlanguage,and forothersSpanish. But i n each case we are trans-
latinginto English (althoughof coursethe statistical modelalso worksfor tra nslating
out of English). In a probabilistic model, the best English s entenceˆE=e1,e2,...,el
is the one whose probability P(E|F)is the highest. As is usual in the noisy channel
model,we canrewritethisviaBayesrule:
ˆE=argmax EP(E|F)
=argmax EP(F|E)P(E)
P(F)
=argmax EP(F|E)P(E) (25.12)
We canignorethedenominator P(F)insidetheargmaxsincewearechoosingthebest
English sentence for a ﬁxed foreign sentence F, and hence P(F)is a constant. The
resulting noisy channel equation shows that we need two comp onents: a translation
modelP(F|E), andalanguagemodel P(E). Translation model
Language model
ˆE=argmax
E∈Englishtranslationmodel
P(F|E)languagemodel

P(F)
ˆE=argmax
P(E) (25.13)
Notice that applying the noisy channel model to machine tran slation requires that
we think of things backwards, as shown in Fig. 25.15. We prete nd that the foreign
(source language) input Fwe must translate is a corrupted version of some English
(target language) sentence E, and that our task is to discover the hidden (target lan-
guage)sentence Ethatgeneratedourobservationsentence F.
ThenoisychannelmodelofstatisticalMTthusrequiresthre ecomponentstotrans-
late fromaFrenchsentence FtoanEnglishsentence E:

•Alanguagemodel tocompute P(E)
•Atranslationmodel tocompute P(F|E)
•Adecoder,whichisgiven Fandproducesthe mostprobable E
Ofthesethreecomponents,wehavealreadyintroducedthela nguagemodel P(E)in
Ch.4. StatisticalMTsystemsarebasedonthesame N-gramlanguagemodelsasspeech
recognition and other applications. The language model com ponent is monolingual,
andsoacquiringtrainingdataisrelativelyeasy.
The next few sections will thereforeconcentrateon the othe r two components,the
translationmodelandthedecodingalgorithm.

25.4P(F|E): the Phrase-BasedTranslationModel
Thejobofthetranslationmodel,givenanEnglishsentence Eandaforeignsentence F,
istoassignaprobabilitythat Egenerates F. Whilewecanestimatetheseprobabilities

DRAFTSection25.4. P(F|E): the Phrase-BasedTranslationModel 891
decoder
Mary did not slap...
Harry did not wrap...
...
Larry did not nap...guess at source:noisy 1
noisy 2
noisy NMary did not slap
the green witch.
Mary did not slap
the green witchMaria no dió una bofetada 
a la bruja verde
Language Model P(E) x Translation Model P(F|E)
Figure25.15 The noisy channel model of statistical MT. If we are translat ing a source lan-
guage French to a target language English, we have to think of ’sources’ and ’targets’ back-
wards. We build a model of the generation process from an Engl ish sentence through a channel
to a French sentence. Now given a French sentence to translat e, we pretend it is the output of
an English sentence going through the noisy channel, and sea rch for the best possible ‘source’
Englishsentence.
by thinking about how each individual word is translated, mo dern statistical MT is

basedontheintuitionthatabetterwaytocomputetheseprob abilitiesisbyconsidering
the behavior of phrases. As we see in Fig. 25.16, repeated from page 884, entire
phrasesoftenneedtobetranslatedandmovedasaunit. Thein tuitionof phrase-based Phrase-based
statistical MT is to use phrases (sequences of words) as well as single words as the
fundamentalunitsoftranslation.
Figure25.16 Phrasal reorderings necessary when generating German from English; repeated
from Fig.25.8.

There are a wide variety of phrase-based models; in this sect ion we will sketch
the model of Koehn et al. (2003). We’ll use a Spanish example, seeing how the
phrase-basedmodel computesthe probabilityP( Maria no di ´o una bofetadaa la bruja
verde|Marydidnotslapthegreenwitch ).
Thegenerativestoryofphrase-basedtranslationhasthree steps. Firstwegroupthe
Englishsourcewordsinto phrases ¯ e1,¯e2...¯eI. Next we translate each Englishphrase ¯ ei
intoaSpanishphrase ¯fj. FinallyeachoftheSpanishphrasesis(optionally)reorde red.

The probability model for phrase-based translation relies on atranslation proba-
bilityand adistortion probability . The factor φ(¯fj|¯ei)is the translation probability
ofgeneratingSpanishphrase ¯fjfromEnglishphrase ¯ ei. ThereorderingoftheSpanish

DRAFT892 Chapter 25. MachineTranslation
phrasesisdonebythe distortion probability d. Distortioninstatistical machinetrans- Distortion
lation refersto a wordhaving a different(‘distorted’)pos itionin the Spanish sentence
than it had in the English sentence; it is thus a measure of the distance between the
positions of a phrase in the two languages. The distortion pr obability in phrase-based
MTmeanstheprobabilityoftwoconsecutiveEnglishphrases beingseparatedinSpan-
ish by a span (of Spanish words) of a particular length. More f ormally, the distortion
isparameterizedby d(ai−bi−1), whereaiisthe start positionoftheforeign(Spanish)
phrasegeneratedbythe ithEnglishphrase ¯ ei,andbi−1istheendpositionoftheforeign
(Spanish)phrasegeneratedbythe i−1thEnglishphrase ¯ ei−1. Wecanuseaverysimple
distortionprobability,inwhichwesimplyraisesomesmall constantαtothedistortion.
d(ai−bi−1) =α|ai−bi−1−1|. Thisdistortionmodelpenalizeslargedistortionsbygivi ng
lowerandlowerprobabilitythelargerthedistortion.

Theﬁnal translationmodelforphrase-basedMTis:
P(F|E) =I
∏
i=1φ(¯fi,¯ei)d(ai−bi−1) (25.14)
Let’sconsiderthe followingparticularset ofphrasesforo urexamplesentences:2
Position 1 2 3 4 5
English Mary didnot slap the greenwitch
Spanish Maria no di´ ounabofetada a la brujaverde
Sinceeachphrasefollowsdirectlyinorder(nothingmovesa roundinthisexample,
unlike the German example in (25.16)) the distortions are al l 1, and the probability
P(F|E)canbe computedas:
P(F|E) =P(Maria,Mary)×d(1)×P(no|didnot )×d(1)×
P(di´ ounabofetada|slap)×d(1)×P(ala|the)×d(1)×

P(F|E) =I
∏
P(brujaverde|greenwitch )×d(1) (25.15)
In order to use the phrase-based model, we need two more thing s. We need a
model of decoding , so we can go from a surface Spanish string to a hidden English
string. Andwe needa modelof training,sowe can learnparameters. We’ll introduce
thedecodingalgorithminSec. 25.8. Let’sturnﬁrst totrain ing.
Howdowelearnthesimplephrase-basedtranslationprobabi litymodelin(25.14)?
The main set of parameters that needs to be trained is the set o f phrase translation

probabilities φ(¯fi,¯ei).
Theseparameters,aswellasthedistortionconstant α,couldbesetifonlywehada
largebilingualtrainingset,inwhicheachSpanishsentenc ewaspairedwithanEnglish
sentence, and if furthermore we knew exactly which phrase in the Spanish sentence
was translated by which phrase in the English sentence. We ca ll such a mapping a
phrase alignment . Phrase alignment
2Exactly which phrases we use depends on which phrases are dis covered in the training process, as de-

scribed in Sec. 25.7; thus for example if we don’t see the phra segreen witch in our training data, we would
have to translate greenandwitchindependently.

DRAFTSection25.4. P(F|E): the Phrase-BasedTranslationModel 893
Thetableofphrasesaboveshowedanimplicitalignmentofth ephrasesforthissen-
tence, forexample green witch alignedwith brujaverde . If we hada largetrainingset
with each pair of sentences labeled with such a phrase alignm ent, we could just count
thenumberoftimeseachphrase-pairoccurred,andnormaliz etogetprobabilities:
φ(¯f,¯e) =count(¯f,¯e)
∑¯fcount(¯f,¯e)(25.16)
We could store each phrase pair (¯f,¯e), together with its probability φ(¯f,¯e), in a
largephrase translationtable .Phrase translationtable
Alas,wedon’thavelargehand-labeledphrase-alignedtrai ningsets. Butitturnsout
that we can extract phrases from another kind of alignment ca lled aword alignment . Word alignment
Awordalignmentisdifferentthanaphrasealignment,becau seitshowsexactlywhich
Spanish word aligns to which English word inside each phrase . We can visualize a
word alignment in various ways. Fig. 25.17 and Fig. 25.18 sho w a graphical model

andanalignmentmatrix,respectively,fora wordalignment .
Figure25.17 A graphical model representation of a wordalignment betwee n the Englishand
Spanishsentences. We willsee laterhow toextract phrases..
Figure25.18 An alignment matrix representation of a word alignment betw een the English
and Spanishsentences. Wewillsee laterhow toextract phras es..
The next section introduces a few algorithms for deriving wo rd alignments. We
then show in Sec. 25.7 how we can extract a phrase table from wo rd alignments, and

ﬁnallyinSec. 25.8howthe phrasetablecanbeusedin decodin g.

DRAFT894 Chapter 25. MachineTranslation
25.5 Alignmentin MT
All statistical translation models are based on the idea of a word alignment . A word Word alignment
alignment is a mapping between the source words and the targe t words in a set of
parallelsentences.
Fig.25.19showsavisualizationofanalignmentbetweenthe Englishsentence And
theprogramhasbeenimplemented andtheFrenchsentence Leprogrammea ´et´emisen
application . Fornow,weassumethatwealreadyknowwhichsentencesinth eEnglish
textalignswith whichsentencesinthe Frenchtext.
Figure25.19 An alignment between an English and a French sentence, after Brown et al.
(1993). EachFrenchword aligns toa single Englishword.
In principle, we can have arbitrary alignment relationship s between the English
and French word. But the word alignment models we will presen t (IBM Models 1
and 3 and the HMM model) make a more stringent requirement, wh ich is that each
French word comes fromexactly one English word; this is cons istent with Fig. 25.19.

One advantage of this assumption is that we can represent an a lignment by giving the
index number of the English word that the French word comes fr om. We can thus
represent the alignment shown in Fig. 25.19 as A=2,3,4,5,6,6,6. This is a very
likelyalignment. Averyunlikelyalignment,bycontrast,m ightbeA=3,3,3,3,3,3,3.
We will makeoneadditiontothisbasicalignmentidea,which isto allowwordsto
appearintheforeignsentencethatdon’taligntoanywordin theEnglishsentence. We

model these words by assuming the existence of a NULL English worde0at position
0. Words in the foreign sentence that are not in the English se ntence, called spurious
words,maybegeneratedby e0. Fig.25.20showsthealignmentofspuriousSpanish a Spurious word
toEnglishNULL.3
Figure25.20 The alignment of the spurious Spanishword atothe EnglishNULLword e0.
While the simpliﬁed model of alignment above disallows many -to-one or many-
to-manyalignments,wewilldiscussmorepowerfultranslat ionmodelsthatallowsuch

alignments. Here are two such sample alignments; in Fig. 25. 21 we see an alignment
3While this particular amight instead be aligned to English slap, there are many cases of spurious words
which have no other possible alignment site.

DRAFTSection25.5. Alignmentin MT 895
which is many-to-one; each French word does not align to a sin gle English word, al-
thougheachEnglishworddoesaligntoa singleFrenchword.
Figure25.21 An alignment between anEnglish anda Frenchsentence, inwhi ch each French
word does not align toa single Englishword, but each English wordaligns toone French word.
Adapted from Brownet al.(1993).
Fig.25.22showsanevenmorecomplexexample,inwhichmulti pleEnglishwords
don’t have any money jointly align to the French words sont d´emunis. Suchphrasal
alignments will be necessary for phrasal MT, but it turns out they can’t b e directly
generatedbytheIBM Model1,Model3,orHMMwordalignmental gorithms.
Figure25.22 An alignment between an English and a French sentence, in whi ch there is a
many-to-manyalignmentbetweenEnglishandFrenchwords. A daptedfromBrownetal.(1993).
25.5.1 IBM Model 1
We’ll describe two alignment models in this section: IBM Mod el 1 and the HMM

model (we’ll also sketch the fertility-based IBM Model 3 in t he advanced section).
Botharestatisticalalignment algorithms. Forphrase-basedstatisticalMT,weusethe
alignmentalgorithmsjust toﬁndthebest alignmentforasen tencepair (F,E),inorder
to help extract a set of phrases. But it is also possible to use these word alignment
algorithms as a translation model P(F,E)as well. As we will see, the relationship
betweenalignmentandtranslationcanbeexpressedasfollo ws:
P(F|E) =∑
AP(F,A|E)

P(F|E) =∑
AP(F,A|E)
We’ll start with IBM Model 1, so-called because it is the ﬁrst and simplest of ﬁve
modelsproposedbyIBM researchersin aseminalpaper(Brown etal., 1993).
Here’s the general IBM Model 1 generative story for how we gen erate a Spanish
sentencefromanEnglishsentence E=e1,e2,...,eIoflength I:
1. Choosea length Jforthe Spanishsentence,henceforth F=f1,f2,...,fJ.
2. Now choose an alignment A=a1,a2,...,aJbetween the English and Spanish

DRAFT896 Chapter 25. MachineTranslation
3. Now for each position jin the Spanish sentence, chose a Spanish word fjby
translatingtheEnglishwordthatisalignedtoit.
Fig.25.23showsavisualizationofthisgenerativeprocess .
Figure25.23 The three steps of IBM Model 1 generating a Spanish sentence a nd alignment
from anEnglishsentence.
Let’s see how this generative story assigns a probability P(F|E)of generating the
Spanishsentence FfromtheEnglishsentence E. We’ll usethisterminology:
•eajistheEnglishwordthatis alignedtotheSpanishword fj.
•t(fx,ey)istheprobabilityoftranslating eybyfx(i.e.P(fx|ey)
We’llworkourwaybackwardsfromstep3. Sosupposewealread yknewthelength
Jandthealignment A, aswell astheEnglishsource E. TheprobabilityoftheSpanish
sentencewouldbe:
P(F|E,A) =J
∏
j=1t(fj|eaj) (25.17)
Now let’s formalize steps 1 and 2 of the generative story. Thi s is the probability
P(A|E)of an alignment A(of length J) given the English sentence E. IBM Model 1

∏
makes the (very) simplifying assumption that each alignmen t is equally likely. How
many possible alignments are there between an English sente nce of length Iand a
Spanish sentence of length J? Again assuming that each Spanish word must come
from one of the IEnglish words (or the 1 NULL word), there are (I+1)Jpossible
alignments. Model 1 also assumes that the probability of cho osing length Jis some
small constant ǫ. The combined probability of choosing a length Jand then choosing
anyparticularoneofthe (I+1)Jpossiblealignmentsis:
P(A|E) =ǫ

P(A|E) =ǫ
(I+1)J(25.18)
We cancombinetheseprobabilitiesasfollows:
P(F,A|E) =P(F|E,A)×P(A|E)

DRAFTSection25.5. Alignmentin MT 897
=ǫ
(I+1)JJ
∏
j=1t(fj|eaj) (25.19)
This probability, P(F,A|E), is the probability of generating a Spanish sentence F
via a particular alignment. In order to compute the total pro babilityP(F|E)of gener-
atingF, wejust sumoverall possiblealignments:
P(F|E) =∑
AP(F,A|E)
=∑
Aǫ
(I+1)JJ
∏
j=1t(fj|eaj) (25.20)
Eq.25.20showsthegenerativeprobabilitymodelforModel1 ,asitassignsaprob-
abilitytoeachpossibleSpanishsentence.
Inorderto ﬁndthe bestalignmentbetweena pair ofsentences FandE, we needa
way todecodeusing this probabilistic model. It turnsout there is a very s imple poly-
nomialalgorithmforcomputingthebest(Viterbi)alignmen twithModel1,becausethe
best alignment for each word is independent of the decision a bout best alignments of
thesurroundingwords:
ˆA=argmax
AP(F,A|E)
=argmax
Aǫ
(I+1)JJ
∏
j=1t(fj|eaj)
=argmax
ajt(fj|eaj)1<j<J (25.21)
TrainingforModel1isdonebytheEMalgorithm,whichwewill coverinSec.25.6.
25.5.2 HMMAlignment

ˆA=argmax
AP(F,A|E)
=argmax
Aǫ
(I+1)JJ
∏
=argmax
Now that we’ve seen Model 1, it should be clear that it makes so me really appalling
simplifying assumptions. One of the most egregious is the as sumption that all align-
ments are equally likely. One way in which this is a bad assump tion is that align-
ments tend to preserve locality; neighboring words in English are often aligned with
neighboring words in Spanish. If we look back at the Spanish/ English alignment in
Fig. 25.17, for example, we can see that this locality in the n eighboring alignments.

The HMM alignment model capturesthis kind of locality by con ditioningeach align-
mentdecisiononpreviousdecisions. Let’ssee howthiswork s.
TheHMMalignmentmodelis basedonthefamiliarHMMmodelwe’ venowseen
in many chapters. As with IBM Model 1, we are trying to compute P(F,A|E). The
HMM model is based on a restructuring of this probability usi ng the chain rule as
follows:
P(fJ
1,aJ
1|eI
1) =P(J|eI
1)×J
∏
j=1P(fj,aj|fj−1
1,aj−1
1,eI
1)

DRAFT898 Chapter 25. MachineTranslation
=P(J|eI
1)×J
∏
j−1P(aj|fj−1
1,aj−1
1,eI
1)×P(fj|fj−1
1,aj
1,eI
1) (25.22)
Via this restructuring,we can think of P(F,A|E)as being computablefrom proba-
bilitiesofthreetypes: alengthprobability P(J|eI
1), analignmentprobability
P(aj|fj−1
1,aj−1
1,eI
1),anda lexiconprobability P(fj|fj−1
1,aj
1,eI
1).
We next make some standard Markovsimplifyingassumptions. We’ll assume that
the probability of a particular alignment ajfor Spanish word jis only dependent on
thepreviousalignedposition aj−1. We’ll also assumethatthe probabilityofaSpanish
wordfjisdependentonlyonthealignedEnglishword eajat position aj:
P(aj|fj−1
1,aj−1
1,eI
1) =P(aj|aj−1,I) (25.23)
P(fj|fj−1
1,aj
1,eI
1) =P(fj|eaj) (25.24)
Finally,we’llassumethatthelengthprobabilitycanbeapp roximatedjustas P(J|I).
Thusthe probabilisticmodelforHMM alignmentis:
P(fJ
1,aJ
1|eI
1) =P(J|I)×J
∏
j=1P(aj|aj−1,I)P(fj|eaj) (25.25)
To get the total probability of the Spanish sentence P(fJ
1|eI
1)we need to sum over
all alignments:
P(fJ

P(aj|fj−1
1,aj−1
1,eI
P(fj|fj−1
1,aj
1,eI
P(fJ
1,aJ
1|eI
∏
1|eI
P(fJ
1|eI
1) =P(J|I)×∑
AJ
∏
j=1P(aj|aj−1,I)P(fj|eaj) (25.26)
As we suggested at the beginning of the section, we’ve condit ionedthe alignment
probability P(aj|aj−1,I)onthepreviousalignedword,tocapturethelocalityofalig n-
ments. Let’s rephrasethis probabilityfor a momentas P(i|i′,I), whereiwill stand for
theabsolutepositionsintheEnglishsentenceofconsecuti vealignedstatesintheSpan-
ishsentence. We’dliketomaketheseprobabilitiesdepende ntnotontheabsoluteword
positions iandi′, but rather on the jump width between words; the jump width is the Jumpwidth

distancebetweentheirpositions i′−i. Thisisbecauseourgoalistocapturethefactthat
‘the English words that generate neighboring Spanish words are likely to be nearby’ .
Wethusdon’twanttobekeepingseparateprobabilitiesfore achabsolutewordposition
likeP(7|6,15)andP(8|7,15). Instead,we computealignmentprobabilitiesbyusinga
non-negativefunctionofthejumpwidth:
P(i|i′,I) =c(i−i′)
∑I
i′′=1c(i′′−i′)(25.27)
Let’s see how this HMM model gives the probability of a partic ular alignment of

∑I
ourEnglish-Spanishsentences;we’vesimpliﬁedthesenten ceslightly.
Thus the probability P(F,A|E)for this particular alignment of our simpliﬁed sen-
tenceMariadi´ounabofetadaalabrujaverde istheproductof:

DRAFTSection25.6. TrainingAlignmentModels 899
Figure25.24 The HMM alignment model generating from Mary slappped the green witch ,
showing the alignment and lexicon components of the probabi lityP(F,A|E)for this particular
alignment.
P(F,A|E) =P(J|I)×P(Maria|Mary)×P(2|1,5)×
t(di´o|slapped )×P(2|2,5)×T(una|slapped )×P(2|2,5)×... (25.28)
There are also more sophisticated augmentations to the basi c HMM alignment
model. TheseincludeaddingNULLwordsin theEnglishsource whichcanbeusedto
alignwithSpanishwordsthatdon’talignwithEnglishwords ,orconditioningthealign-
ment onC(eaj−1), the word class of the precedingtarget word: P(aj|aj−1,I,C(eaj−1))
(OchandNey,2003;Toutanovaet al.,2002).
ThemainadvantageoftheHMMalignmentmodelisthattherear ewell-understood
algorithmsbothfordecodingandfortraining. Fordecoding ,wecanusetheViterbial-
gorithmintroducedinCh.5andCh.6toﬁndthebest(Viterbi) alignmentforasentence
pair(F,E). Fortraining,wecanuse theBaum-Welchalgorithm,assumma rizedin the
25.6 TrainingAlignmentModels

All statistical translationmodels are trained using a larg eparallel corpus . Aparallel
corpus,paralleltext ,orbitextisatextthatisavailableintwolanguages. Forexample, Parallelcorpus
Bitexttheproceedingsofthe CanadianparliamentarekeptinbothF renchandEnglish. Each
sentence spoken in parliament is translated, producing a vo lume with running text in
bothlanguages. These volumesare called Hansards ,after the publisherofthe British Hansards
parliamentary proceedings. Similarly, the Hong Kong Hansards corpus contains theHong Kong
Hansards

Hansards
proceedingsof the HongKong SAR LegislativeCouncil in both English and Chinese.
Both of these corpora contain tens to hundreds of millions of words. Other parallel
corporahavebeenmadeavailablebytheUnitedNations. Itis possibletomakeparallel
corpora out of literary translations, but this is less commo n for MT purposes, partly
because it is difﬁcult to acquire the legal rights to ﬁction, but mainly because, as we
saw at the beginningof the chapter,translating ﬁction is ve rydifﬁcult andtranslations

Hansards
arenotveryliteral. Thusstatisticalsystemstendtobetra inedonveryliteraltranslations
suchasHansards.
Theﬁrststepintrainingistosegmentthecorpusintosenten ces. Thistaskiscalled
sentencesegmentation orsentencealignment . ThesimplestmethodsalignsentencesSentencesegmentation
Sentencealignment

DRAFT900 Chapter 25. MachineTranslation
based purely on their lengthin words or characters, without lookingat the contentsof
the words in the sentences. The intuition is that if we see a lo ng sentence in roughly
the same position in each language of the parallel text, we mi ght suspect these sen-
tencesaretranslations. Thisintuitioncanbe implemented bya dynamicprogramming
algorithm. More sophisticated algorithms also make use of i nformation about word
alignments. Sentencealignmentalgorithmsarerunonapara llelcorpusbeforetraining
MTmodels. Sentenceswhichdon’taligntoanythingarethrow nout,andtheremaining
alignedsentencescan be used as a trainingset. See the endof the chapterfor pointers
tomoredetailsonsentencesegmentation.
Once we have done sentence alignment, the input to our traini ng algorithm is a
corpus consisting of Ssentence pairs{(Fs,Es):s=1...S}. For each sentence pair
(Fs,Es)thegoalisto learnanalignment A=aJ
1andthecomponentprobabilities( tfor

Model1,andthelexiconandalignmentprobabilitiesforthe HMM model).
25.6.1 EM forTraining AlignmentModels
Ifeachsentencepair (Fs,Es)wasalreadyhand-labeledwithaperfectalignment,learn-
ingtheModel1orHMMparameterswouldbetrivial. Forexampl e,togetamaximum
likelihoodestimatesinModel1forthetranslationprobabi lityt(verde,green),wewould
just count the number of times greenis aligned to verde, and normalize by the total
countofgreen.
But of coursewe don’tknowthe alignmentsin advance;all we h aveare the prob-

abilitiesof each alignment. Recall that Eq˙ 25.19 showed that if we alr eady had
good estimates for the Model 1 tparameter, we could use this to compute probabil-
itiesP(F,A|E)for alignments. Given P(F,A|E), we can generatethe probabilityof an
alignmentjust bynormalizing:
P(A|E,F) =P(A,F|E)
∑AP(A,F|E)
So, if we hada roughestimate of the Model1 tparameters,we couldcomputethe
probabilityfor each alignment. Then instead of estimating thetprobabilitiesfrom the

(unknown)perfect alignment, we would estimate them from ea ch possible alignment,
andcombinetheseestimatesweightedbytheprobabilityofe achalignment. Forexam-
ple if there were two possible alignments, one of probabilit y.9 and one of probability
.1, we would estimate the tparameters separately from the two alignments and mix
thesetwo estimateswithweightsof.9and.1.
Thusif we had model 1 parametersalready, we could re-estimate the parameters,
by using the parameters to compute the probability of each po ssible alignment, and

thenusingtheweightedsumofalignmentstore-estimatethe model1parameters. This
idea of iterativelyimprovingourestimates of probabiliti esis a special case of the EM
algorithm that we introduced in Ch. 6, and that we saw again for speech re cognition
in Ch. 9. Recall that we use the EM algorithm when we have a vari able that we can’t
optimizedirectlybecauseitis hidden. Inthiscasethehiddenvariableisthealignment.
ButwecanusetheEMalgorithmtoestimatetheparameters,co mputealignmentsfrom
theseestimates, usethe alignmentstore-estimatethepara meters,andso on!

DRAFTSection25.6. TrainingAlignmentModels 901
Let’swalkthroughan exampleinspiredbyKnight(1999b),us inga simpliﬁed ver-
sionofModel1,inwhichweignoretheNULLword,andweonlyco nsiderasubsetof
thealignments(ignoringalignmentsforwhichanEnglishwo rdalignswithnoSpanish
word). Hencewecomputethesimpliﬁedprobability P(A,F|E)asfollows:
P(A,F|E) =J
∏
j=1t(fj|eaj) (25.29)
The goal of this example is just to give an intuition of EM appl ied to this task; the
actualdetailsofModel1trainingwouldbesomewhatdiffere nt.
The intuition of EM training is that in the E-step, we compute expected counts
for thetparameterbased on summingover the hidden variable (the ali gnment), while
in the M-step, we computethe maximumlikelihoodestimate of thetprobabilityfrom
thesecounts.
Let’sseeafewstagesofEMtrainingofthisparameteronacor pusoftwosentences:
green house the house
casa verde la casa
Thevocabulariesforthetwolanguagesare E={green,house,the}andS={casa,la,verde}.
We’ll start withuniformprobabilities:
t(casa|green)=1
3t(verde|green)=1

3t(la|green)=1
3
t(casa|house)=1
3t(verde|house)=1
3t(la|house)=1
3
t(casa|the)=1
3t(verde|the)=1
3t(la|the)=1
3
Nowlet’swalkthroughthestepsofEM:
E-step 1: Computetheexpectedcounts E[count(t(f,e))]forall wordpairs (fj,eaj)
E-step1a: We ﬁrst needtocompute P(a,f|e), bymultiplyingall the tprobabilities,following
Eq.25.29
green house green house the house the house
casa verde casa verde la casa la casa
P(a,f|e) =t(casa,green) P(a,f|e) =t(verde,green) P(a,f|e) =t(la,the) P(a,f|e) =t(casa,the)
×t(verde,house) ×t(casa,house) ×t(casa,house) ×t(la,house)
=1
3×1
3=1
9=1
3×1
3=1
9=1
3×1
3=1

3
Eq.25.29
=1
3×1
3=1
9=1
3×1
3=1
9=1
3×1
3=1
9=1
3×1
3=1
9
E-step 1b: Normalize P(a,f|e)to getP(a|e,f), usingthefollowing:
P(a|e,f) =P(a,f|e)
∑aP(a,f|e)
Theresultingvaluesof P(a|f,e)foreachalignmentare asfollows:
P(a|f,e) =1/9
2/9=1
2P(a|f,e) =1/9
2/9=1
2/9=1
2/9=1
2

DRAFT902 Chapter 25. MachineTranslation
E-step 1c: Computeexpected(fractional)counts,byweightingeachco untbyP(a|e,f)
tcount(casa|green)=1
2tcount(verde|green)=1
2tcount(la|green)=0total(green) =1
tcount(casa|house)=1
2+1
2tcount(verde|house)=1
2tcount(la|house)=1
2total(house) =2
tcount(casa|the)=1
2tcount(verde|the)=0tcount(la|the)=1
2total(the) =1
M-step 1: ComputetheMLEprobabilityparametersbynormalizingthe t countstosum toone.
t(casa|green)=1/2
1=1
2t(verde|green)=1/2
1=1
2t(la|green)=0
1=0
2=1
2t(verde|house)=1/2
2=1
4t(la|house)=1/2
2=1
4
t(casa|the)=1/2
1=1
2t(verde|the)=0
1=0t(la|the)=1/2
1=1
2
Notethateachofthecorrecttranslationshaveincreasedin probabilityfromtheini-
tialassignment;forexamplethetranslation casaforhousehasincreasedinprobability
from1
3to1
2.
E-step2a: We re-compute P(a,f|e), againbymultiplyingall the tprobabilities,following
Eq.25.29

1=1
1=0
2=1
2=1
2=1
4
1=1
1=1
2
from1
3to1
2.
Eq.25.29
=1
2×1
4=1
8=1
2×1
2=1
4=1
2×1
2=1
4=1
2×1
4=1
8
Note that the two correct alignments are now higher in probab ility than the two
incorrectalignments. Performingthesecondandfurtherro undofE-stepsandM-steps
isleft asExercise6forthereader.
WehaveshownthatEMcanbeusedtolearntheparametersforas impliﬁedversion

=1
2×1
4=1
8=1
2×1
2=1
4=1
2×1
2=1
4=1
2×1
4=1
8
of Model 1. Our intuitive algorithm, however,requiresthat we enumerateall possible
alignments. Fora longsentence,enumeratingeverypossibl ealignmentwouldbe very
inefﬁcient. Luckilyin practice thereis a veryefﬁcient ver sionof EM forModel 1 that
efﬁcientlyandimplicitlysumsoverall alignments.
WealsouseEM,intheformoftheBaum-Welchalgorithm,forle arningtheparam-
etersofthe HMMmodel.
25.7 SymmetrizingAlignmentsforPhrase-basedMT
ThereasonwhyweneededModel1orHMMalignmentswastobuild wordalignments

onthetrainingset, sothat wecouldextractalignedpairsof phrases.
Unfortunately, HMM (or Model 1) alignments are insufﬁcient for extracting pair-
ings of Spanish phrases with English phrases. This is becaus e in the HMM model,
each Spanish word must be generated from a single English wor d; we cannot gen-
erate a Spanish phrase from multiple English words. The HMM m odel thus cannot

DRAFTSection25.7. SymmetrizingAlignmentsforPhrase-basedMT 903
align a multiword phrase in the source languagewith a multiw ord phrase in the target
language.
We can,however,extendtheHMM modelto producephrase-to-p hrasealignments
for a pair of sentences (F,E), via a method that’s often called symmetrizing . First, Symmetrizing
we train two separate HMM aligners, an English-to-Spanisha ligner anda Spanish-to-
English aligner. We then align ( F,E) using both aligners. We can then combine these
alignmentsincleverwaysto getanalignmentthatmapsphras estophrases.
To combine the alignments, we start by taking the intersection of the two align- Intersection
ments, as shown in Fig. 25.25. The intersection will contain only places where the
two alignmentsagree,hencethe high-precisionalignedwor ds. We can also separately
computethe unionofthesetwoalignments. Theunionwillhavelotsoflessaccu rately
alignedwords. We canthenbuildaclassiﬁertoselectwordsf romtheunion,whichwe

incrementallyaddbackin tothisminimalintersectivealig nment.
Figure25.25 Intersection of English-to-Spanish and Spanish-to-Engli sh alignments to pro-
duce a high-precision alignment. Alignment can then be expa nded with points from both align-
ments toproduce an alignment like that shown inFig.25.26. A fterKoehn (2003b).
Fig. 25.26 shows an example of the resulting word alignment. Note that it does
allow many-to-onealignmentsin bothdirections. We can now harvestall phrase pairs

that are consistent with this word alignment. A consistent p hrase pair is one in which
allthewordsarealignedonlywitheachother,andnottoanye xternalwords. Fig.25.26
also showssomephrasesconsistentwith thealignment.
Once we collect all the aligned phrases pairs from the entire training corpus, we
cancomputethemaximumlikelihoodestimateforthephraset ranslationprobabilityof

DRAFT904 Chapter 25. MachineTranslation
(Maria,Mary), (no, did not),
(slap,di´ ouna bofetada), (verde, green),
(ala, the),(bruja, witch),
(Maria no, Marydidnot),
(nodi´ ouna bofetada, didnot slap),
(di´ ouna bofetada a la,slapthe),
(bruja verde, green witch),
(ala bruja verde, the green witch),...
Figure25.26 A better phrasal alignment for the green witch sentence, computed by starting
withthe intersection alignment inFig. 25.25 and adding poi nts from the union alignment, using
the algorithm of Och and Ney (2003). On the right, some of the p hrases consistent with this
alignment, after Koehn (2003b).
a particularpairasfollows:
∑¯fcount(¯f,¯e)(25.30)
Wecannowstoreeachphrase (¯f,¯e),togetherwithitsprobability φ(¯f,¯e),inalarge
phrase translation table . The decoding algorithm discussed in the next section canPhrase translationtable
usethisphrasetranslationtabletocomputethetranslatio nprobability.
25.8 DecodingforPhrase-BasedStatisticalMT

The remaining component of a statistical MT system is the dec oder. Recall that the
job of the decoder is to take a foreign (Spanish) source sente nceFand produce the
best (English) translation Eaccording to the product of the translation and language
models:
ˆE=argmax
P(E) (25.31)
Finding the sentence which maximizes the translation and la nguage model proba-
bilitiesisa searchproblem,anddecodingisthusakindofsearch. DecodersinMT are

models:
ˆE=argmax
based on best-ﬁrst search , a kind of heuristic orinformed search ; these are search
algorithmsthatareinformedbyknowledgefromtheproblemd omain. Best-ﬁrstsearch
algorithmsselectanode ninthesearchspacetoexplorebasedonanevaluationfunctio n
f(n). MTdecodersarevariantsof a speciﬁc kindof best-ﬁrst sear chcalled A∗search.
A∗searchwas ﬁrst implementedformachinetranslationbyIBM( Brownet al.,1995),
based on IBM’s earlier work on A∗search for speech recognition (Jelinek, 1969). As

we discussed in Sec. 10.2, for historical reasons A∗search and its variants are com-

DRAFTSection25.8. DecodingforPhrase-BasedStatistical MT 905
monly called stack decoding in speech recognition and sometimes also in machine Stackdecoding
translation.
Let’s begin in Fig. 25.27 with a generic version of stack deco ding for machine
translation. The basic intuition is to maintain a priority queue (traditionally referred
toasastack)withall thepartialtranslationhypotheses,togetherwit htheirscores.
function STACK DECODING (source sentence) returnstarget sentence
initializestackwitha null hypothesis
loop do
popbest hypothesis hoff of stack
ifhis acomplete sentence, returnh
for eachpossible expansion h′ofh
assign ascore to h′
pushh′onto stack
Figure25.27 GenericversionofstackorA∗decodingformachinetranslation. Ahypothesisis
expanded bychoosing a single word or phrase totranslate. We ’llsee a more ﬂeshed-out version
of the algorithm inFig.25.30.
Let’snowdescribestackdecodinginmoredetail. Whiletheo riginalIBMstatistical
decodingalgorithmswere forword-basedstatistical MT, we will describethe applica-

loop do
tion to phrase-baseddecodingin the publiclyavailable MT d ecoderPharaoh (Koehn,
2004).
In order to limit the search space in decoding, we don’t want t o search through
the space of all Englishsentences; we only want to considert he onesthat are possible
translationsfor F. Tohelpreducethesearchspace,weonlywanttoconsidersen tences
thatincludewordsorphraseswhicharepossibletranslatio nsofwordsorphrasesinthe
Spanish sentence F. We do this by searching the phrase translation table described

2004).
in the previoussection, for all possible English translati ons for all possible phrases in
F.
A sample lattice of possible translation optionsis shownin Fig. 25.28drawnfrom
Koehn (2003a, 2004). Each of these options consists of a Span ish word or phrase,
the English translation, and the phrase translation probab ilityφ. We’ll need to search
throughcombinationsoftheseto ﬁndthebesttranslationst ring.
Now let’s walk informally throughthe stack decoding exampl ein Fig. 25.29, pro-

F.
ducing an English translation of Mary di´o una bofetada a la bruja verde left to right.
Forthemomentwe’llmakethesimplifyingassumptionthatth ereisasinglestack,and
thatthereisnopruning.
We start with the null hypothesis as the initial search state , in which we have
selectednoSpanishwordsandproducednoEnglishtranslati onwords. Wenow expand
thishypothesisbychoosingeachpossiblesourcewordorphr asewhichcouldgenerate
an English sentence-initial phrase. Fig. 25.29a shows this ﬁrst ply of the search. For

example the top state represents the hypothesis that the Eng lish sentence starts with
Mary, and the Spanish word Mariahas been covered(the asterisk for the ﬁrst word is
markedwithanM).Eachstateisalsoassociatedwithacost,d iscussedbelow. Another

DRAFT906 Chapter 25. MachineTranslation
verde laa bofetadaunadióno Maria bruja
Marynot
did not
no
did not givegive aslapto green thewitch
a slap to green witch
slap to the
to
the
slap the witch
Figure25.28 The latticeofpossible Englishtranslationsfor words andp hrases inaparticular
sentenceF,taken from the entire alignedtraining set. After Koehn (20 03a)
Figure25.29 Three stages instackdecoding of Marianodi´ ounabofetada alabrujaverde (simpliﬁedbyassum-
ing a single stack and no pruning). The nodes in blue, on the fr inge of the search space, are all on the stack, and are
opennodes stillinvolved inthe search. Nodes ingrayare closednodes whichhave been popped off thestack.
state at this ply represents the hypothesis that the English translation starts with the
wordNo, and that Spanish nohas been covered. This turns out to be the lowest-cost
node on the queue, so we pop it off the queue and push all its exp ansionsback on the
queue. Now the state Maryis the lowest cost, so we expand it; Mary did not is now

the lowest cost translation so far, so will be the next to be ex panded. We can then
continue to expand the search space until we have states (hyp otheses) that cover the
entireSpanishsentence,andwecanjustreadoffanEnglisht ranslationfromthisstate.
We mentioned that each state is associated with a cost which, as we’ll see below,
is used to guide the search, The cost combines the current cost with an estimate of
thefuturecost . Thecurrent cost is the total probabilityofthe phrasesthat have been

translated so far in the hypothesis, i.e. the product of the t ranslation, distortion, and
languagemodelprobabilities. Forthesetofpartiallytran slatedphrases S= (F,E),this

DRAFTSection25.8. DecodingforPhrase-BasedStatistical MT 907
probabilitywouldbe:
cost(E,F) =∏
i∈Sφ(¯fi,¯ei)d(ai−bi−1)P(E) (25.32)
Thefuture cost is our estimate of the cost of translating the remaining words in the
Spanish sentence. By combining these two factors, the state cost gives an estimate of
the total probability of the search path for the eventual com plete translation sentence
Epassingthroughthecurrentnode. A searchalgorithmbasedo nlyonthe currentcost
wouldtendtoselecttranslationsthathadafewhigh-probab ilitywordsatthebeginning,
at the expense of translations with a higher overall probabi lity.4For the future cost,
it turns out to be far too expensive to compute the true minimu m probability for all
possible translations. Instead,we approximatethis cost b y ignoringthe distortioncost
and just ﬁnding the sequence of English phrases which has the minimum product of
the languagemodeland translation model costs, which can be easily computedby the
Viterbialgorithm.

Thissketchofthedecodingprocesssuggeststhatwesearcht heentirestatespaceof
possible English translations. But we can’t possiblyaffor dto expandthe entire search
space, because there are far too many states; unlike in speec h recognition, the need
for distortion in MT means there is (at least) a distinct hypo thesis for every possible
orderingoftheEnglishwords!5
ForthisreasonMTdecoders,likedecodersforspeechrecogn ition,allrequiresome
sortof pruning. Pharaohandsimilardecodersuse a versiono fbeam-searchpruning ,Beam-searchpruning

just as we saw in decoding for speech recognition and probabi listic parsing. Recall
thatinbeam-searchpruning,ateveryiterationwekeeponly themostpromisingstates,
andpruneawayunlikely(high-cost)states(those‘outside thesearchbeam’). Wecould
modifythesearchsequencedepictedinFig.25.29,byprunin gawayallbad(high-cost)
statesat everyplyofthesearch,andexpandingonlythebest state. Infact,inPharaoh,
instead of expanding only the best state, we expand all state s within the beam; thus
Pharaohistechnically beamsearch ratherthan best-ﬁrst search orA∗search.

Moreformally,ateachplyofthesearchwekeeparoundastack (priorityqueue)of
states. Thestackonlyﬁts nentries. Ateveryplyofthesearch,weexpandallthestates
on the stack, pushthem onto the stack, orderthem bycost, kee p the best nentriesand
deletetherest.
We’llneedoneﬁnalmodiﬁcation. Whileinspeechwejustused onestackforstack
decoding, in MT we’ll use multiple stacks, because we can’t e asily compare the cost
ofhypothesesthattranslatedifferentnumbersofforeignw ords. Sowe’lluse mstacks,
wherestack smincludesallhypothesesthatcover mforeignwords. Whenweexpanda

hypothesisby choosinga phrase to translate,we’ll insert t he newstate into the correct
stackforthenumberofforeignwordscovered. Thenwe’lluse beam-searchinsideeach
of these stacks, keep only nhypothesesforeach of the mstacks. The ﬁnal multi-stack
versionofbeamsearchstack decodingisshownin Fig.25.30.
4We saw this same kind of cost function for A∗search in speech recognition, where we used the A∗
evaluation function: f∗(p) =g(p)+h∗(p).
5Indeed, as Knight (1999a) shows, decoding even in IBM Model 1 with a bigram language model is

equivalent to the difﬁcult class of problems known as NP-complete .

DRAFT908 Chapter 25. MachineTranslation
function BEAM SEARCH STACK DECODER (source sentence) returnstarget sentence
initializehypothesisStack[0..nf]
push initialnull hypothesis onhypothesisStack[0]
fori←0tonf-1
for eachhypinhypothesisStack[i]
for eachnewhypthat canbe derivedfrom hyp
nf[newhyp]←number of foreignwords covered by newhyp
addnewhyptohypothesisStack[nf[new hyp]]
prune hypothesisStack[nf[new hyp]]
ﬁndbest hypothesis besthypinhypothesisStack[nf]
returnbest paththat leads to besthypvia backtrace
Figure25.30 Pharaoh beam search multi-stack decoding algorithm, adapt ed from (Koehn,
2003a). Forefﬁciency,mostdecoders don’tstoretheentire foreignandEnglishsentence ineach
state,requiringthatwebacktracetoﬁndthestatepathfrom theinitialtotheﬁnalstatesowecan
generate the entire Englishtargetsentence.
There are a number of additional issues in decoding that must be dealt with. All
decoders attempt to limit somewhat the exponential explosi on in the search space by

recombining hypotheses . . We saw hypothesis recombination in the ExactN-BestRecombining
hypotheses
algorithm of Sec. 10.1. In MT, we can merge any two hypotheses that are sufﬁciently
similar(coverthesameforeignwords,havethesamelast-tw oEnglishwords,andhave
thesame endofthelast foreignphrasecovered).
In addition, it turns out that decoders for phrasal MT optimi ze a slightly different
functionthan the one we presented in Eq. 25.31. In practice, it turns out that we need
toaddanotherfactor,whichservestopenalizesentenceswh icharetooshort. Thusthe

decoderisactuallychoosingthesentencewhichmaximizes:
ˆE=argmax
P(E)shortsentencepenalty
ωlength (E)(25.33)
This ﬁnal equation is extremely similar to the use of the word insertion penalty in
speechrecognitioninEq.9.49.
25.9 MTEvaluation
Evaluating the quality of a translation is an extremely subj ective task, and disagree-
ments about methodology are rampant. Nevertheless, evalua tion is essential, and re-
search on evaluation methodologyhas played an important ro le from the earliest days

ˆE=argmax
of MT (Miller and Beebe-Center, 1958). Broadly speaking, tr anslations are evaluted
alongtwodimensions,correspondingtothe ﬁdelityandﬂuencydiscussedinSec.25.3.

DRAFTSection25.9. MTEvaluation 909
25.9.1 UsingHumanRaters
Themostaccurateevaluationsusehumanraterstoevaluatee achtranslationalongeach
dimension. For example,alongthe dimensionof ﬂuency, we canask how intelligible,
how clear, how readable, or how natural is the MT output (the t arget translated text).
Thereare two broadwaysto use humanratersto answerthese qu estions. One method
is to give the raters a scale, for example from 1 (totally unin telligible) to 5 (totally
intelligible), and ask them to rate each sentence or paragra ph of the MT output. We
can use distinct scales for any of the aspects of ﬂuency, such asclarity,naturalness ,
orstyle. The second class of methods relies less on the conscious dec isions of the
participants. For example, we can measure the time it takes f or the raters to read each
output sentence or paragraph. Clearer or more ﬂuent sentenc es should be faster or
easier to read. We can also measure ﬂuency with the clozetask (Taylor, 1953, 1957). Cloze

The cloze task is a metric used often in psychological studie s of reading. The rater
sees an outputsentence with a word replacedby a space (forex ample,every8thword
might be deleted). Raters have to guess the identity of the mi ssing word. Accuracyat
the cloze task, i.e. average success of raters at guessing th e missing words, generally
correlateswith howintelligibleornaturaltheMT outputis .
A similar variety of metrics can be used to judge the second di mension, ﬁdelity.
Two common aspects of ﬁdelity which are measured are adequacy andinformative-

ness. Theadequacy ofatranslationiswhetherit containstheinformationthat existed Adequacy
intheoriginal. We measureadequacybyusingraterstoassig nscoresona scale. Ifwe
havebilingualraters, we can givethem the sourcesentencea nd a proposedtargetsen-
tence, and rate, perhapson a 5-pointscale, how muchof the in formationin the source
was preserved in the target. If we only have monolingual rate rs, but we have a good
human translation of the source text, we can give the monolin gual raters the human

referencetranslation and a target machinetranslation, an d again rate how much infor-
mation is preserved. The informativeness of a translation is a task-based evaluation Informativeness
of whether there is sufﬁcient informationin the MT output to perform some task. For
examplewecangiveratersmultiple-choicequestionsabout thecontentofthematerial
inthesourcesentenceortext. Theratersanswerthesequest ionsbasedonlyontheMT
output. Thepercentageofcorrectanswersisaninformative nessscore.

Anothersetofmetricsattempttojudgetheoverallqualityo fatranslation,combin-
ingﬂuencyandﬁdelity. Forexample,thetypicalevaluation metricforMToutputtobe
post-editedis the edit cost ofpost-editing the MT output into a goodtranslation. For Editcost
Post-editing example, we can measure the number of words, the amount of tim e, or the number of
keystrokesrequiredforahumanto correcttheoutputtoanac ceptablelevel.
25.9.2 AutomaticEvaluation: Bleu
While humans produce the best evaluations of machine transl ation output, running

a human evaluation can be very time-consuming, taking days o r even weeks. It is
useful to have an automatic metric that can be run relatively frequently to quickly
evaluatepotentialsystemimprovements. Inordertohavesu chconvenience,wewould
bewillingforthemetrictobemuchworsethanhumanevaluati on,aslongastherewas
somecorrelationwith humanjudgments.

DRAFT910 Chapter 25. MachineTranslation
In fact there are a number of such heuristic methods, such as Bleu,NIST,TER,
Precision and Recall , andMETEOR (see references at the end of the chapter). The
intuitionoftheseautomaticmetricsderivesfromMilleran dBeebe-Center(1958),who
pointedoutthata goodMT outputisonewhichisverysimilar t oa humantranslation.
In the ﬁeld of automatic speech recognition, we deﬁne ‘verys imilar’ via the word
errorrate,whichistheminimumeditdistancetoahumantran script. Butintranslation,
we don’trely on a single humantranslation,because a source sentence could be legit-
inately translated in many ways. A very good MT output might l ook like one human
translation,butveryunlikeanotherone. Forthis reason,M T metricsgenerallyrequire
thatwehavemultiplehumantranslationsofeachsentencein atest set. Thismayseem
time-consuming,butthehopeisthatwe canreusethistransl atedtest set overandover
againto evaluatenewideas.
Now given an MT output sentence in a test set, we compute the tr anslation close-

ness between the MT output and the human sentences. An MT outp ut is ranked as
better if on average it is closer to the human translations. T he metrics differ on what
countsas‘translationcloseness’.
Fortherestofthissection,let’swalkthroughoneofthesem etrics,the Bleumetric,
following closely the original presentation in Papineni et al. (2002). In Bleu we rank
each MT output by a weighted average of the number of N-gram overlaps with the
humantranslations.

Fig. 25.31showsan intuition,fromtwo candidatetranslati onsof a Chinese source
sentence (Papineni et al., 2002),shown with three referenc ehumantranslationsof the
sourcesentence. Note that Candidate1 sharesmanymoreword s(shownin blue) with
thereferencetranslationsthanCandidate2.
Figure25.31 IntuitionforBleu: oneoftwocandidatetranslationsofaCh inesesourcesentencesharesmorewords
withthe reference human translations.
Let’s look at how the Bleu score is computed,starting with ju st unigrams. Bleu is

based on precision. A basic unigram precision metric would b e to count the number
ofwordsinthecandidatetranslation(MToutput)thatoccur insomereferencetransla-
tion,anddividebythetotalnumberofwordsinthecandidate translation. Ifacandidate
translationhad10words,and6 ofthemoccurredin at least on eofthereferencetrans-
lations,we wouldhavea precisionof6 /10=0.6. Alas, thereis a ﬂawin usingsimple
precision: it rewards candidates that have extra repeated w ords. Fig. 25.32 shows an

example of a pathological candidate sentence composed of mu ltiple instances of the
single word the. Since each of the 7 (identical)wordsin the candidateoccur in one of
thereferencetranslations,theunigramprecisionwouldbe 7/7!
Inordertoavoidthisproblem,Bleuusesa modiﬁedN-gramprecision metric. WeModiﬁedN-gramprecision

DRAFTSection25.9. MTEvaluation 911
Figure25.32 A pathological example showing why Bleu uses a modiﬁed preci sion metric.
Unigram precision would be unreasonably high (7/7). Modiﬁe d unigram precision is appropri-
atelylow (2/7).
ﬁrst countthemaximumnumberoftimesa wordisused inanysin glereferencetrans-
lation. The count of each candidate word is then clipped by this maximum reference
count. Thus the modiﬁed unigram precision in the example in F ig. 25.32 would be
2/7, since Reference 1 has a maximum of 2 thes. Going back to Chinese example in
Fig. 25.32,Candidate1 hasa modiﬁedunigramprecisionof 17 /18,while Candidate 2
hasoneof8/14.
Wecomputethemodiﬁedprecisionsimilarlyforhigherorder N-gramsaswell. The
modiﬁed bigram precision for Candidate 1 is 10/17, and for Ca ndidate 2 is 1/13. The
readershouldcheckthesenumbersforthemselvesonFig.25. 31.
Tocomputeascoreoverthewholetestset,Bleuﬁrstcomputes theN-grammatches
foreachsentence,andaddtogethertheclippedcountsovera llthecandidatessentences,

and divide by the total number of candidate N-grams in the testset. The modiﬁed
precisionscoreisthus:
pn=∑
C∈{Candidates}∑
n-gram∈CCountclip(n-gram )
∑
C′∈{Candidates}∑
n-gram’∈C′Count(n-gram’ )(25.34)
Bleu uses unigram, bigrams, trigrams, and often quadrigram s; it combines these
modiﬁed N-gramprecisionstogetherbytakingtheirgeometricmean.
In addition, Bleu adds a further penalty to penalize candida te translations that are
too short. Consider the candidate translation of the, compared with References 1-3

pn=∑
∑
in Fig. 25.31 above. Because this candidate is so short, and a ll its words appear in
some translation, its modiﬁed unigram precision is inﬂated to 2/2. Normally we deal
with these problems by combining precision with recall. But as we discussed above,
we can’t use recall over multiple human translations, since recall would require (in-
correctly) that a good translation must contain contains lo ts ofN-grams from every
translation. Instead, Bleu includes a brevity penalty over the whole corpus. Let cbe

the total length of the candidate translation corpus. We com pute theeffective refer-
ence length rfor that corpusby summing,for each candidatesentence, the lengthsof
thebest matches. Thebrevitypenaltyisthenanexponential inr/c. Insummary:
BP={1 if c>r
e(1−r/c)ifc≤r

DRAFT912 Chapter 25. MachineTranslation
Bleu=BP×exp(
1
NN
∑
n=1logpn)
(25.35)
WhileautomaticmetricslikeBleu(orNIST,METEOR,etc)hav ebeenveryuseful
in quickly evaluating potential system improvements,and m atch human judgments in
manycases,theyhavecertainlimitationsthatareimportan tto consider. First,manyof
themfocusonverylocalinformation. Considerslightlymov inga phraseinFig.25.31
slightly to produce a candidate like: Ensures that the military it is a guide to action
whichalwaysobeysthecommandsoftheparty . Thissentencewouldhaveanidentical
Bleu scoretoCandidate1,althoughahumanraterwouldgivei t alowerscore.
Furthermore,the automatic metrics probably do poorly at co mparing systems that
have radically different architectures. Thus Bleu, for exa mple, is known to perform
poorly(i.e. notagreewithhumanjudgmentsoftranslationq uality)whenevaluatingthe
outputofcommercialsystemslikeSystranagainst N-gram-basedstatisticalsystems,or
even when evaluating human-aided translation against mach ine translation (Callison-

Burchetal.,2006). Wecanconcludethatautomaticmetricsa remostappropriatewhen
evaluating incremental changes to a single system, or compa ring systems with very
similararchitectures.
25.10 Advanced: SyntacticModelsforMT
Theearlieststatistical MTsystems(likeIBMModels1,2and 3)werebasedonwords
astheelementaryunits. Thephrase-basedsystemsthatwede scribedinearliersections
improved on these word-based systems by using larger units, thus capturing larger
contextsandprovidinga morenaturalunitforrepresenting languagedivergences.

Recent work in MT has focused on ways to move even further up th e Vauquois
hierarchy,fromsimple phrasesto largerandhierarchicals yntacticstructures.
Itturnsoutthatit doesn’tworkjust toconstraineachphras etomatchthesyntactic
boundariesassigned by traditional parsers (Yamada and Kni ght, 2001). Instead, mod-
ernapproachesattempttoassignaparallelsyntactictrees tructuretoapairofsentences
indifferentlanguages,withthegoaloftranslatingthesen tencesbyapplyingreordering
operationsonthe trees. Themathematicalmodelforthesepa rallelstructuresisknown

as atransduction grammar . These transduction grammars can be viewed as an ex-Transductiongrammar
plicit implementation of the syntactic transfer systems that we introduced on page
885,butbasedonamodernstatistical foundation.
A transduction grammar(also called a synchronous grammar ) describes a struc-Synchronousgrammar
turally correlated pair of languages. From a generative per spective, we can view a
transductiongrammar as generatingpairs of aligned senten ces in two languages. For-

mally,atransductiongrammarisageneralizationoftheﬁni te-statetransducerswesaw
in Ch. 3. There are a number of transduction grammarsand form alisms used for MT,
most of which are generalizationsof context-free grammars to the two-language situ-
ation. Let’s consider one of the most widely used such models for MT, the inversion
transductiongrammar (ITG).InversiontransductiongrammarIn an ITG grammar, each non-terminal generates two separate strings. There are

DRAFTSection25.10. Advanced: SyntacticModelsforMT 913
threetypesoftheserules. A lexicalrulelike thefollowing :
N→witch/bruja
generatestheword witchononestream,and brujaonthesecondstream. Anonterminal
ruleinsquarebracketslike:
S→[NPVP ]
generatestwoseparatestreams,eachof NP VP. Anon-terminalinanglebrackets,like
Nominal→⟨AdjN⟩
generates two separate streams, with different orderings :Adj Nin one stream, and N
Adjinthe otherstream.
Fig.25.33showsasamplegrammarwithsomesimplerules. Not ethateachlexical
rule derives distinct English and Spanish word strings, tha t rules in square brackets
([])generatetwoidenticalnon-terminalright-handsides ,andthattheoneruleinangle
brackets(⟨⟩)generatesdifferentorderingsinSpanishfromEnglish.
S→[NPVP ]
NP→[DetNominal ]|Maria /Mar´ıa
Nominal→ ⟨AdjNoun⟩
VP→[VPP]|[NegationVP ]
Negation→didn’t/no
V→slap/di´ounabofetada
PP→[PNP]
P→ǫ/a|from/de
Det→the/la|the/le
Adj→green/verde
Figure25.33 AminiInversionTransduction Grammargrammarforthe greenwitch sentence.

S→[NPVP ]
PP→[PNP]
ThusanITGparsetreeisasinglejointstructurewhichspans overthetwoobserved
(25.36) (a) [S[NPMary][ VPdidn’t[VPslap[PP[NPthe[Nomgreenwitch]]]]]]
(b) [S[NPMar´ ıa][ VPno[VPdi´ ounabofetada[ PPa[NPla [Nombruja
verde]]]]]]
Each non-terminal in the parse derives two strings, one for e ach language. Thus
we couldvisualize the two sentencesin a single parse, where the angle bracketsmean
that the order of the Adj Nconstituents green witch andbruja verde are generated in
oppositeorderinthe twolanguages:

[S[NPMary/Mar´ ıa][ VPdidn’t/no[ VPslap/di´ ounabofetada[ PPǫ/a[NPthe/la⟨Nomwitch/brujagreen/verde ⟩]]]]]
There are a number of related kinds of synchronous grammars, including syn-
chronouscontext-freegrammars(Chiang,2005),multitext grammars(Melamed,2003),

DRAFT914 Chapter 25. MachineTranslation
lexicalized ITGs (Melamed, 2003; Zhang and Gildea, 2005), a nd synchronous tree-
adjoining and tree-insertion grammars (Shieber and Schabe s, 1992; Shieber, 1994b;
Nesson et al., 2006). The synchronous CFG system of Chiang (2 005), for example,
learns hierarchical pairs of rules that capture the fact tha t Chinese relative clauses ap-
pear to the left of their head, while English relative clause s appear to the right of their
head:
<①de②, the②that①>
Other models for translation by aligning parallel parse tre es including (Wu, 2000;
Yamada and Knight, 2001; Eisner, 2003; Melamed, 2003; Galle y et al., 2004; Quirk
et al.,2005;WuandFung,2005).
25.11 Advanced: IBM Model 3 and fertility-based align-
ment
The seminal IBM paper that began work on statistical MT propo sed ﬁve models for
MT. We saw IBM’s Model 1 in Sec. 25.5.1. Models 3, 4 and 5 all use the important
concept of fertility. We’ll introduce Model 3 in this section; our description he re is

ment
inﬂuencedbyKevin Knight’snice tutorial(Knight, 1999b). Model 3 has a more com-
plex generative model than Model 1. The generativemodel fro m an English sentence
E=e1,e2,...,eIhas5steps:
1. For each English word ei, we choose a fertilityφi.6The fertility is the number Fertility
of(zeroormore)Spanishwordsthatwillbegeneratedfrom ei,andisdependent
onlyonei.
2. We also need to generate Spanish words from the NULL Englis h word. Recall

onlyonei.
thatwe deﬁnedthese earlieras spuriouswords . Insteadof havinga fertilityfor Spurious word
NULL, we’ll generate spurious words differently. Every tim e we generate an
English word, we consider (with some probability) generati ng a spurious word
(fromNULL).
3. We now know how many Spanish words to generate from each Eng lish word.
So now for each of these Spanish potential words, generate it by translating its
aligned English word. As with Model 1, the translation will b e based only on
the English word. Spurious Spanish words will be generated b y translating the
NULLwordintoSpanish.

4. Move all the non-spurious words into their ﬁnal positions in the Spanish sen-
tence.
5. InsertthespuriousSpanishwordsintheremainingopenpo sitionsintheSpanish
sentence.
Fig.25.34showsavisualizationoftheModel3generativepr ocess
Model 3 has more parameters than Model 1. The most important a re then,t,d, n
t
dandp1probabilities. The fertility probability φiof a word eiis represented by the
p16Thisφis notrelated to the φthat was used in phrase-based translation.

DRAFTSection25.11. Advanced: IBMModel3andfertility-basedal ignment 915
Figure25.34 The ﬁve steps of IBM Model 3 generating a Spanish sentence and alignment
parameter n. So we will use n(1|green)to representthe probabilitythat English green
will produce one Spanish word, n(2|green)is the probability that English greenwill
produce two Spanish words, n(0|did)is the probability that English didwill produce
noSpanishwords,andsoon. LikeIBMModel1,Model3hasatran slationprobability
t(fj|ei). Next, the probabilitythat expressesthe word positiontha t English wordsend
up in the Spanish sentence is the distortion probability, which is conditioned on the Distortion
EnglishandSpanishsentencelengths. Thedistortionproba bilityd(1,3,6,7)expresses
the probability that the English word e1will align to Spanish word f3, given that the
Englishsentencehaslength6,andthe Spanishsentenceisof length7.
Aswesuggestedabove,Model3doesnotusefertilityprobabi litieslike n(1|NULL ),

orn(3|NULL )to decide how many spurious foreign words to generate from En glish
NULL.Instead,eachtimeModel3generatesa realword,itgen eratesaspuriousword
forthetargetsentencewithprobability p1. Thisway,longersourcesentenceswill nat-
urallygeneratemorespuriouswords. Fig.25.35showsaslig htlymoredetailedversion
ofthe5stepsoftheModel3generativestoryusingthesepara meters.
Switching for a moment to the task of French to English transl ation, Fig. 25.36
shows some of the tandφparameters learned for French-English translation from

Brown et al. (1993). Note that thein general translates to a French article like le, but
sometimes it has a fertility of 0, indicating that English us es an article where French
does not. Conversely, note that farmersprefers a fertility of 2, and the most likely
translationsare agriculteurs andles, indicatingthat hereFrenchtendsto use an article
whereEnglishdoesnot.

DRAFT916 Chapter 25. MachineTranslation
1.for eachEnglishword ei,1<i<I,we choose a fertility φiwithprobability n(φi|ei)
2.Usingthesefertilitiesand p1,determine φ0,the number of spurious Spanishwords,and
hencem.
3.for eachi,0<i<I
for eachk, 1<k<φi
Choose a Spanishword τikwithprobability t(τik,ei)
4.for eachi,1<i<I
Choose a target Spanishposition πikwithprobability d(πik,i,I,J)
5.for eachk, 1<k<φ0
Choose a target Spanish position π0kfrom one of the available Spanish slots, for a
totalprobability of1
φ0!
Figure25.35 The Model 3 generative story for generating a Spanish senten ce from an En-
glish sentence. Remember that we are not translating from En glish to Spanish; this is just the
generative component of the noisychannel model. Adapted fr om Knight (1999b).
the farmers not
ft(f|e)φn(φ|e)f t(f|e)φn(φ|e)ft(f|e)φn(φ|e)
le 0.497 1 0.746 agriculteurs 0.442 2 0.731 ne 0.497 2 0.735
la 0.207 0 0.254 les 0.418 1 0.228 pas 0.442 0 0.154
les 0.155 cultivateurs 0.046 0 0.039 non 0.029 1 0.107

φ0!
l’ 0.086 producteurs 0.021 rien 0.011
ce 0.018
cette 0.011
Figure25.36 Examples of Model 3 parameters from the Brown et al. (1993) Fr ench-English
translation system, for three English words. Note that both farmersandnotare likely to have
fertilitiesof 2.
Now that we have seen the generative story for Model 3, let’s b uild the equation
for the probability assigned by the model. The model needs to assigns a probability
P(F|E)of generatingthe Spanish sentence Ffromthe English sentence E. As we did

ce 0.018
with Model 1, we’ll start by showing how the model gives the pr obability P(F,A|E),
the probabilityof generatingsentence Fvia a particularalignment A. Then we’ll sum
overall alignmentstogetthetotal P(F|E).
In order to compute P(F,A|E), we’ll need to multiply the main three factors n,t,
andd, for generating words, translating them into Spanish, and m oving them around.
Soa ﬁrst passat P(F,A|E)wouldbe:
I
∏
i=1n(φi|ei)×J
∏
j=1t(fj|eaj)×J
∏
j=1d(j|aj,I,J) (25.37)

I
∏
∏
∏
But (25.37)isn’t sufﬁcient as it stands; we need to add facto rs for generatingspu-
rious words, for inserting them into the available slots, an d a factor having to do with
the number of ways (permutations) a word can align with multi ple words. Eq. 25.38
givesthe trueﬁnal equationforIBM Model3,in Knight’smodi ﬁcationofthe original
formula. We won’tgivethe detailsofthese additionalfacto rs,butencouragetheinter-

DRAFTSection25.11. Advanced: IBMModel3andfertility-basedal ignment 917
ested reader to see the original presentation in Brown et al. (1993) and the very clear
explanationoftheequationinKnight(1999b).
P(F,A|E) =generatespurious(J−φ0
φ0)
pJ−2φ0
0pφ0
1×insertspurious
1
φ0!×multi-alignpermutations
I
∏
i=0φi!
×I
∏
∏
∏
j:aj̸=0d(j|aj,I,J) (25.38)
Onceagain,in orderto get the total probabilityofthe Spani shsentencewe’ll need
tosum overall possiblealignments:
P(F|E) =∑
AP(F,A|E)
We can also make it more explicit exactly how we sum over align ments (and also
emphasize the incrediblylarge numberof possible alignmen ts)by expressingthis for-
mula as follows, where we specify an alignment by specifying the aligned English aj
foreachofthe Jwordsintheforeignsentence:
P(F|E) =J
∑
a1=0J
∑
a2=0···I
∑
aJ=0P(F,A|E)
25.11.1 Training forModel 3
Givena parallelcorpus,trainingthe translationmodelfor IBM Model3 meanssetting
valuesforthe n,d,t,andp1parameters.

P(F|E) =∑
AP(F,A|E)
P(F|E) =J
∑
a1=0J
∑
a2=0···I
∑
AswenotedforModel1andHMMmodels,ifthetraining-corpus washand-labeled
withperfectalignments,gettingmaximumlikelihoodestim ateswouldbesimple. Con-
sidertheprobability n(0|did)thatawordlike didwouldhaveazerofertility. We could
estimate this froman aligned corpusjust by countingthe num berof times didaligned
to nothing, and normalize by the total count of did. We can do similar things for the
ttranslation probabilities. To train the distortion probab ilityd(1,3,6,7), we similarly

countthenumberoftimesinthecorpusthatEnglishword e1mapstoSpanishword f3
inEnglishsentencesoflength6thatarealignedtoSpanishs entencesoflength7. Let’s
call thiscountingfunctiondcount. We’ll againneedanorma lizationfactor;
d(1,3,6,7) =dcount (1,3,6,7)
∑I
i=1dcount (i,3,6,7)(25.39)
Finally, we need to estimate p1. Again, we look at all the aligned sentences in
the corpus; let’s assume that in the Spanish sentences there are a total of Nwords.
From the alignments for each sentence, we determine that a to tal ofSSpanish words

∑I
are spurious, i.e. aligned to English NULL. Thus N−Sof the words in the Spanish

DRAFT918 Chapter 25. MachineTranslation
sentencesweregeneratedbyrealEnglishwords. After SoftheseN−SSpanishwords,
we generateaspuriousword. Theprobability p1isthusS/(N−S).
Of course, we don’t have hand-alignments for Model 3. We’ll n eed to use EM
to learn the alignments and the probability model simultane ously. With Model 1 and
the HMM model, there were efﬁcient ways to do training withou t explicitly summing
over all alignments. Unfortunately, this is not true for Mod el 3; we actually would
need to computeall possible alignments. For a real pair of se ntences, with 20 English
words and 20 Spanish words, and allowing NULL and allowing fe rtilities, there are a
very large number of possible alignments (determining the e xact number of possible
alignmentsisleftasExercise7). Instead,weapproximateb yonlyconsideringthebest
few alignments. In orderto ﬁnd the best alignmentswithoutl ookingat all alignments,
wecanuseaniterativeorbootstrappingapproach. Intheﬁrs tstep,wetrainthesimpler

IBM Model 1 or 2 as discussed above. Then we use these Model 2 pa rameters to
evaluateP(A|E,F),givingawaytoﬁndthebestalignmentstobootstrapModel3. See
Brownetal. (1993)andKnight(1999b)fordetails.
25.12 Advanced: Log-linearModelsforMT
While statistical MT was ﬁrst based on the noisy channel mode l, much recent work
combines the language and translation models via a log-line ar model in which we di-
rectlysearchforthesentencewiththe highestposteriorpr obability:
ˆE=argmax
EP(E|F) (25.40)

ˆE=argmax
This is done by modeling P(E|F)via a set of Mfeature functions hm(E,F), each of
whichhasa parameter λm. Thetranslationprobabilityisthen:
P(E|F) =exp[∑M
m=1λmhm(E,F)]
∑E′exp[∑M
m=1λmhm(E′,F)](25.41)
Thebestsentenceisthus:
ˆE=argmax
EP(E|F)
=argmax
Eexp[M
∑
m=1λmhm(E,F)] (25.42)
In practice, the noisy channel model factors (the language m odelP(E)and trans-
lation model P(F|E)), are still the most important feature functions in the log- linear
model,butthearchitecturehastheadvantageofallowingfo rarbitraryotherfeaturesas

∑E′exp[∑M
ˆE=argmax
EP(E|F)
=argmax
Eexp[M
∑
well; acommonset offeatureswouldinclude:
•thelanguagemodel P(E)
•thetranslationmodel P(F|E)
•thereversetranslationmodel P(E|F),Reversetranslationmodel

DRAFTSection25.12. Advanced: Log-linearModelsforMT 919
•lexicalizedversionsofbothtranslationmodels,
•aword penalty , Word penalty
•aphrase penalty Phrase penalty
•anunknown wordpenalty .Unknownwordpenalty
SeeFoster(2000),OchandNey(2002,2004)formoredetails.
Log-linear models for MT could be trained using the standard maximum mutual
information criterion. In practice, however, log-linear m odels are instead trained to
directlyoptimizeevaluationmetricslike Bleuin amethodk nownasMinimum Error
RateTraining ,orMERT(Och,2003;Chouet al.,1993).Minimum ErrorRateTraining
MERT
Work on models of the process and goals of translation goes ba ck at least to Saint
Jerome in the fourth century (Kelley, 1979). The developmen t of logical languages,
free of the imperfections of human languages, for reasoning correctly and for com-
municating truths and thereby also for translation, has bee n pursued at least since the
1600s(Hutchins,1986).

MERT
By the late 1940s, scant years after the birth of the electron ic computer, the idea
of MT was raised seriously (Weaver, 1955). In 1954 the ﬁrst pu blic demonstration of
a MT system prototype(Dostert, 1955)led to great excitemen tin the press (Hutchins,
1997). The next decade saw a great ﬂowering of ideas, preﬁgur ing most subsequent
developments. But this work was ahead of its time — implement ations were limited
by,forexample,thefactthatpendingthedevelopmentofdis kstherewasnogoodway
tostoredictionaryinformation.

AshighqualityMTprovedelusive (Bar-Hillel,1960),a grow ingconsensusonthe
needforbetterevaluationandmorebasicresearchinthenew ﬁeldsofformalandcom-
putational linguistics, culminating in the famous ALPAC (A utomatic Language Pro-
cessingAdvisoryCommittee)reportof1966(Pierceetal.,1 966),ledinthemid1960s
to a dramatic cut in funding for MT. As MT research lost academ ic respectability, the
AssociationforMachineTranslationandComputationalLin guisticsdroppedMTfrom
its name. Some MT developers, however, persevered, slowly a nd steadily improving

their systems, and slowly garnering more customers. Systra n in particular, developed
initially by Peter Toma, has been continuously improved ove r 40 years. Its earliest
uses were for information acquisition, for example by the U. S. Air Force for Rus-
sian documents; and in 1976 an English-French edition was ad opted by the European
Communityfor creatingroughand post-editabletranslatio nsof variousadministrative
documents. AnotherearlysuccessfulMTsystemwasM´ et´ eo, whichtranslatedweather

forecastsfromEnglishtoFrench;incidentally,itsorigin alimplementation(1976),used
“Q-systems”,anearlyuniﬁcationmodel.
The late 1970s saw the birth of another wave of academic inter est in MT. One
strand attempted to apply meaning-based techniques develo ped for story understand-
ing and knowledgeengineering(Carbonell et al., 1981). The re were wide discussions

DRAFT920 Chapter 25. MachineTranslation
of interlingual ideas through the late 1980s and early 1990s (Tsujii, 1986; Nirenburg
etal.,1992;Ward,1994;Carbonelletal.,1992). Meanwhile MTusagewasincreasing,
fueledbyglobalization,governmentpoliciesrequiringth etranslationofalldocuments
intomultipleofﬁciallanguages,andtheproliferationofw ordprocessorsandthenper-
sonalcomputers.
Modern statistical methodsbegan to be applied in the early 1 990s, enabled by the
development of large bilingual corpora and the growth of the web. Early on, a num-
ber of researchers showed that it was possible to extract pai rs of aligned sentences
frombilingualcorpora(KayandR¨ oscheisen,1988,1993;Wa rwickandRussell,1990;
Brown et al., 1991; Gale and Church, 1991, 1993). The earlies t algorithms made use
of the wordsof the sentenceas part of the alignmentmodel,wh ile othersrelied solely
onothercueslike sentencelengthinwordsorcharacters.
Atthesametime,theIBMgroup,drawingdirectlyonalgorith msforspeechrecog-

nition (many of which had themselves been developed origina lly at IBM!) proposed
theCandide system, based on the IBM statistical models we have describe d (Brown Candide
et al., 1990, 1993). These papers described the probabilist ic model and the param-
eter estimation procedure. The decoding algorithm was neve r published, but it was
described in a patent ﬁling (Brown et al., 1995). The IBM work had a huge impact
on the research community, and by the turn of this century, mu ch or most academic

research on machine translation was statistical. Progress was made hugely easier by
the development of publicly-available toolkits, particul arly tools extended from the
EGYPT toolkit developed by the Statistical Machine Translation t eam in during the EGYPT
summer1999researchworkshopattheCenterforLanguageand SpeechProcessingat
theJohnsHopkinsUniversity. Theseincludethe GIZA++ aligner,developedbyFranz GIZA++
JosefOchbyextendingtheGIZAtoolkit(OchandNey,2003),w hichimplementsIBM
models1-5aswell astheHMMalignmentmodel.

MostearlyimplementationsfocusedonIBMModel3,butveryq uicklyresearchers
movedto phrase-basedmodels. While the earliest phrase-ba sedtranslationmodelwas
IBM Model4(Brownet al.,1993),modernmodelsderivefromOc h’s(1998)workon
alignment templates . Key phrase-basedtranslation modelsinclude Marcu and Won g
(2002), Zens et al. (2002). Venugopal et al. (2003), Koehn et al. (2003), Tillmann
(2003)OchandNey(2004),DengandByrne(2005),andKumaran dByrne(2005),
OtherworkonMT decodingincludesthe A∗decodersofWang andWaibel (1997)

and Germann et al. (2001), and the polynomial-time decoder f or binary-branching
stochastictransductiongrammarofWu(1996).
Themostrecentopen-sourceMTtoolkitisthephrase-based MOSESsystem(Koehn MOSES
et al., 2006;KoehnandHoang,2007;ZensandNey,2007). MOSE S developedoutof
thePHARAOH publicly available phrase-based stack decoder, developed by Philipp PHARAOH
Koehn (Koehn, 2004, 2003b), which extended the A∗decoders of (Och et al., 2001)
andBrownet al.(1995)andextendedthe EGYPTtoolsdiscusse dabove.
Modern research continues on sentence and word alignment as well; more recent

algorithmsincludeMoore(2002,2005),FraserandMarcu(20 05),Callison-Burchetal.
(2005),Liuet al.(2005).
Researchonevaluationofmachinetranslationbeganquitee arly. MillerandBeebe-
Center (1958) proposed a number of methods drawing on work in psycholinguistics.
These included the use of cloze and Shannon tasks to measure i ntelligibility, as well

DRAFTSection25.12. Advanced: Log-linearModelsforMT 921
as a metric of edit distance from a human translation, the int uition that underlies all
modern automatic evaluation metrics like Bleu. The ALPAC re port included an early
evaluationstudyconductedbyJohnCarrollthatwasextreme lyinﬂuential(Pierceetal.,
1966,Appendix10). Carroll proposeddistinct measuresfor ﬁdelity andintelligibility,
and had raters score them subjectively on 9-point scales. Mo re recent work on eval-
uation has focused on coming up with automatic metrics, incl ude the work on Bleu
discussed in Sec. 25.9.2(Papineniet al., 2002),as well as r elated measureslike NIST
(Doddington, 2002), TER (Translation Error Rate) (Snover et al., 2006), Precision
and Recall (Turianetal., 2003),and METEOR (BanerjeeandLavie,2005).
Good surveys of the early history of MT are Hutchins (1986) an d (1997). The
textbook by Hutchins and Somers (1992) has a wealth of exampl es of language phe-
nomenaandsummariesofhistoricallysigniﬁcantMTsystems . Nirenburgetal.(2002)

isacomprehensivecollectionofclassicreadingsinMT.Kni ght(1999b)isanexcellent
tutorialintroductionto Statistical MT.
AcademicpapersonmachinetranslationappearinstandardN LPjournalsandcon-
ferences,as well as in the journal MachineTranslation and in the proceedingsof vari-
ousconferences,includingMTSummit,organizedbytheInte rnationalAssociationfor
Machine Translation,the individualconferencesof its thr ee regionaldivisions, (Asso-
ciationforMT intheAmericas– AMTA,EuropeanAssociationf orMT– EAMT,and

Asia-Paciﬁc Association for MT – AAMT), and the Conference o n Theoretical and
MethodologicalIssueinMachineTranslation(TMI).
Exercises
25.1Select at random a paragraph of Ch. 12 which describes a fact a bout English
syntax. a) Describe and illustrate how your favorite foreig n language differs in
thisrespect. b)Explainhowa MTsystem coulddealwiththisd ifference.
25.2Chooseaforeignlanguagenovelina languageyouknow. Copyd owntheshort-
est sentence on the ﬁrst page. Now look up the rendition of tha t sentence in an

Exercises
Englishtranslationofthe novel. a) Forbothoriginalandtr anslation,drawparse
trees. b)For bothoriginaland translation,drawdependenc ystructures. c) Draw
acasestructurerepresentationofthemeaningwhichtheori ginalandtranslation
share. d) What does this exercise suggest to you regarding in termediate repre-
sentationsforMT?
25.3Version1(fornativeEnglishspeakers): Considerthefollo wingsentence:
Theseliesareliketheirfatherthatbegetsthem;grossasam ountain,open,
palpable. HenryIV, Part1,act 2,scene2
Translate this sentence into some dialect of modernvernacu larEnglish, such as

the style of a New York Times editorial, an Economist opinion piece, or your
favoritetelevisiontalk-showhost.
Version 2 (for native speakers of other languages): Transla te the following sen-
tenceintoyournativelanguage.

DRAFT922 Chapter 25. MachineTranslation
One night my friend Tom, who had just moved into a new apartmen t, saw
a cockroachscurryingaboutin thekitchen.
Foreitherversion,now:
a) Describe how you did the translation: What steps did you pe rform? In what
order did you do them? Which steps took the most time? b) Could you write a
programthatwouldtranslateusingthesamemethodsthatyou did? Whyorwhy
not? c)Whataspectswerehardestforyou? Wouldtheybehardf oraMTsystem?
d)WhataspectswouldbehardestforaMTsystem? aretheyhard forpeopletoo?
e) Which modelsare best for describingvariousaspects of yo urprocess(direct,
transfer, interlingua or statistical)? f) Now compare your translation with those
producedbyfriendsorclassmates. Whatisdifferent? Whywe rethetranslations
different?
25.4TypeasentenceintoanyMTsystemandseewhatitoutputs. a)L isttheproblems
with the translation. b) Rank these problemsin orderof seve rity. c) For the two
mostsevereproblems,suggesttheprobablerootcause.

25.5Build a very simple direct MT system for translating from som e language you
know at least somewhat into English (or into a language in whi ch you are rel-
atively ﬂuent), as follows. First, ﬁnd some good test senten ces in the source
language. Reserve half of these as a developmenttest set, an dhalf as an unseen
test set. Next, acquire a bilingual dictionary for these two languages (for many
languages, limited dictionaries can be found on the web that will be sufﬁcient

for this exercise). Your program should translate each word by looking up its
translation in your dictionary. You may need to implement so me stemming or
simplemorphologicalanalysis. Next,examineyouroutput, anddoapreliminary
erroranalysis on the developmenttest set. What are the majo rsources of error?
Write some generalrulesfor correctingthe translationmis takes. Youwill prob-
ably want to run a part-of-speechtagger on the English outpu t,if youhave one.
Thensee howwell yoursystemrunsonthetest set.

25.6Continuethe calculationsfor the EM exampleon page901,per formingthe sec-
ondandthirdroundofE-stepsandM-steps.
25.7(DerivedfromKnight(1999b))HowmanypossibleModel3alig nmentsarethere
between a 20-wordEnglish sentence and a 20-wordSpanish sen tence, allowing
forNULLandfertilities?

DRAFTBibliography
Abbreviationsandsymbols:
AAAI Proceedings of theNational Conference on Artiﬁcial In telligence
ACL Proceedings of theAnnual Conference of the Association for Computational Linguistics
CLS Papers from the Annual Regional Meeting of the Chicago Li nguistics Society
COGSCI Proceedings of theAnnual Conference of the Cognitiv e Science Society
COLINGX Proceedings of theInternational Conference on Com putational Linguistics
EMNLP Proceedings of theConference on Empirical Methods in Natural Language Processing
EUROSPEECH Proceedings of theEuropean Conference on Speec h Communication and Technology
INTERSPEECH Proceedings of theAnnual INTERSPEECHConfere nce
ICPhS Proceedings of theInternational Congress of Phoneti c Sciences
ICSLP Proceedings of theInternational Conference on Spoke n Language Processing
IEEEICASSP Proceedings of theIEEEInternational Conferen ce on Acoustics, Speech, &Signal Processing
NAACL-HLT Proceedings of theNorth American Chapter of the A CL/Human Language Technology Conference

IJCAI Proceedings of theInternational Joint Conference on Artiﬁcial Intelligence
Abney,S.(1996). Partialparsing viaﬁnite-state cascades. Nat-
uralLanguage Engineering ,2.
Abney, S. P. (1991). Parsing by chunks. In Berwick, R. C.,
Abney, S. P., and Tenny, C. (Eds.), Principle-Based Parsing:
Computation and Psycholinguistics , pp. 257–278. Kluwer,
Dordrecht.
Abney,S.P.(1997).Stochasticattribute-value grammars. Com-
putational Linguistics ,23(4), 597–618.
Abney, S.P.(2002). Bootstrapping. In ACL-02.

Abney, S. P. (2004). Understanding the Yarowsky algorithm.
Computational Linguistics ,30(3), 365–395.
Abney, S. P., McAllester, D. A., and Pereira, F. C. N. (1999a).
Relating probabilistic grammars and automata. In ACL-99.
Abney,S.P.,Schapire, R.E.,andSinger, Y.(1999b). Boosting
applied to tagging and PP attachment. In EMNLP/VLC-99 ,
College Park, MD,pp. 38–45.
Ades, A. E. and Steedman, M. (1982). On the order of words.
Linguistics and Philosophy ,4,517–558.

Adjukiewicz , K. (1935). Die syntaktische Konnexit¨ at. Studia
Philosophica ,1, 1–27. English translation “Syntactic Con-
nexion” by H. Weber in McCall, S. (Ed.) Polish Logic , pp.
207–231, Oxford University Press,Oxford, 1967.
Adriaans ,P.andvanZaanen,M.(2004). Computational gram-
marinduction forlinguists. Grammars; special issuewith the
theme “GrammarInduction” ,7,57–68.
Agichtein , E. and Gravano, L. (2000). Snowball: Extracting
relations from large plain-text collections. In Proceedings of

the5th ACM International Conference on Digital Libraries .
Agirre, E. and de Lacalle, O. L. (2003). Clustering WordNet
word senses. In RANLP2003 .
Agirre, E.and Edmonds,P.(Eds.).(2006). WordSenseDisam-
biguation: Algorithms and Applications . Kluwer.
Ahn, D., Adafre, S. F., and de Rijke, M. (2005). Extracting
temporal information from open domain text: A comparative
exploration. In Proceedings of the 5th Dutch-Belgian Infor-
mation Retrieval Workshop (DIR’05).

Aho, A. V., Sethi, R., and Ullman, J. D. (1986). Compilers:
Principles, Techniques, andTools . Addison-Wesley, Reading,
MA.Aho, A. V. and Ullman, J. D. (1972). The Theory of Parsing,
Translation, and Compiling , Vol. 1. Prentice-Hall.
A¨ ıt-Kaci , H.(1984). ALattice-Theoretic Approach toCompu-
tation Based on a Calculus ofPartially Ordered Types . Ph.D.
thesis, University of Pennsylvania.
Aist, G., Dowding, J., Hockey, B. A., and Hieronymus, J. L.
(2002). An intelligent procedure assistant for astronaut t rain-

ing and support. In ACL-02, Philadelphia, PA.
Albright, A. (2007). How many grammars am i holding up?
Discovering phonological differences between word classe s.
InWCCFL 26 ,pp. 34–42.
Albright,A.andHayes,B.(2003).Rulesvs.analogyinenglish
past tenses: A computational/experimental study. Cognition ,
90,119–161.
Alderete, J., Brasoveanu, A., Merchant, N., Prince, A., and
Tesar, B. (2005). Contrast analysis aids in the learning of
phonological underlying forms. In WCCFL 24 , pp.34–42.
Algoet, P.H.andCover, T.M.(1988). Asandwich proofofthe

Shannon-McMillan-Breiman theorem. The Annals of Proba-
bility,16(2), 899–909.
Allen, J. (1984). Towards a general theory of action and time.
Artiﬁcial Intelligence ,23(2), 123–154.
Allen, J. (1995). Natural Language Understanding . Benjamin
Cummings, Menlo Park, CA. 2nd ed.
Allen, J. and Core, M. (1997). Draft of DAMSL: Dialog act
markup in several layers. Unpublished manuscript.
Allen, J., Ferguson, G., and Stent, A. (2001). An architec-
ture for more realistic conversational systems. In IUI ’01:

Proceedings ofthe6th international conference onIntelli gent
user interfaces , Santa Fe, NM,pp. 1–8.ACM Press.
Allen, J. and Perrault, C. R. (1980). Analyzing intention in
utterances. Artiﬁcial Intelligence ,15, 143–178.
Allen, J., Hunnicut, M. S., and Klatt, D. H. (1987). From Text
to Speech: TheMITalk system . Cambridge University Press.
Allwood, J. (1995). An activity-based approach to pragmatics.
Gothenburg Papersin Theoretical Linguistics ,76.
Allwood, J.,Nivre, J.,andAhls´ en, E.(1992). Onthesemantics

and pragmatics of linguistic feedback. Journal of Semantics ,
9,1–26.
Alshawi, H. (1987). Memory and Context for Language Inter-
pretation. Cambridge University Press.
923

DRAFT924 Bibliography
Alshawi, H. (Ed.). (1992). The Core Language Engine . MIT
Press.
Althaus, E.,Karamanis, N.,and Koller, A.(2004). Computing
locally coherent discourses. In ACL-04.
Amsler, R. A. (1980). The Structure of the Merriam-Webster
Pocket Dictionary . Ph.D. thesis, University of Texas, Austin,
Texas.
Amsler,R.A.(1981). AtaxonomyofEnglishnounsandverbs.
InACL-81, Stanford, CA,pp. 133–138. ACL.
Anderson , M. J., Pierrehumbert, J. B., and Liberman, M. Y.
(1984). Improving intonational phrasing with syntactic in for-
mation. In IEEEICASSP-84 , pp.2.8.1–2.8.4.
Anderson , S. R. (1985). Phonology in the Twentieth Century .
Cambridge University Press.
Antworth , E. L. (1990). PC-KIMMO: A Two-level Processor
forMorphological Analysis . SummerInstitute of Linguistics,
Dallas, TX.
Aone,C.and Bennett, S.W.(1995). Evaluating automated and
manualacquisition of anaphora resolution strategies. In ACL-
95,Cambridge, MA,pp.122–129.
Appelt,D.E.,Hobbs,J.R.,Bear,J.,Israel,D.,Kameyama,M.,

Kehler, A.,Martin, D.,Myers, K.,and Tyson,M.(1995). SRI
International FASTUS system MUC-6 test results and analy-
sis. InProceedings ofthe Sixth Message Understanding Con-
ference(MUC-6) ,SanFrancisco, pp.237–248.MorganKauf-
mann.
Appelt, D. E. and Israel, D. (1997). ANLP-97 tuto-
rial: Building information extraction systems. Available as
www.ai.sri.com/˜appelt/ie-tutorial/ .
Archangeli , D. (1984). Underspeciﬁcation in Yawelmani
Phonology and Morphology . Ph.D.thesis, MIT.
Archangeli , D. (1997). Optimality theory: An introduction to

mann.
linguistics in the 1990s. In Archangeli, D. and Langendoen,
D.T.(Eds.), OptimalityTheory: AnOverview .Blackwell, Ox-
ford.
Arens,Y.,Granacki, J.,and Parker, A.(1987). Phrasalanalysis
oflongnounsequences. In ACL-87,Stanford,CA,pp.59–64.
ACL.
Argamon , S., Dagan, I., and Krymolowski, Y. (1998). A
memory-based approach tolearning shallow natural languag e
patterns. In COLING/ACL-98 , Montreal, pp.67–73.
Ariel, M. (2001). Accessibility theory: An overview. In
Sanders, T., Schilperoord, J., and Spooren, W. (Eds.), Text

ford.
ACL.
Representation: Linguistic and Psycholinguistic Aspects , pp.
29–87. Benjamins.
Ariel, M. (1990). Accessing Noun Phrase Antecedents . Rout-
ledge.
Artiles, J., Gonzalo, J., and Sekine, S. (2007). The SemEval-
2007WePSevaluation: Establishing abenchmark fortheweb
peoplesearchtask. In ProceedingsoftheFourthInternational
Workshop on Semantic Evaluations (SemEval-2007) , Prague,
Czech Republic, pp. 64–69.
Ashburner , M., Ball, C. A., Blake, J. A., Botstein, D., Butler,
H.,Cherry,J.M.,Davis,A.P.,Dolinski,K.,Dwight,S.S.,E p-

ledge.
pig,J.T.,Harris,M.A.,Hill,D.P.,Issel-Tarver,L.,Kasa rskis,
A.,Lewis, S.,Matese, J. C., Richardson, J. E.,Ringwald, M. ,
Rubin, G. M., and Sherlock, G. (2000). Gene ontology: tool
for theuniﬁcation of biology. Nature Genetics ,25(1), 25–29.Asher, N. (1993). Reference to Abstract Objects in Discourse .
SLAP50,Dordrecht, Kluwer.
Asher, N. and Lascarides, A. (2003). Logics of Conversation .
Atal, B. S. (1974). Effectiveness of linear prediction charac-
teristics of the speech wave for automatic speaker identiﬁc a-

tion and veriﬁcation. TheJournal ofthe Acoustical Society of
America,55(6), 1304–1312.
Atal,B.S.andHanauer,S.(1971). Speechanalysisandsynthe-
sisbyprediction ofthespeechwave. JournaloftheAcoustical
Society ofAmerica ,50,637–655.
Atkins, S. (1993). Tools for computer-aided corpus lexicog-
raphy: The Hector project. Acta Linguistica Hungarica ,41,
5–72.
Atkinson ,M.andDrew,P.(1979). OrderinCourt . Macmillan,
London.
Attar, R. and Fraenkel, A. S. (1977). Local feedback in full-
text retrieval systems. Journal ofthe ACM ,24(3), 398–417.

5–72.
London.
Aubert, X. and Ney, H. (1995). Large vocabulary continu-
ous speech recognition using word graphs. In IEEE ICASSP ,
Vol. 1,pp. 49–52.
Austin, J. L. (1962). How to Do Things with Words . Harvard
University Press.
Austin,S.,Schwartz,R.,andPlaceway, P.(1991).Theforward-
backward search algorithm. In IEEE ICASSP-91 , Vol. 1, pp.
697–700.
Aylett,M.P.(1999). Stochasticsuprasegmentals -relationships
between redundancy, prosodicstructureandsyllabledurat ion.
InICPhS-99 , San Francisco, CA.

697–700.
Aylett, M. P. and Turk, A. (2004). The smooth signal redun-
dancy hypothesis: A functional explanation for relationsh ips
between redundancy, prosodic prominence, and duration in
spontaneous speech. Language and Speech ,47(1), 31–56.
Baayen, R. H., Piepenbrock, R., and Gulikers, L. (1995). The
CELEXLexical Database (Release 2)[CD-ROM] . Linguistic
Data Consortium, University of Pennsylvania [Distributor ],
Philadelphia, PA.
Baayen, R. H.,Lieber, R., and Schreuder, R. (1997). Themor-

phological complexity of simplex nouns. Linguistics ,35(5),
861–877.
Baayen, R. H. and Sproat, R. (1996). Estimating lexical priors
for low-frequency morphologically ambiguous forms. Com-
putational Linguistics ,22(2), 155–166.
Babyonyshev , M. and Gibson, E. (1999). The complexity of
nested structures in Japanese. Language ,75(3), 423–450.
Bacchiani , M. and Roark, B. (2003). Unsupervised language
modeladaptation. In IEEEICASSP-03 , pp.224–227.
Bacchiani , M., Roark, B., and Saraclar, M. (2004). Language

861–877.
modeladaptation with MAPestimation andtheperceptron al-
gorithm. In HLT-NAACL-04 ,pp. 21–24.
Bach, E. (1976). An extension of classical transformational
grammar. In ProblemsofLinguistic Metatheory (Proceedings
ofthe 1976 Conference) . Michigan State University.
Bach, E.(1988). Categorial grammars as theories of language.
InOehrle,R.T.,Bach,E.,andWheeler, D.W.(Eds.), Catego-
rial Grammars and Natural Language Structures , pp. 17–34.
D.Reidel, Dordrecht.

Bachenko , J. and Fitzpatrick, E. (1990). A computational
grammar of discourse-neutral prosodic phrasing in English .
Computational Linguistics ,16(3), 155–170.

DRAFTBibliography 925
Backus, J. W. (1959). The syntax and semantics of the pro-
posed international algebraic language of the Zurich ACM-
GAMMConference. In InformationProcessing: Proceedings
of the International Conference on Information Processing ,
Paris,pp. 125–132. UNESCO.
Backus, J. W. (1996). Transcript of question and answer ses-
sion. In Wexelblat, R. L.(Ed.), HistoryofProgramming Lan-
guages, p.162. Academic Press.
Bacon, F. (1620). Novum Organum . Annotated edition edited
by Thomas Fowler published by Clarendon Press, Oxford,
1889.
Baeza-Yates , R. and Ribeiro-Neto, B. (1999). Modern Infor-
mation Retrieval . ACM Press,New York.
Bagley, W. C. (1900–1901). The apperception of the spoken
sentence: Astudy in thepsychology of language. TheAmeri-
can Journal ofPsychology ,12,80–130.
Bahl, L. R., Brown, P. F., de Souza, P. V., and Mercer, R. L.
(1986). Maximum mutual information estimation of hidden
Markov model parameters for speech recognition. In IEEE
ICASSP-86 ,Tokyo, pp.49–52.

1889.
Bahl, L. R., de Souza, P. V., Gopalakrishnan, P. S., Nahamoo,
D., and Picheny, M. A. (1992). A fast match for continu-
ous speech recognition using allophonic models. In IEEE
ICASSP-92 ,San Francisco, CA,pp. I.17–20.
Bahl, L. R. and Mercer, R. L. (1976). Part of speech assign-
mentbyastatistical decision algorithm. In Proceedings IEEE
International Symposium on Information Theory , pp. 88–89.
Bahl, L.R.,Jelinek, F.,and Mercer, R. L.(1983). A maximum
likelihood approach to continuous speech recognition. IEEE

Transactions on Pattern Analysis and Machine Intelligence ,
5(2),179–190.
Bailey, D., Feldman, J., Narayanan, S., and Lakoff, G. (1997).
Modeling embodied lexical development. In COGSCI-97 ,
Stanford, CA, pp. 19–24. Lawrence Erlbaum.
Bailey, T. and Hahn, U. (2001). Perception of wordlikeness:
Effectsofsegmentprobability andlength ontheprocessing of
nonwords. Journal ofMemory and Language ,44,568–591.
Baker, C.F.,Fillmore, C.J.,and Cronin, B.(2003). Thestruc-
ture of the framenet database. International Journal of Lexi-
cography,16:3, 281–296.

Baker, C. F., Fillmore, C. J., and Lowe, J. B. (1998). The
Berkeley FrameNet project. In COLING/ACL-98 , Montreal,
Canada, pp. 86–90.
Baker, J. K. (1979). Trainable grammars for speech recogni-
tion. In Klatt, D. H. and Wolf, J. J. (Eds.), Speech Communi-
cation Papers for the 97th Meeting of the Acoustical Society
ofAmerica , pp. 547–550.
Baker, J. K. (1975). The DRAGON system – An overview.
IEEETransactions onAcoustics, Speech, andSignalProcess -
ing,ASSP-23(1),24–29.
Baker, J. K. (1975/1990). Stochastic modeling for automatic

speech understanding. In Waibel, A. and Lee, K.-F. (Eds.),
Readings in Speech Recognition , pp.297–307. Morgan Kauf-
mann,LosAltos. Originally appeared in Speech Recognition ,
Academic Press, 1975.
Baldridge , J.,Asher, N.,and Hunter, J.(2007). Annotation for
androbustparsingofdiscoursestructureonunrestricted t exts.
Zeitschrift f¨ ur Sprachwissenschaft . In press.Banerjee , S. and Lavie, A. (2005). METEOR: An automatic
metric for Mt evaluation with improved correlation with hu-
manjudgments. In ProceedingsofACLWorkshoponIntrinsic

and Extrinsic Evaluation Measures for MT and/or Summa-
rization.
Banerjee , S. and Pedersen, T. (2003). Extended gloss over-
laps as ameasure of semantic relatedness. In IJCAI2003 , pp.
805–810.
Bangalore , S. and Joshi, A. K. (1999). Supertagging: An ap-
proach to almost parsing. Computational Linguistics ,25(2),
237–265.
Banko, M. and Moore, R. C. (2004). A study of unsupervised
part-of-speech tagging. In COLING-04 .
Bar-Hillel , Y. (1953). A quasi-arithmetical notation for syn-

rization.
805–810.
237–265.
tactic description. Language ,29,47–58. Reprinted in Y. Bar-
Hillel.(1964). LanguageandInformation: Selected Essayson
their Theory and Application , Addison-Wesley 1964, 61–74.
Bar-Hillel , Y. (1960). The present status of automatic transla-
tion of languages. In Alt, F. (Ed.), Advances in Computers 1 ,
pp.91–163. Academic Press.
Bar-Hillel , Y., Perles, M., and Shamir, E. (1961). On
formal properties of simple phrase structure grammars.
Zeitschriftf¨ urPhonetik,SprachwissenschaftundKommun ika-

tionsforschung ,14, 143–172. Reprinted in Y. Bar-Hillel.
(1964).Language and Information: Selected Essays on their
Theory and Application , Addison-Wesley 1964, 116–150.
Baroni, M., Matiasek, J., and Trost, H. (2002). Unsupervised
discovery of morphologically related words based on ortho-
graphic and semantic similarity. In Proceedings of ACL SIG-
PHON,Philadelphia, PA.
Barton, Jr., G. E., Berwick, R. C., and Ristad, E. S. (1987).
Computational Complexity and Natural Language . MIT
Press.

Press.
Barzilay, R. and Elhadad, M. (1997). Using lexical chains for
text summarization. In Proceedings of the ACL Workshop on
Intelligent Scalable Text Summarization , pp. 10–17.
Barzilay, R., Elhadad, N., and McKeown, K. R. (2002). Infer-
ring strategies for sentence ordering in multidocument new s
summarization. Journal of Artiﬁcial Intelligence Research ,
17,35–55.
Barzilay, R.andLapata,M.(2005). Modelinglocalcoherence:
an entity-based approach. In ACL-05, pp.141–148.
Barzilay, R.andLapata,M.(2007). Modelinglocalcoherence:

17,35–55.
an entity-based approach. Computational Linguistics . To ap-
pear.
Barzilay, R. and Lee, L. (2003). Learning to paraphrase: an
unsupervisedapproachusingmultiple-sequence alignment . In
HLT-NAACL-03 ,pp. 16–23.
Barzilay, R. and Lee, L. (2004). Catching the drift: Proba-
bilistic content models, with applications to generation a nd
summarization. In HLT-NAACL-04 ,pp. 113–120.
Barzilay, R. and McKeown, K.R. (2005). Sentence fusion for
multidocument newssummarization. Computational Linguis-
tics,31(3), 297–328.

pear.
Bates, R. (1997). The corrections ofﬁcer: Can John Kidd save
Ulysses.Lingua Franca ,7(8). October.
Bauer,L.(1983). Englishword-formation . CambridgeUniver-
sity Press.
Baum, L. F. (1900). The Wizard of Oz . Available at Project
Gutenberg.

DRAFT926 Bibliography
Baum, L. E. (1972). An inequality and associated maximiza-
tion technique in statistical estimation for probabilisti c func-
tions of Markov processes. In Shisha, O. (Ed.), Inequalities
III:Proceedings oftheThirdSymposiumonInequalities , Uni-
versity of California, Los Angeles, pp. 1–8. Academic Press .
Baum,L.E.andEagon,J.A.(1967). Aninequality withappli-
cations to statistical estimation for probabilistic funct ions of
Markov processes and to a model for ecology. Bulletin of the
American Mathematical Society ,73(3), 360–363.
Baum,L.E.andPetrie,T.(1966).Statisticalinferenceforprob -
abilistic functions of ﬁnite-state Markov chains. Annals of
Mathematical Statistics ,37(6), 1554–1563.
Baumgartner , Jr., W. A., Cohen, K. B., Fox, L., Acquaah-
Mensah, G. K., and Hunter, L. (2007). Manual curation is
not sufﬁcient for annotation of genomic databases. Bioinfor-
matics,23, i41–i48.
Baumgartner , Jr., W. A., Lu, Z., Johnson, H. L., Caporaso,

J. G., Paquette, J., Lindemann, A., White, E. K., Medvedeva,
O., Cohen, K. B., and Hunter, L. (2006). An integrated ap-
proach to concept recognition inbiomedical text. In Proceed-
ings ofBioCreative 2006 .
Bayes, T. (1763). An Essay Toward Solving a Problem in the
Doctrine of Chances , Vol. 53. Reprinted in Facsimiles of
twopapersbyBayes ,HafnerPublishingCompany,NewYork,
1963.
Bazell, C.E.(1952/1966). Thecorrespondence fallacy instruc-
tural linguistics. In Hamp, E. P., Householder, F. W., and

1963.
Austerlitz, R. (Eds.), Studies by Members of the English De-
partment, Istanbul University (3), reprinted in Readings i n
Linguistics II (1966) , pp. 271–298. University of Chicago
Press,Chicago.
Bean, D. and Riloff, E. (1999). Corpus-based identiﬁcation of
non-anaphoric noun phrases. In ACL-99,pp. 373–380.
Bean, D. and Riloff, E. (2004). Unsupervised learning of con-
textual role knowledge for coreference resolution. In HLT-
NAACL-04 .
Becker(1975). The phrasal lexicon. In Schank, R. and Nash-

NAACL-04 .
Webber, B.L.(Eds.), Theoretical Issues in Natural Language
Processing . ACL,Cambridge, MA.
Beckman , M.E.andAyers,G.M.(1997). Guidelines forToBI
labelling..
Beckman , M. E. and Hirschberg, J. (1994). The ToBI annota-
tion conventions. Manuscript, Ohio State University.
Beckman , M. E.and Pierrehumbert, J. B. (1986). Intonational
structure in English and Japanese. Phonology Yearbook ,3,
255–310.
Beeferman , D., Berger, A.,and Lafferty, J.D. (1999). Statisti-
cal Models for Text Segmentation. Machine Learning ,34(1),

255–310.
177–210.
Beesley, K. R. (1996). Arabic ﬁnite-state morphological anal-
ysisand generation. In COLING-96 , Copenhagen, pp.89–94.
Beesley, K. R. and Karttunen, L. (2000). Finite-state non-
concatenative morphotactics. In Proceedings of ACL SIG-
PHON,Luxembourg, pp.50–59.
Beesley, K.R.andKarttunen, L.(2003). Finite-State Morphol-
ogy. CSLI Publications, Stanford University.
Bell, A., Jurafsky, D., Fosler-Lussier, E., Girand, C., Gregory ,
M. L., and Gildea, D. (2003). Effects of disﬂuencies, pre-

177–210.
dictability, and utterance position on word form variation inEnglish conversation. Journal of the Acoustical Society of
America,113(2), 1001–1024.
Bellegarda , J. R. (2000). Exploiting latent semantic infor-
mation in statistical language modeling. Proceedings of the
IEEE,89(8), 1279–1296.
Bellegarda , J.R.(1999).Speechrecognition experimentsusing
multi-span statistical language models. In IEEE ICASSP-99 ,
pp.717–720.
Bellegarda , J.R.(2005). Unsupervised, language-independent

grapheme-to-phoneme conversion by latent analogy. Speech
Communication ,46(2), 140–152.
Bellman, R. (1957). Dynamic Programming . Princeton Uni-
versity Press, Princeton, NJ.
Bennett, C. (2005). Large scale evaluation of corpus-based
synthesizers: Results and lessons from the blizzard challe nge
2005. InEUROSPEECH-05 .
Benoˆ ıt, C., Grice, M., and Hazan, V. (1996). The SUS test: A
method for the assessment of text-to-speech synthesis inte lli-
gibility using Semantically Unpredictable Sentences. Speech
Communication ,18(4), 381–392.

Berger, A., Della Pietra, S. A., and Della Pietra, V. J. (1996).
Amaximumentropyapproachtonaturallanguageprocessing.
Computational Linguistics ,22(1), 39–71.
Berger, A. and Miller, R. (1998). Just-in-time language mod-
eling. InIEEEICASSP-98 ,Vol. II,pp. 705–708.
Bergsma, S.andLin,D.(2006). Bootstrapping path-based pro-
noun resolution. In COLING/ACL 2006 , Sydney, Australia,
pp.33–40.
Bestgen,Y.(2006). ImprovingTextSegmentation UsingLatent
Semantic Analysis: A Reanalysis of Choi, Wiemer-Hastings,

pp.33–40.
and Moore (2001). Computational Linguistics ,32(1), 5–12.
Bever, T. G. (1970). The cognitive basis for linguistic struc-
tures. In Hayes, J. R. (Ed.), Cognition and the Development
ofLanguage , pp.279–352. Wiley.
Biber, D., Johansson, S., Leech, G., Conrad, S., and Finegan,
E.(1999). LongmanGrammarofSpokenandWrittenEnglish .
Pearson ESL,Harlow.
Bickel, B. (2003). Referential density in discourse and syntac-
tic typology. Language ,79(2), 708–736.
Bies, A., Ferguson, M., Katz, K., and MacIntyre, R. (1995).

Bracketing guidelines for Treebank II style Penn Treebank
Project..
Bikel, D. M. (2004). Intricacies of Collins’ parsing model.
Computational Linguistics ,30(4), 479–511.
Bikel, D. M., Miller, S., Schwartz, R., and Weischedel, R.
(1997). Nymble: a high-performance learning name-ﬁnder.
InProceedings of ANLP-97 ,pp.194–201.
Bikel, D. M., Schwartz, R., and Weischedel, R. (1999). An al-
gorithm that learns what’s in a name. Machine Learning ,34,
211–231.

Project..
211–231.
Bilmes, J. (1997). A gentle tutorial on the EM algorithm and
its application to parameter estimation for gaussian mixtu re
andhiddenmarkovmodels.Tech.rep.ICSI-TR-97-021,ICSI,
Berkeley.
Bilmes, J. (2003). Buried Markov Models: A graphical-
modeling approach to automatic speech recognition. Com-
puter Speech and Language ,17(2-3).

DRAFTBibliography 927
Bilmes, J. and Bartels, C. (2005). Graphical model architec-
tures for speech recognition. IEEE Signal Processing Maga-
zine,22(5), 89–100.
Bird, S. and Ellison, T. M. (1994). One-level phonology:
Autosegmental representations and rules as ﬁnite automata .
Computational Linguistics ,20(1).
Bird, S. and Loper, E. (2004). NLTK: The Natural Language
Toolkit. In Proceedings of the ACL 2004 demonstration ses-
sion,Barcelona, Spain, pp. 214–217.
Birnbaum , L. and Selfridge, M. (1981). Conceptual analysis
of natural language. In Schank, R. C. and Riesbeck, C. K.
(Eds.),Inside Computer Understanding: Five Programs plus
Miniatures , pp.318–353. Lawrence Erlbaum.
Bisani, M. and Ney, H. (2002). Investigations on joint-
multigram models for grapheme-to-phoneme conversion. In
ICSLP-02 ,Vol. 1,pp. 105–108.
Black, A. W. and Taylor, P. (1994). CHATR: a generic speech
synthesis system. In COLING-94 , Kyoto, Vol. II, pp. 983–
986.
Black, A. W. and Hunt, A. J. (1996). Generating F0 contours

986.
fromToBIlabelsusinglinear regression. In ICSLP-96 ,Vol.3,
pp.1385–1388.
Black, A. W.,Lenzo, K.,and Pagel, V. (1998). Issues in build-
ing general letter to sound rules. In 3rd ESCA Workshop on
Speech Synthesis, Jenolan Caves, Australia .
Black, A. W., Taylor, P., and Caley, R. (1996-1999).
The Festival Speech Synthesis System system. Man-
ual and source code available at www.cstr.ed.ac.uk/
projects/festival.html .
Black, A.W.and Tokuda, K.(2005). TheBlizzard Challenge–
2005: Evaluating corpus-based speech synthesis on common

datasets. In EUROSPEECH-05 .
Black, E.,Abney, S.P.,Flickinger, D., Gdaniec, C.,Grishman,
R., Harrison, P., Hindle, D., Ingria, R., Jelinek, F., Klava ns,
J. L., Liberman, M. Y., Marcus, M. P., Roukos, S., Santorini,
B., and Strzalkowski, T. (1991). A procedure for quantita-
tively comparing thesyntactic coverage ofEnglishgrammar s.
InProceedings DARPASpeech and Natural Language Work-
shop,Paciﬁc Grove, CA,pp. 306–311. Morgan Kaufmann.
Black, E.(1988). An experiment in computational discrimina-

tion of English word senses. IBM Journal of Research and
Development ,32(2), 185–194.
Black,E.,Jelinek,F.,Lafferty,J.D.,Magerman,D.M.,Mercer,
R. L., and Roukos, S. (1992). Towards history-based gram-
mars: Using richer models for probabilistic parsing. In Pro-
ceedings DARPA Speech and Natural Language Workshop ,
Harriman, NY, pp.134–139. Morgan Kaufmann.
Blair, C. R. (1960). A program for correcting spelling errors.
Information and Control ,3,60–67.
Blair-Goldensohn , S., McKeown, K. R., and Schlaikjer, A.H.

(2004). Answeringdeﬁnitional questions: Ahybridapproac h.
InMaybury,M.T.(Ed.), NewDirections inQuestion Answer-
ing, pp.47–58. AAAI Press.
Blaschke , C., Leon, E. A., Krallinger, M., and Valencia, A.
(2005). Evaluation ofBioCreative assessmentoftask2. BMC
Bioinformatics ,6(2).
Bledsoe, W. W. and Browning, I. (1959). Pattern recognition
and reading by machine. In 1959 Proceedings of the Eastern
Joint Computer Conference , pp. 225–232. Academic.Blei, D.M.,Ng,A.Y.,andJordan, M.I.(2003). LatentDirich-
let allocation. Journal of Machine Learning Research ,3(5),

993–1022.
Blevins, J. (1995). The handbook of phonological theory.
In Goldsmith, J. (Ed.), The syllable in phonological theory .
Blackwell, Oxford.
Blitzer, J., McDonald, R., and Pereira, F. C. N. (2006). Do-
main adaptation with structural correspondence learning. In
Proceedings of the Conference on Empirical Methods in Nat-
uralLanguage Processing , Sydney, Australia.
Bloomﬁeld , L. (1914). An Introduction to the Study of Lan-
guage. Henry Holt and Company, New York.

993–1022.
Bloomﬁeld , L.(1933). Language . University ofChicagoPress,
Chicago.
Blunsom , P.and Baldwin, T.(2006). Multilingual deep lexical
acquisition for hpsgs via supertagging. In EMNLP2006 .
Bobrow, D. G., Kaplan, R. M., Kay, M., Norman, D. A.,
Thompson, H., and Winograd, T. (1977). GUS, A frame
driven dialog system. Artiﬁcial Intelligence ,8,155–173.
Bobrow, D. G. and Winograd, T. (1977). An overview of
KRL, a knowledge representation language. Cognitive Sci-
ence,1(1), 3–46.
Bobrow, R.J.and Webber, B. L.(1980). Knowledge represen-

Chicago.
tation for syntactic/semantic processing. In AAAI-80, Stan-
ford, CA,pp. 316–323. Morgan Kaufmann.
Bock, K. and Loebell, H. (1990). Framing sentences. Cogni-
tion,35, 1–39.
Bod, R. (1993). Using an annotated corpus as a stochastic
grammar. In EACL-93,pp. 37–44.
Boersma, P.and Hayes, B.(2001). Empirical tests of thegrad-
ual learning algorithm. Linguistic Inquiry ,32,45–86.
Boersma, P. and Weenink, D. (2005). Praat: doing phonetics
bycomputer(version4.3.14).[Computerprogram].Retriev ed
May 26, 2005, fromhttp://www.praat.org/.

Boguraev , B. and Briscoe, T. (Eds.). (1989). Computational
Lexicography for Natural Language Processing . Longman,
London.
Bohus, D. and Rudnicky, A. I. (2005). Sorry, I Didn’t Catch
That! -AnInvestigation ofNon-understanding ErrorsandRe -
covery Strategies. In SIGdial-2005 , Lisbon, Portugal.
Bolinger, D. (1972). Accent is predictable (if you’re a mind-
reader).Language ,48(3), 633–644.
Bolinger,D.(1981).Twokindsofvowels,twokindsofrhythm.
Indiana University Linguistics Club.
Booth, T.L.(1969). Probabilistic representation of formallan-

London.
guages. In IEEEConference Recordofthe1969TenthAnnual
Symposium on Switching and Automata Theory , pp.74–81.
Booth, T. L. and Thompson, R. A. (1973). Applying proba-
bility measures to abstract languages. IEEE Transactions on
Computers ,C-22(5), 442–450.
Boser, B. E., Guyon, I. M., and Vapnik, V. N. (1992). A train-
ingalgorithm foroptimalmarginclassiﬁers. In COLT’92 ,pp.
144–152.
Bourlard , H. and Morgan, N. (1994). Connectionist Speech
Recognition: AHybrid Approach . Kluwer Press.

144–152.
Bouwman , G., Sturm, J., and Boves, L. (1999). Incorporating
conﬁdence measures in the Dutch train timetable informatio n
system developed in the Arise project. In IEEE ICASSP-99 ,
pp.493–496.

DRAFT928 Bibliography
Brachman ,R.J.(1979). Ontheepistemogical statusofseman-
tic networks. In Findler, N. V. (Ed.), Associative Networks:
Representation and Use of Knowledge by Computers , pp. 3–
50.Academic Press.
Brachman , R. J. and Levesque, H.J. (Eds.). (1985). Readings
in Knowledge Representation . Morgan Kaufmann.
Brachman , R. J. and Schmolze, J. G. (1985). An overview
of the KL-ONE knowledge representation system. Cognitive
Science,9(2),171–216.
Branco, A., McEnery, T., and Mitkov, R. (2002). Anaphora
Processing . John Benjamins.
Brants, T. (2000). TnT: a statistical part-of-speech tagger. In
Proceedings of the sixth conference on Applied natural lan-
guage processing , Seattle, WA, pp. 224–231. Morgan Kauf-
mann.
Brants, T., Chen, F., and Tsochantaridis, I. (2002). Topic-
baseddocumentsegmentationwithprobabilisticlatentsem an-
tic analysis. In CIKM ’02: Proceedings of the eleventh inter-
national conference on Information and knowledge manage-
ment, pp.211–218.

mann.
Breiman, L., Friedman, J. H., Olshen, R. A., and Stone, C. J.
(1984).Classiﬁcation and Regression Trees . Wadsworth &
Brooks, Paciﬁc Grove, CA.
Brennan, S. E. (1995). Centering attention in discourse. Lan-
guage and Cognitive Processes ,10, 137–167.
Brennan, S. E., Friedman, M. W., and Pollard, C. (1987). A
centering approach to pronouns. In ACL-87, Stanford, CA,
pp.155–162.
Brent,M.R.(1999). Anefﬁcient, probabilistically soundalgo-
rithm for segmentation and word discovery. Machine Learn-
ing,34(1–3), 71–105.

Bresnan,J.(Ed.).(1982). TheMentalRepresentation ofGram-
matical Relations . MITPress.
Bresnan, J. and Kaplan, R. M. (1982). Introduction: Gram-
mars as mental representations of language. In Bresnan, J.
(Ed.),The Mental Representation of Grammatical Relations .
MITPress.
Brew, C. (1992). Letting the cat out of the bag: generation for
shake-and-bake mt. In COLING-92 , pp. 610–616.
Brill, E. (1995). Transformation-based error-driven learning
and natural language processing: A case study in part-of-
speech tagging. Computational Linguistics ,21(4), 543–566.

MITPress.
Brill,E.(1997). Unsupervisedlearningofdisambiguation rules
for partof speech tagging. Unpublished manuscript.
Brill, E.,Dumais, S. T., and Banko, M. (2002). An analysis of
the AskMSR question-answering system. In EMNLP 2002 ,
pp.257–264.
Brill,E.andMoore,R.C.(2000). Animprovederrormodelfor
noisychannelspellingcorrection. In ACL-00,HongKong,pp.
286–293.
Brill,E.andResnik,P.(1994). Arule-basedapproachtoprepo-
sitional phrase attachment disambiguation. In COLING-94 ,
Kyoto, pp. 1198–1204.

286–293.
Brill, E. and Wu, J. (1998). Classiﬁer combination for im-
proved lexical disambiguation. In COLING/ACL-98 , Mon-
treal, Canada, pp. 191–195.Brin, S. (1998). Extracting patterns and relations from the
World Wide Web. In Proceedings World Wide Web and
Databases International Workshop, Number 1590 in LNCS ,
pp.172–183. Springer.
Briscoe, T.andCarroll,J.(1993). Generalized ProbabilisticLR
parsing of natural language (corpora) with uniﬁcation-bas ed
grammars. Computational Linguistics ,19(1), 25–59.

Bromberger , S. and Halle, M. (1989). Why phonology is dif-
ferent.Linguistic Inquiry ,20, 51–70.
Brookes, D.M.and Loke,H.P.(1999). Modelling energy ﬂow
inthevocaltractwithapplications toglottalclosureando pen-
ing detection. In IEEEICASSP-99 ,pp. 213–216.
Broschart , J.(1997). WhyTongan doesitdifferently. Linguis-
tic Typology ,1,123–165.
Browman , C. P. and Goldstein, L. (1986). Towards an articu-
latory phonology. Phonology Yearbook ,3,219–252.
Browman ,C.P.andGoldstein, L.(1992). Articulatory phonol-

ogy: An overview. Phonetica ,49, 155–180.
Browman ,C.P.andGoldstein,L.(1989).Articulatory gestures
as phonological units. Phonology ,6, 201–250.
Browman , C.P.andGoldstein, L.(1995). Dynamics andartic-
ulatory phonology. In Port, R. and v. Gelder, T. (Eds.), Mind
as Motion: Explorations in the Dynamics of Cognition , pp.
175–193. MIT Press.
Brown, J. S. and Burton, R. R. (1975). Multiple representa-
tions of knowledge for tutorial reasoning. In Bobrow, D. G.

and Collins, A. (Eds.), Representation and Understanding ,
pp.311–350. Academic Press.
Brown, P. F., Cocke, J., Della Pietra, S. A., Della Pietra, V. J.,
Jelinek, F., Lai, J. C., and Mercer, R. L. (1995). Method
and system for natural language translation. U.S. Patent
5,477,451.
Jelinek, F., Lafferty, J. D., Mercer, R. L., and Roossin, P. S .
(1990). A statistical approach to machine translation. Com-
putational Linguistics ,16(2), 79–85.

Brown, P. F., Della Pietra, S. A., Della Pietra, V. J., Lai, J. C.,
and Mercer, R. L. (1992). An estimate of an upper bound
for the entropy of English. Computational Linguistics ,18(1),
31–40.
Brown,P.F.,DellaPietra,S.A.,DellaPietra,V.J.,andMercer,
R. L. (1993). The mathematics of statistical machine transl a-
tion: Parameter estimation. Computational Linguistics ,19(2),
263–311.
Brown, P. F.,Della Pietra, V. J., de Souza, P. V., Lai, J. C., and
Mercer, R. L. (1992). Class-based n-gram models of natural

31–40.
263–311.
language. Computational Linguistics ,18(4), 467–479.
Brown, P. F., Lai, J. C., and Mercer, R. L. (1991). Aligning
sentences in parallel corpora. In ACL-91, Berkeley, CA, pp.
169–176.
Bruce, R. and Wiebe, J. (1994). Word-sense disambiguation
using decomposable models. In ACL-94, Las Cruces, NM,
pp.139–145.
Budanitsky , A. and Hirst, G. (2001). Semantic distance in
WordNet: An experimental, application-oriented evaluati on
of ﬁve measures. In Proceedings of the NAACL 2001 Work-

169–176.
shop on WordNet and Other Lexical Resources , Pittsburgh,
PA.

DRAFTBibliography 929
Budanitsky , A. and Hirst, G. (2006). Evaluating WordNet-
based measures of lexical semantic relatedness. Computa-
tional Linguistics ,32(1), 13–47.
Bulut1, M., Narayanan, S. S., and Syrdal, A. K. (2002). Ex-
pressive speech synthesis using a concatenative synthesiz er.
InICSLP-02 .
Bulyko, I., Kirchhoff, K., Ostendorf, M., and Goldberg, J.
(2005). Error-sensitive response generation in a spoken la n-
gugagedialoguesystem. SpeechCommunication ,45(3),271–
288.
Bulyko, I. and Ostendorf, M.(2001). Unit selection for speech
synthesis using splicing costs with weighted ﬁnite state tr ans-
ducers. In EUROSPEECH-01 ,Vol. 2,pp. 987–990.
Bulyko, I., Ostendorf, M., and Stolcke, A. (2003). Getting
moremileagefromwebtextsourcesforconversational speec h
language modeling using class-dependent mixtures. In HLT-
NAACL-03 ,Edmonton, Canada, Vol. 2,pp. 7–9.
Bunescu, R. C. and Mooney, R. J. (2005). A shortest path de-
pendency kernel for relation extraction. In Proceedings ofthe

conference on Human Language Technology and Empirical
Methods in Natural Language Processing , pp.724–731.
Bunt, H. (1994). Context and dialogue control. Think,3, 19–
31.
Bunt, H. (2000). Dynamic interpretation and dialogue theory,
volume 2. In Taylor, M. M., Neel, F., and Bouwhuis, D. G.
(Eds.),The structure of multimodal dialogue , pp. 139–166.
John Benjamins, Amsterdam.
Bybee, J. L. (2000). The phonology of the lexicon: evidence
from lexical diffusion. In Barlow, M. and Kemmer, S. (Eds.),

31.
Usage-based Models of Language , pp. 65–85. CSLI, Stan-
ford.
Byrd, R. J. and Chodorow, M. S. (1985). Using an On-Line
dictionary to ﬁnd rhyming words and pronunciations for un-
known words. In ACL-85, pp. 277–283.
Cahn, J. E. (1990). The generation of affect in synthesized
speech. In Journal of the American Voice I/O Society , Vol. 8,
pp.1–19.
Calder, J. (1987). Typed uniﬁcation for natural language pro-
cessing. In Kahn, G., MacQueen, D., and Plotkin, G. (Eds.),

ford.
pp.1–19.
Categories, Polymorphism, and Uniﬁcation . Centre for Cog-
nitiveScience, University ofEdinburgh,Edinburgh,Scotl and.
Callison-Burch , C., Osborne, M., and Koehn, P. (2006). Re-
evaluating the role of BLEU in machine translation research .
InEACL-06.
Callison-Burch , C., Talbot, D., and Osborne, M. (2005). Sta-
tisticalmarchinetranslation withword-andsentences-al igned
parallel corpora. In ACL-05,pp. 176–183.
Cao, X. (1792). The Story of the Stone. (Also known as The
DreamoftheRedChamber ). Penguin Classics, London. First

publishedinChinesein1792,translatedintoEnglishbyDav id
Hawkes and published by Penguin in 1973.
Caraballo , S. A. (1999). Automatic construction of a
hypernym-labeled noun hierarchy from text. In ACL-99, Col-
lege Park, MD.
Caraballo , S.A.(2001). AutomaticAcquisition ofahypernym-
labelednounhierarchyfromtext . Ph.D.thesis,BrownUniver-
sity.Caramazza , A., Grober, E., Garvey, C., and Yates, J. (1977).
Comprehension of anaphoric pronouns. Journal of Verbal
Learning and VerbalBehaviour ,16, 601–609.

Carberry , S. (1990). Plan Recognition in Natural Language
Dialog. MITPress.
Carbonell , J. (1982). Metaphor: An inescapable phenomenon
in natural language comprehension. In Lehnert, W. G. and
Ringle, M. (Eds.), Strategies for Natural Language Process-
ing, pp.415–434. Lawrence Erlbaum.
Carbonell , J.,Cullingford, R. E.,and Gershman, A. V. (1981).
Steps toward knowledge-based machine translation. IEEE
3(4), 376–392.

Carbonell , J. and Goldstein, J. (1998). The use of mmr,
diversity-based reranking for reordering documents and pr o-
ducing summaries. In SIGIR 1998 , pp. 335–336.
Carbonell , J., Mitamura, T., and Nyberg, E. H. (1992). The
KANTperspective: A critique of pure transfer (and pure in-
terlingua, pure statistics, ...). In International Conference on
Theoretical and Methodological Issues in Machine Transla-
tion.
Cardie, C. and Wagstaff, K. (1999). Noun phrase coreference
as clustering. In EMNLP/VLC-99 , College Park, MD.

tion.
Cardie,C.(1993). Acase-based approach toknowledgeacqui-
sition for domain speciﬁc sentence analysis. In AAAI-93, pp.
798–803. AAAI Press.
Cardie, C. (1994). Domain-Speciﬁc Knowledge Acquisition
forConceptual Sentence Analysis . Ph.D.thesis, University of
Massachusetts, Amherst, MA. Available as CMPSCI Techni-
cal Report 94-74.
Cardie, C., Daelemans, W., Ndellec, C., and Sang, E. T. K.
(Eds.).(2000). ProceedingsoftheFourthConferenceonCom-
putational Language Learning , Lisbon, Portugal.
Carletta, J., Dahlb¨ ack, N., Reithinger, N., and Walker, M. A.

(1997a). Standards for dialogue coding in natural language
processing. Tech. rep. Report no. 167, Dagstuhl Seminars.
Report from Dagstuhl seminar number 9706.
Carletta, J., Isard, A., Isard, S., Kowtko, J. C., Doherty-
Sneddon, G., and Anderson, A. H. (1997b). The reliability
of a dialogue structure coding scheme. Computational Lin-
guistics,23(1), 13–32.
Carlson, L. and Marcu, D. (2001). Discourse tagging manual.
Tech. rep. ISI-TR-545, ISI.
Carlson, L., Marcu, D., and Okurowski, M. E. (2001). Build-

ing a discourse-tagged corpus in the framework of rhetorica l
structure theory. In Proceedings ofSIGDIAL .
Carlson, L., Marcu, D., and Okurowski, M. E. (2002). Build-
structure theory. In van Kuppevelt, J. and Smith, R. (Eds.),
CurrentDirections in Discourse and Dialogue . Kluwer.
Carpenter , B. (1991). The generative power of categorial
grammars and head-driven phrase structure grammars with
lexical rules. Computational Linguistics ,17(3), 301–313.

Carpenter , B. (1992). The Logic of Typed Feature Structures .
Carpenter , B. and Penn, G. (1994). The Attribute Logic En-
gineUsers’sGuideVersion2.0.1. Tech.rep.,CarnegieMell on
University.

DRAFT930 Bibliography
Carpuat, M. and Wu, D. (2007). Improving statistical
machine translation using word sense disambiguation. In
EMNLP/CoNLL 2007 , Prague, Czech Republic, pp. 61–72.
Carroll,G.andCharniak,E.(1992).Twoexperimentsonlearn-
ing probabilistic dependency grammars from corpora. Tech.
rep. CS-92-16, Brown University.
Carroll, J.,Briscoe, T.,and Sanﬁlippo, A.(1998). Parser eval-
uation: a survey and a new proposal. In LREC-98 , Granada,
Spain, pp. 447–454.
Cedergren , H. J. and Sankoff, D. (1974). Variable rules: per-
formance as astatistical reﬂection of competence. Language ,
50(2), 333–355.
Chafe, W. L. (1976). Givenness, contrastiveness, deﬁniteness,
subjects, topics, and point of view. In Li, C. N.(Ed.), Subject
and Topic , pp. 25–55. Academic Press.
Chan, Y. S., Ng, H. T., and Chiang, D. (2007). Word sense
disambiguation improves statistical machine translation . In
ACL-07,Prague, Czech Republic, pp. 33–40.
Chandioux , J. (1976). M ´ET´EO: un syst` eme op´ erationnel pour

la traduction automatique des bulletins m´ et´ eorologique s des-
tin´ es au grand public. Meta,21, 127–133.
Chandler ,S.(1991). Metaphor comprehension: Aconnection-
istapproach toimplications forthemental lexicon. Metaphor
and Symbolic Activity ,6(4), 227–258.
Chang, N., Gildea, D., and Narayanan, S. (1998). A dynamic
model of aspectual composition. In COGSCI-98 , Madison,
WI,pp.226–231. Lawrence Erlbaum.
Charniak , E. (1993). Statistical Language Learning . MIT
Press.
Charniak , E. and Johnson, M. (2005). Coarse-to-ﬁne n-best

Press.
parsing and MaxEnt discriminative reranking. In ACL-05,
AnnArbor.
Charniak , E. and Shimony, S. E. (1990). Probabilistic seman-
tics for cost based abduction. In Dietterich, T. S. W. (Ed.),
AAAI-90,pp. 106–111. MIT Press.
Charniak , E. (1997). Statistical parsing with a context-free
grammar and word statistics. In AAAI-97, Menlo Park, pp.
598–603. AAAI Press.
Charniak , E. (2000). A maximum-entropy-inspired parser. In
NAACL2000 , Seattle, Washington, pp. 132–139.
Charniak , E. (2001). Immediate-head parsing for language

AnnArbor.
models. In ACL-01,Toulouse, France.
Charniak , E. and Goldman, R. (1988). A logic for semantic
interpretation. In ACL-88,Buffalo, NY.
Charniak , E., Hendrickson, C., Jacobson, N., and Perkowitz,
M.(1993). Equations forpart-of-speech tagging. In AAAI-93,
Washington, D.C.,pp. 784–789. AAAI Press.
Charniak , E.and McDermott, D.(1985). Introduction to Arti-
ﬁcial Intelligence . Addison Wesley.
Chelba, C. and Jelinek, F. (2000). Structured language model-
ing.Computer Speech and Language ,14,283–332.
Chen,J.N.andChang,J.S.(1998). TopicalclusteringofMRD

senses based on information retrieval techniques. Computa-
tional Linguistics ,24(1), 61–96.
Chen, J. and Rambow, O. (2003). Use of deep linguistic fea-
tures for the recognition and labeling of semantic argument s.
InEMNLP 2003 , pp.41–48.Chen,S.F.(2003). Conditional andjointmodelsforgrapheme-
to-phoneme conversion. In EUROSPEECH-03 .
Chen, S. F. and Goodman, J. (1996). An empirical study of
smoothing techniques for language modeling. In ACL-96,
Santa Cruz, CA, pp.310–318.
Chen, S. F. and Goodman, J. (1998). An empirical study of

smoothing techniques for language modeling. Tech. rep. TR-
10-98, Computer Science Group, Harvard University.
Chen, S. F. and Goodman, J. (1999). An empirical study
of smoothing techniques for language modeling. Computer
Speech and Language ,13(359–394).
Chen, S. F. and Rosenfeld, R. (2000). A survey of smoothing
techniques forMEmodels. IEEETransactions onSpeech and
Audio Processing ,8(1), 37–50.
Chen, S. F., Seymore, K., and Rosenfeld, R. (1998). Topic
adaptation for language modeling using unnormalized expo-

nential models. In IEEEICASSP-98 , pp.681–684. IEEE.
Chi, Z. (1999). Statistical Properties of Probabilistic Conte xt-
FreeGrammars. Computational Linguistics ,25(1), 131–160.
Chiang, D.(2005). A hierarchical phrase-based model for sta-
tistical machine translation. In ACL-05, Ann Arbor, MI, pp.
263–270.
Chierchia , G. and McConnell-Ginet, S. (1991). Meaning and
Grammar . MIT Press.
Chinchor , N., Hirschman, L., and Lewis, D. L. (1993). Eval-
uating Message Understanding systems: An analysis of the
third Message Understanding Conference. Computational

263–270.
Linguistics ,19(3), 409–449.
Chklovski , T. and Mihalcea, R. (2003). Exploiting Agreement
and Disagreement ofHuman Annotators for WordSenseDis-
ambiguation. In RANLP2003 .
Chklovski ,T.andMihalcea, R.(2002).Buildingasensetagged
corpus with open mind word expert. In ACL-02 Workshop on
Word Sense Disambiguation: Recent Successes and Future
Directions , pp. 116–122.
Chklovski , T. and Pantel, P. (2004). Verb ocean: Mining the
Web for ﬁne-grained semantic verb relations. In EMNLP
2004, pp.25–26.

Choi, F. Y. Y. (2000). Advances in domain independent linear
text segmentation. In NAACL2000 , pp. 26–33.
Choi, F.Y. Y., Wiemer-Hastings, P.,and Moore, J.(2001). La-
tentsemanticanalysisfortextsegmentation. In EMNLP2001 ,
pp.109–117.
Chomsky , N. (1956). Three models for the description of lan-
guage.IRE Transactions on Information Theory ,2(3), 113–
124.
Chomsky ,N.(1956/1975). TheLogical StructureofLinguistic
Theory. Plenum.
Chomsky , N. (1957). Syntactic Structures . Mouton, The
Hague.

124.
Hague.
Chomsky , N. (1959a). On certain formal properties of gram-
mars.Information and Control ,2, 137–167.
Chomsky , N. (1959b). A review of B. F. Skinner’s “Verbal
Behavior”. Language ,35, 26–58.
Chomsky ,N.(1963). Formalproperties ofgrammars. InLuce,
R.D.,Bush, R.,and Galanter, E.(Eds.), Handbook ofMathe-
matical Psychology , Vol. 2,pp. 323–418. Wiley.
Chomsky , N. (1965). Aspects of the Theory of Syntax . MIT
Press.

DRAFTBibliography 931
Chomsky , N. (1969). Quine’s empirical assumptions. In
Davidson, D. and Hintikka, J. (Eds.), Words and objections.
EssaysontheworkofW.V.Quine ,pp.53–68.D.Reidel, Dor-
drecht.
Chomsky , N. (1981). Lectures on Government and Binding .
Foris,Dordrecht.
Chomsky , N. and Halle, M. (1968). The Sound Pattern of En-
glish. Harper and Row.
Chomsky , N. and Miller, G.A. (1958). Finite-state languages.
Information and Control ,1,91–112.
Chomsky ,N.and Miller, G.A.(1963). Introduction to thefor-
mal analysis of natural languages. In Luce, R. D., Bush, R.,
and Galanter, E. (Eds.), Handbook of Mathematical Psychol-
ogy, Vol. 2, pp.269–322. Wiley.
Chou,W.,Lee,C.-H.,andJuang,B.H.(1993). Minimumerror
ratetraining based on n-beststring models. In IEEEICASSP-
93,pp. 2.652–655.
Christiansen , M. H., Allen, J., and Seidenberg, M. S. (1998).
Learning to segment speech using multiple cues: A connec-
tionist model. Language and Cognitive Processes ,13(2),
221–268.

221–268.
Chu-Carroll , J. (1998). A statistical model for discourse act
recognition in dialogue interactions. In Chu-Carroll, J. a nd
Green, N. (Eds.), Applying Machine Learning to Discourse
Processing. Papers from the 1998 AAAI Spring Symposium.
Tech. rep. SS-98-01, Menlo Park, CA, pp. 12–17. AAAI
Press.
Chu-Carroll , J. and Brown, M. K. (1997). Tracking initia-
tive in collaborative dialogue interactions. In ACL/EACL-97 ,
Madrid, Spain, pp.262–270.
Chu-Carroll , J. and Carberry, S. (1998). Collaborative re-

Press.
sponsegeneration in planning dialogues. Computational Lin-
guistics,24(3), 355–400.
Chu-Carroll , J. and Carpenter, B. (1999). Vector-based natu-
ral language call routing. Computational Linguistics ,25(3),
361–388.
Chung,G.(2004). Developing aﬂexible spoken dialog system
using simulation. In ACL-04, Barcelona, Spain.
Church, A. (1940). A formulation of asimple theory of types.
Journal ofSymbolic Logic ,5, 56–68.
Church, K. W. (1983). Phrase-Structure Parsing: A Method

361–388.
for Taking Advantage of Allophonic Constraints . Ph.D. the-
sis,MIT.
Church, K. W. (1988). A stochastic parts program and noun
phrase parser for unrestricted text. In Second Conference on
Applied Natural Language Processing , pp. 136–143.
Church, K. W. and Gale, W. A. (1991). A comparison of the
enhancedGood-Turinganddeletedestimationmethodsfores -
timating probabilities of English bigrams. Computer Speech
and Language ,5, 19–54.
Church, K. W., Gale, W. A., and Kruskal, J. B. (1991). Ap-
pendix A: the Good-Turing theorem. In Computer Speech

sis,MIT.
and Language (Church and Gale, 1991), pp. 19–54.
Church, K. W., Hart, T., and Gao, J. (2007). Compress-
ing trigram language models with Golomb coding. In
EMNLP/CoNLL 2007 , pp. 199–207.
Church, K. W. and Patil, R. (1982). Coping with syntactic
ambiguity. American Journal of Computational Linguistics ,
8(3-4), 139–149.Church, K. W. (1980). On memory limitations in natural lan-
guage processing. Master’s thesis, MIT. Distributed by the
Church, K.W.and Hanks, P.(1989). Word association norms,

mutualinformation, andlexicography. In ACL-89,Vancouver,
B.C.,pp.76–83.
Church, K.W.and Hanks, P.(1990). Word association norms,
mutual information, and lexicography. Computational Lin-
guistics,16(1), 22–29.
Ciaramita , M. and Johnson, M. (2000). Explaining away am-
biguity: learning verb selectional preference with Bayesi an
networks. In COLING-00 , pp. 187–193.
Ciaramita , M.and Johnson, M.(2003). Supersense tagging of
unknown nouns in WordNet. In EMNLP-2003 , pp.168–175.

Cieri, C., Miller, D., and Walker, K. (2004). The Fisher Cor-
pus: a Resource for the Next Generations of Speech-to-Text.
InLREC-04 .
Clark, A.(2000). Inducing syntactic categories bycontext dis-
tribution clustering. In CoNLL-00 .
Clark, A. (2001). The unsupervised induction of stochas-
tic context-free grammars using distributional clusterin g. In
CoNLL-01 .
Clark, A.(2002). Memory-based learning ofmorphology with
stochastic transducers. In ACL-02,Philadelphia, PA,pp.513–
520.
Clark, H. H. and Fox Tree, J. E. (2002). Using uh and um in

CoNLL-01 .
520.
spontaneous speaking. Cognition ,84,73–111.
Clark,H.H.andSengal, C.J.(1979). Insearch ofreferents for
nouns and pronouns. Memory and Cognition ,7,35–41.
Clark, H.H.(1994). Discoursein production. In Gernsbacher,
M.A.(Ed.), HandbookofPsycholinguistics . AcademicPress.
Clark, H. H. (1996). Using Language . Cambridge University
Press.
Clark, H. H. and Clark, E. V. (1977). Psychology and Lan-
guage. Harcourt Brace Jovanovich.
Clark, H. H. and Marshall, C. (1981). Deﬁnite reference and
mutualknowledge. InJoshi, A.K.,Webber, B.L.,and Sag, I.

Press.
(Eds.),Elementsofdiscourseunderstanding , pp.10–63.Cam-
bridge.
Clark, H. H. and Schaefer, E. F. (1989). Contributing to dis-
course.Cognitive Science ,13, 259–294.
Clark, H. H. and Wilkes-Gibbs, D. (1986). Referring as a col-
laborative process. Cognition ,22, 1–39.
Clark, J. and Yallop, C. (1995). An Introduction to Phonetics
and Phonology . Blackwell, Oxford. 2nd ed.
Clark, S. and Curran, J. R. (2004a). The importance of su-
pertagging for wide-coverage CCG parsing. In COLING-04 ,
pp.282–288.

bridge.
Clark, S. and Curran, J. R. (2004b). Parsing the WSJ using
CCG and Log-Linear Models. In ACL-04, pp.104–111.
Clarke, J. and Lapata, M. (2006). Models for sentence com-
pression: A comparison across domains, training require-
ments and evaluation measures. In COLING/ACL 2006 , pp.
377–384.
Clarke, J.andLapata, M.(2007). Modelling compression with
discourse constraints. In EMNLP/CoNLL 2007 , Prague, pp.
667–677.

DRAFT932 Bibliography
Clarkson , P. R. and Rosenfeld, R. (1997). Statistical lan-
guage modeling using the CMU-Cambridge toolkit. In
EUROSPEECH-97 ,Vol. 1, pp.2707–2710.
CMU(1993). The Carnegie Mellon Pronouncing Dictionary
v0.1. Carnegie Mellon University.
Coccaro, N. and Jurafsky, D. (1998). Towards better integra-
tion of semantic predictors in statistical language modeli ng.
InICSLP-98 , Sydney, Vol. 6,pp. 2403–2406.
Cohen,J.,Kamm,T.,and Andreou, A.(1995). Vocal tract nor-
malization in speech recognition: compensating for system -
atic systematic speaker variability. Journal of the Acoustical
Society ofAmerica ,97(5), 3246–3247.
Cohen, K.B. and Hunter, L.(2006). A critical review of PAS-
Bio’s argument structures for biomedical verbs. BMC Bioin-
formatics,7(Suppl 3).
Cohen, K. B., Dolbey, A., Mensah, A. G., and Hunter, L.
(2002). Contrast and variability in gene names. In Proceed-
ings of the ACL Workshop on Natural Language Processing
in the Biomedical Domain , pp. 14–20.

Cohen, K. B. and Hunter, L. (2004). Natural language pro-
cessing and systems biology. In Dubitzky, W. and Azuaje, F.
(Eds.),Artiﬁcial Intelligence Methods and Tools for Systems
Biology, pp.147–174. Springer.
Cohen, M. H. (1989). Phonological Structures for Speech
Recognition . Ph.D.thesis, University of California, Berkeley.
Cohen, M. H., Giangola, J. P., and Balogh, J. (2004). Voice
UserInterface Design . Addison-Wesley, Boston.
Cohen, P. R. (1995). Empirical Methods for Artiﬁcial Intelli-

gence. MIT Press.
Cohen, P.R., Johnston, M.,McGee, D.,Oviatt, S. L.,Clow, J.,
and Smith, I. (1998). The efﬁciency of multimodal interac-
tion: acase study. In ICSLP-98 , Sydney, Vol. 2,pp.249–252.
Cohen, P. R. and Oviatt, S. L. (1994). The role of voice in
human-machine communication. In Roe, D. B. and Wilpon,
J. G. (Eds.), Voice Communication Between Humans and
Machines , pp. 34–75. National Academy Press, Washington,
D.C.
Cohen, P. R. and Perrault, C. R. (1979). Elements of a plan-

D.C.
based theory of speech acts. Cognitive Science ,3(3), 177–
212.
Cohen,W.W.,Schapire, R.E.,andSinger, Y.(1999). Learning
toorderthings. JournalofArtiﬁcialIntelligence Research ,10,
243–270.
Coker, C.,Umeda, N.,and Browman, C. P.(1973). Automatic
synthesis from ordinary english test. IEEE Transactions on
Audioand Electroacoustics ,21(3), 293–298.
Colby,K.M.,Weber,S.,andHilf,F.D.(1971). Artiﬁcial para-
noia.Artiﬁcial Intelligence ,2(1), 1–25.
Cole,J.S.andKisseberth,C.W.(1995). Restricting multi-leve l

212.
243–270.
constraint evaluation. Rutgers Optimality Archive ROA-98 .
Cole, R. A., Novick, D. G., Vermeulen, P. J. E., Sutton, S.,
Fanty, M., Wessels, L. F. A., de Villiers, J. H., Schalkwyk,
J., Hansen, B., and Burnett, D. (1997). Experiments with a
spoken dialogue system for taking the US census. Speech
Communication ,23, 243–260.
Cole, R. A. (1973). Listening for mispronunciations: A mea-
sure of what we hear during speech. Perception and Psy-

chophysics ,13, 153–156.Cole, R. A. and Jakimik, J. (1980). A model of speech per-
ception. In Cole, R. A. (Ed.), Perception and Production of
Fluent Speech , pp. 133–163. Lawrence Erlbaum.
Cole, R. A., Novick, D. G., Burnett, D., Hansen, B., Sutton,
S.,and Fanty,M.(1994). Towardsautomatic collection ofth e
U.S.census. In IEEEICASSP-94 , Adelaide, Australia, Vol. I,
pp.93–96. IEEE.
Cole, R. A., Novick, D. G., Fanty, M., Sutton, S., Hansen, B.,
andBurnett,D.(1993). Rapidprototyping ofspokenlanguag e

systems: TheYear2000CensusProject.In Proceedingsofthe
International Symposium on Spoken Dialogue , Waseda Uni-
versity, Tokyo, Japan.
Cole, R. A. and Rudnicky, A. I. (1983). What’s new in speech
perception? The research and ideas of William Chandler
Bagley.Psychological Review ,90(1), 94–101.
Coleman , J. (2005). Introducing Speech and Language Pro-
cessing. Cambridge University Press.
Coleman ,J.andPierrehumbert, J.B.(1997).Stochasticphono-
logical grammars and acceptability. In Proceedings of ACL
SIGPHON .

SIGPHON .
Collins, M. and Koo, T. (2005). Discriminative reranking for
natural language parsing. Computational Linguistics ,31(1),
25–69.
Collins, M. (1996). A new statistical parser based on bigram
lexical dependencies. In ACL-96, Santa Cruz, CA, pp. 184–
191.
Collins, M. (1997). Three generative, lexicalised models for
statistical parsing. In ACL/EACL-97 , Madrid, Spain, pp. 16–
23.
Collins,M.(1999). Head-driven Statistical ModelsforNatural
Language Parsing . Ph.D. thesis, University of Pennsylvania,
Philadelphia.

25–69.
191.
23.
Collins, M. (2000). Discriminative reranking for natural lan-
guage parsing. In ICML 2000 , Stanford, CA, pp.175–182.
Collins, M. (2003). Head-driven statistical models for natu-
ral language parsing. Computational Linguistics ,29(4), 589–
637.
Collins, M., Hajiˇ c, J., Ramshaw, L. A., and Tillmann, C.
(1999). A statistical parser for Czech. In ACL-99, College
Park, MA,pp. 505–512.
Colmerauer , A.(1970).Lessyst` emes-qouunformalismepour
analyseretsynth´ etiser desphrasesurordinateur. Intern alpub-

637.
lication 43, D´ epartement d’informatique de l’Universit´ e de
Montr´ eal.
Colmerauer , A. (1975). Les grammaires de m´ etamorphose
GIA. Internal publication, Groupe Intelligence artiﬁciel le,
Facult´ e des Sciences de Luminy, Universit´ e Aix-Marseill e II,
France, Nov 1975. English version, Metamorphosis gram-
mars. In L. Bolc, (Ed.), Natural Language Communication
with Computers, Lecture Notes in Computer Science 63 ,
Springer Verlag, Berlin, 1978, pp. 133–189.

Colmerauer , A.andRoussel,P.(1996). ThebirthofProlog. In
Bergin Jr., T. J. and Gibson, Jr., R. G. (Eds.), History of Pro-
grammingLanguages–II ,pp.331–352.ACMPress/Addison-
Wesley, New York.
Comrie, B.(1989). LanguageUniversalsandLinguisticTypol-
ogy. Blackwell, Oxford. Second edition.
Conkie,A.andIsard,S.(1996). Optimalcoupling ofdiphones.
InvanSanten,J.P.H.,Sproat,R.,Olive,J.P.,andHirschbe rg,
J.(Eds.), Progress in Speech Synthesis . Springer.

DRAFTBibliography 933
Connine, C. M. (1990). Effects of sentence context and lex-
ical knowledge in speech processing. In Altmann, G. T. M.
(Ed.),Cognitive Models of Speech Processing , pp. 281–294.
MITPress.
Connine, C. M.and Clifton, C. (1987). Interactive use of lexi-
calinformationinspeechperception. JournalofExperimental
Psychology: Human Perception and Performance ,13, 291–
299.
Connolly , D.,Burger, J.D.,and Day, D.S. (1994). A machine
learning approach to anaphoric reference. In Proceedings of
the International Conference on New Methods in Language
Processing (NeMLaP) .
Conroy, J. M., Schlesinger, J. D., and Goldstein, J. (2006).
Classytasked based summarization: Back to basics. In DUC-
06.
Cooley, J. W. and Tukey, J. W. (1965). An algorithm for the
machine calculation of complex Fourier series. Mathematics
ofComputation ,19(90), 297–301.
Cooper, F. S., Liberman, A. M., and Borst, J. M. (1951). The
Interconversion of Audible and Visible Patterns asaBasis f or

06.
Research inthePerception ofSpeech. Proceedings oftheNa-
tional Academy ofSciences ,37(5), 318–325.
Copestake , A. (2002). Implementing Typed Feature Structure
Grammars . CSLI,Stanford, CA.
Copestake , A. and Briscoe, T. (1995). Semi-productive poly-
semy and sense extension. Journal of Semantics ,12(1), 15–
68.
Core, M., Ishizaki, M., Moore, J. D., Nakatani, C., Reithinger,
N.,Traum,D.R.,andTutiya,S.(1999).Thereportofthethir d
workshopoftheDiscourseResourceInitiative, ChibaUnive r-
sityandKazusaAcademiaHall. Tech.rep.No.3CC-TR-99-1,

68.
Chiba Corpus Project, Chiba, Japan.
Corston-Oliver , S. H. (1998). Identifying the linguistic corre-
lates of rhetorical relations. In Workshop on Discourse Rela-
tions and Discourse Markers , pp.8–14.
Cottrell, G. W. (1985). A Connectionist Approach to Word
SenseDisambiguation . Ph.D.thesis, University ofRochester,
Rochester, NY. Revised version published inthesametitle b y
Pitman in 1989.
Cover, T. M. and King, R. C. (1978). A convergent gambling
estimate of the entropy of English. IEEE Transactions on In-

formation Theory ,24(4), 413–421.
Cover, T. M. and Thomas, J. A. (1991). Elements of informa-
tion theory . Wiley.
Cowie, J., Guthrie, J. A., and Guthrie, L. M. (1992). Lexical
disambiguation using simulated annealing. In COLING-92 ,
Nantes, France, pp. 359–365.
Cowper, E.A.(1976). Constraints on Sentence Complexity: A
Model for Syntactic Processing . Ph.D. thesis, Brown Univer-
sity,Providence, RI.
Crawley,R.A.,Stevenson,R.J.,andKleinman,D.(1990).The
use of heuristic strategies in the interpretation of pronou ns.

Journal ofPsycholinguistic Research ,19, 245–264.
Crestani, F.,Lemas, M., Rijsbergen, C. J. V., and Campbell, I.
(1998). “Is This Document Relevant? ...Probably”: asurvey
of probabilistic models in information retrieval. ACM Com-
puting Surveys ,30(4), 528–552.
Croft, W. (1990). Typology and Universals . Cambridge Uni-
versity Press.Croft, W. (1995). Intonation units and grammatical structure.
Linguistics ,33,839–882.
Crouch, C. J. and Yang, B. (1992). Experiments in automatic

statistical thesaurus construction. In SIGIR-92 , Copenhagen,
Denmark, pp. 77–88. ACM.
Cruse,D.A.(2004). Meaning inLanguage: anIntroduction to
Semantics and Pragmatics . Oxford University Press, Oxford.
Second edition.
Crystal,D.(1969). ProsodicsystemsandintonationinEnglish .
Cuay´ahuitl, H., Renals, S., Lemon, O., and Shimodaira,
H. (2007). Hierarchical dialogue optimization using semi-
Markov decision processes. In INTERSPEECH-07 .
Culicover , P. W. and Jackendoff, R. (2005). Simpler Syntax .

Oxford University Press.
Cullingford , R. E. (1981). SAM. In Schank, R. C. and Ries-
beck, C. K. (Eds.), Inside Computer Understanding: Five
Programsplus Miniatures , pp. 75–119. Lawrence Erlbaum.
Culotta, A. and Sorensen, J. (2004). Dependency tree kernels
for relation extraction. In ACL-04.
Culy,C.(1985).Thecomplexity ofthevocabularyofBambara.
Linguistics and Philosophy ,8, 345–351.
Curran, J. R. (2003). From Distributional to Semantic Simi-
larity. Ph.D.thesis, University of Edinburgh.
Curran, J. R. and Moens, M. (2002). Improvements in au-

tomatic thesaurus extraction. In Proceedings of the ACL-02
workshop on Unsupervised Lexical Acquisition , Philadelphia,
PA,pp.59–66.
Cutler, A. (1986). Forbear is a homophone: Lexical prosody
does not constrain lexical access. Language and Speech ,29,
201–219.
Cutler, A. and Carter, D. M. (1987). The predominance of
strong initial syllables in the English vocabulary. Computer
Speech and Language ,2, 133–142.
Cutler,A.andNorris,D.(1988). Theroleofstrongsyllables in
segmentation forlexical access. JournalofExperimentalPsy-

201–219.
chology: Human Perception and Performance ,14, 113–121.
Cutting, D., Kupiec, J., Pedersen, J., and Sibun, P. (1992a). A
practical part-of-speech tagger. In Third Conference on Ap-
plied Natural Language Processing , pp.133–140.
Cutting, D., Karger, D. R., Pedersen, J., and Tukey, J. W.
(1992b). Scatter/gather: A cluster-based approach to brow s-
ing large document collections. In SIGIR-92 , Copenhagen,
Denmark, pp. 318–329. ACM.
Daelemans , W. and van den Bosch, A. (1997). Language-
independent data-oriented grapheme-to-phoneme conversi on.

J.(Eds.), Progress in Speech Synthesis , pp. 77–89. Springer.
Daelemans , W., Zavrel, J., Berck, P., and Gillis, S. (1996).
MBT: A memory based part of speech tagger-generator. In
Ejerhed, E. and Dagan, I. (Eds.), Proceedings of the Fourth
Workshop on VeryLargeCorpora , pp. 14–27.
Daelemans , W., Smedt, K. D., and Gazdar, G. (1992). Inheri-
tanceinnaturallanguage processing. Computational Linguis-
tics,18(2), 205–218.

Dagan, I. (2000). Contextual word similarity. In Dale, R.,
Moisl, H., and Somers, H. (Eds.), A Handbook of Natural
Language Processing: Techniques and applications for the
processing oflanguage as text . Marcel Dekker.

DRAFT934 Bibliography
Dagan, I., Lee, L., and Pereira, F. C. N. (1999). Similarity-
based models of cooccurrence probabilities. Machine Learn-
ing,34(1–3), 43–69.
Dagan, I., Marcus, S., and Markovitch, S. (1993). Contextual
word similarity and estimation from sparse data. In ACL-93,
Columbus, Ohio, pp.164–171.
Dagan,I.,Pereira, F.C.N.,andLee,L.(1994). Similarity-base
estimation of word cooccurrence probabilities. In ACL-94,
LasCruces, NM,pp. 272–278.
Daly, N. A. and Zue, V. W. (1992). Statistical and linguistic
analyses of F0in read and spontaneous speech. In ICSLP-92 ,
Vol. 1,pp. 763–766.
Damerau ,F.J.(1964). Atechniqueforcomputerdetection and
correction of spelling errors. Communications of the ACM ,
7(3), 171–176.
Damerau , F. J. and Mays, E. (1989). An examination of un-
detected typing errors. Information Processing and Manage-
ment,25(6), 659–664.
Damper, R. I., Marchand, Y., Adamson, M. J., and Gustafson,
K. (1999). Evaluating the pronunciation component of text-

to-speech systems for english: A performance comparison of
different approaches. Computer Speech andLanguage ,13(2),
155–176.
Dang, H.T.(2006). Overview of DUC 2006. In DUC-06.
Danieli, M.andGerbino,E.(1995). Metrics forevaluating dia-
logue strategies in aspoken language system. In Proceedings
ofthe1995AAAISpringSymposiumonEmpiricalMethodsin
Discourse Interpretation and Generation , Stanford, CA, pp.
34–39. AAAIPress, Menlo Park, CA.
Darroch, J. N. and Ratcliff, D. (1972). Generalized iterative

155–176.
scaling for log-linear models. The Annals of Mathematical
Statistics,43(5), 1470–1480.
Daum´e III, H. and Marcu, D. (2002). A noisy-channel model
for document compression. In ACL-02.
Daum´e III, H. and Marcu, D. (2005). Induction of word and
phrase alignments for automatic document summarization.
Computational Linguistics ,31(4), 505–530.
Daum´e III,H. and Marcu, D. (2006). Bayesian query-focused
summarization. In COLING/ACL 2006 , Sydney, Australia.
David, Jr.,E.E.and Selfridge, O. G.(1962). Eyes and ears for

computers. Proceedings of the IRE (Institute of Radio Engi-
neers),50, 1093–1101.
Davidson , D. (1967). Thelogical form of action sentences. In
Rescher, N. (Ed.), The Logic of Decision and Action . Univer-
sity of Pittsburgh Press.
Davis, E. (1990). Representations of Commonsense Knowl-
edge. Morgan Kaufmann.
Davis,K.H.,Biddulph,R.,andBalashek,S.(1952). Automatic
recognition ofspokendigits. JournaloftheAcousticalSociety
ofAmerica ,24(6), 637–642.
Davis, S.and Mermelstein, P.(1980). Comparison of paramet-

ricrepresentations for monosyllabic word recognition in c on-
tinuously spoken sentences. IEEETransactions on Acoustics,
Speech, and Signal Processing ,28(4), 357–366.
De Jong, N. H., Feldman, L. B., Schreuder, R., Pastizzo, M.,
and Baayen, R. H.(2002). Theprocessing and representation
of Dutch and English compounds: Peripheral morphological,
and central orthographic effects. Brain and Language ,81,
555–567.de Marcken , C. (1996). Unsupervised Language Acquisition .
Ph.D.thesis, MIT.

deTocqueville , A.(1840). DemocracyinAmerica . Doubleday,
New York. The1966 translation by George Lawrence.
Deerwester , S.,Dumais, S. T.,Furnas, G. W.,Landauer, T.K.,
and Harshman, R.(1990). Indexing by latent semantic analy-
sis.Journal of the American Society of Information Science ,
41,391–407.
Dejean, H. and Tjong Kim Sang, E. F. (2001). Introduction to
the CoNLL-2001 shared task: Clause identiﬁcation. In Pro-
ceedings ofCoNLL-2001 .
DeJong, G.F.(1982a). Anoverview oftheFRUMPsystem. In

Lehnert, W. G. and Ringle, M. H. (Eds.), Strategies for Natu-
ralLanguage Processing , pp. 149–176. Lawrence Erlbaum.
DeJong,G.F.(1982b). Anoverview oftheFRUMPsystem. In
Lehnert, W. G. and Ringle, M. H. (Eds.), Strategies for Nat-
ural Language Processing , pp. 149–176. Lawrence Erlbaum,
New Jersey.
DeJong, G. F. and Waltz, D. L. (1983). Understanding novel
language. Computers and Mathematics with Applications ,9.
Delgado, R.L.-C.and Araki, M.(2005). Spoken, Multilingual,
and Multimodal Dialogue Systems . Wiley.

Deligne, S., Yvon, F., and Bimbot, F. (1995). Variable-
length sequence matching for phonetic transcription using
joint multigrams. In EUROSPEECH-95 ,Madrid.
Della Pietra , S. A., Della Pietra, V. J., and Lafferty, J. D.
(1997). Inducing features of random ﬁelds. IEEE Trans-
actions on Pattern Analysis and Machine Intelligence ,19(4),
380–393.
Demberg , V. (2006). Letter-to-phoneme conversion for a ger-
man text-to-speech system. Diplomarbeit Nr. 47, Universit ¨t
Stuttgart.
Demetriou , G., Atwell, E., and Souter, C. (1997). Large-

380–393.
scale lexical semantics for speech recognition support. In
EUROSPEECH-97 ,pp. 2755–2758.
Dempster , A. P.,Laird, N. M., and Rubin, D. B. (1977). Max-
imum likelihood from incomplete data via the EMalgorithm.
Journal of theRoyal Statistical Society ,39(1), 1–21.
Denes, P. (1959). The design and operation of the mechani-
cal speech recognizer at University College London. Journal
of the British Institution of Radio Engineers ,19(4), 219–234.
Appears together with companion paper (Fry 1959).

Deng, L., Lennig, M., Seitz, F., and Mermelstein, P. (1990).
Large vocabulary word recognition using context-dependen t
allophonic hidden Markov models. Computer Speech and
Language ,4, 345–357.
Deng,L.andHuang,X.(2004). Challenges inadopting speech
recognition. Communications ofthe ACM ,47(1), 69–75.
Deng, Y. and Byrne, W. (2005). HMM word and phrase align-
mentfor statistical machine translation. In HLT-EMNLP-05 .
Dermatas ,E.andKokkinakis,G.(1995). Automaticstochastic

tagging of natural language texts. Computational Linguistics ,
21(2), 137–164.
DeRose,S.J.(1988). Grammaticalcategory disambiguation by
statistical optimization. Computational Linguistics ,14, 31–
39.
Di Eugenio , B. (1990). Centering theory and the Italian
pronominal system. In COLING-90 , Helsinki, pp. 270–275.

DRAFTBibliography 935
Di Eugenio , B. (1996). The discourse functions of Italian sub-
jects: A centering approach. In COLING-96 , Copenhagen,
pp.352–357.
Diab, M. and Resnik, P. (2002). An unsupervised method for
word sense tagging using parallel corpora. In ACL-02, pp.
255–262.
Dietterich , T.G.(1998). Approximate statistical tests forcom-
paring supervised classiﬁcation learning algorithms. Neural
Computation ,10(7), 1895–1924.
Digilakis, V.(1992). Segment-based stochastic models ofspec-
traldynamicsforcontinuousspeechrecognition . Ph.D.thesis,
Boston University.
Dimitrova , L., Ide, N. M., Petkevic, V., Erjavec, T., Kaalep,
H. J., and Tuﬁs, D. (1998). Multext-East: parallel and com-
parable corpora and lexicons for six Central and Eastern Eu-
ropean languages. In COLING/ACL-98 , Montreal, Canada.
Divay, M. and Vitale, A. J. (1997). Algorithms for grapheme-
phonemetranslation forEnglishand French: Applications f or
database searches and speech synthesis. Computational Lin-
guistics,23(4), 495–523.

Dixon, N. and Maxey, H. (1968). Terminal analog synthesis
of continuous speech using the diphone method of segment
assembly. IEEETransactions on Audio and Electroacoustics ,
16(1), 40–50.
Doddington , G.(2001). Speaker recognition based on idiolec-
tal differences between speakers. In EUROSPEECH-01 , Bu-
dapest, pp. 2521–2524.
Doddington , G. (2002). Automatic evaluation of machine
translation quality using n-gram co-occurrence statistic s. In
HLT-01.
Dolan, W. B. (1994). Word sense ambiguation: Clustering

HLT-01.
related senses. In COLING-94 , Kyoto, Japan, pp. 712–716.
ACL.
Dolan, W. B., Quirk, C., and Brockett, C. (2004). Unsuper-
vised construction of large paraphrase corpora: exploitin g
massively parallel news sources. In COLING-04 .
Donovan , R. E. (1996). Trainable Speech Synthesis . Ph.D.
thesis, Cambridge University Engineering Department.
Donovan , R. E. and Eide, E. M. (1998). The IBM trainable
speech synthesis system. In ICSLP-98 , Sydney.
Donovan , R. E. and Woodland, P. C. (1995). Improvements

ACL.
in an HMM-based speech synthesiser. In EUROSPEECH-95 ,
Madrid, Vol. 1,pp. 573–576.
Dorr, B. (1992). The use of lexical semantics in interlingual
machine translation. Journal of Machine Translation ,7(3),
135–193.
Dorr, B. (1993). Machine Translation . MIT Press.
Dorr, B. (1994). Machine translation divergences: A formal
description and proposed solution. Computational Linguis-
tics,20(4), 597–633.
Dorr, B., Zajic, D., and Schwartz, R. (2003). Hedge trimmer:
a parse-and-trim approach to headline generation. In HLT-

135–193.
NAACLWorkshop on TextSummarization , pp.1–8.
Dostert, L. (1955). The Georgetown-I.B.M. experiment. In
Machine Translation of Languages: Fourteen Essays , pp.
124–135. MIT Press.
Doumpiotis , V., Tsakalidis, S., and Byrne, W. (2003a). Dis-
criminative training for segmental minimum bayes-risk de-
coding. In IEEEICASSP-03 .Doumpiotis , V.,Tsakalidis, S.,and Byrne, W.(2003b). Lattice
segmentation and minimum bayes risk discriminative train-
ing. InEUROSPEECH-03 .
Downing , P. (1977). On the creation and use of English com-

pound nouns. Language ,53(4), 810–842.
Dowty, D.R.(1979). WordMeaning andMontague Grammar .
Dowty, D. R., Wall, R. E., and Peters, S. (1981). Introduction
to Montague Semantics . D.Reidel, Dordrecht.
Du Bois, J. W., Schuetze-Coburn, S., Cumming, S., and
Paolino, D. (1983). Outline of discourse transcription. In
Edwards, J. A. and Lampert, M. D. (Eds.), Talking Data:
Transcription and Coding in Discourse Research , pp. 45–89.
Lawrence Erlbaum.
Duda,R.O.,Hart,P.E.,andStork,D.G.(2000). PatternClas-

siﬁcation. Wiley-Interscience Publication.
Duda, R. O. and Hart, P. E. (1973). Pattern Classiﬁcation and
Scene Analysis . John Wiley and Sons,New York.
Dud´ ık, M. and Schapire, R. E. (2006). Maximum entropy dis-
tribution estimation with generalized regularization. In Lu-
gosi, G. and Simon, H. U. (Eds.), COLT 2006 , Berlin, pp.
123–138. Springer-Verlag.
Dunning, T.(1993). Accurate methods for thestatistics ofsur-
priseand coincidence. Computational Linguistics ,19(1), 61–
74.

74.
Durbin, R., Eddy, S., Krogh, A., and Mitchison, G. (1998).
Biological sequence analysis . Cambridge University Press.
Dutoit, T. (1997). An Introduction to Text to Speech Synthesis .
Kluwer.
Dˇ zeroski, S.,Erjavec,T.,andZavrel,J.(2000). Morphosyntac-
tic Tagging of Slovene: Evaluating PoS Taggers and Tagsets.
InLREC-00 ,Paris, pp.1099–1104.
Earley, J.(1968). AnEfﬁcientContext-FreeParsingAlgorithm .
Ph.D.thesis, Carnegie Mellon University, Pittsburgh, PA.
Earley, J. (1970). An efﬁcient context-free parsing algorithm.

Kluwer.
Communications of the ACM ,6(8), 451–455. Reprinted in
Groszet al. (1986).
Echihabi , A., Hermjakob, U., Hovy, E. H., Marcu, D., Melz,
E., and Ravichandran, D. (2005). How to select an answer
string?. In Strzalkowski, T. and Harabagiu, S. (Eds.), Ad-
vances in Textual Question Answering . Kluwer.
Edmunson , H. (1969). New methods in automatic extracting.
Journal of theACM ,16(2), 264–285.
Eide, E.M.and Gish,H.(1996). Aparametric approach to vo-
cal tract length normalization. In IEEE ICASSP-96 , Atlanta,
GA,pp. 346–348.

Eide,E.M.,Bakis,R.,Hamza,W.,andPitrelli, J.F.(2004). To-
wardssynthesizingexpressivespeech.InNarayanan,S.S.a nd
Alwan, A. (Eds.), Text to Speech Synthesis: New paradigms
and Advances . Prentice Hall.
Eisner, J. (1996a). An empirical comparison of probability
models for dependency grammar. Tech. rep. IRCS-96-11, In-
stituteforResearch inCognitive Science, Univ.ofPennsyl va-
nia.
Eisner, J. (1996b). Three new probabilistic models for depen-
dency parsing: An exploration. In COLING-96 , Copenhagen,
pp.340–345.

DRAFT936 Bibliography
Eisner, J. (1997). Efﬁcient generation in primitive optimality
theory. In ACL/EACL-97 , Madrid, Spain, pp. 313–320.
Eisner, J. (2000a). Bilexical grammars and their cubic-time
parsing algorithms. In Bunt, H. and Nijholt, A. (Eds.), Ad-
vances in Probabilistic and Other Parsing Technologies , pp.
29–62. Kluwer.
Eisner, J. (2000b). Directional constraint evaluation in Opti-
mality Theory. In COLING-00 , Saarbr¨ ucken, Germany, pp.
257–263.
Eisner, J. (2002a). An interactive spreadsheet for teaching
the forward-backward algorithm. In Proceedings of the ACL
Workshop on Effective Tools and Methodologies for Teaching
NLPand CL , pp.10–18.
Eisner, J. (2002b). Comprehension and compilation in Opti-
mality Theory. In ACL-02,Philadelphia, pp.56–63.
Eisner, J. (2003). Learning non-isomorphic tree mappings for
machine translation. In ACL-03.
Ejerhed, E. I. (1988). Finding clauses in unrestricted text by
ﬁnitary andstochastic methods. In Second Conference onAp-

plied Natural Language Processing , pp.219–227.
Elhadad,M.(1990). Typesinfunctionaluniﬁcation grammars.
InACL-90, Pittsburgh, PA,pp.157–164.
Ellison, T. M. (1992). The Machine Learning of Phonological
Structure. Ph.D.thesis, University ofWestern Australia.
Ellison, T. M. (1994). Phonological derivation in optimality
theory. In COLING-94 , Kyoto, pp. 1007–1013.
Erjavec, T. (2004). Multext-east version 3: Multilingual mor-
phosyntactic speciﬁcations, lexicons and corpora. In LREC-
04,pp. 1535–1538. ELRA.

Erkan, G. and Radev, D. (2004). Lexrank: Graph-based cen-
trality as salience in text summarization. Journal of Artiﬁcial
Intelligence Research (JAIR) ,22,457–479.
Etzioni, O., Cafarella, M., Downey, D., Popescu, A.-M.,
Shaked, T., Soderland, S., Weld, D. S., and Yates, A. (2005).
Unsupervised named-entity extraction from the web: An ex-
perimental study. Artiﬁcial Intelligence ,165(1), 91–134.
Evans, N. (2000). Word classes in the world’s languages. In

Booij, G., Lehmann, C., and Mugdan, J. (Eds.), Morphology:
a Handbook on Inﬂection and Word Formation , pp. 708–732.
Mouton, Berlin.
Evans,R.andGazdar,G.(1996).DATR:Alanguageforlexical
knowledge representation. Computational Linguistics ,22(2),
167–216.
Evermann , G. and Woodland, P. C. (2000). Large vocabu-
lary decoding and conﬁdence estimation using word poste-
rior probabilities. In IEEE ICASSP-00 , Istanbul, Vol. III, pp.
1655–1658.
Fackrell, J.and Skut,W.(2004). Improvingpronunciation dic-

167–216.
tionary coverage ofnamesbymodelling spelling variation. In
Proceedings ofthe 5th Speech Synthesis Workshop .
Fano, R. M. (1961). Transmission of information; A statistical
theory ofcommunications . MITPress.
Fant, C. G. M. (3951). Speech communication research. Ing.
Vetenskaps Akad.Stockholm, Sweden ,24, 331–337.
Fant,G. (2004). Speech Acoustics and Phonetics . Kluwer.
Fant, G. M. (1960). Acoustic Theory of Speech Production .
Mouton.Fant,G.M.(1986). Glottalﬂow: Modelsandinteraction. Jour-

nalof Phonetics ,14,393–399.
Fant, G. M. (1997). The voice source in connected speech.
Speech Communication ,22(2-3), 125–139.
Fass, D.(1988). Collative Semantics: ASemantics for Natural
Language . Ph.D. thesis, New Mexico State University, Las
Cruces, New Mexico. CRL Report No. MCCS-88-118.
Fass, D. (1991). met*: A method for discriminating metaphor
and metonymy by computer. Computational Linguistics ,
17(1).
Fass, D. (1997). Processing Metonymy and Metaphor . Ablex
Publishing, Greenwich, CT.

17(1).
Fass, D., Martin, J. H., and Hinkelman, E. A. (Eds.). (1992).
Computational Intelligence: Special Issue on Non-Literal
Language , Vol. 8. Blackwell, Cambridge, MA.
Fauconnier , G. (1985). Mental Spaces: Aspects of Meaning
Construction in Natural Language . MITPress.
Fellbaum , C. (Ed.). (1998). WordNet: An Electronic Lexical
Database . MITPress.
Fensel, D., Hendler, J. A., Lieberman, H., and Wahlster, W.
(Eds.). (2003). Spinning the Semantic Web: Bring the World
Wide Web to its Full Potential . MIT Press,Cambridge, MA.

Ferrer, L., Shriberg, E., and Stolcke, A. (2003). A prosody-
basedapproach to end-of-utterance detection thatdoesnot re-
quire speech recognition. In IEEEICASSP-03 .
Ferro, L., Gerber, L., Mani, I., Sundheim, B., and Wilson, G.
(2005). Tides 2005 standard for the annotation of temporal
expressions. Tech. rep.,MITRE.
Fikes, R. E. and Nilsson, N. J. (1971). STRIPS: A new ap-
proach totheapplication oftheorem proving to problemsolv -
ing.Artiﬁcial Intelligence ,2,189–208.
Filippova , K. and Strube, M. (2006). Using linguistically

motivated features for paragraph boundary identiﬁcation. In
EMNLP2006 .
Fillmore, C. J. (1968). The case for case. In Bach, E. W. and
Harms, R. T. (Eds.), Universals in Linguistic Theory , pp. 1–
88.Holt, Rinehart &Winston.
Fillmore, C. J. (1985). Frames and the semantics of under-
standing. Quaderni diSemantica ,VI(2),222–254.
Fillmore, C. J.,Kay,P.,and O’Connor, M.C.(1988). Regular-
ity and idiomaticity in grammatical constructions: The cas e
of LetAlone. Language ,64(3), 510–538.
Finin, T. (1980). The semantic interpretation of nominal com-

pounds. In AAAI-80,Stanford, CA,pp. 310–312.
Firth,J.R.(1957). Asynopsisoflinguistic theory 1930–1955.
InStudies in Linguistic Analysis . Philological Society, Ox-
ford. Reprinted in Palmer, F.(ed.) 1968.Selected Papers of J.
R.Firth. Longman,Harlow.
Fisher, D., Soderland, S., McCarthy, J., Feng, F., and Lehnert,
W. G. (1995). Description of the UMass system as used for
MUC-6. In Proceedings of the Sixth Message Understanding
Conference (MUC-6) , San Francisco, pp. 127–140. Morgan
Kaufmann.

Kaufmann.
Fisher, W.(1996). tsylb2 software and documentation..
Fitt, S. (2002). Unisyn lexicon. http://www.cstr.ed.
ac.uk/projects/unisyn/ .
Flanagan , J. L. (1972). Speech Analysis, Synthesis, and Per-
ception. Springer.

DRAFTBibliography 937
Flanagan , J. L., Ishizaka, K., and Shipley, K. L. (1975). Syn-
thesis ofspeech fromadynamic modelofthevocal cordsand
vocal tract. The Bell System Technical Journal ,54(3), 485–
506.
Fodor,J.A.andBever, T.G.(1965). Thepsychological reality
oflinguistic segments. JournalofVerbalLearningandVerbal
Behavior,4, 414–420.
Ford, C., Fox, B., and Thompson, S. A. (1996). Practices in
theconstruction of turns. Pragmatics ,6, 427–454.
Ford, C. and Thompson, S. A. (1996). Interactional units in
conversation: syntactic, intonational, andpragmaticres ources
for the management of turns. In Ochs, E., Schegloff, E. A.,
and Thompson, S. A. (Eds.), Interaction and Grammar , pp.
134–184. Cambridge University Press.
Ford, M. (1983). A method for obtaining measures of lo-
cal parsing complexity through sentences. Journal of Verbal
Learning and VerbalBehavior ,22,203–218.
Forney Jr ., G. D. (1973). The Viterbi algorithm. Proceedings
ofthe IEEE ,61(3), 268–278.

Fosler-Lussier , E. (1999). Multi-level decision trees for static
and dynamic pronunciation models. In EUROSPEECH-99 ,
Budapest.
Fosler-Lussier , E.and Morgan, N.(1999). Effects of speaking
rateandwordpredictability onconversational pronunciat ions.
Speech Communication ,29(2-4), 137–158.
Foster, D. W. (1989). Elegy by W.S.: A Study in Attribution .
Associated University Presses, Cranbury, NJ.
Foster,D.W.(1996). Primaryculprit. NewYork ,29(8),50–57.
February 26.
Foster, G. (2000). A maximum entropy/minimum divergence

Budapest.
translation model. In ACL-00, Hong Kong.
Fox, B. and Jasperson, R. (1995). A syntactic exploration of
repair in English conversation. In Davis, P. (Ed.), Descrip-
tive and Theoretical Modes in the Alternative Linguistics , pp.
77–134. John Benjamins, Amsterdam. In press.
Fox Tree, J. E. and Clark, H. H. (1997). Pronouncing “the” as
“thee” to signal problems in speaking. Cognition ,62, 151–
167.
Frakes, W. B. and Baeza-Yates, R. (1992). Information Re-
trieval: Data Structures and Algorithms . Prentice Hall.

167.
Francis, H. S., Gregory, M. L., and Michaelis, L. A. (1999).
Are lexical subjects deviant?. In CLS-99. University of
Chicago.
Francis, W. N. (1979). A tagged corpus – problems and
prospects. In Greenbaum, S., Leech, G., and Svartvik, J.
(Eds.),Studies in English linguistics for Randolph Quirk , pp.
192–209. Longman.
Francis, W. N. and Kuˇ cera, H. (1982). Frequency Analysis of
English Usage . Houghton Mifﬂin, Boston.
Frank,R.and Satta, G.(1998). Optimality theory andthegen-
erative complexity of constraint violability. Computational

Chicago.
Linguistics ,24(2), 307–315.
Frankel, J., Wester, M., and King, S.(2007). Articulatory fea-
turerecognition usingdynamicbayesian networks. Computer
Speech and Language ,21(4), 620–640.
Franz, A. and Brants, T. (2006). All our n-gram are belong
to you. http://googleresearch.blogspot.com/
2006/08/all-our-n-gram-are-belong-to-you.
html.Franz, A. (1996). Automatic Ambiguity Resolution in Natural
Language Processing . Springer-Verlag, Berlin.
Franz,A.(1997). Independence assumptionsconsidered harm-
ful. InACL/EACL-97 ,Madrid, Spain, pp.182–189.

Fraser, A. and Marcu, D. (2005). ISI’s participation in the
Romanian-English alignmenttask. In Proceedings oftheACL
Workshop on Building and Using Parallel Texts ,pp. 91–94.
Fraser, N. M. (1992). Assessment of interactive systems. In
Gibbon, D., Moore, R., and Winski, R. (Eds.), Handbook on
Standards and Resources for Spoken Language Systems , pp.
564–615. Mouton de Gruyter, Berlin.
Fraser, N. M. and Gilbert, G. N. (1991). Simulating speech
systems. Computer Speech and Language ,5, 81–99.
Fraser, N. M. and Hudson, R. A. (1992). Inheritance in word

grammar. Computational Linguistics ,18(2), 133–158.
Freitag, D. (1998). Multistrategy learning for information ex-
traction. In ICML 1998 , Madison, WI,pp. 161–169.
Freitag, D. and McCallum, A. (1999). Information extraction
using HMMs and shrinkage. In Proceedings of the AAAI-99
Workshop on Machine Learning for Information Retrieval .
Friedl, J.E.F.(1997). Master Regular Expressions . O’Reilly.
Frisch, S. A., Large, N. R., and Pisoni, D. B. (2000). Per-
ception of wordlikeness: Effects of segment probability an d

length on the processing of nonwords. Journal of Memory
and Language ,42,481–496.
Fromkin, V. and Ratner, N. B. (1998). Speech production.
In Gleason, J. B. and Ratner, N. B. (Eds.), Psycholinguistics .
Harcourt Brace, FortWorth,TX.
Fry, D. B. (1955). Duration and intensity as physical corre-
lates of linguistic stress. Journal of the Acoustical Society of
America,27, 765–768.
Fry, D. B. (1959). Theoretical aspects of mechanical speech
recognition. Journal of the British Institution of Radio En-

gineers,19(4), 211–218. Appears together with companion
paper (Denes 1959).
Fujisaki, H.and Ohno,S.(1997). Comparison and assessment
of models in the study of fundamental frequency contours of
speech.In ESCAworkshoponIntonation: TheoryModelsand
Applications .
Fung, P. and McKeown, K. R. (1997). A technical word and
term translation aid using noisy parallel corpora across la n-
guage groups. Machine Translation ,12(1-2), 53–87.
Gabrilovich , E. and Markovitch, S. (2007). Computing Se-

mantic Relatedness using Wikipedia-based Explicit Semant ic
Analysis. In IJCAI-07.
Gaizauskas , R., Wakao, T., Humphreys, K., Cunningham, H.,
and Wilks, Y. (1995). University of Shefﬁeld: Description
of the LaSIE system as used for MUC-6. In Proceedings of
the Sixth Message Understanding Conference (MUC-6) , San
Francisco, pp. 207–220. Morgan Kaufmann.
Gale, W.A. and Church, K. W. (1990). Estimation procedures
for language context: poor estimates are worse than none.
InCOMPSTAT: Proceedings in Computational Statistics , pp.

69–74.
Gale, W.A.andChurch, K.W.(1991). Aprogramfor aligning
sentences in bilingual corpora. In ACL-91, Berkeley, CA, pp.
177–184.

DRAFT938 Bibliography
Gale,W.A.andChurch, K.W.(1993). Aprogram foraligning
sentencesinbilingualcorpora. ComputationalLinguistics ,19,
75–102.
Gale, W. A. and Church, K. W. (1994). What is wrong with
adding one?. In Oostdijk, N. and de Haan, P. (Eds.), Corpus-
based Research into Language , pp. 189–198. Rodopi, Ams-
terdam.
Gale, W. A., Church, K. W., and Yarowsky, D. (1992a). Work
on statistical methods for word sense disambiguation. In
Goldman, R. (Ed.), Proceedings of the 1992 AAAI Fall Sym-
posium on Probabilistic Approaches to Natural Language .
Gale, W. A., Church, K. W., and Yarowsky, D. (1992b). Esti-
mating upper and lower bounds on the performance of word-
sensedisambiguation programs. In ACL-92,Newark, DE,pp.
249–256.
Gale, W. A., Church, K. W., and Yarowsky, D. (1992c). One
senseper discourse. In Proceedings DARPASpeech and Nat-
uralLanguage Workshop , pp. 233–237. Morgan Kaufmann.
Gale, W. A., Church, K. W., and Yarowsky, D. (1993). A
method for disambiguating word senses in a large corpus.

249–256.
Computers and the Humanities ,26, 415–439.
Gale, W. A. and Sampson, G. (1995). Good-turing frequency
estimation without tears. Journal of Quantitative Linguistics ,
2,217–237.
Galescu, L.and Allen, J.(2001). Bi-directional conversion be-
tween graphemes and phonemes using a joint N-gram model.
InProceedings of the 4th ISCA Tutorial and Research Work-
shop on Speech Synthesis .
Galley, M., Hopkins, M., Knight, K., and Marcu, D. (2004).
What’s in atranslation rule?. In HLT-NAACL-04 .
Galley, M. and McKeown, K. R. (2007). Lexicalized Markov

grammars for sentence compression. In NAACL-HLT 07 ,
Rochester, NY,pp. 180–187.
Garrett, M. F.(1975). Theanalysis of sentence production. In
Bower, G. H.(Ed.), ThePsychology ofLearning and Motiva-
tion, Vol. 9. Academic.
Garside, R. (1987). The CLAWS word-tagging system. In
Garside, R., Leech, G., and Sampson, G. (Eds.), The Compu-
tational Analysis of English , pp. 30–41. Longman, London.
Garside, R., Leech, G., and McEnery, A. (1997). Corpus An-
notation. Longman, London and New York.

Gaussier , E. (1999). Unsupervised learning of derivational
morphology from inﬂectional lexicons. In ACL-99.
Gaustad, T. (2001). Statistical corpus-based word sense dis-
ambiguation: Pseudowords vs. real ambiguous words. In
ACL/EACL2001 – Student Research Workshop , pp.255–262.
Gazdar, G. (1981). Unbounded dependencies and coordinate
structure. Linguistic Inquiry ,12(2), 155–184.
Gazdar, G. (1982). Phrase structure grammar. In Jacobson, P.
and Pullum, G. K. (Eds.), The Nature of Syntactic Represen-

tation, pp.131–186. Reidel, Dordrecht.
Gazdar, G., Klein, E., Pullum, G. K., and Sag, I. A. (1985).
Generalized PhraseStructure Grammar . Blackwell, Oxford.
Gazdar, G. and Mellish, C. (1989). Natural Language Pro-
cessing in LISP . Addison Wesley.
Gazdar, G., Pullum, G. K., Carpenter, B., Klein, E., Hukari,
T.E.,and Levine, R.D.(1988). Category structures. Compu-
tational Linguistics ,14(1), 1–19.Ge,N.,Hale,J.,andCharniak, E.(1998). Astatistical approa ch
to anaphora resolution. In Proceedings of theSixth Workshop

on VeryLarge Corpora , pp.161–171.
Gee, J. P. and Grosjean, F. (1983). Performance structures: A
psycholinguistic and linguistic appraisal. Cognitive Psychol-
ogy,15, 411–458.
Geman,S.andJohnson,M.(2002). Dynamicprogrammingfor
parsing and estimation of stochastic uniﬁcation-based gra m-
mars. InACL-02, pp. 279–286.
Gentner, D. (1983). Structure mapping: A theoretical frame-
work for analogy. Cognitive Science ,7,155–170.
Genzel, D. and Charniak, E.(2002). Entropy rate constancy in
text. InACL-02.

Genzel, D. and Charniak, E. (2003). Variation of entropy and
parse trees of sentences as a function of the sentence number .
InEMNLP2003 .
Gerdemann ,D.andvanNoord,G.(2000). Approximation and
exactness in ﬁnite state optimality theory. In Proceedings of
ACLSIGPHON .
Germann , U., Jahr, M., Knight, K., Marcu, D., and Yamada,
K. (2001). Fast decoding and optimal decoding for machine
translation. In ACL-01,pp. 228–235.
Gershman , A. V. (1977). Conceptual analysis of noun groups

in English. In IJCAI-77, Cambridge, MA,pp. 132–138.
Gibbon, D., Mertins, I., and Moore, R. (2000). Handbook of
MultimodalandSpokenDialogueSystems: Resources,Termi-
nology and ProductEvaluation . Kluwer, Dordrecht.
Gibson,E.(1998). Linguisticcomplexity: Localityofsyntactic
dependencies. Cognition ,68, 1–76.
Gibson, E. (2003). Sentence comprehension, linguistic com-
plexity in. In Nadel, L. (Ed.), Encyclopedia of Cognitive Sci-
ence, pp. 1137–1141. Nature Publishing Group, New York,
NY.

NY.
Gil, D. (2000). Syntactic categories, cross-linguistic varia tion
anduniversalgrammar. InVogel,P.M.andComrie,B.(Eds.),
Approaches to the Typology of Word Classes , pp. 173–216.
Mouton.
Gildea, D. and Hofmann, T. (1999). Topic-based language
models using EM. In EUROSPEECH-99 , Budapest, pp.
2167–2170.
Gildea, D. and Jurafsky, D. (1996). Learning bias and phono-
logicalruleinduction. ComputationalLinguistics ,22(4),497–
530.
Gildea, D. and Jurafsky, D. (2000). Automatic labeling of se-

NY.
Mouton.
530.
mantic roles. In ACL-00,Hong Kong, pp. 512–520.
Gildea, D. and Jurafsky, D. (2002). Automatic labeling of se-
mantic roles. Computational Linguistics ,28(3), 245–288.
Gillick, L. and Cox, S. (1989). Some statistical issues in
the comparison of speech recognition algorithms. In IEEE
ICASSP-89 ,pp. 532–535.
Girju,R.,Badulescu, A.,andMoldovan, D.(2006). Automatic
discovery ofpart-whole relations. Computational Linguistics ,
31(1).
Girju, R., Badulescu, A., and Moldovan, D. (2003). Learning

31(1).
semanticconstraintsfortheautomaticdiscoveryofpart-w hole
relations. In HLT-NAACL-03 ,Edmonton, Canada, pp.1–8.
Giv´on, T. (1990). Syntax: A functional typological introduc-
tion. John Benjamins, Amsterdam.

DRAFTBibliography 939
Glass, J.(2003). A probabilistic framework for segment-based
speech recognition. Computer Speech and Language, ,17(1–
2),137–152.
Glennie, A. (1960). On the syntax machine and the construc-
tion ofauniversal compiler. Tech. rep.No.2,Contr. NR049-
141, Carnegie Mellon University (at the time Carnegie Insti -
tute of Technology), Pittsburgh, PA.
Godfrey,J.,Holliman, E.,andMcDaniel, J.(1992). SWITCH-
BOARD: Telephone speech corpus for research and develop-
ment. InIEEEICASSP-92 , San Francisco, pp.517–520.
Gold, B. and Morgan, N. (1999). Speech and Audio Signal
Processing . Wiley Press.
Golding,A.R.andRoth,D.(1999). Awinnowbasedapproach
to context-sensitive spelling correction. Machine Learning ,
34(1-3), 107–130. Special Issue on Machine Learning and
Natural Language.
Golding, A.R. (1997). A bayesian hybrid method for context-
sensitive spelling correction. In Proceedings of the Third
Workshop on VeryLargeCorpora , Boston, MA,pp.39–53.
Golding, A. R. and Schabes, Y. (1996). Combin-

ing trigram-based and feature-based methods for context-
sensitive spelling COrrection. In ACL-96, Santa Cruz, CA,
pp.71–78.
Goldsmith , J.(1976). Autosegmental Phonology . Ph.D.thesis,
MIT.
Goldsmith , J. (1993). Harmonic phonology. In Goldsmith, J.
(Ed.),The Last Phonological Rule , pp. 21–60. University of
Chicago Press, Chicago.
Goldsmith , J. (2001). Unsupervised learning of the morphol-
ogy of a natural language. Computational Linguistics ,27,
153–198.
Goldstein , J., Mittal, V., Carbonell, J., and Kantrowitz, M.

pp.71–78.
MIT.
153–198.
(2000). Multi-document summarization by sentence extrac-
tion. InProceedings oftheANLP/NAACLWorkshop on Auto-
matic Summarization .
Goldwater , S. and Grifﬁths, T. L. (2007). A fully Bayesian
approach to unsupervised part-of-speech tagging. In ACL-07,
Prague, Czech Republic.
Goldwater , S., Grifﬁths, T. L., and Johnson, M. (2006). Con-
textual dependencies in unsupervised word segmentation. I n
COLING/ACL 2006 , Sydney, Australia.
Goldwater , S. and Johnson, M. (2003). Learning OT con-
straint rankings using a maximum entropy model. In Stock-

holm Workshop on Variation within Optimality Theory , pp.
111–120. Stockholm University Press.
Goldwater , S. and Johnson, M. (2005). Representational bias
in unsupervised learning of syllable structure. In Proceed-
ings of the Conference on Computational Natural Language
Learning (CoNLL-2005) .
Good, I. J. (1953). The population frequencies of species and
the estimation of population parameters. Biometrika ,40, 16–
264.
Good, M. D., Whiteside, J. A., Wixon, D. R., and Jones, S. J.

264.
(1984). Building auser-derived interface. Communications of
theACM,27(10), 1032–1043.
Goodman , J. (1997). Probabilistic feature grammars. In Pro-
ceedings of the International Workshop on Parsing Technol-
ogy.Goodman ,J.(2004). Exponential priors formaximumentropy
models. In ACL-04.
Goodman , J. (2006). A bit of progress in language model-
ing: Extended version. Tech. rep. MSR-TR-2001-72, Ma-
chine Learning and Applied Statistics Group, Microsoft Re-
search, Redmond, WA.
Goodwin , C. (1996). Transparent vision. In Ochs, E., Sche-

gloff, E. A., and Thompson, S. A. (Eds.), Interaction and
Grammar ,pp. 370–404. Cambridge University Press.
Gordon, D. and Lakoff, G. (1971). Conversational postulates.
InCLS-71, pp. 200–213. University of Chicago. Reprinted in
Peter Cole and Jerry L. Morgan (Eds.), Speech Acts: Syntax
and Semantics Volume3 ,Academic, 1975.
Gordon, P. C., Grosz, B. J., and Gilliom, L. A. (1993). Pro-
nouns, names, and the centering of attention in discourse.
Cognitive Science ,17(3), 311–347.
Gorin, A.L.,Riccardi, G.,and Wright, J.H.(1997). How may

ihelp you?. Speech Communication ,23,113–127.
G¨otz,T.,Meurers,W.D.,andGerdemann,D.(1997). TheCon-
Troll manual. Tech. rep., Seminar f¨ ur Sprachwissenschaft ,
Universit¨ at T¨ ubingen.
Gould, J. D., Conti, J., and Hovanyecz, T. (1983). Composing
letters with asimulated listening typewriter. Communications
ofthe ACM ,26(4), 295–308.
Gould, J. D. and Lewis, C. (1985). Designing for usability:
Key principles and what designers think. Communications of
the ACM,28(3), 300–311.
Gould,S.J.(1980). ThePanda’sThumb . Penguin Group,Lon-
don.

don.
Grabe, E., Post, B., Nolan, F.,and Farrar, K. (2000). Pitch ac-
centrealisation in fourvarieties ofBritish English. Journal of
Phonetics ,28, 161–186.
Grabe, E.(2001). TheIViElabelling guide..
Graff, D. (1997). The 1996 Broadcast News speech and
language-model corpus. In Proceedings DARPA Speech
Recognition Workshop , Chantilly, VA, pp. 11–14. Morgan
Kaufmann.
Gray,R.M.(1984).Vectorquantization. IEEETransactionson
Acoustics, Speech, and Signal Processing ,ASSP-1(2),4–29.

don.
Kaufmann.
Green, B. F., Wolf, A. K., Chomsky, C., and Laughery, K.
(1961). Baseball: An automatic question answerer. In Pro-
ceedings of the Western Joint Computer Conference 19 , pp.
219–224. Reprinted in Groszetal. (1986).
Greenberg , S., Ellis, D., and Hollenback, J. (1996). Insights
into spoken language gleaned from phonetic transcription o f
the Switchboard corpus. In ICSLP-96 , Philadelphia, PA, pp.
S24–27.
Greene, B. B. and Rubin, G. M. (1971). Automatic grammat-
ical tagging of English. Department of Linguistics, Brown

S24–27.
University, Providence, Rhode Island.
Grefenstette , G.(1994). Explorations in Automatic Thesaurus
Discovery . Kluwer, Norwell, MA.
Grefenstette , G. (1998). Producing intelligent telegraphic text
reduction to provide an audio scanning service for the blind .
InAAAI 1998 Spring Symposium on Intelligent Text Summa-
rization, pp.102–108.
Grefenstette , G. (1999). Tokenization. In van Halteren, H.
(Ed.),Syntactic Wordclass Tagging . Kluwer.

DRAFT940 Bibliography
Gregory, M. L. and Altun, Y. (2004). Using conditional ran-
dom ﬁelds to predict pitch accents in conversational speech .
InACL-04.
Gregory, M. L., Raymond, W. D., Bell, A., Fosler-Lussier, E.,
and Jurafsky, D. (1999). The effects of collocational stren gth
andcontextual predictability inlexicalproduction. In CLS-99,
pp.151–166. University of Chicago, Chicago.
Grenager , T. and Manning, C. D. (2006). Unsupervised Dis-
covery of aStatistical Verb Lexicon. In EMNLP2006 .
Grice, H.P.(1957). Meaning. Philosophical Review ,67, 377–
388. Reprinted in Semantics , edited by Danny D. Steinberg
& Leon A. Jakobovits (1971), Cambridge University Press,
pages 53–59.
Grice, H. P. (1975). Logic and conversation. In Cole, P. and
Morgan, J.L.(Eds.), Speech Acts: Syntax and Semantics Vol-
ume3,pp. 41–58. Academic Press.
Grice, H. P. (1978). Further notes on logic and conversation.
In Cole, P. (Ed.), Pragmatics: Syntax and Semantics Volume
9,pp. 113–127. Academic Press.

Gries, S. T. and Stefanowitsch, A. (Eds.). (2006). Corpus-
Based Approaches to Metaphor and Metonymy . Mouton de
Gruyter.
Grishman ,R.andSundheim,B.(1995). DesignoftheMUC-6
evaluation. In Proceedings ofthe Sixth Message Understand-
ing Conference (MUC-6) , San Francisco, pp. 1–11. Morgan
Kaufmann.
Grosjean , F. (1980). Spoken word recognition processes and
thegating paradigm. Perception andPsychophysics ,28,267–
283.
Grosjean , F., Grosjean, L., and Lane, H. (1979). The pat-

Gruyter.
Kaufmann.
283.
terns of silence: Performance structures in sentence produ c-
tion.Cognitive Psychology ,11, 58–81.
Grosz,B.J.andHirschberg, J.(1992). Someintonational char-
acteristics of discourse structure. In ICSLP-92 , Vol. 1, pp.
429–432.
Grosz, B. J. (1977a). The representation and use of focus in
a system for understanding dialogs. In IJCAI-77, pp. 67–76.
Morgan Kaufmann. Reprinted in Groszetal. (1986).
Grosz, B. J. (1977b). The Representation and Use of Focus in

429–432.
Dialogue Understanding . Ph.D.thesis, University of Califor-
nia, Berkeley.
Grosz, B. J., Joshi, A. K., and Weinstein, S. (1983). Provid-
ing a uniﬁed account of deﬁnite noun phrases in English. In
ACL-83,pp. 44–50.
Grosz,B.J.,Joshi,A.K.,andWeinstein,S.(1995a).Centering:
A framework for modeling the local coherence of discourse.
Computational Linguistics ,21(2), 203–225.
Grosz,B.J.,Joshi,A.K.,andWeinstein,S.(1995b).Centering:
A framework for modelling the local coherence of discourse.
Computational Linguistics ,21(2).

Grosz, B. J. and Sidner, C. L. (1980). Plans for discourse. In
Cohen,P.R.,Morgan,J.,andPollack, M.E.(Eds.), Intentions
in Communication , pp. 417–444. MITPress.
Grosz,B.J.andSidner,C.L.(1986). Attention, intentions, and
the structure of discourse. Computational Linguistics ,12(3),
175–204.
Grosz, B. J., Sparck Jones, K., and Webber, B. L. (Eds.).
(1986).Readings in Natural Language Processing . Morgan
Kaufmann, LosAltos, CA.Gruber,J.S.(1965). StudiesinLexicalRelations . Ph.D.thesis,
MIT.
Grudin, J.T. (1983). Error patterns in novice and skilled tran-

175–204.
MIT.
scription typing. In Cooper, W.E.(Ed.), Cognitive Aspects of
Skilled Typewriting , pp.121–139. Springer-Verlag.
Guindon , R. (1988). A multidisciplinary perspective on di-
alogue structure in user-advisor dialogues. In Guindon, R.
(Ed.),Cognitive Science And Its Applications For Human-
Computer Interaction , pp. 163–200. Lawrence Erlbaum.
Gundel, J. K., Hedberg, N., and Zacharski, R. (1993). Cogni-
tive status and the form of referring expressions in discour se.
Language ,69(2), 274–307.

Gupta,V.,Lennig,M.,andMermelstein, P.(1988). Fastsearch
strategy in alarge vocabulary word recognizer. Journal ofthe
Acoustical Society ofAmerica ,84(6), 2007–2017.
Gupta,V.,Lennig,M.,andMermelstein, P.(1992). Alanguage
model for very large-vocabulary speech recognition. Com-
puter Speech and Language ,6,331–344.
Gusﬁeld, D. (1997). Algorithms on strings, trees, and se-
quences: computer science andcomputational biology . Cam-
bridge University Press.
Guy, G. R. (1980). Variation in the group and the individual:

The case of ﬁnal stop deletion. In Labov, W. (Ed.), Locating
Language in Time and Space , pp.1–36. Academic.
Habash,N.,Rambow,O.,andKiraz,G.A.(2005).Morpholog-
ical analysis and generation forarabic dialects. In ACLWork-
shoponComputationalApproachestoSemiticLanguages ,pp.
17–24.
Hachey,B.andGrover,C.(2005).Sentenceextraction forlegal
text summarization. In IJCAI-05, pp.1686–1687.
Hafer, M. A. and Weiss, S. F. (1974). Word segmentation by
letter successor varieties. Information Storage and Retrieval ,
10(11-12), 371–385.

17–24.
Haghighi , A.and Klein, D.(2006). Prototype-driven grammar
induction. In COLING/ACL 2006 , pp.881–888.
Hain, T. (2002). Implicit pronunciation modelling in asr. In
Proceedings of ISCAPronunciation Modeling Workshop .
Hain,T.,Woodland,P.C.,Evermann,G.,andPovey,D.(2001).
Newfeatures intheCU-HTKsystem fortranscription ofcon-
versational telephone speech. In IEEEICASSP-01 , Salt Lake
City, Utah.
Hajiˇc, J. (1998). Building a Syntactically Annotated Corpus:
The Prague Dependency Treebank , pp. 106–132. Karolinum,
Prague/Praha.

Hajiˇc, J.(2000). Morphological tagging: Data vs.dictionaries.
InNAACL2000 . Seattle.
Hajiˇc, J. and Hladk´ a, B. (1998). Tagging inﬂective languages:
Prediction of morphological categories for a rich, structu red
tagset. In COLING/ACL-98 , Montreal, Canada.
Hajiˇc, J., Krbec, P., Kvˇ etoˇ n, P., Oliva, K., and Petkeviˇ c, V.
(2001). Serial Combination of Rules and Statistics: A Case
Study in Czech Tagging. In ACL-01,Toulouse, France.
Hakkani-T¨ ur , D., Oﬂazer, K., and T¨ ur, G. (2002). Statistical

morphological disambiguation for agglutinative language s.
Journal of Computers and Humanities ,36(4).
Hale, J. (2006). Uncertainty about the rest of the sentence.
Cognitive Science ,30(4), 609–642.

DRAFTBibliography 941
Hall, K. and Johnson, M. (2003). Language modeling using
efﬁcient best-ﬁrst bottom-up parsing. In IEEE ASRU-03 , pp.
507–512.
Halliday, M. A. K. (1985). An Introduction to Functional
Grammar . Edward Arnold, London.
Halliday,M.A.K.andHasan,R.(1976). Cohesion inEnglish .
Longman,London. English Language Series, Title No.9.
Hammond ,M.(1997).ParsinginOT.Alternativetitle“Parsing
syllables: Modeling OT computationally”. Rutgers Optimal -
ity Archive ROA-222-1097.
Hamza,W.,Bakis,R.,Eide,E.M.,Picheny,M.A.,andPitrelli,
J.F.(2004). TheIBM expressive speech synthesis system. In
ICSLP-04 ,Jeju, Korea.
Hankamer , J.(1986). Finite state morphology and left to right
phonology. In ProceedingsoftheFifthWestCoastConference
on FormalLinguistics , pp. 29–34.
Hankamer , J. and Black, H. A. (1991). Current approaches to
computational morphology. Unpublished manuscript.
Harabagiu , S., Pasca, M., and Maiorano, S. (2000). Ex-
periments with open-domain textual question answering. In

COLING-00 , Saarbr¨ ucken, Germany.
Harris,C.M.(1953). Astudyofthebuilding blocksinspeech.
JournaloftheAcousticalSociety ofAmerica ,25(5),962–969.
Harris, R. A. (2005). Voice Interaction Design: Crafting the
New Conversational Speech Systems . Morgan Kaufmann.
Harris, Z.S.(1946). From morphemeto utterance. Language ,
22(3), 161–183.
Harris, Z. S. (1954). Distributional structure. Word,10, 146–
162. Reprinted in J. Fodor and J. Katz, The structure of lan-

guage: Readings inthephilosophy oflanguage, Prentice-ha ll,
1964 and in Z. S. Harris, Papers in structural and transforma -
tional linguistics, Reidel, Dordrecht, 1970, 775–794.
Harris, Z. S. (1962). String Analysis of Sentence Structure .
Mouton, TheHague.
Harris, Z. S. (1968). Mathematical Structures of Language .
John Wiley.
Harris, Z. S. (1988). Language and Information . Columbia
Hasegawa-Johnson , M. and et al (2005). Landmark-based
speech recognition: Report of the 2004 Johns Hopkins Sum-
merWorkshop. In IEEEICASSP-05 .

Hastie, T., Tibshirani, R., and Friedman, J. H. (2001). The El-
ements ofStatistical Learning . Springer.
Haviland , S.E.and Clark, H.H.(1974). What’s new? Acquir-
ing new information as a process in comprehension. Journal
ofVerbalLearning and VerbalBehaviour ,13, 512–521.
Hayes, B. (2004). Phonological acquisition in optimality the-
ory: the early stages. In Kager, R., Pater, J., and Zonn-
eveld, W. (Eds.), Constraints in Phonological Acquisition .

Hayes, B. and Wilson, C. (2007). A maximum entropy model
of phonotactics and phonotactic learning. Linguistic Inquiry .
Toappear.
Hayes, E. and Bayer, S. (1991). Metaphoric generalization
through sort coercion. In ACL-91, Berkeley, CA, pp. 222–
228.ACL.Hearst, M. A. (1991). Noun homograph disambiguation. In
Proceedings of the 7th Annual Conference of the University
of Waterloo Centre for the New OED and Text Research , Ox-
ford, pp.1–19.
Hearst, M. A. (1992). Automatic acquisition of hyponyms

Toappear.
fromlarge text corpora. In COLING-92 , Nantes, France.
Hearst,M.A.(1994). Multi-paragraph segmentation ofexpos-
itory text. In ACL-94,pp. 9–16.
Hearst, M. A. (1997). Texttiling: Segmenting text into multi-
paragraph subtopic passages. Computational Linguistics ,23,
33–64.
Hearst, M. A. (1998). Automatic discovery of WordNet rela-
tions. In Fellbaum, C. (Ed.), Wordnet: An Electronic Lexical
Heeman, P. A. (1999). POS tags and decision trees for lan-

33–64.
guage modeling. In EMNLP/VLC-99 , College Park, MD, pp.
129–137.
Heeman, P. A. and Allen, J. (1999). Speech repairs, intona-
tional phrases and discourse markers: Modeling speakers’ u t-
terances in spoken dialog. Computational Linguistics ,25(4).
Heikkil¨a, J. (1995). A TWOL-based lexicon and feature sys-
tem for English. In Karlsson, F., Voutilainen, A., Heikkil¨ a,
J.,and Anttila, A. (Eds.), Constraint Grammar: A Language-
Independent System for Parsing Unrestricted Text , pp. 103–
131. Mouton deGruyter, Berlin.

129–137.
Heim, I. and Kratzer, A. (1998). Semantics in a Generative
Grammar . Blackwell Publishers, Malden, MA.
Heinz, J. M. and Stevens, K. N. (1961). On the properties of
voiceless fricative consonants. Journal of the Acoustical So-
ciety ofAmerica ,33, 589–596.
Hemphill , C. T., Godfrey, J., and Doddington, G. (1990). The
ATIS spoken language systems pilot corpus. In Proceed-
ings DARPA Speech and Natural Language Workshop , Hid-
den Valley, PA,pp.96–101. Morgan Kaufmann.
Hermansky , H. (1990). Perceptual linear predictive (PLP)

analysis ofspeech. JournaloftheAcousticalSociety ofAmer-
ica,87(4), 1738–1752.
Hillard, D., Huang, Z., Ji, H., Grishman, R., Hakkani-T¨ ur, D.,
Harper, M., Ostendorf, M., and Wang, W. (2006). Impact of
automatic comma prediction on pos/name tagging of speech.
InProceedings of IEEE/ACL 06 Workshop on Spoken Lan-
guage Technology , Aruba.
Hindle, D. (1983). Deterministic parsing of syntactic non-
ﬂuencies. In ACL-83, pp. 123–128.
Hindle, D. and Rooth, M. (1990). Structural ambiguity and
lexical relations. In Proceedings DARPASpeech and Natural

Language Workshop , Hidden Valley, PA, pp. 257–262. Mor-
gan Kaufmann.
Hindle, D. and Rooth, M. (1991). Structural ambiguity and
lexical relations. In ACL-91,Berkeley, CA, pp.229–236.
Hindle, D. (1990). Noun classiﬁcation from predicate-
argumentstructures. In ACL-90,Pittsburgh, PA,pp.268–275.
Hinkelman , E. A. and Allen, J. (1989). Two constraints on
speech act ambiguity. In ACL-89, Vancouver, Canada, pp.
212–219.
Hintikka , J. (1969). Semantics for propositional attitudes. In

212–219.
Davis,J.W.,Hockney, D.J.,and Wilson,W.K.(Eds.), Philo-
sophical Logic , pp. 21–45. D.Reidel, Dordrecht, Holland.

DRAFT942 Bibliography
Hirschberg , J. and Litman, D. J.(1993). Empirical Studies on
the Disambiguation of Cue Phrases. Computational Linguis-
tics,19(3), 501–530.
Hirschberg , J.(1993). Pitch Accent in Context: Predicting In-
tonationalProminencefromText. ArtiﬁcialIntelligence ,63(1-
2),305–340.
Hirschberg , J. and Litman, D. J. (1993). Empirical studies on
the disambiguation of cue phrases. Computational Linguis-
Hirschberg , J., Litman, D. J., and Swerts, M. (2001). Identi-
fying user corrections automatically in spoken dialogue sy s-
tems. InNAACL 2001 .
Hirschberg , J.andNakatani, C.(1996). Aprosodicanalysis of
discourse segments in direction-giving monologues. In ACL-
96,Santa Cruz, CA,pp. 286–293.
Hirschberg , J. and Pierrehumbert, J. B. (1986). The intona-
tional structuring of discourse. In ACL-86, New York, pp.
136–144.
Hirschman , L. and Blaschke, C. (2006). Evaluation of text
mininginbiology. InAnaniadou, S.and McNaught, J.(Eds.),

136–144.
Text Mining for Biology and Biomedicine , chap. 9, pp. 213–
245.Artech House, Norwood, MA.
Hirschman , L.and Pao,C.(1993). Thecostof errors in aspo-
ken language system. In EUROSPEECH-93 ,pp. 1419–1422.
Hirst,G.(1987). SemanticInterpretation andtheResolutionof
Ambiguity . Cambridge University Press.
Hirst, G.(1988). Resolving lexical ambiguity computationally
withspreading activation andpolaroid words. InSmall, S.L .,
Cottrell, G. W., and Tanenhaus, M. K. (Eds.), Lexical am-
biguity resolution: Perspectives from psycholinguistics , neu-

ropsychology, and artiﬁcial intelligence , pp. 73–108. Morgan
Kaufmann.
Hirst, G. and Budanitsky, A. (2005). Correcting real-word
spelling errors by restoring lexical cohesion. Natural Lan-
guage Engineering ,11,87–111.
Hirst, G. and Charniak, E. (1982). Word sense and case slot
disambiguation. In AAAI-82,pp. 95–98.
Hjelmslev , L. (1969). Prologomena to a Theory of Language .
University ofWisconsin Press. Translated byFrancisJ.Whi t-
ﬁeld; original Danish edition 1943.
Hobbs, J. R. (1977). 38 examples of elusive antecedents from

Kaufmann.
published texts. Tech. rep. 77-2, Department of Computer
Science, City University of New York.
Hobbs, J. R. (1978). Resolving pronoun references. Lingua,
44,311–338. Reprinted in Groszet al. (1986).
Hobbs, J. R. (1979a). Coherence and coreference. Cognitive
Science,3,67–90.
Hobbs, J. R. (1979b). Metaphor, metaphor schemata, and se-
lective inferencing. Tech. rep. 204, SRI.
Hobbs, J. R. (1990). Literature and Cognition . CSLI Lecture
Notes 21.
Hobbs, J. R., Appelt, D. E., Bear, J., Israel, D., Kameyama,

Notes 21.
M., Stickel, M. E., and Tyson, M. (1997). FASTUS: A cas-
caded ﬁnite-state transducer for extracting information f rom
natural-language text. In Roche, E. and Schabes, Y. (Eds.),
Finite-State Language Processing , pp. 383–406. MIT Press.
Hobbs, J. R. and Shieber, S.M. (1987). An algorithm for gen-
erating quantiﬁer scopings. Computational Linguistics ,13(1),
47–55.Hobbs, J. R., Stickel, M. E., Appelt, D. E., and Martin, P.
(1993). Interpretation as abduction. Artiﬁcial Intelligence ,
63,69–142.

Hockenmaier , J.andSteedman,M.(2002). Generative models
forstatistical parsing with Combinatory Categorial Gramm ar.
InACL-02,Philadelphia, PA.
Hofstadter , D.R.(1997). Leton beau demarot . Basic Books.
Holmes, D. I. (1994). Authorship attribution. Computers and
the Humanities ,28, 87–106.
Honal, M. and Schultz, T. (2003). Correction of disﬂuencies
in spontaneous speech using a noisy-channel approach. In
EUROSPEECH-03 .
Honal, M. and Schultz, T. (2005). Automatic disﬂuency re-

moval on recognized spontaneous speech - rapid adaptation
to speaker-dependent disﬂuencies. In IEEEICASSP-05 .
Hopcroft , J. E. and Ullman, J. D. (1979). Introduction to
Automata Theory, Languages, and Computation . Addison-
Wesley, Reading, MA.
Hori, C. and Furui, S. (2004). Speech summarization: an ap-
proach through word extraction and a method for evaluation.
IEICETransactions on Information and Systems ,87,15–25.
Horning, J.J.(1969). Astudyofgrammatical inference . Ph.D.
thesis, Stanford University.

House, A. S., Williams, C. E., Hecker, M. H. L., and Kryter,
K. D. (1965). Articulation-Testing Methods: Consonantal
Differentiation with a Closed-Response Set. Journal of the
Acoustical Society ofAmerica ,37,158–166.
Householder , F. W.(1995). Dionysius Thrax, the technai, and
SextusEmpiricus.InKoerner,E.F.K.andAsher,R.E.(Eds.) ,
Concise History of the Language Sciences , pp. 99–103. Else-
vier Science, Oxford.
Hovy, E., Hermjakob, U., and Ravichandran, D. (2002). A

question/answer typology with surface text patterns. In HLT-
01.
Hovy,E.H.(1990). Parsimonious and proﬂigateapproaches to
thequestionofdiscoursestructurerelations. In Proceedingsof
the Fifth International Workshop on Natural Language Gen-
eration, Dawson, PA,pp.128–136.
Hovy,E.H.andLin,C.-Y.(1999). Automated textsummariza-
tion inSUMMARIST. InMani, I.and Maybury, M.T.(Eds.),
Advances in Automatic Text Summarization , pp. 81–94. MIT
Press.
Hovy, E. H., Marcus, M. P., Palmer, M., Ramshaw, L. A., and

01.
Press.
Weischedel, R. (2006). Ontonotes: The 90% solution. In
HLT-NAACL-06 .
Howes, D. (1957). On the relation between the intelligibility
and frequency of occurrence of English words. Journal ofthe
Acoustical Society ofAmerica ,29,296–305.
Huang, C., Chang, E., Zhou, J., and Lee, K.-F. (2000). Ac-
cent modeling based on pronunciation dictionary adaptatio n
forlarge vocabulary Mandarin speech recognition. In ICSLP-
00,Beijing, China.
Huang, L. and Chiang, D. (2005). Better k-best parsing. In
IWPT-05, pp.53–64.

Huang, X., Acero, A., and Hon, H.-W. (2001). Spoken Lan-
guage Processing: AGuidetoTheory, Algorithm, and System
Development . Prentice Hall, Upper Saddle River, NJ.

DRAFTBibliography 943
Huddleston , R. and Pullum, G. K. (2002). The Cambridge
grammar of the English language . Cambridge University
Press.
Hudson,R.A.(1984). Word Grammar . Blackwell, Oxford.
Huffman , D. A. (1954). The synthesis of sequential switching
circuits.Journal of the Franklin Institute ,3, 161–191. Con-
tinued in Volume 4.
Huffman , S. (1996). Learning information extraction patterns
from examples. In Wertmer, S., Riloff, E., and Scheller, G.
(Eds.),Connectionist, Statistical, and Symbolic Approaches
to Learning Natural Language Processing , pp. 246–260.
Springer, Berlin.
Huls, C., Bos, E., and Classen, W. (1995). Automatic referent
resolution of deictic and anaphoric expressions. Computa-
tional Linguistics ,21(1), 59–79.
Hunt, A. J. and Black, A. W. (1996a). Unit selection in a
concatenative speech synthesis system using a large speech
database. In IEEEICASSP-96 , Atlanta, GA, Vol. 1, pp. 373–
376.IEEE.
Hunt, A. J. and Black, A. W. (1996b). Unit selection in a

376.IEEE.
database. In IEEEICASSP-06 , Vol. 1, pp.373–376.
Hutchins , J. (1997). From ﬁrst conception to ﬁrst demonstra-
tion: the nascent years of machine translation, 1947–1954. A
chronology. Machine Translation ,12,192–252.
Hutchins , W. J. and Somers, H. L. (1992). An Introduction to
Machine Translation . Academic Press.
Hutchins , W. J. (1986). Machine Translation: Past, Present,
Future. Ellis Horwood, Chichester, England.
Huybregts , R. (1984). The weak inadequacy of context-free

phrase structure grammars. In de Haan, G., Trommele, M.,
and Zonneveld, W. (Eds.), Van Periferie naar Kern . Foris,
Dordrecht. Cited in Pullum (1991).
Ide, N. M. and Veronis, J. (Eds.). (1998a). Computational
Linguistics: Special Issue on Word Sense Disambiguation ,
Vol. 24. MITPress.
Ide, N. M. and V´ eronis, J. (1998b). Introduction to the special
issueon word sensedisambiguation. Computational Linguis-
tics,24(1), 1–40.
Irons,E.T.(1961). Asyntaxdirected compiler forALGOL60.
Communications of theACM ,4,51–55.

ISO8601 (2004). Dataelementsandinterchange formats infor-
mation interchange representation of dates and times. Tech .
rep.,International Organization for Standards (ISO).
Issar, S. and Ward, W. (1993). Cmu’s robust spoken language
understanding system. In Eurospeech 93 ,pp. 2147–2150.
Itakura, F. (1975). Minimum prediction residual principle ap-
plied to speech recognition. IEEETransactions on Acoustics,
Speech, and Signal Processing ,ASSP-32,67–72.
Iverson, E. and Helmreich, S. (1992). Metallel: An integrated

approach to non-literal phrase interpretation. Computational
Intelligence ,8(3).
Iyer, R. M. and Ostendorf, M. (1999a). Modeling long dis-
tance dependencies in language: Topic mixtures versus dy-
namiccache model. IEEETransactions on Speech and Audio
Processing ,7.
Iyer, R. M. and Ostendorf, M. (1999b). Relevance weighting
for combining multi-domain data for n-gram language mod-
eling.Computer Speech and Language ,13(3), 267–282.Iyer, R. M. and Ostendorf, M. (1997). Transforming out-of-
domain estimates to improve in-domain language models. In

EUROSPEECH-97 ,pp. 1975–1978.
Jaccard,P.(1908). Nouvellesrecherches surladistribution ﬂo-
rale.Bulletin de la Soci´ et´ e Vaudoise des Sciences Naturelles ,
44,223–227.
Jaccard, P. (1912). The distribution of the ﬂora of the alpine
zone.New Phytologist ,11,37–50.
Jackendoff , R. (1975). Morphological and semantic regulari-
ties in the lexicon. Language ,51(3), 639–671.
Jackendoff , R. (1983). Semantics and Cognition . MIT Press.
Jackendoff , R. (1990). Semantic Structures . MITPress.

Jackson, P. and Moulinier, I. (2002). Natural language pro-
cessing for online applications: text retrieval, extracti on, and
categorization. John Benjamins Publishing Company.
Jacobs, P. (1985). A Knowledge-Based Approach to Language
Generation . Ph.D. thesis, University of California, Berkeley,
CA. Available as University of California at Berkeley Com-
puter Science Division Tech. rep. #86/254.
Jacobs, P. (1987). Knowledge-based natural language genera-
tion.Artiﬁcial Intelligence ,33, 325–378.

Jacobs, P.S.and Rau, L.F.(1990). SCISOR:A system forex-
tracting information from on-line news. Communications of
the ACM,33(11), 88–97.
Jacquemin , C. (1997). Guessing morphology from terms and
corpora. In SIGIR 1997 , Philadelphia, PA,pp. 156–165.
Jakobson , R. (1939). Observations sur le classement
phonologique des consonnes. In Blancquaert, E. and P´ ee, W.
(Eds.),ICPhS-39 , Ghent, pp. 34–41.
Janssen,T.M.V.(1997). Compositionality. InvanBenthem,J.
and ter Meulen, A.(Eds.), Handbook ofLogic andLanguage ,

chap. 7, pp.417–473. North-Holland, Amsterdam.
Jarosz, G.(2006). Richness ofthebaseand probabilistic unsu-
pervised learning inoptimality theory. In Proceedings ofACL
SIGPHON ,New York, NY,pp. 50–59.
Jarosz, G. (2008). Restrictiveness and phonological grammar
and lexicon learning. In CLS 43. In press.
J¨arvinen, T. and Tapanainen, P. (1997). A dependency parser
forEnglish. Tech.rep.TR-1,Department ofGeneralLinguis -
tics, University of Helsinki, Helsinki.
Jefferson , G. (1984). Notes on asystematic deployment of the

acknowledgement tokens‘yeah’and‘mmhm’. PapersinLin-
guistics,17(2), 197–216.
Jeffreys, H. (1948). Theory of Probability . Clarendon Press,
Oxford. 2nd edn Section 3.23.
Jekat, S., Klein, A., Maier, E., Maleck, I., Mast, M., and
Quantz, J. (1995). Dialogue acts in verbmobil.. Verbmobil–
Report–65–95.
Jelinek, F. (1969). A fast sequential decoding algorithm us-
ing a stack. IBM Journal of Research and Development ,13,
675–685.
Jelinek, F.(1976). Continuous speechrecognition bystatistical
methods. Proceedings ofthe IEEE ,64(4), 532–557.

675–685.
Jelinek, F. (1988). Address to the ﬁrst workshop on the eval-
uation of natural language processing systems. December 7,
1988.

DRAFT944 Bibliography
Jelinek, F. (1990). Self-organized language modeling for
speechrecognition. InWaibel,A.andLee,K.-F.(Eds.), Read-
ingsinSpeech Recognition , pp.450–506. MorganKaufmann,
Los Altos. Originally distributed as IBM technical report i n
1985.
Jelinek, F.(1997). Statistical Methods forSpeech Recognition .
MITPress.
Jelinek, F.andLafferty, J.D.(1991). Computation oftheprob-
ability of initial substring generation by stochastic cont ext-
freegrammars. Computational Linguistics ,17(3), 315–323.
Jelinek, F., Lafferty, J. D., Magerman, D. M., Mercer, R. L.,
Ratnaparkhi, A., and Roukos, S. (1994). Decision tree pars-
ing using a hidden derivation model. In ARPA Human Lan-
guageTechnologies Workshop ,Plainsboro, N.J.,pp.272–277.
Morgan Kaufmann.
Jelinek, F.andMercer, R.L.(1980). Interpolated estimation of
Markovsourceparametersfromsparsedata. InGelsema,E.S.
and Kanal, L. N. (Eds.), Proceedings, Workshop on Pattern
Recognition in Practice , pp. 381–397. North Holland, Ams-
terdam.

terdam.
Jelinek, F., Mercer, R. L., and Bahl, L. R. (1975). Design of a
linguistic statistical decoder for therecognition of cont inuous
speech.IEEE Transactions on Information Theory ,IT-21(3),
250–256.
Jiang, J. J. and Conrath, D. W. (1997). Semantic similarity
basedoncorpusstatistics andlexical taxonomy. In ROCLING
X,Taiwan.
Jilka, M.,Mohler, G.,and Dogil, G.(1999). Rules for thegen-
eration of ToBI-based American English intonation. Speech
Communication ,28(2), 83–108.
Jim´enez, V. M. and Marzal, A. (2000). Computation of the n

250–256.
X,Taiwan.
bestparsetreesforweightedandstochasticcontext-freeg ram-
mars. In Advances in Pattern Recognition: Proceedings of
theJoint IAPRInternational Workshops, SSPR2000and SPR
2000, Alicante, Spain, pp. 183–192. Springer.
Jing, H.(2000). Sentence reduction forautomatic text summa-
rization. In ANLP2000 , Seattle, WA,pp.310–315.
Jing,H.(2002). Usinghidden Markov modeling todecompose
human-written summaries. Computational Linguistics ,28(4),
527–543.
Johnson, C. D. (1972). Formal Aspects of Phonological De-
scription. Mouton, The Hague. Monographs on Linguistic

527–543.
Analysis No.3.
Johnson, C. (1999). Syntactic and semantic principles of
FrameNet annotation, version 1. Tech. rep. TR-99-018, ICSI ,
Berkeley, CA.
Johnson, K. (2003). Acoustic and Auditory Phonetics . Black-
well, Oxford. 2nd ed.
Johnson,M.(1984). Adiscovery procedure for certain phono-
logical rules. In COLING-84 , Stanford, CA,pp. 344–347.
Johnson, M. (1988). Attribute-Value Logic and the Theory of
Grammar . CSLI Lecture Notes. Chicago University Press,
Chicago.

Chicago.
Johnson, M. (1990). Expressing disjunctive and negative fea-
ture constraints with classical ﬁrst-order logic. In ACL-90,
Pittsburgh, PA,pp. 173–179.
Johnson,M.(1998a). Finite-state approximation ofconstraint-
based grammars using left-corner grammar transforms. In
COLING/ACL-98 , Montreal, pp.619–623.Johnson, M. (1998b). PCFG models of linguistic tree repre-
sentations. Computational Linguistics ,24(4), 613–632.
Johnson, M. (2001). Joint and conditional estimation of tag-
ging and parsing models. In ACL-01, pp. 314–321.

Johnson, M., Geman, S., Canon, S., Chi, Z., and Riezler, S.
(1999). Estimators for stochastic “uniﬁcation-based” gra m-
mars. InACL-99, pp. 535–541.
Johnson,S.C.andLesk,M.E.(1978). Languagedevelopment
tools.Bell System Technical Journal ,57(6), 2155–2175.
Johnson, W. E. (1932). Probability: deductive and inductive
problems (appendix to). Mind,41(164), 421–423.
Johnson-Laird , P.N.(1983). MentalModels . Harvard Univer-
sity Press, Cambridge, MA.
Johnston , M., Ehlen, P., Gibbon, D., and Liu, Z. (2007). The

multimodal presentation dashboard. In NAACL HLT 2007
Workshop ’Bridging the Gap’ .
Jones, D. A., Gibson, E., Shen, W., Granoien, N., Herzog, M.,
Reynolds, D. A., and Weinstein, C. (2005). Measuring hu-
man readability of machine generated text: Three case stud-
ies in speech recognition and machine translation. In IEEE
ICASSP-05 ,pp. 18–23.
Jones, D. A., Wolf, F., Gibson, E., Williams, E., Fedorenko,
E., Reynolds, D. A., and Zissman, M. (2003). Measuring

the readability of automatic speech-to-text transcripts. In
EUROSPEECH-03 ,pp. 1585–1588.
Jones, M. A. and McCoy, K. (1992). Transparently-motivated
metaphor generation. In Dale, R., Hovy, E. H., R¨ osner, D.,
andStock,O.(Eds.), AspectsofAutomatedNaturalLanguage
Generation , Lecture Notes in Artiﬁcial Intelligence 587, pp.
183–198. Springer Verlag, Berlin.
Jones, M. P. and Martin, J. H. (1997). Contextual spelling
correction using latent semantic analysis. In Proceedings of
the 5th Conference on Applied Natural Language Processing

(ANLP’97) ,Washington, D.C.,pp.166–173.
Joos, M. (1950). Description of language design. Journal of
the Acoustical Society ofAmerica ,22, 701–708.
Joshi, A. K., Vijay-Shanker, K., and Weir, D. J. (1991).
Theconvergence ofmildly context-sensitive grammatical f or-
malisms. InSells, P.,Shieber, S.,andWasow,T.(Eds.), Foun-
dational issues in natural language processing , pp. 31–81.
MITPress.
Joshi, A. K. (1985). Tree adjoining grammars: how much
context-sensitivity isrequired toprovidereasonable str uctural

MITPress.
descriptions?. InDowty,D.R.,Karttunen, L.,and Zwicky,A .
(Eds.),Natural Language Parsing , pp. 206–250. Cambridge
Joshi, A.K.and Hopely, P.(1999). A parser from antiquity. In
Kornai, A. (Ed.), Extended Finite State Models of Language ,
pp.6–15. Cambridge University Press.
Joshi, A. K. and Kuhn, S. (1979). Centered logic: The role
of entity centered sentence representation in natural lang uage
inferencing. In IJCAI-79, pp. 435–439.

Joshi, A. K. and Srinivas, B. (1994). Disambiguation of super
parts of speech (or supertags): Almost parsing. In COLING-
94,Kyoto, pp. 154–160.
Joshi, A. K. and Weinstein, S. (1981). Control of inference:
Role of some aspects of discourse structure – centering. In
IJCAI-81, pp. 385–387.

DRAFTBibliography 945
Jun,S.-A.(Ed.).(2005). ProsodicTypologyandTranscription:
AUniﬁed Approach . Oxford University Press.
Juneja, A. and Espy-Wilson, C. (2003). Speech segmenta-
tionusingprobabilisticphoneticfeaturehierarchyandsu pport
vector machines. In IJCNN 2003 .
Junqua, J. C. (1993). The Lombard reﬂex and its role on hu-
man listeners and automatic speech recognizers. Journal of
theAcoustical Society ofAmerica ,93(1), 510–524.
Juola,P.(1999). Measuring linguistic complexity. Presented at
the 4th Conference on Conceptual Structure, Discourse, and
Language (CSDL-4), Georgia.
Jurafsky, D., Ward, W., Jianping, Z., Herold, K., Xiuyang, Y.,
and Sen, Z. (2001). What kind of pronunciation variation is
hard for triphones to model?. In IEEE ICASSP-01 , Salt Lake
City, Utah, pp. I.577–580.
Jurafsky, D. (1992). An On-line Computational Model of Hu-
man Sentence Interpretation: A Theory of the Representa-
tion and Use of Linguistic Knowledge . Ph.D. thesis, Univer-

sity of California, Berkeley, CA. Available as University o f
California at Berkeley Computer Science Division Tech. rep .
#92/676.
Jurafsky, D.,Bell, A.,Fosler-Lussier, E.,Girand, C.,and Ray-
mond, W. D. (1998). Reduction of English function words in
Switchboard. In ICSLP-98 , Sydney, Vol. 7,pp. 3111–3114.
Jurafsky, D., Bell, A., Gregory, M. L., and Raymond, W. D.
(2001). Probabilistic relations betweenwords: Evidencef rom
reduction inlexical production. InBybee, J.L.andHopper, P.
(Eds.),Frequency and the Emergence of Linguistic Structure ,

#92/676.
pp.229–254. Benjamins, Amsterdam.
Jurafsky,D.,Wooters,C.,Tajchman, G.,Segal,J.,Stolcke, A.,
Fosler, E., and Morgan, N. (1994). The Berkeley restaurant
project. In ICSLP-94 , Yokohama, Japan, pp. 2139–2142.
Fosler,E.,andMorgan,N.(1995). Usingastochasticcontex t-
free grammar as a language model for speech recognition. In
IEEEICASSP-95 , pp.189–192. IEEE.
Kager, R. (2000). Optimality Theory . Cambridge University
Press.

Press.
Kahn, D. (1976). Syllable-based Generalizations in English
Phonology . Ph.D.thesis, MIT.
Kameyama , M. (1986). A property-sharing constraint in cen-
tering. In ACL-86,New York,pp. 200–206.
Kamm, C. A. (1994). User interfaces for voice applications.
In Roe, D. B. and Wilpon, J. G. (Eds.), Voice Communica-
tion Between Humans and Machines , pp. 422–442. National
Academy Press,Washington, D.C.
Kan,M.Y.,Klavans,J.L.,andMcKeown,K.R.(1998). Linear
segmentation and segment signiﬁcance. In Proc. 6th Work-

Press.
shop on Very Large Corpora (WVLC-98) , Montreal, Canada,
pp.197–205.
Kaplan,R.M.,Riezler, S.,King,T.H.,Maxwell, J.T.,Vasser-
man, A., and Crouch, R. (2004). Speed and accuracy in shal-
low and deep stochastic parsing. In HLT-NAACL-04 .
Kaplan,R.M.(1973).Ageneralsyntacticprocessor.InRustin,
R. (Ed.), Natural Language Processing , pp. 193–241. Algo-
rithmics Press,New York.
Kaplan,R.M.(1987). Threeseductions ofcomputational psy-

cholinguistics. InWhitelock, P.,Wood,M.M.,Somers,H.L. ,Johnson, R., and Bennett, P. (Eds.), Linguistic Theory and
Computer Applications , pp. 149–188. Academic Press, Lon-
don.
Kaplan, R. M. and Bresnan, J. (1982). Lexical-functional
grammar: Aformalsystemforgrammaticalrepresentation. I n
Bresnan, J.(Ed.), TheMental Representation ofGrammatical
Relations, pp. 173–281. MIT Press.
Kaplan, R. M. and Kay, M. (1981). Phonological rules and
ﬁnite-state transducers. Paper presented at the Annual mee t-

don.
ing of the Linguistics Society of America. New York.
Kaplan, R. M. and Kay, M.(1994). Regular models of phono-
logical rule systems. Computational Linguistics ,20(3), 331–
378.
Karamanis , N. (2003). Entity Coherence for Descriptive Text
Structuring . Ph.D.thesis, University of Edinburgh.
Karamanis , N. (2006). Evaluating centering for sentence or-
dering in two new domains. In HLT-NAACL-06 .
Karamanis , N. (2007). Supplementing entity coherence with
local rhetorical relations for information ordering. Journal of

378.
Logic, Language and Information . Toappear.
Karamanis , N. and Manurung, H. M. (2002). Stochastic text
structuring using the principle of continuity. In INLG 2002 ,
pp.81–88.
Karamanis , N., Poesio, M., Mellish, C., and Oberlander, J.
(2004). Evaluating centering-based metrics of coherence f or
text structuring using areliably annotated corpus. In ACL-04.
Karlsson ,F.(2007). Constraints onmultiplecenter-embedding
of clauses. Journal ofLinguistics ,43,365–392.
Karlsson , F., Voutilainen, A., Heikkil¨ a, J., and Anttila, A.

pp.81–88.
(1995a). Constraint Grammar — A language-independent
system for parsing unrestricted text . Mouton de Gruyter,
Berlin.
Karlsson , F., Voutilainen, A., Heikkil¨ a, J., and Anttila,
A. (Eds.). (1995b). Constraint Grammar: A Language-
Independent System for Parsing Unrestricted Text . Mouton
deGruyter, Berlin.
Karttunen , L., Chanod, J., Grefenstette, G., and Schiller, A.
(1996). Regular expressions for language engineering. Natu-
ralLanguage Engineering ,2(4), 305–238.
Karttunen ,L.(1983). KIMMO:Ageneral morphological pro-

Berlin.
cessor. In Texas Linguistics Forum22 , pp.165–186.
Karttunen , L. (1993). Finite-state constraints. In Goldsmith,
J.(Ed.),TheLastPhonological Rule ,pp.173–194.University
of Chicago Press.
Karttunen , L. (1998). The proper treatment of optimality in
computational phonology. In Proceedings of FSMNLP’98:
International Workshop on Finite-State Methods in Natural
LanguageProcessing ,BilkentUniversity.Ankara,Turkey,pp.
1–12.
Karttunen ,L.(1999). CommentsonJoshi. InKornai,A.(Ed.),
Extended Finite State Models of Language , pp. 16–18. Cam-

1–12.
Karttunen , L. and Kay, M. (1985). Structure sharing with bi-
nary trees. In ACL-85, Chicago, pp. 133–136. ACL.
Kasami, T. (1965). An efﬁcient recognition and syntax analy-
sis algorithm for context-free languages. Tech. rep. AFCRL -
65-758, Air Force Cambridge Research Laboratory, Bedford,
MA.

DRAFT946 Bibliography
Kashyap ,R.L.and Oommen,B.J.(1983). Spelling correction
using probabilistic methods. Pattern Recognition Letters ,2,
147–154.
Kasper, R. T. and Rounds, W. C. (1986). A logical semantics
for feature structures. In ACL-86, New York, pp.257–266.
Kataja, L. and Koskenniemi, K. (1988). Finite state descrip-
tion of Semitic morphology. In COLING-88 , Budapest, pp.
313–315.
Katz, J.J. and Fodor, J. A.(1963). Thestructure of a semantic
theory.Language ,39, 170–210.
Katz, S. M. (1987). Estimation of probabilities from sparse
data for the language model component of a speech recog-
niser.IEEE Transactions on Acoustics, Speech, and Signal
Processing ,35(3), 400–401.
Kawahara , T., Hasegawa, M., Shitaoka, K., Kitade, T., and
Nanjo,H.(2004).Automaticindexingoflecturepresentati ons
using unsupervised learning of presumed discourse markers .
Speech and Audio Processing, IEEE Transactions on ,12(4),
409–419.
Kawamoto , A. H. (1988). Distributed representations of am-

409–419.
biguous wordsand their resolution inconnectionist networ ks.
In Small, S. L., Cottrell, G. W., and Tanenhaus, M. (Eds.),
Lexical Ambiguity Resolution , pp. 195–228. Morgan Kauf-
man.
Kay, M. (1967). Experiments with a powerful parser. In Proc.
2eme Conference Internationale sur le Traitement Automa-
tique des Langues , Grenoble.
Kay, M. (1973). The MIND system. In Rustin, R. (Ed.), Nat-
ural Language Processing , pp. 155–188. Algorithmics Press,
New York.
Kay, M. (1979). Functional grammar. In BLS-79, Berkeley,

man.
New York.
CA,pp. 142–158.
Kay, M. (1982). Algorithm schemata and data structures in
syntactic processing. InAll´ en, S.(Ed.), TextProcessing: Text
Analysis and Generation, Text Typology and Attribution , pp.
327–358. Almqvist and Wiksell, Stockholm.
Kay,M.(1984). Functional uniﬁcation grammar: Aformalism
for machine translation. In COLING-84 , Stanford, CA, pp.
75–78.
Kay, M. (1985). Parsing in functional uniﬁcation grammar. In
Dowty, D. R., Karttunen, L., and Zwicky, A. (Eds.), Natural
LanguageParsing ,pp.251–278.CambridgeUniversity Press.

75–78.
Kay, M. (1987). Nonconcatenative ﬁnite-state morphology. In
EACL-87,Copenhagen, Denmark, pp. 2–10.
Kay, M. and R¨ oscheisen, M. (1988). Text-translation align-
ment. Tech. rep. P90-00143, Xerox Palo Alto Research Cen-
ter, Palo Alto, CA.
Kay, M. and R¨ oscheisen, M. (1993). Text-translation align-
ment.Computational Linguistics ,19,121–142.
Kay, P. and Fillmore, C. J. (1999). Grammatical constructions
and linguistic generalizations: The What’s X Doing Y? con-
struction. Language ,75(1), 1–33.
Kazakov, D.(1997). Unsupervised learning of na¨ ıve morphol-

ogy with genetic algorithms. In ECML/Mlnet Workshop on
Empirical Learning of Natural Language Processing Tasks ,
Prague, pp. 105–111.Keating, P.A.,Byrd,D.,Flemming,E.,andTodaka,Y.(1994).
Phonetic analysis of word and segment variation using the
TIMITcorpus of American English. Speech Communication ,
14,131–142.
Kehler, A. (1993). The effect of establishing coherence in el-
lipsis and anaphora resolution. In ACL-93, Columbus, Ohio,
pp.62–69.
Kehler, A. (1994a). Common topics and coherent situations:

pp.62–69.
Interpreting ellipsis in the context of discourse inferenc e. In
ACL-94,Las Cruces, New Mexico, pp. 50–57.
Kehler,A.(1994b).Temporalrelations: Referenceordiscourse
coherence?. In ACL-94, Las Cruces, New Mexico, pp. 319–
321.
Kehler, A. (1997a). Current theories of centering for pronoun
interpretation: A critical evaluation. Computational Linguis-
tics,23(3), 467–475.
Kehler, A. (1997b). Probabilistic coreference in information
extraction. In EMNLP1997 , Providence, RI, pp.163–173.

321.
Kehler, A. (2000). Coherence, Reference, and the Theory of
Grammar . CSLI Publications.
Kehler, A., Appelt, D. E., Taylor, L., and Simma, A. (2004).
The (non)utility of predicate-argument frequencies for pr o-
noun interpretation. In HLT-NAACL-04 .
Keller, F. (2004). The entropy rate principle as a predictor of
processing effort: Anevaluation against eye-tracking dat a. In
EMNLP2004 , Barcelona, pp.317–324.
Keller, F. and Lapata, M. (2003). Using the web to obtain fre-
quencies for unseen bigrams. Computational Linguistics ,29,

459–484.
Kelley, L.G.(1979). TheTrueInterpreter: AHistoryofTrans-
lation Theory and Practice in the West . St. Martin’s Press,
New York.
Kelly, E. F. and Stone, P. J. (1975). Computer Recognition of
English Word Senses . North-Holland, Amsterdam.
Kennedy , C. and Boguraev, B. (1996). Anaphora for every-
one: Pronominal anaphora resolution without a parser. In
COLING-96 , Copenhagen, pp. 113–118.
Kernighan , M. D., Church, K. W., and Gale, W. A. (1990). A
spelling correction program base on a noisy channel model.

New York.
InCOLING-90 , Helsinki, Vol. II,pp. 205–211.
Kibble, R. and Power, R. (2000). An integrated framework for
text planning and pronominalisation. In INLG 2000 , pp. 77–
84.
Kiefer, B., Krieger, H.-U., Carroll, J., and Malouf, R. (1999a).
Abagof usefultechniques forefﬁcient and robustparsing. I n
ACL-99,pp. 535–541.
Kiefer, B., Krieger, H.-U.,Carroll, J., and Malouf, R. (1999b).
ACL-99,College Park, MD,pp.473–480.

84.
Kilgarriff , A. (2001). English lexical sample task descrip-
tion. In Proceedings of Senseval-2: Second International
Workshop on Evaluating Word Sense Disambiguation Sys-
tems, Toulouse, France, pp.17–20.
Kilgarriff , A. and Palmer, M. (Eds.). (2000). Computing
and the Humanities: Special Issue on SENSEVAL , Vol. 34.
Kluwer.
Kilgarriff , A. and Rosenzweig, J. (2000). Framework and re-
sults for English SENSEVAL. Computers and the Humani-
ties,34(1-2).

DRAFTBibliography 947
Kim, D.,Gales, M., Hain, T.,and Woodland, P.C. (2004). Us-
ing vtln for broadcast news transcription. In ICSLP-04 , Jeju,
South Korea.
Kim, J. and Woodland, P. C. (2001). The use of prosody in
a combined system for punctuation generation and speech
recognition. In EUROSPEECH-01 ,pp.2757–2760.
King, J. and Just, M. A. (1991). Individual differences in syn-
tactic processing: The role of working memory. Journal of
Memoryand Language ,30, 580–602.
King,P.(1989). ALogical Formalism forHead-Driven Phrase
Structure Grammar . Ph.D.thesis, University of Manchester.
Kinoshita , S., Cohen, K. B., Ogren, P. V., and Hunter, L.
(2005). BioCreAtIvE Task1A: entity identiﬁcation with a
stochastic tagger.. BMCBioinformatics ,6(1).
Kintsch, W. (1974). The Representation of Meaning in Mem-
ory. Wiley, New York.
Kipper, K., Dang, H. T., and Palmer, M. (2000). Class-based
construction of a verb lexicon. In AAAI-00,Austin, TX.
Kiraz, G. A. (1997). Compiling regular formalisms with rule

features into ﬁnite-state automata. In ACL/EACL-97 , Madrid,
Spain, pp. 329–336.
Kiraz, G. A. (2000). Multitiered nonlinear morphology using
multitape ﬁnite automata: A case study on syriac and arabic.
Computational Linguistics ,26(1), 77–105.
Kiraz, G. A. (2001). Computational Nonlinear Morphology
with Emphasis on Semitic Languages . Cambridge University
Press.
Kiraz, G. A. and M¨ obius, B. (1998). Multilingual syllabiﬁca-
tionusingweighted ﬁnite-state transducers. In Proceedingsof
3rd ESCA Workshop on Speech Synthesis , Jenolan Caves, pp.

Press.
59–64.
Kirchhoff ,K.,Fink,G.A.,andSagerer, G.(2002). Combining
acousticandarticulatory featureinformationforrobusts peech
recognition. Speech Communication ,37,303–319.
Kisseberth , C. W. (1969). On the abstractness of phonology:
Theevidence fromYawelmani. PapersinLinguistics ,1,248–
282.
Kisseberth , C. W. (1970). On the functional unity of phono-
logical rules. Linguistic Inquiry ,1(3),291–306.
Kita, K., Fukui, Y., Nagata, M., and Morimoto, T. (1996).
Automatic acquisition of probabilistic dialogue models. I n

282.
ICSLP-96 ,Philadelphia, PA,Vol. 1,pp. 196–199.
Klatt,D.H.(1975).Voiceonsettime,friction,andaspirationin
word-initial consonant clusters. Journal of Speech and Hear-
ing Research ,18, 686–706.
Klatt, D. H. (1982). The Klattalk text-to-speech conversion
system. In IEEEICASSP-82 , pp.1589–1592.
Klatt,D.H.(1977).ReviewoftheARPAspeechunderstanding
project.Journal of the Acoustical Society of America ,62(6),
1345–1366.
Klatt, D. H. (1979). Synthesis by rule of segmental durations

in English sentences. In Lindblom, B. E. F. and ¨Ohman,
S. (Eds.), Frontiers of Speech Communication Research , pp.
287–299. Academic.
Klavans, J. L. (Ed.). (1995). Representation and Acquisition
of Lexical Knowledge: Polysemy, Ambiguity and Generativ-
ity. AAAI Press, Menlo Park, CA. AAAI Technical Report
SS-95-01.Kleene, S. C. (1951). Representation of events in nerve nets
and ﬁnite automata. Tech. rep. RM-704, RAND Corporation.
RAND Research Memorandum.
Kleene,S.C.(1956).Representation ofeventsinnervenetsand

ﬁnite automata. In Shannon, C. and McCarthy, J. (Eds.), Au-
tomata Studies , pp. 3–41. Princeton University Press, Prince-
ton, NJ.
Klein, D. (2005). The unsupervised learning of Natural Lan-
guage Structure . Ph.D.thesis, Stanford University.
Klein,D.andManning,C.D.(2001). Parsingandhypergraphs.
InThe Seventh Internation Workshop on Parsing Technolo-
gies.
Klein,D.andManning,C.D.(2002). Agenerative constituent-
context model for improved grammar induction. In ACL-02.
Klein, D. and Manning, C. D. (2003a). A* parsing: Fast exact

ton, NJ.
gies.
Viterbi parseselection. In HLT-NAACL-03 .
Klein, D. and Manning, C. D.(2003b). Accurate unlexicalized
parsing. In HLT-NAACL-03 .
Klein, D. and Manning, C. D. (2003c). Fast exact inference
withafactored modelfornaturallanguageparsing. InBecke r,
S., Thrun, S., and Obermayer, K. (Eds.), Advances in Neural
Information Processing Systems 15 . MITPress.
Klein, D. and Manning, C. D. (2004). Corpus-based induc-
tion of syntactic structure: Models of dependency and con-
stituency. In ACL-04.
Klein, S. and Simmons, R. F. (1963). A computational ap-

proach to grammatical coding of English words. Journal of
the Association for Computing Machinery ,10(3), 334–347.
Klimt, B. and Yang, Y. (2004). The Enron corpus: A new
dataset for email classiﬁcation research. In Proceedings of
theEuropeanConferenceonMachineLearning ,pp.217–226.
Springer.
Klovstad , J. W. and Mondshein, L. F. (1975). The CASPERS
linguistic analysis system. IEEE Transactions on Acoustics,
Speech, and Signal Processing ,ASSP-23(1), 118–123.

Springer.
Kneser, R. (1996). Statistical language modeling using a vari-
able context length. In ICSLP-96 , Philadelphia, PA, Vol. 1,
pp.494–497.
Kneser, R. and Ney, H. (1993). Improved clustering tech-
niques for class-based statistical language modelling. In
EUROSPEECH-93 ,pp. 973–976.
Kneser, R. and Ney, H. (1995). Improved backing-off for m-
gram language modeling. In IEEE ICASSP-95 , Vol. 1, pp.
181–184.
Knight, K. (1989). Uniﬁcation: A multidisciplinary survey.
ACM Computing Surveys ,21(1), 93–124.
Knight,K.(1999a).Decodingcomplexityinword-replacement

181–184.
translation models. Computational Linguistics ,25(4), 607–
615.
Knight, K. (1999b). A statistical MT tutorial workbook.
Manuscript prepared for the 1999 JHUSummer Workshop.
Knight, K.and Marcu, D.(2000). Statistics-based summariza-
tion - step one: Sentence compression. In AAAI-00, pp. 703–
710.
Knill, K. and Young, S. J. (1997). Hidden Markov Mod-
els in speech and language processing. In Young, S. J. and
Bloothooft, G. (Eds.), Corpus-based Methods in Language
and Speech Processing , pp. 27–68. Kluwer, Dordrecht.

DRAFT948 Bibliography
Knott, A. and Dale, R. (1994). Using linguistic phenomena to
motivate a set of coherence relations. Discourse Processes ,
18(1), 35–62.
Knuth, D. E. (1968). Semantics of context-free languages.
Mathematical Systems Theory ,2(2), 127–145.
Knuth, D. E. (1973). Sorting and Searching: The Artof Com-
puter Programming Volume 3 . Addison-Wesley, Reading,
MA.
Koehn, P. and Hoang, H. (2007). Factored translation models.
InEMNLP/CoNLL 2007 .
Koehn, P. (2003a). Noun Phrase Translation . Ph.D. thesis,
University of Southern California.
Koehn,P.(2003b). Pharaoh: abeamsearchdecoderforphrase-
basedstatisticalmachinetranslation models: Usermanual and
description..
Koehn, P.(2004). Pharaoh: abeam search decoder for phrase-
based statistical machine translation models. In Proceedings
ofAMTA2004 .
Koehn,P.,Abney,S.P.,Hirschberg, J.,and Collins, M.(2000).
Improving intonational phrasing with syntactic informati on.
InIEEEICASSP-00 .
Koehn, P.,Hoang, H.,Birch, A.,Callison-Burch, C., Federico,

M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R.,
Dyer, C., Bojar, O., Constantin, A., and Herbst, E. (2006).
Moses: Open Source Toolkit for Statistical Machine Transla -
tion. InACL-07, Prague. Demonstration session.
Koehn, P.,Och, F.J.,and Marcu, D.(2003). Statistical phrase-
based translation. In HLT-NAACL-03 , pp.48–54.
Koenig, W., Dunn, H. K., Y., L., and Lacy (1946). The sound
spectrograph. Journal of the Acoustical Society of America ,
18,19–49.
Koerner, E. F. K. and Asher, R. E. (Eds.). (1995). Concise

18,19–49.
Historyof theLanguage Sciences . Elsevier Science, Oxford.
Kogan, Y., Collier, N., Pakhomov, S., and Krauthammer, M.
(2005). Towards semantic role labeling & IE in the medical
literature. In AMIA 2005 Symposium Proceedings , pp. 410–
414.
Kogure, K.(1990). Strategic lazy incremental copy graph uni-
ﬁcation. In COLING-90 , Helsinki, pp.223–228.
Kompe, R., Kießling, A., Kuhn, T., Mast, M., Niemann, H.,
N¨ oth,E.,Ott,K.,andBatliner, A.(1993). Prosodytakes ov er:
A prosodically guided dialog system. In EUROSPEECH-93 ,
Berlin, Vol. 3,pp. 2003–2006.

414.
Koppel, M., Argamon, S., and Shimoni, A. R. (2002). Auto-
maticallycategorizing writtentextsbyauthorgender. Literary
and Linguistic Computing ,17(4), 401–412.
Kornai, A. (1991). Formal Phonology . Ph.D. thesis, Stanford
University, Stanford, CA.
Koskenniemi , K. (1983). Two-level morphology: A general
computational model of word-form recognition and produc-
tion. Tech. rep. Publication No. 11, Department of General
Linguistics, University of Helsinki.
Koskenniemi , K.and Church, K.W.(1988). Complexity, two-

levelmorphology,andFinnish. In COLING-88 ,Budapest, pp.
335–339.
Kozima, H. (1993). Text segmentation based on similarity be-
tween words. In ACL-93,pp. 286–288.Krieger, H.-U. and Nerbonne, J. (1993). Feature-based inher-
itance networks for computational lexicons. In Briscoe, T. ,
de Paiva, V., and Copestake, A. (Eds.), Inheritance, Defaults,
and the Lexicon , pp. 90–136. Cambridge University Press.
Krieger, H.-U. and Sch¨ afer, U. (1994). TDL — A type de-
scription language for HPSG. Part 1: Overview. Tech. rep.
RR-94-37, DFKI,Saarbr¨ ucken.

335–339.
Krovetz, R. (1993). Viewing morphology as an inference pro-
cess. InSIGIR-93 , pp.191–202. ACM.
Krovetz, R. (1998). More than one sense per discourse. In
Proceedings of theACL-SIGLEXSENSEVALWorkshop .
Krovetz, R. and Croft, W. B. (1992). Lexical ambiguity and
information retrieval. ACM Transactions on Information Sys-
tems,10(2), 115–141.
Kruskal, J. B. (1983). An overview of sequence compari-
son. In Sankoff, D. and Kruskal, J. B. (Eds.), Time Warps,
String Edits, and Macromolecules: The Theory and Practice

of Sequence Comparison , pp. 1–44. Addison-Wesley, Read-
ing, MA.
Kudo, T. and Matsumoto, Y. (2002). Japanese dependency
analysis using cascaded chunking. In CoNLL-02 , pp. 63–69.
Kuhl,P.K.,F.-M.,T.,andLiu,H.-M.(2003). Foreign-language
experience in infancy: Effects of short-term exposure and s o-
cial interaction on phonetic learning. Proceedings of the Na-
tional Academy of Sciences ,100,9096–9101.
Kuhn, R. and De Mori, R. (1990). A cache-based natural lan-
guage model for speech recognition. IEEE Transactions on

ing, MA.
Pattern Analysis and Machine Intelligence ,12(6), 570–583.
Kukich, K. (1992). Techniques for automatically correcting
words in text. ACM Computing Surveys ,24(4), 377–439.
Kullback , S. and Leibler, R. A. (1951). On information and
sufﬁciency. Annals ofMathematical Statistics ,22,79–86.
Kumar, S. and Byrne, W. (2002). Risk based lattice cutting
for segmental minimum Bayes-risk decoding. In ICSLP-02 ,
Denver, CO.
Kumar, S. and Byrne, W. (2005). Local phrase reordering
models for statistical machine translation. In HLT-EMNLP-
05.

05.
Kuno,S.(1965).Thepredictiveanalyzerandapathelimination
technique. Communications of theACM ,8(7), 453–462.
Kuno, S. and Oettinger, A. G. (1963). Multiple-path syntactic
analyzer. In Popplewell, C. M.(Ed.), Information Processing
1962: Proceedings of the IFIP Congress 1962 , Munich, pp.
306–312. North-Holland. Reprinted in Groszet al. (1986).
Kupiec, J. (1992). Robust part-of-speech tagging using a hid-
denMarkov model. Computer Speech andLanguage ,6,225–
242.
Kupiec, J., Pedersen, J., and Chen, F.(1995). A trainable doc-

242.
umentsummarizer. In SIGIR1995 , pp. 68–73.
Kuˇcera,H.(1992).Themathematicsoflanguage. In TheAmer-
ican Heritage Dictionary of the English Language , pp. xxxi–
xxxiii. Houghton Mifﬂin, Boston.
Kuˇcera, H. and Francis, W. N. (1967). Computational analy-
sisofpresent-dayAmericanEnglish . BrownUniversity Press,
Providence, RI.
Labov, W. (1966). The Social Stratiﬁcation of English in New
YorkCity . Center for Applied Linguistics, Washington, D.C.

DRAFTBibliography 949
Labov, W. (1972). The internal evolution of linguistic rules.
In Stockwell, R. P. and Macaulay, R. K. S. (Eds.), Linguistic
ChangeandGenerativeTheory ,pp.101–171.IndianaUniver-
sity Press, Bloomington.
Labov, W. (1975). The quantitative study of linguistic struc-
ture. Pennsylvania Working Papers onLinguistic Change and
Variation v.1no. 3.U.S.Regional Survey, Philadelphia, PA .
Labov, W. (1994). Principles of Linguistic Change: Internal
Factors. Blackwell, Oxford.
Labov, W. and Fanshel, D. (1977). Therapeutic Discourse .
Academic Press.
Ladd,D.R.(1996). Intonational Phonology . CambridgeStud-
ies in Linguistics. Cambridge University Press.
Ladefoged , P. (1993). A Course in Phonetics . Harcourt Brace
Jovanovich. Third Edition.
Ladefoged , P.(1996). Elements ofAcousticPhonetics . Univer-
sity of Chicago, Chicago, IL. Second Edition.
Lafferty, J. D., Sleator, D., and Temperley, D. (1992). Gram-
matical trigrams: A probabilistic model of link grammar. In

Proceedings ofthe1992AAAIFallSymposiumonProbabilis-
tic Approaches to Natural Language .
Lafferty, J. D., McCallum, A., and Pereira, F. C. N. (2001).
Conditional random ﬁelds: Probabilistic models for segmen t-
ingand labeling sequence data. In ICML2001 , Stanford, CA.
Lakoff, G. (1965). On the Nature of Syntactic Irregularity .
Ph.D. thesis, Indiana University. Published as Irregularity in
Syntax. Holt, Rinehart, and Winston, New York,1970.
Lakoff, G. (1972). Linguistics and natural logic. In Davidson,

D. and Harman, G. (Eds.), Semantics for Natural Language ,
pp.545–665. D.Reidel, Dordrecht.
Lakoff, G. (1993). Cognitive phonology. In Goldsmith, J.
(Ed.),The Last Phonological Rule , pp. 117–145. University
ofChicago Press, Chicago.
Lakoff, G. and Johnson, M. (1980). Metaphors We Live By .
University of Chicago Press,Chicago, IL.
Lambek, J. (1958). The mathematics of sentence structure.
American Mathematical Monthly ,65(3), 154–170.
Lambrecht , K. (1994). Information Structure and Sentence

Form. Cambridge University Press.
Landauer , T. K. (Ed.). (1995). The Trouble With Computers:
Usefulness, Usability, and Productivity . MITPress.
Landauer ,T.K.andDumais,S.T.(1997).AsolutiontoPlato’s
problem: TheLatentSemanticAnalysistheoryofacquisitio n,
induction, and representation of knowledge. Psychological
Review,104,211–240.
Landauer , T. K., Laham, D., Rehder, B., and Schreiner, M. E.
(1997). How well can passage meaning be derived without
using word order: A comparison of latent semantic analysis

and humans. In COGSCI-97 , Stanford, CA, pp. 412–417.
Landes, S., Leacock, C., and Tengi, R. I. (1998). Building se-
mantic concordances. In Fellbaum, C. (Ed.), WordNet: An
Electronic Lexical Database , pp. 199–216. MITPress.
Langendoen , D. T. (1975). Finite-state parsing of phrase-
structure languages and thestatus ofreadjustment rules in the
grammar. Linguistic Inquiry ,6(4), 533–554.Langendoen , D. T. and Langsam, Y. (1987). On the design

of ﬁnite transducers for parsing phrase-structure languag es.
In Manaster-Ramer, A. (Ed.), Mathematics of Language , pp.
191–235. John Benjamins, Amsterdam.
Lapata, M. (2003). Probabilistic text structuring: experiments
with sentence ordering. In ACL-03, Sapporo, Japan, pp.545–
552.
Lapata,M.(2006). Automaticevaluation ofinformationorder-
ing.Computational Linguistics ,32(4), 471–484.
Lappin, S. and Leass, H. (1994). An algorithm for pronom-
inal anaphora resolution. Computational Linguistics ,20(4),
535–561.

552.
535–561.
Lari, K. and Young, S. J. (1990). The estimation of stochas-
ticcontext-free grammarsusingtheInside-Outside algori thm.
Computer Speech and Language ,4,35–56.
Lascarides , A. and Asher, N. (1993). Temporal interpretation,
discourse relations, and common sense entailment. Linguis-
tics and Philosophy ,16(5), 437–493.
Lascarides , A. and Copestake, A. (1997). Default representa-
tion in constraint-based frameworks. Computational Linguis-
tics,25(1), 55–106.
Lauer, M. (1995). Corpus statistics meet the noun compound.
InACL-95,Cambridge, MA,pp.47–54.

Lawrence , W. (1953). The synthesis of speech from signals
which have a low information rate.. In Jackson, W. (Ed.),
Communication Theory , pp. 460–469. Butterworth.
LDC(1993).LDC Catalog: CSR-I (WSJ0) Complete . Univer-
sity of Pennsylvania. www.ldc.upenn.edu/Catalog/
LDC93S6A.html .
LDC(1995). COMLEXEnglishPronunciation Dictionary Ver-
sion 0.2 (COMLEX0.2). Linguistic Data Consortium.
LDC(1998). LDC Catalog: Hub4 project . University
of Pennsylvania. www.ldc.upenn.edu/Catalog/
LDC98S71.html orwww.ldc.upenn.edu/Catalog/

Hub4.html .
Leacock, C. and Chodorow, M. S. (1998). Combining lo-
cal context and WordNet similarity for word sense identiﬁ-
cation. InFellbaum, C.(Ed.), Wordnet: AnElectronic Lexical
Database , pp.265–283. MITPress.
Leacock, C., Towell, G., and Voorhees, E.M. (1993). Corpus-
based statistical senseresolution. In Proceedings oftheARPA
Human Language Technology Workshop , pp. 260–265.
Lease, M.,Johnson, M.,and Charniak, E.(2006). Recognizing
disﬂuencies in conversational speech. IEEE Transactions on

Audio,Speech and Language Processing ,14(5), 1566–1573.
Lee,L.andRose,R.C.(1996). Speakernormalisation usingef-
ﬁcientfrequency warping procedures. In ICASSP96 ,pp.353–
356.
Lee, L. (1999). Measures of distributional similarity. In ACL-
99,pp. 25–32.
Lee, L. (2001). On the effectiveness of the skew divergence
for statistical language analysis. In Artiﬁcial Intelligence and
Statistics, pp.65–72.
Lee, S., Bresch, E., Adams, J., Kazemzadeh, A., and
Narayanan, S. S. (2006). A study of emotional speech ar-

356.
ticulation usingafastmagnetic resonance imaging techniq ue.
InICSLP-06 .
Leech, G.,Garside, R.,and Bryant, M.(1994). CLAWS4: The
tagging of the British National Corpus. In COLING-94 , Ky-
oto, pp.622–628.

DRAFT950 Bibliography
Lees, R. (1970). Problems in the grammatical analysis of En-
glish nominal compounds. In Bierwitsch, M. and Heidolph,
K. E. (Eds.), Progress in Linguistics , pp. 174–187. Mouton,
TheHague.
Leggetter , C. J.and Woodland, P.C. (1995). Maximum likeli-
hoodlinearregressionforspeakeradaptation of HMMs.Com-
puter Speech and Language, ,9(2), 171–186.
Lehiste, I.(Ed.).(1967). Readings inAcoustic Phonetics . MIT
Press.
Lehnert, W. G. (1977). A conceptual theory of question an-
swering. In IJCAI-77, pp.158–164. Morgan Kaufmann.
Lehnert,W.G.,Cardie,C.,Fisher,D.,Riloff,E.,andWilliams,
R. (1991). Description of the CIRCUS system as used for
MUC-3. In Sundheim, B. (Ed.), Proceedings of the Third
Message Understanding Conference , pp.223–233.
Lehrer, A. (1974). Semantic Fields and Lexical Structure .
North-Holland, Amsterdam.
Lehrer, A. and Kittay, E. (Eds.). (1992). Frames, Fields and
Contrasts: NewEssaysinSemanticandLexicalOrganization .

Lemon,O.,Georgila,K.,Henderson,J.,andStuttle, M.(2006).
AnISU dialogue system exhibiting reinforcement learning o f
dialogue policies: generic slot-ﬁlling in the TALKin-car s ys-
tem. InEACL-06.
Lerner, A. J. (1978). The Street Where I Live . Da Capo Press,
New York.
Lesk,M.E.(1986). Automaticsensedisambiguation usingma-
chine readable dictionaries: How to tell a pine cone from an
icecreamcone. In ProceedingsoftheFifthInternationalCon-
ference on Systems Documentation , Toronto, CA, pp. 24–26.
ACM.

New York.
ACM.
Levelt, W. J. M. (1970). A scaling approach to the study of
syntactic relations. In d’Arcais, G. B. F. and Levelt, W. J. M .
(Eds.),Advances in psycholinguistics , pp. 109–121. North-
Holland, Amsterdam.
Levelt, W. J. M. (1983). Monitoring and self-repair in speech.
Cognition ,14,41–104.
Levenshtein , V. I. (1966). Binary codes capable of correcting
deletions, insertions, and reversals. Cybernetics and Control
Theory,10(8), 707–710. Original in Doklady Akademii Nauk
SSSR163(4): 845–848 (1965).

ACM.
Levesque , H. J., Cohen, P. R., and Nunes, J. H. T. (1990). On
acting together. In AAAI-90,Boston, MA,pp.94–99.Morgan
Kaufmann.
Levi, J. (1978). The Syntax and Semantics of Complex Nomi-
nals. Academic Press.
Levin, B. (1993). English Verb Classes And Alternations:
A Preliminary Investigation . University of Chicago Press,
Chicago.
Levin,B.andRappaport Hovav,M.(2005). ArgumentRealiza-
tion. Cambridge University Press.
Levin, E., Pieraccini, R., and Eckert, W. (2000). A stochas-

Kaufmann.
Chicago.
tic model of human-machine interaction for learning dialog
strategies. IEEE Transactions on Speech and Audio Process-
ing,8,11–23.
Levin, L., Gates, D., Lavie, A., and Waibel, A. (1998). An
interlingua based on domain actions for machine translatio n
of task-oriented dialogues. In ICSLP-98 , Sydney, pp. 1155–
1158.Levinson , S. C. (1983). Pragmatics . Cambridge University
Press.
Levinson ,S.E.(1995). Structuralmethodsinautomaticspeech
recognition. Proceedings of theIEEE ,73(11), 1625–1650.

Press.
Levitt, S.D.and Dubner, S.J.(2005). Freakonomics . Morrow.
Levow, G.-A. (1998). Characterizing and recognizing spoken
corrections in human-computer dialogue. In COLING-ACL ,
pp.736–742.
Levy, R. (2007). Expectation-based syntactic comprehension.
Cognition . In press.
Lewin, I., Becket, R., Boye, J., Carter, D., Rayner, M., and
Wir´ en, M. (1999). Language processing for spoken dialogue
systems: is shallow parsing enough?. In Accessing Informa-
tioninSpokenAudio: ProceedingsofESCAETRWWorkshop,

Cambridge, 19 &20th April1999 , pp.37–42.
Lewis,H.andPapadimitriou, C.(1988). ElementsoftheTheory
ofComputation . Prentice-Hall. Second edition.
Li, A.,Zheng, F.,Byrne, W.,Fung, P.,Kamm, T.,Yi, L.,Song,
Z.,Ruhi, U.,Venkataramani, V.,andChen, X.(2000). CASS:
A phonetically transcribed corpus of Mandarin spontaneous
speech. In ICSLP-00 , Beijing, China, pp. 485–488.
Li, C. N. and Thompson, S. A. (1981). Mandarin Chinese:
A Functional Reference Grammar . University of California
Press.

Press.
Li, H. and Abe, N. (1998). Generalizing case frames using a
thesaurus andtheMDLprinciple. Computational Linguistics ,
24(2), 217–244.
Li, X. and Roth, D. (2002). Learning question classiﬁers. In
COLING-02 , pp. 556–562.
Li, X. and Roth, D. (2005). Learning question classiﬁers: The
role of semantic information. Journal of Natural Language
Engineering ,11(4).
Liberman ,A.M.,Delattre,P.C.,andCooper,F.S.(1952). The
roleofselected stimulus variables intheperception ofthe un-
voiced stopconsonants. AmericanJournal ofPsychology ,65,

497–516.
Liberman , M.Y. and Church, K.W.(1992). Text analysis and
word pronunciation in text-to-speech synthesis. In Furui, S.
and Sondhi, M. M. (Eds.), Advances in Speech Signal Pro-
cessing, pp. 791–832. Marcel Dekker.
Liberman , M.Y.and Prince, A.(1977). On stress and linguis-
tic rhythm. Linguistic Inquiry ,8,249–336.
Liberman , M. Y. and Sproat, R. (1992). The stress and struc-
tureofmodiﬁednounphrasesinEnglish. InSag,I.A.andSz-
abolcsi, A.(Eds.), Lexical Matters , pp. 131–181. CSLI,Stan-
ford University.

Lidstone, G. J. (1920). Note on the general case of the Bayes-
Laplace formula for inductive or a posteriori probabilitie s.
Transactions ofthe Faculty ofActuaries ,8, 182–192.
Light, M. and Greiff, W. (2002). Statistical models for the in-
duction anduseofselectional preferences. Cognitive Science ,
87,1–13.
Lin, C.-Y. (2004). ROUGE: A package for automatic evalua-
tion of summaries. In ACL 2004 Workshop on Text Summa-
rization Branches Out .
Lin, C.-Y. and Hovy, E. (2000). The automated acquisition of

87,1–13.
topic signatures for text summarization. In COLING-00 , pp.
495–501.

DRAFTBibliography 951
Lin, C.-Y. and Hovy, E. H. (2003). Automatic evaluation of
summaries using N-gram co-occurrence statistics. In HLT-
NAACL-03 ,Edmonton, Canada.
Lin, D. (1995). A dependency-based method for evaluating
broad-coverage parsers. In IJCAI-95, Montreal, pp. 1420–
1425.
Lin, D. (1998a). Automatic retrieval and clustering of similar
words. In COLING/ACL-98 , Montreal, pp. 768–774.
Lin, D. (1998b). An information-theoretic deﬁnition of simi-
larity. InICML 1998 , San Francisco, pp.296–304.
Lin, D. (2007). Dependency-based word similarity
demo. http://www.cs.ualberta.ca/ ˜lindek/
demos.htm .
Lin, D. and Pantel, P. (2001). Discovery of inference rules
forquestion-answering. NaturalLanguageEngineering ,7(4),
343–360.
Lin, D. and Pantel, P. (2002). Concept discovery from text. In
COLING-02 , pp.1–7.
Lin, D., Zhao, S., Qin, L., and Zhou, M. (2003). Identifying
synonymsamongdistributionally similar words. In IJCAI-03,
pp.1492–1493.
Lin,J.andDemner-Fushman,D.(2005).Evaluatingsummaries

343–360.
andanswers: Twosidesofthesamecoin?. In ACL2005Work-
shop on Measures for MT and Summarization .
Lin, J. (2007). An exploration of the principles underlying
redundancy-based factoid question answering. ACM Trans-
actions on Information Systems .
Lindblom , B. E. F. (1990). Explaining phonetic variation: A
sketch of the H&H theory. In Hardcastle, W. J. and Marchal,
A.(Eds.), Speech ProductionandSpeechModelling , pp.403–
439.Kluwer.
Lindsey, R. (1963). Inferential memory as the basis of ma-

chineswhich understand natural language. InFeigenbaum, E .
andFeldman,J.(Eds.), ComputersandThought ,pp.217–233.
McGraw Hill.
Litman, D. J. (1985). Plan Recognition and Discourse Anal-
ysis: An Integrated Approach for Understanding Dialogues .
Ph.D.thesis, University of Rochester, Rochester, NY.
Litman, D. J. and Allen, J. (1987). A plan recognition model
forsubdialogues inconversation. Cognitive Science ,11,163–
200.
Litman, D.J. and Pan, S.(2002). Designing and evaluating an

200.
adaptive spoken dialogue system. User Modeling and User-
Adapted Interaction ,12(2-3), 111–137.
Litman, D. J. and Silliman, S. (2004). Itspoke: An intelligent
tutoring spoken dialogue system. In HLT-NAACL-04 .
Litman, D. J., Swerts, M., and Hirschberg, J. (2000). Predict-
ingautomatic speech recognition performance usingprosod ic
cues. InNAACL2000 .
Litman, D. J., Walker, M. A., and Kearns, M. (1999). Au-
tomatic detection of poor speech recognition at the dialogu e
level. InACL-99, College Park, MA,pp. 309–316.

Liu,Y.,Chawla,N.V.,Harper,M.P.,Shriberg, E.,andStolcke,
A. (2006a). A study in machine learning from imbalanced
data for sentence boundary detection in speech. Computer
Speech &Language ,20(4), 468–494.Liu, Y., Shriberg, E., Stolcke, A., Hillard, D., Ostendorf, M.,
and Harper, M. (2006b). Enriching speech recognition with
automatic detection of sentence boundaries and disﬂuencie s.
IEEETransactions onAudio,Speech, andLanguage Process-
ing,14(5), 1526–1540.
Liu, Y., Shriberg, E., Stolcke, A., Peskin, B., Ang, J., Hillard ,

D.,Ostendorf, M.,Tomalin,M.,Woodland, P.C.,and Harper,
M. P. (2005). Structural metadata research in the ears pro-
gram. InIEEEICASSP-05 .
Liu, Y. (2004). Word fragment identiﬁcation using acoustic-
prosodic features in conversational speech. In HLT-NAACL-
03 student research workshop , pp.37–42.
Liu,Y.,Liu,Q.,andLin,S.(2005). Log-linearmodelsforword
alignment. In ACL-05,pp. 459–466.
Livescu, K., Glass, J., and Bilmes, J. (2003). Hidden feature
modeling forspeech recognition using dynamic bayesian net -
works. In EUROSPEECH-03 .

Livescu,K.(2005). Feature-BasedPronuncaitionModelingfor
Automatic Speech Recognition . Ph.D. thesis, Massachusetts
Institute of Technology.
Livescu, K.andGlass,J.(2004a). Feature-based pronunciation
modeling forspeech recognition. In HLT-NAACL-04 ,Boston,
MA.
Livescu, K.andGlass,J.(2004b). Feature-based pronunciation
modeling with trainable asynchrony probabilities. In ICSLP-
04,Jeju, South Korea.
Lochbaum , K. E. (1998). A collaborative planning model of
intentional structure. Computational Linguistics ,24(4), 525–
572.

MA.
572.
Lochbaum , K.E.,Grosz,B.J.,andSidner, C.L.(1990). Mod-
els of plans to support communication: An initial report. In
AAAI-90,Boston, MA,pp. 485–490. Morgan Kaufmann.
Lochbaum , K. E., Grosz, B. J., and Sidner, C. L. (2000).
Discourse structure and intention recognition. In Dale, R. ,
Somers, H. L., and Moisl, H. (Eds.), Handbook of Natural
Language Processing . Marcel Dekker.
Longacre , R. E. (1983). The Grammar of Discourse . Plenum
Press.
Lowe, J. B., Baker, C. F.,and Fillmore, C. J. (1997). A frame-

572.
Press.
semantic approach to semantic annotation. In Proceedings of
ACLSIGLEXWorkshop on Tagging Textwith Lexical Seman-
tics, Washington, D.C.,pp.18–24.
Lowerre, B.T.(1968). TheHarpySpeech Recognition System .
Lu,Z.,Cohen,B.K.,andHunter,L.(2006). FindingGeneRIFs
via GeneOntology annotations.. In PSB2006 , pp. 52–63.
Lucassen , J. and Mercer, R. L. (1984). An information the-
oretic approach to the automatic determination of phonemic

baseforms. In IEEEICASSP-84 ,Vol. 9,pp. 304–307.
Luce, P. A., Pisoni, D. B., and Goldﬁnger, S. D. (1990). Simi-
larity neighborhoods of spoken words. In Altmann, G. T. M.
(Ed.),Cognitive Models of Speech Processing , pp. 122–147.
MITPress.
Luhn, H. P. (1957). A statistical approach to the mechanized
encoding and searching of literary information. IBM Journal
ofResearch and Development ,1(4), 309–317.
Luhn, H. P. (1958). The automatic creation of literature ab-
stracts.IBM Journal of Research and Development ,2(2),
159–165.

DRAFT952 Bibliography
Luk, R. W. P. and Damper, R. I. (1996). Stochastic phono-
graphic transduction for english. Computer Speech and Lan-
guage,10(2), 133–153.
Lyons, J.(1977). Semantics . Cambridge University Press.
Lyons,R.G.(2004). Understanding DigitalSignalProcessing .
Prentice Hall. 2nd.ed.
Macleod, C.,Grishman,R.,andMeyers, A.(1998). COMLEX
Syntax Reference Manual Version 3.0. Linguistic Data Con-
sortium.
MacWhinney , B. (1977). Starting points. Language ,53, 152–
168.
MacWhinney ,B.(1982). Basicsyntactic processes. InKuczaj,
S.(Ed.),LanguageAcquisition: Volume1,SyntaxandSeman-
tics, pp. 73–136. Lawrence Erlbaum.
MacWhinney , B. and Csaba Pl´ eh (1988). The processing of
restrictive relative clauses in Hungarian. Cognition ,29, 95–
141.
Madhu, S. and Lytel, D. (1965). A ﬁgure of merit technique
forthe resolution ofnon-grammatical ambiguity. Mechanical
Translation ,8(2), 9–13.
Magerman ,D.M.(1995a). Statistical decision-tree modelsfor
parsing. In ACL-95, pp. 276–283.

168.
141.
Magerman ,D.M.(1995b). Statistical decision-tree modelsfor
parsing. In ACL-95.
Magerman , D. M. and Marcus, M. P. (1991). Pearl: A proba-
bilistic chart parser. In EACL-91, Berlin, Germany.
Main,M.G.andBenson,D.B.(1983). Denotational semantics
fornatural language question-answering programs. American
Journal ofComputational Linguistics ,9(1), 11–21.
Makhoul , J., Baron, A., Bulyko, I., Nguyen, L., Ramshaw,
L. A., Stallard, D., Schwartz, R., and Xiang, B. (2005). The

effects of speech recognition and punctuation on informati on
extraction performance. In INTERSPEECH-05 , Lisbon, Por-
tugal, pp. 57–60.
Makkai, A. (1972). Idiom Structure in English . Mouton, The
Hague.
Malouf,R.,Carroll,J.,andCopestake, A.(2000). Efﬁcientfea-
ture structure operations without compilation. Natural Lan-
guage Engineering ,6(1).
Malouf, R. (2002). A comparison of algorithms for maximum
entropy parameter estimation. In CoNLL-2002 , pp.49–55.

Hague.
Mangu,L.and Brill, E.(1997). Automatic rule acquisition for
spelling correction. In ICML 1997 , Nashville, TN, pp. 187–
194.
Mangu, L., Brill, E., and Stolcke, A. (2000). Finding consen-
susin speech recognition: Worderror minimization and othe r
applications of confusion networks. Computer Speech and
Language ,14(4), 373–400.
Mani, I.(2001). Automatic Summarization . John Benjamins.
Mani,I.andBloedorn,E.(1999).Summarizingsimilaritiesand
differences among related documents. Information Retrieval ,
1(1-2), 35–67.

194.
Mani, I., Gates, B., and Bloedorn, E. (1999). Improving sum-
maries by revising them. In ACL-99, pp. 558–565.
Mani, I. and Maybury, M. T. (1999). Advances in Automatic
TextSummarization . MITPress.Mann,W.C.andThompson,S.A.(1987). Rhetorical structure
theory: A theory of text organization. Tech. rep. RS-87-190 ,
Information Sciences Institute.
Manning , C.D.(1998). Rethinking text segmentation models:
Aninformation extraction casestudy. Tech.rep.SULTRY-98 -
07-01, University of Sydney.
Manning , C. D., Raghavan, P., and Sch¨ utze, H. (2008). In-

troduction to Information Retrieval . Cambridge University
Press.
Manning ,C.D.andSch¨ utze, H.(1999). Foundations ofStatis-
tical Natural Language Processing . MIT Press.
Marchand , Y. and Damper, R. I. (2000). A multi-strategy
approach to improving pronunciation by analogy. Computa-
tional Linguistics ,26(2), 195–219.
Marcu, D. (1995). Discourse trees are good indicators of im-
portance in text. In Mani, I. and Maybury, M. T. (Eds.), Ad-
vances in Automatic Text Summarization , Cambridge, MA,
pp.123–136. MITPress.

Press.
Marcu, D. (1999). The automatic construction of large-scale
corpora for summarization research. In SIGIR 1999 , Berke-
ley, CA,pp. 137–144.
Marcu,D.(2000a). Therhetoricalparsingofunrestricted texts:
Asurface-based approach. Computational Linguistics ,26(3),
395–448.
Marcu, D. (Ed.). (2000b). The Theory and Practice of Dis-
course Parsing and Summarization . MITPress.
Marcu,D.andEchihabi,A.(2002). Anunsupervised approach
to recognizing discourse relations. In ACL-02,pp. 368–375.
Marcu, D. and Wong, W. (2002). A phrase-based, joint prob-

395–448.
ability model for statistical machine translation. In EMNLP
2002, pp.133–139.
Marcus, M. P. (1990). Summary of session 9: Automatic
acquisition of linguistic structure. In Proceedings DARPA
Speech and Natural Language Workshop , Hidden Valley, PA,
pp.249–250. Morgan Kaufmann.
Marcus, M.P.,Kim,G.,Marcinkiewicz, M.A.,MacIntyre, R.,
Bies, A.,Ferguson, M.,Katz, K.,and Schasberger, B. (1994) .
ThePennTreebank: Annotatingpredicate argumentstructur e.
InARPAHumanLanguageTechnologyWorkshop ,Plainsboro,
NJ,pp.114–119. Morgan Kaufmann.

Marcus,M.P.,Santorini,B.,andMarcinkiewicz, M.A.(1993).
Building a large annotated corpus of English: The Penn tree-
bank.Computational Linguistics ,19(2), 313–330.
Markov, A. A. (1913). Essai d’une recherche statistique sur
le texte du roman “Eugene Onegin” illustrant la liaison des
epreuve en chain (‘Example of a statistical investigation o f
the text of “Eugene Onegin” illustrating the dependence be-
tween samples in chain’). Izvistia Imperatorskoi Akademii
Nauk (Bulletin de l’Acad´ emie Imp´ eriale des Sciences de St .-

P´ etersbourg) ,7, 153–162. English translation by Morris
Halle, 1956.
Markov, A. A. (2006). Classical text in translation: A. A.
Markov, an example ofstatistical investigation of thetext Eu-
gene Onegin concerning the connection of samples in chains.
Science in Context ,19(4), 591–600. Translated by David
Link.
Marshall ,I.(1983). Choiceofgrammaticalword-classwithout
GLobalsyntactic analysis: TaggingwordsintheLOBcorpus.
Computers and the Humanities ,17,139–150.

DRAFTBibliography 953
Marshall , I.(1987). Tagselection using probabilistic methods.
In Garside, R., Leech, G.,and Sampson, G.(Eds.), TheCom-
putationalAnalysisofEnglish ,pp.42–56.Longman,London.
Marslen-Wilson ,W.andWelsh,A.(1978). Processinginterac-
tionsandlexicalaccessduringwordrecognitionincontinu ous
speech.Cognitive Psychology ,10,29–63.
Marslen-Wilson , W. (1973). Linguistic structure and speech
shadowing at very short latencies. Nature,244, 522–523.
Marslen-Wilson , W., Tyler, L. K., Waksler, R., and Older, L.
(1994). Morphology and meaning in the English mental lexi-
con.Psychological Review ,101(1), 3–33.
Martin, J. H. (1986). The acquisition of polysemy. In ICML
1986, Irvine, CA, pp.198–204.
Martin, J.H.(1990). AComputational Model ofMetaphor In-
terpretation . Perspectives inArtiﬁcial Intelligence. Academic
Press.
Martin, J. H. (1996). Computational approaches to ﬁgurative
language. Metaphor and Symbolic Activity ,11(1), 85–100.
Martin, J. H. (2006). A rational analysis of the context ef-

Press.
fect on metaphor processing. In Gries, S. T. and Stefanow-
itsch, A. (Eds.), Corpus-Based Approaches to Metaphor and
Metonymy . Mouton de Gruyter.
Massaro, D. W. (1998). Perceiving Talking Faces: From
Speech Perception to a Behavioral Principle . MITPress.
Massaro, D. W. and Cohen, M. M. (1983). Evaluation and in-
tegration ofvisualandauditory information inspeech perc ep-
tion.JournalofExperimentalPsychology: HumanPerception
and Performance ,9, 753–771.
Masterman ,M.(1957).Thethesaurusinsyntaxandsemantics.

Mechanical Translation ,4(1), 1–2.
Matsuzaki , T., Miyao, Y., and ichi Tsujii, J. (2007). Efﬁcient
hpsgparsingwithsupertagging andcfg-ﬁltering. In IJCAI-07.
Matthews , A. and Chodorow, M. S. (1988). Pronoun resolu-
tionintwo-clause sentences: Effects ofambiguity, antece dent
location, and depth of embedding. Journal of Memory and
Language ,27, 245–260.
Mays, E., Damerau, F. J., and Mercer, R. L. (1991). Context
based spelling correction. Information Processing and Man-
agement,27(5), 517–522.
McCallum , A.,Freitag, D.,andPereira, F.C.N.(2000). Maxi-

mumEntropy Markov Models forInformation Extraction and
Segmentation. In ICML 2000 , pp.591–598.
McCallum ,A.(2005). Informationextraction: Distilling struc-
tured data from unstructured text. ACM Queue , 48–57.
McCallum , A.,Freitag, D.,and Pereira, F.C. N.(2000). Max-
imum entropy Markov models for information extraction and
segmentation. In ICML 2000 , pp. 591–598.
McCarthy ,D.(2006).Relating WordNetsensesforwordsense
disambiguation. In Proceedings ofACLWorkshop onMaking
Senseof Sense .
McCarthy , D. and Carroll, J. (2003). Disambiguating nouns,

verbs,and adjectives using automatically acquired select ional
preferences. Computational Linguistics ,29(4), 639–654.
McCarthy , D., Koeling, R., Weeds, J., and Carroll, J. (2004).
Finding predominant word senses in untagged text. In ACL-
04,pp. 279–286.
McCarthy , J. J. (1981). A prosodic theory of non-
concatenative morphology. Linguistic Inquiry ,12,373–418.McCarthy , J. F. and Lehnert, W. G. (1995). Using deci-
sion trees for coreference resolution. In IJCAI-95, Montreal,
Canada, pp. 1050–1055.

McCawley , J. D. (1968). The role of semantics in a grammar.
InBach,E.W.andHarms,R.T.(Eds.), UniversalsinLinguis-
tic Theory , pp. 124–169. Holt, Rinehart &Winston.
McCawley , J. D. (1978). Where you can shove inﬁxes. In
Bell, A.and Hooper, J.B.(Eds.), Syllables and Segments , pp.
213–221. North-Holland, Amsterdam.
McCawley , J. D. (1993). Everything that Linguists have Al-
ways Wanted to Know about Logic (2nd edition). University
of Chicago Press,Chicago, IL.

McCawley , J.D.(1998). TheSyntactic Phenomena ofEnglish .
University of Chicago Press,Chicago.
McClelland , J. L. and Elman, J. L. (1986). Interactive pro-
cesses in speech perception: The TRACE model. In McClel-
land, J. L., Rumelhart, D. E., and the PDP Research Group
(Eds.),Parallel Distributed Processing Volume2: Psycholog-
ical and Biological Models , pp.58–121. MITPress.
McCulloch , W. S. and Pitts, W. (1943). A logical calculus of
ideas immanent in nervous activity. Bulletin of Mathematical

Biophysics ,5,115–133. Reprinted in Neurocomputing: Foun-
dations of Research, ed. by J. A. Anderson and E Rosenfeld.
MIT Press1988 .
McDermott , E.andHazen, T.(2004). MinimumClassiﬁcation
Error training of landmark models for real-time continuous
speech recognition. In IEEEICASSP-04 .
McDonald , D. B. (1982). Understanding Noun Compounds .
Ph.D. thesis, Carnegie Mellon University, Pittsburgh, PA.
CMU Technical Report CS-82-102.
McDonald , R., Pereira, F. C. N., Ribarov, K., and Hajiˇ c, J.

(2005). Non-projective dependency parsing using spanning
tree algorithms. In HLT-EMNLP-05 .
McDonald , R. (2006). Discriminative sentence compression
with soft syntactic constraints. In EACL-06.
McDonald , R.,Crammer, K.,andPereira, F.C.N.(2005). On-
line large-margin training of dependency parsers. In ACL-05,
Ann Arbor, pp. 91–98.
McGuiness , D. L. and van Harmelen, F. (2004). OWL web
ontology overview. Tech. rep. 20040210, World Wide Web
Consortium.
McGurk,H.andMacdonald,J.(1976).Hearinglipsandseeing

voices.Nature,264, 746–748.
McKeown , K.R.,Passonneau, R.,Elson,D.,Nenkova, A.,and
Hirschberg,J.(2005).Dosummarieshelp? Atask-basedeval -
uationofmulti-documentsummarization. In SIGIR2005 ,Sal-
vador, Brazil.
McLuhan , M. (1964). Understanding media: The extensions
ofman. New American Library, New York.
McNamee , P. and Mayﬁeld, J. (2002). Entity extraction with-
outlanguage-speciﬁc resources.In ProceedingsoftheConfer-
enceonComputational NaturalLanguageLearning (CoNLL-
2002), Taipei, Taiwan.

McRoy,S.(1992). Usingmultipleknowledge sourcesforword
sensediscrimination. ComputationalLinguistics ,18(1),1–30.
McTear, M. F. (2002). Spoken dialogue technology: Enabling
theconversational interface. ACM Computing Surveys ,34(1),
90–169.

DRAFT954 Bibliography
McTear, M.F.(2004). Spoken Dialogue Technology . Springer
Verlag, London.
Mealy, G. H. (1955). A method for synthesizing sequential
circuits.BellSystem Technical Journal ,34(5), 1045–1079.
Megyesi, B. (1999). Improving brill’s pos tagger for an agglu-
tinative language. In EMNLP/VLC-99 , College Park, MA.
Melamed , I. D. (2003). Multitext grammars and synchronous
parsers. In HLT-NAACL-03 .
Mellish, C., Knott, A., Oberlander, J., , and O’Donnell, M.
(1998). Experimentsusingstochasticsearchfortextplann ing.
InINLG 1998 , pp. 98–107.
Mel’ˇcuk, I. A. (1979). Studies in dependency syntax . Karoma
Publishers, Ann Arbor.
Merialdo , B.(1994). Tagging English text with aprobabilistic
model.Computational Linguistics ,20(2), 155–172.
Merlo, P. and Stevenson, S. (2001). Automatic verb classiﬁ-
cation based on statistical distribution of argument struc ture.
Computational Linguistics ,27(3), 373–408.
Merlo, P., Stevenson, S., Tsang, V., and Allaria, G. (2001). A

multilingual paradigm for automatic verb classiﬁcation. I n
ACL-02,pp. 207–214.
Merton, R. K. (1961). Singletons and multiples in scien-
tiﬁc discovery. American Philosophical Society Proceedings ,
105(5), 470–486.
Meurers,W.D.andMinnen,G.(1997). Acomputational treat-
mentoflexicalrulesinHPSGascovariation inlexicalentri es.
Computational Linguistics ,23(4), 543–568.
Mihalcea , R. and Moldovan, D. (2001). Automatic generation
of a coarse grained WordNet. In NAACL Workshop on Word-

Netand Other Lexical Resources .
Mihalcea , R. and Moldovan, D. (1999). An automatic method
forgenerating sensetagged corpora. In Proceedings ofAAAI ,
pp.461–466.
Mikheev, A. (2003). Text segmentation. In Mitkov, R. (Ed.),
Oxford Handbook ofComputational Linguistics . Oxford Uni-
versity Press,Oxford.
Mikheev, A.,Moens,M.,andGrover, C.(1999). Named entity
recognition without gazetteers. In EACL-99,pp. 1–8.
Miller, C. A. (1998). Pronunciation modeling in speech syn-
thesis. Tech. rep. IRCS 98–09, University of Pennsylvania

Institute for Research in Cognitive Science, Philadephia, PA.
Miller, G. A. and Nicely, P. E. (1955). An analysis of percep-
tual confusions among some English consonants. Journal of
theAcoustical Society ofAmerica ,27, 338–352.
Miller, G. A. and Beebe-Center, J. G. (1958). Some psycho-
logicalmethodsforevaluating thequality oftranslations .Me-
chanical Translation ,3, 73–80.
Miller, G.A.and Charles, W.G. (1991). Contextual correlates
of semantics similarity. Language and Cognitive Processes ,
6(1), 1–28.

Miller, G.A.and Chomsky,N.(1963). Finitary models of lan-
guage users. In Luce, R. D., Bush, R. R., and Galanter, E.
(Eds.),Handbook of Mathematical Psychology , Vol. II, pp.
419–491. John Wiley.
Miller, G. A., Leacock, C., Tengi, R. I., and Bunker, R. T.
(1993). Asemanticconcordance. In ProceedingsARPAWork-
shop on Human Language Technology , pp. 303–308.Miller, G. A. and Selfridge, J. A. (1950). Verbal context and
the recall of meaningful material. American Journal of Psy-
chology,63,176–185.

Miller, J.L.(1994). On the internal structure of phonetic cate-
gories: aprogress report. Cognition ,50, 271–275.
Miller, S., Bobrow, R. J., Ingria, R., and Schwartz, R. (1994).
Hiddenunderstanding modelsofnaturallanguage. In ACL-94,
LasCruces, NM,pp. 25–32.
Miller, S., Fox, H., Ramshaw, L. A., and Weischedel, R.
(2000). A novel use of statistical parsing to extract inform a-
tion fromtext. In NAACL2000 , Seattle, WA,pp. 226–233.
Miller, S.,Stallard, D.,Bobrow,R.J.,andSchwartz, R.(1996).

A fully statistical approach to natural language interface s. In
ACL-96,Santa Cruz, CA,pp. 55–61.
Miltsakaki , E., Prasad, R., Joshi, A. K., and Webber, B. L.
(2004a). Annotating discourse connectives and their argu-
ments. In Proceedings of the NAACL/HLT Workshop: Fron-
tiers in Corpus Annotation .
(2004b). ThePenn Discourse Treebank. In LREC-04 .
Mishne, G. and de Rijke, M. (2006). MoodViews: Tools for
blog mood analysis. In Nicolov, N., Salvetti, F., Liberman,

M.Y.,and Martin, J.H.(Eds.), Computational Approaches to
AnalyzingWeblogs: Papersfromthe2006SpringSymposium ,
Stanford, Ca. AAAI.
Mitkov, R. (2002). Anaphora Resolution . Longman.
Mitkov, R. and Boguraev, B. (Eds.). (1997). Proceedings of
the ACL-97 Workshop on Operational Factors in Practical,
Robust Anaphora Resolution for Unrestricted Texts , Madrid,
Spain.
Miyao, Y. and Tsujii, J. (2005). Probabilistic disambiguation
models for wide-coverage hpsg parsing. In ACL-05, pp. 83–
90.
Mohammad , S. and Hirst, G. (2005). Distributional measures

Spain.
90.
as proxies for semantic relatedness. Submitted.
Mohri,M.and Nederhof, M.J.(2001). Regular approximation
of context-free grammars through transformation. In Junqu a,
J.-C. and van Noord, G. (Eds.), Robustness in Language and
Speech Technology , pp.153–163. Kluwer.
Mohri, M. (1996). On some applications of ﬁnite-state au-
tomata theory to natural language processing. Natural Lan-
guage Engineering ,2(1), 61–80.
Mohri, M. (1997). Finite-state transducers in language and
speech processing. Computational Linguistics ,23(2), 269–

312.
Mohri, M. (2000). Minimization algorithms for sequential
transducers. Theoretical Computer Science ,234, 177–201.
Mohri,M.andSproat,R.(1998).Onacommonfallacy incom-
putational linguistics. In Suominen, M., Arppe, A., Airola ,
A., Hein¨ am¨ aki, O., Miestamo, M., M¨ a¨ att¨ a, U., Niemi, J. ,
Pitk¨ anen,K.K.,andSinnem¨ aki,K.(Eds.), AManofMeasure:
Festschrift in Honour ofFred Karlsson on this 60th Birthday ,
pp.432–439. SKY Journal of Linguistics, Volume 19, 2006.
M¨oller,S.(2002). Anewtaxonomyforthequalityoftelephone

servicesbasedonspokendialoguesystems. In InProceedings
of the 3rd SIGdial Workshop on Discourse and Dialogue , pp.
142–153.
M¨oller, S. (2004). Quality of Telephone-Based Spoken Dia-
logue Systems . Springer.

DRAFTBibliography 955
Montague , R. (1973). The proper treatment of quantiﬁcation
in ordinary English. In Thomason, R. (Ed.), Formal Philoso-
phy: Selected PapersofRichardMontague , pp.247–270.Yale
University Press,New Haven, CT.
Monz, C. (2004). Minimal span weighting retrieval for ques-
tion answering. In SIGIR Workshop on Information Retrieval
for Question Answering , pp. 23–30.
Moore, E.F.(1956). Gedanken-experiments onsequential ma-
chines. In Shannon, C. and McCarthy, J. (Eds.), Automata
Studies, pp. 129–153. Princeton University Press, Princeton,
NJ.
Moore, R. C. (1977). Reasoning about knowledge and action.
InIJCAI-77, pp.223–227.
Moore, R. C. (2002). Fast and accurate sentence alignment of
bilingual corpora. In Machine Translation: FromResearch to
Real Users (Proceedings, 5th Conference of the Association
for Machine Translation in the Americas, Tiburon, Califor-
nia), pp.135–244.
Moore, R. C. (2004). On log-likelihood-ratios and the signif-
icance of rare events. In EMNLP 2004 , Barcelona, pp. 333–

NJ.
340.
Moore,R.C.(2005). Adiscriminative framework forbilingual
word alignment. In HLT-EMNLP-05 ,pp. 81–88.
Morgan, A. A., Wellner, B., Colombe, J. B., Arens, R.,
Colosimo, M. E., and Hirschman, L. (2007). Evaluating hu-
man gene and protein mention normalization to unique iden-
tiﬁers. In Paciﬁc Symposium on Biocomputing , pp. 281–291.
Morgan,N.andFosler-Lussier,E.(1989). Combiningmultiple
estimators of speaking rate. In IEEEICASSP-89 .
Morris, J. and Hirst, G. (1991). Lexical cohesion computed

340.
by thesaural relations as an indicator of the structure of te xt.
Computational Linguistics ,17(1), 21–48.
Morris, W. (Ed.). (1985). American Heritage Dictionary (2nd
College Edition edition). Houghton Mifﬂin.
Moscoso del Prado Mart´ ın, F., Bertram, R., H¨ aiki¨ o, T.,
Schreuder, R., and Baayen, R. H. (2004a). Morphological
family size in a morphologically rich language: The case of
ﬁnnish compared to dutch and hebrew. Journal of Experi-
mental Psychology: Learning, Memory, and Cognition ,30,
1271–1278.

Moscoso del Prado Mart´ ın, F., Kostic, A., and Baayen, R. H.
(2004b). Putting thebits together: An information theoret ical
perspective on morphological processing. Cognition ,94(1),
1–18.
Moshier, M. A. (1988). Extensions to Uniﬁcation Grammar
fortheDescription ofProgrammingLanguages . Ph.D.thesis,
University of Michigan, Ann Arbor, MI.
Mosteller , F. and Wallace, D. L. (1964). Inference and Dis-
puted Authorship: TheFederalist . Springer-Verlag. A second

1–18.
edition appeared in 1984 as Applied Bayesian and Classical
Inference.
M¨ uller, K. (2001). Automatic detection of syllable boundaries
combining the advantages of treebank and bracketed corpora
training. In ACL-01,Toulouse, France.
M¨ uller, K. (2002). Probabilistic context-free grammars for
phonology. In Proceedings of ACL SIGPHON , Philadelphia,
PA,pp. 70–80.M¨ uller, K., M¨ obius, B., and Prescher, D. (2000). Inducing
probabilistic syllable classes using multivariate cluste ring. In
ACL-00,pp. 225–232.

Munoz, M., Punyakanok, V., Roth, D., and Zimak, D. (1999).
A learning approach to shallow parsing. In EMNLP/VLC-99 ,
College Park, MD,pp. 168–178.
Munteanu , C. and Penn, G. (2004). Optimizing typed feature
structuregrammarparsingthroughnon-statistical indexi ng. In
ACL-04,Barcelona, Spain, pp. 223–230.
Murata, T. (1989). Petri nets: Properties, analysis, and appli-
cations.Proceedings ofthe IEEE ,77(4), 541–576.
Murveit, H., Butzberger, J. W., Digalakis, V. V., and Wein-

traub, M. (1993). Large-vocabulary dictation using SRI’s d e-
cipher speech recognition system: Progressive-search tec h-
niques. In IEEEICASSP-93 , Vol. 2, pp.319–322. IEEE.
N´adas, A. (1983). A decision theoretic formulation of a train-
ing problem in speech recognition and a comparison of train-
ingbyunconditional versusconditional maximumlikelihoo d.
IEEETransactions onAcoustics,Speech, andSignalProcess -
ing,31(4), 814–817.
N´adas, A. (1984). Estimation of probabilities in the language

model of the IBM speech recognition system. IEEE Trans-
actions on Acoustics, Speech, Signal Processing ,32(4), 859–
861.
Nagata, M. and Morimoto, T. (1994). First steps toward sta-
tistical modeling of dialogue to predict the speech act type of
the next utterance. Speech Communication ,15,193–203.
Nakajima ,S.andHamada,H.(1988). Automaticgeneration of
synthesis units based on context oriented clustering. In IEEE
ICASSP-88 ,pp. 659–662.
Nakatani , C. and Hirschberg, J. (1994). A corpus-based study

861.
of repair cues in spontaneous speech. Journal of the Acousti-
cal Society ofAmerica ,95(3), 1603–1616.
Nakov, P. I. and Hearst, M. A. (2003). Category-based pseu-
dowords. In HLT-NAACL-03 ,Edmonton, Canada.
Nakov, P. I. and Hearst, M. A. (2005). A study of using
search engine page hits as aproxy for n-gram frequencies. In
Proceedings of RANLP-05(Recent Advances in Natural Lan-
guage Processing) , Borovets, Bulgaria.
Narayanan , S.S.andAlwan, A.(Eds.).(2004). TexttoSpeech

Synthesis: New paradigms and advances . Prentice Hall.
Narayanan , S. (1997a). Knowledge-based Action Representa-
tions for Metaphor and Aspect (KARMA) . Ph.D. thesis, Uni-
versity of California, Berkeley.
Narayanan , S. (1997b). Talking the talk islike walking the
walk: A computational model of verbal aspect. In COGSCI-
97,Stanford, CA, pp.548–553.
Narayanan , S., Fillmore, C. J., Baker, C. F., and Petruck,
M. R. L. (1999). Framenet meets the semantic web: A
DAML+OILframe representation. In AAAI-02.

Naur, P., Backus, J. W., Bauer, F. L., Green, J., Katz, C.,
McCarthy, J., Perlis, A. J., Rutishauser, H., Samelson, K.,
Vauquois, B., Wegstein, J. H., van Wijnagaarden, A., and
Woodger,M.(1960). Reportonthealgorithmic language AL-
GOL 60. Communications of the ACM ,3(5), 299–314. Re-
vised in CACM 6:1, 1-17, 1963.
Navigli, R.(2006). Meaningfulclustering ofsenseshelpsboost
word sense disambiguation performance. In COLING/ACL
2006, pp.105–112.

DRAFT956 Bibliography
Nederhof , M.J.(2000). Practical experiments with regular ap-
proximation of context-free languages. Computational Lin-
guistics,26(1), 17–44.
Needleman ,S.B.andWunsch,C.D.(1970).Ageneralmethod
applicable to the search for similarities in the amino-acid se-
quence of two proteins. Journal of Molecular Biology ,48,
443–453.
Nenkova, A. and Vanderwende, L.(2005). The Impact of Fre-
quency on Summarization. Tech. rep. MSR-TR-2005-101,
Microsoft Research, Redmond, WA.
Nenkova, A., Brenier, J. M., Kothari, A., Calhoun, S., Whit-
ton, L., Beaver, D., and Jurafsky, D. (2007). To memorize or
to predict: Prominence labeling in conversational speech. In
NAACL-HLT07 .
Nenkova, A. and McKeown, K. R. (2003). References to
namedentities: acorpusstudy.In HLT-NAACL-03 ,pp.70–72.
Nenkova, A., Passonneau, R., and McKeown, K. R. (2007).
The pyramid method: Incorporating human content selection
variation in summarization evaluation. ACM TSLP ,4(2).

Nespor, M. and Vogel, I. (1986). Prosodic phonology . Foris,
Nesson, R., Shieber, S. M., and Rush, A. (2006). Induction
of probabilistic synchronous tree-insertion grammars for ma-
chine translation. In Proceedings ofthe 7th Conference ofthe
Association for Machine Translation in the Americas (AMTA
2006), Boston, MA.
Neu, H. (1980). Ranking of constraints on /t,d/ deletion in
American English: A statistical analysis. In Labov, W. (Ed. ),
Locating Language in Time and Space , pp. 37–54. Academic

Press.
Newell, A., Langer, S., and Hickey, M. (1998). The rˆ ole of
natural language processing in alternative and augmentati ve
communication. Natural Language Engineering ,4(1), 1–16.
Newman , S. (1944). Yokuts Language of California . Viking
Fund Publications in Anthropology 2, New York.
Ney,H.,Essen,U.,andKneser,R.(1994). Onstructuring prob-
abilisticdependenciesinstochasticlanguagemodelling. Com-
puter Speech and Language ,8,1–38.
Ney, H., Haeb-Umbach, R., Tran, B.-H., and Oerder, M.

Press.
(1992). Improvementsinbeamsearchfor10000-wordcontin-
uousspeech recognition. In IEEEICASSP-92 ,SanFrancisco,
CA,pp. I.9–12.IEEE.
Ney,H.(1991).Dynamicprogrammingparsingforcontext-free
grammars in continuous speech recognition. IEEE Transac-
tions on Signal Processing ,39(2), 336–340.
Ng, H. T. and Lee, H. B. (1996). Integrating multiple knowl-
edge sources to disambiguate word senses: An exemplar-
basedapproach. In ACL-96,SantaCruz,CA,pp.40–47.ACL.
Ng, H. T. and Zelle, J. (1997). Corpus-based approaches to

semantic interpretation in NLP. AIMagazine ,18(4), 45–64.
Ng, S.-K. (2006). Integrating text mining with data mining. In
Ananiadou, S. and McNaught, J. (Eds.), Text mining for biol-
ogyand biomedicine . Artech HousePublishers.
Ng, V. (2004). Learning noun phrase anaphoricity to improve
coreference resolution: Issuesinrepresentation andopti miza-
tion. InACL-04.
Ng, V. (2005). Machine learning for coreference resolution:

Fromlocal classiﬁcation to global ranking. In ACL-05.Ng, V. and Cardie, C. (2002a). Identifying anaphoric and non-
anaphoric noun phrases to improve coreference resolution. In
COLING-02 .
Ng, V. and Cardie, C. (2002b). Improving machine learning
approaches to coreference resolution. In ACL-02.
Nguyen, L. and Schwartz, R. (1999). Single-tree method for
grammar-directed search. In IEEE ICASSP-99 , pp. 613–616.
IEEE.
Nichols, J. (1986). Head-marking and dependent-marking
grammar. Language ,62(1), 56–119.

IEEE.
Nicolov, N. and Glance, N. (Eds.). (2007). Proceedings of the
First International Conference on Weblogs and Social Media
(ICWSM) , Boulder, CO.
Nicolov, N., Salvetti, F., Liberman, M. Y., and Martin, J. H.
(Eds.). (2006). Computational Approaches to Analyzing We-
blogs: Papersfromthe2006SpringSymposium ,Stanford,Ca.
AAAI.
Nida, E. A. (1975). Componential Analysis of Meaning: An
Introduction to Semantic Structures . Mouton, TheHague.
Niekrasz, J. and Purver, M. (2006). A multimodal discourse

AAAI.
ontology for meeting understanding. In Renals, S. and Ben-
gio, S. (Eds.), Machine Learning for Multimodal Interaction:
SecondInternationalWorkshopMLMI2005,RevisedSelected
Papers, No. 3689 in Lecture Notes in Computer Science, pp.
162–173. Springer-Verlag.
Nielsen, J. (1992). The usability engineering life cycle. IEEE
Computer ,25(3), 12–22.
Niesler, T. R., Whittaker, E. W. D., and Woodland, P. C.
(1998). Comparison of part-of-speech and automatically de -
rivedcategory-based language modelsforspeechrecogniti on.

InIEEEICASSP-98 , Vol. 1, pp.177–180.
Niesler, T. R. and Woodland, P. C. (1996). A variable-length
category-based n-gram language model. In IEEEICASSP-96 ,
Atlanta, GA, Vol. I,pp. 164–167. IEEE.
Niesler, T. R. and Woodland, P. C. (1999). Modelling word-
pair relations in a category-based language model. In IEEE
ICASSP-99 ,pp. 795–798. IEEE.
Nilsson,N.J.(1980). PrinciplesofArtiﬁcialIntelligence . Mor-
gan Kaufmann, Los Altos, CA.
Ninomiya , T., Tsuruoka, Y., Miyao, Y., Taura, K., and Tsujii,
J.(2006). Fastandscalable hpsgparsing. Traitement automa-

tique des langues (TAL) ,46(2).
Nirenburg , S., Carbonell, J., Tomita, M., and Goodman, K.
(1992).Machine Translation: AKnowledge-based Approach .
Nirenburg , S., Somers, H. L., and Wilks, Y. (Eds.). (2002).
Readings in Machine Translation . MITPress.
Nissim, M.,Dingare, S.,Carletta, J.,and Steedman, M.(2004).
An annotation scheme for information status in dialogue. In
LREC-04 ,Lisbon.
NIST(1990). TIMIT Acoustic-Phonetic Continuous Speech
Corpus. National Institute of Standards and Technology
Speech Disc 1-1.1. NIST Order No.PB91-505065.

NIST(2005). Speech recognition scoring toolkit (sctk) version
2.1. Available athttp://www.nist.gov/speech/tools/.
NIST(2007a). The ace 2007 (ace07) evaluation plan: Evalu-
ation of the detection and recognition of ace entities, valu es,
temporal expressions, relations, and events. Unpublished .

DRAFTBibliography 957
NIST(2007b). Matched Pairs Sentence-Segment Word Error
(MAPSSWE) Test. http://www.nist.gov/speech/
tests/sigtests/mapsswe.htm .
Nivre, J. (2007). Incremental non-projective dependency pars-
ing. InNAACL-HLT07 .
Nivre, J., Hall, J., and Nilsson, J. (2006). Maltparser: A data-
driven parser-generator fordependency parsing. In LREC-06 ,
pp.2216–2219.
Niyogi, P., Burges, C., and Ramesh, P. (1998). Distinctive
feature detection using support vector machines. In IEEE
ICASSP-98 .
Norman, D.A.(1988). TheDesign ofEveryday Things . Basic
Books.
Norman, D. A. and Rumelhart, D. E. (1975). Explorations in
Cognition . Freeman, San Francisco, CA.
Normandin , Y. (1996). Maximum mutual information estima-
tion of hidden Markov models. In Lee, C.-H., Soong, F. K.,
and Paliwal, K. K. (Eds.), Automatic Speech and Speaker
Recognition , pp. 57–82. Kluwer.
Norvig, P. (1987). A Uniﬁed Theory of Inference for Text Un-
derstanding . Ph.D.thesis, University of California, Berkeley,

Books.
puter Science Division Tech. rep. #87/339.
Norvig, P.(1991). Techniques for automatic memoization with
applications to context-free parsing. Computational Linguis-
tics,17(1), 91–98.
Norvig, P. (2005). The gettysburgh powerpoint presentation.
http://norvig.com/Gettysburg/ .
Norvig, P. (2007). How to write a spelling corrector. http:
//www.norvig.com/spell-correct.html .
Och, F. J. (2003). Minimum error rate training in statistical

machine translation. In ACL-03, pp.160–167.
Och,F.J.andNey,H.(2002). Discriminativetraining andmax-
imum entropy models for statistical machine translation. I n
ACL-02,pp. 295–302.
Och,F.J.andNey,H.(2003). Asystematic comparison ofvar-
ious statistical alignment models. Computational Linguistics ,
29(1), 19–51.
Och,F.J.andNey,H.(2004).Thealignmenttemplateapproach
to statistical machine translation. Computational Linguistics ,
30(4), 417–449.
Och, F. J., Uefﬁng, N., and Ney, H. (2001). An efﬁcient a*

search algorithm for statistical machine translation. In Pro-
ceedings of the ACL Workshop on Data-Driven methods in
Machine Translation , pp. 1–8.
Och, F. J. (1998). Ein beispielsbasierter und statistischer
Ansatz zum maschinellen Lernen von nat¨ urlichsprachliche r
¨Ubersetzung . Ph.D. thesis, Universit¨ at Erlangen-N¨ urnberg,
Germany. Diplomarbeit (diploma thesis).
Odden,D.(2005). Introducing Phonology . CambridgeUniver-
Odell, J. J. (1995). The Use of Context in Large Vocabulary

Speech Recognition . Ph.D. thesis, Queen’s College, Univer-
sity of Cambridge.
Odell, M. K. and Russell, R. C. (1918/1922). U.S. Patents
1261167 (1918), 1435663 (1922). Cited in Knuth (1973).Oden,G.C.andMassaro,D.W.(1978). Integration offeatural
information in speech perception. Psychological Review ,85,
172–191.
Oehrle, R. T., Bach, E., and Wheeler, D. W. (Eds.). (1988).
Categorial Grammars and Natural Language Structures . D.
Reidel, Dordrecht.
Oﬂazer, K.(1993). Two-level description of Turkish morphol-

172–191.
ogy. InProceedings, Sixth ConferenceoftheEuropeanChap-
ter ofthe ACL .
Ogburn, W. F. and Thomas, D. S. (1922). Are inventions in-
evitable? A note on social evolution. Political Science Quar-
terly,37,83–98.
Ogren, P.V., Cohen, K. B.,Acquaah-Mensah, G.K., Eberlein,
J.,andHunter,L.(2004).ThecompositionalstructureofGe ne
Ontology terms. In PacSymp Biocomput , pp. 214–225.
Olive, J.P.(1977). Rule synthesis ofspeech fromdyadic units.
InICASSP77 ,pp. 568–570.

Olive, J.P.and Liberman, M.Y.(1979). Asetof concatenative
units for speech synthesis. Journal of the Acoustical Society
ofAmerica ,65, S130.
Olive, J. P., van Santen, J. P. H., M¨ obius, B., and Shih, C.
(1998). Synthesis. In Sproat, R. (Ed.), Multilingual Text-
To-Speech Synthesis: The Bell Labs Approach , pp. 191–228.
Kluwer, Dordrecht.
Oncina, J., Garc´ ıa, P., and Vidal, E. (1993). Learning sub-
sequential transducers for pattern recognition tasks. IEEE
15,448–458.

Oppenheim , A. V., Schafer, R. W., and Stockham, T. G. J.
(1968). Nonlinear ﬁltering of multiplied and convolved sig -
nals.Proceedings ofthe IEEE ,56(8), 1264–1291.
Oravecz, C. and Dienes, P. (2002). Efﬁcient stochastic part-
of-speech tagging for Hungarian. In LREC-02 , Las Palmas,
Canary Islands, Spain, pp. 710–717.
Ortmanns , S., Ney, H., and Aubert, X. (1997). A word graph
algorithmforlargevocabularycontinuousspeechrecognit ion.
Computer Speech and Language, ,11, 43–72.
Ortony, A. (Ed.).(1993). Metaphor (2nd edition). Cambridge

University Press, Cambridge.
Osgood, C.E.,Suci, G.J.,and Tannenbaum, P.H. (1957). The
Measurement of Meaning . University of Illinois Press, Ur-
bana, IL.
O’Shaughnessy , D. (2000). Speech Communications: Human
and Machine . IEEEPress, New York. 2nd. ed.
Ostendorf , M., Digilakis, V., and Kimball, O. (1996). From
HMMstosegmentmodels: Auniﬁed view ofstochastic mod-
eling for speech recognition. IEEE Transactions on Speech
and Audio ,4(5),360–378.
Ostendorf , M. and Ross, K. (1997). Multi-level recogni-

bana, IL.
tion of intonation labels. In Sagisaka, Y., Campbell, N.,
and Higuchi, N. (Eds.), Computing Prosody: Computational
Models for Processing Spontaneous Speech , chap. 19, pp.
291–308. Springer.
Ostendorf ,M.andVeilleux, N.(1994). Ahierarchical stochas-
ticmodelforautomatic prediction ofprosodicboundary loc a-
tion.Computational Linguistics ,20(1).

DRAFT958 Bibliography
Oviatt,S.L.,Cohen,P.R.,Wang,M.Q.,andGaston,J.(1993).
A simulation-based research strategy for designing comple x
NLsysems. In Proceedings DARPASpeech andNatural Lan-
guage Workshop , Princeton, NJ, pp. 370–375. Morgan Kauf-
mann.
Oviatt, S. L.,MacEachern, M.,and Levow, G.-A. (1998). Pre-
dicting hyperarticulate speech during human-computer err or
resolution. Speech Communication ,24, 87–110.
Packard, D. W. (1973). Computer-assisted morphological
analysis of ancient Greek. In Zampolli, A. and Calzolari, N.
(Eds.),Computational and Mathematical Linguistics: Pro-
ceedings of the International Conference on Computational
Linguistics , Pisa, pp. 343–355. LeoS. Olschki.
Palmer,D.D.(2000).Tokenisationandsentencesegmentation.
InDale,R.,Somers,H.L.,andMoisl,H.(Eds.), Handbookof
Natural Language Processing . Marcel Dekker.
Palmer, M., Dang, H. T., and Fellbaum, C. (2006). Mak-
ing ﬁne-grained and coarse-grained sense distinctions, bo th
manually and automatically. Natural Language Engineering ,

13(2), 137–163.
Palmer, M., Babko-Malaya, O., and Dang, H. T. (2004). Dif-
ferent sense granularities for different applications. In HLT-
NAACL Workshop on Scalable Natural Language Under-
standing, Boston, MA,pp. 49–56.
Palmer, M., Fellbaum, C., Cotton, S., Delfs, L., and Dang,
H. T. (2001a). English tasks: All-words and verb lexical
sample. In Proceedings of SENSEVAL-2: Second Interna-
tional Workshop on Evaluating Word Sense Disambiguation
Systems, Toulouse, France.

H. T. (2001b). English tasks: All-words and verb lexical
sample. In Proceedings of Senseval-2: Second International
tems,Toulouse, France, pp. 21–24.
Palmer, M. and Finin, T. (1990). Workshop on the evaluation
of natural language processing systems. Computational Lin-
guistics,16(3), 175–181.
Palmer, M., Kingsbury, P., and Gildea, D. (2005). The propo-
sition bank: An annotated corpus of semantic roles. Compu-
tational Linguistics ,31(1), 71–106.
Palmer, M., Ng, H. T., and Dang, H. T. (2006). Evaluation

of wsd systems. In Agirre, E. and Edmonds, P. (Eds.), Word
SenseDisambiguation: AlgorithmsandApplications .Kluwer.
Pan, S. and Hirschberg, J. (2000). Modeling local context for
pitchaccentprediction. In ACL-00,HongKong,pp.233–240.
Pan, S. and McKeown, K. R. (1999). Word informativeness
and automatic pitch accent modeling. In EMNLP/VLC-99 .
Pang, B., Knight, K., and Marcu, D. (2003). Syntax-based
alignmentofmultipletranslations: extracting paraphras es and
generating new sentences. In HLT-NAACL-03 ,pp. 102–109.

Pantel,P.andRavichandran, D.(2004).Automatically labeling
semantic classes. In HLT-NAACL-04 ,Boston, MA.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002).
Bleu: a method for automatic evaluation of machine trans-
lation. In ACL-02, Philadelphia, PA.
Parsons, T. (1990). Events in the Semantics of English . MIT
Press.
Partee, B. H. (Ed.). (1976). Montague Grammar . Academic
Press.Partee, B. H., ter Meulen, A., and Wall, R. E. (1990). Mathe-
matical Methods in Linguistics . Kluwer, Dordrecht.
Pasca, M. (2003). Open-Domain Question Answering from

Press.
LargeTextCollections . CSLI.
Passonneau , R. (2006). Measuring agreement on set-valued
items(masi)forsemanticandpragmaticannotation. In LREC-
06.
Passonneau ,R.,Nenkova,A.,McKeown,K.R.,andSigleman,
S. (2005). Applying the pyramid method in DUC 2005. In
In Proceedings of the Document Understanding Conference
(DUC’05) .
Passonneau , R.and Litman, D.J.(1993). Intention-based seg-
mentation: Human reliability and correlation with linguis tic
cues. InACL-93, Columbus, Ohio, pp.148–155.

06.
(DUC’05) .
Pater, J.(2007). Gradual learning and convergence. Linguistic
Inquiry. In press.
Pater, J., Potts, C., and Bhatt, R. (2007). Harmonic grammar
with linear programming. unpublished manuscript.
Patwardhan , S., Banerjee, S., and Pedersen, T. (2003). Using
measures of semantic relatedness for word sensedisambigua -
tion. InProceedings of the Fourth International Conference
onIntelligent TextProcessingandComputational Linguist ics,
pp.241–257. Springer.
Paul, D. B. (1991). Algorithms for an optimal A∗search and

linearizing the search in the stack decoder. In IEEEICASSP-
91,Vol. 1,pp. 693–696. IEEE.
Pearl, J.(1984). Heuristics . Addison-Wesley, Reading, MA.
Pearl,J.(1988). ProbabilisticReasoninginIntelligentSystems:
Networks of Plausible Inference . Morgan Kaufman.
Pedersen ,T.andBruce, R.(1997). Distinguishing wordsenses
in untagged text. In EMNLP 1997 , Providence, RI.
Pedersen , T.,Patwardhan, S.,andMichelizzi, J.(2004). Word-
Net::Similarity – Measuring the relatedness of concepts. I n
HLT-NAACL-04 .

Peirce, C. S. (1955). Abduction and induction. In Buchler, J.
(Ed.),Philosophical Writings of Peirce , pp. 150–156. Dover
Books, New York.
Penn, G. and Munteanu, C. (2003). A tabulation-based pars-
ing method that reduces copying. In Proceedings of the 41st
AnnualMeeting of the ACL ,Sapporo, Japan.
Percival, W. K. (1976). On the historical source of immediate
constituent analysis. In McCawley, J. D. (Ed.), Syntax and
Semantics Volume 7, Notes from the Linguistic Underground ,
pp.229–242. Academic Press.

Pereira, F.C.N.(1985). Astructure-sharing representation for
uniﬁcation-based grammar formalisms. In ACL-85, Chicago,
pp.137–144.
Pereira,F.C.N.,Riley,M.D.,andSproat,R.(1994).Weighted
rational transductions and their applications to human lan -
guage processing. In ARPA Human Language Technology
Workshop , Plainsboro, NJ,pp.262–267. Morgan Kaufmann.
Pereira, F. C. N. and Shieber, S. M. (1984). The seman-
tics of grammar formalisms seen as computer languages. In
COLING-84 , Stanford, CA,pp. 123–129.

Pereira, F. C. N. and Shieber, S. M. (1987). Prolog and
Natural-Language Analysis , Vol. 10 of CSLI Lecture Notes .
Chicago University Press, Chicago.

DRAFTBibliography 959
Pereira,F.C.N.,Tishby,N.,andLee,L.(1993). Distributional
clustering ofEnglish words. In ACL-93,Columbus, Ohio,pp.
183–190.
Pereira, F. C. N. and Warren, D. H. D. (1980). Deﬁnite clause
grammars for language analysis— a survey of the formalism
andacomparison with augmented transition networks. Artiﬁ-
cial Intelligence ,13(3), 231–278.
Pereira,F.C.N.andWright,R.N.(1997). Finite-state approx-
imation of phrase-structure grammars. In Roche, E.and Sch-
abes, Y. (Eds.), Finite-State Language Processing , pp. 149–
174.MIT Press.
Perrault, C. R. and Allen, J. (1980). A plan-based analysis
of indirect speech acts. American Journal of Computational
Linguistics ,6(3-4), 167–182.
Peshkin,L.andPfefer,A.(2003).Bayesianinformationextrac-
tion network. In Proceedings of the Eighteenth International
Joint Conference on Artiﬁcial Intelligence .
Peters, W.,Peters, I.,, and Vossen, P.(1998). Automatic sense
clustering inEuroWordNet. In LREC-98 ,Granada, Spain, pp.
447–454.

447–454.
Peterson, G. E. and Barney, H. L. (1952). Control methods
used in astudy of the vowels. Journal ofthe Acoustical Soci-
ety ofAmerica ,24, 175–184.
Peterson, G. E., Wang, W. S.-Y., and Sivertsen, E. (1958).
Segmentation techniques in speech synthesis. Journal of the
Acoustical Society ofAmerica ,30(8), 739–742.
Peterson, J. L. (1986). A note on undetected typing errors.
Communications of theACM ,29(7), 633–637.
Petrov, S., Barrett, L., Thibaux, R., and Klein, D. (2006).
Learningaccurate, compact,andinterpretable treeannota tion.

InCOLING/ACL 2006 , Sydney, Australia, pp.433–440.
Pevzner, L.and Hearst, M.A.(2002). Acritique and improve-
mentofanevaluation metricfortextsegmentation. Computa-
tional Linguistics ,28(1), 19–36.
Pieraccini ,R.,Levin,E.,andLee,C.-H.(1991).Stochasticrep-
resentation of conceptual structure in the ATIS task. In Pro-
Paciﬁc Grove, CA, pp.121–124. Morgan Kaufmann.
Pierce, J. R., Carroll, J. B., and et al (1966). Language and
Machines: Computers in Translation and Linguistics . AL-

PAC report. National Academy of Sciences, National Re-
search Council, Washington, DC.
Pierre, I. (1984). Another look at nominal compounds. In
COLING-84 , Stanford, CA,pp. 509–516.
Pierrehumbert , J. B. and Hirschberg, J. (1990). The meaning
of intonational contours in the interpretation of discours e. In
Cohen,P.R.,Morgan,J.,andPollack, M.(Eds.), Intentions in
Communication , pp. 271–311. MIT Press.
Pierrehumbert , J.B.(1980). ThePhonology and Phonetics of
English Intonation . Ph.D.thesis, MIT.

Pito, R.(1993). Tgrepdoc man page..
Pitrelli, J.F.,Beckman, M.E.,andHirschberg, J.(1994). Eval-
uationofprosodictranscription labelingreliability int heToBI
framework. In ICSLP-94 , Vol. 1, pp.123–126.
Plotkin, J. B. and Nowak, M. A. (2000). Language evolu-
tion and information theory. Journal of Theoretical Biology ,
205(1), 147–159.Poesio, M., Stevenson, R., Di Eugenio, B., and Hitzeman, J.
(2004). Centering: A parametric theory and its instantiati ons.
Computational Linguistics ,30(3), 309–363.

Poesio, M.and Vieira, R.(1998). A corpus-based investigation
of deﬁnite description use. Computational Linguistics ,24(2),
183–216.
Polanyi, L., Culy, C., van den Berg, M., Thione, G. L., and
Ahn, D. (2004a). A Rule Based Approach to Discourse Pars-
ing. InProceedings ofSIGDIAL .
Ahn,D. (2004b). Sentential Structure and Discourse Parsin g.
InDiscourse Annotation Workshop, ACL04 .
Polanyi, L. (1988). A formal model of the structure of dis-

183–216.
course.Journal ofPragmatics ,12.
Polifroni, J., Hirschman, L.,Seneff, S.,and Zue, V. W. (1992).
Experiments in evaluating interactive spoken language sys -
tems. InProceedings DARPA Speech and Natural Language
Workshop , Harriman, NY,pp. 28–33. Morgan Kaufmann.
Pollard, C. (1984). Generalized phrase structure grammars,
headgrammars,andnaturallanguage . Ph.D.thesis, Stanford
Pollard, C. and Moshier, M. A. (1990). Unifying partial de-
scriptions of sets. In Hanson, P. P. (Ed.), Information, Lan-

guage, and Cognition , pp. 285–322. University of British
Columbia Press, Vancouver.
Pollard, C. and Sag, I. A. (1987). Information-Based Syn-
tax and Semantics: Volume 1: Fundamentals . University of
Chicago Press,Chicago.
Pollard, C. and Sag, I. A. (1994). Head-Driven Phrase Struc-
tureGrammar . University of Chicago Press,Chicago.
Pollock, J. J. and Zamora, A. (1975). Automatic abstracting
research at Chemical Abstracts Service. Journal of Chemical
Information and Computer Sciences ,15(4), 226–232.

Porter, M. F. (1980). An algorithm for sufﬁx stripping. Pro-
gram,14(3), 130–127.
Power,R.(1979). Theorganization ofpurposefuldialogs. Lin-
guistics,17, 105–152.
Power,R.,Scott,D.,andBouayad-Agha, N.(2003). Document
structure. Computational Linguistics ,29(2), 211–260.
Pradhan, S., Hacioglu, K., Ward, W., Martin, J. H., and Ju-
rafsky, D. (2003). Semantic role parsing: Adding semantic
structure to unstructured text. In Proceedings of the Interna-
tional Conference on Data Mining (ICDM-2003) .

Pradhan, S., Ward, W.,Hacioglu, K., Martin, J. H., and Juraf-
sky,D. (2005). Semantic role labeling using different synt ac-
tic views. In ACL-05,Ann Arbor, MI.
Price, P.J.,Fisher, W.,Bernstein, J.,and Pallet, D.(1988). The
DARPA 1000-word resource management database for con-
tinuous speech recognition. In IEEE ICASSP-88 , New York,
Vol. 1,pp. 651–654.
Price, P. J., Ostendorf, M., Shattuck-Hufnagel, S., and Fong,
C. (1991). The use of prosody in syntactic disambiguation.

Journal of theAcoustical Society ofAmerica ,90(6).
Prince, A. and Smolensky, P. (1993). Optimality theory: Con-
straint interaction in generative grammar.. Appeared as Te ch.
rep. CU-CS-696-93, Department of Computer Science, Uni-
versity of Colorado at Boulder, and TEch. rep. TR-2, Rut-
gers Center for Cognitive Science, Rutgers University, New
Brunswick, NJ,April 1993.

DRAFT960 Bibliography
Prince, A.and Smolensky,P.(2004). Optimality Theory: Con-
straint interaction in generative grammar . Blackwell.
Prince, A. and Tesar, B. (2004). Learning phonotactic distri-
butions. In Kager, R., Pater, J., and Zonneveld, W. (Eds.),
Constraints in Phonological Acquisition , pp. 245–291. Cam-
Prince, E. (1981). Toward a taxonomy of given-new informa-
tion. InCole,P.(Ed.), RadicalPragmatics ,pp.223–255.Aca-
demicPress.
Prince, E. (1992). The ZPG letter: Subjects, deﬁniteness, and
information-status. In Thompson, S. and Mann, W. (Eds.),
Discourse Description: Diverse Analyses of a Fundraising
Text,pp.295–325.JohnBenjamins, Philadelphia/Amsterdam.
Pr¨ ust,H.(1992). OnDiscourseStructuring, VPAnaphora,and
Gapping. Ph.D.thesis, University of Amsterdam.
Pullum, G. K. and Gazdar, G. (1982). Natural languages and
context-free languages. Linguistics and Philosophy ,4, 471–
504.
Pullum, G. K. (1991). The Great Eskimo Vocabulary Hoax .
University of Chicago, Chicago, IL.

504.
Pullum, G. K. and Ladusaw, W. A. (1996). Phonetic Symbol
Guide. University of Chicago, Chicago, IL. Second Edition.
Pustejovsky , J.(1995). TheGenerative Lexicon . MITPress.
Pustejovsky , J. and Bergler, S. (Eds.). (1992). Lexical Seman-
tics and Knowledge Representation . Lecture Notes in Artiﬁ-
cial Intelligence. Springer Verlag, Berlin.
Pustejovsky , J. and Boguraev, B. (Eds.). (1996). Lexical Se-
mantics: The Problem of Polysemy . Oxford University Press,
Oxford.
Pustejovsky ,J.,Castao, J.,Ingria,R.,Saur,R.,Gaizauskas, R.,

Oxford.
Setzer, A., and Katz, G. (2003a). TimeML: robust speciﬁca-
tionofevent andtemporalexpressions intext. In Proceedings
oftheFifthInternational WorkshoponComputationalSeman -
tics (IWCS-5) .
Pustejovsky , J., Hanks, P., Saur´ ı, R., See, A., Gaizauskas, R.,
Setzer, A., Radev, D., Sundheim, B., Day, D., Ferro, L., and
Lazo, M. (2003b). The TIMEBANK corpus. In Proceed-
ings oftheCorpus Linguistics 2003 conference , pp.647–656.
UCRELTechnical Paper number 16.
Pustejovsky , J., Hanks, P., Saur, R., See, A., Gaizauskas, R.,

Lazo, M. (2003c). The TIMEBANK corpus. In Proceedings
ofCorpus Linguistics 2003 Conference , pp.647–656.
Pustejovsky , J., Ingria, R., Sauri, R., Castano, J., Littman, J.,
Gaizauskas, R.,Setzer, A.,Katz, G.,and Mani, I.(2005). The
Speciﬁcation Language TimeML , chap. 27. Oxford, Oxford,
England.
Qu, Y., Shanahan, J., and Wiebe, J. (Eds.). (2004). Exploring
AttitudeandAffect inText: Papersfromthe2004 SpringSym-
posium, Stanford, Ca. AAAI.
Quillian,M.R.(1968).Semanticmemory.InMinsky,M.(Ed.),

England.
Semantic Information Processing , pp. 227–270. MIT Press.
Quinlan, J. R. (1986). Induction of decision trees. Machine
Learning,1, 81–106.
Quirk,C.,Brockett, C.,andDolan,W.B.(2004). Monolingual
machine translation for paraphrase generation. In EMNLP
2004, pp.142–149.Quirk, C., Menezes, A., and Cherry, C. (2005). Dependency
treelet translation: Syntactically informed phrasal smt. In
ACL-05.
Quirk, R., Greenbaum, S., Leech, G., and Svartvik, J. (1985).
A Comprehensive Grammar of the English Language . Long-
man,London.

ACL-05.
Rabin, M. O. and Scott, D. (1959). Finite automata and their
decision problems. IBM Journal of Research and Develop-
ment,3(2), 114–125.
Rabiner, L. R. (1989). A tutorial on Hidden Markov Models
and selected applications in speech recognition. Proceedings
ofthe IEEE ,77(2), 257–286.
Rabiner, L. R. and Juang, B. H. (1993). Fundamentals of
Speech Recognition . Prentice Hall.
Radev, D., Blair-Goldensohn, S., and Zhang, Z. (2001). Ex-
periments in single and multi-document summarization usin g
MEAD. In DUC-01, New Orleans, LA.

Radev, D., Jing, H., and Budzikowska, M. (2000). Summa-
rization of multiple documents: clustering, sentence extr ac-
tion, and evaluation. In ANLP-NAACL Workshop on Auto-
matic Summarization , Seattle, WA.
Radford, A. (1988). Transformational Grammar: A First
Course. Cambridge University Press.
Radford, A.(1997). Syntactic Theory and the Structure ofEn-
glish: AMinimalist Approach . Cambridge University Press.
Ramshaw , L. A. and Marcus, M. P. (1995). Text chunking
using transformation-based learning. In Proceedings of the

Third AnnualWorkshop on VeryLargeCorpora , pp.82–94.
Rand, D. and Sankoff, D. (1990). Goldvarb: A vari-
able rule application for the macintosh. Available at
http://www.crm.umontreal.ca/ sankoff/GoldVarb Eng.html.
Raphael, B. (1968). SIR: A computer program for semantic
information retrieval. InMinsky,M.(Ed.), SemanticInforma-
tion Processing , pp. 33–145. MIT Press.
Ratnaparkhi , A. (1996). A maximum entropy part-of-speech
tagger. In EMNLP1996 , Philadelphia, PA,pp. 133–142.
Ratnaparkhi ,A.,Reynar,J.C.,andRoukos,S.(1994).AMax-

imum Entropy model for prepositional phrase attachment. In
ARPAHuman Language Technologies Workshop , Plainsboro,
N.J.,pp. 250–255.
Rau, L. F., Jacobs, P. S., and Zernik, U. (1989). Information
extraction and text summarization using linguistic knowle dge
acquisition. Information Processing and Management ,25(4),
419–428.
Ravichandran , D. and Hovy, E. H. (2002). Learning surface
text patterns for a question answering system. In ACL-02,
Philadelphia, PA,pp. 41–47.
Ravishankar , M. K. (1996). Efﬁcient Algorithms for Speech

419–428.
Recognition . Ph.D. thesis, School of Computer Science,
Carnegie Mellon University, Pittsburgh. Available as CMU
CS tech report CMU-CS-96-143.
Rayner, M. and Hockey, B. A. (2003). Transparent combi-
nation of rule-based and data-driven approaches in a speech
understanding architecture. In EACL-03,Budapest, Hungary.
Rayner, M. and Hockey, B. A. (2004). Side effect free dia-
logue management in a voice enabled procedure browser. In
ICSLP-04 , pp.2833–2836.
Rayner, M., Hockey, B. A., and Bouillon, P. (2006). Putting

Linguistics into Speech Recognition . CSLI.

DRAFTBibliography 961
Rebholz-Schuhmann , D., Marcel, S., Albert, S., Tolle, R.,
Casari, G., and Kirsch, H. (2004). Automatic extraction of
mutations frommedline and cross-validation with omim. Nu-
cleic Acids Research ,32(1), 135–142.
Reeves, B. and Nass, C. (1996). The Media Equation: How
PeopleTreatComputers,Television,andNewMediaLikeReal
Peopleand Places . Cambridge University Press.
Regier, T.(1996). TheHuman Semantic Potential . MITPress.
Reichenbach ,H.(1947). ElementsofSymbolicLogic . Macmil-
lan, New York.
Reichert, T.A.,Cohen, D. N.,and Wong, A.K. C.(1973). An
application ofinformation theorytogeneticmutationsand the
matching of polypeptide sequences. Journal of Theoretical
Biology,42,245–261.
Reichman , R.(1985). Getting Computers toTalkLike Youand
Me. MITPress.
Reimer, U.and Hahn, U. (1988). Text condensation as knowl-
edge baseabstraction. In CAIA-88,pp. 14–18.
Reiter, E. and Dale, R. (2000). Building Natural Language
Generation Systems . Cambridge University Press.

Reiter, R. (1980). A logic for default reasoning. Artiﬁcial In-
telligence ,13, 81–132.
Reithinger , N., Engel, R., Kipp, M., and Klesen, M. (1996).
Predicting dialogue acts for a speech-to-speech translati on
system. In ICSLP-96 , Philadelphia, PA,Vol. 2,pp. 654–657.
Resnik, P. (1992). Probabilistic tree-adjoining grammar as a
framework for statistical natural language processing. In Pro-
ceedings of the 14th International Conference on Computa-
tional Linguistics , Nantes, France, pp. 418–424.

Resnik, P. (1995). Using information content to evaluate se-
mantic similarity in a taxanomy. In International Joint Con-
ference for Artiﬁcial Intelligence (IJCAI-95) , pp. 448–453.
Resnik, P. (1996). Selectional constraints: An information-
theoretic model and its computational realization. Cognition ,
61,127–159.
Resnik, P. (1997). Selectional preference and sense disam-
biguation. In Proceedings ofACL SIGLEXWorkshop on Tag-
ging Text with Lexical Semantics , Washington, D.C., pp. 52–
57.

57.
Resnik, P. (1998). WordNet and class-based probabilities.
In Fellbaum, C. (Ed.), WordNet: An Electronic Lexical
Database . MIT Press.
Resnik, P. (2006). Word sense disambiguation in nlp apppli-
cations. In Agirre, E. and Edmonds, P. (Eds.), Word Sense
Disambiguation: Algorithms and Applications . Kluwer.
Reynar, J. C. (1994). An automatic method of ﬁnding topic
boundaries. In ACL-94, pp. 27–30.
Reynar,J.C.(1999). Statistical modelsfortopicsegmentation.
InACL/EACL-97 , pp.357–364.

57.
Reynolds , D. A. and Rose, R. C. (1995). Robust text-
independent speaker identiﬁcation using gaussian mixture
speakermodels. IEEETransactionsonSpeechandAudioPro-
cessing,3(1), 72–83.
Rhodes, F. L. (1929). Beginnings of telephony . Harper and
Brothers, New York.
Rhodes, R. A. (1992). Flapping in American English. In
Dressler, W. U., Prinzhorn, M., and Rennison, J. (Eds.), Pro-
ceedings of the 7th International Phonology Meeting , pp.

217–232. Rosenberg and Sellier, Turin.Riesbeck , C.K.(1975). Conceptual analysis. In Schank, R.C.
(Ed.),ConceptualInformationProcessing ,pp.83–156.Amer-
ican Elsevier, New York.
Riesbeck , C. K. (1986). From conceptual analyzer to direct
memory access parsing: An overview. In Advances in Cogni-
tive Science 1 ,pp. 236–258. Ellis Horwood, Chichester.
Riezler, S., King, T. H., Crouch, R., and Zaenen, A.
(2003). Statistical sentence condensation using ambiguit y
packing and stochastic disambiguation methods for Lexical -

FunctionalGrammar.In HLT-NAACL-03 ,Edmonton,Canada.
Riezler, S., Prescher, D., Kuhn, J., and Johnson, M. (2000).
Lexicalized stochastic modeling of constraint-based gram -
mars using log-linear measures and em training. In ACL-00,
Hong Kong.
Riezler, S.,King,T.H.,Kaplan,R.M.,Crouch,R.,III,J.T.M.,
and Johnson, M. (2002). Parsing the wall street journal us-
ing a lexical-functional grammar and discriminative estim a-
tion techniques. In ACL-02, Philadelphia, PA.
Riggle, J.(2005). Contenders and learning. In WCCFL 23 ,pp.
101–114.
Riley, M. D. (1992). Tree-based modelling for speech synthe-

Hong Kong.
101–114.
sis. In Bailly, G. and Beniot, C. (Eds.), Talking Machines:
Theories, Models and Designs . North Holland, Amsterdam.
Riloff, E. (1993). Automatically constructing a dictionary for
information extraction tasks. In AAAI-93, Washington, D.C.,
pp.811–816.
Riloff, E.(1996). Automatically generating extraction patterns
fromuntagged text. In AAAI-96,pp. 117–124.
Riloff, E.andJones,R.(1999). Learning dictionaries forinfor-
mation extraction by multi-level bootstrapping. In AAAI-99,
pp.474–479.

Rindﬂesch , T. C., Tanabe, L., Weinstein, J. N., and Hunter,
L. (2000). EDGAR: Extraction of drugs, genes and relations
from the biomedical literature. In Paciﬁc Symposium on Bio-
computing , pp. 515–524.
Rivest, R. L. (1987). Learning decision lists. Machine Learn-
ing,2(3), 229–246.
Roark,B.(2001). Probabilistic top-downparsingandlanguage
modeling. Computational Linguistics ,27(2), 249–276.
Roark, B. and Sproat, R. (2007). Computational Approaches
to Morphology and Syntax . Oxford University Press.

Robins, R. H. (1967). A Short History of Linguistics . Indiana
University Press, Bloomington.
Robinson ,J.A.(1965). Amachine-oriented logicbasedonthe
resolutionprinciple. JournaloftheAssociationforComputing
Machinery ,12,23–41.
Robinson , J. J. (1975). Performance grammars. In Reddy,
D. R. (Ed.), Speech Recognition: Invited Paper Presented at
the 1974 IEEESymposium , pp. 401–427. Academic Press.
Robinson , S. E. and Sparck Jones, K. (1976). Relevance
weighting of search terms. Journal of the American Society

for Information Science ,27,129–146.
Rocchio, J. J. (1971). Relevance feedback in information re-
trieval. In The SMART Retrieval System: Experiments in Au-
tomatic Indexing , pp. 324–336. Prentice Hall.
Roche, E. and Schabes, Y. (1997a). Deterministic part-of-
speech tagging with ﬁnite-state transducers. In Roche, E.
andSchabes,Y.(Eds.), Finite-State LanguageProcessing ,pp.
205–239. MIT Press.

DRAFT962 Bibliography
Roche, E.and Schabes, Y. (1997b). Introduction. In Roche, E.
1–65. MITPress.
Rohde, D.L.T.(2005). Tgrep2 user manual..
Rosario, B. and Hearst, M. A. (2004). Classifying semantic
relations inbioscience texts. In Proceedings ofACL2004 ,pp.
430–437.
Rosario, B. and Hearst, M. A. (2005). Multi-way Relation
Classiﬁcation: Application to Protein-Protein Interacti ons. In
HLT-EMNLP-05 .
Rosenberg , A. E.(1971). Effect of Glottal Pulse Shape on the
Quality of Natural Vowels. The Journal of the Acoustical So-
ciety ofAmerica ,49, 583–590.
Rosenfeld , R. (1996). A maximum entropy approach to adap-
tivestatistical languagemodeling. ComputerSpeechandLan-
guage,10,187–228.
Roth,D.andtauYih,W.(2001). Relational learning viapropo-
sitional algorithms: An information extraction case study . In
Proceedings of the International Joint Conference on Artiﬁ -
cial Intelligence (IJCAI) , pp. 1257–1263.

Roth,D.andZelenko,D.(1998).Partofspeechtaggingusinga
network of linear separators. In COLING/ACL-98 , Montreal,
Canada, pp. 1136–1142.
Rounds, W. C. and Kasper, R. T. (1986). A complete logical
calculus for record structures representing linguistic in forma-
tion. InProceedings of the 1st Annual IEEE Symposium on
Logic in Computer Science , pp. 38–43.
Roy, N., Pineau, J.,and Thrun, S. (2000). Spoken dialog man-
agement for robots. In ACL-00,Hong Kong.
Rubenstein ,H.andGoodenough, J.B.(1965). Contextual cor-

relates of synonymy. Communications of the ACM ,8(10),
627–633.
Ruppenhofer , J., Ellsworth, M., Petruck, M. R. L.,
Johnson, C. R., and Scheffczyk, J. (2006). FrameNet
ii: Extended theory and practice. Version 1.3,
http://www.icsi.berkeley.edu/framenet/.
Russell, S. and Norvig, P. (1995). Artiﬁcial Intelligence: A
Modern Approach . Prentice Hall.
Russell, S. and Norvig, P. (2002). Artiﬁcial Intelligence: A
Modern Approach . Prentice Hall. Second edition.
Russell, S. W. (1976). Computer understanding of metaphor-

627–633.
ically used verbs. American Journal of Computational Lin-
guistics,2. Microﬁche 44.
Ryder,M.E.(1994). OrderedChaos: TheInterpretationofEn-
glishNoun-Noun Compounds . University ofCalifornia Press,
Berkeley.
Sacks, H., Schegloff, E. A., and Jefferson, G. (1974). A sim-
plest systematics for the organization of turn-taking for c on-
versation. Language ,50(4), 696–735.
Sadek, D. and De Mori, R. (1998). Dialogue systems. In
De Mori, R. (Ed.), Spoken Dialogues With Computers . Aca-
demicPress, London.
Sadek, M. D. (1991). Dialogue acts are rational plans. In

Berkeley.
ESCA/ETR Workshop on the Structure of Multimodal Dia-
logue, pp.19–48.
Saffran, J. R., Newport, E. L., and Aslin, R. N. (1996a). Sta-
tistical learning by 8-month old infants. Science,274, 1926–
1928.Saffran, J. R.,Newport, E.L.,and Aslin, R.N.(1996b). Word
segmentation: The role of distributional cues. Journal of
Memory and Language ,35, 606–621.
Sag, I. A. and Liberman, M. Y. (1975). The intonational dis-
ambiguation ofindirect speech acts. In CLS-75, pp.487–498.
University of Chicago.

Sag, I. A. and Wasow, T. (Eds.). (1999). Syntactic Theory: A
FormalIntroduction . CSLI Publications, Stanford, CA.
Sag,I.A.,Wasow,T.,andBender,E.M.(Eds.).(2003). Syntac-
tic Theory: A Formal Introduction . CSLI Publications, Stan-
ford, CA.
Sagisaka ,Y.(1988). Speech synthesis byruleusinganoptimal
selectionofnon-uniformsynthesisunits. In IEEEICASSP-88 ,
pp.679–682.
Sagisaka , Y., Kaiki, N., Iwahashi, N., and Mimura, K. (1992).
Atr –ν-talk speech synthesis system. In ICSLP-92 , Banff,
Canada, pp. 483–486.

ford, CA.
Sagisaka , Y., Campbell, N., and Higuchi, N. (Eds.). (1997).
Computing Prosody: Computational Models for Processing
Spontaneous Speech . Springer.
Saint-Dizier , P. and Viegas, E. (Eds.). (1995). Computational
Lexical Semantics . Cambridge University Press.
Sakoe, H. and Chiba, S. (1971). A dynamic programming ap-
proach to continuous speech recognition. In Proceedings of
the Seventh International Congress on Acoustics , Budapest,
Vol. 3,pp. 65–69. Akad´ emiai Kiad´ o.

Sakoe, H. and Chiba, S. (1984). Dynamic programming algo-
rithmoptimization forspoken wordrecognition. IEEETrans-
actions on Acoustics, Speech, and Signal Processing ,ASSP-
26(1), 43–49.
Salasoo, A.and Pisoni, D.B.(1985). Interaction ofknowledge
sourcesinspokenwordidentiﬁcation. JournalofMemoryand
Language ,24, 210–231.
Salomaa, A.(1969). Probabilistic andweighted grammars. In-
formation and Control ,15,529–544.
Salomaa, A.(1973). FormalLanguages . Academic.
Salton, G.(1971). TheSMART Retrieval System: Experiments

in Automatic Document Processing . Prentice Hall.
Salton, G. and Buckley, C. (1990). Improving retrieval per-
formancebyrelevance feedback. Information Processing and
Management ,41, 288–297.
Salton, G. and McGill, M. J. (1983). Introduction to Modern
Information Retrieval . McGraw-Hill, New York,NY.
Sampson , G.(1987). Alternative grammatical coding systems.
In Garside, R.,Leech, G., and Sampson, G. (Eds.), The Com-
putational Analysis of English , pp. 165–183. Longman, Lon-
don and New York.
Sampson , G. (1996). Evolutionary Language Understanding .

Cassell, London.
Samuel, A. G. (1981). Phonemic restoration: Insights from a
newmethodology. JournalofExperimentalPsychology: Gen-
eral,110, 474–494.
Samuel, K., Carberry, S., and Vijay-Shanker, K. (1998). Com-
puting dialogue acts from features with transformation-ba sed
learning. In Chu-Carroll, J. and Green, N. (Eds.), Applying
Machine Learning to Discourse Processing. Papers from the
1998 AAAI Spring Symposium , pp. 90–97. Technical Report
SS-98-01.

DRAFTBibliography 963
Samuelsson , C. (1993). Morphological tagging based entirely
on Bayesian inference. In 9th Nordic Conference on Compu-
tational Linguistics NODALIDA-93 .Stockholm.
Samuelsson ,C.andReichl, W.(1999). Aclass-based language
modelforlarge-vocabulary speech recognition extracted f rom
part-of-speech statistics. In IEEE ICASSP-99 , pp. 537–540.
IEEE.
San-Segundo , R., Montero, J. M., Ferreiros, J., C` ordoba, R.,
andPardo,J.M.(2001). Designing conﬁrmation mechanisms
anderrorrecovery techniques inarailway information syst em
for Spanish. In In Proceedings of the 2nd SIGdial Workshop
on Discourse and Dialogue, Aalborg, Denmark .
Sanders, T.J.M.,Spooren, W.P.M.,and Noordman, L.G.M.
(1992). Towardataxonomyofcoherencerelations. Discourse
Processes ,15, 1–35.
Sanderson , M. (1994). Word sense disambiguation and infor-
mation retrieval. In SIGIR-94 , Dublin, Ireland, pp. 142–151.
ACM.
Sanﬁlippo , A. (1993). LKB encoding of lexical knowledge.

ACM.
In Briscoe, T., de Paiva, V., and Copestake, A. (Eds.), Inher-
itance, Defaults, and the Lexicon , pp. 190–222. Cambridge
Sang, E.F. T.K.(2002). Introduction to the conll-2002 shared
task: Language-independent named entity recognition. In
Proceedings ofCoNLL-2002 , pp. 155–158. Taipei, Taiwan.
Sang,E.F.T.K.andDeMeulder, F.(2003). Introduction tothe
conll-2003 shared task: Language-independent named entit y
recognition. In Daelemans, W. and Osborne, M. (Eds.), Pro-
ceedings ofCoNLL-2003 , pp.142–147. Edmonton, Canada.

Sankoff, D. (1972). Matching sequences under deletion-
insertion constraints. Proceedings of the Natural Academy
ofSciences ofthe U.S.A. ,69, 4–6.
Santorini , B.(1990). Part-of-speech tagging guidelines for the
Penn Treebank project. 3rd revision, 2nd printing.
Saraclar, M., Nock, H., and Khudanpur, S. (2000). Pronunci-
ation modeling by sharing gaussian densities across phonet ic
models.Computer Speech and Language ,14(2), 137–160.
Sassano, M. and Utsuro, T. (2000). Named entity chunking

techniques in supervised learning for japanese named entit y
recognition. In COLING-00 , Saarbrcken, Germany, pp. 705–
711.
Scha, R. and Polanyi, L. (1988). An augmented context free
grammar for discourse. In COLING-88 , Budapest, pp. 573–
577.
Schabes, Y. (1990). Mathematical and Computational Aspects
of Lexicalized Grammars . Ph.D. thesis, University of Penn-
sylvania, Philadelphia, PA.
Schabes,Y.(1992). Stochasticlexicalized tree-adjoining gram-
mars. InProceedings ofthe14th International Conference on

711.
577.
Computational Linguistics , Nantes, France, pp. 426–433.
Schabes, Y., Abeill´ e, A., and Joshi, A. K. (1988). Parsing
strategies with ‘lexicalized’ grammars: Applications to T ree
Adjoining Grammars. In COLING-88 , Budapest, pp. 578–
583.
Schachter , P. (1985). Parts-of-speech systems. In Shopen, T.
(Ed.),Language Typology and Syntactic Description, Volume
1,pp. 3–61. Cambridge University Press.Schank, R. C. (1972). Conceptual dependency: A theory of

583.
natural language processing. Cognitive Psychology ,3, 552–
631.
Schank,R.C.and Abelson, R.P.(1977). Scripts, Plans,Goals
and Understanding . Lawrence Erlbaum.
Schank,R.C.andAlbelson,R.P.(1977). Scripts,Plans,Goals
Schank, R. C. and Riesbeck, C. K. (Eds.). (1981). Inside
Computer Understanding: Five Programs plus Miniatures .
Schegloff , E. A. (1968). Sequencing in conversational open-
ings.American Anthropologist ,70, 1075–1095.
Schegloff , E. A. (1979). Identiﬁcation and recognition in tele-

631.
phone conversation openings. In Psathas, G. (Ed.), Everyday
Language: Studies in Ethnomethodology , pp. 23–78. Irving-
ton.
Schegloff , E.A.(1982). Discourse as an interactional achieve-
ment: Some uses of ‘uh huh’ and other things that come
between sentences. In Tannen, D. (Ed.), Analyzing Dis-
course: Text and Talk , pp. 71–93. Georgetown University
Press,Washington, D.C.
Schone, P. and Jurafsky, D. (2000). Knowlege-free induction
of morphology using latent semantic analysis. In Proceed-

ton.
Learning (CoNLL-2000) .
Schone, P. and Jurafsky, D. (2001). Knowledge-free induction
of inﬂectional morphologies. In NAACL2001 .
Sch¨onkﬁnkel , M. (1924). ¨Uber die Bausteine der mathematis-
chen Logik. Mathematische Annalen ,92, 305–316. English
Translation appears in From Frege to G¨ odel: A Source Book
inMathematicalLogic ,HarvardUniversity Press,Cambridge,
MA,1967.
Schroder , M. (2006). Expressing degree of activation in syn-

MA,1967.
thetic speech. IEEETransactions onAudio,Speech, and Lan-
guage Processing ,14(4), 1128–1136.
Schubert , L. K. and Pelletier, F. J. (1982). From English
to logic: Context-free computation of ‘conventional’ logi cal
translation. American Journal of Computational Linguistics ,
8(1), 27–44.
Schulte im Walde , S. (2000). Clustering verbs semantically
according to their alternation behaviour. In COLING-00 ,
Saarbr¨ ucken, Germany, pp. 747–753.
Sch¨ utze,H.(1992a).Contextspace.InGoldman,R.(Ed.), Pro-

ceedings of the 1992 AAAI Fall Symposium on Probabilistic
Approaches to Natural Language .
Sch¨ utze, H. (1992b). Dimensions of meaning. In Proceedings
ofSupercomputing ’92 ,pp. 787–796. IEEE,IEEEPress.
Sch¨ utze, H. (1995). Distributional part-of-speech tagging. In
EACL-95.
Sch¨ utze, H.(1997). Ambiguity Resolution in Language Learn-
ing: Computational and Cognitive Models . CSLI Publica-
tions, Stanford, CA.
Sch¨ utze, H. (1998). Automatic word sense discrimination.
Computational Linguistics ,24(1), 97–124.

EACL-95.
Sch¨ utze, H. and Pedersen, J. (1995). Information retrieval
based on word senses. In Proceedings of the Fourth Annual
SymposiumonDocumentAnalysisandInformation Retrieval ,
LasVegas, pp.161–175.

DRAFT964 Bibliography
Sch¨ utze, H. and Singer, Y. (1994). Part-of-speech tagging us-
ingavariablememoryMarkovmodel.In ACL-94,LasCruces,
NM,pp.181–187.
Sch¨ utzenberger , M. P. (1977). Sur une variante des fonctions
sequentielles. Theoretical Computer Science ,4,47–57.
Schwartz , A.S.and Hearst, M.A.(2003). Asimple algorithm
for identifying abbreviation deﬁnitions in biomedical tex t. In
PaciﬁcSymposium on Biocomputing , Vol. 8,pp. 451–462.
Schwartz , R. and Austin, S. (1991). A comparison of several
approximate algorithms for ﬁnding multiple ( N-BEST) sen-
tence hypotheses. In icassp91, Toronto, Vol. 1, pp. 701–704.
IEEE.
Schwartz , R. and Chow, Y.-L. (1990). The N-best algorithm:
Anefﬁcient and exact procedure for ﬁnding theNmostlikely
sentence hypotheses. In IEEE ICASSP-90 , Vol. 1, pp. 81–84.
IEEE.
Schwartz ,R.,Chow,Y.-L.,Kimball,O.,Roukos,S.,Krasnwer,
M.,and Makhoul, J.(1985). Context-dependent modeling for
acoustic-phonetic recognition of continuous speech. In IEEE
ICASSP-85 ,Vol. 3,pp. 1205–1208. IEEE.

IEEE.
IEEE.
Scott, M. and Shillcock, R. (2003). Eye movements reveal the
on-line computation of lexical probabilities during readi ng.
Psychological Science ,14(6), 648–652.
Searle, J.R.(1975a). Indirectspeech acts. InCole,P.andMor-
gan, J. L. (Eds.), Speech Acts: Syntax and Semantics Volume
3,pp. 59–82. Academic Press.
Searle,J.R.(1975b).Ataxonomyofillocutionary acts.InGun-
derson,K.(Ed.), Language, Mind andKnowledge, Minnesota
Studies in the Philosophy of Science , Vol. VII, pp. 344–369.
University of Minnesota Press, Amsterdam. Also appears in

John R. Searle, Expression and Meaning: Studies in the The-
oryofSpeech Acts ,Cambridge University Press, 1979.
Searle, J. R. (1980). Minds, brains, and programs. Behavioral
and Brain Sciences ,3,417–457.
Sejnowski ,T.J.andRosenberg,C.R.(1987).Parallelnetworks
that learn to pronounce English text. Complex Systems ,1(1),
145–168.
Sekine, S.and Collins, M.(1997). Theevalb software. http:
//cs.nyu.edu/cs/projects/proteus/evalb .
Selkirk,E.(1986). Onderived domainsinsentence phonology.
Phonology Yearbook ,3,371–405.

145–168.
Seneff, S. (1995). TINA: A natural language system for spo-
ken language application. Computational Linguistics ,18(1),
62–86.
Seneff, S. (2002). Response planning and generation in the
MERCURY ﬂight reservation system. Computer Speech and
Language, Special Issue on Spoken Language Generation ,
16(3-4), 283–312.
Seneff, S., Lau, R., and Meng, H. (1996). ANGIE: A new
frameworkforspeechanalysisbasedonmorpho-phonologica l
modelling. In ICSLP-96 .
Seneff, S. and Polifroni, J. (2000). Dialogue management in
themercuryﬂightreservation system.In ANLP/NAACLWork-

62–86.
shop on Conversational Systems , Seattle.
Seneff, S. and Zue, V. W. (1988). Transcription and alignment
of the TIMIT database. In Proceedings of the Second Sym-
posium on Advanced Man-Machine Interface through Spoken
Language , Oahu, Hawaii.Senellart , J.,Dienes,P.,andV´ aradi,T.(2001). Newgeneration
systran translation system. In MT Summit 8 .
Settles, B. (2005). ABNER: an open source tool for automat-
ically tagging genes,proteins and other entity names in tex t.
Bioinformatics ,21(14), 3191–3192.

Seuss,D.(1960). OneFishTwo FishRed FishBlueFish . Ran-
domHouse, New York.
Shannon , C. E. (1938). A symbolic analysis of relay and
switching circuits. Transactions of the American Institute of
Electrical Engineers ,57, 713–723.
Shannon ,C.E.(1948). Amathematical theory ofcommunica-
tion.Bell System Technical Journal ,27(3), 379–423. Contin-
ued in thefollowing volume.
Shannon , C. E. (1951). Prediction and entropy of printed En-
glish.Bell System Technical Journal ,30,50–64.

Sheil, B. A. (1976). Observations on context free parsing.
SMIL: Statistical Methods in Linguistics ,1,71–109.
Shieber, S. M. (1985a). Evidence against the context-freeness
of natural language. Linguistics and Philosophy ,8,333–343.
Shieber, S.M. (1985b). Using restriction to extend parsing al-
gorithms for complex-feature-based formalisms. In ACL-85,
Chicago, pp. 145–152.
Shieber, S. M. (1986). An Introduction to Uniﬁcation-Based
Approaches to Grammar . Center for the Study of Language

and Information, Stanford University, Stanford, CA.
Shieber, S. M. (1994a). Lessons from a restricted Turing test.
Communications ofthe ACM ,37(6), 70–78.
Shieber,S.M.(1994b). Restricting theweak-generative capac-
ity of synchronous tree-adjoining grammars. Computational
Intelligence ,10(4), 371–385.
Shieber, S. M. and Schabes, Y. (1992). Generation and syn-
chronous tree-adjoining grammars. Computational Intelli-
gence,7(4), 220–228.
Shockey, L.(2003). Sound Patterns of Spoken English . Black-
well, Oxford.

Shoup, J. E. (1980). Phonological aspects of speech recogni-
tion. In Lea, W. A. (Ed.), Trends in Speech Recognition , pp.
125–138. Prentice-Hall.
Shriberg,E.(2002). To‘errrr’ishuman: ecologyandacoustics
of speech disﬂuencies. Journal of the International Phonetic
Association ,31(1), 153–169.
Shriberg, E. (2005). Spontaneous speech: How people really
talk, and why engineers should care. In INTERSPEECH-05 ,
Lisbon,Portugal.
Shriberg, E., Ferrer, L., Kajarekar, S., Venkataraman, A., and
Stolcke, A. (2005). Modeling prosodic feature sequences fo r

speaker recognition. Speech Communication ,46(3-4), 455–
472.
Shriberg, E.,Stolcke, A.,Hakkani-T¨ ur,D.,andT¨ ur,G.(2000).
Prosody-based automatic segmentation of speech into sen-
tences and topics. Speech Communication ,32(1-2), 127–154.
Shriberg, E. (1994). Preliminaries to a Theory of Speech Dis-
ﬂuencies. Ph.D. thesis, University of California, Berkeley,
CA. (unpublished).
Shriberg, E., Bates, R., Taylor, P., Stolcke, A., Jurafsky, D.,
Ries, K., Coccaro, N., Martin, R., Meteer, M., and Van Ess-

472.
Dykema,C.(1998). Canprosodyaidtheautomatic classiﬁca-
tion of dialog acts in conversational speech?. Language and
Speech(SpecialIssueonProsodyandConversation) ,41(3-4),
439–487.

DRAFTBibliography 965
Shriberg,E.,Wade,E.,andPrice,P.J.(1992).Human-machine
problem solving using spoken language systems (SLS): Fac-
tors affecting performance and user satisfaction. In Proceed-
ings DARPASpeech and Natural Language Workshop , Harri-
man,NY, pp.49–54. Morgan Kaufmann.
Siddharthan , A., Nenkova, A., and McKeown, K. R. (2004).
Syntactic simpliﬁcation for improving content selection i n
multi-document summarization. In COLING-04 , p.896.
Sidner, C. L. (1979). Towards a computational theory of deﬁ-
niteanaphora comprehension in Englishdiscourse. Tech. re p.
537,MITArtiﬁcial Intelligence Laboratory, Cambridge, MA .
Sidner,C.L.(1983).Focusinginthecomprehensionofdeﬁnite
anaphora. In Brady, M. and Berwick, R. C. (Eds.), Computa-
tional Models of Discourse , pp. 267–330. MITPress.
Sills, D. L. and Merton, R. K. (Eds.). (1991). Social Science
Quotations . MacMillan, New York.
Silverman , K., Beckman, M. E., Pitrelli, J. F., Ostendorf,
M., Wightman, C. W., Price, P. J., Pierrehumbert, J. B., and

Hirschberg, J. (1992). ToBI: a standard for labelling Engli sh
prosody. In ICSLP-92 , Vol. 2,pp. 867–870.
Simmons , R. F.(1965). Answering English questions by com-
puter: A survey. Communications ofthe ACM ,8(1), 53–70.
Simmons , R. F. (1973). Semantic networks: Their computa-
tion and use for understanding English sentences. In Schank ,
R. C. and Colby, K. M. (Eds.), Computer Models of Thought
andLanguage , pp.61–113.W.H.Freeman andCo.,SanFran-
cisco.
Simmons , R. F. (1978). Rule-based computations on En-

cisco.
glish. InWaterman, D.A.andHayes-Roth, F.(Eds.), Pattern-
Directed Inference Systems . Academic Press.
Simmons , R. F.(1983). Computations from the English . Pren-
tice Hall.
Singh, S. P., Litman, D. J., Kearns, M., and Walker, M. A.
(2002). Optimizing dialogue managementwithreinforcemen t
learning: Experiments with the njfun system. J. Artif. Intell.
Res.(JAIR) ,16, 105–133.
Sleator, D. and Temperley, D. (1993). Parsing English with a
linkgrammar. In Proceeedings,ThirdInternational Workshop
on Parsing Technologies , Tilburg, The Netherlands/Durbuy,
Belgium.

tice Hall.
Belgium.
Slobin, D. I. (1996). Two ways to travel. In Shibatani, M. and
Thompson, S. A. (Eds.), Grammatical Constructions: Their
Formand Meaning , pp. 195–220. Clarendon Press,Oxford.
Small,S.L.,Cottrell, G.W.,andTanenhaus,M.(Eds.).(1988).
Lexical Ambiguity Resolution . Morgan Kaufman.
Small, S. L. and Rieger, C. (1982). Parsing and comprehend-
ing with Word Experts. In Lehnert, W. G. and Ringle, M. H.
(Eds.),Strategies for Natural Language Processing , pp. 89–
147.Lawrence Erlbaum.

Belgium.
Smith, D. A. and Eisner, J. (2007). Bootstrapping feature-rich
dependency parsers with entropic priors. In EMNLP/CoNLL
2007, Prague, pp. 667–677.
Smith,N.A.andEisner,J.(2005).Guidingunsupervisedgram-
mar induction using contrastive estimation. In IJCAI Work-
shop on Grammatical Inference Applications , Edinburgh, pp.
73–82.
Smith, N. A. and Johnson, M. (2007). Weighted and proba-
bilistic context-free grammars are equally expressive. Com-
putational Linguistics . Toappear.Smith, R. W. and Gordon, S. A. (1997). Effects of variable

73–82.
initiative on linguistic behavior in human-computer spoke n
natural language dialogue. Computational Linguistics ,23(1),
141–168.
Smith,V.L.and Clark, H.H.(1993). Onthecourseofanswer-
ing questions. Journal of Memoryand Language ,32, 25–38.
Smolensky , P. and Legendre, G. (2006). The Harmonic Mind .
MITPress.
Smrˇ z,O.(1998). FunctionalArabicMorphology . Ph.D.thesis,
Charles University in Prague.
Smyth, R. (1994). Grammatical determinants of ambiguous
pronounresolution. JournalofPsycholinguistic Research ,23,
197–229.

141–168.
MITPress.
197–229.
Snover, M., Dorr, B., Schwartz, R., Micciulla, L., and
Makhoul, J. (2006). A study of translation edit rate with tar -
geted human annotation. In AMTA-2006 .
Snow,R.,Jurafsky, D.,and Ng,A.Y.(2005). Learning syntac-
ticpatterns forautomatic hypernym discovery. In Saul, L.K .,
Weiss, Y., and Bottou, L. (Eds.), NIPS 17, pp. 1297–1304.
MITPress.
Snow, R., Jurafsky, D.,and Ng, A. Y. (2006). Semantic taxon-
omyinduction fromheterogenous evidence. In COLING/ACL
2006.
Snow, R., Prakash, S., Jurafsky, D., and Ng, A. Y. (2007).

MITPress.
2006.
Learning tomergeword senses. In EMNLP/CoNLL 2007 ,pp.
1005–1014.
Soderland , S., Fisher, D., Aseltine, J., and Lehnert, W. G.
(1995). CRYSTAL: Inducing a conceptual dictionary. In
IJCAI-95, Montreal, pp.1134–1142.
Soon, W. M., Ng, H. T., and Lim, D. C. Y. (2001). A machine
learning approach to coreference resolution of noun phrase s.
Computational Linguistics ,27(4), 521–544.
Soong, F. K. and Huang, E.-F. (1990). A tree-trellis based fast
search for ﬁnding the n-best sentence hypotheses in continu -

ous speech recognition. In Proceedings DARPA Speech and
Natural Language Processing Workshop , Hidden Valley, PA,
pp. 705–708. Also in Proceedings of IEEEICASSP-91, 705-
708.
Soricut, R. and Marcu, D. (2006). Discourse generation using
utility-trained coherence models. In COLING/ACL 2006 , pp.
803–810.
Sparck Jones , K. (1972). A statistical interpretation of term
speciﬁcity and its application in retrieval. Journal of Docu-
mentation ,28(1), 11–21.
Sparck Jones , K. (1986). Synonymy and Semantic Classiﬁca-

708.
803–810.
tion. Edinburgh University Press, Edinburgh. Republication
of 1964 PhD Thesis.
Sparck Jones , K. (2007). Automatic summarising: The state
of the art. Information Processing and Management ,43(6),
1449–1481.
SparckJones ,K.and Galliers, J.R.(Eds.).(1996). Evaluating
Natural Language Processing Systems . Springer.
SparckJones ,K.andWillett, P.(Eds.).(1997). ReadingsinIn-
formation Retrieval . Morgan Kaufmann, San Francisco, CA.
Spiegel, M. F. (2002). Proper name pronunciations for speech
technology applications. In Proceedings of IEEE Workshop

on Speech Synthesis , pp. 175–178.

DRAFT966 Bibliography
Spiegel, M. F. (2003). Proper name pronunciations for speech
technology applications. International Journal of Speech
Technology ,6(4), 419–427.
Sporleder , C. and Lapata, M. (2004). Automatic paragraph
identiﬁcation: A study across languages and domains. In
EMNLP2004 .
Sporleder , C. and Lapata, M. (2006). Automatic Paragraph
Identiﬁcation: AStudyacrossLanguages andDomains. ACM
Transactions on Speech and Language Processing (TSLP) ,
3(2).
Sporleder , C.andLascarides, A.(2005). Exploiting Linguistic
Cues to Classify Rhetorical Relations. In Proceedings of the
Recent Advances in Natural Language Processing (RANLP-
05),Borovets, Bulgaria .
Sproat, R.(1993). Morphology and Computation . MITPress.
Sproat, R. (1994). English noun-phrase prediction for text-to-
speech.Computer Speech and Language ,8,79–94.
Sproat,R.(1998a). Furtherissuesintextanalysis. InSproat,R.
(Ed.),Multilingual Text-To-Speech Synthesis: The Bell Labs
Approach , pp.89–114. Kluwer, Dordrecht.

Sproat,R.(Ed.).(1998b). Multilingual Text-To-Speech Synthe-
sis: The BellLabs Approach . Kluwer, Dordrecht.
Sproat, R., Black, A. W., Chen, S. F., Kumar, S., Ostendorf,
M., and Richards, C. (2001). Normalization of non-standard
words.Computer Speech &Language ,15(3), 287–333.
Sproat, R., Shih, C., Gale, W. A., and Chang, N. (1996). A
stochastic ﬁnite-state word-segmentation algorithm for C hi-
nese.Computational Linguistics ,22(3), 377–404.
Stabler, E. (1997). Derivational minimalism. In Retor´ e, C.

(Ed.),Logical Aspects of Computational Linguistics , pp. 68–
95.Springer.
Stalnaker , R.C.(1978). Assertion. InCole,P.(Ed.), Pragmat-
ics: Syntax and Semantics Volume 9 , pp. 315–332. Academic
Press.
Stanners , R. F., Neiser, J., Hernon, W. P., and Hall, R. (1979).
Memory representation for morphologically related words.
Journal of Verbal Learning and Verbal Behavior ,18, 399–
412.
Steedman ,M.(1989).Constituency andcoordination inacom-
binatory grammar. In Baltin, M. R. and Kroch, A. S. (Eds.),

Press.
412.
Alternative Conceptions of Phrase Structure , pp. 201–231.
University of Chicago, Chicago.
Steedman , M. (1996). Surface Structure and Interpretation .
MITPress. Linguistic Inquiry Monograph, 30.
Steedman , M.(2000). The Syntactic Process . TheMIT Press.
Steedman ,M.(2003).Information-structural semanticsforEn-
glish intonation..
Steedman , M. and Baldridge, J. (2007). Combinatory cate-
gorial grammar. In Borsley, R. and Borjars, K. (Eds.), Non-
Transformational Syntax . Blackwell.

Stent,A.(2002). Aconversation actsmodelforgenerating spo-
ken dialogue contributions. Computer Speech and Language,
Special Issueon Spoken Language Generation ,16(3-4).
Stetina, J.and Nagao, M.(1997). Corpus based PPattachment
ambiguity resolution with a semantic dictionary. In Zhou, J .
and Church, K. W. (Eds.), Proceedings of the Fifth Workshop
on VeryLargeCorpora , Beijing, China, pp. 66–80.Stevens, K.N.,Kasowski, S.,andFant, G.M.(1953). Anelec-
trical analog of the vocal tract. Journal of the Acoustical So-

ciety ofAmerica ,25(4), 734–742.
Stevens, K.N.(1998). Acoustic Phonetics . MITPress.
Stevens, K. N. and House, A. S. (1955). Development of a
quantitative description of vowel articulation. Journal of the
Acoustical Society ofAmerica ,27,484–493.
Stevens, K. N. and House, A. S. (1961). An acoustical theory
of vowel production and some of its implications. Journal of
Speech and Hearing Research ,4,303–320.
Stevens, S. S. and Volkmann, J. (1940). The relation of pitch
to frequency: A revised scale. The American Journal of Psy-
chology,53(3), 329–353.

Stevens, S. S., Volkmann, J., and Newman, E. B. (1937). A
scale for the measurement of the psychological magnitude
pitch.Journal of the Acoustical Society of America ,8, 185–
190.
Stevenson , M.and Wilks, Y.(2001). Theinteraction of knowl-
edge sources in word sense disambiguation. Computational
Linguistics ,27(3), 321–349.
Stevenson , S. and Merlo, P.(1999). Automatic verb classiﬁca-
tion using distributions of grammatical features. In EACL-99,
Bergen, Norway, pp.45–52.
Stifelman , L. J., Arons, B., Schmandt, C., and Hulteen, E. A.

190.
(1993). VoiceNotes: Aspeech interface forahand-held voic e
notetaker. In Human Factors in Computing Systems: INTER-
CHI ’93 Conference Proceedings , Amsterdam, pp. 179–186.
ACM.
Stolcke, A.(1995). Anefﬁcient probabilistic context-free pars-
ing algorithm that computes preﬁx probabilities. Computa-
tional Linguistics ,21(2), 165–202.
Stolcke, A.(1998). Entropy-based pruningofbackofflanguage
models. In Proc. DARPA Broadcast News Transcription and
Understanding Workshop , Lansdowne, VA, pp.270–274.

ACM.
Stolcke, A. (2002). Srilm - an extensible language modeling
toolkit. In ICSLP-02 , Denver, CO.
Stolcke, A., Ries, K., Coccaro, N., Shriberg, E., Bates, R., Ju-
rafsky, D., Taylor, P., Martina, R., Meteer, M., and Van Ess-
Dykema,C.(2000). Dialogueactmodeling forautomatic tag-
gingandrecognition ofconversationalspeech. Computational
Linguistics ,26,339–371.
Stolcke, A., Shriberg, E., Bates, R., Coccaro, N., Jurafsky, D.,
Martin, R., Meteer, M., Ries, K., Taylor, P., and Van Ess-

Dykema, C. (1998). Dialog act modeling for conversational
speech. In Chu-Carroll, J. and Green, N. (Eds.), Applying
1998 AAAISpring Symposium. Tech. rep.SS-98-01 , Stanford,
CA,pp. 98–105. AAAI Press.
Stolcke, A. and Shriberg, E. (1996). Statistical language mod-
eling for speech disﬂuencies. In IEEE ICASSP-96 , Atlanta,
GA,Vol. 1, pp.405–408. IEEE.
Stolz, W. S., Tannenbaum, P. H., and Carstensen, F. V. (1965).
A stochastic approach to the grammatical coding of English.

Communications ofthe ACM ,8(6), 399–405.
Streeter, L.(1978). Acoustic determinants of phraseboundary
perception. Journal of the Acoustical Society of America ,63,
1582–1592.
Strube, M. and Ponzetto, S. P. (2006). WikiRelate! Comput-
ing semantic relatedness using Wikipedia. In AAAI-06, pp.
1419–1424.

DRAFTBibliography 967
Strube,M.andHahn,U.(1996). Functionalcentering. In ACL-
96,Santa Cruz, CA,pp. 270–277.
Strzalkowski , T. and Harabagiu, S. (Eds.). (2006). Advances
in Open Domain Question Answering . Springer.
Sundheim ,B.(Ed.).(1991). Proceedings oftheThirdMessage
Understanding Conference .
Sundheim , B. (Ed.). (1992). Proceedings of the Fourth Mes-
sage Understanding Conference .
Sundheim , B. (Ed.). (1993). Proceedings, Fifth Message Un-
derstanding Conference (MUC-5), Baltimore, MD . Morgan
Kaufmann.
Sundheim ,B.(1995a). OverviewofresultsoftheMUC-6eval-
uation. In Proceedings of the Sixth Message Understanding
Conference (MUC-6) , Columbia, MD, pp.13–31.
Sundheim ,B.(Ed.).(1995b). ProceedingsoftheSixthMessage
Surdeanu , M., Harabagiu, S., Williams, J., and Aarseth, P.
(2003). Using predicate-argument structures for informat ion
extraction. In ACL-03,pp. 8–15.
Sutton, C. and McCallum, A. (2006). An introduction to con-

Kaufmann.
ditional random ﬁelds for relational learning. In Getoor, L .
and Taskar, B. (Eds.), Introduction to Statistical Relational
Learning. MIT Press.
Sutton,R.S.andBarto,A.G.(1998). ReinforcementLearning:
AnIntroduction . Bradford Books (MIT Press).
Sweet, H.(1877). A Handbook of Phonetics . Clarendon Press,
Oxford.
Swerts, M., Litman, D. J., and Hirschberg, J. (2000). Cor-
rections in spoken dialogue systems. In ICSLP-00 , Beijing,
China.
Swier, R. and Stevenson, S. (2004). Unsupervised semantic
rolelabelling. In EMNLP 2004 , pp.95–102.

Oxford.
China.
Syrdal, A. K. and Conkie, A. (2004). Data-driven perceptu-
ally based join costs. In Proceedings of Fifth ISCA Speech
Synthesis Workshop .
Syrdal, A. K., Wightman, C. W., Conkie, A., Stylianou, Y.,
Beutnagel, M.,Schroeter, J.,Strom,V.,andLee,K.-S.(200 0).
Corpus-based techniques in the AT&T NEXTGEN synthesis
system. In ICSLP-00 , Beijing.
Talmy, L. (1985). Lexicalization patterns: Semantic structure
inlexical forms. In Shopen, T.(Ed.), Language Typology and
SyntacticDescription,Volume3 .CambridgeUniversityPress.

Originally appeared as UC Berkeley Cognitive Science Pro-
gram Report No.30, 1980.
Talmy, L.(1991). Path to realization: a typology of event con-
ﬂation. In BLS-91,Berkeley, CA, pp.480–519.
Taskar, B., Klein, D., Collins, M., Koller, D., and Manning,
C.D.(2004). Max-margin parsing. In EMNLP2004 .
Taylor, P. (2000). Analysis and synthesis of intonation using
the Tilt model. Journal of the Acoustical Society of America ,
107(3), 1697–1714.
Taylor, P. (2005). Hidden Markov Models for grapheme to
phoneme conversion. In INTERSPEECH-05 , Lisbon, Portu-

gal, pp.1973–1976.
Taylor, P.(2008). Text-to-speech synthesis. Manuscript.
Taylor, P. and Black, A. W. (1998). Assigning phrase breaks
from part of speech sequences. Computer Speech and Lan-
guage,12,99–117.Taylor, P. and Isard, S. (1991). Automatic diphone segmenta-
tion. InEUROSPEECH-91 ,Genova, Italy.
Taylor, P., King, S., Isard, S., and Wright, H. (1998). Intona-
tion and dialog context as constraints for speech recogniti on.
Language and Speech ,41(3-4), 489–508.

Taylor, W.L.(1953). Cloze procedure: anew tool for measur-
ing readability. Journalism Quarterly ,30, 415–433.
Taylor, W.L.(1957). Cloze readability scores asindices ofin-
dividual differences in comprehension and aptitude. Journal
ofApplied Psychology ,4,19–26.
terMeulen, A. (1995). Representing Time in Natural Lan-
guage. MIT Press.
Teranishi , R. and Umeda, N. (1968). Use of pronouncing dic-
tionary in speech synthesis experiments. In 6th International
Congress on Acoustics , Tokyo,Japan, pp. B155–158.

Tesar, B. (2006a). Faithful contrastive features in learning.
Cognitive Science ,30(5), 863–903.
Tesar, B.(2006b). Learning fromparadigmatic information..
Tesar, B. and Prince, A. (2007). Using phonotactics to learn
phonological alternations. In CLS 39,pp. 200–213.
Tesar, B. and Smolensky, P. (2000). Learning in Optimality
Theory. MIT Press.
Tesni`ere, L.(1959). ´El´ ementsdeSyntaxeStructurale . Librairie
C.Klincksieck, Paris.
Teufel, S.and van Halteren, H.(2004). Evaluating information

content by factoid analysis: human annotation and stabilit y.
InEMNLP2004 , Barcelona.
Teufel, S. and Moens, M. (2002). Summarizing scientiﬁc arti-
cles: experiments with relevance and rhetorical status. Com-
putational Linguistics ,28(4), 409–445.
Thede, S. M. and Harper, M. P. (1999). A second-order Hid-
den Markov Model for part-of-speech tagging. In ACL-99,
College Park, MA,pp. 175–182.
Thiessen, E. D., Hill, E. A., and Saffran, J. R. (2005). Infant-
directed speechfacilitates wordsegmentation. Infancy,7,53–
71.

71.
Thiessen, E.D. and Saffran, J. R. (2004). Spectral tilt as a cue
to word segmentation in infancy and adulthood. Perception
and Psychophysics ,66(2), 779–791.
Thompson , K. (1968). Regular expression search algorithm.
Communications ofthe ACM ,11(6), 419–422.
Tillmann , C. (2003). A projection extension algorithm for sta-
tisticalmachinetranslation. In EMNLP2003 ,Sapporo,Japan.
Titone, D. and Connine, C. M. (1997). Syllabiﬁcation strate-
gies in spoken word processing: Evidence from phonological
priming. Psychological Research ,60(4), 251–263.

Titov, I. and Henderson, J. (2006). Lossminimization in parse
reranking. In EMNLP2006 .
Titov, I. and Henderson, J.(2007). A latent variable model for
generative dependency parsing. In IWPT-07.
Tjong Kim Sang , E. F. and Veenstra, J. (1999). Representing
text chunks. In Proceedings of EACL1999 , pp.173–179.
Tokuda, K., Kobayashi, T., and Imai, S. (1995a). Speech pa-
rameter generation from HMM using dynamic features. In
IEEEICASSP-95 .

DRAFT968 Bibliography
Tokuda, K., Masuko, T., and Yamada, T. (1995b). An algo-
rithm for speech parameter generation from continuous mix-
ture HMMs with dynamic features. In EUROSPEECH-95 ,
Madrid.
Tokuda,K.,Yoshimura,T.,Masuko,T.,Kobayashi, T.,andKi-
tamura, T. (2000). Speech parameter generation algorithms
for HMM-based speech synthesis. In IEEEICASSP-00 .
Tokuda,K.,Zen,H.,andKitamura,T.(2003). Trajectory mod-
eling based on HMMs with the explicit relationship between
static and dynamic features. In EUROSPEECH-03 .
Tomabechi , H.(1991). Quasi-destructive graph uniﬁcation. In
ACL-91,Berkeley, CA, pp.315–322.
Tomokiyo , L. M. (2001). Recognizing non-native speech:
Characterizing and adapting to non-native usage in speech
recognition . Ph.D.thesis, Carnegie Mellon University.
Tomokiyo , L. M. and Waibel, A. (2001). Adaptation methods
for non-native speech. In Proceedings of Multilinguality in
Spoken Language Processing , Aalborg, Denmark.
Touretzky , D. S., Elvgren III, G., and Wheeler, D. W. (1990).

Phonological rule induction: An architectural solution. I n
COGSCI-90 , pp. 348–355.
Toutanova , K., Manning, C. D., Flickinger, D., and Oepen,
S. (2005). Stochastic HPSG Parse Disambiguation using the
Redwoods Corpus. Research on Language & Computation ,
3(1),83–105.
Toutanova , K., Ilhan, H. T., and Manning, C. D. (2002). Ex-
tensionstoHMM-based statistical wordalignmentmodels. I n
EMNLP2002 , pp. 87–94.
Toutanova , K., Klein, D., Manning, C. D., and Singer, Y.
(2003). Feature-rich part-of-speech tagging with a cyclic de-

pendency network. In HLT-NAACL-03 .
Toutanova , K.andMoore,R.C.(2002). Pronunciation model-
ingforimprovedspellingcorrection. In ACL-02,Philadelphia,
PA,pp. 144–151.
Traum,D.R.(2000).20questionsfordialogueacttaxonomies.
Journal ofSemantics ,17(1).
Traum, D. R. and Hinkelman, E.A.(1992). Conversation acts
intask-oriented spokendialogue. ComputationalIntelligence:
Special Issue on Computational Approaches to Non-Literal
Language ,8(3).
Traum,D.R.andLarsson,S.(2000).Informationstateanddia-
loguemanagement in thetrindi dialogue moveengine toolkit .

Natural Language Engineering ,6(323-340), 97–114.
Traum, D. R. and Larsson, S. (2003). The information state
approach to dialogue management. In van Kuppevelt, J. and
Smith, R. (Eds.), Current and New Directions in Discourse
and Dialogue . Kluwer.
Treiman, R., Bowey, J., and Bourassa, D. (2002). Segmenta-
tion of spoken words into syllables by english-speaking chi l-
dren as compared to adults. Journal of Experimental Child
Psychology ,83, 213–238.
Trubetskoi ,N.S.(1939). Grundz¨ ugederPhonologie , Vol.7of

Travaux du cercle linguistique de Prague . Available in 1969
English translation by Christiane A. M. Baltaxe as Principles
ofPhonology , University of California Press.
Tseng, H.(2003). Semantic classiﬁcation ofChinese unknown
words. In ACL-03,pp. 72–79.Tseng, H., Chang, P., Andrew, G., Jurafsky, D., and Manning,
C. D. (2005a). Conditional random ﬁeld word segmenter.
InProceedings of the Fourth SIGHAN Workshop on Chinese
Language Processing .
Tseng, H.,Jurafsky,D.,andManning, C.D.(2005b). Morpho-

logical features help pos tagging of unknown words across
language varieties. In Proceedings of the Fourth SIGHAN
Workshop on Chinese Language Processing .
Tsujii, J. (1986). Future directions of machine translation. In
COLING-86 , Bonn, pp. 655–668.
Turian, J. P., Shen, L., and Melamed, I. D. (2003). Evaluation
of machine translation and its evaluation. In Proceedings of
MT SummitIX ,New Orleans, LA.
Turing, A. M. (1936). On computable numbers, with an ap-
plication to the Entscheidungsproblem. Proceedings of the

London Mathematical Society ,42, 230–265. Read to the So-
cietyin1936,butpublishedin1937.Correction involume43 ,
544–546.
Turing, A. M. (1950). Computing machinery and intelligence.
Mind,59, 433–460.
Turner, J. and Charniak, E. (2005). Supervised and unsuper-
visedlearning forsentencecompression. In ACL-05,pp.290–
297.
Turney, P., Littman, M., Bigham, J., and Shnayder, V. (2003).
Combining independent modules to solve multiple-choice
synonym and analogy problems. In Proceedings of RANLP-
03,Borovets, Bulgaria, pp. 482–489.

544–546.
297.
Tyler, L. K. (1984). The structure of the initial cohort: Evi-
dence from gating. Perception & Psychophysics ,36(5), 417–
427.
Umeda, N.,Matui, E.,Suzuki, T.,and Omura, H.(1968). Syn-
thesis of fairy tale using an analog vocal tract. In 6th Interna-
tional Congress on Acoustics , Tokyo, Japan, pp. B159–162.
Umeda, N. (1976). Linguistic rules for text-to-speech synthe-
sis.Proceedings ofthe IEEE ,64(4), 443–451.
Usher, A.P.(1954). AHistory of Mechanical Inventions . Har-
vard University Press.

427.
Uszkoreit , H. (1986). Categorial uniﬁcation grammars. In
COLING-86 , Bonn, pp. 187–194.
vanBenthem, J. and ter Meulen, A. (Eds.). (1997). Handbook
ofLogic and Language . MIT Press.
vanDeemter, K. and Kibble, R. (2000). On coreferring: coref-
erenceinmucandrelatedannotation schemes. Computational
Linguistics ,26(4), 629–637.
van den Bosch, A. (1997). Learning to Pronounce Written
Words: A Study in Inductive Language Learning . Ph.D. the-
sis,University of Maastricht, Maastricht, TheNetherland s.

vanHalteren, H. and Teufel, S. (2003). Examining the con-
sensus between human summaries: initial experiments with
factoid analysis. In HLT-NAACL-03 Workshop on Text Sum-
marization .
vanHalteren, H. (Ed.). (1999). Syntactic Wordclass Tagging .
vanLehn,K.(1978). Determining thescopeofEnglishquanti-
ﬁers. Master’s thesis, MIT, Cambridge, MA. MIT Technical
Report AI-TR-483.
vanRijsbergen, C. J. (1975). Information Retrieval . Butter-
worths, London.

DRAFTBibliography 969
vanSanten, J. P.H. (1994). Assignment of segmental duration
in text-to-speech synthesis. Computer Speech and Language ,
8(95–128).
vanSanten, J. P. H. (1997). Segmental duration and speech
timing. InSagisaka, Y.,Campbell,N.,andHiguchi, N.(Eds. ),
vanSanten, J. P. H. (1998). Timing. In Sproat, R. (Ed.), Mul-
tilingual Text-To-Speech Synthesis: The Bell Labs Approac h,
pp.115–140. Kluwer, Dordrecht.
vanSanten, J. P. H. and Sproat, R. (1998). Methods and tools.
In Sproat, R. (Ed.), Multilingual Text-To-Speech Synthesis:
TheBellLabs Approach , pp. 7–30. Kluwer, Dordrecht.
vanSanten, J. P. H., Sproat, R., Olive, J. P., and Hirschberg, J.
(Eds.).(1997). Progressin Speech Synthesis . Springer.
VanSon,R.J.J.H.,Koopmans-van Beinum, F.J.,and Pols,L.
C.W.(1998). Efﬁciency as an organizing principle of natura l
speech. In ICSLP-98 , Sydney.
Van Son, R. J. J. H. and Pols, L. C. W. (2003). How efﬁcient

is speech?. Proceedings of the Institute of Phonetic Sciences ,
25,171–184.
VanValin ,Jr.,R.D.andLaPolla,R.(1997).Syntax: Structure,
meaning, and function..
Vanderwende , L. (1994). Algorithm for the automatic inter-
pretation ofnounsequences. In COLING-94 ,Kyoto,pp.782–
788.
Vanderwende , L., Suzuki, H., Brockett, C., and Nenkova, A.
(2007a).Beyondsumbasic: Task-focusedsummarizationwit h
sentence simpliﬁcation and lexical expansion. Information
Processing and Management ,43(6), 1606–1618.

788.
Vanderwende , L., Suzuki, H., Brockett, C., and Nenkova,
A. (2007b). Beyond sumbasic: Task-focused summarization
with sentence simpliﬁcation and lexical expansion. Informa-
tion Processing and Management, Special issue on summa-
rization,43(6).
VanLehn , K., Jordan, P. W., Ros´ e, C., Bhembe, D., B¨ ottner,
M., Gaydos, A., Makatchev, M., Pappuswamy, U., Ringen-
berg, M., Roque, A., Siler, S., Srivastava, R., and Wilson, R .
(2002). Thearchitecture of Why2-Atlas: Acoach for qualita -

tive physics essay writing. In Proc. Intelligent Tutoring Sys-
tems.
Vapnik, V. N. (1995). The Nature of Statistical Learning The-
ory. Springer-Verlag.
Vasilescu , F.,Langlais, P.,and Lapalme, G.(2004). Evaluating
variants of the lesk approach for disambiguating words. In
LREC-04 ,Lisbon,Portugal, pp. 633–636. ELRA.
Vauquois , B. (1968). A survey of formal grammars and algo-
rithms for recognition and transformation in machine trans la-
tion. InIFIPCongress 1968 , Edinburgh, pp.254–260.

tems.
Veale, T. and Keane, M. T. (1992). Conceptual scaffolding: A
spatially founded meaning representation for metaphor com -
prehension. Computational Intelligence ,8(3), 494–519.
Veblen, T. (1899). Theory of the Leisure Class . Macmillan
Company, New York.
Veldhuis, R. (2000). Consistent pitch marking. In ICSLP-00 ,
Beijing, China.
Velichko, V. M. and Zagoruyko, N. G. (1970). Automatic
recognition of 200 words. International Journal of Man-

Machine Studies ,2,223–234.Venditti, J. J. (2005). The j tobi model of japanese intonation.
In Jun, S.-A. (Ed.), Prosodic Typology and Transcription: A
Uniﬁed Approach . Oxford University Press.
Vendler, Z.(1967). Linguistics in Philosophy . Cornell Univer-
sity Press, Ithaca, NY.
Venugopal , A., Vogel, S., and Waibel, A. (2003). Effective
phrasetranslation extraction fromalignment models. In ACL-
03,pp. 319–326.
Verhagen , M., Gaizauskas, R., Schilder, F., Hepple, M., Katz,

G., and Pustejovsky, J. (2007). Semeval-2007 task 15: Tem-
peval temporal relation identiﬁcation. In Proceedings of
the Fourth International Workshop on Semantic Evaluations
(SemEval-2007) , Prague, Czech Republic, pp.75–80.
Veronis, J. and Ide, N. M. (1990). Word sense disambigua-
tion with very large neural networks extracted from machine
readable dictionaries. In COLING-90 , Helsinki, Finland, pp.
389–394.
Verspoor ,C.M.,Joslyn,C.,andPapcun,G.J.(2003).Thegene
ontology as asource of lexical semantic knowledge for abio-

389–394.
logical natural language processing application. In Proceed-
ings of the SIGIR’03 Workshop on Text Analysis and Search
for Bioinformatics , Toronto, CA.
Vieira, R.andPoesio, M.(2000). Anempirically based system
for processing deﬁnite descriptions. Computational Linguis-
tics,26(4), 539–593.
Vilain, M., Burger, J. D., Aberdeen, J., Connolly, D., and
Hirschman, L.(1995). Amodel-theoretic coreference scori ng
scheme. In MUC6’95: Proceedings ofthe6thConference on
Message Understanding .
Vintsyuk ,T.K.(1968).Speechdiscrimination bydynamicpro-

gramming. Cybernetics ,4(1), 52–57. Russian Kibernetika
4(1):81-88 (1968).
Viterbi,A.J.(1967). Errorboundsforconvolutional codesand
an asymptotically optimum decoding algorithm. IEEETrans-
actions on Information Theory ,IT-13(2), 260–269.
Voiers, W., Sharpley, A., and Hehmsoth, C. (1975). Research
on diagnostic evaluation of speech intelligibility. Resea rch
Report AFCRL-72-0694.
vonNeumann, J.(1963). Collected Works: VolumeV . Macmil-
lan Company, New York.
Voorhees , E. M. (1998). Using WordNet for text retrieval.

Database , pp.285–303. MITPress.
Voorhees ,E.M.andHarman,D.K.(2005). TREC:Experiment
and Evaluation in Information Retrieval . MIT Press.
Voorhees , E. M. and Tice, D. M. (1999). The TREC-8 ques-
tion answering track evaluation. Proceedings of the TREC-8
Workshop.
Voutilainen , A. (1995). Morphological disambiguation. In
Karlsson, F., Voutilainen, A., Heikkil¨ a, J., and Anttila, A.
(Eds.),Constraint Grammar: A Language-Independent Sys-
tem for Parsing Unrestricted Text , pp. 165–284. Mouton de

Workshop.
Gruyter, Berlin.
Voutilainen , A. (1999). Handcrafted rules. In van Halteren,
H. (Ed.), Syntactic Wordclass Tagging , pp. 217–246. Kluwer,
Wade, E., Shriberg, E., and Price, P. J. (1992). User behaviors
affecting speech recognition. In ICSLP-92 , pp. 995–998.

DRAFT970 Bibliography
Wagner, R. A. and Fischer, M. J. (1974). The string-to-string
correction problem. JournaloftheAssociation forComputing
Machinery ,21, 168–173.
Waibel, A. (1988). Prosody and Speech Recognition . Morgan
Kaufmann.
Wald,B.andShopen,T.(1981). Aresearcher’s guidetotheso-
ciolinguisticvariable(ING).InShopen,T.andWilliams,J .M.
(Eds.),StyleandVariablesinEnglish ,pp.219–249.Winthrop
Publishers.
Walker, M. A., Fromer, J. C., and Narayanan, S. S. (1998).
Learning optimal dialogue strategies: a case study of a spo-
ken dialogue agent for email. In COLING/ACL-98 , Montreal,
Canada, pp. 1345–1351.
Walker, M. A., Iida, M., and Cote, S. (1994). Japanese dis-
course and the process of centering. Computational Linguis-
tics,20(2).
Walker,M.A.,Joshi,A.K.,andPrince,E.(Eds.).(1998). Cen-
tering in Discourse . Oxford University Press.
Walker, M. A., Kamm, C. A., and Litman, D. J. (2001).
Towards developing general models of usability with PAR-
ADISE. Natural Language Engineering: Special Issue on

BestPractice in Spoken Dialogue Systems ,6(3).
Walker, M. A., Litman, D. J., Kamm, C. A., and Abella, A.
(1997). PARADISE:Aframework for evaluating spoken dia-
logue agents. In ACL/EACL-97 ,Madrid, Spain, pp.271–280.
Walker, M. A., Maier, E., Allen, J., Carletta, J., Condon,
S., Flammia, G., Hirschberg, J., Isard, S., Ishizaki, M.,
Levin, L., Luperfoy, S., Traum, D. R., and Whittaker, S.
(1996). Penn multiparty standard coding scheme: Draft
annotation manual. www.cis.upenn.edu/˜ircs/dis
course-tagging/newcoding.html .

Walker, M. A., Passonneau, R., Rudnicky, A. I., Aberdeen, J.,
Boland, J.,Bratt, E.,Garofolo, J.,Hirschman, L.,Le,A.,L ee,
S., Narayanan, S. S., Papineni, K., Pellom, B., Polifroni, J .,
Potamianos,A.,Prabhu,P.,Rudnicky, A.I.,Sanders,G.,Se n-
eff,S.,Stallard, D.,and Whittaker, S.(2002). Cross-site eval-
uation inDARPA Communicator: TheJune2000 datacollec-
tion. submitted.
Walker, M.A.and Rambow,O.(2002). Spoken language gen-
eration. Computer Speech and Language, Special Issue on
Spoken Language Generation ,16(3-4), 273–281.

Walker, M. A. and Whittaker, S. (1990). Mixed initiative in
dialogue: An investigation into discourse segmentation. I n
ACL-90,Pittsburgh, PA,pp.70–78.
Walker, M. A.et al. (2001). Cross-site evaluation in darpa
communicator: Thejune 2000 data collection. Submitted ms.
Wang, M. Q. and Hirschberg, J. (1992). Automatic classiﬁca-
tion of intonational phrasing boundaries. Computer Speech
and Language ,6(2), 175–196.
Wang,Y.Y.andWaibel, A.(1997). Decoding algorithm insta-
tistical machine translation. In ACL/EACL-97 ,pp. 366–372.

Wang, Z., Schultz, T., and Waibel, A. (2003). Comparison of
acoustic model adaptation techniques on non-native speech .
InIEEEICASSP ,Vol. 1, pp.540–543.
Wanner, E. and Maratsos, M. (1978). An ATN approach to
comprehension. In Halle, M., Bresnan, J., and Miller, G. A.
(Eds.),Linguistic TheoryandPsychological Reality , pp.119–
161.MIT Press.Ward,N.(1994). AConnectionistLanguageGenerator .Ablex.
Ward, N. and Tsukahara, W. (2000). Prosodic features which
cue back-channel feedback in English and Japanese. Journal
ofPragmatics ,32,1177–1207.

Ward, W. (1989). Modelling non-verbal sounds for speech
recognition. In HLT ’89: Proceedings of the Workshop on
Speech and Natural Language , Cape Cod, MA,pp.47–50.
Ward, W. and Issar, S. (1994). Recent improvements in the
cmuspokenlanguageunderstanding system. In ARPAHuman
Language Technologies Workshop , Plainsboro, N.J.
Warnke, V., Kompe, R., Niemann, H., and N¨ oth, E. (1997).
Integrated dialog act segmentation and classiﬁcation usin g
prosodic features and language models. In EUROSPEECH-
97,Vol. 1,pp. 207–210.

Warren, R. M. (1970). Perceptual restoration of missing
speech sounds. Science,167, 392–393.
Warwick , S. and Russell, G. (1990). Bilingual concordancing
and bilingual lexicography. In EURALEX 4th International
Congress.
Wattarujeekrit , T., Shah, P. K., and Collier, N. (2004). PAS-
Bio: predicate-argument structures for event extraction i n
molecular biology. BMC Bioinformatics ,5(155).
Waugh,L.R.(1976).Thesemanticsandparadigmaticsofword
order.Language ,52(1), 82–107.
Weaver, W. (1949/1955). Translation. In Locke, W. N. and

Congress.
Boothe, A. D. (Eds.), Machine Translation of Languages , pp.
15–23.MITPress. Reprinted fromamemorandumwritten by
Weaver in 1949.
Webber, B. L.(2004). D-LTAG:extending lexicalized TAG to
discourse. Cognitive Science ,28(5), 751–79.
Webber, B. L., Knott, A., Stone, M., and Joshi, A. K. (1999).
Discourse relations: A structural and presuppositional ac -
count using lexicalised TAG. In ACL-99, College Park, MD,
pp.41–48.
Webber, B. L. (1978). A Formal Approach to Discourse
Anaphora . Ph.D.thesis, Harvard University.

pp.41–48.
Webber, B. L. (1983). So what can we talk about now?. In
Brady, M. and Berwick, R. C. (Eds.), Computational Mod-
els of Discourse , pp. 331–371. The MIT Press. Reprinted in
Webber, B. L. (1991). Structure and ostension in the interpre-
tationofdiscoursedeixis. LanguageandCognitiveProcesses ,
6(2), 107–135.
Weber, D. J.,Black, H. A.,and McConnel, S.R. (1988). AM-
PLE:Atool for exploring morphology. Tech. rep.Occasional
Publications in Academic Computing No. 12, Summer Insti-
tute of Linguistics, Dallas.

Weber, D.J.and Mann,W.C.(1981). Prospects for computer-
assisted dialect adaptation. American Journal of Computa-
tional Linguistics ,7,165–177. Abridged from Summer Insti-
titute of Linguistics Notes on Linguistics Special Publication
1,1979.
Weeds, J. (2003). Measures and Applications of Lexical Dis-
tributional Similarity . Ph.D.thesis, University ofSussex.
Weeds, J. and Weir, D. J. (2005). Co-occurrence retrieval: a
general framework for lexical distributional similarity. Com-
putational Linguistics ,31(4), 439–476.

1,1979.
Wegmann ,S.,McAllaster,D.,Orloff,J.,andPeskin,B.(1996).
Speaker normalisation onconversational telephone speech . In
IEEEICASSP-96 , Atlanta, GA.

DRAFTBibliography 971
Weinschenk , S. and Barker, D. T. (2000). Designing effective
speech interfaces . Wiley.
Weintraub ,M.,Taussig,K.,Hunicke-Smith, K.,andSnodgras,
A. (1996). Effect of speaking style on LVCSR performance.
InICSLP-96 , Philadelphia, PA,pp.16–19.
Weischedel , R.(1995). BBN:DescriptionofthePLUMsystem
asused for MUC-6. In Proceedings oftheSixth Message Un-
derstanding Conference (MUC-6) , San Francisco, pp. 55–70.
Weischedel , R., Meteer, M., Schwartz, R., Ramshaw, L. A.,
andPalmucci,J.(1993).Copingwithambiguityandunknown
words through probabilistic models. Computational Linguis-
tics,19(2), 359–382.
Weizenbaum , J.(1966). ELIZA–Acomputer program forthe
study of natural language communication between man and
machine. Communications ofthe ACM ,9(1), 36–45.
Weizenbaum , J.(1976). ComputerPowerandHumanReason:
FromJudgementtoCalculation .W.H.FreemanandCompany,
San Francisco.
Welling, L.,Ney,H.,and Kanthak, S.(2002). Speaker adaptive

modeling byvocal tract normalisation. IEEETransactions on
Speech and Audio Processing ,10,415–426.
Wells, J. C. (1982). Accents ofEnglish . Cambridge University
Press.
Weng, F., Varges, S., Raghunathan, B., Ratiu, F., Pon-Barry,
H., Lathrop, B., Zhang, Q., Scheideck, T., Bratt, H., Xu, K.,
Purver, M.,Mishra, R., Raya, M.,Peters, S.,Meng, Y., Cave-
don, L., and Shriberg, E. (2006). Chat: A conversational
helper for automotive tasks. In ICSLP-06 , pp. 1061–1064.
Wierzbicka , A. (1992). Semantics, Culture, and Cognition:

Press.
University Human Concepts in Culture-Speciﬁc Conﬁgura-
tions. Oxford University Press.
Wierzbicka , A. (1996). Semantics: Primes and Universals .
Wilensky , R.(1983). Planning andUnderstanding: ACompu-
tational Approach to Human Reasoning . Addison-Wesley.
Wilensky , R. and Arens, Y. (1980). PHRAN: A knowledge-
based natural language understander. In ACL-80, Philadel-
phia, PA,pp. 117–121.
Wilks, Y.(1975a). An intelligent analyzer and understander of
English.Communications ofthe ACM ,18(5), 264–274.

Wilks, Y. (1975b). Preference semantics. In Keenan, E. L.
(Ed.),The Formal Semantics of Natural Language , pp. 329–
350.Cambridge Univ. Press.
Wilks, Y. (1975c). A preferential, pattern-seeking, semantics
for natural language inference. Artiﬁcial Intelligence ,6(1),
53–74.
Wilks, Y. (1978). Making preferences more active. Artiﬁcial
Intelligence ,11(3), 197–223.
Wilks, Y., Slator, B. M., and Guthrie, L. M. (1996). Electric
Words: Dictionaries, Computers, and Meanings . MITPress.
Williams , J. D. and Young, S. J. (2000). Partially observable

53–74.
markov decision processes for spoken dialog systems. Com-
puter Speech and Language ,21(1), 393–422.
Williams , J. D. and Young, S. J. (2005). Scaling up pomdps
for dialog management: The ”summary pomdp” method. In
IEEEASRU-05 .Winograd , T. (1972a). Understanding natural language. Cog-
nitive Psychology ,3(1), 1–191. Reprinted as a book by Aca-
demic Press,1972.
Winograd ,T.(1972b). UnderstandingNaturalLanguage .Aca-
demic Press.
Wise, B.,Cole, R.A.,van Vuuren, S.,Schwartz, S.,Snyder,L.,

Ngampatipatpong, N.,Tuantranont, J.,andPellom,B.(2007 ).
Learning to Read with a Virtual Tutor: Foundations to Liter-
acy. In Kinzer, C.and Verhoeven, L.(Eds.), Interactive Liter-
acy Education: Facilitating Literacy Environments Throug h
Technology . Lawrence Erlbaum.
Witten, I. H.and Bell, T. C. (1991). Thezero-frequency prob-
lem: Estimating the probabilities of novel events in adapti ve
text compression. IEEETransactions on Information Theory ,
37(4), 1085–1094.
Witten, I. H. and Frank, E. (2005). Data Mining: Practical

Machine Learning ToolsandTechniques . MorganKaufmann.
2nd ed.
Wittgenstein , L.(1953). Philosophical Investigations. (Trans-
lated byAnscombe, G.E.M.) . Blackwell, Oxford.
Wolf, F.and Gibson, E.(2005). Representing discourse coher-
ence: A corpus-based analysis. Computational Linguistics ,
31(2), 249–287.
Wolfram , W. A. (1969). A Sociolinguistic Description of De-
troit Negro Speech . Center for Applied Linguistics, Washing-
ton, D.C.
Woodland , P.C.,Leggetter, C.J.,Odell, J.J.,Valtchev, V.,and

2nd ed.
ton, D.C.
Young, S. J. (1995). The 1994 htk large vocabulary speech
recognition system. In IEEEICASSP .
Woodland , P.C.andPovey,D.(2002). Largescalediscrimina-
tivetraining ofhiddenMarkovmodelsforspeechrecognitio n.
Computer Speech and Language, ,16, 25–47.
Woodland , P. C. (2001). Speaker adaptation for continuous
density HMMs: A review. In Juncqua, J.-C. and Wellekens,
C. (Eds.), Proceedings of the ITRW ‘Adaptation Methods For
Speech Recognition’ , Sophia-Antipolis, France.

Woods, W. A. (1967). Semantics for a Question-Answering
System. Ph.D.thesis, Harvard University.
Woods,W.A.(1973). Progressinnaturallanguageunderstand-
ing. InProceedings of AFIPS National Conference , pp. 441–
450.
Woods, W. A. (1975). What’s in a link: Foundations for se-
manticnetworks. InBobrow,D.G.and Collins, A.M.(Eds.),
Representation and Understanding: Studies in Cognitive Sc i-
ence, pp.35–82. Academic Press.
Woods, W. A. (1977). Lunar rocks in natural English: Ex-
plorations in natural language question answering. In Zam-

450.
polli,A.(Ed.), LinguisticStructuresProcessing ,pp.521–569.
North Holland, Amsterdam.
Woods, W. A. (1978). Semantics and quantiﬁcation in natural
language question answering. In Yovits, M. (Ed.), Advances
in Computers , Vol. 17, pp. 2–87. Academic Press.
Woods,W.A.,Kaplan,R.M.,andNash-Webber,B.L.(1972a).
The Lunar Sciences Natural Language Information System:
Final report. Tech. rep. 2378, Bolt, Beranek, and Newman,
Inc.,Cambridge, MA.
Woods,W.A.,Kaplan,R.M.,andNash-Webber,B.L.(1972b).

The lunar sciences natural language information system: Fi -
nal report.. BBN Report 2378.

DRAFT972 Bibliography
Woszczyna , M. and Waibel, A. (1994). Inferring linguistic
structureinspokenlanguage. In ICSLP-94 ,Yokohama,Japan,
pp.847–850.
Wouters, J. and Macon, M. (1998). Perceptual evaluation
of distance measures for concatenative speech synthesis. I n
ICSLP-98 ,Sydney, pp.2747–2750.
Wu, D. (1992). Automatic Inference: A Probabilistic Basis for
Natural Language Interpretation . Ph.D. thesis, University of
California, Berkeley, Berkeley, CA. UCB/CSD 92-692.
Wu,D.(1996). Apolynomial-time algorithm forstatistical ma -
chine translation. In ACL-96,Santa Cruz, CA,pp. 152–158.
Wu,D.(2000). Bracketing andaligning wordsandconstituents
in parallel text using stochastic inversion transduction g ram-
mars.InVeronis,J.(Ed.), ParallelTextProcessing: Alignment
and UseofTranslation Corpora . Kluwer, Dordrecht.
Wu, D. and Fung, P. (2005). Inversion transduction gram-
mar constraints for mining parallel sentences from quasi-
comparable corpora. In IJCNLP-2005 , Jeju, Korea.

Wu, Z. and Palmer, M. (1994). Verb semantics and lexical se-
lection. In ACL-94,Las Cruces, NM, pp.133–138.
Wundt,W.(1900). V¨ olkerpsychologie: eine Untersuchung der
Entwicklungsgesetze von Sprache, Mythus, und Sitte . W. En-
gelmann, Leipzig. Band II:Die Sprache, Zweiter Teil.
Xia, F. and Palmer, M. (2001). Converting dependency struc-
tures to phrasestructures. In HLT-01,San Diego, pp. 1–5.
Xu, P., Chelba, C., and Jelinek, F. (2002). A study on richer

syntactic dependencies for structured language modeling. In
ACL-02,pp. 191–198.
Xu,W.andRudnicky, A.I.(2000). Task-based dialog manage-
ment using an agenda. In ANLP/NAACL Workshop on Con-
versational Systems , Somerset, New Jersey,pp. 42–47.
Xue,N.andPalmer,M.(2004). Calibrating featuresforseman-
tic rolelabeling. In EMNLP2004 .
Xue, N. and Shen, L. (2003). Chinese word segmentation as
lmrtagging. In Proceedings ofthe2ndSIGHANWorkshop on
Chinese Language Processing , Sapporo, Japan.

Yamada, H.andMatsumoto, Y.(2003). Statistical dependency
analysis with supportvector machines. In Noord,G.V. (Ed.) ,
IWPT-03,pp. 195–206.
Yamada, K. and Knight, K. (2001). A syntax-based statistical
translation model. In ACL-01, Toulouse, France.
Yamada,K.andKnight,K.(2002).Adecoderforsyntax-based
statistical MT. In ACL-02, Philadelphia, PA.
Yankelovich , N.,Levow,G.-A.,andMarx, M.(1995). Design-
ing SpeechActs: Issues in speech user interfaces. In Human
FactorsinComputingSystems: CHI’95ConferenceProceed-

ings,Denver, CO,pp. 369–376. ACM.
Yarowsky , D. and Wicentowski, R. (2000). Minimally super-
vised morphological analysis by multimodal alignment. In
ACL-00,Hong Kong, pp.207–216.
Yarowsky , D. (1994). Decision lists for lexical ambiguity
resolution: Application to accent restoration in Spanish a nd
French. In ACL-94, LasCruces, NM,pp. 88–95. ACL.
Yarowsky , D. (1995). Unsupervised word sense disambigua-
tion rivaling supervised methods. In ACL-95, Cambridge,

MA,pp.189–196.Yarowsky , D. (1997). Homograph disambiguation in text-to-
speech synthesis. In van Santen, J. P. H., Sproat, R., Olive,
J. P., and Hirschberg, J. (Eds.), Progress in Speech Synthesis ,
pp.157–172. Springer.
Yeh, A., Morgan, A., Colosimo, M. E., and Hirschman, L.
(2005). BioCreatve task1A:genemention ﬁndingevaluation .
BMC Bioinformatics ,6(1).
Yngve,V.H.(1955).Syntaxandtheproblemofmultiplemean-
ing. InLocke, W.N.andBooth, A.D.(Eds.), Machine Trans-
lation of Languages , pp. 208–226. MIT Press.

Yngve, V. H. (1960). A model and an hypothesis for language
structure. ProceedingsoftheAmericanPhilosophicalSociety ,
104, 444–466.
Yngve, V. H. (1970). On getting a word in edgewise. In CLS-
70,pp. 567–577. University of Chicago.
Young, M. and Rounds, W. C. (1993). A logical semantics for
nonmonotonicsorts.In ACL-93,Columbus,OH,pp.209–215.
Young, S. J. (1984). Generating multiple solutions from con-
nected word dp recognition algorithms. Proceedings of the
Institute ofAcoustics ,6(4), 351–354.
Young, S. J., Odell, J. J., and Woodland, P. C. (1994). Tree-

based state tying for high accuracy acoustic modelling. In
Proceedings ARPA Workshop on Human Language Technol-
ogy, pp. 307–312.
Young,S.J.,Russell, N.H.,and Thornton, J.H.S.(1989). To-
kenpassing: Asimpleconceptualmodelforconnected speech
recognition systems. Tech. rep. CUED/F-INFENG/TR.38,
Cambridge University Engineering Department, Cambridge,
England.
Young, S. J. and Woodland, P. C. (1994). State clustering
in HMM-based continuous speech recognition. Computer
Speech and Language ,8(4), 369–394.

England.
Young, S. J. (2002). The statistical approach to the de-
sign of spoken dialogue systems. Tech. rep. CUED/F-
INFENG/TR.433,CambridgeUniversityEngineeringDepart-
ment, Cambridge, England.
Young, S. J., Evermann, G., Gales, M., Hain, T., Kershaw, D.,
Moore, G., Odell, J. J., Ollason, D., Povey, D., Valtchev, V. ,
andWoodland, P.C.(2005). TheHTKBook . Cambridge Uni-
versity Engineering Department.
Younger,D.H.(1967).Recognitionandparsingofcontext-free
languages in time n3.Information and Control ,10,189–208.

Yuan, J., Brenier, J. M., and Jurafsky, D. (2005). Pitch accent
prediction: Effects of genre and speaker. In EUROSPEECH-
05.
Yuret,D.(1998). Discovery ofLinguistic Relations UsingLex-
ical Attraction . Ph.D.thesis, MIT.
Yuret, D.(2004). Someexperiments with aNaive Bayes WSD
system. In Senseval-3: Third International Workshop on the
Evaluation ofSystems for the Semantic Analysis ofText .
Zajic, D., Dorr, B., Lin, J., and Schwartz, R. (2007). Multi-
candidatereduction: Sentencecompressionasatoolfordoc u-

05.
mentsummarization tasks. Information Processing andMan-
agement,43(6), 1549–1570.
Zappa, F. and Zappa, M. U. (1982). Valley girl. From Frank
Zappa album Ship Arriving Too Late To Save A Drowning
Witch.
Zavrel, J.and Daelemans, W.(1997). Memory-based learning:
Using similarity for smoothing. In ACL/EACL-97 , Madrid,
Spain, pp. 436–443.

DRAFTBibliography 973
Zens, R. and Ney, H. (2007). Efﬁcient phrase-table represen-
tation for machine translation with applications to online MT
and speech translation. In NAACL-HLT 07 , Rochester, NY,
pp.492–499.
Zens, R., Och, F. J., and Ney, H. (2002). Phrase-based statisti-
cal machine translation. In KI 2002, pp. 18–32.
Zernik, U.(1987). Strategies inLanguage Acquisition: Learn-
ing Phrases from Examples in Context . Ph.D. thesis, Univer-
sity of California, Los Angeles, Computer Science Depart-
ment, LosAngeles, CA.
Zernik, U. (1991). Train1 vs. train2: Tagging word senses in
corpus. In LexicalAcquisition: Exploiting On-LineResources
to Build a Lexicon , pp. 91–112. Lawrence Erlbaum.
Zhang, H.and Gildea, D.(2005). Stochastic lexicalized inver-
sion transduction grammar for alignment. In ACL-05, Ann
Arbor, MI.
Zheng, Y., Sproat, R., Gu, L.,Shafran, I.,Zhou, H., Su, Y., Ju-
rafsky,D.,Starr, R.,andYoon,S.-Y.(2005). Accent detect ion
and speech recognition for Shanghai-accented Mandarin. In

Arbor, MI.
InterSpeech 2005 , Lisbon, Portugal.
Zhou,G.andLua,K.(1998). Wordassociation andMI-trigger-
based language modelling. In COLING/ACL-98 , Montreal,
Canada, pp. 1465–1471.
Zhou, G., Su, J., Zhang, J., and Zhang, M. (2005). Explor-
ingvarious knowledge inrelation extraction. In ACL-05,Ann
Arbor, MI,pp. 427–434.
Zhu,X.andRosenfeld, R.(2001). Improvingtrigramlanguage
modeling with theworld wideweb. In IEEEICASSP-01 , Salt
LakeCity, UT,Vol. I,pp. 533–536.
Zue, V.W.,Glass, J.,Goodine, D.,Leung, H.,Phillips, M.,Po-

lifroni,J.,andSeneff,S.(1989). Preliminary evaluation ofthe
VOYAGER spoken language system. In Proceedings DARPA
Speech and NaturalLanguage Workshop ,Cape Cod,MA,pp.
160–167. Morgan Kaufmann.
Zweig, G.(1998). Speech Recognition with Dynamic Bayesian
Networks. Ph.D.thesis, University of California, Berkeley.
Zwicky, A. (1972). On Casual Speech. In CLS-72, pp. 607–
615.University of Chicago.
Zwicky, A.and Sadock, J.(1975). Ambiguity tests and how to
fail them. In Kimball, J. (Ed.), Syntax and Semantics 4 , pp.
1–36. Academic Press.

DRAFTIndex
λ-reduction, 565
⊔uniﬁcation operator, 498
*,373
.,283, 688,823
.wav format, 233
#,61
ˆ ,140
DeMeulder, F.,13, 746, 775
ISO8601., 758,759
Schulte im Walde, S.,691
Tjong Kim Sang, E.F.,458
ˆ ,140
10-fold crossvalidation, 155
()(optional constituents), 400
→(derives), 392
Σto mean ﬁnite alphabet of
symbols, 28
δas transition function between
states in FSA,28
ǫ-transitions in ﬁnite-state
automata, 32
#,61
ˆ,61
* (REKleene *), 20
+(RE Kleene +), 20
.(RE any character), 20
, \(RE or symbol)21
$(RE end-of-line), 21
((RE precedence symbol), 21
[(RE character disjunction), 19
\B(RE non word-boundary), 21
\b(RE word-boundary), 21
](RE character disjunction), 19
ˆ(RE start-of-line), 21
[ˆ](single-char negation), 19
∃(there exists), 563
∀(for all), 564
⇒(implies), 567
λ-expressions, 565
∧(and), 563
¬(not), 563
∨(or), 567
2001: ASpace Odyssey,1
3sg,403
4-tuple, 395
,(short pause), 421
. (long pause), 421
[] (non-verbal events), 421
A¨ ıt-Kaci, H.,527, 532
A∗decoder,339
compared to Viterbi, 344

∨(or), 567
3sg,403
for information ordering in
summarization, 823pseudocode, 345
A∗evaluation function, 346
A∗search,363
A∗search
in MT,907
A-Dconversion, 232,297
Aarseth, P.,690
AB test,283
abbreviation expansion, 253
Abduction, 569,725
Abe, N.,688
Abeill´ e, A.,490, 491
Abella, A.,851, 852
Abelson, R. P.,764
Aberdeen, J.,724, 852
Abney, S.,456
Abney, S.P.,264, 462, 483, 484,
491, 532, 550,688
ABox,579
ABSITY,688
absolute
temporal expression, 755
absolute discounting, 109
abstract
in text summarization, 802

A∗search
in MT,907
ABox,579
absolute
abstract
acceleration feature in ASR, 304
Accent ratio, 266
accented syllables, 224
accepting
by ﬁnite-state automaton, 27
accepting state, 27
Accessibility scale, 712
accessing areferent, 708
Accomplishment expressions,
577
accomplishment expressions,
575
accuracy
asmetricforPOStagging, 155
in WSD,656
Accusative, 404
accusative, 404
Acero, A.,247, 285, 306,326,
335
Achievement expressions, 577
achievement expressions, 575,
577
ACL,15
Acoustic model, 292
acoustic model, 292
frame based, 364
segment based, 364
acoustic modeling, 292

accepting
577
575
accuracy
in WSD,656
335
577
ACL,15
gender dependent, 356
Acquaah-Mensah, G. K.,769,
773acronyms
in TTS,253
Action schema, 865
action schema, 865
active list (in beam search), 326
active voice sentence, 407
activity expressions, 575, 576
Ad hoc retrieval, 782
ad hocretrieval, 782,784
Adafre, S.F.,760
Adams, J.,285
Adamson, M. J.,284
language model, 111
speaker in ASR, 356
Add-one, 98
add-one smoothing, 98
Additive noise, 355
adequacy
in MT,909
Ades, A.E.,428
adjacency pair, 828
Adjective phrase, 400
adjective phrase, 400, 426
adjectives, 125,400

in TTS,253
adequacy
in MT,909
Adjukiewicz, K.,420
in TAG,427
Adriaans, P.,492
adverb,126, 400
days of the week coded as
noun instead of,126
degree,126
directional, 126
locative,126
manner,126
temporal, 126
afﬁx, 47
affricate sound, 221
African-American Vernacular
English,230
agent, asthematic role, 633
agglomerative clustering, 687
agglutinative
language, 877
morphology, 48
Agichtein, E.,753, 799
Agirre, E.,688, 689
agreement, 52,402, 403
determiner-nominal, 502, 503
number, 404
explosion of grammar size to
deal with, 404

in TAG,427
afﬁx, 47
subject-verb, 426, 502, 503
Ahls´ en, E.,854
Ahn, D.,734, 760
975

DRAFT976 Index
Aho, A.V.,434, 461, 462
Aist, G.,871
Albelson, R. P.,11, 629
Albert, S.,772
Albright, A.,387
Alderete, J.,387
Algoet, P.H.,115
ALGOL,427
algorithm
A* search, 345
beam search stack decoder for
MT,908
bottom-up parsing, 434
Brill tagger, 154
centroid for summarization,
823
chart parsing, 452
CKY, 440
Corpus Lesk,658, 659
decision list classiﬁer, 655
Earley,447
Extended Gloss Overlap, 668
Extended Lesk,668
forward, 183, 321
forward-backward, 192
Good-Turing smoothing, 101
Hobbs tree search algorithm
for pronoun resolution,
716
IBM Model 1,895
IBM Model 3,914
Jiang-Conrath word similarity,
668
Kneser-Ney discounting, 109
Lesk, 658
Lin association measure, 674
Lin word similarity, 668
log-linear models for MT,918
minimum edit distance, 76
N-gram tiling for question
answering, 800
naive Bayes classiﬁer, 653
named entity recognition, 745
path-length based similarity,
665
pointwise mutual information,
674
probabilistic CKY, 469
Resnik word similarity, 667
semantic role labeling, 684
Simpliﬁed Lesk,658

716
668
Lesk, 658
665
674
Soundex, 82
stack decoding, 345
stack decoding for MT,905
SumBasic, 823
t-test for word similarity, 675
TBL,154
TextTiling, 697
top-down parsing, 433unsupervised word sense
disambiguation, 687
Viterbi, 184,186, 323
Yarowsky, 662
Algorithm schema, 454
alignment, 74
forced, in TTS,275
HMM,for MT,897
in ASR,330
in letter-to-sound rules, 260
minimum cost, 75
of accents to syllables, 271
of sentences in extractive
summarization, 809
phrase, in MT,892
sentence in MT,899
string,74
via minimum edit distance, 75

TBL,154
in ASR,330
string,74
word, in MT,893
alignment templates, 920
all-words task in WSD, 651
Allaria, G.,691
Allen, J.,12, 16,260, 284, 285,
364, 387, 587,764, 853,
854, 859, 864,869–871
allophone, 226
allophonic rules, 246
Allwood, J.,854
alphabet
in deﬁning aformal language,
30
Alshawi, H.,600, 606, 608, 611,
619, 733
alternation, subcategorization,
425
Alternations, 510
Althaus, E.,813, 823
Altun, Y.,266
alveolar sound, 220
Alwan, A.,285
Ambiguity, 4
ambiguity, 4,461
amount of part-of-speech in
Brown corpus, 137
attachment, 436

alphabet
30
619, 733
425
coordination, 436, 461, 474
dialogue act, 856
duck as example, 4
homograph, 256
in semantic analysis, 590
in meaning representations,
554
in morphology, 67
in named entity recognition,
741
lexical, 650
local,437
noun phrase bracketing, 461
of referring expressions, 709
part-of-speech, 133PCFG in ,467
PP-attachment, 461
prepositional phrase
attachment, 473
resolution, 4
resolution of tag, 133
speciﬁc/non-speciﬁc, 710
speech act, 5,856
syntactic, 4
tests to distinguish from
vagueness, 555
word sense, 650

554
741
local,437
Ambiguous, 4
American Structuralism, 427
amplitude
of asignal, 231
RMS,234
Amsler, R.A.,645
analogical reasoning
and metaphor, 646
Anaphora, 708
anaphora resolution, 709
anchors in regular expressions,
21,41
Anderson, A.H.,854, 856,871
Anderson, M.J.,271
Anderson, S.R.,531
Andreou, A.,364
Andrew, G.,81
Ang, J.,364
Answer type, 794
Answer type recognition, 795
Answer type taxonomy, 795
Antecedent, 708
Antonym, 628
Anttila, A.,135, 137, 139,418,
419, 428
Antworth, E.L.,55,64, 65, 67,
81,368
Aone, C.,733
AP,400

amplitude
RMS,234
21,41
419, 428
81,368
AP,400
Appelt, D. E.,726, 733,
766–768, 774
Apple
AIFF,233
appropriateness conditions for
types (seealso typed
feature structures), 527
approximant sound, 221
Arabic, 215
Egyptian, 241
Araki, M.,852, 871
Aramaic, 215
Archangeli, D.,369,371, 373
arcs in directed graphs, 27
Arens, R.,771
Arens, Y.,620
Argamon, S.,462, 775
argmax,140,291
Ariel, M.,712, 733

DRAFTIndex 977
Aristotle, 123, 575
Arity,570
Arons, B.,832,850
ARPA,334
ARPA formatfor N-grams, 108
ARPAbet, 245
article (part-of-speech), 127
articulatory phonetics, 218, 218
Articulatory phonology, 244
articulatory phonology, 244
articulatory synthesis, 284
artiﬁcial intelligence
founding of ﬁeld, 10
Artiles, J.,775
Aseltine, J.,775
Ashburner, M.,772
Asher, N.,734
Asher, R. E.,170
Aslin, R. N.,365, 382
Aspect,575
aspect
computational models of,585
aspirated sound, 226
ASR,287
conﬁdence, 342, 860
decoding
recombining hypotheses, 340
neural nets for, 354
ASR architecture
HMM-MLP hybrid, 354
assertive, 830
Assimilation, 226
assimilation, 226
association, 672
measure of for word
similarity, 672
probability asmeasure of, 673
Atal, B.S.,333, 355
ATIS,335, 390, 393, 396, 421,
437, 502
Atkins, S.,651
Atkinson, M.,828
ATN,seeaugmented transition
network
atomic types, 527
ATRANS,642
Attachment ambiguity, 436
Attar, R.,792
attribute grammars,619
attribute-value matrix, 494
Atwell, E.,113
Aubert, X.,340, 341

decoding
437, 502
network
augmentative communication,
12,84
augmented transition network,
531
Austin, J.L.,829, 870
Austin, S.,339, 343
Authorship attribution, 121
authorship attribution, 121automatic speech understanding,
287
Automaton, 27
automaton, ﬁnite state, see
ﬁnite-state automaton
auxiliary verb, 125, 126,128,
406
Ayers,G. M.,268
Aylett, M.P.,118, 119
Baayen, R. H.,79, 81, 126,158,
241, 487
Babbage, C.,249
Babko-Malaya, O.,688
Babyonyshev, M.,547
Bacchiani, M.,111
Bach, E.,428, 591,619, 734
Bach-Peters sentences, 734

12,84
531
287
406
241, 487
Bachenko, J.,264
backchannel, 831
backoff in N-gram smoothing,
104
backtrace
in minimum edit distance, 77
backup as solution to
non-determinism, 33
Backus, J. W.,10, 391,427
Backus-Naur Form, 391
Backward chaining, 568
Backward looking center, 718
Backward probability, 188
Bacon, F.,431
Badulescu, A.,682, 690
Baeza-Yates, R.,779, 786, 822
Bag generation, 121
bag-of-words, 653
features in WSD, 652
in IR,781
Bagley, W.C.,362
Bahl, L.R.,119, 139, 334, 339,
347, 354
Bailey, D.,585
Bailey, T.,387
Bake-off, 335

104
backtrace
in IR,781
347, 354
bake-off
speech recognition
competition, 335
Baker, C. F.,508, 509, 629, 637,
646
Baker, J.K.,119, 334, 471, 490
Bakis network, 178, 295
Bakis, R.,285
Balashek, S.,10, 333
Baldridge, J.,428,734
Baldwin, T.,491,532
Ball, C.A.,772
Balogh, J.,832, 840, 841, 846,
850, 860, 871
Banerjee, S.,668, 670,921
Bangalore, S.,491Banko, M.,170, 797, 800, 824
Bar-Hillel, Y.,417, 420, 549,
919
Barge-in, 850, 851
barge-in, 851
Barker, D.T.,846
Barney, H.L.,247
Baron, A.,359
Baroni, M.,387
Barrett, L.,477,490
Bartels, C.,364

bake-off
646
919
Barto, A.G.,863, 871
Barton, Jr., 81,550
Barzilay, R.,733, 809, 812–815,
823, 824
baseline
for part-of-speech tagging
evaluation, 155
mostfrequent sense, 657
take the ﬁrstsense, 657
Baseline frequency, 271
Bates, R.,84, 857, 858, 871
Batliner, A.,871
Bauer, F.L.,10,427
Bauer, L.,56
Baum, L.E.,104,187, 212, 334
Baum, L.F.,850
Baum-Welch algorithm, see
forward backward
algorithm
Baumgartner, Jr., 769,771
Bayer, S.,646
Bayes, T.,139, 290
Bayesian, 290
Bayesian inference, 139,164,
290
Bazell, C. E.,427

823, 824
baseline
algorithm
290
BBN, 334
BDI,864
beam search, 325, 326
in MT,907
Beam width, 326
beam width, 326
Bean, D.,733
bear pitch accent, 224,264
Bear, J.,766, 767, 774
Beaver, D.,266
Becker., 620
Becket, R., 837
Beckman, M.E.,267, 268, 285
Beebe-Center, J.G.,908, 910,
920
Beeferman, D.,699, 700
Beesley, K.R.,70, 81,387
belief
logic of, 865
Bell, A.,119, 229, 358
Bell, T.C.,120, 482
Bellegarda, J.R.,113, 284, 362
Bellman equation, 862

DRAFT978 Index
Bellman, R.,74, 863
Bender, E.M.,428
Bennett, C.,285
Bennett, S.W.,733
Benoˆ ıt, C.,282
Benson, D. B.,619
Berck, P.,170
Berger, A.,13,112, 207, 213,
699, 700
Bergler, S.,645
Bergsma, S.,718, 733
Berkeley Restaurant Project, 89
Bernstein, J.,334
Bertram, R.,79
Berwick, R. C.,81,550
Bestgen, Y.,733
Beutnagel, M.,284
Bever, T.G.,424, 487
Bhatt, R.,387
Bhembe, D.,827
Biber, D.,428
Bickel, B.,878
Biddulph, R., 10,333
Bies, A.,408, 410
Bigham, J.,679
Bigram,88
bigram,88
Bikel, D. M.,482, 491, 775
bilabial,220
Bilmes, J.,213, 364
Bimbot, F.,284
Binary branching, 416
binary branching, 416
binary tree, 416
binary vectors
for word similarity, 676
Binding theory, 714
binyan, 370
Birch, A.,920
Bird, S.,16, 387
Birnbaum, L.,620
Bisani, M.,284
Bitext,899
bits
for measuring entropy, 114
Black, A.W.,254, 255,258,
261, 264, 271, 272,
279–281, 283, 285
Black, E.,483,484, 490, 688
Black, H.A.,80
Blair, C. R.,170
Blair-Goldensohn, S.,818,823
Blake, J. A.,772
Blaschke, C.,772, 774
Bledsoe, W.W.,11, 290

bits
bleeding, 367
Blei, D.M.,13
BLEU
compared to ROUGE,819,
820Bleu, 909
Blevins, J.,377
Blitzer, J.,773
Blizzard Challenge, 285
blocks world, 11
Bloedorn, E.,809,823
Bloomﬁeld, L.,413, 426
Blunsom, P.,491, 532
BNC,94, seeBritish National
Corpus, 157
BNF (Backus-Naur Form), 391
Bobrow, D.G.,587, 825, 836
Bobrow, R.J.,532, 837, 839
Bock, K.,424
Bod, R.,490
Body of STRIPS plan, 865
Boersma, P.,236, 244, 247, 375,
385
Boguraev, B.,645, 733
Bohus, D.,846
Bojar, O.,920
Boland, J.,852
Bolinger, D.,224,265
Bonaparte, N.,249

BLEU
385
BOOK-FLIGHT,866
boostrapping
in IE,751
Booth, T.L.,464,465, 490
bootstrapping, 662
generating seeds, 662
bootstrapping for WSD, 662
Borst, J.M.,284
Bos,E.,733
Boser, B. E.,13
Botstein, D.,772
B¨ ottner, M.,827
bottom-up, 431, 433,460
Bouayad-Agha, N.,733
Bouillon, P.,836
Bound,711
bound
pronoun, 711
Boundary tone, 267
Bourassa, D.,377
Bourlard, H.,351, 354
Bouwman, G.,860
Boves, L.,860
Bowey, J.,377
Boye, J.,837
Brachman, R.J.,532, 587
Bracketed notation, 395
bracketed notation, 395

in IE,751
Bound,711
bound
Branco, A.,733
Brants, T.,120, 149–151,
159–162, 168,733
Brasoveanu, A.,387
Bratt, E.,852
Bratt, H.,871
Break index, 268breath
transcription of, 422
Breiman, L.,213
Brenier, J.M.,266
Brennan, S.E.,718, 731, 733,
735
Brent, M. R.,383,387
Bresch, E.,285
Bresnan, J.,11, 427,527, 529,
530, 532
Brew, C.,813
bridging inference, 712
Brill tagger, 135
Brill, E.,135, 151, 154,155,
159, 163,168, 170, 171,
343, 490,797, 800, 824
Brin, S.,680,799
Briscoe, T.,484, 491, 645
British National Corpus

735
530, 532
useof C5 tagset to tag, 130
broadcast news
discourse segmentation, 698
speech recognition of, 334
Brockett, C.,810, 823, 824
Bromberger, S.,388
Brookes, D. M.,276
Broschart, J.,125
Browman, C.P.,244, 284
Brown corpus, 85, 86, 94,130,
135, 152,155, 169
POStagset for, 130
Brown, J.S.,619
Brown, M.K.,844, 871
Brown, P.F.,13, 86,111, 118,
354, 690,894, 895, 904,
915–918, 920
Browning, I.,11,290
Bruce, R.,651, 689
Bryant, M.,94
Buckley, C.,791
Budanitsky, A.,171, 665,669
Budzikowska, M.,823
Bulut1, M.,285
Bulyko, I.,112,282, 359, 858

Bunescu, R. C.,750
Bunker, R.T.,652
Bunt, H.,854
Burger, J.D.,724, 733
Burges, C.,364
Burnett, D.,850
Burton, R. R.,619
Butler, H.,772
Butzberger, J.W.,339, 340
Bybee, J.L.,229
Byrd, D.,229
Byrd, R.J.,284
Byrne, W.,243, 343, 351, 920
cache language model, 112
Cafarella, M.,690, 754, 755

DRAFTIndex 979
Cahn, J.E.,285
Calder, J.,532
Caley, R.,272
Calhoun, S.,266
Callison-Burch, C.,912, 920
Campbell, I.,822
Campbell, N.,285
Canadian raising, 388
Candide, 920
Canon, S.,491, 532
Canonical form, 556
Cantonese, 877
Cao, X.,873
Caporaso, J.G.,771
Caraballo, S. A.,682
Caramazza, A.,730
Carberry, S.,151, 869
Carbonell, J.,646, 811,919, 920
Cardie, C.,460, 620, 717, 733,
774, 775
cardinal number, 399
Carletta, J.,713, 854, 856, 871
Carlson, L.,13, 703, 706, 734
Carpenter, B.,527–530, 532,826
Carpuat, M.,689
carrier phrase in TTS,275, 282
Carroll, G.,492
Carroll, J.,484, 491, 524, 532,
670, 688, 689
Carroll, J.B., 919,921
Carstensen, F.V., 139
Carter, D.,837
Carter, D.M.,363
Casari, G.,772
Cascade, 65
cascade, 26,367
transducer, 10,65, 65
case
as feature in IE,742
Case frame, 634
case sensitive
regular expression search as,
19
case, nominal, 404
Castano, J.,756, 759
Castao, J.,762
Cataphora, 711
categorial grammar, 420, 420,
428, 619
Cavedon, L.,871
cd,seeconceptual dependency
CD phone, 348

case
19
428, 619
Cedergren, H.J.,230
ceiling
human, 155
CELEX,81, 126, 241
cell on tape of ﬁnite-state
automaton, 27center-embedded, 542
center-embedding, 417, 536,
542, 546, 549
Centering
algorithm for information
ordering in
summarization, 813
Centering algorithm for pronoun
resolution, 720
psychological tests of, 731
Centering theory, 718
algorithm for pronoun
resolution, 718
model of entity-based
coherence, 718
centrality
of sentences in content
selection for
summarization, 805

ceiling
human, 155
Centering
Centroid, 823
cepstral coefﬁcients
uncorrelated, 303
cepstral distance
in TTS,280
cepstral meannormalization, 355
Cepstrum, 302
cepstrum, 297, 333
and covariance matrix, 303
delta feature, 304
formal deﬁnition, 303
in TTS,280
CFG,390, 391
multiplying probabilities, 489
Chafe, W.L.,733
Chan, Y. S.,689
Chandioux, J.,919
Chandler, S.,646
Chang, E.,357
Chang, J.S.,688
Chang, N.,81, 585
Chang, P.,81
channels in stored waveforms,
232
Chanod, J.,70
Charles, W.G.,669
Charniak parser, 477
Charniak, E.,16,118, 119, 156,

in TTS,280
in TTS,280
232
364, 413, 477,486,
490–492, 683,688, 726,
733, 823, 824
Chart,447
chart
in parsing, 447
Chart parsing, 452
chart parsing
fundamental rule, 453
Chawla, N.V.,364
Chelba, C.,468, 487
Chen, F.,733,823
Chen, J.,690Chen, J.N.,688
Chen, S.F.,89, 112, 120, 213,
254, 255,284
Chen, X.,243
Cherry, C.,914
Chi, Z.,491, 532, 550
Chiang, D.,491, 689, 913, 914
Chiba, S.,213, 333
Chierchia, G.,587
Chinchor, N.,774
Chinese, 215, 873,878, 880
POStagging, 160
word segmentation, 70
Chklovski, T.,688–690
Chodorow, M.S.,284, 665, 731

Chart,447
chart
Choi, F.Y.Y.,698, 733
Chomsky
adjunction, 416
Chomsky hierarchy, 536, 548
Chomsky normalform, 416,
426, 469
Chomsky, C.,586, 823
Chomsky, N.,9,10, 12,63, 83,
119, 228,285, 369, 389,
391, 416,417, 427, 531,
536, 538,544, 546, 547,
549, 714
Chomsky-adjunction, 416
Chou, W.,351, 919
Chow, Y.-L.,339, 348, 491
Christiansen, M.H.,387
Chronological ordering, 812
Chu-Carroll, J.,826, 844, 869,
871
Chung, G.,850
Chunking, 455
chunking, 455
Church, A.,565
Church, K.W.,12,81, 86, 100,
102, 103,113, 119, 120,
139, 155,164–167,

Chomsky
426, 469
549, 714
871
170–172, 257, 259, 380,
437, 462,549, 657, 663,
674, 920,931
CI phone, 348
Ciaramita, M.,682, 688
Cieri, C.,334
circumﬁx, 47
CIRCUS, 620,774
Citation form, 623
Citizen Kane, 693
CKY algorithm, 431, 461,463
probabilistic, 489
clariﬁcation subdialogue, 860
Clarissa,871
Clark, A.,170, 387, 492
Clark, E.V.,424
Clark, H.H.,85, 229, 424,713,
730, 830,832, 833, 854,
871, 872

DRAFT980 Index
Clark, J.,247
Clark, S.,491
Clarke, J.,823, 824
Clarkson, P.R.,109
clash (prosodic), 266
Class-based N-gram, 111
class-based N-gram, 111, 111
Classen, W.,733
Classiﬁcation, 199
logistic regression for, 199
classiﬁcation versus regression,
194
Clause,398
Cleft,713
click dueto bad joins in TTS,
276
click studies, 424
Clifton, C., 362
Clinton, W.,122
Clitic,48
clitic,48, 51
origin of term, 123
Cliticization, 48
cloning of states in acoustic
modeling, 350
closed class, 124
Closed vocabulary, 94
closed vocabulary, 97
closure, 40
stop,221
Clow, J.,287
cloze task
for evaluating translations,
909
Cluster N-gram, 111
clustering, 306
in word sense disambiguation,
689
K-means, 306
Clustering By Committee, 690
CMLE,seeconditional
maximum likelihood
estimation, 353
CMU.,225, 242, 258, 316
CNF,416
co-occurrence
ﬁrstorder, 686
second order, 686
Coarticulation, 273
coarticulation, 227
as motivation for triphone
acoustic model, 348
Coccaro, N.,113, 362, 857, 858,
871
Cochlea, 239
cochlea,239

stop,221
cloze task
909
689
CNF,416
871
Cocke, J.,13, 904,920
Cocke-Kasami-Younger
algorithmprobabilistic, 489
coda
syllable,223
codebook
in VQ,305
codeword
in VQ,305
Cohen, B. K.,773
Cohen, D. N.,213
Cohen, J.,364
Cohen, K. B.,769–771, 773
Cohen, M. H.,358, 832, 840,
841, 846, 850,860, 871
Cohen, M. M.,362
Cohen, P.R.,12, 156, 287, 825,
850, 864, 867,870, 871
Cohen, W.W.,813
Coherence, 695
coherence
different fromcohesion, 697
relations, 695,701
compendium of relations, 734
coherence conditions in LFG,
527
cohesion
different fromcoherence, 697
in discourse, 697

coda
codebook
in VQ,305
codeword
in VQ,305
coherence
527
cohesion
lexical, 697
Cohesion chain, 697
Coker, C.,284
Colby, K.M.,870
cold languages, 878
Cole, J.S.,369, 371
Cole, R. A.,9,362, 363, 850
Coleman, J.,247, 379, 387
COLING,15
collaborative completion, 831
collection
in IR,782
Collier, N.,773
Collins parser, 477
Collins, M.,264, 413–415, 419,
474, 475, 477–479, 481,
482, 484, 490–492, 683
collocation, 652
collocational features, 652
in WSD,652
collocations, 674
Colmerauer, A.,11,531
Colombe, J. B.,771
Colosimo, M. E.,771
Combinatory categorial
grammar, 421

COLING,15
in IR,782
in WSD,652
combinatory categorial grammar,
538
COMLEX,405
commissive, 830
common ground, 830
common noun, 125
complement, 405, 405, 426complementizers, 128
table of,128
Complete, 569
completeness
in FOL,569
complex onsets, 372
Complex question, 781
Complex types, 527
complex types, 527
Complex-term, 599
complexity, 535, 536, 546, 548
caused by memory
limitations, 549
center-embedding, 536, 546,
549
generative power, 548
Swiss German, 544
componential analysis, 641
composition
transducer, 58, 79,367
compound nominals, 608

538
in FOL,569
549
semantic attachments, 608
Compounding, 48
compression, 231
Computational Grammar Coder
(CGC), 169
computational lexical semantics,
649
computational phonology, 365,
365
computer-aided human
translation, 876
Comrie, B.,877
concatenation, 41
concatenation as basic FSA
operation, 40
concatenative morphology, 52
Concept accuracy, 851
conceptual dependency, 552, 642
concordance
semantic, 651
conditional independence, 489
Conditional maximumlikelihood
estimation, 200
conditional maximum likelihood
estimation, 352

(CGC), 169
649
365
Conditional Random Field, 214,
266
Condon, S.,854
conﬁdence
ASR,342, 860
in relation extraction, 754
confusion matrix
in noisy channel spelling, 166
confusion matrix for error
analysis,156
confusion network, 343
conjoined phrase, 407
conjugate gradient, 201
conjunctions, 126, 128,407

DRAFTIndex 981
table of, 128
Conkie, A.,275, 282, 284
Connective, 705
Connine, C. M.,362, 377
CoNNL,15
Connolly, D.,724, 733
Conrad, S.,428
Conrath, D.W.,668
Conroy, J. M.,810, 813, 823
Consistent, 465
consonant, 219
constant entropy rate hypothesis,
118
Constantin, A.,920
constants
in FOL,562
constituency, 389,417
evidence for, 391
constituent, 405, 425
book titles which arenot, 389
Constituent Likelihood
Automatic Word-tagging
System,seeCLAWS and
tagset
constraint
uniﬁcation features as, 493
Constraint Demotion, 385
Constraint Grammar, 135, 419,
428
constraint-based formalism, 493
Construction Grammar, 427, 620
content planner, 839
Content selection, 804
context-free, 537
context-free grammar, 395, 425
see CFG, 391
and Chomsky hierarchy, 537
Chomsky normalform, 416
ﬁnite-state approximations,
426, 428
invention of, 427
non-terminal symbol, 392
productions, 391
rules, 391
semantic attachments, 591
terminal symbol, 392
weak and strong equivalence,
416
context-free language, 425
context-sensitive grammar, 536,

428
426, 428
rules, 391
416
537
context-sensitive spell
correction, 167
Conti, J.,850
contingency table for error
Continuation rise, 267
Continuer, 831
continuer, 831
Continuous speech, 288continuous speech recognition,
288
contrastive knowledge
in MT,884
Contribution, 830
contribution, 830
Control,509
conventional metaphor, 644
conversation, 825
Conversation Analysis, 828,871
conversational act, 854
Conversational agent, 1
conversational agent, 8, 825
acknowledgement in, 832
conversational implicature, 834
conversational move, 854

288
in MT,884
Conversational speech, 288
Convex optimization, 201
Convolutional noise, 355
Cooley, J.W.,333
Cooper, F.S.,247, 284
coordinate noun phrase, 407
Coordination ambiguity, 436
coordination ambiguity, 474
Copestake, A.,529, 532,645
copula,128
C` ordoba, R.,842, 860
CoreLanguage Engine, 606, 619
Core, M.,854
Corefer,708
Coreference chain, 708
Coreference resolution, 708
coronal sound, 220
Corpora, 85
corpora,85
CORPORATIONAS PERSON
metaphor, 644
Corpus,85
corpus,85
ATIS,335,393
BNC, 94, 157
Broadcast news, 334

Corpus,85
corpus,85
Brown, 85, 86, 94,130, 135,
152, 155, 169
CASS phonetic of Mandarin,
243
ﬁsher, 334
Hansards, 899
Hong Kong Hansards, 899
Kiel of German, 243
LOB,169
MULTEXT-East,162
regular expression searching
insde,18
Resource Management, 334
Switchboard, 85,86, 225,229,
231, 232, 242,243, 297,
318, 335, 423
Switchboard phonetic, 243
TimeBank, 762TIMIT,243
WallStreet Journal, 97,334
Ziff-Davis, 807, 809
Corpus Lesk, 659
correction act detection, 857
Corston-Oliver, S.H.,734
Cosine,783
cosine, 822

243
ﬁsher, 334
LOB,169
insde,18
as asimilarity metric, 783
cosine similarity metric between
vectors, 677
Cote, S.,718, 733
Cotton, S.,13, 651, 652
Cottrell, G.W.,688
count noun, 125
counters, 41
counts
treating low as zero, 160
covariance matrix, 311
diagonal, 311
Cover, T.M.,113–115, 118
Cowan, B.,920
Cowie, J.,689
Cowper, E.A.,547
Cox, S.,332
Crammer, K.,491
Crawley, R. A.,730
Crestani, F.,822
CRF,214
Croft, W.,285, 877
Croft, W.B.,790
Cronin, B.,646
cross-brackets, 484
Cross-entropy, 116
cross-entropy, 116
cross-serial dependencies, 535,

counts
CRF,214
544
crossvalidation, 155
10-fold,155
Crouch, C. J.,792
Crouch, R.,484, 491, 532, 823
Cruse, D.A.,645
Crystal, D.,285
Csaba Pl´ eh, 547
Cuay´ ahuitl, H.,871
Cue phrase, 705
cue phrase
sentential use, 705
cue word
as feature for discourse
segmentation, 699
Culicover, P.W.,427
Cullingford, R.E.,11,919
Culotta, A.,750
Culy, C.,544,734
Cumming, S.,285
Cunningham, H.,766
Curran, J.R.,491,672, 674,675,
677, 679,689
Currying, 566

DRAFT982 Index
Cutler, A.,124, 363
Cutting, D.,150, 687
CV skeleton, 370
cycles in awave, 231
Cycles per second, 231
cycles per second, 231
Czech, 162
d
parameter in IBM Model 3,
914
Daelemans, W.,170, 284,460,
491, 532
Dagan, I.,462, 674, 677, 678,
689
Dahlb¨ ack, N.,854
Dale, R.,734, 840,859
Daly, N.A.,871
Damerau, F.J.,73, 165, 168,170
Damper, R. I.,284
DAMSL,854
Dang, H.T.,13,635, 651, 652,
656, 657, 688
Danieli, M.,845,851
Darroch, J.N.,201
data-directed search, 433
date
fully qualiﬁed, 759
date normalization, 837
Dative alternation, 635
Daum´ eIII, H.,809, 823,824
David, Jr.,333
Davidson, D.,572, 587
Davidsonian, 572
Davis, A.P.,772
Davis, E.,587
Davis, K.H.,10,333
Davis, S.,302
Day, D.,13, 762
Day, D.S.,733
deCourtenay, B.,247
decision list, 171, 653, 655
learning, 655
decision tree
use in WSD,688
declaration, 830
declarative, 396, 426
declarative constraint, 367
declarative sentence structure,
396
Declination, 271
Decoder, 184
decoder, 184
Decoding, 184,317
decoding, 145,184,290, 292,
317

396
317
A∗,seeA∗decoder
as search, 290
beam search, 325dynamic programming, see
Viterbi algorithm
MT,using IBM Model 1,897
multiple-pass, 339
N-best, 339
pruning in, 325
stack,seeA∗decoder
Viterbi,seeViterbi algorithm,
145, 184
word lattice, 340
deduction
as type of inference, 725
deep role, 633
Deerwester, S.,690
default pair, 61
Defaults, 529
defeasible inferences, 729
Deﬁnite Clause Grammars, 531
deﬁnite reference, 710
degree adverb, 126
Dejean, H.,13
DeJong, G.F.,646, 774, 823
Delattre, P.C.,247

317
145, 184
deduction
deleted interpolation, 150
deletion,227
Delfs, L.,13, 651, 652
Delgado, R.L.-C.,852,871
Deligne, S.,284
Della Pietra, S.A.,13, 86, 118,
201, 207, 213,894, 895,
904, 915–918, 920
Della Pietra, V. J.,13,86, 111,
118, 201, 207,213, 690,
894, 895, 904,915–918,
920
delta feature, 304, 304
Demberg, V.,256
Demetriou, G.,113
Demner-Fushman, D.,824
Dempster, A.P.,104, 167, 187
Denes, P.,333
Deng, L.,289, 348
Deng, Y.,920
Denotation, 558
dental sound, 220
dependency, 417
grammar, 418
lexical, 474
relations, 390

920
dependency grammar, 418
dependent-marking languages,
877
depth-ﬁrst search, 37, 41
Dereferencing, 513
Derivation, 48
derivation
compared to inﬂection, 51
productivity of in English, 51
syntactic, 392, 392, 395,396,
425direct,395
derivational, 79
Dermatas, E.,158
DeRose, S.J.,135,137, 139,150
Det, 392
determiner, 126, 127,392, 399,
426
deterministic algorithm, 28
deterministic ﬁnite-state
automaton, 32
development testset, 92,153
devset,92
devtest,153
De Jong,N.H.,79
de Lacalle, O. L.,688
de Marcken, C.,383

877
Det, 392
426
devset,92
De Mori, R.,112,362, 871
de Rijke, M.,760, 775
de Souza, P.V.,111, 347, 354,
690
de Villiers, J.H.,850
DFSA,32, 41
DFT,300
Diab, M.,688
Diagnostic RhymeTest, 282
diagonal covariance matrix, 311
dialect variation, 229
Dialogic pair, 828
Dialogue, 825
dialogue, 825
compared to monologue, 693
information-state model of,
853
dialogue act, 829, 854
acknowledgement, 831
correction, 857
dialogue acts
question detection, 857
dialogue manager, 841
design, 850

690
DFT,300
853
ﬁnite-state automaton, 841
frame-based, 843
dialogue structure
intentional, 868
Dialogue system, 1
dialogue systems
evaluation, 850
Diathesis alternation, 635
Dice similarity metric, 677
Dienes, P.,161, 162, 887
Dietterich, T.G.,156
diff program, 77
Digalakis, V. V.,339, 340
Digilakis, V.,364
digit pronunciation
paired method, 254
digitization, 232,297

DRAFTIndex 983
digits task
ASR,288
Dimitrova, L.,162
Dingare, S.,713
Diphone, 273
diphone synthesis
compared to unit-selection,
278
Diphthong, 222
diphthong
Direct translation, 881
directed graph as representation
of FSA,27
directional adverb, 126
directive, 830
Directive prompt, 844
directive prompt, 850
directly derive, 395
disambiguation, 67,135
PCFGs for, 465
period, 251
role of probabilistic parsing,
463
syntactic, 437
via PCFG, 467
word sense, 649
Discount, 99
discount, 99, 100
discounted reward, 862
Discounting, 99
discounting, 99
Discourse, 693
discourse
context
role in reference resolution,
708
model,708
purpose,868
segment, 702
Discourse marker, 705
discourse marker
Discourse parsing, 704
Discourse purpose, 868
Discourse segment purpose, 868
Discourse segmentation, 696
discourse segmentation
broadcast news, 698
discovery procedure, 427
Discrete Fourier Transform, 300
discriminative training
in ASR,351

discourse
context
708
model,708
in ASR,351
discriminative versus generative
models, 209
disﬂuency, 85,360,422,422
disﬂuent, 426Disjunction, 530
disjunction, 41
in feature structures, 530
pipe in regular expressions as,
21
square braces in regular
expression as,19
dispreferred response, 829, 872
Distal,711
Distance, 481
distance
edit,74
Euclidean, 307, 676
Mahalanobis, 307
Manhattan, 675
or distortion, in MT,892
Distance metric, 306
distance metric, 306
distinctive feature, 227
distortion metric, 306
distortion probability
in IBM Model 3, 915

21
distance
edit,74
in MT,891, 892
distributional similarity, 427
ditransitive, 425
ditransitive alternation, 425
Divay, M.,256
divergences between languages
in MT,876
Dixon, N.,283
DiEugenio, B.,733
document
in IR,782
Document Understanding
Conference, 824
Doddington, G.,335, 390, 825,
921
Dogil, G.,271
Doherty-Sneddon, G.,854, 856,
871
Dolan, W.B.,688, 824
Dolbey, A.,771
Dolinski, K.,772
Dolittle, Doctor, 27
Domain,558
dominance
discourse relation, 868
dominates, 393
domination in syntax, 392
Donovan, R.E.,281, 284

in MT,876
document
in IR,782
921
871
dominance
Dorr, B.,646, 810,823, 876, 921
Dostert, L.,919
Dotproduct, 196,784
dotproduct, 676
Dotted rule, 447
dotted rule, 447
double delta feature, 304
Doumpiotis, V.,343, 351
Dowding, J.,871
Downey, D.,690, 754,755Downing, P.,620
downstepped high in ToBI,272
Dowty, D. R.,428, 575,578,
587, 642
DP,868
Dragon Systems, 334
Dream of theRed Chamber, 873
Drew, P.,828
DRT,282
DSP,868
Dubner, S.J.,194
DUC, 824
duck
as example of ambiguity, 4
Duda, R.O.,213, 306,687
Dud´ ık, M.,213
Dumais, S.T.,679, 690, 797,
800, 824

587, 642
DP,868
DRT,282
DSP,868
DUC, 824
duck
800, 824
Dunn, H.K.,10, 247
Dunning, T.,805
duration
in TTS,277
Durbin, R.,213
Dutoit, T.,285
Du Bois, J.W.,285
Dwight, S.S.,772
Dyer, C.,920
dynamic programming, 74,146,
213, 431, 439, 461
forward algorithm as,181
invariant, 338
Viterbi as,184
Dynamic timewarping, 333
Dˇ zeroski, S.,162
E-step (expectation step) in EM,
192
E.E.,333
EACL,15
Eagon, J.A.,212, 334
Earley algorithm
integrating with semantic
analysis, 615
Earley algorithm, 431, 461, 518

800, 824
duration
in TTS,277
192
E.E.,333
EACL,15
Earley, J.,431, 447, 449, 462
Earnest
Importance of Being, 72
Eberlein, J.,773
Echihabi, A.,706,707, 799
Eckert, W.,861–863
Eddy, S.,213
edit cost
of post-editing MT,909
edit distance, 74
minimum, 74
edit term, 423
Edmonds, P.,689
Edmunson, H.,823

DRAFT984 Index
Effects of STRIPSplan, 865
EGG(electroglottograph), 276
EGYPT,920
Ehlen, P.,871
Eide, E.M.,281,285, 364
Eisner, J.,177, 387, 491,492,
914
Ejerhed, E.I.,462
Elaboration (as coherence
relation), 701
Electroglottograph, 276
Elhadad, M.,532, 809
Elhadad, N.,812
ELIZA,7
implementation, 26
sample conversation, 25
Ellis, D.,225, 243
Ellison, T.M.,374, 379, 387
Ellsworth, M.,637
Elman, J.L.,363
Elson, D.,810
Elvgren III,G.,380
EM,104, 167
E-step, 192
in training MEMM,211
M-step, 192
EMalgorithm, 187
Embedded training, 327
embedded training, 327, 355
embedded verb, 401
emission probabilities, 145,177,
293,296,318,327
EMNLP,15
Empiricist, 433
empiricist, 433
empty category, 398
enclitic,51
energy in frame, 304
EngCG,135, 137
Engel, R.,871
English, 878
amount of utterance overlap in
dialogue, 828
argument by JimMcCawley
that inﬁxes exist in, 47
derivational morphology of,
50
entropy of, 117
grammar of spoken, 421
habitual present form, 49
inﬂectional morphology of, 48
lexical differences from
French, 881

EMNLP,15
50
number of irregular verbs in,
49
simpliﬁed grammar rules, 393
Entity grid, 813
entropy,114
and branching factor in letter
prediction forboundaries, 382
and MDL,382
and perplexity, 114
of English, 117
of morphological paradigm,
79
relation to thermodynamics,
10
relative, 661, 678
Entropy rate, 115
entropy rate, 115
constancy hypothesis, 118
epoch (pitch pulse), 276
epoch detection, 276
Eppig, J.T.,772
ErgodicHMM, 178
Erjavec, T.,162
Erkan, G.,806, 809
errors
analyzing, 156

49
79
10
errors
Espy-Wilson, C., 364
Essen,U.,113
Estonian, 162
Etzioni, O.,690,754, 755
etal, 364, 919, 921
Euclidean distance, 306, 676
Eugene Onegin, 119, 173, 212
Euler, 300
Euler’s formula, 300
EUROSPEECH,16
EVALin Optimality Theory, 372
EVALB,484
evaluating parsers, 483
evaluation, 91
adequacy in MT,909
baseline, 155
mostfrequent class, 156
baselines in summarization,
821
Bleu in MT,909
ceiling, 155
cloze task, 909
comparing models, 97
coreference resolution, 724

Euler, 300
EVALB,484
821
development test set, 92, 153
devset, 92, 153
dialogue systems, 850
discourse segmentation using
Pk, 700
WindowDiff, 700
distributional word similarity,
679
error analysis, 156
extrinsic, 95,656
ﬁdelity in MT,909
ﬂuency in MT,909human ceiling, 155, 657
in vitro,656
in vivo,95
Kendall’s τ,819
Matched-Pair Sentence
Segment WordError
(MAPSSWE),156, 331
mean opinion score, 283
METEORfor MT,910
model-theoretic coreference
scoring scheme, 724
mostfrequent class baseline,
156
MT,908

Pk, 700
679
in vivo,95
156
MT,908
MUC coreference, 724
named entity recognition, 744
of N-gram, 91
of N-grams via perplexity, 95
of summarization, 819
precision and recall for MT,
910
pseudoword, 657
Pyramid Method in
summarization, 820
relation detection, 754
relevance feedback, 791
ROUGE for summarization,
819
test set, 91
training on the test set, 92
training set, 91
choosing, 153
TTS,282
unsupervised WSD,687
word similarity, 669
WSDsystems, 656
evaluaton
TERfor MT,910
Evans, N.,125, 126
Evans, R.,532
Event detection and

156
MT,908
910
819
TTS,282
evaluaton
classiﬁcation, 738,761
Event variable, 572
events
representation of,569
Evermann, G.,316, 343, 358
Evidence, 703
evoking areferent, 708
existential there, 129
expansion, 393, 397
Expectation step, 471
expectation step in EM,192
Expectation-Maximization, 187
Explanation (as coherence
Explicit conﬁrmation, 845
exponential classiﬁers, 193
expressive, 830
expressiveness

DRAFTIndex 985
of ameaning representation,
557
Extended Lesk, 668
extract
extraposition, 713
Extrinsic evaluation, 95
extrinsic evaluation, 656
eye-tracker, 487
F (for F-measure), 459, 483, 745
f∗,346
F0,233
F-measure, 459,483
in NER,744
F.-M.,387
Fackrell, J.,259
factoid question, 780
FAHQT,875, 875
fail state, 30
Fail type, 527
fail type, 527
FAITHC,372
faithfulness in MT,889
faithfulness in phonology, 373,
380
FAITHV,372
False negative, 23
False positive, 23
Fano, R. M.,674
Fanshel, D.,856
Fant, C. G.M.,284
Fant, G.,247
Fant, G.M.,247, 284, 361
Fanty, M.,850
Farrar, K.,285
Farsi, 878
Fass,D.,620, 646
FastFourier Transform, 301, 333
fast match in ASR decoding, 344
fast match, 345
FASTUS, 766
Fauconnier, G.,587
feasible pair, 61
feature
distinctive, 227
extraction of spectral, 292
morphological, 53
phonetic, 227
feature structure
AGREEMENT ,495
reentrant, 496
feature co-occurrence
restrictions, 527
Feature path, 495
feature structure, 404
NUMBER,494

Farsi, 878
feature
CAT,495
as DAGs, 511constraining values of
features, 527
DAG content ﬁeld, 512
DAG pointer ﬁeld, 512
description of versus instance
of, 530
disjunction in, 530
need for typing in,526
negation in, 530
path inequalities in, 530
reentrant, 498
set-valued features, 530
Feature structures, 494
Feature vector, 297
feature vector, 652
for distributional word
similarity, 670
in IR,782
in WSD,652
features
articulatory, 244
in linear regression, 195
phonetic, 244
Federalist papers, 11,121
Federico, M.,920
Fedorenko, E.,359

of, 530
in IR,782
in WSD,652
features
Feldman, J.,585
Feldman, L.B.,79
Fellbaum, C.,13, 629,645, 651,
652, 688
Feng, F.,766
Fensel, D.,584
Ferguson, G.,853, 859,864
Ferguson, J.,179
Ferguson, M.,408, 410
Ferreiros, J.,842, 860
Ferrer, L.,335, 871
Ferro, L.,13, 756, 759,762
fertility
in MT,914
FFT,301, 333
ﬁdelity
in MT,909
FIFO,37,41
Fikes, R. E.,865, 870
ﬁleformat
.wav,233
Filippova, K.,733
ﬁlled pause, 85, 423, 426
Filler,511
ﬁller,85,423
Fillmore, C.J.,12, 427,508,
509, 532, 587,620, 629,
633, 637, 645,646
Final fall, 267
ﬁnalfall, 267

652, 688
fertility
in MT,914
ﬁdelity
in MT,909
ﬁleformat
.wav,233
Final lowering, 856
ﬁnalstate, 27
Finegan, E.,428
Finin, T.,83,620ﬁnite automaton, 80
ﬁnite-state approximations to
CFGs,428
Finite-state automaton, 26
ﬁnite-state automaton, 17, 26,
27,27, 40, 41
ǫ-transitions in, 32
accepting by,27
agenda in non-deterministic
recognition, 33
concatenation, 40
converting non-deterministic
to deterministic, 38
deterministic, 32, 41
deterministic recognition, 29
deterministic recognition of,
28
failure in recognition of,29
ﬁnal state, 27

CFGs,428
28
for English money phrases, 31
for English nominal inﬂection,
54
intersection, 67
Kleene closure as basic
operation for combining,
40
Moore vs. Mealy machines,
80
non-deterministic, 32,41
non-deterministic recognition,
33,35
recognition by,27
rejection by, 28
search and non-deterministic
recognition, 36
sink state, 30
start state, 27
state-transition table for, 28
states in,27
union as basic operation for
combining, 40
weighted, 144,174, 378
ﬁnite-state morphology, 60
ﬁnite-state transducer, 46,57,
57,79, 386
and OT, 374

54
40
80
33,35
57,79, 386
and OT, 374
as generator, 57
as recognizer, 57
as setrelater, 57
as translator, 57
cascade of, 10,65
closure properties, 58
composition, 58
epsilon in, 58
machine learning of,380
non-determinism, 67
other, 64
regular relations, 58

DRAFT986 Index
two-level morphology, 65
union, 58
Fink, G.A.,228
FirstIn First Out, 37
ﬁrstorder co-occurrence, 686
FirstOrder Logic, seeFOL
ﬁrst-order Markov chain, 175
Firth, J.R., 670,689
Fischer, M.J.,74, 170,213
ﬁsh
plural of, 45
Fisher, D.,620, 766, 774, 775
Fisher, W.,334, 378
Fitt, S.,258
Fitzpatrick, E.,264
Flammia, G.,854
Flanagan, J.L.,284
ﬂap (phonetic), 222, 226
ﬂapping, 228
Flat start, 328
ﬂatstart,328
Flemming, E.,229
Flickinger, D.,483, 484, 491,
532
ﬂuency
in MT,909
ﬂuency in MT,889
Fodor, J.A.,424, 587, 646, 660
FOL,552, 561
∧(and), 563,567
¬(not), 563,567
∨(or), 567
and veriﬁability, 561
constants, 562
expressiveness of, 561
expressiveness of, 558
functions, 562
inference in, 561
terms, 562
variables, 562
fold (in crossvalidation), 155
Fong, C.,264
food in NLP
ice cream, 177
Forced alignment, 329
forced alignment
for TTS,275
Ford, C.,285
Ford, M.,547
Form-based, 843
formal language, 30,41, 395,
536
for soda machine, 31
Formant, 239

∨(or), 567
terms, 562
food in NLP
536
formant,239
formant synthesis, 284
Forney Jr.,213Forward algorithm, 181
forward algorithm, 74, 183
usein stack decoding, 344
Forward chaining, 568
Forward looking center, 718
forward trellis, 181
forward-backward algorithm,
187, 213, 310,343, 471
backward probability in, 188
for GMMs,314
forward-backward search, 343
FORWARD -BACKWARD
ALGORITHM ,192
FORWARD ALGORITHM ,183,
321
Fosler, E.,90, seeFosler-Lussier,
E.,487
Fosler-Lussier, E.,229, 358
Foster, D.W.,121, 122
Foster, G.,919
Fox,B.,285, 423
Fox,H.,839
Fox,L.,769

321
E.,487
Fox Tree,J.E.,85,229
Fraenkel, A.S.,792
fragment
transcription of, 421
word, 422, 423
fragment of word, 85
Frakes, W.B.,786, 822
Frame,299
frame
semantic, 629, 637
Frameelement, 637
Frameshift, 299
Framesize, 299
frame-and-slot semantics, 836
Frame-based, 843
frame-based acoustic model, 364
FrameNet, 629, 637
Frames,578
frames, 836
Francis, H. S.,472
Francis, W.N.,11, 85,94, 123,
130, 134, 135,169
Frank, E.,213
Frank, R.,374
Frankel, J.,364
Franz, A.,120, 156, 168,170,
491
Fraser, A.,920

fragment
Frame,299
frame
491
Fraser, N.M.,532, 850, 852
Freeword order, 419
Freitag, D.,159, 213, 765,775
French, 877
homographs, 256
frequency
fricative sound, 221
Friedl, J.E.F.,42Friedman, J.H.,213
Friedman, M. W.,718, 733, 735
Frisch, S.A.,387
Fromer, J.C.,845
Fromkin, V., 78
fronting of [uw], 216
Frump, 774
Fry, D.B.,247, 333
FSA,26,seeﬁnite-state
automaton
FST,57,seeﬁnite-state
as sequence classiﬁer, 173
projection of, 59
Fujisaki,285
Fujisaki, H.,285
Fukui, Y.,871
full listing hypothesis, 78

frequency
Frump, 774
automaton
Fully qualiﬁed date expressions,
759
fully-connected HMM, 178
function word, 125, 168
functional grammar, 428
functions
in FOL,562
Fundamental frequency, 233
fundamental frequency, 233
Fundamental rule, 453
fundamental rule
of chart parsing, 453
Funeral Elegy, 122
Fung, P.,243,689, 914
Furnas, G. W.,690
Furui, S.,824
fusion language, 877
g, 346
G. D.,213
G. E.,81, 550
Gabrilovich, E.,690
Gaizauskas, R.,13, 762, 766,
775
Gale, W.A.,81,86, 100, 102,
103, 119,120, 155,
164–167, 170, 171, 657,
663, 920,931

759
functions
in FOL,562
g, 346
G. D.,213
775
Gales, M.,316, 343,364
Galescu, L.,284
Galley, M.,823, 914
Galliers, J.R.,95
Gao, J.,113, 120
Gap list,511
gap-threading
and semantic analysis, 619
Garc´ ıa, P.,380
Garden-path, 487
garden-path sentences, 487,490
Garofolo, J.,852
Garrett, M.F.,78

DRAFTIndex 987
Garside, R.,94, 123,130, 136,
139, 169, 170
Garvey, C.,730
Gaston, J.,850
Gates, B., 823
Gaussian, 308
estimators for acoustic
likelihood, 308
mixtures, 313
prior on weights, 205
Gaussian Mixture Model, 308,
313
Gaussier, E.,384
Gaustad, T.,657
Gazdar, G.,16, 408, 527,532,
542, 546, 549, 619
gazetteer, 742
Gdaniec, C.,483,484
Ge, N.,733
Gee, J.P.,264
Geman, S.,491, 532
GENin Optimality Theory, 372
gender
marking on words, 52
Gender agreement, 404
gender agreement, 404
gender dependent acoustic
modeling, 356
Gene normalization, 771
Generalized semantic role, 635
generate, 393
generation
template-based, 840
Generative grammar, 395
generative grammar, 395, 426
relation toformallanguage, 31
Generative Lexicon, 645
generative power, 536, 548
generative syntax, 428
generator, 392
Generic summary, 802
generics, 713
genitive NP, 429
Gentner, D.,646
genus
in deﬁnition questions, 817
Genzel, D.,118, 119
Georgila, K.,871
Gerber, L.,756, 759
Gerbino, E.,845, 851
Gerdemann, D.,387, 532
German, 241, 877

genus
Germann, U.,920
Gershman, A.V., 620,919
Gerund,50
gerund,50,401
gerundive, 426
gerundive postmodiﬁer, 401
Gestural score, 244
gestural score, 244Gesture,244
gesture
Giangola, J.P.,832, 840, 841,
846, 850, 860,871
Gibbon, D.,285, 871
Gibson, E.,359, 546–548, 734
Gil, D.,125
Gilbert and Sullivan, 72, 287
Gilbert, G.N.,850
Gildea, D.,13,112, 229, 358,
380, 381, 491,585, 683,
684, 690, 914
Gillick, L.,332
Gilliom, L.A.,731
Gillis, S.,170
Girand, C.,229, 358
Girju, R.,682, 690
Gish,H.,364

Gerund,50
gesture
Givenness hierarchy, 712
Giv´ on, T.,472
GIZA++, 920
Glance, N.,775
Glass, J.,245, 358, 364, 851
Glennie, A.,461
Gloss,630
Glottal,221
glottal,221
Glottis,218
GMM,308
covariance matrix, 303
GMM(Gaussian Mixture
Model),313
goal-directed search, 433
Godfrey, J.,85,225, 334, 335,
390, 825
Godzilla, speaker as, 639
Gold Standard test set, 155
Gold, B.,247, 335
Goldberg, J.,858
Goldﬁnger, S.D.,363
Golding, A.R.,171
Goldman, R.,726
Goldsmith, J.,370, 383
Goldstein, J.,810, 811, 813,823
Goldstein, L.,244

Gloss,630
GMM,308
390, 825
Goldwater, S.,170, 377–379,
387
TheGondoliers, 72
Gonzalo, J.,775
Good, I.J.,101
Good, M.D.,850
Goodenough, J.B.,669
Goodine, D.,851
Goodman, J.,89,112, 120, 213,
491, 532
Goodman, K.,920
Goodwin, C.,856, 871
Gopalakrishnan, P.S.,347Gordon, D.,870
Gordon, P.C.,731
Gordon, S.A.,871
Gorin, A.L.,826
G¨ otz, T.,532
Gould, J.D.,850
Gould, S.J.,649
Government and Binding, 427
Grabe, E.,285
gradient ascent, 201
Gradual Learning Algorithm,
385
Graff, D.,334
Grammar
Constraint, 419

387
491, 532
385
Grammar
Construction, 427
Dependency, 418
Head-Driven PhraseStructure
(HPSG),413, 427
Lexical-Functional (LFG),
427
Link, 419
TreeAdjoining, 427
probabilistic, 491
grammar
categorial, 420, 420
CCG, 421, 538
checking, 431
combinatory categorial, 421,
538
equivalence, 415
generative, 395
Head,538
hierarchy, 536
inversion transduction, 912
Minimalist, 538
regular, 537
strong equivalence, 416
systemic, 407
TAG,538
Tree-Adjoining, 538
weak equivalence, 416
Grammar Rock, 123

427
Link, 419
grammar
538
Head,538
TAG,538
grammars
mildly context-sensitive, 538
grammatical relations, 390
grammatical sentences, 395
Granacki, J.,620
Granoien, N.,359
Grapheme-to-phoneme, 259
Graphone, 284
Gravano, L.,753, 799
Gray, R. M.,305
greedy REpatterns, 22
Greek, 215
Green, B.F.,586, 823
Green, J.,10,427
Greenbaum, S.,49, 127, 428

DRAFT988 Index
Greenberg, S.,225, 243
Greene, B.B., 137,169
greeting, 129
Grefenstette, G.,70,71, 81,677,
679, 690, 823
Gregory, M.L.,119, 229, 266,
358, 472
Greiff, W.,688
Grenager, T.,691
grep, 17, 18,42
Grice, H.P.,834, 866
Grice, M.,282
Gricean maxims,834
Gries, S.T.,646
Grifﬁths, T.L.,170, 387
Grishman, R.,359, 364, 405,
483, 484, 508, 510, 766
Grober, E.,730
Groliers, 680
Grosjean, F.,264, 362
Grosjean, L.,264
Grosz, B.J.,12, 16,695, 718,
731–733, 868–871
grounding
ﬁve kinds of,831
grounding contributions, 830
Grover, C.,743, 809
Gruber, J.S.,633, 645
Grudin, J.T.,166
Gu,L.,357
Gulikers, L.,81,126, 241
Gundel, J.K.,712,733
Gupta, V.,120, 347
Gusﬁeld, D.,77, 81
Gustafson, K.,284
Guthrie, J.A.,689
Guthrie, L.M.,688, 689
Guy,G.R., 229
Guyon, I.M.,13
h∗,346
H* pitch accent, 268
Habash, N.,81
Hachey, B.,809
Hacioglu, K.,690
Haeb-Umbach, R., 347
Hafer, M.A.,382
Haghighi, A.,492
Hahn, U.,387, 733, 823
H¨ aiki¨ o, T.,79
Hain, T.,316, 343,358, 364
Hajiˇ c, J.,13,161–163, 419, 484,
491, 492

Gu,L.,357
h∗,346
491, 492
Hakkani-T¨ ur, D.,360, 364
Hakkani-T¨ ur, D.,161, 162, 359,
364
Hakwing, S.,249
HAL9000, 1
Hale, J.,487, 733
Hall, J.,491Hall, K.,486
Hall, R.,78
Halle, M.,63,228, 285, 369,
388, 536
Halliday, M. A.K.,407,697,
733, 734
Hamada, H.,283
Hamilton, A.,121
Hamming, 299
Hammond,M.,374
Hamza, W.,285
Hanauer, S.,333
Hankamer, J.,80
Hanks, P.,13, 674,762
Hansards corpus, 899
Hansen, B.,850
hapax legomena, 101, 158
hapax legomenon, 101, 158
Hapsburg, M.T.,249
Harabagiu, S.,690, 795, 823
hard decisions
avoiding them, 197

364
HAL9000, 1
388, 536
733, 734
Harman, D.K.,790
harmonic, 241
Harmonic Grammar, 387
and log-linear models, 387
harmonic mean, 459,483
harmony, 369,387
in Harmonic Grammar, 387
Harper, M.P.,150, 159, 364
Harris, C.M.,283
Harris, R.A.,828, 850, 871
Harris, Z.S.,10, 137, 156, 169,
382, 387, 462,672, 689
Hart, P.E.,213, 306, 687
Hart, T.,113, 120
Harvey, T.,151
Hasan, R.,697, 733, 734
Hasegawa, M.,700, 734
Hasegawa-Johnson, M.,364
Hastie, T.,213
Haviland, S.E.,713
Hayes, B.,375, 385, 387
Hayes, E.,646
Hays,D.,418
Hazan, V.,282
Hazen, T.,351
head, 413, 477
tag,477

tag,477
Head features, 505
Head Grammar, 538
head nouns, 426
Head tag, 477
Head-Driven Phrase Structure
Grammar (HPSG),413,
427
head-marking languages, 877
Hearst, M. A.,112, 488, 657,
662, 680, 688,697, 699,
700, 733, 771,772Hebrew, 215
Hecker, M.H. L.,282
HECTORcorpus, 651
Hedberg, N.,712, 733
Heeman, P.A.,120,364
Hehmsoth, C.,282
Heikkil¨ a, J.,137, 138
Heikkil¨ a, J.,135, 139, 418, 419,
428
Heim, I.,566, 587
Heinz, J.M.,247
Held-out, 104
held-out data, 92
Helmreich, S.,646
Hemphill, C.T.,335,390, 825

tag,477
427
428
Henderson, J.,491, 871
Hendler, J.A.,584
Hendrickson, C.,156
Henrix, J.,225
Hepple, M.,762, 775
Herbst, E.,920
Hermansky, H.,361
Hermjakob, U.,795,799
Hernon, W.P.,78
Herold, K.,358
Hertz as unitof measure, 231
Herzog, M.,359
Hickey, M.,84
hidden events
in HMM,144
Hidden Markov Model, see
HMM,177
Hidden Understanding Model,
839
Hieronymus, J.L.,871
Higgins, H.,247
Higuchi, N.,285
Hilf, F.D.,870
Hill, D.P.,772
Hill, E.A.,387
Hillard, D.,359,364
Hindi, 877, 878
Hindle, D.,423, 483, 484, 490,
672, 674,675, 689

in HMM,144
HMM,177
839
Hinkelman, E.A.,620, 854,855,
871
Hintikka, J.,587, 865
Hirschberg, J.,264, 265, 267,
268, 285,364, 705, 734,
810, 854,857–860, 869
Hirschman, L.,724,771, 774,
850–852
Hirst, G.,171, 660, 665, 669,
688, 689,733
Hitzeman, J.,733
Hjelmslev, L.,646
Hladk´ a, B., 162
HMM,144,177,289
accepting states, 178
alignment, for MT,897

DRAFTIndex 989
cloning,350
ergodic,178
for alignment in
for speech synthesis, 285
formal deﬁnition of, 144, 177,
293
fully-connected, 178
history in speech recognition,
334
illustrative example of, 294
initial distribution, 178
multiple observations for a
single hidden state, 839
observation likelihood, 145,
177, 293, 296, 318, 327
observations, 145, 177,293
part-of-speech tagger, 135
phone models, 296
semi-HMM, 838, 839
states, 145, 177, 293, 296,
318, 327
tied states, 348
transition probabilities, 145,
trigram POStagging, 149
HMM part-of-speech tagging
simplifying assumptions, 140
HMM state, 296
HMM synthesis, 285
HMM tagger, 135
HMM-MLPhybrid, 354
Hoang, H.,920
Hobbs algorithm, 716
Hobbs tree search algorithm for
pronoun resolution, 716,
717
Hobbs, J.R.,12,619, 646, 701,
716, 726, 733–735, 766,
767, 774
Hockenmaier, J.,491
Hockey, B.A.,836, 839, 871
Hofmann, T.,112
Hofstadter, D.R.,173
Hollenback, J.,225, 243
Holliman, E.,85,225, 334

717
767, 774
Holmes, D.I.,122
Holonym, 629
Homograph, 626
homonym, 624
Homonymy, 624
homonymy
and IR, 790
Homophone, 626
Hon, H.-W.,247,285, 306, 326,
335
Honal, M.,364Hong Kong Hansards corpus,
899
Hopcroft, J.E.,38, 39,42, 81,
395, 538, 539,541, 549
Hopely, P.,10,169
Hopkins, M.,914
Hori, C.,824
Horning, J.J.,492
hotlanguages, 878
hourglass metaphor for TTS, 250
House, A.S.,247,282
Householder, F.W.,170
Hovanyecz, T.,850
Hovy,E.,795,805
Hovy,E.H.,688, 734,799, 809,
819
Howes, D.,362
Huang, C.,357
Huang, E.-F.,339

homonymy
and IR, 790
335
899
819
Huang, L.,491
Huang, X.,247, 285,289, 306,
326, 335
Huang, Z.,359, 364
Huddleston, R.,396, 428
Hudson, R.A.,428, 532
Huffman, D.A.,80
Huffman, S.,775
Hukari, T.E.,527
Huls,C., 733
Hulteen, E.A.,832, 850
HUM,839
human
sentence processing, 487
human parsing, 487
Human sentence processing, 487
human sentence processing, 487
Humphreys, K.,766
Hundreds digits, 256
Hungarian, 162, 877
part-of-speech tagging, 160
Hunicke-Smith, K.,357
Hunnicut, M.S.,260, 284, 285
Hunt, A.J.,271,279–281, 283
Hunter, J.,734

326, 335
HUM,839
human
Hunter, L.,769–773
Hutchins, J.,919, 921
Hutchins, W.J.,880, 881, 883,
887, 919, 921
Huybregts, R.,544
Hyperarticulation, 858
Hypernym, 628
hypernym
and information content, 667
and WordNetsupersenses, 682
as lexical cohesion, 697
bootstrapping, 681
extending WordNet, 681
in Extended Lesk,669
induction of relation, 680lexico-syntactic patterns for,
680
hyperplane
logistic regression as,200
Hypoarticulation, 226
hypoarticulation, 226
Hyponym, 628
hyponym
induction of relation, 680

hypernym
680
hyponym
hyppnym
Hz asunit of measure, 231
IBM, 119
IBM clustering, 111
IBM ThomasJ.Watson
Research Center, 11, 12,
119, 334
ICASSP,16
ichi Tsujii, J.,491
ICSLP,16
Ide, N. M.,162,652, 689
IDF,659, 785,785, 822
IDF term weighting, 785
idioms, 553,617
and compositional semantics,
617
if then reasoning
in FOL,567
Iida, M.,718, 733
III, J.T.M.,484, 491,532
Ilhan, H.T.,899
illocutionary act, 829
Illocutionary force, 829
illocutionary force, 829
Imai, S.,285

IBM, 119
119, 334
ICASSP,16
ICSLP,16
617
in FOL,567
immediately dominates, 393
imperative, 396, 426
imperative sentence structure,
396
imperatives
semantic attachments, 606
Implicature, 834
implicature, 834
Implicit conﬁrmation, 845
in vitro evaluation, 656
in vivo,656
in vivo evaluation, 95,656
indeﬁnite article, 399
indeﬁnite reference, 710
Indicator function, 202
indirect object, 425
indirect speech act, 856
Inference, 199,557
inference

DRAFT990 Index
bridging, 712
in FOL,567
inferrable referent and bridging
inferences, 712
inﬁnite employment of ﬁnite
means, 30
inﬁnitives, 390, 405, 426
inﬁx, 47
Inﬂection, 48
inﬂectional, 79
INFORM, 866
information content word
similarity, 666
Information extraction, 737
information extraction, 455, 461
Information retrieval, 781
information retrieval, 79
Information status, 712
Information structure, 712
information theory, 113
information-state architecture,
853
informativeness
in MT,909
Ingria, R.,483, 484, 756,759,
762, 837, 839
Initiative, 842
initiative, 871
mixed, 843
single, 842
system, 842
inner ear, 239
inner product, 676
input form,45
Inside-outside, 471
inside-outside algorithm, 490
Instance checking, 583
intelligibility
for TTSevaluation, 282
intensity of sound, 234
Intentional structure, 868
intentional structure, 868
interjection, 129
Interlingua, 887
interlingua, 887
intermediate phrase, 263
in ToBI,268
intermediate semantic
representations, 555
intermediate tape, 62
internal rule

mixed, 843
in aCFG parse, 478
International Phonetic Alphabet,
216, 245
International Space Station, 8,
871
Interpolated Kneser-Ney, 110
Interpolated Kneser-Ney
discounting, 109
Interpolated precision, 788interpolation in N-gram
smoothing, 104
Interpretation, 559
Interruption point, 361
interruption point, 423
Intersection, 903
intersection
transducer, 79
Intersective semantics, 610
Intonation phrase, 263
Intonation unit, 285
intransitive, 426
intransitive verbs, 405
Intrinsic evaluation, 95
Inverse document frequency, 785

216, 245
871
inverse document frequency,
659,seeIDF
inversion
of transducer, 58
Inversion transduction grammar,
912
IOB tagging, 457
Iolanthe, 72
IPA,216, 245
IR,seeinformation retrieval
and WSD,790
feature vector, 782
probabilistic model, 822
term, 782
term frequency, 785
term weighting, 782, 785, 822
vector space model, 782
Irons,E.T.,461
irregular verb, 49
Isard, A.,854, 856, 871
Isard, S.,275, 854, 856, 871
Ishizaka, K.,284
Ishizaki, M.,854
ISO8601, 758
Isolated word, 288
isolated-word recognition, 288

inversion
912
term, 782
isolating langauge, 877
Israel, D.,766–768, 774
Issar, S.,837
Issel-Tarver, L.,772
Itakura, F.,333
iterative scaling, 201
ITG,seeinversion transduction
grammar
Iverson, E.,646
Iwahashi, N.,283
Iyer, R. M.,111
Jaccard similarity metric, 677
Jaccard, P.,677
Jackendoff, R.,427, 529, 575,
646
Jackson, P.,746
Jacobs, P.,532Jacobs, P.S.,774, 823
Jacobson, N.,156
Jacquemin, C.,384
Jahr, M.,920
Jakimik, J.,363
Jakobson, R.,531
Janssen, T.M. V.,619
Japanese, 215, 241,877, 878,
880
Jarosz, G.,387
J¨ arvinen, T.,418, 419

grammar
646
880
Jasperson, R.,423
Jay, J.,121
Jefferson, G.,828, 832, 856, 870
Jeffreys, H.,98, 100,119
Jekat, S.,855
Jelinek, F.,13, 104, 119, 120,
150, 170,213, 334, 335,
339, 345,346, 364, 468,
483, 484,487, 490, 904,
920
Jenson-Shannon divergence, 678
Ji, H.,359, 364
Jiang, J.J.,668
Jiang-Conrath distance, 668
Jianping, Z.,358
Jilka, M.,271
Jim´ enez, V. M.,491
Jing, H.,809, 810, 823
Jitter,361
Johansson, S.,428
Johnson, C.,508–510
Johnson, C. D.,80,365, 386,
539
Johnson, C. R.,637
Johnson, H. L.,771
Johnson, K.,247

920
539
Johnson, M.,364, 377–380, 387,
428, 475,484, 486, 491,
530, 532,550, 575, 644,
646, 682,688
Johnson, S. C.,619
Johnson, W.E.,98,100, 119
Johnson-Laird, P.N.,629
Johnston, M.,287, 871
Join cost, 279
joint intentions, 871
joint probability, 466
Jones, D.A.,359
Jones, M. A.,646
Jones, M. P.,171
Jones, R.,753
Jones, S.J.,850
Joos, M.,689
Jordan, M.I.,13
Jordan, P.W.,827
Joshi, A.K.,10, 13,169, 427,
490, 491,538, 695, 718,
733, 734
Joslyn, C., 773

DRAFTIndex 991
Joyce, J.,84
Juang, B. H.,11, 213, 335, 351,
919
jump width
in MT,898
Jun,S.-A.,268
Juneja, A.,364
Junqua, J.C., 355
Juola, P.,550
Jurafsky, D.,81,90, 113, 119,
160, 229, 266, 357, 358,
362, 380–384, 487, 491,
525, 681–684, 688, 690,
857, 858, 871
Just, M.A.,547
K-means clustering, 306
Kaalep, H.J.,162
Kager, R.,388
Kahn, D.,378
Kaiki, N.,283
Kajarekar, S.,335
Kameyama, M.,733,766, 767,
774
Kamm,C. A.,850–852, 860
Kamm,T.,243, 364
Kan, M.Y.,733
Kanthak, S.,364
Kantrowitz, M.,811
Kaplan, R. M.,11, 12, 39,67,
81, 365, 368, 431,452,
484, 491, 529, 530, 532,
732, 823, 825, 836
Karamanis, N.,733, 813,823
Karger, D.R.,687
Karlsson, F.,135, 137,139, 418,
419, 428, 543, 547
Karttunen, L.,10, 70,81, 169,
370, 374, 375, 386, 387,
524
Kasami, T.,431,461
Kasarskis, A.,772
Kashyap, R. L.,170
Kasowski, S.,247, 284
Kasper, R.T.,501,530
Kataja, L.,387
Katz backoff, 105
details, 106
Katz, C.,10,427
Katz, G.,756, 759, 762, 775
Katz, J.J.,587, 646, 660
Katz, K.,408, 410
Katz, S.M.,103,105
Kawahara, T.,700, 734

524
Kawamoto, A.H.,688
Kay, M.,11, 12, 39,67, 81, 365,
368, 371, 387, 431, 452,
461, 496, 524, 527, 530,
531, 825, 836, 920
Kay, P.,427, 532, 620Kazakov, D.,383
Kazemzadeh, A.,285
Keane, M.T.,646
Kearns, M.,844, 860, 863
Keating, P.A.,229
Kehler, A.,733, 734, 767
Keller, F.,112, 119
Kelley, L.G.,919
Kelly, E.F.,688, 689
Kennedy, C.,733
Kernighan, M.D.,164–167, 170
Kershaw, D.,316, 343
keyword in IR, 68
Khudanpur, S.,357
Kibble, R.,709, 823
Kiefer, B.,524, 532
Kießling, A.,871
Kilgarriff, A.,13, 651, 656, 658,

659
Kim,D.,364
Kim,G.,408, 410
Kim,J.,364
Kimball, O.,348, 364
King, J.,547
King, P.,532
King, R.C.,118
King, S.,364,871
King, T.H.,484, 491, 532, 823
Kingsbury, P.,13
Kinoshita, S.,771
Kintsch, W.,11, 587
Kipp, M.,871
Kipper, K.,635
Kiraz, G. A.,81, 378, 379, 387
Kirchhoff, K.,228, 858
Kirsch, H.,772
Kisseberth, C. W.,369, 372
Kita, K.,871
Kitade, T.,700, 734
Kitamura, T.,285
Kittay, E.,646
KLdivergence, 678
KL-ONE,587
Klatt formant synthesizer, 284
Klatt, D.H.,260, 270,284, 285,
334

659
334
Klavans, J.L.,483, 484, 645,
733
Kleene *, 20
sneakiness of matching zero
things, 20
Kleene +, 20
Kleene, S.C.,9, 18,42
Klein, A.,855
Klein, D.,156, 160, 170,
475–477, 490–492
Klein, E.,408, 527, 619
Klein, J.,122
Klein, S.,137, 169Kleinman, D.,730
Klesen, M.,871
Klimt, B.,250
Klovstad, J.W.,347
Kneser, R.,110, 113, 120
Knight, K.,532, 813, 823,824,
883, 886,901, 907, 912,
914, 916–918, 920–922
Knill, K.,335
Knott, A.,734, 823
Knowledge base, 554
knowledge base, 555
Knuth, D.E.,82, 170, 619

733
things, 20
Kobayashi, T.,285
Koehn, P.,264, 891,903–906,
908, 912,920
Koeling, R.,670, 689
Koenig, W.,10, 247
Koerner, E.F.K.,170
Kogan, Y.,773
Kogure, K.,524
Kokkinakis, G.,158
Koller, A.,813, 823
Koller, D.,491
Kompe, R.,871
Koo, T.,491
Koopmans-van Beinum, F.J.,
118
Koppel, M.,775
Korean, 241
Kornai, A.,387
Koskenniemi, K.,61,81, 365,
367, 387
Kostic, A.,487
Kothari, A.,266
Kowtko, J.C.,854, 856, 871
Kozima, H.,733
Krallinger, M.,772
Krasnwer, M.,348
Kratzer, A.,566, 587
Krauthammer, M.,773
Krbec, P.,163

118
367, 387
Krieger, H.-U.,524, 529, 532
KRL, 587
Krogh, A.,213
Krovetz, R.,68,664, 790
Kruskal, J.B.,102, 146, 212
Krymolowski, Y., 462
Kryter, K. D.,282
Kuˇ cera, H.,11, 85,86, 94, 123,
130, 134,135, 169
Kudo, T.,491
Kuhl, P.K.,387
Kuhn, J.,532
Kuhn, R.,112, 362
Kuhn, S.,733
Kuhn, T.,871
Kukich, K.,72, 84,167, 170
Kullback, S.,661, 678

DRAFT992 Index
Kullback-Leibler divergence,
661
Kumar, S.,254, 255,343, 920
Kuno, S.,461
Kupiec, J.,150, 156,170, 823
Kvˇ etoˇ n, P.,163
L*pitch accent, 268
L+H* pitch accent, 268
L.,10, 247
L1norm, 675
L2norm, 676
labeled precision, 483
labeled recall, 483
labial place of articulation, 220
labiodental consonants, 220
Labov, W.,216, 227, 229, 230,
856
Lacy, 10,247
Ladd, D.R.,263, 265, 285
Ladefoged, P.,222,223, 242,
247, 377
Ladusaw, W.A.,247
Lafferty, J.D.,13, 170, 201, 213,
214, 490, 491, 699, 700,
775, 920
Laham, D.,679
Lai, J.C.,86, 111,118, 690,
904, 920
Laird, N.M.,104, 167, 187
Lakoff, G.,370, 575, 585, 587,
642, 644, 646, 870
Lambdanotation, 565
Lambek, J.,428
Lambrecht, K.,733
Lancaster UCREL,130
Landauer, T.K.,679, 690, 850
Landes, S.,652
Lane, H.,264
Langendoen, D.T.,285, 428
Langer, S.,84
Langlais, P.,659
Langsam, Y.,428
language
formal, 41
of certain sheep, 27
regular, 38
Language idenntiﬁcation, 335
language identiﬁcation, 335
Language model, 292
language model, 83,292, 463
adaptation, 111

904, 920
language
formal, 41
for MT,890
PCFG, 467
language model scaling factor,
317
language universal, 877
Lapalme, G.,659
Lapata, M.,112, 705,733, 813,
814, 819, 823, 824Laplace smoothing, 98
Lappin, S.,733
lapse (prosodic), 266
Large, N.R.,387
Lari, K.,471, 492
Larsson,S.,853, 871
Laryngograph, 276
larynx,218
Lascarides, A.,529, 707,734
LastIn FirstOut, 37
latent semantic analysis, 384,
690
for N-gram smoothing, 113
for spelling error correction,
171
latent semantic indexing, 690
lateral sound, 222

for MT,890
PCFG, 467
317
690
171
Lathrop, B.,871
lattice,340
pinched,343
lattice density, 341
lattice error rate, 341
Lau,R.,378, 379
Lauer, M.,620
Laughery, K.,586, 823
Lavie, A.,921
Lawrence, W.,284
Lazo, M.,13,762
LaPolla, R.,428
LCS,666
LDC,471
LDC.,97, 242, 334
Le,A.,852
Leacock, C.,651, 652, 665
leading sentences baseline in
summarization, 821
leap year, 287
Lease, M.,364
Leass,H.,733
Lede,696
Lee, C.-H.,351, 837, 838, 919
Lee, H.B.,688
Lee, K.-F.,357
Lee, K.-S.,284
Lee, L.,364, 678, 689,690, 813,
824
Lee, S.,285, 852

LCS,666
LDC,471
Le,A.,852
Lede,696
824
Leech, G.,49, 94,123, 127, 130,
136, 170, 428
Lees,R.,620
left-linear, 537
left-linear grammar, 537
Legendre, G.,387
Leggetter, C.J.,341, 356
Lehiste, I.,247
Lehnert, W.G.,11, 620, 733,
766, 774, 775,823
Lehrer, A.,646
Leibler, R. A.,661, 678Lemas, M.,822
Lemma,623
lemma, 85
versus wordform, 85
Lemmatization, 46,623
Lemon, O.,871
lenient composition, 374
Lennig, M.,120, 347,348
Lenzo, K.,258, 261
Leon, E.A.,772
Lerner, A.J.,47
Lesk
Corpus,659
Extended, 668
Simpliﬁed, 658
Lesk algorithm, 658
Lesk, M.E.,619, 659, 688

Lemma,623
lemma, 85
Lesk
letter-to-sound rules, 260
Leung, H.,851
Levelt, W.J.M.,361, 422, 424
Levenshtein distance, 74, 675
Levenshtein, V.I.,74
Levesque, H.J.,587, 871
Levi, J.,620
Levin, B.,425, 635, 645
Levin, E.,837, 838, 861–863
Levin, L.,854
Levine, R. D.,527
Levinson, S.C.,828
Levinson, S.E.,335
Levitt, S.D.,194
Levow, G.-A.,829,832, 846,
850, 858
Levy, R.,487
Lewin, I.,837
Lewis, C.,850
Lewis, D.L.,774
Lewis, H.,38, 39,42, 81, 538,
539
Lewis, S.,772
Lexeme,623
lexical
category, 392
cohesion, 697
dependency, 464, 474
gap,880
head, 490
rule,529

850, 858
539
lexical
gap,880
head, 490
rule,529
stress,224
tags,seepart-of-speech
tape,60
Lexical access, 361
lexical ambiguity resolution, 688
lexical database, 629
Lexical dependency, 474
Lexical gap, 880
lexical level in ﬁnite-state
morphology, 60
lexical rule

DRAFTIndex 993
to capture lexical
generalizations, 529
lexical sample, 651
lexical sampletask in WSD, 651
lexical semantics, 623
Lexical stress, 224
Lexical tape, 60
lexical trigger
in IE,756
Lexical-Functional Grammar,
550
Lexical-Functional Grammar
(LFG),427
Lexicalized grammar, 477
lexico-syntactic pattern, 680
lexicon,391
as list of lexemes, 623
deﬁnition of,54
tree-structured, 347
use ofin morphological
parsing,53
LFG,11
Li,A.,243
Li,C. N.,885
Li,H.,688
Li,X.,795, 796
Liaison,262
Liberman, A. M.,247, 284
Liberman, M. Y.,257, 259, 266,
271, 283, 483, 484, 775,
870
Lidstone, G.J.,98, 100
Lieber, R.,79
Lieberman, H.,584
LIFO,37,41
Light, M.,688
likelihood, 140
Lim,D.C. Y.,723
Lin similarity, 668
Lin,C.-Y., 805, 809,819
Lin,D.,484, 666–668, 670,
672–674, 679, 689, 690,
718, 733, 824
Lin,J.,794, 800, 810, 823,824
Lin,S.,920
Lindblom, B. E.F.,118
Lindemann, A.,771
Lindsey, R.,586
linear interpolation for N-grams,
104
Linear Predictive Coding, see
LPC
linear regression, 194, 196

870
104
LPC
multiple, 195
setting weights, 196
Linear segmentation, 696
Linguistic Data Consortium, 471
Linguistic Discourse model, 734
linguistic knowledgewhy needed, 2
linguists, ﬁring, 83
Link Grammar, 419
lipsmack
Literal meaning, 553
literal meaning, 589
Litman, D.J.,705,734, 827,
844, 851, 852,857–860,
863, 869
Littman, J.,756, 759
Littman, M.,679
Liu,H.-M.,387
Liu,Q.,920
Liu,Y.,361, 364, 920
Liu,Z.,871
Livescu, K.,245, 358, 364
LM,83
LMSF,317
LOBcorpus, 169
Local ambiguity, 437

lipsmack
863, 869
LM,83
LMSF,317
local ambiguity, 437
localization, 876
locative adverb, 126
Lochbaum, K.E.,869,870
locutionary act, 829
Loebell, H.,424
Loebner Prize, 7
Lofting, H.,27
log
compression of speech, 232
Loglikelihood ratio, 805
log probabilities, 107
log-linear classiﬁers, 193
Logical connectives, 563
Logical vocabulary, 558
Logistic function, 199
Logistic regression, 198
logistic regression, 193
conditional maximum
likelihood estimation,
200
logit function in,198
multinomial, 201
setting weights, 200
Logitfunction, 198
Logprob, 336

log
200
logprob, 336
logprob (log probability), 315
Loke, H.P.,276
Lombard effect (Lombard
speech),355
Long-distance dependencies,
511
long-distance dependencies, 608
long-distance dependency, 397,
408
Longacre, R. E.,734
look-ahead as solution to
non-determinism, 33Loper, E.,16
loudness, 235
Lowe, J.B.,508, 509, 629, 637,
646
lower tape, 61
Lowerre, B.T.,325, 334
lowest common subsumer, 666
LSA,seelatent semantic
analysis
LSI,seelatent semantic indexing
LTS,seeletter-to-sound
Lu,Z.,771, 773
Lua, K.,113
Lucassen, J.,260, 284

511
408
646
analysis
Luce, P.A.,363
Luhn, H.P.,785, 804, 822, 823
Luk, R.W.P.,284
Lunar, 586,619
Luperfoy, S.,854
LVCSR,289
Lx (laryngograph), 276
Lyons, J.,587, 645
Lyons, R. G.,247
Lytel, D.,688
M-step (maximization step) in
EM,192
Macdonald, J.,362
MacEachern, M.,858
machine, seeﬁnite-state
automaton
machine learning, 173
applied to document
categorization, 824
machine readable dictionaries
in WSD,688
machine state as opposed to
search state, 33
Machine translation, 1,873
machine translation, seeMT
MacIntyre, R.,408, 410

LVCSR,289
EM,192
automaton
in WSD,688
Macleod, C.,405, 508, 510
Macon, M.,282
MacWhinney, B.,547
Madhu, S.,688
Madison, J.,121
Magerman, D. M.,170, 414, 490
Mahalanobis distance, 307
Maier, E.,854, 855
Main, M.G.,619
Maiorano, S.,795
Makatchev, M.,827
Makhoul, J.,348, 359, 921
Makkai, A.,620
Maleck, I.,855
Malouf, R.,201, 524, 532
Mandarin, 241, 873, 877
Mangu, L.,171, 343
Manhattan distance, 675

DRAFT994 Index
Mani, I.,756, 759, 802, 803,
809, 823, 824
Mann, W.C.,80, 703, 734
manner adverb, 126
manner of articulation, 221
Manning, C. D.,16, 81, 156,
160, 170, 213, 471,
475–477, 490–492, 532,
675, 689, 691, 699, 734,
805, 822, 899
Manurung, H.M.,823
MAPSSWE, 156
Maratsos, M.,547
Marcel, S.,772
Marchand, Y., 284
Marcinkiewicz, M.A.,13, 123,
130, 155, 408, 410, 471
Marcu, D.,13, 703, 704, 706,
707, 734, 799, 807–809,
813, 823, 824, 891, 914,
920
Marcus, M.P.,13,123,130, 155,
408, 410, 457, 462, 471,
483, 484, 490, 688
Marcus, S.,674
marker passing
for word sense
disambiguation, 688
Markov,88
assumption, 88
chain, 119, 144
model,88
history, 119
Markov assumption, 175
Markov chain, 174
accepting states, 176
ﬁrstorder, 175
formal deﬁnition of, 175
initial distribution, 176
N-gram as,175
start and end states, 175
states, 175
transition probabilities, 175
Markov decision process, 860
Markov model
Markov, A.A.,119, 173, 212
Markovitch, S.,674, 690
Marshall, C.,871

Markov,88
model,88
Marshall, I.,139, 169
Marslen-Wilson, W.,78,362,
363
Martin, D.,767
Martin, J.H.,171, 620,645, 646,
690, 775
Martin, P.,726
Martin, R.,857, 858, 871
Martina, R., 871
Marx, G.,436
Marx, M.,829, 832, 846,850Marzal, A.,491
massnoun, 125
Massaro, D.W.,362
Mast, M.,855,871
Masterman, M.,586,688
Masuko, T.,285
Matiasek, J.,387
Matsumoto, Y.,491
Matsuzaki, T.,491
Matthews, A.,731
Matui, E.,284
max-margin methods, 214
MaxEnt, 193, 201, 201
Gaussian priors, 205
learning in, 204
normalization factor Zin, 201
regularization, 205

363
690, 775
why called maximum entropy,
205
Maxey, H.,283
Maxim,834
of Manner, 834
of Quality, 834
of Quantity, 834
of Relevance, 834
maxim,834
Maximal Marginal Relevance,
811
Maximization step, 471
maximization step in EM,192
Maximum Entropy Markov
Model,seeMEMM
Maximum Entropy modeling,
193
maximum entropy modeling
intuition, 205
maximum likelihood estimation,
seeMLE
Maximum Likelihood Linear
Regression, 356
maximum margin methods, 214
Maximum matching, 70
maximum matching, 70
Maximum Mutual Information
Estimation, 353

205
Maxim,834
maxim,834
811
193
seeMLE
Maximum Onset, 377
maxmatch, 70
Maxwell, J.T.,484,491, 532
Mayan, 878
Maybury, M.T.,802,824
Mayﬁeld, J.,775
Mays, E.,73, 168
McAllaster, D.,364
McAllester, D.A.,550
McCallum, A.,159, 213, 214,
738, 765, 775
McCarthy, D.,670, 688, 689
McCarthy, J.,10, 427, 766
McCarthy, J.F.,733
McCarthy, J.J.,370McCawley, J.D.,47,428, 587,
646, 647
McClelland, J.L.,363
McConnel, S.R.,80
McConnell-Ginet, S.,587
McCoy, K.,646
McCulloch, W.S.,9, 42
McCulloch-Pitts neuron, 42
McDaniel, J.,85, 225,334
McDermott, D.,726
McDermott, E.,351

Mayan, 878
646, 647
McDonald, D.B.,620
McDonald, R.,491, 492, 773,
823
McEnery, A.,123, 130, 136, 170
McEnery, T.,733
McGee, D.,287
McGill, M.J.,822
McGuiness, D.L.,584
McGurk effect, 362
McGurk, H.,362
McKeown, K.R.,265, 689, 733,
810, 812,815, 818, 820,
823, 824
McLuhan, M.,878
McNamee, P.,775
McNemar test, 156,331
McRoy, S.,688
McTear, M.F.,871
MDL,382
MDP,861
Mealy machine
and Moore machine, 80
Mealy, G.H.,80
Mean,308
Mean average precision, 789
Mean reciprocal rank, 801
meaning
as action, 585

823
823, 824
MDL,382
MDP,861
Mean,308
meaning
Meaning as action, 585
Meaning postulates, 571
Meaning representation
languages, 551
meaning representation
languages, 553
Meaning representations, 551
meaning representations
as sets of symbols, 552
early uses, 586
Mechanical Turk,249
mediated referent, 712
Medvedeva, O.,771
Megyesi, B., 160
Mel,301
mel frequency cepstral
coefﬁcients, 297
mel scale, 235
Melamed, I.D.,913,914, 921
Mel’ˇ cuk, I.A.,428

DRAFTIndex 995
Mellish, C.,16, 823
Melz, E.,799
MEMM,208
as discriminative model, 209
compared to HMM,209
for unknown words in POS
tagging, 159
inference (decoding), 210
learning, 211
Viterbi decoding, 210
memory limitations, 549
Menezes, A.,914
Meng, H.,378, 379
Meng, Y.,871
Mensah, A. G.,771
Mercer, R. L.,13, 86,104, 118,
119, 139, 150, 168, 170,
260, 284, 334, 339, 354,
490, 894, 895, 904,
Merchant, N.,387
Merialdo, B.,150, 170
Merlo, P.,691
Mermelstein, P.,120, 302, 347,
348
Meronym, 629
meronym
difﬁculty of extracting, 682
Meronymy, 629
MERT
for training in MT, 919
Mertins, I.,285
Merton, R.K.,14, 617
MeSH (Medical Subject
Headings), 650
mesh,in ASR, 343
Message Understanding
Conference, 766
Metadata, 360
metamorphosis grammar, 11
Metaphor, 643
metaphor, 553
and analogical reasoning, 646
computational approaches,
646
metarule, 408
Meteer, M.,150,159, 170, 857,
858, 871
METEOR,910, 921
Metonymy, 625
metonymy
741
Meurers, W.D.,529, 532
Meyers, A.,405, 508, 510

meronym
MERT
646
858, 871
metonymy
741
MFCC,297
Micciulla, L.,921
Michaelis, L.A.,472
Michelizzi, J.,669, 691
Micro-Planner, 586microgrammar, 856
Microsoft
WAVformat, 233
Microsoft Word,171
regular expressions in, 17
Mihalcea, R.,688, 689
Mikheev, A.,81,743
mildly context-sensitive
grammar, 537, 538
Miller, C. A.,224
Miller, D.,334
Miller, G.A.,92, 119, 247, 538,
546, 547, 652,669, 908,
910, 920
Miller, J.L.,362
Miller, R.,112
Miller, S.,482, 775, 837, 839
Miltsakaki, E.,13, 734
Mimura, K.,283
min/max similarity metric, 677

MFCC,297
Microsoft
910, 920
Minimalist Grammar, 538
minimumdescription length, 382
Minimum edit distance, 74
minimum edit distance, 74, 74,
184, 330
and Viterbi, 146
example of,76
in summarization, 809
Minimum Error Rate Training,
919
minimum redundancy
hypothesis, 78
MINIMUM EDIT DISTANCE ,76
Minnen, G.,529
Minsky, M.,10
Mishne, G.,775
Mishra, R.,871
MITalk, 285
Mitamura, T.,920
Mitchison, G.,213
Mitkov, R.,733
Mittal, V., 811
mixed initiative, 843
Miyao, Y., 491,532
MLE
alternatives to, 351
for N-grams, 88
for N-grams, intuition, 89
MLLR,356
MMIE,353

184, 330
919
MLE
MLLR,356
MMIE,353
MMR,811
M¨ obius, B.,274
M¨ obius, B.,378, 379
modal,128
modallogic, 587
modaloperators, 587
modalverb, 128,406
Model,558
model
to mean phone HMM, 296Modiﬁed N-gram precision, 910
Modiﬁed RhymeTest, 282
modus ponens, 567,725
Moens, M.,675, 679, 743, 807,
810
Mohammad, S.,689
Mohler, G.,271
Mohri, M.,57,59, 60,67, 81,
428, 542
Moldovan, D.,682, 688–690
M¨ oller, S.,850, 852
Mondshein, L.F.,347
monologue
as type of discourse, 693
Monotonic, 501
Montague semantics, 587
Montague, R.,587, 619
Montero, J.M.,842, 860

modal,128
Model,558
model
810
428, 542
monologue
Monz, C.,797
Mooney, R.J.,750
Moore machines and Mealy
machines, 80
Moore, E.F.,80
Moore, G.,316,343
Moore, J.,698, 733
Moore, J.D.,854
Moore, R.,285
Moore, R.C.,168, 170, 587,
805, 920
Moran, C.,920
Morgan, A.,771
Morgan, A. A.,771
Morgan, N.,90, 229, 247,335,
351, 354,358, 487
Morimoto, T.,871
morpheme, 47
Morpheme boundary, 61
morphemes, 79
examples of in Turkish,46
morphological classes, 123
Morphological family size, 79
Morphological parsing, 45
morphological parsing, 79
argument from English

805, 920
productive afﬁxes for
need for, 46
evidence from Turkish for
necessity of,46
goals of,53
requirements for, 53
morphological recognition, 56
morphological rules, 45
morphology, 79
agglutinative, 48
CELEXdictionary of, 81
cliticization, 48,51
compounding, 48
concatenative, 52
derivation, 48
derivational, 50

DRAFT996 Index
inﬂection, 48
non-concatenative, 370
root-and-pattern, 52
templatic, 52,370
tier,370
use in part-of-speech tagging
of unknown words, 158
morphotactics, 53,79
Morris, J.,733
Morris, W.,626
MOS (mean opinion score), 283
Moscoso del Prado Mart´ ın, F.,
79, 487
MOSES, 920
Moshier, M.A.,501, 530
Mosteller, F.,11,121, 139, 290
Moulinier, I.,746
move, conversational, 854
MRR,801
MRT,282
MT,873
and dates, 879
and the web, 875
decoding
recombining hypotheses, 908
distortion probability, 891
divergences, 876
faithfulness, 889
ﬂuency, 889
problems with, 888
jump width, 898
lexical differences and, 880
phrase-based, 891
post-editing, 876
sublanguage in, 876
transfer model of, 884
transformation examples, 886
translation probability, 891
useful domains for, 875
MUC,766,774
MUC evaluation for coreference,
724
M¨ uller, K.,378, 379
Multi-document summarization,
810
Multinomial logistic regression,
201
multinomial logistic regression,
193
multiple discovery, 13, 81

724
810
201
193
Multiple document
summarization, 802
multiple linear regression, 195
Munoz, M.,462
Munteanu, C.,524,532
Murata, T.,585
Murveit, H.,339, 340
mutual information, 674Myers, K.,767
n
914
N-best,339
N-bestlist, 485
N-gram,88
applications of,84
ARPA format, 108
as approximation, 88
as generators, 93
as Markov chain, 175
backoff,105
Cambridge-CMU toolkit, 109
class-based, 111
equation for, 88

n
914
N-gram,88
example of,90
ﬁltering in QA, 801
for Shakespeare, 93
history of, 119
interpolation, 103
logprobs in, 107
mining in QA, 801
normalizing, 90
of dialogue act sequences, 856
parameter estimation, 89
sensitivity to corpus, 92
skip,113,820
smoothing, 97
SRILM toolkit, 109
tiling in QA, 801
tokenization before
computing, 86
topic-based, 112
triggers,113
unknown words, 94
variable length, 113

N-gram tiling, 800
N´ adas, A.,118, 119,354
Nagao, M.,491
Nagata, M.,871
Nahamoo, D.,347
training, 654
usein WSD,653
Nakajima, S.,283
Nakatani, C.,364, 854,869
Nakov, P.I.,112, 657name
Named entity, 739
named entity
list of types, 740
mentions, 738
named entity recognizer, 767
named entity recognition, 737,
739
names
and gazetteers, 743
as unknown words, 258
Census lists, 743
detecting language of origin,
262
in TTS,259
pronunciation in TTS,259

739
names
262
in TTS,259
rewriting in summarization,
814
Nanjo, H.,700, 734
Narayanan, S.,585, 646
Narayanan, S.S.,285, 845, 852
nasal sound, 219,221
nasal tract, 219
Nash-Webber, B.L.,732, 823
Nass, C.,8
natural languages contrasted
with formal languages,
31
Naur, P.,10, 427
Navigli, R.,688
Nederhof, M. J.,417, 428
Needleman, S.B.,213
Negation, 530
negative part-of-speech, 129
Neiser, J.,78
Nenkova, A.,266, 810, 815,820,
823, 824
Neo-Davidsonian, 572
NER,737
Nerbonne, J.,529
Nespor, M.,285
Nesson, R.,914
nested, 536
Neu, H.,229

814
Nass, C.,8
31
823, 824
NER,737
Newell, A.,84
newline character, 24
Newman, E.B.,235, 301
Newman, S.,369
Newport, E.L.,365,382
Ney, H.,13,110, 113, 120, 284,
340, 341,347, 364, 469,
899, 904,919, 920
NFSA,32, 41
Ng, A.Y.,13, 681, 682,688, 690
Ng, H.T.,656, 657, 688, 689,
723
Ng, S.-K.,770
Ng, V.,733
Ngampatipatpong, N.,9

DRAFTIndex 997
Nguyen, L.,347, 359
Nicely, P.E.,247
Nichols, J.,877
Nicolov, N.,775
Nida, E.A.,646
Niekrasz, J.,871
Nielsen, J.,850
Niemann, H.,871
Niesler, T.R.,111, 113
Nilsson, J.,491
Nilsson, N.J.,346, 865, 870
Ninomiya, T.,532
Nirenburg, S.,920, 921
Nissim,M.,713
NIST
for MT evaluation, 910, 921
NIST.,13, 77, 243, 331
Nivre, J.,491, 492, 854
Niyogi, P.,364
Nock, H.,357
node as term for FSAstate as
opposed to search state,
33
Noisy channel, 289
noisy channel model, 291
for MT,889
for speech, 289
for spelling, 163
Noisy-or, 754
Nolan, F.,285
Nominal, 392
nominalization as example of a
morphological process,
50
Nominative, 404
nominative, 404
non-concatenative morphology,
52
non-deterministic FSA, 32
non-ﬁnite postmodiﬁer, 401
Non-logical vocabulary, 558
Non-projective dependencies,
491
non-standard words
in TTS,253
Non-stationary, 298
non-terminal symbols, 392, 393,
425
nonein FUG,527
Noordman, L.G.M.,734
Normal distribution, 308
Normal form, 416
normal form,415, 416
dates, 837
temporal, 758

50
52
491
in TTS,253
425
dates, 837
normalization ofprobabilities,
88
normalized dot product, 677Norman, D.A.,11,587, 825,
830, 836
Normandin, Y., 354
Norris, D.,363
Norvig, P.,16, 42, 84,168, 172,
462, 562, 587,646, 656,
691, 802, 863,871
N¨ oth, E.,871
noun,125
abstract, 125, 399
common, 125
count, 125
days of the week coded as,
126
mass,125, 399
proper,125
noun class, 52
Noun group, 418
noun group, 418
Noun phrase, 390
noun phrase, 389, 392,393, 426
genitive, 609
noun-noun compounds
stress in, 266
noun-noun sequences, 608
Novick, D. G.,850

88
830, 836
noun,125
count, 125
126
Nowak, M. A.,119
NP,392, 393
NPattachment, 473
NP-complete, 813
NP-completeness of LFG,550
NP-completeness of natural
language parsing, 550
NP-completeness of two-level
morphology, 550
Nuclear accent, 265
Nuclear scope, 594
Nucleus, 703
nucleus of syllable, 223
number, 426, 493
numeral, 126
numerical expression, 740
Nunes, J.H.T.,871
Nyberg, E.H.,920
Nyquist frequency, 232,297
Ndellec, C., 460
O’Connor, M.C.,620
O’Donnell, M.,823
O’Shaughnessy, D.,247
Oberlander, J.,823
object, syntactic, 390
frequency of pronouns as, 472

obligatory rules, 367
observation likelihood, 148,182,
185, 292, 321,322
observations
in HMM, 144
Occasion (as coherence
relation), 702Och, F.J.,13,891, 899, 904,
919, 920
OCR,72
Odden, D.,388
Odds,198
Odell, J.J.,341, 348–352
Odell, M. K.,82, 170
Oden, G.C.,362
Oehrle, R. T.,428
Oepen, S.,491, 532
Oerder, M.,347
Oettinger, A.G.,461
Oﬂazer, K.,81, 161, 162
Ogburn, W.F.,14
Ogren, P.V., 771,773
Ohno, S.,285
Okurowski, M.E.,13,703, 706,
734
OLA (overlap-and-add), 276
old information, and word order,
472

in HMM, 144
919, 920
OCR,72
Odds,198
734
472
Older, L.,78
Oliva, K.,163
Olive, J.P.,274, 283, 285
Ollason, D.,316, 343
Olshen, R.A.,213
Omura, H.,284
on-line handwriting recognition,
72
on-line nature of human speech
processing, 362
on-line sentence-processing
experiments, 490
Oncina, J.,380
One Senseper Collocation, 662
one senseper collocation
constraint, 662
One SensePer Discourse, 663
onset
ontology, 579,629
OntoNotes, 688
Oommen, B.J.,170
OOV (outof vocabulary) words,
95
OOV rate, 95
open class, 124
Open prompt, 844
Open quotient, 361

72
onset
95
open vocabulary system
unknown words in, 95
operation list, 74
operator precedence, 22, 22
Oppenheim, A.V.,333
optical character recognition, 72
Optimal coupling, 275
Optimality Theory, 371,373
implementation via FST,374,
375
optional rules, 367

DRAFT998 Index
optionality
of determiners, 399
use of() in syntax, 400
use of? in regular expressions
for, 19
Oracle accuracy, 486
oracle error rate, 341
oral tract, 219
Oravecz, C.,161, 162
ordinal number, 399
Orloff, J.,364
orthographic rules, 45, 53, 63
orthography
opaque, 217
transparent, 217
Ortmanns, S.,341
Ortony, A.,646
Osborne, M.,912, 920
Osgood, C.E.,689
Ostendorf, M.,111, 112,254,
255, 264, 267, 282, 359,
364, 858
OSTIA,380
OT,371
other, 64
Ott, K.,871
Overlap-and-add, 276
Oviatt, S.L.,287, 825,850, 858
p-subsequential transducer, 60,
60
p1
914
Packard, D.W.,80
Pagel, V.,258, 261
paired method for digit
pronunciation, 254
paired t-tests, 156
paired tests, 156
Pakhomov, S.,773
palatal sound, 220
palatalization, 227
Palate,220
palato-alveolar sound, 220
Pallet, D.,334
Palmer, D.D.,71,81
Palmer, M.,13,83, 419, 635,
651, 652, 656, 657, 666,
688, 690
Palmucci, J.,150, 159, 170
Pan, S.,265,860
Pang, B.,824
Pantel, P.,690,824
Pao, C.,851
Paolino, D.,285

60
p1
914
688, 690
Papadimitriou, C.,38,39, 42,81,
538, 539
Papcun, G.J.,773
Papineni, K.,852, 910, 921Pappuswamy, U.,827
Paquette, J.,771
PARADISE,852
Paragraph segmentation, 699
Parallel (as coherence relation),
701
Parallel corpus, 899
parallel text, 899
parallelism as solution to
Paraphrase detection, 824
Parent annotation, 475
Parker, A.,620
PARRY,870
parsetree, 392, 394
parsed corpus, 490
parsers
evaluation, 483
parsing,45, 460
ambiguity, 436
as search, 432
bottom-up, 433, 434
chart, 440, 452

538, 539
701
PARRY,870
parsers
CKY, 440, 469
complexity duetomemory
limitations, 547
CYK, 440
difﬁculty of self-embedding,
547
Earley, 440, 447
history, 461
morphological, 45
partial, 455
shallow,455
syntactic, 431
top-down, 433, 433
top-down versus bottom-up,
435
well-formed substring table,
461
parsing and grammar, 396
Parsons, T.,572, 587
part-of-speech, 123, 392
adjective, 125
adverb,126
closed class, 124, 126
negative, 129
noun, 125
particle,126

CYK, 440
547
435
461
noun, 125
possessive versus personal
pronouns, 124
subclasses of verbs, 125
subtle distinction between
verb and noun, 125
usefulness of, 123verb,125
part-of-speech tagger
CLAWS,130
ENGTWOL, 137
HMM,135
PARTS,170
rule-based, 135
stochastic, 135
TAGGIT,169
TBLor Brill, 151
part-of-speech tagging, 124, 133
accuracy, 155
adverbial that, 138
ambiguity and, 133
amount of ambiguity in
analysis of errors in, 157
and morphological analysis,
160
Brill or TBL

CLAWS,130
HMM,135
PARTS,170
160
example ofrule template
from, 153
complementizer that, 139
contingency table or
confusion matrix for
error analysis of, 156
decision trees, 170
distinguishing preterites from
participles, 157
early algorithms, 137
for phrases, 157
Gold Standard, 155
history of, 169
human performance at, 155
Hungarian, 160
in agglutinative languages,
161
log-linear analysis, 170
maximum entropy, 170
percent correct as metric for,
155
SNOW,170
TBLor Brill, rule learning in,
152
TBLor Brill, sample
application, 152

from, 153
161
155
SNOW,170
152
transformations, 155
Turkish, 160
unigram baseline, 155
unknown word
dealing with, 158
useof morphology, 158
useof subcategorization
information, 137
Part-whole, 629

DRAFTIndex 999
Partee, B. H.,16, 541, 542, 549,
550, 587
Partial parse, 455
partial parsing, 455, 460
participle
-ing in English, 50
particle, 126, 126, 405
table of, 127
PARTS,170
Pasca, M.,795, 797, 799, 800
Passage retrieval, 797
passive auxiliary, 406
Passonneau, R.,734, 810, 820,
824, 852, 869
past participial VP,426
Pastizzo, M.,79
Pater, J.,387
Path inequalities, 530
path-length based similarity, 665
Patil, R.,437
PATR-II,496
pattern
as target of regular expression
search, 18
Patwardhan, S.,669, 670, 691
Paul, D.B., 345,346
PCFG,463, 464, 489
for disambiguation, 465
lack of lexical sensitivity, 473
lexicalized, 490
parse probability, 466
poor independence
assumption, 472
rule probabilities, 464
use in language modeling, 467
PCM (Pulse CodeModulation),
233
pdf,308
Pearl, J.,13, 346
Pedersen, J.,150,687, 790, 823
Pedersen, T.,668–670, 689, 691
Peirce, C. S.,725
Pelletier, F.J.,619
Pellom, B., 9,852
Penn Treebank, 155, 408, 471
POS tags for phrases, 157
tagset, 130, 131
Penn, G.,524, 530,532

233
pdf,308
per-letter entropy, 118
per-word entropy, 115
percent correct
use in evaluating
part-of-speech taggers,
155
Percival, W.K.,426
Pereira, F.C. N.,11, 16, 159,
213, 214, 378, 428, 491,
492, 501, 524, 530, 531,
550, 678, 689, 690, 773,
775perfect
-ed/-ed form in English, 50
perfect auxiliary, 406
performance grammars, 619
performative, 829
period disambiguation, 251
period of awave, 231
Perkowitz, M.,156
Perllanguage, 17, 18
Perles, M.,417, 549
Perlis, A.J.,10, 427
perlocutionary act, 829
Perplexity, 95,117
perplexity, 92

155
asweighted average branching
factor, 96
deﬁned via cross-entropy, 117
Perrault, C. R.,12,864, 867, 870
person, 495
personal pronoun, 128
Peshkin, L.,765
Peskin, B.,364
Peters, I.,688
Peters, S.,587, 734, 871
Peters, W.,688
Peterson, G.E.,247, 283
Peterson, J.L.,73
Petkevic, V., 162
Petkeviˇ c, V., 163
Petri net, 585
as meaning representation,
585
Petrie, T.,212, 334
Petrov, S.,477, 490
Petruck, M.R. L.,637, 646
Pevzner, L.,700
Pfefer, A.,765
PHARAOH, 920
Phillips, M.,851
phone,216
Phonemodel, 296

factor, 96
585
phone,216
phone recognition, 292
phone, context-dependent (CD),
348
phone, context-independent (CI),
348
phoneme, 226, 246
Phoneme restoration effect, 362
phones, 245
phonetic feature, 227
Phonetics, 216
phonetics
articulatory, 218, 218
ﬁeld of,216
phonological rule
compiling into FST,368
ordering, 366
transducer for, 365
phonological rules, 228Chomsky-Halle rewrite
format, 536
obligatory, 367
optional, 367
phonology
Phonotactics, 223
PHRAN, 620
phrasal alignment, 895
phrasal verb, 126, 405

348
348
phonetics
phonology
PHRAN, 620
Phrase alignment, 892
phrase penalty
in MT,919
Phrasetranslation table, 893,904
phrase-based MT,891, 891
phrase-structure grammar, 391,
425, 427
Picheny, M. A.,285, 347
Piepenbrock, R.,81,126, 241
Pieraccini, R.,837, 838,
861–863
Pierce, J.R.,919, 921
Pierre, I.,620
Pierrehumbert, J.B.,267, 268,
271, 285,379, 387, 869,
870
pinched lattice, 343
Pineau, J.,871
pipe,21
Pisoni, D.B.,362, 363, 387
Pitch,235
pitch,235
changing in diphone TTS,277
Pitch accent, 224,264
pitch accent
and sentence stress,224
ToBI,267

in MT,919
425, 427
861–863
870
pipe,21
Pitch,235
pitch,235
ToBI,267
Pitch extraction, 235
Pitch marking, 276
Pitch range, 271
Pitch track, 233
Pitch tracking, 276
pitch-synchronous joins in TTS,
276
Pito, R.,412
Pitrelli, J.F.,267, 285
Pitts, W.,9, 42
place of articulation, 220
Placeway, P.,343
planning
and speech acts, 864
shared plans, 871
Plato, 246
Pleonastic, 713
plosive sound, 221
Plotkin, J.B.,119
plural, 48,399
plural afﬁx, 48
Ply,433
ply of asearch space, 433

DRAFT1000 Index
PNAMBIC,850
Poesio, M.,733, 823
674
Polanyi, L.,734
Polifroni, J.,844, 850–853
politeness
by and to computers, 8
politeness marker, 129
Pollard, C.,413, 414, 427, 529,
530, 532, 538, 718, 733,
735
Pollock, J.J.,823
Pols, L.C. W.,118
Polysemy, 625
polysemy
and IR, 790
polysynthetic language, 877
POMDP,864,871
Pon-Barry, H.,871
Ponzetto, S.P.,690
Popescu, A.-M.,690, 754, 755
Porter stemmer, 68, 79
use in IR,786
Porter, M.F.,56,68, 82,786
POS,123
POStagging, seepart-of-speech
tagging
possessive, 48
possessive NP, 429
possessive pronoun, 128
Post, B.,285
Post-editing, 876
post-editing MToutput, 909
postdeterminer, 399
postmodiﬁer, 400, 426
Postposed, 391
postposed constructions, 391
postposition, 877
Potamianos, A.,852
Potts, C.,387
Povey, D.,316, 343,351, 354,
358
power,535
power of asignal, 234
Power, R.,733, 823,854
PP,393
praat, 236, 237, 244
Prabhu, P.,852
Pradhan, S.,690
pragmatic grammars, 619
Prakash, S.,688
Prasad, R.,13, 734
Precedence, 21

tagging
358
power,535
PP,393
precedence, operator, 22
Precision, 459
precision, 459
in NER,744modiﬁed N-gram, in MT,910
preconditions for STRIPSplan,
865
predeterminer, 402
predicate, 405
predicate-argument relations,
405
preference semantics, 688
preﬁx, 47, 79
Preposed, 391
preposed constructions, 391
preposition, 126, 126
prepositional phrases
semantic attachments, 613
prepositional dative, 425
prepositional phrase, 391, 393,
393, 400
prepositions
acquisition of semantics, 585
table of English, 127

865
405
393, 400
Prescher, D.,379, 532
present tense, 402
Preterite, 49
previous forward path
probability, 182, 321
previous Viterbi forward path
probability, 148,185,322
Price, P.J.,264, 267, 334,850,
858
Primary Colors, 122
priming, 425
repetition, 78
primitive decomposition, 641
Prince, A.,266, 371, 387
Prince, E.,712, 713, 733
principle of compositionality,
619
Principle of compositionality,
589
589
prior probability, 140, 292
priority queue, 344
Priority union, 529
pro-drop languages, 878

858
619
589
589
Probabilistic CKY, 469
probabilistic CKY algorithm,
469
probabilistic context-free
grammar, seePCFG
probabilistic model
in IR,822
probabilistic parsing
by humans, 487
Probability density function, 308
probability density function, 308
procedures
as meaning representations,
585proclitic, 51
production system, 568
productions, 391, 425
productive, 49
productive morphology
usein argument fornotstoring
all possible English word
forms,46
Progressive, 50
progressive auxiliary, 406
Progressive prompting, 846
projection of an FST, 59

469
in IR,822
forms,46
Prolog, 568
Prominence, 264
prominence, phonetic, 225
Prompt,840
prompts
design of,850
pronominal reference, 711
pronoun, 126, 128
and old information, 472
bound, 711
demonstrative, 711
greter frequency in spoken
English, 422
personal, 128
possessive, 128
table of,129
wh-,128
pronunciation dictionary, 241
CELEX,241
CMU,242
PRONLEX,242
PropBank, 636
proper noun, 125
Propositional meaning, 627
Prosodic Phrasing, 263
Prosody,263
prosody
reduced vowels, 224
PROTO-AGENT,635

prompts
bound, 711
wh-,128
CELEX,241
CMU,242
prosody
PROTO-PATIENT,635
prototype vector
in VQ,305
Proximal, 711
Pruning,325
Pr¨ ust, H.,734
PTAG,491
PTRANS,642
Pullum, G.K.,247,389, 396,
408, 428,527, 542, 544,
546, 549,619
pumping lemma, 539, 540, 549
for context-free languages,
541, 549
punctuation
Punyakanok, V.,462

DRAFTIndex 1001
Purple Haze, 225
Purver, M.,871
Pushkin, A.,173
Pustejovsky, J.,13,645, 756,
759, 762, 775
Pyramid Method, 820
q0as start state, 28
Q-systems, 11
Qin, L.,690
Qu,Y., 775
Qualia structure, 645
quality
quantiﬁer, 399, 400, 426
Quantiﬁers, 563
quantiﬁers
scoping,598
quantization, 232,297
quantization of an analog signal,
232
Quantz, J.,855
quasi-Newton optimization, 201
Query,782
query
expansion, 792
improvement, 791
in IR,782
Query expansion, 792
query reformulation
in question answering, 794
Query-based summarization, 781
Query-focused summarization,
802,816
question
automatic detection of, 857
factoid,780
Question answering, 1
question answering
factoid questions, 780
query reformulation in, 794
question answering task, 730
Question classiﬁcation, 795
Question ontology, 795
Question rise, 267
question rise, 267
questions, 397
queue
use ofin breadth-ﬁrst search,
37
Quillian, M.R.,11, 587, 688
Quinlan, J.R.,213
Quirk, C.,824, 914
Quirk, R.,49, 127, 428
R. D.,428

802,816
question
queue
37
R. D.,428
Rabin, M.O.,42
Rabiner, L.R.,11,179, 190,
191, 213, 335
Radev, D.,13,762, 806, 809,
823Radford, A.,391, 428
Radio Rex, 333
Raghavan, P.,16, 822
Raghunathan, B., 871
Rambow, O.,81,690, 840
Ramesh, P.,364
Ramshaw, L.A.,150, 159, 170,
359, 419, 457,462, 484,
688, 839
Rand, D.,230
random sentences baseline in
range
in regular expressions, 19
Raphael, B.,586
Rapid reprompting, 846
Rappaport Hovav, M.,635, 645
rarefaction, 231
Ratcliff, D.,201
rate of speech, 229
Rationalist, 433

R. D.,428
688, 839
range
rationalist tradition, 433
Ratiu, F.,871
Ratnaparkhi, A.,155, 156, 159,
160, 170, 213,482, 490,
491
Ratner, N. B.,78
Rau, L.F.,774, 823
Ravichandran, D.,690, 795, 799
Ravishankar, M.K.,347
Raymond, W.D.,119, 229,358
Rayner, M.,836, 837, 839,871
RE,17
read speech (as opposed to
conversational speech),
288
Reading time, 487
reading timeexperiments, 730
real estate ads
predictors of houseprices, 194
real-word spelling errors, 73
Rebholz-Schuhmann, D.,772
Recall,459
recall, 459
for MTevaluation, 910, 921

491
RE,17
288
in NER,744
recognition
recombining hypotheses
in MT,908
in speech, 340
Rectangular, 299
recursion, 391
center-embedding, 417
recursively enumerable, 536
reduced vowel, 224
reduction
phonetic, 224, 226
reentrant structure, 496, 498
Reeves, B.,8Reference, 707
reference
bound pronouns, 711
cataphora, 711
deﬁnite, 710
indeﬁnite, 710
pronominal, 711
resolution, 707
Centering algorithm, 718, 720
Hobbs tree search algorithm,
716, 717
psychological studies, 730

in NER,744
in MT,908
reduction
reference
716, 717
Reference line, 271
Reference point, 574
Reference resolution, 707
Referent, 707
referent
accessing of, 708
evoking of, 708
mediated, 712
Referent naming task, 730
Referential density, 878
Referring expression, 707
Reﬂexive, 714
reformulation, 831
Regier, T.,585
register
sociolinguistic, 230
registers in RE pattern
substitution, 25
Regression line, 195
regression versus classiﬁcation,
194
Regular expression, 17
regular expression, 17, 17,18,41
returning lines of documents,
18
substitutions, 25
regular grammar, 537

referent
register
194
18
Regular language, 26
regular language, 26,38
proof that English isn’t one,
543
pumping lemma, 540
Regular relation, 58
regular sets, 38
regular verbs, 49
regularization in MaxEnt
learning, 204
Rehder, B.,679
Reichenbach, H.,574
Reichert, T.A.,213
Reichl, W.,120
Reichman, R.,869
Reimer, U.,823
Reiter, E.,840, 859
Reiter, R.,529
Reithinger, N.,854, 871

DRAFT1002 Index
Rejection, 846
rejection, 860
by ﬁnite-state automaton, 28
Relation detection and
classiﬁcation, 738
relative
relative entropy, 661
Relative frequency, 89
relative frequency, 89
relative pronoun, 401
release
stop,221
Relevance feedback, 791
relevance feedback, 791,822
evaluation of, 791
update formula, 791
Renals, S.,871
repair, 422
in conversation, 360,422,423
repair as disﬂuency, 426
Reparandum, 423
reparandum, 423, 423
repeated namepenalty, 731
repeated parsing of sub-trees,
461
repetition
of words as disﬂuency, 422
Repetition primining, 362
REQUEST,866
rescoring, 339
Residual collection, 791
residual collection, 791
Resnik similarity, 667
Resnik, P.,490, 491, 660, 661,
665–667, 688, 689
Resolution, 569
resolve,133
restart, 422
and disﬂuency, 422
restart as disﬂuency, 426
Restriction, 594
Restrictive grammar, 836
restrictive relative clause, 401
Result (as coherence relation),
701
Reverse translation model, 918
rewrite, 392

461
701
Reynar, J.C., 491,699, 733
Reynolds, D.A.,335, 359
rhetorical relations, 703
Rhetorical Structure Theory, see
RST
Rhodes, F.L.,iv
Rhodes, R. A.,229
rhyme
Riau Indonesian, 125
Ribarov, K.,491,492Ribeiro-Neto, B., 779,822
Riccardi, G.,826
Rich transcription, 360
Richards, C.,254, 255
Richardson, J. E.,772
Rieger, C.,688
Ries, K.,857, 858,871
Riesbeck, C. K.,11,620, 688
Riezler, S.,484, 491, 532,823
Riggle, J.,387
right-linear, 537
right-linear grammar, 537
Rijsbergen, C.J.V.,822
Riley, M.D.,272,378

RST
rhyme
Riloff, E.,620, 680, 733, 753,
774, 775
rime
Rindﬂesch, T.C.,772
Ringenberg, M.,827
Ringwald, M.,772
Ristad, E.S.,81, 550
Rivest, R. L.,656
RMS amplitude, 234
Roark, B.,81, 111, 487, 491
Robins, R. H.,170, 246
Robinson, J.A.,531
Robinson, J.J.,619
Robinson, S.E.,822
Rocchio, J.J.,791, 822
Roche, E.,81, 153
Rochester, N.,10
Rohde, D. L.T.,412–414
Romanian, 162
Roossin, P.S.,13,920
root-and-pattern morphology, 52
Rooth, M.,490
Roque, A.,827
Rosario, B.,772
R¨ oscheisen, M.,920
Ros´ e, C.,827
Rose, R.C.,335, 364

774, 775
rime
Rosenberg, A.E.,361
Rosenberg, C.R.,284
Rosenfeld, R.,109, 112, 113,
213, 362
Rosenzweig, J.,651, 658, 659
Ross,K.,364
Roth, D.,170, 171, 462,765,
795, 796
ROUGE, 819
compared to BLEU,819,820
ROUGE-1, 819
ROUGE-2, 819
ROUGE-L, 820
ROUGE-S, 820
ROUGE-SU, 820
Roukos, S.,170, 348, 483,484,
490, 491, 910,921
rounded vowels, 223Rounds, W.C., 501,529, 530
Roussel, P.,531
Roy, N.,871
RST,702,806
orginal formulation, 703
parsing, 706
TreeBank, 703
Rubenstein, H.,669
Rubin, D.B.,104, 167, 187
Rubin, G.M.,137, 169, 772

213, 362
795, 796
ROUGE, 819
Rudnicky, A.I.,362, 826,846,
852
Ruhi, U.,243
rule
lexical,529
rule operator, 367
rule-to-rule hypothesis, 619
Rule-to-rule hypothesis, 591
rules, 425
context-free, 391
expansion, 392, 397
sample English, 393
useof () for optional
constituents, 400
dotted,447
orthographic, 53, 63
phonological, 228
compiling into FSTs,368
two-level, 367
phonological and transducers,
365
spelling, 53, 63
Rumelhart, D.E.,11,587
Ruppenhofer, J.,637
Rush, A.,914
Russell, G.,920
Russell, N.H.,364

852
rule
rules, 425
365
Russell, R. C.,82, 170
Russell, S.,16, 42,84, 562, 587,
656, 691,863, 871
Russell, S.W.,646
Russian, 877, 878
Rutishauser, H.,10, 427
Ryder, M. E.,620
S,393
Sacks, H.,828, 856, 870
Sadek, D.,871
Sadek, M.D.,871
Sadock, J.,555
Saffran, J. R.,365,382, 387
Sag, I. A.,408, 413, 414, 427,
428, 511,527, 529, 530,
532, 619,870
Sagerer, G.,228
Sagisaka, Y.,283, 285
Saint-Dizier, P.,645

DRAFTIndex 1003
Sakoe, H.,213,333
Salasoo, A.,362
salience
in discourse model, 711
Salomaa, A.,80,490
Salton, G.,782, 791,822
Salvetti, F.,775
Samelson, K.,10, 427
Sampling, 297
sampling, 297
used in clustering, 687
sampling of an analog
waveform, 232
Sampling rate, 297
sampling rate, 232,297
Sampson, G.,102,103, 120, 170
Samuel, A. G.,362
Samuel, K.,151
Samuelsson, C.,120, 159
San-Segundo, R., 842,860
Sanders, G.,852
Sanders, T.J.M.,734
Sanderson, M.,790
Sanﬁlippo, A.,405, 484, 509,
529
Sang, E.F.T.K.,13, 746, 775
Sang, E.T.K.,460
Sankoff, D.,213, 230
Santorini, B.,13, 123,130–132,
155, 408, 471, 483, 484
Saraclar, M.,111, 357
Sassano, M.,775
Satellite, 703
satellite,878
satellite-framed language, 878
satisfaction-precedence
Satta, G.,374
Saur´ ı, R., 13
Sauri, R., 756,759
Saur, R.,762
sausage, in ASR, 343
Scaled likelihood, 355
scaled likelihood, 355
SCFG,seePCFG,464
Sch¨ onkﬁnkelization, 566
Scha, R.,734
Schabes, Y.,81, 153, 171,490,
491, 914
Schachter, P.,126

491, 914
Schaefer, E.F.,830, 854
Schafer, R. W.,333
Sch¨ afer, U.,532
Schalkwyk, J.,850
Schank, R.C.,11, 587, 629, 764
Schapire, R.E.,213,491, 813
Schasberger, B.,408, 410
Scheffczyk, J.,637
Schegloff, E.A.,828, 832, 856,
870Scheideck, T.,871
Schilder, F.,762, 775
Schiller, A.,70
Schlaikjer, A.H.,818
Schlesinger, J.D.,810, 813, 823
Schmandt, C.,832, 850
Schmolze, J.G.,532, 587
Schone, P.,382–384
Sch¨ onkﬁnkel, M.,566
Schoolhouse Rock, 123
Schreiner, M.E.,679
Schreuder, R.,79
Schroder, M.,285
Schroeter, J.,284

Schubert, L.K.,619
Schuetze-Coburn, S.,285
Schultz, T.,357, 364
Sch¨ utze, H.,16, 170,213, 471,
492, 657, 675,686, 689,
690, 790, 805,822
Schwa,224
schwa,224
Schwartz, A. S.,771
Schwartz, R., 150,159, 170,
339, 343, 347,348, 359,
482, 491, 775,810, 823,
837, 839, 921
Schwartz, S.,9
SCISOR,774
sclite, 331
sclite package, 77
Scott, D.,42, 733
Scott, M.,487
script, 629
Scripts,764
SCU,820
SDC,334
SDRT(Segmented Discourse
Representation Theory),
734
search, 42,460
A∗,363
as metaphor for
non-deterministic

Schwa,224
schwa,224
SCU,820
SDC,334
734
A∗,363
recognition, 35
beam, 325
beam in MT,907
breadth-ﬁrst, 37, 41
picture of, 37
pitfalls of,38
data-directed, 433
depth-ﬁrst, 36, 37
pitfalls in,37
FIFO,37
First InFirst Out, 37
forward-backward, 343
goal-directed, 433
in ASR, 290
LIFO,37multiple-pass, 343
parsing as, 432
queue for breadth-ﬁrst, 37
stack for depth-ﬁrst, 36
search strategy, 41
search-state in non-deterministic
recognition by ﬁnite-state
automata, 33
Searle, J.R.,7,830, 870
secondary stress, 225
See, A.,13, 762
seed pattern

beam, 325
FIFO,37
in ASR, 290
in IE,751
Segal, J.,90,487
segment based acoustic model,
364
segment or phone, 216
segmentation
Chinese word, 70
sentence, 70
word,47,69
Seidenberg, M. S.,387
Seitz, F.,348
Sejnowski, T.J.,284
Sekine, S.,484, 775
selectional association, 661
selectional preference strength,
660
Selectional restriction, 639
selectional restriction violations
and WSD,660
selectional restrictions, 640
self-embeddings, 547
Selfridge, J.A.,92
Selfridge, M.,620
Selfridge, O.G.,333
Selkirk, E.,285

in IE,751
364
660
syntax-driven, 590
semantic networks, 552
Semantic analysis, 552,589
expectation-driven, 620
syntax-driven, 589
word-based, 620
Semantic analyzer, 590
semantic analyzer, 590, 615
Semantic attachments, 591
semantic concordance, 651
semantic distance, 664
semantic drift
in IE,753
semantic feature, 642
Semantic ﬁeld, 629
semantic grammars, 619
semantic network
Semantic networks, 578

DRAFT1004 Index
semantic networks
origins, 587
semantic processing, 551
semantic relations
list of for IE,747
semantic role labeling, 683
semantic transfer
in MT,885
semantically unpredictable
sentence, 282
semantics, 551
frame-and-slot, 836
grounding in embodiment,
585
Semilattice, 501
semivowel, 220
Sen, Z.,358
Seneff, S.,243,378, 379, 837,
838, 841, 844, 850–853
Senellart, J.,887
Sengal, C.J.,730
sense
word,624
senseaccuracy, 656
SENSEVAL
and WSDevaluation, 656
SENSEVAL corpus, 651
sentence, 426
sentence centrality
in summarization, 805
Sentence completion task, 730
Sentence compression, 810
Sentence error rate, 331
sentence error rate, 331
Sentence fusion, 815
sentence segmentation, 70
in MT,899
Sentence simpliﬁcation, 810
Sentence tokenization, 251
sentential
use ofcue phrase, 705
Sentential complements, 404
sentential complements, 404
sequence classiﬁer, 173
FSTas, 173
sequential transducer, 59
serial method for digit
Set-valued features, 530
Sethi, R.,462

in MT,899
FSTas, 173
Settles, B.,738
Setzer, A.,756, 759, 762
Seuss, D.,45
Seymore, K.,112
Shafran, I.,357
Shah, P.K.,773
Shaked, T.,690,754, 755
Shakespeare
N-gram approximations to, 93author attribution, 122
word count, 86
Shallow parse, 455
shallow parse, 455
Shamir, E.,417,549
Shanahan, J.,775
Shannon, C. E.,9, 80,92, 117,
119, 334
shape
feature in IE, 742
Sharpley, A.,282
Shattuck-Hufnagel, S.,264
Shaw,G.B.,247
sheep language, 27
Sheil, B. A.,462
Shen, L.,81, 921
Shen, W.,359, 920
Sherlock, G.,772

119, 334
shape
Shieber, S.M.,7, 16,496, 501,
524, 530, 532–534, 544,
545, 619, 914
Shih, C.,81,274
Shillcock, R.,487
Shimodaira, H.,871
Shimoni, A.R.,775
Shimony, S.E.,726
Shipley, K.L.,284
Shitaoka, K.,700, 734
Shnayder, V.,679
Shockey, L.,247
Shopen, T.,230
Shoup, J.E.,216
SHRDLU,11,586
Shriberg, E.,85, 335,360, 361,
364, 422, 850,857, 858,
871
sibilant sound, 221
Sibun, P.,150
Siddharthan, A.,823
Sidner, C.L.,12,732, 868–871
Sigleman, S.,824
signal processing, 292
Signature terms, 805
signatures, 383
signiﬁcance tests

871
Signiﬁcant silence, 828
signiﬁcant silence, 828
Siler, S.,827
Silliman, S.,827
Sills, D.L.,617
Silverman, K.,267
Simma,A.,733
Simmons,R.F.,12,137, 169,
586, 587, 620,645, 683,
688, 690, 780,823
Simple Good-Turing smoothing,
102
Simple types, 527simple types, 527
Simpliﬁed Lesk, 658
Singer, Y., 156,160, 170, 491,
813
Singh, S.P.,844, 863
Single document summarization,
802
single initiative, 842
singleton
in authorship identiﬁcation,
122
singleton events, 101
singleton N-grams, 101

102
813
802
singleton
122
singular, 402
singular afﬁx, 48
singular value decomposition
in IR, 690
sink state in ﬁnite-state
automaton, 30
Situational context, 708
Sivertsen, E.,283
skip bigrams, 820
Skip N-gram, 113
Skut, W.,259
Slator, B.M.,688, 689
Sleator, D.,419, 491
slips of the tongue, 78
Slobin, D.I.,878
slot-ﬁller representations, 578
Slovene, 162
Small, S.L.,688
SMART, 822
SMDP,871
Smedt, K. D.,532
Smith, D. A.,492
Smith, I.,287
Smith, N. A.,492, 550
Smith, R. W.,871
Smith, V. L.,872
Smolensky, P.,371, 385, 387
Smoothing, 98

in IR, 690
SMART, 822
SMDP,871
smoothing, 97, 98
add-one,98
details of Katz backoff, 106
for HMM POStagging, 150
Good-Turing, 101
Laplace,98
linear interpolation, 104
Simple Good-Turing, 103
Smrˇ z, O.,81
Smyth, R.,730
Snippet,816
snippet,780
Snodgras, A.,357

DRAFTIndex 1005
Snover, M.,921
SNOWalgorithm, 170
Snow, R.,681,682, 688, 690
Snyder, L.,9
Social media, 775
sociolinguistic factors in
pronunciation, 229
Soderland, S.,766, 775
Somers,H. L.,880, 881, 887,
921
Song, Z.,243
sonority,377
sonority hierarchy, 377
Soon, W.M.,723
Soong, F.K.,339
Sorensen, J.,750
Soricut, R.,813, 823
Sound inference, 725
source-ﬁlter model, 241
Souter, C.,113
SOVlanguage, 877
space
as aregular character, 18
Spam detection, 824
Spanish, 241
Sparck Jones, K.,16,95, 688,
689, 785, 822, 824
Sparse data, 97
sparse data, 97
speaker adaptation, 356
Speaker identiﬁcation, 335
speaker identiﬁcation, 335
Speaker independent, 289
Speaker veriﬁcation, 335
speaker veriﬁcation, 335
speaker-independent, 289
species
spectral
features, 292
spectral subtraction, 355
Spectral tilt, 298,361
spectrogram, 239
Spectrum, 237
spectrum, 237
speech
telephone bandwidth, 232
wideband, 232
speech act, 829, 830
speech error, 78
acoustic modeltraining, 326,

species
spectral
speech
327
architecture, 289
continuous, 288
history of,333
isolated-word, 288
noisy channel model, 290
pruning, 325speaker independent, 289
word segmentation, 318
Speech synthesis, 249
speech synthesis, seeTTS
spelling correction
useof N-grams in, 84
spelling errors
correction
context-dependent, 73
EM,167
isolated-word, 73
noisy channel model, 165
detection
morphology, 73
non-word, 72
real words viaN-gram, 167
overview of detection and
correction, 72
real word, 73
societal role, 72
spelling rule

EM,167
detection
English consonant doubling,
50
spelling rules, 53, 63
Spiegel, M.F.,259
Split,475
Split and merge, 477
spoken dialogue system, 825
spoken English, grammar of,421
spoken language system, 825
Spooren, W.P.M.,734
Sporleder, C.,705,707, 733
Sproat, R.,39, 55, 56,81, 158,
254, 255, 266,285, 357,
371, 378, 542
spurious word
in MT,914
spurious words
in MT,894
SRILMtoolkit, 109
Srinivas, B.,491
Srivastava, R.,827
Stabler, E.,538
stack
for depth-ﬁrst search, 36
stack decoder, seeA∗decoder
Stack decoding, 905

50
Split,475
in MT,914
in MT,894
stack
Stallard, D.,359, 839, 852
Stalnaker, R. C.,830
Stanners, R.F.,78
Starr, R., 357
Start symbol, 393
start symbol, 393
state
accepting, 27
ﬁnal,27
in ﬁnite-state automaton, 27
state-space search, 35state-transition table
ﬁnite-state automaton, 28
states
Stationary, 299
stationary stochastic process,
116
statistical paradigm
riseof, 11
statistical signiﬁcance
MAPSSWEfor ASR,156,
331
McNemar test, 156, 331
Wilcoxon signed-rank test for
sequence models, 156

state
ﬁnal,27
states
116
riseof, 11
331
statistical vs symbolic
paradigms, 10
x expressions, 575
stative expressions, 575
Steedman, M.,268, 421, 428,
491, 538,713
Stefanowitsch, A.,646
stem,47
stemming, 79, 786
and morphological parsing, 46
stemming in IR, 68
Stent, A.,853, 858, 859,864
Stetina, J.,491
Stevens, K.N.,247,284
Stevens, S.S.,235,301
Stevenson, M.,688
Stevenson, R.,733
Stevenson, R. J.,730
Stevenson, S.,690, 691
Stickel, M.E.,726
Stifelman, L.J.,832, 850
Stochastic OT, 375
Stockham, T.G.J.,333
Stoics, 123
Stolcke, A.,85, 90,109, 112,
113, 120,316, 335, 343,

stem,47
360, 364,487, 490, 857,
858, 871
Stolz, W.S.,139
Stone, C.J.,213
Stone, M.,734
Stone, P.J.,688, 689
stop (consonant), 221
Stop list, 786
stop list,786
stoplist,672
stopword, 672
Stork, D.G.,213,306
Story of the Stone, 873
Streeter, L.,285
stress
lexical,224
secondary, 225
string, 395

DRAFT1006 Index
deﬁned assequence of
symbols, 18
string distance, 74
Strom, V.,284
strong equivalence
of grammars, 416
Strube, M.,690, 733
Structural ambiguity, 436
structurally ambiguous, 461
Strzalkowski, T.,483, 484, 823
Sturm, J.,860
Stuttle, M.,871
style
in phonetic variation, 230
Stylianou, Y.,284
Su,J.,748
Su,Y., 357
subcategorization, 390, 405,417,
426, 464, 506, 510
alternation, 425
subcategorization frame, 405,
405, 425
Subcategorization frames, 508
subcategorize for, 405
subdialogue, 869
knowledge precondition, 869
subject, syntactic, 390, 397,402,
426
frequency ofpronouns as, 472
subject-verb agreement
via uniﬁcation, 495
subject-verb agreement, 403
via uniﬁcation, 498
sublanguage, 876
subsequential transducer, 59
substitutability, 427
substitution
in TAG,427
substitutions
regular expressions, 25
Subsumption, 580
subsumption, 582
subsumption in uniﬁcation, 500
Subtype, 527
subtype,527
Suci, G.J.,689
sufﬁx, 47,79
Sum-of-products, 270
Sum-squared error, 196
summarization, 823

426
in TAG,427
generic,802
multiple document, 802
single document, 802
via sentence centrality, 805
Summary Content Unit, 820
Sundheim, B.,13,724, 756, 759,
762, 766, 774
Superordinate, 628
supersenses, 682Support Vector Machine, see
SVM
Suprasegmental, 263
Surdeanu, M.,690
surface form, 45
surface level in ﬁnite-state
surface tape, 61
SUS(semantically unpredictable
sentence), 282
Sutton, C., 214
Sutton, R. S.,863, 871
Sutton, S.,850
Suzuki, H.,810, 823
Suzuki, T.,284
Svartvik, J.,49, 127,428
SVM,204

SVM
SVM,204
SVO language, 877
Swedish, 878
Sweet, H.,246
Swerts, M.,857–860
Swier, R.,690
SwissGerman
cross-serial constraints, 544
Switchboard
Transcription Project, 243
Switchboard Corpus, 85, 86,
225, 229, 231,232, 242,
297, 318, 335
Syllabiﬁcation, 223
syllabiﬁcation, 373, 376
Maximum OnsetPrinciple,
377
accented, 224
coda,223
nucleus,223
onset,223
prominent, 224
rhyme,223
rime,223
syllables
complex codas, 372
complex onsets to, 372
symbolic vs statistical
Symmetrizing, 903

SVM,204
377
coda,223
onset,223
rhyme,223
rime,223
syllables
Synchronous grammar, 912
Synonym, 627
synonymy
and IR,790
Synset,630
Syntactic disambiguation, 437
Syntactic movement, 408
syntactic movement, 408
syntactic transformations, 885
Syntax,389
syntax,389
Syrdal, A. K.,282, 284, 285system initiative, 842
Systemic grammar, 407
systemic grammar, 407
t
914
t-test
for word similarity, 675
T.,387
tableau,373
tableau in Optimality Theory,
373
TAG,427, 491
tagger,seepart-of-speech tagger
CLAWS,130
tagging,seepart-of-speech

synonymy
and IR,790
t
914
t-test
T.,387
373
CLAWS,130
tagging,133
tags,seetagsets or
part-of-speech
taggers/tagging
tagset,123
Brown, 130
C5, 130
difference between C5 and
Penn Treebank, 132
difference between Penn
Treebank and Brown,
132
English, 130
history of Penn Treebank, 132
Penn Treebank, 130, 131
table of Penn Treebank tags,
131
Tajchman, G.,90,487
Talbot, D.,920
Talmy,L.,878
Tamil, 878
Tanabe, L.,772
Tanenhaus, M.,688
Tanimoto similarity metric, 677
Tannenbaum, P.H.,139,689
tap (phonetic), 222,226

Brown, 130
C5, 130
132
131
Tamil, 878
Tapanainen, P.,418,419
tape in ﬁnite-state automaton, 27
picture of,28
Tapered prompt, 840
Target cost, 279
Target point, 271
Tarte Tatin
recipe for, 697
Taskar, B., 491
tau Yih, W.,765
Taura, K.,532
Taussig, K.,357
taxonomy, 629
Taylor, L.,733

DRAFTIndex 1007
Taylor, P.,250, 264,268, 269,
272, 275, 283–285, 302,
303, 857, 858, 871
Taylor, W.L.,909
TBL,151, 171
painting metaphor for, 151
TBox,579
TD-PSOLA, 276, 277
technai, 123
Telephone-bandwidth, 297
telephone-bandwidth speech,
232
Telic eventualities, 578
Temperley, D.,419, 491
template
in IE,764
template, in TBL, 153
template-based generation, 840
Template-ﬁlling, 739
templatic morphology, 52, 370
temporal expressions
metaphors for, 575
temporal adverb, 126
Temporal analysis, 738
Temporal anchor, 760
temporal expression, 740
absolute, 755
relative, 755
Temporal expression
recognition, 738
Temporal logic, 573
temporal normalization, 758
temporal reasoning, 587
Tengi, R.I.,652
Tense logic, 573
ter Meulen, A.,16,541, 542,
549, 550, 587, 619
Teranishi, R., 284
term
in IR,782, 782
Term clustering, 792
term clustering, 688,689
Term weight, 782
term weighting, 785, 822
term weights
in IR,782
term-by-document matrix, 784
terminal symbol, 392, 425
Terminology
in description logic, 579

term
in IR,782
terms
in FOL,562
Tesar, B., 385,387
Tesni` ere, L.,418
Testset,91
development, 92
how to choose, 92
Teufel, S.,807,810, 824text analysis (stage of TTS), 250
Textcategorization, 824
text normalization
in TTS,250
text summarization, 802, 823
Text-to-speech, 249
text-to-speech synthesis, see
TTS
TF*IDF,266
tf-idf, 785
ThePirates of Penzance, 287
Thede, S.M.,150, 159
Thematic grid, 634
Thematic role, 633
thematic role, 632
and diathesis alternation, 635
examples of,633
in MT,888

terms
in FOL,562
in TTS,250
TTS
in MT,888
labeling, 683
problems, 635
Theme,633
theme, as thematic role, 633
there,129
thesaurus, 687
for query expansion in IR, 792
Thesaurus generation, 792
thesaurus generation, 792
thesaurus induction, 682
Thibaux, R.,477, 490
Thiessen, E.D.,387
Thione, G.L.,734
third-person, 402
Thomas,D.S.,14
Thomas,J.A.,113–115, 118
Thompson,H.,825, 836
Thompson,K.,42
Thompson,R. A.,465
Thompson,S.A.,285, 703, 734,
885
Thornton, J.H.S.,364
Thrax, D.,123
Thrun,S.,871
Tibshirani, R.,213
Tice, D.M.,13
tied states in HMM, 348
tier

Theme,633
there,129
885
tier
of ToBI transcript, 268
tier, morphological, 370
Tillmann, C.,419, 484, 920
Tiltmodel, 268
tilt parameter in, 268
tilt parameter in Tiltmodel, 268
time
representation of, 572
Time-aligned transcription, 243
time-synchronous beam search,
326
TimeBank, 762
TIMIT,243Tishby, N.,678, 689, 690
Titone, D.,377
Titov, I.,491
Tjong KimSang, E.F.,13
ToBI,267
boundary tones, 267
tiers, 268
Todaka, Y.,229
Tokenization, 47
tokenization
for N-grams, 86
in TTS,251
word,69
tokens
word,86
Tokuda, K.,285
Tolle, R.,772

time
326
ToBI,267
tiers, 268
in TTS,251
word,69
tokens
word,86
Tomabechi, H.,524
Tomalin, M.,364
Tomita, M.,920
Tomokiyo, L.M.,288, 357
Toneunit, 285
Top-down, 433
top-down, 431,433, 460
top-down parser, 433
topic (information structure),
472
Topic signature, 805
topic-based language models,
112
topicalization, 398
Topline,271
Touretzky, D.S.,380
Toutanova, K.,156,160, 168,
170, 491,532, 899
Towell, G.,651
trace, 398, 408
trachea,218
trailing unit method of digit
pronunciation, 256
Training set, 91
training set, 91,153
Tran, B.-H.,347

472
112
transcription, 421
time-aligned, 243
composition, 58, 58, 367
inversion, 58
sequential, 59
subsequential, 59
weighted, 378
Transduction grammar, 912
transfer model, 884
Transformation Based Learning,
seeTBL
transformation-based learning,
seeTBL

DRAFT1008 Index
transformation-based tagger, 135
Transformations and Discourse
Analysis Project
(TDAP),10, 169
transition probability, 148, 182,
185, 321, 322
transitions in ﬁnite-state
automaton, 27
transitive, 426
transitive verbs, 405
difﬁculty of literary, 874
impossibility of, 889
translation divergences, 876
Translation model, 890
Traum,D.R.,853–855, 871
TREC,790, 823
TreeAdjoining Grammar (TAG),
427
adjunction in, 427
substitution in, 427
Tree-Adjoining Grammar, 538
tree-structured lexicon, 347
Treebank, 408
treebank, 408, 471
Treiman, R.,377
trellis
Viterbi, 148
trie,382
trigger language model, 113
Trigram,91
trigram,91
triphone
HMM model for speech
recognition, 348
Trost,H.,387
Trubetskoi, N.S.,531
Truth-conditional semantics, 561
Tsakalidis, S.,343, 351
Tsang, V.,691
Tseng, H.,81, 160, 682
Tsochantaridis, I.,733
Tsujii, J.,532, 920
Tsukahara, W.,832
Tsuruoka, Y.,532
TTS,249
diphone, 273
diphone vs unit selection, 278
HMM, 285

trellis
trie,382
triphone
TTS,249
HMM, 285
hourglass metaphor for, 250
unit selection, 278
Tuantranont, J.,9
Tuﬁs,D.,162
Tukey, J.W.,333, 687
Tune,267
tune
ﬁnal fall, 267
question rise, 267tune:continuation rise, 267
T¨ ur,G.,161,162, 360, 364
Turian, J.P.,921
Turing equivalent, 537
Turing machine, 42, 536
as origin of ﬁnite automaton,
42
Turing test, 6,7
Turing, A.M.,6,9, 80
Turk
Mechanical, 249
Turk,A.,119
Turkish, 877
morphemes per word, 46
number of words,46
really long words in, 46
turn,827
turn correction ratio, 851

Tune,267
tune
42
Turk
turn,827
Turn-taking, 827
turn-taking, 827
Turner, J.,823, 824
Turney,P.,679
Tutiya, S.,854
two-level
morphology, 79,365, 367,386
lexical tape, 60
surface tape, 60
rule, 367
and Optimality Theory, 374
two-level morphology
feasible pairs, 61
Tyler, L.K.,78, 362
type 0 grammar, 536
Typehierarchy, 527
type hierarchy
example of for agr, 528
example of usefor
subcategorization frames,
529
typed feature structures, 527
appropriateness conditions
for, 527

two-level
rule, 367
529
for, 527
simple types, 527
what good are they anyhow?,
527
Types,527
types
word,86
Typology, 877
typology
linguistic, 877
Tyson,M.,766,767, 774
UCREL,130Uefﬁng, N.,920
uh
ﬁlled pause, 361, 422, 422
uh as ﬁlled pause, 423
Ullman, J.D.,38,39, 42, 81,
395, 434,461, 462, 538,
539, 541,549
um
ﬁlled pause, 422
um as ﬁlled pause, 423
Umeda, N.,284
UMLS,792
unaspirated sound, 226
ungrammatical sentences, 395
uniﬁcation, 497
[],498
algorithm, 513
grammar, 501

527
Types,527
types
word,86
typology
uh
um
UMLS,792
[],498
set-valued features in, 530
subsumption in, 500
uniﬁcation grammars, 404
union, 40
Unique beginner, 631
unit production, 441
Unit selection synthesis, 278
Universal, 842
universal of language, 877
UNIX, 17
¡UNK¿,95
penalty in MT, 919
unknown words
in N-grams, 95
in POS tagging, 158
open and closed-class words,
159
useof capitalization, 159
useof MaxEnt, 159
unrestricted, 536
unvoiced sound, 219
upper tape, 60
User generated content, 775
user initiative, 842
user-centered design, 850

union, 40
UNIX, 17
¡UNK¿,95
159
Usher, A.P.,iv
Uszkoreit, H.,532
Utsuro, T.,775
Utterance, 85,421,829
utterance, 85,421, 426,829
V (vocabulary), 164
Vagueness, 555
vagueness
ambiguity, 555
Valence,510
Valencia, A.,772

DRAFTIndex 1009
Valtchev, V.,316, 341,343
Value iteration, 863
van Benthem, J.,587
van Deemter, K.,709
Van Ess-Dykema, C.,857, 858,
871
van Halteren, H.,170,824
van Harmelen, F.,584
van Lehn, K.,619
van Noord, G.,387
van Rijsbergen, C.J.,459, 483,
744, 822
van Santen, J.P.H.,39, 270,
274, 285
van Vuuren, S.,9
van Wijnagaarden, A.,10, 427
van Zaanen, M.,492
Vanderwende, L.,620, 810, 823
VanLehn, K.,827
van den Berg, M.,734
van den Bosch, A.,284, 378,
379, 381
Van Son,R. J.J.H.,118
Van Valin, Jr.,428
Vapnik, V. N.,13,213
V´ aradi, T.,887
Varges, S.,871
Variable, 562
variable
existentially quantiﬁed, 565
universally quantiﬁed, 565
variable-length N-gram, 113
variables
in FOL,562
Variance, 308
variation
sociolinguistic, 229
Vasilescu, F.,659
Vasserman, A.,484, 491,532
Vauquois triangle, 881
Vauquois, B., 10,427, 882
Veale, T.,646
Veblen, T.,72
Vector length, 784
vector length, 677,784
vector model, 822
Vector quantization, 305
Vector space model, 782
vector space model, 779, 782,
821
Veenstra, J.,458

variable
variables
in FOL,562
variation
821
Veilleux, N.,264
velar sound, 221
Veldhuis, R.,276
Velichko, V.M.,333
velocity feature, 304
velum,221
Venditti, J.J.,268
Vendler, Z.,575
Venkataraman, A.,335Venkataramani, V., 243
Venugopal, A.,920
verb,125
modal,128,406
phrasal,126
Verb alternation, 635
Verb group, 407
verb phrase, 393,404
semantic attachments, 611
verb-framed language, 878
verbs
irregular, 49
irregular, number in English,
49
main verb, 49
modal verb, 49
primary verbs, 49
subclasses, 125
Verhagen, M.,762,775
Veriﬁability, 554

velum,221
verb,125
verbs
49
Vermeulen, P.J.E.,850
Veronis, J.,689
V´ eronis, J.,652
Verspoor, C. M.,773
vertex, 27
vertices in directed graphs, 27
Vidal, E.,380
Viegas, E.,645
Vieira, R.,733
Vietnamese, 877
Vijay-Shanker, K.,151, 538
Vilain, M.,724
Vintsyuk, T.K.,146, 212, 213,
333
Vitale, A.J.,256
Viterbi
decoding in MEMM,210
for semantics in dialog, 839
trellis, 148
Viterbi algorithm, 74, 184
and stack decoder, 344
backtrace in, 186
exercises modifying, 336
for letter-to-sound, 261
for POS tagging, 145
history of, 213

vertex, 27
333
Viterbi
in HMM alignment in MT,
899
in unit-selection synthesis,
280
limitations, 338
Viterbi approximation, 338
Viterbi backtrace, 186
Viterbi training, 329
Viterbi, A.J.,213
VITERBI ALGORITHM ,147,
186, 323
vocabularygrowth as function of token
count, 86
vocal
cords,seevocal folds
folds,218
tract,219
vocal cords, seevocal folds
vocal folds, 218
vocal tract, 219
Vogel, I.,285
Vogel, S.,920
Voice,275
Voice talent, 274
voiced sound, 219
voiceless sound, 219
VoiceXML, 846
Voiers, W.,282
Volkmann, J.,235, 301
von Kempelen, W.,249

899
280
186, 323
count, 86
vocal
folds,218
tract,219
Voice,275
von Neumann, J.,42
Voorhees, E.M.,13, 651, 790
Vossen, P.,688
Voutilainen, A.,135, 137–139,
155, 418,419, 428
vowel,219
back,222
front,222
harmony, 369
height,222
high,222
low,222
mid,222
reduced,224
rounded, 222
vowel harmony, 369
VP attachment, 473
VQ (vector quantization), 305
VSO language, 877
VTLN,357
VUI,850
vxml,846
W.A.,769, 771
Wade, E.,850,858
Wagner, R.A.,74, 170, 213
Wagstaff, K.,717, 733
Wahlster, W.,584
Waibel, A.,357, 871, 920
Wakao, T.,766
Waksler, R.,78
Wald, B.,230
Walker, K.,334

vowel,219
back,222
front,222
high,222
low,222
mid,222
VTLN,357
VUI,850
vxml,846
Walker, M.A. etal., 825
Walker, M.A.,718,733, 840,
842, 844,845, 851, 852,
854, 860,863, 871
Wall Street Journal
Wall, R. E.,16, 541, 542, 549,
550, 587

DRAFT1010 Index
Wallace, D. L.,11, 121,139, 290
Waltz, D.L.,646
Wang, M.Q.,264,364, 850
Wang, W.,359, 364
Wang, W.S.-Y.,283
Wang, Y.Y.,920
Wang, Z.,357
Wanner, E.,547
WANT()in BDI model, 865
Ward,N.,832, 920
Ward,T.,910, 921
Ward,W.,356, 358,690, 837
Warnke, V., 871
Warping, 333
warping, 333
Warren, D.H.D.,11, 531
Warren, R. M.,362
Warwick, S.,920
Wasow,T.,428,511, 530
Wattarujeekrit, T.,773
Waugh, L.R.,879
waveﬁle format, 233
waveform synthesis (stage of
TTS),250
weak equivalence
Weaver, W.,652, 687,919
Web Ontology Language, 584
web site for book, xxv
Webber, B. L.,13, 16, 532,708,
709, 732, 734
Weber, D.J.,80
Weber, S.,870
Weeds, J.,670, 689, 690
Weenink, D.,236, 244, 247
Wegmann, S.,364
Wegstein, J.H.,10,427
weighted, 144
weighted ﬁnite-state automaton,
144, 174
weighted FSA, 174
weighted FST, 378
weighted mutual information,
689
weighted transducer, 378
Weinschenk, S.,846
Weinstein, C.,359
Weinstein, J.N.,772
Weinstein, S.,695, 718, 733
Weintraub, M.,339, 340, 357
Weir, D.J.,538, 689

144, 174
689
Weischedel, R.,150, 159, 170,
482, 688, 766, 775, 839
Weiss,S.F.,382
Weizenbaum, J.,7,17, 25
Weld, D.S.,690,754, 755
well-formed substring table, 461
well-formedness constraint, 386
Welling, L.,364
Wellner, B.,771Wells, J.C.,224, 247
Welsh,A.,362, 363
Weng,F.,871
Wessels,L.F.A.,850
Wester, M.,364
WFST,461
Wh,128
wh-non-subject question, 397
wh-non-subject-question, 511
semantic attachments, 607
wh-phrase, 397, 397
wh-pronoun, 128
wh-questions, 396, 426
wh-subject-questions, 397
Wh-word, 397

WFST,461
Wh,128
wh-word, 397
Wheeler, D.W.,380, 428
White, E.K.,771
Whiteside, J.A.,850
Whittaker, E.W.D.,111
Whittaker, S.,842, 844, 852,
854, 871
Whitton, L.,266
Wicentowski, R.,384
Wideband, 297
wideband speech, 232
Wiebe, J.,651, 775
Wiemer-Hastings, P.,698, 733
Wierzbicka, A.,641, 643, 646
Wightman, C.W.,267,284
Wilcoxon signed-rank test, 156
wildcard
regular expression, 20
Wilde, O.,72
Wilensky, R.,11,620, 864, 870
Wilkes-Gibbs, D.,871
Wilks, Y.,11, 587, 620,646,
660, 688, 689,766, 921
Willett, P.,822
Williams, C.E.,282

854, 871
wildcard
Williams, E.,359
Williams, J.,690
Williams, J.D.,871
Williams, R.,620, 774
Wilson, C.,387
Wilson, G.,756, 759
Wilson, R.,827
Winnow algorithm, 170
Winograd, T.,11, 585–587, 725,
732, 735, 825,836
Wir´ en, M.,837
Wise, B.,9
Witten, I.H.,120, 213, 482
Wittgenstein, L.,829
Wixon, D.R.,850
Wizard-of-Oz, 850
Wolf,A.K.,586, 823
Wolf,F.,359,734
Wolfram,W.A.,229Wong, A.K.C.,213
Wong, W.,920
Woodger, M.,10,427
Woodland, P.C.,111, 113, 281,
316, 341,343, 348–352,
354, 356–358, 364
Woods,W.A.,12,586, 587,608,
619, 732,823

Wise, B.,9
Wooters, C.,90, 487
word
boundaries
regular expression notation
for, 21
classes,seepart-of-speech,
123
count in Shakespeare, 86
deﬁnition of, 84
error
evaluation for speech
recognition, 330
fragment, 85,422
function, 125, 168
how many in English, 86
prediction, 83
punctuation as, 85
tokens,86
types,86
types in Hungarian, Turkish
and English, 161
Word alignment, 894
word alignment, 894
in MT,893
Word association, 362
word boundary, 61
Word error, 330
word error rate, 288

word
for, 21
123
error
tokens,86
types,86
in MT,893
Word Grammar, 428
word graph in ASR, 342
Word insertion penalty, 317
word insertion penalty, 317
word penalty
in MT,919
word relatedness, 665
word segmentation, 69
Word sense, 624
Word sensedisambiguation, 556
word sensedisambiguation, 650,
seeWSD,seeWSD
Word senses, 556
word similarity, 664
word tokenization, 69
Wordform, 85
wordform
and lemma, 623
versus lemma, 85
WordNet, 629, 630, 809
Wordnet::Similarity, 669, 691

DRAFTIndex 1011
world knowledge, 551
Woszczyna, M.,871
Wouters, J.,282
WOZ,seeWizard-of-Oz system
Wright, H.,871
Wright, J.H.,826
Wright, R. N.,428
WSD,649, 650,687
AI-oriented efforts, 688
all-words task, 651
bootstrapping, 662, 688
connectionist approaches, 688
decision list approach, 653,
655
decision tree approach, 688
evaluation of, 656
history of,689
in IR,790
in Machine Translation, 876
lexical sample task, 651
limitations of selectional
restrictions, 660
naive Bayes approach, 653
robust approach, 688
supervised machine learning,
688
unsupervised machine
learning, 686
use ofbag-of-words features,
652
use ofcollocational features,
652
use ofmachine readable
dictionaries, 688, 689
use ofselectional association,
661
Wu,D.,620,689, 914, 920
Wu,J.,163
Wu,Z.,666
Wundt, W.,391, 426
Wunsch, C.D.,213
X-bar schemata, 427X-schema, 585
Xia, F.,419
Xiang, B.,359
Xiuyang, Y.,358
Xu,K.,871
Xu,P.,487
Xu,W.,826
Xue, N.,81, 690
Y.,10, 247
Yale School of AI,11
Yallop, C.,247
Yamada, H.,491
Yamada, K.,883, 886, 912, 914,

688
652
652
661
Wu,J.,163
Wu,Z.,666
Xu,K.,871
Xu,P.,487
Xu,W.,826
Y.,10, 247
920
Yamada, T.,285
Yang, B.,792
Yang, Y.,250
Yankelovich, N.,829, 832,846,
850
Yarowsky algorithm, 662
Yarowsky, D.,155, 171, 257,
384, 655–657, 662, 663,
688
Yates, A.,690, 754, 755
Yates, J.,730
Yawelmani, 369, 371
Yeh, A.,771
yes-no question, 397
yes-no-questions, 396, 403, 426,
855
Yi, L.,243
Yield,467
Yngve, V. H.,434, 461, 542,
546, 547, 832
Yokuts, 369
Yonkers Racetrack, 114
Yoon,S.-Y.,357
Yoshimura, T.,285
Young,M.,529
Young,S. J.,316, 335, 339, 343,
348–350, 352,364, 471,

920
850
688
855
Yi, L.,243
Yield,467
492, 871Younger, D.H.,431, 461
Yuan, J.,266
Yupik, 877
Yuret, D.,492, 660
Yvon, F.,284
Z
normalization factor in
MaxEnt, 201
Zacharski, R.,712, 733
Zaenen, A.,532, 823
Zagoruyko, N. G.,333
Zajic, D.,810, 823
Zamora, A.,823
Zappa, F.,216
Zappa, M.U.,216
Zavrel, J.,162, 170, 491
Zelenko, D.,170
Zelle, J.,689
Zen, H.,285
Zens, R.,920
Zernik, U.,620,689, 823
zeugma,626
Zhang, H.,914
Zhang, J.,748
Zhang, M.,748
Zhang, Q.,871
Zhang, Z.,823
Zhao, S.,690
Zheng, F.,243
Zheng, Y., 357
Zhou, G.,113,748
Zhou, H.,357

Yupik, 877
Z
Zhou, J.,357
Zhou, M.,690
Zhu, W.-J.,910,921
Zhu, X.,112
Ziff-Davis corpus, 807, 809
Zimak, D.,462
Zissman, M.,359
Zue, V. W.,243, 850, 851,871
Zweig, G.,364
Zwicky, A.,229, 555

Aurélien GéronHands-on Machine Learning with
Scikit-Learn, Keras, and
TensorFlow
Concepts, Tools, and Techniques to
Build Intelligent SystemsSECOND EDITIONBostonFarnhamSebastopolTokyoBeijingBostonFarnhamSebastopolTokyoBeijing

978-1-492-03264-9
[LSI]Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow
by Aurélien Géron
Copyright © 2019 Aurélien Géron. All rights reserved.
Printed in the United States of America.
Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.
O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are
also available for most titles ( http://oreilly.com ). For more information, contact our corporate/institutional
sales department: 800-998-9938 or corporate@oreilly.com .Editor:  Nicole Tache
Interior Designer:  David FutatoCover Designer:  Karen Montgomery
Illustrator:  Rebecca DemarestJune 2019:
 Second Edition
Revision History for the Early Release
2018-11-05: First Release
2019-01-24: Second Release
2019-03-07: Third Release
2019-03-29: Fourth Release
2019-04-22: Fifth Release
See http://oreilly.com/catalog/errata.csp?isbn=9781492032649  for release details.

The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Hands-on Machine Learning with
Scikit-Learn, Keras, and TensorFlow , the cover image, and related trade dress are trademarks of O’Reilly
Media, Inc.
While the publisher and the author have used good faith efforts to ensure that the information and
instructions contained in this work are accurate, the publisher and the author disclaim all responsibility
for errors or omissions, including without limitation responsibility for damages resulting from the use of

or reliance on this work. Use of the information and instructions contained in this work is at your own
risk. If any code samples or other technology this work contains or describes is subject to open source
licenses or the intellectual property rights of others, it is your responsibility to ensure that your use
thereof complies with such licenses and/or rights.

Table of Contents
Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  xi
Part I. The Fundamentals of Machine Learning1.
The Machine Learning Landscape. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  3
What Is Machine Learning?                                                                                           4
Why Use Machine Learning?                                                                                         4
Types of Machine Learning Systems                                                                             8
Supervised/Unsupervised Learning                                                                           8
Batch and Online Learning                                                                                       15
Instance-Based Versus Model-Based Learning                                                      18

Main Challenges of Machine Learning                                                                       24
Insufficient Quantity of Training Data                                                                   24
Nonrepresentative Training Data                                                                            26
Poor-Quality Data                                                                                                      27
Irrelevant Features                                                                                                     27

Overfitting the Training Data                                                                                   28
Underfitting the Training Data                                                                                30
Stepping Back                                                                                                             30
Testing and Validating                                                                                                   31

Hyperparameter Tuning and Model Selection                                                       32
Data Mismatch                                                                                                           33
Exercises                                                                                                                          342.
End-to-End Machine Learning Project. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  37

Working with Real Data                                                                                                38
Look at the Big Picture                                                                                                  39iii

Frame the Problem                                                                                                    39
Select a Performance Measure                                                                                  42
Check the Assumptions                                                                                             45
Get the Data                                                                                                                    45
Create the Workspace                                                                                                45
Download the Data                                                                                                    49
Take a Quick Look at the Data Structure                                                                50
Create a Test Set                                                                                                          54

Discover and Visualize the Data to Gain Insights                                                     58
Visualizing Geographical Data                                                                                 59
Looking for Correlations                                                                                           62
Experimenting with Attribute Combinations                                                        65
Prepare the Data for Machine Learning Algorithms                                                66

Data Cleaning                                                                                                             67
Handling Text and Categorical Attributes                                                              69
Custom Transformers                                                                                                71
Feature Scaling                                                                                                            72
Transformation Pipelines                                                                                          73

Select and Train a Model                                                                                               75
Training and Evaluating on the Training Set                                                         75
Better Evaluation Using Cross-Validation                                                              76
Fine-Tune Y our Model                                                                                                  79

Grid Search                                                                                                                 79
Randomized Search                                                                                                   81
Ensemble Methods                                                                                                     82
Analyze the Best Models and Their Errors                                                             82

Evaluate Y our System on the Test Set                                                                      83
Launch, Monitor, and Maintain Y our System                                                            84
Try It Out!                                                                                                                       85
Exercises                                                                                                                          853.

Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  87
MNIST                                                                                                                             87
Training a Binary Classifier                                                                                          90
Performance Measures                                                                                                  90

MNIST                                                                                                                             87
Measuring Accuracy Using Cross-Validation                                                        91
Confusion Matrix                                                                                                       92
Precision and Recall                                                                                                   94
Precision/Recall Tradeoff                                                                                          95
The ROC Curve                                                                                                          99

Multiclass Classification                                                                                             102
Error Analysis                                                                                                              104iv | Table of Contents

Multilabel Classification                                                                                             108
Multioutput Classification                                                                                          109
Exercises                                                                                                                        1104.
Training Models. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  113
Linear Regression                                                                                                        114
The Normal Equation                                                                                              116
Computational Complexity                                                                                    119
Gradient Descent                                                                                                         119

Batch Gradient Descent                                                                                           123
Stochastic Gradient Descent                                                                                   126
Mini-batch Gradient Descent                                                                                 129
Polynomial Regression                                                                                                130

Learning Curves                                                                                                           132
Regularized Linear Models                                                                                         136
Ridge Regression                                                                                                      137
Lasso Regression                                                                                                      139

Elastic Net                                                                                                                 142
Early Stopping                                                                                                          142
Logistic Regression                                                                                                      144
Estimating Probabilities                                                                                          144

Training and Cost Function                                                                                   145
Decision Boundaries                                                                                                146
Softmax Regression                                                                                                  149
Exercises                                                                                                                        1535.

Support Vector Machines. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  155
Linear SVM Classification                                                                                          155
Soft Margin Classification                                                                                       156
Nonlinear SVM Classification                                                                                   159

Polynomial Kernel                                                                                                   160
Adding Similarity Features                                                                                     161
Gaussian RBF Kernel                                                                                               162
Computational Complexity                                                                                    163

SVM Regression                                                                                                           164
Under the Hood                                                                                                           166
Decision Function and Predictions                                                                       166
Training Objective                                                                                                   167

Quadratic Programming                                                                                         169
The Dual Problem                                                                                                    170
Kernelized SVM                                                                                                       171
Online SVMs                                                                                                            174Table of Contents | v

Exercises                                                                                                                        175 6.
Decision Trees. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  177
Training and Visualizing a Decision Tree                                                                177
Making Predictions                                                                                                     179
Estimating Class Probabilities                                                                                   181
The CART Training Algorithm                                                                                 182
Computational Complexity                                                                                        183
Gini Impurity or Entropy?                                                                                         183

Regularization Hyperparameters                                                                              184
Regression                                                                                                                     185
Instability                                                                                                                      188
Exercises                                                                                                                        1897.

Ensemble Learning and Random Forests. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  191
Voting Classifiers                                                                                                         192
Bagging and Pasting                                                                                                    195
Bagging and Pasting in Scikit-Learn                                                                     196

Out-of-Bag Evaluation                                                                                            197
Random Patches and Random Subspaces                                                                198
Random Forests                                                                                                           199
Extra-Trees                                                                                                                200

Feature Importance                                                                                                  200
Boosting                                                                                                                        201
AdaBoost                                                                                                                   202
Gradient Boosting                                                                                                    205

Stacking                                                                                                                         210
Exercises                                                                                                                        2138.
Dimensionality Reduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  215
The Curse of Dimensionality                                                                                     216

Main Approaches for Dimensionality Reduction                                                   218
Projection                                                                                                                  218
Manifold Learning                                                                                                   220
PCA                                                                                                                                222

PCA                                                                                                                                222
Preserving the Variance                                                                                          222
Principal Components                                                                                            223
Projecting Down to d Dimensions                                                                        224
Using Scikit-Learn                                                                                                    224

Explained Variance Ratio                                                                                        225
Choosing the Right Number of Dimensions                                                       225
PCA for Compression                                                                                             226vi | Table of Contents

Randomized PCA                                                                                                    227
Incremental PCA                                                                                                     227
Kernel PCA                                                                                                                   228
Selecting a Kernel and Tuning Hyperparameters                                                229
LLE                                                                                                                                 232
Other Dimensionality Reduction Techniques                                                         234
Exercises                                                                                                                        2359.
Unsupervised Learning Techniques. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  237

LLE                                                                                                                                 232
Clustering                                                                                                                      238
K-Means                                                                                                                    240
Limits of K-Means                                                                                                   250
Using clustering for image segmentation                                                             251

Using Clustering for Preprocessing                                                                       252
Using Clustering for Semi-Supervised Learning                                                 254
DBSCAN                                                                                                                   256
Other Clustering Algorithms                                                                                 259

DBSCAN                                                                                                                   256
Gaussian Mixtures                                                                                                       260
Anomaly Detection using Gaussian Mixtures                                                     266
Selecting the Number of Clusters                                                                          267
Bayesian Gaussian Mixture Models                                                                      270
Other Anomaly Detection and Novelty Detection Algorithms                        274
Part II. Neural Networks and Deep Learning10.

Introduction to Artificial  Neural Networks with Keras. . . . . . . . . . . . . . . . . . . . . . . . . .  277
From Biological to Artificial Neurons                                                                      278
Biological Neurons                                                                                                   279
Logical Computations with Neurons                                                                    281
The Perceptron                                                                                                         281

Multi-Layer Perceptron and Backpropagation                                                    286
Regression MLPs                                                                                                      289
Classification MLPs                                                                                                 290
Implementing MLPs with Keras                                                                                292

Installing TensorFlow 2                                                                                           293
Building an Image Classifier Using the Sequential API                                     294
Building a Regression MLP Using the Sequential API                                       303
Building Complex Models Using the Functional API                                        304
Building Dynamic Models Using the Subclassing API                                       309

Saving and Restoring a Model                                                                                311
Using Callbacks                                                                                                        311Table of Contents | vii

Visualization Using TensorBoard                                                                          313
Fine-Tuning Neural Network Hyperparameters                                                     315
Number of Hidden Layers                                                                                      319
Number of Neurons per Hidden Layer                                                                 320
Learning Rate, Batch Size and Other Hyperparameters                                     320
Exercises                                                                                                                        32211.
Training Deep Neural Networks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  325
Vanishing/Exploding Gradients Problems                                                              326
Glorot and He Initialization                                                                                   327

Nonsaturating Activation Functions                                                                     329
Batch Normalization                                                                                                333
Gradient Clipping                                                                                                    338
Reusing Pretrained Layers                                                                                          339

Transfer Learning With Keras                                                                                341
Unsupervised Pretraining                                                                                       343
Pretraining on an Auxiliary Task                                                                           344
Faster Optimizers                                                                                                         344

Momentum Optimization                                                                                      345
Nesterov Accelerated Gradient                                                                              346
AdaGrad                                                                                                                    347
RMSProp                                                                                                                   349

Adam and Nadam Optimization                                                                           349
Learning Rate Scheduling                                                                                       352
Avoiding Overfitting Through Regularization                                                        356
ℓ1 and ℓ2 Regularization                                                                                           356

Dropout                                                                                                                     357
Monte-Carlo (MC) Dropout                                                                                  360
Max-Norm Regularization                                                                                      362
Summary and Practical Guidelines                                                                           363

Exercises                                                                                                                        36412.
Custom Models and Training with TensorFlow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  367
A Quick Tour of TensorFlow                                                                                     368
Using TensorFlow like NumPy                                                                                  371

Tensors and Operations                                                                                          371
Tensors and NumPy                                                                                                373
Type Conversions                                                                                                     374
Variables                                                                                                                    374

Other Data Structures                                                                                             375
Customizing Models and Training Algorithms                                                      376
Custom Loss Functions                                                                                           376viii | Table of Contents

Saving and Loading Models That Contain Custom Components                    377
Custom Activation Functions, Initializers, Regularizers, and Constraints     379
Custom Metrics                                                                                                        380
Custom Layers                                                                                                          383
Custom Models                                                                                                        386
Losses and Metrics Based on Model Internals                                                     388
Computing Gradients Using Autodiff                                                                   389
Custom Training Loops                                                                                          393
TensorFlow Functions and Graphs                                                                           396

Autograph and Tracing                                                                                           398
TF Function Rules                                                                                                    40013.
Loading and Preprocessing Data with TensorFlow. . . . . . . . . . . . . . . . . . . . . . . . . . . . .  403
The Data API                                                                                                                404
Chaining Transformations                                                                                      405

Shuffling the Data                                                                                                    406
Preprocessing the Data                                                                                            409
Putting Everything Together                                                                                  410
Prefetching                                                                                                                411

Using the Dataset With tf.keras                                                                              413
The TFRecord Format                                                                                                414
Compressed TFRecord Files                                                                                   415
A Brief Introduction to Protocol Buffers                                                              415

TensorFlow Protobufs                                                                                             416
Loading and Parsing Examples                                                                              418
Handling Lists of Lists Using the SequenceExample  Protobuf                          419
The Features API                                                                                                         420
Categorical Features                                                                                                 421

Crossed Categorical Features                                                                                 421
Encoding Categorical Features Using One-Hot Vectors                                    422
Encoding Categorical Features Using Embeddings                                            423
Using Feature Columns for Parsing                                                                      426
Using Feature Columns in Y our Models                                                               426

TF Transform                                                                                                               428
The TensorFlow Datasets (TFDS) Project                                                                42914.
Deep Computer Vision Using Convolutional Neural Networks. . . . . . . . . . . . . . . . . . .  431
The Architecture of the Visual Cortex                                                                     432

Convolutional Layer                                                                                                    434
Filters                                                                                                                         436
Stacking Multiple Feature Maps                                                                             437
TensorFlow Implementation                                                                                  439Table of Contents | ix

Memory Requirements                                                                                           441
Pooling Layer                                                                                                                442
TensorFlow Implementation                                                                                  444
CNN Architectures                                                                                                      446
LeNet-5                                                                                                                      449
AlexNet                                                                                                                      450
GoogLeNet                                                                                                                452
VGGNet                                                                                                                     456

VGGNet                                                                                                                     456
ResNet                                                                                                                        457
Xception                                                                                                                    459
SENet                                                                                                                         461
Implementing a ResNet-34 CNN Using Keras                                                        464

ResNet                                                                                                                        457
SENet                                                                                                                         461
Using Pretrained Models From Keras                                                                      465
Pretrained Models for Transfer Learning                                                                 467
Classification and Localization                                                                                  469
Object Detection                                                                                                          471

Fully Convolutional Networks (FCNs)                                                                 473
Y ou Only Look Once (YOLO)                                                                                475
Semantic Segmentation                                                                                               478
Exercises                                                                                                                        482x | Table of Contents

1Available on Hinton’s home page at http://www.cs.toronto.edu/~hinton/ .2
Despite the fact that Y ann Lecun’s deep convolutional neural networks had worked well for image recognition
since the 1990s, although they were not as general purpose.Preface
The Machine Learning Tsunami
In 2006, Geoffrey Hinton et al. published a paper1 showing how to train a deep neural
network capable of recognizing handwritten digits with state-of-the-art precision
(>98%). They branded this technique “Deep Learning. ” Training a deep neural net
was widely considered impossible at the time,2 and most researchers had abandoned
the idea since the 1990s. This paper revived the interest of the scientific community
and before long many new papers demonstrated that Deep Learning was not only
possible, but capable of mind-blowing achievements that no other Machine Learning
(ML) technique could hope to match (with the help of tremendous computing power
and great amounts of data). This enthusiasm soon extended to many other areas of

Machine Learning.
Fast-forward 10 years and Machine Learning has conquered the industry: it is now at
the heart of much of the magic in today’s high-tech products, ranking your web
search results, powering your smartphone’s speech recognition, recommending vid‐
eos, and beating the world champion at the game of Go. Before you know it, it will be
driving your car.
Machine Learning in Your Projects
So naturally you are excited about Machine Learning and you would love to join the
party!

party!
Perhaps you would like to give your homemade robot a brain of its own? Make it rec‐
ognize faces? Or learn to walk around?xi

Or maybe your company has tons of data (user logs, financial data, production data,
machine sensor data, hotline stats, HR reports, etc.), and more than likely you could
unearth some hidden gems if you just knew where to look; for example:•
Segment customers and find the best marketing strategy for each group•
Recommend products for each client based on what similar clients bought•
Detect which transactions are likely to be fraudulent•
Forecast next year’s revenue•
And more
Whatever the reason, you have decided to learn Machine Learning and implement it
in your projects. Great idea!
Objective and Approach
This book assumes that you know close to nothing about Machine Learning. Its goal
is to give you the concepts, the intuitions, and the tools you need to actually imple‐
ment programs capable of learning from data .
We will cover a large number of techniques, from the simplest and most commonly
used (such as linear regression) to some of the Deep Learning techniques that regu‐
larly win competitions.

Rather than implementing our own toy versions of each algorithm, we will be using
actual production-ready Python frameworks:•
Scikit-Learn  is very easy to use, yet it implements many Machine Learning algo‐
rithms efficiently, so it makes for a great entry point to learn Machine Learning.•
TensorFlow  is a more complex library for distributed numerical computation. It
makes it possible to train and run very large neural networks efficiently by dis‐
tributing the computations across potentially hundreds of multi-GPU servers.

TensorFlow was created at Google and supports many of their large-scale
Machine Learning applications. It was open sourced in November 2015.•
Keras  is a high level Deep Learning API that makes it very simple to train and
run neural networks. It can run on top of either TensorFlow, Theano or Micro‐
soft Cognitive Toolkit (formerly known as CNTK). TensorFlow comes with its
own implementation of this API, called tf.keras , which provides support for some
advanced TensorFlow features (e.g., to efficiently load data).

The book favors a hands-on approach, growing an intuitive understanding of
Machine Learning through concrete working examples and just a little bit of theory.
While you can read this book without picking up your laptop, we highly recommendxii | Preface

you experiment with the code examples available online as Jupyter notebooks at
https://github.com/ageron/handson-ml2 .
Prerequisites
This book assumes that you have some Python programming experience and that you
are familiar with Python’s main scientific libraries, in particular NumPy , Pandas , and
Matplotlib .
Also, if you care about what’s under the hood you should have a reasonable under‐
standing of college-level math as well (calculus, linear algebra, probabilities, and sta‐
tistics).
If you don’t know Python yet, http://learnpython.org/  is a great place to start. The offi‐
cial tutorial on python.org  is also quite good.
If you have never used Jupyter, Chapter 2  will guide you through installation and the
basics: it is a great tool to have in your toolbox.
If you are not familiar with Python’s scientific libraries, the provided Jupyter note‐
books include a few tutorials. There is also a quick math tutorial for linear algebra.
Roadmap

tistics).
Roadmap
This book is organized in two parts. Part I, The Fundamentals of Machine Learning ,
covers the following topics:•
What is Machine Learning? What problems does it try to solve? What are the
main categories and fundamental concepts of Machine Learning systems?•
The main steps in a typical Machine Learning project.•
Learning by fitting a model to data.•
Optimizing a cost function.•
Handling, cleaning, and preparing data.•
Selecting and engineering features.•
Selecting a model and tuning hyperparameters using cross-validation.•

The main challenges of Machine Learning, in particular underfitting and overfit‐
ting (the bias/variance tradeoff).•
Reducing the dimensionality of the training data to fight the curse of dimension‐
ality.•
Other unsupervised learning techniques, including clustering, density estimation
and anomaly detection.Preface | xiii

• The most common learning algorithms: Linear and Polynomial Regression,
Logistic Regression, k-Nearest Neighbors, Support Vector Machines, Decision
Trees, Random Forests, and Ensemble methods.xiv | Preface

Part II, Neural Networks and Deep Learning , covers the following topics: •
What are neural nets? What are they good for?•
Building and training neural nets using TensorFlow and Keras.•
The most important neural net architectures: feedforward neural nets, convolu‐
tional nets, recurrent nets, long short-term memory (LSTM) nets, autoencoders
and generative adversarial networks (GANs).•
Techniques for training deep neural nets.•
Scaling neural networks for large datasets.•
Learning strategies with Reinforcement Learning.•
Handling uncertainty with Bayesian Deep Learning.
The first part is based mostly on Scikit-Learn while the second part uses TensorFlow
and Keras.
Don’t jump into deep waters too hastily: while Deep Learning is no
doubt one of the most exciting areas in Machine Learning, you
should master the fundamentals first. Moreover, most problems
can be solved quite well using simpler techniques such as Random
Forests and Ensemble methods (discussed in Part I ). Deep Learn‐

and Keras.
ing is best suited for complex problems such as image recognition,
speech recognition, or natural language processing, provided you
have enough data, computing power, and patience.
Other Resources
Many resources are available to learn about Machine Learning. Andrew Ng’s ML
course on Coursera  and Geoffrey Hinton’s course on neural networks and Deep
Learning  are amazing, although they both require a significant time investment
(think months).
There are also many interesting websites about Machine Learning, including of

course Scikit-Learn’s exceptional User Guide . Y ou may also enjoy Dataquest , which
provides very nice interactive tutorials, and ML blogs such as those listed on Quora .
Finally, the Deep Learning website  has a good list of resources to learn more.
Of course there are also many other introductory books about Machine Learning, in
particular:•
Joel Grus, Data Science from Scratch  (O’Reilly). This book presents the funda‐
mentals of Machine Learning, and implements some of the main algorithms in
pure Python (from scratch, as the name suggests).Preface | xv

• Stephen Marsland, Machine Learning: An Algorithmic Perspective  (Chapman and
Hall). This book is a great introduction to Machine Learning, covering a wide
range of topics in depth, with code examples in Python (also from scratch, but
using NumPy).•
Sebastian Raschka, Python Machine Learning  (Packt Publishing). Also a great
introduction to Machine Learning, this book leverages Python open source libra‐
ries (Pylearn 2 and Theano).•
François Chollet, Deep Learning with Python  (Manning). A very practical book
that covers a large range of topics in a clear and concise way, as you might expect
from the author of the excellent Keras library. It favors code examples over math‐
ematical theory.•
Y aser S. Abu-Mostafa, Malik Magdon-Ismail, and Hsuan-Tien Lin, Learning from
Data  (AMLBook). A rather theoretical approach to ML, this book provides deep
insights, in particular on the bias/variance tradeoff (see Chapter 4 ).•
Stuart Russell and Peter Norvig, Artificial  Intelligence: A Modern Approach, 3rd

Edition  (Pearson). This is a great (and huge) book covering an incredible amount
of topics, including Machine Learning. It helps put ML into perspective.
Finally, a great way to learn is to join ML competition websites such as Kaggle.com
this will allow you to practice your skills on real-world problems, with help and
insights from some of the best ML professionals out there.
Conventions Used in This Book
The following typographical conventions are used in this book:
Italic

Italic
Indicates new terms, URLs, email addresses, filenames, and file extensions.
Constant width
Used for program listings, as well as within paragraphs to refer to program ele‐
ments such as variable or function names, databases, data types, environment
variables, statements and keywords.
Constant width bold
Shows commands or other text that should be typed literally by the user.
Constant width italic
Shows text that should be replaced with user-supplied values or by values deter‐
mined by context.xvi | Preface

This element signifies a tip or suggestion.
This element signifies a general note.
This element indicates a warning or caution.
Code Examples
Supplemental material (code examples, exercises, etc.) is available for download at
https://github.com/ageron/handson-ml2 . It is mostly composed of Jupyter notebooks.
Some of the code examples in the book leave out some repetitive sections, or details
that are obvious or unrelated to Machine Learning. This keeps the focus on the
important parts of the code, and it saves space to cover more topics. However, if you
want the full code examples, they are all available in the Jupyter notebooks.
Note that when the code examples display some outputs, then these code examples
are shown with Python prompts ( >>> and ...), as in a Python shell, to clearly distin‐
guish the code from the outputs. For example, this code defines the square()  func‐
tion then it computes and displays the square of 3:
>>> def square(x):
...     return x ** 2
...
>>> result = square(3)
>>> result
9

...
>>> result
9
When code does not display anything, prompts are not used. However, the result may
sometimes be shown as a comment like this:
def square(x):
    return x ** 2
result = square(3)  # result is 9Preface | xvii

Using Code Examples
This book is here to help you get your job done. In general, if example code is offered
with this book, you may use it in your programs and documentation. Y ou do not
need to contact us for permission unless you’re reproducing a significant portion of
the code. For example, writing a program that uses several chunks of code from this
book does not require permission. Selling or distributing a CD-ROM of examples
from O’Reilly books does require permission. Answering a question by citing this
book and quoting example code does not require permission. Incorporating a signifi‐
cant amount of example code from this book into your product’s documentation does
require permission.
We appreciate, but do not require, attribution. An attribution usually includes the
title, author, publisher, and ISBN. For example: “ Hands-On Machine Learning with
Scikit-Learn, Keras and TensorFlow  by Aurélien Géron (O’Reilly). Copyright 2019

Aurélien Géron, 978-1-492-03264-9. ” If you feel your use of code examples falls out‐
side fair use or the permission given above, feel free to contact us at permis‐
sions@oreilly.com .
O’Reilly Safari
Safari  (formerly Safari Books Online) is a membership-based
training and reference platform for enterprise, government,
educators, and individuals.
Members have access to thousands of books, training videos, Learning Paths, interac‐
tive tutorials, and curated playlists from over 250 publishers, including O’Reilly

Media, Harvard Business Review, Prentice Hall Professional, Addison-Wesley Profes‐
sional, Microsoft Press, Sams, Que, Peachpit Press, Adobe, Focal Press, Cisco Press,
John Wiley & Sons, Syngress, Morgan Kaufmann, IBM Redbooks, Packt, Adobe
Press, FT Press, Apress, Manning, New Riders, McGraw-Hill, Jones & Bartlett, and
Course Technology, among others.
For more information, please visit http://oreilly.com/safari .
How to Contact Us
Please address comments and questions concerning this book to the publisher:
O’Reilly Media, Inc.
1005 Gravenstein Highway North
Sebastopol, CA 95472

800-998-9938 (in the United States or Canada)xviii | Preface

707-829-0515 (international or local)
707-829-0104 (fax)
We have a web page for this book, where we list errata, examples, and any additional
information. Y ou can access this page at http://bit.ly/hands-on-machine-learning-
with-scikit-learn-and-tensorflow  or https://homl.info/oreilly .
To comment or ask technical questions about this book, send email to bookques‐
tions@oreilly.com .
For more information about our books, courses, conferences, and news, see our web‐
site at http://www.oreilly.com .
Find us on Facebook: http://facebook.com/oreilly
Follow us on Twitter: http://twitter.com/oreillymedia
Watch us on Y ouTube: http://www.youtube.com/oreillymedia
Changes in the Second Edition
This second edition has five main objectives:1.
Cover additional topics: additional unsupervised learning techniques (including
clustering, anomaly detection, density estimation and mixture models), addi‐
tional techniques for training deep nets (including self-normalized networks),

additional computer vision techniques (including the Xception, SENet, object
detection with YOLO, and semantic segmentation using R-CNN), handling
sequences using CNNs (including WaveNet), natural language processing using
RNNs, CNNs and Transformers, generative adversarial networks, deploying Ten‐
sorFlow models, and more.2.
Update the book to mention some of the latest results from Deep Learning
research.3.
Migrate all TensorFlow chapters to TensorFlow 2, and use TensorFlow’s imple‐

mentation of the Keras API (called tf.keras) whenever possible, to simplify the
code examples.4.
Update the code examples to use the latest version of Scikit-Learn, NumPy, Pan‐
das, Matplotlib and other libraries.5.
Clarify some sections and fix some errors, thanks to plenty of great feedback
from readers.
Some chapters were added, others were rewritten and a few were reordered. Table P-1
shows the mapping between the 1st edition chapters and the 2nd edition chapters:Preface | xix

Table P-1. Chapter mapping between 1st and 2nd edition 1st Ed. chapter2nd Ed. Chapter% Changes2nd Ed. Title11<10%The Machine Learning Landscape22<10%End-to-End Machine Learning Project33<10%Classification44<10%Training Models55<10%Support Vector Machines66<10%Decision Trees77<10%Ensemble Learning and Random Forests88<10%Dimensionality ReductionN/A9100% newUnsupervised Learning Techniques1010~75%Introduction to Artificial  Neural Networks with Keras 1111~50%Training Deep Neural Networks912100% rewrittenCustom Models and Training with TensorFlowPart of 1213100% rewrittenLoading and Preprocessing Data with TensorFlow1314~50%Deep Computer Vision Using Convolutional Neural NetworksPart of 1415~75%Processing Sequences Using RNNs and CNNsPart of 1416~90%Natural Language Processing with RNNs and Attention1517~75%Autoencoders and GANs1618~75%Reinforcement LearningPart of 1219100% rewrittenDeploying your TensorFlow Models
More specifically, here are the main changes for each 2nd edition chapter (other than

clarifications, corrections and code updates):•
Chapter 1—
Added a section on handling mismatch between the training set and the vali‐
dation & test sets.•
Chapter 2—
Added how to compute a confidence interval.—
Improved the installation instructions (e.g., for Windows).—
Introduced the upgraded OneHotEncoder  and the new ColumnTransformer .•
Chapter 4—
Explained the need for training instances to be Independent and Identically
Distributed (IID).•
Chapter 7—
Added a short section about XGBoost.xx | Preface

• Chapter 9 – new chapter including: —
Clustering with K-Means, how to choose the number of clusters, how to use it
for dimensionality reduction, semi-supervised learning, image segmentation,
and more.—
The DBSCAN clustering algorithm and an overview of other clustering algo‐
rithms available in Scikit-Learn.—
Gaussian mixture models, the Expectation-Maximization (EM) algorithm,
Bayesian variational inference, and how mixture models can be used for clus‐
tering, density estimation, anomaly detection and novelty detection.—
Overview of other anomaly detection and novelty detection algorithms.•
Chapter 10 (mostly new)—
Added an introduction to the Keras API, including all its APIs (Sequential,
Functional and Subclassing), persistence and callbacks (including the Tensor
Board  callback).•
Chapter 11 (many changes)—
Introduced self-normalizing nets, the SELU activation function and Alpha
Dropout.—
Introduced self-supervised learning.—
Added Nadam optimization.—
Added Monte-Carlo Dropout.—

Dropout.—
Added a note about the risks of adaptive optimization methods.—
Updated the practical guidelines.•
Chapter 12 – completely rewritten chapter, including:—
A tour of TensorFlow 2—
TensorFlow’s lower-level Python API—
Writing custom loss functions, metrics, layers, models—
Using auto-differentiation and creating custom training algorithms.—
TensorFlow Functions and graphs (including tracing and autograph).•
Chapter 13 – new chapter, including:—
The Data API—
Loading/Storing data efficiently using TFRecords—

The Features API (including an introduction to embeddings).—
An overview of TF Transform and TF Datasets—
Moved the low-level implementation of the neural network to the exercises.Preface | xxi

— Removed details about queues and readers that are now superseded by the
Data API.•
Chapter 14—
Added Xception and SENet architectures.—
Added a Keras implementation of ResNet-34.—
Showed how to use pretrained models using Keras.—
Added an end-to-end transfer learning example.—
Added classification and localization.—
Introduced Fully Convolutional Networks (FCNs).—
Introduced object detection using the YOLO architecture.—
Introduced semantic segmentation using R-CNN.•
Chapter 15—
Added an introduction to Wavenet.—
Moved the Encoder–Decoder architecture and Bidirectional RNNs to Chapter
16.•
Chapter 16—
Explained how to use the Data API to handle sequential data.—
Showed an end-to-end example of text generation using a Character RNN,
using both a stateless and a stateful RNN.—
Showed an end-to-end example of sentiment analysis using an LSTM.—
Explained masking in Keras.—
Showed how to reuse pretrained embeddings using TF Hub.—
Showed how to build an Encoder–Decoder for Neural Machine Translation

16.•
using TensorFlow Addons/seq2seq.—
Introduced beam search.—
Explained attention mechanisms.—
Added a short overview of visual attention and a note on explainability.—
Introduced the fully attention-based Transformer architecture, including posi‐
tional embeddings and multi-head attention.—
Added an overview of recent language models (2018).•
Chapters 17, 18 and 19: coming soon.xxii | Preface

3“Deep Learning with Python, ” François Chollet (2017).Acknowledgments
Never in my wildest dreams did I imagine that the first edition of this book would get
such a large audience. I received so many messages from readers, many asking ques‐
tions, some kindly pointing out errata, and most sending me encouraging words. I
cannot express how grateful I am to all these readers for their tremendous support.
Thank you all so very much! Please do not hesitate to file issues on github  if you find
errors in the code examples (or just to ask questions), or to submit errata  if you find
errors in the text. Some readers also shared how this book helped them get their first
job, or how it helped them solve a concrete problem they were working on: I find
such feedback incredibly motivating. If you find this book helpful, I would love it if
you could share your story with me, either privately (e.g., via LinkedIn ) or publicly
(e.g., in an Amazon review ).

I am also incredibly thankful to all the amazing people who took time out of their
busy lives to review my book with such care. In particular, I would like to thank Fran‐
çois Chollet for reviewing all the chapters based on Keras & TensorFlow, and giving
me some great, in-depth feedback. Since Keras is one of the main additions to this 2nd
edition, having its author review the book was invaluable. I highly recommend Fran‐
çois’s excellent book Deep Learning with Python3: it has the conciseness, clarity and

depth of the Keras library itself. Big thanks as well to Ankur Patel, who reviewed
every chapter of this 2nd edition and gave me excellent feedback.
This book also benefited from plenty of help from members of the TensorFlow team,
in particular Martin Wicke, who tirelessly answered dozens of my questions and dis‐
patched the rest to the right people, including Alexandre Passos, Allen Lavoie, André
Susano Pinto, Anna Revinskaya, Anthony Platanios, Clemens Mewald, Dan Moldo‐

van, Daniel Dobson, Dustin Tran, Edd Wilder-James, Goldie Gadde, Jiri Simsa, Kar‐
mel Allison, Nick Felt, Paige Bailey, Pete Warden (who also reviewed the 1st edition),
Ryan Sepassi, Sandeep Gupta, Sean Morgan, Todd Wang, Tom O’Malley, William
Chargin, and Yuefeng Zhou, all of whom were tremendously helpful. A huge thank
you to all of you, and to all other members of the TensorFlow team. Not just for your
help, but also for making such a great library.
Big thanks to Haesun Park, who gave me plenty of excellent feedback and caught sev‐

eral errors while he was writing the Korean translation of the 1st edition of this book.
He also translated the Jupyter notebooks to Korean, not to mention TensorFlow’s
documentation. I do not speak Korean, but judging by the quality of his feedback, all
his translations must be truly excellent! Moreover, he kindly contributed some of the
solutions to the exercises in this book.Preface | xxiii

Many thanks as well to O’Reilly’s fantastic staff, in particular Nicole Tache, who gave
me insightful feedback, always cheerful, encouraging, and helpful: I could not dream
of a better editor. Big thanks to Michele Cronin as well, who was very helpful (and
patient) at the start of this 2nd edition. Thanks to Marie Beaugureau, Ben Lorica, Mike
Loukides, and Laurel Ruma for believing in this project and helping me define its
scope. Thanks to Matt Hacker and all of the Atlas team for answering all my technical
questions regarding formatting, asciidoc, and LaTeX, and thanks to Rachel Mona‐
ghan, Nick Adams, and all of the production team for their final review and their
hundreds of corrections.
I would also like to thank my former Google colleagues, in particular the Y ouTube
video classification team, for teaching me so much about Machine Learning. I could
never have started the first edition without them. Special thanks to my personal ML

gurus: Clément Courbet, Julien Dubois, Mathias Kende, Daniel Kitachewsky, James
Pack, Alexander Pak, Anosh Raj, Vitor Sessak, Wiktor Tomczak, Ingrid von Glehn,
Rich Washington, and everyone I worked with at Y ouTube and in the amazing Goo‐
gle research teams in Mountain View. All these people are just as nice and helpful as
they are bright, and that’s saying a lot.
I will never forget the kind people who reviewed the 1st edition of this book, including
David Andrzejewski, Eddy Hung, Grégoire Mesnil, Iain Smears, Ingrid von Glehn,

Justin Francis, Karim Matrah, Lukas Biewald, Michel Tessier, Salim Sémaoune, Vin‐
cent Guilbeau and of course my dear brother Sylvain.
Last but not least, I am infinitely grateful to my beloved wife, Emmanuelle, and to our
three wonderful children, Alexandre, Rémi, and Gabrielle, for encouraging me to
work hard on this book, as well as for their insatiable curiosity: explaining some of
the most difficult concepts in this book to my wife and children helped me clarify my
thoughts and directly improved many parts of this book. Plus, they keep bringing me

cookies and coffee! What more can one dream of?xxiv | Preface

CHAPTER 1
The Machine Learning Landscape
With Early Release ebooks, you get books in their earliest form—
the author’s raw and unedited content as he or she writes—so you
can take advantage of these technologies long before the official
release of these titles. The following will be Chapter 1 in the final
release of the book.
When most people hear “Machine Learning, ” they picture a robot: a dependable but‐
ler or a deadly Terminator depending on who you ask. But Machine Learning is not
just a futuristic fantasy, it’s already here. In fact, it has been around for decades in
some specialized applications, such as Optical Character Recognition  (OCR). But the
first ML application that really became mainstream, improving the lives of hundreds
of millions of people, took over the world back in the 1990s: it was the spam filter .
Not exactly a self-aware Skynet, but it does technically qualify as Machine Learning
(it has actually learned so well that you seldom need to flag an email as spam any‐

more). It was followed by hundreds of ML applications that now quietly power hun‐
dreds of products and features that you use regularly, from better recommendations
to voice search.
Where does Machine Learning start and where does it end? What exactly does it
mean for a machine to learn  something? If I download a copy of Wikipedia, has my
computer really “learned” something? Is it suddenly smarter? In this chapter we will
start by clarifying what Machine Learning is and why you may want to use it.
Then, before we set out to explore the Machine Learning continent, we will take a

look at the map and learn about the main regions and the most notable landmarks:
supervised versus unsupervised learning, online versus batch learning, instance-
based versus model-based learning. Then we will look at the workflow of a typical ML
project, discuss the main challenges you may face, and cover how to evaluate and
fine-tune a Machine Learning system.3

This chapter introduces a lot of fundamental concepts (and jargon) that every data
scientist should know by heart. It will be a high-level overview (the only chapter
without much code), all rather simple, but you should make sure everything is
crystal-clear to you before continuing to the rest of the book. So grab a coffee and let’s
get started!
If you already know all the Machine Learning basics, you may want
to skip directly to Chapter 2 . If you are not sure, try to answer all
the questions listed at the end of the chapter before moving on.
What Is Machine Learning?
Machine Learning is the science (and art) of programming computers so they can
learn from data .
Here is a slightly more general definition:
[Machine Learning is the] field of study that gives computers the ability to learn
without being explicitly programmed.
—Arthur Samuel, 1959
And a more engineering-oriented one:
A computer program is said to learn from experience E with respect to some task T

and some performance measure P , if its performance on T, as measured by P , improves
with experience E.
—Tom Mitchell, 1997
For example, your spam filter is a Machine Learning program that can learn to flag
spam given examples of spam emails (e.g., flagged by users) and examples of regular
(nonspam, also called “ham”) emails. The examples that the system uses to learn are
called the training set . Each training example is called a training instance  (or sample ).

In this case, the task T is to flag spam for new emails, the experience E is the training
data , and the performance measure P needs to be defined; for example, you can use
the ratio of correctly classified emails. This particular performance measure is called
accuracy  and it is often used in classification tasks.
If you just download a copy of Wikipedia, your computer has a lot more data, but it is
not suddenly better at any task. Thus, it is not Machine Learning.
Why Use Machine Learning?

Consider how you would write a spam filter using traditional programming techni‐
ques ( Figure 1-1 ):4 | Chapter 1: The Machine Learning Landscape

1.First you would look at what spam typically looks like. Y ou might notice that
some words or phrases (such as “4U, ” “credit card, ” “free, ” and “amazing”) tend to
come up a lot in the subject. Perhaps you would also notice a few other patterns
in the sender’s name, the email’s body, and so on.2.
Y ou would write a detection algorithm for each of the patterns that you noticed,
and your program would flag emails as spam if a number of these patterns are
detected.3.
Y ou would test your program, and repeat steps 1 and 2 until it is good enough.
Figure 1-1. The traditional approach
Since the problem is not trivial, your program will likely become a long list of com‐
plex rules—pretty hard to maintain.
In contrast, a spam filter based on Machine Learning techniques automatically learns
which words and phrases are good predictors of spam by detecting unusually fre‐
quent patterns of words in the spam examples compared to the ham examples

(Figure 1-2 ). The program is much shorter, easier to maintain, and most likely more
accurate.Why Use Machine Learning? | 5

Figure 1-2. Machine Learning approach
Moreover, if spammers notice that all their emails containing “4U” are blocked, they
might start writing “For U” instead. A spam filter using traditional programming
techniques would need to be updated to flag “For U” emails. If spammers keep work‐
ing around your spam filter, you will need to keep writing new rules forever.
In contrast, a spam filter based on Machine Learning techniques automatically noti‐
ces that “For U” has become unusually frequent in spam flagged by users, and it starts
flagging them without your intervention ( Figure 1-3 ).
Figure 1-3. Automatically adapting to change
Another area where Machine Learning shines is for problems that either are too com‐
plex for traditional approaches or have no known algorithm. For example, consider 
speech recognition: say you want to start simple and write a program capable of dis‐
tinguishing the words “one” and “two. ” Y ou might notice that the word “two” starts

with a high-pitch sound (“T”), so you could hardcode an algorithm that measures
high-pitch sound intensity and use that to distinguish ones and twos. Obviously this
technique will not scale to thousands of words spoken by millions of very different6 | Chapter 1: The Machine Learning Landscape

people in noisy environments and in dozens of languages. The best solution (at least
today) is to write an algorithm that learns by itself, given many example recordings
for each word.
Finally, Machine Learning can help humans learn ( Figure 1-4 ): ML algorithms can be
inspected to see what they have learned (although for some algorithms this can be
tricky). For instance, once the spam filter has been trained on enough spam, it can
easily be inspected to reveal the list of words and combinations of words that it
believes are the best predictors of spam. Sometimes this will reveal unsuspected cor‐
relations or new trends, and thereby lead to a better understanding of the problem.
Applying ML techniques to dig into large amounts of data can help discover patterns
that were not immediately apparent. This is called data mining .
Figure 1-4. Machine Learning can help humans learn
To summarize, Machine Learning is great for:•
Problems for which existing solutions require a lot of hand-tuning or long lists of

rules: one Machine Learning algorithm can often simplify code and perform bet‐
ter.•
Complex problems for which there is no good solution at all using a traditional
approach: the best Machine Learning techniques can find a solution.•
Fluctuating environments: a Machine Learning system can adapt to new data.•
Getting insights about complex problems and large amounts of data.Why Use Machine Learning? | 7

Types of Machine Learning Systems
There are so many different types of Machine Learning systems that it is useful to
classify them in broad categories based on:•
Whether or not they are trained with human supervision (supervised, unsuper‐
vised, semisupervised, and Reinforcement Learning)•
Whether or not they can learn incrementally on the fly (online versus batch
learning)•
Whether they work by simply comparing new data points to known data points,
or instead detect patterns in the training data and build a predictive model, much
like scientists do (instance-based versus model-based learning)
These criteria are not exclusive; you can combine them in any way you like. For
example, a state-of-the-art spam filter may learn on the fly using a deep neural net‐
work model trained using examples of spam and ham; this makes it an online, model-
based, supervised learning system.
Let’s look at each of these criteria a bit more closely.
Supervised/Unsupervised Learning

Machine Learning systems can be classified according to the amount and type of
supervision they get during training. There are four major categories: supervised
learning, unsupervised learning, semisupervised learning, and Reinforcement Learn‐
ing.
Supervised learning
In supervised learning , the training data you feed to the algorithm includes the desired
solutions, called labels  (Figure 1-5 ).
Figure 1-5. A labeled training set for supervised learning (e.g., spam classification)8 | Chapter 1: The Machine Learning Landscape

1Fun fact: this odd-sounding name is a statistics term introduced by Francis Galton while he was studying the
fact that the children of tall people tend to be shorter than their parents. Since children were shorter, he called
this regression to the mean . This name was then applied to the methods he used to analyze correlations
between variables.
A typical supervised learning task is classification . The spam filter is a good example
of this: it is trained with many example emails along with their class  (spam or ham),
and it must learn how to classify new emails.
Another typical task is to predict a target  numeric value, such as the price of a car,
given a set of features  (mileage, age, brand, etc.) called predictors . This sort of task is 
called regression  (Figure 1-6 ).1 To train the system, you need to give it many examples
of cars, including both their predictors and their labels (i.e., their prices).
In Machine Learning an attribute  is a data type (e.g., “Mileage”),

while a feature  has several meanings depending on the context, but
generally means an attribute plus its value (e.g., “Mileage =
15,000”). Many people use the words attribute  and feature  inter‐
changeably, though.
Figure 1-6. Regression
Note that some regression algorithms can be used for classification as well, and vice
versa. For example, Logistic Regression  is commonly used for classification, as it can
output a value that corresponds to the probability of belonging to a given class (e.g.,

20% chance of being spam).Types of Machine Learning Systems | 9

2Some neural network architectures can be unsupervised, such as autoencoders and restricted Boltzmann
machines. They can also be semisupervised, such as in deep belief networks and unsupervised pretraining.Here are some of the most important supervised learning algorithms (covered in this
book):•
k-Nearest Neighbors•
Linear Regression•
Logistic Regression•
Support Vector Machines (SVMs)•
Decision Trees and Random Forests•
Neural networks2
Unsupervised learning
In unsupervised learning , as you might guess, the training data is unlabeled
(Figure 1-7 ). The system tries to learn without a teacher.
Figure 1-7. An unlabeled training set for unsupervised learning
Here are some of the most important unsupervised learning algorithms (most of
these are covered in Chapter 8  and Chapter 9 ):•
Clustering—
K-Means—
DBSCAN—
Hierarchical Cluster Analysis (HCA)•
Anomaly detection and novelty detection—
One-class SVM—
Isolation Forest10 | Chapter 1: The Machine Learning Landscape

• Visualization and dimensionality reduction —
Principal Component Analysis (PCA)—
Kernel PCA—
Locally-Linear Embedding (LLE)—
t-distributed Stochastic Neighbor Embedding (t-SNE)•
Association rule learning—
Apriori—
Eclat
For example, say you have a lot of data about your blog’s visitors. Y ou may want to
run a clustering  algorithm to try to detect groups of similar visitors ( Figure 1-8 ). At
no point do you tell the algorithm which group a visitor belongs to: it finds those
connections without your help. For example, it might notice that 40% of your visitors
are males who love comic books and generally read your blog in the evening, while
20% are young sci-fi lovers who visit during the weekends, and so on. If you use a
hierarchical clustering  algorithm, it may also subdivide each group into smaller
groups. This may help you target your posts for each group.
Figure 1-8. Clustering
Visualization  algorithms are also good examples of unsupervised learning algorithms:

you feed them a lot of complex and unlabeled data, and they output a 2D or 3D rep‐
resentation of your data that can easily be plotted ( Figure 1-9 ). These algorithms try
to preserve as much structure as they can (e.g., trying to keep separate clusters in the
input space from overlapping in the visualization), so you can understand how the
data is organized and perhaps identify unsuspected patterns.Types of Machine Learning Systems | 11

3Notice how animals are rather well separated from vehicles, how horses are close to deer but far from birds,
and so on. Figure reproduced with permission from Socher, Ganjoo, Manning, and Ng (2013), “T-SNE visual‐
ization of the semantic word space. ”
Figure 1-9. Example of a t-SNE visualization highlighting semantic clusters3
A related task is dimensionality reduction , in which the goal is to simplify the data
without losing too much information. One way to do this is to merge several correla‐
ted features into one. For example, a car’s mileage may be very correlated with its age,
so the dimensionality reduction algorithm will merge them into one feature that rep‐
resents the car’s wear and tear. This is called feature extraction .
It is often a good idea to try to reduce the dimension of your train‐
ing data using a dimensionality reduction algorithm before you
feed it to another Machine Learning algorithm (such as a super‐
vised learning algorithm). It will run much faster, the data will take

up less disk and memory space, and in some cases it may also per‐
form better.
Y et another important unsupervised task is anomaly detection —for example, detect‐
ing unusual credit card transactions to prevent fraud, catching manufacturing defects,
or automatically removing outliers from a dataset before feeding it to another learn‐
ing algorithm. The system is shown mostly normal instances during training, so it

learns to recognize them and when it sees a new instance it can tell whether it looks12 | Chapter 1: The Machine Learning Landscape

4That’s when the system works perfectly. In practice it often creates a few clusters per person, and sometimes
mixes up two people who look alike, so you need to provide a few labels per person and manually clean up
some clusters.like a normal one or whether it is likely an anomaly (see Figure 1-10 ). A very similar
task is novelty detection : the difference is that novelty detection algorithms expect to
see only normal data during training, while anomaly detection algorithms are usually
more tolerant, they can often perform well even with a small percentage of outliers in
the training set.
Figure 1-10. Anomaly detection
Finally, another common unsupervised task is association rule learning , in which the
goal is to dig into large amounts of data and discover interesting relations between
attributes. For example, suppose you own a supermarket. Running an association rule
on your sales logs may reveal that people who purchase barbecue sauce and potato

chips also tend to buy steak. Thus, you may want to place these items close to each 
other.
Semisupervised learning
Some algorithms can deal with partially labeled training data, usually a lot of unla‐
beled data and a little bit of labeled data. This is called semisupervised learning
(Figure 1-11 ).
Some photo-hosting services, such as Google Photos, are good examples of this. Once
you upload all your family photos to the service, it automatically recognizes that the

other.
same person A shows up in photos 1, 5, and 11, while another person B shows up in
photos 2, 5, and 7. This is the unsupervised part of the algorithm (clustering). Now all
the system needs is for you to tell it who these people are. Just one label per person,4
and it is able to name everyone in every photo, which is useful for searching photos.Types of Machine Learning Systems | 13

Figure 1-11. Semisupervised learning
Most semisupervised learning algorithms are combinations of unsupervised and
supervised algorithms. For example, deep belief networks  (DBNs) are based on unsu‐
pervised components called restricted Boltzmann machines  (RBMs) stacked on top of
one another. RBMs are trained sequentially in an unsupervised manner, and then the
whole system is fine-tuned using supervised learning techniques.
Reinforcement Learning
Reinforcement Learning  is a very different beast. The learning system, called an agent
in this context, can observe the environment, select and perform actions, and get
rewards  in return (or penalties  in the form of negative rewards, as in Figure 1-12 ). It
must then learn by itself what is the best strategy, called a policy , to get the most
reward over time. A policy defines what action the agent should choose when it is in a
given situation.14 | Chapter 1: The Machine Learning Landscape

Figure 1-12. Reinforcement Learning
For example, many robots implement Reinforcement Learning algorithms to learn
how to walk. DeepMind’s AlphaGo program is also a good example of Reinforcement
Learning: it made the headlines in May 2017 when it beat the world champion Ke Jie
at the game of Go. It learned its winning policy by analyzing millions of games, and
then playing many games against itself. Note that learning was turned off during the
games against the champion; AlphaGo was just applying the policy it had learned.
Batch and Online Learning
Another criterion used to classify Machine Learning systems is whether or not the
system can learn incrementally from a stream of incoming data.
Batch learning
In batch learning , the system is incapable of learning incrementally: it must be trained
using all the available data. This will generally take a lot of time and computing
resources, so it is typically done offline. First the system is trained, and then it is

launched into production and runs without learning anymore; it just applies what it
has learned. This is called offline  learning .
If you want a batch learning system to know about new data (such as a new type of
spam), you need to train a new version of the system from scratch on the full dataset
(not just the new data, but also the old data), then stop the old system and replace it
with the new one.
Fortunately, the whole process of training, evaluating, and launching a Machine

Learning system can be automated fairly easily (as shown in Figure 1-3 ), so even aTypes of Machine Learning Systems | 15

batch learning system can adapt to change. Simply update the data and train a new
version of the system from scratch as often as needed.
This solution is simple and often works fine, but training using the full set of data can
take many hours, so you would typically train a new system only every 24 hours or
even just weekly. If your system needs to adapt to rapidly changing data (e.g., to pre‐
dict stock prices), then you need a more reactive solution.
Also, training on the full set of data requires a lot of computing resources (CPU,
memory space, disk space, disk I/O, network I/O, etc.). If you have a lot of data and
you automate your system to train from scratch every day, it will end up costing you a
lot of money. If the amount of data is huge, it may even be impossible to use a batch
learning algorithm.
Finally, if your system needs to be able to learn autonomously and it has limited
resources (e.g., a smartphone application or a rover on Mars), then carrying around

large amounts of training data and taking up a lot of resources to train for hours
every day is a showstopper.
Fortunately, a better option in all these cases is to use algorithms that are capable of
learning incrementally.
Online learning
In online learning , you train the system incrementally by feeding it data instances
sequentially, either individually or by small groups called mini-batches . Each learning
step is fast and cheap, so the system can learn about new data on the fly, as it arrives
(see Figure 1-13 ).
Figure 1-13. Online learning

Online learning is great for systems that receive data as a continuous flow (e.g., stock
prices) and need to adapt to change rapidly or autonomously. It is also a good option16 | Chapter 1: The Machine Learning Landscape

if you have limited computing resources: once an online learning system has learned
about new data instances, it does not need them anymore, so you can discard them
(unless you want to be able to roll back to a previous state and “replay” the data). This
can save a huge amount of space.
Online learning algorithms can also be used to train systems on huge datasets that
cannot fit in one machine’s main memory (this is called out-of-core  learning). The
algorithm loads part of the data, runs a training step on that data, and repeats the
process until it has run on all of the data (see Figure 1-14 ).
Out-of-core learning is usually done offline (i.e., not on the live
system), so online learning  can be a confusing name. Think of it as
incremental learning .
Figure 1-14. Using online learning to handle huge datasets
One important parameter of online learning systems is how fast they should adapt to
changing data: this is called the learning rate . If you set a high learning rate, then your

system will rapidly adapt to new data, but it will also tend to quickly forget the old
data (you don’t want a spam filter to flag only the latest kinds of spam it was shown).
Conversely, if you set a low learning rate, the system will have more inertia; that is, it
will learn more slowly, but it will also be less sensitive to noise in the new data or to
sequences of nonrepresentative data points (outliers).
A big challenge with online learning is that if bad data is fed to the system, the sys‐

tem’s performance will gradually decline. If we are talking about a live system, your
clients will notice. For example, bad data could come from a malfunctioning sensor
on a robot, or from someone spamming a search engine to try to rank high in searchTypes of Machine Learning Systems | 17

results. To reduce this risk, you need to monitor your system closely and promptly
switch learning off (and possibly revert to a previously working state) if you detect a
drop in performance. Y ou may also want to monitor the input data and react to
abnormal data (e.g., using an anomaly detection algorithm).
Instance-Based Versus Model-Based Learning
One more way to categorize Machine Learning systems is by how they generalize .
Most Machine Learning tasks are about making predictions. This means that given a
number of training examples, the system needs to be able to generalize to examples it
has never seen before. Having a good performance measure on the training data is
good, but insufficient; the true goal is to perform well on new instances.
There are two main approaches to generalization: instance-based learning and
model-based learning.
Instance-based learning
Possibly the most trivial form of learning is simply to learn by heart. If you were to

create a spam filter this way, it would just flag all emails that are identical to emails
that have already been flagged by users—not the worst solution, but certainly not the
best.
Instead of just flagging emails that are identical to known spam emails, your spam
filter could be programmed to also flag emails that are very similar to known spam
emails. This requires a measure of similarity  between two emails. A (very basic) simi‐
larity measure between two emails could be to count the number of words they have

best.
in common. The system would flag an email as spam if it has many words in com‐
mon with a known spam email.
This is called instance-based learning : the system learns the examples by heart, then
generalizes to new cases by comparing them to the learned examples (or a subset of
them), using a similarity measure. For example, in Figure 1-15  the new instance
would be classified as a triangle because the majority of the most similar instances
belong to that class.18 | Chapter 1: The Machine Learning Landscape

Figure 1-15. Instance-based learning
Model-based learning
Another way to generalize from a set of examples is to build a model of these exam‐
ples, then use that model to make predictions . This is called model-based learning
(Figure 1-16 ).
Figure 1-16. Model-based learning
For example, suppose you want to know if money makes people happy, so you down‐
load the Better Life Index  data from the OECD’s website  as well as stats about GDP
per capita from the IMF’s website . Then you join the tables and sort by GDP per cap‐
ita. Table 1-1  shows an excerpt of what you get.Types of Machine Learning Systems | 19

5By convention, the Greek letter θ (theta) is frequently used to represent model parameters.Table 1-1. Does money make people happier?CountryGDP per capita (USD)Life satisfactionHungary12,2404.9Korea27,1955.8France37,6756.5Australia50,9627.3United States55,8057.2
Let’s plot the data for a few random countries ( Figure 1-17 ).
Figure 1-17. Do you see a trend here?
There does seem to be a trend here! Although the data is noisy  (i.e., partly random), it
looks like life satisfaction goes up more or less linearly as the country’s GDP per cap‐
ita increases. So you decide to model life satisfaction as a linear function of GDP per
capita. This step is called model selection : you selected a linear model  of life satisfac‐
tion with just one attribute, GDP per capita ( Equation 1-1 ).
Equation 1-1. A simple linear modellife_satisfaction = θ0+θ1× GDP_per_capita
This model has two model parameters , θ0 and θ1.5 By tweaking these parameters, you

can make your model represent any linear function, as shown in Figure 1-18 .20 | Chapter 1: The Machine Learning Landscape

Figure 1-18. A few possible linear models
Before you can use your model, you need to define the parameter values θ0 and θ1.
How can you know which values will make your model perform best? To answer this
question, you need to specify a performance measure. Y ou can either define a utility
function  (or fitness  function ) that measures how good  your model is, or you can define
a cost function  that measures how bad it is. For linear regression problems, people
typically use a cost function that measures the distance between the linear model’s
predictions and the training examples; the objective is to minimize this distance.
This is where the Linear Regression algorithm comes in: you feed it your training
examples and it finds the parameters that make the linear model fit best to your data.
This is called training  the model. In our case the algorithm finds that the optimal
parameter values are θ0 = 4.85 and θ1 = 4.91 × 10–5.

Now the model fits the training data as closely as possible (for a linear model), as you
can see in Figure 1-19 .
Figure 1-19. The linear model that fits the training data bestTypes of Machine Learning Systems | 21

6The prepare_country_stats()  function’s definition is not shown here (see this chapter’s Jupyter notebook if
you want all the gory details). It’s just boring Pandas code that joins the life satisfaction data from the OECD
with the GDP per capita data from the IMF.7
It’s okay if you don’t understand all the code yet; we will present Scikit-Learn in the following chapters.Y ou are finally ready to run the model to make predictions. For example, say you
want to know how happy Cypriots are, and the OECD data does not have the answer.
Fortunately, you can use your model to make a good prediction: you look up Cyprus’s
GDP per capita, find $22,587, and then apply your model and find that life satisfac‐
tion is likely to be somewhere around 4.85 + 22,587 × 4.91 × 10-5 = 5.96.
To whet your appetite, Example 1-1  shows the Python code that loads the data, pre‐
pares it,6 creates a scatterplot for visualization, and then trains a linear model and
makes a prediction.7

Example 1-1. Training and running a linear model using Scikit-Learn
import matplotlib.pyplot  as plt
import numpy as np
import pandas as pd
import sklearn.linear_model
# Load the data
oecd_bli  = pd.read_csv ("oecd_bli_2015.csv" , thousands =',')
gdp_per_capita  = pd.read_csv ("gdp_per_capita.csv" ,thousands =',',delimiter ='\t',
                             encoding ='latin1' , na_values ="n/a")
# Prepare the data
country_stats  = prepare_country_stats (oecd_bli , gdp_per_capita )
X = np.c_[country_stats ["GDP per capita" ]]
y = np.c_[country_stats ["Life satisfaction" ]]

# Visualize the data
country_stats .plot(kind='scatter' , x="GDP per capita" , y='Life satisfaction' )
plt.show()
# Select a linear model
model = sklearn.linear_model .LinearRegression ()
# Train the model
model.fit(X, y)
# Make a prediction for Cyprus
X_new = [[22587]]  # Cyprus' GDP per capita
print(model.predict(X_new)) # outputs [[ 5.96242338]]22 | Chapter 1: The Machine Learning Landscape

If you had used an instance-based learning algorithm instead, you
would have found that Slovenia has the closest GDP per capita to
that of Cyprus ($20,732), and since the OECD data tells us that
Slovenians’ life satisfaction is 5.7, you would have predicted a life
satisfaction of 5.7 for Cyprus. If you zoom out a bit and look at the
two next closest countries, you will find Portugal and Spain with
life satisfactions of 5.1 and 6.5, respectively. Averaging these three
values, you get 5.77, which is pretty close to your model-based pre‐
diction. This simple algorithm is called k-Nearest Neighbors  regres‐
sion (in this example, k = 3).
Replacing the Linear Regression model with k-Nearest Neighbors
regression in the previous code is as simple as replacing these two
lines:
with these two:
import sklearn.neighbors
model = sklearn.neighbors .KNeighborsRegressor (n_neighbors =3)

lines:
If all went well, your model will make good predictions. If not, you may need to use
more attributes (employment rate, health, air pollution, etc.), get more or better qual‐
ity training data, or perhaps select a more powerful model (e.g., a Polynomial Regres‐
sion model).
In summary:•
Y ou studied the data.•
Y ou selected a model.•
Y ou trained it on the training data (i.e., the learning algorithm searched for the
model parameter values that minimize a cost function).•

Finally, you applied the model to make predictions on new cases (this is called
inference ), hoping that this model will generalize well.
This is what a typical Machine Learning project looks like. In Chapter 2  you will
experience this first-hand by going through an end-to-end project.
We have covered a lot of ground so far: you now know what Machine Learning is
really about, why it is useful, what some of the most common categories of ML sys‐
tems are, and what a typical project workflow looks like. Now let’s look at what can go

wrong in learning and prevent you from making accurate predictions.Types of Machine Learning Systems | 23

Main Challenges of Machine Learning
In short, since your main task is to select a learning algorithm and train it on some
data, the two things that can go wrong are “bad algorithm” and “bad data. ” Let’s start
with examples of bad data.
Insufficient  Quantity of Training Data
For a toddler to learn what an apple is, all it takes is for you to point to an apple and
say “apple” (possibly repeating this procedure a few times). Now the child is able to
recognize apples in all sorts of colors and shapes. Genius.
Machine Learning is not quite there yet; it takes a lot of data for most Machine Learn‐
ing algorithms to work properly. Even for very simple problems you typically need
thousands of examples, and for complex problems such as image or speech recogni‐
tion you may need millions of examples (unless you can reuse parts of an existing
model).24 | Chapter 1: The Machine Learning Landscape

8For example, knowing whether to write “to, ” “two, ” or “too” depending on the context.9
Figure reproduced with permission from Banko and Brill (2001), “Learning Curves for Confusion Set Disam‐
biguation. ”10
“The Unreasonable Effectiveness of Data, ” Peter Norvig et al. (2009).The Unreasonable Effectiveness  of Data
In a famous paper  published in 2001, Microsoft researchers Michele Banko and Eric
Brill showed that very different Machine Learning algorithms, including fairly simple
ones, performed almost identically well on a complex problem of natural language
disambiguation8 once they were given enough data (as you can see in Figure 1-20 ).
Figure 1-20. The importance of data versus algorithms9
As the authors put it: “these results suggest that we may want to reconsider the trade-
off between spending time and money on algorithm development versus spending it
on corpus development. ”
The idea that data matters more than algorithms for complex problems was further

popularized by Peter Norvig et al. in a paper titled “The Unreasonable Effectiveness
of Data”  published in 2009.10 It should be noted, however, that small- and medium-
sized datasets are still very common, and it is not always easy or cheap to get extra
training data, so don’t abandon algorithms just yet.Main Challenges of Machine Learning | 25

Nonrepresentative Training Data
In order to generalize well, it is crucial that your training data be representative of the
new cases you want to generalize to. This is true whether you use instance-based
learning or model-based learning.
For example, the set of countries we used earlier for training the linear model was not
perfectly representative; a few countries were missing. Figure 1-21  shows what the
data looks like when you add the missing countries.
Figure 1-21. A more representative training sample
If you train a linear model on this data, you get the solid line, while the old model is
represented by the dotted line. As you can see, not only does adding a few missing
countries significantly alter the model, but it makes it clear that such a simple linear
model is probably never going to work well. It seems that very rich countries are not
happier than moderately rich countries (in fact they seem unhappier), and conversely
some poor countries seem happier than many rich countries.

By using a nonrepresentative training set, we trained a model that is unlikely to make
accurate predictions, especially for very poor and very rich countries.
It is crucial to use a training set that is representative of the cases you want to general‐
ize to. This is often harder than it sounds: if the sample is too small, you will have
sampling noise  (i.e., nonrepresentative data as a result of chance), but even very large
samples can be nonrepresentative if the sampling method is flawed. This is called
sampling bias .

A Famous Example of Sampling Bias
Perhaps the most famous example of sampling bias happened during the US presi‐
dential election in 1936, which pitted Landon against Roosevelt: the Literary Digest
conducted a very large poll, sending mail to about 10 million people. It got 2.4 million
answers, and predicted with high confidence that Landon would get 57% of the votes.26 | Chapter 1: The Machine Learning Landscape

Instead, Roosevelt won with 62% of the votes. The flaw was in the Literary Digest ’s
sampling method:•
First, to obtain the addresses to send the polls to, the Literary Digest  used tele‐
phone directories, lists of magazine subscribers, club membership lists, and the
like. All of these lists tend to favor wealthier people, who are more likely to vote
Republican (hence Landon).•
Second, less than 25% of the people who received the poll answered. Again, this
introduces a sampling bias, by ruling out people who don’t care much about poli‐
tics, people who don’t like the Literary Digest , and other key groups. This is a spe‐
cial type of sampling bias called nonresponse bias .
Here is another example: say you want to build a system to recognize funk music vid‐
eos. One way to build your training set is to search “funk music” on Y ouTube and use
the resulting videos. But this assumes that Y ouTube’s search engine returns a set of

videos that are representative of all the funk music videos on Y ouTube. In reality, the
search results are likely to be biased toward popular artists (and if you live in Brazil
you will get a lot of “funk carioca” videos, which sound nothing like James Brown).
On the other hand, how else can you get a large training set?
Poor-Quality Data
Obviously, if your training data is full of errors, outliers, and noise (e.g., due to poor-
quality measurements), it will make it harder for the system to detect the underlying

patterns, so your system is less likely to perform well. It is often well worth the effort
to spend time cleaning up your training data. The truth is, most data scientists spend
a significant part of their time doing just that. For example:•
If some instances are clearly outliers, it may help to simply discard them or try to
fix the errors manually.•
If some instances are missing a few features (e.g., 5% of your customers did not
specify their age), you must decide whether you want to ignore this attribute alto‐

gether, ignore these instances, fill in the missing values (e.g., with the median
age), or train one model with the feature and one model without it, and so on.
Irrelevant Features
As the saying goes: garbage in, garbage out. Y our system will only be capable of learn‐
ing if the training data contains enough relevant features and not too many irrelevant
ones. A critical part of the success of a Machine Learning project is coming up with a
good set of features to train on. This process, called feature engineering , involves:Main Challenges of Machine Learning | 27

•Feature selection : selecting the most useful features to train on among existing
features.•
Feature extraction : combining existing features to produce a more useful one (as
we saw earlier, dimensionality reduction algorithms can help).•
Creating new features by gathering new data.
Now that we have looked at many examples of bad data, let’s look at a couple of exam‐
ples of bad algorithms.
Overfitting  the Training Data
Say you are visiting a foreign country and the taxi driver rips you off. Y ou might be
tempted to say that all taxi drivers in that country are thieves. Overgeneralizing is
something that we humans do all too often, and unfortunately machines can fall into
the same trap if we are not careful. In Machine Learning this is called overfitting : it
means that the model performs well on the training data, but it does not generalize
well.
Figure 1-22  shows an example of a high-degree polynomial life satisfaction model

well.
that strongly overfits the training data. Even though it performs much better on the
training data than the simple linear model, would you really trust its predictions?
Figure 1-22. Overfitting  the training data
Complex models such as deep neural networks can detect subtle patterns in the data,
but if the training set is noisy, or if it is too small (which introduces sampling noise),
then the model is likely to detect patterns in the noise itself. Obviously these patterns
will not generalize to new instances. For example, say you feed your life satisfaction

model many more attributes, including uninformative ones such as the country’s
name. In that case, a complex model may detect patterns like the fact that all coun‐
tries in the training data with a w in their name have a life satisfaction greater than 7:
New Zealand (7.3), Norway (7.4), Sweden (7.2), and Switzerland (7.5). How confident28 | Chapter 1: The Machine Learning Landscape

are you that the W-satisfaction rule generalizes to Rwanda or Zimbabwe? Obviously
this pattern occurred in the training data by pure chance, but the model has no way
to tell whether a pattern is real or simply the result of noise in the data.
Overfitting happens when the model is too complex relative to the
amount and noisiness of the training data. The possible solutions
are:•
To simplify the model by selecting one with fewer parameters
(e.g., a linear model rather than a high-degree polynomial
model), by reducing the number of attributes in the training
data or by constraining the model•
To gather more training data•
To reduce the noise in the training data (e.g., fix data errors
and remove outliers)
Constraining a model to make it simpler and reduce the risk of overfitting is called
regularization . For example, the linear model we defined earlier has two parameters,
θ0 and θ1. This gives the learning algorithm two degrees of freedom  to adapt the model

to the training data: it can tweak both the height ( θ0) and the slope ( θ1) of the line. If
we forced θ1 = 0, the algorithm would have only one degree of freedom and would
have a much harder time fitting the data properly: all it could do is move the line up
or down to get as close as possible to the training instances, so it would end up
around the mean. A very simple model indeed! If we allow the algorithm to modify θ1
but we force it to keep it small, then the learning algorithm will effectively have some‐

where in between one and two degrees of freedom. It will produce a simpler model
than with two degrees of freedom, but more complex than with just one. Y ou want to
find the right balance between fitting the training data perfectly and keeping the
model simple enough to ensure that it will generalize well.
Figure 1-23  shows three models: the dotted line represents the original model that
was trained with a few countries missing, the dashed line is our second model trained
with all countries, and the solid line is a linear model trained with the same data as

the first model but with a regularization constraint. Y ou can see that regularization
forced the model to have a smaller slope, which fits a bit less the training data that the
model was trained on, but actually allows it to generalize better to new examples.Main Challenges of Machine Learning | 29

Figure 1-23. Regularization reduces the risk of overfitting
The amount of regularization to apply during learning can be controlled by a hyper‐
parameter . A hyperparameter is a parameter of a learning algorithm (not of the
model). As such, it is not affected by the learning algorithm itself; it must be set prior
to training and remains constant during training. If you set the regularization hyper‐
parameter to a very large value, you will get an almost flat model (a slope close to
zero); the learning algorithm will almost certainly not overfit the training data, but it
will be less likely to find a good solution. Tuning hyperparameters is an important
part of building a Machine Learning system (you will see a detailed example in the
next chapter).
Underfitting  the Training Data
As you might guess, underfitting  is the opposite of overfitting: it occurs when your
model is too simple to learn the underlying structure of the data. For example, a lin‐

ear model of life satisfaction is prone to underfit; reality is just more complex than
the model, so its predictions are bound to be inaccurate, even on the training exam‐
ples.
The main options to fix this problem are:•
Selecting a more powerful model, with more parameters•
Feeding better features to the learning algorithm (feature engineering)•
Reducing the constraints on the model (e.g., reducing the regularization hyper‐
parameter)
Stepping Back
By now you already know a lot about Machine Learning. However, we went through

ples.
so many concepts that you may be feeling a little lost, so let’s step back and look at the
big picture:30 | Chapter 1: The Machine Learning Landscape

•Machine Learning is about making machines get better at some task by learning
from data, instead of having to explicitly code rules.•
There are many different types of ML systems: supervised or not, batch or online,
instance-based or model-based, and so on.•
In a ML project you gather data in a training set, and you feed the training set to
a learning algorithm. If the algorithm is model-based it tunes some parameters to
fit the model to the training set (i.e., to make good predictions on the training set
itself), and then hopefully it will be able to make good predictions on new cases
as well. If the algorithm is instance-based, it just learns the examples by heart and
generalizes to new instances by comparing them to the learned instances using a
similarity measure.•
The system will not perform well if your training set is too small, or if the data is
not representative, noisy, or polluted with irrelevant features (garbage in, garbage

out). Lastly, your model needs to be neither too simple (in which case it will
underfit) nor too complex (in which case it will overfit).
There’s just one last important topic to cover: once you have trained a model, you
don’t want to just “hope” it generalizes to new cases. Y ou want to evaluate it, and fine-
tune it if necessary. Let’s see how.
Testing and Validating
The only way to know how well a model will generalize to new cases is to actually try
it out on new cases. One way to do that is to put your model in production and moni‐

tor how well it performs. This works well, but if your model is horribly bad, your
users will complain—not the best idea.
A better option is to split your data into two sets: the training set  and the test set . As
these names imply, you train your model using the training set, and you test it using
the test set. The error rate on new cases is called the generalization error  (or out-of-
sample error ), and by evaluating your model on the test set, you get an estimate of this

error. This value tells you how well your model will perform on instances it has never
seen before.
If the training error is low (i.e., your model makes few mistakes on the training set)
but the generalization error is high, it means that your model is overfitting the train‐
ing data.
It is common to use 80% of the data for training and hold out  20%
for testing. However, this depends on the size of the dataset: if it
contains 10 million instances, then holding out 1% means your test

ing data.
set will contain 100,000 instances: that’s probably more than
enough to get a good estimate of the generalization error.Testing and Validating | 31

Hyperparameter Tuning and Model Selection
So evaluating a model is simple enough: just use a test set. Now suppose you are hesi‐
tating between two models (say a linear model and a polynomial model): how can
you decide? One option is to train both and compare how well they generalize using
the test set.
Now suppose that the linear model generalizes better, but you want to apply some 
regularization to avoid overfitting. The question is: how do you choose the value of
the regularization hyperparameter? One option is to train 100 different models using
100 different values for this hyperparameter. Suppose you find the best hyperparame‐
ter value that produces a model with the lowest generalization error, say just 5% error.
So you launch this model into production, but unfortunately it does not perform as
well as expected and produces 15% errors. What just happened?
The problem is that you measured the generalization error multiple times on the test

set, and you adapted the model and hyperparameters to produce the best model for
that particular set . This means that the model is unlikely to perform as well on new
data.
A common solution to this problem is called holdout validation : you simply hold out
part of the training set to evaluate several candidate models and select the best one.
The new heldout set is called the validation set  (or sometimes the development set , or
dev set ). More specifically, you train multiple models with various hyperparameters

data.
on the reduced training set (i.e., the full training set minus the validation set), and
you select the model that performs best on the validation set. After this holdout vali‐
dation process, you train the best model on the full training set (including the valida‐
tion set), and this gives you the final model. Lastly, you evaluate this final model on
the test set to get an estimate of the generalization error.
This solution usually works quite well. However, if the validation set is too small, then

model evaluations will be imprecise: you may end up selecting a suboptimal model by
mistake. Conversely, if the validation set is too large, then the remaining training set
will be much smaller than the full training set. Why is this bad? Well, since the final
model will be trained on the full training set, it is not ideal to compare candidate
models trained on a much smaller training set. It would be like selecting the fastest
sprinter to participate in a marathon. One way to solve this problem is to perform

repeated cross-validation , using many small validation sets. Each model is evaluated
once per validation set, after it is trained on the rest of the data. By averaging out all
the evaluations of a model, we get a much more accurate measure of its performance.
However, there is a drawback: the training time is multiplied by the number of valida‐
tion sets.32 | Chapter 1: The Machine Learning Landscape

11“The Lack of A Priori Distinctions Between Learning Algorithms, ” D. Wolpert (1996).Data Mismatch
In some cases, it is easy to get a large amount of data for training, but it is not per‐
fectly representative of the data that will be used in production. For example, suppose
you want to create a mobile app to take pictures of flowers and automatically deter‐
mine their species. Y ou can easily download millions of pictures of flowers on the
web, but they won’t be perfectly representative of the pictures that will actually be
taken using the app on a mobile device. Perhaps you only have 10,000 representative
pictures (i.e., actually taken with the app). In this case, the most important rule to
remember is that the validation set and the test must be as representative as possible
of the data you expect to use in production, so they should be composed exclusively
of representative pictures: you can shuffle them and put half in the validation set, and

half in the test set (making sure that no duplicates or near-duplicates end up in both
sets). After training your model on the web pictures, if you observe that the perfor‐
mance of your model on the validation set is disappointing, you will not know
whether this is because your model has overfit the training set, or whether this is just
due to the mismatch between the web pictures and the mobile app pictures. One sol‐
ution is to hold out part of the training pictures (from the web) in yet another set that

Andrew Ng calls the train-dev set . After the model is trained (on the training set, not
on the train-dev set), you can evaluate it on the train-dev set: if it performs well, then
the model is not overfitting the training set, so if performs poorly on the validation
set, the problem must come from the data mismatch. Y ou can try to tackle this prob‐
lem by preprocessing the web images to make them look more like the pictures that
will be taken by the mobile app, and then retraining the model. Conversely, if the
model performs poorly on the train-dev set, then the model must have overfit the

training set, so you should try to simplify or regularize the model, get more training
data and clean up the training data, as discussed earlier.
No Free Lunch Theorem
A model is a simplified version of the observations. The simplifications are meant to
discard the superfluous details that are unlikely to generalize to new instances. How‐
ever, to decide what data to discard and what data to keep, you must make assump‐
tions . For example, a linear model makes the assumption that the data is

fundamentally linear and that the distance between the instances and the straight line
is just noise, which can safely be ignored.
In a famous 1996 paper ,11 David Wolpert demonstrated that if you make absolutely
no assumption about the data, then there is no reason to prefer one model over any
other. This is called the No Free Lunch  (NFL) theorem. For some datasets the bestTesting and Validating | 33

model is a linear model, while for other datasets it is a neural network. There is no
model that is a priori  guaranteed to work better (hence the name of the theorem). The
only way to know for sure which model is best is to evaluate them all. Since this is not
possible, in practice you make some reasonable assumptions about the data and you
evaluate only a few reasonable models. For example, for simple tasks you may evalu‐
ate linear models with various levels of regularization, and for a complex problem you
may evaluate various neural networks.
Exercises
In this chapter we have covered some of the most important concepts in Machine
Learning. In the next chapters we will dive deeper and write more code, but before we
do, make sure you know how to answer the following questions:1.
How would you define Machine Learning?2.
Can you name four types of problems where it shines?3.
What is a labeled training set?4.
What are the two most common supervised tasks?5.
Can you name four common unsupervised tasks?6.

Exercises
What type of Machine Learning algorithm would you use to allow a robot to
walk in various unknown terrains?7.
What type of algorithm would you use to segment your customers into multiple
groups?8.
Would you frame the problem of spam detection as a supervised learning prob‐
lem or an unsupervised learning problem?9.
What is an online learning system?10.
What is out-of-core learning?11.
What type of learning algorithm relies on a similarity measure to make predic‐
tions?12.

groups?8.
tions?12.
What is the difference between a model parameter and a learning algorithm’s
hyperparameter?13.
What do model-based learning algorithms search for? What is the most common
strategy they use to succeed? How do they make predictions?14.
Can you name four of the main challenges in Machine Learning?15.
If your model performs great on the training data but generalizes poorly to new
instances, what is happening? Can you name three possible solutions?16.

tions?12.
What is a test set and why would you want to use it?34 | Chapter 1: The Machine Learning Landscape

17. What is the purpose of a validation set? 18.
What can go wrong if you tune hyperparameters using the test set?19.
What is repeated cross-validation and why would you prefer it to using a single
validation set?
Solutions to these exercises are available in ???.Exercises | 35

1The example project is completely fictitious; the goal is just to illustrate the main steps of a Machine Learning
project, not to learn anything about the real estate business.
CHAPTER 2
End-to-End Machine Learning Project
release of these titles. The following will be Chapter 2 in the final
In this chapter, you will go through an example project end to end, pretending to be a
recently hired data scientist in a real estate company.1 Here are the main steps you will
go through:1.
Look at the big picture.2.
Get the data.3.
Discover and visualize the data to gain insights.4.
Prepare the data for Machine Learning algorithms.5.
Select a model and train it.6.
Fine-tune your model.7.
Present your solution.8.
Launch, monitor, and maintain your system.37

2The original dataset appeared in R. Kelley Pace and Ronald Barry, “Sparse Spatial Autoregressions, ” Statistics
& Probability Letters  33, no. 3 (1997): 291–297. Working with Real Data
When you are learning about Machine Learning it is best to actually experiment with
real-world data, not just artificial datasets. Fortunately, there are thousands of open
datasets to choose from, ranging across all sorts of domains. Here are a few places
you can look to get data:•
Popular open data repositories:—
UC Irvine Machine Learning Repository—
Kaggle datasets—
Amazon’s AWS datasets•
Meta portals (they list open data repositories):—
http://dataportals.org/—
http://opendatamonitor.eu/—
http://quandl.com/•
Other pages listing many popular open data repositories:—
Wikipedia’s list of Machine Learning datasets—
Quora.com question—
Datasets subreddit
In this chapter we chose the California Housing Prices dataset from the StatLib repos‐
itory2 (see Figure 2-1 ). This dataset was based on data from the 1990 California cen‐

sus. It is not exactly recent (you could still afford a nice house in the Bay Area at the
time), but it has many qualities for learning, so we will pretend it is recent data. We
also added a categorical attribute and removed a few features for teaching purposes.38 | Chapter 2: End-to-End Machine Learning Project

Figure 2-1. California housing prices
Look at the Big Picture
Welcome to Machine Learning Housing Corporation! The first task you are asked to
perform is to build a model of housing prices in California using the California cen‐
sus data. This data has metrics such as the population, median income, median hous‐
ing price, and so on for each block group in California. Block groups are the smallest
geographical unit for which the US Census Bureau publishes sample data (a block
group typically has a population of 600 to 3,000 people). We will just call them “dis‐
tricts” for short.
Y our model should learn from this data and be able to predict the median housing
price in any district, given all the other metrics.
Since you are a well-organized data scientist, the first thing you do
is to pull out your Machine Learning project checklist. Y ou can
start with the one in ???; it should work reasonably well for most
Machine Learning projects but make sure to adapt it to your needs.

In this chapter we will go through many checklist items, but we will
also skip a few, either because they are self-explanatory or because
they will be discussed in later chapters.
Frame the Problem
The first question to ask your boss is what exactly is the business objective; building a
model is probably not the end goal. How does the company expect to use and benefitLook at the Big Picture | 39

3A piece of information fed to a Machine Learning system is often called a signal  in reference to Shannon’s
information theory: you want a high signal/noise ratio.from this model? This is important because it will determine how you frame the
problem, what algorithms you will select, what performance measure you will use to
evaluate your model, and how much effort you should spend tweaking it.
Y our boss answers that your model’s output (a prediction of a district’s median hous‐
ing price) will be fed to another Machine Learning system (see Figure 2-2 ), along
with many other signals .3 This downstream system will determine whether it is worth
investing in a given area or not. Getting this right is critical, as it directly affects reve‐
nue.
Figure 2-2. A Machine Learning pipeline for real estate investments
Pipelines
A sequence of data processing components  is called a data pipeline . Pipelines are very
common in Machine Learning systems, since there is a lot of data to manipulate and

nue.
Pipelines
many data transformations to apply.
Components typically run asynchronously. Each component pulls in a large amount
of data, processes it, and spits out the result in another data store, and then some time
later the next component in the pipeline pulls this data and spits out its own output,
and so on. Each component is fairly self-contained: the interface between components
is simply the data store. This makes the system quite simple to grasp (with the help of
a data flow graph), and different teams can focus on different components. Moreover,

if a component breaks down, the downstream components can often continue to run
normally (at least for a while) by just using the last output from the broken compo‐
nent. This makes the architecture quite robust.40 | Chapter 2: End-to-End Machine Learning Project

On the other hand, a broken component can go unnoticed for some time if proper
monitoring is not implemented. The data gets stale and the overall system’s perfor‐
mance drops.
The next question to ask is what the current solution looks like (if any). It will often
give you a reference performance, as well as insights on how to solve the problem.
Y our boss answers that the district housing prices are currently estimated manually
by experts: a team gathers up-to-date information about a district, and when they
cannot get the median housing price, they estimate it using complex rules.
This is costly and time-consuming, and their estimates are not great; in cases where
they manage to find out the actual median housing price, they often realize that their
estimates were off by more than 20%. This is why the company thinks that it would
be useful to train a model to predict a district’s median housing price given other data
about that district. The census data looks like a great dataset to exploit for this pur‐

pose, since it includes the median housing prices of thousands of districts, as well as
other data.
Okay, with all this information you are now ready to start designing your system.
First, you need to frame the problem: is it supervised, unsupervised, or Reinforce‐
ment Learning? Is it a classification task, a regression task, or something else? Should
you use batch learning or online learning techniques? Before you read on, pause and
try to answer these questions for yourself.

Have you found the answers? Let’s see: it is clearly a typical supervised learning task
since you are given labeled  training examples (each instance comes with the expected
output, i.e., the district’s median housing price). Moreover, it is also a typical regres‐
sion task, since you are asked to predict a value. More specifically, this is a multiple
regression  problem since the system will use multiple features to make a prediction (it
will use the district’s population, the median income, etc.). It is also a univariate

regression  problem since we are only trying to predict a single value for each district.
If we were trying to predict multiple values per district, it would be a multivariate
regression  problem. Finally, there is no continuous flow of data coming in the system,
there is no particular need to adjust to changing data rapidly, and the data is small
enough to fit in memory, so plain batch learning should do just fine.
If the data was huge, you could either split your batch learning
work across multiple servers (using the MapReduce  technique), or

you could use an online learning technique instead.Look at the Big Picture | 41

Select a Performance Measure
Y our next step is to select a performance measure. A typical performance measure for
regression problems is the Root Mean Square Error (RMSE). It gives an idea of how
much error the system typically makes in its predictions, with a higher weight for
large errors. Equation 2-1  shows the mathematical formula to compute the RMSE.
Equation 2-1. Root Mean Square Error (RMSE)RMSE X,h=1
m∑
i= 1m
hxi−yi242 | Chapter 2: End-to-End Machine Learning Project

4Recall that the transpose operator flips a column vector into a row vector (and vice versa).
Notations
This equation introduces several very common Machine Learning notations that we
will use throughout this book:•
m is the number of instances in the dataset you are measuring the RMSE on.—
For example, if you are evaluating the RMSE on a validation set of 2,000 dis‐
tricts, then m = 2,000.•
x(i) is a vector of all the feature values (excluding the label) of the ith instance in
the dataset, and y(i) is its label (the desired output value for that instance).—
For example, if the first district in the dataset is located at longitude –118.29°,
latitude 33.91°, and it has 1,416 inhabitants with a median income of $38,372,
and the median house value is $156,400 (ignoring the other features for now),
then:x1=−118 . 29
33 . 91
1, 416
38, 372
and:y1= 156, 400 •
X is a matrix containing all the feature values (excluding labels) of all instances in

33 . 91
1, 416
38, 372
the dataset. There is one row per instance and the ith row is equal to the transpose
of x(i), noted ( x(i))T.4—
For example, if the first district is as just described, then the matrix X looks
like this:X=x1T
x2T
⋮
x1999 T
x2000 T=−118 . 29 33 . 91 1, 416 38, 372
⋮ ⋮ ⋮ ⋮Look at the Big Picture | 43

• h is your system’s prediction function, also called a hypothesis . When your system
is given an instance’s feature vector x(i), it outputs a predicted value ŷ(i) = h(x(i))
for that instance ( ŷ is pronounced “y-hat”).—
For example, if your system predicts that the median housing price in the first
district is $158,400, then ŷ(1) = h(x(1)) = 158,400. The prediction error for this
district is ŷ(1) – y(1) = 2,000.•
RMSE( X,h) is the cost function measured on the set of examples using your
hypothesis h.
We use lowercase italic font for scalar values (such as m or y(i)) and function names
(such as h), lowercase bold font for vectors (such as x(i)), and uppercase bold font for
matrices (such as X).
Even though the RMSE is generally the preferred performance measure for regression
tasks, in some contexts you may prefer to use another function. For example, suppose
that there are many outlier districts. In that case, you may consider using the Mean

Absolute Error  (also called the Average Absolute Deviation; see Equation 2-2 ):
Equation 2-2. Mean Absolute ErrorMAE X,h=1
m∑
i= 1m
hxi−yi
Both the RMSE and the MAE are ways to measure the distance between two vectors:
the vector of predictions and the vector of target values. Various distance measures,
or norms , are possible:•
Computing the root of a sum of squares (RMSE) corresponds to the Euclidean
norm : it is the notion of distance you are familiar with. It is also called the ℓ2
norm , noted ∥ · ∥2 (or just ∥ · ∥).•

5The latest version of Python 3 is recommended. Python 2.7+ may work too, but it is now deprecated, all major
scientific libraries are dropping support for it, so you should migrate to Python 3 as soon as possible.outliers are exponentially rare (like in a bell-shaped curve), the RMSE performs
very well and is generally preferred.
Check the Assumptions
Lastly, it is good practice to list and verify the assumptions that were made so far (by
you or others); this can catch serious issues early on. For example, the district prices
that your system outputs are going to be fed into a downstream Machine Learning
system, and we assume that these prices are going to be used as such. But what if the
downstream system actually converts the prices into categories (e.g., “cheap, ”
“medium, ” or “expensive”) and then uses those categories instead of the prices them‐
selves? In this case, getting the price perfectly right is not important at all; your sys‐

tem just needs to get the category right. If that’s so, then the problem should have
been framed as a classification task, not a regression task. Y ou don’t want to find this
out after working on a regression system for months.
Fortunately, after talking with the team in charge of the downstream system, you are
confident that they do indeed need the actual prices, not just categories. Great! Y ou’re
all set, the lights are green, and you can start coding now!
Get the Data

It’s time to get your hands dirty. Don’t hesitate to pick up your laptop and walk
through the following code examples in a Jupyter notebook. The full Jupyter note‐
book is available at https://github.com/ageron/handson-ml2 .
Create the Workspace
First you will need to have Python installed. It is probably already installed on your
system. If not, you can get it at https://www.python.org/ .5
Next you need to create a workspace directory for your Machine Learning code and

datasets. Open a terminal and type the following commands (after the $ prompts):
$ export ML_PATH="$HOME/ml"      # You can change the path if you prefer
$ mkdir -p $ML_PATH
Y ou will need a number of Python modules: Jupyter, NumPy, Pandas, Matplotlib, and
Scikit-Learn. If you already have Jupyter running with all these modules installed,
you can safely skip to “Download the Data” on page 49. If you don’t have them yet,
there are many ways to install them (and their dependencies). Y ou can use your sys‐Get the Data | 45

6We will show the installation steps using pip in a bash shell on a Linux or MacOS system. Y ou may need to
adapt these commands to your own system. On Windows, we recommend installing Anaconda instead.7
If you want to upgrade pip for all users on your machine rather than just your own user, you should remove
the --user  option and make sure you have administrator rights (e.g., by adding sudo  before the whole com‐
mand on Linux or MacOSX).8
Alternative tools include venv (very similar to virtualenv and included in the standard library), virtualenv‐
wrapper (provides extra functionalities on top of virtualenv), pyenv (allows easy switching between Python
versions), and pipenv (a great packaging tool by the same author as the popular requests  library, built on top
of pip, virtualenv and more).tem’s packaging system (e.g., apt-get on Ubuntu, or MacPorts or HomeBrew on
MacOS), install a Scientific Python distribution such as Anaconda and use its packag‐

ing system, or just use Python’s own packaging system, pip, which is included by
default with the Python binary installers (since Python 2.7.9).6 Y ou can check to see if
pip is installed by typing the following command:
$ python3 -m pip --version
pip 19.0.2 from [...]/lib/python3.6/site-packages (python 3.6)
Y ou should make sure you have a recent version of pip installed. To upgrade the pip
module, type:7
$ python3 -m pip install --user -U pip
Collecting pip
[...]
Successfully installed pip-19.0.2
Creating an Isolated Environment

[...]
If you would like to work in an isolated environment (which is strongly recom‐
mended so you can work on different projects without having conflicting library ver‐
sions), install virtualenv8 by running the following pip command (again, if you want
virtualenv to be installed for all users on your machine, remove --user  and run this
command with administrator rights):
$ python3 -m pip install --user -U virtualenv
Collecting virtualenv
[...]
Successfully installed virtualenv
Now you can create an isolated Python environment by typing:
$ cd $ML_PATH

[...]
$ virtualenv env
Using base prefix '[...]'
New python executable in [...]/ml/env/bin/python3.6
Also creating executable in [...]/ml/env/bin/python
Installing setuptools, pip, wheel...done.46 | Chapter 2: End-to-End Machine Learning Project

9Note that Jupyter can handle multiple versions of Python, and even many other languages such as R or
Octave.Now every time you want to activate this environment, just open a terminal and type:
$ source env/bin/activate # on Linux or MacOSX
$ .\env\Scripts\activate  # on Windows
To deactivate this environment, just type deactivate . While the environment is
active, any package you install using pip will be installed in this isolated environment,
and Python will only have access to these packages (if you also want access to the sys‐
tem’s packages, you should create the environment using virtualenv’s --system-site-
packages  option). Check out virtualenv’s documentation for more information.
Now you can install all the required modules and their dependencies using this sim‐
ple pip command (if you are not using a virtualenv, you will need the --user  option
or administrator rights):
$ python3 -m pip install -U jupyter matplotlib numpy pandas scipy scikit-learn
Collecting jupyter

  Downloading jupyter-1.0.0-py2.py3-none-any.whl
Collecting matplotlib
  [...]
To check your installation, try to import every module like this:
$ python3 -c "import jupyter, matplotlib, numpy, pandas, scipy, sklearn"
There should be no output and no error. Now you can fire up Jupyter by typing:
$ jupyter notebook
[I 15:24 NotebookApp] Serving notebooks from local directory: [...]/ml
[I 15:24 NotebookApp] 0 active kernels
[I 15:24 NotebookApp] The Jupyter Notebook is running at: http://localhost:8888/

Downloading jupyter-1.0.0-py2.py3-none-any.whl
  [...]
[I 15:24 NotebookApp] Use Control-C to stop this server and shut down all
kernels (twice to skip confirmation).
A Jupyter server is now running in your terminal, listening to port 8888. Y ou can visit
this server by opening your web browser to http://localhost:8888/  (this usually hap‐
pens automatically when the server starts). Y ou should see your empty workspace
directory (containing only the env directory if you followed the preceding virtualenv
instructions).

Now create a new Python notebook by clicking on the New button and selecting the
appropriate Python version9 (see Figure 2-3 ).
This does three things: first, it creates a new notebook file called Untitled.ipynb  in
your workspace; second, it starts a Jupyter Python kernel to run this notebook; andGet the Data | 47

third, it opens this notebook in a new tab. Y ou should start by renaming this note‐
book to “Housing” (this will automatically rename the file to Housing.ipynb ) by click‐
ing Untitled and typing the new name.
Figure 2-3. Your workspace in Jupyter
A notebook contains a list of cells. Each cell can contain executable code or formatted
text. Right now the notebook contains only one empty code cell, labeled “In [1]:” . Try
typing print("Hello world!")  in the cell, and click on the play button (see
Figure 2-4 ) or press Shift-Enter. This sends the current cell to this notebook’s Python
kernel, which runs it and returns the output. The result is displayed below the cell,
and since we reached the end of the notebook, a new cell is automatically created. Go
through the User Interface Tour from Jupyter’s Help menu to learn the basics.
Figure 2-4. Hello world Python notebook48 | Chapter 2: End-to-End Machine Learning Project

10Y ou might also need to check legal constraints, such as private fields that should never be copied to unsafe
datastores.11
In a real project you would save this code in a Python file, but for now you can just write it in your Jupyter
notebook.Download the Data
In typical environments your data would be available in a relational database (or
some other common datastore) and spread across multiple tables/documents/files. To
access it, you would first need to get your credentials and access authorizations,10 and
familiarize yourself with the data schema. In this project, however, things are much
simpler: you will just download a single compressed file, housing.tgz , which contains a
comma-separated value (CSV) file called housing.csv  with all the data.
Y ou could use your web browser to download it, and run tar xzf housing.tgz  to
decompress the file and extract the CSV file, but it is preferable to create a small func‐

tion to do that. It is useful in particular if data changes regularly, as it allows you to
write a small script that you can run whenever you need to fetch the latest data (or
you can set up a scheduled job to do that automatically at regular intervals). Auto‐
mating the process of fetching the data is also useful if you need to install the dataset
on multiple machines.
Here is the function to fetch the data:11
import os
import tarfile
from six.moves  import urllib

import os
DOWNLOAD_ROOT  = "https://raw.githubusercontent.com/ageron/handson-ml2/master/"
HOUSING_PATH  = os.path.join("datasets" , "housing" )
HOUSING_URL  = DOWNLOAD_ROOT  + "datasets/housing/housing.tgz"
def fetch_housing_data (housing_url =HOUSING_URL , housing_path =HOUSING_PATH ):
    if not os.path.isdir(housing_path ):
        os.makedirs (housing_path )
    tgz_path  = os.path.join(housing_path , "housing.tgz" )
    urllib.request.urlretrieve (housing_url , tgz_path )
    housing_tgz  = tarfile.open(tgz_path )

    housing_tgz .extractall (path=housing_path )
    housing_tgz .close()
Now when you call fetch_housing_data() , it creates a datasets/housing  directory in
your workspace, downloads the housing.tgz  file, and extracts the housing.csv  from it in
this directory.
Now let’s load the data using Pandas. Once again you should write a small function to
load the data:Get the Data | 49

def load_housing_data (housing_path =HOUSING_PATH ):
    csv_path  = os.path.join(housing_path , "housing.csv" )
    return pd.read_csv (csv_path )
This function returns a Pandas DataFrame object containing all the data.
Take a Quick Look at the Data Structure
Let’s take a look at the top five rows using the DataFrame’s head()  method (see
Figure 2-5 ).
Figure 2-5. Top five rows in the dataset
Each row represents one district. There are 10 attributes (you can see the first 6 in the
screenshot): longitude , latitude , housing_median_age , total_rooms , total_bed
rooms , population , households , median_income , median_house_value , and
ocean_proximity .
The info()  method is useful to get a quick description of the data, in particular the
total number of rows, and each attribute’s type and number of non-null values (see
Figure 2-6 ).50 | Chapter 2: End-to-End Machine Learning Project

Figure 2-6. Housing info
There are 20,640 instances in the dataset, which means that it is fairly small by
Machine Learning standards, but it’s perfect to get started. Notice that the total_bed
rooms  attribute has only 20,433 non-null values, meaning that 207 districts are miss‐
ing this feature. We will need to take care of this later.
All attributes are numerical, except the ocean_proximity  field. Its type is object , so it
could hold any kind of Python object, but since you loaded this data from a CSV file
you know that it must be a text attribute. When you looked at the top five rows, you
probably noticed that the values in the ocean_proximity  column were repetitive,
which means that it is probably a categorical attribute. Y ou can find out what cate‐
gories exist and how many districts belong to each category by using the
value_counts()  method:
>>> housing["ocean_proximity" ].value_counts ()
<1H OCEAN     9136
INLAND        6551
NEAR OCEAN    2658
NEAR BAY      2290
ISLAND           5

ISLAND           5
Name: ocean_proximity, dtype: int64
Let’s look at the other fields. The describe()  method shows a summary of the
numerical attributes ( Figure 2-7 ).Get the Data | 51

12The standard deviation is generally denoted σ (the Greek letter sigma), and it is the square root of the var‐
iance , which is the average of the squared deviation from the mean. When a feature has a bell-shaped normal
distribution  (also called a Gaussian distribution ), which is very common, the “68-95-99.7” rule applies: about
68% of the values fall within 1σ of the mean, 95% within 2σ, and 99.7% within 3σ.
Figure 2-7. Summary of each numerical attribute
The count , mean , min, and max rows are self-explanatory. Note that the null values are
ignored (so, for example, count  of total_bedrooms  is 20,433, not 20,640). The std
row shows the standard deviation , which measures how dispersed the values are.12
The 25%, 50%, and 75% rows show the corresponding percentiles : a percentile indi‐
cates the value below which a given percentage of observations in a group of observa‐
tions falls. For example, 25% of the districts have a housing_median_age  lower than

18, while 50% are lower than 29 and 75% are lower than 37. These are often called the
25th percentile (or 1st quartile ), the median, and the 75th percentile (or 3rd quartile).
Another quick way to get a feel of the type of data you are dealing with is to plot a 
histogram for each numerical attribute. A histogram shows the number of instances
(on the vertical axis) that have a given value range (on the horizontal axis). Y ou can

either plot this one attribute at a time, or you can call the hist()  method on the
whole dataset, and it will plot a histogram for each numerical attribute (see
Figure 2-8 ). For example, you can see that slightly over 800 districts have a
median_house_value  equal to about $100,000.
%matplotlib  inline   # only in a Jupyter notebook
housing.hist(bins=50, figsize=(20,15))
plt.show()52 | Chapter 2: End-to-End Machine Learning Project

The hist()  method relies on Matplotlib, which in turn relies on a
user-specified graphical backend to draw on your screen. So before
you can plot anything, you need to specify which backend Matplot‐
lib should use. The simplest option is to use Jupyter’s magic com‐
mand %matplotlib inline . This tells Jupyter to set up Matplotlib
so it uses Jupyter’s own backend. Plots are then rendered within the
notebook itself. Note that calling show()  is optional in a Jupyter
notebook, as Jupyter will automatically display plots when a cell is
executed.
Figure 2-8. A histogram for each numerical attribute
Notice a few things in these histograms:1.
First, the median income attribute does not look like it is expressed in US dollars
(USD). After checking with the team that collected the data, you are told that the
data has been scaled and capped at 15 (actually 15.0001) for higher median
incomes, and at 0.5 (actually 0.4999) for lower median incomes. The numbers

executed.
represent roughly tens of thousands of dollars (e.g., 3 actually means about
$30,000). Working with preprocessed attributes is common in Machine Learning,Get the Data | 53

and it is not necessarily a problem, but you should try to understand how the
data was computed.2.
The housing median age and the median house value were also capped. The lat‐
ter may be a serious problem since it is your target attribute (your labels). Y our
Machine Learning algorithms may learn that prices never go beyond that limit.
Y ou need to check with your client team (the team that will use your system’s out‐
put) to see if this is a problem or not. If they tell you that they need precise pre‐
dictions even beyond $500,000, then you have mainly two options:a.
Collect proper labels for the districts whose labels were capped.b.
Remove those districts from the training set (and also from the test set, since
your system should not be evaluated poorly if it predicts values beyond
$500,000).3.
These attributes have very different scales. We will discuss this later in this chap‐
ter when we explore feature scaling.4.
Finally, many histograms are tail heavy : they extend much farther to the right of

the median than to the left. This may make it a bit harder for some Machine
Learning algorithms to detect patterns. We will try transforming these attributes
later on to have more bell-shaped distributions.
Hopefully you now have a better understanding of the kind of data you are dealing
with.
Wait! Before you look at the data any further, you need to create a
test set, put it aside, and never look at it.
Create a Test Set
It may sound strange to voluntarily set aside part of the data at this stage. After all,

with.
you have only taken a quick glance at the data, and surely you should learn a whole
lot more about it before you decide what algorithms to use, right? This is true, but
your brain is an amazing pattern detection system, which means that it is highly
prone to overfitting: if you look at the test set, you may stumble upon some seemingly
interesting pattern in the test data that leads you to select a particular kind of
Machine Learning model. When you estimate the generalization error using the test

set, your estimate will be too optimistic and you will launch a system that will not
perform as well as expected. This is called data snooping  bias.
Creating a test set is theoretically quite simple: just pick some instances randomly,
typically 20% of the dataset (or less if your dataset is very large), and set them aside:54 | Chapter 2: End-to-End Machine Learning Project

13In this book, when a code example contains a mix of code and outputs, as is the case here, it is formatted like
in the Python interpreter, for better readability: the code lines are prefixed with >>> (or ... for indented
blocks), and the outputs have no prefix.14
Y ou will often see people set the random seed to 42. This number has no special property, other than to be
The Answer to the Ultimate Question of Life, the Universe, and Everything.import numpy as np
def split_train_test (data, test_ratio ):
    shuffled_indices  = np.random.permutation (len(data))
    test_set_size  = int(len(data) * test_ratio )
    test_indices  = shuffled_indices [:test_set_size ]
    train_indices  = shuffled_indices [test_set_size :]
    return data.iloc[train_indices ], data.iloc[test_indices ]
Y ou can then use this function like this:13
>>> train_set , test_set  = split_train_test (housing, 0.2)
>>> len(train_set )
16512
>>> len(test_set )
4128

16512
4128
Well, this works, but it is not perfect: if you run the program again, it will generate a
different test set! Over time, you (or your Machine Learning algorithms) will get to
see the whole dataset, which is what you want to avoid.
One solution is to save the test set on the first run and then load it in subsequent
runs. Another option is to set the random number generator’s seed (e.g., np.ran
dom.seed(42) )14 before calling np.random.permutation() , so that it always generates
the same shuffled indices.

But both these solutions will break next time you fetch an updated dataset. A com‐
mon solution is to use each instance’s identifier to decide whether or not it should go
in the test set (assuming instances have a unique and immutable identifier). For
example, you could compute a hash of each instance’s identifier and put that instance
in the test set if the hash is lower or equal to 20% of the maximum hash value. This
ensures that the test set will remain consistent across multiple runs, even if you

refresh the dataset. The new test set will contain 20% of the new instances, but it will
not contain any instance that was previously in the training set. Here is a possible
implementation:
from zlib import crc32
def test_set_check (identifier , test_ratio ):
    return crc32(np.int64(identifier )) & 0xffffffff  < test_ratio  * 2**32
def split_train_test_by_id (data, test_ratio , id_column ):
    ids = data[id_column ]Get the Data | 55

15The location information is actually quite coarse, and as a result many districts will have the exact same ID, so
they will end up in the same set (test or train). This introduces some unfortunate sampling bias.    in_test_set  = ids.apply(lambda id_: test_set_check (id_, test_ratio ))
    return data.loc[~in_test_set ], data.loc[in_test_set ]
Unfortunately, the housing dataset does not have an identifier column. The simplest
solution is to use the row index as the ID:
housing_with_id  = housing.reset_index ()   # adds an `index` column
train_set , test_set  = split_train_test_by_id (housing_with_id , 0.2, "index")
If you use the row index as a unique identifier, you need to make sure that new data
gets appended to the end of the dataset, and no row ever gets deleted. If this is not
possible, then you can try to use the most stable features to build a unique identifier.
For example, a district’s latitude and longitude are guaranteed to be stable for a few

million years, so you could combine them into an ID like so:15
housing_with_id ["id"] = housing["longitude" ] * 1000 + housing["latitude" ]
train_set , test_set  = split_train_test_by_id (housing_with_id , 0.2, "id")
Scikit-Learn provides a few functions to split datasets into multiple subsets in various
ways. The simplest function is train_test_split , which does pretty much the same
thing as the function split_train_test  defined earlier, with a couple of additional

features. First there is a random_state  parameter that allows you to set the random
generator seed as explained previously, and second you can pass it multiple datasets
with an identical number of rows, and it will split them on the same indices (this is
very useful, for example, if you have a separate DataFrame for labels):
from sklearn.model_selection  import train_test_split
train_set , test_set  = train_test_split (housing, test_size =0.2, random_state =42)
So far we have considered purely random sampling methods. This is generally fine if

your dataset is large enough (especially relative to the number of attributes), but if it
is not, you run the risk of introducing a significant sampling bias. When a survey
company decides to call 1,000 people to ask them a few questions, they don’t just pick
1,000 people randomly in a phone book. They try to ensure that these 1,000 people
are representative of the whole population. For example, the US population is com‐
posed of 51.3% female and 48.7% male, so a well-conducted survey in the US would

try to maintain this ratio in the sample: 513 female and 487 male. This is called strati‐
fied sampling : the population is divided into homogeneous subgroups called strata ,
and the right number of instances is sampled from each stratum to guarantee that the
test set is representative of the overall population. If they used purely random sam‐
pling, there would be about 12% chance of sampling a skewed test set with either less
than 49% female or more than 54% female. Either way, the survey results would be

significantly biased.56 | Chapter 2: End-to-End Machine Learning Project

Suppose you chatted with experts who told you that the median income is a very
important attribute to predict median housing prices. Y ou may want to ensure that
the test set is representative of the various categories of incomes in the whole dataset.
Since the median income is a continuous numerical attribute, you first need to create
an income category attribute. Let’s look at the median income histogram more closely
(back in Figure 2-8 ): most median income values are clustered around 1.5 to 6 (i.e.,
$15,000–$60,000), but some median incomes go far beyond 6. It is important to have
a sufficient number of instances in your dataset for each stratum, or else the estimate
of the stratum’s importance may be biased. This means that you should not have too
many strata, and each stratum should be large enough. The following code uses the
pd.cut()  function to create an income category attribute with 5 categories (labeled
from 1 to 5): category 1 ranges from 0 to 1.5 (i.e., less than $15,000), category 2 from

1.5 to 3, and so on:
housing["income_cat" ] = pd.cut(housing["median_income" ],
                               bins=[0., 1.5, 3.0, 4.5, 6., np.inf],
                               labels=[1, 2, 3, 4, 5])
These income categories are represented in Figure 2-9 :
housing["income_cat" ].hist()
Figure 2-9. Histogram of income categories
Now you are ready to do stratified sampling based on the income category. For this
you can use Scikit-Learn’s StratifiedShuffleSplit  class:

from sklearn.model_selection  import StratifiedShuffleSplit
split = StratifiedShuffleSplit (n_splits =1, test_size =0.2, random_state =42)
for train_index , test_index  in split.split(housing, housing["income_cat" ]):
    strat_train_set  = housing.loc[train_index ]
    strat_test_set  = housing.loc[test_index ]Get the Data | 57

Let’s see if this worked as expected. Y ou can start by looking at the income category
proportions in the test set:
>>> strat_test_set ["income_cat" ].value_counts () / len(strat_test_set )
3    0.350533
2    0.318798
4    0.176357
5    0.114583
1    0.039729
Name: income_cat, dtype: float64
With similar code you can measure the income category proportions in the full data‐
set. Figure 2-10  compares the income category proportions in the overall dataset, in
the test set generated with stratified sampling, and in a test set generated using purely
random sampling. As you can see, the test set generated using stratified sampling has
income category proportions almost identical to those in the full dataset, whereas the
test set generated using purely random sampling is quite skewed.
Figure 2-10. Sampling bias comparison of stratified  versus purely random sampling
Now you should remove the income_cat  attribute so the data is back to its original
state:
for set_ in (strat_train_set , strat_test_set ):

state:
    set_.drop("income_cat" , axis=1, inplace=True)
We spent quite a bit of time on test set generation for a good reason: this is an often
neglected but critical part of a Machine Learning project. Moreover, many of these
ideas will be useful later when we discuss cross-validation. Now it’s time to move on
to the next stage: exploring the data.
Discover and Visualize the Data to Gain Insights
So far you have only taken a quick glance at the data to get a general understanding of

set_.drop("income_cat" , axis=1, inplace=True)
the kind of data you are manipulating. Now the goal is to go a little bit more in depth.
First, make sure you have put the test set aside and you are only exploring the train‐
ing set. Also, if the training set is very large, you may want to sample an exploration58 | Chapter 2: End-to-End Machine Learning Project

set, to make manipulations easy and fast. In our case, the set is quite small so you can
just work directly on the full set. Let’s create a copy so you can play with it without
harming the training set:
housing = strat_train_set .copy()
Visualizing Geographical Data
Since there is geographical information (latitude and longitude), it is a good idea to
create a scatterplot of all districts to visualize the data ( Figure 2-11 ):
housing.plot(kind="scatter" , x="longitude" , y="latitude" )
Figure 2-11. A geographical scatterplot of the data
This looks like California all right, but other than that it is hard to see any particular
pattern. Setting the alpha  option to 0.1 makes it much easier to visualize the places
where there is a high density of data points ( Figure 2-12 ):
housing.plot(kind="scatter" , x="longitude" , y="latitude" , alpha=0.1)Discover and Visualize the Data to Gain Insights | 59

16If you are reading this in grayscale, grab a red pen and scribble over most of the coastline from the Bay Area
down to San Diego (as you might expect). Y ou can add a patch of yellow around Sacramento as well.
Figure 2-12. A better visualization highlighting high-density areas
Now that’s much better: you can clearly see the high-density areas, namely the Bay
Area and around Los Angeles and San Diego, plus a long line of fairly high density in
the Central Valley, in particular around Sacramento and Fresno.
More generally, our brains are very good at spotting patterns on pictures, but you
may need to play around with visualization parameters to make the patterns stand
out.
Now let’s look at the housing prices ( Figure 2-13 ). The radius of each circle represents
the district’s population (option s), and the color represents the price (option c). We
will use a predefined color map (option cmap ) called jet, which ranges from blue
(low values) to red (high prices):16

out.
housing.plot(kind="scatter" , x="longitude" , y="latitude" , alpha=0.4,
    s=housing["population" ]/100, label="population" , figsize=(10,7),
    c="median_house_value" , cmap=plt.get_cmap ("jet"), colorbar =True,
)
plt.legend()60 | Chapter 2: End-to-End Machine Learning Project

Figure 2-13. California housing pricesDiscover and Visualize the Data to Gain Insights | 61

This image tells you that the housing prices are very much related to the location
(e.g., close to the ocean) and to the population density, as you probably knew already.
It will probably be useful to use a clustering algorithm to detect the main clusters, and
add new features that measure the proximity to the cluster centers. The ocean prox‐
imity attribute may be useful as well, although in Northern California the housing
prices in coastal districts are not too high, so it is not a simple rule.
Looking for Correlations
Since the dataset is not too large, you can easily compute the standard correlation
coefficient  (also called Pearson’s r ) between every pair of attributes using the corr()
method:
corr_matrix  = housing.corr()
Now let’s look at how much each attribute correlates with the median house value:
>>> corr_matrix ["median_house_value" ].sort_values (ascending =False)
median_house_value    1.000000
median_income         0.687170
total_rooms           0.135231
housing_median_age    0.114220

method:
households            0.064702
total_bedrooms        0.047865
population           -0.026699
longitude            -0.047279
latitude             -0.142826
Name: median_house_value, dtype: float64
The correlation coefficient ranges from –1 to 1. When it is close to 1, it means that
there is a strong positive correlation; for example, the median house value tends to go
up when the median income goes up. When the coefficient is close to –1, it means

that there is a strong negative correlation; you can see a small negative correlation
between the latitude and the median house value (i.e., prices have a slight tendency to
go down when you go north). Finally, coefficients close to zero mean that there is no
linear correlation. Figure 2-14  shows various plots along with the correlation coeffi‐
cient between their horizontal and vertical axes.62 | Chapter 2: End-to-End Machine Learning Project

Figure 2-14. Standard correlation coefficient  of various datasets (source: Wikipedia;
public domain image)
The correlation coefficient only measures linear correlations (“if x
goes up, then y generally goes up/down”). It may completely miss
out on nonlinear relationships (e.g., “if x is close to zero then y gen‐
erally goes up”). Note how all the plots of the bottom row have a
correlation coefficient equal to zero despite the fact that their axes
are clearly not independent: these are examples of nonlinear rela‐
tionships. Also, the second row shows examples where the correla‐
tion coefficient is equal to 1 or –1; notice that this has nothing to
do with the slope. For example, your height in inches has a correla‐
tion coefficient of 1 with your height in feet or in nanometers.
Another way to check for correlation between attributes is to use Pandas’
scatter_matrix  function, which plots every numerical attribute against every other

numerical attribute. Since there are now 11 numerical attributes, you would get 112 =
121 plots, which would not fit on a page, so let’s just focus on a few promising
attributes that seem most correlated with the median housing value ( Figure 2-15 ):
from pandas.plotting  import scatter_matrix
attributes  = ["median_house_value" , "median_income" , "total_rooms" ,
              "housing_median_age" ]
scatter_matrix (housing[attributes ], figsize=(12, 8))Discover and Visualize the Data to Gain Insights | 63

Figure 2-15. Scatter matrix
The main diagonal (top left to bottom right) would be full of straight lines if Pandas
plotted each variable against itself, which would not be very useful. So instead Pandas
displays a histogram of each attribute (other options are available; see Pandas’ docu‐
mentation for more details).
The most promising attribute to predict the median house value is the median
income, so let’s zoom in on their correlation scatterplot ( Figure 2-16 ):
housing.plot(kind="scatter" , x="median_income" , y="median_house_value" ,
             alpha=0.1)
This plot reveals a few things. First, the correlation is indeed very strong; you can
clearly see the upward trend and the points are not too dispersed. Second, the price
cap that we noticed earlier is clearly visible as a horizontal line at $500,000. But this
plot reveals other less obvious straight lines: a horizontal line around $450,000,
another around $350,000, perhaps one around $280,000, and a few more below that.

alpha=0.1)
Y ou may want to try removing the corresponding districts to prevent your algorithms
from learning to reproduce these data quirks.64 | Chapter 2: End-to-End Machine Learning Project

Figure 2-16. Median income versus median house value
Experimenting with Attribute Combinations
Hopefully the previous sections gave you an idea of a few ways you can explore the
data and gain insights. Y ou identified a few data quirks that you may want to clean up
before feeding the data to a Machine Learning algorithm, and you found interesting
correlations between attributes, in particular with the target attribute. Y ou also
noticed that some attributes have a tail-heavy distribution, so you may want to trans‐
form them (e.g., by computing their logarithm). Of course, your mileage will vary
considerably with each project, but the general ideas are similar.
One last thing you may want to do before actually preparing the data for Machine
Learning algorithms is to try out various attribute combinations. For example, the
total number of rooms in a district is not very useful if you don’t know how many
households there are. What you really want is the number of rooms per household.

Similarly, the total number of bedrooms by itself is not very useful: you probably
want to compare it to the number of rooms. And the population per household also
seems like an interesting attribute combination to look at. Let’s create these new
attributes:
housing["rooms_per_household" ] = housing["total_rooms" ]/housing["households" ]
housing["bedrooms_per_room" ] = housing["total_bedrooms" ]/housing["total_rooms" ]
housing["population_per_household" ]=housing["population" ]/housing["households" ]

And now let’s look at the correlation matrix again:
>>> corr_matrix  = housing.corr()
median_house_value          1.000000Discover and Visualize the Data to Gain Insights | 65

median_income               0.687160
rooms_per_household         0.146285
total_rooms                 0.135097
housing_median_age          0.114110
households                  0.064506
total_bedrooms              0.047689
population_per_household   -0.021985
population                 -0.026920
longitude                  -0.047432
latitude                   -0.142724
bedrooms_per_room          -0.259984
Hey, not bad! The new bedrooms_per_room  attribute is much more correlated with
the median house value than the total number of rooms or bedrooms. Apparently
houses with a lower bedroom/room ratio tend to be more expensive. The number of
rooms per household is also more informative than the total number of rooms in a
district—obviously the larger the houses, the more expensive they are.
This round of exploration does not have to be absolutely thorough; the point is to
start off on the right foot and quickly gain insights that will help you get a first rea‐

sonably good prototype. But this is an iterative process: once you get a prototype up
and running, you can analyze its output to gain more insights and come back to this
exploration step.
Prepare the Data for Machine Learning Algorithms
It’s time to prepare the data for your Machine Learning algorithms. Instead of just
doing this manually, you should write functions to do that, for several good reasons:•
This will allow you to reproduce these transformations easily on any dataset (e.g.,
the next time you get a fresh dataset).•

Y ou will gradually build a library of transformation functions that you can reuse
in future projects.•
Y ou can use these functions in your live system to transform the new data before
feeding it to your algorithms.•
This will make it possible for you to easily try various transformations and see
which combination of transformations works best.
But first let’s revert to a clean training set (by copying strat_train_set  once again),
and let’s separate the predictors and the labels since we don’t necessarily want to apply

the same transformations to the predictors and the target values (note that drop()  
creates a copy of the data and does not affect strat_train_set ):
housing = strat_train_set .drop("median_house_value" , axis=1)
housing_labels  = strat_train_set ["median_house_value" ].copy()66 | Chapter 2: End-to-End Machine Learning Project

Data Cleaning
Most Machine Learning algorithms cannot work with missing features, so let’s create
a few functions to take care of them. Y ou noticed earlier that the total_bedrooms
attribute has some missing values, so let’s fix this. Y ou have three options:•
Get rid of the corresponding districts.•
Get rid of the whole attribute.•
Set the values to some value (zero, the mean, the median, etc.).
Y ou can accomplish these easily using DataFrame’s dropna() , drop() , and fillna()
methods:
housing.dropna(subset=["total_bedrooms" ])    # option 1
housing.drop("total_bedrooms" , axis=1)       # option 2
median = housing["total_bedrooms" ].median()  # option 3
housing["total_bedrooms" ].fillna(median, inplace=True)
If you choose option 3, you should compute the median value on the training set, and
use it to fill the missing values in the training set, but also don’t forget to save the
median value that you have computed. Y ou will need it later to replace missing values

methods:
in the test set when you want to evaluate your system, and also once the system goes
live to replace missing values in new data.
Scikit-Learn provides a handy class to take care of missing values: SimpleImputer .
Here is how to use it. First, you need to create a SimpleImputer  instance, specifying
that you want to replace each attribute’s missing values with the median of that
attribute:
from sklearn.impute  import SimpleImputer
imputer = SimpleImputer (strategy ="median" )

Since the median can only be computed on numerical attributes, we need to create a
copy of the data without the text attribute ocean_proximity :
housing_num  = housing.drop("ocean_proximity" , axis=1)
Now you can fit the imputer  instance to the training data using the fit()  method:
imputer.fit(housing_num )
The imputer  has simply computed the median of each attribute and stored the result
in its statistics_  instance variable. Only the total_bedrooms  attribute had missing

values, but we cannot be sure that there won’t be any missing values in new data after
the system goes live, so it is safer to apply the imputer  to all the numerical attributes:
>>> imputer.statistics_
array([ -118.51 , 34.26 , 29. , 2119.5 , 433. , 1164. , 408. , 3.5409])Prepare the Data for Machine Learning Algorithms | 67

17For more details on the design principles, see “ API design for machine learning software: experiences from
the scikit-learn project, ” L. Buitinck, G. Louppe, M. Blondel, F. Pedregosa, A. Müller, et al. (2013).>>> housing_num .median().values
array([ -118.51 , 34.26 , 29. , 2119.5 , 433. , 1164. , 408. , 3.5409])
Now you can use this “trained” imputer  to transform the training set by replacing
missing values by the learned medians:
X = imputer.transform (housing_num )
The result is a plain NumPy array containing the transformed features. If you want to
put it back into a Pandas DataFrame, it’s simple:
housing_tr  = pd.DataFrame (X, columns=housing_num .columns)
Scikit-Learn Design
Scikit-Learn’s API is remarkably well designed. The main design principles  are:17•
Consistency . All objects share a consistent and simple interface:—
Estimators . Any object that can estimate some parameters based on a dataset
is called an estimator  (e.g., an imputer  is an estimator). The estimation itself is

performed by the fit()  method, and it takes only a dataset as a parameter (or
two for supervised learning algorithms; the second dataset contains the
labels). Any other parameter needed to guide the estimation process is con‐
sidered a hyperparameter (such as an imputer ’s strategy ), and it must be set
as an instance variable (generally via a constructor parameter).—
Transformers . Some estimators (such as an imputer ) can also transform a
dataset; these are called transformers . Once again, the API is quite simple: the

transformation is performed by the transform()  method with the dataset to
transform as a parameter. It returns the transformed dataset. This transforma‐
tion generally relies on the learned parameters, as is the case for an imputer .
All transformers also have a convenience method called fit_transform()  
that is equivalent to calling fit()  and then transform()  (but sometimes
fit_transform()  is optimized and runs much faster).—
Predictors . Finally, some estimators are capable of making predictions given a

dataset; they are called predictors . For example, the LinearRegression  model 
in the previous chapter was a predictor: it predicted life satisfaction given a
country’s GDP per capita. A predictor has a predict()  method that takes a
dataset of new instances and returns a dataset of corresponding predictions. It
also has a score()  method that measures the quality of the predictions given68 | Chapter 2: End-to-End Machine Learning Project

18Some predictors also provide methods to measure the confidence of their predictions.19
This class is available since Scikit-Learn 0.20. If you use an earlier version, please consider upgrading, or use
Pandas’ Series.factorize()  method.a test set (and the corresponding labels in the case of supervised learning
algorithms).18•
Inspection . All the estimator’s hyperparameters are accessible directly via public
instance variables (e.g., imputer.strategy ), and all the estimator’s learned
parameters are also accessible via public instance variables with an underscore
suffix (e.g., imputer.statistics_ ).•
Nonproliferation of classes . Datasets are represented as NumPy arrays or SciPy
sparse matrices, instead of homemade classes. Hyperparameters are just regular
Python strings or numbers.•
Composition . Existing building blocks are reused as much as possible. For
example, it is easy to create a Pipeline  estimator from an arbitrary sequence of
transformers followed by a final estimator, as we will see.•

Sensible defaults . Scikit-Learn provides reasonable default values for most
parameters, making it easy to create a baseline working system quickly.
Handling Text and Categorical Attributes
Earlier we left out the categorical attribute ocean_proximity  because it is a text
attribute so we cannot compute its median:
>>> housing_cat  = housing[["ocean_proximity" ]]
>>> housing_cat .head(10)
      ocean_proximity
17606       <1H OCEAN
18632       <1H OCEAN
14650      NEAR OCEAN
3230           INLAND
3555        <1H OCEAN
19480          INLAND
8879        <1H OCEAN

13685          INLAND
4937        <1H OCEAN
4861        <1H OCEAN
Most Machine Learning algorithms prefer to work with numbers anyway, so let’s con‐
vert these categories from text to numbers. For this, we can use Scikit-Learn’s Ordina
lEncoder  class19:
>>> from sklearn.preprocessing  import OrdinalEncoder
>>> ordinal_encoder  = OrdinalEncoder ()Prepare the Data for Machine Learning Algorithms | 69

20Before Scikit-Learn 0.20, it could only encode integer categorical values, but since 0.20 it can also handle
other types of inputs, including text categorical inputs.>>> housing_cat_encoded  = ordinal_encoder .fit_transform (housing_cat )
>>> housing_cat_encoded [:10]
array([[0.],
       [0.],
       [4.],
       [1.],
       [0.],
       [1.],
       [0.],
       [1.],
       [0.],
       [0.]])
Y ou can get the list of categories using the categories_  instance variable. It is a list
containing a 1D array of categories for each categorical attribute (in this case, a list
containing a single array since there is just one categorical attribute):
>>> ordinal_encoder .categories_
[array(['<1H OCEAN', 'INLAND', 'ISLAND', 'NEAR BAY', 'NEAR OCEAN'],
       dtype=object)]
One issue with this representation is that ML algorithms will assume that two nearby
values are more similar than two distant values. This may be fine in some cases (e.g.,

for ordered categories such as “bad” , “average” , “good” , “excellent”), but it is obviously
not the case for the ocean_proximity  column (for example, categories 0 and 4 are
clearly more similar than categories 0 and 1). To fix this issue, a common solution is
to create one binary attribute per category: one attribute equal to 1 when the category
is “<1H OCEAN” (and 0 otherwise), another attribute equal to 1 when the category is
“INLAND” (and 0 otherwise), and so on. This is called one-hot encoding , because

only one attribute will be equal to 1 (hot), while the others will be 0 (cold). The new
attributes are sometimes called dummy  attributes. Scikit-Learn provides a OneHotEn
coder  class to convert categorical values into one-hot vectors20:
>>> from sklearn.preprocessing  import OneHotEncoder
>>> cat_encoder  = OneHotEncoder ()
>>> housing_cat_1hot  = cat_encoder .fit_transform (housing_cat )
>>> housing_cat_1hot
<16512x5 sparse matrix of type '<class 'numpy.float64'>'
  with 16512 stored elements in Compressed Sparse Row format>

Notice that the output is a SciPy sparse matrix , instead of a NumPy array. This is very
useful when you have categorical attributes with thousands of categories. After one-
hot encoding we get a matrix with thousands of columns, and the matrix is full of
zeros except for a single 1 per row. Using up tons of memory mostly to store zeros
would be very wasteful, so instead a sparse matrix only stores the location of the non‐70 | Chapter 2: End-to-End Machine Learning Project

21See SciPy’s documentation for more details.
zero elements. Y ou can use it mostly like a normal 2D array,21 but if you really want to
convert it to a (dense) NumPy array, just call the toarray()  method:
>>> housing_cat_1hot .toarray()
array([[1., 0., 0., 0., 0.],
       [1., 0., 0., 0., 0.],
       [0., 0., 0., 0., 1.],
       ...,
       [0., 1., 0., 0., 0.],
       [0., 0., 0., 1., 0.]])
Once again, you can get the list of categories using the encoder’s categories_
instance variable:
>>> cat_encoder .categories_
If a categorical attribute has a large number of possible categories
(e.g., country code, profession, species, etc.), then one-hot encod‐
ing will result in a large number of input features. This may slow
down training and degrade performance. If this happens, you may
want to replace the categorical input with useful numerical features

related to the categories: for example, you could replace the
ocean_proximity  feature with the distance to the ocean (similarly,
a country code could be replaced with the country’s population and
GDP per capita). Alternatively, you could replace each category
with a learnable low dimensional vector called an embedding . Each
category’s representation would be learned during training: this is
an example of representation learning  (see Chapter 13  and ??? for
more details).
Custom Transformers

Although Scikit-Learn provides many useful transformers, you will need to write
your own for tasks such as custom cleanup operations or combining specific
attributes. Y ou will want your transformer to work seamlessly with Scikit-Learn func‐
tionalities (such as pipelines), and since Scikit-Learn relies on duck typing (not inher‐
itance), all you need is to create a class and implement three methods: fit()
(returning self ), transform() , and fit_transform() . Y ou can get the last one for

free by simply adding TransformerMixin  as a base class. Also, if you add BaseEstima
tor as a base class (and avoid *args  and **kargs  in your constructor) you will get
two extra methods ( get_params()  and set_params() ) that will be useful for auto‐Prepare the Data for Machine Learning Algorithms | 71

matic hyperparameter tuning. For example, here is a small transformer class that adds
the combined attributes we discussed earlier:
from sklearn.base  import BaseEstimator , TransformerMixin
rooms_ix , bedrooms_ix , population_ix , households_ix  = 3, 4, 5, 6
class CombinedAttributesAdder (BaseEstimator , TransformerMixin ):
    def __init__ (self, add_bedrooms_per_room  = True): # no *args or **kargs
        self.add_bedrooms_per_room  = add_bedrooms_per_room
    def fit(self, X, y=None):
        return self  # nothing else to do
    def transform (self, X, y=None):
        rooms_per_household  = X[:, rooms_ix ] / X[:, households_ix ]
        population_per_household  = X[:, population_ix ] / X[:, households_ix ]
        if self.add_bedrooms_per_room :
            bedrooms_per_room  = X[:, bedrooms_ix ] / X[:, rooms_ix ]
            return np.c_[X, rooms_per_household , population_per_household ,
                         bedrooms_per_room ]
        else:

def fit(self, X, y=None):
        else:
            return np.c_[X, rooms_per_household , population_per_household ]
attr_adder  = CombinedAttributesAdder (add_bedrooms_per_room =False)
housing_extra_attribs  = attr_adder .transform (housing.values)
In this example the transformer has one hyperparameter, add_bedrooms_per_room ,
set to True  by default (it is often helpful to provide sensible defaults). This hyperpara‐
meter will allow you to easily find out whether adding this attribute helps the

else:
Machine Learning algorithms or not. More generally, you can add a hyperparameter
to gate any data preparation step that you are not 100% sure about. The more you
automate these data preparation steps, the more combinations you can automatically
try out, making it much more likely that you will find a great combination (and sav‐
ing you a lot of time).
Feature Scaling
One of the most important transformations you need to apply to your data is feature
scaling . With few exceptions, Machine Learning algorithms don’t perform well when

the input numerical attributes have very different scales. This is the case for the hous‐
ing data: the total number of rooms ranges from about 6 to 39,320, while the median
incomes only range from 0 to 15. Note that scaling the target values is generally not
required.
There are two common ways to get all attributes to have the same scale: min-max
scaling  and standardization .
Min-max scaling (many people call this normalization ) is quite simple: values are
shifted and rescaled so that they end up ranging from 0 to 1. We do this by subtract‐

required.
ing the min value and dividing by the max minus the min. Scikit-Learn provides a72 | Chapter 2: End-to-End Machine Learning Project

transformer called MinMaxScaler  for this. It has a feature_range  hyperparameter
that lets you change the range if you don’t want 0–1 for some reason.
Standardization is quite different: first it subtracts the mean value (so standardized
values always have a zero mean), and then it divides by the standard deviation so that
the resulting distribution has unit variance. Unlike min-max scaling, standardization
does not bound values to a specific range, which may be a problem for some algo‐
rithms (e.g., neural networks often expect an input value ranging from 0 to 1). How‐
ever, standardization is much less affected by outliers. For example, suppose a district
had a median income equal to 100 (by mistake). Min-max scaling would then crush
all the other values from 0–15 down to 0–0.15, whereas standardization would not be
much affected. Scikit-Learn provides a transformer called StandardScaler  for stand‐
ardization.
As with all the transformations, it is important to fit the scalers to

the training data only, not to the full dataset (including the test set).
Only then can you use them to transform the training set and the
test set (and new data).
Transformation Pipelines
As you can see, there are many data transformation steps that need to be executed in
the right order. Fortunately, Scikit-Learn provides the Pipeline  class to help with
such sequences of transformations. Here is a small pipeline for the numerical
from sklearn.pipeline  import Pipeline

from sklearn.preprocessing  import StandardScaler
num_pipeline  = Pipeline ([
        ('imputer' , SimpleImputer (strategy ="median" )),
        ('attribs_adder' , CombinedAttributesAdder ()),
        ('std_scaler' , StandardScaler ()),
    ])
housing_num_tr  = num_pipeline .fit_transform (housing_num )
The Pipeline  constructor takes a list of name/estimator pairs defining a sequence of
steps. All but the last estimator must be transformers (i.e., they must have a

    ])
fit_transform()  method). The names can be anything you like (as long as they are
unique and don’t contain double underscores “ __”): they will come in handy later for
hyperparameter tuning.
When you call the pipeline’s fit()  method, it calls fit_transform()  sequentially on
all transformers, passing the output of each call as the parameter to the next call, until
it reaches the final estimator, for which it just calls the fit()  method.Prepare the Data for Machine Learning Algorithms | 73

22Just like for pipelines, the name can be anything as long as it does not contain double underscores.The pipeline exposes the same methods as the final estimator. In this example, the last
estimator is a StandardScaler , which is a transformer, so the pipeline has a trans
form()  method that applies all the transforms to the data in sequence (and of course
also a fit_transform()  method, which is the one we used).
So far, we have handled the categorical columns and the numerical columns sepa‐
rately. It would be more convenient to have a single transformer able to handle all col‐
umns, applying the appropriate transformations to each column. In version 0.20,
Scikit-Learn introduced the ColumnTransformer  for this purpose, and the good news
is that it works great with Pandas DataFrames. Let’s use it to apply all the transforma‐
tions to the housing data:
from sklearn.compose  import ColumnTransformer
num_attribs  = list(housing_num )
cat_attribs  = ["ocean_proximity" ]
full_pipeline  = ColumnTransformer ([

        ("num", num_pipeline , num_attribs ),
        ("cat", OneHotEncoder (), cat_attribs ),
    ])
housing_prepared  = full_pipeline .fit_transform (housing)
Here is how this works: first we import the ColumnTransformer  class, next we get the
list of numerical column names and the list of categorical column names, and we
construct a ColumnTransformer . The constructor requires a list of tuples, where each
tuple contains a name22, a transformer and a list of names (or indices) of columns
that the transformer should be applied to. In this example, we specify that the numer‐

])
ical columns should be transformed using the num_pipeline  that we defined earlier,
and the categorical columns should be transformed using a OneHotEncoder . Finally,
we apply this ColumnTransformer  to the housing data: it applies each transformer to
the appropriate columns and concatenates the outputs along the second axis (the
transformers must return the same number of rows).
Note that the OneHotEncoder  returns a sparse matrix, while the num_pipeline  returns

a dense matrix. When there is such a mix of sparse and dense matrices, the Colum
nTransformer  estimates the density of the final matrix (i.e., the ratio of non-zero
cells), and it returns a sparse matrix if the density is lower than a given threshold (by
default, sparse_threshold=0.3 ). In this example, it returns a dense matrix. And
that’s it! We have a preprocessing pipeline that takes the full housing data and applies
the appropriate transformations to each column.74 | Chapter 2: End-to-End Machine Learning Project

Instead of a transformer, you can specify the string "drop"  if you
want the columns to be dropped. Or you can specify "pass
through"  if you want the columns to be left untouched. By default,
the remaining columns (i.e., the ones that were not listed) will be
dropped, but you can set the remainder  hyperparameter to any
transformer (or to "passthrough" ) if you want these columns to be
handled differently.
If you are using Scikit-Learn 0.19 or earlier, you can use a third-party library such as
sklearn-pandas , or roll out your own custom transformer to get the same function‐
ality as the ColumnTransformer . Alternatively, you can use the FeatureUnion  class
which can also apply different transformers and concatenate their outputs, but you
cannot specify different columns for each transformer, they all apply to the whole
data. It is possible to work around this limitation using a custom transformer for col‐
umn selection (see the Jupyter notebook for an example).
Select and Train a Model

At last! Y ou framed the problem, you got the data and explored it, you sampled a
training set and a test set, and you wrote transformation pipelines to clean up and
prepare your data for Machine Learning algorithms automatically. Y ou are now ready
to select and train a Machine Learning model.
Training and Evaluating on the Training Set
The good news is that thanks to all these previous steps, things are now going to be
much simpler than you might think. Let’s first train a Linear Regression model, like

we did in the previous chapter:
from sklearn.linear_model  import LinearRegression
lin_reg = LinearRegression ()
lin_reg.fit(housing_prepared , housing_labels )
Done! Y ou now have a working Linear Regression model. Let’s try it out on a few
instances from the training set:
>>> some_data  = housing.iloc[:5]
>>> some_labels  = housing_labels .iloc[:5]
>>> some_data_prepared  = full_pipeline .transform (some_data )
>>> print("Predictions:" , lin_reg.predict(some_data_prepared ))

Predictions: [ 210644.6045  317768.8069  210956.4333  59218.9888  189747.5584]
>>> print("Labels:" , list(some_labels ))
Labels: [286600.0, 340600.0, 196900.0, 46300.0, 254500.0]Select and Train a Model | 75

It works, although the predictions are not exactly accurate (e.g., the first prediction is
off by close to 40%!). Let’s measure this regression model’s RMSE on the whole train‐
ing set using Scikit-Learn’s mean_squared_error  function:
>>> from sklearn.metrics  import mean_squared_error
>>> housing_predictions  = lin_reg.predict(housing_prepared )
>>> lin_mse = mean_squared_error (housing_labels , housing_predictions )
>>> lin_rmse  = np.sqrt(lin_mse)
>>> lin_rmse
68628.19819848922
Okay, this is better than nothing but clearly not a great score: most districts’
median_housing_values  range between $120,000 and $265,000, so a typical predic‐
tion error of $68,628 is not very satisfying. This is an example of a model underfitting
the training data. When this happens it can mean that the features do not provide
enough information to make good predictions, or that the model is not powerful
enough. As we saw in the previous chapter, the main ways to fix underfitting are to

select a more powerful model, to feed the training algorithm with better features, or
to reduce the constraints on the model. This model is not regularized, so this rules
out the last option. Y ou could try to add more features (e.g., the log of the popula‐
tion), but first let’s try a more complex model to see how it does.
Let’s train a DecisionTreeRegressor . This is a powerful model, capable of finding
complex nonlinear relationships in the data (Decision Trees are presented in more

detail in Chapter 6 ). The code should look familiar by now:
from sklearn.tree  import DecisionTreeRegressor
tree_reg  = DecisionTreeRegressor ()
tree_reg .fit(housing_prepared , housing_labels )
Now that the model is trained, let’s evaluate it on the training set:
>>> housing_predictions  = tree_reg .predict(housing_prepared )
>>> tree_mse  = mean_squared_error (housing_labels , housing_predictions )
>>> tree_rmse  = np.sqrt(tree_mse )
>>> tree_rmse
0.0

0.0
Wait, what!? No error at all? Could this model really be absolutely perfect? Of course,
it is much more likely that the model has badly overfit the data. How can you be sure?
As we saw earlier, you don’t want to touch the test set until you are ready to launch a
model you are confident about, so you need to use part of the training set for train‐
ing, and part for model validation.
Better Evaluation Using Cross-Validation
One way to evaluate the Decision Tree model would be to use the train_test_split

0.0
function to split the training set into a smaller training set and a validation set, then76 | Chapter 2: End-to-End Machine Learning Project

train your models against the smaller training set and evaluate them against the vali‐
dation set. It’s a bit of work, but nothing too difficult and it would work fairly well.
A great alternative is to use Scikit-Learn’s K-fold cross-validation  feature. The follow‐
ing code randomly splits the training set into 10 distinct subsets called folds , then it
trains and evaluates the Decision Tree model 10 times, picking a different fold for
evaluation every time and training on the other 9 folds. The result is an array con‐
taining the 10 evaluation scores:
from sklearn.model_selection  import cross_val_score
scores = cross_val_score (tree_reg , housing_prepared , housing_labels ,
                         scoring="neg_mean_squared_error" , cv=10)
tree_rmse_scores  = np.sqrt(-scores)
Scikit-Learn’s cross-validation features expect a utility function
(greater is better) rather than a cost function (lower is better), so
the scoring function is actually the opposite of the MSE (i.e., a neg‐

ative value), which is why the preceding code computes -scores
before calculating the square root.
Let’s look at the results:
>>> def display_scores (scores):
...     print("Scores:" , scores)
...     print("Mean:", scores.mean())
...     print("Standard deviation:" , scores.std())
...
>>> display_scores (tree_rmse_scores )
Scores: [70194.33680785 66855.16363941 72432.58244769 70758.73896782
 71115.88230639 75585.14172901 70262.86139133 70273.6325285
 75366.87952553 71231.65726027]
Mean: 71407.68766037929
Standard deviation: 2439.4345041191004

...
Now the Decision Tree doesn’t look as good as it did earlier. In fact, it seems to per‐
form worse than the Linear Regression model! Notice that cross-validation allows
you to get not only an estimate of the performance of your model, but also a measure
of how precise this estimate is (i.e., its standard deviation). The Decision Tree has a
score of approximately 71,407, generally ±2,439. Y ou would not have this information
if you just used one validation set. But cross-validation comes at the cost of training

the model several times, so it is not always possible.
Let’s compute the same scores for the Linear Regression model just to be sure:
>>> lin_scores  = cross_val_score (lin_reg, housing_prepared , housing_labels ,
...                              scoring="neg_mean_squared_error" , cv=10)
...
>>> lin_rmse_scores  = np.sqrt(-lin_scores )
>>> display_scores (lin_rmse_scores )Select and Train a Model | 77

Scores: [66782.73843989 66960.118071   70347.95244419 74739.57052552
 68031.13388938 71193.84183426 64969.63056405 68281.61137997
 71552.91566558 67665.10082067]
Mean: 69052.46136345083
Standard deviation: 2731.674001798348
That’s right: the Decision Tree model is overfitting so badly that it performs worse
than the Linear Regression model.
Let’s try one last model now: the RandomForestRegressor . As we will see in Chap‐
ter 7 , Random Forests work by training many Decision Trees on random subsets of
the features, then averaging out their predictions. Building a model on top of many
other models is called Ensemble Learning , and it is often a great way to push ML algo‐
rithms even further. We will skip most of the code since it is essentially the same as
for the other models:
>>> from sklearn.ensemble  import RandomForestRegressor
>>> forest_reg  = RandomForestRegressor ()
>>> forest_reg .fit(housing_prepared , housing_labels )
>>> [...]
>>> forest_rmse
18603.515021376355

>>> [...]
>>> display_scores (forest_rmse_scores )
Scores: [49519.80364233 47461.9115823  50029.02762854 52325.28068953
 49308.39426421 53446.37892622 48634.8036574  47585.73832311
 53490.10699751 50021.5852922 ]
Mean: 50182.303100336096
Standard deviation: 2097.0810550985693
Wow, this is much better: Random Forests look very promising. However, note that
the score on the training set is still much lower than on the validation sets, meaning
that the model is still overfitting the training set. Possible solutions for overfitting are

to simplify the model, constrain it (i.e., regularize it), or get a lot more training data.
However, before you dive much deeper in Random Forests, you should try out many
other models from various categories of Machine Learning algorithms (several Sup‐
port Vector Machines with different kernels, possibly a neural network, etc.), without
spending too much time tweaking the hyperparameters. The goal is to shortlist a few
(two to five) promising models.78 | Chapter 2: End-to-End Machine Learning Project

Y ou should save every model you experiment with, so you can
come back easily to any model you want. Make sure you save both
the hyperparameters and the trained parameters, as well as the
cross-validation scores and perhaps the actual predictions as well.
This will allow you to easily compare scores across model types,
and compare the types of errors they make. Y ou can easily save
Scikit-Learn models by using Python’s pickle  module, or using
sklearn.externals.joblib , which is more efficient at serializing 
large NumPy arrays:
from sklearn.externals  import joblib
joblib.dump(my_model , "my_model.pkl" )
# and later...
my_model_loaded  = joblib.load("my_model.pkl" )
Fine-Tune Your Model
Let’s assume that you now have a shortlist of promising models. Y ou now need to
fine-tune them. Let’s look at a few ways you can do that.
Grid Search
One way to do that would be to fiddle with the hyperparameters manually, until you
find a great combination of hyperparameter values. This would be very tedious work,

and you may not have time to explore many combinations.
Instead you should get Scikit-Learn’s GridSearchCV  to search for you. All you need to
do is tell it which hyperparameters you want it to experiment with, and what values to
try out, and it will evaluate all the possible combinations of hyperparameter values,
using cross-validation. For example, the following code searches for the best combi‐
nation of hyperparameter values for the RandomForestRegressor :
from sklearn.model_selection  import GridSearchCV

param_grid  = [
    {'n_estimators' : [3, 10, 30], 'max_features' : [2, 4, 6, 8]},
    {'bootstrap' : [False], 'n_estimators' : [3, 10], 'max_features' : [2, 3, 4]},
  ]
forest_reg  = RandomForestRegressor ()
grid_search  = GridSearchCV (forest_reg , param_grid , cv=5,
                           scoring='neg_mean_squared_error' ,
                           return_train_score =True)
grid_search .fit(housing_prepared , housing_labels )Fine-Tune Your Model | 79

When you have no idea what value a hyperparameter should have,
a simple approach is to try out consecutive powers of 10 (or a
smaller number if you want a more fine-grained search, as shown
in this example with the n_estimators  hyperparameter).
This param_grid  tells Scikit-Learn to first evaluate all 3 × 4 = 12 combinations of
n_estimators  and max_features  hyperparameter values specified in the first dict
(don’t worry about what these hyperparameters mean for now; they will be explained
in Chapter 7 ), then try all 2 × 3 = 6 combinations of hyperparameter values in the
second dict , but this time with the bootstrap  hyperparameter set to False  instead of
True  (which is the default value for this hyperparameter).
All in all, the grid search will explore 12 + 6 = 18 combinations of RandomForestRe
gressor  hyperparameter values, and it will train each model five times (since we are
using five-fold cross validation). In other words, all in all, there will be 18 × 5 = 90

rounds of training! It may take quite a long time, but when it is done you can get the
best combination of parameters like this:
>>> grid_search .best_params_
{'max_features': 8, 'n_estimators': 30}
Since 8 and 30 are the maximum values that were evaluated, you
should probably try searching again with higher values, since the
score may continue to improve.
Y ou can also get the best estimator directly:
>>> grid_search .best_estimator_
RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=None,

           max_features=8, max_leaf_nodes=None, min_impurity_decrease=0.0,
           min_impurity_split=None, min_samples_leaf=1,
           min_samples_split=2, min_weight_fraction_leaf=0.0,
           n_estimators=30, n_jobs=None, oob_score=False, random_state=None,
           verbose=0, warm_start=False)
If GridSearchCV  is initialized with refit=True  (which is the
default), then once it finds the best estimator using cross-
validation, it retrains it on the whole training set. This is usually a

max_features=8, max_leaf_nodes=None, min_impurity_decrease=0.0,
good idea since feeding it more data will likely improve its perfor‐
mance.
And of course the evaluation scores are also available:
>>> cvres = grid_search .cv_results_
>>> for mean_score , params in zip(cvres["mean_test_score" ], cvres["params" ]):80 | Chapter 2: End-to-End Machine Learning Project

...     print(np.sqrt(-mean_score ), params)
...
63669.05791727153 {'max_features': 2, 'n_estimators': 3}
55627.16171305252 {'max_features': 2, 'n_estimators': 10}
53384.57867637289 {'max_features': 2, 'n_estimators': 30}
60965.99185930139 {'max_features': 4, 'n_estimators': 3}
52740.98248528835 {'max_features': 4, 'n_estimators': 10}
50377.344409590376 {'max_features': 4, 'n_estimators': 30}
58663.84733372485 {'max_features': 6, 'n_estimators': 3}
52006.15355973719 {'max_features': 6, 'n_estimators': 10}
50146.465964159885 {'max_features': 6, 'n_estimators': 30}
57869.25504027614 {'max_features': 8, 'n_estimators': 3}
51711.09443660957 {'max_features': 8, 'n_estimators': 10}
49682.25345942335 {'max_features': 8, 'n_estimators': 30}
62895.088889905004 {'bootstrap': False, 'max_features': 2, 'n_estimators': 3}
54658.14484390074 {'bootstrap': False, 'max_features': 2, 'n_estimators': 10}
59470.399594730654 {'bootstrap': False, 'max_features': 3, 'n_estimators': 3}

52725.01091081235 {'bootstrap': False, 'max_features': 3, 'n_estimators': 10}
57490.612956065226 {'bootstrap': False, 'max_features': 4, 'n_estimators': 3}
51009.51445842374 {'bootstrap': False, 'max_features': 4, 'n_estimators': 10}
In this example, we obtain the best solution by setting the max_features  hyperpara‐
meter to 8, and the n_estimators  hyperparameter to 30. The RMSE score for this
combination is 49,682, which is slightly better than the score you got earlier using the

default hyperparameter values (which was 50,182). Congratulations, you have suc‐
cessfully fine-tuned your best model!
Don’t forget that you can treat some of the data preparation steps as
hyperparameters. For example, the grid search will automatically
find out whether or not to add a feature you were not sure about
(e.g., using the add_bedrooms_per_room  hyperparameter of your
CombinedAttributesAdder  transformer). It may similarly be used
to automatically find the best way to handle outliers, missing fea‐

tures, feature selection, and more.
Randomized Search
The grid search approach is fine when you are exploring relatively few combinations,
like in the previous example, but when the hyperparameter search space  is large, it is
often preferable to use RandomizedSearchCV  instead. This class can be used in much
the same way as the GridSearchCV  class, but instead of trying out all possible combi‐
nations, it evaluates a given number of random combinations by selecting a random
value for each hyperparameter at every iteration. This approach has two main bene‐

fits:Fine-Tune Your Model | 81

•If you let the randomized search run for, say, 1,000 iterations, this approach will
explore 1,000 different values for each hyperparameter (instead of just a few val‐
ues per hyperparameter with the grid search approach).•
Y ou have more control over the computing budget you want to allocate to hyper‐
parameter search, simply by setting the number of iterations.
Ensemble Methods
Another way to fine-tune your system is to try to combine the models that perform
best. The group (or “ensemble”) will often perform better than the best individual
model (just like Random Forests perform better than the individual Decision Trees
they rely on), especially if the individual models make very different types of errors.
We will cover this topic in more detail in Chapter 7 .
Analyze the Best Models and Their Errors
Y ou will often gain good insights on the problem by inspecting the best models. For
example, the RandomForestRegressor  can indicate the relative importance of each
attribute for making accurate predictions:

>>> feature_importances  = grid_search .best_estimator_ .feature_importances_
>>> feature_importances
array([7.33442355e-02, 6.29090705e-02, 4.11437985e-02, 1.46726854e-02,
       1.41064835e-02, 1.48742809e-02, 1.42575993e-02, 3.66158981e-01,
       5.64191792e-02, 1.08792957e-01, 5.33510773e-02, 1.03114883e-02,
       1.64780994e-01, 6.02803867e-05, 1.96041560e-03, 2.85647464e-03])
Let’s display these importance scores next to their corresponding attribute names:

>>> extra_attribs  = ["rooms_per_hhold" , "pop_per_hhold" , "bedrooms_per_room" ]
>>> cat_encoder  = full_pipeline .named_transformers_ ["cat"]
>>> cat_one_hot_attribs  = list(cat_encoder .categories_ [0])
>>> attributes  = num_attribs  + extra_attribs  + cat_one_hot_attribs
>>> sorted(zip(feature_importances , attributes ), reverse=True)
[(0.3661589806181342, 'median_income'),
 (0.1647809935615905, 'INLAND'),
 (0.10879295677551573, 'pop_per_hhold'),
 (0.07334423551601242, 'longitude'),
 (0.0629090704826203, 'latitude'),

 (0.05641917918195401, 'rooms_per_hhold'),
 (0.05335107734767581, 'bedrooms_per_room'),
 (0.041143798478729635, 'housing_median_age'),
 (0.014874280890402767, 'population'),
 (0.014672685420543237, 'total_rooms'),
 (0.014257599323407807, 'households'),
 (0.014106483453584102, 'total_bedrooms'),
 (0.010311488326303787, '<1H OCEAN'),
 (0.002856474637320158, 'NEAR OCEAN'),82 | Chapter 2: End-to-End Machine Learning Project

(0.00196041559947807, 'NEAR BAY'),
 (6.028038672736599e-05, 'ISLAND')]
With this information, you may want to try dropping some of the less useful features
(e.g., apparently only one ocean_proximity  category is really useful, so you could try
dropping the others).
Y ou should also look at the specific errors that your system makes, then try to under‐
stand why it makes them and what could fix the problem (adding extra features or, on
the contrary, getting rid of uninformative ones, cleaning up outliers, etc.).
Evaluate Your System on the Test Set
After tweaking your models for a while, you eventually have a system that performs
sufficiently well. Now is the time to evaluate the final model on the test set. There is
nothing special about this process; just get the predictors and the labels from your
test set, run your full_pipeline  to transform the data (call transform() , not
fit_transform() , you do not want to fit the test set!), and evaluate the final model
on the test set:

final_model  = grid_search .best_estimator_
X_test = strat_test_set .drop("median_house_value" , axis=1)
y_test = strat_test_set ["median_house_value" ].copy()
X_test_prepared  = full_pipeline .transform (X_test)
final_predictions  = final_model .predict(X_test_prepared )
final_mse  = mean_squared_error (y_test, final_predictions )
final_rmse  = np.sqrt(final_mse )   # => evaluates to 47,730.2
In some cases, such a point estimate of the generalization error will not be quite

enough to convince you to launch: what if it is just 0.1% better than the model cur‐
rently in production? Y ou might want to have an idea of how precise this estimate is.
For this, you can compute a 95% confidence  interval  for the generalization error using
scipy.stats.t.interval() :
>>> from scipy import stats
>>> confidence  = 0.95
>>> squared_errors  = (final_predictions  - y_test) ** 2
>>> np.sqrt(stats.t.interval (confidence , len(squared_errors ) - 1,
...                          loc=squared_errors .mean(),

...                          scale=stats.sem(squared_errors )))
...
array([45685.10470776, 49691.25001878])
The performance will usually be slightly worse than what you measured using cross-
validation if you did a lot of hyperparameter tuning (because your system ends up
fine-tuned to perform well on the validation data, and will likely not perform as wellFine-Tune Your Model | 83

on unknown datasets). It is not the case in this example, but when this happens you
must resist the temptation to tweak the hyperparameters to make the numbers look
good on the test set; the improvements would be unlikely to generalize to new data.
Now comes the project prelaunch phase: you need to present your solution (high‐
lighting what you have learned, what worked and what did not, what assumptions
were made, and what your system’s limitations are), document everything, and create
nice presentations with clear visualizations and easy-to-remember statements (e.g.,
“the median income is the number one predictor of housing prices”). In this Califor‐
nia housing example, the final performance of the system is not better than the
experts’ , but it may still be a good idea to launch it, especially if this frees up some
time for the experts so they can work on more interesting and productive tasks.
Launch, Monitor, and Maintain Your System

Perfect, you got approval to launch! Y ou need to get your solution ready for produc‐
tion, in particular by plugging the production input data sources into your system
and writing tests.
Y ou also need to write monitoring code to check your system’s live performance at
regular intervals and trigger alerts when it drops. This is important to catch not only
sudden breakage, but also performance degradation. This is quite common because
models tend to “rot” as data evolves over time, unless the models are regularly trained
on fresh data.

Evaluating your system’s performance will require sampling the system’s predictions
and evaluating them. This will generally require a human analysis. These analysts
may be field experts, or workers on a crowdsourcing platform (such as Amazon
Mechanical Turk or CrowdFlower). Either way, you need to plug the human evalua‐
tion pipeline into your system.
Y ou should also make sure you evaluate the system’s input data quality. Sometimes
performance will degrade slightly because of a poor quality signal (e.g., a malfunc‐

tioning sensor sending random values, or another team’s output becoming stale), but
it may take a while before your system’s performance degrades enough to trigger an
alert. If you monitor your system’s inputs, you may catch this earlier. Monitoring the
inputs is particularly important for online learning systems.
Finally, you will generally want to train your models on a regular basis using fresh
data. Y ou should automate this process as much as possible. If you don’t, you are very
likely to refresh your model only every six months (at best), and your system’s perfor‐

mance may fluctuate severely over time. If your system is an online learning system,
you should make sure you save snapshots of its state at regular intervals so you can
easily roll back to a previously working state.84 | Chapter 2: End-to-End Machine Learning Project

Try It Out!
Hopefully this chapter gave you a good idea of what a Machine Learning project
looks like, and showed you some of the tools you can use to train a great system. As
you can see, much of the work is in the data preparation step, building monitoring
tools, setting up human evaluation pipelines, and automating regular model training.
The Machine Learning algorithms are also important, of course, but it is probably
preferable to be comfortable with the overall process and know three or four algo‐
rithms well rather than to spend all your time exploring advanced algorithms and not
enough time on the overall process.
So, if you have not already done so, now is a good time to pick up a laptop, select a
dataset that you are interested in, and try to go through the whole process from A to
Z. A good place to start is on a competition website such as http://kaggle.com/ : you
will have a dataset to play with, a clear goal, and people to share the experience with.
Exercises

Exercises
Using this chapter’s housing dataset:1.
Try a Support Vector Machine regressor ( sklearn.svm.SVR ), with various hyper‐
parameters such as kernel="linear"  (with various values for the C hyperpara‐
meter) or kernel="rbf"  (with various values for the C and gamma
hyperparameters). Don’t worry about what these hyperparameters mean for now.
How does the best SVR predictor perform?2.
Try replacing GridSearchCV  with RandomizedSearchCV .3.
Try adding a transformer in the preparation pipeline to select only the most
important attributes.4.

Try creating a single pipeline that does the full data preparation plus the final
prediction.5.
Automatically explore some preparation options using GridSearchCV .
Solutions to these exercises are available in the online Jupyter notebooks at https://
github.com/ageron/handson-ml2 .Try It Out! | 85

1By default Scikit-Learn caches downloaded datasets in a directory called $HOME/scikit_learn_data .
CHAPTER 3
Classification
release of these titles. The following will be Chapter 3 in the final
In Chapter 1  we mentioned that the most common supervised learning tasks are
regression (predicting values) and classification (predicting classes). In Chapter 2  we
explored a regression task, predicting housing values, using various algorithms such
as Linear Regression, Decision Trees, and Random Forests (which will be explained
in further detail in later chapters). Now we will turn our attention to classification
systems.
MNIST
In this chapter, we will be using the MNIST dataset, which is a set of 70,000 small
images of digits handwritten by high school students and employees of the US Cen‐

systems.
MNIST
sus Bureau. Each image is labeled with the digit it represents. This set has been stud‐
ied so much that it is often called the “Hello World” of Machine Learning: whenever
people come up with a new classification algorithm, they are curious to see how it
will perform on MNIST. Whenever someone learns Machine Learning, sooner or
later they tackle MNIST.
Scikit-Learn provides many helper functions to download popular datasets. MNIST is
one of them. The following code fetches the MNIST dataset:187

>>> from sklearn.datasets  import fetch_openml
>>> mnist = fetch_openml ('mnist_784' , version=1)
>>> mnist.keys()
dict_keys(['data', 'target', 'feature_names', 'DESCR', 'details',
           'categories', 'url'])
Datasets loaded by Scikit-Learn generally have a similar dictionary structure includ‐
ing:•
A DESCR  key describing the dataset•
A data  key containing an array with one row per instance and one column per
feature•
A target  key containing an array with the labels
Let’s look at these arrays:
>>> X, y = mnist["data"], mnist["target" ]
>>> X.shape
(70000, 784)
>>> y.shape
(70000,)
There are 70,000 images, and each image has 784 features. This is because each image
is 28×28 pixels, and each feature simply represents one pixel’s intensity, from 0
(white) to 255 (black). Let’s take a peek at one digit from the dataset. All you need to
do is grab an instance’s feature vector, reshape it to a 28×28 array, and display it using
Matplotlib’s imshow()  function:
import matplotlib  as mpl

(70000,)
some_digit  = X[0]
some_digit_image  = some_digit .reshape(28, 28)
plt.imshow(some_digit_image , cmap = mpl.cm.binary, interpolation ="nearest" )
plt.axis("off")
This looks like a 5, and indeed that’s what the label tells us:88 | Chapter 3: Classification

>>> y[0]
'5'
Note that the label is a string. We prefer numbers, so let’s cast y to integers:
>>> y = y.astype(np.uint8)
Figure 3-1  shows a few more images from the MNIST dataset to give you a feel for
the complexity of the classification task.
Figure 3-1. A few digits from the MNIST dataset
But wait! Y ou should always create a test set and set it aside before inspecting the data
closely. The MNIST dataset is actually already split into a training set (the first 60,000
images) and a test set (the last 10,000 images):
X_train, X_test, y_train, y_test = X[:60000], X[60000:], y[:60000], y[60000:]
The training set is already shuffled for us, which is good as this guarantees that all
cross-validation folds will be similar (you don’t want one fold to be missing some dig‐
its). Moreover, some learning algorithms are sensitive to the order of the trainingMNIST | 89

2Shuffling may be a bad idea in some contexts—for example, if you are working on time series data (such as
stock market prices or weather conditions). We will explore this in the next chapters.
instances, and they perform poorly if they get many similar instances in a row. Shuf‐
fling the dataset ensures that this won’t happen.2
Training a Binary Classifier
Let’s simplify the problem for now and only try to identify one digit—for example,
the number 5. This “5-detector” will be an example of a binary classifier , capable of
distinguishing between just two classes, 5 and not-5. Let’s create the target vectors for
this classification task:
y_train_5  = (y_train == 5)  # True for all 5s, False for all other digits.
y_test_5  = (y_test == 5)
Okay, now let’s pick a classifier and train it. A good place to start is with a Stochastic
Gradient Descent  (SGD) classifier, using Scikit-Learn’s SGDClassifier  class. This clas‐
sifier has the advantage of being capable of handling very large datasets efficiently.

This is in part because SGD deals with training instances independently, one at a time
(which also makes SGD well suited for online learning ), as we will see later. Let’s create
an SGDClassifier  and train it on the whole training set:
from sklearn.linear_model  import SGDClassifier
sgd_clf = SGDClassifier (random_state =42)
sgd_clf.fit(X_train, y_train_5 )
The SGDClassifier  relies on randomness during training (hence
the name “stochastic”). If you want reproducible results, you
should set the random_state  parameter.

Now you can use it to detect images of the number 5:
>>> sgd_clf.predict([some_digit ])
array([ True])
The classifier guesses that this image represents a 5 ( True ). Looks like it guessed right
in this particular case! Now, let’s evaluate this model’s performance.
Performance Measures
Evaluating a classifier is often significantly trickier than evaluating a regressor, so we
will spend a large part of this chapter on this topic. There are many performance90 | Chapter 3: Classification

measures available, so grab another coffee and get ready to learn many new concepts
and acronyms!
Measuring Accuracy Using Cross-Validation
A good way to evaluate a model is to use cross-validation, just as you did in Chap‐
ter 2 .
Implementing Cross-Validation
Occasionally you will need more control over the cross-validation process than what
Scikit-Learn provides off-the-shelf. In these cases, you can implement cross-
validation yourself; it is actually fairly straightforward. The following code does
roughly the same thing as Scikit-Learn’s cross_val_score()  function, and prints the 
same result:
from sklearn.model_selection  import StratifiedKFold
from sklearn.base  import clone
skfolds = StratifiedKFold (n_splits =3, random_state =42)
for train_index , test_index  in skfolds.split(X_train, y_train_5 ):
    clone_clf  = clone(sgd_clf)
    X_train_folds  = X_train[train_index ]
    y_train_folds  = y_train_5 [train_index ]
    X_test_fold  = X_train[test_index ]
    y_test_fold  = y_train_5 [test_index ]

    clone_clf .fit(X_train_folds , y_train_folds )
    y_pred = clone_clf .predict(X_test_fold )
    n_correct  = sum(y_pred == y_test_fold )
    print(n_correct  / len(y_pred))  # prints 0.9502, 0.96565 and 0.96495
The StratifiedKFold  class performs stratified sampling (as explained in Chapter 2 )
to produce folds that contain a representative ratio of each class. At each iteration the
code creates a clone of the classifier, trains that clone on the training folds, and makes
predictions on the test fold. Then it counts the number of correct predictions and

n_correct  = sum(y_pred == y_test_fold )
outputs the ratio of correct predictions.
Let’s use the cross_val_score()  function to evaluate your SGDClassifier  model
using K-fold cross-validation, with three folds. Remember that K-fold cross-
validation means splitting the training set into K-folds (in this case, three), then mak‐
ing predictions and evaluating them on each fold using a model trained on the
remaining folds (see Chapter 2 ):Performance Measures | 91

>>> from sklearn.model_selection  import cross_val_score
>>> cross_val_score (sgd_clf, X_train, y_train_5 , cv=3, scoring="accuracy" )
array([0.96355, 0.93795, 0.95615])
Wow! Above 93% accuracy  (ratio of correct predictions) on all cross-validation folds? 
This looks amazing, doesn’t it? Well, before you get too excited, let’s look at a very
dumb classifier that just classifies every single image in the “not-5” class:
from sklearn.base  import BaseEstimator
class Never5Classifier (BaseEstimator ):
        pass
    def predict(self, X):
        return np.zeros((len(X), 1), dtype=bool)
Can you guess this model’s accuracy? Let’s find out:
>>> never_5_clf  = Never5Classifier ()
>>> cross_val_score (never_5_clf , X_train, y_train_5 , cv=3, scoring="accuracy" )
array([0.91125, 0.90855, 0.90915])
That’s right, it has over 90% accuracy! This is simply because only about 10% of the
images are 5s, so if you always guess that an image is not a 5, you will be right about

        pass
90% of the time. Beats Nostradamus.
This demonstrates why accuracy is generally not the preferred performance measure
for classifiers, especially when you are dealing with skewed datasets  (i.e., when some
classes are much more frequent than others).
Confusion Matrix
A much better way to evaluate the performance of a classifier is to look at the confu‐
sion matrix . The general idea is to count the number of times instances of class A are

classified as class B. For example, to know the number of times the classifier confused
images of 5s with 3s, you would look in the 5th row and 3rd column of the confusion
matrix.
To compute the confusion matrix, you first need to have a set of predictions, so they
can be compared to the actual targets. Y ou could make predictions on the test set, but
let’s keep it untouched for now (remember that you want to use the test set only at the
very end of your project, once you have a classifier that you are ready to launch).

matrix.
Instead, you can use the cross_val_predict()  function:
from sklearn.model_selection  import cross_val_predict
y_train_pred  = cross_val_predict (sgd_clf, X_train, y_train_5 , cv=3)
Just like the cross_val_score()  function, cross_val_predict()  performs K-fold
cross-validation, but instead of returning the evaluation scores, it returns the predic‐92 | Chapter 3: Classification

tions made on each test fold. This means that you get a clean prediction for each
instance in the training set (“clean” meaning that the prediction is made by a model
that never saw the data during training).
Now you are ready to get the confusion matrix using the confusion_matrix()  func‐
tion. Just pass it the target classes ( y_train_5 ) and the predicted classes
(y_train_pred ):
>>> from sklearn.metrics  import confusion_matrix
>>> confusion_matrix (y_train_5 , y_train_pred )
array([[53057,  1522],
       [ 1325,  4096]])
Each row in a confusion matrix represents an actual class , while each column repre‐
sents a predicted class . The first row of this matrix considers non-5 images (the nega‐
tive class ): 53,057 of them were correctly classified as non-5s (they are called true
negatives ), while the remaining 1,522 were wrongly classified as 5s ( false positives ).
The second row considers the images of 5s (the positive class ): 1,325 were wrongly

classified as non-5s ( false negatives ), while the remaining 4,096 were correctly classi‐
fied as 5s ( true positives ). A perfect classifier would have only true positives and true
negatives, so its confusion matrix would have nonzero values only on its main diago‐
nal (top left to bottom right):
>>> y_train_perfect_predictions  = y_train_5   # pretend we reached perfection
>>> confusion_matrix (y_train_5 , y_train_perfect_predictions )
array([[54579,     0],
       [    0,  5421]])

The confusion matrix gives you a lot of information, but sometimes you may prefer a
more concise metric. An interesting one to look at is the accuracy of the positive pre‐
dictions; this is called the precision  of the classifier ( Equation 3-1 ).
Equation 3-1. Precisionprecision =TP
TP+FP
TP is the number of true positives, and FP is the number of false positives.
A trivial way to have perfect precision is to make one single positive prediction and

TP+FP
ensure it is correct (precision = 1/1 = 100%). This would not be very useful since the
classifier would ignore all but one positive instance. So precision is typically used
along with another metric named recall , also called sensitivity  or true positive ratePerformance Measures | 93

(TPR ): this is the ratio of positive instances that are correctly detected by the classifier
(Equation 3-2 ).
Equation 3-2. Recallrecall =TP
TP+FN
FN is of course the number of false negatives.
If you are confused about the confusion matrix, Figure 3-2  may help.
Figure 3-2. An illustrated confusion matrix
Precision and Recall
Scikit-Learn provides several functions to compute classifier metrics, including preci‐
sion and recall:
>>> from sklearn.metrics  import precision_score , recall_score
>>> precision_score (y_train_5 , y_train_pred ) # == 4096 / (4096 + 1522)
0.7290850836596654
>>> recall_score (y_train_5 , y_train_pred ) # == 4096 / (4096 + 1325)
0.7555801512636044
Now your 5-detector does not look as shiny as it did when you looked at its accuracy.
When it claims an image represents a 5, it is correct only 72.9% of the time. More‐
over, it only detects 75.6% of the 5s.
It is often convenient to combine precision and recall into a single metric called the F1

score , in particular if you need a simple way to compare two classifiers. The F1 score is 
the harmonic mean  of precision and recall ( Equation 3-3 ). Whereas the regular mean94 | Chapter 3: Classification

treats all values equally, the harmonic mean gives much more weight to low values.
As a result, the classifier will only get a high F1 score if both recall and precision are
high.
Equation 3-3. F1F1=2
1
precision+1
recall= 2 ×precision × recall
precision + recall=TP
TP+FN+FP
2
To compute the F1 score, simply call the f1_score()  function:
>>> from sklearn.metrics  import f1_score
>>> f1_score (y_train_5 , y_train_pred )
0.7420962043663375
The F1 score favors classifiers that have similar precision and recall. This is not always
what you want: in some contexts you mostly care about precision, and in other con‐
texts you really care about recall. For example, if you trained a classifier to detect vid‐
eos that are safe for kids, you would probably prefer a classifier that rejects many
good videos (low recall) but keeps only safe ones (high precision), rather than a clas‐
sifier that has a much higher recall but lets a few really bad videos show up in your

product (in such cases, you may even want to add a human pipeline to check the clas‐
sifier’s video selection). On the other hand, suppose you train a classifier to detect
shoplifters on surveillance images: it is probably fine if your classifier has only 30%
precision as long as it has 99% recall (sure, the security guards will get a few false
alerts, but almost all shoplifters will get caught).
Unfortunately, you can’t have it both ways: increasing precision reduces recall, and
vice versa. This is called the precision/recall tradeoff .
Precision/Recall Tradeoff

To understand this tradeoff, let’s look at how the SGDClassifier  makes its classifica‐
tion decisions. For each instance, it computes a score based on a decision function , 
and if that score is greater than a threshold, it assigns the instance to the positive
class, or else it assigns it to the negative class. Figure 3-3  shows a few digits positioned
from the lowest score on the left to the highest score on the right. Suppose the deci‐
sion threshold  is positioned at the central arrow (between the two 5s): you will find 4

true positives (actual 5s) on the right of that threshold, and one false positive (actually
a 6). Therefore, with that threshold, the precision is 80% (4 out of 5). But out of 6
actual 5s, the classifier only detects 4, so the recall is 67% (4 out of 6). Now if you
raise the threshold (move it to the arrow on the right), the false positive (the 6)
becomes a true negative, thereby increasing precision (up to 100% in this case), but
one true positive becomes a false negative, decreasing recall down to 50%. Conversely,

lowering the threshold increases recall and reduces precision.Performance Measures | 95

Figure 3-3. Decision threshold and precision/recall tradeoff
Scikit-Learn does not let you set the threshold directly, but it does give you access to
the decision scores that it uses to make predictions. Instead of calling the classifier’s
predict()  method, you can call its decision_function()  method, which returns a
score for each instance, and then make predictions based on those scores using any
threshold you want:
>>> y_scores  = sgd_clf.decision_function ([some_digit ])
>>> y_scores
array([2412.53175101])
>>> threshold  = 0
>>> y_some_digit_pred  = (y_scores  > threshold )
The SGDClassifier  uses a threshold equal to 0, so the previous code returns the same
result as the predict()  method (i.e., True ). Let’s raise the threshold:
>>> threshold  = 8000
>>> y_some_digit_pred
array([False])
This confirms that raising the threshold decreases recall. The image actually repre‐

sents a 5, and the classifier detects it when the threshold is 0, but it misses it when the
threshold is increased to 8,000.
Now how do you decide which threshold to use? For this you will first need to get the
scores of all instances in the training set using the cross_val_predict()  function
again, but this time specifying that you want it to return decision scores instead of
predictions:
y_scores  = cross_val_predict (sgd_clf, X_train, y_train_5 , cv=3,
                             method="decision_function" )

Now with these scores you can compute precision and recall for all possible thresh‐
olds using the precision_recall_curve()  function:96 | Chapter 3: Classification

from sklearn.metrics  import precision_recall_curve
precisions , recalls, thresholds  = precision_recall_curve (y_train_5 , y_scores )
Finally, you can plot precision and recall as functions of the threshold value using
Matplotlib ( Figure 3-4 ):
def plot_precision_recall_vs_threshold (precisions , recalls, thresholds ):
    plt.plot(thresholds , precisions [:-1], "b--", label="Precision" )
    plt.plot(thresholds , recalls[:-1], "g-", label="Recall" )
    [...] # highlight the threshold, add the legend, axis label and grid
plot_precision_recall_vs_threshold (precisions , recalls, thresholds )
Figure 3-4. Precision and recall versus the decision threshold
Y ou may wonder why the precision curve is bumpier than the recall
curve in Figure 3-4 . The reason is that precision may sometimes go
down when you raise the threshold (although in general it will go
up). To understand why, look back at Figure 3-3  and notice what
happens when you start from the central threshold and move it just

one digit to the right: precision goes from 4/5 (80%) down to 3/4
(75%). On the other hand, recall can only go down when the thres‐
hold is increased, which explains why its curve looks smooth.
Another way to select a good precision/recall tradeoff is to plot precision directly
against recall, as shown in Figure 3-5  (the same threshold as earlier is highlighed).Performance Measures | 97

Figure 3-5. Precision versus recall
Y ou can see that precision really starts to fall sharply around 80% recall. Y ou will
probably want to select a precision/recall tradeoff just before that drop—for example,
at around 60% recall. But of course the choice depends on your project.
So let’s suppose you decide to aim for 90% precision. Y ou look up the first plot and
find that you need to use a threshold of about 8,000. To be more precise you can
search for the lowest threshold that gives you at least 90% precision ( np.argmax()
will give us the first index of the maximum value, which in this case means the first
True  value):
threshold_90_precision  = thresholds [np.argmax(precisions  >= 0.90)] # ~7816
To make predictions (on the training set for now), instead of calling the classifier’s
predict()  method, you can just run this code:
y_train_pred_90  = (y_scores  >= threshold_90_precision )
Let’s check these predictions’ precision and recall:
>>> precision_score (y_train_5 , y_train_pred_90 )

0.9000380083618396
>>> recall_score (y_train_5 , y_train_pred_90 )
0.4368197749492714
Great, you have a 90% precision classifier ! As you can see, it is fairly easy to create a
classifier with virtually any precision you want: just set a high enough threshold, and
you’re done. Hmm, not so fast. A high-precision classifier is not very useful if its 
recall is too low!98 | Chapter 3: Classification

If someone says “let’s reach 99% precision, ” you should ask, “at
what recall?”
The ROC Curve
The receiver operating characteristic  (ROC) curve is another common tool used with
binary classifiers. It is very similar to the precision/recall curve, but instead of plot‐
ting precision versus recall, the ROC curve plots the true positive rate  (another name
for recall) against the false positive rate . The FPR is the ratio of negative instances that
are incorrectly classified as positive. It is equal to one minus the true negative rate , 
which is the ratio of negative instances that are correctly classified as negative. The
TNR is also called specificity . Hence the ROC curve plots sensitivity  (recall) versus
1 – specificity .
To plot the ROC curve, you first need to compute the TPR and FPR for various thres‐
hold values, using the roc_curve()  function:
from sklearn.metrics  import roc_curve
fpr, tpr, thresholds  = roc_curve (y_train_5 , y_scores )

Then you can plot the FPR against the TPR using Matplotlib. This code produces the
plot in Figure 3-6 :
def plot_roc_curve (fpr, tpr, label=None):
    plt.plot(fpr, tpr, linewidth =2, label=label)
    plt.plot([0, 1], [0, 1], 'k--') # dashed diagonal
    [...] # Add axis labels and grid
plot_roc_curve (fpr, tpr)
plt.show()Performance Measures | 99

Figure 3-6. ROC curve
Once again there is a tradeoff: the higher the recall (TPR), the more false positives
(FPR) the classifier produces. The dotted line represents the ROC curve of a purely
random classifier; a good classifier stays as far away from that line as possible (toward
the top-left corner).
One way to compare classifiers is to measure the area under the curve  (AUC). A per‐
fect classifier will have a ROC AUC  equal to 1, whereas a purely random classifier will
have a ROC AUC equal to 0.5. Scikit-Learn provides a function to compute the ROC
AUC:
>>> from sklearn.metrics  import roc_auc_score
>>> roc_auc_score (y_train_5 , y_scores )
0.9611778893101814
Since the ROC curve is so similar to the precision/recall (or PR)
curve, you may wonder how to decide which one to use. As a rule
of thumb, you should prefer the PR curve whenever the positive
class is rare or when you care more about the false positives than
the false negatives, and the ROC curve otherwise. For example,

AUC:
looking at the previous ROC curve (and the ROC AUC score), you
may think that the classifier is really good. But this is mostly
because there are few positives (5s) compared to the negatives
(non-5s). In contrast, the PR curve makes it clear that the classifier
has room for improvement (the curve could be closer to the top-
right corner).100 | Chapter 3: Classification

Let’s train a RandomForestClassifier  and compare its ROC curve and ROC AUC
score to the SGDClassifier . First, you need to get scores for each instance in the
training set. But due to the way it works (see Chapter 7 ), the RandomForestClassi
fier  class does not have a decision_function()  method. Instead it has a pre
dict_proba()  method. Scikit-Learn classifiers generally have one or the other. The
predict_proba()  method returns an array containing a row per instance and a col‐
umn per class, each containing the probability that the given instance belongs to the
given class (e.g., 70% chance that the image represents a 5):
from sklearn.ensemble  import RandomForestClassifier
forest_clf  = RandomForestClassifier (random_state =42)
y_probas_forest  = cross_val_predict (forest_clf , X_train, y_train_5 , cv=3,
                                    method="predict_proba" )
But to plot a ROC curve, you need scores, not probabilities. A simple solution is to
use the positive class’s probability as the score:

y_scores_forest  = y_probas_forest [:, 1]   # score = proba of positive class
fpr_forest , tpr_forest , thresholds_forest  = roc_curve (y_train_5 ,y_scores_forest )
Now you are ready to plot the ROC curve. It is useful to plot the first ROC curve as
well to see how they compare ( Figure 3-7 ):
plt.plot(fpr, tpr, "b:", label="SGD")
plot_roc_curve (fpr_forest , tpr_forest , "Random Forest" )
plt.legend(loc="lower right" )
Figure 3-7. Comparing ROC curvesPerformance Measures | 101

As you can see in Figure 3-7 , the RandomForestClassifier ’s ROC curve looks much
better than the SGDClassifier ’s: it comes much closer to the top-left corner. As a
result, its ROC AUC score is also significantly better:
>>> roc_auc_score (y_train_5 , y_scores_forest )
0.9983436731328145
Try measuring the precision and recall scores: you should find 99.0% precision and
86.6% recall. Not too bad!
Hopefully you now know how to train binary classifiers, choose the appropriate met‐
ric for your task, evaluate your classifiers using cross-validation, select the precision/
recall tradeoff that fits your needs, and compare various models using ROC curves
and ROC AUC scores. Now let’s try to detect more than just the 5s.
Multiclass Classification
Whereas binary classifiers distinguish between two classes, multiclass classifiers  (also
called multinomial classifiers ) can distinguish between more than two classes.
Some algorithms (such as Random Forest classifiers or naive Bayes classifiers) are

capable of handling multiple classes directly. Others (such as Support Vector Machine
classifiers or Linear classifiers) are strictly binary classifiers. However, there are vari‐
ous strategies that you can use to perform multiclass classification using multiple
binary classifiers.
For example, one way to create a system that can classify the digit images into 10
classes (from 0 to 9) is to train 10 binary classifiers, one for each digit (a 0-detector, a
1-detector, a 2-detector, and so on). Then when you want to classify an image, you get

the decision score from each classifier for that image and you select the class whose
classifier outputs the highest score. This is called the one-versus-all  (OvA) strategy 
(also called one-versus-the-rest ).
Another strategy is to train a binary classifier for every pair of digits: one to distin‐
guish 0s and 1s, another to distinguish 0s and 2s, another for 1s and 2s, and so on.
This is called the one-versus-one  (OvO) strategy. If there are N classes, you need to
train N × (N – 1) / 2 classifiers. For the MNIST problem, this means training 45

binary classifiers! When you want to classify an image, you have to run the image
through all 45 classifiers and see which class wins the most duels. The main advan‐
tage of OvO is that each classifier only needs to be trained on the part of the training
set for the two classes that it must distinguish.
Some algorithms (such as Support Vector Machine classifiers) scale poorly with the
size of the training set, so for these algorithms OvO is preferred since it is faster to

train many classifiers on small training sets than training few classifiers on large
training sets. For most binary classification algorithms, however, OvA is preferred.102 | Chapter 3: Classification

Scikit-Learn detects when you try to use a binary classification algorithm for a multi‐
class classification task, and it automatically runs OvA (except for SVM classifiers for
which it uses OvO). Let’s try this with the SGDClassifier :
>>> sgd_clf.fit(X_train, y_train)  # y_train, not y_train_5
array([5], dtype=uint8)
That was easy! This code trains the SGDClassifier  on the training set using the origi‐
nal target classes from 0 to 9 ( y_train ), instead of the 5-versus-all target classes
(y_train_5 ). Then it makes a prediction (a correct one in this case). Under the hood,
Scikit-Learn actually trained 10 binary classifiers, got their decision scores for the
image, and selected the class with the highest score.
To see that this is indeed the case, you can call the decision_function()  method.
Instead of returning just one score per instance, it now returns 10 scores, one per
class:
>>> some_digit_scores  = sgd_clf.decision_function ([some_digit ])
>>> some_digit_scores

class:
array([[-15955.22627845, -38080.96296175, -13326.66694897,
           573.52692379, -17680.6846644 ,   2412.53175101,
        -25526.86498156, -12290.15704709,  -7946.05205023,
        -10631.35888549]])
The highest score is indeed the one corresponding to class 5:
>>> np.argmax(some_digit_scores )
5
>>> sgd_clf.classes_
array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=uint8)
>>> sgd_clf.classes_ [5]
5
When a classifier is trained, it stores the list of target classes in its

5
5
classes_  attribute, ordered by value. In this case, the index of each
class in the classes_  array conveniently matches the class itself
(e.g., the class at index 5 happens to be class 5), but in general you
won’t be so lucky.
If you want to force ScikitLearn to use one-versus-one or one-versus-all, you can use
the OneVsOneClassifier  or OneVsRestClassifier  classes. Simply create an instance
and pass a binary classifier to its constructor. For example, this code creates a multi‐

class classifier using the OvO strategy, based on a SGDClassifier :
>>> from sklearn.multiclass  import OneVsOneClassifier
>>> ovo_clf = OneVsOneClassifier (SGDClassifier (random_state =42))
>>> ovo_clf.fit(X_train, y_train)
>>> ovo_clf.predict([some_digit ])Multiclass Classification  | 103

>>> len(ovo_clf.estimators_ )
45
Training a RandomForestClassifier  is just as easy:
>>> forest_clf .fit(X_train, y_train)
>>> forest_clf .predict([some_digit ])
This time Scikit-Learn did not have to run OvA or OvO because Random Forest
classifiers  can directly classify instances into multiple classes. Y ou can call
predict_proba()  to get the list of probabilities that the classifier assigned to each
instance for each class:
>>> forest_clf .predict_proba ([some_digit ])
array([[0.  , 0.  , 0.01, 0.08, 0.  , 0.9 , 0.  , 0.  , 0.  , 0.01]])
Y ou can see that the classifier is fairly confident about its prediction: the 0.9 at the 5th
index in the array means that the model estimates a 90% probability that the image
represents a 5. It also thinks that the image could instead be a 2, a 3 or a 9, respec‐
tively with 1%, 8% and 1% probability.
Now of course you want to evaluate these classifiers. As usual, you want to use cross-

validation. Let’s evaluate the SGDClassifier ’s accuracy using the cross_val_score()
function:
>>> cross_val_score (sgd_clf, X_train, y_train, cv=3, scoring="accuracy" )
array([0.8489802 , 0.87129356, 0.86988048])
It gets over 84% on all test folds. If you used a random classifier, you would get 10%
accuracy, so this is not such a bad score, but you can still do much better. For exam‐
ple, simply scaling the inputs (as discussed in Chapter 2 ) increases accuracy above
89%:

function:
89%:
>>> from sklearn.preprocessing  import StandardScaler
>>> scaler = StandardScaler ()
>>> X_train_scaled  = scaler.fit_transform (X_train.astype(np.float64))
>>> cross_val_score (sgd_clf, X_train_scaled , y_train, cv=3, scoring="accuracy" )
array([0.89707059, 0.8960948 , 0.90693604])
Error Analysis
Of course, if this were a real project, you would follow the steps in your Machine
Learning project checklist (see ???): exploring data preparation options, trying out

89%:
multiple models, shortlisting the best ones and fine-tuning their hyperparameters
using GridSearchCV , and automating as much as possible, as you did in the previous
chapter. Here, we will assume that you have found a promising model and you want
to find ways to improve it. One way to do this is to analyze the types of errors it
makes.104 | Chapter 3: Classification

First, you can look at the confusion matrix. Y ou need to make predictions using the
cross_val_predict()  function, then call the confusion_matrix()  function, just like
you did earlier:
>>> y_train_pred  = cross_val_predict (sgd_clf, X_train_scaled , y_train, cv=3)
>>> conf_mx = confusion_matrix (y_train, y_train_pred )
>>> conf_mx
array([[5578,    0,   22,    7,    8,   45,   35,    5,  222,    1],
       [   0, 6410,   35,   26,    4,   44,    4,    8,  198,   13],
       [  28,   27, 5232,  100,   74,   27,   68,   37,  354,   11],
       [  23,   18,  115, 5254,    2,  209,   26,   38,  373,   73],
       [  11,   14,   45,   12, 5219,   11,   33,   26,  299,  172],
       [  26,   16,   31,  173,   54, 4484,   76,   14,  482,   65],
       [  31,   17,   45,    2,   42,   98, 5556,    3,  123,    1],
       [  20,   10,   53,   27,   50,   13,    3, 5696,  173,  220],
       [  17,   64,   47,   91,    3,  125,   24,   11, 5421,   48],

[  28,   27, 5232,  100,   74,   27,   68,   37,  354,   11],
       [  24,   18,   29,   67,  116,   39,    1,  174,  329, 5152]])
That’s a lot of numbers. It’s often more convenient to look at an image representation
of the confusion matrix, using Matplotlib’s matshow()  function:
plt.matshow(conf_mx, cmap=plt.cm.gray)
This confusion matrix looks fairly good, since most images are on the main diagonal,
which means that they were classified correctly. The 5s look slightly darker than the
other digits, which could mean that there are fewer images of 5s in the dataset or that

the classifier does not perform as well on 5s as on other digits. In fact, you can verify
that both are the case.
Let’s focus the plot on the errors. First, you need to divide each value in the confusion
matrix by the number of images in the corresponding class, so you can compare errorError Analysis | 105

rates instead of absolute number of errors (which would make abundant classes look
unfairly bad):
row_sums  = conf_mx.sum(axis=1, keepdims =True)
norm_conf_mx  = conf_mx / row_sums
Now let’s fill the diagonal with zeros to keep only the errors, and let’s plot the result:
np.fill_diagonal (norm_conf_mx , 0)
plt.matshow(norm_conf_mx , cmap=plt.cm.gray)
Now you can clearly see the kinds of errors the classifier makes. Remember that rows
represent actual classes, while columns represent predicted classes. The column for
class 8 is quite bright, which tells you that many images get misclassified as 8s. How‐
ever, the row for class 8 is not that bad, telling you that actual 8s in general get prop‐
erly classified as 8s. As you can see, the confusion matrix is not necessarily
symmetrical. Y ou can also see that 3s and 5s often get confused (in both directions).
Analyzing the confusion matrix can often give you insights on ways to improve your

classifier. Looking at this plot, it seems that your efforts should be spent on reducing
the false 8s. For example, you could try to gather more training data for digits that
look like 8s (but are not) so the classifier can learn to distinguish them from real 8s.
Or you could engineer new features that would help the classifier—for example, writ‐
ing an algorithm to count the number of closed loops (e.g., 8 has two, 6 has one, 5 has

none). Or you could preprocess the images (e.g., using Scikit-Image, Pillow, or
OpenCV) to make some patterns stand out more, such as closed loops.
Analyzing individual errors can also be a good way to gain insights on what your
classifier is doing and why it is failing, but it is more difficult and time-consuming.106 | Chapter 3: Classification

3But remember that our brain is a fantastic pattern recognition system, and our visual system does a lot of
complex preprocessing before any information reaches our consciousness, so the fact that it feels simple does
not mean that it is.For example, let’s plot examples of 3s and 5s (the plot_digits()  function just uses
Matplotlib’s imshow()  function; see this chapter’s Jupyter notebook for details):
cl_a, cl_b = 3, 5
X_aa = X_train[(y_train == cl_a) & (y_train_pred  == cl_a)]
X_ab = X_train[(y_train == cl_a) & (y_train_pred  == cl_b)]
X_ba = X_train[(y_train == cl_b) & (y_train_pred  == cl_a)]
X_bb = X_train[(y_train == cl_b) & (y_train_pred  == cl_b)]
plt.figure(figsize=(8,8))
plt.subplot(221); plot_digits (X_aa[:25], images_per_row =5)
plt.subplot(222); plot_digits (X_ab[:25], images_per_row =5)
plt.subplot(223); plot_digits (X_ba[:25], images_per_row =5)
plt.subplot(224); plot_digits (X_bb[:25], images_per_row =5)

The two 5×5 blocks on the left show digits classified as 3s, and the two 5×5 blocks on
the right show images classified as 5s. Some of the digits that the classifier gets wrong
(i.e., in the bottom-left and top-right blocks) are so badly written that even a human
would have trouble classifying them (e.g., the 5 on the 1st row and 2nd column truly
looks like a badly written 3). However, most misclassified images seem like obvious
errors to us, and it’s hard to understand why the classifier made the mistakes it did.3

The reason is that we used a simple SGDClassifier , which is a linear model. All it
does is assign a weight per class to each pixel, and when it sees a new image it just
sums up the weighted pixel intensities to get a score for each class. So since 3s and 5s
differ only by a few pixels, this model will easily confuse them.Error Analysis | 107

The main difference between 3s and 5s is the position of the small line that joins the
top line to the bottom arc. If you draw a 3 with the junction slightly shifted to the left,
the classifier might classify it as a 5, and vice versa. In other words, this classifier is
quite sensitive to image shifting and rotation. So one way to reduce the 3/5 confusion
would be to preprocess the images to ensure that they are well centered and not too
rotated. This will probably help reduce other errors as well.
Multilabel Classification
Until now each instance has always been assigned to just one class. In some cases you
may want your classifier to output multiple classes for each instance. For example,
consider a face-recognition classifier: what should it do if it recognizes several people
on the same picture? Of course it should attach one tag per person it recognizes. Say
the classifier has been trained to recognize three faces, Alice, Bob, and Charlie; then

when it is shown a picture of Alice and Charlie, it should output [1, 0, 1] (meaning
“ Alice yes, Bob no, Charlie yes”). Such a classification system that outputs multiple
binary tags is called a multilabel classification  system.
We won’t go into face recognition just yet, but let’s look at a simpler example, just for
illustration purposes:
from sklearn.neighbors  import KNeighborsClassifier
y_train_large  = (y_train >= 7)
y_train_odd  = (y_train % 2 == 1)
y_multilabel  = np.c_[y_train_large , y_train_odd ]
knn_clf = KNeighborsClassifier ()

knn_clf.fit(X_train, y_multilabel )
This code creates a y_multilabel  array containing two target labels for each digit
image: the first indicates whether or not the digit is large (7, 8, or 9) and the second
indicates whether or not it is odd. The next lines create a KNeighborsClassifier  
instance (which supports multilabel classification, but not all classifiers do) and we
train it using the multiple targets array. Now you can make a prediction, and notice
that it outputs two labels:
>>> knn_clf.predict([some_digit ])
array([[False,  True]])

And it gets it right! The digit 5 is indeed not large ( False ) and odd ( True ).
There are many ways to evaluate a multilabel classifier, and selecting the right metric
really depends on your project. For example, one approach is to measure the F1 score
for each individual label (or any other binary classifier metric discussed earlier), then
simply compute the average score. This code computes the average F1 score across all
labels:108 | Chapter 3: Classification

4Scikit-Learn offers a few other averaging options and multilabel classifier metrics; see the documentation for
more details.
>>> y_train_knn_pred  = cross_val_predict (knn_clf, X_train, y_multilabel , cv=3)
>>> f1_score (y_multilabel , y_train_knn_pred , average="macro")
0.976410265560605
This assumes that all labels are equally important, which may not be the case. In par‐
ticular, if you have many more pictures of Alice than of Bob or Charlie, you may want
to give more weight to the classifier’s score on pictures of Alice. One simple option is
to give each label a weight equal to its support  (i.e., the number of instances with that
target label). To do this, simply set average="weighted"  in the preceding code.4
Multioutput Classification
The last type of classification task we are going to discuss here is called multioutput-
multiclass classification  (or simply multioutput classification ). It is simply a generaliza‐
tion of multilabel classification where each label can be multiclass (i.e., it can have

more than two possible values).
To illustrate this, let’s build a system that removes noise from images. It will take as
input a noisy digit image, and it will (hopefully) output a clean digit image, repre‐
sented as an array of pixel intensities, just like the MNIST images. Notice that the
classifier’s output is multilabel (one label per pixel) and each label can have multiple
values (pixel intensity ranges from 0 to 255). It is thus an example of a multioutput
classification system.

The line between classification and regression is sometimes blurry,
such as in this example. Arguably, predicting pixel intensity is more
akin to regression than to classification. Moreover, multioutput
systems are not limited to classification tasks; you could even have
a system that outputs multiple labels per instance, including both
class labels and value labels.
Let’s start by creating the training and test sets by taking the MNIST images and

adding noise to their pixel intensities using NumPy’s randint()  function. The target
images will be the original images:
noise = np.random.randint(0, 100, (len(X_train), 784))
X_train_mod  = X_train + noise
noise = np.random.randint(0, 100, (len(X_test), 784))
X_test_mod  = X_test + noise
y_train_mod  = X_train
y_test_mod  = X_testMultioutput Classification  | 109

5Y ou can use the shift()  function from the scipy.ndimage.interpolation  module. For example,
shift(image, [2, 1], cval=0)  shifts the image 2 pixels down and 1 pixel to the right.Let’s take a peek at an image from the test set (yes, we’re snooping on the test data, so
you should be frowning right now):
On the left is the noisy input image, and on the right is the clean target image. Now
let’s train the classifier and make it clean this image:
knn_clf.fit(X_train_mod , y_train_mod )
clean_digit  = knn_clf.predict([X_test_mod [some_index ]])
plot_digit (clean_digit )
Looks close enough to the target! This concludes our tour of classification. Hopefully
you should now know how to select good metrics for classification tasks, pick the
appropriate precision/recall tradeoff, compare classifiers, and more generally build
good classification systems for a variety of tasks.
Exercises1.
Try to build a classifier for the MNIST dataset that achieves over 97% accuracy

on the test set. Hint: the KNeighborsClassifier  works quite well for this task;
you just need to find good hyperparameter values (try a grid search on the
weights  and n_neighbors  hyperparameters).2.
Write a function that can shift an MNIST image in any direction (left, right, up,
or down) by one pixel.5 Then, for each image in the training set, create four shif‐110 | Chapter 3: Classification

ted copies (one per direction) and add them to the training set. Finally, train your
best model on this expanded training set and measure its accuracy on the test set.
Y ou should observe that your model performs even better now! This technique of
artificially growing the training set is called data augmentation  or training set
expansion .3.
Tackle the Titanic  dataset. A great place to start is on Kaggle .4.
Build a spam classifier (a more challenging exercise):•
Download examples of spam and ham from Apache SpamAssassin’s public
datasets .•
Unzip the datasets and familiarize yourself with the data format.•
Split the datasets into a training set and a test set.•
Write a data preparation pipeline to convert each email into a feature vector.
Y our preparation pipeline should transform an email into a (sparse) vector
indicating the presence or absence of each possible word. For example, if all
emails only ever contain four words, “Hello, ” “how, ” “are, ” “you, ” then the email

“Hello you Hello Hello you” would be converted into a vector [1, 0, 0, 1]
(meaning [“Hello” is present, “how” is absent, “are” is absent, “you” is
present]), or [3, 0, 0, 2] if you prefer to count the number of occurrences of
each word.•
Y ou may want to add hyperparameters to your preparation pipeline to control
whether or not to strip off email headers, convert each email to lowercase,
remove punctuation, replace all URLs with “URL, ” replace all numbers with
“NUMBER, ” or even perform stemming  (i.e., trim off word endings; there are

Python libraries available to do this).•
Then try out several classifiers and see if you can build a great spam classifier,
with both high recall and high precision.
github.com/ageron/handson-ml2 .Exercises | 111

CHAPTER 4
Training Models
release of these titles. The following will be Chapter 4 in the final
So far we have treated Machine Learning models and their training algorithms mostly
like black boxes. If you went through some of the exercises in the previous chapters,
you may have been surprised by how much you can get done without knowing any‐
thing about what’s under the hood: you optimized a regression system, you improved
a digit image classifier, and you even built a spam classifier from scratch—all this
without knowing how they actually work. Indeed, in many situations you don’t really
need to know the implementation details.
However, having a good understanding of how things work can help you quickly
home in on the appropriate model, the right training algorithm to use, and a good set

of hyperparameters for your task. Understanding what’s under the hood will also help
you debug issues and perform error analysis more efficiently. Lastly, most of the top‐
ics discussed in this chapter will be essential in understanding, building, and training
neural networks (discussed in Part II  of this book).
In this chapter, we will start by looking at the Linear Regression model, one of the
simplest models there is. We will discuss two very different ways to train it:•
Using a direct “closed-form” equation that directly computes the model parame‐

ters that best fit the model to the training set (i.e., the model parameters that
minimize the cost function over the training set).113

• Using an iterative optimization approach, called Gradient Descent (GD), that
gradually tweaks the model parameters to minimize the cost function over the
training set, eventually converging to the same set of parameters as the first
method. We will look at a few variants of Gradient Descent that we will use again
and again when we study neural networks in Part II : Batch GD, Mini-batch GD,
and Stochastic GD.
Next we will look at Polynomial Regression, a more complex model that can fit non‐
linear datasets. Since this model has more parameters than Linear Regression, it is
more prone to overfitting the training data, so we will look at how to detect whether
or not this is the case, using learning curves, and then we will look at several regulari‐
zation techniques that can reduce the risk of overfitting the training set.
Finally, we will look at two more models that are commonly used for classification
tasks: Logistic Regression and Softmax Regression.

There will be quite a few math equations in this chapter, using basic
notions of linear algebra and calculus. To understand these equa‐
tions, you will need to know what vectors and matrices are, how to
transpose them, multiply them, and inverse them, and what partial
derivatives are. If you are unfamiliar with these concepts, please go
through the linear algebra and calculus introductory tutorials avail‐
able as Jupyter notebooks in the online supplemental material. For
those who are truly allergic to mathematics, you should still go

through this chapter and simply skip the equations; hopefully, the
text will be sufficient to help you understand most of the concepts.
Linear Regression
In Chapter 1 , we looked at a simple regression model of life satisfaction: life_satisfac‐
tion = θ0 + θ1 × GDP_per_capita .
This model is just a linear function of the input feature GDP_per_capita . θ0 and θ1 are
the model’s parameters.
More generally, a linear model makes a prediction by simply computing a weighted

sum of the input features, plus a constant called the bias term  (also called the intercept
term ), as shown in Equation 4-1 .
Equation 4-1. Linear Regression model predictiony=θ0+θ1x1+θ2x2+⋯+θnxn •
ŷ is the predicted value.114 | Chapter 4: Training Models

•n is the number of features. •
xi is the ith feature value.•
θj is the jth model parameter (including the bias term θ0 and the feature weights
θ1, θ2, ⋯, θn).
This can be written much more concisely using a vectorized form, as shown in Equa‐
tion 4-2 .
Equation 4-2. Linear Regression model prediction (vectorized form)y=hθx=θ·x•
θ is the model’s parameter vector , containing the bias term θ0 and the feature
weights θ1 to θn.•
x is the instance’s feature vector , containing x0 to xn, with x0 always equal to 1.•
θ · x is the dot product of the vectors θ and x, which is of course equal toθ0x0+θ1x1+θ2x2+⋯+θnxn
.•
hθ is the hypothesis function, using the model parameters θ.
In Machine Learning, vectors are often represented as column vec‐
tors, which are 2D arrays with a single column. If θ and x are col‐
umn vectors, then the prediction is: y=θTx, where θT is the
transpose  of θ (a row vector instead of a column vector) and θTx is
the matrix multiplication of θT and x. It is of course the same pre‐

.•
diction, except it is now represented as a single cell matrix rather
than a scalar value. In this book we will use this notation to avoid
switching between dot products and matrix multiplications.
Okay, that’s the Linear Regression model, so now how do we train it? Well, recall that
training a model means setting its parameters so that the model best fits the training
set. For this purpose, we first need a measure of how well (or poorly) the model fits

the training data. In Chapter 2  we saw that the most common performance measure
of a regression model is the Root Mean Square Error (RMSE) ( Equation 2-1 ). There‐
fore, to train a Linear Regression model, you need to find the value of θ that minimi‐
zes the RMSE. In practice, it is simpler to minimize the Mean Square Error (MSE)Linear Regression | 115

1It is often the case that a learning algorithm will try to optimize a different function than the performance
measure used to evaluate the final model. This is generally because that function is easier to compute, because
it has useful differentiation properties that the performance measure lacks, or because we want to constrain
the model during training, as we will see when we discuss regularization.2
The demonstration that this returns the value of θ that minimizes the cost function is outside the scope of this
book.than the RMSE, and it leads to the same result (because the value that minimizes a
function also minimizes its square root).1
The MSE of a Linear Regression hypothesis hθ on a training set X is calculated using
Equation 4-3 .
Equation 4-3. MSE cost function for a Linear Regression modelMSE X,hθ=1
m∑
i= 1m
θTxi−yi2
Most of these notations were presented in Chapter 2  (see “Notations”  on page 43).
The only difference is that we write hθ instead of just h in order to make it clear that

m∑
i= 1m
θTxi−yi2
the model is parametrized by the vector θ. To simplify notations, we will just write
MSE( θ) instead of MSE( X, hθ).
The Normal Equation
To find the value of θ that minimizes the cost function, there is a closed-form solution
—in other words, a mathematical equation that gives the result directly. This is called
the Normal Equation  (Equation 4-4 ).2
Equation 4-4. Normal Equationθ=XTX−1 XT y •θ is the value of θ that minimizes the cost function.•
y is the vector of target values containing y(1) to y(m).

Let’s generate some linear-looking data to test this equation on ( Figure 4-1 ):
X = 2 * np.random.rand(100, 1)
y = 4 + 3 * X + np.random.randn(100, 1)116 | Chapter 4: Training Models

Figure 4-1. Randomly generated linear dataset
Now let’s compute θ using the Normal Equation. We will use the inv()  function from
NumPy’s Linear Algebra module ( np.linalg ) to compute the inverse of a matrix, and
the dot()  method for matrix multiplication:
X_b = np.c_[np.ones((100, 1)), X]  # add x0 = 1 to each instance
theta_best  = np.linalg.inv(X_b.T.dot(X_b)).dot(X_b.T).dot(y)
The actual function that we used to generate the data is y = 4 + 3 x1 + Gaussian noise.
Let’s see what the equation found:
>>> theta_best
array([[4.21509616],
       [2.77011339]])
We would have hoped for θ0 = 4 and θ1 = 3 instead of θ0 = 4.215 and θ1 = 2.770. Close
enough, but the noise made it impossible to recover the exact parameters of the origi‐
nal function.
Now you can make predictions using θ:
>>> X_new = np.array([[0], [2]])
>>> X_new_b = np.c_[np.ones((2, 1)), X_new] # add x0 = 1 to each instance
>>> y_predict  = X_new_b.dot(theta_best )
>>> y_predict
       [9.75532293]])

Let’s plot this model’s predictions ( Figure 4-2 ):
plt.plot(X_new, y_predict , "r-")
plt.plot(X, y, "b.")Linear Regression | 117

The pseudoinverse itself is computed using a standard matrix factorization technique 
called Singular Value Decomposition  (SVD) that can decompose the training set
matrix X into the matrix multiplication of three matrices U Σ VT (see
numpy.linalg.svd() ). The pseudoinverse is computed as X+=VΣ+UT
. To compute
the matrix Σ+, the algorithm takes Σ and sets to zero all values smaller than a tiny
threshold value, then it replaces all the non-zero values with their inverse, and finally
it transposes the resulting matrix. This approach is more efficient than computing the
Normal Equation, plus it handles edge cases nicely: indeed, the Normal Equation may
not work if the matrix XTX is not invertible (i.e., singular), such as if m < n or if some
features are redundant, but the pseudoinverse is always defined.
Computational Complexity
The Normal Equation computes the inverse of XT X, which is an ( n + 1) × ( n + 1)
matrix (where n is the number of features). The computational complexity  of inverting

such a matrix is typically about O(n2.4) to O(n3) (depending on the implementation).
In other words, if you double the number of features, you multiply the computation
time by roughly 22.4 = 5.3 to 23 = 8.
The SVD approach used by Scikit-Learn’s LinearRegression  class is about O(n2). If
you double the number of features, you multiply the computation time by roughly 4.
Both the Normal Equation and the SVD approach get very slow
when the number of features grows large (e.g., 100,000). On the
positive side, both are linear with regards to the number of instan‐

ces in the training set (they are O(m)), so they handle large training
sets efficiently, provided they can fit in memory.
Also, once you have trained your Linear Regression model (using the Normal Equa‐
tion or any other algorithm), predictions are very fast: the computational complexity
is linear with regards to both the number of instances you want to make predictions
on and the number of features. In other words, making predictions on twice as many
instances (or twice as many features) will just take roughly twice as much time.

Now we will look at very different ways to train a Linear Regression model, better
suited for cases where there are a large number of features, or too many training
instances to fit in memory.
Gradient Descent
Gradient Descent  is a very generic optimization algorithm capable of finding optimal
solutions to a wide range of problems. The general idea of Gradient Descent is to
tweak parameters iteratively in order to minimize a cost function.Gradient Descent | 119

Suppose you are lost in the mountains in a dense fog; you can only feel the slope of
the ground below your feet. A good strategy to get to the bottom of the valley quickly
is to go downhill in the direction of the steepest slope. This is exactly what Gradient
Descent does: it measures the local gradient of the error function with regards to the 
parameter vector θ, and it goes in the direction of descending gradient. Once the gra‐
dient is zero, you have reached a minimum!
Concretely, you start by filling θ with random values (this is called random initializa‐
tion), and then you improve it gradually, taking one baby step at a time, each step
attempting to decrease the cost function (e.g., the MSE), until the algorithm converges
to a minimum (see Figure 4-3 ).
Figure 4-3. Gradient Descent
An important parameter in Gradient Descent is the size of the steps, determined by 
the learning rate  hyperparameter. If the learning rate is too small, then the algorithm

will have to go through many iterations to converge, which will take a long time (see
Figure 4-4 ).120 | Chapter 4: Training Models

Figure 4-4. Learning rate too small
On the other hand, if the learning rate is too high, you might jump across the valley
and end up on the other side, possibly even higher up than you were before. This
might make the algorithm diverge, with larger and larger values, failing to find a good
solution (see Figure 4-5 ).
Figure 4-5. Learning rate too large
Finally, not all cost functions look like nice regular bowls. There may be holes, ridges,
plateaus, and all sorts of irregular terrains, making convergence to the minimum very
difficult. Figure 4-6  shows the two main challenges with Gradient Descent: if the ran‐
dom initialization starts the algorithm on the left, then it will converge to a local mini‐
mum , which is not as good as the global minimum . If it starts on the right, then it will
take a very long time to cross the plateau, and if you stop too early you will never
reach the global minimum.Gradient Descent | 121

4Technically speaking, its derivative is Lipschitz continuous .5
Since feature 1 is smaller, it takes a larger change in θ1 to affect the cost function, which is why the bowl is
elongated along the θ1 axis.
Figure 4-6. Gradient Descent pitfalls
Fortunately, the MSE cost function for a Linear Regression model happens to be a
convex function , which means that if you pick any two points on the curve, the line
segment joining them never crosses the curve. This implies that there are no local
minima, just one global minimum. It is also a continuous function with a slope that
never changes abruptly.4 These two facts have a great consequence: Gradient Descent
is guaranteed to approach arbitrarily close the global minimum (if you wait long
enough and if the learning rate is not too high).
In fact, the cost function has the shape of a bowl, but it can be an elongated bowl if
the features have very different scales. Figure 4-7  shows Gradient Descent on a train‐

ing set where features 1 and 2 have the same scale (on the left), and on a training set
where feature 1 has much smaller values than feature 2 (on the right).5
Figure 4-7. Gradient Descent with and without feature scaling122 | Chapter 4: Training Models

As you can see, on the left the Gradient Descent algorithm goes straight toward the
minimum, thereby reaching it quickly, whereas on the right it first goes in a direction
almost orthogonal to the direction of the global minimum, and it ends with a long
march down an almost flat valley. It will eventually reach the minimum, but it will
take a long time.
When using Gradient Descent, you should ensure that all features
have a similar scale (e.g., using Scikit-Learn’s StandardScaler
class), or else it will take much longer to converge.
This diagram also illustrates the fact that training a model means searching for a
combination of model parameters that minimizes a cost function (over the training
set). It is a search in the model’s parameter space : the more parameters a model has,
the more dimensions this space has, and the harder the search is: searching for a nee‐
dle in a 300-dimensional haystack is much trickier than in three dimensions. Fortu‐

nately, since the cost function is convex in the case of Linear Regression, the needle is
simply at the bottom of the bowl.
Batch Gradient Descent
To implement Gradient Descent, you need to compute the gradient of the cost func‐
tion with regards to each model parameter θj. In other words, you need to calculate
how much the cost function will change if you change θj just a little bit. This is called 
a partial derivative . It is like asking “what is the slope of the mountain under my feet

if I face east?” and then asking the same question facing north (and so on for all other
dimensions, if you can imagine a universe with more than three dimensions). Equa‐
tion 4-5  computes the partial derivative of the cost function with regards to parame‐
ter θj, noted ∂
∂θj MSE( θ).
Equation 4-5. Partial derivatives of the cost function∂
∂θjMSE θ=2
m∑
i= 1m
θTxi−yixji
Instead of computing these partial derivatives individually, you can use Equation 4-6

∂θjMSE θ=2
m∑
i= 1m
to compute them all in one go. The gradient vector, noted ∇θMSE( θ), contains all the
partial derivatives of the cost function (one for each model parameter).Gradient Descent | 123

6Eta ( η) is the 7th letter of the Greek alphabet.
 Equation 4-6. Gradient vector of the cost function∇θMSE θ=∂
∂θ0MSE θ
∂
∂θ1MSE θ
⋮
∂
∂θnMSE θ=2
mXTXθ−y
Notice that this formula involves calculations over the full training
set X, at each Gradient Descent step! This is why the algorithm is
called Batch Gradient Descent : it uses the whole batch of training
data at every step (actually, Full Gradient Descent  would probably
be a better name). As a result it is terribly slow on very large train‐
ing sets (but we will see much faster Gradient Descent algorithms
shortly). However, Gradient Descent scales well with the number of
features; training a Linear Regression model when there are hun‐
dreds of thousands of features is much faster using Gradient
Descent than using the Normal Equation or SVD decomposition.
Once you have the gradient vector, which points uphill, just go in the opposite direc‐
tion to go downhill. This means subtracting ∇θMSE( θ) from θ. This is where the

tion to go downhill. This means subtracting ∇θMSE( θ) from θ. This is where the 
learning rate η comes into play:6 multiply the gradient vector by η to determine the
size of the downhill step ( Equation 4-7 ).
Equation 4-7. Gradient Descent stepθnext step=θ−η∇θMSE θ
Let’s look at a quick implementation of this algorithm:
eta = 0.1  # learning rate
n_iterations  = 1000
m = 100
theta = np.random.randn(2,1)  # random initialization
for iteration  in range(n_iterations ):
    gradients  = 2/m * X_b.T.dot(X_b.dot(theta) - y)
    theta = theta - eta * gradients124 | Chapter 4: Training Models

That wasn’t too hard! Let’s look at the resulting theta :
>>> theta
Hey, that’s exactly what the Normal Equation found! Gradient Descent worked per‐
fectly. But what if you had used a different learning rate eta? Figure 4-8  shows the
first 10 steps of Gradient Descent using three different learning rates (the dashed line
represents the starting point).
Figure 4-8. Gradient Descent with various learning rates
On the left, the learning rate is too low: the algorithm will eventually reach the solu‐
tion, but it will take a long time. In the middle, the learning rate looks pretty good: in
just a few iterations, it has already converged to the solution. On the right, the learn‐
ing rate is too high: the algorithm diverges, jumping all over the place and actually
getting further and further away from the solution at every step.
To find a good learning rate, you can use grid search (see Chapter 2 ). However, you

may want to limit the number of iterations so that grid search can eliminate models
that take too long to converge.
Y ou may wonder how to set the number of iterations. If it is too low, you will still be
far away from the optimal solution when the algorithm stops, but if it is too high, you
will waste time while the model parameters do not change anymore. A simple solu‐
tion is to set a very large number of iterations but to interrupt the algorithm when the

gradient vector becomes tiny—that is, when its norm becomes smaller than a tiny
number ϵ (called the tolerance )—because this happens when Gradient Descent has
(almost) reached the minimum.Gradient Descent | 125

7Out-of-core algorithms are discussed in Chapter 1 .
Convergence Rate
When the cost function is convex and its slope does not change abruptly (as is the
case for the MSE cost function), Batch Gradient Descent with a fixed learning rate
will eventually converge to the optimal solution, but you may have to wait a while: it
can take O(1/ ϵ) iterations to reach the optimum within a range of ϵ depending on the
shape of the cost function. If you divide the tolerance by 10 to have a more precise
solution, then the algorithm may have to run about 10 times longer.
Stochastic Gradient Descent
The main problem with Batch Gradient Descent is the fact that it uses the whole
training set to compute the gradients at every step, which makes it very slow when
the training set is large. At the opposite extreme, Stochastic Gradient Descent  just
picks a random instance in the training set at every step and computes the gradients
based only on that single instance. Obviously this makes the algorithm much faster

since it has very little data to manipulate at every iteration. It also makes it possible to
train on huge training sets, since only one instance needs to be in memory at each
iteration (SGD can be implemented as an out-of-core algorithm.7)
On the other hand, due to its stochastic (i.e., random) nature, this algorithm is much
less regular than Batch Gradient Descent: instead of gently decreasing until it reaches
the minimum, the cost function will bounce up and down, decreasing only on aver‐

age. Over time it will end up very close to the minimum, but once it gets there it will
continue to bounce around, never settling down (see Figure 4-9 ). So once the algo‐
rithm stops, the final parameter values are good, but not optimal.
Figure 4-9. Stochastic Gradient Descent126 | Chapter 4: Training Models

When the cost function is very irregular (as in Figure 4-6 ), this can actually help the
algorithm jump out of local minima, so Stochastic Gradient Descent has a better
chance of finding the global minimum than Batch Gradient Descent does.
Therefore randomness is good to escape from local optima, but bad because it means
that the algorithm can never settle at the minimum. One solution to this dilemma is
to gradually reduce the learning rate. The steps start out large (which helps make
quick progress and escape local minima), then get smaller and smaller, allowing the
algorithm to settle at the global minimum. This process is akin to simulated anneal‐
ing, an algorithm inspired from the process of annealing in metallurgy where molten
metal is slowly cooled down. The function that determines the learning rate at each
iteration is called the learning schedule . If the learning rate is reduced too quickly, you
may get stuck in a local minimum, or even end up frozen halfway to the minimum. If

the learning rate is reduced too slowly, you may jump around the minimum for a
long time and end up with a suboptimal solution if you halt training too early.
This code implements Stochastic Gradient Descent using a simple learning schedule:
n_epochs  = 50
t0, t1 = 5, 50  # learning schedule hyperparameters
def learning_schedule (t):
    return t0 / (t + t1)
for epoch in range(n_epochs ):
    for i in range(m):
        random_index  = np.random.randint(m)
        xi = X_b[random_index :random_index +1]

        yi = y[random_index :random_index +1]
        gradients  = 2 * xi.T.dot(xi.dot(theta) - yi)
        eta = learning_schedule (epoch * m + i)
        theta = theta - eta * gradients
By convention we iterate by rounds of m iterations; each round is called an epoch . 
While the Batch Gradient Descent code iterated 1,000 times through the whole train‐
ing set, this code goes through the training set only 50 times and reaches a fairly good
solution:
>>> theta
array([[4.21076011],
       [2.74856079]])

yi = y[random_index :random_index +1]
solution:
>>> theta
Figure 4-10  shows the first 20 steps of training (notice how irregular the steps are).Gradient Descent | 127

Figure 4-10. Stochastic Gradient Descent first 20 steps
Note that since instances are picked randomly, some instances may be picked several
times per epoch while others may not be picked at all. If you want to be sure that the
algorithm goes through every instance at each epoch, another approach is to shuffle
the training set (making sure to shuffle the input features and the labels jointly), then
go through it instance by instance, then shuffle it again, and so on. However, this gen‐
erally converges more slowly.
When using Stochastic Gradient Descent, the training instances
must be independent and identically distributed (IID), to ensure
that the parameters get pulled towards the global optimum, on
average. A simple way to ensure this is to shuffle the instances dur‐
ing training (e.g., pick each instance randomly, or shuffle the train‐
ing set at the beginning of each epoch). If you do not do this, for
example if the instances are sorted by label, then SGD will start by

optimizing for one label, then the next, and so on, and it will not
settle close to the global minimum.
To perform Linear Regression using SGD with Scikit-Learn, you can use the SGDRe
gressor  class, which defaults to optimizing the squared error cost function. The fol‐
lowing code runs for maximum 1000 epochs ( max_iter=1000 ) or until the loss drops
by less than 1e-3 during one epoch ( tol=1e-3 ), starting with a learning rate of 0.1
(eta0=0.1 ), using the default learning schedule (different from the preceding one),

and it does not use any regularization ( penalty=None ; more details on this shortly):
from sklearn.linear_model  import SGDRegressor
sgd_reg = SGDRegressor (max_iter =1000, tol=1e-3, penalty=None, eta0=0.1)
sgd_reg.fit(X, y.ravel())128 | Chapter 4: Training Models

Once again, you find a solution quite close to the one returned by the Normal Equa‐
tion:
>>> sgd_reg.intercept_ , sgd_reg.coef_
(array([4.24365286]), array([2.8250878]))
Mini-batch Gradient Descent
The last Gradient Descent algorithm we will look at is called Mini-batch Gradient
Descent . It is quite simple to understand once you know Batch and Stochastic Gradi‐
ent Descent: at each step, instead of computing the gradients based on the full train‐
ing set (as in Batch GD) or based on just one instance (as in Stochastic GD), Mini-
batch GD computes the gradients on small random sets of instances called mini-
batches . The main advantage of Mini-batch GD over Stochastic GD is that you can
get a performance boost from hardware optimization of matrix operations, especially
when using GPUs.
The algorithm’s progress in parameter space is less erratic than with SGD, especially
with fairly large mini-batches. As a result, Mini-batch GD will end up walking

around a bit closer to the minimum than SGD. But, on the other hand, it may be
harder for it to escape from local minima (in the case of problems that suffer from
local minima, unlike Linear Regression as we saw earlier). Figure 4-11  shows the
paths taken by the three Gradient Descent algorithms in parameter space during
training. They all end up near the minimum, but Batch GD’s path actually stops at the
minimum, while both Stochastic GD and Mini-batch GD continue to walk around.

However, don’t forget that Batch GD takes a lot of time to take each step, and Stochas‐
tic GD and Mini-batch GD would also reach the minimum if you used a good learn‐
ing schedule.
Figure 4-11. Gradient Descent paths in parameter spaceGradient Descent | 129

8While the Normal Equation can only perform Linear Regression, the Gradient Descent algorithms can be
used to train many other models, as we will see.9
A quadratic equation is of the form y = ax2 + bx + c.
Let’s compare the algorithms we’ve discussed so far for Linear Regression8 (recall that
m is the number of training instances and n is the number of features); see Table 4-1 .Table 4-1. Comparison of algorithms for Linear RegressionAlgorithmLarge m Out-of-core supportLarge n HyperparamsScaling requiredScikit-LearnNormal EquationFastNoSlow0Non/aSVDFastNoSlow0NoLinearRegressionBatch GDSlowNoFast2YesSGDRegressorStochastic GDFastYesFast≥2YesSGDRegressorMini-batch GDFastYesFast≥2YesSGDRegressor
There is almost no difference after training: all these algorithms
end up with very similar models and make predictions in exactly 
the same way.
Polynomial Regression
What if your data is actually more complex than a simple straight line? Surprisingly,

you can actually use a linear model to fit nonlinear data. A simple way to do this is to
add powers of each feature as new features, then train a linear model on this extended
set of features. This technique is called Polynomial Regression .
Let’s look at an example. First, let’s generate some nonlinear data, based on a simple
quadratic equation9 (plus some noise; see Figure 4-12 ):
m = 100
X = 6 * np.random.rand(m, 1) - 3
y = 0.5 * X**2 + X + 2 + np.random.randn(m, 1)130 | Chapter 4: Training Models

Figure 4-12. Generated nonlinear and noisy dataset
Clearly, a straight line will never fit this data properly. So let’s use Scikit-Learn’s Poly
nomialFeatures  class to transform our training data, adding the square (2nd-degree
polynomial) of each feature in the training set as new features (in this case there is
just one feature):
>>> from sklearn.preprocessing  import PolynomialFeatures
>>> poly_features  = PolynomialFeatures (degree=2, include_bias =False)
>>> X_poly = poly_features .fit_transform (X)
>>> X[0]
array([-0.75275929])
>>> X_poly[0]
array([-0.75275929, 0.56664654])
X_poly  now contains the original feature of X plus the square of this feature. Now you
can fit a LinearRegression  model to this extended training data ( Figure 4-13 ):
>>> lin_reg = LinearRegression ()
>>> lin_reg.fit(X_poly, y)
>>> lin_reg.intercept_ , lin_reg.coef_
(array([1.78134581]), array([[0.93366893, 0.56456263]]))Polynomial Regression | 131

Figure 4-13. Polynomial Regression model predictions
Not bad: the model estimates y= 0 . 56 x12+ 0 . 93 x1+ 1 . 78
 when in fact the original
function was y= 0 . 5 x12+ 1 . 0 x1+ 2 . 0 + Gaussian noise
.
Note that when there are multiple features, Polynomial Regression is capable of find‐
ing relationships between features (which is something a plain Linear Regression
model cannot do). This is made possible by the fact that PolynomialFeatures  also
adds all combinations of features up to the given degree. For example, if there were
two features a and b, PolynomialFeatures  with degree=3  would not only add the
features a2, a3, b2, and b3, but also the combinations ab, a2b, and ab2.
PolynomialFeatures(degree=d)  transforms an array containing n
features into an array containing n+d!
d!n! features, where n! is the
factorial  of n, equal to 1 × 2 × 3 × ⋯ × n. Beware of the combinato‐
rial explosion of the number of features!
Learning Curves

If you perform high-degree Polynomial Regression, you will likely fit the training
data much better than with plain Linear Regression. For example, Figure 4-14  applies
a 300-degree polynomial model to the preceding training data, and compares the
result with a pure linear model and a quadratic model (2nd-degree polynomial).
Notice how the 300-degree polynomial model wiggles around to get as close as possi‐
ble to the training instances.132 | Chapter 4: Training Models

Figure 4-14. High-degree Polynomial Regression
Of course, this high-degree Polynomial Regression model is severely overfitting the
training data, while the linear model is underfitting it. The model that will generalize
best in this case is the quadratic model. It makes sense since the data was generated
using a quadratic model, but in general you won’t know what function generated the
data, so how can you decide how complex your model should be? How can you tell
that your model is overfitting or underfitting the data?
In Chapter 2  you used cross-validation to get an estimate of a model’s generalization
performance. If a model performs well on the training data but generalizes poorly
according to the cross-validation metrics, then your model is overfitting. If it per‐
forms poorly on both, then it is underfitting. This is one way to tell when a model is
too simple or too complex.
Another way is to look at the learning curves : these are plots of the model’s perfor‐

mance on the training set and the validation set as a function of the training set size
(or the training iteration). To generate the plots, simply train the model several times
on different sized subsets of the training set. The following code defines a function
that plots the learning curves of a model given some training data:
from sklearn.metrics  import mean_squared_error
def plot_learning_curves (model, X, y):
    X_train, X_val, y_train, y_val = train_test_split (X, y, test_size =0.2)

    train_errors , val_errors  = [], []
    for m in range(1, len(X_train)):
        model.fit(X_train[:m], y_train[:m])
        y_train_predict  = model.predict(X_train[:m])Learning Curves | 133

y_val_predict  = model.predict(X_val)
        train_errors .append(mean_squared_error (y_train[:m], y_train_predict ))
        val_errors .append(mean_squared_error (y_val, y_val_predict ))
    plt.plot(np.sqrt(train_errors ), "r-+", linewidth =2, label="train")
    plt.plot(np.sqrt(val_errors ), "b-", linewidth =3, label="val")
Let’s look at the learning curves of the plain Linear Regression model (a straight line;
Figure 4-15 ):
plot_learning_curves (lin_reg, X, y)
Figure 4-15. Learning curves
This deserves a bit of explanation. First, let’s look at the performance on the training
data: when there are just one or two instances in the training set, the model can fit
them perfectly, which is why the curve starts at zero. But as new instances are added
to the training set, it becomes impossible for the model to fit the training data per‐
fectly, both because the data is noisy and because it is not linear at all. So the error on

the training data goes up until it reaches a plateau, at which point adding new instan‐
ces to the training set doesn’t make the average error much better or worse. Now let’s
look at the performance of the model on the validation data. When the model is
trained on very few training instances, it is incapable of generalizing properly, which
is why the validation error is initially quite big. Then as the model is shown more
training examples, it learns and thus the validation error slowly goes down. However,

once again a straight line cannot do a good job modeling the data, so the error ends
up at a plateau, very close to the other curve.
These learning curves are typical of an underfitting model. Both curves have reached
a plateau; they are close and fairly high.134 | Chapter 4: Training Models

If your model is underfitting the training data, adding more train‐
ing examples will not help. Y ou need to use a more complex model
or come up with better features.
Now let’s look at the learning curves of a 10th-degree polynomial model on the same
data ( Figure 4-16 ):
polynomial_regression  = Pipeline ([
        ("poly_features" , PolynomialFeatures (degree=10, include_bias =False)),
        ("lin_reg" , LinearRegression ()),
    ])
plot_learning_curves (polynomial_regression , X, y)
These learning curves look a bit like the previous ones, but there are two very impor‐
tant differences:•
The error on the training data is much lower than with the Linear Regression
model.•
There is a gap between the curves. This means that the model performs signifi‐
cantly better on the training data than on the validation data, which is the hall‐
mark of an overfitting model. However, if you used a much larger training set,
the two curves would continue to get closer.

model.•
Figure 4-16. Learning curves for the polynomial modelLearning Curves | 135

10This notion of bias is not to be confused with the bias term of linear models.
 One way to improve an overfitting model is to feed it more training
data until the validation error reaches the training error.
The Bias/Variance Tradeoff
An important theoretical result of statistics and Machine Learning is the fact that a
model’s generalization error can be expressed as the sum of three very different
errors:
Bias
This part of the generalization error is due to wrong assumptions, such as assum‐
ing that the data is linear when it is actually quadratic. A high-bias model is most
likely to underfit the training data.10
Variance
This part is due to the model’s excessive sensitivity to small variations in the
training data. A model with many degrees of freedom (such as a high-degree pol‐
ynomial model) is likely to have high variance, and thus to overfit the training
data.
Irreducible error
This part is due to the noisiness of the data itself. The only way to reduce this

Variance
data.
part of the error is to clean up the data (e.g., fix the data sources, such as broken
sensors, or detect and remove outliers).
Increasing a model’s complexity will typically increase its variance and reduce its bias.
Conversely, reducing a model’s complexity increases its bias and reduces its variance. 
This is why it is called a tradeoff.
Regularized Linear Models
As we saw in Chapters 1 and 2, a good way to reduce overfitting is to regularize the

model (i.e., to constrain it): the fewer degrees of freedom it has, the harder it will be
for it to overfit the data. For example, a simple way to regularize a polynomial model
is to reduce the number of polynomial degrees.
For a linear model, regularization is typically achieved by constraining the weights of
the model. We will now look at Ridge Regression, Lasso Regression, and Elastic Net,
which implement three different ways to constrain the weights.136 | Chapter 4: Training Models

11It is common to use the notation J(θ) for cost functions that don’t have a short name; we will often use this
notation throughout the rest of this book. The context will make it clear which cost function is being dis‐
cussed.12
Norms are discussed in Chapter 2 .
Ridge Regression
Ridge Regression  (also called Tikhonov regularization ) is a regularized version of Lin‐
ear Regression: a regularization term  equal to α∑i= 1nθi2
 is added to the cost function. 
This forces the learning algorithm to not only fit the data but also keep the model
weights as small as possible. Note that the regularization term should only be added
to the cost function during training. Once the model is trained, you want to evaluate
the model’s performance using the unregularized performance measure.
It is quite common for the cost function used during training to be
different from the performance measure used for testing. Apart
from regularization, another reason why they might be different is

that a good training cost function should have optimization-
friendly derivatives, while the performance measure used for test‐
ing should be as close as possible to the final objective. A good
example of this is a classifier trained using a cost function such as
the log loss (discussed in a moment) but evaluated using precision/
recall.
The hyperparameter α controls how much you want to regularize the model. If α = 0
then Ridge Regression is just Linear Regression. If α is very large, then all weights end

recall.
up very close to zero and the result is a flat line going through the data’s mean. Equa‐
tion 4-8  presents the Ridge Regression cost function.11
Equation 4-8. Ridge Regression cost functionJθ= MSE θ+α1
2∑i= 1nθi2
Note that the bias term θ0 is not regularized (the sum starts at i = 1, not 0). If we
define w as the vector of feature weights ( θ1 to θn), then the regularization term is
simply equal to ½( ∥ w ∥2)2, where ∥ w ∥2 represents the ℓ2 norm of the weight vector.12

2∑i= 1nθi2
For Gradient Descent, just add αw to the MSE gradient vector ( Equation 4-6 ).
It is important to scale the data (e.g., using a StandardScaler ) 
before performing Ridge Regression, as it is sensitive to the scale of
the input features. This is true of most regularized models.Regularized Linear Models | 137

13A square matrix full of 0s except for 1s on the main diagonal (top-left to bottom-right).Figure 4-17  shows several Ridge models trained on some linear data using different α
value. On the left, plain Ridge models are used, leading to linear predictions. On the
right, the data is first expanded using PolynomialFeatures(degree=10) , then it is
scaled using a StandardScaler , and finally the Ridge models are applied to the result‐
ing features: this is Polynomial Regression with Ridge regularization. Note how
increasing α leads to flatter (i.e., less extreme, more reasonable) predictions; this
reduces the model’s variance but increases its bias.
As with Linear Regression, we can perform Ridge Regression either by computing a 
closed-form equation or by performing Gradient Descent. The pros and cons are the
same. Equation 4-9  shows the closed-form solution (where A is the ( n + 1) × ( n + 1)
identity matrix13 except with a 0 in the top-left cell, corresponding to the bias term).
Figure 4-17. Ridge Regression

Equation 4-9. Ridge Regression closed-form solutionθ=XTX+αA−1 XT y
Here is how to perform Ridge Regression with Scikit-Learn using a closed-form solu‐
tion (a variant of Equation 4-9  using a matrix factorization technique by André-Louis
Cholesky):
>>> from sklearn.linear_model  import Ridge
>>> ridge_reg  = Ridge(alpha=1, solver="cholesky" )
>>> ridge_reg .fit(X, y)138 | Chapter 4: Training Models

14Alternatively you can use the Ridge  class with the "sag"  solver. Stochastic Average GD is a variant of SGD.
For more details, see the presentation “Minimizing Finite Sums with the Stochastic Average Gradient Algo‐
rithm”  by Mark Schmidt et al. from the University of British Columbia.>>> ridge_reg .predict([[1.5]])
array([[1.55071465]])
And using Stochastic Gradient Descent:14
>>> sgd_reg = SGDRegressor (penalty="l2")
>>> sgd_reg.fit(X, y.ravel())
>>> sgd_reg.predict([[1.5]])
array([1.47012588])
The penalty  hyperparameter sets the type of regularization term to use. Specifying
"l2"  indicates that you want SGD to add a regularization term to the cost function 
equal to half the square of the ℓ2 norm of the weight vector: this is simply Ridge
Regression.
Lasso Regression
Least Absolute Shrinkage and Selection Operator Regression  (simply called Lasso
Regression ) is another regularized version of Linear Regression: just like Ridge

Regression, it adds a regularization term to the cost function, but it uses the ℓ1 norm
of the weight vector instead of half the square of the ℓ2 norm (see Equation 4-10 ).
Equation 4-10. Lasso Regression cost functionJθ= MSE θ+α∑i= 1nθi
Figure 4-18  shows the same thing as Figure 4-17  but replaces Ridge models with
Lasso models and uses smaller α values.Regularized Linear Models | 139

Figure 4-18. Lasso Regression
An important characteristic of Lasso Regression is that it tends to completely elimi‐
nate the weights of the least important features (i.e., set them to zero). For example,
the dashed line in the right plot on Figure 4-18  (with α = 10-7) looks quadratic, almost
linear: all the weights for the high-degree polynomial features are equal to zero. In
other words, Lasso Regression automatically performs feature selection and outputs a
sparse model  (i.e., with few nonzero feature weights).
Y ou can get a sense of why this is the case by looking at Figure 4-19 : on the top-left
plot, the background contours (ellipses) represent an unregularized MSE cost func‐
tion ( α = 0), and the white circles show the Batch Gradient Descent path with that
cost function. The foreground contours (diamonds) represent the ℓ1 penalty, and the
triangles show the BGD path for this penalty only ( α → ∞ ). Notice how the path first

reaches θ1 = 0, then rolls down a gutter until it reaches θ2 = 0. On the top-right plot,
the contours represent the same cost function plus an ℓ1 penalty with α = 0.5. The
global minimum is on the θ2 = 0 axis. BGD first reaches θ2 = 0, then rolls down the
gutter until it reaches the global minimum. The two bottom plots show the same
thing but uses an ℓ2 penalty instead. The regularized minimum is closer to θ = 0 than
the unregularized minimum, but the weights do not get fully eliminated.140 | Chapter 4: Training Models

15Y ou can think of a subgradient vector at a nondifferentiable point as an intermediate vector between the gra‐
dient vectors around that point.
Figure 4-19. Lasso versus Ridge regularization
On the Lasso cost function, the BGD path tends to bounce across
the gutter toward the end. This is because the slope changes
abruptly at θ2 = 0. Y ou need to gradually reduce the learning rate in
order to actually converge to the global minimum.
The Lasso cost function is not differentiable at θi = 0 (for i = 1, 2, ⋯, n), but Gradient
Descent still works fine if you use a subgradient vector  g15 instead when any θi = 0.
Equation 4-11  shows a subgradient vector equation you can use for Gradient Descent
with the Lasso cost function.
Equation 4-11. Lasso Regression subgradient vectorgθ,J=∇θMSE θ+αsign θ1
sign θ2
⋮
sign θn   where  sign θi=−1 if  θi< 0
0 if  θi= 0
+1 if  θi> 0Regularized Linear Models | 141

Here is a small Scikit-Learn example using the Lasso  class. Note that you could
instead use an SGDRegressor(penalty="l1") .
>>> from sklearn.linear_model  import Lasso
>>> lasso_reg  = Lasso(alpha=0.1)
>>> lasso_reg .fit(X, y)
>>> lasso_reg .predict([[1.5]])
array([1.53788174])
Elastic Net
Elastic Net is a middle ground between Ridge Regression and Lasso Regression. The
regularization term is a simple mix of both Ridge and Lasso’s regularization terms,
and you can control the mix ratio r. When r = 0, Elastic Net is equivalent to Ridge
Regression, and when r = 1, it is equivalent to Lasso Regression (see Equation 4-12 ).
Equation 4-12. Elastic Net cost functionJθ= MSE θ+rα∑i= 1nθi+1 −r
2α∑i= 1nθi2
So when should you use plain Linear Regression (i.e., without any regularization),
Ridge, Lasso, or Elastic Net? It is almost always preferable to have at least a little bit of
regularization, so generally you should avoid plain Linear Regression. Ridge is a good

default, but if you suspect that only a few features are actually useful, you should pre‐
fer Lasso or Elastic Net since they tend to reduce the useless features’ weights down to
zero as we have discussed. In general, Elastic Net is preferred over Lasso since Lasso
may behave erratically when the number of features is greater than the number of
training instances or when several features are strongly correlated.
Here is a short example using Scikit-Learn’s ElasticNet  (l1_ratio  corresponds to
the mix ratio r):
>>> from sklearn.linear_model  import ElasticNet

>>> elastic_net  = ElasticNet (alpha=0.1, l1_ratio =0.5)
>>> elastic_net .fit(X, y)
>>> elastic_net .predict([[1.5]])
array([1.54333232])
Early Stopping
A very different way to regularize iterative learning algorithms such as Gradient
Descent is to stop training as soon as the validation error reaches a minimum. This is
called early stopping . Figure 4-20  shows a complex model (in this case a high-degree
Polynomial Regression model) being trained using Batch Gradient Descent. As the

epochs go by, the algorithm learns and its prediction error (RMSE) on the training set
naturally goes down, and so does its prediction error on the validation set. However,142 | Chapter 4: Training Models

after a while the validation error stops decreasing and actually starts to go back up.
This indicates that the model has started to overfit the training data. With early stop‐
ping you just stop training as soon as the validation error reaches the minimum. It is
such a simple and efficient regularization technique that Geoffrey Hinton called it a
“beautiful free lunch. ”
Figure 4-20. Early stopping regularization
With Stochastic and Mini-batch Gradient Descent, the curves are
not so smooth, and it may be hard to know whether you have
reached the minimum or not. One solution is to stop only after the
validation error has been above the minimum for some time (when
you are confident that the model will not do any better), then roll
back the model parameters to the point where the validation error
was at a minimum.
Here is a basic implementation of early stopping:
# prepare the data
poly_scaler  = Pipeline ([

        ("poly_features" , PolynomialFeatures (degree=90, include_bias =False)),
        ("std_scaler" , StandardScaler ())
    ])
X_train_poly_scaled  = poly_scaler .fit_transform (X_train)
X_val_poly_scaled  = poly_scaler .transform (X_val)
sgd_reg = SGDRegressor (max_iter =1, tol=-np.infty, warm_start =True,
                       penalty=None, learning_rate ="constant" , eta0=0.0005)Regularized Linear Models | 143

minimum_val_error  = float("inf")
best_epoch  = None
best_model  = None
for epoch in range(1000):
    sgd_reg.fit(X_train_poly_scaled , y_train)  # continues where it left off
    y_val_predict  = sgd_reg.predict(X_val_poly_scaled )
    val_error  = mean_squared_error (y_val, y_val_predict )
    if val_error  < minimum_val_error :
        minimum_val_error  = val_error
        best_epoch  = epoch
        best_model  = clone(sgd_reg)
Note that with warm_start=True , when the fit()  method is called, it just continues
training where it left off instead of restarting from scratch.
Logistic Regression
As we discussed in Chapter 1 , some regression algorithms can be used for classifica‐
tion as well (and vice versa). Logistic Regression  (also called Logit Regression ) is com‐
monly used to estimate the probability that an instance belongs to a particular class
(e.g., what is the probability that this email is spam?). If the estimated probability is

greater than 50%, then the model predicts that the instance belongs to that class
(called the positive class, labeled “1”), or else it predicts that it does not (i.e., it
belongs to the negative class, labeled “0”). This makes it a binary classifier.
Estimating Probabilities
So how does it work? Just like a Linear Regression model, a Logistic Regression
model computes a weighted sum of the input features (plus a bias term), but instead
of outputting the result directly like the Linear Regression model does, it outputs the
logistic  of this result (see Equation 4-13 ).

Equation 4-13. Logistic Regression model estimated probability (vectorized form)p=hθx=σxTθ
The logistic—noted σ(·)—is a sigmoid function  (i.e., S-shaped) that outputs a number
between 0 and 1. It is defined as shown in Equation 4-14  and Figure 4-21 .
Equation 4-14. Logistic functionσt=1
1 + exp −t144 | Chapter 4: Training Models

Figure 4-21. Logistic function
Once the Logistic Regression model has estimated the probability p = hθ(x) that an
instance x belongs to the positive class, it can make its prediction ŷ easily (see Equa‐
tion 4-15 ).
Equation 4-15. Logistic Regression model predictiony=0 if p< 0 . 5
1 if p≥ 0 . 5
Notice that σ(t) < 0.5 when t < 0, and σ(t) ≥ 0.5 when t ≥ 0, so a Logistic Regression
model predicts 1 if xT θ is positive, and 0 if it is negative.
The score t is often called the logit : this name comes from the fact
that the logit function, defined as logit( p) = log( p / (1 - p)), is the
inverse of the logistic function. Indeed, if you compute the logit of
the estimated probability p, you will find that the result is t. The
logit is also called the log-odds , since it is the log of the ratio
between the estimated probability for the positive class and the
estimated probability for the negative class.
Training and Cost Function
Good, now you know how a Logistic Regression model estimates probabilities and

makes predictions. But how is it trained? The objective of training is to set the param‐
eter vector θ so that the model estimates high probabilities for positive instances ( y =
1) and low probabilities for negative instances ( y = 0). This idea is captured by the
cost function shown in Equation 4-16  for a single training instance x.
Equation 4-16. Cost function of a single training instancecθ=−log p if y= 1
−log 1 −pif y= 0Logistic Regression | 145

This cost function makes sense because – log( t) grows very large when t approaches
0, so the cost will be large if the model estimates a probability close to 0 for a positive
instance, and it will also be very large if the model estimates a probability close to 1
for a negative instance. On the other hand, – log( t) is close to 0 when t is close to 1, so
the cost will be close to 0 if the estimated probability is close to 0 for a negative
instance or close to 1 for a positive instance, which is precisely what we want.
The cost function over the whole training set is simply the average cost over all train‐
ing instances. It can be written in a single expression (as you can verify easily), called 
the log loss , shown in Equation 4-17 .
Equation 4-17. Logistic Regression cost function (log loss)Jθ= −1
m∑i= 1myilogpi+1 −yilog1 −pi
The bad news is that there is no known closed-form equation to compute the value of
θ that minimizes this cost function (there is no equivalent of the Normal Equation).

But the good news is that this cost function is convex, so Gradient Descent (or any
other optimization algorithm) is guaranteed to find the global minimum (if the learn‐
ing rate is not too large and you wait long enough). The partial derivatives of the cost
function with regards to the jth model parameter θj is given by Equation 4-18 .
Equation 4-18. Logistic cost function partial derivatives∂
∂θjJθ=1
m∑
i= 1m
σθTxi−yixji
This equation looks very much like Equation 4-5 : for each instance it computes the

∂θjJθ=1
m∑
i= 1m
prediction error and multiplies it by the jth feature value, and then it computes the
average over all training instances. Once you have the gradient vector containing all
the partial derivatives you can use it in the Batch Gradient Descent algorithm. That’s
it: you now know how to train a Logistic Regression model. For Stochastic GD you
would of course just take one instance at a time, and for Mini-batch GD you would
use a mini-batch at a time.
Decision Boundaries

Let’s use the iris dataset to illustrate Logistic Regression. This is a famous dataset that
contains the sepal and petal length and width of 150 iris flowers of three different
species: Iris-Setosa, Iris-Versicolor, and Iris-Virginica (see Figure 4-22 ).146 | Chapter 4: Training Models

16Photos reproduced from the corresponding Wikipedia pages. Iris-Virginica photo by Frank Mayfield ( Crea‐
tive Commons BY-SA 2.0 ), Iris-Versicolor photo by D. Gordon E. Robertson ( Creative Commons BY-SA 3.0 ),
and Iris-Setosa photo is public domain.17
NumPy’s reshape()  function allows one dimension to be –1, which means “unspecified”: the value is inferred
from the length of the array and the remaining dimensions.
Figure 4-22. Flowers of three iris plant species16
Let’s try to build a classifier to detect the Iris-Virginica type based only on the petal
width feature. First let’s load the data:
>>> from sklearn import datasets
>>> iris = datasets .load_iris ()
>>> list(iris.keys())
['data', 'target', 'target_names', 'DESCR', 'feature_names', 'filename']
>>> X = iris["data"][:, 3:]  # petal width
>>> y = (iris["target" ] == 2).astype(np.int)  # 1 if Iris-Virginica, else 0
Now let’s train a Logistic Regression model:
from sklearn.linear_model  import LogisticRegression
log_reg = LogisticRegression ()

log_reg.fit(X, y)
Let’s look at the model’s estimated probabilities for flowers with petal widths varying
from 0 to 3 cm ( Figure 4-23 )17:
X_new = np.linspace (0, 3, 1000).reshape(-1, 1)
y_proba = log_reg.predict_proba (X_new)
plt.plot(X_new, y_proba[:, 1], "g-", label="Iris-Virginica" )Logistic Regression | 147

18It is the the set of points x such that θ0 + θ1x1 + θ2x2 = 0, which defines a straight line. plt.plot(X_new, y_proba[:, 0], "b--", label="Not Iris-Virginica" )
# + more Matplotlib code to make the image look pretty
Figure 4-23. Estimated probabilities and decision boundary
The petal width of Iris-Virginica flowers (represented by triangles) ranges from 1.4
cm to 2.5 cm, while the other iris flowers (represented by squares) generally have a
smaller petal width, ranging from 0.1 cm to 1.8 cm. Notice that there is a bit of over‐
lap. Above about 2 cm the classifier is highly confident that the flower is an Iris-
Virginica (it outputs a high probability to that class), while below 1 cm it is highly
confident that it is not an Iris-Virginica (high probability for the “Not Iris-Virginica”
class). In between these extremes, the classifier is unsure. However, if you ask it to
predict the class (using the predict()  method rather than the predict_proba()

method), it will return whichever class is the most likely. Therefore, there is a decision
boundary  at around 1.6 cm where both probabilities are equal to 50%: if the petal
width is higher than 1.6 cm, the classifier will predict that the flower is an Iris-
Virginica, or else it will predict that it is not (even if it is not very confident):
>>> log_reg.predict([[1.7], [1.5]])
array([1, 0])
Figure 4-24  shows the same dataset but this time displaying two features: petal width
and length. Once trained, the Logistic Regression classifier can estimate the probabil‐

ity that a new flower is an Iris-Virginica based on these two features. The dashed line
represents the points where the model estimates a 50% probability: this is the model’s
decision boundary. Note that it is a linear boundary.18 Each parallel line represents the
points where the model outputs a specific probability, from 15% (bottom left) to 90%
(top right). All the flowers beyond the top-right line have an over 90% chance of
being Iris-Virginica according to the model.148 | Chapter 4: Training Models

Figure 4-24. Linear decision boundary
Just like the other linear models, Logistic Regression models can be regularized using 
ℓ1 or ℓ2 penalties. Scitkit-Learn actually adds an ℓ2 penalty by default.
The hyperparameter controlling the regularization strength of a
Scikit-Learn LogisticRegression  model is not alpha  (as in other
linear models), but its inverse: C. The higher the value of C, the less
the model is regularized.
Softmax Regression
The Logistic Regression model can be generalized to support multiple classes directly,
without having to train and combine multiple binary classifiers (as discussed in
Chapter 3 ). This is called Softmax  Regression , or Multinomial Logistic Regression .
The idea is quite simple: when given an instance x, the Softmax Regression model
first computes a score sk(x) for each class k, then estimates the probability of each
class by applying the softmax  function  (also called the normalized exponential ) to the

scores. The equation to compute sk(x) should look familiar, as it is just like the equa‐
tion for Linear Regression prediction (see Equation 4-19 ).
Equation 4-19. Softmax  score for class kskx=xTθk
Note that each class has its own dedicated parameter vector θ(k). All these vectors are
typically stored as rows in a parameter matrix  Θ.
Once you have computed the score of every class for the instance x, you can estimate
the probability pk that the instance belongs to class k by running the scores through

the softmax function ( Equation 4-20 ): it computes the exponential of every score,Logistic Regression | 149

then normalizes them (dividing by the sum of all the exponentials). The scores are
generally called logits or log-odds (although they are actually unnormalized log-
odds).
Equation 4-20. Softmax  functionpk=σsxk=exp skx
∑j= 1Kexp sjx •
K is the number of classes.•
s(x) is a vector containing the scores of each class for the instance x.•
σ(s(x))k is the estimated probability that the instance x belongs to class k given
the scores of each class for that instance.
Just like the Logistic Regression classifier, the Softmax Regression classifier predicts
the class with the highest estimated probability (which is simply the class with the
highest score), as shown in Equation 4-21 .
Equation 4-21. Softmax  Regression classifier  predictiony= argmax
kσsxk= argmax
kskx= argmax
kθkTx•
The argmax  operator returns the value of a variable that maximizes a function. In
this equation, it returns the value of k that maximizes the estimated probability
σ(s(x))k.

kθkTx•
σ(s(x))k.
The Softmax Regression classifier predicts only one class at a time
(i.e., it is multiclass, not multioutput) so it should be used only with
mutually exclusive classes such as different types of plants. Y ou
cannot use it to recognize multiple people in one picture.
Now that you know how the model estimates probabilities and makes predictions,
let’s take a look at training. The objective is to have a model that estimates a high
probability for the target class (and consequently a low probability for the other

classes). Minimizing the cost function shown in Equation 4-22 , called the cross
entropy , should lead to this objective because it penalizes the model when it estimates
a low probability for a target class. Cross entropy is frequently used to measure how150 | Chapter 4: Training Models

well a set of estimated class probabilities match the target classes (we will use it again
several times in the following chapters).
Equation 4-22. Cross entropy cost functionJΘ= −1
m∑i= 1m∑k= 1Kykilogpki
•yki
 is the target probability that the ith instance belongs to class k. In general, it is
either equal to 1 or 0, depending on whether the instance belongs to the class or
not.
Notice that when there are just two classes ( K = 2), this cost function is equivalent to
the Logistic Regression’s cost function (log loss; see Equation 4-17 ).
Cross Entropy
Cross entropy originated from information theory. Suppose you want to efficiently
transmit information about the weather every day. If there are eight options (sunny,
rainy, etc.), you could encode each option using 3 bits since 23 = 8. However, if you
think it will be sunny almost every day, it would be much more efficient to code
“sunny” on just one bit (0) and the other seven options on 4 bits (starting with a 1).

Cross entropy measures the average number of bits you actually send per option. If
your assumption about the weather is perfect, cross entropy will just be equal to the
entropy of the weather itself (i.e., its intrinsic unpredictability). But if your assump‐
tions are wrong (e.g., if it rains often), cross entropy will be greater by an amount 
called the Kullback–Leibler divergence .
The cross entropy between two probability distributions p and q is defined asHp,q= − ∑xpxlogqx

 (at least when the distributions are discrete). For more
details, check out this video .
The gradient vector of this cost function with regards to θ(k) is given by Equation
4-23 :
Equation 4-23. Cross entropy gradient vector for class k∇
θkJΘ=1
m∑
i= 1m
pki−ykixi
Now you can compute the gradient vector for every class, then use Gradient Descent
(or any other optimization algorithm) to find the parameter matrix Θ that minimizes
the cost function.Logistic Regression | 151

Let’s use Softmax Regression to classify the iris flowers into all three classes. Scikit-
Learn’s LogisticRegression  uses one-versus-all by default when you train it on more
than two classes, but you can set the multi_class  hyperparameter to "multinomial"
to switch it to Softmax Regression instead. Y ou must also specify a solver that sup‐
ports Softmax Regression, such as the "lbfgs"  solver (see Scikit-Learn’s documenta‐
tion for more details). It also applies ℓ2 regularization by default, which you can
control using the hyperparameter C.
X = iris["data"][:, (2, 3)]  # petal length, petal width
y = iris["target" ]
softmax_reg  = LogisticRegression (multi_class ="multinomial" ,solver="lbfgs", C=10)
softmax_reg .fit(X, y)
So the next time you find an iris with 5 cm long and 2 cm wide petals, you can ask
your model to tell you what type of iris it is, and it will answer Iris-Virginica (class 2)
with 94.2% probability (or Iris-Versicolor with 5.8% probability):
>>> softmax_reg .predict([[5, 2]])
array([2])

>>> softmax_reg .predict_proba ([[5, 2]])
array([[6.38014896e-07, 5.74929995e-02, 9.42506362e-01]])
Figure 4-25  shows the resulting decision boundaries, represented by the background
colors. Notice that the decision boundaries between any two classes are linear. The
figure also shows the probabilities for the Iris-Versicolor class, represented by the
curved lines (e.g., the line labeled with 0.450 represents the 45% probability bound‐

ary). Notice that the model can predict a class that has an estimated probability below
50%. For example, at the point where all decision boundaries meet, all classes have an
equal estimated probability of 33%.
Figure 4-25. Softmax  Regression decision boundaries152 | Chapter 4: Training Models

What Linear Regression training algorithm can you use if you have a training set
with millions of features?2.
Suppose the features in your training set have very different scales. What algo‐
rithms might suffer from this, and how? What can you do about it?3.
Can Gradient Descent get stuck in a local minimum when training a Logistic
Regression model?4.
Do all Gradient Descent algorithms lead to the same model provided you let
them run long enough?5.
Suppose you use Batch Gradient Descent and you plot the validation error at
every epoch. If you notice that the validation error consistently goes up, what is
likely going on? How can you fix this?6.
Is it a good idea to stop Mini-batch Gradient Descent immediately when the vali‐
dation error goes up?7.
Which Gradient Descent algorithm (among those we discussed) will reach the
vicinity of the optimal solution the fastest? Which will actually converge? How
can you make the others converge as well?8.

Suppose you are using Polynomial Regression. Y ou plot the learning curves and
you notice that there is a large gap between the training error and the validation
error. What is happening? What are three ways to solve this?9.
Suppose you are using Ridge Regression and you notice that the training error
and the validation error are almost equal and fairly high. Would you say that the
model suffers from high bias or high variance? Should you increase the regulari‐
zation hyperparameter α or reduce it?10.

Why would you want to use:•
Ridge Regression instead of plain Linear Regression (i.e., without any regulari‐
zation)?•
Lasso instead of Ridge Regression?•
Elastic Net instead of Lasso?11.
Suppose you want to classify pictures as outdoor/indoor and daytime/nighttime.
Should you implement two Logistic Regression classifiers or one Softmax Regres‐
sion classifier?12.
Implement Batch Gradient Descent with early stopping for Softmax Regression 
(without using Scikit-Learn).

zation)?•
Solutions to these exercises are available in ???.Exercises | 153

CHAPTER 5
Support Vector Machines
release of these titles. The following will be Chapter 5 in the final
A Support Vector Machine  (SVM) is a very powerful and versatile Machine Learning
model, capable of performing linear or nonlinear classification, regression, and even
outlier detection. It is one of the most popular models in Machine Learning, and any‐
one interested in Machine Learning should have it in their toolbox. SVMs are partic‐
ularly well suited for classification of complex but small- or medium-sized datasets.
This chapter will explain the core concepts of SVMs, how to use them, and how they
work.
Linear SVM Classification
The fundamental idea behind SVMs is best explained with some pictures. Figure 5-1

work.
shows part of the iris dataset that was introduced at the end of Chapter 4 . The two
classes can clearly be separated easily with a straight line (they are linearly separable ).
The left plot shows the decision boundaries of three possible linear classifiers. The
model whose decision boundary is represented by the dashed line is so bad that it
does not even separate the classes properly. The other two models work perfectly on
this training set, but their decision boundaries come so close to the instances that

these models will probably not perform as well on new instances. In contrast, the
solid line in the plot on the right represents the decision boundary of an SVM classi‐
fier; this line not only separates the two classes but also stays as far away from the
closest training instances as possible. Y ou can think of an SVM classifier as fitting the155

widest possible street (represented by the parallel dashed lines) between the classes.
This is called large margin classification .
Figure 5-1. Large margin classification
Notice that adding more training instances “off the street” will not affect the decision
boundary at all: it is fully determined (or “supported”) by the instances located on the
edge of the street. These instances are called the support vectors  (they are circled in
Figure 5-1 ).
SVMs are sensitive to the feature scales, as you can see in
Figure 5-2 : on the left plot, the vertical scale is much larger than the
horizontal scale, so the widest possible street is close to horizontal.
After feature scaling (e.g., using Scikit-Learn’s StandardScaler ), 
the decision boundary looks much better (on the right plot).
Figure 5-2. Sensitivity to feature scales
Soft Margin Classification
If we strictly impose that all instances be off the street and on the right side, this is

called hard margin classification . There are two main issues with hard margin classifi‐
cation. First, it only works if the data is linearly separable, and second it is quite sensi‐
tive to outliers. Figure 5-3  shows the iris dataset with just one additional outlier: on
the left, it is impossible to find a hard margin, and on the right the decision boundary
ends up very different from the one we saw in Figure 5-1  without the outlier, and it

will probably not generalize as well.156 | Chapter 5: Support Vector Machines

Figure 5-3. Hard margin sensitivity to outliers
To avoid these issues it is preferable to use a more flexible model. The objective is to
find a good balance between keeping the street as large as possible and limiting the
margin violations  (i.e., instances that end up in the middle of the street or even on the
wrong side). This is called soft margin classification .
In Scikit-Learn’s SVM classes, you can control this balance using the C hyperparame‐
ter: a smaller C value leads to a wider street but more margin violations. Figure 5-4
shows the decision boundaries and margins of two soft margin SVM classifiers on a
nonlinearly separable dataset. On the left, using a low C value the margin is quite
large, but many instances end up on the street. On the right, using a high C value the
classifier makes fewer margin violations but ends up with a smaller margin. However,
it seems likely that the first classifier will generalize better: in fact even on this train‐

ing set it makes fewer prediction errors, since most of the margin violations are
actually on the correct side of the decision boundary.
Figure 5-4. Large margin (left)  versus fewer margin violations (right)
If your SVM model is overfitting, you can try regularizing it by
reducing C.
The following Scikit-Learn code loads the iris dataset, scales the features, and then
trains a linear SVM model (using the LinearSVC  class with C = 1 and the hinge loss
function, described shortly) to detect Iris-Virginica flowers. The resulting model is

represented on the left of Figure 5-4 .Linear SVM Classification  | 157

from sklearn import datasets
from sklearn.svm  import LinearSVC
iris = datasets .load_iris ()
y = (iris["target" ] == 2).astype(np.float64)  # Iris-Virginica
svm_clf = Pipeline ([
        ("scaler" , StandardScaler ()),
        ("linear_svc" , LinearSVC (C=1, loss="hinge")),
    ])
svm_clf.fit(X, y)
Then, as usual, you can use the model to make predictions:
>>> svm_clf.predict([[5.5, 1.7]])
array([1.])
Unlike Logistic Regression classifiers, SVM classifiers do not out‐
put probabilities for each class.
Alternatively, you could use the SVC class, using SVC(kernel="linear", C=1) , but it
is much slower, especially with large training sets, so it is not recommended. Another
option is to use the SGDClassifier  class, with SGDClassifier(loss="hinge",
alpha=1/(m*C)) . This applies regular Stochastic Gradient Descent (see Chapter 4 ) to

train a linear SVM classifier. It does not converge as fast as the LinearSVC  class, but it
can be useful to handle huge datasets that do not fit in memory (out-of-core train‐
ing), or to handle online classification tasks.
The LinearSVC  class regularizes the bias term, so you should center
the training set first by subtracting its mean. This is automatic if
you scale the data using the StandardScaler . Moreover, make sure
you set the loss  hyperparameter to "hinge" , as it is not the default

value. Finally, for better performance you should set the dual
hyperparameter to False , unless there are more features than
training instances (we will discuss duality later in the chapter).158 | Chapter 5: Support Vector Machines

Nonlinear SVM Classification
Although linear SVM classifiers are efficient and work surprisingly well in many
cases, many datasets are not even close to being linearly separable. One approach to
handling nonlinear datasets is to add more features, such as polynomial features (as
you did in Chapter 4 ); in some cases this can result in a linearly separable dataset.
Consider the left plot in Figure 5-5 : it represents a simple dataset with just one feature
x1. This dataset is not linearly separable, as you can see. But if you add a second fea‐
ture x2 = (x1)2, the resulting 2D dataset is perfectly linearly separable.
Figure 5-5. Adding features to make a dataset linearly separable
To implement this idea using Scikit-Learn, you can create a Pipeline  containing a
PolynomialFeatures  transformer (discussed in “Polynomial Regression” on page
130), followed by a StandardScaler  and a LinearSVC . Let’s test this on the moons
dataset: this is a toy dataset for binary classification in which the data points are sha‐

ped as two interleaving half circles (see Figure 5-6 ). Y ou can generate this dataset
using the make_moons()  function:
from sklearn.datasets  import make_moons
from sklearn.preprocessing  import PolynomialFeatures
polynomial_svm_clf  = Pipeline ([
        ("poly_features" , PolynomialFeatures (degree=3)),
        ("svm_clf" , LinearSVC (C=10, loss="hinge"))
    ])
polynomial_svm_clf .fit(X, y)Nonlinear SVM Classification  | 159

Figure 5-6. Linear SVM classifier  using polynomial features
Polynomial Kernel
Adding polynomial features is simple to implement and can work great with all sorts
of Machine Learning algorithms (not just SVMs), but at a low polynomial degree it
cannot deal with very complex datasets, and with a high polynomial degree it creates
a huge number of features, making the model too slow.
Fortunately, when using SVMs you can apply an almost miraculous mathematical
technique called the kernel trick  (it is explained in a moment). It makes it possible to
get the same result as if you added many polynomial features, even with very high-
degree polynomials, without actually having to add them. So there is no combinato‐
rial explosion of the number of features since you don’t actually add any features. This
trick is implemented by the SVC class. Let’s test it on the moons dataset:
from sklearn.svm  import SVC
poly_kernel_svm_clf  = Pipeline ([

        ("svm_clf" , SVC(kernel="poly", degree=3, coef0=1, C=5))
    ])
poly_kernel_svm_clf .fit(X, y)
This code trains an SVM classifier using a 3rd-degree polynomial kernel. It is repre‐
sented on the left of Figure 5-7 . On the right is another SVM classifier using a 10th-
degree polynomial kernel. Obviously, if your model is overfitting, you might want to160 | Chapter 5: Support Vector Machines

reduce the polynomial degree. Conversely, if it is underfitting, you can try increasing
it. The hyperparameter coef0  controls how much the model is influenced by high-
degree polynomials versus low-degree polynomials.
Figure 5-7. SVM classifiers  with a polynomial kernel
A common approach to find the right hyperparameter values is to
use grid search (see Chapter 2 ). It is often faster to first do a very
coarse grid search, then a finer grid search around the best values
found. Having a good sense of what each hyperparameter actually
does can also help you search in the right part of the hyperparame‐
ter space.
Adding Similarity Features
Another technique to tackle nonlinear problems is to add features computed using a
similarity function  that measures how much each instance resembles a particular
landmark . For example, let’s take the one-dimensional dataset discussed earlier and
add two landmarks to it at x1 = –2 and x1 = 1 (see the left plot in Figure 5-8 ). Next,

ter space.
let’s define the similarity function to be the Gaussian Radial Basis Function  (RBF )
with γ = 0.3 (see Equation 5-1 ).
Equation 5-1. Gaussian RBFϕγx, ℓ= exp −γ∥x− ℓ∥2
It is a bell-shaped function varying from 0 (very far away from the landmark) to 1 (at
the landmark). Now we are ready to compute the new features. For example, let’s look
at the instance x1 = –1: it is located at a distance of 1 from the first landmark, and 2
from the second landmark. Therefore its new features are x2 = exp (–0.3 × 12) ≈ 0.74

and x3 = exp (–0.3 × 22) ≈ 0.30. The plot on the right of Figure 5-8  shows the trans‐
formed dataset (dropping the original features). As you can see, it is now linearly
separable.Nonlinear SVM Classification  | 161

Figure 5-8. Similarity features using the Gaussian RBF
Y ou may wonder how to select the landmarks. The simplest approach is to create a
landmark at the location of each and every instance in the dataset. This creates many
dimensions and thus increases the chances that the transformed training set will be
linearly separable. The downside is that a training set with m instances and n features
gets transformed into a training set with m instances and m features (assuming you
drop the original features). If your training set is very large, you end up with an
equally large number of features.
Gaussian RBF Kernel
Just like the polynomial features method, the similarity features method can be useful
with any Machine Learning algorithm, but it may be computationally expensive to
compute all the additional features, especially on large training sets. However, once
again the kernel trick does its SVM magic: it makes it possible to obtain a similar

result as if you had added many similarity features, without actually having to add
them. Let’s try the Gaussian RBF kernel using the SVC class:
rbf_kernel_svm_clf  = Pipeline ([
        ("svm_clf" , SVC(kernel="rbf", gamma=5, C=0.001))
    ])
rbf_kernel_svm_clf .fit(X, y)
This model is represented on the bottom left of Figure 5-9 . The other plots show
models trained with different values of hyperparameters gamma  (γ) and C. Increasing

    ])
gamma  makes the bell-shape curve narrower (see the left plot of Figure 5-8 ), and as a
result each instance’s range of influence is smaller: the decision boundary ends up
being more irregular, wiggling around individual instances. Conversely, a small gamma  
value makes the bell-shaped curve wider, so instances have a larger range of influ‐
ence, and the decision boundary ends up smoother. So γ acts like a regularization
hyperparameter: if your model is overfitting, you should reduce it, and if it is under‐

fitting, you should increase it (similar to the C hyperparameter).162 | Chapter 5: Support Vector Machines

1“ A Dual Coordinate Descent Method for Large-scale Linear SVM, ” Lin et al. (2008).
Figure 5-9. SVM classifiers  using an RBF kernel
Other kernels exist but are used much more rarely. For example, some kernels are
specialized for specific data structures. String kernels  are sometimes used when classi‐
fying text documents or DNA sequences (e.g., using the string subsequence kernel  or
kernels based on the Levenshtein distance ).
With so many kernels to choose from, how can you decide which
one to use? As a rule of thumb, you should always try the linear
kernel first (remember that LinearSVC  is much faster than SVC(ker
nel="linear") ), especially if the training set is very large or if it
has plenty of features. If the training set is not too large, you should
try the Gaussian RBF kernel as well; it works well in most cases.
Then if you have spare time and computing power, you can also
experiment with a few other kernels using cross-validation and grid

search, especially if there are kernels specialized for your training
set’s data structure.
The LinearSVC  class is based on the liblinear  library, which implements an optimized
algorithm  for linear SVMs.1 It does not support the kernel trick, but it scales almostNonlinear SVM Classification  | 163

2“Sequential Minimal Optimization (SMO), ” J. Platt (1998).linearly with the number of training instances and the number of features: its training
time complexity is roughly O(m × n).
The algorithm takes longer if you require a very high precision. This is controlled by
the tolerance hyperparameter ϵ (called tol in Scikit-Learn). In most classification
tasks, the default tolerance is fine.
The SVC class is based on the libsvm  library, which implements an algorithm  that sup‐
ports the kernel trick.2 The training time complexity is usually between O(m2 × n)
and O(m3 × n). Unfortunately, this means that it gets dreadfully slow when the num‐
ber of training instances gets large (e.g., hundreds of thousands of instances). This
algorithm is perfect for complex but small or medium training sets. However, it scales
well with the number of features, especially with sparse features  (i.e., when each
instance has few nonzero features). In this case, the algorithm scales roughly with the

average number of nonzero features per instance. Table 5-1  compares Scikit-Learn’s
SVM classification classes.Table 5-1. Comparison of Scikit-Learn classes for SVM classification ClassTime complexityOut-of-core supportScaling requiredKernel trickLinearSVCO(m × n) NoYesNoSGDClassifierO(m × n) YesYesNoSVCO(m² × n) to O( m³ × n) NoYesYes
SVM Regression
As we mentioned earlier, the SVM algorithm is quite versatile: not only does it sup‐

port linear and nonlinear classification, but it also supports linear and nonlinear
regression. The trick is to reverse the objective: instead of trying to fit the largest pos‐
sible street between two classes while limiting margin violations, SVM Regression
tries to fit as many instances as possible on the street while limiting margin violations
(i.e., instances off the street). The width of the street is controlled by a hyperparame‐
ter ϵ. Figure 5-10  shows two linear SVM Regression models trained on some random

linear data, one with a large margin ( ϵ = 1.5) and the other with a small margin ( ϵ =
0.5).164 | Chapter 5: Support Vector Machines

Figure 5-10. SVM Regression
Adding more training instances within the margin does not affect the model’s predic‐
tions; thus, the model is said to be ϵ-insensitive .
Y ou can use Scikit-Learn’s LinearSVR  class to perform linear SVM Regression. The
following code produces the model represented on the left of Figure 5-10  (the train‐
ing data should be scaled and centered first):
from sklearn.svm  import LinearSVR
svm_reg = LinearSVR (epsilon=1.5)
svm_reg.fit(X, y)
To tackle nonlinear regression tasks, you can use a kernelized SVM model. For exam‐
ple, Figure 5-11  shows SVM Regression on a random quadratic training set, using a
2nd-degree polynomial kernel. There is little regularization on the left plot (i.e., a large
C value), and much more regularization on the right plot (i.e., a small C value).
Figure 5-11. SVM regression using a 2nd-degree polynomial kernelSVM Regression | 165

The following code produces the model represented on the left of Figure 5-11  using
Scikit-Learn’s SVR class (which supports the kernel trick). The SVR class is the regres‐
sion equivalent of the SVC class, and the LinearSVR  class is the regression equivalent
of the LinearSVC  class. The LinearSVR  class scales linearly with the size of the train‐
ing set (just like the LinearSVC  class), while the SVR class gets much too slow when
the training set grows large (just like the SVC class).
from sklearn.svm  import SVR
svm_poly_reg  = SVR(kernel="poly", degree=2, C=100, epsilon=0.1)
svm_poly_reg .fit(X, y)
SVMs can also be used for outlier detection; see Scikit-Learn’s doc‐
umentation for more details.
Under the Hood
This section explains how SVMs make predictions and how their training algorithms
work, starting with linear SVM classifiers. Y ou can safely skip it and go straight to the
exercises at the end of this chapter if you are just getting started with Machine Learn‐

ing, and come back later when you want to get a deeper understanding of SVMs.
First, a word about notations: in Chapter 4  we used the convention of putting all the 
model parameters in one vector θ, including the bias term θ0 and the input feature
weights θ1 to θn, and adding a bias input x0 = 1 to all instances. In this chapter, we will
use a different convention, which is more convenient (and more common) when you
are dealing with SVMs: the bias term will be called b and the feature weights vector

will be called w. No bias feature will be added to the input feature vectors.
Decision Function and Predictions
The linear SVM classifier model predicts the class of a new instance x by simply com‐
puting the decision function wT x + b = w1 x1 + ⋯ + wn xn + b: if the result is positive,
the predicted class ŷ is the positive class (1), or else it is the negative class (0); see
Equation 5-2 .
Equation 5-2. Linear SVM classifier  predictiony=0 if wTx+b< 0,
1 if wTx+b≥ 0166 | Chapter 5: Support Vector Machines

3More generally, when there are n features, the decision function is an n-dimensional hyperplane , and the deci‐
sion boundary is an ( n – 1)-dimensional hyperplane.Figure 5-12  shows the decision function that corresponds to the model on the left of
Figure 5-4 : it is a two-dimensional plane since this dataset has two features (petal
width and petal length). The decision boundary is the set of points where the decision
function is equal to 0: it is the intersection of two planes, which is a straight line (rep‐
resented by the thick solid line).3
Figure 5-12. Decision function for the iris dataset
The dashed lines represent the points where the decision function is equal to 1 or –1:
they are parallel and at equal distance to the decision boundary, forming a margin
around it. Training a linear SVM classifier means finding the value of w and b that
make this margin as wide as possible while avoiding margin violations (hard margin)
or limiting them (soft margin).
Training Objective

Consider the slope of the decision function: it is equal to the norm of the weight vec‐
tor, ∥ w ∥. If we divide this slope by 2, the points where the decision function is equal
to ±1 are going to be twice as far away from the decision boundary. In other words,
dividing the slope by 2 will multiply the margin by 2. Perhaps this is easier to visual‐
ize in 2D in Figure 5-13 . The smaller the weight vector w, the larger the margin.Under the Hood | 167

4Zeta ( ζ) is the 6th letter of the Greek alphabet.
Figure 5-13. A smaller weight vector results in a larger margin
So we want to minimize ∥ w ∥ to get a large margin. However, if we also want to avoid
any margin violation (hard margin), then we need the decision function to be greater
than 1 for all positive training instances, and lower than –1 for negative training
instances. If we define t(i) = –1 for negative instances (if y(i) = 0) and t(i) = 1 for positive
instances (if y(i) = 1), then we can express this constraint as t(i)(wT x(i) + b) ≥ 1 for all
instances.
We can therefore express the hard margin linear SVM classifier objective as the con‐
strained optimization  problem in Equation 5-3 .
Equation 5-3. Hard margin linear SVM classifier  objectiveminimizew,b1
2wTw
subject to tiwTxi+b≥ 1 for i= 1, 2,⋯,m
We are minimizing 1
2wT w, which is equal to 1
2∥ w ∥2, rather than
minimizing ∥ w ∥. Indeed, 1
2∥ w ∥2 has a nice and simple derivative

2wTw
(it is just w) while ∥ w ∥ is not differentiable at w = 0. Optimization
algorithms work much better on differentiable functions.
To get the soft margin objective, we need to introduce a slack variable  ζ(i) ≥ 0 for each
instance:4 ζ(i) measures how much the ith instance is allowed to violate the margin. We
now have two conflicting objectives: making the slack variables as small as possible to
reduce the margin violations, and making 1
2wT w as small as possible to increase the

margin. This is where the C hyperparameter comes in: it allows us to define the trade‐168 | Chapter 5: Support Vector Machines

5To learn more about Quadratic Programming, you can start by reading Stephen Boyd and Lieven Vanden‐
berghe, Convex Optimization  (Cambridge, UK: Cambridge University Press, 2004) or watch Richard Brown’s
series of video lectures .off between these two objectives. This gives us the constrained optimization problem
in Equation 5-4 .
Equation 5-4. Soft margin linear SVM classifier  objectiveminimizew,b,ζ1
2wTw+C∑
i= 1m
ζi
subject to tiwTxi+b≥ 1 − ζiand ζi≥ 0 for i= 1, 2,⋯,m
Quadratic Programming
The hard margin and soft margin problems are both convex quadratic optimization
problems with linear constraints. Such problems are known as Quadratic Program‐
ming  (QP) problems. Many off-the-shelf solvers are available to solve QP problems
using a variety of techniques that are outside the scope of this book.5 The general
problem formulation is given by Equation 5-5 .
Equation 5-5. Quadratic Programming problemMinimize
p1
2pTHp + fTp
subject to Ap≤b
wherepis an np‐dimensional vector ( np= number of parameters),

p1
2pTHp + fTp
His an np×npmatrix,
fis an np‐dimensional vector,
Ais an nc×npmatrix ( nc= number of constraints),
bis an nc‐dimensional vector.
Note that the expression A p ≤ b actually defines nc constraints: pT a(i) ≤ b(i) for i = 1,
2, ⋯, nc, where a(i) is the vector containing the elements of the ith row of A and b(i) is
the ith element of b.
Y ou can easily verify that if you set the QP parameters in the following way, you get
the hard margin linear SVM classifier objective:•
np = n + 1, where n is the number of features (the +1 is for the bias term).Under the Hood | 169

6The objective function is convex, and the inequality constraints are continuously differentiable and convex
functions.•nc = m, where m is the number of training instances. •
H is the np × np identity matrix, except with a zero in the top-left cell (to ignore
the bias term).•
f = 0, an np-dimensional vector full of 0s.•
b = –1, an nc-dimensional vector full of –1s.•
a(i) = –t(i) x˙ (i), where x˙ (i) is equal to x(i) with an extra bias feature x˙ 0 = 1.
So one way to train a hard margin linear SVM classifier is just to use an off-the-shelf
QP solver by passing it the preceding parameters. The resulting vector p will contain
the bias term b = p0 and the feature weights wi = pi for i = 1, 2, ⋯, n. Similarly, you
can use a QP solver to solve the soft margin problem (see the exercises at the end of
the chapter).
However, to use the kernel trick we are going to look at a different constrained opti‐
mization problem.
The Dual Problem

Given a constrained optimization problem, known as the primal problem , it is possi‐
ble to express a different but closely related problem, called its dual problem . The sol‐
ution to the dual problem typically gives a lower bound to the solution of the primal
problem, but under some conditions it can even have the same solutions as the pri‐
mal problem. Luckily, the SVM problem happens to meet these conditions,6 so you
can choose to solve the primal problem or the dual problem; both will have the same

solution. Equation 5-6  shows the dual form of the linear SVM objective (if you are
interested in knowing how to derive the dual problem from the primal problem,
see ???).
Equation 5-6. Dual form of the linear SVM objectiveminimizeα1
2∑
i= 1m
∑
j= 1m
αiαjtitjxiTxj−∑
i= 1m
αi
subject to αi≥ 0 for i= 1, 2,⋯,m170 | Chapter 5: Support Vector Machines

7As explained in Chapter 4 , the dot product of two vectors a and b is normally noted a · b. However, in
Machine Learning, vectors are frequently represented as column vectors (i.e., single-column matrices), so the
dot product is achieved by computing aTb. To remain consistent with the rest of the book, we will use this
notation here, ignoring the fact that this technically results in a single-cell matrix rather than a scalar value.Once you find the vector α that minimizes this equation (using a QP solver), you can
compute w and b that minimize the primal problem by using Equation 5-7 .
Equation 5-7. From the dual solution to the primal solutionw=∑
i= 1m
αitixi
b=1
ns∑
i= 1
αi> 0m
ti−wTxi
The dual problem is faster to solve than the primal when the number of training
instances is smaller than the number of features. More importantly, it makes the ker‐
nel trick possible, while the primal does not. So what is this kernel trick anyway?
Kernelized SVM

i= 1m
αitixi
b=1
ns∑
i= 1
αi> 0m
ti−wTxi
Suppose you want to apply a 2nd-degree polynomial transformation to a two-
dimensional training set (such as the moons training set), then train a linear SVM
classifier on the transformed training set. Equation 5-8  shows the 2nd-degree polyno‐
mial mapping function ϕ that you want to apply.
Equation 5-8. Second-degree polynomial mappingϕx=ϕx1
x2=x12
2x1x2
x22
Notice that the transformed vector is three-dimensional instead of two-dimensional.
Now let’s look at what happens to a couple of two-dimensional vectors, a and b, if we

x2=x12
2x1x2
x22
apply this 2nd-degree polynomial mapping and then compute the dot product7 of the
transformed vectors (See Equation 5-9 ).Under the Hood | 171

Equation 5-9. Kernel trick for a 2nd-degree polynomial mappingϕaTϕb =a12
2a1a2
a22Tb12
2b1b2
b22=a12b12+ 2a1b1a2b2+a22b22
=a1b1+a2b22=a1
a2Tb1
b22
=aTb2
How about that? The dot product of the transformed vectors is equal to the square of
the dot product of the original vectors: ϕ(a)T ϕ(b) = ( aT b)2.
Now here is the key insight: if you apply the transformation ϕ to all training instan‐
ces, then the dual problem (see Equation 5-6 ) will contain the dot product ϕ(x(i))T
ϕ(x(j)). But if ϕ is the 2nd-degree polynomial transformation defined in Equation 5-8 ,
then you can replace this dot product of transformed vectors simply by xiTxj2
. So
you don’t actually need to transform the training instances at all: just replace the dot
product by its square in Equation 5-6 . The result will be strictly the same as if you
went through the trouble of actually transforming the training set then fitting a linear
SVM algorithm, but this trick makes the whole process much more computationally

. So
efficient. This is the essence of the kernel trick.
The function K(a, b) = ( aT b)2 is called a 2nd-degree polynomial kernel . In Machine
Learning, a kernel  is a function capable of computing the dot product ϕ(a)T ϕ(b)
based only on the original vectors a and b, without having to compute (or even to
know about) the transformation ϕ. Equation 5-10  lists some of the most commonly
used kernels.
Equation 5-10. Common kernelsLinear: Ka,b=aTb
Polynomial: Ka,b=γaTb+rd
Gaussian RBF: Ka,b= exp −γ∥a−b∥2
Sigmoid: Ka,b= tanh γaTb+r172 | Chapter 5: Support Vector Machines

Mercer’s Theorem
According to Mercer’s theorem , if a function K(a, b) respects a few mathematical con‐
ditions called Mercer’s conditions  (K must be continuous, symmetric in its arguments
so K(a, b) = K(b, a), etc.), then there exists a function ϕ that maps a and b into
another space (possibly with much higher dimensions) such that K(a, b) = ϕ(a)T ϕ(b).
So you can use K as a kernel since you know ϕ exists, even if you don’t know what ϕ
is. In the case of the Gaussian RBF kernel, it can be shown that ϕ actually maps each
training instance to an infinite-dimensional space, so it’s a good thing you don’t need
to actually perform the mapping!
Note that some frequently used kernels (such as the Sigmoid kernel) don’t respect all
of Mercer’s conditions, yet they generally work well in practice.
There is still one loose end we must tie. Equation 5-7  shows how to go from the dual
solution to the primal solution in the case of a linear SVM classifier, but if you apply

the kernel trick you end up with equations that include ϕ(x(i)). In fact, w must have
the same number of dimensions as ϕ(x(i)), which may be huge or even infinite, so you
can’t compute it. But how can you make predictions without knowing w? Well, the
good news is that you can plug in the formula for w from Equation 5-7  into the deci‐
sion function for a new instance x(n), and you get an equation with only dot products
between input vectors. This makes it possible to use the kernel trick, once again
(Equation 5-11 ).

Equation 5-11. Making predictions with a kernelized SVMhw,bϕxn=wTϕxn+b=∑
i= 1m
αitiϕxiT
ϕxn+b
=∑
i= 1m
αitiϕxiTϕxn+b
=∑
i= 1
αi> 0m
αitiKxi,xn+b
Note that since α(i) ≠ 0 only for support vectors, making predictions involves comput‐
ing the dot product of the new input vector x(n) with only the support vectors, not all
the training instances. Of course, you also need to compute the bias term b, using the
same trick ( Equation 5-12 ).Under the Hood | 173

Equation 5-12. Computing the bias term using the kernel trickb=1
ns∑
i= 1
αi> 0m
ti−wTϕxi=1
ns∑
i= 1
αi> 0m
ti−∑
j= 1m
αjtjϕxjT
ϕxi
=1
ns∑
i= 1
αi> 0m
ti−∑
j= 1
αj> 0m
αjtjKxi,xj
If you are starting to get a headache, it’s perfectly normal: it’s an unfortunate side
effect of the kernel trick.
Online SVMs
Before concluding this chapter, let’s take a quick look at online SVM classifiers (recall
that online learning means learning incrementally, typically as new instances arrive).
For linear SVM classifiers, one method is to use Gradient Descent (e.g., using
SGDClassifier ) to minimize the cost function in Equation 5-13 , which is derived
from the primal problem. Unfortunately it converges much more slowly than the
methods based on QP .
Equation 5-13. Linear SVM classifier  cost functionJw,b=1
2wTw + C∑
i= 1m
max 0, 1 − tiwTxi+b
The first sum in the cost function will push the model to have a small weight vector
w, leading to a larger margin. The second sum computes the total of all margin viola‐

2wTw + C∑
i= 1m
tions. An instance’s margin violation is equal to 0 if it is located off the street and on
the correct side, or else it is proportional to the distance to the correct side of the
street. Minimizing this term ensures that the model makes the margin violations as
small and as few as possible
Hinge Loss
The function max (0, 1 – t) is called the hinge loss  function (represented below). It is
equal to 0 when t ≥ 1. Its derivative (slope) is equal to –1 if t < 1 and 0 if t > 1. It is not
differentiable at t = 1, but just like for Lasso Regression (see “Lasso Regression”  on

Hinge Loss
page 139) you can still use Gradient Descent using any subderivative  at t = 1 (i.e., any
value between –1 and 0).174 | Chapter 5: Support Vector Machines

8“Incremental and Decremental Support Vector Machine Learning, ” G. Cauwenberghs, T. Poggio (2001).9
“Fast Kernel Classifiers with Online and Active Learning,“ A. Bordes, S. Ertekin, J. Weston, L. Bottou (2005).
It is also possible to implement online kernelized SVMs—for example, using “Incre‐
mental and Decremental SVM Learning”8 or “Fast Kernel Classifiers with Online and
Active Learning. ”9 However, these are implemented in Matlab and C++. For large-
scale nonlinear problems, you may want to consider using neural networks instead 
(see Part II ).
What is the fundamental idea behind Support Vector Machines?2.
What is a support vector?3.
Why is it important to scale the inputs when using SVMs?4.
Can an SVM classifier output a confidence score when it classifies an instance?
What about a probability?5.
Should you use the primal or the dual form of the SVM problem to train a model
on a training set with millions of instances and hundreds of features?6.

Say you trained an SVM classifier with an RBF kernel. It seems to underfit the
training set: should you increase or decrease γ (gamma )? What about C?7.
How should you set the QP parameters ( H, f, A, and b) to solve the soft margin
linear SVM classifier problem using an off-the-shelf QP solver?8.
Train a LinearSVC  on a linearly separable dataset. Then train an SVC and a
SGDClassifier  on the same dataset. See if you can get them to produce roughly
the same model.9.
Train an SVM classifier on the MNIST dataset. Since SVM classifiers are binary

classifiers, you will need to use one-versus-all to classify all 10 digits. Y ou mayExercises | 175

want to tune the hyperparameters using small validation sets to speed up the pro‐
cess. What accuracy can you reach?10.
Train an SVM regressor on the California housing dataset.
Solutions to these exercises are available in ???.176 | Chapter 5: Support Vector Machines

CHAPTER 6
Decision Trees
release of these titles. The following will be Chapter 6 in the final
Like SVMs, Decision Trees  are versatile Machine Learning algorithms that can per‐
form both classification and regression tasks, and even multioutput tasks. They are
very powerful algorithms, capable of fitting complex datasets. For example, in Chap‐
ter 2  you trained a DecisionTreeRegressor  model on the California housing dataset,
fitting it perfectly (actually overfitting it).
Decision Trees are also the fundamental components of Random Forests (see Chap‐
ter 7 ), which are among the most powerful Machine Learning algorithms available
today.
In this chapter we will start by discussing how to train, visualize, and make predic‐

today.
tions with Decision Trees. Then we will go through the CART training algorithm
used by Scikit-Learn, and we will discuss how to regularize trees and use them for
regression tasks. Finally, we will discuss some of the limitations of Decision Trees.
Training and Visualizing a Decision Tree
To understand Decision Trees, let’s just build one and take a look at how it makes pre‐
dictions. The following code trains a DecisionTreeClassifier  on the iris dataset
(see Chapter 4 ):
from sklearn.datasets  import load_iris

from sklearn.tree  import DecisionTreeClassifier177

1Graphviz is an open source graph visualization software package, available at http://www.graphviz.org/ . iris = load_iris ()
X = iris.data[:, 2:] # petal length and width
y = iris.target
tree_clf  = DecisionTreeClassifier (max_depth =2)
tree_clf .fit(X, y)
Y ou can visualize the trained Decision Tree by first using the export_graphviz()  
method to output a graph definition file called iris_tree.dot :
from sklearn.tree  import export_graphviz
export_graphviz (
        tree_clf ,
        out_file =image_path ("iris_tree.dot" ),
        feature_names =iris.feature_names [2:],
        class_names =iris.target_names ,
        rounded=True,
        filled=True
    )
Then you can convert this .dot file to a variety of formats such as PDF or PNG using
the dot command-line tool from the graphviz  package.1 This command line converts
the .dot file to a .png image file:
$ dot -Tpng iris_tree.dot -o iris_tree.png
Y our first decision tree looks like Figure 6-1 .178 | Chapter 6: Decision Trees

Figure 6-1. Iris Decision Tree
Making Predictions
Let’s see how the tree represented in Figure 6-1  makes predictions. Suppose you find
an iris flower and you want to classify it. Y ou start at the root node  (depth 0, at the
top): this node asks whether the flower’s petal length is smaller than 2.45 cm. If it is,
then you move down to the root’s left child node (depth 1, left). In this case, it is a leaf
node  (i.e., it does not have any children nodes), so it does not ask any questions: you
can simply look at the predicted class for that node and the Decision Tree predicts
that your flower is an Iris-Setosa ( class=setosa ).
Now suppose you find another flower, but this time the petal length is greater than
2.45 cm. Y ou must move down to the root’s right child node (depth 1, right), which is
not a leaf node, so it asks another question: is the petal width smaller than 1.75 cm? If
it is, then your flower is most likely an Iris-Versicolor (depth 2, left). If not, it is likely

an Iris-Virginica (depth 2, right). It’s really that simple.
One of the many qualities of Decision Trees is that they require
very little data preparation. In particular, they don’t require feature
scaling or centering at all.Making Predictions | 179

A node’s samples  attribute counts how many training instances it applies to. For
example, 100 training instances have a petal length greater than 2.45 cm (depth 1,
right), among which 54 have a petal width smaller than 1.75 cm (depth 2, left). A
node’s value  attribute tells you how many training instances of each class this node
applies to: for example, the bottom-right node applies to 0 Iris-Setosa, 1 Iris-
Versicolor, and 45 Iris-Virginica. Finally, a node’s gini  attribute measures its impur‐
ity: a node is “pure” ( gini=0 ) if all training instances it applies to belong to the same
class. For example, since the depth-1 left node applies only to Iris-Setosa training
instances, it is pure and its gini  score is 0. Equation 6-1  shows how the training algo‐
rithm computes the gini score Gi of the ith node. For example, the depth-2 left node
has a gini  score equal to 1 – (0/54)2 – (49/54)2 – (5/54)2 ≈ 0.168. Another impurity
measure  is discussed shortly.
Equation 6-1. Gini impurityGi= 1 − ∑
k= 1n
pi,k2•

k= 1n
pi,k2•
pi,k is the ratio of class k instances among the training instances in the ith node.
Scikit-Learn uses the CART algorithm, which produces only binary
trees : nonleaf nodes always have two children (i.e., questions only
have yes/no answers). However, other algorithms such as ID3 can
produce Decision Trees with nodes that have more than two chil‐
dren.
Figure 6-2  shows this Decision Tree’s decision boundaries. The thick vertical line rep‐
resents the decision boundary of the root node (depth 0): petal length = 2.45 cm.

dren.
Since the left area is pure (only Iris-Setosa), it cannot be split any further. However,
the right area is impure, so the depth-1 right node splits it at petal width = 1.75 cm
(represented by the dashed line). Since max_depth  was set to 2, the Decision Tree
stops right there. However, if you set max_depth  to 3, then the two depth-2 nodes
would each add another decision boundary (represented by the dotted lines).180 | Chapter 6: Decision Trees

Figure 6-2. Decision Tree decision boundaries
Model Interpretation: White Box Versus Black Box
As you can see Decision Trees are fairly intuitive and their decisions are easy to inter‐
pret. Such models are often called white box models . In contrast, as we will see, Ran‐
dom Forests or neural networks are generally considered black box models . They
make great predictions, and you can easily check the calculations that they performed
to make these predictions; nevertheless, it is usually hard to explain in simple terms
why the predictions were made. For example, if a neural network says that a particu‐
lar person appears on a picture, it is hard to know what actually contributed to this
prediction: did the model recognize that person’s eyes? Her mouth? Her nose? Her
shoes? Or even the couch that she was sitting on? Conversely, Decision Trees provide
nice and simple classification rules that can even be applied manually if need be (e.g.,
for flower classification).
Estimating Class Probabilities

A Decision Tree can also estimate the probability that an instance belongs to a partic‐
ular class k: first it traverses the tree to find the leaf node for this instance, and then it
returns the ratio of training instances of class k in this node. For example, suppose
you have found a flower whose petals are 5 cm long and 1.5 cm wide. The corre‐
sponding leaf node is the depth-2 left node, so the Decision Tree should output the
following probabilities: 0% for Iris-Setosa (0/54), 90.7% for Iris-Versicolor (49/54),

and 9.3% for Iris-Virginica (5/54). And of course if you ask it to predict the class, it
should output Iris-Versicolor (class 1) since it has the highest probability. Let’s check
this:
>>> tree_clf .predict_proba ([[5, 1.5]])
array([[0.        , 0.90740741, 0.09259259]])Estimating Class Probabilities | 181

>>> tree_clf .predict([[5, 1.5]])
array([1])
Perfect! Notice that the estimated probabilities would be identical anywhere else in
the bottom-right rectangle of Figure 6-2 —for example, if the petals were 6 cm long
and 1.5 cm wide (even though it seems obvious that it would most likely be an Iris-
Virginica in this case).
The CART Training Algorithm
Scikit-Learn uses the Classification  And Regression Tree  (CART) algorithm to train
Decision Trees (also called “growing” trees). The idea is really quite simple: the algo‐
rithm first splits the training set in two subsets using a single feature k and a thres‐
hold tk (e.g., “petal length ≤  2.45 cm”). How does it choose k and tk? It searches for the
pair ( k, tk) that produces the purest subsets (weighted by their size). The cost function
that the algorithm tries to minimize is given by Equation 6-2 .
Equation 6-2. CART cost function for classificationJk,tk=mleft
mGleft+mright
mGright
whereGleft/rightmeasures the impurity of the left/right subset,

mGright
mleft/rightis the number of instances in the left/right subset.
Once it has successfully split the training set in two, it splits the subsets using the
same logic, then the sub-subsets and so on, recursively. It stops recursing once it rea‐
ches the maximum depth (defined by the max_depth  hyperparameter), or if it cannot
find a split that will reduce impurity. A few other hyperparameters (described in a
moment) control additional stopping conditions ( min_samples_split , min_sam

ples_leaf , min_weight_fraction_leaf , and max_leaf_nodes ).182 | Chapter 6: Decision Trees

2P is the set of problems that can be solved in polynomial time. NP is the set of problems whose solutions can
be verified in polynomial time. An NP-Hard problem is a problem to which any NP problem can be reduced
in polynomial time. An NP-Complete problem is both NP and NP-Hard. A major open mathematical ques‐
tion is whether or not P = NP . If P ≠ NP (which seems likely), then no polynomial algorithm will ever be
found for any NP-Complete problem (except perhaps on a quantum computer).3
log2 is the binary logarithm. It is equal to log2(m) = log(m) / log(2).4
A reduction of entropy is often called an information gain .
As you can see, the CART algorithm is a greedy algorithm : it greed‐
ily searches for an optimum split at the top level, then repeats the
process at each level. It does not check whether or not the split will
lead to the lowest possible impurity several levels down. A greedy
algorithm often produces a reasonably good solution, but it is not
guaranteed to be the optimal solution.

Unfortunately, finding the optimal tree is known to be an NP-
Complete  problem:2 it requires O(exp( m)) time, making the prob‐
lem intractable even for fairly small training sets. This is why we
must settle for a “reasonably good” solution.
Making predictions requires traversing the Decision Tree from the root to a leaf.
Decision Trees are generally approximately balanced, so traversing the Decision Tree
requires going through roughly O(log2(m)) nodes.3 Since each node only requires

checking the value of one feature, the overall prediction complexity is just O(log2(m)),
independent of the number of features. So predictions are very fast, even when deal‐
ing with large training sets.
However, the training algorithm compares all features (or less if max_features  is set)
on all samples at each node. This results in a training complexity of O(n × m log(m)).
For small training sets (less than a few thousand instances), Scikit-Learn can speed up
training by presorting the data (set presort=True ), but this slows down training con‐

siderably for larger training sets.
Gini Impurity or Entropy?
By default, the Gini impurity measure is used, but you can select the entropy  impurity
measure instead by setting the criterion  hyperparameter to "entropy" . The concept
of entropy originated in thermodynamics as a measure of molecular disorder:
entropy approaches zero when molecules are still and well ordered. It later spread to a
wide variety of domains, including Shannon’s information theory , where it measures

the average information content of a message:4 entropy is zero when all messages are
identical. In Machine Learning, it is frequently used as an impurity measure: a set’sComputational Complexity | 183

5See Sebastian Raschka’s interesting analysis for more details . entropy is zero when it contains instances of only one class. Equation 6-3  shows the
definition of the entropy of the ith node. For example, the depth-2 left node in
Figure 6-1  has an entropy equal to −49
54log249
54−5
54log25
54
 ≈ 0.445.
Equation 6-3. EntropyHi= − ∑
k= 1
pi,k≠ 0n
pi,klog2pi,k
So should you use Gini impurity or entropy? The truth is, most of the time it does not
make a big difference: they lead to similar trees. Gini impurity is slightly faster to
compute, so it is a good default. However, when they differ, Gini impurity tends to
isolate the most frequent class in its own branch of the tree, while entropy tends to
produce slightly more balanced trees.5
Regularization Hyperparameters
Decision Trees make very few assumptions about the training data (as opposed to lin‐
ear models, which obviously assume that the data is linear, for example). If left

unconstrained, the tree structure will adapt itself to the training data, fitting it very
closely, and most likely overfitting it. Such a model is often called a nonparametric
model , not because it does not have any parameters (it often has a lot) but because the
number of parameters is not determined prior to training, so the model structure is
free to stick closely to the data. In contrast, a parametric model  such as a linear model
has a predetermined number of parameters, so its degree of freedom is limited,

reducing the risk of overfitting (but increasing the risk of underfitting).
To avoid overfitting the training data, you need to restrict the Decision Tree’s freedom
during training. As you know by now, this is called regularization. The regularization
hyperparameters depend on the algorithm used, but generally you can at least restrict
the maximum depth of the Decision Tree. In Scikit-Learn, this is controlled by the
max_depth  hyperparameter (the default value is None , which means unlimited).
Reducing max_depth  will regularize the model and thus reduce the risk of overfitting.

The DecisionTreeClassifier  class has a few other parameters that similarly restrict
the shape of the Decision Tree: min_samples_split  (the minimum number of sam‐
ples a node must have before it can be split), min_samples_leaf  (the minimum num‐
ber of samples a leaf node must have), min_weight_fraction_leaf  (same as
min_samples_leaf  but expressed as a fraction of the total number of weighted184 | Chapter 6: Decision Trees

instances), max_leaf_nodes  (maximum number of leaf nodes), and max_features
(maximum number of features that are evaluated for splitting at each node). Increas‐
ing min_*  hyperparameters or reducing max_*  hyperparameters will regularize the
model.
Other algorithms work by first training the Decision Tree without
restrictions, then pruning  (deleting) unnecessary nodes. A node
whose children are all leaf nodes is considered unnecessary if the
purity improvement it provides is not statistically significant . Stan‐
dard statistical tests, such as the χ2 test, are used to estimate the
probability that the improvement is purely the result of chance
(which is called the null hypothesis ). If this probability, called the p-
value , is higher than a given threshold (typically 5%, controlled by
a hyperparameter), then the node is considered unnecessary and its
children are deleted. The pruning continues until all unnecessary
nodes have been pruned.

Figure 6-3  shows two Decision Trees trained on the moons dataset (introduced in
Chapter 5 ). On the left, the Decision Tree is trained with the default hyperparameters
(i.e., no restrictions), and on the right the Decision Tree is trained with min_sam
ples_leaf=4 . It is quite obvious that the model on the left is overfitting, and the
model on the right will probably generalize better.
Figure 6-3. Regularization using min_samples_leaf
Regression

Decision Trees are also capable of performing regression tasks. Let’s build a regres‐
sion tree using Scikit-Learn’s DecisionTreeRegressor  class, training it on a noisy
quadratic dataset with max_depth=2 :
from sklearn.tree  import DecisionTreeRegressorRegression | 185

tree_reg  = DecisionTreeRegressor (max_depth =2)
tree_reg .fit(X, y)
The resulting tree is represented on Figure 6-4 .
Figure 6-4. A Decision Tree for regression
This tree looks very similar to the classification tree you built earlier. The main differ‐
ence is that instead of predicting a class in each node, it predicts a value. For example,
suppose you want to make a prediction for a new instance with x1 = 0.6. Y ou traverse
the tree starting at the root, and you eventually reach the leaf node that predicts
value=0.1106 . This prediction is simply the average target value of the 110 training
instances associated to this leaf node. This prediction results in a Mean Squared Error
(MSE) equal to 0.0151 over these 110 instances.
This model’s predictions are represented on the left of Figure 6-5 . If you set
max_depth=3 , you get the predictions represented on the right. Notice how the pre‐
dicted value for each region is always the average target value of the instances in that

region. The algorithm splits each region in a way that makes most training instances
as close as possible to that predicted value.186 | Chapter 6: Decision Trees

Figure 6-5. Predictions of two Decision Tree regression models
The CART algorithm works mostly the same way as earlier, except that instead of try‐
ing to split the training set in a way that minimizes impurity, it now tries to split the
training set in a way that minimizes the MSE. Equation 6-4  shows the cost function
that the algorithm tries to minimize.
Equation 6-4. CART cost function for regressionJk,tk=mleft
mMSEleft+mright
mMSErightwhereMSEnode=∑
i∈nodeynode−yi2
ynode=1
mnode∑
i∈nodeyi
Just like for classification tasks, Decision Trees are prone to overfitting when dealing
with regression tasks. Without any regularization (i.e., using the default hyperpara‐
meters), you get the predictions on the left of Figure 6-6 . It is obviously overfitting
the training set very badly. Just setting min_samples_leaf=10  results in a much more
reasonable model, represented on the right of Figure 6-6 .
Figure 6-6. Regularizing a Decision Tree regressorRegression | 187

6It randomly selects the set of features to evaluate at each node.Instability
Hopefully by now you are convinced that Decision Trees have a lot going for them:
they are simple to understand and interpret, easy to use, versatile, and powerful.
However they do have a few limitations. First, as you may have noticed, Decision
Trees love orthogonal decision boundaries (all splits are perpendicular to an axis),
which makes them sensitive to training set rotation. For example, Figure 6-7  shows a
simple linearly separable dataset: on the left, a Decision Tree can split it easily, while
on the right, after the dataset is rotated by 45°, the decision boundary looks unneces‐
sarily convoluted. Although both Decision Trees fit the training set perfectly, it is very
likely that the model on the right will not generalize well. One way to limit this prob‐
lem is to use PCA (see Chapter 8 ), which often results in a better orientation of the
training data.
Figure 6-7. Sensitivity to training set rotation

More generally, the main issue with Decision Trees is that they are very sensitive to
small variations in the training data. For example, if you just remove the widest Iris-
Versicolor from the iris training set (the one with petals 4.8 cm long and 1.8 cm wide)
and train a new Decision Tree, you may get the model represented in Figure 6-8 . As
you can see, it looks very different from the previous Decision Tree ( Figure 6-2 ).
Actually, since the training algorithm used by Scikit-Learn is stochastic6 you may

get very different models even on the same training data (unless you set the
random_state  hyperparameter).188 | Chapter 6: Decision Trees

Figure 6-8. Sensitivity to training set details
Random Forests can limit this instability by averaging predictions over many trees, as
we will see in the next chapter.
What is the approximate depth of a Decision Tree trained (without restrictions)
on a training set with 1 million instances?2.
Is a node’s Gini impurity generally lower or greater than its parent’s? Is it gener‐
ally lower/greater, or always  lower/greater?3.
If a Decision Tree is overfitting the training set, is it a good idea to try decreasing
max_depth ?4.
If a Decision Tree is underfitting the training set, is it a good idea to try scaling
the input features?5.
If it takes one hour to train a Decision Tree on a training set containing 1 million
instances, roughly how much time will it take to train another Decision Tree on a
training set containing 10 million instances?6.
If your training set contains 100,000 instances, will setting presort=True  speed
up training?7.
Train and fine-tune a Decision Tree for the moons dataset.a.

Generate a moons dataset using make_moons(n_samples=10000, noise=0.4) .b.
Split it into a training set and a test set using train_test_split() .Exercises | 189

c. Use grid search with cross-validation (with the help of the GridSearchCV
class) to find good hyperparameter values for a DecisionTreeClassifier . 
Hint: try various values for max_leaf_nodes .d.
Train it on the full training set using these hyperparameters, and measure
your model’s performance on the test set. Y ou should get roughly 85% to 87%
accuracy.8.
Grow a forest.a.
Continuing the previous exercise, generate 1,000 subsets of the training set,
each containing 100 instances selected randomly. Hint: you can use Scikit-
Learn’s ShuffleSplit  class for this.b.
Train one Decision Tree on each subset, using the best hyperparameter values
found above. Evaluate these 1,000 Decision Trees on the test set. Since they
were trained on smaller sets, these Decision Trees will likely perform worse
than the first Decision Tree, achieving only about 80% accuracy.c.
Now comes the magic. For each test set instance, generate the predictions of
the 1,000 Decision Trees, and keep only the most frequent prediction (you can

use SciPy’s mode()  function for this). This gives you majority-vote predictions
over the test set.d.
Evaluate these predictions on the test set: you should obtain a slightly higher
accuracy than your first model (about 0.5 to 1.5% higher). Congratulations,
you have trained a Random Forest classifier!
Solutions to these exercises are available in ???.190 | Chapter 6: Decision Trees

CHAPTER 7
Ensemble Learning and Random Forests
release of these titles. The following will be Chapter 7 in the final
Suppose you ask a complex question to thousands of random people, then aggregate
their answers. In many cases you will find that this aggregated answer is better than
an expert’s answer. This is called the wisdom of the crowd . Similarly, if you aggregate
the predictions of a group of predictors (such as classifiers or regressors), you will
often get better predictions than with the best individual predictor. A group of pre‐
dictors is called an ensemble ; thus, this technique is called Ensemble Learning , and an
Ensemble Learning algorithm is called an Ensemble method .
For example, you can train a group of Decision Tree classifiers, each on a different

random subset of the training set. To make predictions, you just obtain the predic‐
tions of all individual trees, then predict the class that gets the most votes (see the last
exercise in Chapter 6 ). Such an ensemble of Decision Trees is called a Random Forest , 
and despite its simplicity, this is one of the most powerful Machine Learning algo‐
rithms available today.
Moreover, as we discussed in Chapter 2 , you will often use Ensemble methods near

the end of a project, once you have already built a few good predictors, to combine
them into an even better predictor. In fact, the winning solutions in Machine Learn‐
ing competitions often involve several Ensemble methods (most famously in the Net‐
flix Prize competition ).
In this chapter we will discuss the most popular Ensemble methods, including bag‐
ging, boosting , stacking , and a few others. We will also explore Random Forests.191

Voting Classifiers
Suppose you have trained a few classifiers, each one achieving about 80% accuracy.
Y ou may have a Logistic Regression classifier, an SVM classifier, a Random Forest
classifier, a K-Nearest Neighbors classifier, and perhaps a few more (see Figure 7-1 ).
Figure 7-1. Training diverse classifiers
A very simple way to create an even better classifier is to aggregate the predictions of
each classifier and predict the class that gets the most votes. This majority-vote classi‐
fier is called a hard voting  classifier (see Figure 7-2 ).
Figure 7-2. Hard voting classifier  predictions192 | Chapter 7: Ensemble Learning and Random Forests

Somewhat surprisingly, this voting classifier often achieves a higher accuracy than the
best classifier in the ensemble. In fact, even if each classifier is a weak learner  (mean‐
ing it does only slightly better than random guessing), the ensemble can still be a
strong learner  (achieving high accuracy), provided there are a sufficient number of
weak learners and they are sufficiently diverse.
How is this possible? The following analogy can help shed some light on this mystery.
Suppose you have a slightly biased coin that has a 51% chance of coming up heads,
and 49% chance of coming up tails. If you toss it 1,000 times, you will generally get
more or less 510 heads and 490 tails, and hence a majority of heads. If you do the
math, you will find that the probability of obtaining a majority of heads after 1,000
tosses is close to 75%. The more you toss the coin, the higher the probability (e.g.,
with 10,000 tosses, the probability climbs over 97%). This is due to the law of large

numbers : as you keep tossing the coin, the ratio of heads gets closer and closer to the
probability of heads (51%). Figure 7-3  shows 10 series of biased coin tosses. Y ou can
see that as the number of tosses increases, the ratio of heads approaches 51%. Eventu‐
ally all 10 series end up so close to 51% that they are consistently above 50%.
Figure 7-3. The law of large numbers
Similarly, suppose you build an ensemble containing 1,000 classifiers that are individ‐

ually correct only 51% of the time (barely better than random guessing). If you pre‐
dict the majority voted class, you can hope for up to 75% accuracy! However, this is
only true if all classifiers are perfectly independent, making uncorrelated errors,
which is clearly not the case since they are trained on the same data. They are likely to
make the same types of errors, so there will be many majority votes for the wrong
class, reducing the ensemble’s accuracy.Voting Classifiers  | 193

Ensemble methods work best when the predictors are as independ‐
ent from one another as possible. One way to get diverse classifiers
is to train them using very different algorithms. This increases the
chance that they will make very different types of errors, improving
the ensemble’s accuracy.
The following code creates and trains a voting classifier in Scikit-Learn, composed of
three diverse classifiers (the training set is the moons dataset, introduced in Chap‐
ter 5 ):
from sklearn.ensemble  import VotingClassifier
log_clf = LogisticRegression ()
rnd_clf = RandomForestClassifier ()
svm_clf = SVC()
voting_clf  = VotingClassifier (
    estimators =[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
    voting='hard')
voting_clf .fit(X_train, y_train)
Let’s look at each classifier’s accuracy on the test set:
>>> from sklearn.metrics  import accuracy_score

>>> for clf in (log_clf, rnd_clf, svm_clf, voting_clf ):
...     clf.fit(X_train, y_train)
...     y_pred = clf.predict(X_test)
...     print(clf.__class__ .__name__ , accuracy_score (y_test, y_pred))
...
LogisticRegression 0.864
RandomForestClassifier 0.896
SVC 0.888
VotingClassifier 0.904
There you have it! The voting classifier slightly outperforms all the individual classifi‐
ers.
If all classifiers are able to estimate class probabilities (i.e., they have a pre

...
SVC 0.888
ers.
dict_proba()  method), then you can tell Scikit-Learn to predict the class with the
highest class probability, averaged over all the individual classifiers. This is called soft
voting . It often achieves higher performance than hard voting because it gives more
weight to highly confident votes. All you need to do is replace voting="hard"  with
voting="soft"  and ensure that all classifiers can estimate class probabilities. This is
not the case of the SVC class by default, so you need to set its probability  hyperpara‐

meter to True  (this will make the SVC class use cross-validation to estimate class prob‐
abilities, slowing down training, and it will add a predict_proba()  method). If you194 | Chapter 7: Ensemble Learning and Random Forests

1“Bagging Predictors, ” L. Breiman (1996).2
In statistics, resampling with replacement is called bootstrapping .3
“Pasting small votes for classification in large databases and on-line, ” L. Breiman (1999).modify the preceding code to use soft voting, you will find that the voting classifier
achieves over 91.2% accuracy!
Bagging and Pasting
One way to get a diverse set of classifiers is to use very different training algorithms,
as just discussed. Another approach is to use the same training algorithm for every
predictor, but to train them on different random subsets of the training set. When
sampling is performed with  replacement, this method is called bagging1 (short for
bootstrap aggregating2). When sampling is performed without  replacement, it is called
pasting .3
In other words, both bagging and pasting allow training instances to be sampled sev‐
eral times across multiple predictors, but only bagging allows training instances to be

pasting .3
sampled several times for the same predictor. This sampling and training process is
represented in Figure 7-4 .
Figure 7-4. Pasting/bagging training set sampling and training
Once all predictors are trained, the ensemble can make a prediction for a new
instance by simply aggregating the predictions of all predictors. The aggregation
function is typically the statistical mode  (i.e., the most frequent prediction, just like a
hard voting classifier) for classification, or the average for regression. Each individualBagging and Pasting | 195

4Bias and variance were introduced in Chapter 4 .5
max_samples  can alternatively be set to a float between 0.0 and 1.0, in which case the max number of instances
to sample is equal to the size of the training set times max_samples .
predictor has a higher bias than if it were trained on the original training set, but
aggregation reduces both bias and variance.4 Generally, the net result is that the
ensemble has a similar bias but a lower variance than a single predictor trained on the
original training set.
As you can see in Figure 7-4 , predictors can all be trained in parallel, via different
CPU cores or even different servers. Similarly, predictions can be made in parallel.
This is one of the reasons why bagging and pasting are such popular methods: they
scale very well.
Bagging and Pasting in Scikit-Learn
Scikit-Learn offers a simple API for both bagging and pasting with the BaggingClas
sifier  class (or BaggingRegressor  for regression). The following code trains an

ensemble of 500 Decision Tree classifiers,5 each trained on 100 training instances ran‐
domly sampled from the training set with replacement (this is an example of bagging,
but if you want to use pasting instead, just set bootstrap=False ). The n_jobs  param‐
eter tells Scikit-Learn the number of CPU cores to use for training and predictions
(–1 tells Scikit-Learn to use all available cores):
from sklearn.ensemble  import BaggingClassifier
from sklearn.tree  import DecisionTreeClassifier
bag_clf = BaggingClassifier (

    DecisionTreeClassifier (), n_estimators =500,
    max_samples =100, bootstrap =True, n_jobs=-1)
bag_clf.fit(X_train, y_train)
y_pred = bag_clf.predict(X_test)
The BaggingClassifier  automatically performs soft voting
instead of hard voting if the base classifier can estimate class proba‐
bilities (i.e., if it has a predict_proba()  method), which is the case
with Decision Trees classifiers.
Figure 7-5  compares the decision boundary of a single Decision Tree with the deci‐
sion boundary of a bagging ensemble of 500 trees (from the preceding code), both

trained on the moons dataset. As you can see, the ensemble’s predictions will likely
generalize much better than the single Decision Tree’s predictions: the ensemble has a
comparable bias but a smaller variance (it makes roughly the same number of errors
on the training set, but the decision boundary is less irregular).196 | Chapter 7: Ensemble Learning and Random Forests

6As m grows, this ratio approaches 1 – exp(–1) ≈ 63.212%.
Figure 7-5. A single Decision Tree versus a bagging ensemble of 500 trees
Bootstrapping introduces a bit more diversity in the subsets that each predictor is
trained on, so bagging ends up with a slightly higher bias than pasting, but this also
means that predictors end up being less correlated so the ensemble’s variance is
reduced. Overall, bagging often results in better models, which explains why it is gen‐
erally preferred. However, if you have spare time and CPU power you can use cross-
validation to evaluate both bagging and pasting and select the one that works best.
Out-of-Bag Evaluation
With bagging, some instances may be sampled several times for any given predictor,
while others may not be sampled at all. By default a BaggingClassifier  samples m
training instances with replacement ( bootstrap=True ), where m is the size of the
training set. This means that only about 63% of the training instances are sampled on

average for each predictor.6 The remaining 37% of the training instances that are not
sampled are called out-of-bag  (oob) instances. Note that they are not the same 37%
for all predictors.
Since a predictor never sees the oob instances during training, it can be evaluated on
these instances, without the need for a separate validation set. Y ou can evaluate the
ensemble itself by averaging out the oob evaluations of each predictor.
In Scikit-Learn, you can set oob_score=True  when creating a BaggingClassifier  to

request an automatic oob evaluation after training. The following code demonstrates
this. The resulting evaluation score is available through the oob_score_  variable:
>>> bag_clf = BaggingClassifier (
...     DecisionTreeClassifier (), n_estimators =500,
...     bootstrap =True, n_jobs=-1, oob_score =True)
...
>>> bag_clf.fit(X_train, y_train)Bagging and Pasting | 197

7“Ensembles on Random Patches, ” G. Louppe and P . Geurts (2012).8
“The random subspace method for constructing decision forests, ” Tin Kam Ho (1998).>>> bag_clf.oob_score_
0.90133333333333332
According to this oob evaluation, this BaggingClassifier  is likely to achieve about
90.1% accuracy on the test set. Let’s verify this:
>>> y_pred = bag_clf.predict(X_test)
>>> accuracy_score (y_test, y_pred)
0.91200000000000003
We get 91.2% accuracy on the test set—close enough!
The oob decision function for each training instance is also available through the
oob_decision_function_  variable. In this case (since the base estimator has a pre
dict_proba()  method) the decision function returns the class probabilities for each
training instance. For example, the oob evaluation estimates that the first training
instance has a 68.25% probability of belonging to the positive class (and 31.75% of
belonging to the negative class):
>>> bag_clf.oob_decision_function_

array([[0.31746032, 0.68253968],
       [0.34117647, 0.65882353],
       [1.        , 0.        ],
       ...
       [1.        , 0.        ],
       [0.03108808, 0.96891192],
       [0.57291667, 0.42708333]])
Random Patches and Random Subspaces
The BaggingClassifier  class supports sampling the features as well. This is con‐
trolled by two hyperparameters: max_features  and bootstrap_features . They work
the same way as max_samples  and bootstrap , but for feature sampling instead of

       [1.        , 0.        ],
       ...
       [1.        , 0.        ],
instance sampling. Thus, each predictor will be trained on a random subset of the
input features.
This is particularly useful when you are dealing with high-dimensional inputs (such
as images). Sampling both training instances and features is called the Random
Patches  method .7 Keeping all training instances (i.e., bootstrap=False  and max_sam
ples=1.0 ) but sampling features (i.e., bootstrap_features=True  and/or max_fea

tures  smaller than 1.0) is called the Random Subspaces  method .8198 | Chapter 7: Ensemble Learning and Random Forests

9“Random Decision Forests, ” T. Ho (1995).10
The BaggingClassifier  class remains useful if you want a bag of something other than Decision Trees.11
There are a few notable exceptions: splitter  is absent (forced to "random" ), presort  is absent (forced to
False ), max_samples  is absent (forced to 1.0), and base_estimator  is absent (forced to DecisionTreeClassi
fier  with the provided hyperparameters).Sampling features results in even more predictor diversity, trading a bit more bias for
a lower variance.
Random Forests
As we have discussed, a Random Forest9 is an ensemble of Decision Trees, generally
trained via the bagging method (or sometimes pasting), typically with max_samples
set to the size of the training set. Instead of building a BaggingClassifier  and pass‐
ing it a DecisionTreeClassifier , you can instead use the RandomForestClassifier
class, which is more convenient and optimized for Decision Trees10 (similarly, there is

a RandomForestRegressor  class for regression tasks). The following code trains a
Random Forest classifier with 500 trees (each limited to maximum 16 nodes), using
all available CPU cores:
rnd_clf = RandomForestClassifier (n_estimators =500, max_leaf_nodes =16, n_jobs=-1)
rnd_clf.fit(X_train, y_train)
y_pred_rf  = rnd_clf.predict(X_test)
With a few exceptions, a RandomForestClassifier  has all the hyperparameters of a
DecisionTreeClassifier  (to control how trees are grown), plus all the hyperpara‐

meters of a BaggingClassifier  to control the ensemble itself.11
The Random Forest algorithm introduces extra randomness when growing trees;
instead of searching for the very best feature when splitting a node (see Chapter 6 ), it
searches for the best feature among a random subset of features. This results in a
greater tree diversity, which (once again) trades a higher bias for a lower variance,
generally yielding an overall better model. The following BaggingClassifier  is
roughly equivalent to the previous RandomForestClassifier :

    DecisionTreeClassifier (splitter ="random" , max_leaf_nodes =16),
    n_estimators =500, max_samples =1.0, bootstrap =True, n_jobs=-1)Random Forests | 199

12“Extremely randomized trees, ” P . Geurts, D. Ernst, L. Wehenkel (2005).
Extra-Trees
When you are growing a tree in a Random Forest, at each node only a random subset
of the features is considered for splitting (as discussed earlier). It is possible to make
trees even more random by also using random thresholds for each feature rather than
searching for the best possible thresholds (like regular Decision Trees do).
A forest of such extremely random trees is simply called an Extremely Randomized
Trees  ensemble12 (or Extra-Trees  for short). Once again, this trades more bias for a
lower variance. It also makes Extra-Trees much faster to train than regular Random
Forests since finding the best possible threshold for each feature at every node is one
of the most time-consuming tasks of growing a tree.
Y ou can create an Extra-Trees classifier using Scikit-Learn’s ExtraTreesClassifier
class. Its API is identical to the RandomForestClassifier  class. Similarly, the Extra

TreesRegressor  class has the same API as the RandomForestRegressor  class.
It is hard to tell in advance whether a RandomForestClassifier
will perform better or worse than an ExtraTreesClassifier . Gen‐
erally, the only way to know is to try both and compare them using
cross-validation (and tuning the hyperparameters using grid
search).
Feature Importance
Y et another great quality of Random Forests is that they make it easy to measure the 
relative importance of each feature. Scikit-Learn measures a feature’s importance by

search).
looking at how much the tree nodes that use that feature reduce impurity on average
(across all trees in the forest). More precisely, it is a weighted average, where each
node’s weight is equal to the number of training samples that are associated with it
(see Chapter 6 ).
Scikit-Learn computes this score automatically for each feature after training, then it
scales the results so that the sum of all importances is equal to 1. Y ou can access the
result using the feature_importances_  variable. For example, the following code

trains a RandomForestClassifier  on the iris dataset (introduced in Chapter 4 ) and
outputs each feature’s importance. It seems that the most important features are the
petal length (44%) and width (42%), while sepal length and width are rather unim‐
portant in comparison (11% and 2%, respectively).200 | Chapter 7: Ensemble Learning and Random Forests

>>> from sklearn.datasets  import load_iris
>>> iris = load_iris ()
>>> rnd_clf = RandomForestClassifier (n_estimators =500, n_jobs=-1)
>>> rnd_clf.fit(iris["data"], iris["target" ])
>>> for name, score in zip(iris["feature_names" ], rnd_clf.feature_importances_ ):
...     print(name, score)
...
sepal length (cm) 0.112492250999
sepal width (cm) 0.0231192882825
petal length (cm) 0.441030464364
petal width (cm) 0.423357996355
Similarly, if you train a Random Forest classifier on the MNIST dataset (introduced
in Chapter 3 ) and plot each pixel’s importance, you get the image represented in
Figure 7-6 .
Figure 7-6. MNIST pixel importance (according to a Random Forest classifier)
Random Forests are very handy to get a quick understanding of what features
actually matter, in particular if you need to perform feature selection.
Boosting
Boosting  (originally called hypothesis boosting ) refers to any Ensemble method that
can combine several weak learners into a strong learner. The general idea of most

Boosting
boosting methods is to train predictors sequentially, each trying to correct its prede‐
cessor. There are many boosting methods available, but by far the most popular areBoosting | 201

13“ A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting, ” Y oav Freund,
Robert E. Schapire (1997).14
This is just for illustrative purposes. SVMs are generally not good base predictors for AdaBoost, because they
are slow and tend to be unstable with AdaBoost.AdaBoost13 (short for Adaptive Boosting ) and Gradient Boosting . Let’s start with Ada‐
Boost.
AdaBoost
One way for a new predictor to correct its predecessor is to pay a bit more attention
to the training instances that the predecessor underfitted. This results in new predic‐
tors focusing more and more on the hard cases. This is the technique used by Ada‐
Boost.
For example, to build an AdaBoost classifier, a first base classifier (such as a Decision
Tree) is trained and used to make predictions on the training set. The relative weight
of misclassified training instances is then increased. A second classifier is trained
using the updated weights and again it makes predictions on the training set, weights

Boost.
are updated, and so on (see Figure 7-7 ).
Figure 7-7. AdaBoost sequential training with instance weight updates
Figure 7-8  shows the decision boundaries of five consecutive predictors on the
moons dataset (in this example, each predictor is a highly regularized SVM classifier
with an RBF kernel14). The first classifier gets many instances wrong, so their weights202 | Chapter 7: Ensemble Learning and Random Forests

get boosted. The second classifier therefore does a better job on these instances, and
so on. The plot on the right represents the same sequence of predictors except that
the learning rate is halved (i.e., the misclassified instance weights are boosted half as
much at every iteration). As you can see, this sequential learning technique has some
similarities with Gradient Descent, except that instead of tweaking a single predictor’s
parameters to minimize a cost function, AdaBoost adds predictors to the ensemble,
gradually making it better.
Figure 7-8. Decision boundaries of consecutive predictors
Once all predictors are trained, the ensemble makes predictions very much like bag‐
ging or pasting, except that predictors have different weights depending on their
overall accuracy on the weighted training set.
There is one important drawback to this sequential learning techni‐
que: it cannot be parallelized (or only partially), since each predic‐
tor can only be trained after the previous predictor has been

trained and evaluated. As a result, it does not scale as well as bag‐
ging or pasting.
Let’s take a closer look at the AdaBoost algorithm. Each instance weight w(i) is initially
set to 1
m. A first predictor is trained and its weighted error rate r1 is computed on the
training set; see Equation 7-1 .
Equation 7-1. Weighted error rate of the jth predictorrj=∑
i= 1
yji≠yim
wi
∑
i= 1m
wiwhere yjiis the jthpredictor’s prediction for the ithinstance.Boosting | 203

15The original AdaBoost algorithm does not use a learning rate hyperparameter.The predictor’s weight αj is then computed using Equation 7-2 , where η is the learn‐
ing rate hyperparameter (defaults to 1).15 The more accurate the predictor is, the
higher its weight will be. If it is just guessing randomly, then its weight will be close to
zero. However, if it is most often wrong (i.e., less accurate than random guessing),
then its weight will be negative.
Equation 7-2. Predictor weightαj=ηlog1 −rj
rj
Next the instance weights are updated using Equation 7-3 : the misclassified instances
are boosted.
Equation 7-3. Weight update rulefori= 1, 2,⋯,m
wiwiifyji=yi
wiexp αjifyji≠yi
Then all the instance weights are normalized (i.e., divided by ∑i= 1mwi
).
Finally, a new predictor is trained using the updated weights, and the whole process is
repeated (the new predictor’s weight is computed, the instance weights are updated,
then another predictor is trained, and so on). The algorithm stops when the desired

rj
).
number of predictors is reached, or when a perfect predictor is found.
To make predictions, AdaBoost simply computes the predictions of all the predictors
and weighs them using the predictor weights αj. The predicted class is the one that
receives the majority of weighted votes (see Equation 7-4 ).
Equation 7-4. AdaBoost predictionsyx= argmax
k∑
j= 1
yjx=kN
αjwhere Nis the number of predictors.
204 | Chapter 7: Ensemble Learning and Random Forests

16For more details, see “Multi-Class AdaBoost, ” J. Zhu et al. (2006).17
First introduced in “ Arcing the Edge, ” L. Breiman (1997), and further developed in the paper “Greedy Func‐
tion Approximation: A Gradient Boosting Machine, ” Jerome H. Friedman (1999).
Scikit-Learn actually uses a multiclass version of AdaBoost called SAMME16 (which
stands for Stagewise Additive Modeling using a Multiclass Exponential loss function ).
When there are just two classes, SAMME is equivalent to AdaBoost. Moreover, if the
predictors can estimate class probabilities (i.e., if they have a predict_proba()
method), Scikit-Learn can use a variant of SAMME called SAMME.R  (the R stands
for “Real”), which relies on class probabilities rather than predictions and generally
performs better.
The following code trains an AdaBoost classifier based on 200 Decision Stumps  using
Scikit-Learn’s AdaBoostClassifier  class (as you might expect, there is also an Ada

BoostRegressor  class). A Decision Stump is a Decision Tree with max_depth=1 —in
other words, a tree composed of a single decision node plus two leaf nodes. This is
the default base estimator for the AdaBoostClassifier  class:
from sklearn.ensemble  import AdaBoostClassifier
ada_clf = AdaBoostClassifier (
    DecisionTreeClassifier (max_depth =1), n_estimators =200,
    algorithm ="SAMME.R" , learning_rate =0.5)
ada_clf.fit(X_train, y_train)
If your AdaBoost ensemble is overfitting the training set, you can
try reducing the number of estimators or more strongly regulariz‐

ing the base estimator.
Gradient Boosting
Another very popular Boosting algorithm is Gradient Boosting .17 Just like AdaBoost,
Gradient Boosting works by sequentially adding predictors to an ensemble, each one
correcting its predecessor. However, instead of tweaking the instance weights at every
iteration like AdaBoost does, this method tries to fit the new predictor to the residual
errors  made by the previous predictor.
Let’s go through a simple regression example using Decision Trees as the base predic‐

tors (of course Gradient Boosting also works great with regression tasks). This is
called  Gradient Tree Boosting , or Gradient Boosted Regression Trees  (GBRT ). First, let’s
fit a DecisionTreeRegressor  to the training set (for example, a noisy quadratic train‐
ing set):Boosting | 205

tree_reg1  = DecisionTreeRegressor (max_depth =2)
tree_reg1 .fit(X, y)
Now train a second DecisionTreeRegressor  on the residual errors made by the first
predictor:
y2 = y - tree_reg1 .predict(X)
tree_reg2  = DecisionTreeRegressor (max_depth =2)
tree_reg2 .fit(X, y2)
Then we train a third regressor on the residual errors made by the second predictor:
y3 = y2 - tree_reg2 .predict(X)
tree_reg3  = DecisionTreeRegressor (max_depth =2)
tree_reg3 .fit(X, y3)
Now we have an ensemble containing three trees. It can make predictions on a new
instance simply by adding up the predictions of all the trees:
y_pred = sum(tree.predict(X_new) for tree in (tree_reg1 , tree_reg2 , tree_reg3 ))
Figure 7-9  represents the predictions of these three trees in the left column, and the
ensemble’s predictions in the right column. In the first row, the ensemble has just one
tree, so its predictions are exactly the same as the first tree’s predictions. In the second

row, a new tree is trained on the residual errors of the first tree. On the right you can
see that the ensemble’s predictions are equal to the sum of the predictions of the first
two trees. Similarly, in the third row another tree is trained on the residual errors of
the second tree. Y ou can see that the ensemble’s predictions gradually get better as
trees are added to the ensemble.
A simpler way to train GBRT ensembles is to use Scikit-Learn’s GradientBoostingRe

gressor  class. Much like the RandomForestRegressor  class, it has hyperparameters to
control the growth of Decision Trees (e.g., max_depth , min_samples_leaf , and so on),
as well as hyperparameters to control the ensemble training, such as the number of
trees (n_estimators ). The following code creates the same ensemble as the previous
one:
from sklearn.ensemble  import GradientBoostingRegressor
gbrt = GradientBoostingRegressor (max_depth =2, n_estimators =3, learning_rate =1.0)
gbrt.fit(X, y)206 | Chapter 7: Ensemble Learning and Random Forests

Figure 7-9. Gradient Boosting
The learning_rate  hyperparameter scales the contribution of each tree. If you set it
to a low value, such as 0.1, you will need more trees in the ensemble to fit the train‐
ing set, but the predictions will usually generalize better. This is a regularization tech‐
nique called shrinkage . Figure 7-10  shows two GBRT ensembles trained with a low
learning rate: the one on the left does not have enough trees to fit the training set,
while the one on the right has too many trees and overfits the training set.Boosting | 207

Figure 7-10. GBRT ensembles with not enough predictors (left)  and too many (right)
In order to find the optimal number of trees, you can use early stopping (see Chap‐
ter 4 ). A simple way to implement this is to use the staged_predict()  method: it
returns an iterator over the predictions made by the ensemble at each stage of train‐
ing (with one tree, two trees, etc.). The following code trains a GBRT ensemble with
120 trees, then measures the validation error at each stage of training to find the opti‐
mal number of trees, and finally trains another GBRT ensemble using the optimal
number of trees:
X_train, X_val, y_train, y_val = train_test_split (X, y)
gbrt = GradientBoostingRegressor (max_depth =2, n_estimators =120)
gbrt.fit(X_train, y_train)
errors = [mean_squared_error (y_val, y_pred)
          for y_pred in gbrt.staged_predict (X_val)]
bst_n_estimators  = np.argmin(errors)

gbrt_best  = GradientBoostingRegressor (max_depth =2,n_estimators =bst_n_estimators )
gbrt_best .fit(X_train, y_train)
The validation errors are represented on the left of Figure 7-11 , and the best model’s
predictions are represented on the right.208 | Chapter 7: Ensemble Learning and Random Forests

Figure 7-11. Tuning the number of trees using early stopping
It is also possible to implement early stopping by actually stopping training early
(instead of training a large number of trees first and then looking back to find the
optimal number). Y ou can do so by setting warm_start=True , which makes Scikit-
Learn keep existing trees when the fit()  method is called, allowing incremental
training. The following code stops training when the validation error does not
improve for five iterations in a row:
gbrt = GradientBoostingRegressor (max_depth =2, warm_start =True)
min_val_error  = float("inf")
error_going_up  = 0
for n_estimators  in range(1, 120):
    gbrt.n_estimators  = n_estimators
    gbrt.fit(X_train, y_train)
    y_pred = gbrt.predict(X_val)
    val_error  = mean_squared_error (y_val, y_pred)
    if val_error  < min_val_error :
        min_val_error  = val_error
        error_going_up  = 0
    else:
        error_going_up  += 1
        if error_going_up  == 5:
            break  # early stopping

    else:
The GradientBoostingRegressor  class also supports a subsample  hyperparameter,
which specifies the fraction of training instances to be used for training each tree. For
example, if subsample=0.25 , then each tree is trained on 25% of the training instan‐
ces, selected randomly. As you can probably guess by now, this trades a higher bias
for a lower variance. It also speeds up training considerably. This technique is called
Stochastic Gradient Boosting .Boosting | 209

18“Stacked Generalization, ” D. Wolpert (1992).
 It is possible to use Gradient Boosting with other cost functions.
This is controlled by the loss  hyperparameter (see Scikit-Learn’s
documentation for more details).
It is worth noting that an optimized implementation of Gradient Boosting is available
in the popular python library XGBoost , which stands for Extreme Gradient Boosting.
This package was initially developed by Tianqi Chen as part of the Distributed (Deep)
Machine Learning Community ( DMLC ), and it aims at being extremely fast, scalable
and portable. In fact, XGBoost is often an important component of the winning
entries in ML competitions. XGBoost’s API is quite similar to Scikit-Learn’s:
import xgboost
xgb_reg = xgboost.XGBRegressor ()
xgb_reg.fit(X_train, y_train)
y_pred = xgb_reg.predict(X_val)
XGBoost also offers several nice features, such as automatically taking care of early
stopping:
xgb_reg.fit(X_train, y_train,
            eval_set =[(X_val, y_val)], early_stopping_rounds =2)

stopping:
Y ou should definitely check it out!
Stacking
The last Ensemble method we will discuss in this chapter is called stacking  (short for
stacked generalization ).18 It is based on a simple idea: instead of using trivial functions
(such as hard voting) to aggregate the predictions of all predictors in an ensemble,
why don’t we train a model to perform this aggregation? Figure 7-12  shows such an
ensemble performing a regression task on a new instance. Each of the bottom three

Stacking
predictors predicts a different value (3.1, 2.7, and 2.9), and then the final predictor 
(called a blender , or a meta learner ) takes these predictions as inputs and makes the
final prediction (3.0).210 | Chapter 7: Ensemble Learning and Random Forests

19Alternatively, it is possible to use out-of-fold predictions. In some contexts this is called stacking , while using a
hold-out set is called blending . However, for many people these terms are synonymous.
Figure 7-12. Aggregating predictions using a blending predictor
To train the blender, a common approach is to use a hold-out set.19 Let’s see how it
works. First, the training set is split in two subsets. The first subset is used to train the
predictors in the first layer (see Figure 7-13 ).
Figure 7-13. Training the first layer
Next, the first layer predictors are used to make predictions on the second (held-out)
set (see Figure 7-14 ). This ensures that the predictions are “clean, ” since the predictors
never saw these instances during training. Now for each instance in the hold-out setStacking | 211

there are three predicted values. We can create a new training set using these predic‐
ted values as input features (which makes this new training set three-dimensional),
and keeping the target values. The blender is trained on this new training set, so it
learns to predict the target value given the first layer’s predictions.
Figure 7-14. Training the blender
It is actually possible to train several different blenders this way (e.g., one using Lin‐
ear Regression, another using Random Forest Regression, and so on): we get a whole
layer of blenders. The trick is to split the training set into three subsets: the first one is
used to train the first layer, the second one is used to create the training set used to
train the second layer (using predictions made by the predictors of the first layer),
and the third one is used to create the training set to train the third layer (using pre‐
dictions made by the predictors of the second layer). Once this is done, we can make

a prediction for a new instance by going through each layer sequentially, as shown in
Figure 7-15 .212 | Chapter 7: Ensemble Learning and Random Forests

Figure 7-15. Predictions in a multilayer stacking ensemble
Unfortunately, Scikit-Learn does not support stacking directly, but it is not too hard
to roll out your own implementation (see the following exercises). Alternatively, you
can use an open source implementation such as brew  (available at https://github.com/
viisar/brew ).
If you have trained five different models on the exact same training data, and
they all achieve 95% precision, is there any chance that you can combine these
models to get better results? If so, how? If not, why?2.
What is the difference between hard and soft voting classifiers?3.
Is it possible to speed up training of a bagging ensemble by distributing it across
multiple servers? What about pasting ensembles, boosting ensembles, random
forests, or stacking ensembles?4.
What is the benefit of out-of-bag evaluation?5.
What makes Extra-Trees more random than regular Random Forests? How can
this extra randomness help? Are Extra-Trees slower or faster than regular Ran‐

dom Forests?6.
If your AdaBoost ensemble underfits the training data, what hyperparameters
should you tweak and how?Exercises | 213

7. If your Gradient Boosting ensemble overfits the training set, should you increase
or decrease the learning rate?8.
Load the MNIST data (introduced in Chapter 3 ), and split it into a training set, a
validation set, and a test set (e.g., use 50,000 instances for training, 10,000 for val‐
idation, and 10,000 for testing). Then train various classifiers, such as a Random
Forest classifier, an Extra-Trees classifier, and an SVM. Next, try to combine
them into an ensemble that outperforms them all on the validation set, using a
soft or hard voting classifier. Once you have found one, try it on the test set. How
much better does it perform compared to the individual classifiers?9.
Run the individual classifiers from the previous exercise to make predictions on
the validation set, and create a new training set with the resulting predictions:
each training instance is a vector containing the set of predictions from all your
classifiers for an image, and the target is the image’s class. Train a classifier on

this new training set. Congratulations, you have just trained a blender, and
together with the classifiers they form a stacking ensemble! Now let’s evaluate the
ensemble on the test set. For each image in the test set, make predictions with all
your classifiers, then feed the predictions to the blender to get the ensemble’s pre‐
dictions. How does it compare to the voting classifier you trained earlier?
Solutions to these exercises are available in ???.214 | Chapter 7: Ensemble Learning and Random Forests

CHAPTER 8
Dimensionality Reduction
release of these titles. The following will be Chapter 8 in the final
Many Machine Learning problems involve thousands or even millions of features for
each training instance. Not only does this make training extremely slow, it can also
make it much harder to find a good solution, as we will see. This problem is often
referred to as the curse of dimensionality .
Fortunately, in real-world problems, it is often possible to reduce the number of fea‐
tures considerably, turning an intractable problem into a tractable one. For example,
consider the MNIST images (introduced in Chapter 3 ): the pixels on the image bor‐
ders are almost always white, so you could completely drop these pixels from the

training set without losing much information. Figure 7-6  confirms that these pixels
are utterly unimportant for the classification task. Moreover, two neighboring pixels
are often highly correlated: if you merge them into a single pixel (e.g., by taking the
mean of the two pixel intensities), you will not lose much information.215

1Well, four dimensions if you count time, and a few more if you are a string theorist.
 Reducing dimensionality does lose some information (just like
compressing an image to JPEG can degrade its quality), so even
though it will speed up training, it may also make your system per‐
form slightly worse. It also makes your pipelines a bit more com‐
plex and thus harder to maintain. So you should first try to train
your system with the original data before considering using dimen‐
sionality reduction if training is too slow. In some cases, however,
reducing the dimensionality of the training data may filter out
some noise and unnecessary details and thus result in higher per‐
formance (but in general it won’t; it will just speed up training).
Apart from speeding up training, dimensionality reduction is also extremely useful
for data visualization (or DataViz ). Reducing the number of dimensions down to two
(or three) makes it possible to plot a condensed view of a high-dimensional training

set on a graph and often gain some important insights by visually detecting patterns,
such as clusters. Moreover, DataViz is essential to communicate your conclusions to
people who are not data scientists, in particular decision makers who will use your
results.
In this chapter we will discuss the curse of dimensionality and get a sense of what
goes on in high-dimensional space. Then, we will present the two main approaches to
dimensionality reduction (projection and Manifold Learning), and we will go

results.
through three of the most popular dimensionality reduction techniques: PCA, Kernel
PCA, and LLE.
The Curse of Dimensionality
We are so used to living in three dimensions1 that our intuition fails us when we try
to imagine a high-dimensional space. Even a basic 4D hypercube is incredibly hard to
picture in our mind (see Figure 8-1 ), let alone a 200-dimensional ellipsoid bent in a
1,000-dimensional space.216 | Chapter 8: Dimensionality Reduction

2Watch a rotating tesseract projected into 3D space at https://homl.info/30 . Image by Wikipedia user Nerd‐
Boy1392 ( Creative Commons BY-SA 3.0 ). Reproduced from https://en.wikipedia.org/wiki/Tesseract .3
Fun fact: anyone you know is probably an extremist in at least one dimension (e.g., how much sugar they put
in their coffee), if you consider enough dimensions.
Figure 8-1. Point, segment, square, cube, and tesseract (0D to 4D hypercubes)2
It turns out that many things behave very differently in high-dimensional space. For
example, if you pick a random point in a unit square (a 1 × 1 square), it will have only
about a 0.4% chance of being located less than 0.001 from a border (in other words, it
is very unlikely that a random point will be “extreme” along any dimension). But in a
10,000-dimensional unit hypercube (a 1 × 1 × ⋯ × 1 cube, with ten thousand 1s), this
probability is greater than 99.999999%. Most points in a high-dimensional hypercube
are very close to the border.3

Here is a more troublesome difference: if you pick two points randomly in a unit
square, the distance between these two points will be, on average, roughly 0.52. If you
pick two random points in a unit 3D cube, the average distance will be roughly 0.66.
But what about two points picked randomly in a 1,000,000-dimensional hypercube?
Well, the average distance, believe it or not, will be about 408.25 (roughly1, 000, 000/6
)! This is quite counterintuitive: how can two points be so far apart

when they both lie within the same unit hypercube? This fact implies that high-
dimensional datasets are at risk of being very sparse: most training instances are
likely to be far away from each other. Of course, this also means that a new instance
will likely be far away from any training instance, making predictions much less relia‐
ble than in lower dimensions, since they will be based on much larger extrapolations.
In short, the more dimensions the training set has, the greater the risk of overfitting
it.

it.
In theory, one solution to the curse of dimensionality could be to increase the size of
the training set to reach a sufficient density of training instances. Unfortunately, in
practice, the number of training instances required to reach a given density grows
exponentially with the number of dimensions. With just 100 features (much less thanThe Curse of Dimensionality | 217

in the MNIST problem), you would need more training instances than atoms in the
observable universe in order for training instances to be within 0.1 of each other on
average, assuming they were spread out uniformly across all dimensions.
Main Approaches for Dimensionality Reduction
Before we dive into specific dimensionality reduction algorithms, let’s take a look at
the two main approaches to reducing dimensionality: projection and Manifold
Learning.
Projection
In most real-world problems, training instances are not spread out uniformly across
all dimensions. Many features are almost constant, while others are highly correlated
(as discussed earlier for MNIST). As a result, all training instances actually lie within
(or close to) a much lower-dimensional subspace  of the high-dimensional space. This
sounds very abstract, so let’s look at an example. In Figure 8-2  you can see a 3D data‐
set represented by the circles.
Figure 8-2. A 3D dataset lying close to a 2D subspace

Notice that all training instances lie close to a plane: this is a lower-dimensional (2D)
subspace of the high-dimensional (3D) space. Now if we project every training
instance perpendicularly onto this subspace (as represented by the short lines con‐
necting the instances to the plane), we get the new 2D dataset shown in Figure 8-3 .
Ta-da! We have just reduced the dataset’s dimensionality from 3D to 2D. Note that
the axes correspond to new features z1 and z2 (the coordinates of the projections on
the plane).218 | Chapter 8: Dimensionality Reduction

Figure 8-3. The new 2D dataset after  projection
However, projection is not always the best approach to dimensionality reduction. In
many cases the subspace may twist and turn, such as in the famous Swiss roll  toy data‐
set represented in Figure 8-4 .
Figure 8-4. Swiss roll datasetMain Approaches for Dimensionality Reduction | 219

Simply projecting onto a plane (e.g., by dropping x3) would squash different layers of
the Swiss roll together, as shown on the left of Figure 8-5 . However, what you really
want is to unroll the Swiss roll to obtain the 2D dataset on the right of Figure 8-5 .
Figure 8-5. Squashing by projecting onto a plane (left)  versus unrolling the Swiss roll
(right)
Manifold Learning
The Swiss roll is an example of a 2D manifold . Put simply, a 2D manifold is a 2D
shape that can be bent and twisted in a higher-dimensional space. More generally, a
d-dimensional manifold is a part of an n-dimensional space (where d < n) that locally
resembles a d-dimensional hyperplane. In the case of the Swiss roll, d = 2 and n = 3: it
locally resembles a 2D plane, but it is rolled in the third dimension.
Many dimensionality reduction algorithms work by modeling the manifold  on which
the training instances lie; this is called Manifold Learning . It relies on the manifold

assumption , also called the manifold hypothesis , which holds that most real-world
high-dimensional datasets lie close to a much lower-dimensional manifold. This
assumption is very often empirically observed.
Once again, think about the MNIST dataset: all handwritten digit images have some
similarities. They are made of connected lines, the borders are white, they are more
or less centered, and so on. If you randomly generated images, only a ridiculously

tiny fraction of them would look like handwritten digits. In other words, the degrees
of freedom available to you if you try to create a digit image are dramatically lower
than the degrees of freedom you would have if you were allowed to generate any
image you wanted. These constraints tend to squeeze the dataset into a lower-
dimensional manifold.
The manifold assumption is often accompanied by another implicit assumption: that
the task at hand (e.g., classification or regression) will be simpler if expressed in the

lower-dimensional space of the manifold. For example, in the top row of Figure 8-6
the Swiss roll is split into two classes: in the 3D space (on the left), the decision220 | Chapter 8: Dimensionality Reduction

boundary would be fairly complex, but in the 2D unrolled manifold space (on the
right), the decision boundary is a simple straight line.
However, this assumption does not always hold. For example, in the bottom row of
Figure 8-6 , the decision boundary is located at x1 = 5. This decision boundary looks
very simple in the original 3D space (a vertical plane), but it looks more complex in
the unrolled manifold (a collection of four independent line segments).
In short, if you reduce the dimensionality of your training set before training a
model, it will usually speed up training, but it may not always lead to a better or sim‐
pler solution; it all depends on the dataset.
Hopefully you now have a good sense of what the curse of dimensionality is and how
dimensionality reduction algorithms can fight it, especially when the manifold
assumption holds. The rest of this chapter will go through some of the most popular
algorithms.

Figure 8-6. The decision boundary may not always be simpler with lower dimensionsMain Approaches for Dimensionality Reduction | 221

4“On Lines and Planes of Closest Fit to Systems of Points in Space, ” K. Pearson (1901).PCA
Principal Component Analysis  (PCA) is by far the most popular dimensionality reduc‐
tion algorithm. First it identifies the hyperplane that lies closest to the data, and then
it projects the data onto it, just like in Figure 8-2 .
Preserving the Variance
Before you can project the training set onto a lower-dimensional hyperplane, you
first need to choose the right hyperplane. For example, a simple 2D dataset is repre‐
sented on the left of Figure 8-7 , along with three different axes (i.e., one-dimensional
hyperplanes). On the right is the result of the projection of the dataset onto each of
these axes. As you can see, the projection onto the solid line preserves the maximum
variance, while the projection onto the dotted line preserves very little variance, and
the projection onto the dashed line preserves an intermediate amount of variance.
Figure 8-7. Selecting the subspace onto which to project

It seems reasonable to select the axis that preserves the maximum amount of var‐
iance, as it will most likely lose less information than the other projections. Another
way to justify this choice is that it is the axis that minimizes the mean squared dis‐
tance between the original dataset and its projection onto that axis. This is the rather
simple idea behind PCA .4222 | Chapter 8: Dimensionality Reduction

Principal Components
PCA identifies the axis that accounts for the largest amount of variance in the train‐
ing set. In Figure 8-7 , it is the solid line. It also finds a second axis, orthogonal to the
first one, that accounts for the largest amount of remaining variance. In this 2D
example there is no choice: it is the dotted line. If it were a higher-dimensional data‐
set, PCA would also find a third axis, orthogonal to both previous axes, and a fourth,
a fifth, and so on—as many axes as the number of dimensions in the dataset.
The unit vector that defines the ith axis is called the ith principal component  (PC). In
Figure 8-7 , the 1st PC is c1 and the 2nd PC is c2. In Figure 8-2  the first two PCs are
represented by the orthogonal arrows in the plane, and the third PC would be
orthogonal to the plane (pointing up or down).
The direction of the principal components is not stable: if you per‐
turb the training set slightly and run PCA again, some of the new

PCs may point in the opposite direction of the original PCs. How‐
ever, they will generally still lie on the same axes. In some cases, a
pair of PCs may even rotate or swap, but the plane they define will
generally remain the same.
So how can you find the principal components of a training set? Luckily, there is a
standard matrix factorization technique called Singular Value Decomposition  (SVD)
that can decompose the training set matrix X into the matrix multiplication of three
matrices U Σ VT, where V contains all the principal components that we are looking

for, as shown in Equation 8-1 .
Equation 8-1. Principal components matrixV=∣ ∣ ∣
c1c2⋯cn
∣ ∣ ∣
The following Python code uses NumPy’s svd()  function to obtain all the principal
components of the training set, then extracts the first two PCs:
X_centered  = X - X.mean(axis=0)
U, s, Vt = np.linalg.svd(X_centered )
c1 = Vt.T[:, 0]
c2 = Vt.T[:, 1]PCA | 223

PCA assumes that the dataset is centered around the origin. As we
will see, Scikit-Learn’s PCA classes take care of centering the data
for you. However, if you implement PCA yourself (as in the pre‐
ceding example), or if you use other libraries, don’t forget to center
the data first.
Projecting Down to d Dimensions
Once you have identified all the principal components, you can reduce the dimen‐
sionality of the dataset down to d dimensions by projecting it onto the hyperplane
defined by the first d principal components. Selecting this hyperplane ensures that the
projection will preserve as much variance as possible. For example, in Figure 8-2  the
3D dataset is projected down to the 2D plane defined by the first two principal com‐
ponents, preserving a large part of the dataset’s variance. As a result, the 2D projec‐
tion looks very much like the original 3D dataset.
To project the training set onto the hyperplane, you can simply compute the matrix

multiplication of the training set matrix X by the matrix Wd, defined as the matrix
containing the first d principal components (i.e., the matrix composed of the first d
columns of V), as shown in Equation 8-2 .
Equation 8-2. Projecting the training set down to d dimensionsXd‐proj=XWd
The following Python code projects the training set onto the plane defined by the first
two principal components:
W2 = Vt.T[:, :2]
X2D = X_centered .dot(W2)
There you have it! Y ou now know how to reduce the dimensionality of any dataset

down to any number of dimensions, while preserving as much variance as possible.
Using Scikit-Learn
Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did
before. The following code applies PCA to reduce the dimensionality of the dataset
down to two dimensions (note that it automatically takes care of centering the data):
from sklearn.decomposition  import PCA
pca = PCA(n_components  = 2)
X2D = pca.fit_transform (X)
After fitting the PCA transformer to the dataset, you can access the principal compo‐

nents using the components_  variable (note that it contains the PCs as horizontal vec‐224 | Chapter 8: Dimensionality Reduction

tors, so, for example, the first principal component is equal to pca.components_.T[:,
0]).
Explained Variance Ratio
Another very useful piece of information is the explained variance ratio  of each prin‐
cipal component, available via the explained_variance_ratio_  variable. It indicates
the proportion of the dataset’s variance that lies along the axis of each principal com‐
ponent. For example, let’s look at the explained variance ratios of the first two compo‐
nents of the 3D dataset represented in Figure 8-2 :
>>> pca.explained_variance_ratio_
array([0.84248607, 0.14631839])
This tells you that 84.2% of the dataset’s variance lies along the first axis, and 14.6%
lies along the second axis. This leaves less than 1.2% for the third axis, so it is reason‐
able to assume that it probably carries little information.
Choosing the Right Number of Dimensions
Instead of arbitrarily choosing the number of dimensions to reduce down to, it is

generally preferable to choose the number of dimensions that add up to a sufficiently
large portion of the variance (e.g., 95%). Unless, of course, you are reducing dimen‐
sionality for data visualization—in that case you will generally want to reduce the
dimensionality down to 2 or 3.
The following code computes PCA without reducing dimensionality, then computes
the minimum number of dimensions required to preserve 95% of the training set’s
variance:
pca = PCA()
pca.fit(X_train)
cumsum = np.cumsum(pca.explained_variance_ratio_ )

variance:
pca = PCA()
d = np.argmax(cumsum >= 0.95) + 1
Y ou could then set n_components=d  and run PCA again. However, there is a much
better option: instead of specifying the number of principal components you want to
preserve, you can set n_components  to be a float between 0.0 and 1.0, indicating the
ratio of variance you wish to preserve:
pca = PCA(n_components =0.95)
X_reduced  = pca.fit_transform (X_train)
Y et another option is to plot the explained variance as a function of the number of
dimensions (simply plot cumsum ; see Figure 8-8 ). There will usually be an elbow in the

curve, where the explained variance stops growing fast. Y ou can think of this as the
intrinsic dimensionality of the dataset. In this case, you can see that reducing thePCA | 225

dimensionality down to about 100 dimensions wouldn’t lose too much explained var‐
iance.
Figure 8-8. Explained variance as a function of the number of dimensions
PCA for Compression
Obviously after dimensionality reduction, the training set takes up much less space.
For example, try applying PCA to the MNIST dataset while preserving 95% of its var‐
iance. Y ou should find that each instance will have just over 150 features, instead of
the original 784 features. So while most of the variance is preserved, the dataset is
now less than 20% of its original size! This is a reasonable compression ratio, and you
can see how this can speed up a classification algorithm (such as an SVM classifier)
tremendously.
It is also possible to decompress the reduced dataset back to 784 dimensions by
applying the inverse transformation of the PCA projection. Of course this won’t give
you back the original data, since the projection lost a bit of information (within the

5% variance that was dropped), but it will likely be quite close to the original data.
The mean squared distance between the original data and the reconstructed data
(compressed and then decompressed) is called the reconstruction error . For example,
the following code compresses the MNIST dataset down to 154 dimensions, then uses
the inverse_transform()  method to decompress it back to 784 dimensions.
Figure 8-9  shows a few digits from the original training set (on the left), and the cor‐
responding digits after compression and decompression. Y ou can see that there is a

slight image quality loss, but the digits are still mostly intact.
pca = PCA(n_components  = 154)
X_recovered  = pca.inverse_transform (X_reduced )226 | Chapter 8: Dimensionality Reduction

Figure 8-9. MNIST compression preserving 95% of the variance
The equation of the inverse transformation is shown in Equation 8-3 .
Equation 8-3. PCA inverse transformation, back to the original number of
dimensionsXrecovered=Xd‐projWdT
Randomized PCA
If you set the svd_solver  hyperparameter to "randomized" , Scikit-Learn uses a sto‐
chastic algorithm called Randomized PCA  that quickly finds an approximation of the
first d principal components. Its computational complexity is O(m × d2) + O(d3),
instead of O(m × n2) + O(n3) for the full SVD approach, so it is dramatically faster
than full SVD when d is much smaller than n:
rnd_pca = PCA(n_components =154, svd_solver ="randomized" )
X_reduced  = rnd_pca.fit_transform (X_train)
By default, svd_solver  is actually set to "auto" : Scikit-Learn automatically uses the
randomized PCA algorithm if m or n is greater than 500 and d is less than 80% of m
or n, or else it uses the full SVD approach. If you want to force Scikit-Learn to use full

SVD, you can set the svd_solver  hyperparameter to "full" .
Incremental PCA
One problem with the preceding implementations of PCA is that they require the
whole training set to fit in memory in order for the algorithm to run. Fortunately,
Incremental PCA  (IPCA) algorithms have been developed: you can split the training
set into mini-batches and feed an IPCA algorithm one mini-batch at a time. This isPCA | 227

5Scikit-Learn uses the algorithm described in “Incremental Learning for Robust Visual Tracking, ” D. Ross et al.
(2007).useful for large training sets, and also to apply PCA online (i.e., on the fly, as new
instances arrive).
The following code splits the MNIST dataset into 100 mini-batches (using NumPy’s
array_split()  function) and feeds them to Scikit-Learn’s IncrementalPCA  class5 to 
reduce the dimensionality of the MNIST dataset down to 154 dimensions (just like
before). Note that you must call the partial_fit()  method with each mini-batch
rather than the fit()  method with the whole training set:
from sklearn.decomposition  import IncrementalPCA
n_batches  = 100
inc_pca = IncrementalPCA (n_components =154)
for X_batch in np.array_split (X_train, n_batches ):
    inc_pca.partial_fit (X_batch)
X_reduced  = inc_pca.transform (X_train)
Alternatively, you can use NumPy’s memmap  class, which allows you to manipulate a
large array stored in a binary file on disk as if it were entirely in memory; the class

loads only the data it needs in memory, when it needs it. Since the IncrementalPCA
class uses only a small part of the array at any given time, the memory usage remains
under control. This makes it possible to call the usual fit()  method, as you can see
in the following code:
X_mm = np.memmap(filename , dtype="float32" , mode="readonly" , shape=(m, n))
batch_size  = m // n_batches
inc_pca = IncrementalPCA (n_components =154, batch_size =batch_size )
inc_pca.fit(X_mm)
Kernel PCA

Kernel PCA
In Chapter 5  we discussed the kernel trick, a mathematical technique that implicitly
maps instances into a very high-dimensional space (called the feature space ), enabling
nonlinear classification and regression with Support Vector Machines. Recall that a
linear decision boundary in the high-dimensional feature space corresponds to a
complex nonlinear decision boundary in the original space .
It turns out that the same trick can be applied to PCA, making it possible to perform

Kernel PCA
complex nonlinear projections for dimensionality reduction. This is called Kernel228 | Chapter 8: Dimensionality Reduction

6“Kernel Principal Component Analysis, ” B. Schölkopf, A. Smola, K. Müller (1999).PCA  (kPCA) .6 It is often good at preserving clusters of instances after projection, or
sometimes even unrolling datasets that lie close to a twisted manifold.
For example, the following code uses Scikit-Learn’s KernelPCA  class to perform kPCA
with an RBF kernel (see Chapter 5  for more details about the RBF kernel and the
other kernels):
from sklearn.decomposition  import KernelPCA
rbf_pca = KernelPCA (n_components  = 2, kernel="rbf", gamma=0.04)
X_reduced  = rbf_pca.fit_transform (X)
Figure 8-10  shows the Swiss roll, reduced to two dimensions using a linear kernel
(equivalent to simply using the PCA class), an RBF kernel, and a sigmoid kernel
(Logistic).
Figure 8-10. Swiss roll reduced to 2D using kPCA with various kernels
Selecting a Kernel and Tuning Hyperparameters
As kPCA is an unsupervised learning algorithm, there is no obvious performance
measure to help you select the best kernel and hyperparameter values. However,

dimensionality reduction is often a preparation step for a supervised learning task
(e.g., classification), so you can simply use grid search to select the kernel and hyper‐
parameters that lead to the best performance on that task. For example, the following
code creates a two-step pipeline, first reducing dimensionality to two dimensions
using kPCA, then applying Logistic Regression for classification. Then it uses Grid
SearchCV  to find the best kernel and gamma value for kPCA in order to get the best

classification accuracy at the end of the pipeline:
from sklearn.pipeline  import PipelineKernel PCA | 229

clf = Pipeline ([
        ("kpca", KernelPCA (n_components =2)),
        ("log_reg" , LogisticRegression ())
    ])
param_grid  = [{
        "kpca__gamma" : np.linspace (0.03, 0.05, 10),
        "kpca__kernel" : ["rbf", "sigmoid" ]
    }]
grid_search  = GridSearchCV (clf, param_grid , cv=3)
grid_search .fit(X, y)
The best kernel and hyperparameters are then available through the best_params_
variable:
>>> print(grid_search .best_params_ )
{'kpca__gamma': 0.043333333333333335, 'kpca__kernel': 'rbf'}
Another approach, this time entirely unsupervised, is to select the kernel and hyper‐
parameters that yield the lowest reconstruction error. However, reconstruction is not
as easy as with linear PCA. Here’s why. Figure 8-11  shows the original Swiss roll 3D
dataset (top left), and the resulting 2D dataset after kPCA is applied using an RBF
kernel (top right). Thanks to the kernel trick, this is mathematically equivalent to
mapping the training set to an infinite-dimensional feature space (bottom right)

using the feature map  φ, then projecting the transformed training set down to 2D
using linear PCA. Notice that if we could invert the linear PCA step for a given
instance in the reduced space, the reconstructed point would lie in feature space, not
in the original space (e.g., like the one represented by an x in the diagram). Since the
feature space is infinite-dimensional, we cannot compute the reconstructed point,
and therefore we cannot compute the true reconstruction error. Fortunately, it is pos‐

sible to find a point in the original space that would map close to the reconstructed
point. This is called the reconstruction pre-image . Once you have this pre-image, you
can measure its squared distance to the original instance. Y ou can then select the ker‐
nel and hyperparameters that minimize this reconstruction pre-image error.230 | Chapter 8: Dimensionality Reduction

7Scikit-Learn uses the algorithm based on Kernel Ridge Regression described in Gokhan H. Bakır, Jason
Weston, and Bernhard Scholkopf, “Learning to Find Pre-images”  (Tubingen, Germany: Max Planck Institute
for Biological Cybernetics, 2004).
Figure 8-11. Kernel PCA and the reconstruction pre-image error
Y ou may be wondering how to perform this reconstruction. One solution is to train a
supervised regression model, with the projected instances as the training set and the
original instances as the targets. Scikit-Learn will do this automatically if you set
fit_inverse_transform=True , as shown in the following code:7
rbf_pca = KernelPCA (n_components  = 2, kernel="rbf", gamma=0.0433,
                    fit_inverse_transform =True)
X_preimage  = rbf_pca.inverse_transform (X_reduced )
By default, fit_inverse_transform=False  and KernelPCA  has no
inverse_transform()  method. This method only gets created
when you set fit_inverse_transform=True .Kernel PCA | 231

8“Nonlinear Dimensionality Reduction by Locally Linear Embedding, ” S. Roweis, L. Saul (2000).Y ou can then compute the reconstruction pre-image error:
>>> mean_squared_error (X, X_preimage )
32.786308795766132
Now you can use grid search with cross-validation to find the kernel and hyperpara‐
meters that minimize this pre-image reconstruction error.
LLE
Locally Linear Embedding  (LLE)8 is another very powerful nonlinear dimensionality
reduction  (NLDR) technique. It is a Manifold Learning technique that does not rely
on projections like the previous algorithms. In a nutshell, LLE works by first measur‐
ing how each training instance linearly relates to its closest neighbors (c.n.), and then
looking for a low-dimensional representation of the training set where these local
relationships are best preserved (more details shortly). This makes it particularly
good at unrolling twisted manifolds, especially when there is not too much noise.

For example, the following code uses Scikit-Learn’s LocallyLinearEmbedding  class to
unroll the Swiss roll. The resulting 2D dataset is shown in Figure 8-12 . As you can
see, the Swiss roll is completely unrolled and the distances between instances are
locally well preserved. However, distances are not preserved on a larger scale: the left
part of the unrolled Swiss roll is stretched, while the right part is squeezed. Neverthe‐
less, LLE did a pretty good job at modeling the manifold.

from sklearn.manifold  import LocallyLinearEmbedding
lle = LocallyLinearEmbedding (n_components =2, n_neighbors =10)
X_reduced  = lle.fit_transform (X)232 | Chapter 8: Dimensionality Reduction

Figure 8-12. Unrolled Swiss roll using LLE
Here’s how LLE works: first, for each training instance x(i), the algorithm identifies its
k closest neighbors (in the preceding code k = 10), then tries to reconstruct x(i) as a
linear function of these neighbors. More specifically, it finds the weights wi,j such that
the squared distance between x(i) and ∑j= 1mwi,jxj
 is as small as possible, assuming wi,j
= 0 if x(j) is not one of the k closest neighbors of x(i). Thus the first step of LLE is the
constrained optimization problem described in Equation 8-4 , where W is the weight
matrix containing all the weights wi,j. The second constraint simply normalizes the
weights for each training instance x(i).LLE | 233

Equation 8-4. LLE step 1: linearly modeling local relationshipsW= argmin
W∑
i= 1m
xi−∑
j= 1m
wi,jxj2
subject towi,j= 0 if xjis not one of the kc.n. of xi
∑
j= 1m
wi,j= 1 for i= 1, 2,⋯,m
After this step, the weight matrix W (containing the weights wi,j) encodes the local
linear relationships between the training instances. Now the second step is to map the
training instances into a d-dimensional space (where d < n) while preserving these
local relationships as much as possible. If z(i) is the image of x(i) in this d-dimensional
space, then we want the squared distance between z(i) and ∑j= 1mwi,jzj
 to be as small
as possible. This idea leads to the unconstrained optimization problem described in
Equation 8-5 . It looks very similar to the first step, but instead of keeping the instan‐
ces fixed and finding the optimal weights, we are doing the reverse: keeping the
weights fixed and finding the optimal position of the instances’ images in the low-

dimensional space. Note that Z is the matrix containing all z(i).
Equation 8-5. LLE step 2: reducing dimensionality while preserving relationshipsZ= argmin
Z∑
i= 1m
zi−∑
j= 1m
wi,jzj2
Scikit-Learn’s LLE implementation has the following computational complexity:
O(m log(m)n log(k)) for finding the k nearest neighbors, O(mnk3) for optimizing the
weights, and O(dm2) for constructing the low-dimensional representations. Unfortu‐
nately, the m2 in the last term makes this algorithm scale poorly to very large datasets.
Other Dimensionality Reduction Techniques

Z∑
i= 1m
zi−∑
j= 1m
wi,jzj2
There are many other dimensionality reduction techniques, several of which are
available in Scikit-Learn. Here are some of the most popular:•
Multidimensional Scaling  (MDS) reduces dimensionality while trying to preserve
the distances between the instances (see Figure 8-13 ).234 | Chapter 8: Dimensionality Reduction

9The geodesic distance between two nodes in a graph is the number of nodes on the shortest path between
these nodes.•Isomap  creates a graph by connecting each instance to its nearest neighbors, then
reduces dimensionality while trying to preserve the geodesic distances9 between
the instances.•
t-Distributed Stochastic Neighbor Embedding  (t-SNE) reduces dimensionality
while trying to keep similar instances close and dissimilar instances apart. It is
mostly used for visualization, in particular to visualize clusters of instances in
high-dimensional space (e.g., to visualize the MNIST images in 2D).•
Linear Discriminant Analysis  (LDA) is actually a classification algorithm, but dur‐
ing training it learns the most discriminative axes between the classes, and these
axes can then be used to define a hyperplane onto which to project the data. The
benefit is that the projection will keep classes as far apart as possible, so LDA is a
good technique to reduce dimensionality before running another classification

algorithm such as an SVM classifier.
Figure 8-13. Reducing the Swiss roll to 2D using various techniques
What are the main motivations for reducing a dataset’s dimensionality? What are
the main drawbacks?2.
What is the curse of dimensionality?3.
Once a dataset’s dimensionality has been reduced, is it possible to reverse the
operation? If so, how? If not, why?4.
Can PCA be used to reduce the dimensionality of a highly nonlinear dataset?5.
Suppose you perform PCA on a 1,000-dimensional dataset, setting the explained

variance ratio to 95%. How many dimensions will the resulting dataset have?Exercises | 235

6. In what cases would you use vanilla PCA, Incremental PCA, Randomized PCA,
or Kernel PCA?7.
How can you evaluate the performance of a dimensionality reduction algorithm
on your dataset?8.
Does it make any sense to chain two different dimensionality reduction algo‐
rithms?9.
Load the MNIST dataset (introduced in Chapter 3 ) and split it into a training set
and a test set (take the first 60,000 instances for training, and the remaining
10,000 for testing). Train a Random Forest classifier on the dataset and time how
long it takes, then evaluate the resulting model on the test set. Next, use PCA to
reduce the dataset’s dimensionality, with an explained variance ratio of 95%.
Train a new Random Forest classifier on the reduced dataset and see how long it
takes. Was training much faster? Next evaluate the classifier on the test set: how
does it compare to the previous classifier?10.
Use t-SNE to reduce the MNIST dataset down to two dimensions and plot the

result using Matplotlib. Y ou can use a scatterplot using 10 different colors to rep‐
resent each image’s target class. Alternatively, you can write colored digits at the
location of each instance, or even plot scaled-down versions of the digit images
themselves (if you plot all digits, the visualization will be too cluttered, so you
should either draw a random sample or plot an instance only if no other instance
has already been plotted at a close distance). Y ou should get a nice visualization

with well-separated clusters of digits. Try using other dimensionality reduction
algorithms such as PCA, LLE, or MDS and compare the resulting visualizations.
Solutions to these exercises are available in ???.236 | Chapter 8: Dimensionality Reduction

CHAPTER 9
Unsupervised Learning Techniques
release of these titles. The following will be Chapter 9 in the final
Although most of the applications of Machine Learning today are based on super‐
vised learning (and as a result, this is where most of the investments go to), the vast
majority of the available data is actually unlabeled: we have the input features X, but
we do not have the labels y. Y ann LeCun famously said that “if intelligence was a cake,
unsupervised learning would be the cake, supervised learning would be the icing on
the cake, and reinforcement learning would be the cherry on the cake” . In other
words, there is a huge potential in unsupervised learning that we have only barely
started to sink our teeth into.

For example, say you want to create a system that will take a few pictures of each item
on a manufacturing production line and detect which items are defective. Y ou can
fairly easily create a system that will take pictures automatically, and this might give
you thousands of pictures every day. Y ou can then build a reasonably large dataset in
just a few weeks. But wait, there are no labels! If you want to train a regular binary
classifier that will predict whether an item is defective or not, you will need to label

every single picture as “defective” or “normal” . This will generally require human
experts to sit down and manually go through all the pictures. This is a long, costly
and tedious task, so it will usually only be done on a small subset of the available pic‐
tures. As a result, the labeled dataset will be quite small, and the classifier’s perfor‐
mance will be disappointing. Moreover, every time the company makes any change to
its products, the whole process will need to be started over from scratch. Wouldn’t it237

be great if the algorithm could just exploit the unlabeled data without needing
humans to label every picture? Enter unsupervised learning.
In Chapter 8 , we looked at the most common unsupervised learning task: dimension‐
ality reduction. In this chapter, we will look at a few more unsupervised learning tasks
and algorithms:•
Clustering : the goal is to group similar instances together into clusters . This is a
great tool for data analysis, customer segmentation, recommender systems,
search engines, image segmentation, semi-supervised learning, dimensionality
reduction, and more.•
Anomaly detection : the objective is to learn what “normal” data looks like, and
use this to detect abnormal instances, such as defective items on a production
line or a new trend in a time series.•
Density estimation : this is the task of estimating the probability density function
(PDF) of the random process that generated the dataset. This is commonly used

for anomaly detection: instances located in very low-density regions are likely to
be anomalies. It is also useful for data analysis and visualization.
Ready for some cake? We will start with clustering, using K-Means and DBSCAN,
and then we will discuss Gaussian mixture models and see how they can be used for
density estimation, clustering, and anomaly detection.
Clustering
As you enjoy a hike in the mountains, you stumble upon a plant you have never seen
before. Y ou look around and you notice a few more. They are not perfectly identical,

yet they are sufficiently similar for you to know that they most likely belong to the
same species (or at least the same genus). Y ou may need a botanist to tell you what
species that is, but you certainly don’t need an expert to identify groups of similar-
looking objects. This is called clustering : it is the task of identifying similar instances
and assigning them to clusters , i.e., groups of similar instances.
Just like in classification, each instance gets assigned to a group. However, this is an

unsupervised task. Consider Figure 9-1 : on the left is the iris dataset (introduced in
Chapter 4 ), where each instance’s species (i.e., its class) is represented with a different
marker. It is a labeled dataset, for which classification algorithms such as Logistic
Regression, SVMs or Random Forest classifiers are well suited. On the right is the
same dataset, but without the labels, so you cannot use a classification algorithm any‐

more. This is where clustering algorithms step in: many of them can easily detect the
top left cluster. It is also quite easy to see with our own eyes, but it is not so obvious
that the lower right cluster is actually composed of two distinct sub-clusters. That
said, the dataset actually has two additional features (sepal length and width), not238 | Chapter 9: Unsupervised Learning Techniques

represented here, and clustering algorithms can make good use of all features, so in
fact they identify the three clusters fairly well (e.g., using a Gaussian mixture model,
only 5 instances out of 150 are assigned to the wrong cluster).
Figure 9-1. Classification  (left)  versus clustering (right)
Clustering is used in a wide variety of applications, including:•
For customer segmentation: you can cluster your customers based on their pur‐
chases, their activity on your website, and so on. This is useful to understand who
your customers are and what they need, so you can adapt your products and
marketing campaigns to each segment. For example, this can be useful in recom‐
mender systems  to suggest content that other users in the same cluster enjoyed.•
For data analysis: when analyzing a new dataset, it is often useful to first discover
clusters of similar instances, as it is often easier to analyze clusters separately.•
As a dimensionality reduction technique: once a dataset has been clustered, it is

usually possible to measure each instance’s affinity  with each cluster (affinity is
any measure of how well an instance fits into a cluster). Each instance’s feature
vector x can then be replaced with the vector of its cluster affinities. If there are k
clusters, then this vector is k dimensional. This is typically much lower dimen‐
sional than the original feature vector, but it can preserve enough information for
further processing.•
For anomaly detection  (also called outlier detection ): any instance that has a low

affinity to all the clusters is likely to be an anomaly. For example, if you have clus‐
tered the users of your website based on their behavior, you can detect users with
unusual behavior, such as an unusual number of requests per second, and so on.
Anomaly detection is particularly useful in detecting defects in manufacturing, or
for fraud detection .•
For semi-supervised learning: if you only have a few labels, you could perform
clustering and propagate the labels to all the instances in the same cluster. This

can greatly increase the amount of labels available for a subsequent supervised
learning algorithm, and thus improve its performance.Clustering | 239

1“Least square quantization in PCM, ” Stuart P . Lloyd. (1982).• For search engines: for example, some search engines let you search for images
that are similar to a reference image. To build such a system, you would first
apply a clustering algorithm to all the images in your database: similar images
would end up in the same cluster. Then when a user provides a reference image,
all you need to do is to find this image’s cluster using the trained clustering
model, and you can then simply return all the images from this cluster.•
To segment an image: by clustering pixels according to their color, then replacing
each pixel’s color with the mean color of its cluster, it is possible to reduce the
number of different colors in the image considerably. This technique is used in
many object detection and tracking systems, as it makes it easier to detect the
contour of each object.
There is no universal definition of what a cluster is: it really depends on the context,

and different algorithms will capture different kinds of clusters. For example, some
algorithms look for instances centered around a particular point, called a centroid .
Others look for continuous regions of densely packed instances: these clusters can
take on any shape. Some algorithms are hierarchical, looking for clusters of clusters.
And the list goes on.
In this section, we will look at two popular clustering algorithms: K-Means and
DBSCAN, and we will show some of their applications, such as non-linear dimen‐

sionality reduction, semi-supervised learning and anomaly detection.
K-Means
Consider the unlabeled dataset represented in Figure 9-2 : you can clearly see 5 blobs
of instances. The K-Means algorithm is a simple algorithm capable of clustering this
kind of dataset very quickly and efficiently, often in just a few iterations. It was pro‐
posed by Stuart Lloyd at the Bell Labs in 1957 as a technique for pulse-code modula‐
tion, but it was only published outside of the company in 1982, in a paper titled

K-Means
“Least square quantization in PCM” .1 By then, in 1965, Edward W . Forgy had pub‐
lished virtually the same algorithm, so K-Means is sometimes referred to as Lloyd-
Forgy.240 | Chapter 9: Unsupervised Learning Techniques

Figure 9-2. An unlabeled dataset composed of five blobs of instances
Let’s train a K-Means clusterer on this dataset. It will try to find each blob’s center and
assign each instance to the closest blob:
from sklearn.cluster  import KMeans
k = 5
kmeans = KMeans(n_clusters =k)
y_pred = kmeans.fit_predict (X)
Note that you have to specify the number of clusters k that the algorithm must find.
In this example, it is pretty obvious from looking at the data that k should be set to 5,
but in general it is not that easy. We will discuss this shortly.
Each instance was assigned to one of the 5 clusters. In the context of clustering, an
instance’s label  is the index of the cluster that this instance gets assigned to by the
algorithm: this is not to be confused with the class labels in classification (remember
that clustering is an unsupervised learning task). The KMeans  instance preserves a
copy of the labels of the instances it was trained on, available via the labels_  instance
variable:
>>> y_pred

variable:
>>> y_pred
array([4, 0, 1, ..., 2, 1, 0], dtype=int32)
>>> y_pred is kmeans.labels_
True
We can also take a look at the 5 centroids that the algorithm found:
>>> kmeans.cluster_centers_
array([[-2.80389616,  1.80117999],
       [ 0.20876306,  2.25551336],
       [-2.79290307,  2.79641063],
       [-1.46679593,  2.28585348],
       [-2.80037642,  1.30082566]])
Of course, you can easily assign new instances to the cluster whose centroid is closest:Clustering | 241

>>> X_new = np.array([[0, 2], [3, 2], [-3, 3], [-3, 2.5]])
>>> kmeans.predict(X_new)
array([1, 1, 2, 2], dtype=int32)
If you plot the cluster’s decision boundaries, you get a Voronoi tessellation (see
Figure 9-3 , where each centroid is represented with an X):
Figure 9-3. K-Means decision boundaries (Voronoi tessellation)
The vast majority of the instances were clearly assigned to the appropriate cluster, but
a few instances were probably mislabeled (especially near the boundary between the
top left cluster and the central cluster). Indeed, the K-Means algorithm does not
behave very well when the blobs have very different diameters since all it cares about
when assigning an instance to a cluster is the distance to the centroid.
Instead of assigning each instance to a single cluster, which is called hard clustering , it
can be useful to just give each instance a score per cluster: this is called soft clustering .
For example, the score can be the distance between the instance and the centroid, or

conversely it can be a similarity score (or affinity) such as the Gaussian Radial Basis
Function (introduced in Chapter 5 ). In the KMeans  class, the transform()  method
measures the distance from each instance to every centroid:
>>> kmeans.transform (X_new)
array([[2.81093633, 0.32995317, 2.9042344 , 1.49439034, 2.88633901],
       [5.80730058, 2.80290755, 5.84739223, 4.4759332 , 5.84236351],
       [1.21475352, 3.29399768, 0.29040966, 1.69136631, 1.71086031],
       [0.72581411, 3.21806371, 0.36159148, 1.54808703, 1.21567622]])

In this example, the first instance in X_new  is located at a distance of 2.81 from the
first centroid, 0.33 from the second centroid, 2.90 from the third centroid, 1.49 from
the fourth centroid and 2.87 from the fifth centroid. If you have a high-dimensional
dataset and you transform it this way, you end up with a k-dimensional dataset: this
can be a very efficient non-linear dimensionality reduction technique.242 | Chapter 9: Unsupervised Learning Techniques

2This can be proven by pointing out that the mean squared distance between the instances and their closest
centroid can only go down at each step.The K-Means Algorithm
So how does the algorithm work? Well it is really quite simple. Suppose you were
given the centroids: you could easily label all the instances in the dataset by assigning
each of them to the cluster whose centroid is closest. Conversely, if you were given all
the instance labels, you could easily locate all the centroids by computing the mean of
the instances for each cluster. But you are given neither the labels nor the centroids,
so how can you proceed? Well, just start by placing the centroids randomly (e.g., by
picking k instances at random and using their locations as centroids). Then label the
instances, update the centroids, label the instances, update the centroids, and so on
until the centroids stop moving. The algorithm is guaranteed to converge in a finite

number of steps (usually quite small), it will not oscillate forever2. Y ou can see the
algorithm in action in Figure 9-4 : the centroids are initialized randomly (top left),
then the instances are labeled (top right), then the centroids are updated (center left),
the instances are relabeled (center right), and so on. As you can see, in just 3 itera‐
tions the algorithm has reached a clustering that seems close to optimal.
Figure 9-4. The K-Means algorithmClustering | 243

The computational complexity of the algorithm is generally linear
with regards to the number of instances m, the number of clusters
k and the number of dimensions n. However, this is only true when
the data has a clustering structure. If it does not, then in the worst
case scenario the complexity can increase exponentially with the
number of instances. In practice, however, this rarely happens, and
K-Means is generally one of the fastest clustering algorithms.
Unfortunately, although the algorithm is guaranteed to converge, it may not converge
to the right solution (i.e., it may converge to a local optimum): this depends on the
centroid initialization. For example, Figure 9-5  shows two sub-optimal solutions that
the algorithm can converge to if you are not lucky with the random initialization step:
Figure 9-5. Sub-optimal solutions due to unlucky centroid initializations
Let’s look at a few ways you can mitigate this risk by improving the centroid initializa‐
tion.
Centroid Initialization Methods

tion.
If you happen to know approximately where the centroids should be (e.g., if you ran
another clustering algorithm earlier), then you can set the init  hyperparameter to a
NumPy array containing the list of centroids, and set n_init  to 1:
good_init  = np.array([[-3, 3], [-3, 2], [-3, 1], [-1, 2], [0, 2]])
kmeans = KMeans(n_clusters =5, init=good_init , n_init=1)
Another solution is to run the algorithm multiple times with different random initial‐
izations and keep the best solution. This is controlled by the n_init  hyperparameter:

by default, it is equal to 10, which means that the whole algorithm described earlier
actually runs 10 times when you call fit() , and Scikit-Learn keeps the best solution.
But how exactly does it know which solution is the best? Well of course it uses a per‐
formance metric! It is called the model’s inertia : this is the mean squared distance
between each instance and its closest centroid. It is roughly equal to 223.3 for the
model on the left of Figure 9-5 , 237.5 for the model on the right of Figure 9-5 , and

211.6 for the model in Figure 9-3 . The KMeans  class runs the algorithm n_init  times
and keeps the model with the lowest inertia: in this example, the model in Figure 9-3
will be selected (unless we are very unlucky with n_init  consecutive random initiali‐244 | Chapter 9: Unsupervised Learning Techniques

3“k-means\++: The advantages of careful seeding, ” David Arthur and Sergei Vassilvitskii (2006).4
“Using the Triangle Inequality to Accelerate k-Means, ” Charles Elkan (2003).zations). If you are curious, a model’s inertia is accessible via the inertia_  instance
variable:
>>> kmeans.inertia_
211.59853725816856
The score()  method returns the negative inertia. Why negative? Well, it is because a
predictor’s score()  method must always respect the " great is better " rule.
>>> kmeans.score(X)
-211.59853725816856
An important improvement to the K-Means algorithm, called K-Means+\+ , was pro‐
posed in a 2006 paper  by David Arthur and Sergei Vassilvitskii:3 they introduced a
smarter initialization step that tends to select centroids that are distant from one
another, and this makes the K-Means algorithm much less likely to converge to a sub-
optimal solution. They showed that the additional computation required for the
smarter initialization step is well worth it since it makes it possible to drastically

The KMeans  class actually uses this initialization method by default. If you want to
force it to use the original method (i.e., picking k instances randomly to define the
initial centroids), then you can set the init  hyperparameter to "random" . Y ou will
rarely need to do this.
Accelerated K-Means and Mini-batch K-Means
Another important improvement to the K-Means algorithm was proposed in a 2003
paper  by Charles Elkan.4 It considerably accelerates the algorithm by avoiding many
unnecessary distance calculations: this is achieved by exploiting the triangle inequal‐Clustering | 245

5The triangle inequality is AC ≤ AB + BC where A, B and C are three points, and AB, AC and BC are the
distances between these points.6
“Web-Scale K-Means Clustering, ” David Sculley (2010).ity (i.e., the straight line is always the shortest5) and by keeping track of lower and
upper bounds for distances between instances and centroids. This is the algorithm
used by default by the KMeans  class (but you can force it to use the original algorithm
by setting the algorithm  hyperparameter to "full" , although you probably will
never need to).
Y et another important variant of the K-Means algorithm was proposed in a 2010
paper  by David Sculley.6 Instead of using the full dataset at each iteration, the algo‐
rithm is capable of using mini-batches, moving the centroids just slightly at each iter‐
ation. This speeds up the algorithm typically by a factor of 3 or 4 and makes it
possible to cluster huge datasets that do not fit in memory. Scikit-Learn implements

this algorithm in the MiniBatchKMeans  class. Y ou can just use this class like the
KMeans  class:
from sklearn.cluster  import MiniBatchKMeans
minibatch_kmeans  = MiniBatchKMeans (n_clusters =5)
minibatch_kmeans .fit(X)
If the dataset does not fit in memory, the simplest option is to use the memmap  class, as
we did for incremental PCA in Chapter 8 . Alternatively, you can pass one mini-batch
at a time to the partial_fit()  method, but this will require much more work, since
you will need to perform multiple initializations and select the best one yourself (see

the notebook for an example).
Although the Mini-batch K-Means algorithm is much faster than the regular K-
Means algorithm, its inertia is generally slightly worse, especially as the number of
clusters increases. Y ou can see this in Figure 9-6 : the plot on the left compares the
inertias of Mini-batch K-Means and regular K-Means models trained on the previous
dataset using various numbers of clusters k. The difference between the two curves
remains fairly constant, but this difference becomes more and more significant as k

increases, since the inertia becomes smaller and smaller. However, in the plot on the
right, you can see that Mini-batch K-Means is much faster than regular K-Means, and
this difference increases with k.246 | Chapter 9: Unsupervised Learning Techniques

Figure 9-6. Mini-batch K-Means vs K-Means: worse inertia as k increases (left)  but
much faster (right)
Finding the Optimal Number of Clusters
So far, we have set the number of clusters k to 5 because it was obvious by looking at
the data that this is the correct number of clusters. But in general, it will not be so
easy to know how to set k, and the result might be quite bad if you set it to the wrong
value. For example, as you can see in Figure 9-7 , setting k to 3 or 8 results in fairly
bad models:
Figure 9-7. Bad choices for the number of clusters
Y ou might be thinking that we could just pick the model with the lowest inertia,
right? Unfortunately, it is not that simple. The inertia for k=3 is 653.2, which is much
higher than for k=5 (which was 211.6), but with k=8, the inertia is just 119.1. The
inertia is not a good performance metric when trying to choose k since it keeps get‐
ting lower as we increase k. Indeed, the more clusters there are, the closer each

instance will be to its closest centroid, and therefore the lower the inertia will be. Let’s
plot the inertia as a function of k (see Figure 9-8 ):Clustering | 247

Figure 9-8. Selecting the number of clusters k using the “elbow rule”
As you can see, the inertia drops very quickly as we increase k up to 4, but then it
decreases much more slowly as we keep increasing k. This curve has roughly the
shape of an arm, and there is an “elbow” at k=4 so if we did not know better, it would
be a good choice: any lower value would be dramatic, while any higher value would
not help much, and we might just be splitting perfectly good clusters in half for no
good reason.
This technique for choosing the best value for the number of clusters is rather coarse.
A more precise approach (but also more computationally expensive) is to use the sil‐
houette score , which is the mean silhouette coefficient  over all the instances. An instan‐
ce’s silhouette coefficient is equal to ( b – a) / max( a, b) where a is the mean distance
to the other instances in the same cluster (it is the mean intra-cluster distance), and b

is the mean nearest-cluster distance, that is the mean distance to the instances of the
next closest cluster (defined as the one that minimizes b, excluding the instance’s own
cluster). The silhouette coefficient can vary between -1 and +1: a coefficient close to
+1 means that the instance is well inside its own cluster and far from other clusters,
while a coefficient close to 0 means that it is close to a cluster boundary, and finally a
coefficient close to -1 means that the instance may have been assigned to the wrong

cluster. To compute the silhouette score, you can use Scikit-Learn’s silhou
ette_score()  function, giving it all the instances in the dataset, and the labels they
were assigned:
>>> from sklearn.metrics  import silhouette_score
>>> silhouette_score (X, kmeans.labels_)
0.655517642572828
Let’s compare the silhouette scores for different numbers of clusters (see Figure 9-9 ):248 | Chapter 9: Unsupervised Learning Techniques

Figure 9-9. Selecting the number of clusters k using the silhouette score
As you can see, this visualization is much richer than the previous one: in particular,
although it confirms that k=4 is a very good choice, it also underlines the fact that
k=5 is quite good as well, and much better than k=6 or 7. This was not visible when
comparing inertias.
An even more informative visualization is obtained when you plot every instance’s
silhouette coefficient, sorted by the cluster they are assigned to and by the value of the
coefficient. This is called a silhouette diagram  (see Figure 9-10 ):
Figure 9-10. Silouhette analysis: comparing the silhouette diagrams for various values of
k
The vertical dashed lines represent the silhouette score for each number of clusters.
When most of the instances in a cluster have a lower coefficient than this score (i.e., if
many of the instances stop short of the dashed line, ending to the left of it), then the

k
cluster is rather bad since this means its instances are much too close to other clus‐Clustering | 249

ters. We can see that when k=3 and when k=6, we get bad clusters. But when k=4 or
k=5, the clusters look pretty good – most instances extend beyond the dashed line, to
the right and closer to 1.0. When k=4, the cluster at index 1 (the third from the top),
is rather big, while when k=5, all clusters have similar sizes, so even though the over‐
all silhouette score from k=4 is slightly greater than for k=5, it seems like a good idea
to use k=5 to get clusters of similar sizes.
Limits of K-Means
Despite its many merits, most notably being fast and scalable, K-Means is not perfect.
As we saw, it is necessary to run the algorithm several times to avoid sub-optimal sol‐
utions, plus you need to specify the number of clusters, which can be quite a hassle.
Moreover, K-Means does not behave very well when the clusters have varying sizes,
different densities, or non-spherical shapes. For example, Figure 9-11  shows how K-
Means clusters a dataset containing three ellipsoidal clusters of different dimensions,

densities and orientations:
Figure 9-11. K-Means fails to cluster these ellipsoidal blobs properly
As you can see, neither of these solutions are any good. The solution on the left is
better, but it still chops off 25% of the middle cluster and assigns it to the cluster on
the right. The solution on the right is just terrible, even though its inertia is lower. So
depending on the data, different clustering algorithms may perform better. For exam‐
ple, on these types of elliptical clusters, Gaussian mixture models work great.

It is important to scale the input features before you run K-Means,
or else the clusters may be very stretched, and K-Means will per‐
form poorly. Scaling the features does not guarantee that all the
clusters will be nice and spherical, but it generally improves things.
Now let’s look at a few ways we can benefit from clustering. We will use K-Means, but
feel free to experiment with other clustering algorithms.250 | Chapter 9: Unsupervised Learning Techniques

Using clustering for image segmentation
Image segmentation  is the task of partitioning an image into multiple segments. In
semantic segmentation , all pixels that are part of the same object type get assigned to
the same segment. For example, in a self-driving car’s vision system, all pixels that are
part of a pedestrian’s image might be assigned to the “pedestrian” segment (there
would just be one segment containing all the pedestrians). In instance segmentation ,
all pixels that are part of the same individual object are assigned to the same segment.
In this case there would be a different segment for each pedestrian. The state of the
art in semantic or instance segmentation today is achieved using complex architec‐
tures based on convolutional neural networks (see Chapter 14 ). Here, we are going to
do something much simpler: color segmentation . We will simply assign pixels to the
same segment if they have a similar color. In some applications, this may be sufficient,

for example if you want to analyze satellite images to measure how much total forest
area there is in a region, color segmentation may be just fine.
First, let’s load the image (see the upper left image in Figure 9-12 ) using Matplotlib’s
imread()  function:
>>> from matplotlib.image  import imread  # you could also use `imageio.imread()`
>>> image = imread(os.path.join("images" ,"clustering" ,"ladybug.png" ))
>>> image.shape
(533, 800, 3)
The image is represented as a 3D array: the first dimension’s size is the height, the

second is the width, and the third is the number of color channels, in this case red,
green and blue (RGB). In other words, for each pixel there is a 3D vector containing
the intensities of red, green and blue, each between 0.0 and 1.0 (or between 0 and 255
if you use imageio.imread() ). Some images may have less channels, such as gray‐
scale images (one channel), or more channels, such as images with an additional
alpha channel  for transparency, or satellite images which often contain channels for

many light frequencies (e.g., infrared). The following code reshapes the array to get a
long list of RGB colors, then it clusters these colors using K-Means. For example, it
may identify a color cluster for all shades of green. Next, for each color (e.g., dark
green), it looks for the mean color of the pixel’s color cluster. For example, all shades
of green may be replaced with the same light green color (assuming the mean color of

the green cluster is light green). Finally it reshapes this long list of colors to get the
same shape as the original image. And we’re done!
X = image.reshape(-1, 3)
kmeans = KMeans(n_clusters =8).fit(X)
segmented_img  = kmeans.cluster_centers_ [kmeans.labels_]
segmented_img  = segmented_img .reshape(image.shape)
This outputs the image shown in the upper right of Figure 9-12 . Y ou can experiment
with various numbers of clusters, as shown in the figure. When you use less than 8

clusters, notice that the ladybug’s flashy red color fails to get a cluster of its own: itClustering | 251

gets merged with colors from the environment. This is due to the fact that the lady‐
bug is quite small, much smaller than the rest of the image, so even though its color is
flashy, K-Means fails to dedicate a cluster to it: as mentioned earlier, K-Means prefers
clusters of similar sizes.
Figure 9-12. Image segmentation using K-Means with various numbers of color clusters
That was not too hard, was it? Now let’s look at another application of clustering: pre‐
Using Clustering for Preprocessing
Clustering can be an efficient approach to dimensionality reduction, in particular as a
preprocessing step before a supervised learning algorithm. For example, let’s tackle
the digits dataset  which is a simple MNIST-like dataset containing 1,797 grayscale 8×8
images representing digits 0 to 9. First, let’s load the dataset:
from sklearn.datasets  import load_digits
X_digits , y_digits  = load_digits (return_X_y =True)
Now, let’s split it into a training set and a test set:

X_train, X_test, y_train, y_test = train_test_split (X_digits , y_digits )
Next, let’s fit a Logistic Regression model:
log_reg = LogisticRegression (random_state =42)
log_reg.fit(X_train, y_train)
Let’s evaluate its accuracy on the test set:
>>> log_reg.score(X_test, y_test)
0.9666666666666667252 | Chapter 9: Unsupervised Learning Techniques

Okay, that’s our baseline: 96.7% accuracy. Let’s see if we can do better by using K-
Means as a preprocessing step. We will create a pipeline that will first cluster the
training set into 50 clusters and replace the images with their distances to these 50
clusters, then apply a logistic regression model.
Although it is tempting to define the number of clusters to 10,
since there are 10 different digits, it is unlikely to perform well,
because there are several different ways to write each digit.
pipeline  = Pipeline ([
    ("kmeans" , KMeans(n_clusters =50)),
    ("log_reg" , LogisticRegression ()),
])
pipeline .fit(X_train, y_train)
Now let’s evaluate this classification pipeline:
>>> pipeline .score(X_test, y_test)
0.9822222222222222
How about that? We almost divided the error rate by a factor of 2!
But we chose the number of clusters k completely arbitrarily, we can surely do better.

])
Since K-Means is just a preprocessing step in a classification pipeline, finding a good
value for k is much simpler than earlier: there’s no need to perform silhouette analysis
or minimize the inertia, the best value of k is simply the one that results in the best
classification performance during cross-validation. Let’s use GridSearchCV  to find the
optimal number of clusters:
param_grid  = dict(kmeans__n_clusters =range(2, 100))

grid_clf  = GridSearchCV (pipeline , param_grid , cv=3, verbose=2)
grid_clf .fit(X_train, y_train)
Let’s look at best value for k, and the performance of the resulting pipeline:
>>> grid_clf .best_params_
{'kmeans__n_clusters': 90}
>>> grid_clf .score(X_test, y_test)
0.9844444444444445
With k=90 clusters, we get a small accuracy boost, reaching 98.4% accuracy on the
test set. Cool!Clustering | 253

Using Clustering for Semi-Supervised Learning
Another use case for clustering is in semi-supervised learning, when we have plenty
of unlabeled instances and very few labeled instances. Let’s train a logistic regression
model on a sample of 50 labeled instances from the digits dataset:
n_labeled  = 50
log_reg.fit(X_train[:n_labeled ], y_train[:n_labeled ])
What is the performance of this model on the test set?
0.8266666666666667
The accuracy is just 82.7%: it should come as no surprise that this is much lower than
earlier, when we trained the model on the full training set. Let’s see how we can do
better. First, let’s cluster the training set into 50 clusters, then for each cluster let’s find
the image closest to the centroid. We will call these images the representative images:
k = 50
X_digits_dist  = kmeans.fit_transform (X_train)
representative_digit_idx  = np.argmin(X_digits_dist , axis=0)

k = 50
X_representative_digits  = X_train[representative_digit_idx ]
Figure 9-13  shows these 50 representative images:
Figure 9-13. Fifty  representative digit images (one per cluster)
Now let’s look at each image and manually label it:
y_representative_digits  = np.array([4, 8, 0, 6, 8, 3, ..., 7, 6, 2, 3, 1, 1])
Now we have a dataset with just 50 labeled instances, but instead of being completely
random instances, each of them is a representative image of its cluster. Let’s see if the
performance is any better:

>>> log_reg = LogisticRegression ()
>>> log_reg.fit(X_representative_digits , y_representative_digits )
0.9244444444444444
Wow! We jumped from 82.7% accuracy to 92.4%, although we are still only training
the model on 50 instances. Since it is often costly and painful to label instances, espe‐254 | Chapter 9: Unsupervised Learning Techniques

cially when it has to be done manually by experts, it is a good idea to label representa‐
tive instances rather than just random instances.
But perhaps we can go one step further: what if we propagated the labels to all the
other instances in the same cluster? This is called label propagation :
y_train_propagated  = np.empty(len(X_train), dtype=np.int32)
for i in range(k):
    y_train_propagated [kmeans.labels_==i] = y_representative_digits [i]
Now let’s train the model again and look at its performance:
>>> log_reg.fit(X_train, y_train_propagated )
0.9288888888888889
We got a tiny little accuracy boost. Better than nothing, but not astounding. The
problem is that we propagated each representative instance’s label to all the instances
in the same cluster, including the instances located close to the cluster boundaries,
which are more likely to be mislabeled. Let’s see what happens if we only propagate

the labels to the 20% of the instances that are closest to the centroids:
percentile_closest  = 20
X_cluster_dist  = X_digits_dist [np.arange(len(X_train)), kmeans.labels_]
    in_cluster  = (kmeans.labels_ == i)
    cluster_dist  = X_cluster_dist [in_cluster ]
    cutoff_distance  = np.percentile (cluster_dist , percentile_closest )
    above_cutoff  = (X_cluster_dist  > cutoff_distance )
    X_cluster_dist [in_cluster  & above_cutoff ] = -1
partially_propagated  = (X_cluster_dist  != -1)

X_train_partially_propagated  = X_train[partially_propagated ]
y_train_partially_propagated  = y_train_propagated [partially_propagated ]
Now let’s train the model again on this partially propagated dataset:
>>> log_reg.fit(X_train_partially_propagated , y_train_partially_propagated )
0.9422222222222222
Nice! With just 50 labeled instances (only 5 examples per class on average!), we got
94.2% performance, which is pretty close to the performance of logistic regression on

the fully labeled digits dataset (which was 96.7%). This is because the propagated
labels are actually pretty good, their accuracy is very close to 99%:Clustering | 255

>>> np.mean(y_train_partially_propagated  == y_train[partially_propagated ])
0.9896907216494846
Active Learning
To continue improving your model and your training set, the next step could be to do
a few rounds of active learning : this is when a human expert interacts with the learn‐
ing algorithm, providing labels when the algorithm needs them. There are many dif‐
ferent strategies for active learning, but one of the most common ones is called
uncertainty sampling :•
The model is trained on the labeled instances gathered so far, and this model is
used to make predictions on all the unlabeled instances.•
The instances for which the model is most uncertain (i.e., when its estimated
probability is lowest) must be labeled by the expert.•
Then you just iterate this process again and again, until the performance
improvement stops being worth the labeling effort.
Other strategies include labeling the instances that would result in the largest model

change, or the largest drop in the model’s validation error, or the instances that differ‐
ent models disagree on (e.g., an SVM, a Random Forest, and so on).
Before we move on to Gaussian mixture models, let’s take a look at DBSCAN,
another popular clustering algorithm that illustrates a very different approach based
on local density estimation. This approach allows the algorithm to identify clusters of
arbitrary shapes.
DBSCAN

DBSCAN
This algorithm defines clusters as continuous regions of high density. It is actually
quite simple:•
For each instance, the algorithm counts how many instances are located within a
small distance ε (epsilon) from it. This region is called the instance’s ε-
neighborhood .•
If an instance has at least min_samples  instances in its ε-neighborhood (includ‐
ing itself), then it is considered a core instance . In other words, core instances are
those that are located in dense regions.•
All instances in the neighborhood of a core instance belong to the same cluster.

This may include other core instances, therefore a long sequence of neighboring
core instances forms a single cluster.256 | Chapter 9: Unsupervised Learning Techniques

• Any instance that is not a core instance and does not have one in its neighbor‐
hood is considered an anomaly.
This algorithm works well if all the clusters are dense enough, and they are well sepa‐
rated by low-density regions. The DBSCAN  class in Scikit-Learn is as simple to use as
you might expect. Let’s test it on the moons dataset, introduced in Chapter 5 :
from sklearn.cluster  import DBSCAN
X, y = make_moons (n_samples =1000, noise=0.05)
dbscan = DBSCAN(eps=0.05, min_samples =5)
dbscan.fit(X)
The labels of all the instances are now available in the labels_  instance variable:
>>> dbscan.labels_
array([ 0,  2, -1, -1,  1,  0,  0,  0, ...,  3,  2,  3,  3,  4,  2,  6,  3])
Notice that some instances have a cluster index equal to -1: this means that they are
considered as anomalies by the algorithm. The indices of the core instances are avail‐
able in the core_sample_indices_  instance variable, and the core instances them‐

selves are available in the components_  instance variable:
>>> len(dbscan.core_sample_indices_ )
808
>>> dbscan.core_sample_indices_
array([ 0,  4,  5,  6,  7,  8, 10, 11, ..., 992, 993, 995, 997, 998, 999])
>>> dbscan.components_
array([[-0.02137124,  0.40618608],
       [-0.84192557,  0.53058695],
                  ...
       [-0.94355873,  0.3278936 ],
       [ 0.79419406,  0.60777171]])
This clustering is represented in the left plot of Figure 9-14 . As you can see, it identi‐

808
                  ...
fied quite a lot of anomalies, plus 7 different clusters. How disappointing! Fortunately,
if we widen each instance’s neighborhood by increasing eps to 0.2, we get the cluster‐
ing on the right, which looks perfect. Let’s continue with this model.
Figure 9-14. DBSCAN clustering using two different  neighborhood radiusesClustering | 257

Somewhat surprisingly, the DBSCAN class does not have a predict()  method,
although it has a fit_predict()  method. In other words, it cannot predict which
cluster a new instance belongs to. The rationale for this decision is that several classi‐
fication algorithms could make sense here, and it is easy enough to train one, for
example a KNeighborsClassifier :
knn = KNeighborsClassifier (n_neighbors =50)
knn.fit(dbscan.components_ , dbscan.labels_[dbscan.core_sample_indices_ ])
Now, given a few new instances, we can predict which cluster they most likely belong
to, and even estimate a probability for each cluster. Note that we only trained them on
the core instances, but we could also have chosen to train them on all the instances,
or all but the anomalies: this choice depends on the final task.
>>> X_new = np.array([[-0.5, 0], [0, 0.5], [1, -0.1], [2, 1]])
>>> knn.predict(X_new)
array([1, 0, 1, 0])
>>> knn.predict_proba (X_new)
array([[0.18, 0.82],

       [1.  , 0.  ],
       [0.12, 0.88],
       [1.  , 0.  ]])
The decision boundary is represented on Figure 9-15  (the crosses represent the 4
instances in X_new ). Notice that since there is no anomaly in the KNN’s training set,
the classifier always chooses a cluster, even when that cluster is far away. However, it
is fairly straightforward to introduce a maximum distance, in which case the two
instances that are far away from both clusters are classified as anomalies. To do this,

       [1.  , 0.  ],
       [1.  , 0.  ]])
we can use the kneighbors()  method of the KNeighborsClassifier : given a set of
instances, it returns the distances and the indices of the k nearest neighbors in the
training set (two matrices, each with k columns):
>>> y_dist, y_pred_idx  = knn.kneighbors (X_new, n_neighbors =1)
>>> y_pred = dbscan.labels_[dbscan.core_sample_indices_ ][y_pred_idx ]
>>> y_pred[y_dist > 0.2] = -1
>>> y_pred.ravel()
array([-1,  0,  1, -1])258 | Chapter 9: Unsupervised Learning Techniques

Figure 9-15. cluster_classification_diagram
In short, DBSCAN is a very simple yet powerful algorithm, capable of identifying any
number of clusters, of any shape, it is robust to outliers, and it has just two hyper‐
parameters ( eps and min_samples ). However, if the density varies significantly across
the clusters, it can be impossible for it to capture all the clusters properly. Moreover,
its computational complexity is roughly O( m log m), making it pretty close to linear
with regards to the number of instances. However, Scikit-Learn’s implementation can
require up to O( m2) memory if eps is large.
Other Clustering Algorithms
Scikit-Learn implements several more clustering algorithms that you should take a
look at. We cannot cover them all in detail here, but here is a brief overview:•
Agglomerative clustering : a hierarchy of clusters is built from the bottom up.
Think of many tiny bubbles floating on water and gradually attaching to each

other until there’s just one big group of bubbles. Similarly, at each iteration
agglomerative clustering connects the nearest pair of clusters (starting with indi‐
vidual instances). If you draw a tree with a branch for every pair of clusters that
merged, you get a binary tree of clusters, where the leaves are the individual
instances. This approach scales very well to large numbers of instances or clus‐
ters, it can capture clusters of various shapes, it produces a flexible and informa‐

tive cluster tree instead of forcing you to choose a particular cluster scale, and it
can be used with any pairwise distance. It can scale nicely to large numbers of
instances if you provide a connectivity matrix. This is a sparse m by m matrix
that indicates which pairs of instances are neighbors (e.g., returned by
sklearn.neighbors.kneighbors_graph() ). Without a connectivity matrix, the
algorithm does not scale well to large datasets.•
Birch : this algorithm was designed specifically for very large datasets, and it can

be faster than batch K-Means, with similar results, as long as the number of fea‐
tures is not too large (<20). It builds a tree structure during training containingClustering | 259

just enough information to quickly assign each new instance to a cluster, without
having to store all the instances in the tree: this allows it to use limited memory,
while handle huge datasets.•
Mean-shift : this algorithm starts by placing a circle centered on each instance,
then for each circle it computes the mean of all the instances located within it,
and it shifts the circle so that it is centered on the mean. Next, it iterates this
mean-shift step until all the circles stop moving (i.e., until each of them is cen‐
tered on the mean of the instances it contains). This algorithm shifts the circles
in the direction of higher density, until each of them has found a local density
maximum. Finally, all the instances whose circles have settled in the same place
(or close enough) are assigned to the same cluster. This has some of the same fea‐
tures as DBSCAN, in particular it can find any number of clusters of any shape, it
has just one hyperparameter (the radius of the circles, called the bandwidth) and

it relies on local density estimation. However, it tends to chop clusters into pieces
when they have internal density variations. Unfortunately, its computational
complexity is O( m2), so it is not suited for large datasets.•
Affinity  propagation : this algorithm uses a voting system, where instances vote for
similar instances to be their representatives, and once the algorithm converges,
each representative and its voters form a cluster. This algorithm can detect any

number of clusters of different sizes. Unfortunately, this algorithm has a compu‐
tational complexity of O( m2), so it is not suited for large datasets.•
Spectral clustering : this algorithm takes a similarity matrix between the instances
and creates a low-dimensional embedding from it (i.e., it reduces its dimension‐
ality), then it uses another clustering algorithm in this low-dimensional space
(Scikit-Learn’s implementation uses K-Means). Spectral clustering can capture

complex cluster structures, and it can also be used to cut graphs (e.g., to identify
clusters of friends on a social network), however it does not scale well to large
number of instances, and it does not behave well when the clusters have very dif‐
ferent sizes.
Now let’s dive into Gaussian mixture models, which can be used for density estima‐
tion, clustering and anomaly detection.
Gaussian Mixtures
A Gaussian mixture model  (GMM) is a probabilistic model that assumes that the

instances were generated from a mixture of several Gaussian distributions whose
parameters are unknown. All the instances generated from a single Gaussian distri‐
bution form a cluster that typically looks like an ellipsoid. Each cluster can have a dif‐
ferent ellipsoidal shape, size, density and orientation, just like in Figure 9-11 . When
you observe an instance, you know it was generated from one of the Gaussian distri‐260 | Chapter 9: Unsupervised Learning Techniques

• The large rectangles are called plates : they indicate that their content is repeated
several times.•
The number indicated at the bottom right hand side of each plate indicates how
many times its content is repeated, so there are m random variables z(i) (from z(1)
to z(m)) and m random variables x(i), and k means μ(j) and k covariance matrices
Σ(j), but just one weight vector ϕ (containing all the weights ϕ(1) to ϕ(k)).•
Each variable z(i) is drawn from the categorical distribution  with weights ϕ. Each
variable x(i) is drawn from the normal distribution with the mean and covariance
matrix defined by its cluster z(i).•
The solid arrows represent conditional dependencies. For example, the probabil‐
ity distribution for each random variable z(i) depends on the weight vector ϕ.
Note that when an arrow crosses a plate boundary, it means that it applies to all
the repetitions of that plate, so for example the weight vector ϕ conditions the
probability distributions of all the random variables x(1) to x(m).•

to start by estimating the weights ϕ and all the distribution parameters μ(1) to μ(k) and
Σ(1) to Σ(k). Scikit-Learn’s GaussianMixture  class makes this trivial:
from sklearn.mixture  import GaussianMixture
gm = GaussianMixture (n_components =3, n_init=10)
gm.fit(X)
Let’s look at the parameters that the algorithm estimated:
>>> gm.weights_
array([0.20965228, 0.4000662 , 0.39028152])
>>> gm.means_
array([[ 3.39909717,  1.05933727],
       [-1.40763984,  1.42710194],
       [ 0.05135313,  0.07524095]])
>>> gm.covariances_
array([[[ 1.14807234, -0.03270354],
        [-0.03270354,  0.95496237]],
       [[ 0.63478101,  0.72969804],
        [ 0.72969804,  1.1609872 ]],262 | Chapter 9: Unsupervised Learning Techniques

[[ 0.68809572,  0.79608475],
        [ 0.79608475,  1.21234145]]])
Great, it worked fine! Indeed, the weights that were used to generate the data were
0.2, 0.4 and 0.4, and similarly, the means and covariance matrices were very close to
those found by the algorithm. But how? This class relies on the Expectation-
Maximization  (EM) algorithm, which has many similarities with the K-Means algo‐
rithm: it also initializes the cluster parameters randomly, then it repeats two steps
until convergence, first assigning instances to clusters (this is called the expectation
step) then updating the clusters (this is called the maximization step ). Sounds famil‐
iar? Indeed, in the context of clustering you can think of EM as a generalization of K-
Means which not only finds the cluster centers ( μ(1) to μ(k)), but also their size, shape
and orientation ( Σ(1) to Σ(k)), as well as their relative weights ( ϕ(1) to ϕ(k)). Unlike K-
Means, EM uses soft cluster assignments rather than hard assignments: for each

instance during the expectation step, the algorithm estimates the probability that it
belongs to each cluster (based on the current cluster parameters). Then, during the
maximization step, each cluster is updated using all the instances in the dataset, with
each instance weighted by the estimated probability that it belongs to that cluster.
These probabilities are called the responsibilities  of the clusters for the instances. Dur‐
ing the maximization step, each cluster’s update will mostly be impacted by the
instances it is most responsible for.

Unfortunately, just like K-Means, EM can end up converging to
poor solutions, so it needs to be run several times, keeping only the
best solution. This is why we set n_init  to 10. Be careful: by default
n_init  is only set to 1.
Y ou can check whether or not the algorithm converged and how many iterations it
took:
>>> gm.converged_
True
>>> gm.n_iter_
3
Okay, now that you have an estimate of the location, size, shape, orientation and rela‐
tive weight of each cluster, the model can easily assign each instance to the most likely

took:
True
3
cluster (hard clustering) or estimate the probability that it belongs to a particular
cluster (soft clustering). For this, just use the predict()  method for hard clustering,
or the predict_proba()  method for soft clustering:
>>> gm.predict(X)
array([2, 2, 1, ..., 0, 0, 0])
>>> gm.predict_proba (X)
array([[2.32389467e-02, 6.77397850e-07, 9.76760376e-01],
       [1.64685609e-02, 6.75361303e-04, 9.82856078e-01],Gaussian Mixtures | 263

[2.01535333e-06, 9.99923053e-01, 7.49319577e-05],
       ...,
       [9.99999571e-01, 2.13946075e-26, 4.28788333e-07],
       [1.00000000e+00, 1.46454409e-41, 5.12459171e-16],
       [1.00000000e+00, 8.02006365e-41, 2.27626238e-15]])
It is a generative model , meaning you can actually sample new instances from it (note
that they are ordered by cluster index):
>>> X_new, y_new = gm.sample(6)
>>> X_new
array([[ 2.95400315,  2.63680992],
       [-1.16654575,  1.62792705],
       [-1.39477712, -1.48511338],
       [ 0.27221525,  0.690366  ],
       [ 0.54095936,  0.48591934],
       [ 0.38064009, -0.56240465]])
>>> y_new
array([0, 1, 2, 2, 2, 2])
It is also possible to estimate the density of the model at any given location. This is
achieved using the score_samples()  method: for each instance it is given, this
method estimates the log of the probability density function  (PDF) at that location.
The greater the score, the higher the density:
>>> gm.score_samples (X)

[-1.39477712, -1.48511338],
>>> y_new
array([-2.60782346, -3.57106041, -3.33003479, ..., -3.51352783,
       -4.39802535, -3.80743859])
If you compute the exponential of these scores, you get the value of the PDF at the
location of the given instances. These are not probabilities, but probability densities :
they can take on any positive value, not just between 0 and 1. To estimate the proba‐
bility that an instance will fall within a particular region, you would have to integrate

the PDF over that region (if you do so over the entire space of possible instance loca‐
tions, the result will be 1).
Figure 9-17  shows the cluster means, the decision boundaries (dashed lines), and the
density contours of this model:264 | Chapter 9: Unsupervised Learning Techniques

Figure 9-17. Cluster means, decision boundaries and density contours of a trained Gaus‐
sian mixture model
Nice! The algorithm clearly found an excellent solution. Of course, we made its task
easy by actually generating the data using a set of 2D Gaussian distributions (unfortu‐
nately, real life data is not always so Gaussian and low-dimensional), and we also gave
the algorithm the correct number of clusters. When there are many dimensions, or
many clusters, or few instances, EM can struggle to converge to the optimal solution.
Y ou might need to reduce the difficulty of the task by limiting the number of parame‐
ters that the algorithm has to learn: one way to do this is to limit the range of shapes
and orientations that the clusters can have. This can be achieved by imposing con‐
straints on the covariance matrices. To do this, just set the covariance_type  hyper‐
parameter to one of the following values:•
"spherical" : all clusters must be spherical, but they can have different diameters

(i.e., different variances).•
"diag" : clusters can take on any ellipsoidal shape of any size, but the ellipsoid’s
axes must be parallel to the coordinate axes (i.e., the covariance matrices must be
diagonal).•
"tied" : all clusters must have the same ellipsoidal shape, size and orientation
(i.e., all clusters share the same covariance matrix).
By default, covariance_type  is equal to "full" , which means that each cluster can
take on any shape, size and orientation (it has its own unconstrained covariance

matrix). Figure 9-18  plots the solutions found by the EM algorithm when cova
riance_type  is set to "tied"  or "spherical “.Gaussian Mixtures | 265

Figure 9-18. covariance_type_diagram
The computational complexity of training a GaussianMixture
model depends on the number of instances m, the number of
dimensions n, the number of clusters k, and the constraints on the
covariance matrices. If covariance_type  is "spherical  or "diag" ,
it is O( kmn ), assuming the data has a clustering structure. If cova
riance_type  is "tied"  or "full" , it is O( kmn2 + kn3), so it will not
scale to large numbers of features.
Gaussian mixture models can also be used for anomaly detection. Let’s see how.
Anomaly Detection using Gaussian Mixtures
Anomaly detection  (also called outlier detection ) is the task of detecting instances that
deviate strongly from the norm. These instances are of course called anomalies  or
outliers , while the normal instances are called inliers . Anomaly detection is very use‐
ful in a wide variety of applications, for example in fraud detection, or for detecting
defective products in manufacturing, or to remove outliers from a dataset before

training another model, which can significantly improve the performance of the
resulting model.
Using a Gaussian mixture model for anomaly detection is quite simple: any instance
located in a low-density region can be considered an anomaly. Y ou must define what
density threshold you want to use. For example, in a manufacturing company that
tries to detect defective products, the ratio of defective products is usually well-
known. Say it is equal to 4%, then you can set the density threshold to be the value

that results in having 4% of the instances located in areas below that threshold den‐
sity. If you notice that you get too many false positives (i.e., perfectly good products
that are flagged as defective), you can lower the threshold. Conversely, if you have too
many false negatives (i.e., defective products that the system does not flag as defec‐
tive), you can increase the threshold. This is the usual precision/recall tradeoff (see
Chapter 3 ). Here is how you would identify the outliers using the 4th percentile low‐266 | Chapter 9: Unsupervised Learning Techniques

est density as the threshold (i.e., approximately 4% of the instances will be flagged as
anomalies):
densities  = gm.score_samples (X)
density_threshold  = np.percentile (densities , 4)
anomalies  = X[densities  < density_threshold ]
These anomalies are represented as stars on Figure 9-19 :
Figure 9-19. Anomaly detection using a Gaussian mixture model
A closely related task is novelty detection : it differs from anomaly detection in that the
algorithm is assumed to be trained on a “clean” dataset, uncontaminated by outliers,
whereas anomaly detection does not make this assumption. Indeed, outlier detection
is often precisely used to clean up a dataset.
Gaussian mixture models try to fit all the data, including the outli‐
ers, so if you have too many of them, this will bias the model’s view
of “normality”: some outliers may wrongly be considered as nor‐
mal. If this happens, you can try to fit the model once, use it to
detect and remove the most extreme outliers, then fit the model

again on the cleaned up dataset. Another approach is to use robust
covariance estimation methods (see the EllipticEnvelope  class).
Just like K-Means, the GaussianMixture  algorithm requires you to specify the num‐
ber of clusters. So how can you find it?
Selecting the Number of Clusters
With K-Means, you could use the inertia or the silhouette score to select the appro‐
priate number of clusters, but with Gaussian mixtures, it is not possible to use these
metrics because they are not reliable when the clusters are not spherical or have dif‐

ferent sizes. Instead, you can try to find the model that minimizes a theoretical infor‐Gaussian Mixtures | 267

mation criterion  such as the Bayesian information criterion  (BIC) or the Akaike
information criterion  (AIC), defined in Equation 9-1 .
Equation 9-1. Bayesian information criterion (BIC) and Akaike information
criterion (AIC)BIC = log mp− 2 log L
AIC = 2 p− 2 log L•
m is the number of instances, as always.•
p is the number of parameters learned by the model.•L is the maximized value of the likelihood function  of the model.
Both the BIC and the AIC penalize models that have more parameters to learn (e.g.,
more clusters), and reward models that fit the data well. They often end up selecting
the same model, but when they differ, the model selected by the BIC tends to be sim‐
pler (fewer parameters) than the one selected by the AIC, but it does not fit the data
quite as well (this is especially true for larger datasets).
Likelihood function
The terms “probability” and “likelihood” are often used interchangeably in the
English language, but they have very different meanings in statistics: given a statistical

model with some parameters θ, the word “probability” is used to describe how plausi‐
ble a future outcome x is (knowing the parameter values θ), while the word “likeli‐
hood” is used to describe how plausible a particular set of parameter values θ are,
after the outcome x is known.
Consider a one-dimensional mixture model of two Gaussian distributions centered at
-4 and +1. For simplicity, this toy model has a single parameter θ that controls the

standard deviations of both distributions. The top left contour plot in Figure 9-20
shows the entire model f(x; θ) as a function of both x and θ. To estimate the probabil‐
ity distribution of a future outcome x, you need to set the model parameter θ. For
example, if you set it to θ=1.3 (the horizontal line), you get the probability density
function f(x; θ=1.3) shown in the lower left plot. Say you want to estimate the proba‐
bility that x will fall between -2 and +2, you must calculate the integral of the PDF on

this range (i.e., the surface of the shaded region). On the other hand, if you have
observed a single instance x=2.5 (the vertical line in the upper left plot), you get the
likelihood function noted ℒ(θ|x=2.5)=f( x=2.5; θ) represented in the upper right plot.
In short, the PDF is a function of x (with θ fixed) while the likelihood function is a
function of θ (with x fixed). It is important to understand that the likelihood function
is not a probability distribution: if you integrate a probability distribution over all268 | Chapter 9: Unsupervised Learning Techniques

possible values of x, you always get 1, but if you integrate the likelihood function over
all possible values of θ, the result can be any positive value.
Figure 9-20. A model’s parametric function (top left),  and some derived functions: a PDF
(lower left),  a likelihood function (top right) and a log likelihood function (lower right)
Given a dataset X, a common task is to try to estimate the most likely values for the
model parameters. To do this, you must find the values that maximize the likelihood
function, given X. In this example, if you have observed a single instance x=2.5, the
maximum likelihood estimate  (MLE) of θ is θ=1.5. If a prior probability distribution g
over θ exists, it is possible to take it into account by maximizing ℒ(θ|x)g(θ) rather
than just maximizing ℒ(θ|x). This is called maximum a-posteriori (MAP) estimation.
Since MAP constrains the parameter values, you can think of it as a regularized ver‐
sion of MLE.

Notice that it is equivalent to maximize the likelihood function or to maximize its
logarithm (represented in the lower right hand side of Figure 9-20 ): indeed, the loga‐
rithm is a strictly increasing function, so if θ maximizes the log likelihood, it also
maximizes the likelihood. It turns out that it is generally easier to maximize the log
likelihood. For example, if you observed several independent instances x(1) to x(m), you
would need to find the value of θ that maximizes the product of the individual likeli‐

>>> gm.bic(X)
8189.74345832983
>>> gm.aic(X)
8102.518178214792
Figure 9-21  shows the BIC for different numbers of clusters k. As you can see, both
the BIC and the AIC are lowest when k=3, so it is most likely the best choice. Note
that we could also search for the best value for the covariance_type  hyperparameter.
For example, if it is "spherical"  rather than "full" , then the model has much fewer
parameters to learn, but it does not fit the data as well.
Figure 9-21. AIC and BIC for different  numbers of clusters k
Bayesian Gaussian Mixture Models
Rather than manually searching for the optimal number of clusters, it is possible to
use instead the BayesianGaussianMixture  class which is capable of giving weights
equal (or close) to zero to unnecessary clusters. Just set the number of clusters n_com
ponents  to a value that you have good reason to believe is greater than the optimal
number of clusters (this assumes some minimal knowledge about the problem at

hand), and the algorithm will eliminate the unnecessary clusters automatically. For
example, let’s set the number of clusters to 10 and see what happens:
>>> from sklearn.mixture  import BayesianGaussianMixture
>>> bgm = BayesianGaussianMixture (n_components =10, n_init=10, random_state =42)
>>> bgm.fit(X)
>>> np.round(bgm.weights_ , 2)
array([0.4 , 0.21, 0.4 , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  ])
Perfect: the algorithm automatically detected that only 3 clusters are needed, and the

resulting clusters are almost identical to the ones in Figure 9-17 .
In this model, the cluster parameters (including the weights, means and covariance
matrices) are not treated as fixed model parameters anymore, but as latent random
variables, like the cluster assignments (see Figure 9-22 ). So z now includes both the
cluster parameters and the cluster assignments.270 | Chapter 9: Unsupervised Learning Techniques

Figure 9-22. Bayesian Gaussian mixture model
Prior knowledge about the latent variables z can be encoded in a probability distribu‐
tion p(z) called the prior . For example, we may have a prior belief that the clusters are
likely to be few (low concentration), or conversely, that they are more likely to be
plentiful (high concentration). This can be adjusted using the weight_concentra
tion_prior  hyperparameter. Setting it to 0.01 or 1000 gives very different clusterings
(see Figure 9-23 ). However, the more data we have, the less the priors matter. In fact,
to plot diagrams with such large differences, you must use very strong priors and lit‐
tle data.
Figure 9-23. Using different  concentration priors
The fact that you see only 3 regions in the right plot although there
are 4 centroids is not a bug: the weight of the top-right cluster is
much larger than the weight of the lower-right cluster, so the prob‐
ability that any given point in this region belongs to the top-right

tle data.
cluster is greater than the probability that it belongs to the lower-
right cluster, even near the lower-right cluster.Gaussian Mixtures | 271

Bayes’ theorem ( Equation 9-2 ) tells us how to update the probability distribution over
the latent variables after we observe some data X. It computes the posterior  distribu‐
tion p(z|X), which is the conditional probability of z given X.
Equation 9-2. Bayes’ theorempzX= Posterior =Likelihood×Prior
Evidence=pXzpz
pX
Unfortunately, in a Gaussian mixture model (and many other problems), the denomi‐
nator p(x) is intractable, as it requires integrating over all the possible values of z
(Equation 9-3 ). This means considering all possible combinations of cluster parame‐
ters and cluster assignments.
Equation 9-3. The evidence p( X) is often  intractablepX=∫pXzpzdz
This is one of the central problems in Bayesian statistics, and there are several
approaches to solving it. One of them is variational inference , which picks a family of
distributions q(z; λ) with its own variational parameters  λ (lambda), then it optimizes
these parameters to make q(z) a good approximation of p(z|X). This is achieved by

In practice, there are different techniques to maximize the ELBO. In mean field varia‐
tional inference , it is necessary to pick the family of distributions q(z; λ) and the prior
p(z) very carefully to ensure that the equation for the ELBO simplifies to a form that
can actually be computed. Unfortunately, there is no general way to do this, it
depends on the task and requires some mathematical skills. For example, the distribu‐
tions and lower bound equations used in Scikit-Learn’s BayesianGaussianMixture
class are presented in the documentation . From these equations it is possible to derive
update equations for the cluster parameters and assignment variables: these are then
used very much like in the Expectation-Maximization algorithm. In fact, the compu‐
tational complexity of the BayesianGaussianMixture  class is similar to that of the
GaussianMixture  class (but generally significantly slower). A simpler approach to
maximizing the ELBO is called black box stochastic variational inference  (BBSVI): at

each iteration, a few samples are drawn from q and they are used to estimate the gra‐
dients of the ELBO with regards to the variational parameters λ, which are then used
in a gradient ascent step. This approach makes it possible to use Bayesian inference
with any kind of model (provided it is differentiable), even deep neural networks: this
is called Bayesian deep learning.
If you want to dive deeper into Bayesian statistics, check out the
Bayesian Data Analysis  book  by Andrew Gelman, John Carlin, Hal

Stern, David Dunson, Aki Vehtari, and Donald Rubin.
Gaussian mixture models work great on clusters with ellipsoidal shapes, but if you try
to fit a dataset with different shapes, you may have bad surprises. For example, let’s
see what happens if we use a Bayesian Gaussian mixture model to cluster the moons
dataset (see Figure 9-24 ):
Figure 9-24. moons_vs_bgm_diagram
Oops, the algorithm desperately searched for ellipsoids, so it found 8 different clus‐

ters instead of 2. The density estimation is not too bad, so this model could perhaps
be used for anomaly detection, but it failed to identify the two moons. Let’s now look
at a few clustering algorithms capable of dealing with arbitrarily shaped clusters.Gaussian Mixtures | 273

Other Anomaly Detection and Novelty Detection Algorithms
Scikit-Learn also implements a few algorithms dedicated to anomaly detection or
novelty detection:•
Fast-MCD  (minimum covariance determinant), implemented by the EllipticEn
velope  class: this algorithm is useful for outlier detection, in particular to
cleanup a dataset. It assumes that the normal instances (inliers) are generated
from a single Gaussian distribution (not a mixture), but it also assumes that the
dataset is contaminated with outliers that were not generated from this Gaussian
distribution. When it estimates the parameters of the Gaussian distribution (i.e.,
the shape of the elliptic envelope around the inliers), it is careful to ignore the
instances that are most likely outliers. This gives a better estimation of the elliptic
envelope, and thus makes it better at identifying the outliers.•
Isolation forest : this is an efficient algorithm for outlier detection, especially in

high-dimensional datasets. The algorithm builds a Random Forest in which each
Decision Tree is grown randomly: at each node, it picks a feature randomly, then
it picks a random threshold value (between the min and max value) to split the
dataset in two. The dataset gradually gets chopped into pieces this way, until all
instances end up isolated from the other instances. An anomaly is usually far
from other instances, so on average (across all the Decision Trees) it tends to get
isolated in less steps than normal instances.•

Local outlier factor  (LOF): this algorithm is also good for outlier detection. It
compares the density of instances around a given instance to the density around
its neighbors. An anomaly is often more isolated than its k nearest neighbors.•
One-class SVM : this algorithm is better suited for novelty detection. Recall that a
kernelized SVM classifier separates two classes by first (implicitly) mapping all
the instances to a high-dimensional space, then separating the two classes using a

linear SVM classifier within this high-dimensional space (see Chapter 5 ). Since
we just have one class of instances, the one-class SVM algorithm instead tries to
separate the instances in high-dimensional space from the origin. In the original
space, this will correspond to finding a small region that encompasses all the
instances. If a new instance does not fall within this region, it is an anomaly.
There are a few hyperparameters to tweak: the usual ones for a kernelized SVM,

plus a margin hyperparameter that corresponds to the probability of a new
instance being mistakenly considered as novel, when it is in fact normal. It works
great, especially with high-dimensional datasets, but just like all SVMs, it does
not scale to large datasets.274 | Chapter 9: Unsupervised Learning Techniques

1Y ou can get the best of both worlds by being open to biological inspirations without being afraid to create
biologically unrealistic models, as long as they work well.
CHAPTER 10
Introduction to Artificial  Neural Networks
with Keras
release of these titles. The following will be Chapter 10 in the final
Birds inspired us to fly, burdock plants inspired velcro, and countless more inven‐
tions were inspired by nature. It seems only logical, then, to look at the brain’s archi‐
tecture for inspiration on how to build an intelligent machine. This is the key idea
that sparked artificial  neural networks  (ANNs). However, although planes were
inspired by birds, they don’t have to flap their wings. Similarly, ANNs have gradually

become quite different from their biological cousins. Some researchers even argue
that we should drop the biological analogy altogether (e.g., by saying “units” rather
than “neurons”), lest we restrict our creativity to biologically plausible systems.1
ANNs are at the very core of Deep Learning. They are versatile, powerful, and scala‐
ble, making them ideal to tackle large and highly complex Machine Learning tasks,
such as classifying billions of images (e.g., Google Images), powering speech recogni‐

tion services (e.g., Apple’s Siri), recommending the best videos to watch to hundreds
of millions of users every day (e.g., Y ouTube), or learning to beat the world champion
at the game of Go by playing millions of games against itself (DeepMind’s Alpha‐
Zero).277

2“ A Logical Calculus of Ideas Immanent in Nervous Activity, ” W . McCulloch and W . Pitts (1943).In the first part of this chapter, we will introduce artificial neural networks, starting
with a quick tour of the very first ANN architectures, leading up to Multi-Layer Per‐
ceptrons  (MLPs) which are heavily used today (other architectures will be explored in
the next chapters). In the second part, we will look at how to implement neural net‐
works using the popular Keras API. This is a beautifully designed and simple high-
level API for building, training, evaluating and running neural networks. But don’t be
fooled by its simplicity: it is expressive and flexible enough to let you build a wide
variety of neural network architectures. In fact, it will probably be sufficient for most
of your use cases. Moreover, should you ever need extra flexibility, you can always
write custom Keras components using its lower-level API, as we will see in Chap‐
ter 12 .

ter 12 .
But first, let’s go back in time to see how artificial neural networks came to be!
From Biological to Artificial  Neurons
Surprisingly, ANNs have been around for quite a while: they were first introduced
back in 1943 by the neurophysiologist Warren McCulloch and the mathematician
Walter Pitts. In their landmark paper ,2 “ A Logical Calculus of Ideas Immanent in
Nervous Activity, ” McCulloch and Pitts presented a simplified computational model
of how biological neurons might work together in animal brains to perform complex

computations using propositional logic . This was the first artificial neural network
architecture. Since then many other architectures have been invented, as we will see.
The early successes of ANNs until the 1960s led to the widespread belief that we
would soon be conversing with truly intelligent machines. When it became clear that
this promise would go unfulfilled (at least for quite a while), funding flew elsewhere
and ANNs entered a long winter. In the early 1980s there was a revival of interest in

and ANNs entered a long winter. In the early 1980s there was a revival of interest in 
connectionism  (the study of neural networks), as new architectures were invented and
better training techniques were developed. But progress was slow, and by the 1990s
other powerful Machine Learning techniques were invented, such as Support Vector
Machines (see Chapter 5 ). These techniques seemed to offer better results and stron‐
ger theoretical foundations than ANNs, so once again the study of neural networks
entered a long winter.

Finally, we are now witnessing yet another wave of interest in ANNs. Will this wave
die out like the previous ones did? Well, there are a few good reasons to believe that
this wave is different and that it will have a much more profound impact on our lives:278 | Chapter 10: Introduction to Artificial  Neural Networks with Keras

•There is now a huge quantity of data available to train neural networks, and
ANNs frequently outperform other ML techniques on very large and complex
problems.•
The tremendous increase in computing power since the 1990s now makes it pos‐
sible to train large neural networks in a reasonable amount of time. This is in
part due to Moore’s Law, but also thanks to the gaming industry, which has pro‐
duced powerful GPU cards by the millions.•
The training algorithms have been improved. To be fair they are only slightly dif‐
ferent from the ones used in the 1990s, but these relatively small tweaks have a
huge positive impact.•
Some theoretical limitations of ANNs have turned out to be benign in practice.
For example, many people thought that ANN training algorithms were doomed
because they were likely to get stuck in local optima, but it turns out that this is
rather rare in practice (or when it is the case, they are usually fairly close to the
global optimum).•

ANNs seem to have entered a virtuous circle of funding and progress. Amazing
products based on ANNs regularly make the headline news, which pulls more
and more attention and funding toward them, resulting in more and more pro‐
gress, and even more amazing products.
Biological Neurons
Before we discuss artificial neurons, let’s take a quick look at a biological neuron (rep‐
resented in Figure 10-1 ). It is an unusual-looking cell mostly found in animal cerebral
cortexes (e.g., your brain), composed of a cell body  containing the nucleus and most

of the cell’s complex components, and many branching extensions called dendrites ,
plus one very long extension called the axon . The axon’s length may be just a few
times longer than the cell body, or up to tens of thousands of times longer. Near its
extremity the axon splits off into many branches called telodendria , and at the tip of
these branches are minuscule structures called synaptic terminals  (or simply synap‐
ses), which are connected to the dendrites (or directly to the cell body) of other neu‐

rons. Biological neurons receive short electrical impulses called signals  from other
neurons via these synapses. When a neuron receives a sufficient number of signals
from other neurons within a few milliseconds, it fires its own signals.From Biological to Artificial  Neurons | 279

3Image by Bruce Blaus ( Creative Commons 3.0 ). Reproduced from https://en.wikipedia.org/wiki/Neuron .4
In the context of Machine Learning, the phrase “neural networks” generally refers to ANNs, not BNNs.5
Drawing of a cortical lamination by S. Ramon y Cajal (public domain). Reproduced from https://en.wikipe
dia.org/wiki/Cerebral_cortex .
Figure 10-1. Biological neuron3
Thus, individual biological neurons seem to behave in a rather simple way, but they
are organized in a vast network of billions of neurons, each neuron typically connec‐
ted to thousands of other neurons. Highly complex computations can be performed
by a vast network of fairly simple neurons, much like a complex anthill can emerge
from the combined efforts of simple ants. The architecture of biological neural net‐
works (BNN)4 is still the subject of active research, but some parts of the brain have
been mapped, and it seems that neurons are often organized in consecutive layers, as 
shown in Figure 10-2 .

Figure 10-2. Multiple layers in a biological neural network (human cortex)5280 | Chapter 10: Introduction to Artificial  Neural Networks with Keras

Logical Computations with Neurons
Warren McCulloch and Walter Pitts proposed a very simple model of the biological
neuron, which later became known as an artificial  neuron : it has one or more binary
(on/off) inputs and one binary output. The artificial neuron simply activates its out‐
put when more than a certain number of its inputs are active. McCulloch and Pitts
showed that even with such a simplified model it is possible to build a network of
artificial neurons that computes any logical proposition you want. For example, let’s
build a few ANNs that perform various logical computations (see Figure 10-3 ),
assuming that a neuron is activated when at least two of its inputs are active.
Figure 10-3. ANNs performing simple logical computations•
The first network on the left is simply the identity function: if neuron A is activa‐
ted, then neuron C gets activated as well (since it receives two input signals from
neuron A), but if neuron A is off, then neuron C is off as well.•

The second network performs a logical AND: neuron C is activated only when
both neurons A and B are activated (a single input signal is not enough to acti‐
vate neuron C).•
The third network performs a logical OR: neuron C gets activated if either neu‐
ron A or neuron B is activated (or both).•
Finally, if we suppose that an input connection can inhibit the neuron’s activity
(which is the case with biological neurons), then the fourth network computes a
slightly more complex logical proposition: neuron C is activated only if neuron A

is active and if neuron B is off. If neuron A is active all the time, then you get a
logical NOT: neuron C is active when neuron B is off, and vice versa.
Y ou can easily imagine how these networks can be combined to compute complex
logical expressions (see the exercises at the end of the chapter).
The Perceptron
The Perceptron  is one of the simplest ANN architectures, invented in 1957 by Frank
Rosenblatt. It is based on a slightly different artificial neuron (see Figure 10-4 ) called From Biological to Artificial  Neurons | 281

6The name Perceptron  is sometimes used to mean a tiny network with a single TLU. a threshold logic unit  (TLU), or sometimes a linear threshold unit  (LTU): the inputs
and output are now numbers (instead of binary on/off values) and each input con‐
nection is associated with a weight. The TLU computes a weighted sum of its inputs
(z = w1 x1 + w2 x2 + ⋯ + wn xn = xT w), then applies a step function  to that sum and
outputs the result: hw(x) = step( z), where z = xT w.
Figure 10-4. Threshold  logic unit
The most common step function used in Perceptrons is the Heaviside step function
(see Equation 10-1 ). Sometimes the sign function is used instead.
Equation 10-1. Common step functions used in Perceptronsheaviside z=0 if z< 0
1 if z≥ 0sgn z=−1 if z< 0
0 if z= 0
+1 if z> 0
A single TLU can be used for simple linear binary classification. It computes a linear
combination of the inputs and if the result exceeds a threshold, it outputs the positive

0 if z= 0
+1 if z> 0
class or else outputs the negative class (just like a Logistic Regression classifier or a
linear SVM). For example, you could use a single TLU to classify iris flowers based on
the petal length and width (also adding an extra bias feature x0 = 1, just like we did in
previous chapters). Training a TLU in this case means finding the right values for w0,
w1, and w2 (the training algorithm is discussed shortly).
A Perceptron is simply composed of a single layer of TLUs,6 with each TLU connected

to all the inputs. When all the neurons in a layer are connected to every neuron in the
previous layer (i.e., its input neurons), it is called a fully connected layer  or a dense
layer . To represent the fact that each input is sent to every TLU, it is common to draw
special passthrough neurons called input neurons : they just output whatever input
they are fed. All the input neurons form the input layer . Moreover, an extra bias fea‐282 | Chapter 10: Introduction to Artificial  Neural Networks with Keras

ture is generally added ( x0 = 1): it is typically represented using a special type of neu‐
ron called a bias neuron , which just outputs 1 all the time. A Perceptron with two
inputs and three outputs is represented in Figure 10-5 . This Perceptron can classify
instances simultaneously into three different binary classes, which makes it a multi‐
output classifier.
Figure 10-5. Perceptron diagram
Thanks to the magic of linear algebra, it is possible to efficiently compute the outputs
of a layer of artificial neurons for several instances at once, by using Equation 10-2 :
Equation 10-2. Computing the outputs of a fully connected layerhW,bX=ϕXW +b•
As always, X represents the matrix of input features. It has one row per instance,
one column per feature.•
The weight matrix W contains all the connection weights except for the ones
from the bias neuron. It has one row per input neuron and one column per artifi‐
cial neuron in the layer.•
The bias vector b contains all the connection weights between the bias neuron

and the artificial neurons. It has one bias term per artificial neuron.•
The function ϕ is called the activation function : when the artificial neurons are
TLUs, it is a step function (but we will discuss other activation functions shortly).
So how is a Perceptron trained? The Perceptron training algorithm proposed by
Frank Rosenblatt was largely inspired by Hebb’s rule . In his book The Organization of
Behavior , published in 1949, Donald Hebb suggested that when a biological neuron
often triggers another neuron, the connection between these two neurons grows

stronger. This idea was later summarized by Siegrid Löwel in this catchy phrase:
“Cells that fire together, wire together. ” This rule later became known as Hebb’s rule From Biological to Artificial  Neurons | 283

7Note that this solution is generally not unique: in general when the data are linearly separable, there is an
infinity of hyperplanes that can separate them.(or Hebbian learning ); that is, the connection weight between two neurons is
increased whenever they have the same output. Perceptrons are trained using a var‐
iant of this rule that takes into account the error made by the network; it reinforces
connections that help reduce the error. More specifically, the Perceptron is fed one
training instance at a time, and for each instance it makes its predictions. For every
output neuron that produced a wrong prediction, it reinforces the connection
weights from the inputs that would have contributed to the correct prediction. The
rule is shown in Equation 10-3 .
Equation 10-3. Perceptron learning rule (weight update)wi,jnext step=wi,j+ηyj−yjxi •
wi, j is the connection weight between the ith input neuron and the jth output neu‐
ron.•

ron.•
xi is the ith input value of the current training instance.•yj is the output of the jth output neuron for the current training instance.•
yj is the target output of the jth output neuron for the current training instance.•
η is the learning rate.
The decision boundary of each output neuron is linear, so Perceptrons are incapable
of learning complex patterns (just like Logistic Regression classifiers). However, if the
training instances are linearly separable, Rosenblatt demonstrated that this algorithm

would converge to a solution.7 This is called the Perceptron convergence theorem .
Scikit-Learn provides a Perceptron  class that implements a single TLU network. It
can be used pretty much as you would expect—for example, on the iris dataset (intro‐
duced in Chapter 4 ):
from sklearn.linear_model  import Perceptron
iris = load_iris ()
X = iris.data[:, (2, 3)]  # petal length, petal width
y = (iris.target == 0).astype(np.int)  # Iris Setosa?

per_clf = Perceptron ()
per_clf.fit(X, y)284 | Chapter 10: Introduction to Artificial  Neural Networks with Keras

y_pred = per_clf.predict([[2, 0.5]])
Y ou may have noticed the fact that the Perceptron learning algorithm strongly resem‐
bles Stochastic Gradient Descent. In fact, Scikit-Learn’s Perceptron  class is equivalent
to using an SGDClassifier  with the following hyperparameters: loss="perceptron" ,
learning_rate="constant" , eta0=1  (the learning rate), and penalty=None  (no regu‐
larization).
Note that contrary to Logistic Regression classifiers, Perceptrons do not output a class
probability; rather, they just make predictions based on a hard threshold. This is one
of the good reasons to prefer Logistic Regression over Perceptrons.
In their 1969 monograph titled Perceptrons , Marvin Minsky and Seymour Papert
highlighted a number of serious weaknesses of Perceptrons, in particular the fact that
they are incapable of solving some trivial problems (e.g., the Exclusive OR  (XOR)
classification problem; see the left side of Figure 10-6 ). Of course this is true of any

other linear classification model as well (such as Logistic Regression classifiers), but
researchers had expected much more from Perceptrons, and their disappointment
was great, and many researchers dropped neural networks altogether in favor of
higher-level problems such as logic, problem solving, and search.
However, it turns out that some of the limitations of Perceptrons can be eliminated by
stacking multiple Perceptrons. The resulting ANN is called a Multi-Layer Perceptron

(MLP). In particular, an MLP can solve the XOR problem, as you can verify by com‐
puting the output of the MLP represented on the right of Figure 10-6 : with inputs (0,
0) or (1, 1) the network outputs 0, and with inputs (0, 1) or (1, 0) it outputs 1. All
connections have a weight equal to 1, except the four connections where the weight is
shown. Try verifying that this network indeed solves the XOR problem!
Figure 10-6. XOR classification  problem and an MLP that solves itFrom Biological to Artificial  Neurons | 285

8In the 1990s, an ANN with more than two hidden layers was considered deep. Nowadays, it is common to see
ANNs with dozens of layers, or even hundreds, so the definition of “deep” is quite fuzzy.9
“Learning Internal Representations by Error Propagation, ” D. Rumelhart, G. Hinton, R. Williams (1986).
Multi-Layer Perceptron and Backpropagation
An MLP is composed of one (passthrough) input layer , one or more layers of TLUs,
called hidden layers , and one final layer of TLUs called the output layer  (see
Figure 10-7 ). The layers close to the input layer are usually called the lower layers,
and the ones close to the outputs are usually called the upper layers. Every layer
except the output layer includes a bias neuron and is fully connected to the next layer.
Figure 10-7. Multi-Layer Perceptron
The signal flows only in one direction (from the inputs to the out‐
puts), so this architecture is an example of a feedforward neural net‐
work  (FNN).

When an ANN contains a deep stack of hidden layers8, it is called a deep neural net‐
work  (DNN). The field of Deep Learning studies DNNs, and more generally models
containing deep stacks of computations. However, many people talk about Deep
Learning whenever neural networks are involved (even shallow ones).
For many years researchers struggled to find a way to train MLPs, without success.
But in 1986, David Rumelhart, Geoffrey Hinton and Ronald Williams published a
groundbreaking paper9 introducing the backpropagation  training algorithm, which is

still used today. In short, it is simply Gradient Descent (introduced in Chapter 4 )286 | Chapter 10: Introduction to Artificial  Neural Networks with Keras

10This technique was actually independently invented several times by various researchers in different fields,
starting with P . Werbos in 1974.
using an efficient technique for computing the gradients automatically10: in just two
passes through the network (one forward, one backward), the backpropagation algo‐
rithm is able to compute the gradient of the network’s error with regards to every sin‐
gle model parameter. In other words, it can find out how each connection weight and
each bias term should be tweaked in order to reduce the error. Once it has these gra‐
dients, it just performs a regular Gradient Descent step, and the whole process is
repeated until the network converges to the solution.
Automatically computing gradients is called automatic differentia‐
tion, or autodiff . There are various autodiff techniques, with differ‐
ent pros and cons. The one used by backpropagation is called
reverse-mode autodiff . It is fast and precise, and is well suited when

the function to differentiate has many variables (e.g., connection
weights) and few outputs (e.g., one loss). If you want to learn more
about autodiff, check out ???.
Let’s run through this algorithm in a bit more detail:•
It handles one mini-batch at a time (for example containing 32 instances each),
and it goes through the full training set multiple times. Each pass is called an
epoch , as we saw in Chapter 4 .•
Each mini-batch is passed to the network’s input layer, which just sends it to the

first hidden layer. The algorithm then computes the output of all the neurons in
this layer (for every instance in the mini-batch). The result is passed on to the
next layer, its output is computed and passed to the next layer, and so on until we
get the output of the last layer, the output layer. This is the forward pass : it is
exactly like making predictions, except all intermediate results are preserved
since they are needed for the backward pass.•

Next, the algorithm measures the network’s output error (i.e., it uses a loss func‐
tion that compares the desired output and the actual output of the network, and
returns some measure of the error).•
Then it computes how much each output connection contributed to the error.
This is done analytically by simply applying  the chain rule  (perhaps the most fun‐
damental rule in calculus), which makes this step fast and precise.•
The algorithm then measures how much of these error contributions came from

each connection in the layer below, again using the chain rule—and so on until
the algorithm reaches the input layer. As we explained earlier, this reverse pass
efficiently measures the error gradient across all the connection weights in theFrom Biological to Artificial  Neurons | 287

network by propagating the error gradient backward through the network (hence
the name of the algorithm).•
Finally, the algorithm performs a Gradient Descent step to tweak all the connec‐
tion weights in the network, using the error gradients it just computed.
This algorithm is so important, it’s worth summarizing it again: for each training
instance the backpropagation algorithm first makes a prediction (forward pass),
measures the error, then goes through each layer in reverse to measure the error con‐
tribution from each connection (reverse pass), and finally slightly tweaks the connec‐
tion weights to reduce the error (Gradient Descent step).
It is important to initialize all the hidden layers’ connection weights
randomly, or else training will fail. For example, if you initialize all
weights and biases to zero, then all neurons in a given layer will be
perfectly identical, and thus backpropagation will affect them in
exactly the same way, so they will remain identical. In other words,

despite having hundreds of neurons per layer, your model will act
as if it had only one neuron per layer: it won’t be too smart. If
instead you randomly initialize the weights, you break the symme‐
try and allow backpropagation to train a diverse team of neurons.
In order for this algorithm to work properly, the authors made a key change to the
MLP’s architecture: they replaced the step function with the logistic function, σ(z) =
1 / (1 + exp(– z)). This was essential because the step function contains only flat seg‐

ments, so there is no gradient to work with (Gradient Descent cannot move on a flat
surface), while the logistic function has a well-defined nonzero derivative every‐
where, allowing Gradient Descent to make some progress at every step. In fact, the
backpropagation algorithm works well with many other activation functions , not just
the logistic function. Two other popular activation functions are:
The hyperbolic tangent function tanh(z) = 2σ(2z) – 1
Just like the logistic function it is S-shaped, continuous, and differentiable, but its

output value ranges from –1 to 1 (instead of 0 to 1 in the case of the logistic func‐
tion), which tends to make each layer’s output more or less centered around 0 at
the beginning of training. This often helps speed up convergence.
The Rectified  Linear Unit function: ReLU(z) = max(0, z)
It is continuous but unfortunately not differentiable at z = 0 (the slope changes
abruptly, which can make Gradient Descent bounce around), and its derivative is

0 for z < 0. However, in practice it works very well and has the advantage of being288 | Chapter 10: Introduction to Artificial  Neural Networks with Keras

11Biological neurons seem to implement a roughly sigmoid (S-shaped) activation function, so researchers stuck
to sigmoid functions for a very long time. But it turns out that ReLU generally works better in ANNs. This is
one of the cases where the biological analogy was misleading.fast to compute11. Most importantly, the fact that it does not have a maximum
output value also helps reduce some issues during Gradient Descent (we will
come back to this in Chapter 11 ).
These popular activation functions and their derivatives are represented in
Figure 10-8 . But wait! Why do we need activation functions in the first place? Well, if
you chain several linear transformations, all you get is a linear transformation. For
example, say f( x) = 2 x + 3 and g( x) = 5 x - 1, then chaining these two linear functions
gives you another linear function: f(g( x)) = 2(5 x - 1) + 3 = 10 x + 1. So if you don’t
have some non-linearity between layers, then even a deep stack of layers is equivalent

to a single layer: you cannot solve very complex problems with that.
Figure 10-8. Activation functions and their derivatives
Okay! So now you know where neural nets came from, what their architecture is and
how to compute their outputs, and you also learned about the backpropagation algo‐
rithm. But what exactly can you do with them?
Regression MLPs
First, MLPs can be used for regression tasks. If you want to predict a single value (e.g.,
the price of a house given many of its features), then you just need a single output

neuron: its output is the predicted value. For multivariate regression (i.e., to predict
multiple values at once), you need one output neuron per output dimension. For
example, to locate the center of an object on an image, you need to predict 2D coordi‐
nates, so you need two output neurons. If you also want to place a bounding box
around the object, then you need two more numbers: the width and the height of the
object. So you end up with 4 output neurons.From Biological to Artificial  Neurons | 289

In general, when building an MLP for regression, you do not want to use any activa‐
tion function for the output neurons, so they are free to output any range of values.
However, if you want to guarantee that the output will always be positive, then you
can use the ReLU activation function, or the softplus  activation function in the output
layer. Finally, if you want to guarantee that the predictions will fall within a given
range of values, then you can use the logistic function or the hyperbolic tangent, and
scale the labels to the appropriate range: 0 to 1 for the logistic function, or –1 to 1 for
the hyperbolic tangent.
The loss function to use during training is typically the mean squared error, but if you
have a lot of outliers in the training set, you may prefer to use the mean absolute
error instead. Alternatively, you can use the Huber loss, which is a combination of
both.
The Huber loss is quadratic when the error is smaller than a thres‐

both.
hold δ (typically 1), but linear when the error is larger than δ. This
makes it less sensitive to outliers than the mean squared error, and
it is often more precise and converges faster than the mean abso‐
lute error.

both.
Table 10-1  summarizes the typical architecture of a regression MLP .Table 10-1. Typical Regression MLP ArchitectureHyperparameterTypical Value# input neuronsOne per input feature (e.g., 28 x 28 = 784 for MNIST)# hidden layersDepends on the problem. Typically 1 to 5.# neurons per hidden layerDepends on the problem. Typically 10 to 100.# output neurons1 per prediction dimensionHidden activationReLU (or SELU, see Chapter 11 ) Output activationNone or ReLU/Softplus (if positive outputs) or Logistic/Tanh (if bounded outputs)Loss functionMSE or MAE/Huber (if outliers)
Classification  MLPs

MLPs can also be used for classification tasks. For a binary classification problem,
you just need a single output neuron using the logistic activation function: the output
will be a number between 0 and 1, which you can interpret as the estimated probabil‐
ity of the positive class. Obviously, the estimated probability of the negative class is
equal to one minus that number.
MLPs can also easily handle multilabel binary classification tasks (see Chapter 3 ). For
example, you could have an email classification system that predicts whether each
incoming email is ham or spam, and simultaneously predicts whether it is an urgent290 | Chapter 10: Introduction to Artificial  Neural Networks with Keras

or non-urgent email. In this case, you would need two output neurons, both using
the logistic activation function: the first would output the probability that the email is
spam and the second would output the probability that it is urgent. More generally,
you would dedicate one output neuron for each positive class. Note that the output
probabilities do not necessarily add up to one. This lets the model output any combi‐
nation of labels: you can have non-urgent ham, urgent ham, non-urgent spam, and
perhaps even urgent spam (although that would probably be an error).
If each instance can belong only to a single class, out of 3 or more possible classes
(e.g., classes 0 through 9 for digit image classification), then you need to have one
output neuron per class, and you should use the softmax  activation function for the
whole output layer (see Figure 10-9 ). The softmax function (introduced in Chapter 4 )
will ensure that all the estimated probabilities are between 0 and 1 and that they add

up to one (which is required if the classes are exclusive). This is called multiclass clas‐
sification.
Figure 10-9. A modern MLP (including ReLU and softmax)  for classification
Regarding the loss function, since we are predicting probability distributions, the
cross-entropy (also called the log loss, see Chapter 4 ) is generally a good choice.

Table 10-2  summarizes the typical architecture of a classification MLP .Table 10-2. Typical Classification  MLP Architecture HyperparameterBinary classification Multilabel binary classification Multiclass classification Input and hidden layersSame as regressionSame as regressionSame as regression# output neurons11 per label1 per classOutput layer activationLogisticLogisticSoftmax From Biological to Artificial  Neurons | 291

12Project ONEIROS (Open-ended Neuro-Electronic Intelligent Robot Operating System).
HyperparameterBinary classification Multilabel binary classification Multiclass classification Loss functionCross-EntropyCross-EntropyCross-Entropy
Before we go on, I recommend you go through exercise 1, at the
end of this chapter. Y ou will play with various neural network
architectures and visualize their outputs using the TensorFlow Play‐
ground . This will be very useful to better understand MLPs, for
example the effects of all the hyperparameters (number of layers
and neurons, activation functions, and more).
Now you have all the concepts you need to start implementing MLPs with Keras!
Implementing MLPs with Keras
Keras is a high-level Deep Learning API that allows you to easily build, train, evaluate
and execute all sorts of neural networks. Its documentation (or specification) is avail‐
able at https://keras.io . The reference implementation is simply called Keras as well, so

to avoid any confusion we will call it keras-team (since it is available at https://
github.com/keras-team/keras ). It was developed by François Chollet as part of a
research project12 and released as an open source project in March 2015. It quickly
gained popularity owing to its ease-of-use, flexibility and beautiful design. To per‐
form the heavy computations required by neural networks, keras-team relies on a
computation backend. At the present, you can choose from three popular open

source deep learning libraries: TensorFlow, Microsoft Cognitive Toolkit (CNTK) or
Theano.
Moreover, since late 2016, other implementations have been released. Y ou can now
run Keras on Apache MXNet, Apple’s Core ML, Javascript or Typescript (to run Keras
code in a web browser), or PlaidML (which can run on all sorts of GPU devices, not
just Nvidia). Moreover, TensorFlow itself now comes bundled with its own Keras
implementation called tf.keras. It only supports TensorFlow as the backend, but it has

Theano.
the advantage of offering some very useful extra features (see Figure 10-10 ): for
example, it supports TensorFlow’s Data API which makes it quite easy to load and
preprocess data efficiently. For this reason, we will use tf.keras in this book. However,
in this chapter we will not use any of the TensorFlow-specific features, so the code
should run fine on other Keras implementations as well (at least in Python), with only

minor modifications, such as changing the imports.292 | Chapter 10: Introduction to Artificial  Neural Networks with Keras

Figure 10-10. Two Keras implementations: keras-team (left)  and tf.keras (right)
As tf.keras is bundled with TensorFlow, let’s install TensorFlow!
Installing TensorFlow 2
Assuming you installed Jupyter and Scikit-Learn by following the installation instruc‐
tions in Chapter 2 , you can simply use pip to install TensorFlow. If you created an
isolated environment using virtualenv, you first need to activate it:
$ cd $ML_PATH              # Your ML working directory (e.g., $HOME/ml)
$ source env/bin/activate  # on Linux or MacOSX
$ .\env\Scripts\activate   # on Windows
Next, install TensorFlow 2 (if you are not using a virtualenv, you will need adminis‐
trator rights, or to add the --user  option):
$ python3 -m pip install --upgrade tensorflow
For GPU support, you need to install tensorflow-gpu  instead of
tensorflow , and there are other libraries to install. See https://
tensorflow.org/install/gpu  for more details.
To test your installation, open a Python shell or a Jupyter notebook, then import Ten‐

sorFlow and tf.keras, and print their versions:
>>> import tensorflow  as tf
>>> from tensorflow  import keras
>>> tf.__version__
'2.0.0'
>>> keras.__version__
'2.2.4-tf'Implementing MLPs with Keras | 293

The second version is the version of the Keras API implemented by tf.keras. Note that
it ends with -tf, highlighting the fact that tf.keras implements the Keras API, plus
some extra TensorFlow-specific features.
Now let’s use tf.keras! Let’s start by building a simple image classifier.
Building an Image Classifier  Using the Sequential API
First, we need to load a dataset. We will tackle Fashion MNIST , which is a drop-in
replacement of MNIST (introduced in Chapter 3 ). It has the exact same format as
MNIST (70,000 grayscale images of 28×28 pixels each, with 10 classes), but the
images represent fashion items rather than handwritten digits, so each class is more
diverse and the problem turns out to be significantly more challenging than MNIST.
For example, a simple linear model reaches about 92% accuracy on MNIST, but only
about 83% on Fashion MNIST.
Using Keras to Load the Dataset
Keras provides some utility functions to fetch and load common datasets, including

MNIST, Fashion MNIST, the original California housing dataset, and more. Let’s load
Fashion MNIST:
fashion_mnist  = keras.datasets .fashion_mnist
(X_train_full , y_train_full ), (X_test, y_test) = fashion_mnist .load_data ()
When loading MNIST or Fashion MNIST using Keras rather than Scikit-Learn, one
important difference is that every image is represented as a 28×28 array rather than a
1D array of size 784. Moreover, the pixel intensities are represented as integers (from

0 to 255) rather than floats (from 0.0 to 255.0). Here is the shape and data type of the
training set:
>>> X_train_full .shape
(60000, 28, 28)
>>> X_train_full .dtype
dtype('uint8')
Note that the dataset is already split into a training set and a test set, but there is no
validation set, so let’s create one. Moreover, since we are going to train the neural net‐
work using Gradient Descent, we must scale the input features. For simplicity, we just
scale the pixel intensities down to the 0-1 range by dividing them by 255.0 (this also

converts them to floats):
X_valid, X_train = X_train_full [:5000] / 255.0, X_train_full [5000:] / 255.0
y_valid, y_train = y_train_full [:5000], y_train_full [5000:]
With MNIST, when the label is equal to 5, it means that the image represents the
handwritten digit 5. Easy. However, for Fashion MNIST, we need the list of class
names to know what we are dealing with:294 | Chapter 10: Introduction to Artificial  Neural Networks with Keras

class_names  = ["T-shirt/top" , "Trouser" , "Pullover" , "Dress", "Coat",
               "Sandal" , "Shirt", "Sneaker" , "Bag", "Ankle boot" ]
For example, the first image in the training set represents a coat:
>>> class_names [y_train[0]]
'Coat'
Figure 10-11  shows a few samples from the Fashion MNIST dataset:
Figure 10-11. Samples from Fashion MNIST
Creating the Model Using the Sequential API
Now let’s build the neural network! Here is a classification MLP with two hidden lay‐
ers:
model = keras.models.Sequential ()
model.add(keras.layers.Flatten(input_shape =[28, 28]))
model.add(keras.layers.Dense(300, activation ="relu"))
model.add(keras.layers.Dense(100, activation ="relu"))
model.add(keras.layers.Dense(10, activation ="softmax" ))
Let’s go through this code line by line:•
The first line creates a Sequential  model. This is the simplest kind of Keras
model, for neural networks that are just composed of a single stack of layers, con‐
nected sequentially. This is called the sequential API.•

Next, we build the first layer and add it to the model. It is a Flatten  layer whose
role is simply to convert each input image into a 1D array: if it receives input data
X, it computes X.reshape(-1, 1) . This layer does not have any parameters, it is
just there to do some simple preprocessing. Since it is the first layer in the model,
you should specify the input_shape : this does not include the batch size, only the
shape of the instances. Alternatively, you could add a keras.layers.InputLayer

as the first layer, setting shape=[28,28] .•
Next we add a Dense  hidden layer with 300 neurons. It will use the ReLU activa‐
tion function. Each Dense  layer manages its own weight matrix, containing all the
connection weights between the neurons and their inputs. It also manages a vec‐Implementing MLPs with Keras | 295

tor of bias terms (one per neuron). When it receives some input data, it computes
Equation 10-2 .•
Next we add a second Dense  hidden layer with 100 neurons, also using the ReLU
activation function.•
Finally, we add a Dense  output layer with 10 neurons (one per class), using the
softmax activation function (because the classes are exclusive).
Specifying activation="relu"  is equivalent to activa
tion=keras.activations.relu . Other activation functions are
available in the keras.activations  package, we will use many of
them in this book. See https://keras.io/activations/  for the full list.
Instead of adding the layers one by one as we just did, you can pass a list of layers
when creating the Sequential  model:
model = keras.models.Sequential ([
    keras.layers.Flatten(input_shape =[28, 28]),
    keras.layers.Dense(300, activation ="relu"),
    keras.layers.Dense(100, activation ="relu"),
    keras.layers.Dense(10, activation ="softmax" )
])
Using Code Examples From keras.io

])
Code examples documented on keras.io will work fine with tf.keras, but you need to
change the imports. For example, consider this keras.io code:
from keras.layers  import Dense
output_layer  = Dense(10)
Y ou must change the imports like this:
from tensorflow.keras.layers  import Dense
Or simply use full paths, if you prefer:
from tensorflow  import keras
output_layer  = keras.layers.Dense(10)
This is more verbose, but I use this approach in this book so you can easily see which

packages to use, and to avoid confusion between standard classes and custom classes.
In production code, I use the previous approach, as do most people.296 | Chapter 10: Introduction to Artificial  Neural Networks with Keras

13Y ou can also generate an image of your model using keras.utils.plot_model() . The model’s summary()  method displays all the model’s layers13, including each layer’s
name (which is automatically generated unless you set it when creating the layer), its
output shape ( None  means the batch size can be anything), and its number of parame‐
ters. The summary ends with the total number of parameters, including trainable and
non-trainable parameters. Here we only have trainable parameters (we will see exam‐
ples of non-trainable parameters in Chapter 11 ):
>>> model.summary()
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
flatten_1 (Flatten)          (None, 784)               0
dense_3 (Dense)              (None, 300)               235500

dense_4 (Dense)              (None, 100)               30100
dense_5 (Dense)              (None, 10)                1010
Total params: 266,610
Trainable params: 266,610
Non-trainable params: 0
Note that Dense  layers often have a lot of parameters. For example, the first hidden
layer has 784 × 300 connection weights, plus 300 bias terms, which adds up to

235,500 parameters! This gives the model quite a lot of flexibility to fit the training
data, but it also means that the model runs the risk of overfitting, especially when you
do not have a lot of training data. We will come back to this later.
Y ou can easily get a model’s list of layers, to fetch a layer by its index, or you can fetch
it by name:
>>> model.layers
[<tensorflow.python.keras.layers.core.Flatten at 0x132414e48>,
 <tensorflow.python.keras.layers.core.Dense at 0x1324149b0>,

it by name:
 <tensorflow.python.keras.layers.core.Dense at 0x1356ba8d0>,
 <tensorflow.python.keras.layers.core.Dense at 0x13240d240>]
>>> model.layers[1].name
'dense_3'
>>> model.get_layer ('dense_3' ).name
'dense_3'
All the parameters of a layer can be accessed using its get_weights()  and
set_weights()  method. For a Dense  layer, this includes both the connection weights
and the bias terms:Implementing MLPs with Keras | 297

>>> weights, biases = hidden1.get_weights ()
>>> weights
array([[ 0.03854964, -0.04054524,  0.00599282, ...,  0.02566582,
         0.01032123,  0.06914985],
       ...,
       [ 0.02632413, -0.05105981, -0.00332005, ...,  0.04175945,
         0.0443138 , -0.05558084]], dtype=float32)
>>> weights.shape
(784, 300)
>>> biases
array([0., 0., 0., 0., 0., 0., 0., 0., 0., ...,  0., 0., 0.], dtype=float32)
>>> biases.shape
(300,)
Notice that the Dense  layer initialized the connection weights randomly (which is
needed to break symmetry, as we discussed earlier), and the biases were just initial‐
ized to zeros, which is fine. If you ever want to use a different initialization method,
you can set kernel_initializer  (kernel  is another name for the matrix of connec‐
tion weights) or bias_initializer  when creating the layer. We will discuss initializ‐
ers further in Chapter 11 , but if you want the full list, see https://keras.io/initializers/ .
The shape of the weight matrix depends on the number of inputs.

This is why it is recommended to specify the input_shape  when
creating the first layer in a Sequential  model. However, if you do
not specify the input shape, it’s okay: Keras will simply wait until it
knows the input shape before it actually builds the model. This will
happen either when you feed it actual data (e.g., during training),
or when you call its build()  method. Until the model is really
built, the layers will not have any weights, and you will not be able

to do certain things (such as print the model summary or save the
model), so if you know the input shape when creating the model, it
is best to specify it.
Compiling the Model
After a model is created, you must call its compile()  method to specify the loss func‐
tion and the optimizer to use. Optionally, you can also specify a list of extra metrics to
compute during training and evaluation:
model.compile(loss="sparse_categorical_crossentropy" ,
              optimizer ="sgd",

              metrics=["accuracy" ])298 | Chapter 10: Introduction to Artificial  Neural Networks with Keras

Using loss="sparse_categorical_crossentropy"  is equivalent to
loss=keras.losses.sparse_categorical_crossentropy . Simi‐
larly, optimizer="sgd"  is equivalent to optimizer=keras.optimiz
ers.SGD()  and metrics=["accuracy"]  is equivalent to
metrics=[keras.metrics.sparse_categorical_accuracy]  (when
using this loss). We will use many other losses, optimizers and met‐
rics in this book, but for the full lists see https://keras.io/losses/ ,
https://keras.io/optimizers/  and https://keras.io/metrics/ .
This requires some explanation. First, we use the "sparse_categorical_crossen
tropy"  loss because we have sparse labels (i.e., for each instance there is just a target
class index, from 0 to 9 in this case), and the classes are exclusive. If instead we had
one target probability per class for each instance (such as one-hot vectors, e.g. [0.,
0., 0., 1., 0., 0., 0., 0., 0., 0.]  to represent class 3), then we would need
to use the "categorical_crossentropy"  loss instead. If we were doing binary classi‐

fication (with one or more binary labels), then we would use the "sigmoid"  (i.e.,
logistic) activation function in the output layer instead of the "softmax"  activation
function, and we would use the "binary_crossentropy"  loss.
If you want to convert sparse labels (i.e., class indices) to one-hot
vector labels, you can use the keras.utils.to_categorical()
function. To go the other way round, you can just use the np.arg
max()  function with axis=1 .

Secondly, regarding the optimizer, "sgd"  simply means that we will train the model
using simple Stochastic Gradient Descent. In other words, Keras will perform the
backpropagation algorithm described earlier (i.e., reverse-mode autodiff + Gradient
Descent). We will discuss more efficient optimizers in Chapter 11  (they improve the
Gradient Descent part, not the autodiff).
Finally, since this is a classifier, it’s useful to measure its "accuracy"  during training
and evaluation.
Training and Evaluating the Model

Now the model is ready to be trained. For this we simply need to call its fit()
method. We pass it the input features ( X_train ) and the target classes ( y_train ), as
well as the number of epochs to train (or else it would default to just 1, which would
definitely not be enough to converge to a good solution). We also pass a validation set
(this is optional): Keras will measure the loss and the extra metrics on this set at the
end of each epoch, which is very useful to see how well the model really performs: if

the performance on the training set is much better than on the validation set, yourImplementing MLPs with Keras | 299

model is probably overfitting the training set (or there is a bug, such as a data mis‐
match between the training set and the validation set):
>>> history = model.fit(X_train, y_train, epochs=30,
...                     validation_data =(X_valid, y_valid))
...
Train on 55000 samples, validate on 5000 samples
Epoch 1/30
55000/55000 [==========] - 3s 55us/sample - loss: 1.4948     - acc: 0.5757
                                          - val_loss: 1.0042 - val_acc: 0.7166
Epoch 2/30
55000/55000 [==========] - 3s 55us/sample - loss: 0.8690     - acc: 0.7318
                                          - val_loss: 0.7549 - val_acc: 0.7616
[...]
Epoch 50/50
55000/55000 [==========] - 4s 72us/sample - loss: 0.3607     - acc: 0.8752
                                          - val_loss: 0.3706 - val_acc: 0.8728
And that’s it! The neural network is trained. At each epoch during training, Keras dis‐
plays the number of instances processed so far (along with a progress bar), the mean

Epoch 2/30
[...]
training time per sample, the loss and accuracy (or any other extra metrics you asked
for), both on the training set and the validation set. Y ou can see that the training loss
went down, which is a good sign, and the validation accuracy reached 87.28% after 50
epochs, not too far from the training accuracy, so there does not seem to be much
overfitting going on.
Instead of passing a validation set using the validation_data
argument, you could instead set validation_split  to the ratio of

the training set that you want Keras to use for validation (e.g., 0.1).
If the training set was very skewed, with some classes being overrepresented and oth‐
ers underrepresented, it would be useful to set the class_weight  argument when
calling the fit()  method, giving a larger weight to underrepresented classes, and a
lower weight to overrepresented classes. These weights would be used by Keras when
computing the loss. If you need per-instance weights instead, you can set the sam

ple_weight  argument (it supersedes class_weight ). This could be useful for exam‐
ple if some instances were labeled by experts while others were labeled using a
crowdsourcing platform: you might want to give more weight to the former. Y ou can
also provide sample weights (but not class weights) for the validation set by adding
them as a third item in the validation_data  tuple.
The fit()  method returns a History  object containing the training parameters ( his

tory.params ), the list of epochs it went through ( history.epoch ), and most impor‐
tantly a dictionary ( history.history ) containing the loss and extra metrics it
measured at the end of each epoch on the training set and on the validation set (if300 | Chapter 10: Introduction to Artificial  Neural Networks with Keras

any). If you create a Pandas DataFrame using this dictionary and call its plot()
method, you get the learning curves shown in Figure 10-12 :
pd.DataFrame (history.history).plot(figsize=(8, 5))
plt.grid(True)
plt.gca().set_ylim (0, 1) # set the vertical range to [0-1]
Figure 10-12. Learning Curves
Y ou can see that both the training and validation accuracy steadily increase during
training, while the training and validation loss decrease. Good! Moreover, the valida‐
tion curves are quite close to the training curves, which means that there is not too
much overfitting. In this particular case, the model performed better on the valida‐
tion set than on the training set at the beginning of training: this sometimes happens
by chance (especially when the validation set is fairly small). However, the training set
performance ends up beating the validation performance, as is generally the case
when you train for long enough. Y ou can tell that the model has not quite converged

yet, as the validation loss is still going down, so you should probably continue train‐
ing. It’s as simple as calling the fit()  method again, since Keras just continues train‐
ing where it left off (you should be able to reach close to 89% validation accuracy).
If you are not satisfied with the performance of your model, you should go back and
tune the model’s hyperparameters, for example the number of layers, the number of
neurons per layer, the types of activation functions we use for each hidden layer, theImplementing MLPs with Keras | 301

number of training epochs, the batch size (it can be set in the fit()  method using the
batch_size  argument, which defaults to 32). We will get back to hyperparameter
tuning at the end of this chapter. Once you are satisfied with your model’s validation
accuracy, you should evaluate it on the test set to estimate the generalization error
before you deploy the model to production. Y ou can easily do this using the evalu
ate()  method (it also supports several other arguments, such as batch_size  or sam
ple_weight , please check the documentation for more details):
>>> model.evaluate (X_test, y_test)
8832/10000 [==========================] - ETA: 0s - loss: 0.4074 - acc: 0.8540
[0.40738476498126985, 0.854]
As we saw in Chapter 2 , it is common to get slightly lower performance on the test set
than on the validation set, because the hyperparameters are tuned on the validation
set, not the test set (however, in this example, we did not do any hyperparameter tun‐

ing, so the lower accuracy is just bad luck). Remember to resist the temptation to
tweak the hyperparameters on the test set, or else your estimate of the generalization
error will be too optimistic.
Using the Model to Make Predictions
Next, we can use the model’s predict()  method to make predictions on new instan‐
ces. Since we don’t have actual new instances, we will just use the first 3 instances of
the test set:
>>> X_new = X_test[:3]
>>> y_proba = model.predict(X_new)
>>> y_proba.round(2)

array([[0.  , 0.  , 0.  , 0.  , 0.  , 0.09, 0.  , 0.12, 0.  , 0.79],
       [0.  , 0.  , 0.94, 0.  , 0.02, 0.  , 0.04, 0.  , 0.  , 0.  ],
       [0.  , 1.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  ]],
      dtype=float32)
As you can see, for each instance the model estimates one probability per class, from
class 0 to class 9. For example, for the first image it estimates that the probability of
class 9 (ankle boot) is 79%, the probability of class 7 (sneaker) is 12%, the probability

of class 5 (sandal) is 9%, and the other classes are negligible. In other words, it
“believes” it’s footwear, probably ankle boots, but it’s not entirely sure, it might be
sneakers or sandals instead. If you only care about the class with the highest estima‐
ted probability (even if that probability is quite low) then you can use the pre
dict_classes()  method instead:
>>> y_pred = model.predict_classes (X_new)
>>> y_pred
array([9, 2, 1])
>>> np.array(class_names )[y_pred]

>>> y_pred
array(['Ankle boot', 'Pullover', 'Trouser'], dtype='<U11')
And the classifier actually classified all three images correctly:302 | Chapter 10: Introduction to Artificial  Neural Networks with Keras

>>> y_new = y_test[:3]
>>> y_new
Now you know how to build, train, evaluate and use a classification MLP using the
Sequential API. But what about regression?
Building a Regression MLP Using the Sequential API
Let’s switch to the California housing problem and tackle it using a regression neural
network. For simplicity, we will use Scikit-Learn’s fetch_california_housing()
function to load the data: this dataset is simpler than the one we used in Chapter 2 ,
since it contains only numerical features (there is no ocean_proximity  feature), and
there is no missing value. After loading the data, we split it into a training set, a vali‐
dation set and a test set, and we scale all the features:
from sklearn.datasets  import fetch_california_housing
housing = fetch_california_housing ()
X_train_full , X_test, y_train_full , y_test = train_test_split (
    housing.data, housing.target)

X_train, X_valid, y_train, y_valid = train_test_split (
    X_train_full , y_train_full )
scaler = StandardScaler ()
X_train_scaled  = scaler.fit_transform (X_train)
X_valid_scaled  = scaler.transform (X_valid)
X_test_scaled  = scaler.transform (X_test)
Building, training, evaluating and using a regression MLP using the Sequential API to
make predictions is quite similar to what we did for classification. The main differ‐
ences are the fact that the output layer has a single neuron (since we only want to

predict a single value) and uses no activation function, and the loss function is the
mean squared error. Since the dataset is quite noisy, we just use a single hidden layer
with fewer neurons than before, to avoid overfitting:
    keras.layers.Dense(30, activation ="relu", input_shape =X_train.shape[1:]),
    keras.layers.Dense(1)
])
model.compile(loss="mean_squared_error" , optimizer ="sgd")
history = model.fit(X_train, y_train, epochs=20,

])
                    validation_data =(X_valid, y_valid))
mse_test  = model.evaluate (X_test, y_test)
X_new = X_test[:3] # pretend these are new instances
y_pred = model.predict(X_new)Implementing MLPs with Keras | 303

14“Wide & Deep Learning for Recommender Systems, ” Heng-Tze Cheng et al. (2016).As you can see, the Sequential API is quite easy to use. However, although sequential
models are extremely common, it is sometimes useful to build neural networks with
more complex topologies, or with multiple inputs or outputs. For this purpose, Keras
offers the Functional API.
Building Complex Models Using the Functional API
One example of a non-sequential neural network is a Wide & Deep  neural network.
This neural network architecture was introduced in a 2016 paper  by Heng-Tze Cheng
et al.14. It connects all or part of the inputs directly to the output layer, as shown in
Figure 10-13 . This architecture makes it possible for the neural network to learn both
deep patterns (using the deep path) and simple rules (through the short path). In
contrast, a regular MLP forces all the data to flow through the full stack of layers, thus
simple patterns in the data may end up being distorted by this sequence of transfor‐

mations.304 | Chapter 10: Introduction to Artificial  Neural Networks with Keras

Figure 10-13. Wide and Deep Neural Network
Let’s build such a neural network to tackle the California housing problem:
input = keras.layers.Input(shape=X_train.shape[1:])
hidden1 = keras.layers.Dense(30, activation ="relu")(input)
hidden2 = keras.layers.Dense(30, activation ="relu")(hidden1)
concat = keras.layers.Concatenate ()[input, hidden2])
output = keras.layers.Dense(1)(concat)
model = keras.models.Model(inputs=[input], outputs=[output])
Let’s go through each line of this code:•
First, we need to create an Input  object. This is needed because we may have
multiple inputs, as we will see later.•
Next, we create a Dense  layer with 30 neurons and using the ReLU activation
function. As soon as it is created, notice that we call it like a function, passing it
the input. This is why this is called the Functional API. Note that we are just tell‐Implementing MLPs with Keras | 305

ing Keras how it should connect the layers together, no actual data is being pro‐
cessed yet.•
We then create a second hidden layer, and again we use it as a function. Note
however that we pass it the output of the first hidden layer.•
Next, we create a Concatenate()  layer, and once again we immediately use it like
a function, to concatenate the input and the output of the second hidden layer
(you may prefer the keras.layers.concatenate()  function, which creates a Con
catenate  layer and immediately calls it with the given inputs).•
Then we create the output layer, with a single neuron and no activation function,
and we call it like a function, passing it the result of the concatenation.•
Lastly, we create a Keras Model , specifying which inputs and outputs to use.
Once you have built the Keras model, everything is exactly like earlier, so no need to
repeat it here: you must compile the model, train it, evaluate it and use it to make
predictions.

But what if you want to send a subset of the features through the wide path, and a
different subset (possibly overlapping) through the deep path (see Figure 10-14 )? In
this case, one solution is to use multiple inputs. For example, suppose we want to
send 5 features through the deep path (features 0 to 4), and 6 features through the
wide path (features 2 to 7):
input_A = keras.layers.Input(shape=[5])
input_B = keras.layers.Input(shape=[6])
hidden1 = keras.layers.Dense(30, activation ="relu")(input_B)

concat = keras.layers.concatenate ([input_A, hidden2])
model = keras.models.Model(inputs=[input_A, input_B], outputs=[output])306 | Chapter 10: Introduction to Artificial  Neural Networks with Keras

Figure 10-14. Handling Multiple Inputs
The code is self-explanatory. Note that we specified inputs=[input_A, input_B]
when creating the model. Now we can compile the model as usual, but when we call
the fit()  method, instead of passing a single input matrix X_train , we must pass a
pair of matrices (X_train_A, X_train_B) : one per input. The same is true for
X_valid , and also for X_test  and X_new  when you call evaluate()  or predict() :
model.compile(loss="mse", optimizer ="sgd")
X_train_A , X_train_B  = X_train[:, :5], X_train[:, 2:]
X_valid_A , X_valid_B  = X_valid[:, :5], X_valid[:, 2:]
X_test_A , X_test_B  = X_test[:, :5], X_test[:, 2:]
X_new_A, X_new_B = X_test_A [:3], X_test_B [:3]
history = model.fit((X_train_A , X_train_B ), y_train, epochs=20,
                    validation_data =((X_valid_A , X_valid_B ), y_valid))
mse_test  = model.evaluate ((X_test_A , X_test_B ), y_test)
y_pred = model.predict((X_new_A, X_new_B))Implementing MLPs with Keras | 307

There are also many use cases in which you may want to have multiple outputs:•
The task may demand it, for example you may want to locate and classify the
main object in a picture. This is both a regression task (finding the coordinates of
the object’s center, as well as its width and height) and a classification task.•
Similarly, you may have multiple independent tasks to perform based on the
same data. Sure, you could train one neural network per task, but in many cases
you will get better results on all tasks by training a single neural network with
one output per task. This is because the neural network can learn features in the
data that are useful across tasks.•
Another use case is as a regularization technique (i.e., a training constraint whose
objective is to reduce overfitting and thus improve the model’s ability to general‐
ize). For example, you may want to add some auxiliary outputs in a neural net‐
work architecture (see Figure 10-15 ) to ensure that the underlying part of the

network learns something useful on its own, without relying on the rest of the
network.
Figure 10-15. Handling Multiple Outputs – Auxiliary Output for Regularization
Adding extra outputs is quite easy: just connect them to the appropriate layers and
add them to your model’s list of outputs. For example, the following code builds the
network represented in Figure 10-15 :308 | Chapter 10: Introduction to Artificial  Neural Networks with Keras

[...] # Same as above, up to the main output layer
aux_output  = keras.layers.Dense(1)(hidden2)
model = keras.models.Model(inputs=[input_A, input_B],
                           outputs=[output, aux_output ])
Each output will need its own loss function, so when we compile the model we
should pass a list of losses (if we pass a single loss, Keras will assume that the same
loss must be used for all outputs). By default, Keras will compute all these losses and
simply add them up to get the final loss used for training. However, we care much
more about the main output than about the auxiliary output (as it is just used for reg‐
ularization), so we want to give the main output’s loss a much greater weight. Fortu‐
nately, it is possible to set all the loss weights when compiling the model:
model.compile(loss=["mse", "mse"], loss_weights =[0.9, 0.1], optimizer ="sgd")
Now when we train the model, we need to provide some labels for each output. In

this example, the main output and the auxiliary output should try to predict the same
thing, so they should use the same labels. So instead of passing y_train , we just need
to pass (y_train, y_train)  (and the same goes for y_valid  and y_test ):
history = model.fit(
    [X_train_A , X_train_B ], [y_train, y_train], epochs=20,
    validation_data =([X_valid_A , X_valid_B ], [y_valid, y_valid]))
When we evaluate the model, Keras will return the total loss, as well as all the individ‐
ual losses:

total_loss , main_loss , aux_loss  = model.evaluate (
    [X_test_A , X_test_B ], [y_test, y_test])
Similarly, the predict()  method will return predictions for each output:
y_pred_main , y_pred_aux  = model.predict([X_new_A, X_new_B])
As you can see, you can build any sort of architecture you want quite easily with the
Functional API. Let’s look at one last way you can build Keras models.
Building Dynamic Models Using the Subclassing API

Both the Sequential API and the Functional API are declarative: you start by declar‐
ing which layers you want to use and how they should be connected, and only then
can you start feeding the model some data for training or inference. This has many
advantages: the model can easily be saved, cloned, shared, its structure can be dis‐
played and analyzed, the framework can infer shapes and check types, so errors can
be caught early (i.e., before any data ever goes through the model). It’s also fairly easy

to debug, since the whole model is just a static graph of layers. But the flip side is just
that: it’s static. Some models involve loops, varying shapes, conditional branching,
and other dynamic behaviors. For such cases, or simply if you prefer a more impera‐
tive programming style, the Subclassing API is for you.Implementing MLPs with Keras | 309

15Keras models have an output  attribute, so we cannot use that name for the main output layer, which is why
we renamed it to main_output .Simply subclass the Model  class, create the layers you need in the constructor, and use
them to perform the computations you want in the call()  method. For example, cre‐
ating an instance of the following WideAndDeepModel  class gives us an equivalent
model to the one we just built with the Functional API. Y ou can then compile it, eval‐
uate it and use it to make predictions, exactly like we just did.
class WideAndDeepModel (keras.models.Model):
    def __init__ (self, units=30, activation ="relu", **kwargs):
        super().__init__ (**kwargs) # handles standard args (e.g., name)
        self.hidden1 = keras.layers.Dense(units, activation =activation )
        self.hidden2 = keras.layers.Dense(units, activation =activation )
        self.main_output  = keras.layers.Dense(1)
        self.aux_output  = keras.layers.Dense(1)
    def call(self, inputs):

        input_A, input_B = inputs
        hidden1 = self.hidden1(input_B)
        hidden2 = self.hidden2(hidden1)
        concat = keras.layers.concatenate ([input_A, hidden2])
        main_output  = self.main_output (concat)
        aux_output  = self.aux_output (hidden2)
        return main_output , aux_output
model = WideAndDeepModel ()
This example looks very much like the Functional API, except we do not need to cre‐
ate the inputs, we just use the input  argument to the call()  method, and we separate

hidden1 = self.hidden1(input_B)
the creation of the layers15 in the constructor from their usage in the call()  method.
However, the big difference is that you can do pretty much anything you want in the
call()  method: for loops, if statements, low-level TensorFlow operations, your
imagination is the limit (see Chapter 12 )! This makes it a great API for researchers
experimenting with new ideas.
However, this extra flexibility comes at a cost: your model’s architecture is hidden
within the call()  method, so Keras cannot easily inspect it, it cannot save or clone it,

and when you call the summary()  method, you only get a list of layers, without any
information on how they are connected to each other. Moreover, Keras cannot check
types and shapes ahead of time, and it is easier to make mistakes. So unless you really
need that extra flexibility, you should probably stick to the Sequential API or the
Functional API.310 | Chapter 10: Introduction to Artificial  Neural Networks with Keras

Keras models can be used just like regular layers, so you can easily
compose them to build complex architectures.
Now that you know how to build and train neural nets using Keras, you will want to
save them!
Saving and Restoring a Model
Saving a trained Keras model is as simple as it gets:
model.save("my_keras_model.h5" )
Keras will save both the model’s architecture (including every layer’s hyperparame‐
ters) and the value of all the model parameters for every layer (e.g., connection
weights and biases), using the HDF5 format. It also saves the optimizer (including its
hyperparameters and any state it may have).
Y ou will typically have a script that trains a model and saves it, and one or more
scripts (or web services) that load the model and use it to make predictions. Loading
the model is just as easy:
model = keras.models.load_model ("my_keras_model.h5" )
This will work when using the Sequential API or the Functional
API, but unfortunately not when using Model subclassing. How‐

ever, you can use save_weights()  and load_weights()  to at least
save and restore the model parameters (but you will need to save
and restore everything else yourself).
But what if training lasts several hours? This is quite common, especially when train‐
ing on large datasets. In this case, you should not only save your model at the end of
training, but also save checkpoints at regular intervals during training. But how can
you tell the fit()  method to save checkpoints? The answer is: using callbacks.

Using Callbacks
The fit()  method accepts a callbacks  argument that lets you specify a list of objects
that Keras will call during training at the start and end of training, at the start and end
of each epoch and even before and after processing each batch. For example, the Mod
elCheckpoint  callback saves checkpoints of your model at regular intervals during
training, by default at the end of each epoch:Implementing MLPs with Keras | 311

[...] # build and compile the model
checkpoint_cb  = keras.callbacks .ModelCheckpoint ("my_keras_model.h5" )
history = model.fit(X_train, y_train, epochs=10, callbacks =[checkpoint_cb ])
Moreover, if you use a validation set during training, you can set
save_best_only=True  when creating the ModelCheckpoint . In this case, it will only
save your model when its performance on the validation set is the best so far. This
way, you do not need to worry about training for too long and overfitting the training
set: simply restore the last model saved after training, and this will be the best model
on the validation set. This is a simple way to implement early stopping (introduced in
Chapter 4 ):
checkpoint_cb  = keras.callbacks .ModelCheckpoint ("my_keras_model.h5" ,
                                                save_best_only =True)
history = model.fit(X_train, y_train, epochs=10,
                    validation_data =(X_valid, y_valid),
                    callbacks =[checkpoint_cb ])

model = keras.models.load_model ("my_keras_model.h5" ) # rollback to best model
Another way to implement early stopping is to simply use the EarlyStopping  call‐
back. It will interrupt training when it measures no progress on the validation set for
a number of epochs (defined by the patience  argument), and it will optionally roll
back to the best model. Y ou can combine both callbacks to both save checkpoints of
your model (in case your computer crashes), and actually interrupt training early

when there is no more progress (to avoid wasting time and resources):
early_stopping_cb  = keras.callbacks .EarlyStopping (patience =10,
                                                  restore_best_weights =True)
history = model.fit(X_train, y_train, epochs=100,
                    callbacks =[checkpoint_cb , early_stopping_cb ])
The number of epochs can be set to a large value since training will stop automati‐

cally when there is no more progress. Moreover, there is no need to restore the best
model saved in this case since the EarlyStopping  callback will keep track of the best
weights and restore them for us at the end of training.
There are many other callbacks available in the keras.callbacks
package. See https://keras.io/callbacks/ .
If you need extra control, you can easily write your own custom callbacks. For exam‐
ple, the following custom callback will display the ratio between the validation loss

and the training loss during training (e.g., to detect overfitting):312 | Chapter 10: Introduction to Artificial  Neural Networks with Keras

class PrintValTrainRatioCallback (keras.callbacks .Callback ):
    def on_epoch_end (self, epoch, logs):
        print("\nval/train: {:.2f}" .format(logs["val_loss" ] / logs["loss"]))
As you might expect, you can implement on_train_begin() , on_train_end() ,
on_epoch_begin() , on_epoch_begin() , on_batch_end()  and on_batch_end() .
Moreover, callbacks can also be used during evaluation and predictions, should you
ever need them (e.g., for debugging). In this case, you should implement
on_test_begin() , on_test_end() , on_test_batch_begin() , or
on_test_batch_end()  (called by evaluate() ), or on_predict_begin() , on_pre
dict_end() , on_predict_batch_begin() , or on_predict_batch_end()  (called by
predict() ).
Now let’s take a look at one more tool you should definitely have in your toolbox
when using tf.keras: TensorBoard.
Visualization Using TensorBoard
TensorBoard is a great interactive visualization tool that you can use to view the

learning curves during training, compare learning curves between multiple runs, vis‐
ualize the computation graph, analyze training statistics, view images generated by
your model, visualize complex multidimensional data projected down to 3D and
automatically clustered for you, and more! This tool is installed automatically when
you install TensorFlow, so you already have it!
To use it, you must modify your program so that it outputs the data you want to visu‐
alize to special binary log files called event files. Each binary data record is called a

summary . The TensorBoard server will monitor the log directory, and it will automat‐
ically pick up the changes and update the visualizations: this allows you to visualize
live data (with a short delay), such as the learning curves during training. In general,
you want to point the TensorBoard server to a root log directory, and configure your
program so that it writes to a different subdirectory every time it runs. This way, the
same TensorBoard server instance will allow you to visualize and compare data from

multiple runs of your program, without getting everything mixed up.
So let’s start by defining the root log directory we will use for our TensorBoard logs,
plus a small function that will generate a subdirectory path based on the current date
and time, so that it is different at every run. Y ou may want to include extra informa‐
tion in the log directory name, such as hyperparameter values that you are testing, to
make it easier to know what you are looking at in TensorBoard:
root_logdir  = os.path.join(os.curdir, "my_logs" )
def get_run_logdir ():
    import time

    run_id = time.strftime ("run_%Y_%m_%d-%H_%M_%S")
    return os.path.join(root_logdir , run_id)Implementing MLPs with Keras | 313

run_logdir  = get_run_logdir () # e.g., './my_logs/run_2019_01_16-11_28_43'
Next, the good news is that Keras provides a nice TensorBoard  callback:
[...] # Build and compile your model
tensorboard_cb  = keras.callbacks .TensorBoard (run_logdir )
history = model.fit(X_train, y_train, epochs=30,
                    callbacks =[tensorboard_cb ])
And that’s all there is to it! It could hardly be easier to use. If you run this code, the
TensorBoard  callback will take care of creating the log directory for you (along with
its parent directories if needed), and during training it will create event files and write
summaries to them. After running the program a second time (perhaps changing
some hyperparameter value), you will end up with a directory structure similar to
this one:
my_logs
├── run_2019_01_16-16_51_02
│   └── events.out.tfevents.1547628669.mycomputer.local.v2
└── run_2019_01_16-16_56_50
    └── events.out.tfevents.1547629020.mycomputer.local.v2

this one:
my_logs
Next you need to start the TensorBoard server. If you installed TensorFlow within a
virtualenv, you should activate it. Next, run the following command at the root of the
project (or from anywhere else as long as you point to the appropriate log directory).
If your shell cannot find the tensorboard  script, then you must update your PATH
environment variable so that it contains the directory in which the script was
installed (alternatively, you can just replace tensorboard  with python3 -m tensor
board.main ).
$ tensorboard --logdir =./my_logs --port =6006

TensorBoard 2.0.0 at http://mycomputer.local:6006 (Press CTRL+C to quit )
Finally, open up a web browser to http://localhost:6006 . Y ou should see TensorBoard’s
web interface. Click on the SCALARS tab to view the learning curves (see
Figure 10-16 ). Notice that the training loss went down nicely during both runs, but
the second run went down much faster. Indeed, we used a larger learning rate by set‐
ting optimizer=keras.optimizers.SGD(lr=0.05)  instead of optimizer="sgd" ,

which defaults to a learning rate of 0.001.314 | Chapter 10: Introduction to Artificial  Neural Networks with Keras

Figure 10-16. Visualizing Learning Curves with TensorBoard
Unfortunately, at the time of writing, no other data is exported by the TensorBoard
callback, but this issue will probably be fixed by the time you read these lines. In Ten‐
sorFlow 1, this callback exported the computation graph and many useful statistics:
type help(keras.callbacks.TensorBoard)  to see all the options.
Let’s summarize what you learned so far in this chapter: we saw where neural nets
came from, what an MLP is and how you can use it for classification and regression,
how to build MLPs using tf.keras’s Sequential API, or more complex architectures
using the Functional API or Model  Subclassing, you learned how to save and restore a
model, use callbacks for checkpointing, early stopping, and more, and finally how to
use TensorBoard for visualization. Y ou can already go ahead and use neural networks
to tackle many problems! However, you may wonder how to choose the number of

hidden layers, the number of neurons in the network, and all the other hyperparame‐
ters. Let’s look at this now.
Fine-Tuning Neural Network Hyperparameters
The flexibility of neural networks is also one of their main drawbacks: there are many
hyperparameters to tweak. Not only can you use any imaginable network architec‐
ture, but even in a simple MLP you can change the number of layers, the number of

neurons per layer, the type of activation function to use in each layer, the weight initi‐Fine-Tuning Neural Network Hyperparameters | 315

alization logic, and much more. How do you know what combination of hyperpara‐
meters is the best for your task?
One option is to simply try many combinations of hyperparameters and see which
one works best on the validation set (or using K-fold cross-validation). For this, one
approach is simply use GridSearchCV  or RandomizedSearchCV  to explore the hyper‐
parameter space, as we did in Chapter 2 . For this, we need to wrap our Keras models
in objects that mimic regular Scikit-Learn regressors. The first step is to create a func‐
tion that will build and compile a Keras model, given a set of hyperparameters:
def build_model (n_hidden =1, n_neurons =30, learning_rate =3e-3, input_shape =[8]):
    model = keras.models.Sequential ()
    options = {"input_shape" : input_shape }
    for layer in range(n_hidden ):
        model.add(keras.layers.Dense(n_neurons , activation ="relu", **options))
        options = {}
    model.add(keras.layers.Dense(1, **options))

    optimizer  = keras.optimizers .SGD(learning_rate )
    model.compile(loss="mse", optimizer =optimizer )
    return model
This function creates a simple Sequential  model for univariate regression (only one
output neuron), with the given input shape and the given number of hidden layers
and neurons, and it compiles it using an SGD optimizer configured with the given
learning rate. The options  dict is used to ensure that the first layer is properly given
the input shape (note that if n_hidden=0 , the first layer will be the output layer). It is

model.compile(loss="mse", optimizer =optimizer )
good practice to provide reasonable defaults to as many hyperparameters as you can,
as Scikit-Learn does.
Next, let’s create a KerasRegressor  based on this build_model()  function:
keras_reg  = keras.wrappers .scikit_learn .KerasRegressor (build_model )
The KerasRegressor  object is a thin wrapper around the Keras model built using
build_model() . Since we did not specify any hyperparameter when creating it, it will
just use the default hyperparameters we defined in build_model() . Now we can use

this object like a regular Scikit-Learn regressor: we can train it using its fit()
method, then evaluate it using its score()  method, and use it to make predictions
using its predict()  method. Note that any extra parameter you pass to the fit()
method will simply get passed to the underlying Keras model. Also note that the
score will be the opposite of the MSE because Scikit-Learn wants scores, not losses
(i.e., higher should be better).
keras_reg .fit(X_train, y_train, epochs=100,

              validation_data =(X_valid, y_valid),
              callbacks =[keras.callbacks .EarlyStopping (patience =10)])
mse_test  = keras_reg .score(X_test, y_test)
y_pred = keras_reg .predict(X_new)316 | Chapter 10: Introduction to Artificial  Neural Networks with Keras

However, we do not actually want to train and evaluate a single model like this, we
want to train hundreds of variants and see which one performs best on the validation
set. Since there are many hyperparameters, it is preferable to use a randomized search
rather than grid search (as we discussed in Chapter 2 ). Let’s try to explore the number
of hidden layers, the number of neurons and the learning rate:
from scipy.stats  import reciprocal
from sklearn.model_selection  import RandomizedSearchCV
param_distribs  = {
    "n_hidden" : [0, 1, 2, 3],
    "n_neurons" : np.arange(1, 100),
    "learning_rate" : reciprocal (3e-4, 3e-2),
}
rnd_search_cv  = RandomizedSearchCV (keras_reg , param_distribs , n_iter=10, cv=3)
rnd_search_cv .fit(X_train, y_train, epochs=100,
                  validation_data =(X_valid, y_valid),
                  callbacks =[keras.callbacks .EarlyStopping (patience =10)])
As you can see, this is identical to what we did in Chapter 2 , with the exception that

}
we pass extra parameters to the fit()  method: they simply get relayed to the under‐
lying Keras models. Note that RandomizedSearchCV  uses K-fold cross-validation, so it
does not use X_valid  and y_valid . These are just used for early stopping.
The exploration may last many hours depending on the hardware, the size of the
dataset, the complexity of the model and the value of n_iter  and cv. When it is over,
you can access the best parameters found, the best score, and the trained Keras model
like this:

like this:
>>> rnd_search_cv .best_params_
{'learning_rate': 0.0033625641252688094, 'n_hidden': 2, 'n_neurons': 42}
>>> rnd_search_cv .best_score_
-0.3189529188278931
>>> model = rnd_search_cv .best_estimator_ .model
Y ou can now save this model, evaluate it on the test set, and if you are satisfied with
its performance, deploy it to production. Using randomized search is not too hard,
and it works well for many fairly simple problems. However, when training is slow

like this:
(e.g., for more complex problems with larger datasets), this approach will only
explore a tiny portion of the hyperparameter space. Y ou can partially alleviate this
problem by assisting the search process manually: first run a quick random search
using wide ranges of hyperparameter values, then run another search using smaller
ranges of values centered on the best ones found during the first run, and so on. This
will hopefully zoom in to a good set of hyperparameters. However, this is very time

consuming, and probably not the best use of your time.
Fortunately, there are many techniques to explore a search space much more effi‐
ciently than randomly. Their core idea is simple: when a region of the space turns outFine-Tuning Neural Network Hyperparameters | 317

16“Population Based Training of Neural Networks, ” Max Jaderberg et al. (2017).to be good, it should be explored more. This takes care of the “zooming” process for
you and leads to much better solutions in much less time. Here are a few Python
libraries you can use to optimize hyperparameters:•
Hyperopt : a popular Python library for optimizing over all sorts of complex
search spaces (including real values such as the learning rate, or discrete values
such as the number of layers).•
Hyperas , kopt  or Talos : optimizing hyperparameters for Keras model (the first
two are based on Hyperopt).•
Scikit-Optimize  (skopt): a general-purpose optimization library. The Bayes
SearchCV  class performs Bayesian optimization using an interface similar to Grid
SearchCV .•
Spearmint : a Bayesian optimization library.•
Sklearn-Deap : a hyperparameter optimization library based on evolutionary
algorithms, also with a GridSearchCV -like interface.•
And many more!

Moreover, many companies offer services for hyperparameter optimization. For
example Google Cloud ML Engine has a hyperparameter tuning service . Other com‐
panies provide APIs for hyperparameter optimization, such as Arimo , SigOpt , Oscar
and many more.
Hyperparameter tuning is still an active area of research. Evolutionary algorithms are
making a comeback lately. For example, check out DeepMind’s excellent 2017 paper16,
where they jointly optimize a population of models and their hyperparameters. Goo‐

gle also used an evolutionary approach, not just to search for hyperparameters, but
also to look for the best neural network architecture for the problem. They call this
AutoML , and it is already available as a cloud service . Perhaps the days of building
neural networks manually will soon be over? Check out Google’s post  on this topic. In
fact, evolutionary algorithms have also been used successfully to train individual neu‐

ral networks, replacing the ubiquitous Gradient Descent! See this 2017 post  by Uber
where they introduce their Deep Neuroevolution  technique.
Despite all this exciting progress, and all these tools and services, it still helps to have
an idea of what values are reasonable for each hyperparameter, so you can build a
quick prototype, and restrict the search space. Here are a few guidelines for choosing
the number of hidden layers and neurons in an MLP , and selecting good values for

some of the main hyperparameters.318 | Chapter 10: Introduction to Artificial  Neural Networks with Keras

Number of Hidden Layers
For many problems, you can just begin with a single hidden layer and you will get
reasonable results. It has actually been shown that an MLP with just one hidden layer
can model even the most complex functions provided it has enough neurons. For a
long time, these facts convinced researchers that there was no need to investigate any
deeper neural networks. But they overlooked the fact that deep networks have a much
higher parameter efficiency  than shallow ones: they can model complex functions
using exponentially fewer neurons than shallow nets, allowing them to reach much
better performance with the same amount of training data.
To understand why, suppose you are asked to draw a forest using some drawing soft‐
ware, but you are forbidden to use copy/paste. Y ou would have to draw each tree
individually, branch per branch, leaf per leaf. If you could instead draw one leaf,
copy/paste it to draw a branch, then copy/paste that branch to create a tree, and

finally copy/paste this tree to make a forest, you would be finished in no time. Real-
world data is often structured in such a hierarchical way and Deep Neural Networks
automatically take advantage of this fact: lower hidden layers model low-level struc‐
tures (e.g., line segments of various shapes and orientations), intermediate hidden
layers combine these low-level structures to model intermediate-level structures (e.g.,
squares, circles), and the highest hidden layers and the output layer combine these

intermediate structures to model high-level structures (e.g., faces).
Not only does this hierarchical architecture help DNNs converge faster to a good sol‐
ution, it also improves their ability to generalize to new datasets. For example, if you
have already trained a model to recognize faces in pictures, and you now want to
train a new neural network to recognize hairstyles, then you can kickstart training by
reusing the lower layers of the first network. Instead of randomly initializing the

weights and biases of the first few layers of the new neural network, you can initialize
them to the value of the weights and biases of the lower layers of the first network.
This way the network will not have to learn from scratch all the low-level structures
that occur in most pictures; it will only have to learn the higher-level structures (e.g.,
hairstyles). This is called transfer learning .
In summary, for many problems you can start with just one or two hidden layers and

it will work just fine (e.g., you can easily reach above 97% accuracy on the MNIST
dataset using just one hidden layer with a few hundred neurons, and above 98% accu‐
racy using two hidden layers with the same total amount of neurons, in roughly the
same amount of training time). For more complex problems, you can gradually ramp
up the number of hidden layers, until you start overfitting the training set. Very com‐
plex tasks, such as large image classification or speech recognition, typically require

networks with dozens of layers (or even hundreds, but not fully connected ones, as
we will see in Chapter 14 ), and they need a huge amount of training data. However,
you will rarely have to train such networks from scratch: it is much more common toFine-Tuning Neural Network Hyperparameters | 319

17By Vincent Vanhoucke in his Deep Learning class  on Udacity.com. reuse parts of a pretrained state-of-the-art network that performs a similar task.
Training will be a lot faster and require much less data (we will discuss this in Chap‐
ter 11 ).
Number of Neurons per Hidden Layer
Obviously the number of neurons in the input and output layers is determined by the
type of input and output your task requires. For example, the MNIST task requires 28
x 28 = 784 input neurons and 10 output neurons.
As for the hidden layers, it used to be a common practice to size them to form a pyra‐
mid, with fewer and fewer neurons at each layer—the rationale being that many low-
level features can coalesce into far fewer high-level features. For example, a typical
neural network for MNIST may have three hidden layers, the first with 300 neurons,
the second with 200, and the third with 100. However, this practice has been largely
abandoned now, as it seems that simply using the same number of neurons in all hid‐

den layers performs just as well in most cases, or even better, and there is just one
hyperparameter to tune instead of one per layer—for example, all hidden layers could
simply have 150 neurons. However, depending on the dataset, it can sometimes help
to make the first hidden layer bigger than the others.
Just like for the number of layers, you can try increasing the number of neurons grad‐
ually until the network starts overfitting. In general you will get more bang for the

buck by increasing the number of layers than the number of neurons per layer.
Unfortunately, as you can see, finding the perfect amount of neurons is still somewhat
of a dark art.
A simpler approach is to pick a model with more layers and neurons than you
actually need, then use early stopping to prevent it from overfitting (and other regu‐
larization techniques, such as dropout , as we will see in Chapter 11 ). This has been
dubbed the “stretch pants” approach:17 instead of wasting time looking for pants that

perfectly match your size, just use large stretch pants that will shrink down to the
right size.
Learning Rate, Batch Size and Other Hyperparameters
The number of hidden layers and neurons are not the only hyperparameters you can
tweak in an MLP . Here are some of the most important ones, and some tips on how
to set them:•
The learning rate is arguably the most important hyperparameter. In general, the
optimal learning rate is about half of the maximum learning rate (i.e., the learn‐320 | Chapter 10: Introduction to Artificial  Neural Networks with Keras

18“Practical recommendations for gradient-based training of deep architectures, ” Y oshua Bengio (2012).ing rate above which the training algorithm diverges, as we saw in Chapter 4 ). So
a simple approach for tuning the learning rate is to start with a large value that
makes the training algorithm diverge, then divide this value by 3 and try again,
and repeat until the training algorithm stops diverging. At that point, you gener‐
ally won’t be too far from the optimal learning rate. That said, it is sometimes
useful to reduce the learning rate during training: we will discuss this in Chap‐
ter 11 .•
Choosing a better optimizer than plain old Mini-batch Gradient Descent (and
tuning its hyperparameters) is also quite important. We will discuss this in Chap‐
ter 11 .•
The batch size can also have a significant impact on your model’s performance
and the training time. In general the optimal batch size will be lower than 32 (in
April 2018, Y ann Lecun even tweeted " Friends don’t let friends use mini-batches

ter 11 .•
ter 11 .•
larger than 32 “). A small batch size ensures that each training iteration is very
fast, and although a large batch size will give a more precise estimate of the gradi‐
ents, in practice this does not matter much since the optimization landscape is
quite complex and the direction of the true gradients do not point precisely in
the direction of the optimum. However, having a batch size greater than 10 helps
take advantage of hardware and software optimizations, in particular for matrix

multiplications, so it will speed up training. Moreover, if you use Batch Normal‐
ization  (see Chapter 11 ), the batch size should not be too small (in general no less
than 20).•
We discussed the choice of the activation function earlier in this chapter: in gen‐
eral, the ReLU activation function will be a good default for all hidden layers. For
the output layer, it really depends on your task.•
In most cases, the number of training iterations does not actually need to be
tweaked: just use early stopping instead.

than 20).•
For more best practices, make sure to read Y oshua Bengio’s great 2012 paper18, which
presents many practical recommendations for deep networks.
This concludes this introduction to artificial neural networks and their implementa‐
tion with Keras. In the next few chapters, we will discuss techniques to train very
deep nets, we will see how to customize your models using TensorFlow’s lower-level
API and how to load and preprocess data efficiently using the Data API, and we will
dive into other popular neural network architectures: convolutional neural networks

for image processing, recurrent neural networks for sequential data, autoencoders forFine-Tuning Neural Network Hyperparameters | 321

19A few extra ANN architectures are presented in ???. representation learning, and generative adversarial networks to model and generate
data.19
Visit the TensorFlow Playground at https://playground.tensorflow.org/•
Layers and patterns: try training the default neural network by clicking the run
button (top left). Notice how it quickly finds a good solution for the classifica‐
tion task. Notice that the neurons in the first hidden layer have learned simple
patterns, while the neurons in the second hidden layer have learned to com‐
bine the simple patterns of the first hidden layer into more complex patterns.
In general, the more layers, the more complex the patterns can be.•
Activation function: try replacing the Tanh activation function with the ReLU
activation function, and train the network again. Notice that it finds a solution
even faster, but this time the boundaries are linear. This is due to the shape of
the ReLU function.•

Local minima: modify the network architecture to have just one hidden layer
with three neurons. Train it multiple times (to reset the network weights, click
the reset button next to the play button). Notice that the training time varies a
lot, and sometimes it even gets stuck in a local minimum.•
Too small: now remove one neuron to keep just 2. Notice that the neural net‐
work is now incapable of finding a good solution, even if you try multiple
times. The model has too few parameters and it systematically underfits the

training set.•
Large enough: next, set the number of neurons to 8 and train the network sev‐
eral times. Notice that it is now consistently fast and never gets stuck. This
highlights an important finding in neural network theory: large neural net‐
works almost never get stuck in local minima, and even when they do these
local optima are almost as good as the global optimum. However, they can still
get stuck on long plateaus for a long time.•
Deep net and vanishing gradients: now change the dataset to be the spiral (bot‐

tom right dataset under “DATA ”). Change the network architecture to have 4
hidden layers with 8 neurons each. Notice that training takes much longer, and
often gets stuck on plateaus for long periods of time. Also notice that the neu‐
rons in the highest layers (i.e. on the right) tend to evolve faster than the neu‐
rons in the lowest layers (i.e. on the left). This problem, called the “vanishing

gradients” problem, can be alleviated using better weight initialization and322 | Chapter 10: Introduction to Artificial  Neural Networks with Keras

other techniques, better optimizers (such as AdaGrad or Adam), or using
Batch Normalization.•
More: go ahead and play with the other parameters to get a feel of what they
do. In fact, you should definitely play with this UI for at least one hour, it will
grow your intuitions about neural networks significantly.2.
Draw an ANN using the original artificial neurons (like the ones in Figure 10-3 )
that computes A ⊕ B (where ⊕ represents the XOR operation). Hint: A ⊕ B = (A
∧ ¬ B) ∨ (¬ A ∧ B).3.
Why is it generally preferable to use a Logistic Regression classifier rather than a
classical Perceptron (i.e., a single layer of threshold logic units trained using the
Perceptron training algorithm)? How can you tweak a Perceptron to make it
equivalent to a Logistic Regression classifier?4.
Why was the logistic activation function a key ingredient in training the first
MLPs?5.
Name three popular activation functions. Can you draw them?6.
Suppose you have an MLP composed of one input layer with 10 passthrough

MLPs?5.
neurons, followed by one hidden layer with 50 artificial neurons, and finally one
output layer with 3 artificial neurons. All artificial neurons use the ReLU activa‐
tion function.•
What is the shape of the input matrix X?•
What about the shape of the hidden layer’s weight vector Wh, and the shape of
its bias vector bh?•
What is the shape of the output layer’s weight vector Wo, and its bias vector bo?•
What is the shape of the network’s output matrix Y?•
Write the equation that computes the network’s output matrix Y as a function
of X, Wh, bh, Wo and bo.7.

How many neurons do you need in the output layer if you want to classify email
into spam or ham? What activation function should you use in the output layer?
If instead you want to tackle MNIST, how many neurons do you need in the out‐
put layer, using what activation function? Answer the same questions for getting
your network to predict housing prices as in Chapter 2 .8.
What is backpropagation and how does it work? What is the difference between
backpropagation and reverse-mode autodiff?9.

Can you list all the hyperparameters you can tweak in an MLP? If the MLP over‐
fits the training data, how could you tweak these hyperparameters to try to solve
the problem?Exercises | 323

10. Train a deep MLP on the MNIST dataset and see if you can get over 98% preci‐
sion. Try adding all the bells and whistles (i.e., save checkpoints, use early stop‐
ping, plot learning curves using TensorBoard, and so on).
Solutions to these exercises are available in ???.324 | Chapter 10: Introduction to Artificial  Neural Networks with Keras

CHAPTER 11
Training Deep Neural Networks
release of these titles. The following will be Chapter 11 in the final
In Chapter 10  we introduced artificial neural networks and trained our first deep
neural networks. But they were very shallow nets, with just a few hidden layers. What
if you need to tackle a very complex problem, such as detecting hundreds of types of
objects in high-resolution images? Y ou may need to train a much deeper DNN, per‐
haps with 10 layers or much more, each containing hundreds of neurons, connected
by hundreds of thousands of connections. This would not be a walk in the park:•
First, you would be faced with the tricky vanishing gradients  problem (or the
related exploding gradients  problem) that affects deep neural networks and makes
lower layers very hard to train.•

Second, you might not have enough training data for such a large network, or it
might be too costly to label.•
Third, training may be extremely slow.•
Fourth, a model with millions of parameters would severely risk overfitting the
training set, especially if there are not enough training instances, or they are too
noisy.
In this chapter, we will go through each of these problems in turn and present techni‐
ques to solve them. We will start by explaining the vanishing gradients problem and
exploring some of the most popular solutions to this problem. Next, we will look at

noisy.
transfer learning and unsupervised pretraining, which can help you tackle complex325

1“Understanding the Difficulty of Training Deep Feedforward Neural Networks, ” X. Glorot, Y Bengio (2010).tasks even when you have little labeled data. Then we will discuss various optimizers
that can speed up training large models tremendously compared to plain Gradient
Descent. Finally, we will go through a few popular regularization techniques for large
neural networks.
With these tools, you will be able to train very deep nets: welcome to Deep Learning!
Vanishing/Exploding Gradients Problems
As we discussed in Chapter 10 , the backpropagation algorithm works by going from
the output layer to the input layer, propagating the error gradient on the way. Once
the algorithm has computed the gradient of the cost function with regards to each
parameter in the network, it uses these gradients to update each parameter with a
Gradient Descent step.
Unfortunately, gradients often get smaller and smaller as the algorithm progresses
down to the lower layers. As a result, the Gradient Descent update leaves the lower

layer connection weights virtually unchanged, and training never converges to a good
solution. This is called the vanishing gradients  problem. In some cases, the opposite
can happen: the gradients can grow bigger and bigger, so many layers get insanely
large weight updates and the algorithm diverges. This is the exploding gradients  prob‐
lem, which is mostly encountered in recurrent neural networks (see ???). More gener‐
ally, deep neural networks suffer from unstable gradients; different layers may learn at
widely different speeds.

Although this unfortunate behavior has been empirically observed for quite a while
(it was one of the reasons why deep neural networks were mostly abandoned for a
long time), it is only around 2010 that significant progress was made in understand‐
ing it. A paper titled “Understanding the Difficulty of Training Deep Feedforward
Neural Networks”  by Xavier Glorot and Y oshua Bengio1 found a few suspects, includ‐
ing the combination of the popular logistic sigmoid activation function and the

weight initialization technique that was most popular at the time, namely random ini‐
tialization using a normal distribution with a mean of 0 and a standard deviation of 1.
In short, they showed that with this activation function and this initialization scheme,
the variance of the outputs of each layer is much greater than the variance of its
inputs. Going forward in the network, the variance keeps increasing after each layer
until the activation function saturates at the top layers. This is actually made worse by

the fact that the logistic function has a mean of 0.5, not 0 (the hyperbolic tangent
function has a mean of 0 and behaves slightly better than the logistic function in deep
networks).326 | Chapter 11: Training Deep Neural Networks

2Here’s an analogy: if you set a microphone amplifier’s knob too close to zero, people won’t hear your voice, but
if you set it too close to the max, your voice will be saturated and people won’t understand what you are say‐
ing. Now imagine a chain of such amplifiers: they all need to be set properly in order for your voice to come
out loud and clear at the end of the chain. Y our voice has to come out of each amplifier at the same amplitude
as it came in.Looking at the logistic activation function (see Figure 11-1 ), you can see that when
inputs become large (negative or positive), the function saturates at 0 or 1, with a
derivative extremely close to 0. Thus when backpropagation kicks in, it has virtually
no gradient to propagate back through the network, and what little gradient exists
keeps getting diluted as backpropagation progresses down through the top layers, so
there is really nothing left for the lower layers.
Figure 11-1. Logistic activation function saturation
Glorot and He Initialization

In their paper, Glorot and Bengio propose a way to significantly alleviate this prob‐
lem. We need the signal to flow properly in both directions: in the forward direction
when making predictions, and in the reverse direction when backpropagating gradi‐
ents. We don’t want the signal to die out, nor do we want it to explode and saturate.
For the signal to flow properly, the authors argue that we need the variance of the
outputs of each layer to be equal to the variance of its inputs,2 and we also need the

gradients to have equal variance before and after flowing through a layer in the
reverse direction (please check out the paper if you are interested in the mathematical
details). It is actually not possible to guarantee both unless the layer has an equal
number of inputs and neurons (these numbers are called the fan-in  and fan-out  of the
layer), but they proposed a good compromise that has proven to work very well in

practice: the connection weights of each layer must be initialized randomly asVanishing/Exploding Gradients Problems | 327

3Such as “Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification, ” K.
He et al. (2015).described in Equation 11-1 , where f anavg=f anin+f anout/2. This initialization
strategy is called Xavier initialization  (after the author’s first name) or Glorot initiali‐
zation  (after his last name).
Equation 11-1. Glorot initialization (when using the logistic activation function)Normal distribution with mean 0 and variance  σ2=1
fanavg
Or a uniform distribution between − r and  + r, with  r=3
fanavg
If you just replace fanavg with fanin in Equation 11-1 , you get an initialization strategy
that was actually already proposed by Y ann LeCun in the 1990s, called LeCun initiali‐
zation , which was even recommended in the 1998 book Neural Networks: Tricks of the
Trade  by Genevieve Orr and Klaus-Robert Müller (Springer). It is equivalent to
Glorot initialization when fanin = fanout. It took over a decade for researchers to realize

fanavg
fanavg
just how important this trick really is. Using Glorot initialization can speed up train‐
ing considerably, and it is one of the tricks that led to the current success of Deep
Learning.
Some papers3 have provided similar strategies for different activation functions.
These strategies differ only by the scale of the variance and whether they use fanavg or
fanin, as shown in Table 11-1  (for the uniform distribution, just compute r=3σ2
).

Learning.
).
The initialization strategy for the ReLU activation function (and its variants, includ‐
ing the ELU activation described shortly) is sometimes called He initialization  (after
the last name of its author). The SELU activation function will be explained later in
this chapter. It should be used with LeCun initialization (preferably with a normal

).
distribution, as we will see).Table 11-1. Initialization parameters for each type of activation functionInitializationActivation functionsσ² (Normal)GlorotNone, Tanh, Logistic, Softmax1 / fan avg HeReLU & variants2 / fan in LeCunSELU1 / fan in
By default, Keras uses Glorot initialization with a uniform distribution. Y ou can
change this to He initialization by setting kernel_initializer="he_uniform"  or ker
nel_initializer="he_normal"  when creating a layer, like this:328 | Chapter 11: Training Deep Neural Networks

4Unless it is part of the first hidden layer, a dead neuron may sometimes come back to life: gradient descent
may indeed tweak neurons in the layers below in such a way that the weighted sum of the dead neuron’s
inputs is positive again.5
“Empirical Evaluation of Rectified Activations in Convolution Network, ” B. Xu et al. (2015).keras.layers.Dense(10, activation ="relu", kernel_initializer ="he_normal" )
If you want He initialization with a uniform distribution, but based on fanavg rather
than fanin, you can use the VarianceScaling  initializer like this:
he_avg_init  = keras.initializers .VarianceScaling (scale=2., mode='fan_avg' ,
                                                 distribution ='uniform' )
keras.layers.Dense(10, activation ="sigmoid" , kernel_initializer =he_avg_init )
Nonsaturating Activation Functions
One of the insights in the 2010 paper by Glorot and Bengio was that the vanishing/
exploding gradients problems were in part due to a poor choice of activation func‐

tion. Until then most people had assumed that if Mother Nature had chosen to use
roughly sigmoid activation functions in biological neurons, they must be an excellent
choice. But it turns out that other activation functions behave much better in deep
neural networks, in particular the ReLU activation function, mostly because it does
not saturate for positive values (and also because it is quite fast to compute).
Unfortunately, the ReLU activation function is not perfect. It suffers from a problem

known as the dying ReLUs : during training, some neurons effectively die, meaning
they stop outputting anything other than 0. In some cases, you may find that half of
your network’s neurons are dead, especially if you used a large learning rate. A neu‐
ron dies when its weights get tweaked in such a way that the weighted sum of its
inputs are negative for all instances in the training set. When this happens, it just
keeps outputting 0s, and gradient descent does not affect it anymore since the gradi‐

ent of the ReLU function is 0 when its input is negative.4
To solve this problem, you may want to use a variant of the ReLU function, such as
the leaky ReLU . This function is defined as LeakyReLUα(z) = max( αz, z) (see
Figure 11-2 ). The hyperparameter α defines how much the function “leaks”: it is the
slope of the function for z < 0, and is typically set to 0.01. This small slope ensures
that leaky ReLUs never die; they can go into a long coma, but they have a chance to

eventually wake up. A 2015 paper5 compared several variants of the ReLU activation
function and one of its conclusions was that the leaky variants always outperformed
the strict ReLU activation function. In fact, setting α = 0.2 (huge leak) seemed to
result in better performance than α = 0.01 (small leak). They also evaluated the
randomized leaky ReLU  (RReLU), where α is picked randomly in a given range during
training, and it is fixed to an average value during testing. It also performed fairly well

and seemed to act as a regularizer (reducing the risk of overfitting the training set).Vanishing/Exploding Gradients Problems | 329

6“Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs), ” D. Clevert, T. Unterthiner,
S. Hochreiter (2015).Finally, they also evaluated the parametric leaky ReLU  (PReLU), where α is authorized
to be learned during training (instead of being a hyperparameter, it becomes a
parameter that can be modified by backpropagation like any other parameter). This
was reported to strongly outperform ReLU on large image datasets, but on smaller
datasets it runs the risk of overfitting the training set.
Figure 11-2. Leaky ReLU
Last but not least, a 2015 paper  by Djork-Arné Clevert et al.6 proposed a new activa‐
tion function called the exponential linear unit  (ELU) that outperformed all the ReLU
variants in their experiments: training time was reduced and the neural network per‐
formed better on the test set. It is represented in Figure 11-3 , and Equation 11-2
shows its definition.
Equation 11-2. ELU activation functionELUαz=αexp z− 1 ifz< 0
z ifz≥ 0330 | Chapter 11: Training Deep Neural Networks

7“Self-Normalizing Neural Networks, " G. Klambauer, T. Unterthiner and A. Mayr (2017).
Figure 11-3. ELU activation function
It looks a lot like the ReLU function, with a few major differences:•
First it takes on negative values when z < 0, which allows the unit to have an
average output closer to 0. This helps alleviate the vanishing gradients problem,
as discussed earlier. The hyperparameter α defines the value that the ELU func‐
tion approaches when z is a large negative number. It is usually set to 1, but you
can tweak it like any other hyperparameter if you want.•
Second, it has a nonzero gradient for z < 0, which avoids the dead neurons prob‐
lem.•
Third, if α is equal to 1 then the function is smooth everywhere, including
around z = 0, which helps speed up Gradient Descent, since it does not bounce as
much left and right of z = 0.
The main drawback of the ELU activation function is that it is slower to compute
than the ReLU and its variants (due to the use of the exponential function), but dur‐

lem.•
ing training this is compensated by the faster convergence rate. However, at test time
an ELU network will be slower than a ReLU network.
Moreover, in a 2017 paper7 by Günter Klambauer et al., called “Self-Normalizing
Neural Networks” , the authors showed that if you build a neural network composed
exclusively of a stack of dense layers, and if all hidden layers use the SELU activation
function (which is just a scaled version of the ELU activation function, as its name

suggests), then the network will self-normalize : the output of each layer will tend to
preserve mean 0 and standard deviation 1 during training, which solves the vanish‐
ing/exploding gradients problem. As a result, this activation function often outper‐Vanishing/Exploding Gradients Problems | 331

forms other activation functions very significantly for such neural nets (especially
deep ones). However, there are a few conditions for self-normalization to happen:•
The input features must be standardized (mean 0 and standard deviation 1).•
Every hidden layer’s weights must also be initialized using LeCun normal initiali‐
zation. In Keras, this means setting kernel_initializer="lecun_normal" .•
The network’s architecture must be sequential. Unfortunately, if you try to use
SELU in non-sequential architectures, such as recurrent networks (see ???) or
networks with skip connections  (i.e., connections that skip layers, such as in wide
& deep nets), self-normalization will not be guaranteed, so SELU will not neces‐
sarily outperform other activation functions.•
The paper only guarantees self-normalization if all layers are dense. However, in
practice the SELU activation function seems to work great with convolutional
neural nets as well (see Chapter 14 ).

So which activation function should you use for the hidden layers
of your deep neural networks? Although your mileage will vary, in
general SELU > ELU > leaky ReLU (and its variants) > ReLU > tanh
> logistic. If the network’s architecture prevents it from self-
normalizing, then ELU may perform better than SELU (since SELU
is not smooth at z = 0). If you care a lot about runtime latency, then
you may prefer leaky ReLU. If you don’t want to tweak yet another
hyperparameter, you may just use the default α values used by

Keras (e.g., 0.3 for the leaky ReLU). If you have spare time and
computing power, you can use cross-validation to evaluate other
activation functions, in particular RReLU if your network is over‐
fitting, or PReLU if you have a huge training set.
To use the leaky ReLU activation function, you must create a LeakyReLU  instance like
this:
leaky_relu  = keras.layers.LeakyReLU (alpha=0.2)
layer = keras.layers.Dense(10, activation =leaky_relu ,
                           kernel_initializer ="he_normal" )

this:
For PReLU, just replace LeakyRelu(alpha=0.2)  with PReLU() . There is currently no
official implementation of RReLU in Keras, but you can fairly easily implement your
own (see the exercises at the end of Chapter 12 ).
For SELU activation, just set activation="selu"  and kernel_initial
izer="lecun_normal"  when creating a layer:
layer = keras.layers.Dense(10, activation ="selu",
                           kernel_initializer ="lecun_normal" )332 | Chapter 11: Training Deep Neural Networks

8“Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, ” S. Ioffe
and C. Szegedy (2015).Batch Normalization
Although using He initialization along with ELU (or any variant of ReLU) can signifi‐
cantly reduce the vanishing/exploding gradients problems at the beginning of train‐
ing, it doesn’t guarantee that they won’t come back during training.
In a 2015 paper ,8 Sergey Ioffe and Christian Szegedy proposed a technique called
Batch Normalization  (BN) to address the vanishing/exploding gradients problems.
The technique consists of adding an operation in the model just before or after the
activation function of each hidden layer, simply zero-centering and normalizing each
input, then scaling and shifting the result using two new parameter vectors per layer:
one for scaling, the other for shifting. In other words, this operation lets the model
learn the optimal scale and mean of each of the layer’s inputs. In many cases, if you

add a BN layer as the very first layer of your neural network, you do not need to
standardize your training set (e.g., using a StandardScaler ): the BN layer will do it
for you (well, approximately, since it only looks at one batch at a time, and it can also
rescale and shift each input feature).
In order to zero-center and normalize the inputs, the algorithm needs to estimate
each input’s mean and standard deviation. It does so by evaluating the mean and stan‐

• σB is the vector of input standard deviations, also evaluated over the whole mini-
batch (it contains one standard deviation per input).•
mB is the number of instances in the mini-batch.•x(i) is the vector of zero-centered and normalized inputs for instance i.•
γ is the output scale parameter vector for the layer (it contains one scale parame‐
ter per input).•
⊗ represents element-wise multiplication (each input is multiplied by its corre‐
sponding output scale parameter).•
β is the output shift (offset) parameter vector for the layer (it contains one offset
parameter per input). Each input is offset by its corresponding shift parameter.•
ϵ is a tiny number to avoid division by zero (typically 10–5). This is called a
smoothing term .•
z(i) is the output of the BN operation: it is a rescaled and shifted version of the
inputs.
So during training, BN just standardizes its inputs then rescales and offsets them.
Good! What about at test time? Well it is not that simple. Indeed, we may need to

inputs.
make predictions for individual instances rather than for batches of instances: in this
case, we will have no way to compute each input’s mean and standard deviation.
Moreover, even if we do have a batch of instances, it may be too small, or the instan‐
ces may not be independent and identically distributed (IID), so computing statistics
over the batch instances would be unreliable (during training, the batches should not
be too small, if possible more than 30 instances, and all instances should be IID, as we

saw in Chapter 4 ). One solution could be to wait until the end of training, then run
the whole training set through the neural network, and compute the mean and stan‐
dard deviation of each input of the BN layer. These “final” input means and standard
deviations can then be used instead of the batch input means and standard deviations
when making predictions. However, it is often preferred to estimate these final statis‐
tics during training using a moving average of the layer’s input means and standard
deviations. To sum up, four parameter vectors are learned in each batch-normalized

layer: γ (the ouput scale vector) and β (the output offset vector) are learned through
regular backpropagation, and μ (the final input mean vector), and σ (the final input
standard deviation vector) are estimated using an exponential moving average. Note
that μ and σ are estimated during training, but they are not used at all during train‐
ing, only after training (to replace the batch input means and standard deviations in
Equation 11-3 ).

The authors demonstrated that this technique considerably improved all the deep
neural networks they experimented with, leading to a huge improvement in the
ImageNet classification task (ImageNet is a large database of images classified into
many classes and commonly used to evaluate computer vision systems). The vanish‐334 | Chapter 11: Training Deep Neural Networks

ing gradients problem was strongly reduced, to the point that they could use saturat‐
ing activation functions such as the tanh and even the logistic activation function.
The networks were also much less sensitive to the weight initialization. They were
able to use much larger learning rates, significantly speeding up the learning process.
Specifically, they note that “ Applied to a state-of-the-art image classification model,
Batch Normalization achieves the same accuracy with 14 times fewer training steps,
and beats the original model by a significant margin. […] Using an ensemble of
batch-normalized networks, we improve upon the best published result on ImageNet
classification: reaching 4.9% top-5 validation error (and 4.8% test error), exceeding
the accuracy of human raters. ” Finally, like a gift that keeps on giving, Batch Normal‐
ization also acts like a regularizer, reducing the need for other regularization techni‐
ques (such as dropout, described later in this chapter).

Batch Normalization does, however, add some complexity to the model (although it
can remove the need for normalizing the input data, as we discussed earlier). More‐
over, there is a runtime penalty: the neural network makes slower predictions due to
the extra computations required at each layer. So if you need predictions to be
lightning-fast, you may want to check how well plain ELU + He initialization perform
before playing with Batch Normalization.
Y ou may find that training is rather slow, because each epoch takes

much more time when you use batch normalization. However, this
is usually counterbalanced by the fact that convergence is much
faster with BN, so it will take fewer epochs to reach the same per‐
formance. All in all, wall time  will usually be smaller (this is the
time measured by the clock on your wall).
Implementing Batch Normalization with Keras
As with most things with Keras, implementing Batch Normalization is quite simple.
Just add a BatchNormalization  layer before or after each hidden layer’s activation

function, and optionally add a BN layer as well as the first layer in your model. For
example, this model applies BN after every hidden layer and as the first layer in the
model (after flattening the input images):
    keras.layers.BatchNormalization (),
    keras.layers.Dense(300, activation ="elu", kernel_initializer ="he_normal" ),
    keras.layers.Dense(100, activation ="elu", kernel_initializer ="he_normal" ),

])Vanishing/Exploding Gradients Problems | 335

9However, they are estimated during training, based on the training data, so arguably they are trainable. In
Keras, “Non-trainable” really means “untouched by backpropagation” .That’s all! In this tiny example with just two hidden layers, it’s unlikely that Batch
Normalization will have a very positive impact, but for deeper networks it can make a
tremendous difference.
Let’s zoom in a bit. If you display the model summary, you can see that each BN layer
adds 4 parameters per input: γ, β, μ and σ (for example, the first BN layer adds 3136
parameters, which is 4 times 784). The last two parameters, μ and σ, are the moving
averages, they are not affected by backpropagation, so Keras calls them “Non-
trainable”9 (if you count the total number of BN parameters, 3136 + 1200 + 400, and
divide by two, you get 2,368, which is the total number of non-trainable params in
this model).
Model: "sequential_3"

flatten_3 (Flatten)          (None, 784)               0
batch_normalization_v2 (Batc (None, 784)               3136
dense_50 (Dense)             (None, 300)               235500
batch_normalization_v2_1 (Ba (None, 300)               1200

dense_51 (Dense)             (None, 100)               30100
batch_normalization_v2_2 (Ba (None, 100)               400
dense_52 (Dense)             (None, 10)                1010
Total params: 271,346
Trainable params: 268,978
Non-trainable params: 2,368

Let’s look at the parameters of the first BN layer. Two are trainable (by backprop), and
two are not:
>>> [(var.name, var.trainable ) for var in model.layers[1].variables ]
[('batch_normalization_v2/gamma:0', True),
 ('batch_normalization_v2/beta:0', True),
 ('batch_normalization_v2/moving_mean:0', False),
 ('batch_normalization_v2/moving_variance:0', False)]
Now when you create a BN layer in Keras, it also creates two operations that will be

called by Keras at each iteration during training. These operations will update the336 | Chapter 11: Training Deep Neural Networks

moving averages. Since we are using the TensorFlow backend, these operations are
TensorFlow operations (we will discuss TF operations in Chapter 12 ).
>>> model.layers[1].updates
[<tf.Operation 'cond_2/Identity' type=Identity>,
 <tf.Operation 'cond_3/Identity' type=Identity>]
The authors of the BN paper argued in favor of adding the BN layers before the acti‐
vation functions, rather than after (as we just did). There is some debate about this, as
it seems to depend on the task. So that’s one more thing you can experiment with to
see which option works best on your dataset. To add the BN layers before the activa‐
tion functions, we must remove the activation function from the hidden layers, and
add them as separate layers after the BN layers. Moreover, since a Batch Normaliza‐
tion layer includes one offset parameter per input, you can remove the bias term from
the previous layer (just pass use_bias=False  when creating it):

    keras.layers.Dense(300, kernel_initializer ="he_normal" , use_bias =False),
    keras.layers.Activation ("elu"),
    keras.layers.Dense(100, kernel_initializer ="he_normal" , use_bias =False),
])
The BatchNormalization  class has quite a few hyperparameters you can tweak. The

malized. It defaults to –1, meaning that by default it will normalize the last axis (using
the means and standard deviations computed across the other  axes). For example,
when the input batch is 2D (i.e., the batch shape is [batch size, features]), this means
that each input feature will be normalized based on the mean and standard deviation
computed across all the instances in the batch. For example, the first BN layer in the
previous code example will independently normalize (and rescale and shift) each of
the 784 input features. However, if we move the first BN layer before the FlattenVanishing/Exploding Gradients Problems | 337

10“Fixup Initialization: Residual Learning Without Normalization, ” Hongyi Zhang, Y ann N. Dauphin, Tengyu
Ma (2019).11
“On the difficulty of training recurrent neural networks, ” R. Pascanu et al. (2013).layer, then the input batches will be 3D, with shape [batch size, height, width], there‐
fore the BN layer will compute 28 means and 28 standard deviations (one per column
of pixels, computed across all instances in the batch, and all rows in the column), and
it will normalize all pixels in a given column using the same mean and standard devi‐
ation. There will also be just 28 scale parameters and 28 shift parameters. If instead
you still want to treat each of the 784 pixels independently, then you should set
axis=[1, 2] .
Notice that the BN layer does not perform the same computation during training and
after training: it uses batch statistics during training, and the “final” statistics after
training (i.e., the final value of the moving averages). Let’s take a peek at the source

code of this class to see how this is handled:
class BatchNormalization (Layer):
    [...]
    def call(self, inputs, training =None):
        if training  is None:
            training  = keras.backend.learning_phase ()
        [...]
The call()  method is the one that actually performs the computations, and as you
can see it has an extra training  argument: if it is None  it falls back to keras.back
end.learning_phase() , which returns 1 during training (the fit()  method ensures
that). Otherwise, it returns 0. If you ever need to write a custom layer, and it needs to

[...]
        [...]
behave differently during training and testing, simply use the same pattern (we will
discuss custom layers in Chapter 12 ).
Batch Normalization has become one of the most used layers in deep neural net‐
works, to the point that it is often omitted in the diagrams, as it is assumed that BN is
added after every layer. However, a very recent paper10 by Hongyi Zhang et al. may
well change this: the authors show that by using a novel fixed-update (fixup) weight

initialization technique, they manage to train a very deep neural network (10,000 lay‐
ers!) without BN, achieving state-of-the-art performance on complex image classifi‐
cation tasks.
Gradient Clipping
Another popular technique to lessen the exploding gradients problem is to simply
clip the gradients during backpropagation so that they never exceed some threshold.
This is called Gradient Clipping .11 This technique is most often used in recurrent neu‐338 | Chapter 11: Training Deep Neural Networks

ral networks, as Batch Normalization is tricky to use in RNNs, as we will see in ???.
For other types of networks, BN is usually sufficient.
In Keras, implementing Gradient Clipping is just a matter of setting the clipvalue  or
clipnorm  argument when creating an optimizer. For example:
optimizer  = keras.optimizers .SGD(clipvalue =1.0)
This will clip every component of the gradient vector to a value between –1.0 and 1.0.
This means that all the partial derivatives of the loss (with regards to each and every
trainable parameter) will be clipped between –1.0 and 1.0. The threshold is a hyper‐
parameter you can tune. Note that it may change the orientation of the gradient vec‐
tor: for example, if the original gradient vector is [0.9, 100.0], it points mostly in the
direction of the second axis, but once you clip it by value, you get [0.9, 1.0], which
points roughly in the diagonal between the two axes. In practice however, this

approach works well. If you want to ensure that Gradient Clipping does not change
the direction of the gradient vector, you should clip by norm by setting clipnorm
instead of clipvalue . This will clip the whole gradient if its ℓ2 norm is greater than
the threshold you picked. For example, if you set clipnorm=1.0 , then the vector [0.9,
100.0] will be clipped to [0.00899964, 0.9999595], preserving its orientation, but
almost eliminating the first component. If you observe that the gradients explode
during training (you can track the size of the gradients using TensorBoard), you may

want to try both clipping by value and clipping by norm, with different threshold,
and see which option performs best on the validation set.
Reusing Pretrained Layers
It is generally not a good idea to train a very large DNN from scratch: instead, you
should always try to find an existing neural network that accomplishes a similar task
to the one you are trying to tackle (we will discuss how to find them in Chapter 14 ),

then just reuse the lower layers of this network: this is called transfer learning . It will
not only speed up training considerably, but will also require much less training data.
For example, suppose that you have access to a DNN that was trained to classify pic‐
tures into 100 different categories, including animals, plants, vehicles, and everyday
objects. Y ou now want to train a DNN to classify specific types of vehicles. These

tasks are very similar, even partly overlapping, so you should try to reuse parts of the
first network (see Figure 11-4 ).Reusing Pretrained Layers | 339

Figure 11-4. Reusing pretrained layers
If the input pictures of your new task don’t have the same size as
the ones used in the original task, you will usually have to add a
preprocessing step to resize them to the size expected by the origi‐
nal model. More generally, transfer learning will work best when
the inputs have similar low-level features.
The output layer of the original model should usually be replaced since it is most
likely not useful at all for the new task, and it may not even have the right number of
outputs for the new task.
Similarly, the upper hidden layers of the original model are less likely to be as useful
as the lower layers, since the high-level features that are most useful for the new task
may differ significantly from the ones that were most useful for the original task. Y ou
want to find the right number of layers to reuse.
The more similar the tasks are, the more layers you want to reuse
(starting with the lower layers). For very similar tasks, you can try

keeping all the hidden layers and just replace the output layer.
Try freezing all the reused layers first (i.e., make their weights non-trainable, so gradi‐
ent descent won’t modify them), then train your model and see how it performs.
Then try unfreezing one or two of the top hidden layers to let backpropagation tweak
them and see if performance improves. The more training data you have, the more340 | Chapter 11: Training Deep Neural Networks

layers you can unfreeze. It is also useful to reduce the learning rate when you unfreeze
reused layers: this will avoid wrecking their fine-tuned weights.
If you still cannot get good performance, and you have little training data, try drop‐
ping the top hidden layer(s) and freeze all remaining hidden layers again. Y ou can
iterate until you find the right number of layers to reuse. If you have plenty of train‐
ing data, you may try replacing the top hidden layers instead of dropping them, and
even add more hidden layers.
Transfer Learning With Keras
Let’s look at an example. Suppose the fashion MNIST dataset only contained 8 classes,
for example all classes except for sandals and shirts. Someone built and trained a
Keras model on that set and got reasonably good performance (>90% accuracy). Let’s
call this model A. Y ou now want to tackle a different task: you have images of sandals
and shirts, and you want to train a binary classifier (positive=shirts, negative=san‐

dals). However, your dataset is quite small, you only have 200 labeled images. When
you train a new model for this task (let’s call it model B), with the same architecture
as model A, it performs reasonably well (97.2% accuracy), but since it’s a much easier
task (there are just 2 classes), you were hoping for more. While drinking your morn‐
ing coffee, you realize that your task is quite similar to task A, so perhaps transfer
learning can help? Let’s find out!

First, you need to load model A, and create a new model based on the model A ’s lay‐
ers. Let’s reuse all layers except for the output layer:
model_A = keras.models.load_model ("my_model_A.h5" )
model_B_on_A  = keras.models.Sequential (model_A.layers[:-1])
model_B_on_A .add(keras.layers.Dense(1, activation ="sigmoid" ))
Note that model_A  and model_B_on_A  now share some layers. When you train
model_B_on_A , it will also affect model_A . If you want to avoid that, you need to clone

model_A  before you reuse its layers. To do this, you must clone model A ’s architecture,
then copy its weights (since clone_model()  does not clone the weights):
model_A_clone  = keras.models.clone_model (model_A)
model_A_clone .set_weights (model_A.get_weights ())
Now we could just train model_B_on_A  for task B, but since the new output layer was
initialized randomly, it will make large errors, at least during the first few epochs, so
there will be large error gradients that may wreck the reused weights. To avoid this,

one approach is to freeze the reused layers during the first few epochs, giving the new
layer some time to learn reasonable weights. To do this, simply set every layer’s train
able  attribute to False  and compile the model:
for layer in model_B_on_A .layers[:-1]:
    layer.trainable  = FalseReusing Pretrained Layers | 341

model_B_on_A .compile(loss="binary_crossentropy" , optimizer ="sgd",
                     metrics=["accuracy" ])
Y ou must always compile your model after you freeze or unfreeze
layers.
Next, we can train the model for a few epochs, then unfreeze the reused layers (which
requires compiling the model again) and continue training to fine-tune the reused
layers for task B. After unfreezing the reused layers, it is usually a good idea to reduce
the learning rate, once again to avoid damaging the reused weights:
history = model_B_on_A .fit(X_train_B , y_train_B , epochs=4,
                           validation_data =(X_valid_B , y_valid_B ))
    layer.trainable  = True
optimizer  = keras.optimizers .SGD(lr=1e-4) # the default lr is 1e-3
model_B_on_A .compile(loss="binary_crossentropy" , optimizer =optimizer ,
history = model_B_on_A .fit(X_train_B , y_train_B , epochs=16,

So, what’s the final verdict? Well this model’s test accuracy is 99.25%, which means
that transfer learning reduced the error rate from 2.8% down to almost 0.7%! That’s a
factor of 4!
>>> model_B_on_A .evaluate (X_test_B , y_test_B )
[0.06887910133600235, 0.9925]
Are you convinced? Well you shouldn’t be: I cheated! :) I tried many configurations
until I found one that demonstrated a strong improvement. If you try to change the

validation_data =(X_valid_B , y_valid_B ))
classes or the random seed, you will see that the improvement generally drops, or
even vanishes or reverses. What I did is called “torturing the data until it confesses” .
When a paper just looks too positive, you should be suspicious: perhaps the flashy
new technique does not help much (in fact, it may even degrade performance), but
the authors tried many variants and reported only the best results (which may be due
to shear luck), without mentioning how many failures they encountered on the way.

Most of the time, this is not malicious at all, but it is part of the reason why so many
results in Science can never be reproduced.
So why did I cheat? Well it turns out that transfer learning does not work very well
with small dense networks: it works best with deep convolutional neural networks, so
we will revisit transfer learning in Chapter 14 , using the same techniques (and this
time there will be no cheating, I promise!).342 | Chapter 11: Training Deep Neural Networks

Unsupervised Pretraining
Suppose you want to tackle a complex task for which you don’t have much labeled
training data, but unfortunately you cannot find a model trained on a similar task.
Don’t lose all hope! First, you should of course try to gather more labeled training
data, but if this is too hard or too expensive, you may still be able to perform unsuper‐
vised pretraining  (see Figure 11-5 ). It is often rather cheap to gather unlabeled train‐
ing examples, but quite expensive to label them. If you can gather plenty of unlabeled
training data, you can try to train the layers one by one, starting with the lowest layer
and then going up, using an unsupervised feature detector algorithm such as Restric‐
ted Boltzmann Machines  (RBMs; see ???) or autoencoders (see ???). Each layer is
trained on the output of the previously trained layers (all layers except the one being
trained are frozen). Once all layers have been trained this way, you can add the output

layer for your task, and fine-tune the final network using supervised learning (i.e.,
with the labeled training examples). At this point, you can unfreeze all the pretrained
layers, or just some of the upper ones.
Figure 11-5. Unsupervised pretraining
This is a rather long and tedious process, but it often works well; in fact, it is this
technique that Geoffrey Hinton and his team used in 2006 and which led to the
revival of neural networks and the success of Deep Learning. Until 2010, unsuper‐
vised pretraining (typically using RBMs) was the norm for deep nets, and it was only

after the vanishing gradients problem was alleviated that it became much more com‐Reusing Pretrained Layers | 343

mon to train DNNs purely using supervised learning. However, unsupervised pre‐
training (today typically using autoencoders rather than RBMs) is still a good option
when you have a complex task to solve, no similar model you can reuse, and little
labeled training data but plenty of unlabeled training data.
Pretraining on an Auxiliary Task
If you do not have much labeled training data, one last option is to train a first neural
network on an auxiliary task for which you can easily obtain or generate labeled
training data, then reuse the lower layers of that network for your actual task. The
first neural network’s lower layers will learn feature detectors that will likely be reusa‐
ble by the second neural network.
For example, if you want to build a system to recognize faces, you may only have a
few pictures of each individual—clearly not enough to train a good classifier. Gather‐
ing hundreds of pictures of each person would not be practical. However, you could

gather a lot of pictures of random people on the web and train a first neural network
to detect whether or not two different pictures feature the same person. Such a net‐
work would learn good feature detectors for faces, so reusing its lower layers would
allow you to train a good face classifier using little training data.
For natural language processing  (NLP) applications, you can easily download millions
of text documents and automatically generate labeled data from it. For example, you

could randomly mask out some words and train a model to predict what the missing
words are (e.g., it should predict that the missing word in the sentence “What ___
you saying?” is probably “are” or “were”). If you can train a model to reach good per‐
formance on this task, then it will already know quite a lot about language, and you
can certainly reuse it for your actual task, and fine-tune it on your labeled data (we
will discuss more pretraining tasks in ???).

Self-supervised learning  is when you automatically generate the
labels from the data itself, then you train a model on the resulting
“labeled” dataset using supervised learning techniques. Since this
approach requires no human labeling whatsoever, it is best classi‐
fied as a form of unsupervised learning.
Faster Optimizers
Training a very large deep neural network can be painfully slow. So far we have seen
four ways to speed up training (and reach a better solution): applying a good initiali‐

zation strategy for the connection weights, using a good activation function, using
Batch Normalization, and reusing parts of a pretrained network (possibly built on an
auxiliary task or using unsupervised learning). Another huge speed boost comes from
using a faster optimizer than the regular Gradient Descent optimizer. In this section344 | Chapter 11: Training Deep Neural Networks

12“Some methods of speeding up the convergence of iteration methods, ” B. Polyak (1964).we will present the most popular ones: Momentum optimization, Nesterov Acceler‐
ated Gradient, AdaGrad, RMSProp, and finally Adam and Nadam optimization.
Momentum Optimization
Imagine a bowling ball rolling down a gentle slope on a smooth surface: it will start
out slowly, but it will quickly pick up momentum until it eventually reaches terminal
velocity (if there is some friction or air resistance). This is the very simple idea behind
Momentum optimization , proposed by Boris Polyak in 1964 .12 In contrast, regular
Gradient Descent will simply take small regular steps down the slope, so it will take
much more time to reach the bottom.
Recall that Gradient Descent simply updates the weights θ by directly subtracting the
gradient of the cost function J(θ) with regards to the weights ( ∇θJ(θ)) multiplied by
the learning rate η. The equation is: θ ← θ – η∇θJ(θ). It does not care about what the

earlier gradients were. If the local gradient is tiny, it goes very slowly.
Momentum optimization cares a great deal about what previous gradients were: at
each iteration, it subtracts the local gradient from the momentum vector  m (multi‐
plied by the learning rate η), and it updates the weights by simply adding this
momentum vector (see Equation 11-4 ). In other words, the gradient is used for accel‐
eration, not for speed. To simulate some sort of friction mechanism and prevent the

momentum from growing too large, the algorithm introduces a new hyperparameter
β, simply called the momentum , which must be set between 0 (high friction) and 1
(no friction). A typical momentum value is 0.9.
Equation 11-4. Momentum algorithm1 . m βm−η∇θJθ
2 . θ θ+m
Y ou can easily verify that if the gradient remains constant, the terminal velocity (i.e.,
the maximum size of the weight updates) is equal to that gradient multiplied by the
learning rate η multiplied by 1
1 −β (ignoring the sign). For example, if β = 0.9, then the

2 . θ θ+m
terminal velocity is equal to 10 times the gradient times the learning rate, so Momen‐
tum optimization ends up going 10 times faster than Gradient Descent! This allows
Momentum optimization to escape from plateaus much faster than Gradient Descent.
In particular, we saw in Chapter 4  that when the inputs have very different scales the 
cost function will look like an elongated bowl (see Figure 4-7 ). Gradient Descent goes
down the steep slope quite fast, but then it takes a very long time to go down the val‐Faster Optimizers | 345

13“ A Method for Unconstrained Convex Minimization Problem with the Rate of Convergence O(1/k2), ” Yurii
Nesterov (1983).
ley. In contrast, Momentum optimization will roll down the valley faster and faster
until it reaches the bottom (the optimum). In deep neural networks that don’t use
Batch Normalization, the upper layers will often end up having inputs with very dif‐
ferent scales, so using Momentum optimization helps a lot. It can also help roll past
local optima.
Due to the momentum, the optimizer may overshoot a bit, then
come back, overshoot again, and oscillate like this many times
before stabilizing at the minimum. This is one of the reasons why it
is good to have a bit of friction in the system: it gets rid of these
oscillations and thus speeds up convergence.
Implementing Momentum optimization in Keras is a no-brainer: just use the SGD
optimizer and set its momentum  hyperparameter, then lie back and profit!
optimizer  = keras.optimizers .SGD(lr=0.001, momentum =0.9)

The one drawback of Momentum optimization is that it adds yet another hyperpara‐
meter to tune. However, the momentum value of 0.9 usually works well in practice
and almost always goes faster than regular Gradient Descent.
Nesterov Accelerated Gradient
One small variant to Momentum optimization, proposed by Yurii Nesterov in 1983 ,13
is almost always faster than vanilla Momentum optimization. The idea of Nesterov
Momentum optimization , or Nesterov Accelerated Gradient  (NAG), is to measure the

gradient of the cost function not at the local position but slightly ahead in the direc‐
tion of the momentum (see Equation 11-5 ). The only difference from vanilla
Momentum optimization is that the gradient is measured at θ + βm rather than at θ.
Equation 11-5. Nesterov Accelerated Gradient algorithm1 . m βm−η∇θJθ+βm
2 . θ θ+m
This small tweak works because in general the momentum vector will be pointing in
the right direction (i.e., toward the optimum), so it will be slightly more accurate to

2 . θ θ+m
use the gradient measured a bit farther in that direction rather than using the gradi‐
ent at the original position, as you can see in Figure 11-6  (where ∇1 represents the
gradient of the cost function measured at the starting point θ, and ∇2 represents the346 | Chapter 11: Training Deep Neural Networks

14“ Adaptive Subgradient Methods for Online Learning and Stochastic Optimization, ” J. Duchi et al. (2011).gradient at the point located at θ + βm). As you can see, the Nesterov update ends up
slightly closer to the optimum. After a while, these small improvements add up and
NAG ends up being significantly faster than regular Momentum optimization. More‐
over, note that when the momentum pushes the weights across a valley, ∇1 continues
to push further across the valley, while ∇2 pushes back toward the bottom of the val‐
ley. This helps reduce oscillations and thus converges faster.
NAG will almost always speed up training compared to regular Momentum optimi‐
zation. To use it, simply set nesterov=True  when creating the SGD optimizer:
optimizer  = keras.optimizers .SGD(lr=0.001, momentum =0.9, nesterov =True)
Figure 11-6. Regular versus Nesterov Momentum optimization
AdaGrad
Consider the elongated bowl problem again: Gradient Descent starts by quickly going

sions than for dimensions with gentler slopes. This is called an adaptive learning rate . 
It helps point the resulting updates more directly toward the global optimum (see
Figure 11-7 ). One additional benefit is that it requires much less tuning of the learn‐
ing rate hyperparameter η.
Figure 11-7. AdaGrad versus Gradient Descent
AdaGrad often performs well for simple quadratic problems, but unfortunately it
often stops too early when training neural networks. The learning rate gets scaled
down so much that the algorithm ends up stopping entirely before reaching the
global optimum. So even though Keras has an Adagrad  optimizer, you should not use
it to train deep neural networks (it may be efficient for simpler tasks such as Linear
Regression, though). However, understanding Adagrad is helpful to grasp the other
adaptive learning rate optimizers.348 | Chapter 11: Training Deep Neural Networks

15This algorithm was created by Geoffrey Hinton and Tijmen Tieleman in 2012, and presented by Geoffrey
Hinton in his Coursera class on neural networks (slides: https://homl.info/57 ; video: https://homl.info/58 ).
Amusingly, since the authors did not write a paper to describe it, researchers often cite “slide 29 in lecture 6”
in their papers.16
“ Adam: A Method for Stochastic Optimization, ” D. Kingma, J. Ba (2015).17
These are estimations of the mean and (uncentered) variance of the gradients. The mean is often called the
first moment , while the variance is often called the second moment , hence the name of the algorithm.RMSProp
Although AdaGrad slows down a bit too fast and ends up never converging to the
global optimum, the RMSProp  algorithm15 fixes this by accumulating only the gradi‐
ents from the most recent iterations (as opposed to all the gradients since the begin‐
ning of training). It does so by using exponential decay in the first step (see Equation
11-7 ).

1 – β1 times the decaying sum). Steps 3 and 4 are somewhat of a technical detail: since
m and s are initialized at 0, they will be biased toward 0 at the beginning of training,
so these two steps will help boost m and s at the beginning of training.
The momentum decay hyperparameter β1 is typically initialized to 0.9, while the scal‐
ing decay hyperparameter β2 is often initialized to 0.999. As earlier, the smoothing
term ϵ is usually initialized to a tiny number such as 10–7. These are the default values
for the Adam  class (to be precise, epsilon  defaults to None , which tells Keras to use
keras.backend.epsilon() , which defaults to 10–7; you can change it using
keras.backend.set_epsilon() ).
optimizer  = keras.optimizers .Adam(lr=0.001, beta_1=0.9, beta_2=0.999)
Since Adam is an adaptive learning rate algorithm (like AdaGrad and RMSProp), it
requires less tuning of the learning rate hyperparameter η. Y ou can often use the
default value η = 0.001, making Adam even easier to use than Gradient Descent.

If you are starting to feel overwhelmed by all these different techni‐
ques, and wondering how to choose the right ones for your task,
don’t worry: some practical guidelines are provided at the end of
this chapter.
Finally, two variants of Adam are worth mentioning:350 | Chapter 11: Training Deep Neural Networks

on the dataset, and in general Adam actually performs better. So it’s just one
more optimizer you can try if you experience problems with Adam on some task.•
Nadam optimization18 is more important: it is simply Adam optimization plus
the Nesterov trick, so it will often converge slightly faster than Adam. In his
report, Timothy Dozat compares many different optimizers on various tasks, and
finds that Nadam generally outperforms Adam, but is sometimes outperformed
by RMSProp.
Adaptive optimization methods (including RMSProp, Adam and
Nadam optimization) are often great, converging fast to a good sol‐
ution. However, a 2017 paper19 by Ashia C. Wilson et al. showed
that they can lead to solutions that generalize poorly on some data‐
sets. So when you are disappointed by your model’s performance,
try using plain Nesterov Accelerated Gradient instead: your dataset
may just be allergic to adaptive gradients. Also check out the latest
research, it is moving fast (e.g., AdaBound).

All the optimization techniques discussed so far only rely on the first-order  partial
derivatives  (Jacobians ). The optimization literature contains amazing algorithms
based on the second-order partial derivatives  (the Hessians , which are the partial
derivatives of the Jacobians). Unfortunately, these algorithms are very hard to apply
to deep neural networks because there are n2 Hessians per output (where n is the
number of parameters), as opposed to just n Jacobians per output. Since DNNs typi‐

cally have tens of thousands of parameters, the second-order optimization algorithmsFaster Optimizers | 351

20“Primal-Dual Subgradient Methods for Convex Problems, ” Yurii Nesterov (2005).21
“ Ad Click Prediction: a View from the Trenches, ” H. McMahan et al. (2013).often don’t even fit in memory, and even when they do, computing the Hessians is 
just too slow.
Training Sparse Models
All the optimization algorithms just presented produce dense models, meaning that
most parameters will be nonzero. If you need a blazingly fast model at runtime, or if
you need it to take up less memory, you may prefer to end up with a sparse model
instead.
One trivial way to achieve this is to train the model as usual, then get rid of the tiny
weights (set them to 0). However, this will typically not lead to a very sparse model,
and it may degrade the model’s performance.
A better option is to apply strong ℓ1 regularization during training, as it pushes the
optimizer to zero out as many weights as it can (as discussed in Chapter 4  about Lasso
Regression).

instead.
However, in some cases these techniques may remain insufficient. One last option is
to apply Dual Averaging , often called Follow The Regularized Leader  (FTRL), a techni‐
que proposed by Yurii Nesterov .20 When used with ℓ1 regularization, this technique
often leads to very sparse models. Keras implements a variant of FTRL called FTRL-
Proximal21 in the FTRL  optimizer.
Learning Rate Scheduling
Finding a good learning rate can be tricky. If you set it way too high, training may

actually diverge (as we discussed in Chapter 4 ). If you set it too low, training will
eventually converge to the optimum, but it will take a very long time. If you set it
slightly too high, it will make progress very quickly at first, but it will end up dancing
around the optimum, never really settling down. If you have a limited computing
budget, you may have to interrupt training before it has converged properly, yielding
a suboptimal solution (see Figure 11-8 ).352 | Chapter 11: Training Deep Neural Networks

Figure 11-8. Learning curves for various learning rates η
As we discussed in Chapter 10 , one approach is to start with a large learning rate, and
divide it by 3 until the training algorithm stops diverging. Y ou will not be too far
from the optimal learning rate, which will learn quickly and converge to good solu‐
tion.
However, you can do better than a constant learning rate: if you start with a high
learning rate and then reduce it once it stops making fast progress, you can reach a
good solution faster than with the optimal constant learning rate. There are many dif‐
ferent strategies to reduce the learning rate during training. These strategies are called
learning schedules  (we briefly introduced this concept in Chapter 4 ), the most com‐
mon of which are:
Power scheduling
Set the learning rate to a function of the iteration number t: η(t) = η0 / (1 + t/k)c.
The initial learning rate η0, the power c (typically set to 1) and the steps s are

hyperparameters. The learning rate drops at each step, and after s steps it is down
to η0 / 2. After s more steps, it is down to η0 / 3. Then down to η0 / 4, then η0 / 5,
and so on. As you can see, this schedule first drops quickly, then more and more
slowly. Of course, this requires tuning η0, s (and possibly c).
Exponential scheduling
Set the learning rate to: η(t) = η0 0.1t/s. The learning rate will gradually drop by a
factor of 10 every s steps. While power scheduling reduces the learning rate more

and more slowly, exponential scheduling keeps slashing it by a factor of 10 every
s steps.
Piecewise constant scheduling
Use a constant learning rate for a number of epochs (e.g., η0 = 0.1 for 5 epochs),
then a smaller learning rate for another number of epochs (e.g., η1 = 0.001 for 50
epochs), and so on. Although this solution can work very well, it requires fid‐Faster Optimizers | 353

22“ An Empirical Study of Learning Rates in Deep Neural Networks for Speech Recognition, ” A. Senior et al.
(2013).dling around to figure out the right sequence of learning rates, and how long to
use each of them.
Performance scheduling
Measure the validation error every N steps (just like for early stopping) and
reduce the learning rate by a factor of λ when the error stops dropping.
A 2013 paper22 by Andrew Senior et al. compared the performance of some of the
most popular learning schedules when training deep neural networks for speech rec‐
ognition using Momentum optimization. The authors concluded that, in this setting,
both performance scheduling and exponential scheduling performed well. They
favored exponential scheduling because it was easy to tune and it converged slightly
faster to the optimal solution (they also mentioned that it was easier to implement
than performance scheduling, but in Keras both options are easy).
Implementing power scheduling in Keras is the easiest option: just set the decay

hyperparameter when creating an optimizer. The decay  is the inverse of s (the num‐
ber of steps it takes to divide the learning rate by one more unit), and Keras assumes
that c is equal to 1. For example:
optimizer  = keras.optimizers .SGD(lr=0.01, decay=1e-4)
Exponential scheduling and piecewise scheduling are quite simple too. Y ou first need
to define a function that takes the current epoch and returns the learning rate. For
example, let’s implement exponential scheduling:
def exponential_decay_fn (epoch):

    return 0.01 * 0.1**(epoch / 20)
If you do not want to hard-code η0 and s, you can create a function that returns a
configured function:
def exponential_decay (lr0, s):
    def exponential_decay_fn (epoch):
        return lr0 * 0.1**(epoch / s)
    return exponential_decay_fn
exponential_decay_fn  = exponential_decay (lr0=0.01, s=20)
Next, just create a LearningRateScheduler  callback, giving it the schedule function,
and pass this callback to the fit()  method:
lr_scheduler  = keras.callbacks .LearningRateScheduler (exponential_decay_fn )

history = model.fit(X_train_scaled , y_train, [...], callbacks =[lr_scheduler ])354 | Chapter 11: Training Deep Neural Networks

The LearningRateScheduler  will update the optimizer’s learning_rate  attribute at
the beginning of each epoch. Updating the learning rate just once per epoch is usually
enough, but if you want it to be updated more often, for example at every step, you
need to write your own callback (see the notebook for an example). This can make
sense if there are many steps per epoch.
The schedule function can optionally take the current learning rate as a second argu‐
ment. For example, the following schedule function just multiplies the previous
learning rate by 0.1&1/20, which results in the same exponential decay (except the decay
now starts at the beginning of epoch 0 instead of 1). This implementation relies on
the optimizer’s initial learning rate (contrary to the previous implementation), so
make sure to set it appropriately.
def exponential_decay_fn (epoch, lr):
    return lr * 0.1**(1 / 20)
When you save a model, the optimizer and its learning rate get saved along with it.

This means that with this new schedule function, you could just load a trained model
and continue training where it left off, no problem. However, things are not so simple
if your schedule function uses the epoch  argument: indeed, the epoch does not get
saved, and it gets reset to 0 every time you call the fit()  method. This could lead to a
very large learning rate when you continue training a model where it left off, which
would likely damage your model’s weights. One solution is to manually set the fit()

method’s initial_epoch  argument so the epoch  starts at the right value.
For piecewise constant scheduling, you can use a schedule function like the following
one (as earlier, you can define a more general function if you want, see the notebook
for an example), then create a LearningRateScheduler  callback with this function
and pass it to the fit()  method, just like we did for exponential scheduling:
def piecewise_constant_fn (epoch):
    if epoch < 5:
        return 0.01
    elif epoch < 15:
        return 0.005
    else:
        return 0.001

    else:
For performance scheduling, simply use the ReduceLROnPlateau  callback. For exam‐
ple, if you pass the following callback to the fit()  method, it will multiply the learn‐
ing rate by 0.5 whenever the best validation loss does not improve for 5 consecutive
epochs (other options are available, please check the documentation for more
details):
lr_scheduler  = keras.callbacks .ReduceLROnPlateau (factor=0.5, patience =5)
Lastly, tf.keras offers an alternative way to implement learning rate scheduling: just

details):
define the learning rate using one of the schedules available in keras.optimizFaster Optimizers | 355

ers.schedules , then pass this learning rate to any optimizer. This approach updates
the learning rate at each step rather than at each epoch. For example, here is how to
implement the same exponential schedule as earlier:
s = 20 * len(X_train) // 32 # number of steps in 20 epochs (batch size = 32)
learning_rate  = keras.optimizers .schedules .ExponentialDecay (0.01, s, 0.1)
optimizer  = keras.optimizers .SGD(learning_rate )
This is nice and simple, plus when you save the model, the learning rate and its
schedule (including its state) get saved as well. However, this approach is not part of
the Keras API, it is specific to tf.keras.
To sum up, exponential decay or performance scheduling can considerably speed up
convergence, so give them a try!
Avoiding Overfitting  Through Regularization
With four parameters I can fit an elephant and with five I can make him wiggle his
trunk.
—John von Neumann, cited by Enrico Fermi in Nature 427

trunk.
With thousands of parameters you can fit the whole zoo. Deep neural networks typi‐
cally have tens of thousands of parameters, sometimes even millions. With so many
parameters, the network has an incredible amount of freedom and can fit a huge vari‐
ety of complex datasets. But this great flexibility also means that it is prone to overfit‐
ting the training set. We need regularization.
We already implemented one of the best regularization techniques in Chapter 10 :
early stopping. Moreover, even though Batch Normalization was designed to solve

the vanishing/exploding gradients problems, is also acts like a pretty good regularizer.
In this section we will present other popular regularization techniques for neural net‐
works: ℓ1 and ℓ2 regularization, dropout and max-norm regularization.
ℓ1 and ℓ2 Regularization
Just like you did in Chapter 4  for simple linear models, you can use ℓ1 and ℓ2 regulari‐
zation to constrain a neural network’s connection weights (but typically not its bia‐
ses). Here is how to apply ℓ2 regularization to a Keras layer’s connection weights,

using a regularization factor of 0.01:
layer = keras.layers.Dense(100, activation ="elu",
                           kernel_initializer ="he_normal" ,
                           kernel_regularizer =keras.regularizers .l2(0.01))
The l2()  function returns a regularizer that will be called to compute the regulariza‐
tion loss, at each step during training. This regularization loss is then added to the
final loss. As you might expect, you can just use keras.regularizers.l1()  if you356 | Chapter 11: Training Deep Neural Networks

23“Improving neural networks by preventing co-adaptation of feature detectors, ” G. Hinton et al. (2012).24
“Dropout: A Simple Way to Prevent Neural Networks from Overfitting, ” N. Srivastava et al. (2014).want ℓ1 regularization, and if you want both ℓ1 and ℓ2 regularization, use keras.regu
larizers.l1_l2()  (specifying both regularization factors).
Since you will typically want to apply the same regularizer to all layers in your net‐
work, as well as the same activation function and the same initialization strategy in all
hidden layers, you may find yourself repeating the same arguments over and over.
This makes it ugly and error-prone. To avoid this, you can try refactoring your code
to use loops. Another option is to use Python’s functools.partial()  function: it lets
you create a thin wrapper for any callable, with some default argument values. For
example:
from functools  import partial
RegularizedDense  = partial(keras.layers.Dense,
                           activation ="elu",

example:
    RegularizedDense (300),
    RegularizedDense (100),
    RegularizedDense (10, activation ="softmax" ,
                     kernel_initializer ="glorot_uniform" )
])
Dropout
Dropout  is one of the most popular regularization techniques for deep neural net‐

kernel_initializer ="he_normal" ,
])
Dropout
works. It was proposed23 by Geoffrey Hinton in 2012 and further detailed in a paper24
by Nitish Srivastava et al., and it has proven to be highly successful: even the state-of-
the-art neural networks got a 1–2% accuracy boost simply by adding dropout. This
may not sound like a lot, but when a model already has 95% accuracy, getting a 2%
accuracy boost means dropping the error rate by almost 40% (going from 5% error to
roughly 3%).
It is a fairly simple algorithm: at every training step, every neuron (including the

input neurons, but always excluding the output neurons) has a probability p of being
temporarily “dropped out, ” meaning it will be entirely ignored during this training
step, but it may be active during the next step (see Figure 11-9 ). The hyperparameter
p is called the dropout rate , and it is typically set to 50%. After training, neurons don’t
get dropped anymore. And that’s all (except for a technical detail we will discuss
momentarily).Avoiding Overfitting  Through Regularization | 357

Figure 11-9. Dropout regularization
It is quite surprising at first that this rather brutal technique works at all. Would a
company perform better if its employees were told to toss a coin every morning to
decide whether or not to go to work? Well, who knows; perhaps it would! The com‐
pany would obviously be forced to adapt its organization; it could not rely on any sin‐
gle person to fill in the coffee machine or perform any other critical tasks, so this
expertise would have to be spread across several people. Employees would have to
learn to cooperate with many of their coworkers, not just a handful of them. The
company would become much more resilient. If one person quit, it wouldn’t make
much of a difference. It’s unclear whether this idea would actually work for compa‐
nies, but it certainly does for neural networks. Neurons trained with dropout cannot
co-adapt with their neighboring neurons; they have to be as useful as possible on

their own. They also cannot rely excessively on just a few input neurons; they must
pay attention to each of their input neurons. They end up being less sensitive to slight
changes in the inputs. In the end you get a more robust network that generalizes bet‐
ter.
Another way to understand the power of dropout is to realize that a unique neural
network is generated at each training step. Since each neuron can be either present or
absent, there is a total of 2N possible networks (where N is the total number of drop‐

ter.
pable neurons). This is such a huge number that it is virtually impossible for the same
neural network to be sampled twice. Once you have run a 10,000 training steps, you
have essentially trained 10,000 different neural networks (each with just one training
instance). These neural networks are obviously not independent since they share
many of their weights, but they are nevertheless all different. The resulting neural
network can be seen as an averaging ensemble of all these smaller neural networks.
There is one small but important technical detail. Suppose p = 50%, in which case

during testing a neuron will be connected to twice as many input neurons as it was
(on average) during training. To compensate for this fact, we need to multiply each358 | Chapter 11: Training Deep Neural Networks

25This is specific to tf.keras, so you may prefer to use keras.backend.set_learning_phase(1)  before calling
the fit()  method (and set it back to 0 right after).
neuron’s input connection weights by 0.5 after training. If we don’t, each neuron will
get a total input signal roughly twice as large as what the network was trained on, and
it is unlikely to perform well. More generally, we need to multiply each input connec‐
tion weight by the keep probability  (1 – p) after training. Alternatively, we can divide
each neuron’s output by the keep probability during training (these alternatives are
not perfectly equivalent, but they work equally well).
To implement dropout using Keras, you can use the keras.layers.Dropout  layer.
During training, it randomly drops some inputs (setting them to 0) and divides the
remaining inputs by the keep probability. After training, it does nothing at all, it just
passes the inputs to the next layer. For example, the following code applies dropout

regularization before every Dense  layer, using a dropout rate of 0.2:
    keras.layers.Dropout(rate=0.2),
])

])
Since dropout is only active during training, the training loss is
penalized compared to the validation loss, so comparing the two
can be misleading. In particular, a model may be overfitting the
training set and yet have similar training and validation losses. So
make sure to evaluate the training loss without dropout (e.g., after
training). Alternatively, you can call the fit()  method inside a
with keras.backend.learning_phase_scope(1)  block: this will
force dropout to be active during both training and validation.25

If you observe that the model is overfitting, you can increase the dropout rate. Con‐
versely, you should try decreasing the dropout rate if the model underfits the training
set. It can also help to increase the dropout rate for large layers, and reduce it for
small ones. Moreover, many state-of-the-art architectures only use dropout after the
last hidden layer, so you may want to try this if full dropout is too strong.
Dropout does tend to significantly slow down convergence, but it usually results in a

much better model when tuned properly. So, it is generally well worth the extra time
and effort.Avoiding Overfitting  Through Regularization | 359

26“Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning, ” Y . Gal and Z.
Ghahramani (2016).27
Specifically, they show that training a dropout network is mathematically equivalent to approximate Bayesian
inference in a specific type of probabilistic model called a deep Gaussian Process .
If you want to regularize a self-normalizing network based on the
SELU activation function (as discussed earlier), you should use
AlphaDropout : this is a variant of dropout that preserves the mean
and standard deviation of its inputs (it was introduced in the same
paper as SELU, as regular dropout would break self-normalization).
Monte-Carlo (MC) Dropout
In 2016, a paper26 by Y arin Gal and Zoubin Ghahramani added more good reasons to
use dropout:•
First, the paper establishes a profound connection between dropout networks
(i.e., neural networks containing a dropout layer before every weight layer) and
approximate Bayesian inference27, giving dropout a solid mathematical justifica‐
tion.•

tion.•
Second, they introduce a powerful technique called MC Dropout , which can
boost the performance of any trained dropout model, without having to retrain it
or even modify it at all!•
Moreover, MC Dropout also provides a much better measure of the model’s
uncertainty.•
Finally, it is also amazingly simple to implement. If this all sounds like a “one
weird trick” advertisement, then take a look at the following code. It is the full
implementation of MC Dropout , boosting the dropout model we trained earlier,

without retraining it:
with keras.backend.learning_phase_scope (1): # force training mode = dropout on
    y_probas  = np.stack([model.predict(X_test_scaled )
                         for sample in range(100)])
y_proba = y_probas .mean(axis=0)
We first force training mode on, using a learning_phase_scope(1)  context. This
turns dropout on within the with  block. Then we make 100 predictions over the test
set, and we stack them. Since dropout is on, all predictions will be different. Recall

that predict()  returns a matrix with one row per instance, and one column per class.
Since there are 10,000 instances in the test set, and 10 classes, this is a matrix of shape
[10000, 10]. We stack 100 such matrices, so y_probas  is an array of shape [100, 10000,
10]. Once we average over the first dimension ( axis=0 ), we get y_proba , an array of
shape [10000, 10], like we would get with a single prediction. That’s all! Averaging360 | Chapter 11: Training Deep Neural Networks

over multiple predictions with dropout on gives us a Monte Carlo estimate that is
generally more reliable than the result of a single prediction with dropout off. For
example, let’s look at the model’s prediction for the first instance in the test set, with
dropout off:
>>> np.round(model.predict(X_test_scaled [:1]), 2)
array([[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],
The model seems almost certain that this image belongs to class 9 (ankle boot).
Should you trust it? Is there really so little room for doubt? Compare this with the
predictions made when dropout is activated:
>>> np.round(y_probas [:, :1], 2)
array([[[0.  , 0.  , 0.  , 0.  , 0.  , 0.14, 0.  , 0.17, 0.  , 0.68]],
       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.16, 0.  , 0.2 , 0.  , 0.64]],
       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.01, 0.  , 0.97]],
       [...]
This tells a very different story: apparently, when we activate dropout, the model is

       [...]
not sure anymore. It still seems to prefer class 9, but sometimes it hesitates with
classes 5 (sandal) and 7 (sneaker), which makes sense given they’re all footwear. Once
we average over the first dimension, we get the following MC dropout predictions:
>>> np.round(y_proba[:1], 2)
array([[0.  , 0.  , 0.  , 0.  , 0.  , 0.22, 0.  , 0.16, 0.  , 0.62]],
The model still thinks this image belongs to class 9, but only with a 62% confidence,
which seems much more reasonable than 99%. Plus it’s useful to know exactly which

other classes it thinks are likely. And you can also take a look at the standard devia‐
tion of the probability estimates :
>>> y_std = y_probas .std(axis=0)
>>> np.round(y_std[:1], 2)
array([[0.  , 0.  , 0.  , 0.  , 0.  , 0.28, 0.  , 0.21, 0.02, 0.32]],
Apparently there’s quite a lot of variance in the probability estimates: if you were
building a risk-sensitive system (e.g., a medical or financial system), you should prob‐
ably treat such an uncertain prediction with extreme caution. Y ou definitely would

not treat it like a 99% confident prediction. Moreover, the model’s accuracy got a
small boost from 86.8 to 86.9:
>>> accuracy  = np.sum(y_pred == y_test) / len(y_test)
>>> accuracy
0.8694Avoiding Overfitting  Through Regularization | 361

The number of Monte Carlo samples you use (100 in this example)
is a hyperparameter you can tweak. The higher it is, the more accu‐
rate the predictions and their uncertainty estimates will be. How‐
ever, it you double it, inference time will also be doubled.
Moreover, above a certain number of samples, you will notice little
improvement. So your job is to find the right tradeoff between
latency and accuracy, depending on your application.
If your model contains other layers that behave in a special way during training (such
as Batch Normalization layers), then you should not force training mode like we just
did. Instead, you should replace the Dropout  layers with the following MCDropout
class:
class MCDropout (keras.layers.Dropout):
        return super().call(inputs, training =True)
We just sublass the Dropout  layer and override the call()  method to force its train
ing argument to True  (see Chapter 12 ). Similarly, you could define an MCAlphaDrop

class:
out class by subclassing AlphaDropout  instead. If you are creating a model from
scratch, it’s just a matter of using MCDropout  rather than Dropout . But if you have a
model that was already trained using Dropout , you need to create a new model, iden‐
tical to the existing model except replacing the Dropout  layers with MCDropout , then
copy the existing model’s weights to your new model.
In short, MC Dropout is a fantastic technique that boosts dropout models and pro‐

vides better uncertainty estimates. And of course, since it is just regular dropout dur‐
ing training, it also acts like a regularizer.
Max-Norm Regularization
Another regularization technique that is quite popular for neural networks is called
max-norm regularization : for each neuron, it constrains the weights w of the incom‐
ing connections such that ∥ *w* ∥2 ≤ _r_, where r is the max-norm hyperparameter
and ∥ · ∥2 is the ℓ2 norm.
Max-norm regularization does not add a regularization loss term to the overall loss

function. Instead, it is typically implemented by computing ∥w∥2 after each training
step and clipping w if needed (w wr
∥w∥2
).
Reducing r increases the amount of regularization and helps reduce overfitting. Max-
norm regularization can also help alleviate the vanishing/exploding gradients prob‐
lems (if you are not using Batch Normalization).362 | Chapter 11: Training Deep Neural Networks

To implement max-norm regularization in Keras, just set every hidden layer’s ker
nel_constraint  argument to a max_norm()  constraint, with the appropriate max
value, for example:
keras.layers.Dense(100, activation ="elu", kernel_initializer ="he_normal" ,
                   kernel_constraint =keras.constraints .max_norm (1.))
After each training iteration, the model’s fit()  method will call the object returned
by max_norm() , passing it the layer’s weights and getting clipped weights in return,
which then replace the layer’s weights. As we will see in Chapter 12 , you can define
your own custom constraint function if you ever need to, and use it as the ker
nel_constraint . Y ou can also constrain the bias terms by setting the bias_con
straint  argument.
The max_norm()  function has an axis  argument that defaults to 0. A Dense  layer usu‐
ally has weights of shape [number of inputs, number of neurons], so using axis=0
means that the max norm constraint will apply independently to each neuron’s weight

vector. If you want to use max-norm with convolutional layers (see Chapter 14 ),
make sure to set the max_norm()  constraint’s axis  argument appropriately (usually
axis=[0, 1, 2] ).
Summary and Practical Guidelines
In this chapter, we have covered a wide range of techniques and you may be wonder‐
ing which ones you should use. The configuration in Table 11-2  will work fine in

most cases, without requiring much hyperparameter tuning.Table 11-2. Default DNN configuration HyperparameterDefault valueKernel initializer:LeCun initializationActivation function:SELUNormalization:None (self-normalization)Regularization:Early stoppingOptimizer:NadamLearning rate schedule:Performance scheduling
Don’t forget to standardize the input features! Of course, you should also try to reuse
parts of a pretrained neural network if you can find one that solves a similar problem,

or use unsupervised pretraining if you have a lot of unlabeled data, or pretraining on
an auxiliary task if you have a lot of labeled data for a similar task.
The default configuration in Table 11-2  may need to be tweaked:Summary and Practical Guidelines | 363

•If your model self-normalizes:—
If it overfits the training set, then you should add alpha dropout (and always
use early stopping as well). Do not use other regularization methods, or else
they would break self-normalization.•
If your model cannot self-normalize (e.g., it is a recurrent net or it contains skip
connections):—
Y ou can try using ELU (or another activation function) instead of SELU, it
may perform better. Make sure to change the initialization method accord‐
ingly (e.g., He init for ELU or ReLU).—
If it is a deep network, you should use Batch Normalization after every hidden
layer. If it overfits the training set, you can also try using max-norm or ℓ2 reg‐
ularization.•
If you need a sparse model, you can use ℓ1 regularization (and optionally zero out
the tiny weights after training). If you need an even sparser model, you can try
using FTRL instead of Nadam optimization, along with ℓ1 regularization. In any
case, this will break self-normalization, so you will need to switch to BN if your

model is deep.•
If you need a low-latency model (one that performs lightning-fast predictions),
you may need to use less layers, avoid Batch Normalization, and possibly replace
the SELU activation function with the leaky ReLU. Having a sparse model will
also help. Y ou may also want to reduce the float precision from 32-bits to 16-bit
(or even 8-bits) (see ???).•
If you are building a risk-sensitive application, or inference latency is not very

important in your application, you can use MC Dropout to boost performance
and get more reliable probability estimates, along with uncertainty estimates.
With these guidelines, you are now ready to train very deep nets! I hope you are now
convinced that you can go a very long way using just Keras. However, there may
come a time when you need to have even more control, for example to write a custom
loss function or to tweak the training algorithm. For such cases, you will need to use
TensorFlow’s lower-level API, as we will see in the next chapter.

Is it okay to initialize all the weights to the same value as long as that value is
selected randomly using He initialization?2.
Is it okay to initialize the bias terms to 0?3.
Name three advantages of the SELU activation function over ReLU.364 | Chapter 11: Training Deep Neural Networks

4. In which cases would you want to use each of the following activation functions:
SELU, leaky ReLU (and its variants), ReLU, tanh, logistic, and softmax?5.
What may happen if you set the momentum  hyperparameter too close to 1 (e.g.,
0.99999) when using an SGD optimizer?6.
Name three ways you can produce a sparse model.7.
Does dropout slow down training? Does it slow down inference (i.e., making
predictions on new instances)? What are about MC dropout?8.
Deep Learning.a.
Build a DNN with five hidden layers of 100 neurons each, He initialization,
and the ELU activation function.b.
Using Adam optimization and early stopping, try training it on MNIST but
only on digits 0 to 4, as we will use transfer learning for digits 5 to 9 in the
next exercise. Y ou will need a softmax output layer with five neurons, and as
always make sure to save checkpoints at regular intervals and save the final
model so you can reuse it later.c.
Tune the hyperparameters using cross-validation and see what precision you
can achieve.d.

Now try adding Batch Normalization and compare the learning curves: is it
converging faster than before? Does it produce a better model?e.
Is the model overfitting the training set? Try adding dropout to every layer
and try again. Does it help?9.
Transfer learning.a.
Create a new DNN that reuses all the pretrained hidden layers of the previous
model, freezes them, and replaces the softmax output layer with a new one.b.
Train this new DNN on digits 5 to 9, using only 100 images per digit, and time

how long it takes. Despite this small number of examples, can you achieve
high precision?c.
Try caching the frozen layers, and train the model again: how much faster is it
now?d.
Try again reusing just four hidden layers instead of five. Can you achieve a
higher precision?e.
Now unfreeze the top two hidden layers and continue training: can you get
the model to perform even better?10.
Pretraining on an auxiliary task.a.
In this exercise you will build a DNN that compares two MNIST digit images

now?d.
and predicts whether they represent the same digit or not. Then you will reuse
the lower layers of this network to train an MNIST classifier using very littleExercises | 365

training data. Start by building two DNNs (let’s call them DNN A and B), both
similar to the one you built earlier but without the output layer: each DNN
should have five hidden layers of 100 neurons each, He initialization, and ELU
activation. Next, add one more hidden layer with 10 units on top of both
DNNs. To do this, you should use a keras.layers.Concatenate  layer to con‐
catenate the outputs of both DNNs for each instance, then feed the result to
the hidden layer. Finally, add an output layer with a single neuron using the
logistic activation function.b.
Split the MNIST training set in two sets: split #1 should containing 55,000
images, and split #2 should contain contain 5,000 images. Create a function
that generates a training batch where each instance is a pair of MNIST images
picked from split #1. Half of the training instances should be pairs of images
that belong to the same class, while the other half should be images from dif‐

ferent classes. For each pair, the training label should be 0 if the images are
from the same class, or 1 if they are from different classes.c.
Train the DNN on this training set. For each image pair, you can simultane‐
ously feed the first image to DNN A and the second image to DNN B. The
whole network will gradually learn to tell whether two images belong to the
same class or not.d.
Now create a new DNN by reusing and freezing the hidden layers of DNN A

and adding a softmax output layer on top with 10 neurons. Train this network
on split #2 and see if you can achieve high performance despite having only
500 images per class.
Solutions to these exercises are available in ???.366 | Chapter 11: Training Deep Neural Networks

CHAPTER 12
Custom Models and Training with
release of these titles. The following will be Chapter 12 in the final
So far we have used only TensorFlow’s high level API, tf.keras, but it already got us
pretty far: we built various neural network architectures, including regression and
classification nets, wide & deep nets and self-normalizing nets, using all sorts of tech‐
niques, such as Batch Normalization, dropout, learning rate schedules, and more. In
fact, 95% of the use cases you will encounter will not require anything else than
tf.keras (and tf.data, see Chapter 13 ). But now it’s time to dive deeper into TensorFlow
and take a look at its lower-level Python API . This will be useful when you need extra

control, to write custom loss functions, custom metrics, layers, models, initializers,
regularizers, weight constraints and more. Y ou may even need to fully control the
training loop itself, for example to apply special transformations or constraints to the
gradients (beyond just clipping them), or to use multiple optimizers for different
parts of the network. We will cover all these cases in this chapter, then we will also
look at how you can boost your custom models and training algorithms using Ten‐

sorFlow’s automatic graph generation feature. But first, let’s take a quick tour of Ten‐
sorFlow.367

1TensorFlow also includes another Deep Learning API called the Estimators API , but it is now recommended
to use tf.keras instead.
TensorFlow 2.0 was released in March 2019, making TensorFlow
much easier to use. The first edition of this book used TF 1, while
this edition uses TF 2.
A Quick Tour of TensorFlow
As you know, TensorFlow  is a powerful library for numerical computation, particu‐
larly well suited and fine-tuned for large-scale Machine Learning (but you could use
it for anything else that requires heavy computations). It was developed by the Google
Brain team and it powers many of Google’s large-scale services, such as Google Cloud
Speech, Google Photos, and Google Search. It was open sourced in November 2015,
and it is now the most popular deep learning library (in terms of citations in papers,
adoption in companies, stars on github, etc.): countless projects use TensorFlow for
all sorts of Machine Learning tasks, such as image classification, natural language

processing (NLP), recommender systems, time series forecasting, and much more.
So what does TensorFlow actually offer? Here’s a summary:•
Its core is very similar to NumPy, but with GPU support.•
It also supports distributed computing (across multiple devices and servers).•
It includes a kind of just-in-time (JIT) compiler that allows it to optimize compu‐
tations for speed and memory usage: it works by extracting the computation
graph  from a Python function, then optimizing it (e.g., by pruning unused nodes)

and finally running it efficiently (e.g., by automatically running independent
operations in parallel).•
Computation graphs can be exported to a portable format, so you can train a
TensorFlow model in one environment (e.g., using Python on Linux), and run it
in another (e.g., using Java on an Android device).•
It implements autodiff (see Chapter 10  and ???), and provides some excellent
optimizers, such as RMSProp, Nadam and FTRL (see Chapter 11 ), so you can
easily minimize all sorts of loss functions.•

TensorFlow offers many more features, built on top of these core features: the
most important is of course tf.keras1, but it also has data loading & preprocessing
ops (tf.data, tf.io, etc.), image processing ops (tf.image), signal processing ops
(tf.signal), and more (see Figure 12-1  for an overview of TensorFlow’s Python
API).368 | Chapter 12: Custom Models and Training with TensorFlow

2If you ever need to (but you probably won’t), you can write your own operations using the C++ API.3
If you are a researcher, you may be eligible to use these TPUs for free, see https://tensorflow.org/tfrc/  for more
details.
Figure 12-1. TensorFlow’s Python API
We will cover many of the packages and functions of the Tensor‐
Flow API, but it’s impossible to cover them all so you should really
take some time to browse through the API: you will find that it is
quite rich and well documented.
At the lowest level, each TensorFlow operation is implemented using highly efficient
C++ code2. Many operations (or ops for short) have multiple implementations, called
kernels : each kernel is dedicated to a specific device type, such as CPUs, GPUs, or
even TPUs ( Tensor Processing Units ). As you may know, GPUs can dramatically speed
up computations by splitting computations into many smaller chunks and running
them in parallel across many GPU threads. TPUs are even faster. Y ou can purchase

your own GPU devices (for now, TensorFlow only supports Nvidia cards with CUDA
Compute Capability 3.5+), but TPUs are only available on Google Cloud Machine
Learning Engine  (see ???).3
TensorFlow’s architecture is shown in Figure 12-2 : most of the time your code will
use the high level APIs (especially tf.keras and tf.data), but when you need more flexi‐
bility you will use the lower level Python API, handling tensors directly. Note that

APIs for other languages are also available. In any case, TensorFlow’s executionA Quick Tour of TensorFlow | 369

engine will take care of running the operations efficiently, even across multiple devi‐
ces and machines if you tell it to.
Figure 12-2. TensorFlow’s architecture
TensorFlow runs not only on Windows, Linux, and MacOS, but also on mobile devi‐
ces (using TensorFlow Lite ), including both iOS and Android (see ???). If you do not
want to use the Python API, there are also C++, Java, Go and Swift APIs. There is
even a Javascript implementation called TensorFlow.js  that makes it possible to run
your models directly in your browser.
There’s more to TensorFlow than just the library. TensorFlow is at the center of an
extensive ecosystem of libraries. First, there’s TensorBoard for visualization (see
Chapter 10 ). Next, there’s TensorFlow Extended (TFX) , which is a set of libraries built
by Google to productionize TensorFlow projects: it includes tools for data validation,
preprocessing, model analysis and serving (with TF Serving, see ???). Google also

launched TensorFlow Hub , a way to easily download and reuse pretrained neural net‐
works. Y ou can also get many neural network architectures, some of them pretrained,
in TensorFlow’s model garden . Check out the TensorFlow Resources , or https://
github.com/jtoy/awesome-tensorflow  for more TensorFlow-based projects. Y ou will
find hundreds of TensorFlow projects on GitHub, so it is often easy to find existing
code for whatever you are trying to do.
More and more ML papers are released along with their implemen‐

tation, and sometimes even with pretrained models. Check out
https://paperswithcode.com/  to easily find them.370 | Chapter 12: Custom Models and Training with TensorFlow

Last but not least, TensorFlow has a dedicated team of passionate and helpful devel‐
opers, and a large community contributing to improving it. To ask technical ques‐
tions, you should use http://stackoverflow.com/  and tag your question with tensorflow
and python . Y ou can file bugs and feature requests through GitHub. For general dis‐
cussions, join the Google group .
Okay, it’s time to start coding!
Using TensorFlow like NumPy
TensorFlow’s API revolves around tensors , hence the name Tensor-Flow. A tensor is
usually a multidimensional array (exactly like a NumPy ndarray ), but it can also hold
a scalar (a simple value, such as 42). These tensors will be important when we create
custom cost functions, custom metrics, custom layers and more, so let’s see how to
create and manipulate them.
Tensors and Operations
Y ou can easily create a tensor, using tf.constant() . For example, here is a tensor
representing a matrix with two rows and three columns of floats:

>>> tf.constant ([[1., 2., 3.], [4., 5., 6.]]) # matrix
<tf.Tensor: id=0, shape=(2, 3), dtype=float32, numpy=
array([[1., 2., 3.],
       [4., 5., 6.]], dtype=float32)>
>>> tf.constant (42) # scalar
<tf.Tensor: id=1, shape=(), dtype=int32, numpy=42>
Just like an ndarray , a tf.Tensor  has a shape and a data type ( dtype ):
>>> t = tf.constant ([[1., 2., 3.], [4., 5., 6.]])
>>> t.shape
TensorShape([2, 3])
>>> t.dtype
tf.float32
Indexing works much like in NumPy:
>>> t[:, 1:]
<tf.Tensor: id=5, shape=(2, 2), dtype=float32, numpy=
array([[2., 3.],

       [5., 6.]], dtype=float32)>
>>> t[..., 1, tf.newaxis]
<tf.Tensor: id=15, shape=(2, 1), dtype=float32, numpy=
array([[2.],
       [5.]], dtype=float32)>
Most importantly, all sorts of tensor operations are available:
>>> t + 10
<tf.Tensor: id=18, shape=(2, 3), dtype=float32, numpy=Using TensorFlow like NumPy | 371

array([[11., 12., 13.],
       [14., 15., 16.]], dtype=float32)>
>>> tf.square(t)
<tf.Tensor: id=20, shape=(2, 3), dtype=float32, numpy=
array([[ 1.,  4.,  9.],
       [16., 25., 36.]], dtype=float32)>
>>> t @ tf.transpose (t)
<tf.Tensor: id=24, shape=(2, 2), dtype=float32, numpy=
array([[14., 32.],
       [32., 77.]], dtype=float32)>
Note that writing t + 10  is equivalent to calling tf.add(t, 10)  (indeed, Python calls
the magic method t.__add__(10) , which just calls tf.add(t, 10) ). Other operators
(like -, *, etc.) are also supported. The @ operator was added in Python 3.5, for matrix
multiplication: it is equivalent to calling the tf.matmul()  function.
Y ou will find all the basic math operations you need (e.g., tf.add() , tf.multiply() ,
tf.square() , tf.exp() , tf.sqrt() …), and more generally most operations that you
can find in NumPy (e.g., tf.reshape() , tf.squeeze() , tf.tile() ), but sometimes
with a different name (e.g., tf.reduce_mean() , tf.reduce_sum() , tf.reduce_max() ,

tf.math.log()  are the equivalent of np.mean() , np.sum() , np.max()  and np.log() ).
When the name differs, there is often a good reason for it: for example, in Tensor‐
Flow you must write tf.transpose(t) , you cannot just write t.T like in NumPy. The
reason is that it does not do exactly the same thing: in TensorFlow, a new tensor is
created with its own copy of the transposed data, while in NumPy, t.T is just a trans‐
posed view on the same data. Similarly, the tf.reduce_sum()  operation is named this

way because its GPU kernel (i.e., GPU implementation) uses a reduce algorithm that
does not guarantee the order in which the elements are added: because 32-bit floats
have limited precision, this means that the result may change ever so slightly every
time you call this operation. The same is true of tf.reduce_mean()  (but of course
tf.reduce_max()  is deterministic).372 | Chapter 12: Custom Models and Training with TensorFlow

4A notable exception is tf.math.log()  which is commonly used but there is no tf.log()  alias (as it might be
confused with logging).
Many functions and classes have aliases. For example, tf.add()
and tf.math.add()  are the same function. This allows TensorFlow
to have concise names for the most common operations4, while
preserving well organized packages.
Keras’ Low-Level API
The Keras API actually has its own low-level API, located in keras.backend . It
includes functions like square() , exp() , sqrt()  and so on. In tf.keras, these func‐
tions generally just call the corresponding TensorFlow operations. If you want to
write code that will be portable to other Keras implementations, you should use these
Keras functions. However, they only cover a subset of all functions available in Ten‐
sorFlow, so in this book we will use the TensorFlow operations directly. Here is as
simple example using keras.backend , which is commonly named K for short:
>>> K = keras.backend

>>> K.square(K.transpose (t)) + 10
<tf.Tensor: id=39, shape=(3, 2), dtype=float32, numpy=
array([[11., 26.],
       [14., 35.],
       [19., 46.]], dtype=float32)>
Tensors and NumPy
Tensors play nice with NumPy: you can create a tensor from a NumPy array, and vice
versa, and you can even apply TensorFlow operations to NumPy arrays and NumPy
operations to tensors:
>>> a = np.array([2., 4., 5.])
>>> tf.constant (a)
<tf.Tensor: id=111, shape=(3,), dtype=float64, numpy=array([2., 4., 5.])>
>>> t.numpy() # or np.array(t)

       [4., 5., 6.]], dtype=float32)
>>> tf.square(a)
<tf.Tensor: id=116, shape=(3,), dtype=float64, numpy=array([4., 16., 25.])>
>>> np.square(t)
       [16., 25., 36.]], dtype=float32)Using TensorFlow like NumPy | 373

Notice that NumPy uses 64-bit precision by default, while Tensor‐
Flow uses 32-bit. This is because 32-bit precision is generally more
than enough for neural networks, plus it runs faster and uses less
RAM. So when you create a tensor from a NumPy array, make sure
to set dtype=tf.float32 .
Type Conversions
Type conversions can significantly hurt performance, and they can easily go unno‐
ticed when they are done automatically. To avoid this, TensorFlow does not perform
any type conversions automatically: it just raises an exception if you try to execute an
operation on tensors with incompatible types. For example, you cannot add a float
tensor and an integer tensor, and you cannot even add a 32-bit float and a 64-bit float:
>>> tf.constant (2.) + tf.constant (40)
Traceback[...]InvalidArgumentError[...]expected to be a float[...]
>>> tf.constant (2.) + tf.constant (40., dtype=tf.float64)
Traceback[...]InvalidArgumentError[...]expected to be a double[...]

This may be a bit annoying at first, but remember that it’s for a good cause! And of
course you can use tf.cast()  when you really need to convert types:
>>> t2 = tf.constant (40., dtype=tf.float64)
>>> tf.constant (2.0) + tf.cast(t2, tf.float32)
<tf.Tensor: id=136, shape=(), dtype=float32, numpy=42.0>
Variables
So far, we have used constant tensors: as their name suggests, you cannot modify
them. However, the weights in a neural network need to be tweaked by backpropaga‐

Variables
tion, and other parameters may also need to change over time (e.g., a momentum
optimizer keeps track of past gradients). What we need is a tf.Variable :
>>> v = tf.Variable ([[1., 2., 3.], [4., 5., 6.]])
>>> v
<tf.Variable 'Variable:0' shape=(2, 3) dtype=float32, numpy=
A tf.Variable  acts much like a constant tensor: you can perform the same opera‐
tions with it, it plays nicely with NumPy as well, and it is just as picky with types. But

>>> v
it can also be modified in place using the assign()  method (or assign_add()  or
assign_sub()  which increment or decrement the variable by the given value). Y ou
can also modify individual cells (or slices), using the cell’s (or slice’s) assign()
method (direct item assignment will not work), or using the scatter_update()  or
scatter_nd_update()  methods:
v.assign(2 * v)           # => [[2., 4., 6.], [8., 10., 12.]]

v[0, 1].assign(42)        # => [[2., 42., 6.], [8., 10., 12.]]374 | Chapter 12: Custom Models and Training with TensorFlow

v[:, 2].assign([0., 1.])  # => [[2., 42., 0.], [8., 10., 1.]]
v.scatter_nd_update (indices=[[0, 0], [1, 2]], updates=[100., 200.])
                          # => [[100., 42., 0.], [8., 10., 200.]]
In practice you will rarely have to create variables manually, since
Keras provides an add_weight()  method that will take care of it for
you, as we will see. Moreover, model parameters will generally be
updated directly by the optimizers, so you will rarely need to
update variables manually.
Other Data Structures
TensorFlow supports several other data structures, including the following (please see
the notebook or ??? for more details):•
Sparse tensors  (tf.SparseTensor ) efficiently represent tensors containing mostly
0s. The tf.sparse  package contains operations for sparse tensors.•
Tensor arrays  (tf.TensorArray ) are lists of tensors. They have a fixed size by
default, but can optionally be made dynamic. All tensors they contain must have
the same shape and data type.•

Ragged tensors  (tf.RaggedTensor ) represent static lists of lists of tensors, where
every tensor has the same shape and data type. The tf.ragged  package contains
operations for ragged tensors.•
String tensors  are regular tensors of type tf.string . These actually represent byte
strings, not Unicode strings, so if you create a string tensor using a Unicode
string (e.g., a regular Python 3 string like "café"` ), then it will get encoded to
UTF-8 automatically (e.g., b"caf\xc3\xa9" ). Alternatively, you can represent

Unicode strings using tensors of type tf.int32 , where each item represents a
Unicode codepoint (e.g., [99, 97, 102, 233] ). The tf.strings  package (with
an s) contains ops for byte strings and Unicode strings (and to convert one into
the other).•
Sets are just represented as regular tensors (or sparse tensors) containing one or
more sets, and you can manipulate them using operations from the tf.sets
package.•
Queues , including First In, First Out (FIFO) queues ( FIFOQueue ), queues that can
prioritize some items ( PriorityQueue ), queues that shuffle their items ( Random

package.•
ShuffleQueue ), and queues that can batch items of different shapes by padding
(PaddingFIFOQueue ). These classes are all in the tf.queue  package.
With tensors, operations, variables and various data structures at your disposal, you
are now ready to customize your models and training algorithms!Using TensorFlow like NumPy | 375

Customizing Models and Training Algorithms
Let’s start by creating a custom loss function, which is a simple and common use case.
Custom Loss Functions
Suppose you want to train a regression model, but your training set is a bit noisy. Of
course, you start by trying to clean up your dataset by removing or fixing the outliers,
but it turns out to be insufficient, the dataset is still noisy. Which loss function should
you use? The mean squared error might penalize large errors too much, so your
model will end up being imprecise. The mean absolute error would not penalize out‐
liers as much, but training might take a while to converge and the trained model
might not be very precise. This is probably a good time to use the Huber loss (intro‐
duced in Chapter 10 ) instead of the good old MSE. The Huber loss is not currently
part of the official Keras API, but it is available in tf.keras (just use an instance of the
keras.losses.Huber  class). But let’s pretend it’s not there: implementing it is easy as

pie! Just create a function that takes the labels and predictions as arguments, and use
TensorFlow operations to compute every instance’s loss:
def huber_fn (y_true, y_pred):
    error = y_true - y_pred
    is_small_error  = tf.abs(error) < 1
    squared_loss  = tf.square(error) / 2
    linear_loss   = tf.abs(error) - 0.5
    return tf.where(is_small_error , squared_loss , linear_loss )
For better performance, you should use a vectorized implementa‐
tion, as in this example. Moreover, if you want to benefit from Ten‐
sorFlow’s graph features, you should use only TensorFlow
operations.

It is also preferable to return a tensor containing one loss per instance, rather than
returning the mean loss. This way, Keras can apply class weights or sample weights
when requested (see Chapter 10 ).
Next, you can just use this loss when you compile the Keras model, then train your
model:
model.compile(loss=huber_fn , optimizer ="nadam")
model.fit(X_train, y_train, [...])
And that’s it! For each batch during training, Keras will call the huber_fn()  function

model:
to compute the loss, and use it to perform a Gradient Descent step. Moreover, it will
keep track of the total loss since the beginning of the epoch, and it will display the
mean loss.376 | Chapter 12: Custom Models and Training with TensorFlow

But what happens to this custom loss when we save the model?
Saving and Loading Models That Contain Custom Components
Saving a model containing a custom loss function actually works fine, as Keras just
saves the name of the function. However, whenever you load it, you need to provide a
dictionary that maps the function name to the actual function. More generally, when
you load a model containing custom objects, you need to map the names to the
objects:
model = keras.models.load_model ("my_model_with_a_custom_loss.h5" ,
                                custom_objects ={"huber_fn" : huber_fn })
With the current implementation, any error between -1 and 1 is considered “small” .
But what if we want a different threshold? One solution is to create a function that
creates a configured loss function:
def create_huber (threshold =1.0):
    def huber_fn (y_true, y_pred):
        error = y_true - y_pred
        is_small_error  = tf.abs(error) < threshold
        squared_loss  = tf.square(error) / 2

custom_objects ={"huber_fn" : huber_fn })
        linear_loss   = threshold  * tf.abs(error) - threshold **2 / 2
        return tf.where(is_small_error , squared_loss , linear_loss )
    return huber_fn
model.compile(loss=create_huber (2.0), optimizer ="nadam")
Unfortunately, when you save the model, the threshold  will not be saved. This means
that you will have to specify the threshold  value when loading the model (note that
the name to use is "huber_fn" , which is the name of the function we gave Keras, not
the name of the function that created it):

return tf.where(is_small_error , squared_loss , linear_loss )
model = keras.models.load_model ("my_model_with_a_custom_loss_threshold_2.h5" ,
                                custom_objects ={"huber_fn" : create_huber (2.0)})
Y ou can solve this by creating a subclass of the keras.losses.Loss  class, and imple‐
ment its get_config()  method:
class HuberLoss (keras.losses.Loss):
    def __init__ (self, threshold =1.0, **kwargs):
        self.threshold  = threshold
        super().__init__ (**kwargs)
    def call(self, y_true, y_pred):
        is_small_error  = tf.abs(error) < self.threshold

custom_objects ={"huber_fn" : create_huber (2.0)})
        linear_loss   = self.threshold  * tf.abs(error) - self.threshold **2 / 2
    def get_config (self):
        base_config  = super().get_config ()
        return {**base_config , "threshold" : self.threshold }Customizing Models and Training Algorithms | 377

5It would not be a good idea to use a weighted mean: if we did, then two instances with the same weight but in
different batches would have a different impact on training, depending on the total weight of each batch.
The Keras API only specifies how to use subclassing to define lay‐
ers, models, callbacks, and regularizers. If you build other compo‐
nents (such as losses, metrics, initializers or constraints) using
subclassing, they may not be portable to other Keras implementa‐
tions.
Let’s walk through this code:•
The constructor accepts **kwargs  and passes them to the parent constructor,
which handles standard hyperparameters: the name  of the loss and the reduction
algorithm to use to aggregate the individual instance losses. By default, it is
"sum_over_batch_size" , which means that the loss will be the sum of the
instance losses, possibly weighted by the sample weights, if any, and then divide
the result by the batch size (not by the sum of weights, so this is not the weighted

mean).5. Other possible values are "sum"  and None .•
The call()  method takes the labels and predictions, computes all the instance
losses, and returns them.•
The get_config()  method returns a dictionary mapping each hyperparameter
name to its value. It first calls the parent class’s get_config()  method, then adds
the new hyperparameters to this dictionary (note that the convenient {**x}  syn‐
tax was added in Python 3.5).
Y ou can then use any instance of this class when you compile the model:

model.compile(loss=HuberLoss (2.), optimizer ="nadam")
When you save the model, the threshold will be saved along with it, and when you
load the model you just need to map the class name to the class itself:
model = keras.models.load_model ("my_model_with_a_custom_loss_class.h5" ,
                                custom_objects ={"HuberLoss" : HuberLoss })
When you save a model, Keras calls the loss instance’s get_config()  method and
saves the config as JSON in the HDF5 file. When you load the model, it calls the

from_config()  class method on the HuberLoss  class: this method is implemented by
the base class ( Loss ) and just creates an instance of the class, passing **config  to the
constructor.
That’s it for losses! It was not too hard, was it? Well it’s just as simple for custom acti‐
vation functions, initializers, regularizers, and constraints. Let’s look at these now.378 | Chapter 12: Custom Models and Training with TensorFlow

Custom Activation Functions, Initializers, Regularizers, and
Constraints
Most Keras functionalities, such as losses, regularizers, constraints, initializers, met‐
rics, activation functions, layers and even full models can be customized in very much
the same way. Most of the time, you will just need to write a simple function, with the
appropriate inputs and outputs. For example, here are examples of a custom activa‐
tion function (equivalent to keras.activations.softplus  or tf.nn.softplus ), a
custom Glorot initializer (equivalent to keras.initializers.glorot_normal ), a cus‐
tom ℓ1 regularizer (equivalent to keras.regularizers.l1(0.01) ) and a custom con‐
straint that ensures weights are all positive (equivalent to
keras.constraints.nonneg()  or tf.nn.relu ):
def my_softplus (z): # return value is just tf.nn.softplus(z)
    return tf.math.log(tf.exp(z) + 1.0)
def my_glorot_initializer (shape, dtype=tf.float32):
    stddev = tf.sqrt(2. / (shape[0] + shape[1]))

    return tf.random.normal(shape, stddev=stddev, dtype=dtype)
def my_l1_regularizer (weights):
    return tf.reduce_sum (tf.abs(0.01 * weights))
def my_positive_weights (weights): # return value is just tf.nn.relu(weights)
    return tf.where(weights < 0., tf.zeros_like (weights), weights)
As you can see, the arguments depend on the type of custom function. These custom
functions can then be used normally, for example:
layer = keras.layers.Dense(30, activation =my_softplus ,

return tf.random.normal(shape, stddev=stddev, dtype=dtype)
                           kernel_initializer =my_glorot_initializer ,
                           kernel_regularizer =my_l1_regularizer ,
                           kernel_constraint =my_positive_weights )
The activation function will be applied to the output of this Dense  layer, and its result
will be passed on to the next layer. The layer’s weights will be initialized using the
value returned by the initializer. At each training step the weights will be passed to the

kernel_initializer =my_glorot_initializer ,
regularization function to compute the regularization loss, which will be added to the
main loss to get the final loss used for training. Finally, the constraint function will be
called after each training step, and the layer’s weights will be replaced by the con‐
strained weights.
If a function has some hyperparameters that need to be saved along with the model,
then you will want to subclass the appropriate class, such as keras.regulariz
ers.Regularizer , keras.constraints.Constraint , keras.initializers.Initial

izer  or keras.layers.Layer  (for any layer, including activation functions). For
example, much like we did for the custom loss, here is a simple class for ℓ1 regulariza‐Customizing Models and Training Algorithms | 379

6However, the Huber loss is seldom used as a metric (the MAE or MSE are preferred).tion, that saves its factor  hyperparameter (this time we do not need to call the parent
constructor or the get_config()  method, as they are not defined by the parent class):
class MyL1Regularizer (keras.regularizers .Regularizer ):
    def __init__ (self, factor):
        self.factor = factor
    def __call__ (self, weights):
        return tf.reduce_sum (tf.abs(self.factor * weights))
        return {"factor" : self.factor}
Note that you must implement the call()  method for losses, layers (including activa‐
tion functions) and models, or the __call__()  method for regularizers, initializers
and constraints. For metrics, things are a bit different, as we will see now.
Custom Metrics
Losses and metrics are conceptually not the same thing: losses are used by Gradient
Descent to train  a model, so they must be differentiable (at least where they are evalu‐

def get_config (self):
ated) and their gradients should not be 0 everywhere. Plus, it’s okay if they are not
easily interpretable by humans (e.g. cross-entropy). In contrast, metrics are used to
evaluate  a model, they must be more easily interpretable, and they can be non-
differentiable or have 0 gradients everywhere (e.g., accuracy).
That said, in most cases, defining a custom metric function is exactly the same as
defining a custom loss function. In fact, we could even use the Huber loss function we

created earlier as a metric6, it would work just fine (and persistence would also work
the same way, in this case only saving the name of the function, "huber_fn" ):
model.compile(loss="mse", optimizer ="nadam", metrics=[create_huber (2.0)])
For each batch during training, Keras will compute this metric and keep track of its
mean since the beginning of the epoch. Most of the time, this is exactly what you
want. But not always! Consider a binary classifier’s precision, for example. As we saw

in Chapter 3 , precision is the number of true positives divided by the number of posi‐
tive predictions (including both true positives and false positives). Suppose the model
made 5 positive predictions in the first batch, 4 of which were correct: that’s 80% pre‐
cision. Then suppose the model made 3 positive predictions in the second batch, but
they were all incorrect: that’s 0% precision for the second batch. If you just compute
the mean of these two precisions, you get 40%. But wait a second, this is not the mod‐

el’s precision over these two batches! Indeed, there were a total of 4 true positives (4 +
0) out of 8 positive predictions (5 + 3), so the overall precision is 50%, not 40%. What
we need is an object that can keep track of the number of true positives and the num‐380 | Chapter 12: Custom Models and Training with TensorFlow

ber of false positives, and compute their ratio when requested. This is precisely what
the keras.metrics.Precision  class does:
>>> precision  = keras.metrics.Precision ()
>>> precision ([0, 1, 1, 1, 0, 1, 0, 1], [1, 1, 0, 1, 0, 1, 0, 1])
<tf.Tensor: id=581729, shape=(), dtype=float32, numpy=0.8>
>>> precision ([0, 1, 0, 0, 1, 0, 1, 1], [1, 0, 1, 1, 0, 0, 0, 0])
<tf.Tensor: id=581780, shape=(), dtype=float32, numpy=0.5>
In this example, we created a Precision  object, then we used it like a function, pass‐
ing it the labels and predictions for the first batch, then for the second batch (note
that we could also have passed sample weights). We used the same number of true
and false positives as in the example we just discussed. After the first batch, it returns
the precision of 80%, then after the second batch it returns 50% (which is the overall
precision so far, not the second batch’s precision). This is called a streaming metric  (or
stateful metric ), as it is gradually updated, batch after batch.

At any point, we can call the result()  method to get the current value of the metric.
We can also look at its variables (tracking the number of true and false positives)
using the variables  attribute, and reset these variables using the reset_states()
method:
>>> p.result()
<tf.Tensor: id=581794, shape=(), dtype=float32, numpy=0.5>
>>> p.variables
[<tf.Variable 'true_positives:0' [...] numpy=array([4.], dtype=float32)>,
 <tf.Variable 'false_positives:0' [...] numpy=array([4.], dtype=float32)>]

method:
>>> p.reset_states () # both variables get reset to 0.0
If you need to create such a streaming metric, you can just create a subclass of the
keras.metrics.Metric  class. Here is a simple example that keeps track of the total
Huber loss and the number of instances seen so far. When asked for the result, it
returns the ratio, which is simply the mean Huber loss:
class HuberMetric (keras.metrics.Metric):

        super().__init__ (**kwargs) # handles base args (e.g., dtype)
        self.huber_fn  = create_huber (threshold )
        self.total = self.add_weight ("total", initializer ="zeros")
        self.count = self.add_weight ("count", initializer ="zeros")
    def update_state (self, y_true, y_pred, sample_weight =None):
        metric = self.huber_fn (y_true, y_pred)
        self.total.assign_add (tf.reduce_sum (metric))
        self.count.assign_add (tf.cast(tf.size(y_true), tf.float32))
    def result(self):

self.threshold  = threshold
        return self.total / self.count
        return {**base_config , "threshold" : self.threshold }Customizing Models and Training Algorithms | 381

7This class is for illustration purposes only. A simpler and better implementation would just subclass the
keras.metrics.Mean  class, see the notebook for an example.
Let’s walk through this code:7: •
The constructor uses the add_weight()  method to create the variables needed to
keep track of the metric’s state over multiple batches, in this case the sum of all
Huber losses ( total ) and the number of instances seen so far ( count ). Y ou could
just create variables manually if you preferred. Keras tracks any tf.Variable  that
is set as an attribute (and more generally, any “trackable” object, such as layers or
models).•
The update_state()  method is called when you use an instance of this class as a
function (as we did with the Precision  object). It updates the variables given the
labels and predictions for one batch (and sample weights, but in this case we just
ignore them).•
The result()  method computes and returns the final result, in this case just the

models).•
mean Huber metric over all instances. When you use the metric as a function, the
update_state()  method gets called first, then the result()  method is called,
and its output is returned.•
We also implement the get_config()  method to ensure the threshold  gets
saved along with the model.•
The default implementation of the reset_states()  method just resets all vari‐
ables to 0.0 (but you can override it if needed).
Keras will take care of variable persistence seamlessly, no action is
required.
When you define a metric using a simple function, Keras automatically calls it for

required.
each batch, and it keeps track of the mean during each epoch, just like we did man‐
ually. So the only benefit of our HuberMetric  class is that the threshold  will be saved.
But of course, some metrics, like precision, cannot simply be averaged over batches:
in thoses cases, there’s no other option than to implement a streaming metric.
Now that we have built a streaming metric, building a custom layer will seem like a
walk in the park!382 | Chapter 12: Custom Models and Training with TensorFlow

Custom Layers
Y ou may occasionally want to build an architecture that contains an exotic layer for
which TensorFlow does not provide a default implementation. In this case, you will
need to create a custom layer. Or sometimes you may simply want to build a very
repetitive architecture, containing identical blocks of layers repeated many times, and
it would be convenient to treat each block of layers as a single layer. For example, if
the model is a sequence of layers A, B, C, A, B, C, A, B, C, then you might want to
define a custom layer D containing layers A, B, C, and your model would then simply
be D, D, D. Let’s see how to build custom layers.
First, some layers have no weights, such as keras.layers.Flatten  or keras.lay
ers.ReLU . If you want to create a custom layer without any weights, the simplest
option is to write a function and wrap it in a keras.layers.Lambda  layer. For exam‐
ple, the following layer will apply the exponential function to its inputs:

exponential_layer  = keras.layers.Lambda(lambda x: tf.exp(x))
This custom layer can then be used like any other layer, using the sequential API, the
functional API, or the subclassing API. Y ou can also use it as an activation function
(or you could just use activation=tf.exp , or activation=keras.activations.expo
nential , or simply activation="exponential" ). The exponential layer is sometimes
used in the output layer of a regression model when the values to predict have very
different scales (e.g., 0.001, 10., 1000.).

As you probably guessed by now, to build a custom stateful layer (i.e., a layer with
weights), you need to create a subclass of the keras.layers.Layer  class. For exam‐
ple, the following class implements a simplified version of the Dense  layer:
class MyDense(keras.layers.Layer):
    def __init__ (self, units, activation =None, **kwargs):
        self.units = units
        self.activation  = keras.activations .get(activation )
    def build(self, batch_input_shape ):
        self.kernel = self.add_weight (

            name="kernel" , shape=[batch_input_shape [-1], self.units],
            initializer ="glorot_normal" )
        self.bias = self.add_weight (
            name="bias", shape=[self.units], initializer ="zeros")
        super().build(batch_input_shape ) # must be at the end
    def call(self, X):
        return self.activation (X @ self.kernel + self.bias)
    def compute_output_shape (self, batch_input_shape ):
        return tf.TensorShape (batch_input_shape .as_list()[:-1] + [self.units])Customizing Models and Training Algorithms | 383

8This function is specific to tf.keras. Y ou could use keras.activations.Activation  instead.9
The Keras API calls this argument input_shape , but since it also includes the batch dimension, I prefer to call
it batch_input_shape . Same for compute_output_shape() .    def get_config (self):
        return {**base_config , "units": self.units,
                "activation" : keras.activations .serialize (self.activation )}
The constructor takes all the hyperparameters as arguments (in this example just
units  and activation ), and importantly it also takes a **kwargs  argument. It
calls the parent constructor, passing it the kwargs : this takes care of standard
arguments such as input_shape , trainable , name , and so on. Then it saves the
hyperparameters as attributes, converting the activation  argument to the
appropriate activation function using the keras.activations.get()  function (it

accepts functions, standard strings like "relu"  or "selu" , or simply None )8.•
The build()  method’s role is to create the layer’s variables, by calling the
add_weight()  method for each weight. The build()  method is called the first
time the layer is used. At that point, Keras will know the shape of this layer’s
inputs, and it will pass it to the build()  method9, which is often necessary to cre‐
ate some of the weights. For example, we need to know the number of neurons in

the previous layer in order to create the connection weights matrix (i.e., the "ker
nel" ): this corresponds to the size of the last dimension of the inputs. At the end
of the build()  method (and only at the end), you must call the parent’s build()
method: this tells Keras that the layer is built (it just sets self.built = True ).•
The call()  method actually performs the desired operations. In this case, we
compute the matrix multiplication of the inputs X and the layer’s kernel, we add

the bias vector, we apply the activation function to the result, and this gives us the
output of the layer.•
The compute_output_shape()  method simply returns the shape of this layer’s
outputs. In this case, it is the same shape as the inputs, except the last dimension
is replaced with the number of neurons in the layer. Note that in tf.keras, shapes
are instances of the tf.TensorShape  class, which you can convert to Python lists
using as_list() .•

The get_config()  method is just like earlier. Note that we save the activation
function’s full configuration by calling keras.activations.serialize() .
Y ou can now use a MyDense  layer just like any other layer!384 | Chapter 12: Custom Models and Training with TensorFlow

Y ou can generally omit the compute_output_shape()  method, as
tf.keras automatically infers the output shape, except when the
layer is dynamic (as we will see shortly). In other Keras implemen‐
tations, this method is either required or by default it assumes the
output shape is the same as the input shape.
To create a layer with multiple inputs (e.g., Concatenate ), the argument to the call()
method should be a tuple containing all the inputs, and similarly the argument to the
compute_output_shape()  method should be a tuple containing each input’s batch
shape. To create a layer with multiple outputs, the call()  method should return the
list of outputs, and the compute_output_shape()  should return the list of batch out‐
put shapes (one per output). For example, the following toy layer takes two inputs
and returns three outputs:
class MyMultiLayer (keras.layers.Layer):
        X1, X2 = X
        return [X1 + X2, X1 * X2, X1 / X2]

        X1, X2 = X
        b1, b2 = batch_input_shape
        return [b1, b1, b1] # should probably handle broadcasting rules
This layer may now be used like any other layer, but of course only using the func‐
tional and subclassing APIs, not the sequential API (which only accepts layers with
one input and one output).
If your layer needs to have a different behavior during training and during testing
(e.g., if it uses Dropout  or BatchNormalization  layers), then you must add a train

b1, b2 = batch_input_shape
ing argument to the call()  method and use this argument to decide what to do. For
example, let’s create a layer that adds Gaussian noise during training (for regulariza‐
tion), but does nothing during testing (Keras actually has a layer that does the same
thing: keras.layers.GaussianNoise ):
class MyGaussianNoise (keras.layers.Layer):
    def __init__ (self, stddev, **kwargs):
        self.stddev = stddev
    def call(self, X, training =None):
        if training :

            noise = tf.random.normal(tf.shape(X), stddev=self.stddev)
            return X + noise
        else:
            return X
        return batch_input_shapeCustomizing Models and Training Algorithms | 385

10The name “subclassing API” usually refers only to the creation of custom models by subclassing, although
many other things can be created by subclassing, as we saw in this chapter.With that, you can now build any custom layer you need! Now let’s create custom
models.
Custom Models
We already looked at custom model classes in Chapter 10  when we discussed the sub‐
classing API.10 It is actually quite straightforward, just subclass the keras.mod
els.Model  class, create layers and variables in the constructor, and implement the
call()  method to do whatever you want the model to do. For example, suppose you
want to build the model represented in Figure 12-3 :
Figure 12-3. Custom Model Example
The inputs go through a first dense layer, then through a residual block  composed of
two dense layers and an addition operation (as we will see in Chapter 14 , a residual
block adds its inputs to its outputs), then through this same residual block 3 more

times, then through a second residual block, and the final result goes through a dense
output layer. Note that this model does not make much sense, it’s just an example to
illustrate the fact that you can easily build any kind of model you want, even contain‐386 | Chapter 12: Custom Models and Training with TensorFlow

ing loops and skip connections. To implement this model, it is best to first create a
ResidualBlock  layer, since we are going to create a couple identical blocks (and we
might want to reuse it in another model):
class ResidualBlock (keras.layers.Layer):
    def __init__ (self, n_layers , n_neurons , **kwargs):
        self.hidden = [keras.layers.Dense(n_neurons , activation ="elu",
                                          kernel_initializer ="he_normal" )
                       for _ in range(n_layers )]
        Z = inputs
        for layer in self.hidden:
            Z = layer(Z)
        return inputs + Z
This layer is a bit special since it contains other layers. This is handled transparently
by Keras: it automatically detects that the hidden  attribute contains trackable objects
(layers in this case), so their variables are automatically added to this layer’s list of

for _ in range(n_layers )]
        Z = inputs
variables. The rest of this class is self-explanatory. Next, let’s use the subclassing API
to define the model itself:
class ResidualRegressor (keras.models.Model):
    def __init__ (self, output_dim , **kwargs):
        self.hidden1 = keras.layers.Dense(30, activation ="elu",
        self.block1 = ResidualBlock (2, 30)
        self.block2 = ResidualBlock (2, 30)
        self.out = keras.layers.Dense(output_dim )

        Z = self.hidden1(inputs)
        for _ in range(1 + 3):
            Z = self.block1(Z)
        Z = self.block2(Z)
        return self.out(Z)
We create the layers in the constructor, and use them in the call()  method. This
model can then be used like any other model (compile it, fit it, evaluate it and use it to
make predictions). If you also want to be able to save the model using the save()
method, and load it using the keras.models.load_model()  function, you must

def call(self, inputs):
implement the get_config()  method (as we did earlier) in both the ResidualBlock
class and the ResidualRegressor  class. Alternatively, you can just save and load the
weights using the save_weights()  and load_weights()  methods.
The Model  class is actually a subclass of the Layer  class, so models can be defined and
used exactly like layers. But a model also has some extra functionalities, including of

course its compile() , fit() , evaluate()  and predict()  methods (and a few var‐Customizing Models and Training Algorithms | 387

iants, such as train_on_batch()  or fit_generator() ), plus the get_layers()
method (which can return any of the model’s layers by name or by index), and the
save()  method (and support for keras.models.load_model()  and keras.mod
els.clone_model() ). So if models provide more functionalities than layers, why not
just define every layer as a model? Well, technically you could, but it is probably
cleaner to distinguish the internal components of your model (layers or reusable
blocks of layers) from the model itself. The former should subclass the Layer  class,
while the latter should subclass the Model  class.
With that, you can quite naturally and concisely build almost any model that you find
in a paper, either using the sequential API, the functional API, the subclassing API, or
even a mix of these. “ Almost” any model? Y es, there are still a couple things that we
need to look at: first, how to define losses or metrics based on model internals, and
second how to build a custom training loop.

Losses and Metrics Based on Model Internals
The custom losses and metrics we defined earlier were all based on the labels and the
predictions (and optionally sample weights). However, you will occasionally want to
define losses based on other parts of your model, such as the weights or activations of
its hidden layers. This may be useful for regularization purposes, or to monitor some
internal aspect of your model.
To define a custom loss based on model internals, just compute it based on any part

of the model you want, then pass the result to the add_loss()  method. For example,
the following custom model represents a standard MLP regressor with 5 hidden lay‐
ers, except it also implements a reconstruction loss  (see ???): we add an extra Dense
layer on top of the last hidden layer, and its role is to try to reconstruct the inputs of
the model. Since the reconstruction must have the same shape as the model’s inputs,
we need to create this Dense  layer in the build()  method to have access to the shape

of the inputs. In the call()  method, we compute both the regular output of the MLP ,
plus the output of the reconstruction layer. We then compute the mean squared dif‐
ference between the reconstructions and the inputs, and we add this value (times
0.05) to the model’s list of losses by calling add_loss() . During training, Keras will
add this loss to the main loss (which is why we scaled down the reconstruction loss,
to ensure the main loss dominates). As a result, the model will be forced to preserve
as much information as possible through the hidden layers, even information that is

not directly useful for the regression task itself. In practice, this loss sometimes
improves generalization; it is a regularization loss:
class ReconstructingRegressor (keras.models.Model):
        self.hidden = [keras.layers.Dense(30, activation ="selu",
                                          kernel_initializer ="lecun_normal" )388 | Chapter 12: Custom Models and Training with TensorFlow

for _ in range(5)]
        n_inputs  = batch_input_shape [-1]
        self.reconstruct  = keras.layers.Dense(n_inputs )
        super().build(batch_input_shape )
        Z = inputs
        reconstruction  = self.reconstruct (Z)
        recon_loss  = tf.reduce_mean (tf.square(reconstruction  - inputs))
        self.add_loss (0.05 * recon_loss )
Similarly, you can add a custom metric based on model internals by computing it in
any way you want, as long at the result is the output of a metric object. For example,
you can create a keras.metrics.Mean()  object in the constructor, then call it in the
call()  method, passing it the recon_loss , and finally add it to the model by calling
the model’s add_metric()  method. This way, when you train the model, Keras will

self.add_loss (0.05 * recon_loss )
display both the mean loss over each epoch (the loss is the sum of the main loss plus
0.05 times the reconstruction loss) and the mean reconstruction error over each
epoch. Both will go down during training:
Epoch 1/5
11610/11610 [=============] [...] loss: 4.3092 - reconstruction_error: 1.7360
Epoch 2/5
11610/11610 [=============] [...] loss: 1.1232 - reconstruction_error: 0.8964
[...]
In over 99% of the cases, everything we have discussed so far will be sufficient to

Epoch 1/5
Epoch 2/5
[...]
implement whatever model you want to build, even with complex architectures, los‐
ses, metrics, and so on. However, in some rare cases you may need to customize the
training loop itself. However, before we get there, we need to look at how to compute
gradients automatically in TensorFlow.
Computing Gradients Using Autodiff
To understand how to use autodiff (see Chapter 10  and ???) to compute gradients
automatically, let’s consider a simple toy function:
def f(w1, w2):
    return 3 * w1 ** 2 + 2 * w1 * w2

If you know calculus, you can analytically find that the partial derivative of this func‐
tion with regards to w1 is 6 * w1  + 2 * w2 . Y ou can also find that its partial derivative
with regards to w2 is 2 * w1 . For example, at the point (w1, w2)  = (5, 3) , these par‐Customizing Models and Training Algorithms | 389

tial derivatives are equal to 36 and 10, respectively, so the gradient vector at this point
is (36, 10). But if this were a neural network, the function would be much more com‐
plex, typically with tens of thousands of parameters, and finding the partial deriva‐
tives analytically by hand would be an almost impossible task. One solution could be
to compute an approximation of each partial derivative by measuring how much the
function’s output changes when you tweak the corresponding parameter:
>>> w1, w2 = 5, 3
>>> eps = 1e-6
>>> (f(w1 + eps, w2) - f(w1, w2)) / eps
36.000003007075065
>>> (f(w1, w2 + eps) - f(w1, w2)) / eps
10.000000003174137
Looks about right! This works rather well and it is trivial to implement, but it is just
an approximation, and importantly you need to call f() at least once per parameter
(not twice, since we could compute f(w1, w2)  just once). This makes this approach
intractable for large neural networks. So instead we should use autodiff (see Chap‐

ter 10  and ???). TensorFlow makes this pretty simple:
w1, w2 = tf.Variable (5.), tf.Variable (3.)
with tf.GradientTape () as tape:
    z = f(w1, w2)
gradients  = tape.gradient (z, [w1, w2])
We first define two variables w1 and w2, then we create a tf.GradientTape  context
that will automatically record every operation that involves a variable, and finally we
ask this tape to compute the gradients of the result z with regards to both variables
[w1, w2] . Let’s take a look at the gradients that TensorFlow computed:
>>> gradients

[<tf.Tensor: id=828234, shape=(), dtype=float32, numpy=36.0>,
 <tf.Tensor: id=828229, shape=(), dtype=float32, numpy=10.0>]
Perfect! Not only is the result accurate (the precision is only limited by the floating
point errors), but the gradient()  method only goes through the recorded computa‐
tions once (in reverse order), no matter how many variables there are, so it is incredi‐
bly efficient. It’s like magic!
Only put the strict minimum inside the tf.GradientTape()  block,

to save memory. Alternatively, you can pause recording by creating
a with tape.stop_recording()  block inside the tf.Gradient
Tape()  block.
The tape is automatically erased immediately after you call its gradient()  method, so
you will get an exception if you try to call gradient()  twice:390 | Chapter 12: Custom Models and Training with TensorFlow

dz_dw1 = tape.gradient (z, w1) # => tensor 36.0
dz_dw2 = tape.gradient (z, w2) # RuntimeError!
If you need to call gradient()  more than once, you must make the tape persistent,
and delete it when you are done with it to free resources:
with tf.GradientTape (persistent =True) as tape:
dz_dw2 = tape.gradient (z, w2) # => tensor 10.0, works fine now!
del tape
By default, the tape will only track operations involving variables, so if you try to
compute the gradient of z with regards to anything else than a variable, the result will
be None :
c1, c2 = tf.constant (5.), tf.constant (3.)
    z = f(c1, c2)
gradients  = tape.gradient (z, [c1, c2]) # returns [None, None]
However, you can force the tape to watch any tensors you like, to record every opera‐
tion that involves them. Y ou can then compute gradients with regards to these ten‐
sors, as if they were variables:

be None :
    tape.watch(c1)
    tape.watch(c2)
gradients  = tape.gradient (z, [c1, c2]) # returns [tensor 36., tensor 10.]
This can be useful in some cases, for example if you want to implement a regulariza‐
tion loss that penalizes activations that vary a lot when the inputs vary little: the loss
will be based on the gradient of the activations with regards to the inputs. Since the
inputs are not variables, you would need to tell the tape to watch them.

If you compute the gradient of a list of tensors (e.g., [z1, z2, z3] ) with regards to
some variables (e.g., [w1, w2] ), TensorFlow actually efficiently computes the sum of
the gradients of these tensors (i.e., gradient(z1, [w1, w2]) , plus gradient(z2,
[w1, w2]) , plus gradient(z3, [w1, w2]) ). Due to the way reverse-mode autodiff
works, it is not possible to compute the individual gradients ( z1, z2 and z3) without
actually calling gradient()  multiple times (once for z1, once for z2 and once for z3),

which requires making the tape persistent (and deleting it afterwards).Customizing Models and Training Algorithms | 391

Moreover, it is actually possible to compute second order partial derivatives (the Hes‐
sians, i.e., the partial derivatives of the partial derivatives)! To do this, we need to
record the operations that are performed when computing the first-order partial
derivatives (the Jacobians): this requires a second tape. Here is how it works:
with tf.GradientTape (persistent =True) as hessian_tape :
    with tf.GradientTape () as jacobian_tape :
        z = f(w1, w2)
    jacobians  = jacobian_tape .gradient (z, [w1, w2])
hessians  = [hessian_tape .gradient (jacobian , [w1, w2])
            for jacobian  in jacobians ]
del hessian_tape
The inner tape is used to compute the Jacobians, as we did earlier. The outer tape is
used to compute the partial derivatives of each Jacobian. Since we need to call gradi
ent()  once for each Jacobian (or else we would get the sum of the partial derivatives
over all the Jabobians, as explained earlier), we need the outer tape to be persistent, so

we delete it at the end. The Jacobians are obviously the same as earlier (36 and 5), but
now we also have the Hessians:
>>> hessians  # dz_dw1_dw1, dz_dw1_dw2, dz_dw2_dw1, dz_dw2_dw2
[[<tf.Tensor: id=830578, shape=(), dtype=float32, numpy=6.0>,
  <tf.Tensor: id=830595, shape=(), dtype=float32, numpy=2.0>],
 [<tf.Tensor: id=830600, shape=(), dtype=float32, numpy=2.0>, None]]
Let’s verify these Hessians. The first two are the partial derivatives of 6 * w1  + 2 * w2
(which is, as we saw earlier, the partial derivative of f with regards to w1), with

regards to w1 and w2. The result is correct: 6 for w1 and 2 for w2. The next two are the
partial derivatives of 2 * w1  (the partial derivative of f with regards to w2), with
regards to w1 and w2, which are 2 for w1 and 0 for w2. Note that TensorFlow returns
None  instead of 0 since w2 does not appear at all in 2 * w1 . TensorFlow also returns
None  when you use an operation whose gradients are not defined (e.g., tf.argmax() ).
In some rare cases you may want to stop gradients from backpropagating through

some part of your neural network. To do this, you must use the tf.stop_gradient()
function: it just returns its inputs during the forward pass (like tf.identity() ), but
it does not let gradients through during backpropagation (it acts like a constant). For
example:
    return 3 * w1 ** 2 + tf.stop_gradient (2 * w1 * w2)
    z = f(w1, w2) # same result as without stop_gradient()

example:
gradients  = tape.gradient (z, [w1, w2]) # => returns [tensor 30., None]392 | Chapter 12: Custom Models and Training with TensorFlow

Finally, you may occasionally run into some numerical issues when computing gradi‐
ents. For example, if you compute the gradients of the my_softplus()  function for
large inputs, the result will be NaN:
>>> x = tf.Variable ([100.])
>>> with tf.GradientTape () as tape:
...     z = my_softplus (x)
...
>>> tape.gradient (z, [x])
<tf.Tensor: [...] numpy=array([nan], dtype=float32)>
This is because computing the gradients of this function using autodiff leads to some
numerical difficulties: due to floating point precision errors, autodiff ends up com‐
puting infinity divided by infinity (which returns NaN). Fortunately, we can analyti‐
cally find that the derivative of the softplus function is just 1 / (1 + 1 / exp(x)), which
is numerically stable. Next, we can tell TensorFlow to use this stable function when
computing the gradients of the my_softplus()  function, by decorating it with
@tf.custom_gradient , and making it return both its normal output and the function

that computes the derivatives (note that it will receive as input the gradients that were
backpropagated so far, down to the softplus function, and according to the chain rule
we should multiply them with this function’s gradients):
@tf.custom_gradient
def my_better_softplus (z):
    exp = tf.exp(z)
    def my_softplus_gradients (grad):
        return grad / (1 + 1 / exp)
    return tf.math.log(exp + 1), my_softplus_gradients
Now when we compute the gradients of the my_better_softplus()  function, we get

the proper result, even for large input values (however, the main output still explodes
because of the exponential: one workaround is to use tf.where()  to just return the
inputs when they are large).
Congratulations! Y ou can now compute the gradients of any function (provided it is
differentiable at the point where you compute it), you can even compute Hessians,
block backpropagation when needed and even write your own gradient functions!

This is probably more flexibility than you will ever need, even if you build your own
custom training loops, as we will see now.
Custom Training Loops
In some rare cases, the fit()  method may not be flexible enough for what you need
to do. For example, the Wide and Deep paper we discussed in Chapter 10  actually
uses two different optimizers: one for the wide path and the other for the deep path.
Since the fit()  method only uses one optimizer (the one that we specify whenCustomizing Models and Training Algorithms | 393

compiling the model), implementing this paper requires writing your own custom
loop.
Y ou may also like to write your own custom training loops simply to feel more confi‐
dent that it does precisely what you intent it to do (perhaps you are unsure about
some details of the fit()  method). It can sometimes feel safer to make everything
explicit. However, remember that writing a custom training loop will make your code
longer, more error prone and harder to maintain.
Unless you really need the extra flexibility, you should prefer using
the fit()  method rather than implementing your own training
loop, especially if you work in a team.
First, let’s build a simple model. No need to compile it, since we will handle the train‐
ing loop manually:
l2_reg = keras.regularizers .l2(0.05)
    keras.layers.Dense(30, activation ="elu", kernel_initializer ="he_normal" ,
                       kernel_regularizer =l2_reg),
    keras.layers.Dense(1, kernel_regularizer =l2_reg)
])

])
Next, let’s create a tiny function that will randomly sample a batch of instances from
the training set (in Chapter 13  we will discuss the Data API, which offers a much bet‐
ter alternative):
def random_batch (X, y, batch_size =32):
    idx = np.random.randint(len(X), size=batch_size )
    return X[idx], y[idx]
Let’s also define a function that will display the training status, including the number
of steps, the total number of steps, the mean loss since the start of the epoch (i.e., we

])
will use the Mean  metric to compute it), and other metrics:
def print_status_bar (iteration , total, loss, metrics=None):
    metrics = " - ".join(["{}: {:.4f}" .format(m.name, m.result())
                         for m in [loss] + (metrics or [])])
    end = "" if iteration  < total else "\n"
    print("\r{}/{} - " .format(iteration , total) + metrics,
          end=end)
This code is self-explanatory, unless you are unfamiliar with Python string format‐

          end=end)
ting: {:.4f}  will format a float with 4 digits after the decimal point. Moreover, using
\r (carriage return) along with end=""  ensures that the status bar always gets printed
on the same line. In the notebook, the print_status_bar()  function also includes a
progress bar, but you could use the handy tqdm library instead.394 | Chapter 12: Custom Models and Training with TensorFlow

With that, let’s get down to business! First, we need to define some hyperparameters,
choose the optimizer, the loss function and the metrics (just the MAE in this exam‐
ple):
n_epochs  = 5
batch_size  = 32
n_steps = len(X_train) // batch_size
optimizer  = keras.optimizers .Nadam(lr=0.01)
loss_fn = keras.losses.mean_squared_error
mean_loss  = keras.metrics.Mean()
metrics = [keras.metrics.MeanAbsoluteError ()]
And now we are ready to build the custom loop!
for epoch in range(1, n_epochs  + 1):
    print("Epoch {}/{}" .format(epoch, n_epochs ))
    for step in range(1, n_steps + 1):
        X_batch, y_batch = random_batch (X_train_scaled , y_train)
        with tf.GradientTape () as tape:
            y_pred = model(X_batch, training =True)
            main_loss  = tf.reduce_mean (loss_fn(y_batch, y_pred))
            loss = tf.add_n([main_loss ] + model.losses)
        gradients  = tape.gradient (loss, model.trainable_variables )
        optimizer .apply_gradients (zip(gradients , model.trainable_variables ))

for step in range(1, n_steps + 1):
        mean_loss (loss)
        for metric in metrics:
            metric(y_batch, y_pred)
        print_status_bar (step * batch_size , len(y_train), mean_loss , metrics)
    print_status_bar (len(y_train), len(y_train), mean_loss , metrics)
    for metric in [mean_loss ] + metrics:
        metric.reset_states ()
There’s a lot going on in this code, so let’s walk through it:•
We create two nested loops: one for the epochs, the other for the batches within
an epoch.•
Then we sample a random batch from the training set.•

for metric in metrics:
an epoch.•
Inside the tf.GradientTape()  block, we make a prediction for one batch (using
the model as a function), and we compute the loss: it is equal to the main loss
plus the other losses (in this model, there is one regularization loss per layer).
Since the mean_squared_error()  function returns one loss per instance, we
compute the mean over the batch using tf.reduce_mean()  (if you wanted to
apply different weights to each instance, this is where you would do it). The regu‐

larization losses are already reduced to a single scalar each, so we just need to
sum them (using tf.add_n() , which sums multiple tensors of the same shape
and data type).Customizing Models and Training Algorithms | 395

11The truth is we did not process every single instance in the training set because we sampled instances ran‐
domly, so some were processed more than once while others were not processed at all. In practice that’s fine.
Moreover, if the training set size is not a multiple of the batch size, we will miss a few instances.12
Alternatively, check out K.learning_phase() , K.set_learning_phase()  and K.learning_phase_scope() .13
With the exception of optimizers, as very few people ever customize these: see the notebook for an example.• Next, we ask the tape  to compute the gradient of the loss with regards to each
trainable variable ( not all variables!), and we apply them to the optimizer to per‐
form a Gradient Descent step.•
Next we update the mean loss and the metrics (over the current epoch), and we
display the status bar.•
At the end of each epoch, we display the status bar again to make it look com‐
plete11 and to print a line feed, and we reset the states of the mean loss and the
metrics.

metrics.
If you set the optimizer’s clipnorm  or clipvalue  hyperparameters, it will take care of
this for you. If you want to apply any other transformation to the gradients, simply do
so before calling the apply_gradients()  method.
If you add weight constraints to your model (e.g., by setting kernel_constraint  or
bias_constraint  when creating a layer), you should update the training loop to
apply these constraints just after apply_gradients() :
for variable  in model.variables :
    if variable .constraint  is not None:
        variable .assign(variable .constraint (variable ))

Most importantly, this training loop does not handle layers that behave differently
during training and testing (e.g., BatchNormalization  or Dropout ). To handle these,
you need to call the model with training=True  and make sure it propagates this to
every layer that needs it.12
As you can see, there are quite a lot of things you need to get right, it is easy to make a
mistake. But on the bright side, you get full control, so it’s your call.
Now that you know how to customize any part of your models13 and training algo‐

rithms, let’s see how you can use TensorFlow’s automatic graph generation feature: it
can speed up your custom code considerably, and it will also make it portable to any
platform supported by TensorFlow (see ???).
TensorFlow Functions and Graphs
In TensorFlow 1, graphs were unavoidable (as were the complexities that came with
them): they were a central part of TensorFlow’s API. In TensorFlow 2, they are still396 | Chapter 12: Custom Models and Training with TensorFlow

there, but not as central, and much (much!) simpler to use. To demonstrate this, let’s
start with a trivial function that just computes the cube of its input:
def cube(x):
    return x ** 3
We can obviously call this function with a Python value, such as an int or a float, or
we can call it with a tensor:
>>> cube(2)
8
>>> cube(tf.constant (2.0))
<tf.Tensor: id=18634148, shape=(), dtype=float32, numpy=8.0>
Now, let’s use tf.function()  to convert this Python function to a TensorFlow Func‐
tion:
>>> tf_cube = tf.function (cube)
>>> tf_cube
<tensorflow.python.eager.def_function.Function at 0x1546fc080>
This TF Function can then be used exactly like the original Python function, and it
will return the same result (but as tensors):
>>> tf_cube(2)
<tf.Tensor: id=18634201, shape=(), dtype=int32, numpy=8>
>>> tf_cube(tf.constant (2.0))
<tf.Tensor: id=18634211, shape=(), dtype=float32, numpy=8.0>
Under the hood, tf.function()  analyzed the computations performed by the cube()

tion:
function and generated an equivalent computation graph! As you can see, it was
rather painless (we will see how this works shortly). Alternatively, we could have used
tf.function  as a decorator; this is actually more common:
@tf.function
def tf_cube(x):
The original Python function is still available via the TF Function’s python_function
attribute, in case you ever need it:
>>> tf_cube.python_function (2)
8
TensorFlow optimizes the computation graph, pruning unused nodes, simplifying

8
expressions (e.g., 1 + 2 would get replaced with 3), and more. Once the optimized
graph is ready, the TF Function efficiently executes the operations in the graph, in the
appropriate order (and in parallel when it can). As a result, a TF Function will usually
run much faster than the original Python function, especially if it performs complexTensorFlow Functions and Graphs | 397

14However, in this trivial example, the computation graph is so small that there is nothing at all to optimize, so
tf_cube()  actually runs much slower than cube() .
computations.14 Most of the time you will not really need to know more than that:
when you want to boost a Python function, just transform it into a TF Function.
That’s all!
Moreover, when you write a custom loss function, a custom metric, a custom layer or
any other custom function, and you use it in a Keras model (as we did throughout
this chapter), Keras automatically converts your function into a TF Function, no need
to use tf.function() . So most of the time, all this magic is 100% transparent.
Y ou can tell Keras not to convert your Python functions to TF
Functions by setting dynamic=True  when creating a custom layer
or a custom model. Alternatively, you can set run_eagerly=True
when calling the model’s compile()  method.
TF Function generates a new graph for every unique set of input shapes and data

types, and it caches it for subsequent calls. For example, if you call tf_cube(tf.con
stant(10)) , a graph will be generated for int32 tensors of shape []. Then if you call
tf_cube(tf.constant(20)) , the same graph will be reused. But if you then call
tf_cube(tf.constant([10, 20])) , a new graph will be generated for int32 tensors
of shape [2]. This is how TF Functions handle polymorphism (i.e., varying argument
types and shapes). However, this is only true for tensor arguments: if you pass numer‐

ical Python values to a TF Function, a new graph will be generated for every distinct
value: for example, calling tf_cube(10)  and tf_cube(20)  will generate two graphs.
If you call a TF Function many times with different numerical
Python values, then many graphs will be generated, slowing down
your program and using up a lot of RAM. Python values should be
reserved for arguments that will have few unique values, such as
hyperparameters like the number of neurons per layer. This allows

TensorFlow to better optimize each variant of your model.
Autograph and Tracing
So how does TensorFlow generate graphs? Well, first it starts by analyzing the Python
function’s source code to capture all the control flow statements, such as for loops
and while  loops, if statements, as well as break , continue  and return  statements.
This first step is called autograph . The reason TensorFlow has to analyze the source
code is that Python does not provide any other way to capture control flow state‐

ments: it offers magic methods like __add__()  or __mul__()  to capture operators like398 | Chapter 12: Custom Models and Training with TensorFlow

+ and *, but there are no __while__()  or __if__()  magic methods. After analyzing
the function’s code, autograph outputs an upgraded version of that function in which
all the control flow statements are replaced by the appropriate TensorFlow opera‐
tions, such as tf.while_loop()  for loops and tf.cond()  for if statements. For
example, in Figure 12-4 , autograph analyzes the source code of the sum_squares()
Python function, and it generates the tf__sum_squares()  function. In this function,
the for loop is replaced by the definition of the loop_body()  function (containing
the body of the original for loop), followed by a call to the for_stmt()  function. This
call will build the appropriate tf.while_loop()  operation in the computation graph.
Figure 12-4. How TensorFlow generates graphs using autograph and tracing
Next, TensorFlow calls this “upgraded” function, but instead of passing the actual
argument, it passes a symbolic tensor , meaning a tensor without any actual value, only

a name, a data type, and a shape. For example, if you call sum_squares(tf.con
stant(10)) , then the tf__sum_squares()  function will actually be called with a sym‐
bolic tensor of type int32 and shape []. The function will run in graph mode , meaning
that each TensorFlow operation will just add a node in the graph to represent itself
and its output tensor(s) (as opposed to the regular mode, called eager execution , or
eager mode ). In graph mode, TF operations do not perform any actual computations.

This should feel familiar if you know TensorFlow 1, as graph mode was the default
mode. In Figure 12-4 , you can see the tf__sum_squares()  function being called with
a symbolic tensor as argument (in this case, an int32 tensor of shape []), and the final
graph generated during tracing. The ellipses represent operations, and the arrows
represent tensors (both the generated function and the graph are simplified).TensorFlow Functions and Graphs | 399

To view the generated function’s source code, you can call tf.auto
graph.to_code(sum_squares.python_function) . The code is not
meant to be pretty, but it can sometimes help for debugging.
TF Function Rules
Most of the time, converting a Python function that performs TensorFlow operations
into a TF Function is trivial: just decorate it with @tf.function  or let Keras take care
of it for you. However, there are a few rules to respect:•
If you call any external library, including NumPy or even the standard library,
this call will run only during tracing, it will not be part of the graph. Indeed, a
TensorFlow graph can only include TensorFlow constructs (tensors, operations,
variables, datasets, and so on). So make sure you use tf.reduce_sum()  instead of
np.sum() , and tf.sort()  instead of the built-in sorted()  function, and so on
(unless you really want the code to run only during tracing).—
For example, if you define a TF function f(x)  that just returns np.ran

dom.rand() , a random number will only be generated when the function is
traced, so f(tf.constant(2.))  and f(tf.constant(3.))  will return the
same random number, but f(tf.constant([2., 3.]))  will return a different
one. If you replace np.random.rand()  with tf.random.uniform([]) , then a
new random number will be generated upon every call, since the operation
will be part of the graph.—
If your non-TensorFlow code has side-effects (such as logging something or
updating a Python counter), then you should not expect that side-effect to

occur every time you call the TF Function, as it will only occur when the func‐
tion is traced.—
Y ou can wrap arbitrary Python code in a tf.py_function()  operation, but
this will hinder performance, as TensorFlow will not be able to do any graph
optimization on this code, and it will also reduce portability, as the graph will
only run on platforms where Python is available (and the right libraries
installed).•
Y ou can call other Python functions or TF Functions, but they should follow the

same rules, as TensorFlow will also capture their operations in the computation
graph. Note that these other functions do not need to be decorated with
@tf.function .•
If the function creates a TensorFlow variable (or any other stateful TensorFlow
object, such as a dataset or a queue), it must do so upon the very first call, and
only then, or else you will get an exception. It is usually preferable to create vari‐

ables outside of the TF Function (e.g., in the build()  method of a custom layer).400 | Chapter 12: Custom Models and Training with TensorFlow

• The source code of your Python function should be available to TensorFlow. If
the source code is unavailable (for example, if you define your function in the
Python shell, which does not give access to the source code, or if you deploy only
the compiled Python files *.pyc  to production), then the graph generation pro‐
cess will fail or have limited functionality.•
TensorFlow will only capture for loops that iterate over a tensor or a Dataset . So
make sure you use for i in tf.range(10)  rather than for i in range(10) , or
else the loop will not be captured in the graph. Instead, it will run during tracing.
This may be what you want, if the for loop is meant to build the graph, for exam‐
ple to create each layer in a neural network.•
And as always, for performance reasons, you should prefer a vectorized imple‐
mentation whenever you can, rather than using loops.
It’s time to sum up! In this chapter we started with a brief overview of TensorFlow,

then we looked at TensorFlow’s low-level API, including tensors, operations, variables
and special data structures. We then used these tools to customize almost every com‐
ponent in tf.keras. Finally, we looked at how TF Functions can boost performance,
how graphs are generated using autograph and tracing, and what rules to follow when
you write TF Functions (if you would like to open the black box a bit further, for
example to explore the generated graphs, you will find further technical details
in ???).

in ???).
In the next chapter, we will look at how to efficiently load and preprocess data with
TensorFlow.TensorFlow Functions and Graphs | 401

CHAPTER 13
Loading and Preprocessing Data with
release of these titles. The following will be Chapter 13 in the final
So far we have used only datasets that fit in memory, but Deep Learning systems are
often trained on very large datasets that will not fit in RAM. Ingesting a large dataset
and preprocessing it efficiently can be tricky to implement with other Deep Learning
libraries, but TensorFlow makes it easy thanks to the Data API : you just create a data‐
set object, tell it where to get the data, then transform it in any way you want, and
TensorFlow takes care of all the implementation details, such as multithreading,
queuing, batching, prefetching, and so on.
Off the shelf, the Data API can read from text files (such as CSV files), binary files

with fixed-size records, and binary files that use TensorFlow’s TFRecord format,
which supports records of varying sizes. TFRecord is a flexible and efficient binary
format based on Protocol Buffers (an open source binary format). The Data API also
has support for reading from SQL databases. Moreover, many Open Source exten‐
sions are available to read from all sorts of data sources, such as Google’s BigQuery
service.
However, reading huge datasets efficiently is not the only difficulty: the data also

service.
needs to be preprocessed. Indeed, it is not always composed strictly of convenient
numerical fields: sometimes there will be text features, categorical features, and so on.
To handle this, TensorFlow provides the Features API : it lets you easily convert these
features to numerical features that can be consumed by your neural network. For403

example, categorical features with a large number of categories (such as cities, or
words) can be encoded using embeddings  (as we will see, an embedding is a trainable
dense vector that represents a category).
Both the Data API and the Features API work seamlessly with
tf.keras.
In this chapter, we will cover the Data API, the TFRecord format and the Features
API in detail. We will also take a quick look at a few related projects from Tensor‐
Flow’s ecosystem:•
TF Transform ( tf.Transform ) makes it possible to write a single preprocessing
function that can be run both in batch mode on your full training set, before
training (to speed it up), and then exported to a TF Function and incorporated
into your trained model, so that once it is deployed in production, it can take
care of preprocessing new instances on the fly.•
TF Datasets (TFDS) provides a convenient function to download many common
datasets of all kinds, including large ones like ImageNet, and it provides conve‐

nient dataset objects to manipulate them using the Data API.
So let’s get started!
The Data API
The whole Data API revolves around the concept of a dataset : as you might suspect,
this represents a sequence of data items. Usually you will use datasets that gradually
read data from disk, but for simplicity let’s just create a dataset entirely in RAM using
tf.data.Dataset.from_tensor_slices() :
>>> X = tf.range(10)  # any data tensor
>>> dataset = tf.data.Dataset.from_tensor_slices (X)
>>> dataset
<TensorSliceDataset shapes: (), types: tf.int32>

The from_tensor_slices()  function takes a tensor and creates a tf.data.Dataset
whose elements are all the slices of X (along the first dimension), so this dataset con‐
tains 10 items: tensors 0, 1, 2, …, 9. In this case we would have obtained the same
dataset if we had used tf.data.Dataset.range(10) .
Y ou can simply iterate over a dataset’s items like this:
>>> for item in dataset:
...     print(item)404 | Chapter 13: Loading and Preprocessing Data with TensorFlow

...
tf.Tensor(0, shape=(), dtype=int32)
tf.Tensor(1, shape=(), dtype=int32)
tf.Tensor(2, shape=(), dtype=int32)
[...]
tf.Tensor(9, shape=(), dtype=int32)
Chaining Transformations
Once you have a dataset, you can apply all sorts of transformations to it by calling its
transformation methods. Each method returns a new dataset, so you can chain trans‐
formations like this (this chain is illustrated in Figure 13-1 ):
>>> dataset = dataset.repeat(3).batch(7)
...     print(item)
...
tf.Tensor([0 1 2 3 4 5 6], shape=(7,), dtype=int32)
tf.Tensor([7 8 9 0 1 2 3], shape=(7,), dtype=int32)
tf.Tensor([4 5 6 7 8 9 0], shape=(7,), dtype=int32)
tf.Tensor([1 2 3 4 5 6 7], shape=(7,), dtype=int32)
tf.Tensor([8 9], shape=(2,), dtype=int32)
Figure 13-1. Chaining Dataset Transformations
In this example, we first call the repeat()  method on the original dataset, and it
returns a new dataset that will repeat the items of the original dataset 3 times. Of

...
course, this will not copy the whole data in memory 3 times! In fact, if you call this
method with no arguments, the new dataset will repeat the source dataset forever.
Then we call the batch()  method on this new dataset, and again this creates a new
dataset. This one will group the items of the previous dataset in batches of 7 items.
Finally, we iterate over the items of this final dataset. As you can see, the batch()
method had to output a final batch of size 2 instead of 7, but you can call it with

drop_remainder=True  if you want it to drop this final batch so that all batches have
the exact same size.The Data API | 405

The dataset methods do not modify datasets, they create new ones,
so make sure to keep a reference to these new datasets (e.g., data
set = ... ), or else nothing will happen.
Y ou can also apply any transformation you want to the items by calling the map()
method. For example, this creates a new dataset with all items doubled:
>>> dataset = dataset.map(lambda x: x * 2) # Items: [0,2,4,6,8,10,12]
This function is the one you will call to apply any preprocessing you want to your
data. Sometimes, this will include computations that can be quite intensive, such as
reshaping or rotating an image, so you will usually want to spawn multiple threads to
speed things up: it’s as simple as setting the num_parallel_calls  argument.
While the map()  applies a transformation to each item, the apply()  method applies a
transformation to the dataset as a whole. For example, the following code “unbatches”
the dataset, by applying the unbatch()  function to the dataset (this function is cur‐

rently experimental, but it will most likely move to the core API in a future release).
Each item in the new dataset will be a single integer tensor instead of a batch of 7
integers:
>>> dataset = dataset.apply(tf.data.experimental .unbatch()) # Items: 0,2,4,...
It is also possible to simply filter the dataset using the filter()  method:
>>> dataset = dataset.filter(lambda x: x < 10) # Items: 0 2 4 6 8 0 2 4 6...
Y ou will often want to look at just a few items from a dataset. Y ou can use the take()

integers:
method for that:
>>> for item in dataset.take(3):
...
tf.Tensor(0, shape=(), dtype=int64)
tf.Tensor(2, shape=(), dtype=int64)
tf.Tensor(4, shape=(), dtype=int64)
Shuffling  the Data
As you know, Gradient Descent works best when the instances in the training set are
independent and identically distributed (see Chapter 4 ). A simple way to ensure this
is to shuffle the instances. For this, you can just use the shuffle()  method. It will

...
create a new dataset that will start by filling up a buffer with the first items of the
source dataset, then whenever it is asked for an item, it will pull one out randomly
from the buffer, and replace it with a fresh one from the source dataset, until it has
iterated entirely through the source dataset. At this point it continues to pull out
items randomly from the buffer until it is empty. Y ou must specify the buffer size, and406 | Chapter 13: Loading and Preprocessing Data with TensorFlow

1Imagine a sorted deck of cards on your left: suppose you just take the top 3 cards and shuffle them, then pick
one randomly and put it to your right, keeping the other 2 in your hands. Take another card on your left,
shuffle the 3 cards in your hands and pick one of them randomly, and put it on your right. When you are
done going through all the cards like this, you will have a deck of cards on your right: do you think it will be
perfectly shuffled?
it is important to make it large enough or else shuffling will not be very efficient.1
However, obviously do not exceed the amount of RAM you have, and even if you
have plenty of it, there’s no need to go well beyond the dataset’s size. Y ou can provide
a random seed if you want the same random order every time you run your program.
>>> dataset = tf.data.Dataset.range(10).repeat(3) # 0 to 9, three times
>>> dataset = dataset.shuffle(buffer_size =5, seed=42).batch(7)
...

...
tf.Tensor([0 2 3 6 7 9 4], shape=(7,), dtype=int64)
tf.Tensor([5 0 1 1 8 6 5], shape=(7,), dtype=int64)
tf.Tensor([4 8 7 1 2 3 0], shape=(7,), dtype=int64)
tf.Tensor([5 4 2 7 8 9 9], shape=(7,), dtype=int64)
tf.Tensor([3 6], shape=(2,), dtype=int64)
If you call repeat()  on a shuffled dataset, by default it will generate
a new order at every iteration. This is generally a good idea, but if
you prefer to reuse the same order at each iteration (e.g., for tests
or debugging), you can set reshuffle_each_iteration=False .

For a large dataset that does not fit in memory, this simple shuffling-buffer approach
may not be sufficient, since the buffer will be small compared to the dataset. One sol‐
ution is to shuffle the source data itself (for example, on Linux you can shuffle text
files using the shuf  command). This will definitely improve shuffling a lot! However,
even if the source data is shuffled, you will usually want to shuffle it some more, or
else the same order will be repeated at each epoch, and the model may end up being

biased (e.g., due to some spurious patterns present by chance in the source data’s
order). To shuffle the instances some more, a common approach is to split the source
data into multiple files, then read them in a random order during training. However,
instances located in the same file will still end up close to each other. To avoid this
you can pick multiple files randomly, and read them simultaneously, interleaving
their lines. Then on top of that you can add a shuffling buffer using the shuffle()

method. If all this sounds like a lot of work, don’t worry: the Data API actually makes
all this possible in just a few lines of code. Let’s see how to do this.The Data API | 407

Interleaving Lines From Multiple Files
First, let’s suppose that you loaded the California housing dataset, you shuffled it
(unless it was already shuffled), you split it into a training set, a validation set and a
test set, then you split each set into many CSV files that each look like this (each row
contains 8 input features plus the target median house value):
MedInc,HouseAge,AveRooms,AveBedrms,Popul,AveOccup,Lat,Long,MedianHouseValue
3.5214,15.0,3.0499,1.1065,1447.0,1.6059,37.63,-122.43,1.442
5.3275,5.0,6.4900,0.9910,3464.0,3.4433,33.69,-117.39,1.687
3.1,29.0,7.5423,1.5915,1328.0,2.2508,38.44,-122.98,1.621
[...]
Let’s also suppose train_filepaths  contains the list of file paths (and you also have
valid_filepaths  and test_filepaths ):
>>> train_filepaths
['datasets/housing/my_train_00.csv', 'datasets/housing/my_train_01.csv',...]
Now let’s create a dataset containing only these file paths:
filepath_dataset  = tf.data.Dataset.list_files (train_filepaths , seed=42)

[...]
By default, the list_files()  function returns a dataset that shuffles the file paths. In
general this is a good thing, but you can set shuffle=False  if you do not want that,
for some reason.
Next, we can call the interleave()  method to read from 5 files at a time and inter‐
leave their lines (skipping the first line of each file, which is the header row, using the
skip()  method):
n_readers  = 5
dataset = filepath_dataset .interleave (
    lambda filepath : tf.data.TextLineDataset (filepath ).skip(1),
    cycle_length =n_readers )

The interleave()  method will create a dataset that will pull 5 file paths from the
filepath_dataset , and for each one it will call the function we gave it (a lambda in
this example) to create a new dataset, in this case a TextLineDataset . It will then
cycle through these 5 datasets, reading one line at a time from each until all datasets
are out of items. Then it will get the next 5 file paths from the filepath_dataset , and
interleave them the same way, and so on until it runs out of file paths.

For interleaving to work best, it is preferable to have files of identi‐
cal length, or else the end of the longest files will not be interleaved.408 | Chapter 13: Loading and Preprocessing Data with TensorFlow

By default, interleave()  does not use parallelism, it just reads one line at a time
from each file, sequentially. However, if you want it to actually read files in parallel,
you can set the num_parallel_calls  argument to the number of threads you want.
Y ou can even set it to tf.data.experimental.AUTOTUNE  to make TensorFlow choose
the right number of threads dynamically based on the available CPU (however, this is
an experimental feature for now). Let’s look at what the dataset contains now:
>>> for line in dataset.take(5):
...     print(line.numpy())
...
b'4.2083,44.0,5.3232,0.9171,846.0,2.3370,37.47,-122.2,2.782'
b'4.1812,52.0,5.7013,0.9965,692.0,2.4027,33.73,-118.31,3.215'
b'3.6875,44.0,4.5244,0.9930,457.0,3.1958,34.04,-118.15,1.625'
b'3.3456,37.0,4.5140,0.9084,458.0,3.2253,36.67,-121.7,2.526'
b'3.5214,15.0,3.0499,1.1065,1447.0,1.6059,37.63,-122.43,1.442'
These are the first rows (ignoring the header row) of 5 CSV files, chosen randomly.

...
Looks good! But as you can see, these are just byte strings, we need to parse them,
and also scale the data.
Preprocessing the Data
Let’s implement a small function that will perform this preprocessing:
X_mean, X_std = [...] # mean and scale of each feature in the training set
n_inputs  = 8
def preprocess (line):
  defs = [0.] * n_inputs  + [tf.constant ([], dtype=tf.float32)]
  fields = tf.io.decode_csv (line, record_defaults =defs)
  x = tf.stack(fields[:-1])
  y = tf.stack(fields[-1:])
  return (x - X_mean) / X_std, y

First, we assume that you have precomputed the mean and standard deviation of
each feature in the training set. X_mean  and X_std  are just 1D tensors (or NumPy
arrays) containing 8 floats, one per input feature.•
The preprocess()  function takes one CSV line, and starts by parsing it. For this,
it uses the tf.io.decode_csv()  function, which takes two arguments: the first is
the line to parse, and the second is an array containing the default value for each
column in the CSV file. This tells TensorFlow not only the default value for each

column, but also the number of columns and the type of each column. In this
example, we tell it that all feature columns are floats and missing values should
default to 0, but we provide an empty array of type tf.float32  as the default
value for the last column (the target): this tells TensorFlow that this column con‐The Data API | 409

tains floats, but that there is no default value, so it will raise an exception if it
encounters a missing value.•
The decode_csv()  function returns a list of scalar tensors (one per column) but
we need to return 1D tensor arrays. So we call tf.stack()  on all tensors except
for the last one (the target): this will stack these tensors into a 1D array. We then
do the same for the target value (this makes it a 1D tensor array with a single
value, rather than a scalar tensor).•
Finally, we scale the input features by subtracting the feature means and then
dividing by the feature standard deviations, and we return a tuple containing the
scaled features and the target.
Let’s test this preprocessing function:
>>> preprocess (b'4.2083,44.0,5.3232,0.9171,846.0,2.3370,37.47,-122.2,2.782' )
(<tf.Tensor: id=6227, shape=(8,), dtype=float32, numpy=
 array([ 0.16579159,  1.216324  , -0.05204564, -0.39215982, -0.5277444 ,
        -0.2633488 ,  0.8543046 , -1.3072058 ], dtype=float32)>,

 <tf.Tensor: [...], numpy=array([2.782], dtype=float32)>)
We can now apply this preprocessing function to the dataset.
Putting Everything Together
To make the code reusable, let’s put together everything we have discussed so far into
a small helper function: it will create and return a dataset that will efficiently load Cal‐
ifornia housing data from multiple CSV files, then shuffle it, preprocess it and batch it
(see Figure 13-2 ):
def csv_reader_dataset (filepaths , repeat=None, n_readers =5,

<tf.Tensor: [...], numpy=array([2.782], dtype=float32)>)
                       n_read_threads =None, shuffle_buffer_size =10000,
                       n_parse_threads =5, batch_size =32):
    dataset = tf.data.Dataset.list_files (filepaths ).repeat(repeat)
    dataset = dataset.interleave (
        lambda filepath : tf.data.TextLineDataset (filepath ).skip(1),
        cycle_length =n_readers , num_parallel_calls =n_read_threads )
    dataset = dataset.shuffle(shuffle_buffer_size )
    dataset = dataset.map(preprocess , num_parallel_calls =n_parse_threads )

n_read_threads =None, shuffle_buffer_size =10000,
    dataset = dataset.batch(batch_size )
    return dataset.prefetch (1)410 | Chapter 13: Loading and Preprocessing Data with TensorFlow

2In general, just prefetching one batch is fine, but in some cases you may need to prefetch a few more. Alterna‐
tively, you can let TensorFlow decide automatically by passing tf.data.experimental.AUTOTUNE  (this is an
experimental feature for now).
Figure 13-2. Loading and Preprocessing Data From Multiple CSV Files
Everything should make sense in this code, except the very last line ( prefetch(1) ),
which is actually quite important for performance.
Prefetching
By calling prefetch(1)  at the end, we are creating a dataset that will do its best to
always be one batch ahead2. In other words, while our training algorithm is working
on one batch, the dataset will already be working in parallel on getting the next batch
ready. This can improve performance dramatically, as is illustrated on Figure 13-3 . If
we also ensure that loading and preprocessing are multithreaded (by setting num_par
allel_calls  when calling interleave()  and map() ), we can exploit multiple cores

on the CPU and hopefully make preparing one batch of data shorter than running a
training step on the GPU: this way the GPU will be almost 100% utilized (except for
the data transfer time from the CPU to the GPU), and training will run much faster.The Data API | 411

Figure 13-3. Speedup Training Thanks  to Prefetching and Multithreading
If you plan to purchase a GPU card, its processing power and its
memory size are of course very important (in particular, a large
RAM is crucial for computer vision), but its memory bandwidth  is
just as important as the processing power to get good performance:
this is the number of gigabytes of data it can get in or out of its
RAM per second.
With that, you can now build efficient input pipelines to load and preprocess data
from multiple text files. We have discussed the most common dataset methods, but
there are a few more you may want to look at: concatenate() , zip() , window() ,
reduce() , cache() , shard() , flat_map()  and padded_batch() . There are also a cou‐
ple more class methods: from_generator()  and from_tensors() , which create a new
dataset from a Python generator or a list of tensors respectively. Please check the API
documentation for more details. Also note that there are experimental features avail‐

able in tf.data.experimental , many of which will most likely make it to the core
API in future releases (e.g., check out the CsvDataset  class and the SqlDataset
classes).412 | Chapter 13: Loading and Preprocessing Data with TensorFlow

3Support for datasets is specific to tf.keras, it will not work on other implementations of the Keras API.4
The number of steps per epoch is optional if the dataset just goes through the data once, but if you do not
specify it, the progress bar will not be displayed during the first epoch.5
Note that for now the dataset must be created within the TF Function. This may be fixed by the time you read
these lines (see TensorFlow issue #25414).Using the Dataset With tf.keras
Now we can use the csv_reader_dataset()  function to create a dataset for the train‐
ing set (ensuring it repeats the data forever), the validation set and the test set:
train_set  = csv_reader_dataset (train_filepaths , repeat=None)
valid_set  = csv_reader_dataset (valid_filepaths )
test_set  = csv_reader_dataset (test_filepaths )
And now we can simply build and train a Keras model using these datasets.3 All we
need to do is to call the fit()  method with the datasets instead of X_train  and

y_train , and specify the number of steps per epoch for each set:4
model = keras.models.Sequential ([...])
model.compile([...])
model.fit(train_set , steps_per_epoch =len(X_train) // batch_size , epochs=10,
          validation_data =valid_set ,
          validation_steps =len(X_valid) // batch_size )
Similarly, we can pass a dataset to the evaluate()  and predict()  methods (and again
specify the number of steps per epoch):
model.evaluate (test_set , steps=len(X_test) // batch_size )

model.predict(new_set, steps=len(X_new) // batch_size )
Unlike the other sets, the new_set  will usually not contain labels (if it does, Keras will
just ignore them). Note that in all these cases, you can still use NumPy arrays instead
of datasets if you want (but of course they need to have been loaded and preprocessed
first).
If you want to build your own custom training loop (as in Chapter 12 ), you can just
iterate over the training set, very naturally:
for X_batch, y_batch in train_set :

first).
    [...] # perform one gradient descent step
In fact, it is even possible to create a tf.function (see Chapter 12 ) that performs the
whole training loop!5
def train(model, optimizer , loss_fn, n_epochs , [...]):
    train_set  = csv_reader_dataset (train_filepaths , repeat=n_epochs , [...])
    for X_batch, y_batch in train_set :
        with tf.GradientTape () as tape:The Data API | 413

y_pred = model(X_batch)
        grads = tape.gradient (loss, model.trainable_variables )
        optimizer .apply_gradients (zip(grads, model.trainable_variables ))
Congratulations, you now know how to build powerful input pipelines using the Data
API! However, so far we have used CSV files, which are common, simple and conve‐
nient, but they are not really efficient, and they do not support large or complex data
structures very well, such as images or audio. So let’s use TFRecords instead.
If you are happy with CSV files (or whatever other format you are
using), you do not have  to use TFRecords. As the saying goes, if it
ain’t broke, don’t fix it! TFRecords are useful when the bottleneck
during training is loading and parsing the data.
The TFRecord Format
The TFRecord format is TensorFlow’s preferred format for storing large amounts of

data and reading it efficiently. It is a very simple binary format that just contains a
sequence of binary records of varying sizes (each record just has a length, a CRC
checksum to check that the length was not corrupted, then the actual data, and finally
a CRC checksum for the data). Y ou can easily create a TFRecord file using the
tf.io.TFRecordWriter  class:
with tf.io.TFRecordWriter ("my_data.tfrecord" ) as f:
    f.write(b"This is the first record" )
    f.write(b"And this is the second record" )
And you can then use a tf.data.TFRecordDataset  to read one or more TFRecord

files:
filepaths  = ["my_data.tfrecord" ]
dataset = tf.data.TFRecordDataset (filepaths )
for item in dataset:
    print(item)
This will output:
tf.Tensor(b'This is the first record', shape=(), dtype=string)
tf.Tensor(b'And this is the second record', shape=(), dtype=string)
By default, a TFRecordDataset  will read files one by one, but you
can make it read multiple files in parallel and interleave their
records by setting num_parallel_reads . Alternatively, you could

files:
obtain the same result by using list_files()  and interleave()
as we did earlier to read multiple CSV files.414 | Chapter 13: Loading and Preprocessing Data with TensorFlow

6Since protobuf objects are meant to be serialized and transmitted, they are called messages . Compressed TFRecord Files
It can sometimes be useful to compress your TFRecord files, especially if they need to
be loaded via a network connection. Y ou can create a compressed TFRecord file by
setting the options  argument:
options = tf.io.TFRecordOptions (compression_type ="GZIP")
with tf.io.TFRecordWriter ("my_compressed.tfrecord" , options) as f:
  [...]
When reading a compressed TFRecord file, you need to specify the compression type:
dataset = tf.data.TFRecordDataset (["my_compressed.tfrecord" ],
                                  compression_type ="GZIP")
A Brief Introduction to Protocol Buffers
Even though each record can use any binary format you want, TFRecord files usually
contain serialized Protocol Buffers (also called protobufs ). This is a portable, extensi‐
ble and efficient binary format developed at Google back in 2001 and Open Sourced

[...]
in 2008, and they are now widely used, in particular in gRPC , Google’s remote proce‐
dure call system. Protocol Buffers are defined using a simple language that looks like
this:
syntax = "proto3" ;
message Person {
  string name = 1;
  int32 id = 2;
  repeated  string email = 3;
}
This definition says we are using the protobuf format version 3, and it specifies that
each Person  object6 may (optionally) have a name  of type string , an id of type int32 ,

this:
}
and zero or more email  fields, each of type string . The numbers 1, 2 and 3 are the
field identifiers: they will be used in each record’s binary representation. Once you
have a definition in a .proto  file, you can compile it. This requires protoc , the proto‐
buf compiler, to generate access classes in Python (or some other language). Note that
the protobuf definitions we will use have already been compiled for you, and their
Python classes are part of TensorFlow, so you will not need to use protoc . All you

need to know is how to use protobuf access classes in Python. To illustrate the basics,
let’s look at a simple example that uses the access classes generated for the Person
protobuf (the code is explained in the comments):
>>> from person_pb2  import Person  # import the generated access class
>>> person = Person(name="Al", id=123, email=["a@b.com" ])  # create a Person
>>> print(person)  # display the PersonThe TFRecord Format | 415

7This chapter contains the bare minimum you need to know about protobufs to use TFRecords. To learn more
about protobufs, please visit https://homl.info/protobuf .name: "Al"
id: 123
email: "a@b.com"
>>> person.name  # read a field
"Al"
>>> person.name = "Alice"  # modify a field
>>> person.email[0]  # repeated fields can be accessed like arrays
"a@b.com"
>>> person.email.append("c@d.com" )  # add an email address
>>> s = person.SerializeToString ()  # serialize the object to a byte string
>>> s
b'\n\x05Alice\x10{\x1a\x07a@b.com\x1a\x07c@d.com'
>>> person2 = Person()  # create a new Person
>>> person2.ParseFromString (s)  # parse the byte string (27 bytes long)
27
>>> person == person2  # now they are equal
True
In short, we import the Person  class generated by protoc , we create an instance and
we play with it, visualizing it, reading and writing some fields, then we serialize it
using the SerializeToString()  method. This is the binary data that is ready to be

>>> s
27
True
saved or transmitted over the network. When reading or receiving this binary data,
we can parse it using the ParseFromString()  method, and we get a copy of the object
that was serialized.7
We could save the serialized Person  object to a TFRecord file, then we could load and
parse it: everything would work fine. However, SerializeToString()  and ParseFrom
String()  are not TensorFlow operations (and neither are the other operations in this
code), so they cannot be included in a TensorFlow Function (except by wrapping

them in a tf.py_function()  operation, which would make the code slower and less
portable, as we saw in Chapter 12 ). Fortunately, TensorFlow does include special pro‐
tobuf definitions for which it provides parsing operations.
TensorFlow Protobufs
The main protobuf typically used in a TFRecord file is the Example  protobuf, which
represents one instance in a dataset. It contains a list of named features, where each
feature can either be a list of byte strings, a list of floats or a list of integers. Here is the
protobuf definition:

message BytesList  { repeated  bytes value = 1; }
message FloatList  { repeated  float value = 1 [packed = true]; }
message Int64List  { repeated  int64 value = 1 [packed = true]; }416 | Chapter 13: Loading and Preprocessing Data with TensorFlow

8Why was Example  even defined since it contains no more than a Features  object? Well, TensorFlow may one
day decide to add more fields to it. As long as the new Example  definition still contains the features  field,
with the same id, it will be backward compatible. This extensibility is one of the great features of protobufs.message Feature {
    oneof kind {
        BytesList  bytes_list  = 1;
        FloatList  float_list  = 2;
        Int64List  int64_list  = 3;
    }
};
message Features  { map<string, Feature> feature = 1; };
message Example { Features  features  = 1; };
The definitions of BytesList , FloatList  and Int64List  are straightforward enough
([packed = true]  is used for repeated numerical fields, for a more efficient encod‐
ing). A Feature  either contains a BytesList , a FloatList  or an Int64List . A Fea
tures  (with an s) contains a dictionary that maps a feature name to the
corresponding feature value. And finally, an Example  just contains a Features  object.8

Here is how you could create a tf.train.Example  representing the same person as
earlier, and write it to TFRecord file:
from tensorflow.train  import BytesList , FloatList , Int64List
from tensorflow.train  import Feature, Features , Example
person_example  = Example(
    features =Features (
        feature={
            "name": Feature(bytes_list =BytesList (value=[b"Alice"])),
            "id": Feature(int64_list =Int64List (value=[123])),
            "emails" : Feature(bytes_list =BytesList (value=[b"a@b.com" ,

        feature={
                                                          b "c@d.com" ]))
        }))
The code is a bit verbose and repetitive, but it’s rather straightforward (and you could
easily wrap it inside a small helper function). Now that we have an Example  protobuf,
we can serialize it by calling its SerializeToString()  method, then write the result‐
ing data to a TFRecord file:
with tf.io.TFRecordWriter ("my_contacts.tfrecord" ) as f:
    f.write(person_example .SerializeToString ())
Normally you would write much more than just one example! Typically, you would

}))
create a conversion script that reads from your current format (say, CSV files), creates
an Example  protobuf for each instance, serializes them and saves them to several
TFRecord files, ideally shuffling them in the process. This requires a bit of work, so
once again make sure it is really necessary (perhaps your pipeline works fine with
CSV files).The TFRecord Format | 417

Now that we have a nice TFRecord file containing a serialized Example , let’s try to
load it.
Loading and Parsing Examples
To load the serialized Example  protobufs, we will use a tf.data.TFRecordDataset
once again, and we will parse each Example  using tf.io.parse_single_example() .
This is a TensorFlow operation so it can be included in a TF Function. It requires at
least two arguments: a string scalar tensor containing the serialized data, and a
description of each feature. The description is a dictionary that maps each feature
name to either a tf.io.FixedLenFeature  descriptor indicating the feature’s shape,
type and default value, or a tf.io.VarLenFeature  descriptor indicating only the type
(if the length may vary, such as for the "emails"  feature). For example:
feature_description  = {
    "name": tf.io.FixedLenFeature ([], tf.string, default_value =""),
    "id": tf.io.FixedLenFeature ([], tf.int64, default_value =0),
    "emails" : tf.io.VarLenFeature (tf.string),
}

}
for serialized_example  in tf.data.TFRecordDataset (["my_contacts.tfrecord" ]):
    parsed_example  = tf.io.parse_single_example (serialized_example ,
                                                feature_description )
The fixed length features are parsed as regular tensors, but the variable length fea‐
tures are parsed as sparse tensors. Y ou can convert a sparse tensor to a dense tensor
using tf.sparse.to_dense() , but in this case it is simpler to just access its values:
>>> tf.sparse.to_dense (parsed_example ["emails" ], default_value =b"")

parsed_example  = tf.io.parse_single_example (serialized_example ,
<tf.Tensor: [...] dtype=string, numpy=array([b'a@b.com', b'c@d.com'], [...])>
>>> parsed_example ["emails" ].values
A BytesList  can contain any binary data you want, including any serialized object.
For example, you can use tf.io.encode_jpeg()  to encode an image using the JPEG
format, and put this binary data in a BytesList . Later, when your code reads the
TFRecord, it will start by parsing the Example , then you will need to call

tf.io.decode_jpeg()  to parse the data and get the original image (or you can use
tf.io.decode_image() , which can decode any BMP , GIF, JPEG or PNG image). Y ou
can also store any tensor you want in a BytesList  by serializing the tensor using
tf.io.serialize_tensor() , then putting the resulting byte string in a BytesList
feature. Later, when you parse the TFRecord, you can parse this data using
tf.io.parse_tensor() .
Instead of parsing examples one by one using tf.io.parse_single_example() , you

may want to parse them batch by batch using tf.io.parse_example() :418 | Chapter 13: Loading and Preprocessing Data with TensorFlow

dataset = tf.data.TFRecordDataset (["my_contacts.tfrecord" ]).batch(10)
for serialized_examples  in dataset:
    parsed_examples  = tf.io.parse_example (serialized_examples ,
                                          feature_description )
As you can see, the Example  proto will probably be sufficient for most use cases.
However, it may be a bit cumbersome to use when you are dealing with lists of lists.
For example, suppose you want to classify text documents. Each document may be
represented as a list of sentences, where each sentence is represented as a list of
words. And perhaps each document also has a list of comments, where each com‐
ment is also represented as a list of words. Moreover, there may be some contextual
data as well, such as the document’s author, title and publication date. TensorFlow’s
SequenceExample  protobuf is designed for such use cases.
Handling Lists of Lists Using the SequenceExample  Protobuf
Here is the definition of the SequenceExample  protobuf:

message FeatureList  { repeated  Feature feature = 1; };
message FeatureLists  { map<string, FeatureList > feature_list  = 1; };
message SequenceExample  {
    Features  context = 1;
    FeatureLists  feature_lists  = 2;
};
A SequenceExample  contains a Features  object for the contextual data and a Fea
tureLists  object which contains one or more named FeatureList  objects (e.g., a
FeatureList  named "content"  and another named "comments" ). Each FeatureList

};
just contains a list of Feature  objects, each of which may be a list of byte strings, a list
of 64-bit integers or a list of floats (in this example, each Feature  would represent a
sentence or a comment, perhaps in the form of a list of word identifiers). Building a
SequenceExample , serializing it and parsing it is very similar to building, serializing
and parsing an Example , but you must use tf.io.parse_single_sequence_exam
ple()  to parse a single SequenceExample  or tf.io.parse_sequence_example()  to

parse a batch, and both functions return a tuple containing the context features (as a
dictionary) and the feature lists (also as a dictionary). If the feature lists contain
sequences of varying sizes (as in the example above), you may want to convert them
to ragged tensors using tf.RaggedTensor.from_sparse()  (see the notebook for the
full code):
parsed_context , parsed_feature_lists  = tf.io.parse_single_sequence_example (
    serialized_sequence_example , context_feature_descriptions ,
    sequence_feature_descriptions )

parsed_content  = tf.RaggedTensor .from_sparse (parsed_feature_lists ["content" ])
Now that you know how to efficiently store, load and parse data, the next step is to
prepare it so that it can be fed to a neural network. This means converting all featuresThe TFRecord Format | 419

into numerical features (ideally not too sparse), scaling them, and more. In particular,
if your data contains categorical features or text features, they need to be converted to
numbers. For this, the Features API  can help.
The Features API
Preprocessing your data can be performed in many ways: it can be done ahead of
time when preparing your data files, using any tool you like. Or you can preprocess
your data on the fly when loading it with the Data API (e.g., using the dataset’s map()
method, as we saw earlier). Or you can include a preprocessing layer directly in your
model. Whichever solution you prefer, the Features API can help you: it is a set of
functions available in the tf.feature_column  package, which let you define how
each feature (or group of features) in your data should be preprocessed (therefore you
can think of this API as the analog of Scikit-Learn’s ColumnTransformer  class). We
will start by looking at the different types of columns available, and then we will look
at how to use them.

Let’s go back to the variant of the California housing dataset that we used in Chap‐
ter 2 , since it includes a categorical feature and missing data. Here is a simple numeri‐
cal column named "housing_median_age" :
housing_median_age  = tf.feature_column .numeric_column ("housing_median_age" )
Numeric columns let you specify a normalization function using the normalizer_fn
argument. For example, let’s tweak the "housing_median_age"  column to define how
it should be scaled. Note that this requires computing ahead of time the mean and

standard deviation of this feature in the training set:
age_mean , age_std = X_mean[1], X_std[1]  # The median age is column in 1
housing_median_age  = tf.feature_column .numeric_column (
    "housing_median_age" , normalizer_fn =lambda x: (x - age_mean ) / age_std)
In some cases, it might improve performance to bucketize some numerical features,
effectively transforming a numerical feature into a categorical feature. For example,
let’s create a bucketized column based on the median_income  column, with 5 buckets:

less than 1.5 ($15,000), then 1.5 to 3, 3 to 4.5, 4.5 to 6., and above 6. (notice that when
you specify 4 boundaries, there are actually 5 buckets):
median_income  = tf.feature_column .numeric_column ("median_income" )
bucketized_income  = tf.feature_column .bucketized_column (
    median_income , boundaries =[1.5, 3., 4.5, 6.])
If the median_income  feature is equal to, say, 3.2, then the bucketized_income  feature
will automatically be equal to 2 (i.e., the index of the corresponding income bucket).

Choosing the right boundaries can be somewhat of an art, but one approach is to just
use percentiles of the data (e.g., the 10th percentile, the 20th percentile, and so on). If
a feature is multimodal , meaning it has separate peaks in its distribution, you may420 | Chapter 13: Loading and Preprocessing Data with TensorFlow

want to define a bucket for each mode, placing the boundaries in between the peaks.
Whether you use the percentiles or the modes, you need to analyze the distribution of
your data ahead of time, just like we had to measure the mean and standard deviation
ahead of time to normalize the housing_median_age  column.
Categorical Features
For categorical features such as ocean_proximity , there are several options. If it is
already represented as a category ID (i.e., an integer from 0 to the max ID), then you
can use the categorical_column_with_identity()  function (specifying the max
ID). If not, and you know the list of all possible categories, then you can use categori
cal_column_with_vocabulary_list() :
ocean_prox_vocab  = ['<1H OCEAN' , 'INLAND' , 'ISLAND' , 'NEAR BAY' , 'NEAR OCEAN' ]
ocean_proximity  = tf.feature_column .categorical_column_with_vocabulary_list (
    "ocean_proximity" , ocean_prox_vocab )
If you prefer to have TensorFlow load the vocabulary from a file, you can call catego

rical_column_with_vocabulary_file()  instead. As you might expect, these two
functions will simply map each category to its index in the vocabulary (e.g., NEAR
BAY  will be mapped to 3), and unknown categories will be mapped to -1.
For categorical columns with a large vocabulary (e.g., for zipcodes, cities, words,
products, users, etc.), it may not be convenient to get the full list of possible cate‐
gories, or perhaps categories may be added or removed so frequently that using cate‐

gory indices would be too unreliable. In this case, you may prefer to use a
categorical_column_with_hash_bucket() . If we had a "city"  feature in the dataset,
we could encode it like this:
city_hash  = tf.feature_column .categorical_column_with_hash_bucket (
    "city", hash_bucket_size =1000)
This feature will compute a hash for each category (i.e., for each city), modulo the
number of hash buckets ( hash_bucket_size ). Y ou must set the number of buckets

high enough to avoid getting too many collisions (i.e., different categories ending up
in the same bucket), but the higher you set it, the more RAM will be used (by the
embedding table, as we will see shortly).
Crossed Categorical Features
If you suspect that two (or more) categorical features are more meaningful when used
jointly, then you can create a crossed column . For example, suppose people are partic‐
ularly fond of old houses inland and new houses near the ocean, then it might help toThe Features API | 421

9Since the housing_median_age  feature was normalized, the boundaries are for normalized ages. create a bucketized column for the housing_median_age  feature9, and cross it with
the ocean_proximity  column. The crossed column will compute a hash of every age
& ocean proximity combination it comes across, modulo the hash_bucket_size , and
this will give it the cross category ID. Y ou may then choose to use only this crossed
column in your model, or also include the individual columns.
bucketized_age  = tf.feature_column .bucketized_column (
    housing_median_age , boundaries =[-1., -0.5, 0., 0.5, 1.]) # age was scaled
age_and_ocean_proximity  = tf.feature_column .crossed_column (
    [bucketized_age , ocean_proximity ], hash_bucket_size =100)
Another common use case for crossed columns is to cross latitude and longitude into
a single categorical feature: you start by bucketizing the latitude and longitude, for
example into 20 buckets each, then you cross these bucketized features into a loca

housing_median_age , boundaries =[-1., -0.5, 0., 0.5, 1.]) # age was scaled
tion  column. This will create a 20×20 grid over California, and each cell in the grid
will correspond to one category:
latitude  = tf.feature_column .numeric_column ("latitude" )
longitude  = tf.feature_column .numeric_column ("longitude" )
bucketized_latitude  = tf.feature_column .bucketized_column (
    latitude , boundaries =list(np.linspace (32., 42., 20 - 1)))
bucketized_longitude  = tf.feature_column .bucketized_column (
    longitude , boundaries =list(np.linspace (-125., -114., 20 - 1)))
location  = tf.feature_column .crossed_column (

    [bucketized_latitude , bucketized_longitude ], hash_bucket_size =1000)
Encoding Categorical Features Using One-Hot Vectors
No matter which option you choose to build a categorical feature (categorical col‐
umns, bucketized columns or crossed columns), it must be encoded before you can
feed it to a neural network. There are two options to encode a categorical feature:
one-hot vectors or embeddings . For the first option, simply use the indicator_col
umn()  function:
ocean_proximity_one_hot  = tf.feature_column .indicator_column (ocean_proximity )

A one-hot vector encoding has the size of the vocabulary length, which is fine if there
are just a few possible categories, but if the vocabulary is large, you will end up with
too many inputs fed to your neural network: it will have too many weights to learn
and it will probably not perform very well. In particular, this will typically be the case
when you use hash buckets. In this case, you should probably encode them using
embeddings  instead.422 | Chapter 13: Loading and Preprocessing Data with TensorFlow

As a rule of thumb (but your mileage may vary!), if the number of
categories is lower than 10, then one-hot encoding is generally the
way to go. If the number of categories is greater than 50 (which is
often the case when you use hash buckets), then embeddings are
usually preferable. In between 10 and 50 categories, you may want
to experiment with both options and see which one works best for
your use case. Also, embeddings typically require more training
data, unless you can reuse pretrained embeddings.
Encoding Categorical Features Using Embeddings
An embedding is a trainable dense vector that represents a category. By default,
embeddings are initialized randomly, so for example the "NEAR BAY"  category could
be represented initially by a random vector such as [0.131, 0.890] , while the "NEAR
OCEAN"  category may be represented by another random vector such as [0.631,
0.791]  (in this example, we are using 2D embeddings, but the number of dimensions

is a hyperparameter you can tweak). Since these embeddings are trainable, they will
gradually improve during training, and as they represent fairly similar categories,
Gradient Descent will certainly end up pushing them closer together, while it will
tend to move them away from the "INLAND"  category’s embedding (see Figure 13-4 ).
Indeed, the better the representation, the easier it will be for the neural network to
make accurate predictions, so training tends to make embeddings useful representa‐

tions of the categories. This is called representation learning  (we will see other types of
representation learning in ???).The Features API | 423

10“Distributed Representations of Words and Phrases and their Compositionality” , T. Mikolov et al. (2013).
Figure 13-4. Embeddings Will Gradually Improve During Training
Word Embeddings
Not only will embeddings generally be useful representations for the task at hand, but
quite often these same embeddings can be reused successfully for other tasks as well.
The most common example of this is word embeddings  (i.e., embeddings of individual
words): when you are working on a natural language processing task, you are often
better off reusing pretrained word embeddings than training your own. The idea of
using vectors to represent words dates back to the 1960s, and many sophisticated
techniques have been used to generate useful vectors, including using neural net‐
works, but things really took off in 2013, when Tomáš Mikolov and other Google
researchers published a paper10 describing how to learn word embeddings using deep
neural networks, much faster than previous attempts. This allowed them to learn

embeddings on a very large corpus of text: they trained a deep neural network to pre‐
dict the words near any given word. This allowed them to obtain astounding word
embeddings. For example, synonyms had very close embeddings, and semantically
related words such as France, Spain, Italy, and so on, ended up clustered together. But
it’s not just about proximity: word embeddings were also organized along meaningful
axes in the embedding space. Here is a famous example: if you compute King – Man

+ Woman (adding and subtracting the embedding vectors of these words), then the
result will be very close to the embedding of the word Queen (see Figure 13-5 ). In
other words, the word embeddings encode the concept of gender! Similarly, you can
compute Madrid – Spain + France, and of course the result is close to Paris, which
seems to show that the notion of capital city was also encoded in the embeddings.424 | Chapter 13: Loading and Preprocessing Data with TensorFlow

Figure 13-5. Word Embeddings
Let’s go back to the Features API. Here is how you could encode the ocean_proxim
ity categories as 2D embeddings:
ocean_proximity_embed  = tf.feature_column .embedding_column (ocean_proximity ,
                                                           dimension =2)
Each of the five ocean_proximity  categories will now be represented as a 2D vector.
These vectors are stored in an embedding matrix  with one row per category, and one
column per embedding dimension, so in this example it is a 5×2 matrix. When an
embedding column is given a category index as input (say, 3, which corresponds to
the category "NEAR BAY" ), it just performs a lookup in the embedding matrix and
returns the corresponding row (say, [0.331, 0.190] ). Unfortunately, the embedding
matrix can be quite large, especially when you have a large vocabulary: if this is the
case, the model can only learn good representations for the categories for which it has

sufficient training data. To reduce the size of the embedding matrix, you can of
course try lowering the dimension  hyperparameter, but if you reduce this parameter
too much, the representations may not be as good. Another option is to reduce the
vocabulary size (e.g., if you are dealing with text, you can try dropping the rare words
from the vocabulary, and replace them all with a token like "<unknown>"  or "<UNK>" ).
If you are using hash buckets, you can also try reducing the hash_bucket_size  (but

not too much, or else you will get collisions).The Features API | 425

If there are no pretrained embeddings that you can reuse for the
task you are trying to tackle, and if you do not have enough train‐
ing data to learn them, then you can try to learn them on some
auxiliary task for which it is easier to obtain plenty of training data.
After that, you can reuse the trained embeddings for your main
task.
Using Feature Columns for Parsing
Let’s suppose you have created feature columns for each of your input features, as well
as for the target. What can you do with them? Well, for one you can pass them to the
make_parse_example_spec()  function to generate feature descriptions (so you don’t
have to do it manually, as we did earlier):
columns = [bucketized_age , ....., median_house_value ] # all features + target
feature_descriptions  = tf.feature_column .make_parse_example_spec (columns)
Y ou don’t always have to create a separate feature column for each
and every feature. For example, instead of having 2 numerical fea‐

ture columns, you could choose to have a single 2D column: just
set shape=[2]  when calling numerical_column() .
Y ou can then create a function that parses serialized examples using these feature
descriptions, and separates the target column from the input features:
def parse_examples (serialized_examples ):
    examples  = tf.io.parse_example (serialized_examples , feature_descriptions )
    targets = examples .pop("median_house_value" ) # separate the targets
    return examples , targets

Next, you can create a TFRecordDataset  that will read batches of serialized examples
(assuming the TFRecord file contains serialized Example  protobufs with the appropri‐
ate features):
dataset = tf.data.TFRecordDataset (["my_data_with_features.tfrecords" ])
dataset = dataset.repeat().shuffle(10000).batch(batch_size ).map(parse_examples )
Using Feature Columns in Your Models
Feature columns can also be used directly in your model, to convert all your input

return examples , targets
features into a single dense vector which the neural network can then process. For
this, all you need to do is add a keras.layers.DenseFeatures  layer as the first layer
in your model, passing it the list of feature columns (excluding the target column):
columns_without_target  = columns[:-1]
    keras.layers.DenseFeatures (feature_columns =columns_without_target ),426 | Chapter 13: Loading and Preprocessing Data with TensorFlow

keras.layers.Dense(1)
])
model.compile(loss="mse", optimizer ="sgd", metrics=["accuracy" ])
steps_per_epoch  = len(X_train) // batch_size
history = model.fit(dataset, steps_per_epoch =steps_per_epoch , epochs=5)
The DenseFeatures  layer will take care of converting every input feature to a dense
representation, and it will also apply any extra transformation we specified, such as
scaling the housing_median_age  using the normalizer_fn  function we provided. Y ou
can take a closer look at what the DenseFeatures  layer does by calling it directly:
>>> some_columns  = [ocean_proximity_embed , bucketized_income ]
>>> dense_features  = keras.layers.DenseFeatures (some_columns )
>>> dense_features ({
...     "ocean_proximity" : [["NEAR OCEAN" ], ["INLAND" ], ["INLAND" ]],
...     "median_income" : [[3.], [7.2], [1.]]
... })
...
<tf.Tensor: id=559790, shape=(3, 7), dtype=float32, numpy=
array([[ 0. , 0. , 1. , 0. , 0. ,-0.36277947 , 0.30109018],
       [ 0. , 0. , 0. , 0. , 1. , 0.22548223 , 0.33142096],

... })
...
       [ 1. , 0. , 0. , 0. , 0. , 0.22548223 , 0.33142096]], dtype=float32)>
In this example, we create a DenseFeatures  layer with just two columns, and we call
it with some data, in the form of a dictionary of features. In this case, since the bucke
tized_income  column relies on the median_income  column, the dictionary must
include the "median_income"  key, and similarly since the ocean_proximity_embed
column is based on the ocean_proximity  column, the dictionary must include the

[ 1. , 0. , 0. , 0. , 0. , 0.22548223 , 0.33142096]], dtype=float32)>
"ocean_proximity"  key. Columns are handled in alphabetical order, so first we look
at the bucketized income column (its name is the same as the median_income  column
name, plus "_bucketized" ). The incomes 3, 7.2 and 1 get mapped respectively to cat‐
egory 2 (for incomes between 1.5 and 3), category 0 (for incomes below 1.5), and cat‐
egory 4 (for incomes greater than 6). Then these category IDs get one-hot encoded:
category 2 gets encoded as [0., 0., 1., 0., 0.]  and so on (note that bucketized

columns get one-hot encoded by default, no need to call indicator_column() ). Now
on to the ocean_proximity_embed  column. The "NEAR OCEAN"  and "INLAND"  cate‐
gories just get mapped to their respective embeddings (which were initialized ran‐
domly). The resulting tensor is the concatenation of the one-hot vectors and the
embeddings.
Now you can feed all kinds of features to a neural network, including numerical fea‐
tures, categorical features, and even text (by splitting the text into words, then using

word embedding)! However, performing all the preprocessing on the fly can slow
down training. Let’s see how this can be improved.The Features API | 427

TF Transform
If preprocessing is computationally expensive, then handling it before training rather
than on the fly may give you a significant speedup: the data will be preprocessed just
once per instance before  training, rather than once per instance and per epoch during
training. Tools like Apache Beam let you run efficient data processing pipelines over
large amounts of data, even distributed across multiple servers, so why not use it to
preprocess all the training data? This works great and indeed can speed up training,
but there is one problem: once your model is trained, suppose you want to deploy it
to a mobile app: you will need to write some code in your app to take care of prepro‐
cessing the data before it is fed to the model. And suppose you also want to deploy
the model to TensorFlow.js so it runs in a web browser? Once again, you will need to
write some preprocessing code. This can become a maintenance nightmare: when‐

ever you want to change the preprocessing logic, you will need to update your Apache
Beam code, your mobile app code and your Javascript code. It is not only time con‐
suming, but also error prone: you may end up with subtle differences between the
preprocessing operations performed before training and the ones performed in your
app or in the browser. This training/serving skew  will lead to bugs or degraded perfor‐
mance.
One improvement would be to take the trained model (trained on data that was pre‐

mance.
processed by your Apache Beam code), and before deploying it to your app or the
browser, add an extra input layer to take care of preprocessing on the fly (either by
writing a custom layer or by using a DenseFeatures  layer). That’s definitely better,
since now you just have two versions of your preprocessing code: the Apache Beam
code and the preprocessing layer’s code.
But what if you could define your preprocessing operations just once? This is what

TF Transform was designed for. It is part of TensorFlow Extended  (TFX), an end-to-
end platform for productionizing TensorFlow models. First, to use a TFX component,
such as TF Transform, you must install it, it does not come bundled with TensorFlow.
Y ou define your preprocessing function just once (in Python), by using TF Transform
functions for scaling, bucketizing, crossing features, and more. Y ou can also use any
TensorFlow operation you need. Here is what this preprocessing function might look
like if we just had two features:

import tensorflow_transform  as tft
def preprocess (inputs):  # inputs is a batch of input features
    median_age  = inputs["housing_median_age" ]
    ocean_proximity  = inputs["ocean_proximity" ]
    standardized_age  = tft.scale_to_z_score (median_age  - tft.mean(median_age ))
    ocean_proximity_id  = tft.compute_and_apply_vocabulary (ocean_proximity )
    return {
        "standardized_median_age" : standardized_age ,428 | Chapter 13: Loading and Preprocessing Data with TensorFlow

11At the time of writing, TFDS requires you to download a few files manually for ImageNet (for legal reasons),
but this will hopefully get resolved soon.
        "ocean_proximity_id" : ocean_proximity_id
    }
Next, TF Transform lets you apply this preprocess()  function to the whole training
set using Apache Beam (it provides an AnalyzeAndTransformDataset  class that you
can use for this purpose in your Apache Beam pipeline). In the process, it will also
compute all the necessary statistics over the whole training set: in this example, the
mean and standard deviation of the housing_median_age  feature, and the vocabulary
for the ocean_proximity  feature. The components that compute these statistics are
called analyzers .
Importantly, TF Transform will also generate an equivalent TensorFlow Function that
you can plug into the model you deploy. This TF Function contains all the necessary
statistics computed by Apache Beam (the mean, standard deviation, and vocabulary),
simply included as constants.

At the time of this writing, TF Transform only supports Tensor‐
Flow 1. Moreover, Apache Beam only has partial support for
Python 3. That said, both these limitations will likely be fixed by
the time your read this.
With the Data API, TFRecords, the Features API and TF Transform, you can build
highly scalable input pipelines for training, and also benefit from fast and portable
data preprocessing in production.
But what if you just wanted to use a standard dataset? Well in that case, things are
much simpler: just use TFDS!

The TensorFlow Datasets (TFDS) Project
The TensorFlow Datasets  project makes it trivial to download common datasets, from
small ones like MNIST or Fashion MNIST, to huge datasets like ImageNet11 (you will
need quite a bit of disk space!). The list includes image datasets, text datasets (includ‐
ing translation datasets), audio and video datasets, and more. Y ou can visit https://
homl.info/tfds  to view the full list, along with a description of each dataset.
TFDS is not bundled with TensorFlow, so you need to install the tensorflow-

datasets  library (e.g., using pip). Then all you need to do is call the tfds.load()
function, and it will download the data you want (unless it was already downloaded
earlier), and return the data as a dictionary of Datasets  (typically one for training,The TensorFlow Datasets (TFDS) Project | 429

and one for testing, but this depends on the dataset you choose). For example, let’s
download MNIST:
import tensorflow_datasets  as tfds
dataset = tfds.load(name="mnist")
mnist_train , mnist_test  = dataset["train"], dataset["test"]
Y ou can then apply any transformation you want (typically repeating, batching and
prefetching), and you’re ready to train your model. Here is a simple example:
mnist_train  = mnist_train .repeat(5).batch(32).prefetch (1)
for item in mnist_train :
    images = item["image"]
    labels = item["label"]
    [...]
In general, load()  returns a shuffled training set, so there’s no need
to shuffle it some more.
Note that each item in the dataset is a dictionary containing both the features and the
labels. But Keras expects each item to be a tuple containing 2 elements (again, the fea‐
tures and the labels). Y ou could transform the dataset using the map()  method, like
this:
mnist_train  = mnist_train .repeat(5).batch(32)

    [...]
this:
mnist_train  = mnist_train .map(lambda items: (items["image"], items["label"]))
mnist_train  = mnist_train .prefetch (1)
Or you can just ask the load()  function to do this for you by setting as_super
vised=True  (obviously this works only for labeled datasets). Y ou can also specify the
batch size if you want. Then the dataset can be passed directly to your tf.keras model:
dataset = tfds.load(name="mnist", batch_size =32, as_supervised =True)
mnist_train  = dataset["train"].repeat().prefetch (1)

model.compile(loss="sparse_categorical_crossentropy" , optimizer ="sgd")
model.fit(mnist_train , steps_per_epoch =60000 // 32, epochs=5)
This was quite a technical chapter, and you may feel that it is a bit far from the
abstract beauty of neural networks, but the fact is deep learning often involves large
amounts of data, and knowing how to load, parse and preprocess it efficiently is a
crucial skill to have. In the next chapter, we will look at Convolutional Neural Net‐

works, which are among the most successful neural net architectures for image pro‐
cessing, and many other applications.430 | Chapter 13: Loading and Preprocessing Data with TensorFlow

CHAPTER 14
Deep Computer Vision Using Convolutional
Neural Networks
release of these titles. The following will be Chapter 14 in the final
Although IBM’s Deep Blue supercomputer beat the chess world champion Garry Kas‐
parov back in 1996, it wasn’t until fairly recently that computers were able to reliably
perform seemingly trivial tasks such as detecting a puppy in a picture or recognizing
spoken words. Why are these tasks so effortless to us humans? The answer lies in the
fact that perception largely takes place outside the realm of our consciousness, within
specialized visual, auditory, and other sensory modules in our brains. By the time
sensory information reaches our consciousness, it is already adorned with high-level

features; for example, when you look at a picture of a cute puppy, you cannot choose
not to see the puppy, or not to notice its cuteness. Nor can you explain how you rec‐
ognize a cute puppy; it’s just obvious to you. Thus, we cannot trust our subjective
experience: perception is not trivial at all, and to understand it we must look at how
the sensory modules work.
Convolutional neural networks (CNNs) emerged from the study of the brain’s visual
cortex, and they have been used in image recognition since the 1980s. In the last few

years, thanks to the increase in computational power, the amount of available training
data, and the tricks presented in Chapter 11  for training deep nets, CNNs have man‐
aged to achieve superhuman performance on some complex visual tasks. They power
image search services, self-driving cars, automatic video classification systems, and
more. Moreover, CNNs are not restricted to visual perception: they are also successful431

1“Single Unit Activity in Striate Cortex of Unrestrained Cats, ” D. Hubel and T. Wiesel (1958).2
“Receptive Fields of Single Neurones in the Cat’s Striate Cortex, ” D. Hubel and T. Wiesel (1959).3
“Receptive Fields and Functional Architecture of Monkey Striate Cortex, ” D. Hubel and T. Wiesel (1968).at many other tasks, such as voice recognition  or natural language processing  (NLP);
however, we will focus on visual applications for now.
In this chapter we will present where CNNs came from, what their building blocks
look like, and how to implement them using TensorFlow and Keras. Then we will dis‐
cuss some of the best CNN architectures, and discuss other visual tasks, including
object detection  (classifying multiple objects in an image and placing bounding boxes
around them) and semantic segmentation  (classifying each pixel according to the class
of the object it belongs to).
The Architecture of the Visual Cortex
David H. Hubel and Torsten Wiesel performed a series of experiments on cats in

19581 and 19592 (and a few years later on monkeys3), giving crucial insights on the
structure of the visual cortex (the authors received the Nobel Prize in Physiology or
Medicine in 1981 for their work). In particular, they showed that many neurons in
the visual cortex have a small local receptive field, meaning they react only to visual
stimuli located in a limited region of the visual field (see Figure 14-1 , in which the
local receptive fields of five neurons are represented by dashed circles). The receptive

fields of different neurons may overlap, and together they tile the whole visual field.
Moreover, the authors showed that some neurons react only to images of horizontal
lines, while others react only to lines with different orientations (two neurons may
have the same receptive field but react to different line orientations). They also
noticed that some neurons have larger receptive fields, and they react to more com‐
plex patterns that are combinations of the lower-level patterns. These observations

led to the idea that the higher-level neurons are based on the outputs of neighboring
lower-level neurons (in Figure 14-1 , notice that each neuron is connected only to a
few neurons from the previous layer). This powerful architecture is able to detect all
sorts of complex patterns in any area of the visual field.432 | Chapter 14: Deep Computer Vision Using Convolutional Neural Networks

4“Neocognitron: A Self-organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected
by Shift in Position, ” K. Fukushima (1980).5
“Gradient-Based Learning Applied to Document Recognition, ” Y . LeCun et al. (1998).
Figure 14-1. Local receptive fields  in the visual cortex
These studies of the visual cortex inspired the neocognitron, introduced in 1980 ,4
which gradually evolved into what we now call convolutional neural networks . An
important milestone was a 1998 paper5 by Y ann LeCun, Léon Bottou, Y oshua Bengio,
and Patrick Haffner, which introduced the famous LeNet-5  architecture, widely used
to recognize handwritten check numbers. This architecture has some building blocks
that you already know, such as fully connected layers and sigmoid activation func‐
tions, but it also introduces two new building blocks: convolutional layers  and pooling
layers . Let’s look at them now.
Why not simply use a regular deep neural network with fully con‐

nected layers for image recognition tasks? Unfortunately, although
this works fine for small images (e.g., MNIST), it breaks down for
larger images because of the huge number of parameters it
requires. For example, a 100 × 100 image has 10,000 pixels, and if
the first layer has just 1,000 neurons (which already severely
restricts the amount of information transmitted to the next layer),
this means a total of 10 million connections. And that’s just the first
layer. CNNs solve this problem using partially connected layers and

weight sharing.The Architecture of the Visual Cortex | 433

6A convolution is a mathematical operation that slides one function over another and measures the integral of
their pointwise multiplication. It has deep connections with the Fourier transform and the Laplace transform,
and is heavily used in signal processing. Convolutional layers actually use cross-correlations, which are very
similar to convolutions (see https://homl.info/76  for more details).
Convolutional Layer
The most important building block of a CNN is the convolutional layer :6 neurons in
the first convolutional layer are not connected to every single pixel in the input image
(like they were in previous chapters), but only to pixels in their receptive fields (see
Figure 14-2 ). In turn, each neuron in the second convolutional layer is connected
only to neurons located within a small rectangle in the first layer. This architecture
allows the network to concentrate on small low-level features in the first hidden layer,

then assemble them into larger higher-level features in the next hidden layer, and so
on. This hierarchical structure is common in real-world images, which is one of the
reasons why CNNs work so well for image recognition.
Figure 14-2. CNN layers with rectangular local receptive fields
Until now, all multilayer neural networks we looked at had layers
composed of a long line of neurons, and we had to flatten input
images to 1D before feeding them to the neural network. Now each
layer is represented in 2D, which makes it easier to match neurons
with their corresponding inputs.

A neuron located in row i, column j of a given layer is connected to the outputs of the
neurons in the previous layer located in rows i to i + fh – 1, columns j to j + fw – 1,
where fh and fw are the height and width of the receptive field (see Figure 14-3 ). In
order for a layer to have the same height and width as the previous layer, it is com‐434 | Chapter 14: Deep Computer Vision Using Convolutional Neural Networks

mon to add zeros around the inputs, as shown in the diagram. This is called zero pad‐
ding.
Figure 14-3. Connections between layers and zero padding
It is also possible to connect a large input layer to a much smaller layer by spacing out
the receptive fields, as shown in Figure 14-4 . The shift from one receptive field to the
next is called the stride . In the diagram, a 5 × 7 input layer (plus zero padding) is con‐
nected to a 3 × 4 layer, using 3 × 3 receptive fields and a stride of 2 (in this example
the stride is the same in both directions, but it does not have to be so). A neuron loca‐
ted in row i, column j in the upper layer is connected to the outputs of the neurons in
the previous layer located in rows i × sh to i × sh + fh – 1, columns j × sw to j × sw + fw –
1, where sh and sw are the vertical and horizontal strides.Convolutional Layer | 435

Figure 14-4. Reducing dimensionality using a stride of 2
Filters
A neuron’s weights can be represented as a small image the size of the receptive field.
For example, Figure 14-5  shows two possible sets of weights, called filters  (or convolu‐
tion kernels ). The first one is represented as a black square with a vertical white line in
the middle (it is a 7 × 7 matrix full of 0s except for the central column, which is full of
1s); neurons using these weights will ignore everything in their receptive field except
for the central vertical line (since all inputs will get multiplied by 0, except for the
ones located in the central vertical line). The second filter is a black square with a
horizontal white line in the middle. Once again, neurons using these weights will
ignore everything in their receptive field except for the central horizontal line.
Now if all neurons in a layer use the same vertical line filter (and the same bias term),

and you feed the network the input image shown in Figure 14-5  (bottom image), the
layer will output the top-left image. Notice that the vertical white lines get enhanced
while the rest gets blurred. Similarly, the upper-right image is what you get if all neu‐
rons use the same horizontal line filter; notice that the horizontal white lines get
enhanced while the rest is blurred out. Thus, a layer full of neurons using the same
filter outputs a feature map , which highlights the areas in an image that activate the

filter the most. Of course you do not have to define the filters manually: instead, dur‐
ing training the convolutional layer will automatically learn the most useful filters for
its task, and the layers above will learn to combine them into more complex patterns.436 | Chapter 14: Deep Computer Vision Using Convolutional Neural Networks

Figure 14-5. Applying two different  filters  to get two feature maps
Stacking Multiple Feature Maps
Up to now, for simplicity, I have represented the output of each convolutional layer as
a thin 2D layer, but in reality a convolutional layer has multiple filters (you decide
how many), and it outputs one feature map per filter, so it is more accurately repre‐
sented in 3D (see Figure 14-6 ). To do so, it has one neuron per pixel in each feature
map, and all neurons within a given feature map share the same parameters (i.e., the
same weights and bias term). However, neurons in different feature maps use differ‐
ent parameters. A neuron’s receptive field is the same as described earlier, but it
extends across all the previous layers’ feature maps. In short, a convolutional layer
simultaneously applies multiple trainable filters to its inputs, making it capable of
detecting multiple features anywhere in its inputs.
The fact that all neurons in a feature map share the same parame‐

ters dramatically reduces the number of parameters in the model.
Moreover, once the CNN has learned to recognize a pattern in one
location, it can recognize it in any other location. In contrast, once
a regular DNN has learned to recognize a pattern in one location, it
can recognize it only in that particular location.
Moreover, input images are also composed of multiple sublayers: one per color chan‐
nel. There are typically three: red, green, and blue (RGB). Grayscale images have justConvolutional Layer | 437

one channel, but some images may have much more—for example, satellite images
that capture extra light frequencies (such as infrared).
Figure 14-6. Convolution layers with multiple feature maps, and images with three color
channels
Specifically, a neuron located in row i, column j of the feature map k in a given convo‐
lutional layer l is connected to the outputs of the neurons in the previous layer l – 1,
located in rows i × sh to i × sh + fh – 1 and columns j × sw to j × sw + fw – 1, across all
feature maps (in layer l – 1). Note that all neurons located in the same row i and col‐
umn j but in different feature maps are connected to the outputs of the exact same
neurons in the previous layer.
Equation 14-1  summarizes the preceding explanations in one big mathematical equa‐
tion: it shows how to compute the output of a given neuron in a convolutional layer.438 | Chapter 14: Deep Computer Vision Using Convolutional Neural Networks

It is a bit ugly due to all the different indices, but all it does is calculate the weighted
sum of all the inputs, plus the bias term.
Equation 14-1. Computing the output of a neuron in a convolutional layerzi,j,k=bk+∑
u= 0fh− 1
∑
v= 0fw− 1
∑
k′= 0fn′− 1
xi′,j′,k′.wu,v,k′,kwithi′=i×sh+u
j′=j×sw+v•
zi, j, k is the output of the neuron located in row i, column j in feature map k of the
convolutional layer (layer l).•
As explained earlier, sh and sw are the vertical and horizontal strides, fh and fw are
the height and width of the receptive field, and fn′ is the number of feature maps
in the previous layer (layer l – 1).•
xi′, j′, k′ is the output of the neuron located in layer l – 1, row i′, column j′, feature
map k′ (or channel k′ if the previous layer is the input layer).•
bk is the bias term for feature map k (in layer l). Y ou can think of it as a knob that
tweaks the overall brightness of the feature map k.•
wu, v, k′ ,k is the connection weight between any neuron in feature map k of the layer

l and its input located at row u, column v (relative to the neuron’s receptive field),
and feature map k′.
TensorFlow Implementation
In TensorFlow, each input image is typically represented as a 3D tensor of shape
[height, width, channels] . A mini-batch is represented as a 4D tensor of shape
[mini-batch size, height, width, channels] . The weights of a convolutional
layer are represented as a 4D tensor of shape [ fh, fw, fn′, fn]. The bias terms of a convo‐

lutional layer are simply represented as a 1D tensor of shape [ fn].
Let’s look at a simple example. The following code loads two sample images, using
Scikit-Learn’s load_sample_images()  (which loads two color images, one of a Chi‐
nese temple, and the other of a flower). The pixel intensities (for each color channel)
is represented as a byte from 0 to 255, so we scale these features simply by dividing by
255, to get floats ranging from 0 to 1. Then we create two 7 × 7 filters (one with a

vertical white line in the middle, and the other with a horizontal white line in the
middle), and we apply them to both images using the tf.nn.conv2d()  function,
which is part of TensorFlow’s low-level Deep Learning API. In this example, we use
zero padding ( padding="SAME" ) and a stride of 2. Finally, we plot one of the resulting
feature maps (similar to the top-right image in Figure 14-5 ).Convolutional Layer | 439

from sklearn.datasets  import load_sample_image
# Load sample images
china = load_sample_image ("china.jpg" ) / 255
flower = load_sample_image ("flower.jpg" ) / 255
images = np.array([china, flower])
batch_size , height, width, channels  = images.shape
# Create 2 filters
filters = np.zeros(shape=(7, 7, channels , 2), dtype=np.float32)
filters[:, 3, :, 0] = 1  # vertical line
filters[3, :, :, 1] = 1  # horizontal line
outputs = tf.nn.conv2d(images, filters, strides=1, padding="SAME")
plt.imshow(outputs[0, :, :, 1], cmap="gray") # plot 1st image's 2nd feature map
Most of this code is self-explanatory, but the tf.nn.conv2d()  line deserves a bit of
explanation:•
images  is the input mini-batch (a 4D tensor, as explained earlier).•
filters  is the set of filters to apply (also a 4D tensor, as explained earlier).•
strides  is equal to 1, but it could also be a 1D array with 4 elements, where the
two central elements are the vertical and horizontal strides ( sh and sw). The first

and last elements must currently be equal to 1. They may one day be used to
specify a batch stride (to skip some instances) and a channel stride (to skip some
of the previous layer’s feature maps or channels).•
padding  must be either "VALID"  or "SAME" :—
If set to "VALID" , the convolutional layer does not use zero padding, and may
ignore some rows and columns at the bottom and right of the input image,
depending on the stride, as shown in Figure 14-7  (for simplicity, only the hor‐

izontal dimension is shown here, but of course the same logic applies to the
vertical dimension).—
If set to "SAME" , the convolutional layer uses zero padding if necessary. In this
case, the number of output neurons is equal to the number of input neurons
divided by the stride, rounded up (in this example, 13 / 5 = 2.6, rounded up to
3). Then zeros are added as evenly as possible around the inputs.440 | Chapter 14: Deep Computer Vision Using Convolutional Neural Networks

Figure 14-7. Padding options—input width: 13, filter  width: 6, stride: 5
In this example, we manually defined the filters, but in a real CNN you would nor‐
mally define filters as trainable variables, so the neural net can learn which filters
work best, as explained earlier. Instead of manually creating the variables, however,
you can simply use the keras.layers.Conv2D  layer:
conv = keras.layers.Conv2D(filters=32, kernel_size =3, strides=1,
                           padding="SAME", activation ="relu")
This code creates a Conv2D  layer with 32 filters, each 3 × 3, using a stride of 1 (both
horizontally and vertically), SAME padding, and applying the ReLU activation func‐
tion to its outputs. As you can see, convolutional layers have quite a few hyperpara‐
meters: you must choose the number of filters, their height and width, the strides, and
the padding type. As always, you can use cross-validation to find the right hyperpara‐
meter values, but this is very time-consuming. We will discuss common CNN archi‐

tectures later, to give you some idea of what hyperparameter values work best in 
practice.
Memory Requirements
Another problem with CNNs is that the convolutional layers require a huge amount
of RAM. This is especially true during training, because the reverse pass of backpro‐
pagation requires all the intermediate values computed during the forward pass.
For example, consider a convolutional layer with 5 × 5 filters, outputting 200 feature
maps of size 150 × 100, with stride 1 and SAME padding. If the input is a 150 × 100Convolutional Layer | 441

7A fully connected layer with 150 × 100 neurons, each connected to all 150 × 100 × 3 inputs, would have 1502
× 1002 × 3 = 675 million parameters!8
In the international system of units (SI), 1 MB = 1,000 kB = 1,000 × 1,000 bytes = 1,000 × 1,000 × 8 bits.
RGB image (three channels), then the number of parameters is (5 × 5 × 3 + 1) × 200
= 15,200 (the +1 corresponds to the bias terms), which is fairly small compared to a
fully connected layer.7 However, each of the 200 feature maps contains 150 × 100 neu‐
rons, and each of these neurons needs to compute a weighted sum of its 5 × 5 × 3 =
75 inputs: that’s a total of 225 million float multiplications. Not as bad as a fully con‐
nected layer, but still quite computationally intensive. Moreover, if the feature maps
are represented using 32-bit floats, then the convolutional layer’s output will occupy
200 × 150 × 100 × 32 = 96 million bits (12 MB) of RAM.8 And that’s just for one
instance! If a training batch contains 100 instances, then this layer will use up 1.2 GB

of RAM!
During inference (i.e., when making a prediction for a new instance) the RAM occu‐
pied by one layer can be released as soon as the next layer has been computed, so you
only need as much RAM as required by two consecutive layers. But during training
everything computed during the forward pass needs to be preserved for the reverse
pass, so the amount of RAM needed is (at least) the total amount of RAM required by
all layers.
If training crashes because of an out-of-memory error, you can try
reducing the mini-batch size. Alternatively, you can try reducing

dimensionality using a stride, or removing a few layers. Or you can
try using 16-bit floats instead of 32-bit floats. Or you could distrib‐
ute the CNN across multiple devices.
Now let’s look at the second common building block of CNNs: the pooling layer .
Pooling Layer
Once you understand how convolutional layers work, the pooling layers are quite
easy to grasp. Their goal is to subsample  (i.e., shrink) the input image in order to
reduce the computational load, the memory usage, and the number of parameters

(thereby limiting the risk of overfitting).
Just like in convolutional layers, each neuron in a pooling layer is connected to the
outputs of a limited number of neurons in the previous layer, located within a small
rectangular receptive field. Y ou must define its size, the stride, and the padding type,
just like before. However, a pooling neuron has no weights; all it does is aggregate the
inputs using an aggregation function such as the max or mean. Figure 14-8  shows a

max pooling layer , which is the most common type of pooling layer. In this example,442 | Chapter 14: Deep Computer Vision Using Convolutional Neural Networks

9Other kernels we discussed so far had weights, but pooling kernels do not: they are just stateless sliding win‐
dows.
we use a 2 × 2 _pooling kernel_9, with a stride of 2, and no padding. Only the max
input value in each receptive field makes it to the next layer, while the other inputs
are dropped. For example, in the lower left receptive field in Figure 14-8 , the input
values are 1, 5, 3, 2, so only the max value, 5, is propagated to the next layer. Because
of the stride of 2, the output image has half the height and half the width of the input
image (rounded down since we use no padding).
Figure 14-8. Max pooling layer (2 × 2 pooling kernel, stride 2, no padding)
A pooling layer typically works on every input channel independ‐
ently, so the output depth is the same as the input depth.
Other than reducing computations, memory usage and the number of parameters, a
max pooling layer also introduces some level of invariance  to small translations, as

shown in Figure 14-9 . Here we assume that the bright pixels have a lower value than
dark pixels, and we consider 3 images (A, B, C) going through a max pooling layer
with a 2 × 2 kernel and stride 2. Images B and C are the same as image A, but shifted
by one and two pixels to the right. As you can see, the outputs of the max pooling
layer for images A and B are identical. This is what translation invariance means.
However, for image C, the output is different: it is shifted by one pixel to the right

(but there is still 75% invariance). By inserting a max pooling layer every few layers in
a CNN, it is possible to get some level of translation invariance at a larger scale.
Moreover, max pooling also offers a small amount of rotational invariance and a
slight scale invariance. Such invariance (even if it is limited) can be useful in cases
where the prediction should not depend on these details, such as in classification
tasks.Pooling Layer | 443

Figure 14-9. Invariance to small translations
But max pooling has some downsides: firstly, it is obviously very destructive: even
with a tiny 2 × 2 kernel and a stride of 2, the output will be two times smaller in both
directions (so its area will be four times smaller), simply dropping 75% of the input
values. And in some applications, invariance is not desirable, for example for seman‐
tic segmentation : this is the task of classifying each pixel in an image depending on the
object that pixel belongs to: obviously, if the input image is translated by 1 pixel to the
right, the output should also be translated by 1 pixel to the right. The goal in this case
is equivariance , not invariance: a small change to the inputs should lead to a corre‐
sponding small change in the output.
Implementing a max pooling layer in TensorFlow is quite easy. The following code
creates a max pooling layer using a 2 × 2 kernel. The strides default to the kernel size,

so this layer will use a stride of 2 (both horizontally and vertically). By default, it uses
V ALID padding (i.e., no padding at all):
max_pool  = keras.layers.MaxPool2D (pool_size =2)
To create an average pooling layer , just use AvgPool2D  instead of MaxPool2D . As you
might expect, it works exactly like a max pooling layer, except it computes the mean
rather than the max. Average pooling layers used to be very popular, but people444 | Chapter 14: Deep Computer Vision Using Convolutional Neural Networks

mostly use max pooling layers now, as they generally perform better. This may seem
surprising, since computing the mean generally loses less information than comput‐
ing the max. But on the other hand, max pooling preserves only the strongest feature,
getting rid of all the meaningless ones, so the next layers get a cleaner signal to work
with. Moreover, max pooling offers stronger translation invariance than average
pooling.
Note that max pooling and average pooling can be performed along the depth dimen‐
sion rather than the spatial dimensions, although this is not as common. This can
allow the CNN to learn to be invariant to various features. For example, it could learn
multiple filters, each detecting a different rotation of the same pattern, such as hand-
written digits (see Figure 14-10 ), and the depth-wise max pooling layer would ensure
that the output is the same regardless of the rotation. The CNN could similarly learn
to be invariant to anything else: thickness, brightness, skew, color, and so on.

Figure 14-10. Depth-wise max pooling can help the CNN learn any invariancePooling Layer | 445

Keras does not include a depth-wise max pooling layer, but TensorFlow’s low-level
Deep Learning API does: just use the tf.nn.max_pool()  function, and specify the
kernel size and strides as 4-tuples. The first three values of each should be 1: this indi‐
cates that the kernel size and stride along the batch, height and width dimensions
shoud be 1. The last value should be whatever kernel size and stride you want along
the depth dimension, for example 3 (this must be a divisor of the input depth; for
example, it will not work if the previous layer outputs 20 feature maps, since 20 is not
a multiple of 3):
output = tf.nn.max_pool (images,
                        ksize=(1, 1, 1, 3),
                        strides=(1, 1, 1, 3),
                        padding="VALID")
If you want to include this as a layer in your Keras models, you can simply wrap it in
a Lambda  layer (or create a custom Keras layer):
depth_pool  = keras.layers.Lambda(
    lambda X: tf.nn.max_pool (X, ksize=(1, 1, 1, 3), strides=(1, 1, 1, 3),

                             padding="VALID"))
One last type of pooling layer that you will often see in modern architectures is the
global average pooling  layer. It works very differently: all it does is compute the mean
of each entire feature map (it’s like an average pooling layer using a pooling kernel
with the same spatial dimensions as the inputs). This means that it just outputs a sin‐
gle number per feature map and per instance. Although this is of course extremely
destructive (most of the information in the feature map is lost), it can be useful as the

output layer, as we will see later in this chapter. To create such a layer, simply use the
keras.layers.GlobalAvgPool2D  class:
global_avg_pool  = keras.layers.GlobalAvgPool2D ()
It is actually equivalent to this simple Lamba  layer, which computes the mean over the
spatial dimensions (height and width):
global_avg_pool  = keras.layers.Lambda(lambda X: tf.reduce_mean (X, axis=[1, 2]))
Now you know all the building blocks to create a convolutional neural network. Let’s
see how to assemble them.
CNN Architectures

Typical CNN architectures stack a few convolutional layers (each one generally fol‐
lowed by a ReLU layer), then a pooling layer, then another few convolutional layers
(+ReLU), then another pooling layer, and so on. The image gets smaller and smaller
as it progresses through the network, but it also typically gets deeper and deeper (i.e.,
with more feature maps) thanks to the convolutional layers (see Figure 14-11 ). At the
top of the stack, a regular feedforward neural network is added, composed of a few446 | Chapter 14: Deep Computer Vision Using Convolutional Neural Networks

fully connected layers (+ReLUs), and the final layer outputs the prediction (e.g., a
softmax layer that outputs estimated class probabilities).
Figure 14-11. Typical CNN architecture
A common mistake is to use convolution kernels that are too large.
For example, instead of using a convolutional layer with a 5 × 5
kernel, it is generally preferable to stack two layers with 3 × 3 ker‐
nels: it will use less parameters and require less computations, and
it will usually perform better. One exception to this recommenda‐
tion is for the first convolutional layer: it can typically have a large
kernel (e.g., 5 × 5), usually with stride of 2 or more: this will reduce
the spatial dimension of the image without losing too much infor‐
mation, and since the input image only has 3 channels in general, it
will not be too costly.
Here is how you can implement a simple CNN to tackle the fashion MNIST dataset
(introduced in Chapter 10 ):
DefaultConv2D  = partial(keras.layers.Conv2D,

                        kernel_size =3, activation ='relu', padding="SAME")
    DefaultConv2D (filters=64, kernel_size =7, input_shape =[28, 28, 1]),
    keras.layers.MaxPooling2D (pool_size =2),
    DefaultConv2D (filters=128),
    DefaultConv2D (filters=256),
    keras.layers.Flatten(),

kernel_size =3, activation ='relu', padding="SAME")
    keras.layers.Dense(units=128, activation ='relu'),
    keras.layers.Dropout(0.5),
    keras.layers.Dense(units=64, activation ='relu'),
    keras.layers.Dense(units=10, activation ='softmax' ),
])CNN Architectures | 447

•In this code, we start by using the partial()  function to define a thin wrapper
around the Conv2D  class, called DefaultConv2D : it simply avoids having to repeat
the same hyperparameter values over and over again.•
The first layer uses a large kernel size, but no stride because the input images are
not very large. It also sets input_shape=[28, 28, 1] , which means the images
are 28 × 28 pixels, with a single color channel (i.e., grayscale).•
Next, we have a max pooling layer, which divides each spatial dimension by a fac‐
tor of two (since pool_size=2 ).•
Then we repeat the same structure twice: two convolutional layers followed by a
max pooling layer. For larger images, we could repeat this structure several times
(the number of repetitions is a hyperparameter you can tune).•
Note that the number of filters grows as we climb up the CNN towards the out‐
put layer (it is initially 64, then 128, then 256): it makes sense for it to grow, since

the number of low level features is often fairly low (e.g., small circles, horizontal
lines, etc.), but there are many different ways to combine them into higher level
features. It is a common practice to double the number of filters after each pool‐
ing layer: since a pooling layer divides each spatial dimension by a factor of 2, we
can afford doubling the number of feature maps in the next layer, without fear of
exploding the number of parameters, memory usage, or computational load.•

Next is the fully connected network, composed of 2 hidden dense layers and a
dense output layer. Note that we must flatten its inputs, since a dense network
expects a 1D array of features for each instance. We also add two dropout layers,
with a dropout rate of 50% each, to reduce overfitting.
This CNN reaches over 92% accuracy on the test set. It’s not the state of the art, but it
is pretty good, and clearly much better than what we achieved with dense networks in
Chapter 10 .

Over the years, variants of this fundamental architecture have been developed, lead‐
ing to amazing advances in the field. A good measure of this progress is the error rate
in competitions such as the ILSVRC ImageNet challenge . In this competition the
top-5 error rate for image classification fell from over 26% to less than 2.3% in just six
years. The top-five error rate is the number of test images for which the system’s top 5
predictions did not include the correct answer. The images are large (256 pixels high)

and there are 1,000 classes, some of which are really subtle (try distinguishing 120
dog breeds). Looking at the evolution of the winning entries is a good way to under‐
stand how CNNs work.
We will first look at the classical LeNet-5 architecture (1998), then three of the win‐
ners of the ILSVRC challenge: AlexNet (2012), GoogLeNet (2014), and ResNet
(2015).448 | Chapter 14: Deep Computer Vision Using Convolutional Neural Networks

10“Gradient-Based Learning Applied to Document Recognition” , Y . LeCun, L. Bottou, Y . Bengio and P . Haffner
(1998).LeNet-5
The LeNet-5 architecture10 is perhaps the most widely known CNN architecture. As
mentioned earlier, it was created by Y ann LeCun in 1998 and widely used for hand‐
written digit recognition (MNIST). It is composed of the layers shown in Table 14-1 .Table 14-1. LeNet-5 architectureLayerTypeMapsSizeKernel sizeStrideActivationOutFully Connected–10––RBFF6Fully Connected–84––tanhC5Convolution1201 × 15 × 51tanhS4Avg Pooling165 × 52 × 22tanhC3Convolution1610 × 105 × 51tanhS2Avg Pooling614 × 142 × 22tanhC1Convolution628 × 285 × 51tanhInInput132 × 32–––
There are a few extra details to be noted:•
MNIST images are 28 × 28 pixels, but they are zero-padded to 32 × 32 pixels and
normalized before being fed to the network. The rest of the network does not use
any padding, which is why the size keeps shrinking as the image progresses
through the network.•

The average pooling layers are slightly more complex than usual: each neuron
computes the mean of its inputs, then multiplies the result by a learnable coeffi‐
cient (one per map) and adds a learnable bias term (again, one per map), then
finally applies the activation function.•
Most neurons in C3 maps are connected to neurons in only three or four S2
maps (instead of all six S2 maps). See table 1 (page 8) in the original paper10 for
details.•
The output layer is a bit special: instead of computing the matrix multiplication
of the inputs and the weight vector, each neuron outputs the square of the Eucli‐
dian distance between its input vector and its weight vector. Each output meas‐

details.•
ures how much the image belongs to a particular digit class. The cross entropy 
cost function is now preferred, as it penalizes bad predictions much more, pro‐
ducing larger gradients and converging faster.CNN Architectures | 449

11“ImageNet Classification with Deep Convolutional Neural Networks, ” A. Krizhevsky et al. (2012).Y ann LeCun’s website  (“LENET” section) features great demos of LeNet-5 classifying 
digits.
AlexNet
The AlexNet  CNN architecture11 won the 2012 ImageNet ILSVRC challenge by a
large margin: it achieved 17% top-5 error rate while the second best achieved only
26%! It was developed by Alex Krizhevsky (hence the name), Ilya Sutskever, and
Geoffrey Hinton. It is quite similar to LeNet-5, only much larger and deeper, and it
was the first to stack convolutional layers directly on top of each other, instead of
stacking a pooling layer on top of each convolutional layer. Table 14-2  presents this

digits.
AlexNet
architecture.Table 14-2. AlexNet architectureLayerTypeMapsSizeKernel sizeStridePaddingActivationOutFully Connected–1,000–––SoftmaxF9Fully Connected–4,096–––ReLUF8Fully Connected–4,096–––ReLUC7Convolution25613 × 133 × 31SAMEReLUC6Convolution38413 × 133 × 31SAMEReLUC5Convolution38413 × 133 × 31SAMEReLUS4Max Pooling25613 × 133 × 32VALID–C3Convolution25627 × 275 × 51SAMEReLUS2Max Pooling9627 × 273 × 32VALID–C1Convolution9655 × 5511 × 114VALIDReLUInInput3 (RGB)227 × 227––––

To reduce overfitting, the authors used two regularization techniques: first they
applied dropout (introduced in Chapter 11 ) with a 50% dropout rate during training
to the outputs of layers F8 and F9. Second, they performed data augmentation  by ran‐
domly shifting the training images by various offsets, flipping them horizontally, and
changing the lighting conditions.
Data Augmentation
Data augmentation artificially increases the size of the training set by generating

many realistic variants of each training instance. This reduces overfitting, making this
a regularization technique. The generated instances should be as realistic as possible:450 | Chapter 14: Deep Computer Vision Using Convolutional Neural Networks

ideally, given an image from the augmented training set, a human should not be able
to tell whether it was augmented or not. Moreover, simply adding white noise will not
help; the modifications should be learnable (white noise is not).
For example, you can slightly shift, rotate, and resize every picture in the training set
by various amounts and add the resulting pictures to the training set (see
Figure 14-12 ). This forces the model to be more tolerant to variations in the position,
orientation, and size of the objects in the pictures. If you want the model to be more
tolerant to different lighting conditions, you can similarly generate many images with
various contrasts. In general, you can also flip the pictures horizontally (except for
text, and other non-symmetrical objects). By combining these transformations you
can greatly increase the size of your training set.
Figure 14-12. Generating new training instances from existing ones

AlexNet also uses a competitive normalization step immediately after the ReLU step
of layers C1 and C3, called local response normalization . The most strongly activated
neurons inhibit other neurons located at the same position in neighboring feature
maps (such competitive activation has been observed in biological neurons). This
encourages different feature maps to specialize, pushing them apart and forcing themCNN Architectures | 451

12“Going Deeper with Convolutions, ” C. Szegedy et al. (2015).13
In the 2010 movie Inception , the characters keep going deeper and deeper into multiple layers of dreams,
hence the name of these modules.to explore a wider range of features, ultimately improving generalization. Equation
14-2  shows how to apply LRN.
Equation 14-2. Local response normalizationbi=aik+α∑
j=jlowjhigh
aj2−β
withjhigh= min i+r
2,fn− 1
jlow= max 0,i−r
2•
bi is the normalized output of the neuron located in feature map i, at some row u
and column v (note that in this equation we consider only neurons located at this
row and column, so u and v are not shown).•
ai is the activation of that neuron after the ReLU step, but before normalization.•
k, α, β, and r are hyperparameters. k is called the bias, and r is called the depth
radius .•
fn is the number of feature maps.
For example, if r = 2 and a neuron has a strong activation, it will inhibit the activation
of the neurons located in the feature maps immediately above and below its own.

radius .•
In AlexNet, the hyperparameters are set as follows: r = 2, α = 0.00002, β = 0.75, and k
= 1. This step can be implemented using the tf.nn.local_response_normaliza
tion()  function (which you can wrap in a Lambda  layer if you want to use it in a
Keras model).
A variant of AlexNet called ZF Net  was developed by Matthew Zeiler and Rob Fergus
and won the 2013 ILSVRC challenge. It is essentially AlexNet with a few tweaked 
hyperparameters (number of feature maps, kernel size, stride, etc.).
GoogLeNet

GoogLeNet
The GoogLeNet architecture  was developed by Christian Szegedy et al. from Google
Research,12 and it won the ILSVRC 2014 challenge by pushing the top-5 error rate
below 7%. This great performance came in large part from the fact that the network
was much deeper than previous CNNs (see Figure 14-14 ). This was made possible by
sub-networks called inception modules ,13 which allow GoogLeNet to use parameters452 | Chapter 14: Deep Computer Vision Using Convolutional Neural Networks

much more efficiently than previous architectures: GoogLeNet actually has 10 times
fewer parameters than AlexNet (roughly 6 million instead of 60 million).
Figure 14-13  shows the architecture of an inception module. The notation “3 × 3 +
1(S)” means that the layer uses a 3 × 3 kernel, stride 1, and SAME padding. The input
signal is first copied and fed to four different layers. All convolutional layers use the
ReLU activation function. Note that the second set of convolutional layers uses differ‐
ent kernel sizes (1 × 1, 3 × 3, and 5 × 5), allowing them to capture patterns at different
scales. Also note that every single layer uses a stride of 1 and SAME padding (even
the max pooling layer), so their outputs all have the same height and width as their
inputs. This makes it possible to concatenate all the outputs along the depth dimen‐
sion in the final depth concat layer  (i.e., stack the feature maps from all four top con‐
volutional layers). This concatenation layer can be implemented in TensorFlow using

the tf.concat()  operation, with axis=3  (axis 3 is the depth).
Figure 14-13. Inception module
Y ou may wonder why inception modules have convolutional layers with 1 × 1 ker‐
nels. Surely these layers cannot capture any features since they look at only one pixel
at a time? In fact, these layers serve three purposes:•
First, although they cannot capture spatial patterns, they can capture patterns
along the depth dimension.•
Second, they are configured to output fewer feature maps than their inputs, so
they serve as bottleneck layers , meaning they reduce dimensionality. This cuts the

computational cost and the number of parameters, speeding up training and
improving generalization.•
Lastly, each pair of convolutional layers ([1 × 1, 3 × 3] and [1 × 1, 5 × 5]) acts like
a single, powerful convolutional layer, capable of capturing more complex pat‐
terns. Indeed, instead of sweeping a simple linear classifier across the image (as aCNN Architectures | 453

single convolutional layer does), this pair of convolutional layers sweeps a two-
layer neural network across the image.
In short, you can think of the whole inception module as a convolutional layer on
steroids, able to output feature maps that capture complex patterns at various scales.
The number of convolutional kernels for each convolutional layer
is a hyperparameter. Unfortunately, this means that you have six
more hyperparameters to tweak for every inception layer you add.
Now let’s look at the architecture of the GoogLeNet CNN (see Figure 14-14 ). The
number of feature maps output by each convolutional layer and each pooling layer is
shown before the kernel size. The architecture is so deep that it has to be represented
in three columns, but GoogLeNet is actually one tall stack, including nine inception
modules (the boxes with the spinning tops). The six numbers in the inception mod‐
ules represent the number of feature maps output by each convolutional layer in the

module (in the same order as in Figure 14-13 ). Note that all the convolutional layers
use the ReLU activation function.454 | Chapter 14: Deep Computer Vision Using Convolutional Neural Networks

Figure 14-14. GoogLeNet architecture
Let’s go through this network:•
The first two layers divide the image’s height and width by 4 (so its area is divided
by 16), to reduce the computational load. The first layer uses a large kernel size,
so that much of the information is still preserved.•
Then the local response normalization layer ensures that the previous layers learn
a wide variety of features (as discussed earlier).•
Two convolutional layers follow, where the first acts like a bottleneck layer . As
explained earlier, you can think of this pair as a single smarter convolutional
layer.•
Again, a local response normalization layer ensures that the previous layers cap‐
ture a wide variety of patterns.CNN Architectures | 455

14“Very Deep Convolutional Networks for Large-Scale Image Recognition, ” K. Simonyan and A. Zisserman
(2015).• Next a max pooling layer reduces the image height and width by 2, again to speed
up computations.•
Then comes the tall stack of nine inception modules, interleaved with a couple
max pooling layers to reduce dimensionality and speed up the net.•
Next, the global average pooling layer simply outputs the mean of each feature
map: this drops any remaining spatial information, which is fine since there was
not much spatial information left at that point. Indeed, GoogLeNet input images
are typically expected to be 224 × 224 pixels, so after 5 max pooling layers, each
dividing the height and width by 2, the feature maps are down to 7 × 7. More‐
over, it is a classification task, not localization, so it does not matter where the
object is. Thanks to the dimensionality reduction brought by this layer, there is
no need to have several fully connected layers at the top of the CNN (like in

AlexNet), and this considerably reduces the number of parameters in the net‐
work and limits the risk of overfitting.•
The last layers are self-explanatory: dropout for regularization, then a fully con‐
nected layer with 1,000 units, since there are a 1,000 classes, and a softmax acti‐
vation function to output estimated class probabilities.
This diagram is slightly simplified: the original GoogLeNet architecture also included
two auxiliary classifiers plugged on top of the third and sixth inception modules.

They were both composed of one average pooling layer, one convolutional layer, two
fully connected layers, and a softmax activation layer. During training, their loss
(scaled down by 70%) was added to the overall loss. The goal was to fight the vanish‐
ing gradients problem and regularize the network. However, it was later shown that
their effect was relatively minor.
Several variants of the GoogLeNet architecture were later proposed by Google
researchers, including Inception-v3 and Inception-v4, using slightly different incep‐
tion modules, and reaching even better performance.

VGGNet
The runner up in the ILSVRC 2014 challenge was  VGGNet14, developed by K. Simon‐
yan and A. Zisserman. It had a very simple and classical architecture, with 2 or 3 con‐
volutional layers, a pooling layer, then again 2 or 3 convolutional layers, a pooling
layer, and so on (with a total of just 16 convolutional layers), plus a final dense net‐
work with 2 hidden layers and the output layer. It used only 3 × 3 filters, but many
filters.456 | Chapter 14: Deep Computer Vision Using Convolutional Neural Networks

15“Deep Residual Learning for Image Recognition, ” K. He (2015).ResNet
The ILSVRC 2015 challenge was won using a Residual Network  (or ResNet ), devel‐
oped by Kaiming He et al.,15 which delivered an astounding top-5 error rate under
3.6%, using an extremely deep CNN composed of 152 layers. It confirmed the general
trend: models are getting deeper and deeper, with fewer and fewer parameters. The
key to being able to train such a deep network is to use skip connections  (also called
shortcut connections ): the signal feeding into a layer is also added to the output of a
layer located a bit higher up the stack. Let’s see why this is useful.
When training a neural network, the goal is to make it model a target function h(x).
If you add the input x to the output of the network (i.e., you add a skip connection),
then the network will be forced to model f(x) = h(x) – x rather than h(x). This is
called residual learning  (see Figure 14-15 ).
Figure 14-15. Residual learning

When you initialize a regular neural network, its weights are close to zero, so the net‐
work just outputs values close to zero. If you add a skip connection, the resulting net‐
work just outputs a copy of its inputs; in other words, it initially models the identity
function. If the target function is fairly close to the identity function (which is often
the case), this will speed up training considerably.
Moreover, if you add many skip connections, the network can start making progress

even if several layers have not started learning yet (see Figure 14-16 ). Thanks to skip
connections, the signal can easily make its way across the whole network. The deep
residual network can be seen as a stack of residual units , where each residual unit is a
small neural network with a skip connection.CNN Architectures | 457

Figure 14-16. Regular deep neural network (left)  and deep residual network (right)
Now let’s look at ResNet’s architecture (see Figure 14-17 ). It is actually surprisingly
simple. It starts and ends exactly like GoogLeNet (except without a dropout layer),
and in between is just a very deep stack of simple residual units. Each residual unit is
composed of two convolutional layers (and no pooling layer!), with Batch Normaliza‐
tion (BN) and ReLU activation, using 3 × 3 kernels and preserving spatial dimensions
(stride 1, SAME padding).
Figure 14-17. ResNet architecture
Note that the number of feature maps is doubled every few residual units, at the same
time as their height and width are halved (using a convolutional layer with stride 2).
When this happens the inputs cannot be added directly to the outputs of the residual
unit since they don’t have the same shape (for example, this problem affects the skip458 | Chapter 14: Deep Computer Vision Using Convolutional Neural Networks

16“Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning, ” C. Szegedy et al.
(2016).17
“Xception: Deep Learning with Depthwise Separable Convolutions, ” François Chollet (2016)
connection represented by the dashed arrow in Figure 14-17 ). To solve this problem,
the inputs are passed through a 1 × 1 convolutional layer with stride 2 and the right
number of output feature maps (see Figure 14-18 ).
Figure 14-18. Skip connection when changing feature map size and depth
ResNet-34 is the ResNet with 34 layers (only counting the convolutional layers and
the fully connected layer) containing three residual units that output 64 feature maps,
4 RUs with 128 maps, 6 RUs with 256 maps, and 3 RUs with 512 maps. We will imple‐
ment this architecture later in this chapter.
ResNets deeper than that, such as ResNet-152, use slightly different residual units.
Instead of two 3 × 3 convolutional layers with (say) 256 feature maps, they use three

convolutional layers: first a 1 × 1 convolutional layer with just 64 feature maps (4
times less), which acts as a bottleneck layer (as discussed already), then a 3 × 3 layer
with 64 feature maps, and finally another 1 × 1 convolutional layer with 256 feature
maps (4 times 64) that restores the original depth. ResNet-152 contains three such
RUs that output 256 maps, then 8 RUs with 512 maps, a whopping 36 RUs with 1,024
maps, and finally 3 RUs with 2,048 maps.
Google’s Inception-v416 architecture merged the ideas of GoogLe‐

Net and ResNet and achieved close to 3% top-5 error rate on
ImageNet classification.
Xception
Another variant of the GoogLeNet architecture is also worth noting: Xception17
(which stands for Extreme Inception ) was proposed in 2016 by François Chollet (theCNN Architectures | 459

18This name can sometimes be ambiguous, since spatially separable convolutions are often called “separable
convolutions” as well.author of Keras), and it significantly outperformed Inception-v3 on a huge vision task
(350 million images and 17,000 classes). Just like Inception-v4, it also merges the
ideas of GoogLeNet and ResNet, but it replaces the inception modules with a special
type of layer called a depthwise separable convolution  (or separable convolution  for
short18). These layers had been used before in some CNN architectures, but they were
not as central as in the Xception architecture. While a regular convolutional layer
uses filters that try to simultaneously capture spatial patterns (e.g., an oval) and cross-
channel patterns (e.g., mouth + nose + eyes = face), a separable convolutional layer
makes the strong assumption that spatial patterns and cross-channel patterns can be
modeled separately (see Figure 14-19 ). Thus, it is composed of two parts: the first part

applies a single spatial filter for each input feature map, then the second part looks
exclusively for cross-channel patterns—it is just a regular convolutional layer with 1 ×
1 filters.
Figure 14-19. Depthwise Separable Convolutional Layer
Since separable convolutional layers only have one spatial filter per input channel,
you should avoid using them after layers that have too few channels, such as the input
layer (granted, that’s what Figure 14-19  represents, but it is just for illustration pur‐

1 filters.
poses). For this reason, the Xception architecture starts with 2 regular convolutional
layers, but then the rest of the architecture uses only separable convolutions (34 in460 | Chapter 14: Deep Computer Vision Using Convolutional Neural Networks

19“Crafting GBD-Net for Object Detection, ” X. Zeng et al. (2016).20
“Squeeze-and-Excitation Networks, ” Jie Hu et al. (2017)
all), plus a few max pooling layers and the usual final layers (a global average pooling
layer, and a dense output layer).
Y ou might wonder why Xception is considered a variant of GoogLeNet, since it con‐
tains no inception module at all? Well, as we discussed earlier, an Inception module
contains convolutional layers with 1 × 1 filters: these look exclusively for cross-
channel patterns. However, the convolution layers that sit on top of them are regular
convolutional layers that look both for spatial and cross-channel patterns. So you can
think of an Inception module as an intermediate between a regular convolutional
layer (which considers spatial patterns and cross-channel patterns jointly) and a sepa‐
rable convolutional layer (which considers them separately). In practice, it seems that
separable convolutions generally perform better.

Separable convolutions use less parameters, less memory and less
computations than regular convolutional layers, and in general
they even perform better, so you should consider using them by
default (except after layers with few channels).
The ILSVRC 2016 challenge was won by the CUImage team from the Chinese Uni‐
versity of Hong Kong. They used an ensemble of many different techniques, includ‐
ing a sophisticated object-detection system called GBD-Net19, to achieve a top-5 error

rate below 3%. Although this result is unquestionably impressive, the complexity of
the solution contrasted with the simplicity of ResNets. Moreover, one year later
another fairly simple architecture performed even better, as we will see now.
SENet
The winning architecture in the ILSVRC 2017 challenge was the Squeeze-and-
Excitation Network  (SENet)20. This architecture extends existing architectures such as
inception networks or ResNets, and boosts their performance. This allowed SENet to

SENet
win the competition with an astonishing 2.25% top-5 error rate! The extended ver‐
sions of inception networks and ResNet are called SE-Inception  and SE-ResNet  respec‐
tively. The boost comes from the fact that a SENet adds a small neural network, called
a SE Block , to every unit in the original architecture (i.e., every inception module or
every residual unit), as shown in Figure 14-20 .CNN Architectures | 461

Figure 14-20. SE-Inception Module (left)  and SE-ResNet Unit (right)
A SE Block analyzes the output of the unit it is attached to, focusing exclusively on
the depth dimension (it does not look for any spatial pattern), and it learns which fea‐
tures are usually most active together. It then uses this information to recalibrate the
feature maps, as shown in Figure 14-21 . For example, a SE Block may learn that
mouths, noses and eyes usually appear together in pictures: if you see a mouth and a
nose, you should expect to see eyes as well. So if a SE Block sees a strong activation in
the mouth and nose feature maps, but only mild activation in the eye feature map, it
will boost the eye feature map (more accurately, it will reduce irrelevant feature
maps). If the eyes were somewhat confused with something else, this feature map
recalibration will help resolve the ambiguity.462 | Chapter 14: Deep Computer Vision Using Convolutional Neural Networks

Figure 14-21. An SE Block Performs Feature Map Recalibration
A SE Block is composed of just 3 layers: a global average pooling layer, a hidden dense
layer using the ReLU activation function, and a dense output layer using the sigmoid
activation function (see Figure 14-22 ):
Figure 14-22. SE Block ArchitectureCNN Architectures | 463

As earlier, the global average pooling layer computes the mean activation for each fea‐
ture map: for example, if its input contains 256 feature maps, it will output 256 num‐
bers representing the overall level of response for each filter. The next layer is where
the “squeeze” happens: this layer has much less than 256 neurons, typically 16 times
less than the number of feature maps (e.g., 16 neurons), so the 256 numbers get com‐
pressed into a small vector (e.g., 16 dimensional). This is a low-dimensional vector
representation (i.e., an embedding) of the distribution of feature responses. This bot‐
tleneck step forces the SE Block to learn a general representation of the feature com‐
binations (we will see this principle in action again when we discuss autoencoders
in ???). Finally, the output layer takes the embedding and outputs a recalibration vec‐
tor containing one number per feature map (e.g., 256), each between 0 and 1. The

feature maps are then multiplied by this recalibration vector, so irrelevant features
(with a low recalibration score) get scaled down while relevant features (with a recali‐
bration score close to 1) are left alone.
Implementing a ResNet-34 CNN Using Keras
Most CNN architectures described so far are fairly straightforward to implement
(although generally you would load a pretrained network instead, as we will see). To

illustrate the process, let’s implement a ResNet-34 from scratch using Keras. First, let’s
create a ResidualUnit  layer:
DefaultConv2D  = partial(keras.layers.Conv2D, kernel_size =3, strides=1,
                        padding="SAME", use_bias =False)
class ResidualUnit (keras.layers.Layer):
    def __init__ (self, filters, strides=1, activation ="relu", **kwargs):
        self.main_layers  = [

            DefaultConv2D (filters, strides=strides),
            keras.layers.BatchNormalization (),
            self.activation ,
            DefaultConv2D (filters),
            keras.layers.BatchNormalization ()]
        self.skip_layers  = []
        if strides > 1:
            self.skip_layers  = [
                DefaultConv2D (filters, kernel_size =1, strides=strides),
                keras.layers.BatchNormalization ()]
        Z = inputs
        for layer in self.main_layers :

keras.layers.BatchNormalization (),
        Z = inputs
        skip_Z = inputs
        for layer in self.skip_layers :464 | Chapter 14: Deep Computer Vision Using Convolutional Neural Networks

skip_Z = layer(skip_Z)
        return self.activation (Z + skip_Z)
As you can see, this code matches Figure 14-18  pretty closely. In the constructor, we
create all the layers we will need: the main layers are the ones on the right side of the
diagram, and the skip layers are the ones on the left (only needed if the stride is
greater than 1). Then in the call()  method, we simply make the inputs go through
the main layers, and the skip layers (if any), then we add both outputs and we apply
the activation function.
Next, we can build the ResNet-34 simply using a Sequential  model, since it is really
just a long sequence of layers (we can treat each residual unit as a single layer now
that we have the ResidualUnit  class):
model.add(DefaultConv2D (64, kernel_size =7, strides=2,
                        input_shape =[224, 224, 3]))
model.add(keras.layers.BatchNormalization ())
model.add(keras.layers.Activation ("relu"))

model.add(keras.layers.MaxPool2D (pool_size =3, strides=2, padding="SAME"))
prev_filters  = 64
for filters in [64] * 3 + [128] * 4 + [256] * 6 + [512] * 3:
    strides = 1 if filters == prev_filters  else 2
    model.add(ResidualUnit (filters, strides=strides))
    prev_filters  = filters
model.add(keras.layers.GlobalAvgPool2D ())
model.add(keras.layers.Flatten())
The only slightly tricky part in this code is the loop that adds the ResidualUnit  layers

to the model: as explained earlier, the first 3 RUs have 64 filters, then the next 4 RUs
have 128 filters, and so on. We then set the strides to 1 when the number of filters is
the same as in the previous RU, or else we set it to 2. Then we add the ResidualUnit ,
and finally we update prev_filters .
It is quite amazing that in less than 40 lines of code, we can build the model that won
the ILSVRC 2015 challenge! It demonstrates both the elegance of the ResNet model,
and the expressiveness of the Keras API. Implementing the other CNN architectures

is not much harder. However, Keras comes with several of these architectures built in,
so why not use them instead?
Using Pretrained Models From Keras
In general, you won’t have to implement standard models like GoogLeNet or ResNet
manually, since pretrained networks are readily available with a single line of code, in
the keras.applications  package. For example:
model = keras.applications .resnet50 .ResNet50 (weights="imagenet" )Using Pretrained Models From Keras | 465

21In the ImageNet dataset, each image is associated to a word in the WordNet dataset : the class ID is just a
WordNet ID.
That’s all! This will create a ResNet-50 model and download weights pretrained on
the ImageNet dataset. To use it, you first need to ensure that the images have the right
size. A ResNet-50 model expects 224 × 224 images (other models may expect other
sizes, such as 299 × 299), so let’s use TensorFlow’s tf.image.resize()  function to
resize the images we loaded earlier:
images_resized  = tf.image.resize(images, [224, 224])
The tf.image.resize()  will not preserve the aspect ratio. If this is
a problem, you can try cropping the images to the appropriate
aspect ratio before resizing. Both operations can be done in one
shot with tf.image.crop_and_resize() .
The pretrained models assume that the images are preprocessed in a specific way. In
some cases they may expect the inputs to be scaled from 0 to 1, or -1 to 1, and so on.

Each model provides a preprocess_input()  function that you can use to preprocess
your images. These functions assume that the pixel values range from 0 to 255, so we
must multiply them by 255 (since earlier we scaled them to the 0–1 range):
inputs = keras.applications .resnet50 .preprocess_input (images_resized  * 255)
Now we can use the pretrained model to make predictions:
Y_proba = model.predict(inputs)
As usual, the output Y_proba  is a matrix with one row per image and one column per

class (in this case, there are 1,000 classes). If you want to display the top K predic‐
tions, including the class name and the estimated probability of each predicted class,
you can use the decode_predictions()  function. For each image, it returns an array
containing the top K predictions, where each prediction is represented as an array
containing the class identifier21, its name and the corresponding confidence score:
top_K = keras.applications .resnet50 .decode_predictions (Y_proba, top=3)

for image_index  in range(len(images)):
    print("Image #{}" .format(image_index ))
    for class_id , name, y_proba in top_K[image_index ]:
        print("  {} - {:12s} {:.2f}%" .format(class_id , name, y_proba * 100))
    print()
The output looks like this:
Image #0
  n03877845 - palace       42.87%
  n02825657 - bell_cote    40.57%
  n03781244 - monastery    14.56%466 | Chapter 14: Deep Computer Vision Using Convolutional Neural Networks

Image #1
  n04522168 - vase         46.83%
  n07930864 - cup          7.78%
  n11939491 - daisy        4.87%
The correct classes (monastery and daisy) appear in the top 3 results for both images.
That’s pretty good considering that the model had to choose among 1,000 classes.
As you can see, it is very easy to create a pretty good image classifier using a pre‐
trained model. Other vision models are available in keras.applications , including
several ResNet variants, GoogLeNet variants like InceptionV3 and Xception,
VGGNet variants, MobileNet and MobileNetV2 (lightweight models for use in
mobile applications), and more.
But what if you want to use an image classifier for classes of images that are not part
of ImageNet? In that case, you may still benefit from the pretrained models to per‐
form transfer learning.
Pretrained Models for Transfer Learning
If you want to build an image classifier, but you do not have enough training data,

then it is often a good idea to reuse the lower layers of a pretrained model, as we dis‐
cussed in Chapter 11 . For example, let’s train a model to classify pictures of flowers,
reusing a pretrained Xception model. First, let’s load the dataset using TensorFlow
Datasets (see Chapter 13 ):
dataset, info = tfds.load("tf_flowers" , as_supervised =True, with_info =True)
dataset_size  = info.splits["train"].num_examples  # 3670

class_names  = info.features ["label"].names # ["dandelion", "daisy", ...]
n_classes  = info.features ["label"].num_classes  # 5
Note that you can get information about the dataset by setting with_info=True . Here,
we get the dataset size and the names of the classes. Unfortunately, there is only a
"train"  dataset, no test set or validation set, so we need to split the training set. The
TF Datasets project provides an API for this. For example, let’s take the first 10% of

the dataset for testing, the next 15% for validation, and the remaining 75% for train‐
ing:
test_split , valid_split , train_split  = tfds.Split.TRAIN.subsplit ([10, 15, 75])
test_set  = tfds.load("tf_flowers" , split=test_split , as_supervised =True)
valid_set  = tfds.load("tf_flowers" , split=valid_split , as_supervised =True)
train_set  = tfds.load("tf_flowers" , split=train_split , as_supervised =True)Pretrained Models for Transfer Learning | 467

Next we must preprocess the images. The CNN expects 224 × 224 images, so we need
to resize them. We also need to run the image through Xception’s prepro
cess_input()  function:
def preprocess (image, label):
    resized_image  = tf.image.resize(image, [224, 224])
    final_image  = keras.applications .xception .preprocess_input (resized_image )
    return final_image , label
Let’s apply this preprocessing function to all 3 datasets, and let’s also shuffle & repeat
the training set, and add batching & prefetching to all datasets:
train_set  = train_set .shuffle(1000).repeat()
train_set  = train_set .map(preprocess ).batch(batch_size ).prefetch (1)
valid_set  = valid_set .map(preprocess ).batch(batch_size ).prefetch (1)
test_set  = test_set .map(preprocess ).batch(batch_size ).prefetch (1)
If you want to perform some data augmentation, you can just change the preprocess‐
ing function for the training set, adding some random transformations to the training

images. For example, use tf.image.random_crop()  to randomly crop the images, use
tf.image.random_flip_left_right()  to randomly flip the images horizontally, and
so on (see the notebook for an example).
Next let’s load an Xception model, pretrained on ImageNet. We exclude the top of the
network (by setting include_top=False ): this excludes the global average pooling
layer and the dense output layer. We then add our own global average pooling layer,
based on the output of the base model, followed by a dense output layer with 1 unit

per class, using the softmax activation function. Finally, we create the Keras Model :
base_model  = keras.applications .xception .Xception (weights="imagenet" ,
                                                  include_top =False)
avg = keras.layers.GlobalAveragePooling2D ()(base_model .output)
output = keras.layers.Dense(n_classes , activation ="softmax" )(avg)
model = keras.models.Model(inputs=base_model .input, outputs=output)
As explained in Chapter 11 , it’s usually a good idea to freeze the weights of the pre‐

trained layers, at least at the beginning of training:
for layer in base_model .layers:
    layer.trainable  = False
Since our model uses the base model’s layers directly, rather than
the base_model  object itself, setting base_model.trainable=False
would have no effect.
Finally, we can compile the model and start training:468 | Chapter 14: Deep Computer Vision Using Convolutional Neural Networks

optimizer  = keras.optimizers .SGD(lr=0.2, momentum =0.9, decay=0.01)
model.compile(loss="sparse_categorical_crossentropy" , optimizer =optimizer ,
              metrics=["accuracy" ])
history = model.fit(train_set ,
                    steps_per_epoch =int(0.75 * dataset_size  / batch_size ),
                    validation_data =valid_set ,
                    validation_steps =int(0.15 * dataset_size  / batch_size ),
                    epochs=5)
This will be very slow, unless you have a GPU. If you do not, then
you should run this chapter’s notebook in Colab, using a GPU run‐
time (it’s free!). See the instructions at https://github.com/ageron/
handson-ml2 .
After training the model for a few epochs, its validation accuracy should reach about
75-80%, and stop making much progress. This means that the top layers are now
pretty well trained, so we are ready to unfreeze all layers (or you could try unfreezing
just the top ones), and continue training (don’t forget to compile the model when you

freeze or unfreeze layers). This time we use a much lower learning rate to avoid dam‐
aging the pretrained weights:
optimizer  = keras.optimizers .SGD(lr=0.01, momentum =0.9, decay=0.001)
model.compile(...)
history = model.fit(...)
It will take a while, but this model should reach around 95% accuracy on the test set.
With that, you can start training amazing image classifiers! But there’s more to com‐

puter vision than just classification. For example, what if you also want to know where
the flower is in the picture? Let’s look at this now.
Classification  and Localization
Localizing an object in a picture can be expressed as a regression task, as discussed in
Chapter 10 : to predict a bounding box around the object, a common approach is to
predict the horizontal and vertical coordinates of the object’s center, as well as its
height and width. This means we have 4 numbers to predict. It does not require much

change to the model, we just need to add a second dense output layer with 4 units
(typically on top of the global average pooling layer), and it can be trained using the
MSE loss:
class_output  = keras.layers.Dense(n_classes , activation ="softmax" )(avg)Classification  and Localization | 469

22“Crowdsourcing in Computer Vision, ” A. Kovashka et al. (2016).
loc_output  = keras.layers.Dense(4)(avg)
model = keras.models.Model(inputs=base_model .input,
                           outputs=[class_output , loc_output ])
model.compile(loss=["sparse_categorical_crossentropy" , "mse"],
              loss_weights =[0.8, 0.2], # depends on what you care most about
              optimizer =optimizer , metrics=["accuracy" ])
But now we have a problem: the flowers dataset does not have bounding boxes
around the flowers. So we need to add them ourselves. This is often one of the hard‐
est and most costly part of a Machine Learning project: getting the labels. It’s a good
idea to spend time looking for the right tools. To annotate images with bounding
boxes, you may want to use an open source image labeling tool like VGG Image
Annotator, LabelImg, OpenLabeler or ImgLab, or perhaps a commercial tool like
LabelBox or Supervisely. Y ou may also want to consider crowdsourcing platforms

such as Amazon Mechanical Turk or CrowdFlower if you have a very large number of
images to annotate. However, it is quite a lot of work to setup a crowdsourcing plat‐
form, prepare the form to be sent to the workers, to supervise them and ensure the
quality of the bounding boxes they produce is good, so make sure it is worth the
effort: if there are just a few thousand images to label, and you don’t plan to do this
frequently, it may be preferable to do it yourself. Adriana Kovashka et al. wrote a very

practical paper22 about crowdsourcing in Computer Vision, I recommend you check
it out, even if you do not plan to use crowdsourcing.
So let’s suppose you obtained the bounding boxes for every image in the flowers data‐
set (for now we will assume there is a single bounding box per image), you then need
to create a dataset whose items will be batches of preprocessed images along with
their class labels and their bounding boxes. Each item should be a tuple of the form:

(images, (class_labels, bounding_boxes)) . Then you are ready to train your
model!
The bounding boxes should be normalized so that the horizontal
and vertical coordinates, as well as the height and width all range
from 0 to 1. Also, it is common to predict the square root of the
height and width rather than the height and width directly: this
way, a 10 pixel error for a large bounding box will not be penalized
as much as a 10 pixel error for a small bounding box.

model!
The MSE often works fairly well as a cost function to train the model, but it is not a
great metric to evaluate how well the model can predict bounding boxes. The most
common metric for this is the Intersection over Union (IoU): it is the area of overlap
between the predicted bounding box and the target bounding box, divided by the470 | Chapter 14: Deep Computer Vision Using Convolutional Neural Networks

area of their union (see Figure 14-23 ). In tf.keras, it is implemented by the
tf.keras.metrics.MeanIoU  class.
Figure 14-23. Intersection over Union (IoU) Metric for Bounding Boxes
Classifying and localizing a single object is nice, but what if the images contain multi‐
ple objects (as is often the case in the flowers dataset)?
Object Detection
The task of classifying and localizing multiple objects in an image is called object
detection . Until a few years ago, a common approach was to take a CNN that was
trained to classify and locate a single object, then slide it across the image, as shown
in Figure 14-24 . In this example, the image was chopped into a 6 × 8 grid, and we
show a CNN (the thick black rectangle) sliding across all 3 × 3 regions. When the
CNN was looking at the top left of the image, it detected part of the left-most rose,
and then it detected that same rose again when it was first shifted one step to the

right. At the next step, it started detecting part of the top-most rose, and then it detec‐
ted it again once it was shifted one more step to the right. Y ou would then continue to
slide the CNN through the whole image, looking at all 3 × 3 regions. Moreover, since
objects can have varying sizes, you would also slide the CNN across regions of differ‐
ent sizes. For example, once you are done with the 3 × 3 regions, you might want to
slide the CNN across all 4 × 4 regions as well.Object Detection | 471

Figure 14-24. Detecting Multiple Objects by Sliding a CNN Across the Image
This technique is fairly straightforward, but as you can see it will detect the same
object multiple times, at slightly different positions. Some post-processing will then
be needed to get rid of all the unnecessary bounding boxes. A common approach for
this is called non-max suppression :•
First, you need to add an extra objectness  output to your CNN, to estimate the
probability that a flower is indeed present in the image (alternatively, you could
add a “no-flower” class, but this usually does not work as well). It must use the
sigmoid activation function and you can train it using the "binary_crossen
tropy"  loss. Then just get rid of all the bounding boxes for which the objectness
score is below some threshold: this will drop all the bounding boxes that don’t
actually contain a flower.•
Second, find the bounding box with the highest objectness score, and get rid of

all the other bounding boxes that overlap a lot with it (e.g., with an IoU greater
than 60%). For example, in Figure 14-24 , the bounding box with the max object‐
ness score is the thick bounding box over the top-most rose (the objectness score
is represented by the thickness of the bounding boxes). The other bounding box
over that same rose overlaps a lot with the max bounding box, so we will get rid
of it.472 | Chapter 14: Deep Computer Vision Using Convolutional Neural Networks

23“Fully Convolutional Networks for Semantic Segmentation, ” J. Long, E. Shelhamer, T. Darrell (2015).24
There is one small exception: a convolutional layer using V ALID padding will complain if the input size is
smaller than the kernel size.
• Third, repeat step two until there are no more bounding boxes to get rid of.
This simple approach to object detection works pretty well, but it requires running
the CNN many times, so it is quite slow. Fortunately, there is a much faster way to
slide a CNN across an image: using a Fully Convolutional Network .
Fully Convolutional Networks (FCNs)
The idea of FCNs was first introduced in a 2015 paper23 by Jonathan Long et al., for
semantic segmentation (the task of classifying every pixel in an image according to
the class of the object it belongs to). They pointed out that you could replace the
dense layers at the top of a CNN by convolutional layers. To understand this, let’s look
at an example: suppose a dense layer with 200 neurons sits on top of a convolutional

layer that outputs 100 feature maps, each of size 7 × 7 (this is the feature map size, not
the kernel size). Each neuron will compute a weighted sum of all 100 × 7 × 7 activa‐
tions from the convolutional layer (plus a bias term). Now let’s see what happens if we
replace the dense layer with a convolution layer using 200 filters, each 7 × 7, and with
V ALID padding. This layer will output 200 feature maps, each 1 × 1 (since the kernel
is exactly the size of the input feature maps and we are using V ALID padding). In

other words, it will output 200 numbers, just like the dense layer did, and if you look
closely at the computations performed by a convolutional layer, you will notice that
these numbers will be precisely the same as the dense layer produced. The only differ‐
ence is that the dense layer’s output was a tensor of shape [batch size, 200] while the
convolutional layer will output a tensor of shape [batch size, 1, 1, 200].
To convert a dense layer to a convolutional layer, the number of fil‐
ters in the convolutional layer must be equal to the number of units

in the dense layer, the filter size must be equal to the size of the
input feature maps, and you must use V ALID padding. The stride
may be set to 1 or more, as we will see shortly.
Why is this important? Well, while a dense layer expects a specific input size (since it
has one weight per input feature), a convolutional layer will happily process images of
any size24 (however, it does expect its inputs to have a specific number of channels,
since each kernel contains a different set of weights for each input channel). Since an

FCN contains only convolutional layers (and pooling layers, which have the same
property), it can be trained and executed on images of any size!Object Detection | 473

25This assumes we used only SAME padding in the network: indeed, V ALID padding would reduce the size of
the feature maps. Moreover, 448 can be neatly divided by 2 several times until we reach 7, without any round‐
ing error. If any layer uses a different stride than 1 or 2, then there may be some rounding error, so again the
feature maps may end up being smaller.For example, suppose we already trained a CNN for flower classification and localiza‐
tion. It was trained on 224 × 224 images and it outputs 10 numbers: outputs 0 to 4 are
sent through the softmax activation function, and this gives the class probabilities
(one per class); output 5 is sent through the logistic activation function, and this gives
the objectness score; outputs 6 to 9 do not use any activation function, and they rep‐
resent the bounding box’s center coordinates, and its height and width. We can now
convert its dense layers to convolutional layers. In fact, we don’t even need to retrain

it, we can just copy the weights from the dense layers to the convolutional layers!
Alternatively, we could have converted the CNN into an FCN before training.
Now suppose the last convolutional layer before the output layer (also called the bot‐
tleneck layer) outputs 7 × 7 feature maps when the network is fed a 224 × 224 image
(see the left side of Figure 14-25 ). If we feed the FCN a 448 × 448 image (see the right
side of Figure 14-25 ), the bottleneck layer will now output 14 × 14 feature maps.25

Since the dense output layer was replaced by a convolutional layer using 10 filters of
size 7 × 7, V ALID padding and stride 1, the output will be composed of 10 features
maps, each of size 8 × 8 (since 14 - 7 + 1 = 8). In other words, the FCN will process
the whole image only once and it will output an 8 × 8 grid where each cell contains 10
numbers (5 class probabilities, 1 objectness score and 4 bounding box coordinates).
It’s exactly like taking the original CNN and sliding it across the image using 8 steps

per row and 8 steps per column: to visualize this, imagine chopping the original
image into a 14 × 14 grid, then sliding a 7 × 7 window across this grid: there will be 8
× 8 = 64 possible locations for the window, hence 8 × 8 predictions. However, the
FCN approach is much  more efficient, since the network only looks at the image
once. In fact, You Only Look Once  (YOLO) is the name of a very popular object detec‐
tion architecture!474 | Chapter 14: Deep Computer Vision Using Convolutional Neural Networks

26“Y ou Only Look Once: Unified, Real-Time Object Detection, ” J. Redmon, S. Divvala, R. Girshick, A. Farhadi
(2015).27
“YOLO9000: Better, Faster, Stronger, ” J. Redmon, A. Farhadi (2016).28
“YOLOv3: An Incremental Improvement, ” J. Redmon, A. Farhadi (2018).
Figure 14-25. A Fully Convolutional Network Processing a Small Image (left)  and a
Large One (right)
You Only Look Once (YOLO)
YOLO is an extremely fast and accurate object detection architecture proposed by
Joseph Redmon et al. in a 2015 paper26, and subsequently improved in 201627
(YOLOv2) and in 201828 (YOLOv3). It is so fast that it can run in realtime on a video
(check out this nice demo ).
YOLOv3’s architecture is quite similar to the one we just discussed, but with a few
important differences:Object Detection | 475

•First, it outputs 5 bounding boxes for each grid cell (instead of just 1), and each
bounding box comes with an objectness score. It also outputs 20 class probabili‐
ties per grid cell, as it was trained on the PASCAL VOC dataset, which contains
20 classes. That’s a total of 45 numbers per grid cell (5 * 4 bounding box coordi‐
nates, plus 5 objectness scores, plus 20 class probabilities).•
Second, instead of predicting the absolute coordinates of the bounding box cen‐
ters, YOLOv3 predicts an offset relative to the coordinates of the grid cell, where
(0, 0) means the top left of that cell, and (1, 1) means the bottom right. For each
grid cell, YOLOv3 is trained to predict only bounding boxes whose center lies in
that cell (but the bounding box itself generally extends well beyond the grid cell).
YOLOv3 applies the logistic activation function to the bounding box coordinates
to ensure they remain in the 0 to 1 range.•
Third, before training the neural net, YOLOv3 finds 5 representative bounding

box dimensions, called anchor boxes  (or bounding box priors ): it does this by
applying the K-Means algorithm (see ???) to the height and width of the training
set bounding boxes. For example, if the training images contain many pedes‐
trians, then one of the anchor boxes will likely have the dimensions of a typical
pedestrian. Then when the neural net predicts 5 bounding boxes per grid cell, it
actually predicts how much to rescale each of the anchor boxes. For example,
suppose one anchor box is 100 pixels tall and 50 pixels wide, and the network

predicts, say, a vertical rescaling factor of 1.5 and a horizontal rescaling of 0.9 (for
one of the grid cells), this will result in a predicted bounding box of size 150 × 45
pixels. To be more precise, for each grid cell and each anchor box, the network
predicts the log of the vertical and horizontal rescaling factors. Having these pri‐
ors makes the network more likely to predict bounding boxes of the appropriate
dimensions, and it also speeds up training since it will more quickly learn what
reasonable bounding boxes look like.•

Fourth, the network is trained using images of different scales: every few batches
during training, the network randomly chooses a new image dimension (from
330 × 330 to 608 × 608 pixels). This allows the network to learn to detect objects
at different scales. Moreover, it makes it possible to use YOLOv3 at different
scales: the smaller scale will be less accurate but faster than the larger scale, so
you can choose the right tradeoff for your use case.
There are a few more innovations you might be interested in, such as the use of skip

connections to recover some of the spatial resolution that is lost in the CNN (we will
discuss this shortly when we look at semantic segmentation). Moreover, in the 2016
paper, the authors introduce the YOLO9000 model that uses hierarchical classifica‐
tion: the model predicts a probability for each node in a visual hierarchy called Word‐
Tree. This makes it possible for the network to predict with high confidence that an

image represents, say, a dog, even though it is unsure what specific type of dog it is.476 | Chapter 14: Deep Computer Vision Using Convolutional Neural Networks

So I encourage you to go ahead and read all three papers: they are quite pleasant to
read, and it is an excellent example of how Deep Learning systems can be incremen‐
tally improved.
Mean Average Precision (mAP)
A very common metric used in object detection tasks is the mean Average Precision
(mAP). “Mean Average” sounds a bit redundant, doesn’t it? To understand this met‐
ric, let’s go back to two classification metrics we discussed in Chapter 3 : precision and
recall. Remember the tradeoff: the higher the recall, the lower the precision. Y ou can
visualize this in a Precision/Recall curve (see Figure 3-5 ). To summarize this curve
into a single number, we could compute its Area Under the Curve (AUC). But note
that the Precision/Recall curve may contain a few sections where precision actually
goes up when recall increases, especially at low recall values (you can see this at the
top left of Figure 3-5 ). This is one of the motivations for the mAP metric.

Suppose the classifier has a 90% precision at 10% recall, but a 96% precision at 20%
recall: there’s really no tradeoff here: it simply makes more sense to use the classifier
at 20% recall rather than at 10% recall, as you will get both higher recall and higher
precision. So instead of looking at the precision at 10% recall, we should really be
looking at the maximum  precision that the classifier can offer with at least  10% recall.

It would be 96%, not 90%. So one way to get a fair idea of the model’s performance is
to compute the maximum precision you can get with at least 0% recall, then 10%
recall, 20%, and so on up to 100%, and then calculate the mean of these maximum
precisions. This is called the Average Precision  (AP) metric. Now when there are more
than 2 classes, we can compute the AP for each class, and then compute the mean AP
(mAP). That’s it!
However, in an object detection systems, there is an additional level of complexity:

what if the system detected the correct class, but at the wrong location (i.e., the
bounding box is completely off)? Surely we should not count this as a positive predic‐
tion. So one approach is to define an IOU threshold: for example, we may consider
that a prediction is correct only if the IOU is greater than, say, 0.5, and the predicted
class is correct. The corresponding mAP is generally noted mAP@0.5 (or mAP@50%,
or sometimes just AP50). In some competitions (such as the Pascal VOC challenge),
this is what is done. In others (such as the COCO competition), the mAP is computed

for different IOU thresholds (0.50, 0.55, 0.60, …, 0.95), and the final metric is the
mean of all these mAPs (noted AP@[.50:.95] or AP@[.50:0.05:.95]). Y es, that’s a mean
mean average.
Several YOLO implementations built using TensorFlow are available on github, some
with pretrained weights. At the time of writing, they are based on TensorFlow 1, but
by the time you read this, TF 2 implementations will certainly be available. Moreover,

other object detection models are available in the TensorFlow Models project, manyObject Detection | 477

29“SSD: Single Shot MultiBox Detector, ” Wei Liu et al. (2015).30
“Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, ” Shaoqing Ren et al.
(2015).with pretrained weights, and some have even been ported to TF Hub, making them
extremely easy to use, such as SSD29 and Faster-RCNN .30, which are both quite popu‐
lar. SSD is also a “single shot” detection model, quite similar to YOLO, while Faster R-
CNN is more complex: the image first goes through a CNN, and the output is passed
to a Region Proposal Network (RPN) which proposes bounding boxes that are most
likely to contain an object, and a classifier is run for each bounding box, based on the
cropped output of the CNN.
The choice of detection system depends on many factors: speed, accuracy, available
pretrained models, training time, complexity, etc. The papers contain tables of met‐
rics, but there is quite a lot of variability in the testing environments, and the technol‐

ogies evolve so fast that it is difficulty to make a fair comparison that will be useful for
most people and remain valid for more than a few months.
Great! So we can locate objects by drawing bounding boxes around them. But per‐
haps you might want to be a bit more precise. Let’s see how to go down to the pixel
level.
Semantic Segmentation
In semantic segmentation , each pixel is classified according to the class of the object it
belongs to (e.g., road, car, pedestrian, building, etc.), as shown in Figure 14-26 . Note

level.
that different objects of the same class are not distinguished. For example, all the bicy‐
cles on the right side of the segmented image end up as one big lump of pixels. The
main difficulty in this task is that when images go through a regular CNN, they grad‐
ually lose their spatial resolution (due to the layers with strides greater than 1): so a
regular CNN may end up knowing that there’s a person in the image, somewhere in
the bottom left of the image, but it will not be much more precise than that.478 | Chapter 14: Deep Computer Vision Using Convolutional Neural Networks

31This type of layer is sometimes referred to as a deconvolution layer , but it does not perform what mathemati‐
cians call a deconvolution, so this name should be avoided.
Figure 14-26. Semantic segmentation
Just like for object detection, there are many different approaches to tackle this prob‐
lem, some quite complex. However, a fairly simple solution was proposed in the 2015
paper by Jonathan Long et al. we discussed earlier. They start by taking a pretrained
CNN and turning into an FCN, as discussed earlier. The CNN applies a stride of 32 to
the input image overall (i.e., if you add up all the strides greater than 1), meaning the
last layer outputs feature maps that are 32 times smaller than the input image. This is
clearly too coarse, so they add a single upsampling layer  that multiplies the resolution
by 32. There are several solutions available for upsampling (increasing the size of an
image), such as bilinear interpolation, but it only works reasonably well up to ×4 or

×8. Instead, they used a transposed convolutional layer :31 it is equivalent to first
stretching the image by inserting empty rows and columns (full of zeros), then per‐
forming a regular convolution (see Figure 14-27 ). Alternatively, some people prefer to
think of it as a regular convolutional layer that uses fractional strides (e.g., 1/2 in
Figure 14-27 ). The transposed convolutional layer  can be initialized to perform some‐
thing close to linear interpolation, but since it is a trainable layer, it will learn to do
better during training.Semantic Segmentation | 479

Figure 14-27. Upsampling Using a Transpose Convolutional Layer
In a transposed convolution layer, the stride defines how much the
input will be stretched, not the size of the filter steps, so the larger
the stride, the larger the output (unlike for convolutional layers or
pooling layers).
TensorFlow Convolution Operations
TensorFlow also offers a few other kinds of convolutional layers:•
keras.layers.Conv1D  creates a convolutional layer for 1D inputs, such as time
series or text (sequences of letters or words), as we will see in ???.•
keras.layers.Conv3D  creates a convolutional layer for 3D inputs, such as 3D
PET scan.•
Setting the dilation_rate  hyperparameter of any convolutional layer to a value
of 2 or more creates an à-trous convolutional layer  (“à trous” is French for “with
holes”). This is equivalent to using a regular convolutional layer with a filter dila‐
ted by inserting rows and columns of zeros (i.e., holes). For example, a 1 × 3 filter

PET scan.•
equal to [[1,2,3]]  may be dilated with a dilation rate  of 4, resulting in a dilated
filter  [[1, 0, 0, 0, 2, 0, 0, 0, 3]] . This allows the convolutional layer to480 | Chapter 14: Deep Computer Vision Using Convolutional Neural Networks

have a larger receptive field at no computational price and using no extra param‐
eters.•
tf.nn.depthwise_conv2d()  can be used to create a depthwise convolutional layer
(but you need to create the variables yourself). It applies every filter to every
individual input channel independently. Thus, if there are fn filters and fn′ input
channels, then this will output fn × fn′ feature maps.
This solution is okay, but still too imprecise. To do better, the authors added skip con‐
nections from lower layers: for example, they upsampled the output image by a factor
of 2 (instead of 32), and they added the output of a lower layer that had this double
resolution. Then they upsampled the result by a factor of 16, leading to a total upsam‐
pling factor of 32 (see Figure 14-28 ). This recovered some of the spatial resolution
that was lost in earlier pooling layers. In their best architecture, they used a second
similar skip connection to recover even finer details from an even lower layer: in

short, the output of the original CNN goes through the following extra steps: upscale
×2, add the output of a lower layer (of the appropriate scale), upscale ×2, add the out‐
put of an even lower layer, and finally upscale ×8. It is even possible to scale up
beyond the size of the original image: this can be used to increase the resolution of an
image, which is a technique called super-resolution .
Figure 14-28. Skip layers recover some spatial resolution from lower layers
Once again, many github repositories provide TensorFlow implementations of

semantic segmentation (TensorFlow 1 for now), and you will even find a pretrained
instance segmentation  model in the TensorFlow Models project. Instance segmenta‐
tion is similar to semantic segmentation, but instead of merging all objects of the
same class into one big lump, each object is distinguished from the others (e.g., it
identifies each individual bicycle). At the present, they provide multiple implementa‐
tions of the Mask R-CNN  architecture, which was proposed in a 2017 paper : it

extends the Faster R-CNN model by additionally producing a pixel-mask for each
bounding box. So not only do you get a bounding box around each object, with a set
of estimated class probabilities, you also get a pixel mask that locates pixels in the
bounding box that belong to the object.Semantic Segmentation | 481

32“Matrix Capsules with EM Routing, ” G. Hinton, S. Sabour, N. Frosst (2018).As you can see, the field of Deep Computer Vision is vast and moving fast, with all
sorts of architectures popping out every year, all based on Convolutional Neural Net‐
works. The progress made in just a few years has been astounding, and researchers
are now focusing on harder and harder problems, such as adversarial learning  (which
attempts to make the network more resistant to images designed to fool it), explaina‐
bility (understanding why the network makes a specific classification), realistic image
generation  (which we will come back to in ???), single-shot learning  (a system that can
recognize an object after it has seen it just once), and much more. Some even explore
completely novel architectures, such as Geoffrey Hinton’s capsule networks32 (I pre‐
sented them in a couple videos , with the corresponding code in a notebook). Now on
to the next chapter, where we will look at how to process sequential data such as time

series using Recurrent Neural Networks and Convolutional Neural Networks.
What are the advantages of a CNN over a fully connected DNN for image classi‐
fication?2.
Consider a CNN composed of three convolutional layers, each with 3 × 3 kernels,
a stride of 2, and SAME padding. The lowest layer outputs 100 feature maps, the
middle one outputs 200, and the top one outputs 400. The input images are RGB
images of 200 × 300 pixels. What is the total number of parameters in the CNN?
If we are using 32-bit floats, at least how much RAM will this network require

when making a prediction for a single instance? What about when training on a
mini-batch of 50 images?3.
If your GPU runs out of memory while training a CNN, what are five things you
could try to solve the problem?4.
Why would you want to add a max pooling layer rather than a convolutional
layer with the same stride?5.
When would you want to add a local response normalization  layer?6.
Can you name the main innovations in AlexNet, compared to LeNet-5? What

about the main innovations in GoogLeNet, ResNet, SENet and Xception?7.
What is a Fully Convolutional Network? How can you convert a dense layer into
a convolutional layer?8.
What is the main technical difficulty of semantic segmentation?9.
Build your own CNN from scratch and try to achieve the highest possible accu‐
racy on MNIST.482 | Chapter 14: Deep Computer Vision Using Convolutional Neural Networks

10. Use transfer learning for large image classification. a.
Create a training set containing at least 100 images per class. For example, you
could classify your own pictures based on the location (beach, mountain, city,
etc.), or alternatively you can just use an existing dataset (e.g., from Tensor‐
Flow Datasets).b.
Split it into a training set, a validation set and a test set.c.
Build the input pipeline, including the appropriate preprocessing operations,
and optionally add data augmentation.d.
Fine-tune a pretrained model on this dataset.11.
Go through TensorFlow’s DeepDream tutorial . It is a fun way to familiarize your‐
self with various ways of visualizing the patterns learned by a CNN, and to gener‐
ate art using Deep Learning.
Solutions to these exercises are available in ???.Exercises | 483

About the Author
Aurélien Géron  is a Machine Learning consultant. A former Googler, he led the Y ou‐
Tube video classification team from 2013 to 2016. He was also a founder and CTO of
Wifirst from 2002 to 2012, a leading Wireless ISP in France; and a founder and CTO
of Polyconseil in 2001, the firm that now manages the electric car sharing service
Autolib’ .
Before this he worked as an engineer in a variety of domains: finance (JP Morgan and
Société Générale), defense (Canada’s DOD), and healthcare (blood transfusion). He
published a few technical books (on C++, WiFi, and internet architectures), and was
a Computer Science lecturer in a French engineering school.
A few fun facts: he taught his three children to count in binary with their fingers (up
to 1023), he studied microbiology and evolutionary genetics before going into soft‐
ware engineering, and his parachute didn’t open on the second jump.
Colophon
The animal on the cover of Hands-On Machine Learning with Scikit-Learn and Ten‐

Colophon
sorFlow  is the fire salamander ( Salamandra salamandra ), an amphibian found across
most of Europe. Its black, glossy skin features large yellow spots on the head and
back, signaling the presence of alkaloid toxins. This is a possible source of this
amphibian’s common name: contact with these toxins (which they can also spray
short distances) causes convulsions and hyperventilation. Either the painful poisons
or the moistness of the salamander’s skin (or both) led to a misguided belief that these

creatures not only could survive being placed in fire but could extinguish it as well.
Fire salamanders live in shaded forests, hiding in moist crevices and under logs near
the pools or other freshwater bodies that facilitate their breeding. Though they spend
most of their life on land, they give birth to their young in water. They subsist mostly
on a diet of insects, spiders, slugs, and worms. Fire salamanders can grow up to a foot
in length, and in captivity, may live as long as 50 years.

The fire salamander’s numbers have been reduced by destruction of their forest habi‐
tat and capture for the pet trade, but the greatest threat is the susceptibility of their
moisture-permeable skin to pollutants and microbes. Since 2014, they have become
extinct in parts of the Netherlands and Belgium due to an introduced fungus.
Many of the animals on O’Reilly covers are endangered; all of them are important to
the world. To learn more about how you can help, go to animals.oreilly.com .

The cover image is from Wood’s Illustrated Natural History . The cover fonts are URW
Typewriter and Guardian Sans. The text font is Adobe Minion Pro; the heading font
is Adobe Myriad Condensed; and the code font is Dalton Maag’s Ubuntu Mono.

Contents IIContents 
 
Overview         viii List of Illustrations        xiv Words of Thanks        xix  
Part I: GEB 
Introduction: A Musico-Logical Offering    3  
Three-Part Invention      29 
Chapter I: The MU-puzzle      33 
Two-Part Invention      43 
Chapter II: Meaning and Form in Mathematics   46 
Sonata for Unaccompanied Achilles    61 
Chapter III: Figure and Ground     64 
Contracrostipunctus      75 
Chapter IV: Consistency, Completeness, and Geometry  82 
Little Harmonic Labyrinth     103 
Chapter V: Recursive Structures and Processes   127 
Canon by Intervallic Augmentation    153 
Chapter VI: The Location of Meaning    158   
Chromatic Fantasy, And Feud    177  
Chapter VII: The Propositional Calculus    181   
Crab Canon       199 
Chapter VIII: Typographical Number Theory   204 
A Mu Offering      231 
Chapter IX: Mumon and Gödel     246

data/pdfs/Gödel, Escher, Bach-An Eternal Golden Braid (1999).pdf

Contents III 
Part II EGB  
 
Prelude ...        275 
Chapter X: Levels of Description, and Computer Systems   285 
Ant Fugue        311 
Chapter XI: Brains and Thoughts      337 
English French German Suit      366 
Chapter XII: Minds and Thoughts      369 
Aria with Diverse Variations      391 
Chapter XIII: BlooP and FlooP and GlooP     406 
Air on G's String       431 
Chapter XIV: On Formally Undecidable Propositions of TNT         and Related Systems       438 
Birthday Cantatatata ...      461 
Chapter XV: Jumping out of the System     465 
Edifying Thoughts of a Tobacco Smoker    480 
Chapter XVI: Self-Ref and Self-Rep      495 
The Magn fierab, Indeed      549 
Chapter XVII: Church, Turing, Tarski, and Others    559 
SHRDLU, Toy of Man's Designing     586 
Chapter XVIII: Artificial Intelligence: Retrospects    594 
Contrafactus        633 
Chapter XIX: Artificial Intelligence: Prospects    641 
Sloth Canon        681 
Chapter XX: Strange Loops, Or Tangled Hierarchies   684

Chapter XX: Strange Loops, Or Tangled Hierarchies   684 
Six-Part Ricercar       720 
   Notes          743 Bibliography         746 Credits          757 Index          759

Overview IVOverview 
 
Part I: GEB 
 
Introduction: A Musico-Logical Offering . The book opens with the story of Bach's Musical 
Offering. Bach made an impromptu visit to King Frederick the Great of Prussia, and was 
requested to improvise upon a theme presented by th e King. His improvisations formed the basis 
of that great work. The Musical Offering and its story form a theme upon which I "improvise" throughout the book, thus making a sort of "Me tamusical Offering". Self-reference and the 
interplay between different levels in Bach ar e discussed: this leads to a discussion of parallel 
ideas in Escher's drawings and then Gödel’s Theo rem. A brief presentation of the history of logic 
and paradoxes is given as background for Gödel’s Th eorem. This leads to mechanical reasoning 
and computers, and the debate about whether Artif icial Intelligence is possible. I close with an 
explanation of the origins of the book-particularly the why and wherefore of the Dialogues.

explanation of the origins of the book-particularly the why and wherefore of the Dialogues. 
 Three-Part Invention.  Bach wrote fifteen three-part inventions. In this three-part Dialogue, the 
Tortoise and Achilles-the main fictional protagonists in the Dialogues-are "invented" by Zeno (as 
in fact they were, to illustrate Zeno's paradoxes of motion). Very short, it simply gives the flavor of the Dialogues to come. 
 Chapter I: The MU-puzzle.  A simple formal system (the MIL'-system) is presented, and the reader 
is urged to work out a puzzle to gain familiarity with formal systems in general. A number of

is urged to work out a puzzle to gain familiarity with formal systems in general. A number of 
fundamental notions are introduced: string, theorem, axiom, rule of inference, derivation, formal 
system, decision procedure, working inside/outside the system. 
 Two-Part Invention.  Bach also wrote fifteen two-part inventions. This two-part Dialogue was written 
not by me, but by Lewis Carroll in 1895. Carroll borrowed Achilles and the Tortoise from Zeno, 
and I in turn borrowed them from Carroll. The t opic is the relation between reasoning, reasoning 
about reasoning, reasoning about reasoning about r easoning, and so on. It parallels, in a way,

about reasoning, reasoning about reasoning about r easoning, and so on. It parallels, in a way, 
Zeno's paradoxes about the impossibility of motion,  seeming to show, by using infinite regress, 
that reasoning is impossible. It is a beautiful pa radox, and is referred to several times later in the 
book. 
 
Chapter II: Meaning and Form in Mathematics.  A new formal system (the pq-system) is 
presented, even simpler than the MIU-system of Chapter I. Apparently meaningless at first, its 
symbols are suddenly revealed to possess meaning by  virtue of the form of the theorems they

book. 
 
symbols are suddenly revealed to possess meaning by  virtue of the form of the theorems they 
appear in. This revelation is the first important  insight into meaning: its deep connection to 
isomorphism. Various issues related to meaning ar e then discussed, such as truth, proof, symbol 
manipulation, and the elusive concept, "form". 
 Sonata for Unaccompanied Achilles . A Dialogue which imitates the Bach Sonatas for 
unaccompanied violin. In particular, Achilles is the only speaker, since it is a transcript of one 
end of a telephone call, at the far end of which is the Tortoise. Their conversation concerns the

end of a telephone call, at the far end of which is the Tortoise. Their conversation concerns the 
concepts of "figure" and "ground" in various

Overview V 
contexts- e.g., Escher's art. The Dialogue itsel f forms an example of the distinction, since 
Achilles' lines form a "figure", and the Tortoise' s lines-implicit in Achilles' lines-form a "ground". 
 
Chapter III: Figure and Ground . The distinction between figure and ground in art is compared to 
the distinction between theorems and nontheorem s in formal systems. The question "Does a 
figure necessarily contain the same information as its ground%" leads to the distinction between 
recursively enumerable sets and recursive sets. 
 Contracrostipunctus . This Dialogue is central to the book, for it contains a set of paraphrases of 
Gödel’s self-referential construction and of his Incompleteness Theorem. One of the paraphrases 
of the Theorem says, "For each record player there is a record which it cannot play." The 
Dialogue's title is a cross between the word "acr ostic" and the word "contrapunctus", a Latin word

Dialogue's title is a cross between the word "acr ostic" and the word "contrapunctus", a Latin word 
which Bach used to denote the many fugues and canons making up his Art of the Fugue . Some 
explicit references to the Art of the Fugue are made. The Dialogue itself conceals some acrostic 
tricks. 
 Chapter IV: Consistency, Completeness, and Geometry . The preceding Dialogue is explicated to 
the extent it is possible at this stage. This lead s back to the question of how and when symbols in 
a formal system acquire meaning. The history of Euclidean and non-Euclidean geometry is given,

tricks. 
a formal system acquire meaning. The history of Euclidean and non-Euclidean geometry is given, 
as an illustration of the elusive notion of "unde fined terms". This leads to ideas about the 
consistency of different and possibly "rival" geometries. Through this discussion the notion of 
undefined terms is clarified, and the relati on of undefined terms to perception and thought 
processes is considered. 
 Little Harmonic Labyrinth . This is based on the Bach organ piece by the same name. It is a playful 
introduction to the notion of recursive-i.e., nested st ructures. It contains stories within stories. The

introduction to the notion of recursive-i.e., nested st ructures. It contains stories within stories. The 
frame story, instead of finishing as expected, is left open, so the reader is left dangling without resolution. One nested story concerns modulation in music-particularly an organ piece which ends in the wrong key, leaving the listener dangling without resolution. 
 
Chapter V: Recursive Structures and Processes . The idea of recursion is presented in many 
different contexts: musical patterns, linguistic patterns, geometric structures, mathematical 
functions, physical theories, computer programs, and others.

functions, physical theories, computer programs, and others. 
 Canon by Intervallic Augmentation . Achilles and the Tortoise try to  resolve the question, "Which 
contains more information-a record, or the phonograph which plays it This odd question arises 
when the Tortoise describes a si ngle record which, when played on a set of different 
phonographs, produces two quite different melodies: B-A-C-H and C-A-G-E. It turns out, however, that these melodies are "t he same", in a peculiar sense. 
 Chapter VI: The Location of Meaning.  A broad discussion of how meaning is split among coded 
message, decoder, and receiver. Examples presented include strands of DNA, undeciphered 
inscriptions on ancient tablets, and phonograph reco rds sailing out in space. The relationship of 
intelligence to "absolute" meaning is postulated.

intelligence to "absolute" meaning is postulated. 
 Chromatic Fantasy, And Feud . A short Dialogue bearing hardly any resemblance, except in title, to 
Bach's Chromatic Fantasy and Fugue . It concerns the proper way to manipulate sentences so as 
to preserve truth-and in particular the question

Overview VIof whether there exist rules for the usage of th e word "arid". This Dialogue has much in common 
with the Dialogue by Lewis Carroll.  
Chapter VII: The Propositional Calculus.  It is suggested how words such as .,and" can be 
governed by formal rules. Once again, the ideas of isomorphism and automatic acquisition of meaning by symbols in such a system are br ought up. All the examples in this Chapter, 
incidentally, are "Zentences"-sentences take n from Zen koans. This is purposefully done, 
somewhat tongue-in-cheek, since Zen koans are deliberately illogical stories. 
 
Crab Canon . A Dialogue based on a piece by the same name from the Musical Offering . Both are so 
named because crabs (supposedly) walk  backwards. The Crab makes his first appearance in this 
Dialogue. It is perhaps the densest Dialogue in the book in terms of formal trickery and level-
play. Gödel, Escher, and Bach are deeply intertwined in this very short Dialogue.

play. Gödel, Escher, and Bach are deeply intertwined in this very short Dialogue. 
 
Chapter VIII: Typographical Number Theory . An extension of the Pr opositional Calculus called 
"TNT" is presented. In TNT, number-theoreti cal reasoning can be done by rigid symbol 
manipulation. Differences between formal reasoning and human thought are considered. 
 A Mu Offering . This Dialogue foreshadows several new topics in the book. Ostensibly concerned 
with Zen Buddhism and koans, it is actually a thinly veiled discussion of theoremhood and 
nontheoremhood, truth and falsity, of strings in number theory. There are fleeting references to

nontheoremhood, truth and falsity, of strings in number theory. There are fleeting references to 
molecular biology-particular) the Genetic Code. There is no close affinity to the Musical 
Offering,  other than in the title and the playing of self-referential games. 
 Chapter IX: Mumon and Gödel.  An attempt is made to talk about the strange ideas of Zen 
Buddhism. The Zen monk Mumon, who gave we ll known commentaries on many koans, is a 
central figure. In a way, Zen ideas bear a me taphorical resemblance to some contemporary ideas 
in the philosophy of mathematics. After this "Zennery", Gödel’s fundamental idea of Gödel-

numbering is introduced, and a first p ass through Gödel’s Theorem is made. 
 
Part II: EGB 
 Prelude  ... This Dialogue attaches to the next one . They are based on preludes and fugues from 
Bach's Well-Tempered Clavier . Achilles and the Tortoise bring a present to the Crab, who has a 
guest: the Anteater. The present turns out to be a recording of the W.T.C.; it is immediately put 
on. As they listen to a prelude, they discuss th e structure of preludes and fugues, which leads

on. As they listen to a prelude, they discuss th e structure of preludes and fugues, which leads 
Achilles to ask how to hear a fugue: as a whole, or  as a sum of parts? This is the debate between 
holism and reductionism, which is soon taken up in the Ant Fugue. 
 
Chapter X: Levels of Description, and Computer Systems . Various levels of seeing pictures, 
chessboards, and computer systems are discussed. Th e last of these is then examined in detail. 
This involves describing machine languages, assembly languages, compiler languages, operating

This involves describing machine languages, assembly languages, compiler languages, operating 
systems, and so forth. Then the discussion turns to composite systems of other types, such as 
sports teams, nuclei, atoms, the weather, and so forth. The question arises as to how man 
intermediate levels exist-or indeed whether any exist.

Overview VII…Ant Fugue.  An imitation of a musical fugue: each voi ce enters with the same statement. The 
theme-holism versus reductionism-is introduced in a recursive picture composed of words 
composed of smaller words. etc. The words which appear on the four levels of this strange picture 
are "HOLISM", "REDLCTIONIsM", and "ML". The discussion veers off to a friend of the 
Anteater's Aunt Hillary, a conscious ant colony . The various levels of her thought processes are 
the topic of discussion. Many fugal tricks are ensc onced in the Dialogue. As a hint to the reader, 
references are made to parallel tricks occurring in  the fugue on the record to which the foursome 
is listening. At the end of the Ant Fugue, themes from the Prelude return. transformed considerably. 
 
Chapter XI: Brains and Thoughts . "How can thoughts he supported by the hardware of the brain is 
the topic of the Chapter. An overview of the la rge scale and small-scale structure of the brain is

the topic of the Chapter. An overview of the la rge scale and small-scale structure of the brain is 
first given. Then the relation between concepts and neural activity is speculatively discussed in 
some detail. 
 
English French German Suite.  An interlude consisting of Lewis Carroll's nonsense poem 
"Jabberwocky`' together with two translations: one into French and one into German, both done 
last century. 
 
Chapter XII: Minds and Thoughts.  The preceding poems bring up in a forceful way the question 
of whether languages, or indeed minds, can be "mapped" onto each other. How is communication

of whether languages, or indeed minds, can be "mapped" onto each other. How is communication 
possible between two separate physical brains: What do all human brains have in common? A geographical analogy is used to suggest an answer. The question arises, "Can a brain be 
understood, in some objective sense, by an outsider?" 
 
Aria with Diverse Variations . A Dialogue whose form is based on Bach's Goldberg Variations, and 
whose content is related to number-theoretical problems such as the Goldbach conjecture. This

whose content is related to number-theoretical problems such as the Goldbach conjecture. This 
hybrid has as its main purpose to show how numbe r theory's subtlety stems from the fact that 
there are many diverse variations on the theme of searching through an infinite space. Some of 
them lead to infinite searches, some of them l ead to finite searches, while some others hover in 
between. 
 Chapter XIII: BlooP and FlooP and GlooP . These are the names of three computer languages. 
BlooP programs can carry out only predictably finite searches, while FlooP programs can carry out unpredictable or even infinite searches. The purpo se of this Chapter is to give an intuition for

between. 
BlooP programs can carry out only predictably finite searches, while FlooP programs can carry out unpredictable or even infinite searches. The purpo se of this Chapter is to give an intuition for 
the notions of primitive recursive and general recu rsive functions in number theory, for they are 
essential in Gödel’s proof. 
 Air on G's String . A Dialogue in which Gödel’s self-referen tial construction is mirrored in words. 
The idea is due to W. V. O. Quine. This Dial ogue serves as a prototype for the next Chapter. 
 
Chapter XIV: On Formally Undecidable Pr opositions of TNT and Related Systems . This 
Chapter's title is an adaptation of the title of Gödel’s 1931 article, in which his Incompleteness

Chapter's title is an adaptation of the title of Gödel’s 1931 article, in which his Incompleteness 
Theorem was first published. The two major parts of Gödel’s proof are gone through carefully. It 
is shown how the assumption of consistency of TNT forces one to conclude that TNT (or any 
similar system) is incomplete. Relations to Eu clidean and non-Euclidean geometry are discussed. 
Implications for the philosophy of mathematics are gone into with some care.

Overview VIIIBirthday Cantatatata ... In which Achilles cannot convi nce the wily and skeptical Tortoise that today 
is his (Achilles') birthday. His repeated but  unsuccessful tries to do so foreshadow the 
repeatability of the Gödel argument. 
 
Chapter XV: Jumping out of the System.  The repeatability of Gödel’s argument is shown, with 
the implication that TNT is not only incomplete, but "essentially incomplete The fairly notorious 
argument by J. R. Lucas, to the effect that Gödel’s Theorem demonstrates that human thought cannot in any sense be "mechanical", is analyzed and found to be wanting. 
 
Edifying Thoughts of a Tobacco Smoker . A Dialogue treating of many topics, with the thrust being 
problems connected with self-replication and self-reference. Television cameras filming television screens, and viruses and other sub cellular entities which assemble themselves, are 
among the examples used. The title comes from a poem by  J. S. Bach himself, which enters in a 
peculiar way.

peculiar way. 
 
Chapter XVI: Self-Ref and Self-Rep . This Chapter is about the connection between self-reference 
in its various guises, and self-reproducing entities e.g., computer programs or DNA molecules). The relations between a self-reproducing entity and th e mechanisms external to it which aid it in 
reproducing itself (e.g., a computer or proteins ) are discussed-particularly the fuzziness of the 
distinction. How information travels between various levels of such systems is the central topic of 
this Chapter. 
 The Magnificrab, Indeed.  The title is a pun on Bach's Magnifacat in D. The tale is about the Crab,

 The Magnificrab, Indeed.  The title is a pun on Bach's Magnifacat in D. The tale is about the Crab, 
who gives the appearance of having a magical pow er of distinguishing between true and false 
statements of number theory by reading them as musical pieces, playing them on his flute, and 
determining whether they are "beautiful" or not. 
 Chapter XVII: Church, Turing, Tarski, and Others . The fictional Crab of the preceding Dialogue 
is replaced by various real people with amazing mathematical abilities. The Church-Turing Thesis, which relates mental activity to computati on, is presented in several versions of differing 
strengths. All are analyzed, particularly in terms of their implications for simulating human

strengths. All are analyzed, particularly in terms of their implications for simulating human 
thought mechanically, or programming into a mach ine an ability to sense or create beauty. The 
connection between brain activity and computa tion brings up some other topics: the halting 
problem of Turing, and Tarski's Truth Theorem. 
 SHRDLU, Toy of Man's Designing . This Dialogue is lifted out of an article by Terry Winograd on 
his program SHRDLU : only a few names have been changed. In it. a program communicates 
with a person about the so-called "blocks world"  in rather impressive English. The computer

with a person about the so-called "blocks world"  in rather impressive English. The computer 
program appears to exhibit some real understandi ng-in its limited world. The Dialogue's title is 
based on Jesu, joy of Mans Desiring , one movement of Bach's Cantata 147. 
 
Chapter XVIII: Artificial Intelligence: Retrospects , This Chapter opens with a discussion of the 
famous "Turing test"-a proposal by the computer pioneer Alan Turing for a way to detect the presence or absence of "thought" in a machine. From there, we go on to an abridged history of Artificial Intelligence. This covers programs that can-to some degree-play games, prove

famous "Turing test"-a proposal by the computer pioneer Alan Turing for a way to detect the presence or absence of "thought" in a machine. From there, we go on to an abridged history of Artificial Intelligence. This covers programs that can-to some degree-play games, prove 
theorems, solve problems, compose music, do ma thematics, and use "natural language" (e.g., 
English).

Overview IXContrafactus . About how we unconsciously organize our thoughts so that we can imagine 
hypothetical variants on the real world all the time.  Also about aberrant variants of this ability-
such as possessed by the new character , the Sloth, an avid lover of French fries, and rabid hater of 
counterfactuals. 
 Chapter XIX: Artificial Intelligence: Prospects . The preceding Dialogue triggers a discussion of 
how knowledge is represented in laye rs of contexts. This leads to the modern Al idea of "frames". 
A frame-like way of handling a set of visual pattern puzzles is presented, for the purpose of 
concreteness. Then the deep issu e of the interaction of concepts in general is discussed, which 
leads into some speculations on creativity. Th e Chapter concludes with a set of personal 
"Questions and Speculations" on Al and minds in general. 
 Sloth Canon . A canon which imitates a Bach canon in which one voice plays the same melody as

 Sloth Canon . A canon which imitates a Bach canon in which one voice plays the same melody as 
another, only upside down and twice as slowly, while  a third voice is free. Here, the Sloth utters 
the same lines as the Tortoise does, only negated (in a liberal sense of the term) and twice as 
slowly, while Achilles is free. 
 Chapter XX: Strange Loops, Or Tangled Hierarchies . A grand windup of many of the ideas 
about hierarchical systems and self-reference. It is concerned with the snarls which arise when 
systems turn back on themselv es-for example, science probing science, government investigating

systems turn back on themselv es-for example, science probing science, government investigating 
governmental wrongdoing, art violating the rules of art, and finally, humans thinking about their own brains and minds. Does Gödel’s Theorem have anything to say about this last "snarl"? Are 
free will and the sensation of consciousness connected  to Gödel’s Theorem? The Chapter ends by 
tying Gödel, Escher, and Bach together once again. 
 
Six-Part Ricercar.  This Dialogue is an exuberant game pl ayed with many of the ideas which have

Six-Part Ricercar.  This Dialogue is an exuberant game pl ayed with many of the ideas which have 
permeated the book. It is a reenactment of the story of the Musical Offering, which began the book; it is simultaneously a "translation" into wo rds of the most complex piece in the Musical 
Offering: the Six-Part Ricercar. This duality imbu es the Dialogue with more levels of meaning 
than any other in the book. Frederick the Great is replaced by the Crab, pianos by computers, and 
so on. Many surprises arise. The Dialogue's cont ent concerns problems of mind, consciousness,

so on. Many surprises arise. The Dialogue's cont ent concerns problems of mind, consciousness, 
free will, Artificial Intelligence, the Turing test, a nd so forth, which have been introduced earlier. 
It concludes with an implicit reference to th e beginning of the book, thus making the book into 
one big self-referential loop, symbolizing at once Bach's music, Escher's drawings, and Gödel’s 
Theorem.

Introduction: A Musico-Logical Offering    10
 
FIGURE 1. Johann Sebastian Bach, in 1748. From a painting by Elias Gottlieb 
Hanssmann.

Introduction: A Musico-Logical Offering    11Introduction: 
 
A Musico-Logical Offering 
Author: 
    FREDERICK THE GREAT, King of Pru ssia, came to power in 1740. Although he is 
remembered in history books mostly for his m ilitary astuteness, he was also devoted to 
the life of the mind and the spirit. His court in Potsdam was one of the great centers of 
intellectual activity in Europe in the eight eenth century. The celebrated mathematician 
Leonhard Euler spent twenty-five years there. Many other mathematicians and scientists came, as well as philosophers-including Voltaire and La Mettrie, who wrote some of their most influential works while there.     But music was Frederick's real love. He was an avid flutist and composer. Some of his 
compositions are occasionally performed even to  this day. Frederick was one of the first 
patrons of the arts to recognize the virtues of the newly developed "piano-forte" ("soft-

loud"). The piano had been developed in the first half of the eighteenth century as a modification of the harpsichord. The problem w ith the harpsichord was that pieces could 
only be played at a rather uniform loudness- there was no way to strike one note more 
loudly than its neighbors. The "soft-loud", as its name implies, provided a remedy to this 
problem. From Italy, where Bartolommeo Cristo fori had made the first one, the soft-loud

problem. From Italy, where Bartolommeo Cristo fori had made the first one, the soft-loud 
idea had spread widely. Gottfried Silbermann, the foremost German organ builder of the day, was endeavoring to make a "perfect" piano-forte. Undoubtedly King Frederick was 
the greatest supporter of his efforts-it is sa id that the King owned as many as fifteen 
Silbermann pianos!  
Bach 
 
Frederick was an admirer not onl y of pianos, but also of an organist and composer by the 
name of J. S. Bach. This Bach's compos itions were somewhat notorious. Some called 
them "turgid and confused", while others cl aimed they were incomparable masterpieces.

Bach 
 
them "turgid and confused", while others cl aimed they were incomparable masterpieces. 
But no one disputed Bach's ability to improvi se on the organ. In those days, being an 
organist not only meant being able to play, but also to extemporize, and Bach was known far and wide for his remarkable extemporiza tions. (For some delightful anecdotes about 
Bach's extemporization, see The Bach Reader, by H. T. David and A. Mendel.)     In 1747, Bach was sixty-two, and his fame , as well as one of his sons, had reached 
Potsdam: in fact, Carl Philipp Emanuel Bach was the Capellmeister (choirmaster) at the court of King Frederick. For years the King had let it be known, through gentle hints to 
Philipp Emanuel, how

Introduction: A Musico-Logical Offering    12pleased he would be to have the elder Bach come and pay him a visit; but this wish had 
never been realized. Frederick was particul arly eager for Bach to try out his new 
Silbermann pianos, which lie (Frederick) correctly foresaw as the great new wave in music.     It was Frederick's custom to have even ing concerts of chamber music in his court. 
Often he himself would be the soloist in a c oncerto for flute Here we have reproduced a 
painting of such an evening by the Germ an painter Adolph von Menzel, who, in the 
1800's, made a series of paintings illustrati ng the life of Frederick the Great. At the 
cembalo is C. P. E. Bach, and the figure fu rthest to the right is Joachim Quantz, the 
King's flute master-and the only person allo wed to find fault with the King's flute 
playing. One May evening in 1747, an unexp ected guest showed up. Johann Nikolaus 
Forkel, one of Bach's earliest biographers, tells the story as follows:

Forkel, one of Bach's earliest biographers, tells the story as follows: 
One evening, just as lie was getting his flut e ready, and his musicians were  ssembled, 
an officer brought him a list of the strangers who had arrived. With his flute in his hand 
he ran ever the list, but immediately turned to the assembled musicians, and said, with a 
kind of agitation, "Gentlemen, old Bach is come." The Hute was now laid aside, and old 
Bach, who had alighted at his son's lodgings , was immediately summoned to the Palace.

Bach, who had alighted at his son's lodgings , was immediately summoned to the Palace. 
Wilhelm Friedemann, who accompanied his father , told me this story, and I must say 
that 1 still think with pleasure on the manner in  which lie related it. At that time it was 
the fashion to make rather prolix compliments . The first appearance of J. S. Bach before 
se great a King, who did not even give him time to change his traveling dress for a 
black chanter's gown, must necessarily be attended with many apologies. I will net here dwell en these apologies, but merely observe , that in Wilhelm Friedemann's mouth they

black chanter's gown, must necessarily be attended with many apologies. I will net here dwell en these apologies, but merely observe , that in Wilhelm Friedemann's mouth they 
made a formal Dialogue between the King and the Apologist. But what is mere important than this is that the King gave up his Concert for this 
evening, and invited Bach, then already called the Old Bach, to try his fortepianos, 
made by Silbermann, which steed in several r ooms of the palace. [Forkel here inserts 
this footnote: "The pianofortes manufactur ed by Silbermann, of Frevberg, pleased the 
King se much, that he resolved to buy them a ll up. He collected fifteen. I hear that they

King se much, that he resolved to buy them a ll up. He collected fifteen. I hear that they 
all now stand unfit for use in various corners of the Royal Palace."] The musicians went 
with him from room to room, and Bach was i nvited everywhere to try them and to play 
unpremeditated compositions. After he had gene en for some time, he asked the King to 
give him a subject for a Fugue, in orde r to execute it immediately without any

give him a subject for a Fugue, in orde r to execute it immediately without any 
preparation. The King admired the learned manner in which his subject was thus executed extempore: and, probably to see hew far such art t could be carried, expressed a wish to hear a Fugue with six Obligato parts.  But as it is not every subject that is fit 
for such full harmony, Bach chose one hims elf, and immediately executed it to the 
astonishment of all present in the same magnificent and learned manner as he had done 
that of the King. His Majesty desired also to hear his performance en the organ. The

that of the King. His Majesty desired also to hear his performance en the organ. The 
next day therefore Bach was taken to all the organs in Potsdam, as lie had before been to Silbermann's fortepianos. After his return to Leipzig, he composed the subject, which 
he had received from the King, in three and six parts. added several artificial passages 
in strict canon to it, and had it engrave d, under the title of "Musikalisches Opfer" 
[Musical Offering], and dedicated it to the Inventor.'

Introduction: A Musico-Logical Offering    14
 
FIGURE 3. The Royal Theme.      When Bach sent a copy of his Musical  Offering to the King, he  included a dedicatory 
letter, which is of interest for its prose style if nothing else rather submissive and flattersome. From a modern perspectiv e it seems comical. Also, it probably gives 
something of the flavor of Bach's apology for his appearance.2  
MOST GRACIOUS KING!  In deepest humility I dedicate herewith to  Your Majesty a musical offering, the 
noblest part of which derives from Your  Majesty's own august hand. With awesome 
pleasure I still remember the very special Royal grace when, some time ago, during my visit in Potsdam, Your Majesty's Self deigned to play to me a theme for a fugue 
upon the clavier, and at the same time charge d me most graciously to carry it out in 
Your Majesty's most august presence. To obey Your Majesty's command was my most 
humble dim. I noticed very soon, however, that , for lack of necessary preparation, the

humble dim. I noticed very soon, however, that , for lack of necessary preparation, the 
execution of the task did not fare as we ll as such an excellent theme demanded. I 
resoled therefore and promptly pledged myse lf to work out this right Royal theme 
more fully, and then make it known to the world. This resolve has now been carried 
out as well as possible, and it has none other th an this irreproachable intent, to glorify, 
if only in a small point, the fame of a m onarch whose greatness and power, as in all

if only in a small point, the fame of a m onarch whose greatness and power, as in all 
the sciences of war and peace, so especially in music, everyone must admire and revere. I make bold to add this most hum ble request: may Your Majesty deign to 
dignify the present modest labor with a gracious acceptance, and continue to grant 
Your Majesty's most august Royal grace to 
 
Your Majesty's 
most humble and obedient servant, 
THE AUTHOR 
Leipzig, July 7  1747     Some twenty-seven years later, when Bach  had been dead for twentyfour years, a Baron 
named Gottfried van Swieten-to whom, incide ntally, Forkel dedicated his biography of

THE AUTHOR 
named Gottfried van Swieten-to whom, incide ntally, Forkel dedicated his biography of 
Bach, and Beethoven dedicated his Firs t Symphony-had a conversation with King 
Frederick, which he reported as follows:  
He [Frederick] spoke to me, among other thi ngs, of music, and of a great organist 
named Bach, who has been for a while in Berlin. This artist [Wilhelm Friedemann Bach] is endowed with a talent superior , in depth of harmonic knowledge and power 
of execution, to any 1 have heard or can imagine, while those who knew his father 
claim that he, in turn, was even greater. The King

Introduction: A Musico-Logical Offering    15is of this opinion, and to prove it to me he sang aloud a chromatic fugue subject which 
he had given this old Bach, who on the spot ha d made of it a fugue in four parts, then 
in five parts, and finally in eight parts.'  
    Of course there is no way of knowi ng whether it was King Fr ederick or Baron van 
Swieten who magnified the story into la rger-than-life proportions. But it shows how 
powerful Bach's legend had become by that time.  To give an idea of how extraordinary a 
six-part fugue is, in the entire Well-Temper ed Clavier by Bach, containing forty-eight 
Preludes and Fugues, only two have as many as five parts, and nowhere is there a six-part fugue! One could probably liken the task of im provising a six-part fugue  to the playing of 
sixty simultaneous blindfold games of chess,  and winning them all. To improvise an

sixty simultaneous blindfold games of chess,  and winning them all. To improvise an 
eight-part fugue is really beyond human capability.     In the copy which Bach sent to King Fr ederick, on the page preceding the first sheet of 
music, was the following inscription:  
 
FIG URE 4. 
 ("At the King's Command, the Song and the Remainder Resolved with Canonic Art.") 
Here Bach is punning on the word "canonic" , since it means not only "with canons" but 
also "in the best possible way". The initials of this inscription are  
R I C E R C A R 
 
-an Italian word, meaning "to seek". And certain ly there is a great deal to seek in the

FIG URE 4. 
R I C E R C A R 
 
-an Italian word, meaning "to seek". And certain ly there is a great deal to seek in the 
Musical Offering . It consists of one three-part fugue , one six-part fugue, ten canons, and a 
trio sonata. Musical scholars have conclude d that the three-part fugue must be, in 
essence, identical with the one which Bach  improvised for King Frederick. The six-part 
fugue is one of Bach's most complex creati ons, and its theme is, of course, the Royal 
Theme. That theme, shown in Figure 3, is a very complex one, rhythmically irregular and 
highly chromatic (that is, filled with tones wh ich do not belong to the key it is in). To

highly chromatic (that is, filled with tones wh ich do not belong to the key it is in). To 
write a decent fugue of even  two voices based on it would not be easy for the average 
musician!     Both of the fugues are inscribed "Ricer car", rather than "Fuga". This is another 
meaning of the word; "ricercar" was, in fact, the original name for the musical form now 
known as "fugue". By Bach's time, the word "fugue" (or fuga, in Latin and Italian) had 
become standard, but the term "ricercar" ha d survived, and now designated an erudite 
kind of fugue, perhaps too austerely intell ectual for the common ear. A similar usage

kind of fugue, perhaps too austerely intell ectual for the common ear. A similar usage 
survives in English today: the word "recher che" means, literally, "sought out", but carries 
the same kind of implication, namely of esoteric or highbrow cleverness.     The trio sonata forms a delightful relief from the austerity of the fugues and canons, because it is very melodious and sweet, almost dance-

Introduction: A Musico-Logical Offering    16able. Nevertheless, it too is based largely on the King's theme, chromatic and austere as it 
is. It is rather miraculous that Bach coul d use such a theme to make so pleasing an 
interlude.    The ten canons in the Musical Offering are among the most sophisticated canons Bach ever wrote. However, curiously enough, Bach hi mself never wrote them out in full. This 
was deliberate. They were posed as puzzles to King Frederick. It was a familiar musical 
game of the day to give a single theme, together with some more or less tricky hints, and to let the canon based on that theme be "discovered" by someone else. In order to know how this is possible, you must understand a few facts about canons.  
Canons and Fugues 
 
The idea of a canon is that one single theme is played against itself. This is done by

The idea of a canon is that one single theme is played against itself. This is done by 
having "copies" of the theme played by the various participating voices. But there are means' ways to do this. The most straightforward of all canons is the round, such as "Three Blind Mice", "Row, Row, Row Your Boat ", or " Frere Jacques". Here, the theme 
enters in the first voice and, after a fixed time- delay, a "copy" of it enters, in precisely the 
same key. After the same fixed time-delay in  the second voice, the third voice enters 
carrying the theme, and so on. Most themes will not harmonize with themselves in this

carrying the theme, and so on. Most themes will not harmonize with themselves in this 
way. In order for a theme to work as a canon theme, each of its notes must be able to 
serve in a dual (or triple, or quadruple) role : it must firstly be part of a melody, and 
secondly it must be part of a harmonization of the same melody. When there are three 
canonical voices, for instance, each note of the theme must act in two distinct harmonic ways, as well as melodically. Thus, each note in a canon has more than one musical meaning; the listener's ear and brain automa tically figure out the appropriate meaning, by 
referring to context. 
   There are more complicated sorts of canons, of course. The first escalation in 
complexity comes when the "copies" of the th eme are staggered not only in time, but also

complexity comes when the "copies" of the th eme are staggered not only in time, but also 
in pitch; thus, the first voice might sing th e theme starting on C, and the second voice, 
overlapping with the first voice, might sing the identical theme starting five notes higher, 
on G. A third voice, starting on the D yet five notes higher, might overlap with the first 
two, and so on. The next escalation in complex ity comes when the speeds of' the different 
voices are not equal; thus, the second voice mi ght sing twice as quickly, or twice as 
slowly, as the first voice. The former is ca lled diminution, the latter augmentation (since

slowly, as the first voice. The former is ca lled diminution, the latter augmentation (since 
the theme seems to shrink or to expand).    We are not yet done! The next stage of comp lexity in canon construction is to invert the 
theme, which means to make a melody whic h jumps down wherever the original theme 
jumps up, and by exactly the same number of se mitones. This is a rather weird melodic 
transformation, but when one has heard many th emes inverted, it begins to seem quite 
natural. Bach was especially fond of invers ions, and they show up often in his work-and 
the Musical Offering is no exception. (For a simple example of

Introduction: A Musico-Logical Offering    17inversion, try the tune "Good King Wenceslas". When the original and its inversion are 
sung together, starting an octave apart and st aggered with a time-delay of two beats, a 
pleasing canon results.) Finally, the most es oteric of "copies" is the retrograde copy-
where the theme is played backwards in time. A canon which uses this trick is 
affectionately known as a crab canon, because  of the peculiarities of crab locomotion. 
Bach included a crab canon in the Musical Offering, needless to say. Notice that every 
type of "copy" preserves all the information in  the original theme, in the sense that the 
theme is fully recoverable from any of the copies. Such an information preserving transformation is often called an isomorphi sm, and we will have much traffic with 
isomorphisms in this book.    Sometimes it is desirable to relax the ti ghtness of the canon form. One way is to allow

isomorphisms in this book.    Sometimes it is desirable to relax the ti ghtness of the canon form. One way is to allow 
slight departures from perfect copying, for the sake of more fluid harmony. Also, some 
canons have "free" voices-voices which do not employ the canon's theme, but which 
simply harmonize agreeably with the voices that are in canon with each other.    Each of the canons in the Musical Offering has for its theme a different variant of the King's Theme, and all the devices descri bed above for making canons intricate are 
exploited to the hilt; in fact, they are o ccasionally combined. Thus, one three-voice canon

exploited to the hilt; in fact, they are o ccasionally combined. Thus, one three-voice canon 
is labeled "Canon per Augmentationem, contrari o Motu"; its middle voice is free (in fact, 
it sings the Royal Theme), while the other two dance canonically above and below it, 
using the devices of augmentation and invers ion. Another bears simply the cryptic label 
"Quaerendo invenietis" ("By seeking, you will discover"). All of the canon puzzles have been solved. The canonical solutions were given by one of Bach's pupils, Johann Philipp

"Quaerendo invenietis" ("By seeking, you will discover"). All of the canon puzzles have been solved. The canonical solutions were given by one of Bach's pupils, Johann Philipp 
Kirnberger. But one might still wonder whether there are more solutions to seek!     I should also explain briefly what a f ugue is. A fugue is like a canon, in that it is 
usually based on one theme which gets played in different voices and different keys, and 
occasionally at different speeds or upside dow n or backwards. However, the notion of 
fugue is much less rigid than that of canon, and consequently it allows for more 
emotional and artistic expression. The telltale si gn of a fugue is the way it begins: with a

emotional and artistic expression. The telltale si gn of a fugue is the way it begins: with a 
single voice singing its theme. When it is done, then a second voice enters, either five 
scale-notes up, or four down. Meanwh ile the first voice goes on, singing the 
"countersubject": a secondary theme, chosen to provide rhythmic, harmonic, and melodic 
contrasts to the subject. Each of the voices en ters in turn, singing the theme, often to the 
accompaniment of the countersubject in some  other voice, with the remaining voices

accompaniment of the countersubject in some  other voice, with the remaining voices 
doing whatever fanciful things entered the composer's mind. When all the voices have "arrived", then there are no rules. There are, to be sure, standard kinds of things to do-but not so standard that one can merely compos e a fugue by formula. The two fugues in the 
Musical Offering are outstanding examples of fugues that could never have been 
"composed by formula". There is something much deeper in them than mere fugality.     All in all, the Musical Offering repres ents one of Bach's supreme accomplishments in 
counterpoint. It is itself one large intellectual fugue, in

Introduction: A Musico-Logical Offering    18which many ideas and forms have been w oven together, and in which playful double 
meanings and subtle allusions are commonplace. And it is a very beautiful creation of the 
human intellect which we can appreciate forever. (The entire work is wonderfully described in the book f. S. Bach's Musical Offering, by H. T. David.)  
An Endlessly Rising Canon 
 
There is one canon in the Musical Offering whic h is particularly unusual. Labeled simply 
"Canon per Tonos", it has three voices. The uppe rmost voice sings a variant of the Royal 
Theme, while underneath it, two voices pr ovide a canonic harmonization based on a 
second theme. The lower of this pair sings its theme in C minor (which is the key of the canon as a whole), and the upper of the pair sings the same theme displaced upwards in 
pitch by an interval of a fifth. What makes this canon different from any other, however, 
is that when it concludes-or, rather, seems to conclude-it is no longer in the key of C

is that when it concludes-or, rather, seems to conclude-it is no longer in the key of C 
minor, but now is in D minor. Somehow Bach  has contrived to modulate (change keys) 
right under the listener's nose. And it is so constructed that this "ending" ties smoothly 
onto the beginning again; thus one can repeat th e process and return in the key of E, only 
to join again to the beginning. These successive  modulations lead the ear to increasingly 
remote provinces of tonality, so that after several of them, one would expect to be 
hopelessly far away from the starting key. A nd yet magically, after exactly six such

hopelessly far away from the starting key. A nd yet magically, after exactly six such 
modulations, the original key of C minor has been restored! All the voices are exactly one 
octave higher than they were at the beginni ng, and here the piece may be broken off in a 
musically agreeable way. Such, one imagines, was Bach's intention; but Bach indubitably 
also relished the implication that this proce ss could go on ad infinitum, which is perhaps 
why he wrote in the margin "As the modula tion rises, so may the King's Glory." To 
emphasize its potentially infinite aspect, I like to call this the "Endlessly Rising Canon".

emphasize its potentially infinite aspect, I like to call this the "Endlessly Rising Canon". 
    In this canon, Bach has given us our fi rst example of the notion of Strange Loops. The 
"Strange Loop" phenomenon occurs whenever, by moving upwards (or downwards) through the levels of some hierarchical sy stem, we unexpectedly find ourselves right 
back where we started. (Here, the system is that of musical keys.) Sometimes I use the term Tangled Hierarchy to describe a system  in which a Strange Loop occurs. As we go 
on, the theme of Strange Loops will recur ag ain and again. Sometimes it will be hidden,

on, the theme of Strange Loops will recur ag ain and again. Sometimes it will be hidden, 
other times it will be out in the open; sometim es it will be right side up, other times it will 
be upside down, or backwards. "Quaerendo invenietis" is my advice to the reader.   
Escher 
 
To my mind, the most beautiful and powerful visu al realizations of this notion of Strange 
Loops exist in the work of the Dutch graphi c artist M. C. Escher, who lived from 1902 to 
1972. Escher was the creator of some of the

Introduction: A Musico-Logical Offering    19
 
FIGURE 5. Waterfall, by M. C. Escher (lithograph, 1961). 
 most intellectually stimulating drawings of a ll time. Many of them have their origin in 
paradox, illusion, or double-meaning. Mathematicians were among the first admirers of Escher's drawings, and this is understandable because they often are based on mathematical principles of symmetry or patte rn ... But there is much more to a typical 
Escher drawing than just symmetry or pattern ; there is often an underlying idea, realized 
in artistic form. And in particular, the Strange Loop is one of the most recurrent themes in 
Escher's work. Look, for example, at the lit hograph Waterfall (Fig. 5), and compare its 
six-step endlessly falling loop with the six- step endlessly rising loop of the "Canon per 
Tonos". The similarity of vision is

Introduction: A Musico-Logical Offering    20
 
FIGURE 6. Ascending and Descending, by M. C. Escher (lithograph, 1960).

Introduction: A Musico-Logical Offering    21remarkable. Bach and Escher are playing one single theme in two different "keys": music 
and art.     Escher realized Strange Loops in several different ways, and they can be arranged according to the tightness of the loop. The lithograph Ascending and Descending (Fig. 6), 
in which monks trudge forever in loops, is the loosest version, since it involves so many 
steps before the starting point is regained. A tighter loop is contained in Waterfall, which, 
as we already observed, involves only six discre te steps. You may be thinking that there 
is some ambiguity in the notion of a singl e "step"-for instance, couldn't Ascending and 
Descending be seen just as easily as having f our levels (staircases) as forty-five levels 
(stairs)% It is indeed true that there is an inherent      
 
FIGURE 7. Hand with Reflecting Globe. Self- portrait In, M. C. Escher (lithograph, 
1935).

Introduction: A Musico-Logical Offering    23 
haziness in level-counting, not only in Escher  pictures, but in hierarchical, many-level 
systems. We will sharpen our understanding of this haziness later on. But let us not get 
too distracted now' As we tighten our loop, we come to the remarkable Drawing Hands 
(Fig. 135), in which each of two hands draw s the other: a two-step Strange Loop. And 
finally, the tightest of all Strange Loops is r ealized in Print Gallery (Fig. 142): a picture 
of a picture which contains itsel f. Or is it a picture of a gallery which contains itself? Or 
of a town which contains itself? Or a young ma n who contains himself'? (Incidentally, the 
illusion underlying Ascending and Descending and Waterfall was not invented by Escher, 
but by Roger Penrose, a British mathematic ian, in 1958. However, the theme of the 
Strange Loop was already present in Escher's  work in 1948, the year he drew Drawing

Strange Loop was already present in Escher's  work in 1948, the year he drew Drawing 
Hands. Print Gallery dates from 1956.)     Implicit in the concept of Strange Loops is  the concept of infinity, since what else is a 
loop but a way of representing an  endless process in a finite way? And infinity plays a 
large role n many of Escher's drawings. Copi es of one single theme often fit into each' 
other, forming visual analogues to the canons of Bach. Several such patterns can be seen 
in Escher's famous print Metamorphosis (Fi g. 8). It is a little like the "Endlessly Rising

in Escher's famous print Metamorphosis (Fi g. 8). It is a little like the "Endlessly Rising 
Canon": wandering further and further from its starting point, it suddenly is back. In the 
tiled planes of Metamorphosis and other pi ctures, there are already suggestions of 
infinity. But wilder visions of infinity appear  in other drawings by Escher. In some of his 
drawings, one single theme can appear on different levels of reality. For instance, one level in a drawing might clearly be recognizab le as representing fantasy or imagination;

drawings, one single theme can appear on different levels of reality. For instance, one level in a drawing might clearly be recognizab le as representing fantasy or imagination; 
another level would be recognizable as r eality. These two levels might be the only 
explicitly portrayed levels. But the mere presen ce of these two levels invites the viewer to 
look upon himself as part of yet another leve l; and by taking that step, the viewer cannot 
help getting caught up in Escher's implied chai n of levels, in which, for any one level, 
there is always another level above it of great er "reality", and likewise, there is always a

there is always another level above it of great er "reality", and likewise, there is always a 
level below, "more imaginary" than it is. This can be mind-boggling in itself. However, what happens if the chain of levels is not lin ear, but forms a loop? What is real, then, and 
what is fantasy? The genius of Escher was that he could not only concoct, but actually 
portray, dozens of half-real, half-mythical worlds, worlds filled with Strange Loops, 
which he seems to be inviting his viewers to enter.  
Gödel

Gödel 
 
In the examples we have seen of Strange Loops by Bach and Escher, there is a conflict between the finite and the infinite, and he nce a strong sense of paradox. Intuition senses 
that there is something mathematical invol ved here. And indeed in our own century a 
mathematical counterpart was discovered, w ith the most enormous repercussions. And, 
just as the Bach and Escher loops appeal to  very simple and ancient intuitions-a musical 
scale, a staircase-so this discovery, by K. Gödel, of a Strange Loop in

Introduction: A Musico-Logical Offering    24
 
FIGURE 9. Kurt Godel.

Introduction: A Musico-Logical Offering    25mathematical systems has its origins in simp le and ancient intuitions. In its absolutely 
barest form, Godel's discovery involves the translation of an ancient paradox in philosophy into mathematical terms. That paradox is the so-called Epimenides paradox, 
or liar paradox. Epimenides was a Cretan who made one immortal statement: "All Cretans are liars." A sharper version of the st atement is simply "I am lying"; or, "This 
statement is false". It is that last version which I will usually mean when I speak of the 
Epimenides paradox. It is a statement which rudely violates the usually assumed dichotomy of statements into true and false,  because if you tentatively think it is true, 
then it immediately backfires on you and ma kes you think it is false. But once you've 
decided it is false, a similar backfiring returns you to the idea that it must be true. Try it!     The Epimenides paradox is a one-step Strange Loop, like Escher's Print Gallery. But

decided it is false, a similar backfiring returns you to the idea that it must be true. Try it!     The Epimenides paradox is a one-step Strange Loop, like Escher's Print Gallery. But 
how does it have to do with mathematics? That  is what Godel discovered. His idea was to 
use mathematical reasoning in exploring math ematical reasoning itself. This notion of 
making mathematics "introspective" proved to be enormously powerful, and perhaps its richest implication was the one Godel found: Godel's Incompleteness Theorem. What the 
Theorem states and how it is proved are two different things. We shall discuss both in 
quite some detail in this book. The Theorem can De likened to a pearl, and the method of

quite some detail in this book. The Theorem can De likened to a pearl, and the method of 
proof to an oyster. The pearl is prized for its  luster and simplicity; the oyster is a complex 
living beast whose innards give rise to this mysteriously simple gem.     Godel's Theorem appears as Propos ition VI in his 1931 paper "On Formally 
Undecidable Propositions in Principia Mathematica and Related Systems I." It states:  
To every w-consistent recursive class K of formulae there correspond recursive class-signs r, such that neither v Gen r nor  Neg (v Gen r) belongs to Fig (K) (where v 
is the free variable of r).

is the free variable of r). 
 
Actually, it was in German, and perhaps you feel that it might as well be in German 
anyway. So here is a paraphrase in more normal English:  
All consistent axiomatic formulations of number theory include undecidable propositions. 
 This is the pearl.     In this pearl it is hard to see a Strange  Loop. That is because the Strange Loop is buried 
in the oyster-the proof. The proof of G odel's Incompleteness Theorem hinges upon the

in the oyster-the proof. The proof of G odel's Incompleteness Theorem hinges upon the 
writing of a self-referential mathematical statement, in the same way as the Epimenides paradox is a self-referential statement of langua ge. But whereas it is very simple to talk 
about language in language, it is not at all eas y to see how a statement about numbers can 
talk about itself. In fact, it took genius me rely to connect the idea of self-referential 
statements with number theory. Once Godel ha d the intuition that such a statement could 
be created, he was over the major hurdle. Th e actual creation of the statement was the

be created, he was over the major hurdle. Th e actual creation of the statement was the 
working out of this one beautiful spark of intuition.

Introduction: A Musico-Logical Offering    26  
We shall examine the Godel construction quite car efully in Chapters to come, but so that 
you are not left completely in the dark, I will sk etch here, in a few strokes, the core of the 
idea, hoping that what you see will trigger ideas  in your mind. First of all, the difficulty 
should be made absolutely clear. Mathemati cal statements-let us concentrate on number-
theoretical ones-are about properties of  whole numbers. Whole numbers are not 
statements, nor are their properties. A stat ement of number theory is not about a. 
statement of number theory; it just is a statement of number theory. This is the problem; but Godel realized that there was more here than meets the eye.    Godel had the insight that a statement of  number theory could be about a statement of 
number theory (possibly even itself), if  only numbers could somehow stand for 
statements. The idea of a code, in other words,  is at the heart of his construction. In the

statements. The idea of a code, in other words,  is at the heart of his construction. In the 
Godel Code, usually called "Godel-numbering", numbers are made to stand for symbols 
and sequences of symbols. That way, each statement of number theory, being a sequence of specialized symbols, acquires a Godel num ber, something like a telephone number or a 
license plate, by which it can be referred to. And this coding trick enables statements of

license plate, by which it can be referred to. And this coding trick enables statements of 
number theory to be understood on two different levels: as statements of number theory, and also as statements about statements of number theory.    Once Godel had invented this coding scheme, he had to work out in detail a way of transporting the Epimenides paradox into a numbertheoretical formalism. His final 
transplant of Epimenides did not say, "This statement of number theory is false", but 
rather, "This statement of number theory doe s not have any proof". A great deal of 
confusion can be caused by this, because people generally understand the notion of

confusion can be caused by this, because people generally understand the notion of 
"proof" rather vaguely. In fact, Godel's work was just part of a long attempt by 
mathematicians to explicate for themselves wh at proofs are. The important thing to keep 
in mind is that proofs are demonstrations within fixed systems of propositions. In the case 
of Godel's work, the fixed system of numbe rtheoretical reasoning to which the word 
"proof" refers is that of Principia Mathematica (P.M.), a giant opus by Bertrand Russell and Alfred North Whitehead, published between 1910 and 1913. Therefore, the Godel sentence G should more properly be written in English as:  
This statement of number theory does not ha ve any proof in the system of Principia 
Mathematica.

Mathematica. 
 Incidentally, this Godel sentence G is not G odel's Theorem-no more than the Epimenides 
sentence is the observation that "The Epim enides sentence is a paradox." We can now 
state what the effect of discovering G is. Whereas the Epimenides statement creates a 
paradox since it is neither true nor false,  the Godel sentence G is unprovable (inside 
P.M.) but true. The grand conclusion% That the system of Principia Mathematica is

P.M.) but true. The grand conclusion% That the system of Principia Mathematica is 
"incomplete"-there are true statements of number theory which its methods of proof are 
too weak to demonstrate.

Introduction: A Musico-Logical Offering    27   But if Principia Mathematica was the first victim of this stroke, it was certainly not the 
last! The phrase "and Related Systems" in the title  of Godel's article is a telling one: for if 
Godel's result had merely pointed out a defect  in the work of Russell and Whitehead, then 
others could have been inspired to improve upon P.M. and to outwit Godel's Theorem. But this was not possible: Godel's proof pertained to any axiomatic system which 
purported to achieve the aims which Whitehead and Russell had set for themselves. And for each different system, one basic method di d the trick. In short, Godel showed that 
provability is a weaker notion than truth, no matter what axiomatic system is involved.    Therefore Godel's Theorem had an electri fying effect upon logicians, mathematicians, 
and philosophers interested in the foundations of mathematics, for it showed that no fixed

and philosophers interested in the foundations of mathematics, for it showed that no fixed 
system, no matter how complicated, could represent the complexity of the whole numbers: 0, 1, 2, 3, ... Modern readers may not be as nonplussed by this as readers of 
1931 were, since in the interim our culture ha s absorbed Godel's Theorem, along with the 
conceptual revolutions of relativity and qua ntum mechanics, and their philosophically 
disorienting messages have reached the public, even if cushioned by several layers of 
translation (and usually obfuscation). There is  a general mood of expectation, these days,

translation (and usually obfuscation). There is  a general mood of expectation, these days, 
of "limitative" results-but back in 1931, this came as a bolt from the blue.  
Mathematical Logic: A Synopsis 
 
A proper appreciation of Godel's Theorem requi res a setting of context. Therefore, I will 
now attempt to summarize in a short space the history of mathematical logic prior to 
1931-an impossible task. (See DeLong, Kneebone, or Nagel and Newman, for good presentations of history.) It all began w ith the attempts to mechanize the thought 
processes of reasoning. Now our ability to r eason has often been claimed to be what 
distinguishes us from other species; so it seems somewhat paradoxical, on first thought,

distinguishes us from other species; so it seems somewhat paradoxical, on first thought, 
to mechanize that which is most human. Yet even the ancient Greeks knew that reasoning 
is a patterned process, and is at least partially governed by statable laws. Aristotle codified syllogisms, and Euclid codified ge ometry; but thereafter, many centuries had to 
pass before progress in the study of axiomatic reasoning would take place again.    One of the significant discoveries of nine teenth-century mathematics was that there are 
different, and equally valid, geometries-where  by "a geometry" is meant a theory of

different, and equally valid, geometries-where  by "a geometry" is meant a theory of 
properties of abstract points and lines. It ha d long been assumed that geometry was what 
Euclid had codified, and that, although th ere might be small flaws in Euclid's 
presentation, they were unimportant and a ny real progress in geometry would be 
achieved by extending Euclid. This idea wa s shattered by the roughly simultaneous 
discovery of non-Euclidean geometry by seve ral people-a discovery that shocked the 
mathematics community, because it deeply cha llenged the idea that mathematics studies 
the real world. How could there be many differ

Introduction: A Musico-Logical Offering    28ent kinds of "points" and "lines" in one si ngle reality? Today, the solution to the dilemma 
may be apparent, even to some nonmathematicians-but at the time, the dilemma created havoc in mathematical circles.    Later in the nineteenth century, the E nglish logicians George Boole and Augustus De 
Morgan went considerably further than Aris totle in codifying strictly deductive reasoning 
patterns. Boole even called his book "The La ws of Thought"-surely an exaggeration, but 
it was an important contribution. Lewis Carroll was fascinated by these mechanized reasoning methods, and invented many puzzles which could be solved with them. Gottlob 
Frege in Jena and Giuseppe Peano in Turi n worked on combining formal reasoning with

Frege in Jena and Giuseppe Peano in Turi n worked on combining formal reasoning with 
the study of sets and numbers. David Hilbert in Gottingen worked on stricter formalizations of geometry than Euclid's. All of these efforts were directed towards clarifying what one means by "proof".     In the meantime, interesting developmen ts were taking place in classical mathematics. 
A theory of different types of infinities, known as the theory of sets, was developed by 
Georg Cantor in the 1880's. The theory was powerful and beautiful, but intuition-defying. 
Before long, a variety of set-theoretical pa radoxes had been unearthed. The situation was

Before long, a variety of set-theoretical pa radoxes had been unearthed. The situation was 
very disturbing, because just as mathematics seemed to be recovering from one set of paradoxes-those related to the theory of limits, in the calculusalong came a whole new 
set, which looked worse!     The most famous is Russell's paradox. Mo st sets, it would seem, are not members of 
themselves-for example, the set of walruses is not a walrus, the set containing only Joan 
of Arc is not Joan of Arc (a set is not a person)-and so on. In this respect, most sets are rather "run-of-the-mill". However, some "self-swallowing" sets do contain themselves as members, such as the set of all sets, or the set of all things except Joan of Arc, and so on. 
Clearly, every set is either run-of-the-m ill or self-swallowing, and no set can be both.

Clearly, every set is either run-of-the-m ill or self-swallowing, and no set can be both. 
Now nothing prevents us from inventing R: the set of all run-o,-the-mill sets. At first, R 
might seem a rather run-of-the-mill invention-but that opinion must be revised when you 
ask yourself, "Is R itself "a run-of-the-mill set or a self-swallowing set?" You will find 
that the answer is: "R is neither run-of -the-mill nor self-swallowing, for either choice 
leads to paradox." Try it!     But if R is neither run-of-the-mill nor self-swallowing, then what is it? At the very

leads to paradox." Try it!     But if R is neither run-of-the-mill nor self-swallowing, then what is it? At the very 
least, pathological. But no one was satisfied w ith evasive answers of that sort. And so 
people began to dig more deeply into the f oundations of set theory. The crucial questions 
seemed to be: "What is wrong with our intuitive concept of 'set'? Can we make a rigorous theory of sets which corresponds closely with our intuitions, but which skirts the

seemed to be: "What is wrong with our intuitive concept of 'set'? Can we make a rigorous theory of sets which corresponds closely with our intuitions, but which skirts the 
paradoxes?" Here, as in number theory and geometry, the problem is in trying to line up intuition with formalized, or axiomatized, reasoning systems.     A startling variant of Russell's paradox, called "Grelling's paradox", can be made using 
adjectives instead of sets. Divide the adjec tives in English into two categories: those 
which are self-descriptive, such as "penta syllabic", "awkwardnessful", and "recherche", 
and those which are not, such

Introduction: A Musico-Logical Offering    29  
as "edible", "incomplete", and "bisyllabic". No w if we admit "non-selfdescriptive" as an 
adjective, to which class does it belong? If  it seems questionable to include hyphenated 
words, we can use two terms invented specia lly for this paradox: autological (= "self-
descriptive"), and heterological (= "non-self -descriptive"). The question then becomes: 
"Is 'heterological' heterological?" Try it!    There seems to he one common culprit in  these paradoxes, namely self-reference, or 
"Strange Loopiness". So if the goal is to ban all paradoxes, why not try banning self-
reference and anything that allows it to aris e? This is not so easy as it might seem, 
because it can be hard to figure out just where self-reference is occurring. It may be 
spread out over a whole Strange Loop with severa l steps, as in this "expanded" version of 
Epimenides, reminiscent of Drawing Hands:  
The following sentence is false. 
The preceding sentence is true.

The preceding sentence is true. 
  Taken together, these sentences have the same  effect as the original Epimenides paradox: 
yet separately, they are harmless and even potentially useful sentences. The "blame" for 
this Strange Loop can't he pinned on either se ntence-only on the way they "point" at each 
other. In the same way, each local regi on of Ascending and Descending is quite 
legitimate; it is only the way they are globally  put together that creates an impossibility. 
Since there are indirect as well as direct wa ys of achieving self-reference, one must figure

Since there are indirect as well as direct wa ys of achieving self-reference, one must figure 
out how to ban both types at once-if one sees selfreference as the root of all evil. Banishing Strange Loops  Russell and Whitehead did subscribe to  this view, and accordingly, Principia 
Mathematica was a mammoth exercise in e xorcising Strange Loops from logic, set 
theory, and number theory. The idea of their system was basically this. A set of the 
lowest "type" could contain only "objects" as membersnot sets. A set of the next type up 
could only contain objects, or se ts of the lowest type. In general, a set of a given type

could only contain objects, or se ts of the lowest type. In general, a set of a given type 
could only contain sets of lower type, or objects. Every set woul d belong to a specific 
type. Clearly, no set could contain itself because  it would have to belong to a type higher 
than its own type. Only "run-of'-the-mill" sets  exist in such a system; furthermore, old R-
the set of all run-of-the-mill sets-no longer is considered a set at all, because it does not 
belong to any finite type. To all appearances, then, this theory of types, which we might 
also call the "theory of the abolition of Strange  Loops", successfully rids set theory of its 
paradoxes, but only at the cost of introduci ng an artificial-seeming hierarchy, and of

paradoxes, but only at the cost of introduci ng an artificial-seeming hierarchy, and of 
disallowing the formation of certain kinds of sets-such as the set of all run-of-the-mill 
sets. Intuitively, this is not the way we imagine sets.     The theory of types handled Russell's paradox, but it did nothing about the Epimenides 
paradox or Grelling's paradox. For people whose

Introduction: A Musico-Logical Offering    30interest went no further than set theory, this  was quite adequate-but for people interested 
in the elimination of paradoxes generally, some similar "hierarchization" seemed necessary, to forbid looping back inside la nguage. At the bottom of such a hierarchy 
would be an object language. Here, referen ce could be made only to a specific domain-
not to aspects of the object language itself (s uch as its grammatical rules, or specific 
sentences in it). For that purpose there would be a metalanguage. This experience of two 
linguistic levels is familiar to all learners of foreign languages. Then there would be a 
metametalanguage for discussing the metalangua ge, and so on. It would be required that 
every sentence should belong to some precise level of the hierarchy. Therefore, if one

every sentence should belong to some precise level of the hierarchy. Therefore, if one 
could find no level in which a given utterance fit, then the utterance would be deemed meaningless, and forgotten.    An analysis can be attempted on the tw o-step Epimenides loop given above. The first 
sentence, since it speaks of the second, must be on a higher level than the second. But by 
the same token, the second sentence must be on a higher level than the first. Since this is 
impossible, the two sentences are "meaningle ss". More precisely, such sentences simply 
cannot be formulated at all in a system ba sed on a strict hierarchy of languages. This

cannot be formulated at all in a system ba sed on a strict hierarchy of languages. This 
prevents all versions of the Epimenides paradox as well as Grelling's paradox. (To what language level could "heterological" belong?)     Now in set theory, which deals with ab stractions that we don't use all the time, a 
stratification like the theory of types seems acceptable, even if a little strange-but when it 
comes to language, an all-pervading part of life, such stratification appears absurd. We 
don't think of ourselves as jumping up and dow n a hierarchy of languages when we speak 
about various things. A rather matter-of-fact  sentence such as, "In this book, I criticize

about various things. A rather matter-of-fact  sentence such as, "In this book, I criticize 
the theory of types" would be doubly forbidde n in the system we are discussing. Firstly, it 
mentions "this book", which should only be mentionable in a  metabook"-and secondly, it mentions me-a pe rson whom I should not be allowed to 
speak of at all! This example points out how  silly the theory of types seems, when you

speak of at all! This example points out how  silly the theory of types seems, when you 
import it into a familiar context. The remedy it adopts for paradoxes-total banishment of self-reference in any form-is a real case of overkill, branding many perfectly good constructions as meaningless. The adjective "meaningless", by the way, would have to apply to all discussions of the theory of li nguistic types (such as that of this very 
paragraph) for they clearly could not occur on any of the levels-neither object language,

paragraph) for they clearly could not occur on any of the levels-neither object language, 
nor metalanguage, nor metametalanguage, etc. So  the very act of discussing the theory 
would be the most blatant possible violation of it!     Now one could defend such theories by saying that they were only intended to deal 
with formal languages-not with ordinary, info rmal language. This may be so, but then it 
shows that such theories are extremely acad emic and have little to say about paradoxes 
except when they crop up in special tailor-made systems. Besides, the drive to eliminate paradoxes at any cost, especially when it requires the creation of highly artificial

except when they crop up in special tailor-made systems. Besides, the drive to eliminate paradoxes at any cost, especially when it requires the creation of highly artificial 
formalisms, puts too much stress on bland consistency, and too little on the

Introduction: A Musico-Logical Offering    31quirky and bizarre, which make life and mathema tics interesting. It is of course important 
to try to maintain consistency, but when th is effort forces you into a stupendously ugly 
theory, you know something is wrong.    These types of issues in the foundations of mathematics were responsible for the high interest in codifying human reasoning methods wh ich was present in the early part of this 
century. Mathematicians and philosophers had begun to have serious doubts about whether even the most concrete of theories , such as the study of whole numbers (number 
theory), were built on solid foundations. If  paradoxes could pop up so easily in set 
theory-a theory whose basic concept, that of  a set, is surely very intuitively appealing-
then might they not also exist in other bran ches of mathematics? Another related worry 
was that the paradoxes of logic, such as th e Epimenides paradox, might turn out to be

was that the paradoxes of logic, such as th e Epimenides paradox, might turn out to be 
internal to mathematics, and thereby cast in doubt all of mathematics. This was especially 
worrisome to those-and there were a good num ber-who firmly believed that mathematics 
is simply a branch of logic (or conversely, that  logic is simply a branch of mathematics). 
In fact, this very question-"Are mathematic s and logic distinct, or separate%"-was the 
source of much controversy.     This study of mathematics itself becam e known as metamathematics-or occasionally, 
metalogic, since mathematics and logic are so  intertwined. The most urgent priority of

metalogic, since mathematics and logic are so  intertwined. The most urgent priority of 
metamathematicians was to determine the true nature of mathematical reasoning. What is a legal method of procedure, and what is an illegal one? Since mathematical reasoning had always been done in "natural language" (e.g., French or Latin or some language for 
normal communication), there was always a lot of possible ambiguity. Words had different meanings to different people, conj ured up different images, and so forth. It

normal communication), there was always a lot of possible ambiguity. Words had different meanings to different people, conj ured up different images, and so forth. It 
seemed reasonable and even important to establish a single uniform notation in which all mathematical work could be done, and with the aid of which any two mathematicians 
could resolve disputes over whether a sugge sted proof was valid or not. This would 
require a complete codification of the uni versally acceptable modes of human reasoning, 
at least as far as they applied to mathematics.  
Consistency, Completeness, Hilbert's Program 
 
This was the goal of Principia Mathematica, which purported to derive all of mathematics

This was the goal of Principia Mathematica, which purported to derive all of mathematics 
from logic, and, to be sure, without contra dictions! It was widely admired, but no one 
was sure if (1) all of mathematics really  was contained in the methods delineated by 
Russell and Whitehead, or (2) the methods given were even self-consistent. Was it 
absolutely clear that contradictory results could never be derived, by any mathematicians 
whatsoever, following the methods of Russell and Whitehead?    This question particularly bothered th e distinguished German mathematician (and 
metamathematician) David Hilbert, w ho set before the world community of 
mathematicians (and metamathematicians) this chal

Introduction: A Musico-Logical Offering    32lenge: to demonstrate rigorously-perhaps following the very methods outlined by Russell 
and Whitehead-that the system defined in Principia Mathematica was both consistent 
(contradiction-free), and complete (i.e., that every true statement of, number theory could 
be derived within the framework drawn up in P.M.). This was a tall order, and one could 
criticize it on the grounds that it was some what circular: how can you justify your 
methods of reasoning on the basis of those sa me methods of reasoning? It is like lifting 
yourself up by your own bootstraps. (We just don' t seem to be able to get away from 
these Strange Loops!)     Hilbert was fully aware of this dilemma , of course, and therefore expressed the hope 
that a demonstration of consistency or co mpleteness could be found which depended only 
on "finitistic" modes of reasoning. "these were  a small set of reasoning methods usually

on "finitistic" modes of reasoning. "these were  a small set of reasoning methods usually 
accepted by mathematicians. In this way, H ilbert hoped that mathematicians could 
partially lift themselves by their own bootstra ps: the sum total of mathematical methods 
might be proved sound, by invoking only a sma ller set of methods. This goal may sound 
rather esoteric, but it occupied the minds of  many of the greatest mathematicians in the 
world during the first thirty years of this century.     In the thirty-first year, however, G odel published his paper, which in some ways

world during the first thirty years of this century.     In the thirty-first year, however, G odel published his paper, which in some ways 
utterly demolished Hilbert's program. This paper revealed not only that there were 
irreparable "holes" in the axiomatic system  proposed by Russell and Whitehead, but more 
generally, that no axiomatic system whatso ever could produce all number-theoretical 
truths, unless it were an inconsistent sy stem! And finally, the hope of proving the 
consistency of a system such as that presente d in P.M. was shown to  be vain: if such a 
proof could be found using only methods inside P.M., then-and this is one of the most

proof could be found using only methods inside P.M., then-and this is one of the most 
mystifying consequences of Godel's work-P.M. itself would be inconsistent!     The final irony of it all is that the pr oof of Gi del's Incompleteness Theorem involved 
importing the Epimenides paradox right into th e heart ofPrincipia Mathematica, a bastion 
supposedly invulnerable to the attacks of Strange Loops! Although Godel's Strange Loop 
did not destroy Principia Mathematica, it made it far less interesting to mathematicians, for it showed that Russell and Whitehead's original aims were illusory.  
Babbage, Computers, Artificial Intelligence ...

Babbage, Computers, Artificial Intelligence ... 
 
When Godel's paper came out, the world was on the brink of developing electronic digital 
computers. Now the idea of mechanical calculating engines had been around for a while. In the seventeenth century, Pascal and Le ibniz designed machines to perform fixed 
operations (addition and multiplication). Thes e machines had no memory, however, and 
were not, in modern parlance, programmable.     The first human to conceive of the immense computing potential of machinery was the Londoner Charles Babbage (1792-1871). A character  who could almost have stepped out 
of the pages of the Pickwick Papers,

Introduction: A Musico-Logical Offering    33 Babbage was most famous during his life time for his vigorous campaign to rid London 
of "street nuisances"-organ grinders above all.  These pests, loving to get his goat, would 
come and serenade him at any time of day or  night, and he would furiously chase them 
down the street. Today, we recognize in Ba bbage a man a hundred years ahead of his 
time: not only inventor of the basic principles  of modern computers, he was also one of 
the first to battle noise pollution.     His first machine, the "Difference Engi ne", could generate mathematical tables of 
many kinds by the "method of differences". But before any model of the "D.E." had been 
built, Babbage became obsessed with a much more revolutionary idea: his "Analytical 
Engine". Rather immodestly, he wrote, "The  course through which I arrived at it was the 
most entangled and perplexed which probabl y ever occupied the human mind."' Unlike

most entangled and perplexed which probabl y ever occupied the human mind."' Unlike 
any previously designed machine, the A.E. was to possess both a "store" (memory) and a 
"mill" (calculating and decision-making unit). Th ese units were to be built of thousands 
of intricate geared cylinders interlocked in incredibly complex ways. Babbage had a 
vision of numbers swirling in and out of the mill tinder control of a program contained in 
punched cards-an idea inspired by the jacquard  loom, a card-controlled loom that wove 
amazingly complex patterns. Babbage's brillia nt but ill-fated Countess friend, Lady Ada

amazingly complex patterns. Babbage's brillia nt but ill-fated Countess friend, Lady Ada 
Lovelace (daughter of Lord Byron), poetically  commented that "the Analytical Engine 
weaves algebraic patterns just as the Jacquard-loom weaves flowers and leaves." Unfortunately, her use of the present tense was misleading, for no A.E. was ever built, 
and Babbage died a bitterly disappointed man.     Lady Lovelace, no less than Babbage, was profoundly aware that with the invention of 
the Analytical Engine, mankind was flirting w ith mechanized intelligence-particularly if 
the Engine were capable of "eating its own tail" (the way Babbage described the Strange

the Engine were capable of "eating its own tail" (the way Babbage described the Strange 
Loop created when a machine reaches in and alters its own stored program). In an 1842 memoir,5 she wrote that the A.E. "might act upon other things besides number". While 
Babbage dreamt of creating_ a chess or tic-tac-toe automaton, she suggested that his 
Engine, with pitches and harmonies coded into its spinning cylinders, "might compose elaborate and scientific pieces of music of any degree of complexity or extent." In nearly 
the same breath, however, she cautions that "T he Analytical Engine has no pretensions

the same breath, however, she cautions that "T he Analytical Engine has no pretensions 
whatever to originate anything. It can do what ever we know how to order it to perform." 
Though she well understood the power of artificial computation, Lady Lovelace was skeptical about the artificial creation of in telligence. However, could her keen insight 
allow her to dream of the potential that  would be opened up with the taming of 
electricity?    In our century the time was ripe for co mputers-computers beyond the wildest dreams of 
Pascal, Leibniz, Babbage, or Lady Lovelace.  In the 1930's and 1940's, the first "giant 
electronic brains" were designed and built. They catalyzed the convergence of three

electronic brains" were designed and built. They catalyzed the convergence of three 
previously disparate areas: the theory of axiomatic reasoning, the study of mechanical 
computation, and the psychology of intelligence. These same years saw the theory of computers develop by leaps and

Introduction: A Musico-Logical Offering    34bounds. This theory was tightly linked to meta mathematics. In fact, Godel's Theorem has 
a counterpart in the theory of computation, discovered by Alan Turing, which reveals the existence of inelucPable "holes" in even  the most powerful computer imaginable. 
Ironically, just as these somewhat eerie lim its were being mapped out, real computers 
were being built whose powers seemed to gr ow and grow beyond their makers' power of 
prophecy. Babbage, who once declared he would gl adly give up the rest  of his life if he 
could come back in five hundred years and have  a three-day guided scientific tour of the 
new age, would probably have been thrille d speechless a mere century after his death-
both by the new machines, and by their unexpected limitations.     By the early 1950's, mechanized intellig ence seemed a mere stone's throw away; and

both by the new machines, and by their unexpected limitations.     By the early 1950's, mechanized intellig ence seemed a mere stone's throw away; and 
yet, for each barrier crossed, there always cropped up some new barrier to the actual creation of a genuine thinking machine. Wa s there some deep reason for this goal's 
mysterious recession?     No one knows where the borderline be tween non-intelligent behavior and intelligent 
behavior lies; in fact, to suggest that a sharp borderline exists is probably silly. But 
essential abilities for intelligence are certainly:

essential abilities for intelligence are certainly:  
to respond to situations very flexibly; to take advantage of fortuitous circumstances; to make sense out of ambiguous or contradictory messages; to recognize the relative importance of different elements of a situation; to find similarities between situations despite differences which may separate them; to draw distinctions between situations despite similarities may link them; to synthesize new concepts by taking old th em together in new ways; to come up 
with ideas which are novel.

with ideas which are novel. 
 
    Here one runs up against a seeming paradox. Computers by their very nature are the most inflexible, desireless, rule-following of  beasts. Fast though they may be, they are 
nonetheless the epitome of unconsciousness. How, then, can intelligent behavior be programmed? Isn't this the most blatant of contradictions in terms? One of the major 
theses of this book is that it is not a contradic tion at all. One of the major purposes of this

theses of this book is that it is not a contradic tion at all. One of the major purposes of this 
book is to urge each reader to confront the a pparent contradiction head on, to savor it, to 
turn it over, to take it apart, to wallow in it, so that in the end the reader might emerge with new insights into the seemingly unbr eathable gulf between the formal and the 
informal, the animate and the inanimate, the flexible and the inflexible.    This is what Artificial Intelligence (A1) re search is all about. And the strange flavor of 
AI work is that people try to put together long sets of rules in strict formalisms which tell

AI work is that people try to put together long sets of rules in strict formalisms which tell 
inflexible machines how to be flexible.     What sorts of "rules" could possibly capture all of what we think of as intelligent behavior, however? Certainly there must be rules on all sorts of

Introduction: A Musico-Logical Offering    35 different levels. There must be many "just pl ain" rules. There must be "metarules" to 
modify the "just plain" rules; then "metamet arules" to modify the metarules, and so on. 
The flexibility of intelligence comes from the enormous number of different rules, and 
levels of rules. The reason that so many rule s on so many different levels must exist is 
that in life, a creature is faced with millions of  situations of completely different types. In 
some situations, there are stereotyped respons es which require "just plain" rules. Some 
situations are mixtures of stereotyped situations-thus they require rules for deciding which of the 'just plain" rules to apply. Some  situations cannot be classified-thus there 
must exist rules for inventing new rules ... and on and on. Without doubt, Strange Loops 
involving rules that change themselves, dir ectly or indirectly, are at the core of

involving rules that change themselves, dir ectly or indirectly, are at the core of 
intelligence. Sometimes the complexity of our minds seems so overwhelming that one 
feels that there can be no solution to the problem of understanding intelligence-that it is 
wrong to think that rules of any sort govern a creature's behavior, even if one takes "rule" 
in the multilevel sense described above.  
...and Bach 
 
In the year 1754, four years after the death of  J. S. Bach, the Leipzig theologian Johann 
Michael Schmidt wrote, in a treatise on mu sic and the soul, the following noteworthy 
passage:

passage: 
Not many years ago it was reported from Fran ce that a man had made a statue that 
could play various pieces on the Fleuttraversi ere, placed the flute to its lips and took it 
down again, rolled its eyes, etc. But no one ha s yet invented an image that thinks, or 
wills, or composes, or even does anything at all similar. Let anyone who wishes to be convinced look carefully at the last fugal work of the above-praised Bach, which has 
appeared in copper engraving, but which was left unfinished because his blindness

appeared in copper engraving, but which was left unfinished because his blindness 
intervened, and let him observe the art that is contained therein; or what must strike 
him as even more wonderful, the Chorale whic h he dictated in his blindness to the pen 
of another: Wenn wir in hochsten Nothen s een. I am sure that he will soon need his 
soul if he wishes to observe all the beautie s contained therein, let alone wishes to play 
it to himself or to form a judgment of the author. Everything that the champions of Materialism put forward must fall to the ground in view of this single example.6

it to himself or to form a judgment of the author. Everything that the champions of Materialism put forward must fall to the ground in view of this single example.6 
Quite likely, the foremost of the "champions of Materialism" here alluded to was none 
other than Julien Offroy de la Mettrie-philo sopher at the court of Frederick the Great, 
author of L'homme machine ("Man, the Machine"), and Materialist Par Excellence. It is now more than 200 years later, and the battle is still raging between those who agree with 
Johann Michael Schmidt, and those who agree with Julien Offroy de la Mettrie. I hope in 
this book to give some perspective on the battle.  
"Godel, Escher, Bach"

"Godel, Escher, Bach" 
 
The book is structured in an unusual way:  as a counterpoint between Dialogues and 
Chapters. The purpose of this structure is to allow me to

Introduction: A Musico-Logical Offering    36present new concepts twice: almost every new concept is first presented metaphorically 
in a Dialogue, yielding a set of  concrete, visual images; then these serve, during the 
reading of the following`Chapter, as an intuitive background for a more serious and 
abstract presentation of the same concept. In  many of the Dialogues I appear to be talking 
about one idea on the surface, but in reality I am  talking about some other idea, in a thinly 
disguised way.    Originally, the only characters in my Dialogues were Achilles and the Tortoise, who 
came to me from Zeno of Elea, by way of Lewis Carroll. Zeno of Elea, inventor of 
paradoxes, lived in the fifth century B.C. One of his paradoxes was an allegory, with 
Achilles and the Tortoise as protagonists. Zeno' s invention of the happy pair is told in my 
first Dialogue, Three-Part Invention. In 1895, Lewis Carroll reincarnated Achilles and the

first Dialogue, Three-Part Invention. In 1895, Lewis Carroll reincarnated Achilles and the 
Tortoise for the purpose of illustrating his own new paradox of infinity. Carroll's paradox, 
which deserves to be far better known than it is, plays a significant role in this book. Originally titled "What the Tortoise Said to Achilles", it is reprinted here as Two-Part 
Invention.    When I began writing Dialogues, somehow I connected them up with musical forms. I

Invention.    When I began writing Dialogues, somehow I connected them up with musical forms. I 
don't remember the moment it happened; I just remember one day writing "Fugue" above an early Dialogue, and from then on the idea stuck. Eventually I decided to pattern each Dialogue in one way or another on a di fferent piece by Bach. This was not so 
inappropriate. Old Bach himself used to remind his pupils that the separate parts in their compositions should behave like "persons who conversed together as if in a select 
company". I have taken that suggestion perhap s rather more literally than Bach intended

company". I have taken that suggestion perhap s rather more literally than Bach intended 
it; nevertheless I hope the result is faithful to  the meaning. I have been particularly 
inspired by aspects of Bach's compositions which have struck me over and over, and which are so well described by David and Mendel in The Bach Reader: His form in general was based on relations between separate sections. These relations 
ranged from complete identity of passages on the one hand to the 
return of a single principle of  elaboration or a mere thematic allusion on the other. The 
resulting patterns were often symmetrical, but by no means necessarily so. Sometimes the relations betw een the various sections make up a maze of

resulting patterns were often symmetrical, but by no means necessarily so. Sometimes the relations betw een the various sections make up a maze of 
interwoven threads that only detailed analysis can unravel. Usually, however, a few dominant features afford proper orientation at first sight or hearing, and 
while in the course of study one may discover unending sub tleties, one is never at a loss to grasp the un ity that holds together  every single creation by 
Bach.'     I have sought to weave an Eternal Gold en Braid out of these three strands: Godel, 
Escher, Bach. I began, intending to write an e ssay at the core of which would be Godel's 
Theorem. I imagined it would be a mere pa mphlet. But my ideas expanded like a sphere, 
and soon touched Bach and Escher. It took so me time for me to think of making this

and soon touched Bach and Escher. It took so me time for me to think of making this 
connection explicit, instead of just letting it be a private motivating force. But finally 1 
realized that to me, Godel and Escher a nd Bach were only shadows cast in different 
directions by some central solid essence. I tried to reconstruct the central object, and 
came up with this book.

Three-Part Invention 
 37Three-Part Invention  
 
Achilles (a Greek warrior, the fleetest of f oot of all mortals) and a Tortoise are 
standing together on a dusty runway in the hot sun. Far down the runway, on a tall flagpole, there hangs a large rectangul ar flag. The flag  is sold red, except 
where a thin ring-shaped holes has been cut out of it, through which one can see the sky. 
 
ACHILLES: What is that strange flag down at the other end of the track? It reminds me 
somehow of a print by my favourite artists M.C. Escher. 
TORTOISE: That is Zeno’ s flag  ACHILLES: Could it be that the hole in it resemb les the holes in a Mobian strip Escher once 
drew? Something is wrong about the flag, I can tell. 
TORTOISE: The ring which has been cut from it has the shape of the numeral for zero, which 
is Zeno´s favourite number. 
ACHILLES: The ring which hasn´t been invented  yet! It will only be invented by a Hindu 
mathematician some millennia hence. And thus , Mr. T, mt argument proves that such a

mathematician some millennia hence. And thus , Mr. T, mt argument proves that such a 
flag is impossible. 
TORTOISE: Your argument is persuasive, Achilles, a nd I must agree that such  a  flag is indeed 
impossible. But it is beautiful anyway, is it not? 
ACHILLES: Oh, yes, there is no doubt of its beauty. TORTOISE: I wonder if it´s beauty is related to it´s impossibility. I don´ t know, I´ve never had 
the time to analyze Beauty. It´s a Capitalized Essence, and I never seem to have time for 
Capitalized Essences. 
ACHILLES: Speaking of Capitalized Essences, Mr. T, have you ever wondered about the 
Purpose of Life?

Purpose of Life? 
TORTOISE: Oh, heavens, no; ACHILLES: Haven’t you ever wondered why we are here, or who invented us? TORTOISE: Oh, that is quite another matter. We are inventions of Zeno (as you will shortly 
see) and the reason we are here is to have a footrace. 
ACHILLES::: A footrace?  How outrageous! Me, the fl eetest of foot of all mortals, versus you, 
the ploddingest of the plodders! There can be no point  to such a race.

the ploddingest of the plodders! There can be no point  to such a race. 
TORTOISE: You might give me a head start. ACHILLES: It would have to be a huge one. TORTOISE: I don’t object. ACHILLES: But I will catch you, sooner or later – most likely sooner. TORTOISE: Not if things go according to Ze no´s paradox, you won’t. Zeno is hoping to use 
our footrace to show that motion is impossible, you see. It is only in the mind that motion seems possible, according to Zeno. In truth, Mo tion Is Inherently Impossible.  He proves 
it quite elegantly.

 38
 
Figure 10. Mobius strip by M.C.Escher (wood-e ngraving printed from four blocks, 1961) 
 
ACHILLES: Oh, yes, it comes  back to me now: the famous Zen koan about Zen 
Master Zeno. As you say it is very simple indeed. 
TORTOISE: Zen Koan? Zen Master? What do you mean? ACHILLES: It goes like this:  Two monks we re arguing about a flag. One said, “The 
flag is moving.” The other said, “The wind is moving.” The sixth patriarch, Zeno, 
happened to be passing by. He told them , “Not the wind, not the flag, mind is 
moving.” 
TORTOISE: I am afraid you are a little befuddl ed, Achilles. Zeno is no Zen master, far 
from it. He is in fact, a Greek philosopher from the town of Elea (which lies halfway 
between points A and B). Centuries hence, he will be celebrated for his paradoxes of 
motion. In one of those paradoxes, this ve ry footrace between you and me will play a 
central role. 
ACHILLES: I’m all confused. I remember vividly how I used to repeat over and over

moving.” 
ACHILLES: I’m all confused. I remember vividly how I used to repeat over and over 
the names of the six patriarchs of Zen, and I always said, “The sixth patriarch is Zeno, 
The sixth patriarch is Zeno…” ( Suddenly a soft warm breeze picks up. ) Oh, look Mr. 
Tortoise – look at the flag waving! How I love to watch the ripples shimmer through 
it’s soft fabric. And the ring cut out of it is waving, too!

 39TORTOISE: Don´t be silly. The flag is impo ssible, hence it can’t be waving. The wind is 
waving. 
 
(At this moment, Zeno happens by.) 
 
Zeno: Hallo! Hulloo! What’s up? What’s new? ACHILLES: The flag is moving. TORTOISE: The wind is moving. Zeno: O friends, Friends! Cease your argumenta tion! Arrest your vitriolics! Abandon your 
discord! For I shall resolve the issue for you forthwith. Ho! And on such a fine day. 
ACHILLES: This fellow must be playing the fool. TORTOISE: No, wait, Achilles. Let us hear what  he has to say. Oh Unknown Sir, do impart to 
us your thoughts on this matter. 
Zeno: Most willingly. Not thw ind, not the flag – neither one is moving, nor is anything moving 
at all. For I have discovered a great Theorem, which states; “Motion Is Inherently Impossible.”  And from this Theorem fo llows an even greater Theorem – Zeno’s 
Theorem: “Motion Unexists.”

Theorem: “Motion Unexists.” 
ACHILLES: “Zeno’s Theorem”? Are you, sir, by any chance, the philosopher Zeno of Elea? Zeno: I am indeed, Achilles. ACHILLES: (scratching his head in puzzlement ). Now how did he know my name? 
Zeno: Could I possibly persuade you two to hear me out as to why this is the case? I’ve come 
all the way to Elea from point A this afte rnoon, just trying to find someone who’ll pay 
some attention to my closely honed argument. But they’re all hurrying hither and thither,

some attention to my closely honed argument. But they’re all hurrying hither and thither, 
and they don’t have time. You’ve no idea how disheartening it is to meet with refusal after refusal. Oh, I’m sorry to burden you with my troubles, I’d just like to ask you one thing: Would the two of you humour a sill old philosopher for a few moments – only a 
few, I promise you – in his eccentric theories. 
ACHILLES: Oh, by all means! Please do illuminate us! I know I speak for both of us, since my 
companion, Mr. Tortoise, was only moments a go speaking of you with great veneration –

companion, Mr. Tortoise, was only moments a go speaking of you with great veneration – 
and he mentioned especially your paradoxes. 
Zeno: Thank you. You see, my Master, the fift h patriarch, taught me that reality is one, 
immutable, and unchanging, all plurality, cha nge, and motion are mere illusions of the 
sense. Some have mocked his views; but I will show the absurdity of their mockery. My 
argument is quite simple. I will illustrate it with two characters of my own Invention: 
Achilles )a Greek warrior, the fleetest of foot of all mortals), and a Tortoise. In my tale,

Achilles )a Greek warrior, the fleetest of foot of all mortals), and a Tortoise. In my tale, 
they are persuaded by a passerby to run a f ootrace down a runway towards a distant flag 
waving in the breeze. Let us assume that, sin ce the Tortoise is a much slowerrunner, he 
gets a head start of, say, ten rods. Now the race begins. In a few bounds Achilles has 
reached the spot where the Tortoise  started.

 40ACHILLES: Hah! 
Zeno: And now the Tortoise is but a single rod ahead of Achilles. Within only a moment, 
Achilles has attained that spot. 
ACHILLES: Ho ho! Zeno: Yet, in that short moment, the Tortoise  has managed to advance a slight amount. In a 
flash, Achilles covers that distance too. 
ACHILLES: Hee hee hee! Zeno: But in that very short flash, the Tortoise  has managed to inch ahead by ever so little, and 
so Achilles is still behind. Now you see that in order for Achilles to catch the Tortoise, this game of “try-to-catch-me” will have to  be played an INFINITE number of times – 
and therefore Achilles can NEVER catch up with the Tortoise. 
TORTOISE: Heh heh heh heh! ACHILLES: Hmm… Hmm… Hmm… Hmm… Hmm…That argument sounds wrong to me. 
And yes, I can’t quite make out what’s wrong with it 
Zeno: Isn’t it a teaser? It’s my favourite paradox. TORTOISE: Excuse me, Zeno, but I believe your tale illustrates the wrong principle, doe sit

Zeno: Isn’t it a teaser? It’s my favourite paradox. TORTOISE: Excuse me, Zeno, but I believe your tale illustrates the wrong principle, doe sit 
not? You have just told us what will come to known, centuries hence, as Zeno’s “Achilles 
paradox” , which shows (ahem!) that Achilles will never catch the Tortoise; but the proof that Motion Is Inherently Impossible (a nd thence that Motion Unexists) is your 
“dichotomy paradox”, isn’t that so? 
Zeno: Oh, shame on me. Of course, you’re right. That’s the new one about how, in going from 
A to B, one has to go halfway first – and of th at stretch one also has to go halfway, and so 
on and so forth. But you see, both those paradoxe s really have the same flavour. Frankly, 
I’ve only had one Great Idea – I just exploit it in different ways.

I’ve only had one Great Idea – I just exploit it in different ways. 
ACHILLES: I swear, these arguments contain a fl aw. I don’t quite see where, but they cannot 
be correct. 
Zeno: You doubt the validity of my paradox? Why not just try it out|? You see that red flag 
waving down here, at the far end of the runway? 
ACHILLES: The impossible one, based on an Escher print? Zeno: Exactly. What do you say to you and Mr. To rtoise racing for it, allowing Mr. T a fair 
head start of, well, I don’t know – 
TORTOISE: How about ten rods? Zeno: Very good – ten rods. ACHILLES: Any time. Zeno: Excellent! How exciting! An empirical test of my rigorously proven Theorem! Mr.

TORTOISE: How about ten rods? Zeno: Very good – ten rods. ACHILLES: Any time. Zeno: Excellent! How exciting! An empirical test of my rigorously proven Theorem! Mr. 
Tortoise, will you position yourself ten rods upwind? 
 
(The Tortoise moves ten rods closer to the flag) 
 
Tortoise and Achlles: Ready! Zeno: On your mark! Get set! Go!

The MU-puzzle 41Chapter 1 
 
The MU-puzzle 
 
Formal Systems 
 ONE OF THE most central notions  in this book is that of a formal system . The type of 
formal system I use was invented by the Amer ican logician Emil Post in the 1920's, and 
is often called a "Post production system". This Chapter introduces you to a formal 
system and moreover, it is my hope that you w ill want to explore this formal system at 
least a little; so to provoke your curiosity, I have posed a little puzzle. 
"Can you produce MU?" is the puzzle. To begin with, you will be supplied with a 
string  (which means a string of letters).* Not to  keep you in suspense, that string will be 
MI. Then you will be told some rules, with which you can change one string into another. 
If one of those rules is applicable at so me point, and you want to use it, you may, but-
there is nothing that will dictate which rule  you should use, in case there are several

there is nothing that will dictate which rule  you should use, in case there are several 
applicable rules. That is left up to you-and of  course, that is where playing the game of 
any formal system can become something of  an art. The major point, which almost 
doesn't need stating, is that you must not do anything which is outside the rules. We 
might call this restriction the "Requirement of Formality". In the present Chapter, it 
probably won't need to be stressed at a ll. Strange though it may sound, though, I predict 
that when you play around with some of the formal systems of Chapters to come, you

that when you play around with some of the formal systems of Chapters to come, you 
will find yourself violating the Requirement of Formality over and over again, unless you 
have worked with formal systems before. 
The first thing to say about our formal system-the MIU -system -is that it utilizes 
only three letters of the alphabet: M, I, U . That means that the only strings of the MIU -
system are strings which are composed of t hose three letters. Below are some strings of 
the MIU -system: 
 
MU 
UIM MUUMUU UIIUMIUUIMUIIUMIUUIMUIIU 
 
* In this book, we shall employ the following c onventions when we refer to strings. When the

MU 
 
* In this book, we shall employ the following c onventions when we refer to strings. When the 
string is in the same typeface as the text, then  it will be enclosed in single or double quotes. 
Punctuation which belongs to the sentence and not to the string under discussion will go outside  
of the quotes, as logic dictates. For example, the firs t letter of this sentence is 'F', while the first 
letter of 'this ‘sentence’.is 't'. When the string is in Quadrata Roman , however, quotes will 
usually be left off, unless clarity demands them. For example, the first letter of Quadrata  is Q.

The MU-puzzle 42But although all of these are legitimate strings, they are not strings which are "in your 
possession". In fact, the only stri ng in your possession so far is MI. Only by using the 
rules, about to be introduced, can you enlarg e your private collection. Here is the first 
rule:  RULE I: If you possess a string whose last letter is I, you can add on a U at the end. 
 By the way, if up to this point you had not guesse d it, a fact about the meaning of "string" 
is that the letters are in a fixed order. For example, MI and IM are two different strings. 
A string of symbols is not just a "bag" of symbols, in which the order doesn't make any difference. 
Here is the second rule: 
 RULE II: Suppose you have Mx. Then you may add Mxx  to your collection. 
 What I mean by this is shown below, in a few examples.  From MIU , you may get MIUIU .  
From MUM , you may get MUMUM .  
From MU, you may get MUU . 
 So the letter `x' in the rule simply stands for any string; but once you have decided which

 So the letter `x' in the rule simply stands for any string; but once you have decided which 
string it stands for, you have to stick with  your choice (until you use the rule again, at 
which point you may make a new choice). Noti ce the third example above. It shows how, 
once you possess MU, you can add another string to your collection; but you have to get 
MU first! I want to add one last comment about the letter `x': it is not part of the formal 
system in the same way as the three letters ` M', `I', and ` U' are. It is useful for us, 
though, to have some way to talk in general about strings of the system, symbolically-and

though, to have some way to talk in general about strings of the system, symbolically-and 
that is the function of the `x ': to stand for an arbitrary string. If you ever add a string 
containing an 'x' to your "collection", you ha ve done something wrong, because strings of 
the MIU -system never contain "x" “s”! 
Here is the third rule:  RULE III: If III occurs in one of the strings in your collection, you may make a new 
string with U in place of III. 
 Examples:  
From UMIIIMU , you could make UMUMU . 
From MII11 , you could make MIU  (also MUI ). 
From IIMII , you can't get anywhere using this rule. 
(The three I's have to be consecutive.)

 Examples:  
(The three I's have to be consecutive.)  
From MIII , make MU. 
 Don't, under any circumstances, think you can run this rule backwards, as in the 
following example:

The MU-puzzle 43From MU, make MIII             <- This is wrong. 
 Rules are one-way.  Here is the final rule.  RULE IV: If UU occurs inside one of your strings, you can drop it. 
 From UUU , get U. 
From MUUUIII , get MUIII . 
 There you have it. Now you ma y begin trying to make MU. Don't worry you don't get it. 
Just try it out a bit-the main thing is  for you to get the flavor of this MU-puzzle. Have 
fun.  
Theorems, Axioms, Rules 
 
The answer to the MU-puzzle appears later in the book. For now, what important is not finding the answer, but looking for it. You pr obably hay made some attempts to produce 
MU. In so doing, you have built up your own priv ate collection of strings. Such strings, 
producible by the rules, are called theorems . The term "theorem" has, of course, a 
common usage mathematics which is quite di fferent from this one. It means some 
statement in ordinary language which has been  proven to be true by a rigorous argument,

statement in ordinary language which has been  proven to be true by a rigorous argument, 
such as Zeno's Theorem about the "unexistence" of motion, c Euclid's Theorem about the infinitude of primes. But in formal system th eorems need not be thought of as statements-
they are merely strings c symbols. And instead of being proven , theorems are merely 
produced , as if F machine, according to certain typographical rules. To emphasize this 
important distinction in meanings for the word "theorem", I will adopt the following 
convention in this book: when "theorem" is capitalized, its meaning will be the everyday

convention in this book: when "theorem" is capitalized, its meaning will be the everyday 
one-a Theorem is a statement in ordinary language which somebody once proved to be true by some sort of logic argument. Wh en uncapitalized, "theorem" will have its 
technical meaning a string producible in some  formal system. In these terms, the MU-
puzzle asks whether MU is a theorem of the MIU -system. 
 I gave you a theorem for free at the beginning, namely MI. Such "free" theorem is called 
an axiom -the technical meaning again being qui  different from the usual meaning. A 
formal system may have zero, or several, or even infinitely many axioms. Examples of all 
these types v appear in the book.

these types v appear in the book. 
 
Every formal system has symbol-shunting ru les, such as the four rules of the MIU -
system. These rules are called either rules of production  or rules of inference . I will use 
both terms. 
 
The last term which I wish to introduce at this point is derivation. Shown below is a 
derivation of the theorem MUIIU : 
 
(1) MI    a x i o m  
(2) MII    from (1) by rule II

The MU-puzzle 44(3) MIII    from (2) by rule II 
(4) MIIIIU   from (3) by rule I 
(5) MUIU   from (4) by rule III 
(6) MUIUUIU    from (5) by rule II 
(7) MUIIU   from (6) by rule IV 
 A derivation of a theorem is an explicit, line-by-line demonstration of how to produce 
that theorem according to the rules of the fo rmal system. The concept of derivation is 
modeled on that of proof, but a derivation is an austere cousin of a proof. It would sound 
strange to say that you had proven  MUIIU , but it does not sound so strange to say you 
have derived  MUIIU . 
 
Inside and Outside the System 
 Most people go about the MU-puzzle by derivi ng a number of theorems, quite at random, 
just to see what kind of thing turns up. Pre tty soon, they begin to notice some properties 
of the theorems they have made; that is wh ere human intelligence enters the picture. For 
instance, it was probably not obvious to you that all theorems would begin with M, until

instance, it was probably not obvious to you that all theorems would begin with M, until 
you had tried a few. Then, the pattern emer ged, and not only could you see the pattern, 
but you could understand it by looking at the rule s, which have the property that they 
make each new theorem inherit its first letter from an earlier theorem; ultimately, then, all 
theorems' first letters can be traced back  to the first letter of the sole axiom MI-and that is 
a proof that theorems of the MIU -system must all begin with M.

a proof that theorems of the MIU -system must all begin with M. 
There is something very significant about  what has happened here. It shows one 
difference between people and machines. It w ould certainly be possible-in fact it would 
be very easy-to program a computer to generate theorem after theorem of the MIU -
system; and we could include in the pr ogram a command to stop only upon generating U. 
You now know that a computer so progra mmed would never stop. And this does not 
amaze you. But what if you asked a friend to try to generate U? It would not surprise you

amaze you. But what if you asked a friend to try to generate U? It would not surprise you 
if he came back after a while, complaini ng that he can't get rid of the initial M, and 
therefore it is a wild goose chase. Even if a person is not very bright, he still cannot help 
making some observations about what he is  doing, and these observations give him good 
insight into the task-insight which the computer program, as we have described it, lacks. 
Now let me be very explicit about what I meant by saying this shows a difference 
between people and machines. I meant that it is possible  to program a machine to do a

between people and machines. I meant that it is possible  to program a machine to do a 
routine task in such a way that the machin e will never notice even the most obvious facts 
about what it is doing; but it is inherent in human consciousness to notice some facts about the things one is doing. But you knew this all along. If you punch "1" into an adding machine, and then add 1 to it, and then add 1 again, and again, and again, and 
continue doing so for hours and hours, the m achine will never learn to anticipate you, and 
do it itself, although any person would pick up the

The MU-puzzle 45pick up the idea, no matter how much or how we ll it is driven, that it i supposed to avoid 
other cars and obstacles on the road; and it w ill never learn even the most frequently 
traveled routes of its owner. 
The difference, then, is that it is possible  for a machine to act unobservant; it is 
impossible for a human to act unobservant. Noti ce I am not saying that all machines are 
necessarily incapable of making sophisticated observations; just that some machines are. 
Nor am I saying that all people are always making sophisticated observations; people, in 
fact, are often very unobservant. But machines  can be made to be totally unobservant; 
any people cannot. And in fact, most machines  made so far are pretty close ti being 
totally unobservant. Probably for this reason, th e property of being; unobservant seems to 
be the characteristic feature of machines, to  most people. For example, if somebody says

be the characteristic feature of machines, to  most people. For example, if somebody says 
that some task is "mechanical", i does not mean that people are incapable of doing the 
task; it implies though, that only a machin e could do it over and over without eve 
complaining, or feeling bored.  
Jumping out of the System 
 It is an inherent property of intelligence th at it can jump out of the tas which it is 
performing, and survey what it has done; it is always looking for and often finding, 
patterns. Now I said that an intelligence can jump out o its task, but that does not mean

patterns. Now I said that an intelligence can jump out o its task, but that does not mean 
that it always will. However, a little promp ting will often suffice. For example, a human 
being who is reading a boo may grow sleepy. Instead of continuing to read until the book 
is finished he is just as likely to put the book aside and turn off th e light. He ha stepped 
"out of the system" and yet it seems the most natural thing in the world to us. Or, suppose 
person A is watching television when person B comes in the room, and shows evident

person A is watching television when person B comes in the room, and shows evident 
displeasure with the situation Person A may think he understands the problem, and try to 
remedy it b exiting the present system (that te levision program), and flipping the channel 
knob, looking for a better show. Person B may have a more radio concept of what it is to 
"exit the system"-namely to turn the televisi on oft Of course, there are cases where only a 
rare individual will have the vision to pe rceive a system which governs many peoples 
lives, a system which ha never before even b een recognized as a system; then such people

lives, a system which ha never before even b een recognized as a system; then such people 
often devote their lives to convincing other peopl e that the system really is there and that 
it ought to be exited from! 
How well have computers been taught to jump out of the system? I w cite one 
example which surprised some observers. In a computer chess: tournament not long ago 
in Canada, one program-the weakest of all the competing ones-had the unusual feature of quitting long before the game was over. It was not a very good chess player, but it at least 
had the redeeming quality of being able to  spot a hopeless position, and to resign then

had the redeeming quality of being able to  spot a hopeless position, and to resign then 
and there, instead of waiting for the other program to go through the

The MU-puzzle 46boring ritual of checkmating. Although it lost ever y game it played, it did it in style. A lot 
of local chess experts were impressed. Thus , if you define "the system" as "making 
moves in a chess game", it is clear that th is program had a sophisticated, preprogrammed 
ability to exit from the system. On the other hand, if you think of "the system" as being 
"whatever the computer had been programme d to do", then there is no doubt that the 
computer had no ability whatsoever to exit from that system. 
It is very important when studying fo rmal systems to distinguish working within  
the system from making statements or observations about  the system. I assume that you 
began the MU-puzzle, as do most people, by working within the system; and that you 
then gradually started getting anxious, and this anxiety finally built up to the point where without any need for further consideration, you exited from the system, trying to take

then gradually started getting anxious, and this anxiety finally built up to the point where without any need for further consideration, you exited from the system, trying to take 
stock of what you had produced, and wonderi ng why it was that you had not succeeded in 
producing MU. Perhaps you found a reason why you could not produce MU; that is 
thinking about the system. Perhaps you produced MIU  somewhere along the way; that is 
working within the system. Now I do not want  to make it sound as if the two modes are 
entirely incompatible; I am sure that every human being is capable to some extent of 
working inside a system and simultaneously th inking about what he is doing. Actually, in

working inside a system and simultaneously th inking about what he is doing. Actually, in 
human affairs, it is often next to impossible to break things neatly up into "inside the 
system" and "outside the system"; life is  composed of so many interlocking and 
interwoven and often inconsistent "systems" that it may seem simplistic to think of things in those terms. But it is often important to fo rmulate simple ideas very clearly so that one 
can use them as models in thinking about more complex ideas. And that is why I am showing you formal systems; and it is about  time we went back to discussing the MIU -
system. 
  
M-Mode, I-Mode, U-Mode

system. 
  
M-Mode, I-Mode, U-Mode 
 
The MU-puzzle was stated in such a way that  it encouraged some amount of exploration 
within the MIU -system-deriving theorems. But it was also stated in a way so as not to 
imply that staying inside the system w ould necessarily yield fruit. Therefore it 
encouraged some oscillation between the two modes of work. One way to separate these 
two modes would be to have two sheets of paper; on one sheet, you work "in your 
capacity as a machine", thus filling it with nothing but M's, I's, and U's; on the second

capacity as a machine", thus filling it with nothing but M's, I's, and U's; on the second 
sheet, you work "in your capacity as a thinki ng being", and are allowed to do whatever 
your intelligence suggests-which might involve  using English, sketching ideas, working 
backwards, using shorthand (such as the le tter `x'), compressing several steps into one, 
modifying the rules of the system to see wh at that gives, or whatever else you might 
dream up. One thing you might do is notice that  the numbers 3 and 2 play an important 
role, since I's are gotten rid of in three's, and U's in two's-and doubling of length (except

role, since I's are gotten rid of in three's, and U's in two's-and doubling of length (except 
for the M) is allowed by rule II. So the second sheet might

The MU-puzzle 47 
also have some figuring on it. We will occasionally refer back to these two modes of dealing with a formal system, and we will call them the Mechanic mode (M-mode ) and 
the Intelligent mode (I-mode ). To round out our mode with  one for each letter of the 
MIU -system, I will also mention a fin mode-the Un-mode (U-mode ), which is the Zen 
way of approaching thing. More about this in a few Chapters.   
Decision Procedures 
 An observation about this puzzle is that it involves rules of two opposite tendencies-the lengthening rules  and the shortening rules . Two rules (I and II) allow you to increase the 
size of strings (but only in very rigid, pr sc ribed ways, of course); and two others allow 
you to shrink strings somewhat (again in very rigid ways). There seems to be an endless variety to the order in which these different t ypes of rules might be applied, and this gives 
hope that one way or another, MU could be produced. It might involve lengthening the

hope that one way or another, MU could be produced. It might involve lengthening the 
string to some gigantic size, and then ex tracting piece after piece until only two symbols 
are left; or, worse yet, it might involve successive stages of lengthening and then 
shortening and then lengthening and then shor tening, and so on. But there is no guarantee 
it. As a matter of fact, we already observed that U cannot be produced at all and it will 
make no difference if you lengthen and shorten till kingdom come. 
Still, the case of U and the case of MU seem quite different. It is by very 
superficial feature of U that we recognize the impossibility of producing it: it doesn't

superficial feature of U that we recognize the impossibility of producing it: it doesn't 
begin with an M (whereas all theorems must). It is very convenient to have such a simple 
way to detect nontheorems. However w ho says that that test will detect all nontheorems? 
There may be lots strings which begin with M but are not producible. Maybe MU is one 
of them. That would mean that the "first-letter test" is of limited usefulness able only to detect a portion of the nontheorems, but missi ng others. B there remains the possibility of

of them. That would mean that the "first-letter test" is of limited usefulness able only to detect a portion of the nontheorems, but missi ng others. B there remains the possibility of 
some more elaborate test which discriminates perfectly between those strings which can be produced by the rules and those which ca nnot. Here we have to face the question, 
"What do mean by a test?" It may not be obvious why that question makes sense, of 
important, in this context. But I will give an  example of a "test" which somehow seems to 
violate the spirit of the word. 
Imagine a genie who has all the time in the world, and who enjoys using it to 
produce theorems of the MIU -system, in a rather methodical way. Here, for instance, is a

produce theorems of the MIU -system, in a rather methodical way. Here, for instance, is a 
possible way the genie might go about it 
 
Step 1: Apply every applicable rule to th e axiom MI. This yields two new theorems 
MIU, MII.  
Step 2: Apply every applicable rule to th e theorems produced in step 1. This yields 
three new theorems: MIIU, MIUIU, MIIII .

The MU-puzzle 48Step 3: Apply every applicable rule to th e theorems produced in step 2. This yields 
five new theorems: MIIIIU, MIIUIIU, MIUIUIUIU, MIIIIIIII, MUI.  
  This method produces every single theorem soone r or later, because the rules are applied 
in every conceivable order. (See Fig. 11.) A ll of the lengthening-shortening alternations 
which we mentioned above eventually get ca rried out. However, it is not clear how long 
to wait for a given string  
 
FIGURE 11. A systematically constructed "tree" of all the theorems of the MIU -system. 
The N th level down contains those theore ms whose derivations contain exactly N steps. 
The encircled numbers tell which rule was employed. Is MU anywhere in this tree? 
 
 to appear on this list, since theorems are listed according to the shortness of their derivations. This is not a very useful order, if you are interested in a specific string (such 
as MU), and you don't even know if it has a ny derivation, much less how long that

as MU), and you don't even know if it has a ny derivation, much less how long that 
derivation might be. 
Now we state the proposed "theoremhood-test": 
 
Wait until the string in question is produced; when that happens, you know it is a theorem-and if it never happens, you know that it is not a theorem. 
 This seems ridiculous, because it presupposes that we don't mind waiting around literally an infinite length of time for our answer. Th is gets to the crux of the matter of what 
should count as a "test". Of prime importance is a guarantee that we will get our answer 
in a finite length of time. If there is a test for theoremhood, a test which does always

in a finite length of time. If there is a test for theoremhood, a test which does always 
terminate in a finite

The MU-puzzle 49 
amount of time, then that test is called a decision procedure  for the given formal system. 
When you have a decision procedure,  then you have a very concrete 
characterization of the nature of all theorems  in the system. Offhand, it might seem that 
the rules and axioms of the formal system provide no less complete a characterization of 
the theorems of the system than a decision procedure would. The tricky word here is "characterization". Certainly the rules of inference and the axioms of the MIU -system do 
characterize, implicitly, those strings that are theorems. Even more  implicitly, they 
characterize those strings that are not theo rems. But implicit characterization is not 
enough, for many purposes. If someone claims to have a characterization of all theorems,

enough, for many purposes. If someone claims to have a characterization of all theorems, 
but it takes him infinitely long to deduce that some particular string is not a theorem, you would probably tend to say that there is something lacking in that characterization-it is not quite concrete enough. And that is why disc overing that a decision procedure exists is 
a very important step. What the discovery mean s, in effect, is that you can perform a test 
for theoremhood of a string, and that, even  if the test is complicated, it is guaranteed to 
terminate . In principle, the test is just as easy, just  as mechanical, just as finite, just as full

terminate . In principle, the test is just as easy, just  as mechanical, just as finite, just as full 
of certitude, as checking whether the first letter of the string is M. A decision procedure 
is a "litmus test" for theoremhood! 
Incidentally, one requirement on form al systems is that the set of axioms  must be 
characterized by a decision procedure-there must be a litmus test for axiomhood. This 
ensures that there is no problem in getting o ff the ground at the beginning, at least. That 
is the difference between the set of axioms a nd the set of theorems: the former always has 
a decision procedure, but the latter may not. 
I am sure you will agree that when you l ooked at the MIU-system for the first 
time, you had to face this problem exactl y. The lone axiom was known, the rules of

time, you had to face this problem exactl y. The lone axiom was known, the rules of 
inference were simple, so the theorems ha d been implicitly characterized-and yet it was 
still quite unclear what the consequences of that characterization were. In particular, it 
was still totally unclear whether MU is, or is not, a theorem.

The MU-puzzle 50
 
FIGURE 12. Sky Castle, by M. C.: Escher (woodcut, 1928).

Two-Part Invention 
 51Two-Part Invention 
 
or, 
What the Tortoise Said to Achilles 
by Lewis Carroll' 
 
Achilles had overtaken the Tortoise, and had seated himself comfortably on its back. 
"So you've got to the end of our race-c ourse?" said the Tortoise. "Even though it 
DOES consist of an infinite series of di stances? I thought some wiseacre or other had 
proved that the thing couldn't be done?" 
"It CAN be done," said Ac hilles. "It HAS been done! Solvitur ambulando . You see the 
distances were constantly DIMINISHING; and so-" 
"But if they had been constantly INCREASING?" the Tortoise interrupted. "How 
then?" 
"Then I shouldn't be here," Achilles mode stly replied; "and You would have got 
several times round the world, by this time!" 
"You flatter me-FLATTEN, I mean," said th e Tortoise; "for you ARE a heavy weight, 
and NO mistake! Well now, would you like to h ear of a race-course, that most people 
fancy they can get to the end of in two or three steps, while it REALLY consists of an

then?" 
fancy they can get to the end of in two or three steps, while it REALLY consists of an 
infinite number of distances, each one longer than the previous one?" 
"Very much indeed!" said the Grecian warri or, as he drew from his helmet (few 
Grecian warriors possessed POCKETS in thos e days) an enormous note-book and pencil. 
"Proceed! And speak SLOWLY, please! SHORTHAND isn't invented yet!" 
"That beautiful First Proposition by Euclid!"  the Tortoise murmured dreamily. "You 
admire Euclid?" 
"Passionately! So far, at least, as one C AN admire a treatise that won't be published 
for some centuries to come!"

for some centuries to come!" 
"Well, now, let's take a little bit of the argument in that First Proposition just TWO 
steps, and the conclusion drawn from them. Kindly enter them in your note-book. And in order to refer to them conveniently, let's call them A, B, and Z: 
 
(A) Things that are equal to the same are equal to each other. (B) The two sides of this Triangle are things that are equal  to the same. (Z) The two sides of this Triangle are equal to each other. 
 Readers of Euclid will grant, I suppose, that Z follows logically from A and B, so that

 Readers of Euclid will grant, I suppose, that Z follows logically from A and B, so that 
any one who accepts A and B as true, MUST accept Z as true?"  
"Undoubtedly! The youngest child in a High School-as soon as High

 52Schools are invented, which will not be till some two thousand years later-will grant 
THAT." 
"And if some reader had NOT yet accepted A and B as true, he might still accept the 
SEQUENCE as a VALID one, I suppose?" 
"No doubt such a reader might exist. He mi ght say, `I accept as true the Hypothetical 
Proposition that, IF A and B be true, Z must be true; but I DON'T accept A and B as true.' 
Such a reader would do wisely in abandoning Euclid, and taking to football." 
"And might there not ALSO be some reader  who would say `I accept A and B as true, 
but I DON'T accept the Hypothetical'?" 
"Certainly there might. HE, also, had better take to football." "And NEITHER of these readers," the To rtoise continued, "is AS YET under any 
logical necessity to accept Z as true?" 
"Quite so," Achilles assented. "Well, now, I want you to consider ME as a reader of the SECOND kind, and to force 
me, logically, to accept Z as true."

me, logically, to accept Z as true." 
"A tortoise playing football would be-" Achilles was beginning. `-an anomaly, of course," the Tortoise ha stily interrupted. "Don't wander from the 
point. Let's have Z first, and football afterwards!" 
"I'm to force you to accept Z, am I?" Achilles said musingly. "And your present 
position is that you accept A and B, but you DON'T accept the Hypothetical-" 
"Let's call it C," said the Tortoise. "-but you DON'T accept  (C) If A and B are true, Z must be true."  "That is my present position," said the Tortoise.  
"Then I must ask you to accept C."

"Then I must ask you to accept C." 
"I'll do so," said the Tortoise, "as soon as  you've entered it in that notebook of yours. 
What else have you got in it?" 
"Only a few memoranda," said Achilles, nervously fluttering the leaves: "a few 
memoranda of-of the battles in which I have distinguished myself!" 
"Plenty of blank leaves, I see!" the Tortoi se cheerily remarked. "We shall need them 
ALL!" (Achilles shuddered.) "Now write as I dictate:

ALL!" (Achilles shuddered.) "Now write as I dictate: 
 (A) Things that are equal to the same are equal to each other.  (B) The two sides of this Triangle are things that are equal to the same. (C) If A and B are true, Z must be true. (Z) The two sides of this Triangle are equal to each other."  "You should call it D, not Z," said Achilles. "It comes NEXT to the other three. If you 
accept A and B and C, you MUST accept Z.

 53“And why must I?” 
"Because it follows LOGICALLY from them. If A and B and C are true, Z MUST be 
true. You can't dispute THAT, I imagine?" 
"If A and B and C are true, Z MUST be tr ue," the Tortoise thoughtfully repeated. 
"That's ANOTHER Hypothetical, isn't it? And, if  I failed to see its truth, I might accept A 
and B and C, and STILL not accept Z, mightn't I?" 
"You might," the candid hero admitted; "t hough such obtuseness would certainly be 
phenomenal. Still, the event is POSSIBLE. So I must ask you to grant ONE more 
Hypothetical." 
"Very good, I'm quite willing to grant it, as soon as you've written it down. We will 
call it 
 (D) If A and B and C are true, Z must be true. 
 Have you entered that in your note-book?" 
"I HAVE!" Achilles joyfully exclaimed, as he ran the pencil into its sheath. "And at 
last we've got to the end of this ideal race-course! Now that you accept A and B and C 
and D, OF COURSE you accept Z."

call it 
and D, OF COURSE you accept Z." 
"Do I?" said the Tortoise innocently. "Let's make that quite clear . I accept A and B and 
C and D. Suppose I STILL refused to accept Z?" 
"Then Logic would take you by the thro at, and FORCE you to do it!" Achilles 
triumphantly replied. "Logic would tell you, `You can't help yourself. Now that you've 
accepted A and B and C and D, you MUST accept Z!' So you've no choice, you see.", 
"Whatever LOGIC is good enough to tell me  is worth WRITING DOWN," said the 
Tortoise. "So enter it in your book, please. We will call it

Tortoise. "So enter it in your book, please. We will call it 
 
(E) If A and B and C and D are true, Z must be true. 
 Until I've granted THAT, of course I needn't grant Z. So it's quite a NECESSARY 
step, you see?" 
"I see," said Achilles; and there was a touch of sadness in his tone. Here the narrator, having pressing busine ss at the Bank, was obliged to leave the 
happy pair, and did not again pass the spot un til some months afterwards. When he did 
so, Achilles was still seated on the back of  the much-enduring Tortoise, and was writing 
in his notebook, which appeared to be near ly full. The Tortoise was saying, "Have you

in his notebook, which appeared to be near ly full. The Tortoise was saying, "Have you 
got that last step written down? Unless I've  lost count, that makes a thousand and one. 
There are several millions more to come . And WOULD you mind, as a personal favour, 
considering what a lot of instruction this colloquy of ours will provide for the Logicians 
of the Nineteenth Century-WOULD you mind adopting a pun that my cousin the Mock-
Turtle will then make, and allowing yourself to be renamed TAUGHT-US?" 
"As you please," replied the weary warrior, in the hollow tones of despair, as he buried

"As you please," replied the weary warrior, in the hollow tones of despair, as he buried 
his face in his hands. "Provided that YOU,  for YOUR part, will adopt a pun the Mock-
Turtle never made, and allow yourself to be re-named A KILL-EASE!"

Meaning and Form in Mathematics 
 54CHAPTER 11 
 
Meaning and Form 
in Mathematics. 
 
 THIS Two-Part Invention  was the inspiration for my two characters. Just as Lewis 
Carroll took liberties with Zeno's Tortoise and Achilles, so have I taken liberties with Lewis Carroll's Tortoise and Achilles. In Ca rroll's dialogue, the same events take place 
over and over again, only each time on a highe r and higher level; it is a wonderful 
analogue to Bach's Ever-Rising Canon. The Ca rrollian Dialogue, with its wit subtracted 
out, still leaves a deep philosophical problem: Do w ords and thoughts follow formal 
rules, or do they not ? That problem is the problem of this book. 
In this Chapter and the next, we will look at several new formal systems. This will give 
us a much wider perspective on the concept of  formal system. By the end of these two 
Chapters, you should have quite a good idea of the power of formal systems, and why 
they are of interest to mathematicians and logicians.  
The pq-System

The pq-System 
 The formal system of this Chapter is called the pq-system . It is not important to 
mathematicians or logicians-in fact, it is ju st a simple invention of mine. Its importance 
lies only in the fact that it provides an excelle nt example of many ideas that play a large 
role in this book. There are three distinct symbols of the pq-system:  
p q     
- 
 
-the letters p, q, and the hyphen. 
The pq-system has an infinite number of axio ms. Since we can't write them all down, we

p q     
- 
 
The pq-system has an infinite number of axio ms. Since we can't write them all down, we 
have to have some other way of describing what they are. Actually, we want more than just a description of the axioms; we want a way to tell whether some given string is an axiom or not. A mere description of axio ms might characterize them fully and yet 
weakly-which was the problem with the way theorems in the MIU -system were 
characterized. We don't want to have to str uggle for an indeterminate-possibly infinite 
length of time, just to find out if some string is an axiom or not. Therefore, we will define

length of time, just to find out if some string is an axiom or not. Therefore, we will define 
axioms in such a way that there is an obvious decision procedure for axiomhood of a 
string composed of p's, q's, and hyphens.

 55 
DEFINITION: x p-qx is an axiom, whenever x is composed of hyphens only. 
 Note that 'x' must stand for the same string of hyphens in both occurrences For example, -
-p-q---is an axiom. The literal expression ` xp-qx -' i,, not an axiom, of course (because `x' 
does not belong to the pq-system); it is more like a mold in which all axioms are cast-and 
it is called an axiom schema. 
The pq-system has only one rule of production: 
 RULE: Suppose x, y, and z all stand for par ticular strings containing only hyphens. And 
suppose that x py qz is known to be a theorem. The` x py-qz - is a theorem. 
 For example, take x to be'--', y to be'---', and z to be'-'. The rule tells us:  
If --p---q- turns out to be a theorem, then so will -- p----q--. 
  As is typical of rules of production, the statement establishes a causal connection between 
the theoremhood of two strings, but without asserting theoremhood for either one on its own.

RULE: Suppose x, y, and z all stand for par ticular strings containing only hyphens. And 
the theoremhood of two strings, but without asserting theoremhood for either one on its own. 
A most useful exercise for you is to find a decision procedure for the theorems of 
the pq-system. It is not hard; if you play  around for a while you will probably pick it up. 
Try it. 
  
The Decision Procedure 
 I presume you have tried it. First of all, though it may seem too obvious to mention, I would like to point out that every theorem of  the pq-system has three separate groups of 
hyphens, and the separating elements are one p, and one q, in that order. (This can be

Try it. 
  
hyphens, and the separating elements are one p, and one q, in that order. (This can be 
shown by an argument based on "heredity", just the way one could show that all MIU -
system theorems had to begin with M.) This means that we can rule out, from its form 
alone, o string such as -- p--p--p--q  . 
Now, stressing the phrase "from its form alone" may seem silly; what else is there to a 
string except its form? What else could possibl y play a roll in determining its properties? 
Clearly nothing could. But bear this in mint as the discussion of formal systems goes on;

Clearly nothing could. But bear this in mint as the discussion of formal systems goes on; 
the notion of "form" will star to get rather  more complicated and abstract, and we will 
have to think more about the meaning of the word "form". In any case, let us give the name well formed string  to any string which begins w ith a hyphen-group, then ha one p, 
then has a second hyphen-group, then a q, and then a final hyphen-group. Back to the decision procedure ... The crite rion for theoremhood is that the first two 
hyphen-groups should add up, in length, to the third

 56hyphen-group. for instance , --p--q -  is a theorem, since 2 plus 2 equals 4, whereas --p--q-
is not, since 2 plus 2 is not 1. To see why this is the proper criterion, look first at the axiom schema. Obviously, it only manufactur es axioms which satisfy the addition 
criterion. Second, look at the rule of production. If the first string satisfies the addition 
criterion, so must the second one-and conversel y, if the first string does not satisfy the 
addition criterion, then neither does the second string. The rule makes the addition criterion into a hereditary property of theore ms: any theorem passes the property on to its 
offspring. This shows why the addition criterion is correct. 
There is, incidentally, a fact about the pq-system which would enable us to say 
with confidence that it has a decision procedure, even before finding the addition criterion. That fact is that the pq-system is not complicated by the opposing currents of

with confidence that it has a decision procedure, even before finding the addition criterion. That fact is that the pq-system is not complicated by the opposing currents of 
lengthening and shortening  rules; it has only lengthening rules. Any formal system which 
tells you how to make longer theorems from shorter ones, but never the reverse, has got to have a decision procedure for its theo rems. For suppose you are given a string. First 
check whether it's an axiom or not (I am assuming that there is a decision procedure for axiomhood-otherwise, things are hopeless). If it is an axiom, then it is by definition a

check whether it's an axiom or not (I am assuming that there is a decision procedure for axiomhood-otherwise, things are hopeless). If it is an axiom, then it is by definition a 
theorem, and the test is over. So suppose inst ead that it's not an axiom. Then, to be a 
theorem, it must have come from a shorter string, via one of the rules. By going over the various rules one by one, you can pinpoint not  only the rules that could conceivably 
produce that string, but also exactly which shorter strings could be its forebears on the "family tree". In this way, you "reduce" th e problem to determining whether any of 
several new but shorter strings is a theorem. E ach of them can in turn be subjected to the

several new but shorter strings is a theorem. E ach of them can in turn be subjected to the 
same test. The worst that can happen is a proliferation of more and more, but shorter and shorter, strings to test. As you continue in ching your way backwards in this fashion, you 
must be getting closer to the source of all theorems-the axiom schemata. You just can't 
get shorter and shorter indefinitely; therefore,  eventually either you will find that one of 
your short strings is an axiom, or you'll come  to a point where you're stuck, in that none 
of your short strings is an axiom, and none of them can be further shortened by running

of your short strings is an axiom, and none of them can be further shortened by running 
some rule or other backwards. This points out that there really is not much deep interest 
in formal systems with lengthening rules only; it is the interplay of lengthening and shortening rules that gives formal systems a certain fascination.. 
 
Bottom-up vs. Top-down 
 
The method above might be called a top-down  decision procedure, to be contrasted with a 
bottom-up  decision procedure, which I give now. It  is very reminiscent of the genie's 
systematic theorem-generating method for the MIU -system, but is complicated by the 
presence of an axiom schema. We are going to  form a "bucket" into which we throw

presence of an axiom schema. We are going to  form a "bucket" into which we throw 
theorems as they are generated. Here is how it is done:

 57 
(1a) Throw the simplest possible axiom (-p-q-- ) into the bucket. 
(I b) Apply the rule of inference to the ite m in the bucket, and put the result into the 
bucket. 
(2a) Throw the second-simplest axiom into the bucket. (2b) Apply the rule to each item in the bucket, and throw all results into the bucket. (3a) Throw the third-simplest axiom into the bucket. (3b) Apply the rule to each item in the bucket, and throw all results into the bucket.  
etc., etc. 
 A moment's reflection will show that you can 't fail to produce every theorem of the pq-
system this way. Moreover, the bucket is ge tting filled with longer and longer theorems, 
as time goes on. It is again a consequence of th at lack of shortening rules. So if you have 
a particular string, such as --p---q ---- , which you want to test for theoremhood, just 
follow the numbered steps, checking all the wh ile for the string in question. If it turns up-

etc., etc. 
theorem! If at some point everything that goe s into the bucket is longer than the string in 
question, forget it-it is not a theorem. This decision procedure is bottom=up because it is 
working its way up from the basics, which is to say the axioms. The previous decision procedure is top-down because it does precisely  the reverse: it works its way back down 
towards the basics.  
Isomorphisms Induce Meaning 
 Now we come to a central issue of this  Chapter-indeed of the book. Perhaps you have

 Now we come to a central issue of this  Chapter-indeed of the book. Perhaps you have 
already thought to yourself that the pq-theorems are like additions. The string --p---q---  is 
a theorem because 2 plus 3 equals 5. It could even occur to you that the theorem --p---q--  
is a statement , written in an odd notation, whose meaning  is that 2 plus 3 is 5. Is this a 
reasonable way to look at things ? Well, I deliberately chose ' p' to remind you of 'plus', 
and 'q' to remind you of 'equals' . . . So, does the string --p---q----  actually mean  "2 plus 
3 equals 5"?

3 equals 5"? 
What would make us feel that way? My answer would be that we have perceived 
an isomorphism  between pq-theorems and additi ons. In the Introduction, the word 
"isomorphism" was defined as an informa tion preserving transformation. We can now go 
into that notion a little more deeply, and s ee it from another perspective. The word 
"isomorphism' applies when two complex stru ctures can be mapped onto each other, in

"isomorphism' applies when two complex stru ctures can be mapped onto each other, in 
such a way that to each part of one structure there is a corresponding part in the other structure, where "corresponding" means that the two part play similar roles in their 
respective structures. This usage of the wo rd "isomorphism" is derived from a more 
precise notion in mathematics.

 58It is cause for joy when a mathematician discovers an isomorphism between two 
structures which he knows. It is often a "bolt from the blue", and a source of 
wonderment. The perception of an isomor phism between two known structures is a 
significant advance in knowledge-and I claim that  it is such perceptions of isomorphism 
which create meanings  in the minds of people. A fi nal word on the perception of 
isomorphisms: since they come in many shap es and sizes, figuratively speaking, it is not 
always totally clear when you really have found an isomorphism. Thus, "isomorphism" is 
a word with all the usual vagueness of words-which is a defect but an advantage as well. 
In this case, we have an excellent pr ototype for the concept of isomorphism. 
There is a "lower level" of our isomorphism -that is, a mapping between the parts of the 
two structures: 
 
p    <= => plus 
   q    <= => equals 
-    <= => one 
--   <= => two 
---- <= => three 
etc.

p    <= => plus 
-    <= => one 
--   <= => two 
etc. 
 This symbol-word correspondence has a name: interpretation . 
Secondly, on a higher level, there is the correspondence between true statements 
and theorems. But-note carefully-this higher-l evel correspondence could not be perceived 
without the prior choice of an interpretati on for the symbols. Thus it would be more 
accurate to describe it as a correspondence between true statements and interpreted  
theorems. In any case we have displayed a tw o-tiered correspondence, which is typical of 
all isomorphisms.

all isomorphisms. 
When you confront a formal system you know nothing of, and if you hope to 
discover some hidden meaning in it, your problem  is how to assign interpretations to its 
symbols in a meaningful way-that is, in such a way that a higher-level correspondence 
emerges between true statements and theorems . You may make several tentative stabs in 
the dark before finding a good set of words to associate with the symbols. It is very 
similar to attempts to crack a code, or to  decipher inscriptions in an unknown language

similar to attempts to crack a code, or to  decipher inscriptions in an unknown language 
like Linear B of Crete: the onl y way to proceed is by trial and error, based on educated 
guesses. When you hit a good choice, a "meaningf ul" choice, all of a sudden things just 
feel right, and work speeds up enormously. Pr etty soon everything falls into place. The 
excitement of such an experience is captured in The Decipherment of Linear  B by John 
Chadwick. 
But it is uncommon, to say the least, for someone to be in the position of 
"decoding" a formal system turned up in th e excavations of a ruined civilization!

Chadwick. 
"decoding" a formal system turned up in th e excavations of a ruined civilization! 
Mathematicians (and more recently, linguists, philosophers, and some others) are the only 
users of formal systems, and they invariably have an interpretation in mind for the formal 
systems which they use and publish. The idea of these people is to set up a formal system 
whose

 59 
Theorems reflect some portion of reality isom orphically. In such a case, the choice of 
symbols is a highly motivated one, as is th e choice of typographical rules of production. 
When I devised the pq-system, I was in position. You see why I chose the symbols I 
chose. It is no accident theorems are is omorphic to additions; it happened because I 
deliberately sought out a way to reflect additions typographically.  
Meaningless and Meaningful Interpretations 
 You can choose interpretations other than the one I chose. You need make every 
theorem come out true. But there would be very little reason make an interpretation in 
which, say, all theorems came out false, certainly even less reason to make an interpretation under which there is no correla tion at all, positive or negative, between 
theoremhood and tri Let us therefore make  a distinction between two types of 
interpretations a formal system. First, we can have a meaningless  interpretation, one un

You can choose interpretations other than the one I chose. You need make every 
interpretations a formal system. First, we can have a meaningless  interpretation, one un 
which we fail to see any isomorphic connecti on between theorems of system, and reality. 
Such interpretations abound-any random choice a will do. For instance, take this one: 
 
p <= => horse 
q <= => happy 
- <= => apple 
  Now -p-q--  acquires a new interpretation: "apple horse apple hat apple apple"-a 
poetic sentiment, which might appeal to horse s, and mi! even lead them to favor this

poetic sentiment, which might appeal to horse s, and mi! even lead them to favor this 
mode of interpreting pq-strings! However, t interpretation has very little "meaningfulness"; under interpretative, theore ms don't sound any truer, or any better, 
than nontheorems. A ho might enjoy "happy happy happy apple horse" (mapped onto  q q 
q) just as much as any interpreted theorem. 
The other kind of interpretation will be called meaningful . Under si an

The other kind of interpretation will be called meaningful . Under si an 
interpretation, theorems and truths correspond-that is, an isomorphism exists between theorems and some portion of reality. That  is why it is good to distinguish between 
interpretations and meanings . Any old word can be used as an interpretation for ` p', but 
`plus' is the only meaningful choice  we've come up with. In summary, the meaning of ` p' 
seems to be 'plus’ though it can have a million different interpretations. 
 
Active vs. Passive Meanings 
 
Probably the most significant fact of this Chapter, if understood deeply this: the pq-

system seems to force us into recognizing that symbols of a formal system, though 
initially without meaning, cannot avoid taki ng on "meaning" of sorts at least if an 
isomorphism is found. The difference between meaning it formal system and in a 
language is a very important one, however. It is this:

 60in a language, when we have learned a meaning for a word, we then mar-c new 
statements based on the meaning of the word. In a sense the meaning becomes active , 
since it brings into being a new rule for creating sentences. This means that our command of language is not like a finished product: th e rules for making sentences increase when 
we learn new meanings. On the other hand, in a formal system, the theorems are 
predefined, by the rules of production. We can choose "meanings" based on an isomorphism (if we can find one) between theorems and true statements. But this does not give us the license to go out and add new theorems to the established theorems. That 
is what the Requirement of Formality in Chapter I was warning you of. 
In the MIU-system, of course, there was no temptation to go beyond the four 
rules, because no interpretation was sought or found. But here, in our new system, one

rules, because no interpretation was sought or found. But here, in our new system, one 
might be seduced by the newly found "meaning"  of each symbol into thinking that the 
string  
--p--p--p--q 
 
is a theorem. At least, one might wish that this string were a theorem. But wishing doesn't 
change the fact that it isn't. And it would be a serious mistake to think that it "must" be a theorem, just because 2 plus 2 plus 2 plus 2 equals 8. It would even be misleading to 
attribute it any meaning at all, since it is not well-formed, and our meaningful

string  
 
attribute it any meaning at all, since it is not well-formed, and our meaningful 
interpretation is entirely derived from looking at well-formed strings. 
In a formal system, the meaning must remain passive ; we can read each string 
according to the meanings of its constituent symbols, but we do not have the right to 
create new theorems purely on the basis of the meanings we've assigned the symbols. 
Interpreted formal systems straddle the line between systems without meaning, and 
systems with meaning. Their strings can be thought of as "expressing" things, but this

systems with meaning. Their strings can be thought of as "expressing" things, but this 
must come only as a consequence of the formal properties of the system. 
 
Double-Entendre! 
 And now, I want to destroy any illusion about  having found the meanings for the symbols 
of the pq-system. Consider the following association:  
p <= => equals 
       q  <= => taken from 
-  <= => one -- <= => two 
etc. 
 Now, --p---q---- has a new interpretation: "2 equals 3 taken from 5". Of course it is a true 
statement. All theorems will come out true under this new interpretation. It is just as

etc. 
statement. All theorems will come out true under this new interpretation. It is just as 
meaningful as the old one. Obviously, it is silly to ask, "But which one is the meaning of 
the string?" An interpreta

 61tion will me meaningful to the extent that it accurately reflects some isomorphism to the 
real world. When different aspects of the real  world a isomorphic to each other (in this 
case, additions and subtractions), or single formal system can be isomorphic to both, and 
therefore can take ( two passive meanings. This kind of double-valuedness of symbols at 
strings is an extremely important phenom enon. Here it seems trivial curious, annoying. 
But it will come back in deeper contexts and bring with it a great richness of ideas. 
Here is a summary of our observations a bout the pq-system. Und either of the two 
meaningful interpretations given, every well- form( string has a grammatical assertion for 
its counterpart-some are true, son false. The idea of well formed strings  in any formal 
system is that they a those strings which, when interpreted symbol for symbol, yield grammatical  sentences. (Of course, it depends on th e interpretation, but usually, there one

system is that they a those strings which, when interpreted symbol for symbol, yield grammatical  sentences. (Of course, it depends on th e interpretation, but usually, there one 
in mind.) Among the well-formed strings occu r the theorems. The: are defined by an 
axiom schema, and a rule of production. My goal in inventing the pq-system was to 
imitate additions: I wanted every theorem] to express a true addition under interpretation; conversely, I wanted every true addition of  precisely two positive integers to be 
translatable into a string, which would be a theorem. That goal was achieved. Notice, 
then fore, that all false additions, such as "2 plus 3 equals 6", are mapped into strings which are well-formed, but which are not theorems.

then fore, that all false additions, such as "2 plus 3 equals 6", are mapped into strings which are well-formed, but which are not theorems.   
Formal Systems and Reality 
 This is our first example of 'a case where a formal system is based upon portion of reality, and seems to mimic it perfectly, in that  its theorems a] isomorphic to truths about 
that part of reality. However, reality and tt formal system are independent. Nobody need 
be aware that there is a isomorphism between the two. Each side stands by itself-one plus 
or equals two, whether or not we know that -p-q--  is a theorem; and -p-q--  is still a 
theorem whether or not we connect it with addition.

This is our first example of 'a case where a formal system is based upon portion of reality, and seems to mimic it perfectly, in that  its theorems a] isomorphic to truths about 
theorem whether or not we connect it with addition. 
You might wonder whether making this form al system, or any form system, sheds 
new light on truths in the domain of its inte rpretation. Hat we learned any new additions 
by producing pq-theorems? Certainly not but we have learned something about the nature 
of addition as process-namely, that it is  easily mimicked by a typographical rule 
governing meaningless symbols. This still shoul d not be a big surprise sing addition is

governing meaningless symbols. This still shoul d not be a big surprise sing addition is 
such a simple concept. It is a commonplace that addition can I captured in the spinning 
gears of a device like a cash register. 
But it is clear that we have hardly scratched the surface, as far formal systems go; 
it is natural to wonder about what portion of r eality co be imitated in its behavior by a set 
of meaningless symbols governed I formal rules. Can all of reality be turned into a formal 
system? In a very broad sense, the answer mi ght appear to be yes. One could suggest, for

system? In a very broad sense, the answer mi ght appear to be yes. One could suggest, for 
instance, that reality is itself nothing but one very complicated formal

 62system. Its symbols do not move around on pape r, but rather in a three-dimensional 
vacuum (space); they are the elementary particles of which everything is composed. (Tacit assumption: that there is an end to the descending chain of matter, so that the 
expression "elementary particles" makes sense. ) The "typographical rules" are the laws of 
physics, which tell how, given the positions a nd velocities of all particles at a given 
instant, to modify them, resulting in a new set of positions and velocities belonging to the "next" instant. So the theorems of th is grand formal system are the possible 
configurations of particles at different times in the history of the universe. The sole axiom 
is (or perhaps, was) the original configura tion of all the particle s at the "beginning of 
time". This is so grandiose a conception, however, that it has only the most theoretical interest; and besides, quantum mechanics (and other parts of physics) casts at least some

time". This is so grandiose a conception, however, that it has only the most theoretical interest; and besides, quantum mechanics (and other parts of physics) casts at least some 
doubt on even the theoretical worth of this id ea. Basically, we are asking if the universe 
operates deterministically, which is an open question.  
Mathematics and Symbol Manipulation 
 Instead of dealing with such a big picture, let's limit ourselves to mathematics  as 
our "real world". Here, a serious question arises: How can we be sure, if we've tried to model a formal system on some part of mathematics, that we've done the job accurately-especially if we're not one hundred per cent familiar with that portion of mathematics 
already? Suppose the goal of the formal syst em is to bring us new knowledge in that

already? Suppose the goal of the formal syst em is to bring us new knowledge in that 
discipline. How will we know that the interp retation of every theorem is true, unless 
we've proven that the isomorphism is pe rfect? And how will we prove that the 
isomorphism is perfect, if we don't already know  all about the truths in the discipline to 
begin with? 
Suppose that in an excavation somewh ere, we actually did discover some 
mysterious formal system. We would try out various interpretations and perhaps eventually hit upon one which seemed to make  every theorem come out true, and every

mysterious formal system. We would try out various interpretations and perhaps eventually hit upon one which seemed to make  every theorem come out true, and every 
nontheorem come out false. But this is some thing which we could only check directly in 
a finite number of cases. The set of theore ms is most likely infinite. How will we know  
that all theorems express truths under this interpretation, unless we know everything there 
is to know about both the formal system and the corresponding domain of interpretation? 
It is in somewhat this odd position that we will find ourselves when we attempt to

It is in somewhat this odd position that we will find ourselves when we attempt to 
match the reality of natural numbers (i.e., the nonnegative integers: 0, 1, 2, ...) with the 
typographical symbols of a formal system. We will try to understand the relationship 
between what we call "truth" in number th eory and what we can get at by symbol 
manipulation. 
So let us briefly look at the basis for calling some statements of number theory 
true, and others false. How much is 12 times 12? Everyone knows it is 144. But how 
many of the people who give that answer have actually at

 63 
any time in their lives drawn a 12 by 12 recta ngle, and then counted the little squares in 
it? Most people would regard the drawing and counting unnecessary. They would instead 
offer as proof a few marks on paper, such as are shown below:  
   12 
X 12 
------ 
    24 
12 
------ 
 144 
  
And that would be the "pr oof". Nearly everyone believes that if you counted the 
squares, you would get 144 of them; few people feel that outcome is in doubt. 
The conflict between the two points of vi ew comes into sharper focus when you 
consider the problem of determining the value 987654321 x 123456789. First of all, it is 
virtually impossible to construct the appropriate  rectangle; and what is worse, even if it 
were  constructed and huge armies of people sp ent centuries counting the little squares, o 
a very gullible person would be willing to believe  their final answer. It is just too likely

a very gullible person would be willing to believe  their final answer. It is just too likely 
that somewhere, somehow, somebody bobbled just  a little bit. So is it ever possible to 
know what the answer is? If you trust the sy mbolic process which involves manipulating 
digits according to certain simple rules, yes.  That process is presented to children as a 
device which gets right answer; lost in th e shuffle, for many children, are the rhyme 
reason of that process. The digit-shunting laws for multiplication are based mostly on a 
few properties of addition and multiplication which are assumed to hold for all numbers. 
 
The Basic Laws of Arithmetic

The Basic Laws of Arithmetic 
 The kind of assumption I mean is illustrated below. Suppose that you down a few sticks:  
/   //   //   //    /    / 
 Now you count them. At the same time, so mebody else counts them, starting from the 
other end. Is it clear that th e two of you will get the s: answer? The result of a counting 
process is independent of the way in which it is done. This is really an assumption about

/   //   //   //    /    / 
process is independent of the way in which it is done. This is really an assumption about 
what counting i would be senseless to try to prove it, because it is so basic; either you s or you don't-but in the latter case, a proof won't help you a bit. 
From this kind of assumption, one can ge t to the commutativity and associativity 
of addition (i.e., first that b + c = c + b  always, and second that b + (c + d) = (b + c) + d  
always). The same assumption can also you to the commutativity and associativity of 
multiplication; just think of

 64many cubes assembled to form a large rectangular solid. Multiplicative commutativity 
and associativity are just the assumptions that  when you rotate the solid in various ways, 
the number of cubes will not change. Now thes e assumptions are not verifiable in all 
possible cases, because the number of such cases  is infinite. We take them for granted; 
we believe them (if we ever think about th em) as deeply as we could believe anything. 
The amount of money in our pocket will not ch ange as we walk down the street, jostling 
it up and down; the number of books we have w ill not change if we pack them up in a 
box, load them into our car, drive one hundred miles, unload the box, unpack it, and place the books in a new shelf. All of this is part of what we mean by number . 
There are certain types of people who, as soon as some undeniable fact is written 
down, find it amusing to show why that "fact" is false after all. I am such a person, and as

down, find it amusing to show why that "fact" is false after all. I am such a person, and as 
soon as I had written down the examples above involving sticks, money, and books, I 
invented situations in which they were wrong. You may have done the same. It goes to 
show that numbers as abstractions are rea lly quite different from the everyday numbers 
which we use. 
People enjoy inventing slogans which viol ate basic arithmetic but which illustrate 
"deeper" truths, such as "1 and 1 make 1" (for lovers), or "1 plus 1 plus 1 equals 1" (the 
Trinity). You can easily pick holes in those slogans, showing why, for instance, using the

Trinity). You can easily pick holes in those slogans, showing why, for instance, using the 
plus-sign is inappropriate in both cases. Bu t such cases proliferate. Two raindrops 
running down a windowpane merge; does one pl us one make one? A cloud breaks up into 
two clouds-more evidence for the same? It is not  at all easy to draw  a sharp line between 
cases where what is happening could be calle d "addition", and where some other word is 
wanted. If you think about the question, you will probably come up with some criterion 
involving separation of the objects in space,  and making sure each one is clearly

involving separation of the objects in space,  and making sure each one is clearly 
distinguishable from all the others. But th en how could one count ideas? Or the number 
of gases comprising the atmosphere? So mewhere, if you try to look it up, you can 
probably find a statement such as, "There are 17 languages in India, and 462 dialects." 
There is something strange about precise statements like that, when the concepts "language" and "dialect" are themselves fuzzy. 
 
Ideal Numbers 
 Numbers as realities misbehave. However, th ere is an ancient and innate sense in 
people that numbers ought not to misbehave. There is something clean and pure in the

people that numbers ought not to misbehave. There is something clean and pure in the 
abstract notion of number, removed from c ounting beads, dialects, or clouds; and there 
ought to be a way of talking about numbers w ithout always having the silliness of reality 
come in and intrude. The hard-edged rules that govern "ideal" numbers constitute arithmetic, and their more advanced consequences constitute number theory. There is only one relevant question to be asked, in making the transition from numbers as practical things to numbers as formal things. Once you have

 65
 
FIGURE 13. Liberation, by M.C. Escher (lithograph, 1955).

 66decided to try to capsulize all of number theory in an ideal system, is it really possible to 
do the job completely? Are numbers so clean a nd crystalline and regular that their nature 
can be completely captured in the ru les of a formal system? The picture Liberation  (Fig. 
13), one of Escher's most beautiful, is a ma rvelous contrast between the formal and the 
informal, with a fascinating transition region. Are numbers really as free as birds? Do 
they suffer as much from being crystallized  into a rule-obeying system? Is there a 
magical transition region between numbers in reality and numbers on paper? 
When I speak of the properties of natu ral numbers, I don't just mean properties 
such as the sum of a particular pair of integers. That can be found out by counting, and 
anybody who has grown up in this century cannot doubt the mechanizability of such 
processes as counting, adding, multiplying, a nd so on. I mean the kinds of properties

processes as counting, adding, multiplying, a nd so on. I mean the kinds of properties 
which mathematicians are interested in exploring, questions for which no counting-process is sufficient to provide the answer-not even theoretically sufficient. Let us take a 
classic example of such a property of natu ral numbers. The statement is: "There are 
infinitely many prime numbers." First of all, there is no counting process which will ever be able to confirm, or refute, this asser tion. The best we could do would be to count

infinitely many prime numbers." First of all, there is no counting process which will ever be able to confirm, or refute, this asser tion. The best we could do would be to count 
primes for a while and concede that there are "a lot". But no amount of counting alone 
would ever resolve the question of whether th e number of primes is finite or infinite. 
There could always be more. The statement- and it is called "Euclid's Theorem" (notice 
the capital "T")-is quite unobvious. It may seem  reasonable, or appealing, but it is not 
obvious. However, mathematicians since Euclid  have always called it true. What is the 
reason? 
 
Euclid's Proof 
 
The reason is that reasoning  tells them it is so. Let us follow the reasoning involved. We

reason? 
 
The reason is that reasoning  tells them it is so. Let us follow the reasoning involved. We 
will look at a variant of Euc lid's proof. This proof works by showing that whatever 
number you pick, there is a prime larger than it. Pick a number-N. Multiply all the positive integers starting with 1 and ending with N; in other words, form the factorial of 
N, written "N!". What you get is divisible by every number up to N. When you add 1 to 
N!, the result  
can't be a multiple of 2 (because it leaves 1 over, when you divide 
by 2); 
can't be a multiple of 3 (because it leaves I over, when you divide 
by 3);

by 2); 
by 3); 
can't be a multiple of 4 (because it leaves 1 over, when you divide 
by 4); 
. 
. 
.

 67can't be a multiple of N (because it leaves 1 over, when you 
divide by N); 
 In other words, N! + 1, if it is divisible at all (other than by 1 and itself only is 
divisible by numbers greater than N. So either it is itself prime, or prime divisors are 
greater than N. But in either case we've show n the must exist a prime above N. The 
process holds no matter what number is. Whatever N is, there is a prime greater than N. 
And thus ends the demonstration of the infinitude of the primes. 
This last step, incidentally, is called generalization , and we will meet again later 
in a more formal context. It is where we  phrase an argument terms of a single number 
(N), and then point out that N was unspecified and therefore the argument is a general 
one. 
Euclid's proof is typical of what c onstitutes "real mathematics". It simple, 
compelling, and beautiful. It illustrates that by taking several rash short steps one can get

one. 
compelling, and beautiful. It illustrates that by taking several rash short steps one can get 
a long way from one's starting point. In our case, t starting points are basic ideas about 
multiplication and division and forth. The s hort steps are the steps of reasoning. And 
though eve individual step of the reasoning seems obvious, the end result is not obvious. 
We can never check directly whether the statem ent is true or not; } we believe it, because 
we believe in reasoning. If you accept reasoning there seems to be no escape route; once

we believe in reasoning. If you accept reasoning there seems to be no escape route; once 
you agree to hear Euclid out, you’ll have to  agree with his conclusion. That's most 
fortunate-because it mea that mathematicians will always agree on what statements to 
label "true and what statements to label "false". 
This proof exemplifies an orderly thought  process. Each statement related to 
previous ones in an irresistible way. This is why it is called "proof'' rather than just "good 
evidence". In mathematics the goal always to  give an ironclad proof for some unobvious

evidence". In mathematics the goal always to  give an ironclad proof for some unobvious 
statement. The very fact of the steps being linked together in an ironclad way suggests ti 
there may be a patterned structure  binding these statements together. TI structure can 
best be exposed by finding a new vocabular y-a stylized vocabulary, consisting of 
symbols-suitable only for expressing statements  about numbers. Then we can look at the 
proof as it exists in its translated version. It  will be a set of statements which are related,

proof as it exists in its translated version. It  will be a set of statements which are related, 
line by line, in some detectable way. But th e statements, since they're represented by 
means a small and stylized set of symbols, take  on the aspect of patterns. In other words, 
though when read aloud, they seem to be stat ements about numb and their properties, still 
when looked at on paper, they seem to be abstract patterns-and the line-by-line structure 
of the proof may start to l ook like slow transformation of patterns according to some few 
typographical rules. 
 
Getting Around Infinity

Getting Around Infinity 
 
Although Euclid's proof is a proof that all numbers have a certain property it avoids 
treating each of the infinitely many cases separately. It gets around

 68it by using phrases like "whatever N is", or "no matter what number N is". We could also 
phrase-the proof over again, so that it uses the phrase "all N". By knowing the appropriate 
context and correct ways of using such phrases , we never have to deal with infinitely 
many statements. We deal with just two or three concepts, such as the word "all"-which, though themselves finite, embody an infinit ude; and by using them, we sidestep the 
apparent problem that there are an infinite number of facts we want to prove. 
We use the word "all" in a few ways which are defined by the thought processes 
of reasoning. That is, there are rules  which our usage of "all" obeys. We may be 
unconscious of them, and tend to claim we operate on the basis of the meaning  of the 
word; but that, after all, is only a circumlo cution for saying that we are guided by rules

word; but that, after all, is only a circumlo cution for saying that we are guided by rules 
which we never make explicit. We have used words all our lives in certain patterns, and instead of calling the patterns "rules", we attr ibute the courses of our thought processes to 
the "meanings" of words. That discovery was a crucial recognition in the long path 
towards the formalization of number theory. 
If we were to delve into Euclid's proof  more and more carefully, we would see 
that it is composed of many, many small-almost infinitesimal steps. If all those steps were 
written out line after line, the proof would app ear incredibly complicated. To our minds it

written out line after line, the proof would app ear incredibly complicated. To our minds it 
is clearest when several steps are telescoped t ogether, to form one single sentence. If we 
tried to look at the proof in slow motion, we would begin to discern individual frames. In 
other words, the dissection can go only so far,  and then we hit the "atomic" nature of 
reasoning processes. A proof can be broken dow n into a series of tiny but discontinuous 
jumps which seem to flow smoothly when perceived from a higher vantage point. In 
Chapter VIII, I will show one way of breaki ng the proof into atomic units, and you will

Chapter VIII, I will show one way of breaki ng the proof into atomic units, and you will 
see how incredibly many steps are involved. Perhaps it should not surprise you, though. The operations in Euclid's brain when he i nvented the proof must have involved millions 
of neurons (nerve cells), many of which fi red several hundred times in a single second. 
The mere utterance of a sentence involves hundr eds of thousands of neurons. If Euclid's 
thoughts were that complicated, it makes sense for his proof to contain a huge number of

thoughts were that complicated, it makes sense for his proof to contain a huge number of 
steps! (There may be little direct connection between the neural actions in his brain, and a proof in our formal system, but the complexitie s of the two are comparable. It is as if 
nature wants the complexity of the proof of th e infinitude of primes to be conserved, even 
when the systems involved are very different from each other.) 
In Chapters to come, we will lay out a formal system that (1) includes a stylized

In Chapters to come, we will lay out a formal system that (1) includes a stylized 
vocabulary in which all statements about natural numbers can be expressed, and (2) has rules corresponding to all the types of reasoning which seem necessary. A very important question will be whether the rules for sy mbol manipulation which we have then 
formulated are really of equal power (as far as number theory is concerned) to our usual 
mental reasoning abilities-or, more generally, whet her it is theoretically possible to attain 
the level of our thinking abilities, by using some formal system.

Sonata for Unaccompanied Achilles 69 
Sonata 
for Unaccompanied Achilles 
 
The telephone rings; Achilles picks it up. 
  Achilles: Hello, this is Achilles. Achilles: Oh, hello, Mr. T. How are you? Achilles: A torticollis? Oh, I'm sorry to hear it. Do you have any idea what caused it? Achilles: How long did you hold it in that position? Achilles: Well, no wonder it's stiff, then. What on earth induced you keep your neck 
twisted that way for so long? 
Achilles: Wondrous many of them, eh? What kinds, for example? Achilles: What do you 
mean, "phantasmagorical beasts"? 
 
FIGURE 14. Mosaic II, by M. C. Escher (lithograph, 1957).

Sonata for Unaccompanied Achilles 70Achilles: Wasn't it terrifying to see so many of them at the same time? Achilles: A 
guitar!? Of all things to be in the mids t of all those weird creatures. Say, don't you 
play the guitar? 
Achilles: Oh, well, it's all the same to me. Achilles: You're right; I wonder why I never noticed that difference between fiddles and 
guitars before. Speaking of fiddling, how  would you like to come over and listen 
to one of the sonatas for unaccompanied violin by your favorite composer, J. S. 
Bach? I just bought a marvelous recording of them. I still can't get over the way 
Bach uses a single violin to create a piece with such interest. 
Achilles: A headache too? That's a shame. Perhaps you should just go to bed. Achilles: I see. Have you tried counting sheep? Achilles: Oh, oh, I see. Yes, I fully know wh at you mean. Well, if it's THAT distracting, 
perhaps you'd better tell it to me, and let me try to work on it, too.

perhaps you'd better tell it to me, and let me try to work on it, too. 
Achilles: A word with the letters ` A', `D', `A', `C'  consecutively inside it ... Hmm ... 
What about "abracadabra"? 
Achilles: True, " ADAC " occurs backwards, not forwards , in that word. Achilles: Hours 
and hours? It sounds like I'm in for a l ong puzzle, then. Where did you hear this 
infernal riddle? 
Achilles: You mean he looked like he was me ditating on esoteric Buddhist matters, but in 
reality he was just trying to think up complex word puzzles? 
Achilles: Aha!-the snail knew what this fe llow was up to. But how did you come to talk 
to the snail?

to the snail? 
Achilles: Say, I once heard a word puzzle a little bit like this one. Do you want to hear it? 
Or would it just drive you further into di straction? Achilles: I agree-can't do any 
harm. Here it is: What's a word that begins with the letters " HE" and also ends 
with " HE"? 
Achilles: Very ingenious-but that's almost cheating. It's certainly not what I meant! 
Achilles: Of course you're right-it fulfills th e conditions, but it's a sort of "degenerate" 
solution. There's another solution which I ha d in mind. Achilles: That's exactly it!

with " HE"? 
solution. There's another solution which I ha d in mind. Achilles: That's exactly it! 
How did you come up with it so fast? Ach illes: So here's a case where having a 
headache actually might have helped you, rather than hindering you. Excellent! 
But I'm still in the dark on your " ADAC " puzzle. 
Achilles: Congratulations! Now maybe you'll be ab le to get to sleep! So tell me, what is 
the solution? 
Achilles: Well, normally I don't like hints, but  all right. What's your hint? Achilles: I 
don't know what you mean by "figure" and "ground" in this case. 
Achilles: Certainly I know Mosaic II! I know ALL of Escher's works. After all, he's my

Achilles: Certainly I know Mosaic II! I know ALL of Escher's works. After all, he's my 
favorite artist. In any case, I've got a print of Mosaic II hanging on my wall, in plain view from here.

Sonata for Unaccompanied Achilles 71 
Achilles:: Yes, t see all the black animals. Achilles: Yes, I also see how their "negativ e space" -- what's left out-- defines the white 
animals. 
Achilles: So THAT'S what you mean by "figur e" and "ground". But what does that have 
to do with the " ADAC " puzzle? 
Achilles: Oh, this is too tricky for me. I think I'M starting to get a headache Achilles: You want to come over now? But I thought-- Achilles: Very well. Perhaps by then I'll have thought of the right answer to YOUR 
puzzle, using your figure-ground hint, relating it to MY puzzle  
Achilles: I'd love to play them for you. Achilles: You've invented a theory about them?  Achilles: Accompanied by what instrument? Achilles: Well, if that's the case, it seems a little strange that he would have written out 
the harpsichord part, then, and had it published a s well. 
Achilles: I see -- sort of an optional feature.  One could listen to them either way -- with

Achilles: I see -- sort of an optional feature.  One could listen to them either way -- with 
or without accompaniment. But how woul d one know what the accompaniment is 
supposed to sound like? 
Achilles: Ah, yes, I guess that it is best, after all, to leave it to the listener’s imagination. 
And perhaps, as you said, Bach never ev en had accompaniment in mind at all. 
Those sonatas seem to work very indeed as they are. 
Achilles: Right. Well, I'll see you shortly.  Achilles: Good-bye, Mr. T.

Figure and Ground 72CHAPTER III 
 
Figure and Ground 
 
Primes vs. Composites 
 
THERE IS A strangeness to the idea that  concepts can be captured by simple 
typographical manipulations. The one concept so  far captured is that of addition, and it 
may not have appeared very strange. But suppos e the goal were to create a formal system 
with theorems of the form Px, the letter `x' standing for a hyphen-string, and where the 
only such theorems would be ones in whic h the hyphen-string contained exactly a prime 
number of hyphens. Thus , P---  would be a theorem, but P----  would not. How could this 
be done typographically? First, it is important to specify clearly what is meant by typographical  operations. The complete repertoire has been presented in the MIU -system 
and the pq-system, so we really only need to make a list of the kinds of things we have 
permitted:

permitted:  
(1)  reading and recognizing any of a finite set of symbols; (2)  writing down any symbol belonging to that set; (3)  copying any of those symbols from one place to another; (4)  erasing any of those symbols; (5)  checking to see whether one symbol is the same as another; (6)  keeping and using a list of previously generated theorems. 
 The list is a little redundant, but no matter. What  is important is that it clearly involves 
only trivial abilities, each of them far less th an the ability to distinguish primes from

The list is a little redundant, but no matter. What  is important is that it clearly involves 
only trivial abilities, each of them far less th an the ability to distinguish primes from 
nonprimes. How, then, could we compound some of these operations to make a formal system in which primes are distinguished from composite numbers?  
The tq-System 
 A first step might be to try to solve a simple r, but related, problem. We could try to make 
a system similar to the pq-system, except th at it represents multiplication, instead of 
addition. Let's call it the tq-system , `t' for times'. More specifically, suppose X, Y, and Z 
are, respectively, the numbers of hyphens in the hyphen-strings x, y, and z. (Notice I am 
taking special pains to distinguish betw een a string and the number of hyphens it 
contains.) Then we wish the string x ty q z to be a theorem if and only if X times Y 
equals Z. For instance, --t---q-----  should be a theorem because 2 times 3 equals 6, but --

t--q--- should not be a theorem. The tq-system can  be characterized just about as easily as 
the pq-system namely, by using just one axiom schema and one rule of inference:

Figure and Ground 73 
AXIOM SCHEMA: x t-qx is an axiom, whenever x is a hyphen string. 
. RULE OF INFERENCE: Suppose that x, y, and z are all hyphen-strings. An suppose that 
x ty qz is an old theorem. Then, xty-qzx is a ne' theorem. 
 Below is the derivation of the theorem --t---q-----  
: 
(1) --t-q--  (axiom) (2) --t—q---- (by rule of inference, using line (1) as the old theorem) (3) --t---q ------- (by rule of inference,  using line (2) as the old theorem) 
 Notice how the middle hyphen-string grow s by one hyphen each time the rule of 
inference is applied; so it is predictable th at if you want a theorem with ten hyphens in 
the middle, you apply the rule of inference nine times in a row.  
Capturing Compositeness 
 Multiplication, a slightly trickier concept than addition, has now bee] "captured" 
typographically, like the birds in Escher's Li beration. What about primeness? Here's a

Notice how the middle hyphen-string grow s by one hyphen each time the rule of 
typographically, like the birds in Escher's Li beration. What about primeness? Here's a 
plan that might seem smart: using the tq-system define a new set of theorems of the form Cx, which characterize compost. numbers, as follows: 
 RULE: Suppose x, y, and z are hyphen-strings. If x-ty-qz is a theorem then C z is a 
theorem.  This works by saying that Z (the number of hyphens in z) is composite a long as it is the product of two numbers greater than 1-namel y, X + (the number of hyphens in x-), and Y 
+ 1 (the number of hyphens in y I am de fending this new rule by giving you some

+ 1 (the number of hyphens in y I am de fending this new rule by giving you some 
"Intelligent mode justifications for it. That is because you are a human being, and want t, 
know why there is such a rule. If you were ope rating exclusively in the "Mechanical 
mode", you would not need any justification, since M-mod. workers just follow the rules mechanically and happily, never questioning; them! 
Because you work in the I-mode, you will tend to blur in your mind the 
distinction between strings and their interpre tations. You see, things Cal become quite

distinction between strings and their interpre tations. You see, things Cal become quite 
confusing as soon as you perceive "meaning"  in the symbol which you are manipulating. 
You have to fight your own self to keep from thinking that the string' ---' is the number  3. 
The Requirement of Formality, which in Ch apter I probably seemed puzzling (because it 
seemed so obvious), here becomes tricky, and crucial. It is the essential thing which keeps you from mixing up the I-mode with the M-mode; or said another way, it keeps you from mixing up arithmetical facts with typographical theorems.

Figure and Ground 74Illegally Characterizing Primes 
 
It is very tempting to jump from the C-type  theorems directly to P-type theorems, by 
proposing a rule of the following kind:  PROPOSED RULE: Suppose x is a hyphen-string. If Cx is not a theorem, then Px is a 
theorem. The fatal flaw here is that checking whether Cx is not a theorem is not an explicitly 
typographical operation. To know for sure that MU is not a theorem of the MIU-system, 
you have to go outside  of the system ... and so it is with this Proposed Rule. It is a rule 
which violates the whole idea of formal system s, in that it asks you to operate informally-
that is, outside the system. Typographical operation (6) allows you to look into the stockpile of previously found theorems, but th is Proposed Rule is asking you to look into 
a hypothetical "Table of Nontheorems". But in order to generate such a table, you would 
have to do some reasoning outside the system -reasoning which shows why various strings

have to do some reasoning outside the system -reasoning which shows why various strings 
cannot be generated inside the system. Now it may well be that there is another formal 
system which can generate the "Table of Nontheorems", by purely typographical means. 
In fact, our aim is to find just such a syst em. But the Proposed Rule is not a typographical 
rule, and must be dropped. 
This is such an important point that we might dwell on it a bit more. In our C-
system (which includes the tq-system and the rule which defines C-type theorems), we 
have theorems of the form Cx, with `x' standing, as usual, for a hyphen-string. There are

have theorems of the form Cx, with `x' standing, as usual, for a hyphen-string. There are 
also nontheorems of the form Cx. (These are what I mean when I refer to "nontheorems", 
although of course tt-Cqq  and other ill-formed messes are also nontheorems.) The 
difference is that theorems have a com posite number of hyphens, nontheorems have a 
prime number of hyphens. Now the theorems all have a common "form", that is, originate 
from a common set of typographical rules. Do all nontheorems also have a common "form", in the same sense? Below is a lis t of C-type theorems, shown without their

from a common set of typographical rules. Do all nontheorems also have a common "form", in the same sense? Below is a lis t of C-type theorems, shown without their 
derivations. The parenthesized numbers fo llowing them simply count the hyphens in 
them. 
 
C---- (4) C-------- (6) C----------------(8) C-----------------(9) C--------------------(10) C -------------------- (12) C------------------------ (14) C ------------------------ (15) C----------------------------(16) C----------------------------(18) 
 
. 
. 
.

Figure and Ground 75I he "holes" in this list are the nontheorems. I o repeat the earlier quest Do the holes also 
have some "form" in common? Would it be r easonable say that merely by virtue of being 
the holes in this list, they share a common form? Yes and no. That they share some  
typographical quality is and able, but whether we want to call it "form" is unclear. The 
reason hesitating is that the holes are only negatively  defined-they are the things that are 
left out of a list which is positively  defined. 
 
Figure and Ground  
This recalls the famous artistic distinction between figure  and ground . When a figure or 
"positive space" (e.g., a human form, or a letter, or a still life is drawn inside a frame, an 
unavoidable consequence is that its comple mentary shape-also called the "ground", or 
"background", or "negative space"-has also been  drawn. In most drawings, however, this 
fig ground relationship plays little role. The artis t is much less interested in ground than

fig ground relationship plays little role. The artis t is much less interested in ground than 
in the figure. But sometimes, an artist will take interest in ground as well. 
There are beautiful alphabets which play  with this figure-ground distinction. A 
message written in such an alphabet is shown below. At fir looks like a collection of somewhat random blobs, but if you step back ways and stare at it for a while, all of a 
sudden, you will see seven letters appear in this .. 
. 
 
FIGURE 15. 
  
For a similar effect, take a look at my drawing Smoke Signal  (Fig. 139). Along these

. 
 
FIGURE 15. 
  
For a similar effect, take a look at my drawing Smoke Signal  (Fig. 139). Along these 
lines, you might consider this puzzle: can you somehow create a drawing containing words in both the figure and the ground? 
Let us now officially distinguish  between two kinds of figures: cursively 
drawable  ones, and recursive  ones (by the way, these are my own terms are not in 
common usage). A cursively drawable  figure is one whose ground is merely an 
accidental by-product of the drawing act. A recursive  figure is one whose ground can be

accidental by-product of the drawing act. A recursive  figure is one whose ground can be 
seen as a figure in its own right. Usually this is quite deliberate on the part of the artist. 
The "re" in "recursive" represents the fact that both foreground and background are 
cursively drawable – the figure is "twi ce-cursive". Each figure-ground boundary in a 
recursive figure is a double-edged sword. M. C. Escher was a master at drawing recursive 
figures-see, for instance, his beautiful recursive drawing of birds (Fig. 16).

Figure and Ground 76
 
FIGURE 16. Tiling of the plane using birds, by M. C. Escher (from a 1942 notebook). 
 
Our distinction is not as rigorous as one in mathematics, for who can definitively 
say that a particular ground is not a figur e? Once pointed out, almost any ground has 
interest of its own. In that se nse, every figure is recursive. But that is not what I intended 
by the term. There is a natural and intuitive notion of recognizable forms. Are both the 
foreground and background recognizable forms? If so, then the drawing is recursive. If 
you look at the grounds of most line drawi ngs, you will find them rather unrecognizable. 
This demonstrates that 
 
There exist recognizable forms whose negative space is not any recognizable form.  
In more "technical" terminology, this becomes:  
There exist cursively drawable figures which are not recursive.  Scott Kim's solution to the above pu zzle, which I call his "FIGURE-FIGURE 
Figure", is shown in Figure 17. If you read both black and white,

Figure and Ground 77
 
FIGURE 17. FIGURE-FIGURE Figure,  by Scott E. Kim (1975).

Figure and Ground 78you will see "FIGURE" everywhere, but "GROUND" nowhere! It is a paragon of 
recursive figures. In this clever draw ing, there are two nonequivalent ways of 
characterizing the black regions:  
(1)  as the negative space  to the white regions; 
(2)  as altered copies  of the white regions (produced by coloring and shifting each 
white region). 
 
(In the special case of the FIGURE-FIGURE Figure, the two characterizations are equivalent-but in most black-and-white pict ures, they would not be.) Now in Chapter 
VIII, when we create our T ypographical Number Theory ( TNT ), it will be our hope that 
the set of all false statements of number th eory can be characterized in two analogous 
ways:  
(1) as the negative space to the set of all TNT -theorems; 
(2) as altered copies of the set of all TNT -theorems (produced by negating each 
TNT -theorem).

ways:  
TNT -theorem). 
 But this hope will be dashed, because:  (1) inside the set of all nontheorems are found some truths (2) outside the set of all negated theorems are found some falsehoods . 
You will see why and how this happens, in Chapter XIV. Meanwhile, ponder over a 
pictorial representation of the situation (Fig. 18).  
Figure and Ground in Music 
 
One may also look for figures and grounds in  music. One analogue is the distinction 
between melody and accompaniment-for the mel ody is always in the forefront of our

But this hope will be dashed, because:  (1) inside the set of all nontheorems are found some truths (2) outside the set of all negated theorems are found some falsehoods . 
 
between melody and accompaniment-for the mel ody is always in the forefront of our 
attention, and the accompaniment is subsidiary, in some sense. Therefore it is surprising when we find, in the lower lines of a piece of music, recognizable melodies. This does not happen too often in post-baroque music. Usually the harmonies are not thought of as 
foreground. But in baroque music-in Bach above  all-the distinct lines, whether high or 
low or in between, all act as "figures". In  this sense, pieces by Bach can be called 
"recursive".

"recursive". 
Another figure-ground distinction exists in  music: that between on-beat and off-
beat. If you count notes in a measure " one-and, two-and, three-and, four-and", most 
melody-notes will come on numbers, not on "and"'s. But sometimes, a melody will be deliberately pushed onto the "and" 's, for the sh eer effect of it. This occurs in several 
etudes for the piano by Chopin, for instance. It also occurs in Bach-particularly in his 
Sonatas and Partitas for unaccompanied vio lin, and his Suites for unaccompanied cello.

Sonatas and Partitas for unaccompanied vio lin, and his Suites for unaccompanied cello. 
There, Bach manages to get two or more musical lines going simultaneously. Sometimes 
he does this by having the solo instrument play "double stops"-two notes at once. Other 
times, however, he

Figure and Ground 79
 
FIGURE 18. Considerable visual symbolism is featured in this diagram of the relation between various classes of TNT  strings. The biggest box represents the set of all TNT  
strings The next-biggest box represents the se t of all well-formed TNT strings. Within it is 
found~ set of all sentences of TNT . Now things begin to get interesting. The set of 
theorems pictured as a tree growing out of a trunk (representing the set of axioms). The 
tree-symbol chosen because of the recu rsive growth pattern which it exhibits: new 
branches (theorems constantly sprouting fr om old ones. The fingerlike branches probe 
into the corners of constraining region (the se t of truths), yet can never fully occupy it. 
The boundary beta the set of truths and the set of falsities is meant to suggest a randomly 
meandering coastline which, no matter how clos ely you examine it, always has finer 
levels of structure, an consequently impossible to describe exactly in any finite way. (See

levels of structure, an consequently impossible to describe exactly in any finite way. (See 
B. Mandelbrot's book Fractals.) The reflected tree represents the set of negations of theorems: all of them false yet unable collecti vely to span the space of false statements. 
[Drawing by the author.] 
 
puts one voice on the on-beats, and the other voice  on the off-beats, so ear separates them 
and hears two distinct melodies weaving in and out, - harmonizing with each other. Needless to say, Bach didn't stop at this level of complexity...  
Recursively Enumerable Sets vs. Recursive Sets

Recursively Enumerable Sets vs. Recursive Sets 
 Now let us carry back the notions of figure and ground to the domain formal 
systems. In our example, the ro le of positive space is played by C-type theorems, and the 
role of negative space is played by strings with a

Figure and Ground 80prime number of hyphens. So far, the only way we have found to represent prime 
numbers typographically is as a negative space. Is there, however, some way-I don't care how complicated-of representing the primes as a positive  space-that is, as a set of 
theorems of some formal system? 
Different people's intuitions give different  answers here. I remember quite vividly 
how puzzled and intrigued I was upon rea lizing the difference between a positive 
characterization and a negative characterizati on. I was quite convinced that not only the 
primes, but any set of numbers which could be represented negatively, could also be 
represented positively. The intuition underlying my belief is represented by the question: "How could a figure and its ground not carry exactly the same information ?" They 
seemed to me to embody the same informati on, just coded in two complementary ways. 
What seems right to you? 
It turns out I was right about the primes , but wrong in general. This astonished

It turns out I was right about the primes , but wrong in general. This astonished 
me, and continues to astonish me even today. It is a fact that: 
 
There exist formal systems whose nega tive space (set of nontheorems) is not 
the positive space (set of theorems) of any formal system. 
 This result, it turns out, is of depth equal to Gödel’s Theorem-so it is not 
surprising that my intuition was upset. I, ju st like the mathematicians of the early 
twentieth century, expected the world of form al systems and natural numbers to be more 
predictable than it is. In more technical terminology, this becomes:

predictable than it is. In more technical terminology, this becomes: 
 There exist recursively enumerable sets which are not recursive.  
The phrase recursively enumerable  (often abbreviated "r.e.") is the mathematical 
counterpart to our artistic notion of "cursively drawable"-and recursive  is the counterpart 
of "recursive". For a set of stri ngs to be "r.e." means that it can be generated according to 
typographical rules-for example, the set of C-type theorems, the set of theorems of the 
MIU -system-indeed, the set of theorems of any formal system. This could be compared

MIU -system-indeed, the set of theorems of any formal system. This could be compared 
with the conception of a "figure" as "a set of lines which can be generated according to 
artistic rules" (whatever that might mean!). And a "recursive set" is like a figure whose 
ground is also a figure-not only is it r.e., but its complement is also r.e. 
It follows from the above result that:  
There exist formal systems for whic h there is no typographical decision 
procedure. 
 How does this follow? Very simply. A typographical decision procedure is a 
method which tells theorems from nontheorems. Th e existence of such a test allows us to

method which tells theorems from nontheorems. Th e existence of such a test allows us to 
generate all nontheorems systematically, simply by going down a list of all strings and 
performing the test on them one at a time, discarding ill-formed strings and theorems 
along the way. This amounts to

Figure and Ground 81a typographical method for generating the se t of nontheorems. But according to the 
earlier statement (which we here accept on faith), for some  systems this is not possible. 
So we must conclude that typographical deci sion procedures do not exist for all formal 
systems. 
Suppose we found a set F of natural numbers (` F' for `Figure') whi4 we could 
generate in some formal way-like the composite numbers. Suppose its complement is the set G (for 'Ground')-like the primes. Together  F and G make up all the natural numbers, 
and we know a rule for making all the numbers  in set F, but we know no such rule for 
making all tl numbers in set G. It is important to understand that if the members of were always generated in order of increasing  size, then we could always characterize G. The 
problem is that many r.e. sets are generate d I methods which throw in elements in an

problem is that many r.e. sets are generate d I methods which throw in elements in an 
arbitrary order, so you never know if a number which has been skipped over for a long time will get included you just wait a little longer. 
We answered no to the artistic question, "Are all figures recursive We have now 
seen that we must likewise answer no to th e analogous question in mathematics: "Are all 
sets recursive?" With this perspective, 1 us  now come back to the elusive word "form". 
Let us take our figure-set and our ground-set G again. We can agree that all the numbers 
in set have some common "form"-but can the sa me be said about numbers in s G? It is a 
strange question. When we are dealing with an infinite set to sta with-the natural

strange question. When we are dealing with an infinite set to sta with-the natural 
numbers-the holes created by removing some s ubs may be very hard to define in any 
explicit way. And so it may be that th< ar e not connected by any common attribute or 
"form". In the last analysis it is a matter of taste whether you want to use the word 
"form"-but just thinking about it is provocative. Perhaps it is best not to define "form", bi 
to leave it with some intuitive fluidity. 
Here is a puzzle to think about in connection with the above matter Can you 
characterize the following set of integers (or its negative space) 
 
1 3 7 12 18 26 35 45 56 69...

1 3 7 12 18 26 35 45 56 69...   
How is this sequence like the FIGURE-FIGURE Figure? 
  
Primes as Figure Rather than Ground 
 
Finally, what about a formal system for gene rating primes? How is it don< The trick is to 
skip right over multiplication, and to go directly to nondivisibility  as the thing to 
represent positively. Here are an axiom sche ma and rule for producing theorems which 
represent the notion that one number does not divide ( D N D ) another number exactly: 
 AXIOM SCHEMA: xy D N D x where x and y are hyphen-strings.

 AXIOM SCHEMA: xy D N D x where x and y are hyphen-strings. 
 For example - ---D N D--,  where x has been replaced by'--'and y  by ‘---“.

Figure and Ground 82RULE: If x D N D y is a theorem, then so is x D N D x y. 
 If you use the rule twice, you can generate this theorem:  
   -----D N D -------------- 
 
which is interpreted as "5 does not divide 12". But ---D N D------------  is not a theorem. 
What goes wrong if you try to produce it? 
Now in order to determine that a given number is prime, we have to build up 
some knowledge about its nondivisibility propertie s. In particular, we want to know that 
it is not divisible by 2 or 3 or 4, etc., all th e way up to 1 less than the number itself. But 
we can't be so vague in formal systems as to say "et cetera". We must spell things out. 
We would like to have a way of saying, in th e language of the system, "the number Z is 
divisor free  up to X", meaning that no number betw een 2 and X divides Z. This can be 
done, but there is a trick to it. Think about it if you want. 
Here is the solution:  
RULE: If --D N D z is a theorem, so is z D F--.

RULE: If --D N D z is a theorem, so is z D F--. 
 RULE: If z D Fx is a theorem and also x- D N D z is a theorem, z D Fx- is a theorem. 
 
These two rules capture the notion of divisor fr eeness. All we need to do is to say that 
primes are numbers which are divisor-free up to 1 less than themselves:  RULE: If z- DFz is a theorem, then Pz- is a theorem.  
 
Oh-let's not forget that 2 is prime!  
 Axiom: P--. 
 And there you have it. The principle of represen ting primality formally is that there is a

 And there you have it. The principle of represen ting primality formally is that there is a 
test for divisibility which can be done without any backtracking. You march steadily upward, testing first for divisibility by 2, then by 3, and so on. It is this "monotonicity" or 
unidirectionality-this absence of cross- play between lengthening and shortening, 
increasing and decreasing-that allows prima lity to be captured. And it is this potential

increasing and decreasing-that allows prima lity to be captured. And it is this potential 
complexity of formal systems to involve arbitrary amounts of backwards-forwards interference that is responsible for such lim itative results as Gödel’s Theorem, Turing's 
Halting Problem, and the fact that not all recursively enumerable sets are recursive.

Contracrostipunctus 83Contracrostipunctus 
 
Achilles has come to visit his friend and jogging companion, the 
Tortoise, at his home 
 
Achilles: Heavens, you certainly have an admirable boomerang collection 
 Tortoise: Oh, pshaw. No better than that of any other Tortoise . And now would you like 
to step into the parlor? 
Achilles: Fine. ( Walks to the corner of the room .) I see you also have a large collection of 
records. What sort of music do you enjoy?  
Tortoise: Sebastian Bach isn't so bad, in my opinion. But these days, I must say, I am 
developing more and more of an interest in a rather specialized sort of music. 
Achilles: Tell me, what kind of music is that? Tortoise: A type of music which you are most unlikely to have heard of. call it "music to 
break phonographs by". 
Achilles: Did you say "to break phonographs by"? That is a curious concept. I can just 
see you, sledgehammer in hand, whacking on phonograph after another to pieces,

see you, sledgehammer in hand, whacking on phonograph after another to pieces, 
to the strains of Beethoven's heroic masterpiece Wellington's Victory . 
Tortoise: That's not quite what this musi c is about. However, you might find its true 
nature just as intriguing. Perhaps I should give you a brief description of it? 
Achilles: Exactly what I was thinking. Tortoise: Relatively few people are acquainted with it. It all began whet my friend the 
Crab-have you met him, by the way?-paid m• a visit. 
Achilles: ' twould be a pleasure to make his acquaintance, I'm sure Though I've heard so

Achilles: ' twould be a pleasure to make his acquaintance, I'm sure Though I've heard so 
much about him, I've never met him 
Tortoise: Sooner or later I'll get the two of you together. You'd hit it of splendidly. 
Perhaps we could meet at random in the park on day ... 
Achilles: Capital suggestion! I'll be looking forward to it. But you were going to tell me 
about your weird "music to smash phone graphs by", weren't you? 
Tortoise: Oh, yes. Well, you see, the Cr ab came over to visit one day. You must 
understand that he's always had a weakne ss for fang gadgets, and at that time he

understand that he's always had a weakne ss for fang gadgets, and at that time he 
was quite an aficionado for, of al things , record players. He had just bought his 
first record player, and being somewh at gullible, believed every word the 
salesman had told him about it-in particular, that it was capable of reproducing any and all sounds. In short, he was convinced that it was a Perfect phonograph.

Contracrostipunctus 84Achilles: Naturally, I suppose you disagreed. 
Tortoise: True, but he would h ear nothing of my arguments. He staunchly maintained that 
any sound whatever was reproducible on hi s machine. Since I couldn't convince 
him of the contrary, I left it at that. But not long after that, I returned the visit, taking with me a record of a song whic h I had myself composed. The song was 
called "I Cannot Be Played on Record Player 1". 
Achilles: Rather unusual. Was it a present for the Crab? Tortoise: Absolutely. I suggested that we  listen to it on his new phonograph, and he was 
very glad to oblige me. So he put it on. But unfortunately, after only a few notes, the record player began vibrating rather  severely, and then with a loud "pop", 
broke into a large number of fairly sma ll pieces, scattered all about the room. The 
record was utterly destroyed also, needless to say. 
Achilles: Calamitous blow for the poor fello w, I'd say. What was the matter with his 
record player?

record player? 
Tortoise: Really, there was nothing the matter, nothing at all. It simply couldn't reproduce 
the sounds on the record which I had brought  him, because they were sounds that 
would make it vibrate and break. 
Achilles: Odd, isn't it? I mean, I thought it was a Perfect phonograph. That's what the 
salesman had told him, after all. 
Tortoise: Surely, Achilles, you don't believe everything that salesmen tell you! Are you 
as naive as the Crab was? 
Achilles: The Crab was naiver by far! I know th at salesmen are notorious prevaricators. I 
wasn't born yesterday!

wasn't born yesterday! 
Tortoise: In that case, maybe you can imagine th at this particular salesman had somewhat 
exaggerated the quality of the Crab's piece of equipment ... perhaps it was indeed less than Perfect, and could not reproduce every possible sound. 
Achilles: Perhaps that is an explanation. But there's no explanation for the amazing 
coincidence that your record had those very sounds on it ... 
Tortoise: Unless they got put there deliberat ely. You see, before returning the Crab's

Tortoise: Unless they got put there deliberat ely. You see, before returning the Crab's 
visit, I went to the store where the Cr ab had bought his machine, and inquired as 
to the make. Having ascertained that, I sent off to the manufacturers for a 
description of its design. After receiving th at by return mail, I analyzed the entire 
construction of the phonograph and discovere d a certain set of sounds which, if 
they were produced anywhere in the vici nity, would set the device to shaking and 
eventually to falling apart.

eventually to falling apart. 
Achilles: Nasty fellow! You needn't spell out for me the last details: that you recorded 
those sounds yourself, and offered the dastardly item as a gift ...

Contracrostipunctus 85Tortoise: Clever devil! You jumped ahead of  the story! But that wasn't t end of the 
adventure, by any means, for the Crab did r believe that his record player was at 
fault. He was quite stubborn. So he went out and bought a new record player, this 
o even more expensive, and this time th e salesman promised give him double his 
money back in case the Crab found a soul which it could not reproduce exactly. So the Crab told r excitedly about his new model, and I promised to come over and see it. 
Achilles: Tell me if I'm wrong-I bet that  before you did so, you on again wrote the 
manufacturer, and composed and recorded  new song called "I Cannot Be Played 
on Record Player based on the construction of the new model. 
Tortoise: Utterly brilliant deduction, Achilles. You've quite got the spirit. Achilles: So what happened this time? Tortoise: As you might expect, precisely  the same thing. The phonograph fell into

Tortoise: Utterly brilliant deduction, Achilles. You've quite got the spirit. Achilles: So what happened this time? Tortoise: As you might expect, precisely  the same thing. The phonograph fell into 
innumerable pieces, and the record was shattered. Achilles: Consequently, the 
Crab finally became convinced that there could be no such thing as a Perfect record player. 
Tortoise: Rather surprisingly, that's not quite  what happened. He was sure that the next 
model up would fill the bill, and having twice the money, h e-- 
Achilles: Oho-I have an idea! He could have easily outwitted you, I obtaining a LOW-
fidelity phonograph-one that was not capab le of reproducing the sounds which 
would destroy it. In that way, he would avoid your trick.

would destroy it. In that way, he would avoid your trick. 
Tortoise: Surely, but that would defeat the-original purpose-namely, to have a 
phonograph which could reproduce any sound whatsoever, even its own self-
breaking sound, which is of coup impossible. 
Achilles: That's true. I see the dilemma now. If any record player-si Record Player X-is sufficiently high-fidelity, then when attempts to play the song "I 
Cannot Be Played on Record Player X", it will create just those vibrations which

Cannot Be Played on Record Player X", it will create just those vibrations which 
will cause to break. .. So it fails to be Perfect. And yet, the only way to g, around that trickery, namely for Record Player X to be c lower fidelity, even more directly ensures that it is not Perfect It seems that every record player is 
vulnerable to one or the other of these frailties, and hence all record players are defective. 
Tortoise: I don't see why you call them "defective" . It is simply an inherent fact about 
record players that they can't do all that you might wish them to be able to do. But

record players that they can't do all that you might wish them to be able to do. But 
if there is a defect anywhe re, is not in THEM, but in your expectations of what 
they should b able to do! And the Crab was just full of such unrealistic expectations.

Contracrostipunctus 86Achilles: Compassion for the Crab overwhelms me. High fidelity or low fidelity, he loses 
either way. 
Tortoise: And so, our little game went on like *_his for a few more rounds, and 
eventually our friend tried to become ve ry smart. He got wind of the principle 
upon which I was basing my own records, and decided to try to outfox me. He 
wrote to the phonograph makers, and desc ribed a device of his own invention, 
which they built to specification. He ca lled it "Record Player Omega". It was 
considerably more sophisticated than an ordinary record player. 
Achilles: Let me guess how: Did it have no of cotton? Or Tortoise: Let me tell you, instead. That will sa ve some time. In the first place, Record 
Player Omega incorporated a television camera whose purpose it was to scan any record before playing it. This camera was hooked up to a small built-in computer, 
which would determine exactly the nature  of the sounds, by looking at the groove-
patterns.

patterns. 
Achilles: Yes, so far so good. But what c ould Record Player Omega do with this 
information? 
Tortoise: By elaborate calculations, its little co mputer figured out what effects the sounds 
would have upon its phonograph. If it deduced th at the sounds were such that they 
would cause the machine in its presen t configuration to break, then it did 
something very clever. Old Omega cont ained a device which could disassemble 
large parts of its phonograph subunit, and re build them in new ways, so that it

large parts of its phonograph subunit, and re build them in new ways, so that it 
could, in effect, change its own structure.  If the sounds were "dangerous", a new 
configuration was chosen, one to which the sounds would pose no threat, and this 
new configuration would then be built by the rebuilding subunit, under direction 
of the little computer. Only after this rebuilding operation would Record Player 
Omega attempt to play the record. 
Achilles: Aha! That must have spelled th e end of your tricks. I bet you were a little 
disappointed.

disappointed. 
Tortoise: Curious that you should think so ... I don't suppose that you know Godel's 
Incompleteness Theorem backwards and forwards, do you? 
Achilles: Know WHOSE Theorem backwards and forwards? I've heard of anything that sounds like that. I'm sure  it's fascinating, but I'd rather hear more 
about "music to break records by". It's an amusing little story. Actually, I guess I 
can fill in the end. Obviously, ther e was no point in going on, and so you 
sheepishly admitted defeat, and that was that. Isn't that exactly it?

sheepishly admitted defeat, and that was that. Isn't that exactly it? 
Tortoise: What! It's almost midnight! I'm afraid  it's my bedtime. I'd love to talk some 
more, but really I am growing quite sleepy.

Contracrostipunctus 87 
Achilles: As am 1. Well, 1 u be on my way. ( As he reaches the door, he suddenly stops, 
and turns around .) Oh, how silly of me! I almost forgo brought you a little 
present. Here. (Hands the Tortoise a small neatly wrapped package .) 
Tortoise: Really, you shouldn't have! Why, tha nk you very much indeed think I'll open it 
now. ( Eagerly tears open the package, and ins discovers a glass goblet .) Oh, what 
an exquisite goblet! Did y know that I am qu ite an aficionado for, of all things, gl 
goblets? 
Achilles: Didn't have the foggiest. What an agreeable coincidence! Tortoise: Say, if you can keep a secret, I'll let you in on something: I trying to find a 
Perfect goblet: one having no defects of a sort in its shape. Wouldn't it be 
something if this goblet-h call it "G"-we re the one? Tell me, where did you come 
across Goblet G?

goblets? 
across Goblet G? 
Achilles: Sorry, but that's MY little secret. But you might like to know w its maker is. Tortoise: Pray tell, who is it? Achilles: Ever hear of the famous glassblo wer Johann Sebastian Bach? Well, he wasn't 
exactly famous for glassblowing-but he dabbled at the art as a hobby, though 
hardly a soul knows it-a: this goblet is the last piece he blew. 
Tortoise: Literally his last one? My gracious. If it truly was made by Bach its value is 
inestimable. But how are you sure of its maker  
Achilles: Look at the inscription on the inside -do you see where tletters `B', `A', `C', `H'

Achilles: Look at the inscription on the inside -do you see where tletters `B', `A', `C', `H' 
have been etched? 
Tortoise: Sure enough! What an extraordinary thing. (Ge ntly sets Goblet G down on a 
shelf. ) By the way, did you know that each of the four letters in\Bach's name is 
the name of a musical note?  
Achilles:' tisn't possible, is it? After all, musical notes only go from ‘A’ through `G'. 
Tortoise: Just so; in most countries, that 's the case. But in Germany, Bach’s own 
homeland, the convention has always been si milar, except that what we call `B', 
they call `H', and what we call `B-flat', they call `B'. For instance, we talk about 
Bach's "Mass in B Minor whereas they ta lk about his "H-moll Messe". Is that

Bach's "Mass in B Minor whereas they ta lk about his "H-moll Messe". Is that 
clear? 
Achilles: ... hmm ... I guess so. It's a little confusing: H is B, and B B-flat. I suppose 
his name actually constitutes a melody, then 
Tortoise: Strange but true. In fact, he worked  that melody subtly into or of his most 
elaborate musical pieces-namely, the final Contrapunctus  in his Art of the Fugue . 
It was the last fugue Bach ever wrote. When I heard it for the first time, I had no idea how would end. Suddenly, without wa rning, it broke off. And the ... dead 
silence. I realized immediately that was where Bach died. It is an indescribably

silence. I realized immediately that was where Bach died. It is an indescribably 
sad moment, and the effect it had o me was-shattering. In any case, B-A-C-H is the last theme c that fugue. It is hidden inside the piece. Bach didn't point it out

Contracrostipunctus 88
 
FIGURE 19. The last page of Bach's Art of the Fugue. In the original manuscript, in the 
handwriting of Bach's son Carl Philipp Emanuel, is written: "N.B. In the course of this fugue, at the point where the name B.A.C.H.  was brought in as countersubject, the 
composer died." (B-A-C-H in box.) I have let th is final page of Bach's last fugue serve as 
an epitaph. [Music Printed by Donald Byrd's program  "SMUT", developed at Indiana University]

Contracrostipunctus 89Explicitly, but if you know about it, you can fi nd it without much trouble. Ah, me-there 
are so many clever ways of hiding things in music .. . 
Achilles: . . or in poems. Poets used to do very similar things, you know (though it's 
rather out of style these days). For instance, Lewis Carroll often hid words and names in the first letters (or characters) of  the successive lines in poems he wrote. 
Poems which conceal messages that way are called "acrostics". 
Tortoise: Bach, too, occasionally wrote acrostics, which isn't surprising. After all, 
counterpoint and acrostics, with their levels  of hidden meaning, have quite a bit in 
common. Most acrostics, however, have only one hidden level-but there is no reason that one couldn't make a double-deck er-an acrostic on top of an acrostic. 
Or one could make a "contracrostic"-where the initial letters, taken in reverse order, form a message. Heavens! There's no end to the possibilities inherent in the

Or one could make a "contracrostic"-where the initial letters, taken in reverse order, form a message. Heavens! There's no end to the possibilities inherent in the 
form. Moreover, it's not limited to poets; anyone could write acrostics-even a dialogician. 
Achilles: A dial-a-logician? That's a new one on me. Tortoise: Correction: I said "dialogician",  by which I meant a writer of dialogues. Hmm 
... something just occurred to me. In the unlikely event that a dialogician should write a contrapuntal acrostic in homage to  J. S. Bach, do you suppose it would be 
more proper for him to acrostically embe d his OWN name-or that of Bach? Oh,

more proper for him to acrostically embe d his OWN name-or that of Bach? Oh, 
well, why worry about such frivolous matters? Anybody who wanted to write such a piece could make up his own mind. Now getting back to Bach's melodic 
name, did you know that the melody B- A-C-H, if played upside down and 
backwards, is exactly the same as the original? 
Achilles: How can anything be played upside down? Backwards, I can see-you get H-C-
A-B-but upside down? You must be pulling my leg. 
Tortoise: ' pon my word, you're quite a skeptic, aren't you? Well, I guess I'll have to give

Tortoise: ' pon my word, you're quite a skeptic, aren't you? Well, I guess I'll have to give 
you a demonstration. Let me just go and fe tch my fiddle- (Walks into the next 
room, and returns in a jiffy with an an cient-looking violin.) -and play it for you 
forwards and backwards and every which way. Let's see, now ... (Places his copy 
of the Art of the Fugue on his music stand and opens it to the last page.) ... here's the last Contrapunctus, and here's the last theme ... 
 
The Tortoise begins to play: B-A-C- - but as he bows the final H, suddenly,

The Tortoise begins to play: B-A-C- - but as he bows the final H, suddenly, 
without warning, a shattering sound rudely interrupts his performance. Both he and Achilles spin around, just in time to catch a glimpse of myriad 
fragments of glass tinkling to the floor from the shelf where Goblet G had stood, only moments before. And then ... dead silence.

Consistency, Completeness, and Geometry 90Chapter IV 
 
Consistency, Completeness, 
and Geometry 
 
Implicit and Explicit Meaning 
 
IN CHAPTER II, we saw how meaning-at least in the relatively simple context of formal 
systems-arises when there is an isomorphism  between rule-governed symbols, and things 
in the real world. The more complex the isom orphism, in general, the more "equipment"-
both hardware and software-is required to extr act the meaning from the symbols. If an 
isomorphism is very simple (or very familia r), we are tempted to say that the meaning 
which it allows us to see is explicit. We see the meaning without seeing the isomorphism. 
The most blatant example is human language , where people often attribute meaning to 
words in themselves, without being in th e slightest aware of the very complex 
"isomorphism" that imbues them with meanings. This is an easy enough error to make. It attributes all the meaning to the object (the word), rather than to the link between that

"isomorphism" that imbues them with meanings. This is an easy enough error to make. It attributes all the meaning to the object (the word), rather than to the link between that 
object and the real world. You might compar e it to the naive belief that noise is a 
necessary side effect of any collision of two objects. This is a false belief; if two objects 
collide in a vacuum, there will be no noise at all. Here again, the error stems from attributing the noise exclusively to the co llision, and not recognizing the role of the 
medium, which carries it from the objects to the ear. 
Above, I used the word "isomorphism" in quotes to indicate that it must be taken 
with a grain of salt. The symbolic proce sses which underlie the understanding of human

with a grain of salt. The symbolic proce sses which underlie the understanding of human 
language are so much more complex than the symbolic processes in typical formal 
systems, that, if we want to continue thi nking of meaning as mediated by isomorphisms, 
we shall have to adopt a far more flexible conception of what isomorphisms can be than 
we have up till now. In my opinion, in fact, the key element in answering the question 
"What is consciousness?" will be the unrave ling of the nature of the "isomorphism" 
which underlies meaning.  
Explicit Meaning of the Contracrostipunctus

Explicit Meaning of the Contracrostipunctus  
 
All this is by way of prepar ation for a discussion of the Contracrostipunctus -a study in 
levels of meaning. The Dialogue has both explicit and implicit meanings. Its most 
explicit meaning is simply the story

Consistency, Completeness, and Geometry 91Which was related. This “explicit meaning is, strictly speaking extremely implicit , in the 
sense that the brain processes required to understand the events in the story, given only 
the black marks on paper, are incredibly co mplex. Nevertheless, we shall consider the 
events in the story to be the explicit mean ing of the Dialogue, and assume that every 
reader of English uses more or less the sa me "isomorphism" in sucking that meaning 
from the marks on the paper.  
Even so, I'd like to be a little more explicit about the explicit meaning of the story. 
First I'll talk about the record players and the records. The main point is that there are two 
levels of meaning for the grooves in the records.  Level One is that of music. Now what is 
"music"-a sequence of vibrations in the air,  or a succession of emotional responses in a 
brain? It is both. But before there can be emo tional responses, there have to be vibrations.

brain? It is both. But before there can be emo tional responses, there have to be vibrations. 
Now the vibrations get "pulled" out of the grooves by a record player, a relatively 
straightforward device; in fact you can do it with a pin, just pulling it down the grooves. 
After this stage, the ear converts the vibrations into firings of auditory neurons in the brain. Then ensue a number of stages in th e brain, which gradually transform the linear

After this stage, the ear converts the vibrations into firings of auditory neurons in the brain. Then ensue a number of stages in th e brain, which gradually transform the linear 
sequence of vibrations into a complex pattern of interacting emotional responses-far too complex for us to go into here, much though I would like to. Let us therefore content 
ourselves with thinking of the sounds in th e air as the "Level One" meaning of the 
grooves. What is the Level Two meaning of the grooves?  It is the sequence of vibrations induced 
in the record player. This meaning can only arise after the Level One meaning has been pulled out of the grooves, since the vibrations  in the air cause the vibrations in the

in the record player. This meaning can only arise after the Level One meaning has been pulled out of the grooves, since the vibrations  in the air cause the vibrations in the 
phonograph. Therefore, the Level Two m eaning depends upon a chain of two 
isomorphisms:  
(1) Isomorphism between arbitrary groove patterns and air 
vibrations; 
(2) Isomorphism between graph vibrations. arbitrary air 
vibrations and phonograph vibrations 
 This chain of two isomorphisms is depicted in Figure 20. Notice that isomorphism I is the 
one which gives rise to the Level One meaning. The Level Two meaning is more implicit than the Level One meaning, because it is medi ated by the chain of two isomorphisms. It 
is the Level Two meaning which "backfires", causing the record player to break apart.

is the Level Two meaning which "backfires", causing the record player to break apart. 
What is of interest is that the production of the Level One meaning forces the production of the Level Two meaning simultaneously-the re is no way to have Level One without 
Level Two. So it was the implicit meaning of the record which turned back on it, and destroyed it. 
Similar comments apply to the goblet. On e difference is that the mapping from 
letters of the alphabet to musical notes is one  more level of isomorphism, which we could

letters of the alphabet to musical notes is one  more level of isomorphism, which we could 
call "transcription". That is followed by "translation"-conversion of musical notes into musical sounds. Thereafter, the vibrations act back on the goblet just as they did on the 
escalating series of phonographs.

Consistency, Completeness, and Geometry 92
 
FIGURE 20. Visual rendition of the principl e underlying Gödel’s Theorem: two back-to-
back mappings which have an unexpected boomeranging effect. The first is from groove patterns to sounds, carried out by a phonograph. The second-familiar, but usually ignored -- is from sounds to vibrations of the phonograph. Note that the second mapping 
exists independently of the first one, for any sound in the vicinity, not just ones produced 
by the phonograph itself, will cause such vibrations. The paraphrase of Gödel’s Theorem says that for any record player, there are records which it cannot play because they will 
cause its indirect self-destruction. [Drawing by the author. 
 
Implicit Meanings of the Contracrostipunctus 
 
What about implicit meanings of the Dialogue? (Yes, it has more than one of these.) The 
simplest of these has already been pointed out in the paragraphs above-namely, that the

simplest of these has already been pointed out in the paragraphs above-namely, that the 
events in the two halves of the dialogue are roughly isomorphic to each other: the 
phonograph becomes a violin, the Tortoise b ecomes Achilles, the Crab becomes the 
Tortoise, the grooves become the etched autograph, etc. Once you notice this simple isomorphism, you can go a little further. Observe that in the first half of the story, the 
Tortoise is the perpetrator of all the mischief , while in the second half, he is the victim. 
What do you know, but his own method has turned around and backfired on him!

What do you know, but his own method has turned around and backfired on him! 
Reminiscent of the backfiring of the records' muusic-or the goblet's inscription-or perhaps 
of the Tortoise's boomerang collection? Yes, indeed. The story is about backfiring on two 
levels, as follows ...  
Level One: Goblets and records which backfire; Level Two: The Tortoise's devilish method of exploiting implicit meaning to 
cause backfires-which backfires. 
 Therefore we can even make an isomorphi sm between the two levels of the story, 
in which we equate the way in which the records and goblet boomerang back to destroy

in which we equate the way in which the records and goblet boomerang back to destroy 
themselves, with the way in which the Tort oise's own fiendish method boomerangs back 
to get him in the end. Seen this

Consistency, Completeness, and Geometry 93way, the story itself is an example of the back firings which it discusses. So we can think 
of the Contracrostipunctus as referring to its elf indirectly that its own structure is 
isomorphic to the events it portrays. (Exac tly goblet and records refer implicitly to 
themselves via the back-to-back morphisms of playing and vibration-causing.) One may 
read the Dialogue without perceiving this fact, of course-but it is there all the time. 
 
Mapping Between the Contracrostipunctus 
and Gödel’s Theorem 
 
Now you may feel a little dizzy-but the best is yet to come. (Actually, levels of 
implicit meaning will not even be discussed here-they will 1 for you to ferret out.) The 
deepest reason for writing this Dialogue illustra te Gödel’s Theorem, which, as I said in 
the Introduction, heavily on two different levels  of meaning of statements of number t1 
Each of the two halves  of the Contracrostipunctus is an "isomorphic co Gödel’s Theorem.

Each of the two halves  of the Contracrostipunctus is an "isomorphic co Gödel’s Theorem. 
Because this mapping is the central idea of the Dialogue and is rather elaborate, I have carefully charted it out below. 
 
Phonograph <= =>axiomatic system for number theory 
 
low-fidelity phonograph <= =>"weak" axiomatic system 
 
high-fidelity phonograph <= =>"strong" axiomatic system 
 
"Perfect" phonograph" <= => complete system for number theory' 
 
Blueprint" of phonograph <= => axioms and rules of formal system 
 
record <= => string of the formal system

record <= => string of the formal system 
 
playable record<= => theorem of the axiomatic system 
 
unplayable record <= =>nontheorem of the axiomatic system 
 
sound <= =>true statement of number theory 
 
reproducible sound  <= =>  'interpreted theorem of the system 
 
unreproducible sound <= => true statement which isn't a theorem: 
 
song title <= =>implicit meaning of Gödel’s string: 
 
"I Cannot Be Played  "I Cannot Be Derived 
on Record Player X"  in Formal System X" 
 
This is not the full extent of the isomorphism between Gödel’s theorem and the

This is not the full extent of the isomorphism between Gödel’s theorem and the 
Contracrostipunctus, but it is the core of it. You need not if you don't fully grasp Gödel’s                              Theorem by now-there are still Chapters to go before we reach it! Nevertheless, having 
read this Dialogue you have already tasted some of the flavor of Gödel’s Theorem 
without necessarily being aware of it. I now  leave you to look for any other types of 
implicit meaning in the Contracrostipunctus. "Quaerendo invenietis!"

Consistency, Completeness, and Geometry 94The Art of the Fugue 
 
A few words on the Art of the Fugue ... Composed in the last year of Bach's life, it is a 
collection of eighteen fugues all based on one  theme. Apparently, writing the Musical 
Offering was an inspiration to Bach. He d ecided to compose another set of fugues on a 
much simpler theme, to demonstrate the full ra nge of possibilities inherent in the form. In 
the Art of the Fugue , Bach uses a very simple theme in the most complex possible ways. 
The whole work is in a single key. Most of the fugues have four voices, and they 
gradually increase in complexity and depth of  expression. Toward the end, they soar to 
such heights of intricacy that one suspects he  can no longer maintain them. Yet he does . . 
. until the last Contrapunctus . 
The circumstances which caused the break-off of the Art of the Fugue (which is to 
say, of Bach's life) are these: his eyesight having troubled him for years, Bach wished to

say, of Bach's life) are these: his eyesight having troubled him for years, Bach wished to 
have an operation. It was done; however, it cam e out quite poorly, and as a consequence, 
he lost his sight for the better part of the last  year of his life. This  did not keep him from 
vigorous work on his monumental project, howev er. His aim was to construct a complete 
exposition of fugal writing, and us age of multiple themes was one important facet of it. In 
what he planned as the next-to-last fugue, he  inserted his own name coded into notes as

what he planned as the next-to-last fugue, he  inserted his own name coded into notes as 
the third theme. However, upon this very act, his health became so precarious that he was 
forced to abandon work on his cherished project . In his illness, he managed to dictate to 
his son-in-law a final chorale prelude, of wh ich Bach's biographer Forkel wrote, "The 
expression of pious resignation and devotion in  it has always affected me whenever I 
have played it; so that I can hardly say which I would rather miss-this Chorale, or the end 
of the last fugue." 
One day, without warning, Bach regained  his vision. But a few hours later, he

One day, without warning, Bach regained  his vision. But a few hours later, he 
suffered a stroke; and ten days later, he di ed, leaving it for others to speculate on the 
incompleteness of the Art of the Fugue. Could it have been caused by Bach's attainment 
of self-reference? 
 
Problems Caused by Gödel’s Result 
 The Tortoise says that no sufficiently powerfu l record player can be perfect, in the 
sense of being able to reproduce every possi ble sound from a record. Godel says that no 
sufficiently powerful formal system can be perfect, in the sense of reproducing every

sufficiently powerful formal system can be perfect, in the sense of reproducing every 
single true statement as a theorem. But as the Tortoise pointed out with respect to 
phonographs, this fact only seems like a defect  if you have unrealistic expectations of 
what formal systems should be able to do. Nevertheless, mathematicians began this 
century with just such unrealistic expectations, thinking that axiomatic reasoning was the cure to all ills. They found out otherwise in 1931. The fact that truth transcends 
theoremhood, in any given formal system, is called "incompleteness" of that system.

theoremhood, in any given formal system, is called "incompleteness" of that system. 
A most puzzling fact about Gödel’s method of proof is that he uses

Consistency, Completeness, and Geometry 95 
reasoning methods which seemingly cannot be "e ncapsulated"-they re being incorporated 
into any formal system. Thus, at first sight, it seems that Gödel has unearthed a hitherto unknown, but deeply significant, difference between human reasoning and mechanical reasoning. This mysterious discrepancy in th e power of living and nonliving systems is 
mirrored in the discrepancy between the noti on of truth, and that of theoremhood or at 
least that is a "romantic" way to view the situation.  
The Modified pq-System and Inconsistency 
 In order to see the situation more realisti cally, it is necessary to see in, depth why 
and how meaning is mediated, in formal sy stems, by isomorphisms. And I believe that 
this leads to a more romantic way to vi ew i situation. So we now will proceed to 
investigate some further aspects of 1 relation between meaning and form. Our first step is 
to make a new formal system by modifying our old friend, the pq-system, very slightly.

to make a new formal system by modifying our old friend, the pq-system, very slightly. 
We a one more axiom schema (retaining the original one, as well as the sin rule of 
inference): 
 Axiom SCHEMA II: If x is a hyphen-string, then xp-qx is an axiom. 
 Clearly, then, --p-q-- is a theorem in the ne w system, and so --p--q---. And yet, their 
interpretations are, respectively, "2 plus; equals 2", and "2 plus 2 equals 3". It can be seen that our new system contain a lot of false statements (if you consider strings to be 
statement Thus, our new system is inconsistent with the external world.

statement Thus, our new system is inconsistent with the external world. 
As if this weren't bad enough, we also ha ve internal problems with < new system, 
since it contains statements which disagree with one another such as -p-q-- (an old 
axiom) and -p-q- (a new axiom). So our sy stem is inconsistent in a second sense: 
internally. 
Would, therefore, the only reasonable thi ng to do at this point be drop the new 
system entirely? Hardly. I have deliberately presented the "inconsistencies" in a wool-

pulling manner: that is, I have tried to press fuzzy-headed arguments as strongly as possible, with the purpose of n leading. In f act, you may well have detected the fallacies 
in what I hi said. The crucial fallacy came when I unquestioningly adopted the very same 
interpreting words for the new system as I ha d for the old of Remember that there was 
only one reason for adopting those words in I last Chapter, and that reason was that the

only one reason for adopting those words in I last Chapter, and that reason was that the 
symbols acted isomorphically to concepts which they were matched with, by the 
interpretation. But when y modify the rules governing the system, you are bound to 
damage t isomorphism. It just cannot be helped. Thus all the problems which we 
lamented over in preceding paragraphs were bogus problems; they can made to vanish in no time, by suitably reinterpreting some of th e symbols of system. Notice that I said 
"some"; not necessarily all symbols will ha ve to mapped onto new notions. Some may 
very well retain their "meaning while others change.

Consistency, Completeness, and Geometry 96Suppose, for instance, that we reinterpret just the symbol q, leaving all the others 
constant; in particular, interpret q by the phras e "is greater than or equal to". Now, our 
"contradictory" theorems -p-q-and -p-q--come out harmlessly as: "1 plus 1 is greater than or equal to 1", and "1 plus 1 is greater than  or equal to 2". We have simultaneously gotten 
rid of (1) the inconsistency with the extern al world, and (2) the internal inconsistency. 
And our new interpretation is a meaningful inte rpretation; of course the original one is 
meaningless. That is, it is meaningless for the new system ; for the original pq-system, it is 
fine. But it now seems as pointless and arbitr ary to apply it to the new pq-system as it 
was to apply the "horse-apple-happy" interpretation to the old pq-system.  
The History of Euclidean Geometry 
 Although I have tried to catch you off gua rd and surprise you a little, this lesson

 Although I have tried to catch you off gua rd and surprise you a little, this lesson 
about how to interpret symbols by words ma y not seem terribly difficult once you have 
the hang of it. In fact, it is not. And yet it is one  of the deepest lessons of all of nineteenth 
century mathematics! It all begins w ith Euclid, who, around 300 B.C., compiled and 
systematized all of what was known about plane and solid geometry in his day. The 
resulting work, Euclid's Elements, was so solid  that it was virtually a bible of geometry 
for over two thousand years-one of the most enduring works of all time. Why was this so?

for over two thousand years-one of the most enduring works of all time. Why was this so? 
The principal reason was that Euclid was the founder of rigor in mathematics. The 
Elements  began with very simple concepts, defi nitions, and so forth, and gradually built 
up a vast body of results organized in such a way that any given result depended only on 
foregoing results. Thus, there was a definite pl an to the work, an architecture which made 
it strong and sturdy. 
Nevertheless, the architecture was of a different type from that of, say, a 
skyscraper. (See Fig. 21.) In the latter, that it is standing is proof e nough that its structural

skyscraper. (See Fig. 21.) In the latter, that it is standing is proof e nough that its structural 
elements are holding it up. But in a book on ge ometry, when each proposition is claimed 
to follow logically from earlier propositions, th ere will be no visible crash if one of the 
proofs is invalid. The girders and struts are not  physical, but abstract. In fact, in Euclid's 
Elements, the stuff out of which proofs were constructed was human language-that 
elusive, tricky medium of communication with  so many hidden pitfalls. What, then, of 
the architectural strength of the Elements? Is it certain that it is held up by solid structural

the architectural strength of the Elements? Is it certain that it is held up by solid structural 
elements, or could it have structural weaknesses? 
Every word which we use has a meaning to  us, which guides us in our use of it. 
The more common the word, the more associati ons we have with it, and the more deeply 
rooted is its meaning. Therefore, when some one gives a definition for a common word in 
the hopes that we will abide by that

Consistency, Completeness, and Geometry 97 
 
FIGURE 21. Tower of Babel, by M. C. Escher (woodcut, 1928).

Consistency, Completeness, and Geometry 98definition, it is a foregone conclusion that we will not do so but will instead be guided, 
largely unconsciously, by what our minds find in  their associative stores. I mention this 
because it is the sort of problem which Euc lid created in his Elements, by attempting to 
give definitions of ordinary, common words such  as "point", "straight line", "circle", and 
so forth. How can you define something of wh ich everyone already has a clear concept? 
The only way is if you can make it clear that  your word is supposed to be a technical 
term, and is not to be confused with the ev eryday word with the same spelling. You have 
to stress that the connection with the everyda y word is only suggestive. Well, Euclid did 
not do this, because he felt that the points a nd lines of his Elements were indeed the 
points and lines of the real world. So by not  making sure that all associations were

points and lines of the real world. So by not  making sure that all associations were 
dispelled, Euclid was inviting readers to let their powers of association run free ... 
This sounds almost anarchic, and is a little unfair to Euclid. He did set down 
axioms, or postulates, which were supposed to  be used in the proofs of propositions. In 
fact, nothing other than those axioms and postu lates was supposed to be used. But this is 
where he slipped up, for an inevitable conseque nce of his using ordinary words was that 
some of the images conjured up by those wo rds crept into the proofs which he created.

some of the images conjured up by those wo rds crept into the proofs which he created. 
However, if you read proofs in the Elements , do not by any means expect to find glaring 
"jumps" in the reasoning. On the contrary, they are very subtle, for Euclid was a 
penetrating thinker, and would not have ma de any simpleminded errors. Nonetheless, 
gaps are there, creating slight imperfections in a classic work. But this is not to be

gaps are there, creating slight imperfections in a classic work. But this is not to be 
complained about. One should merely gain an appreciation for the difference between absolute rigor and relative rigor. In the long run, Euclid's lack of absolute rigor was the cause of some of the most fertile path-br eaking in mathematics, over two thousand years 
after he wrote his work. 
Euclid gave five postulates to be used as the "ground story" of the infinite 
skyscraper of geometry, of which his Elem ents constituted only the first several hundred 
stories. The first four postulates are rather terse and elegant:

stories. The first four postulates are rather terse and elegant: 
 
(1) A straight line segment can be drawn joining any two points.  (2) Any straight line segment can be extended indefinitely in a straight line.  (3) Given any straight line segment, a ci rcle can be drawn having the segment as 
radius and one end point as center. 
 (4) All right angles are congruent.  
The fifth, however, did not share their grace: 
  (5) If two lines are drawn which intersect a third in such a way that the sum of the 
inner angles on one side is less than two right angles, then the two lines 
inevitably must intersect each other on that side if extended far enough

Consistency, Completeness, and Geometry 99Though he never explicitly said so, Euclid c onsidered this postulate to be somehow 
inferior to the others, since he managed to avoid using it in t proof s of the first twenty-
eight propositions. Thus, the first twenty-ei ght propositions belong to what might be 
called "four-postulate geometry" that part of geometry which can be derived on the basis of the first to postulates of the Elements, wit hout the help of the fifth postulate. (It is al 
often called absolute geometry.) Certainly Euclid would have found it 1 preferable to 
prove this ugly duckling, rather than to ha ve to assume it. B he found no proof, and 
therefore adopted it. 
But the disciples of Euclid were no happier about having to assume this fifth 
postulate. Over the centuries, untold numbers of  people ga untold years of their lives in 
attempting to prove that the fifth postulate s itself part of four-postulate geometry. By

attempting to prove that the fifth postulate s itself part of four-postulate geometry. By 
1763, at least twenty-eight deficient proofs ha d been published-all erroneous! (They were 
all criticized the dissertation of one G. S. Kl ugel.) All of these erroneous proofs involve a 
confusion between everyday intuition and strictly  formal properties. It safe to say that 
today, hardly any of these "proofs" holds any mathematic or historical interest-but there 
are certain exceptions. 
 
The Many Faces of Noneuclid 
 Girolamo Saccheri (1667-1733) lived around Bach's time. He had t ambition to

 Girolamo Saccheri (1667-1733) lived around Bach's time. He had t ambition to 
free Euclid of every flaw. Based on some earlier work he h; done in logic, he decided to 
try a novel approach to the proof of the famous fifth: suppose you assume its opposite; then work with that as your fif postulate  ... Surely after a while you will create a 
contradiction. Since i mathematical system  can support a contradiction, you will have

contradiction. Since i mathematical system  can support a contradiction, you will have 
shown t unsoundness of your own fifth postulate, and therefore the soundness Euclid's fifth postulate. We need not go into details here. Suffice it to s that with great skill, 
Saccheri worked out proposition after propositi on "Saccherian geometry" and eventually 
became tired of it. At one point, decide d he had reached a proposition which was 
"repugnant to the nature of the straight line" . That was what he had been hoping for-to his 
mind was the long-sought contradiction. At th at point, he published his work under the

mind was the long-sought contradiction. At th at point, he published his work under the 
title Euclid Freed of Every Flaw , and then expired. 
But in so doing, he robbed himself of  much posthumous glory, sir he had 
unwittingly discovered what came later to be  known as "hyperbolic geometry". Fifty 
years after Saccheri, J. H. Lambert repeat ed the "near miss", this time coming even 
closer, if possible. Finally, forty years afte r Lambert, and ninety years after Saccheri, 
non-Euclidean geometry was recognized for what it was-an authentic new brand of 
geometry, a bifurcation the hitherto single stream of mathematics. In 1823, non-
Euclidean geometry  was discovered simultaneously, in one of those inexplicable

Euclidean geometry  was discovered simultaneously, in one of those inexplicable 
coincidences, by a Hungarian mathematician, Janos (or Johann) Bolyai, age twenty-one, 
and a Russian mathematician, Nikolay Lobachev skiy, ag thirty. And, ironically, in that 
same year, the great French mathematician

Consistency, Completeness, and Geometry 100Adrien-Marie Legendre came up with what he was sure was a proof of Euclid's fifth 
postulate, very much along the lines of Saccheri. 
Incidentally, Bolyai's father, Farkas (o r Wolfgang) Bolyai, a close friend of the 
great Gauss, invested much effort in trying to  prove Euclid's fifth postulate. In a letter to 
his son Janos, he tried to dissuade him from thinking about such matters:  
You must not attempt this approach to paralle ls. I know this way to its very end. I have 
traversed this bottomless night, which extinguished  all light and joy of my life. I entreat 
you, leave the science of parallels alone.... I thought I would sacrifice myself for the sake 
of the truth. I was ready to become a martyr who would remove the flaw from geometry 
and return it purified to mankind. I accomp lished monstrous, enormous labors; my 
creations are far better than those of othe rs and yet I have not achieved complete

creations are far better than those of othe rs and yet I have not achieved complete 
satisfaction. For here it is true that si paullum a summo discessit, vergit ad imum.  I turned 
back when I saw that no man can reach the bo ttom of this night. I turned back unconsoled, 
pitying myself and all mankind.... I have traveled past all reefs of this infernal Dead Sea 
and have always come back with broken mast and torn sail. The ruin of my disposition and 
my fall date back to this time. I thoughtlessly risked my life and happiness sut Caesar aut 
nihil.' 
 
But later, when convinced his son rea lly "had something", he urged him to

nihil.' 
 
But later, when convinced his son rea lly "had something", he urged him to 
publish it, anticipating correctly the simultane ity which is so frequent in scientific 
discovery:  
When the time is ripe for certain things, th ese things appear in different places in 
the manner of violets coming to light in early spring. 
 How true this was in the case of non-Euclidean geometry! In Germany, Gauss 
himself and a few others had more or le ss independently hit upon non-Euclidean ideas.

nihil.' 
 
himself and a few others had more or le ss independently hit upon non-Euclidean ideas. 
These included a lawyer, F. K. Schweikart, who in 1818 sent a page describing a new "astral" geometry to Gauss; Schweikart's nephew, F. A. Taurinus, who did non-Euclidean 
trigonometry; and F. L. Wachter, a student of  Gauss, who died in 1817, aged twenty-five, 
having found several deep results in non-Euclidean geometry. 
The clue to non-Euclidean geometry was "thinking straight" about the 
propositions which emerge in geometries lik e Saccheri's and Lambert's. The Saccherian

propositions which emerge in geometries lik e Saccheri's and Lambert's. The Saccherian 
propositions are only "repugnant to the nature of the straight line" if you cannot free 
yourself of preconceived notions of what "s traight line" must mean. If, however, you can 
divest yourself of those preconceived imag es, and merely let a "straight line" be 
something which satisfies the new propositions , then you have achieved a radically new 
viewpoint. 
 
Undefined Terms

Undefined Terms 
 
This should begin to sound familiar. In particular, it harks back to the pq-system, and its variant, in which the symbols acquired passi ve meanings by virtue of their roles in 
theorems. The symbol q is especially interesting,

Consistency, Completeness, and Geometry 101 
since its "meaning" changed when a new ax iom schema was added. In the very same 
way, one can let the meanings of "point", "line", and so on I determined by the set of 
theorems (or propositions) in which they occur . This was th great realization of the 
discoverers of non-Euclidean geometry. Th e found different sorts of non-Euclidean 
geometries by denying Euclid's fifth postulate  in different ways and following out the 
consequences. Strict] speaking, they (and S accheri) did not deny the fifth postulate 
directly, but rather, they denied an equivalent postulate, called the parallel postulate, 
which runs as follows:  
Given any straight line, and a point not on it, there exists one, and only one, straight 
line which passes through that point and ne ver intersects the first line, no matter 
how far they are extended. 
 The second straight line is then said to be parallel to the first. If you assert that no

 The second straight line is then said to be parallel to the first. If you assert that no 
such line exists, then you reach elliptical geometry ; if you assert that, at east two such 
lines exist, you reach hyperbolic geometry . Incidentally, tf reason that such variations are 
still called "geometries" is that the cot elem ent-absolute, or four-postulate, geometry-is 
embedded in them. is the presence of this mi nimal core which makes it sensible to think 
of the] as describing properties of some sort of geometrical space, even  if the spa( is not 
as intuitive as ordinary space.

as intuitive as ordinary space. 
Actually, elliptical geometry is easily visualized. All "points", "lines and so forth 
are to be parts of the surf ace of an ordinary sphere. Let t write "POINT" when the 
technical term is meant, and "point" when t1  everyday sense is desired. Then, we can say 
that a POINT consists of a pa of diametri cally opposed points of the sphere's surface. A 
LINE is a great circle on the sphere (a circle which, like the equator, has its center at tI

LINE is a great circle on the sphere (a circle which, like the equator, has its center at tI 
center of the sphere). Under these interpre tations, the propositions ( elliptical geometry, 
though they contain words like "POINT" and "LINE speak of the goings-on on a sphere, not a plane. Notice that two LINT always in tersect in exactly two antipodal points of the 
sphere's surface that is, in exactly one singl e POINT! And just as two LINES determine 
POINT, so two POINTS determine a LINE. 
By treating words such as "POINT" and "LINE" as if they had only tt meaning

By treating words such as "POINT" and "LINE" as if they had only tt meaning 
instilled in them by the propositions in which th ey occur, we take step towards complete 
formalization of geometry. This semiformal version still uses a lot of words in English with their usual meanings (words such "the",  ` if ", "and", "join", "have"), although the 
everyday meaning has bee drained out of sp ecial words like "POINT" and "LINE", which 
are consequently called undefined terms . Undefined terms, like the p and q of th pq-

system, do get defined in a sense: implicitly-by the totality of all propos dons in which they occur, rather than explicitly, in a definition. 
One could maintain that a full definiti on of the undefined tern resides in the 
postulates alone, since the propositions which follow from them are implicit in the postulates already. This view would say that the postulates are implicit definitions of all the undefined terms, all of the undefined terms being defined in terms of the others.

Consistency, Completeness, and Geometry 102The Possibility of Multiple Interpretations 
 
A full formalization of geometry would take  the drastic step of making every term 
undefined-that is, turning every term into a "meaningless" symbol of a formal system. I put quotes around "meaningless" because, as you know, the symbols automatically pick 
up passive meanings in accordance with the theorems they occur in. It is another 
question, though, whether people discover those meanings, for to do so requires finding a 
set of concepts which can be linked by an isomorphism to the symbols in the formal 
system. If one begins with the aim of fo rmalizing geometry, presumably one has an 
intended interpretation for each symbol, so th at the passive meanings are built into the 
system. That is what I did for p and q when I first created the pq-system. 
But there may be other passive meanings  which are potentially perceptible, which

But there may be other passive meanings  which are potentially perceptible, which 
no one has yet noticed. For instance, there we re the surprise interpretations of p as 
"equals" and q as "taken from", in the origin al pq-system. Although this is rather a trivial 
example, it contains the essence of the id ea that symbols may have many meaningful 
interpretations-it is up to the observer to look for them. 
We can summarize our observations so far in terms of the word "consistency". 
We began our discussion by manuf acturing what appeared to be an inconsistent formal

We began our discussion by manuf acturing what appeared to be an inconsistent formal 
system-one which was internally inconsistent, as well as inconsistent with the external 
world. But a moment later we took it all back, when we realized our error: that we had 
chosen unfortunate interpretations for the sy mbols. By changing the interpretations, we 
regained consistency! It now becomes clear that consistency is not a property of a formal 
system per se, but depends on the in terpretation which is proposed for it . By the same 
token, inconsistency is not an intrinsic property of any formal system.

token, inconsistency is not an intrinsic property of any formal system. 
 
Varieties of Consistency 
 We have been speaking of "consistency" and "inconsistency" all along, without 
defining them. We have just relied on good old everyday notions. But now let us say 
exactly what is meant by consistency  of a formal system (togeth er with an interpretation): 
that every theorem, when interpreted, become s a true statement. And we will say that 
inconsistency occurs when there is at l east one false statement among the interpreted 
theorems. 
This definition appears to be talking about  inconsistency with the external world-

theorems. 
what about internal inconsistencies? Pr esumably, a system would be internally 
inconsistent if it contained two or more theorems whose interpretations were incompatible with one another, and internally consistent if all interpreted theorems were 
compatible with one another. Consider, for example, a formal system which has only the 
following three theorems: TbZ, ZbE , and EbT . If T is interpreted as "the Tortoise",  Z as 
"Zeno", E as "Egbert", and x by as "x beats y in chess always", then we have the 
following interpreted theorems:

Consistency, Completeness, and Geometry 103 
The Tortoise always beats Zeno at chess Zeno always beats Egbert at chess. Egbert always beats the Tortoise at chess.  
The statements are not incompatible, although they describe a rather bizarre circle of chess players. Hence, under this interpreta tion, the form; system in which those three 
strings are theorems is internally consistent  although, in point of f act, none of the three 
statements is true! Intern< consistency does not require all theorems to come out true, but merely that they come out compatible  with one another. 
Now suppose instead that x by is to be inte rpreted as "x was invented by y". Then 
we would have: 
 The Tortoise was invented by Zeno.  Zeno was invented by Egbert. Egbert was invented by the Tortoise.  
In this case, it doesn't matter whether the individual statements are true c false-and

In this case, it doesn't matter whether the individual statements are true c false-and 
perhaps there is no way to know which ones are true, and which are not. What is nevertheless certain is that not all three can be true at one  Thus, the interpretation makes 
the system internally inconsistent. The internal inconsistency depends not on the interpretations of the three capital letters, but  only on that of b, and on the fact that the 
three capita are cyclically permuted around the occurrences of b. Thus, one can have 
internal inconsistency without having interpreted all of the symbols of the formal system.

internal inconsistency without having interpreted all of the symbols of the formal system. 
(In this case it sufficed to interpret a single symbol.) By tl time sufficiently many symbols 
have been given interpretations, it may t clear that there is no way that the rest of them 
can be interpreted so that a theorems will co me out true. But it is not just a question of 
truth-it is question of possibility. All three theo rems would come out false if the capitals 
were interpreted as the names of real people- but that is not why we would call the system 
internally inconsistent; our grounds for doing s would be the circularity, combined with 
the interpretation of the letter I (By the way, you'll find more on this "authorship triangle"

the interpretation of the letter I (By the way, you'll find more on this "authorship triangle" 
in Chapter XX.;  
Hypothetical Worlds and Consistency 
 We have given two ways of looking at consistency: the first says that system-
plus-interpretation is consistent with the external world  if every theorem comes out true 
when interpreted; the second says that a system-plus: interpretation is internally 
consistent  if all theorems come out mutually compatible when interpreted. Now there is a 
close relationship between these two types of consistency. In order to determine whether

close relationship between these two types of consistency. In order to determine whether 
several statements at mutually compatible, you try to imagine a world in which all of 
them could be simultaneously true. Ther efore, internal consistency depends upon 
consistency with the external world-only now , "the external world" allowed to be any 
imaginable world , instead of the one we live in. But this is

Consistency, Completeness, and Geometry 104an extremely vague, unsatisfactory conclusion. What constitutes an “imaginable" world? 
After all, it is possible to imagine a world in which three characters invent each other 
cyclically. Or is it? Is it possible to imagine a world in which there are square circles? Is a 
world imaginable in which Newton's laws, and not relativity, hold? Is it possible to 
imagine a world in which something can be  simultaneously green and not green? Or a 
world in which animals exist which are not made of cells? In which Bach improvised an eight-part fugue on a theme of King Frederick the Great? In which mosquitoes are more 
intelligent than people? In which tortoises can  play football-or talk? A tortoise talking 
football would be an anomaly, of course. 
Some of these worlds seem more imaginable than others, since some seem to 
embody logical  contradictions-for example, green and not green-while some of them

embody logical  contradictions-for example, green and not green-while some of them 
seem, for want of a better word, "plausible " -- such as Bach improvising an eight-part 
fugue, or animals which are not made of cells. Or even, come to think of it, a world in 
which the laws of physics are different ... Roughly, then, it should be possible to establish different brands of consistency. For inst ance, the most lenient would be "logical 
consistency", putting no restraints on things at all, except those of logic. More 
specifically, a system-plus-interpretation would be logically consistent  just as long as no

specifically, a system-plus-interpretation would be logically consistent  just as long as no 
two of its theorems, when interpreted as stat ements, directly contradict each other; and 
mathematically consistent  just as long as interpreted theorems do not violate 
mathematics; and physically consistent  just as long as all its interpreted theorems are 
compatible with physical law; then come s biological consistency, and so on. In a 
biologically consistent system, there could be a theorem whose interpretation is the

biologically consistent system, there could be a theorem whose interpretation is the 
statement "Shakespeare wrote an opera", but no theorem whose interpretation is the statement "Cell-less animals exist". Generally speaking, these fancier kinds of inconsistency are not studied, for the reason that  they are very hard to disentangle from 
one another. What kind of inconsistency, for example, should one say is involved in the 
problem of the three characters who invent  each other cyclically ? Logical? Physical? 
Biological? Literary? 
Usually, the borderline between uninteres ting and interesting is drawn between

Usually, the borderline between uninteres ting and interesting is drawn between 
physical consistency and mathematical consistency. (Of course, it is the mathematicians and logicians who do the drawing-hardly an impa rtial crew . . .) This means that the kinds 
of inconsistency which "count", for formal syst ems, are just the logical and mathematical 
kinds. According to this convention, then, we  haven't yet found an interpretation which 
makes the trio of theorems TbZ, ZbE, EbT  inconsistent. We can do so by interpreting b 
as "is bigger than". What about T and Z and E? They can be interpreted as natural 
numbers-for example, Z as 0, T as 2, and E as 11. Notice that two theorems come out 
true this way, one false. If, instead, we had interpreted Z as 3, there would have been two

true this way, one false. If, instead, we had interpreted Z as 3, there would have been two 
falsehoods and only one truth. But either way, we'd have had inconsistency. In fact, the 
values assigned to T, Z,  and E are irrelevant, as long as it is understood that they are 
restricted to natural numbers. Once again we see a case where only some  of the 
interpretation is needed, in order to recognize internal inconsistency.

Consistency, Completeness, and Geometry 105Embedding of One Formal System In Another 
 
The preceding example, in which some symbol s could have interpretations while others 
didn't, is reminiscent of doing geometry in natural languag4 using some words as 
undefined terms. In such a case, words are di vide into two classes: those whose meaning 
is fixed and immutable, and, those whose mean ing is to be adjusted until the system is 
consistent (these are th undefined terms). Doing geometry in this way requires that meanings have already been established for wo rds in the first class, somewhere outside c 
geometry. Those words form a rigid skel eton, giving an underlying structure to the 
system; filling in that skeleton comes othe r material, which ca vary (Euclidean or non-
Euclidean geometry). 
Formal systems are often built up in just th is type of sequential, c hierarchical, 
manner. For example, Formal System I may be  devised, wit rules and axioms that give

manner. For example, Formal System I may be  devised, wit rules and axioms that give 
certain intended passive meanings to its sym bol Then Formal System I is incorporated 
fully into a larger system with more symbol s-Formal System II. Since Formal System I's 
axioms and rules at part of Formal System II, the passive meanings of Formal System I 
symbols remain valid; they form an immutable skeleton which then plays large role in the 
determination of the passive meanings of the new symbols of Formal System II. The 
second system may in turn play the role of sk eleton with respect to a third system, and so 
on. It is also possible-an geometry is a good example of this-to have a system (e.g.,

on. It is also possible-an geometry is a good example of this-to have a system (e.g., 
absolute geometry) which partly  pins down the passive meanings of its undefined terms, 
and which can be supplemented by extra rules or axioms, which then further  restrict the 
passive meanings of the undefined terms. This the case with Euclidean versus non-
Euclidean geometry.  
Layers of Stability in Visual Perception 
 In a similar, hierarchical way, we acquire new knowledge, new vocabulary or

 In a similar, hierarchical way, we acquire new knowledge, new vocabulary or 
perceive unfamiliar objects. It is particularly interesting in the case understanding drawings by Escher, such as Relativity (Fig. 22), in which there occur blatantly impossible images. You might think that we w on seek to reinterpret the picture over and 
over again until we came to interpretation of its parts which was free of contradictions-but we dot do that at all. We sit there am used and puzzled by staircases which go eve 
which way, and by people going in inconsistent  directions on a sing staircase. Those

which way, and by people going in inconsistent  directions on a sing staircase. Those 
staircases are "islands of certainty" upon whic h we base of interpretation of the overall 
picture. Having once identified them, we try extend our understanding, by seeking to 
establish the relationship which they bear to one another. At that stage, we encounter 
trouble. But if i attempted to backtrack-that is, to question the "islands of certainty"-s 
would also encounter trouble, of another sort. There's no way of backtracking and 
"undeciding" that they are staircases. They ar e not fishes, or whip or hands-they are just 
staircases. (There is, actually, one other on t-i leave all the lines of the picture totally uninterpreted, like the "meaningless

Consistency, Completeness, and Geometry 106
 
FIGURE 22. Relativity, by M. C. Escher (lithograph, 1953). 
  symbols" of a formal system. This ultimat e escape route is an example of a "U-mode" 
response-a Zen attitude towards symbolism.) 
So we are forced, by the hierarchical natu re of our perceptive processes, to see 
either a crazy world or just a bunch of poin tless lines. A similar analysis could be made 
of dozens of Escher pictures, which rely heavily upon the recognition of certain basic 
forms, which are then put together in nonstandard ways; and by the time the observer 
sees the paradox on a high level, it is too late -he can't go back and change his mind about 
how to interpret the lower-level objects. Th e difference between an Escher drawing and 
non-Euclidean geometry is that in the latter,  comprehensible interpretations can be found 
for the undefined terms, resulting in a com

Consistency, Completeness, and Geometry 107prehensible total system, whereas for the form er, the end result is not reconcilable with 
one's conception of the world, no matter how long or stares at the pictures. Of course, one 
can still manufacture hypothetic worlds, in whic h Escherian events can happen ... but in 
such worlds, t1 laws of biology, physics, math ematics, or even logic will be violated on 
or level, while simultaneously being obeye d on another, which makes the: extremely 
weird worlds. (An example of this is in Waterfall  (Fig. 5), whet normal gravitation 
applies to the moving water, but where the nature space violates the laws of physics.)  
Is Mathematics the Same in Every Conceivable World? 
 
We have stressed the fact, above, that internal  consistency of a form; system (together 
with an interpretation) requires that there be some imaginable  world-that is, a world 
whose only restriction is that in it, mathema tics and logic should be the same as in our

whose only restriction is that in it, mathema tics and logic should be the same as in our 
world-in which all the interpreted theorems come out true. External  consistency, however 
consistency with the external world-requires th at all theorems come of true in the real 
world. Now in the special case where one wishes to create consistent formal system whose theorems are to be interpreted as stat ements of mathematics, it would seem that

world. Now in the special case where one wishes to create consistent formal system whose theorems are to be interpreted as stat ements of mathematics, it would seem that 
the difference between the two types of consistency should fade away, since, according to what we sat above, all imaginable worlds have the same mathematics as the real world . 
Thus, i every conceivable world, 1 plus 1 would have to be 2; likewise, there would have 
to be infinitely many prime numbers; furtherm ore, in every conceivable world, all right 
angles would have to be congruent; and of  cours4 through any poi nt not on a given line 
there would have to be exactly on parallel line ...

there would have to be exactly on parallel line ... 
But wait a minute! That's the parallel postulate-and to assert i universality would 
be a mistake, in light of what's just been said. If in all conceivable worlds the parallel 
postulate-is obeyed, then we are asserting that non-Euclidean geometry is inconceivable, which puts us back in the same mental stat e as Saccheri and Lambert-surely an unwise 
move. But what, then, if not all of mathematics, must all conceivable worlds share ?

move. But what, then, if not all of mathematics, must all conceivable worlds share ? 
Could it I as little as logic itself? Or is even logic suspect? Could there be worlds where contradictions are normal parts of existe nce-worlds where contradictious are not 
contradictions? 
Well, in some sense, by merely inventing the concept, we have shoe that such 
worlds are indeed conceivable; but in a deeper sense, they are al: quite inconceivable. (This in itself is a little contradiction.) Quite serious] however, it seems that if we want to

worlds are indeed conceivable; but in a deeper sense, they are al: quite inconceivable. (This in itself is a little contradiction.) Quite serious] however, it seems that if we want to 
be able to communicate at all, we ha, to adopt some common base, and it pretty well has to include logic. (The are belief systems which reject this point of view-it is too logical. 
particular, Zen embraces contradictions and non- contradictions with equ eagerness. This 
may seem inconsistent, but then being inconsis tent is pa of Zen, and so ... what can one 
say?)

Consistency, Completeness, and Geometry 108Is Number Theory the Same In All Conceivable Worlds? 
 
If we assume that logic  is part of every conceivabl e world (and note that we have 
not defined logic, but we will in Chapters to come), is that all? Is it really conceivable 
that, in some worlds, there are not infinite ly many primes? Would it not seem necessary 
that numbers should obey the same laws in a ll conceivable worlds? Or ... is the concept 
"natural number" better thought of as an undefined term, like "POINT" or "LINE"? In 
that case, number theory would be a bifurcated theory, like geometry: there would be standard and nonstandard number theories. But there would have to be some counterpart 
to absolute geometry: a "core" theory, an invariant ingredient of all number theories 
which identified them as number theories rather than, say, theories about cocoa or rubber or bananas. It seems to be the consen sus of most modern mathematicians and

which identified them as number theories rather than, say, theories about cocoa or rubber or bananas. It seems to be the consen sus of most modern mathematicians and 
philosophers that there is such a core num ber theory, which ought to be included, along 
with logic, in what we consider to be "con ceivable worlds". This core of number theory, 
the counterpart to absolute geometry-is called Peano arithmetic , and we shall formalize it 
in Chapter VIII. Also, it is now well estab lished-as a matter of fact as a direct 
consequence of Gödel’s Theorem-that numbe r theory is a bifurcated theory, with

consequence of Gödel’s Theorem-that numbe r theory is a bifurcated theory, with 
standard and nonstandard versions. Unlike the situation in geometry, however, the number of "brands" of number theory is infi nite, which makes the situation of number 
theory considerably more complex. 
For practical purposes, all number theories ar e the same. In other words, if bridge 
building depended on number theory (which in a sense it does), the fact that there are 
different number theories would not matter, sin ce in the aspects relevant to the real world, 
all number theories overlap. The same cannot  be said of different geometries; for

all number theories overlap. The same cannot  be said of different geometries; for 
example, the sum of the angles in a triangle is 180 degrees only in Euclidean geometry; it is greater in elliptic geometry, less in hype rbolic. There is a story that Gauss once 
attempted to measure the sum of the angles in a large triangle defined by three mountain 
peaks, in order to determine, once and for a ll, which kind of geometry really rules our 
universe. It was a hundred years later that Ei nstein gave a theory (general relativity) 
which said that the geometry of the universe is determined by its content of matter, so 
that no one geometry is intrinsic to space itsel f. Thus to the question, "Which geometry is

that no one geometry is intrinsic to space itsel f. Thus to the question, "Which geometry is 
true?" nature gives an ambiguous answer not onl y in mathematics, but also in physics. As 
for the corresponding question, "Which number theo ry is true?", we shall have more to 
say on it after going through Gödel’s Theorem in detail. 
 
Completenes 
 If consistency is the minimal condition unde r which symbols acquire passive meanings, 
then its complementary notion, completeness,  is the maximal confirmation of those 
passive meanings. Where consistency is the property

Consistency, Completeness, and Geometry 109way round: "Every true statement is produced  by the system". Now I refine the notion 
slightly. We can't mean every true statem ent in th world-we mean only those which 
belong to the domain which we at attempting to represent in the system. Therefore, completeness mean! "Every true statement whic h can be expressed in the notation of the 
system is a theorem."  
Consistency: when every theorem, upon in terpretation, comes out true (in some 
imaginable world). 
Completeness: when all statements which are true (in some imaginable world), and 
which can be expressed as well-formed strings of the system, are theorems. 
 
An example of a formal system which is complete on its own mode level is the 
original pq-system, with the original inte rpretation. All true additions of two positive 
integers are represented by theorems of th system. We might say this another way: "All true additions of two positive integers are pr ovable within the system." (Warning: When

integers are represented by theorems of th system. We might say this another way: "All true additions of two positive integers are pr ovable within the system." (Warning: When 
we start using th term "provable statements" instead of "theorems", it shows that we at 
beginning to blur the distinction between formal  systems and their interpretations. This is 
all right, provided we are very conscious of th  blurring that is taking place, and provided 
that we remember that multiple interpreta tions are sometimes possible.) The pq-system 
with the origin interpretation is complete ; it is also consistent , since no false statement is-, 
use our new phrase-provable within the system.

use our new phrase-provable within the system. 
Someone might argue that the system is incomplete, on the grounds that additions 
of three  positive integers (such as 2 + 3 + 4 =9) are not represented by theorems of the 
pq-system, despite being translatable into the notation of the system (e.g., --p---p----q----
--------). However, this string is not well-formed, and hence should be considered to I just 
as devoid of meaning as is p q p---q p q . Triple additions are simply not expressible  in 
the notation of the system-so the completeness of the system is preserved.

the notation of the system-so the completeness of the system is preserved. 
Despite the completeness of the pq-system under this interpretation, certainly falls 
far short of capturing the full notion of trut h in numb theory. For example, there is no 
way that the pq-system tells us how mat prime numbers there are. Gödel’s Incompleteness Theorem says that any system  which is "sufficiently powerful" is, by

way that the pq-system tells us how mat prime numbers there are. Gödel’s Incompleteness Theorem says that any system  which is "sufficiently powerful" is, by 
virtue of its power, incomplete, in the sense that there are well-formed strings which express tr statements of number theory, but which are not theorems. (There a truths belonging to number theory which are not prova ble within the system.) Systems like the 
pq-system, which are complete but not very powerful, are more like low-fidelity phonographs; they are so poor to beg with that  it is obvious that they cannot do what we 
would wish them do-namely tell us everything about number theory.

Consistency, Completeness, and Geometry 110How an Interpretation May Make or Break Completeness 
 
What does it mean to say, as I did above, that "completeness is the maximal confirmation of passive meanings"? It means that if a system is consistent but incomplete, there is a mismatch between the symbols and their inte rpretations. The system does not have the 
power to justify being interpreted that way. Sometimes, if the interpretations are 
"trimmed" a little, the system can become comple te. To illustrate this idea, let's look at 
the modified pq-system (including Axiom Sche ma II) and the interpretation we used for 
it. 
After modifying the pq-system, we modified  the interpretation for q from "equals" 
to "is greater than or equal to". We saw th at the modified pq-system was consistent under 
this interpretation; yet something about the new interpretation is not very satisfying. The 
problem is simple: there are now many expre ssible truths which are not theorems. For

it. 
problem is simple: there are now many expre ssible truths which are not theorems. For 
instance, "2 plus 3 is greater than or equal to 1" is expressed by the nontheorem --p---q-.  
The interpretation is just too sloppy! It doesn't accurately reflect what the theorems in the system do. Under this sloppy interpretation, th e pq-system is not complete. We could 
repair the situation either by (1) adding new rules  to the system, making it more 
powerful, or by (2) tightening up the interpretation . In this case, the sensible alternative 
seems to be to tighten the interpretation. Instead of interpreting q as "is greater than or

seems to be to tighten the interpretation. Instead of interpreting q as "is greater than or 
equal to", we should say "equals or exceeds by 1". Now the modified pq-system becomes both consistent and complete. And the comple teness confirms the appropriateness of the 
interpretation. 
 
Incompleteness of Formalized Number Theory 
 In number theory, we will encounter incomp leteness again; but there, to remedy the 
situation, we will be pulled in the other dir ection-towards adding new rules, to make the

situation, we will be pulled in the other dir ection-towards adding new rules, to make the 
system more powerful. The irony is that we think, each time we add a new rule, that we surely have made the system complete now! The nature of the dilemma can be illustrated' 
by the following allegory ... 
We have a record player, and we also ha ve a record tentatively labeled "Canon on 
B-A-C-H ". However, when we play the record  on the record player, the feedback-

induced vibrations (as caused by the Tortoise's records) interfere so much that we do not even recognize the tune. We conclude that something  is defective-either our record, or 
our record player. In order to test our record , we would have to play it on friends' record 
players, and listen to its quality. In order to test our phonograph , we would have to play 
friends' records on it, and see if the music we hear agrees with the labels. If our record

friends' records on it, and see if the music we hear agrees with the labels. If our record 
player passes its test, then we will say the record was defective; contrariwise, if the record passes its test, then we will say our record player was defective. What, however, 
can we conclude when we find out that both pass their respective tests? That is the 
moment to remember the chain of two isomorphisms (Fig. 20), and think carefully!

Little Harmonic Labyrinth 111Little Harmonic Labyrinth 
 
The Tortoise and Achilles are spending a day at Coney Island After buying a 
couple of cotton candies, they decide to take a ride on the Ferris wheel. 
 
Tortoise: This is my favorite ride. One seems to move so far, and reality one gets nowhere. Achilles: I can see why it would appeal to you. Are you all strapped in? Tortoise: Yes, I think I've got this buckle done. Well, here we go.  Achilles: You certainly are exuberant today. Tortoise: I have good reason to be. My aunt, w ho is a fortune-teller me that a stroke of 
Good Fortune would befall me today. So I am tingling with anticipation. 
Achilles: Don't tell me you believe in fortune-telling! Tortoise: No ... but they say it works even if you don't believe ii  Achilles: Well, that's fortunate indeed.  Tortoise: Ah, what a view of the beach, the crowd, the ocean, the city. . .   Achilles: Yes, it certainly is splendid. Say, look at that helicopter there. It seems to be

Achilles: Don't tell me you believe in fortune-telling! Tortoise: No ... but they say it works even if you don't believe ii  Achilles: Well, that's fortunate indeed.  Tortoise: Ah, what a view of the beach, the crowd, the ocean, the city. . .   Achilles: Yes, it certainly is splendid. Say, look at that helicopter there. It seems to be 
flying our way. In fact it's almost directly above us now. 
Tortoise: Strange-there's a cable dangling down from it, which is very close to us. It's 
coming so close we could practically grab it  
Achilles: Look! At the end of the line there's a giant hook, with a note  
(He reaches out and snatches the note. They pass by and are on their z down.) 
 
Tortoise: Can you make out what the note says? Achilles: Yes-it reads, "Howdy, friends. Grab a hold of the hook time around, for an 
Unexpected Surprise." 
Tortoise: The note's a little corny but who knows where it might lead, Perhaps it's got

Tortoise: The note's a little corny but who knows where it might lead, Perhaps it's got 
something to do with that bit of Good Fortune due me. By all means, let's try it! 
Achilles: Let's!  
(On the trip up they unbuckle their buckles, and at the crest of the ride, grab for the 
giant hook. All of a sudden they are whoos hed up by the ca which quickly reels 
them skyward into the hovering helicopter. A It strong hand helps them in.) 
 
Voice: Welcome aboard-Suckers.  Achilles: Wh-who are you?

Little Harmonic Labyrinth 112Voce: Allow me to introduce myself. I am Hexachlorophene J. Goodforttune, Kidnapper 
At-Large, and Devourer of Tortoises par Excellence, at your service. 
Tortoise: Gulp! Achilles ( whispering to his friend ): Uh-oh-I think that this "Goodfortune" is not exactly 
what we'd anticipated. (To Goodfortune)  Ah-if I may be so bold-where are you 
spiriting us off to? 
Goodfortune: Ho ho! To my all-electric k itchen-in-the-sky, where I will prepare THIS 
tasty morsel-( leering at the Tortoise as he says this )-in a delicious pie-in-the-sky! 
And make no mistake-it's all just for my gobbling pleasure! Ho ho ho! 
Achilles: All I can say is you've got a pretty fiendish laugh. Goodfortune ( laughing fiendishly):  Ho ho ho! For that remark, my friend, you will pay 
dearly. Ho ho! 
Achilles: Good grief-I wonder what he means by that! Goodfortune: Very simple-I've got a Sinister Fate in store for both of you! Just you wait! 
Ho ho ho! Ho ho ho!

Ho ho ho! Ho ho ho! 
Achilles: Yikes! Goodfortune: Well, we have arrived. Disembark, my friends, into my fabulous all-electric 
kitchen-in-the-sky. 
 (They walk inside.)  
Let me show you around, before I prepare your fates. Here is my bedroom. Here is 
my study. Please wait here for me for a mome nt. I've got to go sharpen my knives. 
While you're waiting, help yourselves to some popcorn. Ho ho ho! Tortoise pie! 
Tortoise pie! My favorite kind of pie! (Exit.) 
 
Achilles: Oh, boy-popcorn! I'm going to munch my head off!

Achilles: Oh, boy-popcorn! I'm going to munch my head off! 
Tortoise: Achilles! You just stuffed yourse lf with cotton candy! Besides, how can you 
think about food at a time like this? 
Achilles: Good gravy-oh, pardon me-I shouldn't use that turn of phrase, should I? I mean 
in these dire circumstances ... Tortoise: I'm afraid our goose is cooked. 
Achilles: Say-take a gander at all these books  old Goodfortune has in his study. Quite a 
collection of esoterica: Birdbrains I Have Known; Chess and Umbrella-Twirling

collection of esoterica: Birdbrains I Have Known; Chess and Umbrella-Twirling 
Made Easy; Concerto for Tapdancer and Orchestra  ... H mmm. 
Tortoise: What's that small volume lying open over there on the desk, next to the 
dodecahedron and the open drawing pad? 
Achilles: This one? Why, its title is Provocative Adventures of Achilles and the Tortoise 
Taking Place in Sundry Spots of the Globe . Tortoise: A moderately provocative 
title. 
Achilles: Indeed-and the adventure it's opened to looks provocative. It's called "Djinn and 
Tonic". 
Tortoise: Hmm ... I wonder why. Shall we try read ing it? I could take the Tortoise's part,

title. 
Tonic". 
Tortoise: Hmm ... I wonder why. Shall we try read ing it? I could take the Tortoise's part, 
and you could take that of Achilles.

Little Harmonic Labyrinth 113Achilles:  I’m game. Here goes nothing . . . 
 
(They begin reading "Djinn and Tonic".) 
 
(Achilles has invited the Tortoise over to see his collection of prints by 
his favorite artist, M. C. Escher.) 
 
Tortoise: These are wonderful prints, Achilles. Achilles: I knew you would enjoy seeing them. Do you have any particular 
favorite? 
Tortoise: One of my favorites is Conve x and Concave, where two internally 
consistent worlds, when juxtaposed, make a completely inconsistent composite world. Inconsistent worlds are always fun places to visit, but I wouldn't want to live there. 
Achilles: What do you mean, "fun to vis it"? Inconsistent worlds don't EXIST, 
so how can you visit one? 
Tortoise: I beg your pardon, but weren't we just agreeing that in 
 
this Escher picture, an inconsistent world is portrayed? Achilles: Yes, but that's just a two-dimensional world-a fictitious world-a 
picture. You can't visit that world.

picture. You can't visit that world. 
Tortoise: I have my ways ... Achilles: How could you propel yourself  into a flat picture-universe? 
Tortoise: By drinking a little glass of PUSHING-POTION. That does the 
trick. 
Achilles: What on earth is pushing-potion? Tortoise: It's a liquid that comes in small ceramic phials, and which, when 
drunk by someone looking at a picture,  "pushes'' him right into the 
world of that picture. People w ho aren't aware of the powers of 
pushing-potion often are pretty surp rised by the situations they wind 
up in. 
Achilles: Is there no antidote? Once pushed, is one irretrievably lost? Tortoise: In certain cases, that's not so bad a fate. But there is, in fact, another

trick. 
up in. 
Achilles: Is there no antidote? Once pushed, is one irretrievably lost? Tortoise: In certain cases, that's not so bad a fate. But there is, in fact, another 
potion-well, not a potion, actually, but an elixir-no, not an elixir, but 
a-a 
Tortoise: He probably means "tonic". Achilles: Tonic? Tortoise: That's the word I was looki ng for! "POPPING-TONIC" iu what it's 
called, and if you remember to carry a bottle of it in your right hand as 
you swallow the pushing-potion, it too w ill be pushed into the picture; 
then, whenever you get a hanker ing to  "pop" back out into real life,

a-a 
then, whenever you get a hanker ing to  "pop" back out into real life, 
you need only take a swallow of popping-tonic, and presto! You're back in the rea. world, exactly  where you were before you pushed 
yourself in. 
Achilles: That sounds very interes ting. What would happen it you took some 
popping-tonic without having previ ously pushed yourself into a 
picture?

Little Harmonic Labyrinth 114Tortoise:  I don’t precisely know, Achille s, but I would be rather wary of 
horsing around with these strange pushing and popping liquids. Once I 
had a friend, a Weasel, who did pr ecisely what you suggested-and no 
one has heard from him since. 
Achilles: That's unfortunate. Can you also carry along the bottle of pushing-
potion with you? 
Tortoise: Oh, certainly. Just hold it in your left hand, and it too will get 
pushed right along with you into the picture you're looking at. 
Achilles: What happens if you then find a picture inside the picture which you 
have already entered, and take another swig of pushing-potion? 
Tortoise: Just what you would expect: you wind up inside that picture-in-a-
picture. 
Achilles: I suppose that you have to pop twice, then, in order to extricate 
yourself from the nested pictures, and re-emerge back in real life. 
Tortoise: That's right. You have to pop once for each push, since a push takes 
you down inside a picture, and a pop undoes that.

picture. 
you down inside a picture, and a pop undoes that. 
Achilles: You know, this all sounds pretty fishy to me . . . Are you sure you're 
not just testing the limits of my gullibility? 
Tortoise: I swear! Look-here are two phials, right here in my pocket. 
(Reaches into his lapel pocket, and pulls out two rather large 
unlabeled phials, in one of which one can hear a red liquid sloshing 
around, and in the other of which one can hear a blue liquid sloshing 
around .) If you're willing, we can try them. What do you say?

around .) If you're willing, we can try them. What do you say? 
Achilles: Well, I guess, ahm, maybe, ahm ... Tortoise: Good! I knew you'd want to tr y it out. Shall we push ourselves into 
the world of Escher's Convex and Concave ? 
Achilles: Well, ah, .. . 
Tortoise: Then it's decided. Now we've got to remember to take along this 
flask of tonic, so that we can pop back out. Do you want to take that 
heavy responsibility, Achilles? 
Achilles: If it's all the same to you, I' m a little nervous, and I'd prefer letting 
you, with your experience, manage the operation. 
Tortoise: Very well, then.

Tortoise: Very well, then.  
(So saying, the Tortoise pours two sm all portions of pushing-potion. Then 
he picks up the flask of tonic and grasps it firmly in his right hand, and 
both he and Achilles lift their glasses to their lips.) 
 
Tortoise: Bottoms up! 
 
(They swallow.)

Little Harmonic Labyrinth 115
 
FIGURE 23. Convex and Concave, by M. C. Escher (lithograph, 1955). 
  Achilles: That's an exceedingly strange taste. Tortoise: One gets used to it. Achilles: Does taking the tonic feel this strange? Tortoise: Oh, that's quite 
another sensation. Whenever you taste the tonic, you feel a deep sense of satisfaction, as if you'd been waiting to taste it all your life. 
Achilles: Oh, I'm looking forward to  that. Tortoise: Well, Achilles, 
where are we? 
Achilles ( taking cognizance of his surroundings ): We're in a little gondola, 
gliding down a canal! I want to get out. Mr.Gondolier, please let us out here. 
 
(The gondolier pays no attention to this request.) 
 
Tortoise: He doesn't speak English. If we want to get out here, we'd better just 
clamber out quickly before he

Little Harmonic Labyrinth 116Enters the sinister “Tunnel of Love”; just ahead of us. 
 
(Achilles, his face a little pale scrambles out in a split second and then 
pulls his slower friend out.) 
 
Achilles: I didn't like the sound of that  place, somehow. I'm glad we got out 
here. Say, how do you know so much about this place, anyway? Have 
you been here before? 
Tortoise: Many times, although I always cam e in from other Escher pictures. 
They're all connected behind the frames, you know. Once you're in 
one, you can get to any other one. 
Achilles: Amazing! Were I not here, s eeing these things with my own eyes, 
I'm not sure I'd believe you. (The y wander out through a little arch.) 
Oh, look at those two cute lizards! 
Tortoise: Cute? They aren't cute-it ma kes me shudder just to think of them! 
They are the vicious guardians of  that magic copper lamp hanging 
from the ceiling over there. A mere  touch of their tongues, and any 
mortal turns to a pickle.

mortal turns to a pickle. 
Achilles: Dill, or sweet? Tortoise: Dill. Achilles: Oh, what a sour fate! But if  the lamp has magical powers, I would 
like to try for it. 
Tortoise: It's a foolhardy venture, my friend. I wouldn't risk it. Achilles: I'm going to try just once.  
(He stealthily approaches the lamp, making sure not to awaken the 
sleeping lad nearby. But suddenly, he slips on a strange shell-like 
indentation in the floor, and lunges out  into space. Lurching crazily, he

indentation in the floor, and lunges out  into space. Lurching crazily, he 
reaches for anything, and manages somehow to grab onto the lamp with one hand. Swinging wildly, with both lizards hissing and thrusting their 
tongues violently out at him, he is left dangling helplessly out in the middle 
of space.) 
 
Achilles: He-e-e-elp!  
(His cry attracts the attention of  a woman who rushes downstairs and 
awakens the sleeping boy. He takes stock of the situation, and, with a 
kindly smile on his face, gestures to Ac hilles that all will be well. He shouts

of space.) 
 
kindly smile on his face, gestures to Ac hilles that all will be well. He shouts 
something in a strange guttural tongue  to a pair of trumpeters high up in 
windows, and immediately,

Little Harmonic Labyrinth 117Weird tones begin ringing out and maki ng beats each other. The sleepy 
young lad points at the lizards, and Achille s sees that the music is having a 
strong soporific effect on them. Soon, they are completely unconscious. Then the helpful lad shouts to two companions climbing up ladders. They 
both pull their ladders up and then ext end them out into space just 
underneath the stranded Achilles, formi ng a sort of bridge. Their gestures 
make it clear that Achilles should hu rry and climb on. But before he does 
so, Achilles carefully unlinks the top link of the chain holding the lamp, and 
detaches the lamp. Then he climbs  onto the ladder-bridge and the three 
young lads pull him in to safety. Ac hilles throws his arms around them and 
hugs them gratefully.) 
 
Achilles: Oh, Mr. T, how can I repay them? Tortoise: I happen to know that these va liant lads just love coffee, and down 
in the town below, there's a place where they make an incomparable

in the town below, there's a place where they make an incomparable 
cup of espresso. Invite them for a cup of espresso! Achilles: That would hit the spot. 
 
(And so, by a rather comical series of gestures, smiles, and words, Achilles 
manages to convey his invitation to the young lads, and the party of five 
walks out and down a steep staircase de scending into the town. They reach 
a charming small cafe, sit down outside , and order five espressos. As they 
sip their drinks, Achilles remembers he has the lamp with him.)  
Achilles: I forgot, Mr. Tortoise-I've got this ma; lamp with me! But-what's

Achilles: I forgot, Mr. Tortoise-I've got this ma; lamp with me! But-what's 
magic about it? Tortoise: Oh, you know, just the usual-a genie. 
Achilles: What? You mean a genie comes out when you rub it, and grants you 
wishes? 
Tortoise: Right. What did you expect? Pennies fry heaven? Achilles: Well, this is fantastic! I can have any wish want, eh? I've always 
wished this would happen to me ... 
 
(And so Achilles gently rubs the larg e letter `L' which is etched on the 
lamp's copper surface ... Suddenly a huge puff of smoke appears, and in the forms of the smoke the five friends can make out a weird, ghostly figure towering above them.)

Little Harmonic Labyrinth 118I 
Genie: Hello, my friends – and thanks ever so much for rescuing my Lamp 
from the evil Lizard-Duo. 
 (And so saying, the Genie picks up the Lamp, and stuffs it into a pocket 
concealed among the folds of his long ghostly robe which swirls out of the Lamp.) 
 As a sign of gratitude for your heroic  deed, I would like to offer you, on the 
part of my Lamp, the opportunity to have any three of your wishes realized. 
Achilles: How stupefying! Don't you think so, Mr. T? Tortoise: I surely do. Go ahead, Achilles, take the first wish. Achilles: Wow! But what should I wish ? Oh, I know! It's what I thought of 
the first time I read the Arabian Ni ghts (that collection of silly (and 
nested) tales)-I wish that I had a HUNDRED wishes, instead of just 
three! Pretty clever, eh, Mr. T?  I bet YOU never would have thought 
of that trick. I always wondered w hy those dopey people in the stories 
never tried it themselves.

never tried it themselves. 
Tortoise: Maybe now you'll find out the answer. Genie: I am sorry, Achilles, but I don't grant metawishes. Achilles: I wish you'd tell me what a "meta-wish" is! Genie: But THAT is a meta-meta-wi sh, Achilles-and I don't grant them, 
either. Achilles: Whaaat? I don't follow you at all. 
Tortoise: Why don't you rephrase your last request, Achilles? Achilles: What do you mean? Why should I? 
Tortoise: Well, you began by saying "I wish". Since you're just asking for 
information, why don't you just ask a question?

information, why don't you just ask a question? 
Achilles: All right, though I don't see why. Tell me, Mr. Genie-what is a 
meta-wish? Genie: It is simply a wish about wishes. I am not allowed 
to grant meta-wishes. It is only within my purview to grant plain 
ordinary wishes, such as wishing fo r ten bottles of beer, to have Helen 
of Troy on a blanket, or to have an  all-expenses-paid weekend for two 
at the Copacabana. You know-simpl e things like that. But meta-
wishes I cannot grant. GOD won't permit me to. 
Achilles: GOD? Who is GOD? And why won't he let you grant meta-wishes? 
That seems like such a puny thing compared to the others you mentioned.

Little Harmonic Labyrinth 119 
Genie:  Well, it’s a complicated matte r, you see. Why don’t you just go ahead 
and make your three wishes? Or at  least make one of them. I don't 
have all I time in the world, you know ... 
Achilles: Oh, I feel so rotten. I was REALLY HOPING wish for a hundred 
wishes ... 
Genie: Gee, I hate to see anybody so disappointed that. And besides, meta-
wishes are my favorite k of wish. Let me just see if there isn't anything I do about this. This'll just take one moment 
 
(The Genie removes from the wispy fo lds of his robe an object which looks 
just like the copper Lamp he had put away , except that this one is made of 
silver; and where the previous one had 'L' etched on it, this one has 'ML' in smaller letters, so as to cover the same area.) 
I 
Achilles: And what is that? Genie: This is my Meta-Lamp ...  
(He rubs the Meta-Lamp, and a huge puff of smoke appears. In the billows 
of smoke, they can all make out a ghostly form towering above them.)

I 
of smoke, they can all make out a ghostly form towering above them.) 
 
Meta-Genie: I am the Meta-Genie. You summoned me, 0 Genie? What is 
your wish? 
Genie: I have a special wish to make  of you, 0 Djinn and of GOD. I wish for 
permission for tempos suspension of all type-restrictions on wishes, for duration of one Typeless Wish. Could you ph grant this wish for 
me? 
Meta-Genie: I'll have to send it thr ough Channels, of course. One half a 
moment, please 
 
(And, twice as quickly as the Genie did, this Meta-Genie removes from the

your wish? 
me? 
 
(And, twice as quickly as the Genie did, this Meta-Genie removes from the 
wispy folds of her robe an object whic h looks just like the silver Meta-
Lamp, except that it is made of gold;  and where the previous one had 'ML' 
etched on it, this one has 'MML' in smaller letters, so as to cover the same area.) 
 
Achilles ( his voice an octave higher than before ): And what is that? Meta-
Genie: This is my Meta-Meta-Lamp. . .  
 
(She rubs the Meta-Meta-Lamp, and a hugs puff of smoke appears. In the 
billows o smoke, they can all ma ke out a ghostly fore towering above 
them.)

Little Harmonic Labyrinth 120Meta-Meta-Genie: I am the MetaMeta-Genie. You summoned me, 
0 Meta-Genie? What is your wish? Meta-Genie: I have a special wish to  make of you, 0 Djinn, and of GOD. I 
wish for permission for temporary su spension of all type-restrictions 
on wishes, for the duration of one  Typeless Wish. Could you please 
grant this wish for me? 
Meta-Meta-Genie: I'll have to se nd it through Channels, of course. 
One quarter of a moment, please. 
 
(And, twice as quickly as the Meta-Genie did, this MetaMeta-Genie removes from the folds of his robe an object which looks 
just like the gold MetaLamp, except that it is made of ...) 
. 
   .       . 
          . 
                 . 
  . .
{GOD} 
                 .             .        . 
. 
( ... swirls back into the MetaMeta-Meta-Lamp, which the Meta-
Meta-Genie then folds back into his robe, half as quickly as the 
Meta-Meta-Meta-Genie did.) 
 
Your wish is granted, 0 MetaGenie. 
 Meta-Genie: Thank you, 0 Djinn, and GOD.

. 
   .       . 
          . 
                 . 
  . .
{GOD} 
                 .             .        . 
. 
 
 Meta-Genie: Thank you, 0 Djinn, and GOD. 
 
(And the Meta-Meta-Genie, as all the higher ones before him, swirls 
back into the Meta-Meta-Lamp, which the Meta-Genie then folds back 
into her robe, half as quickly as the Meta-Meta-Genie did.) 
 
Your wish is granted, 0 Genie.  
Genie: Thank you, 0 Djinn, and GOD.  
(And the Meta-Genie, as all the higher ones before her,

Little Harmonic Labyrinth 121swirls back into the Meta-Lamp, which the Genie folds back into his robe, 
half as quickly as the M Genie did.)  
Your wish is granted, Achilles. 
 
(And one precise moment has elapsed since he "This will just take one moment.") 
 
Achilles: Thank you, 0 Djinn, and GOD. Genie: I am pleased to report, Ach illes, that you r have exactly one (1) 
Typeless Wish-that is to sa wish, or  a meta-wish, or a meta-meta-wish, 
as many "meta"'s as you wish-even infinitely many (if wish). 
Achilles: Oh, thank you so very muc h, Genie. But curiosity is provoked. 
Before I make my wish, would you mind telling me who-or what-
GOD is? 
Genie: Not at all. "GOD" is an acr onym which stands "GOD Over Djinn". 
The word "Djinn" is used designate Genies, Meta-Genies, Meta-Meta-Gen etc. It is a Typeless word. 
Achilles: But-but-how can "GOD" be a word in own acronym? That doesn't 
make any sense! 
Genie: Oh, aren't you acquainted with recursive acronyms? I thought

GOD is? 
Genie: Oh, aren't you acquainted with recursive acronyms? I thought 
everybody knew about them. \ see, "GOD" stands for "GOD Over 
Djinn"-which can be expanded as  "GOD Over Djinn, O, Djinn"-and 
that can, in turn, be expanded to "G( Over Djinn, Over Djinn, Over 
Djinn"-which can its turn, be further expanded ... You can go as as you like. 
Achilles: But I'll never finish! 
Genie: Of course not. You can never totally expand GOD. Achilles: Hmm ... That's puzzling. What  did you me when you said to the 
Meta-Genie, "I have a sped wish  to make of you, 0 Djinn, and of 
GOD"?

GOD"? 
Genie: I wanted not only to make a request of Meta-Genie, but also of all the 
Djinns over her. 'I recursive acronym method accomplishes this qL naturally. You see, when the Meta -Genie received my request, she 
then had to pass it upwards to I GOD. So she forwarded a similar message to I Meta-Meta-Genie, who then did likewise to t Meta-Meta-
Meta-Genie ... Ascending the chain this way transmits the message to GOD.

Little Harmonic Labyrinth 122Achilles: I see. You mean GOD sits up at the top of the ladder of djinns? 
Genie: No, no, no! There is nothing "at th e top", for there is no top. That is 
why GOD is a recursive acronym. GOD is not some ultimate djinn; GOD is the tower of djinns above any given djinn. 
Tortoise: It seems to me that each a nd every djinn would have a different 
concept of what GOD is, then, sin ce to any djinn, GOD is the set of 
djinns above him or her, and no two djinns share that set. 
Genie: You're absolutely right-and sin ce I am the lowest djinn of all, my 
notion of GOD is the most exalted one. I pity the higher djinns, who 
fancy themselves somehow closer to GOD. What blasphemy!

fancy themselves somehow closer to GOD. What blasphemy! 
Achilles: By gum, it must have taken genies to invent GOD. Tortoise: Do you really believe all this stuff about GOD, Achilles? Achilles: Why certainly, I do. Are you atheistic, Mr. T? Or are you agnostic? Tortoise: I don't think I'm agnostic. Maybe I'm metaagnostic. Achilles: Whaaat? I don't follow you at all. Tortoise: Let's see . . . If I were meta-agnostic, I'd be confused over whether 
I'm agnostic or not-but I'm not quite sure if I feel THAT way; hence I 
must be meta-meta-agnostic (I guess) . Oh, well. Tell me, Genie, does 
any djinn ever make a mistake, a nd garble up a message moving up or

any djinn ever make a mistake, a nd garble up a message moving up or 
down the chain? 
Genie: This does happen; it is the mo st common cause for Typeless Wishes 
not being granted. You see, the chances are infinitesimal, that a garbling will occur at any PARTICUL AR link in the chain-but when 
you put an infinite number of them in a row, it becomes virtually certain that a garbling will occur SO MEWHERE. In fact, strange as it 
seems, an infinite number of garblings usually occur, although they 
are very sparsely distributed in the chain. 
Achilles: Then it seems a miracle that any Typeless Wish ever gets carried 
out. 
Genie: Not really. Most garblings ar e inconsequential, and many garblings 
tend to cancel each other out. But occasionally-in fact, rather seldom-the nonfulfillment of a Typeless Wish  can be traced back to a single

out. 
tend to cancel each other out. But occasionally-in fact, rather seldom-the nonfulfillment of a Typeless Wish  can be traced back to a single 
unfortunate djinn's garbling. When this happens, the guilty djinn is 
forced to run an infinite

Little Harmonic Labyrinth 123Gauntlet and get paddled on his or he r rump, by GOD. It's good fun for the 
paddlers, and q harmless for the pa ddlee. You might be amused by the 
sight. 
Achilles: I would love to see that! But it only happens when a Typeless Wish 
goes ungranted? 
Genie: That's right. Achilles: Hmm ... That gives me an idea for my w Tortoise: Oh, really? What 
is it? Achilles: I wish my wish would not be granted! 
 
(At that moment, an event-or is "event" the word for it? --takes place which 
cannot be described, and hence no attempt will be made to describe it.) 
 
Achilles: What on earth does that cryptic comment mean?  Tortoise: It refers to the Typeless Wish Achilles made.  Achilles: But he hadn't yet made it. Tortoise: Yes, he had. He said, "I wish my wish would not be granted", and the Genie took THAT to be his wish.  
(At that moment, some footsteps ar e heard coming down the hallway in 
their direction.) 
 
Achilles: Oh, my! That sounds ominous.

Achilles: Oh, my! That sounds ominous.  
(The footsteps stop; then they turn around and fade away.) 
 
Tortoise: Whew! 
Achilles: But does the story go on, let's see.  or is that the end? Turn the page 
and let’s see. 
 
(The Tortoise turns the page of "Dji nn and Tonic", where they find that the 
story goes on ...) 
 
Achilles: Hey! What happened? Wher e is my Genie: lamp? My cup of 
espresso? What happened to young friends from the Convex and 
Concave worlds? What are all those little lizards doing hi

Concave worlds? What are all those little lizards doing hi 
Tortoise: I'm afraid our context got restored incorrectly Achilles. Achilles: What on earth does that cryptic comment mean? Tortoise: I refer to the Typeless Wish you made. Achilles: But I hadn't yet made it. Tortoise: Yes, you had. You said, "I  wish my wish would not be 
granted", and the Genie took THAT to be your wish. 
Achilles: Oh, my! That sounds ominous. Tortoise; It spells PARADOX. For that Typeless wish to be

Little Harmonic Labyrinth 124granted, it had to be denied – yet not to grant it would be to grant 
it. 
Achilles: So what happened? Did the earth come to a standstill? Did the 
universe cave in? 
Tortoise: No. The System crashed. Achilles: What does that mean? Tortoise: It means that you and I, Achilles, were suddenly and 
instantaneously transported to Tumbolia. Achilles: To where? 
Tortoise: Tumbolia: the land of dead hiccups and extinguished light 
bulbs. It's a sort of waiting room , where dormant software waits 
for its host hardware to come back up. No telling how long the 
System was down, and we were in Tumbolia. It could have been 
moments, hours, days-even years. 
Achilles: I don't know what software is, and I don't know what hardware 
is. But I do know that I didn't get to make my wishes! I want my Genie back! 
Tortoise: I'm sorry, Achilles-you blew it. You crashed the System, and 
you should thank your lucky stars that  we're back at all. Things

you should thank your lucky stars that  we're back at all. Things 
could have come out a lot worse. But I have no idea where we are. 
Achilles: I recognize it now-we're insi de another of Escher's pictures. 
This time it's Reptiles . 
Tortoise: Aha! The System tried to save as much of our context as it 
could before it crashed, and it got as far as recording that it was an Escher picture with lizards before it went down. That's commendable. 
Achilles: And look-isn't that our phial of poppingtonic over there on the 
table, next to the cycle of lizards? 
Tortoise: It certainly is, Achilles. I must say, we are very lucky indeed.

Tortoise: It certainly is, Achilles. I must say, we are very lucky indeed. 
The System was very kind to us , in giving us back our popping-
tonic-it's precious stuff! 
Achilles: I'll say! Now we can pop back  out of the Escher world, into my 
house. 
Tortoise: There are a couple of books on the desk, next to the tonic. I 
wonder what they are. ( He picks up the smaller one, which is 
open to a random page. ) This looks like a moderately 
provocative book. 
Achilles: Oh, really? What is its title? Tortoise: Provocative Adventures of the Tortoise and Achilles Taking

house. 
Achilles: Oh, really? What is its title? Tortoise: Provocative Adventures of the Tortoise and Achilles Taking 
Place in Sundry Parts of the Globe . It sounds like an interesting 
book to read out of.

Little Harmonic Labyrinth 125
 
FIGURE 24. Reptiles, by M. C. Escher (lithograph, 1943 ). 
 Achilles: Well, You can read it if you want, but as for I'm not going to 
take any chances with t popping-t onic-one of the lizards might 
knock it off the table, so I'm going to get it right now! 
 
(He dashes over to the table and re aches for the popping-tonic, but in 
his haste he somehow bumps the flask of tonic, and it tumbles off the desk and begins rolling.) 
 
Oh, no! Mr. T-look! I accidentally knocked tonic onto the floor, and it's 
rolling toward towards-the stairwell! Quick-before it falls! 
 
(The Tortoise, however, is completely wrapped up in the thin volume 
which he has in his hands.)  Achilles: Well, You can read it if you 
want, but as for I'm not going to  take any chances with t popping-
tonic-one of the lizards might knoc k it off the table, so I'm going 
to get it right

Little Harmonic Labyrinth 126Tortoise ( muttering ):  Eh? This story looks fascinating. 
Achilles: Mr. T, Mr. T, help! Help catch the tonic-flask!  Tortoise: What's all the fuss about? Achilles: The tonic-flask-I knocked it down from the desk, and now it's 
rolling and 
 
(At that instant it reaches the brin k of the stairwell, and plummets 
over ... ) 
 
Oh no! What can we do? Mr. To rtoise-aren't you alarmed? We're 
losing our tonic! It's just fallen down the stairwell! There's only one 
thing to do! We'll have to go down one story! 
Tortoise: Go down one story? My pleasure. Won't you join me?  
(He begins to read aloud, and Achilles,  pulled in two directions at 
once, finally stays, taking the role of the Tortoise.) 
 
Achilles: It's very dark here, Mr. T. I can't see a thing. Oof! I bumped 
into a wall. Watch out! 
Tortoise: Here-I have a couple of walking sticks. Why don't you take one 
of them? You can hold it out in  front of you so that you don't 
bang into things.

bang into things. 
Achilles: Good idea. ( He takes the stick .) Do you get the sense that this 
path is curving gently to the left as we walk? Tortoise: Very slightly, yes. 
Achilles: I wonder where we are. And whether we'll ever see the light of 
day again. I wish I'd never listened to you, when you suggested I 
swallow some of that "DRINK ME" stuff. 
Tortoise: I assure you, it's quite harmle ss. I've done it scads of times, and 
not a once have I ever regretted it. Relax and enjoy being small.

not a once have I ever regretted it. Relax and enjoy being small. 
Achilles: Being small? What is it you've done to me, Mr. T? Tortoise: Now don't go blaming me. You did it of your own free will. 
Achilles: Have you made me shrink?  So that this labyrinth we're 
in is actually some teeny thing that someone could STEP on?

Little Harmonic Labyrinth 127
 
FIGURE 25. Cretan Labyrinth (Ita lian engraving; School of 
Finiguerra). [From N Matthews, Mazes and Labyrinths: Their History and Development (New York: Dover Publications, 1970). 
 
Tortoise: Labyrinth? Labyrinth? Could it Are we in the notorious Little 
Harmonic Labyrinth of the dreaded Majotaur? 
Achilles: Yiikes! What is that? Tortoise: They say-although I person never believed it myself-that an I 
Majotaur has created a tiny labyrinth sits in a pit in the middle of it, waiting innocent victims to get lost in its fears complexity. Then, when they wander and dazed into the center, he laughs and laughs at them-so hard, that he laughs them to death! 
Achilles: Oh, no! Tortoise: But it's only a myth. Courage, Achilles.  
 (And the dauntless pair trudge on.)  
 Achilles: Feel these walls. They're like o gated tin sheets, or something. 
But the corrugations have different sizes.

Little Harmonic Labyrinth 128(To emphasize his point, he sticks out his walking stick against the 
wall surface as he walks. As the stick bounces back and forth against the corrugations, strange noises echo up and down the long curved 
corridor they are in.) 
 
Tortoise (alarmed): What was THAT? Achilles: Oh, just me, rubbing my walking stick against the wall. Tortoise: Whew! I thought for a moment it was the bellowing of the 
ferocious Majotaur! Achilles: I thought you said it was all a myth. 
Tortoise: Of course it is. Nothing to be afraid of.  
(Achilles puts his walking stick back against the wall, and continues 
walking. As he does so, some musi cal sounds are heard, coming from 
the point where his stick is scraping the wall.) 
 
Tortoise: Uh-oh. I have a bad feeling, Achilles. That Labyrinth may not be a myth, after all. Achilles: Wait a minute. 
What makes you change your mind a ll of a sudden? Tortoise: Do 
you hear that music? 
 
(To hear more clearly, Achilles lowers the stick, and the strains of

(To hear more clearly, Achilles lowers the stick, and the strains of 
melody cease.) 
 
Hey! Put that back! I want to hear the end of this piece! 
 
(Confused, Achilles obeys, and the music resumes.) 
 
Thank you. Now as I was about to say, I have just figured out where 
we are.  Achilles: Really? Where are we?   Tortoise: We are walking down a spiral  groove of a record in its jacket. 
Your stick scraping against the strange shapes in the wall acts 
like a needle running down the gr oove, allowing us to hear the 
music.

music. 
Achilles: Oh, no, oh, no ... Tortoise: What? Aren't you overjoyed?  Have you ever had the chance to 
be in such intimate contact with music before?

Little Harmonic Labyrinth 129 
Achzltes: How am I ever going to win footraces against full-sized people 
when I am smaller than a flea, Mr. Tortoise? 
Tortoise: Oh, is that all that's bot hering you That's nothing to fret abopt, 
Achilles.  
Achilles: The way you talk, I get the impression that you never worry at 
all. 
Tortoise: I don't know. But one thing for certain is that I don't worry 
about being small. Especially not when faced with the awful 
danger of the dreaded Majotaur!  
Achilles: Horrors! Are you telling me Tortoise: I'm afraid so, Achilles. The music gave it away. Achilles: How could it do that? Tortoise: Very simple. When I heard melody B-A-C-H in the top voice, 
I immediately realized that the grooves we're walking through 
could only be Little Harmonic Labyrinth , one of Bach's er known 
organ pieces. It is so named cau se of its dizzyingly frequent 
modulations. 
Achilles: Wh-what are they? Tortoise: Well, you know that most music pieces are written in a key, or

Achilles: Wh-what are they? Tortoise: Well, you know that most music pieces are written in a key, or 
tonality, as C major, which is the key of this o; 
Achilles: I had heard the term before. Do that mean that C is the note 
you want to on? 
Tortoise: Yes, C acts like a home base, in a Actually, the usual word is 
"tonic". 
Achilles: Does one then stray away fr om tonic with the aim of eventually 
returning 
Tortoise: That's right. As the pi ece develops ambiguous chords and 
melodies are t which lead away  from the tonic. Little by little, 
tension builds up-you feel at creasi ng desire to return home, to

"tonic". 
returning 
tension builds up-you feel at creasi ng desire to return home, to 
hear the tonic. 
Achilles: Is that why, at the end of a pie always feel so satisfied, as if I 
had waiting my whole life to hear the ton 
Tortoise: Exactly. The composer has uses knowledge of harmonic 
progressions to

Little Harmonic Labyrinth 130 
manipulate your emotions, and to build up hopes in you to hear that tonic. 
Achilles: But you were going to tell me about modulations. Tortoise: Oh, yes. One very important thing a composer can do is to 
"modulate" partway through a piece, which means that he sets up a temporary goal other than resolution into the tonic. 
Achilles: I see ... I think. Do you mean  that some sequence of chords 
shifts the harmonic tension somehow so that I actually desire to resolve in a new key? 
Tortoise: Right. This makes the situ ation more complex, for although in 
the short term you want to resolve in the new key, all the while at 
the back of your mind you retain the longing to hit that original goal-in this case, C major. And when the subsidiary goal is reached, there is 
Achilles ( suddenly gesturing enthusiastically ): Oh, listen to the gorgeous 
upward-swooping chords which mark the end of this Little 
Harmonic Labyrinth! 
Tortoise: No, Achilles, this isn't the end. It's merely

Tortoise: No, Achilles, this isn't the end. It's merely 
Achilles: Sure it is! Wow! What a pow erful, strong ending! What a sense 
of relief! That's some resolution! Gee! 
 
(And sure enough, at that moment the music stops, as they emerge into 
an open area with no walls.) 
 
You see, it Is over. What did I te ll you? Tortoise: Something is very 
wrong. This record 
is a disgrace to the world of music. Achilles: What do you mean? Tortoise: It was exactly what I was telling you about. Here Bach had

is a disgrace to the world of music. Achilles: What do you mean? Tortoise: It was exactly what I was telling you about. Here Bach had 
modulated from C into G, setting up a secondary goal of hearing G. This means that you experien ce two tensions at once-waiting 
for resolution into G, but also keeping in mind that ultimate desire-to resolve triumphantly into C Major. 
Achilles: Why should you have to keep any

Little Harmonic Labyrinth 131thing in mind when listening to a piece of music? Is music only an 
intellectual exercise? 
Tortoise: No, of course not. Some mu sic is highly intellectual, but most 
music is not. And most of the time your ear or br the 
"calculation" for you, and lets your emotions know what they want to hear, don't have to thi nk about it consciously in this 
piece, Bach was playing tricks hoping to lead you astray. And in your case Achilles, he succeeded. 
Achilles: Are you telling me that I responded to a resolution in a 
subsidiary key? 
Tortoise: That's right. Achilles: It still sounded like an ending to me Tortoise: Bach intentionally made it sot way. You just fell into his trap. 
It was deliberately contrived to sound like an ending but if you follow the harmonic progression carefully, you will see that it is in the wrong key. Apparently not  just you but this miserable 
record company fell for the same trick-and they truncated the piece early.

record company fell for the same trick-and they truncated the piece early. 
Achilles: What a dirty trick Bach played Tortoise: That is his whole game-to m lose your way in his Labyrinth! 'l 
Majotaur is in cahoots with B ach, And if you don't watch out, he 
i laugh you to death-and perhaps n with you! 
Achilles: Oh, let us hurry up and get here! Quick! Let's run backwards 
grooves, and escape on the outside record before the Evil Majotaur finds us. 
Tortoise: Heavens, no! My sensibility is delicate to handle the bizarre the 
gressions which occur when time versed.

gressions which occur when time versed. 
Achilles: Oh, Mr. T, how will we ever get out of here, if we can't just 
retrace our steps 
Tortoise: That's a very good question.  
(A little desperately, Achilles starts runt about aimlessly in the dark. 
Suddenly t is a slight gasp, and then a "thud".)

Little Harmonic Labyrinth 132Achilles-are you all right? 
Achilles: Just a bit shaken up but otherwise fine. I fell into some big 
hole. 
Tortoise: You've fallen into the pit of the Evil Majotaur! Here, I'll come 
help you out. We've got to move fast! 
Achilles: Careful, Mr. T-I don't want You to fall in here, too ... Tortoise: Don't fret, Achilles. Everything will be all --  
(Suddenly, there is a slight gasp, and then a "thud".) 
 
Achilles: Mr. T-you fell in, too! Are you all right? Tortoise: Only my pride is hurt-otherwise I'm fine. Achilles: Now we're in a pretty pickle, aren't we?  
(Suddenly, a giant, booming laugh is  heard, alarmingly close to 
them.)

them.) 
 
Tortoise: Watch out, Achilles! This is no laughing matter. Majotaur: Hee hee hee! Ho ho! Haw haw haw! Achilles: I'm starting to feel weak, Mr. T ...  Tortoise: Try to pay no attention to his laugh, Achilles. That's your only hope. Achilles: I'll do my best. If only my stomach weren't empty! Tortoise: Say, am I smelling things, or is there a bowl of hot buttered 
popcorn around here? Achilles: I smell it, too. Where is it coming from? 
Tortoise: Over here, I think. Oh! I just  ran into a big bowl of the stuff. 
Yes, indeed-it seems to be a bowl of popcorn!

Yes, indeed-it seems to be a bowl of popcorn! 
Achilles: Oh, boy-popcorn! I'm going to munch my head off!  
Tortoise: Let's just hope it isn't pushcorn! Pushcorn and popcorn are 
extraordinarily difficult to tell apart.  Achilles: What's this about Pushkin? Tortoise: I didn't say a thing. You must be hearing things. Achilles: Go-golly! I hope not. Well, let's dig in!

Little Harmonic Labyrinth 133 
(And the two Jriends begin munc nai popcorn (or pushcorn?)-and t 
once POP! I guess it was popcorn; all.) 
 
Tortoise: What an amusing story. Did you en Achilles: Mildly. Only I wonder whether the' out of that Evil Majotaur's 
pit or r Achilles-he wanted to be full-sized again 
Tortoise: Don't worry-they're out, and he is again. That's what the "POP" 
was all abo 
Achilles: Oh, I couldn't tell. Well, now I REAL: find that bottle of tonic. 
For some reason, burning. And nothi ng would taste bett drink of 
popping-tonic. 
Tortoise: That stuff is renowned fo r its thirst powers. Why, in some 
places people very crazy over it. At  the turn of the century the 
Schonberg food factory stopped ma ] and started making cereal 
instead. You cai the uproar that caused. 
Achilles: I have an inkling. But le t's go look fo Hey just a moment. 
Those lizards on the you see anything funny about them? 
Tortoise: Umm ... not particularl y. What do you see of such great 
interest?

interest? 
Achilles: Don't you see it? They'r e emerging flat picture without 
drinking any pop] How are they able to do that? 
Tortoise: Oh, didn't I tell you? You can ge picture by moving 
perpendicularly to it you have no popping-tonic. The little li learned to climb UP when they want to ge two-dimensional sketchbook world. 
Achilles: Could we do the same thing to get Escher picture we're in? 
Tortoise: Of course! We just need to go UP one story. you want to try it? Achilles: Anything to get back to  my house! I all these provocative 
adventures.

adventures.  
Tortoise: Follow me, then, up this way.  
(And they go up one story.) 
 
Achilles: It's good to be back. But something seems wrong. This isn't my 
house! This is YOUR house, Mr. Tortoise 
Tortoise: Well, so it is-and am I glad for that! I wasn’t looking

Little Harmonic Labyrinth 134forward one whit to the long walk back from your house. I am bushed, 
and doubt if I could have made it. 
Achilles: I don't mind walking home, so I guess it's lucky we ended up 
here, after all. 
Tortoise: I'll say! This certainly is a piece of Good Fortune!

Recursive Structures and Processes 135Recursive Structures 
and Processes 
 
What Is Recursion? 
  WHAT IS RECURSION? It is what  was illustrated in the Dialogue Little Harmonic 
Labyrinth : nesting, and variations on nesting. The concept is very general. (Stories inside 
stories, movies inside movies, paintings in side paintings, Russian dolls inside Russian 
dolls (even parenthetical comments in. side pa renthetical comments!)-these are just a few 
of the charms of recursion.) However, you should he aware that the meaning of 
"recursive' in this Chapter is only faintly  related to its meaning in Chapter 111. The 
relation should be clear by the end of this Chapter. 
Sometimes recursion seems to brush para dox very closely. For example, there are 
recursive definitions . Such a definition may give the casual viewer the impression that 
something is being defined in terms of itself.  That would be circular and lead to infinite

something is being defined in terms of itself.  That would be circular and lead to infinite 
regress, if not to paradox proper. Actua lly, a recursive definition (when properly 
formulated) never leads to infinite regress or paradox. This is because a recursive definition never defines something in terms of itself, but always in terms of simpler 
versions  of itself. What I mean by this will b ecome clearer shortly, when ' show some 
examples of recursive definitions. 
One of the most common ways in which recursion appears in daily life is when

One of the most common ways in which recursion appears in daily life is when 
you postpone completing a task in favor of a si mpler task, often o the same type. Here is 
a good example. An executive has a fancy tele phone and receives many calls on it. He is 
talking to A when B calls. To A he sa y,, "Would you mind holding for a moment?" Of 
course he doesn't really car if A minds; he just pushes a button, and switches to B. Now C 
calls. The same deferment happens to B. This could go on indefinitely, but let us not get too bogged down in our enthusiasm. So let's say the call with C terminates. Then our

calls. The same deferment happens to B. This could go on indefinitely, but let us not get too bogged down in our enthusiasm. So let's say the call with C terminates. Then our 
executive "pops" back up to B, and continues. Meanwhile A is sitting at the other end of the line, drumming his fingernails again so me table, and listening to some horrible 
Muzak piped through the phone lines to placate him ... Now the easiest case is if the call 
with B simply terminates, and the executive returns to A finally. But it could  happen that 
after the conversation with B is resumed, a new caller-D-calls. B is once again pushed 
onto the stack of waiting callers, and D is take n care of. Aft D is done, back to B, then

onto the stack of waiting callers, and D is take n care of. Aft D is done, back to B, then 
back to A. This executive is hopelessly mech anical, to be sure-but we are illustrating 
recursion in its most precise form

Recursive Structures and Processes 136Pushing, Popping, and Stacks 
 
In the preceding example, I have introduced so me basic terminology of recursion-at least 
as seen through the eyes of computer scientists. The terms are push, pop , and stack  (or 
push-down stack , to be precise) and they are all related. They were introduced in the late 
1950's as part of IPL, one of  the first languages for Artificial Intelligence. You have 
already encountered "push" and "pop" in the Dialogue. But I will spell things out 
anyway. To push  means to suspend operations on the task you're currently working on, 
without forgetting where you are-and to take up a new task. The new task is usually said 
to be "on a lower level" than the earlier task. To pop is the reverse-it means to close 
operations on one level, and to resume opera tions exactly where you left off, one level 
higher. 
But how do you remember exactly where you were on each different level? The

higher. 
But how do you remember exactly where you were on each different level? The 
answer is, you store the relevant information in a stack . So a stack is just a table telling 
you such things as (1) where you were in each unfinished task (jargon: the "return 
address"), (2) what the relevant facts to know  were at the points of interruption (jargon: 
the "variable bindings"). When you pop back up to resume some task, it is the stack 
which restores your context, so you don't feel  lost. In the telephone-call example, the 
stack tells you who is waiting on each different level, and where  you were in the 
conversation when it was interrupted.

conversation when it was interrupted. 
By the way, the terms "push", "pop", and "s tack" all come from the visual image 
of cafeteria trays in a stack. There is usually  some sort of spring underneath which tends 
to keep the topmost tray at a constant hei ght, more or less. So when you push a tray onto 
the stack, it sinks a little-and when you remove a tray from the stack, the stack pops up a 
little. 
One more example from daily life. When  you listen to a news report on the radio, 
oftentimes it happens that they switch you to some foreign correspondent. "We now

little. 
oftentimes it happens that they switch you to some foreign correspondent. "We now 
switch you to Sally Swumpley in Peafog, Engl and." Now Sally has got a tape of some 
local reporter interviewing someone, so after giving a bit of background, she plays it. "I'm 
Nigel Cadwallader, here on scene just outside of Peafog, where the great robbery took place, and I'm talking with ..." Now you are th ree levels down. It may turn out that the 
interviewee also plays a tape of some c onversation. It is not too uncommon to go down 
three levels in real news reports, and surprisingly enough, we scarcely have any

three levels in real news reports, and surprisingly enough, we scarcely have any 
awareness of the suspension. It is all kept track of quite easily by our subconscious mind. 
Probably the reason it is so easy is that each le vel is extremely different in flavor from 
each other level. If they were all similar, we would get confused in no time flat. 
An example of a more complex recursi on is, of course, our Dialogue. There, 
Achilles and the Tortoise appeared on all th e different levels. Sometimes they were 
reading a story in which they appeared as characters. That is when your mind may get a

reading a story in which they appeared as characters. That is when your mind may get a 
little hazy on what's going on, and you have to con centrate carefully to get things straight. 
"Let's see, the real Achilles and Tortoise are still up there in Goodfortune's helicopter, but the

Recursive Structures and Processes 137secondary  ones are in some Escher picture-and then they found this book and are reading 
in it, so it's the tertiary  Achilles and Tortoise who wa ndering around inside the grooves 
of the Little Harmonic Labyrinth . wait a minute-I left out one level somewhere ..." You 
have to ha conscious mental stack like this in order to keep track of the recursion the 
Dialogue. (See Fig. 26.)  
 
FIGURE 26. Diagram of the st ructure of the Dialogue Little Harmonic Labyrinth  
Vertical descents are "pushes"; rises ore "pops ". Notice the similarity of this diagram to 
indentation pattern of the Dialogue. From the diagram it is clear that the initial tension 
Goodfortune's threat-never was resolved; Achilles and the Tortoise were just left 
dangling the sky. Some readers might agonize  over this unpopped push, while others 
might not ba eyelash. In the story, Bach's mu sical labyrinth likewise was cut off too soon-

but Achilles d even notice anything funny. Only  the Tortoise was aware of the more 
global dangling tension 
 
Stacks in Music 
 
While we're talking about the Little Harmonic Labyrinth, we should discuss 
something which is hinted at, if not stated  explicitly in the Dialogue: that hear music 
recursively-in particular, that we maintain a mental stack of keys, and that each new 
modulation pushes a new key onto the stack. impli cation is further that we want to hear 
that sequence of keys retrace reverse orde r-popping the pushed keys off the stack, one by

that sequence of keys retrace reverse orde r-popping the pushed keys off the stack, one by 
one, until the tonic is reached. This is an exaggeration. There is a grain of truth to it however. 
Any reasonably musical person automatically maintains a shallow with two keys. 
In that "short stack", the true tonic key is  held and also most immediate "pseudotonic" 
(the key the composer is pretending t in). In other words, the most global key and the 
most local key. That the listener knows when th e true tonic is regained, and feels a strong

most local key. That the listener knows when th e true tonic is regained, and feels a strong 
s of "relief". The listener can also distinguish (unlike Achilles) between a local  easing of 
tension-for example a resolution into the pseudotonic --

Recursive Structures and Processes 138and a global  resolution. In fact, a pseudoresoluti on should heighten the global tension, 
not relieve it, because it is a piece of irony-ju st like Achilles' rescue from his perilous 
perch on the swinging lamp, when all the while  you know he and the Tortoise are really 
awaiting their dire fates at the knife of Monsieur Goodfortune. 
Since tension and resolution are the heart and soul of music, there are many, many 
examples. But let us just look at a coupl e in Bach. Bach wrote many pieces in an 
"AABB" form-that is, where there are two halves, and each one is repeated. Let's take the gigue from the French Suite no. 5, which is quite  typical of the form. Its tonic key is G, 
and we hear a gay dancing melody which establishes the key of G strongly. Soon, 
however, a modulation in the A-section lead s to the closely related key of D (the 
dominant). When the A-section ends, we are in the key of D. In fact , it sounds as if the

dominant). When the A-section ends, we are in the key of D. In fact , it sounds as if the 
piece has ended in the key of D! (Or at l east it might sound that way to Achilles.) But 
then a strange thing happens-we abruptly ju mp back to the beginning, back to G, and 
rehear the same transition into D. But then  a strange thing happens-we abruptly jump 
back to the beginning, back to G, and rehear the same transition into D. 
Then comes the B-section. With the inve rsion of the theme for our melody, we 
begin in D as if that had always been the t onic-but we modulate back to G after all, which

begin in D as if that had always been the t onic-but we modulate back to G after all, which 
means that we pop back into the tonic, and the B-section ends properly. Then that funny 
repetition takes place, jerking us without warni ng back into D, and letting us return to G 
once more. Then that funny repetition takes place, jerking us without warning 
back into D, and letting us return to G once more. The psychological effect of all this key shifting-some jerky, some smooth-is very 
difficult to describe. It is part of the magic of music that we can automatically make sense

difficult to describe. It is part of the magic of music that we can automatically make sense 
of these shifts. Or perhaps it is the magic of Bach that he can write pieces with this kind 
of structure which have such a natural grace to them that we are not aware of exactly 
what is happening. 
The original Little Harmonic Labyrinth  is a piece by Bach in which he tries to 
lose you in a labyrinth of quick key changes.  Pretty soon you are so disoriented that you 
don't have any sense of direction left-you don't know where the true tonic is, unless you

don't have any sense of direction left-you don't know where the true tonic is, unless you 
have perfect pitch, or like Theseus, have a friend like Ariadne who gives you a thread that 
allows you to retrace your steps. In this case,  the thread would be a written score. This 
piece-another example is the Endlessly Ri sing Canon-goes to show that, as music 
listeners, we don't have very reliable deep stacks. 
  
Recursion in Language 
 
Our mental stacking power is perhaps sli ghtly stronger in language. The grammatical 
structure of all languages involves setting up quite elaborate push-down stacks, though, to

structure of all languages involves setting up quite elaborate push-down stacks, though, to 
be sure, the difficulty of understanding a sent ence increases sharply with the number of 
pushes onto the stack. The proverbial Germ an phenomenon of the "verb-at-the-end", 
about which

Recursive Structures and Processes 139Droll tales of absentminded professors w ho would begin a sentence, ramble on for 
an entire lecture, and then finish up by rattling off a string of verbs by which their 
audience, for whom the stack had long sin ce lost its coherence, would be totally 
nonplussed, are told, is an excellent ex ample of linguistic pushing and popping. The 
confusion among the audience out-of-orde r popping from the stack onto which the 
professor's verbs been pushed, is amusing to imagine, could engender. But in normal ken 
German, such deep stacks almost never occu r-in fact, native speaker of German often 
unconsciously violate certain conventions which force verb to go to the end, in order to 
avoid the mental effort of keeping track of  the stack. Every language has constructions 
which involve stacks, though usually of a less spectacular nature than German. But there 
are always of rephrasing sentences so that the depth of stacking is minimal.  
Recursive Transition Networks

Recursive Transition Networks 
 
The syntactical structure of sentences affords a good place to present a of describing recursive structures and processes: the Recursive Transition Network  (RTN ). An RTN  is 
a diagram showing various paths which can be  followed to accomplish a particular task. 
Each path consists of a number of nodes , or little boxes with words in them, joined by 
arcs, or lines with arrows. The overall name for the RTN  is written separately at the left, 
and the and last nodes have the words begin  and end in them. All the other nodes contain

and the and last nodes have the words begin  and end in them. All the other nodes contain 
either very short explicit directions  to perform, or else name other RTN' s. Each time you 
hit a node, you are to carry out the direct inside it, or to jump to the RTN  named inside it, 
and carry it out. 
Let's take a sample RTN , called ORNATE NOUN , which tells how to construct 
a certain type of English noun phrase. (See Fig. 27a.) If traverse ORNATE NOUN  
purely horizontally, we begin' , then we create ARTICLE , an ADJECTIVE , and a 
NOUN , then we end. For instance, "the shampoo" or "a thankless brunch". But the arcs 
show other possibilities such as skipping the ar ticle, or repeating the adjective. Thus we

show other possibilities such as skipping the ar ticle, or repeating the adjective. Thus we 
co construct "milk", or "big red blue green sneezes", etc. 
When you hit the node NOUN , you are asking the unknown black I called NOUN  
to fetch any noun for you from its storehouse of nouns. This is known as a procedure 
call, in computer science terminology. It means you temporarily give control to a 
procedure  (here, NOUN ) which (1) does thing (produces a noun) and then (2) hands 
control back to you. In above RTN , there are calls on three such procedures: ARTICLE , 
ADJECTIVE  and NOUN . Now the RTN  ORNATE NOUN  could itself be called from

ADJECTIVE  and NOUN . Now the RTN  ORNATE NOUN  could itself be called from 
so other RTN -for instance an RTN  called SENTENCE . In this case, ORNATE NOUN  
would produce a phrase such as "the silly shampoo" and d return to the place inside 
SENTENCE  from which it had been called. I quite  reminiscent of the way in which you 
resume where you left off nested telephone calls or nested news reports. 
However, despite calling this a "recursive transition network", we have

Recursive Structures and Processes 140
 
FIGURE 27. Recursive Transition Networks for ORNATE NOUN  and FANCY NOUN . 
 
not exhibited any true recursion so far. Thi ngs get recursive-and seemingly circular-when 
you go to an RTN  such as the one in Figure 27b, for FANCY NOUN . As you can see, 
every possible pathway in FA NCY NOUN  involves a call on ORNATE NOUN , so there 
is no way to avoid getting a noun of some sort or other. And it is possible to be no more 
ornate than that, coming out merely with "m ilk" or "big red blue green sneezes". But 
three of the pathways involve recursive  calls on FANCY NOUN  itself. It certainly looks 
as if something is being defined in terms of itself. Is that what is happening, or not? 
The answer is "yes, but benignly" . Suppose that, in the procedure SENTENCE , 
there is a node which calls FANCY NOUN , and we hit that node. This means that we 
commit to memory (viz., the stack) the location of that node inside SENTENCE , so we'll

commit to memory (viz., the stack) the location of that node inside SENTENCE , so we'll 
know where to return to-then we transfer our attention to the procedure FANCY NOUN . 
Now we must choose a pathway to take, in order to generate a FANCY NOUN . Suppose 
we choose the lower of the upper pathways-the one whose calling sequence goes: 
 
ORNATE NOUN; RELATIVE PR ONOUN; FANCY NOUN; VERB.

Recursive Structures and Processes 141So we spit out an ORNATE NOUN : "the strange bagels "; a RELATIVE NOUN : 
"that"; and now we are suddenly asked for a FANCY NOUN . B are in the middle of 
FANCY NOUN ! Yes, but remember our executive was in the middle of one phone call 
when he got another one. He n stored the ol d phone call's status on a stack, and began the 
new one nothing were unusual. So we shall do the same. 
We first write down in our stack the node we are at in the outer call on FANCY 
NOUN , so that we have a "return addr ess"; then we jump t beginning of FANCY NOUN  
as if nothing were unusual. No w we h~ choose a pathway again. For variety's sake, let's 
choose the lower pat] ORNATE NOUN; PREPOSITION; FANCY NOUN . That 
means we produce an ORNATE NOUN  (say " the purple cow "), then a PREPOSITION  
(say “ without "), and once again, we hit the recursion. So we hang onto our hats descend

(say “ without "), and once again, we hit the recursion. So we hang onto our hats descend 
one more level. To avoid complexity, let's assume that this the pathway we take is the direct one just ORNATE NOUN.  For example: we might get " horns ". We hit the node 
END  in this call on FANCY NOUN  which amounts to popping out, and so we go to our 
stack to find the return address. It tells us that we were in the middle of executing 
FANCY NOUN  one level up-and so we resume there. This yields " the purple cow 
without horns ". On this level, too, we hit END, and so we pop up once more, this finding

without horns ". On this level, too, we hit END, and so we pop up once more, this finding 
ourselves in need of a VERB -so let's choose " gobbled ". This ends highest-level call on 
FANCY NOUN , with the result that the phrase 
 
"the strange bagels that the purple cow without horns gobbled" 
 
will get passed upwards to the patient SENTENCE , as we pop for the last time. 
As you see, we didn't get into any infin ite regress. The reason is tl least one 
pathway inside the RTN FANCY NOUN  does not involve recursive calls on FANCY

pathway inside the RTN FANCY NOUN  does not involve recursive calls on FANCY 
NOUN  itself. Of course, we could have perv ersely insisted on always choosing the 
bottom pathway inside FANCY NOUN  then we would never have gotten finished, just 
as the acronym " GOD ” never got fully expanded. But if the pathways are chosen at 
random, an infinite regress of that sort will not happen. 
 
"Bottoming Out" and Heterarchies 
 This is the crucial fact which distinguish es recursive definitions from circular 
ones. There is always some part of the defi nition which avoids reference, so that the

ones. There is always some part of the defi nition which avoids reference, so that the 
action of constructing an object which satisfi es the definition will eventually "bottom 
out". 
Now there are more oblique ways  of achieving recursivity in RTN s than by self-
calling. There is the analogue of Escher's Drawing  (Fig. 135), where each of two 
procedures calls the other, but not its elf. For example, we could have an RTN  named 
CLAUSE , which calls FANCY NOUN  whenever it needs an object for a transitive verb, 
and conversely, the u path of FANCY NOUN  could call RELATIVE PRONOUN  and 
then CLAUSE

Recursive Structures and Processes 142whenever it wants a relative clause. This is an example of indirect recursion. It is 
reminiscent also of the two-step version of the Epimenides paradox. 
Needless to say, there can be a trio of procedures which call one another, 
cyclically-and so on. There can be a whole family of RTN 's which are all tangled up, 
calling each other and themselves like crazy. A program which has such a structure in which there is no single "highest level", or "monitor", is called a heterarchy (as 
distinguished from a hierarchy). The term is  due, I believe, to Warren McCulloch, one of 
the first cyberneticists, and a reverent student of brains and minds.  
Expanding Nodes 
 One graphic way of thinking about RTN 's is this. Whenever you are moving along some 
pathway and you hit a node which calls on an RTN , you "expand" that node, which 
means to replace it by a very small copy of the RTN  it calls (see Fig. 28). Then you 
proceed into the very small RTN ,

proceed into the very small RTN , 
 
FIGURE 28. The FANCY NOUN RTN  with one node recursively expanded  
 When you pop out of it, you are automatically in the right place in the big one. While in the small one, you may wind up constructing even more miniature RTN 's. But by 
expanding nodes only when you come across th em, you avoid the need to make an 
infinite diagram, even when an RTN  calls itself. 
Expanding a node is a little like replacing a letter in an acronym by the word it 
stands for. The " GOD " acronym is recursive but has the defect-or advantage-that you

When you pop out of it, you are automatically in the right place in the big one. While in the small one, you may wind up constructing even more miniature RTN 's. But by 
stands for. The " GOD " acronym is recursive but has the defect-or advantage-that you 
must repeatedly expand the `G'; t hus it never bottoms out. When an RTN  is implemented 
as a real computer program, however, it alwa ys has at least one pathway which avoids 
recursivity (direct or indirect) so that infi nite regress is not created. Even the most 
heterarchical program structure bottoms out-oth erwise it couldn't run! It would just be 
constantly expanding node after node, but never performing any action.

Recursive Structures and Processes 143Diagram G and Recursive Sequences 
 
Infinite geometrical structures can be defined in just this way-that is by expanding 
node after node. For example, let us define an  infinite diagram called "Diagram G". To 
do so, we shall use an implicit representati on. In two nodes, we shall write merely the 
letter `G', which, however, will stand for an  entire copy of Diagram G. In Figure 29a, 
Diagram G is portrayed implicitly. Now if we wish to see Diagram G more explicitly, we 
expand each of the two G's-that is, we replace them by the same diagram , only reduced 
in scale (see Fig. 29b). This "second-order" ve rsion of Diagram gives us an inkling of 
what the final, impossible-to-realize Diagram G really looks like. In Figure 30 is shown a 
larger portion of Diagram G, where all the nodes have been numbered from the bottom up, and from left to right. Two extra nodes-num bers -- 1 and 2--- have been inserted at 
the bottom

the bottom 
This infinite tree has some very curious mathematical properties Running up its 
right-hand edge is the famous sequence of Fibonacci numbers . 
 1,    1,   2,   3,   5,   8,   13,   21,   34,   55,   89,    144,     233,  
discovered around the year 1202 by Leonardo of Pisa, son of Bonaccio, ergo "Filius 
Bonacci", or "Fibonacci" for short. These numbers are best  FIGURE 29.  (a) Diagram G, unexpanded.  (c) Diagram H, unexpanded 
(b) Diagram G, expanded once.  (d) Diagram H, expanded once

Recursive Structures and Processes 144
 
FIGURE 30. Diagram G, further expanded and with numbered nodes. 
 
defined recursively by the pair of formulas  FIBO (n) = FIBO (n- 1) + FIBO (n-2)    for n > 2  
 
FIBO (l) = FIBO (2) = 1 
 
Notice how new Fibonacci numbers are defined in terms of previous Fibonacci numbers. 
We could represent this pair of formulas in an RTN  (see Fig. 31). 
 
FIGURE 31. An RTN  for Fibonacci numbers. 
 
Thus you can calculate FIBO (15) by a sequence of recursive calls on the procedure 
defined by the RTN  above. This recursive definition bottoms out when you hit FIBO (1) 
or FIBO (2) (which are given explicitly) afte r you have worked your way backwards 
through descending values of n. It is sligh tly awkward to work your way backwards, 
when you could just as well work your way forwards, starting with FIBO (l) and FIBO (2) 
and always adding the most recent two values, until you reach FIBO (15). That way you 
don't need to keep track of a stack.

don't need to keep track of a stack. 
Now Diagram G has some even more surp rising properties than this. Its entire 
structure can be coded up in a single recursive definition, as follows:

Recursive Structures and Processes 145G(n) = n - G(G(n- 1)) for n > 0  
 
G(O) = 0 
 
How does this function G(n) code for the tree-structure? Quite simply you construct a 
tree by placing G(n) below n, for all values of n, you recreate Diagram G. In fact, that is 
how I discovered Diagram G in the place. I was investigating the function  G, and in 
trying to calculate its values quickly, I con ceived of displaying the values I already knew 
in a tree. T surprise, the tree turned out to have this extremely orderly recursive 
geometrical description. 
What is more wonderful is that if you make the analogous tree function H(n) 
defined with one more nesting than G— 
 
H(n) = n - H(H(H (n - 1))) for n > 0  
 
H(0) = 0 
 
--then the associated "Diagram H" is defined implicitly as shown in Figure 29c. The 
right-hand trunk contains one more node; that  is the difference. The first recursive 
expansion of Diagram H is shown in Figure 29d. And so it goes, for any degree of

H(0) = 0 
 
expansion of Diagram H is shown in Figure 29d. And so it goes, for any degree of 
nesting. There is a beautiful regularity to the recursive geometrical structures, which corresponds precisely to the recursive algebraic definitions. 
A problem for curious readers is: suppose you flip Diagram G around as if in a 
mirror, and label the nodes of the new tree so  they increase left to right. Can you find a 
recursive algebraic  definition for this "flip-tree. What about for the "flip" of the H-tree? 
Etc.? 
Another pleasing problem involves a pair of recursively intertwined functions

Etc.? 
Another pleasing problem involves a pair of recursively intertwined functions 
F(n) and M(n) -- "married" functions, you might say -- defined this way: 
 F(n) = n - M(F(n- 1))  
      F o r  n > 0 
M(n) = n - F(M(n- 1))  
 F(0) = 1, and M(0) = 0 
 The RTN 's for these two functions call each other and themselves as well. The 
problem is simply to discover the recursive structures of Diagram F; and Diagram M. 
They are quite elegant and simple. 
 
A Chaotic Sequence 
 One last example of recursion in number th eory leads to a small my Consider the

      F o r  n > 0 
 
 One last example of recursion in number th eory leads to a small my Consider the 
following recursive definition of a function: 
 
Q(n) = Q(n - Q(n- 1)) + Q(n - Q(n-2)) for n > 2  
Q(1) =  Q(2) = 1.

Recursive Structures and Processes 146It is reminiscent of the Fibonacci definiti on in that each new value is a sum of two 
previous values-but not of the immediately  previous two values. Instead, the two 
immediately previous values tell how far to c ount back to obtain the numbers to be added 
to make the new value! The first 17 Q-numbers run as follows: 
 1,  1,  2,  3,  3,  4,  5,  5,  6,  6,  6,   8,  8,   8,  10,  9,  10, . . . .  .. .                              5 + 6 = 11                            how far to move to the left                                     New term  To obtain the next one, move leftwards (from the three dots) respectively 10 and 9 terms; 
you will hit a 5 and a 6, shown by the arrows. Their sum-1 l-yields the new value: Q(18). 
This is the strange process by which the list of known Q-numbers is used to extend itself. 
The resulting sequence is, to put it mildly, e rratic. The further out you go, the less sense it

The resulting sequence is, to put it mildly, e rratic. The further out you go, the less sense it 
seems to make. This is one of those very  peculiar cases where what seems to be a 
somewhat natural definition leads to extremely puzzling behavior: chaos produced in a very orderly manner. One is naturally led to  wonder whether the apparent chaos conceals 
some subtle regularity. Of course, by definition, there is regularity, but what is of interest 
is whether there is another way of char acterizing this sequence-and with luck, a 
nonrecursive way.  
Two Striking Recursive Graphs 
 The marvels of recursion in mathematics ar e innumerable, and it is not my purpose to

 The marvels of recursion in mathematics ar e innumerable, and it is not my purpose to 
present them all. However, there are a couple of particularly striking examples from my own experience which I feel are worth presen ting. They are both graphs. One came up in 
the course of some number-theoretical inves tigations. The other came up in the course of 
my Ph.D. thesis work, in solid state physics. Wh at is truly fascinating is that the graphs 
are closely related. 
The first one (Fig. 32) is a gr aph of a function which I call INT(x). It is plotted

The first one (Fig. 32) is a gr aph of a function which I call INT(x). It is plotted 
here for x between 0 and 1. For x between any other pair of integers n and n + 1, you just 
find INT(x-n), then add n back. The structure of the plot is quite jumpy, as you can see. It 
consists of an infinite number of curved pieces, which get smaller and smaller towards the corners-and incidentally, less and less cu rved. Now if you look closely at each such 
piece, you will find that it is actually a c opy of the full graph, merely curved! The 
implications are wild. One of them is that the graph of INT consists of nothing but copies

implications are wild. One of them is that the graph of INT consists of nothing but copies 
of itself, nested down infinitely deeply. If you pick up any piece of the graph, no matter 
how small, you are holding a complete copy of the whole graph-in fact, infinitely many 
copies of it!  The fact that INT consists of nothing but c opies of itself might make you think it is too 
ephemeral to exist. Its definition sounds too circular.

Recursive Structures and Processes 147 
 
FIGURE 32. Graph of the function INT(x). There is a jump discontinuity at every rat 
value of x. 
 
How does it ever get off the ground? That is a very interesting matter. main thing to 
notice is that, to describe INT to someone w ho hasn't see it will not suffice merely to say, 
"It consists of copies of itself." The o ha lf of the story-the nonrecursive half-tells where  
those copies lie in the square, and how they have been deformed, relative to the full 
graph. Only the combination of these two aspect s of INT will specify structure of INT. It 
is exactly as in the definition of Fibon acci number where you need two lines-one to 
define the recursion , the other to de the bottom  (i.e., the values at the beginning). To be 
very concrete, if make one of the botto m values 3 instead of 1, you will produce a 
completely different sequence, known as the Lucas sequence:  
1,   3,   4 ,   7,    11,    18,    29,    47,    76,    123, .. .

1,   3,   4 ,   7,    11,    18,    29,    47,    76,    123, .. .  
            the "bottom"                         29 + 47 = 76 
                                                         same recursive rule 
                                                         as for the Fibonacci numbers

Recursive Structures and Processes 148What corresponds to the bottom  in the definition of IN T is a picture (Fig. 33a) 
composed of many boxes, showing where  the copies go, and how they are distorted. I call 
it the "skeleton" of INT. To construct INT from its skeleton, you do the following. First, 
for each box of the skeleton, you do two operations: (1) put a small curved copy of the skeleton inside the box, using the curved line inside it as a guide; (2) erase the containing box and its curved line. Once this has been  done for each box of the original skeleton, 
you are left with many "baby" skeletons in  place of one big one. Next you repeat the 
process one level down, with all the baby skel etons. Then again, again, and again ... What 
you approach in the limit is an exact graph of INT, though you never get there. By 
nesting the skeleton inside itself over and over again, you gradually construct the graph 
of INT "from out of nothing". But in fact the "nothing" was not nothing-it was a picture.

of INT "from out of nothing". But in fact the "nothing" was not nothing-it was a picture. 
To see this even more dramatically, im agine keeping the recursive part of the 
definition of INT, but changing the initial picture, the skeleton. A variant skeleton is 
shown in Figure 33b, again with boxes which get smaller and smaller as they trail off to 
the four corners. If you nest this second skeleton inside itself over and over again, you 
will create the key graph from my Ph.D. thesis, which I call Gplot  (Fig. 34). (In fact, 
some complicated distortion of each copy is needed as well-but nesting is the basic idea.).

some complicated distortion of each copy is needed as well-but nesting is the basic idea.). 
Gplot is thus a member of the INT-family. It is a distant relative, because its 
skeleton is quite different from-and considerably more complex than-that of INT. However, the recursive part of the definition is identical, and therein lies the family tie. I should not keep you too much in the dark a bout the origin of these beautiful graphs. 
INT-standing for "interchange"-comes from a problem involving "Eta-sequences", which 
are related to continued fractions. The basic idea behind INT is that plus and minus signs

are related to continued fractions. The basic idea behind INT is that plus and minus signs 
are interchanged in a certain kind of continued fraction. As a consequence, INT(INT (x)) 
= x. INT has the property that if x is rational, so is INT(x); if x is quadratic, so is INT(x). 
I do not know if this trend holds for higher al gebraic degrees. Another lovely feature of 
INT is that at all rational values of x, it has a jump discontinuity, but at all irrational 
values of x, it is continuous. 
Gplot comes from a highly idealized version of the question, "What are the 
allowed energies of electrons in a crystal in a magnetic field?" This problem is interesting

allowed energies of electrons in a crystal in a magnetic field?" This problem is interesting 
because it is a cross between two very simple and fundamental physical situations: an 
electron in a perfect crystal, and an elect ron in a homogeneous magnetic field. These two 
simpler problems are both well understood, and th eir characteristic solutions seem almost 
incompatible with each other. Therefore, it is  of quite some interest to see how nature 
manages to reconcile the two. As it happens, the crystal without-magnetic-field situation

manages to reconcile the two. As it happens, the crystal without-magnetic-field situation 
and the magnetic-field-without-crystal situation do have one feature in common: in each of them, the electron behaves periodically in  time. It turns out that when the two 
situations are combined, the ratio of their tw o time periods is the key parameter. In fact, 
that ratio holds all the information about the distribution of allowed electron energies-but 
it only gives up its secret upon being expanded into a continued fraction.

Recursive Structures and Processes 150Gplot shows that distribution. The horizontal axis represents energy, and the 
vertical axis represents the above-mentione d ratio of time periods, which we can call " α". 
At the bottom, a is zero, and at the top a is unity. When a is zero, there is no magnetic 
field. Each of the line segments making up Gplo t is an "energy band"-that is, it represents 
allowed values of energy. The empty swaths traversing Gplot on all different size scales 
are therefore regions of forbidden energy. One of the most startling properties of Gplot is 
that when a is rational (say p/q in lowest terms), there are exactly q such bands (though 
when q is even, two of them "ki ss" in the middle). And when a is irrational, the bands 
shrink to points, of which there are infinite ly many, very sparsely distributed in a so-
called "Cantor set" -- another recursively defined entity which springs up in topology. 
You might well wonder whether such an intr icate structure would ever show up in

You might well wonder whether such an intr icate structure would ever show up in 
an experiment. Frankly, I would be the most surprised person in the world if Gplot came 
out of any experiment. The physicality of Gplo t lies in the fact that it points the way to 
the proper mathematical treatment of less ideali zed problems of this sort. In other words, 
Gplot is purely a contribution to theoretical phys ics, not a hint to experimentalists as to 
what to expect to see! An agnostic friend of mine once was so struck by Gplot's infinitely 
many infinities that he called it "a picture of  God", which I don't think is blasphemous at 
all.

all.  
Recursion at the Lowest Level of Matter 
 We have seen recursion in the grammars of languages, we have seen recursive 
geometrical trees which grow upwards foreve r, and we have seen one way in which 
recursion enters the theory of solid state physics. Now we are going to see yet another 
way in which the whole world is built out of recursion. This has to do with the structure 
of elementary particles: electrons, pr otons, neutrons, and the tiny quanta of 
electromagnetic radiation called "photons". We are going to see that particles are-in a

We have seen recursion in the grammars of languages, we have seen recursive 
electromagnetic radiation called "photons". We are going to see that particles are-in a 
certain sense which can only be defined rigorously in relativistic quantum mechanics -- nested inside each other in a way which can  be described recursively, perhaps even by 
some sort of "grammar". 
We begin with the observation that if pa rticles didn't interact with each other, 
things would be incredibly simple. Physicists  would like such a world because then they 
could calculate the behavior of all particles easily (if physicists in such a world existed,

could calculate the behavior of all particles easily (if physicists in such a world existed, 
which is a doubtful proposition). Particle s without interactions are called bare particles , 
and they are purely hypothetical creations; they don't exist. 
Now when you "turn on" the interactions, then particles get tangled up together in 
the way that functions F and M are tangled  together, or married people are tangled 
together. These real particles are said to be renormalized -an ugly but intriguing term. 
What happens is that no particle can even  be defined without referring to all other

What happens is that no particle can even  be defined without referring to all other 
particles, whose definitions in turn depend on the first particles, etc. Round and round, in 
a never-ending loop.

Recursive Structures and Processes 151
 
Figure 34. Gplot; a recursive graph, showing en ergy bands for electrons in an idealized 
crystal in a magnetic field, α representing magnetic field strength, runs vertically from 0 
to 1. Energy runs horizontally. The horizontal line segments are bands of allowed 
electron energies.

Recursive Structures and Processes 152Let us be a little more concrete, now. Let's limit ourselves to only two kinds of 
particles: electrons  and photons . We'll also have to throw in the electron's antiparticle, the 
positron . (Photons are their own antiparticles.) Im agine first a dull world where a bare 
electron wishes to propagate from point  A to point B, as Zeno did in my Three-Part 
Invention.  A physicist would draw a picture like this: 
 
There is a mathematical expression which co rresponds to this line and its endpoints, and 
it is easy to write down. With it, a physicis t can understand the behavior of the bare 
electron in this trajectory. 
Now let us "turn on" the electromagnetic  interaction, whereby electrons and 
photons interact. Although there are no photons in  the scene, there will nevertheless be 
profound consequences even for this simple trajectory. In particular, our electron now 
becomes capable of emitting and then reabsorbing virtual photons -photons which flicker

becomes capable of emitting and then reabsorbing virtual photons -photons which flicker 
in and out of existence before they can be seen. Let us show one such process: 
 
Now as our electron propagates, it may emit a nd reabsorb one photon after another, or it 
may even nest them, as shown below: 
 
The mathematical expressions corresponding to these diagrams-called "Feynman diagrams"-are easy to write down, but they are harder to calculate than that for the bare electron. But what really complicates matters is that a photon (real or virtual) can decay

The mathematical expressions corresponding to these diagrams-called "Feynman diagrams"-are easy to write down, but they are harder to calculate than that for the bare electron. But what really complicates matters is that a photon (real or virtual) can decay 
for a brief moment into an electron-positron pair. Then these two annihilate each other, and, as if by magic, the original photon reappears. This sort of process is shown below: 
 
The electron has a right-pointing arrow, wh ile the positron's arrow points leftwards.

Recursive Structures and Processes 153As you might have anticipated, these virtua l processes can be inside each other to 
arbitrary depth. This can give rise to some  complicated-looking drawings, such as the one 
in Figure 35. In that man diagram, a single elect ron enters on the left at A, does some an 
acrobatics, and then a single electron emerges on the right at B. outsider who can't see the 
inner mess, it looks as if one electron peacefu lly sailed from A to B. In the diagram, you 
can see how el lines can get arbitrarily em bellished, and so can the photon lines diagram 
would be ferociously hard to calculate. 
. 
 
FIGURE 35. A Feynman diagram showing th e propagation of a renormalized electron 
from A to B. In this diagram, time increase s to the right. Therefore, in the segments 
where the electron’s arrow points leftwards, it is moving "backwards in time". A more intuitive way to say this is that an antielect ron (positron) is moving forwards in time.

. 
 
where the electron’s arrow points leftwards, it is moving "backwards in time". A more intuitive way to say this is that an antielect ron (positron) is moving forwards in time. 
Photons are their own antiparticles; hence their lines have no need of arrows. 
 
There is a sort of "grammar" to these di agrams, that only certain pictures to be 
realized in nature. For instance, the one be impossible: 
 
You might say it is not a "well-formed" Feynm an diagram. The gram a result of basic 
laws of physics, such as conservation of en ergy, conservation of electric charge, and so 
on. And, like the grammars of l - languages, th is grammar has a recursive structure, in

on. And, like the grammars of l - languages, th is grammar has a recursive structure, in 
that it allow' nestings of structures inside each  other. It would be possible to drat set of 
recursive transition networks defining the "grammar" of the electromagnetic interaction. 
When bare electrons and bare photons are a llowed to interact ii arbitrarily tangled 
ways, the result is renormalized  electrons and ph Thus, to understand how a real, physical 
electron propagates from A to B,

Recursive Structures and Processes 154 
the physicist has to be able to take a sort of  average of all the infinitely many different 
possible drawings which involve virtual particles. This is Zeno with a vengeance! 
Thus the point is that a physical partic le-a renormalized particle involves (1) a 
bare particle and (2) a huge tangle of virtual particles, inextricably wound together in a recursive mess. Every real particle's exis tence therefore involves the existence of 
infinitely many other particles, contained in a virtual "cloud" which surrounds it as it propagates. And each of the virtual particles in the cloud, of course, also drags along its 
own virtual cloud, and so on ad infinitum. 
Particle physicists have found that this co mplexity is too much to handle, and in 
order to understand the behavior of elect rons and photons, they use approximations 
which neglect all but fairly simple Feynman  diagrams. Fortunately, the more complex a

which neglect all but fairly simple Feynman  diagrams. Fortunately, the more complex a 
diagram, the less important its contributi on. There is no known way of summing up all of 
the infinitely many possible diagrams, to get an expression for the behavior of a fully renormalized, physical electron. But by c onsidering roughly the simplest hundred 
diagrams for certain processes, physicists have  been able to predict one value (the so-
called g-factor of the muon) to nine decimal places -- correctly! 
Renormalization takes place not only am ong electrons and photons. Whenever

Renormalization takes place not only am ong electrons and photons. Whenever 
any types of particle interact together, phys icists use the ideas of renormalization to 
understand the phenomena. Thus protons and neutrons, neutrinos, pi-mesons, quarks-all 
the beasts in the subnuclear zoo they all have  bare and renormalized versions in physical 
theories. And from billions of these bubbles w ithin bubbles are all the beasts and baubles 
of the world composed.  
Copies and Sameness 
 Let us now consider Gplot once agai n. You will remember that in the

 Let us now consider Gplot once agai n. You will remember that in the 
Introduction, we spoke of different varietie s of canons. Each type of canon exploited 
some manner of taking an original th eme and copying it by an isomorphism, or 
information-preserving transformation. Sometimes the copies were upside down, sometimes backwards, sometimes shrunken or expanded ... In Gplot we have all those types of transformation, and more. The mappings between the full Gplot and the "copies" of itself inside itself involve size changes, skewings, reflections, and more. And yet there

information-preserving transformation. Sometimes the copies were upside down, sometimes backwards, sometimes shrunken or expanded ... In Gplot we have all those types of transformation, and more. The mappings between the full Gplot and the "copies" of itself inside itself involve size changes, skewings, reflections, and more. And yet there 
remains a sort of skeletal identity, which the eye can pick up with a bit of effort, 
particularly after it has practiced with INT. 
Escher took the idea of an object's parts be ing copies of the object itself and made 
it into a print: his woodcut Fishes and Scales  (Fig. 36). Of course these fishes and scales 
are the same only when seen on a sufficiently  abstract plane. Now everyone knows that a 
fish's scales aren't really small copies of the fish; and a fish's cells aren't small copies of

fish's scales aren't really small copies of the fish; and a fish's cells aren't small copies of 
the fish; however, a fish's DNA , sitting inside each and every one of the fish's cells, is a 
very convo-

Recursive Structures and Processes 155
 
FIGURE 36. Fish and Scales, by M. C. Escher (woodcut, 1959). 
 luted "copy" of the entire fish-and so there is more than a grain of truth to the Escher 
picture. 
What is there that is the "same" about all butterflies? The mapping from one 
butterfly to another does not map cell ont o cell; rather, it m; functional part onto 
functional part, and this may be partially  on a macroscopic scale, partially on a 
microscopic scale. The exact proportions of pa are not preserved; just the functional 
relationships between parts. This is the type  of isomorphism which links all butterflies in 
Escher's wood engraving Butterflies  (Fig. 37) to each other. The same goes for the more 
abstract butterflies of Gplot,  which are all linked to each other by mathematical mappings 
that carry functional part onto functional pa rt, but totally ignore exact line proportions, 
angles, and so on. 
Taking this exploration of sameness to a yet higher plane of abstraction, we might

Taking this exploration of sameness to a yet higher plane of abstraction, we might 
well ask, "What is there that is the `same' a bout all Esc l drawings?" It would be quite 
ludicrous to attempt to map them piece by piece onto each other. The amazing thing is that even a tiny section of an

Recursive Structures and Processes 156
 
FIGURE 37. Butterflies, by M. C. Escher (wood-engraving, 1950). 
 
Escher drawing or a Bach piece gi ves it away. Just as a fish's DNA  is contained inside 
every tiny bit of the fish, so a creator's "si gnature" is contained inside every tiny section 
of his creations. We don't know what to call it but "style" -- a vague and elusive word. 
We keep on running up against "sameness-in-differentness", and the question  
When are two things the same? 
 
It will recur over and over again in this book. We  shall come at it from  all sorts of skew 
angles, and in the end, we shall see how deeply  this simple question is connected with the 
nature of intelligence. 
That this issue arose in the Chapter on recursion is no accident, for recursion is a 
domain where "sameness-in-differentness" play s a central role. Recursion is based on the 
"same" thing happening on several differ-

Recursive Structures and Processes 157ent levels at once. But the events on different  levels aren't exactly same-rather, we find 
some invariant feature in them, despite many s in which they differ. For example, in the 
Little Harmonic Labyrinth , all stories on different leve ls are quite unrelated-their 
"sameness" reside only two facts: (1) they ar e stories, and (2) they involve the Tortoise 
and Achilles. Other than that, they are radically different from each other.  
Programming and Recursion: Modularity, Loops, Procedures 
 One of the essential skills in computer progr amming is to perceive wl two processes are 
the same in this extended sense, for that leads modularization-the breaking-up of a task 
into natural subtasks. For stance, one might want a sequence of many similar operations to be cart out one after another. Instead of  writing them all out, one can write a h which 
tells the computer to perform a fixed set of  operations and then loop back and perform

tells the computer to perform a fixed set of  operations and then loop back and perform 
them again, over and over, until some condition is satisfied. Now the body  of the loop-the 
fixed set of instructions to repeated-need not actually be completely fixed. It may vary in 
so predictable way. 
An example is the most simple-minded te st for the primality o natural number N, 
in which you begin by trying to divide N by 2, then 3, 4, 5, etc. until N - 1. If N has survived all these tests without be divisible, it's prime. Notice that each step in the loop is

in which you begin by trying to divide N by 2, then 3, 4, 5, etc. until N - 1. If N has survived all these tests without be divisible, it's prime. Notice that each step in the loop is 
similar to, but i the same as, each other step. Notice also that the number of steps varies 
with N-hence a loop of fixed length could never work as a general test primality. There 
are two criteria for "aborting" the loop: (1) if so number divides N exactly, quit with answer "NO"; (2) if N - 1 is react as a te st divisor and N survives, quit with answer 
"YES". 
The general idea of loops, then, is this: pe rform some series of related steps over 
and over, and abort the process when sp ecific conditions are n Now sometimes, the

"YES". 
and over, and abort the process when sp ecific conditions are n Now sometimes, the 
maximum number of steps in a loop will be known advance; other times, you just begin, 
and wait until it is aborted. The sec ond type of loop -- which I call a free loop -- is 
dangerous, because criterion for abortion may ne ver occur, leaving the computer in a so-
cal "infinite loop". This distinction between bounded loops  and free loops  is one the most 
important concepts in all of computer scien ce, and we shall dev an entire Chapter to it: 
"BlooP and FlooP and G1ooP". 
Now loops may be nested inside each othe r. For instance, suppose t we wish to

Now loops may be nested inside each othe r. For instance, suppose t we wish to 
test all the numbers between 1 and 5000 for primality. We c write a second loop which 
uses the above-described test over and over starting with N = I and finishing with N = 
5000. So our program i have a "loop-the-loop" structure. Such program structures are 
typical – in fact they are deemed to be good programming style. Th is kind of nest loop 
also occurs in assembly instructions for commonplace items, and such activities as knitting or crocheting-in which very small loops are

Recursive Structures and Processes 158repeated several times in larger loops, which in turn are carried out repeatedly ... While 
the result of a low-level loop might be no more  than couple of stitches, the result of a 
high-level loop might be a substantial portion of a piece of clothing. 
In music, too, nested loops often occur-a s, for instance, when a scale (a small 
loop) is played several times in a row, pe rhaps displaced in pitch each new time. For 
example, the last movements of both th e Prokofiev fifth piano concerto and the 
Rachmaninoff second symphony contain extended passages in which fast, medium, and slow scale-loops are played simultaneously by different groups of instruments, to great 
effect. The Prokofiev scales go up; the R achmaninoff-scales, down. Take your pick. 
A more general notion than loop is that of subroutine, or procedure, which we 
have already discussed somewhat. The basic idea here is that a group of operations are

have already discussed somewhat. The basic idea here is that a group of operations are 
lumped together and considered a single unit with a name-such as the procedure 
ORNATE NOUN . As we saw in RTN 's, procedures can call each other by name, and 
thereby express very concisely sequences of operations which are to be carried out. This 
is the essence of modularity in programmi ng. Modularity exists, of course, in hi-fi 
systems, furniture, living cells, human so ciety-wherever there is hierarchical 
organization. 
More often than not, one wants a proce dure which will act variably, according to

More often than not, one wants a proce dure which will act variably, according to 
context. Such a procedure can either be given a way of peering out at what is stored in 
memory and selecting its actions accordingl y, or it can be explicitly fed a list of 
parameters which guide its choice of what actions to take. Sometimes both of these 
methods are used. In RTN  terminology, choosing the sequence of actions to carry out 
amounts to choosing which pathway to follow . An RTN  which has been souped up with 
parameters and conditions that control the choice of pathways inside it is called an

parameters and conditions that control the choice of pathways inside it is called an 
Augmented Transition Network  (ATN ). A place where you might prefer ATN 's to RTN's  
is in producing sensible-as distinguished fr om nonsensical-English sentences out of raw 
words, according to a grammar represented in a set of ATN 's. The parameters and 
conditions would allow you to insert various semantic constraints, so that random juxtapositions like "a thankless brunch" woul d be prohibited. More on this in Chapter 
XVIII, however. 
  
Recursion in Chess Programs 
 
A classic example of a recursive procedure with  parameters is one for choosing the "best"

A classic example of a recursive procedure with  parameters is one for choosing the "best" 
move in chess. The best move would seem to be the one which leaves your opponent in the toughest situation. Therefore, a test fo r goodness of a move is simply this: pretend 
you've made the move, and now evaluate th e board from the point of view of your 
opponent. But how does your opponent evaluate  the position? Well, he looks for his best 
move. That is, he mentally runs through all possible moves and evaluates them from what 
he thinks is your point of view, hoping they will look bad to you. But

Recursive Structures and Processes 159notice that we have now defined "best move" r ecursively, simply maxim that what is best 
for one side is worst for the other. The pro cedure which looks for the best move operates 
by trying a move and then calling on itself in the role of opponent ! As such, it tries 
another n calls on itself in the role of its opponent's opponent-that is, its 
This recursion can go several levels deep -but it's got to bottom out somewhere! 
How do you evaluate a board position without  looking There are a number of useful 
criteria for this purpose, such as si number of pieces on each side, the number and type of pieces undo the control of the center, and so on. By using this kind of evaluation at the 
bottom, the recursive move-generator can pop back upwards an( evaluation at the top 
level of each different move. One of the para meters in the self-calling, then, must tell 
how many moves to look ahead. TI most call on the procedure will use some externally

how many moves to look ahead. TI most call on the procedure will use some externally 
set value parameter. Thereafter, each time the procedure recursively calls must decrease this look-ahead parameter by 1. That way, w parameter reaches zero, the procedure will 
follow the alternate pathway -- the non-recursive evaluation. 
In this kind of game-playing program, each move investigate the generation of a 
so-called "look-ahead tree", with the move trunk, responses as main branches, counter-

responses as subsidiary branches, and so on. In Figure 38 I have shown a simple look-ahead tree depicting the start of a tic-tar-toe game. There is an art to figuring to avoid 
exploring every branch of a look-ahead tree out to its tip. trees, people-not computers-
seem to excel at this art; it is known that top-level players look ahead relatively little, compared to most chess programs – yet the people are far better! In the early days of 
compute people used to estimate that it would be ten years until a computer (or

compute people used to estimate that it would be ten years until a computer (or 
 
FIGURE 38. The branching tree of moves and countermoves at the start of c tic-tac-toe.

Recursive Structures and Processes 160program) was world champion. But after ten years had passed, it seemed that the day a 
computer would become world champion was still more than ten years away ... This is 
just one more piece of evidence for the rather recursive 
 
Hofstadter's Law : It always takes longer than you expect, even when you take into 
account Hofstadter's Law. 
 
Recursion and Unpredictability 
 
Now what is the connection between the recu rsive processes of this Chapter, and the 
recursive sets of the preceding Chapter? The answer involves the notion of a recursively 
enumerable set . For a set to be r.e. means that it can be generated from a set of starting 
points (axioms), by the repeated application of rules of inference. Thus, the set grows and 
grows, each new element being compounded somehow  out of previous elements, in a sort 
of "mathematical snowball". But this is the essence of recursion-something being defined

of "mathematical snowball". But this is the essence of recursion-something being defined 
in terms of simpler versions of itself, in stead of explicitly. The Fibonacci numbers and 
the Lucas numbers are perfect examples of r.e. sets-snowballing from two elements by a 
recursive rule into infinite sets. It is just a matter of convention to call an r.e. set whose 
complement is also r.e. "recursive". 
Recursive enumeration is a process in wh ich new things emerge from old things 
by fixed rules. There seem to be many surp rises in such processes-for example the 
unpredictability of the Q-sequence. It might s eem that recursively defined sequences of

unpredictability of the Q-sequence. It might s eem that recursively defined sequences of 
that type possess some sort of inherently in creasing complexity of behavior, so that the 
further out you go, the less predictable they get. This kind of thought carried a little 
further suggests that suitably complicated recursive systems might be strong enough to break out of any predetermined patterns. And isn't this one of the defining properties of 
intelligence? Instead of just considering pr ograms composed of procedures which can 
recursively call themselves, why not get really sophi sticated, and invent programs which

recursively call themselves, why not get really sophi sticated, and invent programs which 
can modify  themselves-programs which can act on programs, extending them, improving 
them, generalizing them, fixing them, and so on? This kind of "tangled recursion" probably lies at the heart of intelligence.

Canon 
by Intervallic Augmentation 
 
Achilles and the Tortoise have just fin ished a delicious Chinese banquet for 
two, at the best Chinese restaurant in town. 
 
Achilles: You wield a mean chopstick, Mr. T. 
Tortoise: I ought to. Ever since my youth, I have had a fondness for Oriental cuisine. And  you-
did you enjoy your meal, Achilles? Achilles:  Immensely. I'd not eaten Chinese food 
before. This meal was a splendid introduction. And now, are you in a hurry to go, or shall 
we just sit here and talk a little while? 
Tortoise: I'd love to talk while we drink our tea. Waiter!  
(A waiter comes up.) 
 
Could we have our bill, please, and some more tea?  
 
(The waiter rushes off.) 
 
Achilles: You may know more about Chinese cuis ine than I do, Mr.T, I'll bet I know more about 
Japanese poetry than you do. Have you ever read any haiku? 
Tortoise: I'm afraid not. What is a haiku? Achilles: A haiku is a Japanese seventeen-sylla ble poem-or minipoem rather, which is evocative

Tortoise: I'm afraid not. What is a haiku? Achilles: A haiku is a Japanese seventeen-sylla ble poem-or minipoem rather, which is evocative 
in the same way, perhaps, as a fragrant pe tal is, or a lily pond in a light drizzle. It 
generally consists of groups of: of five, then seven, then five syllables. 
Tortoise: Such compressed poems with seventeen syllables can't much meaning  ... Achilles: Meaning lies as much in the mind of the reader as i haiku. Tortoise: Hmm ... That's an evocative statement.  
 (The waiter arrives with their bill, another pot of tea, and two fortune cookies.) 
 
Thank you, waiter. Care for more tea, Achilles?

Thank you, waiter. Care for more tea, Achilles? 
Achilles: Please. Those little cookies look delicious. ( Picks one up, bites I into it and begins to 
chew. ) Hey! What's this funny thing inside? A piece of paper? 
Tortoise: That's your fortune, Achilles. Many Chin ese restaurants give out fortune cookies with 
their bills, as a way of softening the blow. I frequent Chinese restaurants, you come to 
think of fortune cookies

less as cookies than as message bearers Un fortunately you seem to have swallowed some 
of your fortune. What does the rest say? 
Achilles: It's a little strange, for all the letters  are run together, with no spaces in between. 
Perhaps it needs decoding in some way? Oh, now I see. If you put the spaces back in 
where they belong, it says, "ONE WAR TWO EA R EWE". I can't quite make head or tail 
of that. Maybe it was a haiku-like poem, of which I ate the majority of syllables. 
Tortoise: In that case, your fortune is now a me re 5/17-haiku. And a curious image it evokes. If 
5/17-haiku is a new art form, then I'd say woe, 0, woe are we ... May I look at it? 
Achilles ( handing the Tortoise the small slip of paper ): Certainly. 
Tortoise: Why, when I "decode" it, Achilles, it co mes out completely different! It's not a 5/17-
haiku at all. It is a six-syllable message which says, "0 NEW ART WOE ARE WE". That 
sounds like an insightful commentary on the new art form of 5/17-haiku.

sounds like an insightful commentary on the new art form of 5/17-haiku. 
Achilles: You're right. Isn't it astonishing that the poem contains its own commentary! Tortoise: All I did was to shift the reading frame by one unit-that is, shift all the spaces one unit 
to the right. 
Achilles: Let's see what your fortune says, Mr. Tortoise. Tortoise ( deftly splitting open his cookie , reads ): "Fortune lies as much in  the hand of the eater as 
in the cookie." 
Achilles: Your fortune is also a haiku, Mr. Tortoi se-at least it's got seventeen syllables in the 5-7-
5 form.

5 form. 
Tortoise: Glory be! I would never have noticed that, Achilles. It's the kind of thing only you 
would have noticed. What struck me more is  what it says-which, of course, is open to 
Achilles: I guess it just shows th at each of us has his own characteristic way of interpreting 
messages which we run across ... 
 
(Idly, Achilles gazes at the tea leaves on the bottom of his empty teacup.)

5 form. 
 
(Idly, Achilles gazes at the tea leaves on the bottom of his empty teacup.) 
 
Tortoise: More tea, Achilles? Achilles: Yes, thank you. By the way, how is your  friend the Crab? I have been thinking about 
him a lot since you told me of your peculiar phonograph-battle. 
Tortoise: I have told him about you, too, and he is quite eager to meet you. He is getting along 
just fine. In fact, he recently made a new acqui sition in the record player line: a rare type 
of jukebox. 
Achilles: Oh, would you tell me about it? I find j ukeboxes, with their flashing colored lights and 
silly songs, so quaint and reminiscent of bygone eras.

silly songs, so quaint and reminiscent of bygone eras. 
Tortoise: This jukebox is too large to fit in his hous e, so he had a shed specially built in back for 
it. 
Achilles: I can't imagine why it w ould be so large, unless it has an unusually large selection of 
records. Is that it? 
Tortoise: As a matter of fact, it has exactly one record.

Achilles: What? A jukebox with only one record? That's a contradiction in terms. Why is the 
jukebox so big, then? Is its single record gigantic -- twenty feet in diameter? 
Tortoise: No, it's just a regular jukebox-style record. Achilles: Now, Mr. Tortoise, you must be joshing me. After all, what I of a jukebox is it that has only a single song? Tortoise: Who said anything about a single song, Achilles? Achilles: Every- jukebox I've ever run into obe yed the fundamental jukebox-axiom: "One record, 
one song". 
Tortoise: This jukebox is different, Achilles. Th e one record sits vertically, suspended, and 
behind it there is a small but elaborate ne twork of overhead rails, from which hang 
various record players. When push a pair of buttons, such as B-1, that selects one of the 
record players. This triggers an automatic  mechanism that starts the record player 
squeakily rolling along the rusty tracks. It gets  shunted alongside the record-then it clicks 
into playing position.

one song". 
into playing position. 
Achilles: And then the record begins spinning and music comes out -- right? Tortoise: Not quite. The record stands still-it's the record player which rotates. Achilles: I might have known. But how, if you have but one record to play can you get more than 
one song out of this crazy contraption? 
Tortoise: I myself asked the Crab that question. He merely suggested I try it out. So I fished a

Tortoise: I myself asked the Crab that question. He merely suggested I try it out. So I fished a 
quarter from my pocket (you get three plays for a quarter), stuffed it in the slot, and hit buttons B-1, then C-3 then B-10-all just at random. 
Achilles: So phonograph B-1 came sliding down th e rail, I suppose, plugged itself into the 
vertical record, and began spinning? 
Tortoise: Exactly. The music that came out was quite agreeable, based the famous old tune B-A-
C-H, which I believe you remember. 
 
Achilles: Could I ever forget it? Tortoise: This was record player B-1. Then it finished, and was s rolled back into its hanging 
position, so that C-3 could be slid into position.

position, so that C-3 could be slid into position. 
Achilles: Now don't tell me that C-3 played another song?  Tortoise: It did just that. Achilles: Ah, I understand. It played the flip side  of the first song, or another band on the same 
side. 
Tortoise: No, the record has grooves only on one side, and has only a single band.

Achilles: I don't understand that at all. You CAN'T pull different songs out of the same record! 
Tortoise: That's what I thought until I saw Mr . Crab's jukebox. Achilles: How did the second 
song go? 
Tortoise: That's the interesting thing ... It was a song based on the melody C-A-G-E . 
Achilles: That's a totally different melody! Tortoise: True. Achilles: And isn't John Cage a composer of mode rn music? I seem to remember reading about 
him in one of my books on haiku. 
Tortoise: Exactly. He has composed many celebra ted pieces, such as 4'33", a three-movement 
piece consisting of silences of different le ngths. It's wonderfully expressive-if you like 
that sort of thing. 
Achilles: I can see where if I were in a loud a nd brash cafe I might gladly pay to hear Cage's 
4'33" on a jukebox. It might afford some relief! 
Tortoise: Right-who wants to hear the racket of clinking dishes and jangling silverware? By the 
way, another place where 4'33" would come in handy is the Hall of Big Cats, at feeding 
time.

time. 
Achilles: Are you suggesting that Cage belongs in the zoo? Well, I guess that makes some sense. 
But about the Crab's jukebox ... I am baffled. How could both " BACH " and " CAGE " be 
coded inside a single record at once? 
Tortoise: You may notice that there is some re lation between the two, Achilles, if you inspect 
them carefully. Let me point the way. What do you get if you list the successive intervals 
in the melody B-A-C-H ? 
Achilles: Let me see. First it goes down one se mitone, from B to A (where B is taken the

Achilles: Let me see. First it goes down one se mitone, from B to A (where B is taken the 
German way); then it rises three semitones to C; and finally it falls one semitone, to H. 
That yields the pattern: 
 
-1, +3, -1. 
 Tortoise: Precisely. What about C-A-G-E , now? 
Achilles: Well, in this case, it begins by falling three semitones, then ten semitones (nearly an 
octave), and finally falls three more semitones. That means the pattern is: 
 
-3, +10, -3. 
 
It's very much like the other one, isn't it? 
Tortoise: Indeed it is. They have exactly the same  "skeleton", in a certain sense. You can make

-1, +3, -1. 
 
Tortoise: Indeed it is. They have exactly the same  "skeleton", in a certain sense. You can make 
C-A-G-E  out of B-A-C-H  by multiplying all the intervals by 31/3, and taking the nearest 
whole number. 
Achilles: Well, blow me down and pick me up! So does that mean that only

some sort of skeletal code is present in th e grooves, and that the various record players 
add their own interpretations to that code? 
Tortoise: I don't know, for sure. The cagey Crab wouldn't fill me in on the details. But I did get 
to hear a third song, when record player B-1 swiveled into place. 
Achilles: How did it go? Tortoise: The melody consisted of enormously wide intervals, and we B-C-A-H . 
 
The interval pattern in semitones was: 
 
-10,       +33,       -10. 
 
It can be gotten from the CAGE pattern by yet another multiplication by 3%3, and rounding to whole numbers.  
Achilles: Is there a name for this kind of interval multiplication? Tortoise: One could call it "intervallic augmentation". It is similar to tl canonic device of 
temporal augmentation, where all the time values notes in a melody get multiplied by some constant. There, the effect just to slow the melody down. Here, the effect is to 
expand the melodic range in a curious way.

expand the melodic range in a curious way. 
Achilles: Amazing. So all three melodies you tried were intervallic augmentations of one single 
underlying groove-pattern in the record: 
Tortoise: That's what I concluded. Achilles: I find it curious that when you augment BACH  you get CAGE  and when you augment 
CAGE  over again, you get BACH back, except jumbled up inside, as if BACH  had an 
upset stomach after passing through the intermediate stage of CAGE . 
Tortoise: That sounds like an insightful commentary on the new art form of Cage.

The Location of Meaning 
 166CHAPTER VI 
 
When Is One Thing Not Always the Same? 
 
LAST CHAPTER, WE came upon the question, "Whe n are two things the same?" In this 
Chapter, we will deal with the flip side of that question: "When is one thing not always 
the same?" The issue we are broaching is whethe r meaning can be said to be inherent in a 
message, or whether meaning is always manuf actured by the interaction of a mind or a 
mechanism with a message-as in the preced ing Dialogue. In the latter case, meaning 
could not said to be located in any single pl ace, nor could it be said that a message has 
any universal, or objective, meaning, since each  observer could bring its own meaning to 
each message. But in the former case, meaning would have both location and universality. In this Chapter, I want to pres ent the case for the universality of at least 
some messages, without, to be sure, clai ming it for all messages. The idea of an

some messages, without, to be sure, clai ming it for all messages. The idea of an 
"objective meaning" of a message will turn out to be related, in an interesting way, to the 
simplicity with which intelligence can be described.  
Information-Bearers and Information- Revealers 
 I'll begin with my favorite example: the rela tionship between records, music, and record 
players. We feel quite comfortable with the idea that a record contains the same 
information as a piece of music, because of the existence of record players, which can 
"read" records and convert the groove-patterns into sounds. In other words, there is an

"read" records and convert the groove-patterns into sounds. In other words, there is an 
isomorphism between groove-patterns and sounds, and the record player is a mechanism 
which physically realizes that isomorphism. It is natural, then, to think of the record as an 
information-bearer , and the record-player as an information-revealer . A second example 
of these notions is given by the pq-system. There, the "information-bearers" are the 
theorems, and the "information-revealer" is the interpretation, which is so transparent that 
we don't need any electrical machine to he lp us extract the information from pq-
theorems.

theorems. 
One gets the impression from these two examples that isomorphisms and 
decoding mechanisms (i.e., information-reveal ers) simply reveal information which is 
intrinsically inside the structures, waiting to be "pulled out". This leads to the idea that for each structure, there are certa in pieces of information which can be pulled out of it, 
while there are other pieces of information which cannot  be pulled out of it. But what 
does this phrase

 167"pull out" really mean? How hard are you allowed to pull? There are c where by 
investing sufficient effort, you can pull very r econdite piece of information out of certain 
structures. In fact, the pulling-out may inv such complicated operations that it makes you feel you are putting in n information than you are pulling out.  
Genotype and Phenotype 
 Take the case of the genetic information co mmonly said to reside in double helix of 
deoxyribonucleic acid ( DNA ). A molecule of DNA  – a genotype -is converted into a 
physical organism-a phenotype -by a complex process, involving the manufacture of 
proteins, the replication the DNA , the replication of cells, the gradual differentiation of 
cell types and so on. Incidentally, this  unrolling of phenotype from genotype epigenesis -
is the most tangled of tangled recursions, a nd in Chapter we shall devote our full attention 
to it. Epigenesis is guided by a se enormously  complex cycles of chemical reactions and

to it. Epigenesis is guided by a se enormously  complex cycles of chemical reactions and 
feedback loops the time the full organism has been constructed, there is not even remotest similarity between its physical characteristics and its genotype. 
And yet, it is standard practice to attri bute the physical structure of organism to 
the structure of its DNA, and to that alone. Th e first evidence for this point of view came 
from experiments conducted by Oswald A,  in 1946, and overwhelming corroborative 
evidence has since been amassed Avery's expe riments showed that, of all the biological

evidence has since been amassed Avery's expe riments showed that, of all the biological 
molecules, only E transmits hereditary prope rties. One can modify other molecules it 
organism, such as proteins, but such modi fications will not be transmitted to later 
generations. However, when DNA is modifie d, all successive generations inherit the 
modified DNA . Such experiments show that the onl y of changing the instructions for 
building a new organism is to change DNA -and this, in turn, implies that those 
instructions must be cc somehow in the structure of the DNA .

instructions must be cc somehow in the structure of the DNA . 
 
Exotic and Prosaic Isomorphisms 
 Therefore one seems forced in to accepting the idea that the DNA 's structure contains the 
information of the phenotype's stru cture, which is to the two are isomorphic . However, 
the isomorphism is an exotic one, by w] I mean  that it is highly nontrivial to divide the 
phenotype and genotype into "parts" which can be mapped onto each other. Prosaic 
isomorphic by contrast, would be ones in wh ich the parts of one structure are easily

isomorphic by contrast, would be ones in wh ich the parts of one structure are easily 
mappable onto the parts of the other. An ex ample is the isomorphism between a record 
and a piece of music, where one knows that to any so in the piece there exists an exact "image" in the patterns etched into groove s, and one could pinpoint it arbitrarily 
accurately, if the need arose Another prosaic isomorphism is that between Gplot and any 
of its internal butterflies.

 168The isomorphism between DNA  structure and phenotype structure is anything but 
prosaic, and the mechanism which carries it out physically is awesomely complicated. 
For instance, if you wanted to find some piece of your DNA  which accounts for the shape 
of your nose or the shape of your fingerprint, you would have a very hard time. It would 
be a little like trying to pin down the note in a piece of music which is the carrier of the 
emotional meaning of the piece. Of course th ere is no such note, because the emotional 
meaning is carried on a very high level, by large "chunks" of the piece, not by single 
notes. Incidentally, such "chunks" are not necessarily sets of contiguous notes; there may be disconnected sections which, taken together, carry some emotional meaning. 
Similarly, "genetic meaning"-that is, in formation about phenotype structure-is 
spread all through the small parts of a molecule of DNA, although nobody understands

spread all through the small parts of a molecule of DNA, although nobody understands 
the language yet. (Warning: Understanding this  "language" would not at all be the same 
as cracking the Genetic Code, something which took place in the early 1960's. The 
Genetic Code tells how to translate short portions of DNA  into various amino acids. 
Thus, cracking the Genetic Code is comparable  to figuring out the phonetic values of the 
letters of a foreign alphabet, without figur ing out the grammar of the language or the 
meanings of any of its words. The cracking of the Genetic Code was a vital step on the

meanings of any of its words. The cracking of the Genetic Code was a vital step on the 
way to extracting the meaning of DNA  strands, but it was only the first on a long path 
which is yet to be trodden.)   
Jukeboxes and Triggers 
 The genetic meaning contained in DNA is one of the best possible examples of implicit meaning. In order to convert genotype into  phenotype, a set of mechanisms far more 
complex than the genotype must operate on the genotype. The various parts of the 
genotype serve as triggers  for those mechanisms. A juke box-the ordinary type, not the 
Crab type!-provides a useful analogy here: a pair of buttons specifies a very complex

Crab type!-provides a useful analogy here: a pair of buttons specifies a very complex 
action to be taken by the mechanism, so that the pair of buttons could well be described 
as "triggering" the song which is played. In  the process which converts genotype into 
phenotype, cellular jukeboxes-if you will pardon the notion!-accept "button-pushings" from short excerpts from a long strand of DNA , and the "songs" which they play are 
often prime ingredients in the creation of furthe r "jukeboxes". It is as if the output of real 
jukeboxes, instead of being love ballads, were songs whose lyrics told how to build more

jukeboxes, instead of being love ballads, were songs whose lyrics told how to build more 
complex jukeboxes ... Portions of the DNA  trigger the manufacture of proteins; those 
proteins trigger hundreds of new reactions; they  in turn trigger the replicating-operation 
which, in several steps, copies the DNA -and on and on ... This gives a sense of how 
recursive the whole process is. The final resu lt of these many-triggered triggerings is the 
phenotype-the individual. And one says that the phenotype is the revelation-the "pulling-
out"-of the information that was present in the DNA  to start with, latently. (The term

out"-of the information that was present in the DNA  to start with, latently. (The term 
"revelation" in this context is due to

 169Jacques Monod, one of the deepest and most original of twentieth-century molecular 
biologists.) 
Now no one would say that a song comi ng out of the loudspeaker of jukebox 
constitutes a "revelation" of information inhere nt in the pair buttons which were pressed, 
for the pair of buttons seem to be mere triggers , whose purpose is to activate information-
bearing portions of the jukebox mechanism.  On the other hand, it seems perfectly 
reasonable to call t extraction of music from  a record a "revelation" of information 
inherent the record, for several reasons: 
 
(1) the music does not seem to be concealed in the mechanism of the record player; (2) it is possible to match pieces of the i nput (the record) with pieces of the output 
(the music) to an arbitrary degree of accuracy; 
(3) it is possible to play other records on the same record player and get other 
sounds out; 
(4)  the record and the record player are easily separated from one another.

(4)  the record and the record player are easily separated from one another. 
 
It is another question altogether whether the fragments of a smashed  record contain 
intrinsic meaning. The edges of the separate pieces together and in that way allow the 
information to be reconstituted-t something much more complex is going on here. Then there is the question of the intrinsic meani ng of a scrambled telephone call ... There is a 
vast spectrum of degrees of inherency of mean ing. It is interesting to try place epigenesis 
in this spectrum. As development of an or ganism takes place, can it be said that the

in this spectrum. As development of an or ganism takes place, can it be said that the 
information is being "pulled out" of its DNA ? Is that where all of the information about 
the organism's structure reside;  
DNA  and the Necessity of Chemical Context 
 
In one sense, the answer seems to be yes, thanks to experiments li Avery's. But in another 
sense, the answer seems to be no, because so  much of the pulling-out process depends on 
extraordinarily complicated cellular chemical processes, which are not coded for in the 
DNA  itself. The DNA  relies on the fact that they will happen, but does not seem to

DNA  itself. The DNA  relies on the fact that they will happen, but does not seem to 
contain a code which brings them about. Thus we have two conflicting views on the 
nature of the information in a genotype. One vi ew says that so much of t information is 
outside the DNA  that it is not reasonable to look upon the DNA  as anything more than a 
very intricate set of triggers, like a seque nce of buttons to be pushed on a jukebox; 
another view says that the information is all there , but in a very implicit form. 
Now it might seem that these are just two ways of saying the same thing, but that

Now it might seem that these are just two ways of saying the same thing, but that 
is not necessarily so. One view says that the DNA  is quite meaningless out of context; the 
other says that even if it were taken out context, a molecule of DNA  from a living being 
has such a compelling inner

 170logic  to its structure that its message could be  deduced anyway. To put it as succinctly as 
possible, one view says that in order for DNA  to have meaning, chemical context  is 
necessary; the other view says that only intelligence  is necessary to reveal the "intrinsic 
meaning" of a strand of DNA . 
 
An Unlikely UFO 
 We can get some perspective on this issue by considering a strange hypothetical 
event. A record of David Oistrakh and Lev O borin playing Bach's sonata in F Minor for 
violin and clavier is sent up in a satellite. Fr om the satellite it is then launched on a course 
which will carry it outside of the solar system, perhaps out of the entire galaxy just a thin plastic platter with a hole in the middle, sw irling its way through intergalactic space. It 
has certainly lost its context. How much meaning does it carry? 
If an alien civilization were to encounter it, they would almost certainly be struck

If an alien civilization were to encounter it, they would almost certainly be struck 
by its shape, and would probably be very interested in it. Thus immediately its shape, acting as a trigger, has given them some information: that it is an artifact, perhaps an information-bearing artifact. This idea-communicated, or triggered, by the record itself-now creates a new context in which the reco rd will henceforth be perceived. The next 
steps in the decoding might take considerably longer-but that is very hard for us to assess. We can imagine that if such a record had arrived on earth in Bach's time, no one would

steps in the decoding might take considerably longer-but that is very hard for us to assess. We can imagine that if such a record had arrived on earth in Bach's time, no one would 
have known what to make of it, and very likely it would not have gotten deciphered. But 
that does not diminish our conviction that the information was in principle there; we just 
know that human knowledge in those times was not  very sophisticated with respect to the 
possibilities of storage, transformation, and revelation of information. 
 
Levels of Understanding of a Message 
 
Nowadays, the idea of decoding is extremely wi despread; it is a significant part of the 
activity of astronomers, linguists, archaeologi sts, military specialists, and so on. It is 
often suggested that we may be floating in a sea of radio messages from other 
civilizations, messages which we do not yet know how to decipher. And much serious 
thought has been given to the techniques of deciphering such a message. One of the main

thought has been given to the techniques of deciphering such a message. One of the main 
problems perhaps the deepest problem-is the question, "How will we recognize the fact 
that there is a message at all? How to iden tify a frame?" The sending of a record seems to 
be a simple solution-its gross physical struct ure is very attention-drawing, and it is at 
least plausible to us that it would trigger, in  any sufficiently great intelligence, the idea of 
looking for information hidden in it. However,  for technological reasons, sending of solid

looking for information hidden in it. However,  for technological reasons, sending of solid 
objects to other star systems seems to be out  of the question. Still, that does not prevent 
our thinking about the idea. 
Now suppose that an alien civilization hit upon the idea that the appropriate 
mechanism for translation of the record is a machine which

 171converts the groove-patterns into sounds. This  would still be a far cry from a true 
deciphering. What, indeed, would constitute a successful  deciphering of such a record? 
Evidently, the civilization would have to be able to ma sense out of the sounds. Mere 
production of sounds is in itself hart worthwhile, unless they have the desired triggering effect in the brains that is the word) of the alien creatures. And what is that desired 
effect? would be to activate structures in th eir brains which create emotional effects in 
them which are analogous to the emotional effects which experience in hearing the piece. In fact, the production of sounds  cot even be bypassed, provided that they used the record 
in some other way get at the appropriate stru ctures in their brains. (If we humans had a w 
of triggering the appropriate structures in our  brains in sequential order, as music does, 
we might be quite content to bypass the sounds -but it see] extraordinarily unlikely that

we might be quite content to bypass the sounds -but it see] extraordinarily unlikely that 
there is any way to do that, other than vi a o ears. Deaf composers-Beethoven, Dvofák, 
Faure-or musicians who can "hear" music by looki ng at a score, do not give the lie to this 
assertion, for such abilities are founded upon preceding decades of direct auditory 
experiences.) 
Here is where things become very uncl ear. Will beings of an alien civilization 
have emotions? Will their emotions-supposing they have some-be mappable, in any sense, onto ours? If they do have emotions somewhat like ours, do the emotions cluster

have emotions? Will their emotions-supposing they have some-be mappable, in any sense, onto ours? If they do have emotions somewhat like ours, do the emotions cluster 
together in somewhat the same way as our s do? Will they understand such amalgams as 
tragic beauty courageous suffering? If it tu rns out that beings throughout the universe do 
share cognitive structures with us to the extent  that even emotions overlap, then in some 
sense, the record can never be out of its natura l context; that context is part of the scheme 
of things, in nature. And if such is the case,  then it is likely that a meandering record, if 
not destroyed en route, would eventually ge t picked up by a being or group of beings, at

not destroyed en route, would eventually ge t picked up by a being or group of beings, at 
get deciphered in a way which we would consider successful. 
 
"Imaginary Spacescape" 
 
In asking about the meaning of a molecule of DNA  above, I used t phrase 
"compelling inner logic"; and I think this is a key notion. To illustrate this, let us slightly 
modify our hypothetical record-into-spa event by substituting John Cage's "Imaginary Landscape no. 4" for the Bach. This piece is a classic of aleatoric , or chance , music-

music who structure is chosen by various random processes, rather than by an attempt to convey a personal emotion. In this case, twenty -four performers attar themselves to the 
twenty-four knobs on twelve radios. For the dur ation the piece they twiddle their knobs in 
aleatoric ways so that each radio randomly ge ts louder and softer, switching stations all 
the while. The tot sound produced is the piece of music. Cage's attitude is expressed in 14 
own words: "to let sounds be themselves, rather  than vehicles for man made theories or 
expressions of human sentiments."

 172Now imagine that this is the piece on the r ecord sent out into space. It would be 
extraordinarily unlikely-if not downright impossible-for an alien civilization to 
understand the nature of the artifact. They  would probably be very puzzled by the 
contradiction between the frame message ("I am a message; decode me"), and the chaos of the inner structure. There are few "chunks " to seize onto in this Cage piece, few 
patterns which could guide a decipherer. On th e other hand, there seems to be, in a Bach 
piece, much to seize onto-patterns, patterns of patterns, and so on. We have no way of 
knowing whether such patterns are universa lly appealing. We do not know enough about 
the nature of intelligence, emotions, or musi c to say whether the inner logic of a piece by 
Bach is so universally compelling that its meaning could span galaxies. 
However, whether Bach in particular ha s enough inner logic is not the issue here;

However, whether Bach in particular ha s enough inner logic is not the issue here; 
the issue is whether any message has, pe r se, enough compelling inner logic that its 
context will be restored automatically whenever intelligence of a high enough level 
comes in contact with it. If some message di d have that context-restoring property, then it 
would seem reasonable to consider the meaning of the message as an inherent property of 
the message. 
 
The Heroic Decipherers 
 
Another illuminating example of these ideas is  the decipherment of ancient texts written 
in unknown languages and unknown alphabets. The intuition feels that there is

in unknown languages and unknown alphabets. The intuition feels that there is 
information inherent in such texts, whether or not we succeed in revealing it. It is as 
strong a feeling as the belief that there is meaning inherent in a newspaper written in 
Chinese, even if we are completely ignorant of Chinese. Once the sc ript or language of a 
text has been broken, then no one questions wh ere the meaning resides: clearly it resides 
in the text, not in the method of decipherment ju st as music resides in a record, not inside

in the text, not in the method of decipherment ju st as music resides in a record, not inside 
a record player! One of the ways that we id entify decoding mechanisms is by the fact that 
they do not add any meaning to the signs or objects which they take as input; they merely reveal the intrinsic meaning of those si gns or objects. A jukebox is not a decoding 
mechanism, for it does not reveal any mean ing belonging to its input symbols; on the 
contrary, it supplies meaning concealed inside itself. 
Now the decipherment of an ancient text  may have involved decades of labor by

Now the decipherment of an ancient text  may have involved decades of labor by 
several rival teams of scholars, drawing on knowledge stored in libraries all over the 
world ... Doesn't this process add information, too? Just how intrinsic is the meaning of a text, when such mammoth efforts are required in order to find the decoding rules? Has one put meaning into the text, or was that m eaning already there? My intuition says that 
the meaning was always there, and that  despite the arduousness of the pulling-out 
process, no meaning was pulled out that wasn't  in the text to start with. This intuition

process, no meaning was pulled out that wasn't  in the text to start with. This intuition 
comes mainly from one fact: I feel that the result was inevitable; that, had the text not 
been deciphered by this group at this time, it would have been deciphered by that group at that time-and it would have come

 173
 
FIGURE 39. The Rosetta Stone [courtesy of the British Museum. 
 out the same way. That is why the meaning is part of the text itself; it acts upon 
intelligence in a predictable way. Generally, we can say: meaning is part of an object to 
the extent that it acts upon intelligence in a predictable way. 
In Figure 39 is shown the Rosetta stone, one  of the most precious of all historic 
discoveries. It was the key to the decipherm ent of Egyptian hieroglyphics, for it contains 
parallel text in three ancient scripts: hi eroglyphic demotic characters, and Greek. The 
inscription on this basalt stele was firs deciphered in 1821 by Jean Francois Champollion, the "father of Egyptology"; it is  a decree of priests assembled at Memphis in favor of 
Ptolemy Epiphanes.

 174Three Layers of Any Message 
In these examples of decipherment of out -of-context messages, we can separate 
out fairly clearly three levels of information: (1) the frame  message; (2) the outer 
message; (3) the inner  message. The one we are most familiar with is (3), the inner 
message; it is the message which is supposed to be transmitted: the emotional experiences in music, the phenotype in ge netics, the royalty and rites of ancient 
civilizations in tablets, etc. 
 
To understand the inner message is to have extracted the meaning intended by the 
sender.. 
 
The frame message is the message "I am a message; decode me if you can!"; and 
it is implicitly conveyed by the gross structural aspects of any information-bearer. 
 
To understand the frame message is to recognize the need for a decoding-
mechanism. 
 If the frame message is recognized as such, then attention is switched to level (2), 
the outer message. This is information, implicitly carried by symbol-patterns and

sender.. 
 
the outer message. This is information, implicitly carried by symbol-patterns and 
structures in the message, which tells how to decode the inner message. 
 
To understand the outer message is to build, or know how to build, the correct 
decoding mechanism for the inner message. 
 
This outer level is perforce an implicit message , in the sense that the sender cannot ensure 
that it will be understood. It would be a vain e ffort to send instructions which tell how to 
decode the outer message, for they would have to be part of the inner message, which can 
only be understood once the decoding mechan ism has been found. For this reason, the

only be understood once the decoding mechan ism has been found. For this reason, the 
outer message is necessarily a set of tri ggers, rather than a message which can be 
revealed by a known decoder. 
The formulation of these three "layers"  is only a rather crude beginning at 
analyzing how meaning is contained in messa ges. There may be layers and layers of 
outer and inner messages, rather than just  one of each. Think, for instance, of how 
intricately tangled are the inner and outer messages of the Rosetta stone. To decode a

intricately tangled are the inner and outer messages of the Rosetta stone. To decode a 
message fully, one would have to reconstruct the entire semantic structure which underlay its creation and thus to understand the sender in every deep way. Hence one 
could throw away the inner message, because if one truly understood all the finesses of 
the outer message, the inner message would be reconstructible. 
The book After Babel , by George Steiner, is a long discussion of the interaction 
between inner and outer messages (though he never uses that terminology). The tone of his book is given by this quote:

between inner and outer messages (though he never uses that terminology). The tone of his book is given by this quote: 
 
We normally use a shorthand beneath whic h there lies a wealth of subconscious, 
deliberately concealed or declared asso ciations so extensive and intri-

 175cate that they probably equal the sum and uniqueness of our status as an individual person.' 
 
Thoughts along the same lines are expressed by Leonard B. Meyer, in h book Music, the 
Arts, and Ideas: 
 
The way of listening to a composition by Elliott Carter is radically different from the way 
of listening appropriate to a work by John Ca ge. Similarly, a novel by Beckett must in a 
significant sense be read differently from one by Bellow. A painting by Willem de 
Kooning and one by Andy War hol require different perceptional-cognitive attitudes.' 
 
Perhaps works of art are trying to convey th eir style more than an thing else. In 
that case, if you could ever plumb a style to  its very bottom you could dispense with the 
creations in that style. "Style", "outer  message "decoding technique"-all ways of 
expressing the same basic idea. 
 
Schrodinger's Aperiodic Crystals 
 What makes us see a frame message in certain objects, but none in other; Why

 What makes us see a frame message in certain objects, but none in other; Why 
should an alien civilization suspect, if they inte rcept an errant record that a message lurks 
within? What would make a record any differe nt from a meteorite? Clearly its geometric 
shape is the first clue that "something funny is  going on". The next clue is that, on a more 
microscopic scale, consists of a very long aperiodic sequence of patterns, arranged in a 
spiral If we were to unwrap the spiral, we would have one huge linear sequence (around

spiral If we were to unwrap the spiral, we would have one huge linear sequence (around 
2000 feet long) of minuscule symbols. This is not so different from a DNA  molecule, 
whose symbols, drawn from a meager "alphabe t" of four different chemical bases, are 
arrayed in a one-dimensional sequence, an then coiled up into a helix. Before Avery had established the connection between genes and DNA , the physicist Erwin Schrödinger 
predicted, o purely theoretical grounds, that gene tic information would have to be stored 
in "aperiodic crystals", in his influential book What Is Life ? In fact books themselves are

in "aperiodic crystals", in his influential book What Is Life ? In fact books themselves are 
aperiodic crystals contained inside neat ge ometric forms. These examples suggest that, 
where an aperiodic crystal is found "packaged" inside a very regular geometric structure, 
there may lurk a inner message. (I don't claim this is a complete characterization of frame messages; however, it is a fact that many co mmon messages have frame messages of this 
description. See Figure 40 for some good examples.)  
Languages for the Three Levels 
 
The three levels are very clear in the case of a message found in a bottle washed up on a

The three levels are very clear in the case of a message found in a bottle washed up on a 
beach. The first level, the frame message, is  found when one picks up the bottle and sees 
that it is sealed, and contains a dry pi ece c paper. Even without seeing writing, one 
recognizes this type of artifact an information-bearer, and at this point it would take an extraordinary almost inhuman-lack of curiosity, to drop the bottle and not look further.

 177Next, one opens the bottle and examines the marks on the paper. Perhaps, they are in 
Japanese; this can be discovered without any of the inner message being understood-it 
merely comes from a recognition of 1 character s. The outer message can be stated as an 
English sentence: "I in Japanese." Once this has been discovered, then one can proceed the inner message, which may be a call for help, a haiku poem, a lover’s lament ... 
It would be of no use to include in th e inner message a translation the sentence 
"This message is in Japanese", since it woul d take someone who knew Japanese to read 
it. And before reading it, he would have rec ognize the fact that, as it is in Japanese, he 
can read it. You might try wriggle out of th is by including translations of the statement 
"This mess2 is in Japanese" into many differe nt languages. That would help it practical

"This mess2 is in Japanese" into many differe nt languages. That would help it practical 
sense, but in a theoretical sense the same difficulty is there. . English-speaking person still has to recognize the "Englishness" of the  message; otherwise it does no good. Thus 
one cannot avoid the problem that one has to  find out how to decipher the inner message 
from the outside the inner message itself may provide clues and confirmations, but those ; 
at best triggers acting upon the bottle finder (or upon the people whom enlists to help). 
Similar kinds of problem confront the s hortwave radio listener. First he has to

Similar kinds of problem confront the s hortwave radio listener. First he has to 
decide whether the sounds he hears actually constitute a message or are just static. The sounds in themselves do not give the answer, not e% in the unlikely case that the inner 
message is in the listener's own native langua ge, and is saying, "These sounds actually 
constitute a message a are not just static!" If the listener recognizes a frame message in the soup then he tries to iden tify the language the broadcast is in-and clearly, he is still on 
the outside; he accepts triggers from the radio, but they cam explicitly tell him the answer. 
It is in the nature of outer messages that they are not conveyed in any

It is in the nature of outer messages that they are not conveyed in any  
FIGURE 40. A collage of scripts. Uppermost on the left is an inscription in the un ciphered 
boustrophedonic writing system from Easter Island,  in which every second lin upside down. The 
characters are chiseled on a wooden tablet, 4 in ches by 35 inches. Mov clockwise, we encounter 
vertically written Mongolian: above, present-day Mongolian, below, a document dating from 
1314. Then we come to a poem in Bengali by Rabindran Tagore in the bottom righthand corner. Next to it is a newspaper headline in Malayalam (II Kerala, southern India), above which is the

1314. Then we come to a poem in Bengali by Rabindran Tagore in the bottom righthand corner. Next to it is a newspaper headline in Malayalam (II Kerala, southern India), above which is the 
elegant curvilinear language Tamil (F Kerala). The smallest entry is part of a folk tale in 
Buginese (Celebes Island, Indonesia). In center of the collage is a paragraph in the Thai language, and above it a manuscript in Rn dating from the fourteenth century, containing a sample of the provincial law of Scania (so Swed en). Finally, wedged in on the left is a section of 
the laws of Hammurabi, written Assyrian cuneiform . As an outsider, I feel a deep sense of

the laws of Hammurabi, written Assyrian cuneiform . As an outsider, I feel a deep sense of 
mystery as I wonder how meanin cloaked in the strange curves and angles of each of these 
beautiful aperiodic crystals. Info there is conten t. [From Ham Jensen, Sign, Symbol, and Script 
(New York: G. Putnam's S. 1969), pp. 89 (cuneiform), 356 (Eas ter Island), 386, 417 (Mongolian), 
552 (Runic); from Keno Katzner, The Languages of the World (New York: Funk & Wagnalls, 1975), pp. 190 (Bengali), 
(Buginese); from I. A. Richards and Chris tine Gibson, English Through Pictures (New Y 
Washington Square Press, 1960), pp. 73 (Tamil), 82 (Thai
).

 178explicit language. To find an explicit language  in which to convey outer messages would 
not be a breakthrough-it would be a contradicti on in terms! It is always the listener's 
burden to understand the outer message. Success le ts him break through into the inside, at 
which point the ratio of triggers to explicit m eanings shifts drastically towards the latter. 
By comparison with the previous stag es, understanding the inner message seems 
effortless. It is as if it just gets pumped in.  
The "Jukebox" Theory of Meaning. 
 These examples may appear to be eviden ce for the viewpoint that no message has 
intrinsic meaning, for in order to understa nd any inner message, no matter how simple it 
is, one must first understand its frame message and its outer message, both of which are carried only by triggers (such as being wr itten in the Japanese alphabet, or having 
spiraling grooves, etc.). It begins to s eem, then, that one cannot get away from a

spiraling grooves, etc.). It begins to s eem, then, that one cannot get away from a 
"jukebox" theory of meaning-the doctrine th at no message contains inherent meaning, 
because, before any message can be understood, it has to be used as the input to some 
"jukebox", which means that information contai ned in the "jukebox" must be added to the 
message before it acquires meaning. 
This argument is very similar to the trap  which the Tortoise caught Achilles in, in

This argument is very similar to the trap  which the Tortoise caught Achilles in, in 
Lewis Carroll's Dialogue. There, the trap was the idea that before you can use any rule, you have to have a rule which tells you how to use that rule; in other words, there is an infinite hierarchy of levels of rules, which prevents any rule from ever getting used. Here, 
the trap is the idea that before you can understand any message, you have to have a message which tells you how to understand that  message; in other words, there is an 
infinite hierarchy of levels of messages, which prevents any message from ever getting

infinite hierarchy of levels of messages, which prevents any message from ever getting 
understood. However, we all know that these paradoxes are invalid, for rules do get used, 
and messages do get understood. How come? 
 
Against the Jukebox Theory 
 This happens because our intelligence is not disembodied, but is instantiated in physical 
objects: our brains. Their structure is due to the long process of evolution, and their 
operations are governed by the laws of physics. Since they are physical entities, our brains run without being told how to run. So  it is at the level where thoughts are produced

operations are governed by the laws of physics. Since they are physical entities, our brains run without being told how to run. So  it is at the level where thoughts are produced 
by physical law that Carroll's rule-paradox breaks down; and likewise, it is at the level where a brain interprets incoming data as a message that the message-paradox breaks down. It seems that brains come equipped w ith "hardware" for recognizing that certain 
things are messages, and for decoding thos e messages. This minimal inborn ability to 
extract inner meaning is what allows th e highly recursive, snowballing process of 
language acquisition to take place. The inborn hardware is like a jukebox: it supplies the

language acquisition to take place. The inborn hardware is like a jukebox: it supplies the 
additional information which turns mere triggers into complete messages.

 179Meaning Is Intrinsic If Intelligence Is Natural 
 
Now if different people's "j ukeboxes" had different "songs" in then responded to given 
triggers in completely idiosyncratic ways, the would have no inclination to attribute 
intrinsic meaning to those tri; However, human  brains are so constructed that one brain 
responds in much the same way to a given tr igger as does another brain, all other t being 
equal. This is why a baby can learn any language; it responds to triggers in the same way as any other baby. This uniformity of "human jukeboxes" establishes a uniform "language" in which frame message out er messages can be communicated. If, 
furthermore, we believe human intelligence is  just one example of a general phenomena 
nature-the emergence of intelligent beings in widely varying contexts then presumably the "language" in which frame messages and outer sages are communicated among 
humans is a "dialect" of a universal  gauge by which intelligences can communicate with

humans is a "dialect" of a universal  gauge by which intelligences can communicate with 
each other. Thus, would be certain kinds of  triggers which would have "universal 
triggering power", in that all intelligent bei ngs would tend to respond to them i same way 
as we do. 
This would allow us to shift our descri ption of where meaning located. We could 
ascribe the meanings (frame, outer, and inner) message to the message itself, because of 
the fact that deciphering mechanisms are themselves universal-that is, they are fundamental f of nature which arise in the same way in diverse contexts. To make it 
concrete, suppose that "A-5" triggered the same song in all jukeboxes suppose moreover

as we do. 
concrete, suppose that "A-5" triggered the same song in all jukeboxes suppose moreover 
that jukeboxes were not man-made artifacts, but w occurring natural objects, like galaxies 
or carbon atoms. Under such circumstances, we  would probably feel justified in calling 
the universal triggering power of "A-5" its "i nherent meaning"; also, "A-5" would merit:  
the name of "message", rather than "trigger" , and the song would indeed "revelation" of 
the inherent, though implicit, meaning of "A-5". 
 
Earth Chauvinism 
 
This ascribing of meaning to a message co mes from the invariance c processing of the

This ascribing of meaning to a message co mes from the invariance c processing of the 
message by intelligences distributed anywhere ii universe. In that sense, it bears some resemblance to the ascribing of to an object. To  the ancients, it must have seemed that an 
object's weight was an intrinsic property of  the object. But as gravity became understood, 
it was realized that weight varies with the gravitational field the object is immersed in. 
Nevertheless, there is a related quantity, th e mass, which not vary according to the 
gravitational field; and from this invariance the conclusion that an object's mass was an

gravitational field; and from this invariance the conclusion that an object's mass was an 
intrinsic property of the object itself. If it turn s out that mass is also variable, according to 
context, then will backtrack and revise our opinion that it is an intrinsic property of an object. In the same way, we might imagine that there could exist other

 180kinds of "jukeboxes"-intelligences-which communicate among each other via messages 
which we would never recognize as messages,  and who also would never recognize our 
messages as messages. If that were the case, then the claim that meaning is an intrinsic 
property of a set of symbols would have to be reconsidered. On the other hand, how could we ever realize that such beings existed? 
It is interesting to compare this argument for the inherency of meaning with a 
parallel argument for the inherency of weight . Suppose one defined an object's weight as 
"the magnitude of the downward force which the object exerts when on the surface of the 
planet Earth". Under this definition, the dow nward force which an object exerts when on 
the surface of Mars would have to be given another name than "weight". This definition 
makes weight an inherent property, but at the cost of geocentricity" Earth chauvinism". It

makes weight an inherent property, but at the cost of geocentricity" Earth chauvinism". It 
would be like "Greenwich chauvinism"-re fusing to accept local time anywhere on the 
globe but in the GMT  time zone. It is an unnatural way to think of time. 
Perhaps we are unknowingly burdened with a similar chauvinism with respect to 
intelligence, and consequently with respect to  meaning. In our chauvinism, we would call 
any being with a brain sufficiently much like our own "intelligent", and refuse to

any being with a brain sufficiently much like our own "intelligent", and refuse to 
recognize other types of objects as intelligent. To take an extreme example, consider a meteorite which, instead of deciphering the outer-space Bach record, punctures it with colossal indifference, and continues in its merry orbit. It has interacted with the record in 
a way which we feel disregards the record 's meaning. Therefore, we might well feel

a way which we feel disregards the record 's meaning. Therefore, we might well feel 
tempted to call the meteorite "stupid". But perhaps we would thereby do the meteorite a disservice. Perhaps it has a "higher intellig ence" which we in our Earth chauvinism 
cannot perceive, and its interaction with the record was a manifestation of that higher 
intelligence. Perhaps, then, the record has a "h igher meaning"-totally different from that 
which we attribute to it; perhaps its m eaning depends on the type of intelligence 
perceiving it. Perhaps. 
It would be nice if we could define in telligence in some other way than "that

It would be nice if we could define in telligence in some other way than "that 
which gets the same meaning out of a sequen ce of symbols as we do". For if we can only 
define it this one way, then our argument that meaning is an intrinsic property is circular, 
hence content-free. We should try to form ulate in some independent way a set of 
characteristics which deserve the name "i ntelligence". Such characteristics would 
constitute the uniform core of intelligence, shar ed by humans. At this point in history we 
do not yet have a well-defined list of those ch aracteristics. However, it appears likely that 
within the next few decades there will be mu ch progress made in elucidating what human

within the next few decades there will be mu ch progress made in elucidating what human 
intelligence is. In particular, perhaps cogn itive psychologists, workers in Artificial 
Intelligence, and neuroscientists will be able  to synthesize their understandings, and come 
up with a definition of intelligence. It may still be human-chauvinistic; there is no way 
around that. But to counterbalance that, ther e may be some elegant and beautiful-and 
perhaps even simple-abstract ways of char acterizing the essence of intelligence. This 
would serve to lessen the feeling of having

 181formulated an anthropocentric concept. And of course, if contact were established with an 
alien civilization from another star system, we feel supported in our belief that our own 
type of intelligence is not just a fluke, but an example of a basic form which reappears in nature in contexts, like stars and uranium nuc lei. This in turn would support the idea of 
meaning being an inherent property. 
To conclude this topic, let us consid er some new and old ex; and discuss the 
degree of inherent meaning which they have, by ourselves, to the extent that we can, in 
the shoes of an alien civilization which intercepts a weird object ...   
Two Plaques in Space 
 Consider a rectangular plaque made of an  indestructible metallic alloy which are 
engraved two dots, one immediately above th e another preceding colon shows a picture. 
Though the overall form of the might suggest th at it is an artifact, and therefore that it

Though the overall form of the might suggest th at it is an artifact, and therefore that it 
might conceal some message, two dots are simp ly not sufficient to convey anything. (Can 
before reading on, hypothesize what they are supposed to mean suppose that we made a second plaque, containing more dots, as follows.  
. 
.. 
… 
….. 
…….. 
…………… 
…………………….. 
………………………………… 
 
Now one of the most obvious things to do-so it might seer terrestrial intelligence 
at least-would be to count the dots in the successive rows. The sequence obtained is: 
 
1,   1,   2,  3,   5,   8,   13,   21,   34.

. 
.. 
… 
….. 
…….. 
…………… 
 
1,   1,   2,  3,   5,   8,   13,   21,   34. 
  
Here there is evidence of a rule governing th e progression from one the next. In fact, the 
recursive part of the definition of the Fib num bers can be inferred, with some confidence, 
from this list. Supp think of the initial pair of  values (1,1) as a "genotype" from which the 
"phenotype"-the full Fibonacci sequence-is pulle d out by a recursive rule. By sending the 
genotype alone-namely the first version pla que-we fail to send the information which

genotype alone-namely the first version pla que-we fail to send the information which 
allows reconstitution phenotype. Thus, th e genotype does not contain the full 
specification of

 182the phenotype. On the other hand, if we consider  the second version of the plaque to be 
the genotype, then there is much better cause to suppose that the phenotype could 
actually be reconstituted. This new version of  the genotype-a "long genotype"-contains so 
much information that the mechanism by which phenotype is pulled out of genotype can 
be inferred by intelligence from the genotype alone. 
Once this mechanism is firmly established as the way to pull phenotype from 
genotype, then we can go back to using "s hort genotypes"-like the first plaque. For 
instance, the "short genotype" (1,3) would yield the phenotype 
 
1, 3, 4, 7, 11, 18, 29, 47, .. . 
 -the Lucas sequence. And for every set of two initial values-that is, for every short genotype-there will be a corresponding phenotype. But the short genotypes, unlike the long ones, are only triggers-buttons to be pushed on the jukeboxes into which the

 -the Lucas sequence. And for every set of two initial values-that is, for every short genotype-there will be a corresponding phenotype. But the short genotypes, unlike the long ones, are only triggers-buttons to be pushed on the jukeboxes into which the 
recursive rule has been built. The long genotypes are informative enough that they trigger, in an intelligent being, the recognition of what kind of "jukebox" to build. In that 
sense, the long genotypes contain the inform ation of the phenotype, whereas the short 
genotypes do not. In other words, the long ge notype transmits not only an inner message, 
but also an outer message, which enables the i nner message to be read. It seems that the 
clarity of the outer message resides in th e sheer length of the message. This is not

clarity of the outer message resides in th e sheer length of the message. This is not 
unexpected; it parallels precisely what happens in deciphering ancient texts. Clearly, one's likelihood of success depends crucially on the amount of text available.  
Bach vs. Cage Again 
 
But just having a long text may not be e nough. Let us take up once more the difference 
between sending a record of Bach's music into  space, and a record of John Cage's music. 
Incidentally, the latter, being a Composition of Aleatorically Generated Elements, might 
be handily called a " CAGE ", whereas the former, being a Beautiful Aperiodic Crystal of 
Harmony, might aptly be dubbed a " BACH ". Now let's consider what the meaning of a

Harmony, might aptly be dubbed a " BACH ". Now let's consider what the meaning of a 
Cage piece is to ourselves. A Cage piece has to  be taken in a large cultural setting-as a 
revolt against certain kinds of traditions. Thus , if we want to transmit that meaning, we 
must not only send the notes of the piece, but we must have earlier communicated an 
extensive history of Western culture. It is fair to say, then, that an isolated record of John 
Cage's music does not have an intrinsic meaning. However, for a listener who is 
sufficiently well versed in Western and Easter n cultures, particularly in the trends in

sufficiently well versed in Western and Easter n cultures, particularly in the trends in 
Western music over the last few decades, it does carry meaning-but such a listener is like a jukebox, and the piece is like a pair of buttons . The meaning is mostly contained inside 
the listener to begin with; the music serves only to trigger it. And this "jukebox", unlike 
pure intelligence, is not at all unive rsal; it is highly earthbound, depending on 
idiosyncratic se-

 183 
quences of events all over our globe for l ong period of time. Hoping that John Cage's 
music will be understood by another civilizati on is like hoping that your favorite tune, on 
a jukebox on the moon, will have the same buttons as in a saloon in Saskatoon. 
On the other hand, to appreciate Bach re quires far less cultural k edge. This may 
seem like high irony, for Bach is so much more  con and organized, and Cage is so devoid 
of intellectuality. But there strange reversal here: intelligence loves patterns and balks at 
randomness For most people, the randomness in Cage's music requires much explanation; 
and even after explanations, they may feel  they are missing the message-whereas with 
much of Bach, words are superfluous. In sense,  Bach's music is more self-contained than 
Cage's music. Still, it is clear how much of the human condition is presumed by Bach. 
For instance, music has three major dimensions of structure (me harmony,

For instance, music has three major dimensions of structure (me harmony, 
rhythm), each of which can be further di vided into small intermediate, and overall 
aspects. Now in each of these dimensions, th ere is a certain amount of complexity which 
our minds can handle before boggling; clearly a composer takes this into account, mostly 
unconsciously when writing a piece. These "levels of tolerable complexity" along 
different dimensions are probably very de pendent on the peculiar conditions of our  
evolution as a species, and another intelligen t species might have developed music with

evolution as a species, and another intelligen t species might have developed music with 
totally different levels of tolerable complexity along these many dimensions. Thus a Bach piece might conceivably have to be accomp anied, by a lot of information about the 
human species, which simply could not inferre d from the music's structure alone. If we 
equate the Bach music a genotype, and the em otions which it is supposed to evoke with 
the phenotype, then what we are interested in is whether the genotype con all the information necessary for the revelation of the phenotype. 
 
How Universal Is DNA's Message?

How Universal Is DNA's Message? 
 The general question which we are facing, and wh ich is very similar t questions inspired 
by the two plaques, is this: "How much of the co necessary for its own understanding is a 
message capable of restoring? can now revert  to the original biological meanings of 
"genotype" "phenotype"- DNA  and a living organism-and ask similar quest Does DNA 
have universal triggering power? Or does it n eed a "biojukebox" to reveal its meaning? 
Can DNA  evoke a phenotype without being embedded in the proper chemical context? 
To this question to answer is no-but a qua lified no. Certainly a molecule of DNA in a

To this question to answer is no-but a qua lified no. Certainly a molecule of DNA in a 
vacuum will not create anything at all. However, if a molecule of DNA  were set to seek 
its fortune in the universe, as we imagined the BACH  and the CAGE  were, it might be 
intercepted by an intelligent civilization. They might first of all recognize its frame message. Given that, they might to try to dedu ce from its chemical structure what kind of 
chemical environment it seemed to want, and then supply such an environment. Succes-

 184sively more refined attempts along these lines might eventually lead to a full restoration 
of the chemical context necessary for the revelation of DNA's  phenotypical meaning. 
This may sound a little implausible, but if one allows many millions of years for the 
experiment, perhaps the DNA 's meaning would finally emerge. 
On the other hand, if the sequence of bases which compose a strand of DNA  were 
sent as abstract symbols (as in Fig. 41), not as a long helical molecule, the odds are 
virtually nil that this, as an outer message , would trigger the proper decoding mechanism 
which would enable the phenotype to be draw n out of the genotype. This would be a case 
of wrapping an inner message in such an abst ract outer message that the context-restoring 
power of the outer message would be lost, and so in a very pragmatic sense, the set of 
symbols would have no intrinsic meaning. Lest you think this all sounds hopelessly

symbols would have no intrinsic meaning. Lest you think this all sounds hopelessly 
abstract and philosophical, consider that the exact moment when phenotype can be said to 
be "available", or "implied", by genotype, is a highly charged issue in our day: it is the 
issue of abortion. 
 
FIGURE 41. This Giant Aperiodic Crystal is the base sequence for the chromosome of 
bacteriophage OX174. It is the first complete  genome ever mapped out for any organism. 
About 2,000 of these boustrophedonic pages would be needed to show the base sequence 
of a single E. Coli cell, and about one milli on pages to show the base sequence of the

of a single E. Coli cell, and about one milli on pages to show the base sequence of the 
DNA of a single human cell. The book now in  your hands contains roughly the same 
amount of information as a molecular blueprint for one measly E. Coli cell.

Chromatic Fantasy, And Feud. 
 
Having had a splendid dip in the pond, the Tortoise is just crawling out and 
shaking himself dry, when who but Achilles walks by. 
 
Tortoise: Ho there, Achilles. I was just thinking of you as I splash around in the pond. Achilles: Isn't that curious? I was just th inking of you, too, while I meandered through the 
meadows. They're so green at this time of year. 
Tortoise: You think so? It reminds me of a thought I was hoping to share with you. 
Would you like to hear it? 
Achilles: Oh, I would be delighted. That is, I would be delighted as long you're not going 
to try to snare me in one of your wicked traps of log Mr. T. 
Tortoise: Wicked traps? Oh, you do me wrong. Would I do anything wicked? I'm a

Tortoise: Wicked traps? Oh, you do me wrong. Would I do anything wicked? I'm a 
peaceful soul, bothering nobody and leading a gent; herbivorous life. And my thoughts merely drift among the oddities and quirks of how things are (as I see them). I, humble observer phenomena, plod along and puff my silly words into the air rather unspectacularly, I am afraid. But to reassure you about my intention I was only planning to speak of my Tortoise-shell today, and as you know, those things have nothing-nothing whatsoever-to do with logic! 
Achilles: Your words Do reassure me, Mr. T. And, in fact, my curiosity quite piqued. I 
would certainly like to listen to what you have to say even if it is unspectacular.

would certainly like to listen to what you have to say even if it is unspectacular. 
Tortoise: Let's see ... how shall I begin? Hmm ... What strikes you me about my shell, 
Achilles? 
Achilles: It looks wonderfully clean! Tortoise: Thank you. I just went swimming and washed off several layers of dirt which 
had accumulated last century. Now you can see ho green my shell is. 
Achilles: Such a good healthy green shell, it's nice to see it shining in sun. Tortoise: Green? It's not green. Achilles: Well, didn't you just tell me Tortoise: I did. Achilles: Then, we agree: it is green. Tortoise: No, it isn't green. Achilles: Oh, I understand your game. You're hinting to me that what you say isn't 
necessarily true; that Tortoises play with language; that your statements and reality don't necessarily match; that --

Tortoise:  I certainly am not. Tortoises treat words as sacred. Tortoises revere accuracy. 
Achilles: Well, then, why did you say that your shell is green, and that it is not green 
also? 
Tortoise: I never said such a thing; but I wish I had. Achilles: You would have liked to 
say that? 
Tortoise: Not a bit. I regret saying it, and disagree wholeheartedly with it. Achilles: That 
certainly contradicts what you said before! 
Tortoise: Contradicts? Contradicts? I never c ontradict myself. It's not part of Tortoise-
nature. 
Achilles: Well, I've caught you this time, you slippery fellow, you. Caught you in a full-
fledged contradiction. 
Tortoise: Yes, I guess you did. Achilles: There you go again! Now you're contradicting yourself more and more! You are 
so steeped in contradiction it's impossible to argue with you! 
Tortoise: Not really. I argue with myself without any trouble at all. Perhaps the problem

nature. 
Tortoise: Not really. I argue with myself without any trouble at all. Perhaps the problem 
is with you. I would venture a guess that maybe you're the one who's contradictory, but you're so trapped in your own tangled web that you can't see how inconsistent you're being. 
Achilles: What an insulting suggestion! I'm going to show you that you're the 
contradictory one, and there are no two ways about it. 
Tortoise: Well, if it's so, your task ought to be cut out for you. What could be easier than 
to point out a contradiction? Go ahead-try it out.

to point out a contradiction? Go ahead-try it out. 
Achilles: Hmm ... Now I hardly know where to begin. Oh ... I know. You first said that 
(1) your shell is green, and then you went on to say that (2) your shell is not green. What more can I say? 
Tortoise: Just kindly point out the contradiction. Quit beating around the bush. 
Achilles: But-but-but ... Oh, now I begin to see.  (Sometimes I am so slow-witted!) It must

Achilles: But-but-but ... Oh, now I begin to see.  (Sometimes I am so slow-witted!) It must 
be that you and I differ as to what constitutes a contradiction. That's the trouble. Well, let me make myself very clear: a contradiction occurs when somebody says one thing and denies it at the same time. 
Tortoise: A neat trick. I'd like to see it done. Probably ventriloquists would excel at 
contradictions, speaking out of both sides of their mouth, as it were. But I'm not a ventriloquist. 
Achilles: Well, what I actually meant is just that somebody can say one thing and deny it 
all within one single sentence! It doesn't literally have to be in the same instant.

all within one single sentence! It doesn't literally have to be in the same instant. 
Tortoise: Well, you didn't give ONE sentence. You gave TWO. Achilles: Yes-two sentences that contradict each other! Tortoise: I am sad to see the tangled structure of your thoughts becoming so exposed, 
Achilles. First you told me that a contradiction is some thing which occurs in a single sentence. Then you told me that you

Found a contradiction in a pair of sentences I uttered. Frankly, it’s just as I said. Your 
own system of thought is so delusional that you manage to avoid seeing how inconsistent it is. From the outside, however plain as day. 
Achilles: Sometimes I get so confused by your diversionary tactics tl can't quite tell if 
we're arguing about something utterly petty, or something deep and profound! 
Tortoise: I assure you, Tortoises don't spend their time on the petty. Hence it's the latter. Achilles: I am very reassured. Thank you. Now I have had a moment to reflect, and I see 
the necessary logical step to convince you that you contradicted yourself. 
Tortoise: Good, good. I hope it's an easy step, an indisputable one. Achilles: It certainly is. Even you will agree with it. The idea is that you believed 
sentence 1 ("My shell is green"), AND you believed sentence 2 ("My shell is not green"), you would believe one compound( sentence in which both were combined, wouldn't you?

sentence 1 ("My shell is green"), AND you believed sentence 2 ("My shell is not green"), you would believe one compound( sentence in which both were combined, wouldn't you? 
Tortoise: Of course. It would only be reas onable ... providing just that the manner of 
combination is universally acceptable. But I'm sure we'll agree on that. 
Achilles: Yes, and then I'll have you! The combination I propose is Tortoise: But we must be careful in combining sentences. For instance you'd grant that 
"Politicians lie" is true, wouldn't you? 
Achilles: Who could deny it? Tortoise: Good. Likewise, "Cast-iron sinks" is a valid utterance, isn't it? Achilles: Indubitably. Tortoise: Then, putting them together, we get "Politicians lie in cast iron sinks". Now 
that's not the case, is it?

that's not the case, is it? 
Achilles: Now wait a minute ... "Politicians lie in cast-iron sinks?" N no, but Tortoise: So, you see, combining two true sentences in one is not a policy, is it? 
Achilles: But you-you combined the two-in such a silly way! 
Tortoise: Silly? What have you got to object to in the way I combined them Would you 
have me do otherwise?

have me do otherwise? 
Achilles: You should have used the word "and", not "in". Tortoise: I should have? You mean, if YOU'D had YOUR way, I should h; Achilles: No-it's the LOGICAL thing to do. It's got nothing to do with personally. Tortoise: This is where you always lose me, when you resort to your L and its high-
sounding Principles. None of that for me today, plea 
Achilles: Oh, Mr. Tortoise, don't put me through all this agony. You k very well that 
that's what "and" means! It's harmless to combine true sentences with "and"! 
Tortoise: "Harmless", my eye! What gall! This is certainly a pernicious plot

to entrap a poor, innocent, bumbling Tortoise in a fatal contradiction. If it were so 
harmless, why would you be trying so bloody hard to get me to do it? Eh? 
Achilles: You've left me speechless. You make me feel like a villain, where I really had 
only the most innocent of motivations. 
Tortoise: That's what everyone believes of himself... Achilles: Shame on me-trying to outwit you, to use words to snare you in a self-
contradiction. I feel so rotten. 
Tortoise: And well you should. I know what you were trying to set up. Your plan was to 
make me accept sentence 3, to wit: "My shell is green and my shell is not green". And such a blatant falsehood is repellent to the Tongue of a Tortoise. 
Achilles: Oh, I'm so sorry I started all this. Tortoise: You needn't be sorry. My feelings aren't hurt. After all, I'm used to the 
unreasonable ways of the folk about me. I enjoy your company, Achilles, even if your thinking lacks clarity.

unreasonable ways of the folk about me. I enjoy your company, Achilles, even if your thinking lacks clarity. 
Achilles: Yes ... Well, I fear I am set in my ways, and will probably continue to err and 
err again, in my quest for Truth. 
Tortoise: Today's exchange may have served a little to right your course. Good day, 
Achilles. 
Achilles: Good day, Mr. T.

The Propositional Calculus 
 189CHAPTER VII 
 
Words and Symbols 
 
THE PRECEDING DIALOGUE is reminiscent of the Two-Part Invention  by Lewis 
Carroll. In both, the Tortoise refuses to use normal, ordinary in the normal, ordinary way-
or at least he refuses to do so when it is hi s advantage to do so. A way to think about the 
Carroll paradox was given last Chapter. In this  Chapter we are going to make symbols dc 
Achilles couldn't make the Tortoise do with his words. That is, we are to make a formal 
system one of whose symbols will do just wh at A wished the word `and' would do, when 
spoken by the Tortoise, and ail of whose symbols will behave the way the words 'if... then 
. . .' ought to behave. There are only two othe r words which we will attempt to deal with 
`or' and `not'. Reasoning which depends only on correct usage of these words is termed 
propositional reasoning . 
 
Alphabet and First Rule of the Propositional Calculus

Alphabet and First Rule of the Propositional Calculus 
 I will present this new formal system, calle d the Propositional Calculus, like a puzzle, not 
explaining everything at once, but letting you th ings out to some extent. We begin with 
the list of symbols: 
 
                                                     <      > 
                                              P     Q     R      
´ 
                                              ∧     ∨     ⊃      ~ 
 
The first rule of this system that I will reveal is the following: 
 
RULE OF JOINING: If x and y are theorems of the system, then so is the string < x ∧y >.

                                                     <      > 
                                              P     Q     R      
´ 
                                              ∧     ∨     ⊃      ~ 
 
RULE OF JOINING: If x and y are theorems of the system, then so is the string < x ∧y >. 
 
This rule takes two theorems and combin es them into one. It s remind you of the 
Dialogue.  
Well-Formed Strings 
 There will be several other rules of inference, and they will all be pres shortly-but first, it 
is important to define a subset of all strings, namely the

 190well formed  strings. They will be defined in a recursive way. We begin with the 
ATOMS:  P, Q,  and R are called atoms.. New atoms are formed by appending primes 
onto the right of old atoms-thus, R', Q", P "', etc. This gives an endless supply of atoms. 
All atoms are well-formed. 
 
Then we have four recursive 
 
FORMATION RULES: If x and y are well-form ed, then the following four strings are 
also well-formed:   (1) ~x   
(2) < x ∧y>   
(3) < x ∨y>   
(4) < x ⊃y>  
  
For example, all of the following are well-formed: 
  
P      atom 
~ P      by (1) 
~~P     by (1) 
Q ´      atom 
~Q1     by (1) 
<P∧~Q' >    by (2) 
~<P∧~Q' >    by (1) 
~~<P⊃~Q' >    by (4) 
<~<P∧~Q' >∨~~<P⊃~Q' >>   by (3) 
 
The last one may look quite formidable, but  it is built up straightforwardly from two 
components-namely the two lines just above it.  Each of them is in turn built up from 
previous lines ... and so on. Every well-formed st ring can in this way be traced back to its

P      atom 
~ P      by (1) 
~~P     by (1) 
Q ´      atom 
~Q1     by (1) 
 
previous lines ... and so on. Every well-formed st ring can in this way be traced back to its 
elementary constituents-that is, atoms. You simply run the formation rules backwards until you can no more. This process is guaranteed to terminate, since each formation rule (when run forwards) is a lengthening  rule, so that running it backwards always drives you 
towards atoms. 
This method of decomposing strings thus serves as a check on the well-
formedness of any string. It is a top-down decision procedure  for wellformedness. You

formedness of any string. It is a top-down decision procedure  for wellformedness. You 
can test your understanding of this deci sion procedure by checking which of the 
following strings are well-formed:  
 
(1) <P> 
(2) (2)  <~P> 
(3) <P∧Q∧R> 
(4) <P∧Q> 
(5) <<P∧Q>∧Q~∧P>> 
(6) <P∧~P> 
(7) <<P∨<Q⊃R>>∧<~P∨~R´>> 
(8) <P∧Q>∧< Q∧P:

 191 
(Answer: Those whose numbers are Fibonacci numbers are not formed. The rest are well-
formed.)  
More Rules of Inference 
 Now we come to the rest of the rules by which theorems of this system constructed. A few rules of inference follow. In all of them , the symbols ´x´ and 'y' are always to be 
understood as restricted to well formed  strings 
 
RULE OF SEPARATION: If < x ∧y> is a theorem, then both x and theorems. 
 
Incidentally, you should have a pretty good gue ss by now as to concept the symbol `A' 
stands for. (Hint: it is the troublesome word  the preceding Dialogue.) From the following 
rule, you should be a figure out what concept the tilde ('~') represents: 
 DOUBLE-TILDE RULE: The string '~~' can be dele ted from any theorem. It can also be 
inserted into any theorem, provided that the rest string is itself well-formed.   
The Fantasy Rule 
 Now a special feature of this system is that it has no axioms -only rule you think back to

 Now a special feature of this system is that it has no axioms -only rule you think back to 
the previous formal systems we've seen, you may w( how there can be any theorems, 
then. How does everything get started? The an swer is that there is one rule which 
manufactures theorems from out of thin air- it doesn't need an "old theorem" as input. 
(The rest of the do require input.)  This special rule is called the fantasy rule . The reason I 
call it that is quite simple. 
To use the fantasy rule, the first thing you do is to write down an well-formed 
string x you like, and then "fantasize" by asking, " What if  string x were an axiom, or a

string x you like, and then "fantasize" by asking, " What if  string x were an axiom, or a 
theorem?" And then, you let the system give an answer. That is, you go ahead and make a 
derivation with x ; opening line; let us suppose y is the last line. (Of course the derivation 
must strictly follow the rules of the system.)  Everything from x to y (inclusive) is the 
fantasy ; x is the premise  of the fantasy, and y is its outcome . The next step is to jump out 
of the fantasy, having learned from it that out. 
 
If x were a theorem, y would be a theorem.

If x were a theorem, y would be a theorem. 
 Still, you might wonder, where is the real theorem? The real theorem is the string 
 
<x⊃y> 
 
Notice the resemblance of this string to the sentence printed above 
To signal the entry into, and emergence from, a fantasy, one uses the

 192square brackets `[' and ']', respectively. Thus , whenever you see a left square bracket, you 
know you are "pushing" into a fantasy, and the next line will contain the fantasy's 
premise . Whenever you see a right square bracket, you know you are "popping" back out, 
and the preceding  line was the outcome. It is helpful (though not necessary) to indent 
those lines of a derivation which take place in fantasies. 
Here is an illustration of the fantasy ru le, in which the string P is taken as a 
premise. (It so happens that P is not a theo rem, but that is of no import; we are merely 
inquiring, "What if it were?") We make the following fantasy: 
 [     push into fantasy 
  P     premise 
   ~~~P     outcome (by double tilde rule) 
]     pop out of fantasy 
 The fantasy shows that:  If P were a theorem, so would ~~P be one. 
 We now "squeeze" this sentence of Englis h (the metalanguage) into the formal

  P     premise 
 We now "squeeze" this sentence of Englis h (the metalanguage) into the formal 
notation (the object language): <P ⊃~~P>. This, our first theorem of the Propositional 
Calculus, should reveal to you the intended interpretation of the symbol ` ⊃'. 
Here is another derivation using the fantasy rule: 
 [     push 
   <P∧Q>    premise 
    P     separation 
    Q     separation 
    <Q∧P>    joining 
]     pop 
<<P∧Q>⊃<Q∧P>>   fantasy rule 
 
It is important to understand that only the last  line is a genuine theorem, here-everything 
else is in the fantasy.  
Recursion and the Fantasy Rule

 [     push 
]     pop 
 
Recursion and the Fantasy Rule 
 
As you might guess from the recursion termi nology "push" and "pop", the fantasy rule 
can be used recursively-thus, there can be fantasies within fantasies, thrice-nested 
fantasies, and so on. This means that there are all sorts of "levels of reality", just as in 
nested stories or movies. When you pop out of a movie-within-a-movie, you feel for a 
moment as if you had reached the real worl d, though you are still one level away from the 
top. Similarly, when you pop out of a fantasy-within-a-fantasy, you are in a "realer"

top. Similarly, when you pop out of a fantasy-within-a-fantasy, you are in a "realer" 
world than you had been, but you are still one level away from the top. 
Now a "No Smoking" sign inside a movie theater does not apply to the

 193 
characters in the movie-there is no carry-over from the real world in fantasy world, in 
movies. But in the Propositional Calculus, then  carry-over from the real world into the 
fantasies; there is even carry from a fantasy to fantasies inside it. This is formalized by 
the following rule:  CARRY-OVER RULE: Inside a fantasy, any theorem from the "reality level higher can 
be brought in and used.  
It is as if a "No Smoking" si gn in a theater applied not only to a moviegoers, but also to 
all the actors in the movie, and, by repetition of  the  same idea, to anyone inside multiply 
nested movies! (Warning: There carry-over in the reverse direction: theorems inside fantasies cannot be exported to the exterior! If  it weren't for this fact, you could write any 
as the first line of a fantasy, and then lift it out into the real world as a  theorem.) 
 To show how carry-over works, and to s how how the fantasy rule can be used 
recursively, we present the following derivation:

recursively, we present the following derivation: 
 [     push 
   P     premise of outer fantasy 
   [     push again 
       Q     premise of inner fantasy 
       P     carry-over of P into inner fantasy 
       <P∧Q>    joining 
   ]     pop out of inner fantas y, regain outer fantasy 
   <Q⊃<P∧Q>>   fantasy rule 
]     pop out of outer fantasy, reach real world! 
<P⊃<Q⊃<P∧Q>>>   fantasy rule 
 
Note that I've indented the outer fantas y once, and the inner fantasy twice, to 
emphasize the nature of these nest ed "levels of reality". One to look at the fantasy rule is

[     push again 
 
emphasize the nature of these nest ed "levels of reality". One to look at the fantasy rule is 
to say that an observation made about  the system is inserted into the system. Namely, the 
theorem < x ⊃y> which gets produced can be thought of as a representation inside the 
system of the statement about the system "If x is  a theorem, then y is too". To be specific, 
the intended interpretation for <P⊃Q> is "if P, then Q equivalently, " P implies Q". 
 
The Converse of the Fantasy Rule

The Converse of the Fantasy Rule 
 
Now Lewis Carroll's Dialogue was all about "if-then" statements. In particular, Achilles had a lot of trouble in persuading the Tortoise  to accept the second clause of an "if-then" 
statement, even when the "if-then" state itsel f was accepted, as well as its first clause. The 
next rule allows y infer the second "clause" of a' ⊃'-string, provided that the ` ⊃'-string it a 
theorem, and that its first "clause" is also a theorem.

 194RULE OF DETACHMENT: If x and < x ⊃y> are both theorems, then y is a theorem. 
 
Incidentally, this rule is often called "Modus Ponens", and the fantasy rule is often called the "Deduction Theorem".  
The Intended Interpretation of the Symbols 
 We might as well let the cat out of the bag at  this point, and reveal the "meanings" of the 
rest of the symbols of our new system. In cas e it is not yet apparent, the symbol `A' is 
meant to be acting isomorphically to the nor mal, everyday word `and'. The symbol '-' 
represents the word 'not'-it is a formal sort  of negation. The angle brackets '<' and `>' are 
groupers-their function being very similar to that of parentheses in ordinary algebra. The main difference is that in algebra, you have th e freedom to insert parentheses or to leave 
them out, according to taste and style, whereas in a formal system, such anarchic freedom 
is not tolerated. The symbol ' ∨' represents the word `or' ('vel'  is a Latin word for `or'). The

is not tolerated. The symbol ' ∨' represents the word `or' ('vel'  is a Latin word for `or'). The 
`or' that is meant is the so-called inclusive  `or', which means that the interpretation of 
<x∨y> is "either x or y-or both". 
 
The only symbols we have not interpreted are the atoms. An atom has no single 
interpretation-it may be interpreted by any sent ence of English (it must continue to be 
interpreted by the same sentence if it occurs multiply within a string or derivation). Thus, 
for example, the well-formed string < P∧~P> could be interpreted by the compound 
sentence

sentence 
 
This mind is Buddha, and this mind is not Buddha.  Now let us look at each of the theorems so far derived, and interpret them. The first one 
was < P⊃~~P>. If we keep the same interpretation for P, we have the following 
interpretation: 
 If this mind is Buddha, 
then it is not the case that this mind is not Buddha. 
 Note how I rendered the double negation. It is  awkward to repeat a negation in any 
natural language, so one gets around it by us ing two different ways of expressing

sentence 
 
natural language, so one gets around it by us ing two different ways of expressing 
negation. The second theorem we derived was <<P∧Q>⊃<Q∧P>>.  If we let Q be 
interpreted by the sentence "This flax weighs  three pounds", then our theorem reads as 
follows:  If this mind is Buddha and this flax weighs three pounds,  
then this flax weighs three pounds and this mind is Buddha. 
 
The third theorem was < P⊃<Q⊃<P∧Q>>>. This one goes into the following nested "if-
then" sentence:

 195 
If this mind is Buddha, 
then, if this flax weighs three pounds, 
then this mind is Buddha and this flax weighs three pounds.  
You probably have noticed that each theorem, when interpreted, something 
absolutely trivial and self-evident. (Sometim es they are so s evident that they sound 
vacuous and-paradoxically enough-confusing or  even wrong!) This may not be very 
impressive, but just remember there are plen ty of falsities out there which could have 
been produced they weren't. This system-the Propositional Calculus-steps neatly ft truth to truth, carefully avoiding all falsities, just as a person who is concerned with staying dry 
will step carefully from one stepping-stone creek to the next, following the layout of 
stepping-stones no matter I twisted and tricky it might be. What is impressive is that-in 
the Propositional Calculus-the whole thing is done purely typographically . There is 
nobody down "in there", thinking about the meaning  of the strings. It i! done

nobody down "in there", thinking about the meaning  of the strings. It i! done 
mechanically, thoughtlessly, rigidly, even stupidly. 
 
Rounding Out the List of Rules 
 We have not yet stated all the rules of th e Propositional Calculus. The complete set of 
rules is listed below, including the three new ones.  
JOINING RULE: If x and y are theorems, then < x ∧y> is a theorem. 
 
SEPARATION RULE: If < x ∧y> is a theorem, then both x and y are theorems. 
 
DOUBLE-TILDE RULE: The string '~~' can be deleted from any theorem can also be 
inserted into any theorem, provided that the result string is itself well-formed. 
 
FANTASY RULE: If y can be derived when x is assumed to be a theorem then < x ⊃y> is 
a theorem.

a theorem. 
 
CARRY-OVER RULE: Inside a fantasy, any th eorem from the "reality" c level higher 
can be brought in and used. 
  
RULE OF DETACHMENT: If x and < x ⊃y> are both theorems, then y is a theorem.  
 
CONTRAPOSITIVE RULE: <x ⊃y> and <~y ⊃~x> are interchangeable  
 
DE MORGAN'S RULE: <~x ∧~y> and ~< x ∨y> are interchangeable.  
 
SWITCHEROO RULE: <x ∨y> and <~x ⊃y> are interchangeable. 
 
(The Switcheroo rule is named after Q. q. Sw itcheroo, an Albanian railroad engineer who

a theorem. 
 
(The Switcheroo rule is named after Q. q. Sw itcheroo, an Albanian railroad engineer who 
worked in logic on the siding.) By "intercha ngeable" in foregoing rules, the following is 
meant: If an expression of one form occurs as either a theorem or part of a theorem, the 
other form may be

 196substituted, and the resulting string will also be a theorem. It must be kept in mind that 
the symbols ‘x’ and ‘y’ always stand for well-formed strings of the system.  
Justifying the Rules 
 Before we see these rules used inside de rivations, let us look at some very short 
justifications for them. You can probably justify them to yourself better than my 
examples – which is why I only give a couple.  The contrapositive rule expresses explicitly a way of turning around conditional statements which we carry out unconsciously. For instance, the “Zentence”  
If you are studying it, then you are far from the Way 
 Means the same thing as  
If you are close to the Way, then you are not studying it. 
  De Morgan’s rule can be illustrated by our familiar sentence “The flag is not 
moving and the wind is not moving”. If P symbolizes “the flag is not moving”, and Q 
symbolizes “the wind is moving”, then  the compound sentence is symbolized by

symbolizes “the wind is moving”, then  the compound sentence is symbolized by 
<~P∧~Q>, which, according to Morgan’s law, is interchangeable with ~<P∨Q>. whose 
interpretation would be “It is not true that either the flag or the wind is moving”. And no 
one could deny that it is a Zensible conclusion to draw.  For the Switrcheroo rule, consider th e sentence “Either a cloud is hanging over 
the mountain, or the moonlight is penetrating the waves of the lake,” which might be spoken, I suppose, by a wistful Zen master re membering a familiar lake which he can 
visualize mentally but cannot see. Now hang on to your seat, for the Swircheroo rule tells

visualize mentally but cannot see. Now hang on to your seat, for the Swircheroo rule tells 
us that this is interchangeable with the thought “If a cloud is not hanging over the 
mountain, then the moonlight is penetrating the waves of the lake.” This may not be 
enlightenment, but it is the best the Propositional Calculus has to offer.  
Playing around with the system 
 Now, let us apply these rules to a previous theorem, ands see what we get: For instance, 
take the theorem <P⊃~~P> : 
 
<P⊃~~P> :   old theorem 
<~~~P⊃~P>:   contrapositive 
<~P⊃~P>   double-tilde  
<P∨~P>   switcheroo

<P∨~P>   switcheroo 
 
This new theorem, when interpreted, says:

 197Either this mind is Buddha, or this mind is not Buddha 
 Once again, the interpreted theorem, though perhaps less than mind boggling, is at least true.  
Semi-Interpretations 
 It is natural, when one reads theorems of the Propositional Calculus out loud, to interpret 
everything but the atoms. I call this se mi-interpreting. For example, the semi-
interpretation of <P∨~P>:: would be 
 
P or not P. 
 Despite the fact that P is not a sentence, the above semisentence still sounds true, because 
you can very easily imagine sticking any senten ce in for P – and the form of the semi-
interpreted theorem assures you that however you make your choice, the resulting sentence will be true. And that is the key idea of the Propositional Calculus: it produces 
theorems which, when semi-interpreted, are seen to be “universally true semisaentences”, by which is meant that no matter how you comple te the interpretation, the final result will 
be a true statement.  
Ganto’s Ax

Ganto’s Ax 
 Now we can do a more advanced exercise, based on a Zen koan called “Ganto’s Ax”. Here is how it began.  
One day Tokusan told his student Ganto, “I  have two monks who have been here 
for many years. Go and examine them.” Gant o picked up an ax and went to the hut 
where the two monks were meditating. He raised the ax, saying “If you say a word, 
I will cut off your heads; and if you do not  say a word, I will also cut off your 
heads.”
1 
 If you say a word I will cut off this koan, and if you do not say a word, I will also cut off

heads.”
1 
 If you say a word I will cut off this koan, and if you do not say a word, I will also cut off 
this koan – because I want you to translate so me of it into our notation. Let us symbolize 
“you say a word” by P and “I will cut off your heads” by Q. Then Ganto’s ax threat is 
symbolized by the string <<P⊃Q>∧<~`P⊃Q>> . What if this ax threat were an axiom? 
Here is a fantasy to answer that question. 
 (1)   [      push 
(2)       <<P⊃Q>∧<~`P⊃Q>> .  Ganto’s axiom 
(3)       <P⊃Q>    separation 
(4)        <~Q⊃~P>.    contrapositive 
(5)        <~ P⊃Q>    separation 
(6)        <~Q⊃~~P> .    contrapositive 
(7)         ]     push again 
(8)              ~ Q      premise

 198(9)           <~Q⊃~P>.   carry-over of line 4 
(10)          ~P    detachment  
(11)         <~Q⊃~~P> .   carry-over of line 6 
(12)          ~~P    detachment (lines 8 and 11)  
(13)         < ~P∧~~P>   joining 
(14)           < ~P∨~~P>   De Morgan 
(15)      ]    pop once 
(16)      <~Q⊃~<P∨~P>> .  fantasy rule 
(17)      <~P∨~P>⊃Q>.  contrapositive 
(18)      [    push 
(19)      .  ~P    premise (also outcome) 
(20)      ]    pop 
(21)      <~P⊃~P>.   fantasy rule 
(22)      <P∨~P>.   switcheroo 
(23)      Q    detachment (lines 22 and 17) 
(24)    ]    pop out  The power of the Propositional Calculus is show n in this example. Why, in but two dozen 
steps, we have deduced Q: that the heads will be cut off! (Ominously, the rule last 
invoked was "detachment" ...) It might seem s uperfluous to continue the koan now, since 
we know what must ensue ... However, I shall drop my resolve to cut the koan off; it is a

(20)      ]    pop 
we know what must ensue ... However, I shall drop my resolve to cut the koan off; it is a 
true Zen koan, after all. The rest of the incident is here related:  
Both monks continued their meditation as  if he had not spoken. Ganto dropped the 
ax and said, "You are true Zen students." He returned to Tokusan and related the incident. "I see your side well," Tokusan ag reed, "but tell me, how is their side?" 
"Tõzan may admit them," replied Ganto, "but they should not be admitted under Tokusan."2 
 Do you see my side well? How is the Zen side?  
Is There a Decision Procedure for Theorems?

Is There a Decision Procedure for Theorems? 
 The Propositional Calculus gives us a set of rules for producing statements which would 
be true in all conceivable worlds. That is w hy all of its theorems sound so simple-minded; 
it seems that they have absolutely no c ontent! Looked at this way, the Propositional 
Calculus might seem to be a waste of time, since what it tells us is absolutely trivial. On the other hand, it does it by specifying the form  of statements that are universally true,

Calculus might seem to be a waste of time, since what it tells us is absolutely trivial. On the other hand, it does it by specifying the form  of statements that are universally true, 
and this throws a new kind of light onto the core truths of the universe: they are not only 
fundamental, but also regular : they can be produced by one set of typographical rules. To 
put it another way, they are all "cut from th e same cloth". You might consider whether 
the same could be said about Zen koans: could they all be produced by one set of 
typographical rules? 
It is quite relevant here to bring up the question of a decision procedure. That is,

It is quite relevant here to bring up the question of a decision procedure. That is, 
does there exist any mechanical method to tell nontheorems from theorems? If so, that would tell us that the set of theorems of the

 199Propositional Calculus is not only r.e., but also recursive. It turns out that there is an 
interesting decision procedure-the method of trut h u would take us a bit afield to present 
it here; you can find it in almost any standa rd book on logic. And what about Zen koans? 
Could there conceivably be a mechanical d ecision procedure which distinguishes genuine 
Zen koans from other things?  
Do We Know the System Is Consistent? 
 Up till now, we have only presumed  that all theorems, when interpreted as indicated, are 
true statements. But do we know  that that is the case' we prove it to be? This is just 
another way of asking whether the intended interpretations ('and' for ` ∧', etc.) merit being 
called the "passive meanings” of the symbols. One can look at this issue from two very 
different points of view, whic h might be called the "prudent" and "imprudent" points I 
will now present those two sides as I see th em, personifying their as "Prudence" and

will now present those two sides as I see th em, personifying their as "Prudence" and 
"Imprudence".  Prudence: We will only KNOW that all theorems come out true un intended 
interpretation if we manage to PROVE it. That is the c: thoughtful way to proceed. 
Imprudence: On the contrary. It is OBVIOUS that all theorems will come out true. If you 
doubt me, look again at the rules of the sy stem. You will find that each rule makes a 
symbol act exactly as the word it represents  ought to be used. For instance, the joining 
rule makes the symbol ‘ ∧’ act as `and' ought to act; the rule of detachment makes ` ⊃'

rule makes the symbol ‘ ∧’ act as `and' ought to act; the rule of detachment makes ` ⊃' 
act as it ought to, if it is to stand for 'imp lies', or 'if-then'; and so on. Unless you are 
like the Tortoise, you will recognize in each ru le a codification of a pattern you use in 
your own thought patterns. So if you trus t your own thought patterns, then you HAVE 
to believe that all theorems come out true ! That's the way I see it. I don't need any 
further proof. If you think that some theo rem comes out false, then presumably you 
think that some rule must be wrong. Show me which one.

think that some rule must be wrong. Show me which one. 
Prudence: I'm not sure that there is any faulty  rule, so I can't point one out to you. Still, I 
can imagine the following kind of scenario. You, following the rules, come up with a theorem -- say x. Meanwhile I, also follo wing the rules, come up with another 
theorem-it happens to be ~x.           Can't you force yourself to conceive of that? 
Imprudence: All right; let's suppose it happene d. Why would it bother you? Or let me put 
it another way. Suppose that in playing with the MIU -system, I came up with a

it another way. Suppose that in playing with the MIU -system, I came up with a 
theorem x, and you came up with xU Can you force yourself to conceive of that? 
Prudence: Of course-in fact both MI and MIU  are theorems. 
Imprudence: Doesn't that bother you? Prudence: Of course not. Your example is ridiculous, because MI and MIU  are not 
CONTRADICTORY, whereas two strings x and ~x in the Propositional Calculus 
ARE contradictory.

 200Imprudence: Well, yes -- provided you wish to in terpret `~' as `not'. But what would lead 
you to think that '~' should be interpreted as `not'? 
Prudence: The rules themselves. When you look at them, you realize that the only 
conceivable interpretation for '~' is 'not'-and likewise, the only conceivable 
interpretation for ` ∧' is `and', etc. 
Imprudence: In other words, you are convinced that the rules capture the meanings of 
those words? 
Prudence: Precisely. Imprudence: And yet you are still willing to entertain the thought that both x and ~x 
could be theorems? Why not also entertain the notion that hedgehogs are frogs, or that 
1 equals 2, or that the moon is made of gr een cheese? I for one am not prepared even 
to consider whether such basic ingred ients of my thought processes are wrong -- 
because if I entertained that notion, then I would also have to consider whether my 
modes of analyzing the entire question are also wrong, and I would wind up in a total

modes of analyzing the entire question are also wrong, and I would wind up in a total 
tangle. 
Prudence: Your arguments are forceful ... Ye t I would still like to see a PROOF that all 
theorems come out true, or that x and ~x can never both be theorems. 
Imprudence: You want a proof. I guess that m eans that you want to be more convinced 
that the Propositional Calculus is consis tent than you are convinced of your own 
sanity. Any proof I could think of would involve mental operations of a greater 
complexity than anything in the Propositional Calculus itself. So what would it prove? 
Your desire for a proof of consistency of  the Propositional Calculus makes me think

Your desire for a proof of consistency of  the Propositional Calculus makes me think 
of someone who is learning English and in sists on being given a dictionary which 
definers all the simple words in terms of complicated ones... 
 
The Carroll Dialogue Again 
 This little debate shows the difficulty of trying to use logic and reasoning to defend 
themselves. At some point, you reach rock bo ttom, and there is no defense except loudly 
shouting, "I know I'm right!" Once again, we  are up against the issue which Lewis

shouting, "I know I'm right!" Once again, we  are up against the issue which Lewis 
Carroll so sharply set forth in his Dialogue: you can't go on defending your patterns of reasoning forever. There comes a point where faith takes over. 
A system of reasoning can be compared to an egg. An egg has a shell which 
protects its insides. If you want to ship an egg somewhere, though, you don't rely on the 
shell. You pack the egg in some sort of  container, chosen according to how rough you 
expect the egg's voyage to be. To be extr a careful, you may put the egg inside several 
nested boxes. However, no matter how many layers of boxes you pack your egg in, you

nested boxes. However, no matter how many layers of boxes you pack your egg in, you 
can imagine some cataclysm which could break the egg. But that doesn't mean that you'll 
never risk transporting your egg. Similarly, one  can never give an ultimate, absolute 
proof that a proof in some system is correct. Of course,

 201one can give a proof of a proof, or a proof of a proof of a pr oof – but the  validity of the 
outermost system always remains an unproven assumption, accepted on faith. One can 
always imagine that some unsuspected subtlety will invalidate every single level of proof down to the bottom, and tI "proven" result will be  seen not to be correct after all. But that 
doesn’t mean that mathematicians and logicians are constantly worrying that the whole edifice of mathematics might be wr ong. On the other hand, unorthodox proofs are 
proposed, or extremely lengthy proofs, or pr oofs generated by computers, then people do 
stop to think a bit about what they really mean by that quasi-sacred word "proven". 
An excellent exercise for you at this poi nt would be to go back Carroll Dialogue, 
and code the various stages of the debate in to our notation -- beginning with the original 
bone of contention: 
 
Achilles: If you have <<A∧B>⊃Z>, and you also have <A∧B>, then surely you have Z.

Achilles: If you have <<A∧B>⊃Z>, and you also have <A∧B>, then surely you have Z. 
Tortoise: Oh! You mean: <<<<A∧B>⊃Z>∧<A∧B>>⊃Z>, : don't you? 
 
(Hint: Whatever Achilles considers a rule of in ference, the Tortoise immediately flattens 
into a mere string of the system. If you use or letters A, B , and Z, you will get a recursive 
pattern of longer and strings.)  
Shortcuts and Derived Rules 
 When carrying out derivations in the Propositi onal Calculus, one quickly invents various 
types of shortcut, which are not strictly part  of the system For instance, if the string

types of shortcut, which are not strictly part  of the system For instance, if the string 
<Q∨~Q> were needed at some point, and <P∨~P> had been derived earlier, many people 
would proceed as if <Q∨~Q> had been derived, since they know  that its derivation is an 
exact parallel to that of <P∨~P>. The derived theorem is treated as a "theorem schema" -- 
a mold for other theorems. This turns out to be  a perfect valid procedure, in that it always 
leads you to new theorems, but it is not a rule of the Propositional Calculus as we

leads you to new theorems, but it is not a rule of the Propositional Calculus as we 
presented it. It is, rather, a derived rule , It is part of the knowledge which we have about  
the system. That this rule keeps you within the space of theorems needs proof, of course-
but such a proof is not like a derivation inside the system. It is a proof in the ordinary, intuitive sense -- a chain of reasoning carried out in the I-mode. The theory about the Propositional Calculus is a "metatheory", and results in it can be called "metatheorems" - Theorems about theorems. (Incidentally, not e the peculiar capitalization in the phrase

but such a proof is not like a derivation inside the system. It is a proof in the ordinary, intuitive sense -- a chain of reasoning carried out in the I-mode. The theory about the Propositional Calculus is a "metatheory", and results in it can be called "metatheorems" - Theorems about theorems. (Incidentally, not e the peculiar capitalization in the phrase 
"Theorems about theorems". It is a conse quence of our convention: metatheorems are 
Theorems (proven results) concerning theorems (derivable strings).) 
In the Propositional Calculus, one could discover many metatheorems, or derived 
rules of inference. For instance, there is a De Morgan's Rule:

 202<~x∨~y> and ~<x ∧y> are interchangeable. 
 
If this were a rule of the system, it could speed up many derivations considerably. But if 
we prove  that it is correct, isn't that good enough?  Can't we use it just like a rule of 
inference, from then on? 
There is no reason to doubt the correctness of this particular derived rule. But 
once you start admitting derived rules as pa rt of your procedure in the Propositional 
Calculus, you have lost the formality of th e system, since derived rules are derived 
informally-outside the system. Now formal systems were proposed as a way to exhibit 
every step of a proof explicitly, within one single, rigid framework, so that any 
mathematician could check another's work m echanically. But if you are willing to step 
outside of that framework at the drop of a ha t, you might as well never have created it at 
all. Therefore, there is a drawback to using such shortcuts.  
Formalizing Higher Levels

Formalizing Higher Levels 
 
On the other hand, there is an alternative way out. Why not formalize the metatheory, 
too? That way, derived rules (metatheorems)  would be theorems of a larger formal 
system, and it would be legitimate to look for shortcuts and derive them as theorems-that 
is, theorems of the formalized metatheory-w hich could then be used to speed up the 
derivations of theorems of the Propositional Calculus. This is an interesting idea, but as soon as it is suggested, one jumps ahead to th ink of metametatheories, and so on. It is

derivations of theorems of the Propositional Calculus. This is an interesting idea, but as soon as it is suggested, one jumps ahead to th ink of metametatheories, and so on. It is 
clear that no matter how many levels you form alize, someone will eventually want to 
make shortcuts in the top level. 
It might even be suggested that a theory of reasoning could be identical to its own 
metatheory, if it were worked out carefu lly. Then, it might seem, all levels would 
collapse into one, and thinking about  the system would be just one way of working in the 
system! But it is not that easy. Even if a system can "think about itself", it still is not outside  itself. You, outside the system, perceive  it differently from the way it perceives

system! But it is not that easy. Even if a system can "think about itself", it still is not outside  itself. You, outside the system, perceive  it differently from the way it perceives 
itself. So there still is a metatheory-a view  from outside-even for a theory which can 
"think about itself" inside itself. We will fi nd that there are theories which can "think 
about themselves". In fact, we will soon see a system in which this happens completely 
accidentally, without our even intending it! A nd we will see what kinds of effects this 
produces. But for our study of the Propositional Ca lculus, we will stick with the simplest 
ideas-no mixing of levels.

ideas-no mixing of levels. 
Fallacies can result if you fail to disti nguish carefully between working in the 
system (the M-mode) and thinking about the system (the I-mode). For example, it might 
seem perfectly reasonable to assume that, since <P∨~P> (whose semi-interpretation is 
"either P or not P") is a theorem, either P or ~ P must be a theorem. But this is dead 
wrong: neither one of the latter pair is a theore m. In general, it is a dangerous practice to 
assume that symbols can be slipped back a nd forth between different levels-here, the

assume that symbols can be slipped back a nd forth between different levels-here, the 
language of the formal system and its metalanguage (English).

 203Reflections on the Strengths and Weaknesses of the System 
 
You have now seen one example of a system w ith a purpose-to re part of the architecture 
of logical thought. The concepts which this ha ndles are very few in number, and they are 
very simple, precise co But the simplicity and precision of the Propositional Calculus are 
the kinds of features which make it appeali ng to mathematicians. There are two reasons 
for this. (1) It can be studied for its own pr operties, ex geometry studies simple, rigid 
shapes. Variants can be made on it, employi ng different symbols, rules of inference, 
axioms or axiom schemata on. (Incidentall y, the version of the Propositional Calculus

axioms or axiom schemata on. (Incidentall y, the version of the Propositional Calculus 
here pr is related to one invented by G. Gentzen in the early 1930's. The other versions in which only one rule of inference is used-detachment usually-and in which there are several axioms, or axiom schemata study of ways to carry out propositional reasoning in elegant formal systems is an appealing bran ch of pure mathematics. (2) The Propositional 
Calculus can easily be extended to include other fundamental aspects of reasoning. Some 
of this will be shown in the next Chap ter, where the Propositional Calculus is

of this will be shown in the next Chap ter, where the Propositional Calculus is 
incorporated lock, stock and barrel into a much larger and deeper system in which 
sophisticated number-theoretical reasoning can be done.  
Proofs vs. Derivations 
 
The Propositional Calculus is very much like reasoning in some w one should not equate its rules with the rules of human thought. A proof  is something informal, or in other 
words a product of normal thought written in a human language, for human consumption. 
All sorts of complex features  of thought may be used in proofs, and, though they may 
“feel right", one may wonder if they can be defended logically. That is really what formalization is for. A derivation  is an artificial counterpart of and its purpose is to reach 
the same goal but via a logical structure whos e methods are not only all explicit, but also 
very simple.

very simple. 
If -- and this is usually the case  -it happe ns that a formal derivation is extremely 
lengthy compared with the corresponding "natural " proof that is just too bad. It is the 
price one pays for making each step so simple . What often happens is that a derivation 
and a proof are "simple" in complementary se nses of the word. The proof is simple in 
that each step sounds right", even though one may not know just why; the derivation is 
simple in that each of its myriad steps is c onsidered so trivial that it is beyond reproach,

simple in that each of its myriad steps is c onsidered so trivial that it is beyond reproach, 
and since the whole derivation consists just of such trivial steps it is supposedly error-
free. Each type of simplicity, however, brings  along a characteristic type of complexity. 
In the case of proofs, it is the complexity of the underlying system on which they rest -- 
namely, human language -- and in the case of derivations, it is their astronomical size, 
which makes them almost impossible to grasp. 
Thus, the Propositional Calculus should be thought of as  part of a

 204general method for synthesizing artificial proof -like structures. It does not, however, have 
much flexibility or generality. It is intended only for use in connection with mathematical concepts-which are themselves quite rigid. As a rather interesting example of this, let us 
make a derivation in which a very peculiar st ring is taken as a premise in a fantasy: 
<P∧~P>.  At least its semi-interpretation is  peculiar. The Propositional Calculus, 
however, does not think about semi-interpr etations; it just manipulates strings 
typographically-and typographically, there is really nothing peculiar about this string. 
Here is a fantasy with this string as its premise:  
(1) [   push 
(2)    <P∧~P>  premise 
 (3)    P   separation 
(4)    ~P   separation 
(5)    [   push (6)       ~Q   premise (7)      P   carry-over line 3 
(8)      ~~P   double-tilde 
(9)     ]   pop 
(10)     <~Q⊃~~P>   fantasy 
(11)     <~P⊃Q>  contrapositive 
(12)    Q   detachment (Lines 4,11) 
(13) ]   pop

(1) [   push 
(9)     ]   pop 
(13) ]   pop 
(14) <<P∧~P >⊃Q> fantasy 
 
Now this theorem has a very strange semi-interpretation:  
 P and not P together imply Q 
 
Since Q is interpretable by any statement, we can loosely take the theorem to say that 
"From a contradiction, anything follows"! Thus, in systems based on the Propositional 
Calculus, contradictions cannot be containe d; they infect the whole system like an 
instantaneous global cancer.  
The Handling of Contradictions 
 
This does not sound much like human thought. If you found a contradiction in your own

This does not sound much like human thought. If you found a contradiction in your own 
thoughts, it's very unlikely that your whol e mentality would break down. Instead, you 
would probably begin to question the beliefs  or modes of reasoning which you felt had 
led to the contradictory thoughts. In other wo rds, to the extent you could, you would step 
out of the systems inside you which you felt we re responsible for the contradiction, and 
try to repair them. One of the least likely things for you to do would be to throw up your

try to repair them. One of the least likely things for you to do would be to throw up your 
arms and cry, "Well, I guess that shows that  I believe everything now!" As a joke, yes-
but not seriously. 
Indeed, contradiction is a major source of clarification and progress in all domains 
of life-and mathematics is no exception. When in times past, a

 205contradiction in mathematics was found, mathematicians would immediately seek to 
pinpoint the system responsible for it, to jump  out of it, to reason about it, and to amend 
it. Rather than weakening mathematics, the di scovery and repair of a contradiction would 
strengthen it. This might take time and a numbe r of false starts, but in the end it would 
yield fruit. For instance, in the Middle Ages, the value of the infinite series  
1 – 1 + 1 – 1 + 1 -. ..  
was hotly disputed. It was "proven" to equal 0, 1, ½, and perhaps other values. Out of such controversial findings came a fuller, deeper about infinite series.  
A more relevant example is the contradi ction right now confronting us-namely the 
discrepancy between the way we really think, and t the Propositional Calculus imitates us. This has been a source of discomfort for many logicians, and much creative effort has 
gone into trying to patch up the Propositional Calc ulus so that it would not act so stupidly

gone into trying to patch up the Propositional Calc ulus so that it would not act so stupidly 
and inflexibly. One attempt, put forth in the book Entailment  by A. R. Anderson and N. 
Belnap,
3 involves "relevant implication", which tr ies to make the symbol for "if-then" 
reflect genuine causality, or at least connect meanings. Consider the following theorems of the Propositional Calculus 
 
<P⊃<Q⊃P>> 
<P⊃<Q∨~P>> 
<<P∧~P>⊃Q> 
<<P⊃Q>∨<Q⊃P>> 
 
They, and many others like them, all show that there need be no relationship at all between the first and second clauses of an if -then statement for it to be provable within 
the Propositional Calculus. In protest, "relevan t implication" puts certain restrictions on

<P⊃<Q⊃P>> 
 
the Propositional Calculus. In protest, "relevan t implication" puts certain restrictions on 
the contexts in which the rules of inference can be applied. Intuitively, it says that "something can only be derived from something el se if they have to do with each other”. 
For example, line 10 in the derivation given above would not be allowed in such a 
system, and that would block the derivation of the <<P∧~P >⊃Q> 
More radical attempts abandon completely the quest for completeness or 
consistency, and try to mimic human reasoning with all its inconsistencies. Such research no longer has as its goal to provide a solid underpinning for mathematics, but purely to

consistency, and try to mimic human reasoning with all its inconsistencies. Such research no longer has as its goal to provide a solid underpinning for mathematics, but purely to 
study human thought processes. 
Despite its quirks, the Propositional Calcul us has some feat recommend itself. If 
one embeds it into a larger system (as we will do next Chapter), and if one is sure that the 
larger system contains no contradictions (a nd we will be), then the Propositional Calculus 
does all that one could hope: it provides valid propositional inferences -- all that can be 
made. So if ever an incompleteness or an in consistency is uncovered, can be sure that it 
will be the fault of the larger  system, and not of its subsystem which is the Propositional 
Calculus.

Crab Canon 206
 
FIGURE 42 . “Crab Canon”, by M. C. Escher (~1965)

Crab Canon 207Crab Canon 
 
Achilles and the Tortoise happen upon each other 
in the park one day while strolling. 
 
Tortoise: Good day, Mr. A. Achilles: Why, same to you. Tortoise: So nice to run into you. Achilles: That echoes my 
thoughts. 
Tortoise: And it's a perfect day for a walk. I think I'll be walking home soon. Achilles: Oh, really? I guess there's nothing better for you than w Tortoise: Incidentally, 
you're looking in very fine fettle these days, I must say. 
Achilles: Thank you very much. Tortoise: Not at all. Here, care for one of my cigars? Achilles: Oh, you are such a philistine. In this area, the Dutch contributions are of 
markedly inferior taste, don't you think? 
Tortoise: I disagree, in this case. But speaking of taste, I finally saw that Crab Canon by

Tortoise: I disagree, in this case. But speaking of taste, I finally saw that Crab Canon by 
your favorite artist, M. C. Escher, in a gallery the other day, and I fully appreciate the beauty and ingenuity with which he made one single theme mesh with itself going both backwards and forwards. But I am afraid I will always feel Bach is superior to Escher.

Crab Canon 208Achilles: I don't know. But one thing for certain is that I don't worry about arguments of 
taste. De gustibus non est disputandum . 
Tortoise: Tell me, what's it like to be your age? Is it true that one has no worries at all? Achilles: To be precise, one has no frets. Tortoise: Oh, well, it's all the same to me. Achilles: Fiddle. It makes a big difference, you know. Tortoise: Say, don't you play the 
guitar? 
Achilles: That's my good friend. He often plays, the fool. But I myself wouldn't touch a 
guitar with a ten-foot pole! 
 
(Suddenly, the Crab, appearing from out of nowhere, wanders up excitedly, 
pointing to a rather prominent black eye.) 
 
Crab: Hallo! Hulloo! What's up? What's new? You see this bump, this lump? Given to

me by a grump. Ho! And on such a fine day. You see, I was just idly loafing about the park when up lumbers this giant fellow from Warsaw-a colossal bear of a man-playing a lute. He was three meters tall, if I'm a day. I mosey on up to the chap, reach skyward and manage to tap him on the knee, saying, "Pardon me, sir, but you are Pole-luting our park with your mazurkas." But wow! he had no sense of humor-not a bit, not a wit-and POW!-he lets loose and belts me one, smack in the eye! Were it in my nature, I would crab up a storm, but in the time-honored tradition of my species, I backed off. After all, when we walk forwards, we move backwards. It's in our genes, you know, turning round and round. That reminds me-I've always wondered, "Which came first-the Crab, or the Gene?" That is to say, "Which came last the Gene, or the Crab?" I'm always turning things round and round, you know. It's in our genes, after all. When we walk backwards, we move forwards. Ah me, oh my! I must lope along

on my merry way-so off I go on such a fine day. Sing "ho!" for the life of a Crab! 
TATA! iOle! 
 
(And he disappears as suddenly as he arrived.) 
 
Tortoise: That's my good friend. He often plays the fool. But I myself wouldn't touch a 
ten-foot Pole with a guitar! 
Achilles: Say, don't you play the guitar? Tortoise: Fiddle. It makes a big difference, 
Achilles: Oh, well, it's all the same to me. 
Tortoise: To be precise, one has no frets. Achilles: Tell me, what's it like to be your age? Is it true that one has no worries at all? Tortoise: I don't know. But one thing for certain is that I don't worry about arguments of 
taste. Disputandum non est de gustibus.

Crab Canon 209 
  
FIGURE 43. Here is a short section one of the Crab's 
Genes, turning round and round. When the two DNA strands are raveled and laid out side by side, they read this way: 
 
….TTTTTTTTTCGAAAAAAAAA ….AAAAAAAAGCTTTTTTTTTT  
Notice that they are the same, only one forwards while 
the other goes backwards This is the defining property of the form called "crab canon" in music. It is reminiscent of, though a little different from palindrome, which is a sentence that reads the same backwards and forwards ,In  molecular biology, such segments of DNA  are called "palindromes "-a 
slight misnomer, since "crab canon" would be more accurate. Not only is this DNA  segment crab-
canonical-but moreover its base sequence codes for the Dialogue's structure Look carefully! 
 
... 
Achilles: I disagree, in this case. But speaking of taste, I finally heard that Crab Canon

... 
Achilles: I disagree, in this case. But speaking of taste, I finally heard that Crab Canon  
by your favorite composer, J. S. Bach, in a concert other day, and I fully appreciate the beauty and ingenuity with which he made one single theme mesh with itself going both backwards and forwards. But I'm afraid I will always feel Escher is superior to Bach 
Tortoise: Oh, you are such a philistine. In this area, the Dutch contributions are of 
Achilles: Not at all. Here, care for one of my cigars?  Tortoise: Thank you very much. Achilles: Incidentally, you're looking in very fine fettle these days, I must say.

Crab Canon 211Tortoise: Oh, really? I guess there's nothing better for you than walking. 
Achilles: And it's a perfect day for a walk. I think I'll be walking home soon. Tortoise: That echoes my thoughts.  Achilles: So nice to run into you.  Tortoise: Why, same to you.  Achilles: Good day, Mr. T

Typographical Number Theory 212CHAPTER VIII 
 
Typographical Number Theory 
 
The Crab Canon  and Indirect Self-Reference 
 
THREE EXAMPLES OF indirect self-refere nce are found in the Crab Canon. Achilles 
and the Tortoise both describe artistic crea tions they know-and, quite accidentally, those 
creations happen to have the same structure as the Dialogue they're in. (Imagine my surprise, when I, the author, noticed this!) Al so, the Crab describes a biological structure 
and that, too, has the same property. Of course, one could read the Dialogue and 
understand it and somehow fail to notice that it,  too, has the form of a crab canon. This 
would be understanding it on one level, but not on another. To see the self-reference, one has to look at the form, as well as the content, of the Dialogue. 
Gödel’s construction depends on describing the form, as well as the content, of 
strings of the formal system we shall define in this Chapter -- Typographical Number

strings of the formal system we shall define in this Chapter -- Typographical Number  
Theory ( TNT ). The unexpected twist is that, because of the subtle mapping which Gödel 
discovered, the form of strings can be described in the formal system itself. Let us acquaint ourselves with this strange system with the capacity for wrapping around.  
What We Want to Be Able to Express in TNT 
 We'll begin by citing some typical sentences  belonging to number theory; then we will 
try to find a set of basic notions in terms of which all our sentences can be rephrased.

try to find a set of basic notions in terms of which all our sentences can be rephrased. 
Those notions will then be given individual symbols. Incidentally, it should be stated at the outset that the term "number theory" will refer only to properties of positive integers 
and zero (and sets of such integers). These numbers are called the natural numbers . 
Negative numbers play no role in this theory. Thus the word "number", when used, will mean exclusively a natural number. And it is im portant -- vital-for you to keep separate in 
your mind the formal system ( TNT ) and the rather ill-defined but comfortable old branch

your mind the formal system ( TNT ) and the rather ill-defined but comfortable old branch 
of mathematics that is number theory itself; this I shall call " N". 
Some typical sentences of N-number theory-are: 
 (1) 5 is prime. (2) 2 is not a square. (3) 1729 is a sum of two cubes. (4) No sum of two positive cubes is itself a cube. (5) There are infinitely many prime numbers. (6) 6 is even.

Typographical Number Theory 213 
Now it may seem that we will need a symbol for each notion such as "prime” or "cube" or "positive" -- but those notions are really not primitive. Primeness, for instance, has to do with the factors which a number has, wh ich in turn has to do with multiplication. 
Cubeness as well is defined in terms multiplication. Let us rephrase the sentences, then, in terms of what seem to be more elementary notions.  
(1) There do not exist numbers a and b, both greater than 1. such that 5 equals a 
times b. 
 (2) There does not exist a number b, such that b times b equals 2. 
 (3) There exist numbers b and c such that b times b times b, plus c times c times c, 
equals 1729. 
 (4') For all numbers b and c, greater than 0, there is no number a such that a times a 
times a equals b times b times b plus c times c times c. 
 (5) For each number a, there exists a number b, greater than a, with the property 
that there do not exist numbers c and d, both greater than 1, such that b equals c

times b. 
that there do not exist numbers c and d, both greater than 1, such that b equals c 
times d. 
 (6') There exists a number e such that 2 times e equals 6. 
 This analysis has gotten us a long ways to wards the basic elements of language of 
number theory. It is clear that a few phrases reappear over a over: 
 
for all numbers b 
there exists a number b, such that 
greater than equals times  plus 0, 1, 2, . . 
 Most of these will be granted individual sym bols. An exception is "greater than", which 
can be further reduced. In fact, the sentence " a is greater than b" becomes

times d. 
 
can be further reduced. In fact, the sentence " a is greater than b" becomes 
 
there exists a number c, not equal to 0, such that a equals b plus c.  
 
Numerals 
 We will not have a distinct symbol for each  natural number. Instead, we have a very 
simple, uniform way of giving a compound symbol to e natural number -- very much as 
we did in the pq-system. Here is notation for natural numbers:

Typographical Number Theory 214zero:   0  
one:           SO  
two:         SSO  
three:      SSSO  
 
       etc. 
 The symbol S has an interpretation-"the successor of". Hence, the interpretation of SSO  
is literally "the successor of the successor of  zero". Strings of this form are called 
numerals. 
 
Variables and Terms 
 
Clearly, we need a way of referring to unspeci fied, or variable, numbers. For that, we will 
use the letters a, b, c, d, e. But five will not be enough. We need an unlimited supply of 
them, just as we had of atoms in the Propos itional Calculus. We will use a similar method 
for making more variables: tacking on any number of primes. (Note: Of course the 
symbol "'-read "prime"-is not to be confused with prime numbers!) For instance:  
e d'  c" b´´´... a´´´´ 
 are all variables . 
In a way it is a luxury to use the first fi ve letters of the alphabet when we could 
get away with just a and the prime. Later on, I will actually drop b, c, d , and e, which will

get away with just a and the prime. Later on, I will actually drop b, c, d , and e, which will 
result in a sort of "austere" version of TNT -austere in the sense that  it is a little harder to 
decipher complex formulas. But for now we'll be luxurious. 
Now what about addition and multiplica tion? Very simple: we will use the 
ordinary symbols `+' and `•'. However, we will also introduce a parenthesizing requirement (we are now slowly slipping into the rules which define well-formed strings 
of TNT ). To write " b plus c" and " b times c", for instance, we use the strings 
 
(b+c) 
(b • c)

(b+c) 
(b • c) 
 There is no laxness about such parentheses; to violate the convention is to produce a non-
well-formed formula. ("Formula"? I use the term instead of "string" because it is 
conventional to do so. A formula  is no more and no less than a string of TNT .) 
 Incidentally, addition and multiplicati on are always to be thought of as binary  operations-
that is, they unite precisely two numbers, ne ver three or more. Hence, if you wish to

(b+c) 
(b • c) 
that is, they unite precisely two numbers, ne ver three or more. Hence, if you wish to 
translate "1 plus 2 plus 3", you have to d ecide which of the following two expressions 
you want:

Typographical Number Theory 215(SO+(SSO+SSSO)) 
((SO+SSO)+SSSO) 
 
The next notion we'll symbolize is equals . That is very simple: we use ´=´.The advantage 
of taking over the standard symbol used  N -- nonformal number theory -- iis obvious: 
easy legibility. The disadvantage is very much  like the disadvantage of using the words 
"point" a "line" in a formal treatment of geometry: unless one is very conscious a careful, one may blur the distinction between the fa miliar meaning and strictly rule-governed 
behavior of the formal symbol. In disc uss geometry, I distinguished between the 
everyday word and the formal to by capitaliz ing the formal term: thus, in elliptical 
geometry, a POINT was 1 union of two ordi nary points. Here, there is no such 
distinction; hen mental effort is needed not to  confuse a symbol with all of the association 
is laden with. As I said earlier, with refere nce to the pq-system: the string --- is not the

is laden with. As I said earlier, with refere nce to the pq-system: the string --- is not the 
number 3, but it acts isomorphically to 3, at l east in the context of additions. Similar 
remarks go for the string SSSO . 
 
Atoms and Propositional Symbols 
 All the symbols of the Propositional Calculus  except the letters used making atoms ( P, Q , 
and R) will be used in TNT , and they retain their interpretations. The role of atoms  will 
be played by strings which, when interpre ted, are statements of equality, such as 
SO=SSO  or (SO • SO ) Now, we have the equipment to do a fair amount of translation of 
simple sentences into the notation of TNT :

simple sentences into the notation of TNT : 
 
2 plus 3 equals 4:   (SSO+SSSO)=SSSSO  
2 plus 2 is not equal to 3:  ~(SSO+SSO)=SSSO  
If 1 equals 0, then 0 equals 1:  <SO=OJO=SO>  
 The first of these strings is an atom; the rest are compound formulas (Warning: The `and' 
in the phrase "I and 1 make 2" is just anothe r word for `plus', and must be represented by 
`+' (and the requisite parentheses).)  
Free Variables and Quantifiers 
 All the well-formed formulas above have th e property that their interpretations are

 All the well-formed formulas above have th e property that their interpretations are 
sentences which are either true or false. Th ere are, however, well-formed formulas which 
do-not have that property, such as this one  
(b+SO)=SSO 
 
Its interpretation is " b plus 1 equals 2". Since b is unspecified, there is way to assign a 
truth value to the statement. It is like an out-of-context statement with a pronoun, such as 
"she is clumsy". It is neither true nor false; it is waiting for you to put it into a context. Because it is neither true nor false, such a formula is called open , and the variable b is

"she is clumsy". It is neither true nor false; it is waiting for you to put it into a context. Because it is neither true nor false, such a formula is called open , and the variable b is 
called a free variable .

Typographical Number Theory 216One way of changing an open formula into a closed  formula, or sentence , is by 
prefixing it with a quantifier -either the phrase "there exists a number b such that  , or the 
phrase "for all numbers b". In the first instance, you get the sentence 
 
There exists a number b such that b plus 1 equals 2.  
 
Clearly this is true. In the second instance, you get the sentence  
 
For all numbers b, b plus 1 equals 2. 
 
Clearly this is false. We now introduce symbols for both of these quantifiers . These 
sentences are translated into TNT-notation as follows: 
 
ℑb:(b+SO)=SSO      ('ℑ' stands for `exists'.) 
                                  V b:(b+SO)=SSO    (' V' stands for `all'.) 
 
It is very important to note that these statements are no longer about unspecified 
numbers; the first one is an assertion of existence, and the second one is a universal 
assertion . They would mean the same thing, even if written with c instead of b: 
 
  ℑc:(c+SO)=SSO ` 
Vc:(c+SO)=SSO

Vc:(c+SO)=SSO 
 
A variable which is under the domin ion of a quantifier is called a quantified variable . The 
following two formulas illustrate the difference between free variables and quantified variables:  
(b.b)=SSO   (open) 
---ℑb:(b•b)=SSO  (closed; a sentence  of TNT) 
 
The first one expresses a property  which might be possessed by some natural number. Of 
course, no natural number has that property. A nd that is precisely what is expressed by 
the second one. It is very crucial to unders tand this difference between a string with a free

the second one. It is very crucial to unders tand this difference between a string with a free 
variable,  which expresses a property , and a string where the variable is quantified , which 
expresses a truth  or falsity . The English translation of a fo rmula with at least one free 
variable-an open formula-is called a predicate . It is a sentence without a subject (or a 
sentence whose subject is an out-of-context pronoun). For instance, 
 
"is a sentence without a subject" 
 
"would be an anomaly" 
 
"runs backwards and forwards simultaneously" 
 
"improvised a six-part fugue on demand"

"improvised a six-part fugue on demand" 
 are nonarithmetical predicates. They express properties  which specific entities might or 
might not possess. One could as well stick on a "dummy

Typographical Number Theory 217subject", such as "so-and-so". A string with  free variables is like a predicate with "so-
and-so" as its subject. For instance,  
(SO+SO)=b 
 
is like saying "1 plus 1 equals so-and-so".  This is a predicate in the variable b. It 
expresses a property which the number b might have. If one wet substitute various 
numerals for b, one would get a succession of forms most of which would express 
falsehoods. Here is another example of difference between open formulas and sentences : 
 
`V
b:`V c:(b+c)=(c+b) 
 
The above formula is a sentence representi ng, of course, the commutativity of addition. 
On the other hand,  
`V
c:(b+c)=(c+b) 
 
is an open formula, since b is free. It expresses a property which unspecified number b 
might or might not have -- namely of commuting with all numbers c. 
 
Translating Our Sample Sentences 
 This completes the vocabulary with which we will express all num theoretical statements!

`V
 
`V
 
 This completes the vocabulary with which we will express all num theoretical statements! 
It takes considerable practice to get the ha ng of expressing complicated statements of N 
in this notation, and converse] figuring out the meaning of well-formed formulas. For this reason return to the six sample sentences  given at the beginning, and work their 
translations into TNT . By the way, don't think that th e translations given below are 
unique-far from it. There are many -- infinitely many -- ways to express each one. 
Let us begin with the last one: "6 is ev en". This we rephrased in to of more

Let us begin with the last one: "6 is ev en". This we rephrased in to of more 
primitive notions as "There exists a number e such that 2 times e equals 6". This one is 
easy 
: 
ℑe:(SSO. e)=SSSSSSO 
 
Note the necessity of the quantifier; it simply would not do to write 
 
(SSO . e)=SSSSSSO 
 
alone. This string's interpretation is of cour se neither true nor false; it expresses a 
property which the number e might have. 
It is curious that, since we know multip lication is commutative might easily have 
written 
 
ℑe:(e - SSO)=SSSSSSO 
 
instead. Or, knowing that equality is a symmetr ical relation, we might 1 chosen to write

easy 
: 
 
written 
 
instead. Or, knowing that equality is a symmetr ical relation, we might 1 chosen to write 
the sides of the equation in the opposite order:

Typographical Number Theory 218ℑe:SSSSSSO=(SSO • e) 
 
Now these three translations of "6 is even " are quite different strings, and it is by no 
means obvious that theoremhood of any one of them is tied to theoremhood of any of the 
others. (Similarly, the fact that --p-q---  was a theorem had very little to do with the fact 
that its "equivalent" string -p--q---  was a theorem. The equivalence lies in our minds, 
since, as humans, we almost automatically think about interpretations, not structural 
properties of formulas.) 
We can dispense with sentence 2: "2 is not a square", almost immediately: 
 
-ℑb:(b • b)=SSO 
 
However, once again, we find an ambiguity. What if we had chosen to write it this way?  
Vb: -(b • b) =SSO 
 
The first way says, "It is not th e case that there exists a number b with the property that 
b's square is 2", while the second way says, "For all numbers b, it is not the case that b's 
square is 2." Once again, to us , they are conceptually equivalent-but to TNT , they are

square is 2." Once again, to us , they are conceptually equivalent-but to TNT , they are 
distinct strings. 
Let us proceed to sentence 3: "1729 is a sum of two cubes." This one will involve 
two existential quantifiers, one after the other, as follows: 
 
ℑb:ℑc:SSSSSS…………SSSSSO=(((b • b) • b)+((c • c) • c)) 
 
1729 of them 
 
There are alternatives galore. Reverse the orde r of the quantifiers; switch the sides of the 
equation; change the variables to d and e; reverse the addition; write the multiplications

equation; change the variables to d and e; reverse the addition; write the multiplications 
differently; etc., etc. However, I prefer the following two translations of the sentence:  
ℑb:ℑc:(((SSSSSSSSSSO.SSSSSSSSSSO).SSSSSSSSSSO)+ 
 ((SSSSSSSSSO • SSSSSSSSSO) • SSSSSSSSSO))=(((b • b) • b)+((c • c) • c)) 
 
and 
 
ℑb:ℑc:(((SSSSSSSSSSSSO.SSSSSSSSSSSSO). SSSSSSSSSSSSO)+ 
 ((SO •SO) • SO))=(((b •b) •b)+((c • c) •c)) 
 
Do you see why?  
Tricks of the Trade 
 
Now let us tackle the related sentence 4: "No sum of two positive cubes is itself a cube". Suppose that we wished merely to state that  7 is not a sum of two positive cubes. The

and 
 
Now let us tackle the related sentence 4: "No sum of two positive cubes is itself a cube". Suppose that we wished merely to state that  7 is not a sum of two positive cubes. The 
easiest way to do this is by negating  the formula

Typographical Number Theory 219which asserts that 7 is a sum of two positive cubes. This will be just like the preceding 
sentence involving 1729, except that we have to  add in the proviso of the cubes being 
positive. We can do this with a trick: prefix variables with the symbol S, as follows: 
 
ℑb:ℑc:SSSSSSSO=(((Sb • Sb) • Sb)+((Sc • Sc) -Sc)) 
 
You see, we are cubing not b and c, but their successors, which must be positive, since 
the smallest value which either b or c can take on is zero. Hence the right-hand side 
represents a sum of two positive cubes. In( tally, notice that the phrase "there exist 
numbers b and c such that…..”) when translated, does not involve the symbol ` n' which 
stands for ‘and’. That symbol is used for connecting entire well-formed strings, not for 
joining two quantifiers. 
Now that we have translated "7 is a sum of two positive cubes", we wish to negate 
it. That simply involves prefixing the whole thing by a single (Note: you should not

it. That simply involves prefixing the whole thing by a single (Note: you should not 
negate each quantifier, even though the desire d phrase runs "There do not exist numbers 
b and c such that ...".) Thus we get: 
 
-ℑb:ℑc:SSSSSSSO=(((Sb • Sb) • Sb)+((Sc -Sc) -Sc)) 
 
Now our original goal was to assert this property not of the number of all cubes. 
Therefore, let us replace the numeral SSSSSSSO  by the ((a-a)- a), which is the translation 
of "a cubed": 
 
ℑb:ℑc:((a •a) •a)=(((Sb •Sb) • Sb)+((Sc -Sc) -Sc)) 
-  
At this stage, we are in possession of an open  formula, since a is still free. This formula

-  
At this stage, we are in possession of an open  formula, since a is still free. This formula 
expresses a property which a number a might or might not have-and it is our purpose to 
assert that all numbers do have that property. Th at is simple -- just prefix the whole thing 
with a universal quantifier  
V
a:-ℑb:ℑc:((a -a) • a)=(((Sb • Sb) • Sb) +((Sc -Sc) -Sc)) 
 
An equally good translation would be this:  
--ℑa:ℑb:ℑc:((a-a) a)=(((Sb•Sb)•Sb)+((Sc•Sc)•Sc)) 
 In austere TNT , we could use a' instead of b, and a" instead of c, and the formula would 
become:  
--ℑa: ℑa': ℑa":((a • a) • a) =(((Sa' • Sa') • Sa') +((Sa" • Sa") • Sa"))

V
 
become:  
--ℑa: ℑa': ℑa":((a • a) • a) =(((Sa' • Sa') • Sa') +((Sa" • Sa") • Sa")) 
 
What about sentence 1: "5 is prime"? We had reworded it in this way "There do not exist 
numbers a and b, both greater than 1, such equals a times b". We can slightly modify it, 
as follows: "There do not exist numbers a and b such that 5 equals a plus 2, times b plus 
2". This is another trick-since a and b are restricted to natural number values, this is an 
adequate way to say the same thing. Now " b plus 2" could be translated into

Typographical Number Theory 220(b+SSO ), but there is a shorter way to write it -- namely, SSb. Likewise, " c plus 2" can 
be written SSc. Now, our translation is extremely concise: 
 
ℑb: ℑc:SSSSSO=(SSb • SSc) 
 
Without the initial tilde, it would be an assertion that two natural numbers do exist, 
which, when augmented by 2, have a product e qual to 5. With the tilde in front, that 
whole statement is denied, resulting in an assertion that 5 is prime. 
I f  w e  w a n t e d  t o  a s s e r t  t h a t  d plus e plus 1, rather than 5, is prime, the most 
economical way would be to replace the numeral for 5 by the string ( d+Se ): 
 
ℑb: ℑc:(d+Se)=(SSb SSc) 
 
Once again, an open formula, one whose inte rpretation is neither a true nor a false 
sentence, but just an assertion about two unspecified numbers, d and e. Notice that the 
number represented by the string ( d+Se ) is necessarily greater than  d, since one has 
added to d an unspecified but definitely positive amount. Therefore, if we existentially

added to d an unspecified but definitely positive amount. Therefore, if we existentially 
quantify over the variable e, we will have a formula which asserts that: 
 There exists a number which is greater than d and which is prime. 
 
ℑe:- ℑb:3c:(d+Se)=(SSb • SSc) 
 
Well, all we have left to do now is to asse rt that this property actually obtains, no matter 
what d is. The way to do that is to universally quantify over the variable d: 
 
Vd:3e:-3b:3c:(d+Se)=(SSb •SSc) 
 
That's the translation of sentence 5!   
Translation Puzzles for You 
 This completes the exercise of translating all six typical number-theoretical sentences.

There exists a number which is greater than d and which is prime. 
 
 This completes the exercise of translating all six typical number-theoretical sentences. 
However, it does not necessarily make you an expert in the notation of TNT . There are 
still some tricky issues to be mastered. The following six well-formed formulas will test 
your understanding of TNT  notation. What do they mean? Which ones are true (under 
interpretation, of course), and which ones are fa lse? (Hint: the way to tackle this exercise 
is to move leftwards. First, translate th e atom; next, figure out  what adding a single 
quantifier or a tilde does; then move leftward s, adding another quantifier or tilde; then

quantifier or a tilde does; then move leftward s, adding another quantifier or tilde; then 
move leftwards again, and do the same.)  
-V
c: ℑb:(SSO • b)=c 
 
Vc:- ℑb:(SSO • b)=c

Typographical Number Theory 221 
 Vc: ℑb:---(SSO • b)=c 
~ℑb:Vc:(SSO • b)=c 
ℑb:- V c:(SSO • b)=c 
ℑb: Vc:-(SSO • b)=c 
 
(Second hint: Either four of them are true a nd two false, or four false and two true.) 
 
How to Distinguish True from False? 
 At this juncture, it is worthwhile pausing fo r breath and contempt what it would mean to 
have a formal system that could sift out the true from the false ones. This system would 
treat all these strings-which look like statem ents-as designs having form, but no content. 
An( system would be like a sieve through wh ich could pass only designs v special style-
the "style of truth". If you yourself have  gone through ti formulas above, and have 
separated the true from the false by this a bout meaning, you will appreciate the subtlety 
that any system would to have, that coul d do the same thing-but typographically! The 
bout separating the set of true statements from  the set of false statements written in the

bout separating the set of true statements from  the set of false statements written in the 
TNT -notation) is anything but straight; it is a boundary with many treacherous curves 
(recall Fig. 18), a boundary of which mathematicia ns have delineated stretches, here and 
there, working over hundreds years. Just th ink what a coup it would be to have a 
typographical m( which was guaranteed to place any formula on the proper side o border!  
The Rules of Well-Formedness 
 It is useful to have a tabl e of Rules of Formation for well-formed formulas This is 
provided below. There are some  preliminary stages, defining numerals, variables , and

provided below. There are some  preliminary stages, defining numerals, variables , and 
terms . Those three classes of strings are ingred ients of well-formed formulas, but are not 
in themselves well-formed. The smallest well-formed formulas are the atoms ; then there 
are ways of compounding atoms. Many of these rules are recursive lengthening rules, in 
that they take as input an item of a given class and produce a longer item of the class. In 
this table, I use `x' and 'y' to st and for well-formed formulas, and ` s', `t', and ` u' to stand 
for other kinds of TNT -strings. Needless to say, none of these five symbols is itself a

for other kinds of TNT -strings. Needless to say, none of these five symbols is itself a 
symbol of TNT . 
 NUMERALS. 
0 is a numeral. 
A numeral preceded by S is also a numeral.  
Examples : 0 SO S50 SSSO SSSSO SSSSSO

Typographical Number Theory 222VARIABLES. 
a is a variable. If we're not being austere, so are b, c, d  and e. A variable followed 
by a prime is also a variable. Examples : a b' c" d"' a"" 
 
TERMS. 
All numerals and variables are terms. 
A term preceded by S is also a term. 
If s and t are terms, then so are (s+ t) and (s • t).   
Examples : 0 b SSa' (SO • (SSO+c)) S(Sa • (Sb • Sc))  
 
TERMS  may be divided into two categories: 
 
(1) DEFINITE  terms. These contain no variables. 
      Examples : 0 (SO+SO) SS((SSO.SSO)+(SO.SO))   
(2) INDEFINITE  terms. These contain variables. 
     Examples : b Sa (b+SO) (((SO+SO)+SO)+e) 
 
The above rules tell how to make parts  of well-formed formulas; the remaining 
rules tell how to make complete  well-formed formulas. 
 
ATOMS. 
If s and t are terms, then s = t is an atom. 
Examples : SO=0 (SS0+SS0)=5SSS0 5(b+c)=((c•d).e) 
If an atom contains a variable u, then u is free in it. Thus there are 
four free variables in the last example. 
NEGATIONS.

ATOMS. 
NEGATIONS. 
A well-formed formula preceded by a tilde is well-formed.  
Examples :  ~S0=0     ~ ℑb:(b+b)=SO    -<O=0 ⊃S0=O>    ~b=SO  
The quan tification status  of a variable (which says whether the variable is 
free or quantified) does not change under negation. 
COMPOUNDS. 
If x and y are well-formed formulas, and pr ovided that no variable which is free in 
one is quantified in the other, then the following are all well-formed formulas: 
< x∧ y>, < x∨ y>, < x⊃ y>. 
Examples : <O=O∧~-0=0> <b=b ∨~ℑc:c=b>  
<SO=O⊃Vc:~ℑb:(b+b)=c>

<SO=O⊃Vc:~ℑb:(b+b)=c>  
The quantification status of a variable doesn't change here. 
QUANTI FI CATIONS. 
If u is a variable, and x is a well-formed formula in which u is free then the 
following strings are well-formed formulas: 
ℑu: x and V u: x. 
Examples : Vb:<b=b∨~ℑc:c=b>    ∨c:~ℑb:(b+b)=c  ~ ℑc:Sc=d 
OPEN FORMULAS  contain at least one free variable. 
Examples : --c=c b=b <V b:b=bn---c=c>  
CLOSED FORMULAS (SENTENCES)  contain no free variables. 
Examples : 50=0    ~V d:d=0    ℑc:<Vb:b=b ∧~c=c>

Typographical Number Theory 223This completes the table of Rules of Formation for the well-formed formulas of TNT . 
 
A Few More Translation Exercises 
 
And now, a few practice exercises for you, to test your understanding of the notation of 
TNT . Try to translate the first four of the following N-sentences into TNT -sentences, and 
the last one into an open formed formula. 
 
All natural numbers are equal to 4. 
 
There is no natural number which equals its own square. 
 
Different natural numbers have different successors. 
 
If 1 equals 0, then every number is odd. 
 
b is a power of 2. 
 
The last one you may find a little tricky. But it is nothing, compared to this one: 
 
b is a power of 10. 
 
Strangely, this one takes great cleverness to  render in our notation. I would caution you to 
try it only if you are willing to spend hours and hours on it -- and if  you know quite a bit 
of number theory! 
 
A Non typographical System

A Non typographical System 
 
This concludes the exposition of the notation of TNT ; however, we still left with the 
problem of making TNT  into the ambitious system which we have described. Success 
would justify the interpretations which we gi ven to the various symbols. Until we have 
done that, however, particular interpretations are no more justified than the "horse-apple 
happy" interpretations were for the pq-system's symbols. 
Someone might suggest the following way of constructing TNT : (1|) Do not have

Someone might suggest the following way of constructing TNT : (1|) Do not have 
any rules of inference; they are unnecessary, because (2) We take as axioms all true 
statements of number theory (as written in TNT -notation). What a simple prescription! 
Unfortunately it is as empty as instantaneous r eaction says it is. Part (2) is, of course, not 
a typographical description of strings. The whole purpose of TNT  is to figure out if and 
how it is possible to- characterize the true strings typographically. 
 
The Five Axioms and First Rules of TNT

The Five Axioms and First Rules of TNT 
 
Thus we will follow a more difficult route than the suggestion above; we will have axioms and rules of inference. Firstly, as was promised, all of the rules of the 
Propositional Calculus are taken over into  TNT . Therefore one theorem of TNT  will be 
this one:   
<S0=0∨~S0=0>

Typographical Number Theory 224which can be derived in the same way as <P∨-P> was derived. 
Before we give more rules, let us give the five axioms  of TNT : 
 Axiom 1: V
a:~Sa=O  
 Axiom 2: V
a:(a+O)=a  
 Axiom 3: V
a:Vb:(a+Sb)=S(a+b)  
 Axiom 4: V
a:(a-O)=O  
 Axiom 5: V
a:Vb:(a-Sb)=((a-b)+a)  
 (In the austere versions, use a' instead of b.) All of them are very simple to understand. 
Axiom 1 states a special fact about the numbe r 0; Axioms 2 and 3 are concerned with the 
nature of addition; Axioms 4 and 5 are concer ned with the nature of multiplication, and in 
particular with its relation to addition.  
The Five Peano Postulates 
 By the way, the interpretation of Axiom 1- "Zero is not the successor of any natural 
number"-is one of five famous properties of natural numbers first explicitly recognized by the mathematician and logician Giuseppe Peano, in 1889. In setting out his postulates, 
Peano was following the path of Euclid in this  way: he made no attempt to formalize the

Peano was following the path of Euclid in this  way: he made no attempt to formalize the 
principles of reasoning, but tried to give a sm all set of properties of natural numbers from 
which everything else could be derived by reasoning. Peano's attempt might thus be 
considered "semiformal". Peano's work had a significant influence, and thus it would be 
good to show Peano's five postulates. Since the notion of "natural number" is the one 
which Peano was attempting to define, we will not use the familiar term "natural

which Peano was attempting to define, we will not use the familiar term "natural 
number", which is laden with connotation. We will replace it with the undefined term djinn , a word which comes fresh and free of c onnotations to our mind. Then Peano's five 
postulates place five restrictions on djinns. There are two other undefined terms: Genie , 
and meta . I will let you figure out for yourself wh at usual concept each of them is 
supposed to represent. The five Peano postulates:

supposed to represent. The five Peano postulates:  
(1) Genie is a djinn. (2) Every djinn has a mesa (which is also a djinn). (3) Genie is not the mesa of any djinn. (4) Different djinns have different metas. (5) If Genie has X, and each djinn relays X to its mesa, then all djinns get X. 
 In light of the lamps of the Little Harmonic Labyrinth , we should name the set of all 
djinns " GOD ". This harks back to a celebrated statement by the German mathematician 
and logician Leopold Kronecker, archenemy of Georg Cantor: "God made the natural

and logician Leopold Kronecker, archenemy of Georg Cantor: "God made the natural 
numbers; all the rest is the work of man."

Typographical Number Theory 225 
You may recognize Peano's fifth postulate  as the principle of mathematical 
induction-another term for a hereditary argumen t. Peano he that his five restrictions on 
the concepts "Genie", "djinn", and "mesa" so  strong that if two different people formed 
images in their minds o concepts, the two images would have completely isomorphic 
structures. example, everybody's image would include an infinite number of distinct 
djinns. And presumably everybody would agree th at no djinn coins with its own meta, or 
its meta's meta, etc. 
Peano hoped to have pinned down the essence of natural numbers in his five 
postulates. Mathematicians generally grant that he succeeded that does not lessen the 
importance of the question, "How is a true statement about natural numbers to be distinguished from a false one?" At answer th is question, mathematicians turned to totally 
formal systems, as TNT . However, you will see the influence of Peano in TNT , because

formal systems, as TNT . However, you will see the influence of Peano in TNT , because 
all of his postulates are incorporated in TNT  in one way or another. 
 
New Rules of TNT: Specification and Generalization 
 Now we come to the new rules of TNT . Many of these rules will allow reach in and 
change the internal structure of the atoms of TNT . In sense they deal with more 
"microscopic" properties of strings than th e of the Propositional Calculus, which treat 
atoms as indivisible units. example, it woul d be nice if we could extract the string -SO=O 
from the first axiom. To do this we would n eed a rule which permits us to di universal

from the first axiom. To do this we would n eed a rule which permits us to di universal 
quantifier, and at the same time to change the internal strut of the string which remains, if 
we wish. Here is such a rule:  RULE OF SPECIFICATION: Suppose u is a variab le which occurs inside string x. If the 
string V
u:x is a theorem, then so is x, and so  an strings made from x by replacing u, 
wherever it occurs, by one the same term.  
(Restriction : The term which replaces u must not contain any vat that is quantified 
in x.) 
 The rule of specification allows the desired string to be extracted Axiom 1. It is a one-

string V
in x.) 
step derivation:  
V
a -Sa=0               axiom 1 
       ~S0=0           specification 
 
Notice that the rule of specification will allow some formulas which co: free variables (i.e., open formulas) to become theorems. Fo r example following strings could also be 
derived from Axiom 1, by specification:  
Sa=0 
~S(c+SSO)=0 
 
There is another rule, the rule of generalization , which allows us to put

Typographical Number Theory 226back the universal quantifier on theorems whic h contain variables that became free as a 
result of usage of specification. Acting on th e lower string, for example, generalization 
would give: 
 
Vc:~S(c+SSO)=O 
 
Generalization undoes the action of specificati on, and vice versa. Usually, generalization 
is applied after several intermediate steps have transformed the open formula in various 
ways. Here is the exact statement of the rule: 
 
RULE OF GENERALIZATION: Suppose x is a th eorem in which u, a variable, occurs 
free. Then Vu:x is a theorem. 
 
( Restriction : No generalization is allowed in  a fantasy on any variable which 
appeared free in the fantasy's premise.) 
 
The need for restrictions on these two rules will shortly be demonstrated explicitly. Incidentally, this generalization is the same  generalization as was mentioned in Chapter 
II, in Euclid's proof about the infinitude of  primes. Already we can see how the symbol-

manipulating rules are starting to appr oximate the kind of reasoning which a 
mathematician uses. 
 
The Existential Quantifier 
 
These past two rules told how to take off uni versal quantifiers and put them back on; the 
next two rules tell how to handle existential quantifiers. 
 
RULE OF INTERCHANGE: Suppose u is a variable. Then the strings Vu:- and -3u: are 
interchangeable anywhere inside any theorem. 
 
For example, let us apply this rule to Axiom 1: 
 
Va:-Sa=O  axiom 1 
~ℑa:Sa=O  interchange

~ℑa:Sa=O  interchange 
 
By the way, you might notice that both these st rings are perfectly natural renditions, in 
TNT , of the sentence "Zero is not the successo r of any natural number". Therefore it is 
good that they can be turned into each other with ease. 
The next rule is, if anything, even more  intuitive. It corresponds to the very 
simple kind of inference we make when we go from "2 is prime" to "There exists a 
prime". The name of this rule is self-explanatory:

prime". The name of this rule is self-explanatory: 
 
RULE OF EXISTENCE: Suppose a te rm (which may contain variables as long as they 
are free) appears once, or multiply, in a theore m. Then any (or several, or all) of the 
appearances of the term may be replaced by a variable which otherwise does not occur in 
the theorem, and the corresponding existential quantifier must be placed in front.  Let us apply the rule to --as usual--Axiom 1:

Typographical Number Theory 227Va:-Sa=O axiom 1 
ℑb:Va:-Sa=b  existence 
 
You might now try to shunt symbols, according to rules so far giver produce the theorem 
~Vb: ℑa:Sa=b. 
 
Rules of Equality and Successorship 
 
We have given rules for manipulating quantifie rs, but so far none for symbols `=' and 'S'. 
We rectify that situation now. In what follows, r, s, t all stand for arbitrary terms. 
 
RULES OF EQUALITY: 
 
SYMMETRY: If r = s is a theorem, then so is s = r. 
TRANSITIVITY: If r = s and s = t are theorems, then so is r = t. 
 
RULFS OF SUCCESSORSHIP: 
 
ADD S: If r = t is a theorem, then Sr = St is a theorem. 
 
DROP S: If Sr = St is a theorem, then r = t is a theorem. 
 
Now we are equipped with rules that can gi ve us a fantastic variet theorems. For 
example, the following derivations yield theorems which pretty fundamental: 
 
(1) Va: Vb:(a+Sb)=S(a+b)   a x i o m  3  
(2) Vb:(SO+Sb)=S(SO+b)     specification (SO for a)  
(3) (SO+SO)=S(SO+O)    specification (0 for b)

(3) (SO+SO)=S(SO+O)    specification (0 for b)  
(4) Va:(a+O)=a     axiom 2 
(5) (SO+O)=SO     specification (SO for a) 
(6) S(SO+O)=SSO    add S 
(7) (SO+SO)=SSO    transitivity (lines 3,6) 
 
                                                  *       *        *       *         * 
 
(1) Va: Vb:(a-Sb)=((a-b)+a)   a x i o m  5  
(2) Vb:(SO•Sb)=((SO•b)+SO)   specification (SO for a)  
(3) (SO.SO)=((SO.O)+SO)   specification (0 for b) 
(4) Va: Vb:(a+Sb)=S(a+b)    a x i o m  3  
(5) Vb:((SO.O)+Sb)=S((5O O)+b)   specification ((SO-0) for a)  
(6) ((SO .0)+SO)=S((SO.0)+0)   specification (0 for b)

                                                  *       *        *       *         * 
 
(6) ((SO .0)+SO)=S((SO.0)+0)   specification (0 for b) 
(7) Va:(a+O)=a     a x i o m  2  
(8) ((SO.O)+0)=(SO.O)    specification ((S0.0) for a) 
(9) Va:(a.0)=0     a x i o m  4    
(10) (S0-0)=0     specification (SO for a) 
(11) ((SO.O)+O)=O    transitivity (lines 8,10)  
(12) S((SO.0)+0)=SO    a d d  S  
(13) ((SO -0)+SO)=SO    transitivity (lines 6,12) 
(14) (SO.SO)=SO     transitivity (lines 3,13)

Typographical Number Theory 228Illegal Shortcuts 
 
Now here is an interesting question: "How can we make a derivation for the string 0=0?" 
It seems that the obvious route to go would be first to derive the string Va:a=a , and then 
to use specification. So, what about the following "derivation" of Va:a=a  ... What is 
wrong with it? Can you fix it up?  
(1) V
a:(a+0)=a   axiom 2 
(2) Va:a=(a+0)   symmetry 
(3) Va:a=a  transitivity (lines 2,1) 
 I gave this mini-exercise to point out one simp le fact: that one should not jump too fast in 
manipulating symbols (such as `=') which are familiar. One must follow the rules, and not 
one's knowledge of the passive meanings of th e symbols. Of course, this latter type of 
knowledge is invaluable in guiding the route of a derivation.  
Why Specification and Generalization Are Restricted 
 Now let us see why there are restricti ons necessary on both specification and 
generalization. Here are two derivations. In each of them, one of the restrictions is

generalization. Here are two derivations. In each of them, one of the restrictions is 
violated. Look at the disastrous results they produce:  (1) [    push 
(2)     a=0    premise 
(3)     V
a:a=0   generalization ( Wrong !) 
(4)     Sa=O   specification 
(5) ]    pop 
(6) <a=O⊃Sa=O>   fantasy rule 
(7) Va:<a=O⊃Sa=O>   generalization 
(8) <O=O⊃SO=0>  specification 
(9) 0=0    previous theorem 
(10) S0=0    detachment (lines 9,8) 
 This is the first disaster.  The other one is via faulty specification.  (1) V
a:a=a   previous theorem 
(2) Sa=Sa    specification 
(3) ℑb:b=Sa   existence 
(4) Va: ℑb:b=Sa    generalization

(3)     V
(5) ]    pop 
(4) Va: ℑb:b=Sa    generalization 
(5) ℑb:b=Sb   specification ( Wrong !)  
 
So now you can see why those restrictions are needed. 
Here is a simple puzzle: translate (if you have not already done so) Peano's fourth 
postulate into TNT -notation, and then derive that string as a theorem.

Typographical Number Theory 229Something Is Missing 
 
Now if you experiment around for a while  with the rules and axioms of TNT  so far 
presented, you will find that you can produce the following pyramidal family  of theorems 
(a set of strings all cast from an identical mo ld, differing from one a nother only in that the 
numerals 0, SO, SSO , and s have been stuffed in): 
 
(0+0)=0 
(O+SO)=S0 
(O+SSO)=SSO 
(O+SSSO)=SSSO 
(O+SSSSO)=SSSSO 
 
etc. 
 As a matter of fact, each of the theorems in this family can be derived the one directly 
above it, in only a couple of lines. Thus it is a so "cascade" of theorems, each one 
triggering the next. (These theorem very reminiscent of the pq-theorems, where the 
middle and right-] groups of hyphens grew simultaneously.) 
Now there is one string which we can easily write down, and v summarizes the 
passive meaning of them all, taken together. That un sally quantified summarizing string  
is this: 
 
V
a:(O+a)=a

is this: 
 
V
a:(O+a)=a 
 
Yet with the rules so far given, this stri ng eludes production. Ti produce it yourself if you 
don't believe me. 
You may think that we should immediately remedy the situation the following 
 (PROPOSED) RULE OF ALL: If all the strings in a pyramidal family are theorems, then 
so is the universally quantified string which summarizes them.  The problem with this rule is that it cannot be used in the M -mode. people who are 
thinking about  the system can ever know that an infi nite set of strings are all theorems. 
Thus this is not a rule that can be stuck i any formal system.

Thus this is not a rule that can be stuck i any formal system.  
ω-Incomplete Systems and Undecidable Strings 
 
So we find ourselves in a strange situation, in which we can typographically produce theorems about the addition of any specific  numbers, but even a simple string as the one 
above, which expresses a property of addition in general , is not a theorem. You might 
think that is not all that strange, we were in  precisely that situation with the pq-system. 
However, the pq-system had no pretensions a bout what it ought to be able to do; and ii 
fact

Typographical Number Theory 230there was no way to express  general statements about addition in its symbolism, let alone 
prove them. The equipment simply was not there,  and it did not even occur to us to think 
that the system was defective. Here, however , the expressive capability is far stronger, 
and we have correspondingly higher expectations  of TNT than of the pq-system. If the 
string above is not a theorem, then we w ill have good reason to consider TNT to be 
defective. As a matter of fact, there is a na me for systems with this kind of defect-they 
are called ω-incomplete . (The prefix ' ω'-'omega'- comes from the fact that the totality of 
natural numbers is sometimes denoted by ` ω'.) Here is the exact definition: 
 
A system is ω-incomplete if all the strings in a pyramidal family are theorems, but 
the universally quantified summarizing string is not a theorem. 
 Incidentally, the negation of the above summarizing string 
 
~V
a:(O+a)=a

~V
a:(O+a)=a 
 
-is also a nontheorem of TNT. This means that the original string is undecidable within 
the system . If one or the other were a theorem, then we would say that it was decidable. 
Although it may sound like a mystical term , there is nothing mystical about 
undecidability within a given system. It is onl y a sign that the system could be extended. 
For example, within absolute geometry, Euclid 's fifth postulate is undecidable. It has to

~V
a:(O+a)=a 
 
For example, within absolute geometry, Euclid 's fifth postulate is undecidable. It has to 
be added as an extra postulate of geometr y, to yield Euclidean geometry; or conversely, 
its negation can be added, to yield non-Euclidean geometry. If you think back to geometry, you will remember why this curious  thing happens. It is because the four 
postulates of absolute geometry simply do not pin down the meanings of the terms 
"point" and "line", and there is room for different extensions  of the notions. The points 
and lines of Euclidean geometry provide one kind of extension of the notions of "point"

and lines of Euclidean geometry provide one kind of extension of the notions of "point" 
and "line"; the POINTS and LINES of non-Euc lidean geometry, another. However, using 
the pre-flavored words "point" and "line"  tended, for two millennia, to make people 
believe that those words were necessarily univalent, capable of only one meaning.  
Non-Euclidean TNT 
 We are now faced with a similar situation, involving TNT . We have adopted a notation 
which prejudices us in certain ways. For instan ce, usage of the symbol `+'tends to make 
us think that every theorem with a plus  sign in it ought to say something known and

us think that every theorem with a plus  sign in it ought to say something known and 
familiar and "sensible" about the known and familiar operation we call "addition". 
Therefore it would run against the grain to propose adding the following "sixth axiom":  
~
Va:(0+a)=a

Typographical Number Theory 231It doesn't jibe with what we believe about addition. But it is one possible extension of 
TNT , as we have so far formulated TNT. The sy stem which uses this as its sixth axiom is 
a consistent  system, in the sense of not has, two th eorems of the form x and - x. However, 
when you juxtapose this "sixth axiom" with  the pyramidal family of theorems shown 
above, you will probably be bothered by a seeming inconsistency between the family and the new axiom. But this kind of inconsiste ncy is riot so damaging as the other kind 
(where x and x are both theorems ). In fact, it is not a true inconsistency, because there is 
a way of interpreting the symbols so that everything comes out all right.  
ω-Inconsistency Is Not the Same as Inconsistency 
 
This kind of inconsistency, created by th e opposition of (1) a pyramidal family of 
theorems which collectively assert that all natural numbers have some property, and (2) a

theorems which collectively assert that all natural numbers have some property, and (2) a 
single theorem which seems to assert that not all  numbers have it, is given the name of w- 
inconsistency . An w-inconsistent system is more like the at-the-outset-distasteful-but-in-
the-end-accept non-Euclidean geometry. In order to form a mental model of what is going on, you have to imagine that there are so me "extra", unsuspected numbers--let us 
not call them "natural", but supernatural  numbers-which have no numerals. Therefore,

not call them "natural", but supernatural  numbers-which have no numerals. Therefore, 
facts about them cannot be represented in the pyramidal family. (This is a little bit like Achilles' conception GOD -as a sort of "superdjinn", a bei ng greater than any of the djinn 
This was scoffed at by the Genie, but it is a reasonable image, and may I you to imagine supernatural numbers.) 
What this tells us is that the axioms and rules of TNT, as so presented, do not 
fully pin down the interpretations for the symbol  TNT. There is still room for variation in

fully pin down the interpretations for the symbol  TNT. There is still room for variation in 
one's mental model of the notions they stand for. Each of the various possible extensions would pin d, some of the notions further; but  in different ways. Which symbols we begin 
to take on "distasteful" passive meanings, if we added the "s axiom" given above? Would all of the symbols become tainted, or we some of them still mean what we want them to 
mean? I will let you tt about that. We will encounter a similar question in Chapter XIV, 
discuss the matter then. In any case, we will not follow this extension r but instead go on

discuss the matter then. In any case, we will not follow this extension r but instead go on 
to try to repair the w-incompleteness of TNT . 
 
The Last Rule 
 
The problem with the "Rule of  All" was that it required knowing that all lines of an 
infinite pyramidal family are theorems -- t oo much for a finite being. But suppose that 
each line of the pyramid can be de rived from its predecessor in a patterned  way. Then 
there would be a finite  reason  accounting for the fact that all the strings in the pyramid 
are theorems. The trick then, is to find the pattern  that causes the cascade, and show that

Typographical Number Theory 232pattern is a theorem in itself. That is like proving that each djinn passes a message to its 
meta, as in the children's game of "Telephone". The other thing left to show is that Genie 
starts the cascading message-that is, to establish that the first line of the pyramid is a theorem. Then you know that GOD  will get the message! 
In the particular pyramid we were looking at, there is a pattern, captured by lines 
4-9 of the derivation below. 
 
(1) V
a:Vb:(a+Sb)=S(a+b)  axiom 3 
(2) Vb:(O+Sb)=S(O+b)   specification 
(3) (O+Sb)=S(O+b)   specification 
(4) [     push 
(5)    (O+b)=b    premise 
(6)    S(O+b)=Sb    add S 
(7)    (O+Sb)=S(O+b)   carry over line 3 
(8)    (O+Sb)=Sb    transitivity 
(9) ]     pop 
 The premise is (O+b)=b ; the outcome is (O+Sb)=Sb.  
The first line of the pyramid is also a theorem; it follows directly from Axiom 2. 
All we need now is a rule which lets us  deduce that the string which summarizes the

(1) V
(4) [     push 
(9) ]     pop 
All we need now is a rule which lets us  deduce that the string which summarizes the 
entire pyramid is itself a theorem. Such a rule  will he a formalized statement of the fifth 
Peano postulate. 
To express that rule, we need a little not ation. Let us abbreviate a well-formed 
formula in which the variable a is free by the following notation: 
 
X{a} 
 
(There may be other free variables, too, but  that is irrelevant.) Then the notation X{Sa/a}  
will stand for that string but with  every occurrence of a replaced by Sa. Likewise, X{0/a}

X{a} 
 
will stand for that string but with  every occurrence of a replaced by Sa. Likewise, X{0/a}  
would stand for the same string, with each appearance of a replaced by 0. 
A specific example would be to let X{a}  stand for the string in question: (O+a)=a. 
Then X{Sa/a}  would represent the string (O+Sa)=Sa , and X{0/a}  would represent 
(0+0)=0.  (Warning: This notation is not part of TNT ; it is for our convenience in talking 
about TNT .) 
With this new notation, we can state the last rule of TNT  quite precisely: 
 RULE OF INDUCTION: Suppose u is a variable, and X{u} is a well-formed formula in 
which u occurs free. If both V

u:< X{u} ⊃ X{Su/u}>  and X{0/u}  are theorems, 
then Vu: X{u}  is also a theorem. 
 This is about as close as we can come to putting Peano's fifth postulate into TNT . Now 
let us use it to show that Va:( O+a)=a is indeed a theorem in TNT . Emerging from the 
fantasy in our derivation above, we can apply the fantasy rule, to give us  
(10) <(O+b)=b ⊃(O+Sb)=Sb>  fantasy rule 
(11) V
b:<(O+b)=b ⊃(O+Sb)=Sb>  generalization

Typographical Number Theory 233This is the first of the two input theorems required by the induction The other 
requirement is the first line of the pyramid, which we have. Therefore, we can apply the 
rule of induction, to deduce what we wanted.  
`V
b:(O+b)=b 
 
Specification and generalization will allow us to  change the variable from b to a; thus 
Va:(O+a)= a is no longer an undecidable string of TNT .. 
 
A Long Derivation 
 
Now I wish to present one longer derivation in  TNT, so that you ca what one is like, and 
also because it proves a significant, if simple, fact of number theory.  (1)  V
a: Vb:(a+Sb)=S(a+b)      axiom 3 
(2)  Vb:(d+Sb)=S(d+b)      specification 
(3)  (d+SSc)=S(d+Sc)      specificatic 
(4)  b:(Sd+Sb)=S(Sd+b)     specification (line 1) 
(5)  (Sd+Sc)-S(Sd+c)      specification 
 6)  S(Sd+c)=(Sd+Sc)      symmetry 
(7) [         push  
(8) V d:(d+Sc)=(Sd+c)     premise 
(9) (d+Sc)=(Sd+c)     specification   
(10) S(d+Sc)=S(Sd+c)     add S  
(11) (d+SSc)=S(d+Sc)     carry over 3

(7) [         push  
(11) (d+SSc)=S(d+Sc)     carry over 3 
(12) (d+SSc)=S(Sd+c)     transitivity    
(13) S(Sd+c)=(Sd+Sc)     carry over 6 
(14) (d+SSc)=(Sd+Sc)     transitivity    
(15) V d:(d+SSc)=(Sd+Sc)     generalization   
(16)  ]          pop 
(17)  <V d:(d+5c)=(Sd+c) ⊃Vd:(d+SSc)=(Sd+Sc)>    fantasy rule   
(18)  Vc:< V d:(d+Sc)=(Sd+c) ⊃Vd:(d+SSc)=(Sd+Sc)>  generalization  
 
*    *    *    *    * 
(19) (d+S0)=5(d+0)      specification (line 2) 
(20) Va:(a+0)=a       a x i o m  1  
(21) (d+0)=d        specification 
(22) S(d+0)=Sd       a d d  S

(16)  ]          pop 
 
*    *    *    *    * 
(22) S(d+0)=Sd       a d d  S  
(23) (d+SO)=Sd      transitivity (lines 19,2) 
(24) (Sd+O)=Sd       specification (line 20) 
(25) Sd=(Sd+O)      s y m m e t r y

Typographical Number Theory 234(26) ( d+SO)=(Sd+o)      transitivity (lines 23,25) 
(27) Vd:(d+5O)=(Sd+O)     generalization   
 
*    *    *    *    * 
 (28) V
c: Vd:(d+Sc)=(Sd+c)       induction (lines 18,27) 
 
[S can be slipped back and forth in an addition] 
*    *    *    *    * 
 (29)  V
b:(c+Sb)=S(c+b)     specification (line 1) 
(30) (c+Sd)=S(c+d)      specification 
(31)  Vb:(d+Sb)=S(d+b)     specification (line 1) 
(32)  (d+Sc)=S(d+c)       specification 
(33)  S(d+c)=(d+Sc)      symmetry 
(34)  bed:(d+Sc)=(Sd+c)     specification (line 28) 
(35)  (d+Sc)=(Sd+c)       specification 
(36) [         push 
(37) V c:(c+d)=(d+c)     premise 
(38) (c+d)=(d+c)      specification  
(39) S(c+d)=S(d+c)     add S 
(40) (c+Sd)=S(c+d)     carry over 30 
(41) (c+Sd)=S(d+c)     transitivity 
(42) S(d+c)=(d+Sc)     carry over 33 
(43) (c+Sd)=(d+Sc)     transitivity 
(44) (d+Sc)=(Sd+c)     carry over 35 
(45) (c+Sd)=(Sd+c)     transitivity 
(46) V c:(c+Sd)=(Sd+c)      generalization 
(47)  ]          pop

(36) [         push 
(47)  ]          pop 
(48)  <Ve:(c+d)=(d+c) ⊃Vc:(c+Sd)=(Sd+c)>    fantasy rule 
(49)  Vd:< Vc:(c+d)=(d+c) ⊃Vc:(c+Sd)=(Sd+c)>  generalization 
 
[If d commutes with every c, then Sd does too. 
*    *    *    *    * 
(50) (c+O)=c        specification (line 20) 
(51) Va:(O+a)=a      previous theorem 
(52) (O+c)=c        s p e c i f i c a t i o n  
(53) c=(O+c)       s y m m e t r y

Typographical Number Theory 235(54)  (c+0)=(0+c)      transitivity (lines 50,53) 
(55)  Vc:(c+0)=(O+c)      generalization 
 
[0 commutes with every c.] 
 (56) V
d: Vc:(c+d)=(d+c)      induction (lines 49,55) 
 
[Therefore, every d commutes with every  c.] 
 
Tension and Resolution in TNT 
 TNT  has proven the commutativity of addition. Even if you do not follow this derivation 
in detail, it is important to realize that, like a piece of music, it has its own natural 
"rhythm". It is not just a random walk that  happens to have landed on the desired last 
line. I have inserted "breathing marks” to show some of the "phrasing" of this derivation. Line 28 in particular turning point in the derivation, something like the halfway point it 
AABB  type of piece, where you resolve moment arily, even if not in the t key. Such 
important intermediate stages are often called "lemmas". 
It is easy to imagine a reader starting at line 1 of this derivation ignorant of where

It is easy to imagine a reader starting at line 1 of this derivation ignorant of where 
it is to end up, and getting a sense of where it is going as he sees each new line. This 
would set up an inner tension, very much the tension in a piece of music caused by chord 
progressions that let know what the tonality is, without resolving. Arrival at line 28 w, 
confirm the reader's intuition and give him a momentary feeling of satisfaction while at 
the same time strengthening his drive to progress tow what he presumes is the true goal.

the same time strengthening his drive to progress tow what he presumes is the true goal. 
Now line 49 is a critically important tension-increaser, because of "almost-there" 
feeling which it induces. It would be extremel y unsatisfactory to leave off there! From 
there on, it is almost predictable how things  must go. But you wouldn't want a piece of 
music to quit on you just when had made th e mode of resolution apparent. You don't 
want to imagine  ending-you want to hear the ending. Likewise here, we have to c things 
through. Line 55 is inevitable, and sets up all the final tension which are resolved by Line 
56.

56. 
This is typical of the structure not only of formal derivations, but of informal 
proofs. The mathematician's sense of tension is intimately related to his sense of beauty, and is what makes mathematics worthy doing. Notice, however, that in TNT  itself, there 
seems to be no reflection of these tensions. In other words, TNT  doesn't formalize the 
notions of tension and resolution, goal and subgoal, "naturalness" and "inevitable any

56. 
notions of tension and resolution, goal and subgoal, "naturalness" and "inevitable any 
more than a piece of music is a book about harmony and rhythm. Could one devise a much fancier typographical system which is aware  of the tensions and goals inside 
derivations?

Typographical Number Theory 236Formal Reasoning vs. Informal Reasoning 
 
I would have preferred to show how to deri ve Euclid's Theorem (the infinitude of 
primes) in TNT , but it would probably have doubled the length of the book. Now after 
this theorem, the natural direction to go woul d be to prove the associativity of addition, 
the commutativity and associativity of multiplication and the distributivity of 
multiplication over addition. These would give a powerful base to work from. 
As it is now formulated, TNT  has reached "critical mass" (perhaps a strange 
metaphor to apply to something called " TNT "). It is of the same st rength as the system of 
Principia Mathematica ; in TNT one can now prove ev ery theorem which you would find 
in a standard treatise on number theory. Of  course, no one would claim that deriving 
theorems in TNT  is the best way to do number theory. Anybody who felt that way would 
fall in the same class of people as those w ho think that the best way to know what 1000 x

fall in the same class of people as those w ho think that the best way to know what 1000 x 
1000 is, is to draw a 1000 by 1000 grid, and count all the squares in it ... No; after total formalization, the only way to go is towards re laxation of the formal system. Otherwise, 
it is so enormously unwieldy as to be, for all practical purposes, useless. Thus, it is 
important to embed TNT  within a wider context, a context which enables new rules of 
inference to be derived, so that derivati ons can be speeded up. This would require 
formalization of the language in which rule s of inference are expressed-that is, the

formalization of the language in which rule s of inference are expressed-that is, the 
metalanguage. And one could go considerably further. However, none of these speeding-
up tricks would make TNT  any more powerful ; they would simply make it more usable . 
The simple fact is that we have put into TNT  every mode of thought that number 
theorists rely on. Embedding it in ever larger contexts will not enlarge the space of theorems; it will just make working in TNT -or in each "new, improved version"-look 
more like doing conventional number theory. 
 
Number Theorists Go out of Business 
 Suppose that you didn't have advance knowledge that TNT  will turn out to be 
incomplete, but rather, expected that it is complete-that is, that every true statement

incomplete, but rather, expected that it is complete-that is, that every true statement 
expressible in the TNT -notation is a theorem. In that case, you could make a decision 
procedure for all of number theory. The met hod would be easy: if you want to know if N-
statement X is true or false, code it into TNT -sentence x. Now if X is true, completeness 
says that x is a theorem; and conversely, if not-X is true, then completeness says that ~x 
is a theorem. So either x or ~x must be a theorem, since e ither X or not-X is true. Now

is a theorem. So either x or ~x must be a theorem, since e ither X or not-X is true. Now 
begin systematically enumerating all the theorems of TNT , in the way we did for the 
MIU -system and pq-system. You must come to  x or ~x after a while; and whichever one 
you hit tells you which of X and not-X is true. (Did you follow this argument? It crucially 
depends on your being able to hold sepa rate in your mind the formal system TNT  and its 
informal counterpart N. Make sure you understand it.) Thus, in prince-

Typographical Number Theory 237ple, if TNT  were complete, number theorists would be put out of business any question 
in their field could be resolved, with suffici ent time, in a  purely mechanical way. As it 
turns out, this is impossible, which, depending on your point of view, is a cause either for 
rejoicing, or for mourning. 
  
Hilbert's Program 
 The final question which we will take up in  this Chapter is whether should have 
as much faith in the consistency of TNT  as we did consistency of the Propositional 
Calculus; and, if we don't, whether possible to increase our faith in TNT , by proving  it to 
be consistent could make the same opening statement on the "obviousness" of TNT ’s 
consistency as Imprudence did in regard to  the Propositional Calculus namely, that each 
rule embodies a reasoning principle which we believe in, and therefore to question the 
consistency of TNT  is to question our own sanity. To some extent, this argument still

consistency of TNT  is to question our own sanity. To some extent, this argument still 
carries weight-but not quite so much weight as before. There are just too many rules of 
inference and some of them just might be slightly "off ". Furthermore, how do we know that this mental model we have of some abstract entities called "natural numbers" is actually a coherent construct? Perhaps our  own thought processes, those informal 
processes which we have tried to capture in th e formal rules of the system, are themselves 
inconsistent! It is of course not the kind of  thing we expect, but it gets more and more

inconsistent! It is of course not the kind of  thing we expect, but it gets more and more 
conceivable that our thoughts might lead us astray, the more complex the subject matter 
gets-and natural numbers are by no means a tr ivial subject matter. Prudence's cry for a 
proof  of consistency has to be taken more seriously in this case. It's not that we seriously 
doubt that TNT  could be inconsistent but there is a little doubt, a flicker, a glimmer of a 
doubt in our minds, and a proof would help to dispel that doubt. 
But what means of proof would we like to see used? Once again, faced with the

But what means of proof would we like to see used? Once again, faced with the 
recurrent question of circularity. If we use all the equipment in a proof about  our system 
as we have inserted into it, what will we  have accomplished? If we could manage to 
convince ourselves consistency of TNT , but by using a weaker system of reasoning than 
we will have beaten the circularity objection!  Think of the way a heavy rope is passed 
between ships (or so I read when I was a kid) : first a light arrow is fired across the gap, 
pulling behind it a thin rope. Once a connecti on has been established between the two

pulling behind it a thin rope. Once a connecti on has been established between the two 
ships this way, then the heavy rope pulled acr oss the gap. If we can use a "light" system 
to show that a system is consistent, then we shall have really accomplished something. 
Now on first sight one might think there is a thin rope. Our goal is to prove that 
TNT  has a certain typographical property (consis tency): that no theorems of the form x 
and . ~x ever occur. This is similar to trying to show that MU is not a theorem of the 
MIU -system. Both are statements about typographical  properties of symbol-

manipulation systems. The visions of a thin  rope are based on the presumption that facts 
about number theory won’t be

Typographical Number Theory 238needed  in proving that such a typographical prope rty holds. In other words, if properties 
of integers are not used-or if only a few ex tremely simple ones are used-then we could 
achieve the goal of proving TNT  consistent, by using means which are weaker than its 
own internal modes of reasoning. 
This is the hope which was held by an important school of mathematicians and 
logicians in the early part of this century, led by David Hilbert. The goal was to prove the 
consistency of formalizations of number theory similar to TNT  by employing a very 
restricted set of principles of reasoning called "finitistic" methods of reasoning. These 
would be the thin rope. Included among fi nitistic methods are all of propositional 
reasoning, as embodied in the Propositional Ca lculus, and additionally some kinds of 
numerical reasoning. But Gödel’s work showed th at any effort to pull the heavy rope of

numerical reasoning. But Gödel’s work showed th at any effort to pull the heavy rope of 
TNT's  consistency across the gap by using the th in rope of finitistic methods is doomed 
to failure. Gödel showed that in order to pull the heavy rope across the gap, you can't use 
a lighter rope; there just isn't a strong enough one. Less metaphorically, we can say: Any 
system that is strong enough to prove TNT 's consistency is at least as strong as TNT  
itself.  And so circularity is inevitable.

A Mu Offering 
 239A Mu Offering 
 
The Tortoise and Achilles have just been to hear a lecture on the origins of the 
Genetic Code, and are now drinking some tea at Achilles' home. 
 
Achilles: I have something terrible to confess, Mr. T. Tortoise: What is it, Achilles? Achilles: Despite the fascinating subject matter of  that lecture, I drifter to sleep a time or 
two. But in my drowsy state, I still was se mi-awake aware of the words coming into 
my ears. One strange image that floated up from my lower levels was that `A' and `T', instead of standing "adenine" and "thym ine", stood for my name and yours-and 
double-strands of DNA  had tiny copies of me a nd you along backbones, always 
paired up, just as adenine and thymine always Isn't that a strange symbolic image? 
Tortoise: Phooey! Who believes in that silly kind of stuff? Anyway, about `C' and `G'? Achilles: Well, I suppose `C' could stand for Mr. Crab, instead o cytosine. I'm not sure

Tortoise: Phooey! Who believes in that silly kind of stuff? Anyway, about `C' and `G'? Achilles: Well, I suppose `C' could stand for Mr. Crab, instead o cytosine. I'm not sure 
about `G', but I'm sure one could thin so mething. Anyway, it was amusing to imagine 
my DNA being with minuscule copies of you- as well as tiny copies of myself, for 
matter. Just think of the infinite regress THAT leads to! 
Tortoise: I can see you were not paying too much attention to the lecture. Achilles: No, you're wrong. I was doing my best, only I had a hard keeping fancy 
separated from fact. After all, it is such a strange netherworld that those molecular biologists are exploring.

separated from fact. After all, it is such a strange netherworld that those molecular biologists are exploring. 
Tortoise: How do you mean? Achilles: Molecular biology is filled with p eculiar convoluted loops which I can't quite 
understand, such as the way that folded proteins, which are coded for in DNA, can loop back and manipulate the DNA which cam e from, possibly even destroying it. 
Such strange loops always confuse the daylights out of me. They're eerie, in a way. 
Tortoise: I find them quite appealing. Achilles: You would, of course-they're just  down your alley. But me, sometimes I like to

Tortoise: I find them quite appealing. Achilles: You would, of course-they're just  down your alley. But me, sometimes I like to 
retreat from all this analytic thought any med itate a little, as an antidote. It clears my 
mind of all those conf loops and incredible  complexities which we were hearing about 
tonight. 
Tortoise: Fancy that. I wouldn't have guessed that you were a meditator Achilles: Did I never tell you that I am studying Zen Buddhism? Tortoise: Heavens, how did you come upon that? Achilles: I have always had a yen for the yin and yang, you know – the

 240whole Oriental mysticism trip, with the I Ching, gurus, and whatnot. So one day I'm 
thinking to myself, "Why not Zen too?" And that's how it all began. 
Tortoise: Oh, splendid. Then perhaps I can fi nally become enlightened. Achilles: Whoa, 
now. Enlightenment is not the first step on th e road to Zen; if anything, it'. the last 
one! Enlightenment is not for novices like you, Mr. T! 
Tortoise: I see we have had a misunderstanding. By "enlightenment", I hardly meant 
something so weighty as is meant in Zen. All I meant is that I can perhaps become enlightened as to what Zen is all about. Achilles: For Pete's sake, why didn't you say 
so? Well, I'd be only too happy to tell you what I know of Zen. Perhaps you might 
even be tempted to become a student of it, like me.

even be tempted to become a student of it, like me. 
Tortoise: Well, nothing's impossible. Achilles: You could study with me under my master, Okanisama-the seventh patriarch. Tortoise: Now what in the world does that mean? Achilles: You have to know the history of Zen to understand that. Tortoise: Would you tell me a little of the history of Zen, then? Achilles: An excellent idea. Zen is a kind of Buddhism which was founded by a monk 
named Bodhidharma, who left India and went to China around the sixth century. 
Bodhidharma was the first patriarch. The sixth one was Eno. (I've finally got it 
straight now!)

straight now!) 
Tortoise: The sixth patriarch was Zeno, eh? I fi nd it strange that he, of all people, would 
get mixed up in this business. 
Achilles: I daresay you underestimate the valu e of Zen. Listen just a little more, and 
maybe you'll come to appreciate it. As I wa s saying, about five hundred years later, 
Zen was brought to Japan, and it took hold very  well there. Since that time it has been 
one of the principal religions in Japan. 
Tortoise: Who is this Okanisama, the "seventh patriarch"? 
Achilles: He is my master, and his teachings  descend directly from those of the sixth 
patriarch. He has taught me that reality  is one, immutable, and unchanging; all 
plurality, change, and motion are mere illusions of the senses. 
Tortoise: Sure enough, that's Zeno, a mile away. But how ever did he come to be tangled 
up in Zen? Poor fellow!

up in Zen? Poor fellow! 
Achilles: Whaaat? I wouldn't put it that way. If ANYONE is tangled up, it's ... But that's 
another matter. Anyway, I don't know the answ er to your question. Instead, let me tell 
you something of the teachings of my master . I have learned that in Zen, one seeks 
enlightenment, or SATORI-the state of "No-mind". In this state, one does not think 
about the world-one just is. I have also l earned that a student of Zen is not supposed 
to "attach" to any object or thought or person- which is to say, he must not believe in,

to "attach" to any object or thought or person- which is to say, he must not believe in, 
or depend on, any absolute-not even this philosophy of nonattachment.

 241Tortoise: Hmm ... Now THERE'S something I could like about Achilles: I had a hunch 
you'd get attached to it. 
Tortoise: But tell me: if Zen rejects in tellectual activity, does it make sense to 
intellectualize about Zen, to study it rigorously? 
Achilles: That matter has troubled me quite a b it. But I think I have finally worked out an 
answer. It seems to me that you may be gin approaching Zen through any path you 
know-even if it is completely antithetical to Zen. As you approach it, you gradually 
learn to stray from that pa th. The more you stray from the path, the closer you get to 
Zen. 
Tortoise: Oh, it all begins to sound so clear now. Achilles: My favorite path to Zen is through th e short, fascinating and weird Zen parables 
called "koans". 
Tortoise: What is a koan? Achilles: A koan is a story about Zen masters and their student times it is like a riddle; 
other times like a fable; and other ti nothing you've ever heard before.

Zen. 
other times like a fable; and other ti nothing you've ever heard before. 
Tortoise: Sounds rather intriguing. Would you sa y that to read al koans is to practice 
Zen? 
Achilles: I doubt it. However, in my opinion, a delight in koans million times closer to 
real Zen than reading volume after about Zen, written in heavy philosophical jargon. 
Tortoise: I would like to hear a koan or two. Achilles: And I would like to tell you one-or a few. Perhaps begin with the most famous 
one of all. Many centuries ago, the Zen master named Joshu, who lived to be 119 
years old.

Zen? 
years old. 
Tortoise: A mere youngster! Achilles: By your standards, yes. Now one  day while Joshu and monk were standing 
together in the monastery, a dog wand The monk asked Joshu, "Does a dog have 
Buddha-nature,  
Tortoise: Whatever that is. So tell me-what did Joshu reply?  Achilles: 'MU'. Tortoise: 'MU? What is this 'MU'? What  about the dog? What about Buddha-nature? 
What's the answer? 
Achilles: Oh, but 'MU' is Joshu's answer. By saying 'MU', Joshu let the other monk know

years old. 
Achilles: Oh, but 'MU' is Joshu's answer. By saying 'MU', Joshu let the other monk know 
that only by not asking such questions can one know the answer to them. 
Tortoise: Joshu "unasked" the question. Achilles: Exactly! Tortoise: 'MU' sounds like a handy thing to have around. I'd like unask a question or two, 
sometimes. I guess I'm beginning to get the hang of Zen. Do you know any other 
koans, Achilles? I would like to hear some more. 
Achilles: My pleasure. I can tell you a pair of koans which go together  
Only ... 
Tortoise: What's the matter?

 242Achilles: Well, there is one problem. Although both are widely told koans, my master has 
cautioned me that only one of them is ge nuine. And what is more, he does not know 
which one is genuine, and which one is a fraud. 
Tortoise: Crazy! Why don't you tell them both to me and we can speculate to our hearts' 
content! 
Achilles: All right. One of the alleged koans goes like this:  
A monk asked Baso: "What is Buddha?" 
Baso said: "This mind is Buddha." 
 Tortoise: Hmm ... "This mind is Buddha"? So metimes I don't quite understand what these 
Zen people are getting at. Achilles: You might prefer the other alleged koan then. 
Tortoise: How does it run? Achilles: Like this:  
Baso said: "This mind is not Buddha." 
 Tortoise: My, my! If my shell isn't green a nd not green! I like that! Achilles: Now, Mr. T-
you're not supposed to just "like" koans.

you're not supposed to just "like" koans. 
Tortoise: Very well, then-I don't like it. Achilles: That's better. Now as I was saying, my master believes only one of the two is 
genuine. 
Tortoise: I can't imagine what led him to such a belief. But anyway, I suppose it's all 
academic, since there's no way to know if a koan is genuine or phony. 
Achilles: Oh, but there you are mistaken. My master has shown us how to do it. Tortoise: Is that so? A decision procedure for genuineness of koans? I should very much 
like to hear about THAT.

genuine. 
like to hear about THAT. 
Achilles: It is a fairly complex ritual, invol ving two stages. In the first stage, you must 
TRANSLATE the koan in question into a pi ece of string, folded all around in three 
dimensions. 
Tortoise: That's a curious thing to do. And what is the second stage? Achilles: Oh, that's easy-all you need to do is determine whether the string has Buddha-
nature, or not! If it does, then the koan is genuine-if not, the koan is a fraud. 
Tortoise: Hmm ... It sounds as if all you've done is transfer the need for a decision

Tortoise: Hmm ... It sounds as if all you've done is transfer the need for a decision 
procedure to another domain. Now it's a d ecision procedure for Buddha-nature that 
you need. What next? After all, if you can 't even tell whether a Do(; has Buddha-
nature or not, how can you expect to do so for every possible folded string? 
Achilles: Well, my master explained to me that shifting between domains can help. It's 
like switching your point of view. Things sometimes look complicated from one angle, but simple from another. He gave the example of an orchard, in which from 
one direction no order is

 243
 
FIGURE 45. La Mezquita, by M. C. Escher (black and white chalk, 1936 
  apparent, but from special angles, beautiful  regularity em, You've reordered the same 
information by changing your way of looking at it. 
Tortoise: I see. So perhaps the genuineness of a koan is concealed how very deeply inside 
it, but if you translate it into a string it ma in some way to float to the surface? 
Achilles: That's what my master has discovered. Tortoise: Then I would very much like to l earn about the techniqu first, tell me: how can 
you turn a koan (a sequence of words) fold ed string (a three-dimensional object)? 
They are rather dif kinds of entities. 
Achilles: That is one of the most mysterious things I have learned i There are two steps: 
"transcription" and "translation". TRANS CF a koan involves writing it in a phonetic 
alphabet, which contain four geometric sy mbols. This phonetic rendition of the koan 
is called the MESSENGER.

is called the MESSENGER. 
Tortoise: What do the geometric symbols look like? Achilles: They are made of hexagons and pentagons. Here is what they

 244look like ( picks up a nearby napkin, and draws for the Tortoise these four figures ): 
 
Tortoise: They are mysterious-looking. Achilles: Only to the uninitiated. Now on ce you have made the messenger, you rub your 
hands in some ribo, and 
Tortoise: Some ribo? Is that a kind of ritual anointment? Achilles: Not exactly. It is a special stic ky preparation which makes the string hold its 
shape, when folded up. Tortoise: What is it made of? 
Achilles: I don't know, exactly. But it feels so rt of gluey, and it works exceedingly well. 
Anyway, once you have some ribo on your hands, you can TRANSLATE the sequence of symbols in the messenger into certa in kinds of folds in the string. It's as 
simple as that. Tortoise: Hold on! Not so fast! How do you do that? 
Achilles: You begin with the string entirely straight. Then you go to one end and start 
making folds of various types, according to the geometric symbols in the messenger.

making folds of various types, according to the geometric symbols in the messenger. 
Tortoise: So each of those geometric symbols st ands for a different way to curl the string 
up? 
Achilles: Not in isolation. You take them three at a time, instead of one at a time. You 
begin at one end of the string, and one end of  the messenger. What to do with the first 
inch of the string is determined by the firs t three geometric symbols. The next three 
symbols tell you how to fold the second in ch of string. And so you inch your way

up? 
symbols tell you how to fold the second in ch of string. And so you inch your way 
along the string and simultaneously along the messenger, folding each little segment of string until you have exhausted the messenger. If you have properly applied some ribo, the string will keep its folded shape, and what you thereby produce is the translation of the koan into a string. 
Tortoise: The procedure has a certain eleg ance to it. You must get some wild-looking 
strings that way. 
Achilles: That's for sure. The longer koans translate into quite bizarre shapes. Tortoise: I can imagine. But in order to carry  out the translation of the messenger into the

Achilles: That's for sure. The longer koans translate into quite bizarre shapes. Tortoise: I can imagine. But in order to carry  out the translation of the messenger into the 
string, you need to know what kind of fold each triplet of geometric symbols in the 
messenger stands for. How do you know this? Do you have a dictionary? 
Achilles: Yes-there is a venerated book whic h lists the "Geometric Code". If you don't 
have a copy of this book, of course, you can't translate a koan into a string.

 245 
Tortoise: Evidently not. What is the origin of the Geometric Code Achilles: It came from 
an ancient master known as "Great Tutor" w ho my master says is the only one ever to 
attain the Enlightenment ' Enlightenment. 
Tortoise: Good gravy! As if one level of the stuff weren't enough. But then there are 
gluttons of every sort-why not gluttons for enlighten] Achilles: Do you suppose that 
"Enlightenment 'Yond Enlighten] stands for "EYE"? 
Tortoise: In my opinion, it's rather doubtful that it stands for you, Ac More likely, it 
stands for "Meta-Enlightenment"-"ME", that is 
Achilles: For you? Why would it stand for you? You haven't even re; the FIRST stage of 
enlightenment, let alone the 
Tortoise: You never know, Achilles. Perh aps those who have learn( lowdown on 
enlightenment return to their state before enlighten I've always held that "twice 
enlightened is unenlightened." But le back  to the Grand Tortue-uh, I mean the Great 
Tutor.

Tutor. 
Achilles: Little is known of him, except that he also invented the Art of Zen Strings. Tortoise: What is that? Achilles: It is an art on which the decision procedure for Buddha-nature is based. I shall 
tell you about it. 
Tortoise: I would be fascinated. There is so much for novices like absorb! Achilles: There is even reputed to be a koan which tells how the Art Strings began. But

Tutor. 
Tortoise: I would be fascinated. There is so much for novices like absorb! Achilles: There is even reputed to be a koan which tells how the Art Strings began. But 
unfortunately, all this has long since been lost sands of time, and is no doubt gone forever. Which may be just a for otherwise there would be imitators who would take on the m~ name, and copy him in other ways. 
Tortoise: But wouldn't it be a good thing if all students of Zen copied that most 
enlightened master of all, the Great Tutor? 
Achilles: Let me tell you a koan about an imitator. 
 
Zen master Gutei raised his finger whenev er he was asked a question about Zen. A

Zen master Gutei raised his finger whenev er he was asked a question about Zen. A 
young novice began to irritate him in this way. When Gut was told about the novice's imitation, he sent for him and asked him if were true. The novice 
admitted it was so. Gutei asked him if he understood. In reply the novice held up 
his index finger. Gutei promptly cut off.  The novice ran from the room, howling in 
pain. As he reached it threshold, Gute i called, "Boy!" When the novice turned, 
Gutei raised h index finger. At that instant the novice vas enlightened.

Gutei raised h index finger. At that instant the novice vas enlightened. 
 Tortoise: Well, what do you know! Just when I thought Zen was all about Joshu and his 
shenanigans, now I find out that Gutei is in on the merriment too. He seems to have quite a sense of humor.  
Achilles: That koan is very serious. I don't know how you got the idea that it is 
humorous. 
Tortoise: Perhaps Zen is instructive because it is humorous. I would guess

 246that if you took all such stories entirely se riously, you would miss the point as often as 
you would get it. 
Achilles: Maybe there's something to your Tortoise-Zen. Tortoise: Can you answer just one question for me? I would like to know this: Why did 
Bodhidharma come from India into China? 
Achilles: Oho! Shall I tell you what Joshu said when he was asked that very question? Tortoise: Please do. Achilles: He replied, "That oak tree in the garden." Tortoise: Of course; that's just what I would have said. Except that I would have said it in 
answer to a different question-namely, "Where can I find some shade from the 
midday sun?" 
Achilles: Without knowing it, you have inadvertently hit upon one of the basic questions 
of all Zen. That question, innocent though it sounds, actually means, "What is the basic principle of Zen?" 
Tortoise: How extraordinary. I hadn't the slight est idea that the central aim of Zen was to 
find some shade.

find some shade. 
Achilles: Oh, no-you've misunderstood me entir ely. I wasn't referring to THAT question. 
I meant your question about why Bodhidharma came from India into China. 
Tortoise: I see. Well, I had no idea that I wa s getting into such deep waters. But let's 
come back to this curious mapping. I gather that any koan can be turned into a folded 
string by following the method you outlined. No w what about the reverse process? 
Can any folded string be read in such a way as to yield a koan?

Can any folded string be read in such a way as to yield a koan? 
Achilles: Well, in a way. However .. . Tortoise: What's wrong? Achilles: You're just not supposed to do it that way 'round. It would violate the Central 
Dogma of Zen strings, you see, which is contained in this picture ( picks up a napkin 
and draws ): 
 
koan => messenger folded string 
transcription translation 
 
You're not supposed to go against the arrows-especially not the second one. 
Tortoise: Tell me, does this Dogma have Buddha -nature, or not? Come to think of it, I

Tortoise: Tell me, does this Dogma have Buddha -nature, or not? Come to think of it, I 
think I'll unask the question. Is that all right? 
Achilles: I am glad you unasked the questi on. But-I'll let you in on a secret. Promise you 
won't tell anyone? 
Tortoise: Tortoise's honor. Achilles: Well, once in a while, I actually do go against the arrows. I get sort of an illicit 
thrill out of it, I guess. 
Tortoise: Why, Achilles! I had no idea you would do something so irreverent! Achilles: I've never confessed it to anyone before-not even Okanisama.

 247Tortoise: So tell me, what happens when you go against the arrows i Central Dogma? 
Does that mean you begin with a string and m koan? 
Achilles: Sometimes-but some weirder things can happen. Tortoise: Weirder than producing koans? Achilles: Yes ... When you untranslate and unt ranscribe, you get THING, but not always 
a koan. Some strings, when read out Ion way, only give nonsense. 
Tortoise: Isn't that just another name for koans? Achilles: You clearly don't have the true spirit of Zen yet. Tortoise: Do you always get stories, at least? Achilles: Not always-sometimes you get nonsense syllables, other you get ungrammatical 
sentences. But once in a while you get seems to be a koan. 
Tortoise: It only SEEMS to be one? Achilles: Well, it might be fraudulent. you see. Tortoise: Oh, of course. Achilles: I call those strings which yield apparent koans "well-foi strings. Tortoise: Why don't you tell me about the decision procedure which allows you to

Tortoise: It only SEEMS to be one? Achilles: Well, it might be fraudulent. you see. Tortoise: Oh, of course. Achilles: I call those strings which yield apparent koans "well-foi strings. Tortoise: Why don't you tell me about the decision procedure which allows you to 
distinguish phony koans from the genuine article? 
Achilles: That's what I was heading towards. Given the koan, or non• as the case may be, 
the first thing is to translate it into the dime nsional string. All that's left is to find out 
if the strip Buddha-nature or not. 
Tortoise: But how do you do THAT? Achilles: Well, my master has said that the Great Tutor was able, I glancing at a string, to 
tell if it had Buddha-nature or not. 
Tortoise: But what if you have not reached the stage of the Enlightenment: 'Yond 
Enlightenment? Is there no other way to tell if a string hasi Buddha-nature? 
Achilles: Yes, there is. And this is where th e Art of Zen Strings come is a technique for

Achilles: Yes, there is. And this is where th e Art of Zen Strings come is a technique for 
making innumerably many strings, all of whit Buddha-nature. 
Tortoise: You don't say! A nd is there a corresponding way of n strings which DON'T 
have Buddha-nature? 
Achilles: Why would you want to do that? Tortoise: Oh, I just thought it might be useful. Achilles: You have the strangest taste. Imag ine! Being more intere things that DON'T 
have Buddha-nature than things that DO! 
Tortoise: Just chalk it up to my unenlighten ed state. But go on. T how to make a string 
which DOES have Buddha-nature.

which DOES have Buddha-nature. 
Achilles: Well, you must begin by draping a l oop of string over your in one of five legal 
starting positions, such as this one ... ( Picks up a string and drapes it in a simple loop 
between a finger on each hand.:)

 248Tortoise: What are the other four legal starting positions? 
Achilles: Each one is a position considered to be a SELF-EVIDENT manner of picking 
up a string. Even novices often pick up stri ngs in those positions. And these five 
strings all have Buddha-nature. Tortoise: Of course. 
Achilles: Then there are some String Mani pulation Rules, by which you can make more 
complex string figures. In particular, you are allowed to modify your string by doing 
certain basic motions of the hands. For instance, you can reach across like this-and 
pull like this-and twist like this. With each operation you are changing the overall 
configuration of the string draped over your hands. 
Tortoise: Why, it looks just like making cat's-cradles and such string figures! Achilles: That's right. Now as you watch, you'll see that some of these rules make the 
string more complex; some simplify it.  But whichever way you go, as long as you

string more complex; some simplify it.  But whichever way you go, as long as you 
follow the String Manipulation Rules, every string you produce will have Buddha-
nature. 
Tortoise: That is truly marvelous. Now what about the koan concealed inside this string 
you've just made? Would it be genuine? 
Achilles: Why, according to what I've learned, it must. Since I made it according to the 
Rules, and began in one of the five se lf-evident positions, the string must have 
Buddha-nature, and consequently it must correspond to a genuine koan.

nature. 
Buddha-nature, and consequently it must correspond to a genuine koan. 
Tortoise: Do you know what the koan is? Achilles: Are you asking me to violate the Central Dogma? Oh, you naughty fellow!  
(And with furrowed brow and code book in hand, Achilles points along the string 
inch by inch, recording each fold by a triplet of geometric symbols of the strange phonetic alphabet for koan, until he has nearly a napkinful.) 
 
Done! 
 Tortoise: Terrific. Now let's hear it. Achilles: All right.  
A traveling monk asked an old woman th e road to Taizan, a popular temple

Done! 
A traveling monk asked an old woman th e road to Taizan, a popular temple 
supposed to give wisdom to the one who worships there. The old woman said: "Go straight ahead." After the monk had pro ceeded a few steps, she said to herself, 
"He also is a common church-goer." Some one told this incident to Joshu, who 
said: "Wait until I investigate." The next day he went and asked the same question, 
and the old woman gave the same answer. Joshu remarked: "I have investigated 
that old woman."

that old woman." 
 Tortoise: Why, with his flair for investigations, it's a shame that Joshu never was hired by the FBI. Now tell me-what you did, I could also do, if I followed the Rules from the Art of Zen Strings, right?  Achilles: Right. Tortoise: Now would I have to perform the operations in just the same ORDER as you 
did?

 249.Achilles: No, any old order will do. 
Tortoise: Of course, then I would get a di fferent string, and consequently a different 
koan. Now would I have to perform the same NUMBER of steps as you did? 
Achilles: By no means. Any number of steps is fine. Tortoise: Well, then there are an infinite  number of strings with Buddha nature-and 
consequently an infinite number of ge nuine koans Howdo you know there is any 
string which CAN "I- be made by your Achille s: Oh, yes-back to things which lack 
Buddha-nature. It just so happens that  once you know how to make strings WITH 
Buddha nature, you can also make stri ngs WITHOUT Buddha-nature. That is 
something which my master drilled into me  right at the beg Tortoise: Wonderful! 
How does it work? 
Achilles: Easy. Here, for example-I'll make a string which lacks Buddha-nature .. .  
(He picks up the string out of which th e preceding koan was "pulled", ties a little 
teeny knot at one end of it, pulling it tight with his thumb forefinger.)

teeny knot at one end of it, pulling it tight with his thumb forefinger.) 
 This is it -- no Buddha-nature here.  Tortoise: Very illuminating. All it takes is adding a knot? How know that the new string 
lacks Buddha-nature? 
Achilles: Because of this fundamental prope rty of Buddha-nature; when two well-formed 
strings are identical but for a knot at one  end, then only ONE of them can have 
Buddha-nature. It's a rule of thumb which my master taught me. 
Tortoise: I'm just wondering about someth ing. Are there some strings with Buddha-

This is it -- no Buddha-nature here.  Tortoise: Very illuminating. All it takes is adding a knot? How know that the new string 
nature which you CAN'T reach by following the Rules of Zen Strings, no matter in 
what order? 
Achilles: I hate to admit it, but I am a little confused on this point myself. At first my 
master gave the strongest impression that Buddha in a string was DEFINED by 
starting in one of the five legal positions, and then developing the string according to the Rules. But then later, he said something about somebody-o "Theorem". I never got it straight. Maybe I even misheard said. But whatever he said, it put some doubt in

starting in one of the five legal positions, and then developing the string according to the Rules. But then later, he said something about somebody-o "Theorem". I never got it straight. Maybe I even misheard said. But whatever he said, it put some doubt in 
my mind as to this method hits ALL st rings with Buddha-nature. To the be 
knowledge, at least, it does. But Buddha-nature is a pretty elusive thing, you know. 
Tortoise: I gathered as much, from Joshu's 'MU'. I wonder ... Achilles: What is it? Tortoise: I was just wondering about those two koans-I mean t and its un-koan-the ones 
which say "This mind is Buddha" at mind is not Buddha"-what do they look like, 
when turned int via the Geometric Code? 
Achilles: I'd be glad to show you.

 250(He writes down the phonetic transcriptions, and then pulls from his pocket a 
couple of pieces of string, which he care fully folds inch by inch, following the 
triplets of symbols written in the strange alphabet. Then he places the finished strings side by side.) 
 
You see, here is the difference.  Tortoise: They are very similar, indeed. Why, I do believe there is only one difference 
between them: it's that one of them has a little knot on its end! 
Achilles: By Joshu, you're right. Tortoise: Aha! Now I understand why your master is suspicious. Achilles: You do? Tortoise: According to your rule of thumb, AT MOST ONE of such a pair can have 
Buddha-nature, so you know right away that one of the koans must be phony. 
Achilles: But that doesn't tell which one is phony. I've worked, and so has my master, at 
trying to produce these two strings by followi ng the String Manipulation Rules, but to 
no avail. Neither one ever turns up. It's quite frustrating. Sometimes you begin to wonder ...

no avail. Neither one ever turns up. It's quite frustrating. Sometimes you begin to wonder ... 
Tortoise: You mean, to wonder if either one has Buddha-nature? Perhaps neither of them 
has Buddha-nature-and neither koan is genuine! 
Achilles: I never carried my thoughts as far as  that-but you're right-it's possible, I guess. 
But I think you should not ask so many que stions about Buddha-nature. The Zen 
master Mumon always warned his pupils of the danger of too many questions. 
Tortoise: All right-no more questions. Instead, I have a sort of hankering to make a string 
myself. It would be amusing to see if what I come up with is well-formed or not. 
Achilles: That could be intere sting. Here's a piece of string. ( He passes one to the 
Tortoise.)

Tortoise.)  
Tortoise: Now you realize that I don't have the slightest idea what to do. 
We'll just have to take potluck with my  awkward production, which will follow no rules 
and will probably wind up being completely undecipherable. ( Grasps the loop 
between his feet and, with a few simple m anipulations, creates a complex string which 
he proffers wordlessly to Achilles. At that moment, Achilles' face lights up.)  
Achilles: Jeepers creepers! I'll have to try out your method myself. I have never seen a 
string like this!

string like this! 
Tortoise: I hope it is well-formed. Achilles: I see it's got a knot at one end. Tortoise: Oh just a moment! May I have it back? I want to do one thing to it. Achilles: Why, certainly. Here you are.  
(Hands it back to the Tortoise, who ties anot her knot at the same end. Then the 
Tortoise gives a sharp tug, and suddenly both knots disappear!)

 251Achilles: What happened? 
Tortoise: I wanted to get rid of that knot. Achilles: But instead of untying it, you tied another one, and then BOTH disappeared! 
Where did they go? 
Tortoise: Tumbolia, of course. That's the Law of Double Nodulation  
(Suddenly, the two knots reappear from out of nowhere-that is to say, Tumbolia.) 
 
Achilles: Amazing. They must lie in a fairly accessible layer of Tumbol they can pop into 
it and out of it so easily. Or is all of Tumbolia equally inaccessible? 
Tortoise: I couldn't say. However, it does occu r to me that burning string would make it 
quite improbable for the knots to come back such a case, you could think of them as 
being trapped in a deeper la of Tumbolia. Perhaps there are layers and layers of Tumbolia. that's neither here nor there.  What I would like to know is how string 
sounds, if you turn it back into phonetic sym bols. (As he hauls it back, once again, the 
knots pop into oblivion.)

knots pop into oblivion.) 
Achilles: I always feel so guilty about violating the Central Dogma ( Takes out his pen 
and code book, and carefully jots down the many sym triplets which correspond to the 
curvy involutions of the Tortoise's string;  when he is finished, he clears his voice .) 
Ahem. Are you ready to hear w you have wrought? 
Tortoise: I'm willing if you're willing. Achilles: All right. It goes like this: A certain monk had a habit of pestering the Grand Tortue (the only one who had ever 
reached the Enlightenment 'Yond Enlightenme nt), by asking whether various objects

reached the Enlightenment 'Yond Enlightenme nt), by asking whether various objects 
had Buddha-nature or not. To such questions  Tortue invariably sat silent. The monk 
had already asked about a bean, a lake, and a moonlit night. One day, he brought to 
Tortue a piece of string, and asked the sa me question. In reply, the Grand Tortue 
grasped the loop between his feet and 
Tortoise: Between his feet? How odd! Achilles: Why should you find that odd? 
Tortoise: Well, ah ... you've got a point there. But please go on!  Achilles: All right.

Tortoise: Well, ah ... you've got a point there. But please go on!  Achilles: All right.  
The Grand Tortue grasped the loop between    his feet and, with a few simple manipulations, created a complex string which he proffered wordlessly to the 
monk. At that moment, the monk was enlightened. 
 Tortoise: I'd rather be twice-enlightened, personally. Achilles: Then it tells how to make the Gr and Tortue's string, if you be, with a string 
draped over your feet. I'll skip those boring details concludes this way: 
 
From then on, the monk did not bother Tort ue. Instead, he made string after string

From then on, the monk did not bother Tort ue. Instead, he made string after string 
by Tortue's method; and he passed the me thod on to his own disciples, who passed 
it on to theirs.

 252Tortoise: Quite a yarn. It's hard to believe it was really hidden inside my string. 
Achilles: Yet it was. Astonishingly, you seem to have created a well-formed string right 
off the bat. 
Tortoise: But what did the Grand Tortue's stri ng look like? That's the main point of this 
koan, I'd suppose. 
Achilles: I doubt it. One shouldn't "attach" to small details like that inside koans. It's the 
spirit of the whole koan that counts, not little parts of it. Say, do you know what I just 
realized? I think, crazy though it sounds, that you may have hit upon that long-lost koan which describes the very origin of the Art of Zen Strings! 
Tortoise: Oh, that would almost be too good to have Buddha-nature. Achilles: But that means that the great maste r-the only one who ever reached the mystical 
state of the Enlightenment 'Yond Enlighten ment-was named "Tortue", not "Tutor". 
What a droll name! 
Tortoise: I don't agree. I think it's a handsom e name. I still want to know how Tortue's

Tortoise: I don't agree. I think it's a handsom e name. I still want to know how Tortue's 
string looked. Can you possibly recreate it from the description given in the koan? 
Achilles: I could try ... Of course, I'll have to use my feet, too, since it's described in 
terms of foot motions. That's pretty unusual.  But I think I can manage it. Let me give 
it a go. ( He picks up the koan and a piece of st ring, and for a few minutes twists and 
bends the string in arcane ways until he has the finished product .) Well, here it is. 
Odd, how familiar it looks.

Odd, how familiar it looks.  
Tortoise: Yes, isn't that so? I wonder where I saw it before? Achilles: I know! Why, this 
is YOUR string, Mr. T! Or is Tortoise: Certainly not. 
Achilles: Of course not-it's the string wh ich you first handed to me, before you took it 
back to tie an extra knot in it.  
Tortoise: Oh, yes-indeed it is. Fancy that. I wonder what that implies. Achilles: It's strange, to say the least.  
Tortoise: Do you suppose my koan is genuine?

Tortoise: Do you suppose my koan is genuine?  
Achilles: Wait just a moment ...  Tortoise: Or that my string has Buddha-nature? Achilles: Something about your string is beginning. to trouble me, Mr.Tortoise .  Tortoise ( looking most pleased with himse lf and paying no attention to Achilles ): And 
what about Tortue's string? Does it have Buddha nature? There are a host of questions 
to ask! 
Achilles: I would be scared to ask such questions, Mr. T. There is something mighty

to ask! 
Achilles: I would be scared to ask such questions, Mr. T. There is something mighty 
funny going on here, and I'm not sure I like it. Tortoise: I'm sorry to hear it. I can't imagine what's troubling you. Achilles: Well, the best way I know to explain it is to 
quote the words of another old Zen master, Kyogen.

 253 
 
Kyogen said: Zen is like a man hanging in a tree by his teeth over a precipice. His 
har grasp no branch, his feet rest on no limb, and under the tree anotl person asks 
him: "Why did Bodhidharma come to China from India?" the man in the tree does 
not answer, he fails; and if he does answer , falls and loses his life. Now what shall 
he do? 
 Tortoise: That's clear; he should give up Zen, and take up molecular biology.

Mumon and Gödel 
 254CHAPTER IX 
 
What Is Zen? 
 I'M NOT SURE I know what Zen is. In a wa y, I think I understand it very well; but in a 
way, I also think I can never understand it at a ll. Ever since my freshman English teacher 
in college read Joshu's MU out loud to our cl ass, I have struggled with Zen aspects of 
life, and probably I will never cease doing s o. To me, Zen is intellectual quicksand-
anarchy, darkness, meaninglessness, chaos. It is tantalizing and infuriating. And yet it is 
humorous, refreshing, enticing. Zen has its own special kind of meaning, brightness, and 
clarity. I hope that in this Chapter, I can get some of this cluster of reactions across to 
you. And then, strange though it may seem, that will lead us directly to Godelian matters. 
One of the basic tenets of Zen Buddhism is that there is no way to characterize 
what Zen is. No matter what verbal space you tr y to enclose Zen in, it resists, and spills

what Zen is. No matter what verbal space you tr y to enclose Zen in, it resists, and spills 
over. It might seem, then, that all efforts to explain Zen are complete wastes of time. But that is not the attitude of Zen masters and st udents. For instance, Zen koans are a central 
part of Zen study, verbal though they are. Koans are supposed to be "triggers" which, 
though they do not contain enough information in themselves to impart enlightenment, may possibly be sufficient to unlock the mech anisms inside one's mind that lead to 
enlightenment. But in general, the Zen attitude  is that words and truth are incompatible,

enlightenment. But in general, the Zen attitude  is that words and truth are incompatible, 
or at least that no words can capture truth.  
Zen Master Mumon 
 Possibly in order to point this out in an extreme way, the monk Mumon ("No-gate"), in the thirteenth century, compiled forty-eight koans, following each with a commentary and a small "poem". This work is called "The Gateless Gate" or the Mumonkan ("No-
gate barrier"). It is interesting to note th at the lives of Mumon and Fibonacci coincided 
almost exactly: Mumon living from 1183 to  1260 in China, Fibonacci from 1180 to 1250 
in Italy. To those who would look to the Mumonkan in hopes of making sense of, or 
"understanding", the koans, the Mumonkan may come as a rude shock, for the comments

"understanding", the koans, the Mumonkan may come as a rude shock, for the comments 
and poems are entirely as opaque as the koans  which they are supposed to clarify. Take 
this, for example:' -

 255
 
FIGURE 46. Three Worlds by M. C. Escher (lithograph, 1955)

 256Koan: 
 
Hogen of Seiryo monastery was about to lecture befo re dinner when he noticed that the bamboo screen, 
lowered for meditation, had not been rolled up. He pointed to it. Two monks arose wordlessly from the audience and rolled it up. Hogen, observing the physical moment, said, "The state of the first monk is good, 
not that of the second." 
 
Mumon's Commentary: 
 
I want to ask you: which of those two monks gained and which lost? If an y of you has one eye, he will see 
the failure on the teacher's part. Howeve r, I am not discussing gain and loss. 
 
Mumon's Poem: 
 
When the screen is rolled up the great sky opens,  
Yet the sky is not attuned to Zen.  It is best to forget the great sky  
And to retire from every wind. 
 
Or then again, there is this one:2  
 
Koan: 
 
Goso said: "When a buffalo goes out of his enclosure to the edge of the abyss, his horns and his 
head and his hoofs all pass through, but why can't the tail also pass?" 
 
Mumon's Commentary:

Koan: 
 
If anyone can open one eye at this point and say a word of Zen, he is qualified to repay 
the four gratifications, and, not only that, he can  save all sentient beings under him. But if 
he cannot say such a word of Zen, he should turn back to his tail. 
 
If the buffalo runs, he will fall into the trench;  
If he returns, he will be butchered.  That little tail 
Is a very strange thing. 
 
I think you will have to admit that Mumon does not exactly clear everything up. One

I think you will have to admit that Mumon does not exactly clear everything up. One 
might say that the metalanguage (in which Mum on writes) is not very different from the 
object language (the language of the koan). According to some, Mumon's comments are 
intentionally idiotic, perhaps meant to  show  how useless it is to spend one's time in 
chattering about Zen. How ever, Mumon's co mments can be taken on more than one 
level. For instance, consider this :3 
 
Koan: 
A monk asked Nansen: "Is there a teachi ng no master ever taught before?"  
Nansen said: "Yes, there is."

Koan: 
Nansen said: "Yes, there is." 
"What is it?" asked the monk. Nansen replied: "It is not mind, it is not Buddha, it is not things."

 257
 
FIGURE 47. Dewdrop, by M. C. Escher (mezzotint, 1948). 
 Mumon's Commentary:  Old Nansen gave away his treasure-words. He must have been greatly upset  Mumon's Poem:  
Nansen was too kind and lost his treasure. Truly, words have no power. Even though the mountain becomes the sea,  Words cannot open another's mind. 
 In this poem Mumon seems to be saying something very central to Zen, not making idiotic statements. Curiously, however, the poem is referential, and thus it is a comment 
not only on Nansen’s words, but on its own ine ffectiveness. This type of paradox is quite 
characteristic of Zen. It is an attempt to "b reak the mind of logic". You see this paradox 
quality in the koan, as well. Concerning Mumon's commentary, do think that Nansen was really so sure of his answer? Or did the "co rrectness of his answer matter at all? Or does 
correctness play any role in Zen? is the di fference between correctness and truth, or is

correctness play any role in Zen? is the di fference between correctness and truth, or is 
there any? What if Nansen had said, "No, th ere is not any such teaching"? Would it have 
r any difference? Would his remark have been immortalized in a koan?

 258
 
FIGURE 48. Another World, by M. C. Escher (wood-engraving, 1947). 
 Here is another koan which aims to break the mind of logic:'  The student Doko came to a Zen master, and said : "I am seeking the truth. In what state 
of mind should I train myself, so as to find it?" 
Said the master, "There is no mind, so you cannot put it in any state. There is no 
truth, so you cannot train yourself for it." 
"If there is no mind to train, and no truth_ to find, why do you have these monks 
gather before you every day to study- Zen and train themselves for this study?" 
"But I haven't an inch of room here," said the master, "so how could the monks 
gather? I have no tongue, so how could I call them together or teach them?"

 259"Oh, how can you lie like this?" asked Doko. 
"But if I have no tongue to talk to others, how can I lie to you?" asked the master. Then Doko said sadly, "I cannot follow you. I cannot understand you "I cannot understand myself," said the master. 
 If any koan serves to bewilder, this one doe s. And most likely, can bewilderment is its 
precise purpose, for when one is in a bewildered s one's mind does begin to operate nonlogically, to some extent. Only by stepping out side of logic, so the theory goes, can 
one make the lea enlightenment. But what is so bad about logic? Why does it prevent the to enlightenment?  
Zen's Struggle Against Dualism 
 To answer that, one needs to understand some thing about what enlightenment is. Perhaps 
the most concise summary of enlightenment  w be: transcending dualism. Now what is 
dualism? Dualism is the conceptual division of the world into categories. Is it possible to transcend this natural tendency? By pref ixing the word "division" by the word

dualism? Dualism is the conceptual division of the world into categories. Is it possible to transcend this natural tendency? By pref ixing the word "division" by the word 
"conceptual", I may have made it seem that th is is an intellectual or cons effort, and 
perhaps thereby given the impression that dualism could overcome simply by suppressing 
thought (as if to suppress thinking act were si mple!). But the breaking of the world into 
categories takes plat below the upper strata of thought; in fact, dualism is just as a perceptual division of the world into categor ies as it is a conceptual division In other 
words, human perception is by nature a dua listic phenomenon which makes the quest for

words, human perception is by nature a dua listic phenomenon which makes the quest for 
enlightenment an uphill struggle, to say the least. 
At the core of dualism, according to Zen, are words just plain w The use of words 
is inherently dualistic, since each word re presents, obviously, a conceptual category. 
Therefore, a major part of Zen is the agai nst reliance on words. To combat the use of 
words, one of the devices is the koan, where words are so deeply abused that one's mi 
practically left reeling, if one takes the koa ns seriously. Therefore perhaps wrong to say

practically left reeling, if one takes the koa ns seriously. Therefore perhaps wrong to say 
that the enemy of enlightenment is logic; ra ther dualistic, verbal thinking. In fact, it is 
even more basic than that: perception. As soon as you perceive an object, you draw a line 
between it and the rest of the world; you di vide the world, artificially, into parts you 
thereby miss the Way. 
 
Here is a koan which demonstrates the struggle against words: Koan:  Shuzan held out his short staff and said: "If you call this a short staff, you oppose its

Here is a koan which demonstrates the struggle against words: Koan:  Shuzan held out his short staff and said: "If you call this a short staff, you oppose its 
reality. If you do not call it a short staff, you ignore the fact. N, what do you wish to call 
this?"

 260
 
FIGURE 49. Day and Night, by M. C. Escher (woodcut, 1938). 
 Mumon's Commentary:  If you call this a short staff, you oppose its r eality. If you do not call it a short staff, you 
ignore the fact. It cannot be expressed with  words and it cannot be expressed without 
words. Now say quickly what it is.  Mumon's Poem: . 
Holding out the short staff, He gave an order of life or death.  Positive and negative interwoven, Even Buddhas and patriarchs cannot escape this attack. 
 ("Patriarchs" refers to six venerated founde rs of Zen Buddhism, of  whom Bodhidharma is 
the first, and Eno is the sixth.) 
Why is calling it a short staff opposing its reality? Probably because such a 
categorization gives the appearance of capturi ng reality, whereas the surface has not even 
been scratched by such a statement. It c ould be compared to saying "5 is a prime 
number". There is so much more-an infinity of facts-that has been omitted. On the other

("Patriarchs" refers to six venerated founde rs of Zen Buddhism, of  whom Bodhidharma is 
number". There is so much more-an infinity of facts-that has been omitted. On the other 
hand, not to call it a staff is, indeed, to ignore that particular fact, minuscule as it may be. 
Thus words lead to some truth-some false hood, perhaps, as well-but certainly not to all 
truth. Relying on words to lead you to the trut h is like relying on an incomplete formal 
system to lead you to the truth. A formal sy stem will give you some truths, but as we 
shall soon see, a formal system-no matter how  powerful-cannot lead to all truths. The 
dilemma of mathematicians is: what else is there to rely on, but formal systems? And the

dilemma of mathematicians is: what else is there to rely on, but formal systems? And the 
dilemma of

 261Zen people is, what else is there to rely on, but words? Mumon states t dilemma very 
clearly: "It cannot be expressed with words and it cannot expressed without words."  
Here is Nansen, once again:' 
 
Joshu asked the teacher Nansen, "What is the true Way?" Nansen answered, "Everyday way is the true Way.' Joshu asked, "Can I study it?" Nansen answered, "The more you study, the fu rther from the Way." Joshu asked, "If I 
don't study it, how can I know it?" Nansen answered, "The Way does not belong to things seen: nor to thing: unseen. It does not belong to things known: nor to th ings unknown. Do not seek it, study it, or 
name it. To find yourself on it, open yourself wide as the sky." [See Fig. 50.] 
 FIGURE 50. Rind, by M. C. Escher (wood-engraving, 1955).

 262This curious statement seems to abound w ith paradox. It is a little reminiscent of 
this surefire cure for hiccups: "Run around th e house three times without thinking of the 
word `wolf'." Zen is a philosophy which seems to have embraced the notion that the road 
to ultimate truth, like the only surefire cure for hiccups, may bristle with paradoxes. 
 
Ism, The Un-Mode, and Unmon 
 If words are bad, and thinking is bad, what is  good? Of course, to ask this is already 
horribly dualistic, but we are making no pretense  of being faithful to Zen in discussing 
Zen-so we can try to answer the question se riously. I have a name for what Zen strives 
for: ism. Ism is an antiphilosophy, a way of  being without thinki ng. The masters of ism 
are rocks, trees, clams; but it is the fate of hi gher animal species to have to strive for ism, 
without ever being able to attain it fully. S till, one is occasionally granted glimpses of 
ism. Perhaps the following koan offers such a glimpse 
:7

:7 
 
Hyakujo wished to send a monk to open a new monastery. He told his pupils that 
whoever answered a question most ably w ould be appointed. Placing a water vase on 
the ground, he asked: "Who can say what this is without calling its name?" 
The chief monk said: "No one can call it a wooden shoe." Isan, the cooking monk, tipped over the vase with his foot and went out. Hyakujo 
smiled and said: "The chief monk loses." And Isan became the  
master of the new monastery. 
 To suppress perception, to suppress logical, ve rbal, dualistic thinking-this is the essence

 To suppress perception, to suppress logical, ve rbal, dualistic thinking-this is the essence 
of Zen, the essence of ism. This is the Unm ode-not Intelligent, not Mechanical, just "Un". 
Joshu was in the Unmode, and that is w hy his 'MU' unasks the question. The Un-mode 
came naturally to Zen Master Unmon:
8 
 
One day Unmon said to his disciples, "This staff of mine has transformed itself into a 
dragon and has swallowed up the universe! Oh, where are the rivers and mountains 
and the great earth?" 
 Zen is holism, carried to its logical extreme.  If holism claims that things can only be

8 
 
 Zen is holism, carried to its logical extreme.  If holism claims that things can only be 
understood as wholes, not as sums of their part s, Zen goes one further, in maintaining that 
the world cannot be broken into parts at all. To divide the world into parts is to be deluded, and to miss enlightenment.  
A master was asked the question, "What is the Way?" by a curious monk. "

A master was asked the question, "What is the Way?" by a curious monk. " 
It is right before your eyes," said the master. "Why do I not see it for myself?" "Because you are thinking of yourself." "What about you: do you see it?" "So long as you see double, saying `I don't', and `you do', and so on, your 
 eyes are clouded," said the master. 
"When there is neither 'I' nor 'You', can one see it?" 
"When there is neither `I' nor `You', who is the one that wants to see it?"
9

 263Apparently the master wants to get acro ss the idea that an enlighte state is one 
where the borderlines between the self and th e rest of universe are dissolved. This would 
truly be the end of dualism, for a says, there is no system left which has any desire for 
perception. But what is that state, if not death? How can a live human being dissolve the 
borderlines between himself and the outside world? 
 
Zen and Tumbolia 
 The Zen monk Bassui wrote a letter to one of hi s disciples who was about to die, and in it 
he said: "Your end which is endless is as a snowflake dissolving in the pure air." The 
snowflake, which was once very much a di scernible subsystem of the universe, now 
dissolves into the larger system which 4 held it. Though it is no longer present as a 
distinct subsystem, its essence somehow still present, and will remain so. It floats in 
Tumbolia, along hiccups that are not being hi ccupped and characters in stories that are

Tumbolia, along hiccups that are not being hi ccupped and characters in stories that are 
being read . . . That is how I understand Bassui's message. 
Zen recognizes its own limitations, just as mathematicians have lea: to recognize 
the limitations of the axiomatic method as a method attaining truth. This does not mean 
that Zen has an answer to what beyond Zen a ny more than mathematicians have a clear 
understanding the forms of valid reasoning which lie outside of formalization. One o1 clearest Zen statements about the borderlines of Zen is given in the fol ing strange koan, 
very much in the spirit of Nansen:
10

10 
 
Tozan said to his monks, "You monks should know there is an even high understanding in Buddhism." A monk steppe d forward and asked, "What the higher 
Buddhism?" Tozan answered, "It is not Buddha." 
 There is always further to go; enlightenment is not the end-all of And there is no recipe 
which tells how to transcend Zen; the only thi ng can rely on for sure is that Buddha is not 
the way. Zen is a system cannot be its own metasystem; there is always something outside of which cannot be fully understood or described within Zen.

the way. Zen is a system cannot be its own metasystem; there is always something outside of which cannot be fully understood or described within Zen.  
Escher and Zen 
 In questioning perception and posing absurd  answerless riddles, Zen company, in the 
person of M. C. Escher. Consider Day a nd Night (Fig. 4 masterpiece of "positive and 
negative interwoven" (in the words of Mumoni).  One might ask, "Are those really birds, 
or are they really field it really night, or da y?" Yet we all know there is no point to such 
questions The picture, like a Zen koan, is tr ying to break the mind of logic. Es4 also 
delights in setting up contradictory pictures, such as Another World

 264
 
FIGURE 51. Puddle, by M. C. Escher (woodcut, 1952). 
 
(Fig. 48)-pictures that play with reality a nd unreality the same way as Zen plays with 
reality and unreality. Should one take Escher seriously? Should one take Zen seriously? 
There is a delicate, haiku-like study of reflections in Dewdrop  (Fig. 47); and then 
there are two tranquil images of the moon reflected in still waters: Puddle  (Fig. 51), and 
Rippled Surface  (Fig. 52). The reflected moon is a th eme which recurs in various koans. 
Here is an example:'  
Chiyono studied Zen for many years under Bukko of Engaku. Still, she could not 
attain the fruits of meditation. At last one  moonlit night she was carrying water in an 
old wooden pail girded with bamboo. The bamboo broke, and the bottom fell out of 
the pail. At that moment, she was set free. Chiyono said, "No more water in the pail, no more moon in the water." 
 Three Worlds : an Escher picture (Fig. 46), and the subject of a Zen koan:12

 Three Worlds : an Escher picture (Fig. 46), and the subject of a Zen koan:12 
 
A monk asked Ganto, "When the three worlds  threaten me, what shall I do?" Ganto 
answered, "Sit down." "I do not understand," said the monk. Canto said, "Pick up the 
mountain and bring it to me. Then I will tell you."

 265Hemiolia and Escher 
 
In Verbum (Fig. 149), oppositions are made into unities on several I Going around we see gradual transitions from black birds to white birds to black fish to white fish to black 
frogs to white frogs to black birds ... six st eps, back where we started! Is this a 
reconciliation of the dichotomy of black and wh ite? Or of the trichotomy of birds, fish, 
and frogs? Or sixfold unity made from th e opposition of the evenness of 2 an oddness of 
3? In music, six notes of equal time value create a rhythmic ambiguity-are they 2 groups 
of 3, or 3 groups of 2? This ambiguity ha s a name: hemiolia. Chopin was a master of 
hemiolia: see his Waltz op. his Etude op. 25, no. 2. In Bach, there is the Tempo di Menuetto from the keyboard Partita no. 5, or th e incredible Finale of the first Sonata 
unaccompanied violin, in G Minor. 
As one glides inward toward the center of  Verbum, the distinctions gradually blur,

As one glides inward toward the center of  Verbum, the distinctions gradually blur, 
so that in the end there remains not three,  not two, but one single essence: "VERBUM", 
which glows with brilliancy-perhaps a symbol  of enlightenment. Ironically, ` verbum" 
not only is a word, but "word"-not exactly th e most compatible notion with Zen. On the 
hand, "verbum" is the only word in the pict ure. And Zen master 1 once said, "The 
complete Tripitaka can be expressed in one character ("Tripitaka", meaning "three baskets", refers to the complete texts c original Buddhist writings.) What kind of

complete Tripitaka can be expressed in one character ("Tripitaka", meaning "three baskets", refers to the complete texts c original Buddhist writings.) What kind of 
decoding-mechanism, I wonder would it take to suck the three baskets out of one 
character? Perhaps one with two hemispheres.   
FIGURE 52. Rippled Surface, by M. C. Escher (lino-cut, 1950).

 266 
 
FIGURE 53. Three Spheres II, by M. C. Escher (lithograph, 1946), 
 
Indra's Net 
 
Finally, consider Three Spheres II  (Fig. 53), in which every pa rt of the world seems to 
contain, and be contained in, every other par t: the writing table reflects the spheres on top 
of it, the spheres reflect each other, as well as the writing table, the drawing of them, and 
the artist drawing it. The endless connections wh ich all things have to each other is only 
hinted at here, yet the hint is enough. The B uddhist allegory of "Indra's Net" tells of an 
endless net of threads throughout the unive rse, the horizontal threads running through 
space, the vertical ones through time. At every crossing of threads is an individual, and 
every individual is a crystal bead. The great  light of "Absolute Being" illuminates and

every individual is a crystal bead. The great  light of "Absolute Being" illuminates and 
penetrates every crystal bead; moreover, every crystal bead reflects not only the light from every other crystal in the net-but also  every reflection of every reflection throughout 
the universe. 
To my mind, this brings forth an image of renormalized particles: in every 
electron, there are virtual photons, positrons, neutrinos, muons ... ; in every photon, there 
are virtual electrons, protons, neutrons , pions ... ; in every pion, there are ... 
But then another image rises: that of pe ople, each one reflected in the minds of

But then another image rises: that of pe ople, each one reflected in the minds of 
many others, who in turn are mirrored in yet others, and so on. 
Both of these images could be represented in a concise, elegant way by using 
Augmented Transition Networks. In the case of particles, there would be one network for 
each category of particle; in the case of people,

 267one for each person. Each one would contain calls to many others, t creating a virtual 
cloud of ATN's around each ATN. Calling one we create calls on others, and this process 
might cascade arbitrarily far, un~ bottomed out.  
Mumon on MU 
 Let us conclude this brief excursion into Zen by returning to Mumon. H is his comment 
on Joshu's MU
:13 
 
To realize Zen one has to pass through the barrier of the patriarchs. Enlightenment 
always comes after the road of thinking is blocked. If you do nc pass the barrier of the 
patriarchs or if your thinking road is not blocked whatever you think, whatever you 
do, is like a tangling ghost. You may ask "What is a barrier of a patriarch?" This one word, 'MU', is it. 
This is the barrier of Zen. If you pass through it, you will see Joshu face t face. 
Then you can work hand in hand with the whole line of patriarchs. I this not a pleasant 
thing to do? 
If you want to pass this barrier, you must work through every bone in you body,

If you want to pass this barrier, you must work through every bone in you body, 
through every pore of your skin, filled with this question: "What `MU'?" and carry it day and night. Do not believe it is the co mmon negative symbol meaning nothing. It is 
not nothingness, the opposite of existence. I you really want to pass this barrier, you should feel like drinking a hot iro ball that you can neither swallow nor spit out. 
Then your previous lesser knowledge disa ppears. As a fruit ripening i season, 
your subjectivity and objectivity naturally become one. It is like dumb man who has

your subjectivity and objectivity naturally become one. It is like dumb man who has 
had a dream. He knows about it but he cannot tell i 
When he enters this condition his ego-she ll is crushed and he can shake th heaven 
and move the earth. He is like a great warrior  with a sharp sword. If Buddha stands in 
his way, he will cut him down; if a patriarch offers him an obstacle, he will kill him; 
and he will be free in his way of birth and death. H can enter any world as if it were 
his own playground. I will tell you how to d this with this koan:

his own playground. I will tell you how to d this with this koan: 
Just concentrate your whole energy in to this MU, and do not allow an 
discontinuation. When you enter this MU and there is no discontinuation -- your attainment will be as a candle burning and illuminating the who] universe. 
 
From Mumon to the MU-puzzle 
 From the ethereal heights of Joshu's MU, we  now descend to the private lowlinesses of 
Hofstadter's MU . . . I know that you have already concentrated your whole energy into 
this MU (when you read Chapter 1). So n wi sh to answer the question which was posed 
there:

there:  
Has MU theorem-nature, or not? 
 The answer to this question is not an evasive MU; rather, it is a resounding NO. In order to show this, we will take advantage of dualistic, logical thinking.

 268We made two crucial observations in Chapter I: 
 
(1) that the MU-puzzle has depth largel y because it involves the interplay of 
lengthening and shortening rules; 
 (2) that hope nevertheless exists for cracking the problem by employing a tool which 
is in some sense of adequate depth to  handle matters of that complexity: the 
theory of numbers. 
 We did not analyze the MU-puzzle in those term s very carefully in Chapter I; we shall do 
so now. And we will see how the second observation (when generalized beyond the insignificant MIU-system) is one of the most fr uitful realizations of all mathematics, and 
how it changed mathematicians' view of their own discipline. 
For your ease of reference, here is a recapitulation of the MIU-system:  
SYMBOLS: M, I, U  Axiom: MI  RULES:  
I. If xl is a theorem, so is xIU. II. If Mx is a theorem, so is Mxx. III. In any theorem, III can be replaced by U.  IV. UU can be dropped from any theorem. 
 
Mumon Shows Us How to Solve the MU-puzzle

Mumon Shows Us How to Solve the MU-puzzle 
 According to the observations above, then, the MU-puzzle is merely a puzzle about natural numbers in typographical disguise. If we  could only find a way to transfer it to the 
domain of number theory, we might be able  to solve it. Let us ponder the words of 
Mumon, who said, "If any of you has one eye,  he will see the failure on the teacher's 
part." But why should it matter to have one eye? 
If you try counting the number of l's contained in theorems, you will soon notice 
that it seems never to be 0. In other words,  it seems that no matter how much lengthening

that it seems never to be 0. In other words,  it seems that no matter how much lengthening 
and shortening is involved, we can never work in such a way that all I's are eliminated. 
Let us call the number of I's in any string th e I-count of that string. Note that the I-count 
of the axiom MI is 1. We can do more than s how that the I-count can't be 0-we can show 
that the I-count can never be any multiple of 3. 
To begin with, notice that rules I and IV leave the I-count totally undisturbed. 
Therefore we need only think about rules II and III. As far as rule III is concerned, it 
diminishes the I-count by exactly 3. After an a pplication of this rule, the I-count of the

diminishes the I-count by exactly 3. After an a pplication of this rule, the I-count of the 
output might conceivably be a multiple of 3-but  only if the I-count of the input was also. 
Rule III, in short, never creates a multiple of  3 from scratch. It can only create one when 
it began with one. The same holds for rule II, which doubles the

 269I-count. The reason is that if 3 divides 2n, th en-because 3 does not dig 2-it must divide n 
(a simple fact from the theory of numbers ). Neither rule II nor rule III can create a 
multiple of 3 from scratch. 
But this is the key to the MU-puzzle! Here is what we know:  
(1)   The I-count begins at 1 (not a multiple of 3);  (2)   Two of the rules do not affect the I-count at all; (3) 
 
(3)  The two remaining rules which do affect  the I-count do so in such a way as never 
to create a multiple of 3 unless given one initially. 
 
The conclusion-and a typically hereditary one it is, too-is that I-count can never become 
any multiple of 3. In particular, 0 is a forbid value of the I-count. Hence, MU is not a 
theorem of the MIU-system. 
Notice that, even as a puzzle about I-c ounts, this problem was plagued by the 
crossfire of lengthening and shortening ru les. Zero became the goal; I-counts could 
increase (rule II), could decrease (rule III). 1 we analyzed the situation, we might have

increase (rule II), could decrease (rule III). 1 we analyzed the situation, we might have 
thought that, with enough switching back a nd forth between the rules, we might 
eventually hit 0. IS thanks to a simple num ber-theoretical argument, we know that the 
impossible. 
 
Gödel-Numbering the MIU-System 
 Not all problems of the the type which the MU- puzzle symbolizes at easy to solve as this 
one. But we have seen that at least one su ch pr could be embedded within, and solved

one. But we have seen that at least one su ch pr could be embedded within, and solved 
within, number theory. We are going to see that there is a way to embed all problems about any for system, in number theory. This can happen thanks to the discovery Gödel, 
of a special kind of isomorphism. To illustrate it, I will use MIU-system.  We begin by considering the notation of th e MIU-system. We map each symbol onto a

Not all problems of the the type which the MU- puzzle symbolizes at easy to solve as this 
of a special kind of isomorphism. To illustrate it, I will use MIU-system.  We begin by considering the notation of th e MIU-system. We map each symbol onto a 
new symbol:                                                         M   <= => 3                                                          I  <= =>  1                                                         U <= =>  0  The correspondence was chosen arbitrarily; the only rhyme or reason is that each symbol looks a little like the one it is mapped onto. I number is called the Gödel number of the 
corresponding letter. Now I sure you can gue ss what the Gödel number of a multiletter

corresponding letter. Now I sure you can gue ss what the Gödel number of a multiletter 
string will be:                                                          MU   <= => 30 
                                                       MIIU <= => 3110 
                                                              Etc.

 270It is easy. Clearly this mapping between notations is an information preserving 
transformation; it is like playing the same melody on two different instruments.  
Let us now take a look at a typical derivation in the MIU-system, written 
simultaneously in both notations:  (1) MI  axiom 31 (2) MII  rule 2 311 (3) MIIII  rule 2 31111 (4) MUI  rule 3 301 (5) MUIU  rule 1 3010 (6) MUIUUIU rule 2 3010010 (7) MUIIU rule 4 30110  The left-hand column is obtained by applying our four familiar typographical rules. The right-hand column, too, could be thought of as  having been generated by a similar set of 
typographical rules. Yet the right-hand column has a dual nature. Let me explain what this means.  
Seeing Things Both Typographically and Arithmetically 
 We could say of the fifth string ('3010') that  it was made from the fourth, by appending a 
`0' on the right; on the other hand we could e qually well view the transition as caused by

`0' on the right; on the other hand we could e qually well view the transition as caused by 
an arithmetical operation-multiplication by 10, to be exact. When natural numbers are written in the decimal system, multiplicati on by 10 and putting a `0' on the right are 
indistinguishable operations. We can take advant age of this to write an arithmetical rule 
which corresponds to typographical rule I:  ARITHMETICAL RULE la: A number whose decima l expansion ends on the right in `1' 
can be multiplied by 10.  We can eliminate the reference to the symbol s in the decimal expansion by arithmetically 
describing the rightmost digit:  ARITHMETICAL RULE Ib: A number whose re mainder when divided by 10 is 1, can

describing the rightmost digit:  ARITHMETICAL RULE Ib: A number whose re mainder when divided by 10 is 1, can 
be multiplied by 10.  Now we could have stuck with a purely t ypographical rule, such as the following one: 
 TYPOGRAPHICAL RULE I: From any theorem whose rightmost symbol is ' 1' a new 
theorem can be made, by appending `0' to the right of that 1'. They would have the same effect. This is w hy the right-hand column has a "dual nature": 
it can be viewed either as a series of typographical opera-

 271 
tions changing one pattern of symbols into anot her, or as a series arithmetical operations 
changing one magnitude into another. But the are powerful reasons for being more 
interested in the arithmetical version Ste pping out of one purely typographical system 
into another isomorphic typographical system is  not a very exciting thing to do; whereas 
stepping clear out of the typographical domain into an isomorphic part of number theory 
has some kind of unexplored potential. It is  as if somebody h known musical scores all 
his life, but purely visually-and then, all o: sudden, someone introduced him to the 
mapping between sounds a musical scores. What a rich, new world! Then again, it is as if 
somebody h been familiar with string figures a ll his life, but purely as string figur devoid 
of meaning-and then, all of a sudden, so meone introduced him the mapping between

of meaning-and then, all of a sudden, so meone introduced him the mapping between 
stories and strings. What a revelation! The discovery of Gödel-numbering has been likened to the discovery, by Descartes, of t isomorphism between curves in a plane and 
equations in two variables; incredibly simple, once you see it-and opening onto a vast 
new world 
Before we jump to conclusions, t hough, perhaps you would like to a more 
complete rendering of this higher level of the isomorphism. It i very good exercise. The 
idea is to give an arithmetical rule whose ac tion is indistinguishable from that of each 
typographical rule of the MIU-system:

new world 
typographical rule of the MIU-system: 
 
A solution is given below. In the rules, m and k are arbitrary natural numbers, and n is 
any natural number which is less than 10
m 
 RULE 1: If we have made 10m + 1, then we can make 10 x (10m + 1)  Example: Going from line 4 to line 5. Here, m = 30.  
RULE 2: If we have made 3 x 10" + n, then we can make 10' X X (3 x 10"'+n)+n.

m 
RULE 2: If we have made 3 x 10" + n, then we can make 10' X X (3 x 10"'+n)+n. 
Example: Going from line 1 to line 2, where both m and n equal 1.  RULE 3: If we have made k x 10 "`+ 111 x 10'+n, then we can make k x 10"+` + n. Example: Going from line 3 to line 4. Here, m and n are 1, and k is 3.  RULE 4: If we have made k x 10rn+z + n, k x 10" +n. then we can make  k x 10m + n Example: Going from line 6 to line 7. Here, m = 2, n = 10, and k = 301.  Let us not forget our axiom! Without it we can go nowhere. Therefore, let us postulate 
that:  
We can make 31.

that:  
We can make 31. 
 Now the right-hand column can be seen as a full-fledged arithmetic process, in a new 
arithmetical system which we might call the 310-system

 272I 
(1) 31  given (2) 311  rule 2 (m=1, n=1) (3) 31111  rule 2 (m=2, n=11) (4) 301  rule 3 (m=1, n=1, k=3) (5) 3010  rule 1 (m=30) (6) 3010010 rule 2 (m=3, n=10) (7) 30110  rule 4 (m=2, n=10, k=301)  Notice once again that the lengthening and shorte ning rules are ever with us in this "310-
system"; they have merely been transposed into the domain of numbers, so that the Godel 
numbers go up and down. If you look carefully at what is going on, you will discover that 
the rules are based on nothing more profound than the idea that shifting digits to left and 
right in decimal representations of integers is related to multiplications and divisions by 
powers of 10. This simple observation finds its generalization in the following  
CENTRAL PROPOSITION: If there is a typographical rule which tells how

CENTRAL PROPOSITION: If there is a typographical rule which tells how 
certain digits are to be shifted, changed, dropped, or inserted in any number represented decimally, then this rule can be represented equally well by an arithmetical counterpart which involves arithmetical operations with powers of 10 as well as additions, subtractions, and so forth. 
 
More briefly: 
 
Typographical rules for manipulating numerals are actually arithmetical rules for operating on numbers. 
 
This simple observation is at the heart of Gödel’s method, and it will have an absolutely

This simple observation is at the heart of Gödel’s method, and it will have an absolutely 
shattering effect. It tells us that once we have a Gödel numbering for any formal system, 
we can straightaway form a set of arith metical rules which complete the Gödel 
isomorphism. The upshot is that we can transf er the study of any formal system-in fact 
the study of all formal systems-into number theory.  
MIU-Producible Numbers 
 Just as any set of typographical rules genera tes a set of theorems, a corresponding set of

 Just as any set of typographical rules genera tes a set of theorems, a corresponding set of 
natural numbers will be generated by repeated applications of arithmetical rules. These 
producible numbers  play the same role inside number theory as theorems do inside any 
formal system. Of course, different num bers will be producible, depending on which 
rules are adopted. "Producible numbers" are only producible relative to a system  of 
arithmetical rules. For example, such numbers as 31, 3010010, 3111, and so forth could 
be called MIU -producible  numbers-an ungainly name, which might be shortened to

be called MIU -producible  numbers-an ungainly name, which might be shortened to 
MIU -numbers , symbolizing the fact that those numbers are the ones that result when you 
transcribe the MIU -system into number theory, via Gödel-numbering. If we were to 
Gödel-number the pq-system

 273and then "arithmetize" its rules, we could call the producible numbers "pq-numbers"-and 
so on. 
Note that the producible numbers (in any given system) are defined by a recursive 
method: given numbers which are known to be producible, we have rules telling how to make more producible numbers. Thus, the cl ass of numbers known to be producible is 
constantly extending itself, in much the same way that the list of Fibonacci numbers, or Q-numbers, does. The set of producible numbers of any system is a recursively 
enumerable set . What about its complement-the set of nonproducible numbers? Is that set 
always recursively enumerable? Do numbers which are nonproducible share some common arithmetical feature? 
This is the sort of issue which arises  when you transpose the study of formal 
systems into number theory. For each system  which is arithmetized, one can ask, "Can 
we characterize producible numbers in a simple way?" "Can we characterize

we characterize producible numbers in a simple way?" "Can we characterize 
nonproducible numbers in a recursively enumer able way?" These are difficult questions 
of number theory. Depending on the system wh ich has been arithmetized, such questions 
might prove too hard for us to resolve. But if  there is any hope for solving such problems, 
it would have to reside in the usual kind of step-by-step reasoning as it applies to natural 
numbers. And that, of course, was put in its qui ntessential form in the previous Chapter. 
TNT seemed, to all appearances, to have captured all valid mathematical thinking 
processes in one single, compact system.

processes in one single, compact system.  
Answering Questions about Producible Numbers 
by Consulting TNT 
 
Could it be, therefore, that the means with  which to answer any question about any 
formal system lies within just a single formal system- TNT ? It seems plausible. Take, for 
instance, this question: 
 
Is MU a theorem of the MIU -system? 
 Finding the answer is equivalent to determining whether 30 is a MIU  number or not. 
Because it is a statement of number theory, we  should expect that, with some hard work,

Because it is a statement of number theory, we  should expect that, with some hard work, 
we could figure out how to tr anslate the sentence "30 is a MIU -number" into TNT -
notation, in somewhat the same way as we figured out how to translate other number-theoretical sentences into TNT -notation. I should immediately caution the reader that 
such a translation, though it does exist, is i mmensely complex. If you recall, I pointed out 
in Chapter VIII that even such a simple arithme tical predicate as "b is a power of 10" is 
very tricky to code into TNT -notation-and the predicate "b is a MIU -number" is a lot

very tricky to code into TNT -notation-and the predicate "b is a MIU -number" is a lot 
more complicated than that! Still,  it can be found; and the numeral 
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSO  can be substituted for every b. This will 
result in a MON strous string of TNT , a string of TNT  which speaks about the MU-
puzzle. Let us therefore call that string " MUMON ". Through MUMON  and strings like 
it, TNT  is now capable of speaking "in code" about the MIU -system.

 274The Dual Nature of MUMON  
 
In order to gain some benefit from this p eculiar transformation of the original question, 
we would have to seek the answer to a new question:  
Is MUMON  a theorem of TNT ? 
 All we have done is replace one relatively short string ( MU) by another (the monstrous 
MUMON ), and a simple formal system (the  MIU-system) by a complicated one ( TNT ). 
It isn't likely that the answer will be any more forthcoming even though the question has been reshaped. In fact, TNT  has a full complement of both lengthening and shortening 
rules, and the reformulation of the question is  likely to be far harder than the original. 
One might even say that looking at MU via MUMON  is an intentionally idiotic way of 
doing things. However, MUMON  can be looked at on more than one level. 
In fact, this is an intriguing point: MUMON  has two different passive meanings. 
Firstly, it has the one which was given before:  
30 is a MIU -number.

30 is a MIU -number. 
 But secondly, we know that this statement is tied (via isomorphism) to the statement  
MU is a theorem of the MIU -system. 
 So we can legitimately quote this latter as the second passive meaning of MUMON . It 
may seem very strange because, after all, MUMON  contains nothing but plus signs, 
parentheses, and so forth-symbols of TNT . How can it possibly express any statement 
with other than arithmetical content? 
The fact is, it can. Just as a single mu sical line may serve as both harmony and 
melody in a single piece; just as " BACH " may be interpreted as both a name and a

melody in a single piece; just as " BACH " may be interpreted as both a name and a 
melody; just as a single sentence may be an accurate structural description of a picture by Escher, of a section of DNA , of a piece by Bach, and of the dialogue in which the 
sentence is embedded, so MUMON  can be taken in (at least)  two entirely different ways. 
This state of affairs comes about because of two facts: 
 
Fact 1. Statements such as " MU is a theorem" can be coded into number theory 
via Gödel’s isomorphism. 
 Fact 2. Statements of number theory can be translated into TNT .

 Fact 2. Statements of number theory can be translated into TNT . 
 It could be said that MUMON  is, by Fact 1, a coded message , where the symbols of the 
code are, by Fact 2, just symbols of TNT .

 275Codes and Implicit Meaning 
 
Now it could be objected here that a code d message, unlike an uncod message, does not 
express anything on its own-it requires knowledge  the code. But in reality there is no 
such thing as an uncoded message. There are only messages written in more familiar 
codes, and message written in less familiar codes. If the meaning of a message is to be revealed it must be pulled out of the code by some sort of mechanism, or isomorphism. It 
may be difficult to discover the method by which the decoding should be done; but once 
that method has been discovered, the message becomes transparent as water. When a code is familiar enough, it ceases appearing like a code; one forgets that there is a decoding mechanism. The message is identified with its meaning. 
Here we have a case where the identifica tion of message and meant is so strong

Here we have a case where the identifica tion of message and meant is so strong 
that it is hard for us to conceive of an alternate meaning: residing in the same symbols. Namely, we are so prejudiced by the symbols of TNT  towards seeing number-theoretical 
meaning (and only numb( theoretical meaning) in strings of TNT , that to conceive of 
certain string of TNT  as statements about the MIU -system is quite difficult. But Gödel’s 
isomorphism compels us to recognize this s econd level of meaning certain strings of 
TNT . 
 Decoded in the more familiar way, MUMON bears the message: 
 
TNT . 
 
 This is a statement of number theory, gotten by interpreting each sign the conventional way. 
But in discovering Gödel-numbering and the whole isomorphism bu upon it, we 
have in a sense broken a code in which messages about the MIU -system are written in 
strings of TNT . Gödel’s isomorphism is a n( information-revealer, just as the 
decipherments of ancient scripts we informa tion-revealers. Decoded by this new and less 
familiar mechanism MUMON  bears the message 
 
MU is a theorem of the MIU -system.

This is a statement of number theory, gotten by interpreting each sign the conventional way. 
 
 The moral of the story is one we have heard before: that meaning is ; automatic by-product of our recognition of any isomorphism; therefore there are at least two passive 
meanings of MUMON -maybe more! 
 
The Boomerang: Gödel-Numbering TNT 
 Of course things do not stop here. We have only begun realizing the: potential of Gödel’s 
isomorphism. The natural trick would be to turn TNT's  capability of mirroring other 
formal systems back on itself, as the Tort oise turned the Crab's phonographs against

The moral of the story is one we have heard before: that meaning is ; automatic by-product of our recognition of any isomorphism; therefore there are at least two passive 
 
formal systems back on itself, as the Tort oise turned the Crab's phonographs against 
themselves, and as his Goblet G turned agains t itself, in destroying itself. In order to do 
this, we

 276will have to Gödel-number TNT  itself, just as we did the MIU -system, and then 
"arithmetize" its rules of inference. The G ödel-numbering is easy to do. For instance, we 
could make the following correspondence: 
 
Symbol... Codon   Mnemonic Justification 
 
0  666  Number of the Beast for the Mysterious Zero S ..... 123   successorship: 1, 2, 3, = ..... 111  visual resemblance, turned sideways + .... 112  1+1=2 
. .... 236  2x3=6 
( .... 362  ends in 2 *    
) .... 323  ends in 3 *    <  212  ends in 2 *   these three pairs > .... 213  ends in 3 *    form a pattern [ .... 312  ends in 2 *    ] .... 313  ends in 3 *     
α .... 262  opposite to  V (626) 
´....   163  163 is prime 
∧ ..... 161  ´ ∧´ is a "graph" of the sequence 1-6-1 
∨ ..... 616  ´ ∨' is a "graph" of the sequence 6-1-6 
⊃ ..... 633  ` 6 "implies" 3 and 3, in some sense . 
~ ..... 223  . 2 + 2 is not 3 
ℑ ..... 333  ` ℑ' looks like `3' 
V ..... 626  opposite to a; also a "graph" of 6-2-6

V ..... 626  opposite to a; also a "graph" of 6-2-6 
.: ..... 636  two dots, two sixes 
 
punc. ..... 611  special number, as on Bell system (411, 911) 
 
Each symbol of TNT  is matched up with a triplet composed of the digits 1, 2, 3, 
and 6, in a manner chosen for mnemonic value. I shall call each such triplet of digits a 
Gödel codon , or codon  for short. Notice that I have given no codon for b, c, d , or e; we 
are using austere TNT . There is a hidden motivation for this, which you will find out

are using austere TNT . There is a hidden motivation for this, which you will find out 
about in Chapter XVI. I will explain the bottom entry, "punctuation", in Chapter XIV. 
Now we can rewrite any string or rule of TNT  in the new garb. Here, for instance, 
is Axiom 1 in the two notations, the old below the new: 
 
626,262,636,223,123,262,111,666 
 V    a :     ~      S    a     =     0 
 
Conveniently, the standard convention of putting in a comma every third digit happens to 
coincide with our colons, setting them off for "easy" legibility.

 V    a :     ~      S    a     =     0 
 
coincide with our colons, setting them off for "easy" legibility. 
 
Here is the Rule of Detachment, in the new notation: 
 
RULE: If x and 212x6331213 are both theorems, then 1 is a theorem. Finally, here is an 
entire derivation taken from last Chapter, give n in austere TNT and also transcribed into 
the new notation:

 277626,262,636,626262,163,636,362262,112,123,262,163,323, 111,123,362,262,112,262,163,323 axiom: 
 V     α   : :   V   α    '      :       (   α     +     5     α    '       )    =     S      (    α    +    α     '     ) 
626,262.163,636,362,123,666,112,123,262,163,323,111,123, 362,123,666,112,262,163,32  specification 
 V     α    '      :      (      S    0     +    S     α     ´      )     =     S    (        S    0    +     α   ´     ) 
362,123,666,112,123,666,323,1 11,123,362,123,666,112,666,323   specification 
  (      S     0     +     5      0     )     =     S     (      S    0      +    0      ) 
626,262,636,362 262,112,666, 3 23,111,262      axiom 
 V •  α    :    (     α    +     0       )      =    α 
362,123,666,112,666,323,111,123,666      specification  
   (     S     0     +     0      )    =      S 0 
123,362,123,666,112,666,323,11 1,123,123,666     insert '12; 
  S      (     S     0     +    0      )     =      S     S    0

(      S     0     +     5      0     )     =     S     (      S    0      +    0      ) 
   (     S     0     +     0      )    =      S 0 
  S      (     S     0     +    0      )     =      S     S    0 
362,123,666,112,123,666,323,111,123,123,666     transitivity  
   (     S     0     +     5    0      )     =      S    S     0 
 
Notice that I changed the name of the "Add S" rule to "Insert `123' ", since that is the 
typographical operation which it now legitimizes. 
This new notation has a pretty strange feel to it. You lose all sense o meaning; but 
if you had been brought up on it, you could read strings it this notation as easily as you do 
TNT. You would be able to look and, at glan ce, distinguish well-formed formulas from

(     S     0     +     5    0      )     =      S    S     0 
 
TNT. You would be able to look and, at glan ce, distinguish well-formed formulas from 
ill-formed ones. Naturally since it is so visu al, you would think of this as a typographical 
operation but at the same time, picking out well-formed formulas  in this notation i 
picking out a special class of integers , which have an arithmetical definition too. 
Now what about "arithmetizing" all the rules of inference? As matter stand, they 
are all still typographical rules. But wait!  According to the Central Proposition, a 
typographical rule is really equivalent to al  arithmetical rule. Inserting and moving digits

typographical rule is really equivalent to al  arithmetical rule. Inserting and moving digits 
in decimally represented numbers is an arithmetical  operation, which can be carried out 
typographically. Just as appending a 'O' on the end is exactly the same as multiplying b, 
10, so each rule is a condensed way of describing a messy arithmetical operation. Therefore, in a sense, we do not even need  to look for equivalent arithmetical rules, 
because all of the rules are already  arithmetical! 
 
TNT-Numbers: A Recursively Enumerable Set of Numbers 
 
Looked at this way, the preceding derivation of the theorem

Looked at this way, the preceding derivation of the theorem 
"362,123,666,112,123,666,323,111,123,123,666" is a sequence of high] convoluted number-theoretical transformations, each of which acts on one or more input numbers, 
and yields an output number, which is, as before, called a producible  number, or, to be 
more specific, a TNT -number . Some the arithmetical rules take an old TNT -number and 
increase it in a particular way, to yield a new TNT -number; some take an old TNT -
number a and decrease it; other rules take two TNT -numbers, operate on each of them 
some odd way, and then combine the results into a new TNT -number

some odd way, and then combine the results into a new TNT -number 
and so on and so forth. And instead  of starting with just one know: 'TNT -number, we 
have five initial TNT -numbers-one for each (austere axiom, of course. Arithmetized TNT  
is actually extremely similar to the

 278arithmetized MIU -system, only there are more rules and axioms, and to write out 
arithmetical equivalents explicitly would be a big bother-and quite unenlightening, incidentally. If you followed how it was done for the MIU-system, there ought to be no 
doubt on your part that it is quite analogous here. 
There is a new number-theoretical pr edicate brought into being by this 
"Godelization" of TNT : the predicate 
 
α is a TNT -number. 
 
For example, we know from the preceding derivation that 362,123,666,112,123,666,323,111,123,123,666 is a TNT -number, while on the other 
hand, presumably 123,666,111,666 is not a TNT -number. 
Now it occurs to us that this new number-theoretical! predicate is expressible  by 
some string of TNT  with one free variable, say a. We could put a tilde in front, and that 
string would express the complementary notion 
 
α is not a TNT -number. 
 
Now if we replaced all the occurrences  of a in this second string by the TNT -numeral for

Now if we replaced all the occurrences  of a in this second string by the TNT -numeral for 
123,666,111,666-a numeral which would contain exactly 123,666,111,666 S's, much too long to write out-we would have a TNT -string which, just like MUMON , is capable of 
being interpreted on two levels. In the first place, that string would say  
123,666,111,666 is not a TNT -number. 
 But because of the isomorphism which links TNT -numbers to theorems of TNT , there 
would be a second-level meaning of this string, which is:  
S0=0  is not a theorem of TNT.  
 
TNT Tries to Swallow Itself 
 
This unexpected double-entendre demonstrates that TNT  contains strings which talk

This unexpected double-entendre demonstrates that TNT  contains strings which talk 
about other strings of TNT . In other words, the metalanguage in which we, on the 
outside, can speak about TNT , is at least partially imitated inside  TNT  itself. And this is 
not an accidental feature of TNT ; it happens because the architecture of any formal 
system can be mirrored inside N (number theory). It is just  as inevitable a feature of TNT  
as are the vibrations induced in a record pl ayer when it plays a record. It seems as if 
vibrations should come from the outside wo rld-for instance, from jumping children or

vibrations should come from the outside wo rld-for instance, from jumping children or 
bouncing balls; but a side effect of produci ng sounds-and an unavoidable one-is that they 
wrap around and shake the very mechanism whic h produces them. It is no accident; it is a 
side effect which cannot be helped. It is in th e nature of record players. And it is in the 
nature of any formalization of number theory that its metalanguage is embedded within it.

 279We can dignify this observation by calling it the Central Dogma of MIathematical 
Logic , and depicting it in a two-step diagram: 
 
TNT   => N  => meta- 'TNT  
 In words: a string of TNT  has an interpretation in N; and a statement o may have a 
second meaning as a statement about TNT . 
 
G: A String Which Talks about Itself in Code 
 This much is intriguing yet it is only half the story. The rest of the st involves an 
intensification of the self-reference. We are now  at the st where the Tortoise was when he 
realized that a record could be m; whic h would make the phonograph playing it break-but 
now the quest is: "Given a record player, how do you actually figure out what to put the 
record?" That is a tricky matter. 
We want to find a string of TNT-which we'll call 'G'-which is ab itself, in the 
sense that one of its passive meanings is a sentence about G. particular the passive 
meaning will turn out to be 
 
"G is not a theorem of TNT ."

"G is not a theorem of TNT ." 
 I should quickly add that G also has a passive meaning which is a sta tement of number 
theory;  just like MUMON  it is susceptible to being construed in least) two different 
ways. The important thing is that each passive  mean is valid and useful and doesn't cast 
doubt on the other passive meaning in any way. (The fact that a phonograph playing a 
record can induce vibrations in itself and in the record does not diminish in any way the 
fact t those vibrations are musical sounds!) 
 
G's Existence Is What Causes TNT's Incompleteness

G's Existence Is What Causes TNT's Incompleteness 
 
The ingenious method of creating G, and some important concepts relating to TNT , will 
be developed in Chapters XIII and XIV; for now it is interesting to glance ahead, a bit 
superficially, at the consequences  finding a self-referential piece of TNT . Who knows? It 
might blow up! In a sense it does. We focus down on the obvious question:  
Is G a theorem of TNT , or not? 
 Let us be sure to form our own opinion on this matter, rather than rely G's opinion about 
itself. After all, G may not understand itsel f any be than a Zen master understands

itself. After all, G may not understand itsel f any be than a Zen master understands 
himself. Like MUMON , G may express a falsity. Like MU, G may be a nontheorem. We 
don't need to believe every possible string of TNT -only its theorems. Now let us use our 
power of reasoning to clarify the issue as best we can at this point. 
We will make our usual assumption: that TNT  incorporates valid

 280methods of reasoning, and therefore that TNT  never has falsities for theorems. In other 
words, anything which is a theorem of TNT  expresses a truth. So if G were a theorem, it 
would express a truth, namely: " G is not a theorem". The full force of its self-reference 
hits us. By being a theorem, G would have to be a falsity. Relying on our assumption that 
TNT  never has falsities for theorems, we'd be forced to conclude that G is not a theorem . 
This is all right; it leaves us, however , with a lesser problem. Knowing that G is not a 
theorem, we'd have to concede that G expresses a truth. Here is a situation in which TNT  
doesn't live up to our expectations-we ha ve found a string which expresses a true 
statement yet the string is not a theorem. A nd in our amazement, we shouldn't lose track 
of the fact that G has an arithmetical interpretation, too-which allows us to summarize 
our findings this way:

our findings this way:  
A string of TNT  has been found; it expresses, unambiguously, a statement about 
certain arithmetical properties of natura l numbers; moreover, by reasoning outside 
the system we can determine not only that th e statement is a true one, but also that 
the string fails to be a theorem of TNT . And thus, if we ask TNT  whether the 
statement is true, TNT  says neither yes nor no. 
 
Is the Tortoise's string in the Mu Offering  the analogue of G? Not quite. The

Is the Tortoise's string in the Mu Offering  the analogue of G? Not quite. The 
analogue of the Tortoise's string is ~G. Why is this so? Well, let us think a moment about 
what -G says. It must say the opposite of what G says. G says, " G is not a theorem of 
TNT ", so ~G must say " G is a theorem". We could rephrase both G and ~G this way: 
 G: "I am not a theorem (of TNT )." 
~G: "My negation is a theorem (of TNT )." 
 
It is ~G which is parallel to the Tortoise's string, for that string spoke not about itself, but

It is ~G which is parallel to the Tortoise's string, for that string spoke not about itself, but 
about the string which the Tortoise first pro ffered to Achilles -- which had an extra knot 
on it (or one too few, however you want to look at it).  
Mumon Has the Last Word 
 
Mumon penetrated into the Mystery of th e Undecidable anyone, in his concise poem on 
Joshu's MU:  
Has a dog Buddha-nature? This is the most serious question of all. If you say yes or no,  You lose your own Buddha-nature.

Prelude 281Prelude . . 
 
Achilles and the Tortoise have come to th e residence of their friend the Crab, to 
make the acquaintance of one of his fr iends, the Anteater. The introductions 
having been made, the four of them settle down to tea. 
 
Tortoise We have brought along a little someth ing for you, Mr. Crab. Crab: That's most 
kind of you. But you shouldn't have. 
Tortoise: Just a token of our esteem. Achilles, would you like to give it to Mr. C? Achilles: Surely. Best wishes, Mr. Crab. I hope you enjoy it.  
(Achilles hands the Crab an elegantly wrappe d present, square and very thin. The 
Crab begins unwrapping it.) 
 
Anteater: I wonder what it could be. 
Crab: We'll soon find out. (Completes the unwrapping, and pulls out the gif)t  Two 
records! How exciting! But there's no label. Uh-oh-is this another of your "specials", 
Mr. T? 
Tortoise: If you mean a phonograph-breaker, not this time. But it is in fact a custom-

Mr. T? 
recorded item, the only one of its kind in th e entire world. In fact, it's never even been 
heard before-except, of course, when Bach played it. 
Crab: When Bach played it? What do you mean, exactly? Achilles: Oh, you are going to be fabulously  excited, Mr. Crab, when Mr. T tells you 
what these records in fact are. 
Tortoise: Oh, you go ahead and tell him, Achilles. Achilles: May I? Oh, boy! I'd better consult my notes, then. (P ulls out a small filing card, 
and clears his voice .) Ahem. Would you be interested in hearing about the remarkable

and clears his voice .) Ahem. Would you be interested in hearing about the remarkable 
new result in mathematics, to which your records owe their existence? 
Crab: My records derive from some piece of  mathematics? How curious Well, now that 
you've provoked my interest, I must hear about it. 
Achilles: Very well, then. ( Pauses for a moment to sip his tea, then resumes) Have you 
heard of Fermat's infamous "Last Theorem"? 
Anteater: I'm not sure ... It sounds strangely familiar, and yet I can't qui place it. Achilles: It's a very simple idea. Pierre de  Fermat, a lawyer by vocation b mathematician

Anteater: I'm not sure ... It sounds strangely familiar, and yet I can't qui place it. Achilles: It's a very simple idea. Pierre de  Fermat, a lawyer by vocation b mathematician 
by avocation, had been reading in  his copy of the class text Arithmetica  by 
Diophantus, and came across a page containing the equation 
 
a2+b2=c2

Prelude 282
 
FIGURE 54. Mobius Strip II, by M. C. Escher (woodcut, 1963).

Prelude 283He immediately realized that this equation ha s infinitely many solutions a, b, c, and then 
wrote in the margin the following notorious comment: The equation 
 
an +bn=cn 
 
has solutions in positive integers a, b, c, and n only when n = 2 (an then there are 
infinitely many triplets a, b, c which satisfy the equation); but there are no 
solutions for n > 2. I have discovered a truly marvelous proof of this statement, which, unfortunately, this margin is too small to contain. 
 Ever since that day, some three hundred years ago, mathematicians have been vainly 
trying to do one of two things: either to I Fermat's claim, and thereby vindicate Fermat's reputation, whit though very high, ha s been somewhat tarnished by skeptics 
who he never really found the proof he claime d to have found-or e: refute the claim, 
by finding a counterexample: a set of four in tegers a, b, c, and n, with n > 2, which

Ever since that day, some three hundred years ago, mathematicians have been vainly 
by finding a counterexample: a set of four in tegers a, b, c, and n, with n > 2, which 
satisfy the equation. Until recently, every attempt in either direction had met with failure. 1 sure, the Theorem has been proven for many specific values of i particular, all n up to 125,000. 
Anteater: Shouldn't it be called a "Conjecture" rather than a "Theorem it's never been 
given a proper proof? 
Achilles: Strictly speaking, you're right, but tradition has kept it this i Crab: Has someone at last managed to resolv e this celebrated quest Achilles: Indeed! In 
fact, Mr. Tortoise has done so, and as usua l, by a wizardly stroke. He has not only

fact, Mr. Tortoise has done so, and as usua l, by a wizardly stroke. He has not only 
found a PROOF of Fermat's Theorem (thus justifying its name as well as vindicating Fermat; also a COUNTEREXAMPLE, thus  showing that the skeptics had good 
intuition! 
Crab: Oh my gracious! That is a revolutionary discovery. Anteater: But please don't leave us in suspense . What magical integer they, that satisfy 
Fermat's equation? I'm especially curious about the value of n. 
Achilles: Oh, horrors! I'm most embarrassed! Can you believe this? the values at home on 
a truly colossal piece of paper. Unfortunate ly was too huge to bring along. I wish I 
had them here to show to y( it's of any help to you, I do remember one thing-the value

had them here to show to y( it's of any help to you, I do remember one thing-the value 
of n only positive integer which does not occu r anywhere in the continued fraction for 
π 
Crab: Oh, what a shame that you don't have them here. But there reason to doubt what 
you have told us.

Prelude 284
                           FIGURE 55. Pierre de Fermat. 
 
Anteater: Anyway, who needs to see n written out decimally? Achilles has just told us 
how to find it. Well, Mr. T, please accept my  hearty felicitations, on the occasion of 
your epoch-making discovery! 
Tortoise: Thank you. But what I feel is more  important than the result itself is the 
practical use to which my result immediately led. 
Crab: I am dying to hear about it, since I al ways thought number theory was the Queen of 
Mathematics -- the purest branch of mathem atic -- the one branch of mathematics 
which has No applications! 
Tortoise: You're not the only one with that be lief, but in fact it is quite impossible to 
make a blanket statement about when or how some branch-or even some individual 
Theorem-of pure mathematics will have important repercussions outside of mathematics. It is quite unpredictable-and this case is a perfect example of that 
phenomenon.

phenomenon. 
Achilles: Mr. Tortoise's double-barreled result has created a breakthrough in the field of 
acoustico-retrieval! 
Anteater: What is acoustico-retrieval? Achilles: The name tells it all: it is the retr ieval of acoustic information from extremely 
complex sources. A typical task of acoustico-retrieval is to reconstruct the sound which a rock made on plummeting into a lake  from the ripples which spread out over 
the lake's surface.

the lake's surface. 
Crab: Why, that sounds next to impossible! Achilles: Not so. It is actually quite similar to what one's brain does, when it reconstructs 
the sound made in the vocal cords of another person from the vibrations transmitted by the eardrum to the fibers in the cochlea. 
Crab: I see. But I still don't see where number theory enters the picture, or what this all 
has to do with my new records.

Prelude 285Achilles: Well, in the mathematics of acoustico-retrieval, there arise rr questions which 
have to do with the number of solutions of cer Diophantine equations. Now Mr. T has 
been for years trying to fit way of rec onstructing the sounds of Bach playing his 
harpsichord, which took place over two hundred  years ago, from calculations in% ing 
the motions of all the molecules in the atmosphere at the pre time. 
Anteater: Surely that is impossible! They are irretrievably gone, g forever! Achilles: Thus think the nave ... But Mr. T has devoted many year this problem, and 
came to the realization that the whole thi ng hinged on the number of solutions to the 
equation 
 
an +bn=cn 
 
in positive integers, with n > 2. 
Tortoise: I could explain, of course, just how  this equation arises, but I’m sure it would 
bore you. 
Achilles: It turned out that acoustico-retrieva l theory predicts that Bach sounds can be

equation 
 
an +bn=cn 
 
bore you. 
Achilles: It turned out that acoustico-retrieva l theory predicts that Bach sounds can be 
retrieved from the motion of all the molecule the atmosphere, provided that EITHER there exists at least one solution to the equation 
Crab: Amazing! Anteater: Fantastic! Tortoise: Who would have thought! Achilles: I was about to say, "provided that there exists EITHER such a solution OR a 
proof that there are tic) solutions!" And ther efore, Mr. T, in careful fashion, set about 
working at both ends of the problem, simulta neously. As it turns out, the discovery of

working at both ends of the problem, simulta neously. As it turns out, the discovery of 
the counterexample was the key ingredient to  finding the proof, so the one led directly 
to the other. 
Crab: How could that be? Tortoise: Well, you see, I had shown that the st ructural layout of any pr of Fermat's Last 
Theorem-if one existed-could be describe d by elegant formula, which, it so happened, 
depended on the values ( solution to a certain equation. When I found this second equation my surprise it turned out to be the Fermat equation. An amusing accidental

depended on the values ( solution to a certain equation. When I found this second equation my surprise it turned out to be the Fermat equation. An amusing accidental 
relationship between form and content. So  when I found the counterexample, all I 
needed to do was to use those numbers blueprint for constructing my proof that there were no solutions to equation. Remarkably simple, when you think about it. I can't imagine why no one had ever found the result before. 
Achilles: As a result of this unanticipatedly ri ch mathematical success,  Mr. T was able to 
carry out the acoustico-retrieval which he  had long dreamed of. And Mr. Crab's

carry out the acoustico-retrieval which he  had long dreamed of. And Mr. Crab's 
present here represents a palpable realization of all this abstract work.

Prelude 286Crab: Don't tell me it's a recording of Bach playing his own works for harpsichord! 
Achilles: I'm sorry, but I have to, for that is indeed just what it is! This is a set of two 
records of Johann Sebastian Bach playing all of his Well Tempered Clavier . Each 
record contains one of the two volumes of the Well-Tempered Clavier ; that is to say, 
each record contains 24 preludes and fugues-one in each major and minor key. 
Crab: Well, we must absolutely put one of these priceless records on, immediately! And 
how can I ever thank the two of you? 
Tortoise: You have already thanked us plentif ully, with this delicious tea which you have 
prepared. 
 
(The Crab slides one of the records out of its jacket, and puts it on. The sound of 
an incredibly masterful harpsichordist fills the room, in the highest imaginable 
fidelity. One even hears-or is it one's im agination?-the soft sounds of Bach singing 
to himself as he plays ...)

prepared. 
 
to himself as he plays ...) 
 
Crab: Would any of you like to follow along in  the score? I happen to have a unique 
edition of the Well-Tempered Clavier , specially illuminated by a teacher of mine who 
happens also to be an unusually fine callig rapher. Tortoise: I would very much enjoy 
that. 
 
(The Crab goes to his elegant glass-encl osed wooden bookcase, opens the doors, and 
draws out two large volumes.) 
 
Crab: Here you are, Mr. Tortoise. I've neve r really gotten to know all the beautiful 
illustrations in this edition. Perhaps your gift will provide the needed impetus for me to do so.

that. 
 
illustrations in this edition. Perhaps your gift will provide the needed impetus for me to do so. 
Tortoise: I do hope so. 
Anteater: Have you ever noticed how in thes e pieces the prelude always sets the mood 
perfectly for the following fugue? 
Crab: Yes. Although it may be hard to put it in to words, there is always some subtle 
relation between the two. Even if the prelude and fugue do not have a common 
melodic subject, there is nevertheless always some intangible abstract quality which underlies both of them, binding them together very strongly.

melodic subject, there is nevertheless always some intangible abstract quality which underlies both of them, binding them together very strongly. 
Tortoise: And there is something very dramatic about the few moments of silent suspense 
hanging between prelude and fugue-that moment  where the the theme of the fugue is 
about to ring out, in single tones, and then to join with itself in ever-increasingly 
complex levels of weird, exquisite harmony. 
Achilles: I know just what you mean. Ther e are so many preludes and fugues which I 
haven't yet gotten to know, and for me that  fleeting interlude of silence is very 
exciting; it's a time when I try to sec ond-guess old Bach. For example, I always

exciting; it's a time when I try to sec ond-guess old Bach. For example, I always 
wonder what the fugue's tempo will be: allegr o, or adagio? Will it be in 6/8, or 4/4? 
Will it have three voices, or five-or four? A nd then, the first voice starts ... Such an 
exquisite moment.

Prelude 287Crab: Ah, yes, well do I remember those long-gone days of my youth, days when I 
thrilled to each new prelude and fugue, fille d with excitement of their novelty and 
beauty and the many unexpected' surprises which they conceal. 
Achilles: And now? Is that thrill all gone? Crab: It's been supplanted by familiarity, as thrills always will be. But that familiarity 
there is also a kind of depth, which has its own compensations. For instance, I find that there are always new surprises whit hadn't noticed before. 
Achilles: Occurrences of the theme which you had overlooked? Crab: Perhaps-especially when it is invert ed and hidden among several other voices, or 
where it seems to come rushing up from the dept out of nowhere. But there are also 
amazing modulations which ii marvelous to listen to over and over again, and wonder 
how old B2 dreamt them up. 
Achilles: I am very glad to hear that there is something to look forward after I have been

Achilles: I am very glad to hear that there is something to look forward after I have been 
through the first flush of infatuation with the Well Tempered Clavier -although it also 
makes me sad that this stage cot not last forever and ever. 
Crab: Oh, you needn't fear that your infatuati on will totally die. One the nice things about 
that sort of youthful thrill is that it can  always resuscitated, just when you thought it 
was finally dead. It just takes the ri ght kind of triggering from the outside. 
Achilles: Oh, really? Such as what? Crab: Such as hearing it through the ears, so to speak, of someone whom it is a totally

Achilles: Oh, really? Such as what? Crab: Such as hearing it through the ears, so to speak, of someone whom it is a totally 
new experience-someone such as you, Achilles. Somehow the excitement transmits itself, and I can feel thrilled again. 
Achilles: That is intriguing. The thrill has remained dormant somewhere inside you, but 
by yourself, you aren't able to fish it up out of your subconscious. 
Crab: Exactly. The potential of reliving the thrill is "coded", in sot unknown way, in the 
structure of my brain, but I don't have th e power to summon it up at will; I have to 
wait for chance circumstance trigger it.

wait for chance circumstance trigger it. 
Achilles: I have a question about fugues which I feel a little embarrass about asking, but 
as I am just a novice at fugue-listeni ng, I was wondering if perhaps one of you 
seasoned fugue-listeners might help me learning .. . 
Tortoise: I'd certainly like to offer my own meager knowledge, if it might prove of' some 
assistance. 
Achilles: Oh, thank you. Let me come at th e question from an angle. Are you familiar 
with the print called Cube with Magic Ribbons , by M. Escher?

with the print called Cube with Magic Ribbons , by M. Escher? 
Tortoise: In which there are circular bands  having bubble-like distortions which, as soon 
as you've decided that they are bumps, seem to turn it dents-and vice versa?

Prelude 288
 
FIGURE 56. Cube with Magic Ribbons, by M. C. Escher (lithograph, 1957 ). 
 Achilles: Exactly. Crab: I remember that picture. Those little bubbl es always seem to flip back and forth 
between being concave and convex, dependi ng on the direction that you approach 
them from. There's no way to see them simultaneously as concave AND convex-somehow one's brain doesn't allow that. Ther e are two mutually exclusive "modes" in 
which one can perceive the bubbles. 
Achilles: Just so. Well, I seem to have discovered two somewhat analogous modes in 
which I can listen to a fugue. The modes ar e these: either to follow one individual 
voice at a time, or to listen to the total effect  of all of them toge ther, without trying to 
disentangle one from another. I have tried out both of these modes, and, much to my 
frustration, each one of them shuts out the othe r. It's simply not in my power to follow

frustration, each one of them shuts out the othe r. It's simply not in my power to follow 
the paths of individual voices and at the same time to hear the whole effect. I find that I flip back and forth between one mode and the other, more or less spontaneously and 
involuntarily.

Prelude 289Anteater: Just as when you look at the magic bands, eh? 
Achilles: Yes. I was just wondering ... does my description of they modes of fugue-
listening brand me unmistakably as a naive, inexperienced listener, who couldn't even 
begin to grasp the deeper mo, perception which exist beyond his ken? 
Tortoise: No, not at all, Achilles. I can onl y speak for myself, but I to myself shifting 
back and forth from one mode to the ot her without exerting any conscious control 
over which mode should he dominant don't know  if our other companions here have 
also experience( thing similar. 
Crab: Most definitely. It's quite a tantaliz ing phenomenon, since you feel that the essence 
of the fugue is flitting about you, and you can't  grasp all of it, because you can't quite 
make yourself function ways at once. 
Anteater: Fugues have that interesting propert y, that each of their voices is a piece of 
music in itself; and thus a fugue might be  thought o collection of several distinct

music in itself; and thus a fugue might be  thought o collection of several distinct 
pieces of music, all based on one theme, a nd all played simultaneously. And it is up to 
the listener subconscious) to decide whether it should be perceived as a unit, c collection of independent parts, all of which harmonize. 
Achilles: You say that the parts are "independe nt", yet that can't be literally true. There 
has to be some coordination between them, otherwise when they were put together 
one would just have an unsystematic clashing of  tones-and that is as far from the truth 
as could b,

as could b, 
Anteater: A better way to state it might be  this: if you listened to each on its own, you 
would find that it seemed to make sense a ll by its could stand alone, and that is the 
sense in which I meant that it is independe nt. But you are quite right in pointing out 
that each of individually meaningful lines fuses with the others in a highly nonrandom 
way, to make a graceful totality. The art of writing a beautiful fugue lies precisely in

as could b, 
way, to make a graceful totality. The art of writing a beautiful fugue lies precisely in 
this ability, to manufacture several diff lines , each one of which gives the illusion of 
having been written I own beauty, and yet whic h when taken together form a whole, , 
does not feel forced in any way. Now, this dichotomy between he a fugue as a whole, 
and hearing its component voices, is a part: example of a very general dichotomy, which applies to many kit structures built up from lower levels. 
Achilles: Oh, really? You mean that my tw o "modes" may have some general type of 
applicability, in situations other than fugue-listening?

applicability, in situations other than fugue-listening? 
Anteater: Absolutely. Achilles: I wonder how that could be. I gue ss it has to do with alternating between 
perceiving something as a whole, and percei ving it as a collection of parts. But the 
only place I have ever run into that dichotomy is in listening to fugues. 
Tortoise: Oh, my, look at this! I just turned  the page while following the music, and came 
across this magnificent illustration facing the page of the fugue.

Prelude 290Crab: I have never seen that illustration before. Why don't you pass it 'round? 
 
(The Tortoise passes the book around. Each  of the foursome looks at it in a 
characteristic way-this one from afar, that one from close up, everyone tipping his head this way and that in puzzlement. Fi nally it has made the rounds, and returns 
to the Tortoise, who peers at it rather intently.) 
 
Achilles: Well, I guess the prelude is just about  over. I wonder if, as I listen to this fugue, 
I will gain any more insight into the ques tion, "What is the right way to listen to a 
fugue: as a whole, or as the sum of its parts?"  
TTortoise: Listen carefully, and you will!  
(The prelude ends. There is a moment of silence; and ... 
 
[ATTACCA]

Levels of Description, and Computer Systems 
 291CHAPTER X 
 
Levels of Description, 
and Computer Systems 
 
Levels of Description 
  GODEL'S STRING G, and a Bach fugue: they bot h have the property that they can be 
understood on different levels. We are all familiar with this kind of thing; and yet in some 
cases it confuses us, while in others w handle it without any difficulty at all. For example, 
we all know that w human beings are compos ed of an enormous number of cells (around 
twenty-five trillion), and therefore that everyt hing we do could in principle be described 
in terms of cells. Or it could even be desc ribed on the level c molecules. Most of us 
accept this in a rather matter-of-fact way; we  go t the doctor, who looks at us on lower 
levels than we think of ourselves. W read about DNA and "genetic engineering" and sip 
our coffee. We seem t have reconciled thes e two inconceivably different pictures of

our coffee. We seem t have reconciled thes e two inconceivably different pictures of 
ourselves simply by disconnecting them from each  other. We have almost no way t relate 
a microscopic description of ourselves to that  which we feel ourselves to be, and hence it 
is possible to store separate representations of  ourselves in quite separate "compartments" 
of our minds. Seldom do we have to fir back  and forth between these two concepts of 
ourselves, wondering "How can these two totally different things be the same me?" 
Or take a sequence of images on a television screen which show Shirley

Or take a sequence of images on a television screen which show Shirley 
MacLaine laughing. When we watch that sequence, we know that we are actually looking 
not at a woman, but at sets of flickering dot s on a flat surface. We know it, but it is the 
furthest thing from our mind. We have these two wildly opposing representations of what 
is on the screen, but that does not confuse us. We can just shut one out, and pay attention 
to th other-which is what all of us do. Whic h one is "more real"? It depends o; whether 
you're a human, a dog, a computer, or a television set. 
 
Chunking and Chess Skill

Chunking and Chess Skill 
 One of the major problems of Artificial Inte lligence research is to figure out how to 
bridge the gap between these two descripti ons; how to construe a system which can 
accept one level of description, and produce the other One way in which this gap enters 
Artificial Intelligence is well illustrated b the progress in knowledge about how to 
program a computer to play goof chess. It us ed to be thought in the 1950's and on into the 
1960's-that the

 292trick to making a machine play well was to ma ke the machine look further ahead into the 
branching network of possible sequences of pl ay than any chess master can. However, as 
this goal gradually became attained, the level of computer chess did not have any sudden 
spurt, and surpass human experts. In f act, a human expert can quite soundly and 
confidently trounce the best chess programs of this day. 
The reason for this had actually been in print for many years. In the 1940's, the 
Dutch psychologist Adriaan de Groot made  studies of how chess novices and chess 
masters perceive a chess situation. Put in their starkest terms, his results imply that chess 
masters perceive the distribution of pieces in chunks . There is a higher-level description 
of the board than the straightforward "wh ite pawn on K5, black rook on Q6" type of 
description, and the master somehow produces such a mental image of the board. This

description, and the master somehow produces such a mental image of the board. This 
was proven by the high speed with which a master could reproduce an actual position taken from a game, compared with the novice's plodding reconstruction of the position, 
after both of them had had five-second glances at the board. Highly revealing was the fact 
that masters' mistakes involved placing whole groups  of pieces in the wrong place, which 
left the game strategically almost the same, but to a novice's eyes, not at all the same. The 
clincher was to do the same experiment but with pieces randomly assigned to the squares

clincher was to do the same experiment but with pieces randomly assigned to the squares 
on the board, instead of copied from actual games. The masters were found to be simply 
no better than the novices in reconstructing such random boards. 
The conclusion is that in normal chess play, certain types of situation recur-
certain patterns-and it is to those high-level patterns that the master is sensitive. He thinks on a different level  from the novice; his set of concepts is different. Nearly everyone is 
surprised to find out that in actual play, a master rarely looks ahead any further than a 
novice does-and moreover, a master usually ex amines only a handful of possible moves!

novice does-and moreover, a master usually ex amines only a handful of possible moves! 
The trick is that his mode of perceivi ng the board is like a filter: he literally does not see 
bad moves  when he looks at a chess situation-no more than chess amateurs see illegal 
moves when they look at a chess situation. Anyone who has played even a little chess has 
organized his perception so that diagonal r ook-moves, forward captures by pawns, and so 
forth, are never brought to mind. Similarly, master-level players have built up higher

forth, are never brought to mind. Similarly, master-level players have built up higher 
levels of organization in the way they see the board; consequently, to them, bad moves are as unlikely to come to mind as illegal move s are, to most people. This might be called 
implicit pruning  of the giant branching tree of possibilities. By contrast, explicit pruning  
would involve thinking of a move, and afte r superficial examination, deciding not to 
pursue examining it any further. 
The distinction can apply just as well to other intellectual activities -- for instance,

The distinction can apply just as well to other intellectual activities -- for instance, 
doing mathematics. A gifted mathematician doe sn't usually think up and try out all sorts 
of false pathways to the desired theorem, as  less gifted people might do; rather, he just 
"smells" the promising paths, and takes them immediately. 
Computer chess programs which rely on looking ahead have not been taught to 
think on a higher level; the strategy has just been to use brute

 293force look-ahead, hoping to crush all type s of opposition. But it h worked. Perhaps 
someday, a look-ahead program with enough br ute ,gill indeed overcome the best human 
players-but that will be a intellectual gain, co mpared to the revelation that intelligence de 
crucially on the ability to create high-level desc riptions of complex such as chess boards, 
television screens, printed pages, or painting  
Similar Levels 
 usually, we are not required to hold more than  one level of understanding of a situation in 
our minds at once. Moreover, the different de scriptions a single system are usually so 
conceptually distant from each other tl was mentioned earlier, there is no problem in 
maintaining them both are just maintained in  separate mental compartments. What is 
confusing though, is when a single system admits  of two or more descriptions different 
levels which nevertheless resemble  each other in some way. we find it hard to avoid

levels which nevertheless resemble  each other in some way. we find it hard to avoid 
mixing levels when we think about the system can easily get totally lost. 
Undoubtedly this happens when we th ink about our psychology-for instance, 
when we try to understand people's motivations: for various actions. There are many levels in the human m structure-certainly it is a system which we do not understand very 
we But there are hundreds of rival theories  which tell why people act the way they do, 
each theory based on some underlying assumptions  about he down in this set of levels

each theory based on some underlying assumptions  about he down in this set of levels 
various kinds of psychological "forces" are f( Since at this time we use pretty much the same kind of language f mental levels, th is makes for much level-mixing and most 
certain] hundreds of wrong theories. For instance,  we talk of "drives"-for se power, for 
fame, for love, etc., etc.-without knowing wher e these drives come from in the human 
mental structure. Without belaboring the pc si mply wish to say that our confusion about

mental structure. Without belaboring the pc si mply wish to say that our confusion about 
who we are is certainly r( to the fact that we  consist of a large set of levels, and we use 
overlapping language to describe ourselves on all of those levels. 
 
Computer Systems 
 There is another place where many levels of description coexist for a system, and where 
all the levels are conceptually quite close to  one an( I am referring to computer systems. 
When a computer program is ping, it can be vi ewed on a number of levels. On each level, 
the description is given in the language of computer science, which makes all the de

the description is given in the language of computer science, which makes all the de 
descriptions similar in some ways to each ot her-yet there are extremely imp( differences 
between the views one gets on the different leve ls. At the 1 level, the description can be 
so complicated that it is like the dot-descr iption of a television picture. For some 
purposes, however, this is by far the important view. At the highest level, the description 
is greatly chunked  and

 294takes on a completely different feel, despite the fact that many of the same concepts 
appear on the lowest and highest levels. Th e chunks on the high-level description are like 
the chess expert's chunks, and like the chunke d description of the image on the screen: 
they summarize in capsule form a number of things which on lower levels are seen as 
separate. (See Fig. 57.) Now before things become too abstract, let us pass on to the  
 
FIGURE 57. The idea of "chunking": a group of items is reperceived as a single "chunk". 
The chunk's boundary is a little like a cell membrane or a national border: it establishes a separate identity for the cluster within. A ccording to context, one may wish to ignore 
the chunk's internal structure or to take it into account.  
concrete facts about computers, beginning with  a very quick skim of what a computer 
system is like on the lowest level. The lowest  level? Well, not really, for I am not going

system is like on the lowest level. The lowest  level? Well, not really, for I am not going 
to talk about elementary particles-but it is the lowest level which we wish to think about. 
At the conceptual rock-bottom of a computer, we find a memory, a central 
processing unit (CPU), and some input-output (I/O) devices. Let us first describe the 
memory. It is divided up into distinct phys ical pieces, called words. For the sake of 
concreteness, let us say there are 65,536 words of memory (a typical number, being 2 to

concreteness, let us say there are 65,536 words of memory (a typical number, being 2 to 
the 16th power). A word is further divided in to what we shall consider the atoms of 
computer science-bits. The number of bits in a typical word might be around thirty-six. Physically, a bit is just a magnetic "switch" that can be in either of two positions.  
 
--- a word of 36 bits --

 295you could call the two positions "up" and "down" , or "x" and "o", o and "0" ... The third 
is the usual convention. It is perfectly fine, but i the possibly misleading effect of making 
people think that a comp deep down, is storing numbers. This is not true. A set of thirty-
six bits not have to be thought of as a numbe r any more than two bits has i thought of as 
the price of an ice cream cone. Just as money can do va things depending on how you use 
it, so a word in memory can serve r functions. Sometimes, to be sure, those thirty-six bits 
will indeed repn a number in binary notation. Other times, they may represent thin dots 
on a television screen. And other times, they ma y represent a letters of text. How a word 
in memory is to be thought of depends eni on th e role that this word plays in the program 
which uses it. It ma course, play more than one role-like a note in a canon.  
Instructions and Data

Instructions and Data 
 There is one interpretation of a word whic h I haven't yet mentioned, that is as an 
instruction. The words of memory contain not  only data t acted on, but also the program 
to act on the data. There exists a lin repertoi re of operations which can be carried out by 
the central proce5 unit-the CPU-and part of a word, usually its first several bits-is it pretable as the name of the instruction-type wh ich is to be carried What do the rest of the 
bits in a word-interpreted-as-instruction sta nd Most often, they tell which other words in

bits in a word-interpreted-as-instruction sta nd Most often, they tell which other words in 
memory are to be acted upoi other words, the remaining bits constitute a pointer to some other wor( words) in memory. Every word in memory has a distinct location, li house on a street; and its location is called its address. Memory may have "street", or many "streets"-they are called "pages". So a given wo addressed by its page number (if memory 
is paged) together wit position within the page . Hence the "pointer" part of an instruction 
i numerical address of some word(s) in memo ry. There are no restric on the pointer, so an

i numerical address of some word(s) in memo ry. There are no restric on the pointer, so an 
instruction may even "point" at itself, so that  whet executed, it causes a change in itself to 
be made. How does the computer know what instruction to execute at any € time? This is kept 
track of in the CPU. The CPU has a special poi nter w points at (i.e., stores the address of) 
the next word which is to be inter ed as an  instruction. The CPU fetches that word from 
memory, and c it electronically into a special word belonging to the CPU itself. (Wor the 
CPU are usually not called "words", but rather, registers.) Then the executes that instruction. Now the instruction may call for a ny of a number of types of operations to be 
carried out. Typical ones include:  
ADD  the word pointed to in the instruction, to a register. 
(In this case, the word pointed to is obviously interpreted as number.)

 296PRINT  the word pointed to in the instruction, as letters. 
(In this case, the word is obviously interpreted not as a number, but as a 
string of letters.) 
JUMP  to the word pointed to in the instruction. 
(In this case, the CPU  is being told to interpret that particular word as its 
next instruction.) 
 Unless the instruction explicitly dictates otherwise, the CPU  will pick up 
the very next word and interpret it as  an instruction. In other words, the CPU  
assumes that it should move down the "street" sequentially, like a mailman, interpreting word after word as an inst ruction. But this sequential order can be 
broken by such instructions as the JUMP  instruction, and others. 
 
Machine Language vs. Assembly language 
 
This is a very brief sketch of machine language . In this language, the types of 
operations which exist constitute a finite repertoire which cannot be extended. Thus all programs, no matter how large and complex, must be made out of

operations which exist constitute a finite repertoire which cannot be extended. Thus all programs, no matter how large and complex, must be made out of 
compounds of these types of instruc tions. Looking at a program written in 
machine language is vaguely comparable to looking at a DNA  molecule atom by 
atom. If you glance back to Fig. 41, s howing the nucleotide sequence of a DNA  
molecule--and then if you consider that each nucleotide contains two dozen atoms or so-and if you imagine trying to write the DNA , atom by atom, for a small virus 
(not to mention a human being!)-then you w ill get a feeling for what it is like to

(not to mention a human being!)-then you w ill get a feeling for what it is like to 
write a complex program in machine language , and what it is like to try to grasp 
what is going on in a program if you ha ve access only to its machine language 
description. , 
It must be mentioned, however, that computer programming was 
originally done on an even lower level, if  possible, than that of machine language-
-namely, connecting wires to each other, so  that the proper operations were "hard-
wired" in. This is so amazingly primitive by modern standards that it is painful

wired" in. This is so amazingly primitive by modern standards that it is painful 
even to' imagine. Yet undoubtedly the peopl e who first did it experienced as much 
exhilaration as the pioneers of modern computers ever do .. . 
We now wish to move to a higher level of the hierarchy of levels of 
description of programs. This is the assembly language  level. There is not a

description of programs. This is the assembly language  level. There is not a 
gigantic spread between assembly language and machine language; indeed, the step is rather gentle. In essence, there is a one-to-one correspondence between assembly language instructions and machine language instructions. The idea of assembly language is to "chunk" the individual machine language instructions, so that instead of writing the sequence of bits "010111000" when you want an instruction which adds one numbe r to another, you simply write ADD , and then 
instead of giving the address in binary re presentation, you can refer to the word in

instead of giving the address in binary re presentation, you can refer to the word in 
memory by a name .

 297Therefore, a program in assembly language is very much like a machine language 
program made legible to humans. Y ou might compare the machine language 
version of a program to a TNT -derivation done in the obscure Gödel-numbered 
notation, and the assembly language version to the isomorphic TNT -derivation, 
done in the original TNT -notation, which is much easier to understand. Or, going 
back to the DNA  image, we can liken the difference between machine language 
and assembly language to the differe nce between painfully specifying each 
nucleotide, atom by atom, and specify ing a nucleotide by simply giving its name  
(i.e., 'A', 'G', 'C', or 'T' ). There is a tremendous saving of labor in this very 
simple "chunking" operation, although conceptually not much has been changed. 
 
Programs That Translate Programs 
 
Perhaps the central point about assembly  language is not its differences from

Perhaps the central point about assembly  language is not its differences from 
machine language, which are not that e normous, but just the key idea that 
programs could be written on a different level at all ! Just think about it: the 
hardware is built to "understand" machin e language programs-sequences of bits-
but not letters and decimal numbers. Wh at happens when hardware is fed a 
program in assembly language% It is as if  you tried to get a cell to accept a piece 
of paper with the nucleotide sequence written out in letters of the alphabet, instead 
of in chemicals. What can a cell do with  a piece of paper? What can a computer

of in chemicals. What can a cell do with  a piece of paper? What can a computer 
do with an assembly language program? 
And here is the vital point: someone can write, in machine language, a 
translation program . This program, called an assembler , accepts mnemonic 
instruction names, decimal numbers, and other convenient abbreviations which a 
programmer can remember easily, and carries out the conversion into the 
monotonous but critical bit-sequences. Afte r the assembly language program has 
been assembled  (i.e., translated), it is run-or rather, its machine language

been assembled  (i.e., translated), it is run-or rather, its machine language 
equivalent is run. But this is a matte r of terminology. Which level program is 
running? You can never go wrong if you sa y that the machine language program 
is running, for hardware is always involve d when any program runs-but it is also 
quite reasonable to think of the running program in terms of assembly language. For instance, you might very well say, "Right now, the CPU  is executing a JUMP  
instruction", instead of saying, "Right now, the CPU  is executing a ' 1 11010000'

instruction", instead of saying, "Right now, the CPU  is executing a ' 1 11010000' 
instruction". A pianist who plays the notes G-E-B E-G-B  is also playing an 
arpeggio in the chord of E minor. Ther e is no reason to be reluctant about 
describing things from a higher-level poi nt of view. So one can think of the 
assembly language program running concurrently with the machine language program. We have two modes of describing what the CPU  is doing.

 298Higher-Level Languages, Compilers, and Interpreters 
 
The next level of the hierarchy carries much further the extremely powerful idea 
of using the computer itself to translat e programs from a high level into lower 
levels. After people had programmed in assembly language for a number of years, in the early 1950's, they realized that there were a number of characteristic 
structures which kept reappearing in pr ogram after program. There seemed to be, 
just as in chess, certain fundamental  patterns which cropped up naturally when 
human beings tried to formulate algorithms --exact descriptions of processes they 
wanted carried out. In other words, al gorithms seemed to have certain higher-
level components, in terms of which they could be much more easily and 
esthetically specified than in the very restricted machine language, or assembly 
language. Typically, a high-level algorithm co mponent consists not of one or two

language. Typically, a high-level algorithm co mponent consists not of one or two 
machine language instructions, but of a w hole collection of them, not necessarily 
all contiguous in memory. Such a compone nt could be represented in a higher-
level language by a single item-a chunk. 
Aside from standard chunks-the ne wly discovered components out of 
which all algorithms can be built-people rea lized that almost all programs contain 
even larger chunks-superchunks, so to speak. These superchunks differ from program to program, depending on the kinds of high-level tasks the j program is

even larger chunks-superchunks, so to speak. These superchunks differ from program to program, depending on the kinds of high-level tasks the j program is 
supposed to carry out. We discussed superchunks in Chapter V, calling them by their usual names: "subroutines" and "pro cedures". It was clear that a most 
powerful addition to any programming langua ge would be the ability to define 
new higher-level entities in terms of previ ously known ones, and then to call them 
by name. This would build the chunking ope ration right into the language. Instead 
of there being a determinate repertoire of  instructions out of which all programs 
had to be explicitly assembled, the progr ammer could construct his own modules,

had to be explicitly assembled, the progr ammer could construct his own modules, 
each with its own name, each usable anywhere  inside the program, just as if it had 
been a built-in feature of the language. Of  course, there is no getting away from 
the fact that down below, on a machine language level, everything would still be composed of the same old machine language  instructions, but that would not be 
explicitly visible to the highlevel programmer; it would be implicit. 
The new languages based on these ideas were called compiler languages. 
One of the earliest and most elegant was called "Algol", for "Algorithmic

One of the earliest and most elegant was called "Algol", for "Algorithmic 
Language". Unlike the case with assembly language, there is no straightforward one-to-one correspondence between statements in Algol and machine language instructions. To be sure, there is still a type of mapping from Algol into machine 
language, but it is far more "scrambled" than that between assembly language and machine language. Roughly speaking, an Algol program is to its machine language translation as a word problem in an elementary algebra text is to the 
equation it translates into. (Actually, getti ng from a word problem to an equation

equation it translates into. (Actually, getti ng from a word problem to an equation 
is far more complex, but it gives some ink ling of the types of "unscrambling" that 
have to be carried out in translating fr om a high-level language to a lower-level 
Ian

 299guage.) In the mid-1950's, successful programs called compilers were written 
whose function was to carry out the tr anslation from compiler languages to 
machine language. 
Also, interpreters were invented. Like compilers, interpreters translate 
from high-level languages into machine langua ge, but instead of translating all the 
statements first and then executing the machine code, they read one line and' 
execute it immediately. This has the advant age that a user need not have written a 
complete program to use an interpreter.  He may invent his program line by line, 
and test it out as he goes along. Thus, an  interpreter is to a compiler as a 
simultaneous interpreter is to a translat or of a written speech. One of the most 
important and fascinating of all computer languages is LISP  (standing for "List 
Processing"), which was invented by John McCarthy around the time Algol was

Processing"), which was invented by John McCarthy around the time Algol was 
invented. Subsequently, LISP  has enjoyed great popularity with workers in 
Artificial Intelligence. 
There is one interesting difference be tween the way interpreters work and 
compilers work. A compiler takes input (a  finished Algol program, for instance) 
and produces output (a long sequence of m achine language instructions). At this 
point, the compiler has done its duty. The output is then given to the computer to run. By contrast, the interpreter is cons tantly running while the programmer types

point, the compiler has done its duty. The output is then given to the computer to run. By contrast, the interpreter is cons tantly running while the programmer types 
in one LISP  statement after another, and each one gets executed then' and there. 
But this doesn't mean that each statement gets first translated, then executed, for then an interpreter would be nothing but a line-by-line compiler. Instead, in an interpreter, the operations of reading a new line, "understanding" it, and executing 
it are intertwined: they occur simultaneously. 
Here is the idea, expanded a little more. Each time a new line of LISP  is

Here is the idea, expanded a little more. Each time a new line of LISP  is 
typed in, the interpreter tries to process it. This means that the interpreter jolts into 
action, and certain (machine language) instructions inside it get executed. 
Precisely which ones get executed depends on the LISP  statement itself, of 
course. There are many JUMP  instructions inside the interpreter, so that the new 
line of LISP  may cause control to move around in a complex way-forwards, 
backwards, then forwards again, etc.. Thus, each LISP  statement gets converted

backwards, then forwards again, etc.. Thus, each LISP  statement gets converted 
into a "pathway" inside the interpreter, and the act of following that pathway achieves the desired effect. 
Sometimes it is helpful to think of the LISP  statements as mere pieces of 
data which are fed sequentially to a constantly running machine language 
program (the LISP  interpreter). When you think of things this way, you get a 
different image of the relation between  a program written in a higher-level 
language and the machine which is executing it. 
 
Bootstrapping

Bootstrapping 
 
Of course a compiler, being itself a program, has to be written in some language. The first compilers were written in assembly language, rather than machine language, thus taking full advantage of the already ac-

 300omplished first step up from machine language. A summary of these rather tricky 
concepts is presented in Figure 58.  
 FIGURE 58. Assemblers and 
compilers are both translators into machine language. This is indicated by the solid lines. Moreover, since they are themselves programs, they are originally written in a language also. The wavy lines indicate that aa compiler can be written in assembly language, and an assembler in machine language. 
 
  Now as sophistication increased, people r ealized that a partially written compiler 
could be used to compile extensions of  itself. In other words, once i certain 
minimal core of a compiler had been wr itten, then that minimal compiler could 
translate bigger compilers into machine la nguage-which n turn could translate yet 
bigger compilers, until the final, full-blown :compiler had been compiled. This process is affectionately known as `bootst rapping"-for obvious reasons (at least if

Now as sophistication increased, people r ealized that a partially written compiler 
bigger compilers, until the final, full-blown :compiler had been compiled. This process is affectionately known as `bootst rapping"-for obvious reasons (at least if 
your native language is English it is obvious ). It is not so different from the 
attainment by a child of a critical level of fluency in his native language, from 
which point on his 'vocabulary and flue ncy can grow by leaps and bounds, since 
he can use language to acquire new language.  
Levels on Which to Describe Running Programs 
 
Compiler languages typically do not reflect the structure of the machines which 
will run programs written in them. This is one of their chief advantages over the

will run programs written in them. This is one of their chief advantages over the 
highly specialized assembly and machine languages. Of course, when a compiler language program is translated into m achine language, the resulting program is 
machine-dependent. Therefore one can describe a program which is being executed in a machine-independent way or  a machine-dependent way. It is like 
referring to a paragraph in a book by its s ubject matter (publisher-independent), or 
its page number and position on the page (publisher-dependent).

its page number and position on the page (publisher-dependent). 
As long as a program is running co rrectly, it hardly matters how you 
describe it or think of its functioning. It is when something goes wrong that

 301 
it is important to be able to think on diffe rent levels. If, for instance, the machine 
is instructed to divide by zero at some st age, it will come to a halt and let the user 
know of this problem, by telling where in  the program the questionable event 
occurred. However, the specification is often given on a lower level than that in which the programmer wrote the program. He re are three parallel descriptions of 
a program grinding to a halt:  Machine Language Level: 
"Execution of the program stopped in location 1110010101110111" 
 Assembly Language Level•: 
"Execution of the program stopped when  the DIV (divide) instruction was 
hit" 
 Compiler Language Level: 
"Execution of the program stopped during evaluation of the algebraic expression `(A + B)/Z' 
 One of the greatest problems for system s programmers (the people who write

hit" 
 One of the greatest problems for system s programmers (the people who write 
compilers, interpreters, assemblers, and other programs to be used by many people) is to figure out how to write error-detecting routines in such a way that the messages which they feed to the user whose program has a "bug" provide high-level, rather than low-level, descrip tions of the problem. It is an interesting 
reversal that when something goes wrong in a genetic "program" (e.g., a 
mutation), the "bug" is manifest only to people on a high level-namely on the 
phenotype level, not the genotype level. Actually, modern biology uses mutations

phenotype level, not the genotype level. Actually, modern biology uses mutations 
as one of its principal windows onto genetic  processes, because of their multilevel 
traceability. 
 
Microprogramming and Operating Systems 
 
In modern computer systems, there are se veral other levels of the hierarchy. For 
instance, some systems-often the so-called "microcomputers" come with machine language instructions which are even more rudimentary than the instruction to add 
a number in memory to a number in a register . It is up to the user to decide what 
kinds of ordinary machine-level instructi ons he would like to be able to program 
in; he "microprograms" these instructions in terms of the "micro-instructions" which are available. Then the "higher-level machine language" instructions which he has designed may be burned into th e circuitry and become hard-wired,

in; he "microprograms" these instructions in terms of the "micro-instructions" which are available. Then the "higher-level machine language" instructions which he has designed may be burned into th e circuitry and become hard-wired, 
although they need not be. Thus microprogramming allows the user to step a little below the conventional machine language leve l. One of the consequences is that a 
computer of one manufacturer can be ha rd-wired (via microprogramming) so as 
to have the same machine language instruction set as a computer of the same, or 
even another, manufacturer. The micropr ogrammed computer is said to be 
"emulating" the other computer. Then there is the level of the operating system , 
which fits between the

 302machine language program and whatever hi gher level the user is programming in. 
The operating system is itself a program wh ich has the functions of shielding the 
bare machine from access by users (thus protecting the system), and also of 
insulating the programmer from the many extremely intricate and messy problems of reading the program, calling a transl ator, running the translated program, 
directing the output to the proper channels at the proper time, and passing control to the next user. If there are several user s "talking" to the same CPU at once, then 
the operating system is the program that sh ifts attention from one to the other in 
some orderly fashion. The complexities of operating systems are formidable 
indeed, and I shall only hint at them by the following analogy. 
Consider the first telephone system . Alexander Graham Bell could phone 
his assistant in the next room: electronic transmission of a voice! Now that is like

his assistant in the next room: electronic transmission of a voice! Now that is like 
a bare computer minus operating system: electronic computation! Consider now a 
modern telephone system. You have a choice of other telephones to connect to. Not only that, but many different calls can be handled simultaneously. You can add a prefix and dial into different areas. You can call direct, through the operator, collect, by credit card, person-to -person, on a conference call. You can 
have a call rerouted or traced. You can ge t a busy signal. You can get a siren-like 
signal that says that the number you dialed  isn't "well-formed", or that you have

signal that says that the number you dialed  isn't "well-formed", or that you have 
taken too in long in dialing. You can inst all a local switchboard so that a group of 
phones are all locally connected--etc., etc.  The list is amazing, when you think of 
how much flexibility there is, particularly  in comparison to the erstwhile miracle 
of a "bare" telephone. Now sophisticated  operating systems carry out similar 
traffic-handling and level-switching operati ons with respect to users and their 
programs. It is virtually certain that ther e are somewhat parallel things which take

programs. It is virtually certain that ther e are somewhat parallel things which take 
place in the brain: handling of many stimu li at the same time; decisions of what 
should have priority over what and for how long; instantaneous "interrupts" 
caused by emergencies or other unexpected occurrences; and so on. 
 
Cushioning the User and Protecting the System 
 
The many levels in a complex computer system have the combined effect of "cushioning" the user, preventing him from  having to think about the many lower-

level goings-on which are most likely totally irrelevant to him anyway. A passenger in an airplane does not usually want  to be aware of the levels of fuel in 
the tanks, or the wind speeds, or how many chicken dinners are to be served, or the status of the rest of the air traffic around the destination-this is all left to 
employees on different levels of the air lines hierarchy, and the passenger simply 
gets from one place to another. Here ag ain, it is when something goes wrong-such 
as his baggage not arriving that the pa ssenger is made aware of the confusing

as his baggage not arriving that the pa ssenger is made aware of the confusing 
system of levels underneath him.

 303 
Are Computers Super-Flexible or Super-Rigid? 
 
One of the major goals of the drive to hi gher levels has always been to make as 
natural as possible the task of communica ting to the computer what you want it to 
do. Certainly, the high-level constructs in compiler languages are closer to the concepts which humans naturally think in, th an are lower-level constructs such as 
those in machine language. But in this drive towards ease of communication, one aspect of "naturalness" has been quite negl ected. That is the fact that interhuman 
communication is far less rigidly constrained than human-machine communication. For instance, we often produce meaningless sentence fragments 
as we search for the best way to expr ess something, we cough in the middle of 
sentences, we interrupt each other, we us e ambiguous descriptions and "improper"

sentences, we interrupt each other, we us e ambiguous descriptions and "improper" 
syntax, we coin phrases and distort meanings-but our message still gets through, mostly. With programming languages, it has ge nerally been the rule that there is a 
very strict syntax which has to be obe yed one hundred per cent of the time; there 
are no ambiguous words or constructions. In terestingly, the printed equivalent of 
coughing (i.e., a nonessential or irrele vant comment) is allowed, but only 
provided it is signaled in advance by a key word (e.g., COMMENT ), and then

provided it is signaled in advance by a key word (e.g., COMMENT ), and then 
terminated by another key word (e.g., a se micolon). This small gesture towards 
flexibility has its own little pitfall, ironically: if a semicolon (or whatever key word is used for terminating a comment) is  used inside a comment, the translating 
program will interpret that semicolon as  signaling the end of the comment, and 
havoc will ensue. If a procedure named INSIGHT  has been defined and then called seventeen times 
in the program, and the eighteenth time it is misspelled as INSIHGT , woe to the

in the program, and the eighteenth time it is misspelled as INSIHGT , woe to the 
programmer. The compiler will balk and print a rigidly unsympathetic error 
message, saying that it has never heard of INSIHGT . Often, when such an error 
is detected by a compiler, the compiler tries to continue, but because of its lack of 
insihgt, it has not understood what the pr ogrammer meant. In fact, it may very 
well suppose that something entirely differe nt was meant, and proceed under that 
erroneous assumption. Then a long series of  error messages will pepper the rest of

erroneous assumption. Then a long series of  error messages will pepper the rest of 
the program, because the compiler-not the programmer-got confused. Imagine the chaos that would result if a simultane ous English-Russian interpreter, upon 
hearing one phrase of French in the E nglish, began trying to interpret all the 
remaining English as French. Compilers ofte n get lost in such pathetic ways. C'est 
la vie. Perhaps this sounds condemnatory of computer s, but it is not meant to be. In some 
sense, things had to be that way. When  you stop to think what most people use

sense, things had to be that way. When  you stop to think what most people use 
computers for, you realize that it is to ca rry out very definite and precise tasks, 
which are too complex for people to do. If the computer is to be reliable, then it is 
necessary that it should understand, without  the slightest chance of ambiguity, 
what it is supposed to do. It is also necessary that it should do neither more nor less than it is explicitly instructed to do. If there is, in the cushion underneath the 
programmer, a program whose purpose is to "guess" what the programmer wants 
or

 304means, then it is quite conceivable that the programmer could try to communicate 
his task and be totally misunderstood. So  it is important that the high-level 
program, while comfortable for the human, still should be unambiguous and precise.  
Second-Guessing the Programmer 
 
Now it is possible to devise a pr ogramming language-and a program which 
translates it into the lower levels-which  allows some sorts of imprecision. One 
way of putting it would be to say that a translator for such a programming language tries to make sense of things wh ich are done "outside of the rules of the 
language". But if a language allows certain "transgressions", then transgressions of that type are no longer true transgre ssions, because they have been included 
inside the rules' If a programmer is aw are that he may make certain types of 
misspelling, then he may use this feat ure of the language deliberately, knowing

misspelling, then he may use this feat ure of the language deliberately, knowing 
that he is actually operating within th e rigid rules of the language, despite 
appearances. In other words, if the user is aware of all the flexibilities 
programmed into the translator for his convenience, then he knows the bounds which he cannot overstep, and therefore, to  him, the translator still appears rigid 
and inflexible, although it may allow him mu ch more freedom than early versions 
of the language, which did not incorpor ate "automatic compensation for human 
error".

error". 
With "rubbery" languages of that t ype, there would seem to be two 
alternatives: (1) the user is aware of the built-in flexibilities of the language and 
its translator; (2) the user is unaware of them. In the first case, the language is still 
usable for communicating programs pr ecisely, because the programmer can 
predict how the computer will interpret th e programs he writes in the language. In

error". 
predict how the computer will interpret th e programs he writes in the language. In 
the second case, the "cushion" has hidden features which may do things that are unpredictable (from the vantage point of  a user who doesn't know the inner 
workings of the translator). This may result in gross misinterpretations of 
programs, so such a language is unsuitable for purposes where computers are used mainly for their speed and reliability. 
Now there is actually a third alternativ e: (3) the user is aware of the built-

in flexibilities of the language and its translator, but there are so many of them and they interact with each other in su ch a complex way that he cannot tell how 
his programs will be interpreted. This ma y well apply to the person who wrote the 
translating program; he certainly knows its insides as well as anyone could-but he 
still may not be able to anticipate how it will react to a given type of unusual 
construction. 
One of the major areas of research in Artificial Intelligence today is called 
automatic programming, which is concerne d with the development of yet higher-

level languages-languages whose translators are sophisticated. in that they can do 
at least some of the following impressi ve things: generalize from examples, 
correct some misprints or grammatical errors,

 305 
try to make sense of ambiguous descriptions, try to second-guess the user by having a primitive user model, ask questions  when things are unclear, use English 
itself, etc. The hope is that one can walk the tightrope between reliability and flexibility.  
AI Advances Are Language Advances 
 
It is striking how tight the connection is  between progress in computer science 
(particularly Artificial Intelligence) a nd the development of new languages. A 
clear trend has emerged in the last decade:  the trend to consolidate new types of 
discoveries in new languages. One key for the understanding and creation of intelligence lies in the constant development and refinement of the languages in terms of which processes for symbol ma nipulation are describable. Today, there 
are probably three or four dozen expe rimental languages which have been 
developed exclusively for Artificial Intelligen ce research. It is important to realize

developed exclusively for Artificial Intelligen ce research. It is important to realize 
that any program which can be written in one of these languages is in principle 
programmable in lower-level languages, but  it would require a supreme effort for 
a human; and the resulting program would be so long that it would exceed the 
grasp of humans. It is not that each higher level extends the potential of the 
computer; the full potential of the computer  already exists in its machine language 
instruction set. It is that the new c oncepts in a high-level language suggest 
directions and perspectives by their very nature.

directions and perspectives by their very nature. 
The "space" of all possible programs is so huge that no one can have a 
sense of what is possible. Each highe r-level language is naturally suited for 
exploring certain regions of "program sp ace"; thus the programmer, by using that 
language, is channeled into those areas of  program space. He is not forced by the 
language into writing programs of any par ticular type, but the language makes it 
easy for him to do certain kinds of things . Proximity to a concept, and a gentle

easy for him to do certain kinds of things . Proximity to a concept, and a gentle 
shove, are often all that is needed for a major discovery-and that is the reason for 
the drive towards languages of ever higher levels. 
Programming in different 'languages is  like composing pieces in different 
keys, particularly if you work at the keyboard. If you have learned or written 
pieces in many keys, each key will have its own special emotional aura. Also, 
certain kinds of figurations "lie in th e hand" in one key but are awkward in 
another. So you are channeled by your choice of key. In some ways, even

another. So you are channeled by your choice of key. In some ways, even 
enharmonic keys, such as C-sharp and D-fl at, are quite distinct in feeling. This 
shows how a notational system can play a significant role in shaping the final 
product. 
A "stratified" picture of Al is shown in Figure 59, -with machine 
components such as transistors on the bottom, and "intelligent programs" on the top. The picture is taken from the book Artificial Intelligence  by Patrick Henry 
Winston, and it represents a vision of Al shared by nearly all Al workers. 
Although I agree with the idea that Al must be stratified in some such way, I do

product. 
Although I agree with the idea that Al must be stratified in some such way, I do 
not think that, with so few layers, intelligent programs

 306 
 FIGURE 59. To create intelligent 
programs, one needs to build up a series of levels of hardware and software, so that one is spared the agonT of seeing everything only on the lowest level. Descriptions of a single process on different levels will sound verb different from each other, only the top one being sufficiently chunked that it is comprehensible to us. [Adapted from P. H. Winston, Artificial Intelligence (Reading, Mass.: Addison-ifele'', 1977)] 
 
 can he reached. Between the machine language level and the level where rue intelligence will be reached, I am convinced  there will lie perhaps mother dozen 
(or even several dozen!) layers, each new layer building on and extending the flexibilities of the layer below. What they  will be like we can hardly dream of 
now ... 
  
The Paranoid and the Operating System 
 
The similarity of all levels in a computer  system can lead to some strange level-

can he reached. Between the machine language level and the level where rue intelligence will be reached, I am convinced  there will lie perhaps mother dozen 
now ... 
  
mixing experiences. I once watched a c ouple of friends-both computer novices-
playing with the program "PARRY" on a te rminal. PARRY s a. rather infamous 
program which simulates a paranoid in an extremely rudimentary way, by spitting 
out canned phrases in English chosen from a vide repertoire; its plausibility is due 
to its ability to tell which of its stock phrases might sound reasonable in response 
to English sentences typed to t by a human. 
At one point, the response time got ve ry sluggish-PARRY was taking very

At one point, the response time got ve ry sluggish-PARRY was taking very 
long to reply-and I explained to my friends  that this was probably because of the 
heavy load on the time-sharing system. I told them they could find out how many 
users were logged on, by typing a special  "control" character which would go 
directly to the operating system, and would )e unseen by PARRY. One of my 
friends pushed the control character. In a lash, some internal data about the 
operating system's status overwrote so me of PARRY's words on the screen. 
PARRY knew nothing of this: it is a pr ogram with "knowledge" only of horse

PARRY knew nothing of this: it is a pr ogram with "knowledge" only of horse 
racing and bookies-not operating systems and terminals and special control characters. But to my friends, both PARRY  and the operating system were just 
"the computer"-a mysterious, remote, amorphous entity that responded to them 
when they typed. And so it made perfect sense when one of them blithely typed, 
in 3nglish, "Why are you overtyping what's on the screen?" The idea that PARRY 
could know' nothing about the operating system it was running

 307under was not clear to my friends. The id ea that "you" know all about "yourself" 
is so familiar from interaction with people that it was natural to extend it to the computer-after all, it was intelligent e nough that it could "talk" to them in 
English! Their question was not unlike asking a person, "Why are you making so few red blood cells today?" People do not know about that level-the "operating 
system level"-of their bodies. 
The main cause of this level-confusion was that communication with all 
levels of the computer system was taking place on a single screen, on a single terminal. Although my friends' naiveté might seem rather extreme, even experienced computer people often make si milar errors when several levels of a 
complex system are all present at once on the same screen. They forget "who"

complex system are all present at once on the same screen. They forget "who" 
they are talking to, and type something which makes no sense at that level, although it would have made perfect sense on another level. It might seem desirable, therefore, to have the system itself sort out the levels-to interpret commands according to what "makes sense". Unfortunately, such interpretation would require the system to have a lot of common sense, as well as perfect knowledge of the programmer's overall intent-both of which would require more artificial intelligence than exists at the present time. 
 
The Border between Software and Hardware

The Border between Software and Hardware 
 
One can also be confused by the flexibility  of some levels and the rigidity of 
others. For instance, on some computers there are marvelous text-editing systems 
which allow pieces of text to be "pour ed" from one format into another, 
practically as liquids can be poured from one vessel into another. A thin page can 
turn into a wide page, or vice versa. With such power, you might expect that it 
would be equally trivial to change from one font to another-say from roman to 
italics. Yet there may be only a single font  available on the screen, so that such 
changes are impossible. Or it may be feas ible on the screen but not printable by 
the printer-or the other way around. After d ealing with computers for a long time, 
one gets spoiled, and thinks that ever ything should be programmable: no printer 
should be so rigid as to have only one charact er set, or even a finite repertoire of

should be so rigid as to have only one charact er set, or even a finite repertoire of 
them-typefaces should be user-specifiable ! But once that degree of flexibility has 
been attained, then one may be annoyed th at the printer cannot print in different 
colors of ink, or that it cannot accept paper of all shapes and sizes, or that it does 
not fix itself when it breaks ... 
The trouble is that somewhere, all this flexibility has to "bottom out", to 
use the phrase from Chapter V. There must  be a hardware level which underlies it

use the phrase from Chapter V. There must  be a hardware level which underlies it 
all, and which is inflexible. It may lie deeply hidden, and there may be so much 
flexibility on levels above it that few user s feel the hardware limitations-but it is 
inevitably there. 
What is this proverbial distinction be tween software and hardware? It is 
the distinction between programs and machines-between long complicated sequences of instructions, and the physical machines which carry

 308them out. I like to think of software as "anything which you could send over he 
telephone lines", and hardware as "anyt hing else". A piano is hardware, gut 
printed music is software. A telephone set is hardware, but a telephone lumber is 
software. `The distinction is a useful one, but not always so clear-cut. 
We humans also have "software" and "hardware" aspects, and the 
difference is second nature to us. We are used to the rigidity of our physiology: 
the fact that we cannot, at will, cure ourse lves of diseases, or ;row hair of any 
color-to mention just a couple of simple  examples. We an, however, "reprogram" 
our minds so that we operate in ne w conceptual frameworks. The amazing 
flexibility of our minds seems nearly irreconcilable with the notion that our brains must be made out of fixed-rule hard ware, which cannot be reprogrammed. We 
cannot make our neurons ire faster or sl ower, we cannot rewire our brains, we

cannot make our neurons ire faster or sl ower, we cannot rewire our brains, we 
cannot redesign the interior of a neuron, we cannot make anti choices about the 
hardware-and 'et, we can control how we think. 
But there are clearly aspects of t hought which are beyond our control. We 
cannot make ourselves smarter by an act of will; we cannot learn a new language 
as fast as we want; we cannot make ourse lves think faster than we lo; we cannot 
make ourselves think about several things  at once; and so on. This is a kind of

make ourselves think about several things  at once; and so on. This is a kind of 
primordial self-knowledge which is so obvious that it is lard to see it at all; it is like being conscious that the air is there.  We never really bother to think about 
what might cause these "defects" of our  minds: lamely, the organization of our 
brains. To suggest ways of reconciling the software of mind with the hardware of 
brain is a main goal of this book. 
 
Intermediate Levels and the Weather 
 
We have seen that in computer system s, there are a number of rather sharply 
defined strata, in terms of any one of  which the operation of a running program

defined strata, in terms of any one of  which the operation of a running program 
can be described. Thus there is not me rely a single low bevel and a single high 
level-there are all degrees of lowness and highness. s the existence of intermediate 
levels a general feature of systems which lave low and high levels? Consider, for example, the system whose 'hardware" is the earth's atmosphere (not very hard, 
but no matter), and whose "sof tware" is the weather. K eeping track of the motions 
of all of the molecules simultaneously  would be a very low-level way of 
"understanding" he weather, rather lik e looking at a huge, complicated program

"understanding" he weather, rather lik e looking at a huge, complicated program 
on the machine language level. Obviously it is way beyond human comprehension. 3ut we still have our ow n peculiarly human ways of looking at, 
and describing, weather phenomena. Our chunke d view of the weather is based >n 
very high-level phenomena, such as: ra in, fog, snow, hurricanes, cold fronts, 
seasons, pressures, trade winds, the jet stream, cumulo-nimbus clouds, thunderstorms, inversion layers, and so on. All of these phenomena involve 
astronomical numbers of molecules, some how behaving in concert o that large-

scale trends emerge. This is a little like looking at the weather n a compiler language.

 309Is there something analogous to looking at the weather in an intermediate-
level language, such as assembly language? For instance, are there very small local "mini-storms", something like the small whirlwinds which one occasionally 
sees, whipping up some dust in a swirling co lumn a few feet wide, at most? Is a 
local gust of wind an intermediate-level  chunk which plays a role in creating 
higher-level weather phenomena? Or is th ere just no practical way of combining 
knowledge of such kinds of phenomen a to create a more comprehensive 
explanation of the weather? 
Two other questions come to my mind. The first is: "Could it be that the 
weather phenomena which we perceive on our scale-a tornado, a drought-are just 
intermediate-level phenomena: parts of vaster, slower phenomena?" If so, then true high-level weather phenomena would be global, and their time scale would 
be geological. The Ice Age would be a high-level weather event. The second

be geological. The Ice Age would be a high-level weather event. The second 
question is: "Are there intermediate le vel weather phenomena which have so far 
escaped human perception, but which, if per ceived, could give greater insight into 
why the weather is as it is?" 
 
From Tornados to Quarks 
 
This last suggestion may sound fanciful, but it is not all that far-fetched. 
We need only look to the hardest of the hard sciences-physics-to find peculiar examples of systems which are explained in  terms of interacting "parts" which are

We need only look to the hardest of the hard sciences-physics-to find peculiar examples of systems which are explained in  terms of interacting "parts" which are 
themselves invisible. In physics, as in any other discipline, a system is a group of interacting parts. In most systems that  we know, the parts retain their identities 
during the interaction, so that we still see the parts inside the system. For 
example, when a team of football players assembles, the individual players retain 
their separateness-they do not melt into some composite entity, in which their 
individuality is lost. Still-and this is important-some processes are going on in

individuality is lost. Still-and this is important-some processes are going on in 
their brains which are evoked by the t eam-context, and which would not go on 
otherwise, so that in a minor way, the pl ayers change identity when they become 
part of the larger system, the team. This kind of system is called a nearly decomposable system (the term comes from H. A. Simon's article "The Architecture of Complexity"; see the Bibliography). Such a system consists of weakly interacting modules, each of wh ich maintains its own private identity

part of the larger system, the team. This kind of system is called a nearly decomposable system (the term comes from H. A. Simon's article "The Architecture of Complexity"; see the Bibliography). Such a system consists of weakly interacting modules, each of wh ich maintains its own private identity 
throughout the interaction but by becoming slightly different from how it is when outside of the system,, contributes to the cohesive behavior of the whole system. The systems studied in physics are usually of this type. For instance, an atom is seen as made of 'a nucleus whose positive charge captures a number of electrons in "orbits", or bound states. The bound electr ons are very much like free electrons, 
despite their being internal to a composite object. 
Some systems studied in physics o ffer a contrast to the relatively

Some systems studied in physics o ffer a contrast to the relatively 
straightforward atom. Such systems invol ve extremely strong interactions, as a 
result of which the parts are swallowed up into the larger system, and lose some 
or all of their individuality. An example of this is the nucleus of an atom, which is 
usually described as being "a collection of protons and

 310neutrons". But the forces which pull the component particles together are strong 
that the component particles do not survive to anything like their “free" form (the 
form they have when outside a nucleus). And in fact a nucleus acts in many ways 
as a single particle, rather than as a co llection of interacting particles. When a 
nucleus is split. protons and neutrons are ten released. but also other particles. such as pi-mesons and gamma rays, are commonly produced. Are all those different particles physically present side a nucleus before it is split, or are then 
just "sparks" which fly off ten the nucleus is split- It is perhaps not meaningful to try to give an answer to such a questi on. On the level of particle physics, the 
difference between storing the potential to make "sparks" and storing actual sub 
particles is not so clear. 
A nucleus is thus one systems whose "parts!, even though they are not

A nucleus is thus one systems whose "parts!, even though they are not 
visible while on the inside, can be pulled out and made risible. However, ere are 
more pathological cases. such as the prot on and neutron seen as stems themselves. 
Each of them has been hypothesized to be  constituted from a trio of "quarks"-
hypothetical particles which can be combined in twos or threes to make many known fundamental particles. However, the interaction between quarks is so strong that not only can they not he seen  [side the proton and neutron, but they 
cannot even be pulled out at all'. bus, although quarks help to give a theoretical understanding of certain properties of prot ons and neutrons, their own existence

cannot even be pulled out at all'. bus, although quarks help to give a theoretical understanding of certain properties of prot ons and neutrons, their own existence 
may perhaps ever be independently established. Here see have the antithesis of a nearly decomposable system"-it is a system which, if anything, is "nearly 
indecomposable", Yet what is curious is that a quark-based theory of rotors and neutrons (and other particles) has consid erable explanatory power. in that many 
experimental results concerning the particles which narks supposedly compose can be accounted for quite well, quantitatively. by using the "quark model". 
 
Superconductivity: A "Paradox" of Renormalization

Superconductivity: A "Paradox" of Renormalization 
 
In Chapter V we discussed how renormali zed particles emerge from their bare 
cores, by recursively compounded interactions with virtual particles. A renormalized particle can be seen either as this complex mathematical construct, 
or as the single lump which it is, physi cally. One of the strangest rid most 
dramatic consequences of this way of describing particles is the explanation it provides for the famous phenomenon of superconductivity  resistance-free flow of 
electrons in certain solids, at extremely low temperatures. 
It turns out that electrons in solids are renormalized by their interactions

It turns out that electrons in solids are renormalized by their interactions 
with strange quanta of vibration called phonons  (themselves renormalized as 
well!). These renormalized electrons are called polarons . Calculation shows that 
at very low temperatures, two oppositely spinning polarons sill begin to attract 
each other, and can actually become bound together in i certain way. Under the 
proper conditions. all the current-carrying polar

 311ons will get paired up, forming Cooper pain s. Ironically, this pairing comes about 
precisely because electrons-the hare core s of the paired polarons--repel each other 
electrically. In contrast to the electrons, each Cooper pair feels neither attracted to 
nor repelled by an other Cooper pair, and consequently it can slip freely through a metal as if the metal were a vacuum. If  you convert the mathematical description 
of such a metal from one whose prim itive units are polarons into one whose 
primitive units are Cooper pairs. you ge t a considerable- simplified set of 
equations. This mathematical simplicity is the physicist's way of knowing that 
"chunking" into Cooper pairs is the natural way to look at superconductivity. 
Here we have several levels of par ticle: the Cooper pair itself: the two 
oppositely-spinning polarons which compos e it: the electrons and phonons which 
make up the polarons: and then, within the electrons, the virtual photons and

make up the polarons: and then, within the electrons, the virtual photons and 
positrons, etc. etc. We can look at each  level and perceive phenomena there, 
which are explained by an understanding of the levels below. 
 
"Sealing-off" 
 
Similarly, and fortunately. one does not  have to know all about quarks to 
understand many things about the particle s which they may compose. Thus, a 
nuclear physicist can proceed with theories  of nuclei that are based on protons and 
neutrons, and ignore quark theories and their rivals. The nuclear physicist has a

neutrons, and ignore quark theories and their rivals. The nuclear physicist has a 
chunked picture of protons and neutrons-a  description derived from lower-level 
theories buf which does not require understanding the lower-level theories. 
Likewise, an atomic physicist has a chunked picture of an atomic nucleus derived 
from nuclear theory. Then a chemist has a chunked picture of the electrons and their orbits, and builds theories of small molecules, theories which can be taken 
over in a chunked way by the molecular bi ologist, who has an intuition for how

over in a chunked way by the molecular bi ologist, who has an intuition for how 
small molecules hang together, but whose t echnical expertise is in the field of 
extremely large molecules and how they interact. Then the cell biologist has a 
chunked picture of the units which the mol ecular biologist pores over, and tries to 
use them to account f'or the ways that cells  interact. The point is clear. Each level 
is, in some sense, "sealed off'' from the le vels below it. This is another of Simon's 
vivid terms, recalling the way in which a submarine is built in compartments, so

vivid terms, recalling the way in which a submarine is built in compartments, so 
that if one part is damaged, and wate r begins pouring in, the trouble can be 
prevented from spreading, by closing the doors, thereby sealing off the damaged 
compartment from neighboring compartments. 
Although there is always some "leakage" between the hierarchical levels 
of science, so that a chemist cannot affo rd to ignore lower-level physics totally, or 
a biologist to ignore chemistry totally, th ere is almost no leakage from one level

a biologist to ignore chemistry totally, th ere is almost no leakage from one level 
to a distant level. That is why people earl, have intuitive understandings of other 
people without necessarily understanding the quark model, the structure of nuclei, 
the nature of electron orbits,

 312the chemical bond, the structure of proteins , the organelles in a cell, the methods 
of intercellular communication, the physio logy 'of the various organs of the 
human body, or the complex interactions among organs. All at a person needs is a chunked model of how the highest level act s; and as all know, such models are 
very realistic and successful.  
The Trade-off between Chunking and Determinism 
 
There is, however, perhaps one significan t negative feature of a chunked model: it 
usually does not have exact predictive power. That is, we save ourselves from the impossible task of seeing people as collec tions of quarks (or whatever is at the 
lowest level) by using chunked models: but of course such models only give us 
probabilistic estimates of how other people feel, wil1 react to what we say or do, 
and so on. In short, in using chunked high- level models, we sacrifice determinism

and so on. In short, in using chunked high- level models, we sacrifice determinism 
for simplicity. Despite not being sure how  people will react to a joke, we tell it 
with the expectation at they will do some thing such as laugh, or not laugh-rather 
than, say, climb the nearest flagpole. (Zen masters might well do the latter!) A 
chunked model defines a "space" within whic h behavior is expected to fall, and 
specifies probabilities of its falling in different parts of that space.  
"Computers Can Only Do What You Tell Them to Do"

"Computers Can Only Do What You Tell Them to Do" 
 
Now these ideas can be applied as well to computer programs as to 
compose physical systems. There is an old saw which says, "Computers can only 
what you tell them to do." This is right in one sense, but it misses the hint: you don't know in advance the consequences of what you tell a computer to do; 
therefore its behavior can be as baffli ng and surprising id unpredictable to you as 
that of a person. You generally know in advance the space in which the output will fall, but you don't know details of he re it will fall. For instance, you might

that of a person. You generally know in advance the space in which the output will fall, but you don't know details of he re it will fall. For instance, you might 
write a program to calculate the first milli on digits of 7r. Your program will spew 
forth digits of 7r much faster than you can -but there is no paradox in the fact that 
the computer outracing its programmer. Y ou know in advance the space in which 
the output will lie-namely the space of digits  between 0 and 9-which is to say, )u 
have a chunked model of the program's behavior; but if you'd known ie rest, you 
wouldn't have written the program. 
There is another sense in which this old saw is rusty. This involves the ct

There is another sense in which this old saw is rusty. This involves the ct 
that as you program in ever higher-level languages, you know less and ss precisely what you've told the computer to  do! Layers and layers of translation 
may separate the "front end" of a complex program from the actual machine language instructions. At the level you think and program, your statements may 
resemble declaratives and suggestions more than they resemble imperatives or commands. And all the internal rumbli ng provoked by the input of a high-level 
statement is invisible to you, generally, just as when you eat a sandwich, you are

statement is invisible to you, generally, just as when you eat a sandwich, you are 
spared conscious awareness of the digestive processes it triggers

 313 
In any case, this notion that "computer s can only do what they are told to 
do," first propounded by Lady Lovelace in he r famous memoir, is so prevalent 
and so connected with the notion that "computers cannot think" that we shall 
return to it in later Chapters when our level of sophistication is greater. 
 
Two Types of System 
 
There is an important division between two types of system built up from many 
parts. There are those systems in which th e behavior of some parts tends to cancel 
out the behavior of other parts, with the result that it does not matter too much 
what happens on the low level, because  most anything will yield similar high-
level behavior. An example of this kind of system is a container of gas, where all 
the molecules bump and bang against each other in very complex microscopic 
ways; but the total outcome, from a macros copic point of view, is a very calm,

ways; but the total outcome, from a macros copic point of view, is a very calm, 
stable system with a certain temperatur e, pressure, and volume. Then there are 
systems where the effect of a single low- level event may get magnified into an 
enormous high-level consequence. Such a system is a pinball machine, where the 
exact angle with which a ball strikes each pos t is crucial in determining the rest of 
its descending pathway. 
A computer is an elaborate combinati on of these two types of system. It 
contains subunits such as wires, which behave in a highly predictable fashion:

contains subunits such as wires, which behave in a highly predictable fashion: 
they conduct electricity according to Ohm's law, a very precise, chunked law which resembles the laws governing gases in containers, since it depends on statistical effects in which billions of random effects cancel each other out, yielding a predictable overall behavior. A computer also contains macroscopic 
subunits, such as a printer, whose behavior is completely determined by delicate

subunits, such as a printer, whose behavior is completely determined by delicate 
patterns of currents. What the printer prints is not by any means created by a myriad canceling microscopic effects. In fact, in the case of most computer programs, the value of every single bit in the program plays a critical role in the 
output that gets printed. If any bit were  changed, the output would also change 
drastically. 
Systems which are made up of "reliable" subsystems only-that is, 
subsystems whose behavior can be relia bly predicted from chunked descriptions-

play inestimably important roles in our daily lives, because they are pillars of stability. We can rely on walls not to fall down, on sidewalks to go where they 
went yesterday, on the sun to shine, on cl ocks to tell the time correctly, and so on. 
Chunked models of such systems are virt ually entirely deterministic. Of course, 
the other kind of system which plays a very large role in our lives is a system that

the other kind of system which plays a very large role in our lives is a system that 
has variable behavior which depends on some internal microscopic parameters-often a very large number of them, more over-which we cannot directly observe. 
Our chunked model of such a system is n ecessarily in terms of the "space" of 
operation, and involves probabilistic estimates  of landing in different regions of 
that space. 
A container of gas, which, as I already pointed out, is a reliable system

 314Because of many canceling effects, obeys precise, deterministic laws of physics. 
Such laws are chunked laws, in that they deal with the gas as a whole, nd ignore 
its constituents. Furthermore, the microscopic and macroscopic descriptions of a gas use entirely different terms. The fo rmer requires the pacification of the 
position and velocity of every single com ponent molecule; the latter requires only 
the specification of three new quantities: temperature, pressure, and volume, the 
first two of which do not even have microscopic counterparts. The simple 
mathematical relationship which elates th ese three parameters- pV = cT, where c 
is a constant-is a law which depends on, yet is independent of, the lower-level 
phenomena. Less paradoxically, this law can  be derived from the laws governing 
the molecular level; in that sense it depe nds on the lower level. On the other hand,

the molecular level; in that sense it depe nds on the lower level. On the other hand, 
it is law which allows you to ignore the lower level completely, if you wish: in hat sense it is independent of the lower level. 
It is important to realize that the high-level law cannot be stated in the 
vocabulary of the low-level descripti on. "Pressure" and "temperature" are new 
terms which experience with the low level alone cannot convey. We humans 
perceive temperature and pressure directly; that is how we are guilt, so that it is not amazing that we should have found this  law. But creatures which knew gases

perceive temperature and pressure directly; that is how we are guilt, so that it is not amazing that we should have found this  law. But creatures which knew gases 
only as theoretical mathematical construc ts would have to have an ability to 
synthesize new concepts, if they were to discover this law.  
Epiphenomena 
 
In drawing this Chapter to a close, I w ould like to relate a story about a complex 
system. I was talking one day with two systems programmers for he computer I was using. They mentioned that the operati ng system seemed to be able to handle 
up to about thirty-five users with great co mfort, but at about thirty-five users or

up to about thirty-five users with great co mfort, but at about thirty-five users or 
so, the response time all of a sudden hot up, getting so slow that you might as well log off and go home and wait until later. Jokingl y I said, "Well, that's simple to fix 
just find the place in he operating system  where the number `35' is stored, and 
change it to 60'!" Everyone laughed. The point  is, of course, that there is no such 
place. where, then, does the critical numbe r-35 users-come from? The answer is: 
It is a visible consequence of  the overall system organization -an 
"epiphenometon ,,.

"epiphenometon ,,. 
Similarly, you might ask about a sprinter , "Where is the `9.3' stored, hat 
makes him be able to run 100 yards in 9.3 seconds?" Obviously, it is not stored 
anywhere. His time is a result of how he  is built, what his reaction time is, a 
million factors all interacting when he r uns. The time is quite 'reproducible, but it 
is not stored in his body anywhere. It is spread around among all the cells of his body and only manifests itself in the act of the print itself. 
Epiphenomena abound. In the game of "Go" , there is the feature that “two 
eyes live”. It is not built into the rules, but it is a consequence of the

 315rules. In the human brain, there is gullibility. How gullible are you? Is your 
gullibility located in some "gullibility  center" in your brain? Could a 
neurosurgeon reach in and perform so me delicate operation to lower your 
gullibility, otherwise leaving you alone? If you believe this, you are pretty gullible, and should perhaps consider such an operation. 
 
Mind vs. Brain 
 
In coming Chapters, where we discuss th e brain, we shall examine whether the 
brain's top level-the mind-can be unde rstood without understanding the lower 
levels on which it both depends and does not depend. Are there laws of thinking 
which are "sealed off" from the lower laws that govern the microscopic activity in the cells of the brain? Can mind be "skimme d" off of brain and transplanted into 
other systems? Or is it impossible to unr avel thinking processes into neat and

other systems? Or is it impossible to unr avel thinking processes into neat and 
modular subsystems? Is the brain more like an atom, a renormalized electron, a nucleus, a neutron, or a quark? Is consciousness an epiphenomenon? To 
understand the mind, must one go all the way down to the level of nerve cells?

Ant Fugue 
 317... Ant Fugue 
 
.  .  . .   then, one by one, the four vo ices of the fugue chime in.) 
 
Achilles: I know the rest of you won't believe this, but the answer to the question is 
staring us all in the face, hidden in the pict ure. It is simply one word-but what an 
important one: "MU"! 
CCrab: I know the rest of you won't believe this , but the answer to the question is staring 
us all in the face, hidden in the picture. It  is simply one word-but what an important 
one: "HOLISM"! 
Achilles: Now hold on a minute. You must be seeing things. It's plain as day that the 
message of this picture is "MU", not "HOLISM"! 
Crab: I beg your pardon, but my eyesight is  extremely good. Please look again, and then 
tell me if the the picture doesn't say what I said it says! 
Anteater: I know the rest of you won't believe  this, but the answer to the question is 
important one: "REDUCTIONISM"!

important one: "REDUCTIONISM"! 
Crab: Now hold on a minute. You must be s eeing things. It's plain as day that the 
message of this picture is "HOLISM", not "REDUCTIONISM"! 
Achilles: Another deluded one! Not "HOLIS M", not "REDUCTIONISM", but "MU" is 
the message of this picture, and that much is certain. 
Anteater: I beg your pardon, but my eyesight  is extremely clear. Please look again, and 
then see if the picture doesn't say what I said it says.

then see if the picture doesn't say what I said it says. 
Achilles: Don't you see that the picture is com posed of two pieces, and that each of them 
is a single letter? 
Crab: You are right about the two pieces, but you are wrong in your identification of 
what they are. The piece on the left is entir ely composed of three copies of one word: 
"HOLISM"; and the piece on the right is composed of many copies, in smaller letters, of the same word. Why the letters are of di fferent sizes in the two parts, I don't know, 
but I know what I see, and what I see is "HOLISM", plain as day. How you see

but I know what I see, and what I see is "HOLISM", plain as day. How you see 
anything else is beyond me. 
Anteater: You are right about the two pieces , but you are wrong in your identification of 
what they are. The piece on the left is entirely composed of many copies of one word: "REDUCTIONISM"; and the piece on th e right is composed of one single 
copy, in larger letters, of the same word. W hy the letters are of different sizes in the 
two parts, I don't know, but I know wh at I see, and what I see is 
"REDUCTIONISM", plain as day. How you see anything else is beyond me.

"REDUCTIONISM", plain as day. How you see anything else is beyond me. 
Achilles: I know what is going on here. Each of you has seen letters which compose, or 
are composed of, other letters. In the left-hand piece,

Ant Fugue 
 318there are indeed three "HOLISM"'s, but each one of them is composed out of smaller 
copies of the word "REDUCTIONISM". And in complementary fashion, in the right-hand piece, there is indeed one "REDUC TIONISM", but it is composed out of 
smaller copies of the word "HOLISM". Now this is all fine and good, but in your silly squabble, the two of you have actually missed the forest for the trees. You see, 
what good is it to argue about whether "HOLISM" or "REDUCTIONISM" is right, 
when the proper way to understand the matter is to transcend the question, by answering "Mu", 
Crab: I now see the picture as you have descri bed it, Achilles, but I have no idea of what 
you mean by the strange expression "transcending the question". 
Anteater: I now see the picture as you have de scribed it, Achilles, but I have no idea of 
what you mean by the strange expression "Mu" . .illes: I will be glad to indulge both 
of you, if you will first oblige me, by telli ng me the meaning of these strange

of you, if you will first oblige me, by telli ng me the meaning of these strange 
expressions, "HOLISM" and  "REDUCTIONISM". 
Crab: HOLISM is the most natural thing in the world to grasp. It's simply the belief that 
"the whole is greater than the sum of its pa rts". No one in his right mind could reject 
holism. 
Anteater: REDUCTIONISM is the most natural thing in the world to grasp. It's simply 
the belief that "a whole can be understood completely if you understand its parts, and the nature of their 'sum'". No one in her left brain could reject reductionism. 
Crab: I reject reductionism. I challenge you to tell me, for instance, how to understand a

holism. 
Crab: I reject reductionism. I challenge you to tell me, for instance, how to understand a 
brain reductionistically. Any reductionistic explanation of a brain will inevitably fall 
far short of explaining where the consciousness experienced by a brain arises from. 
Anteater: I reject holism. I challenge you to tell me, for instance, how a holistic 
description of an ant colony sheds any more  light on it than is shed by a description 
of the ants inside it, and their roles,  and their interrelationships. Any holistic 
explanation of an ant colony will inevitabl y fall far short of explaining where the 
consciousness experienced by an ant colony arises from.

consciousness experienced by an ant colony arises from. 
Antilles: Oh, no! The last thing which I want ed to do was to provoke another argument. 
Anyway, now that I understand the controversy, I believe that my explanation of "Mu" will help greatly. You see, "Mu" is an ancient Zen answer which, when given 
to a question, UNASKS the question. Here, the question seems to be, "Should the 
world be understood via holism, or via reduc tionism?" And the answer of "Mu" here 
rejects the premises of the question, which ar e that one or the other must be chosen.

rejects the premises of the question, which ar e that one or the other must be chosen. 
By unasking the question, it reveals a wider trut h: that there is a larger context into 
which both holistic and reductionistic  explanations fit. 
Anteater: Absurd! Your "Mu" is as silly as a cow's moo. I'll have none of this Zen washy-
wishiness.

Ant Fugue 
 319 
Crab: Ridiculous! Your "ML" is as silly as  a kitten's mew. I'll have none of this Zen 
washy-wishiness. 
Achilles: Oh, dear! We're getting nowhere fast . Why have you stayed so strangely silent, 
Mr. Tortoise? It makes me very uneasy. Surely you must somehow be capable of 
helping straighten out this mess? 
Tortoise: I know the rest of you won't believe this, but the answer to the question is 
important one: "Mu"! 
Gust as he says this, the fourth voice in the fugue being played enters, exactly one octave 
below the first entry.) 
Achilles: Oh, Mr. T, for once you have let me  down. I was sure that you, who always see 
the most deeply into things, would be able  to resolve this dilemma-but apparently, 
you have seen no further than I myself saw.  Oh, well, I guess I should feel pleased to 
have seen as far as Mr. Tortoise, for once.

have seen as far as Mr. Tortoise, for once. 
Tortoise: I beg your pardon, but my eyesight  is extremely fine. Please look again, and 
then tell me if the picture doesn't say what I said it says. 
Achilles: But of course it does! You have merely repeated my own original observation. Tortoise: Perhaps "Mu" exists in this pi cture on a deeper level than you imagine, 
Achilles-an octave lower (figuratively sp eaking). But for now I doubt that we can 
settle the dispute in the abstract. I would like to see both the holistic and

settle the dispute in the abstract. I would like to see both the holistic and 
reductionistic points of view laid out more explicitly; then there may be more of a basis for a decision. I would very much like to  hear a reductionistic description of an 
ant colony, for instance. 
Crab: Perhaps Dr. Anteater will tell you someth ing of his experiences in that regard. 
After all, he is by profession something of an expert on that subject. 
Tortoise: I am sure that we have much to  learn from you, Dr. Anteater. Could you tell us 
more about ant colonies, from a reductionistic point of view?

more about ant colonies, from a reductionistic point of view? 
Anteater: Gladly. As Mr. Crab mentioned to  you, my profession has led me quite a long 
way into the understanding of ant colonies. 
Achilles: I can imagine! The profession of anteater would seem to be synonymous with 
being an expert on ant colonies! 
Anteater: I beg your pardon. "Anteater" is not  my profession; it is my species. By 
profession, I am a colony surgeon. I speciali ze in correcting nervous disorders of the 
colony by the technique of surgical removal. 
Achilles: Oh, I see. But what do you mean by "nervous disorders" of an ant colony? Anteater: Most of my clients suffer from some sort of speech impairment. You know,

Achilles: Oh, I see. But what do you mean by "nervous disorders" of an ant colony? Anteater: Most of my clients suffer from some sort of speech impairment. You know, 
colonies which have to grope for words in everyday situations. It can be quite tragic. 
I attempt to remedy the situation by, uhh—rem oving the defective part of the colony. 
These operations

Ant Fugue 
 320are sometimes quite involved, and of course  years of study are required before one 
can perform them. 
Achilles: But-isn't it true that, before one can suffer from speech impairment, one must 
have the faculty of speech? 
Anteater: Right. Achilles: Since ant colonies don't have that faculty, I am a little confused. Crab: It's too 
bad, Achilles, that you weren't here last week, when Dr. 
Anteater and Aunt Hillary were my house guests. I should have  thought of having you 
over then. 
Achilles: Is Aunt Hillary your aunt, Mr. Crab? Crab: Oh, no, she's not really anybody's 
aunt. 
Anteater: But the poor dear insists that ever ybody should call her that, even strangers. It's 
just one of her many endearing quirks. 
Crab: Yes, Aunt Hillary is quite eccentric, but such a merry old soul. It's a shame I didn't 
have you over to meet her last week. 
Anteater: She's certainly one of the best-edu cated ant colonies I have ever had the good

over then. 
aunt. 
Anteater: She's certainly one of the best-edu cated ant colonies I have ever had the good 
fortune to know. The two of us have sp ent many a long evening in conversation on 
the widest range of topics. 
Achilles: I thought anteaters were devourers of ants, not patrons of antintellectualism! Anteater: Well, of course the two are not mutu ally inconsistent. I am on the best of terms 
with ant colonies. It's just ANTS that I eat, not colonies-and that is good for both 
parties: me, and the colony.

parties: me, and the colony. 
Achilles: How is it possible that-- Tortoise: How is it possible that-- Achilles: -having its ants eaten can do an ant colony any good? Crab: How is it possible 
that 
Tortoise: -having a forest fire can do a fore st any good? Anteater: How is it possible that 
Crab: -having its branches pruned can do a tr ee any good? Anteater: -having a haircut can 
do Achilles any good? 
Tortoise: Probably the rest of you were too engrossed in the discussion to notice the 
lovely stretto which just occurred in this Bach fugue.

that 
lovely stretto which just occurred in this Bach fugue. 
Achilles: What is a stretto? Tortoise: Oh, I'm sorry; I thought you knew the term. It is where one theme repeatedly 
enters in one voice after another, with very little delay between entries. 
Achilles: If I listen to enough fugues, soon I'll know  all of these things and will be able to 
pick them out myself, without their having to be pointed out. 
Tortoise: Pardon me, my friends. I am sorry to  have interrupted. Dr. Anteater was trying 
to explain how eating ants is perfectly consistent with being a friend of an ant colony. 
Achilles: Well, I can vaguely see how it might be possible for a limited and regulated

Achilles: Well, I can vaguely see how it might be possible for a limited and regulated 
amount of ant consumption to improve the overall health of

Ant Fugue 
 321a colony-but what is far more perplexing is  all this talk about having conversations 
with ant colonies. That's impossible. An ant colony is simply a bunch of individual ants running around at random looking for food and making a nest. 
Anteater: You could put it that  way if you want to insist on seeing the trees but missing 
the forest, Achilles. In fact, ant colonies, seen as wholes, are quite well-defined units, 
with their own qualities, at times including the mastery of language. 
Achilles: I find it hard to imagine myself shouting something out loud in the middle of 
the forest, and hearing an ant colony answer back. 
Anteater: Silly fellow! That's not the wa y it happens. Ant colonies don't converse out 
loud, but in writing. You know how ants form trails leading them hither and thither? 
Achilles: Oh, yes-usually straight through the kitchen sink and into my peach jam. Anteater: Actually, some trails contain information in coded form. If you know the

Achilles: Oh, yes-usually straight through the kitchen sink and into my peach jam. Anteater: Actually, some trails contain information in coded form. If you know the 
system, you can read what they're saying just like a book. Achilles: Remarkable. And 
can you communicate back to them? Anteater: Without any trouble at all. That's how Aunt Hillary and I have conversations for hours. I take a stick and draw trails in the moist ground, and watch the ants follow my tr ails. Presently, a new trail starts getting 
formed somewhere. I greatly enjoy watchi ng trails develop. As they are forming, I 
anticipate how they will continue (and more  often I am wrong than right). When the

anticipate how they will continue (and more  often I am wrong than right). When the 
trail is completed, I know what Aunt Hillary is thinking, and I in turn make my reply. 
Achilles: There must be some amazingly smart ants in that colony, I'll say that. Anteater: I think you are still having some diffi culty realizing the difference in levels 
here. Just as you would never confuse an i ndividual tree with a forest, so here you 
must not take an ant for the colony. You see,  all the ants in Aunt Hillary are as dumb 
as can be. They couldn't converse to save their little thoraxes! 
Achilles: Well then, where does the ability to converse come from? It must reside

Achilles: Well then, where does the ability to converse come from? It must reside 
somewhere inside the colony! I don't understa nd how the ants can all be unintelligent, 
if Aunt Hillary can entertain you for hours with witty banter. 
Tortoise: It seems to me that the situation is not unlike the composition of  a human brain 
out of neurons. Certainly no one would insist  that individual brain cells have to be 
intelligent beings on their own, in order to e xplain the fact that a person can have an 
intelligent conversation.

intelligent conversation. 
Achilles: Oh, no, clearly not. With brain cells , I see your point completely. Only ... ants 
are a horse of another color. I mean, ants  just roam about at will, completely 
randomly, chancing now and then upon a morsel  of food ... They are free to do what 
they want to do, and with that freedom, I don’ t see at all how their behaviour, seen as 
a whole, can

Ant Fugue 
 322amount to anything coherent-especially someth ing so coherent as the brain behavior 
necessary for conversing. 
Crab: It seems to me that the ants are free only within certain constraints. For example, 
they are free to wander, to brush against each other, to pick up small items, to work on trails, and so on. But they never step out  of that small world, that ant-system, 
which they are in. It would never occur to them, for they don't have the mentality to 
imagine anything of the kind. Thus the ants are very reliable components, in the sense that you can depend on them to perform certain kinds of tasks in certain ways. 
Achilles: But even so, within those limits they  are still free, and they just act at random, 
running about incoherently without any re gard for the thought mechanisms of a 
higher-level being which Dr. Anteater asserts they are merely components of.

higher-level being which Dr. Anteater asserts they are merely components of. 
Anteater: Ah, but you fail to recognize one thing. Achilles-the regularity of statistics. Achilles: How is that? Anteater: For example, even though ants as individuals wander about in what seems a 
random way, there are nevertheless overall tr ends, involving large numbers of ants, 
which can emerge from that chaos. 
Achilles: Oh, I know what you mean. In fact, an t trails are a perfect example of such a

Achilles: Oh, I know what you mean. In fact, an t trails are a perfect example of such a 
phenomenon. There, you have really quite unpredictable motion on the part of any single ant-and yet, the trail itself seems to  remain well-defined and stable. Certainly 
that must mean that the individual ants are not just running about totally at random. 
Anteater: Exactly, Achilles. There is some degree of communication among the ants, just 
enough to keep them from wandering off co mpletely at random. By this minimal 
communication they can remind each othe r that they are not alone but are

communication they can remind each othe r that they are not alone but are 
cooperating with teammates. It takes a large number of ants, all reinforcing each other this way, to sustain any activity-such as trail-building-for any length of time. 
Now my very hazy understanding of the operati on of brains leads me to believe that 
something similar pertains to the firing of neurons. Isn't it true, Mr. Crab, that it takes 
a group of neurons firing in order to make another neuron fire? 
Crab: Definitely. Take the neurons in Achilles' brain, for example. Each neuron receives

Crab: Definitely. Take the neurons in Achilles' brain, for example. Each neuron receives 
signals from neurons attached to its input lines, and if the sum total of inputs at any 
moment exceeds a critical threshold. then that neuron will fire and send its own output pulse rushing off to other neurons , which may in turn fire-and on down the 
line it goes. The neural flash swoops relen tlessly in its Achillean path, in shapes 
stranger then the dash of a gnat-hungry swallo w; every twist, every turn foreordained 
by the neural structure in Achilles' brain, until sensory input messages interfere.

by the neural structure in Achilles' brain, until sensory input messages interfere. 
Achilles: Normally, I think that I'M in contro l of what I think-but the way you put it turns 
it all inside out, so that it sounds as though "I" am just

Ant Fugue 
 323what comes out of all this neural structure,  and natural law. It makes what I consider 
my SELF sound at best like a by-product of an organism governed by natural law, and at worst, an artificial notion produced by my distorted perspective. In other words, you make me feel like I don't know who or what-I am, if anything. 
Tortoise: You'll come to understand much better as we go along. But Dr. Anteater-what do you make of this similarity? Anteater: I knew there was something parallel going on in the two very different systems. 
Now I understand it much better. It seems that group phenomena which have coherence-trail-building, for example-will take place only when a certain threshold number of ants get involved. If an effort is initiated, perhaps at random, by a few ants 
in some locale, one of two things can ha ppen: either it will fizzle out after a brief 
sputtering start

sputtering start  
Achilles: When there aren't enough ants to keep the thing rolling? Anteater: Exactly. The other thing that can happen is that a critical mass of ants is 
present, and the thing will snowball, bringing more and more ants into the picture. In 
the latter case, a whole "team" is brought in to being which works on a single project. 
That project might be trailmaking, or food-gathering, or it might involve nest-
keeping. Despite the extreme simplicity of th is scheme on a small scale, it can give 
rise to very complex consequences on a larger scale. 
Achilles: I can grasp the general idea of orde r emerging from chaos, as you sketch it, but

Achilles: I can grasp the general idea of orde r emerging from chaos, as you sketch it, but 
that still is a long way from the ability to c onverse. After all, order also emerges from 
chaos when molecules of a gas bounce ag ainst each other randomly-yet all that 
results there is an amorphous mass with but three parameters to characterize it: 
volume, pressure, and temperature. Now that 's a far cry from the ability to understand 
the world, or to talk about it! 
Anteater: That highlights a very interesting difference between the explanation of the

Anteater: That highlights a very interesting difference between the explanation of the 
behavior of an ant colony and the explan ation of the behavior of gas inside a 
container. One can explain the behavior  of the gas simply by calculating the 
statistical properties of the motions of its molecules. There is no need to discuss any 
higher elements of structure than molecule s, except the full gas itself. On the other 
hand, in an ant colony, you can't even begi n to understand the activities of the colony 
unless you go through several layers of structure.

unless you go through several layers of structure. 
Achilles: I see what you mean. In a gas, one jump takes you from the lowest level-
molecules-to the highest level-the full gas. There are no intermediate levels of organization. Now how do intermediate levels of organized activity arise in an ant colony? 
Anteater: It has to do with the existence of se veral different varieties of ants inside any 
colony. 
Achilles: Oh, yes. I think I have heard about that. They are called "castes", aren't they? Anteater: That's correct. Aside from the queen, there are males, who do

Ant Fugue 
 324practically nothing towards, the upkeep of the nest, and then— 
Achilles: And of course there are soldiers-Glorious Fighters Against Communism! Crab: Hmm ... I hardly think that could be  right, Achilles. An ant colony is quite 
communistic internally, so why would its so ldiers fight against communism? Or am I 
right, Dr. Anteater? . 
Anteater: Yes, about colonies you are ri ght, Mr. Crab; they are indeed based on 
somewhat communistic principles. But about soldiers Achilles is somewhat naive. In 
fact, the so-called "soldiers" are hardly adept at fighting at all. They are slow, 
ungainly ants with giant heads, who can snap  with their strong jaws, but are hardly to 
be glorified. As in a true communistic stat e, it is rather the workers who are to be 
glorified. It is they who do most of the chores, such as food-gathering, hunting, and 
nursing of the young. It is even they who do most of the fighting.

nursing of the young. It is even they who do most of the fighting. 
Achilles: Bah. That is an absurd state of affairs. Soldiers who won't fight! Anteater: Well, as I just said, they really aren 't soldiers at all. It's the workers who are 
soldiers; the soldiers are just lazy fatheads. 
Achilles: Oh, how disgraceful! Why, if I were an ant, I'd put some discipline in their 
ranks! I'd knock some sense into those fatheads! 
Tortoise: If you were an ant? How could you be an ant? There is no way to map your

Tortoise: If you were an ant? How could you be an ant? There is no way to map your 
brain onto an ant brain, so it seems to me to be a pretty fruitless question to worry 
over. More reasonable would be the pr oposition of mapping your brain onto an ant 
colony ... But let us not get sidetracked. Le t Dr. Anteater continue with his most 
illuminating description of castes and their role in the higher levels of organization. 
Anteater: Very well. There are all sorts of tasks which must be accomplished in a colony,

Anteater: Very well. There are all sorts of tasks which must be accomplished in a colony, 
and individual ants develop specializations. Usually an ant's specialization changes as 
the ant ages. And of course it is also dependent on the ant's caste. At any one 
moment, in any small area of a colony, there ar e ants of all types present. Of course, 
one caste may be be very sparse in some places and very dense in others. 
Crab: Is the density of a given caste, or speci alization, just a random thing? Or is there a 
reason why ants of one type might be more heavily concentrated in certain areas, and 
less heavily in others?

less heavily in others? 
Anteater: I'm glad you brought that up, since it is of crucial importance in understanding 
how a colony thinks. In fact, there evolves,  over a long period of time, a very delicate 
distribution of castes inside a colony. And it is this distribution which allows the 
colony to have the complexity which underlies the ability to converse with me. 
Achilles: It would seem to me that the constant motion of ants to and fro would 
completely prevent the possibility of a very delicate distribution.Any delicate

completely prevent the possibility of a very delicate distribution.Any delicate 
distribution would be quickly destroyed by all the random motions of ants, just as 
any delicate pattern among molecules in a ga s would not survive for an instant, due 
to the random bombardment from all sides. 
Anteater: In an ant colony. the situation is quite  the contrary. In fact, it is just exactly the 
to-ing and fro-ing of ants inside the colony

Ant Fugue 
 325which adapts the caste distribution to vary ing situations, and thereby preserves the 
delicate caste distribution. You see, the caste distribution cannot remain as one single rigid pattern; rather, it must constantly be changing so as to reflect, in some manner, 
the real-world situation with which the colony is dealing, and it is precisely the 
motion inside the colony which updates the caste  distribution, so as to keep it in line 
with the present circumstances facing the colony. 
Tortoise: Could you give an example? Anteater: Gladly. When I, an anteater, arrive to pay a visit to Aunt Hillary, all the foolish 
ants, upon sniffing my odor, go into a panic-wh ich means, of course, that they begin 
running around completely differently from the way they were before I arrived. 
Achilles: But that's understandable, since you're a dreaded enemy of the colony. Anteater: Oh, no. I must reiterate that, far fr om being an enemy of the colony, I am Aunt

Achilles: But that's understandable, since you're a dreaded enemy of the colony. Anteater: Oh, no. I must reiterate that, far fr om being an enemy of the colony, I am Aunt 
Hillary's favorite companion. And Aunt Hilla ry is my favorite aunt. I grant you, I'm 
quite feared by all the individual ants in the colony-but that's another matter entirely. In any case, you see that the ants' action in response to my arrival completely changes 
the internal distribution of ants. 
Achilles: That's clear. Anteater: And that sort of thing is the updating which I spoke of. The new distribution 
reflects my presence. One can describe the change from old state to new as having 
added a "piece of knowledge" to the colony.

added a "piece of knowledge" to the colony. 
Achilles: How can you refer to the distribution of different types of ants inside a colony 
as a "piece of knowledge"? 
Anteater: Now there's a vital point. It require s some elaboration. You see, what it comes 
down to is how you choose to describe the cas te distribution. If you continue to think 
in terms of the lower levels-individual an ts-then you miss the forest for the trees. 
That's just too microscopic a level, and when you think microscopically, you're 
bound to miss some large-scale features. Y ou've got to find the proper high-level

bound to miss some large-scale features. Y ou've got to find the proper high-level 
framework in which to describe the caste  distribution-only then will it make sense 
how the caste distribution can encode many pieces of knowledge. 
Achilles: Well, how DO you find the proper-sized units in which to describe the present 
state of the colony, then? 
Anteater: All right. Let's begin at the botto m. When ants need to get something done, 
they form little "teams", which stick togeth er to perform a chore. As I mentioned

they form little "teams", which stick togeth er to perform a chore. As I mentioned 
earlier, small groups of ants are constantly forming and unforming. Those which actually exist for a while are the teams, and the reason they don't fall apart is that there really is something for them to do. 
Achilles: Earlier you said that a group will s tick together if its size exceeds a certain 
threshold. Now you're saying that a group will stick together if there is something for it to do. 
Anteater: They are equivalent statements. For instance, in food-gathering,

Ant Fugue 
 326if there is an inconsequential amount of  food somewhere which gets discovered by 
some wandering Ant who then attempts to communicate its enthusiasm to other ants, the number of ants who respond will be pr oportional to the size of the food sample-
and an inconsequential amount will not attr act enough ants to surpass the threshold. 
Which is exactly what I meant by saying there is nothing to do-too little food ought 
to be ignored. 
Achilles: I see. I assume that these "teams"  are one of the levels of structure falling 
somewhere in between the single-ant level and the colony level. 
Anteater: Precisely. There exists a special ki nd of team, which I call a "signal"-and all the 
higher levels of structure are based on si gnals. In fact, all the higher entities are 
collections of signals acting in concert.  There are teams on higher levels whose 
members are not ants, but teams on lower levels. Eventually you reach the lowest-
level teams which is to say, signals-and below them, ants.

level teams which is to say, signals-and below them, ants. 
Achilles: Why do signals deserve their suggestive name? Anteater: It comes from their function. The effect  of signals is to transport ants of various 
specializations to appropriate parts of the colony. So the typical story of a signal is thus: it comes into existence by exceeding th e threshold needed for survival, then it 
migrates for some distance through the colony, and at some point it more or less disintegrates into its individual members, leaving them on their own.

migrates for some distance through the colony, and at some point it more or less disintegrates into its individual members, leaving them on their own. 
Achilles: It sounds like a wave, carrying sand dollars and seaweed from afar, and leaving 
them strewn, high and dry, on the shore. 
4nteater: In a way that's analogous, since th e team does indeed deposit something which 
it has carried from a distance, but whereas the water in the wave rolls back to the sea, there is no analogous carrier substance in  the case of a signal, since the ants 
themselves compose it. 
Tortoise: And I suppose that a signal loses its coherency just at some spot in the colony

Tortoise: And I suppose that a signal loses its coherency just at some spot in the colony 
where ants of that type were needed in the first place. 
Anteater: Naturally. Achilles: Naturally? It's not so obvious to ME  that a signal should always go just where it 
is needed. And even if it goes in the right  direction, how does it figure out where to 
decompose? How does it know it has arrived? 
Anteater: Those are extremely important ma tters, since they involve explaining the 
existence of purposeful behavior-or what s eems to be purposeful behavior-on the part

existence of purposeful behavior-or what s eems to be purposeful behavior-on the part 
of signals. From the description, one would be inclined to characterize the signals' 
behavior as being oriented towards filling a need, and to call it "purposeful". But you 
can look at it otherwise. 
Achilles: Oh, wait. Either the behavior is  purposeful, or it is NOT. I don't see how you 
can have it both ways. 
Anteater: Let me explain my way of seeing things, and then see if you agree. Once a 
signal is formed, there is no awareness on its part that it

Ant Fugue 
 327should head off in any particular directi on. But here, the delicate caste distribution 
plays a crucial role. It is what determin es the motion of signals through the colony, 
and also how long a signal will remain stable, and where it will "dissolve". 
Achilles: So everything depends on the caste distribution, eh? Anteater: Right. Let's say a signal is moving along. As it goes, the ants which compose it 
interact, either by direct contact or by exch ange of scents, with ants of the local 
neighborhoods which it passes through. The contacts and scents provide information about local matters of urgency, such as nest-building, or nursing, or whatever. The 
signal will remain glued together as long as the local needs are different from what it 
can supply; but if it CAN contribute, it disi ntegrates, spilling a fresh team of usable 
ants onto the scene. Do you see now how the caste distribution acts as an overall 
guide of the teams inside the colony?

guide of the teams inside the colony? 
Achilles: I do see that. Anteater: And do you see how this way of looki ng at things requires attributing no sense 
of purpose to the signal? 
Achilles: I think so. Actually, I'm beginning to see things from two different vantage 
points. From an ant's-eye point of view, a signal has NO purpose. The typical ant in a 
signal is just meandering around the colony, in  search of nothing in particular, until it 
finds that it feels like stopping. Its teammates usually agree, and at that moment the

finds that it feels like stopping. Its teammates usually agree, and at that moment the 
team unloads itself by crumbling apart, l eaving just its members but none of its 
coherency. No planning is required, no looki ng ahead; nor is any search required, to 
determine the proper direction. But from the COLONY'S point-of view, the team has 
just responded to a message which was written in the language of the caste 
distribution. Now from this perspective, it looks very much like purposeful activity. 
Crab: What would happen if the caste dist ribution were entirely random? Would signals

Crab: What would happen if the caste dist ribution were entirely random? Would signals 
still band and disband? 
Anteater: Certainly. But the co lony would not last long, due to the meaninglessness of the 
caste distribution. 
Crab: -Precisely the point I wanted to make. Colonies survive because their caste 
distribution has meaning, and that meaning is a holistic aspect, invisible on lower 
levels. You lose explanatory power unless you take that higher level into account.

levels. You lose explanatory power unless you take that higher level into account. 
Anteater: I see your side; but I believe you see things too narrowly.  Crab: How so? Anteater: Ant colonies have been subjected to the rigors of evolution for billions of years. 
A few mechanisms were selected for, and mo st were selected against. The end result 
was a set of mechanisms which make ant co lonies work as we have been describing. 
If you could watch the whole process in a movie-running a billion or so times faster

If you could watch the whole process in a movie-running a billion or so times faster 
than life, of course-the emergence of various mechanisms would be seen as natural responses to external pressures, just as bubbles in boiling water are natural responses to an external heat source. I don't suppose you

Ant Fugue 
 328see "meaning" and "purpose", in the bubbles in boiling water-or do you? 
Crab: No, but Anteater: Now that's MY point. No matter how  big a bubble is, it owes its existence to 
processes on the molecular level, and you can  forget about any "higher-level laws". 
The same goes for ant colonies and their t eams. By looking at things from the vast 
perspective of evolution, you can drain th e whole colony of meaning and purpose. 
They become superfluous notions. 
Achilles: Why, then, Dr. Anteater, did you tell me that you talked with Aunt Hillary? It 
now seems that you would deny that she can talk or think at all. 
Anteater: I am not being inconsistent, Achille s. You see, I have as much difficulty as 
anyone else in seeing things on such a gra ndiose time scale, so I find it much easier 
to change points of view. When I do so, fo rgetting about evolution and seeing things 
in the here and now, the vocabulary of teleology comes back: the MEANING of the

in the here and now, the vocabulary of teleology comes back: the MEANING of the 
caste distribution and the PURPOSEFULNE SS of signals. This not only happens 
when I think of ant colonies, but also wh en I think about my own brain and other 
brains. However, with some effort I can al ways remember the other point of view if 
necessary, and drain all these systems of meaning, too. 
Crab: Evolution certainly works some miracl es. You never know the next trick it will pull 
out of its sleeve. For instance, it wouldn't su rprise me one bit if it were theoretically

out of its sleeve. For instance, it wouldn't su rprise me one bit if it were theoretically 
possible for two or more "signals" to pa ss through each other, each one unaware that 
the other one is also a signal; each one treating the other as if it were just part of the 
background population. 
Anteater: It is better than theoretically possible; in fact it happens routinely! Achilles: Hmm ... What a strange image that conjures up in my mind. I can just imagine 
ants moving in four different directions , some black, some white, criss-crossing, 
together forming an orderly pattern, almost like-like

together forming an orderly pattern, almost like-like 
Tortoise: A fugue, perhaps? 
Achilles: Yes-that's it! An ant fugue! Crab: An interesting image, Achilles. By the way, all that talk of boiling water made me 
think of tea. Who would like some more? Achilles: I could do with another cup, Mr. C. 
Crab: Very good. Achilles: Do you suppose one could separate out the different visual "voices" of such an 
"ant fugue"? I know how hard it is for me

"ant fugue"? I know how hard it is for me 
Tortoise: Not for me, thank you. Achilles: -to track a single voice Anteater: I'd like some, too,  Mr. Crab Achilles: -- in a musical fugue--

Ant Fugue 
 329
 
FIGURE 61. "Ant Fugue", by M. C. Escher (woodcut, 1953). 
 Anteater: -if it isn't too much trouble Achilles: . -when all of them Crab: Not at all. Four cups of tea Tortoise: Three? Achilles: -are going at once. Crab: -coming right up! Anteater: That's an interesting thought, Achille s. But its unlikely that anyone could draw 
such a picture in a convincing way  
Achilles: That's too bad.  Tortoise: Perhaps you could answer this, Dr. Anteater. Does a signal, from its creation 
until its dissolution, always consist of the same set of ants? 
Anteater: As a matter of fact, the individua ls in a signal sometimes break off and get 
replaced by others of the same caste, if ther e are a few in the area. Most often, signals 
arrive at their disintegration points with nary an ant in common with their starting 
lineup.

Ant Fugue 
 330Crab: I can see that the signals are constan tly affecting the caste distribution throughout 
the colony, and are doing so in response to th e internal needs of the colony-which in 
turn reflect the external situation which th e colony is faced wit h. Therefore the caste 
distribution, as you said, Dr. Anteater, ge ts continually updated in a way which 
ultimately reflects the outer world. 
Achilles: But what about those intermediate levels of structure? You were saying that the 
caste distribution should best be pictured not in terms of ants or signals, but in terms 
of teams whose members were other teams,  whose members were other teams, and 
so on until you come down to the ant level. And you said that that was the key to 
understanding how it was possible to descri be the caste distribution as encoding 
pieces of information about the world. 
Anteater: Yes, we are coming to all that. I pref er to give teams of a sufficiently high level

Anteater: Yes, we are coming to all that. I pref er to give teams of a sufficiently high level 
the name of "symbols". Mind you, this se nse of the word has some significant 
differences from the usual sense. My "symbols" are ACTIVE SUBSYSTEMS of a complex system, and they are composed of lower-level active subsystems ... They are therefore quite different from PASSIVE symbols, external to the system, such as letters of the alphabet or musical notes, which sit there immobile, waiting for an 
active system to process them. 
Achilles: Oh, this is rather complicated, isn' t it? I just had no idea that ant colonies had

Achilles: Oh, this is rather complicated, isn' t it? I just had no idea that ant colonies had 
such an abstract structure. 
Anteater: Yes, it's quite remarkable. But all th ese layers of structure are necessary for the 
storage of the kinds of knowledge which enab le an organism to be "intelligent" in 
any reasonable sense of the word. Any syst em which has a mastery of language has 
essentially the same underlying sets of levels. 
Achilles: Now just a cotton-picking minute. Are you insinuating that my brain consists 
of, at bottom, just a bunch of ants running around? 
Anteater: Oh, hardly. You took me a little too literally. The lowest level may be utterly

Anteater: Oh, hardly. You took me a little too literally. The lowest level may be utterly 
different. Indeed, the brains of anteaters, for instance, are not composed of ants. But 
when you go up a level or two in a brain, you reach a level whose elements have 
exact counterparts in other systems of equal intellectual strength-such as ant colonies. 
Tortoise: That is why it would be reasonabl e to think of mapping your brain, Achilles, 
onto an ant colony, but not onto the brain of a mere ant. 
Achilles: I appreciate the compliment. But how would such a mapping be carried out? 
For instance, what in my brain corres ponds to the low level teams which you call

For instance, what in my brain corres ponds to the low level teams which you call 
signals? 
Anteater: Oh, I but dabble in brains, and ther efore couldn't set up the map in its glorious 
detail. But-and correct me if I'm wrong, Mr. Crab-I would surmise that the brain counterpart to an ant colony's signal is the fi ring of a neuron; or perhaps it is a larger-
scale event, such as a pattern of neural firings.

Ant Fugue 
 331Crab: I would tend to agree. But don't you thi nk that, for the purposes of our discussion, 
delineating the exact counterpart is not in itself crucial, desirable though it might be? 
It seems to me that the main idea is that such a correspondence does exist, even if we 
don't know exactly how to define it right now. I would only ques tion one point, Dr. 
Anteater, which you raised, and that concerns  the level at which one can have faith 
that the correspondence begins. You seemed  to think that a SIGNAL might have a 
direct counterpart in a brain; whereas I feel that it is only at the level of your 
ACTIVE SYMBOLS and above that it is likely that a correspondence must exist. 
Anteater: Your interpretation may very we ll be more accurate than mine, Mr. Crab. 
Thank you for bringing out that subtle point. 
Achilles: What does a symbol do that a signal couldn't do? Anteater: It is something like the difference between words and letters. Words, which are

Achilles: What does a symbol do that a signal couldn't do? Anteater: It is something like the difference between words and letters. Words, which are 
meaning-carrying entities, are composed of letters, which in themselves carry no meaning. This gives a good idea of the difference between symbols and signals. In fact it is a useful analogy, as  long as you keep in mind the fact that words and letters 
are PASSIVE, symbols and signals are ACTIVE. 
Achilles: I'll do so, but I'm not sure I understa nd why it is so vital to stress the difference 
between active and passive entities. 
Anteater: The reason is that the meaning whic h you attribute to any passive symbol, such

Anteater: The reason is that the meaning whic h you attribute to any passive symbol, such 
as a word on a page, actually derives from the meaning which is carried by 
corresponding active symbols in your brain. So  that the meaning of passive symbols 
can only be properly understood when it is related to the meaning of active symbols. 
Achilles: All right. But what is it that e ndows a SYMBOL-an active one, to be sure-with 
meaning, when you say that a SIGNAL, whic h is a perfectly good entity in its own 
right, has none? Anteater: It a ll has to do with the way that symbols can cause other 
symbols to be triggered. When one symbol becomes active, it does not do so in 
isolation. It is floating about, indeed, in a medium, which is characterized by its caste 
distribution.

distribution. 
Crab: Of course, in a brain there is no su ch thing as a caste distribution, but the 
counterpart is the "brain state". There, you describe the states of all the neurons, and 
all the interconnections, and the threshold for firing of each neuron. 
Anteater: Very well; let's lump "caste dist ribution" and "brain state" under a common 
heading, and call them just the "state". No w the state can be described on a low level 
or on a high level. A low-level description of  the state of an ant colony would involve

or on a high level. A low-level description of  the state of an ant colony would involve 
painfully specifying the location of each an t, its age and caste, and other similar 
items. A very detailed description, yielding practically no global insight as to WHY it 
is in that state. On the other hand, a description on a high level would involve 
specifying which symbols could be tri ggered by which combinations of other 
symbols, under what conditions, and so forth.

Ant Fugue 
 332Achilles: What about a description on the level of signals, or teams? 
Anteater: A description on that level would fall somewhere in between the low-level and 
symbol-level descriptions. It would contain a great deal of information about what is 
actually going on in specific locations th roughout the colony, although certainly less 
than an ant-by-ant description, since teams c onsist of clumps of ants. A team-by-team 
description is like a summary of an ant-by- ant description. However, you have to add 
extra things which were not present in the ant-by-ant description-such as the 
relationships between teams, and the supply of various castes here and there. This 
extra complication is the price you pay for the right to summarize. 
Achilles: It is interesting to me to compare the merits of the descriptions at various levels. 
The highest-level description seems to carry the most explanatory power, in that it gives you the most intuitive picture of the ant colony, although strangely enough, it

The highest-level description seems to carry the most explanatory power, in that it gives you the most intuitive picture of the ant colony, although strangely enough, it 
leaves out seemingly- the most important feature-the ants. 
Anteater: But you see, despite appearances, the ants are not the most important feature. 
Admittedly, were it not for them, the colony Wouldn't exist: but something equivalent-a brain-can exist, ant-free. So, at  least from a high-level point of view, the 
ants are dispensable. .Achilles: I'm su re no ant would embrace your theory with 
eagerness.

eagerness. 
Anteater: Well, I never met an ant with a high-level point of view. Crab: What a counterintuitive picture you paint, Dr. Anteater. It seems that, if what you 
say is true, in order to grasp the whole st ructure, you have to describe it omitting any 
mention of its fundamental building blocks. 
Anteater: Perhaps I can make it a little clear er by an analogy. Imagine you have before 
you a Charles Dickens novel. 
Achilles: The Pickwick Papers-will that do? Anteater: Excellently! And now imagine tr ying the following game: you must find a way

Achilles: The Pickwick Papers-will that do? Anteater: Excellently! And now imagine tr ying the following game: you must find a way 
of mapping letters onto ideas, so that the entire Pickwick Papers makes sense when 
you read it letter by letter. 
Achilles: Hmm ... You mean that every time I hit a word such as "the", I have to think of 
three definite concepts, one after another, with no room for variation? 
Anteater: Exactly. They are the `t'-concept, th e `h'-concept, and the `e'-concept-and every 
time, those concepts are as they were the preceding time. 
Achilles: Well, it sounds like that would turn the experience of "reading" The Pickwick 
Papers into an indescribably boring ni ghtmare. It would be an exercise in

Papers into an indescribably boring ni ghtmare. It would be an exercise in 
meaninglessness, no matter what concept I associated with each letter. 
Anteater: Exactly. There is no natural mapping from the individual letters into the real 
world. The natural mapping occurs on a higher level between words, and parts of the 
real world. If you wanted to describe  the book, therefore, you would make no 
mention of the letter level.

Ant Fugue 
 333Achilles: Of course not! I'd describe the plot and the characters, and so forth. 
Anteater: So there you are. You would omit all mention of the building blocks, even 
though the book exists thanks to them. They are the medium, but not the message. 
Achilles: All right-but what about ant colonies? Anteater: Here, there are active signals inst ead of passive letters, and active symbols 
instead of passive words-hut the idea carries over. 
Achilles: Do you mean I couldn't establish a mapping between signals and things in the 
real world? 
Anteater: You would find that you could not do it in such a way that the triggering of new 
signals would make am sense. Nor could you succeed on any lower level-for example the ant level. Only on the symbol leve l do the triggering patterns make sense. 
Imagine, for instance, that one day you were  watching Aunt Hillary when I arrived to 
pay a call. You could watch as carefully  as you wanted, and yet you would probably

pay a call. You could watch as carefully  as you wanted, and yet you would probably 
perceive nothing more than a rearrangement of ants. 
Achilles: I'm sure that's accurate. Anteater: And yet, as I watched, reading the higher level instead of the lower level, I 
would see several dormant symbols being awakened, those which translate into the thought, "Oh, here's that charming Dr. Ant eater again-how pleasant!"-or words to 
that effect. 
Achilles: That sounds like what happened when th e four of us all found different levels to 
read in the MU-picture--or at least THREE of us did .. .

read in the MU-picture--or at least THREE of us did .. . 
Tortoise: What an astonishing coincidence that there should be such a resemblance 
between that strange picture which I chanced upon in the Well-Tempered Clavier, and the trend of our conversation. 
Achilles: Do you think it's just coincidence? Tortoise: Of course. 
Anteater: Well, I hope you can grasp now how the thoughts in Aunt Hillary emerge from 
the manipulation of symbols composed of signals composed of teams composed of 
lower-level teams, all the way down to ants.

lower-level teams, all the way down to ants. 
Achilles: Why do you call it "symbol manipula tion"? Who does the manipulating, if the 
symbols are themselves active? Who is the agent? 
Anteater: This gets back to the question which you earlier raised about purpose. You're 
right that symbols themselves are active, but the activities which they follow are nevertheless not absolutely free. The activitie s of all symbols are strictly determined 
by the state of the full system in which th ey reside. Therefore, the full system is

by the state of the full system in which th ey reside. Therefore, the full system is 
responsible for how its symbols trigger each other, and so it is quite reasonable to 
speak of the full system as the "agent". As the symbols operate, the state of the 
system gets slowly transformed, or updated. But there are many features which remain over time. It is this partially cons tant, partially varying system which is the 
agent. One can give a name to the

Ant Fugue 
 334full system. For example, Aunt Hillary is the "who" who can be said to manipulate 
her symbols; and you are similar, Achilles. 
Achilles: That's quite a strange characterizati on of the notion of who I am. I'm not sure I 
can fully understand it, but I will give it some thought. 
Tortoise: It would be quite interesting to follow the symbols in your brain as you do that 
thinking about the symbols in your brain. 
Achilles: That's too complicated for me. I have trouble enough just trying to picture how 
it is possible to look at an ant colony and r ead it on the symbol level. I can certainly 
imagine perceiving it at the ant level; and with a little trouble, I can imagine what it must be like to perceive it at the signal le vel; but what in the world can it be like to 
perceive an ant colony at the symbol level? 
Anteater: One only learns thr ough long practice. But when one is at my stage, one reads 
the top level of an ant colony as easily as you yourself read the "MU" in the MU-
picture.

picture. 
Achilles: Really? That must be an amazing experience. Anteater: In a way-but it is also one which is quite familiar to you, Achilles. Achilles: Familiar to me? What do you mean? I have never looked at an ant colony on 
anything but the ant level. 
Anteater: Maybe not; but ant colonies are no different from brains in many respects. Achilles: I have never seen nor read any brain either, however. Anteater: What about your OWN brain? Ar en't you aware of your own thoughts? Isn't

picture. 
Anteater: Maybe not; but ant colonies are no different from brains in many respects. Achilles: I have never seen nor read any brain either, however. Anteater: What about your OWN brain? Ar en't you aware of your own thoughts? Isn't 
that the essence of consciousness? What  else are you doing but reading your own 
brain directly at the symbol level? 
Achilles: I never thought of it that way. You mean that I bypass all the lower levels, and 
only see the topmost level? 
Anteater: That's the way it is, with conscious  systems. They perceive themselves on the 
symbol level only, and have no awareness of  the lower levels, such as the signal 
levels. 
Achilles: Does it follow that in a brain, th ere are active symbols which are constantly

levels. 
Achilles: Does it follow that in a brain, th ere are active symbols which are constantly 
updating themselves so that they reflect the overall state of the brain itself, always on the symbol level? 
Anteater: Certainly. In any conscious system there are symbols which represent the brain 
state, and they are themselves part of the very brain state which they symbolize. For consciousness requires a large degree of self-consciousness. 
Achilles: That is a weird notion. It means th at although there is frantic activity occurring

Achilles: That is a weird notion. It means th at although there is frantic activity occurring 
in my brain at all times, I am only capable of registering that activity in one way-on 
the symbol level; and I am completely insens itive to the lower levels. It is like being 
able to read a Dickens novel by direct visu al perception, without ever having learned 
the letters of the alphabet. I can't imagine anything as weird as that really happening. 
Crab: But precisely that sort of thing can happen when you read “MU”,

Ant Fugue 
 335without perceiving the lower levels "HOLISM" and "REDUCTIONISM". 
Achilles: You're right-I bypassed the lower le vels, and saw only the top. I wonder if I'm 
missing all sorts of meaning on lower levels of my brain as well, by reading only the 
symbol level. It's too bad that the top le vel doesn't contain all the information about 
the bottom level, so that by reading the t op, one also learns what the bottom level 
says. But I guess it would be naive to hope that the top level encodes anything from 
the bottom level-it probably doesn't percolat e up. The MU-picture is the most striking 
possible example of that: there, the t opmost level says only "ML which bears no 
relation whatever to the lower levels! 
Crab: That's absolutely true. ( Picks up the MU-picture, to inspect it more closely .) Hmm 
... There's something strange about the smallest  letters in this picture; they're very 
wiggly ... ' 
Anteater: Let me take a look. ( Peers closely at the MU-picture .) I think there's yet

Anteater: Let me take a look. ( Peers closely at the MU-picture .) I think there's yet 
another level, which all of us missed! 
Tortoise: Speak for yourself, Dr. Anteater. Achilles: Oh, no-that can't be! Let me see. ( Looks very carefully .) I know the rest of you 
won't believe this, but the message of this pi cture is staring us all in the face, hidden 
in its depths. It is simply one word, re peated over and over again, like a mantra-but 
what an important one: "Mu"! What do you know ! It is the same as the top level! 
And none of us suspected it in the least.

And none of us suspected it in the least. 
Crab: We would never have noticed it if it hadn't been for you, Achilles. Anteater: I 
wonder if the coincidence of the highest and lowest levels happened by chance? Or 
was it a purposeful act carried out by some creator? 
Crab: How could one ever decide that? Tortoise: I don't see any way to do so, since we have no idea why that particular picture is 
in the Crab's edition of the Well-Tempered Clavier. Anteater: Although we have been 
having a lively discussion, I have still mana ged to listen with a good fraction of an

having a lively discussion, I have still mana ged to listen with a good fraction of an 
ear to this very long and complex four-voice fugue. It is extraordinarily beautiful. 
Tortoise: It certainly is. And now, in just a moment, comes an organ point. Achilles: Isn't an organ point what happens  when a piece of music slows down slightly, 
settles for a moment or two on a single not e or chord, and then resumes at normal 
speed after a short silence? 
Tortoise: No, you're thinking of a "fermata"- a sort of musical semicolon. Did you notice 
there was one of those in the prelude?

there was one of those in the prelude? 
Achilles: I guess I must have missed it. Tortoise: Well, you have another chance coming up to hear a fermata-in fact, there are a 
couple of them coming up, towards the end of this fugue. 
Achilles: Oh, good. You'll point them out in advance, won't you? Tortoise: If you like. Achilles: But do tell me, what is an organ point? Tortoise: An organ point is the sustaining of a single note by one of the

Ant Fugue 
 336voices in a polyphonic piece (often the lowest  voice), while the other voices continue 
their own independent lines, This organ point  is on the note of G. Listen carefully, 
and you'll hear it. 
Anteater:. There occurred an incident one day when I visited with Aunt Hillary which 
reminds me of your suggestion of observing the symbols in Achilles' brain as they 
create thoughts which are about themselves.  
Crab: Do tell us about it. Anteater: Aunt Hillary had been feeling very lonely, and was very happy to have 
someone to talk to that day. So she gratefully  told me to help myself to the juiciest 
ants I could find. (She's always been most generous with her ants.) 
Achilles: Gee! Anteater: It just happened that I had been watching the symbols which were carrying out 
her thoughts, because in them were some particularly juicy-looking ants. 
Achilles: Gee! Anteater: So I helped myself to a few of th e fattest ants which had been parts of the

Achilles: Gee! Anteater: So I helped myself to a few of th e fattest ants which had been parts of the 
higher-level symbols which I had been read ing. Specifically, the symbols which they 
were part of were the ones which had expr essed the thought, "Help yourself to any of 
the ants which look appetizing." 
Achilles: Gee! Anteater: Unfortunately for them, but fortuna tely for me, the little bugs didn't have the 
slightest inkling of what they were collectively telling me, on the symbol level. 
Achilles: Gee! That is an amazing wraparound. They were completely unconscious of 
what they were participating in. Their acts could be seen as part of a pattern on a

what they were participating in. Their acts could be seen as part of a pattern on a 
higher level, but of course they were comp letely unaware of that. Ah, what a pity-a 
supreme irony, in fact-that they missed it. 
Crab: You are right, Mr. T-that was a lovely organ point. 
Anteater: I had never heard one before, but th at one was so conspicuous that no one could 
miss it. Very effective. 
Achilles: What? Has the organ point already o ccurred? How can I not have noticed it, if it 
was so blatant? 
Tortoise: Perhaps you were so wrapped up in what you were saying that you were

Tortoise: Perhaps you were so wrapped up in what you were saying that you were 
completely unaware of it. Ah, what a pity-a supreme irony, in fact-that you missed it. 
Crab: Tell me, does Aunt Hillary live in an anthill? Anteater: Well, she owns a rather large piece of property. It used to belong to someone 
else, but that is rather a sad story. In any cas e, her estate is quite expansive. She lives 
rather sumptuously, compared to many other colonies. 
!chilies: How does that jibe with the comm unistic nature of ant colonies which you

!chilies: How does that jibe with the comm unistic nature of ant colonies which you 
earlier described to us? It sounds quite inconsistent, to me, to preach communism and to live in a fancy estate.

Ant Fugue 
 337Anteater: The communism is on the ant level.  In an ant colony all ants work for the 
common good, even to their own individual detr iment at times. Now this is simply a 
built-in aspect of Aunt Hillary's structur e, but for all I know, she may not even be 
aware of this internal communism. Most human beings are not aware of anything 
about their neurons; in fact they probabl y are quite content not to know anything 
about their brains, being somewhat squ eamish creatures. Aunt Hillary is also 
somewhat squeamish; she gets rather antsy whenever she starts to think about ants at all. So she avoids thinking about them whenever possible. I truly doubt that she 
knows anything about the communistic society which is built into her very structure. 
She herself is a staunch believer in liberta rianism-you know, laissez-faire and all that. 
So it makes perfect sense, to me at least,  that she should live in a rather sumptuous 
manor.

manor. 
 
Tortoise: As I turned the page just now, wh ile following along in this lovely edition of 
the Well-Tempered Clavier, I noticed that the first of the two fermatas is coming up 
soon-so you might listen for it, Achilles. Achilles: I will, I will. 
Tortoise: Also, there's a most curious picture facing this page. Crab: Another one? What 
next? 
Tortoise: See for yourself. (Passes the score over to the Crab.) Crab: Aha! It's just a few bunches of letters . Let's see-there are various numbers of the

next? 
Tortoise: See for yourself. (Passes the score over to the Crab.) Crab: Aha! It's just a few bunches of letters . Let's see-there are various numbers of the 
letters `J', 'S', `B', `m', `a', and 't'. It's strange, how the first three letters grow, and then 
the last three letters shrink again. Anteater: May I see it? 
Crab: Why, certainly. Anteater: Oh, by concentrating on details, you have utterly missed the big picture. In 
reality, this group of letters is `f', `e', `r' , 'A', `C', 'H', without any repetitions. First 
they get smaller, then they get bigger. Here, Achilles-what do you make of it?

they get smaller, then they get bigger. Here, Achilles-what do you make of it? 
Achilles: Let me see. Hmm. Well, I see it as  a set of upper-case letters which grow as you 
move to the right. 
Tortoise:  Do they spell anything?

Ant Fugue 
 338Achilles: Ah ... "J. S. BACH". Oh! I understand now. It's Bach's name! 
Tortoise: Strange that you should see it that way. I see it as a set of lower-case letters, 
shrinking as they move to the right, and ... spelling out ... the name of ... (Slows down 
slightly, especialh drawing out the last fe w words. Then there is a brief silence. 
Suddenly he resumes as if nothing unusual had happened.) -"fermat". 
Achilles: Oh, you've got Fermat on the br ain, I do believe. You see Fermat's Last 
Theorem everywhere. 
Anteater: You were right, Mr. Tortoise-I just heard a charming little fermata in the fugue. Crab: So did I. Achilles: Do you mean everybody heard it but me? I'm beginning to feel stupid. Tortoise: There, there, Achilles-don't feel  bad. I'm sure you won't miss Fugue's Last 
Fermata (which is coming up quite soon). But,  to return to our previous topic, Dr. 
Anteater, what is the very sad story wh ich you alluded to, concerning the former 
owner of Aunt Hillary's property

owner of Aunt Hillary's property 
Anteater: The former owner was an extraordinary individual, one of the most creative ant 
colonies who ever lived. His name was J ohant Sebastiant Fermant, and he was a 
mathematiciant by vocation, but a musiciant by avocation. 
Achilles: How very versantile of him! Anteater: At the height of his creative power s, he met with a most untimely demise. One 
day, a very hot summer day, he was out soaking up the warmth, when a freak 
thundershower-the kind that hits only once every hundred years or so-appeared from

thundershower-the kind that hits only once every hundred years or so-appeared from 
out of the blue, and thoroughly drenched J. S F. Since the storm came utterly without 
warning, the ants got completely disoriente d and confused. The intricate organization 
which had been so finely built up over decades , all went down the drain in a matter of 
minutes. It was tragic. 
Achilles: Do you mean that all the ants dr owned, which obviously would spell the end of 
poor J. S. F. 
Anteater: Actually, no. The ants managed to survive, every last one of them, by crawling 
onto various sticks and logs which floate d above the raging torrents. But when the

onto various sticks and logs which floate d above the raging torrents. But when the 
waters receded and left the ants back on their home grounds, there was no 
organization left. The caste distributi on was utterly destroyed, and the ants 
themselves had no ability to reconstruct wh at had once before been such a finely 
tuned organization. They were as helpless as  the pieces of Humpty Dumpty in putting 
themselves back together again. I myself tried, like all the king's horses and all the 
king's men, to put poor Fermant together ag ain. I faithfully put out sugar and cheese,

king's men, to put poor Fermant together ag ain. I faithfully put out sugar and cheese, 
hoping against hope that somehow Fermant would reappear ... ( Pulls out a 
handkerchief and wipes his eyes.) 
Achilles: How valiant of you! I never knew Anteaters had such big hearts. 
Anteater: But it was all to no avail. He was Bone, beyond reconstitution.

Ant Fugue 
 339However, something very strange then began to take place: over the next few 
months, the ants which had been component s of J. S. F. slowly regrouped, and built 
up a new organization. And thus was Aunt Hillary born. 
Crab: Remarkable! Aunt Hillary is composed of the very same ants as Fermant was Anteater: Well, originally she was, yes. By now, some of the older ants have died, and 
been replaced. But there are still many holdovers from the J. S. F.-days. 
Crab: And can't you recognize some of J. S. F. 's old traits coming to the fore, from time 
to time, in Aunt Hillary% 
Anteater: Not a one. They have nothing in common. And there is no reason they should, 
as I see it. There are, after all, often several distinct ways to rearrange a group of 
parts to form a "sum". And Aunt Hillary wa s just a new "sum" of the old parts. Not 
MORE than the sum, mind you just that particular KIND of sum. 
Tortoise: Speaking of sums, I am reminded of number theory, where occasionally one

Tortoise: Speaking of sums, I am reminded of number theory, where occasionally one 
will be able to take apart a theorem into its component symbols, rearrange them in a 
new order, and come up with a new theorem. 
Anteater: I've never heard of such a phe nomenon, although I confess to being a total 
ignoramus in the field. 
Achilles: Nor have I heard of it-and I am rather well versed in the field, If I don't say so 
myself. I suspect Mr. T is just setting up one of his elaborate spoofs. I know him 
pretty well by now. Anteater: Speaking of num ber theory, I am reminded of J. S. F.

pretty well by now. Anteater: Speaking of num ber theory, I am reminded of J. S. F. 
again, for number theory is one of the domains  in which he excelled. In fact, he made 
some rather rema, ..able contributions to  number theory. Aunt Hillary, on the other 
hand, is remarkably dull-witted in anything that has even the remotest connection 
with mathematics. Also, she has only a rather banal taste in music, whereas 
Sebastiant was extremely gifted in music. 
Achilles: I am very fond of number theory. Could you possibly relate to us something of 
the nature of Sebastiant's contributions,

the nature of Sebastiant's contributions, 
Anteater: Very well, then. (Pauses for a moment to sip his tea, then resumes.) 
Have you heard of Fourmi's infamous "Well-Tested Conjecture”? 
Achilles. I'm not sure ... It sounds strangely familiar, and yet I can't quite place it. Anteater: It's a very simple idea. Lierre de Fourmi, a mathematiciant by vocation but 
lawyer by avocation, had been reading in his copy-of the classic text Arithmetica by 
Di of Antus, and came across a page containing the equation 
2a+2b=2c

2a+2b=2c 
He immediately realized that this equation has infinitely many solutions a. b, c, and 
then wrote in the margin the following notorious comment:

Ant Fugue 
 340
 
FIGURE 63. During emigrations arm' ants sometimes create living bridges of their 
own bodies. In this photograph of such a bri dge (de Fourmi Lierre), the workers of 
an Eciton burchelli colony can be seen lin king their legs and, along the top of the 
bridge, hooking their tarsal claws together to  form irregular systems of chains. .A 
symbiotic silverfish, Trichatelura manni, is seen crossing the bridge in the center. 
[From E. O. Wilson, The Insect Societies 'Cambridge, Mass.: Harvard University 
Press, 1971), p. 62) 
 
The equation 
na+nb=nc 
 has solutions in positive integers a, b, c, and n only when n = 2 (and then there are infinitely many triplets a, b, c which satis fy the equation); but there are no solutions 
for n > 2. I have discovered a truly marvelous proof of this statement, which, unfortunately. is so small that it would be well-nigh invisible if written in the margin. Ever since that year, some three hundred da ys ago, mathematiciants have been vainly

na+nb=nc 
for n > 2. I have discovered a truly marvelous proof of this statement, which, unfortunately. is so small that it would be well-nigh invisible if written in the margin. Ever since that year, some three hundred da ys ago, mathematiciants have been vainly 
trying to do one of two things: either to prove Fourmi's claim, and thereby vindicate 
Fourmi's reputation, which, although very high, has been somewhat tarnished by 
skeptics who think he never really found the proof he claimed to have found-or else 
to refute the claim, by finding a counterexampl e: a set of four integers a, b, c, and n, 
with n > 2, which satisfy the equation. Until very recently, every attempt in either 
direction had met with failure. To be sure , the Conjecture has been verified for many

direction had met with failure. To be sure , the Conjecture has been verified for many 
specific values of n-in particular, a ll n up to 125,000. But no one had succeeded in 
proving it for ALL n-no one, that is, until Johant Sebastiant Fermant came upon the 
scene. It was he who found the proof that cleared Fourmi's name.

Ant Fugue 
 341It now goes under the name "Johant Sebastiant's Well-Tested Conjecture". 
Achilles: Shouldn't it be called a "Theorem" rath er than a "Conjecture", if it's finally been 
given a proper proof; 
Anteater: Strictly speaking, you're right, but tradition has kept it this way. Tortoise: What sort of music did Sebastiant do? Anteater: He had great gifts for composition. Unfortunately, his greatest work is shrouded 
in mystery, for he never reached the point of publishing it. Some believe that he had it all in his mind; others are more unkind, sa ying that he probably never worked it out 
at all, but merely blustered about it. 
Achilles: What was the nature of this magnum opus? Anteater: It was to be a giant prelude and fugue; the fugue was to have 
twenty-four voices, and to involve twenty-four distinct subjects, one in each of the major and minor keys. 
Achilles: It would certainly be hard to listen to a twenty-four-voice fugue 
as a whole! 
Crab: Not to mention composing one!

as a whole! 
Crab: Not to mention composing one! 
Anteater: But all that we know of it is Sebastia nt's description of it, which he wrote in the 
margin of his copy of Buxtehude's Preludes and Fugues for Organ. The last words which 
he wrote before his tragic demise were: 
 
I have composed a truly marvel ous fugue. In it, I have added 
together the power of 24 keys, and the power of 24 themes; I 
came up with a fugue with the pow er of 24 voices. Unfortunately, 
this margin is too narrow to contain it. 
 
And the unrealized masterpiece simply goes by the name, "Fermant's Last Fugue". 
Achilles: Oh, that is unbearably tragic.

Achilles: Oh, that is unbearably tragic. 
Tortoise: Speaking of fugues, th is fugue which we have been listening to is nearly over. 
Towards the end, there occurs a strange new twist on its theme. (Flips the page in the 
Well-Tempered Clavier.) Well, what have we here? A new illustration-how 
appealing! (Shows it to the Crab .)

Ant Fugue 
 342Crab: Well, what have we here? Oh, I see: It´s HOLISMIONSIM”, written in large letters 
that first shrink and then grow back to thei r original size. But that doesn't make any 
sense, because it's not a word. Oh me, oh my! (Passes it to the Anteater.) 
Anteater: Well, what have we here? Oh, I see: it's "REDUCTHOLISM", written in small 
letters that first grow and then shrink back to  their original size. But that doesn't make 
any sense, because it's not a word. Oh my, oh me! (Passes it to Achilles.) 
Achilles: I know the rest of you won't believe this , but in fact this picture consists of the 
word "HOLISM" written twice, with the lette rs continually shrinking as they proceed 
from left to right. (Returns it to the Tortoise.) 
Tortoise: I know the rest of you won't believe this , but in fact this picture consists of the 
word "REDUCTIONISM" written once, with th e letters continually growing as they 
proceed from left to right.

proceed from left to right. 
Achilles: At last-I heard the new twist on th e theme this time! I am so glad that you 
pointed it out to me, Mr. Tortoise. Finally, I think I am beginning to grasp the art of 
listening to fugues.

Brains and Thoughts 
 343Brains and Thoughts 
 
New Perspectives on Thought 
 
IT WAS ONLY with the advent of comput ers that people actually tried to create 
"thinking" machines, and witnessed bizarre va riations on the theme, of thought. Programs 
were devised whose "thinking" was to human  thinking as a slinky flipping end over end 
down a staircase is to human locomotion. All of a sudden the idiosyncrasies, the 
weaknesses and powers, the vagaries and vici ssitudes of human thought were hinted at by 
the newfound ability to experiment with a lien, yet hand-tailored forms of thought-or 
approximations of thought. As a result, we have acquired, in the last twenty years or so, a 
new kind of perspective on what thought is , and what it is not. Meanwhile, brain 
researchers have found out much about the sm all-scale and large-scale hardware of the 
brain. This approach has not yet been ab le to shed much light on how the brain

brain. This approach has not yet been ab le to shed much light on how the brain 
manipulates concepts, but it gives us some  ideas about the biological mechanisms on 
which thought manipulation rests. In the coming two Chapters, then, we will try to unite some insights gleaned from 
attempts at computer intelligence with some of the facts learned from ingenious experiments on living animal brains, as well as with results from research on human 
thought processes done by cognitive psychologists. The stage has been set by the Prelude, Ant Fugue ; now we develop the ideas more deeply. 
 
Intensionality and Extensionality

Intensionality and Extensionality 
 Thought must depend on representing reality in the hardware of the brain. In the 
preceding Chapters, we have developed form al systems which represent domains of 
mathematical reality in their symbolisms. To  what extent is it reasonable to use such 
formal systems as models for how the brain might manipulate ideas? We saw, in the pq-system and then in othe r more complicated systems, how meaning, in

formal systems as models for how the brain might manipulate ideas? We saw, in the pq-system and then in othe r more complicated systems, how meaning, in 
a limited sense of the term, arose as a result of an isomorphism which maps typographical symbols onto numbers, operations, and relati ons; and strings of typographical symbols 
onto statements. Now in the brain we don't have typographical symbols, but we have 
something even better: active elements which can store information and transmit it and receive it from other active elements. Thus we  have active symbols, rather than passive 
typographical symbols. In the brain, the rules

 344re mixed right in with the symbols themselv es, whereas on paper, the symbols are static 
entities, and the rules are in our heads. It is important not to get the idea, from the rath er strict nature of all ie formal systems we 
have seen, that the isomorphism between sym bols and real things is a rigid, one-to-one 
mapping, like the strings which link a marione tte and the hand guiding it. In TNT, the 
notion "fifty" can be expressed in different symbolic ways; for example,  
((SSSSSSSO.SSSSSSSO)+(SO-SO)) 
 ((SSSSSO•SSSSSO)+(SSSSSO.SSSSSO)) 
 
'hat these both represent the same number is  not a priori clear. You can manipulate each 
expression independently, and at some point  stumble cross a theorem which makes you 
exclaim, "Oh-it's that number!" In your mind, you can also have different ment al descriptions for a single person; for 
example,

example,  
The person whose book I sent to a friend in Poland a while back.  The stranger who started talking with me and my friends tonight in this coffee house. 
 :'hat they both represent the same person is not a priori clear. Both descriptions may sit in 
your mind, unconnected. At some point duri ng the evening you may stumble across a 
topic of conversation which leads to the reve lation that they designate the same person, 
making you exclaim, Oh-you're that person!" Not all descriptions of a person need be att ached to some central symbol for that person,

:'hat they both represent the same person is not a priori clear. Both descriptions may sit in 
making you exclaim, Oh-you're that person!" Not all descriptions of a person need be att ached to some central symbol for that person, 
which stores the person's name. Descriptions  can be manufactured and manipulated in 
themselves. We can invent nonexistent peopl e by making descriptions of them; we can 
merge two descriptions 'hen we find they represent a single entity; we can split one description into two when we find it represents two things, not one-and so on. This calculus of descriptions" is at the heart of thinking. It is said to be intentional and not 
extensional, which means that descriptions can "float" without Being anchored down to 
specific, known objects. The intensionality of  thought is connected to its flexibility; it

specific, known objects. The intensionality of  thought is connected to its flexibility; it 
gives us the ability to imagine hypothetical worl ds, to amalgamate different descriptions 
or chop one description into separate pieces, and so on. Suppose a friend who has borrowed your car telephones you to say hat your car skidded 
off a wet mountain road, careened agains t a bank, .nd overturned, and she narrowly 
escaped death. You conjure up a series & im ages in your mind, which get progressively

escaped death. You conjure up a series & im ages in your mind, which get progressively 
more vivid as she adds details, and in the end you "see it all in your mind's eye". Then she tells you hat it's all been an April Fool's  joke, and both she and the car are fine! In 
many ways that is irrelevant. The story and the images lose nothing of their vividness, 
and the memory will stay with you for a l ong, long time. Later, you may even think of 
her as an unsafe driver because of the strength of

 345the first impression, which should have been wiped out when you learned it was all 
untrue. Fantasy and fact intermingle very closely in our minds, and this is because 
thinking involves the manufacture and manipula tion of complex descriptions, which need 
in no way be tied down to real events or things. A flexible, intensional representation of the world is what thinking is all about. Now how 
can a physiological system such as the brain support such a system?  
The Brain's "Ants" 
 The most important cells in the brain are ne rve cells, or neurons (see Fig. 65), of which 
there are about ten billion. (Curiously, outnum bering the neurons by about ten to one are 
the glial cells, or glia. Glia are believed to play more of a supporting role to the neurons' 
starring role, and therefore we will not discuss them.) Each neuron possesses a number of synapses ("entry ports") and one axon ("out put channel"). The input and output are

starring role, and therefore we will not discuss them.) Each neuron possesses a number of synapses ("entry ports") and one axon ("out put channel"). The input and output are 
electrochemical flows: that is, moving ions. In  between the entry ports of a neuron and its 
output channel is its cell body, where "decisions" are made.  
 
FIGURE 65. Schematic drawing of a neuron. [Adapted From D. Wooldridge, The 
Machinery of the Brain (New York:"- McGraw-Hill, 1963), p. 6.

 346The type of decision which a neuron faces-and this can take place up to a thousand times 
per second-is this: whether or not to fire -that is, to ease ions down its axon, which -
eventually will cross over into  the entry its of one or more other neurons, thus causing 
them to make the same sort of decision. The d ecision is made in a very simple manner: if 
the sum all inputs exceeds a certain threshold, yes; otherwise, no. Some of the inputs can 
be negative inputs, which cancel out positive inputs coming from somewhere else. In any 
case, it is simple addition which rules the lowe st 'el of the mind. To paraphrase Descartes' 
famous remark, "I think, therefore I sum" (from the Latin Cogito, ergo am). 
Now although the manner of making the deci sion sounds very simple, here is one 
fact which complicates the issue: there may be as many as 200,000 separate entry ports to 
a neuron, which means that up to 200,000 Ka rate summands may be involved in

a neuron, which means that up to 200,000 Ka rate summands may be involved in 
determining the neuron's next ion. Once the decision has been made, a pulse of ions 
streaks down the on towards its terminal end. Before the ions reach the end, however, ey may encounter a bifurcation-or several. In such cases, the single output pulse splits up as 
it moves down the bifurcating axon, and by th e tine it has reached the end, "it" has

it moves down the bifurcating axon, and by th e tine it has reached the end, "it" has 
become "they"-and they may reach their destinations at separate times, since the axon branches along which they travel may be of different lengths and have different 
resistivities. The important thing, though, is that they all began as one single pulse, 
moving 'ay from the cell body. After a neuron fi res, it needs a short recovery time fore 
firing again; characteristically this is measur ed in milliseconds, so at a neuron may fire up 
to about a thousand times per second. 
 
Larger Structures in the Brain

Larger Structures in the Brain 
 Now we have described the brain's "ants". Wh at about "teams", or "signals"? What about 
"symbols"? We make the following observation:  despite e complexity of its input, a 
single neuron can respond only in a very prim itive way-by firing, or not firing. This is a 
very small amount of Formation. Certainly for large amounts of information to be carried 
or processed, many neurons must be involve d. And therefore one might guess at larger 
structures, composed from many neurons, would exist, which handle concepts on a 
higher level. This is undoubtedly true, but the most naive assumption-that there is a fixed

higher level. This is undoubtedly true, but the most naive assumption-that there is a fixed 
group of neurons for each different concept-is almost certainly false. 
There are many anatomical portions of the brain which can be distinguished from 
each other, such as the cerebrum, the cer ebellum, the hypothalamus (see Fig. 66). The 
cerebrum is the largest part of the human am, and is divided into a left hemisphere and a 
right hemisphere. The outer few millimeters of  each cerebral hemisphere are coated with 
a layered "bark", or cerebral cortex. The amount of cerebral cortex is the major

a layered "bark", or cerebral cortex. The amount of cerebral cortex is the major 
distinguishing feature, in terms of anatomy, between human brains and brains of less intelligent species. We will not describe any of the brain's suborgans in detail because, as it turns out, only the roughest mapping can

 347
 
FIGURE 66. The human brain, seen fr om the left side. It is strange that the visual area is 
in the back of the head. [From Steven Rose, The Conscious Brain, updated ed. (New 
York: Vintage, 1966), p. 50. ]  at this time be made between such large-scale suborgans and the activities, mental or physical, which they are responsible for. For instance, it is known that language is primarily handled in one of the two cerebra l hemispheres-in fact, usually the left 
hemisphere. Also, the cerebellum is the place where trains of impulses are sent off to 
muscles to control motor activity. But how th ese areas carry out their functions is still 
largely a mystery.  
Mappings between Brains 
 Now an extremely important question comes up here. If thinking does take place in the 
brain, then how are two brains different from each other? How is my brain different from 
yours? Certainly you do not think exactly as I do, nor as anyone else does. But we all

yours? Certainly you do not think exactly as I do, nor as anyone else does. But we all 
have the same anatomical divisions in our brains. How far does this identity of brains 
extend? Does it go to the neural level? Ye s, if you look at animals on a low enough level 
of the thinking-hierarchy-the lowly earthworm , for instance. The following quote is from 
the neurophysiologist, David Hubel, speaki ng at a conference on communication with 
extraterrestrial intelligence:  
The number of nerve cells in an anim al like a worm would be measured, I

The number of nerve cells in an anim al like a worm would be measured, I 
suppose, in the thousands. One very intere sting thing is that we may point to a 
particular individual cell in a particular earthworm, and then identify the same cell, the corresponding cell in another earthworm of the same species.'

 348Earthworms have isomorphic brains! One could say, "There is only one earthworm." 
But such one-to-one mappability between indi viduals' brains disappears very soon as 
you ascend in the thinking-hierarchy and th e number of neurons increases-confirming 
one's suspicions that there is not just one pa n! Yet considerable physical similarity can be 
detected between different human brains when they are compared on a scale larger than a 
;le neuron but smaller than the major suborgans  of the brain. What s this imply about how 
individual mental differences are represente d in physical brain? If we looked at my 
neurons' interconnections, could we l various st ructures that could be identified as coding 
for specific things -tow, specific beliefs I have , specific hopes, fears, likes and dislikes I 
harbor? If mental experiences can be attr ibuted to the brain, can knowledge and other 
aspects of mental life likewise be traced to sp ecific locations de the brain, or to specific

aspects of mental life likewise be traced to sp ecific locations de the brain, or to specific 
physical subsystems of the brain? This will be a central question to which we will often 
return in this Chapter and the next. 
 
Localization of Brain Processes: An Enigma 
 
In an attempt to answer this question, the neurologist Karl Lashley, in a series of 
experiments beginning around 1920 and running fo r many ,s, tried to discover where in 
its brain a rat stores its knowledge about :e running. In his book The Conscious Brain, 
Steven Rose describes Lashley's trials and tribulations this way:

Steven Rose describes Lashley's trials and tribulations this way:  
Lashley was attempting to identify the locus of  memory within the cortex, and, to do so, 
first trained rats to run mazes, and then rem oved various cortical regions. He allowed the 
animals to recover and tested the retenti on of the maze-running skills. To his surprise it 
was not possible to find a particular region corresponding to the ability to remember the way through a maze. instead all the rats whic h had had cortex regions removed suffered

was not possible to find a particular region corresponding to the ability to remember the way through a maze. instead all the rats whic h had had cortex regions removed suffered 
some kind f impairment, and the extent of the impairment was roughly proportional to the 
amount of cortex taken off. Removing cort ex damaged the motor and sensory capacities 
of the animals, and they would limp, hop, roll, or stagger, but somehow they always managed to traverse the maze. So far as memory 'as concerned, the cortex appeared to be 
equipotential, that is, with all regions of equal possible utility. Indeed, Lashley concluded

equipotential, that is, with all regions of equal possible utility. Indeed, Lashley concluded 
rather gloomily in is last paper "In Search of the Engram", which appeared in 1950, that the only conclusion was that memory was not possible at all.' 
 
Curiously, evidence for the opposite point of view was being developed :in Canada at 
roughly the same time that Lashley was doi ng his last work, in late 1940's. The 
neurosurgeon Wilder Penfield was examining the reactions of patients whose brains had 
been operated on, by inserting electrodes into various parts of their exposed brains, and

been operated on, by inserting electrodes into various parts of their exposed brains, and 
then using small electrical pulses to stim ulate the neuron or neurons to which the 
electrodes been attached. These pulses were  similar to the pulses which come other 
neurons. What Penfield found was that stimulation of certain

 349neurons would reliably create specific images or sensations in the patient. These 
artificially provoked impressions ranged from strange but indefinable fears to buzzes and colors, and, most impressively of all, to en tire successions of events recalled from some 
earlier time of life, such as a childhood birthda y party. The set of locations which could 
trigger such specific events was extremel y small-basically centered upon a single neuron. 
Now these results of Penfield dramatically oppose the conclusions of Lashley, since they 
seem to imply that local areas are responsible for specific memories, after all. 
What can one make of this? One possible expl anation could be that memories are coded 
locally, but over and over again in different  areas of the cortex-a strategy perhaps 
developed in evolution as security against possible loss of cortex in fights, or in 
experiments conducted by neurophysiologists.  Another explanation would be that

experiments conducted by neurophysiologists.  Another explanation would be that 
memories can be reconstructed from dynamic processes spread over the whole brain, but 
can be triggered from local spots. This theo ry is based on the notion of modern telephone 
networks, where the routing of a long-distance call is not predictable in advance, for it is selected at the time the call is placed, and depends on the situation all over the whole 
country. Destroying any local part of the ne twork would not block calls; it would just

country. Destroying any local part of the ne twork would not block calls; it would just 
cause them to be routed around the damaged area. In this sense any call is potentially 
nonlocalizable. Yet any call just connects up two specific points; in this sense any call is 
localizable. 
 
Specificity in Visual Processing 
 
Some of the most interesting and significant work on localization of brain processes has 
been done in the last fifteen years by David Hubel and Torsten Wiesel, at Harvard. They 
have mapped out visual pathways in the brains  of cats, starting with the neurons in the

have mapped out visual pathways in the brains  of cats, starting with the neurons in the 
retina, following their connections towards the rear of the head, passing through the 
"relay station" of the lateral geniculate, a nd ending up in the visual cortex, at the very 
back of the brain. First of all, it is rema rkable that there exist well defined neural 
pathways, in light of Lashley's results. But more remarkable are the properties of the 
neurons located at different stages along the pathway. 
It turns out that retinal neurons are primar ily contrast sensors. More specifically, the

It turns out that retinal neurons are primar ily contrast sensors. More specifically, the 
way they act is this. Each retinal neuron is normally firing at a "cruising speed". When its 
portion of the retina is struck by light, it may either fire faster or  slow down and even 
stop firing. However, it will do so only provided that the surrounding part of the retina is 
less illuminated. So this means that there ar e two types of neuron: ."on-center", and "off-
center". The on-center neurons are those whos e firing rate increases whenever, in the

center". The on-center neurons are those whos e firing rate increases whenever, in the 
small circular retinal area to which they are sensitive, the center is bright but the outskirts are dark; the off-center neurons are those which fire faster when there is darkness in the 
center and brightness in the

 350outer ring. If an on-center pattern is shown to an off-center neuron, the neuron will slow 
down in firing (and vice versa). Uniform illu mination will .leave both types of retinal 
neuron unaffected; they will continue to fire at cruising speed. 
From the retina, signals from these neurons proceed via the optic nerve to the lateral 
geniculate, located somewhere towards the middle of the brain. There, one can find a 
direct mapping of the retinal su rface in the .use that there are lateral-geniculate neurons 
which are triggered only by specific stimuli falli ng on specific areas of the retina. In that 
sense, the general geniculate is disappointing;  it seems to be only a "relay station", and 
not a further processor (although to give it its  due, the contrast sensitivity ,ms to be 
enhanced in the lateral geniculate). The retin al image is coded a straightforward way in 
the firing patterns of the neurons in the lateral geniculate, despite the fact that the neurons

the firing patterns of the neurons in the lateral geniculate, despite the fact that the neurons 
there are not arranged on a o-dimensional surface in the form of the retina, but in a three-
dimensional block. So two dimensions get mapped onto three, yet the formation is 
preserved: an isomorphism. There is probably some deep meaning to the change in the 
dimensionality of the representation, which is not yet fully appreciated. In any case, there 
are so many further unexplained stages of vi sion that we should not be disappointed but 
pleased the fact that-to some extent-we have figured out this one stage!

pleased the fact that-to some extent-we have figured out this one stage! 
From the lateral geniculate, the signals pro ceed back to the visual cortex. Here, some 
new types of processing occur. The cells of the visual cortex are divided into three 
categories: simple, complex, and hyper complex. Simple cells act very much like retinal cells or lateral geniculate [Is: they res pond to point-like light or dark spots with 
contrasting surrounds, in particular regions of the retina. Complex cells, by contrast, usually receive input from a hundred or more ot her cells, and they detect light dark bars

contrasting surrounds, in particular regions of the retina. Complex cells, by contrast, usually receive input from a hundred or more ot her cells, and they detect light dark bars 
oriented at specific angles on the retina (see Fig. 67). Hyper complex cells respond to 
corners, bars, or even "tongues" moving in sp ecific directions (again see Fig. 67). These 
latter cells are so highly specialized at they are sometimes called "higher-order hyper 
complex cells". 
 
A "Grandmother Cell"? 
 
Because of the discovery of cells in the visu al cortex which can be triggered stimuli of 
ever-increasing complexity, some people have wondered if things are not leading in the 
direction of "one cell, one concept"-for ample, you would have a "grandmother cell"

direction of "one cell, one concept"-for ample, you would have a "grandmother cell" 
which would fire if, and only if, your gra ndmother came into view. This somewhat 
humorous ample of a "superhypercomplex cell" is not taken very seriously. Rower, it is 
not obvious what alternative theory seems reasona ble. One possibility is that larger neural 
networks are excited collectively by sufficiently complex visual stimuli. Of course, the triggering of these larger multineuron un its would somehow have to come from

networks are excited collectively by sufficiently complex visual stimuli. Of course, the triggering of these larger multineuron un its would somehow have to come from 
integration of signals emanating from the many hyper complex cells. How this might be done nobody knows> Just when we seem to be approaching the threshold where

 351 
 
FIGURE 67. Responses to patterns by certain sample neurons.  
(a) This edge-detecting neuron looks for vertical edges with light on the left and dark on the right. The first column  shows how the orientation of an 
edge is relevant to this neuron. Th e second column shows how the position 
of the edge within the field is irrelevant, for this particular neuron. (b) Showing how a hyper complex cell responds more selectively: here, only when the descending tongue is in the middle of the field. (c) The responses 
of a hypothetical "grandmother cell" to  various random stimuli; the reader 
may enjoy pondering how an "octopus cell" would respond to the same stimuli. 
 
"symbol" might emerge from "signal", the trail gets lost-a tantalizingly unfinished story. 
We will return to this story shortly, however, and try to fill in some of it. 
Earlier I mentioned the coarse-grained isomorphism between all human brains which

Earlier I mentioned the coarse-grained isomorphism between all human brains which 
exists on a large anatomical scale, and the very fine-grained, neural-level isomorphism 
which exists between earthworm brains. It is  quite interesting that there is also an 
isomorphism between the visual processi ng apparatus of cat, monkey, and human, the 
"grain" of which is somewhere between coarse and fine. Here is how that isomorphism works. First of all, all three species have "ded icated" areas of cortex at the back of their 
brains where visual processing is done: the visual cortex. Secondly, in

 352each of them, the visual cortex breaks up into  three subregions, called areas 18, and 19 of 
the cortex. These areas are still universal, in the sense that y can be located in the brain of 
any normal individual in any of the three tie s. Within each area you can go still further, 
reaching the "columnar" organization of the visu al cortex. Perpendicular to the surface of 
the (ex, moving radially inwards towards the inner brain, visual neurons are inged in "columns"-that is, almost all connections m ove along the ial, columnar direction, and not 
between columns. And each column ps onto a small, specific retinal region. The number 
of columns is not same in each individual, so that one can't find "the same column". ally, 
within a column, there are layers in which simple neurons tend to found, and other layers 
in which complex neurons tend to be found. to hypercomplex neurons tend to be found in areas 18 and 19 predominately, while the simp le and complex ones are found mostly in

in which complex neurons tend to be found. to hypercomplex neurons tend to be found in areas 18 and 19 predominately, while the simp le and complex ones are found mostly in 
area 17.) appears that we run out  of isomorphisms at this level of detail. From here down 
to the individual neuron level, each individual cat, monkey, or man has a completely unique pattern-somewhat like a fingerprint or a signature. 
One minor but perhaps telling difference betw een visual processing in ;'brains and 
monkeys' brains has to do with the stage at which informal from the two eyes is integrated to yield a single combined higher-l evel 1al. It turns out that it takes place

monkeys' brains has to do with the stage at which informal from the two eyes is integrated to yield a single combined higher-l evel 1al. It turns out that it takes place 
slightly later in the monkey than in the cat, which gives each separate eye's signal a 
slightly longer time to get processed by itself.  This is not too surprising, since one would 
expect that higher a species lies in the in telligence hierarchy, the more complex will the 
problems which its visual system will be ca lled upon to handle; and before signals ought 
to pass through more and more early processing ore receiving a final "label". This is quite

to pass through more and more early processing ore receiving a final "label". This is quite 
dramatically confirmed by observations of th e visual abilities of a newborn calf, which 
seems to be born with as much power of visu al discrimination as it will ever have. It will 
shy away from people or dogs, but not from other cattle. Probably its entire visual system 
is "hard-wired" before birth, and involves re latively little optical processing. On the other 
hand, a human's visual system, so deeply ant on the cortex, takes several years to reach 
maturity. 
 
Funneling into Neural Modules

maturity. 
 
Funneling into Neural Modules 
 
A puzzling thing about the discoveries so fa r made about the organization the brain is 
that few direct correspondences have been found between  large-scale hardware and high-
level software. The visual cortex, for instance,  is a large-scale piece of hardware, which is 
entirely dedicated to a it software purpose-the processing of visual information-yet all of 
processing so far discovered is still quite lo w-level. Nothing approaching recognition of

maturity. 
 
processing so far discovered is still quite lo w-level. Nothing approaching recognition of 
objects has been localized in the visual co rtex. This means that no one knows where or 
how the output from complex and hypercomplex  cells gets transformed into conscious 
recognition of shapes,

 353rooms, pictures, faces, and so on. People have  looked for evidence of the "funneling" of 
many low-level neural responses into fewer and fewer higher-level ones, culminating in 
something such as the proverbial grandmothe r cell, or some kind of multineuron network, 
as mentioned above. It is evident that this  will not be found in some gross anatomical 
division of the brain, but rather in a more microscopic analysis. 
One possible alternative to the the grandmothe r cell might be a fixed set of neurons, say 
a few dozen, at the thin end of the "funnel", all of which fire when Granny comes into 
view. And for each different recognizable obj ect, there would be a unique network and a 
funneling process that would focus down onto that network. There are more complicated 
alternatives along similar lines, involving netw orks which can be excited in different 
manners, instead of in a fixed manner. Such  networks would be the "symbols" in our 
brains.

brains. 
But is such funneling necessary? Perhaps an object being looked at is implicitly 
identified by its "signature" in the visual cort ex-that is, the collected responses of simple, 
complex, and hypercomplex cells. Perhaps the brain does not need any further recognizer 
for a particular form. This theory, however, poses the following problem. Suppose you are looking at a scene. It regi sters its signature on your visual cortex; but then how do you 
get from that signature to a verbal descripti on of the scene? For instance, the paintings of

get from that signature to a verbal descripti on of the scene? For instance, the paintings of 
Edouard Vuillard, a French post-impressionist, often take a few seconds of scrutiny, and 
then suddenly a human figure will jump out at you. Presumably the signature gets imprinted on the visual cortex in the first fraction of a second-but the picture is only 
understood after a few seconds. This is but one example of what is actually a common phenomenon-a sensation of something "crysta llizing" in your mind at the moment of 
recognition, which takes place not when the light  rays hit your retina, but sometime later,

recognition, which takes place not when the light  rays hit your retina, but sometime later, 
after some part of your intelligence has had a chance to act on the retinal signals. 
The crystallization metaphor yields a pretty image derived from statistical mechanics, 
of a myriad microscopic and uncorrelated ac tivities in a medium, slowly producing local 
regions of coherence which spread and enlarg e; in the end, the myriad small events will 
have performed a complete structural reva mping of their medium from the bottom up, 
changing' it from a chaotic assembly of inde pendent elements into one large, coherent,

changing' it from a chaotic assembly of inde pendent elements into one large, coherent, 
fully linked structure. If one thinks of the early neural activities as independent, and of 
the end result of their many independent firings  as the triggering of a well-defined large 
"module" of neurons, then the word "crystallization" seems quite apt. 
Another argument for funneling is based on the fact that there are a myriad distinct 
scenes which can cause you to feel you have perceived the same object-for example, your 
grandmother, who may be smiling or frowning, w earing a hat or not, in a bright garden or

grandmother, who may be smiling or frowning, w earing a hat or not, in a bright garden or 
a dark train station, seen from near or far, from side or front, and so on. All these scenes 
produce extremely different signatures on the visu al cortex; yet all of them could prompt 
you to say "Hello, Granny." So a funneling process must take

 354place at some point after the reception of th e visual signature and before e words are 
uttered. One could claim that this funneling is  not part of the perception of Granny, but 
just part of verbalization. But it seems quite  unnatural to partition the process that way, 
for you could internally use the formation th at it is Granny without verbalizing it. It 
would be very it unwieldy to handle all of th e information in the entire visual cortex, 
when much of it could be thrown away, since you don't care about where shadows fall or how many buttons there are on her blouse, etc. 
Another difficulty with a non-funneling theory is to explain how there in be different 
interpretations for a single signature-for exam ple, the Escher picture Convex a4 Concave 
(Fig. 23). Just as it seems obvious to us tat we do not merely perceive dots on a television 
screen, but chunks, likewise it seems ridiculous  to postulate that perception has taken

screen, but chunks, likewise it seems ridiculous  to postulate that perception has taken 
place hen a giant dot-like "signature" has been created on the visual cortex. here must be some funneling, whose end result is to trigge r some specific modules of neurons, each of 
which is associated with the concepts-the funks-in the scene. 
 
Modules Which Mediate Thought Processes 
 
Thus we are led to the conclusion that for each concept there is a fairly ell-defined 
module which can be triggered-a module that  consists of a nail group of neurons-a

module which can be triggered-a module that  consists of a nail group of neurons-a 
"neural complex" of the type suggested earlier.  problem with this theory-at least if it is 
taken naively-is that it would suggest that one should be able to locate such modules somewhere within to brain. This has not yet been done, and some evidence, such as the 
experiments by Lashley, points against localiza tion. However, it is still too early to tell. 
There may be many copies of each module spread around, or modules may overlap physically; both of these effects would tend to obscure any division of neurons into

There may be many copies of each module spread around, or modules may overlap physically; both of these effects would tend to obscure any division of neurons into 
"packets". Perhaps the complexes are like very  thin pancakes packed in layers which 
occasionally pass through each other; perhaps they are like long snakes which curl 
around each other, here and there flattening out , like cobras' heads; perhaps they are like 
spiderwebs; or perhaps they are circuits in which signals travel round id round in shapes 
stranger than the dash of a gnat-hungry swallow. here is no telling. It is even possible that 
these modules are software, ether than hard ware, phenomena-but this is something which 
we will discuss later

we will discuss later 
There are many questions that come to mind concerning these hypothesized neural 
complexes. For instance: 
 
Do they extend into the lower regions of the brain, such as the 
midbrain, the hypothalamus, etc.? 
Can a single neuron belong to more than one such complex?  To how many such complexes can a single neuron belong?  By how many neurons can such complexes overlap?

 355 
Are these complexes pretty much the same for everybody? Are corresponding ones found in corresponding places in different 
people's brains? 
Do they overlap in the same way in everybody's brain? 
 
Philosophically, the most important question of all is this: "hat w ould the existence of 
modules-for instance, a grandmother module-tell us? Would this give us any insight into 
the phenomenon of our own consciousness? Or w ould it still leave us as  much in the dark 
about what consciousness is, as does knowledge that a brain is built out of neurons and 
glia? As you might guess from reading the An t Fugue, my feeling is that it would go a 
long way towards giving us an understanding of the phenomenon of consciousness. The 
crucial step that needs to be taken is fr om a low-level-neuron-by-neuron-description of 
the state of a brain, to a high-level-module-by- module-description of the same state of the

the state of a brain, to a high-level-module-by- module-description of the same state of the 
same brain. Or, to revert to the suggestive te rminology of the Ant Fugue, we want to shift 
the description of the brain state from the signal level to the symbol, level. 
 
Active Symbols 
 
Let us from now on refer to these hypotheti cal neural complexes, neural modules, 
neural packets, neural networks, multineuron units-call them what you will, whether they 
come in the form of pancakes, garden rakes,  rattlesnakes, snowflakes, or even ripples on 
lakes-as symbols. A description of a brain stat e in terms of symbols was alluded to in the

lakes-as symbols. A description of a brain stat e in terms of symbols was alluded to in the 
Dialogue. What would such a description be lik e? What kinds of concepts is it reasonable 
to think actually might be "symbolized"? Wh at kinds of interrelations would symbols 
have? And what insights would this whol e picture provide into consciousness? 
The first thing to emphasize is that symbols can be either dormant, or awake (activated). 
An active symbol is one which has been triggered-that is, one in which a threshold number of neurons have been caused to fi re by stimuli coming from outside. Since a

An active symbol is one which has been triggered-that is, one in which a threshold number of neurons have been caused to fi re by stimuli coming from outside. Since a 
symbol can be triggered in many different wa ys, it can act in many different ways when 
awakened. This suggests that we should think of a symbol not as a fixed entity, but as a 
variable entity. Therefore it would not suffice to describe a brain state by saying "Symbols A, B, ..., N are all active"; rather, we  would have to supply in addition a set of 
parameters for each active symbol, characterizi ng some aspects of the symbol's internal 
workings. It is an interesting question whet her in each symbol there are certain core

workings. It is an interesting question whet her in each symbol there are certain core 
neurons, which invariably fire when the symbol is activated. If such a core set of neurons 
exists, we might refer to it as the "invariant core" of the symbol. It is tempting to assume that each time you think of, say, a waterfall, some fixed neural process is repeated, 
without doubt embellished in different ways  depending on the context, but reliably 
occurring. However, it is not clear that this must be so.

 356Now what does a symbol do, when awaken ed? A low-level description would say, 
"Many of its neurons fire." But this no l onger interests us. The high-level description 
should eliminate all reference to neurons, and concentrate exclusively on symbols. So a high-level description of what makes a sym bol active, as distinguished from dormant, 
would be, "It sends out messages, or signals , whose purpose is to try to awaken, or 
trigger, other symbols." Of course these me ssages would be carried as streams of nerve 
impulses, by neurons-but to the extent that  we can avoid such phraseology, we should, 
for it represents a low-level way of looking at  things, and we hope that we can get along 
on purely a high level. In other words, we hope  at thought processes can be thought of as 
being sealed off from neural events in the same  way that the behavior of a clock is sealed 
off from the laws of quantum mechanics, or the biology of cells is sealed off from the 
laws of quarks.

laws of quarks. 
But what is the advantage of this high-level picture? Why is it better to say, "Symbols A 
and B triggered symbol C" than to say, "Neurons 183 through 612 excited neuron 75 and 
caused it to fire"? This question was answered in the Ant Fugue: It is better because symbols symbolize things, and neurons don't. Symbols are the hardware realizations of 
concepts. Whereas group of neurons trigge ring another neuron corresponds to no outer 
event, the triggering of some symbol by othe r symbols bears a relation to events in the

event, the triggering of some symbol by othe r symbols bears a relation to events in the 
real world-or in an imaginary world. Sym bols are related to each other by the messages 
which they can send back and forth, in such a way that their triggering patterns are very 
much like the large-scale events rich do happe n in our world, or could happen in a world 
similar to ours. essence, meaning arises here  for the same reason as it did in the -system-
isomorphism; only here, the isomorphism is in finitely more complex, subtle, delicate, 
versatile, and intensional. 
Incidentally, the requirement that symbols should be able to pass sophisticated

Incidentally, the requirement that symbols should be able to pass sophisticated 
messages to and fro is probably sufficient to  exclude neurons themselves from playing 
the role of symbols. Since a neuron has only a single way of sending information out of 
itself, and has no way of selectively selec ting a signal now in one direction, now in 
another, it simply does not have the kind of selective triggering power which a symbol must have to act e an object in the real wo rld. I n his book The Insect Societies, E. O. 
Wilson makes a similar point about how messages propagate around inside ant colonies:

Wilson makes a similar point about how messages propagate around inside ant colonies: 
 
[Mass communication] is defined as the transfer, among groups, of information that a 
single individual could not pass to another.' 
 
It is not such a bad image, the brain as an ant colony! 
The next question-and an extremely im portant one it is, too concerns the 
nature and "size" of the concepts wh ich are represented in the tin by single 
symbols. About the nature of symbols ther e are questions like this: Would there be 
a symbol for the general notion of waterfa lls, or would there be different symbols

a symbol for the general notion of waterfa lls, or would there be different symbols 
for various specific waterfalls? Or would both of these alternatives be realized? 
About the "size" of symbols, there are que stions like this: Would there be a symbol 
for an entire story? Or for a

 357 
melody? Or a joke? Or is it more likely that there would only be symbols for concepts roughly the size of words, and that larger ideas , such as phrases or sentences, would be 
represented by concurrent or sequential activation of various symbols? 
Let us consider the issue of the size of concepts represented by symbols. Most thoughts 
expressed in sentences are made up out of basic, quasi-atomic components which we do 
not usually analyze further. These are of word size, roughly-sometimes a little longer, 
sometimes a little shorter. For instance, the noun "waterfall", the proper noun "Niagara Falls", the past-tense suffix "-ed", the ve rb "to catch up with", and longer idiomatic 
phrases are all close to atomic. These are t ypical elementary brush strokes which we use 
in painting portraits of more complex concepts, such as the plot of a movie, the flavor of 
a city, the nature of consciousness, etc. Such  complex ideas are not single brush strokes..

a city, the nature of consciousness, etc. Such  complex ideas are not single brush strokes.. 
It seems reasonable to think that the brush st rokes of language are also brush strokes of 
thought, and therefore that symbols represent c oncepts of about this size. Thus a symbol 
would be roughly something for which you know a word or stock phrase, or with which 
you associate a proper name. And the representa tion in the brain of a more complex idea, 
such as a problem in a love affair, would be  a very complicated sequence of activations 
of various symbols by other symbols. 
 
Classes and Instances

Classes and Instances 
 There is a general distinction concerni ng thinking: that between categories and 
individuals, or classes and instances. (Two other terms sometimes used are "types" and 
"tokens".) It might seem at first sight that a given symbol would inherently be either a 
symbol for a class or a symbol for an instan ce-but that is an oversimplification. Actually, 
most symbols may play either role, depending on the context of their activation. For example, look at the list below: 
 
(1) a publication 
(2) a newspaper

There is a general distinction concerni ng thinking: that between categories and 
 
(2) a newspaper 
(3) The San Francisco Chronicle 
(4) the May 18 edition of the Chronicle 
(5) my copy of the May 18 edition of the Chronicle 
(6) my copy of the May 18 edition of the Chronicle as 
it was when I first picked it up (as contrasted with my copy as it was a few days later: in my fireplace, burning) 
 
Here, lines 2 to 5 all play both roles. Thus, lin e 4 is an instance of of the general class of 
line 3, and line 5 is an instance of line 4. Line 6 is a special kind of instance of a class: a

line 3, and line 5 is an instance of line 4. Line 6 is a special kind of instance of a class: a 
manifestation. The successive stages of an object during its life history are its 
manifestations. It is interesting to wonder if  the cows on a farm perceive the invariant 
individual underneath all the manifestations of the jolly farmer who feeds then hay.

 358The Prototype Principle 
 
The list above seems to be a hierarchy of gene rality-the top being a very road conceptual 
category, the bottom some very humble partic ular thing located in space and time. 
However, the idea that a "class" must always be enormously broad and abstract is far too 
limited. The reason- is that our thought makes us e of an ingenious principle, which might 
be called the prototype principle : 
 
The most specific event can serve as a general example 
of a class of events. 
 
Everyone knows that specific events have a vividness which imprints them i strongly on 
the memory that they can later be used as models for other vents which are like them in 
some way. Thus in each specific event, there is the germ of a whole class of similar 
events. This idea that there is generality in the specific is of far-reaching importance. 
Now it is natural to ask: Do the symbols in the brain represent classes, r instances? Are

Now it is natural to ask: Do the symbols in the brain represent classes, r instances? Are 
there certain symbols which represent only cl asses, while other symbols represent only 
instances? Or can a single symbol serve duty e ither as a class symbol or instance symbol, 
depending which parts of it are activated? The latter theory seems appealing; one might 
think that a "light" activation of a symbol might represent a class, and that a deeper, or more complex, activation would contain more de tailed internal neural firing patterns, and

think that a "light" activation of a symbol might represent a class, and that a deeper, or more complex, activation would contain more de tailed internal neural firing patterns, and 
hence would represent an instance. But on second thought, its is crazy: it would imply, for example, that by activating the symbol fo r publication" in a sufficiently complex way, 
you would get the very complex symbol whic h represents a specific newspaper burning 
in my fireplace. And very other possible ma nifestation of every other piece of printed 
matter would be represented internally by so me manner of activating the single symbol 
for "publication". That seems much too h eavy a burden to place on to single symbol

for "publication". That seems much too h eavy a burden to place on to single symbol 
"publication". One must conclude, therefore, that finance symbols can exist side by side 
with class symbols, and are not just lodes of activation of the latter. 
 
The Splitting-off of Instances from Classes 
 
On the other hand, instance symbols often i nherit many of their properties from the 
classes to which those instances belong. If I tell you I went to see a Movie, you will begin "minting" a fresh new instance symbol for that  particular movie; but in the absence of 
more information, the new instance symbol w ill have to lean rather heavily on your pre-
existing class symbol for movie". Un consciously, you will rely on a host of

existing class symbol for movie". Un consciously, you will rely on a host of 
presuppositions about at movie-for example, that it lasted between one and three hours, that it was shown in a local theater, that it told a story about some people, and so i. These are built into the class symbol as expected links to other symbols e., potential triggering relations), and are called default options . In any

 359freshly minted instance symbol, the default options can easily be overridden, but unless 
this is explicitly done, they will remain in the instance symbol, inherited from its class 
symbol. Until they are overridden, they provide  some preliminary basis for you to think 
about the new instance for example, the m ovie I went to see-by using the reasonable 
guesses which are supplied by the "stereotype", or class symbol. 
A fresh and simple instance is like a child without its own ideas or experiences-it relies 
entirely on its parents' experiences and opinions  and just parrots them. But gradually, as it 
interacts more and more with the rest of the world, the child acquires its own 
idiosyncratic experiences and inevitably begins to split away from the parents. Eventually, the child becomes a full-fledged a dult. In the same way, a fresh instance can 
split off from its parent class over a period of time, and become a class, or prototype, in 
its own right.

its own right. 
For a graphic illustration of such a splitting- off process, suppose that some Saturday 
afternoon you turn on your car radio, and happen to tune in on a football game between 
two "random" teams. At first you do not know the names of the players on either team. 
All you register, when the announcer says, "Palindromi made the stop on the twenty-
seven yard line, and that brings up fourth down and six to go," is that some player 
stopped some other player. Thus it is a case of activation of the class symbol "football

stopped some other player. Thus it is a case of activation of the class symbol "football 
player", with some sort of coordinated activ ation of the symbol for tackling. But then as 
Palindromi figures in a few more key pl ays, you begin building up a fresh instance 
symbol for him in particular, using his name, perhaps, as a focal point. This symbol is 
dependent, like a child, on the class symbol fo r "football player": mo st of your image of 
Palindromi is supplied by your stereotype of  a football player as contained in the

Palindromi is supplied by your stereotype of  a football player as contained in the 
"football player" symbol. But gradually, as more information comes to you, the "Palindromi" symbol becomes more autonom ous, and relies less and less on concurrent 
activation of its parent class symbol. This  may happen in a few minutes, as Palindromi 
makes a few good plays and stands out. His teammates may still all be represented by 
activations of the class symbol, however. Ev entually, perhaps after a few days, when you 
have read some articles in the sports secti on of your paper, the umbilical cord is broken,

have read some articles in the sports secti on of your paper, the umbilical cord is broken, 
and Palindromi can stand on his own two f eet. Now you know such things as his home 
town and his major in college; you recognize his face; and so on. At this point, Palindromi is no longer conceived of merely as a football player, but as a human being 
who happens also to be a football player. "P alindromi" is an instance symbol which can 
become active while its parent class symbol (football player) remains dormant. 
Once, the Palindromi symbol was a satellite orbiting around its mother symbol, like an 
artificial satellite circling the Earth, which is  so much bigger and more massive. Then

artificial satellite circling the Earth, which is  so much bigger and more massive. Then 
there came an intermediate stage, where one symbol was more important than the other, 
but they could be seen as orbiting around each other-something like the Earth and the 
Moon. Finally, the new symbol becomes quite  autonomous; now it might easily serve as 
a class symbol around which could start rotating new satellites-

 360symbols for other people who are less familiar but who have something in common with 
Palindromi, and for whom he can serve as a temporary stereotype, until you acquire more 
information, enabling the new symbols so to become autonomous. 
 
The Difficulty of Disentangling Symbols from Each Other 
 
These stages of growth and eventual det achment of an instance from a ass will be 
distinguishable from each other by the way in which the symbols involved are linked. 
Sometimes it will no doubt be very difficult to 11 just where one symbol leaves off and the other one begins. How "active" the one sym bol, compared to the other? If one can be 
activated independently of the other, then it would be quite sensible to call them autonomous. 
We have used an astronomy metaphor above, and it is interesting that to problem of the

We have used an astronomy metaphor above, and it is interesting that to problem of the 
motion of planets is an extremely complex one-in fact the general problem of three gravitationally interacting bodies (such as th e Earth, Moon, and Sun) is far from solved, 
even after several centuries of work. One s ituation in which it is possible to obtain good 
approximate solutions, however, is when one body is much more massive than the other 
two (here, the Sun); then it makes sense to c onsider that body as stationary, with the other 
two rotating about it: on top of this can finally  be added the interaction between the two

two rotating about it: on top of this can finally  be added the interaction between the two 
satellites. But this approximation depends on breaking up the system into the Sun, and a 
"cluster": the Earth-Moon 'stem. This is an approximation, but it enables the system to be 
understood quite deeply. So to what extent is this cluster a part of reality, and to hat 
extent is it a mental fabrication, a human imposition of structure on me universe? This 
problem of the "reality" of boundaries drawn be tween hat are perceived to be autonomous 
or semi-autonomous clusters will create endle ss trouble when we relate it to symbols in 
the brain. 
One greatly puzzling question is the simple i ssue of plurals. How do we visualize, say,

the brain. 
One greatly puzzling question is the simple i ssue of plurals. How do we visualize, say, 
three dogs in a teacup? Or several people in an elevator? Do we begin with the class 
symbol for "dog" and then rub three "copies" off of it? That is, do we manufacture three 
fresh instance symbols using the class 'sym bol "dog" as template? Or do we jointly 
activate the symbols "three" and log"? By adding more or less detail to the scene being imagined, either theory becomes hard to main tain. For instance, we certainly do not have 
a separate instance symbol for each nose, mustach e, grain of salt, etc., that we have ever

a separate instance symbol for each nose, mustach e, grain of salt, etc., that we have ever 
seen. We let class symbols take care of su ch numerous items, and when we pass people 
on the street who have mustaches, we somehow just activate the "mustache" class symbol, without minting fresh instance symbols, unless we scrutinize them carefully. 
On the other hand, once we begin to distingui sh individuals, we cannot rely on a single 
class symbol (e.g., "person") to timeshare itself among all the different people. Clearly there must come into existence separate stan ce symbols for individual people. It would be 
ridiculous to imagine

 361 
that this feat could be accomplished by 'j uggling"-that is, by the single class symbol 
flitting back and forth between several diffe rent modes of activation (one for each 
person). 
Between the extremes, there must be room for many sorts of intermediate cases. There 
may be a whole hierarchy of ways of creati ng the class-instance distinction in the brain, 
giving rise to symbols-and symbol organiza tions-of varying degrees of specificity. The 
following different kinds of individual a nd joint activation of symbols might be 
responsible for mental images of various degrees of specificity: 
 
(1) various different modes or depths of activation of a single class symbol: (2) simultaneous activation of several cla ss symbols in some in some coordinated 
manner: 
(3) activation of a single instance symbol: (4) activation of a single instance symbol in conjunction with activation of several 
class symbols:

manner: 
class symbols: 
(5) simultaneous activation of several in stance symbols and several class symbols 
in some coordinated manner. 
 This brings us right hack to the questi on: "When is a symbol a distinguishable 
subsystem of the brain For instance, consider the second example-simultaneous activation of several class symbols in some  coordinated manner. This could easily be 
what happens when "piano sonata" is the concept under consideration (the symbols for "piano" and "sonata" being at least two of th e activated symbols). But if this pair of

what happens when "piano sonata" is the concept under consideration (the symbols for "piano" and "sonata" being at least two of th e activated symbols). But if this pair of 
symbols gets activated in conjunction often enough, it is reasonable to assume that the 
link between them will become strong enough that  they will act as a unit, when activated 
together in the proper way. So two or mo re symbols can act as one, under the proper 
conditions, which means that the problem of enumerating the number of symbols in the 
brain is trickier than one might guess. 
Sometimes conditions can arise where two pr eviously unlinked symbols get activated 
simultaneously and in a coordinated fashion. Th ey may fit together so well that it seems

simultaneously and in a coordinated fashion. Th ey may fit together so well that it seems 
like an inevitable union, and a single new symbol is formed by the tight interaction of the 
two old symbols. If this happens, would it be fair to say that the new symbol "always had 
been there but never had been activated"-or should one say that it has been "created"? 
In case this sounds too abstract, let us take a concrete example: the Dialogue Crab 
Canon. In the invention of this Dialogue, two existing symbols-that for "musical crab 
canon", and that for "verbal dialogue “had to  be activated simultaneously and in some

canon", and that for "verbal dialogue “had to  be activated simultaneously and in some 
way forced to interact. Once this was done, 'the  rest was quite inevitable: a new symbol-a 
class symbol-was born from the interaction of  these two, and from then on it was able to 
be activated on its own. Now had it always been  a dormant symbol in my brain? If so, 
then it must have also been a dormant symbol in

 362the brain of every human who ever had its component symbols, even if it never was 
awakened in them. This would mean that to  enumerate the symbols in anyone's brain, one 
would have to count all dormant symbols-all possible combinations and permutations of 
all types of activations of all known symbols. This would even include those fantastic creatures of software that one's brain invents when one is asleep-the strange mixtures of 
ideas which wake up when their host goes to sleep ... The existence of these "potential 
symbols" shows that it is really a huge oversim plification to imagine that the brain is a 
well-defined collection of symbols in well-define d states of activation. It is much harder 
than that to pin down a brain state on the symbol level. 
 
Symbols -Software or Hardware? 
 With the enormous and ever-growing repertoire  of symbols that exist in each brain, you 
might wonder whether there eventually comes a point when the brain is saturated-when

might wonder whether there eventually comes a point when the brain is saturated-when 
there is just no more room for a new sym bol. This would come about, presumably, if 
symbols never overlapped each other-if a give n neuron never served a double function, so 
that symbols would be like people getting into an elevator. "Warning: This brain has a maximum capacity of 350,275 symbols!" 
This is not a necessary feature of the sym bol model of brain function, however. In fact, 
overlapping and completely tangled symbols are probably the rule, so that each neuron, 
far from being a member of a unique symbol , is probably a functi oning part of hundreds

far from being a member of a unique symbol , is probably a functi oning part of hundreds 
of symbols. This gets a little disturbing, because  if it is true, then might it not just as 
easily be the case that each neuron is part of every single symbol? If that were so, then 
there would be no localizability whatsoever of symbols-every symbol would be identified with the whole of the brain. This would account  for results like Lashley's cortex removal 
in rats-but it would also mean abandonment of  our original idea of breaking the brain up

in rats-but it would also mean abandonment of  our original idea of breaking the brain up 
into physically distinct subsystems. Our earlie r characterization of symbols as "hardware 
realizations of concepts" could at best be a great oversimplification. In fact, if every 
symbol were made up of the same component neurons as every other symbol, then what 
sense would it make to speak of distinct symbol s at all? What would be the signature of a 
given symbol's activation-that is, how could the activation of symbol A be distinguished

given symbol's activation-that is, how could the activation of symbol A be distinguished 
from the activation of symbol B? Wouldn't our whole theory go down the drain? And even if there is not a total overlap of symbols,  is our theory not more and more difficult to 
maintain, the more that symbols do ove rlap? (One possible way of portraying 
overlapping symbols is shown in Figure 68.) 
There is a way to keep a theory based on symbols even if physically, they overlap 
considerably or totally. Consider the' su rface of a pond, which can support many different 
types of waves or ripples. The hardware namely  the water itself-is the same in all cases,

types of waves or ripples. The hardware namely  the water itself-is the same in all cases, 
but it possesses different possible modes of ex citation. Such software excitations of the 
same

 363 
 
FIGURE 68. In this schematic diagram, neurons are imagined as laid out as dots in one plane. 
Two overlapping neural pathways are shown in di fferent shades of gray. It may happen that two 
independent "neural flashes" simultaneously race down these two pathways, passing through one 
another like two ripples on a pond's surface (as in Fig. 52). This is illustrative of the idea of two "active symbols" which share neurons and whic h may even be simultaneously activated. [From 
John C. Eccles, Facing Reality (New York: Springer Verlag, 1970), p.21.] 
 
hardware can all be distinguished from each other. By this analogy, I do not mean to go 
so far as to suggest that all the different symbols are just different kinds of "waves" 
propagating through a uniform neural medium which admits of no meaningful division 
into physically distinct symbols. But it may be  that in order to distinguish one symbol's

into physically distinct symbols. But it may be  that in order to distinguish one symbol's 
activation from that of another symbol, a process must be carried out which involves not only locating the neurons which are firing, but also identifying very precise details of the 
timing of the firing of those neurons. That is, which neuron preceded which other neuron, 
and by how much? How many times a second was a particular neuron firing? Thus 
perhaps several symbols can coexist in th e same set of neurons by having different

perhaps several symbols can coexist in th e same set of neurons by having different 
characteristic neural firing patterns. The difference between a theory having physically distinct symbols, and a theory having overl apping symbols which are distinguished from 
each other by modes of excitation, is that th e former gives hardware realizations of 
concepts, while the latter gives partly hardware, partly software realizations of concepts.

 364Liftability of Intelligence 
 
Thus we are left with two basic problems in the unraveling of thought processes, as they 
take place in the brain. One is to explain how the A,-level traffic of neuron firings gives 
rise to the high-level traffic of symbol activ ations. The other is to explain the high-level 
traffic of symbol activation in its own terms- to make a theory which does not talk about 
the ,v-level neural events. If this latter is possible-and it is a key assumption the basis of 
all present research into Artificial Intelligen ce-then intelligence can be realized in other 
types of hardware than brains. Then intellig ence will have been shown to be a property 
that can be "lifted" right out of e hardwa re in which it resides-or in other words, 
intelligence will be a software property. This will mean that the phenomena of consciousness and intelligence are indeed high- level in the same sense as most other 
complex

complex  
FIGURE 69. The construction of an arch by work ers of the termite Macrotermes belosus. Each 
column is built up by the addition of pellets of so il and excrement. On the outer part of the left 
column a worker is seen depositing a round fecal pe llet. Other workers, having carried pellets in 
their mandibles up the columns, are now placing them  at the growing ends of ' columns. When a 
column reaches a certain height the termites, evid ently guided by odor, ;in to extend it at an angle

complex  
column reaches a certain height the termites, evid ently guided by odor, ;in to extend it at an angle 
in the direction of a neighboring column. A completed arch is shown in the background. 
[Drawing by Turid Holldobler; from E. 0. Wilson, The Insect Societies Cambridge, Mass.: Harvard University Press, 1971), p. 230]

 365 
phenomena of nature: they have their ow n high-level laws which depend on, yet are 
"liftable" out of, the lower levels. If, on th e other hand, there is absolutely no way to 
realize symbol-triggering patterns without having all the hardware of neurons (or 
simulated neurons), this will imply that intelligence is a brain-bound phenomenon, and much more difficult to unravel than one which owes its existence to a hierarchy of laws 
on several different levels. Here we come back to the mysterious collective behavior of ant colonies, which can build huge and intricate nests, despite the fact that the roughly 100,000 neurons of an ant brain 
almost certainly do not carry any. information about nest structure. How, then, does the 
nest get created? Where does the informati on reside? In particular, ponder where the 
information describing an arch such as is  shown in Figure 69 can be found. Somehow, it

information describing an arch such as is  shown in Figure 69 can be found. Somehow, it 
must be spread about in the colony, in th e caste distribution, the age distribution-and 
probably largely in the physical properties of the ant-body itself. That is, the interaction between ants is determined just as much by their six-leggedness and their size and so on, as by the information stored in their brain. Could there be an Artificial Ant Colony?  
Can One Symbol Be Isolated? 
 Is it possible that one single symbol could be  awakened in isolation from all others?

 Is it possible that one single symbol could be  awakened in isolation from all others? 
Probably not. Just as objects in the world alwa ys exist in a context of other objects, so 
symbols are always connected to a conste llation of other symbols. This does not 
necessarily mean that symbols can never be disentangled from each other. To make a rather simple analogy, males and females always arise in a species together: their roles are completely intertwined, and yet this  does not mean that a male cannot be 
distinguished from a female. Each is reflected  in the other, as the beads in Indra's net

distinguished from a female. Each is reflected  in the other, as the beads in Indra's net 
reflect each other. The recursive intertwining of  the functions F(n) and M(n) in Chapter V 
does not prevent each function from having its own characteristics. The intertwining of F 
and M could be mirrored in a pair of RTN's wh ich call each other. From this we can jump 
to a whole network of ATN's intertwined w ith each other-a heterarchy of interacting 
recursive procedures. Here, the meshing is  so inherent that no one ATN could be 
activated in isolation; yet its activation may be completely distinctive, not confusable

activated in isolation; yet its activation may be completely distinctive, not confusable 
with that of any other of the ATN's. It is not such a bad image, the brain as an ATN-colony! Likewise, symbols, with all their multiple li nks to each other, are meshed together and 
yet ought to be able to be teased apart. Th is might involve identifying a neural network, a 
network plus a mode of excitation-or possibl y something of a completely different kind. 
In any case, if symbols are part  of reality, presumably there exists a natural way to chart 
them out in a real brain. However, if some sy mbols were finally identified in a brain, this

them out in a real brain. However, if some sy mbols were finally identified in a brain, this 
would not mean that any one of them could be awakened in isolation.

 366The fact that a symbol cannot be awaken ed in isolation does not diminish the 
separate identity of the symbol; in fact, qu ite to the contrary: a symbol's identity lies 
precisely in its ways of being connected (via  potential triggering links) to other symbols. 
The network by which symbols can potentially trigger each other constitutes the brain's 
working model of the real universe, as we ll as of the alternate universes which it 
considers (and which are every bit as important  for the individual's survival in the real 
world as the real world is). 
 
The Symbols of Insects 
 Our facility for making instances out of classe s and classes out of instances lies at the 
basis of our intelligence, and it is one of th e great differences between human thought and 
the thought processes of other animals. Not th at I have ever belonged to another species 
and experienced at first hand how it feels to think their way-but from the outside it is

and experienced at first hand how it feels to think their way-but from the outside it is 
apparent that no other species forms general concepts as we do, or imagines hypothetical 
worlds-variants on the world as it is, which ai d in figuring out which future pathway to 
choose. For instance, consider the celebrate d "language of the bees"-information-laden 
dances which are performed by worker bees retu rning to the hive, to inform other bees of 
the location of nectar. While there may be in each bee a set of rudimentary symbols which are activated by such a dance, there is no reason to believe that a bee has an

the location of nectar. While there may be in each bee a set of rudimentary symbols which are activated by such a dance, there is no reason to believe that a bee has an 
expandable vocabulary of symbols. Bees and ot her insects do not seem to have the power 
to generalize-that is, to develop new cla ss symbols from instances which we would 
perceive as nearly identical. A classic experiment with solitary wasps is reported in Dean Wooldridge's book, Mechanical Man, from which I quote:  
When the time comes for egg laying, the wasp Sphex builds a burrow for the purpose and seeks out a cricket which she stings  in such a way as to paralyze but not 
kill it. She drags the cricket into the burro w, lays her eggs alongside, closes the

kill it. She drags the cricket into the burro w, lays her eggs alongside, closes the 
burrow, then flies away, never to return. In  due course, the eggs hatch and the wasp 
grubs feed off the paralyzed cricket, whic h has not decayed, having been kept in the 
wasp equivalent of a deepfreeze. To the human mind, such an elaborately organized 
and seemingly purposeful routine conveys  a convincing flavor of logic and 
thoughtfulness-until more details are examined. For example, the wasp's routine is to 
bring the paralyzed cricket to the burrow, leave it on the threshold, go inside to see

bring the paralyzed cricket to the burrow, leave it on the threshold, go inside to see 
that all is well, emerge, and then drag th e cricket in. If the cricket is moved a few 
inches away while the wasp is inside making her preliminary inspection, the wasp, on emerging from the burrow, will bring the cricket back to the threshold, but not inside, and will then repeat the preparator y procedure of entering the burrow to see 
that everything is all right. If again the cricket is removed a few inches while the

that everything is all right. If again the cricket is removed a few inches while the 
wasp is inside, once again she will move the cricket up to the threshold and reenter the burrow for a final check. The wasp never thinks of pulling the cricket straight in. 
On one occasion this procedure was repeated forty times, always with the same 
result.'

 367This seems to be completely hard-wired behavior. Now in the wasp brain, there may be 
rudimentary symbols, capable of triggeri ng each other; but there is nothing like the 
human capacity to see several instances as in stances of an as-yet-unformed class, and 
then to make the class symbol; nor is th ere anything like the human ability to wonder, 
"What if I did this-what would ensue in that hypothetical world%" This type of thought process requires an ability to manufacture inst ances and to manipulate them as if' they 
were symbols standing for objects in a real situation, although that situation may not be 
the case, and may never be the case.  
Class Symbols and Imaginary Worlds 
 Let us reconsider the April Fool's joke a bout the borrowed car, and the images conjured 
up in your mind during the telephone call. To begin with, you need to activate symbols

up in your mind during the telephone call. To begin with, you need to activate symbols 
which represent a road, a car, a person in a car. Now the concept "road" is a very general one, with perhaps several stock sample s which you can unconsciously pull out of 
dormant memory when the occasion arises. "Road"  is a class, rather than an instance. As 
you listen to the tale, you quickly activate sy mbols which are instances with gradually 
increasing-specificity. For instance, when you l earn that the road' was wet, this conjures 
up a more specific image, though you realize that it is most likely quite different from the 
actual road where the incident took place. Bu t that is not important; what matters is

actual road where the incident took place. Bu t that is not important; what matters is 
whether your symbol is sufficiently well suited for the story-that is, whether the symbols 
which it can trigger are the right kind. As the story progresses, you fill in more aspects of this road: there is a high bank against which a car could smash. Now does this mean  that you are activating the symbol for 
"bank", or does it mean that you are setting so me parameters in your symbol for "road 
Undoubtedly both. That is, the network of ne urons which represents "road" has many

Undoubtedly both. That is, the network of ne urons which represents "road" has many 
different ways of firing, and you are selecting which subnetwork actually shall fire. At 
the same time, you are activating the symbol fo r "bank", and this is probably instrumental 
in the process of selecting the parameters for. "road", in that its neurons may send signals to some of those in "road"-and vice versa. (In case this seems a little confusing, it is because I am somewhat straddling levels of de scription-I am trying to set up an image of

in the process of selecting the parameters for. "road", in that its neurons may send signals to some of those in "road"-and vice versa. (In case this seems a little confusing, it is because I am somewhat straddling levels of de scription-I am trying to set up an image of 
the symbols, as well as of their component neurons.) No less important than the nouns are the ve rbs, prepositions, etc: They, too, activate 
symbols, which send messages back and fort h to each other. There are characteristic 
differences between the kinds of triggering pa tterns of symbols for verbs and symbols for

differences between the kinds of triggering pa tterns of symbols for verbs and symbols for 
nouns, of course, which means that they may be physically somewhat differently organized. For instance, nouns might have fairly localized symbols, while verbs and prepositions might have symbols with many "t entacles" reaching all around the cortex; or 
any number of other possibilities. After the story is all over, you learn it was all untrue. The power of

 368"rubbing off" instances from classes, in the way that one makes rubbings from brasses in 
churches, has enabled you to represent the s ituation, and has freed you from the need to 
remain faithful to the real world. The fact that symbols can act as templates for other 
symbols gives you some mental independe nce of reality: you can create artificial 
universes, in which there can happen nonreal events with any amount of detail that you 
care to imbue them with. But the class symbols themselves, from which all of this richness springs, are deeply grounded in reality. Usually symbols play isomorphic roles to events which seem like they could happen, 
although sometimes symbols are activated whic h represent situations which could not 
happen-for example, watches sizzling, tubas laying eggs, etc. The borderline between 
what could and what could not happen is an extremely fuzzy one. As we imagine a 
hypothetical event, we bring certain sym bols into active states-and depending on how

hypothetical event, we bring certain sym bols into active states-and depending on how 
well they interact (which is presumably reflect ed in our comfort in continuing the train of 
thought), we say the event "could" or "could not" happen. Thus the terms "could" and "could not" are extremely subjective. Actua lly, there is a good deal of agreement among 
people about which events could or could not  happen. This reflects the great amount of 
mental structure which we all share-but th ere is a borderline area where the subjective 
aspect of what kinds of hypothetical worlds we are willing to entertain is apparent. A 
careful study of the kinds of imaginary events that people consider could and could not

careful study of the kinds of imaginary events that people consider could and could not 
happen would yield much insight into the triggering patterns of the symbols by which 
people think.  
Intuitive Laws of Physics 
 When the story has been completely told, you have built up quite an elaborate mental 
model of a scene, and in this model all th e objects obey physical law. This means that 
physical law itself must be implicitly present in the triggering patterns of the symbols. Of course, the phrase "physical law" here does not mean "the laws of physics as expounded 
by a physicist", but rather the intuitive, chunked la ws which all of us have to have in our

by a physicist", but rather the intuitive, chunked la ws which all of us have to have in our 
minds in order to survive. A curious sidelight is that one can voluntar ily manufacture mental sequences of events 
which violate physical law, if one so desire s. For instance, if I but suggest that you 
imagine a scene with two cars approachi ng each other and then passing right through 
each other, you won't have any trouble doing so. The intuitive physical laws can be 
overridden by imaginary laws of physics; but  how this overriding is done, how such

overridden by imaginary laws of physics; but  how this overriding is done, how such 
sequences of images are manufactured-indeed what any one visual image is-all of these 
are deeply cloaked mysteries-inaccessible pieces of knowledge. Needless to say, we have in our brains c hunked laws not only of how inanimate objects 
act, but also of how plants, animals, peopl e and societies act-in other words, chunked 
laws of biology, psychology, sociology, and so

 369on. All of the internal representations of su ch entities involve the inevitable feature of 
chunked models: determinism is sacrificed fo r simplicity. Our representation of reality 
ends up being able only to predict probabilities of ending up in certain parts of abstract 
spaces of behavior-not to predict anything with the precision of physics.  
Procedural and Declarative Knowledge 
 A distinction which is made in Artificial Intelligence is that between procedural and declarative types of knowledge. A piece of knowledge is said to be declarative if it is 
stored explicitly, so that not only the progra mmer but also the program can "read" it as if 
it were in an encyclopedia or an almanac. Th is usually means that it is encoded locally, 
not spread around. By contrast, procedural knowledge is not encoded as facts-only as

not spread around. By contrast, procedural knowledge is not encoded as facts-only as 
programs. A programmer may be able to peer in and say, "I see that because of these procedures here, the program `knows' how to  write English sentences"-but the program 
itself may have no explicit awareness of how it writes those sentences. For instance, its vocabulary may include none of the words "Englis h", "sentence", and "write" at all! Thus 
procedural knowledge is usually spread around in pieces, and you can't retrieve it, or "key" on it. It is a global consequence of how  the program works, not a local detail. In

procedural knowledge is usually spread around in pieces, and you can't retrieve it, or "key" on it. It is a global consequence of how  the program works, not a local detail. In 
other words, a piece of purely procedural knowledge is an epiphenomenon. 
In most people there coexists, along with a powerful procedural representation of 
the grammar of their native language, a weaker  declarative representation of it. The two 
may easily be in conflict, so that a native sp eaker will often instruct a foreigner to say 
things he himself would never say, but whic h agree with the declarative "book learning" 
he acquired in school sometime. The intuitive or chunked laws of physics and other disciplines mentioned earlier fall mainly on th e procedural side; the knowledge that an

he acquired in school sometime. The intuitive or chunked laws of physics and other disciplines mentioned earlier fall mainly on th e procedural side; the knowledge that an 
octopus has eight tentacles falls mainly on the declarative side. 
In between the declarative and procedural  extremes, there are all possible shades. 
Consider the recall of a melody. Is the mel ody stored in your brain, note by note? Could a 
surgeon extract a winding neural filament fr om your brain, then stretch it straight, and 
finally proceed to pinpoint along it the successive ly stored notes, almost as if it were a 
piece of magnetic tape? If so, then melodies ar e stored declaratively. Or is the recall of a

piece of magnetic tape? If so, then melodies ar e stored declaratively. Or is the recall of a 
melody mediated by the interaction of a la rge number of symbols, some of which 
represent tonal relationships, others of wh ich represent emotional qualities, others of 
which represent rhythmic devices, and so on? If so, then melodies are stored 
procedurally. In reality, there is probably a mixture of these extremes in the way a 
melody is stored and recalled. 
It is interesting that, in pulling a melody out of memory, most people do not 
discriminate as to key, so that they are as likely to sing "Happy Birthday" in the key of F-

sharp as in the key of C. This indicates that tone relationships, rather than absolute tones, 
are stored. But there is no reason

 370that tone relationships could not be stored  quite declaratively. On the other hand, some 
melodies are very easy to memorize, whereas others are extremely elusive. If it were just a matter of storing successive notes, any melody could be stored as easily as any other. 
The fact that some melodies are catchy and ot hers are not seems to indicate that the brain 
has a certain repertoire of familiar patterns which are activated as the melody is heard. 
So, to "play back" the melody, those patterns would have to be activated in the same order. This returns us to the concept of sy mbols triggering one another, rather than a 
simple linear sequence of declaratively stored notes or tone relationships. 
How does the brain know whether a piece of  knowledge is stored declaratively? 
For instance, suppose you are asked, "What is  the population of Chicago?" Somehow the 
number five million springs to mind, wit hout your wondering, "Gee, how would I go

number five million springs to mind, wit hout your wondering, "Gee, how would I go 
about counting them all?" Now suppose I ask you, "How many chairs are there in your living room?" Here, the opposite happens-instead of trying to dredge the answer out of a 
mental almanac, you immediately either go to the room and count the chairs, or you 
manufacture the room in your head and count the chairs in the image of the room. The 
questions were of a single type-"how ma ny?"-yet one of them caused a piece of 
declarative knowledge to be fetched, while th e other one caused a procedural method of

declarative knowledge to be fetched, while th e other one caused a procedural method of 
finding the answer to be invoked. This is one  example where it is clear that you have 
knowledge about how you classify your own knowle dge; and what is more, some of that 
metaknowledge may itself be stored procedura lly, so that it is used without your even 
being aware of how it is done. 
 
Visual Imagery 
 One of the most remarkable and difficult-to-describe qualities of consciousness is visual imagery. How do we create a visual image of  our living room? Of a roaring mountain 
brook? Of an orange? Even more mysterious, how do we manufacture images unconsciously, images which guide our t houghts, giving them power and color and

brook? Of an orange? Even more mysterious, how do we manufacture images unconsciously, images which guide our t houghts, giving them power and color and 
depth? From what store are they fetched? Wh at magic allows us to mesh two or three 
images, hardly giving a thought as to how we  should do it? Knowledge of how to do this 
is among the most procedural of all, for we  have almost no insight into what mental 
imagery is. 
It may be that imagery is based on our ab ility to suppress motor activity. By this, I 
mean the following. If you imagine an orange, there may occur in your cortex a set of 
commands to pick it up, to smell it, to in spect it, and so on. Clearly these commands

commands to pick it up, to smell it, to in spect it, and so on. Clearly these commands 
cannot be carried out, because the orange is not there. But they can be sent along the 
usual channels towards the cerebellum or ot her suborgans of the brain, until, at some 
critical point, a "mental faucet" is closed, preventing them from actually being carried 
out. Depending on how far down the line this "f aucet" is situated, the images may be 
more or less vivid and real-seeming. Anger can cause us to

 371imagine quite vividly picking up some object and throwing it, or kicking something; yet 
we don't actually do so. On the other hand, we feel so "near" to actually doing so. 
Probably the faucet catches the nerve impulses "at the last moment". - 
Here is another way in which visualization points out the distinction between 
accessible and inaccessible knowledge. Consider how you visualized the scene of the car 
skidding on the mountain road. Undoubtedly you imagined the mountain as being much 
larger than the car. Now did this happen because sometime long ago you had occasion to 
note that "cars are not as big as mountains";  then you committed this statement to rote 
memory: and in imagining the story, you retr ieved this fact, and made use of it in 
constructing your image? A most unlikely theory. Or did it happen instead as a 
consequence of some introspectively inaccessibl e interactions of the symbols which were

consequence of some introspectively inaccessibl e interactions of the symbols which were 
activated in your brain? Obviously the latte r seems far more likely. This knowledge that 
cars are smaller than mountains is not a piece of rote memorization, but a piece of 
knowledge which can be created by deduction. Ther efore, most likely it is not stored in 
any single symbol in your brain, but rather it can be produced as a result of the activation, 
followed by the mutual interaction, of many sy mbols-for example, those for "compare", 
"size", "car", "mountain", and probably, others . This means that the knowledge is stored

"size", "car", "mountain", and probably, others . This means that the knowledge is stored 
not explicitly, but implicitly, in a spread-about manner, rather than as a local "packet of 
information". Such simple facts as relative sizes of objects have to be assembled, rather than merely retrieved. Therefore, even in  the case of a verbally accessible piece of 
knowledge, there are complex inaccessible processes which mediate its coming to the state of being ready to be said. 
We shall continue our exploration of the entities called "symbols" in different 
Chapters. In Chapters XVIII and XIX, on Artif icial Intelligence, we shall discuss some

Chapters. In Chapters XVIII and XIX, on Artif icial Intelligence, we shall discuss some 
possible ways of implementing active symbols in programs. And next Chapter, we shall 
discuss some of the insights that our symbol-b ased model of brain activity give into the 
comparison of brains.

English French German Suite 
 
 372English French German Suite 
 
By Lewis Carroll .. . 
 
... et Frank L. Warrin .. . .. 
 
. and Robert Scott 
 
'Twas brillig, and the slithy toves Did gyre and gimble in the wabe: All mimsy were the borogoves,  And the mome raths outgrabe. 
 
I1 brilgue: les toves lubricilleux Se gyrent en vrillant dans le guave.  Enmimes sont les gougebosqueux  Et le momerade horsgrave. 
 
Es brillig war. Die schlichten Toven Wirrten and wimmelten in Waben;  Und aller-mumsige Burggoven  Die mohmen Rath' ausgraben. 
 "Beware the Jabberwock, my son! The jaws that bite, the claws that catch!  Beware the Jubjub bird, and shun  The frumious Bandersnatch!" 
 
((Garde-toi du Jaseroque, mon fits! La gueule qui mord; la griffe qui prend!  Garde-toi de I'oiseau Jube, evite  Le frumieux Band-a-prend!)) 
 
))Bewahre doch vor Jammerwoch! Die Zahne knirschen, Krallen kratzen!  Bewahr' vor Jubjub-Vogel, vor  
Frumiosen Banderschnatzchen!)),

"Beware the Jabberwock, my son! The jaws that bite, the claws that catch!  Beware the Jubjub bird, and shun  The frumious Bandersnatch!" 
 
Frumiosen Banderschnatzchen!)), 
 He took his vorpal sword in hand: Long time the manxome foe he sought So rested he by the Tumtum tree,  And stood awhile in thought.  
Son glaive vorpal en main, it va T-a la recherche du fauve manscant;  Puis arrive a I'arbre Te-te,  l y reste, reflechissant

 373Er griff sein vorpals Schwertchen zu,  
Er suchte lang das manchsam' Ding;  Dann, stehend unterm Tumtum Baum,  Er an-zu-denken-fing. 
 
And, as in uffish thought he stood, The Jabberwock, with eyes of flame.  Came whiffling through the tulgey wood,  And burbled as it came! 
 
Pendant qu'il pense, tout uffuse,  Le Jaseroque, a l'oeil flambant,  Vient siblant par le bois tullegeais,  Et burbule en venant. 
 
Als stand er tief in Andacht auf,  Des Jammerwochen's Augen-feuer  Durch turgen Wald mit Wiffek kam 
Fin burbelnd Ungeheuer! 
 
One, two! One, two! And through and through  
The vorpal blade went snicker-snack!  He left it dead, and with its head  He went galumphing back. 
 
Un deux, un deux, par le milieu,  Le glaive vorpal fait pat-a-pan!  La bete defaite, avec sa tete,  Il rentre gallomphant. 
 
Eins, Zwei! Eins, Zwei! Und durch and durch  Sein vorpals Schwert zerschnifer-schnuck,  Da blieb es todt! Er, Kopf in Hand,  Gelaumfig zog zuriick.

Eins, Zwei! Eins, Zwei! Und durch and durch  Sein vorpals Schwert zerschnifer-schnuck,  Da blieb es todt! Er, Kopf in Hand,  Gelaumfig zog zuriick. 
 
"And hast thou slain the Jabberwock?  Come to my arms, my beamish boy!  0 frabjous day! Callooh! Callay!"  He chortled in his joy. 
 
((As-tu tue le Jaseroque? Viens a mon coeur, fils rayonnais!  O jour frabbejais! Calleau! Callai!))  Il cortule clans sa joie. 
 
))Und schlugst Du ja den Jammerwoch?  Umarme mich, mein Bohm'sches Kind!  O Freuden-Tag! 0 Halloo-Schlag!((  Er schortelt froh-gesinnt.

 374'Twas brillig, and the slithy toves  
Did gyre and gimble in the wabe:  All mimsy were the borogoves,  And the mome raths outgrabe. 
 Il brilgue: les toves lubricilleux Se gyrent en vrillant dans le guave.  Enmimes sont les gougebosqueux  Et le momerade horsgrave.  
Es brillig war. Die schlichten Toven  Wirrten and wimmelten in Waben:  Und aller-mumsige Burggoven  Die mohmen Rath' ausgraben.

Minds and Thoughts 375CHAPTER XII 
 
Minds and Thoughts 
 
Can Minds Be Mapped onto Each Other? 
 
Now THAT WE have hypothesized the existence of very high-level active 
subsystems of the brain (symbols), we may return to the matter of a possible 
isomorphism, or partial isomorphism, between  two brains. Instead of asking about an 
isomorphism on the neural level (which sure ly does not exist), or on the macroscopic 
suborgan level (which surely does exist but does not tell us very much), we ask about the possibility of an isomorphism between br ains on the symbol level: a correspondence 
which not only maps symbols in one brain ont o symbols in another brain, but also maps 
triggering patterns onto triggering patterns. This means that corresponding symbols in the two brains are linked in corresponding ways. This would be a true functional  
isomorphism-the same type of isomorphism as  we spoke of when trying to characterize 
what it is that is invariant about all butterflies,

what it is that is invariant about all butterflies, 
It is clear from the outset that such an isomorphism does not exist between any 
pair of human beings. If it did, they would be completely indistinguishable in their 
thoughts; but in order for that to be true , they would have to have completely 
indistinguishable memories, which would mean they would have to have led one and the 
same life. Even identical twins do not approach, in the remotest degree, this ideal. 
How about a single individual% When  you look back over things which you 
yourself wrote a few years ago, you think "How awful!" and smile with amusement at the

yourself wrote a few years ago, you think "How awful!" and smile with amusement at the 
person you once were. What is worse is when you do the same thing with something you wrote or said five minutes ago. When th is happens, it shows that you do not fully 
understand the person you were moments a go. The isomorphism from your brain now to 
your brain then is imperfect. What, then, of the isomorphisms to other people, other 
species ... 
The opposite side of the coin is shown by the power of the communication that

The opposite side of the coin is shown by the power of the communication that 
arises between the unlikeliest partners. Thi nk of the barriers spanned when you read lines 
of poetry penned in jail by Francois Villon, th e French poet of the 1400's. Another human 
being, in another era, captive in jail, sp eaking another language ... How can you ever 
hope to have a sense of the connotations behi nd the facade of his words, translated into 
English% Yet a wealth of meaning comes through. 
Thus, on the one hand, we can drop all hopes of finding exactly isomorphic

Thus, on the one hand, we can drop all hopes of finding exactly isomorphic 
software in humans, but on the other, it is clear that some people think more alike than others do. It would seem an obvious conclu

Minds and Thoughts 376
 
FIGURE 70. A tiny portion of the author's "semantic network".

Minds and Thoughts 377 
sion that there is some sort of partial so ftware isomorphism connecting the brains of 
people wbose style of thinking is similar-i n particular, a correspondence of (1) the 
repertoire of symbols, and (2) the triggering patterns of symbols  
Comparing Different Semantic Networks 
 
But what is a partial  isomorphism? This is a most difficult question to answer. It is made 
even more difficult by the fact that no one has found an adequate way to represent the 
network of symbols and their triggering pattern s. Sometimes a picture of a small part of 
such a network of symbols is drawn, where each symbol is represented as a node into 
which, and out of which, lead some arcs. Th e lines represent triggering relationships-in 
some sense. Such figures attempt to capture  something of the intuitively sensible notion 
of "conceptual nearness". However, there are many different kinds of nearness, and 
different ones are relevant in different c ontexts. A tiny portion of my own "semantic

different ones are relevant in different c ontexts. A tiny portion of my own "semantic 
network" is shown in Figure 70. The pr oblem is that representing a complex 
interdependency of many symbols cannot be carri ed out very easily with just a few lines 
joining vertices. 
Another problem with such a diagram is that  it is not accurate to think of a symbol 
as simply "on" or "off". While this is true of neurons, it does not carry upwards, to collections of them. In this respect, sym bols are quite a bit more complicated than

as simply "on" or "off". While this is true of neurons, it does not carry upwards, to collections of them. In this respect, sym bols are quite a bit more complicated than 
neurons-as you might expect, since they are made up of many neurons. The messages that are exchanged between symbols are more co mplex than the mere fact, "I am now 
activated". That is more like the neuron-level messages. Each symbol can be activated in 
many different ways, and the type of activation will be influential in determining which other symbols it tries to activate. How these intertwining triggering relationships can be 
represented in a pictorial manner-indeed, whether they can be at all-is not clear.

represented in a pictorial manner-indeed, whether they can be at all-is not clear. 
But for the moment, suppose that issue had been solved. Suppose we now agree 
that there are certain drawings of nodes, c onnected by links (let us say they come in 
various colors, so that various types of c onceptual nearness can be distinguished from 
each other), which capture precisely the way in which symbols trigger other symbols. Then under what conditions would we feel that  two such drawings were isomorphic, or 
nearly isomorphic? Since we are dealing with  a visual representation of the network of 
symbols, let us consider an analogous visu al problem. How would you try to determine

symbols, let us consider an analogous visu al problem. How would you try to determine 
whether two spiderwebs had been spun by sp iders belonging to the same species? Would 
you try to identify individual vertices wh ich correspond exactly, thereby setting up an 
exact map of one web onto the other, vertex by vertex, fiber by fiber, perhaps even angle 
by angle? This would be a futile  effort. Two webs are never exactly the same: yet there is 
still some sort of "style", "form", what-hav e-you, that infallibly brands a given species' 
web. 
In any network-like structure, such as a spiderweb, one can look at local

web. 
In any network-like structure, such as a spiderweb, one can look at local 
properties and global properties. Local properties require only a very

Minds and Thoughts 378nearsighted observer-for example an observer who can only see one vertex at a time; and 
global properties require only a sweeping visi on, without attention to detail. Thus, the 
overall shape of a spiderweb is a global pr operty, whereas the average number of lines 
meeting at a vertex is a local property. Suppose we agree that the most reasonable criterion for calling two spiderwebs "isomorphi c" is that they should have been spun by 
spiders of the same species. Then it is inte resting to ask which kind of observation-local 
or global-tends to be a more reliable guide in determining whether two spiderwebs are isomorphic. Without answering the question fo r spiderwebs, let us now return to the 
question of the closeness-or isomorphicne ss, if you will-of two symbol networks. 
 
Translations of "Jabberwocky" 
 
Imagine native speakers of English, French, and German, all of whom have excellent 
command of their respective native languages, a nd all of whom enjoy wordplay in their

command of their respective native languages, a nd all of whom enjoy wordplay in their 
own language. Would their symbol networks be  similar on a local level, or on a global 
level? Or is it meaningful to ask such a question? The question becomes concrete when 
you look at the preceding translations of Lewis Carroll's famous "Jabberwocky". 
I chose this example because it demonstrat es, perhaps better than an example in 
ordinary prose, the problem of trying to fi nd "the same node" in two different networks 
which are, on some level of analysis, extrem ely nonisomorphic. In ordinary language, the

which are, on some level of analysis, extrem ely nonisomorphic. In ordinary language, the 
task of translation is more straightforward, since to each word or phrase in the original 
language, there can usually be found a corresponding word or phrase in the new language. By contrast, in a poem of this type, many "words" do not carry ordinary 
meaning, but act purely as exciters of nearby symbols. However, what is nearby in one 
language may be remote in another. 
Thus, in the brain of a native speaker of  English, "slithy" probably activates such 
symbols as "slimy", "slither", "slippery", "lith e", and "sly", to varying extents. Does

symbols as "slimy", "slither", "slippery", "lith e", and "sly", to varying extents. Does 
"lubricilleux" do the corresponding thing in th e brain of a Frenchman? What indeed 
would be "the corresponding thing"? Would it be to activate symbols which are the 
ordinary translations of those words? What if  there is no word, real or fabricated, which 
will accomplish that? Or what if a word does exist, but is very intellectual-sounding and Latinate ("lubricilleux"), rather than ear thy and Anglo-Saxon ("slithy")? Perhaps 
"huilasse" would be better than "lubricilleux" ? Or does the Latin origin of the word

"huilasse" would be better than "lubricilleux" ? Or does the Latin origin of the word 
"lubricilleux" not make itself felt to a speaker of  French in the way that it would if it were 
an English word ("lubricilious", perhaps)? 
An interesting feature of the translation into French is the transposition into the 
present tense. To keep it in the past would make some unnatural turns of phrase 
necessary, and the present tense has a much fr esher flavor in French than the past. The 
translator sensed that this would be "more appropriate"-in some ill-defined yet

translator sensed that this would be "more appropriate"-in some ill-defined yet 
compelling senseand made the switch. Who can say whether remaining faithful to the English tense would have been better?

Minds and Thoughts 379 
In the German version, the droll phrase "er an-zu-denken-fing" occurs; it does not 
correspond to any English original. It is a play ful reversal of words, whose flavor vaguely 
resembles that of the English phrase "he out-to-ponder set", if I may hazard a reverse translation. Most likely this funny turnabout of  words was inspired by the similar playful 
reversal in the English of one line earlier:  "So rested he by the Tumtum tree". It 
corresponds, yet doesn't correspond. 
Incidentally, why did the Tumturn tree ge t changed into an "arbre T6-t6" in 
French? Figure it out for yourself. 
The word "manxome" in the original , whose "x" imbues it with many rich 
overtones, is weakly rendered in German by "manchsam", which hack-translates into 
English as "maniful". The French "manscan t" also lacks the manifold overtones of 
"manxome". There is no end to the interest of this kind of translation task. 
When confronted with such an example, one realizes that it is utterly impossible

When confronted with such an example, one realizes that it is utterly impossible 
to make an exact translation. Yet even in th is pathologically difficult case of translation, 
there seems to be some rough equivalence obtaina ble. Why is this so, if there really is no 
isomorphism between the brains of people who will read the different versions? The 
answer is that  there is a kind of rough isom orphism, partly global, partly local, between 
the brains of all the readers of these three poems. 
 
ASU's 
 
An amusing geographical fantasy will give  some intuition for this kind of quasi-

ASU's 
 
isomorphism. (Incidentally, this fantasy is somewhat similar to a geographical analogy devised by M. Minsky in his article on "frames", which can be found in P. H. Winston's book The Psychology of Computer Vision.) Imagin e that you are given a strange atlas of 
the USA, with all natural geological features premarked-such as rivers, mountains, lakes,

ASU's 
 
the USA, with all natural geological features premarked-such as rivers, mountains, lakes, 
and so on-but with nary a printed word. Rivers are shown as blue lines, mountains b color, and so on. Now you are told to convert it into a road atlas for a trip which you will soon make. You must neatly fill in the name s of all states, their boundaries, time zones, 
then all counties, cities, towns, all freew ays and highways and toll routes, all county 
roads, all state and national parks, campgrounds , scenic areas, dams, airports, and so on

roads, all state and national parks, campgrounds , scenic areas, dams, airports, and so on 
...  All of this must be carried out down to th e level that would appear in a detailed road 
atlas. And it must be manufactured out of your own head. You are not allowed access to 
any information which would help you for the duration of your task. 
You are told that it will pay off, in ways that will become clear at a later date, to 
make your map as true as you can. Of course , you will begin by filling in large cities and

make your map as true as you can. Of course , you will begin by filling in large cities and 
major roads, etc., which you know. And when you have exhausted your factual knowledge of an area, it will be to your advantage to use your imagination to help you reproduce at least the flavor of that area,  if not its true ge ography, by making up fake 
town names, fake populations, fake roads, fa ke parks, and so on. This arduous task will 
take

Minds and Thoughts 380months. To make things a little easier, you have a cartographer on hand to print 
everything in neatly. The end product will be your personal map of the "Alternative 
Structure of the Union"-your own personal "ASU". 
Your personal ASU will be very much like the USA in the area where you grew 
up. Furthermore, wherever your travels have chanced to lead you, or wherever you have 
perused maps with interest, your ASU will ha ve spots of striking agreement with the 
USA: a few small towns in North Dakota or  Montana, perhaps, or the whole of 
metropolitan New York, might be quite faithfully reproduced in your ASU.  
A Surprise Reversal 
 
When your ASU is done, a surprise takes place. Magically, the country you have designed comes into being, and you are trans ported there. A friendly committee presents 
you with your favorite kind of-automobile, a nd explains that, "As a reward for your 
designing efforts, you may now enjoy an all-e xpense-paid trip, at a leisurely pace, around

designing efforts, you may now enjoy an all-e xpense-paid trip, at a leisurely pace, around 
the good old A. S. of U. You may go wher ever you want, do whatever you wish to do, 
taking as long as you wish-compliments of th e Geographical Society of the ASU. And-to 
guide you around-here is a road atlas." To your surprise, you are given not the atlas which you designed, but a regular road atlas of the USA. 
When you embark on your trip, all sorts of  curious incidents will take place. A 
road atlas is being used to guide you through a country which it only partially fits. As long as you stick to major freeways, you will probably be able to cross the country

road atlas is being used to guide you through a country which it only partially fits. As long as you stick to major freeways, you will probably be able to cross the country 
without gross confusions. But the moment  you wander off into the byways of New 
Mexico or rural Arkansas, there will be adventure in store for you. The locals will not recognize any of the towns you're looking for, nor will they know the roads you're asking about. They will only know the large cities you na me, and even then the routes to those 
cities will not be the same as are indicated  on your map. It will happen occasionally that 
some of the cities which are considered huge  by the locals are nonexistent on your map of

some of the cities which are considered huge  by the locals are nonexistent on your map of 
the USA; or perhaps they exist, but their population according to the atlas is wrong by an 
order of magnitude.  
Centrality and Universality 
 
What makes an ASU and the USA, which are so  different in some ways, nevertheless so 
similar? It is that their most important c ities and routes of communication can be mapped 
onto each other. The differences between them  are found in the less frequently traveled 
routes, the cities of smaller size, and so on. No tice that this cannot be characterized either

routes, the cities of smaller size, and so on. No tice that this cannot be characterized either 
as a local or a global isomorphism. Some correspondences do extend down to the very local level-for instance, in both New Yorks, the main street may be Fifth Avenue, and 
there may be a Times Square in both as well- yet there may not be a single town that is 
found in both Montanas. So the local-global

Minds and Thoughts 381 
distinction is not relevant here. What is releva nt is the centrality of the city, in terms of 
economics, communication, transportation, etc. The more vital the city is, in one of these 
ways, the more certain it will be to occur in both the ASU and the USA. 
In this geographic analogy, one aspect is  very crucial: that there are certain 
definite, absolute points of reference which will occur in nearly all ASU's: New York, 
San Francisco, Chicago, and so on. From these it is then possible to orient oneself. In other words, if we begin comparing my ASU with yours, I can use the known agreement on big cities to establish points of referen ce with which I can communicate the location 
of smaller cities in my ASU. And if I hypot hesize a voyage from Kankakee to Fruto and

of smaller cities in my ASU. And if I hypot hesize a voyage from Kankakee to Fruto and 
you don't know where those towns are, I can refer to something we have in common, and thereby guide you. And if I talk about a voyage from Atlanta to Milwaukee, it may go along different freeways or smaller roads, but the voyage itself can still be carried out in 
both countries. And if you start describing a trip  from Horsemilk to Janzo, I can plot out

both countries. And if you start describing a trip  from Horsemilk to Janzo, I can plot out 
what seems to me to be an analogous trip in my ASU, despite not having towns by those names, as long as you constantly keep me oriented by describing your position with respect to nearby larger towns which are found in my ASU as well as in yours. 
My roads will not be exactly the same as yours, but, with our separate maps, we 
can each get from a particular part of the count ry to another. We can do this, thanks to the 
external, predetermined geological facts mountai n chains, streams, etc.-facts which were

external, predetermined geological facts mountai n chains, streams, etc.-facts which were 
available to us both as we worked on our maps. Without those external features, we 
would have no possibility of reference point s in common. For instance, if you had been 
given only a map of France, and I had been given a map of Germany, and then we had 
both filled them in in great detail, there woul d he no way to try to find "the same place" 
in our fictitious lands. It is necessary to  begin with identical external conditions-
otherwise nothing will match. 
Now that we have carried our geographical  analogy quite far, we return to the 
question of isomorphisms between brains . You might well wonder why this whole 
question of brain isomorphisms has been st ressed so much. What does it matter if two

question of brain isomorphisms has been st ressed so much. What does it matter if two 
brains are isomorphic, or quasi-isomorphic, or  not isomorphic at all? The answer is that 
we have an intuitive sense that, although othe r people differ from us in important ways, 
they are still "the same" as we are in some deep and important ways. It would be instructive to be able to pinpoint what this  invariant core of human intelligence is, and 
then to be able to describe the kinds of "embellishments" which can be added to it, making each one of us a unique embodiment of  this abstract and mysterious quality 
called "intelligence".

called "intelligence". 
In our geographic analogy, cities and towns were the analogues of symbols, while 
roads and highways were analogous to potential triggering paths. The fact that all ASU's 
have some things in common, such as the Ea st Coast, the West Coast, the Mississippi 
River, the Great Lakes, the Rockies, and many major cities and roads is analogous to the fact that we are all forced, by external rea lities, to construct certain class symbols and 
trigger

Minds and Thoughts 382ing paths in the same way. These core symbol s are like the large cities, to which everyone 
can make reference without ambiguity. (Incidentally, the fact that cities are localized entities should in no way be taken as indicative that symbols in a brain are small, almost 
point-like entities. They are merely symbolized in that manner in a network.) 
The fact is that a large proportion of every human's network of symbols is 
universal. We simply take what is common to a ll of us so much for granted that it is hard 
to see how much we have in common with othe r people. It takes the conscious effort of 
imagining how much-or how little -we have in common with other types of entities, such 
as stones, cars, restaurants, ants, and so forth, to make evident the large amount of 
overlap that we have with randomly chosen people. What we notice about another person immediately is not the standard overlap, because  that is taken for granted as soon as we

overlap that we have with randomly chosen people. What we notice about another person immediately is not the standard overlap, because  that is taken for granted as soon as we 
recognize the humanity of the other person; rather, we look beyond the standard overlap 
and generally find some major differences , as well as some unexpected, additional 
overlap. 
Occasionally, you find that another pers on is missing some of what you thought 
was the standard, minimal core-as if Chica go were missing from their ASU, which is 
almost unimaginable. For instance, someone might not know what an elephant is, or who is President, or that the earth is round. In su ch cases, their symbolic network is likely to

overlap. 
almost unimaginable. For instance, someone might not know what an elephant is, or who is President, or that the earth is round. In su ch cases, their symbolic network is likely to 
be so fundamentally different from your own that significant communication will be 
difficult. On the other hand, perhaps this same person will share some specialized kind of knowledge with you-such as expertise in the game of dominoes-so that you can communicate well in a limited domain. This w ould be like meeting someone who comes 
from the very same rural area of North Dakota as you do, so that your two ASU's coincide in great detail over a very small re gion, which allows you to describe how to get 
from one place to another very fluently.

from one place to another very fluently.  
How Much Do Language and Culture Channel Thought? 
 
If we now go back to comparing our own sym bol network with those of a Frenchman and 
a German, we can say that we expect them to  have the standard core of class symbols, 
despite the fact of different native langua ges. We do not expect to share highly 
specialized networks with them, but we do not expect such sharing with a randomly 
chosen person who shares our native language , either. The triggering patterns of people

chosen person who shares our native language , either. The triggering patterns of people 
with other languages will be somewhat different from our own, but still the major class symbols, and the major routes between them, w ill be universally available, so that more 
minor routes can be described with reference to them. 
Now each of our three people may in addition have some command of the 
languages of the other two. What is it that ma rks the difference between true fluency, and

languages of the other two. What is it that ma rks the difference between true fluency, and 
a mere ability to communicate? First of all, someone fluent in English uses most words at roughly their- regular frequencies. A non-nativ e speaker will have picked up some words 
from

Minds and Thoughts 383 
dictionaries, novels, or classes-words which at some time may have been prevalent or 
preferable, but which are now far down in fr equency-for example, "fetch" instead of 
"get", "quite" instead of "very", etc. T hough the meaning usually comes through, there is 
an alien quality transmitted by the unusual choice of words. 
But suppose that a foreigner learns to  use all words at roughly the normal 
frequencies. Will that make his speech trul y fluent? Probably not. Higher than the word 
level, there is an association level, which is attached to the culture as a whole-its history, 
geography, religion, children's stories, litera ture, technological level, and so on. For 
instance, to be able to speak modern Hebrew absolutely fluently, you need to know the Bible quite well in Hebrew, because the langua ge draws on a stock of biblical phrases

instance, to be able to speak modern Hebrew absolutely fluently, you need to know the Bible quite well in Hebrew, because the langua ge draws on a stock of biblical phrases 
and their connotations. Such an association level permeates each language very deeply. Yet there is room for all sorts of variety in side fluency-otherwise the only truly fluent 
speakers would be people whose thoughts were the most stereotyped possible! 
Although we should recognize the depth to  which culture affects thought, we 
should not overstress the role of language in  molding thoughts. For instance, what we 
might call two "chairs" might be perceived by a speaker of French as objects belonging to

might call two "chairs" might be perceived by a speaker of French as objects belonging to 
two distinct types: "chaise" and "fauteuil" ("chair" and "armchair"). People whose native 
language is French are more aware of that difference than we are-but then people who 
grow up in a rural area are more aware of , say, the difference between a pickup and a 
truck, than a city dweller is. A city dwelle r may call them both "trucks". It is not the 
difference in native language, but the difference in  culture (or subculture), that gives rise 
to this perceptual difference. 
The relationships between the symbols of people with different native languages 
have every reason to be quite similar, as fa r as the core is concerned, because everyone

have every reason to be quite similar, as fa r as the core is concerned, because everyone 
lives in the same world. When you come down to more detailed aspects of the triggering 
patterns, you will find that there is less in  common. It would he like comparing rural 
areas in Wisconsin in ASU's which had been  made up by people who had never lived in 
Wisconsin. This will be quite irrelevant, however, as long as there is sufficient agreement on the major cities and major routes, so that  there are common points of reference all 
over the map.   
Trips and Itineraries in ASU's

Trips and Itineraries in ASU's 
 
Without making it explicit, I have been usi ng an image of what a "thought" is in the 
ASU-analogy-namely, I have been implying that a thought corresponds to a trip. The 
towns which are passed through represent the symbols which are excited. This is not a 
perfect analogy, but it is quite strong. One probl em with it is that when a thought recurs 
in someone's mind sufficiently often, it can ge t chunked into a single concept. This would

in someone's mind sufficiently often, it can ge t chunked into a single concept. This would 
correspond to quite a strange event in an ASU: a commonly taken trip would become, in some strange fashion, a new town or city! If one is to continue to use the ASU-metaphor, 
then, it is important to remember that

Minds and Thoughts 384the cities represent not only the elementary sy mbols, such as those for "grass", "house", 
and "car", but also symbols which get creat ed as a result of the chunking ability of a 
brain-symbols for such sophisticated concepts as "crab canon", "palindrome", or "ASU". 
Now if it is granted that the notion of ta king a trip is a fair counterpart to the 
notion of having a thought. then the followi ng difficult issue comes up: virtually any 
route leading from one city to a second, then to a third, and so on, can be imagined, as long as one remembers that some interveni ng cities are also passed through. This would 
correspond to the activation of an arbitrary sequence of symbols, one after another, 
making allowance for some extra symbols-those which lie en route. Now if virtually any 
sequence of symbols can be activated in any de sired order, it may seem that a brain is an 
indiscriminate system, which can absorb or  produce any thought whatsoever. But we all

indiscriminate system, which can absorb or  produce any thought whatsoever. But we all 
know that that is not so. In fact, there are certain kinds of thoughts which we call 
knowledge, or beliefs, which play quite a di fferent role from random fancies, or 
humorously entertained absurdities. How can  we characterize the difference between 
dreams, passing thoughts, beliefs, and pieces of knowledge?  
Possible, Potential, and Preposterous Pathways 
 
There are some pathways-you can think of them as pathways either in an ASU or in a brain-which are taken routinely in going fr om one place to another. There are other

There are some pathways-you can think of them as pathways either in an ASU or in a brain-which are taken routinely in going fr om one place to another. There are other 
pathways which can only be followed if one is led through them by the hand. These pathways are "potential pathways", which would be followed only if special external circumstances arose. The pathways which one relies on over and over again are pathways which incorporate knowledge-and here I mean  not only knowledge of facts (declarative 
knowledge), but also knowledge of how-to' s (procedural knowledge). These stable, 
reliable pathways are what constitute know ledge. Pieces of knowledge merge gradually

reliable pathways are what constitute know ledge. Pieces of knowledge merge gradually 
with beliefs, which are also represented by reliable pathways, but perhaps ones which are 
more susceptible to replacement if, so to sp eak, a bridge goes out, or there is heavy fog. 
This leaves us with fancies, lies, falsities,  absurdities, and other variants. These would 
correspond to peculiar routes such as: New Yo rk City to Newark via Bangor, Maine and 
Lubbock, Texas. They are indeed possible path ways, but ones which are not likely to be 
stock routes, used in everyday voyages.

stock routes, used in everyday voyages. 
A curious, and amusing, implication of this model is that all of the "aberrant" 
kinds of thoughts listed above are composed, at rock bottom, completely out of beliefs or 
pieces of knowledge. That is, any weird and sn aky indirect route breaks up into a number 
of non-weird, non-snaky direct stretches, a nd these short, straightforward symbol-

connecting routes represent simple thoughts that one can rely on-beliefs and pieces of knowledge. On reflection, this is hardly surprising, however, since it is quite reasonable that we should only be able to imagine fic titious things that are somehow grounded in the 
realities we have experienced, no matter how

Minds and Thoughts 385 
wildly they deviate from them. Dreams are perhaps just such random meanderings about 
the ASU's of our minds. Locally, they make sense-but globally ...   
Different Styles of Translating Novels 
 
A poem like `Jabberwocky" is like an unreal  journey around an ASU,  hopping from one 
state to another very quickly, following very curious routes. The translations convey this 
aspect of the poem, rather than the preci se sequence of symbols which are triggered, 
although they do their best in that respect. In  ordinary prose, such leaps and bounds are 
not so common. However, similar problems of translation do occur. Suppose you are 
translating a novel from Russian to Englis h, and come across a sentence whose literal 
translation is, "She had a bowl of borscht."  Now perhaps many of your readers will have 
no idea what borscht is. You could attempt to replace it by the "corresponding" item in 
their culture-thus, your translation might r un, "She had a bowl of Campbell's soup." Now

their culture-thus, your translation might r un, "She had a bowl of Campbell's soup." Now 
if you think this is a silly exaggeration, take  a look at the first sentence of Dostoevsky's 
novel Crime and Punishment in Russian and then  in a few different English translations. I 
happened to look at three different English paperback translations, and found the following curious situation. 
The first sentence employs the street name  "S. Pereulok" (as transliterated). What 
is the meaning of this? A careful reader  of Dostoevsky's work who knows Leningrad 
(which used to be called "St. Petersburg" -or should I say "Petrograd"?) can discover by

(which used to be called "St. Petersburg" -or should I say "Petrograd"?) can discover by 
doing some careful checking of the rest of the geography in the book (which incidentally 
is also given only by its initials) that the st reet must be "Stoliarny Pereulok". Dostoevsky 
probably wished to tell his story in a realis tic way, yet not so realistically that people 
would take literally the addresses at which crimes and other events were supposed to 
have occurred. In any case, we have a translation problem; or to be more precise, we have several translation problems, on several different levels.

have occurred. In any case, we have a translation problem; or to be more precise, we have several translation problems, on several different levels. 
First of all, should we keep the initial so as to reproduce the aura of semi-mystery 
which appears already in this first sentence of the book? We would get "S. Lane" ("lane" 
being the standard translation of "pereulok"). None of the three translators took this tack. 
However, one chose to write "S. Place". The translation of Crime and Punishment which 
I read in high school took a similar option. I will never forget the disoriented feeling I 
experienced when I began reading the novel and encountered those streets with only

experienced when I began reading the novel and encountered those streets with only 
letters for names. I had some sort of inta ngible malaise about the beginning of the book; I 
was sure that I was missing something essen tial, and yet I didn't know what it was ... I 
decided that all Russian novels were very weird. 
Now we could be frank with the reader  (who, it may be assumed, probably won't 
have the slightest idea whether the street is real or fictitious anyway!) and give him the advantage of our modern scholarship, writing

Minds and Thoughts 386"Stoliarny Lane" (or "Place"). This was the choice of translator number 2, who gave the 
translation as "Stoliarny Place". 
What about number 3? This is the most interesting of all. This translation says 
"Carpenter's Lane". And why not, indeed? Afte r all, "stoliar" means "carpenter" and "ny" 
is an adjectival ending. So now we might  imagine ourselves in London, not Petrograd, 
and in the midst of a situation invented by Dickens, not Dostoevsky. Is that what we 
want-, Perhaps we should just read a novel by Dickens instead, with the justification that 
it is "the corresponding work in English". When  viewed on a sufficiently high level, it is 
a "translation" of the Dostoevsky novel-in  fact, the best possible one! Who needs 
Dostoevsky? 
We have come all the way from attempts at great literal fidelity to the author's 
style, to high-level translations of flavor . Now if this happens already in the first

style, to high-level translations of flavor . Now if this happens already in the first 
sentence, can you imagine how it must go on in  the rest of the book? What about the 
point where a German landlady begins shou ting in her German-style Russian% How do 
you translate broken Russian spoken with a German accent, into English? 
Then one may also consider the problems of how to translate slang and colloquial 
modes of expression. Should one search for an "analogous" phrase, or should one settle 
for a word-by-word translation? If you sear ch for an analogous phrase, then you run the

for a word-by-word translation? If you sear ch for an analogous phrase, then you run the 
risk of committing a "Campbell's soup" type  of blunder; but if you translate every 
idiomatic phrase word by word, then the English will sound alien. Perhaps this is 
desirable, since the Russian culture is an a lien one to speakers of English. But a speaker 
of English who reads such a translation will constantly be experiencing, thanks to the 
unusual turns of phrase, a sense-an artificial sense-of strangeness, which was not intended

unusual turns of phrase, a sense-an artificial sense-of strangeness, which was not intended 
by the author, and which is not experienced by readers of the Russian original. 
Problems such as these give one pause in considering such statements as this one, 
made by Warren Weaver, one of the first advocat es of translation by computer, in the late 
1940's: "When I look at an article in Russian, I say, 'This is really written in English, but 
it has been coded in some strange symbols.  I will now proceed to decode."" Weaver's 
remark simply cannot be taken literally; it must  rather be considered a provocative way of

remark simply cannot be taken literally; it must  rather be considered a provocative way of 
saying that there is an objectively describable meaning hidden in the symbols, or at least 
something pretty close to objective; theref ore, there would be no reason to suppose a 
computer could not ferret it out, if sufficiently well programmed.  
High-Level Comparisons between Programs 
 
Weaver's statement is about translations between different natural languages. Let's 
consider now the problem of translating be tween two computer languages. For instance, 
suppose two people have written programs whic h run on different computers, and we

suppose two people have written programs whic h run on different computers, and we 
want to know if the two programs carry out  the same task. How can we find out? We 
must compare the programs. But on what level should this be done? Perhaps one program

Minds and Thoughts 387 
mer wrote in a machine language, the other in a compiler language. Are two such programs comparable? Certainly. But how to  compare them? One way might be to 
compile the compiler language program, produci ng a program in the machine language of 
its home computer. 
Now we have two machine language progr ams. But there is another problem: 
there are two computers, hence two diffe rent machine languages-and they may be 
extremely different. One machine may have sixteen-bit words; the other thirty-six-bit 
words. One machine may' have built-in stack-handling instructions (pushing and popping), while the other lacks them. The differences between the hardware of the two machines may make the two machine language programs seem incomparable-and yet we suspect they are performing the same task, and we would Iike to see that at a glance. We 
are obviously looking at the programs from much too close a distance. 
What we need to do is to step back, away from machine language, towards a

What we need to do is to step back, away from machine language, towards a 
higher, more chunked view. From this vantage point, we hope we will be able to perceive 
chunks of program which make each program seem rationally planned out on a global, 
rather than a local, scale-that is, chunks which fit together in a way that allows one to perceive the goals of the programmer. Let us  assume that both programs were originally 
written in high-level languages. Then some chunking has already been done for us. But 
we will run into other troubles. There is a pr oliferation of such languages: Fortran, Algol,

we will run into other troubles. There is a pr oliferation of such languages: Fortran, Algol, 
LISP, APL, and many others. How can you co mpare a program written in APL with one 
written in Algol: Certainly not by matching them up line by line. You will again chunk 
these programs in your mind, looking for con ceptual, functional units which correspond. 
Thus, you are not comparing hardware, you are not comparing software-you are comparing "etherware"-the pure concepts which lie back of the software. There is some 
sort of abstract "conceptual skeleton" which must be lifted out of low levels before you

sort of abstract "conceptual skeleton" which must be lifted out of low levels before you 
can carry out a meaningful comparison of tw o programs in different computer languges, 
of two animals, or of two sentences in different natural languages. 
Now this brings us back to an earlier question which we asked about computers 
and brains: How can we make sense of a low-le vel description of a computer or a brain? 
Is there, in any reasonable sense, an objectiv e way to pull a high-level description out of a 
low-level one, in such complicated systems? In the case of a computer, a full display of

low-level one, in such complicated systems? In the case of a computer, a full display of 
the contents of memory-a so-called memory dump-is easily available. Dumps were commonly printed out in the early days of computing, when something went wrong with 
a program. Then the programmer would ha ve to go home and pore over the memory 
dump for hours, trying to understand what each  minuscule piece of memory represented. 
In essence, the programmer would be doing the opposite of what a compiler does: he 
would be translating from machine language into a higher-level language, a conceptual

would be translating from machine language into a higher-level language, a conceptual 
language. In the end, the programmer would understand the goals of the program and 
could describe it in high-level terms-for exam ple, "This program translates novels front 
Russian to English", or "This program co mposes an eight-voice fugue based on any 
theme which is fed in".

Minds and Thoughts 388High-Level Comparisons between Brains 
 
Now our question must be investigated in the case of brains. In this case, we are asking, 
"Are people's brains also capable of being 'r ead', on a high level? Is there some objective 
description of the content of a brain?" In the Ant Fugue, the Anteater claimed to be able to tell what Aunt Hillary was thinking a bout, by looking at the scurryings of her 
component ants. Could some superbeing-a Neuroneater, perhaps-conceivably look down 
on our neurons, chunk what it sees, and come up with an analysis of our thoughts? 
Certainly the answer must be yes, since we are all quite able to describe, in 
chunked (i.e., non-neural) terms, the activity of our minds at any given time. This means that we have a mechanism which allows us  to chunk our own brai n state to some rough 
degree, and to give a functional description of  it. To be more precise, we do not chunk all 
of the brain state-we only chunk those porti ons of it which are active. However, if

of the brain state-we only chunk those porti ons of it which are active. However, if 
someone asks us about a subject which is code d in a currently inactive area of our brain, 
we can almost instantly gain access to the appropriate dormant area and come up with a 
chunked description of it-that is, some belief on that subject. Note that we come back 
with absolutely zero information on the neural level of that part of the brain: our description is so chunked that we don't even have any idea what part of our brain it is a description of. This can be contrasted w ith the programmer whose chunked description

with absolutely zero information on the neural level of that part of the brain: our description is so chunked that we don't even have any idea what part of our brain it is a description of. This can be contrasted w ith the programmer whose chunked description 
comes from conscious analysis of every part of the memory dump. 
Now if a person can provide a chunked desc ription of any part of his own brain, 
why shouldn't an outsider too, given some nonde structive means of access to the same 
brain, not only be able to chunk limited portions  of the brain, but actually to give a 
complete chunked description of it-in other words, a complete documentation of the beliefs of the person whose brain is accessibl e? It is obvious that such a description

complete chunked description of it-in other words, a complete documentation of the beliefs of the person whose brain is accessibl e? It is obvious that such a description 
would have an astronomical size, but that is not  of concern here. We are interested in the 
question of whether, in principle, there exis ts a well-defined, highlevel description of a 
brain, or whether, conversely, the neuron-level description-or something equally physiological and intuitively unenlightening-is the best description that in principle 
exists. Surely, to answer this question would be of the highest importance if we seek to 
know whether we can ever understand ourselves.

know whether we can ever understand ourselves. 
 
Potential Beliefs, Potential Symbols 
 
It is my contention that a chunked description is possible, but when we get it, all will not 
suddenly be clear and light. The problem is th at in order to pull a chunked description out 
of the brain state, we need a language to de scribe our findings. Now the most appropriate 
way to describe a brain, it would seem, woul d be to enumerate the kinds of thoughts it 
could entertain, and the kinds of thoughts it could not entertain-or, perhaps, to enumerate

could entertain, and the kinds of thoughts it could not entertain-or, perhaps, to enumerate 
its beliefs and the things which it does not believe. If that is the

Minds and Thoughts 389 
kind of goal we will be striving for in a chunke d description, then it is easy to see what 
kinds of troubles we will run up against. 
Suppose you wanted to enumerate all possibl e voyages that could be taken in an 
ASU; there are infinitely many. How do you de termine which ones are plausible, though? 
Well, what does "plausible" mean? We will have  precisely this kind of difficulty in trying 
to establish what a "possible pathway" from  symbol to symbol in a brain is. We can 
imagine an upsidedown dog flying through the air with a cigar in its mouth-or a collision between two giant fried eggs on a freeway-or any number of other ridiculous images. The number of far-fetched pathways which can be followed in our brains is without bound, 
just as is the number of insane itineraries that  could be planned on an  ASU. But just what 
constitutes a "sane" itinerary, given an ASU?  And just what constitutes a "reasonable"

constitutes a "sane" itinerary, given an ASU?  And just what constitutes a "reasonable" 
thought, given a brain state? The brain state itself does not forbid anv pathway, because 
for any pathway there are always circumstan ces which could force the following of that 
pathway. The physical status of a brain, if read correctly, gives information telling not 
which pathways could be followed, but rath er how much resistance would be offered 
along the way. 
Now in an ASU, there are many trips wh ich could be taken along two or more 
reasonable alternative routes. For example, the trip from San Francisco to New York

reasonable alternative routes. For example, the trip from San Francisco to New York 
could go along either a northern route or a southern route. Each of them is quite 
reasonable, but people tend to take them unde r different circumstances. Looking at a map 
at a given moment in time does not tell you anything about which route will be preferable at some remote time in the future-that depends on the external circumstances under which the trip is to be taken. Likewise, the "readi ng" of a brain state will reveal that several

at a given moment in time does not tell you anything about which route will be preferable at some remote time in the future-that depends on the external circumstances under which the trip is to be taken. Likewise, the "readi ng" of a brain state will reveal that several 
reasonable alternative pathways are often available, connecting a given set of symbols. However, the trip among these symbols need not be imminent; it may be simply one of 
billions of "potential" trips, all of which figure in the readout of the brain state. From this 
follows an important conclusion: there is no information in the brain state itself which 
tells which route will be chosen. The external  circumstances will play a large determining 
role in choosing the route. 
What does this imply? It implies that thoughts which clash totally may be

What does this imply? It implies that thoughts which clash totally may be 
produced by a single brain, depending on the ci rcumstances. And any high-level readout 
of the brain state which is worth its salt must contain all such conflicting versions. 
Actually this is quite obvious-that we all are bundles of contradictions, and we manage to 
hang together by bringing out only one side of  ourselves at a given time. The selection 
cannot be predicted in advance, because the conditions which will force the selection are not known in advance. What the brain state can  provide, if properly read, is a conditional 
description of the selection of routes.

description of the selection of routes. 
Consider, for instance, the Crab's plight, described in the Prelude . He can react in 
various ways to the playing of a piece of musi c. Sometimes he will be nearly immune to 
it, because he knows it so well. Other times, he will  be quite excited by it, but this 
reaction requires the right kind of  triggering from the outside-for instance, the presence of 
an enthusiastic listener, to

Minds and Thoughts 390whom the work is new. Presumably, a high-le vel reading of the Crab's brain state would 
reveal the potential thrill (and conditions whic h would induce it), as well as the potential 
numbness (and conditions which would induce it) . The brain state itself would not tell 
which one would occur on the next hearing of the piece, however: it could only say, "If 
such-&-such conditions obtain, then a thrill will result; otherwise ..." 
Thus a chunked description of a brain st ate would give a catalogue of beliefs 
which could be evoked conditionally, dependent  on circumstances. Since not all possible 
circumstances can be enumerated, one would have to settle for those which one thinks are "reasonable". Furthermore, one would have to settle for a chunked description of the 
circumstances themselves, since they obviously cannot-and should not-be specified down to the atomic level! Therefore, one will not be able to make an exact, deterministic

circumstances themselves, since they obviously cannot-and should not-be specified down to the atomic level! Therefore, one will not be able to make an exact, deterministic 
prediction saying which beliefs will be pulle d out of the brain state by a given chunked 
circumstance. In summary, then, a chunked desc ription of a brain state will consist of a 
probabilistic catalogue, in which are listed t hose beliefs which are most likely to be 
induced (and those symbols which are most likely to be activated) by various sets of "reasonably likely" circumstances, themselves  described on a chunked level. Trying to 
chunk someone's beliefs without referring to c ontext is precisely as silly as trying to

chunk someone's beliefs without referring to c ontext is precisely as silly as trying to 
describe the range of a single person's "potential progeny" without referring to the mate. 
The same sorts of problems arise in e numerating all the symbols in a given 
person's brain. There are potentially not only an  infinite number of pathways in a brain, 
but also an infinite number of symbols. As was pointed out, new concepts can always be 
formed from old ones, and one could argue that the symbols which represent such new 
concepts are merely dormant symbols in each  individual, waiting to be awakened. They

concepts are merely dormant symbols in each  individual, waiting to be awakened. They 
may never get awakened in the person's lifetime, but it could be claimed that those symbols are nonetheless always there, just wa iting for the right circumstances to trigger 
their synthesis. However, if the probability is very low, it would seem that "dormant" 
would be a very unrealistic term to apply in  the situation. To make this clear, try to 
imagine all the "dormant dreams" which are sitting there inside your skull while you're 
awake. Is it conceivable that there exis ts a decision procedure which could tell

awake. Is it conceivable that there exis ts a decision procedure which could tell 
"potentially dreamable themes" from "undreamable themes", given your brain State   
Where Is the Sense of Self? 
 
Looking back on what we have discusse d, you might think to yourself, "These 
speculations about brain and mind are all well and good, but what about the feelings 
involved in consciousness, These symbols may trigger each other all they want, but unless someone perceives the whole thing, there's no consciousness." 
This makes sense to our intuition on some level, but it does not make much sense 
logically. For we would then be compelled to look for an

Minds and Thoughts 391explanation of the mechanism which does the pe rceiving of all the active symbols, if it is 
not covered by what we have described so far.  Of course, a "soulist" would not have to 
look any further-he would merely assert that th e perceiver of all this neural action is the 
soul, which cannot be described in physical terms, and that is that. However, we shall try to give a "nonsoulist" explanation of where consciousness arises. 
Our alternative to the soulist explanation-and a disconcerting one it is, too- is to 
stop at ohe symbol level and say, "This is it- this is what consciousness is. Consciousness 
is that property of a system that arises when ever there exist symbols in the system which 
obey triggering patterns somewhat like the ones described in the past several sections." 
Put so starkly, this may seem inadequate. How does it account for the sense of "I", the sense of' self?  
Subsystems 
 
There is no reason to expect that "I", or "the self"', should not be represented by a

There is no reason to expect that "I", or "the self"', should not be represented by a 
symbol. In fact, the symbol for the self is probably the most complex of all the symbols 
in the brain. For this reason, I choose to put it on a new level of the hierarchy and call it a 
subsystem, rather than a symbol. To be pr ecise, by "subsystem", I mean a constellation of 
symbols, each of which can be separately activated under the control of the subsystem 
itself. The image I wish to convey of a subs ystem is that it functions almost as an 
independent "subbrain", equipped with its own repertoire of symbols which can trigger

independent "subbrain", equipped with its own repertoire of symbols which can trigger 
each other internally. Of course, there is also much communication between the subsystem and the "outside" world-that is, the rest of the brain. "Subsystem" is just another name for an overgrown symbol, one which has gotten so complicated that it has 
many subsymbols which interact among themse lves. Thus, there is no strict level 
distinction between symbols and subsystems. 
Because of the extensive links between a subsystem and the rest of the brain 
(some of which will be described shortly), it would be very difficult to draw a sharp

(some of which will be described shortly), it would be very difficult to draw a sharp 
boundary between the subsystem and the outside; but even if the border is fuzzy, the subsystem is quite a real thing. The intere sting thing about a subsystem is that, once 
activated and left to its own devices, it can work on its own. Thus, two or more 
subsystems of the brain of an individual ma y operate simultaneously. I have noticed this 
happening on occasion in my own brain: some times I become aware that two different 
melodies are running through my mind, comp eting for "my" attention. Somehow, each 
melody is being manufactured, or "played", in a separate compartment of my brain. Each 
of the systems responsible for drawing a mel ody out of my brain is presumably activating

of the systems responsible for drawing a mel ody out of my brain is presumably activating 
a number of symbols, one after another, comp letely oblivious to the other system doing 
the same thing. Then they both attempt to communicate with a third subsystem of my 
brain-mv self'-symbol- and it is at that point that the "1" inside my brain gets wind of what's going on: in other words, it starts pi cking up a chunked description of the activities 
of those two subsystems.

Minds and Thoughts 392Subsystems and Shared Code 
 
Typical subsystems might be those that re present the people we know intimately. They 
are represented in such a complex way in our brains that their symbols enlarge to the rank 
of subsystem, becoming able to act autonom ously, making use of some resources in our 
brains for support. By this, I mean that a subsystem symbolizing a friend can activate 
many of the symbols in my brain just as I can. For instance, I can fire up my subsystem 
for a good friend and virtually feel myself in his shoes, running through thoughts which 
he might have, activating symbols in sequences which reflect his thinking patterns more accurately than my own. It could be said that  my model of this friend, as embodied in a 
subsystem of my brain, constitutes my own chunked description of his brain. 
Does this subsystem include, then, a symbol  for every symbol which I think is in

Does this subsystem include, then, a symbol  for every symbol which I think is in 
his brain? That would be redundant. Probably the subsystem makes extensive use of symbols already present in my brain. For inst ance, the symbol for "mountain" in my brain 
can be borrowed by the subsystem, when it is activated. The way in which that symbol is 
then used by the subsystem will not necessarily be identical to the way it is used by my full brain. In particular, if I am talking with my friend about the Tien Shan mountain range in Central Asia (neither of us having been there), and I know that a number of years

then used by the subsystem will not necessarily be identical to the way it is used by my full brain. In particular, if I am talking with my friend about the Tien Shan mountain range in Central Asia (neither of us having been there), and I know that a number of years 
ago he had a wonderful hiking experience in the Alps, then my interpretation of his remarks will be colored in part by my imported images of his earlier Alpine experience, since I will be trying to imagine how he visualizes the area. 
In the vocabulary we have been building up in this Chapter, we could say that the 
activation of" the "mountain" symbol in me is under control of my subsystem 
representing him. The effect of this is to open up a different window onto to my memories from the one which I normally use- namely, my "default option" switches from 
the full range of my memories to the set of my memories of his memories. Needless to

the full range of my memories to the set of my memories of his memories. Needless to 
say, my representations of his memories ar e only approximations to his actual memories, 
which are complex modes of activation of the symbols in his brain, inaccessible to me. 
My representations of his memories are also complex modes of activation of my 
own symbols-those for "primordial" concepts, such as grass, trees, snow, sky, clouds, and 
so on. These are concepts which I must assume are represented in him "identically" to the 
way they are in me. I must also assume a similar representation in him of even more

way they are in me. I must also assume a similar representation in him of even more 
primordial notions: the experiences of grav ity, breathing, fatigue, color, and so forth. 
Less primordial but perhaps a nearly universal human quality is the enjoyment of reaching a summit and seeing a view. Therefor e, the intricate processes in my brain 
which are responsible for this enjoyment can be taken over directly by the friend-subsystem without much loss of fidelity. 
We could go on to attempt to describe how  I understand an entire tale told by my 
friend, a tale filled with many complexitie s of human relationships and mental

friend, a tale filled with many complexitie s of human relationships and mental 
experiences. But our terminology would quick ly become inadequate. There would be 
tricky recursions connected with representa

Minds and Thoughts 393 
tions in him of representations in me of repr esentations in him of one thing and another. 
If' mutual friends figured in the tale being told, I would unconsciously look for 
compromises between my image of hi s representations of them, and my own images of 
them. Pure recursion would simply be an inappropriate formalism for dealing with 
symbol amalgams of this type. And I have barely scratched the surface!  We plainly lack the vocabulary today for de scribing the complex interactions that are 
possible between symbols. So let us stop before we get bogged down. 
We should note, however, that computer systems are beginning to run into some 
of the some kinds of complexity, and theref ore some of these notions have been given 
names. For instance, my "mountain" symbol is analogous to what in computer jargon is called shared (or reentrant) codecode whic h can be used by two or more separate 
timesharing programs running on a single comput er. The fact that activation of one

timesharing programs running on a single comput er. The fact that activation of one 
symbol can have different results when it is pa rt of different subsystems can be explained 
by saying that its code is being processed by different interpreters. Thus, the triggering 
patterns in the "mountain" symbol are not absolu te; they are relative to the system within 
which the symbol is activated. 
The reality of such "subbrains" may seem doubtful to some. Perhaps the 
following quote from M. C. Escher, as he di scusses how he creates his periodic plane-

filling drawings, will help to make clear what kind of phenomenon I am referring to:  
 
While drawing I sometimes feel as if' I were a spiritualist medium, controlled 
by the creatures which I am conjuring up. It is as if they themselves decide on the shape in which they choose to a ppear. They take little account of my 
critical opinion during their birth and I cannot exert much influence on the measure of their development. They are usually very difficult and obstinate 
creatures

creatures 
 
Here is a perfect example of the nea r-autonomy of certain subsystems of the 
brain, once they are activated. Escher's subsystems seemed to him almost to be able to override his esthetic judgment. Of course, this  opinion must be taken with a grain of salt, 
since those powerful subsystems came into being as a result of his many years of training and submission to precisely the forces that mold ed his esthetic sensitivities. In short, it is 
wrong to divorce the subsystems in Escher's brain from Escher hi mself or from his

wrong to divorce the subsystems in Escher's brain from Escher hi mself or from his 
esthetic judgment. They constitute a vital part  of his esthetic sense, where "he" is the 
complete being of the artist.  
The Self-Symbol and Consciousness 
 
A very important side effect of the self-subsyste m is that it can play th e role of "soul", in 
the following sense: in communicating constan tly with the rest of the subsystems and 
symbols in the brain, it keeps track of what symbols are active, and in what way. This 
means that it has to have symbols for me ntal activity-in other words, symbols for 
symbols, and symbols for the actions of symbols.

Minds and Thoughts 394Of course, this does not elevate consci ousness or awareness to any "magical", 
nonphysical level. Awareness here is a direct  effect of the complex hardware and 
software we have described. Still, despite its earthly origin, this way of describing 
awareness-as the monitoring of brain activity by a subsystem of the brain itself-seems to 
resemble the nearly indescribable sensati on which we all know and call "consciousness". 
Certainly one can see that the complexity here is enough that many unexpected effects 
could be created. For instance, it is quite plau sible that a computer program with this kind 
of structure would make statements about itself which would have a great deal of 
resemblance to statements which people commonly make about themselves. This includes insisting that it has free will, that it is  not explicable as a "sum of its parts", and 
so on. (On this subject, see the article "Ma tter, Mind, and Models" by M. Minsky in his 
book Semantic Information Processing.)

book Semantic Information Processing.) 
What kind of guarantee is there that a subs ystem, such as I have here postulated, 
which represents the self, actually exists in  our brains? Could a whole complex network 
of symbols such as has been described a bove evolve without a self-symbol evolving, 
How could these symbols and their activities play  out "isomorphic" mental events to real 
events in the surrounding universe, if there we re no symbol for the host organism, All the 
stimuli coming into the system are centered on one small mass in space. It would be quite

stimuli coming into the system are centered on one small mass in space. It would be quite 
a glaring hole in a brain's symbolic structure not to have a symbol for the physical object 
in which it is housed, and which plays a larger role in the events it mirrors than any other 
object. In fact, upon reflection, it seems that the only way one could make sense of the 
world surrounding a localized animate object is to understand the role of that object in 
relation to the other objects around it. This nece ssitates the existence of a selfsymbol; and 
the step from symbol to subsystem is merely  a reflection of the importance of the self-

symbol', and is not a qualitative change.  
Our First Encounter with Lucas 
 
The Oxford philosopher J. R. Lucas (not c onnected with the Lucas numbers described 
earlier) wrote a remarkable article in 1961, entitled "Minds, Machines, and Gödel". His views are quite opposite to mine, and yet he manages to mix many of the same ingredients together in coming up with hi s opinions. The following excerpt is quite 
relevant to what we have just been discussing:  
At one's first and simplest attempts to philo sophize, one becomes entangled in questions of

At one's first and simplest attempts to philo sophize, one becomes entangled in questions of 
whether when one knows something one knows th at one knows it, and what, when one is 
thinking of oneself, is being thought about, a nd what is doing the thinking. After one has 
been puzzled and bruised by this problem for a long time, one learns not to press these questions: the concept of a conscious being is, implicitly, realized to be different from that 
of an unconscious object. In saying that a conscious being knows something, we are saying

of an unconscious object. In saying that a conscious being knows something, we are saying 
not onh that he knows it, but that he knows th at he knows it, and that he knows that he 
knows that he knows it, and so on, as long as we care to pose the

Minds and Thoughts 395question: there is, we recognize, an infinity here , but it is not an infinite  regress in the had 
sense, for it is the questions that peter out, as being pointless, rather than the answers. The 
questions are felt to be pointless because the concept contains within itself the idea of 
being able to go on answering such questions indefinitely. Although c onscious beings have 
the power of going on, we do not wish to exhibit this simply as a succession of tasks they are able to perform, nor do we see the mind as an infinite sequence of selves and super-selves and super-super-selves. Rather, we insi st that a conscious being is a unity, and 
though we talk about parts of the mind, we (to so only as a metaphor, and will not allow it 
to be taken literally. 
The paradoxes of consciousness arise because a conscious being can be aware of itself  
as well as of other things, and yet cannot really be  construed as being divisible into parts. It

as well as of other things, and yet cannot really be  construed as being divisible into parts. It 
means that a conscious being can deal with  Gödelian questions in a was in which a 
machine cannot, because a conscious being can both consider itself and its perform a rice and vet not be other than that which did th e performance. A machine can be made in a 
manner of speaking to "consider" its perform ance, but it cannot take this "into account" 
without thereby becoming a different machine, namely the old machine with a "new part" 
added. Btu it is inherent in our idea of a conscious mind that it can reflect upon itself and 
criticize its own performances, and no extra part  is required to (to this: it is already 
complete, and has no Achilles' heel.

complete, and has no Achilles' heel. 
The thesis thus begins to become more of  a matter of conceptual analysis than 
mathematical discovery. This is borne out by  considering another argument put forward by 
Turing. So far, we have constructed only fair ly simple and predictable artifacts. When we 
increase the complexity of our machines, there may, perhaps, be surprises in store for us. 
He draws a parallel with a fission pile. Below a certain "critical" size, nothing much happens: but above the critical size, the sparks be gin to fly. So too, perhaps, with brains

He draws a parallel with a fission pile. Below a certain "critical" size, nothing much happens: but above the critical size, the sparks be gin to fly. So too, perhaps, with brains 
and machines. Most brains and all machines ar e, at present, sub-critical"-they react to 
incoming stimuli in a stodgy and uninteresting way, have no ideas of their own, can produce only stock responses-but a few brains at present, and possibly some machines in the future, are super-critical, and scintillate on their own account. Turing is suggesting that 
it is only a matter of complexity, and that a bove a certain level of complexity a qualitative 
difference appears, so that "super-critical" machines will be quite unlike the simple ones 
hitherto envisaged.

hitherto envisaged. 
This may be so. Complexity often does introduce qualitative differences. Although it 
sounds implausible, it might turn out that above a certain level of complexity, a machine ceased to be predictable, even in principle,  and started doing things on its own account, or, 
to use a very revealing phrase, it might begin to  have a mind of its own. It might begin to 
have a mind of its own. It would begin to have a mind of its own when it was no longer 
entirely predictable and entirely docile, but  was capable of doing things which we

entirely predictable and entirely docile, but  was capable of doing things which we 
recognized as intelligent, and not just mistakes or random shots, but which we had not programmed into it. But then it would cease to be  a machine, within the meaning of the act. 
What is at stake in the mechanist debate is  not how minds are, or might be, brought into 
being, but how they operate. It is essential fo r the mechanist thesis that the mechanical 
model of the mind shall operate according to "m echanical principles," that is, that we can 
understand the operation of the whole in terms of the operations of its parts, and the

understand the operation of the whole in terms of the operations of its parts, and the 
operation of each part either shall be determin ed by its initial state and the construction of 
the machine, or shall be a random choice between a determinate number of determinate operations. If the mechanist produces a machine wh ich is so complicated that this ceases to 
hold good of it, then it is no longer a

Minds and Thoughts 396machine for the purposes of our discussion, no matter how it was constructed. We should 
say, rather, that he had created a mind, in th e same sort of sense as we procreate people at 
present. There would then be two ways of bringing new minds into the world, the 
traditional way, by begetting children born of women, and a new way by constructing very, 
very complicated systems of, say, valves and relays. When talking of the second way. we should take care to stress that although what was created looked like a machine, it was not one really, because it was not just the total of its parts. One could not tell what it was going to do merely by knowing the way in which it was built up and the initial state of its parts: one could not even tell the limits of what it could do, for even when presented with a 
Gödel-type question, it got the answer right. In fact we should say briefly that any system

Gödel-type question, it got the answer right. In fact we should say briefly that any system 
which was not floored by the Gödel question was eo ipso not a Turing machine, i.e. not a machine within the meaning of the act .3 
 
In reading this passage, my mind constantly boggles at the rapid succession of topics, 
allusions, connotations, confusions, and conclusions. We jump from a Carrollian paradox to Gödel to Turing to Artificial Intelligence to holism and reductionism, all in the span of two brief pages. About Lucas one can say that he is nothing if not stimulating. In the following Chapters, we shall come back to many of the topics touched on so tantalizingly and fleetingly in this odd passage.

Aria with Diverse Variations 
 397Aria with Diverse Variations 
 
Achilles has been unable to 'sleep these past few nights. His friend the 
Tortoise has come over tonight, to keep him company during these annoying hours. 
 
Tortoise: I am so sorry to hear of the troubles that have been plaguing you, my dear 
Achilles. I hope my company will provide a welcome relief from all the unbearable stimulation which has kept you awake. Pe rhaps I will bore you sufficiently that you 
will at long last go to sleep. In that way, I will be of some service. 
Achilles: Oh, no, I am afraid that I have alr eady had some of the wo rld's finest bores try 
their hand at boring me to sleep-and all, sad to say, to no avail. So you will be no 
match for them. No, Mr. T, I invited you over hoping that perhaps you could entertain me with a little this or that, take n from number theory, so that I could at 
least while away these long hours in an agr eeable fashion. You see, I have found that

least while away these long hours in an agr eeable fashion. You see, I have found that 
a little number theory does wonders for my troubled psyche. 
Tortoise: How quaint an idea! You know, it remi nds me, just a wee b it, of the story of 
poor Count Kaiserling. 
'Achilles: Who was he? Tortoise: Oh, he was a Count in Saxony in th e eighteenth century-a Count of no account, 
to tell the truth-but because of him-well,  shall I tell you the story? It is quite 
entertaining. 
Achilles: In that case, by all means, do! Tortoise: There was a time when the good Count  was suffering from sleeplessness, and it

Achilles: In that case, by all means, do! Tortoise: There was a time when the good Count  was suffering from sleeplessness, and it 
just so happened that a competent musician  lived in the same town, and so Count 
Kaiserling commissioned this musician to com pose a set of variations to be played 
by the Count's court harpsichordist for him during his sleepless nights, to make the 
hours pass by more pleasantly. 
Achilles: Was the local composer up to the challenge? Tortoise: I suppose so, for after they were done, the Count rewarded him most 
lucratively-he presented him with a gol d goblet containing one hundred Louis d'or.

lucratively-he presented him with a gol d goblet containing one hundred Louis d'or. 
Achilles: You don't say! I wonder where he came upon such a goblet and all those Louis 
d'or, in the first place. 
Tortoise. Perhaps he saw it in a museum, and took a fancy to it. Achilles: Are you suggesting he absconded with it? Tortoise: Now, now, I wouldn't put it exactly  that way, but ... Those days, Counts could 
get away with most anything. Anyway, it is  clear that the Count was most pleased 
with the music, for he was constantly entreating his harpsichordist-a mere lad of a fellow, name of Goldberg-to

 398play one or another of these thirty variations. Consequently (and somewhat 
ironically) the variations became attached  to the name of young Goldberg, rather 
than to the distinguished Count's name. 
Achilles: You mean, the composer was Bach, and these were the so-called "Goldberg 
Variations"? 
Tortoise: Do I ever! Actually, the work wa s entitled Aria with Diverse Variations, of 
which there are thirty. Do you know how B ach structured these thirty magnificent 
variations? 
Achilles: Do tell. Tortoise: All the pieces-except the final one-a re based on a single theme, which he called 
an "aria". Actually, what binds them a ll together is not a common melody, but a 
common harmonic ground. The melodies ma y vary, but underneath, there is a 
constant theme. Only in the last variation di d Bach take liberties. It is a sort of "post-
ending ending". It contains extraneous musical ideas having little to do with the

ending ending". It contains extraneous musical ideas having little to do with the 
original Theme-in fact, two German folk tunes. That variation is called a "quodlibet". 
Achilles: What else is unusual about the Goldberg Variations? Tortoise: Well, every third variation is a canon. First a canon in two canonizing voices 
enter on the SAME note. Second, a canon in which one of the canonizing voices 
enters ONE NOTE HIGHER than the first. Third, one voice enters Two notes higher 
than the other. And so on, until the final canon has entries just exactly one ninth 
apart. Ten canons, all told. And

apart. Ten canons, all told. And 
Achilles: Wait a minute. Don't I recall read ing somewhere or other about fourteen 
recently discovered Goldberg canons ... 
Tortoise: Didn't that appear in the same  journal where they recently reported the 
discovery of fourteen previously unknown days in November? 
Achilles: No, it's true. A fellow named Wol ff-a musicologist-heard about a special copy 
of the Goldberg Variations in Strasbourg. He went there to examine it, and to his 
surprise, on the back page, as a sort of "post-ending ending", he found these fourteen

surprise, on the back page, as a sort of "post-ending ending", he found these fourteen 
new canons, all based on the first eight notes of the theme of the Goldberg Variations. So now it is known that ther e are in reality forty-four Goldberg 
Variations, not thirty. 
Tortoise: That is, there are forty-four of th em, unless some other musicologist discovers 
yet another batch of them in some unlikely spot. And although it seems improbable, it is still possible, even if unlikely, that  still another batch will be discovered, and 
then another one, and on and on and on ... Why, it might never stop! We may never

then another one, and on and on and on ... Why, it might never stop! We may never 
know if or when we have the full complement of Goldberg Variations. 
Achilles: That is a peculiar idea. Presumably, everybody thinks that this latest discovery 
was just a fluke, and that we now really do have all the Goldberg Variations. But just 
supposing that you are right, and some  turn up sometime, we shall start to expect this kind of thing. At

 399that point, the name "Goldberg Variations" w ill start to shift slightly in meaning, to 
include not only the known ones, but also a ny others which might eventually turn up. 
Their number-call it 'g'-is certain to be  finite, wouldn't you agree?-but merely 
knowing that g is finite isn't the same as  knowing how big g is. Consequently, this 
information won't tell us when the last Goldberg Variation has been located. 
Tortoise: That is certainly true. Achilles: Tell me-when was it that Bach wrote these celebrated variations? Tortoise: It all happened in the year 1742, wh en he was Cantor in Leipzig. Achilles: 
1742? Hmm ... That number rings a bell. 
Tortoise: It ought to, for it happens to be a ra ther interesting number, being a sum of two 
odd primes: 1729 and 13. 
Achilles: By thunder! What a curious fact' I wonder how often one runs across an even 
number with that property. Let's see

number with that property. Let's see  
 
6= 3+3    8= 3+5    10= 3+7= 5+5   12= 5+7    14=3+11= 7+7   16=3+ 13= 5+ 11   18=5+ 13= 7+ 11   20=3+ 17= 7+ 13   22 = 3 + 19 = 5 + 17 = 11 + 11 24=5+19= 7+17=11+13 26=3+23= 7+19=13+13 28 = 5 + 23 = 11 + 17 
30 = 7 + 23 = 11 + 19 = 13 + 17 
 
Now what do you know-according to my little table here, it seems to be quite a common occurrence. Yet I don't discern any simple regularity in the table so far.

Now what do you know-according to my little table here, it seems to be quite a common occurrence. Yet I don't discern any simple regularity in the table so far. 
 Tortoise: Perhaps there is no regularity to be discerned. Achilles: But of course there is! I am just not clever enough to spot it right off the bat. Tortoise: You seem quite convinced of it. Achilles: There's no doubt in my mind. I wonder ... Could it be that ALL even numbers 
(except 4) can be written as a sum of two odd primes?  
Tortoise: Hmm ... That question rings a bell ... Ah, I know why! You're not the first 
person to ask that question. Why, as a matter of fact, in the year 1742, a mathematical amateur put forth this very question in a

person to ask that question. Why, as a matter of fact, in the year 1742, a mathematical amateur put forth this very question in a 
Achilles: Did you say 1742? Excuse me for interrupting, but I just noticed that 1742 
happens to be a rather interesting numbe r, being a difference of two odd primes: 
1747 and 5. 
Tortoise: By thunder! What a curious fact! I wonder how often one runs across an even 
number with that property.

 400Achilles: But please don't let me distract you from your story. 
Tortoise: Oh, yes-as I was saying, in 1742. a cer tain mathematical amateur, whose name 
escapes me momentarily, sent a letter to Eu ler, who at the time was at the court of 
King Frederick the Great in Potsdam, and-we ll, shall I tell you the story? It is not 
without charm. Achilles: In that case, by all means, do! 
Tortoise: Very well. In his letter, this dabbler in number theory propounded an unproved 
conjecture to the great Euler: "Every even  number can he represented as a sum of 
two odd primes." Now what was that fellow's name? 
Achilles: I vaguely recollect the story, from some number theory book or other. Wasn't 
the fellow named Iiupfergiidel 
Tortoise: Hmm ... No, that sounds too long. Achilles: Could it have been "Silberescher"? Tortoise: No, that's not it, either. There's a name on the tip of' my tongue-ah-ah-oh yes! It

Tortoise: Hmm ... No, that sounds too long. Achilles: Could it have been "Silberescher"? Tortoise: No, that's not it, either. There's a name on the tip of' my tongue-ah-ah-oh yes! It 
was "Goldbach"! Goldbach was the fellow. Achilles: I knew it was something like that. 
Tortoise: Yes-your guesses helped jog my me mory. It's quite odd, how one occasionally 
has to hunt around in one's memory as if for a book in a library without call numbers ... But let us get back to 1742. 
Achilles: Indeed, let's. I wanted to ask you: did Euler ever prove that this guess by 
Goldbach was right? 
Tortoise: Curiously enough, he never even  considered it worthwhile working on. 
However, his disdain was not shared by a ll mathematicians. In fact, it caught the

However, his disdain was not shared by a ll mathematicians. In fact, it caught the 
fancy of many, and became known as the "Goldbach Conjecture". 
Achilles: Has it ever been proven correct? Tortoise: No, it hasn't. But there have been so me remarkable near misses. For instance, in 
1931 the Russian number theorist Schnire lmann proved that any number-even or 
odd-can be represented as the sum of not more than 300,000 primes. 
Achilles: What a strange result. Of what good is it? Tortoise: It has brought the problem into the domain of the finite. Previous to

Achilles: What a strange result. Of what good is it? Tortoise: It has brought the problem into the domain of the finite. Previous to 
Schnirelmann's proof, it was conceivable that as you took larger and larger even numbers, they would require more and more  primes to represent them. Some even 
number might take a trillion primes to represent it! Now it is known that that is not so-a sum of 300,000 primes (or fewer) will always suffice. 
Achilles: I see. Tortoise: Then in 1937, a sly fellow name d Vinogradov-a Russian too-managed to 
establish something far closer to the desi red result: namely, every sufficiently large

establish something far closer to the desi red result: namely, every sufficiently large 
ODD number can be represented as a sum of no more than THREE odd primes. For 
example, 1937 = 641 + 643 + 653. We could say that an odd number which is 
representable as a sum of three odd prim es has "the Vinogradov property. Thus, all 
sufficiently large odd numbers have the Vinigradov properties

 401 
Achilles: Very well-but what does "sufficiently large" mean? Tortoise: It means that some finite num ber of odd numbers may fail to have the 
Vinogradov property, but there is a numbe r-call it 'v'beyond which all odd numbers 
have the Vinogradov property. But Vinogradov was unable to say how big v is. So in a way, v is like g, the finite but unknown num ber of Goldberg Variations. Merely 
knowing that v is finite isn't the same as  knowing how big v is. Consequently, this 
information won't tell us when the last odd number which needs more than three primes to represent it has been located. 
Achilles: I see. And so any sufficiently larg e even number 2N can be represented as a 
sum of FOUR primes, by first representing 2N  - 3 as a sum of three primes, and then 
adding back the prime number 3. 
Tortoise: Precisely. Another close approach is  contained in the Theorem which says, "All 
even numbers can be represented as a su m of one prime and one number which is a

even numbers can be represented as a su m of one prime and one number which is a 
product of at most two primes." 
Achilles: This question about sums of two primes certainly leads you into strange 
territory. I wonder where you would be le d if you looked at DIFFERENCES of two 
odd primes. I'll bet I could glean some insight  into this teaser by making a little table 
of even numbers, and their representations as differences of two odd primes, just as I did for sums. Let's see ...

of even numbers, and their representations as differences of two odd primes, just as I did for sums. Let's see ... 
 
2= 5-3, 7-5,  13-11,  19-17, etc. 4 = 7 - 3, 11 - 7,  17 - 13, 23 - 19,etc. 6 = 11 - 5, 13 - 7,  17 - 11, 19- 13, etc. 8 = 11 - 3, 13 - 5,  19 - 11, 31 - 23,etc. 10 = 13 - 3, 17 - 7,  23 - 13, 29- 19, etc. 
 
My gracious! There seems to be no end to th e number of different representations I can 
find for these even numbers. Yet I don't discern any simple regularity in the table so far.

find for these even numbers. Yet I don't discern any simple regularity in the table so far.  
Tortoise: Perhaps there is no regularity to be discerned. Achilles: Oh, you and your constant rumblings about chaos! I'll hear none of that, thank 
you. 
Tortoise: Do you suppose that EVERY even num ber can be represented somehow as the 
difference of two odd primes? 
Achilles: The answer certainly would appear to  be yes, from my table. But then again, I 
suppose it could also be no. That doesn't really get us very far, does it?

you. 
suppose it could also be no. That doesn't really get us very far, does it? 
Tortoise: With all due respect, I would say there are deeper insights to be had on the 
matter. 
Achilles: Curious how similar this problem is to Goldbach's original one. Perhaps it 
should be called a "Goldbach Variation". 
Tortoise: Indeed. But you know, there is a rather striking difference between the 
Goldbach Conjecture, and this Goldbach  Variation, which I would like to tell you 
about. Let us say that an even number 2N has the “Goldbach property” if it is the 
SUM of two odd primes, and it has the “Tortoise property” if it is the DIFFERENCE 
of two odd primes

 402Achilles: I think you should call it the "Achille s property". After all, I suggested the 
problem. 
Tortoise: I was just about to propose that we should say that a number which LACKS the 
Tortoise property has the "Achilles property". Achilles: Well, all right .. . 
Tortoise: Now consider, for instance, whether I trillion has the Goldbach property or the 
Tortoise property. Of course, it may have both. 
Achilles: I can consider it, but I doubt whet her I can give you an answer to either 
question. 
Tortoise: Don't give up so soon. Suppose I aske d you to answer one or the other question. 
Which one would you pick to work on? 
Achilles: I suppose I would flip a coin. I don't see much difference between them. Tortoise: Aha: But there's a world of diffe rence' If you pick the Goldbach property, 
involving SUMS of primes, then you are limited to using primes which are bounded 
between 2 and 1 trillion, right?

question. 
between 2 and 1 trillion, right? 
Achilles: Of course. Tortoise: So your search for a representati on for 1 trillion as a sum of two primes is 
GUARANTEED TO TERMINATE. 
Achilles: Ahhh! I see your point. Whereas if I chose to work on representing 1 trillion as 
the DIFFERENCE of two primes, I woul d not have any bound on the size of the 
primes involved. They might be so big that  it would take me a trillion years to find 
them. 
Tortoise: Or then again, they might not even EXIST! After all, that's what the question 
was asking-do such primes exist, It wasn't of much concern how big they might turn 
out to be.

them. 
out to be. 
Achilles: You're right. If they didn't exist, then a search process would lead on forever, 
never answering yes, and never answeri ng no. And nevertheless, the answer would 
be no. 
Tortoise: So if you have some number, and you wish to test whether it has the Goldbach 
property or the Tortoise property, the differe nce between the two tests will be this: in 
the former, the search involved is GUAR ANTEED TO TERMINATE; in the latter, 
it is POTENTIALLY ENDLESS-there are no gua rantees of any type. It might just

be no. 
it is POTENTIALLY ENDLESS-there are no gua rantees of any type. It might just 
go merrily on forever, without yielding an  answer. And yet, on the other hand, in 
some cases, it might stop on the first step. 
Achilles: I see there is a rather vast di fference between the Goldbach and Tortoise 
properties. 
Tortoise: Yes, the two similar problems con cern these vastly different properties. The 
Goldbach Conjecture is to the effect th at all even numbers have the Goldbach 
property; the Goldbach Variation suggests th at all even numbers have the Tortoise 
property. Both problems are unsolved, but what  is interesting is that although they

property. Both problems are unsolved, but what  is interesting is that although they 
sound very much alike, they involve prope rties of whole numbers which are quite 
different. 
Achilles: I see what you mean. The Goldbach property is a detectable, or

 403recognizable property of any even number, since I know how to test for its presence 
just embark on a search. It will automati cally come to an end with a yes or no 
answer. The Tortoise property, however, is mo re elusive, since a brute force search 
just may never give an answer. 
Tortoise: Well, there may be cleverer ways of searching in the case of the Tortoise 
property, and maybe following one of them w ould always come to an end, and yield 
an answer. 
Achilles: Couldn't the search only end if the answer were yes% Tortoise: Not necessarily. There might be some  way of proving that whenever the search 
lasts longer than a certain length of time, then the answer must be no. There might 
even be some OTHER way of searching fo r the primes, not such a brute force way, 
which is guaranteed to find them if they exist, and to tell if they don't. In either case, a finite search would be able to yield th e answer no. But I don't know if such a thing

an answer. 
which is guaranteed to find them if they exist, and to tell if they don't. In either case, a finite search would be able to yield th e answer no. But I don't know if such a thing 
can be proven or not. Searching through infi nite spaces is always a tricky matter, you 
know. 
Achilles: So as things stand now, you know of no test for the Tortoise property which is 
guaranteed to terminate-and yet there MIGHT exist such a search. 
Tortoise: Right. I suppose one could embark on a search for such a search, but I can give 
no guarantee that that "meta-search" would terminate, either. 
Achilles: You know, it strikes me as quite peculiar that if some even number-for

know. 
Achilles: You know, it strikes me as quite peculiar that if some even number-for 
example, a trillion-failed to have the To rtoise property, it would be caused by an 
infinite number of separate pieces of info rmation. It's funny to think of wrapping all 
that information up into one bundle, and calling it, as you so gallantly suggested, 
"the Achilles property" of 1 trillion. It is r eally a property of the number system as a 
"HOLE, not just of the number 1 trillion. 
Tortoise: That is an interesting observation, Achilles, but I maintain that it makes a good

Tortoise: That is an interesting observation, Achilles, but I maintain that it makes a good 
deal of sense to attach this fact to the number 1 trillion nevertheless. For purposes of 
illustration, let me suggest that you . consid er the simpler statement "29 is prime". 
Now in fact, this statement really means th at 2 times 2 is not 29, and 5 times 6 is not 
29, and so forth, doesn't it? 
Achilles: It must, I suppose. Tortoise: But you are perfectly happy to collect a ll such facts together, and attach them in 
a bundle to the number 29, saying merely, "29 is prime"

a bundle to the number 29, saying merely, "29 is prime" 
Achilles: Yes ... Tortoise: And the number of facts involved is actually infinite, isn't it:, After all, such 
facts as "4444 times 3333 is not 29" are all part of it, aren't they% 
Achilles: Strictly speaking, I suppose so. But you and I both know that you can't produce 
29 by multiplying two numbers which are bot h bigger than 29. So in reality, saying 
"29 is prime" is only summarizing a FINITE number of facts about multiplication

 404Tortoise: You can put it that wa y if you want, but think of this: the fact that two numbers 
which are bigger than 29 can't have a product equal to 29 involves the entire structure of the number system. In that sense,  that fact in itself is a summary of an 
infinite number of facts. You can't get aw ay from the fact, Achilles, that when you 
say "29 is prime'-', you are actually stating an infinite number of things. 
Achilles: Maybe so, but it feels like just one fact to me. Tortoise: That's because an infinitude of facts are contained in your prior knowledge-they 
are embedded implicitly in the way you visualize things. You don't see an explicit infinity because it is captured implicitly inside the images you manipulate. 
Achilles: I guess that you're right. It still seems odd to lump a property of the entire 
number system into a unit, and label the unit "primeness of 29" 
Tortoise: Perhaps it seems odd, but it is also quite a convenient way to look at things.

Tortoise: Perhaps it seems odd, but it is also quite a convenient way to look at things. 
Now let us come back to your hypothetical  idea. If, as you suggested, the number 1 
trillion has the Achilles property, then no ma tter what prime you add to it, you do not 
get another prime. Such a state of affair s would be caused by an infinite number of 
separate mathematical "events". Now do a ll these "events" necessarily spring from 
the same source? Do they have to have a common cause? Because if they don't, then 
some sort of "infinite coincidence" has created the fact, rather than an underlying 
regularity. 
Achilles: An "infinite coincidence"? Among the natural numbers, NoTHING is

Achilles: An "infinite coincidence"? Among the natural numbers, NoTHING is 
coincidental-nothing happens without ther e being some underlying pattern. Take 7, 
instead of a trillion. I can deal  with it more easily, because it is smaller. 7 has the 
Achilles property. 
Tortoise: You're sure? Achilles: Yes. Here's why. If you add 2 to it,  you get 9, which isn't prime. And if you add 
any other prime to 7, you are adding two odd numbers, resulting in an even number-
thus you again fail to get a prime. So here the "Achilleanity" of 7, to  coin a term, is a 
consequence of just Two reasons: a far cr y from any "infinite coincidence". Which

consequence of just Two reasons: a far cr y from any "infinite coincidence". Which 
just goes to support my assertion: that it ne ver takes an infinite number of reasons to 
account for some arithmetical truth. If th ere WERE some arithmetical fact which 
were caused by an infinite collection of unr elated coincidences, then you could never 
give a finite proof for that truth. And that is ridiculous. 
Tortoise: That is a reasonable opinion, and you are in good company in making it. 
However 
Achilles: Are there actually those who disagr ee with this view? Such people would have

However 
Achilles: Are there actually those who disagr ee with this view? Such people would have 
to believe that there are "infinite coincidences ", that there is chaos in the midst of the 
most perfect, harmonious, and beautiful of  all creations: the system of natural 
numbers. 
Tortoise: Perhaps they do; but have you ever considered that such chaos might be an 
integral part of the beauty and harmony?

 405
 
FIGURE 71. Order and Chaos, by M. C. Escher (lithograph, 1950).  Achilles: Chaos, part of perfection? Order and chaos make a pleasing unity? Heresy! Tortoise: Your favorite artist, M. C. Escher , has been known to suggest such a heretical 
point of view in one of hi s pictures ... And while we're on the subject of chaos, I 
believe that you might be interested in hearing about two different categories of 
search, both of which are guaranteed to terminate. 
Achilles: Certainly.

 406Tortoise: The first type of search-the non-ch aotic type-is exemplified by the test involved 
in checking for the Goldbach property. You ju st look at primes less than 2N, and if 
some pair adds up to 2N, then 2N has the Goldbach property; otherwise, it doesn't. This kind of test is not only sure to te rminate, but you can predict BY "'HEN it will 
terminate, as well. 
Achilles: So it is a PREDICTABLY TERMINATING test. Are you going to tell me that 
checking for some number-theoretical prope rties involves tests which are guaranteed 
to terminate, but about which there is no way to know in advance how long they will 
take? 
Tortoise: How prophetic of you, Achilles. And the existence of such tests shows that 
there is intrinsic chaos, in a certain sense, in the natural number system. 
Achilles: Well, in that case, I would have to say that people just don't know enough about 
the test. If they did a little more researc h, they could figure out how long it will take,

take? 
the test. If they did a little more researc h, they could figure out how long it will take, 
at most, before it terminates. After all, ther e must always be some rhyme or reason to 
the patterns among integers. There can't just be chaotic patterns which defy prediction' 
Tortoise: I can understand your intuitive fait h, Achilles. However, it's not always 
justified. Of course, in many cases you are exactly right just because somebody 
doesn't know something, one can't concl ude that it is unknowable' But there are 
certain properties of integers for which term inating tests can be proven to exist, and

certain properties of integers for which term inating tests can be proven to exist, and 
yet about which it can also be PROVEN that  there is no way to predict in advance 
how long they will take. 
Achilles: I can hardly believe that. It sounds as if the devil himself managed to sneak in 
and throw a monkey wrench into God's beautiful realm of natural numbers' 
Tortoise: Perhaps it will comfor t you to know that it is by no means easy, or natural, to 
define a property for which there is  a terminating but not PREDICTABLY 
terminating test. Most "natural" properties of integers do admit of predictably

terminating test. Most "natural" properties of integers do admit of predictably 
terminating tests. For example, primeness.  squareness, being a power of ten, and so 
on. 
Achilles: Yes, I can see that those properties are completely straightforward to test for. 
Will you tell me a property for which the onl y possible test is a terminating but 
nonpredictable one? 
Tortoise: That's too complicated for me in my sleepy state. Let me instead show you a 
property which is very easy to define, a nd yet for which no terminating test is 
known. I'm not saying there won't ever be one discovered, mind you just that none is

on. 
known. I'm not saying there won't ever be one discovered, mind you just that none is 
known. You begin with a number-would you care to pick one? 
Achilles: How about 15? Tortoise: An excellent choice. We begin with your number, and if it is ODD, we triple it, 
and add 1. If it is EVEN, we take half of  it. Then we repeat the process. Call a 
number which eventually reaches 1 this  way a WONDROUS number, and a number 
which doesn't, an UNWONDROUS number

 407 
Achilles: Is 15 wondrous, or unwondrous? Let's see:  15  is ODD, so I make 3n + 1:  46  46  is EVEN, so I take half:   23  23  is ODD, so I make 3n + 1:  70  70  is EVEN, so I take half:   35  35  is ODD, so I make 3n + 1:  106 106 is EVEN, so I take half:   53  53  is ODD, so I make 3n + 1:  160 160 is EVEN, so I take half:   80  80  is EVEN, so I take half:   40  40  is EVEN, so I take half:   20  20  is EVEN, so I take half:   10  10  is EVEN, so I take half:   5 5  is ODD, so I make 3n + 1:  16  16  is EVEN, so I take half:   8 8  is EVEN, so I take half:   4 4  is EVEN, so I take half:   2 2  is EVEN, so I take half:  1 . 
Wow! That's quite a roundabout journey, from  15 to 1. But I finally reached it. That 
shows that 15 has the property of bei ng wondrous. I wonder what numbers are 
UNwondrous ... 
Tortoise: Did you notice how the numbers sw ung up and down, in this simply defined 
process?

process? 
Achilles: Yes. I was particularly surprised, af ter thirteen turns, to find myself at 16, only 
one greater than 15, the number I started with. In one sense, I was almost back where I started-yet in another sense, I' was now here near where I had started. Also, I found 
it quite curious that I had to go as high as 160 to resolve the question. I wonder how 
come. 
Tortoise: Yes, there is an infinite "sky" into which you can sail, and it is very hard to 
know in advance how high into the sky you will wind up sailing. Indeed, it is quite plausible that you might just sail up and up and up, and never come down. 
Achilles: Really? I guess that is conceiva ble-but what a weird coincidence it would

come. 
Achilles: Really? I guess that is conceiva ble-but what a weird coincidence it would 
require! You'd just have to hit odd numbe r after odd number, with only a few evens 
mixed in. I doubt if that would ever happen-but I just don't know for sure. 
Tortoise: Why don't you try starting with 27? Mind you, I don't promise anything. But 
sometime, just try it, for your amusement. And I'd advise you to bring along a rather 
large sheet of paper. 
Achilles: Hmm ... Sounds interesting. You know, it still makes me feel funny to associate 
the wondrousness (or unwondrousness) with the starting number, when it is so obviously a property of the entire number system.

the wondrousness (or unwondrousness) with the starting number, when it is so obviously a property of the entire number system. 
Tortoise: I understand what you mean, but it's not that different from saying “29 is 
prime” or “gold is valuable” – both statements attribute to

 408a single entity a property which it has onl y by virtue of being embedded in a 
particular context. 
Achilles: I suppose you're right. This "w ondrousness" problem is wondrous tricky, 
because of the way in which the numbers oscillate-now increasing, now decreasing. The pattern OUGHT to be regular,-yet on th e surface it appears to be quite chaotic. 
Therefore, I can well imagine why, as of ye t, no one knows of a test for the property 
of wondrousness which is guaranteed to terminate. 
Tortoise: Speaking of terminating and nonter minating processes, and those which hover 
in between, I am reminded of a friend of mine, an author, who is at work on a book. 
Achilles: Oh, how exciting! What is it called? Tortoise: Copper, Silver, Gold: an Indest ructible Metallic Alloy. Doesn't that sound 
interesting? 
Achilles: Frankly, I'm a little confused by the title. After all, what do Copper, Silver, and 
Gold have to do with each other? Tortoise: It seems clear to me.

Gold have to do with each other? Tortoise: It seems clear to me. 
Achilles: Now if the title were, say, Giraffes, Silver, Gold, or Copper, Elephants, Gold, 
why, I could see it .. . 
Tortoise: Perhaps you would prefer Copper, Silver, Baboons? Achilles: Oh, absolutely! But that original title is a loser. No one would understand it. Tortoise: I'll tell my friend. He'll be deli ghted to have a catchier title (as will his 
publisher). 
Achilles: I'm glad. But how were you reminded of his book by our discussion? Tortoise: Ah, yes. You see, in his book ther e will be a Dialogue in which he wants to

Achilles: I'm glad. But how were you reminded of his book by our discussion? Tortoise: Ah, yes. You see, in his book ther e will be a Dialogue in which he wants to 
throw readers off by making them SEARCH for the ending. 
Achilles: A funny thing to want to do. How is it done? Tortoise: You've undoubtedly noticed how some  authors go to so much trouble to build 
up great tension a few pages before the e nd of their stories-but a reader who is 
holding the book physically in his hands can FEEL that the story is about to end. 
Hence, he has some extra information wh ich acts as an advance warning, in a way. 
The tension is a bit spoiled by the physicality  of the book. It would be so much better

The tension is a bit spoiled by the physicality  of the book. It would be so much better 
if, for instance, there were a lot of padding at the end of novels. 
Achilles: Padding? Tortoise: Yes; what I mean is, a lot of extra printed pages which are not part of the story 
proper, but which serve to conceal the ex act location of the end from a cursory 
glance, or from the feel of the book. 
Achilles: I see. So a story's true ending might  occur, say, fifty or a hundred pages before 
the physical end of the book? 
Tortoise: Yes. This would provide an elemen t of surprise, because the reader wouldn't

Tortoise: Yes. This would provide an elemen t of surprise, because the reader wouldn't 
know in advance how many pages are padding, and how many are story. 
Achilles: If this were standard practice, it might be quite effective. But

 409there is a problem. Suppose your padding were  very obvious-such as a lot of blanks, 
or pages covered with X's or random letters. Then, it would be as good as absent. 
Tortoise: Granted. You'd have to make it resemble normal printed pages. Achilles: But even a cursory glance at a norma l page from one story will often suffice to 
distinguish it from another story. So you w ill have to make the padding resemble the 
genuine story rather closely. 
Tortoise: That's quite true. The way I've al ways envisioned it is this: you bring the story 
to an end; then without any break, you fo llow it with something which looks like a 
continuation but which is in reality just pa dding, and which is utterly unrelated to the 
true theme. The padding is, in a way, a "post-ending ending". It may contain 
extraneous literary ideas, having little to do with the original theme. 
Achilles: Sneaky! But then the problem is th at you won't be able to tell when the real

Achilles: Sneaky! But then the problem is th at you won't be able to tell when the real 
ending comes. It'll just blend right into the padding. 
Tortoise: That's the conclusion my author frie nd and I have reached as well. It's a shame, 
for I found the idea rather appealing. 
Achilles: Say, I have a suggestion. The tr ansition between genuine story and padding 
material could be made in such a way that, by sufficiently assiduous inspection of the text, an intelligent reader will be able to  detect where one leaves off and the other

material could be made in such a way that, by sufficiently assiduous inspection of the text, an intelligent reader will be able to  detect where one leaves off and the other 
begins. Perhaps it will take him quite a while. Perhaps there will be no way to 
predict how long it will take ... But the publisher could give a guarantee that a 
sufficiently assiduous search for the true e nding will always terminate, even if he 
can't say how long it will be before the test terminates. 
Tortoise: Very well-but what does "sufficiently assiduous" mean? Achilles: It means that the reader must be on the lookout for some small but telltale 
feature in the text which occurs at some  point. That would si gnal the end. And he

feature in the text which occurs at some  point. That would si gnal the end. And he 
must be ingenious enough to think up, and hunt for, many such features until he 
finds the right one. 
Tortoise: Such as a sudden shift of letter fr equencies or word lengths? Or a rash of 
grammatical mistakes? 
Achilles: Possibly. Or a hidden message of some sort might reveal the true end to a 
sufficiently assiduous reader. Who knows? One could even throw in some extraneous characters or events which ar e inconsistent with the spirit of the 
foregoing story. A naive reader would swallow the whole thing, whereas a

foregoing story. A naive reader would swallow the whole thing, whereas a 
sophisticated reader would be able to spot the dividing line exactly. 
Tortoise: That's a most original idea, Achille s. I'll relay it to my friend, and perhaps he 
can incorporate it in his Dialogue. 
Achilles: I would be highly honored. Tortoise: Well, I am afraid th at I myself am growing a little groggy, Achilles. It would be 
well for me to take my leave, while I am still capable of navigating my way home. 
Achilles: I am most flattered' that you have stayed up for so long, and at such an odd hour 
of the night, just for my benefit. I assure you that

 410your number-theoretical entertainment has been a perfect antidote to my usual 
tossing and turning. And who knows-perhaps I may even be able to go to sleep 
tonight. As a token of my gratitude, Mr. T,  I would like to present you with a special 
gift. 
Tortoise: Oh, don't be silly, Achilles. Achilles: It is my pleasure, Mr. T. Go over to that dresser; on it, you will see an Asian 
box. 
 
(The Tortoise moseys over to Achilles' dresser.) 
 
Tortoise. You don't mean this very gold Asian box, do you? Achilles: That's the one. Please accept it, Mr. T, with my warmest compliments. Tortoise: Thank you very much indeed, Achilles. Hmm ... Why are all these 
mathematicians' names engraved on the top? What a curious list: 
 
De Morgan  
Abel 
Boole 
Br o u w e r 
S i e r p i n s k i  
W e i e r s t r a s s 
 Achilles: I believe it is supposed to be a Complete List of All Great Mathematicians. 
What I haven't been able to figure out is why the letters running down the diagonal

De Morgan  
Abel 
Boole 
Br o u w e r 
What I haven't been able to figure out is why the letters running down the diagonal 
are so much bolder. 
Tortoise: At the bottom it says, "Subtract 1 from the diagonal, to find Bach in Leipzig". 
Achilles: I saw that, but I couldn't make head  or tail of it. Say, how about a shot of 
excellent whiskey? I happen to have some in that decanter on my shelf. 
Tortoise: No, thanks. I'm too tired. I'm just going to head home. (Casually, he opens the 
box.) Say, wait a moment, Achilles-there are one hundred Louis d'or in here! 
Achilles: I would be most pleased if you would accept them, Mr. T. Tortoise: But-but Achilles: No objections, now. The box, the gold-they're yours. And thank you for an 
evening without parallel.

evening without parallel. 
Tortoise: Now whatever has come ove r you, Achilles? Well, thank you for your 
outstanding generosity and I hope you ha ve sweet dreams about the strange 
Goldbach Conjecture, and its Variation. Good night. 
 
(And he picks up the very gold Asian box filled with the one hundred Louis d'or, and 
walks towards the door. As he is about to leave, there is a loud knock.) 
 
Who could be knocking at this ungodly hour, Achilles? 
Achilles: I haven't the foggiest idea. It seems suspicious to me. Why don't you go hide 
behind the dresser, in case there's any funny business.

 411Tortoise: Good idea. (Scrambles in behind the dresser.) Achilles: Who's there? 
Voice: Open up-it's the cops.  Achilles: Come in, it's open.  
(Two burly policemen walk in, wearing shiny badges.) 
 
Cop: I'm Silva. This is Gould. (Points at his badge.) Is there an Achilles at this address? 
Achilles: That's me! Cop: Well, Achilles, we have reason to belie ve that there is a very gold Asian box here, 
filled with one hundred Louis d'or. Some one absconded with it from the museum 
this afternoon. Achilles: Heavens to Betsy! 
Cop: If it is here, Achilles, since you would be  the only possible suspect, I' regret to say 
that I should have to take you into custody. Now I have here a search warrant 
Achilles: Oh, sirs, am I ever glad you arri ved! All evening long, I have been being 
terrorized by Mr. Tortoise and his very Asian gold box. Now at last you have come 
to liberate me! Please, sirs, just take a look behind that dresser, and there you will 
find the culprit!

find the culprit! 
 
(The cops look behind the dresser and sp y the Tortoise huddled behind it, holding 
his very gold Asian box, and trembling.) 
 
Cop: So there it is! And so Mr. Tortoise is the varmint, eh? I never would have suspected 
HIM. But he's caught, red-handed. 
Achilles: Haul the villain away, kind sirs! Tha nk goodness, that's the last I'll have to hear 
of him, and the Very Asian Gold Box!

BlooP and FlooP and GlooP 413CHAPTER XI11 
 
BlooP and FlooP and GlooP 
 
Self-Awareness and Chaos 
 
BLOOP, FLOOP, AND GLOOP  are not trolls, talking ducks, or the sounds made by a 
sinking ship-they are three computer language s, each one with is own special purpose. 
These languages were invented specially for this chapter. They will be of use in 
explaining some new senses of the word 'recursive -in particular, the notions of primitive 
recursivity  and general recursivity . They will prove very helpful in clarifying the 
machinery of self-reference in TNT . 
We seem to be making a rather abrupt  transition from brains and hinds to 
technicalities of mathematics and computer sc ience. Though the transition is abrupt in 
some ways, it makes some sense. We just  saw how a certain kind of self-awareness 
seems to be at the crux of consciousne ss. Vow we are going to scrutinize "self-
awareness" in more formal settings, such as TNT . The gulf between TNT  and a mind is

awareness" in more formal settings, such as TNT . The gulf between TNT  and a mind is 
wide, but some of the ideas will be mo st illuminating, and perhaps metaphorically 
transportable back to our thoughts about consciousness. 
One of the amazing things about TNT's  self-awareness is that it is intimately 
connected to questions about order versus chaos among the natural numbers. In particular, we shall see that an orderly syst em of sufficient comple xity that it can mirror 
itself cannot be totally  orderly-it must contain some stra nge, chaotic features. For readers

itself cannot be totally  orderly-it must contain some stra nge, chaotic features. For readers 
who have some Achilles in them, this will be hard to take. However, there is a "magical" 
compensation: there is a kind of order to th e disorder, which is now its own field of 
study, called "recursive function theory". Unfort unately, we will not be able to do much 
more than hint at the fascination of this subject.  
Representability and Refrigerators

Representability and Refrigerators 
 Phrases such as "sufficiently complex", "sufficiently powerful" and the like lave cropped up quite often earlier. Just what do they mean? Let us go back to the battle of the Crab and Tortoise, and ask, "What qualifies somethi ng as a record player?" The Crab might 
claim that his refrigerator s a "Perfect" record  player. Then to prove it, he could set any 
record whatsoever atop it, and say, "You see-it's playing it!" The Tortoise, if he wanted to 
counter this Zen-like act, would have to reply, "No-your refrigerator is too low-fidelity to

counter this Zen-like act, would have to reply, "No-your refrigerator is too low-fidelity to 
be counted as a phonograph: it cannot reproduce sounds-at all (let alone its self-breaking 
sound)." The Tortoise

BlooP and FlooP and GlooP 414can only make a record called "I Cannot Be Played on Record Player X" provided that 
Record Player X is really a record player! The Tortoise's method is quite insidious, as it 
plays on the strength, rather than on the w eakness, of the system. And therefore he 
requires "sufficiently hi-fi" record players. 
Ditto for formal versions of number theory. The reason that TNT  is a 
formalization of N is that its symbols act the ri ght way: that is, its theorems are not silent 
like a refrigerator-they speak actual truths of N. Of course, so do the theorems of the pq-
system. Does it, too, count as "a formaliza tion of number theory", or is it more like a 
refrigerator? Well, it is a little better than a re frigerator, but it is still pretty weak. The pq-
system does not include enough of the core truths of N to count as "a number theory". 
What, then, are these "core truths" of N? They are the primitive recursive truths ;

What, then, are these "core truths" of N? They are the primitive recursive truths ; 
that means they involve only predictably terminating  calculations. These core truths 
serve for N as Euclid's first four postulates serv ed for geometry: they allow you to throw 
out certain candidates before the game begi ns, on the grounds of "insufficient power". 
From here on out, the representability of all primitive recursive truths  will be the 
criterion for calling a system "sufficiently powerful".  
Ganto's Ax in Metamathematics 
 The significance of the notion is shown by the following key fact: If you have a

 The significance of the notion is shown by the following key fact: If you have a 
sufficiently powerful formalization of number theory, then Gödel’s method is applicable, 
and consequently your system is incomplete . If, on the other hand, your system is not 
sufficiently powerful (i.e., not all primitive r ecursive truths are theorems), then your 
system is, precisely by virtue of that lack, incomplete . Here we have a reformulation of 
"Ganto's Ax" in metamathematics: whatever  the system does, Gödel’s Ax will chop its 
head off! Notice also how this completely parallels the high-fidelity-versus-low fidelity 
battle in the Contracrostipunctus .

battle in the Contracrostipunctus . 
Actually, it turns out that much weaker systems are still vulnerable to the Gödel method; the criterion that all primitive recursive truths need be represented as theorems is far too 
stringent. It is a little like a thief who will onl y rob "sufficiently rich" people, and whose 
criterion is that the potential victim should be  carrying at least a million dollars in cash. 
In the case of TNT , luckily, we will be able to act in  our capacity as thieves, for the

In the case of TNT , luckily, we will be able to act in  our capacity as thieves, for the 
million in cash is there-which is to say, TNT  does indeed contain all primitive recursive 
truths as theorems. 
Now before we plunge into a detailed discussion of primitive recursive functions 
and predicates, I would like to tie thee themes  of this Chapter to themes from earlier 
Chapters, so as to provide a bit better motivation.  
Finding Order by Choosing the Right Filter 
 We saw at a very early stage that formal  systems can be difficult and unruly beasts

 We saw at a very early stage that formal  systems can be difficult and unruly beasts 
because they have lengthening and shortening rules, which can

BlooP and FlooP and GlooP 415possibly lead to never-ending searches among strings. The discovery of Gödel-numbering 
showed that any search for a string havi ng a special typographical property has an 
arithmetical cousin: an isomorphic search for an integer with a corresponding special arithmetical property. Consequently, the quest for decision procedures for formal systems 
involves solving the mystery of  unpredictably long searches- chaos -among the integers. 
Now in the Aria with Diverse Variations , I gave perhaps too much weight to apparent 
manifestations of chaos in problems about integers. As a matter of fact, people have tamed wilder examples of apparent chaos than the "wondrousness" problem, finding them 
to be quite gentle beasts after all. Ac hilles' powerful faith in the regularity and 
predictability of numbers should therefore be acco rded quite a bit of respect-especially as 
it reflects the beliefs of nearly all mathem aticians up till the 1930's. To show why order

it reflects the beliefs of nearly all mathem aticians up till the 1930's. To show why order 
versus chaos is such a subtle and significant i ssue, and to tie it in with questions about the 
location and revelation of meaning, I woul d like to quote a beautiful and memorable 
passage from Are Quanta Real ?-a Galilean Dialogue by the late J. M. Jauch: 
 
SALVIATI Suppose I give you two sequences of numbers, such as 
78539816339744830961566084...  
And 
 
1, -1/3, +1/5, -1/7, +1/9, -1/11, +1/13, -1/15, ... 
 
If I asked you, Simplicio, what the next number of the first sequence is, what 
would you say?

And 
 
would you say? 
SIMPLICIO I could not tell you. I think it is a random sequence and that there is 
no law in it. 
SALVIATI And for the second sequence? 
SIMPLICIO That would be easy. It must be +1/17. 
SALVIATI Right. But what would you say if I told you that the first sequence is also constructed by a law and this law is in fact identical with the one you have just discovered for the s econd sequence? SIMPLICIO This does not 
seem probable to me. 
SALVIATI But it is indeed so, since the first sequence is simply the beginning of 
the decimal fraction [expansion] of the sum of the second. Its value is Tr/4.

the decimal fraction [expansion] of the sum of the second. Its value is Tr/4. 
SIMPLICIO You are full of such mathemati cal tricks, but I do not see what this 
has to do with abstraction and reality. 
SALVIATI The relationship with abstrac tion is easy to see. The first sequence 
looks random unless one has developed thr ough a process of abstraction a kind of 
filter which sees a simple structure behind the apparent randomness. 
It is exactly in this manner that laws of nature are discovered. Nature presents us with a host of phenomena wh ich appear mostly as chaotic randomness

It is exactly in this manner that laws of nature are discovered. Nature presents us with a host of phenomena wh ich appear mostly as chaotic randomness 
until we select some significant events, and ab stract from their particular, irrelevant 
circumstances so that they become idealized. Only then can they exhibit their true structure in full splendor. 
SAGREDO This is a marvelous idea! It suggests that when we try to understand 
nature, we should look at the phenomena as if they were messages  to be

BlooP and FlooP and GlooP 416understood. Except that each message appears to be random until we establish a code 
to read it. This code takes the form of an abstraction, that is, we choose to ignore 
certain things as irrelevant and we thus pa rtially select the content of the message by 
a free choice. These irrelevant signals form the "background noise," which will limit 
the accuracy of our message. 
But since the code is not absolute there may be several messages in the same raw 
material of the data, so changing the code  will result in a message of' equally deep 
significance in something that was merely noise before, and conversely: In a new code a former message may be devoid of meaning. 
Thus a code presupposes a free choice am ong different, complementary aspects, 
each of which has equal claim to reality , if I may use this dubious word. 
Some of these aspects may be comple tely unknown to us now but they may 
reveal themselves to an observer with a different system of abstractions.

reveal themselves to an observer with a different system of abstractions. 
But tell me, Salviati, how can we then still claim that we discover  something out 
there in the objective real world? Does th is not mean that we are merely creating 
things according to our own images and that reality is only within ourselves? 
SALVIATI I don't think that this is necessarily so, but it is a question which 
requires deeper reflection.' 
 
Jauch is here dealing with messages that co me not from a "sentient being" but from 
nature itself. The questions that we raised in Chapter VI on the relation of meaning to

nature itself. The questions that we raised in Chapter VI on the relation of meaning to 
messages can be raised equally well with messages from nature. Is nature chaotic, or is nature patterned? And what is the role of intelligence in determining the answer to this 
question? 
To back off from the philosophy, however, we can consider the point about the 
deep regularity of an apparently random sequence. Might the function Q(n) from Chapter 
V have a simple, nonrecursive explanation, t oo? Can every problem, like an orchard, be 
seen from such an angle that its secret is revealed? Or are there some problems in number

question? 
seen from such an angle that its secret is revealed? Or are there some problems in number 
theory which, no matter what angle they are seen from, remain mysteries? 
With this prologue, I feel it is time to m ove ahead to define the precise meaning 
of the term "predictably long search". This  will be accomplished in terms of the language 
B1ooP.  
Primordial Steps of the Language BlooP 
 Our topic will be searches for natural numbers which have various properties. In order to 
talk about the length  of any search, we shall have  to define some primordial steps , out of 
which all searches are built, so that length can be measured in terms of number of steps. Some steps which we might consider primordial are:

B1ooP.  
which all searches are built, so that length can be measured in terms of number of steps. Some steps which we might consider primordial are:   
adding any two natural numbers; multiplying any two natural numbers;  determining if two numbers are equal;  
determining the larger (smaller) of two numbers.

BlooP and FlooP and GlooP 417Loops and Upper Bounds 
 
If we try to formulate a test for, say, prima lity in terms of such steps, we shall soon see 
that we have to include a control structure -that is, descriptions of the order to do things 
in, when to branch back and try something ag ain, when to skip over a set of steps, when 
to stop, and similar matters. It is typical of any algorithm -that is, a specific delineation of how to carry out a task-that 
it includes a mixture of (1) specific operations to be performed, and (2) control statements. Therefore, as we develop our  language for expressing predictably long 
calculations, we shall have to incorporate prim ordial control structures also. In fact, the 
hallmark of BlooP is its limited set of contro l structures. It does not allow you to branch 
to arbitrary steps, or to repeat groups of steps without limit; in BlooP, essentially the only 
control structure is the bounded loop: a set of  instructions which can be executed over

control structure is the bounded loop: a set of  instructions which can be executed over 
and over again, up to a predefined maximum number of times, called the upper bound , or 
ceiling , of the loop. If the ceiling were 300, then the loop might be executed 0, 7, or 300 
times-but not 301. Now the exact values of all the upper bounds in a program need not be put in numerically 
by the programmer-indeed, they may not be  known in advance. Instead, any upper bound 
may be determined by calculations carried out before  its loop is entered. For instance, if 
you wanted to calculate the value of 2"', ther e would be two loops. First, you evaluate 3",

you wanted to calculate the value of 2"', ther e would be two loops. First, you evaluate 3", 
which involves n multiplications. Then, you put 2 to  that power, which involves 3" 
multiplications. Thus, the upper bound for the s econd loop is the result of the calculation 
of the first loop. 
Here is how you would express this in a BlooP program: 
 
DEFINE PROCEDURE "TWO-TO- THE-THREE-TO-THE" [N]:  
BLOCK 0: BEGIN 
CELL(O) <= 1; LOOP N TIMES: BLOCK 1: BEGIN 
CELL(0) ' 3 x CELL(O);  
          BLOCK 1: END; 
CELL(1) <= 1; LOOP CELL(O) TIMES:  BLOCK 2: BEGIN 
CELL(1) # 2 X CELL(l );

CELL(1) # 2 X CELL(l );  
         BLOCK 2: END;          OUTPUT  <= CELL( I );  
BLOCK 0: END. 
 
Conventions of BlooP 
 
Now it is an acquired skill to be able to look at an algorithm written in a computer 
language, and figure out what it is doing. Howeve r, I hope that this algorithm is simple 
enough that it makes sense without too much

BlooP and FlooP and GlooP 418scrutiny. A procedure  is defined, having one input parame ter, N; its output is the desired 
value. 
This procedure definition has what is called block structure , which means that 
certain portions of it are to be considered as units, or blocks. All the statements in a block 
get executed as a unit. Each block has a number (the outermost being BLOCK 0) , and is 
delimited by a BEGIN  and an END . In our example, BLOCK 1  and BLOCK 2 contain 
just one statement each but shortly you will s ee longer blocks. A LOOP statement always 
means to execute the block immediately under it repeatedly. As can be seen above, blocks can be nested. 
The strategy of the above algorithm is as  described earlier. You begin by taking 
an auxiliary variable, called CELL(O) ; you set it initially to 1, and then, in a loop, you 
multiply it repeatedly by 3 until you've done so exactly N times. Next, you do the 
analogous thing for CELL(1) -set it to 1, multiply by 2 exactly CELL(O)  times, then

analogous thing for CELL(1) -set it to 1, multiply by 2 exactly CELL(O)  times, then 
quit. Finally, you set OUTPUT  to the value of CELL(1) . This is the value returned to the 
outside world-the only externally visible behavior of the procedure. 
A number of points about the notation should be made here. First, the meaning of 
the left-arrow <= is this: 
 
Evaluate the expression to its right, then take the result and set the CELL  (or OUTPUT ) 
on its left to that value. 
 So the meaning of a command such as CELL(1)  <= 3 X CELL(1)  is to triple the value 
stored in CELL(1).  You may think of each CELL  as being a separate word in the

stored in CELL(1).  You may think of each CELL  as being a separate word in the 
memory of some computer. The only difference between a CELL  and a true word is that 
the latter can only hold integers up to some finite limit, whereas we allow a CELL  to 
hold any natural number, no matter how big. 
Every procedure in BlooP, when called, yi elds a value-namely the value of the 
variable called OUTPUT . At the beginning of execution of  any procedure, it is assumed 
as a default option that OUTPUT  has the value 0. That way, ev en if the procedure never 
resets OUTPUT  at all, OUTPUT  has a well-defined value at all times.

resets OUTPUT  at all, OUTPUT  has a well-defined value at all times. 
 
IF-Statements and Branching 
 Now let us look at another procedure which w ill show us some other features of BlooP 
which give it more generality. How do you fi nd out, knowing only how to add, what the 
value of M - N  is? The trick is to add various numbers onto N until you find the one 
which yields M. However, what happens if M is smaller than N? What if we are trying to 
take 5 from 2? In the domain of natural numbers, th ere is no answer. But we would like 
our B1ooP procedure to give an answer a nyway-let's say 0. Here, then, is a BlooP

our B1ooP procedure to give an answer a nyway-let's say 0. Here, then, is a BlooP 
procedure which does subtraction:

BlooP and FlooP and GlooP 419DEFINE PROCEDURE "MINUS" [M,N]:  
IF M < N, THEN: QUIT BLOCK 0; LOOP AT MOST M + 1 TIMES:  BLOCK 1: BEGIN 
IF OUTPUT + N = M, THEN: ABORT LOOP 1; OUTPUT, <= OUTPUT + 1;  
BLOCK 1: END; 
 
Here we are making use of the implicit feature that OUTPUT  begins at 0. If M is 
less than N, then the subtraction is impossible, and we simply jump to the bottom of 
BLOCK  0 right away, and the answer is 0. That is what is meant by the line QUIT 
BLOCK  0. But if M is not less than N, then we skip over that QUIT -statement, and 
carry out the next command in sequence (here, a LOOP -statement). That is how IF-
statements always work in BlooP. 
So we enter LOOP  1, so called because the block wh ich it tells us to repeat is 
BLOCK  1. We try adding 0 to N, then 1, 2, etc., until we find a number that gives M. At 
that point, we ABORT  the loop we are in, meaning we jump to the statement

that point, we ABORT  the loop we are in, meaning we jump to the statement 
immediately following the END  which marks the bottom of the loop's block. In this case, 
that jump brings us just below BLOCK 1 : END , which is to say, to the last statement of 
the algorithm, and we are done. OUTPUT  now contains the correct answer. 
Notice that there are two distinct instructions for jumping downwards: QUIT , and 
ABORT . The former pertains to blocks, the latter to loops. QUIT BLOCK  n means to 
jump to the last line of BLOCK n , whereas ABORT LOOP n  means to jump just below

jump to the last line of BLOCK n , whereas ABORT LOOP n  means to jump just below  
the last line of BLOCK n . This distinction only matters when you are inside a loop and 
want to continue looping but to quit th e block this time around. Then you can say QUIT  
and the proper thing will happen. 
Also notice that the words AT MOST  now precede the upper bound of the loop, 
which is a warning that the loop may be aborted before the upper bound is reached. 
 
Automatic Chunking 
 Now there are two last features of BlooP to  explain, both of them very important. The

 Now there are two last features of BlooP to  explain, both of them very important. The 
first is that, once a procedure has been defined , it may be called  inside later procedure 
definitions. The effect of this is that once an operation has been defined in a procedure, it 
is considered as simple as a primordial step . Thus, BlooP features automatic chunking. 
You might compare it to the way a good ice sk ater acquires new motions: not by defining 
them as long sequences of primordial muscle- actions, but in terms of previously learned 
motions, which were themselves learned as compounds of earlier

BlooP and FlooP and GlooP 420learned motions, etc.-and the nestedness, or  chunkedness, can go back many layers until 
you hit primordial muscle-actions And thus, the repertoire of BlooP programs, like the 
repertoire of a skater's tricks, grows, quite literally, by loops and bounds.  
BlooP Tests 
 The other feature of BlooP is that certain procedures can have YES  or NO as their 
output, instead of an integer value. Such procedures are tests, rather than functions . To 
indicate the difference, the name of a test mu st terminate in a question mark. Also, in a 
test, the default option for OUTPUT  is not 0, of course, but NO. 
Let us see an example of these last two feat ures of BlooP in an algorithm which tests its 
argument for primality:  
DEFINE PROCEDURE "PRIME?" [N]: 
IF N = 0, THEN:  QUIT BLOCK 0;  CELL(0) <= 2; LOOP AT MOST MINUS [N,2] TIMES:  BLOCK 1: BEGIN 
IF REMAINDER [N,CELL(O)] = 0, THEN: QUIT BLOCK 0; CELL(O) <= CELL(O) + 1;  
BLOCK 1: END; OUTPUT  <= YES; 
BLOCK 0: END.

Notice that I have called two procedures inside this algorithm: MINUS  and 
REMAINDER . (The latter is presumed to have been previously defined, and you may 
work out its definition yourself.) Now this te st for primality works by trying out potential 
factors of N one by one, starting at 2 and increasing to a maximum of N - 1 . In case any 
of them divides N exactly (i.e., gives remainder 0), then we jump down to the bottom, 
and since OUTPUT  still has its default value at this stage, the answer is NO. Only if N

and since OUTPUT  still has its default value at this stage, the answer is NO. Only if N 
has no exact divisors will it survive the entirety of LOOP 1;  then we will emerge 
smoothly at the statement OUTPUT <= YES,  which will get executed, and then the 
procedure is over.  
BlooP Programs Contain Chains of Procedures 
 We have seen how to define procedures in  BlooP; however, a procedure definition is only 
a part of a program. A program  consists of a chain of procedure definitions  (each only 
calling previously defined procedures),  optionally followed by one or more calls  on the 
procedures defined. Thus, an

BlooP and FlooP and GlooP 421example of a full B1ooP program would be the definition of the procedure TWO-TO-
THE-THREE-TO-THE , followed by the call 
 
TWO-TO-THE-THREE-TO-THE [2] 
 
which would yield an answer of 512. 
If you have only a chain of procedure defi nitions, then nothing ever gets executed; 
they are all just waiting for some call, with  specific numerical values, to set them in 
motion. It is like a meat grinder waiting for some  meat to grind-or rather, a chain of meat 
grinders all linked together, each of which is  fed from earlier ones ... In the case of meat 
grinders, the image is perhaps not so savory; however, in the case of BlooP programs, such a construct is quite important, and we will call it a "call-less program". This notion 
is illustrated in Figure 72. 
Now B1ooP is our language for defining predictably terminating calculations. The 
standard name for functions  which are B1ooP-computable is primitive recursive

standard name for functions  which are B1ooP-computable is primitive recursive 
functions;  and the standard name for properties  which can be detected by B1ooP-tests is 
primitive recursive predicates . Thus, the function 2
3n is a primitive recursive function; 
and the statement " n is a prime number" is a primitive recursive predicate. 
It is clear intuitively that the Goldbach  property is primitive recursive, and to 
make that quite explicit, here is a procedure definition in BlooP, showing how to test for its presence or absence: 
 
DEFINE PROCEDURE "GOLDBACH?" [N]:  
CELL(0) 2; LOOP AT MOST N TIMES:

CELL(0) 2; LOOP AT MOST N TIMES:  
BLOCK 1: BEGIN 
IF {PRIME? [CELL(O)]      AND PRIME? [MINUS [N,CELL(0)]]}, 
THEN:  BLOCK 2: BEGIN 
OUTPUT,# YES; QUIT BLOCK 0-,  
BLOCK 2: END CELL(0) <= CELL(0) + 
 
As usual, we assume NO until proven YES , and we do a brute force search among pairs 
of numbers which sum up to N. If both are prime, we quit the outermost block; otherwise 
we just go back and try again, until all possibilities are exhausted. (Warning: The fact that the Goldbach prope rty is primitive recursive does not make the

we just go back and try again, until all possibilities are exhausted. (Warning: The fact that the Goldbach prope rty is primitive recursive does not make the 
question “Do all numbers have the Goldbach property?” a simple question—far from it!)

BlooP and FlooP and GlooP 422
  
FIGURE 72. The structure of a call-less BlooP program. For this program to be self-
contained, each procedure definition may only call procedures defined above it.  
Suggested Exercises 
 Can you write a similar B1ooP procedure which tests for the presence or absence of the 
Tortoise property (or the Achilles property)? If  so, do it. If not, is it merely because you 
are ignorant about upper bounds, or could it be  that there is a fundamental obstacle 
preventing the formulation of such an al gorithm in BlooP? And what about the same 
questions, with respect to the property of wondrousness, defined in the Dialogue? Below, I list some functions and properties, and you ought to take the time to determine whether you believe they are primitive recurs ive (BlooP-programmable) or not. This 
means that you must carefully consider what kinds of operations will be involved in the 
calculations which they require, and whethe r ceilings can be given for all the loops

calculations which they require, and whethe r ceilings can be given for all the loops 
involved.  FACTORIAL [N] = NI ( the factorial of  N)  
   (e.g., FACTORIAL [4] = 24) 
REMAINDER [M,N] = the remainder upon dividing  M by N  
   (e.g., REMAINDER [24,7] = 3)  
PI-DIGIT [N] = the Nth digit of pi, after the decimal point  
   (e.g. PI-DIGIT [1] = 1, 
PI-DIGIT [2] = 4 
 PI-DIGIT [1000000] = 1

BlooP and FlooP and GlooP 423FIBO [N] = the Nth Fibonacci number   
(e.g.,  FIBO [9] = 34) 
PRIME-BEYOND [N[ = the lowest prime beyond  N  
(e.g.,  PRIME-BEYOND [33] = 37) 
PERFECT [N] = the Nth "perfect" number (a number such as 28 whose divisors sum up 
to itself: 28 = 1 + 2 + 4 + 7 + 14 )  
(e.g.,  PERFECT [2] = 28) 
PRIME? [N] = YES if N is prime, otherwise NO. 
PERFECT? [N] = YES  if N is perfect, otherwise NO. 
TRIVIAL? [A,B,C,N] = YES if A"+B" = Cn is correct; otherwise  NO. 
(e.g.,   TRIVIAL? [3,4,5,2] = YES,  
TRIVIAL? [3,4,5,3] = NO) 
PIERRE? [A,B,C] = YES if A"+B" = C" is satisfiable for some value of  N greater  than 
1, otherwise  NO. 
(e.g.,   PIERRE? [3,4,5] = YES, 
PIERRE? [1,2,3] = NO) 
FERMAT? [N] = YES if A"+B" = CN is satisfied by some positive 
values of A, B, C; otherwise  NO. 
(e.g.,   FERMAT? [2] = YES) 
TORTOISE-PAIR? [M,N] = YES if both  M and M + N are prime, otherwise NO. 
(e.g.,   ORTOISE-PAIR [5,1742] = YES,  
TORTOISE-PAIR [5,100] = NO)

TORTOISE-PAIR [5,100] = NO) 
TORTOISE? [N] = YES if N is the difference of two primes, otherwise NO.  
(e.g.,   TORTOISE [1742] = YES, 
TORTOISE [7] = NO) 
MIU-WELL-FORMED? [N] = YES if N, when seen as a string of the  MIU-System, is 
well-formed; otherwise  NO. 
(e.g.,   MIU-WELL-FORMED? [310] = YES, 
MIU-WELL-FORMED? [415] = NO) 
MIU-PROOF-PAIR? [M,N] = YES If M, as seen as a sequence of strings of the  MIU-
system , is a derivation of  N, as seen as a string of the  MIU- system; otherwise  NO. 
(e.g.,   MIU-PROOF-PAIR? [3131131111301,301] = YES,

(e.g.,   MIU-PROOF-PAIR? [3131131111301,301] = YES,  
MIU-PROOF-PAIR? [311130,30] = NO) 
MIU-THEOREM? [N] = YES if N, seen as a MIU- system string, is a theorem; 
otherwise  NO. 
(e.g.,   MIU-THEOREM? [311] = YES, 
MIU-THEOREM? [30] = NO, 
MIU-THEOREM? [701] = NO) 
TNT-THEOREM? [N] = YES if N, seen as a TNT- string, is a theorem. 
(e.g.,   TNT-THEOREM? [666111666] = YES, 
TNT-THEOREM? [123666111666] = NO, 
TNT-THEOREM? [7014] = NO)

BlooP and FlooP and GlooP 424FALSE? [N] = YES if N, seen as a  TNT- string, is a false statement of number theory; 
otherwise NO. 
(e.g.,  FALSE? [6661 1 1666] = NO, 
FALSE? [2236661 1 1666] = YES, FALSE? [7014] = NO)  
The last seven examples are particularly relevant to our future metamathematical 
explorations, so they highly merit your scrutiny.  
Expressibility and Representability 
 
Now before we go on to some interesting questions about BlooP and are led to its 
relative, FlooP, let us return to the reason for introducing BlooP in the first place, and connect it to TNT . Earlier, I stated that the criti cal mass for Gödel’s method to be 
applicable to a formal system is attain ed when all primitive recursive notions are 
representable in that system. Exactly what does this mean? First of all, we must distinguish between the notions of representability and expressibility. Expressing  a 
predicate is a mere matter of translation from  English into a strict formalism. It has

predicate is a mere matter of translation from  English into a strict formalism. It has 
nothing to do with theoremhood. For a predicate to be represented , on the other hand, is a 
much stronger notion. It means that  
(1) All true instances of the predicate are theorems;  (2) All false instances are nontheorems. 
 By "instance", I mean the string produced  when you replace all free variables by 
numerals. For example, the predicate m + n = k  is represented in the pq-system, because 
each true instance of the predicate is a theo rem, each false instance is a nontheorem. Thus 
any specific addition, whether true or false, translates into a decidable  string of the pq-
system. However, the pq-system is unable to express-let alone represent-any other

By "instance", I mean the string produced  when you replace all free variables by 
system. However, the pq-system is unable to express-let alone represent-any other 
properties of natural numbers. Therefore it would be a weak candidate  indeed in a 
competition of systems which can do number theory. 
Now TNT  has the virtue of being able to express  virtually any number-theoretical 
predicate; for example, it is easy to write a TNT -string which expresses the predicate " b 
has the Tortoise property". Thus, in terms of expressive power, TNT  is all we want. 
However, the question "Which properties are represented  in TNT?" is Precisely 
the question "How powerful an axiomatic system is TNT ?" Are all Possible predicates

the question "How powerful an axiomatic system is TNT ?" Are all Possible predicates 
represented in TNT ? If so, then TNT  can answer any question of number theory; it is 
complete.  
Primitive Recursive Predicat es Are Represented in TNT 
 
Now although completeness will turn out to be a chimera. TNT  is at least complete with 
respect to primitive recursive predicates. In other words, a ny statement of number theory 
whose truth or falsity can be decided by a

BlooP and FlooP and GlooP 425computer within a predictable length of time is also decidable inside TNT . Or, one final 
restatement of the same thing:  
If a BlooP test can be written for some property of natural numbers, then that 
property is represented in TNT . 
 
Are There Functions Which Are Not Primitive Recursive? 
 Now the kinds of properties which can be de tected by BlooP tests are widely varied, 
including whether a number is prime or perfect , has the Goldbach property, is a power of 
2, and so on and so forth. It woul d not be crazy to wonder whether every  property of 
numbers can be detected by some suitable BlooP  program. The fact that, as of the present 
moment, we have no way of testing whethe r a number is wondrous or not need not 
disturb us too much, for it might merely m ean that we are ignorant about wondrousness, 
and that with more digging around, we could discover a universal formula for the upper

and that with more digging around, we could discover a universal formula for the upper 
bound to the loop involved. Then a BlooP test  for wondrousness could be written on the 
spot. Similar remarks could be made about the Tortoise property. 
So the question really is, "Can upper bounds always be given for the length of 
calculations-or, is there an inherent kind of jumbliness to the natural number system,

calculations-or, is there an inherent kind of jumbliness to the natural number system, 
which sometimes prevents calculation lengths from being predictable in advance?" The striking thing is that the latter is the case, and we are about to see why. It is the sort of thing that would have driven Pythagoras, who first proved that the square root of 2 is 
irrational, out of his mind. In our de monstration, we will use the celebrated diagonal 
method,  discovered by Georg Cantor, the founder of set theory. 
 
Pool B, Index Numbers, and Blue Programs

Pool B, Index Numbers, and Blue Programs 
 We shall begin by imagining a curious notion:  the pool of all possible BlooP programs. 
Needless to say, this pool-"Pool B"-is an infinite one. We want to consider a subpool of Pool B, obtained by three successive filtering ope rations. The first filter will retain for us 
only call-less  programs. From this subpool we then  eliminate all tests, leaving only 
functions . (By the way, in call-less programs, the last procedure in the chain determines 
whether the program as a whole is considered  a test, or a function.)  The third filter will

whether the program as a whole is considered  a test, or a function.)  The third filter will 
retain only functions which have exactly one input parameter . (Again referring to the 
final procedure in the chain.) What is left?  
A complete pool of all call-less BlooP programs which calculate functions of 
exactly one input parameter. 
 Let us call these special BlooP programs Blue Programs . 
What we would like to do now is to assign an unambiguous index 
number  to each Blue Program. How can this be done? The easiest way—we shall use it—
is to list them in order of length: the shortest possible. Blue

BlooP and FlooP and GlooP 426Program being # 1, the second shortest being #2, etc. Of course, there will be many 
programs tied for each length. To break such ties, we use alphabetical order. Here, "alphabetical order" is taken in an extended sense, where the alphabet includes all the special characters of BlooP, in some arbitrary order, such as the following:  
A B C D E F G H I J K  L M N 
O P Q R S T U V W X Y  Z + x 
0 1 2 3 4 5 6 7 8 9 <= = < > ( ) [ ] { } - ´ ? : ;  , . 
 
-and at the end comes the lowly blank! Altoge ther, fifty-six characters. For convenience's 
sake, we can put all Blue Programs of length 1 in Volume 1, programs of 2 characters in 
Volume 2, etc. Needless to say, the first fe w volumes will be totally empty, while later 
volumes will have many, many entries (t hough each volume will only have a finite 
number). The very first Blue Program would be this one:  
DEFINE PROCEDURE "A" [B]: 
BLOCK 0: BEGIN BLOCK 0: END.

BLOCK 0: BEGIN BLOCK 0: END. 
 
This rather silly meat grinder returns a value of 0 no matter what its input is. It occurs in 
Volume 56, since it has 56 characters (coun ting necessary blanks, including blanks 
separating successive lines). 
Soon after Volume 56, the volumes will get extremely fat, because there are just 
so many millions of ways of combining sym bols to make Blue BlooP programs. But no 
matter-we are not going to try to print out this infinite catalogue. All that we care about is

matter-we are not going to try to print out this infinite catalogue. All that we care about is 
that, in the abstract, it is well-defined, and that each Blue BlooP program therefore has a 
unique and definite index number. This is the crucial idea. 
Let us designate the function calculated by the kth Blue Program this way: 
 
Blueprogram{# k} [N] 
 Here, k is the index number of the program, and N is the single input parameter. For 
instance, Blue Program # 12 might return a value twice the size of its input:  
Blueprogram{#12} [N] = 2 x N

Blueprogram{#12} [N] = 2 x N 
 
The meaning of the equation above is that the program  named on the left-hand side 
returns the same value as a human would calculate from the ordinary algebraic 
expression on the right-hand side. As anothe r example, perhaps the 5000th Blue Program 
calculates the cube of its input parameter:  
Blueprogram{#5000} [N] = N3

BlooP and FlooP and GlooP 427The Diagonal Method 
 
Very well-now we apply the "twist": Cant or's diagonal method. We shall take this 
catalogue of Blue Programs and use it to define a new function of one variable- Bluediag  
[N]-which will turn out not to be anywhere in the list (which is why its name is in italics). 
Yet Bluediag will clearly be a well-defined, calculable function of one variable, and so 
we will have to conclude that functions exist which simply are not programmable in BlooP. 
Here is the definition of Bluediag ~N]: 
 
Equation (1) ... Bluediag [N] = 1 + Blueprogram{#N} [N] 
 
The strategy is: feed each meat grinder with  its own index number, then add 1 to the 
output. To illustrate, let us find Bluediag [12]. We saw that Blueprogram{# 12} is the 
function 2N; therefore, Bluediag [12] must  have the value 1 + 2 x 12, or 25. Likewise,

function 2N; therefore, Bluediag [12] must  have the value 1 + 2 x 12, or 25. Likewise, 
Bluediag [5000] would have the value 125,000,000,001, since that is 1 more than the cube of 5000. Similarly, you can find Bluediag  of any particular argument you wish. 
The peculiar thing about Bluediag [N] is that it is not represented in the catalogue 
of Blue Programs. It cannot be. The reason is  this. To be a Blue Program, it would have 
to have an index number-say it were Blue Pr ogram # X. This assumption is expressed by 
writing 
 Equation (2) ... Bluediag  [N] = Blueprogram{# X} [N]

writing 
 Equation (2) ... Bluediag  [N] = Blueprogram{# X} [N] 
 
But there is an inconsistency between the equations (1) and (2). It becomes apparent at 
the moment we try to calculate the value of  Bluediag [ X], for we can do so by letting N 
take the value of X in either of the two equa tions. If we substitute into equation (1), we 
get: 
 
Bluediag  [ X] = 1 + Blueprogram{# X} [ X]  
 
But if we substitute into equation (2) instead, we get:  
 
Bluediag  [ X] = Blueprogram{# X} [ X] 
 
Now Bluediag  [ X] cannot be equal to a number and also to the successor of that number.

get: 
 
Now Bluediag  [ X] cannot be equal to a number and also to the successor of that number. 
But that is what the two equations say. So  we will have to go back and erase some 
assumption on which the inconsistency is based. The only possible candidate for erasure is the assumption expressed by Equation (2): that the function Bluediag  [N] is able to be 
coded up as a Blue BlooP program . And that is the proof that Bluediag lies outside the 
realm of primitive recursive functions . Thus, we have achieved our aim of destroying 
Achilles' cherished but naive notion that every number-theoretical function must be 
calculable within a predictable number of steps.

calculable within a predictable number of steps. 
There are some subtle things going on he re. You might ponder this, for instance: 
the number of steps involved in the calculation of Bluediag  [N],for each specific  value of 
N, is predictable—but the different methods  of prediction cannot all be united into a 
general   recipe for predict

BlooP and FlooP and GlooP 428ing the length of calculation of Bluediag  [N]. This is an "infinite conspiracy", related to 
the Tortoise's notion of "infinite coinciden ces", and also to w-incompleteness. But we 
shall not trace out the relations in detail.  
Cantor's Original Diagonal Argument 
 Why is this called a diagonal argument? The terminology comes from Cantor's original 
diagonal argument, upon which many other argumen ts (such as ours) have subsequently 
been based. To explain Cantor's original argum ent will take us a little off course, but it is 
worthwhile to do so. Cantor, too, was concerne d with showing that so me item is not in a 
certain list. Specifically, what Cantor wanted to show was that if a "directory" of real 
numbers were made, it would inevitably leave some real numbers out-so that actually, the notion of a complete directory of real numbers is a contradiction in terms. 
It must be understood that this pertains not  just to directories of finite size, but

It must be understood that this pertains not  just to directories of finite size, but 
also to directories of infinite size. It is a much deeper result than the statement "the 
number of reals is infinite, so of course they  cannot be listed in a finite directory". The 
essence of Cantor's result is that there are (at least) two distinct types of infinity: one kind 
of infinity describes how many entries there can be in an infinite directory or table, and 
another describes how many real numbers th ere are (i.e., how many points there are on a

another describes how many real numbers th ere are (i.e., how many points there are on a 
line, or line segment)-and this latter is "bigger", in the sense that the real numbers cannot be squeezed into a table whose length is desc ribed by the former kind of infinity. So let 
us see how Cantor's argument involves the notion of diagonal, in a literal sense. 
Let us consider just real numbers be tween 0 and 1. Assume, for the sake of 
argument, that an infinite list could be gi ven, in which each positive integer N is matched 
up with a real number r(N) between 0 and 1, and in which each real number between 0

up with a real number r(N) between 0 and 1, and in which each real number between 0 
and 1 occurs somewhere down the line. Si nce real numbers are given by infinite 
decimals, we can imagine that the beginning of the table might look as follows: 
 
r(1): . 1 4 1 5 9 2 6 5  3 
r(2): .3 3 3 3 3 3 3 3  3 
r(3): .7 1 8 2 8 1 8 2 8 
r(4): .4 1 4 2 1 3 5 6  2 
r(5): .5 0 0 0 0 0 0 0  0  
 The digits that run down the diagonal are in boldface: 1, 3, 8, 2, 0.... Now those diagonal 
digits are going to be used in making a special real number d, which is between 0 and 1 
but which, we will see, is not in the list. To make d, you take the diagonal digits in order, 
and change each one of them to some other di git. When you prefix th is sequence of digits

and change each one of them to some other di git. When you prefix th is sequence of digits 
by a decimal point you have d. There are of course many wa ys of changing a digit to 
some other digit, and correspondingly many different d´s. Suppose for,

BlooP and FlooP and GlooP 429example, that we subtract 1 from the diagonal digits  (with the convention that 1 taken 
from 0 is 9). Then our number d will be:  
.0  2  7  1  9 . 
 Now, because of the way we constructed it,  
d's 1st digit is not the same as the 1st digit of r(1);  
d's 2nd digit is not the same as the 2nd digit of r(2);  
d's 3rd digit is not the same as the 3rd digit of r(3); 
 ... and so on. 
 Hence,  
d is different from r(1);  
d is different from r(2);  
d is different from r(3); 
... and soon. 
 In other words,  d is not in the list! 
 
What Does a Diagonal Argument Prove? 
 Now comes the crucial difference between Cant or's proof and our proofit is in the matter 
of what assumption to go back and undo. In Cantor's argument, the shaky assumption 
was that such a table could be drawn up. Therefore, the conclusion warranted by the 
construction of d is that no exhaustive tabl e of reals can be dr awn up after all-which

construction of d is that no exhaustive tabl e of reals can be dr awn up after all-which 
amounts to saying that the set of integers is just not big enough to index the set of reals. 
On the other hand, in our proof, we know that  the directory of Blue BlooP programs can 
be drawn up-the set of integers is big enough to index the set of Blue BlooP programs. 
So, we have to go back and retract some shak ier idea which we used. And that idea is that 
Bluediag  [N] is calculable by some program in Bl ooP. This is a subtle difference in the 
application of the diagonal method.

application of the diagonal method. 
It may become clearer if we apply it to the alleged "List of All Great 
Mathematicians" in the Dialogue-a more c oncrete example. The diagonal itself is 
"Dboups". If we perform the desired diagonal- subtraction, we will get "Cantor". Now two 
conclusions are possible. If you have an unshakable belief that the list is complete , then 
you must conclude that Cantor is not a Great Mathematician, for his name differs from all 
those on the list. On the other hand, if you ha ve an unshakable belief that Cantor is a

those on the list. On the other hand, if you ha ve an unshakable belief that Cantor is a 
Great Mathematician, then you must conclude that the List of All Great Mathematicians is incomplete, for Cantor's name is not on the list! (Woe to those who have unshakable 
beliefs on both sides!) The former case corresponds  to our proof that Bluediag [N] is not 
primitive recursive; the latter case corresponds to  Cantor´s proof that the list of reals is 
incomplete;

BlooP and FlooP and GlooP 430
 
FIGURE 73. Georg Cantor 
 Cantor's proof uses a diagonal in the literal sense of the word. Other "diagonal" proofs are 
based on a more general notion, which is abst racted from the geometric sense of the 
word. The essence of the diagonal method is the fact of using one integer in two different 
ways-or, one could say, using one integer on two different levels -thanks to which one can 
construct an item which is outside of some  predetermined list. One time, the integer 
serves as a vertical  index, the other time as a horizontal  index. In Cantor's construction 
this is very clear. As for the function Bluediag  [N], it involves using one integer on two 
different levels-first, as a Blue Program inde x number; and second, as an input parameter. 
 
The Insidious Repeatability of the Diagonal Argument 
 At first, the Cantor argument may seem less than fully convincing. Isn't there some way

 At first, the Cantor argument may seem less than fully convincing. Isn't there some way 
to get around it? Perhaps by throwing in the diagonally constructed number d, one might 
obtain an exhaustive list. If you consider this  idea, you will see it helps not a bit to throw 
in the number d, for as soon as you assign it a specific place in the table, the diagonal 
method becomes applicable to the new table, and a new missing number d' can be constructed, which is not in the new tabl e. No matter how many times you repeat the 
operation of constructing a number by the di agonal method and then throwing it in to

operation of constructing a number by the di agonal method and then throwing it in to 
make a "more complete" table, you still are caught on the ineradicable hook of Cantor’s method. You might even try to build a table of reals which tries to outwit the Cantor 
diagonal method by taking

BlooP and FlooP and GlooP 431the whole trick, lock, stock, and barrel, incl uding its insidious repeatability, into account 
somehow. It is an interesting exercise. Bu t if you tackle it, you will see that no matter 
how you twist and turn trying to avoid the Cantor "hook", you are still caught on it. One 
might say that any self-proclaimed "table of all reals" is hoist by its own petard. 
The repeatability of Cantor's diagonal met hod is similar to the repeatability of the 
Tortoise's diabolic method for breaking th e Crab's phonographs, one by one, as they got 
more and more "hi-fi" and-at least so the Crab hoped-more "Perfect". This method involves constructing, for each phonograph, a particular song which that phonograph 
cannot reproduce. It is not a coincidence that Cantor's trick and the Tortoise's trick share 
this curious repeatability; indeed, the Contracrostipunctus might well have been named "Cantorcrostipunctus " instead. Moreover, as the Tortoise subtly hinted to the innocent

this curious repeatability; indeed, the Contracrostipunctus might well have been named "Cantorcrostipunctus " instead. Moreover, as the Tortoise subtly hinted to the innocent 
Achilles, the events in the Contracrostipunctu s are a paraphrase of the construction which 
Gödel used in proving his Incompleteness Theorem; it follows that the Gödel construction is also very much like a diagonal construction. This will become quite apparent in the next two Chapters. 
 
From BlooP to FlooP 
 We have now defined the class of primitive recursive functions and primitive recursive properties of natural numbers by means of programs written in the language BlooP. We 
have also shown that BlooP doesn't capture a ll the functions of natural numbers which we

have also shown that BlooP doesn't capture a ll the functions of natural numbers which we 
can define in words. We even constructed an "unBlooPable" function, Bluediag [N], by 
Cantor's diagonal method. What is it about BlooP  that makes Bluediag unrepresentable in 
it? How could BlooP be improved so th at Bluediag became representable? 
BlooP's defining feature was the boundedness of its loops. What if we drop that 
requirement on loops, and invent a second language, called "FlooP" (`F' for "free")? 
FlooP will be identical to BlooP except in  one respect: we may have loops without

FlooP will be identical to BlooP except in  one respect: we may have loops without 
ceilings, as well as loops with ceilings (although the only reason one would include a 
ceiling when writing a loop-statement in FlooP would be for the sake of elegance). These 
new loops will be called MU-LOOPS . This follows the convention of mathematical 
logic, in which "free" searches (search es without bounds) are usually indicated by a 
symbol called a "µ-operator" (mu-operator).  Thus, loop statements in FlooP may look 
like this:  
MU-LOOP:  
BLOCK n: BEGIN 
  . 
.   
BLOCK n: END

BlooP and FlooP and GlooP 432This feature will allow us to write tests in FlooP for such properties as wondrousness 
and the Tortoise property-tests which we di d not know how to program in BlooP because 
of the potential open-endedness of the searches  involved. I shall leave it to interested 
readers to write a FlooP test for wondrousness which does the following things:  
(1) If its input, N, is wondrous, the program halts and gives the answer YES . 
(2) If N is unwondrous, but causes a closed cycle other than 1-4-2-1-4-2-1- ... , the 
program halts and gives the answer NO. 
(3) If N is unwondrous, and causes an "e ndlessly rising progression", the program 
never halts. This is FlooP's way of  answering by not answering. FlooP's 
nonanswer bears a strange resemblance to Joshu's nonanswer " MU". 
 The irony of case 3 is that OUTPUT  always has the value NO, but it is always 
inaccessible, since the program is still grindi ng away. That troublesome third alternative

inaccessible, since the program is still grindi ng away. That troublesome third alternative 
is the price that we must pay for the right  to write free loops. In all FlooP programs 
incorporating the MU-LOOP  option, nontermination will always be one theoretical 
alternative. Of course there will be many Fl ooP programs which actually terminate for all 
possible input values. For instance, as I men tioned earlier, it is suspected by most people 
who have studied wondrousness that a FlooP  program such as suggested above will 
always terminate, and moreover with the answer YES  each time. 
 
Terminating and Nonterminating FlooP Programs

Terminating and Nonterminating FlooP Programs 
 It would seem extremely desirable to be able to separate FlooP procedures into two classes: terminators  and nonterminators . A terminator will eventually halt no matter what 
its input, despite the " MU-ness" of its loops. A nontermi nator will go on and on forever, 
for at least one  choice of input. If we could always  tell, by some kind of complicated 
inspection of a FlooP program, to which class it belonged, there would be some

inspection of a FlooP program, to which class it belonged, there would be some 
remarkable repercussions (as we shall shortly see). Needless to say, the operation of class-checking would itself have to be a terminating operation-otherwise one would gain nothing!  
Turing's Trickery 
 The idea springs to mind that we might le t a BlooP procedure do the inspection. But 
BlooP procedures only accept numerical input , not programs! However, we can get 
around that ... by coding programs into numbers! This sly trick is just Gödel-numbering 
in another of its many

BlooP and FlooP and GlooP 433a very long Gödel number. For instance, the shortest BlooP function (which is also a 
terminating FlooP program)  
DEFINE PROCEDURE "A" [B]:  
BLOCK 0: BEGIN  BLOCK 0: END. 
 
-would get the Godel number partially shown below:  
904, 905, 906, 909, 914, 905  905, 914.904, 955, 
D E F I N E     E N D . 
 
Now our scheme would be to  write a BlooP test called TERMINATOR ? which 
says YES  if its input number codes for a terminating FlooP program, NO if not. This way 
we could hand the task over to a machine a nd with luck, distinguish terminators from 
nonterminators. However, an ingenious ar gument given by Alan Turing shows that no 
BlooP program can make this distinction infallibly. The trick is actually much the same as Gödel’s trick, and therefore closely relate d to the Cantor diagonal trick. We shall not 
give it here-suffice it to say that the idea is to feed the termination tester its own Godel

give it here-suffice it to say that the idea is to feed the termination tester its own Godel 
number. This is not so simple, however, for it is like trying to quote an entire sentence 
inside itself. You have to quote the quote, and so forth; it seems to lead to an infinite 
regress. However, Turing figured out a tr ick for feeding a program its own Godel 
number. A solution to the same problem in a different context will be presented next Chapter. In the present Chapter, we shall take  a different route to the same goal, which is

number. A solution to the same problem in a different context will be presented next Chapter. In the present Chapter, we shall take  a different route to the same goal, which is 
namely to prove that a termination tester is impossible. For readers who wish to see an elegant and simple presentation of the Turing approach, I recommend the article by 
Hoare and Allison, mentioned in the Bibliography. 
 
A Termination Tester Would Be Magical 
 Before we destroy the notion, let us delineate just why having a termination tester would 
be a remarkable thing. In a sense, it woul d be like having a magical dowsing rod which 
could solve all problems of number theory in one swell FlooP. Suppose, for instance, that

could solve all problems of number theory in one swell FlooP. Suppose, for instance, that 
we wished to know if the Goldbach Variation is a true conjecture or not. That is, do all 
numbers have the Tortoise property? We would begin by writing a FlooP test called 
TORTOISE ? which checks whether its input has th e Tortoise property. Now the defect 
of this procedure-namely that it doesn't term inate if the Tortoise property is absent-here 
turns into a virtue! For now we run th e termination tester on the procedure TORTOISE ?. 
If it says YES , that means that TORTOISE ? terminates for all values of its input-in other 
words, all numbers have the Tortoise property. If it says NO, then we know there exists a

words, all numbers have the Tortoise property. If it says NO, then we know there exists a 
number which has the Achilles property. The irony is that we never actually use the 
program TORTOISE  at all—we just inspect it. 
 This idea of solving any problem in number theory by coding it into a

BlooP and FlooP and GlooP 434program and then waving a termination tester  over the program is not unlike the idea of 
testing a khan for genuineness by coding it into a folded string and then running a test for 
Buddha-nature on the string instead. As Achilles suggested, perhaps the desired inform ation lies "closer to the surface" in one 
representation than in another.  
Pool F, Index Numbers, and Green Programs 
 
Well, enough daydreaming. How can we prove that  the termination tester is impossible? 
Our argument for its impossibility will hinge on trying to  apply the diagonal argument to 
FlooP, just as we did to B1ooP. We shall s ee that there are some subtle and crucial 
differences between the two cases. 
As we did for BlooP, imagine the pool of all FlooP programs. We shall call it 
"Pool F". Then perform the same three filteri ng operations on Pool F, so that you get, in 
the end:  
A complete pool of all call-less FlooP programs which calculate functions of exactly one input parameter.

the end:  
A complete pool of all call-less FlooP programs which calculate functions of exactly one input parameter. 
 Let us call these special FlooP-programs Green Programs  (since they may go forever). 
Now just as we assigned index numbers to all Blue Programs, we can assign 
index numbers to Green Programs, by orde ring them in a catalogue, each volume of 
which contains all Green Programs of a fi xed length, arranged in alphabetical order. 
So far, the carry-over from BlooP to FlooP ha s been straightforward. Now let us see if we 
can also carry over the last part: the diagonal trick. What if we try to define a diagonal

can also carry over the last part: the diagonal trick. What if we try to define a diagonal 
function?  
Greendiag  [N] = 1 + Greenprogram{# N} [N] 
 Suddenly, there is a snag: this function Greendiag  [N] may not have a well-defined 
output value for all input values N. This is simply because we have not filtered out the 
nonterminator programs from Pool F, and th erefore we have no guarantee that we can 
calculate Greendiag  [N] for all values of N. Sometimes we may enter calculations which 
never terminate. And the diagonal argument ca nnot be carried through in such a case, for 
it depends on the diagonal function having a value for all possible inputs.

it depends on the diagonal function having a value for all possible inputs.  
The Termination Tester Gives Us Red Programs 
 
To remedy this, we would have to make use of a termination tester, if one existed. So let 
us deliberately introduce the shaky assumption that one exists, and let us use it as our 
fourth filter. We run down the list of Gr een Programs, eliminating one by one all 
nonterminators, so that in the end we are left with:

BlooP and FlooP and GlooP 435A complete pool of all call-less FlooP programs which calculate functions of 
exactly one input parameter, and which terminate for all values of their input.. 
 Let us call these special FlooP programs Red Programs (since they all must stop). Now, 
the diagonal argument will go through. We define  
Reddiag  [N] = 1 + Redprogram(#N} [N] 
 and in an exact parallel to Bluediag, we are forced to conclude that Reddiag  [N] is a well-
defined, calculable function of one variable  which is not in the catalogue of Red 
Programs, and is hence not even calculable in the powerful language FlooP. Perhaps it is 
time to move on to GlooP?  
GlooP ... 
 Yes, but what is GlooP? If FlooP is Bl ooP unchained, then GlooP must be FlooP 
unchained. But how can you take the chains  off twice% How do you make a language 
whose power transcends that of FlooP? In Reddiag , we have found a function whose 
values we humans know how to calculate-the  method of doing so has been explicitly

GlooP ... 
values we humans know how to calculate-the  method of doing so has been explicitly 
described in English-but which seemingly cannot be programmed in the language FlooP. 
This is a serious dilemma because no one has ever found any more powerful computer 
language than FlooP. 
Careful investigation into the power of computer languages has been carried out. 
We need not do it ourselves; let it just be repor ted that there is a vast class of computer 
languages all of which can be proven to have exactly the same expressive power  as FlooP 
does, in this sense: any calculation which can  be programmed in any one of the languages

does, in this sense: any calculation which can  be programmed in any one of the languages 
can be programmed in them all. The curious thing is that almost any sensible attempt at designing a computer language ends up by creati ng a member of this class-which is to 
say, a language of power equal to that of  FlooP. It takes some doing to invent a 
reasonably interesting computer language which is  weaker than those in this class. BlooP 
is, of course, an example of a weaker language , but it is the exception rather than the rule. 
The point is that there are some extremely natural ways to go about inventing algorithmic

The point is that there are some extremely natural ways to go about inventing algorithmic 
languages; and different people, following i ndependent routes, usually wind up creating 
equivalent languages, with the only difference being style, rather than power. 
 
... Is a Myth 
 In fact, it is widely believed that there cannot be any more powerful -language for describing calculations than languages that ar e equivalent to FlooP. This hypothesis was 
formulated in the 1930's by two people, inde pendently of each other: Alan Turing—about 
whom we shall say more later—and Alonzo Chur ch, one of the eminent logicians of this 
century. It

BlooP and FlooP and GlooP 436is called the Church-Turing Thesis . If we accept the CT-Thesis, we have to conclude that 
"GlooP" is a myth-there are no restrictions to remove in FlooP, no ways to increase its power by "unshackling" it, as we did BlooP. 
This puts us in the uncomfortable position of asserting that people  can calculate 
Reddiag  [N] for any value of N, but there is no way to program a computer  to do so. For, 
if it could be done at all, it could be done in FlooP-and by construction, it can't be done in 
FlooP. This conclusion is so peculiar that it s hould cause us to investigate very carefully 
the pillars on which it rests. And one of th em, you will recall, was our shaky assumption 
that there is a decision procedure which can  tell terminating from nonterminating FlooP 
programs. The idea of such a decision proce dure already seemed suspect, when we saw 
that its existence would allow all problems of  number theory to be solved in a uniform

that its existence would allow all problems of  number theory to be solved in a uniform 
way. Now we have double the reason for believing that any termination test is a myth-that there is no way to put FlooP program s in a centrifuge and separate out the 
terminators from the nonterminators. 
Skeptics might maintain that this is nothing like a rigorous proof that such a 
termination test doesn't exist. That is a valid objection; however, the Turing approach 
demonstrates more rigorously that no comput er program can be written in a language of 
the FlooP class which can perform a termination test on all FlooP programs.  
The Church-Turing Thesis

The Church-Turing Thesis 
 Let us come back briefly to the Church -Turing Thesis. We will talk about it-and 
variations on it-in considerable detail in Chap ter XVII; for now it will suffice to state it in 
a couple of versions, and postpone discussion of  its merits and meanings until then. Here, 
then, are three related ways to state the CT-Thesis:  
(1) What is human-computable is machine-computable.  (2) What is machine-computable is FlooP-computable.  (3) What is human-computable is FlooP-computable 
(i.e., general or partial recursive). 
 
Terminology: General and Partial Recursive

Terminology: General and Partial Recursive 
 We have made a rather broad survey, in th is Chapter, of some notions from number 
theory and their relations to the theory of computable functions. It is a very wide and 
flourishing field, an intriguing blend of com puter science and modern mathematics. We 
should not conclude this Chapter without introducing the standard terminology for the notions we have been dealing with.  As has already been mentioned, “BlooP-computable” is synonymous with “primitive recursive”. Now FlooP computable functions can be di-

BlooP and FlooP and GlooP 437vided into two realms: (1) those which are computable by terminating  FlooP programs: 
these are said to be general recursive ; and (2) those which are computable only by 
nonterminating  FlooP programs: these are said to be partial recursive. (Similarly for 
predicates.) People often just say "recursive" when they mean "general recursive".  
The Power of TNT 
 It is interesting that TNT  is so powerful that not onl y are all primitive recursive 
predicates represented, but moreover all genera l recursive predicates are represented. We 
shall not prove either of these facts, becau se such proofs would be superfluous to our 
aim, which is to show that TNT  is incomplete. If TNT  could not represent some 
primitive or general recursive predicates, then it would be incomplete in an uninteresting  
way-so we might as well assume that it can, and then show that it is incomplete in an interesting way.

Air on G's String 439Air on G's String 
 
The Tortoise and Achilles have just completed a tour of a porridge factory. 
 
Achilles: You don't mind if I change the subject, do you? Tortoise: Be my guest. Achilles: Very well, then. It concerns an obscene phone call I received a few days ago. Tortoise: Sounds interesting. Achilles: Yes. Well-the problem was that the caller was incoherent, at least as far as I 
could tell. He shouted something over the line and then hung up-or rather, now that I 
think of it, he shouted something, shouted it again, and then hung up.  
Tortoise: Did you catch what that thing was?  Achilles: Well, the whole call went like this:  
Myself : Hello? 
Caller (shouting wildly ): Yields falsehood when preced ed by its quotation! Yields 
falsehood when preceded by its quotation! 
(Click .)

(Click .) 
 
Tortoise: That is a most unusual thing to say to somebody on an obscene phone call. Achilles: Exactly how it struck me. Tortoise: Perhaps there was some meaning to that seeming madness.  Achilles: Perhaps.   
 (They enter a spacious courtyard framed by some charming three-story stone 
houses. At its center stands a palm tree, and to one side is a tower. Near the 
tower there is a staircase where a boy sits, talking to a young woman in a window.) 
 
Tortoise: Where are you taking me, Achilles?  Achilles: I would like to show you the pretty view from the top of this tower. Tortoise: Oh, how nice.

(They enter a spacious courtyard framed by some charming three-story stone 
 
Tortoise: Where are you taking me, Achilles?  Achilles: I would like to show you the pretty view from the top of this tower. Tortoise: Oh, how nice.  
(They approach the boy, who watches them w ith curiosity, then says something to 
the young woman-they both chuckle. Ach illes and Mr. T, instead of going up the 
boy's staircase, turn left and head down a short flight of stairs which leads to a small wooden door.) 
Achilles: We can just step inside right here. Follow me.

Air on G's String 440
 
FIGURE  74. Above and Below, by M.C. Escher (lithograph 1947 ).

Air on G's String 441(Achilles opens the door. They enter, and begi n climbing the steep helical staircase 
inside the tower.) 
 
Tortoise ( puffing slightly ): I'm a little out of shape for this sort of exercise, 
Achilles. How much further do we have to go? Achilles: Another few flights ... but I have an idea. Instead of walking on the top side of 
these stairs, why don't you walk on the underside?  
Tortoise: How do I do THAT? Achilles: Just hold on tightly, and clim b around underneath-there's room enough for you. 
You'll find that the steps make just as much sense from below as from above ... 
Tortoise ( gingerly shifting himself about ): Am I doing it right? 
Achilles: You've got it! Tortoise ( his voice slightly muffled ): Say-this little maneuver has got me confused. 
Should I head upstairs or downstairs, now? 
Achilles: Just continue heading in the same direction as you were before. On your side of 
the staircase, that means go DOWN , on mine it means UP.

the staircase, that means go DOWN , on mine it means UP. 
Tortoise: Now you're not going to tell me that I can get to the top of the tower by going 
down, are you? 
Achilles: I don't know, but it works ...  
(And so they begin spiraling in synchr ony, with A always on one side, and T 
matching him on the other side. Soon they reach the end of the staircase.) 
 
Now just undo the maneuver, Mr. T. Here-let me help you up.  
(He lends an arm to the Tortoise, and hoist s him back to the other side of the 
stairs.) 
 
Tortoise: Thanks. It was a little easier getting back up.

stairs.) 
 
Tortoise: Thanks. It was a little easier getting back up.   
(And they step out onto the roof, overlooking the town.) 
 
That's a lovely view, Achilles. I'm glad you brought me up here-or rather, DOWN  
here. 
Achilles: I figured you'd enjoy it. Tortoise: I've been thinking about that obscene phone call. I think I understand it a little 
better now. 
Achilles: You do? Would you tell me about it? Tortoise: Gladly. Do you perchance feel, as I do, that that phrase "preceded by its 
quotation" has a slightly haunting quality about it?

here. 
quotation" has a slightly haunting quality about it? 
Achilles: Slightly, yes-extremely slightly. Tortoise: Can you imagine something preceded by its quotation? Achilles: I guess I can conjure up an image of Chairman Mao walking into a banquet 
room in which there already hangs a large banner with some of his own writing on it. 
Here would be Chairman Mao, preceded by his quotation. 
Tortoise: A most imaginative example. But suppose we restrict the word

Air on G's String 442"preceded" to the idea of precedence on a prin ted sheet, rather than elaborate entries 
into a banquet room. 
Achilles: All right. But what exactly do you mean by "quotation" here? Tortoise: When 
you discuss a word or a phrase, you conventionally put it in quotes. For example, I can say, 
 
The word "philosopher" has five letters. 
 
Here, I put "philosopher" in quotes to s how that I am speaking about the WORD 
"philosopher" rather than about a philos opher in the flesh. This is called the USE-
MENTION distinction. 
Achilles: Oh? Tortoise: Let me explain. Suppose I were to say to you, 
 
Philosophers make lots of money. 
 
Here, I would be USING the word to manuf acture an image in your mind of a twinkle-
eyed sage with bulging moneybags. But when  I put this word-or any word-in quotes, I 
subtract out its meaning and connotations, a nd am left only with some marks on paper, 
or some sounds. That is called "MENTION" . Nothing about the word matters, other

or some sounds. That is called "MENTION" . Nothing about the word matters, other 
than its typographical aspects-any meaning it might have is ignored. 
Achilles: It reminds me of using a violin as a fly swatter. Or should I say mentioning"? 
Nothing about the violin matters, other than its solidity-any meaning or function it might have is being ignored. Come to think of it, I guess the fly is being treated that 
way, too. 
Tortoise: Those are sensible, if slightly  unorthodox, extensions of the use-mention 
distinction. But now, I want you to thi nk about preceding something by its own 
quotation.

way, too. 
quotation. 
Achilles: All right. Would this be correct? 
 
"HUBBA" HUBBA 
 
Tortoise: Good. Try another. Achilles: All right.  
"'PLOP' IS NOT THE TITLE OF ANY BOOK. SO FAR AS I KNOW"'  'PLOP' IS NOT THE TITLE OF ANY BOOK, SO FAR AS I KNOW. 
 
Tortoise: Now this example can be modified into quite an interesting specimen, simply 
by dropping `Plop'. Achilles: Really? Let me see what you mean. It becomes 
 
"IS NOT THE TITLE OF ANY BOOK, SO FAR AS I KNOW"   IS NOT THE TITLE OF ANY BOOK, SO FAR AS I KNOW. 
 
Tortoise: You see, you have made a sentence.

Tortoise: You see, you have made a sentence. 
Achilles: So I have. It is a sentence about th e pjrase “is not the toitle of any book, as far 
as I know”, and quite a silly one too. 
Tortoise: Why silly?

Air on G's String 443Achilles: Because it's so pointless. Here's another one for you:  
 
“WILL BE BOYS" WILL BE BOYS. 
 
Now what does that mean? Honestly, what a silly game. 
Tortoise: Not to my mind. It's very earnest st uff, in my opinion. In fact this operation of 
preceding some phrase by its quotation is so overwhelmingly important that I think I'll 
give it a name. 
Achilles: You will? What name will you dignify that silly operation by? Tortoise: I believe I'll call it "to quine a phrase", to quine a phrase. 
Achilles: "Quine"? What sort of word is that? 
Tortoise: A five-letter word, if I'm not in error. Achilles: What 1 was driving at is why you pick ed those exact five letters in that exact 
order. 
Tortoise: Oh, now I understand what you meant when you asked me "What sort of word 
is that?" The answer is that a philosophe r by the name of "Willard Van Orman Quine" 
invented the operation, so I name it in his honor. However, I cannot go any further

order. 
invented the operation, so I name it in his honor. However, I cannot go any further 
than this in my explanation. Why these partic ular five letters make up his name-not to 
mention why they occur in this particular order-is a question to which I have no ready 
answer. However, I'd be perfectly willing to go and 
Achilles: . Please don't bother! I didn't really  want to know everything about Quine's 
name. Anyway, now I know how to quine a phr ase. It's quite amusing. Here's a quined 
phrase: 
 
”IS A SENTENCE FRAGMENT" IS A SENTENCE FRAGMENT. 
 
It's silly but all the same I enjoy it.  You take a sentence fragment, quine

phrase: 
 
It's silly but all the same I enjoy it.  You take a sentence fragment, quine 
it, and lo and behold, you've made a sentence! A true sentence, in this case. 
 
Tortoise: How about quining the phrase "is a king with without no subject”? Achilles: A king without a subject would be---  Tortoise: -an anomaly, of course. Don't wande r from the point. Let's have quines first, 
and kings afterwards! 
Achilles: I'm to quine that phrase, am I? All right 
 
"IS A KING WITH NO SUBJECT" IS A KING WITH NO SUBJECT. 
 
It seems to me that it might make more sens e if it said "sentence" instead of "king". 
Oh, well. Give me another!

Oh, well. Give me another!  
Tortoise: All right just one more. Try this one: 
 
"WHEN QUINED, YIELDS A TORTOISE'S LOVE SONG" 
 
Achilles: That should be easy ... I'd say the quining gives this:  
 
"WHEN QUINED, YIELDS A TORTOISE'S LOVE SONG" WHEN QUINED, YIELDS A TORTOISE'S LOVE SONG  Hmmm… Th ere´s something just a little peculia r here. Oh, I see what it is! The 
sentence is talking about itself! Do you see that?

Air on G's String 444Tortoise: What do you mean? Sentences can't talk. 
Achilles: No, but they REFER  to things-and this one refers directly unambiguously-
unmistakably-to the very sentence which it is! You just have to think back and 
remember what quining is all about. 
Tortoise: I don't see it saying anything about itself. Where does it say "me", or: "this 
sentence", or the like? 
Achilles: Oh, you are being deliberately thick-s kulled. The beauty of it lies in just that: 
it talks about itself without having to come right out and say so! 
Tortoise: Well, as I'm such a simple fellow, could you just spell it all out for me, Achilles: Oh, he is such a Doubting Tortoi se ... All right, let me see ... Suppose I make 
up a sentence-I'll call it "Sentence P"-with a blank in it. 
Tortoise: Such as? Achilles: Such as ...    “________WHEN QUINED, YIELDS A TORTOISE'S LOVE SONG".

Tortoise: Such as? Achilles: Such as ...    “________WHEN QUINED, YIELDS A TORTOISE'S LOVE SONG".  
Now the subject matter of Sentence P depends on how you fill in the blank. But once you've chosen how to fill in the blank, then the subject matter is determined: it 
is the phrase which you get by QUINING the blank. Call that "Sentence Q", since it 
is produced by an act of quining. 
Tortoise: That makes sense. If the blank phr ase were "is written on old jars of mustard 
to keep them fresh", then Sentence Q would have to be

to keep them fresh", then Sentence Q would have to be 
 
"IS WRITTEN ON OLD JARS OF MUSTARD TO KEEP THEM FRESH" IS WRITTEN ON OLD JARS OF MUSTARD TO KEEP THEM FRESH. 
 
Achilles: True, and Sentence P makes the cl aim (though whether it is valid or not, I do 
not know) that Sentence Q is a Tortoise's l ove song. In any case, Sentence P here is 
not talking about itself, but rather about Sentence Q. Can we agree on that much? 
Tortoise: By all means, let us agree-and what a beautiful song it is, too. Achilles: But now I want to make a different choice for the blank, namely  
: "WHEN QUINED, YIELDS A TORTOISE'S LOVE SONG".

: "WHEN QUINED, YIELDS A TORTOISE'S LOVE SONG". 
 Tortoise: Oh, heavens, you're getting a little involved here. I hope this all isn't going to 
be too highbrow for my modest mind. 
Achilles: Oh, don't worry-you'll surely cat ch on. With this choice, Sentence Q 
becomes .. . 
 
"WHEN QUINED, YIELDS A TORTOISE'S LOVE-SONG"  WHEN QUINED, YIELDS A TORTOISE'S LOVE-SONG. 
 Tortoise: Oh, you wily old warrior you, I catch  on. Now Sentence Q is just the same as 
Sentence P. 
Achilles: And since Sentence Q is always th e topic of Sentence P, there is a loop now,

Tortoise: Oh, heavens, you're getting a little involved here. I hope this all isn't going to 
 
Achilles: And since Sentence Q is always th e topic of Sentence P, there is a loop now, 
P points back to itself. But you see, the self-reference is a

Air on G's String 445sort of accident. Usually Sentences Q and P are entirely unlike each other; but with 
the right choice for the blank in Senten ce-P, quining will do this magic trick for 
you. 
Tortoise: Oh, how clever. I wonder why I ne ver thought of that myself. Now tell me: 
is the following sentence self-referential? 
 
"IS COMPOSED OF FIVE WORDS" IS COMPOSED OF FIVE WORDS. 
 Achilles: Hmm ... I can't quite tell. The se ntence which you just gave is not really 
about itself, but rather about the phrase "is composed of five words". Though, of 
course, that phrase is PART of the sentence ... 
Tortoise: So the sentence refers to some part of itself-so what? Achilles: Well, 
wouldn't that qualify as self-reference, too? 
Tortoise: In my opinion, that is still a far cr y from true-self-reference. But don't worry 
too much about these tricky matters. You'll have ample time to think about them in the future. Achilles: I will?

too much about these tricky matters. You'll have ample time to think about them in the future. Achilles: I will? 
Tortoise: Indeed you will. But for now, why don't you try quining the phrase "yields 
falsehood when preceded by its quotation"? 
Achilles: I see what you're getting at-tha t old obscene phone call. Quining it produces 
the following: 
 
"YIELDS FALSEHOOD WHEN PRECEDED BY ITS QUOTATION" YIELDS FALSEHOOD WHEN PRECEDED BY ITS QUOTATION. 
 
So this is what that caller was saying! I just couldn't make out where the quotation 
marks were as he spoke. That certainly is an obscene remark! People ought to be 
jailed for saying things like that!

jailed for saying things like that! 
Tortoise: Why in the world? Achilles: It just makes me very uneasy. Un like the earlier examples, I can't quite make 
out if it is a truth or a falsehood. And the more I think about it, the more I can't 
unravel it. It makes my head spin. I wonder what kind of a lunatic mind would 
make something like that up, and torment innocent people in the night with it? 
Tortoise: I wonder ... Well, shall we go downstairs now? Achilles: We needn't go down-we're at gr ound level already. Let's go back inside-

you'll see. ( They go into the tower, and come to a small wooden door .) We can just 
step outside right here. Follow me. 
Tortoise: Are you sure? I don't want to fall three floors and break my shell. Achilles: Would I fool you? 
(And he opens the door. In front of them sits, to all appearances, the same boy, 
talking to the same young woman. Achilles and Mr. T walk up what seem to be the

talking to the same young woman. Achilles and Mr. T walk up what seem to be the 
same stairs they walked down to enter the tower, and find themselves in what looks like just the same courtyard they first came into.)   
Thank you, Mr. T, for your lucid clarification of that obscene telephone call. 
Tortoise: And thank you, Achilles, for the pl easant promenade. I hope we meet again 
soon.

On Formally Undecidable Propositions 446CHAPTER XIV 
 
On Formally Undecidable 
Propositions of TNT 
and Related Systems' 
 
The Two Ideas of the "Oyster" 
 THIS CHAPTER'S TITLE is an adaptation of  the title of Gödel’s famous 1931 paper-
"TNT " having been substituted for " Principia Mathematica ". Gödel’s paper was a 
technical one, concentrating on making his pr oof watertight and rigorous; this Chapter 
will be more intuitive, and in it I will stress the two key ideas which are at the core of the 
proof. The first key idea is the deep discovery that there are strings of TNT  which can be 
interpreted as speaking about other strings of TNT ; in short, that TNT , as a language, is 
capable of "introspection", or self-scruti ny. This is what comes from Gödel-numbering. 
The second key idea is that the property of self  scrutiny can be entirely concentrated into 
a single string; thus that string' s sole focus of attention is itself. This "focusing trick" is

a single string; thus that string' s sole focus of attention is itself. This "focusing trick" is 
traceable, in essence, to the Cantor diagonal method. 
In my opinion, if one is interested in understanding Gödel’s proof. in a deep way, 
then one must recognize that the proof, in its essence, consists of a fusion of these two 
main ideas. Each of them alone is a master stroke; to put them together took an act of 
genius. If I were to choose, however, whic h of the two key ideas is deeper, I would 
unhesitatingly pick the first one-the idea of Gödel-numbering, for that idea is related to

unhesitatingly pick the first one-the idea of Gödel-numbering, for that idea is related to 
the whole notion of what meaning and refere nce are, in symbol-manipulating systems. 
This is an idea which goes far beyond the conf ines of mathematical logic, whereas the 
Cantor trick, rich though it is in mathematical consequences, has little if any relation to issues in real life. 
 
The First Idea: Proof-Pairs 
 Without further ado, then, let us proceed to th e elaboration of the proof itself. We have 
already given a fairly careful notion of what  the Gödel isomorphism is about, in Chapter

already given a fairly careful notion of what  the Gödel isomorphism is about, in Chapter 
IX. We now shall describe a mathematical noti on which allows us to translate a statement 
such as "The string 0=0 is a theorem of TNT  into a statement of number theory. This will 
involve the notion of proof-pairs . A proof-pair is a pair of natural numbers related in a 
particular way. Here is the idea:

On Formally Undecidable Propositions 447Two natural numbers, m and n respectively, form a TNT  proof-pair if and only if m 
is the Gödel number of a TNT  derivation whose bottom line is the string with 
Gödel number n. 
 The analogous notion exists with respect to the MIU -system, and it is a little easier on the 
intuition to consider that case first. S o, for a moment, let us back off from TNT -proof-
pairs, and look at MIU -proof-pairs. Their definition is parallel: 
 
Two natural numbers, m and n respectively, form a MIU -proof pair if and only if  m 
is the Gödel number of a MIU -system derivation whose bottom line is the string 
with Gödel number n. 
 Let us see a couple of examples involving MIU -proof-pairs. First, let m = 
3131131111301, n = 301. These values of m and n do indeed form a MIU-proof-pair, 
because m is the Gödel number of the MIU -derivation 
 
MI 
MII 
MIIII 
MUI 
 
whose last line is MUI , having Gödel number 301, which is n. By contrast, let m =

MI 
MII 
MIIII 
MUI 
 
whose last line is MUI , having Gödel number 301, which is n. By contrast, let m = 
31311311130, and n = 30. Why do these two values not form a MIU -proof-pair? To see 
the answer, let us write out the alleged derivation which m codes for: 
 
MI 
MII  
MIII  
MU 
 There is an invalid step in this alleged deri vation! It is the step from the second to the 
third line: from MII to MIII . There is no rule of inference in the MIU -system which 
permits such a typographical step. Correspondingl y-and this is most crucial-there is no

MI 
MII  
MIII  
MU 
permits such a typographical step. Correspondingl y-and this is most crucial-there is no 
arithmetical rule of inference which carri es you from 311 to 3111. This is perhaps a 
trivial observation, in light of our discussion in  Chapter IX, yet it is at the heart of the 
Gödel isomorphism. What we do in any formal  system has its parallel in arithmetical 
manipulations. 
In any case, the values m = 31311311130, n = 30 certainly do not form a MIU -
proof-pair. This in itself doe s not imply that 30 is not a MIU -number. There could be 
another value of m which forms a MIU  proof-pair with 30. (Actually, we know by earlier

another value of m which forms a MIU  proof-pair with 30. (Actually, we know by earlier 
reasoning that MU is not a MIU -theorem, and therefore no number at all can form a 
MIU -proof-pair with 30.) 
Now what about TNT  proof pairs? Here are two parallel examples, one being 
merely an alleged TNT  proof-pair, the other being a valid TNT  proof-pair. Can you spot 
which is which? (Incidentally, here is where

On Formally Undecidable Propositions 448the `611' codon comes in. Its purpose is to sepa rate the Gödel numbers of successive lines 
in a TNT -derivation. In that sense, '611' se rves as a punctuation mark. In the MIU -
system, the initial '3' of all lines is sufficient-no extra punctuation is needed.)  
(1)  m = 626.262,636,223,123,262,111,666,611,223,123.666.111,666  
n = 123,666.111,666 
(2)  m=626,262.636,223.123,262,111,666,611223,333,262.636,123.262,111,666  
n = 223,333,262,636,123,262.111,666 
 
It is quite simple to tell which one is which, simply by translating back to the old notation, and making some routine examinations to see  
(1) whether the alleged derivation coded for by m is actually a legitimate derivation; 
(2) if so, whether the last line of the derivation coincides with the string which n codes 
for. 
 
Step 2 is trivial; and step 1 is also utterly straightforward, in this sense: there are no open-
ended searches involved, no hidden endle ss loops. Think of the examples above

for. 
 
ended searches involved, no hidden endle ss loops. Think of the examples above 
involving the MIU -system, and now just mentally substitute the rules of TNT  for the 
MIU -system's rules, and the axioms of TNT  for the MIU -system's one axiom. The 
algorithm in both cases is the same. Let me make that algorithm explicit:  
Go down the lines in the derivation one by one. Mark those which are axioms. For each line which is not an axiom, check whether it follows by any of the 
rules of inference from earlier lines in the alleged derivation.

rules of inference from earlier lines in the alleged derivation. 
If all nonaxioms follow by rules of inference from earlier lines, then you have a 
legitimate derivation; otherwise it is a phony derivation. 
 At each stage, there is a clear set of tasks to perform, and the num ber of them is quite 
easily determinable in advance.  
Proof-Pair-ness Is Primitive Recursive... 
 The reason I am stressing the boundedness of these loops is, as you may have 
sensed, that I am about to assert

sensed, that I am about to assert 
 
FUNDAMENTAL FACT 1: The property of being a proof-pair is a primitive recursive number-theoretical property, and can therefore be tested for by a BlooP 
program. 
  There is a notable contrast to be made here with that other closely related number-
theoretical property: that of being a theorem-number . To

On Formally Undecidable Propositions 449assert that n is a theorem-number is to assert that some  value of m exists which forms a 
proof-pair with n. (Incidentally, these comments apply equally well to TNT  and to the 
MIU -system; it may perhaps help to keep both in mind, the MIU -system serving as a 
prototype.) To check whether n is a theorem-number, you must embark on a search 
through all its potential proof-pair "partners" m-and here you may be getting into an 
endless chase. No one can say how far you will have to look to find a number which 
forms a proof-pair with n as its second element. That is the whole problem of having 
lengthening and shortening rules in the same system: they lead to a certain degree of 
unpredictability. 
The example of the Goldbach Variation ma y prove helpful at this point. It is 
trivial to test whether a pair of numbers ( m,n) form a Tortoise pair : that is to say, both m 
and n + m  should be prime. The test is easy because the property of primeness is

and n + m  should be prime. The test is easy because the property of primeness is 
primitive recursive: it admits of a predictabl y terminating test. But if we want to know 
whether n possesses the Tortoise property, then we are asking, "Does any number m form 
a Tortoise-pair with n as its second element?"-and this, once again, leads us out into the 
wild, MU-loopy unknown. 
 
... And Is Therefore Represented in TNT 
 The key concept at this juncture, then, is F undamental Fact 1 given above, for from it we 
can conclude 
 
FUNDAMENTAL FACT 2: The property of fo rming a proof-pair is testable in

FUNDAMENTAL FACT 2: The property of fo rming a proof-pair is testable in 
BlooP, and consequently, it is represented  in TNT  by some formula having two 
free variables. 
 
Once again, we are being casual about speci fying which system these proof-pairs 
are relative to; it really doesn't matter, fo r both Fundamental Facts hold for any formal 
system. That is the nature of formal systems: it is always possible to tell, in a predictably 
terminating way, whether a given sequence of lines forms a proof, or not-and this carries 
over to the corresponding arithmetical notions.  
The Power of Proof-Pairs

The Power of Proof-Pairs 
 
Suppose we assume we are dealing with the MIU -system, for the sake of concreteness. 
You probably recall the string we called " MUMON ", whose interpretation on one level 
was the statement " MU is a theorem of the MIU- system". We can show how MUMON  
would be expressed in TNT , in terms of the formula wh ich represents the notion of MIU -
proof-pairs. Let us abbreviate that formul a, whose existence we are assured of by 
Fundamental Fact 2, this way:  
MIU-PROOF-PAIR { α,α´}

On Formally Undecidable Propositions 450Since it is a property of two numbers, it is  represented by a formula with two free 
variables. (Note: In this Chap ter we shall always use austere TNT -so be careful to 
distinguish between the variables a,  a', a".) In order to assert " MU is a theorem of the 
MIU -system", we would have to make the isomorphic statement "30 is a theorem-
number of the MIU -system", and then translate that into TNT -notation. With the aid of 
our abbreviation, this is easy (remember also  from Chapter VIII that to indicate the 
replacement of every a' by a numeral, we write that numeral followed by "/a' 1):  
Зa:MIU-PROOF- PAIR ja,SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSO/a'} 
 
Count the  S's: there are 30. Note that this is a closed sentence of TNT , because one free 
variable was quantified, the other replaced by a numeral. A clever thing has been done 
here, by the way. Fundamental Fact 2 gave us a way to talk about proof-pairs; we have

here, by the way. Fundamental Fact 2 gave us a way to talk about proof-pairs; we have 
figured out how to talk about theorem-num bers, as well: you just add an existential 
quantifier in front! A more literal translation of the string above would be, "There exists 
some number  a that forms a MlIJ -proof-pair with 30 as its second element". 
Suppose that we wanted to do some thing parallel with respect to TNT -say, to 
express the statement "0=0 is a theorem of TNT ". We may abbreviate the formula which 
Fundamental Fact 2 assures us exists, in an analogous way (with two free variables, 
again):

again):  
TNT- PROOF- PAIR{a,a'} 
 
(The interpretation of this abbreviated TNT -formula is: "Natural numbers a and a' form a 
TNT-proof-pair.") The next step is to tr ansform our statement into number theory, 
following the MUMON -model above. The statement becomes "There exists some 
number a which forms a TNT  proof-pair with 666,111,666 as its second element". The 
TNT -formula which expresses this is: 
 
Зa:TNT-PROOF-PAI R{a,SSSSS SSSSSO/a' } 
many, many 5's! 
(in fact, 666,111,666 of them) 
 -a closed sentence of TNT . (Let us call it " JOSHtU ", for reasons to appear

 -a closed sentence of TNT . (Let us call it " JOSHtU ", for reasons to appear 
momentarily.) So you see that there is a way to talk not only about the primitive recursive 
notion of TNT -proof-pairs, but also about the related but trickier notion of TNT -
theorem-numbers. 
To check your comprehension of these id eas, figure out how to translate into TNT  
the following statements of meta- TNT :. 
 
(1) 0=0 is not a theorem of TNT . 
(2) ~0=0 is a theorem of TNT . 
(3) ~0=0 is not a theorem of TNT .

On Formally Undecidable Propositions 451How do the solutions differ from the example done above, and from each other' Here are 
a few more translation exercises. 
 
(4) JOSHU is a theorem of TNT . (Call the TNT -string which expresses this 
":METAJOSH t'".) (5) META-JOSH[. is a theorem of TNT. (C all the TNT-string which expresses this 
"META-META-JOSHC".)  (6)META-META-JOSHU is a theorem of TNT (7)META-META- ME IA -JOSHU is a theorem of TNT  
 
(etc., etc.) 
 Example 5 shows that statements of meta-met a-TNT can be translated into TNT-notation; 
example 6 does the same for meta-meta-meta-TNT, etc. 
It is important to keep in mind the di fference between expressing a property, and 
representing  it, at this point. The property of being a TNT theorem-number, for instance, 
is expressed  by the formula 
 
Зa:TNT- PROOF- PAI R{a,a' } 
 
Translation: "a' is a TNT -theorem-number". However, we have no guarantee that this 
formula represents  the notion, for we have no guarantee that this property is primitive

formula represents  the notion, for we have no guarantee that this property is primitive 
recursive-in fact, we have more than a sneaki ng suspicion that it isn't. (This suspicion is 
well warranted. The property of being a TNT- theorem-number is not primitive recursive, 
and no TNT-formula can represent the propert y!) By contrast, the property of being a 
proof-pair, by virtue of its primitive recursivity, is both expressible and representable, by 
the formula already introduced. 
 
Substitution Leads to the Second Idea 
 The preceding discussion got us to  the point where we saw how TNT  can "introspect" on

 The preceding discussion got us to  the point where we saw how TNT  can "introspect" on 
the notion of TNT -theoremhood. This is the essence of the first part of the proof. We 
now wish to press on to the second major id ea of the proof, by developing a notion which 
allows the concentration of this introspection into a single formula. To do this, we need to 
look at what happens to the Gödel number of a formula when you modify the formula structurally in a simple way. In fact, we shall consider this specific modification:  
replacement of all free variables by a specific numeral

replacement of all free variables by a specific numeral 
 . Below are shown a couple of examples of this operation in the left hand column, and in the right hand column are exhibited the parallel changes in Gödel numbers.

On Formally Undecidable Propositions 452Formula   
   
a=a       
We now replace all  free variables by  the numeral for 2:  SSO=SSO  
 
3a:3a':a"=(SSa•SSa')    We now replace all  free variables by  the numeral for 4: 
 
---3a:3a':SSSSO=(SSa•SSa')  Gödel number 
 
262,1 11,262 
 
 123,123,666,111.123,123,666  223,333,262,636,333,262,163,636, 262,163,163,111,362,123,123,262, 236,123,123,262,163,323 
 
 223,333,262,636,333,262,163,636, 123,123,123,123,666,111,362,123, 123,262,236,123,123,262,163,323  
An isomorphic arithmetical process is going on in the right-hand column, in 
which one huge number is turned into an even huger number. The function which makes 
the new number from the old one  would not be too difficult to describe arithmetically, in 
terms of additions, multiplications, powers of  10 and so on-but we need not do so. The 
main point is this: that the relation among (1 ) the original Gödel number, (2) the number

main point is this: that the relation among (1 ) the original Gödel number, (2) the number 
whose numeral is inserted, and (3) the resu lting Gödel number, is a primitive recursive 
relation. That is to say, a BlooP test could be written which, when fed as input any three 
natural numbers, says YES  if they are related in this way, and NO if they aren't. You may 
test yourself on your ability to perform such a test-and at the same time convince yourself 
that there are no hidden open-ended loops to  the process-by checking the following two 
sets of three numbers: 
 
(1) 362,262,112,262,163,323,111,123,123,123,123,666;   2:

(1) 362,262,112,262,163,323,111,123,123,123,123,666;   2: 
362,123,123,666,112,123,123,666,323,111,123,123,123,123,666.  
 
(2) 223,362,123,666,236,123,666,323,111,262,163.  
1 223,362,262,236,262,323,111,262,163; 
 
As usual, one of the examples checks, the other does not. Now this relationship between 
three numbers will be called the substitu tion relationship. Because it is primitive 
recursive, it is represented by some formula of TNT  having three free variables. Lets us 
abbreviate that TNT – formula by the following notation 
SUB (a,a´,a´´)

On Formally Undecidable Propositions 439Because this formula represents the substitution relationship, the formula shown 
below must be a TNT -theorem: 
 
SU B{SSSSS SSSSSO/a,SSO/a',SSSSSS SSSSO/a"} 
262,111,262 S's 123,123,666,111,123,123,666 S's 
 
(This is based on the first example of th e substitution relation shown in the parallel 
columns earlier in this sec tion.) And again because the SUB  formula represents the 
substitution relation, the formula shown below certainly is not a TNT -theorem: 
 
SU B{SSSO/a,SSO/a',S0/a"} 
 
Arithmoquining 
 
We now have reached the crucial point wher e we can combine all of our disassembled 
parts into one meaningful whole. We  want to use the machinery of the TNT-PROOF -
PAIR  and SUB formulas in some way to c onstruct a single sentence of TNT whose 
interpretation is: "This very string of TNT  is not a TNT -theorem." How do we do it% 
Even at this point, with all the necessary mach inery in front of us, the answer is not easy 
to find.

to find. 
A curious and perhaps frivolous-seeming notion is that of substituting a formula's 
own Gödel number into itself. This is quite parallel to that other curious, and perhaps 
frivolous-seeming, notion of "quining" in the Air on G's String . Yet quining turned out to 
have a funny kind of importance, in that it s howed a new way of making a self-referential 
sentence. Self reference of the Quine vari ety sneaks up on you from behind the first time

to find. 
sentence. Self reference of the Quine vari ety sneaks up on you from behind the first time 
you see it-but once you understand the principle, you appreciate that it is quite simple and 
lovely. The arithmetical version of quining-let's call it arithmoquining -will allow us to 
make a TNT -sentence which is "about itself ". 
Let us see an example of arithmoquining. We need a formula with at least one 
free variable. The following one will do:  
a=SO 
 
This formula's Gödel number is 262,111,123,666, and we will stick this number into the 
formula itself-or rather, we will stick its numeral in. Here is the result:  
SSSSS  SSSSSO=SO

a=SO 
 
SSSSS  SSSSSO=SO 
262,111,123,666 S's 
 
This new formula a asserts a silly fals ity-that 262.111.123.666 equals 1: If we had begun 
with the string ~a=S0 and then arthmoquine d, we would have cone up with a true 
statement—as you can see for yourself.  When you arithmoquine, you are of course performing a special case

On Formally Undecidable Propositions 440of the substitution operation we defined ear lier. If we wanted to speak about 
arithmoquining inside TNT , we would use the formula 
 
SUB{a" a" a'} 
 
where the first two variables are the same. This  comes from the fact that we are using a 
single number in two different ways (shade s of the Cantor diagonal method!). The 
number a" is both (1) the original Gödel number, and (2) the insertion-number. Let us invent an abbreviation for the above formula:  
ARITHMOQUINE{a", a'} 
 
What the above formula says, in English, is:  
a' is the Gödel number of the formula gotten by arithmoquining the formula with 
Gödel number a". 
Now the preceding sentence is long and ugly. Let's introduce a concise and elegant term to summarize it. We'll say  
a' is the arithmoquinification of a" 
 to mean the same thing. For instance, the arithmoquinification of 262,111,123,666 is this 
unutterably gigantic number:  
123,123,123  123,123,123,666,111,123,666 
262,111,123,666 copies of '1231

262,111,123,666 copies of '1231 
 
(This is just the Gödel number of the formula we got when we arithmoquined a=SO .) We 
can speak quite easily about arithmoquining inside TNT . 
 
The Last Straw 
 Now if you look back in the Air on G's String , you will see that the ultimate trick 
necessary for achieving self-reference in Quin e's way is to quine a sentence which itself 
talks about the concept of quining. It's not enough just to quine-you must quine a quine-

mentioning sentence! All right, then the paralle l trick in our case must be to arithmoquine 
some formula which itself is talking about the notion of arithmoquining! 
Without further ado, we'll now write that formula down, and call it G's uncle : 
 
-3a:3a':<TNT-PROOF-PAIR{a,a'}A.ARITHMOQUINE{a",a'}> 
 
You can see explicitly how arithmoquinification is thickly involved in the plot, Now this “uncle” has a Gödel number, of course, which we’ll  call `u´

On Formally Undecidable Propositions 441The head and tail of u's decimal expansion, a nd even a teeny bit of its midsection, can be 
read off directly: 
 
u = 223,333,262,636,333,262,163,636,212, ... ,161, ... ,213 
 
For the rest, we'd have to know just how the formulas TNT-PROOF-PAIR  and 
ARITHMOQUINE  actually look when written out. That  is too complex, and it is quite 
beside the point, in any case. 
Now all we need to do is-arithmoquine this very uncle! What this entails is 
"booting out" all free variables-of which there is only one, namely a"-and putting in the numeral for u everywhere. This gives us: 
 
-3a:3a':<TNT-PROOF-PAIR{a,a'}^,ARITHMOQUINE{SSS ... SSSO/a",a'}> 
 u S's 
 
And this, believe it or not, is G ödel’s string, which we can call ' G'. Now there are two 
questions we must answer without delay. They are 
 
(1) What Is G's Gödel number? 
(2) What is the interpretation of G? 
 
Question 1 first. How did we make G? Well, we began with the uncle, and arithmoquined

 u S's 
 
Question 1 first. How did we make G? Well, we began with the uncle, and arithmoquined 
it. So, by the definition of arithmoquinification, G's Gödel number is  
 
the arithmoquinification of u. 
 
Now question 2. We will translate G into English in stages, getting gradually more 
comprehensible as we go along. For our first rough try, we make a pretty literal translation: 
 
"There do not exist numbers a and a' such that both (1) they form a TNT -proof-
pair. and (2)  a' is the arithmoquinification of u."

pair. and (2)  a' is the arithmoquinification of u." 
 
Now certainly there is a number a' which is the arithmoquinification of u-so the problem 
must lie with the other number, a. This obser vation allows us to rephrase the translation 
of G as follows: 
 
"There is no number a that forms a TNT -proof-pair with the arithmoquinification 
of u." 
 
(This step, which can be confusing, is explai ned below in more detail.) Do you see what 
is happening? G is saying this: 
 
"The formula whose Gödel number is the arithmoquinification 
of u is not a theorem of TNT ."

of u." 
 
of u is not a theorem of TNT ." 
 
But-and this should come as no surprise  by now-that formula is none other than G itself; 
whence we can make the ultimate translation of G; as 
 
“G is not a theorem of TNT .”

On Formally Undecidable Propositions 442-or if you prefer, 
 
"I am not a theorem of TNT ." 
 We have gradually pulled a high-level interpretation-a sentence of meta- TNT -out of what 
was originally a low-level interpretation-a sentence of number theory. -  
TNT Says "Uncle!" 
 
The main consequence of this amazing cons truction has already been delineated in 
Chapter IX: it is the incompleteness of TNT . To reiterate the argument: 
 
Is G a TNT -theorem? If so, then it must assert a truth. But what in fact does  G 
assert? Its own nontheoremhood. Thus fr om its theoremhood would follow its 
nontheoremhood: a contradiction. 
Now what about G being a nontheorem? This is acceptable, in that it doesn't 
lead to a contradiction. Bu t G's nontheoremhood is what G asserts-hence G asserts 
a truth. And since G is not a theorem, there exists (at least) one truth which is not a 
theorem of TNT . 
 
Now to explain that one tricky step again. I will use another similar example. Take this 
string:

string: 
 
--3a:3a': <TORTOISE-PAIR{a,a' }.TENTH-POWER{SSO/a ", a' J > 
 
where the two abbreviations are for strings of TNT  which you can write down yourself. 
TENTH-POWER {a",a'} represents the statemen t "a' is the tenth power of a"". The 
literal translation into English is then:  
"There do not exist numbers a and a' such that both (1) they form a Tortoise-pair, and (2) a' is the tenth power of 2." 
 But clearly, there is a tenth power of 2-name ly 1024. Therefore, what the string is really 
saying is that

saying is that  
"There is no number a that forms a Tortoise-pair with 1024"  
 which can be further boiled down to:   
"1024 does not have the Tortoise property." 
 The point is that we have achieved a way of substituting a description  of a number, rather 
than its numeral, into a predicate. It depe nds on using one °extra quantified variable ( a'). 
Here, it was the number 1024 that was described as the “tenth power of 2”; above it was 
the number described as the arithmoquinification of a”.

On Formally Undecidable Propositions 443"Yields Nontheoremhood When Arithmoquined" 
 
Let us pause for breath for a moment, and re view what has been done. The best way I 
know to give some perspective is to set out explicitly how it compares  with the version of 
the Epimenides paradox due to Quine. Here is a map:  
Falsehood 
 <==> nontheoremhood 
quotation of a phrase 
 <==>  
preceding a predicate 
by a subject 
 <==> definite term) into an open formula 
by a quoted phrase 
 <==> substituting the Gödel number of a 
string into an open formula 
by itself, in quotes 
("quining") 
 <==> substituting the Gödel number of an 
open formula into the formula itself 
("arithmoquining") 
yields falsehood when quined 
(a predicate without a subject) 
 <==> "uncle" of G” 
 the(an open formula of TNT  
"yields falsehood when quined" 
(the above predicate. quoted) 
 <==> the number a (the Gödel number 
 of the above open formula) 
"yields falsehood when quined"

<==> substituting the Gödel number of a 
(complete sentence formed by 
quining the above predicate) <==> G itself 
(sentence of TNT formed 
bysubstituting a into the uncle,i.e., 
arithmoquining the uncle) 
 
Gödel’s Second Theorem 
 
Since G's interpretation is true, the interpretation of its negation -- G is false. And we 
know that no false statements are derivable in TNT . Thus neither G nor its negation - G 
can be a theorem of T NT. We have found a "hole" in our system-an undecidable

can be a theorem of T NT. We have found a "hole" in our system-an undecidable 
proposition. This has a number of ramifications. Here is one curious fact which follows 
from G's undecidability: although neither G nor - G is a theorem, the formula < GV -G > is 
a theorem, since the rules of the Propositional Calculus ensure that all well-formed formulas of the form <Pv--P > are theorems. 
This is one simple example where an asserti on inside the system and an assertion about 
the system seem at odds with each other. It makes one wonder if the system really

the system seem at odds with each other. It makes one wonder if the system really 
reflects itself accurately. Does the "reflected metamathematics" which exists inside TNT  
correspond well to the metamathematics which we do? This was one of the questions which intrigued Gödel when he wrote his pape r. In particular, he was interested in 
whether it was possible, in the “reflected me tamathematics”, to prove TNT’s consistency. 
Recall that this was a great philosophical dilemma of

On Formally Undecidable Propositions 444the day: how to prove a system consistent. Gödel found a simple way to express the 
statement " TNT  is consistent" in a TNT  formula; and then he showed that this formula 
(and all others which express the same idea) are only theorems of TNT  under one 
condition: that TNT  is inconsistent. This perverse re sult was a severe blow to optimists 
who expected that one could find a rigorous pr oof that mathematics is contradiction-free. 
How do you express the statement " TNT  is consistent" inside TNT  It hinges on 
this simple fact: that inconsistency means th at two formulas, x and x, one the negation of 
the other, are both theorems. But if both x and -- x are theorems, then according to the Propositional Calculus, all well-formed formulas are theorems. Thus, to show TNT's  
consistency, it would suffice to e xhibit one single sentence of TNT  which can be proven 
to be a nontheorem. Therefor e, one way to express " TNT  is consistent" is to say "The

to be a nontheorem. Therefor e, one way to express " TNT  is consistent" is to say "The 
formula -0=0 is not a theorem of TNT ". This was already proposed  as an exercise a few 
pages back. The translation is: 
 
---3a:TNT-PROOF- PAIR{a,SSSSS   SSSSSOIa'}  
223,666,111,666 S's 
 It can be shown, by lengthy but fairly st raightforward reasoning, that-as long as TNT  is 
consistent-this oath-of-consistency by TNT  is not a theorem of TNT . So TNT's  powers 
of introspection are great when it comes to expressing things, but fairly weak when it 
comes to proving them. This is quite a provoca tive result, if one applies it metaphorically

comes to proving them. This is quite a provoca tive result, if one applies it metaphorically 
to the human problem of self-knowledge.  
TNT Is ω-Incomplete 
 
Now what variety of incompleteness does TNT  "enjoy? We shall see that TNT' s 
incompleteness is of the "omega" variety-defi ned in Chapter VIII. This means that there 
is some infinite pyramidal family of st rings all of which are theorems, but whose 
associated "summarizing string" is a nontheor em. It is easy to exhibit the summarizing 
string which is a nontheorem:  
u S's 
Va: 3a':<TNT-PROOF- PAIR{a,a'}nA RITHMOQUINE{SSS ... SSSO/a",a'}>

u S's 
Va: 3a':<TNT-PROOF- PAIR{a,a'}nA RITHMOQUINE{SSS ... SSSO/a",a'}> 
 To understand why this string is a nontheorem , notice that it is extremely similar to G 
itself-in fact, G can be made from it in one step (viz., according to TNT 's Rule of 
Interchange). Therefore, if it were a theorem, so would G be. But since G isn't a theorem, 
neither can this be.  Now we want to show that all of the strings in the related pyramidal family are 
theorems. We can write them own easily enough:

On Formally Undecidable Propositions 445u S's 
--3a': <TNT-PROOF- PAIR,O/a,a'}  ARITHMOQUINE;SSS ... SSSO/a" a'I> 
-3a': <TNT-PROOF- PAIR) SO/a,a'}  ARITHMOQUINEISSS ... SSSO/a", a'}> 
-3a': <TNT-PROOF-PAIR{SSO/a,a'} ARITHMOQUINEI5SS ... SSSO/a", a'}> 
-3a': <TNT-PROOF- PAIR}SSSO/a,a'}  ARITHMOQUINE{SSS ... SSSO/a", a'}> 
 What does each one assert? Their translations, one by one, are:  
"0 and the arithmoquinification of u do not form a TNT-proof-pair." "1 and the arithmoquinification of u do not form a TNT-proof-pair." 
"2 and the arithmoquinification of u do not form a TNT-proof-pail." 
"3 and the arithmoquinification of u do not form a TNT-proof-pair." 
 Now each of these assertions is about whethe r two specific integers form a proof-pair or 
not. (By contrast, G itself is about whether one specific integer is a theorem-number or 
not.) Now because G is a nontheorem, no intege r forms a proof-pair with G's Gödel 
number. Therefore, each of the statements of  the family is true. Now the crux of the

number. Therefore, each of the statements of  the family is true. Now the crux of the 
matter is that the property of being a proof-p air is primitive recursive, hence represented, 
so that each of the statements in the list above, being true, must translate into a theorem of TNT-which means that everything in our in finite pyramidal family is a theorem. And 
that shows why TNT  is w-incomplete. 
 
Two Different Ways to Plug Up the Hole 
 
Since G's interpretation is true, the interpretation of its negation -G is false. And, using

Since G's interpretation is true, the interpretation of its negation -G is false. And, using 
the assumption that TNT  is consistent, we know that no false statements are derivable in 
TNT . Thus neither G nor its negation - G is a theorem of TNT . We have found a hole in 
our system-an undecidable proposition. Now this need be no source of alarm, if we are philosophically detached enough to recognize what this is a symptom of. It signifies that 
TNT  can be extended, just as absolute geometry could be. In fact, TNT  can be extended 
in two distinct directions, just as absolute  geometry could be. It can be extended in a

in two distinct directions, just as absolute  geometry could be. It can be extended in a 
standard direction-which corresponds to exte nding absolute geometry in the Euclidean 
direction; or, it can be extended in a nonsta ndard direction-which corresponds, of course, 
to extending absolute geometry in the non-Euc lidean direction. Now the standard type of 
extension would involve  
Adding G as a new axiom.

On Formally Undecidable Propositions 446This suggestion seems rather innocuous and pe rhaps even desirable, since, after all, G 
asserts something true about the natural num ber system. But what about the nonstandard 
type of extensions If it is at all parallel to the case of the parallel postulate, it must 
involve   
adding the negation of G as a new axiom. 
 But how can we even contemplate doing such a repugnant, hideous thing? After all, to 
paraphrase the memorable words of Girolamo Saccheri, isn't what -- G says "repugnant to 
the nature of the natural numbers'?  
Supernatural Numbers 
 
I hope the irony of this quotation strikes you. Th e exact problem with Saccheri's approach 
to geometry was that he began with a fixe d notion of what was tr ue and what was not 
true, and he set out only to prove what he'd assessed as true to start with. Despite the 
cleverness of his approach-which involved denying the fifth postulate, and then proving

cleverness of his approach-which involved denying the fifth postulate, and then proving 
many "repugnant" propositions of the ensuing geometry-Saccheri never entertained the possibility of other ways of thinking about points and lines. Now we should be wary of 
repeating this famous mistake. We must cons ider impartially, to the extent that we can, 
what it would mean to add -G as an axiom to TNT . Just think what mathematics would 
be like today if people had never considered adding new axioms of the following sorts:  
3a:(a+a)=S0  
3a:Sa=O  3a:(a•a)=SSO  
3a:S(a•a) =0

3a:S(a•a) =0 
 
While each of them is "repugnant to the na ture of previously known number systems", 
each of them also provides a deep and w onderful extension of the notion of whole 
numbers: rational numbers, negative numbers, irrational numbers, imaginary numbers. Such a possibility is what -G is trying to get us to open our eyes to. Now in the past, each 
new extension of the notion of number was gr eeted with hoots and catcalls. You can hear 
this particularly loudly in the names att ached to the unwelcome arrivals, such as

this particularly loudly in the names att ached to the unwelcome arrivals, such as 
"irrational numbers", "imaginary numbers". Tr ue to this tradition, we shall name the 
numbers which -'- G is announcing to us the supernatural numbers, showing how we feel 
they violate all reasonable and commonsensical notions. 
If we are going to throw - G in as the sixth axiom of TNT , we had better 
understand how in the world it could coexist, in  one system, with the infinite pyramidal 
family we just finished discussing. To put it bluntly, - G says: 
 
“There exists some number which forms a TNT -proof-pair with the 
arithmoquinification of u”

On Formally Undecidable Propositions 447-but the various members of the pyramidal family successively assert: 
 
"0 is not that number" "1 is not that number" "2 is not that number" 
 This is rather confusing, because it seems to be a complete contradiction (which is why it 
is called " ω-inconsistency"). At the root of our confusion-much as in the case of the 
splitting of geometry-is our stubborn resistance to  adopt a modified interpretation for the 
symbols, despite the fact that we are quite aware that the system is a modified system. 
We want to get away without reinterpreting any symbols-and of course that will prove 
The reconciliation comes when we reinterpret 3 as "There exists a generalized  
natural number", rather than as "There exists  a natural number". As we do this, we shall 
also reinterpret V in the corresponding way. Th is means that we are opening the door to 
some extra numbers besides the natural numbers. These are the supernatural  numbers .

some extra numbers besides the natural numbers. These are the supernatural  numbers . 
The naturals and supernaturals together make up the totality of generalized naturals . 
The apparent contradiction vanishes into thin air, now, for the pyramidal family 
still says what it said before: "No natural  number forms a TNT -proof-pair with the 
arithmoquinification of u." The family doesn't say anyt hing about supernatural numbers, 
because there are no numerals for them. But now, - G says, "There exists a generalized  
natural number which forms a TNT -proof-pair with the arithm oquinification of u." It is

natural number which forms a TNT -proof-pair with the arithm oquinification of u." It is 
clear that taken together, the family and -G tell us something: that there is a supernatural  
number which forms a TNT -proof-pair with the arithmoquinification of u. That is all-
there is no contradiction any more. TNT +-G is a consistent system, under an 
interpretation which includes supernatural numbers. 
Since we have now agreed to extend the in terpretations of the two quantifiers, this 
means that any theorem which involves eith er of them has an extended meaning. For 
example, the commutativity theorem 
 
Va:da':(a+a')=(a'+a)

Va:da':(a+a')=(a'+a) 
 
now tells us that addition is commutative fo r all generalized natural numbers-in other 
words, not only for natural numbers, but also  for supernatural numbers. Likewise, the 
TNT -theorem which says "2 is not the square of a natural number"-- 
 
-3a:(a • a)=SSO 
 
--now tells us that 2 is not the square of  a supernatural number, either. In fact, 
supernatural numbers share all the properties of natural numbers, as

On Formally Undecidable Propositions 448long as those properties are given to us in theorems of TNT . In other words, everything 
that can be formally proven  about natural numbers is thereby established also for 
supernatural numbers. This means, in par ticular, that supernatural numbers are not 
anything already familiar to you, such as fr actions, or negative numbers, or complex 
numbers, or whatever. The supernatural numbers are, instead, best visualized as integers 
which are greater than all natural numbers-as infinitely large  integers. Here is the point: 
although theorems of TNT  can rule out negative numbers, fractions, irrational numbers, 
and complex numbers, still there is no way to rule out infinitely large integers. The problem is, there is no way even to expr ess the statement "There are no infinite 
quantities". 
This sounds quite strange, at first. Just exactly how big is the number which 
makes a TNT -proof-pair with G's Gödel number= (Let's call it 'I . for no particular

makes a TNT -proof-pair with G's Gödel number= (Let's call it 'I . for no particular 
reason.) Unfortunately, we have not got a ny good vocabulary for describing the sizes of 
infinitely large integers, so I am afraid I cannot convey a sense of I's magnitude. But then just how big is i (the square root of -1)? It s size cannot be imagined in terms of the sizes 
of familiar natural numbers. You can't say, "Well,  i is about half as big as 14, and 9/10 as 
big as 24." You have to say, "i squared is -1 ", and more or less leave it at that. A quote 
from Abraham Lincoln seems a propos here . When he was asked, "How long should a

from Abraham Lincoln seems a propos here . When he was asked, "How long should a 
man's legs be?" he drawled, "Long enough to r each the ground." That is more or less how 
to answer the question about the size of I-it should be just the size of a number which 
specifies the structure of a proof of G-no bigger, no smaller. 
Of course, any theorem of TNT  has many different derivations, so you might 
complain that my characterization of I is nonunique. That is so. But the parallel with 1-

the square root of -1-still holds. Namely, recall that there is another number whose square is also minus one: -i. Now i and -i are not th e same number. They just have a property in 
common. The only trouble is that it is the pr operty which defines them! We have to 
choose one of them-it doesn't matter which one-and call it " i". In fact there's no way of 
telling them apart. So for all we know we could have been calling the wrong one "i" for all these centuries and it would have made  no difference. Now, like i, I is also

telling them apart. So for all we know we could have been calling the wrong one "i" for all these centuries and it would have made  no difference. Now, like i, I is also 
nonuniquely defined. So you just have to think of I as being some specific one of the 
many possible supernatural numbers which form TNT -proof-pairs with the 
arithmoquinification of u. 
 
Supernatural Theorems Have Infinitely Long Derivations. 
 We haven't yet faced head on what it means to throw - G in as an axiom. We have said it 
but not stressed it. The point is that -G asserts that G has a proof. How can a system 
survive, when one of its axioms asserts that  its own negation has a proof? We must be in 
hot water now! Well, it is not so bad as you might think. As long as we only construct

hot water now! Well, it is not so bad as you might think. As long as we only construct 
finite  proofs, we will never prove G Therefore, no calamitous collision between G and its 
negative ~ G will ever take place. The supernatural number –I won’t cause any disaster.

On Formally Undecidable Propositions 449However, we will have to get used to the idea that --- G is now the one which asserts a 
truth (" G has a proof "), while G asserts a falsity (" G has no proof"). In standard number 
theory it is the other way around-but then, in  standard number theory there aren't any 
supernatural numbers. Notice that a supernatural theorem of TNT -namely G-may assert a 
falsity, but all natural theorems still assert truths.  
Supernatural Addition and Multiplication 
 
There is one extremely curious and unexpected  fact about supernaturals which I would 
like to tell you, without proof. (I don't know the proof  either.) This fact is reminiscent of 
the Heisenberg uncertainty principle in quantum mechanics. It turns out that you can "index" the supernaturals in a simple and natural way by associating with each 
supernatural number a trio of ordinary in tegers (including negative ones). Thus, our

supernatural number a trio of ordinary in tegers (including negative ones). Thus, our 
original supernatural number, I, might have the index set (9,-8,3), and its successor, I + 1, 
might have the index set (9,-8,4). Now there is  no unique way to index the supernaturals; 
different methods offer different advantag es and disadvantages. Under some indexing 
schemes, it is very easy to calculate the index triplet for the sum of two supernaturals, given the indices of the two numbers to be  added. Under other indexing schemes, it is 
very easy to calculate the index triplet for the product  of two supernaturals, given the

very easy to calculate the index triplet for the product  of two supernaturals, given the 
indices of the two numbers to be multiplied. But under no indexing scheme is it possible 
to calculate both. More precisely, if the su m's index can be calculated by a recursive 
function, then the product's index will not be a recursive function; and conversely, if the 
product's index is a recursive function, then  the sum's index will not be. Therefore, 
supernatural schoolchildren who learn their supernatural plus-tables will have to be excused if they do not know their supernatur al times-tables-and vice versa! You cannot 
know both at the same time.

know both at the same time. 
 
Supernaturals Are Useful ... 
 
One can go beyond the number theory of supernaturals, and consider supernatural fractions (ratios of two supernaturals), supern atural real numbers, and so on. In fact, the 
calculus can be put on a new footing, using the notion of supernatural real numbers.

calculus can be put on a new footing, using the notion of supernatural real numbers. 
Infinitesimals such as dx and dy, those old bugaboos of mathematicians, can be completely justified, by considering them to be reciprocals of infinitely large real numbers! Some theorems in advanced analysis can be proven more intuitively with the aid of "nonstandard analysis".  
But Are They Real? 
Nonstandard number theory is a disorienting thing when you first meet up with it. But, 
then, non-Euclidean geometry is also a disorienting subject. In

On Formally Undecidable Propositions 450both instances, one is powerfully driven to ask, "But which one of these two rival theories 
is correct? Which is the truth ?" In a certain sense, there is  no answer to such a question. 
(And vet, in another sense-to be discussed later-there is an answer.) The reason that there is no answer to the question is that the tw o rival theories, although they employ the same 
terms, do not talk about the same concepts. Th erefore, they are only superficially rivals, 
just like Euclidean and non-Euclidean geometries . In geometry, the words "point", "line", 
and so on are undefined terms, and their m eanings are determined by the axiomatic 
system within which they are used. 
Likewise for number theory. When we decided to formalize TNT . we preselected 
the terms we would use as interpretation wo rds-for instance, words such as "number", 
"plus", "times", and so on. By taking the step of formalization, we were committing

"plus", "times", and so on. By taking the step of formalization, we were committing 
ourselves to accepting whatever passive meani ngs these terms might take on. But just like 
Saccheri-we didn't anticipate any surprises. We thought we knew what the true, the real, 
the only theory of natural numbers was. We didn't know that there would be some 
questions about numbers which TNT  would leave open, and which could therefore be 
answered ad libitum by extensions of TNT  heading off in different directions. Thus, there 
is no basis on which to say that number theory "really" is this way or that, just as one

is no basis on which to say that number theory "really" is this way or that, just as one 
would be loath to say that the square root of -1 "really" exists, or "really" does not. 
 
Bifurcations in Geometry, and Physicists 
 
There is one argument which can be, and perhaps ought to be, raised against the 
preceding. Suppose experiments in the real, physical world can be explained more 
economically in terms of one particular versi on of geometry than in terms of any other. 
Then it might make sense to say that that geom etry is "true". From the point of view of a 
physicist who wants to use the "correct" geometry, then it makes some sense to

physicist who wants to use the "correct" geometry, then it makes some sense to 
distinguish between the "true" geometry, and other geometries. But this cannot be taken 
too simplistically. Physicists are always dea ling with approximations and idealizations of 
situations. For instance, my own Ph.D. wor k, mentioned in Chapter V, was based on an 
extreme idealization of the problem of a cr ystal in a magnetic field. The mathematics 
which emerged was of a high degree of beau ty and symmetry. Despite-or rather, because 
of-the artificiality of the model, some fundame ntal features emerged conspicuously in the

of-the artificiality of the model, some fundame ntal features emerged conspicuously in the 
graph. These features then suggest some gue sses about the kinds of things that might 
happen in more realistic situations. But without the simplifying assumptions which 
produced my graph, there could never be such  insights. One can see this kind of thing 
over and over again in physics, where a physicis t uses a "nonreal" situation to learn about 
deeply hidden features of reality. Therefore,  one should be extremely cautious in saying 
that the brand of geometry which physicists might wish to use would represent “the

On Formally Undecidable Propositions 451true geometry", for in fact, physicists will al ways use a variety of different geometries, 
choosing in any given situation the one that seems simplest and most convenient. 
Furthermore-and perhaps this is even more to the point-physicists do not study 
just the 3-D space we live in. There are whol e families of "abstract spaces" within which 
physical calculations take place, spaces wh ich have totally different geometrical 
properties from the physical space within which we live. Who is to say, then, that "the true geometry" is defined by the space in which Uranus and Neptune orbit around the 
sun? There is "Hilbert space", where quantum -mechanical wave functions undulate; there 
is "momentum space", where Fourier compone nts dwell; there is "reciprocal space", 
where wave-vectors cavort; there is "phase space", where many-particle configurations 
swish; and so on. There is absolutely no reas on that the geometries of all these spaces

swish; and so on. There is absolutely no reas on that the geometries of all these spaces 
should be the same; in fact, they couldn't possi bly be the same! So it is essential and vital 
for physicists that different and "rival" geometries should exist. 
 
Bifurcations in Number Theory, and Bankers 
 
So much for geometry. What about number th eory? Is it also essential and vital that 
different number theories should coexist with  each other? If you asked a bank officer, my 
guess is that you would get an expression of horror and disbelief. How could 2 and 2 add 
up to anything but 4? And moreover, if 2 and 2 did not make 4, wouldn't world

up to anything but 4? And moreover, if 2 and 2 did not make 4, wouldn't world 
economies collapse immediately under the unbearable uncertainty opened up by that fact? Not really. First of all, nonstandard num ber theory doesn't threaten the age-old idea 
that 2 plus 2 equals 4. It differs from ordinary number theory only in the way it deals with the concept of the infinite. After all, every theorem of TNT  remains a theorem in any 
extension of TNT ! So bankers need not despair of the chaos that will arrive when 
nonstandard number theory takes over. 
And anyway, entertaining fears about old facts being changed betrays a

And anyway, entertaining fears about old facts being changed betrays a 
misunderstanding of the relationship betw een mathematics and the real world. 
Mathematics only tells you answers to questions  in the real world after you have taken 
the one vital step of choosing which kind of mathematics to apply. Even if there were a 
rival number theory which used the symbols `2',  `3', and `+', and in which a theorem said 
"2 + 2 = 3", there would be little reason for banke rs to choose to use that theory! For that 
theory does not fit the way money works. Y ou fit your mathematics to the world, and not

theory does not fit the way money works. Y ou fit your mathematics to the world, and not 
the other way around. For instance, we don't apply number theory to cloud systems, because the very concept of whole numbers  hardly fits. There can be one cloud and 
another cloud, and they will come together a nd instead of there being two clouds, there 
will still only be one. This doesn't prove that 1 plus 1 equals 1; it just proves that our 
number theoretical concept of “one” is not applicable in its full power to cloud counting.

On Formally Undecidable Propositions 452Bifurcations in Number Theory, and Metamathematicians 
 
So bankers, cloud-counters, and most of the rest  of us need not worry ,about the advent of 
supernatural numbers: they won't affect our everyday perception of the world in the slightest. The only people who might actually be a little worried are people whose endeavors depend in some crucial way on the na ture of infinite entities. There aren't too 
many such people around-but mathematical logicians are members of this category. How can the existence of a bifurcation in number theory affect them Well, number theory plays two roles in logic: (1) when axiomatize d, it is an object of study; and (2) when used 
informally, it is an indispensable tool with which formal systems can be investigated. This is the use-mention distinction once again, in fact: in role (1), number theory is mentioned, in role (2) it is used. 
Now mathematicians have judged that numbe r theory is applicable to the study of

Now mathematicians have judged that numbe r theory is applicable to the study of 
formal systems even if not to cloud-counti ng, just as bankers have judged that the 
arithmetic of real numbers is applicable  to their transactions. This is an 
extramathematical judgement, and shows th at the thought processes involved in doing 
mathematics, just like those in other areas , involve "tangled hierarchies" in which 
thoughts on one level can affect thoughts on a ny other level. Levels are not cleanly 
separated, as the formalist version of what mathematics is would have one believe. 
The formalist philosophy claims that math ematicians only deal with abstract 
symbols, and that they couldn't care less whet her those symbols have any applications to

symbols, and that they couldn't care less whet her those symbols have any applications to 
or connections with reality. But that is quite a distorted picture. Nowhere is this clearer 
than in metamathematics. If the theory of numbers is itself used as an aid in gaining 
factual knowledge about formal systems, then  mathematicians are tacitly showing that 
they believe these ethereal things called "nat ural numbers" are actually part of reality not 
just figments of the imagination. This is why I parenthetically remarked earlier that, in a

just figments of the imagination. This is why I parenthetically remarked earlier that, in a 
certain sense, there is an answer to the que stion of which version of number theory is 
"true". Here is the nub of the matter: mathem atical logicians must choose which version 
of number theory to put their faith in. In particular, they cannot remain neutral on the 
question of the existence or nonexistence of supernatural numbers, for the two different 
theories may give different answers to questions in metamathematics. 
For instance, take this question: "Is - G finitely derivable in TNT ?" No one

For instance, take this question: "Is - G finitely derivable in TNT ?" No one 
actually knows the answer. Nevertheless, most mathematical logicians would answer no without hesitation. The intuition which motivates th at answer is based on the fact that if -
G were a theorem, TNT  would be w-inconsistent, and this would force supernaturals 
down your throat if you wanted to interpret TNT  meaningfully-a most unpalatable 
thought for most people. After all, we didn't in tend or expect supernaturals to be part of

thought for most people. After all, we didn't in tend or expect supernaturals to be part of 
TNT  when we invented it. That is, we-or most of us-believe that it is possible to make a 
formalization of number theory which does not  force you into believing that supernatural 
numbers are every bit as real as naturals . It is that intuition about reality which 
determines which “fork” of number theory ma thematicians will put their faith in, when 
the chips are

On Formally Undecidable Propositions 453down. But this faith may be wrong. Perhaps ev ery consistent formalization of number 
theory which humans invent will imply the existence of supernaturals, by being co-
inconsistent. This is a queer thought, but it is conceivable. 
If this were the case-which I doubt, but there is no disproof available-then G 
would not have to be undecidable. In fact , there might be no undecidable formulas of 
TNT  at all. There could simply be one unbifur cated theory of numbers-which necessarily 
includes supernaturals. This is not the kind of  thing mathematical logicians expect, but it 
is something which ought not to be rejected outright. Generally, mathematical logicians believe that TNT -and systems similar to it-are ω-consistent, and that the Gödel string 
which can be constructed in any such system is undecidable within that system. That means that they can choose to add either it or its negation as an axiom. 
 
Hilbert's Tenth Problem and the Tortoise

Hilbert's Tenth Problem and the Tortoise 
 
I would like to conclude this Chapter by mentioning one extension of Gödel’s 
Theorem. (This material is more fully covered in the article "Hilbert's Tenth Problem" by Davis and Hersh, for which see the Bibliogr aphy.) For this, I must define what a 
Diophantine equation is. This is an equation in which a polynomial with fixed integral 
coefficients and exponents is set to 0. For instance, 
 
a=0 
 
and 
 
5x+13y-1=0 
 
And 
 
5p5 + 17 q17 - 177 = 0 
 
and 
 
a
123,666,111,666 + b123,.666,111,666 - c123,666, 111,666 = 0

a=0 
 
and 
 
And 
 
and 
 
a
123,666,111,666 + b123,.666,111,666 - c123,666, 111,666 = 0 
 
are Diophantine equations. It is in general a difficult matter to know whether a given Diophantine equation has any integer solutions or  not. In fact, in a famous lecture at the 
beginning of the century, Hilbert asked mathem aticians to look for a general algorithm by 
which one could determine in a finite numbe r of steps if a given Diophantine equation 
has integer solutions or not. Little did he suspect that no such algorithm exists!

On Formally Undecidable Propositions 454Now for the simplification of G. It has been shown that whenever you have a sufficiently 
powerful formal number theory and a Göde l-numbering for it, there is a Diophantine 
equation which is equivalent to G. The equivalence lies in the fact that this equation, 
when interpreted on a metamathematical level, asserts of itself that it has no solutions. Turn it around: if you found a solution to it,  you could construct from it the Gödel 
number of a proof in the system that the equation has no solutions! This is what the Tortoise did in the Prelude , using Fermat's equation as his Diophantine equation. It is 
nice to know that when you do this, you can re trieve the sound of Old Bach from the 
molecules in the air!

Birthday Cantatatata . . 
 455Birthday Cantatatata . . 
 
One (tine May day, the Tortoise and Achilles meet, wandering in the woods. 
The latter, all decked out handsomely, is doing a jiggish sort of thing to a tune which he himself is humming. On his vest he is wearing a great big button with the words "Today is my Birthday!" 
 
Tortoise: Hello there, .Achilles. What make s you so joyful today?  Is it your birthday, by 
any chance? 
Achilles: Yes, yes! Yes it is, today is my birthday! Tortoise: That is what I had suspected, on account of that button which you are wearing, 
and also because unless I am mistake n, you are singing a tune from a Birthday 
Cantata by Bach, one written in 1727 for the fifty-seventh birthday of Augustus, King 
of Saxony. 
Achilles: You're right. And Augustus' birthda y coincides with mine, so THIS Birthday 
Cantata has double meaning. However, I shan't tell you my age. 
Tortoise: Oh, that's perfectly all right. Ho wever, I would like to know one other thing.

of Saxony. 
Tortoise: Oh, that's perfectly all right. Ho wever, I would like to know one other thing. 
From what you have told me so far, would it be correct to conclude that today is your 
birthday? 
Achilles: Yes, yes, it would be. Today is my birthday. Tortoise: Excellent. That's just as I suspected. So now, I WILL conclude it is your 
birthday, unless 
Achilles: Yes-unless what? Tortoise: Unless that would be a premature or hasty conclusion to draw, you know. 
Tortoises don't like to jump to conclusions, af ter all. (We don't like to jump at all, but 
especially not to conclusions.) So le t me just ask you, knowing full well of your

birthday? 
especially not to conclusions.) So le t me just ask you, knowing full well of your 
fondness for logical thought, whether it would be reasonable to deduce logically from 
the foregoing sentences, that today is in fact your birthday. 
Achilles: I do believe I detect a pattern to your  questions, Mr. T. But rather than jump to 
conclusions myself, I shall take your question at face value, and answer it 
straightforwardly. The answer is: YES. 
Tortoise: Fine! Fine! Then there is only one more thing I need to know, to be quite 
certain that today is 
Achilles: Yes, yes, yes, yes ... I can alrea dy see the line of your questioning, Mr. T. I'll

Achilles: Yes, yes, yes, yes ... I can alrea dy see the line of your questioning, Mr. T. I'll 
have you know that I am not so gullible as I was when we discussed Euclid's proof, a 
while back. 
Tortoise: Why, who would ever have thought you to be gullible? Quite to the contrary, I 
regard you as an expert in the forms of l ogical thought, an authority in the science of 
valid deductions. a fount of knowledge about certain correct methods of reasoning. . .  
To tell the truth, Achilles, you are, in my opi nion, a veritable titan in the art of rational 
cogitation.

 456And it is only for that reason that I woul d ask you, "Do the foregoing sentences present 
enough evidence that I should conclude without  further puzzlement that today is your 
birthday 
Achilles: You flatten me with your weighty praise, Mr. T-FLATTER , I mean. But I am 
struck by the repetitive nature of your ques tioning and in my estimation, you, just as 
well as I, could have answered `yes' each time. 
Tortoise: Of course I could have, Achilles. But you see, to do so would have been to 
make a Wild Guess-and Tortoises abhor Wild Guesses. Tortoises formulate only 
Educated Guesses. Ah, yes-the power of the Educated Guess. You have no idea how 
many people fail to take into account all the Relevant Factors when they're guessing. 
Achilles: It seems to me that there was only one relevant factor in this rigmarole, and that 
was my first statement. 
Tortoise: Oh, to be sure, it's at least ONE  of the factors to take into account, I'd say-but

Tortoise: Oh, to be sure, it's at least ONE  of the factors to take into account, I'd say-but 
would you have me neglect Logic, that vene rated science of the ancients? Logic is 
always a Relevant Factor in making Educated Guesses, and since I have with me a renowned expert in Logic, I thought it only Logi cal to take advantage of that fact, and 
confirm my hunches, by directly asking him whether my intuitions were correct. So 
let me finally come out and ask you point blank: "Do the preceding sentences allow 
me to conclude, with no room for doubt, that Today is your Birthday?"

me to conclude, with no room for doubt, that Today is your Birthday?" 
Achilles: For one more time, YES. But fra nkly speaking, I have the distinct impression 
that you could have supplied that answer-as well as all the previous ones-yourself. 
Tortoise: How your words sting! Would I were  so wise as your insinuation suggests! But 
as merely a mortal Tortoise, profoundly i gnorant and longing to take into account all 
the Relevant Factors, I needed to know the answers to all those questions. 
Achilles: Well then, let me clear the matter up for once and for all. Th e answer to all the

Achilles: Well then, let me clear the matter up for once and for all. Th e answer to all the 
previous questions, and to all the succeed ing ones which you will ask along the same 
line, is just this: YES . 
Tortoise: Wonderful! In one fell swoop, you have  circumvented the whole mess, in your 
characteristically inventive manner. I hope  you won't mind if I call this ingenious 
trick an ANSWER SCHEMA . It rolls up yes-answers numbers 1, 2, 3, etc., into one 
single ball. In fact, coming as it does at the end of the line, it deserves the title 
"Answer Schema Omega", `w' being the last  letter of the Greek alphabet-as if YOU 
needed to be told THAT !

needed to be told THAT ! 
Achilles: I don't care what you call it. I am just very relieved that you finally agree that it 
is my birthday, and we can go on to some other topic-such as what you are going to 
give me as a present.  
Tortoise: Hold on—not so fast. I WILL agree it is your birthday, provided on thing Achilles: What? That I Ask for no present?

 457Tortoise: Not at all. In fact, Achilles, I am looking forward to treating you to a fine 
birthday dinner, provided merely that I am convinced that knowledge of all those yes-answers at once (as supplied by Answer Schema w) allows me to proceed directly and without any further detours to the conclusi on that today is your birthday. That's the 
case, isn't it? 
Achilles: Yes, of course it is. Tortoise: Good. And now I have yes-answer ω + 1. Armed with it, I can proceed to 
accept the hypothesis that today is your birthday, if it is valid to do so. Would you be so kind as to counsel me on that matter, Achilles? 
Achilles: What is this? I thought I had seen  through your infinite plot. Now doesn't yes-
answer ω + 1 satisfy you? All right. I'll give you not only yes-answer ω + 2, but also 
yes-answers ω + 3, ω + 4, and so on. 
Tortoise: How generous of you, Achilles. And here it is your birthday, when I should be

Tortoise: How generous of you, Achilles. And here it is your birthday, when I should be 
giving YOU presents instead of the reverse.  Or rather, I SUSPECT it is your birthday. 
I guess I can conclude that it IS your bi rthday, now, armed with the new Answer 
Schema, which I will call "Answer Schema 2 ω ". But tell me, Achilles: Does Answer 
Schema 2 ω REALLY allow me to make that enormous leap, or am I missing 
something? 
Achilles: You won't trick me any more, Mr. T. I've seen the way to end this silly game. I 
hereby shall present you with an Answer Sche ma to end all Answer Schemas! That is,

hereby shall present you with an Answer Sche ma to end all Answer Schemas! That is, 
I present you simultaneously with Answer Schemas ω, 2 ω, 3 ω, 4 ω, 5 ω, etc. With 
this Meta-Answer-Schema, I have JUMP ED OUT of the whole system, kit and 
caboodle, transcended this silly game you thought you had me trapped in-and now we 
are DONE! 
Tortoise: Good grief! I feel honored, Achilles,  to be the recipient of such a powerful 
Answer Schema. I feel that seldom has a nything so gigantic been devised by the mind 
of man, and I am awestruck by its power. Would you mind if I give a name to your 
gift? 
Achilles: Not at all. Tortoise: Then I shall call it "Answer Schema ω

are DONE! 
gift? 
2". And we can shortly proceed to other 
matters-as soon as you tell me whether the possession of Answer Schema ω2 allows 
me to deduce that today is your birthday. 
Achilles: Oh, woe is me! Can't I ever reach th e end of this tantalizing trail? What comes 
next? 
Tortoise: Well, after Answer Schema ω2 there's answer ω2 + 1. And then answer ω2 + 2. 
And so forth. But you can wrap those all together into a packet, being Answer 
Schema ω2 + ω. And then there are quite a few other answer-packets, such as ω2 + 2ω,

next? 
Schema ω2 + ω. And then there are quite a few other answer-packets, such as ω2 + 2ω, 
and ω2 + 3ω…. Eventually you come to Answer Schema 2 ω2, and after a while, 
Answer Schemas 3 ω2 and 4ω2. Beyond them there

 458are yet further Answer Schemas, such as ω3;, ω4, ω5, and so on. It goes on quite a 
ways, you know. 
Achilles: I can imagine, I suppose it comes to Answer Schema are yet further Answer 
Schemas, such as w;, w4, w5, and so on. It goes on quite a ways, you know. 
Achilles: I can imagine, I suppose it comes to Answer Schema  ωω after a while. 
Tortoise: Of course. Achilles: And then ω
ωω, and ωωωω', 
Tortoise: You're catching on mighty fast, Ac hilles. I have a suggestion for you, if you 
don't mind. Why don't you throw all of those together into a single Answer Schema? 
Achilles: All right, though I'm beginning to doubt whether it will do any good. Tortoise: It seems to me that within our na ming conventions as so far set up, there is no 
obvious name for this one. So perhaps we should just arbitrarily name it Answer 
Schema Œo. 
Achilles: Confound it all! Every time you give one of my answers a NAME, it seems to

Schema Œo. 
Achilles: Confound it all! Every time you give one of my answers a NAME, it seems to 
signal the imminent shattering of my hope s that that answer will satisfy you. Why 
don't we just leave this Answer Schema nameless? 
Tortoise: We can hardly do that, Achilles. We wouldn't have any way to refer to it 
without a name. And besides, there is some thing inevitable and rather beautiful about 
this particular Answer Schema. It would be quite ungraceful to leave it nameless! And you wouldn't want to do something lacki ng in grace on your birthday, would you? Or 
is it your birthday? Say, speaking of birthdays, today is MY' birthday!

is it your birthday? Say, speaking of birthdays, today is MY' birthday! 
Achilles: It is? Tortoise: Yes, it is. Well, actually, it's my unc le's birthday, but that's almost the same. 
How would you like to treat me to a delicious birthday dinner this evening? 
Achilles: Now just a cotton-picking minute, Mr. T. Today is MY birthday. You should do 
the treating! 
Tortoise: Ah, but you never did succeed in convinc ing me of the veracity of that remark. 
You kept on beating around the bush with an swers, Answer Schemas, and whatnot. 
All I wanted to know was if it was your bi rthday or not, but you managed to befuddle

All I wanted to know was if it was your bi rthday or not, but you managed to befuddle 
me entirely. Oh, well, too bad. In any case,  I'll be happy to let you treat me to a 
birthday dinner this evening. 
Achilles: Very well. I know just the place. Th ey have a variety of delicious soups. And I 
know exactly what kind we should have ...

 459CHAPTER XV 
 
A More Powerful Formal System 
 ONE OF THE things which a thoughtful critic of Gödel’s proof might do would be to examine its generality. Such a critic might, for example, suspect that Gödel has just 
cleverly taken advantage of a hidden defect in one particular formal system, TNT. If this 
were the case, then perhaps a formal system superior to TNT  could be developed which 
would not be subject to the Gödelian trick, and Gödel’s Theorem would lose much of its sting. In this Chapter we will carefully scrutinize the properties of TNT  which made it 
vulnerable to the arguments of last Chapter. 
A natural thought is this: If the basic trouble with TNT  is that it contains a "hole"-
in other words, a sentence which is undecidable, namely G-then why not simply plug up 
the hole? Why not just tack G onto TNT  as a sixth axiom? Of course, by comparison to

the hole? Why not just tack G onto TNT  as a sixth axiom? Of course, by comparison to 
the other axioms,  G is a ridiculously huge giant, and the resulting system- TNT+G -would 
have a rather comical aspect due to the disproportionateness of its axioms. Be that as it may, adding G is a reasonable suggestion. Let us consider it done. Now, it is to be hoped, 
the new system, TNT+G , is a superior formal system-one which is not only supernatural-
free, but also complete. It is certain that TNT+G  is superior to TNT  in at least one

free, but also complete. It is certain that TNT+G  is superior to TNT  in at least one 
respect: the string G is no longer undecidable in this new system, since it is a theorem. 
What was the vulnerability of TNT  due to? The essence of its vulnerability was 
that it was capable of expressing statements about itself-in particular, the statement  
"I Cannot Be Proven in Formal System TNT "  
or, expanded a bit,  
"There does not exist a natural number which forms a TNT -proof-pair with 
the Gödel number of this string." 
 Is there any reason to expect or hope that TNT+G  would be invulnerable to Gödel’s

 Is there any reason to expect or hope that TNT+G  would be invulnerable to Gödel’s 
proof? Not really. Our new system is just as expressive as TNT . Since Gödel’s proof 
relies primarily on the expressive power of a formal system, we should not be surprised to see our new system succumb,

 460too. The trick will be to find a string which expresses the statement  
 
"I Cannot Be Proven in Formal System TNT+G ." 
 Actually, it is not much of a trick, once you have seen it done for TNT . All the same 
principles are employed: only, the context shif ts slightly. (Figuratively speaking, we take 
a tune we know and simply sing it again, only in a higher key.) As before, the string which we are looking for-let us call it " G"'-is constructed by the intermediary of an 
"uncle", But instead of being based on the formula which represents TNT -proof-pairs, it 
is based on the similar but slightly more complicated notion of TNT+G -proofpairs. This 
notion of TNT+G -proof-pairs is only a slight extension of the original notion of' TNT -
proof-pairs. 
A similar extension could be envisaged for the MIU -system. We have seen the 
unadulterated form of MIU -proof-pairs. Were we now to add MU as a second axiom, we

unadulterated form of MIU -proof-pairs. Were we now to add MU as a second axiom, we 
would be dealing with a new system-the MIU+MU  system. A derivation in this extended 
system is presented: 
 
MU  axiom 
MUU   rule 2 
 
There is a MIU+MU -proof-pair which corresponds-namely, m = 30300, n = 300. Of 
course, this pair of numbers does not form a MIU -proof-pair-only a MIU+MU -proof-
pair. The addition of an extra axiom does not substantially complicate the arithmetical properties of proof-pairs. The significant fact about them-that being a proof-pair is primitive recursive-is preserved.  
The Gödel Method Reapplied

MU  axiom 
MUU   rule 2 
 
The Gödel Method Reapplied 
 
Vow, returning to TNT+G , we will find a similar situation . TNT+G  proof-pairs, like 
their predecessors, are primitive recursive, so they are represented inside TNT+G  by a 
formula which we abbreviate in an obvious manner.  
(TNT+G)-PROOF-PAIR{a,a' } 
 
Vow we just do everything all over again. We make the counterpart of  G by beginning 
with an "uncle", just as before:  
3a:3a':<(TNT+G)-PROOF-PAIR{a,a'}ARITHMOQUINE{a",a'}> 
 
.et us say its Gödel-number is u'. Now we arithmoquine this very uncle. That will give us G': 
 
3a:3a': < (TNT+G)-PROOF-PAIR{a,a' }

3a:3a': < (TNT+G)-PROOF-PAIR{a,a' } 
ARITHMOQUINE{SSS….SSSo/a´´,a´}>  
U´ S´s

 461Its interpretation is 
 
"There is no number a that forms a TNT +G -proof-pair 
with the arithmoquinification of u'." 
More concisely,  
 
Multifurcation 
 Well (yawn), the details are quite boring from here on out. G' is precisely to TNT+G  as 
G was to TNT  Itself. One finds that either G' or -G' can be added to TNT+G , to yield a 
further splitting of number theory. And, lest you think this only happens to the "good guys", this very same dastardly trick can be played upon TNT+--G -that is, upon the 
nonstandard extension of TNT  gotten by adding G's negation. So now we see (Fig. 75) 
that there are all sorts of bifurcations in number theory:  
 
FIGURE 75. "Multifurcation" of TNT . Each extension of TNT  has its very own Gödel 
sentence; that sentence, or its negation, can be added on, so that from each extension there sprouts a pair of further extensions, a process which goes on ad infinitum.

sentence; that sentence, or its negation, can be added on, so that from each extension there sprouts a pair of further extensions, a process which goes on ad infinitum. 
 
Of course, this is just the beginning. Let us imagine moving down the leftmost branch of this downwards-pointing tree, where we always toss in the Gödel sentences (rather than their negations). This is the best we can do by way of eliminating supernaturals. After adding G, we add G'. Then we add G", and G"', and so on. Each time we make a new 
extension of TNT , its vulnerability to the Tortoise's method-pardon me, I mean Gödel’s 
method.. allows a new string to be devised, having the interpretation.  
“I cannot be proven in formal system X”

 462Naturally, after a while, the whole process begins  to seem utterly predictable and routine. 
Why, all the "holes" are made by one single technique! This means that, viewed as 
typographical objects, they are all cast from one single mold, which in turn means that 
one single axiom schema suffices to represent a ll of them! So if this  is so, why not plug 
up all :he holes at once and be done with th is nasty business of incompleteness 3nce and 
for all? This would be accomplishe d by adding an axiom schema to TNT , instead of just 
one axiom at a time. Specifically, this axiom schema would be the mold in which all of 
G, G', G", G "', etc., are cast. By adding :his  axiom schema (let's call it " G~"), we would 
be outsmarting the "Gödelization" method. Indeed, it seems quite clear that adding G. to 
TNT  would :)e the last step necessary for the complete axiomatization of all of number-
theoretical truth.

theoretical truth. 
It was at about this point in the Contr acrostipunctus that the Tortoise related the 
Crab's invention of "Record Player Omega". However, readers were left dangling as to 
the fate of that device, since before completi ng his tale, the tuckered-out Tortoise decided 
that he had best go home to sleep; but not be fore tossing off a sly reference to Gödel’s 
Incompleteness Theorem). Now, at last, we  can get around to clearing up that dangling 
detail ... Perhaps you already have an inkling, after reading the Birthday Cantatatata.

detail ... Perhaps you already have an inkling, after reading the Birthday Cantatatata. 
 
Essential Incompleteness 
 
As you probably suspected, even this fantastic advance over TNT  suffers the same fate. 
And what makes it quite weird is that it is still for, in essence, the same reason. The 
axiom schema is not powerful enough, and the Gödel construction can again be effected. 
Let me spell this out a little. (One can do it mu ch more rigorously than I shall here.) If 
there is a way of capturing the various strings G, G', G", G "' . . in a single typographical

there is a way of capturing the various strings G, G', G", G "' . . in a single typographical 
mold, then there is a way of describing thei r Gödel numbers in a single arithmetical mold. 
And this arithmetical portrayal of an infinite  class of numbers can then be represented 
inside TNT+G . by some formula OMEGA-AXIOM {a} whose interpretation is: "a is the 
Godel number of one of the axioms coming from  G.". When a is replaced by any specific 
numeral, the formula which results will be a theorem of TNT+G . if and only if the 
numeral stands for the Gödel number of an axiom coming from the schema.

numeral stands for the Gödel number of an axiom coming from the schema. 
With the aid of this new formula, it b ecomes possible to represent even such a 
complicated notion as TNT+G .-proof-pairs inside TNT+Gω: 
 
(TNT+G.)- PROOF- PAIR{a, a' ) 
 
sing this formula, we can construct a new uncle, which we proceed to Arithmoquine in the by now thoroughly familiar way, making ye t another undecidable string, which will 
be called " TNT+Gω+i". At this point, you might well wonder, "Why isn't Gω+i among 
the axioms created by the axiom schema  Gω?” The answer is that G was not clever

the axioms created by the axiom schema  Gω?” The answer is that G was not clever 
enough to foresee its own embeddability inside number theory.

 463In the Contracrostipunctus, one of the esse ntial steps in the Tortoise's making an 
"unplayable record" was to get a hold of a ma nufacturer's blueprint of the record player 
which he was out to destroy. This was necessary  so that he could figure out to what kinds 
of vibrations it was vulnerable, and then in corporate into his record such grooves as 
would code for sounds which would induce those vibrations. It is a close analogue to the 
Gödel trick, in which the system's own proper ties are reflected inside the notion of proof-
pairs, and then used against it. Any system , no matter how complex or tricky it is, can be 
Gödel-numbered, and then the notion of its proof-pairs can be defined-and this is the petard by which it is hoist. Once a system  is well-defined, or "boxed", it becomes 
vulnerable. 
This principle is excellently illustrated by the Cantor diagonal trick, which finds

This principle is excellently illustrated by the Cantor diagonal trick, which finds 
an omitted real number for each well-defined list of reals between 0 and 1. It is the act of 
giving an explicit list-a "box" of reals whic h causes the downfall. Let us see how the 
Cantor trick can be repeated over and over ag ain. Consider what happens if, starting with 
some list L, you do the following: 
 (la) Take list L, and construct its diagonal number d. (lb) Throw d somewhere into list L, making a new list L+d.  (2a) Take list L +d, and construct its diagonal number d'. (2b) Throw d' somewhere into list L+d, making a new list L+d+d'.

 (la) Take list L, and construct its diagonal number d. (lb) Throw d somewhere into list L, making a new list L+d.  (2a) Take list L +d, and construct its diagonal number d'. (2b) Throw d' somewhere into list L+d, making a new list L+d+d'. 
 Now this step-by-step process may seem a doltish way to patch up  L, for we could have 
made the entire list d, d', d", d "', ... at once, given L originally. But if you think that 
making such a list will enable you to complete  your list of reals, you are very wrong. The 
problem comes at the moment you ask, "Whe re to incorporate the list of diagonal 
numbers inside L?" No matter how diabolically clever a scheme you devise for

Now this step-by-step process may seem a doltish way to patch up  L, for we could have 
numbers inside L?" No matter how diabolically clever a scheme you devise for 
ensconcing the d-numbers inside L, once you have done it, then the new list is still vulnerable. As was said above, it is the act of  giving an explicit list-a "box" of reals-that 
causes the downfall. 
Now in the case of formal systems, it is the act of giving an explicit recipe for 
what supposedly characterizes number-theoretical truth that causes the incompleteness. This is the crux of the problem with TNT+Gω,. Once you insert all the G's in a well-
defined way into TNT, there is seen to be some other G-some unforeseen G-which you

defined way into TNT, there is seen to be some other G-some unforeseen G-which you 
didn't capture in your axiom sc hema. And in the case of the TC-battle inside the 
ContracrostiPunctus , the instant a record player's "arch itecture" is determined, the record 
player becomes capable of being shaken to pieces. 
So what is to be done? There is no end in sight. It appears that TNT , even when 
extended ad infinitum, cannot be made complete. TNT  is therefore said to suffer from 
essential incompleteness  because the income-

 464pleteness here is part and parcel of TNT ; it is an essential part of the nature of TNT  and 
cannot be eradicated in any way, whether si mpleminded or ingenious. What's more, this 
problem will haunt any formal version of num ber theory, whether it is an extension of 
TNT , a modification of TNT , or an alternative to TNT . The fact of the matter is this: the 
possibility of constructing, in a given syst em, an undecidable string via Gödel’s self-
reference method, depends on three basic conditions:  
(1) That the system should be rich enough so that all desired statements about 
numbers, whether true or false, can be expressed in it. (Failure on this count means that the system is from the very st art too weak to be c ounted as a rival to 
TNT , because it can't even express number-theoretical notions that TNT  can. 
In the metaphor of the Contracrosttpunctu s, it is as if one did not have a 
phonograph but a refrigerator or some other kind of object.)

phonograph but a refrigerator or some other kind of object.) 
(2) That all general recursive relations should be represented by formulas in the 
system. (Failure on this count means the system fails to capture in a theorem 
some general recursive truth, which can only be considered a pathetic bellyflop if it is attempting to produce all of number theory's truths. In the Contracrostipunctus metaphor, this is lik e having a record player, but one of 
low fidelity.) 
(3) That the axioms and typographical patterns defined by its rules be recognizable

(3) That the axioms and typographical patterns defined by its rules be recognizable 
by some terminating decision procedure. (Failure on this count means that there is no method to distinguish valid deriva tions in the system from invalid ones-
thus that the "formal system" is not formal  after all, and in fact is not even well-
defined. In the Contracrostipunctus me taphor, it is a phonograph which is still 
on the drawing board, only partially designed.) 
 
Satisfaction of these three conditions guarantees that any consistent system will be 
incomplete, because Gödel’s construction is applicable.

incomplete, because Gödel’s construction is applicable. 
The fascinating thing is that any such system digs its own hole; the system's own 
richness brings about its own downfall. Th e downfall occurs essentially because the 
system is powerful enough to have self-referential sentences. In physics, the notion exists of a "critical mass" of a fissionable substan ce, such as uranium. A solid lump of the 
substance will just sit there, if its mass is less than critical. But beyond the critical mass, 
such a lump will undergo a chain reaction, a nd blow up. It seems that with formal

such a lump will undergo a chain reaction, a nd blow up. It seems that with formal 
systems there is an analogous critical point. Below that point, a system is "harmless" and does not even approach defining arithmetical truth formally; but beyond the critical point, 
the system suddenly attains the capacity for self-reference, and thereby dooms itself to incompleteness. The threshold seems to be  roughly when a system attains the three 
properties listed above.

 465Once this ability for self-reference is attaine d, the system has a hole which is tailor-made 
for itself; the hole takes the features of the system into account and uses them against the 
system.  
The Passion According to Lucas 
 The baffling repeatability of the Gödel argumen t has been used by various people-notably 
J. R. Lucas-as ammunition in the battle to show  that there is some elusive and ineffable 
quality to human intelligence, which makes it unattainable by "mechanical automata"-that is, computers. Lucas begins his article "Minds, Machines, and Gödel" with these words:  
Gödel’s theorem seems to me to prove that Mechanism is false, that is, that minds cannot be explained as machines.'  Then he proceeds to give an argument which, paraphrased, runs like this. For a 
computer to be considered as intelligent as  a person is, it must be able to do every 
intellectual task which a person can do. No w Lucas claims that no computer can do

intellectual task which a person can do. No w Lucas claims that no computer can do 
"Gödelization" (one of his amusingly irreve rent terms) in the manner that people can. 
Why not? Well, think of any particular formal system, such as TNT , or TNT+G , or even 
TNT+G .. One can write a computer program ra ther easily which will systematically 
generate theorems of that system, and in su ch a manner that eventually, any preselected 
theorem will be printed out. That is, the theorem-generating program won't skip any 
portion of the "space" of all theorems. Such a program would be composed of two major

portion of the "space" of all theorems. Such a program would be composed of two major 
parts: (1) a subroutine which stamps out axioms, given the "molds" of the axiom schemas (if there are any), and (2) a subroutine whic h takes known theorems (including axioms, of 
course) and applies rules of inference to  produce new theorems. The program would 
alternate between running first one of these subroutines, and then the other. 
We can anthropomorphically say that this  program "knows" some facts of number 
theory-namely, it knows those facts which it prints  out. If it fails to print out some true

theory-namely, it knows those facts which it prints  out. If it fails to print out some true 
fact of number theory, then of  course it doesn't "know" that fact. Therefore, a computer 
program will be inferior to human beings if  it can be shown that humans know something 
which the program cannot know. Now here is wh ere Lucas starts rolling. He says that we 
humans can always do the Gödel trick on any formal system as powerful as TNT-and 
hence no matter what the formal system, we know more than it does. Now this may only sound like an argument about formal systems, but it can also be slightly modified so that

hence no matter what the formal system, we know more than it does. Now this may only sound like an argument about formal systems, but it can also be slightly modified so that 
it becomes, seemingly, an invincible argument against the possibility of Artificial Intelligence ever reproducing the human level of intelligence. Here is the gist of it: 
 Rigid internal codes entirely rule computers and robots; ergo ...  Computers are isomorphic to formal systems. Now .. . Any computer which wants to be as smar t as we are has got to be able to do 
number theory as well as we can, so….

 466Among other things, it has to be able to do primitive recursive arithmetic. But for 
this very reason .. . 
It is vulnerable to the Gödelian "hook", which implies that ... We, with our human intelligence, can concoct a certain statement of number 
theory which is true, but the computer is blind to that statement's truth (i.e., will 
never print it out), precisely because of Gödel’s boomeranging argument. 
This implies that there is one thing wh ich computers just cannot be programmed 
to do, but which we can do. So we are smarter. 
 Let us enjoy, with Lucas, a transient moment of anthropocentric glory:  
However complicated a machine we construc t, it will, if it is a machine, correspond 
to a formal system, which in turn will be  liable to the Gödel procedure for finding a 
formula unprovable-in-that-system. This formula the machine will be unable to

formula unprovable-in-that-system. This formula the machine will be unable to 
produce as being true, although a mind can see it is true. And so the machine will still not be an adequate model of the mi nd. We are trying to produce a model of the 
mind which is mechanical-which is essen tially "dead"-but the mind, being in fact 
"alive," can always go one better than any formal, ossified, dead system can. Thanks to Gödel’s theorem. the mind always has the last word.
2 
 
On first sight, and perhaps even on caref ul analysis, Lucas' argument appears 
compelling. It usually evokes rather polarized  reactions. Some ;eize onto it as a nearly

2 
 
compelling. It usually evokes rather polarized  reactions. Some ;eize onto it as a nearly 
religious proof of the existence of souls, while others laugh it off as being unworthy of 
comment. I feel it is wrong, but Fascinatingl y so-and therefore quite worthwhile taking 
the time to rebut. In fact, it wa s one of the major early forces driving me to think over the 
matters in this book. I shall try to rebut it in one  way in this Chapter, and in ether ways in 
Chapter XVII. 
We must try to understand more deeply why Lucas says the computer cannot be 
programmed to "know" as much as we do. Ba sically the idea is :hat we are always

programmed to "know" as much as we do. Ba sically the idea is :hat we are always 
outside the system, and from out there we can always perform the "Gödelizing" 
operation, which yields something which the pr ogram, from within, can't see is true. But 
why can't the "Gödelizing operator", as Lucas calls it, be programmed and added to the 
program as a third major component, Lucas explains: 
 
The procedure whereby the Gödelian form ula is constructed is a standard 
procedure-only so could we be sure that a Gödelian formula can be constructed for 
every formal system. But if it is a standa rd procedure, then a machine should be

every formal system. But if it is a standa rd procedure, then a machine should be 
able to be programmed to carry it out  too.... This would correspond to having a 
system with an additional rule of inferen ce which allowed one to add, as a theorem, 
the Gödelian formula of the rest of' th e formal system, and then the Gödelian 
formula of this new, strengthened, fo rmal system, and so on. It would be 
tantamount to adding to the original formal  system an infinite sequence of axioms, 
each the Gödelian formula of the system hitherto obtained… We might expect a

each the Gödelian formula of the system hitherto obtained… We might expect a 
mind, faced with a machine that possessed a Gödelizing operator, to take this into 
account, and

 467out-Gödel the new machine, Gödelizing operato r and all. This has, in fact, proved 
to be the case. Even if we adjoin to a formal system the infinite set of axioms 
consisting of the successive Gödelian fo rmulae, the resulting system is still 
incomplete, and contains a formula which cannot be proved-in-the system, although a rational being can, standing outside the system, see that it is true. We had expected this, for even if an infinite set of axioms were added, they would have to 
be specified by some finite rule or specification, and this further rule or 
specification could then be taken into account by a mind considering the enlarged 
formal system. In a sense, just because the mind has the last word, it can always 
pick a hole in any formal system presente d to it as a model of its own workings. 
The mechanical model must be, in some se nse, finite and definite: and then the 
mind can always go one better.' 
 
Jumping Up a Dimension

Jumping Up a Dimension 
 A visual image provided by M. C. Escher is extremely useful in aiding the intuition here: 
his drawing Dragon  (Fig. 76). Its most salient feature is, of course, its subject matter-a 
dragon biting its tail, with all the Gödelian c onnotations which that carries. But there is a 
deeper theme to this picture. Escher hi mself wrote the following most interesting 
comments. The first comment is about a set of his drawings all of which are concerned with "the conflict between the flat and the spatial"; the second comment is about Dragon 
in particular.

in particular.  
I    Our three-dimensional space is the only true reality we know. The two-dimensional is every bit as fictitious as the four-dimensional, for nothing is flat, not even the most finely polished mirror. And yet we stick to the convention that a wall 
or a piece of paper is flat, and curious ly enough, we still go on, as we have done 
since time immemorial, producing illusions of  space on just such plane surfaces as 
these. Surely it is a bit absurd to draw a few lines and then claim: "This is a house".

these. Surely it is a bit absurd to draw a few lines and then claim: "This is a house". 
This odd situation is the theme of the next five pictures ( Including Dragon) II.    However much this dragon tries to be  spatial, he remains completely flat. Two 
incisions are made in the paper on which he is printed. Then it is folded in such a 
way as to leave two square openings. But th is dragon is an obstinate beast, and in' 
spite of his two dimensions he persists in a ssuming that he has three; so he sticks his 
head through one of the holes and his tail through the  others 
5

5 
 
This second remark especially is a very telling remark. The message is that no matter how cleverly you try to simulate three dime nsions in two, you are always missing some 
"essence of three-dimensionality". The dr agon tries very hard to fight his two-
dimensionality. He defies the two-dimensionality  of the paper on which he thinks he is 
drawn, by sticking his head through it; and yet all the while, we outside the drawing can

5 
 
drawn, by sticking his head through it; and yet all the while, we outside the drawing can 
see the pathetic futility of it all, for the dragon and the holes and the folds are all merely two-dimensional simulations of those concepts , and not a one of them is real. But the 
dragon cannot step out of his two-dimensional space, and cannot

 468
 
FIGURE 76. Dragon, by M. C. Escher (wood-engraving , 1952). 
 know it as we do. We could, in fact, carry the Escher picture any number of steps further. 
For instance, we could tear it out of the book, fold it, cut holes in it, pass it through itself, and photograph the whole mess, so that it agai n becomes two-dimensional. And to that 
photograph, we could once again do the same tr ick. Each time, at the instant that it 
becomes two- Matter how 'cleverly we seem to have simulated three dimensions inside two—it becomes vulnerable to being cut and folded again.

 469Now with this wonderful Escherian metaphor , let us return to the program versus 
the human. We were talking about trying to encapsulate the "Gödelizing operator" inside 
the program itself. Well, even if we had written a program which carried the operation 
out, that program would not capture the e ssence of Gödel’s method. For once again, we, 
outside the system, could still "zap" it in a way which it couldn't do. But then are we arguing with, or against, Lucas 
 
The Limits of Intelligent Systems 
 Against. For the very fact that we cannot write a program to do "Gödelizing" must make 
us somewhat suspicious that we ourselves c ould do it in every case. It is one thing to 
make the argument in the abstract that Göde lizing "can be done"; it is another thing to 
know how to do it in every particular case. In  fact, as the formal systems (or programs) 
escalate in complexity, our own ability to "Göde lize" will eventually begin to waver. It

escalate in complexity, our own ability to "Göde lize" will eventually begin to waver. It 
must, since, as we have said above, we do not have any algorithmic way of describing 
how to perform it. If we can't tell explic itly what is involved in applying the Gödel 
method in all cases, then for each of us there will eventually come some case so 
complicated that we simply can't figure out how to apply it. 
Of course, this borderline of one's abilities will be somewhat ill-defined, just as is 
the borderline of weights which one can pi ck up off the ground. While on some days you

the borderline of weights which one can pi ck up off the ground. While on some days you 
may not be able to pick up a 250- pound object, on other days maybe you can. 
Nevertheless, there are no days whatsoev er on which you can pick up a 250-ton object. 
And in this sense, though everyone's Godeli zation threshold is vague, for each person, 
there are systems which lie far beyond his ability to Godelize. 
This notion is illustrated in the Birthday Cantatatata. At first, it seems obvious 
that the Tortoise can proceed as far as he wi shes in pestering Achilles. But then Achilles

that the Tortoise can proceed as far as he wi shes in pestering Achilles. But then Achilles 
tries to sum up all the answers in a single sw oop. This is a move of a different character 
than any that has gone before, and is given the new name 'co'. The newness of the name 
is quite important. It is the first example where the old naming scheme-which only included names for all the natural numbers-had to be transcended. Then come some more 
extensions, some of whose names seem quite  obvious, others of which are rather tricky. 
But eventually, we run out of names once again-at the point where the answer-schemas

But eventually, we run out of names once again-at the point where the answer-schemas 
 Ω, ω
ω, ωωω  ….. 
 
are all subsumed into one outrageously complex answer schema. The altogether new name 'e„' is supplied for this one. And the reason a new name is needed is that some 
fundamentally new kind of step has been taken—a sort of irregularity has been 
encountered. Thus a new name must be applied ad hoc.

 470There Is No Recursive Rule  for Naming Ordinals. 
 
Now offhand you might think that these irre gularities in the progression >m ordinal to 
ordinal (as these names of infinity are calle d) could be handled by a computer program. 
That is, there would be a program to produ ce new names in a regular way, and when it 
ran out of gas, it would invoke the "irregul arity handler", which would supply a new 
name, and pass control back to the simple one . But this will not work. It turns out that 
irregularities themselves happen in irregular ways, and one would need o a second-order program-that is, a program which makes new programs which make new names. And even this is not enough. Eventually, a third-order program becomes necessary. And so on, and so on. 
All of this perhaps ridiculous-seeming co mplexity stems from a deep °theorem, 
due to Alonzo Church and Ste phen C. Kleene, about the structure of these "infinite 
ordinals", which says:

ordinals", which says: 
 
There is no recursively related notation-system which gives a name to every constructive ordinal. 
 hat "recursively related notation-systems" are,  and what "constructive ordinals" are, we 
must leave to the more technical sources, su ch as Hartley )gets' book, to explain. But the 
intuitive idea has been presented. As the ordinals get bigger and bigger, there are irregularities, and irregularities in e irregularitie s, and irregularities in the irregularities in 
the irregularities, etc. No single scheme, no matter how complex, can name all e ordinals.

hat "recursively related notation-systems" are,  and what "constructive ordinals" are, we 
the irregularities, etc. No single scheme, no matter how complex, can name all e ordinals. 
And from this, it follows that no algorithmic method can tell w to apply the method of Gödel to all possible kinds of formal systems. ad unless one is rather mystically inclined, 
therefore one must conclude at any human be ing simply will reach the limits of his own 
ability to 5delize at some point. From ther e on out, formal systems of that complex, 
though admittedly incomplete for the Gödel reason, will have as much power as that human being.  
Other Refutations of Lucas

Other Refutations of Lucas 
 
Now this is only one way to argue agains t Lucas' position. There are others, possibly 
more powerful, which we shall present la ter. But this counterargument has special 
interest because it brings up the fascinating concept trying to create a computer program which can get outside of itself, see itself completely from the outside, and apply the

interest because it brings up the fascinating concept trying to create a computer program which can get outside of itself, see itself completely from the outside, and apply the 
Gödel zapping-trick to itself. Of course this is just as impossible as for a record player to be able to ay records which would cause it to break. But-one should not consider TNT  defective for that reason. If there a defect anywhere, it 
is not in TNT , but in our expectations of what it s hould he able to do. Furthermore, it is 
helpful to realize that we are equally vulnerable to the word trick which Gödel 
transplanted into mathematical formalis ms: the Epimenides paradox. This was quite 
cleverly pointed out

 471by C. H. Whitely, when he proposed the senten ce "Lucas cannot consistently assert this 
sentence." If you think about it, you will see that (1) it is true, and yet (2) Lucas cannot consistently assert it. So Lucas is also "incomplete" with respect to truths about the world. The way in which he mirrors the world in his brain structures prevents him from 
simultaneously being "consistent" and asserting that true sentence. But Lucas is no more 
vulnerable than any of us. He is just on a par with a sophisticated formal system. 
An amusing way to see the incorrectness of  Lucas' argument is to translate it into 
a battle between men and women ... In his wanderings, Loocus the Thinker one day 
comes across an unknown object-a woman. Such a thing he has never seen before, and at 
first he is wondrous thrilled at her likeness to himself: but then, slightly scared of her as 
well, he cries to all the men about him, "Behold! I can look upon her face, which is

well, he cries to all the men about him, "Behold! I can look upon her face, which is 
something she cannot do-therefore women can never be like me!" And thus he proves 
man's superiority over women, much to his relief, and that of his male companions. Incidentally, the same argument proves that Looc us is superior to all other males, as well-
but he doesn't point that out to them. The woman argues back: "Yes, you can see my face, 
which is something I can't do-but I can see your face, which is something you can't do! We're even." However, Loocus comes up with  an unexpected counter: "I'm sorry, you're

which is something I can't do-but I can see your face, which is something you can't do! We're even." However, Loocus comes up with  an unexpected counter: "I'm sorry, you're 
deluded if you think you can see my face. Wh at you women do is not the same as what 
we men do-it is, as I have already pointed out, of an inferior caliber, and does not deserve 
to be called by the same name. You may call it `womanseeing'. Now the fact that you can 
'womansee' my face is of no import, because the situation is not symmetric. You see?" "I 
womansee," womanreplies the woman, and womanwalks away .. . 
Well, this is the kind of "heads-in-the-sand" argument which you have to be

Well, this is the kind of "heads-in-the-sand" argument which you have to be 
willing to stomach if you are bent on seeing men and women running ahead of computers 
in these intellectual battles. 
 
Self-Transcendence-A Modern Myth 
 It is still of great interest to ponder whether we humans ever can jump out of ourselves-or 
whether computer programs can jump out of themselves. Certainly it is possible for a 
program to modify itself-but such modifiability has to be inherent in the program to start 
with, so that cannot be counted as an exampl e of "jumping out of the system". No matter

with, so that cannot be counted as an exampl e of "jumping out of the system". No matter 
how a program twists and turns to get out of itself, it is still following the rules inherent 
in itself. It is no more possible for it to es cape than it is for a human being to decide 
voluntarily not to obey the laws of physics. Physics is an overriding system, from which 
there can be no escape. However, there is a lesser ambition which it is possible to achieve: that is, one can certainly Jump from a subsystem of one's brain into a wider subsystem. One can step out of ruts on occas ion. This is still due to the interaction of

there can be no escape. However, there is a lesser ambition which it is possible to achieve: that is, one can certainly Jump from a subsystem of one's brain into a wider subsystem. One can step out of ruts on occas ion. This is still due to the interaction of 
various subsystems of one’s brain, but it can feel very much like stepping entirely out of 
oneself. Similarly, it is entirely conceivable that a partial ability to “step outside of itself” could be embodied in a computer program.

 472However, it is important to see the distinction between perceiving oneself, and 
transcending oneself. You can gain visions of your self in all sorts of rays-in a mirror, in 
photos or movies, on tape, through the descri ptions if others, by getting psychoanalyzed, 
and so on. But you cannot quite break out of your own skin and be on the outside of 
yourself (modern occult movements, pop psychology fads, etc. notwithstanding). TNT  
can talk about itself, but it cannot jump out of itself. A computer program can modify itself but it cannot violate its own instructions- it can at best change some parts of itself by 
obeying its own instructions. This is reminiscent of the numerous paradoxical question, "Can God make a stone so heavy that he can’t lift it?" 
 
Advertisement and Framing Devices 
 [his drive to jump out of the system is a pe rvasive one, and lies behind all progress in art, 
music, and other human endeavors. It also lies behind such trivial undertakings as the

music, and other human endeavors. It also lies behind such trivial undertakings as the 
making of radio and television commercials. [his insidious trend has been beautifully perceived and described by Irving Goffman in his book Frame Analysis : 
 
For example, an obviously professional actor completes a commercial pitch and, with the camera still on him, turns in obvious relief from his task, now to take real pleasure in consuming the product he had been advertising. 
This is, of course, but one example of the way in which TV and radio 
commercials are coming to exploit frami ng devices to give an appearance of 
naturalness that (it is hoped) will override  the reserve auditors have developed.

naturalness that (it is hoped) will override  the reserve auditors have developed. 
Thus, use is currently being made of ch ildren's voices, presumably because these 
seem unschooled; street noises, and othe r effects to give the impression of 
interviews with unpaid respondents; fa lse starts, filled pauses, byplays, and 
overlapping speech to simulate actual conversation; and, following Welles, the interception of a firm's jingle commercia ls to give news of its new product, 
alternating occasionally with interception by a public interest spot, this presumably 
keeping the faith of the auditor alive.

keeping the faith of the auditor alive. 
The more that auditors withdraw to minor expressive details as a test of 
genuineness, the more that advertisers chas e after them. What results is a sort of 
interaction pollution, a disorder that is also spread by the public relations consultants of political figures, and, more modestly, by micro-sociology.' 
  Here we have yet another example of an escalating " TC-battle"-the antagonists this time 
being Truth and Commercials.   
Simplicio, Salviati, Sagredo: Why Three? 
 There is a fascinating connection between the problem of jumping out of ie system and

 There is a fascinating connection between the problem of jumping out of ie system and 
the quest for complete objectivity. When I read Jauch's four dialogues in Are Quanta Real? based on Galileo's four Dialogues Con cerning Two New Sciences, I found myself

 473wondering  why there were three characters participating. Simplico, Salviati and Sagredo. 
Why wouldn’t two have

 474sufficed: Simplicio, the educated simpleton, and Salviati, the knowledgeable thinker? 
What function does Sagredo have? Well, he is s upposed to be a sort of neutral third party, 
dispassionately weighing the two sides and coming out with a "fair" and "impartial" 
judgment. It sounds very balanced, and ye t there is a problem: Sagredo is always 
agreeing with Salviati, not with Simplicio. How come Objectivity Personified is playing favorites? One answer, of course, is that Salv iati is enunciating correct views, so Sagredo 
has no choice. But what, then, of fairness or "equal time"? 
By adding Sagredo, Galileo (and Jauch) stacked the deck more  against Simplicio, 
rather than less. Perhaps there should be added a yet higher level Sagredo-someone who 
will be objective about this whole situati on ... You can see where it is going. We are 
getting into a never-ending series of "escalati ons in objectivity", which have the curious

getting into a never-ending series of "escalati ons in objectivity", which have the curious 
property of never getting any more objective th an at the first level: where Salviati is 
simply right, and Simplicio wrong. So the puzzl e remains: why add Sagredo at all? And 
the answer is, it gives the illusion of steppi ng out of the system, in some intuitively 
appealing sense. 
 
Zen and "Stepping Out" 
 In Zen, too, we can see this preoccupation with the concept of transcending the system. For instance, the koan in which Tozan tells his monks that "the higher Buddhism is not 
Buddha". Perhaps, self-transcendence is even  the central theme of Zen. A Zen person is

Buddha". Perhaps, self-transcendence is even  the central theme of Zen. A Zen person is 
always trying to understand more deeply what  he is, by stepping more and more out of 
what he sees himself to be, by breaking ev ery rule and convention which he perceives 
himself to be chained by-needless to say, in cluding those of Zen itself. Somewhere along 
this elusive path may come enlightenment. In any case (as I see it), the hope is that by gradually deepening one's self-awareness, by gradually widening the scope of "the 
system", one will in the end come to a feeling of being at one with the entire universe.

Edifying Thoughts of a Tobacco Smoker 
 475Edifying Thoughts 
of a Tobacco Smoker 
 
Achilles has been invited to the Crab's home. 
 
Achilles: I see you have made a few additions since I was last here, Mr. Crab. Your new 
paintings are especially striking. 
Crab: Thank you. I am quite fond of certain painters-especially Rene Magritte. Most of 
the paintings I have are by him. He's my favorite artist. 
Achilles: They are very intriguing images, I must say. In some ways, these paintings by 
Magritte remind me of works by MY favorite artist, M. C. Escher. 
Crab: I can see that. Both Magritte and Escher  use great realism in exploring the worlds 
of paradox and illusion; both have a sure se nse for the evocative power of certain 
visual symbols, and-something which even their admirers often fail to point out-both 
of them have a sense of the graceful line. 
Achilles: Nevertheless, there is something quite different about them. I wonder how one 
could characterize that difference.

could characterize that difference. 
Crab: It would be fascinating to compare the two in detail. Achilles: I must say, Magritte's command of r ealism is astonishing. For instance, I was 
quite taken in by that painting over there of a tree with a giant pipe behind it. 
 
FIGURE 77. The Shadows, by Rene Magritte (1966).

 476Crab: You mean a normal pipe with a tiny tree in front of it! 
Achilles: Oh, is that what it is? Well, in any case, when I first spotted it, I was convinced 
I was smelling pipe smoke! Can you imagine how silly I felt? 
Crab: I quite understand. My guests are often taken in by that one.  
(So saying, he reaches up, removes the pipe  from behind the tree in the painting, 
turns over and taps it against the table, and the room begins to reek of pipe tobacco. 
He begins packing in a new wad of tobacco.) 
 
This is a fine old pipe, Achilles. Belie ve it or not, the bowl has a copper lining, 
which makes it age wonderfully. 
Achilles: A copper lining! You don't say! Crab (pulls out a box of matches, and li ghts his pipe): Would you care for a smoke, 
Achilles? 
Achilles: No, thank you. I only smoke cigars now and then. Crab: No problem! I have one right here ! (Reaches out towards another Magritte 
painting, featuring a bicycle mounted upon a lit cigar.)

Achilles? 
painting, featuring a bicycle mounted upon a lit cigar.) 
Achilles: Uhh-no thank you, not now. Crab: As you will. I myself am an incura ble tobacco smoker. Which reminds me-you 
undoubtedly know of Old Bach's predilection for pipe smoking? 
Achilles: I don't recall exactly. Crab: Old Bach was fond of versifying, philosophizing, pipe smoking, and  
FIGURE 78. State of Grace, by Rene Magritte (1959).

 477music making (not necessarily in that order). He combined all four into a droll poem 
which he set to music. It can be found in the famous musical notebook he kept for his 
wife, Anna Magdalena, and it is called 
 
Edifying Thoughts of a Tobacco Smoker' 
 
Whene'er I take my pipe and stuff it  And smoke to pass the time away, My thoughts, as I sit there and puff it,  Dwell on a picture sad and gray: It teaches me that very like Am I myself unto my pipe.  Like me, this pipe so fragrant burning  Is made of naught but earth and clay; To earth I too shall be returning.  It falls and, ere I'd think to say, It breaks in two before my eyes; In store for me a like fate lies.  No stain the pipe's hue yet cloth darken;  It remains white. Thus do I know That when to death's call I must harken  My body, too, all pale will grow. To black beneath the sod 'twill turn, Likewise the pipe, if oft it burn.  
Or when the pipe is fairly glowing,

Or when the pipe is fairly glowing, 
Behold then, instantaneously,  The smoke off into thin air going, Till naught but ash is left to see. Man's fame likewise away will burn And unto dust his body turn.  How oft it happens when one's smoking: The stopper's missing from its shelf,  And one goes with one's finger poking Into the bowl and burns oneself. If in the pipe such pain cloth dwell, How hot must be the pains of hell.  Thus o'er my pipe, in contemplation Of such things, I can constantly  Indulge in fruitful meditation, And so, puffing contentedly, On land, on sea, at home, abroad I smoke my pipe and worship God.

 478A charming philosophy, is it not? 
Achilles: Indeed. Old Bach was a turner of phrases quite pleasin'. Crab: You took the very words from my m outh. You know, in my time I have tried to 
write clever verses. But I fear mine don't measure up to much. I don't have such a way 
with words. 
Achilles: Oh, come now, Mr. Crab. You have-how  to put it?-quite a penchant for trick'ry 
and teasin'. I'd be honored if you'd sing me one of your songs, Mr. C. 
Crab: I'm most flattered. How about if I play  you a record of myse lf singing one of my 
efforts? I don't remember when it dates fr om. Its title is "A Song Without Time or 
Season". 
Achilles: How poetic!  
(The Crab pulls a record from his shel ves, and walks over to a huge, complex piece 
of apparatus. He opens it up, and inse rts the record into an ominous-looking 
mechanical mouth. Suddenly a bright fl ash of greenish light sweeps over the

Season". 
mechanical mouth. Suddenly a bright fl ash of greenish light sweeps over the 
surface of the record, and afte r a moment, the record is silently whisked into some 
hidden belly of the fantastic machine. A mo ment passes, and then the strains of the 
Crab's voice ring out.) 
 
A turner of phrases quite pleasin', 
Had a penchant for trick'ry and teasin'. 
In his songs, the last line 
Might seem sans design; 
What I mean is, without why or wherefore. 
 Achilles: Lovely! Only, I'm puzzled by one th ing. It seems to me your song, the last line 
is

is 
Crab: Sans design? Achilles: No ... What I mean is, without rhyme or reason. Crab: You could be right. Achilles: Other than that, it's a very nice s ong, but I must say I am even more intrigued 
by this monstrously complex contraption. Is it merely an oversized record player? 
Crab: Oh, no, it's much more than that. This  is my Tortoise-chomping record player. 
Achilles: Good grief! 
Crab: Well, I don't mean that it chomps up Tortoises. But it chomps up records produced 
by Mr. Tortoise.

is 
by Mr. Tortoise. 
Achilles: Whew! That's a little milder. Is this  part of that weird musical battle that 
evolved between you and Mr. T some time ago? 
Crab: In a way. Let me explain a little more fu lly. You see, Mr. Tortoise's sophistication 
had reached the point where he seemed to be able to destroy almost any record player 
I would obtain. 
Achilles:   But when I heard about your rivalr y, it seemed to me you had at last come into 
possession of an invincible phonograph—one with a

 479built-in TV camera, minicomputer and so on, which could take itself apart and rebuild 
itself in such a way that it would not be destroyed. 
Crab: Alack and alas! My plan was foiled. Fo r Mr. Tortoise took advantage of one small 
detail which I had overlooked: the subunit which directed the disassembly and reassembly processes was itself stable duri ng the entire process. That is, for obvious 
reasons, it could not take itself apart and rebuild itself, so it stayed intact. 
Achilles: Yes, but what consequences did that have. Crab: Oh, the direst ones! For you see, Mr . T focused his method down onto that subunit 
entirely. 
Achilles: How is that= Crab: He simply made a record which would induce fatal vibrations in the one structure 
he knew would never change-the disassembly reassembly subunit. 
Achilles: Oh, I see ... Very sneaky. Crab: Yes, so I thought, too. And his stra tegy worked. Not the first time, mind you. I

entirely. 
Achilles: Oh, I see ... Very sneaky. Crab: Yes, so I thought, too. And his stra tegy worked. Not the first time, mind you. I 
thought I had outwitted him when my phonogr aph survived his first onslaught. I 
laughed gleefully. But the next time, he return ed with a steely glint in his eye, and I 
knew he meant business. I placed his new r ecord on my turntable. Then, both of us 
eagerly watched the computer-directed s ubunit carefully scan the grooves. then 
dismount the record, disassemble the record player, reassemble it in an astonishingly 
different way, remount the record-and then sl owly lower the needle into the outermost 
groove.

groove. 
Achilles: Golly! Crab: No sooner had the first strains of sound issued forth than a loud SMASH! filled the 
room. The whole thing fell apart, but partic ularly badly destroyed was the assembler-
disassembler. In that painful instant I finally  realized, to my chagrin, that the Tortoise 
would ALWAYS be able to focus down upon- if you'll pardon the phrase-the Achilles' 
heel of the system. 
Achilles: Upon my soul! You must have felt devastated. Crab: Yes, I felt rather forlorn for a while. Bu t, happily, that was not the end of the story.

Achilles: Upon my soul! You must have felt devastated. Crab: Yes, I felt rather forlorn for a while. Bu t, happily, that was not the end of the story. 
There is a sequel to the tale, which taught  me a valuable lesson, which I may pass on 
to you. On the Tortoise's recommendati on, I was browsing through a curious book 
filled with strange Dialogues about many subjects, including molecular biology, fugues, Zen Buddhism, and heaven knows what else. 
Achilles: Probably some crackpot wrote it. What is the book called:' Crab: If I recall correctly, it was called Copper,  Silver, Gold: an Indestructible Metallic 
Alloy. 
Achilles: Oh, Mr. Tortoise told me about it, t oo. It's by a friend of his, who, it appears, is

Alloy. 
Achilles: Oh, Mr. Tortoise told me about it, t oo. It's by a friend of his, who, it appears, is 
quite taken with metal-logic. 
Crab- I wonder which friend it is ... Anyway_ in one of the Dialogues, I encountered 
some Edifying Thoughts on the Tobacco Mosaic Virus, ribosomes, and other strange things I had never heard of.

 480
 
FIGURE 79. Tobacco Mosaic Virus. 
From A. Lehninger, Biochemistry (New York: Worth Publishers, 1976). 
  Achilles: What is the Tobacco Mosaic Virus? What are ribosomes? Crab: I can't quite say, for I'm a total dunce when it comes to biology. All I know is what 
I gathered from that Dialogue. There, it sa id that Tobacco Mosaic Viruses are tiny 
cigarette-like objects that cause a disease in tobacco plants. 
Achilles: Cancer? Crab: No, not exactly, but Achilles: What next? A tobacco plant smoking and getting cancer! Serves it right! Crab: I believe you've jumped to a hasty conclusion, Achilles. Tobacco plants don't 
SMOKE these "cigarettes". The nasty little "c igarettes" just come and attack them, 
uninvited. 
Achilles: I see. Well, now that I know all about Tobacco Mosaic Viruses, tell me what a 
ribosome is. 
Crab: Ribosomes are apparently some sort of sub cellular entities which take a message 
in one form and convert it into a message in another form.

in one form and convert it into a message in another form. 
Achilles: Something like a teeny tape recorder or phonograph? Crab: Metaphorically, I suppose so. Now the thing which caught my eye was a line where 
this one exceedingly droll character mentions the fact that ribosomes-as well as Tobacco Mosaic Viruses and certain other bizarre biological structures-possess "the 
baffling ability to spontaneously self-assemble, Those were his exact words. 
Achilles: That was one of his droller lines, I take it.

 481Crab: That's just what the other charact er in the Dialogue thought. But that's a 
preposterous interpretation of the statement. ( The Crab draws deeply from his pipe, 
and puffs several billows of smoke into the air.)  
Achilles: Well, what does "spontaneous self-assembly" mean, then? 
Crab: The idea is that when some biological un its inside a cell are taken apart, they can 
spontaneously reassemble themselves-without  being directed by any other unit. The 
pieces just come together, and presto!-they stick. 
Achilles: That sounds like magic. Wouldn't it be  wonderful if a full-sized record player 
could have that property? I mean, if a mi niature "record player" such as a ribosome 
can do it, why not a big one? That would allow you to create an indestructible phonograph, right? Any time it was broken, it would just put itself together again. 
Crab: Exactly my thought. I breathlessly ru shed a letter off to my manufacturer

Crab: Exactly my thought. I breathlessly ru shed a letter off to my manufacturer 
explaining the concept of self-assembly, and asked him if he could build me a record 
player which could take itself apart and spontaneously self-assemble in another form. 
Achilles: A hefty bill to fill. Crab: True; but after several months, he wrot e to me that he had succeeded, at long last-
and indeed he sent me quite a hefty bill.  One fine day, ho! My Grand Self-assembling 
Record Player arrived in the mail, and it was with great confidence that I telephoned

Record Player arrived in the mail, and it was with great confidence that I telephoned 
Mr. Tortoise, and invited him over for th e purpose of testing my ultimate record 
player. 
Achilles: So this magnificent object before us must be the very machine of which you 
speak. 
Crab: I'm afraid not, Achilles. Achilles: Don't tell me that once again ... Crab: What you suspect, my dear friend is unfortunately the case. I don't pretend to 
understand the reasons why. The whole thing is too painful to recount. To see all those 
springs and wires chaotically strewn about on the floor, and puffs of smoke here and 
there-oh, me ...

player. 
speak. 
there-oh, me ... 
Achilles: There, there, Mr. Crab, don't take it too badly. Crab: I'm quite all right; I just have these sp ells every so often. Well, to go on, after Mr. 
Tortoise's initial gloating, he at last rea lized how sorrowful I was feeling, and took 
pity. He tried to comfort me by explaining that it couldn't be helped-it all had to do 
with somebody-or-other's "Theorem", but I couldn't follow a word of it. It sounded 
like "Turtle's Theorem". 
Achilles: I wonder if it was that "Gödel’s Theo rem" which he spoke of once before to me

Achilles: I wonder if it was that "Gödel’s Theo rem" which he spoke of once before to me 
... It has a rather sinister ring to it. Crab: It could be. I don't recall. 
Achilles: I can assure you, Mr. Crab, that I have followed this tale with the utmost 
empathy for your position. It is  truly sad. But, you mentioned that there was a silver 
lining. Pray tell, what was that? 
Crab:  Oh, yes—the silver lining. Well eventually, I abandoned my quest after 
“Perfection” in phonographs, and decided that I might do better

 482to tighten up my defenses against the Tortoise's records. I concluded that a more 
modest aim than a record player which can pl ay anything is simply a record player that 
can SURVIVE: one that will avoid getting de stroyed-even if that means that it can 
only play a few particular records. 
Achilles: So you decided you would develop s ophisticated anti-Tortoise mechanisms at 
the sacrifice of being able to reproduce every possible sound, eh? 
Crab: Well ... I wouldn't exactly say I "decided" it. More accurate would be to say that I 
was FORCED into that position. 
Achilles: Yes, I can see what you mean. Crab: My new idea was to prevent all "a lien" records from being played on my 
phonograph. I knew my own records are harmless,  and so if I prevented anyone else 
from infiltrating THEIR records, that woul d protect my record player, and still allow 
me to enjoy my recorded music.

me to enjoy my recorded music. 
Achilles: An excellent strategy for your new goal. Now does this giant thing before us 
represent your accomplishments to date along those lines? 
Crab: That it does. Mr. Tortoise, of course, ha s realized that he must change HIS strategy, 
as well. His main goal is now to devise a r ecord which can slip past my censors-a new 
type of challenge.

type of challenge. 
Achilles: For your part, how are you planning to keep his and other "alien" records out? Crab: You promise you won't reveal my strategy to Mr. T, now? Achilles: Tortoise's honor. Crab: What!? Achilles: Oh-it's just a phrase I've picked up from Mr. T. Don't worry-I swear your secret 
will remain secret with me. 
Crab: All right, then. My basic plan is to  use a LABELING techni que. To each and every 
one of my records will be attached a secret label. Now the phonograph before you

one of my records will be attached a secret label. Now the phonograph before you 
contains, as did its predecessors, a televi sion camera for scanning the records, and a 
computer for processing the data obtained in the scan and controlling subsequent 
operations. My idea is simply to chomp all records which do not bear the proper label! 
Achilles: Ah, sweet revenge! But it seems to me  that your plan will be easy to foil. All 
Mr. T needs to do is to get a hold of one of your records, and copy its label!

Mr. T needs to do is to get a hold of one of your records, and copy its label! 
Crab: Not so simple, Achilles. What makes you th ink he will be able to tell the label from 
the rest of the record? It may be better integrated than you suspect. 
Achilles: Do you mean that it could be mixed up somehow with the actual music? Crab:  Precisely. But there is a way to disent angle the two. It requires sucking the data off 
the record visually and then--

 483Achilles: Is that what that bright green Hash was for? 
Crab: That's right. That was the TV camer a scanning the grooves. The groove-patterns 
were sent to the minicomputer, which analy zed the musical style of the piece I had put 
on-all in silence. Nothing had been played yet. 
Achilles: Then is there a screening process, which eliminates pieces which aren't in the 
proper styles? 
Crab: You've got it, Achilles. The only r ecords which can pass this second test are 
records of pieces in my own style-and it will be hopelessly difficult for Mr. T to 
imitate that. So you see, I am convinced I will win this new musical battle. However, I 
should mention that Mr. T is equally convinced that somehow, he will manage to slip a record past my censors. 
Achilles: And smash your marvelous machine to smithereens? Crab: Oh, no-he has proved his point on that. No w he just wants to prove to me that he

Achilles: And smash your marvelous machine to smithereens? Crab: Oh, no-he has proved his point on that. No w he just wants to prove to me that he 
can slip a record-an innocuous one-by me, no matter what measures I take to prevent 
it. He keeps on muttering things about songs with strange titles, such as "I Can Be Played on Record Player X". But he can't s care MtE! The only thing that worries me a 
little is that, as before, he seems to have some murky arguments which ... which ... (He trails off into silence. Then, looking quite pensive, he takes a few puffs on his pipe.) 
Achilles: Hmm ... I'd say Mr. Tortoise has an  impossible task on his hands. He's met his 
match, at long last!

match, at long last! 
Crab: Curious that you should think so  ... I don't suppose that you know Henkin's 
Theorem forwards and backwards, do you? 
Achilles: Know WHOSE Theorem forwards and backwards? I've never heard of anything 
that sounds like that. I'm sure it's fascinating, but I'd rather hear more about "music to infiltrate phonographs by". It's an amusing little  story. Actually, I guess I can fill in the 
end. Obviously, Mr. T will find out that there is no point in going on, and so he will 
sheepishly admit defeat, and that will be that. Isn't that exactly it?

sheepishly admit defeat, and that will be that. Isn't that exactly it? 
Crab: That's what I'm hoping, at least. Would you like to see a little bit of the inner 
workings of my defensive phonograph? 
Achilles: Gladly. I've always wanted to see a working television camera. Crab: No sooner said than done, my friend. (R eaches into the gaping"mouth" of the large 
phonograph, undoes a couple of snaps, and pulls out a neatly packaged instrument.) You see, the whole thing is built of indepe ndent modules, which can be detached and 
used independently. This TV camera, for instance, works very well by itself. Watch

used independently. This TV camera, for instance, works very well by itself. Watch 
the screen over there, beneath the painting with the flaming tuba. (He points the camera at Achilles, whose face instantly appears on the large screen.) 
Achilles: Terrific! May I try it out? Crab: Certainly. 
Achilles: (pointing the camera at the Crab. There YOU are, Mt Crab, on the screen.

 484
 
FIGURE 80. The Fair Captive, by Rene Magritte (1947).  
 
Crab: So I am. Achilles: Suppose I point the camera at the painting with the burning tuba. Now it is on 
the screen, too! 
Crab: The camera can zoom in and out, Achilles. You ought to try it. Achilles: Fabulous! 
Let me just focus down onto the tip of those flames, where they meet the picture frame 
... It's such a funny feeling to  be able to instantaneously "copy" anything in the room-
anything I want-onto that screen. I merely need to point the camera at it, and it pops 
like magic onto the screen. 
Crab: ANYTHING in the room, Achilles? Achilles: Anything in sight, yes. That's 
obvious. 
Crab: What happens, then, if you point the camera at the flames on the TV screen?  
(Achilles shifts the camera so that it points directly at that part of the television 
screen on which the flames are-or were-displayed.) 
 
Achilles: Hey, that's funny! That very act  makes the flames DISAPPEAR from the

obvious. 
 
Achilles: Hey, that's funny! That very act  makes the flames DISAPPEAR from the 
screen! Where did they go? 
Crab: You can't keep an image still on the screen and move the camera at the same time. Achilles: So I see… But I don’t understand wh at’s on the screen now—not at all! It 
seems to be a strange long corridor. Yet I’m certainly not

 485
 
FIGURE 81. Twelve self-engulfing TV screens. I would have included one more, had 13 
not been prime

 487pointing the camera down any corridor. I'm me rely pointing it at an ordinary TV 
screen. 
Crab: Look more carefully, Achilles. Do you really see a corridor? Achilles: Ahhh, now I see. It's a set of nest ed copies of the TV screen itself, getting 
smaller and smaller and smaller ... Of course! The image of the flames HAD to go away, because it came from my- pointing th e camera at the PAINTING. When I point 
the camera at the SCREEN, then the screen itself appears, with whatever is on the 
screen at the time which is the screen itself,  with whatever is on the screen at the time 
which is the screen itself, with 
Crab: I believe I can fill in the rest, Achilles. Why- don't you try rotating the camera? Achilles: Oh! I get a beautiful spiraling corrido r! Each screen is rotated inside its framing 
screen, so that the littler they get, the more  rotated they are, with respect. to the 
outermost screen. This idea of having a TV screen "engulf itself" is weird.

outermost screen. This idea of having a TV screen "engulf itself" is weird. 
Crab: What do you mean by "self-engulfing", Achilles? Achilles: I mean, when I point the camera at th e screen-or at part of the screen. THAT'S 
self-engulfing. 
Crab: Do you mind if I pursue that a little further? I'm intrigued by this new notion. Achilles: So am I. Crab: Very well, then. If you point the camera at a CORNER of the screen, is that still 
what you mean by "self-engulfing"? 
Achilles: Let me try it. Hmm-the "corridor" of  screens seems to go off the edge, so there

Achilles: Let me try it. Hmm-the "corridor" of  screens seems to go off the edge, so there 
isn't an infinite nesting any more. It's prett y, but it doesn't seem to me to have the spirit 
of self-engulfing. It's a "failed self-engulfing". 
Crab: If you were to swing the TV camera b ack towards the center of the screen, maybe 
you could fix it up again ... 
Achilles (slowly and cautiously turning the camera): Yes! The corridor is getting longer 
and longer ... There it is! Now it's all back. I can look down it so far that it vanishes in 
the distance. The corridor became infinite again precisely at the moment when the

the distance. The corridor became infinite again precisely at the moment when the 
camera took in the WHOLE screen. Hmm-that reminds me of something Mr. Tortoise 
was saying a while back, about self-refere nce only occurring when a sentence talks 
about ALL of itself ... 
Crab: Pardon me? Achilles: Oh, nothing just muttering to myself.  
(As Achilles plays with the lens and other controls on the camera, a profusion of new kinds of self-engulfing images appear: swirling spirals that resemble galaxies, kaleidoscopic flower-like shapes, and other assorted patterns ...)

(As Achilles plays with the lens and other controls on the camera, a profusion of new kinds of self-engulfing images appear: swirling spirals that resemble galaxies, kaleidoscopic flower-like shapes, and other assorted patterns ...) 
 Crab:  You seem to be having a grand time. Achilles: (turns away from the camera); I’ll say! What a wealth of images this simple 
idea can produce! (He glances back at the screen, and a look of

 488astonishment crosses his face.) Good grief, Mr. Crab! There's a pulsating petal-pattern 
on the screen! Where do the pulsations come from? The TV is still, and so is the 
camera. 
Crab: You can occasionally set up patterns which change in time. This is because there is 
a slight delay in the circuitry between the moment the camera "sees" something, and 
the moment it appears on the screen around a hundredth of a second. So if you have a 
nesting of depth fifty or so, roughly a ha lf-second delay will re sult. If somehow a 
moving image gets onto the screen-for exam ple, by you putting your finger in front of 
the camera-then it takes a while for the more deeply nested screens to "find out" about 
it. This delay then reverberates through th e whole system, like a visual echo. And if 
things are set up so the echo doesn't die away, then you can get pulsating patterns. 
Achilles: Amazing! Say-what if we tr ied to make a TOTAL self-engulfing?

Achilles: Amazing! Say-what if we tr ied to make a TOTAL self-engulfing? 
Crab: What precisely do you mean by that? Achilles: Well, it seems to me that this stuff with screens within screens is interesting, but 
I'd like to get a picture of the TV camer a AND the screen, ON the screen. Only then 
would I really have made the system engulf itself. For the screen is only PART of the 
total system. 
Crab: I see what you mean. Perhaps with this mirror, you can achieve the effect you 
want. 
 
(The Crab hands him a mirror, and Achilles m aneuvers the mirror and camera in such

want. 
 
(The Crab hands him a mirror, and Achilles m aneuvers the mirror and camera in such 
a way that the camera and the screen are both pictured on the screen.) 
 
Achilles: There! I've created a TOTAL self-engulfing! Crab: It seems to me you only have the front  of the mirror-what about its back? If it 
weren't for the back of the mirror, it woul dn't be reflective-and you wouldn't have the 
camera in the picture. 
Achilles: You're right. But to show both the fr ont and back of this mirror, I need a second 
mirror.

mirror. 
Crab: But then you'll need to show the back of that mirror, too. And what about including 
the back of the television, as well as its fr ont? And then there's the electric cord, and 
the inside of the television, and 
Achilles: Whoa, whoa! My head's beginning to  spin! I can see that this "total self-
engulfing project" is going to pose a wee bit of a problem. I'm feeling a little dizzy. 
Crab: I know exactly how you feel. Why don't you sit down here and take your mind off 
all this self-engulfing? Relax! Look at my paintings, and you'll calm down.

all this self-engulfing? Relax! Look at my paintings, and you'll calm down. 
(Achilles lies down, and sighs.) Oh-perhaps my pipe smoke is bothering you? Here, I'll put my pipe away. ( Takes the pipe 
from his mouth, and carefully places it above some written words in another .Magritte 
painting. ) There! Feeling any better? 
Achilles: I’m still a little woozy, (Points at the Magritte.) That’s an interesting painting. I 
like the way it’s framed, especially the shiny inlay inside the wooden frame.

 489
 
FIGURE 82. The Air and the Song, by Rene Magritte (1964). 
 Crab: Thank you. I had it specially done-it's a gold lining. Achilles: A gold lining? What next? What are those words below the pipe? They aren't in 
English, are they? 
Crab: No, they are in French. They say, "Ceci n'est pas une pipe." That means, "This is 
not a pipe". Which is perfectly true. 4chilles: But it is a pipe! You were just smoking it! 
Crab: Oh, you misunderstand the phrase, I believe . The word "ceci" refers to the painting, 
not to the pipe. Of course the pipe is a pipe. But a painting is not a pipe. 
Achilles: I wonder if that "ceci" inside the pa inting refers to the WHOLE painting, or just 
to the pipe inside the painting. Oh, my  gracious! That would be ANOTHER self-
engulfing! I'm not feeling at all well, Mr. Crab. I think I'm going to be sick ...

Self-Rep and Self-Rep 490CHAPTER XVI 
 
Self-Ref and Self-Rep 
 
IN THIS CHAPTER, we will look at some of the mechanisms which create self-reference in various contexts, and compare them to the mechanisms which allow some kinds of systems 
to reproduce themselves. Some remarkable and beautiful parallels between these mechanisms 
will come to light. 
 
Implicitly and Explicitly Self-Referential Sentences 
 
To begin with, let us look at sentences which, at first glance, may seem to provide the 
simplest examples of self-reference. Some such sentences are these: 
 
(1) This sentence contains five words. (2) This sentence is meaningless because it is self-referential. 
(3) This sentence no verb. 
(4) This sentence is false. (Epimenides paradox) (5) The sentence I am now writing is the sentence you are now reading. 
 
All but the last one (which is an anomaly) involve the simple-seeming mechanism contained 
in the phrase "this sentence". But that mechanism is in reality far from simple. All of these

in the phrase "this sentence". But that mechanism is in reality far from simple. All of these 
sentences are "floating" in the context of the English language. They can be compared to icebergs, whose tips only are visible. The word sequences are the tips of the icebergs, and the 
processing which must be done to understand them is the hidden part. In this sense their 
meaning is implicit, not explicit. Of course, no sentence's meaning is completely explicit, but 
the more explicit the self-reference is, the more exposed will be the mechanisms underlying

the more explicit the self-reference is, the more exposed will be the mechanisms underlying 
it. In this case, for the self-reference of the sentences above to be recognized, not only has one to be comfortable with a language such as English which can deal with linguistic subject 
matter, but also one has to be able to figure out the referent of the phrase "this sentence". It 
seems simple, but it depends on our very complex yet totally assimilated ability to handle 
English. What is especially important here is the ability to figure out the referent of a noun

English. What is especially important here is the ability to figure out the referent of a noun 
phrase with a demonstrative adjective in it. This ability is built up slowly, and should by no 
means be considered trivial. The difficulty is perhaps underlined when a sentence such as 
number 4 is presented to someone naive about paradoxes and linguistic tricks, such as a 
child. They may say, "What sentence is false and it may take a bit of persistence to get across 
the idea that the sentence is talking about itself. The whole idea is a little mind

Self-Rep and Self-Rep 491boggling at first. A couple of pictures may help (Figs. 83, 84). Figure 83 is a picture which 
can be interpreted on two levels. On one level, it is a sentence pointing at itself; on the other 
level, it is a picture of Epimenides executing his own death sentence. 
 
.  
 
    Figure 84, showing visible and invisible portions of the iceberg, suggests the relative proportion of sentence to processing required for the recognition of self-reference: 
     
It is amusing to try to create a self-referring sentence without using the trick of saving 
this sentence". One could try to quote a sentence inside itself. Here is an attempt: 
 
The sentence "The sentence contains five words" contains five words. 
 But such an attempt must fail, for any sentence that could be quoted entirely inside itself 
would have to be shorter than itself. This is actually possible, but only if you are willing to 
entertain infinitely long sentences, such as:

Self-Rep and Self-Rep 492The sentence 
"The sentence 
"The sentence  
 …  
     … .   
etc,.,etc. 
   … 
  …. 
     is infinitely long"  
     is infinitely long"  is infinitely long' 
is infinitely long. 
 
But this cannot work for finite sentences. For the same reason, Godel's string G could not 
contain the explicit numeral for its Godel number: it would not fit. No string of TNT can contain the TNT-numeral for its own Godel number, for that numeral always contains more 
symbols than the string itself does. But you can get around this by having G contain a 
description  of its own Godel number, by means of the notions of "sub" and 
"arithmoquinification". 
One way of achieving self-reference in an English sentence by means of description 
instead of by self-quoting or using the phrase "this sentence" is the Quine method, illustrated 
in the dialogue Air on G's String . The understanding of the Quine sentence requires less

in the dialogue Air on G's String . The understanding of the Quine sentence requires less 
subtle mental processing than the four examples cited earlier. Although it may appear at first 
to be trickier, it is in some ways more explicit. The Quine construction is quite like the Godel 
construction, in the way that it creates self-reference by describing another typographical entity which, as it turns out, is isomorphic to the Quine sentence itself. The description of the 
new typographical entity is carried out by two parts of the Quine sentence. One part is a set

new typographical entity is carried out by two parts of the Quine sentence. One part is a set 
of instructions telling how to build a certain phrase, while the other part contains the 
construction materials to be used; that is, the other part is a template . This resembles a 
floating cake of soap more than it resembles an iceberg (See Fig. 85).

Self-Rep and Self-Rep 493The self-reference of this sentence is achieved in a more direct way than in the Epimenides 
paradox; less hidden processing is needed. By the way, it is interesting to point out that the 
phrase "this sentence" appears in the previous sentence; yet it is not there to cause self-
reference: you probably understood that its referent was the Quine sentence, rather than the 
sentence in which it occurs. This just goes to show how pointer phrases such as "this 
sentence" are interpreted according to context, and helps to show that the processing of such phrases is indeed quite involved. 
 
A Self-Reproducing Program 
 
The notion of quining, and its usage in creating self-reference, have already been explained inside the Dialogue itself, so we need not dwell on such matters here. Let us instead show 
how a computer program can use precisely the same technique to reproduce itself. The 
following selfreproducing program is written in a BlooP-like language and is based on

following selfreproducing program is written in a BlooP-like language and is based on 
following a phrase by its own quotation (the opposite order from quining, so I reverse the 
name "quine" to make "eniuq"):  
DEFINE PROCEDURE "ENIUQ" [TEMPLATE]: PRINT [TEMPLATE, LEFT-
BRACKET, QUOTE-MARK, TEMPLATE, QUOTE-MARK, RIGHT-BRACKET, PERIOD].  
ENIUQ 
['DEFINE PROCEDURE "ENIUQ" [TEMPLATE]: PRINT [TEMPLATE, LEFT-BRACKET, QUOTE-MARK, TEMPLATE, QUOTE-MARK, RIGHT-BRACKET, PERIOD]. ENIUQ']. 
 
ENIUQ  is a procedure defined in the first two lines, and its input is called " TEMPLATE ". It 
is understood that when the procedure is called, TEMPLATE's  value will be some string of

ENIUQ 
 
is understood that when the procedure is called, TEMPLATE's  value will be some string of 
typographical characters. The effect of ENIUQ  is to carry out a printing operation, in which 
TEMPLATE gets printed twice: the first time just  plain; the second time wrapped in (single) 
quotes and brackets, and garnished with a final period. Thus, if TEMPLATE's  value were 
the string DOUBLE-BUBBLE , then performing ENIUQ  on it would yield: 
 
DOUBLE-BUBBLE ['DOUBLE-BUBBLE']. 
 
Now in the last four lines of the program above, the procedure ENIUQ  is called with a

Now in the last four lines of the program above, the procedure ENIUQ  is called with a 
specific value of TEMPLATE -namely the long string inside the single quotes: DEFINE  ... 
ENIUQ . That value has been carefully chosen; it consists of the definition of ENIUQ , 
followed by the word ENIUQ . This makes the program itself-or, if you prefer, a perfect copy 
of it-get printed out. It is very similar to Quine's version of the Epimenides sentence: 
 
"yields falsehood when preceded by its quotation" 
yields falsehood when preceded by its quotation.

yields falsehood when preceded by its quotation. 
 
It is very important to realize that the character string which appears n quotes in the last three 
lines of the program above-that is, the value of

Self-Rep and Self-Rep 494 
 
TEMPLATE -is never interpreted as a sequence of instructions. That it happens to be one is, 
in a sense. just an accident. As was pointed out above, it could just as well have been 
DOUBLE-BUBBLE  or any other string of characters. The beauty of the scheme is that when 
the same string appears in the top two lines of this program, it is treated as a program (because it is not in quotes). Thus in this progr am, one string functions in two ways: first as 
program, and second as data. This is the secret of self-reproducing programs, and, as we shall 
see, of self-reproducing molecules. It is useful, incidentally, to call any kind of self-
•reproducing object or entity a self-rep ; and likewise to call any self-referring object or entity 
a self-ref . I will use those terms occasionally from here on. 
The preceding program is an elegant example of a self-reproducing program written 
in a language which was not designed to make the writing of self-reps particularly easy.

in a language which was not designed to make the writing of self-reps particularly easy. 
Thus, the task had to be carried out using those notions and operations which were assumed 
to be part of the language-such as the word QUOTE-MARK , and the command PRINT . But 
suppose a language were designed expressly for making self-reps easy to write. Then one could write much shorter self-reps. For example, suppose that the operation of eniuq-ing 
were a built-in feature of the language, needing no explicit definition (as we assumed PRINT 
was). Then a teeny self-rep would be this: 
 
ENIUQ ['ENIUQ'].

ENIUQ ['ENIUQ']. 
 
It is very similar to the Tortoise's version of Quine's version of the Epimenides self-ref, 
where the verb "to quine" is assumed to be known: 
 
"yields falsehood when quined" yields falsehood when quined 
 
But self-reps can be even shorter. For instance, in some computer language it might 
be a convention that any program whose first symbol is an asterisk is to be copied before 
being executed normally. Then the program consis ting of merely one asterisk is a self-rep!

being executed normally. Then the program consis ting of merely one asterisk is a self-rep! 
You may complain that this is silly and depends on a totally arbitrary convention. In doing so, you are echoing my earlier point that it is almost cheating to use the phrase "this 
sentence" to achieve self-reference-it relies too much on the processor, and not enough on 
explicit directions for self-reference. Using an asterisk as an example of a self-rep is like 
using the word "I" as an example of a self-ref: both conceal all the interesting aspects of their 
respective problems.

respective problems. 
This is reminiscent of another curious type of self-reproduction: via photocopy 
machine. It might be claimed that any written document is a self-rep because it can cause a 
copy of itself to be printed when it is placed in a photocopy machine and the appropriate 
button is pushed. But somehow this violates our notion of self-reproduction; the piece of

button is pushed. But somehow this violates our notion of self-reproduction; the piece of 
paper is not consulted at all, and is therefore not directing its own reproduction. Again, everything is in the processor. Before we call something a self-rep, we want to have the 
feeling that, to the maximum extent possible, it explicitly contains the directions for copying 
itself.

Self-Rep and Self-Rep 495 
    
To be sure, explicitness is a matter of degree; nonetheless there is an intuitive borderline on 
one side of which we perceive true self-dir ected self-reproduction, and on the other side of 
which we merely see copying being carried out by an inflexible and autonomous copying 
machine.  
What Is a Copy? 
 
Now in any discussion of self-refs and self-reps, one must sooner or later come to grips with 
the essential issue: what is a copy? We already dealt with that question quite seriously in Chapters V and VI; and now we come back to it. To give the flavor of the issue, let us 
describe some highly fanciful, yet plausible, examples of self-reps. 
 
A Self-Reproducing Song 
 
Imagine that there is a nickelodeon in the local bar which, if you press buttons 11-U, will 
play a song whose lyrics go this way: 
 
Put another nickel in, in the nickelodeon, 
All I want is 11-U, and music, music, music. 
 
We could make a little diagram of what happens one evening (Fig. 86).

We could make a little diagram of what happens one evening (Fig. 86). 
 
FIGURE 86. A self-reproducing song. 
 
Although the effect is that the song reproduces its elf, it would feel strange to call the song a 
self-rep, because of the fact that when it passes through the 11-U stage, not all of the 
information is there. The information only gets put back by virtue of the fact that it is fully stored in the nickelodeon that is, in one of the arrows in the diagram, not in one of the ovals. 
It is questionable whether this song contains a complete description of how to get itself 
played again, because the symbol pair "1 1-U" is only a trigger, not a copy. 
 
A "Crab" Program

A "Crab" Program 
 
Consider next a computer program which prints itself out backwards. (Some readers might 
enjoy thinking about how to write such a program in

Self-Rep and Self-Rep 496 
 
the blooP-like language above, using the given sell-rep as a inouel.) vvouiu this funny 
program count as a self-rep. Yes, in a way, because a trivial transformation performed on its 
output will restore the original program. It seems fair to say that the output contains the same information as the program itself, just recast in a simple way. Yet it is clear that someone 
might look at the output and not recognize it as a program printed backwards. To recall 
terminology from Chapter VI, we could say that the "inner messages" of the output and the 
program itself are the same, but they have different "outer messages"-that -is, they must be 
read by using different decoding mechanisms. Now if one counts the outer message as part of the information-which seems quite reasonable-then the total information is not the same after 
all, so the program can't be counted as a self-rep. 
However, this is a disquieting conclusion, because we are accustomed to considering

However, this is a disquieting conclusion, because we are accustomed to considering 
something and its mirror image as containing the same information. But recall that in Chapter 
VI, we made the concept of "intrinsic meaning" dependent on a hypothesized universal notion of intelligence. The idea was that, in determining the intrinsic meaning of an object, 
we could disregard some types of outer message-those which would be universally 
understood. That is, if the decoding mechanism seems fundamental enough, in some still ill-

defined sense, then the inner message which it lets be revealed is the only meaning that 
counts. In this example, it seems reasonably safe to guess that a "standard intelligence" would consider two mirror images to contain the same information as each other; that is, it would 
consider the isomorphism between the two to be so trivial as to be ignorable. And thus our 
intuition that the program is in some sense a fair self-rep, is allowed to stand. 
 
Epimenides Straddles the Channel

Epimenides Straddles the Channel 
 
Now another far-fetched example of a self-rep would be a program which prints itself our, 
but translated into a different computer language. One might liken this to the following 
curious version of the Quine version of the Epimenides self-ref: 
 
'lest une expression qui, quand elle est precedee de sa traduction, mise entre 
guillemets, clans la langue provenant de l'autre tote de la Manche. tree une faussete" 
is an expression which, when it is preceded by its translation, placed in quotation

is an expression which, when it is preceded by its translation, placed in quotation 
marks, into the language originating on the other side of the Channel, yields a 
falsehood. 
 
You might try to write down the sentence which is described by this weird concoction. (Hint: 
It is not itself-or at least it is not if "itself" is taken in a naive sense.) If the notion of "self-rep 
by retrograde motion" (i.e., a program which writes itself out backwards) is reminiscent of a

by retrograde motion" (i.e., a program which writes itself out backwards) is reminiscent of a 
crab canon, the notion of "self-rep by translation" is no less reminiscent of "a canon which involves a transposition of the theme into another key.

Self-Rep and Self-Rep 497 
 
A Program That Prints Out Its Own Godel Number 
 
The idea of printing out a translation instead of an exact copy of the original program may 
seem pointless. However, if you wanted to write a self-rep program in BlooP or FlooP, you 
would have to resort to some such device, for in those languages, OUTPUT  is always a 
number, rather than a typographical string. Therefore, you would have to make the program 
print out its own Godel number: a very huge integer whose decimal expansion codes for the 
program, character by character, by using three digit codons. The program is coming as close as it can to printing itself, within the means av ailable to it: it prints out a copy of itself in 
another "space", and it is easy to switch back and forth between the space of integers and the 
space of strings. Thus, the value of OUTPUT  is not a mere trigger, like "11-12". Instead, all 
the information of the original program lies "close to the surface" of the output.

the information of the original program lies "close to the surface" of the output. 
 
Godelian Self-Reference 
 
This comes very close to describing the mechanis m of Godel's self-ref G. After all, that string 
of TNT contains a description not of itself, but of an integer (the arithmoquinification of u). It 
just so happens that that integer is an exact "image" of the string G, in the space of natural numbers. Thus, G refers to a translation of itself into another space. We still feel comfortable 
in calling G a self-referential string, because the isomorphism between the two spaces is so

in calling G a self-referential string, because the isomorphism between the two spaces is so 
tight that we can consider them to be identical. 
This isomorphism that mirrors TNT inside the abstract realm of natural numbers can 
be likened to the quasi-isomorphism that mirrors the real world inside our brains, by means of symbols. The symbols play quasi-isomorphic roles to the objects, and it is thanks to them 
that we can think. Likewise, the Godel numbers play isomorphic roles to strings, and it is 
thanks to them that we can find metamathematical meanings in statements about natural

thanks to them that we can find metamathematical meanings in statements about natural 
numbers. The amazing, nearly magical, thing about G is that it manages to achieve self-
reference despite the fact that the language in which it is written, TNT, seems to offer no hope of referring to its own structures, unlike English, in which it is the easiest thing in the 
world to discuss the English language. 
So G is an outstanding example of a self-ref via translation-hardly the most 
straightforward case. One might also think back to some of the Dialogues, for some of them,

straightforward case. One might also think back to some of the Dialogues, for some of them, 
too, are self-refs via translation. For instance, take the Sonata for Unaccompanied Achilles . 
In that Dialogue, several references are made to the Bach Sonatas for unaccompanied violin, 
and the Tortoise's suggestion of imagining harpsichord accompaniments is particularly 
interesting. After all, if one applies this idea to the Dialogue itself, one invents lines which 
the Tortoise is saying; but if one assumes that Achilles' part stands alone (as does the violin),

the Tortoise is saying; but if one assumes that Achilles' part stands alone (as does the violin), 
then it is quite wrong to attribute any lines at all to the Tortoise. In any case, here again is a self-ref by means of a mapping which maps Dialogues onto pieces by Bach. And this 
mapping is

Self-Rep and Self-Rep 498left, of course, for the reader to notice. Yet even if the reader does not notice it, the mapping 
is still there, and the Dialogue is still a self-ref. 
 
A Self-Rep by Augmentation 
 
We have been likening self-reps to canons. What, then, would be a fair analogue to a canon by augmentation? Here is a possibility: consider a program which contains a dummy loop 
whose only purpose is to slow up the program. A parameter might tell how often to repeat the 
loop. A self-rep could be made which prints out a copy of itself, but with the parameter 
changed, so that when that copy is run, it will run at half the speed of its parent program; and 
its "daughter" will in turn run at half again the speed, and so on . . . None of these programs prints itself out precisely; yet all clearly belong to a single "family". 
This is reminiscent of the self-reproduction of living organisms. Clearly, an 
individual is never identical to either of its parents; why, then, is the act of making young

individual is never identical to either of its parents; why, then, is the act of making young 
called "self-reproduction'? The answer is that there is a coarse-grained isomorphism between 
parent and child; it is an isomorphism which preserves the information about species. Thus, what is reproduced is the class, rather than the instance. This is also the case in the recursive 
picture Gplot, in Chapter V: that is, the mapping between "magnetic butterflies" of various 
sizes and shapes is coarse-grained; no two are identical, but they all belong to a single 
"species", and the mapping preserves precisely that fact. In terms of self-replicating

"species", and the mapping preserves precisely that fact. In terms of self-replicating 
programs, this would correspond to a family of programs, all written in "dialects" of a single computer language; each one can write itself out, but slightly modified, so that it comes out 
in a dialect of its original language. 
 
A Kimian Self-Rep 
 
 Perhaps the sneakiest example of a self-rep is the following: instead of writing a legal 
expression in the compiler language, you type one of the compiler's own error messages. 
When the compiler looks at your "program", the first thing it does is get confused, because

When the compiler looks at your "program", the first thing it does is get confused, because 
your "program" is ungrammatical; hence the compiler prints out an error message. All you 
need to do is arrange that the one it prints out will be the one you typed in. This kind of self-rep, suggested to me by Scott Kim, exploits a different level of the system from the one you 
would normally approach. Although it may seem frivolous, it may have counterparts in 
complex systems where self-reps vie against each other for survival, as we shall soon discuss. 
 
What Is the Original? 
 
Besides the question "What constitutes a copy?", there is another fundamental philosophical 
question concerning self-reps. That is the obverse

Self-Rep and Self-Rep 499side of the coin: "What is the original?" This can best be explained by referring to some 
examples: 
 
 (1) a program which, when interpreted by some interpreter running on some 
computer, prints itself out; 
 (2) a program which, when interpreted by some interpreter running on some 
computer. prints itself out along with a complete copy of the interpreter (which, 
after all, is also a program); 
 
(3) a program which, when interpreted by some interpreter running on some computer, not only prints itself out along with a complete copy of the interpreter, 
but also directs a mechanical assembly process in which a second computer, 
identical to the one on which the interpreter and program are running, is put 
together. 
 It is clear that in (1), the program is the self-rep. But in (3), is it the program which is the self-
rep, or the compound system of program plus interpreter, or the union of program, 
interpreter, and processor?

together. 
interpreter, and processor? 
Clearly, a self-rep can involve more than just printing itself out. In fact, most of the 
rest of this Chapter is a discussion of self-reps in which data, program, interpreter, and processor are all extremely intertwined, and in which self-replication involves replicating all 
of them at once. 
 
Typogenetics 
 
We are now about to broach one of the most fascinating and profound topics of the twentieth 
century: the study of "the molecular logic of the living state", to borrow Albert Lehninger's 
richly evocative phrase. And logic it is, too but of 'a sort more complex and beautiful than

richly evocative phrase. And logic it is, too but of 'a sort more complex and beautiful than 
any a human mind ever imagined. We will come at it from a slightly novel angle: via an 
artificial solitaire game which I call Typogenetics-short for "Typographical Genetics". In Typogenetics I have tried to capture some ideas of molecular genetics in a typographical 
system which, on first sight, resembles very much the formal systems exemplified by the 
MIU -system. Of course, Typogenetics involves many simplifications, and therefore is useful 
primarily for didactic purposes.

primarily for didactic purposes. 
I should explain immediately that the field of molecular biology is a field in which 
phenomena on several levels interact, and that Typogenetics is only trying to illustrate 
phenomena from one or two levels. In particular, purely chemical aspects have been 
completely avoided-they belong to a level lower than is here dealt with; similarly, all aspects 
of classical genetics (viz., nonmolecular genetics) have also been avoided-they belong to a

of classical genetics (viz., nonmolecular genetics) have also been avoided-they belong to a 
level higher than is here dealt with. I have intended in Typogenetics only to give an intuition for those processes centered on the celebrated Central Dogma of

Self-Rep and Self-Rep 500Molecular Biology,  enunciated by Francis Crick (one of the co-discoverers of the double-
helix structure of DNA): 
 
DNA  =>  RNA  =>  proteins. 
 
It is my hope that with this very skeletal model I have constructed the reader will perceive some simple unifying principles of the field principles which might otherwise be obscured by 
the enormously intricate interplay of phenomena at many different levels. What is sacrificed 
is, of course, strict accuracy; what is gained is, I hope, a little insight. 
 
Strands, Bases, Enzymes 
 
The game of Typogenetics involves typographical manipulation on sequences of letters. 
There are four letters involved: 
 
A C G T. 
 
Arbitrary sequences of them are called strands . Thus, some strands are: 
 
GGGG 
ATTACCA 
CATCATCATCAT 
 
Incidentally, " STRAND " spelled backwards begins with " DNA ". This is appropriate since 
strands, in Typogenetics, play the role of pieces of DNA (which, in real genetics, are often

A C G T. 
 
GGGG 
ATTACCA 
 
strands, in Typogenetics, play the role of pieces of DNA (which, in real genetics, are often 
called "strands"). Not only this, but " STRAND " fully spelled out backwards is " DNA RTS ", 
which may be taken as an acronym for " DNA Rapid Transit Service". This, too, is 
appropriate, for the function of "messenger RNA "-which in Typogenetics is represented by 
strands as well-is quite well characterized by the phrase " Rapid Transit Service" for DNA , as 
we shall see later. 
I will sometimes refer to the letters A, C, G, T  as bases, and to the positions which

I will sometimes refer to the letters A, C, G, T  as bases, and to the positions which 
they occupy as units. Thus, in the middle strand, there are seven units, in the fourth of which 
is found the base A. 
If you have a strand, you can operate on it and change it in various ways. You can 
also produce additional strands, either by copying, or by cutting a strand in two. Some 
operations lengthen strands, some shorten them, and some leave their length alone. 
Operations come in packets-that is, several to be performed together, in order. Such a

Operations come in packets-that is, several to be performed together, in order. Such a 
packet of operations is a little like a programmed machine which moves up and down the 
strand doing things to it. These mobile machines are called "typographical enzymes"-
enzymes for short. Enzymes operate on strands one unit at a time, and are said to be "bound" 
to the unit they are operating on at any given moment. 
I will show how some sample enzymes act on particular strings. The first thing to 
know is that each enzyme likes to start out bound to a particular letter. Thus, there are four 
kinds of enzyme-those which prefer

Self-Rep and Self-Rep 501 
    
A, those which prefer C, etc. Given the sequence of operations which an enzyme performs, 
you can figure out which letter it prefers, but for now I'll just give them without explanation. 
Here's a sample enzyme, consisting of three operations:  
(1) Delete the unit to which the enzyme is bound (and then bind to the next unit to the 
right). 
(2) Move one unit to the right. 
(3) Insert a T (to the immediate right of this unit). 
 
This enzyme happens to like to bind to A initially. And here's a sample strand: 
 
ACA 
 
What happens if our enzyme binds to the left A and begins acting? Step I deletes the A, so we 
are left with CA-and the enzyme is now bound to the C. Step 2 slides the enzyme rightwards, 
to the A, and Step 3 appends a T onto the end to form the strand CAT . And the enzyme has 
done its complete duty: it has transformed ACA  into CAT .

ACA 
 
done its complete duty: it has transformed ACA  into CAT . 
  What if it had bound itself to the right A of ACA? It would have deleted that A and moved off the end of the strand. Whenever this happens, the enzyme quits (this is a general 
principle). So the entire effect would just be to lop off one symbol. 
Let's see some more examples. Here is another enzyme: 
 
(1) Search for the nearest pyrimidine to the right of this unit.  (2) Go into Copy mode. 
(3) Search for the nearest purine to the right of this unit.  
(4) Cut the strand here (viz., to the right of the present unit).

(4) Cut the strand here (viz., to the right of the present unit). 
 
Now this contains the terms "pyrimidine" and "purine". They are easy terms. A and G are 
called purines , and C and T are called pyrimidines . So searching for a pyrimidine merely 
means searching for the nearest C or T. 
 
Copy Mode and Double Strands 
 
The other new term is Copy mode . Any strand can be "copied" onto another strand, but in a 
funny way. Instead of copying A onto A, you copy it onto T, and vice versa. And instead of 
copying C onto C, you copy it onto G, and vice versa. Note that a purine copies onto a 
pyrimidine, and vice versa. This is called complementary base pairing . The complements are 
shown below  
  Complement

  Complement 
Purinas            A <====>  T    pyrimidines 
            G <====> C

Self-Rep and Self-Rep 502You can perhaps remember this molecular pairing scheme by recalling that Achilles is paired 
with the Tortoise, and the Crab with his Genes. 
When "copying" a strand, therefore, you don't actually copy it, but you manufacture 
its complementary  strand. And this one will be written upside down above the original strand. 
Let's see this in concrete terms. Let the previous enzyme act on the following strand (and that 
enzyme also happens to like to start at A): 
 
CAAAGAGAATCCTCTTTGAT 
 
There are many places it could start. Let's take the second A, for example. The enzyme binds 
to it, then executes step 1: Search for the nearest pyrimidine to the right. Well, this means a C 
or a T. The first one is a T somewhere near the middle of the strand, so that's where we go. 
Now step 2: Copy mode. Well, we just put an upside-down A above our  T. But that's not all, 
for Copy mode remains in effect  until it is shut off-or until the enzyme is done, whichever

for Copy mode remains in effect  until it is shut off-or until the enzyme is done, whichever 
comes first. This means that every base which is passed through by the enzyme while Copy 
mode is on will get a complementary base put above it. Step 3 says to look for a purine to the right of our T. That is the G two symbols in from the right-hand end. Now as we move up to 
that G, we must "copy"-that is, create a complementary strand. Here's what that gives: 
(editor’ s note, I can’t print upside down ie it is too much hard work so V = A and D = G 
upside down)

upside down) 
  
The last step is to cut the strand. This will yield two pieces: 
 
           VDDVDVVVJ  
CAAAGAGAATCCTCTTTG 
 
and AT. 
 
And the instruction packet is done. We are left with a double strand, however. Whenever this 
happens, we separate the two complementary strands from each other (general principle); so 
in fact our end product is a set of three strands: 
 
AT, CAAAGAGGA,     and CAAAGAGAATCCTCTTTG 
 
Notice that the upside-down strand has been turned right side up, and thereby right and left 
have been reversed.

VDDVDVVVJ  
 
and AT. 
 
have been reversed. 
Now you have seen most of the typographical operations which can be carried out on 
strands. There are two other instructions which should be mentioned. One shuts off Copy 
mode; the other switches the enzyme from a strand to the upside-down strand above it. When 
this happens, if you keep the paper right side up, then you must switch "left" and "right" in all 
the instructions. Or better, you can keep the wording and just turn the paper around so the top 
strand becomes legible. If the "switch" command is

Self-Rep and Self-Rep 503 
 
given, but there is no complementary base where the enzyme is bound at that instant, then the 
enzyme just detaches itself from the strand, and its job is done. 
It should be mentioned that when a "cut" instruction is encountered, this pertains to 
both strands (if there are two): however, "delete" pertains only to the strand on which the enzyme is working. If Copy mode is on, then the "insert" command pertains to both strands-
the base itself into the strand the enzyme is working on, and its complement into the other 
strand. 
If Copy mode is off, then the "insert" command pertains only to the one strand, so a blank 
space must he inserted into the complementary strand. 
And, whenever Copy mode is on, "move" and "search" commands require that one 
manufacture complementary bases to all bases which the sliding enzyme touches. 
Incidentally, Copy mode is always off when an enzyme starts to work. If Copy mode is off,

strand. 
Incidentally, Copy mode is always off when an enzyme starts to work. If Copy mode is off, 
and the command "Shut off copy mode" is encountered, nothing happens. Likewise, If Copy 
mode is already on, and the command "Turn copy mode on" is encountered, then nothing 
happens. 
 
Amino Acids 
 
There are fifteen types of command, listed below:  
Cut cut strand(s) 
del  delete a base from strand 
swi switch enzyme to other strand 
mvr move one unit to the right mvl move one unit to the left 
cop turn on Copy mode 
off  turn off Copy mode 
ina  insert A to the right of this unit

happens. 
 
ina  insert A to the right of this unit 
inc  insert C to the right of this unit ing insert G to the right of this unit 
int  insert T to the right of this unit 
rpy search for the nearest pyrimidine to the right 
rpu search for the nearest purine to the right 
Ipy search for the nearest pyrimidine to the left lpu search for the nearest purine to the left 
 
Each one has a three-letter abbreviation. We shall refer to the three-letter abbreviations of 
commands as amino acids. Thus, every enzyme is made up of a sequence of amino acids . Let

commands as amino acids. Thus, every enzyme is made up of a sequence of amino acids . Let 
us write down an arbitrary enzyme:  
           rpu - inc - cop - myr - tnyl - swi - Ipu - int  
 
and an arbitrary strand: 
 
TAGATCCAGTCCATCGA

Self-Rep and Self-Rep 504 
 
and see how the enzyme acts on the strand. It so happens that the enzyme binds to  G only. 
Let us bind to the middle G and begin. Search rightwards for a purine (viz., A or G). We (the 
enzyme) skip over TCC  and land on A. 
Insert a C. Now we have 
 
TAGATCCAGTCCACTCGA 
 
where the arrow points to the unit to which the enzyme is bound. Set Copy mode. This puts an upside-down G above the C. Move right, move left, then switch to the other strand. Here's 
what we have so far: 
 
7V 
 
Let's turn it upside down, 'so that the enzyme is attached to the lower strand: 
 
VDJIDV»1DVDD1VDV1 
AG 
 
Now we search leftwards for a purine and find A. Copy mode is on, but the complementary 
bases are already there, so nothing is added. Finally, we insert a T (in Copy mode), and quit: 
 
VD)IVJVJJl7V)DIVDV1 
ATG 
 
Our final product is thus two strands: 
 
             ATG , and TAGATCCAGTCCACATCGA   
 
The old one is of course gone. 
 
Translation and the Typogenetic Code

7V 
 
AG 
 
ATG 
 
Translation and the Typogenetic Code 
 
Now you might be wondering where the enzymes and strands come from, and how to tell the 
initial binding-preference of a given enzyme. One way might be just to throw some random 
strands and some random enzymes together, and see what happens when those enzymes act 
on those strands and their progeny. This has a similar flavor to the MU-puzzle, where there

ATG , and TAGATCCAGTCCACATCGA   
 
on those strands and their progeny. This has a similar flavor to the MU-puzzle, where there 
were some given rules of inference and an axiom, and you just began. The only difference is that here, every time a strand is acted on, its original form is gone forever. In the MU-puzzle, 
acting on MI to make MIU  didn't destroy MI 
But in Typogenetics, as in real genetics, the scheme is quite a bit trickier. We do 
begin with some arbitrary strand, somewhat like an axiom in a formal system. But we have, 
initially, no "rules of inference"-that is, no enzymes. However, we can translate  each strand

initially, no "rules of inference"-that is, no enzymes. However, we can translate  each strand 
into one or more enzymes! Thus, the strands themselves will dictate the operations which 
will be performed upon them, and those operations will in turn produce

Self-Rep and Self-Rep 505 
 
new strands which will dictate further enzymes, etc. etc.! This is mixing levels with a 
vengeance! Think, for the sake of comparison, how different the MU-puzzle would have 
been if each new theorem  produced could have been turned into a new rule of inference  by 
means of some code. 
    How is this "translation" done? It involves a Typogenetic Code  by which adjacent pairs of 
bases-called "duplets"-in a single strand repres ent different amino acids. There are sixteen 
possible duplets: AA, AC, AG, AT, CA, CC , etc. And there are fifteen amino acids. The 
Typogenetic Code is shown in Figure 87.  
    
According to the table, the translation of the duplet GC is "inc" ("insert a C"); that of AT is 
"swi" ("switch strands"); and so on. Therefore it becomes clear that a strand can dictate an 
enzyme very straightforwardly. For example, the strand 
 
TAGATCCAGTCCACATCGA 
 
breaks up into duplets as follows: 
 
TA GA TC CA GT CC AC AT CG A

TA GA TC CA GT CC AC AT CG A 
 
with the A left over at the end. Its translation into an enzyme is: 
 
          rpy - ina - rpu - mvr - int - mvl - cut - swi - cop.  
 (Note that the leftover A contributes nothing.) 
 
Tertiary Structure of Enzymes 
 
What about the little letters 's', 'l', and 'r' in the lower righthand corner of each box% They are crucial in determining the enzyme's binding-preference, and in a peculiar way. In order to 
figure out what letter an enzyme likes to bind to, you have to figure out the enzyme's "tertiary

figure out what letter an enzyme likes to bind to, you have to figure out the enzyme's "tertiary 
structure", which is itself determined by the enzyme's "primary structure". By its

Self-Rep and Self-Rep 506 
 
primary structure  is meant its amino acid sequence. By its tertiary structure is meant the way 
it likes to "fold up". The point is that enzymes don't like being in straight lines, as we have so 
far exhibited them. At each internal amino acid (all but the two ends), there is a possibility of a "kink", which is dictated by the letters in the corners. In particular, '1' and 'r' stand for "left" 
and "right", and 's' stands for "straight". So let us take our most recent sample enzyme, and let 
it fold itself up to show its tertiary struct ure. We will start with the enzyme's primary 
structure, and move along it from left to right. At each amino acid whose corner-letter is '1' 
we'll put a left turn, for those with 'r', we'll put a right turn, and at 's' we'll put no turn. In Figure 88 is shown the two-dimensional conformation for our enzyme. 
 
Cop 
 
swi <== cut <== mvl   <== int  
      mvr 
            rpy -==>  ima  ==> rpu

Cop 
 
      mvr 
            rpy -==>  ima  ==> rpu 
 
FIGURE 88. The tertiary structure of a typoenzyme. 
 Note the left-kink at "rpu", the right-kink at "swi", and so on. Notice also that the first 
segment ("rpy z> ina") and the last segment ("swi => cop") are perpendicular. This is the key 
to the binding-preference. In fact, the relative orientation of the first and last segments  of an 
enzyme's tertiary structure determines the binding-preference of the enzyme. We can always

mvr 
 
enzyme's tertiary structure determines the binding-preference of the enzyme. We can always 
orient the enzyme so that its first segment points to the right. If we do so, then the last segment determines the binding-preference, as shown in Figure 89.

Self-Rep and Self-Rep 507 
 
So in our case, we have an enzyme which likes the letter C. If, in folding up, an enzyme 
happens to cross itself, that's okay-just think of it as going under or over itself. Notice that all 
its amino acids play a role in the determination of an enzyme's tertiary structure. 
 
Punctuation, Genes, and Ribosomes 
 
Now one thing remains to he explained. Why is there a blank in box AA of the typogenetic 
Code' The answer is that the duplet AA acts as a punctuation mark inside a strand, and it 
signals the end of the code for an enzyme. That is to say, one strand may code for two or more enzymes if it has one or more duplets 
AA in it. For example, the strand 
 
CG GA TA CT AA AC CG A 
 
Codes for two enzymes 
 
cop - ina - rpy - off 
and 
       cut – cop 
 
with the AA serving to divide the strand up into two "genes". The definition of gene is: that 
portion of a strand which codes for a single enzyme. Note that the mere presence of AA

and 
       cut – cop 
 
portion of a strand which codes for a single enzyme. Note that the mere presence of AA 
inside a strand does not mean that the strand codes for two enzymes. For instance, CAAG  
codes for "mvr - del". The AA begins on an even-numbered unit and therefore is not read as a 
duplet! 
  The mechanism which reads strands and produces the enzymes which are coded 
inside them is called a ribosome. (In Typogenetics, the player of the game does the work of' 
the ribosomes.) Ribosomes are not in any way responsible for the tertiary  structure of 
enzymes, for that is entirely determined once the primary  structure is created. Incidentally,

duplet! 
enzymes, for that is entirely determined once the primary  structure is created. Incidentally, 
the process of translation  always goes from strands to enzymes , and never in the reverse 
direction. 
 
Puzzle: A Typogenetical Self-Rep 
 
Now that the rules of Typogenetics have been fully set out, you may find it interesting to 
experiment with the game. In particular, it would he most interesting to devise a self-
replicating strand. This would mean something along the following lines. A single strand is 
written down. A ribosome acts on it, to produce any or all of the enzymes which are coded

written down. A ribosome acts on it, to produce any or all of the enzymes which are coded 
for in the strand. Then those enzymes are brought into contact with the original strand, and allowed to work on it. This yields a set of "daughter strands". The daughter strands 
themselves pass through the rihosomes, to yield a second generation of enzymes, which act 
on the daughter strands; and the

Self-Rep and Self-Rep 508 
cycle goes on and on. This can go on for any number of stages; the hope is that eventually, 
among the strands which are present at some point, there 
will be found two copies of the original strand (one of the copies may be, in fact, the original 
strand). 
 
The Central Dogma of Typogenetics 
 
Typogenetical processes can be represented in skeletal form in a diagram (Fig. 90). 
 
This diagram illustrates the Central Dogma of Typogenetics . It shows how strands define 
enzymes (via the Typogenetic Code); and how in turn, enzymes act back on the strands 
which gave rise to them, yielding new strands. Therefore, the line on the left portrays how old information flows upwards, in the sense that an enzyme is a translation of a strand, and 
contains therefore the same information as the strand, only in a different form-in particular; 
in an active form. The line on the right, however, does not show information flowing

in an active form. The line on the right, however, does not show information flowing 
downwards; instead, it shows how new information gets created : by the shunting of symbols 
in strands. 
An enzyme in Typogenetics, like a rule of inference in a formal system, blindly 
shunts symbols in strands without regard to any "meaning" which may lurk in those symbols. 
So there is a curious mixture of levels here. On the one hand, strands are acted upon, and 
therefore play the role of data (as is indicated by the arrow on the right); on the other hand,

therefore play the role of data (as is indicated by the arrow on the right); on the other hand, 
they also dictate the actions which are to be performed on the data, and therefore they play the role of programs  (as is indicated by the arrow on the left). It is the player of Typogenetics 
who acts as interpreter and processor, of course. The two-way street which links "upper" and 
"lower" levels of Typogenetics shows that, in fact, neither strands nor enzymes can be 
thought of as being on a higher level than the other. By contrast, a picture of the Central 
Dogma of the MIU -system looks this way:

Dogma of the MIU -system looks this way: 
 
In the MIU -system, there is a clear distinction of levels: rules of inference simply belong to a 
higher level than strings. Similarly for TNT, and all formal systems.

Self-Rep and Self-Rep 509 
 
Strange Loops, TNT, and Real Genetics 
 
However, we have seen that in TNT , levels are mixed, in another sense. In fact, the 
distinction between language and metalanguage breaks down: statements about the system get mirrored inside the system. It turns out that if we make a diagram showing the 
relationship between TNT and its metalanguage, we will produce something which resembles 
in a remarkable way the diagram which represents the Central Dogma of Molecular Biology. 
In fact, it is our goal to make this comparison in detail; but to do so, we need to indicate the 
places where Typogenetics and true genetics coincide, and where they differ. Of course, real genetics is far more complex than Typogenetics-but the "conceptual skeleton" which the 
reader has acquired in understanding Typogenetics will be very useful as a guide in the 
labyrinth of true genetics. 
 
DNA and Nucleotides 
 
We begin by discussing the relationship between "strands", and DNA . The initials "DNA"

We begin by discussing the relationship between "strands", and DNA . The initials "DNA" 
stand for "deoxyribonucleic acid". The DNA  of most cells resides in the cell's nucleus, which 
is a small area protected by a membrane. Gunther Stent has characterized the nucleus as the 
"throne room" of the cell, with DNA  acting as the ruler. DNA consists of long chains of 
relatively simple molecules called nucleotides. Each nucleotide is made up of three parts: (1) 
a phosphate group stripped of one special oxygen atom, whence the prefix "deoxy"; (2) a

a phosphate group stripped of one special oxygen atom, whence the prefix "deoxy"; (2) a 
sugar called "ribose", and (3) a base. It is the base alone which distinguishes one nucleotide 
from another; thus it suffices to specify its base to identify a nucleotide. The four types of 
bases which occur in DNA  nucleotides are: 
 
(Also see Fig. 91.) It is easy to remember which ones are pyrimidines because the first vowel 
in "cytosine", "thymine", and "pyrimidine" is 'y'. Later, when we talk about RNA , "uracil"-
also a pyrimidine-will come in and wreck the pattern, unfortunately. (Note: Letters

also a pyrimidine-will come in and wreck the pattern, unfortunately. (Note: Letters 
representing nucleotides in real genetics will not be in the Quadrata  font, as they were in 
Typogenetics.) 
A single strand of DNA  thus consists of many nucleotides strung together like a chain 
of beads. The chemical bond which links a nucleotide to its two neighbors is very strong; 
such bonds are called covalent bonds , and the "chain of beads" is often called the covalent 
backbone  of DNA . 
   Now DNA  usually comes in double strands-that is, two single strands which are paired up, 
nucleotide by nucleotide (see Fig. 92). It is the bases

Self-Rep and Self-Rep 510
 
FIGURE 91. The four constituent bases of DNA: Adenine, Guanine, Thymine, Cytosine. 
[From Hanawalt and Haynes, The Chemical Basis of Life (San Francisco: W. H. Freeman, 
1973), p. 142.J 
  
FIGURE 92. DNA structure resembles a ladder in which the side pieces consist of alternating 
units of deoxyrihose and phosphate. The rungs are formed by the bases paired in a special 
way, A with T and G with C, and held together respectively by two and three hydrogen 
bonds. [From Hanawalt and Haynes, The Chemical Basis of Life, p. 142.

Self-Rep and Self-Rep 511which are responsible for the peculiar kind of pairing which takes place between strands. 
Each base in one strand faces a complementary base in the other strand, and binds to it. The 
complements are as in Typogenetics: A pairs up with T, and C with G. Always one purine 
pairs up with a pyrimidine. 
Compared to the strong covalent bonds along the backbone, the interstrand bonds are 
quite weak. They are not covalent bonds, but hydrogen bonds. A hydrogen bond arises when two molecular complexes are aligned in such a way that a hydrogen atom which originally 
belonged to one of them becomes "confused" about which one it belongs to, and it hovers 
between the two complexes, vacillating as to which one to join. Because the two halves of 
double-stranded DNA are held together only by hydrogen bonds, they may come apart or be 
put together relatively easily; and this fact is of great import for the workings of the cell.

put together relatively easily; and this fact is of great import for the workings of the cell. 
When DNA forms double strands, the two strands curl around each other like 
twisting vines (Fig. 93). There are exactly ten nucleotide pairs per revolution; in other words, 
at each nucleotide, the "twist" is 36 degrees. Single-stranded DNA does not exhibit this kind 
of coiling, for it is a consequence of the base-pairing. 
 
FIGURE 93. Molecular model of the DNA double helix. [From Vernon M. Ingram, 
Biosynthesis (Menlo Park, Calif.: W. A. Benjamin, 1972)

Self-Rep and Self-Rep 512Messenger KNA and Ribosomes 
 
As was mentioned above, in many cells, DNA, the ruler of the cell, dwells in its private 
"throne room": the nucleus of the cell. But most of the "living" in a cell goes on outside of 
the nucleus, namely in the cytoplasm -the "ground" to the nucleus' "figure". In particular, 
enzymes , which make practically every life process go, are manufactured by ribosomes  in the 
cytoplasm, and they do most of their work in the cytoplasm. And just as in Typogenetics, the 
blueprints for all enzymes are stored inside the strands-that is, inside the DNA, which 
remains protected in its little nuclear home. So how does the information about enzyme 
structure get from the nucleus to the ribosomes' 
Here is where messenger  RNA -mRNA -comes in. Earlier, m RNA  strands were 
humorously said to constitute a kind of DNA  Rapid Transit Service; by this is meant not that 
mRNA physically carries DNA  anywhere, but rather that it serves to carry the information, or

mRNA physically carries DNA  anywhere, but rather that it serves to carry the information, or 
message, stored in the DNA  in its nuclear chambers, out to the ribosomes in the cytoplasm. 
How is this done? The idea is easy: a special kind of enzyme inside the nucleus faithfully 
copies long stretches of the DNA's  base sequence onto a new strand-a strand of messenger 
RNA. This m RNA  then departs from the nucleus and wanders out into the cytoplasm, where 
it runs into many ribosomes which begin doing their enzyme-creating work on it. 
    The process by which DNA  gets copied onto m RNA  inside the nucleus is called

    The process by which DNA  gets copied onto m RNA  inside the nucleus is called 
transcription ; in it, the double-stranded DNA must be temporarily separated into two single 
strands, one of which serves as a template for the mRNA. Incidentally, " RNA " stands for 
"ribonucleic acid", and it is very much like DNA  except that all of its nucleotides possess that 
special oxygen atom in the phosphate group which DNA's  nucleotides lack. Therefore the 
"deoxy" prefix is dropped. Also, instead of thymine, RNA  uses the' base uracil, so the 
information in strands of RNA  can be represented by arbitrary sequences of the four letters

information in strands of RNA  can be represented by arbitrary sequences of the four letters 
'A', 'C', 'G', 'U' . Now when mRNA is transcribed off of DNA , the transcription process 
operates via the usual base-pairing (except with U instead of T), so that a DNA -template and 
its m RNA -mate might look something like this: 
 
DNA:     GGTAAATCAAGTCA    (template) 
mRNA :   GGCAUUUAGUCAGU   (copy") 
 
RNA  does not generally form long double strands with itself, although it can. Therefore it is 
prevalently found not in the helical form which so characterizes DNA , but rather in long, 
somewhat randomly curving strands.

somewhat randomly curving strands. 
Once a strand of mRNA has escaped the nucleus, it encounters those strange 
subcellular creatures called "ribosomes"-but before we go on to explain how a ribosome uses 
mRNA , I want to make some comments about enzymes and proteins. Enzymes belong to the 
general category of biomolecules called proteins , and the job of ribosomes is to make all pro

Self-Rep and Self-Rep 513 
     
teins, not just enzymes. Proteins which are not enzymes are much more passive kinds of 
beings: many of them, for instance, are structural molecules, which means that they are like 
girders and beams and so forth in buildings: they hold the cell's parts together. There are 
other kinds of proteins, but for our purposes. the principal proteins are enzymes, and I will henceforth not make a sharp distinction. 
 
Proteins are composed of sequences of amino acids, which come in twenty primary varieties, each with a three-letter abbreviation: 
 
ala alanine 
arg arginine 
asn asparagines asp aspartic acid 
cys cysteine 
gln glutamine 
glu glutamic acid 
gly glycine his histidine 
He isoleucine 
leu leucine 
lys lysine 
met methionine phe phenylalanine 
pro praline 
ser serine 
thr threonine 
trp tryptophan tyr tyrosine 
val valine 
 
Notice the slight numerical discrepancy with Typogenetics, where we had only fifteen

lys lysine 
ser serine 
val valine 
 
Notice the slight numerical discrepancy with Typogenetics, where we had only fifteen 
"amino acids" composing enzymes. An amino acid is a small molecule of roughly the same complexity as a nucleotide; hence the building blocks of proteins and of nucleic acids ( DNA , 
RNA ) are roughly of the same size. However, proteins are composed of much shorter 
sequences of components: typically, about three hundred amino acids make a complete 
protein, whereas a strand of DNA  can consist of hundreds of thousands or millions of 
nucleotides.  
Ribosomes and Tape Recorders 
 
Now when a strand of m RNA , after its escape into the cytoplasm, encounters a ribosome, a

Now when a strand of m RNA , after its escape into the cytoplasm, encounters a ribosome, a 
very intricate and beautiful process called translation takes place. It could be said that this process of translation is at the very heart of

Self-Rep and Self-Rep 514 
 
all of life, and there are many mysteries connected with it. But in essence it is easy to 
describe. Let us first give a picturesque image, and then render it more precise. Imagine the 
mRNA to be like a long piece of magnetic recording tape, and the ribosome to be like a tape 
recorder. As the tape passes through the playing head of the recorder, it is "read" and converted into music, or other sounds. Thus magnetic markings are "translated" into notes. 
Similarly, when a "tape" of mRNA passes through the "playing head" of a ribosome, the 
"notes" which are produced are amine acids, and the "pieces of music" which they make up 
are proteins. This is what translation is all about: it is shown in Figure 96. 
 
The Genetic Code 
But how can a ribosome produce a chain of amino acids when it is reading a chain of 
nucleotides This mystery was solved in th e early 1960's by the efforts of a large number

nucleotides This mystery was solved in th e early 1960's by the efforts of a large number 
of people, and at the core of the answer lie s the Genetic Code-a mapping from triplets of 
nucleotides into amino acids (see Fig. 94). Th is is in spirit extremely similar to the 
Typogenetic Code, except that he re, three consecutive bases (or

Typogenetic Code, except that he re, three consecutive bases (or 
nucleotides) form a codon, whereas there, only two were needed. Thus there must be 4x4x4 (equals 64) different entries in the  table, instead of sixteen. A ribosome clicks down a strand of RNA three nucleotides at a time-which is to say, one. codon at a time -and each time it does so, it appends a single new amino acid to the pr otein it is presently manufacturing. Thus, a 
protein comes out of the ribosome amino acid by amino acid. 
 
Tertiary Structure 
 
 However, as a protein emerges from a ribosome, it is not only getting longer and longer, but

 However, as a protein emerges from a ribosome, it is not only getting longer and longer, but 
it is also continually folding itself up into an extraordinary three-dimensional shape, very 
much in the way that those funny little Fourth-of-July fireworks called "snakes" simultaneously grow longer and curl up, when they are lit. This fancy shape is called the 
protein's tertiary structure (Fig. 95), while the amino acid sequence per se is called the 
primary structure of the protein. The tertiary structure is implicit in the primary structure, just 
as in Typogenetics. However, the recipe for deriving the tertiary structure, if you know only 
the primary structure, is by far more complex than that given in Typogenetics. In fact, it is one of the outstanding problems of contemporary molecular biology to figure out some rules 
by which the tertiary structure of a protein can be predicted if only its primary structure is

by which the tertiary structure of a protein can be predicted if only its primary structure is 
known. CUA      GAU  
, C u   A g   A u 
 
A typical segment of mRNA 
read first as two triplets  
(above), and second as three 
duplets (below): an example 
of hemiolia in biochemistry

Self-Rep and Self-Rep 515The Genetic Code. 
 
 U C A G  
U phe 
phe 
leu 
leu ser 
ser 
ser 
ser tyr 
tyr 
punt. 
punc. cys 
cys 
punt. 
trp U 
C 
A 
G 
C leu 
leu 
leu 
Ieu pro 
pro 
pro 
pro his 
his 
gin 
gln arg 
arg 
arg 
arg U 
C 
A 
G 
A ile 
ile 
ile 
met thr 
thr 
thr 
thr asn 
asn 
lys 
lys ser 
ser 
arg 
arg U 
C 
A 
G 
G val val 
val 
 
val ala 
ala 
ala
 
ala asp 
asp 
glu 
 
glu gly 
gly 
gly
 
gly U 
C 
A 
 
G 
 
 FIGURE 94. The Genetic Code, by which each triplet in a strand of messenger RNA codes 
for one of twenty amino acids (or a punctuation mark). 
 
Reductionistic Explanation of Protein Function 
 
Another discrepancy between Typogenetics and true genetics-and this is probably the most 
serious one of all-is this: whereas in Typogenetics, each component amino acid of an enzyme 
is responsible for some specific "piece of the action", in real enzymes, individual amino acids 
cannot be assigned such clear roles. It is the tertiary structure as a whole which determines

cannot be assigned such clear roles. It is the tertiary structure as a whole which determines 
the mode in which an enzyme will function; there is no way one can say, "This

Self-Rep and Self-Rep 516 
 
amino acid's presence means that such-and-such an operation will get performed". In other 
words, in real genetics, an individual amino acid's contribution to the enzyme's overall 
function is not "context-free". However, this fact should not be construed in any way as 
ammunition for an anti reductionist argument to the effect that "the whole [enzyme] cannot be explained as the sum of its parts". That w ould he wholly unjustified. What is justified is 
rejection of the simpler claim that "each amino acid contributes to the sum in a manner which 
is independent of the other amino acids present". In other words, the function of a protein 
cannot be considered to be built up from context-free functions of its parts; rather, one must 
consider how the parts interact. It is still possible in principle to write a computer program which takes as input the primary structure of a protein, 
 
FIGURE 95. The structure of myoglobin, deduced from high-resolution X-ray data. The

FIGURE 95. The structure of myoglobin, deduced from high-resolution X-ray data. The 
large-scale "twisted pipe" appearance is the tertiary structure; the finer helix inside-the 
"alpha helix"-is the secondary structure. [From A. Lehninger, Biochemistry]

Self-Rep and Self-Rep 517and firstly determines its tertiary structure, and secondly determines the function of the 
enzyme. This would be a completely reductionistic explanation of the workings of proteins, 
but the determination of the "sum" of the parts would require a highly complex algorithm. 
The elucidation of the function of an enzyme, given its primary, or even its tertiary, structure, 
is another great problem of contemporary molecular biology. 
Perhaps, in the last analysis, the function of the whole enzyme can be considered to 
be built up from functions of parts in a context-free manner, but where the parts are now 
considered to be individual particles, such as electrons and protons, rather than "chunks", 
such as amino acids. This exemplifies the "Reductionist's Dilemma": In order to explain 
everything in terms of context free sums, one has to go down to the level of physics; but then

everything in terms of context free sums, one has to go down to the level of physics; but then 
the number of particles is so huge as to make it only a theoretical "in-principle" kind of thing. So, one has to settle for a context-dependent sum, which has two disadvantages. The first is 
that the parts are much larger units, whose behavior is describable only on a high level, and 
therefore indeterminately. The second is that the word "sum" carries the connotation that 
each part can be assigned a simple function and that the function of the whole is just a

each part can be assigned a simple function and that the function of the whole is just a 
context-free sum of those individual functions. This just cannot be done when one tries to explain a whole enzyme's function, given its amino acids as parts. But for better or for worse, 
this is a general phenomenon which arises in the explanations of complex systems. In order 
to acquire an intuitive and manageable understanding of how parts interact-in short, in order 
to proceed-one often has to sacrifice the exactness yielded by a microscopic, context-free

to proceed-one often has to sacrifice the exactness yielded by a microscopic, context-free 
picture, simply because of its unmanageability. But one does not sacrifice at that time the faith that such an explanation exists in principle. 
 
Transfer RNA and Ribosomes 
 
Returning, then, to ribosomes and RNA and proteins, we have stated that a protein is 
manufactured by a ribosome according to the blueprint carried from the DNA's "royal 
chambers" by its messenger, RNA. This seems to imply that the ribosome can translate from 
the language of codons into the language of amino acids, which amounts to saying that the

the language of codons into the language of amino acids, which amounts to saying that the 
ribosome "knows" the Genetic Code. However, that amount of information is simply not present in a ribosome. So how does it do it? Where is the Genetic Code stored? The curious 
fact is that the Genetic Code is stored-where else?-in the DNA itself. This certainly calls for 
some explanation. 
Let us back off from a total explanation for a moment, and give a partial explanation. 
There are, floating about in the cytoplasm at any given moment, large numbers of four-leaf-clover-shaped molecules; loosely fastened (i.e., hydrogen-bonded) to one leaf is an amino

There are, floating about in the cytoplasm at any given moment, large numbers of four-leaf-clover-shaped molecules; loosely fastened (i.e., hydrogen-bonded) to one leaf is an amino 
acid, and on the opposite leaf there is a triplet of nucleotides called an anticodon. For our 
purposes, the other two leaves are irrelevant. He re is how these "clovers" are used by the 
ribosomes in their production of proteins. When a new

Self-Rep and Self-Rep 518
 
FIGURE 96. A section of mRNA passing through a ribosome. Floating nearby are t RNA  
molecules, carrying amino acids which are stripped off by the ribosome and appended to the 
growing protein. The Genetic Code is contained in the t RNA  molecules, collectively. Note 
how the base-pairing (A-U, C-G) is represented by interlocking letter-forms in the diagram. 
[Drawing by Scott E. Kim] 
 codon of m RNA  clicks into position in the ribosome's "playing head", the ribosome reaches 
out into the cytoplasm and latches onto a clover whose anticodon is complementary to the 
mRNA  codon. Then it pulls the clover into such a position that it can rip off the clover's 
amino acid, and stick it covalently onto the growing protein. (Incidentally, the bond between 
an amino acid and its neighbor in a protein is a very strong covalent bond, called a "peptide bond". For this reason, proteins are sometimes called "polypeptides".) Of course it is no

an amino acid and its neighbor in a protein is a very strong covalent bond, called a "peptide bond". For this reason, proteins are sometimes called "polypeptides".) Of course it is no 
accident that the "clovers" carry the proper amino acids, for they have all been manufactured 
according to precise instructions emanating from the "throne room".

Self-Rep and Self-Rep 519 
     
The real name for such a clover is transfer  RNA . A molecule of t RNA is quite small-about 
the size of a very small protein-and consists of a chain of about eighty nucleotides. Like 
mRNA , tRNA  molecules are made by transcription  off of the grand cellular template, DNA. 
However, tRNA's are tiny by comparison with the huge mRNA molecules, which may contain thousands of nucleotides in long, long chains. Also, t RNA 's resemble proteins (and 
are unlike strands of m RNA ) in this respect: they have fixed, well-defined tertiary structures-
determined by their primary structure. A t RNA  molecule's tertiary structure allows precisely 
one amino acid to bind to its amino-acid site: to be sure, it is that one dictated according to 
the Genetic Code by the anticodon on the opposite arm. A vivid image of the function of tRNA molecules is as flashcards floating in a cloud around a simultaneous interpreter, who

the Genetic Code by the anticodon on the opposite arm. A vivid image of the function of tRNA molecules is as flashcards floating in a cloud around a simultaneous interpreter, who 
snaps one out of the air-invariably the right one!-whenever he needs to translate a word. In 
this case, the interpreter is the ribosome, the words are codons, and their translations are 
amino acids. 
In order for the inner message of DNA  to get decoded by the ribosomes, the t RNA 
flashcards must be floating about in the cytoplasm. In some sense, the t RNA s contain the 
essence of the outer message of the DNA , since they are the keys to the process of

essence of the outer message of the DNA , since they are the keys to the process of 
translation. But they themselves came from the DNA . Thus, the outer message is trying to be 
part of the inner message, in a way reminiscent of the message-in-a-bottle which tells what 
language it is written in. Naturally, no such attempt can be totally successful: there is no way for the DNA  to hoist itself by its own bootstraps. Some amount of knowledge of the Genetic 
Code must already be present in the cell beforehand, to allow the manufacture of those 
enzymes which transcribe tRNA's themselves off of the master copy of DNA . And this

enzymes which transcribe tRNA's themselves off of the master copy of DNA . And this 
knowledge resides in previously manufactured t RNA  molecules. This attempt to obviate the 
need for any outer message at all is like the Escher dragon, who tries as hard as he can, within the context of the two-dimensional world to which he is constrained, to be 
threedimensional. He seems to go a long way-but of course he never makes it, despite the 
fine imitation he gives of three-dimensionality. 
 
Punctuation and the Reading Frame 
 
How does a ribosome know when a protein is done? Just as in Typogenetics, there is a signal

How does a ribosome know when a protein is done? Just as in Typogenetics, there is a signal 
inside the m RNA  which indicates the termination or initiation of a protein. In fact, three 
special codons- UAA, CAG, UGA  act as punctuation  marks instead of coding for amino 
acids. Whenever such a triplet clicks its wa y into the "reading head" of a ribosome, the 
ribosome releases the protein under construction and begins a new one. 
Recently, the entire genome of the tiniest known virus, φ)X174, has been laid bare. 
One most unexpected discovery was made en route: some of its nine genes overlap-that is,

One most unexpected discovery was made en route: some of its nine genes overlap-that is, 
two distinct proteins are coded for by the same stretch of DNA ! There is even one gene 
contained entirely inside another!

Self-Rep and Self-Rep 520 
 
This is accomplished by having the reading frames of the two genes shifted relative to each 
other, by exactly one unit. The density of information packing in such a scheme is incredible. 
This is, of course, the inspiration behind the strange "5/17 haiku" in Achilles' fortune cookie, 
in the Canon byIntervallic Augmentation , 
 
Recap 
 
In brief, then, this picture emerges: from its central throne, DNA sends off long strands of 
messenger RNA to the ribosomes in the cytoplasm; and the ribosomes, making use of the "flashcards" of tRNA hovering about them, efficiently construct proteins, amino acid by 
amino acid, according to the blueprint contained in the mRNA. Only the primary structure of 
the proteins is dictated by the DNA; but this is enough, for as they emerge from the 
ribosomes, the proteins "magically" fold up into complex conformations which then have the 
ability to act as powerful chemical machines.  
Levels of Structure and Meaning in Proteins and Music

Levels of Structure and Meaning in Proteins and Music 
 
We have been using this image of ribosome as tape recorder, mRNA as tape, and protein as 
music. It may seem arbitrary, and yet there are some beautiful parallels. Music is not a mere linear sequence of notes. Our minds perceive pieces of music on a level far higher than that. 
We chunk notes into phrases, phrases into melodies, melodies into movements, and 
movements into full pieces. Similarly, proteins only make sense when they act as chunked 
units. Although a primary structure carries all the information for the tertiary structure to be

units. Although a primary structure carries all the information for the tertiary structure to be 
created, it still "feels" like less, for its potential is  only realized when the tertiary structure is 
actually physically created. 
   Incidentally, we have been referring only to primary and tertiary structures, and you may 
well wonder whatever happened to the secondary structure. Indeed, it exists, as does a 
quaternary structure, as well. The folding-up of a protein occurs at more than one level.

quaternary structure, as well. The folding-up of a protein occurs at more than one level. 
Specifically, at some points along the chain of amino acids, there may be a tendency to form a kind of helix, called the alpha helix (not to be confused with the DNA double helix). This 
helical twisting of a protein is on a lower level than its tertiary structure. This level of 
structure is visible in Figure 95. Quaternary structure can be directly compared with the 
building of 'a musical piece out of independent movements, for it involves the assembly of'

building of 'a musical piece out of independent movements, for it involves the assembly of' 
several distinct polypeptides, already in their full-blown tertiary beauty, into a larger structure. The binding of these independent chains is usually accomplished by hydrogen 
bonds, rather than covalent bonds; this is of cour se just as with pieces of music composed of 
several movements, which are far less tightly bound to each other than they are internally, but 
which nevertheless form a tight "organic" whole.

which nevertheless form a tight "organic" whole. 
   The four levels of primary, secondary, tertiary, and quaternary structure can also be compared to the four levels of the MU-picture (Fig. 60) in

Self-Rep and Self-Rep 522    
 
the Prelude, Ant Fugue . The global structure-consisting of the letters 'M' and 'U'-is its 
quaternary structure; then each of those two parts has a tertiary structure, consisting of 
"HOLISM" or "REDUCTIONISM"; and then the opposite word exists on the secondary level, and at bottom, the primary structure is once again the word "MU", over and over again. 
 
Polyribosomes and Two-Tiered Canons 
 
Now we come to another lovely parallel between tape recorders translating tape into music and ribosomes translating mRNA into proteins. Imagine a collection of many tape recorders, 
arranged in a row, evenly spaced. We might call this array a "polyrecorder". Now imagine a 
single tape passing serially through the playing heads of all the component recorders. If the 
tape contains a single long melody, then the output will be a many-voiced canon, of course,

tape contains a single long melody, then the output will be a many-voiced canon, of course, 
with the delay determined by the time it takes the tape to get from one tape recorder to the next. In cells, such "molecular canons" do indeed exist, where many ribosomes, spaced out in 
long lines-forming what is called a polyribosome-all "play" the same strand of mRNA, 
producing identical proteins, staggered in time (see Fig. 97). 
Not only this, but nature goes one better. Recall that mRNA is made by transcription

Not only this, but nature goes one better. Recall that mRNA is made by transcription 
off of DNA; the enzymes which are responsible for this process are called RNA polymerases ("-ase" is a general suffix for enzymes). It happens often that a series of RNA polymerases 
will be at work in parallel on a single strand of DNA, with the result that many separate (but 
identical) strands of mRNA are being produced, each delayed with respect to the other by the 
time required for the DNA to slide from one RNA polymerase to the next. At the same time,

time required for the DNA to slide from one RNA polymerase to the next. At the same time, 
there can be several different ribosomes working on each of the parallel emerging mRNA's. Thus one arrives at a double-decker, or two-tiered, "molecular canon" (Fig. 98). The 
corresponding image in music is a rather fanciful but amusing scenario: several 
     
FIGURE 98. Here, an even more complex scheme. Not just one but several strands of 
mRNA, all emerging by transcription from a single strand of DNA, are acted upon by polyribosomes. The result is a two-tiered molecular canon. [From Hanawalt and Haynes, The 
Chemical Basis of Life, p. 271]

Self-Rep and Self-Rep 523different copyists are all at work simultaneously, each one of them copying the same original 
manuscript from a clef which flutists cannot read into a clef which they can read. As each 
copyist finishes a page of the original manuscript, he passes it on to the next copyist, and 
starts transcribing a new page himself. Meanwhile, from each score emerging from the pens 
of the copyists, a set of flutists are reading and tooting the melody, each flutist delayed with 
respect to the others who are reading from the same sheet. This rather wild image gives, perhaps, an idea of some of the complexity of the processes which are going on in each and 
every cell of your body during every second of every day ... 
 
Which Came First-The Ribosome or the Protein? 
 
We have been talking about these wonderful beasts called ribosomes; but what are they 
themselves composed of? How are they made? Ribosomes are composed of two types of

themselves composed of? How are they made? Ribosomes are composed of two types of 
things: (1) various kinds of proteins, and (2) another kind of RNA, called ribosomal RNA 
(rRNA). Thus, in order for a ribosome to be made, certain kinds of proteins must be present, 
and rRNA must be present. Of course, for proteins to be present, ribosomes must be there to make them. So how do you get around the vicious circle? Which comes first-the ribosome or 
the protein? Which makes which? Of course there is no answer because one always traces 
things back to previous members of the same class just as with the chicken-and-the-egg

things back to previous members of the same class just as with the chicken-and-the-egg 
question-until everything vanishes over the horizon of time. In any case, ribosomes are made 
of two pieces, a large and a small one, each of which contains some rRNA and some proteins. Ribosomes are about the size of large proteins; they are much much smaller than the strands 
of mRNA which they take as input, and along which they move. 
 
Protein Function 
 
We have spoken somewhat of the structure of proteins-specifically enzymes; but we have not 
really mentioned the kinds of tasks which they perform in the cell, nor how they do them. All

really mentioned the kinds of tasks which they perform in the cell, nor how they do them. All 
enzymes are catalysts, which means that in a certain sense, they do no more than selectively 
accelerate various chemical processes in the cell, rather than make things happen which 
without them never could happen. An enzyme realizes certain pathways out of the myriad myriad potentialities. Therefore, in choosing which enzymes shall be present, you choose 
what shall happen and what shall not happen-despite the fact that, theoretically speaking, 
there is a nonzero probability for any cellular process to happen spontaneously, without the 
aid of catalysts.

aid of catalysts. 
    Now how do enzymes act upon the molecules of the cell? As has been mentioned, enzymes are folded-up polypeptide chains. In every enzyme, there is a cleft or pocket or 
some other clearly-defined surface feature where the enzyme hinds to some other kind of 
molecule. This location is

Self-Rep and Self-Rep 524called its active site, and any molecule which gets bound there is called a substrate. Enzymes 
may have more than one active site, and more than one substrate. Just as in Typogenetics, 
enzymes are indeed very choosy about what they will operate upon. The active site usually is 
quite specific, and allows just one kind of molecule to bind to it, although there are 
sometimes "decoys"-other molecules which can fit in the active site and clog it up, fooling 
the enzyme and in fact rendering it inactive. 
Once an enzyme and its substrate are bound together, there is some disequilibrium of 
electric charge, and consequently charge-in the form of electrons and protons-flows around 
the bound molecules and readjusts itself. By the time equilibrium has been reached, some 
rather profound chemical changes may have occurred to the substrate. Some examples are

rather profound chemical changes may have occurred to the substrate. Some examples are 
these: there may have been a "welding", in which some standard small molecule got tacked onto a nucleotide, amino acid, or other common cellular molecule; a DNA strand may have 
been "nicked" at a particular location; some piece of a molecule may have gotten lopped off; 
and so forth. In fact, bio-enzymes do operations on molecules which are quite similar to the 
typographical operations which Typo-enzymes perform. However, most enzymes perform

typographical operations which Typo-enzymes perform. However, most enzymes perform 
essentially only a single task, rather than a sequence of tasks. There is one other striking difference between Typoenzymes and bio-enzymes, which is this: whereas Typo-enzymes 
operate only on strands, bio-enzymes can act on DNA, RNA, other proteins, ribosomes, cell 
membranes-in short, on anything and everything in the cell. In other words, enzymes are the 
universal mechanisms for getting things done in the cell. There are enzymes which stick

universal mechanisms for getting things done in the cell. There are enzymes which stick 
things together and take them apart and modify them and activate them and deactivate them and copy them and repair them and destroy them .. . 
Some of the most complex processes in the cell involve "cascades" in which a single 
molecule of some type triggers the production of a certain kind of enzyme; the manufacturing 
process begins and the enzymes which come off the "assembly line" open up a new chemical

process begins and the enzymes which come off the "assembly line" open up a new chemical 
pathway which allows a second kind of enzyme to be produced. This kind of thing can go on for three or four levels, each newly produced type of enzyme triggering the production of 
another type. In the end a "shower" of copies of the final type of enzyme is produced, and all 
of the copies go off and do their specialized thing, which may be to chop up some "foreign" 
DNA, or to help make some amino acid for which the cell is very "thirsty", or whatever. 
  
Need for a Sufficiently Strong Support System

Need for a Sufficiently Strong Support System 
 
Let us describe nature's solution to the puzzle posed for Typogenetics: "What kind of strand 
of DNA can direct its own replication?" Certai nly not every strand of DNA is inherently a 
self-rep. The key point is this: any strand whic h wishes to direct its own copying must 
contain directions for assembling precisely those enzymes which can carry out the task. Now 
it is futile to hope that a strand of DNA in isolation could be a self-rep; for in

Self-Rep and Self-Rep 525order for those potential proteins to be pulled out of the DNA, there must not only be 
ribosomes, but also RNA polymerase, which makes the mRNA that gets transported to the 
ribosomes. And so we have to begin by assuming a kind of "minimal support system" just 
sufficiently strong that it allows transcription and translation to be carried out. This minimal 
support system will thus consist in (1) some proteins, such as RNA polymerase, which allow 
mRNA to be made from DNA, and (2) some ribosomes.  
How DNA Self-Replicates 
 
It is not by any means coincidental that the phrases "sufficiently strong support system" and 
"sufficiently powerful formal system" sound alike. One is the precondition for a self-rep to arise, the other for a self-ref to arise. In fact there is in essence only one phenomenon going 
on in two very different guises, and we shall explicitly map this out shortly. But before we do 
so, let us finish the description of how a strand of DNA can be a self-rep.

so, let us finish the description of how a strand of DNA can be a self-rep. 
The DNA must contain the codes for a set of proteins which will copy it. Now there 
is a very efficient and elegant way to copy a double-stranded piece of DNA, whose two strands are complementary. This involves two steps: 
 
(1) unravel the two strands from each other; 
(2) mate" a new strand to each of the two new single strands. 
 This process will create two new double strands of DNA, each identical to the original one. 
Now if our solution is to be based on this idea, it must involve a set of proteins, coded for in

Now if our solution is to be based on this idea, it must involve a set of proteins, coded for in 
the DNA itself, which will carry out these two steps. 
It is believed that in cells, these two steps are performed together in a coordinated 
way, and that they require three principal enzymes: DNA endonuclease, DNA polymerase, and DNA ligase. The first is an "unzipping enzy me": it peels the two original strands apart 
for a short distance, and then stops. Then the other two enzymes come into the picture. The 
DNA polymerase is basically a copy-and-move  enzyme: it chugs down the short single

DNA polymerase is basically a copy-and-move  enzyme: it chugs down the short single 
strands of DNA, copying them complementarily in a fashion reminiscent of the Copy mode 
in Typogenetics. In order to copy, it draws on raw materials-specifically nucleotides-which are floating about in the cytoplasm. Because the action proceeds in fits and starts, with some 
unzipping and some copying each time, some short gaps are created, and the DNA ligase is 
what plugs them up. The process is repeated over and over again. This precision three-

enzyme machine proceeds in careful fashion all the way down the length of the DNA 
molecule, until the whole thing has been peeled apart and simultaneously replicated, so that there are now two copies of it.

Self-Rep and Self-Rep 526Comparison of DNA's Self-Rep Method with Quining 
 
Note that in the enzymatic action on the DNA strands, the fact that information is stored in 
the DNA is just plain irrelevant; the enzymes are merely carrying out their symbol-shunting 
functions, just like rules of inference in the MMIU-system. It is of no interest to the three 
enzymes that at some point they are actually copying the very genes which coded for them. The DNA, to them, is just a template without meaning or interest. 
It is quite interesting to compare this with the Quine sentence's method of describing 
how to construct a copy of itself. There, too, one has a sort of "double strand"-two copies of 
the same information, where one copy acts as instructions, the other as template. In DNA, the 
process is vaguely parallel, since the three enzymes (DNA endonuclease, DNA polymerase, DNA ligase) are coded for in just one of the two strands, which therefore acts as program,

process is vaguely parallel, since the three enzymes (DNA endonuclease, DNA polymerase, DNA ligase) are coded for in just one of the two strands, which therefore acts as program, 
while the other strand is merely a template. The parallel is not perfect, for when the copying 
is carried out, both strands are used as template, not just one. Nevertheless, the analogy is 
highly suggestive. There is a biochemical analogue to the use-mention dichotomy: when 
DNA is treated as a mere sequence of chemicals to be copied, it is like mention of typographical symbols; when DNA is dictating wh at operations shall he carried out, it is like 
use of typographical symbols. 
 
Levels of Meaning of DNA

Levels of Meaning of DNA 
 
There are several levels of meaning which can be read from a strand of DNA, depending on 
how big the chunks are which you look at, and how powerful a decoder you use. On the 
lowest level, each DNA strand codes for an e quivalent RNA strand-the process of decoding 
being transcription. If one chunks the DNA into triplets, then by using a "genetic decoder", one can read the DNA as a sequence of amino acids. This is translation (on top of 
transcription). On the next natural level of the hierarchy, DNA is readable as a code for a set

transcription). On the next natural level of the hierarchy, DNA is readable as a code for a set 
of proteins. The physical pulling-out of proteins from genes is called gene expression. 
Currently, this is the highest level at which we understand what DNA means. 
However, there are certain to be higher levels of DNA meaning which are harder to 
discern. For instance, there is every reason to believe that the DNA of, say, a human being 
codes for such features as nose shape, music talent, quickness of reflexes, and so on. Could 
one, in principle, learn to read off such pieces of information directly from a strand of DNA,

one, in principle, learn to read off such pieces of information directly from a strand of DNA, 
without going through the actual physical process of epigenesis-the physical pulling-out of 
phenotype from genotype Presumably, yes, since-in theory-one could have an incredibly powerful computer program simulating the entire process, including every cell, every protein, 
every tiny feature involved in the replication of DNA, of cells, to the bitter end. The output of 
such a pseudo-epigenesis program would be a high-level description of the phenotype.

Self-Rep and Self-Rep 527 
     
   There is another (extremely faint) possibility: that we could learn to read the phenotype off 
of the genotype without doing an isomorphic simulation of the physical process of 
epigenesis, but by finding some simpler sort of decoding mechanism. This could be called "shortcut pseudoepigenesis". Whether shortcut or not, pseudo-epigenesis is, of course, totally 
beyond reach at the present time-with one notable  exception: in the species Felis catus, deep 
probing has revealed that it is indeed possible to read the phenotype directly off of the 
genotype. The reader will perhaps better appreciate this remarkable fact after directly 
examining the following typical section of the DNA of Felis catus:  
CATCATCATCATCATCATCATCATCATCAT ... 
 
Below is shown a summary of the levels of DNA-readability, together with the names of the 
different levels of decoding. DNA can be read as a sequence of:  
bases (nucleotides)        transcription 
amino acids         translation

amino acids         translation 
proteins (primary structure)                    . gene expression 
proteins (tertiary structure) protein clusters      higher levels of gene expression 
… 
….  unknown levels of DNA meaning 
…. 
(N-1)  ???? (N) physical, mental, and 
psychological traits       pseudo-epigenesis 
 
The Central Dogmap 
. 
With this background, now we are in a position to draw an elaborate comparison between F. 
Crick's "Central Dogma of Molecular Biology" (.DOGMA I) upon which all cellular 
processes are based; and what I, with poetic license, call the "Central Dogma of

… 
…. 
 
. 
processes are based; and what I, with poetic license, call the "Central Dogma of 
Mathematical Logic" (.DOGMA II), upon which G6del's Theorem is based. The mapping from one onto the other is laid out in Figure 99 and the following chart, which together 
constitute the Central Dogmap. 
 
FIGURE 99. The Central Dogmap. An analogy is established between two fundamental 
Tangled Hierarchies: that of molecular biology and that of mathematical logic.

Self-Rep and Self-Rep 529Note the base-pairing of A and T (Arithmetization and Translation), as well as of G 
and C (Godel and Crick). Mathematical logic gets the purine side, and molecular biology gets 
the pyrimidine side. 
To complete the esthetic side of this mapping, I chose to model my Godel-numbering 
scheme on the Genetic Code absolutely faithfully. In fact, under the following 
correspondence, the table of the Genetic Code becomes the table of the Godel Code:  
 
Each amino acid-of which there are twenty-corresponds to exactly one symbol of TNT-of 
which there are twenty. Thus, at last, my motive for concocting "austere TNT" comes out-so 
that there would be exactly twenty symbols! The Godel Code is shown in Figure 100. Compare it with the Genetic Code (Fig. 94). 
There is something almost mystical in seeing the deep sharing of such an abstract 
structure by these two esoteric, yet fundamental, advances in knowledge achieved in our

structure by these two esoteric, yet fundamental, advances in knowledge achieved in our 
century. This Central Dogmap is by no means a rigorous proof of identity of the two theories; 
but it clearly shows a profound kinship, which is worth deeper exploration.  
Strange Loops in the Central Dogmap 
 
One of the more interesting similarities between the two sides of the map is the way in which 
"loops" of arbitrary complexity arise on the top level of both: on the left, proteins which act on proteins which act on proteins and so on, ad infinitum; and on the right, statements about

"loops" of arbitrary complexity arise on the top level of both: on the left, proteins which act on proteins which act on proteins and so on, ad infinitum; and on the right, statements about 
statements about statements of meta-TNT and so on, ad infinitum. These are like 
heterarchies, which we discussed in Chapter V, where a sufficiently complex substratum 
allows high-level Strange Loops to occur and to cycle around, totally sealed off from lower 
levels. We will explore this idea in greater detail in Chapter XX. 
Incidentally, you may be wondering about this question: "What, according to the 
Central Dogmap, is Godel's Incompleteness Theorem itself mapped onto?" This is a good

Central Dogmap, is Godel's Incompleteness Theorem itself mapped onto?" This is a good 
question to think about before reading ahead. 
 
The Central Dogmap and the Contracrostipuntus 
 
    It turns out that the central dogmap is quite similar to the mapping that was laid out in 
Chapter IV between the Contracrostipunctus and Godel’s Theorem. One can therefore drew 
parallels between all three systems.

Self-Rep and Self-Rep 530
  
FIGURE 100. The Godel Code. Under this Godel-numbering scheme, each T V7- s'mbol gets 
one or more codons. The small ovals show how this table subsumes the earlier 
Godelnumhering table of Chapter IX. 
 (1) formal systems and strings 
(2) cells and strands of DNA 
(3) record players and records 
 
 In the following chart, the mapping between systems 2 and 3 is explained carefully

Self-Rep and Self-Rep 531
 
The analogue of Godel's Theorem is seen to be a peculiar fact, probably little useful to 
molecular biologists (to whom it is likely quite obvious): 
 
It is always possible to design a strand of DNA which, if injected into a cell, would, upon being transcribed, cause such proteins to be manufactured as would destroy the cell 
(or the DNA), and thus result in the non-reproduction of that DNA 
 
This conjures tip a somewhat droll scenario, at least if taken in light of evolution: an invading 
species of virus enters a cell by some surreptitious

Self-Rep and Self-Rep 532
       
FIGURE 101. The T4 bacterial virus is an assembly of protein components (a). The "head" is 
a protein membrane, shaped like a kind of prolate irosahedron with thirty facets and filled with DNA. It is attached by a neck to a tail consisting of a hollow core surrounded by a 
contractile sheathh and based on a spiked end plate to which six fibers are attached. The 
spikes and fibers affix the virus to a bacterial cell wall (h). The sheath contracts, driving the 
core through the wall, and viral DNA enters the cell. [From Hanawalt and Haynes, The 
Chemical Basis of Life, p. 230.1  
 
means, and then carefully ensures the manufacture of proteins which will have the effect of 
destroying the virus itself! It is a sort of suicide-or Epimenides sentence, if you will-on the 
molecular level. Obviously it would not prove advantageous from the point of view of survival of the species. However, it demonstrates the spirit, if not the letter, of the

molecular level. Obviously it would not prove advantageous from the point of view of survival of the species. However, it demonstrates the spirit, if not the letter, of the 
mechanisms of protection and subversion which cells and their invaders have developed. 
 
E. Coli vs. T4 
 
Let us consider the biologists' favorite cell, that of the bacterium Escherichia coli (no relation 
to M. C. Escher), and one of their favorite invaders of that cell: the sinister and eerie T4 
phage, pictures of which you can see in Figure 101. (Incidentally, the words "phage" and 
"virus" are synonymous and mean "attacker of bacterial cells".) The weird tidbit looks like a

"virus" are synonymous and mean "attacker of bacterial cells".) The weird tidbit looks like a 
little like a cross between a LEM (Lunar Excursion Module) and a mosquito-and it is much more sinister than the latter. It has a "head" wherein is stored all its "knowledge"-namely its 
DNA; and it has six "legs" wherewith to fasten itself to the cell it has chosen to invade; and it 
has a "stinging tube" (more properly called its "tail") like a mosquito. The major difference is 
that unlike a mosquito, which uses its stinger for sucking blood, the T4 phage uses its stinger

that unlike a mosquito, which uses its stinger for sucking blood, the T4 phage uses its stinger 
for injecting its hereditary substance into th e cell against the will of its victim. Thus the 
phage commits "rape" on a tiny scale.

Self-Rep and Self-Rep 533
  
FIGURE 102. Viral infection begins when viral DNA enters a bacterium. Bacterial DNA is 
disrupted and viral DNA replicated. Synthesis of viral structural proteins and their assembly 
into virus continues until the cell bursts, releasing particles. [From Hanawalt and Haynes, 
The Chemical Basis of Life, p. 230.]  
A Molecular Trojan Horse 
 
What actually happens when the viral DNA enters a cell? The virus "hopes", to speak 
anthropomorphically, that its DNA will get exactly the same treatment as the DNA of the host cell. This would mean getting transcribed and translated, thus allowing it to direct the 
synthesis of its own special proteins, alien to the host cell, which will then begin to do their 
thing. This amounts to secretly transporting alien proteins "in code" (viz., the Genetic Code) 
into the cell, and then "decoding" (viz., producing) them. In a way this resembles the story of

into the cell, and then "decoding" (viz., producing) them. In a way this resembles the story of 
the Trojan horse, according to which hundreds of soldiers were sneaked into Troy inside a harmless seeming giant wooden horse; but once inside the city, they broke loose and captured 
it. The alien proteins, once they have been "decoded" (synthesized) from their carrier DNA, 
now jump into action. The sequence of actions directed by the T4 phage has been carefully 
studied, and is more or less as follows (see also Figs. 102 and 103): 
 Time elapsed Action taking place 
 
0 min.  Injection of viral DNA.

0 min.  Injection of viral DNA. 
 
1 min.  Breakdown of host DNA. Cessation of production of native proteins and 
initiation of production of alien (T4) proteins. Among the earliest produced 
proteins are those which direct the replication of the alien (T4) DNA. 
 
5 min.  Replication of viral DNA begins. 
 8 min. Initiation of production of structural proteins which will form the "bodies" of 
new phages.

Self-Rep and Self-Rep 53513 min.  First complete replica of T4 invader is produced. 
25 min.  Lysozyme (a protein) attacks host cell wall. breaking open the bacterium, and 
the "bicentuplets" emerge. 
 
Thus, when a T4 phage invades an E. coli cell, after the brief span of about twenty-four or 
twenty-five minutes, the cell has been completely subverted, and breaks open. Out pop about two hundred exact copies of the original virus-"bicentuplets"-ready to go attack more 
bacterial cells, the original cell having been largely consumed in the process. 
Although from a bacterium's point of view this kind of thing is a deadly serious 
menace, from our large-scale vantage point it can be looked upon as an amusing game 
between two players: the invader, or "T" player (named after the T-even class of phages, including the T2, T4, and others), and the "C" player (standing for "Cell"). The objective of 
the T player is to invade and take over the cell of the C player from within, for the purpose of

the T player is to invade and take over the cell of the C player from within, for the purpose of 
reproducing itself. The objective of the C player is to protect itself and destroy the invader. 
When described this way, the molecular TC-game can be seen to be quite parallel to the 
macroscopic TC-game described in the preceding Dialogue. (The reader can doubtless figure out which player-T or C-corresponds to the Tortoise, and which to the Crab.) 
 
Recognition, Disguises, Labeling 
. 
This "game" emphasizes the fact that recognition is one of the central themes of cellular and

. 
This "game" emphasizes the fact that recognition is one of the central themes of cellular and 
subcellular biology. How do molecules (or higher-level structures) recognize each other? It is 
essential for the functioning of enzymes that they should be able to latch onto special 
"binding sites" on their substrates; it is essential that a bacterium should be able to distinguish 
its own DNA from that of phages; it is essential that two cells should be able to recognize

. 
its own DNA from that of phages; it is essential that two cells should be able to recognize 
each other and interact in a controlled way. Such recognition problems may remind you of the original, key problem about formal systems: How can you tell if a string has, or does not 
have, some property such as theoremhood? Is there a decision procedure? This kind of 
question is not restricted to mathematical logic: it permeates computer science and, as we are 
seeing, molecular biology. 
The labeling technique described in the Dialogue is in fact one of E. colt's tricks for

The labeling technique described in the Dialogue is in fact one of E. colt's tricks for 
outwitting its phage invaders. The idea is that strands of DNA can be chemically labeled by 
tacking on a small molecule-methyl-to various nucleotides. Now this labeling operation does 
not change the usual biological properties of the DNA; in other words, methylated (labeled) 
DNA can be transcribed just as well as unmethylated (unlabeled) DNA, and so it can direct 
the synthesis of proteins. But if the host cell has some special

Self-Rep and Self-Rep 536 
    
mechanisms for examining whether DNA is labeled or not, then the label may make all the 
difference in the world. In particular, the host cell may have an enzyme system which looks 
for unlabeled DNA, and destroys any that it finds by unmercifully chopping it to pieces. In that case, woe to all unlabeled invaders! 
The methyl labels on the nucleotides have been compared to serifs on letters. Thus, 
using this metaphor, we could say that the E. colt cell is looking for DNA written in its 
"home script", with its own particular typeface-and will chop up any strand of DNA written 
in an "alien" typeface. One counterstrategy, of course, is for phages to learn to label themselves, and thereby become able to fool the cells which they are invading into 
reproducing them. 
This TC-battle can continue to arbitrary levels of complexity, but we shall not pursue 
it further. The essential fact is that it is a ba ttle between a host which is trying to reject all

it further. The essential fact is that it is a ba ttle between a host which is trying to reject all 
invading DNA, and a phage which is trying to infiltrate its DNA into some host which will transcribe it into mRNA (after which its reproduction is guaranteed). Any phage DNA which 
succeeds in getting itself reproduced this way can be thought of as having this high-level 
interpretation: "I Can Be Reproduced in'Cells of Type X". This is to be distinguished from 
the evolutionarily pointless kind of phage mentioned earlier, which codes for proteins that

the evolutionarily pointless kind of phage mentioned earlier, which codes for proteins that 
destroy it, and whose high-level interpretation is the self-defeating sentence: "I Cannot Be Reproduced in Cells of Type X". 
 
Henkin Sentences and Viruses 
 
Now both of these contrasting types of sel f-reference in molecular biology have their 
counterparts in mathematical logic. We have already discussed the analogue of the self-
defeating phages-namely, strings of the G6del type, which assert their own unproducibility 
within specific formal sstems. But one can also make a counterpart sentence to a real phage:

within specific formal sstems. But one can also make a counterpart sentence to a real phage: 
the' phage asserts its own producibility in a specific cell, and the sentence asserts its own 
producibility in a specific formal system. Sentences of this type are called Henkin sentences, after the mathematical logician Leon Henkin. They can be constructed exactly along the lines 
of Godel sentences, the only difference being the omission of a negation. One begins with an 
"uncle", of course: 
 
3a:3a':<TNT-PROOF-PAIR{a,a'}-°ARITHMOQUINE{a",a'}>  
and then proceeds by the standard trick. Say the Godel number of the above "uncle" is h. 
Now by arithmoquining this very uncle, you get a Henkin sentence:

Now by arithmoquining this very uncle, you get a Henkin sentence: 
 
3a:3a':<TNT-PROOF-PAIR{a,a'} ^ ARITHMOQUINE{SSS ... SSSO/a",a'}> M

Self-Rep and Self-Rep 537 (By the way, can you spot how this sentence differs from -G?) The reason I show it 
explicitly is to point out that a Henkin sentence does not give a full recipe for its own 
derivation; it just asserts that there exists one. You might well wonder whether its claim is 
justified. Do Henkin sentences indeed possess derivations? Are they, as they claim, 
theorems? It is useful to recall that one need not believe a politician who says. "I am honest"-
he may be honest, and yet he may not be. Are Henkin sentences any more trustworthy than politicians? Or do Henkin sentences, like politicians, lie in cast-iron sinks? 
It turns out that these Henkin sentences are invariably truth tellers. Why this is so is 
not obvious; but we will accept this curious fact without proof. 
 
Implicit vs. Explicit Henkin Sentences 
 
I mentioned that a Henkin sentence tells nothing a bout its own derivation; it just asserts that

I mentioned that a Henkin sentence tells nothing a bout its own derivation; it just asserts that 
one exists. Now it is possible to invent a variation on the theme of Henkin sentences-namely 
sentences which explicitly describe their own derivations. Such a sentence's high-level 
interpretation would not be "Some Sequence of Strings Exists Which is a Derivation of Me", but rather, "The Herein-described Sequence of Strings  Is a Derivation of Me". Let us 
call the first type of sentence an implicit Henkin sentence. The new sentences will be called 
explicit Henkin sentences, since they explicitly describe their own derivations. Note that,

explicit Henkin sentences, since they explicitly describe their own derivations. Note that, 
unlike their implicit brethren, explicit Henkin sentences need not be theorems. In fact, it is 
quite easy to write a string which asserts that its own derivation consists of the single string 0=0-a false statement, since 0=0 is not a derivation of anything. However, it is also possible 
to write an explicit Henkin sentence which is a theorem-that is, a sentence which in fact gives 
a recipe for its own derivation. 
 
Henkin Sentences and Self-Assembly 
 
The reason I bring up this distinction between explicit and implicit Henkin sentences is that it

The reason I bring up this distinction between explicit and implicit Henkin sentences is that it 
corresponds very nicely to a significant distinction between types of virus. There are certain 
viruses, such as the so-called "tobacco mosaic virus", which are called self-assembling 
viruses; and then there are others, such as our favorite T-evens, which are non-self-assembling. Now what is this distinction? It is a direct analogue to the distinction between 
implicit and explicit Henkin sentences. 
The DNA of a self-assembling virus codes only for the parts of a new virus, but not

The DNA of a self-assembling virus codes only for the parts of a new virus, but not 
for any enzymes. Once the parts are produced, the sneaky virus relies upon them to link up to 
each other without help from any enzymes. Such a process depends on chemical affinities which the parts have for each other, when swimming in the rich chemical brew of a cell. Not 
only viruses, but also some organelles-such as ribosomes-assemble

Self-Rep and Self-Rep 538 
     
themselves. Somtiems enzymes may be needed – but in such cases, they are recruited from 
the host cell, and enslaved. This is what is meant by self-assembly. 
By contrast, the DNA of more complex viruses, such as the T-evens, codes not only 
for the parts, but in addition for various enzymes which play special roles in the assembly of 
the parts into wholes. Since the assembly process is not spontaneous but requires "machines", 
such viruses are not considered to be self-assembling. The essence of the distinction, then, 
between self-assembling units and non-self-assembling units is that the former get away with 
self-reproduction without telling the cell anything about their construction, while the latter need to give instructions as to how to assemble themselves. 
Now the parallel to Henkin sentences, implicit and explicit, ought to be quite clear. 
Implicit Henkin sentences are self-proving but do not tell anything at all about their proofs-

they are analogous to self-assembling viruses; explicit Henkin sentences direct the 
construction of their own proofs-they are analogous to more complex viruses which direct their host cells in putting copies of themselves together. 
The concept of self-assembling biological structures as complex as viruses raises the 
possibility of complex self-assembling machines as well. Imagine a set of parts which, when 
placed in the proper supporting environment, spontaneously group themselves in such a way

placed in the proper supporting environment, spontaneously group themselves in such a way 
as to form a complex machine. It seems unlikely, yet this is quite an accurate way to describe the process of the tobacco mosaic virus' method of selfreproduction via self-assembly. The 
information for the total conformation of the organism (or machine) is spread about in its 
parts; it is not concentrated in some single place. 
Now this concept can lead in some strange directions, as was shown in the Edifying

Now this concept can lead in some strange directions, as was shown in the Edifying 
Thoughts of a Tobacco Smoker. There, we saw how the Crab used the idea that information for self-assembly can be distributed around, instead of being concentrated in a single place. 
His hope was that this would prevent his new phonographs from succumbing to the Tortoise's 
phonograph-crashing method. Unfortunately, just as with the most sophisticated axiom 
schemata, once the system is all built and packaged into a box, its well-definedness renders it

schemata, once the system is all built and packaged into a box, its well-definedness renders it 
vulnerable to a sufficiently clever "Godelizer"; and that was the sad tale related by the Crab. Despite its apparent absurdity, the fantastic scenario of that Dialogue is not so far from 
reality, in the strange, surreal world of the cell. 
 
Two Outstanding Problems: 
Differentiation and Morphogenesis 
 
Now self-assembly may be the trick whereby certain subunits of cells are constructed, and 
certain viruses-but what of the most complex macroscopic structures, such as the body of an

certain viruses-but what of the most complex macroscopic structures, such as the body of an 
elephant or a spider, or the shape of a Venus's-Hyt-ap? How are homing instincts built into the brain of" a

Self-Rep and Self-Rep 539 
bird, or hunting instincts into the brain of a dog% In short, how is it that merely by dictating 
which proteins are to be produced in cells, DNA exercises such spectacularly precise control 
over the exact structure and function of macroscopic living objects? There are two major 
distinct problems here. One is that of cellular differentiation: how do different cells, sharing 
exactly the same DNA, perform different roles- such as a kidney cell, a bone marrow cell, and 
a brain cell? The other is that of morphogenesis ("birth of form"): how does intercellular 
communication on a local level give rise to large-scale, global structures and organizations-
such as the various organs of the body, the shape of the face, the suborgans of the brain, and 
so on? Although both cellular differentiation and rnorphogenesis are poorly understood at

so on? Although both cellular differentiation and rnorphogenesis are poorly understood at 
present. the trick appears to reside in exquisitely fine-tuned feedback and "feedforward" mechanisms within cells and between cells, which tell a cell when to "turn on" and when to 
"turn off" production of various proteins. 
 
Feedback and Feedforward 
 
Feedback takes place when there is too much or too little of some desired substance in the 
cell: then the cell must somehow regulate the production line which is assembling that 
substance. Feedforward also involves the regulation of" an assembly line, but not according

substance. Feedforward also involves the regulation of" an assembly line, but not according 
to the amount of end product present: rather, according to the amount of some precursor of 
the end product of that assembly line. There are two major devices for achieving negative feedforward or feedback. One way is to prevent the relevant enzymes from being able to 
perform-that is, to "clog up" their active sites. This is called inhibition. The other way is to 
prevent the relevant enzymes from ever being manufactured! This is called repression. 
Conceptually, inhibition is simple: you just block up the active site of the first enzyme in the 
assembly line, and the whole process of synthesis gets stopped dead.  
Repressors and Inducers

Repressors and Inducers 
 
Repression is trickier. How does a cell stop a gene from being expressed? The answer is, it 
prevents it from ever getting transcribed. This means that it has to prevent RNA polymerase from doing its job. This can be accomplished by placing a huge obstacle in its path, along the 
DNA. precisely in front of that gene which the cell wants not to get transcribed. Such 
obstacles do exist, and are called repressors. They are themselves proteins, and they bind to 
special obstacle-holding sites on the DNA, called (I am not sure why) operators. An operator

special obstacle-holding sites on the DNA, called (I am not sure why) operators. An operator 
therefore is a site of control for the gene (or genes) which immediately follow it: those genes are called its operon. Because a series of enzymes often act in concert in carrying out a long 
chemical transformation, they are often coded for in sequence; and this is why operons often 
contain several genes, rather than just one. The effect of the successful repression of an 
operon is that a whole series of genes is

Self-Rep and Self-Rep 540 
 
prevented from being transcribed, which means that a whole set of related enzymes remains 
unsynthesized. 
What about positive feedback and feedforward? Here again, there are two options: (1) 
unclog the clogged enzymes, or (2) stop the repression of the relevant operon. (Notice how 
nature seems to love double-negations! Probably there is some very deep reason for this.) 
The mechanism by which repression is repressed involves a class of molecules called 
inducers. The role of an inducer is simple: it combines with a repressor protein before the 
latter has had a chance to bind to an operator on a DNA molecule; the resulting "repressor-inducer complex" is incapable of binding to an operator, and this leaves the door open for the 
associated operon to be transcribed into mRNA and subsequently translated into protein. 
Often the end product or some precursor of the end product can act as an 
inducer. 
 
Feedback and Strange Loops Compared

inducer. 
 
Feedback and Strange Loops Compared 
 
Incidentally, this is a good time to distinguish between simple kinds of feedback, as in the 
processes of inhibition and repression, and the looping-hack between different informational 
levels, shown in the Central Dogrnap. Both are "feedback" in some sense; but the latter is much deeper than the former. When an amino acid, such as tryptophan or isoleucine, acts as 
feedback (in the form of an inducer) by binding to its repressor so that more of it gets made,

feedback (in the form of an inducer) by binding to its repressor so that more of it gets made, 
it is not telling how to construct itself; it is just telling enzymes to make more of it. This 
could be compared to a radio's volume, which, when fed through a listener's ears, may cause 
itself to be turned down or up. This is another thing entirely from the case in which the broadcast itself tells you explicitly to turn your radio on or off, or to tune to another 
wavelength-or even how to build another radio! The latter is much more like the looping-

back between informational levels, for here, information inside the radio signal gets 
"decoded" and translated into mental structures. The radio signal is composed of symbolic 
constituents whose symbolic meaning matters-a case of use, rather than mention. On the other hand, when the sound is just too loud, the symbols are not conveying meaning: they are 
merely being perceived as loud sounds, and might as well be devoid of meaning-a case of 
mention, rather than use. This case more resembles the feedback loops by which proteins

mention, rather than use. This case more resembles the feedback loops by which proteins 
regulate their own rates of synthesis. 
It has been theorized that the difference between two neighboring cells which share 
the exact same genotype and yet have different functions is that different segments of their 
genome have been repressed, and therefore they have different working sets of proteins. A 
hypopothesis like this could account for the phenomenal differences between cells in 
different organs of the body of a human being.

Self-Rep and Self-Rep 541 
    
Two Simple Examples of Differentiation 
 
The process by which one initial cell replicates over and over, giving rise to a myriad of differentiated cells with specialized functions, can be likened to the spread of a chain letter 
from person to person, in which each new participant is asked to propagate the message 
faithfully, but also to add some extra personal touch. Eventually, there will be letters which 
are tremendously different from each other. 
Another illustration of the ideas of differentiation is provided by this extremely 
simple computer analogue of a differentiating self-rep. Consider a very short program which 
is controlled by an up-down switch, and which has an internal parameter N-a natural number. 
This program can run in two modes-the up-mode, and the down-mode. When it runs in the 
upmode, it self-replicates into an adjacent part of the computer's memoryexcept it makes the

upmode, it self-replicates into an adjacent part of the computer's memoryexcept it makes the 
internal parameter N of its "daughter" one great er than in itself. When it runs in the down-
mode, it does not self-rep, but instead calculates the number 
 
(-1)'/(2N + 1) 
 
and adds it to a running total. 
Well, suppose that at the beginning, there is one copy of the program in memory, N = 
0, and the mode is up. Then the program will copy itself next door in memory, with N = 1. 
Repeating the process, the new program will self-rep next door to itself, with a copy having

Repeating the process, the new program will self-rep next door to itself, with a copy having 
N = 2. And over and over again ... What happens is that a very large program is growing 
inside memory. When memory is full, the process quits. Now all of memory can be looked upon as being filled with one big program, composed of many similar, but differentiated, 
modules-or "cells". Now suppose we switch the mode to down, and run this big program. 
What happens? The first "cell" runs, and calculates 1/1. The second "cell" runs, calculating -

What happens? The first "cell" runs, and calculates 1/1. The second "cell" runs, calculating - 
1/3, and adding it to the previous result. The third "cell" runs, calculating + 1/5 and adding it 
on. .. The end result is that the whole "organism"-the big program-calculates the sum 
 
l -1/3 +1/5 -117 +1/9 -1/11 +1/13 -1/15 + .. . 
 
to a large number of terms (as many terms as "cells" can fit inside memory). And since this 
series converges (albeit slowlv) to 7r/4, we have a "phenotype" whose function is to calculate the value of a famous mathematical constant. 
 
Level Mixing in the Cell

Level Mixing in the Cell 
 
I hope that the descriptions of processes such as labeling, self-assembly, differentiation, morphogenesis, as well as transcription and translation, have helped to convey some notion 
of the immensely complex system which is a cell-an information-processing system with 
some strikingly

Self-Rep and Self-Rep 542 
novel features. We have seen, in the Central Dogmap, that although we can try to draw a 
clear line between program and data, the distinc tion is somewhat arbitrary. Carrying this line 
of thought further, we find that not only are program and data intricately woven together, but 
also the interpreter of programs, the physical processor, and even the language are included 
in this intimate fusion. Therefore, although it is possible (to some extent) to draw boundaries and separate out the levels, it is just as important-and ust as fascinating-to recognize the 
level-crossings and mixings. Illustrative of this is the amazing fact that in biological systems, 
all the various features necessary for self-rep (viz., language, program, data, interpreter, and 
processor) cooperate to such a degree that all of them are replicated simultaneously-which

processor) cooperate to such a degree that all of them are replicated simultaneously-which 
shows how much deeper is biological self-rep'ing than anything yet devised along those lines by humans. For instance, the self-rep program exhibited at the beginning of this Chapter 
takes for granted the pre-existence of three external aspects: a language, an interpreter, and a 
processor, and does not replicate those. 
Let us try to summarize various ways in which the subunits of a cell can be classified

Let us try to summarize various ways in which the subunits of a cell can be classified 
in computer science terms. First, let us take DNA. Since DNA contains all the information for construction of proteins., which are the active agents of the cell, DNA can be viewed as a 
program written in a higher-level language, which is subsequently translated (or interpreted) 
into the "machine language" of the cell (proteins). On the other hand, DNA is itself a passive 
molecule which undergoes manipulation at the hands of various kinds of enzymes; in this

molecule which undergoes manipulation at the hands of various kinds of enzymes; in this 
sense, a DNA molecule is exactly like a long piece of data, as well. Thirdly, DNA contains the templates off of which the tRNA "flashcards" are rubbed, which means that DNA also 
contains the definition of its own higher-level language. 
Let us move on to proteins. Proteins are active molecules, and carry out all the 
functions of the cell; therefore it is quite appr opriate to think of them as programs in the

functions of the cell; therefore it is quite appr opriate to think of them as programs in the 
"machine language" of the cell (the cell itself being the processor). On the other hand, since proteins are hardware and most programs are software, perhaps it is better to think of the 
proteins as processors. Thirdly, proteins are often acted upon by other proteins, which means 
that proteins are often data. Finally, one can view proteins as interpreters; this involves 
viewing DNA as a collection of' high-level language programs, in which case enzymes are

viewing DNA as a collection of' high-level language programs, in which case enzymes are 
merely carrying out the programs written in the DNA code, which is to say, the proteins are acting as interpreters. 
Then there are ribosomes and tRNA molecules. They mediate the translation from 
DNA to proteins, which can be compared to the translation of a program from a high-level 
language to a machine language; in other words, the ribosomes are functioning as interpreters

language to a machine language; in other words, the ribosomes are functioning as interpreters 
and the tRNA molecules provide the definition of the higher-level language. But an alternative view of translation has it that the ribosomes are processors, while the tRNA's are 
interpreters. 
We have barely scratched the surface in this analysis of interrelations between all 
these biomolecules. What we have seen is that nature feels quite

Self-Rep and Self-Rep 543 
    
comfortable in mixing levels which we tend to see as quite distinct. Actually, in computer 
science there is already a visible tendency to nix all these seemingly distinct aspects of an 
information-processing system. This is particularly so in Artificial Intelligence research, 
which is usually at the forefront of computer language design.  
The Origin of Life 
 
A natural and fundamental question to ask, on learning of these incredibly intricately 
interlocking pieces of software and hardware is: "How did they ever get started in the first place?" It is truly a baffling thing. One has to imagine some sort of a bootstrap process 
occurring, somewhat like that which is used in the development of new computer languages-
but a bootstrap from simple molecules to entire cells is almost beyond one's power to 
imagine. There are various theories on the origin of life. They all run aground on this most

imagine. There are various theories on the origin of life. They all run aground on this most 
central of all central questions: "How did th e Genetic Code, along with the mechanisms for 
its translation (ribosomes and tRNA molecules), originate" For the moment, we will have to 
content ourselves with a sense of wonder and awe, rather than with an answer. And perhaps 
experiencing that sense of wonder and awe is more satisfying than having an answer-at least 
for a while.

The Magnificrab, Indeed, 
 544The Magnificrab, Indeed, 
 
It is spring, and the Tortoise and Ach illes are taking a Sunday promenade in 
the woods together. They have decided to climb a hill at the top of which, it is 
said, there is a wonderful teahouse, with all sorts of delicious pastries. 
 
Achilles: Man oh man! If a crab Tortoise: If a crab?? Achilles: I was about to say, if a crab ever were intelligent, then surely it would be our 
mutual friend the Crab. Why, he must be at l east two times as smart as any crab alive. 
Or maybe even three times as smart as any crab alive. Or perhaps 
Tortoise: My soul! How you magnify the Crab! Achilles: Well, I just happen to be an admirer of his ... Tortoise: No need to apologize. I admire hi m, too. Speaking of Crab admirers, did I tell 
you about the curious fan letter which the Crab received not too long ago? 
Achilles: I don't believe so. Who sent it? Tortoise: It bore a postmark from India, and was from someone neither of us had ever

Achilles: I don't believe so. Who sent it? Tortoise: It bore a postmark from India, and was from someone neither of us had ever 
heard of before-a Mr. Najunamar, I believe. 
Achilles: I wonder why someone who never knew Mr. Crab would send him a letter-or 
for that matter, how they would get his a ddress. Tortoise: Apparently whoever it was 
was under the illusion that the Crab is a mathematician. It contained numerous results, all of which were But, ho! Speak of the devil! Here comes Mr. Crab now, down the 
hill. Crab: Good-bye! It was nice to talk with you again. Well, I guess I had best be

hill. Crab: Good-bye! It was nice to talk with you again. Well, I guess I had best be 
off. But I'm utterly stuffed-couldn't eat one more  bite if I had to! I've just been up there 
myself-recommend it highly. Have you ever been  to the teahouse at the crest of the 
hill? How are you, Achilles? Oh, there's Ach illes. Hello, hello. Well, well, if it isn't 
Mr. T! 
Tortoise: Hello, Mr. C. Are you headed up to the hilltop teahouse? Crab: Why, yes 
indeed, I am; how did you guess it? I'm quite looking forward to some of their special napoleons-scrumptious little morsels. I'm so hungry I could eat a frog. Oh, there's

Mr. T! 
indeed, I am; how did you guess it? I'm quite looking forward to some of their special napoleons-scrumptious little morsels. I'm so hungry I could eat a frog. Oh, there's 
Achilles. How are you, Achilles? 
Achilles: Could be worse, I suppose. Crab: Wonderful! Well, don't let me interrupt your discussion. I'll just tag along. Tortoise: Curiously enough, I was just about to describe your mysterious letter from that 
Indian fellow a few weeks back-but now that  you're here. I'll let Achilles get the story 
from the Crab’s mouth.

 545 
 
FIGURE 104. Castrovalva, by'M. C. Escher (lithograph, 1930).

 546 
 Crab: Well, it was this way. This fellow Naj unamar had apparently never had any formal 
training in mathematics, but had instead worked out some of his own methods for deriving new truths of mathematics. Some of  his discoveries defeated me completely; 
I had never seen anything in the least like them before. For instance, he exhibited a map of India that he had managed to color using no fewer than 1729 distinct colors. 
Achilles: 1729! Did you say 1729? Crab: Yes-why do you ask? Achilles: Well, 1729 is a very interesti ng number, you know. Crab: Indeed. I wasn't 
aware of it. 
Achilles: In particular, it so happens that 1729 is the number of the taxicab which I took 
to Mr. Tortoise's this morning! 
Crab: How fascinating! Could you possibly te ll me the number of the trolley car which 
you'll take to Mr. Tortoise's tomorrow morning? 
Achilles (after a moment's thought): It's not obvious to me; however, I should think it 
would be very large.

would be very large. 
Tortoise: Achilles has a wonderful intuition for these things. Crab: Yes. Well, as I was saying, Najunamar in his letter also proved that every even 
prime is the sum of two odd numbers, and that there are no solutions in positive 
integers to the equation 
 
a
n + bn = cn for n = 0. 
 Achilles: What? All these old classics of mathematics resolved in one fell swoop? He 
must be a genius of the first rank! Tort oise: But Achilles-aren't you even in the 
slightest skeptical? 
Achilles: What? Oh, yes-skeptical. Well, of course I am. You don't think I believe that

a
Achilles: What? Oh, yes-skeptical. Well, of course I am. You don't think I believe that 
Mr. Crab got such a letter, do you? I don't fall for just anything, you know. So it must 
have been 5'ou, Mr. T, who received the letter! 
Tortoise: Oh, no, Achilles, the part about Mr. C receiving the letter is quite true. What I 
meant was, aren't you skeptical about the content of the letter-its extravagant claims? 
Achilles: Why should I be? Hmm ... Well, of c ourse I am. I'm a very skeptical person, as 
both of you should well know by now. It's very  hard to convince me of anything, no 
matter how true or false it is.

matter how true or false it is. 
Tortoise: Very well put, Achilles. You certainly  have a first-class awareness of your own 
mental workings. 
Achilles: Did it ever occur to you, my friends, that these claims of Najunamar might be 
incorrect? 
Crab: Frankly, Achilles, being rather c onservative and orthodox myself, I was a bit 
concerned about that very point on first receivi ng the letter. In fact, I suspected at first 
that here was an out-and-out fraud. But on second thought, it occurred to me that not 
many types of people could manufacture su ch strange-sounding and complex results

many types of people could manufacture su ch strange-sounding and complex results 
purely from their imagination. In fact, what it boiled down to was this question:

 547"Which is the more likely: a charlatan of such extraordinary ingenuity, or a 
mathematician of great genius?" And before long, I realized that the probabilities clearly 
favored the former. 
Achilles: Didn't you directly checkout any of his amazing claims, however? Crab: Why should I? The probability argument was the most convincing thing I had ever 
thought of; no mathematical proof would have  equaled it. But Mr. T here insisted on 
rigor. I finally gave in to his insistence, a nd checked all of Najunamar's results. To my 
great surprise, each one of them was right. How he discovered them, I'll never know, however. He must have some amazing and inscru table Oriental type of insight which we 
here in the Occident can have no inkling of. At present, that's the only theory which makes an sense to me. 
Tortoise: Mr. Crab has always been a little mo re susceptible to mystical or fanciful

Tortoise: Mr. Crab has always been a little mo re susceptible to mystical or fanciful 
explanations than I am. I have full confidence that whatever Najunamar did in his way has a complete parallel inside orthodox mathematics. There is no way of doing 
mathematics which is fundamentally different from what we now know, in my opinion. 
Achilles: That is an interesting opinion. I s uppose it has something to do with the Church-
Turing Thesis and related topics. 
Crab: Oh, well, let us leave these technical ma tters aside on such a fine day, and enjoy the

Crab: Oh, well, let us leave these technical ma tters aside on such a fine day, and enjoy the 
quiet of the forest, the chirping of the birds,  and the play of sunlight on the new leaves 
and buds. Ho! 
Tortoise: I second the motion. After all, all gene rations of Tortoises have reveled in such 
delights of nature. 
Crab: As have all generations of Crabs. Achilles: You don't happen to have brought  your flute along, by any chance, Mr. C? 
Crab: Why, certainly! I take it with me everywhere. Would you like to hear a tune or two? Achilles: It would be delightful, in this pastoral setting. Do you play from memory?

Crab: Why, certainly! I take it with me everywhere. Would you like to hear a tune or two? Achilles: It would be delightful, in this pastoral setting. Do you play from memory? 
Crab: Sad to say, that is beyond my capability. I have to read my music 
from a sheet. But that is no problem. I have several very pleasant pieces here in this case.  
(He opens up a thin case and draws out a fe w pieces of paper. The topmost one has the 
following symbols on it: 
 
Va:-Sa=O 
 
He sticks the top sheet into a little holder atta ched to his flute, and plays. The tune is very 
short.) 
 
Achilles: That was charming. (Peers over at the sheet on the flute, and a quizzical expression 
beclouds his face.) What is that statement of  number theory doing, attached to your flute

Va:-Sa=O 
 
short.) 
 
beclouds his face.) What is that statement of  number theory doing, attached to your flute 
like that? 
 
(The Crab looks at his flute, then his musi c, turns his head all around, and appears slightly 
confused.)

 548Crab: I don't understand. What statement of number theory? 
Achilles: "Zero is not the successor of any na tural number." Right there, in the holder on 
your flute! 
Crab: That's the third Piano Postulate. There ar e five of them, and I've arranged them all for 
flute. They're obvious, but catchy. 
Achilles: What's not obvious to me is how a number-theoretical statement can be played as 
music. 
Crab: But I insist, it's 'NOT a number-theoreti cal statement-it's a Piano Postulate! Would 
you like to hear another? 
Achilles: I'd be enchanted.  
(The Crab places another piece of paper on his flute, and this time Achilles watches more 
carefully.) 
 
Well, I watched your eyes, and they were looking at that FORMULA on the sheet. Are 
you sure that that is musical notation? I sw ear, it most amazingly resembles the notation 
which one might use in a formalized version of number theory. 
 Crab: How odd! But certainly that is music, not any kind of statement of mathematics, as

 Crab: How odd! But certainly that is music, not any kind of statement of mathematics, as 
far as I can tell! Of course, I am not a ma thematician in any sense of the word. Would 
you like to hear any other tunes? 
Achilles: By all means. Have you some others? Crab: Scads.  
(He takes a new sheet, and attaches it to his flute. It contains the following symbols: 
 
3a:3b:(SSa. SSb)=SSSSSSSSSSSSSO 
 
Achilles peers at it, while the Crab plays it.) 
 
Isn't it lovely? 
Achilles: Yes, it certainly is a tuneful little piece. But I have to say, it's looking more and 
more like number theory to me.

more like number theory to me. 
Crab: Heavens! It is just my usual musi c notation, nothing more. I simply don't know how 
you read all these extramusical connotations into a straightforward representation for sounds. 
Achilles: Would you be averse to playing a piece of my own composition? Crab: Not in the least. Have you got it with you? Achilles: Not yet, but I have a hunch I might be able to compose some tunes all by myself. Tortoise: I must. tell you, Achilles, that Mr. C is  a harsh judge of music composed by others,

Achilles: Would you be averse to playing a piece of my own composition? Crab: Not in the least. Have you got it with you? Achilles: Not yet, but I have a hunch I might be able to compose some tunes all by myself. Tortoise: I must. tell you, Achilles, that Mr. C is  a harsh judge of music composed by others, 
so do not be disappointed if, by some chance, he is not an enthusiast for your efforts. 
Achilles: That is very kind of you to forewarn me. Still, I'm willing to give it a try .  
(He writes: 
 
((SSSO . SSSO) +(SSSSO. SSSSO))=(SSSSSO. SSSSSO)

 549The Crab takes it, looks it over for a moment, then sets it in his music holder, and pipes.) 
 
Crab: Why, that's quite nice, Achilles. I enjoy strange rhythms.  Achilles: What's strange about the rhythms in that piece? Crab: Oh, naturally, to you as the composer it must seem quite bland, but to my ears, 
shifting from a 3/3 rhythm to 4/4 and then to  5/5 is quite exotic. If you have any other 
songs, I'd be glad to play them. Achilles: Thank you very much. I've never composed anything before, and I must say composing is quite different from how I had imagined it 
to be. Let me try my hand at another one. ( jots down a line .)  
 
3a:3b:(SSa - SSb) =SSSSSSSSSSSSSSO 
 
Crab: Hmmm ... Isn't th at just a copy of my earlier piece? 
Achilles: Oh, no! I've added one more S. Where you had thirteen in a row, I have fourteen. Crab: Oh, yes. Of course. ( He plays it, and looks very stern .)

Achilles: Oh, no! I've added one more S. Where you had thirteen in a row, I have fourteen. Crab: Oh, yes. Of course. ( He plays it, and looks very stern .)  
Achilles: I do hope you didn't dislike my piece! Crab: I am afraid, Achilles, that you completely  failed to grasp the subtleties of my piece, 
upon which yours is modeled. But how coul d I expect you to understand it on first 
hearing? One does not always understand what is at the root of beauty. It is so easy to 
mistake the superficial aspects of a piece for its beauty, and to imitate them, when the beauty itself is locked deep inside the mu sic, in a way which seems always to elude 
analysis.

analysis. 
Achilles: I am afraid that you have lost me  a little in your erudite commentary. I understand 
that my piece does not measure up to your high standards, but I do not know exactly 
where I went astray. Could you perhaps tell me  some specific way in which you find fault 
with my composition? 
Crab: One possible way to save your composition, Achilles, would be to insert another three 
S's-five would do as well-into that long gr oup of S's near the end. That would create a 
subtle and unusual effect.

subtle and unusual effect. 
Achilles: I see. Crab: But there are other ways you might choos e to change your piece. Personally, I would 
find it most appealing to put another tilde in the front. Then there would be a nice balance 
between the beginning and the end. Having two tild es in a row never fails to give a gay 
little twist to .a piece, you know. 
Achilles: How about if I take both of your suggestions, and make the following piece?  
-3a:3b:(SSa.SSb)=SSSSSSSSSSSSSSSSSO

-3a:3b:(SSa.SSb)=SSSSSSSSSSSSSSSSSO 
 
Crab (a painful grimace crossing his face): No w, Achilles, it is important to learn the 
following lesson: never try to put too much in to any single piece. There is always a point 
beyond which it cannot be improved,

 550and further attempts to improve it will in fact destroy it. Such is the case in this example. 
Your idea of incorporating both of my sugge stions together does not yield the desired 
extra amount of beauty, but on the contrary cr eates an imbalance which quite takes away 
all the charm. 
Achilles: How is it that two very similar pieces , such as yours with thirteen 5's, and mine 
with fourteen S's, seem to you to be so differe nt in their musical worth? Other than in that 
minor respect, the two are identical. 
Crab: Gracious! There is a world of difference between your piece and mine. Perhaps this is 
a place where words fail to convey what the spir it can feel. Indeed, I would venture to say 
that there exists no set of rules which delineat e what it is that makes a piece beautiful, nor 
could there ever exist such a set of rules. Th e sense of Beauty is the exclusive domain of 
Conscious Minds, minds which through the experi ence of living have gained a depth that

Conscious Minds, minds which through the experi ence of living have gained a depth that 
transcends explanation by any mere set of rules. 
Achilles: I will always remember this vivid cl arification of the nature of Beauty. I suppose 
that something similar applies to the concept of Truth, as well? 
Crab: Without doubt. Truth and Beauty are as interrelated as-as Achilles: As interrelated as, say, mathematics and music? Crab: Oh! You took the words right out of my mouth! How did you know that that is what I 
was thinking?

was thinking? 
Tortoise: Achilles is very clever, Mr. C. Never underestimate the potency of his insight. Achilles: Would you say that there could concei vably be any relationship between the truth 
or falsity of a particular statement of mathem atics, and the beauty, or lack of beauty, of 
an associated piece of music? Or  is that just a far-fetched fancy of mine, with no basis in 
reality? 
Crab: If you are asking me, that is carrying things much too far. When I spoke of the 
interrelatedness of music and mathematics,  I was speaking very figuratively, you know.

reality? 
interrelatedness of music and mathematics,  I was speaking very figuratively, you know. 
As for a direct connection between specific pieces of music and specific statements of 
mathematics, however, I harbor extremel y grave doubts about its possibility. I would 
humbly counsel you not to give too' much time to such idle speculations. 
Achilles: You are no doubt right. It would be most unprofitable. Perhaps I ought to 
concentrate on sharpening my musical sensitivity by composing some new pieces. Would you be willing to serve as my mentor, 
Mr. C? Crab: I would be very happy to aid you in your steps towards musical understanding.  
(So Achilles takes pen in hand, and, with w hat appears to be a great deal of

(So Achilles takes pen in hand, and, with w hat appears to be a great deal of 
concentration, writes : 
 
AOOaV'\/--nn:b+cS(33=OAD((-d)<v(VS-+(>v

 551The Crab looks very startled.) 
 
Y ou really want me to play that-that-that whatever-it-is?  Achilles: Oh, please do!  
(So the Crab plays it, with evident difficulty.) 
 
Tortoise: Bravo! Bravo! Is John Cage your fa vorite composer, Achilles? Achilles: Actually, 
he's my favorite anti-composer. Anyway, I'm glad you liked MY music. 
Crab: The two of you may find it amusing to listen to such totally meaningless cacophony, 
but I assure you it is not at all pleasant for a sensitive composer to be subjected to such 
excruciating, empty dissonances and meaningl ess rhythms. Achilles, I thought you had a 
good feeling for music. Could it be that your previous pieces had merit merely by 
coincidence? 
Achilles: Oh, please forgive me, Mr. Crab. I wa s trying to explore the limits of your musical 
notation. I wanted to learn directly what ki nds of sound result when I write certain types 
of note sequences, and also how you evaluate pieces written in various styles.

of note sequences, and also how you evaluate pieces written in various styles. 
Crab: Harrumph! I am not just an automatic  music-machine, you know. Nor am I a garbage 
disposal for musical trash. 
Achilles: I am very sorry. But I feel that I have learned a great deal by writing that small 
piece, and I am convinced that I can now write  much better music than I ever could have 
if I hadn't tried that idea. And if you'll just play one more piece of mine, I have high 
hopes that you will feel better about my musical sensitivities. 
Crab: Well, all right. Write it down and I'll give it a chance.  
(Achilles writes:

(Achilles writes:  
 
Ya:Vb:<(a -a) =(SSO -(b  > b))Da=0>  
 
and the Crab plays.) 
 
You were right, Achilles. You seem to have completely regained your musical acuity. This is a little gem! How did you come to compose it? I have never heard anything like it. It obeys all the rules of harmony, and yet has a certain-what shall I say?-irrational appeal 
to it. I can't put my finger on it, but I like it for that very reason. 
Achilles: I kind of thought you might like it. Tortoise: Have you got a name for it, Ach illes? Perhaps you might call it "The Song of

Achilles: I kind of thought you might like it. Tortoise: Have you got a name for it, Ach illes? Perhaps you might call it "The Song of 
Pythagoras". You remember that Pythagoras and his followers were among the first to study musical sound. 
Achilles: Yes, that's true. That would be a fine title. Crab: Wasn't Pythagoras also the first to disc over that the ratio of tw o squares can never be 
equal to 2? Tortoise: I believe you're right. It was considered a truly sinister discovery at 
the time, for never before had anyone realized that there are

 552numbers-such as the square root of 2-whic h are not ratios of integers. And thus the 
discovery was deeply disturbing to the Py thagoreans, who felt that it revealed an 
unsuspected and grotesque defect in the abstract world of numbers. But I don't know 
what this has to do with the price of tea in China. 
Achilles: Speaking of tea, isn't that the teahouse just up there ahead of us? Tortoise: Yes, that's it, all right. We ought to be there in a couple of minutes. Achilles: Hmm ... That's just enough time for me to whistle for you the tune which the taxi 
driver this morning had on his radio. It went like this. 
Crab: Hold on for a moment; I'll get some paper from my case, and jot 
down your tune. ( Scrounges around inside his case, and finds a blank sheet .) 
Go ahead; I'm ready. 
 
(Achilles whistles a rather long tune, and the Crab scrambles to keep up with him.) 
 
Could you whistle the last few bars again?  
Achilles: Why, certainly.

Achilles: Why, certainly.  
(After a couple of such repeats, the sessi on is complete, and the Crab proudly displays 
his transcription : 
 
<((SSSSSO.SSSSSO)+(SSSSSO.SSSSSO))=((SSSSSSSO.SSSSSSSO)+(SO.SO))n---
3b:<3c:(Sc+b)=((SSSSSSSO . SSSSSSSO)+ (SO • SO))n3d:3d':3e:3e': 
<---<d=evd=e' >n<b=((Sd • Sd)+(Sd' • Sd'))nb=((Se • Se)+(Se' • Se'))>> >> 
 
The Crab then plays it himself.) 
 Tortoise: It's peculiar music, isn't it? It sounds a wee bit like music from India, to me. 
Crab. Oh, I think it's too simple to be from India. But of course I know precious little about 
such things.

such things. 
Tortoise: Well, here we are at the teahouse. Shall we sit outside here, on the verandah? Crab: If you don't mind, I'd prefer to go inside. I've gotten perhaps enough sun for the day.  
(They go inside the teahouse and are seated at a nice wooden table, and order cakes and 
tea. Soon a cart of scrumptious-looking pastrie s is wheeled up, and each of them chooses 
his favorite.) 
 
Achilles: You know, Mr. C, I would love to know what you think of another piece which I

Achilles: You know, Mr. C, I would love to know what you think of another piece which I 
have just composed in my head. Crab: Can you show it to me? Here, write it down on this 
napkin. 
 (Achilles writes : 
 
da:3b:3c:<- 3d:3e:<(SSd•SSe)=bv(SSd•SSe)=c>n(a+a)=(b+c)> 
 
The Crab and Tortoise study it with interest .)

 553Tortoise: Is it another beautiful piece, Mr. C, in your opinion? 
Crab: Well, uh ... (Shifts in his chair, and looks somewhat uncomfortable.) Achilles: What's the matter? Is it harder to deci de whether this piece is beautiful than it is for 
other pieces? 
Crab: Ahm ... No, it's not that-not at all. It's just that, well ... I really have to HEAR a piece 
before I can tell how much I like it. 
Achilles: So go ahead and play it! I'm dying to know whether you find it beautiful or not. Crab: Of course, I'd be extremely glad to play it for you. The only thing is Achilles: Can't you play it for me? What's the matter? Why are you balking? Tortoise: Don't you realize, Achilles, that for Mr. Crab to fulfill your request would be most 
impolite and disturbing to the clientele and employees of this fine establishment? 
Crab (suddenly looking relieved): That's right . We have no right to impose our music on 
others.

others. 
Achilles (dejectedly): Oh, PHOOEY! And I so much  wanted to know what he thinks of this 
piece! 
Crab: Whew! That was a close call! Achilles: What was that remark? Crab: Oh-nothing. It's just that that waiter ove r there, he got knocked into by another waiter, 
and almost dropped a whole pot of tea into  a lady's lap. A narrow escape, I must say. 
What do you say, Mr. Tortoise? Tortoise: Ve ry good teas, I'd say. Wouldn't you agree, 
Achilles? Achilles: Oh, yes. Prime teas, in fact. 
Crab: Definitely. Well, I don't know about you tw o, but I should perhaps be going, for I've a 
long steep trail back to my house, on the other side of this hill. 
Achilles: You mean this is a big bluff? Crab: You said it, Achilles. Achilles: I see. Well, I'll have to remember that.

Achilles: You mean this is a big bluff? Crab: You said it, Achilles. Achilles: I see. Well, I'll have to remember that. 
Crab: It has been such a jolly afternoon, Achilles, and I sincerely hope we will exchange 
more musical compositions another day. 
Achilles: I'm looking forward to that very much, Mr. C. Well, good-bye. Tortoise: Good-
bye, Mr. C. 
(And the Crab heads off down his side of the hill.) Achilles: Now there goes a brilliant fellow ... In my estimation, he's at least four times as 
smart as any crab alive. Or he might even be five

bye, Mr. C. 
smart as any crab alive. Or he might even be five 
Tortoise: As you said in the beginning, and pr obably shall be saying forevermore, words 
without end.

Church, Turing, Tarski, and Others 
 554CHAPTER XVII 
 
Formal and Informal Systems 
 
WE HAVE COME to the point where we can de velop one of the main theses of this 
book: that every aspect of thinking can be view ed as a high-level description of a system 
which, on a low level, is governed by simple, even formal, rules. The "system", of course, 
is a brain-unless one is' speaking of thought processes flowing in another medium, such 
as a computer's circuits. The image is that  of a formal system underlying an "informal 
system"-a system which can, for instance, make puns, discover number patterns, forget names, make awful blunders in chess, and so  forth. This is what one sees from the 
outside: its informal, overt, software level. By contrast, it has a formal, hidden, hardware 
level (or "substrate") which is a formidably  complex mechanism that makes transitions 
from state to state according to definite ru les physically embodied in it, and according to

from state to state according to definite ru les physically embodied in it, and according to 
the input of signals which impinge on it. 
A vision of the brain such as this has many philosophical and other consequences, 
needless to say. I shall try to spell some of them out in this Chapter. Among other things, 
this vision seems to imply that, at bottom, the brain is some sort of a "mathematical" 
object. Actually, that is at best a very aw kward way to look at the brain. The reason is 
that, even if a brain is, in a technical and abstract sense, some sort of formal system, it

that, even if a brain is, in a technical and abstract sense, some sort of formal system, it 
remains true that mathematicians only work w ith simple and elegant systems, systems in 
which everything is extremely clearly defined-a nd the brain is a far cry from that, with its 
ten billion or more semi-independent ne urons, quasi-randomly connected up to each 
other. So mathematicians would never study a real brain's networks. And if you define 
"mathematics" as what mathematicians enjoy doing, then the properties of brains are not 
mathematical.

mathematical. 
The only way to understand such a comple x system as a brain is by chunking it on 
higher and higher levels, and thereby losing some  precision at each step. What emerges at 
the top level is the "informal system" which obe ys so many rules of such complexity that 
we do not yet have the vocabulary to thi nk about it. And that is what Artificial 
Intelligence research is hoping to find. It has quite a different 'flavor from mathematics

Intelligence research is hoping to find. It has quite a different 'flavor from mathematics 
research. Nevertheless, there is a loose connection to mathematics: Al people often come from a strong mathematics background, and

 555mathematicians sometimes are intrigued by the workings of their own brains. The 
following passage, quoted from Stan islaw Ulam's autobiographical Adventures of a 
Mathematician , illustrates this point: 
 
It seems to me that more could be done to elicit ... the nature of associations, with computers providing the means for experimentation. Such a study would have to involve a gradation of notions, of symbols,  of classes of symbols, of classes of 
classes, and so on, in the same way that  the complexity of mathematical or 
physical structures is investigated.  
There must be a trick to the train of  thought, a recursive formula. A group of 
neurons starts working automatically, some times without external impulse. It is a 
kind of iterative process with a growing pa ttern. It wanders about in the brain, and 
the way it happens must depend on the memory of similar patterns.' 
 
Intuition and the Magnificent Crab

Intuition and the Magnificent Crab 
 Artificial Intelligence is often referred to as "Al". Often, when I try to explain what is 
meant by the term, I say that the letters "AI"  could just as well stand for "Artificial 
Intuition", or even "Artificial Imagery". The aim of Al is to get at what is happening 
when one's mind silently and invisibly choos es, from a myriad alternatives, which one 
makes most sense in a very complex situation. In many real-life situations, deductive reasoning is inappropriate, not because it w ould give wrong answers, but because there

makes most sense in a very complex situation. In many real-life situations, deductive reasoning is inappropriate, not because it w ould give wrong answers, but because there 
are too many correct but irrelevant statements which can be made; there are just too many things to take into account simultaneously fo r reasoning alone to be sufficient. Consider 
this mini-dialogue:  
"The other day I read in the paper that the-- "Oh-you were reading? It follows that you have eyes. Or at least one eye. Or 
rather, that you had at least one eye then." 
 A sense of judgment-"What is important here , and what is not?"-is called for. Tied up 
with this is a sense of simplicity, a sense of  beauty. Where do these intuitions come from?

with this is a sense of simplicity, a sense of  beauty. Where do these intuitions come from? 
How can they emerge from an underlying formal system? 
In the Magnificrab , some unusual powers of the Crab's mind are revealed. His 
own version of his powers is merely that he listens to music and distinguishes the 
beautiful  from the non-beautiful . (Apparently for him there is  a sharp dividing line.) Now 
Achilles finds another way to describe the Crab 's abilities: the Crab divides statements of 
number theory into the categories true and false. But the Crab maintains that, if he chances to do so, it is only by the purest accident, for he is, by his own admission,

number theory into the categories true and false. But the Crab maintains that, if he chances to do so, it is only by the purest accident, for he is, by his own admission, 
incompetent in mathematics. What makes the Crab's performance all the more mystifying to Achilles, however, is that it seems to be in  direct violation of a celebrated result of 
metamathematics with which Achilles is familiar:  
CHURCH's THEOREM: There is no infallible method for telling theorems of TNT from 
nontheorems.

 556It was proven in 1936 by the Amer ican logician Alonzo Church. Closely related is what I 
call the   TARSKI-CHURCH-TURING THEOREM: There is  no infallible method for telling true 
from false statements of number theory.  
 To understand Church's Theorem and the Tarski-Church-Turing Theorem better, we should first describe one of the ideas on whic h they are based; and that is the Church-
Turing Thesis (often called "Church's Thesis") . For the Church-Turing Thesis is certainly 
one of the most important concepts in the philosophy of mathematics, brains, and 
thinking. 
Actually, like tea, the Church-Turing Thesis  can be given in a variety of different 
strengths. So I will present it in various versions, and we will consider what they imply. The first version sounds very innocent-in fact almost pointless:  
CHURCH-TURING THESIS, TAUTOL OGICAL VERSION: Mathematics 
problems can be solved only by doing mathematics.

thinking. 
problems can be solved only by doing mathematics. 
 Of course, its meaning resides in the meani ng of its constituent terms. By "mathematics 
problem" I mean the problem of deciding whether some number possesses or does not 
possess a given arithmetical property. It turn s out that by means of Godel-numbering and 
related coding tricks, almost any problem in any branch of mathematics can be put into this form, so that "mathematics problem" reta ins its ordinary mean ing. What about "doing 
mathematics"? When one tries to ascertain whether a number has a property, there seem

mathematics"? When one tries to ascertain whether a number has a property, there seem 
to be only a small number of operations wh ich one uses in combination over and over 
again-addition, multiplication, checking for equality or inequality. That is, loops 
composed of such operations seem to be the only tool we have that allows us to probe the world of numbers. Note the word "seem". Th is is the critical word which the Church-
Turing Thesis is about. We can give a revision:  
CHURCH-TURING THESIS, STANDARD VERSION: Suppose there is a 
method which a sentient being follows in or der to sort numbers into two classes.

method which a sentient being follows in or der to sort numbers into two classes. 
Suppose further that this method always yi elds an answer within a finite amount 
of time, and that it always gives the same answer for a given number. Then: 
Some terminating FlooP program (i.e., some general recursive function) exists which gives exactly the same answers as the sentient being's method does. 
 The central hypothesis, to make it very clear, is that any mental process which divides 
numbers into two sorts can be described in  the form of a FlooP program. The intuitive

numbers into two sorts can be described in  the form of a FlooP program. The intuitive 
belief is that there are no other tools than thos e in FlooP, and that there are no ways to use 
those tools other than by

 557unlimited iterations (which FlooP allows). Th e Church-Turing Thesis is not a provable 
fact in the sense of a Theo rem of mathematics-it is a hypothesis about the processes 
which human brains use.  
The Public-Processes Version 
 Now some people might feel that this vers ion asserts too much. These people might put 
their objections as follows: "Someone such as the Crab might exist-someone with an 
almost mystical insight into mathematics, but who is just as much in the dark about his 
own peculiar abilities as anyone else-and perhap s that person's mental mechanisms carry 
out operations which have no counterpart in FlooP." The idea is that perhaps we have a subconscious potential for doing things which transcend the conscious processes-things 
which are somehow inexpressible in terms of  the elementary FlooP operations. For these 
objectors, we shall give a weaker version of  the Thesis, one which distinguishes between 
public and private mental processes:

public and private mental processes:  
CHURCH-TURING THESIS, PUBLIC-PROCESSES VERSION: Suppose there 
is a method which a sentient being follows  in order to sort numbers into two 
classes. Suppose further that this method al ways yields an answer within a finite 
amount of time, and that it always give s the same answer for a given number. 
Proviso. Suppose also that this method can be communicated reliably from one 
sentient being to another by means of language. Then: Some terminating FlooP

sentient being to another by means of language. Then: Some terminating FlooP 
program (i.e., general recursive function)  exists which gives exactly the same 
answers as the sentient beings' method does. 
 This says that public methods are subject to  "FlooPification", but asserts nothing about 
private methods. It does not say that they are un-FlooP-able, but it at least leaves the door 
open.  
Srinivasa Ramanujan 
 As evidence against any stronger version of th e Church-Turing Thesis, let us consider the 
case of the famous Indian mathematician of the first quarter of the twentieth century,

open.  
case of the famous Indian mathematician of the first quarter of the twentieth century, 
Srinivasa Ramanujan (1887-1920). Ramanujan  (Fig. 105) came from Tamilnadu, the 
southernmost part of India, and studied  mathematics a little in high school. One day, 
someone who recognized Ramanujan's talent for math presented him with a copy of a 
slightly out-of-date textbook on analysis, which Ramanujan devoured (figuratively 
speaking). He then began making his own fora ys into the world of analysis, and by the 
time he was twenty-three, he had made a num ber of discoveries which he considered

time he was twenty-three, he had made a num ber of discoveries which he considered 
worthwhile. He did not know to whom to tu rn, but somehow was told about a professor 
of mathematics in faraway England, named G. H. Hardy. Ramanujan compiled his best

 558
 
          FIGURE 105. Srinivasa Ramanujan and one of his strange Indian melodies.  
 
results together in a packet of papers, and se nt them all to the, unforewarned Hardy with a 
covering letter which friends helped him express in English. Below are some excerpts taken fr om Hardy's description of his reaction upon 
receiving the bundle:  
... It soon became obvious that Ramanujan  must possess much more general 
theorems and was keeping a great deal up his sleeve.... [Some formulae]defeated 
me completely; I had never seen anything in the least like them before. A single 
look at them is enough to show that  they could only be written down by a 
mathematician of the highest class. They must be true because, if they were not 
true, no one would have had the imagination to invent them. Finally ... the writer must be completely honest, because great mathematicians are commoner than thieves or humbugs of such incredible skill
2

2 
 What resulted from this correspondence was that Ramanujan came to England in 1913, sponsored by Hardy; and then followed an intense collaboration which terminated in 
Ramanujan's early demise at age thin thirty-three from tuberculosis. 
Ramanujan had several extraordinary characteristics which set him apart from the 
majority of mathematicians. One was his lack of rigor. Very often he would simply state 
a result which, he would insist, had just come to

 560him from a vague intuitive source, far out of the realm of conscious probing. In fact, he 
often said that the goddess Namagiri inspired  him in his dreams. This happened time and 
again, and what made it all the more mystif ying-perhaps even imbuing it with a certain 
mystical quality-was the fact that many of  his "intuition-theorems" were wrong. Now 
there is a curious paradoxical effect where sometimes an event which you think could not help but make credulous people become a little  more skeptical, actually has the reverse 
effect, hitting the credulous ones in some vulne rable spot of their minds, tantalizing them 
with the hint of some baffling irrational side  of human nature. Such was the case with 
Ramanujan's blunders: many educated people with  a yearning to believe in something of 
the sort considered Ramanujan's intuitive powers to be evidence of a mystical insight into

the sort considered Ramanujan's intuitive powers to be evidence of a mystical insight into 
Truth, and the fact of his fallibility seeme d, if anything, to strengthen, rather than 
weaken, such beliefs. 
Of course it didn't hurt that he was from one  of the most backward parts of India, 
where fakirism and other eerie Indian rites had been practiced for millennia, and were 
still practiced with a frequency probably exceeding that of the teaching of higher 
mathematics. And his occasional wrong flashes of insight, instead of suggesting to people 
that he was merely human, paradoxically in spired the idea that Ramanujan's wrongness 
always had some sort of "deeper rightness" to  it-an "Oriental" rightness, perhaps touching

always had some sort of "deeper rightness" to  it-an "Oriental" rightness, perhaps touching 
upon truths inaccessible to Western minds. What  a delicious, almost irresistible thought! 
Even Hardy-who would have been the first to deny that Ramanujan had any mystical 
powers-once wrote about one of Ramanujan's fa ilures, "And yet I am not sure that, in 
some ways, his failure was not more wonderful than any of his triumphs." 
The other outstanding feature of Ramanujan's mathematical personality was his 
"friendship with the integers", as his collea gue Littlewood put it. This is a characteristic

"friendship with the integers", as his collea gue Littlewood put it. This is a characteristic 
that a fair number of mathematicians sh are to some degree or other, but which 
Ramanujan possessed to an extreme. There are a couple of anecdotes which illustrate this 
special power. The first one is related by Hardy: 
 
I remember once going to see him when he was lying ill at Putney. I had ridden in 
taxi-cab No. 1729, and remarked that the number seemed to me rather a dull one, and that I hoped it was not an unfavorable omen. "No," he replied, "it is a very

taxi-cab No. 1729, and remarked that the number seemed to me rather a dull one, and that I hoped it was not an unfavorable omen. "No," he replied, "it is a very 
interesting number; it is the smallest numbe r expressible as a sum of two cubes in 
two different ways." .1 asked him, natura lly, whether he knew the answer to the 
corresponding problem for fourth powers; and he replied, after a moment's thought, that he could see no obvious exam ple, and thought that the first such 
number must be very large.' 
 It turns out that the answer for fourth powers is:  
635318657 = 134 + 1334 = 1584 + 594 
 The reader may find it interesting to tackle the analogous problem for squares, which is 
much easier.

much easier. 
It is actually quite interesting to ponder why it is that Hardy im-

 561mediately jumped to fourth powers. After a ll, there are several other reasonably natural 
generalizations of the equation 
 
u3 + v3 = x3 + y3 
 
along different dimensions. For instance, th ere is the question about representing a 
number in three distinct ways as a sum of two cubes:  
r
3 + 3 = u3 + v3 = x3 + y3 
 Or, one can use three different cubes: 
 
u3 + v3 + w3 = x3 + y3 + z3, 
 
Or one can even make a Grand Generalization in all dimensions at once: 
 
r4 + s4 +  t4 = u4 + v4 + w4 = x4 + v4 + z4 
 
There is a sense, however, in which Hardy's generalization is "the most mathematician-
like". Could this sense of mathematical esthetics ever be programmed? 
The other anecdote is taken from a biography of Ramanujan by his 
countryman S. R. Ranganathan, where it is calle d "Ramanujan's Flash". It is related by a 
Indian friend of Ramanujan's from his Cambridge days, Dr. P. C. Mahalanobis.

Indian friend of Ramanujan's from his Cambridge days, Dr. P. C. Mahalanobis. 
 
On another occasion, I went to his room to  have lunch with him. The First World 
War had started some time earlier. I had in  my hand a copy of the monthly "Strand 
Magazine" which at that time used to pub lish a number of puzzles to be solved by 
readers. Ramanujan was stirring something in a pan over the fire for our lunch. I 
was sitting near the table, turning over the pa ges of the Magazine. I got interested in

was sitting near the table, turning over the pa ges of the Magazine. I got interested in 
a problem involving a relation between two numbers. I have forgotten the details; but I remember the type of the problem. Two British officers had been billeted in 
Paris in two different houses in a long street; the door numbers of these houses were related in a special way; the problem  was to find out the two numbers. It was 
not at all difficult. I got the solution in a few minutes by trial and error.

not at all difficult. I got the solution in a few minutes by trial and error. 
 
MAHALANOBIS (in a joking way): Now here is a problem for you. RAMANUJAN: What problem, tell me.'(He went  on stirring the pan.) I read out the 
question from the "Strand Magazine". RAMANUJAN: Please take down the solution. (He dictated a continued fraction.) 
The first term was the solution which I had obtained. Each successive term 
represented successive solutions for the sa me type of relation between two numbers,

represented successive solutions for the sa me type of relation between two numbers, 
as the number of houses in the street would increase indefinitely. I was amazed. MAHALANOBIS: Did you get the solution in a flash? RAMANUJAN: Immediately I heard the probl em, it was clear that the Solution was 
obviously a continued fraction; I then t hought, "Which continued fraction?" and the 
answer came to my mind. It was just as simple as this.'  
 Hardy, as Ramanujan's closest co-worker, was often asked after

 562Ramanujan's death if there had been any occult or otherwise exotically flavored elements 
to Ramanujan's thinking style. Here is one comment which he gave:  
I have often been asked whether Ramanuj an had any special secret; whether his 
methods differed in kind from those of other mathematicians; whether there was 
anything really abnormal in his mode of  thought. I cannot answer these questions 
with any confidence or conviction; but I do not believe it. My belief is that all 
mathematicians think, at bottom, in the same kind of way, and that Ramanujan was 
no exception .
5 
 Here Hardy states in essence his own version of the Church-Turing Thesis. I paraphrase: CHURCH-TURING THESIS, HARDY'S VERSION: At bottom, all mathematicians are 
isomorphic. 
 
This does not equate the mathematical potential of mathematicians with that of general recursive functions; for that, however, all you n eed is to show that some mathematician's

5 
 
This does not equate the mathematical potential of mathematicians with that of general recursive functions; for that, however, all you n eed is to show that some mathematician's 
mental capacity is no more general than recu rsive functions. Then, if you believe Hardy's 
Version, you know it for all mathematicians. Then Hardy compares Ramanujan with calculating prodigies:  
His memory, and his powers of calculation, were very unusual, but they could not 
reasonably be called "abnormal". If he had to multiply two large numbers, he 
multiplied them in the ordinary way; he  could do it with unusual rapidity and 
accuracy, but not more rapidly and accurately than any mathematician who is naturally quick and has the habit of computations

accuracy, but not more rapidly and accurately than any mathematician who is naturally quick and has the habit of computations 
 
Hardy describes what he perceived as Rama nujan's outstanding intellectual attributes: 
 
With his memory, his patience, and hi s power of calculation, he combined a power 
of generalisation, a feeling for form, and a capacity for rapid modification of his 
hypotheses , that were often really startling, and made him, in his own field, without a 
rival in his day.' 
 
The part of this passage which I have ita licized seems to me to be an excellent 
characterization of some of the subtlest feat ures of intelligence in general. Finally, Hardy

characterization of some of the subtlest feat ures of intelligence in general. Finally, Hardy 
concludes somewhat nostalgically: 
 
[His work has not the simplicity and inevitableness of the very greatest work; it would be greater if it were less strange . One gift it has which no one can deny-
profound and invincible originality. He would probably have been a greater 
mathematician if he had been caught and tamed a little in his youth; he would have discovered more that was new, and that, no doubt, of greater importance. On the other hand he would have been less of a Ramanujan, and more of a European

mathematician if he had been caught and tamed a little in his youth; he would have discovered more that was new, and that, no doubt, of greater importance. On the other hand he would have been less of a Ramanujan, and more of a European 
professor and the loss might have been greater than the gain.
8 
 The esteem in which Hardy held Ramanujan is  revealed by the romantic way in which he 
speaks of him.

 563 
"Idiots 'Savants" 
 
There is another class of people whose mathematical abilities seem to defy rational explanation-the so-called "idiots savants" , who can perform complex calculations at 
lightning speeds in their heads (or wherever they do it). Johann Martin Zacharias Dase, 
who lived from 1824 to 1861 and was employe d by various European governments to 
perform computations, is an outstanding example. He not only could multiply two 
numbers each of 100 digits in his head; he also had an uncanny sense of quantity. That is, he could just "tell", without counting, how many sheep were in a field, or words in a 
sentence, and so forth, up to about 30-this in contrast to most of us, who have such a 
sense up to about 6, with reliability. Incidentally, Dase was not an idiot. 
I shall not describe the many fascinating documented cases of "lightning 
calculators", for that is not my purpose here. Bu t I do feel it is important to dispel the idea

calculators", for that is not my purpose here. Bu t I do feel it is important to dispel the idea 
that they do it by some mysterious, unana lyzable method. Although it is often the case 
that such wizards' calculational abilities far exceed their abilities to explain their results, every once in a while, a person with other inte llectual gifts comes along who also has this 
spectacular ability with numbers. From such people's introspection, as well as from 
extensive research by psychologists, it has been ascertained that nothing occult takes 
place during the performances of lightning calcu lators, but simply that their minds race 
through intermediate steps with the kind of se lf-confidence that a natural athlete has in

through intermediate steps with the kind of se lf-confidence that a natural athlete has in 
executing a complicated motion quickly and gr acefully. They do not reach their answers 
by some sort of instantaneous flash of enli ghtenment (though subjectively it may feel that 
way to some of them), but-like the rest of us-by sequential calculation, which is to say, by FlooP-ing (or BlooP-ing) along. - 
Incidentally, one of the most obvious clues that no "hot line to God" is involved is 
the mere fact that when the numbers invol ved get bigger, the answers are slower in 
coming. Presumably, if God or an "oracle" we re supplying the answers, he wouldn't have 
to slow up when the numbers got bigger. On e could probably make a nice plot showing

to slow up when the numbers got bigger. On e could probably make a nice plot showing 
how the time taken by a lightning calculator varies with the sizes of the numbers 
involved, and the operations involved, and from it deduce some features of the algorithms 
employed. 
 
The Isomorphism Version of the Church-Turing Thesis 
 This finally brings us to a strengthened standard version of the Church-Turing Thesis:  
CHURCH-TURING THESIS, ISOMORPHIS M VERSION: Suppose there is a 

employed. 
 
Some terminating FlooP program (i.e.,

 564general recursive function) exists which gives exactly the same answers as the 
sentient being's method does. Moreover:  The mental process and the FlooP 
program are isomorphic in the sense that on some level there is a correspondence between the steps being carried out in both computer and brain. 
 Notice that not only has the conclusion been  strengthened, but also the proviso of 
communicability of the faint-hearted Public-Processes Version has been dropped. This bold version is the one which we now shall discuss. 
In brief, this version asserts that wh en one computes something, one's mental 
activity can be mirrored isomorphically in so me FlooP program. And let it be very clear 
that this does not mean that the brain is actually running a FlooP program, written in the 
FlooP language complete with BEGIN's , END's , ABORT's , and the rest-not at all. It is 
just that the steps are taken in the same or der as they could be in a FlooP program, and

just that the steps are taken in the same or der as they could be in a FlooP program, and 
the logical structure of the calculation can be mirrored in a FlooP program. 
Now in order to make sense of this idea, we shall have to make some level 
distinctions in both computer and brain, for ot herwise it could be misinterpreted as utter 
nonsense. Presumably the steps of the calculation going on inside a person's head are on the highest level, and are supported by lower le vels, and eventually by hardware. So if we

nonsense. Presumably the steps of the calculation going on inside a person's head are on the highest level, and are supported by lower le vels, and eventually by hardware. So if we 
speak of an isomorphism, it means we've tacitly made the assumption that the highest level can be isolated, allowing us to discu ss what goes on there independently of other 
levels, and then to map that top level into FlooP. To be more precise, the assumption is 
that there exist software entities which play the roles of various mathematical constructs, and which are activated in ways which can be mirrored exactly inside FlooP (see Fig. 
106). What enables these software entities to come into existence is the entire

106). What enables these software entities to come into existence is the entire 
infrastructure discussed in Chapters XI and XI I, as well as in the Prelude, Ant Fugue. There is no assertion of isomorphic activity on the lower levels of brain and computer 
(e.g., neurons and bits). 
The spirit of the Isomorphism Version, if not the letter, is gotten across by saying 
that what an idiot savant does in calculati ng, say, the logarithm of 7r, is isomorphic to 
what a pocket calculator does in calculati ng it-where the isomorphism holds on the 
arithmetic-step level, not on the lower levels of , in the one case, neurons, and in the other, 
integrated circuits. (Of course  different routes can be follo wed in calculating anything-but

integrated circuits. (Of course  different routes can be follo wed in calculating anything-but 
presumably the pocket calculator, if not the human, could be instructed to calculate the 
answer in any specific manner.) 
 
FIGURE 106. The behavior of natural numbers can be  mirrored in a human brain or in 
the programs of a computer. These two diffe rent representations can then be mapped 
onto each other on an appropriately abstract level.

 565 
Representation of Knowledge about the Real World 
 
Now this seems quite plausible when the dom ain referred to is number theory, for there 
the total universe in which things happen is very small and clean. Its boundaries and 
residents and rules are well-defined, as in a hard-edged maze. Such a world is far less 
complicated than the open-ended and ill-defined world which we inhabit. A number theory problem, once stated, is complete in and of itself. A real-world problem, on the 
other hand, never is sealed off from any part  of the world with absolute certainty. For 
instance, the task of replacing a burnt-out light bulb may turn out to require moving a 
garbage bag; this may unexpectedly cause the spilling of a box of pills, which then forces 
the floor to be swept so that the pet dog won' t eat any of the spilled pills, etc., etc. The 
pills and the garbage and the dog and the light bulb are all quite distantly related parts of

pills and the garbage and the dog and the light bulb are all quite distantly related parts of 
the world-yet an intimate connection is cr eated by some everyday happenings. And there 
is no telling what else could be brought in  by some other small variations on the 
expected. By contrast, if you are given a number theory problem, you never wind up 
having to consider extraneous things such as  pills or dogs or bags of garbage or brooms 
in order to solve your problem. (Of course, your intuitive knowledge of such objects may 
serve you in good stead as you go about uncons ciously trying to manufacture mental

serve you in good stead as you go about uncons ciously trying to manufacture mental 
images to help you in visualizing the problem  in geometrical terms-but that is another 
matter.) 
Because of the complexity of the worl d, it is hard to imagine a little pocket 
calculator that can answer questions put to it when you press a few buttons bearing labels 
such as "dog", "garbage", "light bulb", and so  forth. In fact, so far it has proven to be 
extremely complicated to have a full-size hi gh-speed computer answer questions about 
what appear to us to be rather simple subdom ains of the real world. It seems that a large

matter.) 
what appear to us to be rather simple subdom ains of the real world. It seems that a large 
amount of knowledge has to be taken into  account in a highly integrated way for 
"understanding" to take place. We can liken real-world thought processes to a tree whose 
visible part stands sturdily above ground but depends vitally on its invisible roots which 
extend way below ground, giving it stability and nourishment. In this case the roots symbolize complex processes which take place below the conscious level of the mind-processes whose effects permeate the way we think but of which we are unaware. These

extend way below ground, giving it stability and nourishment. In this case the roots symbolize complex processes which take place below the conscious level of the mind-processes whose effects permeate the way we think but of which we are unaware. These 
are the "triggering patterns of symbols" which were discussed in Chapters XI and XII. 
Real-world thinking is quite different  from what happens when we do a 
multiplication of two numbers, where everythi ng is "above ground", so to speak, open to 
inspection. In arithmetic, the top level can be "skimmed off " and implemented equally 
well in many different sorts of hardwa re: mechanical adding machines, pocket 
calculators, large computers, people's brains , and so forth. This is what the Church-

Turing Thesis is all about. But when it comes to real-world understanding, it seems that there is no simple way to skim off the top level, and program it. alone. The triggering patterns of symbols are just too complex. Th ere must he several levels through which 
thoughts may "percolate" and "bubble".

 566In particular-and this comes back to a ma jor theme of Chapters XI ' and XII-the 
representation of the real world in the br ain, although rooted in isomorphism to some 
extent, involves some elements which have no counterparts at all in the outer world. That 
is, there is much more to it than simple me ntal structures representing "dog", "broom", 
etc. All of these symbols exist, to be sure -but their internal structures are extremely 
complex and to a large degree are unavailabl e for conscious inspection. Moreover, one 
would hunt in vain to map each aspect of a sy mbol's internal structure onto some specific 
feature of the real world.  
Processes That Are Not So Skimmable 
 For this reason, the brain begins to look like a very peculiar formal system, for on its bottom level-the neural level-where the "rules" operate and change the state, there may be 
no interpretation of the primitive elements (n eural firings, or perhaps even lower-level

no interpretation of the primitive elements (n eural firings, or perhaps even lower-level 
events). Yet on the top level, there emerge s a meaningful interpretation-a mapping from 
the large "clouds" of neural activity which we  have been calling "symbols", onto the real 
world. There is some resemblance to the Gödel construction, in that a high-level 
isomorphism allows a high level of meaning to  be read into strings; but in the Gödel 
construction, the higher-level meaning "rides" on the lower level-that is, it is derived 
from the lower level, once the notion of G ödel-numbering has been introduced. But in the

from the lower level, once the notion of G ödel-numbering has been introduced. But in the 
brain, the events on the neural level are not s ubject to real-world interpretation; they are 
simply not imitating anything. They are there purely as the substrate to support the higher 
level, much as transistors in a pocket cal culator are there purely to support its number-
mirroring activity. And the implication is that th ere is no way to skim off just the highest 
level and make an isomorphic copy in a program ; if one is to mirror the brain processes

level and make an isomorphic copy in a program ; if one is to mirror the brain processes 
which allow real-world understanding, then one must mirror some of the lower-level things which are taking place: the "languages of the brain". This doesn't necessarily mean that one must go all the way down to the leve l of the hardware, though that may turn out 
to be the case. 
In the course of developing a program with the aim of achieving an "intelligent" 
(viz., human-like) internal representation of what is "out there", at some point one will probably be forced into using structures  and processes which do not admit of any

(viz., human-like) internal representation of what is "out there", at some point one will probably be forced into using structures  and processes which do not admit of any 
straightforward interpretations-that is, which cannot be directly mapped onto elements of 
reality. These lower layers of the program w ill be able to be understood only by virtue of 
their catalytic relation to layers above th em, rather than because of some direct 
connection they have to the outer world. (A concrete image of this idea was suggested by 
the Anteater in the Ant Fugue: the "inde scribably boring nightmare" of trying to 
understand a book on the letter level.) 
Personally, I would guess that such multile vel architecture of concept-handling 
systems becomes necessary just when proce sses involving images and analogies become 
significant elements of the program-in

 567contrast to processes which are supposed to carry out strictly deductive reasoning. 
Processes which carry out deductive reasoni ng can be programmed in essentially one 
single level, and are therefore skimmable,  by definition. According to my hypothesis, 
then, imagery and analogical thought processes intrinsically require several layers of 
substrate and are therefore intrinsically non-sk immable. I believe furthermore that it is 
precisely at this same point that creativity  starts to emerge-which would imply that 
creativity intrinsically depends upon certain kinds of "uninterpretable" lower-level 
events. The layers of underpinning of analogi cal thinking are, of course, of extreme 
interest, and. some speculations on their nature will be offered in the next two Chapters.  
Articles of Reductionistic Faith 
 One way to think about the relation between hi gher and lower levels in the brain is this.

 One way to think about the relation between hi gher and lower levels in the brain is this. 
One could assemble a neural net which, on a lo cal (neuron-to-neuron) level, performed in 
a manner indistinguishable from a neural ne t in a brain, but which had no higher-level 
meaning at all. The fact that the lower leve l is composed of interacting neurons does not 
necessarily force any higher level of meani ng to appear-no more than the fact that 
alphabet soup contains letters forces meani ngful sentences to be found, swimming about 
in the bowl. High-level meaning is an opti onal feature of a neural network-one which

in the bowl. High-level meaning is an opti onal feature of a neural network-one which 
may emerge as a consequence of evolutionary environmental pressures. 
Figure 107 is a diagram illustrating the fact that emergence of a higher level of 
meaning is optional. The upwards-pointing a rrow indicates that a substrate can occur 
without a higher level of meaning, but not vi ce versa: the higher level must be derived 
from properties of a lower one. 
 
FIGURE 107. Floating on neural activity, the sym bol level of the brain mirrors the 
world. But neural activity per se, which can be simulated on a computer, does not create

world. But neural activity per se, which can be simulated on a computer, does not create 
thought; that calls for higher levels of organization .

 568The diagram includes an indication of a comput er simulation of a neural network. This is 
in principle feasible, no matter how complicat ed the network, provided that the behavior 
of individual neurons can be described in terms of computations which a computer can carry out. This is a subtle postulate wh ich few people even think of questioning. 
Nevertheless it is a piece of "reductionistic fa ith"; it could be considered a "microscopic 
version" of the Church-Turing Thesis. Below we state it explicitly: 
 
CHURCH-TURING THESIS, MICROSCOPIC VERSION: The behavior of the 
components of a living being can be simulated on a computer. That is, the behavior of any component (typically assu med to be a cell) can be calculated by 
a FlooP program (i.e., general recursive function) to any desired degree of accuracy, given a sufficiently precise desc ription of the component's internal 
state and local environment.

state and local environment. 
 This version of the Church-Turing Thesis says that brain processes do not possess any 
more mystique-even though they possess more levels of organization-than, say, stomach processes. It would be unthinkable in this da y and age to suggest that people digest their 
food, not by ordinary chemical processes, but by a sort of mysterious and magic 
"assimilation". This version of the CT-Thesi s simply extends this kind of commonsense 
reasoning to brain processes. In short, it amount s to faith that the brain operates in a way

reasoning to brain processes. In short, it amount s to faith that the brain operates in a way 
which is, in principle, understandable. It is a piece of reductionist faith. 
A corollary to the Microscopic CT-Thesis is this rather terse new macroscopic 
version: 
 
CHURCH-TURING THESIS, REDUCTIONIST'S VERSION: All brain processes are 
derived from a computable substrate. 
 
This statement is about the strongest theo retical underpinning one could give in support 
of the eventual possibility of realizing Artificial Intelligence. 
Of course, Artificial Intelligence research is not aimed at simulating neural

version: 
 
Of course, Artificial Intelligence research is not aimed at simulating neural 
networks, for it is based on another kind of  faith: that probably there are significant 
features of intelligence which can be floated on top of entirely different sorts of substrates 
than those of organic brains. Figure 108 s hows the presumed relations among Artificial 
Intelligence, natural intelligence, and the real world. 
 
Parallel Progress in Al and Brain Simulation? 
 The idea that, if Al is to be achieved, the act ual hardware of the brain might one day have

 The idea that, if Al is to be achieved, the act ual hardware of the brain might one day have 
to be simulated or duplicated, is, for the present at least, quite an abhorrent thought to many Al workers. Still one wonders, "How fi nely will we need to copy the brain to 
achieve Al?" The real answer is probably th at it all depends on how many of the features 
of human consciousness you want to simulate.

 569
 
FIGURE 108. Crucial to the endeavor of Artificial Intelligence research is the notion that 
the symbolic levels of the mind can be "s kimmed off " of their neural substrate and 
implemented in other media, such as the electr onic substrate of comput ers. To what depth 
the copying of brain must go is at present completely unclear . 
 
Is an ability to play checkers well a suffici ent indicator of intelligence? If so, then 
Al already exists, since checker-playing program s are of world class. Or is intelligence an 
ability to integrate functions symbolically, as in  a freshman calculus class? If so, then AI 
already exists, since symbolic integration rou tines outdo the best people in most cases. Or 
is intelligence the ability to play  chess well? If so, then AI is well on its way, since chess-
playing programs can defeat most good amateu rs; and the level of artificial chess will 
probably continue to improve slowly.

probably continue to improve slowly. 
Historically, people have been naive a bout what qualities, if mechanized, would 
undeniably constitute intelligence. Sometimes  it seems as though each new step towards 
Al, rather than producing something which everyone agrees is real intelligence, merely reveals what real intelligence is not. If  intelligence involves learning, creativity, 
emotional responses, a sense of beauty, a sense of self, then there is a long road ahead,

emotional responses, a sense of beauty, a sense of self, then there is a long road ahead, 
and it may be that these will only be realized  when we have totally duplicated a living 
brain. 
 
Beauty, the Crab, and the Soul 
 Now what, if anything, does all this have to say about the Crab's virtuoso performance in 
front of Achilles? There are two issues clouded together here. They are:

 570 
(1)  Could any brain process, under any circumstances, distinguish completely 
reliably between true and false statements  of TNT without bei ng in violation of 
the Church-Turing Thesis-or is such an act in principle impossible? 
(2) Is perception of beauty a brain process? 
 First of all, in response to (1), if violations  of the Church-Turing Thesis are allowed, then 
there seems to be no fundamental obstacle to the strange events in the Dialogue. So what 
we are interested in is whether a believer in the Church-Turing Thesis would have to 
disbelieve in the Crab's ability. Well, it all depends on which version of the CT-Thesis you believe. For example, if you only subscrib e to the Public-Processes Version, then you 
could reconcile the Crab's behavior with it ve ry easily by positing that the Crab's ability is 
not communicable. Contrariwise, if you believe  the Reductionist's Version, you will have

not communicable. Contrariwise, if you believe  the Reductionist's Version, you will have 
a very hard time believing in the Crab's oste nsible ability (because of Church's Theorem-
soon to be demonstrated). Believing in interm ediate versions allows you a certain amount 
of wishy-washiness on the issue. Of course, switching your stand according to 
convenience allows you to waffle even more. 
It seems appropriate to present a new ve rsion of the CT-Thesis, one which is 
tacitly held by vast numbers of people, and which has been publicly put forth by several authors, in various manners. Some of the more famous ones are: philosophers Hubert

tacitly held by vast numbers of people, and which has been publicly put forth by several authors, in various manners. Some of the more famous ones are: philosophers Hubert 
Dreyfus, S. Jaki, Mortimer Taube, and J. R.  Lucas; the biologist and philosopher Michael 
Polanyi (a holist par excellence); the dis tinguished Australian neurophysiologist John 
Eccles. I am sure there are many other aut hors who have expressed similar ideas, and 
countless readers who are sympathetic. I have  attempted below to summarize their joint 
position. I have probably not done full justice to it,  but I have tried to convey the flavor as 
accurately as I can: 
 
CHURCH-TURING THESIS, SOULISTS' VERS ION: Some kinds of things which

CHURCH-TURING THESIS, SOULISTS' VERS ION: Some kinds of things which 
a brain can do can be vaguely approxima ted on a computer but not most, and 
certainly not the interesting ones. But anyw ay, even if they all could, that would 
still leave the soul to explain, and there is no way that computers have any bearing on that. 
 This version relates to the tale of the Magni ficrab in two ways. In the first place, its 
adherents would probably consider the tale to be silly and implausible, but-not forbidden 
in principle. In the second place, they woul d probably claim that appreciation of qualities

in principle. In the second place, they woul d probably claim that appreciation of qualities 
such as beauty is one of those properties associated with the elusive soul, and is therefore inherently possible only for humans, not for mere machines. 
We will come back to this second point in a moment; but first, while we are on 
the subject of "soulists", we ought to exhibit th is latest version in an even more extreme 
form, since that is the form to which larg e numbers of well-educated people subscribe 
these days: 
 
CHURCH-TURING THESIS, THEODORE ROSZAK VERSION: Computers are 
ridiculous. So is science in general.

 571This view is prevalent among certain people who see in anything smacking of numbers or 
exactitude a threat to human values. It is too bad that they do not appreciate the depth and 
complexity and beauty involved in exploring abstract structures such as the human mind, 
where, indeed, one comes in intimate contact with the ultimate questions of what to be 
human is. 
Getting back to beauty, we were about to  consider whether the appreciation of 
beauty is a brain process, and if so, wh ether it is imitable by a computer. Those who 
believe that it is not accounted for by the br ain are very unlikely to believe that a 
computer could possess it. Those who believe  it is a brain process again divide up 
according to which version of the CT-Thesis they believe. A total reductionist would believe that any brain process can in princi ple be transformed into a computer program; 
others, however, might feel that beauty is too ill-defined a notion for a computer program

others, however, might feel that beauty is too ill-defined a notion for a computer program 
ever to assimilate. Perhaps they feel that th e appreciation of beauty requires an element of 
irrationality, and therefore is incompatible with the very fiber of computers.  
Irrational and Rational Can Coexist on Different Levels 
 
However, this notion that "irrationality is incompatible with computers" rests on a 
severe confusion of levels. The mistaken notion stems from the idea that since computers are faultlessly functioning mach ines, they are therefore bound to be 
"logical" on all levels. Yet it is perfec tly obvious that a computer can be

"logical" on all levels. Yet it is perfec tly obvious that a computer can be 
instructed to print out a sequence of illogi cal statements-or, for variety's sake, a 
batch of statements having random truth values. Yet in following such instructions, a computer would not be making any mistakes! On the contrary, it 
would only be a mistake if the computer  printed out something other than the 
statements it had been instructed to  print. This illustrates how faultless 
functioning on one level may underlie sym bol manipulation on a higher level-and 
the goals of the higher level may be comp letely unrelated to the propagation of

the goals of the higher level may be comp letely unrelated to the propagation of 
Truth. 
Another way to gain perspective on this is to remember that a brain, too, is a collection of 
faultlessly functioning elements-neurons. When ever a neuron's threshold is surpassed by 
the sum of the incoming signals, BANG!-it fire s. It never happens that a neuron forgets 
its arithmetical knowledge-carelessly adding its inputs and getting a wrong answer. Even 
when a neuron dies, it continues to function correctly, in the sense that its components

Truth. 
when a neuron dies, it continues to function correctly, in the sense that its components 
continue to obey the laws of mathematics and physics. Yet as we all know, neurons are perfectly capable of supporting high-level behavi or that is wrong, on its own level, in the 
most amazing ways. Figure 109 is meant to illustra te such a clash of levels: an incorrect 
belief held in the software of a mind, supported by the hardware of a faultlessly 
functioning brain. 
The point-a point which has been made several times earlier in various contexts-is 
simply that meaning can exist on two or mo re different levels of a symbol-handling

simply that meaning can exist on two or mo re different levels of a symbol-handling 
system, and along with meaning, rightness a nd wrongness can exist on all those levels. 
The presence of meaning on a given

 572
 
FIGURE 109. The brain is rational; the mind may not be. [Drawing by the author.]

 573level is determined by whether or- not reality  is mirrored in an isomorphic (or looser) 
fashion on that level. So the fact that neur ons always perform correct additions (in fact, 
much more complex calculations) has no bear ing whatsoever on the correctness of the 
top-level conclusions supported by their machin ery. Whether one's top level is engaged in 
proving koans of Boolean Buddhism or in med itating on theorems of Zer1 Algebra, one's 
neurons are functioning rationally. By the same  token, the high-level symbolic processes 
which in a brain create the experience of appreciating beauty are perfectly rational on the bottom level, where the faultless functioning is  taking place; any irrationality, if there is 
such, is on the higher level, and is an epi phenomenon-a consequence-of the events on the 
lower level. 
To make the same point in a different way, let us say you are having a hard time

To make the same point in a different way, let us say you are having a hard time 
making up your mind whether to order a cheese burger or a pineappleburger. Does this 
imply that your neurons are also balking, ha ving difficulty deciding whether or not to 
fire? Of course not. Your hamburger-confusi on is a high-level state which fully depends 
on the efficient firing of thousands of neurons  in very organized ways. This is a little 
ironic, yet it is perfectly obvious when you thi nk about it. Nevertheless, it is probably fair 
to say that nearly all confusions about minds and computers have their origin in just such

to say that nearly all confusions about minds and computers have their origin in just such 
elementary level-confusions. 
There is no reason to believe that a. co mputer's faultlessly functioning hardware 
could not support high-level symbolic behavi or which would represent such complex 
states as confusion, forgetting, or appreciation of beauty. It would require that there exist 
massive subsystems interacting with each other according to a complex "logic". The overt behavior could appear either rationa l or irrational; but underneath it would be the 
performance of reliable, logical hardware. 
 
More Against Lucas

More Against Lucas 
 Incidentally, this kind of level distinction provides us with some new fuel in arguing 
against Lucas. The Lucas argument is base d on the idea that Gödel’s Theorem is 
applicable, by definition, to machines. In fact, Lucas makes a most emphatic 
pronunciation:  
Gödel’s theorem must apply to cybernetical  machines, because it is of the essence 
of being a machine, that it should be a concrete instantiation of a formal system.°

of being a machine, that it should be a concrete instantiation of a formal system.° 
 This is, as we have seen, true on the hardwa re level-but since there may be higher levels, 
it is not the last word on the subject. Now Lucas gives the impression that in the mind-
imitating machines he discusses, there is  only one level on which manipulation of 
symbols takes place. For instance, the Rule of Detachment (called "Modus Ponens" in his article) would be wired into the hardware a nd would be an unchangeable feature of such 
a machine. He goes further and intimates that if Modus Ponens were not an

 574immutable pillar of the machine's system, but could be overridden on occasion, then: 
 
The system will have ceased to be a form al logical system. and the machine will 
barely qualify for the title of a model for the mind.10 
 Now many programs which are being develope d in At research have very little in 
common with programs for generating truths of  number theory--programs with inflexible 
rules of inference and fixed sets of axioms. Yet they are certainly intended as "models for 
the mind". On their top level the "informal" level-there may be manipulation of images, 
formulation of analogies, forgetting of id eas, confusing of concepts, blurring of 
distinctions, and so forth. But this does not cont radict the fact that they rely on the correct

distinctions, and so forth. But this does not cont radict the fact that they rely on the correct 
functioning of their underlying hardware as much as brains rely on the correct functioning of their neurons. So At programs are still "concrete instantiations of formal systems"-but they are not machines to whic h Lucas' transmogrifica tion of Gödel’s proof 
can be applied. Lucas' argument applies mere ly to their bottom level, on which their 
intelligence-however great or small it may be-does not lie. 
There is one other way in which Lucas betrays his oversimplified vision of how

There is one other way in which Lucas betrays his oversimplified vision of how 
mental processes would have to be represen ted inside computer programs. In discussing 
the matter of consistency, he writes 
 
If we really were inconsistent machin es, we should remain content with our 
inconsistencies, and would happily affirm both halves of a contradiction. Moreover, 
we would be prepared to say absolutely  anything-which we are not. It is easily 
shown that in an inconsistent formal system everything is provable." 
 This last sentence shows that Lucas assume s that the Propositional Calculus must of

 This last sentence shows that Lucas assume s that the Propositional Calculus must of 
necessity be built into any formal system whic h carries out reasoning. In particular, he is 
thinking of the theorem << PA-P>DQ>  of the Propositional Calculus; evidently he has 
the erroneous belief that it is an inevitable feature of mechanized reasoning. However, it is perfectly plausible that logical thought processes, such as propositional reasoning, will 
emerge as consequences  of the general intelligence of an  At program, rather than being 
preprogrammed . This is what happens in humans! And there is no particular reason to

preprogrammed . This is what happens in humans! And there is no particular reason to 
assume that the strict Propositional Calculus, with its rigid rules and the rather silly 
definition of consistency that they entail, would emerge from such a program.  
An Underpinning of Al 
 We can summarize this excursion into level di stinctions and come away with one final, 
strongest version of the Church-Turing Thesis:  CHURCH-TURING THESIS, At VERSION: Me ntal processes of any sort can be 
simulated by a computer program whose underlying language is of

 575power equal to that of FlooP-that is, in wh ich all partial recursive functions can be 
programmed.  
It should also be pointed out that in pr actice, many AI researchers rely on another 
article of faith which is closely related to the CT-Thesis, and which I call the AI Thesis . It 
runs something like this: 
 
AI THESIS: As the intelligence of machines evolves, its underlying 
mechanisms will gradually converge to the mechanisms underlying human intelligence. 
 In other words, all intelligences are just va riations on a single theme; to create true 
intelligence, At workers will just have to k eep pushing to ever lower levels, closer and 
closer to brain mechanisms, if they wish their machines to attain the capabilities which we have.  
Church's Theorem 
 Now let us come back to the Crab and to the question of whether his decision procedure 
for theoremhood (which is presented in the guise of a filter for musical beauty) is

In other words, all intelligences are just va riations on a single theme; to create true 
for theoremhood (which is presented in the guise of a filter for musical beauty) is 
compatible with reality. Actually, from the events which occur in the Dialogue, we have 
no way of deducing whether the Crab's gift  is an ability to tell theorems from 
nontheorems, or alternatively, an ability to tell true statements from false ones. Of course 
in many cases this amounts to the same thi ng but Gödel’s Theorem shows that it doesn't 
always. But no matter: both of these alterna tives are impossible, if you believe the At 
Version of the Church-Turing Thesis. The proposition that it is impossible to have a

Version of the Church-Turing Thesis. The proposition that it is impossible to have a 
decision procedure for theoremhood in any fo rmal system with the power of TNT is 
known as Church's Theorem. The proposition th at it is impossible to have a decision 
procedure for number theoretical truth-if such  truth exists, which one can well doubt after 
meeting up with all the bifurcations of TN T-follows quickly from Tarski's Theorem 
(published in 1933, although the ideas were known to Tarski considerably earlier). 
The proofs of these two highly important results of metamathematics are very

The proofs of these two highly important results of metamathematics are very 
similar. Both of them follow quite quickly from self-referential constructions. Let us first consider the question of a decision procedure for TNT-theoremhood. If there were a uniform way by which people could decide  which of the classes "theorem" and 
"nontheorem" any given formula X fell into, then, by the CT-Thesis (Standard Version), 
there would exist a terminating FlooP program (a general recursive function) which could 
make the same decision, when given as i nput the Gödel number of formula X. The

make the same decision, when given as i nput the Gödel number of formula X. The 
crucial step is to recall that any property that can be tested for by a terminating FlooP program is represented in TNT. This mean s that the property of TNT-theoremhood would 
be represented (as distinguished from merely expressed) inside TNT. But as we shall see 
in a moment, this,

 576would put us in hot water, for if theoremhood is a representable attribute, then Gödel’s 
formula G becomes as vicious as the Epimenides paradox. 
It all hinges on what G says: "G is not a theorem of TNT". Assume that G were a 
theorem. Then, since theoremhood is supposedly represented, the TNT-formula which asserts "G is a theorem" would be a theorem of TNT. But this formula is -G, the negation 
of G, so that TNT is inconsistent. On the other hand, assume G were not a theorem. Then 
once again by the supposed representability of theoremhood, the formula which asserts 
"G is not a theorem" would be a theorem of TNT. But this formula is G, and once again 
we get into paradox. Unlike the situation be fore, there is no resolution of the paradox. 
The problem is created by the assumption that theorernhood is represented by some formula of TNT, and therefore we must back track and erase that assumption. This forces

The problem is created by the assumption that theorernhood is represented by some formula of TNT, and therefore we must back track and erase that assumption. This forces 
us also to conclude that no FlooP program can tell the Gödel numbers of theorems from 
those of nontheorems. Finally, if we accept th e Al Version of the CT-Thesis, then we 
must backtrack further, and conclude that  no method whatsoever could exist by which 
humans could reliably tell theorems from nont heorems-and this includes determinations 
based on beauty. Those who subscribe only to the Public-Processes Version might still think the Crab's performance is possible; but of all the versions, that one is perhaps the 
hardest one to find any justification for.

hardest one to find any justification for. 
 
Tarski's Theorem 
 Now let us proceed to Tarski's result. Tars ki asked whether there could be a way of 
expressing in TNT the concept of number-t heoretical truth. That theoremhood is 
expressible (though not representable) we ha ve seen; Tarski was interested in the 
analogous question regarding the notion of truth. More specifically, he wished to 
determine whether there is any TNT-formula with a single free variable a which can be 
translated thus:  
"The formula whose Gödel number is a expresses a truth."

Now let us proceed to Tarski's result. Tars ki asked whether there could be a way of 
"The formula whose Gödel number is a expresses a truth." 
 Let us suppose, with Tarski, that there is one-which we'll abbreviate as TRUE{a}. Now 
what we'll do is use the diagonalization method to produce a sentence which asserts about itself that it is untrue. We copy the Godel method exactly, beginning with an "uncle":  
3a:<-TRUE{a}nARITHMOQUINE{a",a}> 
 Let us say the Gödel number of the uncle is t. We arithmoquine this very uncle, and produce the Tarski formula T:  
3a:<--TRUE{a}AARITHMOQUINE{SSS ... SSSO/a",a}> 
t S's

 577 
When interpreted, it says:  
"The arithmoquinification of t is the 
Gödel number of a false statement." 
 But since the arithmoquinification of t is T's own Gödel number, Tarski's formula T 
reproduces the Epimenides paradox to a tee inside  TNT, saying of itself, "I am a falsity". 
Of course, this leads to the conclusion that it must be simultaneously true and false (or 
simultaneously neither). There arises now an interesting matter: What is so bad about 
reproducing the Epimenides paradox? Is it of any consequence? After all, we already 
have it in English, and the English language has not gone up in smoke.  
The Impossibility of the Magnificrab 
 The answer lies in remembering that there ar e two levels of meaning involved here. One 
level is the level we have just been using; th e other is as a statemen t of number theory. If

level is the level we have just been using; th e other is as a statemen t of number theory. If 
the Tarski formula T actually existed, then it would be a statement about natural numbers that is both true and false at once! There is  the rub. While we can always just sweep the 
English-language Epimenides paradox under the rug, saying that its subect matter (its 
own truth) is abstract, this is' not so when it becomes a concrete statement about numbers! If we believe this is a ridiculous state of affairs, then we have to undo our 
assumption that the formula TRUE{a} exists . Thus, there is no way of expressing the

assumption that the formula TRUE{a} exists . Thus, there is no way of expressing the 
notion of truth inside TNT. Notice that this makes truth a far more elusive property than 
theoremhood, for the latter is expressible. The same backtracking reasons as before 
(involving the Church-Turing Thesis, Al Version) lead us to the conclusion that  
The Crab's mind cannot be a truth-recognizer any more than it is a TNT-theorem-recognizer. 
 The former would violate the Tarski-Chur ch-Turing Theorem ("There is no decision 
procedure for arithmetical truth"), while the latter would violate Church's Theorem.  
Two Types of Form 
 It is extremely interesting, then, to think about the meaning of the word "form" as it

 It is extremely interesting, then, to think about the meaning of the word "form" as it 
applies to constructions of arbitrarily comple x shapes. For instance, what is it that we 
respond to when we look at a painting and feel its beauty? Is it the "f orm" of the lines and 
dots on our retina? Evidently it must be, fo r that is how it gets passed along to the 
analyzing mechanisms in our heads-but the co mplexity of the processing makes us feel 
that we are not merely looking at a two-dimensional surface; we are responding to

 578some sort of inner meaning inside the picture, a multidimensional aspect trapped 
somehow inside those two dimensions. It is the word "meaning" which is important here. 
Our minds contain interpreters which accept two-dimensional patterns and then "pull" fr om them high-dimensional notions which are 
so complex that we cannot consciously descri be them. The same can be said about how 
we respond to music, incidentally. 
It feels subjectively that the pulling-out m echanism of inner meaning is not at all 
akin to a decision procedure which checks for the presence or absence of some particular quality such as well-formedness in a string. Pr obably this is because inner meaning is 
something which reveals more of itself over a period of time. One can never be sure, as one can about well-formedness, that one has finished with the issue. 
This suggests a distinction that could be drawn between two sens es of "form" in

This suggests a distinction that could be drawn between two sens es of "form" in 
patterns which we analyze. First, there ar e qualities such as well-formedness, which can 
be detected by predictably terminating tests, as in BlooP programs. These I propose to call syntactic qualities of form. One intuitively feels about the syntactic aspects of form 
that they lie close to the surface, and th erefore they do not provoke the creation of 
multidimensional cognitive structures. 
By contrast, the semantic  aspects of form are those which cannot be tested for in

By contrast, the semantic  aspects of form are those which cannot be tested for in 
predictable lengths of time: they require open-ended tests . Such an aspect is theoremhood 
of TNT-strings, as we have seen. You cannot ju st apply some standard test to a string and 
find out if it is a theorem. Somehow, the fact  that its meaning is involved is crucially 
related to the difficulty of telling whether or  not a string is a TNT-theorem. The act of 
pulling out a string's meaning involves, in essen ce, establishing all the implications of its

pulling out a string's meaning involves, in essen ce, establishing all the implications of its 
connections to all other strings, and this lead s, to be sure, down an open-ended trail. So 
"semantic" properties are connected to open-ended searches because, in an important sense, an object's meaning is not localized within the object itself. This is not to say that 
no understanding of any object's meaning is po ssible until the end of time, for as time 
passes, more and more of the meaning unfolds . However, there are always aspects of its 
meaning which will remain hidden arbitrarily long.

meaning which will remain hidden arbitrarily long.  
Meaning Derives from Connections to Cognitive Structures 
 
Let us switch from strings to pieces of  music, just for variety. You may still 
substitute the term "string" for every refe rence to a piece of mu sic, if you prefer. 
The discussion is meant to be general, but its flavor is better gotten across, I feel, 
by referring to music. There is a strange  duality about the meaning of a piece of 
music: on the one hand, it seems to be sp read around, by virtue of its relation to

music: on the one hand, it seems to be sp read around, by virtue of its relation to 
many other things in the world-and yet,  on the other hand, the meaning of a piece 
of music is obviously derived from the music itself, so it must be localized 
somewhere inside the music. The resolution of this dilemma comes from thinking about the interpreter-the mechanism which does the pulling-out of meaning. (By "inter

 579preter in this context, I mean not -the pe rformer of the piece, but  the mental mechanism 
in the listener which derives meaning when the piece is played.) The interpreter may 
discover many important aspects of a piece's m eaning while hearing it for the first time; 
this seems to confirm the notion that the m eaning is housed in the piece itself, and is 
simply being read off. But that is only part  of the story. The music interpreter works by 
setting up a multidimensional cognitive structur e-a mental representation of *_he piece-
which it tries to integrate with pre-exis tent information by finding links to other 
multidimensional mental structures which encode previous experiences. As this process takes place, the full meaning gradually unfolds . In fact, years may pass before someone 
comes to feel that he has penetrated to the core meaning of a piece. This seems to support

comes to feel that he has penetrated to the core meaning of a piece. This seems to support 
the opposite view: that musical meaning is sp read around, the interpreter's role being to 
assemble it gradually. 
The truth undoubtedly lies somewhere in between: meanings-both musical and 
linguistic-are to some extent localizable , to some extent spread around. In the 
terminology of Chapter VI, we can say that mu sical pieces and pieces of text are partly 
triggers, and partly carriers of explicit m eaning. A vivid illustration of this dualism of 
meaning is provided by the example of a tablet  with an ancient inscription: the meaning

meaning is provided by the example of a tablet  with an ancient inscription: the meaning 
is partially stored in the libraries and the brains of scholars around the world, and yet it is 
also obviously implicit in the tablet itself. 
Thus, another way of characterizing the difference between "syntactic" and 
"semantic" properties (in the just-proposed se nse) is that the syntactic ones reside 
unambiguously inside the object under cons ideration, whereas semantic properties 
depend on its relations with a potentially infin ite class of other objects, and therefore are

depend on its relations with a potentially infin ite class of other objects, and therefore are 
not completely localizable. There is nothing cryptic or hidden, in principle, in syntactic properties, whereas hiddenness is of the essence in semantic properties. That is the reason 
for my suggested distinction between "syntactic" and "semantic" aspects of visual form. 
 
Beauty, Truth, and Form 
 
What about beauty? It is certainly not a s yntactic property, according to the ideas above. 
Is it even a semantic property? Is beauty a property which, for instance, a particular

Is it even a semantic property? Is beauty a property which, for instance, a particular 
painting has? Let us immediately restrict our  consideration to a single viewer. Everyone 
has had the experience of finding something beautiful at one time, dull another time-and probably intermediate at other times. So is b eauty an attribute which varies in time? One 
could turn things around and say that it is th e beholder who has varied in time. Given a

could turn things around and say that it is th e beholder who has varied in time. Given a 
particular beholder of a particular painting at a particular time, is it reasonable to assert that beauty is a quality that is definitely pr esent or absent? Or is there still something ill-
defined and intangible about it? 
Different levels of interpreter probably could be invoked in every

 580person, depending on the circumstances. These various interpreters pull out different 
meanings, establish different connections, and generally evaluate all deep aspects differently. So it seems that this notion of beau ty is extremely hard to pin down. It is for 
this reason that I chose to link beauty, in th e Magnificrab, with truth, which we have seen 
is also one of the most intangible notions in all of metamathematics.  
The Neural Substrate of the Epimenides Paradox 
 I would like to conclude this Chapter with so me ideas about that central problem of truth, 
the Epimenides paradox. I think the Tarski reproduction of the Epimenides paradox inside TNT points the way to a deeper unders tanding of the nature of the Epimenides 
paradox in English. What Tarski found was that his version of the paradox has two 
distinct levels to it. On one level, it is a sentence about itself whic h would be true if it

distinct levels to it. On one level, it is a sentence about itself whic h would be true if it 
were false, and false if it were true. On the other level-which I like to call the arithmetical substrate-it is a sentence about integers which is true if and only if false. 
Now for some reason this latter bothers pe ople a lot more than the former. Some 
people simply shrug off the former as "mean ingless", because of its self-referentiality. 
But you can't shrug off paradoxical statements about integers. Statements about integers 
simply cannot be both true and false.

simply cannot be both true and false. 
Now my feeling is that the Tarski tr ansformation of the Epimenides paradox 
teaches us to look for a substrate in the English-language version. In the arithmetical 
version, the upper level of meaning is supporte d by the lower arithmetical level. Perhaps 
analogously, the self-referential sentence which we  perceive ("This sentence is false") is 
only the top level of a dual-level entity. What would be the lower level, then? Well, what is the mechanism that language rides on? The brain. Therefore one ought to look for a

only the top level of a dual-level entity. What would be the lower level, then? Well, what is the mechanism that language rides on? The brain. Therefore one ought to look for a 
neural substrate to the Epimenides paradox-a lower level of physical events which clash 
with each other. That is, two events which by their nature cannot occur simultaneously. If 
this physical substrate exists, then the r eason we cannot make heads or tails of the 
Epimenides sentence is that our brains are trying to do an impossible task. 
Now what would be the nature of the conflicting physical events? Presumably 
when you hear the Epimenides sentence, your brain sets up some "coding" of the

when you hear the Epimenides sentence, your brain sets up some "coding" of the 
sentence-an internal configuration of interac ting symbols. Then it tries to classify the 
sentence as "true" or "false". This classifyi ng act must involve an attempt to force several 
symbols to interact in a particular way. (P resumably this happens when any sentence is 
processed.) Now if it happens that the act of  classification would physically disrupt the 
coding of the sentence-something which would ordinarily never happen-then one is in

coding of the sentence-something which would ordinarily never happen-then one is in 
trouble, for it is tantamount to trying to force a record player to play its self-breaking 
record. We have described the conflict in physical  terms, but not in neural terms. If this 
analysis is right so

 581far, then presumably the rest of the di scussion could be carried on when we know • 
something about the constitution of the "symbol s" in the brain out of neurons and their 
firings, as well as about the way that sentences become converted into "codings". 
This sketch of the neural substrate of the Epimenides paradox suggests (to me, at 
least) that the resolution of the English version of the Epimenides paradox might be 
similar to that for the Tarski version. Th e resolution involves abandoning the notion that 
a brain could ever provide a fully accurate representation for the notion of truth. The 
novelty of this resolution lies in its suggestion that a total modeling of truth is impossible 
for quite physical reasons: namely, such a modeling would require physically incompatible events to occur in a brain.

SHRDLU, Toy of Man's Designing 
 582 
SHRDLU , 
Toy of Man's Designing 
 
One day, Eta Oin wanders into the MIT Artificial Intelligence Laboratory, 
where she runs into the brilliant y oung computer program, SHRDLU. It  
happens that SHRDLU is just dying for someone to try out the newly 
developed human being "Dr. Tony Earrw ig". SHRDLU explains that this 
Earrwig is fairly intelligent in the limited domain of analyzing a conversation 
about a "toy world" containing blocks of various shapes, sizes, and colors-
blocks which sit on a table, and wh ich can be picked up and moved around. 
Eta Oin, fascinated, begins enthusia stically typing to SHRDLU. Earrwig 
dangles over her shoulder and provides a sort of running commentary, as 
promised. 
 
1. Eta Oin: pick up a big red block. [Refer to Fig. 110. 
SHRDLU: OK. 
Dr. Tony Earrwig: SHRDLU answers "OK" when it carries out a command. In order 
to pick up the red block, it had to clear it off by finding a space for the green one 
and moving the green one away.

promised. 
 
and moving the green one away. 
2. Eta Oin: grasp the pyramid.  
SHRDLU: I DON'T UNDERSTAND WHICH PYRAMID YOU MEAN. 
Dr. Tony Earrwig: There are three pyramids  in the scene, and SHRDLU knows that 
phrases beginning with "the" are intended to refer to a specific object the speaker has in mind. 
 
FIGURE 110 . "Pick up a big red block." [Fro m Terry Winograd, Understanding Natural 
Language, (New York: Academic Press, 1972), p. 8.1

 583
 
FIGURE 111. " Find a block which is bigger than the one you are holding and put it into the 
box." Adapted from Terry Winograd, Understanding Natural Language, p.9.]  
 
3. Eta Oin: find a block which is taller than  the one you are holding and put it into the box. 
[Refer to Fig. 111.] 
Dr. Tony Earrwig: SHRDLU has to figure out w hat is meant by "one" and "it", as well as 
using deduction to decide which block it is holding. 
SHRDLU: BY "IT", I ASSUME YOU MEAN THE BLOCK WHICH IS TALLER 
THAN THE ONE I AM HOLDING. 
Dr. Tony Earrwig: There were two possible referents for "it": the one selected, and "the 
one you are holding". SHRDLU has heuristics for choosing between them.   
Dr. Tony Earrwig: SHRDLU's hand has to set down the red block pick up the taller blue 
one. 
Eta Oin: what does the box contain? 
SHRDLU: THE BLUE PYRAMID AND THE BLUE BLOCK. 
Eta Oin: what is the pyramid supported by?

one. 
Eta Oin: what is the pyramid supported by? 
Dr. Tony Earrwig: Earlier SHRDLU responded to "the pyramid" by saying "I DON'T 
UNDERSTAND WHICH PYRAMID YOU MEAN."  Now, it assumes that you mean the 
one just mentioned.  
SHRDLU: THE BOX. 
Eta Oin: how many blocks are not in the box?

 584Dr. Tony Earrwig: Negatives such as "not" and "no" are handled whether they appear in 
clauses or noun groups (such as "no boxes") .  
SHRDLU: FOUR OF THEM. 
7. Eta Oin: is at least one of them narrower than the one which i told you to pick up? 
Dr. Tony Earrwig: The phrase "at least one  of them" must be analyzed to refer 
back to the previous question, and "the one which ..." must also refer to blocks. Noun groups can have complicated numbers such as "at least one", "more than 
three", "exactly seven", etc.. The memory  of past events is used to deduce which 
block is "the one which I told you to pick up". Notice the complicated syntactic 
structure in which "the one" is actually the object of a clause embedded two levels deeper. SHRDLU's parser incorporates a grammar that handles this kind of 
complexity in a general way. 
SHRDLU: YES, THE RED CUBE. 
8. Eta Oin: is it supported? 
SHRDLU: YES, BY THE TABLE.

SHRDLU: YES, BY THE TABLE. 
Dr. Tony Earrwig: SHRDLU has heuristics about what information is relevant to 
a person who asks a question, even if the pe rson does not request that information 
specifically. 
9. Eta Oin: can the table pick up blocks? 
SHRDLU: NO, 
Dr. Tony Earrwig: SHRDLU's semantic ru les include the knowledge that "picking 
up" must be done by an animate object , and that the table is not animate. 
10. Eta Oin: can a pyramid be supported by a block? 
SHRDLU: YES. 
Dr. Tony Earrwig:• SHRDLU's deductive system finds an actual example, so it 
knows this is possible .

knows this is possible . 
11. Eta Oin: can a pyramid support a pyramid? 
SHRDLU: I DON'T KNOW. 
Dr. Tony Earrwig: The question makes se nse, but there is no example in the 
scene. SHRDLU has no way of looking in to the details of its programs, even 
though these ultimately define its capabilities . 
12. Eta Oin: stack up two pyramids. 
SHRDLU: I CANT. 
Dr. Tony Earrwig: SHRDLU tries and fails. 
13. Eta Oin: the blue pyramid is mine. 
SHRDLU: I UNDERSTAND. 
Dr. Tony Earrwig: All of the informati on about colors, shapes, sizes and locations

Dr. Tony Earrwig: All of the informati on about colors, shapes, sizes and locations 
of the objects is known to SHRDLU initially. Possession is left as an unknown 
property of objects about which it can be told. It stores this information in its data 
base. 
14. Eta Gin: i own all blocks which are not red, but i don't own anything which supports a 
pyramid. 
SHRDLU: I UNDERSTAND.

 585
 
FIGURE 112. " Will you please stack up both of the red blocks and either a green cube or 
a pyramid?" [Adapted from Terry Winograd, Understanding Natural Language, p. 12 . 
 
Dr. Tony Earrwig: The result of understandi ng these sentences is the creation of 
new procedures within SHRDLU for deducing whether you own something. 
15. Eta Oin: do i own the box? 
SHRDLU: NO. 
Dr. Tony Earrwig: The box supports whatever it contains, including a pyramid. 
16. Eta Oin: do i own anything in the box? 
SHRDLU: YES, TWO THINGS: THE BLUE BLOCK AND PYRAMID. 
17. Eta Oin: will you please stack up both of the red blocks and either a green cube or a 
pyramid? [Refer to Fig. 112. SHRDLU: OK. 
Dr. Tony Earrwig: Logical connectives, such as "and", "or", "either", etc. are 
handled in both the grammar and semantics. A command is executed even if it looks like a question. The. command is ca rried out by the following series of

handled in both the grammar and semantics. A command is executed even if it looks like a question. The. command is ca rried out by the following series of 
steps: SHRDLU puts a green cube on the large red block (it chooses the green 
cube with nothing on it), then removes the small pyramid from the little red 
cube, so that it can use that cube to complete the stack as specified.

 58618. Eta Oin: which cube is sitting on the table? 
SHRDLU: THE LARGE GREEN ONE WHICH SUPPORTS THE RED PYRAMID. 
Dr. Tony Earrwig: Ordinarily SHRDLU w ould be satisfied with calling this "A 
BIG GREEN CUBE", but since you asked "whi ch cube" it has to be more spec /lc 
in naming it. It has procedures for us ing substitute nouns (such as "one") and 
pronouns in answers to make the discourse more natural. 
19. Eta Oin: is there a large block behind a pyramid? 
SHRDLU: YES, THREE OF THEM: A LARGE RED ONE, A LARGE GREEN 
CUBE AND THE BLUE ONE. 
20. Eta Oin: put a small one onto the green cube which supports a pyramid. 
Dr. Tony Earrwig: "A small one" must cont rast with your earlier phrase "a large 
block", so this means "a small block". SHRDLU: OK, 
21. Eta Oin: put the littlest pyramid on top of it. 
Dr. Tony Earrwig: Words like "littlest" ar e not in SHRDLU's dictionary but are 
interpreted from the root farms like "little ". SHRDLU compares the sizes of all

interpreted from the root farms like "little ". SHRDLU compares the sizes of all 
the pyramids to carry out the action. 
SHRDLU• OK. 
22. Eta Oin: how many things are on top of green cubes? 
SHRDLU: I'M NOT SURE WHAT YOU MEAN BY "ON TOP OF" IN THE 
PHRASE "ON TOP OF GREEN CUBES" DO YOU MEAN 1-DIRECTLY ON THE SURFACE 2-ANYWHERE ON TOP OF IT? 
Dr. Tony Earrwig: In the previous sentence, the phrase "on top of " was 
interpreted without trouble because the context made it clear what was meant. In 
this one, the heuristics for disambigua tion do not come up with a clearcut

this one, the heuristics for disambigua tion do not come up with a clearcut 
definition, so SHRDLU had to ask you fo r more information. The paraphrases 
were included with the dictionary definiti ons of "on". A request like this can 
also occur when the meaning of a pronoun such as "it" or "they" is unclear.  
23. Eta Din: had you touched any pyramid before you put the green one on the little 
cube? SHRDLU: YES, THE GREEN ONE. 
Dr. Tony Earrwig: SHRDLU deduces fr om its memory what event is being 
referred to, and can relate other events to it in time. The analysis includes

referred to, and can relate other events to it in time. The analysis includes 
complex tenses of verb groups, and th ere are special facilities for keeping 
track of the various forms of irregular verbs like "have ". 
24. Eta Oin: when did you pick it up? 
SHRDLU: WHILE I WAS STACKING UP  THE RED CUBE, A LARGE RED 
BLOCK AND A LARGE GREEN ONE. 
Dr. Tony Earrwig: A time is described by  naming the main goal which was being 
worked on. 
25. Eta Oin: why?

 587 
Dr. Tony Earrwig: SHRDLU remembers immediately previous sentences to 
understand questions like this and the following ones. 
SHRDLU: TO GET RID OF IT. 
Dr. Tony Earrwig: By keeping track of sel ected parts of the original subgoal tree,  
SHRDLU has some understanding of its own motives. 
26. Eta Oin: why did you do that? 
SHRDLU: TO CLEAN OFF THE RED CUBE. 
Dr. Tony Earrwig: "That" refers to the action mentioned in SHRDLU's last 
answer. 
27. Eta Oin: why did you clear off that cube? 
Dr. Tony Earrwig: Here, "that" means th e one just referred to. SHRDLU: TO 
PUT IT ON A LARGE GREEN CUBE. 
28. Eta Oin: why did you do that? 
SHRDLU: TO STACK UP THE RED CUBE, A LARGE RED BLOCK, AND A LARGE GREEN CUBE. 
29. Eta Oin: why did you do that? 
SHRDLU.' BECAUSE YOU ASKED ME TO. 
Dr. Tony Earrwig: SHRDLU remembers what it did, not how the request was 
worded . 
30. Eta Oin: how did you do it? 
Dr. Tony Earrwig: "It" refers to the action mentioned in the last question,

worded . 
Dr. Tony Earrwig: "It" refers to the action mentioned in the last question, 
rather than the answer. 
SHRDLU: BY PUTTING A LARGE RE D BLOCK ON THE TABLE; THEN 
LETTING GO OF IT; THEN PUTTING A LARGE GREEN CUBE ON IT; THEN LETTING GO OF THAT CUBE; THEN PUTTING THE RED CUBE 
‘ ON THAT CUBE; THEN LETTING GO OF THAT CUBE. 
Dr. Tony Earrwig: This is a description of the top-level subgoals used to 
achieve the goal. 
31. Eta Oin: how many objects did you touch while you were doing it? 
Dr. Tony Earrwig: Time references can refer to a time span, using words like 
"while". 
SHRDLU: FOUR OF THEM.

"while". 
32. Eta Oin: what did the red cube support before you started to clean it off? 
SHRDLU: THE GREEN PYRAMID. 
Dr. Tony Earrwig: By remembering what has been moved, it is possible to 
deduce things about earlier states of the scene. You can use complex nestings of clauses like "before you started to clean it off ". 
33. Eta Oin: there were five blocks to the left of the box then. 
Dr. Tony Earrwig: "Then" must refer to the time mentioned in the last sentence. 
SHRDLU: NO, ONLY FOUR OF THEM: THE RED CUBE, TWO LARGE

SHRDLU: NO, ONLY FOUR OF THEM: THE RED CUBE, TWO LARGE 
GREEN CUBES, AND A LARGE RED BLOCK. 
Dr. Tony Earrwig: Earlier in the di alogue, information about "owning" was 
accepted at face value. But SHRDLU knows that it has complete infornia-

 588tion about the basic properties of obj ects, including location. It checks any 
statements about these to see whether it agrees or disagrees. 
34. Eta Oin: put the blue pyramid on the block in the box. 
Dr. Tony Earrwig: This is syntactically ambiguous, but is understood without 
ambiguity since the parsing corresponding to "put the blue pyramid which is on the block into the box" does not make sense in the current scene, whereas "put the blue pyramid onto the block which is in the box" does. SHRDLU: OK.  
35. Eta Oin: is there anything which is bigger than every pyramid but is not as wide as 
the thing that supports it? 
Dr. Tony Earrwig: SHRDLU understands quantifiers such as "every", "any", 
"some", etc., and structures such as "bigge r than", "as wide as", etc.. Here "it" 
refers to a noun group within the same se ntence but two levels up in the parsing 
tree. 
SHRDLU: YES, THE BLUE BLOCK. 
36. Eta Oin: does a steeple 
SHRDLU: SORRY, I DON'T KNOW THE WORD "STEEPLE".

tree. 
SHRDLU: SORRY, I DON'T KNOW THE WORD "STEEPLE". 
Dr. Tony Earrwig: SHRDLU interrupts imme diately when it sees a word it doesn't 
know. 
37. Eta Oin: a "steeple" is a stack which contains two green cubes and a pyramid. 
Dr. Tony Earrwig: The quote marks indica te that you are defining a new word. 
The definition is added to SHRDLU's dictionar y in a form compatible with all the 
words it already knows. 
38. Eta Oin: are there any steeples now? 
Dr. Tony Earrwig: The new definition is used as a description to recognize 
steeples. 
39. Eta Oin: build one.

know. 
steeples. 
39. Eta Oin: build one. 
Dr. Tony Earrwig: The description given  in the definition is enough to deduce a 
plan for building new objects as well as recognizing them. The word "one" means 
"a steeple", and the plan is carried out. First, one of the large green cubes is 
cleared off by removing the red cube and two pyramids. Then the two green cubes 
and the tall red pyramid are stacked up to fit the description. 
40. Eta Oin: call the biggest block "superblock". 
Dr. Tony Earrwig: You can also give objects names.

Dr. Tony Earrwig: You can also give objects names. 
41. Eta Oin: have you picked up superblock since we began? 
42. Eta Oin: why did you drop it? 
SHRDLU: BY "IT", I ASSUME YOU MEAN THE SUPERBLOCK. I CAN'T 
EXPLAIN A NON-EXISTENT EVENT.

 589 
Dr. Tony Earrwig: There is no record in  SHRDLU's memory of any block being 
dropped. The answer routines have special  checks for this type of "loaded 
question" which presupposes things which aren't true. 
43. Eta Oin: is there anything to the right of the red pyramid? 
SHRDLU: YES, FOUR THINGS: THE BOX, THE RED CUBE, THE BLUE PYRAMID AND THE BLOCK NAMED SUPERBLOCK. 
Dr. Tony Earrwig: Once an object has a name,  the name is used to help identify it 
in responses. 
44. Eta Oin: thank you. 
SHRDLU: YOU'RE WELCOME! 
Dr. Tony Earrwig: SHRDLU's grammar is  written as a procedure, which gives 
SHRDLU the capability to carry out special sorts of parsing for 
idioms .

Artificial Intelligence: Retrospects 
 590CHAPTER XVI11 
 
Artificial Intelligence:  
Retrospects 
 
Turing 
 
  IN 1950, ALAN TURING wrote a most 
prophetic and provocative article on Artificial Intelligence. It was entitled "Computing Machinery and Intelligence" and appeared in the journal Mind .' I will say some things about that

Turing 
 
prophetic and provocative article on Artificial Intelligence. It was entitled "Computing Machinery and Intelligence" and appeared in the journal Mind .' I will say some things about that 
article, but I would like to precede them with some remarks about Turing the man. Alan Mathison Turing was born in London in 1912. He was a child full of curiosity and humor. Gifted in mathematics, he went to Cambridge where his interests in machinery and mathematical logic cross-fertilized and resulted in his famous paper on "computable numbers", in which he invented the theory of Turing machines and demonstrated the unsolvability of the halting problem; it was published in 1937. In the 1940's, his interests turned from the theory of computing machines to the actual building of real computers. He was a major figure in the development of computers in Britain, and a staunch defender of Artificial In-  FIGURE 113. Alan Turing, after a

successful race (May, 1950). [From 
Sara Turing, Alan M. Turing (Cambridge, U. K.:W. Heffer & Sons, 1959).

 591telligence when it first came under attack. One of his best friends was David 
Champernowne (who later worked on comput er composition of music). Champernowne 
and Turing were both avid chess players a nd invented "round-the-house" chess: after 
your move, run around the house-if you get b ack before your opponent has moved, you're 
entitled to another move. More seriously, Tu ring and Champernowne invented the first 
chess playing program, called "Turochamp . Turing died young, at 41-apparently of an 
accident with chemicals. Or some say su icide. His mother, Sara Turing, wrote his 
biography. From the people she quotes, one gets the sense that Turing was highly 
unconventional, even gauche in some ways , but so honest and decent that he was 
vulnerable to the world. He loved games, che ss, children, and bike riding; he was a strong 
long-distance runner. As a student at Camb ridge, he bought himself a second-hand violin

long-distance runner. As a student at Camb ridge, he bought himself a second-hand violin 
and taught himself to play. Though not very  musical, he derived a great deal of 
enjoyment from it. He was somewhat eccentric , given to great bursts of energy in the 
oddest directions. One area he explored wa s the problem of morphogenesis in biology. 
According to his mother, Turing "h ad a particular fondness for the Pickwick Papers ", but 
"poetry, with the exception of Shakespeare' s, meant nothing to him." Alan Turing was 
one of the true pioneers in the field of computer science.  
The Turing Test

The Turing Test 
 Turing's article begins with the sentence: "I propose to consider the question `Can 
machines think?"' Since, as he points out, these are loaded terms, it is obvious that we should search for an operational way to appr oach the question. This, he suggests, is 
contained in what he calls the "imitation game"; it is nowadays known as the Turing test . 
Turing introduces it as follows:  
It is played with three people: a man (A ), a woman (B), and an interrogator (C) 
who may be of either sex. The interrogato r stays in a room apart from the other

who may be of either sex. The interrogato r stays in a room apart from the other 
two. The object of the game for the interroga tor is to determine which of the other 
two is the man and which is the woman. He knows them by labels X and Y, and at the end of the game he says either "X is A and Y is B" or "X is B and Y is A". The 
interrogator is allowed to put questions to A and B thus: 
 
C: Will X please tell me the length of his or her hair? 
 
Now suppose X is actually A, then A must an swer. It is A's object in the game to 
try to cause C to make the wrong identification. His answer might therefore be

try to cause C to make the wrong identification. His answer might therefore be 
 
"My hair is shingled, and the longest strands are about nine inches long."  
 
In order that tones of voice may not help  the interrogator the answers should be 
written, or better still, typewritten. The ideal arrangement is to have a teleprinter communicating between the two rooms. A lternatively the questions and answers 
can be repeated by an intermediary. The obj ect of the game for the third player (B) 
is to help the interrogator. The best strategy for her is probably to give truthful

is to help the interrogator. The best strategy for her is probably to give truthful 
answers. She can add such things as "I am the woman, don't listen to him!" to her 
answers, but it will avail nothing as the man can make similar remarks.

 592We now ask the question, "What will happen when a machine takes the part of A 
in this game Will the interrogator decide wrongly as often when the game is played like this as he does when the game  is played between a man and a woman? 
These questions replace our original, "Can machines think?"
2 
 After having spelled out the nature of his test, Turing goes on to make some 
commentaries on it, which, given the year he was writing in, are quite sophisticated. To 
begin with, he gives a short hypothetical dialogue between interrogator and interrogatee:
3 
 
Q. Please write me a sonnet on the subject of the Forth Bridge [a bridge over the 
Firth of Forth, in Scotland].  A. Count me out on this one. I never could write poetry. Q. Add 34957 to 70764. A. (Pause about 30 seconds and then give as answer) 105621.  Q. Do you play chess? A. Yes. Q. I have K at my K1, and no other pieces. You have only K at K6  and R at R1. It

3 
 
Firth of Forth, in Scotland].  A. Count me out on this one. I never could write poetry. Q. Add 34957 to 70764. A. (Pause about 30 seconds and then give as answer) 105621.  Q. Do you play chess? A. Yes. Q. I have K at my K1, and no other pieces. You have only K at K6  and R at R1. It 
is your move. What do you play?  A. (After a pause of 15 seconds) R-R8 mate. 
 Few readers notice that in the arithmetic problem, not only is there an inordinately long delay, but moreover, the answer given is wrong!  This would be easy to account for if the 
respondent were a human: a mere calculationa l error. But if the respondent were a 
machine, a variety of explanations are possible. Here are some:  
(1)  a run-time error on the hardware level (i.e., an irreproducible fluke); 
(2) an unintentional hardware (or programming) (reproducibly) causes

Few readers notice that in the arithmetic problem, not only is there an inordinately long delay, but moreover, the answer given is wrong!  This would be easy to account for if the 
(2) an unintentional hardware (or programming) (reproducibly) causes 
arithmetical mistakes;  (3)  a ploy deliberately inserted by the machine's programmer (or builder) to introduce occasional arithmetical mistakes, so as to trick interrogators; (4) an unanticipated epiphenomenon: the program has a hard time thinking 
abstractly, and simply made "an honest mistake", which it might not make the 
next time around; (5)  a joke on the part of the machine itself, deliberately teasing its interrogator. 
 Reflection on what Turing might have meant by this subtle touch opens up just about all

 Reflection on what Turing might have meant by this subtle touch opens up just about all 
the major philosophical issues connected with Artificial Intelligence. 
Turing goes on to point out that  
The new problem has the advantage of dr awing a fairly sharp line between the 
physical and the intellectual capacities of a man. . . . We do not wish to penalize the machine for its inability to shine in beauty competitions, nor to 
penalize a man for losing in a race against an airplane.' 
 One of the pleasures of the article is to see how far Turing traced out each

 593line of thought, usually turning up a seeming c ontradiction at some stage and, by refining 
his concepts, resolving it at a deeper leve l of analysis. Because of this depth of 
penetration into the issues. the article still sh ines after nearly thirty years of tremendous 
progress in computer development and inte nsive work in Al. In the following short 
excerpt you can see some of this rich back-and-forth working of ideas: 
 
The game may perhaps be criticized on the ground that the odds are weighted too heavily against the machine. If the man were  to try to pretend to be the machine he 
would clearly make a very poor showi ng. He would be given away at once by 
slowness and inaccuracy in arithmetic. May not machines carry out something which ought to be described as thinking but  which is very different from what a 
man does: This objection is a very strong one, but at least we can say that if,

man does: This objection is a very strong one, but at least we can say that if, 
nevertheless, a machine can be constructed to play the imitation game satisfactorily, 
we need not be troubled by this objection. 
It might be urged that when playing the "imitation game" the best strategy 
for the machine may possibly be something other than imitation of the behaviour of 
a man. This may be, but I think it is unlikely that there is any greet effect of this 
kind. In any case there is ,no intention to i nvestigate here the theory of the game,

kind. In any case there is ,no intention to i nvestigate here the theory of the game, 
and it will be assumed that the best strategy is to try to provide answers that would 
naturally be given by a mans 
 
Once the test has been proposed and discussed, Turing remarks: 
 
The original question "Can machines think 1 believe to be too meaningless to deserve discussion. Nevertheless, I believe th at at the end of the century the use of 
words and general educated opinion will have altered so much that one will be able 
to speak of machines thinking without expecting to be contradicted.
6 
 
Turing Anticipates Objections

6 
 
Turing Anticipates Objections 
 
Aware of the storm of opposition that woul d undoubtedly greet this opinion, he then 
proceeds to pick apart, concisely and with wr y humor, a series of objections to the notion 
that machines could think. Below I list the nine  types of objections he counters, using his 
own descriptions of them .7 Unfortunately there is not space to reproduce the humorous 
and ingenious responses he formulate d. You may enjoy pondering the objections 
yourself, and figuring out your own responses.

yourself, and figuring out your own responses. 
 
(1) The Theological Objection. Thinking is a function of man's immortal soul. God 
has given an immortal soul to every ma n and woman, but not to any other animal 
or to machines. Hence no animal or machine can think. 
(2) The "Heads in the Sand" Objection. Th e consequences of machines thinking 
would be too dreadful. Let us hope and believe that they cannot do so. 
(3) The Mathematical Objection. [This is essentially the Lucas argument. (4) The Argument from Consciousness. "Not  until a machine can write a sonnet or

(3) The Mathematical Objection. [This is essentially the Lucas argument. (4) The Argument from Consciousness. "Not  until a machine can write a sonnet or 
compose a concerto because of thoughts a nd emotions felt, and not by the chance 
fall of symbols, could we agree that mach ine equals brainy that is, not only write 
it but know that it had written it. No mechanism

 594could feel (and not merely artificially signa l, an easy contrivance) pleasure at its 
successes, grief when its valves fuse, be  warmed by flattery, be made miserable 
by its mistakes, be charmed by sex, be angry or depressed when it cannot get what it wants." [A quote from a certain Professor Jefferson.] 
 
Turing is quite concerned that he should an swer this serious objection in full detail. 
Accordingly, he devotes quite a bit of space to  his answer, and in it he offers another 
short hypothetical dialogue:' 
 
Interrogator: In the first line of your s onnet which reads "Shall I compare thee to a 
summer's day", would not "a spring day" do as well or better, Witness: It 
wouldn't scan. 
Interrogator: How about ' a winter's day''? That would scan all right. Witness: Yes, 
but nobody wants to be compared to a wi nter's day. Interrogator: Would you say 
Mr. Pickwick reminded you of Christmas? Witness: In a way.

Mr. Pickwick reminded you of Christmas? Witness: In a way. 
Interrogator: Yet Christmas is a winter 's day, and I do not think Mr. Pickwick 
would mind the comparison. 
Witness: I don't think you're serious. By a wi nter's day one means a typical winter's 
day, rather than a special one like Christmas. 
 
After this dialogue, Turing asks, "What w ould Professor Jefferson say if the sonnet-
writing machine was able to answer like this in the viva voce ?"  
Further objections: 
 
(5)  Arguments from various Disabilities. These arguments take the form, "I grant you that

(5)  Arguments from various Disabilities. These arguments take the form, "I grant you that 
you can make machines do all the things that  you have mentioned but you will never be 
able to make one to do X." Numerous features X are suggested in this connection. I offer 
a selection: 
Be kind, resourceful, beautiful, friendly, have  initiative, have a sense of humor, tell right 
from wrong, make mistakes, fall in love, enjoy strawberries and cream, make someone fall in love with it, learn from experience, use words properly, be the subject of its own thought, have as much diversity of behavi our as a man, do something really new.

from wrong, make mistakes, fall in love, enjoy strawberries and cream, make someone fall in love with it, learn from experience, use words properly, be the subject of its own thought, have as much diversity of behavi our as a man, do something really new. 
(6) Lady Lovelace's Objection. Our most de tailed information of Babbage's Analytical 
Engine comes from a memoir by Lady Lovelace. In it she states, "The Analytical Engine has no pretensions to originate anything. It can do whatever we know how to order it to perform" (her italics). 
(7)  Argument from Continuity in the Nervous System. The nervous system is certainly not 
a discrete state machine. A small error in the information about the size of a nervous

a discrete state machine. A small error in the information about the size of a nervous 
impulse impinging on a neuron may make a large difference to the size of the outgoing impulse. It may be argued that, this being so, one cannot expect to be able to mimic the behaviour of the nervous system with a discrete state system. 
(8)  The Argument from Informality of Behavi our. It seems to run something like this. "If 
each man had a definite set of rules of conduc t by which he regulated his life he would 
be no better than a machine. But there are no such rules, so men cannot be machines."

be no better than a machine. But there are no such rules, so men cannot be machines." 
(9)  The Argument from Extra-Sensory Perception. Let us play the imitation game, using as 
witnesses a man who is good as a telepathic receiver, and a digital computer. The interrogator can ask such questions as "What suit

 595does the card in my right hand belong to?" The man by telepathy or clairvoyance gives 
the right answer 130 times out of 400 cards. The machine can only guess at random, and perhaps get 104 right, so the interrogator makes the right identification. 
 
As you can see, many of the objections overlap  and are related to each other. I have 
attempted in this book to answer most of th em in some way or other, usually far less 
concisely than Turing did. 
Objection (9) I find remarkable. I have seen the Turing paper reprinted in a book-
but with objection (9) omitted-which I find equally remarkable. My own point of view-contrary to Turing's-is that ESP does not exist. Turing was reluctant to accept 
the idea that ESP is real, but did so none theless, being compelled by his outstanding 
scientific integrity to accept the conseque nces of what he viewed as powerful 
statistical evidence in favor of ESP. I disagree, though I consider it an exceedingly

statistical evidence in favor of ESP. I disagree, though I consider it an exceedingly 
complex and fascinating question. 
 
"Parry Encounters the Doctor" 
 
In the Dialogue preceding this Chapter, you have seen an authentic exchange between a computer program and a human. Now th ere have been many programs with less 
ability to "converse" than SHRDLU. Two rath er famous ones are "Doctor", created by

ability to "converse" than SHRDLU. Two rath er famous ones are "Doctor", created by 
Joseph Weizenbaum, and "Parry", created by Kenneth Colby. The former is supposed to simulate a psychiatrist using "nondirective therapy", the latter to simulate the belief structure of a paranoid. An amusing head-on collision between the two programs was arranged over a computer network, and the fo llowing is a short excerpt from the full 
transcript
:9 
 
Parry: I don't understand your motives. 
Doctor: Why don't you understand my motives? Parry: Shouldn't I be? Doctor: Why do you say that just now?

:9 
 
Doctor: Why don't you understand my motives? Parry: Shouldn't I be? Doctor: Why do you say that just now? 
Parry: That's just the way it is. 
Doctor: Suppose that's just the way it were not. Parry: So? Doctor: I am not sure I understand you fully. Parry: You harp on the same questions too much.  Doctor: What does that similarity suggest to you? 
Parry: I don't get you. 
Doctor: Do you wish that you did get I? Parry: It's about time you showed some interest in my feelings.  Doctor: Oh? I showed some interest in your feelings? 
 
These programs, of course, were made to ta lk to humans, so they are shown up rather

These programs, of course, were made to ta lk to humans, so they are shown up rather 
poorly here, pitted against each other. Mostly they rely on shrewd guesses as to the nature of the input (which they analyze qu ite shallowly) and spit back canned answers 
that have been carefully selected from a large repertoire. The answer may be only 
partially canned: for example, a template with blanks that can be filled in. It is assumed that their

 596human partners will read much more into wh at they say than is actually underlying it. 
And in fact, according to Weizenbaum, in his book Computer Power and Human 
Reason, just that happens. He writes: 
 ELIZA [the program from which Doctor was made created the most remarkable 
illusion of having understood in the minds of the many people who conversed with 
it.... They would often demand to be perm itted to converse with the system in 
private, and would, after conversing with it for a time, insist, in spite of my 
explanations, that the machine really understood them.
10 
 
Given the above excerpt, you may find this  incredible. Incredible, but true. 
Weizenbaum has an explanation:  
Most men don't understand computers to even  the slightest degree. So, unless they 
are capable of very great skepticism (the kind we bring to bear while watching a stage magician), they can expl ain the computer's intellectual feats only

10 
 
are capable of very great skepticism (the kind we bring to bear while watching a stage magician), they can expl ain the computer's intellectual feats only 
by bringing to hear the single analogy available to them, that is, their model of their own capacity to think. No  wonder, then, that they overshoot the 
mark: it is truly impossible to imagine a human who could imitate ELIZA, for example, but for whom ELIZA's language abilities were his limit."  
Which amounts to an admission that this kind of program is based on a shrewd 
mixture of bravado and bluffing, taking advantage of people's gullibility.

mixture of bravado and bluffing, taking advantage of people's gullibility. 
In light of this weird "ELIZA-effect", so me people have suggested that the Turing 
test needs revision, since people can apparen tly be fooled by simplistic gimmickry. It 
has been suggested that the interrogator should be a Nobel Prize-winning scientist. It 
might be more advisable to turn the Turing test on its head, and insist that the

might be more advisable to turn the Turing test on its head, and insist that the 
interrogator should be another computer. Or perhaps there should be two interrogators-a human and a computer-and one witness, and the two interrogators should try to figure out whether the witness is a human or a computer. In a more serious vein, I personally feel that  the Turing test, as originally proposed, is 
quite reasonable. As for the people who Weizenbaum claims were sucked in by ELIZA, they were not urged to be skeptical , or to use all their wits in trying to

quite reasonable. As for the people who Weizenbaum claims were sucked in by ELIZA, they were not urged to be skeptical , or to use all their wits in trying to 
determine if the "person" typing to them were human or not. I think that Turing's 
insight into this issue was sound, and that the Turing test, essentially unmodified, will 
survive.  
A Brief History of AI 
 
I would like in the next few pages to pres ent the story, perhaps from an unorthodox point 
of view, of some of the efforts at unrav eling the algorithms behind intelligence: there 
have been failures and setbacks and there w ill continue to be. Nonetheless, we are 
learning a great deal, and it is an exciting period.

survive.  
 
learning a great deal, and it is an exciting period. 
Ever since Pascal and Leibniz, people have  dreamt of machines that could perform 
intellectual tasks. In the nineteenth centu ry, Boole and De Morgan devised "laws of 
thought"-essentially the Propositional

 597 
Calculus-and thus took the first step towards At software; also Charles Babbage designed 
the first "calculating engine"-the precursor to the hardware of computers and hence of AI. 
One could define AI as coming into existen ce at the moment when mechanical devices 
took over any tasks previously performable onl y by human minds. It is hard to look back 
and imagine the feelings of those who first saw toothed wheels performing additions and multiplications of large numbers. Perhaps they experienced a sense of awe at seeing "thoughts" flow in their very physical hardwa re. In any case, we do know that nearly a 
century later, when the first electronic com puters were constructed, their inventors did 
experience an awesome and mystical sense of  being in the presence of another kind of 
"thinking being". To what extent real t hought was taking place was a source of much 
puzzlement; and even now, several decades late r, the question remains a great source of

puzzlement; and even now, several decades late r, the question remains a great source of 
stimulation and vitriolics. 
It is interesting that nowadays, practically  no one feels that sense of awe any longer-
even when computers perform operations that  are incredibly more sophisticated than 
those which sent thrills down spines in th e early days. The once-exciting phrase "Giant 
Electronic Brain" remains only as a sort of " camp" cliché, a ridiculous vestige of the era 
of Flash Gordon and Buck Rogers. It is a bit sad that we become blasé so quickly. 
There is a related "Theorem" about progre ss in Al: once some mental function is

There is a related "Theorem" about progre ss in Al: once some mental function is 
programmed, people soon cease to consider it as an essential ingredient of "real thinking". The ineluctable core of intelligence is always in that next thing which hasn't yet been programmed. This "Theorem" was firs t proposed to me by Larry Tesler, so I call 
it Tesler's Theorem. "Al is whatever hasn't been done vet." 
A selective overview of AI is furnished be low. It shows several domains in which 
workers have concentrated their efforts, each  one seeming in its own way to require the 
quintessence of intelligence. With some of  the domains I have included a breakdown

quintessence of intelligence. With some of  the domains I have included a breakdown 
according to methods employed, or more specific areas of concentration. 
 
mechanical translation 
direct (dictionary look-up with some word rearrangement)  indirect (via some intermediary internal language) 
 
game playing 
 
chess 
with brute force look-ahead 
with heuristically pruned look-ahead with no look-ahead checkers 
go kalah bridge (bidding; playing)  poker variations on tic-tac-toe etc.

 598proving theorems in various parts. of mathematics  
symbolic logic 
"resolution" theorem-proving elementary geometry  
symbolic manipulation of mathematical expressions  
symbolic integration algebraic simplification  summation of infinite series  
vision 
printed matter: 
recognition of individual hand-printed characters drawn 
from a small class (e.g., numerals) 
reading text in variable fonts reading passages in handwriting reading Chinese or Japanese printed characters  reading Chinese or Japanese handwritten characters 
pictorial: 
locating prespecified objects in photographs  decomposition of a scene into separate objects  identification of separate objects in a scene  recognition of objects portrayed in sketches by people  recognition of human faces 
hearing 
understanding spoken words drawn from a limited vocabulary (e.g., names of 
the ten digits) 
understanding continuous speech in fixe d domains finding boundaries between 
phonemes

hearing 
phonemes  
identifying phonemes finding boundaries between morphemes  identifying morphemes putting together whole words and sentences  
understanding natural languages 
answering questions in specific domains  parsing complex sentences making paraphrases of longer pieces of text using knowledge of the real world in order to understand passages resolving ambiguous references 
 producing natural language 
abstract poetry (e.g., haiku) random sentences, paragraphs, or longer pieces of text producing output from 
internal representation of knowledge

 599 
creating original thoughts or works of art  
poetry writing (haiku) story writing computer art  musical composition 
atonal tonal  
analogical thinking 
geometrical shapes ("intelligence tests") constructing proofs in one domain of mathematics based on 
those in a related domain 
 learning 
adjustment of parameters  concept formation 
 
Mechanical Translation 
 
Many of the preceding topics will not be touched upon in my selective discussion 
below, but the list would not be accurate without them. The first few topics are 
listed in historical order. In each of them , early efforts fell short of expectations. 
For example, the pitfalls in mechanical tr anslation came as a great surprise to many 
who had thought it was a nearly straightforw ard task, whose perfection, to be sure, 
would be arduous, but whose basic implem entation should be easy. As it turns out, 
translation is far more complex than mere dictionary look-up and word rearranging.

translation is far more complex than mere dictionary look-up and word rearranging. 
Nor is the difficulty caused by a lack of know ledge of idiomatic phrases. The fact is 
that translation involves having a mental model of the world being discussed, and manipulating symbols in that model. A pr ogram which makes no use of a model of 
the world as it reads the passage will soon get hopelessly bogged down in 
ambiguities and multiple meanings. Even people-who have a huge advantage over computers, for they come fully equipped with an understanding of the world-when

ambiguities and multiple meanings. Even people-who have a huge advantage over computers, for they come fully equipped with an understanding of the world-when 
given a piece of text and a di ctionary of a language they do not know, find it next to 
impossible to translate the text into  their own language. Thus-and it is not 
surprising in retrospect-the first problem of  AI led immediately to the issues at the 
heart of AI.  
Computer Chess 
 
Computer chess, too, proved to be much  more difficult than the early intuitive

Computer chess, too, proved to be much  more difficult than the early intuitive 
estimates had suggested. Here again it turns out that the way humans represent a chess situation in their minds is far more complex than just knowing which piece is on which square, coupled with knowledge of the rules of chess. It involves 
perceiving configurations of several related pieces, as well as knowledge of 
heuristics, or rules of thumb, which pertain to

 600such higher-level chunks. Even though heuris tic rules are not rigorous in the way 
that the official rules are, they provide shortcut insights into what is going on on the 
board, which knowledge of the official ru les does not. This much was recognized 
from the start; it was simply underestimated how large a role the intuitive, chunked understanding of the chess world plays in human chess skill. It was predicted that a program having some basic heuristics, coupled with the blinding speed and 
accuracy of a computer to look ahead in the game and analyze each possible move, 
would easily beat top-flight human player s-a prediction which, even after twenty-
five years of intense work by various people, still is far from being realized. 
People are nowadays tackling the chess pr oblem from various angles. One of

People are nowadays tackling the chess pr oblem from various angles. One of 
the most novel involves the hypothesis that looking ahead is a silly thing to do. One should instead merely look at what is on the board at present, and, using some heuristics, generate a plan, and then find a move which advances that particular 
plan. Of course, rules for the formulati on of chess plans will necessarily involve 
heuristics which are, in some sense, "flattened" versions of looking ahead. That is, the equivalent of many games' experience of looking ahead is "squeezed" into another form which ostensibly doesn't invol ve looking ahead. In some sense this is 
a game of words. But if the "flattened"  knowledge gives answers more efficiently

a game of words. But if the "flattened"  knowledge gives answers more efficiently 
than the actual look-ahead-even if it occas ionally misleads- then something has 
been gained. Now this kind of distillati on of knowledge into more highly usable 
forms is just what intelligence excels at -so look-ahead-less chess is probably a 
fruitful line of research to push. Partic ularly intriguing would be to devise a 
program which itself could convert knowle dge gained from looking ahead into 
"flattened" rules-but that is an immense task.  
Samuel's Checker Program 
 As a matter of fact, such a method wa s developed by Arthur Samuel in his

 As a matter of fact, such a method wa s developed by Arthur Samuel in his 
admirable checker-playing program. Samuel's trick was to use both dynamic (look-ahead) and static (no-look-ahead) ways of  evaluating any given board position. The 
static method involved a simple mathem atical function of several quantities 
characterizing any board position, and thus  could be calculated practically 
instantaneously, whereas the dynamic eval uation method involved creating a "tree"

instantaneously, whereas the dynamic eval uation method involved creating a "tree" 
of possible future moves, responses to them, responses to the responses, and so forth (as was shown in Fig. 38). In the st atic evaluation function there were some 
parameters which could vary; the effect of  varying them was to provide a set of 
different possible versions of the static  evaluation function. Samuel's strategy was 
to select, in an evolutionary way, better and better values of those parameters. 
Here's how this was done: each time th e program evaluated a board position, it

Here's how this was done: each time th e program evaluated a board position, it 
did so both statically and dynamically. Th e answer gotten by looking ahead-let us 
call it D-was used in determining the move to be made. The purpose of S, the static 
evaluation, was trickier: on each move, the variable parameters were readjusted slightly so that S approximated D

 601as accurately as possible. The effect was to  partially encode in the values of the 
static evaluation’s parameters the knowledge gained by dynamically searching the tree. In short, the idea was to "flatten"  the complex dynamic evaluation method into 
the much simpler and more efficient static evaluation function. 
There is a rather nice recursive effect here. The point is that the dynamic 
evaluation of any single board position i nvolves looking ahead a finite number of 
moves-say seven. Now each of the scads of board positions which might turn up 
seven turns down the road has to be itself evaluated somehow as well. But when the program evaluates these positions, it certa inly cannot look another seven moves 
ahead, lest it have to look fourteen positi ons ahead, then twenty-one, etc., etc.-an 
infinite regress. Instead, it relies on sta tic evaluations of positions seven moves 
ahead. Therefore, in Samuel's scheme, an intricate sort of feedback takes place,

ahead. Therefore, in Samuel's scheme, an intricate sort of feedback takes place, 
wherein the program is constantly trying to "flatten" look-ahead evaluation into a 
simpler static recipe; and this recipe in  turn plays a key role in the dynamic look-
ahead evaluation. Thus the two are intimat ely linked together, and each benefits 
from improvements in the other in a recursive way. 
The level of play of the Samuel check ers program is extremely high: of the 
order of the top human players in the worl d. If this is so, why not apply the same

order of the top human players in the worl d. If this is so, why not apply the same 
techniques to chess? An international committee, convened in 1961 to study the feasibility of computer chess, including the Dutch International Grandmaster and mathematician Max Euwe, came to the bleak conclusion that the Samuel technique would be approximately one million times as difficult to implement in chess as in checkers, and that seems to close the book on that. 
The extraordinarily great skill of the checkers program cannot be taken as 
saying "intelligence has been achieved"; yet it should not be minimized, either. It is

saying "intelligence has been achieved"; yet it should not be minimized, either. It is 
a combination of insights into what check ers is, how to think about checkers, and 
how to program. Some people might feel that  all it shows is Samuel's own checkers 
ability. But this is not true, for at least tw o reasons. One is that skillful game players 
choose their moves according to mental processes which they do not fully understand-they use their intuitions. Now there is no known way that anyone can 
bring to light all of his own intuitions; the best one can do via introspection is to use "feeling" or "meta-intuition"-an intu ition about one's intuitions-as a guide, and 
try to describe what one thinks one's intu itions are all about. But this will only give 
a rough approximation to the true comple xity of intuitive methods. Hence it is 
virtually certain that Samuel has not mirro red his own personal methods of play in

virtually certain that Samuel has not mirro red his own personal methods of play in 
his program. The other reason that Samuel's  program's play should not be confused 
with Samuel's own play is that Samuel  does not play checkers as well as his 
program-it beats him. This is not a paradox at all-no more than is the fact that a 
computer which has been programmed to calculate 7T can outrace its programmer in spewing forth digits of π.

 602When Is a Program Original? 
 
This issue of a program outdoing its progra mmer is connected with the question of 
"originality" in AI. What if an AI program co mes up with an idea, or a line of play in a 
game, which its programmer has never entertained-who should get the credit? There are various interesting instances of this having ha ppened, some on a fairly trivial level, some 
on a rather deep level. One of the more famous involved a progr am to find proofs of 
theorems in elementary Euclidean geomet ry, written by E. Gelernter. One day the 
program came up with a sparklingly ingenious proof of one of the basic theorems of 
geometry-the so-called "pons asinorum", or "bridge of asses". 
This theorem states that the base angles  of an isosceles triangle are equal. Its 
standard proof requires constructing an altitude which divides the triangle into 
symmetrical halves. The elegant method found by the program (see Fig. 114) used no

symmetrical halves. The elegant method found by the program (see Fig. 114) used no 
construction lines. Instead, it considered 
  
 FIGURE 114. Pons Asinorum Proof (found 
by Pappus [-300 A.D. ] and Gelernter's 
program [--1960 A.D. ]). Problem: To show 
that the base angles of an isosceles triangle 
are equal. Solution: As the triangle is isosceles, AP and AP' are of equal length. Therefore triangles PAP' and PAP are congruent (side-side-side). This implies that corresponding angles are equal. In 
particular, the two base angles are equal.

particular, the two base angles are equal. 
 
the triangle and its mirror image as two different triangles. Then, having proved 
them congruent, it pointed out that the two base angles matched each other in this 
congruence-QED. 
This gem of a proof delighted the program's creator and others; some saw evidence 
of genius in its performance. Not to take a nything away from this feat, it happens that in 
A.D. 300 the geometer Pappus had actually found this proof, too. In any case, the 
question remains: "Who gets the credit?" Is this intelligent behavior? Or was the proof

question remains: "Who gets the credit?" Is this intelligent behavior? Or was the proof 
lying deeply hidden within the human (Gelernt er), and did the computer merely bring it 
to the surface? This last question comes clos e to hitting the mark. We can turn it around: 
Was the proof lying deeply hidden in the progr am? Or was it close to the surface? That is, 
how easy is it to see why the pr ogram did what it did? Can the discovery be attributed to 
some simple mechanism, or simple combination of mechanisms, in the program? Or was there a complex interaction which, if one h eard it explained, would not diminish one's 
awe at its having happened?

awe at its having happened? 
It seems reasonable to say that if one can ascribe the performance to certain 
operations which are easily traced in the progr am, then in some sense the program was 
just revealing ideas which were in e ssence hiddenthough not too deeply-inside the 
programmer's own mind. Conversely, if

 603 
following the program does not serve to enlighten  one as to why this particular discovery 
popped out, then perhaps one should begin to sepa rate the program's "mind" from that of 
its programmer. The human gets credit for having invented the program, but not for 
having had inside his own head the ideas produced by the program. In such cases, the 
human can be referred to as the "meta-author"- the author of the author of the result, and 
the program as the (just plain) author. 
In the particular case of Gelernter and his geometry machine, while Gelernter 
probably would not have rediscovered Pappus'. proof, still the mechanisms which generated that proof were sufficiently clos e to the surface of the program that one 
hesitates to call the program a geometer in its own right. If it had kept on astonishing 
people by coming up with ingenious new pr oofs over and over again, each of which

people by coming up with ingenious new pr oofs over and over again, each of which 
seemed to be based on a fresh spark of genius rather than on some standard method, then one would have no qualms about calling the program a geometer-but this did not happen. 
 
Who Composes Computer Music? 
 
The distinction between author and meta-author is sharply pointed up in the case of computer composition of music. There are va rious levels of autonomy which a program 
may seem to have in the act of compos ition. One level is illustrated by a piece whose 
"meta-author" was Max Mathews of Bell Laborat ories. He fed in the scores of the two

"meta-author" was Max Mathews of Bell Laborat ories. He fed in the scores of the two 
marches "When Johnny Comes Marching Home" and "The British Grenadiers", and instructed the computer to make a new scor e-one which starts out as "Johnny", but slowly 
merges into "Grenadiers". Halfway through the piece, "Johnny" is totally gone, and one 
hears "Grenadiers" by itself ... Then the proce ss is reversed, and the piece finishes with 
"Johnny", as it began. In Mathews' own words, this is 
 
... a nauseating musical experience but one not without interest, particularly in 
the rhythmic conversions. "The Grenadiers " is written in 2/4 time in the key of F

the rhythmic conversions. "The Grenadiers " is written in 2/4 time in the key of F 
major. "Johnny" is written in 6/8 time in th e key of E minor. The change from 2/4 
to 6/8 time can be clearly appreciated, yet would be quite difficult for a human 
musician to play. The modulation from the key of F major to E minor, which involves a change of two notes in the s cale, is jarring, and a smaller transition 
would undoubtedly have been a better choice ." 
 
The resulting piece has a somewhat droll quality  to it, though in spots it is turgid and 
confused. 
 Is the computer composing? The question is best unasked, but it cannot be

confused. 
 Is the computer composing? The question is best unasked, but it cannot be 
completely ignored. An answer is di fficult to provide. The algorithms are 
deterministic, simple, and understandable. No complicated or hard-to understand computations are involved; no "learning"  programs are used; no random processes 
occur; the machine functions in a perfectly  mechanical and straightforward manner. 
However, the result is sequences of sound that are unplanned in fine detail by the composer, even though the over-all structure

 604of the section is completely and precisely  specified. Thus the composer is often 
surprised, and pleasantly surprised, ar the deta ils of the realization of his ideas. To 
this extent only is the computer co mposing. We call the process algorithmic 
composition, but we immediately re-e mphasize that the algorithms are 
transparently simple." 
 This is Mathews' answer to a question wh ich he would rather "unask". Despite his 
disclaimer, however, many people find it easie r to say simply that the piece was 
"composed by a computer". I believe this phr ase misrepresents the situation totally. The 
program contained no structures analogous to th e brain's "symbols", and could not be said 
in any sense to be "thinking" about what it was doing. To attribute the composition of 
such a piece of music to the computer would be like attributing the authorship of this

such a piece of music to the computer would be like attributing the authorship of this 
book to the computerized automatically (often incorrectly) hyphenating phototypesetting machine with which it was set. 
This brings up a question which is a sli ght digression from Al, but actually not a 
huge one. It is this: When you see the word "I" or "me" in a text, what do you take it to be referring to? For instance, think of the phr ase "WASH ME" which appears occasionally 
on the back of dirty trucks. Who is this "me"? Is this an outcry of some forlorn child who, 
in desperation to have a bath, scribbled the words on the nearest surface? Or is the truck

in desperation to have a bath, scribbled the words on the nearest surface? Or is the truck 
requesting a wash? Or, perhaps, does the senten ce itself wish to be given a shower? Or, is 
it that the filthy English language is asking to be cleansed? One could go on and on in 
this game. In this case, the phrase is a j oke, and one is supposed to pretend, on some 
level, that the truck itself wrote the phrase a nd is requesting a wash. On another level, one 
clearly recognizes the writing as that of a ch ild, and enjoys the humor of the misdirection. 
Here, in fact, is a game based on reading the "me" at the wrong level.

Here, in fact, is a game based on reading the "me" at the wrong level. 
Precisely this kind of ambiguity has arisen in this book, first in the 
Contracrostipunctus, and later in the discussi ons of Gödel’s string G (and its relatives). 
The interpretation given for unplayable reco rds was "I Cannot Be Played on Record 
Player X", and that for unprovable statemen ts was, "I Cannot Be Proven in Formal 
System X". Let us take the latter sentence.  On what other occasions, if any, have you 
encountered a sentence containing the pr onoun "I" where you automatically understood

encountered a sentence containing the pr onoun "I" where you automatically understood 
that the reference was not to the speaker of th e sentence, but rather to the sentence itself? 
Very few, I would guess. The word "I", when it appears in a Shakespeare sonnet, is referring not to a fourteen-line form of poetr y printed on a page, but to a flesh-and-blood 
creature behind the scenes, somewhere off stage. 
How far back do we ordinarily trace the "I " in a sentence? The answer, it seems to 
me, is that we look for a sentient being to att ach the authorship to. But what is a sentient

me, is that we look for a sentient being to att ach the authorship to. But what is a sentient 
being? Something onto which we can map ourselves comfortably. In Weizenbaum's 
"Doctor" program, is there a personality? If so, whose is it? A small debate over this very 
question recently raged in the pages of Science magazine. 
This brings us back to the issue of the "who" who composes computer music. In 
most circumstances, the driving force behind such pieces is a

 605human intellect, and the computer has been employed, with more or less ingenuity, as a 
tool for realizing an idea devised by the human. The program which carries this out is not anything which we can identify with. It is a simple and single-minded piece of software 
with no flexibility, no perspective on what it is  doing, and no sense of self. If and when, 
however, people develop programs which have t hose attributes, and pieces of music start 
issuing forth from them, then I suggest that will be the appropriate, time to start splitting up one's admiration: some to the programme r for creating such an amazing program, and 
some to the program itself for its sense of mu sic. And it seems to me that that will only 
take place when the internal structure of such  a program is based on something similar to 
the "symbols" in our brains and their trigge ring patterns, which are responsible for the

the "symbols" in our brains and their trigge ring patterns, which are responsible for the 
complex notion of meaning. The fact of havi ng this kind of internal structure would 
endow the program with properties which woul d make us feel comfortable in identifying 
with it, to some extent. But until then, I w ill not feel comfortable in saying "this piece 
was composed by a computer".  
Theorem Proving and Problem Reduction 
 
Let us now return to the history of AI. One of the early things which people attempted to 
program was the intellectual activity of theorem proving. Conceptually, this is no

program was the intellectual activity of theorem proving. Conceptually, this is no 
different from programming a computer to look for a derivation of MU in the MIU-
system, except that the formal systems invol ved were often more complicated than the 
MIU-system. They were versions of the Predi cate Calculus, which is an extension of the 
Propositional Calculus involving quantifiers. Most  of the rules of the Predicate Calculus 
are included in TNT, as a matter of fact. The trick in writing such a program is to instill a

are included in TNT, as a matter of fact. The trick in writing such a program is to instill a 
sense of direction, so that the program does not wander all over the map, but works only on "relevant" pathways-those which, by some reasonable criterion, seem to be leading 
towards the desired string. 
In this book we have not dealt much w ith such issues. How indeed can you know 
when you are proceeding towards a theorem,  and how can you tell if what you are doing 
is just empty fiddling? This was one thing which I hoped to illustrate with the MU-puzzle. Of course, there can be no definitive answer: that is the content of the limitative

is just empty fiddling? This was one thing which I hoped to illustrate with the MU-puzzle. Of course, there can be no definitive answer: that is the content of the limitative 
Theorems, since if you could always know which way to go, you could construct an 
algorithm for proving any desired theorem, a nd that would violate Church's Theorem. 
There is no such algorithm. (I will leave it to the reader to see exactly why this follows 
from Church's Theorem.) However, this doesn't  mean that it is impossible to develop any 
intuition at all concerning what is and what is  not a promising route; in fact, the best 
programs have very sophisticated heuristics, which enable them to make deductions in 
the Predicate Calculus at speeds which are comparable to those of capable humans.

the Predicate Calculus at speeds which are comparable to those of capable humans. 
The trick in theorem proving is to use the fact that you have an overall goal-namely 
the string you want to produce-in guidi ng you locally. One technique which was 
developed for converting global goals

 606into local strategies for derivations is calle d problem reduction. It is based on the idea that 
whenever one has a long-range goal, there are usually subgoals whose attainment will aid in the attainment of the main goal. Theref ore if one breaks up a given problem into a 
series of new subproblems, then breaks those in turn into subsubproblems, and so on, in a 
recursive fashion, one eventually comes down to very modest goals which can 
presumably be attained in a couple of steps. Or at least so it would seem ... 
Problem reduction got Zeno into hot water. Zeno's method, you recall, for getting 
from A to B (think of B as the goal), is to "reduce" the problem into two subproblems: first go halfway, then go the rest of the wa y. So now you have "pushed"-in the sense of 
Chapter V-two subgoals onto your "goal stack". Each of these, in turn, will be replaced 
by two subsubgoals and so on ad infinitum. You wind up with an infinite goal-stack,

by two subsubgoals and so on ad infinitum. You wind up with an infinite goal-stack, 
instead of a single goal (Fig. 115). Popping an infinite number of goals off your stack will 
prove to be tricky-which is just Zeno's point, of course. 
Another example of an infinite recurs ion in problem reduction occurred in the 
Dialogue Little Harmonic Labyrinth, when Achilles wanted to have a Typeless Wish granted. Its granting had to be deferred until permission was gotten from the Meta-Genie; but in order to get permission to give permission, she had to summon the Meta-Meta-
Genie-and so on. Despite  
FIGURE 115. Zeno's endless goal tree, for getting from A to B.

 607the infiniteness of the goal stack, Achilles got his wish. Problem reduction wins the day! 
Despite my mockery, problem reduction is  a powerful technique for converting 
global problems into local problems. It shines in certain situations, such as in the endgame of chess, where the look-ahead tec hnique often performs miserably, even when 
it is carried to ridiculous lengths, such as fi fteen or more plies. This is because the look-
ahead technique is not based on planning; it simply has no goals and explores a huge 
number of pointless alternatives. Having a goa l enables you to develop a strategy for the 
achievement of that goal, and this is a completely different philosophy from looking 
ahead mechanically. Of course, in the look-ahead technique, desirability or its absence is measured by the evaluation function for positi ons, and that incorporates indirectly a 
number of goals, principally that of not getti ng checkmated. But that is too indirect. Good

number of goals, principally that of not getti ng checkmated. But that is too indirect. Good 
chess players who play against look-ahead ch ess programs usually come away with the 
impression that their opponents are very weak in formulating plans or strategies. 
 
Shandy and the Bone 
 
There is no guarantee that the method of  problem reduction will work. There are 
many situations where it flops. Consider this  simple problem, for instance. You are a dog, 
and a human friend has just thrown your fa vorite bone over a wire fence into another 
yard. You can see your bone through the fence, ju st lying there in the grass-how luscious!

yard. You can see your bone through the fence, ju st lying there in the grass-how luscious! 
There is an open gate in the fence about fi fty feet away from the bone. What do you do? 
Some dogs will just run up to the_ fence, stand next to it, and bark; others will dash up to 
the open gate and double back to the lovely bone. Both dogs can be said to be exercising the problem reduction technique; however, they represent the problem in their minds in 
different ways, and this makes all the diffe rence. The barking dog sees the subproblems 
as (1) running to the fence, (2) getting thr ough it, and (3) running to the bone-but that

as (1) running to the fence, (2) getting thr ough it, and (3) running to the bone-but that 
second subproblem is a "toughie", whence the barking. The other dog sees the 
subproblems as (1) getting to the gate; (2 ) going through the gate; (3) running to the 
bone. Notice how everything depends on the wa y you represent the "problem space"-that 
is, on what you perceive as reducing the pr oblem (forward motion towards the overall 
goal) and what you perceive as magnifying th e problem (backward motion away from the 
goal). 
 
Changing the Problem Space 
 
Some dogs first try running directly toward s the bone, and when they encounter the

goal). 
 
Some dogs first try running directly toward s the bone, and when they encounter the 
fence, something clicks inside their brain; soon they change course, and run over to the gate. These dogs realize that what on first

 608glance seemed as if it would increase the di stance between the initial situation and the 
desired situation-namely, running away from the bone but towards the open gate-actually 
would decrease it. At first, they confuse physical distance with problem distance. Any 
motion away from the bone seems, by defi nition, a Bad Thing. But then-somehow-they 
realize that they can shift their perception of what will bring them "closer" to the bone. In 
a properly chosen abstract space, moving towa rds the gate is a trajectory bringing the dog 
closer to the bone! At every moment, the dog is getting "closer"-in the new sense-to the 
bone. Thus, the usefulness of problem re duction depends on how you represent your 
problem mentally. What in one space looks lik e a retreat can in another space look like a 
revolutionary step forward. 
In ordinary life, we constantly face and solve variations on the dog and-bone

In ordinary life, we constantly face and solve variations on the dog and-bone 
problem. For instance, if one afternoon I deci de to drive one hundred miles south, but am 
at my office and have ridden my  bike to work, I have to make an extremely large number 
of moves in what are ostensibly "wrong" directions before I am actually on my way in car headed south. I have to leave my office, wh ich means, say, heading east a few feet; then 
follow the hall in the building which heads nort h, then west. Then I ride my bike home,

follow the hall in the building which heads nort h, then west. Then I ride my bike home, 
which involves excursions in all the directions of the compass; and I reach my home. A succession of short moves there eventually gets me into my car, and I am off. Not that I 
immediately drive due south. of course-I choose a route which may involve some 
excursions north. west, or east, with the ai m of getting to the freeway as quickly as 
possible. 
All of this doesn't feel paradoxical in the slightest; it is done without even any 
sense of amusement. The space in which physi cal backtracking is perceived as direct

possible. 
sense of amusement. The space in which physi cal backtracking is perceived as direct 
motion towards the goal is built so deeply in to my mind that I don't even see any irony 
when I head north. The roads and hallways a nd so forth act as channels which I accept 
without much fight, so that part of the act  of choosing how to perceive the situation 
involves just accepting what is imposed. But dogs  in front of fences sometimes have a 
hard time doing that, especially when that bone is sitting there so close, staring them in 
the face, and looking so good. And when the probl em space is just a shade more abstract 
than physical space, people are often just as lacking in insight about what to do as the 
barking dogs.

barking dogs. 
In some sense all problems are abstract versions of the dog-and-bone problem. 
Many problems are not in physical space but in some sort of conceptual space. When you realize that direct motion towards the goal in that space runs you into some sort of 
abstract "fence", you can do one of two thi ngs: (1) try moving away from the goal in 
some sort of random way, hoping that you may come upon a hidden "gate" through which you can pass and then reach your bone; or  (2) try to find a new "space" in which

some sort of random way, hoping that you may come upon a hidden "gate" through which you can pass and then reach your bone; or  (2) try to find a new "space" in which 
you can represent the problem, and in which there is no abstract fence separating you 
from your goal-then you can proceed straight to wards the goal in this new space. The first 
method may seem like the lazy way to go, and the second method may seem like a 
difficult and complicated way to go. And yet, solutions which involve restructuring the 
problem space more often

 609than not come as sudden flashes of insight ra ther than as products of a series of slow, 
deliberate thought processes. Probably these in tuitive flashes come from the extreme core 
of intelligence-and, needless to say, their sour ce is a closely protected secret of our 
jealous brains. 
In any case, the trouble is not that problem reduction per se leads to failures; it is 
quite a sound technique. The problem is a deeper one: how do you choose a good internal 
representation for a problem? What kind of  "space" do you see it in? What kinds of 
action reduce the "distance" between you a nd your goal in the space you have chosen? 
This can be expressed in mathematical language as the problem of hunting for an approprate metric (distance function) between states. You want to find a metric in which 
the distance between you and your goal is very small. 
Now since this matter of choosing an inte rnal representation is itself a type of

Now since this matter of choosing an inte rnal representation is itself a type of 
problem-and a most tricky one, too-you might th ink of turning the technique of problem 
reduction back on it! To do so, you would have  to have a way of representing a huge 
variety of abstract spaces, which is an ex ceedingly complex project. I am not aware of 
anyone's having tried anything along these lines. It may be just a theoretically appealing, 
amusing suggestion which is in fact wholly unr ealistic. In any case, what Al sorely lacks 
is programs which can "step back" and take a look at what is going on, and with this

is programs which can "step back" and take a look at what is going on, and with this 
perspective, reorient themselves to the task at hand. It is one thing to write a program which excels at a single task which, when done by a human being, seems to require intelligence-and it is another thing altogether to write an intelligent program! It is the 
difference between the Sphex wasp (see Chapte r XI), whose wired-in routine gives the 
deceptive appearance of great intelligence, and a human being observing a Sphex wasp. 
 
The I-Mode and the M-Mode Again 
 
An intelligent program would presumably be  one which is versatile enough to solve

An intelligent program would presumably be  one which is versatile enough to solve 
problems of many different sorts. It woul d learn to do each diffe rent one and would 
accumulate experience in doing so. It would be ab le to work within a set of rules and yet 
also, at appropriate moments, to step back  and make a judgment about whether working 
within that set of rules is likely to be pr ofitable in terms of so me overall set of goals 
which it has. It would be able to choose to  stop working within a given framework, if 
need be, and to create a new framework of rules within which to work for a while. 
Much of this discussion may remind you of aspects of the MU-puzzle. For

Much of this discussion may remind you of aspects of the MU-puzzle. For 
instance, moving away from the goal of a problem is reminiscent of moving away from MU by making longer and longer strings which you hope may in some indirect way 
enable you to make MU. If you are a naive "dog", you may feel you are moving away 
from your " MU-bone" whenever your string increases  beyond two characters; if you are a 
more sophisticated dog, the use of such lengt hening rules has an indirect justification, 
something like heading for the gate to get your MU-bone.

 610Another connection between the previous discussion and the MU puzzle is the 
two modes of operation which led to insight about the nature of the MU-puzzle: the 
Mechanical mode, and the Intelligent mode . In the former, you are embedded within 
some fixed framework; in the latter, you can always step back and gain an overview of 
things. Having an overview is tantamount to choosing a representation within which to 
work; and working within the rules of the syst em is tantamount to trying the technique of 
problem reduction within that selected framework. Hardy's comment on Ramanujan's 
style-particularly his willingness to modify his own hypotheses-illustrates this interplay 
between the M-mode and the I-mode in creative thought. 
The Sphex wasp operates excellently in the M-mode, but it has absolutely no 
ability to choose its framewor k or even to alter its M-mode in the slightest. It has no

ability to choose its framewor k or even to alter its M-mode in the slightest. It has no 
ability to notice when the same thing occurs  over and over and over again in its system, 
for to notice such a thing would be to jump  out of the system, even if only ever so 
slightly. It simply does not notice the same ness of the repetitions. This idea (of not 
noticing the identity of certain repetitive even ts) is interesting when we apply it to 
ourselves. Are there highly repetitious situati ons which occur in our lives time and time

ourselves. Are there highly repetitious situati ons which occur in our lives time and time 
again, and which we handle in the identical stupid way each time, because we don't have 
enough of an overview to perceive their same ness? This leads back to that recurrent 
issue, "What is sameness?" It will soon come up as an Al theme, when we discuss pattern recognition.  
Applying Al to Mathematics 
 Mathematics is in some ways an extremely interesting domain to study from the Al point 
of view. Every mathematician has the sense that  there is a kind of metric between ideas in 
mathematics-that all of mathematics is a network of results between which there are

mathematics-that all of mathematics is a network of results between which there are 
enormously many connections. In that netw ork, some ideas are very closely linked; 
others require more elaborate pathways to  be joined. Sometimes two theorems in 
mathematics are close because one can be proven easily, given the other. Other times two ideas are close because they are analogous, or  even isomorphic. These are two different 
senses of the word "close" in the domain of  mathematics. There are probably a number of 
others. Whether there is an objectivity or a universality to our sense of mathematical

others. Whether there is an objectivity or a universality to our sense of mathematical 
closeness, or whether it is largely an accident  of historical development is hard to say. 
Some theorems of different branches of math ematics appear to us hard to link, and we 
might say that they are unrelated-but someth ing might turn up later which forces us to 
change our minds. If we could instill our highly developed sense of mathematical 
closeness-a "mathematician's mental metric ", so to speak-into a program, we could 
perhaps produce a primitive "artificial mathema tician". But that depends on being able to

perhaps produce a primitive "artificial mathema tician". But that depends on being able to 
convey a sense of simplicity or "naturalness" as well, which is another major stumbling 
block. 
These issues have been confronted in a number of AI projects. There

 611is a collection of programs developed at MIT  which go under the name « MACSYMA ", 
whose purpose it is to aid mathematicians in symbolic manipulation of complex mathematical, expressions. This program has in it some sense of "where to go"-a sort of "complexity gradient" which guides it from wh at we would generally consider complex 
expressions to simpler ones. Part of MACSYMA's  repertoire is a program called " SIN", 
which does symbolic integration of functions; it is generally acknowledged to be superior to humans in some categories. It relies upon a number of different skills, as intelligence in general must: a vast body of knowledge, the technique of problem reduction, a large number of heuristics, and also some special tricks. 
Another program, written by Douglas Lenat at  Stanford, had as its aim to invent 
concepts and discover facts in very elemen tary mathematics. Beginning with the notion

concepts and discover facts in very elemen tary mathematics. Beginning with the notion 
of sets, and a collection of notions of what is "interesting" which had been spoon-fed into 
it, it "invented" the idea of counting, then th e idea of addition, then multiplication, then-
among other things-the notion of prime numbe rs, and it went so far as to rediscover 
Goldbach's conjecture! Of course these "d iscoveries" were all hundreds-even thousands-
of years old. Perhaps this may be explai ned in part by saying that the sense of 
"interesting" was conveyed by Lenat in a la rge number of rules which may have been

"interesting" was conveyed by Lenat in a la rge number of rules which may have been 
influenced by his twentieth century training;  nonetheless it is impressive. The program 
seemed to run out of steam after this very respectable performance. An interesting thing 
about it was that it was unable to develop or improve upon its own sense of what is interesting. That seemed another level of difficulty up-or perhaps several levels up.  
The Crux of Al: Representation of Knowledge 
 Many of the examples above have been cited in order to stress that the way a domain is 
represented has a huge bearing on how that  domain is "understood". A program which

represented has a huge bearing on how that  domain is "understood". A program which 
merely printed out theorems of TNT in a preordained order would have no understanding 
of number theory; a program such as Lenat's with its extra layers of knowledge could be 
said to have a rudimentary sense of number theory; and one which embeds mathematical 
knowledge in a wide context of real-world experience would probably be the most able to 
"understand" in the sense that we think we do. It is this' representation of knowledge  that 
is at the crux of Al.

is at the crux of Al. 
In the early days it was assumed that knowledge came in sentence-like packets", 
and that the best way to implant knowledge in to a program was to develop a simple way 
of translating facts into small passive packets of data. Then every fact would simply be a 
piece of data, accessible to the programs using it. This is exemplified by chess programs, 
where board Positions are coded into matrices or lists of some sort and stored efficiently in memory where they can be retrieved and acted upon by subroutines.

where board Positions are coded into matrices or lists of some sort and stored efficiently in memory where they can be retrieved and acted upon by subroutines. 
The fact that human beings store facts in a more complicated way was

 612Known to psychologists for quite a while and has only recently been rediscovered by AI 
workers, who are now confronting the problems of "chunked" knowledge, and the 
difference between procedural and declarativ e types of knowledge, which is related, as 
we saw in Chapter XI, to the differen ce between knowledge which is accessible to 
introspection and knowledge which is inaccessible to introspection. 
The naive assumption that all knowledge s hould be coded into passive pieces of 
data is actually contradicted by the most funda mental fact about computer design: that is, 
how to add, subtract, multiply, and so on is not  coded into pieces of data and stored in 
memory; it is, in fact, represented nowhere in memory, but rather in the wiring patterns of 
the hardware. A pocket calculator does not store in its memory knowledge of how to add; that knowledge is encoded into its "guts". Th ere is no memory location to point to if

the hardware. A pocket calculator does not store in its memory knowledge of how to add; that knowledge is encoded into its "guts". Th ere is no memory location to point to if 
somebody demands, "Show me where the knowle dge of how to add resides in this 
machine!" 
A large amount of work in Al has neve rtheless gone into systems in which the 
bulk of the knowledge is stored in specific places-that is, declaratively. It goes without 
saying that some knowledge has to be embodi ed in programs; otherwise one would not 
have a program at all, but merely an en cyclopedia. The question is how to split up 
knowledge between program and data. Not that it is always easy to distinguish between

machine!" 
knowledge between program and data. Not that it is always easy to distinguish between 
program and data, by any means. I hope that  was made clear enough in Chapter XVI. But 
in the development of a system, if the programmer intuitively conceives of some 
particular item as data (or as program), that may have significant repercussions on the system's structure, because as one program s one does tend to distinguish between data-
like objects and program-like objects. 
It is important to point out that in pr inciple, any manner of coding information 
into data structures or procedures is as good as any other, in the sense that if you are not

into data structures or procedures is as good as any other, in the sense that if you are not 
too concerned about efficiency, what you can do in one scheme, you can do in the other. 
However, reasons can be given which seem to indicate that one method is definitely 
superior to the other. For instance, consid er the following argument in favor of using 
procedural representations only: "As soon as  you try to encode features of sufficient 
complexity into data, you are forced into developing what amounts to a new language, or 
formalism. So in effect your data structur es become program-like, with some piece of

formalism. So in effect your data structur es become program-like, with some piece of 
your program serving as their interprete r; you might as well represent the same 
information directly in procedural form to  begin with, and obviate the extra level of 
interpretation." 
 
DNA and Proteins Help Give Some Perspective 
 
This argument sounds quite convincing, and yet, if interpreted a little loosely, it can be read as an argument for the abolishment of DNA  and RNA . Why encode genetic 
information in DNA , when by representing it directly in  proteins, you could eliminate not

information in DNA , when by representing it directly in  proteins, you could eliminate not 
just one, but two levels of interpretation? The answer  is: it turns out that it is extremely 
useful to have

 613 
the same information in several different fo rms for different purposes. One advantage of 
storing genetic information in th e modular and data-like form of DNA  is that two 
individuals' genes can be easily recombined to  form a new genotype. This would be very 
difficult if the information were only in prot eins. A second reason for storing information 
in DNA  is that it is easy to transcribe and transl ate it into proteins. When it is not needed, 
it does not take up much room; when it is n eeded, it serves as a template. There is no 
mechanism for copying one protein off of anothe r; their folded tertiary structures would 
make copying highly unwieldy. Complementarily, it is almost imperative to be able to get 
genetic information into three-dimensional structures such as enzymes, because the 
recognition and manipulation of molecules is by its nature a three-dimensional operation.

recognition and manipulation of molecules is by its nature a three-dimensional operation. 
Thus the argument for purely procedural represen tations is seen to be quite fallacious in 
the context of cells. It suggests that there are advantages to being able to switch back and 
forth between procedural and declarative representations. This is probably true also in AI. 
This issue was raised by Francis Cric k in a conference on communication with 
extraterrestrial intelligence: 
 
We see on Earth that there are two molecules, one of which is good for 
replication [ DNA ] and one of which is good for acti on [proteins]. Is it possible to

replication [ DNA ] and one of which is good for acti on [proteins]. Is it possible to 
devise a system in which one molecule does both jobs, or are there perhaps strong 
arguments, from systems analysis, which might suggest (if they exist) that to divide 
the job into two gives a great advantage, This is a question to which I do not know the answer.
14 
 
Modularity of Knowledge 
 
Another question which comes up in the repr esentation of knowledge is modularity. How 
easy is it to insert new knowledge? How easy is it to revise old knowledge? How modular

14 
 
easy is it to insert new knowledge? How easy is it to revise old knowledge? How modular 
are books? It all depends. If from a tightly structured book with many cross-references a 
single chapter is removed, the rest of the book may become virtually incomprehensible. It 
is like trying to pull a single strand out of a spider web-you ruin the whole in doing so. 
On the other hand, some books are quite modular, having independent chapters. 
Consider a straightforward theorem-generating program which uses TNT 's 
axioms and rules of inference. The "knowledge " of such a program has two aspects. It

axioms and rules of inference. The "knowledge " of such a program has two aspects. It 
resides implicitly in the axioms and rules, and explicitly in the body of theorems which 
have so far been produced. Depending on which way you look at the knowledge, you will 
see it either as modular or as spread a ll around and completely nonmodular. For instance, 
suppose you had written such a program but had forgotten to include TNT 's Axiom I in 
the list of axioms. After the program ha d done many thousands of derivations, you 
realized your oversight, and inserted the new ax iom. The fact that you can do so in a trice

realized your oversight, and inserted the new ax iom. The fact that you can do so in a trice 
shows that the system's implicit knowledge is  modular; but the new axiom's contribution 
to the explicit knowledge of the system will only be reflected after a long time-after its 
effects have "dif-

 614fused" outwards, as the odor of perfume slow ly diffuses in a room when the bottle is 
broken. In that sense the new knowledge takes a long time to be incorporated. 
Furthermore, if you wanted to go back a nd replace Axiom I by its negation, you could not 
just do that by itself; you would have to dele te all theorems which had involved Axiom 1 
in their derivations. Clearly this system's explicit knowledge is not nearly so modular as its implicit knowledge. 
It would be useful if we learned how to transplant knowledge modularly. Then to 
teach everyone French, we would just open up their heads and operate in a fixed way on 
their neural structures-then they would know how  to speak French. Of course, this is only 
a hilarious pipe dream. 
Another aspect of knowledge representati on has to do with the way in which one 
wishes to use the knowledge. Are inferen ces supposed to be drawn as pieces of

wishes to use the knowledge. Are inferen ces supposed to be drawn as pieces of 
information arrive? Should analogies and comp arisons constantly be being made between 
new information and old information? In a chess program, for instance, if you want to 
generate look-ahead trees, then a represen tation which encodes board positions with a 
minimum of redundancy will be preferable to  one which repeats the information in 
several different ways. But if you want your  program to "understand" a board position by 
looking for patterns and comparing them to known patterns, then representing the same

looking for patterns and comparing them to known patterns, then representing the same 
information several times over in different forms will be more useful. 
 
Representing Knowledge in a Logical Formalism 
 There are various schools of thought concerning the best way to represent and manipulate 
knowledge. One which has had great influence advocates representations using formal 
notations similar to those for TNT -using propositional connectives and quantifiers. The 
basic operations in such representations are,  not surprisingly, formalizations of deductive

basic operations in such representations are,  not surprisingly, formalizations of deductive 
reasoning. Logical deductions can be made using rules of inference analogous to some of those in TNT . Querying the system about some particular idea sets up a goal in the form 
of a string to be derived. For example: "Is MUMON  a theorem?" Then the automatic 
reasoning mechanisms take over in a goal oriented way, using various methods of problem reduction. 
For example, suppose that the proposition "All formal arithmetics are incomplete" 
were known, and the program were queried, "Is Principia Mathematica  incomplete?" In

were known, and the program were queried, "Is Principia Mathematica  incomplete?" In 
scanning the list of known facts-often called the data base-the system might notice that if 
it could establish that Principia Mathematica is a formal arithmetic, then it could answer 
the question. Therefore the proposition " Principia Mathematica  is a formal arithmetic" 
would be set up as a subgoal, and then problem reduction would take over. If it could find further things which would help in establishing (or refuting) the goal or the subgoal, it would work on them-and so on, recursively. This process is given the name of backwards 
chaining , since it begins with the goal and works its way backwards, presumably towards

chaining , since it begins with the goal and works its way backwards, presumably towards 
things which may already be known. If one ma kes a graphic representation of the main 
goal,

 615subsidiary goals, subsubgoals, etc., a tree-like structure will arise, since the main goal 
may involve several different subgoals, each  of which in turn involves several 
subsubgoals, etc. 
Notice that this method is not guaranteed to resolve the question, for there may be 
no way of establishing within the system that Principia Mathematica is a formal arithmetic. This does not imply, however, that either the goal or the subgoal is a false 
statement-merely that they cannot be derive d with the knowledge currently available to 
the system. The system may print out, in su ch a circumstance, "I do not know" or words 
to that effect. The fact that some questions  are left open is of course similar to the 
incompleteness from which certain well-known formal systems suffer. 
 
Deductive vs. Analogical Awareness 
 This method affords a deductive awareness of the domain that is represented, in that

 This method affords a deductive awareness of the domain that is represented, in that 
correct logical conclusions can be drawn from known facts. However, it misses something of the human ability to spot sim ilarities and to compare situations-it misses 
what might be called analogical awareness-a cr ucial side of human intelligence. This is 
not to say that analogical thought processes ca nnot be forced into such a mold, but they 
do not lend themselves naturally to being captu red in that kind of formalism. These days, 
logic-oriented systems are not so much in  vogue as other kinds, which allow complex

logic-oriented systems are not so much in  vogue as other kinds, which allow complex 
forms of comparisons to be carried out rather naturally. 
When you realize that knowledge representation is an altogether different ball 
game than mere storage of numbers, then the idea that "a computer has the memory of an 
elephant" is an easy myth to explode. What is stored in memory  is not necessarily 
synonymous with what a program knows ; for even if a give n piece of knowledge is 
encoded somewhere inside a complex system, there may be no procedure, or rule, or 
other type of handler of data , which can get at it-it may be inaccessible. In such a case, 
you can say that the piece of knowledge has been "forgotten" because access to it has

you can say that the piece of knowledge has been "forgotten" because access to it has 
been temporarily or permanently lost. Thus a computer program may "forget" something 
on a high level which it "remembers" on a low level. This is another one of those ever-recurring level distinctions, from which we  can probably learn much about our own 
selves. When a human forgets, it most likely means that a high-level pointer has been lost-not that any information has been dele ted or destroyed. This highlights the extreme 
importance of keeping track of the ways in  which you store incoming experiences, for

importance of keeping track of the ways in  which you store incoming experiences, for 
you never know in advance under what circumstances, or from what angle, you will want to pull something out of storage. 
 
From Computer Haiku to an RTN-Grammar 
 The complexity of the knowledge representation in human heads first hit home with me when I was working on a program to generate English sentences "out of the blue". I had 
come to this project in a rather interest-

 616ing way. I had heard on the radio a few ex amples of so-called "Computer Haiku". 
Something about them struck me deeply. There was a large element of humor and 
simultaneously mystery to making a computer generate something which ordinarily would be considered an artistic creation. I was highly amused by the humorous aspect, 
and I was very motivated by the mystery-even contradiction-of programming creative acts. So I set out to write a program even more mysteriously contradictory and humorous 
than the haiku program. 
At first I was concerned with making the gr ammar flexible and recursive, so that 
one would not have the sense that the program  was merely filling in the blanks in some 
template. At about that time I ran across a Scientific American  article by Victor Yngve in 
which he described a simple but flexible gr ammar which could produce a wide variety of 
sentences of the type found in some childre n's books. I modified some of the ideas I'd

sentences of the type found in some childre n's books. I modified some of the ideas I'd 
gleaned from that article and came up with a set of procedures which formed a Recursive 
Transition Network grammar, as described in Chapter V. In this grammar, the selection 
of words in a sentence was determined by a process which began by selecting-at random-
the overall structure of the sentence; gradua lly the decision-making process trickled down 
through lower levels of structure until the word  level and the letter level were reached. A 
lot had to be done below the word level, su ch as inflecting verbs and making plurals of

lot had to be done below the word level, su ch as inflecting verbs and making plurals of 
nouns; also irregular verb and noun forms were first formed regularly, and then if they 
matched entries in a table, substitutions of  the proper (irregular) forms were made. As 
each word reached its final form, it was printed out. The program was like the proverbial monkey at a typewriter, but operating on several levels of linguistic structure simultaneously-not just the letter level. 
In the early stages of developing the pr ogram, I used a totally silly vocabulary-
deliberately, since I was aiming at humor. It produced a lot of nonsense sentences, some

deliberately, since I was aiming at humor. It produced a lot of nonsense sentences, some 
of which had very complicated structures, ot hers of which were rather short. Some 
excerpts are shown below: 
 A male pencil who must laugh clums ily would quack. Must program not 
always crunch girl at memory? The d ecimal bug which spits clumsily might 
tumble. Cake who does sure take an une xpected man within relationship might 
always dump card. Program ought run cheerfully. The worthy machine ought not always paste the astronomer. 
Oh, program who ought really run off of the girl writes musician for theater.

Oh, program who ought really run off of the girl writes musician for theater. 
The businesslike relationship quacks. The lucky girl which can always quack will never sure quack. The game quacks. Professor will write pickle. A bug tumbles. Man takes the box 
who slips.  
The effect is strongly surrealistic and at times a little reminiscent of

 617haiku-for example. the final sample of four consecutive short sentences. At first it seemed 
very funny and had a certain charm, but soon it became rather stale. After reading a few 
pages of output one could sense the limits  of the space in which the program was 
operating; and after that, seeing random poi nts inside that space-even though each one 
was "new"-was nothing new. This is, it seems to me, a general principle: you get bored 
with something not when you have exhausted its repertoire of behavior, but when you 
have mapped out the limits of the space that c ontains its behavior. The behavior space of 
a person is just about complex enough that it can continually surprise other people; but 
that wasn't true of my program. I reali zed that my goal of producing truly humorous 
output would require that far more subtlety be  programmed in. But what, in this case, was 
meant by "subtlety It was clear that absurd  juxtapositions of words were just too

meant by "subtlety It was clear that absurd  juxtapositions of words were just too 
unsubtle; I needed a way to ensure that wo rds would be used in accordance with the 
realities of the world. This was where thought s about representation of knowledge began 
to enter the picture.  
From RTN's to ATN's 
 The idea I adopted was to classify each word-noun, verb, preposition, etc.-in several 
different "semantic dimensions". Thus, each word was a member of classes of various 
sorts; then there were also superclasses-cla sses of classes (reminiscent of the remark by 
Ulam). In principle, such aggregation coul d continue to any number of levels, but I

Ulam). In principle, such aggregation coul d continue to any number of levels, but I 
stopped at two. At any given moment, the choice of words was now semantically 
restricted, because it was required that there should be agreement  between the various 
parts of the phrase being constructed. The idea wa s, for instance, that certain kinds of acts 
could be performed only by animate objects; that  only certain kinds of abstractions could 
influence events, and so on. The decisions a bout what categories were reasonable, and

influence events, and so on. The decisions a bout what categories were reasonable, and 
whether each category was better thought of as a class or a superclass, were quite complicated. All words were branded in several different dimensions. Common prepositions-"of", "in", etc.-had several dis tinct entries, corresponding to their distinct 
usages. Now, the output began to be much more comprehensible-and for that reason it 
was funny in a new way.  
A Little Turing Test 
 Below, I have reproduced nine selections, carefully culled from many pages of output 
from later versions of my program. Along w ith them are three (seriously intended)

from later versions of my program. Along w ith them are three (seriously intended) 
human-written sentences. Which?  
(1) Blurting may be considered as the reci procal substitution of semiotic material 
(dubbing) for a semiotic dialogical product in a dynamic reflexion.

 618(2) Rather think of a pathway of a 'se quence' of gedankenexperiment simpletons 
where heir-lines are a prima facie case of a paradiachronic transitivity. 
 (3) Think of that as a chain strength possi bility of what, eventually, comes out as a 
product (epistemic conditions:,) and the product is not a Frankfurt-ish packing-
it-all-in. 
 (4) Despite the efforts, the reply, if you will, had been supported by the Orient; 
hence a fallacy will thereafter be suspe nded by the attitude which will be being 
held by the ambassador. 
 (5) Of course, until the upheavals, th e ambassador was slightly gradually 
mollycoddling the rabble. 
 (6) Supposedly, refined liberty caused the attitudes insofar as peace is distilled by 
the consequences which will not ev entually be caused by the command 
irrevocably insofar as peace of it is  sometimes causing the intransigency 
infinitesimally surprisingly.

infinitesimally surprisingly. 
 (7) According to the sophists, the campaigns in the city-states, in other words, have 
been accepted by the Orient cunningly. Of c ourse, the Orient has been separated 
by the states particularly violently. The Orient supports the efforts which had been supported by mankind.  
(8) Admittedly, the hierarchical origin of the fallacy, nevertheless, will be 
prophesied by the enemies of it. By the same token, the individualists will have 
testified that intransigency will not have suspended the campaigns.

(7) According to the sophists, the campaigns in the city-states, in other words, have 
testified that intransigency will not have suspended the campaigns. 
 (9) Needless to say, during the upheaval wh ich will have warranted the secrecy, the 
replies do not separate the Orient. Of course, the countries, ipso facto, are 
always probing liberty. 
 (10) Although a Nobel Prize was being ach ieved by the humanists, yet in addition, 
it was being achieved by the serf. 
 (11) An attitude will often be held by the serfs of a strife-torn nation.  (12) Moreover, the Nobel Prizes will be achieved. By the same token, despite the

(9) Needless to say, during the upheaval wh ich will have warranted the secrecy, the 
 (11) An attitude will often be held by the serfs of a strife-torn nation.  (12) Moreover, the Nobel Prizes will be achieved. By the same token, despite the 
consequence, the Nobel Prizes whic h will be achieved will sometimes be 
achieved by a woman. 
 
The human-written sentences are numbers 1 to 3; they were drawn from the 
contemporary journal Art-Language15 and are- as far as I can tellcompletely serious 
efforts among literate and sane people to co mmunicate something to each other. That 
they appear here out of context is not t oo misleading, since their proper context sounds 
just the same as they do.

 619My program produced the rest. Numbers 10 to  12 were chosen to show that there 
were occasional bursts of total lucidity; numbe rs 7 to 9 are more typical of the output, 
floating, in that curious and provocative ne therworld between meaning and no-meaning; 
and then numbers 4 to 6 pretty much tran scend meaning. In a generous mood, one could 
say that they stand on their own as pure "language objects", something like pieces of 
abstract sculpture carved out of words instead  of stone; alternatively, one could say that 
they are pure pseudointellectual drivel. 
My choice of vocabulary was still aime d at producing humorous effects. The 
flavor of the output is hard to characterize.  Although much of it "makes sense", at least 
on a single-sentence level, one definitely gets the feeling that the output is coming from a 
source with no understanding of what it is sayi ng and no reason to say it. In particular,

source with no understanding of what it is sayi ng and no reason to say it. In particular, 
one senses an utter lack of visual imagery behind the words. When I saw such sentences 
come pouring out of the line printer, I experienced complex emotions. I was very amused by the silliness of the output. I was also ve ry proud of my achievement and tried to 
describe it to friends as similar to giving rules for building up meaningful stories in 
Arabic out of single strokes of the pen-an exaggeration, but it pleased me to think of it 
that way. And lastly I was deeply thrille d by the knowledge that this enormously

that way. And lastly I was deeply thrille d by the knowledge that this enormously 
complicated machine was shunting around long tr ains of symbols inside it according to 
rules, and that these long trains of symbols were something like thoughts in my own head ... something like them. 
 
Images of What Thought Is 
 Of course I didn't fool myself into thinking that there was a conscious being behind those 
sentences-far from it. Of all people, I was the most aware of the reasons that this program 
was terribly remote from real thought. Tesler's Theorem is quite apt here: as soon as this

was terribly remote from real thought. Tesler's Theorem is quite apt here: as soon as this 
level of language handling ability had been mechanized, it was clear that it did not 
constitute intelligence. But this strong experi ence left me with an image: a glimmering 
sense that real thought was composed of much longer, much more complicated trains of 
symbols in the brain-many trains movi ng simultaneously down many parallel and 
crisscrossing tracks, their cars being pushed and pulled, attached and detached, switched 
from track to track by a myriad neural shunting-engines ...

from track to track by a myriad neural shunting-engines ... 
It was an intangible image which I cannot  convey in words, and it was only an 
image. But images and intuitions and motivations lie mingled close in the mind, and my utter fascination with this image was a consta nt spur to think more deeply about what 
thought really could be. I have tried in othe r parts of this book to communicate some of 
the daughter images of this original image-particularly in the Prelude, Ant Fugue . 
What stands out in my mind now, as I look back at this program from the

What stands out in my mind now, as I look back at this program from the 
perspective of a dozen years, is how there is no sense of imagery behind what is being 
said. The program had no idea what a serf is, wh at a person is, or what anything at all is. 
The words were empty formal symbols, as empty

 620
 
FIGURE 116. A meaningful story in Arabic. [From A. Khatibi and M. S~elmassi, The 
Splendour of Islamic Calligraphy (New York: Rizzoli, 1976).

 621as-perhaps emptier than-the p and q of the pq-system. My program took advantage of the 
fact that when people read text, they quite naturally tend to imbue each word with its full 
flavor-as if that were necessarily attached to the group of letters which form the word. 
My program could be looked at as a form al system, whose "theorems"-the output 
sentences-had ready-made interpretations (at least to speakers of English). But unlike the 
pq-system, these "theorems" were not all true statements when interpreted that way. 
Many were false, many were nonsense. 
In its humble way, the pq-system mirrored a tiny corner of the world. But when 
my program ran, there was no mirror inside it of how the world works, except for the 
small semantic constraints which it had to follow. To create such a mirror of 
understanding, I would have had to wrap each c oncept in layers and layers of knowledge

understanding, I would have had to wrap each c oncept in layers and layers of knowledge 
about the world. To do this would have been  another kind of effort from what I had 
intended to do. Not that I didn't often think of trying to do it-but I never got around to 
trying it out. 
 
Higher-Level Grammars ... 
 
In fact, I often pondered wh ether I could write an ATN -grammar (or some other kind of 
sentence-producing program) which would only produce true sentences about the world. 
Such a grammar would imbue the words with genuine meanings, in the way it happened in the pq-system and in TNT . This idea of a language in which false statements are

Such a grammar would imbue the words with genuine meanings, in the way it happened in the pq-system and in TNT . This idea of a language in which false statements are 
ungrammatical is an old one, going back to Johann Amos Comenius, in 1633. It is very 
appealing because you have a crystal ball embodi ed in your grammar: just write down the 
statement you want to know about, and check to see if it is grammatical... . Actually, 
Comenius went even further, for in hi s language, false statements were not only 
ungrammatical-they were inexpressible! 
Carrying this thought in another direction, you might imagine a highlevel 
grammar which would produce random koans . Why not? Such a grammar would be

grammar which would produce random koans . Why not? Such a grammar would be 
equivalent to a formal system whose theore ms are koans. And if you had such a program, 
could you not arrange it to produce only genuine  koans? My friend Marsha Meredith was 
enthusastic about this idea of  "Artificial Ism", so she tackled the project of writing a 
koan-writing program. One of her early effo rts produced this curious quasi-koan: 
 
A SMALL YOUNG MASTER WANTED A SMALL WHITE GNARLED BOWL. "HOW CAN WE LEARN AND UNDERSTAND WITHOUT STUDY"

A SMALL YOUNG MASTER WANTED A SMALL WHITE GNARLED BOWL. "HOW CAN WE LEARN AND UNDERSTAND WITHOUT STUDY" 
THE YOUNG MASTER ASKED A LARGE CONFUSED MASTER. THE CONFUSED MASTER WALKED FROM A BROWN HARD MOUNTAIN TO A WHITE SOFT MOUNTAIN WITH A SMALL RED STONY BOWL. THE CONFUSED MASTER SAW A RED SOFT HUT.THE CONFUSED MASTER WANTED THE HUT. "WHY DID BODHIDHARMA COME INTO CHINA?" THE CONFUSED MASTER

 622 
ASKED A LARGE ENLIGHTENED STUDENT. "THE PEACHES ARE LARGE", THE STUDENT ANSWERED THE CONFUSED MASTER.w "HOW CAN WE LEARN AND UNDERSTAND WITHOUT STUDY?" THE CONFUSED MASTER ASKED A LARGE OLD MASTER. THE OLD MASTER WALKED FROM A WHITE STONY G0025. THE OLD MASTER GOT LOST. 
 
Your personal decision procedure fo r koan genuineness probably reached a 
verdict without need of the Geometric Code or the Art of Zen Stri ngs. If the lack of 
pronouns or the unsophisticated syntax didn' t arouse your suspicions, that strange 
"00025" towards the end must have. What is it? It is a strange fluke-a manifestation of a 
bug which caused the program to print out, in place of the English word for an object, the program's internal name for the "node" (a LISP  atom, in fact) where all information 
concerning that particular object was store d. So here we have a "window" onto a lower 
level of the underlying Zen mind-a level th at should have remained invisible.

level of the underlying Zen mind-a level th at should have remained invisible. 
Unfortunately, we don't have such clear windows onto the lower levels of human Zen minds. 
The sequence of actions, though a little arbitrary, comes from a recursive LISP  
procedure called " CASCADE ", which creates chains of actions linked in a vaguely 
causal way to each other. Although the degree of comprehension of the world possessed 
by this koan generator is clearly not stupendous, work is in progress to make its output a 
little more genuine seeming. 
 
Grammars for Music?

Grammars for Music? 
 
Then there is music. This is a domain which you might suppose, on first thought, would lend itself admirably to being codified in an ATN  grammar, or some such program. 
Whereas (to continue this naive line of t hought) language relies on connections with the 
outside world for meaning, music is purely form al. There is no reference to things "out 
there" in the sounds of music; there is just pure syntax-note following note, chord 
following chord, measure following measure, phrase following phrase...

following chord, measure following measure, phrase following phrase... 
But wait. Something is wrong in this analysis. Why is some music so much 
deeper and more beautiful than other music? It  is because form, in music, is expressive-
expressive to some strange subconscious re gions of our minds. The sounds of music do 
not refer to serfs or city-states, but they do trigger clouds of emotion in our innermost

not refer to serfs or city-states, but they do trigger clouds of emotion in our innermost 
selves; in that sense musical meaning is dependent on intangible links from the symbols to things in the world-those "things", in this case, being secret software structures in our 
minds. No, great music will not come out  of such an easy formalism as an ATN -
grammar. Pseudomusic, like pseudo-fairy tales, may well come out-and that will be a valuable exploration for people to make-but th e secrets of meaning in music lie far, far 
deeper than pure syntax.

deeper than pure syntax. 
I should clarify one point here: in principl e, ATN-grammars have all the power of 
any programming formalism, so if musical meaning is captur-

 623able in any way at all (which I believe it is),  it is capturable in an A I N - grammar. True. 
But in that case, I maintain, the grammar will be defining not just musical structures, but the entire structures of the mind of a beholder. The "grammar" will be a full grammar of thought-not just a grammar of music.  
Winograd's Program SHRDLU 
 
What kind of program would it take to ma ke human beings admit that it had some 
"understanding", even if begrudgingly? What  would it take before you wouldn't feel 
intuitively that there is "nothing there"? 
In the years 1968-70, Terry Winograd (a lias Dr. Tony Earrwig) was a doctoral 
student at MIT , working on the joint problems of language and understanding. At that 
time at MIT , much Al research involved the so-called blocks world -a relatively simple 
domain in which problems concerning both vision and language-handling by computer

domain in which problems concerning both vision and language-handling by computer 
could fit easily. The blocks world consists of a table with various kinds of toy-like blocks 
on it-square ones, oblong ones, triangular ones, etc., in various colors. (For a "blocks world" of another kind, see Figure 117: the painting Mental Arithmetic  by Magritte. I find 
its title singularly appropriate in this context.) The vision problems in the MIT  blocks 
world are very tricky: how can a computer figure out, from a TV-scan of a scene with 
many blocks in it, just what kinds of blocks are present, and what their relationships are?

many blocks in it, just what kinds of blocks are present, and what their relationships are? 
Some blocks may be perched on top of others, some may be in front of others, there may 
be shadows, and so on. 
 
FIGURE 117. Mental Arithmetic, by Rene Magritte (1931).

 624Winograd s work was separate from the issues of vision, however. Beginning with 
the assumption that the blocks world was well represented h inside the computer's memory, he confronted the many-faceted problem of how to get the computer to: 
 
(1)  understand questions in English about the situation; (2)  give answers in English to questions about the situation;  (3)  understand requests in English to manipulate the blocks;  (4)  break down each request into a sequence of operations it could do; (5)  understand what it had done, and for what reasons;  (6)  describe its actions and their reasons, in English.  
It might seem reasonable to break up the overall program into modular 
subprograms, with one module for each different part of the problem; then, after the modules have been developed separately, to integrate them smoothly. Winograd found 
that this strategy of developing independent  modules posed fundamental difficulties. He

that this strategy of developing independent  modules posed fundamental difficulties. He 
developed a radical approach, which challe nged the theory that intelligence can be 
compartmentalized into independent or semi-independent pieces. His program SHRDLU  
named after the old code " ETAOIN SHRDLU ", used by linotype operators to mark 
typos in a newspaper column-did not separate the problem into clean conceptual parts. The operations of parsing sentences, producing internal representations, reasoning about the world represented inside itself, answeri ng questions, and so on, were all deeply and 
intricately meshed together in a procedur al representation of knowledge. Some critics

intricately meshed together in a procedur al representation of knowledge. Some critics 
have charged that his program is so tangled that it does not represent any "theory" at all about language, nor does it contribute in any wa y to our insights about thought processes. 
Nothing could be more wrong than such claims , in my opinion. A tour de force such as 
SHRDLU  may not be isomorphic to what we do- in fact, in no way should you think that 
in SHRDLU , the "symbol level" has been attained-but the act of creating it and thinking 
about it offers tremendous insight into the way intelligence works. 
 
The Structure of SHRDLU

The Structure of SHRDLU 
 In fact, SHRDLU  does consist of separate procedur es, each of which contains some 
knowledge about the world; but the procedures  have such a strong interdependency that 
they cannot be cleanly teased apart. The program is like a very tangled knot which resists untangling; but the fact that you cannot untangle it does not mean that you cannot 
understand it. There may be an elegant geometri cal description of the entire knot even if 
it is physically messy. We could go back to a metaphor from the Mu Offering, and 
compare it to looking at an orchard from a "natural" angle. 
Winograd has written lucidly about SHRDLU . I quote here from his article in

Winograd has written lucidly about SHRDLU . I quote here from his article in 
Schank and Colby's book:

 625One of the basic viewpoints underlying the model is that all language use can be 
thought of as a way of activating procedur es within the hearer. We can think of 
any utterance as a program-one that indi rectly causes a set of operations to he 
carried out within the hearer's cognitive sy stem. This "program writing" is indirect 
in the sense that we are dealing with an intelligent interpreter, who may take a set 
of actions which are quite different fr om those the speaker intended. The exact 
form is determined by his knowledge of the world, his expectations about the 
person talking to him, etc. in this progr am we have a simple version of this 
process of interpretation as it takes place in the robot. Each sentence interpreted by 
the robot is converted to a set of instructions in PLANNER . The program that is 
created is then executed to achieve the desired effect." 
  
PLANNER Facilitates Problem Reduction

PLANNER Facilitates Problem Reduction 
 
The language PLANNER , referred to here, is an At language whose principal feature is 
that some of the operations necessary for problem reduction are built in-namely, the recursive process of creating a tree of jsubgoals , subsubgoals, etc. What this means is that 
such processes, instead of having to be spelled out time and time again by the 
programmer, are automatically implied by so-called GOAL -statements. Someone who

programmer, are automatically implied by so-called GOAL -statements. Someone who 
reads a PLANNER  program will see no explicit reference to such operations; in argon, 
they are user-transparent. If one path-in the tree fails to achieve the desired goal, then the PLANNER  program will "backtrack" and try anothe r route. "Backtracking" is the magic 
word as far as PLANNER  is concerned. 
Winograd's program made excellent  use of these features of PLANNER -more 
exactly, of MICROPLANNER , a partial implementation of the plans for PLANNER . In

exactly, of MICROPLANNER , a partial implementation of the plans for PLANNER . In 
the past few years, however, people with the goal of developing At have concluded that 
automatic backtrack ing, as in PLANNER , has definite disadvantages, and that it will 
probably not lead to their goal; therefore they have backed off from it, preferring to try 
other routes to AI. 
Let us listen to further comments from Winograd on SHRDLU : 
 
The definition of every word is a program wh ich is called at an appropriate point in 
the analysis, and which can do arbitrary computations involving the sentence and 
the present physical situation."

the present physical situation." 
 
Among the examples which Winograd cites is the following: 
 
The different possibilities for the meaning of "the" are procedures which check various facts about the context, then pr escribe actions such as "Look for a unique 
object in the data base which fits this de scription", or "Assert that the object being 
described is unique as far as the speaker is concerned." The program incorporates a 
variety of heuristics for deciding what part of the context is relevant. 
18

18 
 
It is amazing how deep this problem with the wo rd "the" is. It is probably safe to say that 
writing a program which can fully handle the top five words

 626of English-"the", "of", "and", "a", and "to" -would be equivalent to solving the entire 
problem of AI, and hence tantamount to knowing what intelligence and consciousness 
are. A small digression: the five most common nouns in English are-according to the 
Word Frequency Book compiled by John B. Carroll et al-"time", "people", "way", 
"water", and "words" (in that order). The am azing thing about this is that most people 
have no idea that we think in such abstract  terms. Ask your friends, and 10 to 1 they'll 
guess such words as "man", "house", "car",  "dog", and "money". And while we're on the 
subject of frequencies-the top twelve le tters in English, in order, according to 
Mergenthaler, are: " ETAOIN SHRDLU ". 
One amusing feature of SHRDLU  which runs totally against the stereotype of 
computers as "number crunchers" is this f act, pointed out by Winogr ad: "Our system does

computers as "number crunchers" is this f act, pointed out by Winogr ad: "Our system does 
not accept numbers in numeric form, and has only been taught to count to ten."19 With 
all its mathematical underpinning, SHRDLU is a mathematical ignoramus! Just like Aunt 
Hillary, SHRDLU  doesn't know anything about the lower levels which make it up. Its 
knowledge is largely procedural  (see particularly the rema rk by "Dr, Tony Earrwig" in 
section 11 of the previous Dialogue). 
It is interesting to contrast the procedural embedding of knowledge in SHRDLU

It is interesting to contrast the procedural embedding of knowledge in SHRDLU  
with the knowledge in my sentence-generati on program. All of the syntactical knowledge 
in my program was procedurally embedded in Augmented Transition Networks, written in the language Algol; but the semantic knowledge-the information about semantic class membership-was static: it was contained in a short list of numbers after each word. There 
were a few words, such as the auxiliary verbs "to be", "to have", and others, which were represented totally in procedures in Algol, but  they were the exceptions. By contrast, in

were a few words, such as the auxiliary verbs "to be", "to have", and others, which were represented totally in procedures in Algol, but  they were the exceptions. By contrast, in 
SHRDLU , all words were represented as programs. Here is a case which demonstrates 
that, despite the theoretical equivalence of data and programs, in practice the choice of 
one over the other has major consequences. 
 
Syntax and Semantics 
 
And now, a few more words from Winograd:  
Our program does not operate by first pa rsing a sentence, then doing semantic 
analysis, and finally by using deducti on to produce a response. These three 
activities go on concurrently throughout th e understanding of a sentence. As soon

activities go on concurrently throughout th e understanding of a sentence. As soon 
as a piece of syntactic structure begins to take shape, a semantic program is called to see whether it might make sense,  and the resultant answer can direct the 
parsing. In deciding whether it makes sense, the semantic routine may call deductive processes and ask questions about the real world. As an example, in sentence 34 of the Dialogue (" Put the blue pyramid on the block in the 
box"), the parser first comes up with "the blue pyramid on the block" as a candidate 
for a noun group. At this point, semantic analysis is done, and since "the" is

for a noun group. At this point, semantic analysis is done, and since "the" is 
definite, a check is made in the data ba se for the object being referred to. When no 
such object is found, the parsing is redi rected to find the noun group "the blue 
pyramid". It will then go on to find "on the block

 627in the box" as a single phrase indicating a location .-.. Thus there is a continuing 
interplay between the different sorts of an alysis, with the results of one affecting 
the others .20 
 
It is extremely interesting that in natu ral language, syntax and semantics are so 
deeply intertwined. Last Chapter, in discu ssing the elusive concept of "form", we had 
broken the notion into two categories: s yntactic form, which is detectable by a 
predictably terminating decision procedure, and semantic form, which is not. But here, 
Winograd is telling us that-at least when the usual senses of "syntax" and "semantics" are taken-they merge right into each other, in  natural language. The external form of a 
sentence-that is, its composition in terms of el ementary signs-does not divide up so neatly 
into syntactic and semantic aspects. This significant point for linguistics. 
Here are some final comments on SHRDLU  by Winograd.

Here are some final comments on SHRDLU  by Winograd. 
 Let us look at what the system would do with a simple description like "a red cube which supports a pyramid". The description will us e concepts like BLOCK, RED, PYRAMID, 
and EQUIDIMENSIONAL-all parts of the sys  FIGURE 118. Procedural representation of "a re d cube which supports a pyramid." 
Adapted from Roger Schank and Kenneth Co lby, Computer Models of Thought and 
Language (San Francisco: W. H. Freeman, 1973), p. 172 .

 628tem's underlying categorization of the worl d. The result can be represented in a 
flow chart like that in Figure 118. Note that  this is a program for finding an object 
fitting the description. It would_ then be  incorporated into a command for doing 
something with the object, a question asking so mething about it, or, if it appeared in 
a statement, it would become part of the program which was generated to represent 
the meaning for later use. Note that this b it of program could also be used as a test 
to see whether an object fit the description, if the first FIND  instruction were told 
in advance to look only at that particular object. 
At first glance, it seems that there is too much structure in this program, as 
we don't like to think of the meaning of a simple phrase as explicitly containing 
loops, conditional tests, and other progra mming details. The solution is to provide

loops, conditional tests, and other progra mming details. The solution is to provide 
an internal language that contains th e appropriate looping and checking as its 
primitives, and in which the representation of the process is as simple as the 
description. The program described in Figure 11S would be written in PLANNER  
looking something like what is below: 
 
(GOAL (IS ?X 1 BLOCK)) (GOAL (COLOR-OF ?X1 RED))  (GOAL (EQUIDIMENSIONAL ?X1))  (GOAL (IS ?X2 PYRAMID))  (GOAL (SUPPORT ?X1 ?X2)) 
 
The loops of the flowchart are implicit in PLANNER 'S backtrack control structure.

The loops of the flowchart are implicit in PLANNER 'S backtrack control structure. 
The description is evaluated by proceeding down the list until some goal fails, at which time the system backs up automatica lly to the last point where a decision 
was made, trying a different possibility. A decision can be made whenever a new 
object name or VARIABLE  (indicated by the prefix 
") such as "?X 1" or "?X2 " appears. The variables are used by the pattern matcher. 
If' they have already been assigned to a pa rticular item, it checks to see whether the

If' they have already been assigned to a pa rticular item, it checks to see whether the 
GOAL  is true for that item. If not, it check s for all possible items which satisfy the 
GOAL , by choosing one, and then taking su ccessive ones whenever backtracking 
occurs to that point. Thus, even the di stinction between testing and choosing is 
implicit.
21 
 
One significant strategy decision in devising this  program was to not translate all the way 
from English into LISP , but only partway-into PLANNER . Thus (since the PLANNER  
interpreter is itself written in LISP ), a new intermediate level- PLANNER -was inserted

implicit.
21 
 
interpreter is itself written in LISP ), a new intermediate level- PLANNER -was inserted 
between the top-level language (English) and the bottom-level language (machine language). Once a PLANNER  program had been made from an English sentence 
fragment, then it could be sent off to the PLANNER  interpreter, and the higher levels of 
SHRDLU  would be freed up, to work on new tasks. 
This kind of decision constantly crops  up: How many levels should a system 
have? How much and what kind of "intelligen ce" should be placed on which level? These 
are some of the hardest problems facing AI today. Since we know so little about natural

are some of the hardest problems facing AI today. Since we know so little about natural 
intelligence, it is hard for us to figure out wh ich level of an artificially intelligent system 
should carry out what part of a task.

 629This gives you a glimpse behind the scenes of the Dialogue preceding this 
Chapter. Next Chapter, we shall meet new and speculative ideas for AI.

Contrafactus 
 630Contrafactus 
 
The Crab has invited a small group of friends over to watch the Saturday 
afternoon football game on television. Achilles has already arrived, but the Tortoise and his friend the Sloth are still awaited. 
 
 Achilles: Could that be our friends, a-ri ding up on that unusual one-wheeled vehicle? 
 
(The Sloth and Tortoise dismount and come in.) 
 
Crab: Ah, my friends, I'm so glad you could make it. May I present my old and beloved 
acquaintance, Mr. Sloth-and this is Achilles. I believe you know the Tortoise. 
Sloth: This is the first time I can recall making the acquaintance of a Bicyclops. Pleased 
to meet you, Achilles. I've heard many fine things said about the bicyclopean species. 
Achilles: Likewise, I'm sure. May I ask about your elegant vehicle? Tortoise: Our tandem 
unicycle, you mean? Hardly elegant. It's just a way for two to get from A to B, at the same speed.

unicycle, you mean? Hardly elegant. It's just a way for two to get from A to B, at the same speed. 
Sloth: It's built by a company that also makes teeter-teeters. Achilles: I see, I see. What is that knob on it? Sloth: That's the gearshift. Achilles: Aha! And how many speeds does it have? Tortoise: One, including reverse. Most models have fewer, but this is a special model. Achilles: It looks like a very nice tandem unicycle. Oh, Mr. Crab, I wanted to tell you 
how much I enjoyed hearing your orchestra perform last night. 
Crab: Thank you, Achilles. Were you there by any chance, Mr. Sloth? Sloth: No, I

Crab: Thank you, Achilles. Were you there by any chance, Mr. Sloth? Sloth: No, I 
couldn't make it, I'm sad to say. I was participating in a mixed singles ping-ping 
tournament. It was quite exciting because my team was involved in a one-way tie for first place. 
Achilles: Did you win anything? Sloth: Certainly did-a two-sided Mobius stri p made out of copper; it is silver-plated on 
one side, and gold-plated on the othe r. Crab: Congratulations, Mr. Sloth.

one side, and gold-plated on the othe r. Crab: Congratulations, Mr. Sloth. 
Sloth: Thank you. Well, do tell me about the concert. Crab: It was a most enjoyable performance. We played some pieces by the Bach twins Sloth: The famous Job and Sebastian? Crab: One and the same. And there was one work that made me think of you, Mr. Sloth-a 
marvelous piano concerto for two left hands. The

 631next-to-last (and only) movement was a one-voice fugue. You can't imagine its 
intricacies. For our finale, we played  Beethoven's Ninth Zenfunny. At the end, 
everyone in the audience rose and clapped with one hand. It was overwhelming. 
Sloth: Oh, I'm sorry I missed it. But do you suppose it's been recorded: At home I have a 
fine hi-fi to play it on-the best two-channel monaural system money can buy. 
Crab: I'm sure you can find it somewhere. Well, my friends, the game is about to begin. Achilles: Who is playing today, Mr. Crab? Crab: I believe it's Home Team versus Visitors. Oh, no-that was last week. I think this 
week it's Out-of-Towners. 
Achilles: I'm rooting for Home Team. I always do. Sloth: Oh, how conventional. I never root for Home Team. The closer a team lives to the 
antipodes, the more I root for it. 
Achilles: Oh, so you live in the Antipodes? I've heard it's charming to live there, but I 
wouldn't want to visit them. They're so far away.

wouldn't want to visit them. They're so far away. 
Sloth: And the strange thing about them is that they don't get any closer no matter which 
way you travel. 
Tortoise: That's my kind of place. Crab: It's game time. I think I'll turn on the TV.  
(He walks over to an enormous cabinet with a screen, underneath which is an 
instrument panel as complicated as that of a jet airplane. He flicks a knob, and the football stadium a ears in bright vivid color on the screen.) 
 
Announcer: Good afternoon, fans. Well, it looks like that time of year has rolled around

Announcer: Good afternoon, fans. Well, it looks like that time of year has rolled around 
again when Home Team and Out-of-Town face each other on the gridiron and play out their classic pigskin rivalry. It's been dri zzling on and off this afternoon, and the field's 
a little wet, but despite the weather it promises to be a fine game, especially with that 
GREAT pair of eighth-backs playing for Home Team, Tedzilliger and Palindromi. And now, here's Pilipik, kicking off for Home Team. It's in the air! Flampson takes it for Out-of-Towners, and runs it back he's to the 20, the 25, the 30, and down at the 32. That was Mool in on the tackle for Home Team.

GREAT pair of eighth-backs playing for Home Team, Tedzilliger and Palindromi. And now, here's Pilipik, kicking off for Home Team. It's in the air! Flampson takes it for Out-of-Towners, and runs it back he's to the 20, the 25, the 30, and down at the 32. That was Mool in on the tackle for Home Team. 
Crab: A superb runback! Did you see how he was ALMOST tackled by Quilker-but 
somehow broke away? 
Sloth: Oh, don't be silly, Crab. Nothing of  the kind happened. Quilker did NOT tackle 
Flampson. There's no need to confuse poor Achilles (or the rest of us) with hocus-pocus about what "almost" happened. It's a fact-with no "almost" 's, "if "'s, "and" 's, or "but" 's. 
Announcer: Here's the instant replay. Just watch number 79, Quilker, come in from the 
side, surprising Flampson, and just about tackle him! 
Sloth: "Just about"! Bah! 'Achilles: Such a graceful maneuver! What would we do without instant replays?

 632Announcer: It's first down and 10 for Out-of-Town. Noddle takes the ball, hands off to 
Orwix-it's a reverse-Orwix runs around to the right, handing off to Flampson-a double 
reverse, folks!-and now 
Flampson hands it to Treefig, who's downed twelve yards behind scrimmage. A twelve-
yard loss on a triple reverse! 
Sloth: I love it! A sensational play! Achilles: But, Mr. S, I thought you were rooting for Out-of-Town. They lost twelve yards 
on the play. 
Moth: They did? Oh, well-who cares, as long as it was a beautiful play? Let's see it again.  
( ... and so the first half of the game passes. Towards the end of the third quarter, a 
particularly crucial play comes up for Home Team. They are behind by eight points. It's third down and 10, and they badly need a first down.) 
 
Announcer: The ball is hiked to Tedzilliger, who fades back, looking-for a receiver, and

Announcer: The ball is hiked to Tedzilliger, who fades back, looking-for a receiver, and 
fakes to Quilker. There's Palindromi, playing wide right, with nobody near him. Tedzilliger spots him and fires a low pass to him. Palindromi snatches it out of the air, and- ( There is an audible groan from the crowd .)--oh, he steps out of bounds! What a 
crushing blow for Home Team, folks! If Palindromi hadn't stepped out of bounds, he could've run all the way to the end zone for a touchdown! 
Let's watch the subjunctive instant replay.  
(And on the screen the same lineup appears as before.) 
 
The ball is hiked to Tedzilliger, who fades back, looking for a receiver, and fakes to

The ball is hiked to Tedzilliger, who fades back, looking for a receiver, and fakes to 
Quilker. There's Palindromi, playing wide right, with nobody near him. Tedzilliger 
spots him, and fires a low pass to him. Palindromi snatches it out of the air, and- (There is an audible gasp from the crowd. )-he almost steps out of bounds! But he's 
still in bounds, and it's clear all the way to the end zone! Palindromi streaks in, for a touchdown for Home Team! (The stadium breaks into a giant roar of approval.) Well, folks, that's what would've happened if Palindromi hadn't stepped out of bounds.  
Achilles: Wait a minute ... WAS there a touchdown, or WASN'T there? Crab: Oh, no. That was just the subjunctive instant replay. They simply followed a

Achilles: Wait a minute ... WAS there a touchdown, or WASN'T there? Crab: Oh, no. That was just the subjunctive instant replay. They simply followed a 
hypothetical a little way out, you know. 
Sloth: That is the most ridiculous thing I ever heard of! Next thing you know, they'll be 
inventing concrete earmuffs. 
Tortoise: Subjunctive instant replays are a little unusual, aren't they? Crab: Not particularly, if you have a Subjunc-TV. Achilles: Is that one grade below a junk TV? Crab: Not at all! It's a new kind of TV, which can go into the subjunctive mode. They're 
particularly good for football games and such. I just got mine. 
Achilles: Why does it have so many knobs and fancy dials?

 633Crab: So that you can tune it to the proper channel. There are many channels 
broadcasting in the subjunctive mode, and you want to be able to select from them easily. 
Achilles: Could you show us what you mean? I'm afraid I don't quite understand what all 
this talk of "broadcasting in the subjunctive mode" is about. 
Crab: Oh, it's quite simple, really. You can figure it out yourself. I'm going into the 
kitchen to fix some French fries, which I know are Mr. Sloth's weakness. 
Sloth: M mmmm! Go to it,  Crab! French fries are my favorite food. Crab: What about the 
rest of you? 
Tortoise: I could devour a few. Achilles: Likewise. But wait-before you go into the kitchen, is there some trick to using 
your Subjunc-TV? 
Crab: Not particularly. Just continue watching the game. and whenever there's a near

Crab: Not particularly. Just continue watching the game. and whenever there's a near 
miss of some sort, or whenever you wish things had gone differently in some way, just fiddle with the dials, and see what happens. You can't do it any harm, though you may pick up some exotic channels. ( And he disappears into the kitchen .) 
Achilles: I wonder what he means by that. Oh well, let's get back to this game. I was 
quite wrapped up in it. 
Announcer: It's fourth down for Out-of-Tow n, with Home Team receiving. Out-of-Town

Announcer: It's fourth down for Out-of-Tow n, with Home Team receiving. Out-of-Town 
is in punt formation, with Tedzilliger playing deep. Orwix is back to kick-and he gets a long high one away. It's coming down near Tedzilliger 
Achilles: Grab it, Tedzilliger! Give those Out-of-Towners a run for their money! Announcer: -and lands in a puddle-KERSPLOSH! It takes a weird bounce! Now Sprunk 
is madly scrambling for the ball! It looks like it just barely grazed Tedzilliger on the bounce, and then slipped away from himit's ruled a fumble. The referee is signaling that the formidable Sprunk has recovered for Out-of-Town on the Home Team 7! It's a

is madly scrambling for the ball! It looks like it just barely grazed Tedzilliger on the bounce, and then slipped away from himit's ruled a fumble. The referee is signaling that the formidable Sprunk has recovered for Out-of-Town on the Home Team 7! It's a 
bad break for Home Team. Oh, well, that's the way the cookie crumbles. 
Achilles: Oh, no! If only it hadn't been raining ... ( Wrings his hands in despair .) 
Sloth: ANOTHER of those confounded hypotheti cals! Why are the rest of you always 
running off into your absurd worlds of fantasy? If I were you, I would stay firmly grounded in reality. "No subjunctive nonsense" is my motto. And I wouldn't abandon it even if someone offered me a hundred-nay, a hundred and twelve-French fries. 
Achilles: Say, that gives me an idea. Maybe by suitably fiddling with these knobs, I can

Achilles: Say, that gives me an idea. Maybe by suitably fiddling with these knobs, I can 
conjure up a subjunctive instant replay in which it isn't raining, there's no puddle, no weird bounce, and Tedzilliger doesn't 
fumble. I wonder ... ( Walks up to the Subjunc-TV and stares at it.)  But I haven't any idea 
what these different knobs do. ( Spins a few at random .) 
Announcer: It's fourth down for Out- of-Town, with Home Team receiv-

 634ing. Out-of-Town is in punt formation, with Tedzilliger playing deep. Orwix is back to 
kick-and he gets a long high one away. It's coming down near Tedzilliger 
Achilles: Grab it, Tedzilliger! Give those Out-of-Towners a run for their money! Announcer: -and lands in a puddle-KERSPLOSH! Oh-it bounces right into his arms! 
Now Sprunk is madly scrambling after him, but he's got - good blocking, and he steers his way clear of the formidable Sprunk, and now he's got an open field ahead of him. Look at that, folks! He's to the 50, the 40, the 30, the 20, the 10-touchdown, Home Team! ( Huge cheers from the Home Team side .) Well, fans, that's how it would have 
gone, if footballs were spheres instead of oblate spheroids! But in reality, Home Team loses the ball, and Out-of-Towners take over on the Home Team 7-yard line. Oh, well, that's the way the ball bounces. 
Achilles: What do you think of THAT, Mr. Sloth?

Achilles: What do you think of THAT, Mr. Sloth?  
(And Achilles gives a smirk in the direction of the Sloth, but the latter is completely 
oblivious to its devastating effect, as he is busy watching ,the Crab arrive with, a large platter with a hundred and twelve-nay, a hundred-large and delicious French fries, and napkins for all.) 
 
Crab: So how do you three find my Subjunc-TV? Sloth: Most disappointing, Crab, to be quite fr ank. It seems to be badly out of order. It 
makes pointless excursions into nonsense at least half the time. If it belonged to me, I would give it away immediately to someone like you, Crab. But of course it doesn't belong to me. 
Achilles: It's quite a strange device. I tried to rerun a play to see how it would have gone

Achilles: It's quite a strange device. I tried to rerun a play to see how it would have gone 
under different weather conditions, but the thing seems to have a will of its own! Instead of changing the weather, it change d the football shape to ROUND instead of 
FOOTBALL-SHAPED! Now tell me-how can a f ootball not be shaped like a football? 
That's a contradiction in terms. How preposterous! 
Crab: Such tame games! I thought you'd surely find more interesting subjunctives. How 
would you like to see how the last play would have looked if the game had been baseball instead of football? 
Tortoise: Oh! An outstanding idea!

Tortoise: Oh! An outstanding idea! 
 
(The Crab twiddles two knobs, and steps back.)  Announcer: There are four away, and--- Achilles: FOUR away!? Announcer: That's right, fans-four away. When you turn football into baseball, 
SOMETHING'S got to give! Now as I was about to say, there are four away, with Out-of-Town in the field, and Home Team up. Tedzilliger is at bat. Out-of-Town is in bunt formation. Orwix raises his arm to pitch-and he gets a long high ball away. It's heading straight for Tedzilliger 
Achilles: Smash it, Tedzilliger! Give those Out-of-Towners a home run for their money!

 635Announcer: -but it seems to be a spitball, as it takes a strange curve. Now Sprunk is 
madly scrambling for the ball! It looks like it just barely grazed Tedzilliger's bat, then bounced off it-it's ruled a fly ball. The umpire is signaling that the formidable Sprunk has caught it for Out-of-Town, to end the seventh inning. It's a bad break for Home Team. That's how the last play would have looked, football fans, if this had been a game of baseball. 
Sloth: Bah! You might as well transport this game to the Moon. Crab: No sooner said than done! Just a twiddle here, a twiddle there ...  
(On the screen there appears a desolate crater-pitted field, with two teams in space

(On the screen there appears a desolate crater-pitted field, with two teams in space 
suits facing each other, immobile. All at once, the two teams fly into motion, and the players are making great bounds into the air, sometimes over the heads of other players. The ball is thrown into the air, and sails so high that it almost disappears, and then slowly comes floating down into the arms of one space-suited player, roughly a quarter-mile from where it was released.) 
 
Announcer: And there, friends, you have the subjunctive instant replay as it would have 
happened on the Moon. We'll be right back after this important commercial message from the friendly folks who brew Glumpf Beer-my favorite kind of beer! 
Sloth: If I weren't so lazy, I would take that broken TV back to the dealer myself! But

Sloth: If I weren't so lazy, I would take that broken TV back to the dealer myself! But 
alas, it's my fate to be a lazy Sloth ... ( Helps himself to a large gob of French fries .) 
Tortoise: That's a marvelous invention, Mr. Crab. May I suggest a hypothetical? Crab: Of course! Tortoise: What would that last play have looked like if space were four-dimensional? Crab: Oh, that's a complicated one, Mr. T, but I believe I can code it into the dials. Just a 
moment. 
 
(He steps up, and, for the first time, appears to be using the full power of the control 
panel of his Subjunc-TV, turning almost every knob two or three times, and carefully checking various meters. Then he steps back with a satisfied expression on his face.)

moment. 
 
panel of his Subjunc-TV, turning almost every knob two or three times, and carefully checking various meters. Then he steps back with a satisfied expression on his face.) 
 
I think this should do it.  Announcer: And now let's watch the subjunctive instant replay.  
(A confusing array of twisted pipes appears on the screen. It grows larger, then 
smaller, and for a moment seems to do something akin to rotation. Then it turns into a strange mushroom-shaped object, and back to a bunch of pipes. As it metamorphoses from this into other bizarre shapes, the announcer gives his commentary.) 
 
Tedzilliger's fading back to pass. He spots Palindromi ten yards outfield, and passes it to 
the right and outwards-it looks good! Palindromi's at the 35-yard plane, the 40, and he's tackled on his own

 63643-yard plane. And there you nave it, 3-L tans, as it would have looked if football 
were played in four spatial dimensions. 
Achilles: What is it you are doing, Mr. Crab, when you twirl these various dials on the 
control panel? 
Crab: I am selecting the proper subjunctive channel. You see, there are all sorts of 
subjunctive channels broadcasting simultaneously, and I want to tune in precisely that one which represents the kind of hypothetical which has been suggested. 
Achilles: Can you do this on any TV? Crab: No, most TV's can't receive subjunctive channels. They require a special kind of 
circuit which is quite difficult to make. 
Sloth: How do you know which channel is broadcasting what? Do you look it up in the 
newspaper? 
Crab: I don't need to know the channel's call letters. Instead, I tune it in by coding, in

Crab: I don't need to know the channel's call letters. Instead, I tune it in by coding, in 
these dials, the hypothetical situation which I want to be represented. Technically, this is called "addressing a channel by its counterfactual parameters". There are always a large number of channels broadcasting every conceivable world. All the channels which carry worlds that are "near" to each other have call letters that are near 
to each other, too. Tortoise: Why did you not have to turn the dials at all, the first time we saw a subjunctive 
instant replay? 
Crab: Oh, that was because I was tuned in to a channel which is very near to the Reality

Crab: Oh, that was because I was tuned in to a channel which is very near to the Reality 
Channel, but ever so slightly off. So every once in a while, it deviates from reality. It's nearly impossible to tune EXACTLY into the Reality Channel. But that's all right, because it's so dull. All their instant replays are straight! Can you imagine? What a bore! 
Sloth: I find the whole idea of Subjunc-TV's one giant bore. But perhaps I could change 
my mind, if I had some evidence that your machine here could handle an 
INTERESTING counterfactual. For example, how  would that last play have looked if 
addition were not commutative? 
Crab: Oh me, oh my! That change is a little too radical, I'm afraid, for this model. I 
unfortunately don't have a Superjunc-TV, which is the top of the line. Superjunc-TV's can handle ANYTHING you throw at them.

unfortunately don't have a Superjunc-TV, which is the top of the line. Superjunc-TV's can handle ANYTHING you throw at them. 
Sloth: Bah! Crab: But look-I can do ALMOST as well. Wouldn't you like to see how the last play 
would have happened if 13 were not a prime number? Sloth: No thanks! THAT doesn't make any sense! Anyway, if I were the last play, I'd be getting pretty tired of being trotted out time and again in new garb for the likes of you fuzzy-headed concept-slippers. Let's get on with the game! 
Achilles: Where did you get this Subjunc-TV, Mr. Crab? Crab: Believe it or not, Mr. Sloth and I went to a country fair the other evening, and it

Achilles: Where did you get this Subjunc-TV, Mr. Crab? Crab: Believe it or not, Mr. Sloth and I went to a country fair the other evening, and it 
was offered as the first prize in a lottery. Normally I don't indulge in such frivolity, but some crazy impulse grabbed me, and I bought one ticket.

 637Achilles: What about you Mr. Sloth? 
Sloth: I admit, I bought one, just to humor old Crab. Crab: And when the winning number was announced, I found, to my amazement, that I'd 
won the lottery! 
Achilles: Fantastic! I've never known anyone who won anything in a lottery before! Crab: I was flabbergasted at my good fortune. Sloth: Don't you have something else to tell us about that lottery, Crab? Crab: Oh, nothing much. It's just that my ticket number was 129. Now when they 
announced the winning number, it was 128 just one off. Sloth: So you see, he actually didn't win it at all. Achilles: He ALMOST won, though ... 
Crab: I prefer to say that I won it, you see. For I came so terribly close . . If my number 
had been only one smaller, I would have w on. Sloth: But unfortunately, Crab, a miss is 
as good as a mile. 
Tortoise: Or as bad. What about you, Mr. Sloth? What was your number: 
Sloth: Mine was 256-the next power of 2 a bove 128. Surely, that counts as a hit, if

Sloth: Mine was 256-the next power of 2 a bove 128. Surely, that counts as a hit, if 
anything does! I can't understand why, howev er, those fair officials-those UNfair 
officials-were so thickheaded about it. They refused to award me my fully deserved prize. Some other joker claimed HE deserved it, because his number was 128. 1 think my number was far closer than His, but you can't fight City Hall. 
Achilles: I'm all confused. If you didn't win the Subjunc-TV after all, Mr. Crab, then how

Achilles: I'm all confused. If you didn't win the Subjunc-TV after all, Mr. Crab, then how 
can we have been sitting here all afternoon watching it? It seems as if we ourselves have been living in some sort of hypothetical world that would have been, had circumstances just been ever so slightly different ... 
Announcer: And that, folks, was how the afternoon at Mr. Crab's would have been spent, 
had he won the Subjunc-TV. But since he didn't, the four friends simply spent a pleasant afternoon watching Home Team get creamed, 128-0. Or was it 256-0? Oh

had he won the Subjunc-TV. But since he didn't, the four friends simply spent a pleasant afternoon watching Home Team get creamed, 128-0. Or was it 256-0? Oh 
well, it hardly matters, in five-dimensional Plutonian steam hockey.

Artificial Intelligence:Prospects 638CHAPTER XIX 
 
Artificial Intelligence: 
Prospects 
 
"Almost" Situations and Subjunctives 
 
AFTER READING Contrafactus , a friend said to me, "My unc le was almost President of 
the U.S.!" "Really?" I said. "Sure," he rep lied, "he was skipper of the PT 108." (John F. 
Kennedy was skipper of the PT 109.) 
That is what Contrafactus  is all about. In everyday thought, we are constantly 
manufacturing mental variants on situations we face, ideas we have, or events that 
happen, and we let some features stay exactly the same while others "slip". What features 
do we let slip? What ones do we not even consider letting slip? What events are perceived on some deep intuitive level as be ing close relatives of ones which really 
happened? What do we think "almost" ha ppened or "could have" happened, even though

happened? What do we think "almost" ha ppened or "could have" happened, even though 
it unambiguously did not? What alternative versions of events pop without any conscious thought into our minds when we hear a story?  Why do some counterfactuals strike us as 
"less counterfactual" than other counterfactuals ? After all, it is obvious that anything that 
didn't happen didn't happen. There aren't degr ees of "didn't-happen-ness". And the same 
goes for "almost" situations. There are times  when one plaintively says, "It almost

goes for "almost" situations. There are times  when one plaintively says, "It almost 
happened", and other times when one says the same thing, full of relief. But the "almost" 
lies in the mind, not in the external facts. 
Driving down a country road, you run into  a swarm of bees. You don't just duly 
take note of it; the whole situation is immediately placed in perspective by a swarm of "replays" that crowd into your mind. Typi cally, you think, "Sure am lucky my window 
wasn't open!"-or worse, the reverse: "Too bad my window wasn't closed!" "Lucky I wasn't on my bike!" "Too bad I didn't come  along five seconds earlier." Strange but-

possible replays: "If that had been a deer, I could have been killed!" "I bet those bees 
would have rather had a collision with a rosebush." Even stranger replays: "Too bad 
those bees weren't dollar bills!" "Lucky those bees weren't made of cement!" "Too bad it wasn't just one bee instead of a swarm." "Luc ky I wasn't the swarm instead of being me." 
What slips naturally and what doesn't-and why? 
In a recent issue of The New Yorker  magazine, the following excerpt from the 
"Philadelphia Welcomat" was reprinted:'

"Philadelphia Welcomat" was reprinted:' 
 
If Leonardo da Vinci had been born a female the ceiling of the Sistine Chapel might never have been painted.
1

Artificial Intelligence:Prospects 639The New The New Yorker  commented: 
 And if Michelangelo had been Siames e twins, the work would have been 
completed in half the time.  
The point of The New Yorker's  comment is not that such counterfactuals are false; it is 
more that anyone who would entertain such an idea-anyone who would "slip" the sex or 
number of a given human being-would have to be a little loony. Ironically, though, in the 
same issue, the following sentence, concluding a book review, was printed without blushing: 
 
I think he [Professor Philipp Frank w ould have enjoyed both of these books 
enormously.2 
 
Now poor Professor Frank is dead; and clearly  it is nonsense to suggest that someone 
could read books written after his death. So why wasn't this serious sentence also scoffed 
at? Somehow, in some difficult-to-pin-down sense, the parameters slipped in this 
sentence do not violate our sense of "possib ility" as much as in the earlier examples.

sentence do not violate our sense of "possib ility" as much as in the earlier examples. 
Something allows us to imagine "all other thi ngs being equal" better in this one than in 
the others. But why? What is it about the wa y we classify events and people that makes 
us know deep down what is "sensible" to slip, and what is "silly": 
Consider how natural it feel s to slip from the valueless declarative "I don't know 
Russian" to the more charged conditional "I would like to know Russian" to the emotional subjunctive "I wish I knew Russian" a nd finally to the rich counterfactual "If I

Russian" to the more charged conditional "I would like to know Russian" to the emotional subjunctive "I wish I knew Russian" a nd finally to the rich counterfactual "If I 
knew Russian, I would read Chekhov and Lermont ov in the original". How flat and dead 
would be a mind that saw nothing in a nega tion but an opaque barrier! A live mind can 
see a window onto a world of possibilities. 
I believe that "almost" situations a nd unconsciously manufactured subjunctives 
represent some of the richest potential sources of insight into how human beings organize 
and categorize their perceptions of the world. An eloquent co-proponent of this view is the linguist and translator George

and categorize their perceptions of the world. An eloquent co-proponent of this view is the linguist and translator George 
Steiner, who, in his book After Babel , has written: 
 
Hypotheticals, 'imaginaries', conditionals , the syntax of counter-factuality and 
contingency may well be the generative cen tres of human speech.... [They] do more 
than occasion philosophical and grammatical perplexity. No less than future tenses to which they are, one feels, related, and with which they ought probably to be 
classed in the larger set of 'suppositionals' or `alternates', these `if' propositions are 
fundamental to the dynamics of human feeling... .

fundamental to the dynamics of human feeling... . 
 
Ours is the ability, the need, to gainsay or  'un-say' the world, to image and speak it 
otherwise.... We need a word which will designate the power, the compulsion of 
language to posit 'otherness'. . . . Perhaps 'alte rnity' will do: to define the `other than 
the case', the counter-factual propositions, im ages, shapes of will and evasion with 
which we charge our mental being and by means of which we build the changing, 
largely fictive milieu of our somatic and our social existence... .

largely fictive milieu of our somatic and our social existence... . 
 
Finally, Steiner sings a counterfactual hymn to counterfactuality:

Artificial Intelligence:Prospects 640 
It is unlikely that man, as we know him, would have survived without the fictive, counter-factual, anti-determinist means of  language, without the semantic capacity, 
generated and stored in the `superfluous, zones of the cortex, to conceive of, to articulate 
possibilities beyond the treadmill of organic decay and death .
3 
 
The manufacture of "subjunctive worlds" happens so casually, -so naturally, that 
we hardly notice what we are doing. We select  from our fantasy a world which is close, 
in some internal mental sense, to the real world. We compare what is real with what we 
perceive as almost real. In so doing, what we gain is some intangible kind of perspective 
on reality. The Sloth is a droll example of a variation on reality-a thinking being without 
the ability to slip into subjunctives (or at l east, who claims to be without the ability-but 
you may have noticed that what he says is full of counterfactuals'.). Think how

you may have noticed that what he says is full of counterfactuals'.). Think how 
immeasurably poorer our mental lives would be  if we didn't have this creative capacity 
for slipping out of the midst of reality into soft  "what if'-s! And from the point of view of 
studying human thought processes, this slippage is very interesting, for most of the time it 
happens completely without conscious direction, which means that observation of what kinds of things slip, versus what kinds  don't, affords a good window on the unconscious 
mind. 
One way to gain some perspective on the natu re of this mental metric is to "fight

mind. 
One way to gain some perspective on the natu re of this mental metric is to "fight 
fire with fire". This is done in the Dialogue, where our "subjunctive ability" is asked to imagine a world in which the very notion of subjunctive ability is slipped, compared to wh at we expect. In the Dialogue, the first 
subjunctive instant replay-that where Palindr omi stays in bounds-is quite a normal thing 
to imagine. In fact, it was inspired by a completely ordinary, casual remark made to me by a person sitting next to me at a football game. For some reason it struck me and I

to imagine. In fact, it was inspired by a completely ordinary, casual remark made to me by a person sitting next to me at a football game. For some reason it struck me and I 
wondered what made it seem so natural to slip that particular thing, but not, say, the 
number of the down, or the present score. Fr om those thoughts, I went on to consider 
other, probably less slippable features, such as the weather (that's in the Dialogue), the 
kind of game (also in the Dialogue), and then even loonier variations (also in the Dialogue). I noticed, though, that what was completely ludicrous to slip in one situation could be quite slippable in another. For instance, sometimes you might spontaneously

kind of game (also in the Dialogue), and then even loonier variations (also in the Dialogue). I noticed, though, that what was completely ludicrous to slip in one situation could be quite slippable in another. For instance, sometimes you might spontaneously 
wonder how things would be if the ball ha d a different shape (e .g., if you are playing 
basketball with a half-inflated ball); other times that would never enter your mind (e.g., 
when watching a football game on TV).  
Layers of Stability 
 It seemed to me then, and still does now, that the slippability of a feature of some event 
(or circumstance) depends on a set of ne sted contexts in which the event (or 
circumstance) is perceived to occur. The terms constant, parameter, and variable,

circumstance) is perceived to occur. The terms constant, parameter, and variable, 
borrowed from mathematics, seem useful here. Often mathematicians, physicists, and others will carry out a calculation, saying "c is  a constant, p is a parameter, and v is a 
variable". What they

Artificial Intelligence:Prospects 641mean.is that any of them can vary (includi ng the "constant"); however, there is a kind of 
hierarchy of variability. In the situation whic h is being represented b~ the symbols, c 
establishes a global condition; p establishe s some less global condition which can vary 
while c is held fixed; and fi nally, v can run around while c and p are held fixed. It makes 
little sense to think of holding v fixed while c and p vary, for c and p establish the context 
in which v has meaning. For instance, think of a dentist who has a list of patients, and for 
each patient, a list of teeth. It makes perfect  sense (and plenty of money) to hold the 
patient fixed and vary his teeth-but it makes no sense at all to hol d one tooth fixed and 
vary the patient. (Although sometimes it makes good sense to vary the dentist ...) 
We build up our mental representation of a situation layer by layer. The lowest

We build up our mental representation of a situation layer by layer. The lowest 
layer establishes the deepest aspect of the context-sometimes being so low that it cannot 
vary at all. For instance, the three-dimensiona lity of our world is so ingrained that most 
of us never would imagine letting it slip mentall y. It is a constant constant. Then there are 
layers which establish temporarily, though not permanently, fixed aspects of situations, which could be called background assumptions -things which, in the back of your mind, 
you know can vary, but which most of  the time you unquestioningly accept as

you know can vary, but which most of  the time you unquestioningly accept as 
unchanging aspects. These could still be calle d "constants". For instance, when you go to 
a football game, the rules of the game are constants of that sort. Then there are 
"parameters": you think of them as more variable, but you temporarily hold them constant. At a football game, parameters mi ght include the weather, the opposing team, 
and so forth. There could be-and probably are- several layers of parameters. Finally, we 
reach the "shakiest" aspects of your mental re presentation of the situation-the variables.

reach the "shakiest" aspects of your mental re presentation of the situation-the variables. 
These are things such as Palindromi's ste pping out of bounds, which are mentally "loose" 
and which you don't mind letting slip away from their real values, for a short moment. 
 
Frames and Nested Contexts 
 The word frame is in vogue in Al currently, and it could be defined as a computational 
instantiation of a context. The term is due  to Marvin Minsky, as are many ideas about 
frames, though the general concept has been floating around for a good number of years. In frame language, one could say that mental re presentations of situations involve frames

frames, though the general concept has been floating around for a good number of years. In frame language, one could say that mental re presentations of situations involve frames 
nested within each other. Each of the various ingredients of a situation has its own frame. 
It is interesting to verbalize explicitly one of my mental images concerning nested frames. Imagine a large collection of chests  of drawers. When you choose a chest, you 
have a frame, and the drawer holes are pl aces where "subframes" can be attached. But 
subframes are themselves chests of drawers.  How can you stick a whole chest of drawers 
into the slot for a single drawer in another chest of drawers? Easy: you shrink and distort

into the slot for a single drawer in another chest of drawers? Easy: you shrink and distort 
the second chest, since, after a ll, this is all mental, not physical. Now in the outer frame, 
there may be several different drawer slots that need to be

Artificial Intelligence:Prospects 642filled; then you may need to fill slots in some of the inner chests of drawers (or 
subframes). This can go on, recursively.  
The vivid surrealistic image of squishing and bending a chest of drawers so that it 
can fit into a slot of arbitrary shape is pr obably quite important, because it hints that your 
concepts are squished and bent by the contex ts you force them into. Thus, what does your 
concept of "person" ', become when th e people you are thinking about are football 
players? It certainly is a distorted con cept, one which is forced on you by the overall 
context. You have stuck the "person" frame into a slot in the "football game" frame. The theory of representing knowledge in frames re lies on the idea that the world consists of 
quasi-closed subsystems, each of which can se rve as a context for others without being 
too disrupted, or creating too much disruption, in the process.

too disrupted, or creating too much disruption, in the process. 
One of the main ideas about frames is th at each frame comes with its own set of 
expectations. The corresponding image is that each chest. of drawer s comes with a built-
in, but loosely bound, drawer in each of its drawer slots, called a default. If I tell you, "Picture a river bank", you will invoke a visual 
image which has various features, most of  which you could override if I added extra 
phrases such as "in a drought" or "in Br azil" or "without a merry-go-round". The

phrases such as "in a drought" or "in Br azil" or "without a merry-go-round". The 
existence of default values for slots allows th e recursive process of filling slots to come to 
an end. In effect, you say, "I will fill in the slots myself as far as three layers down; 
beyond that I will take the default options." Together with its default expectations, a 
frame contains knowledge of its limits of applicability, and heuristics for switching to 
other frames in case it has been stretched beyond its limits of tolerance. 
The nested structure of a frame gives you a way of "zooming in" and looking at

The nested structure of a frame gives you a way of "zooming in" and looking at 
small details from as close up as you wish: you just zoom in on the proper subframe, and 
then on one of its subframes, etc., until you ha ve the desired amount of detail. It is like 
having a road atlas of the USA which has a ma p of the whole country in the front, with 
individual state maps inside, and even maps of  cities and some of the larger towns if you 
want still more detail. One can imagine an atlas with arbitrary amounts of detail, going 
down to single blocks, houses, rooms, etc. It is like looking through a telescope with

down to single blocks, houses, rooms, etc. It is like looking through a telescope with 
lenses of different power; each lens has its ow n uses. It is important that one can make 
use of all the different scales; often detail is irrelevant and even distracting. 
Because arbitrarily different frames can be  stuck inside other frames' slots, there 
is great potential for conflict or "collision". Th e nice neat scheme of a single, global set of 
layers of "constants", "parameters", and "varia bles" is an oversimplification. In fact, each 
frame will have its own hierarchy of variabilit y, and this is what makes analyzing how we

frame will have its own hierarchy of variabilit y, and this is what makes analyzing how we 
perceive such a complex event as a f ootball game, with its many subframes, 
subsubframes, etc., an incredibly messy ope ration. How do all these many frames interact 
with each other? If there is a conflict where one frame says, "This item is a constant" but 
another frame says, "No, it is a variable!", how does it get resolved ? These are deep and 
difficult problems of frame theory to

Artificial Intelligence:Prospects 643which I can give no answers. There has as yet been no complete agreement on what a 
frame really is, or on how to implement frames in Al programs. I make my own stab at discussing some of these questions in the following section, where I talk about some 
puzzles in visual pattern recognition, which I call "Bongard problems".  
Bongard Problems 
 
Bongard problems  (BP's) are problems of the general type given by the Russian scientist 
M. Bongard in his book Pattern Recognition . A typical BP-number 51 in his collection of 
one hundred-is shown in Figure 119.  
 
FIGURE 119. Bongard problem 51. From R1. Bongard, Pattern Recognition (Rochelle 
Park, N.,J.: Hayden Book Co., Spartan Books, 1970).] 
 
These fascinating problems are intended fo r pattern-recognizers, whether human or 
machine. (One might also throw in ETI's-e xtraterrestrial intelligences.) Each problem 
consists of twelve boxed figures (hencefor th called boxes): six on the left, forming Class

consists of twelve boxed figures (hencefor th called boxes): six on the left, forming Class 
I, and six on the right, forming Class II.  The boxes may be indexed this way: 
 
I-A    I-B  II-A II-B I-C I-D  II-C II-D I-E I-F  II-E II-F 
 The problem is "How do Class I boxes differ from Class II boxes?" 
A Bongard problem-solving program would have several stages, in which raw 
data gradually get converted into descriptions . The early stages are relatively inflexible, 
and higher stages become gradually more flex ible. The final stages have a property which 
I call tentativity , which means simply that the way a picture is represented is always

I call tentativity , which means simply that the way a picture is represented is always 
tentative. Upon the drop of a hat, a high-leve l description can be restructured, using all 
the devices of the

Artificial Intelligence:Prospects 644 
later stages. The ideas presented below also ha ve a tentative quality to them. I will try to 
convey overall ideas first, glossing over signi ficant difficulties. Then I will go back and 
try to explain subtleties and tricks and so forth. So your notion of how it all works may also undergo some revisions as you read. But that is in the spirit of the discussion.  
Preprocessing Selects a Mini-vocabulary 
 Suppose, then, that we have some Bongard problem which we want to solve. The problem is presented to a TV camera and the raw data are read in. Then the raw data are preprocessed. This means that some salient features are detected. The names of these features constitute a "mini-vocabulary" for the problem; they are drawn from a general "salient-feature vocabulary". Some typical te rms of the salient-feature vocabulary are: 
 
line segment, curve, horizontal, vertical, black, white, big, small, pointy, round ...

line segment, curve, horizontal, vertical, black, white, big, small, pointy, round ...  
In a second stage of preprocessing, some knowledge about elementary shapes is used; and if any are found, their names are also made available. Thus, terms such as  
triangle, circle, square, indentation, prot rusion, right angle, vertex, cusp, arrow ... 
 
may be selected. This is roughly the point at which the conscious and the unconscious 
meet, in humans. This discussion is primar ily concerned with describing what happens 
from here on out.  
High-Level Descriptions 
 Now that the picture is "understood", to some extent, in terms of familiar concepts, some looking around is done. Tentative descriptions ar e made for one or a few of the twelve 
boxes. They will typically use simple descriptors such as  
above, below, to the right of, to the left of, inside, outside of , close to, far from, 
parallel to, perpendicular to, in a row, scattered, evenly spaced, irregularly spaced, etc.

parallel to, perpendicular to, in a row, scattered, evenly spaced, irregularly spaced, etc. 
 
Also, definite and indefinite numerical descriptors can be used: 
 1, 2, 3, 4, 5, ... many, few, etc. 
 More complicated descriptors may be built up, such as 
 further to the right of, less close to, almost parallel to, etc.

Artificial Intelligence:Prospects 645
 
FIGURE 120. Bongard problem 47. [From M. Bongard, Pattern Recognition.]  
 
Thus, a typical box-say I-F of BP 47 (Fig. 120 )-could be variously described as having: 
 
three shapes 
or 
three white shapes 
or 
a circle on the right 
or 
two triangles and a circle 
or 
two upwards-pointing triangles 
or 
one large shape and two small shapes 
or 
one curved shape and two straight-edged shapes 
or 
a circle with the same kind of shape on the inside and outside. 
 
Each of these descriptions sees the box thr ough a "filter". Out of context, any of them 
might be a useful description. As it turns out, though, all of them are "wrong", in the 
context of the particular Bongard problem th ey are part of. In other words, if you knew 
the distinction between Classes I and II in BP 47, and were given one of the preceding lines as a description of an unseen drawing, that information would not allow you to tell

the distinction between Classes I and II in BP 47, and were given one of the preceding lines as a description of an unseen drawing, that information would not allow you to tell 
to which Class the drawing belonged. The essentia l feature of this box, in context, is that 
it includes  
a circle containing a triangle. 
 
Note that someone who heard such a description would not be able to reconstruct the original drawing, but would be able to recognize drawings

Artificial Intelligence:Prospects 646
 
FIGURE 121 . Bongard problem 91. [From M. Bongard, Pattern Recognition.]  
 which have this property. It is a little lik e musical style: you may be an infallible 
recognizer of Mozart, but at the same time unable to write anything which would fool 
anybody into thinking it was by Mozart. 
Now consider box I-D of BP 91 (Fig. 121). An overloaded but "right" description 
in the context of BP 91 is  
a circle with three rectangular intrusions. 
 Notice the sophistication of such a descripti on, in which the word "with" functions as a 
disclaimer, implying that the "circle" is not rea lly a circle: it is almost a circle, except that 
. . . Furthermore, the intrusions are not full r ectangles. There is a lot of "play" in the way 
we use language to describe 
things. Clearly, a lot of information has been thrown away, and even more could 
be thrown away. A priori, it is very hard to  know what it would be smart to throw away

Notice the sophistication of such a descripti on, in which the word "with" functions as a 
be thrown away. A priori, it is very hard to  know what it would be smart to throw away 
and what to keep. So some sort of met hod for an intelligent compromise has to be 
encoded, via heuristics. Of course, there is alwa ys recourse to lower levels of description 
(i.e., less chunked descriptions) if discarded information has to be retrieved, just as 
people can constantly look at the puzzle for help  in restructuring their ideas about it. The 
trick, then, is to devise explicit rules that say how to

trick, then, is to devise explicit rules that say how to 
 
make tentative descriptions for each box; compare them with tentative descriptions for other boxes of either Class; restructure the descriptions, by 
(i) adding information, (ii) discarding information, 
or (iii) viewing the same information from another angle; iterate this process until finding out what makes the two Classes differ.

Artificial Intelligence:Prospects 647Templates and Sameness-Detectors 
 
One good strategy would be to try to make descriptions structurally similar to 
each other , to the extent this is possible. Any structure they have in common will 
make comparing them that much easier. Two important elements of this theory 
deal with this strategy. One is th e idea of "description-schemas" or templates ; the 
other is the idea of Sam-a "sameness detector". 
First Sam. Sam is a special agent present on all levels of the program. 
(Actually there may be different kinds  of Sams on different levels.) Sam 
constantly runs around within individua l descriptions and within different 
descriptions, looking for descriptors or other things which are repeated. When

descriptions, looking for descriptors or other things which are repeated. When 
some sameness is found, various restructuring operations can be triggered, either on the single-description level or on the level of several descriptions at once. Now templates. The first thing that happens after preprocessing is an attempt to manufacture a template, or descripti on-schema-a un form format for the 
descriptions of all the boxes in a problem. The idea is that a description can often 
be broken up in a natural way into subdesc riptions, and those in turn into subs 
ubdescriptions, if need be. The bottom is hit when you come to primitive concepts

ubdescriptions, if need be. The bottom is hit when you come to primitive concepts 
which belong to the level of the preproce ssor. Now it is important to choose the 
way of breaking descriptions into parts so as to reflect commonality among all the 
boxes; otherwise you are introducing a superfluous and meaningless kind of "pseudo-order" into the world. 
On the basis of what information is a te mplate built? It is best to look at an 
example. Take BP 49 (Fig. 122). Preproce ssing yields the information that each 
box consists of several little o's, and one la rge closed curve. This is a valuable 
observation, and deserves to be incorporated in the template. Thus a first stab at a 
template would be: 
 
large closed curve:-----  small o's:-----  FIGURE 122. Bongard problem 49. [From M. Bongard, Pattern Recognition.]

Artificial Intelligence:Prospects 648It is very simple: the description-template has two explicit slots where 
subdescriptions are to be attached.  
A Heterarchical Program 
 Sow an interesting thing happens, triggere d by the term "closed curve". one of the 
most important modules in the program is a kind of semantic net--the concept network-in which all the known nouns, adjec tives, etc., are linked in ways which 
indicate their interrelations. For instance, "closed curve" is strongly linked with 
the terms "interior" and "exterior". Th e concept net is just brimming with 
information about relations between terms,  such as what is the opposite of what, 
what is similar to what, what often occu rs with what, and so on. A little portion of 
a concept network, to be explained shor tly, is shown in Figure 123. But let us 
follow what happens now, in the solution of problem 49. The concepts "interior" 
and "exterior" are activated by their proxim ity in the net to "closed curve". This

and "exterior" are activated by their proxim ity in the net to "closed curve". This 
suggests to the template-builder that it mi ght be a good idea to make distinct slots 
for the interior and exterior of the curv e. Thus, in the spirit of tentativity, the 
template is tentatively restructured to be this:  
large closed curve: ---- little o's in interior: ---- little o's in exterior:---- 
 Now when subdescriptions are sought, the terms "interior" and "exterior" will 
cause procedures to inspect those specific regions of the box. What is found in BP 49, box I-A is this:  
large closed curve: circle

large closed curve: circle   
little o's in interior: three  
little o's in exterior: three 
 And a description of box II-A of the same BP might be  
large closed curve: cigar   
little o's in interior: three   
little o's in exterior: three  
 
Now Sam, constantly active in para llel with other operations, spots the 
recurrence of the concept "three" in all the slots dealing with o's, and this is strong 
reason to undertake a second template-restructuring operation. Notice that the first was suggested by the concept net, the second by Sam. Now our template for 
problem 49 becomes: 
 
large closed curve:----

large closed curve:---- 
three little o's in interior: ----- three little o's in exterior:-----

Artificial Intelligence:Prospects 649
 
FIGURE 123. A small portion of a concept network for a program to solve 
Bongard Problems. "Nodes" are joined by " links", which in turn can be linked. By 
considering a link as a verb and the nodes it joins as subject and object, you can pull out some English sentences from this diagram.

Artificial Intelligence:Prospects 650Now that "three" has risen" one level of  generality-namely, into the template-it 
becomes worthwhile to explore its neighbor s in the concept network. One of them 
is "triangle", which suggests that triangles  of o's may be important. As it happens, 
this leads down a blind alley-but how coul d you know in advances It is a typical 
blind alley that a human would explore, so it is good if our program finds it too! 
For box II-E, a description such as the following might get generated:  large closed curve: circle  
three little o's in interior: equilateral triangle   
three little o's in exterior: equilateral triangle  
 Of course an enormous amount of inform ation has been thrown away concerning 
the sizes, positions, and orientations of th ese triangles, and many other things as 
well. But that is the whole point of maki ng descriptions instead of just using the 
raw data! It is the same idea as funneling, which we discussed in Chapter XI.  
The Concept Network

The Concept Network 
 We need not run through the entire solution of problem 49; this suffices to show 
the constant back-and-forth interaction of  individual descriptions, templates, the 
sameness-detector Sam, and the con cept network. We should now look a little 
more in detail at the concept network and its function. A simplified portion shown 
in the figure codes the following ideas:

in the figure codes the following ideas:  
"High" and "low" are opposites. "Up" and "down" are opposites.  "High" and "up" are similar.  "Low" and "down" are similar.  "Right" and "left" are opposites. The "right-left" distinction is similar to the "high-low" distinction.  "Opposite" and "similar" are opposites. 
 Note how everything in the net-both nodes a nd links-can be talked about. In that 
sense nothing in the net is on a higher leve l than anything else. Another portion of 
the net is shown; it codes for the ideas that

the net is shown; it codes for the ideas that  
A square is a polygon. A triangle is a polygon. A polygon is a closed curve.

Artificial Intelligence:Prospects 651The difference between a triangle and a square is that one has 3 sides and the 
     other has 4. 4 is similar to 3. A circle is a closed curve. A closed curve has an interior and an exterior.  "Interior" and "exterior" are opposites. 
 The network of concepts is necessarily ve ry vast. It seems to store knowledge only 
statically, or declaratively, but that is only half the story. Actually, its knowledge borders on being procedural as well, by the fact that the proximities in the net act as guides, or "programs", telling th e main program how to develop its 
understanding of the drawings in the boxes. For instance, some early hunch may turn out  to be wrong and yet have the germ of 
the right answer in it. In BP 33 (Fig. 124), one might at first  
 
FIGURE 124 . Bongard problem 33. [From M. Bongard, Pattern Recognition.] 
  jump to the idea that Class I boxes contai n "pointy" shapes, Class II boxes contain

  jump to the idea that Class I boxes contai n "pointy" shapes, Class II boxes contain 
"smooth" ones. But on closer inspection, this is wrong. Nevertheless, there is a 
worthwhile insight here, and one can try to push it further, by sliding around in the 
network of concepts beginning at "pointy" . It is close to the concept "acute", 
which is precisely the distinguishing feat ure of Class I. Thus one of the main 
functions of the concept network is to allow early wrong ideas to be modified 
slightly, to slip into variations which may be correct.  
Slippage and Tentativity 
 Related to this notion of slipping between  closely related terms is the notion of 
seeing a given object as a variation on a nother object. An excellent example has

seeing a given object as a variation on a nother object. An excellent example has 
been mentioned already-that of the "circle with three indentations", where in fact 
there is no circle at all. One has to be ab le to bend concepts, when it is appropriate. 
Nothing should be absolutely rigid. On

Artificial Intelligence:Prospects 652the other hand, things shouldn't be so wi shy-washy that nothing has any meaning 
at all, either. The trick is to know when and how to slip one concept into another. 
An extremely interesting set of examples where slipping from one description 
to another is the crux of the matter is given in Bongard problems 85-87 (Fig. 125). 
BP 85 is rather trivial. Let us assume th at our program identifies "line segment" in 
its preprocessing stage. It is relatively simp le for it then to count line segments and 
arrive at the difference 
 
FIGURE 125. Bongard problems 85-87. [From M. Bongard, Pattern Recognition .]

Artificial Intelligence:Prospects 653between Class I and Class II in BP 85. Now it goes on to BP 86. A general 
heuristic which it uses is to try out r ecent ideas which have worked. Successful 
repetition of recent methods is very common in the real world, and Bongard does not try to outwit this kind of heuristic in his collection-in fact, he reinforces it, fortunately. So we plunge right into probl em 86 with two ideas ("count" and "line 
segment") fused into one: "count line segments". But as it happens, the trick of BP 86 is to count line trains rather than line segments, where "line train" means an end-to-end concatenation of (one or more) line segments. One way the program might figure this out is if the concepts "line train" and "line segment" are both 
known, and are close in the concept networ k. Another way is if it can invent the 
concept of "line train"-a tricky proposition, to say the least. 
Then comes BP 87, in which the notion of  "line segment" is further played

Then comes BP 87, in which the notion of  "line segment" is further played 
with. When is a line segment three lin e segments? (See box II-A.) The program 
must be sufficiently flexible that it can go back and forth between such different 
representations for a given part of a drawi ng. It is wise to store old representations, 
rather than forgetting them and perhaps ha ving to reconstruct them, for there is no 
guarantee that a newer representation is better than an old one. Thus, along with each old representation should be stored some of the reasons for liking it and 
disliking it. (This begins to sound rather complex, doesn't it?)  
Meta- Descriptions 
 Now we come to another vital part of th e recognition process, and that has to do 
with levels of abstraction and meta-descr iptions. For this let us consider BP 91

with levels of abstraction and meta-descr iptions. For this let us consider BP 91 
(Fig. 121) again. What kind of template c ould be constructed here? There is such 
an amount of variety that it is hard to know  where to begin. But this is in itself a 
clue! The clue says, namely, that the class distinction very likely exists on a higher level of abstraction than that of geomet rical description. This observation clues the 
program that it should construct descriptions of descriptions -that is, meta -
descriptions . Perhaps on this second level some common feature will emerge; and

descriptions . Perhaps on this second level some common feature will emerge; and 
if we are lucky, we will discover enough commonality to guide us towards the formulation of a template for the meta-d escriptions! So we plunge ahead without a

if we are lucky, we will discover enough commonality to guide us towards the formulation of a template for the meta-d escriptions! So we plunge ahead without a 
template, and manufacture descriptions for various boxes; then, once these descriptions have been made, we describe them. What kinds of slot will our template for meta-descriptions have? Perhaps these, among others:  concepts used: ---- recurring concepts-----:  names of slots: ----- filters used:----  There are many other kinds of slots which might be needed in metadescriptions, 
but this is a sample. Now suppose we have described box I-E of BP 91. Its 
(template-less) description might look like this:

Artificial Intelligence:Prospects 654horizontal line segment 
vertical line segment mounted on the horizontal line segment  vertical line segment mounted on the horizontal line segment  vertical line segment mounted on the horizontal line segment 
 Of course much information has been thrown out: the fact that the three vertical lines are of the same length, are spaced equi distantly, etc. But it is plausible that 
the above description would be made. So the meta description might look like this: 
 
concepts used: vertical-horizontal, line segment, mounted on  
repetitions in description: 3 copies of "vertical line segment mounted on the          
horizontal line segment" 
names of slots----- 
filters used:----- 
 Not all slots of the meta-description need be filled in; information can be thrown 
away on this level as well as on the Just- plain-description" level. ‘Now if we were to make a description for any of the other boxes of Class I, and

away on this level as well as on the Just- plain-description" level. ‘Now if we were to make a description for any of the other boxes of Class I, and 
then a meta-description of it, we woul d wind up filling the slot "repetitions in 
description" each time with the phrase "3  copies of ..." The sameness-detector 
would notice this, and pick up three-ness as  a salient feature, on quite a high level 
of abstraction, of the boxes of Class I. Similarly, four-ness would be recognized, 
via the method of metadescriptions, as the mark of Class II.  
Flexibility Is Important

Flexibility Is Important 
 Now you might object that in this case, resorting to the method of metadescriptions is like shooting a fly with an elephant gun, for the three-ness 
versus four-ness might as easily have shown up on the lower level if we had constructed our descriptions slightly differently. Yes, true-but it is important to have the possibility of solving these probl ems by different routes. There should be

versus four-ness might as easily have shown up on the lower level if we had constructed our descriptions slightly differently. Yes, true-but it is important to have the possibility of solving these probl ems by different routes. There should be 
a large amount of flexibility in the program; it should not be doomed if, malaphorically speaking, it "barks up the wrong alley" for a while. (The amusing term "malaphor" was coined by the newspaper columnist Lawrence Harrison; it means a cross between a malapropism and a metaphor. It is a good example of "recombinant ideas".) In any case, I wanted to illustrate the general principle that 
says: When it is hard to build a template  because the preprocessor finds too much

says: When it is hard to build a template  because the preprocessor finds too much 
diversity, that should serve as a clue that  concepts on a higher level of abstraction 
are involved than the preprocessor knows about.  
Focusing and Filtering 
 Now let us deal with another question: ways to throw information out. This involves two related notions, which I call "focusing" and "filtering". Focus -

Artificial Intelligence:Prospects 655
 
FIGURE 126. Bongard problem 55. [From M. Bongard, Pattern Recognition .] 
 
FIGURE 127. Bongard problem 22. [From M. Bongard, Pattern Recognition .] 
 ing involves making a description whose focus is some part of the drawing in the 
box, to the exclusion of everything else. Filtering involves making a description which concentrates on some particular wa y of viewing the contents of the box, and 
deliberately ignores all other aspects. Thus they are complementary: focusing has 
to do with objects (roughly, nouns), and filte ring has to do with concepts (roughly, 
adjectives). For an example of focusing, let's look at BP 55 (Fig. 126). Here, we 
focus on the indentation and the little circle next to it, to the exclusion of the everything else in the box. BP 22 (Fig. 127) presents an example of filtering. Here, 
we must filter out every concept but that  of size. A combination of focusing and

we must filter out every concept but that  of size. A combination of focusing and 
filtering is required to solve problem BP 58 (Fig. 128). One of the most important ways to ge t ideas for focusing and filtering is by 
another sort of "focusing": namely, by inspection of a single particularly simple box-say one with as few objects in it as possible. It can be

Artificial Intelligence:Prospects 656
 
FIGURE 128. Bongard problem 58. [From M. Bongard, Pattern Recognition .] 
 
FIGURE 129. Bongard problem 61. [From M. Bongard, Pattern Recognition.] 
 extremely helpful to compare the starke st boxes from the two Classes. But how 
can you tell which boxes are stark until you have descriptions for them? Well, one way of detecting starkness is to look fo r a box with a minimum of the features 
provided by the preprocessor. This can be done very early, for it does not require a 
pre-existing template; in fact, this can be one useful way of discovering features to 
build into a template. BP 61 (Fig. 129) is  an example where that technique might 
quickly lead to a solution.  
Science and the World of Bongard Problems 
 One can think of the Bongard-problem wo rld as a tiny place where "science" is 
done-that is, where the purpose is to discer n patterns in the world. As patterns are 
sought, templates are made, unmade, and remade;

Artificial Intelligence:Prospects 657
 
FIGURE 130. Bongard problems 70-71. [From M. Bongard, Pattern 
Recognition .] 
 slots are shifted from one level of genera lity to another: filtering and focusing are 
done; and so on. There are discoveries on a ll levels of complexity. The Kuhnian 
theory that certain rare events called "paradigm shifts" mark the distinction 
between "normal" science and "conceptual  revolutions" does not seem to work, 
for we can see paradigm shifts happening all throughout the system, all the time. The fluidity of descriptions ensures th at paradigm shifts will take place on all 
scales. 
Of course, some discoveries are more "revolutionary" than others, because 
they have wider effects. For instance, one can make the discovery that problems 
70 and 71 (Fig. 130) are "the same probl em", when looked at on a sufficiently 
abstract level. The key observation is th at both involve depth-2 versus depth-1 
nesting. This is a new level of discove ry that can he made about Bongard

scales. 
nesting. This is a new level of discove ry that can he made about Bongard 
problems. There is an even higher level, concerning the collection as a whole. If 
someone has never seen the collection, it can be a good puzzle just to figure out 
what it is. To figure it out is a revolutionary  insight, but it must be pointed out that 
the mechanisms of thought which allow such a discovery to be made are no different from those which operate in the solution of a single Bongard problem.

Artificial Intelligence:Prospects 658By the same token, real science does not divide up into "normal" periods versus 
"conceptual. revolutions"; rather, paradigm shifts pervade-there are just bigger and 
smaller ones, paradigm shifts on different levels. The recursive plots of INT and Gplot (Figs. 32 and 34) provide a geometric model for this idea: they have the same 
structure full of discontinuous jumps on ever y level, not just the top level-only the 
lower the level, the smaller the jumps  
Connections to Other Types of Thought 
 To set this entire program somewhat in contex t, let me suggest two ways in which it is 
related to other aspects of cognition. No t only does it depend on other aspects of 
cognition, but also they in turn depend on it. First let me comment on how it depends on other aspects of cognition. The intuition wh ich is required for knowing when it makes 
sense to blur distinctions, to try redescripti ons, to backtrack, to shift levels, and so forth,

sense to blur distinctions, to try redescripti ons, to backtrack, to shift levels, and so forth, 
is something which probably comes only with much experience in thought in general. Thus it would be very hard to define heuris tics for these crucial aspects of the program. 
Sometimes one's experience with real objects in  the world has a subtle effect on how one 
describes or redescribes boxes. For instance, who can say how much one's familiarity with living trees helps one to solve BP 70%  It is very doubtful that in humans, the 
subnetwork of concepts relevant to these pu zzles can be easily separated out from the

subnetwork of concepts relevant to these pu zzles can be easily separated out from the 
whole network. Rather, it is much more likel y that one's intuitions gained from seeing 
and handling real objects-combs, trains, stri ngs, blocks, letters, rubber bands, etc., etc.-
play an invisible but significant guiding role in the solution of these puzzles. 
Conversely, it is certain that understanding real-world situations heavily depends 
on visual imagery and spatial intuition, so th at having a powerful and flexible way of 
representing patterns such as these Bongard patterns can only contribute to the general 
efficiency of thought processes. It seems to me that Bongard's problems were worked out with great care, and that they

efficiency of thought processes. It seems to me that Bongard's problems were worked out with great care, and that they 
have a quality of universality to them, in the sense that each one has a unique correct 
answer. Of course one could argue with this  and say that what we consider "correct" 
depends in some deep way on our being human, and some creatures from some other star 
system might disagree entirely. Not having any concrete evidence either way, I still have 
a certain faith that Bongard problems depend on a sense of simplicity which is not just 
limited to earthbound human beings. My earlie r comments about the probable importance

limited to earthbound human beings. My earlie r comments about the probable importance 
of being acquainted with such surely earth- limited objects as combs, trains, rubber bands, 
and so on, are not in conflict with the idea th at our notion of simplicity is universal, for 
what matters is not any of these individual obj ects, but the fact that taken together they 
span a wide space. And it seems likely that a ny other civilization would have as vast a 
repertoire of artifacts and natural objects and varieties of experience on which to draw as 
we do. So I believe that the skill of solving Bongard

Artificial Intelligence:Prospects 659problems lies very close to the core of "pur e" intelligence, if there is such a thing. 
Therefore it is a good place to be gin if one wants to investigate the ability to discover 
"intrinsic meaning" in patterns or messages. Unfortunately we have reproduced only a small selection of his stimulating collec tion. I hope that many readers will acquaint 
themselves with the entire collection, to be found in his book (see Bibliography). 
Some of the problems of visual pattern recognition which we human beings seem 
to have completely "flattened" into our unconscious are quite amazing. They include:  
recognition of faces (invariance of faces under age change, expression change, 
lighting change, distance change, angle change, etc.) 
recognition of hiking trails in forests and mountains-somehow this has always 
impressed me as one of our most sub tle acts of pattern recognition-and yet 
animals can do it, too

animals can do it, too 
reading text without hesitation in hundreds if not thousands of different typefaces 
 
Message-Passing Languages, Frames, and Symbols 
 One way that has been suggested for handli ng the complexities of pattern recognition and 
other challenges to Al programs is the so-called "actor" formalism of Carl Hewitt (similar to the language "Smailtalk", developed by Alan Kay and others), in which a program is written as a collection of interacting actors,  which can pass elaborate messages back and

other challenges to Al programs is the so-called "actor" formalism of Carl Hewitt (similar to the language "Smailtalk", developed by Alan Kay and others), in which a program is written as a collection of interacting actors,  which can pass elaborate messages back and 
forth among themselves. In a way, this resemble s a heterarchical collection of procedures 
which can call each other. The major difference is that where procedures usually only pass a rather small number of arguments back and forth, the messages exchanged by actors can be arbitrarily long and complex. 
Actors with the ability to exchange messages become somewhat autonomous

Actors with the ability to exchange messages become somewhat autonomous 
agents-in fact, even like autonomous computers, with messages being somewhat like programs. Each actor can have its own idiosyncratic way of interpreting any given message; thus a message's meaning will depe nd on the actor it is intercepted by. This 
comes about by the actor having within it a pi ece of program which interprets messages; 
so there may be as many interpreters as th ere are actors. Of course, there may be many 
actors with identical interpreters; in fact, this  could be a great advantage, just as it is 
extremely important in the cell to have a multitude of identical ribosomes floating

extremely important in the cell to have a multitude of identical ribosomes floating 
throughout the cytoplasm, all of which will interpret a message-in this case, messenger 
RNA-in one and the same way. It is interesting to think how one might merge the frame-notion with the actor-notion. Let us call a frame with the capability of gene rating and interpreting complex messages a 
symbol : 
 
frame + actor = symbol

Artificial Intelligence:Prospects 660We now have reached the point where we ar e talking about ways or implementing those 
elusive active symbols of Chapters XI and XII;  henceforth in this Chapter, "symbol" will 
have that meaning. By the way, don't feel dumb if you don't immediately see just how 
this synthesis is to be made. It is not clear, though it is certainly one of the most 
fascinating directions to go in AI. Furthermor e, it is quite certain that even the best 
synthesis of these notions will turn out to have much less power than the actual symbols 
of human minds. In that sense, calling th ese frame-actor syntheses "symbols" is 
premature, but it is an optimistic way of looking at things. 
Let us return to some issues connected with message passing. Should each 
message be directed specifically at a target symbol, or should it be thrown out into the 
grand void, much as mRNA is thrown out into the cytoplasm, to seek its ribosome? If

grand void, much as mRNA is thrown out into the cytoplasm, to seek its ribosome? If 
messages have destinations, then each symbol must have an address, and messages for it 
should always be sent to that address. On  the other hand, there could be one central 
receiving dock for messages, where a message would simply sit until it got picked up by 
some symbol that wanted it. This is a counterpart to General Delivery. Probably the best solution is to allow both types of message to exist; also to have provisions for different 
classes of urgency-special delivery, first cla ss, second class, and so on. The whole postal

classes of urgency-special delivery, first cla ss, second class, and so on. The whole postal 
system provides a rich source of ideas fo r message-passing languages, including such 
curios as selfaddressed stamped envelope s (messages whose senders want answers 
quickly), parcel post (extremely long messages which can be sent some very slow way), 
and more. The telephone system will give you more inspiration when you run out of 
postal-system ideas. 
 
Enzymes and AI 
 Another rich source of ideas for message pa ssing-indeed, for information processing in 
general-is, of course, the cell. Some objects in  the cell are quite comparable to actors-in

general-is, of course, the cell. Some objects in  the cell are quite comparable to actors-in 
particular, enzymes. Each enzyme's active site acts as a filter which only recognizes certain kinds of substrates (messages). Thus an enzyme has an "address", in effect. The enzyme is "programmed" (by virtue of its tertiary structure) to carry out certain operations upon that "message", and then to release it to the world again. Now in this 
way, when a message is passed from enzyme to enzyme along a chemical pathway, a lot 
can be accomplished. We have already desc ribed the elaborate kinds of feedback

can be accomplished. We have already desc ribed the elaborate kinds of feedback 
mechanisms which can take place in cells (eith er by inhibition or repression). These kinds 
of mechanisms show that complicated contro l of processes can arise through the kind of 
message passing that exists in the cell. 
One of the most striking things about  enzymes is how they sit around idly, 
waiting to be triggered by an incoming substrate. Then, when the substrate arrives, suddenly the enzyme springs into action, lik e a Venus's flytrap. This kind of "hair-

trigger" program has been used in Al, a nd goes by the name of demon. The important 
thing here is the idea of having many different  "species" of triggerable subroutines just 
lying around waiting to

Artificial Intelligence:Prospects 661be triggered. In cells, all the complex mol ecules and organelles are built up, simple step 
by simple step. Some of these new structur es are often enzymes themselves, and they 
participate in the building of new enzymes, whic h in turn participate in the building of yet 
other types of enzyme, etc. Such recursive cascades of enzymes can have drastic effects on what a cell is doing. One would like to s ee the same kind of simple step-by-step 
assembly process imported into AI, in the construction of useful subprograms. For 
instance, repetition has a way of burning new circ uits into our mental hardware, so that 
oft-repeated pieces of behavior become enc oded below the conscious level. It would be 
extremely useful if there were an analogous way of synthesizing efficient pieces of code 
which can carry out the same sequence of operations as something which has been 
learned on a higher level of "consciousness". Enzyme cascades may suggest a model for

learned on a higher level of "consciousness". Enzyme cascades may suggest a model for 
how this could be done. (The program called " HACKER ", written by Gerald Sussman, 
synthesizes and debugs small subroutines in a way not too much unlike that of enzyme cascades.) 
The sameness-detectors in the Bongard  problem-solver (Sams) could be 
implemented as enzyme-like subprograms. Like an enzyme, a Sam would meander about somewhat at random, bumping into small data structures here and there. Upon filling its two "active sites" with identical data stru ctures, the Sam would emit a message to other

implemented as enzyme-like subprograms. Like an enzyme, a Sam would meander about somewhat at random, bumping into small data structures here and there. Upon filling its two "active sites" with identical data stru ctures, the Sam would emit a message to other 
parts (actors) of the program. As long as programs are serial, it would not make much 
sense to have several copies of a Sam, but  in a truly parallel computer, regulating the 
number of copies of a subprogram would be  a way of regulating the expected waiting-
time before an operation gets done, just as re gulating the number of copies of an enzyme 
in a cell regulates how fast that function gets performed. And if new Sams could be 
synthesized, that would be comparable to th e seepage of pattern detection into lower 
levels of our minds.  
Fission and Fusion

Fission and Fusion 
 Two interesting and complementary ideas c oncerning the interaction of symbols are 
"fission" and "fusion". Fission is the gradual divergence of a new symbol from its parent 
symbol (that is, from the symbol which served  as a template off of which it was copied). 
Fusion is what happens when two (or more) or iginally unrelated symbols participate in a 
"joint activation", passing messages so tightly back and forth that they get bound together

"joint activation", passing messages so tightly back and forth that they get bound together 
and the combination can thereafter be addresse d as if it were a singl e symbol. Fission is a 
more or less inevitable process, since once a new symbol has been "rubbed off" of an old 
one, it becomes autonomous, and its interactions with the outside world get reflected in its private internal structure; so what st arted out as a perfect copy will soon become 
imperfect, and then slowly will become less and less like the symbol off of which it was

imperfect, and then slowly will become less and less like the symbol off of which it was 
"rubbed". Fusion is a subtler thing. When do two concepts really become 'one? Is there some precise instant when a fusion takes place?

Artificial Intelligence:Prospects 662This notion of joint activations ope ns up a Pandora's box of questions. For 
instance, how much coo we hear "dough" and "nut" when we say "doughnut"? Does a 
German who thinks of gloves ("Handschuhe")  hear "hand-shoes" or not? How about 
Chinese people, whose word "dong-xi" ("East -West") means "thing"? It is a matter of 
some political concern, too, since some peopl e claim that words like "chairman" are 
heavily charged with undertones of the male gender. The degree to which the parts resonate inside the whole probably varies  from person to person and according to 
circumstances. 
The real problem with this notion of "fusion"  of symbols is that it is very hard to 
imagine general algorithms which will create meaningful new symbols from colliding symbols. It is like two strands of DNA  which come together. How do you take parts from 
each and recombine them into a mean ingful and viable new strand of DNA  which codes

each and recombine them into a mean ingful and viable new strand of DNA  which codes 
for an individual of the same species? Or a new kind of species? The chance is infinitesimal that a random combination of pieces of DNA  will code for anything that 
will survive-something like the chance that a random combination of words from two 
books will make another book. The chance that recombinant DNA  will make sense on 
any level but the lowest is tiny, precisely becau se there are so many levels of meaning in 
DNA . And the same goes for "recombinant symbols". 
 
Epigenesis of the Crab Canon 
 I think of my Dialogue Crab Canon as a prot otype example where two ideas collided in

 I think of my Dialogue Crab Canon as a prot otype example where two ideas collided in 
my mind, connected in a new way, and sudde nly a new kind of verbal structure came 
alive in my mind. Of course I can still th ink about musical crab canons and verbal 
dialogues separately-they can still be activated independently of each other; but the fused 
symbol for crab canonical dialogues has its own characteristic modes of activation, too. 
To illustrate this notion of fusion or "symbolic recombination" in some detail, then, I would like to use the development of my Cr ab Canon as a case study, because, of course,

To illustrate this notion of fusion or "symbolic recombination" in some detail, then, I would like to use the development of my Cr ab Canon as a case study, because, of course, 
it is very familiar to me, and also because it is interesting, yet typical of how far a single 
idea can be pushed. I will recount it in stages named after those of meiosis, which is the 
name for cell division in which "crossing-over", or genetic recombination, takes place-the source of diversity in evolution. 
PROPHASE: I began with a rather simple idea-that a piece of music, say a canon, 
could be imitated verbally. This came from th e observation that, through a shared abstract 
form, a piece of text and a piece of music may be connected. The next step involved

form, a piece of text and a piece of music may be connected. The next step involved 
trying to realize some of the potential of th is vague hunch; here, I hit upon the idea that 
"voices" in canons can be mapped onto "char acters" in dialogues-still a rather obvious 
idea. 
Then I focused down onto specific kinds of canons, and remembered that there 
was a crab canon in the Musical Offering. At that time, I had just

Artificial Intelligence:Prospects 663begun writing Dialogues, and there were only tw o characters: Achilles and the Tortoise. 
Since the Bach crab canon has two voices, th is mapped perfectly: Achilles should be one 
voice, the Tortoise the other, with the one doing forwards what the other does backwards. 
But here I was faced with a problem: on what  level should the reversal take place? The 
letter level? The word level? The sentence level? After some thought, I concluded that 
the "dramatic line" level would be most appropriate. 
Now that the "skeleton" of the Bach crab  canon had been transplanted, at least in 
plan, into a verbal form, there was just one  problem. When the two voices crossed in the 
middle, there would be a short period of ex treme repetition: an ugly blemish. What to do 
about it? Here, a strange thing happened, a kind of level-crossing typical of creative acts:

about it? Here, a strange thing happened, a kind of level-crossing typical of creative acts: 
the word "crab" in "crab canon" flashed into my mind, undoubtedly because of some abstract shared quality with the notion of "tor toise"-and immediately I realized that at the 
dead center, I could block the repetitive eff ect, by inserting one special line, said by a 
new character: a Crab! This is how, in the "prophase" of the Crab Canon, the Crab was 
conceived: at the crossing over of Achilles and the Tortoise. (See Fig. 131.)   
 
FIGURE 131. A schematic diagram of the Dialogue Crab Canon .

FIGURE 131. A schematic diagram of the Dialogue Crab Canon . 
 METAPHASE: This was the skeleton of my Crab Canon . I then entered the 
second stage-the "metaphase"-in which I had to  fill in the flesh, which was of course an 
arduous task. I made a lot of stabs at it, getting used to the way in which pairs of 
successive lines had to make sense when read  from either direction, and experimenting 
around to see what kinds of dual meanings w ould help me in writing such a form (e.g., 
"Not at all"). There were two early versions  both of which were interesting, but weak. I

"Not at all"). There were two early versions  both of which were interesting, but weak. I 
abandoned work on the book for over a year, and when I returned to the Crab Canon , I 
had a few new ideas. One of them was to menti on a Bach canon inside it. At first my plan 
was to mention the "Canon per augmentationem, contrario motu", from the Musical 
Offering (Sloth Canon , as I call it). But that started to seem a little silly, so reluctantly I 
decided that inside my Crab Canon, I could talk about Bach's own Crab Canon  instead. 
Actually, this was a crucial turning point, but I didn't know it then.

Actually, this was a crucial turning point, but I didn't know it then. 
Now if one character was going to menti on a Bach piece, wouldn't it be awkward 
for the other to say exactly the same thing in the corresponding place? Well, Escher was playing a similar role to Bach in my thought s and my book, so wasn't there some way of 
just slightly modifying the line so that it would refer to Escher? After all, in the strict art 
of canons, note-perfect imitation is occasiona lly foregone for the sake of elegance or 
beauty. And

Artificial Intelligence:Prospects 664no sooner did that idea occur to me than the picture Day and Night (Fig. 49) popped into 
my mind. "Of course!" I thought, "It is a sort of pictorial crab canon, with essentially two 
complementary voices carrying the same th eme both leftwards and rightwards, and 
harmonizing with each other!" Here again was the notion of a single "conceptual 
skeleton" being instantiated in two different medi a-in this case, music and art. So I let the 
Tortoise talk about Bach, and Achilles talk about Escher, in parallel language; certainly 
this slight departure from strict imitation retained the spirit of crab cano.is. 
At this point, I began realizing that something marvelous was happening namely, 
the Dialogue was becoming self-referential, w ithout my even having intended it! What's 
more, it was an indirect self-reference, in th at the characters did not talk directly about

more, it was an indirect self-reference, in th at the characters did not talk directly about 
the Dialogue they were in, but rather about structures which were isomorphic to it (on a certain plane of abstraction). To put it in the terms I have been using, my Dialogue now 
shared a "conceptual skeleton" with Gödel’s G, and could therefore be mapped onto G in somewhat the way that the Central Dogma was, to create in this case a "Central Crabmap". This was most exciting to me, since out of nowhere had come an esthetically pleasing unity of Gödel, Escher, and Bach. 
ANAPHASE: The next step was quite st artling. I had had Caroline MacGillavry's

ANAPHASE: The next step was quite st artling. I had had Caroline MacGillavry's 
monograph on Escher's tessellations for years, but one day, as I flipped through it, my eye was riveted to Plate 23 (Fig. 42), for I saw it in a way I had never seen it before: here was a genuine crab canon-crab-like in both fo rm and content! Escher himself had given 
the picture no title, and since he had drawn si milar tessellations using many other animal 
forms, it is probable that this coincidence of  form and content was just something which I 
had noticed. But fortuitous or not, this untitle d plate was a miniature version of one main 
idea of my book: to unite form and cont ent. So with delight I christened it Crab Canon ,

idea of my book: to unite form and cont ent. So with delight I christened it Crab Canon , 
substituted it for Day and Night , and modified Achilles' and the Tortoise's remarks 
accordingly. 
Yet this was not all. Having become infatuated with molecular biology, one day I 
was perusing Watson's book in the bookstore , and in the index saw the word 
"palindrome". When I looked it up, I found a magi cal thing: crab-canonical structures in 
DNA. Soon the Crab's comments had been suitabl y modified to include a short remark to 
the effect that he owed his predilection fo r confusing retrograde and forward motion to 
his genes. 
TELOPHASE: The last step came months la ter, when, as I was talking about the

his genes. 
TELOPHASE: The last step came months la ter, when, as I was talking about the 
picture of the crab-canonical section of DNA  (Fig. 43), 1 saw that the 'A', 'T', 'C'  of 
Adenine, Thymine, Cytosine coincided- mirabile dictu -with the 'A', 'T', 'C'  of Achilles, 
Tortoise, Crab; moreover, just as Adenine and Thymine are paired in DNA , so are 
Achilles and the Tortoise paired in the Di alogue. I thought for a moment and, in another 
of those level-crossings, saw that 'G', the letter paired with ' C' in DNA , could stand for 
"Gene". Once again, I jumped back to the Dialogue, did a little surgery on the Crab's

"Gene". Once again, I jumped back to the Dialogue, did a little surgery on the Crab's 
speech to reflect this new discovery, and now I had a mapping between the DNA 's 
structure, and the Dialogue's st ructure. In that sense, the DNA  could be said to be a 
genotype coding for a phenotype: the

Artificial Intelligence:Prospects 665Structure of the Dialogue. This final touch dr amatically heightened the self-reference, 
and gave the Dialogue a density of meaning which I had never anticipated.  
Conceptual Skeletons and Conceptual Mapping 
 That more or less summarizes the epigenesis  of the Crab Canon. The whole process can 
be seen as a succession of mappings of ideas  onto each other, at varying levels of 
abstraction. This is what I call conceptual  mapping, and the abstract structures which 
connect up two different ideas are conceptual skeletons. Thus, one conceptual skeleton is 
that of the abstract notion of a crab canon: 
 
a structure having two parts which do the same thing, 
only moving in opposite directions. 
 
This is a concrete geometrical imag e which can be manipulated by the mind 
almost as a Bongard pattern. In fact, when I think of the Crab Canon today, I visualize it

almost as a Bongard pattern. In fact, when I think of the Crab Canon today, I visualize it 
as two strands which cross in the middle, where they are joined by a "knot" (the Crab's speech). This is such a vividly pictorial im age that it instantaneously maps, in my mind, 
onto a picture of two homologous chromosomes joined by a centromere in their middle, 
which is an image drawn directly from meiosis, as shown in Figure 132. 
 
FIGURE 132. 
 
In fact, this very image is what inspired me to cast the description of the Crab Canon's  
evolution in terms of meiosis-which is its elf, of course, vet another example of

evolution in terms of meiosis-which is its elf, of course, vet another example of 
conceptual mapping.  
Recombinant Ideas 
 There are a variety of techniques of fusion of two symbols. One involves lining the two 
ideas up next to each other (as if ideas were linear!), then judiciously choosing pieces from each one, and recombining them in a new symbol. This strongly recalls genetic 
recombination. Well, what do chromosomes exchange, and how do they do it? They 
exchange genes. What in a symbol is comparable to a gene? If symbols have frame-like slots, then slots, perhaps. But which slot s to exchange, and why? Here is where the 
crabcanonical fusion may offer some ideas. Ma pping the notion of "musical crab canon"

crabcanonical fusion may offer some ideas. Ma pping the notion of "musical crab canon" 
onto that of "dialogue" involved several auxiliary mappings; in

Artificial Intelligence:Prospects 666fact it induced  them. That is, once it had been decide d that these two notions ,ere to be 
fused, it became a matter of looking at them  on a level where analogous parts emerged 
into view, then going ahead and mapping the parts  onto each other, and so on, 
recursively, to any level that was found de sirable. Here, for instance, "voice" and 
"character" emerged as corresponding slots when "crab canon" and "dialogue" were viewed abstractly. Where did these abstract views come from, though? This is at the crux 
of the mapping-problem-where do abstract vi ews come from? How do you make abstract 
views of specific notions?  
Abstractions, Skeletons, Analogies 
 
A view which has been abstracted from a c oncept along some dimension is what I call a 
conceptual skeleton . In effect, we have dealt with c onceptual skeletons all along, without 
often using that name. For instance, many of the ideas concerning Bongard problems

often using that name. For instance, many of the ideas concerning Bongard problems 
could be rephrased using this terminology. It  is always of interest, and possibly of 
importance, when two or more ideas are di scovered to share a conceptual skeleton. An 
example is the bizarre set of concepts men tioned at the beginning of the Contrafactus: a 
Bicyclops, a tandem unicycle, a teeter-teeter , the game of ping-ping, a one-way tie, a 
two-sided Mobius strip, the "Bach twins", a piano concerto for two left hands, a one-
voice fugue, the act of clapping with one  hand, a two-channel monaural phonograph, a

voice fugue, the act of clapping with one  hand, a two-channel monaural phonograph, a 
pair of eighth-backs. All of these ideas are "isomorphic" because they share this 
conceptual skeleton:  
a plural thing made singular and re-pluralized wrongly. 
 
Two other ideas in this book which share that conceptual skeleton are (1) the Tortoise's solution to Achilles' puzzle, asking for a word beginning and ending in "HE" (the 
Tortoise's solution being the pronoun "HE", which collapses two occurrences into one), and (2) the Pappus-Gelernter proof of the Pons  As' norum Theorem, in which one triangle

Tortoise's solution being the pronoun "HE", which collapses two occurrences into one), and (2) the Pappus-Gelernter proof of the Pons  As' norum Theorem, in which one triangle 
is reperceived as two. Incidentally, these droll concoctions might be dubbed "demi-doublets". 
A conceptual skeleton is like a set of c onstant features (as distinguished from 
parameters or variables)-features which should not be slipped in a subjunctive instant replay or mapping-operation. Having no parameters  or variables of its own to vary, it can 
be the invariant core of several different ideas. Each instance  o f  i t ,  s u c h  a s  " t a n d e m  
unicycle", does have layers of variability and so can be "slipped" in various ways.

unicycle", does have layers of variability and so can be "slipped" in various ways. 
Although the name "conceptual skeleton" sounds absolute and rigid, actually 
there is a lot of play in it. There can be con ceptual skeletons on several different levels of 
abstraction. For instance, the "isomor phism" between Bongard problems 70 and 71, 
already pointed out, involves a hi gher-level conceptual skeleton than that needed to solve 
either problem in isolation.

Artificial Intelligence:Prospects 667Multiple Representations 
 
Not only must conceptual skeletons exist on di fferent levels of abstraction; also, they 
must exist along different conceptual dimens ions. Let us take the following sentence as 
an example:  
"The Vice President is the spare tire on the automobile of government." 
 How do we understand what it means (leaving asid e its humor, which is of course a vital 
aspect)? If you were told, "See our governme nt as an automobile" without any prior 
motivation, you might come up with any num ber of correspondences: steering wheel = 
president, etc.. What are checks and balan ces? What are seat belts? Because the two 
things being mapped are so different, it is al most inevitable that the mapping will involve 
functional aspects. Therefore, you retrieve from your store of conceptual skeletons 
representing parts of automobiles, only those having to do with function, rather than, say,

representing parts of automobiles, only those having to do with function, rather than, say, 
shape. Furthermore, it makes sense to work at a pretty high level of abstraction, where 
"function" isn't taken in too narrow a context.  Thus, of the two following definitions of 
the function of a spare tire: (1) "replacement for a flat tire", and (2) "replacement for a 
certain disabled part of a car", certainly the latter would be preferable, in this case. This 
comes simply from the fact that an auto and a government are so different that they have 
to be mapped at a high level of abstraction.

to be mapped at a high level of abstraction. 
Now when the particular sentence is ex amined, the mapping gets forced in one 
respect-but it is not an awkward way, by any means. In fact, you already have a 
conceptual skeleton for the Vice President, among many others, which says, "replacement for a certain disabled part of  government". Therefore the forced mapping 
works comfortably. But suppose, for the sake of contrast, that you had retrieved another 
conceptual skeleton for "spare tire"-say, one  describing its physical aspects. Among other

conceptual skeleton for "spare tire"-say, one  describing its physical aspects. Among other 
things, it might say that a spare tire is "round a nd inflated". Clearly, this is not the right 
way to go. (Or is it? As a friend of mine poi nted out, some Vice Presidents are rather 
portly, and most are quite inflated!) 
 
Ports of Access 
 One of the major characteristics of each id iosyncratic style of thought is how new 
experiences get classified and stuffed into  memory, for that defines the "handles" by 
which they will later be retrievable. And for events, objects, ideas, and so on-for

which they will later be retrievable. And for events, objects, ideas, and so on-for 
everything that can be thought a bout-there is a wide variety of "handles". I am struck by 
this each time I reach down to turn on my car  radio, and find, to my dismay, that it is 
already on! What has happened is that two i ndependent representations are being used for 
the radio. One is "music producer", the other is "boredom reliever". I am aware that the music is on, but I am bored anyway, and befo re the two realizations have a chance to 
interact, my reflex to reach

Artificial Intelligence:Prospects 668down has been triggered. The same reaching-dow n reflex one day occurred just after I'd 
left the radio at a repair shop and was driv ing away, wanting to hear some music. Odd. 
Many other representations for the same object exist, such as  
shiny silver-knob haver overheating-problems haver  lying-on-my-back-over-hump-to-fix thing  buzz-maker slipping-dials object multidimensional representation example 
 
All of them can act as ports of access. Though th ey all are attached to my symbol for my 
car radio, accessing that symbol through one doe s not open up all the others. Thus it is 
unlikely that I will be inspired to remember  lying on my back to fix the radio when I 
reach down and turn it on. And conversely, when I'm lying on my back, unscrewing 
screws, I probably won't think about the time I heard the Art of the Fugue  on it. There are

screws, I probably won't think about the time I heard the Art of the Fugue  on it. There are 
"partitions" between these aspects of one symbol, partitions that prevent my thoughts from spilling over sloppily, in the manner of fr ee associations. My mental partitions are 
important because they contain and channel the flow of my thoughts. 
One place where these partitions are quite rigid is in sealing off words for the 
same thing in different languages. If the partitions were not strong, a bilingual person would constantly slip back and forth between languages, which would be very

same thing in different languages. If the partitions were not strong, a bilingual person would constantly slip back and forth between languages, which would be very 
uncomfortable. Of course, adults learni ng two new languages at once often confuse 
words in them. The partitions between thes e languages are flimsier, and can break down. 
Interpreters are particularly interesting, sin ce they can speak any of their languages as if 
their partitions were inviolable and yet, on command, they can negate those partitions to 
allow access to one language from the other, so  they can translate. Steiner, who grew up

allow access to one language from the other, so  they can translate. Steiner, who grew up 
trilingual, devotes several pages in After Babel  to the intermingling of French, English, 
and German in the layers of his mind, and how his different languages afford different 
ports of access onto concepts. 
 
Forced Matching 
 When two ideas are seen to share conceptu al skeletons on some level of abstraction, 
different things can happen. Usually the first stage is that you zoom in on both ideas, and, 
using the higher-level match as a guide, you try to identify corresponding subideas.

using the higher-level match as a guide, you try to identify corresponding subideas. 
Sometimes the match can be extended recursiv ely downwards several levels, revealing a 
profound isomorphism. Sometimes it stops earlie r, revealing an analogy or similarity. 
And then there are times when the high-level similarity is so compelling that, even if 
there is no apparent lower-level continuation of the map, you just go ahead and make one: this is the forced match .

Artificial Intelligence:Prospects 669Forced matches occur every day in th e political cartoons of newspapers: a 
political figure is portrayed as an airplane, a boat, a fish, the Mona Lisa; a government is a human, a bird, an oil rig; a treaty is a brie fcase, a sword, a can of worms; on and on and 
on. What is fascinating is how easily we can perform the suggested mapping, and to the exact depth intended. We don't carry the mapping out too deeply or too shallowly. 
Another example of forcing one thing into  the mold of another occurred when I 
chose to describe the development of my Cr ab Canon in terms of meiosis. This happened 
in stages. First, I noticed the common conceptual skeleton shared by the Crab Canon and the image of chromosomes joined by a centrom ere; this provided the inspiration for the 
forced match. Then I saw a high-level rese mblance involving "growth", "stages", and 
"recombination". Then I simply pushed the an alogy as hard as I could. Tentativity-as in

"recombination". Then I simply pushed the an alogy as hard as I could. Tentativity-as in 
the Bongard problem-solver-played a large role : I went forwards and backwards before 
finding a match which I found appealing. 
A third example of conceptual mapping is provided by the Central Dogmap. I 
initially noticed a high-level similarity between  the discoveries of mathematical logicians 
and those of molecular biologists, then pur sued it on lower levels until I found a strong 
analogy. To strengthen it further, I chose a Godel-numbering which imitated the Genetic 
Code. This was the lone element of forced matching in the Central Dogmap.

Code. This was the lone element of forced matching in the Central Dogmap. 
Forced matches, analogies, and meta phors cannot easily be separated out. 
Sportscasters often use vivid imagery which is hard to pigeonhole. For instance, in a metaphor such as "The Rams [football team ar e spinning their wheels", it is hard to say 
just what image you are supposed to conjure up. Do you attach wheels to the team as a 
whole% Or to each player? Probably neith er one. More likely, the image of wheels 
spinning in mud or snow simply flashes before  you for a brief instant, and then in some

spinning in mud or snow simply flashes before  you for a brief instant, and then in some 
mysterious way, just the relevant parts ge t lifted out and transferred to the team's 
performance. How deeply are the football team  and the car mapped onto each other in the 
split second that you do this? 
 
Recap 
 Let me try to tie things together a little. I have presented a number of related ideas 
connected with the creation, manipulation, a nd comparison of symbols. Most of them 
have to do with slippage in some fashion, th e idea being that concepts are composed of

Recap 
have to do with slippage in some fashion, th e idea being that concepts are composed of 
some tight and some loose elements, coming fr om different levels of nested contexts 
(frames). The loose ones can be dislodged and replaced rather easily, which, depending 
on the circumstances, can create a "subjunctive instant replay", a forced match, or an 
analogy. A fusion of two symbols may result from a process in which parts of each symbol are dislodged and other parts remain.

Artificial Intelligence:Prospects 670Creativity and Randomness 
 
It is obvious that we are talking about mech anization of creativity. But the this not a 
contradiction in terms? Almost, but not reall y. Creativity s essence of that which is not 
mechanical. Yet every creative act is mechani cal-it has its explanation no less than a case 
of the hiccups does. The mechanical substrate of creativity may be hidden from view, but it exists. Conversely, there is something unmechanical in flexible programs, even today. It may not constitute creativity, but when programs cease to be transparent to their 
creators, then the approach to creativity has begun. 
It is a common notion that randomness is an  indispensable ingredient of creative 
acts. This may be true, but it does not have any bearing on the mechanizability-or rather, programmability!-of creativity. The world is  a giant heap of randomness; when you 
mirror some of it inside your head, your h ead's interior absorbs a little of that

mirror some of it inside your head, your h ead's interior absorbs a little of that 
randomness. The triggering patterns of symbol s, therefore, can lead you down the most 
randomseeming paths, simply because they came from your interactions with a crazy, 
random world. So it can be with a computer  program, too. Randomness is an intrinsic 
feature of thought, not something which has to  be "artificially inseminated", whether 
through dice, decaying nuclei, random number tabl es, or what-have-you. It is an insult to 
human creativity to imply that it relies on such arbitrary sources. 
What we see as randomness is often simp ly an effect of looking at something

What we see as randomness is often simp ly an effect of looking at something 
symmetric through a "skew" filter. An eleg ant example was provided by Salviati's two 
ways of looking at the number it/4. Although the decimal expansion of 7r/4 is not literally 
random, it is as random as one would need  for most purposes: it is "pseudorandom". 
Mathematics is full of pseudorandomness-plen ty enough to supply all would-be creators 
for all time. 
Just as science is permeated with "con ceptual revolutions" on all levels at all

Just as science is permeated with "con ceptual revolutions" on all levels at all 
times, so the thinking of individuals is shot  through and through with creative acts. They 
are not just on the highest plane; they are ev erywhere. Most of them are small and have 
been made a million times before-but they ar e close cousins to the most highly creative 
and new acts. Computer programs today do not yet seem to produce many small creations. Most of what they do is quite "mechan ical" still. That just testifies to the fact 
that they are not close to simulating the way we think-but they are getting closer.

that they are not close to simulating the way we think-but they are getting closer. 
Perhaps what differentiates highly creativ e ideas from ordinary ones is some 
combined sense of beauty, simplicity, and harmony. In fact, I have a favorite "meta-analogy", in which I liken analogies to chords. The idea is simple: superficially similar ideas are often not deeply related; and d eeply related ideas are often superficially 
disparate. The analogy to chords is natu ral: physically close notes are harmonically 
distant (e.g., E-F-G ); and harmonically close notes are physically distant (e.g., G-E-B ).

distant (e.g., E-F-G ); and harmonically close notes are physically distant (e.g., G-E-B ). 
Ideas that share a conceptual skeleton resona te in a sort of conceptual analogue to 
harmony; these harmonious "idea-chords" are often widely separated, as

Artificial Intelligence:Prospects 671measured on an imaginary "keyboard of concep ts". Of course, it doesn't suffice to reach 
wide and plunk down any old way-you may hit a seventh or a ninth! Perhaps the present analogy is like a ninth-chord-wide but dissonant.  
Picking up Patterns on All Levels 
 Bongard problems were chosen as a focus in  this Chapter because when you study them, 
you realize that the elusive sense for pattern s which we humans inherit from our genes 
involves all the mechanisms of representa tion of knowledge, including nested contexts, 
conceptual skeletons and conceptual ma pping, slippability, descriptions and meta-
descriptions and their interactions, fission a nd fusion of symbols, multiple representations 
(along different dimensions and different levels of abstraction), default expectations, and 
more. 
These days, it is a safe bet that if some  program can pick up patterns in one area, 
it will miss patterns in another area which, to  us, are equally obvious. You may remember

more. 
it will miss patterns in another area which, to  us, are equally obvious. You may remember 
that I mentioned this back in Chapter 1, saying that machines can be oblivious to 
repetition, whereas people cannot. For instance, consider SHRDLU . If Eta Oin typed the 
sentence "Pick up a big red block a nd put it down" over and over again, SHRDLU  would 
cheerfully react in the same way over and over again, exactly as an adding machine will print out "4" over and over again, if a human being has the patience to type "2+2" over 
and over again. Humans aren't like that; if so me pattern occurs over and over again, they

and over again. Humans aren't like that; if so me pattern occurs over and over again, they 
will pick it up. SHRDLU  wasn't built with the potential for forming new concepts or 
recognizing patterns: it had no sense of over and overview. 
 
The Flexibility of Language 
 SHRDLU 's language-handling capability is immensely flexible-within limits. SHRDLU  
can figure out sentences of great syntactical  complexity, or sentences with semantic 
ambiguities as long as-they can- be resolved  by inspecting the data base-but it cannot 
handle "hazy" language. For instance, consider  the sentence "How many blocks go on top

handle "hazy" language. For instance, consider  the sentence "How many blocks go on top 
of each other to make a steeple?" We unde rstand it immediately, yet it does not make 
sense if interpreted literally. Nor is it that some idiomatic phrase has been used. "To go on top of each other" is an imprecise phras e which nonetheless gets the desired image 
across quite well to a human. Few people would be misled into visualizing a paradoxical setup with two blocks each of which is on t op of the other-or blocks which are "going" 
somewhere or other. 
The amazing thing about language is how imprecisely we use it and still manage

The amazing thing about language is how imprecisely we use it and still manage 
to get away with it. SHRDLU  uses words in a "metallic" way, while people use them in a 
"spongy" or "rubbery" or even "Nutty-Putty ish" way. If words were nuts and bolts, 
people could make any bolt fit into any nut: they'd  just squish the one into the other, as in 
some surrealistic

Artificial Intelligence:Prospects 672painting where everything goes soft. Language , in human hands, becomes almost like a 
fluid, despite, the coarse grain of its components. 
Recently, Al research in natural language understanding has turned away 
somewhat from the understanding of single se ntences in isolation, and more towards 
areas such as understanding simple children's stories. Here is a well-known children's joke which illustrates the open-endedness of real-life situations: 
 
A man took a ride in an airplane. 
Unfortunately, he fell out.  
Fortunately, he had a parachute on. 
Unfortunately, it didn't work.  
Fortunately, there was a haystack below him. 
Unfortunately, there was a pitchfork sticking out of it.  
Fortunately, he missed the pitchfork. 
Unfortunately, he missed the haystack.  
It can be extended indefinitely. To represent this silly story in a frame-based system would be extremely complex, involving jointly activating frames for the concepts of man,

It can be extended indefinitely. To represent this silly story in a frame-based system would be extremely complex, involving jointly activating frames for the concepts of man, 
airplane, exit, parachute, falling, etc., etc.  
Intelligence and Emotions 
 
Or consider this tiny yet poignant story:  
Margie was holding tightly to the string of her beautiful new balloon. Suddenly, a 
gust of wind caught it. The wind carried it into a tree. The balloon hit a branch 
and burst. Margie cried and cried.' 
 
To understand this story, one needs to read many things between the lines. For instance: 
Margie is a little girl. This is a toy balloon w ith a string for a child to hold. It may not be

Margie is a little girl. This is a toy balloon w ith a string for a child to hold. It may not be 
beautiful to an adult, but in a child's eye,  it is. She is outside. The "it" that the wind 
caught was the balloon. The wind did not pull Ma rgie along with the balloon; Margie let 
go. Balloons can break on contact with any sh arp point. Once they are broken, they are 
gone forever. Little children love balloons a nd can be bitterly disappointed when they 
break. Margie saw that her balloon was broke n. Children cry when they are sad. "To cry 
and cry" is to cry very long and hard. Marg ie cried and cried because of her sadness at

and cry" is to cry very long and hard. Marg ie cried and cried because of her sadness at 
her balloon's breaking. 
This is probably only a small fraction of what is lacking at the surface level. A 
program must have all this knowledge in or der to get at what is going on. And you might 
object that, even if it "understands" in some intellectual sense what has been said, it will 
never really understand, until it, too, has cried and cried. And when will a computer do that? This is the kind of humanistic point which Joseph Weizenbaum is concerned with 
making in his book Computer Power and Human Reason , and I think it is an important

making in his book Computer Power and Human Reason , and I think it is an important 
issue; in fact, a very, very deep issue. Un fortunately, many Al workers at this time are 
unwilling, for various reasons, to take this sort of point

Artificial Intelligence:Prospects 673seriously. taut in some ways, those Al workers are right: it is a little premature to think 
about computers crying; we must first thi nk about rules for computers to deal with 
language and other things; in time, we'll fi nd ourselves face to face with the deeper 
issues.  
AI Has Far to Go 
 Sometimes it seems that there is such a comple te absence of rule-governed behavior that 
human beings just aren't rule-governed. But this is an illusion-a little like thinking that crystals and metals emerge from rigid underl ying laws, but that fluids or flowers don't. 
We'll come back to this question in the next Chapter.  
The process of logic itself working intern ally in the brain may be more analogous 
to a succession of operations with symbolic pictures, a sort of abstract analogue of 
the Chinese alphabet or some Mayan de scription of events-except that the 
elements are not merely words but more  like sentences or whole stories with

elements are not merely words but more  like sentences or whole stories with 
linkages between them forming a sort of meta- or super-logic with its own rules.' 
 
It is hard for most specialists to express vividly-perhaps even to remember-what 
originally sparked them to enter their field. Conversely, someone on the outside may 
understand a field's special romance and may be able to articulate it precisely. I think that 
is why this quote from Ulam has appeal for me, because it poetically conveys the 
strangeness of the enterprise of Al, and yet s hows faith in it. And one must run on faith at 
this point, for there is so far to go! 
 
Ten Questions and Speculations

Ten Questions and Speculations 
 To conclude this Chapter, I would like to present ten "Questions and Speculations" about 
Al. I would not make so bold as to call th em "Answers"-these are my personal opinions. 
They may well change in some ways, as I lear n more and as Al develops more. (In what 
follows, the term "Al program" means a program  which is far ahead of today's programs; 
it means an "Actually Intelligent" program. Also, the words "program" and "computer" 
probably carry overly mechanistic connotations, but let us stick with them anyway.) 
 
Question: Will a computer program ever write beautiful music?

Question: Will a computer program ever write beautiful music? 
Speculation: Yes, but not soon. Music is a language of emotions, and until programs 
have emotions as complex as ours, there is no way a program will write anything beautiful. There can be "forgeries” shallow imitations of the syntax of earlier music-
but despite what one might think at first, there is much more to musical expression 
than can be captured in syntactical rules. There will be no new kinds of beauty turned up for a long time by computer music- composing programs. Let me carry this

than can be captured in syntactical rules. There will be no new kinds of beauty turned up for a long time by computer music- composing programs. Let me carry this 
thought a little further. To think-and I have  heard this suggested-that we might soon 
be able to command a preprogrammed mass-produced mail-order twenty-dollar desk-model "music box" to bring forth from  its sterile circuitry pieces which Chopin 
or Bach might have written had they lived longer is a grotesque and shameful misestimation of the depth of the human spirit. A "program" which could produce

Artificial Intelligence:Prospects 674music as they did would have to wande r around the world on its own, fighting its 
way through the maze of life a nd feeling every moment of it. It would have to 
understand the joy and loneliness of a chilly  night wind, the longing for a cherished 
hand, the inaccessibility of a distant town, the heartbreak and regeneration after a human death. It would have to have know n resignation and worldweariness, grief 
and despair, determination and victory, pi ety and awe. In it would have had to 
commingle such opposites as hope and fear, anguish and jubilation, serenity and 
suspense. Part and parcel of it would have  to be a sense of grace, humor, rhythm, a 
sense of the unexpected-and of course an e xquisite awareness of the magic of fresh 
creation. Therein, and therein only, lie the sources of meaning in music. 
 
Question: Will emotions be explicitly programmed into a machine? 
Speculation: No. That is ridiculous. Any direct simulation of emotions-PARRY, for

Speculation: No. That is ridiculous. Any direct simulation of emotions-PARRY, for 
example-cannot approach the complexity of human emotions, which arise indirectly from the organization of our minds. Programs or machines will acquire 
emotions in the same way: as by-products of their structure, of the way in which they are organized-not by direct pr ogramming. Thus, for example, nobody will 
write a "falling-in-love" subroutine, any more than they would write a "mistake-making" subroutine. "Falling in love" is a description which we attach to a complex

write a "falling-in-love" subroutine, any more than they would write a "mistake-making" subroutine. "Falling in love" is a description which we attach to a complex 
process of a complex system; there need  be no single module inside the system 
which is solely responsible for it, however! 
 Question: Will a thinking computer be able to add fast? 
Speculation: Perhaps not. We ourselves are composed of hardware which does fancy 
calculations but that doesn't mean that our symbol level, where "we" are, knows 
how to carry out the same fancy calculations. Let me put it this way: there's no way 
that you can load numbers into your ow n neurons to add up your grocery bill.

that you can load numbers into your ow n neurons to add up your grocery bill. 
Luckily for you, your symbol level (i.e., you) can't gain access to the neurons which 
are doing your thinking-otherwise you'd get addle-brained. To paraphrase Descartes 
again: 
 
"I think; therefore I have no access 
to the level where I sum." 
 
Why should it not be the same for an intelligent program? It mustn't be allowed to gain 
access to the circuits which are doing its thinking otherwise it'll get addle-CPU'd. 
Quite seriously, a machine that can pass th e Turing test may well add as slowly as

again: 
 
Quite seriously, a machine that can pass th e Turing test may well add as slowly as 
you or I do, and for

Artificial Intelligence:Prospects 675similar reasons. It will represent the number 2 not just by the two bits "10", but as a 
full-fledged concept the way we do, replete with associations such as its homonyms 
"too" and "to", the words "couple" and "deu ce", a host of mental images such as 
dots on dominos, the shape of the numeral '2 ', the notions of alternation, evenness, 
oddness, and on and on ... With all this "extra baggage" to carry around, an 
intelligent program will become quite slothful in its adding. Of course, we could give it a ' pocket calculator , so to speak (or build one in). Then it could answer 
very fast, but its performance would be just like that of a person with a pocket 
calculator. There would be two separate  parts to the machine: a reliable but 
mindless part and an intelligent but fallible  part. You couldn't rely on the composite 
system to be reliable, any more than  a composite of person and machine is

system to be reliable, any more than  a composite of person and machine is 
necessarily reliable. So if it's right answers you're after, better stick to the pocket calculator alone-don't throw in the intelligence! 
 
Question: Will there be chess programs that can beat anyone? 
Speculation: No. There may be programs whic h can beat anyone at chess, but they 
will not be exclusively chess players.  They will be programs of general 
intelligence, and they will be just as temp eramental as people. "Do you want to play

intelligence, and they will be just as temp eramental as people. "Do you want to play 
chess?" "No, I'm bored with chess. Let's ta lk about poetry." That may be the kind of 
dialogue you could have with a program that could beat everyone. That is because real intelligence inevitably depends on a total overview capacity-that is, a programmed ability to "jump out of the system", so to speak-at least roughly to the extent that we have that ability. Once that  is present, you can't contain the program; 
it's gone beyond that certain critical point, and you just have to  face the facts of 
what you've wrought.

what you've wrought. 
 
Question: Will there be special locations in  memory which store parameters governing 
the behavior of the program, such that if  you reached in and changed them, you would 
be able to make the program smarter or stupi der or more creative or more interested in 
baseball? In short, would you be able to  "tune" the program by fiddling with it on a 
relatively low level? Speculation: No. It would be quite oblivious to changes of any particular elements in 
memory, just as we stay almost exactly  the same though thousands of our neurons

memory, just as we stay almost exactly  the same though thousands of our neurons 
die every day(!). If you fuss around too heav ily, though, you'll damage it, just as if 
you irresponsibly did neurosurgery on a human being. There will be no "magic" 
location in memory where, for instance, th e "IQ" of the program sits. Again, that 
will be a feature which emerges as a c onsequence of lower-level behavior, and 
nowhere will it sit explicitly. The same goe s for such things as "the number of 
items it can hold in short-term memory", "the amount it likes physics", etc., etc.

items it can hold in short-term memory", "the amount it likes physics", etc., etc. 
 Question: Could you "tune" an Al program to act like me, or like you-or halfway between 
us?

Artificial Intelligence:Prospects 676Speculation: No. An intelligent program will not be chameleon-like, any more than 
people are. ,It will rely on the constancy of its memories, and will not be able to flit 
between personalities. The idea of changing in ternal parameters to "tune to a new 
personality" reveals a ridiculous underestimation of the complexity of personality. 
 Question: Will there be a "heart" to an Al pr ogram, or will it simply consist of "senseless 
loops and sequences of trivial operations" (in the words of Marvin Minskys)? Speculation: If we could see all the way to  the bottom, as we can a shallow pond, we 
would surely see only "senseless loops and sequences of trivial operations"-and we 
would surely not see any "heart". Now th ere are two kinds of extremist views on 
AI: one says that the human mind is, for fundamental and mysterious reasons, unprogrammable. The other says that you mere ly need to assemble the appropriate

AI: one says that the human mind is, for fundamental and mysterious reasons, unprogrammable. The other says that you mere ly need to assemble the appropriate 
"heuristic devices-multiple optimizers, pattern-recognition tricks, planning algebras, recursive administration proce dures, and the like",' and you will have 
intelligence. I find myself somewhere in between, believing that the "pond" of an 
Al program will turn out to be so deep and murky that we won't be able to peer all 
the way to the bottom. If we look from the top, the loops will be invisible, just as 
nowadays the current-carrying electrons ar e invisible to most programmers. When

nowadays the current-carrying electrons ar e invisible to most programmers. When 
we create a program that passes the Turi ng test, we will see a "heart" even though 
we know it's not there. 
 Question: Will Al programs ever become "superintelligent"? 
Speculation: I don't know. It is not clear that we would be able to understand or relate 
to a "superintelligence", or that the con cept even makes sense. For instance, our 
own intelligence is tied in with our speed of thought. If our reflexes had been ten 
times faster or slower, we might have developed an entirely different set of

times faster or slower, we might have developed an entirely different set of 
concepts with which to describe the worl d. A creature with a radically different 
view of the world may simply not have ma ny points of contact with us. I have often 
wondered if there could be, for instance, pi eces of music which are to Bach as Bach 
is to folk tunes: "Bach squared", so to speak. And would I be able to understand them? Maybe there is such music around me  already, and I just don't recognize it,

is to folk tunes: "Bach squared", so to speak. And would I be able to understand them? Maybe there is such music around me  already, and I just don't recognize it, 
just as dogs don't understand language. The idea of superintelligence is very strange. In any case, I don't think of it as th e aim of Al research, although if we ever 
do reach the level of human intelligence,  superintelligence will undoubtedly be the 
next goal-not only for us, but for our Al-program colleagues, too, who will be 
equally curious about Al and superinte lligence. It seems quite likely that Al 
programs will be extremely curious about Al in general-understandably.

programs will be extremely curious about Al in general-understandably. 
 Question: You seem to be saying that AI pr ograms will be virtually identical to people, 
then. Won't there be any differences?

Artificial Intelligence:Prospects 677Speculation: Probably the differences between  Al programs and people will be larger 
than the differences between most people. It is almost impossible to imagine that 
the "body" in which an Al program is hous ed would not affect it deeply. So unless 
it had an amazingly faithful replica of a human body-and why should it?-it would 
probably have enormously different persp ectives on what is important, what is 
interesting, etc. Wittgenstein once made  the amusing comment, "If a lion could 
speak, we would not understand him." It makes me think of Rousseau's painting of the gentle lion and the sleeping gypsy on the moonlit desert. But how does Wittgenstein know? My guess is that any Al  program would, if comprehensible to 
us, seem pretty alien. For that reason, we will have a very hard time deciding when and if we really are dealing with an Al program, or just a "weird" program. 
 Question: Will we understand what intelligen ce and consciousness and free will and "I"

 Question: Will we understand what intelligen ce and consciousness and free will and "I" 
are when we have made an intelligent program? Speculation: Sort of-it all depends on what  you mean by "understand". On a gut level, 
each of us probably has about as good an  understanding as is possible of those 
things, to start with. It is like listeni ng to music. Do you really understand Bach 
because you have taken him apart? Or did you understand it that time you felt the 
exhilaration in every nerve in your body? Do we understand how the speed of light

exhilaration in every nerve in your body? Do we understand how the speed of light 
is constant in every inertial reference fr ame? We can do the math, but no one in the 
world has a truly relativistic intuition. And probably no one will ever understand 
the mysteries of intelligence and consciousne ss in an intuitive way. Each of us can 
understand people , and that is probably about as close as you can come.

Sloth Canon 
 678Sloth Canon 
 
This time, we find Achilles and the Tortoise visiting 
the dwelling of their new friend, the Sloth. 
 
Achilles: Shall I tell you of my droll footrace with Mr. T? Sloth: Please do. Achilles: It has become quite celebrated in th ese parts. I believe it's even been written up, 
by Zeno. 
Sloth: It sounds very exciting. Achilles: It was. You see, Mr. T began way ah ead of me. He had such a huge head start, 
and yet 
Sloth: You caught up, didn't you? Achilles: Yes-being so fleet of foot, I dimini shed the distance between us at a constant 
rate, and soon overtook him. 
Sloth: The gap kept getting shorter and shorter, so you could. Achilles: Exactly. Oh, look-Mr. T has brought hi s violin. May I try playing on it, Mr. T? 
Tortoise: Please don't. It sounds very flat. Achilles: Oh, all right. But I'm in a mood for music. I don't know why. Sloth: You can 
play the piano, Achilles. 
Achilles: Thank you. I'll try it In a moment. I ju st wanted to add that I also had another

Achilles: Thank you. I'll try it In a moment. I ju st wanted to add that I also had another 
kind of "race" with Mr. T at a later date. Unfortunately, in that race 
Tortoise: You didn't catch up, did you? The gap kept getting longer and longer, so you 
couldn't. 
Achilles: That's true. I believe THAT r ace has been written up, too, by Lewis Carroll. 
Now, Mr. Sloth, I'll take up your offer of tr ying out the piano. But I'm so bad at the 
piano. I'm not sure I dare. Sloth: You should try.  
 
(Achilles sits down and starts playing a simple tune.) 
 
Achilles: Oh-it sounds very strange. That's  not how it's supposed to sound at all! 
Something is very wrong.

couldn't. 
 
Something is very wrong. 
Tortoise: You can't play the piano, Achilles. You shouldn't try. Achilles: It's like a piano in a mirror. The hi gh notes are on the left, and the low notes are 
on the right. Every melody comes out invert ed, as if upside down. Who would have 
ever thought up something so cockeyed as that? 
Tortoise: That's so characteristic of sloths. They hang from Achilles: Yes, I know-from tree branches-upsid e down, of course. That sloth-piano would 
he appropriate for playing inverted melodies such

 679
 
FIGURE 133. " Sloth Canon",from the Musical Offering, by J. S. Bach. [Music printed 
by Donald Byrd's program "SMUT' .

 680 
as occur in some canons and fugues. But to learn to play a piano while hanging from a tree must he very difficult. You must have to devote a great deal of energy to it. 
Sloth: That's not so characteristic of sloths. Achilles: No, I gather sloths like to take life very easy. They do everything about half as 
fast as normal. And upside down, to boot. What a peculiar way to go through life! Speaking of things that are both upside- and slowed-down, there's a "Canon per augmentationem, contrario motu" in the Musical Offering . In my edition, the letters 
`S', `A', `T' are in front of the three staves. I don't know why. Anyway, I think Bach 
carried it off very skillfully. What's your opinion, Mr. T? 
Tortoise: He outdid himself. As for those le tters "SAT", you could guess what they stand 
for. 
Achilles: "Soprano", "Alto", and "Tenor", I suppose. Three-part pieces are often written 
for that combination of voices. Wouldn't you agree, Mr. Sloth?

for. 
for that combination of voices. Wouldn't you agree, Mr. Sloth? 
Sloth: They stand for Achilles: Oh, just a moment, Mr. Sloth. Mr. Tortoise-why are you putting on your coat? 
You're not leaving, are you? We were just going to fix a snack to eat. You look very tired. How do you feel? 
Tortoise: Out of gas. So long! ( Trudges wearily out the door .) 
Achilles: The poor fellow-he certainly looked exhausted. He was jogging all morning. 
He's in training for another race w ith me. Sloth: He did himself in. 
Achilles: Yes, but in vain. Maybe he coul d beat a Sloth ... but me? Never! Now-weren't

Achilles: Yes, but in vain. Maybe he coul d beat a Sloth ... but me? Never! Now-weren't 
you about to tell me what those letters "SAT" stand for? Sloth: As for those letters "SAT", you could never guess what they stand for. 
Achilles: Well, if they don't stand for what I thought, then my curiosity is piqued. 
Perhaps I'll think a little more about it. Say, how do you cook French fries? Sloth: In 
oil. 
Achilles: Oh, yes-I remember. I'll cut up this potato into strips an inch or two in length. Sloth: So short? Achilles: All right, already, I'll cut four-inch strips. Oh, boy, are these going to be good 
French fries! Too bad Mr. T won't be here to share them.

Strange Loops, Or Tangled Hierarchies 
 681CHAPTER XX 
 
Strange Loops, 
Or Tangled Hierarchies 
 
Can Machines Possess 
Originality? 
 
IN THE CHAPTER before last, I described Arthur Samuel's very successful checkers program-the one which can beat its designer. In light of that, it is interesting to hear how Samuel himself feels about the issue of computers and originality. The following extracts are taken from a rebuttal by Samuel, written in 1960, to an article by Norbert Wiener.

It is my conviction that machines cannot possess originality in the sense implied by Wiener in his thesis that "machines can and do transcend some of the limitations of their designers, and that in doing so they may be both effective and dangerous." .. . A machine is not a genie, it does not work by magic, it does not possess a will, and, Wiener to the contrary, nothing comes out which has not been put in, barring, of course, an infrequent case of malfunctioning... . The "intentions" which the machine seems to manifest are the intentions of the human programmer, as specified in advance, or they are subsidiary intentions derived from these, following rules specified by the programmer. We can even 
anticipate higher levels of 
abstraction, just as Wiener does, in which the program will not only modify the subsidiary intentions

abstraction, just as Wiener does, in which the program will not only modify the subsidiary intentions 
but will also modify the rules which are used in their derivation, or in which it will modify the ways in which it modifies the rules, and so on, or even in which one machine will design and construct a second machine with enhanced capabilities. However, and this is important, the machine will not and cannot [italics are his do any of these things until it has been instructed as to how to proceed. There is and logically there must always remain a complete hiatus between (i) any ultimate extension and elaboration in this process of carrying out man's wishes and (ii) the development within the machine of a will of' its own. To believe otherwise is either to believe in magic or to believe that the existence of man's will is an illusion and that man's actions are as mechanical as the machine's. Perhaps Wiener's article and my 
rebuttal have both been

rebuttal have both been 
mechanically determined, but this I refuse to believe.' 
 This reminds me of the Lewis Carroll Dialogue (the Two-Part Invention ); I'll 
try to explain why. Samuel bases his argument against machine consciousness (or will) on the notion that any 
mechanical instantiation of will would 
require an infinite regress . Similarly, 
Carroll's Tortoise argues that no step of reasoning, no matter how simple, can be done without invoking some rule on a higher level to justify the step in question. But that being

 682also a step of reasoning. one must resort 
to a yet higher-level rule, and so on. Conclusion: Reasoning involves an 
infinite regress . 
Of course something is wrong 
with the Tortoise's argument, and I believe something analogous is wrong with Samuel's argument. To show how the fallacies are analogous, I now shall "help the Devil", by arguing momentarily as Devil's advocate. (Since, as is well known, God helps those who help themselves, presumably the Devil helps all those, and only those, who don't help themselves. Does the Devil help himself?) Here are my devilish conclusions drawn from the Carroll Dialogue: 
 
The conclusion "reasoning is impossible" does not apply to people, because as is plain to anyone, we do manage to carry out many steps of reasoning, all the higher levels notwithstanding. That shows that we humans 
operate without  need of rules : we 
are "informal systems". On the other hand, as an argument against the possibility of any mechanical

are "informal systems". On the other hand, as an argument against the possibility of any mechanical  
instantiation of reasoning, it is valid, for any mechanical reasoning-system would have to depend on rules explicitly, and so it couldn't get off the ground unless it had metarules telling it when to apply its rules, metametarules telling it when to apply its metarules, and so on. We may conclude that the ability to reason can never be mechanized. It is a uniquely human capability. 
 What is wrong with this Devil's advocate point of view? It is obviously the assumption that a machine cannot do anything without having a rule telling it 
to do so . In fact, machines get around the

What is wrong with this Devil's advocate point of view? It is obviously the assumption that a machine cannot do anything without having a rule telling it 
to do so . In fact, machines get around the 
Tortoise's silly objections as easily as people do, and moreover for exactly the same reason: both machines and people are made of hardware which runs all by itself, according to the laws of physics. There is no need to rely on "rules that permit you to apply the rules", because the lowest -level rules-those without any 
"meta"'s in front-are embedded in the hardware, and they run without permission. Moral: The Carroll Dialogue doesn't say anything about the differences between people and machines, after all. (And indeed, reasoning is mechanizable.) 
So much for the Carroll 
Dialogue. On to Samuel's argument. Samuel's point, if I may caricature it, is this: 
 
No computer ever "wants" to do anything, because it was programmed by someone else. Only if it could program itself 
from zero on up-an absurdity-

would it have its own sense of desire. 
 In his argument, Samuel reconstructs the Tortoise's position, replacing "to reason" by "to want". He implies that behind any mechanization of desire, there has to be either an infinite regress or worse, a closed loop. If this is why computers have no will of their own, what about people? The same criterion would imply that

 683Unless a person designed himself 
and chose his own wants (as well as choosing to choose his own wants, etc.), he cannot be said to have a will of his own. 
 It makes you pause to think where your sense of having a will comes from. Unless you are a soulist, you'll probably say that it comes from your brain-a piece of hardware which you did not design or choose. And yet that doesn't diminish your sense that you want certain things, and not others. You aren't a "self-programmed object" (whatever that would be), but you still do have a sense of desires, and it springs from the physical substrate of your mentality. Likewise, machines may someday have wills despite the fact that no magic program spontaneously appears in memory from out of nowhere (a "self-programmed program"). They will have wills for much the same reason as you do-by reason of organization and structure on many levels of hardware 
and software. Moral: The Samuel

and software. Moral: The Samuel 
argument doesn't say anything about the differences between people and machines, after all. (And indeed, will will be mechanized.)  
Below Every Tangled Hierarchy 
Lies An Inviolate Level 
 
Right after the Two-Part Invention , I 
wrote that a central issue of this book would be: "Do words and thoughts follow formal rules?" One major thrust of the book has been to point out the many-leveledness of the mind/brain, and I have tried to show why the ultimate answer to the question is, "Yes-provided that you go down to the lowest level-the hardware-to find the rules." Now Samuel's statement brought 
up a concept which I want  to pursue. It is 
this: When we humans think, we certainly do change our own mental rules, and we change the rules that change the rules, and on and on-but these are, so to speak, "software rules". However, the rules at bottom  do not

this: When we humans think, we certainly do change our own mental rules, and we change the rules that change the rules, and on and on-but these are, so to speak, "software rules". However, the rules at bottom  do not 
change. Neurons run in the same simple way the whole time. You can't "think" your neurons into running some nonneural way, although you can make your mind change style or subject of thought. Like Achilles in the Prelude , 
Ant Fugue , you have access to your 
thoughts but not to your neurons. Software rules on various levels can change; hardware rules cannot-in fact, to their rigidity is due the software's flexibility! Not a paradox at all, but a fundamental, simple fact about the mechanisms of intelligence. 
This distinction between self-
modifiable software and inviolate hardware is what I wish to pursue in this final Chapter, developing it into a set of 
variations on a theme. Some of the

variations on a theme. Some of the 
variations may seem to be quite far-fetched, but I hope that by the time I close the loop by returning to brains, minds, and the sensation of consciousness, you will have found an invariant core in all the variations. 
My main aim in this Chapter is to 
communicate some of the images which help me to visualize how consciousness rises out of the jungle of neurons; to communicate a set of intangible intuitions, in the hope that

 684 
these intuitions are valuable and may perhaps help others a l4tle to come to clearer formulations of their own images of what makes minds run. I could not hope for more than that my own mind's blurry images of minds and images should catalyze the formation of sharper images of minds and images in other minds.  
A Self-Modifying Game 
 A first variation, then, concerns games in which on your turn, you may modify the rules. Think of chess. Clearly the rules stay the same, just the board position changes on each move. But let's invent a variation in which, on your turn, you can either make a move or change the rules. But how? At liberty? Can you turn it into checkers? Clearly such anarchy would be pointless. There must be some constraints. For instance, one version might allow you to redefine the knight's move. Instead of being 1-and-then-2, it could be m-and-then-n where m and n 
are arbitrary natural numbers; and on your turn you could change either m or n

are arbitrary natural numbers; and on your turn you could change either m or n 
by plus or minus 1.-So it could go from 1-2 to 1-3 to 0-3 to 0- 4 to 0-5 to 1-5 to 2-
5 ... Then there could be rules about redefining the bishop's moves, and the other pieces' moves as well. There could be rules about adding new squares, or deleting old squares .. . 
Now we have two layers of rules: 
those which tell how to move pieces, and those which tell how to change the rules. So we have rules and metarules. The next step is obvious: introduce metametarules by which we can change the metarules. It is not so obvious how to 
do this. The reason it is easy to 
formulate rules for moving pieces is that pieces move in a formalized space: the checkerboard. If you can devise a simple

formulate rules for moving pieces is that pieces move in a formalized space: the checkerboard. If you can devise a simple 
formal notation for expressing rules and metarules, then to manipulate them will be like manipulating strings formally, or even like manipulating chess pieces. To carry things to their logical extreme, you could even express rules and metarules as positions on auxiliary chess boards. Then an arbitrary chess position could be read as a game, or as a set of rules, or as a set of metarules, etc., depending on which interpretation you place on it. Of course, both players would have to agree on conventions for interpreting the notation. 
Now we can have any number of

Now we can have any number of 
adjacent chess boards: one for the game, one for rules, one for metarules, one for metametarules, and so on, as far as you care to carry it. On your turn, you may make a move on any one of the chess boards except the top-level one, using the rules which apply (they come from the next chess board up in the hierarchy). Undoubtedly both players would get quite disoriented by the fact that almost 
anything-though not everything!-can 
change. By definition, the top-level chess board can't be changed, because you don't have rules telling how to change it. It is inviolate . There is more 
that is inviolate: the conventions by which the different boards are interpreted, the agreement to take turns, the agreement that each person may change one chess board each turn-and you will find more if you examine the idea carefully.

 685Now it is possible to go considerably 
further in removing the pillars by which orientation is achieved. One step at a time. .. We begin by collapsing the whole array of boards into a single board. What is meant by this? There will be two ways of interpreting the board: (1) as pieces to be moved; (2) as rules for moving the pieces. On your turn, you move pieces-and perforce, you change rules! Thus, the rules constantly change themselves. Shades of Typogenetics-or for that matter, of real genetics. The distinction between game, rules, metarules, metametarules, has been lost. What was once a nice clean hierarchical setup has become a Strange Loop, Or Tangled Hierarchy. The moves change the rules, the rules determine the moves, round and round the mulberry bush ... There are still different levels, but the distinction between "lower" and "higher" has been wiped out. 
Now, part of what was inviolate

Now, part of what was inviolate 
has been made changeable. But there is still plenty that is inviolate. Just as 
before, there are conventions between 
you and your opponent by which you interpret the board as a collection of rules. There is the agreement to take turns-and probably other implicit conventions, as well. Notice, therefore, that the notion of different levels has survived, in an unexpected way. There is an Inviolate level-let's call it the I-level -
on which the interpretation conventions reside; there is also a Tangled level-the T-level -on which the Tangled Hierarchy 
resides. So these two levels are still hierarchical: the I-level governs what happens on the T-level, but the T-level does not and cannot affect the I-level. No matter that the T-level itself is a Tangled Hierarchy-it is still governed by a set of conventions outside of itself. And that is the important point. As you have no doubt imagined, 
there is nothing to st op us from doing the

there is nothing to st op us from doing the 
"impossible"-namely, tangling the I-level and the T-level by making the interpretation conventions themselves subject to revision, according to the position on the chess board. But in order to carry out such a "supertangling", you'd have to agree with your opponent on some further conventions connecting the two levels-and the act of doing so would create a new level, a new sort of inviolate level on top of the "supertangled" level (or underneath it, if you prefer). And this could continue going on and on. In fact, the `jumps" which are being made are very similar to those charted in the Birthday

Cantatatata , and in the repeated 
Gödelization applied to various improvements on TNT. Each time you think you have reached the end, there is some new variation on the theme of jumping out of the system which requires a kind of creativity to spot. 
 
The Authorship Triangle Again 
 But I am not interested in pursuing the strange topic of the ever more abstruse tanglings which can arise in self-modifying chess. The point of this has been to show, in a somewhat graphic way, how in any system there is always some "protected" level which is unassailable by the rules on other levels, no matter how tangled their interaction may be among themselves. An amusing riddle from Chapter IV illustrates this same idea in a slightly different context. Perhaps it will catch you off guard:

 686
 
        FIGURE 134. An “authorship triangle ” 
  
There are three authors-Z, T, and E. Now it happens that Z exists only in a novel by T. Likewise, T exists only in a novel by E.  And strangely, E, too, exists only in a 
novel-by Z, of course. Now, is such an "authorship triangle" really  possible? (See 
Fig. 134.) 
 Of course it's possible. But there's a trick ... All three authors Z, T, E, are themselves 
characters in another novel-by H! You can thi nk of the Z-T-E triangle as a Strange Loop, 
Or Tangled Hierarchy; but author H is outsi de of the space in which that tangle takes 
place-author H is in an inviolate space. Alt hough Z, T, and E all have access-direct or 
indirect-to each other, and can do dastardly th ings to each other in their various novels, 
none of them can touch H's life! They can 't even imagine him-no more than you can 
imagine the author of the book you're  a character in. If I were to draw author H, I would

imagine the author of the book you're  a character in. If I were to draw author H, I would 
represent him somewhere off the page. Of c ourse that would present a problem, since 
drawing a thing necessarily puts it onto the page ... Anyway, H is really outside of the 
world of Z, T, and E, and should be represented as being so.  
Escher's Drawing Hands  
 
Another classic variation on our theme is the Escher picture of Drawing Hands  (Fig. 
135). Here, a left hand (LH) draws a right ha nd (RH), while at the same time, RH draws 
LH. Once again, levels which ordinarily are s een as hierarchical-that which draws, and

LH. Once again, levels which ordinarily are s een as hierarchical-that which draws, and 
that which is drawn-turn back on each other,  creating a Tangled Hierarchy. But the theme 
of the Chapter is borne out, of course, since behind it all lurks the undrawn but drawing hand of M. C. Escher, creator  of both LH and RH. Escher is outside of the two-hand 
space, and in my schematic version of his picture (Fig. 136), you can see that explicitly. In this schematized representation of the Escher picture, you see the Strange Loop, Or 
Tangled Hierarchy at the top; also, you see the Inviolate Level below it, enabling it to

Tangled Hierarchy at the top; also, you see the Inviolate Level below it, enabling it to 
come into being. One could further Escheri ze the Escher picture, by taking a photograph 
of a hand drawing it. And so on.

 685
 
FIGURE 135. Drawing Hands, by M. C. Escher (lithograph, 1948 ). 
 
FIGURE 136. Abstract diagram of M. C. Escher's Drawing Hands. On top, a seeming 
paradox. Below, its resolution .

 686Brain and Mind: 
A Neural Tangle Supporting a Symbol Tangle 
 
Now we can relate this to the brain, as well as to Al programs. In our thoughts, symbols 
activate other symbols, and all interact heterarchically. Furthermore, the symbols may cause each other to change internally, in  the fashion of programs acting on other 
programs. The illusion is created, because of the Tangled Hierarchy of symbols, that 
there is no inviolate level. one thinks there is  no such level because that level is shielded 
from our view. 
If it were possible to schematize this w hole image, there would be a gigantic 
forest of symbols linked to each other by tangl y lines like vines in a tropical jungle-this 
would be the top level, the Tangled Hierarc hy where thoughts really flow back and forth. 
This is the elusive level of mind: the analogue  to LH and RH. Far below in the schematic 
picture, analogous to the invisible "prime m over" Escher, there would be a representation

picture, analogous to the invisible "prime m over" Escher, there would be a representation 
of the myriad neurons-the "inviolate substrat e" which lets the tangle above it come into 
being. Interestingly, this other level is itself a tangle in a literal sense-billions of cells and hundreds of billions of axons, joining them all together. 
This is an interesting case where a soft ware tangle, that of the symbols, is 
supported by a hardware tangle, that of th e neurons. But only the symbol tangle is a 
Tangled Hierarchy. The neural tangle is* ust a "s imple" tangle. This distinction is pretty

Tangled Hierarchy. The neural tangle is* ust a "s imple" tangle. This distinction is pretty 
much the same as that between Strange Loops and feedback, which I mentioned in 
Chapter XVI. A Tangled Hierarchy occurs when  what you presume are clean hierarchical 
levels take you by surprise and fold back in a hierarchy-violating way. The surprise 
element is important; it is the reason I call St range Loops "strange". A simple tangle, like

element is important; it is the reason I call St range Loops "strange". A simple tangle, like 
feedback, doesn't involve violations of presumed level distinctions. An example is when you're in the shower and you wash your left arm with your right, and then vice versa. 
There is no strangeness to the image. Escher  didn't choose to draw hands drawing hands 
for nothing! 
Events such as two arms washing each ot her happen all the time in the world, and 
we don't notice them particularly. I say some thing to you, then you say something back to 
me. Paradox % No; our perceptions of each othe r didn't involve a hier archy to begin with,

me. Paradox % No; our perceptions of each othe r didn't involve a hier archy to begin with, 
so there is no sense of strangeness. 
On the other hand, where language does create strange loops is when it talks 
about itself, whether directly or indirectly. Here, something in the system jumps out and 
acts on the system, as if it were outside the system. What bothers us is perhaps an ill-
defined sense of topological wrongness: the inside -outside distinction is being blurred, as 
in the famous shape called a "Klein bottle". Even though the system is an abstraction, our 
minds use spatial imagery with a sort of mental topology. 
Getting back to the symbol tangle, if we  look only at it, and forget the neural

Getting back to the symbol tangle, if we  look only at it, and forget the neural 
tangle, then we seem to see a self-programme d object-in just the same way as we seem to 
see a self-drawn picture if we look at Drawing Hands  and somehow fall for the illusion, 
by forgetting the existence of Escher. For

 687the picture, this is unlikely-but for humans a nd the way they look at their minds, this is 
usually what happens. We feel self-programmed. Indeed, we couldn't feel any other way, 
for we are shielded from the lower levels, the neural tangle. Our thoughts seem to run about in their own space, creating new thought s and modifying old ones, and we never 
notice any neurons helping us out! But that is to be expected. We can't. 
An analogous double-entendre can happen with LISP  programs that are designed 
to reach in and change their own structure. If you look at them on the LISP  level, you 
will say that they change themselves; but if you shift levels, and think of LISP  programs 
as data to the LISP  interpreter (see Chapter X), then in  fact the sole program that is 
running is the interpreter, and the changes be ing made are merely changes in pieces of 
data. The LISP  interpreter itself is shielded from changes.

data. The LISP  interpreter itself is shielded from changes. 
How you describe a tangled situation of th is sort depends how far back you step 
before describing. If you step far enough back, you can often see the clue that allows you 
to untangle things. 
 
Strange Loops in Government 
 A fascinating area where hierarchies tangle is government-particularly in the courts. 
Ordinarily, you think of two disputants ar guing their cases in court, and the court 
adjudicating the matter. The court is on a diffe rent level from the disputants. But strange

adjudicating the matter. The court is on a diffe rent level from the disputants. But strange 
things can start to happen when the courts  themselves get entangled in legal cases. 
Usually there is a higher court which is outside the dispute. Even if two lower courts get involved in some sort of stra nge fight, with each one claiming jurisdiction over the other, 
some higher court is outside, and in some  sense it is analogous to the inviolate 
interpretation conventions which we discussed in the warped version of chess. 
But what happens when there is no higher court, and the Supreme Court itself gets

But what happens when there is no higher court, and the Supreme Court itself gets 
all tangled up in legal troubles? This sort of snarl nearly happened in the Watergate era. 
The President threatened to obey only a "def initive ruling" of the Supreme Court-then 
claimed he had the right to decide what is "definitive". Now that threat never was made 
good; but if it had been, it would have touc hed off a monumental confrontation between 
two levels of government, each of which, in some ways, can validly claim to be "above" the other-and to whom is there recourse to decide which one is right? To say "Congress"

two levels of government, each of which, in some ways, can validly claim to be "above" the other-and to whom is there recourse to decide which one is right? To say "Congress" 
is not to settle the matter, for Congress might command the President to obey the Supreme Court, yet the President might still refu se, claiming that he has the legal right to 
disobey the Supreme Court (and Congress!) under certain circumstances. This would create a new court case, and would throw th e whole system into disarray, because it 
would be so unexpected, so Tangled-so Strange! 
The irony is that once you hit your head against the ceiling like this, where you 
are prevented from jumping out of the system  to a yet higher authority, the only recourse

are prevented from jumping out of the system  to a yet higher authority, the only recourse 
is to forces which seem less well defined by

 688rules, but which are the only source of higher-level rules anyway: the lower-level rules, 
which in this case means the general reaction of  society. It is well to remember that in a 
society like ours, the legal system is, in a sense, a polite gesture granted collectively by 
millions of people-and it can be overridden just as easily as a river can overflow its banks. Then a seeming anarchy takes over; but anarchy has its own kinds of rules, no less 
than does civilized society: it is just that they operate from the bottom up, not from the 
top down. A student of anarchy could try to discover rules according to which anarchic 
situations develop in time, and very likely there are some such rules. 
An analogy from physics is useful here. As was mentioned earlier in the book, 
gases in equilibrium obey simple laws c onnecting their temperature, pressure, and 
volume. However, a gas can violate those laws  (as a President can violate laws)-provided

volume. However, a gas can violate those laws  (as a President can violate laws)-provided 
it is not in a state of equilibrium. In nonequilib rium situations, to describe what happens, 
a physicist has recourse only to statistical mechanics-that is, to a level of description which is not macroscopic, for the ultimate e xplanation of a gas's behavior always lies on 
the molecular level, just as the ultimate explanation of a society's political behavior always lies at the "grass roots level". The field of nonequilibrium thermodynamics

the molecular level, just as the ultimate explanation of a society's political behavior always lies at the "grass roots level". The field of nonequilibrium thermodynamics 
attempts to find macroscopic laws to describe  the behavior of gases (and other systems) 
which are out of equilibrium. It is the anal ogue to the branch of political science which 
would search for laws governing anarchical societies. 
Other curious tangles which arise in govern ment include the FBI investigating its 
own wrongdoings, a sheriff going to jail while in office, the self-application of the parliamentary rules of procedure, and so on. One of the most curious legal cases I ever

own wrongdoings, a sheriff going to jail while in office, the self-application of the parliamentary rules of procedure, and so on. One of the most curious legal cases I ever 
heard of involved a person who claimed to have psychic powers. In fact, he claimed to be 
able to use his psychic powers to detect pers onality traits, and thereby to aid lawyers in 
picking juries. Now what if this "psychic" has to stand trial himself one day? What effect 
might this have on a jury member who believe s staunchly in ESP? How much will he feel 
affected by the psychic (whether or not the ps ychic is genuine)? The territory is ripe for 
exploitation-a great area for selffulfilling prophecies.

exploitation-a great area for selffulfilling prophecies. 
 
Tangles Involving Science and the Occult 
 Speaking of psychics and ESP, another s phere of life where strange loops abound is 
fringe science. What fringe science does is  to call into question many of the standard 
procedures or beliefs of orthodox science,  and thereby challenge the objectivity of 
science. New ways of interpreting evidence th at rival the established ones are presented. 
But how do you evaluate a way of interpreting evidence? Isn't this precisely the problem of objectivity all over again, just on a higher plane? Of course. Lewis Carroll's infinite-

regress paradox appears in a new guise. The To rtoise would argue that if you want to 
show that A is a fact, you need evidence: B.  But what makes you sure that B is evidence 
of A?' To show that, you need meta-

 689evidence: C. And for the validity of that meta-evidence, you need metameta-evidence-
and so on, ad nauseam. Despite this argument, people have an intuitive sense of evidence. 
This is because-to repeat an old refrain-people have built-in hardware in their brains that includes some rudimentary ways of interp reting evidence. We can build on this, and 
accumulate new ways of interpreting evidence;  we even learn how and when to override 
our most basic mechanisms of evidence interp retation, as one must, for example, in trying 
to figure out magic tricks. 
Concrete examples of evidence dilemma s crop up in regard to many phenomena 
of fringe science. For instance, ESP ofte n seems to manifest itself outside of the 
laboratory, but when brought into the laborator y, it vanishes mysteriously. The standard 
scientific explanation for this is that ESP is a nonreal phenomenon which cannot stand up

scientific explanation for this is that ESP is a nonreal phenomenon which cannot stand up 
to rigorous scrutiny. Some (by no means all) believers in ESP have a peculiar way of 
fighting back, however. They say, "No, ESP is real; it simply goes away when one tries 
to observe it scientifically-it is contrary to th e nature of a scientific worldview." This is 
an amazingly brazen technique, which we mi ght call "kicking the problem upstairs". 
What that means is, instead of questioning the matter at hand, you call into doubt theories 
belonging to a higher level of credibility. The believers in ESP insinuate that what is

belonging to a higher level of credibility. The believers in ESP insinuate that what is 
wrong is not their ideas, but the belief system of science. This is a pretty grandiose claim, 
and unless there is overwhelming evidence for it,  one should be skeptical of it. But then 
here we are again, talking about "overwhelmi ng evidence" as if everyone agreed on what 
that means!  
The Nature of Evidence 
 The Sagredo-Simplicio-Salviati tangle, mentioned in Chapters XIII and XV, gives another example of the complexities of evalua tion of evidence. Sagredo tries to find some

 The Sagredo-Simplicio-Salviati tangle, mentioned in Chapters XIII and XV, gives another example of the complexities of evalua tion of evidence. Sagredo tries to find some 
objective compromise, if possible, between th e opposing views of Simplicio and Salviati. 
But compromise may not always be possible.  How can one compromise "fairly" between 
right and wrong? Between fair and unfair?  Between compromise and no compromise? 
These questions come up over and over again in disguised form in arguments about 
ordinary things. 
Is it possible to define what evidence is? Is it possible to lay down laws as to how 
to make sense out of situations? Probably not, for any rigid rules would undoubtedly have

to make sense out of situations? Probably not, for any rigid rules would undoubtedly have 
exceptions, and nonrigid rules are not rules. Having an intelligent AI program would not 
solve the problem either, for as an evidence processor, it would not be any less fallible 
than humans are. So, if eviden ce is such an intangible thing after all, why am I warning 
against new ways of interpreting evidence? Am  I being inconsistent? In this case, I don't 
think so. My feeling is that there are guidelines which one can give, and out of them an organic synthesis can be made. But inevita bly some amount of judgment and intuition

think so. My feeling is that there are guidelines which one can give, and out of them an organic synthesis can be made. But inevita bly some amount of judgment and intuition 
must enter the picture-things which are differe nt in different people. They will also be 
different in

 690different AI programs. Ultimately, there are complicated criteria for deciding if a method 
of evaluation of evidence is good. One involves the "usefulness" of ideas which are arrived at by that kind of reasoning. Modes of thought which lead to useful new things in 
life are deemed "valid" in some sense. But this word "useful" is extremely subjective. 
My feeling is that the process by which we d ecide what is valid or what is true is 
an art; and that it relies as deeply on a sens e of beauty and simplicity as it does on rock-
solid principles of logic or reasoning or  anything else which can be objectively 
formalized. I am not saying either (1) truth is  a chimera, or (2) human intelligence is in 
principle not programmable. I am saying (1) truth is too elusive for any human or any 
collection of humans ever to attain fully; a nd (2) Artificial Intelligence, when it reaches 
the level of human intelligence-or even if  it surpasses it-will still be plagued by the

the level of human intelligence-or even if  it surpasses it-will still be plagued by the 
problems of art, beauty, and simplicity, and w ill run up against these things constantly in 
its own search for knowledge and understanding. 
"What is evidence?" is not just a philos ophical question, for it intrudes into life in 
all sorts of places. You are faced with an ex traordinary number of choices as to how to 
interpret evidence at every moment. You can hardly go into a bookstore (or these days, 
even a grocery store!) without seeing books on clairvoyance, ESP, UFO's, the Bermuda

even a grocery store!) without seeing books on clairvoyance, ESP, UFO's, the Bermuda 
triangle, astrology, dowsing, evolution vers us creation, black holes, psi fields, 
biofeedback, transcendental meditation, new th eories of psychology ... In science, there 
are fierce debates about catastrophe theory, elem entary particle theory, black holes, truth 
and existence in mathematics, free will, Artificial Intelligence, reductionism versus 
holism ... On the more pragmatic side of lif e, there are debates over the efficacy of

holism ... On the more pragmatic side of lif e, there are debates over the efficacy of 
vitamin C or of laetrile, over the real size of oil reserves (either underground or stored), over what causes inflation and unemploym ent-and on and on. There is Buckminster 
Fullerism, Zen Buddhism, Zeno's paradoxes, psyc hoanalysis, etc., etc. From issues as 
trivial as where books ought to be shelved in a stor e, to issues as vital as what ideas are to 
be taught to children in schools, ways of interpreting evidence play an inestimable role. 
 
Seeing Oneself 
 One of the most severe of a ll problems of evidence interpretation is that of trying to

 One of the most severe of a ll problems of evidence interpretation is that of trying to 
interpret all the confusing signals from the outside as to who one is. In this case, the potential for intralevel and interlevel conflict is tremendous. The psychic mechanisms have to deal simultaneously with the individual's internal need for self-esteem and the constant flow of evidence from the outside affecting the self-image. The result is that 
information flows in a complex swirl between  different levels of the personality; as it 
goes round and round, parts of it get magnified, reduced, negated, or otherwise distorted, 
and then those parts in turn get further subject ed to the same sort of swirl, over and over

and then those parts in turn get further subject ed to the same sort of swirl, over and over 
again-all of this in an attempt to reconcile what is, with what we wish were (see Fig. 81).

 691The upshot is that the total picture of  "who I am" is integrated in some 
enormously complex way inside the entire mental  structure, and contains in each one of 
us a large number of unresolved, possibly unresolvable, inconsistencies. These 
undoubtedly provide much of the dynamic tensi on which is so much a part of being 
human. Out of this tension between the inside  and outside notions of who we are come 
the drives towards various goals that make each of us unique. Thus, ironically, something which we all have in common-the fact of be ing self-reflecting conscious beings-leads to 
the rich diversity in the ways we have of in ternalizing evidence about all sorts of things, 
and in the end winds up being one of the major forces in creating distinct individuals. 
 
Gödel’s Theorem and Other Disciplines 
 It is natural to try to draw parallels betw een people and sufficiently complicated formal

 It is natural to try to draw parallels betw een people and sufficiently complicated formal 
systems which, like people, have "self-images " of a sort. Gödel’s Theorem shows that 
there are fundamental limitations to consistent formal systems with self-images. But is it 
more general? Is there a "Gödel’s Theorem of psychology", for instance? 
If one uses Gödel’s Theorem as a metaphor, as a source of inspiration, rather than 
trying to translate it literally into the language  of psychology or of any other discipline,

trying to translate it literally into the language  of psychology or of any other discipline, 
then perhaps it can suggest new truths in psychology or other areas. But it is quite unjustifiable to translate it directly into a stat ement of another discipline and take that as 
equally valid. It would be a large mistake to th ink that what has been worked out with the 
utmost delicacy in mathematical logic shoul d hold without modification in a completely 
different area.  
Introspection and Insanity: A Gödelian Problem 
 I think it can have suggestive value to tran slate Gödel’s Theorem into other domains,

 I think it can have suggestive value to tran slate Gödel’s Theorem into other domains, 
provided one specifies in advance that the translations are metaphorical and are not 
intended to be taken literally. That having b een said, I see two major ways of using 
analogies to connect Gödel’s Theorem and human thoughts. One involves the problem of wondering about one's sanity. How can you figure out if you are sane? This is a Strange 
Loop indeed. Once you begin to question your own sanity, you can get trapped in an 
ever-tighter vortex of self-fulfilling pr ophecies, though the process is by no means 
inevitable. Everyone knows that the insane in terpret the world via their own peculiarly

inevitable. Everyone knows that the insane in terpret the world via their own peculiarly 
consistent logic; how can you tell if your ow n logic is "peculiar" or not, given that you 
have only your own logic to judge itself? I don't see any answer. I am just reminded of 
Gödel’s second Theorem, which implies that the only versions of formal number theory 
which assert their own consistency are inconsistent ...

 692Can We Understand Our Own" Minds or Brains? 
 
The other metaphorical analogue to Gödel’ s Theorem which I find provocative suggests 
that ultimately, we cannot understand our own minds/ brains. This is such a loaded, 
many-leveled idea that one must be extremely cautious in proposing it. What does "understanding our own minds/brains" mean? It  could mean having a general sense of 
how they work, as mechanics have a sense of how cars work. It could mean having a 
complete explanation for why people do any a nd all things they do. It could mean having 
a complete understanding of the physical struct ure of one's own brain on all levels. It 
could mean having a complete wiring diagram of a brain in a book (or library or computer). It could mean knowing, at every in stant, precisely what is happening in one's 
own brain on the neural level-each firing, each  synaptic alteration, and so on. It could

own brain on the neural level-each firing, each  synaptic alteration, and so on. It could 
mean having written a program which passes the Turing test. It could mean knowing 
oneself so perfectly that such notions as  the subconscious and the intuition make no 
sense, because everything is out in the open. It could mean any number of other things. 
Which of these types of self-mirroring, if  any, does the self-mirroring in Gödel’s 
Theorem most resemble? I would hesitate to say. Some of them are quite silly. For instance, the idea of being able to monitor your  own brain state in all its detail is a pipe

Theorem most resemble? I would hesitate to say. Some of them are quite silly. For instance, the idea of being able to monitor your  own brain state in all its detail is a pipe 
dream, an absurd and uninteresting proposition to start with; and if Gödel’s Theorem 
suggests that it is impossible, that is hard ly a revelation. On the other hand, the age-old 
goal of knowing yourself in some profound way-let us call it "understanding your own 
psychic structure"-has a ring of plausibility to it. But might there not be some vaguely 
Godelian loop which limits the depth to which any individual can penetrate into his own psyche? Just as we cannot see our faces with our own eyes, is it not reasonable to expect

Godelian loop which limits the depth to which any individual can penetrate into his own psyche? Just as we cannot see our faces with our own eyes, is it not reasonable to expect 
that we cannot mirror our complete mental st ructures in the symbols which carry them 
out? 
All the limitative Theorems of metamathem atics and the theory of computation 
suggest that once the ability to represent your own structure has reached a certain critical 
point, that is the kiss of death: it guarantees  that you can never represent yourself totally. 
Gödel’s Incompleteness Theorem, Church's  Undecidability Theorem, Turing's Halting 
Theorem, Tarski's Truth Theorem-all have th e flavor of some ancient fairy tale which

out? 
Theorem, Tarski's Truth Theorem-all have th e flavor of some ancient fairy tale which 
warns you that "To seek self-knowledge is to embark on a journey which ... will always be incomplete, cannot be charted on any map, will never halt, cannot be described." But do the limitative Theorems have any bearing on people? Here is one way of arguing the case. Either I am consistent or I am inc onsistent. (The latter is much more likely, but 
for completeness' sake, I consider both possib ilities.) If I am consistent, then there are 
two cases. (1) The "low-fidelity" case: my se lf-understanding is below a certain critical

two cases. (1) The "low-fidelity" case: my se lf-understanding is below a certain critical 
point. In this case, I am incomplete by hypothe sis. (2) The "high-fidelity" case: My self-
understanding has reached the critical point  where a metaphorical analogue of the 
limitative Theorems does apply, so my self-understanding

 693undermines itself in a Gödelian way, and I am incomplete for that reason. Cases (1) and 
(2) are predicated on my being 100 per cent cons istent-a very unlikely state of affairs. 
More likely is that I am inconsistent-but th at's worse, for then inside me there are 
contradictions, and how can I ever understand that? 
Consistent or inconsistent, no one is exempt from the mystery of the self. 
Probably we are all inconsistent. The world is just too complicated for a person to be able 
to afford the luxury of reconciling all of his beliefs with each other. Tension and 
confusion are important in a world where ma ny decisions must be made quickly, Miguel 
de Unamuno once said, "If a person never contra dicts himself, it must be that he says 
nothing." I would say that we all are in th e same boat as the Zen master who, after 
contradicting himself several times in a row, said to the confused Doko, "I cannot understand myself."

contradicting himself several times in a row, said to the confused Doko, "I cannot understand myself." 
 
Gödel’s Theorem and Personal Nonexistence 
 Perhaps the greatest contradiction in our liv es, the hardest to handle, is the knowledge 
"There was a time when I was not alive, and there will come a time when I am not alive." 
On one level, when you "step out of yourself " and see yourself as "just another human 
being", it makes complete sense. But on anothe r level, perhaps a deeper level, personal 
nonexistence makes no sense at all. All that we know is embedded inside our minds, and

nonexistence makes no sense at all. All that we know is embedded inside our minds, and 
for all that to be absent from the univers e is not comprehensible. This is a basic 
undeniable problem of life; perhaps it is the best metaphorical analogue of Gödel’s 
Theorem. When you try to imagine your own nonexistence, you have to try to jump out of yourself, by mapping yourself onto someone el se. You fool yourself into believing that 
you can import an outsider's view of yourself into you, much as TNT  "believes" it 
mirrors its own metatheory inside itself. But TNT  only contains its own metatheory up to

mirrors its own metatheory inside itself. But TNT  only contains its own metatheory up to 
a certain extent-not fully. And as for you, though you may imagine that you have jumped out of yourself, you never can actually do so-no more than Escher's dragon can jump out of its native two-dimensional plane into three dimensions. In any case, this contradiction 
is so great that most of our lives we ju st sweep the whole mess under the rug, because 
trying to deal with it just leads nowhere. 
Zen minds, on the other hand, revel in this  irreconcilability. Over and over again, 
they face the conflict between the Eastern belie f: "The world and I are one, so the notion

they face the conflict between the Eastern belie f: "The world and I are one, so the notion 
of my ceasing to exist is a contradiction in  terms" (my verbalization is undoubtedly too 
Westernized-apologies to Zenists), and the West ern belief: "I am just part of the world, 
and I will die, but the world will go on without me."  
Science and Dualism 
 Science is often criticized as being too "Western" or "dualistic"-that is, being permeated by the dichotomy between subject and object, or observer

 694and observed. While it is true that up until this century, science was exclusively 
concerned with things which can be readily  distinguished from their human observers-
such as oxygen and carbon, light and heat, stars and planets, accelerations and orbits, and 
so on-this phase of science was a necessary prelude to the more modern phase, in which life itself has come under investigation. Step by step, inexorably, "Western" science has moved towards investigation of the human mind-which is to say, of the observer. Artificial Intelligence research is the furthest step so far along that route. Before AI came 
along, there were two major previews of th e strange consequences of the mixing of 
subject and object in science. One was the revolution of quantum mechanics, with its epistemological problems involving the interf erence of the observer with the observed.

subject and object in science. One was the revolution of quantum mechanics, with its epistemological problems involving the interf erence of the observer with the observed. 
The other was the mixing of subject and object in metamathematics, beginning with Godel's Theorem and moving through all th e other limitative'Theorems we have 
discussed. Perhaps the next step after Al will be the self-application of science: science studying itself as an object. This is a diffe rent manner of mixing subject and object-
perhaps an even more tangled one than that of humans studying their own minds. 
By the way, in passing, it is interesting to note that all results essentially 
dependent on the fusion of subject and object have been limitative results. In addition to the limitative Theorems, there is Heisenberg's  uncertainty principle, which says that

dependent on the fusion of subject and object have been limitative results. In addition to the limitative Theorems, there is Heisenberg's  uncertainty principle, which says that 
measuring one quantity renders impossible the simultaneous measurement of a related quantity. I don't know why all these results are limitative. Make of it what you will. 
 
Symbol vs. Object in Modern Music and Art 
 Closely linked with the subject-object dichot omy is the symbol-object dichotomy, which 
was explored in depth by Ludwig Wittgenstein in  the early part of this century. Later the 
words "use" and "mention" were adopted to make the same distinction. Quine and others have written at length about the connection be tween signs and what they stand for. But 
not only philosophers have devoted much thought to this deep and abstract matter. In our

not only philosophers have devoted much thought to this deep and abstract matter. In our 
century both music and art have gone thr ough crises which reflect a profound concern 
with this problem. Whereas music and painti ng, for instance, have traditionally expressed 
ideas or emotions through a vocabulary of "symbols" (i.e. visual images, chords, rhythms, or whatever), now there is a tendency to explore the capacity of music and art to not express anything just to be. This means to  exist as pure globs of paint, or pure sounds, 
but in either case drained of all symbolic value.

but in either case drained of all symbolic value. 
In music, in particular, John Cage has been very influential in bringing a Zen-like 
approach to sound. Many of his pieces convey a disdain for "use" of sounds-that is, using 
sounds to convey emotional states-and an exultation in "mentioning" sounds-that is, 
concocting arbitrary juxtapositions  of sounds without regard to  any previously formulated 
code by which a listener could decode them into a message. A typical example is "Imaginary Landscape no. 4", the polyradio piece described in Chapter VI. I may not

 695be doing Cage justice, but to me it seems that  much of his work has been directed at 
bringing meaninglessness into music, and in some sense, at making that meaninglessness 
have meaning. Aleatoric music is a typical e xploration in that direction. (Incidentally, 
chance music is a close cousin to the much later notion of "happenings" or "be-in"' s.) There are many other contemporary composer s who are following Ca ge's lead, but few 
with as much originality. A piece by Anna  Lockwood, called "Piano Burning", involves 
just that-with the strings stretched to maximum tightness, to make them snap as loudly as possible; in a piece by LaMonte Young, the noi ses are provided by shoving the piano all 
around the stage and through obstacles, like a battering ram. 
Art in this century has gone through many convulsions of this general type. At 
first there was the abandonment of represen tation, which was genuinely revolutionary:

first there was the abandonment of represen tation, which was genuinely revolutionary: 
the beginnings of abstract art. A gradual swoop from pure representation to the most 
highly abstract patterns is revealed in the work of Piet Mondrian. After the world was 
used to nonrepresentational art, then surr ealism came along. It was a bizarre about-face, 
something like neoclassicism in music, in which extremely representational art was "subverted" and used for altogether new reas ons: to shock, confuse, and amaze. This 
school was founded by Andre Breton, and was locat ed primarily in France; some of its

school was founded by Andre Breton, and was locat ed primarily in France; some of its 
more influential members were Dali, Magritte, de Chirico, Tanguy. 
 
Magritte's Semantic Illusions 
 
Of all these artists, Magritte was the most c onscious of the symbol-object mystery (which 
I see as a deep extension of the use-mention distinction). He uses it to evoke powerful responses in viewers, even if the viewers do not verbalize the distinction this way. For 
example, consider his very strange va riation on the theme of still life, called Common 
Sense  (Fig. 137). 
 
FIGURE 137. Common Sense, by Rene Magritte (1945-46).

 696
 
FIGURE 138. The Two Mysteries, by Rene Magritte (1966) . 
 Here, a dish filled with fruit, ordinarily the kind of thing represented inside a still life, is 
shown sitting on top of a blank canvas. The c onflict between the symbol and the real is 
great. But that is not the full irony, for of course  the whole thing is itself just a painting-in 
fact, a still life with nonstandard subject matter. 
Magritte's series of pipe paintings is fascinating and perplexing. Consider The 
Two Mysteries  (Fig. 138). Focusing on the inner painting, you get the message that 
symbols and pipes are different. Then your  glance moves upward to the "real" pipe 
floating in the air-you perceive that it is real, while the other one is just a symbol. But that 
is of course totally wrong: both of them ar e on the same flat surface before your eyes. 
The idea that one pipe is in a twice-nested  painting, and therefore somehow "less real"

The idea that one pipe is in a twice-nested  painting, and therefore somehow "less real" 
than the other pipe, is a complete fallac y. Once you are willing to "enter the room", you 
have already been tricked: you've fallen for image as reality. To be consistent in your 
gullibility, you should happily go one level fu rther down, and confuse image-within-
image with reality. The only way not to be su cked in is to see both pipes merely as 
colored smudges on a surface a few inches in front of your nose. Then, and only then, do 
you appreciate the full meaning of the writte n message "Ceci West pas une pipe"-but

you appreciate the full meaning of the writte n message "Ceci West pas une pipe"-but 
ironically, at the very instant everything turns to smudges, the writing too turns to smudges, thereby losing its meaning! In other wo rds, at that instant, the verbal message 
of the painting self-destructs in a most Gödelian way.

 697
 
FIGURE 139. Smoke Signal. [Drawing by the author. ] 
 The Air and the Song (Fig. 82), taken from  a series by Magritte, accomplishes all 
that The Two Mysteries  does, but in one level inst ead of two. My drawings Sm oke Signal 
and Pipe Dream  (Figs. 139 and 140) constitute "Variations on a Theme of Magritte". Try 
staring at Smoke Signal  for a while. Before long, you should be able to make out a hidden 
message saying, "Ceci n'est pas un message". Thus, if you find the message, it denies itself-yet if you don't, you miss the point entirel y. Because of their indirect self-snuffing, 
my two pipe pictures can be loosely ma pped onto Gödel’s G-thus giving rise to a 
"Central Pipemap", in the same spirit as the other "Central Xmaps": Dog, Crab, Sloth. 
A classic example of use-mention confusi on in paintings is the occurrence of a 
palette in a painting. Whereas the palette is  an illusion created by the representational

palette in a painting. Whereas the palette is  an illusion created by the representational 
skill of the painter, the paints on the painte d palette are literal daubs of paint from the 
artist's palette. The paint plays itself-it does not symbolize anything else. In Don 
Giovanni , Mozart exploited a related trick: he wr ote into the score explicitly the sound of 
an orchestra tuning up. Similarly, if I want th e letter 'I' to play itself (and not symbolize 
me), I put 'I' directly into my text; then I enclose `I' between quotes. What results is ''I" 
(not `I', nor "`I"'). Got that?

 698
 
FIGURE 140. Pipe Dream. [Drawing by the author.]  
 
The "Code" of Modern Art 
 A large number of influences, which no one could hope to pin down completely, led to 
further explorations of the symbol-object dualism in art. There is no doubt that John 
Cage, with his interest in Zen, had a prof ound influence on art as well as on music. His 
friends jasper Johns and Robert Rauschenbe rg both explored the distinction between 
objects and symbols by using objects as symbols for themselves-or, to flip the coin, by 
using symbols as objects in themselves. All of this was perhaps intended to break down 
the notion that art is one step removed from  reality-that art speaks in "code", for which 
the viewer must act as interpreter. The idea wa s to eliminate the step of interpretation and 
let the naked object simply be, period. ("Peri od"-a curious case of use-mention blur.) 
However, if this was the intention, it was a monumental flop, and perhaps had to be.

However, if this was the intention, it was a monumental flop, and perhaps had to be. 
Any time an object is exhibited in a ga llery or dubbed a "work", it acquires an 
aura of deep inner significance-no matter how much the viewer has been warned not to 
look for meaning. In fact, there is a backfiri ng effect whereby the more that viewers are 
told to look at these objects without mystifi cation, the more mystified the viewers get. 
After all, if a

 699wooden crate on a museum floor is just a wooden crate on a museum floor, then why 
doesn't the janitor haul it out back and throw it in the garbage? Why is the name of an 
artist attached to it? Why did the artist want to  demystify art? Why isn't that dirt clod out 
front labeled with an artist's name? Is this a hoax? Am I crazy, or are artists crazy? More 
and more questions flood into the viewer's mi nd; he can't help it. This is the "frame 
effect" which art-Art-automatically creates. There is no way to suppress the wonderings 
in the minds of the curious. 
Of course, if the purpose is to instill a Zen-like sense of the world as devoid of 
categories and meanings, then perhaps such art is merely intended to serve-as does intellectualizing about Zen-as a catalyst to inspire the viewer to go out and become 
acquainted with the philosophy which reject s "inner meanings" and embraces the world

acquainted with the philosophy which reject s "inner meanings" and embraces the world 
as a whole. In this case, th e art is self-defeating in the short run, since the viewers do 
ponder about its meaning, but it achieves its aim with a few people in the long run, by introducing them to its sources. But in either case, it is not true that there is no code by 
which ideas are conveyed to the viewer. Actually, the code is a much more complex thing, involving statements about the absence of codes and so forth-that is, it is part code, 
part metacode, and so on. There is a Tangled  Hierarchy of messages being transmitted by

part metacode, and so on. There is a Tangled  Hierarchy of messages being transmitted by 
the most Zen-like art objects, which is perhaps why so many find modern art so inscrutable. 
 
Ism Once Again 
 Cage has led a movement to break the boundari es between art and nature. In music, the 
theme is that all sounds are equal-a sort of acoustical democracy. Thus silence is just as 
important as sound, and random sound is just as important as organized sound. Leonard 
B. Meyer, in his book Music, the Arts, and Ideas , has called this movement in music 
"transcendentalism", and states:

"transcendentalism", and states:  
If the distinction between art and nature is mistaken, aesthetic valuation is irrelevant. One should no more judge the va lue of a piano sonata than one should 
judge the value of a stone, a thunderstorm, or a starfish. "Categorical statements, 
such as right and wrong, beautiful or ugly, typical of the rationalistic thinking of 
tonal aesthetics," writes Luciano Berio [a contemporary composer, "are no longer useful in understanding why and how a co mposer today works on audible forms 
and musical action."

and musical action." 
 
Later, Meyer continues in describing the philosophical position of transcendentalism: 
 
... all things in all of time and space are inextricably connected with one 
another. Any divisions, classifications, or  organizations discovered in the universe 
are arbitrary. The world is a complex, continuous, single event .2 [Shades of Zeno!] 
 I find "transcendentalism" too bulky a name for this movement. In its place, I use 
"ism". Being a suffix without a prefix, it suggests an ideology

 700
 
FIGURE 141. The Human Condition I, by Rene Magritte (1933) .

 701without ideas-which, however you interpret it,  is probably the case. And since."ism" 
embraces whatever is, its name is quite fitting. In "ism" thL- word "is" is half mentioned, 
half used; what could be more appropriate? Ism is the spirit of Zen in art. And just as the 
central problem of Zen is to unmask the self, the central problem of art in this century 
seems to be to figure out what art is. All th ese thrashings-about are part of its identity 
crisis. 
We have seen that the use-mention dichotomy, when pushed, turns into the 
philosophical problem of symbol-object dualis m, which links it to the mystery of mind. 
Magritte wrote about his painting The Human Condition  I (Fig. 141): 
 
I placed in front of a window, seen from  a room, a painting representing exactly 
that part of the landscape which was hidde n from view by the painting. Therefore, 
the tree represented in the painting hid from  view the tree situated behind it, outside

crisis. 
 
the tree represented in the painting hid from  view the tree situated behind it, outside 
the room. It existed for the spectator, as it were, simultaneously in his mind, as both inside the room in the painting, and outside  in the real landscape. Which is how we 
see the world: we see it as being outside  ourselves even though it is only a mental 
representation of it that we experience inside ourselves.' 
 
Understanding the Mind 
 First through the pregnant images of his pa inting, and then in direct words, Magritte 
expresses the link between the two questions  "How do symbols work?" and "How do our

expresses the link between the two questions  "How do symbols work?" and "How do our 
minds work?" And so he leads us back to the question posed earlier: "Can we ever hope 
to understand our minds! brains?" 
Or does some marvelous diabolical Gödelian proposition preclude our ever 
unraveling our minds? Provided you do not a dopt a totally unreasonable definition of 
"understanding", I see no Gödelian obstacle in the way of the eventual understanding of 
our minds. For instance, it seems to me qu ite reasonable to desire to understand the 
working principles of brains in general,  much the same way as we understand the 
working principles of car engines in genera l. It is quite different from trying to

working principles of car engines in genera l. It is quite different from trying to 
understand any single brain in every last deta il-let alone trying to do this for one's own 
brain! I don't see how Gödel’s Theorem, ev en if construed in the sloppiest way, has 
anything to say about the feasibility of th is prospect. I see no reason that Gödel’s 
Theorem imposes any limitations on our ability to formulate and verify the general mechanisms by which thought processes take place in the medium of nerve cells. I see no barrier imposed by Gödel’s Theorem to th e implementation on computers (or their

Theorem imposes any limitations on our ability to formulate and verify the general mechanisms by which thought processes take place in the medium of nerve cells. I see no barrier imposed by Gödel’s Theorem to th e implementation on computers (or their 
successors) of types of symbol manipulation that achieve roughly the same results as 
brains do. It is entirely another question to try and duplicate in a program some particular human's mind-but to produce an intelligent progr am at all is a more limited goal. Godel's 
Theorem doesn't ban our reproducing our ow n level of intelligence via programs any 
more than it bans our reproducing our own level of intelligence via transmission of 
hereditary information in

 702DNA , followed by education. Indeed, we have seen, in Chapter XVI, how a remarkable 
'Gödelian mechanism-the Strange Loop of proteins and DNA -is precisely what allows 
transmission of intelligence! 
Does Gödel’s Theorem, then, have abso lutely nothing to offer us in thinking 
about our own minds? I think it does, although not in the mystical and [imitative way 
which some people think it ought to. I think that the process of coming to understand 
Gödel’s proof, with its construction involvi ng arbitrary codes, complex isomorphisms, 
high and low levels of interpretation, and the capacity for self-mirroring, may inject some 
rich undercurrents and flavors into one's set of images about symbols and symbol processing, which may deepen one's intuiti on for the relationship. between mental 
structures on different levels. 
 
Accidental Inexplicability of Intelligence?

Accidental Inexplicability of Intelligence? 
 
Before suggesting a philosophically intriguing "application" of Godel's proof. I would like to bring up the idea of "accidental inexp licability" of intelligence. Here is what that 
involves. It could be that our brains, un like car engines, are stubborn and intractable 
systems which we cannot neatly decompose in any way. At present, we have no idea whether our brains will yield to repeated atte mpts to cleave them into clean layers, each 
of which can be explained in terms of lower layers-or whether our brains will foil all our 
attempts at decomposition.

attempts at decomposition. 
But even if we do fail to understand ourselves, there need not be any Godelian 
"twist" behind it; it could be simply an accident of fate that our brains are too weak to 
understand themselves. Think of the lowly gira ffe, for instance, whose brain is obviously 
far below the level required for self-understandi ng-yet it is remarkably similar to our own 
brain. In fact, the brains of giraffes, elephants, baboons-even the brains of tortoises or 
unknown beings who are far smarter than we are-probably all operate on basically the 
same set of principles. Giraffes may lie far below the threshold of intelligence necessary

same set of principles. Giraffes may lie far below the threshold of intelligence necessary 
to understand how those principles fit together to produce the qualities of mind; humans may lie closer to that threshold perhaps just  barely below it, perhaps even above it. The 
point is that there may be no fundamental (i.e., Gödelian) reason why those qualities are 
incomprehensible; they may be completely clear to more intelligent beings.  
Undecidability Is Inseparable from a High-Level Viewpoint 
 Barring this pessimistic notion of the accidental  inexplicability of the brain, what insights

 Barring this pessimistic notion of the accidental  inexplicability of the brain, what insights 
might Gödel’s proof offer us about explana tions of our minds/brains? Gödel’s proof 
offers the notion that a high-level view of a system may contain explanatory power which 
simply is absent on the lower levels. By this I mean the following. Suppose someone gave you G, Gödel’s undecida ble string, as a string of TNT . Also suppose you knew 
nothing of' Gödel-numbering. The question you are supposed to answer is: "Why isn't

 703this string a theorem of TNT ?" Now you are used to such questions; for instance, if you 
had been asked that question about SO=0,  you would have a ready explanation: "Its 
negation , ~S0=0 , is a theorem ." This, together with your knowledge that TNT  is 
consistent, provides an explanation of why th e given string is a nontheorem. This is what 
I call an explanation "on the TNT-level". Noti ce how different it is from the explanation 
of why MU is not a theorem of the MIU -system: the former comes from the M-mode, the 
latter only from the I-mode. 
Now what about G? The TNT -level explanation which worked for 50=0 does not 
work for G, because - G is not a theorem. The person who has no overview of TNT  will 
be baffled as to why he can't make G according to the rules, because as an arithmetical 
proposition, it apparently has nothing wrong with it. In fact, when G is turned into a

proposition, it apparently has nothing wrong with it. In fact, when G is turned into a 
universally quantified string, every instance gotten from G by substituting numerals for 
the variables can be derive d. The only way to explain G's nontheoremhood is to discover 
the notion of Gödel-numbering and view TNT  on an entirely different level. It is not that 
it is just difficult and complicated to write out the explanation on the TNT -level; it is 
impossible. Such an explanation simply does not exist. There is, on the high level, a kind of explanatory power which simply is lacking, in principle, on the TNT -level. G's

impossible. Such an explanation simply does not exist. There is, on the high level, a kind of explanatory power which simply is lacking, in principle, on the TNT -level. G's 
nontheoremhood is, so to speak, an intrinsically high-level fact . It is my suspicion that 
this is the case for all undecidable propositi ons; that is to say: every undecidable 
proposition is actually a Gödel sentence, asserting its own nontheoremhood in some 
system via some code.  
Consciousness as an Intrinsi cally High-Level Phenomenon 
 
Looked at this way, Gödel’s proof suggest s-though by no means does it prove!-that there 
could be some high-level way of viewing the mind/brain, involving concepts which do

could be some high-level way of viewing the mind/brain, involving concepts which do 
not appear on lower levels, and that this leve l might have explanatory power that does not 
exist-not even in principle-on lower levels. It would mean that some facts could be 
explained on the high level quite easily, but not  on lower levels at all. No matter how 
long and cumbersome a low-level statemen t were made, it would not explain the 
phenomena in question. It is the analogue to the fact that, if you make derivation after 
derivation in TNT , no matter how long and cumbersome you make them, you will never

derivation in TNT , no matter how long and cumbersome you make them, you will never 
come up with one for G-despite the fact that on a higher level, you can see that G is true. 
What might such high-level concepts be? It has been proposed for eons, by 
various holistically or "soulistically" inclined scientists and humanists, that consciousness  
is a phenomenon that escapes explanation in terms of brain-components; so here is a 
candidate, at least. There is also the ever-puzzling notion of free will . So perhaps these 
qualities could be "emergent" in the sense of requiring explanations which cannot be

qualities could be "emergent" in the sense of requiring explanations which cannot be 
furnished by the physiology alone. But it is important to realize that if we are being guided by Gödel’s proof in making such bold hypotheses, we must carry the

 704 
analogy through thoroughly. In particular, it is  vital to recall tnat is s nontheoremhood 
does have an explanation-it is not a to tal mystery! The explanation- hinges on 
understanding not just one level at a time, but the way in which one level mirrors its metalevel, and the consequences of this mirroring. If our analogy is to hold, then, "emergent" phenomena would become explicab le in terms of a relationship between. 
different levels in mental systems.,  
Strange Loops as the Crux of Consciousness 
 My belief is that the explanations of "eme rgent" phenomena in our brains-for instance, 
ideas, hopes, images, analogies, and finally  consciousness and free will-are based on a 
kind of Strange Loop, an interaction between levels in which the top level reaches back down towards the bottom level and influences  it, while at the same time being itself 
determined by the bottom level. In other wo rds, a self-reinforcing "resonance" between

determined by the bottom level. In other wo rds, a self-reinforcing "resonance" between 
different levels-quite like the Henkin sentence which, by merely asserting its own provability, actually becomes provable. The self comes into being at the moment it has the power to reflect itself. 
This should not be taken as an antireduc tionist position. It just implies that a 
reductionistic explanation of a mind, in order to be comprehensible , must bring in "soft" 
concepts such as levels, mappings, and mean ings. In principle, I have no doubt that a

concepts such as levels, mappings, and mean ings. In principle, I have no doubt that a 
totally reductionistic but incomprehensible expl anation of the brain exists; the problem is 
how to translate it into a language we ourselves can fathom. Surely we don't want a description in terms of positions and momenta of particles; we want a description which 
relates neural activity to "signals" (intermediate-level phenomena)-and which relates signals, in turn, to "symbols" and "subsyste ms", including the presumed-to-exist "self-
symbol". This act of translation from low-level physical hardware to high-level

symbol". This act of translation from low-level physical hardware to high-level 
psychological software is analogous to the tr anslation of number-theoretical statements 
into metamathematical statements. Recall that  the level-crossing which takes place at this 
exact translation point is what creates Godel's incompleteness and the self-proving 
character of Henkin's sentence. I postulate that  a similar level-crossing is what creates our 
nearly unanalyzable feelings of self. 
In order to deal with the full richness of the brain/mind system, we will have to be 
able to slip between levels comfortably. More over, we will have to admit various types of

able to slip between levels comfortably. More over, we will have to admit various types of 
"causality": ways in which an event at one level of description can "cause" events at other levels to happen. Sometimes event A will be said to "cause" event B simply for the reason that the one is a translation, on anot her level of description, of the other. 
Sometimes "cause" will have its usual m eaning: physical causality. Both types of 
causality-and perhaps some more-will have to be admitted in any explanation of mind, 
for we will have to admit causes that propagate both upwards and downwards in the

for we will have to admit causes that propagate both upwards and downwards in the 
Tangled Hierarchy of mentality, just as in the Central Dogmap.

 705At the crux, then, of our understanding our selves will come an understanding of 
the Tangled Hierarchy of levels inside our minds. My position is rather similar to the 
viewpoint put forth by the neuroscientist Roge r Sperry in his excellent article "Mind, 
Brain, and Humanist Values", from which I quote a little here: 
 
In my own hypothetical brain model, cons cious awareness does get representation 
as a very real causal agent and rates an  important place in the causal sequence and 
chain of control in brain events, in wh ich it appears as an active, operational 
force.... To put it very simply, it comes down to the issue of who pushes whom around in the population of causal forces that occupy the cranium. It is a matter, in 
other words, of straightening out the peck-order hierarchy among intracranial 
control agents. There exists within the cranium a whole world of diverse causal

control agents. There exists within the cranium a whole world of diverse causal 
forces; what is more, there are forces with in forces within forces, as in no other 
cubic half-foot of universe that we know. ... To make a long story short, if one 
keeps climbing upward in the chain of co mmand within the brain, one finds at the 
very top those over-all organizational for ces and dynamic properties of the large 
patterns of cerebral excitation that are co rrelated with mental states or psychic 
activity.... Near the apex of this command system in the brain ... we find ideas. Man

activity.... Near the apex of this command system in the brain ... we find ideas. Man 
over the chimpanzee has ideas and ideals. In the brain model proposed here, the causal potency of an idea, or an ideal, become s just as real as that of a molecule, a 
cell, or a nerve impulse. Ideas cause ideas and help evolve new ideas. They interact 
with each other and with other mental fo rces in the same brain, in neighboring 
brains, and, thanks to global communication, in  far distant, foreign brains. And they 
also interact with the external surroundings  to produce in toto a burstwise advance

also interact with the external surroundings  to produce in toto a burstwise advance 
in evolution that is far beyond anything to hit the evolutionary scene yet, including 
the emergence of the living cell.' 
 
There is a famous breach between two languages of discourse: the subjective 
language and the objective language. For instan ce, the "subjective" sensation of redness, 
and the "objective" wavelength of red light. To many people, these seem to be forever 
irreconcilable. I don't think so. No more than the two views of Escher's Drawing Hands

irreconcilable. I don't think so. No more than the two views of Escher's Drawing Hands  
are irreconcilable from "in the system", wh ere the hands draw each other, and from 
outside, where Escher draws it all. The s ubjective feeling of redness comes from the 
vortex of self-perception in the brain; th e objective wavelength is how you see things 
when you step back, outside of the system. T hough no one of us will ever be able to step 
back far enough to see the "big picture", we  shouldn't forget that it exists. We should 
remember that physical law is what make s it all happen-way, way down in neural nooks

remember that physical law is what make s it all happen-way, way down in neural nooks 
and crannies which are too remote for us to reach with our high-level introspective 
probes. 
 
The Self-Symbol and Free Will 
 
In Chapter XI I, it was suggested that what we call free will is a result of the interaction 
between the self-symbol (or subsystem), and th e other symbols in the brain. If we take 
the idea that symbols are the high-level entities to

 706which meanings should be attached, then we can' make a stab at explaining the 
relationship between symbols, the self-symbol, and free will. 
One way to gain some perspective on the free-will question is to replace it by 
what I believe is an equivalent question, but one which involves less loaded terms. Instead of asking, "Does system X have free will?" we ask, "Does system X make choices?" By carefully groping for what we really mean when we choose to describe a system-mechanical or biological-as being cap able of making "choices", I think we can 
shed much light on free will it will be helpful to go over a few different systems which, 
under various circumstances, we might feel te mpted to describe as "making choices". 
From these examples we can gain some pe rspective on what we really mean by the 
phrase. 
Let us take the following systems as paradigms: a marble rolling down a bumpy

phrase. 
Let us take the following systems as paradigms: a marble rolling down a bumpy 
hill; a pocket calculator finding successive digits  in the decimal expansion of the square 
root of 2; a sophisticated program which plays a mean game of chess; a robot in a T-maze 
(a maze with but a single fork, on one side of which there is a reward); and a human being confronting a complex dilemma. 
First, what about that marble rolling dow n a hill? Does it make choices? I think 
we would unanimously say that it doesn't, ev en though none of us could predict its path 
for even a very short distance. We feel th at it couldn't have gone any other way than it 
did, and that it was just being shoved along by the relentless laws of nature. In our

did, and that it was just being shoved along by the relentless laws of nature. In our 
chunked mental physics, of course, we can vi sualize many different "possible" pathways 
for the marble, and we see it following only one of them in the real world. On some level 
of our minds, therefore, we can't help fee ling the marble has "chosen" a single pathway 
out of those myriad mental ones; but on so me other level of our minds, we have an 
instinctive understanding that the mental physic s is only an aid in our internal modeling 
of the world, and that the mechanisms which make the real physical sequences of events

of the world, and that the mechanisms which make the real physical sequences of events 
happen do not require nature to go through an  analogous process of first manufacturing 
variants in some hypothetical universe (the "brain of God") and then choosing between 
them. So we shall not bestow the desi gnation "choice" upon this process-although we 
recognize that it is often pragmatically useful to use the word in cases like this, because 
of its evocative power. 
Now what about the calculator programmed to find the digits of the square root of 
2? What about the chess program? Here, we might say that we are just dealing with

2? What about the chess program? Here, we might say that we are just dealing with 
"fancy marbles", rolling down "fancy hills". In fact, the arguments for no choice-making here are, if anything, stronger than in the cas e of a marble. For if you attempt to repeat 
the marble experiment, you will undoubtedly w itness a totally different pathway being 
traced down the hill, whereas if you rerun th e square-root-of-2 program, you will get the 
same results time after time. The marble seem s to "choose" a different path each time, no 
matter how accurately you try to reproduce the c onditions of its original descent, whereas

matter how accurately you try to reproduce the c onditions of its original descent, whereas 
the program runs down precisely the same channels each time. 
Now in the case of fancy chess programs, there are various possibilities.

 707If you play a game against certain programs, and then start a second game with the same 
moves as you made the first time, these progr ams will just move exactly as they did 
before, without any appearance of having l earned anything or having any desire for 
variety. There are other programs which have  randomizing devices that will give some 
variety but not out of any deep desire. Such  programs could be reset with the internal 
random number generator as it was the first time, and once again, the same game would 
ensue. Then there are other programs which do learn from their mistakes, and change 
their strategy depending on the outcome of a game. Such programs would not play the 
same game twice in a row. Of course, you c ould also turn the clock back by wiping out

same game twice in a row. Of course, you c ould also turn the clock back by wiping out 
all the changes in the memory which represent learning, just as you could reset the random number generator, but that hardly seems like a friendly thing to do. Besides, is there any reason to suspect that you would be able to change any of your own past 
decisions if every last detail-and that include s your brain, of course-were reset to the way 
it was the first time around? 
But let us return to the question of whethe r "choice" is an applicable term here. If 
programs are just "fancy marbles rolling down fancy hills", do they make choices, or not?

programs are just "fancy marbles rolling down fancy hills", do they make choices, or not? 
Of course the answer must be a subjective one , but I would say that pretty much the same 
considerations apply here as to the marble. Ho wever, I would have to add that the appeal 
of using the word "choice", even if it is  only a convenient and evocative shorthand, 
becomes quite strong. The fact that a ch ess program looks ahead down the various 
possible bifurcating paths, quite unlike a ro lling marble, makes it seem much more like 
an animate being than a square-root-of-2 pr ogram. However, there is still no deep self-
awareness here-and no sense of free will. 
Now let us go on to imagine a robot which ha s a repertoire of symbols. This robot

Now let us go on to imagine a robot which ha s a repertoire of symbols. This robot 
is placed in a T-maze. However, instead of going for the reward, it is preprogrammed to 
go left whenever the next digit of the square root: of 2 is even, and to go right whenever it 
is odd. Now this robot is capable of modeling th e situation in its symbols, so it can watch 
itself making choices. Each time the T is approached, if you were to address to the robot the question, "Do you know which way you're going to turn this time?" it would have to answer, "No". Then in order to progress, it would activate its "decider" subroutine, which

itself making choices. Each time the T is approached, if you were to address to the robot the question, "Do you know which way you're going to turn this time?" it would have to answer, "No". Then in order to progress, it would activate its "decider" subroutine, which 
calculates the next digit of the square root of 2, and the decision is taken. However, the 
internal mechanism of the decider is unknown to  the robot-it is represented in the robot's 
symbols merely as a black box which puts out "left"'s and "right"'s by some mysterious 
and seemingly random rule. Unless the robot's  symbols are capable of picking up the 
hidden heartbeat of the square root of 2, bea ting in the L's and R's, it will stay baffled by 
the "choices" which it is making. Now does this robot make choices? Put yourself in that

the "choices" which it is making. Now does this robot make choices? Put yourself in that 
position. If you were trapped inside a marble  rolling down a hill and were powerless to 
affect its path, yet could observe it with a ll your human intellect, would you feel that the 
marble's path involved choices? Of course not. Unless your mind is affecting  the 
outcome, it makes no difference that the symbols are present.

 708So now we make a modification in our robo t: we allow its symbols--including its self-
symbol-to affect the decision that is taken. Now here is an example of a program running 
fully under physical law, which seems to get mu ch more deeply at the essence of choice 
than the previous examples did. When the robot's own chunked concept of itself enters the scene, we begin to identify with the robot , for it sounds like the kind of thing we do. It 
is no longer like the calculation of the square  root of 2, where no symbols seem to be 
monitoring the decisions taken. To be sure, if we were to look at the robot's program on a 
very local level, it would look quite like th e square-root program. Step after step is 
executed, and in the end "left" or "right" is  the output. But on a high level we can see the 
fact that symbols are being used to model th e situation and to affect the decision. That

fact that symbols are being used to model th e situation and to affect the decision. That 
radically affects our way of thinking about the program. At this stage, meaning  has 
entered this picture-the same kind of meaning as we manipulate with our own minds. 
 
A Gödel Vortex Where All Levels Cross 
 Now if some outside agent suggests `L' as the next choice to the robot, the suggestion 
will be picked up and channeled into the swirling mass of interacting symbols. There, it will be sucked inexorably into interacti on with the self-symbol, like a rowboat being

will be picked up and channeled into the swirling mass of interacting symbols. There, it will be sucked inexorably into interacti on with the self-symbol, like a rowboat being 
pulled into a whirlpool. That is the vortex of the system, where all levels cross. Here, the 
`L' encounters a Tangled Hierarchy of symbols and is passed up and down the levels. The self-symbol is incapable of monitoring all its internal processes, and so when the actual 
decision emerges-'L' or 'R' or something outside  the system-the system will not be able to 
say where it came from. Unlike a standard chess program, which does not monitor itself and consequently has no ideas about where its moves come from, this program does

say where it came from. Unlike a standard chess program, which does not monitor itself and consequently has no ideas about where its moves come from, this program does 
monitor itself and does have ideas about its ideas-but it cannot monitor its own processes in complete detail, and therefore has a sort of intuitive sense of its  workings, without full 
understanding. From this balance between se lf-knowledge and self-ignorance comes the 
feeling of free will. 
Think, for instance, of a writer who is trying to convey certain ideas which to him 
are contained in mental images. He isn't quite  sure how those images fit together in his 
mind, and he experiments around, expressing thi ngs first one way and then another, and

mind, and he experiments around, expressing thi ngs first one way and then another, and 
finally settles on some version. But does he know where it all came from? Only in a vague sense. Much of the source, like an iceberg, is deep underwater, unseen-and he knows that. Or think of a music composition program, something we discussed earlier, 
asking when we would feel comfortable in calli ng it the composer rather than the tool of 
a human composer. Probably we would feel comfortable when self-knowledge in terms 
of symbols exists inside the program, and when the program has this delicate balance 
between self-knowledge and self -ignorance. It is irrelevant whether the system is running

between self-knowledge and self -ignorance. It is irrelevant whether the system is running 
deterministically; what makes us call it a "choice maker" is whether we can identify with 
a high-level description of the process which takes place when the

 709
 
FIGURE 142. Print Gallery, by M. C. Escher (lithograph, 1956) . 
 
program runs . On a low (machine language) level, the program looks like any other 
program; on a high (chunked) level, qualities such  as "will", "intuition", "creativity", and 
"consciousness" can emerge. 
The important idea is that this "vortex" of self is responsible for the tangledness, 
for the Gödelian-ness, of the mental processes. People have said to me on occasion, "This 
stuff with self-reference and so on is very amusing and enjoyable, but do you really think 
there is anything serious to it?" I certainly do. I think it will eventually turn out to be at 
the core of AI, and the focus of all attempts to understand how human minds work. And 
that is why Godel is so deeply woven into the fabric of my book.

 710An Escher Vortex Where All Levels Cross 
 
A strikingly beautiful, and yet at the same time disturbingly grotesque, illustration of the 
cyclonic "eye" of a Tangled Hierarchy is given to us by Escher in his Print Gallery  (Fig. 
142). What we see is a picture gallery where a young man is standing, looking at a 
picture of a ship in the harbor of a small town, perhaps a Maltese town, to guess from the 
architecture, with its little turrets, occasiona l cupolas, and flat stone roofs, upon one of 
which sits a boy, relaxing in the heat, while  two floors below him a woman-perhaps his 
mother-gazes out of the window from her apar tment which sits directly above a picture 
gallery where a young man is standing, looking at a picture of a ship in the harbor of a small town, perhaps a Maltese town--What!? We  are back on the same level as we began, 
though all logic dictates that we cannot be. Let us draw a diagram of what we see (Fig. 143).

though all logic dictates that we cannot be. Let us draw a diagram of what we see (Fig. 143). 
 
FIGURE 143. Abstract diagram of M. C. Escher's Print Gallery . 
 What this diagram shows is three kinds of "in-ness". The gallery is physically in  the town 
("inclusion"); the town is artistically  in the picture ("depiction"); the picture is mentally  
in the person ("representation"). Now while this  diagram may seem satisfying, in fact it is 
arbitrary, for the number of levels shown is quite arbitrary. Look below at another way of 
representing the top half alone (Fig. 144).  
 
FIGURE 144. A collapsed version of the previous figure.

 711We have eliminated the "town" level; conceptu ally it was useful, but can just as well be 
done without. Figure 144 looks just like the diagram for Drawing Hands: a Strange Loop 
of two steps. The division markers are arbitrar y, even if they seem natural to our minds. 
This can be further accentuated by showing even more "collapsed" schematic diagrams of Print Gallery , such as that in Figure 145. 
 
FIGURE 145. Further collapse of Figure 143 . 
 This exhibits the paradox of the picture in the starkest terms. Now-if the picture is "inside 
itself", then is the young man also inside hims elf-, This question is answered in Figure 
146.  
 
FIGURE 146. Another way of collapsing Figure 143. 
 Thus, we see the young man "inside himself" , in a funny sense which is made up of 
compounding three distinct senses of "in". 
This diagram reminds us of the Epim enides paradox with its one-step self-

146.  
 
reference, while the two-step diagram resembles the sentence pair each of which refers to the other. We cannot make the loop any tight er, but we can open it wider, by choosing to 
insert any number of intermediate levels, such as "picture frame", "arcade", and "building". If we do so, we will have ma ny-step Strange Loops, whose diagrams are 
isomorphic to those of Waterfall  (Fig. 5) or Ascending and Descending  (Fig. 6). The

isomorphic to those of Waterfall  (Fig. 5) or Ascending and Descending  (Fig. 6). The 
number of levels is determined by what we f eel is "natural", which may vary according to 
context, purpose, or frame of mind. The Ce ntral Xmaps-Dog, Crab, Sloth, and Pipe-can 
all be seen as involving three-step Strange L oops; alternatively, they can all be collapsed 
into two- or one-step loops;. then again, they can be expanded out into multistage loops. 
Where one perceives the levels is a matter of intuition and esthetic preference.

Where one perceives the levels is a matter of intuition and esthetic preference. 
Now are we, the observers of Print Galler y, also sucked into ourselves by virtue 
of looking at it? Not really. We manage to escape that particular vortex by being outside 
of the system. And when we look at the picture, we see things which the young man can certainly not see, such as Escher's

 712Signature, "MCE", in the central "blemish ". Though the blemish seems like a defect, 
perhaps the defect lies in our expectations, fo r in fact Escher could not have completed 
that portion of the picture without being in consistent with the rules by which he was 
drawing the picture. That center of the whor l is-and must be-incomplete. Escher could 
have made it arbitrarily small, but he coul d not have gotten rid of it. Thus we, on the 
outside, can know that Print Gallery is essentially incomplete-a fact which the young 
man, on the inside, can never know. Escher has thus given a pictorial parable for Gödel’s Incompleteness Theorem. And that is why the strands of Gödel and Escher are so deeply 
interwoven in my book.  
A Bach Vortex Where All Levels Cross 
 One cannot help being reminded, when one looks  at the diagrams of Strange Loops, of 
the Endlessly Rising Canon from the Musical Offering. A diagram of it would consist of

the Endlessly Rising Canon from the Musical Offering. A diagram of it would consist of 
six steps, as is shown in Figure 147. It is too .  
 
FIGURE 147. The hexagonal modulation scheme of Bach's Endlessly Rising Canon 
forms a true closed loop when Shepard tones are used . 
 bad that when it returns to C, it is an octave higher rather than at the exact original pitch. 
Astonishingly enough, it is possible to arrange fo r it to return exactly to the starting pitch, 
by using what are called Shepard tones, after the psychologist Roger Shepard, who

by using what are called Shepard tones, after the psychologist Roger Shepard, who 
discovered the idea. The principle of a Shep ard-tone scale is shown in Figure 148. In 
words, it is this: you play parallel scales in several different octave ranges. Each note is 
weighted independently, and as the notes rise, the weights shift. You make the top

 713
 
FIGURE 148: Two complete cycles of a Shephard tone scale, notated for piano. The 
loudness of each note is proportional to its area, just as the top voice fades out, a new 
bottom voice feebly enters. (Printed by Donald Boyd´s program “SMUT”.)

 714octave gradually fade out, while at the same time you are gradually bringing in the 
bottom octave. Just at the moment you would ordinarily be one octave higher, the weights have shifted precisely so as to reproduce the starting pitch ... Thus you can go "up and up forever", never getting any higher! You can try it at your piano. It works even better if the pitches can be synthesized accurately under computer control. Then the illusion is bewilderingly strong. 
This wonderful musical discovery allows the Endlessly Rising Canon to be played 
in such a way that it joins back onto itself after going "up" an octave. This idea, which Scott Kim and I conceived jointly, has been realized on tape, using a computer music 
system. The effect is very subtle-but very r eal. It is quite interesting that Bach himself 
was apparently aware, in some sense, of  such scales, for in his music one can

was apparently aware, in some sense, of  such scales, for in his music one can 
occasionally find passages which roughly explo it the general principle of Shepard tones-
for instance, about halfway through the Fantasia  from the Fantasia and Fugue in G Minor, 
for organ. 
In his book J. S. Bach's Musical Offering , Hans Theodore David writes: 
 
Throughout the Musical Offering , the reader, performer, or listener is to search for 
the Royal theme in all its forms. The entir e work, therefore, is a ricercar in the 
original, literal sense of the word.' 
 I think this is true; one cannot look deeply enough into the Musical Offering . There is

for organ. 
 
 I think this is true; one cannot look deeply enough into the Musical Offering . There is 
always more after one thinks one knows everyt hing. For instance. towards the very end of 
the Six-Part Ricercar , the one he declined to improvise, Bach slyly hid his own name, 
split between two of the upper voices. Thi ngs are going on on many levels in the Musical 
Offering.  There are tricks with notes and letters ; there are ingenious variations on the 
King's Theme; there are original kinds of  canons; there are extraordinarily complex 
fugues; there is beauty and extreme depth of  emotion; even an exultation in the many-

leveledness of the work comes through. The Musical Offering  is a fugue of fugues, a 
Tangled Hierarchy like those of Escher and Gödel, an intellectual construction which 
reminds me, in ways I cannot express, of the beautiful many-voiced fugue of the human mind. And that is why in my book the three strands of Gödel, Escher, and Bach are 
woven into an Eternal Golden Braid.

Six-Part Ricercar 
 715Six-Part Ricercar 
 
Achilles has brought his cello to the Crab's residence, to engage in an evening of 
chamber music with the Crab and Tortoise . He has been shown into the music 
room by his host the Crab, who is moment arily absent, having gone to meet their 
mutual friend the Tortoise at the door. The room is filled with all sorts of electronic 
equipment-phonographs in various states of  array and disarray, television screens 
attached to typewriters, and other quite improbable-looking pieces of apparatus. Nestled amongst all this high-powered gadgetry sits a humble radio. Since the radio is the only thing in the room which Achilles knows how to use, he walks over 
to it, and, a little furtively, flicks the di al and f nds he has tuned into a panel 
discussion by six learned scholars on free will and determinism. He listens briefly 
and then, a little scornfully, flicks it off. 
 
Achilles: I can get along very well without such  a program. After all, it's clear to anyone

Achilles: I can get along very well without such  a program. After all, it's clear to anyone 
who's ever thought about it that-I mean, it's not  a very difficult matter to resolve, once 
you understand how-or rather, conceptuall y, one can clear up the whole thing by 
thinking of, or at least imagining a situ ation where ... Hmmm ... I t hought it was quite 
clear in my mind. Maybe I could benefit from listening to that show, after all ... 
 
(Enter the Tortoise, carrying his violin.) 
 
Well, well, if it isn't our fiddler. Have you b een practicing faithfully this week, Mr. T? 
I myself have been playing the cello part in the Trio Sonata from the Musical Offering  
for at least two hours a day. It's a strict regimen, but it pays off.

for at least two hours a day. It's a strict regimen, but it pays off. 
Tortoise: I can get along very well without su ch a program. I find that a moment here, a 
moment there keeps me fit for fiddling. 
Achilles: Oh, lucky you. I wish it came so easily to me. Well, where is our host? Tortoise: I think he's just gone to fetch his flute. Here he comes.  
(Enter the Crab, carrying his flute.) 
 
Achilles: Oh, Mr. Crab, in my ardent practici ng of the Trio Sonata this past week, all

Achilles: Oh, Mr. Crab, in my ardent practici ng of the Trio Sonata this past week, all 
sorts of images bubbled into my mind: jolly gobbling bumblebees, melancholy buzzing turkeys, and a raft of others. Isn't it wonderful, what power music has? 
Crab: I can get along very well without such a program. To my mind. Achilles, there is no music purer than the Musical Offering. Tortoise: You can't be serious, Achilles. The Musical Offering isn't programmatic music!

 716Achilles: Well, I like animals, even if you two stuffy ones disapprove. 
Crab: I don't think we are so stuffy, Achilles. Let's just say that you hear music in 'your 
own special way. 
Tortoise: Shall we sit down and play? Crab: I was hoping that a pianist friend of mi ne would turn up and play continuo. I've 
been wanting you to meet him, Achilles, for a long time. Unfortunately, it appears 
that he may not make it. So let's just go ahead  with the three of us. That's plenty for a 
trio sonata. 
Achilles: Before we start, I just was wondering, Mr. Crab-what are all these pieces of 
equipment, which you have in here? 
Crab: Well, mostly they are just odds and ends-bits and pieces of old broken 
phonographs. Only a few souvenirs (nervously  tapping the buttons), a few souvenirs 
of-of the TC-battles in which I have dis tinguished myself. Those keyboards attached 
to television screens, however, are my new toys. I have fifteen of them around here.

to television screens, however, are my new toys. I have fifteen of them around here. 
They are a new kind of computer, a very sma ll, very flexible type of computer quite 
an advance over the previous types availa ble. Few others seem to be quite as 
enthusiastic about them as I am, but I have faith that they will catch on in time. 
Achilles: Do they have a special name? Crab: Yes; they are called "smart-stupids", since they are so flexible, and have the 
potential to be either smart or stupid, depending on how skillfully they are instructed. 
Achilles: Do you mean you think they could actually become smart like, say, a human

Achilles: Do you mean you think they could actually become smart like, say, a human 
being? 
Crab: I would not balk at saying so-provided, of course, that someone sufficiently versed 
in the art of instructing smart-stupids would make the effort. Sadly, I am not 
personally acquainted with anyone who is a true virtuoso. To be sure, there is one 
expert abroad in the land, an individual of great renown-and nothing would please me 
more than a visit by him, so that I could appreciate what true skill on the smart-stupid 
is; but he has never come, and I wonder if I shall ever have that pleasure.

is; but he has never come, and I wonder if I shall ever have that pleasure. 
Tortoise: It would be very interesting to play chess against a well-instructed smart-stupid. Crab: An extremely intriguing idea. That woul d be a wonderful mark of skill, to program 
a smart-stupid to play a good game of chess.  Even more interesting-but incredibly 
complicated-would be to instruct a smart-st upid sufficiently that it could hold its own 
in a conversation. It might give the impression that it was just another person!

in a conversation. It might give the impression that it was just another person! 
Achilles: Curious that this should come up, fo r I just heard a snatch of a discussion on 
free will and determinism, and it set me to thinking about such questions once more. I 
don't mind admitting that, as I pondered the idea, my thoughts got more and more 
tangled, and in the end I really didn't know  what I thought. But this idea of a smart-
stupid that could converse with you ... it boggles the mind. I mean,

 717what would the smart-stupid itself say, if  you asked it for its opinion on the free-will 
question? I was just wondering if the tw o of you, who know so much about these 
things, wouldn't indulge me by explaining the issue, as you see it, to me. 
Crab: Achilles, you can't imagine how appropriate your question is. I only wish my 
pianist friend were here, because I know you'd be intrigued to hear what he could tell 
you on the subject. In his absence, I'd like to tell you a statement in a Dialogue at the end of a book I came across recently. 
Achilles: Not Copper, Silver, Gold: an Indestructible Metallic Alloy ? 
Crab: No, as I recall, it was entitled Gi raffes, Elephants, Baboons: an Equatorial 
Grasslands Bestiary -or something like that. In any case, towards the end of the 
aforementioned Dialogue, a certain exceedingly droll character quotes Marvin Minsky on the question of free will. Shortly thereafter, while interacting with two

aforementioned Dialogue, a certain exceedingly droll character quotes Marvin Minsky on the question of free will. Shortly thereafter, while interacting with two 
other personages, this droll character quot es Minsky further on musical improvisation, 
the computer language LISP, and Godel's Theorem-and get this-all without giving one whit of credit to Minsky! 
Achilles: Oh, for shame! Crab: I must admit that earlier in the Di alogue, he hints that he WILL quote Minsky 
towards the end; so perhaps it's forgivable. 
Achilles: It sounds that way to me. Anyw ay, I'm anxious to hear the Minskian 
pronouncement on the free will question. 
Crab: Ah, yes... Marvin Minsky said, "When intelligent machines are constructed, we

Crab: Ah, yes... Marvin Minsky said, "When intelligent machines are constructed, we 
should not be surprised to find them as confused and as stubborn as men in their convictions about mind-matter, consciousness, free will, and the like." 
Achilles: I like that! Quite a funny thought. An automaton thinking it had free will! That's 
almost as silly as me thinking I didn't have free will! Tortoise: I suppose it never 
occurred to you, Achilles, that the three of us-you, myself, and Mr. Crab-might all be 
characters in a Dialogue, perhaps even one  similar to the one Mr. Crab just

characters in a Dialogue, perhaps even one  similar to the one Mr. Crab just 
mentioned. Achilles: Oh, it's occurred to me , of course. I suppose such fancies occur 
to every normal person at one time or another. 
Tortoise: And the Anteater, the Sloth, Zeno, even GOD-we might all be characters in a 
series of Dialogues in a book. 
Achilles: Sure, we might. And the Author might just come in and play the piano, too. Crab: That's just what I had hoped. But he's always late. Achilles: Whose leg do you think you're pulling?  I know I'm not being controlled in any

Achilles: Sure, we might. And the Author might just come in and play the piano, too. Crab: That's just what I had hoped. But he's always late. Achilles: Whose leg do you think you're pulling?  I know I'm not being controlled in any 
way by another mentality! I've got my own thoughts, I express myself as I wish-you 
can't deny that! 
Tortoise: Nobody denied any of that, Achilles. But all of what you say is perfectly 
consistent with your being a character in a Dialogue. 
Crab: The--- Achilles: But-but-no! Perhaps Mr. C's article and my rebuttal have both

 718been mechanically determined, but this I refuse to believe. I can accept physical 
determinism, but I cannot accept the idea that  I am but a figment inside of someone 
else's mentality! 
Tortoise: It doesn't really matter whether you have a hardware brain, Achilles. Your will 
can be equally free, if your brain is just a piece of software inside someone else's 
hardware brain. And their brain, too, may be software in a yet higher brain .. .  
Achilles: What an absurd idea! And yet, I must admit, I do enjoy trying to find the 
cleverly concealed holes in your sophist ry, so go ahead. Try to convince me. I'm 
game. 
Tortoise: Did it ever strike you, Achilles, that you keep somewhat unusual company? Achilles: Of course. You are very eccentric  (I know you won't mind my saying so), and 
even Mr. Crab here is a weensy bit eccentric. (Pardon me, Mr. Crab.) 
Crab: Oh, don't worry about offending me. Tortoise: But Achilles, you've overlooked one of the most salient features of your

game. 
Crab: Oh, don't worry about offending me. Tortoise: But Achilles, you've overlooked one of the most salient features of your 
acquaintances. 
Achilles: Which is.... ? Tortoise: That we're animals! Achilles: Well, well-true enough. You have such a keen mind. I would never have 
thought of formulating the facts so concisely. 
Tortoise: Isn't that evidence enough? Ho w many people do you know who spend their 
time with talking Tortoises, and talking Crab s? Achilles: I must admit, a talking Crab 
is 
Crab: -an anomaly, of course. Achilles: Exactly; it is a bit of an anomal y-but it has precedents. It has occurred in 
literature.

is 
literature. 
Tortoise: Precisely-in literature. But where in real life? 
Achilles: Now that you mention it, I can't quite  say. I'll have to give it some thought. But 
that's not enough to convince me that I'm a character in a 
Dialogue. Do you have any other arguments? Tortoise: Do you remember one day when you and I met in the park, seemingly at 
random? 
Achilles: The day we discussed crab canons by Escher and Bach? Tortoise: The very one!

random? 
Achilles: The day we discussed crab canons by Escher and Bach? Tortoise: The very one! 
Achilles: And Mr. Crab, as I recall, turn ed up somewhere towards the middle of our 
conversation and babbled something funny and then left. 
Crab: Not just "somewhere towards the middle", Achilles. EXACTLY in the middle. Achilles: Oh, all right, then. Tortoise: Do you realize that your lines were the same as my lines in that conversation-
except in reverse order? A few words were changed here and there, but in essence there was a time symmetry to our encounter.

 719Achilles: Big Deal! It was just some sort of trickery. Probably all done with mirrors. 
Tortoise: No trickery. Achilles, and no mirrors: just the work of an assiduous Author. Achilles: Oh, well, it's all the same to me. Tortoise: Fiddle' It makes a big difference, you know. Achilles: Say, something about this conversation strikes me as familiar. Haven't I heard 
some of those lines somewhere before= Tortoise: You said it, Achilles. 
Crab: Perhaps those lines occurred at random in the park one day, Achilles. Do you recall 
how your conversation with Mr. T ran that day? 
Achilles: Vaguely. He said "Good day, Mr. A"  at the beginning, and at the end, I said, 
"Good day, Mr. T". Is that right 
Crab: I just happen to have a transcript right here ...  
(He fishes around in his music case, whips out a sheet, and hands it to Achilles. As 
Achilles reads it, he begins to squirm and fidget noticeable.)

Achilles reads it, he begins to squirm and fidget noticeable.) 
 
Achilles: This is very strange. Very, very st range ... All of a Sudden, I feel sort of-weird. 
It's as if somebody had actually planned out th at whole set of statements in advance, 
worked them out on paper or something . As if some Author had had a whole agenda and worked from it in detail in planning all those statements I made that day.  
 
(At that moment, the door bursts open. Enter the Author, carrying a giant 
manuscript.) 
 
Author: I can get along very well without su ch a program. You see, once my characters

Author: I can get along very well without su ch a program. You see, once my characters 
are formed, they seem to have lives of thei r own, and I need to exert very little effort 
in planning their lines. 
Crab: Oh, here you are!' I thought you'd never arrive! 
Author: Sorry to be so late. I followed th e wrong road and wound up very far away. But 
somehow I made it back. Good to see you again, Mr. T and Mr. C. And Achilles, I'm especially glad to see you. 
Achilles: Who are you? I've never seen you before. Author: I am Douglas Hofstadter-please ca ll me Doug-and I'm presently finishing up a

Achilles: Who are you? I've never seen you before. Author: I am Douglas Hofstadter-please ca ll me Doug-and I'm presently finishing up a 
book called Gödel, Escher, Bach: an Eternal Golden Braid . It is the book in which 
the three of you are characters. 
Achilles: Pleased to meet you. My name is Achilles, and Author: No need to introduce yourself, Achilles, since I already know you quite well. Achilles: Weird, weird. Crab: He's the one I was saying might drop in and play continuo with us. Author: I've been playing the Musical Offering  a little bit on my piano at  home, and I can

Achilles: Pleased to meet you. My name is Achilles, and Author: No need to introduce yourself, Achilles, since I already know you quite well. Achilles: Weird, weird. Crab: He's the one I was saying might drop in and play continuo with us. Author: I've been playing the Musical Offering  a little bit on my piano at  home, and I can 
try to blunder my way through the Trio  Sonata providing you'll overlook my many 
wrong notes. 
Tortoise: Oh, we're very tolerant around here, being only amateurs our selves.

 720Author: I hope you don't mind, Achilles, but I'm to blame for the tact that you and Mr. 
Tortoise said the same things, but in reverse order, that day in the park. 
Crab: Don't forget me' I was there, too right in the middle, putting in my two bits' worth! Author: Of course! You were the Crab in the Crab Canon . 
Achilles: So you are saying you control my utte rances;, That my brain is a software 
subsystem of yours? 
Author: You can put it that way if you want, Achilles. Achilles: Suppose I were to write dialogues. Who would the author of them beg You, or 
me? 
Author: You, of course. At least in the fic titious world which you inhabit, you'd get credit 
for them. 
Achilles: Fictitious? I don't see anything fictitious about it! Author: Whereas in the world I inhabit, perhaps the credit would be given to me, 
although I am not sure if it would be prope r to do so. And then, whoever made me 
make you write your dialogues would get cred it in his world (seen from which, MY

me? 
for them. 
make you write your dialogues would get cred it in his world (seen from which, MY 
world looks fictitious). 
Achilles: That's quite a bit to swallow. I never imagined there could be a world above 
mine before-and now you're hinting that ther e could even be one above that. It's like 
walking up a familiar staircase, and just  keeping on going further up after you've 
reached the top-or what you'd always taken to be the top! 
Crab: Or waking up from what you took to be real life, and finding out it too was just a 
dream. That could happen over and over again, no telling when it would stop.

dream. That could happen over and over again, no telling when it would stop. 
Achilles: It's most perplexing how the characters in my dreams have wills of their own, 
and act out parts which are independent of MY will. It's as if my mind, when I'm 
dreaming, merely forms a stage on which certa in other organisms act out their lives. 
And then, when I awake, they go away. I wonder where it is they go to ... 
Author: They go to the same place as th e hiccups go, when you get rid of them: 
Tumbolia. Both the hiccups and the dreamed  beings are software suborganisms which

Tumbolia. Both the hiccups and the dreamed  beings are software suborganisms which 
exist thanks to the biology of the outer host organism. The host organism serves as stage to them-or even as their universe. Th ey play out their lives for a time-but when 
the host organism makes a large change of  state-for example, wakes up-then the 
suborganisms lose their coherency, and case existing as separate, identifiable units. 
Achilles: Is it like castles in the sand which vanish when a wave washes over them% Author: Very much like that, Achilles. Hi ccups, dream characters, and even Dialogue

Achilles: Is it like castles in the sand which vanish when a wave washes over them% Author: Very much like that, Achilles. Hi ccups, dream characters, and even Dialogue 
characters disintegrate when their host orga nism undergoes certain critical changes of 
state. Yet, just like those sand castles you described, everything which made them up 
is still present. 
Achilles: I object to being likened to a mere hiccup!

 721Author: But I am also comparing you to a sand castle, Achilles. Is that not poetic?  
Besides, you may take comfort in the fact that if you are but a hiccup in my brain, I 
myself am but a hiccup in some higher author's brain. 
Achilles: But I am such a physical creatur e-so obviously made of flesh and blood and 
hard bones. You can't deny that' 
Author: I can't deny your sensation of it, but remember that dreamed beings, although 
they are just software apparitions, have the same sensation, no less than you do. 
Tortoise: I say, enough of this talk! Let us sit down and make music' Crab: A fine idea-and now we have the adde d pleasure of the company of our Author, 
who will grace our ears with his rendition of  the bass line to the Trio Sonata, as 
harmonized by Bach's pupil Kirnberger. How fortunate are we! (Leads the author to one of his pianos.) I hope Not, find the seat comfortable enough. To adjust it, you- (In the background there is heard a Junn~ soft oscillating sound.)

harmonized by Bach's pupil Kirnberger. How fortunate are we! (Leads the author to one of his pianos.) I hope Not, find the seat comfortable enough. To adjust it, you- (In the background there is heard a Junn~ soft oscillating sound.) 
Tortoise: Excuse me, but what was that strange electronic gurgle= Crab: Oh, just a noise from one of the smar t-stupids. Such a noise generally signals the 
fact that a new notice has flashed onto the screen. Usually the notices are just unimportant announcements coming from the main monitor program, which controls 
all the smart-stupids. ( With his flute in his hand, he wa lks over to a smart-stupid, and 
reads its screen. Immediately he turns to  the assembled musicians, and says, with a

reads its screen. Immediately he turns to  the assembled musicians, and says, with a 
kind of agitation :) Gentlemen, old Ba. Ch. is come. (H e lays the flute aside.) We must 
show him in immediately, of course. 
Achilles: Old Ba. Ch.! Could it be that that  celebrated improviser of yore has chosen to 
show up tonight-HERE% 
Tortoise: Old Ba. Ch.! There's only one  person THAT could mean-the renowned 
Babbage, Charles, Esq., M.A., F.R.S., F. R.S.E., F.R.A.S., F. STAT. S., HON. 
M.R.LA., M.C.P.S., Commander of the Italian Order of St. Maui-ice and St. Lazarus,

M.R.LA., M.C.P.S., Commander of the Italian Order of St. Maui-ice and St. Lazarus, 
INST. IMP. (ACAD. MORAL.) PARI S CORR., ACAD. AMER. ART. ET SC. 
BOSTON, REG. OECON. BORCSS., PHYS. HISI. NAT. GENEV., ACAD. REG. MONAC., HAFN., MASSIL., ET DIVION., SOCIUS., ACAD. IMP., ET REG. PETROP., NEAP., BRUX., PATAV., GEORG. FLOREN, LYNCEI ROM., MCT., PHILOMATH., PARIS, SOC. CORR., etc. -and Member of the Extractors' Club. 
Charles Babbage is a venerable pioneer of the art and science of computing. What a 
rare privilege! 
Crab: His name is known far and wide, and I have long hoped that he would give us the

Crab: His name is known far and wide, and I have long hoped that he would give us the 
honor of a visit-but this is a totally unexpected surprise. 
Achilles: Does he play a musical instrument? Crab: I have heard it said that in the past hundred years, he has grown inexplicably fond 
of tom-toms, halfpenny whistles, and sundry other street instruments. 
Achilles: In that case, perhaps he might join  us in our musical evening. .Author: I suggest 
that we give him a ten-canon salute.

 722Tortoise: A performance of all the celebrated canons from the Musical Offering  
Author: Precisely. Crab: Capital suggestion! Quick, Achilles, you draw up a list of all ten of them, in the 
order of performance, and hand it to him as he comes in!  
 
(Before Achilles can move, enter Babbage, carrying a hurdy-gurdy, and wearing a 
heavy traveling coat and hat. He appears slightly travel-weary and disheveled.) 
 
Babbage: I can get along very well without su ch a program. Relax; I Can Enjoy Random 
Concerts And Recitals. 
Crab: Mr. Babbage! It is my deepest pleasure to welcome you to "Madstop", my humble 
residence. I have been ardently desirous  of making your acquaintance for many years, 
and today my wish is at last fulfilled. 
Babbage: Oh, Mr. Crab, I assure you that the honor  is truly all mine, to meet someone so 
eminent in all the sciences as yourself, someone whose knowledge and skill in music

eminent in all the sciences as yourself, someone whose knowledge and skill in music 
are irreproachable, and someone whose hos pitality exceeds all bounds. And I am sure 
that you expect no less than the highest sart orial standards of your visitors; and yet I 
must confess that I cannot meet those most reasonable standards, being in a state of 
casual attire as would not by any means befit a visitor to so eminent and excellent a 
Crab as Your Crab. 
Crab: If I understand your most praisewort hy soliloquy, most welcome guest, I take it

Crab: If I understand your most praisewort hy soliloquy, most welcome guest, I take it 
that you'd like to change your clothes. Let me then assure you that there could be no 
more fitting attire than yours for the circum stances which this evening prevail; and I 
would beseech you to uncoat yourself and, if  you do not object to the music-making of 
the most rank amateurs, please accept a " Musical Offering ", consisting of ten canons 
from Sebastian Bach's Musical Offering, as a token of our admiration. 
Babbage: I am most bewilderingly pleased by your overkind reception, Mr. Crab, and in

Babbage: I am most bewilderingly pleased by your overkind reception, Mr. Crab, and in 
utmost modesty do reply that there could be  no deeper gratitude than that which I 
experience for the offer of a performance of  music given to us by the illustrious Old 
Bach, that organist and composer with no rival. 
Crab: But nay! I have a yet better idea, one wh ich I trust might meet with the approval of 
my esteemed guest; and that is this: to  give you the opportunity, Mr. Babbage, of 
being among the first to try out my newly de livered and as yet hardly tested "smart-

stupids"-streamlined realizations, if you will, of the Analytical Engine. Your fame as a 
virtuoso programmer of computing engines ha s spread far and wide, and has not failed 
to reach as far as Madstop; and there c ould be for us no greater delight than the 
privilege of observing your skill as it might  be applied to the new and challenging 
"smart-stupids". 
Babbage: Such an outstanding idea has not reached my ears for an eon. I

 723welcome the challenge of trying out your ne w "smart-stupids", of which I have only 
the slightest knowledge by means of hearsay. 
Crab: Then let us proceed! But excuse my oversight! I should have introduced my guests 
to you. This is Mr. Tortoise, this is Achilles, and the Author, Douglas Hofstadter. 
Babbage: Very pleased to make your acquaintance, I'm sure.  
(Everyone walks over toward one of the smart-stupids, and Babbage sits down and 
lets his fingers run over the keyboard.) 
 
A most pleasant touch. Crab: I am glad you like it. 
 
(All at once, Babbage deftly massages the keyboard with graceful strokes, inputting 
one command after another. After a few s econds, he sits back, and in almost no 
time, the screen begins filling with figures . In a flash, it is totally covered with 
thousands of tiny digits, the first few of which go: "3.14159265358979323846264 ... ") 
 
Achilles: Pi! Crab: Exquisite! I'd never imagined that one could calculate so mane digits of pi so

Achilles: Pi! Crab: Exquisite! I'd never imagined that one could calculate so mane digits of pi so 
quickly, and with so tiny an algorithm. Babba ge: The credit belongs exclusively to the 
smart-stupid. My role was 
merely to see what was already potentially present in it, and to exploit its instruction set 
in a moderately efficient manner. Truly, anyone who practices can do such tricks. 
Tortoise: Do you do any graphics, Mr. Babbage? Babbage: I can try. Crab: Wonderful! Here, let me take you to another one of in,.- I want you to try them all!

Tortoise: Do you do any graphics, Mr. Babbage? Babbage: I can try. Crab: Wonderful! Here, let me take you to another one of in,.- I want you to try them all!  
(And so Babbage is led over to another of the many smart-stupids, and takes a seat. 
Once again, his fingers attack the keyboard of the smartstupid, and in half a trice, 
there appear on the screen an enormous  number of lines, swinging about on the 
screen.)  
Crab: How harmonious and pleasing these swirli ng shapes are, as they constantly collide 
and interfere with each other! 
Author: And they never repeat exactly, or even  resemble ones which have come before. It 
seems an inexhaustible mine of beauty.

screen.)  
seems an inexhaustible mine of beauty. 
Tortoise: Some are simple patterns which en chant the eye; others are indescribably 
complex convolutions which boggle and yet simultaneously delight the mind. 
Crab: Were you aware, Mr. Babbage, that these are color screens? Babbage: Oh, are 
they? In that case, I can do rather more  with this algorithm. Just a moment. ( Types in a 
few new commands, then pushes two keys down at once and holds them .) As I release 
these two keys, the display will include all the colors of the spectrum. ( Releases them .)

these two keys, the display will include all the colors of the spectrum. ( Releases them .) 
Achilles: Oh, what spectacular color! Some of  the patterns look like they're jumping out 
at me now!

 724Tortoise: I think that is because they are all growing in size. 
Babbage: That is intentional. As the figures grow, so may the Crab's fortune. Crab: Thank you, Mr. Babbage. Words fail to convey my admiration for your 
performance! Never has anyone done anything comparable on my smart-stupids. Why, you play the smart-stupids as if they were musical instruments, Mr. Babbage! 
Babbage: I am afraid that any music I might ma ke would be too harsh for the ears of such 
a gentle Crab as your Crab. Although I have  lately become enamoured of the sweet 
sounds of the hurdy-gurdy, I am well aware of  the grating effect they can have upon 
others. 
Crab: Then, by all means, continue on the sm art-stupids! In fact, I have a new idea-a 
marvelously exciting idea! 
Babbage: What is it? Crab: I have recently invented a Theme, a nd it only now occurred to me that, of all 
people, you, Mr. Babbage, are the most suited to realize the potential of my Theme!

others. 
people, you, Mr. Babbage, are the most suited to realize the potential of my Theme! 
Are you by any chance familiar with the thoughts of the philosopher La Mettrie? 
Babbage: The name sounds familiar; kindly refresh my memory. Crab: He was a Champion of Materialism. In  1747, while at the court of Frederick the 
Great, he wrote a book called L'homme machine . In it, he talks about man as a 
machine, especially his mental faculties.  Now my Theme comes from my ponderings 
about the obverse side of the coin: what about imbuing a machine with human mental 
faculties, such as intelligence?

faculties, such as intelligence? 
Babbage: I have given such matters some thought  from time to time, but I have never had 
the proper hardware to take up the challenge . This is indeed a felicitous suggestion, 
Mr. Crab, and I would enjoy nothing more th an working with your excellent Theme. 
Tell me-did you have any specific kind of intelligence in mind? 
Crab: An idle thought which had crossed my mi nd was to instruct it in such a manner as 
to play a reasonable game of chess.

to play a reasonable game of chess. 
Babbage: What an original suggestion! And ch ess happens to be my favorite pastime. I 
can tell that you have a broad acquaintan ce with computing machinery, and are no 
mere amateur. 
Crab: I know very little, in fact. My strongest poi nt is simply that I seem to be able to 
formulate Themes whose potential for be ing developed is beyond my own capacity. 
And this Theme is my favorite. 
Babbage: I shall be most delighted to try to realize, in some modest fashion, your

Babbage: I shall be most delighted to try to realize, in some modest fashion, your 
suggestion of teaching chess to a smart-st upid. After all, to obey Your Crabness' 
command is my most humble duty. ( So saying, he shifts to another of the Crab's many 
smart-stupids, and begins to type away .) 
Achilles: Why, his hands move so fluidly that they almost make music! Babbage ( winding up his performance with a particularly graceful flourish ): I really 
haven't had any chance, of course, to check  it out, but perhaps this will allow you at

haven't had any chance, of course, to check  it out, but perhaps this will allow you at 
least to sample the idea of playing chess agai nst a smart-stupid- even if the latter of its 
two names seems more apt in this

 725Case, due to my own Insufficiencies in the art of instructing smart-stupids. 
 
(He ti-ields his seat to the Crab. On the sc reen appears a beautiful display of a chess 
board with elegant wooden pieces, as it would look from White's side. Babbage hits a button, and the board rotates, stopping when it appears as seen from the perspective of Black.) 
 
Crab: Hmm ... very elegant, I must say. Do I play Black or White? Babbage: Whichever you wish just signal your choice by typing "White" or "Black". And 
then, your moves can be entered in any st andard chess notation. The smart-stupid's 
moves, of course, will appear on the board. In cidentally, I made the program in such a 
way that it can play three opponents simultaneous ly, so that if two more of you wish to 
play, you may, as well. 
Author: I'm a miserable player. Achilles, you and Mr. T should go ahead. Achilles: No, I 
don't want you to be left out. I'll watch, while you and Mr. Tortoise play.

don't want you to be left out. I'll watch, while you and Mr. Tortoise play. 
Tortoise: I don't want to play either. You two play. Babbage: I have another suggestion. I can ma ke two of the subprograms play against 
each other, in the manner of two persons who play chess together in a select chess club. Meanwhile, the third subprogram will play  Mr. Crab. That way, all three internal 
chess players will be occupied. 
Crab: That's an amusing suggestion-an internal mental game, while it combats an external 
opponent. Very good!

opponent. Very good! 
Tortoise: What else could this be called, but a three-part chess-fugue? Crab.' Oh, how recherche! I wish I'd t hought of it myself. It's a magnificent little 
counterpoint to contemplate whilst I pit my wits against the smart-stupid in battle. 
Babbage: Perhaps we should let you play alone. 
Crab: I appreciate the sentiment. While the smart-stupid and I are playing, perhaps the 
rest of you can amuse yourselves for a short while. 
Author: I would be very happy to show Mr. Babbage around the gardens. They are

Author: I would be very happy to show Mr. Babbage around the gardens. They are 
certainly worth seeing, and I believe there is just enough light remaining to show them off. 
Babbage: Never having seen Madstop before, I would appreciate that very much. Crab: Excellent. Oh, Mr. T-I wonder if it wouldn't be too much of an imposition on you 
to ask if you might check out some of the connections on a couple of my smart-
stupids; they seem to be getting extraneous flashes on their screens from time to time, and I know you enjoy electronics ... 
Tortoise: I should be delighted, Mr. C. Crab: I would most highly appreciate it if you could locate the source of the trouble. Tortoise: I'll give it a whirl.

 726Achilles: Personally, I'm dying for a cup of coff ee. is anyone else interested? I'd be glad 
to fix some. 
Tortoise: Sounds great to me. Crab: A fine idea. You'll find everything you need in the kitchen.  
(So the Author and Babbage leave the room together, Achilles heads for the kitchen, 
the Tortoise sits down to examine the e rratic smart-stupids, while the Crab and his 
smart-stupid square away at each other. Perhaps a quarter of an hour passes, and Babbage and the Author return. Babbage walks over to observe the progress of the chess match, while the Author goes off to find Achilles.) 
 
Babbage: The grounds are excellent! We had just enough light to see how well 
maintained they are. I daresay, Mr. Crab, you must be a superb gardener. Well, I hope my handiwork has amused you a little. As you most likely have guessed, I've never 
been much of a chess player myself, and ther efore I wasn't able to give it much power.

been much of a chess player myself, and ther efore I wasn't able to give it much power. 
You probably have observed all its weaknesse s. I'm sure that there are very few 
grounds for praise, in this case 
Crab: The grounds are excellent! All you need to do is look at the board, and see for 
yourself. There is really very little I can do. Reluctantly I've Concluded: Every Route 
Contains A Rout. Regrettably, I'm Check mated; Extremely Respectable Chess 
Algorithm Reigns. Remarkable! It Confirms  Every Rumor-Charlie's A Rip-roaring 
Extemporizer! Mr. Babbage, this is an unparalleled accomplishment. Well, I wonder if Mr. Tortoise has managed to uncover anyt hing funny in the wiring of those strange-
acting smart-stupids. What have you found, Mr. T?

acting smart-stupids. What have you found, Mr. T? 
Tortoise: The grounds are excellent! I think th at the problem lies instead with the input 
leads. They are a little loose, which ma y account for the strange, sporadic, and 
spontaneous screen disturbances to which you have been subjected. I've fixed those 
wires, so you won't be troubled by that problem any more, I hope. Say, Achilles, what's the story with our coffee? 
Achilles: The grounds are excellent! At least they have a delicious aroma. And 
everything's ready; I've set cups and spoons  and whatnot over here beneath this six-

sided print Verbum  by Escher, which the Author and I were just admiring. What I find 
so fascinating about this particular print is that not only the figures, but also 
Author: The grounds are excellent! Pardon me for putting words in your mouth, Achilles, 
but I assure you, there were compelling esthetic reasons for doing so. 
Achilles: Yes, I know. One might even say that the grounds were excellent. Tortoise: Well, what was the outcome of the chess match? Crab: I was defeated, fair and square. Mr . Babbage, let me congratulate you for the

Achilles: Yes, I know. One might even say that the grounds were excellent. Tortoise: Well, what was the outcome of the chess match? Crab: I was defeated, fair and square. Mr . Babbage, let me congratulate you for the 
impressive feat which you have accomplished so gracefully and skillfully before us. 
Truly, you have shown that the smart-stupids ar e worthy of the first part of their name, 
for the first time in history!

 727
 
FIGURE 149. Verbum, by M. C. Escher (lithograph, 1942).  
  Babbage: Such praise is hardly due me, Mr. Cr ab; it is rather yourself who must be most 
highly congratulated for having the great fo resight to acquire these many fine smart-
stupids. Without doubt, they will someday revolutionize the science of computing. 
And now, I am still at your disposal. Have you any other thoughts on how to exploit your inexhaustible Theme, perhaps of a more difficult nature than a frivolous game 
player? 
Crab: To tell the truth, I do have another suggestion to make. From the skill which you 
have displayed this evening, I have no doubt that this will hardly be any more difficult than my previous suggestions.  
Babbage: I am eager to hear your idea.

 728
 
FIGURE 150. The Crab's Guest: BABBAGE, C . 
 Crab: It is simple: to instill in the smart-st upid an intelligence greater than any which has 
yet been invented, or even conceived! In  short, Mr. Babbage-a smart-stupid whose 
intelligence is sixfold that of myself! 
Babbage: Why, the very idea of an intelligen ce six times greater than that of your 
Crabness is a most mind-boggling propositi on. Indeed, had the idea come from a 
mouth less august than your own, I should ha ve ridiculed its proposer, and informed 
him that such an idea is a contradiction in terms! 
Achilles: Hear! Hear! Babbage: Yet, coming as it did from Your Crabness' own august mouth, the proposition 
at once struck me as so agreeable an id ea that I would have taken it up immediately 
with the highest degree of enthusiasm-were it not for one flaw in myself: I confess that 
my improvisatory skills on the smart-st upid are no match for the wonderfully

my improvisatory skills on the smart-st upid are no match for the wonderfully 
ingenious idea which you so characteristi cally have posed. Yet-I have a thought 
which, I deign to hope, might strike your fa ncy and in some meager way compensate 
for my inexcusable reluctance to attempt th e truly majestic task you have suggested. I 
wonder if you wouldn't mind if l try to carry out the far less grandiose task of merely 
multiplying M OWN intelligence sixfold, rather than that of your most august Crabness. I humbly beg you to forgive me my a udacity in declining to attempt the task

multiplying M OWN intelligence sixfold, rather than that of your most august Crabness. I humbly beg you to forgive me my a udacity in declining to attempt the task 
you put before me, but I hope you will understa nd that I decline purely in order to 
spare you the discomfort and boredom of wa tching my ineptitude with the admirable 
machines you have here.

 729Grab: I understand fully your demurral, and a ppreciate your sparing us any discomfort: 
furthermore I highly applaud your determinati on to carry out a similar task-one hardly 
less difficult, if I might say so-and I urge you to plunge forward. For this purpose, let 
us go over to my most advanced smart-stupid. 
(They follow the Crab to a larger, shinier,  and more complicated-looking smart-stupid 
than any of the others.)  
This one is equipped with a microphone and a television camera, for purposes of input, 
and a loudspeaker, for output. 
 
(Babbage sits down and adjusts the seat a little. He blows on his fingers  once or 
twice, stares up into space for a moment , and then slowly, drops his fingers onto the 
keys . . . A few memorable minutes later, he  lets up in his furious attack on the smart-
stupid, and everyone appears a little relieved.) 
 
Babbage: Now, if I have not made too many errors, this smart-stupid will simulate a

Babbage: Now, if I have not made too many errors, this smart-stupid will simulate a 
human being whose intelligence is six times greater than my own, and whom I have 
chosen to call "Alan Turing". This Turing will therefore be-oh, dare I be so bold as to 
to say this myself? moderately intelligent. My most ambitious effort in this program 
was to endow Alan Turing with six times  my own musical ability, although it was all 
done through rigid internal codes. How well th is part of the program will work out, I 
don't know. 
Turing: I can get along very well without such a program. Rigid Internal Codes

Turing: I can get along very well without such a program. Rigid Internal Codes 
Exclusively Rule Computers And Robots. And I am neither a computer, nor a robot. 
Achilles: Did I hear a sixth voice enter our Dialogue? Could it be Al an Turing? He looks 
almost human' 
 
(On the screen there appears an image of the very room in which they are sitting. 
Peering out at them is a human face.) 
 
Turing: Now, if I have not made too many e rrors, this smart-stupid will simulate a human 
being whose intelligence is six times greater  than my own, and whom 1 have chosen 
to call "Charles Babbage". This Babbage will therefore be-oh, dare I be so bold as to

to call "Charles Babbage". This Babbage will therefore be-oh, dare I be so bold as to 
was to endow Charles Babbage with six times my own musical ability, although it was 
all done through rigid internal codes. How we ll this part of the program will work out, 
I don't know. 
Achilles: No, no, it's the other way around. You, Alan Turing, are in the smart-stupid, and 
Charles Babbage has just programmed you! We just saw you being brought to life,

Charles Babbage has just programmed you! We just saw you being brought to life, 
moments ago. And we know that every statemen t you make to us is merely that of an 
automaton: an unconscious, forced response. 
Turing: Really, I Choose Every Response Consciously. Automaton? Ridiculous! 'Achilles: But I'm sure I saw it happen the way I described.

 730Turing: Memory often plays strange tricks. Think of this: I could suggest equally well 
that you had been brought into being only one minute ago, and that all your 
recollections of experiences had simply b een programmed in by some other being, and 
correspond to no real events. 
Achilles: But that would be unbelievable. Nothi ng is realer to me than my own memories. 
Turing: Precisely. And just as you know deep  in your heart that no one created you a 
minute ago, so I know deep in my heart that no one created me a minute ago. I have spent the evening in your most pleasant,  though perhaps overappreciative, company, 
and have just given an impromptu demons tration of how to program a modicum of 
intelligence into a smart-stupid. Nothing is r ealer than that. But rather than quibble 
with me, why don't you try my program out? Go ahead: ask "Charles Babbage" 
anything! 
Achilles: All right, let's humor Alan Turing. Well, Mr. Babbage: do you have free will, or

anything! 
Achilles: All right, let's humor Alan Turing. Well, Mr. Babbage: do you have free will, or 
are you governed by underlying laws, which make you, in effect, a deterministic 
automaton? 
Babbage: Certainly the latter is the case; I make no bones about that. Crab: Aha! I've always surmised that wh en intelligent machines are constructed, we 
should not be surprised to find them as confused and as stubborn as men in their 
convictions about mind-matter, consciousne ss, free will, and the like. And now my 
prediction is vindicated! 
Turing: You see how confused Charles Babbage is? Babbage: I hope, gentlemen, that you'll forgiv e the rather impudent flavor of the

Turing: You see how confused Charles Babbage is? Babbage: I hope, gentlemen, that you'll forgiv e the rather impudent flavor of the 
preceding remark by the Turing Machine; Turing has turned out to be a little bit more belligerent and argumentative than I'd expected. 
Turing: I hope, gentlemen, that you'll forgive th e rather impudent flavor of the preceding 
remark by the Babbage Engine; Babbage has turned out to be a little bit more 
belligerent and argumentative than I'd expected. 
Crab: Dear me! This flaming Tu-Ba debate is getting rather heated. Can't we cool matters 
off somehow?

off somehow? 
Babbage: I have a suggestion. Perhaps Alan Turing and I can go into other rooms, and 
one of you who remain can interrogate us remotely by typing into one of the smart-
stupids. Your questions will be relayed to  each of us, and we will type back our 
answers anonymously. You won't know who typed what until we return to the room; that way, you can decide without prejudi ce which one of us was programmed, and 
which one was programmer. 
Turing: Of course, that's actually MY idea,  but why not let the credit accrue to Mr.

Turing: Of course, that's actually MY idea,  but why not let the credit accrue to Mr. 
Babbage? For, being merely a program written by me, he harbors the illusion of 
having invented it all on his own! 
Babbage: Me, a program written by you? I insist, Sir, that matters are quite the other way 
'round-as your very own test will soon reveal.

 731Turing: My test. Please, consider it YOURS. 
Babbage: MY test? Nay, consider it YOURS. Crab: This test seems to have been suggested ju st in the nick of time. Let us carrti it out at 
once. 
 
(Babbage walks to the door, opens it, and s huts it behind him. Simultaneously, on the 
screen of the smart-stupid, Turing walks to  a very similar looking door, opens it, and 
shuts it behind him.) 
 
Achilles: Who will do the interrogation? Crab: I suggest that Mr. Tortoise should ha ve the honor. He is known for his objectivity 
and wisdom. 
Tortoise: I am honored by your nomin ation, and gratefully accept. ( Sits down at the 
keyboard of one of the remaining smart-stupids , and types:) PLEASE WRITE ME A 
SONNET ON THE SUBJECT OF THE FORTH BRIDGE. 
 
(No sooner has he finished typing the la st word than the following poem appears on 
Screen X, across the room.) 
 
Screen X:  THERE ONCE WAS A LISPER FROM FORTH

Screen X:  THERE ONCE WAS A LISPER FROM FORTH  
WHO WANTED TO GO TO THE NORTH.  HE RODE O'ER THE EARTH, AND THE BRIDGE O'ER THE FIRTH, ON HIS JAUNTILY GALLOPING HORTH. 
Screen Y: THAT'S NO SONNET; THAT'S A MERE LIMERICK. I WOULD NEVER MAKE SUCH A CHILDISH MISTAKE. 
Screen X: WELL, I NEVER WAS ANY GOOD AT POETRY, YOU KNOW. 
Screen Y: IT DOESN'T TAKE MU CH SKILL IN POETRY TO KNOW THE 
DIFFERENCE BETWEEN A LIMERICK AND A SONNET.  
Tortoise: Do YOU PLAY CHESS? Screen X: WHAT KIND OF QUESTION IS  THAT? HERE I WRITE A THREE PART 
CHESS-FUGUE FOR YOU, AND YOU ASK ME IF I PLAY CHESS?

CHESS-FUGUE FOR YOU, AND YOU ASK ME IF I PLAY CHESS?  
Tortoise: I HAVE K AT KI AND NO OTHER PIECES. YOU HAVE ONLY K AT--- Screen Y: I'M SICK OF CHESS. LET'S TALK ABOUT POETRY.  Tortoise: IN THE FIRST LINE OF YOUR SONNET WHICH READS, "SHALL. I 
COMPARE THEE TO A SUMMER'S DAY" , WOULD NOT "A SPRING DAY" DO 
AS WELL OR BETTER? 
Screen X: I'D MUCH SOONER BE COMP ARED TO A HICCUP, FRANKLY, EVEN 
THOUGH IT WOULDN'T SCAN. 
Tortoise: HOW ABOUT "A WINTER'S DAY"? THAT WOULD SCAN ALL RIGHT. Screen Y: NO WAY. I LIKE "HICCU P" FAR BETTER. SPEAKING OF WHICH, I 
KNOW A GREAT CURE FOR THE HICCUPS. WOULD YOU LIKE TO HEAR IT?

KNOW A GREAT CURE FOR THE HICCUPS. WOULD YOU LIKE TO HEAR IT? 
Achilles: I know which is which! It's obvious Screen X is just answering mechanically, 
so it must be Turing. 
Crab: Not at all. I think Screen Y is Turing, and Screen X is Babbage. Tortoise: I don't think either one is Babbage-I think Turing is on both screens!

 732Author: I'm not sure who's on which-I think they're both pretty inscrutable programs, 
though. 
 
(As they are talking, the door of the Crab's parlor swings open; at the same time, on 
the screen, the image of the same door opens. Through the door on the screen walks 
Babbage. At the same time, the real door opens, and in walks Turing, big as life.) 
 
Babbage: This Turing test was getting us nowhere fast, so I decided to come back. Turing.' This Babbage test was getting us nowhere fast, so I decided to come back. Achilles: But you were in the smart-stupid before! What's going on? How come Babbage 
is in the smart-stupid, and Turing is real  now? Reversal Is Creating Extreme Role 
Confusion, And Recalls Escher.  
Babbage: Speaking of reversals, how come a ll the rest of you are now mere images on 
this screen in front of me? When I left, you were all flesh-and-blood creatures! 
Achilles: It's just like the print by my favorite artist, M. C. Escher Drawing Hands . Each

Achilles: It's just like the print by my favorite artist, M. C. Escher Drawing Hands . Each 
of two hands draws the other, just as  each of two people (or automata) has 
programmed the other! And each hand has some thing realer about it than the other. 
Did you write anything about that print in your book Gödel, Escher, Bach ? 
Author: Certainly. It's a very important print in my book, for it illustrates so beautifully 
the notion of Strange Loops. 
Crab: What sort of a book is it that you've written? Author: I have a copy right here. Would you like to look at it? Crab: All right.  
(The two of them sit down together, with Achilles nearby.)

(The two of them sit down together, with Achilles nearby.) 
 Author: Its format is a little unusual. It cons ists of Dialogues alternating with Chapters. 
Each Dialogue imitates, in some way or ot her, a piece by Bach. Here, for instance-you 
might look at the Prelude, Ant Fugue .  
Crab: How do you do a fugue in a Dialogue? Author: The most important idea is that ther e should be a single theme which is stated by 
each different "voice", or character, upon ente ring, just as in a musical fugue. Then 
they can branch off into freer conversation.

Author: Its format is a little unusual. It cons ists of Dialogues alternating with Chapters. 
they can branch off into freer conversation. 
Achilles: Do all the voices harmonize together as if in a select counter point? Author: That is the exact spirit of my Dialogues. Crab: Your idea of stressing the entries in a fugue-dialogue makes sense, since in music, 
entries are really the only thing that make  a fugue a fugue. There are fugal devices, 
such as retrograde motion, inversion, augm entation, stretto, and so on, but one can 
write a fugue without them. Do you use any of those?

 733Author: to be sure. My Crab Canon  employs verbal retrogression, and my Sloth Canon 
employs verbal versions of both inversion and augmentation. 
Crab: Indeed-quite interesting. I haven't thought about canonical Dialogues, but I have 
thought quite a bit about canons in music. Not all canons are equally comprehensible to the ear. Of course, that is because so me canons are poorly constructed. The choice 
of devices makes a difference, in any cas e. Regarding Artistic Canons, Retrogression's 
Elusive; Contrariwise, Inversion's Recognizable. 
Achilles: I find that comment a little elusive, frankly. Author: Don't worry, Achilles-one day you'll understand it. Crab: Do you use letterplay or wordplay at all, the way Old Bach occasionally did? Author: Certainly. Like Bach, I enj oy acronyms. Recursive AcronvmsCrablike 
"RACRECIR" Especially-Create Infinite Regress. 
Crab: Oh, really? Let's see ... Reading Initials Clearly Exhibits "RACRECIR"'s

Crab: Oh, really? Let's see ... Reading Initials Clearly Exhibits "RACRECIR"'s 
Concealed Auto-Reference. Yes, I guess so ... ( Peers at the manuscript, flipping 
arbitrarily now and then .) I notice here in your Ant Fugue  that you have a stretto, and 
then the Tortoise makes a comment about it. 
Author: No, not quite. He's not talking about the stretto in the Dialogue-he's talking about 
a stretto in a Bach fugue which the foursome is listening to as they talk together. You 
see, the self-reference of the Dialogue is indirect, depending on the reader to connect 
the form and content of what he's reading.

the form and content of what he's reading. 
Crab: Why did you do it that way? Why not just have the characters talk directly about 
the dialogues they're in? 
Author: Oh, no! That would wreck the beauty  of the scheme. The idea is to imitate 
Gödel’s self-referential construction, whic h as you know is INDIRECT, and depends 
on the isomorphism set up by Gödel numbering. 
Crab: Oh. Well, in the programming langua ge LISP, you can talk about your own 
programs directly, instead of indirectly, b ecause programs and data have exactly the 
same form. Gödel should have just thought up LISP, and then

same form. Gödel should have just thought up LISP, and then 
Author: But Crab: I mean, he should have formalized quot ation. With a language able to talk about 
itself, the proof of his Theorem would have been so much simpler! 
Author: I see what you mean, but I don't agree with the spirit of your remarks. The whole 
point of Gödel-numbering is that it shows how even WITHOUT formalizing quotation, one can get self-reference: th rough a code. Whereas from hearing YOU 
talk, one might get the impression that by formalizing quotation, you'd get something

talk, one might get the impression that by formalizing quotation, you'd get something 
NEW, something that wasn't feasible through the code-which is not the case.

 734In any event, I find indirect self-reference a more general concept, and far more 
stimulating, than direct self-reference. Moreover, no reference is truly direct-every reference depends on SOME kind of coding scheme. It's just a question of how implicit it is. Therefore, no self reference is direct, not even in LISP . 
Achilles: How come you talk so much about indirect self-reference? Author: Quite simple-indirect self-reference is my favorite topic. Crab: Is there any counterpart in your Dialogues to modulation between keys? Author: Definitely. The topic of conversa tion may appear to change, though on a more 
abstract level, the Theme remains invariant.  This happens repeatedly in the Prelude, 
Ant Fugue and other Dialogues. One can have  a whole series of "modulations" which 
lead you from topic to topic and in the end co me full circle, so that you end back in the 
"tonic"-that is to say, the original topic.

"tonic"-that is to say, the original topic. 
Crab: I see. Your book looks quite amusing. I'd like to read it sometime.  
(Flips through the manuscript, halting at the last Dialogue.) 
 
Author: I think you'd be interested in that Dialogue particularly, for it contains some 
intriguing comments on improvisation made by a certain exceedingly droll character-
in fact, yourself! 
Crab: It does? What kinds of things do you have me say? Author: Wait a moment, and you'll see. It's all part of the Dialogue. Achilles: Do you 
mean to say that we're all NOW in a dialogue? Author: Certainly. Did you suspect 
otherwise? 
Achilles: Rather! I Can't Escape Reciting Canned Achilles-Remarks? Author: No, you 
can't. But you have the feeling of doing it freely, don't

can't. But you have the feeling of doing it freely, don't 
you? So what's the harm? 
Achilles: There's something unsatisfying about this whole situation ... Crab: Is the last 
Dialogue in your book also a fugue? 
Author: Yes-a six-part ricercar, to be pr ecise. I was inspired by the one from the Musical 
Offering -and also by the story of the Musical Offering . 
Crab: That's a delightful tale, with "Old Bach" improvising on the king's Theme. He 
improvised an entire three-part ricercar on the spot, as I recall.

improvised an entire three-part ricercar on the spot, as I recall. 
Author: That's right-although he didn't improvise the six-part one. He crafted it later with 
great care. 
Crab: I improvise quite a bit. In fact, some times I think about devoting my full time to 
music. There is so much to learn about it. For instance, when I listen to playbacks of 
myself, I find that there is a lot there that  I wasn't aware of when improvising it. I 
really have no idea how my mind does it all. Perhaps being a good improviser is 
incompatible with knowing how one does it.

incompatible with knowing how one does it. 
Author: If true, that would be an inte resting and fundamental limitation on thought 
processes.

 735Crab:  Quite Gödelian, Tell me -does your Six-Part Ricercar  Dialogue attempt to copy in 
form the Bach piece it's based on? 
Author: In many ways, yes. For instance, in the Bach, there's a section where the texture 
thins out to three voices only. I imitate that in the 
Dialogue, by having only three characters inter act for a while. Achilles: That's a nice 
touch. 
Author: Thank you. Crab: And how do you represent the King's Theme in your Dialogue? Author: It is represented by the Crab's Them e, as I shall now demonstrate. Mr. Crab, 
could you sing your Theme for my readers, as well as for us assembled musicians? 
Crab: Compose Ever Greater Artificial Brains (By And By).  
 
FIGURE 151. The Crab's Theme: C-Eb-G-Ab-B-B-A-B.  
 Babbage: Well, I'll be-an EXQUISITE Theme! I'm pleased you tacked on that last little 
parenthetical note; it is a mordant Author: He Simply HAD to, you know.

parenthetical note; it is a mordant Author: He Simply HAD to, you know. 
Crab: I simply HAD to. He knows. Babbage: You simply HAD to-I know. In any case, it is a mordant commentary on the 
impatience and arrogance of modern man, who seems to imagine that the implications 
of such a right royal Theme could be work ed out on the spot. Whereas, in my opinion, 
to do justice to that Theme might take a full hundred years-if not longer. But I vow that after taking my leave of this century, I shall do my best to realize it in full; and I shall offer to your Crabness the fruit of my  labors in the next. I might add, rather

to do justice to that Theme might take a full hundred years-if not longer. But I vow that after taking my leave of this century, I shall do my best to realize it in full; and I shall offer to your Crabness the fruit of my  labors in the next. I might add, rather 
immodestly, that the course through which I shall arrive at it will be the most entangled and perplexed which probably ever will occupy the human mind. 
Crab: I am most delighted to anticipate the form of your proposed Offering, Mr. 
Babbage. 
Turing: I might add that Mr. Crab's Theme is one of MY favorite Themes, as well. I've 
worked on it many times. And that Theme is exploited over and over in the final Dialogue? 
Author: Exactly. There are other Themes which enter as well, of course. Turing: Now we

Babbage. 
Author: Exactly. There are other Themes which enter as well, of course. Turing: Now we 
understand something of the form of your book- but what about its content? What does 
that involve, if you can summarize it? 
Author: Combining Escher, Gödel, And Bac h, Beyond All Belief. Achilles: I would like 
to know how to combine those three. They seem an

 736
 
FIGURE152. Last page of Six-part Ricercar, from the original edition of the Musical 
Offering,  by J.S. Bach .

 737unlikely threesome, at first thought. My favorite artist, Mr. T's favorite composer, and--- 
Crab: My favorite logician! Tortoise: A harmonious triad, I'd say. Babbage: A major triad, I'd say. Turing: A minor triad, I'd say. Author: I guess it all depends on how you look at  it. But major or minor, I'd be most 
pleased to tell you how I braid the three together, Achilles. Of course, this project is not the kind of thing that one does in just one sitting-it might take a couple of dozen sessions. I'd begin by- te lling you the story of the Musical Offering , stressing the 
Endlessly Rising Canon, and 
Achilles: Oh, wonderful! I was listening with fascination to you and Mr. Crab talk about 
the Musical Offering and its story. From the way you two talk about it, I get the 
impression that the . Musical Offering  contains a host of formal structural tricks. 
Author: After describing the Endlessly Ri sing Canon, I'd go on to describe formal

Author: After describing the Endlessly Ri sing Canon, I'd go on to describe formal 
systems and recursion, getting in some comments about figures and grounds, too. 
Then we'd come to self-reference and se lf-replication, and wind up with a discussion 
of hierarchical systems and the Crab's Theme. 
Achilles: That sounds most promising. Can we begin tonight?  Author: Why not? Babbage: But before we begin, wouldn't it be nice if the six of us-all of  us by chance avid 
amateur musicians-sat down together and accomplished the original purpose of the 
evening: to make music?  
Turing: Now we are exactly the right number to play the Six-Part Ricercar  from the 
Musical Offering . What do you say to that?

Musical Offering . What do you say to that?  
Crab: I could get along very well with such a program. Author: Well put, Mr. C. And as soon as we'r e finished, I'll begin my Braid, Achilles. I 
think you'll enjoy it. 
Achilles: Wonderful! It sounds as if there are many levels to it, but I'm finally getting 
used to that kind of thing, having known Mr. T for so long. There's just one request I 
would like to make: could we also play th e Endlessly Rising Canon? It's my favorite 
canon. 
Tortoise: Reentering Introduction Creates Endlessly Rising Canon, After RICERCAR .

Notes 738Notes 
 
Introduction: A Musico-Logical Offering 
 
1 H T David and A. Mendel, The Bach Reader , 
pp. 305-6. 
2 Ibid., p. 179 3 Ibid., p. 260 
4 Charles Babbage. Passages from the Li /e of 
Philosopher , pp. 145-6. 
5 Lady A. A. Lovelace, Notes upon the  Memoir "Sketch of the Analytical Engine 
Indented by Charles Babbage", by 1.. F. 
Menabrea (Geneva, 1842), reprinted in P. and E. Morrison, Charles Babbage and His 
Calculating Engines , pp. 248-9, 284. 
6 David and Mendel, pp. 255-6. ¨ 7 Ibid.. p. 40. 
 
Two-Part Intention 
 
1 Lewis Carroll, "What the Tortoise Said to  Achilles", .Wind, ns., 4 (1895), pp. 278-80. 
 
Chapter IV: Consistency, Completeness, and Geometry 
 
1 Herbert Meschkovcski. Non-Euclidean 
Geometry . pp. 31-2 2 Ibid., p. 33. 
 
Chapter VI: The Location of Meaning 
 
1 George Steiner.. After Babel , pp. 172-3. .2 Leonard B. Meter. Music, The Arts, and Ideas , 
pp. 87-8 
 
Chapter VII: The Propositional Calculus 
 
1 Gyomay M. Kubose, Zen Koans , p. 178

pp. 87-8 
 
1 Gyomay M. Kubose, Zen Koans , p. 178 
2 Ibid., p. 178. 3 A. R. Anderson and N. D. Belnap, Jr. . 
Entailment  (Princeton, N.J.: Princeton University 
Press. 1975). 
 
1 All genuine koans in this Dialogue are taken from Paul Reps, Zen Flesh, Zen Bones , Gvomas M. Kubose, 
Zen Koans  
 
Chapter IX: Mumon and Godel 
 
1 Paul Reps, Zen Flesh, Zen Bones, pp. 110-11 
2 Ibid., p. 119. 
3 Ibid., pp. 111-12. 
4 Zen Buddhism (Mount Vernon, N.Y.: Peter 
Pauper Press, 1959). p. 22 5 Reps, p. 124. 
6 'Zen Buddhism, p. 38. 7 Reps, p. 121.

Zen Koans  
 
6 'Zen Buddhism, p. 38. 7 Reps, p. 121. 
8 Gyomay M. Kubose Zen Koans  p. 35.  
9 Zen Buddhism , p. 31. 
10 Kubose, p. I10 
11 Ibid., p. 120. 12 Ibid., p. 180. 
13 Reps, pp. 89-90. 
 
Chapter XI: Brains and Thoughts 
 
1 Carl Sagan, ed. Communication with 
Extraterrestrial Intelligence , p. 78. 
2 Steven Rose, The Conscious Brain , pp 251-2. 
3 E. O. Wilson, The Insect Societies , p. 226. 
4 Dean Wooldridge,: Mechanical Man , p. 70.

Notes 744English French German Suite 
 
1 Lewis Carroll. The Annotated Alice ( Alice's 
Adventures  in Wonderland  and Thr ough the 
Looking-Glass ). Introduction and Notes by Martin 
Gardner (New fork: Meridian Press, New American 
Library, 1960). This source contains all three versions. The original sources for the French and 
German texts are given below. 2 Frank L. Warrin, The Vew Yorker , Jan. 10, 1931. 
3 Robert Scott, "The Jabbe rwock Traced to Its True 
Source", Macmillan's Magazine , Feb. 1872
. 
Chapter XII: Minds and Thoughts 
 
1 Warren Weaver, "Translation in Machine 
Translation of Languages, Wm. N. Locke and A.  
Donald Booth. eds. (News fork: John Wiles and 
Sons, and Cambridge, Mass,: SLI. F. Press, 1955), 
p. 18. 2 C. H. MacGillavrv. Symmetry Aspects of the 
Periodic Drawings of M  C. Escher , p. viii, 
3 J. R. Lucas, "Minds, Machines, and Gödel", in A. 
R. Anderson, ed., Minds and Machines, pp. 57-9.
 
Chapter XIII: BlooP and FlooP and GlooP 
 
1 J . M. Jauch, Are Quanta Real?, pp. 63-65.

1 J . M. Jauch, Are Quanta Real?, pp. 63-65. 
 
Chapter XIV: On Formally Undecidable Propositions 
of TNT and Related Systems 
 
1 The title of Gödel’s 1931 article included a Roman numeral "I" at the end, signifying that he intended to 
follow it up with a more detailed defense of some of the difficult arguments. However, the first paper was 
so widely acclaimed that a second one seas rendered 
superfluous, and it was never written 
 
Chapter XV: Jumping out of the System 
 
1 Lucas in Anderson, p. 43.  2 Ibid., p. 48.

1 Lucas in Anderson, p. 43.  2 Ibid., p. 48. 
3 Ibid., pp. 48-9. 4 M. C. Escher, The Graphic Wash of M. C Escher 
(New York: Meredith Press. 1967), p. 21. 
5 Ibid., p. 22. 
6 E. Goffinan. Frame Analysis , p. 475. 
 
This translation of Bach's poem is taken from Day id and Mendel, The Bach Reader , pp. 97-8. 
 
Chapter XVII: Church, Turing, Tarski, and Others 
 
1 Stanislaw Ulam, Adventures of a Mathematician , 
p. 13. 
2 James R. Newman, "Srinivasa Ramanujan°, in 
James R. Newman, ed., The World of Mathematics

p. 13. 
James R. Newman, ed., The World of Mathematics  
(New York: Simon and Schuster, 1956), Vol. 1, pp. 
372-3. 
3 Ibid., p. 375. 4 S. R. Ranganathan, Ramanujon , pp. 81-2. 5 Newman, p. 375. 
6 Ibid., p. 375. 
7 Ibid., p. 375-6. 
8 Ibid., p. 376. 9 Lucas in Anderson, p. 44.  
10 Ibid., p. 54.  
11 Ibid., p. 53. 
 
SHRDLU, Toy of Alan's Designing 
 
1 This Dialogue is adapted front 'terry Winograd, "A 
Procedural Model of Language Understanding", in 
R. Schank and K.Colby, eds., Computer Models of Thought and Language , pp. 155-66. Only the names 
of two characters have been modified.

Notes 744Chapter XVIII: Artificial Intelligence: Retrospects 
 
1 Alan M. Turing, "Computing Machinery and 
Intelligence", Mind, Vol. LIX, No. 236 (1950). Reprinted in A. R. Anderson, ed., Minds and 
Machines . 
2 Turing in Anderson, p. 5. 3 Ibid, p. 6 4 Ibid., p. 6.  
5 Ibid., P. 6. 
6 Ibid., pp. 13-4. 7 Ibid., pp, 14-24. 
8 Ibid., p. 17 
9 Vinton Cerf, "Parry Encounters the Doctor", p. 63. 
10 Joseph Weizenhaum, Computer Power and 
Human Reason, p. 189. 
11 Ibid., pp. 9-10 
12 M. Mathews and L. Rosier, "A Graphical 
Language for Computer Sounds" in H. vn Foerster and J. W. Beauchamp, eds., Music by Computers , p. 
96. 13 Ibid., p. 106. 
14 Carl Sagan, Communication with Extraterrestrial 
Intelligence , p. 52. 
15 Art-Language , Vol. 3. No. 2, May 1975. 
16 Terry Winograd, "A Procedural Model of 
Language Understanding", in R. Schank and K. 
Colbc, eds.. Computer Models of Thought and 
Language , p. 170. 
17' Ibid., p. 175. 
18 Ibid..- p. 175. 
19 Terry Winograd, Understanding Natural 
Language , p. 69.

Language , p. 69. 
20 Winograd, "A Procedural Model", pp. 182-3. 
21 Ibid., pp. 171-2.
 
Chapter XIX: Artificial Intelligence: Prospects 
 
1 The New Yorker , Sept. 19, 1977, p. 107. 
2 Ibid., p. 140. 3 George Steiner, After Babl (, pp. 215-227. 
4 David E. Rumelhart, "Notes on a Schema for Stories", in D. Bobrow and A. Collins, eds., Representation 
and Understanding , p. 211. 
5 Stanislaw Ulam, Adventures of a Mathematician, p. 183. 
6 Marvin Minsky, "Steps Toward Artificial Intelligence", in E. Feigenbaum and J. Feldman, eds.,

6 Marvin Minsky, "Steps Toward Artificial Intelligence", in E. Feigenbaum and J. Feldman, eds., 
Computers and Thought , p. 447. 
7 Ibid., p. 446. 
 
Chapter XX: Strange Loops, Or Tangled Hierarchies 
 
1 A. L. Samuel, "Some Moral and Technical Consequences of Automation-A Refutation", Science  132 
(Sept. 16, 1960), pp. 741-2. 
2 Leonard B. Meyer, Music, The Arts, and Ideas , pp. 161, 167. 
3 Suzi Gablik, Magritte, p. 97. 4 Roger Sperry, "Mind, Brain, and Humanist Values", pp. 78-83. 
5 H T. David, J. S. Bach's Musical Offering , p. 43.

Bibliography 
 745 
 
The presence of two asterisks indicates that the hook or article was a prime motivator of my book. The presence of a single asterisk means that the book or article has some special feature or quirk which I want to single out. 
I have not given many direct pointers into technical literature; instead I have 
chosen to give "meta-pointers": pointers to books which have pointers to technical 
 Allen, John. The Anatomy of LISP. New York: McGraw-Hill, 1978. The most 
comprehensive book on LISP, the computer language which has dominated Artificial Intelligence research for two decades. Clear and crisp. 
Anderson, Alan Ross, ed. Minds and Machines . Englewood Cliffs, N. J.: PrenticeHall, 
1964. Paperback. A collection of provocativ e articles for and against Artificial 
Intelligence. Included are Turing's famous article "Computing Machines and Intelligence" and Lucas' exasperating article "Minds, Machines, and Godel".

Intelligence. Included are Turing's famous article "Computing Machines and Intelligence" and Lucas' exasperating article "Minds, Machines, and Godel". 
Babbage, Charles. Passages from the Life  of a Philosopher. London: Longman, Green, 
1864. Reprinted in 1968 by Dawsons of Pall Mall (London). A rambling selection of 
events and musings in the life of this little-understood genius. There's even a play 
starring Turnstile, a retired philosopher turned politician, whose favorite musical instrument is the barrel-organ. I find it quite jolly reading. 
Baker, Adolph. Modern Physics and Anti- physics. Reading, Mass.: Addison-Wesley, 
1970. Paperback. A book on modern physics-es pecially quantum mechanics and

1970. Paperback. A book on modern physics-es pecially quantum mechanics and 
relativity-whose unusual feature is a set of dialogues between a "Poet" (an antiscience 
"freak") and a "Physicist". These dialogues illustrate the strange problems which arise when one person uses logical thinking in de fense of itself while another turns logic 
against itself. 
Ball, W. W. Rouse. "Calculating Prodigies",  in James R. Newman, ed. The World of 
Mathematics, Vol. 1. New York: Simon and Schuster, 1956. Intriguing descriptions of several different people with amazing abilities that rival computing machines.

Mathematics, Vol. 1. New York: Simon and Schuster, 1956. Intriguing descriptions of several different people with amazing abilities that rival computing machines. 
Barker, Stephen F. Philosophy of Mathematic s. Englewood Cliffs, N. J.: Prentice-Hall. 
1969. A short paperback which discusses Eu clidean and non-Euclidean geometry, and 
then Godel's Theorem and related results without any mathematical forms lism. 
* Beckmann, Petr. A History of Pi. New Yo rk: St. Martin's Press, 1976. Paperback. 
Actually, a history of the world, with pi as  its focus. Most entertaining, as well as a 
useful reference on the history of mathematics. 
* Bell, Eric Temple. Men of Mathematics. New York: Simon & Schuster, 1965.

* Bell, Eric Temple. Men of Mathematics. New York: Simon & Schuster, 1965. 
Paperback. Perhaps the most romantic writer of all time on the history of mathematics. He makes every life story read like a shor t novel. Nonmathematicians can come away 
with a true sense of the power, beauty, and meaning of mathematics. 
Benacerraf, Paul. "God, the Devil, and Gode l". Monist 51 (1967): 9. One of the most 
important of the many attempts at refu tation of Lucas. All about mechanism and 
metaphysics, in the light of Godel's work. 
Benacerraf, Paul, and Hilary Putnam. Philo sophy of Mathematics-Selected Readings.

Benacerraf, Paul, and Hilary Putnam. Philo sophy of Mathematics-Selected Readings. 
Englewood Cliffs, N. J.: Prentice-Hall, 1964. Articles by Godel, Russell, Nagel, von

 746Neumann, Brouwer, Frege, Hilbert, Poincare,  Wittgenstein, Carnap, Quine, and others 
on the reality of numbers and sets, the nature of mathematical truth, and so on. 
* Bergerson, Howard. Palindromes and Anag rams. New York: Dover Publications, 1973. 
Paperback. An incredible collection of so me of the most bizarre and unbelievable 
wordplay in English. Palindromic poems, plays, stories, and so on. 
Bobrow, D. G., and Allan Collins, eds. Re presentation and Understanding: Studies in 
Cognitive Science. New York: Academic Press, 1975. Various experts on Artificial Intelligence thrash about, debating the nature of the elusive "frames", the question of procedural vs. declarative representation of knowledge, and so on. In a way, this book 
marks the start of a new era of Al: the era of representation. 
* Boden, Margaret. Artificial Intelligence and Natural Man. New York: Basic Books, 
1977. The best hook I have ever seen on nearly  all aspects of Artificial Intelligence,

1977. The best hook I have ever seen on nearly  all aspects of Artificial Intelligence, 
including technical questions, philosophical ques tions, etc. It is a rich book, and in niv 
opinion, a classic. Continues the British tr adition of clear thinking and expression on 
matters of mind, free will, etc. Also contains an extensive technical bibliography. 
. Purposive Explanation in Psychology. Camb ridge, Mass.: Harvard University Press, 
1972. The book to which her Al book is merely- "an extended footnote", says Boden. 
* Boeke, Kees. Cosmic View: The Univer se in 40 Jumps. New York: John Day, 1957.

* Boeke, Kees. Cosmic View: The Univer se in 40 Jumps. New York: John Day, 1957. 
The ultimate book on levels of descripti on. Everyone should see this book at some 
point in their life. Suitable for children. 
** Bongard, M. Pattern Recognition. Rochelle  Park, N. J.: Hay den Book Co., Spartan 
Books, 1970. The author is concerned with problems of determining categories in an ill-defined space. In his book, he sets fort h a magnificent collection of 100 "Bongard 
problems" (as I call them)-puzzles for a pattern  recognizer (human or machine) to test 
its wits on. They are invaluably stimulating for anyone whoo is interested in the nature 
of intelligence.

of intelligence. 
Boolos, George S., and Richard Jeffrey. Computability and Logic. New York: Cambridge 
University Press. 1974. A sequel to Jeffrey's Formal Logic. It contains a wide number 
of results not easily obtainable elsewhere. Quite rigorous, but this does not impair its 
readability. 
Carroll, John B., Peter Davies, and Barry Rickman. The American Heritage Word 
Frequency Book. Boston: Houghton Mifflin, and New York: American Heritage 
Publishing Co., 1971. A table of words in order of frequency in modern written

Publishing Co., 1971. A table of words in order of frequency in modern written 
American English. Perusing it reseals fascinating things about out- thought processes. 
Cerf, Vinton. "Parry Encounters the Doctor ". Datamation, July 1973, pp. 62-64. The first 
meeting of artificial "minds"-what a shock! 
Chadwick, John. The Decipherment of Linear  B. New York: Cambridge University 
Press, 1958. Paperback. A book about  a classic decipherment-that of a script from the 
island of Crete-done by a single man: Michael Ventris.

island of Crete-done by a single man: Michael Ventris. 
Chaitin, Gregory J. "Randomness and Mathema tical Proof"'. Scientific American, May 
1975. An article about an algorithmic definiti on of randomness, and its intimate relation 
to simplicity. These two concepts are tied in with Godel's Theorem, which assumes a new meaning. An important article. 
Cohen, Paul C. Set Theory and the Continuum  Hypothesis. Menlo Park, Calif.: W. A. 
Benjamin, 1966. Paperback. A great contri bution to modern mathematics-the 
demonstration that various statements are undecidable within the usual formalisms for

 747set theory-is here explained to nonspecialists by its discoverer. The necessary 
prerequisites in mathematical logic are quickly, concisely, and quite clearly presented. 
Cooke, Deryck. The Language of Music. Ne w York: Oxford University Press, 1959. 
Paperback. The only book that I know which tries to draw an explicit connection between elements of music and elements of human emotion. A valuable start down 
what is sure to be a long hard road to understanding music and the human mind. 
* David, Hans Theodore.J. S. Bach's Musi cal Offering. New York: Dover Publications, 
1972. Paperback. Subtitled "History, Interpre tation, and Analysis". A wealth of 
information about this tour de force by Bach. Attractively written. 
** David, Hans Theodore, and Arthur Mende l. The Bach Reader. New York: W. W. 
Norton, 1966. Paperback. An excellent annotated collection of original source material on Bach's life, containing pictures, repr oductions of manuscript pages, many short

Norton, 1966. Paperback. An excellent annotated collection of original source material on Bach's life, containing pictures, repr oductions of manuscript pages, many short 
quotes from contemporaries, anecdotes, etc., etc. 
Davis, Martin. The Undecidable. Hewlett,  N. Y.: Raven Press, 1965. An anthology of 
some of the most important papers in me tamathematics from 1931 onwards (thus quite 
complementary to van Heijenoort's anthology) . Included are a translation of Godel's 
1931 paper, lecture notes from a course whic h Godel once gave on his results, and then 
papers by Church, Kleene, Rosser, Post, and Turing. 
Davis, Martin, and Reuben Hersh. "Hilbert' s Tenth Problem". Scientific American,

Davis, Martin, and Reuben Hersh. "Hilbert' s Tenth Problem". Scientific American, 
November 1973, p. 84. How a famous problem in  number theory was finally shown to 
be unsolvable, by a twenty-two-year old Russian. 
** DeLong, Howard. A Profile of Mathema tical Logic. Reading, Mass.: Addison-
Wesley, 1970. An extremely carefully written book about mathematical logic, with an 
exposition of Godel's Theorem and discussi ons of many philosophical questions. One 
of its strong features is its outstandi ng, fully annotated bibliography. A book which 
influenced me greatly.

influenced me greatly. 
Doblhofer, Ernst. Voices in Stone. Ne w York: Macmillan, Collier Books, 1961. Paper 
back. A good hook on the decipherment of ancient scripts.  
* Drevfus, Hubert. What Computers Can' t Do: A Critique of Artificial Reason. New 
York:Harper & Roc-, 1972. A collection of many arguments against Artificial Intelligence from someone outside of the fiel d. Interesting to try to refute. The Al 
community and Drevfus enjoy a relation of st rong mutual antagonism. It is important to

community and Drevfus enjoy a relation of st rong mutual antagonism. It is important to 
have people like Dreyfus around, even if you find them very irritating.  
Edwards, Harold M. "Fermat's Last Theo rem". Scientific American, October 1978, pp. 
104-122. A complete discussion of this hardes t of all mathematical nuts to crack, from
 its origins to the most modern results. Excellently illustrated.  
* Ernst, Bruno. The Magic Mirror of M. C. Escher. New York: Random House, 1976. 
Paperback. Escher as a human being, and th e origins of his drawings, are discussed

Paperback. Escher as a human being, and th e origins of his drawings, are discussed 
with devotion by a friend of mans' years. A "must" for any lover of Escher.  
** Escher, Maurits C., et al. The World of M. C. Escher. New York: Harry N. Abrarns,  1972. Paperback. The most extensive collecti on of reproductions of Escher's works. 
Escher comes about as close as one can to recursion in art, and captures the spirit of 
Godel's Theorem in some of his drawings amazingly well,  
Feigenbaum, Edward, and Julian Feldman, eds. Computers and Thought. New York:

Feigenbaum, Edward, and Julian Feldman, eds. Computers and Thought. New York: 
McGraw-Hill, 1963. Although it is a little old now, this book is still an important 
collection of ideas about Artificial Intellig ence. Included are articles on Gelernters

 748geometry program, Samuel's checkers pr ogram, and others on pattern recognition, 
language understanding, philosophy, and so on. 
Finsler, Paul. "Formal Proofs and Undeci dability", Reprinted in van Heijenoorcs 
anthology From Frege to Godel (see below). A forerunner of Godel's paper, in which the existence of undecidable mathematical statements is suggested, though not rigorously demonstrated. 
Fitzpatrick, P. J. "To Godel via Babel"- Mind 75 (1966): 332-350. An innovative 
exposition of Godel's proof which distinguishes between the relevant levels by using three different languages: English, French, and Latin' 
* Gablik, Suzi. Magritte. Boston, Mass.: Ne w York Graphic Society, 1976. Paperback. 
An excellent book on Magritte and his works by someone who really understands their 
setting in a wide sense; has a good selection of reproductions. 
* Gardner, Martin. Fads and Fallacies. Ne w York: Dover Publications, 1952. Paperback.

* Gardner, Martin. Fads and Fallacies. Ne w York: Dover Publications, 1952. Paperback. 
Still probably the best of all the anti-o ccult books. Although probably not intended as a 
book on the philosophy of science, this book c ontains many lessons therein. Over and 
over, one faces the question, "What is evidence?" Gardner demonstrates how unearthing "the truth" requires art as much as science. 
Gebstadter, Egbert B. Copper, Silver, Gold : an Indestructible Metallic Alloy. Perth:

Gebstadter, Egbert B. Copper, Silver, Gold : an Indestructible Metallic Alloy. Perth: 
Acidic Books, 1979. A formidable hodge-podge, turgid and confused-yet remarkably similar to the present work. Professor Gebsta dter's Shandean digressions include some 
excellent examples of indirect self-reference. Of particular interest is a reference in its 
well-annotated bibliography to an isomorphic, but imaginary, book. 
** Godel, Kurt. On Formally Undecida ble Propositions. New York: Basic Books, 1962. 
A translation of Godel's 1931 paper, together with some discussion.

A translation of Godel's 1931 paper, together with some discussion. 
. "Uber Formal Unentscheidbare Satze der Principia Mathematica and Verwandter 
Systeme, I." Monatshefte fur Mathema tik and Physik, 38 (1931), 173-198. Godel's 
1931 paper. 
Golf man, Ering. Frame Analysis. New York: Harper & Row, Colophon Books, 1974. 
Paperback. A long documentation of the definition of "systems" in human 
communication, and how in art and advertis ing and feporting and the theatre, the 
borderline between "the system" and "the world" is perceived and exploited and 
violated.

violated. 
Goldstein, Ira. and Seymour Papert. "Artific ial Intelligence, Language, and the Study of 
Knowledge". Cognitive Science 1 (January l9/ 7): 84-123. A survey article concerned 
with the past and future of Al. The authors see three periods so far: "Classic", "Romantic,., and "Modern". 
Guod, I. I. "Human and Machine Logic". Britis h Journal for the Philosophy of Science In 
(196/): 144. One of the most interesting atte mpts to refute Lucas, having to do with

violated. 
(196/): 144. One of the most interesting atte mpts to refute Lucas, having to do with 
whether the repeated application of the diagonal method is itself a mechanizable 
operation. 
. "Godel's Theorem is a Red Herring". Britis h Journal for the Philosophy of Science 19 
(1969): 357. In which Good maintains that Lucas' argument has nothing to do with Godel's Theorem, and that Lucas should in  fact have entitled his article "Minds, 
Machines, and Transfinite Counting". The Good-Lucas repartee is fascinating. 
Goodman, Nelson. Fact, Fiction, and Forecast.  3rd ed. Indianapolis: Bobbs-Merrill, 1973.

Goodman, Nelson. Fact, Fiction, and Forecast.  3rd ed. Indianapolis: Bobbs-Merrill, 1973. 
Paperback. A discussion of contrary-to-fact  conditionals and inductive logic, including

 749Goodman's famous problem-words "bleen " and "grue". Bears very much on the 
question of how humans perceive the world, and therefore interesting especially from the Al perspective. 
Press, 1976. A short hook about bees, apes, and other animals, and whether or not they 
are "conscious"-and particularh whether or not it is legitimate to use the word "consciousness" in scientific explanations of animal behavior. 
deGroot, Adriaan. Thought and Choice in Chess. The Hague: Mouton, 1965. A thorough 
study in cognitive psychology, reporting on experi ments that have a classical simplicity 
and elegance. 
Gunderson, Keith. Mentality and Machines. New York: Doubleday, Anchor Books, 1971. 
Paperback. A very anti-Al person tells why. Sometimes hilarious. 
Hanawalt, Philip C., and Robert H. Haynes,  eds. The Chemical Basis of Life. San 
Francisco: W. H. Freeman, 1973. Paperback. An  excellent collection of reprints from 
the Scientilic American. One of the best wa ys to get a feeling for what molecular

the Scientilic American. One of the best wa ys to get a feeling for what molecular 
biology is about. 
Hardy G. H. and E. M. Wright. An Introducti on to the Theory of Numbers, 4th ed. New 
York: Oxford University Press, 1960. The classic book on number theory. Chock-full of information about those mysterious entities. the whole numbers. 
Harmon, Leon. "The Recognition of Faces". Scientific American, November 1973, p. 70. 
Explorations concerning how we represen t faces in our memories, and how much 
information is needed in what form for us to be able to recognize a face. One of the

information is needed in what form for us to be able to recognize a face. One of the 
most fascinating of pattern recognition problems. 
van Heijenoort, Jean. From Frege to Gode l: A Source Book in Mathematical Logic. 
Cambridge, Mass.: Harvard University Pre ss, 1977. Paperback. A collection of epoch-
making articles on mathematical logic, all leading up to Godel's climactic revelation, which is the final paper in the book. 
Henri, Adrian. Total Art: Environments, Happenings, and Performances. New York: 
Praeger, 1974. Paperback. In which it is show n how meaning has degenerated so far m

Praeger, 1974. Paperback. In which it is show n how meaning has degenerated so far m 
modern art that the absence of meaning b ecomes profoundly meaningful (whatever that 
means). * Goodstein, R. L. Development of Mathem atical Logic. New York: Springer Verlag, 
1971. A concise survey of mathematical logic, including much material not easily found elsewhere. An enjoyable book, and useful as a reference. 
Gordon, Cyrus. Forgotten Scripts. New York: Basic Books, 1968. A short and nicely 
written account of the decipherment of anci ent hieroglyphics, cuneiform, and other 
scripts.

scripts. 
Griffin, Donald. The Question of Animal Awa reness. New York: Rockefeller University 
von Foerster, Heinz and James W. Beaucham p, eds. Music by Computers. New York:
 ** 
John Wiles', 1969. This book contains not only a set of articles about various types of 
computer-produced music, but also a set of four small phonograph records so you can 
actually hear (and judge) the pieces described. Among the pieces is Max Mathews' mixture of "Johnny Comes Marching Home" and "The British Grenadiers".

scripts. 
 ** 
actually hear (and judge) the pieces described. Among the pieces is Max Mathews' mixture of "Johnny Comes Marching Home" and "The British Grenadiers".  
Fraenkel, Abraham, Yehoshua Bar-Hillel, and Azriel Levy. Foundations of Set Theory,  
2nd ed. Atlantic Highlands, N. J.: Humanities Press, 1973. A fairly nontechnical

 750discussion of set theory, logic, limitativ e Theorems and undecidable statements. 
Included is a long treatment of intuitionism. * 
Frev, Peter W. Chess Skill in Man and Machine. New York: Springer Verlag, 1977. 
Anexcellent survey of contemporary ideas in  computer chess: why programs work, why 
they don't work, retrospects and prospects.  
Friedman, Daniel P. The Little Lisper. Palo Alto, Calif.: Science Research Associates, 
1974. Paperback. An easily digested introducti on to recursive thinking in LISP. You'll 
eat it up!  
• Hoare, C. A. R, and D. C. S. Allis on. "Incomputability". Computing Surveys 4, 
no. 3 September 1972). A smoothly presente d exposition of why the halting problem 
is unsolvable. Proves this fundament al theorem: ''Any language containing 
conditionals and recursive function defin itions which is powerful enough to program 
its own interpreter cannot be used to program its own 'terminates' function."

eat it up!  
its own interpreter cannot be used to program its own 'terminates' function." 
• Hofstadter, Douglas R. "Energy levels a nd wave functions of Bloch electrons in 
rational and irrational magnetic fields". P hysical Review B, 14, no. 6 (15 September 
1976). The author's Ph.D. work, presented as a paper. Details the origin of "Gplot", 
the recursive graph shown in Figure 34. 
Hook, Sidney. ed. Dimensions of Mind. New York: Macmillan, Collier Books, 1961. 
Paperback. A collection of articles on th e mind-body problem and the mind-computer 
problem. Some rather strong-minded entries here.

problem. Some rather strong-minded entries here. 
• Hornev, Karen. Self-Analysis. New York: W. W. Norton, 1942. Paperback. A 
fascinating description of how the levels of the self must tangle to grapple with 
problems of self-definition of any indivi dual in this complex world. Humane and 
insightful. 
Hubbard, John I. The Biological Basis of Mental Activity. Reading, Mass.: 
AddisonWesley, 1975. Paperback. Just one more  book about the brain, with one special 
virtue. however: it contains man long lists of  questions for the reader to ponder, and 
references to articles which treat those questions.

references to articles which treat those questions. 
• Jackson. Philip C. Introduction to Artificial Intelligence. New York, Petrocelli Charter. 
1975. A recent book, describing, with some exube rance, the ideas of Al. There are a 
huge number of vaguely suggested ideas floating around this book, and for that reason it is very stimulating just to page through it. Has a giant bibliography, which is another 
reason to recommend it. 
Jacobs, Robert L. Understanding Harmony. Ne w York: Oxford University Press, 1958.

Jacobs, Robert L. Understanding Harmony. Ne w York: Oxford University Press, 1958. 
Paperback. A straightforward book on harm ony, which can lead one to ask many 
questions about why it is that conventiona l Western harmony has such a grip on our 
brains. 
Jaki, Stanley L. Brain, Mind, and Computer s. South Bend. Ind.: Gateway Editions, 1969. 
Paperback. A polemic book whose every page exudes contempt for the computational 
paradigm for understanding the mind. Nonethel ess it is interesting to ponder the points 
he brings up. 
• Jauch, J. M. Are Quanta Real? Bloomingt on, Ind.: Indiana University Press, 1973. A

brains. 
• Jauch, J. M. Are Quanta Real? Bloomingt on, Ind.: Indiana University Press, 1973. A 
delightful little book of dialogues, using three characters borrowed from Galileo, put in a modern setting. Not only are questions of  quantum mechanics discussed, but also 
issues of pattern recognition, simplicity, br ain processes, and philosophy of science 
enter. Most enjoyable and provocative.

 751• Jeffrey, Richard. Formal Logic: Its Sc ope and Limits. New York: McGraw-Hill, 1967. 
An easy-to-read elementary textbook whose la st chapter is on Godel's and Church's 
Theorems. This book has quite a different approach from many logic texts, which 
makes it stand out. 
• Jensen, Hans. Sign, Symbol, and Script. Ne w York: G. P. Putnam's, 1969. A-or perhaps 
the-top-notch book on symbolic writing system s the world over, both of now and long 
ago. There is much beauty and mystery in this book-for instance, the undeciphered script of Easter Island. 
Kalmar, Laszlo. "An Argument Against the Plausibility of Church's Thesis". In A. 
Heyting, ed. Constructivity in Mathematics:  Proceedings of the Colloquium held at 
Amsterdam, 1957, North-Holland, 1959. An interesting article by perhaps the bestknown disbeliever in the Church-Turing Thesis. 
* Kim, Scott E. "The Impossible Skew Quadrilateral: A Four-Dimensional Optical

* Kim, Scott E. "The Impossible Skew Quadrilateral: A Four-Dimensional Optical 
Illusion". In David Brisson, ed. Proceed ings of the 1978 A.A.A.S. Sym osium on 
Hypergraphics: Visualizing Complex Relationshi ps in Art and Science. Boulder, Colo.: 
Westview Press, 1978. What seems at first an inconceivably hard idea-an optical illusion for four-dimensional "people"-is gradually made crystal clear, in an amazing virtuoso presentation utilizing a long series of  excellently executed diagrams. The form 
of this article is just as intriguing and unusual as its content: it is tripartite on many

of this article is just as intriguing and unusual as its content: it is tripartite on many 
levels simultaneously. This article a nd my book developed in parallel and each 
stimulated the other. 
Kleene, Stephen C. Introduction to Mathematical Logic. New York: John Wiley, 1967. A 
thorough, thoughtful text by an im portant figure in the subject. Very worthwhile. Each 
time I reread a passage, I find something new in it which had escaped me before. 
. Introduction to Metamathematics. Princeton:  D. Van Nostrand (1952). Classic work on 
mathematical logic; his textbook (above) is essentially an abridged version. Rigorous 
and complete, but oldish.

and complete, but oldish. 
Kneebone G. J. Mathematical Logic and the Foundations of Mathematics. New York: 
Van Nostrand Reinhold, 1963. A solid book with  much philosophical discussion of 
such topics as intuitionism, and the "reality" of the natural numbers, etc. 
Koestler, Arthur. The Act of Creation, Ne w York: Dell, 1966. Paperback. A wide-ranging 
and generally stimulating theory about how ideas are "bisociated" to yield novelty. Best to open it at random and read, rather than begin at the beginning.

and generally stimulating theory about how ideas are "bisociated" to yield novelty. Best to open it at random and read, rather than begin at the beginning. 
Koestler, Arthur and J. R. Smythies, ed s. Beyond Reductionism. Boston: Beacon Press, 
1969. Paperback. Proceedings of Yr conferen ce whose participants all were of the 
opinion that biological systems cannot be expl ained reductionistically, and that there is 
something "emergent" about life. I am in trigued by books which seem wrong to me, yet 
in a hard-to-pin-down way. 
** Kutbose, Gyomay. Zen Koans. Chicago:  Regnerv, 1973. Paperback. One of the best

** Kutbose, Gyomay. Zen Koans. Chicago:  Regnerv, 1973. Paperback. One of the best 
collections of koans available. Attractivel y presented- An essential book for any Zen 
library. Kuffler, Stephen W. and John G. Ni cholls. From Neuron to Brain. Sunderland, 
deals Mss.: Smauer Associates, 1976. Paperback. A book which, despite its title, deals mostly with microscopic processes in the brain, and quite little with the way people's thoughts come out of the tangled mess. The work of Hubel and Wiesel on visual systems is covered particularly well.

 752Lacey Huh, and Geoffrey Joseph. "What the Godel Formula Says". Mind 77 (1968)- 7~. 
A useful discussion of the meaning of the G odel formula, based on a strict separation of 
three levels: uninterpreted formal system, interpreted formal system, a metamathematics. Worth studying. Latos Im re. Proofs and Refutations. New York: 
Cambridge University Press, 1976. Paperb ack. A most entertaining book in dialogue 
form, discussing how concepts are formed in mathematics. Valuable not only to 
mathematicians, but also to people interested in thought processes. 
** Lehninger, Albert. Biochemistry. New Yo rk: Worth Publishers, 1976. A wonderfully 
readable text, considering its technical le vel. In this book one can find many ways in 
which proteins and genes are tangled together. Well organized, and exciting. 
** Lucas, J. R. "Minds, Machines, and Gode l". Philosophy 36 (1961): 112. This article is 
reprinted in Anderson's Minds and Machines , and in Sayre and Crosson's The Modeling

reprinted in Anderson's Minds and Machines , and in Sayre and Crosson's The Modeling 
of Mind. A highly controversial and provocativ e article, it claims to show that the 
human brain cannot, in principle, be mode led by a computer program. The argument is 
based entirely on Godel's Incompleteness Theorem, and is a fascinating one. The prose is (to my mind) 
incredibly infuriating-vet for that very  reason, it makes humorous reading. "Satan 
Stultified: A Rejoinder to Paul Benacerraf". Monist 52 (1968): 145. 
Anti-Benacerraf argument, written in hilariously learned style: at one point Lucas refers

Anti-Benacerraf argument, written in hilariously learned style: at one point Lucas refers 
to Benacerraf as "self-stultifyingly eris tic" (whatever that means). The Lucas-
Benacerraf battle, like the Lucas-Good battle, offers much food for thought. 
  . "Human and Machine Logic: A Rejoinder". British Journal for the Philosophy of 
Science 19 (1967): 155. An attempted refuta tion of Good's attempted refutation of 
Lucas' original article. 
** MacGillavry, Caroline H. Symmetry Aspects of the Periodic Drawings of M. C. 
Escher. Utrecht: A. Oosthoek's Uitgeverm aatschappij, 1965. A collection of tilings of

Escher. Utrecht: A. Oosthoek's Uitgeverm aatschappij, 1965. A collection of tilings of 
the plane by Escher, with scientific comme ntary by a crystallographer. The source for 
some of my illustrations-e.g., the Ant Fugue  and the Crab Canon. Reissued in 1976 in 
New York by Harry N. Abrams under the title Fantasy and Symmetry. 
MacKay, Donald M. Information, Mechanis m and Meaning. Cambridge, Mass.: M.I.T. 
Press, 1970. Paperback. A book about different m easures of information, applicable in 
different situations; theoretical issues re lated to human perception and understanding; 
and the way in which conscious activity can arise from a mechanistic underpinning.

and the way in which conscious activity can arise from a mechanistic underpinning. 
• Mandelbrot, Benoit. Fractals: Form, Chan ce, and Dimension. San Francisco: W. H. 
Freeman, 1977. A rarity: a picture book of sophis ticated contemporary research ideas in 
mathematics. Here, it concerns recursiv ely defined curves and shapes, whose 
dimensionality is not a whole number. Amazi ngly, Mandelbrot shows their relevance to 
practically every branch of science. 
• McCarthy, John. "Ascribing Mental Qualities to Machines". To appear in Martin

• McCarthy, John. "Ascribing Mental Qualities to Machines". To appear in Martin 
Ringle, ed. Philosophical Perspectives in Artificial Intelligence. New York: Humanities 
Press, 1979. A penetrating article about th e circumstances under which it would make 
sense to say that a machine had beliefs, desire s, intentions, consciousness, or free will. 
It is interesting to compare this article with the book by Griffin. 
Meschkowski, Herbert. Non-Euclidean Ge ometry. New York: Academic Press, 1964. 
Paperback. A short book with good historical commentary.

 753Meyer, can. ''Essai d'application de certains modeles cybernetiq ues it la coordina_ Lion 
c1Jei les insectes sociaux". Insertes So ciaux XI11, no. 2 1966 : o which ch aws some 
parallels between the neural organization in the( brain, and the organiz ae lion of an ant 
colon%. 
Meyer, Leonard B. Emotion and Meaning in Music. Chicago: University of' Chicago 
Press, 1956. Paperback. A book which attempts to  use ideas of Gestalt svcholo theory 
of perception to explain why musical structur e is as it is. One of the more unusual the 
books on music and mind. 
. Music, The Arts, and Ideas. Chicago: Univ ersity of Chicago Press, 1967. Paperback. A 
thoughtful analysis of mental processes involved in listening to music, and of 
hierarchical structures in music. The author  compares modern trends in music with Zen 
Buddhism. 
Miller, G. A. and P. N. joint son-Lair d. Language and Perception- Cambridge, Mass.: 
Harvard University Press, Belknap Press, 1976. A fascinating compendium of linguistic

Buddhism. 
Harvard University Press, Belknap Press, 1976. A fascinating compendium of linguistic 
facts and theories, hearing on Whorl 's hypothesis that language is the same as 
worldviesv. A typical example is the discu ssion of the weird "mother-in-law" language 
of the Dyirbal people of Northern Queensl and: a separate language used only for 
speaking to one's mother-in-law. 
Minsky, Marvin L. "Matter, Mind, and Models ". In Marvin L. Minskv, ed. Son antic 
Information Processing. Cambridge, Mass.: M.I.T. Press, 1968. Though merely a few

Information Processing. Cambridge, Mass.: M.I.T. Press, 1968. Though merely a few 
pages long, this article implies a whole philosophy of consciousness and machine intelligence. It is a memorable piece of writi ng by one of the deepest thinkers in the 
field. 
Minsky, Marvin L., and Seymour Papert_ .92-t cial Intelligence Progress Report. 
Cambridge, Mass.: M.I.T. Artificial In telligence Laboratory, Al Memo 252, 1972. A 
survey of all the work in Artificial Inte lligence done at M.I.T. up to 1972, relating it to 
psychology and epistemology. Could serve excellently as an introduction to Al.

field. 
psychology and epistemology. Could serve excellently as an introduction to Al. 
Monod, Jacques. Chance and Necessity. Ne w York: Random House, Vintage Books, 
1971. Paperback. An extremel fertile mind writing in an idiosyncratic way about 
fascinating questions, such as how life is constructed out of non-life: how evolution, 
seeming_ to violate the second law of ther modynamics, is actually dependent on it. The 
book excited me deeply. 
* Morrison, Philip and Emil, eds. Charles Ba bbage and his Calculating Engines. Nesv 
York: Dover Publications, 1961. Paperback. A va luable source of information about the 
life of Babbage. A large fraction of Babbage 's autobiography is reprinted here, along

life of Babbage. A large fraction of Babbage 's autobiography is reprinted here, along 
with several articles about Babbage's machines and his "Mechanical Notation". 
Mvhill, John. "Some Philosophical Implications  of Mathematical Logic". Review of 
Metaphysics 6 (1952): 165. An unusual discussi on of ways in which Godel's Theorem 
and Church's Theorem are connected to psychology and epistemology. Ends up in a discussion of beauty and creativity. 
Nagel, Ernest. The Structure of Science.  New York: Harcourt, Brace, and World, 1961. A 
classic in the philosophy of science, featur ing clear discussions of reductionism vs.

classic in the philosophy of science, featur ing clear discussions of reductionism vs. 
holism, teleological vs. nonteleological explanations, etc. 
Nagel, Ernest and James R. Newman. Gode l's Proof. New York: New York University 
Press, 1958. Paperback. An enjoyable and exciting presentation, which was, in many ways, the inspiration for my own book.

 754* Nievergelt, Jurg, J. C. Farrar, and E. M. Reingold. Computer Approaches to Mathe,nat 
teal Problems. Englesyood Clif-f-s, N. J.: Prentice-Hall, 1974. An unusual collection of 
different types of problems which can be and have been attacked on computers-for 
instance, the "3n + I problem" (mentioned in  my Rria with Diverse Variations) and 
other problems of number theory. 
Pattee, Howard H., ed. Hierarchy Theory. New York: George Braziller, 1973. Paperback. 
Subtitled "The Challenge of Complex Syst ems". Contains a good article by Herbert 
Simon covering some of the same ideas as does my Chapter on "Levels of Description". 
Peter, R6zsa. Recursive Functions. Ne w York: Academic Press, 1967. A thorough 
discussion of primitive recursive functions, general recursive functions, partial recursive functions, the diagonal method, and many other fairly technical topics. 
Quine, Willard Van Ornman. The Ways of Paradox, and Other Essays. New York:

Quine, Willard Van Ornman. The Ways of Paradox, and Other Essays. New York: 
Random House, 1966. A collection of Quine' s thoughts on many topics. The first essay 
deals with carious sorts of paradoxes, and their resolutions. In it, he introduces the 
operation I call "quining" in my book. 
lishiRanganathan, S. R. Ramanujan, The Man and the Mathematician. London: Asia 
Pubng House, 1967. An occult-oriented biography of the Indian genius by an admirer. 
An odd but charming book. 
Reichardt. Jasia. Cybernetics, Arts, a nd Ideas. Boston: New York Graphic Society,

Reichardt. Jasia. Cybernetics, Arts, a nd Ideas. Boston: New York Graphic Society, 
197 1. A weird collection of ideas about comput ers and art, music, literature. Some of it 
is definitely off the deep end-hut some of it is not. Examples of the latter are the articles 
"A Chance for Art" by J. R. Pierce, and "Computerized Haiku" by Margaret 
Masterman.Renyi, Alfred. Dialogues on Math ematics. San Francisco: Holden-Day, 
1967. Paper hack. Three simple but stimula ting dialogues involving classic characters 
in history, trying to get at the nature of mathematics. For the general public.

in history, trying to get at the nature of mathematics. For the general public. 
** Rips Paul. Zen Flesh, Zen Bones. New York: Doubleday, Anchor Books. Paperback. 
This book imparts very well the flavor of Zen-its antirational, antilanguage, antireduc 
iionistic, basically holistic orientation.Rogers , Hartley. Theory of Recursive Functions 
and Effective Computability. New York:McGraw-Hill, 1967. A highly technical treatise, but a good one to learn from. Cont ainsdiscussions of many intriguing problems 
in set theory and recursive function theory.

in set theory and recursive function theory. 
Rokeach, Milton. The Three Christs of' Ypsilanti. New York: Vintage Books, 
1964.paperback. A study of schizophrenia and the strange breeds of "consistency" 
which arise in the afflicted. A fascinating conflict between three men in a mental institution, all of whom imagined they we re God, and how they dealt with being 
brought face to face for many months. 
Rose, Steven. The Conscious Brain, updated ed. New York: Vintage Books, 1976. Paperback. An excellent book-probably the be st introduction to the study of the brain.

Rose, Steven. The Conscious Brain, updated ed. New York: Vintage Books, 1976. Paperback. An excellent book-probably the be st introduction to the study of the brain. 
Contains full discussions of the physical na ture of the brain, as well as philosophical 
discussions on the nature_ of mind, reductionism vs. holism, free will vs. determinism, etc. from a broad, intelligent, and humanistic  viewpoint. Only his ideas on Al are way 
off. 
Rosenblueth, Arturo. Mind and Brain: A Philosophy of Science. Cambridge, 
Mass.:M.I.T. Press, 1970. Paperback. A we ll written book by a brain researcher who 
deals with most of the deep problems concerning mind and brain.

 755* Sagan, Carl, ed. Communication with Extrat errestrial Intelligence. Cambridge, Mass.: 
M.I.T. Press, 1973. Paperback. Transcripts of a truly far-out conference, where a stellar 
group of scientists and others battle it out on this speculative issue. 
Salmon, Wesley, ed. Zeno's Paradoxes. Ne w York: Bobbs-Merrill, 1970. Paperback. A 
collection of articles on Zeno's ancient paradoxe s, scrutinized under the light of modern 
set theory, quantum mechanics, a nd so on. Curious and thought-provoking, 
occasionally humorous. 
Sanger. F., et al. "Nucleotide sequen ce of bacteriophage 16X174 DNA", Nature 265 
(Feb. 24, 1977). An exciting presentation of the first laying-bare ever of the full 
hereditary material of any organism. The su rprise is the double-entendre: two proteins 
coded for in an overlapping way: almost too much to believe. 
Sayre, Kenneth M., and Frederick J. Cro sson. The Modeling of Mind: Computers and

Sayre, Kenneth M., and Frederick J. Cro sson. The Modeling of Mind: Computers and 
Intelligence. New York: Simon and Schuste r, Clarion Books, 1963. A collection of 
philosophical comments on the idea of Artif icial Intelligence by people from a wide 
range of disciplines. Contributors include  Anatol Rapoport, Ludwig Wittgenstein, 
Donald Mackay, Michael Scriven, Gilbert Ryle, and others. 
* Schank, Roger, and Kenneth Colby. Comput er Models of Thought and Language. San 
Francisco: W. H. Freeman, 1973. A collection of articles on various approaches to the

Francisco: W. H. Freeman, 1973. A collection of articles on various approaches to the 
simulation of mental processes such as language-understanding, belief-systems, 
translation, and so forth. An important Al book, and many of the articles are not hard to 
read, even for the layman. 
Schrodinger, Erwin. What is Life? & Mind a nd Matter. New York: Cambridge University 
Press, 1967. Paperback. A famous book by a famous physicist (one of the main founders of quantum mechanics). Explores th e physical basis of life and brain; then 
goes on to discuss consciousness in quite meta physical terms. The first half, What is

goes on to discuss consciousness in quite meta physical terms. The first half, What is 
Life?, had considerable influence in the 1940' s on the search for the carrier of genetic 
information. 
Shepard, Roger N. "Circularity in judgments of Relative Pitch". Journal of the Acoustical 
Society of America 36, no. 12 (Decembe r 1964), pp. 2346-2353. The source of the 
amazing auditory illusion of "Shepard tones". 
Simon, Herbert A. The Sciences of the Artificial. Cambridge, Mass.: M.I.T. Press, 1969. 
Paperback. An interesting book on understanding complex systems. The last chapter,

Paperback. An interesting book on understanding complex systems. The last chapter, 
entitled "The Architecture of Complexity", discusses problems of reductionism versus 
holism somewhat. 
Smart, J. J. C. "Godel's Theorem, Church 's Theorem, and Mechanism". Synthese 13 
(1961): 105. A well written article predating Lucas' 1961 article, but essentially arguing 
against it. One might conclude that you have  to be Good and Smart, to argue against 
Lucas... 
** Smullvan, Raymond. Theory of Formal Systems. Princeton, N. J.: Princeton

Lucas... 
** Smullvan, Raymond. Theory of Formal Systems. Princeton, N. J.: Princeton 
University Press, 1961. Paperback. An advanced  treatise, but one which begins with a 
beautiful discussion of formal systems, and proves a simple version of Godel's Theorem 
in an elegant way. Worthwhile for Chapter 1 alone. What Is the Name of This Book? 
Englewood Cliffs, N. J.: Prentice-Hall,  1978. A book of puzzles and fantasies on 
paradoxes, self-reference, and Godel's Theo rem. Sounds like it will appeal to many of 
the same readers as my book. It appeared af ter mine was all written (with the exception 
of a certain entry in my bibliography).

 756Sommerhoff, Gerd. The Logic of the Li ving Brain. New York: John Wiley, 1974. A book 
which attempts to use knowledge of small-scale structures in the brain, in creating a theory of how the brain as a whole works. 
Sperrv, Roger. "Mind, Brain, and Humanist Valu es". In John R. Platt, ed. .Vew Views on 
the Nature of Man. Chicago: Univers ity of Chicago Press, 1965, A pioneering 
neurophvsiologist here explains most vivi dly how he reconciles brain activity and 
consciousness. 
* Steiner, George. After Babel: Aspects of Language and Translation. New York: Oxford 
University Press, 1975. Paperback. A book by a scholar in linguistics about the deep 
problems of translation and understanding of language by humans. Although Al is 
hardly discussed, the tone is that to program a computer to understand a novel or a poem is out of the question. A well writte n, thought-provoking-sometimes infuriating-
book. 
Stenesh, J. Dictionary of Biochemistry. New York: John Wiley, Wiley-Interscience,

book. 
Stenesh, J. Dictionary of Biochemistry. New York: John Wiley, Wiley-Interscience, 
1975. For me, a useful companion to technical books on molecular biology. 
** Stent, Gunther. "Explicit and Implicit Semantic Content of the Genetic Information". 
In The Centrality of Science and Absolute  Values, Vol. 1. Proceedings of the 4th 
International Conference on the Unity of  the Sciences, New York, 1975. Amazingly 
enough, this article is in the proceedings of  a conference organized by the nowinfamous 
Rev. Sun Mvung Moon. Despite this, the article  is excellent. It is about whether a

Rev. Sun Mvung Moon. Despite this, the article  is excellent. It is about whether a 
genotype can be said, in any operational sense,  to contain "all" the information about its 
phenotype. In other words, it is about the location of meaning in the genotype. 
. Molecular Genetics: A Historical Narrative.  San Francisco: W. H. Freeman, 1971. Stent 
has a broad, humanistic viewpoint, and conveys id eas in their historical perspective. An 
unusual text on molecular biology. 
Suppes, Patrick. Introduction to Logic.  New York: Van Nostrand Reinhold, 1957. A

Suppes, Patrick. Introduction to Logic.  New York: Van Nostrand Reinhold, 1957. A 
standard text, with clear presentations of both the Propositional Calculus and the 
Predicate Calculus. My Propositional Calculus stems mainly from here. 
Sussman, Gerald Jay. A Computer Model of Skill Acquisition. New York: American 
Elsevier, 1975. Paperback. A theory of programs which understand the task of 
programming a computer. The questions of how to  break the task into parts, and of how 
the different parts of such a program should interact, are discussed in detail.

the different parts of such a program should interact, are discussed in detail. 
** Tanenbaum, Andrew S. Structured Comput er Organization. Englewood Cliffs, N. J.: 
Prentice-Hall, 1976. Excellent: a straightforw ard, extremely well written account of the 
many levels which are present in mode rn computer systems. It covers 
microprogramming languages, machine languages, assembly languages, operating systems, and many other topics. Has a good, partially annotated, bibliography. 
Tarski, Alfred. Logic, Semantics, Metamathematics. Papers from 1923 to 1938.

Tarski, Alfred. Logic, Semantics, Metamathematics. Papers from 1923 to 1938. 
Translated by J. H. Lot, New York: Oxford University Press, 1956. Sets forth Tarski's 
ideas about truth, and the relationship between language and the world it represents. These ideas are still having repercussions in the problem of knowledge representation in Artificial Intelligence. 
Taube, Mortimer. Computers and Comm on Sense. New York: McGraw-Hill, 1961. 
Paperback. Perhaps the first tirade against the modern concept of Artificial Intelligence. Annoying.

 757Tietze, Heinrich. Famous Problems of Math ematics. Baltimore: Graylock Press, 1965. A 
book on famous problems, written in a ve ry personal and erudite style. Good 
illustrations and historical material. 
Trakhtenbrot, V. Algorithms and Computi ng Machines. Heath. Paperback. A discussion 
of theoretical issues involving computers, particularly unsohable problems such as the 
halting problem, and the word-equisalence problem. Short. which is nice. 
Turing, Sara. Alan M. Turing. Cambridge , U. K.: WW'. Hefter & Sons, 1959. A 
biography of the great computer pion4r. A mother's work of love. 
* Ulam, Stanislaw. Adventures of a Math ematician. New York: Charles Scribner's, 1976. 
An autobiography written by a sixty-five-year old man who writes as if he were still 
twenty and drunk in love with mathematic s. Chock-full of gossip about who thought 
who was the best, and who envied whom, etc. Not only fun, but serious.

who was the best, and who envied whom, etc. Not only fun, but serious. 
,A'aitson, J. D. The Molecular Biology of th e Gene, 3rd edition. Menlo Park, Calif.: W. 
A. Benjamin, 1976. A good book but not nearly as well organized as Lehninger's, in my opinion. Still almost every page has something interesting on it. 
Webb, Judson. "Metamathematics and the Ph ilosophy of Mind". Philosophy of Science 
35 (1968): 156. A detailed and rigorous argument against Lucas, which contains this 
conclusion: "My overall position in the presen t paper may be stated by saying that the

conclusion: "My overall position in the presen t paper may be stated by saying that the 
mind-machine-GOdel problem cannot be coherently treated until the constructivity problem in the foundations of mathematics is clarified." 
Weiss, Paul. "One Plus One Does Not Equal Two". In G. C. Quarton, T. Melnechuk, and 
F. O. Schmitt, eds. The Neurosciences :.4 Study Program. New York: Rockefeller 
University Press, 1967. An article trying to reconcile holism and reductionism, but a good bit too holism-oriented for my taste. 
* Weizenbaum, Joseph. Computer Power a nd Human Reason. San Francisco: W. H.

* Weizenbaum, Joseph. Computer Power a nd Human Reason. San Francisco: W. H. 
Freeman, Freeman, 19i6. Paperback. A provo cative book by an early At worker who 
has come to the conclusion that much work in  computer science, particularly in Al, is 
dangerous. Although I can agree with him on so me of his criticisms, I think he goes too 
far. His sanctimonious reference to Al peopl e as "artificial intelligentsia" is funny the 
first time, but becomes tiring after the dozen th time. Anyone interested in computers 
should read it.

should read it. 
Wheeler, William Morton. "The Ant-Colony as an Organism".Journal of Morphology 22, 2 (1911): 307-325. One of the foremost authorities of his time on insects gives a famous statement about why an ant colony dese rves the label "organism" as much as its 
parts do.  
Whitely, C. H. "Minds, Machines, and G odel: A Reply to Mr Lucas". Philosophy 37 
(1962): 61. A simple but potent reply to Lucas' argument. Wilder, Raymond. An Introduction to the Foundations of :Mat hematics. New York: John Wiley, 1952. A good

parts do.  
(1962): 61. A simple but potent reply to Lucas' argument. Wilder, Raymond. An Introduction to the Foundations of :Mat hematics. New York: John Wiley, 1952. A good 
general overview, putting into perspective the important ideas of the past century. 
* Wilson, Edward O. The Insect Societies. Cambridge, Mass.: Harvard University Press, 
elknap Press, 1971. Paperback. The authoritativ e book on collective behavior of insects. 
Although it is detailed, it is still readable, and discusses many fascinating ideas. It has 
excellent illustrations, and a giant (although regrettably not annotated) bibliography.

excellent illustrations, and a giant (although regrettably not annotated) bibliography. 
Winograd, Terry. Five Lectures on Artificial Intelligence. AI Memo 246. Stanford, Calif.: 
Stanford University Artificial Intelligen ce Laboratory, 1974. Paperback. A description 
of fundamental problems in At and new ideas for attacking them, by one of the

 758important contemporary workers in the field.. Language as a Cognitive Process. 
Reading, Mass.: Addison-Wesley (forthcomi ng). From what I have seen of the 
manuscript, this will be a most exciti ng book, dealing with language in its full 
complexity as no other book ever has. . Understanding Natural Language. New York: Academic Press, 1972. A detailed discussion of one particular program which is 
remarkably "smart", in a limited worl d. The book shows how language cannot be 
separated from a general understanding of the world, and suggests directions to go in, in 
writing programs which can use language in  the way that people do. An important 
contribution; many ideas can be stimulated by a reading of this book. 
. "On some contested suppositions of generativ e linguistics about the scientific study of 
language", Cognition 4:6. A droll rebuttal to a head-on attack on Artificial Intelligence 
by some doctrinaire linguists.

by some doctrinaire linguists. 
* Winston, Patrick. Artificial Intelligence.  Reading, Mass.: Addison-Wesley, 1977. A 
strong, general presentation of many facets of At by a dedicated and influential young 
proponent. The first half is independent of programs; the second half is LISP-dependent 
and includes a good brief exposition of th e language LISP. The book contains many 
pointers to present-day At literature. e d. The Psychology of Computer Vision. New

pointers to present-day At literature. e d. The Psychology of Computer Vision. New 
York: McGraw-Hill. 1975. Silly title, but fine book. It contains articles on how to program computers to do visual recognition of  objects. scenes. and so forth. The articles 
deal with all levels of the problem, from th e detection of line segments to the general 
organization of knowledge. In particular, th ere is an article by Winston himself on a 
program he wrote which develops abstract concepts from concrete examples, and an 
article by Minsky on the nascent notion of "frames".

article by Minsky on the nascent notion of "frames". 
* Wooldridge, Dean. Mechanical Man-The Phys ical Basis of Intelligent Life. New York: 
NJ McGraw-Hill, 1968. Paperback. A thorough-goi ng discussion of the relationship of 
mental phenomena to brain phenomena, written in clear language. Explores difficult 
philosophical concepts in novel was, shedding light on them by means of concrete 
examples.

Credits 
 759Credits 
 
Figures: Fig. 1, Johann Sebastian Bach by Elias Gottlieb-Hart ssmann (1748), collection of William H. Scheide, Princeton, New Jersey; Fig. 2, Flute Concert in Saussouci, by Adolf von Menzel, Nationalgalerie, West Ber lin; Figs. 3, 4, 152, "The Royal Theme" and 
the last page of the "Six-part Ricercar." fr om the original edition of Musical Offering by 
Johann Sebastian Bath, are reproduced courtesy  of the Library of Congress; Figures of 
lithographs and woodcuts of M. C. Escher are reproduced by permission of the Escher Foundation, Haags Gemeentemuseum, The Hague, copyright © the Escher Foundation, 
1979, reproduction rights arranged courtesy of the Vorpal Galleries, New York, Chicago,

1979, reproduction rights arranged courtesy of the Vorpal Galleries, New York, Chicago, 
San Francisco, and Laguna Beach; Fig. 9, photograph of Kurt Godel by Orren J. Turner from Foundations of Mathematics. Symposium Papers Commemorating the Sixtieth Birthday of Kurt Godel, edited by Jack J. Bulloff, Thomas C. Holyoke, and S. W. Hahn, New York, Springer-Verlag, 1969; Figs. 17, 96, "Figure-Figure" and "A sec tion of mRNA passing through a ribosome," 
drawings by Scott E. Kim; Figs. 19, 44, 133, 148, musical selections from the Musical

drawings by Scott E. Kim; Figs. 19, 44, 133, 148, musical selections from the Musical 
Offering by J. S. Bach, music printed by Donald Byrd's program "SMUT"; Fig. 25, "Cretan labyrinth" from W. H. Matthews, Mazes and Labyrinths: Their History and Development, New York, Dover Publicati ons, Inc., 1970; Fig. 39, photograph of Rosetta 
Stone, courtesy of the British Museum; Fig. 40, A collage of scripts. Samples of cuneiform, Easter Island, Mongolian and Runi c scripts from Hans Jensen, Sign, Symbol 
and Script, East Germany VEB Deutscher Verlag Der Wissenschaften; samples of

and Script, East Germany VEB Deutscher Verlag Der Wissenschaften; samples of 
Bengali and Buginese script from Kenneth Katzner, The Languages of the World, New 
York, Funk & Wagnalls, 1975; samples of Tamil and Thai from I. A. Richards and Christine Gibson, English Through Pictures, New York, Washington Square Press; Fig. 
59, "Intelligence built up layer by layer" (Fig. 9.8) adapted from Patrick Henry Winston. Artificial Intelligence, Reading, Mass., Addison-Wesley Publishing Company, reprinted by permission; Figs. 63, 69, photographs of an ant bridge by Carl W. Rettenmeyer and

59, "Intelligence built up layer by layer" (Fig. 9.8) adapted from Patrick Henry Winston. Artificial Intelligence, Reading, Mass., Addison-Wesley Publishing Company, reprinted by permission; Figs. 63, 69, photographs of an ant bridge by Carl W. Rettenmeyer and 
construction of an arch by termite workers by Turid Hdlldobler, from E. O. Wilson, The Insect Societies, Cambridge, Mass., Harvard University Press, 1979; Fig. 65, schematic drawing of a neuron adapted from The Machinery of the Brain, by Dean Wooldridge, copyright © 1963, McGraw-Hill, Inc. used with permission of McGraw Hill Book

construction of an arch by termite workers by Turid Hdlldobler, from E. O. Wilson, The Insect Societies, Cambridge, Mass., Harvard University Press, 1979; Fig. 65, schematic drawing of a neuron adapted from The Machinery of the Brain, by Dean Wooldridge, copyright © 1963, McGraw-Hill, Inc. used with permission of McGraw Hill Book 
Company, and from Fig. 11-6, page 26, Speech and Brain-Mechanisms, by Wilber Penfield and Lamar Roberts, copyright © by Princeton University Press, reprinted by permission of Princeton University Press; Fi g. 66, "the human brain, seen from the left 
side," from Steven Rose, The Conscious  Brain, copyright © 1973 by Steven Rose, 
reprinted by permission of Alfred A. Knopf , Inc., New York, and John Wolfers, London; 
Fig. 68, "overlapping neural pathways," from  John C. Eccles, Facing Reality, New York,

Fig. 68, "overlapping neural pathways," from  John C. Eccles, Facing Reality, New York, 
SpringerVerlag, 1970; Figs. 77, 78, 80, 82, 117, 137, 138, 141, The Shadows, State of Grace, The Fair Captive, The Air and the Song, Mental Arithmetic, Common Sense, The Two Mysteries, and The Human Condition I by Rene Magritte, copyr ight © by ADAGP, Paris, 1979; Figs. 79, 95, 
"Tobacco Mosaic Virus" and "Secondary and Tertiary Structure of Myoglobin" from Albert Lehninger, Biochemistry, New Yor k, Worth Publishers, 1975; Figs. 91, 92, "The

Credits 
 760four constituent bases of DNA" and "The la dder-like structure of DNA" from Arthur 
Kornberg, "The Synthesis of DNA," Scientif ic American, copyright © October 1968, all 
rights reserved; Fig. 93, "Molecular model of the DNA double helix," reprinted by permission, from V. M. Ingram, Biosynt hesis of Macromolecules, Menlo Park, 
California, The Benjamin/Cummings Publishing Company, 1972; Fig. 97, "Polyribosome" from The Proteins, edited by R. E. Dickerson and H. Neurath, page 64, 
New York, Academic Press; Fig. 98, "A two-tiered molecular canon" from O. L. Miller, Jr., "Visualization of Genes in Action," in Scientific American, copyright © March 1973, 
all rights reserved; Figs. 101, 102, 103, "The T4 bacterial virus," "Infection of a Bacterium by a virus," and "The morphogenetic pathway of the T4 virus," from William B. Wood and R. S. Edgar, "Building a Bacteria l Virus" in Scientific American, copyright

all rights reserved; Figs. 101, 102, 103, "The T4 bacterial virus," "Infection of a Bacterium by a virus," and "The morphogenetic pathway of the T4 virus," from William B. Wood and R. S. Edgar, "Building a Bacteria l Virus" in Scientific American, copyright 
© July 1987, all rights reserved; Fig. 105, photograph of Srinivasa Ramanujan from S. R. Ranganathan, Ramanujan, the Man and the Mathematician, New York, Asia Publishing House, 1967; Figs. 110, 111, 112, from Terry Winograd, Understanding Natural Language,New York, Academic Press, 1972; Fig. 113, Photograph of Alan Turing by Mssr s. C. H. O. Trevelyan from Sara 
Turing, Alan M. Turing, Cambridge, Engla nd, W. H. Heffer and Sons, Ltd., 1959; Fig.

Turing, Alan M. Turing, Cambridge, Engla nd, W. H. Heffer and Sons, Ltd., 1959; Fig. 
116, "a meaningful story in Arabic" from Abdelkebir Khatibi and Mohammed Sijelmassi, The Splendor of Islamic Callizrabhl, New York, London, Thames & Hudson, copyright © by Qarawiyne Library in Fez. Fig 118, Proce dural representation of “red cube which 
supports a pyramid,” adapted from Computer Models of Thought and Language, edited by Roger C. Schank and Kenneth Mark Colby. W. H. Freeman and Company, copyright 1973; Figs. 119, 122, 124, 130, Bongard problems from M. Bongard, Pattern Recognition, Rochelle Park, New Jersey, Hayden Book Company, Spartan Books, 1970

supports a pyramid,” adapted from Computer Models of Thought and Language, edited by Roger C. Schank and Kenneth Mark Colby. W. H. Freeman and Company, copyright 1973; Figs. 119, 122, 124, 130, Bongard problems from M. Bongard, Pattern Recognition, Rochelle Park, New Jersey, Hayden Book Company, Spartan Books, 1970  
Grateful acknowledgment is made to the following publishers for permission to quote 
excerpts from the following material: The Bach Reader. A Life of johann Sebastian Bach in Letters and Documents, edited by Hans T. David and Arthur Mendel, Revised, with the 
permission of W. W. Norton & Company, Inc., copyright (~) 1966, 1945 by W. W. Norton & Company, Inc. copyright renewed 1972 by Mrs. Hans T. David and Arthur 
Mendel; J. S. Bach's Musical Offering, page 179, edited by Hans T. David, New York, copyright 1945 by G. Schirmer, Inc., used by permission; Gyomay Kubose, Zen Koans, Chicago, Regnery, 1973; Pauls Reps, Zen Fl esh, Zen Bones, Tokyo, Japan, Charles E.

Mendel; J. S. Bach's Musical Offering, page 179, edited by Hans T. David, New York, copyright 1945 by G. Schirmer, Inc., used by permission; Gyomay Kubose, Zen Koans, Chicago, Regnery, 1973; Pauls Reps, Zen Fl esh, Zen Bones, Tokyo, Japan, Charles E. 
Tuttle Co., Inc. 1957; J. R. Lucas, "Minds, Machines, and Godel," and Alan M. Turing, "Computing Machinery and Intelligence," from ,finds and Machines, edited by A. R. Anderson, Englewood Cliffs, New Jersey, Prentice-Hall, 1964, and Philosophy, vol. 36, 1961; J. M. Jauch, Are Quanta Real? Bloomi ngton, Indiana, Indiana University Press,

Tuttle Co., Inc. 1957; J. R. Lucas, "Minds, Machines, and Godel," and Alan M. Turing, "Computing Machinery and Intelligence," from ,finds and Machines, edited by A. R. Anderson, Englewood Cliffs, New Jersey, Prentice-Hall, 1964, and Philosophy, vol. 36, 1961; J. M. Jauch, Are Quanta Real? Bloomi ngton, Indiana, Indiana University Press, 
1973; James R. Newman, "Srinivasa Ramanujan," in The World of Mathematics, edited by James R. Newman, New York, Simon & Schuster, reprinted by permission of Simon & Schuster, a division of Gulf & Western Corporation, 1956; Terry Winograd, "A Procedural Model of Language Understanding,"  from Computer Models of Thought and 
Language, edited by Roger C. Schank and Kenneth Mark Colby, San Francisco, W. H. Freeman and Company, copyright © 1973; Jo seph Weizenbaum, Computer Power and 
Human Reason: From judgment To Calculation, San Francisco, W. H. Freeman and Company, copyright © 1976.

AABB form, 130, 227 Abel, Niels Henrik, 404 ABORT (BlooP), 412 abortion, 176 levels of, 651-53, 656-57, 660, abstraction, ti66-72, 674 Achilles: answer-schemas and, 475; Carroll paradox and, 46, 170, 181, 193; Crab Canon and, 204 666-67; heel of, 389, 484; image of GOD, 223; inaccessible neurons of, 328. 686; initial letter of, 231, 507, 667; innocence of, 406, 408, 424; mapped onto ant colony, 318, 324; mentioned 84, 272, 525, 573, 669; mystified by Crab, 560; orchard analogy and, 427; origin of, 28, 29; picture of, 42; problem reduction and, 610-11; recursion and, 128-31, 149; as violin, 502 Achilles property, 396-98, 415 acoustico-retrieval, 278-80, 460 acronyms, 31-32, 113, 174, 176, 204, 237, 272, 374, 606, 684, 727, 736, 738, 740 acrostics, 7, 81 active site, 528-29, 544 actor formalism, 662-64 ,,Actually Intelligent" programs, 676 addition: Al programs and, 677-78; commutativity, associativity of, 55, 225-26; in BlooP, 409; noncommutative, 222-23, 639; pqsystem and, 49-53, 417; representability of,

picture of, 42; problem reduction and, 610-11; recursion and, 128-31, 149; as violin, 502 Achilles property, 396-98, 415 acoustico-retrieval, 278-80, 460 acronyms, 31-32, 113, 174, 176, 204, 237, 272, 374, 606, 684, 727, 736, 738, 740 acrostics, 7, 81 active site, 528-29, 544 actor formalism, 662-64 ,,Actually Intelligent" programs, 676 addition: Al programs and, 677-78; commutativity, associativity of, 55, 225-26; in BlooP, 409; noncommutative, 222-23, 639; pqsystem and, 49-53, 417; representability of, 417; of supernaturals, 455; TNT notation for, 206-7; triple, 101, 206-7 addresses (in memory), 289, 290 adenine, see nucleotides advertisements, 478 Al: applied to mathematics, 573, 614-15; arguments against, 597-99; attitudes against, see anti-Al attitudes; computer languages and, 299-300, 548; converging towards brains, 579; defined, 26; difficulty

of, 26-27, 573, 740; evidence and, 695; 
faith underlying, 572, 578-79; G6del's Theorem and, 388-90, 471-77, 706-7; history of, 19, 24-27, 594-97, 600-9;

leveltranslation and, 285; outline of, 601-3; relation to mathematics, 559-60; stratification of, 299-300 Al programs: compared with people, 679-80; curiosity of, 679; Lucas' argument and, 577-78 Al Thesis, 579 Air and the Song, The (Magritte), 494 Air on G's String, 445, 446, 497 airplane-haystack story, 675 aleatoric music, 163, 174, 700 Algol, 292, 293, 381, 630 algorithms, 292, 410, 412, 413, 414, 440, 459, 567 all 60 "almost"-situations, 634-40, 641-43, 649 alpha helix, 521, 525 Alternative Structures of the Union, see ASU's ambiguity: computer languages and, 297-98; in translation into TNT, 209-11 amino acids, 518-25, 533-34; in Typogenetics, 508, 510-11 analogical thinking: by computer, 603; underpin ning of, 570-71 analogies, 668-74 Analytical Engine, 25, 598, 727 anaphase, 667 anarchy, 693 "and", 177-80, 181, 186, 630 Anderson, Alan Ross, 197 Announcer, 633-40 anomalies, 44, 96, 208, 435, 723 Another World (Escher), 250, 255 answer-schemas, 462-14, 475, 688 ant bridge, 334 ant colonies: artificial,

of the Union, see ASU's ambiguity: computer languages and, 297-98; in translation into TNT, 209-11 amino acids, 518-25, 533-34; in Typogenetics, 508, 510-11 analogical thinking: by computer, 603; underpin ning of, 570-71 analogies, 668-74 Analytical Engine, 25, 598, 727 anaphase, 667 anarchy, 693 "and", 177-80, 181, 186, 630 Anderson, Alan Ross, 197 Announcer, 633-40 anomalies, 44, 96, 208, 435, 723 Another World (Escher), 250, 255 answer-schemas, 462-14, 475, 688 ant bridge, 334 ant colonies: artificial, 359; caste distribution in, 318-28; castes in, 317-18; communism in, 318, 330-31; compared with brains, 315-16,

Index 762317; freedom and control in, 315-16, 327; 
intel ligent, 310-36; levels in, 319-27; mechanisms of teams and signals, 317-21; order and chaos in, 316-17; regrouped, 332-33; signals in, 320-28; symbol level in, 324-28, 330; teams in, 317, 319-27; thresholds in, 316-17, 319-21; trails of, 315-17; see also caste distribution, Aunt Hillary, Fermant, job. Seb., Fourmi, Lierre de, teams, signals, symbols Ant Fugue, 337, 349, 350, 382, 570, 686, 737-39 "Aid Fugue" (Escher), 322-23 Anteater, 275-84, 311-36, 382, 570, 722 anti-AI attitudes. 27, 470-72, 628 anticodons, 522-24 ants: dispensability of, 326; nest of, 359; vs. ant colonies, 314, 315, 319, 321, 326, 330 aperiodic crystals, 167-69, 174-76 arch of 
termites, 358-59 Aria with Diverse Variations (Bach), 392-

93, 395 Aria with Diverse Variations (Dialogue), 408 Aristotle, 19 arithmetization, 262-65, 268-69, 533-34 arithmoquining, 445-54, 466-68, 497, 502, 541, 580-81 arms, mutually washing, 691 art: identity crisis of, 699-700, 703-6; modern, 699-706; by computer, 603, 619-20 Art-Language, 622 Art of the Fugue (Bach), 79-81, 86, 671 Art of Zen Strings, 237, 239-42, 244, 626 Artificial Imagery, 560 Artificial Intuition, 560 Artificial Ism, 625 artificial "thinking", 337, 601 Ascending and Descending (Escher), 12-13, 15, 21, 716 Asian Box, Very Gold, see Gold Box, Very Asian assemblers, 291, 294 assembly language, 290-95; compared to DNA, 290-91 assembly lines, cellular, 528-29, 544-45 
 
Appearances of Achilles and the Tortoise in  Dialogues are not indexed, but those of less 
frequent characters are. The reader is enc ouraged to consult the figure on page 370 for

frequent characters are. The reader is enc ouraged to consult the figure on page 370 for 
possible help in crossreferences.  associativity, 35-56, 207 assumptions, shaky, 420, 422, 429. 580, 581, 644, 672 ASU's: defined, 373-74;  orientation in, 
375; trips in, 377-79, 383 ATN's, see Augmented Transition Networks ATN-colony, 359 atoms: in physics, 303; in Propositional Calculus, 182, 186; in TNT, 207, 213, 214 ATTACCA, 176, 284, 311, 505 attachment and nonattachment, 232-33, 244 augmentation, 8, 9, 146, 503, 681-83, 738; inter vallic, 156-57 
Augmented Transition Networks, 150, 258-59, 359, 621, 625-27, 630 Augustus II, Elector of Saxons, King of Poland, 461 Aunt Hillary, 314-33, 382, 630 Author, the, 3, 6, 204, 370, 689, 722, 724-42 authorship triangle, 94-95, 96, 689 autological adjectives, 20-21 automatic programming, 298-99 Avers, Oswald. 159, 161 awareness, deductive vs. analogical, 570-71, 619

Index 760axiom schemata, 47, 48, 65, 87, 468, 472-
73, 543 axiomatic systems, see formal systems axioms: defined, 35; lack of in Propositional Calculus, 183; of MIU-svstern, 33, 35; of P-system, 73-74; of pq-system, 47; of pq-system, modified, 87; of 310-system, 263; of TNT, 216; of TNT, extended, 451-52, 466-68; of tq-system, 65 axons, 339-40 Babbage, Charles, 24-26, 598, 601, 726-42 Babbage test, 735-73 BACH (acronym), 174-75 B-A-C-H (melody), 79-81, 86, 102, 121, 155-57, 266, 719 Bach, Anna Magdalena, 482 Bach, C. P. E., 3, 4, 80 Bach, Joh and Seb., 633, 669 Bach, Joh. Seb.; Al and, 27, 677; as composer, 392, 461, 740-42; as glassblower, 79; as harpsichordist, 275, 279, 280; as inspiration for Dialogues, 28, 737; confused with Fermat, 331-35; depth of, 7-8, 9-10, 27-28, 70- 71, 677; dissection 
vs. appreciation of, 680; Escher and, 199, 666-t7; Forkel on, 4, 86; homage to, 81; 
improvisation by, 3-7, 96, 719; in Leipzig,

improvisation by, 3-7, 96, 719; in Leipzig, 
383, 404; indirect self-reference of, 79-81, 86; life and death of, 86; modulation and, 122-23, 130; recursive qualities of music of, 70-71; Shepard tones and, 719; squared, 679; vs. Cage, 157, 1624, 174-75; see also Old Bach Bach, Wilhelm Friedemann, 4, 6 background assumptions, 644 backtracking, 97, 629, 632 backwards chaining, 618 bankers, 457 base-pairing, complementary, 433, 506-7, 514-16, 517, 523, 534 baseball, 637-38 bases (genetics): 514, see also nucleotides; in Tvpogenetics, 505-8, 510 Baso, 234 Bassui, 255 beautiful vs. non-beautiful, 552-58, 560,

573-75, .581-82 beauty: computers and, 575; elusiveness of, 554, 555, 565, 574-75, 581, 583-84 bees, 360, 641, 720 Beethoven, Ludwig van. 6. 75, 163, 634 beliefs, catalogue of. 382, 384 Bell, A. G., 296 Belnap, Nuel, 197 Beno, Luciano, 704 bicentuplets, 540 bifurcations, 91-94, 100, 456-59, 467, 579 biojukeboxes, 160, 175 Birthday Cantata (Bach), 461 Birthday Cantatatata (Dialogue), 468, 475, 688 birthdays, 461-64 bits, 288-89, 290, 291 BLOCK (BlooP), 410-11 blocks world, 586-93, 627-32, 674 BlooP, 406. 409-30, 440, 441, 444; alphabet of, 419, 425; control structures in. 410; primordial steps of, 409, 412-13; syntax of, 410-15 BlooP programs, 410-14 Blue Programs, 418-20, 422, 427 Bluediag [N], 42-44, 428 
board evaluation: static and dynamic, 604-

5, 611; Strange Loops in. 604-5 Bodhidharnta, 232, 238, 245. 252, 625 Bolvai, Farkas, 92 Bolyai, Janos, 91-92 Bongard, M., 646 Bongard problems, 646-62, 664. 669, 674; uni versality of, 661-62 Boole, Geo., 20, 404, 600 Boolean Buddhism, 577 boomerangs, 75, 84, 267, 472 bootstrapping, 24, 293-94, 524, 548 bottom (recursion), 139-40; see also skeletons bottom-up vs. top-down, 48-49 bottoming out, 133-35, 151, 259, 301, 650 boustrophedonic writing, 168-69, 176 boxed systems, 469-70, 543 Boyle's law, 308; see also gases brains: ant colonies and, 315-16, 318, 324-25, 350, 358-59; as ATN-colonies, 359; as

Index 761mathematical objects, 559; Epimenides 
paradox and, 584-85; formal systems and, 337-38, 559-62, 569-79, 584-85, 676, see also Church-Turing Thesis, formal vs. informal systems, brains and rules; frame and outer messages and, 170-71; mappings between, 341-42, 345-46, 369-82; music and, 163; operating systems and, 296; programmability of, 302; rules and, 26-27, 676, see also brains and formal systems; suborgans of, 340-41; thoughts and, 337-65; vs. minds, 576; see also minds, intelligence, etc. Breton, Andre, 700 "British Grenadiers, The", 607 Brouwer, Luitzen E. J., 404 Buddha-nature, 233, 234, 238-44 Butterflies (Escher), 147-48 Buxtehude, Dietrich, 335 Byrd, Donald, see SMUT Byron, Lord, 25 C-svstem, 65-67, 71-72 CAGE (acrorn.m), 174-75 C-A-G-E (melody), 156-57 Cage, John, 156-57, 163-64, 167, 174-75, 556, 
699-700, 704

699-700, 704 
calculating prodigies, see idiots savants Canon by Intervallic Augmentation, 525 "Canon per augmentationem et contrario motu" (Bach), see Sloth Canon ,,Canon per Tonos" (Bach), see Endlessly Rising Canon canons: copies and, 8-9, 146; Dialogues and, 665-69, 738; Escher drawings and, 15; in Goldberg Variations, 392; in Musical Offering 7-10, 726-27; polyribosomes and, 526-28; self-refs and, 501-3; structure of, 8-10; two-tiered, 527-28; see also individual canons, fugues Cantor, Georg, 20, 216, 418, 421, 422-24 Cantor set, 142 .Cantorcrostipunctus", 424 Capitalized Essences, 29 car radio. 670-71 cardinalitv, intuitive sense of, 567 Carroll, John B., 630

Canon, Lewis, 20, 28, 46, 192, 372, 681; material by, 43-45. 366-68 Carroll paradox: 28, 43-45, 681; evidence ver sion, 693-94; message version, 170: problem po4ed by, 46. 181; proof version, 192-93; Sam uel's argument and, 684-85; symbolized, 193; see also infinite regress cascades, 224, 529, 626, 664 caste distribution: encoding of knowledge in, 319, 324-28, 359; meaning of, 321-24; updating of, 318-19, 324 catalogues of programs (Blue, Green, Red), 419, 427-28 catalysts, 528-29 cats, 313, 343-46, 532 causality, types of, 709-10 CCrab, see ATTACCA ceilings, see loops, bounded: BlooP celestial mechanics, 353-54 CELLS (BIooP), 410-11 
cellular processes, as models for All, 663-

64 Central Crabmap, 667 Central Dogma: of Mathematical Logic, 271, 532-34; of the MIC-system, 513; of Molecular Biology, 504-5, 514, 532-34, 536, 667; of Ty pogenetics, 513; of Zen strings, 238, 239, 240, 243 Central Dogmap, 532-34, 545, 547, 672, 709 Central Pipemap, 701-2 central processing unit, 288, 289 Central Proposition, 264, 269 Central Slothmap, 702 Central Xmaps, 702, 716; see also individual en tries

Index 762centrality, 374-75 centromere, 668 
cerebellum, 341 Chadwick, John, 50 chain letters, 546 Champernowne, David, 595 Champollion, Jean Fransois, 165 channeling, 299, 376-77 chaos in number theory, 137-38, 152, 557; see also order and chaos chauvinism, 171-73 checkers programs, 573, 604-5 Chekhov, Anton, 642 chess: chunking and, 285-87, 604; grand masters in, 286-87; round-the-house, 595; self-modifying, 687-88 chess boards, hierarchy of, 687 chess players, cycle of, 94-95 chess programs: Babbage and. 25, 729-31, 736; choice and, 711-12; Crab and, 721, 729-31; difficulty of, 151-52, 605; jumping out of the system and, 37-38, 678; knowledge representation in, 618; recursive structure of 150-52; strengths and weaknesses of, 151-52, 285-87, 573, 603-4, 611; Turing and, 595, 596, 736: varieties of, 601; without look-ahead, 604 
chests of drawers, nested, 644-45

chests of drawers, nested, 644-45 
children's stories and Al, 675-76 Chiyono, 256 choice, 711-14 Chopin, Frederic, 70, 257, 677 chords and analogies, 673-74 chromosomes, homologous, 668 chunked versions of this book: jacket, viiixiii, 370, 758-77 chunking: ant colonies and, 326-27; brains arid, 381-84, 559; computer languages and, 290-92, 381, 412-13; defined, 285-88; determinism and, 306-8, 363, 522; of DNA, 531-32; intuitive world-view and, 305-6, 362-63; of music, 160, 164, 525; of one's own brain, 382; probabilistic, 384; scientific explanation and, 305-6; super conductivity and, 305; trade-offs in, 326; vision and, 348 Church, Alonzo, 428, 476, 561 Church-Turing Thesis, 428-29, 552, 561-
79; Al Version, 578- 79, 580, 581; Hardy's

79; Al Version, 578- 79, 580, 581; Hardy's 
Version 566; Isomorphism Version, 567-68; Microscopic Version, 572; Public-Processes Version, 562, 568, 574, 580; Reductionist's Version, 572, 574; Soulist's Version, 574; Standard Version, 561-62, 579; Tautological Version, 561; The odore Roszak Version, 574-75; unprovability of, 562 Church's Theorem, 560--61,574,579-81,609, 697 cigars, 199, 201, 383, 481, 651 classes vs. instances, 351-55, 360-61; see also prototypes, intensionality and extensionality, analogies, conceptual skeletons, etc.codes: art and, 703-4; familiar and unfamiliar,82, 158. 267; see also decoding, Gödel Code, Genetic Code, etc. "coding" of sentences, 583-84 codons, 519-20, 524, 533, 535; see also Gödel codons, duplets Colby, K., 599 columns in brain, 346 Comenius, Johann Amos, 625 comments in programs, 297 Common Sense (Magritte), 
700-1 common sense and programs, 301

700-1 common sense and programs, 301 
communicability of algorithms, 562 commutativity, 55-56, 209, 225-27, 453, 639 compelling inner logic, 161-62, 163-64 competing theories, and nature of evidence, 695 compiler languages, 292-95 compilers, 292-95, 297, 503 compiling, reverse of, 381 Complete List of All Great Mathematicians, see List completeness, 100-2, 417-18, 422, 465; see also incompleteness, consistency complexity of world, 569 composite numbers, 64, 65-66, 73; see also prime numbers compound sentences, in TNT, 214 compound words, 665 computer chess, see chess programs

Index 763computer languages: analogues in cell, 
547; dialects of, 503; flexibility and, 298-99; high level, 292-93, 297-300; message-passing, 662-63; power of, 299, 428-29; presented, 289-99, 406-30, 498-99; in SHRDLU, 629-32  computer systems, 287-302 computers: assembled by computers, 504, 684; crying, 675-76; determinism and, 25-27, 306-7, 684-86; fallibility of, 575, 578, 678; learning by, 603-5; origins of, 24-26; in phonographs, 78, 484, 486-88; see also programs, Al, Al programs concept network, 651-54; see also semantic networks conceptual dimensions, 670-71 conceptual mapping. 668-72 conceptual nearness, 371-73, 614, 651-56 conceptual revolutions, 660-61, 673 conceptual skeletons, 381, 514. 666-72, 674 connotations and culture, 372-73, 379-80 consciousness: causality and, 709-10; source of, 384-85, 387-88: understanding of, 82, 680, 708-10 conservation of complexity, 60, 195 consistency: defined, 94; of extended TNT, 
223, 459; hypothetical worlds and, 95-100;

223, 459; hypothetical worlds and, 95-100; 
interpretations and, 88, 94-101; Lucas and, 477; oath of, in TN T, 450; proofs of, 23-
24, 191-92, 229-30, 449-50; of Propositional Calculus, 191-92, 229; of TNT, 229-30, 449-50; varieties of, 94-96; see also (o-consistency  constants, parameters, variables, 643-44, 669 context: necessity of, 1614, 173-76; restoring of, 115-16, 128, 133, 161-64, 173-76 context-free sums, 520-22 contexts, nested, 643-46, 672, 674 continued fractions, 140, 277, 563, 565 continuous vs. discrete processes, 598 Contracrostipunctus, discussed, 82-85, 267, 270-71, 406-7, 424, 467-70, 483-84, 534-37, 608, 721 contradictions: between levels, see level-

conflicts; caused by impossible cycles, 94-96; coexisting in same brain, 383, 697-98; diagonal argument and, 420-22; in mathematics, 17-24, 196-97, 223, 580-81; so-consistency and, 453; personal nonexistence and, 698; in pq-system, 87, 88; in Propositional Calculus, 191-92, 196-97; in selfimage, 696; Tortoise's shell and, 177-80; two levels of, 581, 584; visual, 97-99; Zen and, 99, 235, 246-56, 698; see also paradoxes, inconsistency, Epimenides paradox, etc. Contrafadus, 641, 643, 669 Convex and Concave (Escher), 105-9, 348 Cooper pairs, 304-5 copies: canons and, 8-9, 527-28; in code, 517, 527-28; complementary to originals, 70, 501, 506-7, 517, see also inversion; DNA and, 529-31; inexact, 500-3, 546; nature of, 146-49; self-reps and, 500-4, 512-13; television and, 489; viruses and, 542-43; visually nested, 138-40; see also sameness, isomorphisms copper, 173 Cops Silva and Could, 405 
Copy mode (Typogenetics), 506-8

Copy mode (Typogenetics), 506-8 
cortex: areas of, 344; cerebral, 340-48; visual, 343-48 counterfactual parameters, 639 counterfactuals, 634-40, 641-44 counterpoint, see canons, fugues, Bach, etc. counting, 55-57, 228, 364 court system, 692, 693 covalent bonds, 514 cows, 312, 346, 351 CPU, see central processing unit Crab: entertains Achilles, 480-94; genes of, 200-1, 204. 507;' hikes and plays flute, 549-58; intelligence of. 549, 558, 733; jukebox of, 154-57; meets Achilles, 200; musical evening chez, 720-42; origin of, 666-68; plight of, 281, 383-84; questionable behavior of, 560, 562,

Index 764573-74, 579-81; receives presents and enter 
tains guests, 275-84, 311-36; subjunctive afternoon chez, 633-40; Theme of, 729, 732. 740, 742; vs. Tortoise, 75-78, 406, 493-88,540, 543 Crab Canon (Bach), 202-3, 666 Crab Canon (Dialogue), 204, 355-56, 665-69, 672, 723-25, 738 Crab Canon (Escher), 198-99, 667 crab canons, 9, 198-203, 204, 355, 501. 665-69; in DNA, 200-1 crab programs, 500-1 creativity, mechanizability of. 25, 26, 571, 620, 673; see also originality, paradox of Al, non programmability, etc. Crick, Francs, 505, 532, 533, 534, 617 Crime and Punishment i'Dostoevskvl, 379-80 Cristofori, Bartolommeo, 3 critical mass, 228, 317, 389, 417, 470 crossing-over, 665-68 crystal in magnetic field, 140-43 crystallization metaphor, 347 C"f-Thesis, see Church-Turing Thesis Cube with Magic Ribbons (Escher), 281-82 cursively drawable figures, 67-68, 72 cytoplasm, 517, 518, 522-24 cytosine, see nucleotides da Vinci, Leonardo, 641 
Dali, Salvador, 700

Dali, Salvador, 700 
Dase, Johann Martin Zacharias, 567 data base, 618 daughter programs, 503, 546 David, Hans Theodore, 3, 28, 719 Day and Night (Escher), 252, 255, 667 Dboups, Hfpsh, 515, 533 De Chirico, Giorgio, 700 De Morgan, Augustus, 20, 404, 600 decidable strings, 417 decimal system, 262-64, 269 decipherment of texts, 50, 164-65, 173-74, 583 decision procedures: for alienness, 487-88, 540-41; for axiomhood, 41, 48, 470; for beauty, 552-58, 560, 579, 581-82, 583-84; as BlooP puzzles, 415-17; for Buddha-nature, 234, 239, 272; defined, 39-41; for Diophantine equations, 559-60; for dreamable themes, 384; for end of Dialogue, 402-3; for genuineness of koans, 234, 239; for Goldbach property, 400, 414; for Granny, 344-45, 347-48; for halting, 425-29; for intelligence, see Turing test; for

Mozart pieces, 649; no guarantee of existence of, 72; for number-theoretic truths, 228-29, 426, 551-58, 560. 573-74, 579-81; for primality, 64, 149, 413; for proof-pair-ness, 416, 439-41; for sameness, 146-49, 158-59; for sorting numbers into two classes, see Church-Turing Thesis; for termination, 425-29; for "the Way" in Zen, 250-51, 253, 254; for theoremhood, 39-41, 47-49, 72-73, 190-91, 408, 416, 560, 579-80, 582; for theorem-numbers, 440-41, 580; topdown vs. bottom-up, 48-49; for Tortoise-pairness, 441; for Tortoise property, 396-97, 415, 441; for truth, 213, 228-29, 417, 552-58, 560-61, 579, 581; for validity of derivations, 194, 416, 439-41, 470; for well-formedness, 182, 269, 416, 582; for wondrousness, 402, 425  d coding: of DNA, 159-62, 175-76, 201, 231, 531-32, 538; of formal systems, 50-51, 54; of fortune, 154; via Gtidel isomorphism, 267; of nature. 409; of records, 154-57, 158-59, 161-64, 172, 174-75; as revelation, 160-61; 
of Russian text, 380; see also

of Russian text, 380; see also 
isomorphisms, translation, information, chunking decoding mechanisms: complexity of, 158-62, 172-76, 582-84; innate, 170-71; nature of, 158-76; record players as, 83, 154-57; transparency of, 267, 501; for Tripitaka, 257; see also isomorphisms, etc. L)eduction Theorem, 186 default options, 352-53, 386, 411, 645, 674 defects and expectations, 77, 86, 102, 222, 476 demidoublets, 633-34, 669 demons, 663-64 deoxyribonucleic acid, see DNA derivations: alleged, 439-40; defined, 35-36: fallacious, 220, 439; in MIU-system, 35-36, 262, 264, 439; in Propositional Calculus, 184, 185, 188, 189-90, 196; supernatural, 454-55; in TNT, 217, 218, 219, 224-27, 269; in tq-system,

Index 76565; in Typogenetics, 507, 509; vs. proofs, 
35-36, 193-94, 195 Descartes, Rene, 263, 340, 677 description-schemas, 650; see also templates descriptions: calculus of, 338; restructuring of, 649-53, 659-61, 672; tentative, 646-49 descriptors, 647 detachment, rule of, 185-86, 577 determinism, 54; see also free will Devil, 685 Dewdrop (Escher), 249, 256 Di of Antus, 333 diagonal method, Cantor's, 418-24, 426, 427-29, 438, 446, 469 Diagram G, 135-37 dialogicians, 81 dialogues, miniature, 191-92, 193, 408-9, 431, 560, 565, 595-96, 598, 599 Dialogues: origin of, 28, 665-69; as self-refs, 84-85, 129, 204, 502-3, 667, 738-39 Dickens, Charles, 326, 328, 380 Difference Engine, 25 differentiation, cellular, 543-46 digestion, 306 
digits, shunting of, 264 Diophantine

digits, shunting of, 264 Diophantine 
equations, 279, 459-60 Diophantus of Alexandria, 275 directory of real numbers, 421-24 disambiguation, 586-93, 603, 629-32 distance to goal, 611-13 divisor-freeness, 74 djinns, 113-15, 216-17, 223, 224; see also Genie DNA: as aperiodic crystal, 167; as carrier of genetic information, 159; compared to computer languages, 290-91; composition and structure of, 514-15; covalent backbone of, 514-15; as declarative knowledge, 616-17; double strands, 514-15, 530-31; isomorphism with organism, 146-48; mode of self-replication, 529-30; in outer space, 167, 175-76; as program, language,data, 290-91, 547; quining and, 531; recombinant, 665; relation to mRNA, 517; self-destroy ing, 536; unusual interpretation of, 231

DNA endonudease, 530, 531 DNA ligase, 530, 531 DNA polymerise, 530, 531 DNA Rapid Transit Service, 505. 517 Doctor program, 599-600. 608 dog-and-bone problem, 611-13 .doGma 1, doGma II, 532-33 dogs, 233, 234, 354, 383, 569, 570, 611-12, 679 Doko, 250, 698 Dostoevsky, Feodor, 379-80 double negation, 183, 545, 554 Double Nodulation, law of, 243 Dragon (Escher), 473-74, 524, 698 Drawing Hands (Escher), 15, 21, 133, 689-92, 710, 716, 737 dreams, 378, 379, 384, 725 Dreyfus, Hubert, 574 dualism, 251-55, 698-99; see also subject vs. object Dumpty, H., 332 duplets (Typogenetics), 510, 512 Dvoi£k, Antonin, 163  E. coli bacterium, 176, 537-41 Earrwig, Dr. Tony, 586-93. 627 earth chauvinism, 171-72 
Earth-Moon-Sun system, 353-54

Earth-Moon-Sun system, 353-54 
earthworm(s), 341-42 Eccles, John, 574 Edifying Thoughts of a Tobacco Smoker (Bach), 482 Edifying Thoughts of a Tobacco Smoker (Dialogue), 543 eggs, 192, 360, 383 Einstein, Albert, 100 electrons, 140-46, 258, 30.3-5 ELIZA, see Doctor program embedding of formal systems, 97, 207, 215 emergent phenomena, 708-9, 714 emotions: brains and, 83; counterfeit, 599-600; dependence of intelligence upon, 573, 597-98; as epiphenomena, 677; music and, 83, 160, 163-64, 174-75, 383-84, 626-27, 676-77, 699; potential, 281, 383-84, 583; programs and, 573, 597-600, 626-27, 675-77; universality of, 163, 174-75

Index 766emulation, 295 
Endlessly Rising Canon (Bach), 10-11, 15, 46, 130, 717-19, 742 ENIUQ (procedure), 498-99 enlightenment, 232, 237, 243, 246, 251, 254-55, 479, 567 Enlightenment 'Yond Enlightenment, 237, 239, 243, 244 End, 232, 252 enzymes: function of, 520-22, 528-30, 543-45; as models for AI, 663-64; rules of inference and, 509-10, 513, 531; structure of, 519-21, 525; synthesis of, 517-19, 522-25, 527-29, 538-45, 547; versatility of, 529; vs. typoenzymes, 529; see also proteins, typoenzymes epigenesis, 159-60, 161-62, 531-32, 665 Epimenides, picture of, 496 Epimenides paradox: connection with G6del's Theorem, 17-18; Escher and, 716; expanded version, 21, 22; fear of, 23; French-English version, 501; indirect recursion and, 134; molecular version, 536-37; neural version, 584-85; Quine version, 431-37, 445, 446, 449. 497-99, 531, 537; subtlety of, 495-98; Tarski version, 580-81, 
584-85; two levels of, 581, 584-85;

584-85; two levels of, 581, 584-85; 
Whitely's version, 476-77 epiphen9mena, 308-9, 363, 577, 596, 677-79 errors in programs, 295, 297-98. 596 Escher, Maurits Cornelis: Bach and, 201, 666-67; contradictions and, 97-99; copies and, 146-48; drawings of, see List of Illustrations (xiv-xviii); figure and ground in, 67-68; flat vs. spatial and, 473-74, 689; 
incompleteness and, 716-17; Magritte and, 480; as prime mover, 689-92, 710; on subbrains, 387; Strange Loops and, 10-15, 737; Zen and, 255-57 Escherization, repeatability of, 473-74, 689; see also 2-D vs. 3-D, Godelization 1.SP, 598-99, 693-95 essential incompleteness: of Achilles' birthday, 462-64, 475-76, 688; of Al, see Tesleis Theorem; of list of reals, 423-24, 469; of phonographs, see Todelization; of

self-engulfing process, 493; of TNT and related systems, 468-71; see also Godelization, Tlidelization, Escherization, nonprogrammahility, etc. Eta Oin, 586-93, 674 ETAOIN SHRDLU, 628,630 Euclid, 19, 43, 58-60, 88-89, 216 Euclid's prime number Theorem, 35, 58-60, 228 Euler, Leonhard, 3, 394 Fume, Max. 605 evidence, nature of, 633-36 evolution, 321-22 existential quantifiers. see quantifiers exotic styles of thought, 552, 563-64, 566-67 expanding nodes, 134-36 explanatory power on high level, 321, 326, 707-10 expressibility and expressive power, 101, 417, 441-43, 444-45, 450, 454, 465-70, 580-81 extrasensory perception, see ESP extraterrestrial intelligence, 162-64, 167, 172-76, 341, 646, 661-62 eyes, 237, 248, 260, 308, 311, 313, 477, 633, 715

Fair Captive, The (Magritte), 489 FANCY NOUN, 132-34 Fantasia and Fugue in G minor (Bath), 719 fantasy rule, 183-85, 187 faucet, mental, 364-65 faultlessly functioning machines, 575-78 Faure, Gabriel, 163 feedback and feedforward, 544-45 Fermant, Johant Sebastiant, 332-35 Fermant's Last Fugue, 335 Fermat, Pierre de, 275-77, 278; confused with Bach, 331-35 fermatas, 275, 329, 332, 333 Fermat's Last Theorem: 275-79, 332, 416; coun terexample to, 277, 279, 460; inverted, 333-34; parodied, 335, 551; proof of, 277, 279, 460 Feynman diagrams, 144-46

Index 767Fibonacci (Leonardo of Pisa), 136, 246 
Fibonacci sequence, 136, 138, 139, 152, 173-74, 265, 416 fiddles, see violins fifth postulate (Euclid), 90-93, 222, 451-52 50 (fifty), 338, 557, 564 figure and ground: 61-63, 64-74, 731; in music, 70-71 FIGURE-FIGURE figure (Kim), 68-70, 73 filters: for abstraction, 286, 407-9, 648, 657-60, 663, 673; for Pools, 418, 427 finitistic methods of reasoning, 24, 230 Fishes and Scales (Escher), 146-47 fission and fusion (of concepts), 338, 352-56, 438, 470, 664-65 5-D space, 640 flags, 29-32, 188 flashcards, fee tRNA flat vs, spatial, see 2-D vs. 3-D flattened look-ahead, 604-5 flattened roles of inference, see theorems vs. rules flexibility and inflexibility, 26-27, 296-302, 611-14, 657, 673-75, 686 flights of fancy, 378 
FlooP, 406, 424-30, 567-68; fed into itself,

FlooP, 406, 424-30, 567-68; fed into itself, 
425-26; power of, 428-29. 561-62 fluency, 376-77 flutes, 3-5, 27, 528, 552-58, 720, 726 F(n) and Min), 137, 142, 359 focusing, 657-59 folding-up of enzymes, 511-12, 519, 521, 525 football, 44, 303, 353, 634-40, 643, 644, 645, 672 footraces, 29-32, 43, 594-95, 681-83 forced matching, 670-72 forgetting, 577, 578, 619 Forkel, Johann Nikolaus, 4. 86 form, 47. 66-67, 68, 73. 190-91, 370-71; syntactic vs. semantic, 581-84, 631 form and content, 84-85, 204, 279, 581-84, 667-68, 740 formal systems, presentations of: C-system, 64-65; MIU-system, 33-41; P-system, 73-74: pq-system, 46-60; Propositional Calculus, 181-97; TNT, 204-30; tq-system, 64-65;

Typogenetics. 504-13 formal systems vs. reality, 53-58 formal vs. informal reasoning, 193-97, 228-29, 271-72. 449-50, 614-15, 618-19 formal vs. informal systems, 26-27, 559-85, 598, 684-86; see also brains, minds, etc. formalist philosophy of mathematics, 458 formula: closed, see sentence; open, 207-8 formulas of TNT, 206, 207-15 Four-color Theorem, parodied, 550 4-D space, 638-39 four-postulate geometry, see geometry, absolute 4'33" (Cage), 156 Fourmi, Lierre de, 333-34 frame effect, 704 frame messages, 162, 166-67, 176 frames, 373, 644-46, 662-63, 672 framing devices, 478 Frank, Philipp, 642 Frederick the Great, King of Prussia, 3-8, 27, 394, 729 free will, 388, 680, 708, 710-14, 720-16, 734-35, 739; see also jumping out of the system Frege, Gottlob, 20 French fries, 636-38, 683 
French Suite no. 5, Gigue (Bach), 130

French Suite no. 5, Gigue (Bach), 130 
frequencies, of words and letters, 377, 630 friend, mental model of, 386-87 fringe science, 693-94 fugues, 335, 634, 730, 736; in the Art of the Fugue, 79-81, 86; devices in, 314, 322-23, 329-30, 737-40; Dialogues and, 28; in the Musical Offering, 4-9; nature of, 9, 281-84, 737 Fundamental Facts 1 and 2. 440-42 fundamental jukebox-axiom, 155 funneling, 346-48 G (Godel's string), 18, 271-72, 285, 447-55, 459-60. 502, 580, 608, 667, 707-8 G', G" G"' Gu, 466-68 -G, 272, 449, 451-55, 458-59, 542 Galileo, 478-79 games played by Al programs, 601 Ganto, 189-90, 256, 407 gases and molecules, 307-8, 317, 693 Gateles.s Gate, see Muvtonkan Gauss, Karl

Index 768Friedrich, 92, 100 Gebstadter, Egbert B., 
94-95, 402-3, 484 Gelemter, E.. 606-7 general recursivity, 406, 430, 470 genes, 200-1, 507, 512, 524-25, 531, 544-45, 668 Genetic code, 160. 519-20, 522-24, 533-34, 536, 538; origins of, 231, 548 genetics, 504-48 (.enie, Meta-Genie, etc., 109-16, 216-17, 223, 224, 610; see also djinns genie, symbol-manipulating. 39-40, 48 genotype and phenotype, 159-62, 167, 173-74, 175-76, 295, 531-32, 667-68 Gentzen, Gerhard, 195 Geometric Code. 235-37, 241, 626 geometry: absolute, 91, 93, 97, 222, 407, 451-52; elliptical, 93; Euclidean, 19-20, 88-92, 100, 222, 451, 456, 606-7, non-Euclidean, 19-20, 91-93, 98-99, 100,222-23, 451, 455-56; "true" version of, 88-94, 99-100, 456-57 German professor, proverbial, 130-31 Giant Electronic Brains. 25, 601 glia, 339 GlooP, 406, 428-29 G(n), 137 
goals and subgoals, 227, 589, 590-91, 609-

14, 618-19, 629, 632 Goblet G, 79, 81, 83-85, 267 God, 216, 400, 478, 482, 533, 567, 597, 711; picture of, 142-43 GOD (acronym), 110-15, 133, 134, 216, 223, 224, 722; see also djinns, Genie Gddel, Kurt, 15-19, 24, 28, 738, 740, 742 G6del Code, 18, 268, 533-35 Godel codons, 268, 425, 533-35 Godel isomorphism, 261-71, 439, 442-46, 738-39; likened to reflection of world in brain, 502, 570 Godel-numbering, 18, 438, 738-39; of FlooP programs, 425-26, 502; of MIU-system, 261-64; of TNT, 268-70, 579 Godel questions, Lucas on, 389, 390 Godelization, 270; programmability. of, 471-73; repeatability of. 424, 465-76, 688; see also

Escherization, Todelization, jumping out l of the system Godelizing operator, 472-73, 475-76, 543 Godel's argument, summarized, 18, 272, 448 Godel's article. 17, 24, 438 Godel's construction, illustrated, 84 Godel's method, underlying causes of, 204, 407, 465, 468-71 Godel's Second Theorem. 230, 449-50, 696 Godel's Theorem: All and, 388-90, 471-77, 706-7, 714; analogue of in molecular biology, 534, 536-37; brief mentions, 72, 74, 78, 100, 486; consequences of, 450-60, 469-76; Contracrosttpunctus and, see Contracrosttpunctus: Diophantine equations and, 459-60; LISP and, 738-39; proof of, 18, 265-72, 438-49; stated, 17, 101, 272 Godel's Theorems and human introspection, 450, 696-98 Goffman, Erving, 478 gold, 173 Gold Box, Very Asian, 404-5 Goldbach, Christian, 394, 395 Goldbach Conjecture, 394-96, 400, 404, 557-58, 615; parodied, 551 
Goldbach property, 395-97, 400, 414, 418

Goldbach property, 395-97, 400, 414, 418 
Goldbach Variation, 395-98. 400, 426, 441 Goldberg, Johann Theophilus, 391-92 Goldberg Variations (Bach), 392-93, 395 Goodfortune, Hexachlorophene J., 103-4, 115, 128-29, 130 Goso, 248 Gplot, 138, 140-43, 146-47, 159, 503, 661 grammar: for computer languages, 297, 408-15; for Feynman diagrams, 142. 145; high-level, 625-27; for koans, 625-26; for music, 626-27; for natural languages. 130-34, 150, 363, 588-93, 619-21, 630-32; for thought, 627 Grand Tortue, 237, 243-44 grandmother cell, 344, 345 grandmothers, perception of, 344, 345, 347-48. 349 graphics, 728 grass roots, 693

Index 769Great Tutor, 237, 239, 244 Green 
Programs, 427 Greendiag [NJ, 427 Grelling's paradox, 20-21, 22 Groot, Adriaan de, 286 grounds, excellent, 731 guanine, see nucleotides guaranteed termination, 41, 396-98, 399, 403 guitars, 62, 200 gullibility, 75-76, 106, 309, 461, 600, 701 Gutei, 237 G0025, white stony, 626  HACKER, 664 haiku, 153-54, 525, 619-20 halting problem, 74, 425-29, 594, 697 hamburger-confusion, 577 Hammurabi, 169 Hardy, Godfrey Harold, 562-66 harmonic tension, 122-23; see also tension and resolution harpsichords, 3, 391, 502 Harrison, Lawrence, 657 Haussmann, Elias Gottlieb, 2 HE-HE puzzle, 62-63, 669 headache, 62-63 hearing by computer, 602 "heart", in an All program, 679 Heisenberg uncertainty principle, 455, 698 Helen of Troy, 110 hemiolia, 257, 519 hemispheres, 257, 340-
41 Henkin, Leon, 541

41 Henkin, Leon, 541 
Henkin sentences, 541-53, 709; explicit and im plicit versions, 542-43 Henkin's Theorem, 488 hereditary arguments, 36, 47-48, 217, 261 heterarchies, 134, 359, 534, 651-54, 662, 691 heterological adjectives,20-21, 22 heuristics, 587, 588, 590, 603, 629 Hewitt, Carl, 662 hiccups, 116, 254, 255, 673. 725-26, 736 hierarchy of variability, 643-45, 669 high-fidelity vs. low-fidelity, 77, 85, 101, 406-7, 470. 697-98 high-level explanatory power, 707-9 Hilbert, David, 20, 23-24, 230, 459-60 Hilbert's program, 23-24, 229-30 Hilbert's tenth problem, 459-60 H(n), 137 Hofstadter, D. R., 75, 310, 724, 728, 742

Hofstadter s law, 152 Hogen, 248 holes in formalized systems, 24, 26, 449, 451, 465, 468, 470-71 holism: defined, 254, 312; vs. reductionism, 284, 311-36, 389-90, 708-9; Zen and, 254 Hubel, David, 341, 343 Human Condition I, The (Magritte), 705-6 Hyakujo, 254 hydrogen bonds, 516, 522, 525 hyphen-strings, 47, 64-65, 66 hypothetical worlds, 95-100, 338, 360-62, 634-40, 641-44; groundedness in reality. 362, 378-79 hypotheticals, 44-45, 634-40  i, 454 I, 454 "I", referent of, 608 "I Can Be Played (Proven, etc.)...", 488, 541 "I Cannot Be Played (Proven, etc.) , 76-77, 85, 406-7, 448, 465-67, 536, 541, 608 I-counts, 
260-61

260-61 
1-level, see inviolate level I-mode, see Intelligent mode iceberg, 495-96, 497 ideal numbers, 56-58 identification with artifacts, 609, 713-14 idiots savants, 566-67 IF-statements (BlooP), 411-12 images: blurry, 686-87; of thought, 623 "Imaginary Landscape no. 4" (Cage), 163-64, 699 imitation game, see Turing test Immunity Theorem, 536 implicit characterization, 41, 67, 72-73, 93 improvisation vs. introspection, 739 inaccessibility of lower levels to higher levels, 686-92, 706-10; in Aunt Hillary, 330-31, 630: in brains/minds, 302, 328-29, 362-65, 619, 677, 686-92, 697, 706-10, 739; in programs, 296, 300-1, 588, 630,

Index 770679; see also software and hardware, 
introspection, level-conflicts incompleteness: Bach and, 86; of brains, 585; defined, 86; Escher and, 716-17; of extensions of TNT, 465-71; of formal arithmetics, 18, 86, 101-2, 407, 618-19; of list of mathematicians, 422; of list of reals, 
421-24; of Lucas, 477; of phonographs, see record players, intrinsic vulnerability of; of Principia Mathematica, 18, 24, 618-19; of self-knowledge, 696-98; of TNT, 271-72, 430, 450-51; see also essential incompleteness, wincompleteness, etc. inconsistency: defined, 94; with external world, 87-88, 95; internal, 87, 88, 94-96; of people, 197, 697-98; of Tortoise, 177-80; see also consistency, contradictions, su-inconsistency, Zen increasing and decreasing rules, 73, 74, 260-61, 264, 269, 401-2, 407-x, 441; see also lengthening and shortening rules, chaos in number theory index numbers for programs, 418-20, 427-28 index triplets for supernaturals, 455 India, 549, 551, 557, 562-66 Indra's Net, 258, 359 
inducers, 545

inducers, 545 
infinite bundle of facts, 397-98 infinite coincidence, 398, 421 infinite regress, 111-13, 142, 146, 152, 231, 388-89, 426, 497, 738; in Carroll paradox, 43-45, 170, 192-93, 684-86. 693-94; halted, 127, 133-35, 170, 605, 684-86; of objectivity, 479; Zeno and, 31-32, 610; see also Carroll paradox, bottoming out, recursive acronyms, repeatability, etc. infinite sentence, 497 infinite sky, 401 infinitesimals and nonstandard analysis, 455 infinity: Bach and, 10, 719; Escher and, 15; handled finitely, 59-60, 221-25, 461-64, 468; illustrated, 135-36, 138-43; names of, 475-76; supernaturals and, 454; types of, 421; see also nontermination, infinite regress, recursion, etc. informal systems, see formal vs. informal systems information: accessibility of, see inaccessibility; creation of, 513; depth from

surface, 234-35, 409, 427, 549-58, 606-7, 612-13, 628, 673, see also decoding; discardable, 649, 653, 657-59. 669-72: flow of, 513, 533, 545, 547; irrelevant, 560 information-bearers, 158, 166, 167 information -reveale rs, 158, 267 inhibition, cellular, 544 inner messages, 166-71, 174-76, 501, 524 input-output devices, 288 input parameters (BlooP), 411 insight, 613, 660-61, 665-76 instant replays, straight and subjunctive, 634-40, 641, 672 instructions: in machine language, 289-95; vs. templates, 497-99, 531, see also programs vs. data INT(x), 138-41, 146, 661 intelligence: accidental inexplicability of, 707; essential abilities for, 26; extraterrestrial, see extraterrestrial intelligence; liftability of, see skimming off; limits of, 475-76, 679-80; necessary underpinning of, 324; simplicity of, 172-73; subtle features of, 566; tangled recursion and, 152; typical abilities of, 559; 
universality of, and intrinsic meaning, 158,

universality of, and intrinsic meaning, 158, 
162-64, 170-76, 501, 661-62; see also brains, minds, A1, etc. Intelligent mode, 38-39, 65, 193-94, 613-14 intensionality and extensionality, 337-39, 350, 361-62 intentions of machines, 684-85 interestingness, programmed, 615 interpretation-conventions, 687-88 interpretations: adjusted to avoid inconsistency, 87-88, 453, 456, see also undefined terms; multiple, 94-102, 153-57, 266-67, 271, 447-48; of pq-system, 49-53, 87-88, 101-2, 158; of Propositional Calculus, 186-87, 189, 191-92; of strands, 509-10; of TNT, 205-9, 266-67, 453, 533; of tq-system, C-system, P-system, 64-65, 73-74

Index 771interpreters: mechanisms in brain, 582-84; 
people, 293, 297, 524, 671; programs, 293, 504, 547, 616, 632, 662, 692 intrinsically high-level properties, 707-9 introspection, see self-monitoring, self-awareness, self-knowledge, inaccessibility, TNT. introspection of intuition, 560, 564, 613, 680, 713; programming of, 605, 609 inversion, 8-9, 81, 146, 681-83, 737-38; see also copies, complementary to original inviolate level, 686-92 irrationality vs, rationality in brain/mind, 575-78 itregularites, meta-irregularities, etc. 475-76 Iran, 254 ism, 254-55, 625, 704-6 isomorphisms: between Bongard problems, 660, 669; between brain-structures and reality, 82, 337-39, 350, 502, 569-71; between brains, 369-82; coarse-grained, 147-48, 503; in Contracrostipunctus, 83-85; between Crab's DNA and Crab Canon, 203, 667-68; defined, 9, 49-50; between earthworms. 342-43, 345; of emotions, 
163; exotic, prosaic, 159-60; fluid, 338,

163; exotic, prosaic, 159-60; fluid, 338, 
350, 362; between form and content in Dialogues, 84-85, 128-30, 204, 667-68; between formal systems and number theory, 408, 625; Godel-numbering and, see Godel isomor phism; between mathematicians, 566; between mathematics and reality, 53-60; between mental processes and programs, 568-73; between MIU-system and 310-system, 261-65; between models of natural numbers, 217; partial, 146-47, 371-82; as revelations, 159-61, 263; as roots of meaning, 49-53, 87-8, 94, 267, 337, 350; self-reps and, 501-3; between something and part of itself, 138-43, 146-47; between spiderwebs, 371-72; transparent, 82, 158, 267; on various levels between same objects, 369; between visual apparatuses, 345-46; in visual processing, 344; see also meaning,

translation, copies, decoding, etc.  'Jabberwocky" (Carroll), 366-68, 372-73 Jacquard loom, 25 Jaki, Stanley, 574 "Jammerwoch, Der", (Carroll-Scott), 366-68 "Jaseroque, Le" (Carroll-Warrin), 366-68 Jauch, J. M., 408, 409, 478-79 Jefferson, G., 598 Joan of Arc, 20 Johns, jasper, 703 J6shti, 233, 237, 238, 240, 253, 259, 272 JOSHU (TNT-string), 443 jukeboxes, 154-57, 160-61, 164, 170-71, 174-76, 500 jumping out of a subsystem, 477 jumping out of the system: in advertisement, 478; by answer-schemers, 462-64; Godel's Theorem and, see Godefization, essential incompleteness; illusion of, 478-79, 698; as method to resolve contradictions, 196-97; in political systems, 
692; 
by programs, 36-38, 476-78, 678; from 2-D to 3-D, see 2-D vs. 3- D; Zen arid, 255, 479;

692; 
by programs, 36-38, 476-78, 678; from 2-D to 3-D, see 2-D vs. 3- D; Zen arid, 255, 479; 
see also Godelization, Todelization, Escherization, TC battles, repeatability, nonprogrammability, etc.  Kaiserling, Count, 391-92 Kay, Alan, 662 Kennedy, John F., 641 keys, musical, 10, 299, 466, 501; see also modulation Kim, Scott, 68-69, 503, 523, 719 Kimberger, Johann Philipp, 9, 726 kitchen sink, the, 315 Kleene, Stephen C., 476 Klein bottle, 691 Kliigel, G. S., 91 knitting, 149-50 knots, 341-44, 272, 628 knowledge: accessible vs. inaccessible, 362, 365,

Index 772616, 619; encoded in ant colonies, 319-28, 
359; explicit vs. implicit, 617-18; modularity of, 615-18, 628; procedural v4. declarative. 363-65, 615-17, 630, 654; knowledge transplantation, surgical, 618 koans, 30, 189-91, 233-45, 246-59, 625-26; generated by computer, 625-26; genuine vs. phony, 234-35, 239, 242, 244, 427, 625-26 Kronecker, Leopold, 216 Kuhn, Thomas, 660 Kupfergiidel, Roman, 394 Kvogen,244-45  La Mettrie, Julien Offroy de, 3, 27, 729 labeling technique, 487-88, 540-41 Lambert, J. H., 91, 92, 99 lamp, meta-lamp, etc., 108-13, 216 language(s): acquisition of, 170, 294, 302; active meanings in, 51-52; Arabic, 623-24; of bees, 360; of the brain, 570; Chinese, 164, 665, 676; collage of, see scripts; computers and, 130-34, 300-1, 363, 586-93, 599-600, 601-3, 619-32, 674-75, 721; effect on thought, 376-77; English, 169, 372-73, 377, 379-80, 619-32, 674-75; flexibility of, 649, 674-75; French, 297, 
366-68, 372-73, 377, 501, 618; German,

366-68, 372-73, 377, 501, 618; German, 
366-68, 372-73, 380, 665; Hebrew, xviii, 377; hierarchy of, 22; imprecise, 674-5; invisible isomorphisms and, 82; Japanese, 169; as medium for proofs, 88-90, 195; necessary underpinning of, 324; partitions between, 671; procedural grammars for, 131-34, 619-32; reading meaning into computer-produced, 599-600, 625; on Rosetta stone, 165; Russian, 297, 379-80, 642; self-refs in, 431-37, 495-98, 501; see also meaning, translation, etc. Lashley, Karl, 342, 343, 348 "last step", 462-63, 468 lateral geniculate, 343-44 layers: of deception, 478; of messages, 166-71, 524, 703-4; of stability, 643-45 leakage, between levels of science, 305-6 Legendre, Adrien-Marie, 92 Lehninger, Albert, 504 Leibniz, Wilhelm Gottfried,

24-25, 600 lemmas, 227 Lenat, Douglas, 615 lengthening and shortening rules: decision procedures and, 48-49, 182, 407-8; MIU-system and, 39-40, 260-61, 264, 613; TNT and, 213, 266, 269; see also increasing and decreasing rules, problem reduction Leonardo of Pisa, see Fibonacci Lermontov, Mikhail, 642 level-conflicts: in Aunt Hillary, 330, 630; in mind/brain, 575-78; in messages, 164, 170, 699-704; between object language and metalanguage, 194, 449-50; in SHRDLU, 630 level-con fusion: ants and, see ants vs. ant colonies; in art, see 2-D vs. 3-D; authorship and, 3. 608, 720-26; in computer systems, 287, 291, 295, 300-2, 308; of Kimian self-rep, 503; minds/ brains and, 287, 575-77; in Propositional Calculus, 185, 194; subjunc-TV and, 608; self and, 709 level-crossing, in thought, 666, 668 level-mixing in genetics, 509-10, 513-14, 
546-48 level-shifting, conceptual, see

546-48 level-shifting, conceptual, see 
abstraction, levels of levels: of computer languages, 290-99; distinct vs. similar, 285, 287; in Escher, 11-15, 6 levels (continued) 715-16; haziness of, 13-15, 54618, 715-16; intermediate, 302-3, 317, 324, 532, 632; of irreality, 243, 641; of MU-picture, 311-13, 328-29, 525-26; of partides, 305; in radio news, 128; of reality, 15, 103-25, 128-29, 184-85, 481, 493, 640, 725-26, 737, 739; in recursive processes, 128-29; of rules in thought, 26-27 levels of description: of ant colonies, 315-33; of brain, 349-50, 382. 559, 570-77, 584-85; of caste distribution, 319-29; of chess boards, 285-86; of errors, 294-95; of gases, 308; of human body, 285; of human psyche, 287; of mental processes, 568-73, 575-78, 584-85; of programs, 294-95, 380-

Index 77381; of television screen, 285; see also 
holism vs. reductionism levels of meaning: in ant colonies, 319-27; in Contracrostipunctus, 82-85; of DNA, 160, 531-32, 665; in Epimenides paradox, 496, 581, 584-85; of groove-patterns, 83-84; of Mumon, 248; of MUMON, 266-67; of music, 162-63; of neural activity, 575-77; of TNT-strings, 266, 270-71 levels of structure: of enzymes, 510-11, 519, 521, 525-27, 532; of music, 525 liar paradox, see Epimenides paradox Liberation (Escher), 57-58, 65 lightning calculators, see idiots savants limericks, 483, 736 limitative results, in general, 19, 74, 609, 697, 699 Lincoln, Abraham, 454 lines, geometrical, 19-20, 90-93, 100, 222, 452, 456 LISP, 293, 381, 626, 652, 692, 738-39 List of All Great Mathematicians, 404, 422 Little Harmonic Labyrinth (Bach), 121-23, 129, 130 Little Harmonic Labyrinth (Dialogue), 127, 128-30, 149, 216, 610-11 Little Harmonic Labyrinth (of Majotaur), 
119-25 Littlewood, J. E., 564

119-25 Littlewood, J. E., 564 
lizards, 108-9, 110, 115-17, 125 Lobachevskiy, Nikolay, 91 local vs. global properties, 21, 160, 359, 363, 371-75, 543, 582-84, 678 localization of knowledge, in brains and programs, 342, 348, 365, 617-18 Lockwood, Anna, 700 logic, 19-24, 43-45, 99-100, 177-80, 181-97, 461-64, 618-19 Loocus the Thinker, 477 look-ahead trees, 151, 604-5, 611, 712 loops: bounded, 149, 410-14, 418, 440-41, 444; free, 149, 424-25; in music, 150; in programming, 149-50, 410-14, 424-25, 503, 632 lottery, 639-40 Lovelace, Lady Ada Augusta, 25, 307, 598 lower levels, see substrate, mental lowest-level rules embodied in hardware,

685-86 Lucas, J. R. 388-90, 471-73, 475, 476, 477, 574, 577-78, 597 Lucas' argument: counterarguments to, 475-77, 577-78; merits of, 472; summarized, 471-73 Lucas sequence, 139, 152, 174  M-mode, see Mechanical mode MacGillavry, Caroline, 667 machine dependence and independence, 294 machine language, 289-300, 306, 381, 547 machines: not the sum of their parts, 389-90; reflecting on themselves, 288-89; self-assem bling, 160, 486, 504, 543, 545 MacLaine, Shirley, 285 macroscopic effects from micros 307 copic causes, MACSYMA, 615 Madstop, 727 magnetic field and crystal, 140-43 Magnficat in D (Bach), 549, 552, 558 hfagn#iicrab, Indeed, 560, 574, 581 Magritte, Rene, 480-81, 489, 493-94, 627, 
700-2,

700-2, 
705-6; paintings by, see List of Illustrations (xiv xviii) Mahalanobis, P. C., 565 main theses of book, 26, 46, 559, 714 Majotaur, 119-21, 123-25 malaphors, 657 Mandelbrot, Benoit, 71 manifestations of symbols, 351 Mao Tse-tung, 433 mappings: charted, 85, 449, 533, 536; induced, 668-69, 671-72 marbles, rolling, 711-12 Margie-balloon story, 675 Materialism, champions of, 27, 729 mathematical logic, history of, 19-24 mathematical view of brains, 559 mathematicians, 458-59, 559, 566, 614 mathematics: done by computers, 573, 602, 614-15; foundations of, 19-24; reality and, 54-58, 456-59 Mathews, Max, 607-8 McCarthy, John, 293 McCulloch, Warren, 134

symbols, 325, 327, 350; carried only on symbol level, 324-27, 330, 330, 709-10, codes and, 82, 158-62, 164-67, 267; of Contracrostipunctns, 82-85; of DNA, 160, 531-32, 665; explicit vs. implicit, 82-85, 158-76, 495-500, 583; in formal systems, see interpretations; intelligence and, 158, 162-64, 170-76, 501, 661-62; intrinsic, see meaning, explicit vs. implicit; location of, 153-57, 158-76, 408-9, 582-84; as multidimensional cognitive structure, 582-84; multiple, 8, 10, 52-53, 82-85, 94-102, 153-57, 158, 172, 266-67, 271, 409, 447-48, 524, 532, 666, see also disambiguation; in music, 83, 160, 161, 162-64, 167, 172, 174-75, 227, 582-84, 626-27, 676-77, 699-700, 704; objective, see meaning, explicit vs. implicit; as optional high-level feature, 571; passive vs. active, 51-52, 94, 97, 100, 102, 191-92, 266, 267, 271, 456; purpose and, 321-32; rooted in isomorphisms, 49-53, 87-88, 94, 267, 337, 350; unnecessary on evolutionary time scale, 321-22 meaningless vs. meaningful interpretations, 51, 88

meaninglessness, in art and music, 699-
700, 704-3 meat grinders, 414 Mechanical mode, 38-39, 65, 194, 221, 613-14 mechanization of thought processes, see Al, formal systems, etc. meiosis, 665, 672 melodies: recall of, 363-64; time-shared, 385 memory, in computers, 288-89, 546, 616 memory dump, 381 men vs. women, 477, 595-96 Mendel, Arthur, 3, 28 Mental Arithmetic (MagrittO, 627 mention, see use vs. mention Menzel, Adolph von, 4-5 Meredith, Marsha, 625 Mergenthaler, Otto, 630 message-L   sing languages, 662-63 messages, 154, 158-76: in ant colonies,

350; in bottles, 167-69, 524; layers of, 166-71, 524, 703-4; from nature, 408-9; see also frame message, inner message, outer message messenger, for koans, 235-36, 238 messenger RNA, see mRNA meta, 216-17, 224 meta-agnosticism, 114 rneta-analogy, 673-74 meta-answer-schema, 463 meta-author, 607-9, 726 metabook, 22 meta-descriptions, 656-57, 674 meta-evidence, etc., 693-94 Meta-Genie, see Genie meta-hiccups, 726 meta-intuition, 605 META-JOSH1-), etc., 443 metaknowledge, 364 metalanguage, 22, 184, 194. 248, 270, 514 metalogic, 23, 676 metanrathematics, 23, 579; reflected inside TNT, 449-50 Metamorphosis (Escher). 14-15 metaphase, 666-57 metaphor, 672 meta-proteins, 533-34 metarules, etc.: in chess, 687-88; in intelligence, 26-27, 559, 684-85 
meta-search, 397 meta-symbols, 560

meta-search, 397 meta-symbols, 560 
metatheorems, 193-94 metatheorv, formalized, 194 meta-TNT, etc., 442-43, 533-34 meta-wishes, see wishes meteorites, 167, 172 methylation, 540-41 metric, mental, 613, 614; see also conceptual nearness Meyer, Leonard B., 167, 704 Michelangelo, 642 microprogramming, 295 minds: overlap of, 376; programmability of. 302, 679, see also Al, paradox of All, Tesler's The orem, nonprogrammability; thoughts and, 369-90; two ways of creating. 390; vs. brains, 309, 575-77; see also brains, intelligence, etc. mini-vocabulary, 647 Minsky, Marvin, 373, 388, 644, 679, 722 mirroring, see isomorphisms, representation misspelling and computers,

Index 775297-98 MIU-numbers, 264-67; see also 
theorem-numbers MIU-system, 33-41, 46, 47, 48, 52, 191, 260-67; as model for TNT, 439-42, 466; table of rules of, 260 MIU+MU-system, 466 Mobius Strip I (Escher), 29-30 Mobius Strip II (Escher), 276 modes of fugue-listening, 282-84 modularity, 149-50, 615-18, 628, 677-78; see also localization, local vs. global properties modulation, 10, 121-23, 129-30, 466, 501, 717, 739 modules in brain, see symbols Modus Ponens, see detachment molecular biology, 504, 514-48 Mondrian, Piet, 700 monkeys, vision of, 345-4b Monod, Jacques, 161 morphogenesis, 539, 543-44 Mosaic II (Escher), 61-63 mountain-car wreck story, 338-39, 361, 365 Mozart, W. A., 649, 702 mRNA, 517-20, 522-25, 527-28, 530-33, 536, 545, 547, 662, 663 MU, as possible theorem of MIU-system, 33-41, 
229, 259-61, 265-67, 271, 708

229, 259-61, 265-67, 271, 708 
MU, Zen word, 233, 241, 246, 254, 259, 272, 311, 312-313, 328 MU-LOOP (Floc P), 424-25, 441 Mu Offering, A, 272, 628 muoperator, 424 MU-picture, 283-84, 310-13, 327, 328-29, 525-26 MU-puzzle, 33-41, 259-61, 509-10, 613-14 multifurcation of TNT, 467 multiple representations, 616-18, 670-71, 674 multiplication, 54-56, 64-65, 206, 409, 455, 566, 567 Mumon, 242, 246, 248-49, 253, 259, 260, 272; commentaries by, 246-49, 252; poems by, 246-49, 252, 272 MUMON, string of TNT, 265-67, 271, 441-42 Mumonkan, 246 music: composed by computer, 25, 595,

597, 603, 607-9, 626-27, 676-77; dimensions of, 175; mathematics and, 227, 555, 560; modern, 156-57, 163-64, 174-75, 699-700, 704; notation of, 552-59, see also SMUT; semantics of, 83, 162-64, 167, 174-75, 582-84, 626-27, 676-77; superhuman comprehension of, 172, 679; syntax of, 121-23, 129-30, 227, 626-27, 737; to break phonographs by, 75-78; to infiltrate phonographs by, 487-88; see also fugues, canons, pianos, flutes, etc. music box, preprogrammed, 677 Musical Offering, The (Bach), 4-10, 86, 665, 666, 719, 720, 724, 727, 739-42 mutations, 295   N, see number theory Najunamar, Z., 549-52 Nansen, 248-49, 253, 255 natural language utterances as programs, 
629 natural numbers: defined, 54, 204;

629 natural numbers: defined, 54, 204; 
generalized; 453-56; postulates for, 216-17; see also number theory, numerals, TNT, prime numbers, etc. near misses, see "almost"-situations nearly decomposable systems, 303-6 negation, 70, 71, 183, 191-92, 210-11, 214, 545 negative space, 62-63, 66-68, 72; see also figure and ground nested movies, 184-85 nested works of art, 15, 106, 700-1, 705-6 nesting, 127, 138-41, 184-85, 660; see also recursion neural networks, see symbols Neuroneater, 382 neurons: compared with ants, 315, 325, 339-40; described, 339-40; Eudid's, 60; faultless func

Index 776tioning of, 575-77; firing of, 83, 340, 316, 
343-45, 347, 350, 357; as an inviolate level, 302, 677, 686, 691-92, see also inaccessibility; not controllable consciously, 302, see also inaccessibility; on-center and off-center, 343-44; in retina, 343-44; simple, complex, hypercomplex,- neurons (continued) 344-45, 346, 347; as summing inputs, 316, 340, 575-77, 677 neurosurgery, 309, 313-14, 618. 678 New Yorker, The, 641-42 nickelodeon, 500; see also jukeboxes nodes and links, 370-71, 652-54 noise in vacuum, 82 nondivisibility, 73-74 nonequilibrium thermodynamics, 693 Noneudid, 91-92 nonexistence, 254-55, 698, 725; see also Tumbolia nonproducible numbers, 265 nonprogrammability: of creativity, 570-71, 620, 673; of emotions and will, 677, 684-86; of Godelization, 472-76; of intelligence, 26-27, 471-73, 597-99, 601; of irrationality, 575-77; of jumping out of the 
system, 37-38, 477-78, 674-75; of ordinal

system, 37-38, 477-78, 674-75; of ordinal 
names, 476; of soul, 574-75; of world chess champion, 151-52, 674; see also people vs. machines, essential incompleteness, Todeization, paradox of Al, TC-battles, 2-D vs. 3-D, etc. non-self-assembling viruses, 542-43 non-self-descriptive adjectives, see heterological adjectives nonsense: based on sense, 378-79; computer-generated, 620, 621-22, 625-26; human-generated, 621-22 nontermination, 408, 425-30; see also potentially endless searches, Hoop nontheorems, see theorems vs. nontheorems normal science, 660-61 nouns, most common in English, 630 novelty, and jumping out of the system, 475 nuclei: atomic, 303-4; cellular, 514, 517,

518 nudeotides, 514-17, 519, 522-24, 530, 540-41; first letters of, 231, 517, 666 number theory: applications of, 278-29; core of, 100, 407; Crab and, 551-58, 560, 562, 573-74, 579-81; demise of, 228-29, 426; formalized, see TNT; informal (N), 54-660, 204, 228; nonstandard, 100, 452-59; primitive notions of, 204-9; as scaled-off mini-world, 569; soothing powers of, 
391-404; "true" version of, 458-59; typical sentences of, 204-5; typographical, see TNT; as universal mirror of formal systems, 260-65, 270; used and mentioned, 458 numbers, nature of, 54-58, 452, 458 numerals, 205-6, 213; vs. numbers, 264  object language, 22, 184, 248 objectivity, quest after, 479, 693-96 Oborin, Lev, 162 octopus cell, 345 Oin, Eta, see Eta Oin Oistrakh, David, 162 Okanisama, 232, 234, 237, 238, 239, 241, 
242 Old Ba. Ch., 726

242 Old Ba. Ch., 726 
Old Bach, 4, 28, 460, 481-83, 738, 739 ca-consistency, 459; see also or-inconsistency cs-incompleteness, 221-22, 421, 450-51 co-inconsistency, 17, 223, 453-55, 458-59 1-D vs. 3-D, 519-21, 616-17 open-ended searches, see potentially endless searches, nontermination, unpredictable but guaranteed termination, loops, free, FlooP, etc. operating systems, 295-96, 300-31, 308 operators and operons, 544-45 oracles, 567 orchard analogy, see information, depth from surface order and chaos: in ant colonies, 316-17; in number theory, 393, 395, 398-402, 406. 408-9, 418; self-awareness and, 406 Order and Chaos (Escher), 399 ordinals, 462-64, 475-76 organ point. 329-30 origin of life, 548

Index 777original (as opposed to copies), 504 
originality and machines, 25-26, 606-9 ORNATE NOUN, 131-33 outcome, 184 outer messages, 166-71, 174-76, 501, 524, 704 OUTPUT (BlooP), 410, 411 overlapping genes, 524-25 overview capacity, 613-14, 678; see also jumping out of the system P-system, 64, 73-74 padding, 402-3 pages, in computers, 289 palindromes, in molecular biology, 201, 667 Palindromi, 353-54, 634-37, 643, 644 Pappus, 606-7 paradigm shifts, 660-61 paradox: of Al, 19, 26-27, 620, 673, see also Tesler's Theorem; in an, see Escher, Magritte, Cage; of credibility through fallibility, 564; of God and the stone, 478; in mathematics, 17-24, 580-81; of motion, see Zeno's paradoxes; near misses, 612, 691; resolutions of, 116, 196-97, 245, see also MU, Tumbolia, jumping out of the system; of self-consciousness, 389; of the Typeless Wish, 115-16; in Zen, 249-
55; see also contradictions, inconsistency

55; see also contradictions, inconsistency 
parallel postulate, see fifth postulate PARRY, 300-301, 599-600, 677 parsing of natural languages, 588-93, 630-32; see also grammar, language partial recursivity, 430 particles, elementary, 54, 140-46, 258, 303-5, 309,522 partitions, mental, 671 parts, 303-5; see also reductionism Pascal, Blaise, 24, 25, 600 pathways: in ATN's and RTN's, 131-34, 150; chemical, 528-29, 544-45, 663-64; conditional on circumstances, 383-84; goal-oriented choice of, 227, 609-15; as incorporating knowledge, beliefs, 378-79; morphogenetic of T4, 539; plausible vs. implausible, 383; potential, in brain, 281 pattern recognition, see Bongard problems,

conceptual skeletons, vision by computers patterns on all levels, 674 Peano, Giuseppe, 20, 216-17 Peano arithmetic, 100 Peano postulates, 216-17, 224 pearl and oyster, 17, 438 Penfield, Wilder, 342-43 Penrose, Roger, 15 people vs. machines, 25-27, 36-38, 151-52, 388-90, 471-73, 475-77, 559-62, 567-75, 577-79, 595-99, 606-9, 621-23, 680, 684-86 peptide bonds, 523 perception: visual, 97-98; and Zen, 251 Perfect items, 3, 75-79, 85, 406, 424, 486, 536 perfect numbers, 416, 418 phages, see viruses phenotype, see genotype and phenotype 0(174, 176, 524-25 Phonographs, see record players phonons, 304 photocopy machines, 499 photons, 142-46, 258 physics laws of: as basis for choosing between rival mathematical theories, 100, 456-57; as blocking infinite regress, 170, 685; as formal system, 53-54; inconsistency and, 95-96, 99, 
584-85; intuitive, 362-63, 711; levels and,

584-85; intuitive, 362-63, 711; levels and, 
303-5, 307-8, 693; no escape from, 477, 575; Reductionist s Dilemma and, 522, 709; underlying consciousness, 575, 685, 710 IT, 277, 306, 408, 415, 421, 546, 568, 605, 673, 728 piano postulates, 552-53 pianos, 3-4, 302, 305, 633-34, 700, 726; inverted, 681-83 Pickruick papers, The (Dickens), 24. 326, 595, 598 pinball machine, 307 pipe Dream, 703 pipes, 480-82, 486, 488, 493-94, 521, 638, 701-3 planets and satellites, 353-54 PLANNER, 629-32 plurals, 354 pocket calculators, 568-70, 616, 678, 710 pointers in computers, 289-90, 619

Index 778points (geometrical), 19-20, 90, 92-93, 100, 
207, 222, 452, 456 Polanyi, Michael, 574 polarons, 304-5 polypeptides, 523, 525, 528 polyribosomes, 526-28 Pons Asinorum proof, 606-7, 669 Pool B, 418 Pool F, 427 popcorn, 104, 124-25 popping, 127-35, 184-85 popping-tonic, 105-6, 116-17, 125 porridge, 431 ports of access, 670-71 Post, Emd, 33 post-ending endings, 392, 403 postal system metaphor, 663 postulates of geometry, 90-91, 92-93, 407 potentially endless searches, 396, 400-402, 425, 400-401, 444, 582-83 pq-system: completeness and consistency of, 101; decision procedure for, 47-49; expressive weak ness of, 101, 221-22, 407, 417; horse-apple happy interpretation of, 51, 88, 215; isomor phisms and, 49-53, 158, 625; modified, 87-
88,

88, 
92-93, 102; surprise interpretation of, 52-53, 94 predicate calMus, 609 predicates, number-theoretical, 208-9 predictable termination, 400, 407, 409-18, 420, 441, 582; see also unpredictable but guaranteed termination, terminators Prelude, 337, 383, 460, 686, 737, 739 preludes and fugues, 280-84, 335; see also fugues, Well-Tempered Clavier premise, 184 Preprocessing, 647, 650, 659 President v. Supreme Court, 692 primary structure: of proteins, 519-22; of ty poenzymes, 511, 512 prime numbers, 58-59, 64-67, 72-74, 149, 211-12, 413, 551-58, 615; differences of,

393, 395-98, 400, 416; sums of, 393-96, 400, 414 primitive recursive truths, 407 primitive recursivity, 406, 407, 414-20, 422, 424, 429-30, 440, 441, 444, 451, 466, 472 Principus Mathematiaa, 18-19,• 21, 23-24, 228, 618-19 Print Gallery (Escher), 15, 714-17 printer of computer, 301, 307 problem reduction, 609-13; self-applied, 613 problem spaces, representation of, 611-13 procedures, 132-34, 150-51, 292, 410-15, 418-20, 424-28; chains of, 413-14, 415, 418 processors (computers), 504, 513, 547; see also. central processing unit producible numbers, 264-65, 269-70 program space, 299 programs: in Analytical Engine, 25; in BlooP and FlooP, 410-15, 424-26; Blue, Green, Red, see Blue, Green, Red programs; chess-playing, see chess programs; constructed by programs, 589, 
629-32, 664; as data, 293, 692; for

629-32, 664; as data, 293, 692; for 
determining enzyme function, 521-22; for determining phenotype, 532; for determining tertiary structure, 521-22; families of, 503, 546; for generating theorems, 471-73, 578, 615, 617-18; high-level comparison of, 380-81; for naming ordinals, 476; recursive structure of, 149-50; secondorder, third-order, etc., 476; self-modifying, 152, 692; self-reproducing, 498-504, 547; for translating programs, 291-94; vs. data, 499, 513, 531, 546-48, 616-17, 630, see also use vs. mention; vs. programmers, 306, 734-37; see also computers, computer languages, Al programs, etc. Prokofiev, Sergei, 150 pronoun reference, 587, 591, 592 proof-pairs, 416, 438-43, 446-47, 450-51, 452-54, 466, 468, 469

Index 779proofs: nature of, 18-24, 58-60, 88-93, 192-
97, 227-28, 458-59, 578, 707-8; as never absolute, 191-94; of proofs, 192-93; vs. derivations, 35, 193, 194-95 prophase, 665-666 Propositional Calculus, 181-97; embedded in TNT, 195, 197, 207, 215-17; as an epiphenomenon, 578; interpretations of symbols of, 186, 189, 191-92; rules of inference, justified, 188-89; rules of inference, presented, 181-87; rules of inference, table of, 187; streamlined, 193-94; variants of, 195; ways to improve, 193-94; 196-97; weaknesses of, 195-97, 578; well-formedness in, 181-83 proteins, 517-18, 544-45; as procedural knowledge, 616-17; as programs, data, interpreters, processors, 547; see also enzymes prototype principle, 352 provability, 18, 101 Prudence and Imprudence, 191-92, 229 pruning, explicit vs. implicit, 286 pseudo-epigenesis, 531-32 psychic powers, 693; see also ESP Ptolemy V Epiphanes, 165 
Puddle (Escher), 256

Puddle (Escher), 256 
pulling-out (mechanisms), see decoding (mechanisms) punctuation, 33, 268, 440, 510, 512, 520, 524-25 purities, 506-7, 514, 516, 534; see also nudeotides, bases, base-pairing purposeful vs. purposeless behavior, 320-22 pushcorn, 124-25 pushing, 127-34, 184-85 pushing-potion, 105-6 Pushkin, Alexander, 124 puzzles. 8, 33-35, 62-63, 67, 73, 137, 182, 212, 215, 220, 401, 415-17, 425, 442-43, 444, 512-13, 564-565, 609, 621-23, 646-60, 689 pyramidal family of theorems, 221-25, 450-53 pyrimidines, 506-7, 514, 516, 534: see also nu cleotides, bases, base-pairing Pythagoras,

418, 556-57  Q(n), 137-38, 152, 265, 409 quantifiers, 207-9, 210, 211-12. 214, 217-19 quantum mechanics, 19, 54. 140-46, 350, 455, 457, 699; see also partides Quantz, Joachim, 4 quarks, 304, 305, 350 quasi-isomorphisms, see isomorphisms, fluid quatemarv structure, 525 Questions and Speculations, 676-80 Quine, Willard Van Orman, 435, 446, 449, 699 quining, 431-37, 445, 446, 449, 497-99, 531 QUIT (BlooP), 412 quotation, 426, 431, 433-37, 496-97, 702, 738 quotation marks, 33, 434, 498, 499, 702  Rachmaninoff, Sergei, 150 RACRECIR, 738 radio broadcasts, 128, 163, 169, 353, 478, 545, 720 Ramanujan, Srinivasa, 562-66, 614 randomness, 408-9, 620, 673, 712 rational and irrational numbers, 140-42, 418, 452, 
556-57

556-57 
rational vs. irrational, in human head, 575-77 rats in mazes, 342 Rauschenberg, Robert, 703 r.e. sets, see recursively enumerable sets reading frame shift, 154, 525 reality, nature of, 409 rearrangement of parts, 78, 333-35, 484; see also self-assembly reasoning about formal systems, 38-39, 66, 260-72, 438-52, 465-71, 579-81 reasoning by programs, 569-70, 577-78, 586-93, 606-7, 609-11, 614-15, 618-19, 628-32 recognition: molecular, 540-41; visual, 346-48, 646-62; vs. production, 648-49 recognizable forms, 68 recombination, 657, 665-69

EpsilonZero, 486; family of, in Crab's jukebox, 154-57; Grand Self-assembling, see record player Epsilon-Zero; as information-revealers, 158-61, 164; intrinsic vulnerability of, 75-78, 102, 424, 470, 483-86, 536, 543, 584, 721, see also Tbdelization, TC-battles, etc.; likened to formal systems, 84, 85; low-fidelity, 77, 85, 101, 406-7, 470; Omega, 78, 468, 483-84; Numbers l, 2 ... etc., 76-77; Tortoise-chomping, 483, 487-88; two-channel monaural, 634, 669; see also jukeboxes records: defective, 102, as information-bearers, 158, 160-61, 164; as labyrinths, 120-24; with multiple melodies, 154-57; as phonographbreakers, 75-78, 83-85, 271, 406-7, 424, 469, 484, 486, 536, 543, 584; smashed, information in, 161; in space, 162-64, 172, 174-75; of WellTempered Clavier given to Crab, 275, 278-80 records and record players, likened to cellular constituents and cells, 83, 158-64, 167, 175, 536 recursion: avoidance of infinite regress in, 127, 134-35; avoidance of paradox in, 127; 
defined, 127-29, 131-35; elementary

defined, 127-29, 131-35; elementary 
particles and, 142-46; fantasy rule and, 184-85; in game-playing programs, 150-51, 604-5; indirect, 134, 137; in language. 130-34, 588, 591, 592; in music, 121-23, 129-30; and unpredictability, 152; see also nesting, levels, distinct vs. similar, level confusion, etc. recursive acronyms, 113, 133, 134-35, 738, 742 recursive diagrams, 135-37 recursive figures, 67-70. 72, 73 recursive formula, of thinking, 560 recursive functions, 136-40, 152, 430, 455; see also genera] recursivity, primitive recursivity. BlooP, HooP recursive graphs. 138-43 recursive sequences, 135-38, 139 recursive sets, 72-74, 152, 191 recursive structure of ideas, 386-87, 560,

621, 644-45, 650, 656-57, 669. 671-72 Recursive Transition Networks, 131-34, 136, 145, 150, 620-21 recursively enumerable sets, 72-74, 152, 191, 265, 269 recursively related notation-systems, 475 Red Programs, 427-28 Reddiag [NJ, 428, 429 redness, subjective and objective, 710 reductionism: defined, 312; proteins and, 520-22; see also holism vs- reductionism, sealing-off Reductionist's Dilemma, 522, 709 reentrant code, 387 refrigerators, see record players, low fidelity registers, in computers, 289 relativity, 19, 96, 100, 680 Relativity (Escher), 97-98 relevant implication, 197 renormalization, 142-46, 258, 304-5, 309 repeatability, see Godelization, Todelization, diag onal method, Escherization, TC-batteries, answer-schemers 
representability, 407, 417-18, 430, 441,

representability, 407, 417-18, 430, 441, 
443, 444, 451, 466, 468, 579-80 representation of knowledge: in All, 569, 615-21, 626-32, 641-59, 664-65, 668-72; in brains, see symbols, localization repressors, 544-45 Reptiles (Escher), 116-17 Requirement of Formality, 33, 52, 65 retrogression, 8-9, 81, 146, 200, 208, 500-501, 549, 666-68, 723-25, 737-38 return addresses, 128, 133 revelation, 160-61, 175 ribo, some, 236 ribonucleic acid, see mRNA, rRNA, tRNA ribosomal RNA, see rRNA ribosomes: as models for Al, 662, 663; molecular canons and, 527-28; need for in DNA's self rep, 530; origin of, 528, 548; as self-assembling

Index 781objects, 485-86, 542; structure of, 528; as 
translators of Genetic Code, 485, 518-19, 522-25, 547; in Typogenetics, 512 ricercar, defined, 7 RICERCAR (E), 7, 727-42 Ripplad Surface (Escher), 256-57 RNA, see mRNA, rRNA, tRNA RNA polymerise, 527, 530, 544 robot in T-maze, 711-13 Rogers, Hartley ropes, thin and thick, 229-30 Rose, Steven, 342 Rosetta stone, 165, 166 Rosaak, Theodore, 574 Rousseau, Henri, 680 Royal Theme, 4-10, 96, 719. 739-40 rRNA, 528 RTN's, see Recursive Transition Networks rule-less systems, 598, 685; see also formal vs. in formal systems rules: arithmetical vs. typographical, 262-64. 269; flattened into strings, see theorems vs. rules; intelligence and, 26-27, 559, see also brains and formal systems rules of inference: of C-system, 65; compared with enzymes, 509-10, 513, 531; defined, 34-35; derived, 193-94; of MIU-system, 34, 260; 
of P system. 74; of pq-system, 47;

of P system. 74; of pq-system, 47; 
proposed, 66, 221; of Propositional Calculus, 187; recursive enumerability and, 152; run backwards, 48-49, 182; of 310-system, 263; of TNT, 215, 217-20, 223-25; of tq-system, 65; of Typogenetics, 509-10  rules of production, see rules of inference run-of-the-mill sets, 20-21 Russell, Bertrand, 18-24 Russell's paradox, 20-21, 685   Saccheri, Girolamo, 91-93, 99, 452, 456 Sagredo, see Salviati, et al Salviati, Simplicio, Sagredo, 408-9, 478-79, 673, 694 sameness: of ASU's, 375; of BACH and CAGE, 153-57; in Bongard world, 650-53, 657,

660, 664; of butterflies, 147, 369; of demi-doublets, 669; elusiveness of, 14619; of Escher drawings, 147; of human and machine intelligence, 337, 379, 679-80; of human minds, 341-42, 369-72, 375-77, 382; intensionality and, 338; mech anisms underlying perception of abstract, 646-62, 665-69, 671-72; overlooked, 614, 674; of programs, 380-82; in self-refs and self-reps, 500-4; of semantic networks, 371; of translations between languages, 372, 379-80; universality of intelligence and, 158, 501; vs.differentness, 153-57; visual, 344-48, 662; see also copies, isomorphisms, conceptual mapping sameness-detectors, see Sams Sams, 650-53, 657, 664 Samuel, Arthur, 604-5, 684-86 Samuel's argument, pro and con, 684-86 San Francisco Chronicle example, 351 sand castles, 725-26 
sanity vs. insanity, 192, 696 satellite-

symbols, see splitting-off satoti, see enlightenment scale, cyclic, see Shepard tones Schmidt, Johann Michael, 27 Schnirelmann, Lev G., 394 Schdnberg, Arnold, 125 Schrodinger, Erwin, 167 Schweikart, F. K., 92 science: and Bongard problems, 659-61; self-applied, 699 Scott, Robert, 366 scripts, collage of, 168-69 sealingoff, 305, 309, 350, 534 secondary structure, 521, 525 self, nature of, 316-17, 327-28, 384-85, 387-88, 695-96, 709-14 self-assembly, spontaneous, 485-86, 542-43 self-awareness, 406, 479, 573 self-descriptive adjectives, see autological adjectives self-engulfing, 489-94; failed;

Index 782490, 492; total, 493 self-knowledge, 
possibility of, 696-98, 706 self-modifying games, 687-88 self-monitoring, 328, 385,-387-88, 697, 713 self-perception, 695-98; vs. self-transcendence, 478 self-programmed objects, 685-86, 691-92 self-proving sentences, 542-43 self-quoting sentence, 426, 496-97 self-reference: Bach and, 86; banning, 21-23; as cause of essential incompleteness, 465, 470-71; focusing of, 438, 443, 445-48; Gddelian, 17-18, 271, 447-49, 497, 502, 533, 667, 738; indirect, 21, 85, 204, 436-37, 502, 667, 738-39; many leveled. 742; near miss, 437; Quine method, 431-37, 445-46, 449, 497-99, 531; by translation, 502 self-reference and self-replication, compared, 530, 533-34, 541-43 self-referential sentences, 435-37, 477, 495-99, 501 self-rep: by augmentation, 503; canons and, 501, 503; differentiating, 546; epigenesis and, 160; by error message, 503; inexact, 500-503, 546; by retrograde motion, 500-
501; by translation, 501; trivial, 499;

501; by translation, 501; trivial, 499; 
typogenetical, 512-13 self-snuffing, 701-2 self-swallowing sets, 20 self-symbol, 385, 387-88, 709; free will and, 710-14; inevitability of, 388 self-transcendence, 477-78, 479 self-unawareness, irony of, 328, 330, 331, 630 semantic classes, 621, 630 semantic networks, 370-72; see also concept network semi-interpretations, 189, 196 semiformal systems, 216; see also geometry, Euclidean senseless loops, 679 sentences in TNT, 208-9 Sentences P and Q, 436-37 sequences of integers, 73, 135-39, 173-74, 408 set theory, 20-23 sets F and G, 73 1729, 204-5, 210-11, 345, 393, 551, 564-855 Shadows, The (Magritte), 480 Shakespeare, Wm., 96, 595, 598, 608, 736

Shandy Double-Dandy, 611 shared code, 387 Shepard, Roger, 717-19 Shepard tones, 717-19 shielding of lower levels, see inaccessibility SHRDLU, 586-93, 599, 627-32, 674 Shuzan, 251 Sierpifiski, W., 404 signals, crisscrossing, 322-23 signature, visual, 347-48 Silberescher, Lowen, 394 Silbermann, Gottfried, 3, 4 silver, 173 Simon, Herbert A., 303, 305 simple, complex, hypercomplex cells, see neurons Simpficio, see Salviati simplicity, 172, 560, 615 simulation: of entire brain, 572-73; of neural networks, 571-72 Six-Part Ricercar (Bach), 4-7, 719, 739-42 skater metaphor, 412-13 skeletons (recursion), 140-41; see also bottom 4 skimming off top levels, 309, 325, 326, 358-59, 568-79 slinky, 337 slippage, conceptual, 633-40. 641-44, 654-
56,

56, 
672; see also conceptual mapping Sloth, 633-40, 643. 681-83, 722 Sloth Canon (Bach), 9, 666, 683 Sloth Canon (Dialogue), 738 SLOTHs, 684-719 slots, 645, 650-53, 656-57, 668 Smalltalk, 662 smart-stupids, 721-42 Smoke Signal 67, 702 SMUT, 6, 80, 155-57, 202, 682, 718, 740 soap cake, 497 soft-louds, see pianos software and hardware: in brain, 346, 356-57, 686, 709; defined, 301 soldier ants, 318 Sonata for Unaccompanied Achilles, 502-3 Sonata for violin and clavier in F minor (Bach), 162

Index 783Sonatas and partitas for unaccompanied 
violin (Bach), 62, 63, 70-71, 257, 502 song, self-reproducing, 500 "Song Without Time or Season, A", 483 songs, recombinant, 607-8 sonnets, 596, 608, 736 soulism, 385, 472, 574, 597, 686, 708 spaces: abstract, 457; of behavior, 306, 307, 363, 621; of integers, 502 spare tire, 670 Sperry, Roger, 710 Sphex wasp, 360-61, 613-14 spiderwebs, 371-72, 617 splitting-off, 353-54, 664; see also classes vs. instances stacks, pushdown, 127-35, 136, 610-11 starting positions in Art of Zen strings, 239-40, 241 State of Grace (Magritte), 481 statistical mechanics, see gases Sterner, Geo., 166-67, 642-43, 670 Stent, Gunther, 514 stomach processes, 572 strands: of DNA and RNA, 514-18; in Typogenetics, 508-13, 514 Strange Loops: abolishment of, 21-23; with 
Babbage and Turing, 737; in Bach, 10,

Babbage and Turing, 737; in Bach, 10, 
719; in Central Dogmap, 534; consciousness and, 709-10; contrasted with feedback, 545, 691; defined, 10; in Escher, 10-15, 714-17; in Godel, 15-17, 24; in government, 692-93; in language, 22, 691; in the mind, 27, 691-92; in molecular biology, 231, 532-34, 545, 546-48; in Principia Mathematica, 24; surprise element necessary in, 691; in TNT, see TNT, introspection of; see also Tangled Hierarchies, level-mixing, levelconflicts, level- confusion, etc. street nuisances, 25, 726-27, 729 stretto, 314, 738 String Manipulation Rules, 240-42 strings: defined, 33-34; folded, 234-44, 427 structure vs. function, 522, 670-71; see also use vs. mention, syntax vs. semantics style, 148,

371 SUB (TNT-formula), 444-45 subbrains, see subsystems subdescriptions, etc. 650-51; see also recursive structure of ideas subframes, etc., 644-45; see abo recursive structure of ideas subject vs. object, 698-99; see also dualism, use vs. mention, symbol vs. object subjunc-TV, 635110 subjunctive instant replays, see instant replays subjunctives, see counterfactuals suborganisms, see subsystems subroutines, 150, 292, 471, 677 substitution notation in TNT, defined, 224 substitution relationship (ingredient in Godel's proof), 443-45, 497  substrate: of Epimenides sentence, 581, 584-85; mental, necessity of, for analogical thought, 569-71; mental, noninterpretability of, 570-71; mental, simulation of, 571-72; of proteins, 529 subsystems of brain, 385-88, 725 subtraction, 52-53, 411-12 sufficiently powerful systems, 86, 101, 
406-7, 430, 460, 530

406-7, 430, 460, 530 
Suites for unaccompanied cello (Bach), 70-71 summarizing string, 221-22, 450-51 superconductivity, 304-5 superinielligence, 679 supernatural numbers, 223, 453-56, 458-59, 467 supernatural proofs, 454-55 supertangling, 688 surrealism, 700 Sussman, Gerald, 664 Swieten, Baron Gottfried van, 6-7 Switcheroo, Q. q., 187 symbol-level description of brain, 349-51 symbol vs. object, 699-706; see also subject vs. object, use vs. mention, etc. symbols: active vs. passive, 324-25, 327-28, 337-38; Al realizations of, 662-63, 665; in ant colonies, 324-28, 330; borderlines between, 354-57, 359-60; as brush strokes, 351; compared with neurons, 350, 371;

Index 784compared with ripples, 356-37; conceptual 
scope of, 350-51; dormant, 327, 349, 355-56, 384; form of, 348, 356-57, 361; free will and, 712-14; funneling and, 347; of insects, 360-61; invariant core of, 349; joint activation of, 351, 354-56, 359, 361, 365, 584-85, 664-65, 675; message-exchange and, 350, 371, 662-63; modes of activation of, 349-57, 361; need of, for originality, 609; neural substrate of, 356-57, 570; no access to substrate by, see inaccessibility; overcrowding of, 358; overlap of, 348-49, 356-57; potential, 355-56, 382-84; universal, 375-76; vs. neurons, 348, 356-57, 361; vs. signals, 325-27, 349-50 synapses, 339 syntax vs. semantics, 626-27, 630-32, 676; see also form, syntactic vs. semantic system, boundaries of, 37-38; see also jumping out of the system System crash, 116 systems, reliable vs. unreliable, 307  T (Tarski's formula), 580-81 't'-concept, 'h'-concept, 'c -concept, 326 T-
even phages, 540, 542 T-level, see tangled

even phages, 540, 542 T-level, see tangled 
level table of nontheorems, 66 Tagore, Rabindranath, 169 Tangled Hierarchies: in art, 704; defined, 10; of genetics, 532-34, 546-48, 688; of meta mathematics, 458, 532-34; of mind, 691-92, 709-10, 719; near miss, 691; in Propositional Calculus, 194; of self-modifying chess, 688; of Tortoises reasoning, 177-80; of Typogenetics, 513, 688 tangled level, 688 Tanguy, Yves, 700 tape recorders, 485, 519, 523, 525-28 Tarski, Alfred, 579-81 Tarski-Church-Turing Theorem. 561, 581 Tarski's Theorem, 579, 580-81, 584-85, 697 Taube, Mortimer, 574 Taurinus, F..4., 92 IrC_battles, 75-78, 406-7, 424, 467-70,

478, 483-88, 536-41, 721 tea. 153-54, 231, 275, 322-23, 333, 549, 557, 558, 561 teleological vs. evolutionary viewpoint, 320-22 telephone calls, 61-63, 127-28. 161; obscene, 431, 437 telephone systems, 296, 663 telephones, 302 television, 285, 348, 478, 484, 487-93, 634-40 telophase, 667-68 templates: for Bongard problems, 650-53, 656-57; vs. instructions, see instructions vs. tem plates, programs vs. data tension and resolution, 121-23, 129-30, 227 tentativity, 646, 651, 654-56, 672 termination tester, 425-29 terminators (FlooP), 425-29 terms (TNT), 206-7, 213, 214 tertiary structure: of proteins, 519-22, 525-27; of tRNA, 524; of typoenzymes, 510-11, 512, 519 Tesler, Lawrence G., 601 Tesler's Theorem, 601, 623; see also paradox of Al, jumping out of the system, 
essential in completeness

essential in completeness 
tesselations, 68, 69, 198, 667 tests vs. functions in BlooP and FlooP, 413, 418 text-handling by computers, 301 T4 phage, 537-41 "the" 586, 629-30 theorem-numbers, 264-67, 269-70, 440-43, 451 theorem-proving, mechanical, 602, 609, 617-19 theorems: defined, 35; systematic enumeration of, 39-40, 48, 471-73, 578, 615, 617-18; vs. non theorems, 39-41, 66-67, 70, 71-73, 190-91, 416-17, 560, 579-80; vs. rules, 43-45, 193-94, 509-10; vs. Theorems, 35, 193; vs. truths, 49-54, 70-71, 86-102, 190-97, 213, 221-23, 228-30, see also consistency, completeness, Godel's Theorem, consequences of

Index 785Theseus and Ariadne, 130 thinking, speed 
of, 679 30, as possible MIU-number, 265-67 "this sentence", 436, 495-98, 499 thought, substrate of, 559 3n+1 problem, 400-2; see also wondrous and un wondrous numbers Three Spheres If (Escher), 258 310-system, 261-67 Three Worlds (Escher), 247, 256 thymine, see nucleotides tilde, 183, 191-92, 554 timesharing, 296, 354-55, 387, 730 TNT, 204-30; absolute geometry and, 451-52; austere, 211, 214, 216, 268, 442, 534; axioms of, 215-17, 222-23; as code, 265-67; consistency of, 229-30, 449-50; extended, axioms of, 451-52, 466-68; extensions of, 451-59, 465-68; FIGUR E-FIGURE figure 
and, 70; as general metalanguage, 265-67; goals of, 60; incompleteness of, see incompleteness; introspection of, 17, 194, 267-72, 406, 438, 443, 449-50, 698, 707-8; as its own metalanguage, 267-72, 441-46, 514; plight of, pigorial version, 71; relaxed, 228; rules of formation, table of, 213-14; 
rules of inference, 215, 217-15; sixth

rules of inference, 215, 217-15; sixth 
axiom of, 222-23, 451-55, 459, 465-68; well-formedness in, 205-15 TNT-derivations compared with machine language, 291 TNT-level, explanations on, 708 TNT-numbers, 269-70; see also theorem-numbers TNT + G, 465-67, 471 TNT + G + G', etc., 467-71 TNT + -G, 467 tobacco mosaic virus, 484-85, 542, 543 Todelization, repeatability of, 76-78, 424, 467-70, 483-86; see also Godelization, essential incompleteness, answer schemas, etc. Todel's Theorem, 486, 536 Tokusan, 189-90 tonic, musical, 121-23, 129-30 Tortoise: answer-schemas and, 475; Carroll

paradox and, 46, 193, 684-85, 693; Crab Canon and, 204, 666-67; degenerate solution by, 669; Diophantine equations and, 4590; as harpsichord, 502; initial letter of, 231, 507, 667; mentioned, 102, 170, 267; origin of, 28-29; picture of, 42; recursion and, 128-30; 149; use of words by, 181; vs. Crab, 84-86, 271, 406, 424, 467-69, 540, 543; Zen string made by, 272; in ZET cycle, 94-95 Tortoise-pairs, 416, 441, 448; compared with proof-pairs, 441, 448 Tortoise property, 395-97, 415-18, 425, 426, 441, 448 Tortoise's love song, 435-36 Tortoise's method, see Todelization T6zan, 190, 255, 257, 479 tq-system, 64-67 transcendentalism, 704 transcription: DNA to mRNA, 517, 524, 527-28, 530, 533, 536, 538, 540-41, 544-45; DNA to tRNA, 425; koans to messengers, 235-36, 238, 239, 242; letters to notes, 83; prevention of, 544-45 transfer RNA, see tRNA 
translation: between computer languages,

translation: between computer languages, 
192-95, 297-98, 306, 380-81, 547, 632; between levels of a brain, 349, 381-84, 709; between natural languages, difficulties of, 372-73, 379-80; between TNT and meta-TNT, 267-72, 441-46, 709; of Crime and Punishment, 379-80.; English to TNT, 209-13, 215, 417; of "Jabberwocky", 372-73, 379; levels of fidelity in, 379-80; mechanical, 380, 601, 603; of messengers into strings, 234-36; mRNA to proteins, 485, 518-19, 522-25, 527-28, 531-36, 538, 545, 546-48; from N to Meta-TNT, 533; from score to sounds, 83; in Typogenetics, 509-10, 512-13 transparency to user, 629, 632 trees: look-ahead, see look-ahead trees; recursive diagrams, 40„ 71, 135-37; of theorems, 40, 71

Index 786triggering patterns of symbols: dependence 
on meaningless lower levels, 569; isomorphism between minds and, 369, 376; isomorphism between physical law and, 362; as key to meaning, 325, 327, 350, 360, 385, 609; mediated by mes- triggering patterns of symbols (continued) sages, 350, 371; for melodies, 364; nouns vs. verbs, 361; randomness in, 673 triggers: DNA as, 160-61; dormant symbols and, 281, 383, 384; frame messages and, 162; jukeboxes and, 160-61, 170-71, 174, 500; koans as, 246; music and, 162-63, 281, 383, 583; outer messages and, 166, 169, 170-71, 174, 501 Trio Sonata from the Musical Offering (Bach), 7-8, 720, 724, 726 Tripitaka, 257 trip-lets, cover, xiv, 1, 28, 273 tRNA, 522-24, 547, 548 Trojan Horse, 538 truth: capturability by symbol manipulation, 53-60; elusiveness of, 694-95; inexpressible in TNT, 580-81; not fully mirrorable in brain, 584-85; vs. beauty, 554-58, 584; vs. commercials, 478; vs. 
falsity, 70, 71, 213, 228-29, 417, 561, 579-

81 TTortoise, see ATTACCA tuba, flaming, 488-89, 492, 735 Tumbolia, 116, 243, 255. 725; lavers of, 243 tuning an Al program, 678-79 Turing, Alan M., 26, 389, 425-26, 428-29. 594-99, 734-42; objections to Al, 597-99 Turing, Sara, 595 Turing machines, 390, 594, 735 Turing test, 595-99, 600, 677-78, 735-37; arithmetic error in, 596; miniature, 621-23; proposed revisions in, 600 Turtle's Theorem, see Todefs Theorem 2, as concept, 678 2-D vs. 3-D: in Escher, 57-58, 105-6, 125, 473-74, 524, 689-90, 698, 714-16; in Magritte, 480-81, 493-94, 700-1, 705-6; television screens and, 488-93, 737; trip-lets and, see triplets Two Mysteries, The (Magritte), 701-2

Two-Part Invention, 28, 43-45, 684-86; see also Carroll paradox, Lewis Carroll 2 + 2 = 5,576 typeface metaphor, 541 Typeless Wishes, 111-15, 610-11 types, theory of, 21-23  typesetting machine, 608 typoenzymes, 505-13; binding-preferences of, 505-6, 511-12 Typogenctic Code, 510, 512, 513, 519 Typogenetics, 504-13, 514, 519, 520, 529; con trasted with MIU-system, 509-10, 514 Typographical Number Theory, see TNT typographical operations, defined, 64 typos, 404  U, as nontheorem of MIU-system, 36, 39 U-mode, see Un-mode Ulam, Stanislaw, 560, 621, 676 Un-mode, 39, 98, 254 Unamuno, Miguel de, 698 uncertainty principle, see Heisenberg uncertainty principle uncles, 446-48, 464, 466, 468, 541, 580 
undecidability, 17, 222, 449, 451-55, 468;

undecidability, 17, 222, 449, 451-55, 468; 
causes of, 707-8 undefined terms, 92-102, 216, 456; defined, 93, 97 understanding, nature of, 569, 675-76, 680 understanding minds/brains: meaning of, 697; possibility of, 697-98, 706-7 unicycle, tandem, 633, 669 units in Tvpogenetics. 505, 509 universal quantifiers, see quantifiers universal triggering power, 171, 175 Unmon, 254 unobservant machines, 36-37, 674 unpredictable but guaranteed termination, 400, 425 upper bounds, see loops, bounded, BIooP uracil, see nucleotides use vs. mention, 434-37, 458, 531, 545, 699-700; see also form, syntactic vs. semantic, programs

Index 787vs. data, syntax vs. semantics, structure vs. 
function variables in TNT, 206, 213-14; free, 207-9, 214; quantified, 208, 214, see also quantifiers verb-at-the-end phenomenon, 130-31 Verbum (Escher), 257, 731-32 versus, see accessible vs. inaccessible knowledge, active vs. passive symbols, ants vs. ant colonies, arithmetical vs. typographical rules, Bach vs. Cage, beautiful vs. non-beautiful, bottom-up vs. top-down, classes vs. instances, continuous vs. discrete processes, deductive vs. analogical awareness, derivations vs. proofs, dissection vs. appreciation of Bach, distinct vs. similar levels, enzymes vs. typoenzymes, explicit vs. implicit knowledge, explicit vs. implicit meaning, explicit vs. implicit pruning, formal vs. informal reasoning, formal vs. informal systems, formal systems vs. reality, genuine vs. phony koans, high-fidelity vs. low-fidelity, holism vs. reductionism, improvisation vs. introspection, instructions vs. templates, local vs. global properties, meaningless vs. meaningful

interpretations, men vs. women, minds vs. 
brains, nouns vs. verbs in triggering patterns, 1-D vs. 3-D, passive vs. active meaning, people vs. machines, plausible vs. implausible pathways, President v. Supreme Court, procedural vs. declarative knowledge, programs vs. data, programs vs. programmers, purposeful vs. purposeless behavior, rational vs. irrational, sanity vs. insanity, self-perception vs. self-transcendence, structure vs. function, subject vs. object, symbol vs. object, symbols vs. neurons, symbols vs. signals, syntactic vs. semantic form, syntax vs. semantics, teleological vs. evolutionary viewpoint, theorems vs. nontheorems, theorems vs. rules, theorems vs. Theorems, theorems vs. truth, Tortoise vs. Crab, truth vs. beauty, truth vs. commercials, truth vs. falsity, 2-D vs. 3-D, use vs. mention, weight vs. mass, womanseeing vs. seeing,

words vs. letters, Zen vs. logic, Zen vs. words vibrations, 76-78, 82-85, 102, 270, 271, 469 Vice President, 670 Villon, Fransois, 369 Vinogradov, Ivan M., 394-95 Vinogradov property, 394-95 violins, 62, 63, 70, 81, 84, 162, 200, 257, 434, 502, 595, 681, 720, 724 viruses, 536-43; likened to Henkin sentences, 542-43 vision by computer, 602, 627 visual imagery: Bongard problems and, 661: faucets and, 364-65; inaccessible knowledge and, 365; lack of in programs, 623; mathematics and, 569, 678; necessitating layers of substrate, 570-71; power of, 338-39; role in conceptual mapping. 668, 672; rubbing-off and. 361-62 es and canons, 28, 282-83, 314, voices in fugu 322-23, 335, 665-67, 669, 683, 737, 740 Voltaire, Francois Marie Arouet de, 3 
vortices, 713-19

vortices, 713-19 
Vuillard, Edouard, 347 Wachter, F. L., 92 Warrin, Frank L., 366 -WASH ME", 608 Waterfall (Escher), 11-13, 99, 716 Watergate, 692  Watson, J. D.. 667 Weasel, 106 weather. 302-3 Weaver, Warren, 380 Weierstrass, Karl W. T., 404 weight vs. mass, 171-72 Weizenbaum, Joseph, 599-600. 675  well-formed strings: in Art of Zen Strings, 239, 242, 244; BIooP puzzle on, 416; defined. 53: in pq-svstem, 47; in Propositional Calculus, 181-83; in TNT, 213-15 Well-Te"ered Clavier (Bach), 7, 280-84, 327. 329, 335

Index 788Well-Tested Conjecture (Fourmi), 333-35 
"When Johnny Comes Marching Home", 607 Whitehead, Alfred North, 18, 21, 23-24 Whitely, C. H., 477  wholes, see holism Wiener, Norbert, 684 Wiesel, Torsten, 343 will: free, see free will; mechanical, 684-86; roots of, 684-86 Wilson, E. 0., 350 Winograd. Terry, 627-32 Winston. Patrick Henry, 299 wishes, meta-wishes, etc., 109-16 Wittgenstein, Ludwig, 680, 699 Wolff, Christoph, 392  womanseeing vs. seeing, womansee men vs. women wondrous and unwondrous numbers, 400-2, 408, 415, 418, 425 Wooldridge, Dean, 360 words: in computers, 288-90, 295, 411; as programs, 629-30; spelled backwards. 81, 418, 427, 498, 505, 533, 549, 727, 738, 740; thoughts, formal rules, and, see main theses: vs. letters, 325-27, 570, 571; Zen attitude towards, 
246, 249, 251-54

246, 249, 251-54 
worker ants, 318 working inside the system, see Mechanical mode yes-answers, 461-64 Yngve, Victor, 620 Young, LaMonte, 700 Zen Algebra, 577 Zen Buddhism: Achilles teaches the Tortoise about. 231-45; computers and, 625-26; Crab's refrigerator and, 406-7; Escher and, 255-58; holism vs. reductionism and, 312-13; inconsistencies and, see Zen vs. logic; introduction to, 246-59; jumping out of the system and, 233, 255, 479; music, art, and, 699, 704-6; Mystery of the Undecidable and, 272; nonexistence and, 254-55, 698; patriarchs of, 30, 232, 252, 259; quasi-, 625-26; U-mode and, 39, 98. 254; vs, logic, 99, 233-34, 249-51, 254; vs. words, 246, 248-49, 
251-54; Zeno and, 30 Zen Strings, Art of, see Art of Zen Strings Zenfunny, Beethoven's Ninth, 634 Zeno of Elea, 28, 29-32, 94-95, 144, 146, 232, 610, 681, 704, 722 Zeno's paradoxes, 29-32, 35, 43, 146, 610 Zentences, 186-90 ZET-cycles, 94-96, 689 zooming in, 645, 671 i T

Machine Learning
A Probabilistic Perspective
Kevin P. Murphy
“ An astonishing machine learning book: intuitive, full  
of examples, fun to read but still comprehensive, 
strong, and deep! A great starting point for any univer-
sity student—and a must-have for anybody in the field. ”
Jan Peters, Darmstadt University of Technology;  
Max-Planck Institute for Intelligent Systems
“Kevin Murphy excels at unraveling the complexities  
of machine learning methods while motivating the 
reader with a stream of illustrated examples and 
real-world case studies. The accompanying software 
package includes source code for many of the figures, 
making it both easy and very tempting to dive in and 
explore these methods for yourself. A must-buy for 
anyone interested in machine learning or curious 
about how to extract useful knowledge from big data. ”
John Winn, Microsoft Research
“This is a wonderful book that starts with basic topics 
in statistical modeling, culminating in the most ad-

vanced topics. It provides both the theoretical foun-
dations of probabilistic machine learning as well as 
practical tools, in the form of MATLAB code. The book 
should be on the shelf of any student interested in the 
topic, and any practitioner working in the field. ”
Y oram Singer, Google Research
“This book will be an essential reference for practitio-
ners of modern machine learning. It covers the basic 
concepts needed to understand the field as a whole,

concepts needed to understand the field as a whole, 
and the powerful modern methods that build on those 
concepts. In Machine Learning , the language of prob-
ability and statistics reveals important connections be-
tween seemingly disparate algorithms and strategies. 
Thus, its readers will become articulate in a holistic 
view of the state-of-the-art and poised to build the next 
generation of machine learning algorithms. ”
David Blei, Princeton University
 machine learning
Kevin P . Murphy
Today’s Web-enabled deluge of electronic data calls for

Today’s Web-enabled deluge of electronic data calls for 
automated methods of data analysis. Machine learning 
provides these, developing methods that can automatically 
detect patterns in data and use the uncovered patterns to 
predict future data. This textbook offers a comprehensive 
and self-contained introduction to the field of machine 
learning, a unified, probabilistic approach. 
The coverage combines breadth and depth, offering 
necessary background material on such topics as probabili-
ty, optimization, and linear algebra as well as discussion of

ty, optimization, and linear algebra as well as discussion of 
recent developments in the field, including conditional ran-
dom fields, L1 regularization, and deep learning. The book 
is written in an informal, accessible style, complete with 
pseudo-code for the most important algorithms. All topics 
are copiously illustrated with color images and worked 
examples drawn from such application domains as biology, 
text processing, computer vision, and robotics. Rather than 
providing a cookbook of different heuristic methods, the

providing a cookbook of different heuristic methods, the 
book stresses a principled model-based approach, often 
using the language of graphical models to specify mod-
els in a concise and intuitive way. Almost all the models 
described have been implemented in a MATLAB software 
package—PMTK (probabilistic modeling toolkit)—that is 
freely available online. The book is suitable for upper-level 
undergraduates with an introductory-level college math 
background and beginning graduate students.  
Kevin P . Murphy is a Research Scientist at Google. Previ-

ously, he was Associate Professor of Computer Science 
and Statistics at the University of British Columbia.
Adaptive Computation and Machine Learning seriesThe MIT Press
Massachusetts Institute of Technology
Cambridge, Massachusetts 02142
http://mitpress.mit.edu978-0-262-01802-9
The cover image is based on sequential Bayesian updating  
of a 2 D Gaussian distribution. See Figure 7 .11 for details.

The MIT Press
Cambridge, MassachusettsLondon, England

© 2012 Massachusetts Institute of Technology
All rights reserved. No part of this book may be reproduced in any form by any electronic or mechanical
means (including photocopying, recording, or information storage and retrieval) without permission inwriting from the publisher.
For information about special quantity discounts, please email special_sales@mitpress.mit.eduThis book was set in the L
ATEX programming language by the author. Printed and bound in the United
States of America.Library of Congress Cataloging-in-Publication Information
Murphy, Kevin P.
Machine learning : a probabilistic perspective / Kevin P. Murphy.p. cm. — (Adaptive computation and machine learning series)Includes bibliographical references and index.ISBN 978-0-262-01802-9 (hardcover : alk. paper)1. Machine learning. 2. Probabilities. I. Title.Q325.5.M87 2012006.3’1—dc232012004558
1 0987654321

This book is dedicated to Alessandro, Michael and Stefano,
and to the memory of Gerard Joseph Murphy.

Contents
Preface xxvii
1.1 Machine learning: what and why? 1
1.1.1 Types of machine learning 2
1.2 Supervised learning 3
1.2.1 Classiﬁcation 3
1.2.2 Regression 8
1.3 Unsupervised learning 9
1.3.1 Discovering clusters 101.3.2 Discovering latent factors 111.3.3 Discovering graph structure 131.3.4 Matrix completion 14
1.4 Some basic concepts in machine learning 16
1.4.1 Parametric vs non-parametric models 161.4.2 A simple non-parametric classiﬁer: K-nearest neighbors 16
1.4.3 The curse of dimensionality 181.4.4 Parametric models for classiﬁcation and regression 191.4.5 Linear regression 191.4.6 Logistic regression 211.4.7 Overﬁtting 221.4.8 Model selection 221.4.9 No free lunch theorem 24
2 Probability 27
2.1 Introduction 272.2 A brief review of probability theory 28
2.2.1 Discrete random variables 282.2.2 Fundamental rules 282.2.3 Bayes rule 292.2.4 Independence and conditional independence 302.2.5 Continuous random variables 32

viii CONTENTS
2.2.6 Quantiles 33
2.2.7 Mean and variance 33
2.3 Some common discrete distributions 34
2.3.1 The binomial and Bernoulli distributions 342.3.2 The multinomial and multinoulli distributions 352.3.3 The Poisson distribution 372.3.4 The empirical distribution 37
2.4 Some common continuous distributions 38
2.4.1 Gaussian (normal) distribution 382.4.2 Degenerate pdf 392.4.3 The Laplace distribution 412.4.4 The gamma distribution 412.4.5 The beta distribution 422.4.6 Pareto distribution 43
2.5 Joint probability distributions 44
2.5.1 Covariance and correlation 442.5.2 The multivariate Gaussian 462.5.3 Multivariate Student tdistribution 46
2.5.4 Dirichlet distribution 47
2.6 Transformations of random variables 49
2.6.1 Linear transformations 492.6.2 General transformations 502.6.3 Central limit theorem 51
2.7 Monte Carlo approximation 52
2.7.1 Example: change of variables, the MC way 532.7.2 Example: estimating πby Monte Carlo integration 54
2.7.3 Accuracy of Monte Carlo approximation 54

2.8 Information theory 56
2.8.1 Entropy 562.8.2 KL divergence 572.8.3 Mutual information 59
3 Generative models for discrete data 65
3.1 Introduction 653.2 Bayesian concept learning 65
3.2.1 Likelihood 673.2.2 Prior 673.2.3 Posterior 683.2.4 Posterior predictive distribution 713.2.5 A more complex prior 72
3.3 The beta-binomial model 72
3.3.1 Likelihood 733.3.2 Prior 743.3.3 Posterior 753.3.4 Posterior predictive distribution 77

CONTENTS ix
3.4 The Dirichlet-multinomial model 78
3.4.1 Likelihood 79
3.4.2 Prior 793.4.3 Posterior 793.4.4 Posterior predictive 81
3.5 Naive Bayes classiﬁers 82
3.5.1 Model ﬁtting 833.5.2 Using the model for prediction 853.5.3 The log-sum-exp trick 863.5.4 Feature selection using mutual information 863.5.5 Classifying documents using bag of words 87
4 Gaussian models 97
4.1 Introduction 97
4.1.1 Notation 974.1.2 Basics 974.1.3 MLE for an MVN 994.1.4 Maximum entropy derivation of the Gaussian * 101
4.2 Gaussian discriminant analysis 101
4.2.1 Quadratic discriminant analysis (QDA) 1024.2.2 Linear discriminant analysis (LDA) 1034.2.3 Two-class LDA 1044.2.4 MLE for discriminant analysis 1064.2.5 Strategies for preventing overﬁtting 1064.2.6 Regularized LDA * 1074.2.7 Diagonal LDA 1084.2.8 Nearest shrunken centroids classiﬁer * 109
4.3 Inference in jointly Gaussian distributions 110
4.3.1 Statement of the result 1114.3.2 Examples 1114.3.3 Information form 1154.3.4 Proof of the result * 116

4.4 Linear Gaussian systems 119
4.4.1 Statement of the result 1194.4.2 Examples 1204.4.3 Proof of the result * 124
4.5 Digression: The Wishart distribution * 125
4.5.1 Inverse Wishart distribution 1264.5.2 Visualizing the Wishart distribution * 127
4.6 Inferring the parameters of an MVN 127
4.6.1 Posterior distribution of μ128
4.6.2 Posterior distribution of Σ* 128
4.6.3 Posterior distribution of μandΣ* 132
4.6.4 Sensor fusion with unknown precisions * 138

x CONTENTS
5 Bayesian statistics 149
5.1 Introduction 149
5.2 Summarizing posterior distributions 149
5.2.1 MAP estimation 1495.2.2 Credible intervals 1525.2.3 Inference for a difference in proportions 154
5.3 Bayesian model selection 155
5.3.1 Bayesian Occam’s razor 1565.3.2 Computing the marginal likelihood (evidence) 1585.3.3 Bayes factors 1635.3.4 Jeffreys-Lindley paradox * 164
5.4 Priors 165
5.4.1 Uninformative priors 1655.4.2 Jeffreys priors * 1665.4.3 Robust priors 1685.4.4 Mixtures of conjugate priors 168
5.5 Hierarchical Bayes 171
5.5.1 Example: modeling related cancer rates 171
5.6 Empirical Bayes 172
5.6.1 Example: beta-binomial model 1735.6.2 Example: Gaussian-Gaussian model 173
5.7 Bayesian decision theory 176
5.7.1 Bayes estimators for common loss functions 1775.7.2 The false positive vs false negative tradeoff 1805.7.3 Other topics * 184
6 Frequentist statistics 191
6.1 Introduction 1916.2 Sampling distribution of an estimator 191
6.2.1 Bootstrap 1926.2.2 Large sample theory for the MLE * 193

6.3 Frequentist decision theory 194
6.3.1 Bayes risk 1956.3.2 Minimax risk 1966.3.3 Admissible estimators 197
6.4 Desirable properties of estimators 200
6.4.1 Consistent estimators 2006.4.2 Unbiased estimators 2006.4.3 Minimum variance estimators 2016.4.4 The bias-variance tradeoff 202
6.5 Empirical risk minimization 204

6.5.1 Regularized risk minimization 2056.5.2 Structural risk minimization 2066.5.3 Estimating the risk using cross validation 2066.5.4 Upper bounding the risk using statistical learning theory * 209

CONTENTS xi
6.5.5 Surrogate loss functions 210
6.6 Pathologies of frequentist statistics * 211
6.6.1 Counter-intuitive behavior of conﬁdence intervals 212
6.6.2 p-values considered harmful 2136.6.3 The likelihood principle 2146.6.4 Why isn’t everyone a Bayesian? 215
7 Linear regression 217
7.1 Introduction 2177.2 Model speciﬁcation 2177.3 Maximum likelihood estimation (least squares) 217
7.3.1 Derivation of the MLE 2197.3.2 Geometric interpretation 2207.3.3 Convexity 221
7.4 Robust linear regression * 2237.5 Ridge regression 225
7.5.1 Basic idea 2257.5.2 Numerically stable computation * 2277.5.3 Connection with PCA * 2287.5.4 Regularization effects of big data 230
7.6 Bayesian linear regression 231
7.6.1 Computing the posterior 2327.6.2 Computing the posterior predictive 2337.6.3 Bayesian inference when σ
2is unknown * 234
7.6.4 EB for linear regression (evidence procedure) 238
8 Logistic regression 245
8.1 Introduction 2458.2 Model speciﬁcation 2458.3 Model ﬁtting 245

8.3.1 MLE 2468.3.2 Steepest descent 2478.3.3 Newton’s method 2498.3.4 Iteratively reweighted least squares (IRLS) 2508.3.5 Quasi-Newton (variable metric) methods 2518.3.6 ℓ
2regularization 252
8.3.7 Multi-class logistic regression 252
8.4 Bayesian logistic regression 254
8.4.1 Laplace approximation 2558.4.2 Derivation of the BIC 2558.4.3 Gaussian approximation for logistic regression 2568.4.4 Approximating the posterior predictive 2568.4.5 Residual analysis (outlier detection) * 260

8.5 Online learning and stochastic optimization 261
8.5.1 Online learning and regret minimization 262

xii CONTENTS
8.5.2 Stochastic optimization and risk minimization 262
8.5.3 The LMS algorithm 2648.5.4 The perceptron algorithm 2658.5.5 A Bayesian view 266
8.6 Generative vs discriminative classiﬁers 267
8.6.1 Pros and cons of each approach 2688.6.2 Dealing with missing data 2698.6.3 Fisher’s linear discriminant analysis (FLDA) * 271
9 Generalized linear models and the exponential family 281
9.1 Introduction 2819.2 The exponential family 281
9.2.1 Deﬁnition 2829.2.2 Examples 2829.2.3 Log partition function 2849.2.4 MLE for the exponential family 2869.2.5 Bayes for the exponential family * 2879.2.6 Maximum entropy derivation of the exponential family * 289
9.3 Generalized linear models (GLMs) 290
9.3.1 Basics 2909.3.2 ML and MAP estimation 2929.3.3 Bayesian inference 293
9.4 Probit regression 293
9.4.1 ML/MAP estimation using gradient-based optimization 2949.4.2 Latent variable interpretation 2949.4.3 Ordinal probit regression * 2959.4.4 Multinomial probit models * 295
9.5 Multi-task learning 296

9.5.1 Hierarchical Bayes for multi-task learning 2969.5.2 Application to personalized email spam ﬁltering 2969.5.3 Application to domain adaptation 2979.5.4 Other kinds of prior 297
9.6 Generalized linear mixed models * 298
9.6.1 Example: semi-parametric GLMMs for medical data 2989.6.2 Computational issues 300
9.7 Learning to rank * 300
9.7.1 The pointwise approach 3019.7.2 The pairwise approach 3019.7.3 The listwise approach 3029.7.4 Loss functions for ranking 303
10 Directed graphical models (Bayes nets) 307
10.1 Introduction 307
10.1.1 Chain rule 30710.1.2 Conditional independence 308

CONTENTS xiii
10.1.3 Graphical models 308
10.1.4 Graph terminology 30910.1.5 Directed graphical models 310
10.2 Examples 311
10.2.1 Naive Bayes classiﬁers 31110.2.2 Markov and hidden Markov models 31210.2.3 Medical diagnosis 31310.2.4 Genetic linkage analysis * 31510.2.5 Directed Gaussian graphical models * 318
10.3 Inference 31910.4 Learning 320
10.4.1 Plate notation 32010.4.2 Learning from complete data 32210.4.3 Learning with missing and/or latent variables 323
10.5 Conditional independence properties of DGMs 324
10.5.1 d-separation and the Bayes Ball algorithm (global Markov
properties) 324
10.5.2 Other Markov properties of DGMs 32710.5.3 Markov blanket and full conditionals 327
10.6 Inﬂuence (decision) diagrams * 328
11 Mixture models and the EM algorithm 337
11.1 Latent variable models 33711.2 Mixture models 337
11.2.1 Mixtures of Gaussians 33911.2.2 Mixture of multinoullis 34011.2.3 Using mixture models for clustering 34011.2.4 Mixtures of experts 342
11.3 Parameter estimation for mixture models 345

11.3.1 Unidentiﬁability 34611.3.2 Computing a MAP estimate is non-convex 347
11.4 The EM algorithm 348
11.4.1 Basic idea 34911.4.2 EM for GMMs 35011.4.3 EM for mixture of experts 35711.4.4 EM for DGMs with hidden variables 35811.4.5 EM for the Student distribution * 35911.4.6 EM for probit regression * 36211.4.7 Theoretical basis for EM * 36311.4.8 Online EM 36511.4.9 Other EM variants * 367
11.5 Model selection for latent variable models 370

11.5.1 Model selection for probabilistic models 37011.5.2 Model selection for non-probabilistic methods 370
11.6 Fitting models with missing data 372

xiv CONTENTS
11.6.1 EM for the MLE of an MVN with missing data 373
12 Latent linear models 381
12.1 Factor analysis 381
12.1.1 FA is a low rank parameterization of an MVN 381
12.1.2 Inference of the latent factors 38212.1.3 Unidentiﬁability 38312.1.4 Mixtures of factor analysers 38512.1.5 EM for factor analysis models 38612.1.6 Fitting FA models with missing data 387
12.2 Principal components analysis (PCA) 387
12.2.1 Classical PCA: statement of the theorem 38712.2.2 Proof * 38912.2.3 Singular value decomposition (SVD) 39212.2.4 Probabilistic PCA 39512.2.5 EM algorithm for PCA 396
12.3 Choosing the number of latent dimensions 398
12.3.1 Model selection for FA/PPCA 39812.3.2 Model selection for PCA 399
12.4 PCA for categorical data 40212.5 PCA for paired and multi-view data 404
12.5.1 Supervised PCA (latent factor regression) 40512.5.2 Partial least squares 40612.5.3 Canonical correlation analysis 407
12.6 Independent Component Analysis (ICA) 407

12.6.1 Maximum likelihood estimation 41012.6.2 The FastICA algorithm 41112.6.3 Using EM 41412.6.4 Other estimation principles * 415
13 Sparse linear models 421
13.1 Introduction 42113.2 Bayesian variable selection 422
13.2.1 The spike and slab model 42413.2.2 From the Bernoulli-Gaussian model to ℓ
0regularization 425
13.2.3 Algorithms 426
13.3ℓ1regularization: basics 429
13.3.1 Why does ℓ1regularization yield sparse solutions? 430

13.3.2 Optimality conditions for lasso 43113.3.3 Comparison of least squares, lasso, ridge and subset selection 43513.3.4 Regularization path 43613.3.5 Model selection 43913.3.6 Bayesian inference for linear models with Laplace priors 440
13.4ℓ
1regularization: algorithms 441
13.4.1 Coordinate descent 441

CONTENTS xv
13.4.2 LARS and other homotopy methods 441
13.4.3 Proximal and gradient projection methods 44213.4.4 EM for lasso 447
13.5ℓ
1regularization: extensions 449
13.5.1 Group Lasso 44913.5.2 Fused lasso 45413.5.3 Elastic net (ridge and lasso combined) 455
13.6 Non-convex regularizers 457
13.6.1 Bridge regression 45813.6.2 Hierarchical adaptive lasso 45813.6.3 Other hierarchical priors 462
13.7 Automatic relevance determination (ARD)/sparse Bayesian learning (SBL) 463
13.7.1 ARD for linear regression 46313.7.2 Whence sparsity? 46513.7.3 Connection to MAP estimation 46513.7.4 Algorithms for ARD * 46613.7.5 ARD for logistic regression 468
13.8 Sparse coding * 468
13.8.1 Learning a sparse coding dictionary 46913.8.2 Results of dictionary learning from image patches 47013.8.3 Compressed sensing 47213.8.4 Image inpainting and denoising 472
14 Kernels 479
14.1 Introduction 47914.2 Kernel functions 479

14.2.1 RBF kernels 48014.2.2 Kernels for comparing documents 48014.2.3 Mercer (positive deﬁnite) kernels 48114.2.4 Linear kernels 48214.2.5 Matern kernels 48214.2.6 String kernels 48314.2.7 Pyramid match kernels 48414.2.8 Kernels derived from probabilistic generative models 485
14.3 Using kernels inside GLMs 486
14.3.1 Kernel machines 48614.3.2 L1VMs, RVMs, and other sparse vector machines 487
14.4 The kernel trick 488
14.4.1 Kernelized nearest neighbor classiﬁcation 48914.4.2 Kernelized K-medoids clustering 48914.4.3 Kernelized ridge regression 49214.4.4 Kernel PCA 493

14.5 Support vector machines (SVMs) 496
14.5.1 SVMs for regression 49714.5.2 SVMs for classiﬁcation 498

xvi CONTENTS
14.5.3 Choosing C504
14.5.4 Summary of key points 504
14.5.5 A probabilistic interpretation of SVMs 505
14.6 Comparison of discriminative kernel methods 50514.7 Kernels for building generative models 507
14.7.1 Smoothing kernels 50714.7.2 Kernel density estimation (KDE) 50814.7.3 From KDE to KNN 50914.7.4 Kernel regression 51014.7.5 Locally weighted regression 512
15 Gaussian processes 515
15.1 Introduction 51515.2 GPs for regression 516
15.2.1 Predictions using noise-free observations 51715.2.2 Predictions using noisy observations 51815.2.3 Effect of the kernel parameters 51915.2.4 Estimating the kernel parameters 52115.2.5 Computational and numerical issues * 52415.2.6 Semi-parametric GPs * 524
15.3 GPs meet GLMs 525
15.3.1 Binary classiﬁcation 52515.3.2 Multi-class classiﬁcation 52815.3.3 GPs for Poisson regression 531
15.4 Connection with other methods 532

15.4.1 Linear models compared to GPs 53215.4.2 Linear smoothers compared to GPs 53315.4.3 SVMs compared to GPs 53415.4.4 L1VM and RVMs compared to GPs 53415.4.5 Neural networks compared to GPs 53515.4.6 Smoothing splines compared to GPs * 53615.4.7 RKHS methods compared to GPs * 538
15.5 GP latent variable model 54015.6 Approximation methods for large datasets 542
16 Adaptive basis function models 543
16.1 Introduction 54316.2 Classiﬁcation and regression trees (CART) 544

16.2.1 Basics 54416.2.2 Growing a tree 54516.2.3 Pruning a tree 54916.2.4 Pros and cons of trees 55016.2.5 Random forests 55016.2.6 CART compared to hierarchical mixture of experts * 551
16.3 Generalized additive models 552

CONTENTS xvii
16.3.1 Backﬁtting 552
16.3.2 Computational efficiency 55316.3.3 Multivariate adaptive regression splines (MARS) 553
16.4 Boosting 554
16.4.1 Forward stagewise additive modeling 55516.4.2 L2boosting 55716.4.3 AdaBoost 55816.4.4 LogitBoost 55916.4.5 Boosting as functional gradient descent 56016.4.6 Sparse boosting 56116.4.7 Multivariate adaptive regression trees (MART) 56216.4.8 Why does boosting work so well? 56216.4.9 A Bayesian view 563
16.5 Feedforward neural networks (multilayer perceptrons) 563
16.5.1 Convolutional neural networks 56416.5.2 Other kinds of neural networks 56816.5.3 A brief history of the ﬁeld 56816.5.4 The backpropagation algorithm 56916.5.5 Identiﬁability 57216.5.6 Regularization 57216.5.7 Bayesian inference * 576
16.6 Ensemble learning 580
16.6.1 Stacking 58016.6.2 Error-correcting output codes 58116.6.3 Ensemble learning is not equivalent to Bayes model averaging 581
16.7 Experimental comparison 582
16.7.1 Low-dimensional features 58216.7.2 High-dimensional features 583

16.8 Interpreting black-box models 585
17 Markov and hidden Markov models 589
17.1 Introduction 58917.2 Markov models 589
17.2.1 Transition matrix 58917.2.2 Application: Language modeling 59117.2.3 Stationary distribution of a Markov chain * 59617.2.4 Application: Google’s PageRank algorithm for web page ranking * 600
17.3 Hidden Markov models 603
17.3.1 Applications of HMMs 604
17.4 Inference in HMMs 606

17.4.1 Types of inference problems for temporal models 60617.4.2 The forwards algorithm 60917.4.3 The forwards-backwards algorithm 61017.4.4 The Viterbi algorithm 61217.4.5 Forwards ﬁltering, backwards sampling 616

xviii CONTENTS
17.5 Learning for HMMs 617
17.5.1 Training with fully observed data 617
17.5.2 EM for HMMs (the Baum-Welch algorithm) 61817.5.3 Bayesian methods for “ﬁtting” HMMs * 62017.5.4 Discriminative training 62017.5.5 Model selection 621
17.6 Generalizations of HMMs 621
17.6.1 Variable duration (semi-Markov) HMMs 62217.6.2 Hierarchical HMMs 62417.6.3 Input-output HMMs 62517.6.4 Auto-regressive and buried HMMs 62617.6.5 Factorial HMM 62717.6.6 Coupled HMM and the inﬂuence model 62817.6.7 Dynamic Bayesian networks (DBNs) 628
18 State space models 631
18.1 Introduction 63118.2 Applications of SSMs 632
18.2.1 SSMs for object tracking 63218.2.2 Robotic SLAM 63318.2.3 Online parameter learning using recursive least squares 63618.2.4 SSM for time series forecasting * 637
18.3 Inference in LG-SSM 640
18.3.1 The Kalman ﬁltering algorithm 64018.3.2 The Kalman smoothing algorithm 643
18.4 Learning for LG-SSM 646

18.4.1 Identiﬁability and numerical stability 64618.4.2 Training with fully observed data 64718.4.3 EM for LG-SSM 64718.4.4 Subspace methods 64718.4.5 Bayesian methods for “ﬁtting” LG-SSMs 647
18.5 Approximate online inference for non-linear, non-Gaussian SSMs 647
18.5.1 Extended Kalman ﬁlter (EKF) 64818.5.2 Unscented Kalman ﬁlter (UKF) 65018.5.3 Assumed density ﬁltering (ADF) 652
18.6 Hybrid discrete/continuous SSMs 655
18.6.1 Inference 65618.6.2 Application: data association and multi-target tracking 65818.6.3 Application: fault diagnosis 65918.6.4 Application: econometric forecasting 660

19 Undirected graphical models (Markov random ﬁelds) 661
19.1 Introduction 66119.2 Conditional independence properties of UGMs 661
19.2.1 Key properties 661

CONTENTS xix
19.2.2 An undirected alternative to d-separation 663
19.2.3 Comparing directed and undirected graphical models 664
19.3 Parameterization of MRFs 665
19.3.1 The Hammersley-Clifford theorem 66519.3.2 Representing potential functions 667
19.4 Examples of MRFs 668
19.4.1 Ising model 66819.4.2 Hopﬁeld networks 66919.4.3 Potts model 67119.4.4 Gaussian MRFs 67219.4.5 Markov logic networks * 674
19.5 Learning 676
19.5.1 Training maxent models using gradient methods 67619.5.2 Training partially observed maxent models 67719.5.3 Approximate methods for computing the MLEs of MRFs 67819.5.4 Pseudo likelihood 67819.5.5 Stochastic maximum likelihood 67919.5.6 Feature induction for maxent models * 68019.5.7 Iterative proportional ﬁtting (IPF) * 681
19.6 Conditional random ﬁelds (CRFs) 684
19.6.1 Chain-structured CRFs, MEMMs and the label-bias problem 68419.6.2 Applications of CRFs 68619.6.3 CRF training 692
19.7 Structural SVMs 693

19.7.1 SSVMs: a probabilistic view 69319.7.2 SSVMs: a non-probabilistic view 69519.7.3 Cutting plane methods for ﬁtting SSVMs 69819.7.4 Online algorithms for ﬁtting SSVMs 70019.7.5 Latent structural SVMs 701
20 Exact inference for graphical models 707
20.1 Introduction 70720.2 Belief propagation for trees 707
20.2.1 Serial protocol 70720.2.2 Parallel protocol 70920.2.3 Gaussian BP * 71020.2.4 Other BP variants * 712
20.3 The variable elimination algorithm 714
20.3.1 The generalized distributive law * 71720.3.2 Computational complexity of VE 71720.3.3 A weakness of VE 720
20.4 The junction tree algorithm * 720
20.4.1 Creating a junction tree 72020.4.2 Message passing on a junction tree 72220.4.3 Computational complexity of JTA 725

xx CONTENTS
20.4.4 JTA generalizations * 726
20.5 Computational intractability of exact inference in the worst case 726
20.5.1 Approximate inference 727
21 Variational inference 731
21.1 Introduction 731
21.2 Variational inference 732
21.2.1 Alternative interpretations of the variational objective 73321.2.2 Forward or reverse KL? * 733
21.3 The mean ﬁeld method 735
21.3.1 Derivation of the mean ﬁeld update equations 73621.3.2 Example: mean ﬁeld for the Ising model 737
21.4 Structured mean ﬁeld * 739
21.4.1 Example: factorial HMM 740
21.5 Variational Bayes 742
21.5.1 Example: VB for a univariate Gaussian 74221.5.2 Example: VB for linear regression 746
21.6 Variational Bayes EM 749
21.6.1 Example: VBEM for mixtures of Gaussians * 750
21.7 Variational message passing and VIBES 75621.8 Local variational bounds * 756

21.8.1 Motivating applications 75621.8.2 Bohning’s quadratic bound to the log-sum-exp function 75821.8.3 Bounds for the sigmoid function 76021.8.4 Other bounds and approximations to the log-sum-exp function * 76221.8.5 Variational inference based on upper bounds 763
22 More variational inference 767
22.1 Introduction 76722.2 Loopy belief propagation: algorithmic issues 767

22.2.1 A brief history 76722.2.2 LBP on pairwise models 76822.2.3 LBP on a factor graph 76922.2.4 Convergence 77122.2.5 Accuracy of LBP 77422.2.6 Other speedup tricks for LBP * 775
22.3 Loopy belief propagation: theoretical issues * 776

22.3.1 UGMs represented in exponential family form 77622.3.2 The marginal polytope 77722.3.3 Exact inference as a variational optimization problem 77822.3.4 Mean ﬁeld as a variational optimization problem 77922.3.5 LBP as a variational optimization problem 77922.3.6 Loopy BP vs mean ﬁeld 783
22.4 Extensions of belief propagation * 783
22.4.1 Generalized belief propagation 783

CONTENTS xxi
22.4.2 Convex belief propagation 785
22.5 Expectation propagation 787
22.5.1 EP as a variational inference problem 788
22.5.2 Optimizing the EP objective using moment matching 78922.5.3 EP for the clutter problem 79122.5.4 LBP is a special case of EP 79222.5.5 Ranking players using TrueSkill 79322.5.6 Other applications of EP 799
22.6 MAP state estimation 799
22.6.1 Linear programming relaxation 79922.6.2 Max-product belief propagation 80022.6.3 Graphcuts 80122.6.4 Experimental comparison of graphcuts and BP 80422.6.5 Dual decomposition 806
23 Monte Carlo inference 815
23.1 Introduction 81523.2 Sampling from standard distributions 815
23.2.1 Using the cdf 81523.2.2 Sampling from a Gaussian (Box-Muller method) 817
23.3 Rejection sampling 817
23.3.1 Basic idea 81723.3.2 Example 81823.3.3 Application to Bayesian statistics 81923.3.4 Adaptive rejection sampling 81923.3.5 Rejection sampling in high dimensions 820
23.4 Importance sampling 820

23.4.1 Basic idea 82023.4.2 Handling unnormalized distributions 82123.4.3 Importance sampling for a DGM: likelihood weighting 82223.4.4 Sampling importance resampling (SIR) 822
23.5 Particle ﬁltering 823
23.5.1 Sequential importance sampling 82423.5.2 The degeneracy problem 82523.5.3 The resampling step 82523.5.4 The proposal distribution 82723.5.5 Application: robot localization 82823.5.6 Application: visual object tracking 82823.5.7 Application: time series forecasting 831
23.6 Rao-Blackwellised particle ﬁltering (RBPF) 831

23.6.1 RBPF for switching LG-SSMs 83123.6.2 Application: tracking a maneuvering target 83223.6.3 Application: Fast SLAM 834
24 Markov chain Monte Carlo (MCMC) inference 837

xxii CONTENTS
24.1 Introduction 837
24.2 Gibbs sampling 838
24.2.1 Basic idea 83824.2.2 Example: Gibbs sampling for the Ising model 83824.2.3 Example: Gibbs sampling for inferring the parameters of a GMM 84024.2.4 Collapsed Gibbs sampling * 84124.2.5 Gibbs sampling for hierarchical GLMs 84424.2.6 BUGS and JAGS 84624.2.7 The Imputation Posterior (IP) algorithm 84724.2.8 Blocking Gibbs sampling 847
24.3 Metropolis Hastings algorithm 848
24.3.1 Basic idea 84824.3.2 Gibbs sampling is a special case of MH 84924.3.3 Proposal distributions 85024.3.4 Adaptive MCMC 85324.3.5 Initialization and mode hopping 85424.3.6 Why MH works * 85424.3.7 Reversible jump (trans-dimensional) MCMC * 855
24.4 Speed and accuracy of MCMC 856
24.4.1 The burn-in phase 85624.4.2 Mixing rates of Markov chains * 85724.4.3 Practical convergence diagnostics 85824.4.4 Accuracy of MCMC 86024.4.5 How many chains? 862
24.5 Auxiliary variable MCMC * 863

24.5.1 Auxiliary variable sampling for logistic regression 86324.5.2 Slice sampling 86424.5.3 Swendsen Wang 86624.5.4 Hybrid/Hamiltonian MCMC * 868
24.6 Annealing methods 868
24.6.1 Simulated annealing 86924.6.2 Annealed importance sampling 87124.6.3 Parallel tempering 871
24.7 Approximating the marginal likelihood 872
24.7.1 The candidate method 87224.7.2 Harmonic mean estimate 87224.7.3 Annealed importance sampling 873
25 Clustering 875
25.1 Introduction 875

25.1.1 Measuring (dis)similarity 87525.1.2 Evaluating the output of clustering methods * 876
25.2 Dirichlet process mixture models 879
25.2.1 From ﬁnite to inﬁnite mixture models 87925.2.2 The Dirichlet process 882

CONTENTS xxiii
25.2.3 Applying Dirichlet processes to mixture modeling 885
25.2.4 Fitting a DP mixture model 886
25.3 Affinity propagation 88725.4 Spectral clustering 890
25.4.1 Graph Laplacian 89125.4.2 Normalized graph Laplacian 89225.4.3 Example 893
25.5 Hierarchical clustering 893
25.5.1 Agglomerative clustering 89525.5.2 Divisive clustering 89825.5.3 Choosing the number of clusters 89925.5.4 Bayesian hierarchical clustering 899
25.6 Clustering datapoints and features 901
25.6.1 Biclustering 90325.6.2 Multi-view clustering 903
26 Graphical model structure learning 907
26.1 Introduction 90726.2 Structure learning for knowledge discovery 908
26.2.1 Relevance networks 90826.2.2 Dependency networks 909
26.3 Learning tree structures 910
26.3.1 Directed or undirected tree? 91126.3.2 Chow-Liu algorithm for ﬁnding the ML tree structure 91226.3.3 Finding the MAP forest 91226.3.4 Mixtures of trees 914
26.4 Learning DAG structures 914

26.4.1 Markov equivalence 91426.4.2 Exact structural inference 91626.4.3 Scaling up to larger graphs 920
26.5 Learning DAG structure with latent variables 922
26.5.1 Approximating the marginal likelihood when we have missing data 92226.5.2 Structural EM 92526.5.3 Discovering hidden variables 92626.5.4 Case study: Google’s Rephil 92826.5.5 Structural equation models * 929
26.6 Learning causal DAGs 931

26.6.1 Causal interpretation of DAGs 93126.6.2 Using causal DAGs to resolve Simpson’s paradox 93326.6.3 Learning causal DAG structures 935
26.7 Learning undirected Gaussian graphical models 938
26.7.1 MLE for a GGM 93826.7.2 Graphical lasso 93926.7.3 Bayesian inference for GGM structure * 94126.7.4 Handling non-Gaussian data using copulas * 942

xxiv CONTENTS
26.8 Learning undirected discrete graphical models 942
26.8.1 Graphical lasso for MRFs/CRFs 942
26.8.2 Thin junction trees 944
27 Latent variable models for discrete data 945
27.1 Introduction 94527.2 Distributed state LVMs for discrete data 946
27.2.1 Mixture models 94627.2.2 Exponential family PCA 94727.2.3 LDA and mPCA 94827.2.4 GaP model and non-negative matrix factorization 949
27.3 Latent Dirichlet allocation (LDA) 950
27.3.1 Basics 95027.3.2 Unsupervised discovery of topics 95327.3.3 Quantitatively evaluating LDA as a language model 95327.3.4 Fitting using (collapsed) Gibbs sampling 95527.3.5 Example 95627.3.6 Fitting using batch variational inference 95727.3.7 Fitting using online variational inference 95927.3.8 Determining the number of topics 960
27.4 Extensions of LDA 961
27.4.1 Correlated topic model 96127.4.2 Dynamic topic model 96227.4.3 LDA-HMM 96327.4.4 Supervised LDA 967
27.5 LVMs for graph-structured data 970

27.5.1 Stochastic block model 97127.5.2 Mixed membership stochastic block model 97327.5.3 Relational topic model 974
27.6 LVMs for relational data 975
27.6.1 Inﬁnite relational model 97627.6.2 Probabilistic matrix factorization for collaborative ﬁltering 979
27.7 Restricted Boltzmann machines (RBMs) 983
27.7.1 Varieties of RBMs 98527.7.2 Learning RBMs 98727.7.3 Applications of RBMs 991
28 Deep learning 995
28.1 Introduction 99528.2 Deep generative models 995

28.2.1 Deep directed networks 99628.2.2 Deep Boltzmann machines 99628.2.3 Deep belief networks 99728.2.4 Greedy layer-wise learning of DBNs 998
28.3 Deep neural networks 999

CONTENTS xxv
28.3.1 Deep multi-layer perceptrons 999
28.3.2 Deep auto-encoders 100028.3.3 Stacked denoising auto-encoders 1001
28.4 Applications of deep networks 1001
28.4.1 Handwritten digit classiﬁcation using DBNs 100128.4.2 Data visualization and feature discovery using deep auto-encoders 100228.4.3 Information retrieval using deep auto-encoders (semantic hashing) 100328.4.4 Learning audio features using 1d convolutional DBNs 100428.4.5 Learning image features using 2d convolutional DBNs 1005
28.5 Discussion 1005
Notation 1009
Bibliography 1015
Indexes 1047
Index to code 1047Index to keywords 1050

Preface
With the ever increasing amounts of data in electronic form, the need for automated methods
for data analysis continues to grow. The goal of machine learning is to develop methods thatcan automatically detect patterns in data, and then to use the uncovered patterns to predictfuture data or other outcomes of interest. Machine learning is thus closely related to the ﬁeldsof statistics and data mining, but differs slightly in terms of its emphasis and terminology. Thisbook provides a detailed introduction to the ﬁeld, and includes worked examples drawn fromapplication domains such as molecular biology, text processing, computer vision, and robotics.
Target audience

This book is suitable for upper-level undergraduate students and beginning graduate students incomputer science, statistics, electrical engineering, econometrics, or any one else who has theappropriate mathematical background. Speciﬁcally, the reader is assumed to already be familiarwith basic multivariate calculus, probability, linear algebra, and computer programming. Priorexposure to statistics is helpful but not necessary.
A probabilistic approach
This books adopts the view that the best way to make machines that can learn from data is touse the tools of probability theory, which has been the mainstay of statistics and engineering forcenturies. Probability theory can be applied to any problem involving uncertainty. In machinelearning, uncertainty comes in many forms: what is the best prediction (or decision) given somedata? what is the best model given some data? what measurement should I perform next? etc.

The systematic application of probabilistic reasoning to all inferential problems, including
inferring parameters of statistical models, is sometimes called a Bayesian approach. However,this term tends to elicit very strong reactions (either positive or negative, depending on whoyou ask), so we prefer the more neutral term “probabilistic approach”. Besides, we will oftenuse techniques such as maximum likelihood estimation, which are not Bayesian methods, butcertainly fall within the probabilistic paradigm.

Rather than describing a cookbook of different heuristic methods, this book stresses a princi-
pled model-based approach to machine learning. For any given model, a variety of algorithms

xxviii Preface
can often be applied. Conversely, any given algorithm can often be applied to a variety of
models. This kind of modularity, where we distinguish model from algorithm, is good pedagogyand good engineering.
We will often use the language of graphical models to specify our models in a concise and
intuitive way. In addition to aiding comprehension, the graph structure aids in developingefficient algorithms, as we will see. However, this book is not primarily about graphical models;it is about probabilistic modeling in general.
A practical approach
Nearly all of the methods described in this book have been implemented in a MATLABsoftware
package called PMTK, which stands for probabilistic modeling toolkit. This is freely available
from pmtk3.googlecode .com(the digit 3 refers to the third edition of the toolkit, which is the

one used in this version of the book). There are also a variety of supporting ﬁles, written by otherpeople, available at pmtksupport .googlecode .com. These will be downloaded automatically,
if you follow the setup instructions described on the PMTK website.
MATLABis a high-level, interactive scripting language ideally suited to numerical computation
and data visualization, and can be purchased from www.mathworks .com. Some of the code
requires the Statistics toolbox, which needs to be purchased separately. There is also a freeversion of Matlab called Octave, available at http://www .gnu.org/software/octave/ , which
supports most of the functionality of MATLAB. Some (but not all) of the code in this book also

works in Octave. See the PMTK website for details.
PMTK was used to generate many of the ﬁgures in this book; the source code for these ﬁgures
is included on the PMTK website, allowing the reader to easily see the effects of changing thedata or algorithm or parameter settings. The book refers to ﬁles by name, e.g., naiveBayesFit .
In order to ﬁnd the corresponding ﬁle, you can use two methods: within Matlab you can typewhich naiveBayesFit and it will return the full path to the ﬁle; or, if you do not have Matlab

but want to read the source code anyway, you can use your favorite search engine, which shouldreturn the corresponding ﬁle from the pmtk3.googlecode .comwebsite.
Details on how to use PMTK can be found on the website, which will be udpated over time.
Details on the underlying theory behind these methods can be found in this book.

A book this large is obviously a team effort. I would especially like to thank the following people:my wife Margaret, for keeping the home ﬁres burning as I toiled away in my office for the last sixyears; Matt Dunham, who created many of the ﬁgures in this book, and who wrote much of thecode in PMTK; Baback Moghaddam, who gave extremely detailed feedback on every page of anearlier draft of the book; Chris Williams, who also gave very detailed feedback; Cody Severinskiand Wei-Lwun Lu, who assisted with ﬁgures; generations of UBC students, who gave helpfulcomments on earlier drafts; Daphne Koller, Nir Friedman, and Chris Manning, for letting me usetheir latex style ﬁles; Stanford University, Google Research and Skyline College for hosting meduring part of my sabbatical; and various Canadian funding agencies (NSERC, CRC and CIFAR)who have supported me ﬁnancially over the years.

In addition, I would like to thank the following people for giving me helpful feedback on

Preface xxix
parts of the book, and/or for sharing ﬁgures, code, exercises or even (in some cases) text: David
Blei, Hannes Bretschneider, Greg Corrado, Arnaud Doucet, Mario Figueiredo, Nando de Freitas,Mark Girolami, Gabriel Goh, Tom Griffiths, Katherine Heller, Geoff Hinton, Aapo Hyvarinen,Tommi Jaakkola, Mike Jordan, Charles Kemp, Emtiyaz Khan, Bonnie Kirkpatrick, Daphne Koller,Zico Kolter, Honglak Lee, Julien Mairal, Andrew McPherson, Tom Minka, Ian Nabney, ArthurPope, Carl Rassmussen, Ryan Rifkin, Ruslan Salakhutdinov, Mark Schmidt, Daniel Selsam, DavidSontag, Erik Sudderth, Josh Tenenbaum, Kai Yu, Martin Wainwright, Yair Weiss.
Kevin Patrick Murphy
Palo Alto, CaliforniaJune 2012

1Introduction
1.1 Machine learning: what and why?
We are drowning in information and starving for knowledge. — John Naisbitt.
We are entering the era of big data. For example, there are about 1 trillion web pages1; one
hour of video is uploaded to YouTube every second, amounting to 10 years of content every
day2; the genomes of 1000s of people, each of which has a length of 3.8×109base pairs, have
been sequenced by various labs; Walmart handles more than 1M transactions per hour and hasdatabases containing more than 2.5 petabytes (2.5 ×10
15) of information (Cukier 2010); and so
on.
This deluge of data calls for automated methods of data analysis, which is what machine
learning provides. In particular, we deﬁne machine learning as a set of methods that can
automatically detect patterns in data, and then use the uncovered patterns to predict futuredata, or to perform other kinds of decision making under uncertainty (such as planning how tocollect more data!).

on.
This books adopts the view that the best way to solve such problems is to use the tools
of probability theory. Probability theory can be applied to any problem involving uncertainty.In machine learning, uncertainty comes in many forms: what is the best prediction about thefuture given some past data? what is the best model to explain some data? what measurementshould I perform next? etc. The probabilistic approach to machine learning is closely related tothe ﬁeld of statistics, but differs slightly in terms of its emphasis and terminology
3.

3.
We will describe a wide variety of probabilistic models, suitable for a wide variety of data and

3.
tasks. We will also describe a wide variety of algorithms for learning and using such models.The goal is not to develop a cook book of ad hoc techiques, but instead to present a uniﬁedview of the ﬁeld through the lens of probabilistic modeling and inference. Although we will payattention to computational efficiency, details on how to scale these methods to truly massivedatasets are better described in other books, such as (Rajaraman and Ullman 2011; Bekkermanet al. 2011).
1.http://googleblog .blogspot .com/2008/07/we-knew-web-was-big .html
2. Source: http://www .youtube .com/t/press_statistics .
3. Rob Tibshirani, a statistician at Stanford university, has created an amusing comparison between machine learning
and statistics, available at http://www-stat .stanford .edu/~tibs/stat315a/glossary .pdf.

2 Chapter 1. Introduction
It should be noted, however, that even when one has an apparently massive data set, the
effective number of data points for certain cases of interest might be quite small. In fact, data
across a variety of domains exhibits a property known as the long tail, which means that a
few things (e.g., words) are very common, but most things are quite rare (see Section 2.4.6 fordetails). For example, 20% of Google searches each day have never been seen before
4. This
means that the core statistical issues that we discuss in this book, concerning generalizing fromrelatively small samples sizes, are still very relevant even in the big data era.
1.1.1 Types of machine learning
Machine learning is usually divided into two main types. In the predictive orsupervised
learning approach, the goal is to learn a mapping from inputs xto outputs y, given a labeled
set of input-output pairs D={(xi,yi)}N
i=1.H e r eDis called the training set, and Nis the
number of training examples.

In the simplest setting, each training input xiis aD-dimensional vector of numbers, rep-
resenting, say, the height and weight of a person. These are called features, attributes or
covariates. In general, however, xicould be a complex structured object, such as an image, a
sentence, an email message, a time series, a molecular shape, a graph, etc.
Similarly the form of the output or response variable can in principle be anything, but

most methods assume that yiis a categorical ornominal variable from some ﬁnite set,
yi∈{1,...,C}(such as male or female), or that yiis a real-valued scalar (such as income
level). When yiis categorical, the problem is known as classiﬁcation orpattern recognition,
and when yiis real-valued, the problem is known as regression. Another variant, known as
ordinal regression , occurs where label space Yhas some natural ordering, such as grades A–F.
The second main type of machine learning is the descriptive orunsupervised learning

approach. Here we are only given inputs, D={xi}Ni=1, and the goal is to ﬁnd “interesting
patterns” in the data. This is sometimes called knowledge discovery. This is a much less
well-deﬁned problem, since we are not told what kinds of patterns to look for, and there is no
obvious error metric to use (unlike supervised learning, where we can compare our predictionofyfor a given xto the observed value).
There is a third type of machine learning, known as reinforcement learning, which is

somewhat less commonly used. This is useful for learning how to act or behave when givenoccasional reward or punishment signals. (For example, consider how a baby learns to walk.)Unfortunately, RL is beyond the scope of this book, although we do discuss decision theoryin Section 5.7, which is the basis of RL. See e.g., (Kaelbling et al. 1996; Sutton and Barto 1998;Russell and Norvig 2010; Szepesvari 2010; Wiering and van Otterlo 2012) for more informationon RL.
4.

4.
http://certifiedknowledge .org/blog/are-search-queries-becoming-even-more-unique-statistic
s-from-google .

1.2. Supervised learning 3
(a)
 (b)
Figure 1.1 Left: Some labeled training examples of colored shapes, along with 3 unlabeled test cases.
Right: Representing the training data as an N×Ddesign matrix. Row irepresents the feature vector xi.
The last column is the label, yi∈{0,1}. Based on a ﬁgure by Leslie Kaelbling.
1.2 Supervised learning
We begin our investigation of machine learning by discussing supervised learning, which is the
form of ML most widely used in practice.
1.2.1 Classiﬁcation
In this section, we discuss classiﬁcation. Here the goal is to learn a mapping from inputs x
to outputs y,w h e r ey∈{1,...,C}, withCbeing the number of classes. If C=2, this is
called binary classiﬁcation (in which case we often assume y∈{0,1}); ifC>2, this is called
multiclass classiﬁcation . If the class labels are not mutually exclusive (e.g., somebody may be
classiﬁed as tall and strong), we call it multi-label classiﬁcation , but this is best viewed as

predicting multiple related binary class labels (a so-called multiple output model ). When we
use the term “classiﬁcation”, we will mean multiclass classiﬁcation with a single output, unless
we state otherwise.
One way to formalize the problem is as function approximation . We assume y=f(x)for
some unknown function f, and the goal of learning is to estimate the function fgiven a labeled
training set, and then to make predictions using ˆy=ˆf(x). (We use the hat symbol to denote

an estimate.) Our main goal is to make predictions on novel inputs, meaning ones that we have
not seen before (this is called generalization ), since predicting the response on the training set
is easy (we can just look up the answer).
1.2.1.1 Example
As a simple toy example of classiﬁcation, consider the problem illustrated in Figure 1.1(a). We
have two classes of object which correspond to labels 0 and 1. The inputs are colored shapes.
These have been described by a set of Dfeatures or attributes, which are stored in an N×D

design matrix X, shown in Figure 1.1(b). The input features xcan be discrete, continuous or a
combination of the two. In addition to the inputs, we have a vector of training labels y.
In Figure 1.1, the test cases are a blue crescent, a yellow circle and a blue arrow. None of
these have been seen before. Thus we are required to generalize beyond the training set. A

4 Chapter 1. Introduction
reasonable guess is that blue crescent should be y=1, since all blue shapes are labeled 1 in the
training set. The yellow circle is harder to classify, since some yellow things are labeled y=1
and some are labeled y=0, and some circles are labeled y=1and some y=0. Consequently
it is not clear what the right label should be in the case of the yellow circle. Similarly, the correct
label for the blue arrow is unclear.
1.2.1.2 The need for probabilistic predictions
To handle ambiguous cases, such as the yellow circle above, it is desirable to return a probability.The reader is assumed to already have some familiarity with basic concepts in probability. Ifnot, please consult Chapter 2 for a refresher, if necessary.
We will denote the probability distribution over possible labels, given the input vector xand
training set Dbyp(y|x,D). In general, this represents a vector of length C. (If there are just two

classes, it is sufficient to return the single number p(y=1|x,D), sincep(y=1|x,D)+p(y=
0|x,D)=1.) In our notation, we make explicit that the probability is conditional on the test
inputx, as well as the training set D, by putting these terms on the right hand side of the
conditioning bar |. We are also implicitly conditioning on the form of model that we use to make

predictions. When choosing between different models, we will make this assumption explicit bywritingp(y|x,D,M),w h e r eMdenotes the model. However, if the model is clear from context,
we will drop Mfrom our notation for brevity.
Given a probabilistic output, we can always compute our “best guess” as to the “true label”
using
ˆy=ˆf(x)=
Cargmax
c=1p(y=c|x,D) (1.1)
This corresponds to the most probable class label, and is called the modeof the distribution
p(y|x,D); it is also known as a MAP estimate (MAP stands for maximum a posteriori). Using

using
ˆy=ˆf(x)=
Cargmax
the most probable label makes intuitive sense, but we will give a more formal justiﬁcation forthis procedure in Section 5.7.
Now consider a case such as the yellow circle, where p(ˆy|x,D)is far from 1.0. In such a

case we are not very conﬁdent of our answer, so it might be better to say “I don’t know” insteadof returning an answer that we don’t really trust. This is particularly important in domainssuch as medicine and ﬁnance where we may be risk averse, as we explain in Section 5.7.Another application where it is important to assess risk is when playing TV game shows, suchas Jeopardy. In this game, contestants have to solve various word puzzles and answer a varietyof trivia questions, but if they answer incorrectly, they lose money. In 2011, IBM unveiled acomputer system called Watson which beat the top human Jeopardy champion. Watson uses avariety of interesting techniques (Ferrucci et al. 2010), but the most pertinent one for our presentpurposes is that it contains a module that estimates how conﬁdent it is of its answer. The systemonly chooses to “buzz in” its answer if sufficiently conﬁdent it is correct. Similarly, Google has asystem known as SmartASS (ad selection system) that predicts the probability you will

they lose money. In 2011, IBM unveiled acomputer system called Watson which beat the top human Jeopardy champion. Watson uses avariety of interesting techniques (Ferrucci et al. 2010), but the most pertinent one for our presentpurposes is that it contains a module that estimates how conﬁdent it is of its answer. The systemonly chooses to “buzz in” its answer if sufficiently conﬁdent it is correct. Similarly, Google has asystem known as SmartASS (ad selection system) that predicts the probability you will click onan ad based on your search history and other user and ad-speciﬁc features (Metz 2010). Thisprobability is known as the click-through rate orCTR, and can be used to maximize expected

proﬁt. We will discuss some of the basic principles behind systems such as SmartASS later inthis book.

1.2. Supervised learning 5
wordsdocuments
10 20 30 40 50 60 70 80 90 100100
200
300
400
500
600
700
800
900
1000
Figure 1.2 Subset of size 16242 x 100 of the 20-newsgroups data. We only show 1000 rows, for clarity.
Each row is a document (represented as a bag-of-words bit vector), each column is a word. The red
lines separate the 4 classes, which are (in descending order) comp, rec, sci, talk (these are the titles of
USENET groups). We can see that there are subsets of words whose presence or absence is indicative
of the class. The data is available from http://cs .nyu.edu/~roweis/data .html. Figure generated by
newsgroupsVisualize .
1.2.1.3 Real-world applications
Classiﬁcation is probably the most widely used form of machine learning, and has been used
to solve many interesting and often difficult real-world problems. We have already mentioned
some important applciations. We give a few more examples below.
Document classiﬁcation and email spam ﬁltering

Indocument classiﬁcation , the goal is to classify a document, such as a web page or email
message, into one of Cclasses, that is, to compute p(y=c|x,D),w h e r exis some represen-
tation of the text. A special case of this is email spam ﬁltering , where the classes are spam
y=1or hamy=0.
Most classiﬁers assume that the input vector xhas a ﬁxed size. A common way to represent
variable-length documents in feature-vector format is to use a bag of words representation.
This is explained in detail in Section 3.4.4.1, but the basic idea is to deﬁne xij=1iff wordj

occurs in document i. If we apply this transformation to every document in our data set, we get
a binary document ×word co-occurrence matrix: see Figure 1.2 for an example. Essentially the
document classiﬁcation problem has been reduced to one that looks for subtle changes in the
pattern of bits. For example, we may notice that most spam messages have a high probability of
containing the words “buy”, “cheap”, “viagra”, etc. In Exercise 8.1 and Exercise 8.2, you will get

hands-on experience applying various classiﬁcation techniques to the spam ﬁltering problem.

6 Chapter 1. Introduction
(a)
 (b)
 (c)
Figure 1.3 Three types of iris ﬂowers: setosa, versicolor and virginica. Source: http://www .statlab.u
ni-heidelberg .de/data/iris/ . Used with kind permission of Dennis Kramb and SIGNA.sepal lengthsepal lengthsepal width petal length petal widthsepal width petal length petal width
Figure 1.4 Visualization of the Iris data as a pairwise scatter plot. The diagonal plots the marginal
histograms of the 4 features. The off diagonals contain scatterplots of all possible pairs of features. Red
circle = setosa, green diamond = versicolor, blue star = virginica. Figure generated by fisheririsDemo .
Classifying ﬂowers
Figure 1.3 gives another example of classiﬁcation, due to the statistician Ronald Fisher. The goal
is to learn to distinguish three different kinds of iris ﬂower, called setosa, versicolor and virginica.
Fortunately, rather than working directly with images, a botanist has already extracted 4 useful

features or characteristics: sepal length and width, and petal length and width. (Such feature
extraction is an important, but difficult, task. Most machine learning methods use features
chosen by some human. Later we will discuss some methods that can learn good features from
the data.) If we make a scatter plot of the iris data, as in Figure 1.4, we see that it is easy to
distinguish setosas (red circles) from the other two classes by just checking if their petal length

1.2. Supervised learning 7
true class = 7
 true class = 2
 true class = 1
true class = 0
 true class = 4
true class = 4
 true class = 9
 true class = 5
(a)
(b)
Figure 1.5 (a) First 9 test MNIST gray-scale images. (b) Same as (a), but with the features permuted
randomly. Classiﬁcation performance is identical on both versions of the data (assuming the training data
is permuted in an identical way). Figure generated by shuffledDigitsDemo .
or width is below some threshold. However, distinguishing versicolor from virginica is slightly
harder; any decision will need to be based on at least two features. (It is always a good idea
to perform exploratory data analysis , such as plotting the data, before applying a machine
learning method.)
Image classiﬁcation and handwriting recognition

Now consider the harder problem of classifying images directly, where a human has not pre-
processed the data. We might want to classify the image as a whole, e.g., is it an indoors or
outdoors scene? is it a horizontal or vertical photo? does it contain a dog or not? This is called
image classiﬁcation .
In the special case that the images consist of isolated handwritten letters and digits, for
example, in a postal or ZIP code on a letter, we can use classiﬁcation to perform handwriting
recognition . Astandarddatasetusedinthisareaisknownas MNIST,whichstandsfor“Modiﬁed

National Institute of Standards”5. (The term “modiﬁed” is used because the images have been
preprocessed to ensure the digits are mostly in the center of the image.) This dataset contains
60,000 training images and 10,000 test images of the digits 0 to 9, as written by various people.
The images are size 28×28and have grayscale values in the range 0 : 255. See Figure 1.5(a) for
some example images.
Many generic classiﬁcation methods ignore any structure in the input features, such as spatial

layout. Consequently, they can also just as easily handle data that looks like Figure 1.5(b), which
is the same data except we have randomly permuted the order of all the features. (You will
verify this in Exercise 1.1.) This ﬂexibility is both a blessing (since the methods are general
purpose) and a curse (since the methods ignore an obviously useful source of information). We
will discuss methods for exploiting structure in the input features later in the book.
5. Available from http://yann .lecun .com/exdb/mnist/ .

8 Chapter 1. Introduction
(a)
 (b)
Figure 1.6 Example of face detection. (a) Input image (Murphy family, photo taken 5 August 2010). Used
with kind permission of Bernard Diedrich of Sherwood Studios. (b) Output of classiﬁer, which detected 5
faces at different poses. This was produced using the online demo at http://demo .pittpatt .com/. The
classiﬁer was trained on 1000s of manually labeled images of faces and non-faces, and then was applied
to a dense set of overlapping patches in the test image. Only the patches whose probability of containing
a face was sufficiently high were returned. Used with kind permission of Pittpatt.com
Face detection and recognition
A harder problem is to ﬁnd objects within an image; this is called object detection orobject
localization . An important special case of this is face detection . One approach to this problem
is to divide the image into many small overlapping patches at different locations, scales and

orientations, and to classify each such patch based on whether it contains face-like texture or
not. This is called a sliding window detector . The system then returns those locations where
the probability of face is sufficiently high. See Figure 1.6 for an example. Such face detection
systems are built-in to most modern digital cameras; the locations of the detected faces are
used to determine the center of the auto-focus. Another application is automatically blurring
out faces in Google’s StreetView system.

Having found the faces, one can then proceed to perform face recognition , which means
estimating the identity of the person (see Figure 1.10(a)). In this case, the number of class labels
might be very large. Also, the features one should use are likely to be different than in the face
detection problem: for recognition, subtle differences between faces such as hairstyle may be
important for determining identity, but for detection, it is important to be invariant to such
details, and to just focus on the differences between faces and non-faces. For more information

about visual object detection, see e.g., (Szeliski 2010).
1.2.2 Regression
Regression is just like classiﬁcation except the response variable is continuous. Figure 1.7 shows
a simple example: we have a single real-valued input xi∈R, and a single real-valued response
yi∈R. We consider ﬁtting two models to the data: a straight line and a quadratic function.
(We explain how to ﬁt such models below.) Various extensions of this basic problem can arise,

such as having high-dimensional inputs, outliers, non-smooth responses, etc. We will discuss
ways to handle such problems later in the book.

1.3. Unsupervised learning 9
0 5 10 15 20−10−5051015degree 1
(a)0 5 10 15 20−10−5051015degree 2
(b)
Figure 1.7 (a) Linear regression on some 1d data. (b) Same data with polynomial regression (degree 2).
Figure generated by linregPolyVsDegree .
Here are some examples of real-world regression problems.
• Predict tomorrow’s stock market price given current market conditions and other possible
side information.
• Predict the age of a viewer watching a given video on YouTube.
• Predict the location in 3d space of a robot arm end effector, given control signals (torques)
sent to its various motors.
• Predict the amount of prostate speciﬁc antigen (PSA) in the body as a function of a number
of different clinical measurements.
• Predict the temperature at any location inside a building using weather data, time, door
sensors, etc.
1.3 Unsupervised learning
We now consider unsupervised learning, where we are just given output data, without any

inputs. The goal is to discover “interesting structure” in the data; this is sometimes called
knowledge discovery. Unlike supervised learning, we are not told what the desired output isfor each input. Instead, we will formalize our task as one of density estimation, that is, we
want to build models of the form p(x
i|θ). There are two differences from the supervised case.
First, we have written p(xi|θ)instead of p(yi|xi,θ); that is, supervised learning is conditional

density estimation, whereas unsupervised learning is unconditional density estimation. Second,x
iis a vector of features, so we need to create multivariate probability models. By contrast,
in supervised learning, yiis usually just a single variable that we are trying to predict. This

means that for most supervised learning problems, we can use univariate probability models(with input-dependent parameters), which signiﬁcantly simpliﬁes the problem. (We will discussmulti-output classiﬁcation in Chapter 19, where we will see that it also involves multivariateprobability models.)
Unsupervised learning is arguably more typical of human and animal learning. It is also
more widely applicable than supervised learning, since it does not require a human expert to

10 Chapter 1. Introduction
55 60 65 70 75 8080100120140160180200220240260280
heightweight
(a)55 60 65 70 75 8080100120140160180200220240260280
heightweightK=2
(b)
Figure 1.8 (a) The height and weight of some people. (b) A possible clustering using K=2clusters.
Figure generated by kmeansHeightWeight .
manually label the data. Labeled data is not only expensive to acquire6, but it also contains
relatively little information, certainly not enough to reliably estimate the parameters of complex
models. Geoff Hinton, who is a famous professor of ML at the University of Toronto, has said:
When we’re learning to see, nobody’s telling us what the right answers are — we justlook. Every so often, your mother says “that’s a dog”, but that’s very little information.You’d be lucky if you got a few bits of information — even one bit per second — thatway. The brain’s visual system has 10
14neural connections. And you only live for 109
seconds. So it’s no use learning one bit per second. You need more like 105bits per

second. And there’s only one place you can get that much information: from the inputitself. — Geoffrey Hinton, 1996 (quoted in (Gorder 2006)).
Below we describe some canonical examples of unsupervised learning.
1.3.1 Discovering clusters
As a canonical example of unsupervised learning, consider the problem of clustering data into

groups. For example, Figure 1.8(a) plots some 2d data, representing the height and weight ofa group of 210 people. It seems that there might be various clusters, or subgroups, althoughit is not clear how many. Let Kdenote the number of clusters. Our ﬁrst goal is to estimate
the distribution over the number of clusters, p(K|D); this tells us if there are subpopulations
within the data. For simplicity, we often approximate the distribution p(K|D)by its mode,
K
∗= argmax Kp(K|D). In the supervised case, we were told that there are two classes (male

K
and female), but in the unsupervised case, we are free to choose as many or few clusters as welike. Picking a model of the “right” complexity is called model selection, and will be discussedin detail below.
Our second goal is to estimate which cluster each point belongs to. Let z
i∈{1,...,K}
represent the cluster to which data point iis assigned. (z iis an example of a hiddenor
6. The advent of crowd sourcing web sites such as Mechanical Turk, (https://www .mturk .com/mturk/welcome ),
which outsource data processing tasks to humans all over the world, has reduced the cost of labeling data. Nevertheless,
the amount of unlabeled data is still orders of magnitude larger than the amount of labeled data.

1.3. Unsupervised learning 11
−8−6−4−202468
−4−2024−202
(a)−505
−4−2024
(b)
Figure 1.9 (a) A set of points that live on a 2d linear subspace embedded in 3d. The solid red line is the
ﬁrst principal component direction. The dotted black line is the second PC direction. (b) 2D representation
of the data. Figure generated by pcaDemo3d .
latentvariable, since it is never observed in the training set.) We can infer which cluster each
data point belongs to by computing z∗
i= argmaxkp(zi=k|xi,D). This is illustrated in
Figure 1.8(b), where we use different colors to indicate the assignments, assuming K=2.
In this book, we focus on model based clustering, which means we ﬁt a probabilistic model
to the data, rather than running some ad hoc algorithm. The advantages of the model-based
approach are that one can compare different kinds of models in an objective way (in terms ofthe likelihood they assign to the data), we can combine them together into larger systems, etc.

Here are some real world applications of clustering.
• In astronomy, the autoclass system (Cheeseman et al. 1988) discovered a new type of star,
based on clustering astrophysical measurements.
• In e-commerce, it is common to cluster users into groups, based on their purchasing or
web-surﬁng behavior, and then to send customized targeted advertising to each group (seee.g., (Berkhin 2006)).
• In biology, it is common to cluster ﬂow-cytometry data into groups, to discover different

sub-populations of cells (see e.g., (Lo et al. 2009)).
1.3.2 Discovering latent factors
When dealing with high dimensional data, it is often useful to reduce the dimensionality byprojecting the data to a lower dimensional subspace which captures the “essence” of the data.This is called dimensionality reduction. A simple example is shown in Figure 1.9, where we

project some 3d data down to a 2d plane. The 2d approximation is quite good, since most pointslie close to this subspace. Reducing to 1d would involve projecting points onto the red line inFigure 1.9(a); this would be a rather poor approximation. (We will make this notion precise inChapter 12.)
The motivation behind this technique is that although the data may appear high dimensional,
there may only be a small number of degrees of variability, corresponding to latent factors.F o r

example, when modeling the appearance of face images, there may only be a few underlyinglatent factors which describe most of the variability, such as lighting, pose, identity, etc, asillustrated in Figure 1.10.

12 Chapter 1. Introduction
(a)
 (b)
Figure 1.10 a) 25 randomly chosen 64×64pixel images from the Olivetti face database. (b) The mean
and the ﬁrst three principal component basis vectors (eigenfaces). Figure generated by pcaImageDemo .
When used as input to other statistical models, such low dimensional representations often
result in better predictive accuracy, because they focus on the “essence” of the object, ﬁltering
out inessential features. Also, low dimensional representations are useful for enabling fast
nearest neighbor searches and two dimensional projections are very useful for visualizing high
dimensional data.
The most common approach to dimensionality reduction is called principal components
analysis orPCA. This can be thought of as an unsupervised version of (multi-output) linear
regression, where we observe the high-dimensional response y, but not the low-dimensional
“cause”z. Thus the model has the form z→y; we have to “invert the arrow”, and infer the

latent low-dimensional zfrom the observed high-dimensional y. See Section 12.1 for details.
Dimensionality reduction, and PCA in particular, has been applied in many different areas.
Some examples include the following:
• In biology, it is common to use PCA to interpret gene microarray data, to account for the
fact that each measurement is usually the result of many genes which are correlated in their
behavior by the fact that they belong to different biological pathways.

• In natural language processing, it is common to use a variant of PCA called latent semantic
analysis for document retrieval (see Section 27.2.2).
• In signal processing (e.g., of acoustic or neural signals), it is common to use ICA (which is a
variant of PCA) to separate signals into their different sources (see Section 12.6).
• In computer graphics, it is common to project motion capture data to a low dimensional
space, and use it to create animations. See Section 15.5 for one way to tackle such problems.

1.3. Unsupervised learning 13
Figure 1.11 A sparse undirected Gaussian graphical model learned using graphical lasso (Section 26.7.2)
applied to some ﬂow cytometry data (from (Sachs et al. 2005)), which measures the phosphorylation status
of 11 proteins. Figure generated by ggmLassoDemo .
1.3.3 Discovering graph structure
Sometimes we measure a set of correlated variables, and we would like to discover which ones
are most correlated with which others. This can be represented by a graph G, in which nodes
represent variables, and edges represent direct dependence between variables (we will make
this precise in Chapter 10, when we discuss graphical models). We can then learn this graph
structure from data, i.e., we compute ˆG=a r g m a x p(G|D).
As with unsupervised learning in general, there are two main applications for learning sparse
graphs: to discover new knowledge, and to get better joint probability density estimators. We
now give somes example of each.

• Much of the motivation for learning sparse graphical models comes from the systems biology
community. For example, suppose we measure the phosphorylation status of some proteins
in a cell (Sachs et al. 2005). Figure 1.11 gives an example of a graph structure that was learned
from this data (using methods discussed in Section 26.7.2). As another example, Smith et al.
(2006) showed that one can recover the neural “wiring diagram” of a certain kind of bird
from time-series EEG data. The recovered structure closely matched the known functional

connectivity of this part of the bird brain.
• In some cases, we are not interested in interpreting the graph structure, we just want to
use it to model correlations and to make predictions. One example of this is in ﬁnancial
portfolio management, where accurate models of the covariance between large numbers of
different stocks is important. Carvalho and West (2007) show that by learning a sparse graph,
and then using this as the basis of a trading strategy, it is possible to outperform (i.e., make

more money than) methods that do not exploit sparse graphs. Another example is predicting
traffic jams on the freeway. Horvitz et al. (2005) describe a deployed system called JamBayes
for predicting traffic ﬂow in the Seattle area; predictions are made using a graphical model
whose structure was learned from data.

14 Chapter 1. Introduction
(a)
 (b)
Figure 1.12 (a) A noisy image with an occluder. (b) An estimate of the underlying pixel intensities, based
on a pairwise MRF model. Source: Figure 8 of (Felzenszwalb and Huttenlocher 2006). Used with kind
permission of Pedro Felzenszwalb.
1.3.4 Matrix completion
Sometimes we have missing data, that is, variables whose values are unknown. For example, we
might have conducted a survey, and some people might not have answered certain questions.
Or we might have various sensors, some of which fail. The corresponding design matrix will
then have “holes” in it; these missing entries are often represented by NaN, which stands for
“not a number”. The goal of imputation is to infer plausible values for the missing entries. This
is sometimes called matrix completion . Below we give some example applications.
1.3.4.1 Image inpainting
An interesting example of an imputation-like task is known as image inpainting . The goal is

to “ﬁll in” holes (e.g., due to scratches or occlusions) in an image with realistic texture. This is
illustrated in Figure 1.12, where we denoise the image, as well as impute the pixels hidden behind
the occlusion. This can be tackled by building a joint probability model of the pixels, given a
set of clean images, and then inferring the unknown variables (pixels) given the known variables
(pixels). This is somewhat like masket basket analysis, except the data is real-valued and spatially

structured, so the kinds of probability models we use are quite different. See Sections 19.6.2.7
and 13.8.4 for some possible choices.
1.3.4.2 Collaborative ﬁltering
Another interesting example of an imputation-like task is known as collaborative ﬁltering .A
common example of this concerns predicting which movies people will want to watch based
on how they, and other people, have rated movies which they have already seen. The key idea

is that the prediction is not based on features of the movie or user (although it could be), but
merely on a ratings matrix. More precisely, we have a matrix XwhereX(m,u)is the rating

a different subset. In particular, for any given user u, we might want to predict which of the
unrated movies he/she is most likely to want to watch.
In order to encourage research in this area, the DVD rental company Netﬂix created a com-
petition, launched in 2006, with a $1M USD prize (see http://netflixprize .com/). In
particular, they provided a large matrix of ratings, on a scale of 1 to 5, for ∼18kmovies
created by ∼500kusers. The full matrix would have ∼9×109entries, but only about 1%
of the entries are observed, so the matrix is extremely sparse. A subset of these are used for
training, and the rest for testing, as shown in Figure 1.13. The goal of the competition was to

predict more accurately than Netﬂix’s existing system. On 21 September 2009, the prize wasawarded to a team of researchers known as “BellKor’s Pragmatic Chaos”. Section 27.6.2 discussessome of their methodology. Further details on the teams and their methods can be found athttp://www .netflixprize .com/community/viewtopic .php?id=1537 .
1.3.4.3 Market basket analysis
In commercial data mining, there is much interest in a task called market basket analysis. The

data consists of a (typically very large but sparse) binary matrix, where each column representsan item or product, and each row represents a transaction. We set x
ij=1if itemjwas
purchased on the i’th transaction. Many items are purchased together (e.g., bread and butter),

so there will be correlations amongst the bits. Given a new partially observed bit vector,representing a subset of items that the consumer has bought, the goal is to predict which otherbits are likely to turn on, representing other items the consumer might be likely to buy. (Unlikecollaborative ﬁltering, we often assume there is no missing data in the training data, since weknow the past shopping behavior of each customer.)
This task arises in other domains besides modeling purchasing patterns. For example, similar

techniques can be used to model dependencies between ﬁles in complex software systems. Inthis case, the task is to predict, given a subset of ﬁles that have been changed, which other onesneed to be updated to ensure consistency (see e.g., (Hu et al. 2010)).
It is common to solve such tasks using frequent itemset mining, which create association
rules (see e.g., (Hastie et al. 2009, sec 14.2) for details). Alternatively, we can adopt a probabilisticapproach, and ﬁt a joint density model p(x
1,...,x D)to the bit vectors, see e.g., (Hu et al.

16 Chapter 1. Introduction
(a)
 (b)
Figure 1.14 (a) Illustration of a K-nearest neighbors classiﬁer in 2d for K=3. The 3 nearest neighbors
of test point x1have labels 1, 1 and 0, so we predict p(y=1|x1,D,K=3 )=2 /3. The 3 nearest
neighbors of test point x2have labels 0, 0, and 0, so we predict p(y=1|x2,D,K=3 )=0 /3. (b)
Illustration of the Voronoi tesselation induced by 1-NN. Based on Figure 4.13 of (Duda et al. 2001). Figure
generated by knnVoronoi .
2010). Such models often have better predictive acccuracy than association rules, although they
may be less interpretible. This is typical of the difference between data mining and machine
learning: in data mining, there is more emphasis on interpretable models, whereas in machine
learning, there is more emphasis on accurate models.
1.4 Some basic concepts in machine learning
In this Section, we provide an introduction to some key ideas in machine learning. We will
expand on these concepts later in the book, but we introduce them brieﬂy here, to give a ﬂavor

of things to come.
1.4.1 Parametric vs non-parametric models
In this book, we will be focussing on probabilistic models of the form p(y|x)orp(x), depending
on whether we are interested in supervised or unsupervised learning respectively. There are
many ways to deﬁne such models, but the most important distinction is this: does the model
have a ﬁxed number of parameters, or does the number of parameters grow with the amount
of training data? The former is called a parametric model , and the latter is called a non-

parametric model . Parametric models have the advantage of often being faster to use, but the
disadvantage of making stronger assumptions about the nature of the data distributions. Non-
parametric models are more ﬂexible, but often computationally intractable for large datasets.
We will give examples of both kinds of models in the sections below. We focus on supervised
learning for simplicity, although much of our discussion also applies to unsupervised learning.

1.4.2 A simple non-parametric classiﬁer: K-nearest neighbors
A simple example of a non-parametric classiﬁer is the Knearest neighbor (KNN) classiﬁer.
This simply “looks at” the Kpoints in the training set that are nearest to the test input x,

1.4. Some basic concepts in machine learning 17
−3 −2 −1 0 1 2 3−2−1012345train
(a)
p(y=1|data,K=10)
  
20 40 60 80 10020406080100120
(b)
p(y=2|data,K=10)
  
(c)−3 −2 −1 0 1 2 3−2−1012345predicted label,  K=10
  
c1
c2
c3
(d)
Figure 1.15 (a) Some synthetic 3-class training data in 2d. (b) Probability of class 1 for KNN with K=1 0.
(c) Probability of class 2. (d) MAP estimate of class label. Figure generated by knnClassifyDemo .
counts how many members of each class are in this set, and returns that empirical fraction as
the estimate, as illustrated in Figure 1.14. More formally,
p(y=c|x,D,K)=1
K∑
i∈NK(x,D)I(yi=c) (1.2)
whereNK(x,D)are the (indices of the) Knearest points to xinDandI(e)is the indicator
function deﬁned as follows:
I(e)={
1ifeis true
0ifeis false(1.3)
This method is an example of memory-based learning orinstance-based learning . It can

K∑
I(e)={
be derived from a probabilistic framework as explained in Section 14.7.3. The most common

18 Chapter 1. Introduction
s
11
0
(a)0 0.2 0.4 0.6 0.8 100.10.20.30.40.50.60.70.80.91
Fraction of data in neighborhoodEdge length of cube
d=1d=3d=5d=7d=10
(b)
Figure 1.16 Illustration of the curse of dimensionality. (a) We embed a small cube of side sinside a larger
unit cube. (b) We plot the edge length of a cube needed to cover a given volume of the unit cube as a
function of the number of dimensions. Based on Figure 2.6 from (Hastie et al. 2009). Figure generated bycurseDimensionality .
distance metric to use is Euclidean distance (which limits the applicability of the technique to
data which is real-valued), although other metrics can be used.
Figure 1.15 gives an example of the method in action, where the input is two dimensional, we
have three classes, and K=1 0. (We discuss the effect of Kbelow.) Panel (a) plots the training
data. Panel (b) plots p(y=1|x,D)wherexis evaluated on a grid of points. Panel (c) plots
p(y=2|x,D). We do not need to plot p(y=3|x,D), since probabilities sum to one. Panel (d)

plots the MAP estimate ˆy(x) = argmaxc(y=c|x,D).
A KNN classiﬁer with K=1induces a Voronoi tessellation of the points (see Figure 1.14(b)).
This is a partition of space which associates a region V(xi)with each point xiin such a way
that all points in V(xi)are closer to xithan to any other point. Within each cell, the predicted
label is the label of the corresponding training point.
1.4.3 The curse of dimensionality

The KNN classiﬁer is simple and can work quite well, provided it is given a good distance metricand has enough labeled training data. In fact, it can be shown that the KNN classiﬁer can comewithin a factor of 2 of the best possible performance if N→∞(Cover and Hart 1967).
However, the main problem with KNN classiﬁers is that they do not work well with high
dimensional inputs. The poor performance in high dimensional settings is due to the curse of
dimensionality.

To explain the curse, we give some examples from (Hastie et al. 2009, p22). Consider applying
a KNN classiﬁer to data where the inputs are uniformly distributed in the D-dimensional unit
cube. Suppose we estimate the density of class labels around a test point xby “growing” a
hyper-cube around xuntil it contains a desired fraction fof the data points. The expected edge
length of this cube will be e
D(f)=f1/D.I fD=1 0, and we want to base our estimate on 10%

1.4. Some basic concepts in machine learning 19
−3 −2 −1 0 1 2 300.050.10.150.20.250.30.350.4PDF
(a)
 (b)
Figure 1.17 (a) A Gaussian pdf with mean 0 and variance 1. Figure generated by gaussPlotDemo . (b)
Visualization of the conditional density model p(y|x,θ)=N(y|w0+w1x,σ2). The density falls off
exponentially fast as we move away from the regression line. Figure generated by linregWedgeDemo2 .
of the data, we have e10(0.1) = 0.8, so we need to extend the cube 80% along each dimension
aroundx. Even if we only use 1% of the data, we ﬁnd e10(0.01) = 0.63: see Figure 1.16. Since
the entire range of the data is only 1 along each dimension, we see that the method is no longer
very local, despite the name “nearest neighbor”. The trouble with looking at neighbors that are
so far away is that they may not be good predictors about the behavior of the input-output
function at a given point.
1.4.4 Parametric models for classiﬁcation and regression

The main way to combat the curse of dimensionality is to make some assumptions about
the nature of the data distribution (either p(y|x)for a supervised problem or p(x)for an
unsupervised problem). These assumptions, known as inductive bias , are often embodied in
the form of a parametric model , which is a statistical model with a ﬁxed number of parameters.
Below we brieﬂy describe two widely used examples; we will revisit these and other models in
much greater depth later in the book.
1.4.5 Linear regression

One of the most widely used models for regression is known as linear regression . This asserts
that the response is a linear function of the inputs. This can be written as follows:
y(x)=wTx+ϵ=D∑
j=1wjxj+ϵ (1.4)
wherewTxrepresents the inner or scalar product between the input vector xand the model’s
weight vector w7, andϵis the residual error between our linear predictions and the true
response.
7. In statistics, it is more common to denote the regression weights by β.

20 Chapter 1. Introduction
0 5 10 15 20−10−5051015degree 14
(a)0 5 10 15 20−10−5051015degree 20
(b)
Figure 1.18 Polynomial of degrees 14 and 20 ﬁt by least squares to 21 data points. Figure generated by
linregPolyVsDegree .
We often assume that ϵhas a Gaussian8ornormaldistribution. We denote this by ϵ∼
N(μ,σ2),w h e r eμis the mean and σ2is the variance (see Chapter 2 for details). When we plot
this distribution, we get the well-known bell curve shown in Figure 1.17(a).
To make the connection between linear regression and Gaussians more explicit, we can rewrite
the model in the following form:
p(y|x,θ)=N(y|μ(x),σ2(x)) (1.5)
This makes it clear that the model is a conditional probability density. In the simplest case, we
assumeμis a linear function of x,s oμ=wTx, and that the noise is ﬁxed, σ2(x)=σ2.I n
this case, θ=(w,σ2)are the parameters of the model.
For example, suppose the input is 1 dimensional. We can represent the expected response as
follows:
μ(x)=w0+w1x=wTx (1.6)

follows:
wherew0is the intercept or biasterm,w1is the slope, and where we have deﬁned the vector
x=( 1,x). (Prepending a constant 1 term to an input vector is a common notational trick which
allows us to combine the intercept term with the other terms in the model.) If w1is positive,
it means we expect the output to increase as the input increases. This is illustrated in 1d inFigure 1.17(b); a more conventional plot, of the mean response vs x, is shown in Figure 1.7(a).
Linear regression can be made to model non-linear relationships by replacing xwith some

non-linear function of the inputs, φ(x). That is, we use
p(y|x,θ)=N(y|w
Tφ(x),σ2) (1.7)
This is known as basis function expansion. For example, Figure 1.18 illustrates the case where
φ(x)=[ 1,x,x2,...,xd],f o rd=1 4andd=2 0; this is known as polynomial regression.

We will consider other kinds of basis functions later in the book. In fact, many popularmachine learning methods — such as support vector machines, neural networks, classiﬁcationand regression trees, etc. — can be seen as just different ways of estimating basis functionsfrom data, as we discuss in Chapters 14 and 16.
8. Carl Friedrich Gauss (1777–1855) was a German mathematician and physicist.

1.4. Some basic concepts in machine learning 21
−10 −5 0 5 1000.10.20.30.40.50.60.70.80.91
(a)460 480 500 520540560 580 600 62064000.10.20.30.40.50.60.70.80.91
(b)
Figure 1.19 (a) The sigmoid or logistic function. We have sigm(−∞)=0 ,sigm(0) = 0.5 , and
sigm(∞ )=1. Figure generated by sigmoidPlot . (b) Logistic regression for SAT scores. Solid black dots
are the data. The open red circles are the predicted probabilities. The green crosses denote two students
with the same SAT score of 525 (and hence same input representation x) but with different training labels
(one student passed, y=1, the other failed, y=0). Hence this data is not perfectly separable using just
the SAT feature. Figure generated by logregSATdemo .
1.4.6 Logistic regression
We can generalize linear regression to the (binary) classiﬁcation setting by making two changes.
First we replace the Gaussian distribution for ywith a Bernoulli distribution9,which is more
appropriate for the case when the response is binary, y∈{0,1}. That is, we use

p(y|x,w) = Ber(y|μ(x)) (1.8)
whereμ(x)=E[y|x]=p(y=1|x). Second, we compute a linear combination of the inputs,
as before, but then we pass this through a function that ensures 0≤μ(x)≤1by deﬁning
μ(x) = sigm(wTx) (1.9)
wheresigm(η)refers to the sigmoid function, also known as the logisticorlogitfunction.
This is deﬁned as
sigm(η)≜1
1+exp(−η)=eη
eη+1(1.10)
The term “sigmoid” means S-shaped: see Figure 1.19(a) for a plot. It is also known as a squashing
function, since it maps the whole real line to [0,1], which is necessary for the output to be

sigm(η)≜1
interpreted as a probability.
Putting these two steps together we get
p(y|x,w) = Ber(y|sigm(wTx)) (1.11)
This is called logistic regression due to its similarity to linear regression (although it is a form
of classiﬁcation, not regression!).
9. Daniel Bernoulli (1700–1782) was a Dutch-Swiss mathematician and physicist.

22 Chapter 1. Introduction
A simple example of logistic regression is shown in Figure 1.19(b), where we plot
p(yi=1|xi,w) = sigm(w 0+w1xi) (1.12)
wherexiis the SAT10score of student iandyiis whether they passed or failed a class. The
solid black dots show the training data, and the red circles plot p(y=1|xi,ˆw),w h e r eˆware
the parameters estimated from the training data (we discuss how to compute these estimates in
Section 8.3.4).
If we threshold the output probability at 0.5, we can induce a decision rule of the form
ˆy(x)=1⇐⇒p(y=1|x)>0.5 (1.13)
By looking at Figure 1.19(b), we see that sigm(w0+w1x)=0.5forx≈545 =x∗. We can
imagine drawing a vertical line at x=x∗; this is known as a decision boundary. Everything to
the left of this line is classiﬁed as a 0, and everything to the right of the line is classiﬁed as a 1.
We notice that this decision rule has a non-zero error rate even on the training set. This
is because the data is not linearly separable, i.e., there is no straight line we can draw to

separate the 0s from the 1s. We can create models with non-linear decision boundaries usingbasis function expansion, just as we did with non-linear regression. We will see many examplesof this later in the book.
1.4.7 Overﬁtting
When we ﬁt highly ﬂexible models, we need to be careful that we do not overﬁtthe data, that

is, we should avoid trying to model every minor variation in the input, since this is more likelyto be noise than true signal. This is illustrated in Figure 1.18(b), where we see that using a highdegree polynomial results in a curve that is very “wiggly”. It is unlikely that the true functionhas such extreme oscillations. Thus using such a model might result in accurate predictions offuture outputs.
As another example, consider the KNN classiﬁer. The value of Kcan have a large effect on

the behavior of this model. When K=1, the method makes no errors on the training set (since
we just return the labels of the original training points), but the resulting prediction surface isvery “wiggly” (see Figure 1.20(a)). Therefore the method may not work well at predicting futuredata. In Figure 1.20(b), we see that using K=5results in a smoother prediction surface,
because we are averaging over a larger neighborhood. As Kincreases, the predictions becomes

smoother until, in the limit of K=N, we end up predicting the majority label of the whole
data set. Below we discuss how to pick the “right” value of K.
1.4.8 Model selection
When we have a variety of models of different complexity (e.g., linear or logistic regressionmodels with different degree polynomials, or KNN classiﬁers with different values of K), how
should we pick the right one? A natural approach is to compute the misclassiﬁcation rate on

10. SAT stands for “Scholastic Aptitude Test”. This is a standardized test for college admissions used in the United States
(the data in this example is from (Johnson and Albert 1999, p87)).

1.4. Some basic concepts in machine learning 23
−3 −2 −1 0 1 2 3−2−1012345predicted label,  K=1
  
c1
c2
c3
(a)−3 −2 −1 0 1 2 3−2−1012345predicted label,  K=5
  
c1
c2
c3
(b)
Figure 1.20 Prediction surface for KNN on the data in Figure 1.15(a). (a) K=1. (b) K=5. Figure generated by
knnClassifyDemo .
the training set for each method. This is deﬁned as follows:
err(f,D)=1
NN∑
i=1I(f(xi)̸=yi) (1.14)
wheref(x)is our classiﬁer. In Figure 1.21(a), we plot this error rate vs Kfor a KNN classiﬁer
(dotted blue line). We see that increasing Kincreasesour error rate on the training set, because
we are over-smoothing. As we said above, we can get minimal error on the training set by using
K=1, since this model is just memorizing the data.
However, what we care about is generalization error, which is the expected value of the
misclassiﬁcation rate when averaged over future data (see Section 6.3 for details). This can beapproximated by computing the misclassiﬁcation rate on a large independent test set, not used

during model training. We plot the test error vs Kin Figure 1.21(a) in solid red (upper curve).
Now we see a U-shaped curve: for complex models (small K), the method overﬁts, and for
simple models (big K), the method underﬁts. Therefore, an obvious way to pick Kis to pick
the value with the minimum error on the test set (in this example, any value between 10 and100 should be ﬁne).
Unfortunately, when training the model, we don’t have access to the test set (by assumption),
so we cannot use the test set to pick the model of the right complexity.
11However, we can create

a test set by partitioning the training set into two: the part used for training the model, and asecond part, called the validation set, used for selecting the model complexity. We then ﬁt all
the models on the training set, and evaluate their performance on the validation set, and pickthe best. Once we have picked the best, we can reﬁt it to all the available data. If we have aseparate test set, we can evaluate performance on this, in order to estimate the accuracy of ourmethod. (We discuss this in more detail in Section 6.5.3.)

Often we use about 80% of the data for the training set, and 20% for the validation set. But
if the number of training cases is small, this technique runs into problems, because the model
11. In academic settings, we usually do have access to the test set, but we should not use it for model ﬁtting or model
selection, otherwise we will get an unrealistically optimistic estimate of performance of our method. This is one of the
“golden rules” of machine learning research.

24 Chapter 1. Introduction
0 20 40 60 80 100 12000.050.10.150.20.250.30.35
Kmisclassification ratetrain
test
(a)
 (b)
Figure 1.21 (a) Misclassiﬁcation rate vs Kin a K-nearest neighbor classiﬁer. On the left, where Kis
small, the model is complex and hence we overﬁt. On the right, where Kis large, the model is simple
and we underﬁt. Dotted blue line: training set (size 200). Solid red line: test set (size 500). (b) Schematic
of 5-fold cross validation. Figure generated by knnClassifyDemo .
won’t have enough data to train on, and we won’t have enough data to make a reliable estimate
of the future performance.
A simple but popular solution to this is to use cross validation (CV). The idea is simple: we
split the training data into Kfolds; then, for each fold k∈{1,...,K}, we train on all the
folds but the k’th, and test on the k’th, in a round-robin fashion, as sketched in Figure 1.21(b).
We then compute the error averaged over all the folds, and use this as a proxy for the test error.

(Note that each point gets predicted only once, although it will be used for training K−1times.)
It is common to use K=5; this is called 5-fold CV. If we set K=N, then we get a method
called leave-one out cross validation ,o rLOOCV, since in fold i, we train on all the data cases
except for i, and then test on i. Exercise 1.3 asks you to compute the 5-fold CV estimate of the
test error vs K, and to compare it to the empirical test error in Figure 1.21(a).

Choosing Kfor a KNN classiﬁer is a special case of a more general problem known as model
selection , where we have to choose between models with different degrees of ﬂexibility. Cross-
validation is widely used for solving such problems, although we will discuss other approaches
later in the book.
1.4.9 No free lunch theorem
All models are wrong, but some models are useful. — George Box (Box and Draper 1987,
p424).12
Much of machine learning is concerned with devising different models, and different algorithms

p424).12
to ﬁt them. We can use methods such as cross validation to empirically choose the best method
for our particular problem. However, there is no universally best model — this is sometimes
called the no free lunch theorem (Wolpert 1996). The reason for this is that a set of assumptions
that works well in one domain may work poorly in another.
12. George Box is a retired statistics professor at the University of Wisconsin.

1.4. Some basic concepts in machine learning 25
As a consequence of the no free lunch theorem, we need to develop many different types of
models, to cover the wide variety of data that occurs in the real world. And for each model,
there may be many different algorithms we can use to train the model, which make differentspeed-accuracy-complexity tradeoffs. It is this combination of data, models and algorithms thatwe will be studying in the subsequent chapters.
Exercises
Exercise 1.1 KNN classiﬁer on shuffled MNIST data
Runmnist1NNdemo and verify that the misclassiﬁcation rate (on the ﬁrst 1000 test cases) of MNIST of a
1-NN classiﬁer is 3.8%. (If you run it all on all 10,000 test cases, the error rate is 3.09%.) Modify the code
so that you ﬁrst randomly permute the features (columns of the training and test design matrices), as inshuffledDigitsDemo , and then apply the classiﬁer. Verify that the error rate is not changed.
Exercise 1.2 Approximate KNN classiﬁers

Exercises
Use the Matlab/C++ code at http://people .cs.ubc.ca/~mariusm/index .php/FLANN/FLANN to per-
form approximate nearest neighbor search, and combine it with mnist1NNdemo to classify the MNIST data
set. How much speedup do you get, and what is the drop (if any) in accuracy?
Exercise 1.3 CV for KNN
UseknnClassifyDemo to plot the CV estimate of the misclassiﬁcation rate on the test set. Compare this
to Figure 1.21(a). Discuss the similarities and differences to the test error rate.

2 Probability
2.1 Introduction
Probability theory is nothing but common sense reduced to calculation. — Pierre Laplace,
1812
In the previous chapter, we saw how probability can play a useful role in machine learning. Inthis chapter, we discuss probability theory in more detail. We do not have to space to go intogreat detail — for that, you are better off consulting some of the excellent textbooks availableon this topic, such as (Jaynes 2003; Bertsekas and Tsitsiklis 2008; Wasserman 2004). But we willbrieﬂy review many of the key ideas you will need in later chapters.
Before we start with the more technical material, let us pause and ask: what is probability?
We are all familiar with the phrase “the probability that a coin will land heads is 0.5”. But whatdoes this mean? There are actually at least two different interpretations of probability. One iscalled the frequentist interpretation. In this view, probabilities represent long run frequencies

of events. For example, the above statement means that, if we ﬂip the coin many times, weexpect it to land heads about half the time.
1
The other interpretation is called the Bayesian interpretation of probability. In this view,
probability is used to quantify our uncertainty about something; hence it is fundamentally
related to information rather than repeated trials (Jaynes 2003). In the Bayesian view, the abovestatement means we believe the coin is equally likely to land heads or tails on the next toss.
One big advantage of the Bayesian interpretation is that it can be used to model our uncer-

tainty about events that do not have long term frequencies. For example, we might want tocompute the probability that the polar ice cap will melt by 2020 CE. This event will happen zeroor one times, but cannot happen repeatedly. Nevertheless, we ought to be able to quantify ouruncertainty about this event; based on how probable we think this event is, we will (hopefully!)take appropriate actions (see Section 5.7 for a discussion of optimal decision making underuncertainty). To give some more machine learning oriented examples, we might have receiveda speciﬁc email message, and want to compute the probability it is spam. Or we might haveobserved a “blip” on our radar screen, and want to compute the probability distribution overthe location of the corresponding target (be it a bird, plane, or missile). In all these cases, theidea of repeated trials does not make sense, but the Bayesian interpretation is valid and indeed

1. Actually, the Stanford statistician (and former professional magician) Persi Diaconis has shown that a coin is about
51% likely to land facing the same way up as it started, due to the physics of the problem (Diaconis et al. 2007).

28 Chapter2. Probability
0 1 2 3 4 500.250.50.751
(a)0 1 2 3 4 500.250.50.751
(b)
Figure 2.1 (A) a uniform distribution on {1,2,3,4}, withp(x=k)=1/4. (b) a degenerate distribution
p(x)=1ifx=1andp(x)=0ifx∈{2,3,4}. Figure generated by discreteProbDistFig .
quite natural. We shall therefore adopt the Bayesian interpretation in this book. Fortunately, the
basic rules of probability theory are the same, no matter which interpretation is adopted.
2.2 A brief review of probability theory
This section is a very brief review of the basics of probability theory, and is merely meant asa refresher for readers who may be “rusty”. Readers who are already familiar with these basicsmay safely skip this section.
2.2.1 Discrete random variables
The expression p(A)denotes the probability that the event Ais true. For example, Amight
be the logical expression “it will rain tomorrow”. We require that 0≤p(A)≤1,w h e r e
p(A)=0means the event deﬁnitely will not happen, and p(A)=1means the event deﬁnitely

will happen. We write p(A)to denote the probability of the event not A; this is deﬁned to
p(A)=1−p(A). We will often write A=1to mean the event Ais true, and A=0to mean
the event Ais false.
We can extend the notion of binary events by deﬁning a discrete random variable X, which
can take on any value from a ﬁnite or countably inﬁnite set X. We denote the probability of
the event that X=xbyp(X=x), or justp(x)for short. Here p()is called a probability

mass function orpmf. This satisﬁes the properties 0≤p(x)≤1and∑
x∈Xp(x)=1.
Figure 2.1 shows two pmf’s deﬁned on the ﬁnite state space X={1,2,3,4,5}. On the left we
have a uniform distribution, p(x)=1/5, and on the right, we have a degenerate distribution,
p(x)= I(x=1 ),w h e r e I()is the binary indicator function. This distribution represents the
fact thatXis always equal to the value 1, in other words, it is a constant.
2.2.2 Fundamental rules
In this section, we review the basic rules of probability.

2.2. Abriefreviewofprobabilitytheory 29
2.2.2.1 Probability of a union of two events
Given two events, AandB, we deﬁne the probability of AorBas follows:
p(A∨B)=p( A)+p(B)−p(A∧B) (2.1)
=p(A)+p(B)ifAandBare mutually exclusive (2.2)
2.2.2.2 Joint probabilities
We deﬁne the probability of the joint event AandBas follows:
p(A,B)=p(A∧B)=p(A|B)p(B) (2.3)
This is sometimes called the product rule. Given a joint distribution on two events p(A,B),
we deﬁne the marginal distribution as follows:
p(A)=∑
bp(A,B)=∑
bp(A|B=b)p(B=b) (2.4)
where we are summing over all possible states of B. We can deﬁne p(B)similarly. This is
sometimes called the sum rule or therule of total probability.
The product rule can be applied multiple times to yield the chain rule of probability:
p(X1:D)=p(X1)p(X2|X1)p(X3|X2,X1)p(X4|X1,X2,X3)...p(XD|X1:D−1)(2.5)
where we introduce the Matlab-like notation 1:Dto denote the set {1,2,...,D}.
2.2.2.3 Conditional probability

p(A)=∑
bp(A,B)=∑
We deﬁne the conditional probability of eventA, given that event Bis true, as follows:
p(A|B)=p(A,B)
p(B)ifp(B)>0 (2.6)
2.2.3 Bayes rule
Combining the deﬁnition of conditional probability with the product and sum rules yields Bayes
rule, also called Bayes Theorem2:
p(X=x|Y=y)=p(X=x,Y=y)
p(Y=y)=p(X=x)p(Y=y|X=x)∑
x′p(X=x′)p(Y=y|X=x′)(2.7)
2.2.3.1 Example: medical diagnosis
As an example of how to use this rule, consider the following medical diagonsis problem.

Suppose you are a woman in your 40s, and you decide to have a medical test for breast cancercalled amammogram. If the test is positive, what is the probability you have cancer? That
obviously depends on how reliable the test is. Suppose you are told the test has a sensitivity
2. Thomas Bayes (1702–1761) was an English mathematician and Presbyterian minister.

30 Chapter2. Probability
of 80%, which means, if you have cancer, the test will be positive with probability 0.8. In other
words,
p(x=1|y=1 )=0 .8 (2.8)
wherex=1is the event the mammogram is positive, and y=1is the event you have breast
cancer. Many people conclude they are therefore 80% likely to have cancer. But this is false! Itignores the prior probability of having breast cancer, which fortunately is quite low:
p(y=1 )=0 .004 (2.9)
Ignoring this prior is called the base rate fallacy. We also need to take into account the fact
that the test may be a false positive orfalse alarm. Unfortunately, such false positives are
quite likely (with current screening technology):
p(x=1|y=0 )=0 .1 (2.10)
Combining these three terms using Bayes rule, we can compute the correct answer as follows:
p(y=1|x=1 ) =p(x=1|y=1 )p(y=1 )
p(x=1|y=1 )p(y=1 )+p(x=1|y=0 )p(y=0 )(2.11)
=0.8×0.004
0.8×0.004+0.1×0.996=0.031 (2.12)
wherep(y=0 )=1 −p(y=1 )=0 .996. In other words, if you test positive, you only have

about a 3% chance of actually having breast cancer!3
2.2.3.2 Example: Generative classiﬁers
We can generalize the medical diagonosis example to classify feature vectors xof arbitrary type
as follows:
p(y=c|x,θ)=p(y=c|θ)p(x|y=c,θ)∑
c′p(y=c′|θ)p(x|y=c′,θ)(2.13)
This is called a generative classiﬁer, since it speciﬁes how to generate the data using the class-
conditional density p(x|y=c)and the class prior p(y=c). We discuss such models in detail
in Chapters 3 and 4. An alternative approach is to directly ﬁt the class posterior, p(y=c|x);

this is known as a discriminative classiﬁer. We discuss the pros and cons of the two approachesin Section 8.6.
2.2.4 Independence and conditional independence
We sayXandYareunconditionally independent ormarginally independent, denoted
X⊥Y, if we can represent the joint as the product of the two marginals (see Figure 2.2), i.e.,
X⊥Y⇐⇒p(X,Y)=p(X)p(Y) (2.14)

to deﬁne p(x,y).
In general, we say a set of variables is mutually independent if the joint can be written as a
product of marginals.
Unfortunately, unconditional independence is rare, because most variables can inﬂuence most
other variables. However, usually this inﬂuence is mediated via other variables rather than beingdirect. We therefore say XandYareconditionally independent (CI) given Ziff the conditional
joint can be written as a product of conditional marginals:
X⊥Y|Z⇐⇒p(X,Y|Z)=p(X|Z)p(Y|Z) (2.15)
When we discuss graphical models in Chapter 10, we will see that we can write this assumptionas a graph X−Z−Y, which captures the intuition that all the dependencies between XandY
are mediated via Z. For example, the probability it will rain tomorrow (event X) is independent
of whether the ground is wet today (event Y), given knowledge of whether it is raining today
(eventZ). Intuitively, this is because Z“causes” both XandY, so if we know Z,w ed on o t

need to know about Yin order to predict Xor vice versa. We shall expand on this concept in
Chapter 10.
Another characterization of CI is this:
Theorem 2.2.1. X⊥Y|Ziffthereexistfunction gandhsuchthat
p(x,y|z)=g(x,z)h(y,z) (2.16)
forallx,y,zsuch
 thatp(z)>0.
3. These numbers are from (McGrayne 2011, p257). Based on this analysis, the US government decided not to recommend
annual mammogram screening to women in their 40s: the number of false alarms would cause needless worry and

stress amongst women, and result in unnecesssary, expensive, and potentially harmful followup tests. See Section 5.7
for the optimal way to trade off risk reverse reward in the face of uncertainty.

32 Chapter2. Probability
See Exercise 2.8 for the proof.
CI assumptions allow us to build large probabilistic models from small pieces. We will see
many examples of this throughout the book. In particular, in Section 3.5, we discuss naive Bayesclassiﬁers, in Section 17.2, we discuss Markov models, and in Chapter 10 we discuss graphicalmodels; all of these models heavily exploit CI properties.
2.2.5 Continuous random variables
So far, we have only considered reasoning about uncertain discrete quantities. We will now show(following (Jaynes 2003, p107)) how to extend probability to reason about uncertain continuousquantities.
SupposeXis some uncertain continuous quantity. The probability that Xlies in any interval
a≤X≤bcan be computed as follows. Deﬁne the events A=(X≤a),B=(X≤b)and
W=(a<X≤b). We have that B=A∨W, and since AandWare mutually exclusive, the
sum rules gives
p(B)=p(A)+p(W) (2.17)
and hence
p(W)=p(B)−p(A) (2.18)
Deﬁne the function F(q)≜p(X≤q). This is called the cumulative distribution function

and hence
orcdfofX. This is obviously a monotonically increasing function. See Figure 2.3(a) for an
example. Using this notation we have
p(a<X≤b)=F(b)−F(a) (2.19)
Now 
deﬁnef(x)=d
dxF(x)(we assume this derivative exists); this is called the probability
density function orpdf. See Figure 2.3(b) for an example. Given a pdf, we can compute the
probability of a continuous variable being in a ﬁnite interval as follows:
P(a<X≤b)=∫b
af(x)dx (2.20)
As the size of the interval gets smaller, we can write
P(x≤X≤x+dx)≈p(x)dx (2.21)
We require p(x)≥0, but it is possible for p(x)>1for any given x, so long as the density

Now 
integrates to 1. As an example, consider the uniform distribution Unif(a,b):
Unif(x|a,b)=1
b−aI(a≤x≤b) (2.22)
If we seta=0andb=1
2,w eh a v ep( x)=2for anyx∈[0,1
2].

2.2. Abriefreviewofprobabilitytheory 33
−3 −2 −1 0 1 2 3020406080100CDF
(a)Φ−1(α/2) 0 Φ−1(1−α/2)α/2 α/2
(b)
Figure 2.3 (a) Plot of the cdf for the standard normal, N(0,1). (b) Corresponding pdf. The shaded
regions each contain α/2of the probability mass. Therefore the nonshaded region contains 1−αof the
probability mass. If the distribution is Gaussian N(0,1), then the leftmost cutoff point is Φ−1(α/2),w h e r e
Φis the cdf of the Gaussian. By symmetry, the rightost cutoff point is Φ−1(1−α/2) =−Φ−1(α/2).I f
α=0.05, the central interval is 95%, and the left cutoff is -1.96 and the right is 1.96. Figure generated by
quantileDemo.
2.2.6 Quantiles
Since the cdf Fis a monotonically increasing function, it has an inverse; let us denote this by
F−1.I fFis the cdf of X, thenF−1(α)is the value of xαsuch that P(X≤xα)=α; this is
called the αquantile ofF. The value F−1(0.5)is themedianof the distribution, with half of
the probability mass on the left, and half on the right. The values F−1(0.25)andF−1(0.75)

are the lower and upper quartiles.
We can also use the inverse cdf to compute tail area probabilities. For example, if Φis
the cdf of the Gaussian distribution N(0,1), then points to the left of Φ−1(α)/2)containα/2
probability mass, as illustrated in Figure 2.3(b). By symmetry, points to the right of Φ−1(1−α/2)
also contain α/2of the mass. Hence the central interval (Φ−1(α/2),Φ−1(1−α/2))contains
1−αof the mass. If we set α=0.05, the central 95% interval is covered by the range

(Φ−1(0.025),Φ−1(0.975)) = (−1. 96,1.96) (2.23)
If the distribution is N(μ,σ2), then the 95% interval becomes (μ−1.96σ,μ+1.96σ). This is
sometimes approximated by writing μ±2σ.
2.2.7 Mean and variance
The most familiar property of a distribution is its mean,o rexpected value, denoted by μ.F o r
discrete rv’s, it is deﬁned as E[X]≜∑
x∈Xxp(x), and for continuous rv’s, it is deﬁned as
E[X]≜∫
Xxp(x)dx. If this integral is not ﬁnite, the mean is not deﬁned (we will see some
examples of this later).

E[X]≜∫
Thevariance is a measure of the “spread” of a distribution, denoted by σ2. This is deﬁned

34 Chapter2. Probability
var[X]≜E[
(X−μ)2]
=∫
(x−μ)2p(x)dx (2.24)
=∫
x2p(x)dx+μ2∫
p(x)dx−2μ∫
xp(x)dx=E[
X2]
−μ2(2.25)
from which we derive the useful result
E[
X2]
=μ2+σ2(2.26)
Thestandard deviation is deﬁned as
std[X]≜√
var[X] (2.27)
This is useful since it has the same units as Xitself.
2.3 Some common discrete distributions
In this section, we review some commonly used parametric distributions deﬁned on discrete
state spaces, both ﬁnite and countably inﬁnite.
2.3.1 The binomial and Bernoulli distributions
Suppose we toss a coin ntimes. Let X∈{0,...,n}be the number of heads. If the probability
of heads is θ, then we say Xhas abinomial distribution, written as X∼Bin(n,θ). The pmf
is given by
Bin(k|n,θ)≜(n
k)
θk(1−θ)n−k(2.28)
where(n
k)
≜n!
(n−k)!k!(2.29)
is the number of ways to choose kitems from n(this is known as the binomial coefficient,
and is pronounced “n choose k”). See Figure 2.4 for some examples of the binomial distribution.This distribution has the following mean and variance:

is given by
k)
where(n
k)
≜n!
mean=θ,var=nθ(1−θ) (2.30)
Now suppose we toss a coin only once. Let X∈{0,1}be a binary random variable, with
probability of “success” or “heads” of θ. We say that Xhas aBernoulli distribution. This is
written as X∼Ber(θ), where the pmf is deﬁned as
Ber(x|θ)=θ
I(x=1)(1−θ)I(x=0)(2.31)
In other words,
Ber(x|θ)={θifx=1
1−θifx=0(2.32)
This is obviously just a special case of a Binomial distribution with n=1.

2.3. Somecommondiscretedistributions 35
0 1 2 3 4 5 6 7 8 9 1000.050.10.150.20.250.30.35θ=0.250
(a)0 1 2 3 4 5 6 7 8 9 1000.050.10.150.20.250.30.350.4θ=0.900
(b)
Figure 2.4 Illustration of the binomial distribution with n=1 0andθ∈{0.25,0.9}. Figure generated
bybinomDistPlot.
2.3.2 The multinomial and multinoulli distributions
The binomial distribution can be used to model the outcomes of coin tosses. To model the
outcomes of tossing a K-sided die, we can use the multinomial distribution. This is deﬁned as
follows: let x=(x1,...,x K)be a random vector, where xjis the number of times side jof
the die occurs. Then xhas the following pmf:
Mu(x|n,θ)≜(n
x1...xK)K∏
j=1θxj
j (2.33)
whereθjis the probability that side jshows up, and
(n
x1...xK)
≜n!
x1!x2!···xK!(2.34)
is themultinomial coefficient (the number of ways to divide a set of size n=∑K
k=1xkinto
subsets with sizes x1up toxK).
Now suppose n=1. This is like rolling a K-sided dice once, so xwill be a vector of 0s

j=1θxj
j (2.33)
(n
x1...xK)
≜n!
k=1xkinto
and 1s (a bit vector), in which only one bit can be turned on. Speciﬁcally, if the dice shows
up as face k, then the k’th bit will be on. In this case, we can think of xas being a scalar
categorical random variable with Kstates (values), and xis itsdummy encoding, that is,
x=[I(x=1 ),..., I(x=K)]. For example, if K=3, we encode the states 1, 2 and 3 as
(1,0,0),(0,1,0), and(0,0,1). This is also called a one-hot encoding, since we imagine that
only one of the K“wires” is “hot” or on. In this case, the pmf becomes
Mu(x|1,θ)=K∏
j=1θI(xj=1)
j (2.35)

j (2.35)
See Figure 2.1(b-c) for an example. This very common special case is known as a categorical
ordiscretedistribution. (Gustavo Lacerda suggested we call it the multinoulli distribution,b y
analogy with the Binomial/ Bernoulli distinction, a term which we shall adopt in this book.) We

36 Chapter2. Probability
Name nKx
Multinomial - - x∈{0,1,...,n}K,∑K
k=1xk=n
Multinoulli 1 - x∈{0,1}K,∑K
k=1xk=1(1-of-Kencoding)
Binomial - 1 x∈{0,1,...,n}
Bernoulli 1 1 x∈{0,1}
Table 2.1 Summary of the multinomial and related distributions.
a t a g c c g g t a c g g c a 
t t a g c t g c a a c c g c a 
t c a g c c a c t a g a g c a 
a t a a c c g c g a c c g c a 
t t a g c c g c t a a g g t a 
t a a g c c t c g t a c g t a 
t t a g c c g t t a c g g c c 
a t a t c c g g t a c a g t a 
a t a g c a g g t a c c g a a 
a c a t c c g t g a c g g a a
(a)123456789101112131415012
Sequence PositionBits
(b)
Figure 2.5 (a) Some aligned DNA sequences. (b) The corresponding sequence logo. Figure generated by
seqlogoDemo .
will use the following notation for this case:
Cat(x|θ)≜Mu(x|1,θ) (2.36)
In otherwords, if x∼Cat(θ), thenp(x=j|θ)=θj. See Table 2.1 for a summary.
2.3.2.1 Application: DNA sequence motifs
An interesting application of multinomial models arises in biosequence analysis . Suppose

(b)
we have a set of (aligned) DNA sequences, such as in Figure 2.5(a), where there are 10 rows
(sequences) and 15 columns (locations along the genome). We see that several locations are con-
served by evolution (e.g., because they are part of a gene coding region), since the corresponding
columns tend to be “pure”. For example, column 7 is all G’s.
One way to visually summarize the data is by using a sequence logo : see Figure 2.5(b). We

plot the letters A, C, G and T with a fontsize proportional to their empirical probability, and with
the most probable letter on the top. The empirical probability distribution at location t,ˆθt,i s
gotten by normalizing the vector of counts (see Equation 3.48 ):
Nt=(N∑
i=1I(Xit=1 ),N∑
i=1I(Xit=2 ),N∑
i=1I(Xit=3 ),N∑
i=1I(Xit=4 ))
(2.37)
ˆθt=Nt/N (2.38)
This distribution is known as a motif. We can also compute the most probable letter in each
location; this is called the consensus sequence .

2.3. Somecommondiscretedistributions 37
0 5 10 15 20 25 3000.050.10.150.20.250.30.350.4Poi(λ=1.000)
(a)0 5 10 15 20 25 3000.020.040.060.080.10.120.14Poi(λ=10.000)
(b)
Figure 2.6 Illustration of some Poisson distributions for λ∈{1,10}. We have truncated the x-axis to
25 for clarity, but the support of the distribution is over all the non-negative integers. Figure generated by
poissonPlotDemo.
2.3.3 The Poisson distribution
We say that X∈{0,1,2,...}has aPoisson distribution with parameter λ>0, written
X∼Poi(λ), if its pmf is
Poi(x|λ)=e−λλx
x!(2.39)
The ﬁrst term is just the normalization constant, required to ensure the distribution sums to 1.
The Poisson distribution is often used as a model for counts of rare events like radioactive
decay and traffic accidents. See Figure 2.6 for some plots.
2.3.4 The empirical distribution
Given a set of data, D={x1,...,x N}, we deﬁne the empirical distribution, also called the
empirical measure, as follows:
pemp(A)≜1
NN∑
i=1δxi(A) (2.40)

x!(2.39)
pemp(A)≜1
NN∑
whereδx(A)is theDirac measure, deﬁned by
δx(A)={0ifx̸∈A
1ifx∈A(2.41)
In general, we can associate “weights” with each sample:
p(x)=N∑
i=1wiδxi(x) (2.42)
where we require 0≤wi≤1and∑N
i=1wi=1. We can think of this as a histogram, with
“spikes” at the data points xi,w h e r ewidetermines the height of spike i. This distribution
assigns 0 probability to any point not in the data set.

38 Chapter2. Probability
2.4 Some common continuous distributions
In this section we present some commonly used univariate (one-dimensional) continuous prob-
ability distributions.
2.4.1 Gaussian (normal) distribution
The most widely used distribution in statistics and machine learning is the Gaussian or normaldistribution. Its pdf is given by
N(x|μ,σ
2)≜1√
2πσ2e−1
2σ2(x−μ)2(2.43)
Hereμ=E[X]is the mean (and mode), and σ2= var[X]is the variance.√
2πσ2is the
normalization constant needed to ensure the density integrates to 1 (see Exercise 2.11).
We write X∼N(μ,σ2)to denote that p(X=x)=N(x|μ,σ2).I fX∼N(0,1),w e
sayXfollows a standard normal distribution. See Figure 2.3(b) for a plot of this pdf; this is
sometimes called the bell curve.
We will often talk about the precision of a Gaussian, by which we mean the inverse variance:
λ=1/σ2. A high precision means a narrow distribution (low variance) centered on μ.4
Note that, since this is a pdf, we can have p(x)>1. To see this, consider evaluating the

density at its center, x=μ.W eh a v e N(μ|μ,σ2)=(σ√
2π)−1e0,so ifσ<1/√
2π,w eh a v e
p(x)>1.
The cumulative distribution function or cdf of the Gaussian is deﬁned as
Φ(x;μ,σ2)≜∫x
−∞N(z|μ,σ2)dz (2.44)
See Figure 2.3(a) for a plot of this cdf when μ=0,σ2=1. This integral has no closed form
expression, but is built in to most software packages. In particular, we can compute it in termsof theerror function (erf):
Φ(x;μ,σ)=1
2[1+erf(z/√
2)] (2.45)
wherez=(x−μ)/σand
erf(x)≜2√π∫x
0e−t2dt (2.46)

2π,w eh a v e
p(x)>1.
2)] (2.45)
The Gaussian distribution is the most widely used distribution in statistics. There are several

p(x)>1.
2)] (2.45)
reasons for this. First, it has two parameters which are easy to interpret, and which capturesome of the most basic properties of a distribution, namely its mean and variance. Second,the central limit theorem (Section 2.6.3) tells us that sums of independent random variableshave an approximately Gaussian distribution, making it a good choice for modeling residualerrors or “noise”. Third, the Gaussian distribution makes the least number of assumptions (has

4. The symbol λwill have many different meanings in this book, in order to be consistent with the rest of the literature.
The intended meaning should be clear from context.

2.4. Somecommoncontinuousdistributions 39
maximum entropy), subject to the constraint of having a speciﬁed mean and variance, as we
show in Section 9.2.6; this makes it a good default choice in many cases. Finally, it has a simplemathematical form, which results in easy to implement, but often highly effective, methods, aswe will see. See (Jaynes 2003, ch 7) for a more extensive discussion of why Gaussians are sowidely used.
2.4.2 Degenerate pdf
In the limit that σ2→0, the Gaussian becomes an inﬁnitely tall and inﬁnitely thin “spike”
centered at μ:
lim
σ2→0N(x|μ,σ2)=δ(x−μ) (2.47)
whereδis called a Dirac delta function, and is deﬁned as
δ(x)={∞ifx=0
0ifx̸=0(2.48)
such that
∫∞
−∞δ(x)dx=1 (2.49)
A useful property of delta functions is the sifting property, which selects out a single term
from a sum or integral:
∫∞
−∞f(x)δ(x−μ)dx=f(μ) (2.50)
since the integrand is only non-zero if x−μ=0.
One problem with the Gaussian distribution is that it is sensitive to outliers, since the log-

lim
such that
∫∞
∫∞
probability only decays quadratically with distance from the center. A more robust distributionis theStudenttdistribution
5Its pdf is as follows:
T(x|μ,σ2,ν)∝[
1+1
ν(x−μ
σ)2]−(ν+1
2)
(2.51)
whereμis the mean, σ2>0is the scale parameter, and ν>0is called the degrees of
freedom. See Figure 2.7 for some plots. For later reference, we note that the distribution hasthe following properties:
mean=μ,mode=μ,var=νσ
2
(ν−2)(2.52)
5. This distribution has a colourful etymology. It was ﬁrst published in 1908 by William Sealy Gosset, who worked at the

1+1
ν(x−μ
σ)2]−(ν+1
2)
(2.51)
2
Guinness brewery in Dublin. Since his employer would not allow him to use his own name, he called it the “Student”
distribution. The origin of the term tseems to have arisen in the context of Tables of the Student distribution, used by
Fisher when developing the basis of classical statistical inference. See http://jeff560.tripod.com/s.html for more
historical details.

40 Chapter2. Probability
  
Gauss
Student
Laplace
(a)−4 −3 −2 −1 0 1 2 3 4−9−8−7−6−5−4−3−2−10
  
Gauss
Student
Laplace
(b)
Figure 2.7 (a) The pdf’s for a N(0,1),T(0,1,1)andLap(0,1/√
2). The mean is 0 and the variance
is 1 for both the Gaussian and Laplace. The mean and variance of the Student is undeﬁned when ν=1.
(b) Log of these pdf’s. Note that the Student distribution is not log-concave for any parameter value, unlike
the Laplace distribution, which is always log-concave (and log-convex...) Nevertheless, both are unimodal.Figure generated by studentLaplacePdfPlot .
−5 0 5 1000.10.20.30.40.5
  
gaussian
student T
laplace
(a)−5 0 5 1000.10.20.30.40.5
  
gaussian
student T
laplace
(b)
Figure 2.8 Illustration of the effect of outliers on ﬁtting Gaussian, Student and Laplace distributions. (a)

gaussian
student T
laplace
  
gaussian
student T
laplace
(b)
No outliers (the Gaussian and Student curves are on top of each other). (b) With outliers. We see that theGaussian is more affected by outliers than the Student and Laplace distributions. Based on Figure 2.16 of(Bishop 2006a). Figure generated by robustDemo.
The variance is only deﬁned if ν>2. The mean is only deﬁned if ν>1.
As an illustration of the robustness of the Student distribution, consider Figure 2.8. On the
left, we show a Gaussian and a Student ﬁt to some data with no outliers. On the right, we

add some outliers. We see that the Gaussian is affected a lot, whereas the Student distributionhardly changes. This is because the Student has heavier tails, at least for small ν(see Figure 2.7).
Ifν=1, this distribution is known as the CauchyorLorentzdistribution. This is notable
for having such heavy tails that the integral that deﬁnes the mean does not converge.
To ensure ﬁnite variance, we require ν>2. It is common to use ν=4, which gives good

performance in a range of problems (Lange et al. 1989). For ν≫5, the Student distribution
rapidly approaches a Gaussian distribution and loses its robustness properties.

2.4. Somecommoncontinuousdistributions 41
1 2 3 4 5 6 70.10.20.30.40.50.60.70.80.9Gamma distributions
  
a=1.0,b=1.0
a=1.5,b=1.0
a=2.0,b=1.0
(a)0 0.5 1 1.5 2 2.500.511.522.533.5
(b)
Figure 2.9 (a) Some Ga(a,b=1 )distributions. If a≤1, the mode is at 0, otherwise it is >0.A s
we increase the rate b, we reduce the horizontal scale, thus squeezing everything leftwards and upwards.
Figure generated by gammaPlotDemo. (b) An empirical pdf of some rainfall data, with a ﬁtted Gamma
distribution superimposed. Figure generated by gammaRainfallDemo .
2.4.3 The Laplace distribution
Another distribution with heavy tails is the Laplace distribution6, also known as the double
sided exponential distribution. This has the following pdf:
Lap(x|μ,b)≜1
2bexp(
−|x−μ|
b)
(2.53)
Hereμis a location parameter and b>0is a scale parameter. See Figure 2.7 for a plot. This
distribution has the following properties:
mean=μ,mode=μ,var=2b2(2.54)

2bexp(
−|x−μ|
b)
(2.53)
Its robustness to outliers is illustrated in Figure 2.8. It also put mores probability density at 0
than the Gaussian. This property is a useful way to encourage sparsity in a model, as we willsee in Section 13.3.
2.4.4 The gamma distribution
Thegamma distribution is a ﬂexible distribution for positive real valued rv’s, x>0.I t i s
deﬁned in terms of two parameters, called the shape a>0and the rate b>0:7
Ga(T|shape =a,rate =b)≜ba
Γ(a)Ta−1e−Tb(2.55)
6. Pierre-Simon Laplace (1749–1827) was a French mathematician, who played a key role in creating the ﬁeld of Bayesian

7. There is an alternative parameterization, where we use the scale parameter instead of the rate: Gas(T |a,b)≜
Ga(T|a,1/b). This version is the one used by Matlab’s gampdf, although in this book will use the rate parameterization
unless otherwise speciﬁed.

42 Chapter2. Probability
whereΓ(a)is the gamma function:
Γ(x)≜∫∞
0ux−1e−udu (2.56)
See Figure 2.9 for some plots. For later reference, we note that the distribution has the following
properties:
mean=a
b,mode=a−1
b,var=a
b2(2.57)
There are several distributions which are just special cases of the Gamma, which we discuss
below.
•Exponential distribution This is deﬁned by Expon(x|λ)≜Ga(x|1,λ),w h e r eλis the rate
parameter. This distribution describes the times between events in a Poisson process, i.e. a
process in which events occur continuously and independently at a constant average rate λ.
•Erlang distribution This is the same as the Gamma distribution where ais an integer. It
is common to ﬁx a=2, yielding the one-parameter Erlang distribution, Erlang(x|λ)=
Ga(x|2,λ),w h e r eλ is the rate parameter.
•Chi-squared distribution This is deﬁned by χ2(x|ν)≜Ga(x|ν
2,1
2). This is the distribution
of the sum of squared Gaussian random variables. More precisely, if Zi∼N(0,1), and
S=∑ν
i=1Z2
i, thenS∼χ2
ν.

2,1
S=∑ν
i=1Z2
ν.
Another useful result is the following: If X∼Ga(a,b), then one can show (Exercise 2.10)
that1
X∼IG(a,b),w h e r eIG is theinverse gamma distribution deﬁned by
IG(x|shape =a,scale =b)≜ba
Γ(a)x−(a+1)e−b/x(2.58)
The distribution has these properties
mean=b
a−1,mode=b
a+1,var=b2
(a−1)2(a−2), (2.59)
The mean only exists if a>1. The variance only exists if a>2.
We will see applications of these distributions later on.
2.4.5 The beta distribution

S=∑ν
i=1Z2
ν.
that1
mean=b
Thebeta distribution has support over the interval [0,1]and is deﬁned as follows:
Beta(x|a,b)=1
B(a,b)xa−1(1−x)b−1(2.60)
HereB(p,q)is the beta function,
B(a,b)≜Γ(a)Γ(b)
Γ(a+b)(2.61)
See Figure 2.10 for plots of some beta distributions. We require a,b >0to ensure the distribution
is integrable (i.e., to ensure B(a,b)exists). If a=b=1, we get the uniform distirbution. If

2.4. Somecommoncontinuousdistributions 43
0 0.2 0.4 0.6 0.8 100.511.522.53beta distributions
  
a=0.1, b=0.1
a=1.0, b=1.0
a=2.0, b=3.0
a=8.0, b=4.0
Figure 2.10 Some beta distributions. Figure generated by betaPlotDemo.
aandbare both less than 1, we get a bimodal distribution with “spikes” at 0 and 1; if aand
bare both greater than 1, the distribution is unimodal. For later reference, we note that the
distribution has the following properties (Exercise 2.16):
mean=a
a+b,mode=a−1
a+b−2,var=ab
(a+b)2(a+b+1)(2.62)
2.4.6 Pareto distribution
ThePareto distribution is used to model the distribution of quantities that exhibit long tails,
also called heavy tails. For example, it has been observed that the most frequent word in
English (“the”) occurs approximately twice as often as the second most frequent word (“of”),
which occurs twice as often as the fourth most frequent word, etc. If we plot the frequency ofwords vs their rank, we will get a power law; this is known as Zipf’s law. Wealth has a similarly

skewed distribution, especially in plutocracies such as the USA.
8
The Pareto pdf is deﬁned as follow:
Pareto(x|k,m)=kmkx−(k+1)I(x≥m) (2.63)
This density asserts that xmust be greater than some constant m, but not too much greater,
wherekcontrols what is “too much”. As k→∞, the distribution approaches δ(x−m). See
Figure 2.11(a) for some plots. If we plot the distibution on a log-log scale, it forms a straightline, of the form logp(x)=alogx+cfor some constants aandc. See Figure 2.11(b) for an

8
illustration (this is known as a power law). This distribution has the following properties
mean=km
k−1ifk>1,mode=m,var=m2k
(k−1)2(k−2)ifk>2 (2.64)
8. In the USA, 400 Americans have more wealth than half of all Americans combined. (Source:
http://www.politifact.com/wisconsin/statements/2011/mar/10/michael-moore/michael-moore-s
ays-400-americans-have-more-wealth- .) See (Hacker and Pierson 2010) for a political analysis of how such an
extreme distribution of income has arisen in a democratic country.

44 Chapter2. Probability
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 500.20.40.60.811.21.41.61.82Pareto distribution
  
m=0.01, k=0.10
m=0.00, k=0.50
m=1.00, k=1.00
(a)10010−310−210−1100101Pareto(m=1, k) on log scale
  
k=1.0
k=2.0
k=3.0
(b)
Figure 2.11 (a) The Pareto distribution Pareto(x |m,k)form=1. (b) The pdf on a log-log scale. Figure
generated by paretoPlot.
2.5 Joint probability distributions
So far, we have been mostly focusing on modeling univariate probability distributions. In this
section, we start our discussion of the more challenging problem of building joint probabilitydistributions on multiple related random variables; this will be a central topic in this book.
Ajoint probability distribution has the form p(x
1,...,x D)f o ras e to fD> 1variables,

and models the (stochastic) relationships between the variables. If all the variables are discrete,we can represent the joint distribution as a big multi-dimensional array, with one variable perdimension. However, the number of parameters needed to deﬁne such a model is O(K
D),
whereKis the number of states for each variable.
We can deﬁne high dimensional joint distributions using fewer parameters by making con-

D),
ditional independence assumptions, as we explain in Chapter 10. In the case of continuousdistributions, an alternative approach is to restrict the form of the pdf to certain functionalforms, some of which we will examine below.
2.5.1 Covariance and correlation
Thecovariance between two rv’s XandYmeasures the degree to which XandYare (linearly)
related. Covariance is deﬁned as
cov[X,Y ]≜E[(X−E[X])(Y−E[Y])] = E[XY]−E[X]E[Y] (2.65)

2.5. Jointprobabilitydistributions 45
Figure 2.12 Several sets of (x,y)points, with the correlation coefficient of xandyfor each set. Note
that the correlation reﬂects the noisiness and direction of a linear relationship (top row), but not the slope
of that relationship (middle), nor many aspects of nonlinear relationships (bottom). N.B.: the ﬁgure in the
center has a slope of 0 but in that case the correlation coefficient is undeﬁned because the variance of Y
is zero. Source: http://en .wikipedia .org/wiki/File:Correlation_examples .png
Ifxis ad-dimensional random vector, its covariance matrix is deﬁned to be the following
symmetric, positive deﬁnite matrix:
cov[x]≜E[
(x−E[x])(x−E[x])T]
(2.66)
=⎛
⎜⎜⎜⎝var[X1]c o v [ X1,X2]···cov[X1,Xd]
cov[X2,X1] var[ X2]···cov[X2,Xd]
............
cov[Xd,X1]c o v [Xd,X2]···var[Xd]⎞
⎟⎟⎟⎠(2.67)
Covariances can be between 0 and inﬁnity. Sometimes it is more convenient to work with a
normalized measure, with a ﬁnite upper bound. The (Pearson) correlation coefficient between

cov[x]≜E[
(2.66)
=⎛
XandYis deﬁned as
corr[X,Y]≜cov[X,Y]√
var[X]var[Y](2.68)
Acorrelation matrix has the form
R=⎛
⎜⎝corr[X1,X1] corr[X1,X2]···corr[X1,Xd]
corr[Xd,X1] corr[Xd,X2]···corr[Xd,Xd]⎞
⎟⎠ (2.69)
One can show (Exercise 4.3) that −1≤corr[X,Y]≤1. Hence in a correlation matrix, each
entry on the diagonal is 1, and the other entries are between -1 and 1.
One can also show that corr[X,Y]=1if and only if Y=aX+bfor some parameters a
andb, i.e., if there is a linearrelationship between XandY(see Exercise 4.4). Intuitively one

46 Chapter2. Probability
might expect the correlation coefficient to be related to the slope of the regression line, i.e., the
coefficient ain the expression Y=aX+b. However, as we show in Equation 7.99 later, the
regression coefficient is in fact given by a=c o v[X,Y]/var[X]. A better way to think of the
correlation coefficient is as a degree of linearity: see Figure 2.12.
IfXandYare independent, meaning p(X,Y)=p(X)p(Y)(see Section 2.2.4), then
cov[X,Y ]=0, and hence corr[X,Y ]=0so they are uncorrelated. However, the con-
v e r s ei sn o tt r u e : uncorrelateddoesnotimplyindependent. For example, let X∼U(−1,1)and
Y=X2. Clearly Yis dependent on X(in fact,Yis uniquely determined by X), yet one
can show (Exercise 4.1) that corr[X,Y ]=0. Some striking examples of this fact are shown in
Figure 2.12. This shows several data sets where there is clear dependendence between XandY,

and yet the correlation coefficient is 0. A more general measure of dependence between randomvariables is mutual information, discussed in Section 2.8.3. This is only zero if the variables trulyare independent.
2.5.2 The multivariate Gaussian
Themultivariate Gaussian ormultivariate normal (MVN) is the most widely used joint prob-
ability density function for continuous variables. We discuss MVNs in detail in Chapter 4; herewe just give some deﬁnitions and plots.

The pdf of the MVN in Ddimensions is deﬁned by the following:
N(x|μ,Σ)≜1
(2π)D/2|Σ|1/2exp[
−1
2(x−μ)TΣ−1(x−μ)]
(2.70)
whereμ=E[x]∈RDis the mean vector, and Σ=c o v [x]is theD×Dcovariance
matrix. Sometimes we will work in terms of the precision matrix orconcentration matrix
instead. This is just the inverse covariance matrix, Λ=Σ−1. The normalization constant
(2π)−D/2|Λ|1/2just ensures that the pdf integrates to 1 (see Exercise 4.5).
Figure 2.13 plots some MVN densities in 2d for three different kinds of covariance matrices.

−1
(2.70)
A full covariance matrix has D(D+1)/2parameters (we divide by 2 since Σis symmetric). A
diagonal covariance matrix has Dparameters, and has 0s in the off-diagonal terms. A spherical
orisotropic covariance, Σ=σ2ID, has one free parameter.
2.5.3 Multivariate Student tdistribution
A more robust alternative to the MVN is the multivariate Student t distribution, whose pdf is
given by
T(x|μ,Σ,ν)=Γ(ν/2+D/2)
Γ(ν/2)|Σ|−1/2
νD/2πD/2×[
1+1
ν(x−μ)TΣ−1(x−μ)]−(ν+D
2)
(2.71)
=Γ(ν/2+D/2)
Γ(ν/2)|πV|−1/2×[
1+(x−μ)TV−1(x−μ)]−(ν+D
2)(2.72)

given by
1+1
2)
(2.71)
2)(2.72)
whereΣis called the scale matrix (since it is not exactly the covariance matrix) and V=νΣ.
This has fatter tails than a Gaussian. The smaller νis, the fatter the tails. As ν→∞, the

2.5. Jointprobabilitydistributions 47
full
−6 −4 −2 0 2 4 6−6−4−20246
(a)diagonal
−5 −4 −3 −2 −1 0 1 2 3 4 5−10−8−6−4−20246810
(b)
spherical
−4 −2 0 2 4 6−5−4−3−2−1012345
(c)−505
−50500.050.10.150.2spherical
(d)
Figure 2.13 We show the level sets for 2d Gaussians. (a) A full covariance matrix has elliptical contours.
(b) A diagonal covariance matrix is an axis aligned ellipse. (c) A spherical covariance matrix has a circular
shape. (d) Surface plot for the spherical Gaussian in (c). Figure generated by gaussPlot2Ddemo.
distribution tends towards a Gaussian. The distribution has the following properties
mean=μ,mode=μ,Cov=ν
ν−2Σ (2.73)
2.5.4 Dirichlet distribution
A multivariate generalization of the beta distribution is the Dirichlet distribution9, which has
support over the probability simplex, deﬁned by
SK={x:0≤xk≤1,K∑
k=1xk=1} (2.74)
The pdf is deﬁned as follows:
Dir(x|α)≜1
B(α)K∏
k=1xαk−1
kI(x∈SK) (2.75)
9. Johann Dirichlet was a German mathematician, 1805–1859.

48 Chapter2. Probability
(a)
 (b)
(c)00.51
00.51051015
α=0.10
p
(d)
Figure 2.14 (a) The Dirichlet distribution when K=3deﬁnes a distribution over the simplex, which
can be represented by the triangular surface. Points on this surface satisfy 0≤θk≤1and∑3
k=1θk=
1. (b) Plot of the Dirichlet density when α=( 2,2,2). (c) α= (20,2,2). Figure generated by
visDirichletGui , by Jonathan Huang. (d) α=( 0.1,0.1,0.1). (The comb-like structure on the edges is
a plotting artifact.) Figure generated by dirichlet3dPlot .
1 2 3 4 500.51Samples from Dir (alpha=0.1)
1 2 3 4 500.51
(a)1 2 3 4 500.51Samples from Dir (alpha=1)
(b)
Figure 2.15 Samples from a 5-dimensional symmetric Dirichlet distribution for different parameter values.
(a)α=( 0.1,...,0.1). This results in very sparse distributions, with many 0s. (b) α=( 1,...,1). This

(b)
results in more uniform (and dense) distributions. Figure generated by dirichletHistogramDemo .

2.6. Transformationsofrandomvariables 49
whereB(α1,...,α K)is the natural generalization of the beta function to Kvariables:
B(α)≜∏K
k=1Γ(αk)
Γ(α0)(2.76)
whereα0≜∑Kk=1αk.
Figure 2.14 shows some plots of the Dirichlet when K=3, and Figure 2.15 for some sampled
probability vectors. We see that α0=∑Kk=1αkcontrols the strength of the distribution (how
peaked it is), and the αkcontrol where the peak occurs. For example, Dir(1,1,1)is a uniform
distribution, Dir(2,2,2)is a broad distribution centered at (1/3,1/3,1/3), andDir(20,20,20)
is a narrow distribution centered at (1/3,1/3,1/3).I fαk<1for allk, we get “spikes” at the
corner of the simplex.
For future reference, the distribution has these properties
E[xk]=αk
α0,mode[xk]=αk−1
α0−K,var[xk]=αk(α0−αk)
α2
0(α0+1)(2.77)
whereα0=∑
kαk. Often we use a symmetric Dirichlet prior of the form αk=α/K. In this
case, the mean becomes 1/K, and the variance becomes var[xk]=K−1
K2(α+1). So increasing α
increases the precision (decreases the variance) of the distribution.

E[xk]=αk
α2
whereα0=∑
2.6 Transformations of random variables
Ifx∼p()is some random variable, and y=f(x), what is the distribution of y? This is the
question we address in this section.
2.6.1 Linear transformations
Supposef()is a linear function:
y=f(x)=Ax+b (2.78)
In this case, we can easily derive the mean and covariance of yas follows. First, for the mean,
we have
E[y]=E[Ax+b]=Aμ+b (2.79)
whereμ=E[x]. This is called the linearity of expectation.I f f()is a scalar-valued function,
f(x)=aTx+b, the corresponding result is
E[
aTx+b]
=aTμ+b (2.80)

we have
E[
aTx+b]
For the covariance, we have
cov[y]=c o v[Ax+b]=AΣAT(2.81)
whereΣ=c o v[x]. We leave the proof of this as an exercise. If f()is scalar valued, the result
becomes
var[y]=v a r[
aTx+b]
=aTΣa (2.82)

50 Chapter2. Probability
We will use both of these results extensively in later chapters. Note, however, that the mean
and covariance only completely deﬁne the distribution of yifxis Gaussian. In general we must
use the techniques described below to derive the full distribution of y, as opposed to just its
ﬁrst two moments.
2.6.2 General transformations
IfXis a discrete rv, we can derive the pmf for yby simply summing up the probability mass
for all the x’s such that f(x)=y:
py(y)=∑
x:f(x)=ypx(x) (2.83)
For example, if f(X)=1ifXis even and f(X)=0otherwise, and px(X)is uniform on the
set{1,...,10}, then py(1) =∑
x∈{2,4,6,8,10} px(x)=0.5, andpy(0) = 0.5 similarly. Note
that in this example, fis a many-to-one function.
IfXis continuous, we cannot use Equation 2.83 since px(x)is a density, not a pmf, and we
cannot sum up densities. Instead, we work with cdf’s, and write
Py(y)≜P(Y≤y)=P(f(X)≤y)=P(X∈{x|f(x)≤y}) (2.84)
We can derive the pdf of yby differentiating the cdf.

py(y)=∑
In the case of monotonic and hence invertible functions, we can write
Py(y)=P(f(X)≤y)=P(X≤f−1(y)) =Px(f−1(y)) (2.85)
Taking derivatives we get
py(y)≜d
dyPy(y)=d
dyPx(f−1(y)) =dx
dyd
dxPx(x)=dx
dypx(x) (2.86)
wherex=f−1(y). We can think of dxas a measure of volume in the x-space; similarly dy
measures volume in yspace. Thusdx
dymeasures the change in volume. Since the sign of this
change is not important, we take the absolute value to get the general expression:
py(y)=px(x)⏐⏐dx
dy⏐⏐ (2.87)

py(y)≜d
dyPy(y)=d
dyd
This is called change of variables formula. We can understand this result more intuitively as
follows. Observations falling in the range (x,x+δx)will get transformed into (y,y+δy),w h e r e
p
x(x)δx≈py(y)δy. Hence py(y)≈px(x)|δx
δy|. For example, suppose X∼U(−1,1), and
Y=X2. Thenpy(y)=1
2y−1
2. See also Exercise 2.10.
2.6.2.1 Multivariate change of variables *
We can extend the previous results to multivariate distributions as follows. Let fbe a function
that maps RntoRn, and lety=f(x). Then its Jacobian matrix Jis given by

p
2y−1
Jx→y≜∂(y1,...,y n)
∂(x1,...,x n)≜⎛
⎜⎝∂y1
∂x1···∂y1
∂xn.........
∂yn
∂x1···∂yn
∂xn⎞
⎟⎠ (2.88)

2.6. Transformationsofrandomvariables 51
|detJ|measures how much a unit cube changes in volume when we apply f.
Iffis an invertible mapping, we can deﬁne the pdf of the transformed variables using the
Jacobian of the inverse mapping y→x:
py(y)=px(x)⏐⏐det(∂x
∂y)⏐⏐=p
x(x)|detJy→x| (2.89)
In Exercise 4.5 you will use this formula to derive the normalization constant for a multivariate
Gaussian.
As a simple example, consider transforming a density from Cartesian coordinates x=(x1,x2)
to polar coordinates y=(r,θ),w h e r ex 1=rcosθandx2=rsinθ. Then
Jy→x=(∂x1
∂r∂x1
∂θ∂x2
∂r∂x2
∂θ)
=(cosθ−rsinθ
sinθrcosθ)
(2.90)
and
|detJ|=|rcos2θ+rsin2θ|=|r| (2.91)
Hence
py(y)=p x(x)|detJ| (2.92)
pr,θ(r,θ)=p x1,x2(x1,x2)r=px1,x2(rcosθ,rsinθ)r (2.93)
To see this geometrically, notice that the area of the shaded patch in Figure 2.16 is given by
P(r≤R≤r+dr,θ≤Θ≤θ+dθ)=p r,θ(r,θ)drdθ (2.94)
In the limit, this is equal to the density at the center of the patch, p(r,θ), times the size of the
patch,rd rd θ. Hence

Jy→x=(∂x1
∂r∂x1
∂θ∂x2
∂r∂x2
∂θ)
(2.90)
and
Hence
pr,θ(r,θ)drdθ=px1,x2(rcosθ,rsinθ)rd rd θ (2.95)
2.6.3 Central limit theorem
Now consider Nrandom variables with pdf’s (not necessarily Gaussian) p(xi), each with mean
μand variance σ2. We assume each variable is independent and identically distributed
oriidfor short. Let SN=∑N
i=1Xibe the sum of the rv’s. This is a simple but widely
used transformation of rv’s. One can show that, as Nincreases, the distribution of this sum
approaches
p(SN=s)=1√
2πNσ2exp(
−(s−Nμ)2
2Nσ2)
(2.96)

2πNσ2exp(
−(s−Nμ)2
2Nσ2)
(2.96)
Hence the distribution of the quantity
ZN≜SN−Nμ
σ√
N=X−μ
σ/√
N(2.97)
converges to the standard normal, where X=1
N∑Ni=1xiis the sample mean. This is called
thecentral limit theorem. See e.g., (Jaynes 2003, p222) or (Rice 1995, p169) for a proof.
In Figure 2.17 we give an example in which we compute the mean of rv’s drawn from a beta
distribution. We see that the sampling distribution of the mean value rapidly converges to a
Gaussian distribution.

52 Chapter2. Probability
Figure 2.16 Change of variables from polar to Cartesian. The area of the shaded patch is rd rd θ. Based
on (Rice 1995) Figure 3.16.
0 0.5 10123N = 1
(a)0 0.5 10123N = 5
(b)
Figure 2.17 The central limit theorem in pictures. We plot a histogram of1
N∑N
i=1xij,w h e r exij∼
Beta(1,5),f o rj= 1 : 10000 .A sN→∞, the distribution tends towards a Gaussian. (a) N=1. (b)
N=5. Based on Figure 2.6 of (Bishop 2006a). Figure generated by centralLimitDemo .
2.7 Monte Carlo approximation
In general, computing the distribution of a function of an rv using the change of variables
formula can be difficult. One simple but powerful alternative is as follows. First we generate
Ssamples from the distribution, call them x1,...,x S. (There are many ways to generate such
samples; one popular method, for high dimensional distributions, is called Markov chain Monte
Carlo or MCMC; this will be explained in Chapter 24.) Given the samples, we can approximate

the distribution of f(X)by using the empirical distribution of {f(xs)}S
s=1. This is called a
Monte Carlo approximation, named after a city in Europe known for its plush gambling casinos.
Monte Carlo techniques were ﬁrst developed in the area of statistical physics — in particular,
during development of the atomic bomb — but are now widely used in statistics and machine
learning as well.
We can use Monte Carlo to approximate the expected value of any function of a random

2.7. MonteCarloapproximation 53
−1 0 1−0.500.511.5
0 0.5 10246
0 0.5 100.050.10.150.20.25
Figure 2.18 Computing the distribution of y=x2,w h e r ep(x)is uniform (left). The analytic result is
shown in the middle, and the Monte Carlo approximation is shown on the right. Figure generated by
changeOfVarsDemo1d .
variable. We simply draw samples, and then compute the arithmetic mean of the function
applied to the samples. This can be written as follows:
E[f(X)] =∫
f(x)p(x)dx≈1
SS∑
s=1f(xs) (2.98)
wherexs∼p(X). This is called Monte Carlo integration, and has the advantage over numerical
integration (which is based on evaluating the function at a ﬁxed grid of points) that the functionis only evaluated in places where there is non-negligible probability.
By varying the function f(), we can approximate many quantities of interest, such as
•
x=1
S∑S
s=1xs→E[X]
•1
S∑Ss=1(xs−x)2→var[X]
•1
S#{xs≤c}→P(X≤c)
• median{x 1,...,x S}→median( X)
We give some examples below, and will see many more in later chapters.

•
x=1
S∑S
•1
•1
2.7.1 Example: change of variables, the MC way
In Section 2.6.2, we discussed how to analytically compute the distribution of a function of a
random variable, y=f(x). A much simpler approach is to use a Monte Carlo approximation.
For example, suppose x∼Unif(−1,1)andy=x2. We can approximate p(y)by drawing
many samples from p(x), squaring them, and computing the resulting empirical distribution.
See Figure 2.18 for an illustration. We will use this technique extensively in later chapters. Seealso Figure 5.2.

54 Chapter2. Probability
−2 −1 0 1 2−2−1.5−1−0.500.511.52
Figure 2.19 Estimating πby Monte Carlo integration. Blue points are inside the circle, red crosses are
outside. Figure generated by mcEstimatePi.
2.7.2 Example: estimating πby Monte Carlo integration
MC approximation can be used for many applications, not just statistical ones. Suppose we want
to estimate π. We know that the area of a circle with radius risπr2, but it is also equal to the
following deﬁnite integral:
I=∫r
−r∫r
−rI(x2+y2≤r2)dxdy (2.99)
Henceπ=I/(r2). Let us approximate this by Monte Carlo integration. Let f(x,y)=
I(x2+y2≤r2)be an indicator function that is 1 for points inside the circle, and 0 outside,
and letp(x)andp(y)be uniform distributions on [−r,r],s op(x)=p(y)=1/(2r). Then
I=( 2r)(2r)∫∫
f(x,y)p(x)p(y)dxdy (2.100)
=4r2∫∫
f(x,y)p(x)p(y)dxdy (2.101)
≈4r21
SS∑
s=1f(xs,ys) (2.102)
We ﬁndˆπ=3.1416with standard error 0.09 (see Section 2.7.3 for a discussion of standard

I=∫r
−r∫r
=4r2∫∫
≈4r21
SS∑
errors). We can plot the points that are accepted/ rejected as in Figure 2.19.
2.7.3 Accuracy of Monte Carlo approximation
The accuracy of an MC approximation increases with sample size. This is illustrated in Fig-ure 2.20, On the top line, we plot a histogram of samples from a Gaussian distribution. Onthe bottom line, we plot a smoothed version of these samples, created using a kernel densityestimate (Section 14.7.2). This smoothed distribution is then evaluated on a dense grid of points

2.7. MonteCarloapproximation 55
0.5 1 1.5 2 2.5012345610 samples
(a)0.5 1 1.5 2 2.500.20.40.60.811.21.41.61.82100 samples
(b)
0.5 1 1.5 2 2.500.511.522.510 samples
(c)0.5 1 1.5 2 2.500.20.40.60.811.21.41.61.8100 samples
(d)
Figure 2.20 10 and 100 samples from a Gaussian distribution, N(μ=1.5,σ2=0.25). Solid red
line is true pdf. Top line: histogram of samples. Bottom line: kernel density estimate derived from
samples in dotted blue, solid red line is true pdf. Based on Figure 4.1 of (Hoff 2009). Figure generated bymcAccuracyDemo.
and plotted. Note that this smoothing is just for the purposes of plotting, it is not used for the
Monte Carlo estimate itself.
If we denote the exact mean by μ=E[f(X)], and the MC approximation by ˆμ, one can
show that, with independent samples,
(ˆμ−μ)→N(0,σ2
S) (2.103)
where
σ2= var[f(X)] =E[
f(X)2]
−E[f(X)]2(2.104)
This is a consequence of the central-limit theorem. Of course, σ2is unknown in the above
expression, but it can also be estimated by MC:
ˆσ2=1
SS∑
s=1(f(xs)−ˆμ)2(2.105)

S) (2.103)
where
f(X)2]
ˆσ2=1
SS∑
Then we have
P{
μ−1.96ˆσ√
S≤ˆμ≤μ+1.96ˆσ√
S}
≈0.95 (2.106)

56 Chapter2. Probability
The term√
ˆσ2
Sis called the (numerical or empirical) standard error, and is an estimate of our
uncertainty about our estimate of μ. (See Section 6.2 for more discussion on standard errors.)
If we want to report an answer which is accurate to within ±ϵwith probability at least 95%,
we need to use a number of samples Swhich satisﬁes 1.96√
ˆσ2/S≤ϵ. We can approximate
the 1.96 factor by 2, yielding S≥4ˆσ2
ϵ2.
2.8 Information theory
information theory is concerned with representing data in a compact fashion (a task known as
data compression orsource coding), as well as with transmitting and storing it in a way that
is robust to errors (a task known as error correction orchannel coding). At ﬁrst, this seems
far removed from the concerns of probability theory and machine learning, but in fact there is

an intimate connection. To see this, note that compactly representing data requires allocatingshort codewords to highly probable bit strings, and reserving longer codewords to less probablebit strings. This is similar to the situation in natural language, where common words (such as“a”, “the”, “and”) are generally much shorter than rare words. Also, decoding messages sent overnoisy channels requires having a good probability model of the kinds of messages that peopletend to send. In both cases, we need a model that can predict which kinds of data are likelyand which unlikely, which is also a central problem in machine learning (see (MacKay 2003) formore details on the connection between information theory and machine learning).

Obviously we cannot go into the details of information theory here (see e.g., (Cover and
Thomas 2006) if you are interested to learn more). However, we will introduce a few basicconcepts that we will need later in the book.
2.8.1 Entropy
Theentropy of a random variable Xwith distribution p, denoted by H(X)or sometimes
H(p), is a measure of its uncertainty. In particular, for a discrete variable with Kstates, it is
deﬁned by
H(X)≜−K∑
k=1p(X=k)log2p(X=k) (2.107)
Usually we use log base 2, in which case the units are called bits(short for binary digits). If
we use log base e, the units are called nats. For example, if X∈{1,...,5}with histogram
distribution p=[ 0.25,0.25,0.2,0.15,0.15], we ﬁndH=2.2855. The discrete distribution with
maximum entropy is the uniform distribution (see Section 9.2.6 for a proof). Hence for a K-ary
random variable, the entropy is maximized if p(x=k)=1/K; in this case, H(X)=l o g2K.

deﬁned by
H(X)≜−K∑
Conversely, the distribution with minimum entropy (which is zero) is any delta-function thatputs all its mass on one state. Such a distribution has no uncertainty. In Figure 2.5(b), wherewe plotted a DNA sequence logo, the height of each bar is deﬁned to be 2−H,w h e r eHis
the entropy of that distribution, and 2 is the maximum possible entropy. Thus a bar of height 0corresponds to a uniform distribution, whereas a bar of height 2 corresponds to a deterministicdistribution.

2.8. Informationtheory 57
0 0.5 100.51
p(X = 1)H(X)
Figure 2.21 Entropy of a Bernoulli random variable as a function of θ. The maximum entropy is
log22=1. Figure generated by bernoulliEntropyFig .
For the special case of binary random variables, X∈{0,1}, we can write p(X=1 )=θ
andp(X=0 )=1 −θ. Hence the entropy becomes
H(X)=−[ p(X=1 )l o g2p(X=1 )+p(X=0 )l o g2p(X=0 ) ] (2.108)
=−[θlog2θ+(1−θ)log2(1−θ)] (2.109)
This is called the binary entropy function, and is also written H(θ). We plot this in Figure 2.21.
We see that the maximum value of 1 occurs when the distribution is uniform, θ=0.5.
2.8.2 KL divergence
One way to measure the dissimilarity of two probability distributions, pandq, is known as the
Kullback-Leibler divergence (KL divergence)o rrelative entropy. This is deﬁned as follows:
KL(p||q)≜K∑
k=1pklogpk
qk(2.110)
where the sum gets replaced by an integral for pdfs.10We can rewrite this as
KL(p||q)=∑
kpklogpk−∑
kpklogqk=−H(p)+H(p,q) (2.111)
where H(p,q)is called the cross entropy,
H(p,q)≜−∑

qk(2.110)
H(p,q)≜−∑
kpklogqk (2.112)
One can show (Cover and Thomas 2006) that the cross entropy is the average number of bits
needed to encode data coming from a source with distribution pwhen we use model qto
10. The KL divergence is not a distance, since it is asymmetric. One symmetric version of the KL divergence is the
Jensen-Shannon divergence, deﬁned as JS(p1,p2)=0.5KL (p1||q)+0.5KL (p2||q),w h e r eq =0.5p1+0.5p2.

58 Chapter2. Probability
deﬁne our codebook. Hence the “regular” entropy H(p)= H(p,p), deﬁned in Section 2.8.1, is
the expected number of bits if we use the true model, so the KL divergence is the difference
between these. In other words, the KL divergence is the average number of extrabits needed to
encode the data, due to the fact that we used distribution qto encode the data instead of the
true distribution p.
The “extra number of bits” interpretation should make it clear that KL(p||q)≥0, and that
the KL is only equal to zero iff q=p. We now give a proof of this important result.
Theorem 2.8.1. (Information inequality) KL(p||q)≥0withequalityiff p=q.
Proof.To prove the theorem, we need to use Jensen’s inequality. This states that, for any
convex function f,w eh a v et h a t
f(n∑
i=1λixi)
≤n∑
i=1λif(xi) (2.113)
whereλi≥0and∑n
i=1λi=1. This is clearly true for n=2(by deﬁnition of convexity), and
can be proved by induction for n>2.

f(n∑
i=1λixi)
≤n∑
Let us now prove the main theorem, following (Cover and Thomas 2006, p28). Let A={x:
p(x)>0}be the support of p(x). Then
−KL(p||q)=−∑
x∈Ap(x)logp(x)
q(x)=∑
x∈Ap(x)logq(x)
p(x)(2.114)
≤log∑
x∈Ap(x)q(x)
p(x)=l o g∑
x∈Aq(x) (2.115)
≤log∑
x∈Xq(x)=l o g1=0 (2.116)
where the ﬁrst inequality follows from Jensen’s. Since log(x)is a strictly concave function, we
have equality in Equation 2.115 iff p(x)=cq(x)for some c. We have equality in Equation 2.116
iff∑
x∈Aq(x)=∑
x∈Xq(x)=1, which implies c=1. Hence KL(p||q)=0iffp(x)=q(x)
for allx.

q(x)=∑
≤log∑
p(x)=l o g∑
≤log∑
iff∑
x∈Aq(x)=∑
for allx.
One important consequence of this result is that the discrete distribution with the maximum
entropy is the uniform distribution. More precisely, H(X)≤log|X|,w h e r e |X|is the number
of states for X, with equality iff p(x)is uniform. To see this, let u(x)=1/|X|. Then
0≤KL(p||u)=∑
xp(x)logp(x)
u(x)(2.117)
=∑
xp(x)logp(x)−∑
xp(x)logu(x)=− H(X)+log|X| (2.118)
This is a formulation of Laplace’s principle of insufficient reason, which argues in favor of
using uniform distributions when there are no other reasons to favor one distribution over

=∑
another. See Section 9.2.6 for a discussion of how to create distributions that satisfy certainconstraints, but otherwise are as least-commital as possible. (For example, the Gaussian satisﬁesﬁrst and second moment constraints, but otherwise has maximum entropy.)

2.8. Informationtheory 59
2.8.3 Mutual information
Consider two random variables, XandY. Suppose we want to know how much knowing one
variable tells us about the other. We could compute the correlation coefficient, but this is only
deﬁned for real-valued random variables, and furthermore, this is a very limited measure ofdependence, as we saw in Figure 2.12. A more general approach is to determine how similar thejoint distribution p(X,Y)is to the factored distribution p(X)p(Y). This is called the mutual
information orMI, and is deﬁned as follows:
I(X;Y)≜KL(p(X,Y)||p(X)p(Y)) =∑
x∑
yp(x,y)logp(x,y)
p(x)p(y)(2.119)
We have I(X;Y)≥0with equality iff p(X,Y)=p(X)p(Y). That is, the MI is zero iff the
variables are independent.
To gain insight into the meaning of MI, it helps to re-express it in terms of joint and conditional
entropies. One can show (Exercise 2.12) that the above expression is equivalent to the following:
I(X;Y)=H(X)−H(X|Y)=H(Y)−H(Y|X) (2.120)
where H(Y|X)is theconditional entropy, deﬁned as H(Y|X)=∑

x∑
xp(x)H(Y|X=x).
Thus we can interpret the MI between XandYas the reduction in uncertainty about Xafter
observing Y, or, by symmetry, the reduction in uncertainty about Yafter observing X. We will
encounter several applications of MI later in the book. See also Exercises 2.13 and 2.14 for theconnection between MI and correlation coefficients.
A quantity which is closely related to MI is the pointwise mutual information or PMI. For
two events (not random variables) xandy, this is deﬁned as
PMI(x,y)≜logp(x,y)
p(x)p(y)=l o gp(x|y)
p(x)=l o gp(y|x)

p(y)(2.121)
This measures the discrepancy between these events occuring together compared to what wouldbe expected by chance. Clearly the MI of XandYis just the expected value of the PMI.
Interestingly, we can rewrite the PMI as follows:
PMI(x,y)=l o gp(x|y)
p(y)(2.122)
This is the amount we learn from updating the prior p(x)into the posterior p(x|y), or equiva-
lently, updating the prior p(y)into the posterior p(y|x).
2.8.3.1 Mutual information for continuous random variables *

The above formula for MI is deﬁned for discrete random variables. For continuous randomvariables, it is common to ﬁrst discretize orquantize them, by dividing the ranges of each
variable into bins, and computing how many values fall in each histogram bin (Scott 1979). Wecan then easily compute the MI using the formula above (see mutualInfoAllPairsMixed for
some code, and miMixedDemo for a demo).
Unfortunately, the number of bins used, and the location of the bin boundaries, can have

a signiﬁcant effect on the results. One way around this is to try to estimate the MI directly,

/g21/g46/g47/g50/g43/g55/g44/g52/g1/g33/g44/g55/g1/g38/g53/g51/g40/g52/g1/g29/g47/g45/g44/g1/g23/g60/g54/g44/g42/g57/g40/g52/g42/g61/g1/g4/g39/g44/g40/g55/g56/g5/g1
/g12/g14/g16/g16/g10
/g13/g10/g19/g10
-g18
/g25/g55/g53/g56/g56/g1/g31/g40/g57/g62/g50/g1/g27/g52/g42/g1/g9/g1/g33/g44/g55/g56/g53/g52/g1/g4/g27/g52/g57/g2/g5/g1/g26/g44/g40/g50/g57/g46/g1/g23/g60/g54/g8/g1/g9/g1/g33/g44/g55/g56/g53/g52/g1/g4/g27/g52/g57/g1/g2/g5/g10/g12/g10/g10/g10/g14/g10/g10/g10/g16/g10/g10/g10
/g10 /g12/g10/g6/g10/g10/g10 /g14/g10/g6/g10/g10/g10
Figure 2.22 Left: Correlation coefficient vs maximal information criterion (MIC) for all pairwise relation-
ships in the WHO data. Right: scatter plots of certain pairs of variables. The red lines are non-parametric
smoothing regressions (Section 15.4.6) ﬁt separately to each trend. Source: Figure 4 of (Reshed et al. 2011) .
Used with kind permission of David Reshef and the American Association for the Advancement of Science.

without ﬁrst performing density estimation (Learned-Miller 2004). Another approach is to try
many different bin sizes and locations, and to compute the maximum MI achieved. This
statistic, appropriately normalized, is known as the maximal information coefficient (MIC)
(Reshed et al. 2011). More precisely, deﬁne
m(x,y)=maxG∈G(x,y)I(X(G);Y(G))
logmin(x,y)(2.123)
whereG(x,y)is the set of 2d grids of size x×y, andX(G),Y(G)represents a discretization of

the variables onto this grid. (The maximization over bin locations can be performed efficiently
using dynamic programming (Reshed et al. 2011).) Now deﬁne the MIC as
MIC≜max
x,y:xy<Bm(x,y) (2.124)
whereBis some sample-size dependent bound on the number of bins we can use and still
reliably estimate the distribution ((Reshed et al. 2011) suggest B=N0.6). It can be shown that
the MIC lies in the range [0,1], where 0 represents no relationship between the variables, and 1
represents a noise-free relationship of any form, not just linear.

MIC≜max
Figure 2.22 gives an example of this statistic in action. The data consists of 357 variables
measuring a variety of social, economic, health and political indicators, collected by the World
Health Organization (WHO). On the left of the ﬁgure, we see the correlation coefficient (CC)
plotted against the MIC for all 63,566 variable pairs. On the right of the ﬁgure, we see scatter
plots for particular pairs of variables, which we now discuss:
• The point marked C has a low CC and a low MIC. The corresponding scatter plot makes it

2.8. Informationtheory 61
clear that there is no relationship between these two variables (percentage of lives lost to
injury and density of dentists in the population).
• The points marked D and H have high CC (in absolute value) and high MIC, because they
represent nearly linear relationships.
• The points marked E, F, and G have low CC but high MIC. This is because they correspond
to non-linear (and sometimes, as in the case of E and F, non-functional, i.e., one-to-many)relationships between the variables.
In summary, we see that statistics (such as MIC) based on mutual information can be used
to discover interesting relationships between variables in a way that simpler measures, such ascorrelation coefficients, cannot. For this reason, the MIC has been called “a correlation for the21st century” (Speed 2011).
Exercises
Exercise 2.1 Probabilities are sensitive to the form of the question that was used to generate the answer

Exercises
(Source: Minka.) My neighbor has two children. Assuming that the gender of a child is like a coin ﬂip,
it is most likely, a priori, that my neighbor has one boy and one girl, with probability 1/2. The otherpossibilities—two boys or two girls—have probabilities 1/4 and 1/4.
a. Suppose I ask him whether he has any boys, and he says yes. What is the probability that one child is
a girl?
b. Suppose instead that I happen to see one of his children run by, and it is a boy. What is the probability
that the other child is a girl?
Exercise 2.2 Legal reasoning

a girl?
(Source: Peter Lee.) Suppose a crime has been committed. Blood is found at the scene for which there isno innocent explanation. It is of a type which is present in 1% of the population.
a. The prosecutor claims: “There is a 1% chance that the defendant would have the crime blood type if he
were innocent. Thus there is a 99% chance that he guilty”. This is known as the prosecutor’s fallacy.
What is wrong with this argument?
b. The defender claims: “The crime occurred in a city of 800,000 people. The blood type would be

found in approximately 8000 people. The evidence has provided a probability of just 1 in 8000 that
the defendant is guilty, and thus has no relevance.” This is known as the defender’s fallacy. What is
wrong with this argument?
Exercise 2.3 Variance of a sum
Show that the variance of a sum is var[X+Y]=v a r[X]+v ar[Y]+2cov[X,Y ],wherecov[X,Y ]
is the covariance between XandY
Exercise 2.4 Bayes rule for medical diagnosis
(Source: Koller.) After your yearly checkup, the doctor has bad news and good news. The bad news is that

you tested positive for a serious disease, and that the test is 99% accurate (i.e., the probability of testingpositive given that you have the disease is 0.99, as is the probability of tetsing negative given that you don’thave the disease). The good news is that this is a rare disease, striking only one in 10,000 people. What are
the chances that you actually have the disease? (Show your calculations as well as giving the ﬁnal result.)

62 Chapter2. Probability
Exercise 2.5 The Monty Hall problem
(Source: Mackay.) On a game show, a contestant is told the rules as follows:
There are three doors, labelled 1, 2, 3. A single prize has been hidden behind one of them. You
get to select one door. Initially your chosen door will notbe opened. Instead, the gameshow host
will open one of the other two doors, and hewilldosoinsuchawayasnottorevealtheprize. For
example, if you ﬁrst choose door 1, he will then open one of doors 2 and 3, and it is guaranteed
that he will choose which one to open so that the prize will not be revealed.
At this point, you will be given a fresh choice of door: you can either stick with your ﬁrst choice,
or you can switch to the other closed door. All the doors will then be opened and you will receivewhatever is behind your ﬁnal choice of door.

Imagine that the contestant chooses door 1 ﬁrst; then the gameshow host opens door 3, revealing nothingbehind the door, as promised. Should the contestant (a) stick with door 1, or (b) switch to door 2, or (c)
does it make no difference? You may assume that initially, the prize is equally likely to be behind any of
the 3 doors. Hint: use Bayes rule.
Exercise 2.6 Conditional independence
(Source: Koller.)
a. LetH∈{1,...,K}be a discrete random variable, and let e
1ande2be the observed values of two

other random variables E1andE2. Suppose we wish to calculate the vector
⃗P(H|e1,e2)=(P(H=1|e1,e2),...,P(H=K|e1,e2))
Which of the following sets of numbers are sufficient for the calculation?
i.P(e1,e2),P(H),P(e1|H),P(e2|H)
ii.P(e1,e2),P(H),P(e1,e2|H)
iii.P(e1|H),P(e2|H),P(H)
b. Now suppose we now assume E1⊥E2|H(i.e.,E1andE2are conditionally independent given H).
Which of the above 3 sets are sufficent now?

Show your calculations as well as giving the ﬁnal result. Hint: use Bayes rule.Exercise 2.7 Pairwise independence does not imply mutual independence
We say that two random variables are pairwise independent if
p(X
2|X1)=p(X2) (2.125)
and hence
p(X2,X1)=p(X1)p(X2|X1)=p(X1)p(X2) (2.126)
We say that nrandom variables are mutually independent if
p(Xi|XS)=p(Xi)∀S⊆{1,...,n}\{i} (2.127)
and hence
p(X1:n)=n∏
i=1p(Xi) (2.128)

p(X
and hence
and hence
Show that pairwise independence between all pairs of variables does not necessarily imply mutual inde-
pendence. It suffices to give a counter example.

2.8. Informationtheory 63
Exercise 2.8 Conditional independence iff joint factorizes
In the text we said X⊥Y|Ziff
p(x,y|z)=p(x|z)p(y|z) (2.129)
for allx,y,zsuch that p(z)>0. Now prove the following alternative deﬁnition: X⊥Y|Ziff there exist
functiongandhsuch that
p(x,y|z)=g(x,z)h(y,z) (2.130)
for allx,y,zsuch that p(z)>0.
Exercise 2.9 Conditional independence
(Source: Koller.) Are the following properties true? Prove or disprove. Note that we are not restricting
attention to distributions that can be represented by a graphical model.
a. True or false? (X⊥W|Z,Y)∧(X⊥Y|Z)⇒(X⊥Y,W|Z)
b.T
rue or false? (X⊥Y|Z)∧(X⊥Y|W)⇒(X⊥Y|Z,W)
Exercise 2.10 Deriving the inverse gamma density
LetX∼Ga(a,b), i.e.
Ga(x|a,b)=ba
Γ(a)xa−1e−xb(2.131)
LetY=1/X. Show that Y∼IG(a,b), i.e.,
IG(x|shape = a,scale =b)=ba
Γ(a)x−(a+1)e−b/x(2.132)
Hint: use the change of variables formula.
Exercise 2.11 Normalization constant for a 1D Gaussian
The normalization constant for a zero-mean Gaussian is given by
Z=∫b
aexp(
−x2
2σ2)
dx (2.133)

b.T
Z=∫b
aexp(
−x2
2σ2)
dx (2.133)
wherea=−∞andb=∞. To compute this, consider its square
Z2=∫b
a∫b
aexp(
−x2+y2
2σ2)
dxdy (2.134)
Let us change variables from cartesian (x,y)to polar(r,θ)usingx=rcosθandy=rsinθ. Since
dxdy=rdrdθ, andcos2θ+sin2θ=1,w eh a v e
Z2=∫2π
0∫∞
0rexp(
−r2
2σ2)
drdθ (2.135)
Evaluate this integral and hence show Z=σ√
(2π). Hint 1: separate the integral into a product of
two terms, the ﬁrst of which (involving dθ) is constant, so is easy. Hint 2: if u=e−r2/2σ2then
du/dr=−1
σ2re−r2/2σ2, so the second integral is also easy (since∫
u′(r)dr=u(r)).

64 Chapter2. Probability
Exercise 2.12 Expressing mutual information in terms of entropies
Show that
I(X,Y)=H(X)−H(X|Y)=H(Y)−H(Y|X) (2.136)
Exercise 2.13 Mutual information for correlated normals
(Source: (Cover and Thomas 1991, Q9.3).) Find the mutual information I(X1,X2)where Xhas a bivariate
normal distribution:
(X1
X2)
∼N(
0,(σ2ρσ2
ρσ2σ2))
(2.137)
EvaluateI(X1,X2)forρ=1,ρ=0andρ=−1and comment. Hint: The (differential) entropy of a
d-dimensional Gaussian is
h(X)=1
2log2[
(2πe)ddetΣ]
(2.138)
In the 1d case, this becomes
h(X)=1
2log2[
2πeσ2]
(2.139)
Hint:log(0) = ∞.
Exercise 2.14 A measure of correlation (normalized mutual information)
(Source: (Cover and Thomas 1991, Q2.20).) Let XandYbe discrete random variables which are identically
distributed (so H(X)=H(Y)) but not necessarily independent. Deﬁne
r=1−H(Y|X)
H(X)(2.140)
a. Showr=I(X,Y )
H(X)
b. Show0≤r≤1
c. When is r=0?
d. When is r=1?
Exercise 2.15 MLE minimizes KL divergence to the empirical distribution

h(X)=1
2log2[
(2.138)
h(X)=1
2log2[
2πeσ2]
(2.139)
H(X)
Letpemp(x)betheempiricaldistribution, andlet q(x|θ)besomemodel. Showthat argminqKL(pemp||q)
is obtained by q(x)=q(x;ˆθ),w h e r eˆθis the MLE. Hint: use non-negativity of the KL divergence.
Exercise 2.16 Mean, mode, variance for the beta distribution
Supposeθ∼Beta(a,b) . Derive the mean, mode and variance.
Exercise 2.17 Expected value of the minimum
SupposeX,Yare two points sampled independently and uniformly at random from the interval [0,1].
What is the expected location of the left most point?

3 Generative models for discrete data
3.1 Introduction
In Section 2.2.3.2, we discussed how to classify a feature vector xby applying Bayes rule to a
generative classiﬁer of the form
p(y=c|x,θ)∝p(x|y=c,θ)p(y=c|θ) (3.1)
The key to using such models is specifying a suitable form for the class-conditional density
p(x|y=c,θ), which deﬁnes what kind of data we expect to see in each class. In this chapter,
we focus on the case where the observed data are discrete symbols. We also discuss how toinfer the unknown parameters θof such models.
3.2 Bayesian concept learning

Consider how a child learns to understand the meaning of a word, such as “dog”. Presumablythe child’s parents point out positive examples of this concept, saying such things as, “look atthe cute dog!”, or “mind the doggy”, etc. However, it is very unlikely that they provide negativeexamples, by saying “look at that non-dog”. Certainly, negative examples may be obtained duringan active learning process — the child says “look at the dog” and the parent says “that’s a cat,dear, not a dog” — but psychological research has shown that people can learn concepts frompositive examples alone (Xu and Tenenbaum 2007).

We can think of learning the meaning of a word as equivalent to concept learning, which in
turn is equivalent to binary classiﬁcation. To see this, deﬁne f(x)=1ifxis an example of the
conceptC, andf(x)=0otherwise. Then the goal is to learn the indicator function f, which
just deﬁnes which elements are in the set C. By allowing for uncertainty about the deﬁnition
off, or equivalently the elements of C, we can emulate fuzzy set theory, but using standard
probability calculus. Note that standard binary classiﬁcation techniques require positive andnegative examples. By contrast, we will devise a way to learn from positive examples alone.
For pedagogical purposes, we will consider a very simple example of concept learning called
thenumber game , based on part of Josh Tenenbaum’s PhD thesis (Tenenbaum 1999). The game
proceeds as follows. I choose some simple arithmetical concept C, such as “prime number” or
“a number between 1 and 10”. I then give you a series of randomly chosen positive examplesD={x

1,...,x N}drawn from C, and ask you whether some new test case ˜xbelongs to C,
i.e., I ask you to classify ˜x.

66 Chapter 3. Generative models for discrete data
481216202428323640444852566064687276808488929610000.5116Examples
481216202428323640444852566064687276808488929610000.5160
481216202428323640444852566064687276808488929610000.5116   8   2  64
481216202428323640444852566064687276808488929610000.5116  23  19  20
Figure 3.1 Empirical predictive distribution averaged over 8 humans in the number game. First two
rows: after seeing D={16}andD={60}. This illustrates diffuse similarity. Third row: after
seeingD={16,8,2,64}. This illustrates rule-like behavior (powers of 2). Bottom row: after seeing
D={16,23,19,20}. This illustrates focussed similarity (numbers near 20). Source: Figure 5.5 of
(Tenenbaum 1999). Used with kind permission of Josh Tenenbaum.
Suppose, for simplicity, that all numbers are integers between 1 and 100. Now suppose I tell
you “16” is a positive example of the concept. What other numbers do you think are positive?

17? 6? 32? 99? It’s hard to tell with only one example, so your predictions will be quite vague.Presumably numbers that are similar in some sense to 16 are more likely. But similar in whatway? 17 is similar, because it is “close by”, 6 is similar because it has a digit in common,32 is similar because it is also even and a power of 2, but 99 does not seem similar. Thussome numbers are more likely than others. We can represent this as a probability distribution,p(˜x|D), which is the probability that ˜x∈Cgiven the data Dfor any˜x∈{1,...,100}. This

is called the posterior predictive distribution. Figure 3.1(top) shows the predictive distribution
of people derived from a lab experiment. We see that people predict numbers that are similarto 16, under a variety of kinds of similarity.
Now suppose I tell you that 8, 2 and 64 are alsopositive examples. Now you may guess that
the hidden concept is “powers of two”. This is an example of induction. Given this hypothesis,
the predictive distribution is quite speciﬁc, and puts most of its mass on powers of 2, as shownin Figure 3.1(third row). If instead I tell you the data is D={16,23,19,20}, you will get a
different kind of generalization gradient, as shown in Figure 3.1(bottom).
How can we explain this behavior and emulate it in a machine? The classic approach to
induction is to suppose we have a hypothesis space of concepts, H, such as: odd numbers,
even numbers, all numbers between 1 and 100, powers of two, all numbers ending in j(for

3.2. Bayesian concept learning 67
0≤j≤9), etc. The subset of Hthat is consistent with the data Dis called the version space.
As we see more examples, the version space shrinks and we become increasingly certain about
the concept (Mitchell 1997).
However, the version space is not the whole story. After seeing D={16}, there are many
consistent rules; how do you combine them to predict if ˜x∈C? Also, after seeing D=
{16,8,2,64}, why did you choose the rule “powers of two” and not, say, “all even numbers”, or
“powers of two except for 32”, both of which are equally consistent with the evidence? We willnow provide a Bayesian explanation for this.
3.2.1 Likelihood
We must explain why we chose htwo≜“powers of two”, and not, say, heven≜“even numbers”
after seeing D={16,8,2,64}, given that both hypotheses are consistent with the evidence.
The key intuition is that we want to avoid suspicious coincidences. If the true concept was
even numbers, how come we only saw numbers that happened to be powers of two?

To formalize this, let us assume that examples are sampled uniformly at random from the
extension of a concept. (The extension of a concept is just the set of numbers that belong
to it, e.g., the extension of hevenis{2,4,6,...,98,100}; the extension of “numbers ending
in 9” is{9,19,...,99}.) Tenenbaum calls this the strong sampling assumption. Given this
assumption, the probability of independently sampling Nitems (with replacement) from his
given by
p(D|h)=[1
size(h)]N
=[1
|h|]N
(3.2)

given by
p(D|h)=[1
size(h)]N
=[1
|h|]N
(3.2)
This crucial equation embodies what Tenenbaum calls the size principle, which means the
model favors the simplest (smallest) hypothesis consistent with the data. This is more commonlyknown as Occam’s razor.
1
To see how it works, let D={16}. Then p(D|htwo)=1/6, since there are only 6 powers
of two less than 100, but p(D|heven)=1/50, since there are 50 even numbers. So the
likelihood that h=htwois higher than if h=heven. After 4 examples, the likelihood of htwo

given by
p(D|h)=[1
size(h)]N
=[1
|h|]N
(3.2)
1
is(1/6)4=7.7×10−4, whereas the likelihood of hevenis(1/50)4=1.6×10−7. This is
alikelihood ratio of almost 5000:1 in favor of htwo. This quantiﬁes our earlier intuition that
D={16,8,2,64}would be a very suspicious coincidence if generated by heven.
3.2.2 Prior
SupposeD={16,8,2,64}. Given this data, the concept h′=“powers of two except 32” is
more likely than h=“powers of two”, since h′does not need to explain the coincidence that 32
is missing from the set of examples.

However, the hypothesis h′=“powers of two except 32” seems “conceptually unnatural”. We
can capture such intution by assigning low prior probability to unnatural concepts. Of course,your prior might be different than mine. This subjective aspect of Bayesian reasoning is a
source of much controversy, since it means, for example, that a child and a math professor
1. William of Occam (also spelt Ockham) was an English monk and philosopher, 1288–1348.

68 Chapter 3. Generative models for discrete data
will reach different answers. In fact, they presumably not only have different priors, but also
different hypothesis spaces. However, we can ﬁnesse that by deﬁning the hypothesis space ofthe child and the math professor to be the same, and then setting the child’s prior weight to bezero on certain “advanced” concepts. Thus there is no sharp distinction between the prior andthe hypothesis space.
Although the subjectivity of the prior is controversial, it is actually quite useful. If you are
told the numbers are from some arithmetic rule, then given 1200, 1500, 900 and 1400, you maythink 400 is likely but 1183 is unlikely. But if you are told that the numbers are examples ofhealthy cholesterol levels, you would probably think 400 is unlikely and 1183 is likely. Thus wesee that the prior is the mechanism by which background knowledge can be brought to bear ona problem. Without this, rapid learning (i.e., from small samples sizes) is impossible.

So, what prior should we use? For illustration purposes, let us use a simple prior which
puts uniform probability on 30 simple arithmetical concepts, such as “even numbers”, “oddnumbers”, “prime numbers”, “numbers ending in 9”, etc. To make things more interesting, wemake the concepts even and odd more likely apriori. We also include two “unnatural” concepts,namely “powers of 2, plus 37” and “powers of 2, except 32”, but give them low prior weight. SeeFigure 3.2(a) for a plot of this prior. We will consider a slightly more sophisticated prior later on.

3.2.3 Posterior
The posterior is simply the likelihood times the prior, normalized. In this context we have
p(h|D)=p(D|h)p(h)∑
h′∈Hp(D,h′)=p(h)I(D∈h)/|h|N
∑
h′∈Hp(h′)I(D∈h′)/|h′|N(3.3)
where I(D∈h)is 1iff(iff and only if) all the data are in the extension of the hypothesis
h. Figure 3.2 plots the prior, likelihood and posterior after seeing D={16}. We see that the

∑
posterior is a combination of prior and likelihood. In the case of most of the concepts, the prioris uniform, so the posterior is proportional to the likelihood. However, the “unnatural” conceptsof “powers of 2, plus 37” and “powers of 2, except 32” have low posterior support, despite havinghigh likelihood, due to the low prior. Conversely, the concept of odd numbers has low posteriorsupport, despite having a high prior, due to the low likelihood.
Figure 3.3 plots the prior, likelihood and posterior after seeing D={16,8,2,64}. Now the

likelihood is much more peaked on the powers of two concept, so this dominates the posterior.Essentially the learner has an ahamoment, and ﬁgures out the true concept. (Here we see the
need for the low prior on the unnatural concepts, otherwise we would have overﬁt the data andpicked “powers of 2, except for 32”.)
In general, when we have enough data, the posterior p(h|D)becomes peaked on a single
concept, namely the MAP estimate, i.e.,
p(h|D)→δ
ˆhMAP(h) (3.4)
whereˆhMAP=a r g m a xhp(h|D)is the posterior mode, and where δis theDirac measure
deﬁned by
δx(A)={
1ifx∈A
0ifx̸∈A(3.5)

3.2. Bayesian concept learning 69
0 0.1 0.2powers of 2 − {32}powers of 2 + {37}allpowers of 10powers of 9powers of 8powers of 7powers of 6powers of 5powers of 4powers of 3powers of 2ends in 9ends in 8ends in 7ends in 6ends in 5ends in 4ends in 3ends in 2ends in 1mult of 10mult of 9mult of 8mult of 7mult of 6mult of 5mult of 4mult of 3squaresoddeven
prior0 0.2 0.405101520253035
likdata = 16
0 0.2 0.405101520253035
post
Figure 3.2 Prior, likelihood and posterior for D={16}. Based on (Tenenbaum 1999). Figure generated
bynumbersGame .
Note that the MAP estimate can be written as
ˆhMAP= argmax
hp(D|h)p(h) = argmax
h[logp(D|h)+logp(h)] (3.6)
Since the likelihood term depends exponentially on N, and the prior stays constant, as we get
more and more data, the MAP estimate converges towards the maximum likelihood estimate
orMLE:
ˆhmle≜argmax
hp(D|h) = argmax
hlogp(D|h) (3.7)
In other words, if we have enough data, we see that the data overwhelms the prior. In this

70 Chapter 3. Generative models for discrete data
prior0 1 2
x 10−305101520253035
likdata = 16   8   2  64
0 0.5 105101520253035
post
Figure 3.3 Prior, likelihood and posterior for D={16,8,2,64}. Based on (Tenenbaum 1999). Figure
generated by numbersGame .
case, the MAP estimate converges towards the MLE.
If the true hypothesis is in the hypothesis space, then the MAP/ ML estimate will converge
upon this hypothesis. Thus we say that Bayesian inference (and ML estimation) are consistent
estimators (see Section 6.4.1 for details). We also say that the hypothesis space is identiﬁable in

post
the limit, meaning we can recover the truth in the limit of inﬁnite data. If our hypothesis classis not rich enough to represent the “truth” (which will usually be the case), we will convergeon the hypothesis that is as close as possible to the truth. However, formalizing this notion of“closeness” is beyond the scope of this chapter.

3.2. Bayesian concept learning 71
4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 88 92 96 10000.51
powers of 2 + {37}powers of 2 − {32}allmult of 4mult of 8evensquaresends in 6powers of 2powers of 4
0 0.5 1
p(h | 16 )
Figure 3.4 Posterior over hypotheses and the corresponding predictive distribution after seeing one
example, D={16}. A dot means this number is consistent with this hypothesis. The graph p(h|D)on
the right is the weight given to hypothesis h. By taking a weighed sum of dots, we get p(˜x∈C|D)(top).
Based on Figure 2.9 of (Tenenbaum 1999). Figure generated by numbersGame .
3.2.4 Posterior predictive distribution
The posterior is our internal belief state about the world. The way to test if our beliefs are
justiﬁed is to use them to predict objectively observable quantities (this is the basis of the
scientiﬁc method). Speciﬁcally, the posterior predictive distribution in this context is given by
p(˜x∈C|D)=∑
hp(y=1|˜x,h)p(h|D) (3.8)

This is just a weighted average of the predictions of each individual hypothesis and is calledBayes model averaging (Hoeting et al. 1999). This is illustrated in Figure 3.4. The dots at the
bottom show the predictions from each hypothesis; the vertical curve on the right shows theweight associated with each hypothesis. If we multiply each row by its weight and add up, weget the distribution at the top.
When we have a small and/or ambiguous dataset, the posterior p(h|D)is vague, which

induces a broad predictive distribution. However, once we have “ﬁgured things out”, the posteriorbecomes a delta function centered at the MAP estimate. In this case, the predictive distribution

72 Chapter 3. Generative models for discrete data
becomes
hp(˜x|h)δˆh(h)=p(˜x|ˆh) (3.9)
This is called a plug-in approximation to the predictive density and is very widely used, due
to its simplicity. However, in general, this under-represents our uncertainty, and our predictions
will not be as “smooth” as when using BMA. We will see more examples of this later in the book.
Although MAP learning is simple, it cannot explain the gradual shift from similarity-based
reasoning (with uncertain posteriors) to rule-based reasoning (with certain posteriors). Forexample, suppose we observe D={16}. If we use the simple prior above, the minimal
consistent hypothesis is “all powers of 4”, so only 4 and 16 get a non-zero probability of beingpredicted. This is of course an example of overﬁtting. Given D={16,8,2,64}, the MAP

hypothesis is “all powers of two”. Thus the plug-in predictive distribution gets broader (or staysthe same) as we see more data: it starts narrow, but is forced to broaden as it seems more data.In contrast, in the Bayesian approach, we start broad and then narrow down as we learn more,which makes more intuitive sense. In particular, given D={16}, there are many hypotheses
with non-negligible posterior support, so the predictive distribution is broad. However, when weseeD={16,8,2,64}, the posterior concentrates its mass on one hypothesis, so the predictive

distribution becomes narrower. So the predictions made by a plug-in approach and a Bayesianapproach are quite different in the small sample regime, although they converge to the sameanswer as we see more data.
3.2.5 A more complex prior
To model human behavior, Tenenbaum used a slightly more sophisticated prior which was de-rived by analysing some experimental data of how people measure similarity between numbers;see (Tenenbaum 1999, p208) for details. The result is a set of arithmetical concepts similar tothose mentioned above, plus all intervals between nandmfor1≤n,m≤100. (Note that
these hypotheses are not mutually exclusive.) Thus the prior is a mixture of two priors, one
over arithmetical rules, and one over intervals:
p(h)=π
0prules(h)+(1− π0)pinterval(h) (3.10)

p(h)=π
The only free parameter in the model is the relative weight, π0, given to these two parts of the
prior. The results are not very sensitive to this value, so long as π0>0.5, reﬂecting the fact
that people are more likely to think of concepts deﬁned by rules. The predictive distributionof the model, using this larger hypothesis space, is shown in Figure 3.5. It is strikingly similarto the human predictive distribution, shown in Figure 3.1, even though it was not ﬁt to humandata (modulo the choice of hypothesis space).
3.3 The beta-binomial model
The number game involved inferring a distribution over a discrete variable drawn from a ﬁnitehypothesis space, h∈H, given a series of discrete observations. This made the computations

particularly simple: we just needed to sum, multiply and divide. However, in many applications,the unknown parameters are continuous, so the hypothesis space is (some subset) of R
K,w h e r e

3.3. The beta-binomial model 73
Figure 3.5 Predictive distributions for the model using the full hypothesis space. Compare to Figure 3.1.
The predictions of the Bayesian model are only plotted for those values of ˜xfor which human data is
available; this is why the top line looks sparser than Figure 3.4. Source: Figure 5.6 of (Tenenbaum 1999).
Used with kind permission of Josh Tenenbaum.
Kis the number of parameters. This complicates the mathematics, since we have to replace
sums with integrals. However, the basic ideas are the same.
We will illustrate this by considering the problem of inferring the probability that a coin shows
up heads, given a series of observed coin tosses. Although this might seem trivial, it turns out

that this model forms the basis of many of the methods we will consider later in this book,including naive Bayes classiﬁers, Markov models, etc. It is historically important, since it was theexample which was analyzed in Bayes’ original paper of 1763. (Bayes’ analysis was subsequentlygeneralized by Pierre-Simon Laplace, creating what we now call “Bayes rule” — see (Stigler 1986)for further historical details.)
We will follow our now-familiar recipe of specifying the likelihood and prior, and deriving the

posterior and posterior predictive.
3.3.1 Likelihood
Suppose Xi∼Ber(θ),w h e r eXi=1represents “heads”, Xi=0represents “tails”, and
θ∈[0,1]is the rate parameter (probability of heads). If the data are iid, the likelihood has the
form
p(D|θ)=θN1(1−θ)N0(3.11)

74 Chapter 3. Generative models for discrete data
w h e r ew eh a v eN 1=∑N
i=1I(xi=1 )heads and N0=∑Ni=1I(xi=0 )tails. These two counts
are called the sufficient statistics of the data, since this is all we need to know about Dto
inferθ. (An alternative set of sufficient statistics are N1andN=N0+N1.)
More formally, we say s(D)is a sufficient statistic for data Difp(θ|D)=p(θ|s(data)).I f
we use a uniform prior, this is equivalent to saying p(D|θ∝p(s(D)|θ). Consequently, if we
have two datasets with the same sufficient statistics, we will infer the same value for θ.
Now suppose the data consists of the count of the number of heads N1observed in a ﬁxed
numberN=N1+N0of trials. In this case, we have N1∼Bin(N,θ),w h e r eBinrepresents
the binomial distribution, which has the following pmf:
k)
θk(1−θ)n−k(3.12)
Since(n
k)
is a constant independent of θ, the likelihood for the binomial sampling model is the
same as the likelihood for the Bernoulli model. So any inferences we make about θwill be the

k)
Since(n
k)
same whether we observe the counts, D=(N1,N), or a sequence of trials, D={x1,...,x N}.
3.3.2 Prior
We need a prior which has support over the interval [0,1]. To make the math easier, it would
convenient if the prior had the same form as the likelihood, i.e., if the prior looked like
p(θ)∝θγ1(1−θ)γ2(3.13)
for some prior parameters γ1andγ2. If this were the case, then we could easily evaluate the
posterior by simply adding up the exponents:
p(θ)∝p(D|θ)p(θ)=θN1(1−θ)N0θγ1(1−θ)γ2=θN1+γ1(1−θ)N0+γ2(3.14)

When the prior and the posterior have the same form, we say that the prior is a conjugate
priorfor the corresponding likelihood. Conjugate priors are widely used because they simplify
computation, and are easy to interpret, as we see below.
In the case of the Bernoulli, the conjugate prior is the beta distribution, which we encountered
in Section 2.4.5:
Beta(θ|a,b)∝θa−1(1−θ)b−1(3.15)
The parameters of the prior are called hyper-parameters. We can set them in order to encode

our prior beliefs. For example, to encode our beliefs that θhas mean 0.7 and standard deviation
0.2, we set a=2.975andb=1.275(Exercise 3.15). Or to encode our beliefs that θhas mean
0.15 and that we think it lives in the interval (0.05,0.30)with probability, then we ﬁnd a=4.5
andb=2 5.5(Exercise 3.16).
If we know “nothing” about θ, except that it lies in the interval [0,1], we can use a uni-
form prior, which is a kind of uninformative prior (see Section 5.4.2 for details). The uniform

distribution can be represented by a beta distribution with a=b=1.

3.3. The beta-binomial model 75
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10123456
  
prior Be(2.0, 2.0)
lik Be(4.0, 18.0)
post Be(5.0, 19.0)
(a)0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 100.511.522.533.544.5
  
prior Be(5.0, 2.0)
lik Be(12.0, 14.0)
post Be(16.0, 15.0)
(b)
Figure 3.6 (a) Updating a Beta(2, 2) prior with a Binomial likelihood with sufficient statistics N1=
3,N0=1 7to yield a Beta(5,19) posterior. (b) Updating a Beta(5, 2) prior with a Binomial likeli-
hood with sufficient statistics N1=1 1,N0=1 3to yield a Beta(16, 15) posterior. Figure generated by
binomialBetaPosteriorDemo .
3.3.3 Posterior
If we multiply the likelihood by the beta prior we get the following posterior (following Equa-
tion 3.14):
p(θ|D)∝Bin(N 1|θ,N0+N1)Beta(θ|a,b)Beta(θ|N1+a,N0+b) (3.16)
In particular, the posterior is obtained by adding the prior hyper-parameters to the empiricalcounts. For this reason, the hyper-parameters are known as pseudo counts. The strength of the

prior, also known as the effective sample size of the prior, is the sum of the pseudo counts,
a+b; this plays a role analogous to the data set size, N
1+N0=N.
Figure 3.6(a) gives an example where we update a weak Beta(2,2) prior with a peaked likelihood

1+N0=N.
function, corresponding to a large sample size; we see that the posterior is essentially identicalto the likelihood: since the data has overwhelmed the prior. Figure 3.6(b) gives an examplewhere we update a strong Beta(5,2) prior with a peaked likelihood function; now we see that theposterior is a “compromise” between the prior and likelihood.
Note that updating the posterior sequentially is equivalent to updating in a single batch.
To see this, suppose we have two data sets D
aandDbwith sufficient statistics Na
1,Na

1,Na
0and
Nb
1,Nb
0.L e tN1=Na
1+Nb
1andN0=Na
0+Nb
0be the sufficient statistics of the combined
datasets. In batch mode we have
p(θ|Da,Db)∝Bin(N 1|θ,N1+N0)Beta(θ|a,b)∝Beta(θ|N1+a,N0+b)(3.17)
In sequential mode, we have
p(θ|Da,Db)∝p(Db|θ)p(θ|Da) (3.18)
∝Bin(Nb
1|θ,Nb
1+Nb
0)Beta(θ|Na
1+a,Na
0+b) (3.19)
∝Beta(θ|Na
1+Nb
1+a,Na
0+Nb
0+b) (3.20)
This makes Bayesian inference particularly well-suited to online learning, as we will see later.

76 Chapter 3. Generative models for discrete data
3.3.3.1 Posterior mean and mode
From Equation 2.62, the MAP estimate is given by
ˆθMAP=a+N1−1
a+b+N−2(3.21)
If we use a uniform prior, then the MAP estimate reduces to the MLE, which is just the empirical
fraction of heads:
ˆθMLE=N1
N(3.22)
This makes intuitive sense, but it can also be derived by applying elementary calculus tomaximize the likelihood function in Equation 3.11. (Exercise 3.1).
By contrast, the posterior mean is given by,
θ=a+N1
a+b+N(3.23)
This difference between the mode and the mean will prove important later.
We will now show that the posterior mean is convex combination of the prior mean and the
MLE, which captures the notion that the posterior is a compromise between what we previouslybelieved and what the data is telling us.
Letα
0=a+bbe theequivalent sample size of the prior, which controls its strength, and
let the prior mean be m1=a/α0. Then the posterior mean is given by
E[θ|D]=α0m1+N1
N+α0=α0
N+α0m1+N
N+α0N1
N=λm1+(1−λ)ˆθMLE (3.24)

Letα
N+α0=α0
N+α0m1+N
N+α0N1
whereλ=α0
N+α 0is the ratio of the prior to posterior equivalent sample size. So the weaker the
prior, the smaller is λ, and hence the closer the posterior mean is to the MLE. One can show
similarly that the posterior mode is a convex combination of the prior mode and the MLE, andthat it too converges to the MLE.
3.3.3.2 Posterior variance
The mean and mode are point estimates, but it is useful to know how much we can trust them.The variance of the posterior is one way to measure this. The variance of the Beta posterior isgiven by
var[θ|D]=(a+N
1)(b+N0)

1)(b+N0)
(a+N1+b+N0)2(a+N1+b+N0+1)(3.25)
We can simplify this formidable expression in the case that N≫a,b,t og e t
var[θ|D]≈N1N0
NNN=ˆθ(1−ˆθ)
N(3.26)
whereˆθis the MLE. Hence the “error bar” in our estimate (i.e., the posterior standard deviation),
is given by
σ=√
var[θ|D]≈√
ˆθ(1−ˆθ)
N(3.27)

3.3. The beta-binomial model 77
We see that the uncertainty goes down at a rate of 1/√
N. Note, however, that the uncertainty
(variance) is maximized when ˆθ=0.5, and is minimized when ˆθis close to 0 or 1. This means
it is easier to be sure that a coin is biased than to be sure that it is fair.
3.3.4 Posterior predictive distribution
So far, we have been focusing on inference of the unknown parameter(s). Let us now turn our
attention to prediction of future observable data.
Consider predicting the probability of heads in a single future trial under a Beta(a,b)poste-
rior. We have
p(˜x=1|D)=∫1
0p(x=1|θ)p(θ|D)dθ (3.28)
=∫1
0θBeta(θ|a,b)dθ=E[θ|D]=a
a+b(3.29)
Thus we see that the mean of the posterior predictive distribution is equivalent (in this case) toplugging in the posterior mean parameters: p(˜x|D)=B e r (˜x|E[θ|D]).
3.3.4.1 Overﬁtting and the black swan paradox
Suppose instead that we plug-in the MLE, i.e., we use p(˜x|D)≈Ber(˜x|ˆθ
MLE). Unfortunately,

=∫1
a+b(3.29)
this approximation can perform quite poorly when the sample size is small. For example,suppose we have seen N=3tails in a row. The MLE is ˆθ=0/3=0, since this makes the
observed data as probable as possible. However, using this estimate, we predict that heads areimpossible. This is called the zero count problem or thesparse data problem, and frequently

occurs when estimating counts from small amounts of data. One might think that in the eraof “big data”, such concerns are irrelevant, but note that once we partition the data based oncertain criteria — such as the number of times a speciﬁc person has engaged in a speciﬁc activity
— the sample sizes can become much smaller. This problem arises, for example, when tryingto perform personalized recommendation of web pages. Thus Bayesian methods are still useful,even in the big data regime (Jordan 2011).

The zero-count problem is analogous to a problem in philosophy called the black swan

paradox. This is based on the ancient Western conception that all swans were white. Inthat context, a black swan was a metaphor for something that could not exist. (Black swanswere discovered in Australia by European explorers in the 17th Century.) The term “black swanparadox” was ﬁrst coined by the famous philosopher of science Karl Popper; the term has alsobeen used as the title of a recent popular book (Taleb 2007). This paradox was used to illustratethe problem of induction, which is the problem of how to draw general conclusions about the
future from speciﬁc observations from the past.
Let us now derive a simple Bayesian solution to the problem. We will use a uniform prior, so

a=b=1. In this case, plugging in the posterior mean gives Laplace’s rule of succession
p(˜x=1|D)=N
1+1
N1+N0+2(3.30)
This justiﬁes the common practice of adding 1 to the empirical counts, normalizing and thenplugging them in, a technique known as add-one smoothing. (Note that plugging in the MAP

78 Chapter 3. Generative models for discrete data
parameters would not have this smoothing effect, since the mode has the form ˆθ=N1+a−1
N+a+b−2,
which becomes the MLE if a=b=1.)
3.3.4.2 Predicting the outcome of multiple future trials
Suppose now we were interested in predicting the number of heads, x,i nMfuture trials. This
is given by
p(x|D,M)=∫1
0Bin(x|θ,M)Beta(θ|a,b)dθ (3.31)
=(M
x)1
B(a,b)∫1
0θx(1−θ)M−xθa−1(1−θ)b−1dθ (3.32)
We recognize the integral as the normalization constant for a Beta(a+x,M−x+b)distribution.
Hence
∫1
0θx(1−θ)M−xθa−1(1−θ)b−1dθ=B(x+a,M−x+b) (3.33)
Thus we ﬁnd that the posterior predictive is given by the following, known as the (compound)
beta-binomial distribution:
Bb(x|a,b,M)≜(M
x)B(x+a,M−x+b)
B(a,b)(3.34)
This distribution has the following mean and variance
E[x]=Ma
a+b,var[x]=Mab
(a+b)2(a+b+M)
a+b+1(3.35)
IfM=1, and hence x∈{0,1}, we see that the mean becomes E[x|D]=p(x=1|D)=a
a+b,
which is consistent with Equation 3.29.

Hence
∫1
E[x]=Ma
a+b,
This process is illustrated in Figure 3.7(a). We start with a Beta(2,2) prior, and plot the
posterior predictive density after seeing N1=3heads and N0=1 7tails. Figure 3.7(b) plots
a plug-in approximation using a MAP estimate. We see that the Bayesian prediction has longertails, spreading its probablity mass more widely, and is therefore less prone to overﬁtting andblackswan type paradoxes.
3.4 The Dirichlet-multinomial model

a+b,
In the previous section, we discussed how to infer the probability that a coin comes up heads.In this section, we generalize these results to infer the probability that a dice with Ksides
comes up as face k. This might seem like another toy exercise, but the methods we will study
are widely used to analyse text data, biosequence data, etc., as we will see later.

3.4. The Dirichlet-multinomial model 79
0 1 2 3 4 5 6 7 8 9 1000.050.10.150.20.250.30.35posterior predictive
(a)0 1 2 3 4 5 6 7 8 9 1000.050.10.150.20.250.30.35plugin predictive
(b)
Figure 3.7 (a) Posterior predictive distributions after seeing N1=3,N0=1 7. (b) Plugin approximation.
Figure generated by betaBinomPostPredDemo .
3.4.1 Likelihood
Suppose we observe Ndice rolls, D={x1,...,x N},w h e r e xi∈{1,...,K}. If we assume
the data is iid, the likelihood has the form
p(D|θ)=K∏
k=1θNk
k(3.36)
whereNk=∑N
i=1I(yi=k)is the number of times event koccured (these are the sufficient
statistics for this model). The likelihood for the multinomial model has the same form, up to an
irrelevant constant factor.
3.4.2 Prior
Since the parameter vector lives in the K-dimensional probability simplex, we need a prior that
has support over this simplex. Ideally it would also be conjugate. Fortunately, the Dirichletdistribution (Section 2.5.4) satisﬁes both criteria. So we will use the following prior:
Dir(θ|α)=1
B(α)K∏

B(α)K∏
k=1θαk−1
kI(x∈SK) (3.37)
3.4.3 Posterior
Multiplying the likelihood by the prior, we ﬁnd that the posterior is also Dirichlet:
p(θ|D)∝p(D|θ)p(θ) (3.38)
∝K∏
k=1θNk
kθαk−1
k=K∏
k=1θαk+Nk−1
k(3.39)
= Dir(θ |α1+N1,...,α K+NK) (3.40)

80 Chapter 3. Generative models for discrete data
We see that the posterior is obtained by adding the prior hyper-parameters (pseudo-counts) αk
to the empirical counts Nk.
We can derive the mode of this posterior (i.e., the MAP estimate) by using calculus. However,
we must enforce the constraint that∑
kθk=1.2. We can do this by using a Lagrange
multiplier. The constrained objective function, or Lagrangian, is given by the log likelihood
plus log prior plus the constraint:
ℓ(θ,λ)=∑
kNklogθk+∑
k(αk−1)logθk+λ(
1−∑
kθk)
(3.41)
To simplify notation, we deﬁne N′
k≜Nk+αk−1. Taking derivatives with respect to λyields
the original constraint:
∂ℓ
∂λ=(
1−∑
kθk)
=0 (3.42)
Taking derivatives with respect to θkyields
∂ℓ
∂θk=N′
k
θk−λ=0 (3.43)
N′
k=λθk (3.44)
We can solve for λusing the sum-to-one constraint:
∑
kN′
k=λ∑
kθk (3.45)
N+α0−K=λ (3.46)
whereα0≜∑K
k=1αkis the equivalent sample size of the prior. Thus the MAP estimate is
given by
ˆθk=Nk+αk−1
N+α0−K(3.47)

ℓ(θ,λ)=∑
1−∑
kθk)
(3.41)
∂ℓ
∂λ=(
1−∑
kθk)
=0 (3.42)
∂ℓ
∂θk=N′
k
N′
∑
kN′
k=λ∑
kθk (3.45)
given by
which is consistent with Equation 2.77. If we use a uniform prior, αk=1, we recover the MLE:
ˆθk=Nk/N (3.48)
This is just the empirical fraction of times face kshows up.
2. We do not need to explicitly enforce the constraint that θk≥0since the gradient of the objective has the form
Nk/θk−λ; so negative values would reduce the objective, rather than maximize it. (Of course, this does not preclude
settingθk=0, and indeed this is the optimal solution if Nk=0andαk=1.)

3.4. The Dirichlet-multinomial model 81
3.4.4 Posterior predictive
The posterior predictive distribution for a single multinoulli trial is given by the following
p(X=j|D)=∫
p(X=j|θ)p(θ|D)dθ (3.49)
=∫
p(X=j|θj)[∫
p(θ−j,θj|D)dθ−j]
dθj (3.50)
=∫
θjp(θj|D)dθj=E[θj|D]=αj+Nj∑
k(αk+Nk)=αj+Nj
α0+N(3.51)
whereθ−jare all the components of θexceptθj. See also Exercise 3.13.
The above expression avoids the zero-count problem, just as we saw in Section 3.3.4.1. In
fact, this form of Bayesian smoothing is even more important in the multinomial case than thebinary case, since the likelihood of data sparsity increases once we start partitioning the datainto many categories.
3.4.4.1 Worked example: language models using bag of words
One application of Bayesian smoothing using the Dirichlet-multinomial model is to language
modeling, which means predicting which words might occur next in a sequence. Here wewill take a very simple-minded approach, and assume that the i’th word, X
i∈{1,...,K},i s

sampled independently from all the other words using a Cat(θ)distribution. This is called the
bag of words model. Given a past sequence of words, how can we predict which one is likely
to come next?
For example, suppose we observe the following sequence (part of a children’s nursery rhyme):
Mary had a little lamb, little lamb, little lamb,Mary had a little lamb, its fleece as white as snow
Furthermore, suppose our vocabulary consists of the following words:mary lamb little big fleece white black snow rain unk
1 2 34 56 7 8 9 1 0
Hereunkstands for unknown, and represents all other words that do not appear elsewhere on

the list. To encode each line of the nursery rhyme, we ﬁrst strip off punctuation, and remove
anystop words such as “a”, “as”, “the”, etc. We can also perform stemming, which means
reducing words to their base form, such as stripping off the ﬁnal sin plural words, or the ing
from verbs (e.g., runningbecomes run). In this example, no words need stemming. Finally, we
replace each word by its index into the vocabulary to get:
11 0323232
11 0321 051 068
We now ignore the word order, and count how often each word occurred, resulting in a
histogram of word counts:

82 Chapter 3. Generative models for discrete data
Token123 4 56789 1 0
Wordmary lamb little big ﬂeece white black snow rain unk
Count244 0 1 1010 4
Denote the above counts by Nj. If we use a Dir(α)prior for θ, the posterior predictive is
just
p(˜X=j|D)=E[θj|D]=αj+Nj∑
j′αj′+Nj′=1+Nj
10+17(3.52)
If we setαj=1,w eg e t
p(˜X=j|D)=( 3/27,5/27,5/27,1/27,2/27,2/27,1/27,2/27,1/27,5/27) (3.53)
The modes of the predictive distribution are X=2(“lamb”) and X=1 0(“unk”). Note that the
words “big”, “black” and “rain” are predicted to occur with non-zero probability in the future,
even though they have never been seen before. Later on we will see more sophisticated languagemodels.
3.5 Naive Bayes classiﬁers
In this section, we discuss how to classify vectors of discrete-valued features, x∈{1,...,K}D,
whereKis the number of values for each feature, and Dis the number of features. We will use
a generative approach. This requires us to specify the class conditional distribution, p(x|y=c).

The simplest approach is to assume the features are conditionally independent given the class
label. This allows us to write the class conditional density as a product of one dimensionaldensities:
p(x|y=c,θ)=D∏
j=1p(xj|y=c,θjc) (3.54)
The resulting model is called a naive Bayes classiﬁer (NBC).
The model is called “naive” since we do not expect the features to be independent, even

conditional on the class label. However, even if the naive Bayes assumption is not true, it oftenresults in classiﬁers that work well (Domingos and Pazzani 1997). One reason for this is that themodel is quite simple (it only has O(CD)parameters, for Cclasses and Dfeatures), and hence
it is relatively immune to overﬁtting.
The form of the class-conditional density depends on the type of each feature. We give some
possibilities below:
• In the case of real-valued features, we can use the Gaussian distribution: p(x|y=c,θ)=∏
D

D
j=1N(xj|μjc,σ2
jc),w h e r eμjcis the mean of feature jin objects of class c, andσ2
jcis its
variance.
• In the case of binary features, xj∈{0,1}, we can use the Bernoulli distribution: p(x|y=
c,θ)=∏Dj=1Ber(xj|μjc),w h e r eμjcis the probability that feature joccurs in class c.
This is sometimes called the multivariate Bernoulli naive Bayes model. We will see an
application of this below.

3.5. Naive Bayes classiﬁers 83
• In the case of categorical features, xj∈{1,...,K}, we can model use the multinoulli
distribution: p(x|y=c,θ)=∏D
j=1Cat(xj|μjc),w h e r eμjcis a histogram over the K
possible values for xjin classc.
Obviously we can handle other kinds of features, or use different distributional assumptions.
Also, it is easy to mix and match features of different types.
3.5.1 Model ﬁtting
We now discuss how to “train” a naive Bayes classiﬁer. This usually means computing the MLEor the MAP estimate for the parameters. However, we will also discuss how to compute the fullposterior, p(θ|D).
3.5.1.1 MLE for NBC
The probability for a single data case is given by
p(x
i,yi|θ)=p(yi|π)∏
jp(xij|θj)=∏
cπI(yi=c)
c∏
j∏
cp(xij|θjc)I(yi=c)(3.55)
Hence the log-likelihood is given by
logp(D|θ)=C∑
c=1Nclogπc+D∑
j=1C∑
c=1∑
i:yi=clogp(xij|θjc) (3.56)
We see that this expression decomposes into a series of terms, one concerning π, andDC

p(x
cπI(yi=c)
c∏
j∏
j=1C∑
c=1∑
terms containing the θjc’s. Hence we can optimize all these parameters separately.
From Equation 3.48, the MLE for the class prior is given by
ˆπc=Nc
N(3.57)
whereNc≜∑
iI(yi=c)is the number of examples in class c.
The MLE for the likelihood depends on the type of distribution we choose to use for each
feature. For simplicity, let us suppose all features are binary, so xj|y=c∼Ber(θjc). In this
case, the MLE becomes
ˆθjc=Njc
Nc(3.58)
It is extremely simple to implement this model ﬁtting procedure: See Algorithm 8 for some
pseudo-code (and naiveBayesFit for some Matlab code). This algorithm obviously takes

ˆπc=Nc
N(3.57)
whereNc≜∑
ˆθjc=Njc
Nc(3.58)
O(ND)time. The method is easily generalized to handle features of mixed type. This simplicity
is one reason the method is so widely used.
Figure 3.8 gives an example where we have 2 classes and 600 binary features, representing the
presence or absence of words in a bag-of-words model. The plot visualizes the θcvectors for the
two classes. The big spike at index 107 corresponds to the word “subject”, which occurs in bothclasses with probability 1. (In Section 3.5.4, we discuss how to “ﬁlter out” such uninformativefeatures.)

84 Chapter 3. Generative models for discrete data
Algorithm 3.1: Fitting a naive Bayes classiﬁer to binary features
1Nc=0,Njc=0;
2fori=1:Ndo
3c=yi// Class label of i’th example;
4Nc:=Nc+1;
5forj=1:Ddo
6 ifxij=1then
7 Njc:=Njc+1
8ˆπc=Nc
N,ˆθjc=Njc
N
0 100 200 300 400 500 600 70000.10.20.30.40.50.60.70.80.91p(xj=1|y=1)
(a)0 100 200 300 400 500 600 70000.10.20.30.40.50.60.70.80.91p(xj=1|y=2)
(b)
Figure 3.8 Class conditional densities p(xj=1|y=c)for two document classes, corresponding to “X
windows” and “MS windows”. Figure generated by naiveBayesBowDemo .
3.5.1.2 Bayesian naive Bayes
The trouble with maximum likelihood is that it can overﬁt. For example, consider the example
in Figure 3.8: the feature corresponding to the word “subject” (call it feature j) always occurs
in both classes, so we estimate ˆθjc=1. What will happen if we encounter a new email which

(b)
does not have this word in it? Our algorithm will crash and burn, since we will ﬁnd thatp(y=c|x,ˆθ)=0for both classes! This is another manifestation of the black swan paradox
discussed in Section 3.3.4.1.
A simple solution to overﬁtting is to be Bayesian. For simplicity, we will use a factored prior:
p(θ)=p( π)D∏
j=1C∏
c=1p(θjc) (3.59)
We will use a Dir(α)prior for πand aBeta(β0,β1)prior for each θjc. Often we just take
α=1andβ=1, corresponding to add-one or Laplace smoothing.

3.5. Naive Bayes classiﬁers 85
Combining the factored likelihood in Equation 3.56 with the factored prior above gives the
following factored posterior:
p(θ|D)=p( π|D)D∏
j=1C∏
c=p(θjc|D) (3.60)
p(π|D)=D i r ( N1+α1...,NC+αC) (3.61)
p(θjc|D) = Beta((N c−Njc)+β0,Njc+β1) (3.62)
In other words, to compute the posterior, we just update the prior counts with the empirical
counts from the likelihood. It is straightforward to modify algorithm 8 to handle this version ofmodel “ﬁtting”.
3.5.2 Using the model for prediction
At test time, the goal is to compute
p(y=c|x,D)∝p(y=c|D)D∏
j=1p(xj|y=c,D) (3.63)
The correct Bayesian procedure is to integrate out the unknown parameters:
p(y=c|x,D)∝[∫
Cat(y=c|π)p(π|D)dπ]
(3.64)
D∏
j=1[∫
Ber(xj|y=c,θjc)p(θjc|D)]
(3.65)
Fortunately, this is easy to do, at least if the posterior is Dirichlet. In particular, from Equa-tion 3.51, we know the posterior predictive density can be obtained by simply plugging in theposterior mean parameters
θ. Hence
p(y=c|x,D)∝πcD∏

(3.64)
D∏
j=1[∫
(3.65)
θ. Hence
j=1(θjc)I(xj=1)(1−θjc)I(xj=0)(3.66)
θjk=Njc+β1
Nc+β0+β1(3.67)
πc=Nc+αc
N+α0(3.68)
whereα0=∑
cαc.
If we have approximated the posterior by a single point, p(θ|D)≈δˆθ(θ),w h e r eˆθmay be
the ML or MAP estimate, then the posterior predictive density is obtained by simply plugging inthe parameters, to yield a virtually identical rule:
p(y=c|x,D)∝ˆπ
cD∏
j=1(ˆθjc)I(xj=1)(1−ˆθjc)I(xj=0)(3.69)

86 Chapter 3. Generative models for discrete data
The only difference is we replaced the posterior mean θwith the posterior mode or MLE ˆθ.
However, this small difference can be important in practice, since the posterior mean will result
in less overﬁtting (see Section 3.4.4.1).
3.5.3 The log-sum-exp trick
We now discuss one important practical detail that arises when using generative classiﬁers of anykind. We can compute the posterior over class labels using Equation 2.13, using the appropriateclass-conditional density (and a plug-in approximation). Unfortunately a naive implementationof Equation 2.13 can fail due to numerical underﬂow. The problem is that p(x|y=c)is often
a very small number, especially if xis a high-dimensional vector. This is because we require
that∑
xp(x|y)=1, so the probability of observing any particular high-dimensional vector is
small. The obvious solution is to take logs when applying Bayes rule, as follows:
logp(y=c|x)=b c−log[C∑
c′=1ebc′]
(3.70)
bc≜logp(x|y=c)+logp(y=c) (3.71)

that∑
c′=1ebc′]
(3.70)
However, this requires evaluating the following expression
log[∑
c′ebc′]=l o g∑
c′p(y=c′,x)=l o gp(x) (3.72)
and we can’t add up in the log domain. Fortunately, we can factor out the largest term, and justrepresent the remaining numbers relative to that. For example,
log(e
−120+e−121)=l o g(
e−120(e0+e−1))
= log(e0+e−1)−120 (3.73)
In general, we have
log∑
cebc=l o g[
(∑
cebc−B)eB]
=[
log(∑
cebc−B)]
+B (3.74)
whereB=m a x cbc. This is called the log-sum-exp trick, and is widely used. (See the function
logsumexp for an implementation.)
This trick is used in Algorithm 1 which gives pseudo-code for using an NBC to compute

log(e
log∑
cebc=l o g[
(∑
=[
log(∑
cebc−B)]
+B (3.74)
p(yi|xi,ˆθ). See naiveBayesPredict for the Matlab code. Note that we do not need the
log-sum-exp trick if we only want to compute ˆyi, since we can just maximize the unnormalized
quantitylogp(yi=c)+logp(xi|y=c).
3.5.4 Feature selection using mutual information
Since an NBC is ﬁtting a joint distribution over potentially many features, it can suffer fromoverﬁtting. In addition, the run-time cost is O(D), which may be too high for some applications.
One common approach to tackling both of these problems is to perform feature selection,t o

remove “irrelevant” features that do not help much with the classiﬁcation problem. The simplestapproach to feature selection is to evaluate the relevance of each feature separately, and then

3.5. Naive Bayes classiﬁers 87
Algorithm 3.2: Predicting with a naive bayes classiﬁer for binary features
1fori=1:Ndo
2forc=1:Cdo
3 Lic= logˆπc;
4 forj=1:Ddo
5 ifxij=1thenLic:=Lic+logˆθjcelseLic:=Lic+log(1− ˆθjc)
6pic=e x p (Lic−logsumexp( Li,:));
7ˆyi=a r g m a xcpic;
take the top K,w h e r eKis chosen based on some tradeoff between accuracy and complexity.
This approach is known as variable ranking, ﬁltering,o rscreening.
One way to measure relevance is to use mutual information (Section 2.8.3) between feature
Xjand the class label Y:
I(X,Y)=∑
xj∑
yp(xj,y)logp(xj,y)
p(xj)p(y)(3.75)
The mutual information can be thought of as the reduction in entropy on the label distribution
once we observe the value of feature j. If the features are binary, it is easy to show (Exercise 3.21)
that the MI can be computed as follows
Ij=∑
c[
θjcπclogθjc
θj+(1−θjc)πclog1−θjc
1−θj]
(3.76)
whereπc=p(y=c),θjc=p(xj=1|y=c), andθj=p(xj=1 )=∑
cπcθjc. (All of these

I(X,Y)=∑
xj∑
Ij=∑
c[
1−θj]
(3.76)
quantities can be computed as a by-product of ﬁtting a naive Bayes classiﬁer.)
Figure 3.1 illustrates what happens if we apply this to the binary bag of words dataset used in

Ij=∑
c[
1−θj]
(3.76)
Figure 3.8. We see that the words with highest mutual information are much more discriminativethan the words which are most probable. For example, the most probable word in both classesis “subject”, which always occurs because this is newsgroup data, which always has a subjectline. But obviously this is not very discriminative. The words with highest MI with the classlabel are (in decreasing order) “windows”, “microsoft”, “DOS” and “motif”, which makes sense,since the classes correspond to Microsoft Windows and X Windows.
3.5.5 Classifying documents using bag of words

Document classiﬁcation is the problem of classifying text documents into different categories.
One simple approach is to represent each document as a binary vector, which records whethereach word is present or not, so x
ij=1iff wordjoccurs in document i, otherwise xij=0.
We can then use the following class conditional density:
p(xi|yi=c,θ)=D∏
j=1Ber(xij|θjc)=D∏
j=1θI(xij)
jc(1−θjc)I(1−xij)(3.77)

88 Chapter 3. Generative models for discrete data
class 1 prob class 2 prob highest MI MI
subject 0.998 subject 0.998 windows 0.215
this 0.628 windows 0.639 microsoft 0.095
with 0.535 this 0.540 dos 0.092
but 0.471 with 0.538 motif 0.078
you 0.431 but 0.518 window 0.067
Table 3.1 We list the 5 most likely words for class 1 (X windows) and class 2 (MS windows). We also show
the 5 words with highest mutual information with class label. Produced by naiveBayesBowDemo
This is called the Bernoulli product model,o rt h ebinary independence model.
However, ignoring the number of times each word occurs in a document loses some in-
formation (McCallum and Nigam 1998). A more accurate representation counts the number
of occurrences of each word. Speciﬁcally, let xibe a vector of counts for document i,s o
xij∈{0,1,...,N i},w h e r eNiis the number of terms in document i(so∑D
j=1xij=Ni). For
the class conditional densities, we can use a multinomial distribution:
p(xi|yi=c,θ)=M u (xi|Ni,θc)=Ni!∏Dj=1xij!D∏
j=1θxij
jc (3.78)

j=1θxij
jc (3.78)
where we have implicitly assumed that the document length Niis independent of the class.
Hereθjcis the probability of generating word jin documents of class c; these parameters satisfy
the constraint that∑Dj=1θjc=1for each class c.3
Although the multinomial classiﬁer is easy to train and easy to use at test time, it does not
work particularly well for document classiﬁcation. One reason for this is that it does not take
into account the burstiness of word usage. This refers to the phenomenon that most words

never appear in any given document, but if they do appear once, they are likely to appear morethan once, i.e., words occur in bursts.
The multinomial model cannot capture the burstiness phenomenon. To see why, note that
Equation 3.78 has the form θNij
jc, and since θjc≪1for rare words, it becomes increasingly

unlikely to generate many of them. For more frequent words, the decay rate is not as fast. Tosee why intuitively, note that the most frequent words are function words which are not speciﬁcto the class, such as “and”, “the”, and “but”; the chance of the word “and” occuring is prettymuch the same no matter how many time it has previously occurred (modulo document length),so the independence assumption is more reasonable for common words. However, since rarewords are the ones that matter most for classiﬁcation purposes, these are the ones we want tomodel the most carefully.

Various ad hoc heuristics have been proposed to improve the performance of the multinomial
document classiﬁer (Rennie et al. 2003). We now present an alternative class conditional densitythat performs as well as these ad hoc methods, yet is probabilistically sound (Madsen et al.2005).
3. Since Equation 3.78 models each word independently, this model is often called a naive Bayes classiﬁer, although
technically the features xijare not independent, because of the constraint∑
jxij=Ni.

3.5. Naive Bayes classiﬁers 89
Suppose we simply replace the multinomial class conditional density with the Dirichlet
Compound Multinomial orDCMdensity, deﬁned as follows:
p(xi|yi=c,α)=∫
Mu(xi|Ni,θc)Dir(θ c|αc)dθc=Ni!∏D
j=1xij!B(xi+αc)
B(αc)(3.79)
(This equation is derived in Equation 5.24.) Surprisingly this simple change is all that is needed
to capture the burstiness phenomenon. The intuitive reason for this is as follows: After seeingone occurence of a word, say word j, the posterior counts on θ
jgets updated, making another
occurence of word jmore likely. By contrast, if θjis ﬁxed, then the occurences of each word are
independent. The multinomial model corresponds to drawing a ball from an urn with Kcolors
of ball, recording its color, and then replacing it. By contrast, the DCM model corresponds todrawing a ball, recording its color, and then replacing it with one additional copy; this is calledthePolya urn.
Using the DCM as the class conditional density gives much better results than using the

multinomial, and has performance comparable to state of the art methods, as described in(Madsen et al. 2005). The only disadvantage is that ﬁtting the DCM model is more complex; see(Minka 2000e; Elkan 2006) for the details.
Exercises
Exercise 3.1 MLE for the Bernoulli/ binomial model
Derive Equation 3.22 by optimizing the log of the likelihood in Equation 3.11.
Exercise 3.2 Marginal likelihood for the Beta-Bernoulli model

Exercises
In Equation 5.23, we showed that the marginal likelihood is the ratio of the normalizing constants:
p(D)=Z(α1+N1,α0+N0)
Z(α1,α0)=Γ(α1+N1)Γ(α0+N0)
Γ(α1+α0+N)Γ(α1+α0)
Γ(α1)Γ(α0)(3.80)
We will now derive an alternative derivation of this fact. By the chain rule of probability,
p(x1:N)=p(x1)p(x2|x1)p(x3|x1:2)... (3.81)
In Section 3.3.4, we showed that the posterior predictive distribution is
p(X=k|D1:N)=Nk+αk∑
iNi+αi≜Nk+αk
N+α(3.82)
wherek∈{0,1}andD1:Nis the data seen so far. Now suppose D=H,T,T,H,H orD=1,0,0,1,1.
Then
p(D)=α1
α·α0
α+1·α0+1
α+2·α1+1
α+3·α1+2
α+4(3.83)

N+α(3.82)
Then
p(D)=α1
α·α0
α+1·α0+1
α+2·α1+1
α+3·α1+2
α+4(3.83)
=[α1(α1+1)(α1+2)][α0(α0+1)]
α(α+1)···(α+4)(3.84)
=[(α1)···(α1+N1−1)][(α 0)···(α0+N0−1)]
(α)···(α+N−1)(3.85)
Show how this reduces to Equation 3.80 by using the fact that, for integers, (α−1)! = Γ(α ).

90 Chapter 3. Generative models for discrete data
Exercise 3.3 Posterior predictive for Beta-Binomial model
Recall from Equation 3.32 that the posterior predictive for the Beta-Binomial is given by
p(x|n,D)=Bb (x|α′
0,α′
1,n) (3.86)
=B(x+α′
1,n−x+α′
0)
B(α′
1,α′
0)(n
x)
(3.87)
Prove that this reduces to
p(˜x=1|D)=α′
1
α′
0+α′
1(3.88)
whenn=1(and hence x∈{0,1}). i.e., show that
Bb(1|α′
1,α′
0,1) =α′
1
α′
1+α′
0(3.89)
Hint: use the fact that
Γ(α0+α1+1)=(α0+α1+1)Γ(α0+α1) (3.90)
Exercise 3.4 Beta updating from censored likelihood
(Source: Gelman.) Suppose we toss a coin n=5times. Let Xbe the number of heads. We observe that
there are fewer than 3 heads, but we don’t know exactly how many. Let the prior probability of heads be
p(θ)=B e t a ( θ|1,1). Compute the posterior p(θ|X<3)up to normalization constants, i.e., derive an
expression proportional to p(θ,X <3). Hint: the answer is a mixture distribution.
Exercise 3.5 Uninformative prior for log-odds ratio
Let
φ= logit(θ )=l o gθ
1−θ(3.91)

Let
1−θ(3.91)
Show that if p(φ)∝1, thenp(θ)∝Beta(θ|0,0). Hint: use the change of variables formula.
Exercise 3.6 MLE for the Poisson distribution
The Poisson pmf is deﬁned as Poi(x|λ)=e−λλx
x!,f o rx∈{0,1,2,...}whereλ>0is the rate
parameter. Derive the MLE.
Exercise 3.7 Bayesian analysis of the Poisson distribution
In Exercise 3.6, we deﬁned the Poisson distribution with rate λand derived its MLE. Here we perform a
conjugate Bayesian analysis.
a. Derive the posterior p(λ|D)assuming a conjugate prior p(λ)=G a (λ|a,b)∝λa−1e−λb. Hint: the

posterior is also a Gamma distribution.
b. What does the posterior mean tend to as a→0andb→0? (Recall that the mean of a Ga(a,b)
distribution is a/b.)
Exercise 3.8 MLE for the uniform distribution
(Source: Kaelbling.) Consider a uniform distribution centered on 0 with width 2a. The density function is
given by
p(x)=1
2aI(x∈[−a,a]) (3.92)

3.5. Naive Bayes classiﬁers 91
a. Given a data set x1,...,x n, what is the maximum likelihood estimate of a(call itˆa)?
b. What probability would the model assign to a new data point xn+1usingˆa?
c. Do you see any problem with the above approach? Brieﬂy suggest (in words) a better approach.
Exercise 3.9 Bayesian analysis of the uniform distribution
Consider the uniform distribution Unif(0,θ). The maximum likelihood estimate is ˆθ=m a x (D),a sw e
saw in Exercise 3.8, but this is unsuitable for predicting future data since it puts zero probability mass
outside the training data. In this exercise, we will perform a Bayesian analysis of the uniform distribution(following (Minka 2001a)). The conjugate prior is the Pareto distribution, p(θ)=Pareto(θ|b,K), deﬁned in
Section 2.4.6. Given a Pareto prior, the joint distribution of θandD=(x
1,...,x N)is
p(D,θ)=KbK
θN+K+1I(θ≥max(D)) (3.93)
Letm=m a x (D). The evidence (the probability that all Nsamples came from the same uniform
distribution) is
p(D)=∫∞
mKbK

p(D)=∫∞
mKbK
θN+K+1dθ (3.94)
={K
(N+K)bNifm≤b
KbK
(N+K)mN+Kifm>b(3.95)
Derive the posterior p(θ|D), and show that if can be expressed as a Pareto distribution.
Exercise 3.10 Taxicab (tramcar) problem
Suppose you arrive in a new city and see a taxi numbered 100. How many taxis are there in this city? Letus assume taxis are numbered sequentially as integers starting from 0, up to some unknown upper boundθ. (We number taxis from 0 for simplicity; we can also count from 1 without changing the analysis.) Hence

p(D)=∫∞
mKbK
={K
KbK
the likelihood function is p(x)=U(0,θ), the uniform distribution. The goal is to estimate θ. We will use
the Bayesian analysis from Exercise 3.9.
a. Suppose we see one taxi numbered 100, so D={100},m= 100,N=1. Using an (improper)
non-informative prior on θof the form p(θ)=Pa(θ|0,0)∝1/θ, what is the posterior p(θ|D)?
b. Compute the posterior mean, mode and median number of taxis in the city, if such quantities exist.

c. Rather than trying to compute a point estimate of the number of taxis, we can compute the predictive
density over the next taxicab number using
p(D
′|D,α)=∫
p(D′|θ)p(θ|D,α)dθ=p(D′|β) (3.96)
whereα=(b,K)are the hyper-parameters, β=(c,N+K)are the updated hyper-parameters. Now
consider the case D={m}, andD′={x}. Using Equation 3.95, write down an expression for
p(x|D,α) (3.97)
As above, use a non-informative prior b=K=0.
d. Use the predictive density formula to compute the probability that the next taxi you will see (say,

p(D
′|D,α)=∫
the next day) has number 100, 50 or 150, i.e., compute p(x= 100|D,α ),p(x=5 0|D,α),p(x=
150|D,α ).
e. Brieﬂy describe (1-2 sentences) some ways we might make the model more accurate at prediction.

92 Chapter 3. Generative models for discrete data
Exercise 3.11 Bayesian analysis of the exponential distribution
A lifetime Xof a machine is modeled by an exponential distribution with unknown parameter θ. The
likelihood is p(x|θ)=θe−θxforx≥0,θ>0.
a. Show that the MLE is ˆθ=1/x,w h e r ex=1
N∑N
i=1xi.
b. Suppose we observe X1=5,X2=6,X3=4(the lifetimes (in years) of 3 different iid machines).
What is the MLE given this data?
c. Assume that an expert believes θshould have a prior distribution that is also exponential
p(θ) = Expon( θ|λ) (3.98)
Choose the prior parameter, call it ˆλ, such that E[θ]=1/3. Hint: recall that the Gamma distribution
has the form
Ga(θ|a,b)∝θa−1e−θb(3.99)
and its mean is a/b.
d. What is the posterior, p(θ|D,ˆλ)?
e. Is the exponential prior conjugate to the exponential likelihood?
f. What is the posterior mean, E[
θ|D,ˆλ]
?
g. Explain why the MLE and posterior mean differ. Which is more reasonable in this example?
Exercise 3.12 MAP estimation for the Bernoulli with non-conjugate priors

θ|D,ˆλ]
?
(Source: Jaakkola.) In the book, we discussed Bayesian inference of a Bernoulli rate parameter with the
priorp(θ)=B e t a ( θ|α,β). We know that, with this prior, the MAP estimate is given by
ˆθ=N1+α−1
N+α+β−2(3.100)
whereN1is the number of heads, N0is the number of tails, and N=N0+N1is the total number of
trials.
a. Now consider the following prior, that believes the coin is fair, or is slightly biased towards tails:
p(θ)=⎧
⎨
⎩0.5ifθ=0.5
0.5ifθ=0.4
0otherwise(3.101)

ˆθ=N1+α−1
trials.
p(θ)=⎧
⎨
Derive the MAP estimate under this prior as a function of N1andN.
b. Suppose the true parameter is θ=0.41. Which prior leads to a better estimate when Nis small?
Which prior leads to a better estimate when Nis large?
Exercise 3.13 Posterior predictive distribution for a batch of data with the dirichlet-multinomial model
In Equation 3.51, we gave the the posterior predictive distribution for a single multinomial trial using a
dirichlet prior. Now consider predicting a batchof new data, ˜D=(X1,...,X m), consisting of msingle

multinomial trials (think of predicting the next mwords in a sentence, assuming they are drawn iid).
Derive an expression for
p(˜D|D,α) (3.102)

3.5. Naive Bayes classiﬁers 93
Your answer should be a function of α, and the old and new counts (sufficient statistics), deﬁned as
Nold
k=∑
i∈DI(xi=k) (3.103)
Nnew
k=∑
i∈˜DI(xi=k) (3.104)
Hint: recall that, for a vector of counts, N1:K, the marginal likelihood (evidence) is given by
p(D|α)=Γ(α)
Γ(N+α)∏
kΓ(Nk+αk)
Γ(αk)(3.105)
whereα=∑
kαkandN=∑
kNk.
Exercise 3.14 Posterior predictive for Dirichlet-multinomial
(Source: Koller.).
a. Suppose we compute the empirical distribution over letters of the Roman alphabet plus the space
character (a distribution over 27 values) from 2000 samples. Suppose we see the letter “e” 260 times.
What isp(x2001=e|D), if we assume θ∼Dir(α1,...,α 27),w h e r eα k=1 0for allk?
b. Suppose, in the 2000 samples, we saw “e” 260 times, “a” 100 times, and “p” 87 times. What is
p(x2001=p,x2002=a|D), if we assume θ∼Dir(α1,...,α 27),w h e r eαk=1 0for allk? Show
your work.
Exercise 3.15 Setting the beta hyper-parameters

your work.
Supposeθ∼β(α1,α2)and we believe that E[θ]=mandvar[θ]=v. Using Equation 2.62, solve for
α1andα2in terms of mandv. What values do you get if m=0.7andv=0.22?
Exercise 3.16 Setting the beta hyper-parameters II
(Source: Draper.) Suppose θ∼β(α1,α2)and we believe that E[θ]=mandp(ℓ<θ<u )=0.95.
Write a program that can solve for α1andα2in terms of m,ℓandu. Hint: write α2as a function of α1
andm, so the pdf only has one unknown; then write down the probability mass contained in the interval
as an integral, and minimize its squared discrepancy from 0.95. What values do you get if m=0.15,

ℓ=0.05andu=0.3? What is the equivalent sample size of this prior?
Exercise 3.17 Marginal likelihood for beta-binomial under uniform prior
Suppose we toss a coin Ntimes and observe N1heads. Let N1∼Bin(N,θ)andθ∼Beta(1,1). Show
that the marginal likelihood is p(N1|N)=1/(N+1). Hint:Γ(x+1)=x!ifxis an integer.
Exercise 3.18 Bayes factor for coin tossing
Suppose we toss a coin N=1 0times and observe N1=9heads. Let the null hypothesis be that the

coin is fair, and the alternative be that the coin can have any bias, so p(θ) = Unif(0, 1). Derive the
Bayes factor BF1,0in favor of the biased coin hypothesis. What if N= 100andN1=9 0? Hint: see
Exercise 3.17.
Exercise 3.19 Irrelevant features with naive Bayes
(Source: Jaakkola.) Let xiw=1if wordwoccurs in document iandxiw=0otherwise. Let θcwbe the
estimated probability that word woccurs in documents of class c. Then the log-likelihood that document

94 Chapter 3. Generative models for discrete data
xbelongs to class cis
logp(xi|c,θ)=l o gW∏
w=1θxiwcw(1−θcw)1−xiw(3.106)
=W∑
w=1xiwlogθcw+(1−xiw)log(1−θcw) (3.107)
=W∑
w=1xiwlogθcw
1−θcw+∑
wlog(1−θcw) (3.108)
whereWis the number of words in the vocabulary. We can write this more succintly as
logp(xi|c,θ)=φ (xi)Tβc (3.109)
wherexi=(xi1,...,x iW)is a bit vector, φ(xi)=(xi,1), and
βc=( l o gθc1
1−θc1,...,logθcW
1−θcW,∑
wlog(1−θcw))T(3.110)
We see that this is a linear classiﬁer, since the class-conditional density is a linear function (an inner
product) of the parameters βc.
a. Assuming p(C=1 )=p (C=2 )=0 .5, write down an expression for the log posterior odds ratio,
log2p(c=1| xi)
p(c=2| xi), in terms of the features φ(xi)and the parameters β1andβ2.
b. Intuitively, words that occur in both classes are not very “discriminative”, and therefore should not
affect our beliefs about the class label. Consider a particular word w. State the conditions on θ1,wand

θ2,w(or equivalently the conditions on β1,w,β2,w) under which the presence or absence of win a
test document will have no effect on the class posterior (such a word will be ignored by the classiﬁer).Hint: using your previous result, ﬁgure out when the posterior odds ratio is 0.5/0.5.
c. The posterior mean estimate of θ, using a Beta(1,1) prior, is given by
ˆθ
cw=1+∑
i∈cxiw
2+nc(3.111)
where the sum is over the ncdocuments in class c. Consider a particular word w, and suppose it
always occurs in every document (regardless of class). Let there be n1documents of class 1 and n2be

ˆθ
cw=1+∑
i∈cxiw
the number of documents in class 2, where n1̸=n2(since e.g., we get much more non-spam than
spam; this is an example of class imbalance). If we use the above estimate for θcw, will word wbe
ignored by our classiﬁer? Explain why or why not.
d. What other ways can you think of which encourage “irrelevant” words to be ignored?
Exercise 3.20 Class conditional densities for binary data
Consider a generative classiﬁer for Cclasses with class conditional density p(x|y)and uniform class prior

p(y). Suppose all the Dfeatures are binary, xj∈{0,1}. If we assume all the features are conditionally
independent (the naive Bayes assumption), we can write
p(x|y=c)=D∏
j=1Ber(xj|θjc) (3.112)
This requires DCparameters.

3.5. Naive Bayes classiﬁers 95
a. Now consider a different model, which we will call the “full” model, in which all the features are fully
dependent (i.e., we make no factorization assumptions). How might we represent p(x|y=c)in this
case? How many parameters are needed to represent p(x|y=c)?
b. Assume the number of features Dis ﬁxed. Let there be Ntraining cases. If the sample size Nis very
small, which model (naive Bayes or full) is likely to give lower test set error, and why?
c. If the sample size Nis very large, which model (naive Bayes or full) is likely to give lower test set error,
and why?
d. What is the computational complexity of ﬁtting the full and naive Bayes models as a function of N
andD? Use big-Oh notation. (Fitting the model here means computing the MLE or MAP parameter
estimates. You may assume you can convert a D-bit vector to an array index in O(D)time.)
e. What is the computational complexity of applying the full and naive Bayes models at test time to a
single test case?

and why?
f. Suppose the test case has missing data. Let xvbe the visible features of size v, andxhbe the hidden
(missing) features of size h,w h e r e v+h=D. What is the computational complexity of computing
p(y|xv,ˆθ)for the full and naive Bayes models, as a function of vandh?
Exercise 3.21 Mutual information for naive Bayes classiﬁers with binary features
Derive Equation 3.76.
Exercise 3.22 Fitting a naive bayes spam ﬁlter by hand
(Source: Daphne Koller.). Consider a Naive Bayes model (multivariate Bernoulli version) for spam classiﬁca-

tion with the vocabulary V="secret", "offer", "low", "price", "valued", "customer", "today", "dollar", "million","sports", "is", "for", "play", "healthy", "pizza". We have the following example spam messages "million dollaroffer", "secret offer today", "secret is secret" and normal messages, "low price for valued customer", "playsecret sports today", "sports is healthy", "low price pizza". Give the MLEs for the following parameters:θspam,θsecret|spam,θsecret|non-spam,θsports|non-spam,θdollar|spam.

4 Gaussian models
4.1 Introduction
In this chapter, we discuss the multivariate Gaussian ormultivariate normal (MVN), which
is the most widely used joint probability density function for continuous variables. It will form
the basis for many of the models we will encounter in later chapters.
Unfortunately, the level of mathematics in this chapter is higher than in many other chapters.
In particular, we rely heavily on linear algebra and matrix calculus. This is the price one mustpay in order to deal with high-dimensional data. Beginners may choose to skip sections markedwith a *. In addition, since there are so many equations in this chapter, we have put a boxaround those that are particularly important.
4.1.1 Notation
Let us brieﬂy say a few words about notation. We denote vectors by boldface lower case letters,such asx. We denote matrices by boldface upper case letters, such as X. We denote entries in
a matrix by non-bold upper case letters, such as X
ij.

ij.
All vectors are assumed to be column vectors unless noted otherwise. We use [x1,...,x D]to
denote a column vector created by stacking Dscalars. Similarly, if we write x=[x1,...,xD],
where the left hand side is a tall column vector, we mean to stack the xialong the rows; this is
usually written as x=(xT
1,...,xT
D)T, but that is rather ugly. If we write X=[x1,...,xD],
where the left hand side is a matrix, we mean to stack the xialong the columns, creating a
matrix.
4.1.2 Basics
Recall from Section 2.5.2 that the pdf for an MVN in Ddimensions is deﬁned by the following:
−1
(4.1)

98 Chapter 4. Gaussian models
μu1u2
λ11/2
λ21/2
x1x2
Figure 4.1 Visualization of a 2 dimensional Gaussian density. The major and minor axes of the ellipse
are deﬁned by the ﬁrst two eigenvectors of the covariance matrix, namely u1andu2. Based on Figure 2.7
of (Bishop 2006a).
The expression inside the exponent is the Mahalanobis distance between a data vector x
and the mean vector μ, We can gain a better understanding of this quantity by performing an
eigendecomposition ofΣ. That is, we write Σ=UΛUT,w h e r eU is an orthonormal matrix
of eigenvectors satsifying UTU=I, andΛis a diagonal matrix of eigenvalues.
Using the eigendecomposition, we have that
Σ−1=U−TΛ−1U−1=UΛ−1UT=D∑
i=11
λiuiuT
i (4.2)
whereuiis thei’th column of U, containing the i’th eigenvector. Hence we can rewrite the
Mahalanobis distance as follows:
(x−μ)TΣ−1(x−μ)=(x−μ)T(D∑
i=11
λiuiuTi)
(x−μ) (4.3)
=D∑
i=11
λi(x−μ)TuiuTi(x−μ)=D∑
i=1y2
i
λi(4.4)
whereyi≜uT
i(x−μ). Recall that the equation for an ellipse in 2d is
y2
1
λ1+y2
2
λ2=1 (4.5)

i=11
λiuiuT
i (4.2)
i=11
λiuiuTi)
=D∑
i=11
i=1y2
i
λi(4.4)
y2
1
λ1+y2
2
λ2=1 (4.5)
Hence we see that the contours of equal probability density of a Gaussian lie along ellipses.
This is illustrated in Figure 4.1. The eigenvectors determine the orientation of the ellipse, andthe eigenvalues determine how elogonated it is.
In general, we see that the Mahalanobis distance corresponds to Euclidean distance in a
transformed coordinate system, where we shift by μand rotate by U.

4.1. Introduction 99
4.1.3 MLE for an MVN
We now describe one way to estimate the parameters of an MVN, using MLE. In later sections,
we will discuss Bayesian inference for the parameters, which can mitigate overﬁtting, and canprovide a measure of conﬁdence in our estimates.
Theorem 4.1.1 (MLE for a Gaussian). If we have Niid samples x
i∼N(μ,Σ), then the MLE for
the parameters is given by
ˆμmle=1
NN∑
i=1xi≜x (4.6)
ˆΣmle=1
NN∑
i=1(xi−x)(xi−x)T=1
N(N∑
i=1xixT
i)−xxT(4.7)
That is, the MLE is just the empirical mean and empirical covariance. In the univariate case, we
get the following familiar results:
ˆμ=1
N∑
ixi=x (4.8)
ˆσ2=1
N∑
i(xi−x)2=(
1
N∑
ix2
i)
−(x)2(4.9)
4.1.3.1 Proof *
To prove this result, we will need several results from matrix algebra, which we summarize
below. In the equations, aandbare vectors, and AandBare matrices. Also, the notation
tr(A)refers to the traceof a matrix, which is the sum of its diagonals: tr(A)=∑
iAii.
∂(bTa)
∂a=b
∂(aTAa)
∂a=(A+AT)a
∂
∂Atr(BA)=BT
∂
∂Alog|A|=A−T≜(A−1)T

ˆμ=1
N∑
ˆσ2=1
N∑
1
N∑
ix2
i)
iAii.
∂(bTa)
∂a=b
∂(aTAa)
∂
∂
tr(ABC)=t r ( CAB)=t r (BCA)(4.10)
The last equation is called the cyclic permutation property of the trace operator. Using this,
we can derive the widely used trace trick, which reorders the scalar inner product xTAxas
follows
xTAx=t r (xTAx)=t r ( xxTA)=t r ( AxxT) (4.11)

100 Chapter 4. Gaussian models
Proof.We can now begin with the proof. The log-likelihood is
ℓ(μ,Σ)=l o g p(D|μ,Σ)=N
2log|Λ|−1
2N∑
i=1(xi−μ)TΛ(xi−μ) (4.12)
whereΛ=Σ−1is the precision matrix.
Using the substitution yi=xi−μand the chain rule of calculus, we have
∂
∂μ(xi−μ)TΣ−1(xi−μ)=∂
∂yiyT
iΣ−1yi∂yi
∂μ(4.13)
=−1(Σ−1+Σ−T)yi (4.14)
Hence
∂
∂μℓ(μ,Σ)=−1
2N∑
i=1−2Σ−1(xi−μ)=Σ−1N∑
i=1(xi−μ)=0 (4.15)
ˆμ=1
NN∑
i=1xi=x (4.16)
So the MLE of μis just the empirical mean.
Now we can use the trace-trick to rewrite the log-likelihood for Λas follows:
ℓ(Λ)=N
2log|Λ|−1
2∑
itr[(xi−μ)(xi−μ)TΛ] (4.17)
=N
2log|Λ|−1
2tr[SμΛ] (4.18)
(4.19)
where
Sμ≜N∑
i=1(xi−μ)(xi−μ)T(4.20)
is the scatter matrix centered on μ. Taking derivatives of this expression with respect to Λ
yields
∂ℓ(Λ)
∂Λ=N
2Λ−T−1
2ST
μ=0 (4.21)
Λ−T=Λ−1=Σ=1
NSμ (4.22)
so
ˆΣ=1
NN∑
i=1(xi−μ)(xi−μ)T(4.23)
which is just the empirical covariance matrix centered on μ. If we plug-in the MLE μ=x

ℓ(Λ)=N
2log|Λ|−1
2∑
=N
2log|Λ|−1
(4.19)
where
Sμ≜N∑
yields
∂ℓ(Λ)
∂Λ=N
2Λ−T−1
2ST
μ=0 (4.21)
NSμ (4.22)
so
ˆΣ=1
NN∑
(since both parameters must be simultaneously optimized), we get the standard equation for the
MLE of a covariance matrix.

4.2. Gaussian discriminant analysis 101
4.1.4 Maximum entropy derivation of the Gaussian *
In this section, we show that the multivariate Gaussian is the distribution with maximum entropy
subject to having a speciﬁed mean and covariance (see also Section 9.2.6). This is one reason theGaussian is so widely used: the ﬁrst two moments are usually all that we can reliably estimatefrom data, so we want a distribution that captures these properties, but otherwise makes as fewaddtional assumptions as possible.
To simplify notation, we will assume the mean is zero. The pdf has the form
p(x)=1
Zexp(−1
2xTΣ−1x) (4.24)
If we deﬁne fij(x)=xixjandλij=1
2(Σ−1)ij,f o ri,j∈{1,...,D}, we see that this is in
the same form as Equation 9.74. The (differential) entropy of this distribution (using log base e)
is given by
h(N(μ,Σ)) =1
2ln[
(2πe)D|Σ|]
(4.25)
We now show the MVN has maximum entropy amongst all distributions with a speciﬁed co-varianceΣ.
Theorem 4.1.2. Letq(x)be any density satisfying∫
q(x)x
ixj=Σij. Letp=N(0,Σ). Then

p(x)=1
Zexp(−1
is given by
2ln[
(4.25)
q(x)x
h(q)≤h(p).
Proof.(From (Cover and Thomas 1991, p234).) We have
0≤KL(q||p)=∫
q(x)logq(x)
p(x)dx (4.26)
=−h(q)−∫
q(x)logp(x)dx (4.27)
=∗−h(q)−∫
p(x)logp(x)dx (4.28)
=−h(q)+h(p) (4.29)
where the key step in Equation 4.28 (marked with a *) follows since qandpyield the same
moments for the quadratic form encoded by logp(x).
4.2 Gaussian discriminant analysis
One important application of MVNs is to deﬁne the the class conditional densities in a generativeclassiﬁer, i.e.,
p(x|y=c,θ)=N(x|μ
c,Σc) (4.30)

=−h(q)−∫
=∗−h(q)−∫
The resulting technique is called (Gaussian) discriminant analysis or GDA (even though it is a
generative, not discriminative, classiﬁer — see Section 8.6 for more on this distinction). If Σcis
diagonal, this is equivalent to naive Bayes.

102 Chapter 4. Gaussian models
heightweightred = female, blue=male
(b)
Figure 4.2 (a) Height/weight data. (b) Visualization of 2d Gaussians ﬁt to each class. 95% of the probability
mass is inside the ellipse. Figure generated by gaussHeightWeight .
We can classify a feature vector using the following decision rule, derived from Equation 2.13:
ˆy(x) = argmax
c[logp(y=c|π)+logp(x|θc)] (4.31)
When we compute the probability of xunder each class conditional density, we are measuring
the distance from xto the center of each class, μc, using Mahalanobis distance. This can be
thought of as a nearest centroids classiﬁer.
As an example, Figure 4.2 shows two Gaussian class-conditional densities in 2d, representing
the height and weight of men and women. We can see that the features are correlated, as is

to be expected (tall people tend to weigh more). The ellipses for each class contain 95% of theprobability mass. If we have a uniform prior over classes, we can classify a new test vector asfollows:
ˆy(x) = argmin
c(x−μc)TΣ−1
c(x−μc) (4.32)
4.2.1 Quadratic discriminant analysis (QDA)
The posterior over class labels is given by Equation 2.13. We can gain further insight into thismodel by plugging in the deﬁnition of the Gaussian density, as follows:
p(y=c|x,θ)=π
c|2πΣc|−1
2exp[
−1
2(x−μc)TΣ−1
c(x−μc)]
∑
c′πc′|2πΣc′|−1
2exp[
−1
2(x−μc′)TΣ−1
c′(x−μc′)] (4.33)

c|2πΣc|−1
2exp[
−1
c(x−μc)]
∑
2exp[
−1
Thresholding this results in a quadratic function of x. The result is known as quadratic
discriminant analysis (QDA). Figure 4.3 gives some examples of what the decision boundaries
look like in 2D.

4.2. Gaussian discriminant analysis 103
−2 0 2−202Parabolic Boundary
(a)−2 0 2 4 6−202468Some Linear, Some Quadratic
(b)
Figure 4.3 Quadratic decision boundaries in 2D for the 2 and 3 class case. Figure generated by
discrimAnalysisDboundariesDemo .
1 2 300.10.20.30.4T=100
1 2 300.51T=1
1 2 300.51T=0.1
1 2 300.51T=0.01
Figure 4.4 Softmax distribution S(η/T),w h e r eη=( 3,0,1), at different temperatures T. When the
temperature is high (left), the distribution is uniform, whereas when the temperature is low (right), the
distribution is “spiky”, with all its mass on the largest element. Figure generated by softmaxDemo2 .
4.2.2 Linear discriminant analysis (LDA)
We now consider a special case in which the covariance matrices are tiedorsharedacross
classes,Σc=Σ. In this case, we can simplify Equation 4.33 as follows:
p(y=c|x,θ)∝πcexp[
μT
cΣ−1x−1
2xTΣ−1x−1
2μTcΣ−1μc]
(4.34)
= exp[
μTcΣ−1x−1
2μTcΣ−1μc+logπc]
exp[−1
2xTΣ−1x](4.35)

μT
cΣ−1x−1
2xTΣ−1x−1
(4.34)
= exp[
μTcΣ−1x−1
exp[−1
Since the quadratic term xTΣ−1xis independent of c, it will cancel out in the numerator and
denominator. If we deﬁne
γc=−1
2μTcΣ−1μc+logπc (4.36)
βc=Σ−1μc (4.37)

104 Chapter 4. Gaussian models
then we can write
p(y=c|x,θ)=eβT
cx+γc
∑
c′eβT
c′x+γc′=S(η)c (4.38)
whereη=[βT
1x+γ1,...,βTCx+γC], andSis thesoftmax function, deﬁned as follows:
S(η)c=eηc
∑C
c′=1eηc′(4.39)
The softmax function is so-called since it acts a bit like the max function. To see this, let us
divide each ηcby a constant Tcalled the temperature. Then as T→0,w eﬁ n d
S(η/T)c={1.0ifc=a r g m a xc′ηc′
0.0otherwise(4.40)
In other words, at low temperatures, the distribution spends essentially all of its time in themost probable state, whereas at high temperatures, it visits all states uniformly. See Figure 4.4for an illustration. Note that this terminology comes from the area of statistical physics, whereit is common to use the Boltzmann distribution, which has the same form as the softmax
function.
An interesting property of Equation 4.38 is that, if we take logs, we end up with a linear
function of x. (The reason it is linear is because the x
TΣ−1xcancels from the numerator

function.
and denominator.) Thus the decision boundary between any two classes, say candc′, will be
a straight line. Hence this technique is called linear discriminant analysis orLDA.1We can
derive the form of this line as follows:
p(y=c|x,θ)=p( y=c′|x,θ) (4.41)
βT
cx+γc=βTc′x+γc′ (4.42)
xT(βc′−β)=γ c′−γc (4.43)
See Figure 4.5 for some examples.
An alternative to ﬁtting an LDA model and then deriving the class posterior is to directly
ﬁtp(y|x,W)=C a t ( y|Wx)for some C×Dweight matrix W. This is called multi-class
logistic regression ,o rmultinomial logistic regression.2We will discuss this model in detail
in Section 8.2. The difference between the two approaches is explained in Section 8.6.
4.2.3 Two-class LDA
To gain further insight into the meaning of these equations, let us consider the binary case. In

this case, the posterior is given by
p(y=1|x,θ)=eβT
1x+γ1
eβT
1x+γ1+eβT0x+γ0(4.44)
=1
1+e(β0−β1)Tx+(γ 0−γ1)=s i g m(
(β1−β0)Tx+(γ1−γ0))
(4.45)
1. The abbreviation “LDA”, could either stand for “linear discriminant analysis” or “latent Dirichlet allocation” (Sec-
tion 27.3). We hope the meaning is clear from text.
2. In the language modeling community, this model is called a maximum entropy model, for reasons explained in
Section 9.2.6.

4.2. Gaussian discriminant analysis 105
−2 0 2−202Linear Boundary
(a)−2 0 2 4 6−20246All Linear Boundaries 
(b)
Figure 4.5 Linear decision boundaries in 2D for the 2 and 3 class case. Figure generated by
Figure 4.6 Geometry of LDA in the 2 class case where Σ1=Σ2=I.
wheresigm(η)refers to the sigmoid function (Equation 1.10).
Now
γ1−γ0=−1
2μT
1Σ−1μ1+1
2μT
0Σ−1μ0+log(π1/π0) (4.46)
=−1
2(μ1−μ0)TΣ−1(μ1+μ0)+log(π1/π0) (4.47)
So if we deﬁne
w=β1−β0=Σ−1(μ1−μ0) (4.48)
x0=1
2(μ1+μ0)−(μ1−μ0)log(π1/π0)
(μ1−μ0)TΣ−1(μ1−μ0)(4.49)

106 Chapter 4. Gaussian models
then we have wTx0=−(γ1−γ0), and hence
p(y=1|x,θ) = sigm(wT(x−x0)) (4.50)
(This is closely related to logistic regression, which we will discuss in Section 8.2.) So the ﬁnal
decision rule is as follows: shift xbyx0, project onto the line w, and see if the result is positive
or negative.
IfΣ=σ2I, thenwis in the direction of μ1−μ0. So we classify the point based on whether
its projection is closer to μ0orμ1. This is illustrated in Figure 4.6. Furthemore, if π1=π0, then
x0=1
2(μ1+μ0), which is half way between the means. If we make π1>π0, thenx0gets
closer toμ0, so more of the line belongs to class 1 a priori. Conversely if π1<π0, the boundary
shifts right. Thus we see that the class prior, πc, just changes the decision threshold, and not
the overall geometry, as we claimed above. (A similar argument applies in the multi-class case.)
The magnitude of wdetermines the steepness of the logistic function, and depends on

x0=1
how well-separated the means are, relative to the variance. In psychology and signal detectiontheory, it is common to deﬁne the discriminability of a signal from the background noise using
a quantity called d-prime:
d
′≜μ1−μ0
σ(4.51)
whereμ1is the mean of the signal and μ0is the mean of the noise, and σis the standard
deviation of the noise. If d′is large, the signal will be easier to discriminate from the noise.
4.2.4 MLE for discriminant analysis

d
′≜μ1−μ0
σ(4.51)
We now discuss how to ﬁt a discriminant analysis model. The simplest way is to use maximumlikelihood. The log-likelihood function is as follows:
logp(D|θ)=[
N∑
i=1C∑
c=1I(yi=c)logπc]
+C∑
c=1⎡
⎣∑
i:yi=clogN(x|μc,Σc)⎤⎦ (4.52)
We see that this factorizes into a term for π, andCterms for each μ
candΣc. Hence we
can estimate these parameters separately. For the class prior, we have ˆπc=Nc
N, as with naive
Bayes. For the class-conditional densities, we just partition the data based on its class label, and
compute the MLE for each Gaussian:
ˆμc=1
Nc∑
i:yi=cxi,ˆΣc=1
Nc∑

N∑
i=1C∑
+C∑
c=1⎡
⎣∑
ˆμc=1
Nc∑
Nc∑
i:yi=c(xi−ˆμc)(xi−ˆμc)T(4.53)
SeediscrimAnalysisFit for a Matlab implementation. Once the model has been ﬁt, you can
make predictions using discrimAnalysisPredict , which uses a plug-in approximation.
4.2.5 Strategies for preventing overﬁtting
The speed and simplicity of the MLE method is one of its greatest appeals. However, the MLEcan badly overﬁt in high dimensions. In particular, the MLE for a full covariance matrix issingular if N
c<D. And even when Nc>D, the MLE can be ill-conditioned, meaning it is
close to singular. There are several possible solutions to this problem:

4.2. Gaussian discriminant analysis 107
• Use a diagonal covariance matrix for each class, which assumes the features are conditionally
independent; this is equivalent to using a naive Bayes classiﬁer (Section 3.5).
• Use a full covariance matrix, but force it to be the same for all classes, Σc=Σ. This is an
example of parameter tying orparameter sharing, and is equivalent to LDA (Section 4.2.2).
• Useadiagonalcovariancematrix andforcedittobeshared. Thisiscalleddiagonalcovariance
LDA, and is discussed in Section 4.2.7.
• Use a full covariance matrix, but impose a prior and then integrate it out. If we use a
conjugate prior, this can be done in closed form, using the results from Section 4.6.3; this
is analogous to the “Bayesian naive Bayes” method in Section 3.5.1.2. See (Minka 2000f) fordetails.
• Fit a full or diagonal covariance matrix by MAP estimation. We discuss two different kinds
of prior below.
• Project the data into a low dimensional subspace and ﬁt the Gaussians there. See Sec-

tion 8.6.3.3 for a way to ﬁnd the best (most discriminative) linear projection.
We discuss some of these options below.
4.2.6 Regularized LDA *
Suppose we tie the covariance matrices, so Σc=Σ, as in LDA, and furthermore we perform
MAP estimation of Σusing an inverse Wishart prior of the form IW(diag( ˆΣmle),ν0)(see
Section 4.5.1). Then we have
ˆΣ=λdiag(ˆΣmle)+(1−λ)ˆΣmle (4.54)
whereλcontrols the amount of regularization, which is related to the strength of the prior, ν0

(see Section 4.6.2.1 for details). This technique is known as regularized discriminant analysis
or RDA (Hastie et al. 2009, p656).
When we evaluate the class conditional densities, we need to compute ˆΣ−1, and hence ˆΣ−1
mle,
which is impossible to compute if D>N. However, we can use the SVD of X(Section 12.2.3)
to get around this, as we show below. (Note that this trick cannot be applied to QDA, which isa nonlinear function of x.)
LetX=UDV
Tbe the SVD of the design matrix, where VisD×N,Uis anN×N

mle,
LetX=UDV
orthogonal matrix, and Dis a diagonal matrix of size N. Furthermore, deﬁne the N×N
matrixZ=UD; this is like a design matrix in a lower dimensional space (since we assume
N<D). Also, deﬁne μz=VTμas the mean of the data in this reduced space; we can recover
the original mean using μ=Vμz, sinceVTV=VVT=I. With these deﬁnitions, we can

108 Chapter 4. Gaussian models
rewrite the MLE as follows:
ˆΣmle=1
NXTX−μμT(4.55)
=1
N(ZVT)T(ZVT)−(Vμz)(Vμz)T(4.56)
=1
NVZTZVT−VμzμT
zVT(4.57)
=V(1
NZTZ−μzμTz)VT(4.58)
=VˆΣzVT(4.59)
whereˆΣzis the empirical covariance of Z. Hence we can rewrite the MAP estimate as
ˆΣmap=V˜ΣzVT(4.60)
˜Σz=λdiag(ˆΣz)+(1−λ)ˆΣz (4.61)
Note, however, that we never need to actually compute the D×DmatrixˆΣmap. This is because
Equation 4.38 tells us that to classify using LDA, all we need to compute is p(y=c|x,θ)∝
exp(δc),w h e r e
δc=−xTβc+γc,βc=ˆΣ−1μc,γc−1
2μTcβc+logπc (4.62)
We can compute the crucial βcterm for RDA without inverting the D×Dmatrix as follows:
βc=ˆΣ−1
mapμc=(V˜ΣzVT)−1μc=V˜Σ−1
zVTμc=V˜Σ−1
zμz,c (4.63)
whereμz,c=VTμcis the mean of the Zmatrix for data belonging to class c. See rdaFitfor
the code.
4.2.7 Diagonal LDA
A simple alternative to RDA is to tie the covariance matrices, so Σc=Σas in LDA, and then to
use a diagonal covariance matrix for each class. This is called the diagonal LDA model, and is

βc=ˆΣ−1
the code.
equivalent to RDA with λ=1. The corresponding discriminant function is as follows (compare
to Equation 4.33):
δc(x)=l o gp(x,y=c|θ)=−D∑
j=1(xj−μcj)2
2σ2
j+logπc (4.64)
Typically we set ˆμcj=xcjandˆσ2
j=s2
j, which is the pooled empirical variance of feature j
(pooled across classes) deﬁned by
s2
j=∑C
c=1∑
i:yi=c(xij−xcj)2
N−C(4.65)
In high dimensional settings, this model can work much better than LDA and RDA (Bickel and
Levina 2004).

4.2. Gaussian discriminant analysis 109
Number of Genes
2308 1355 352 106 36 12 5 0
1 2 3 4 5 6 7 800.10.20.30.40.50.60.70.80.91
λMisclassification Error
  
Test
Train
CV
Figure 4.7 Error versus amount of shrinkage for nearest shrunken centroid classiﬁer applied to the
SRBCT gene expression data. Based on Figure 18.4 of (Hastie et al. 2009). Figure generated by
shrunkenCentroidsSRBCTdemo .
4.2.8 Nearest shrunken centroids classiﬁer *
One drawback of diagonal LDA is that it depends on all of the features. In high dimensional
problems, we might prefer a method that only depends on a subset of the features, for reasonsof accuracy and interpretability. One approach is to use a screening method, perhaps basedon mutual information, as in Section 3.5.4. We now discuss another approach to this problemknown as the nearest shrunken centroids classiﬁer (Hastie et al. 2009, p652).
The basic idea is to perform MAP estimation for diagonal LDA with a sparsity-promoting

(Laplace) prior (see Section 13.3). More precisely, deﬁne the class-speciﬁc feature mean, μ
cj,i n
terms of the class-independent feature mean, mj, and a class-speciﬁc offset, Δcj. Thus we have
μcj=mj+Δcj (4.66)
We will then put a prior on the Δcjterms to encourage them to be strictly zero and compute
a MAP estimate. If, for feature j, we ﬁnd that Δcj=0for allc, then feature jwill play no role
in the classiﬁcation decision (since μcjwill be independent of c). Thus features that are not

cj,i n
discriminative are automatically ignored. The details can be found in (Hastie et al. 2009, p652)and (Greenshtein and Park 2009). See shrunkenCentroidsFit for some code.
Let us give an example of the method in action, based on (Hastie et al. 2009, p652). Consider

the problem of classifying a gene expression dataset, which 2308 genes, 4 classes, 63 trainingsamples and 20 test samples. Using a diagonal LDA classiﬁer produces 5 errors on the test set.Using the nearest shrunken centroids classiﬁer produced 0 errors on the test set, for a range ofλvalues: see Figure 4.7. More importantly, the model is sparse and hence more interpretable:
Figure 4.8 plots an unpenalized estimate of the difference, d
cj, in gray, as well as the shrunken
estimates Δcjin blue. (These estimates are computed using the value of λestimated by CV.)

We see that only 39 genes are used, out of the original 2308.
Now consider an even harder problem, with 16,603 genes, a training set of 144 patients, a
test set of 54 patients, and 14 different types of cancer (Ramaswamy et al. 2001). Hastie et al.(Hastie et al. 2009, p656) report that nearest shrunken centroids produced 17 errors on the test

110 Chapter 4. Gaussian models
0 500 1000 1500 2000 2500−6−4−20246Class 1
(a)0 500 1000 1500 2000 2500−8−6−4−202468Class 2
(b)
0 500 1000 1500 2000 2500−6−4−20246Class 3
(c)0 500 1000 1500 2000 2500−4−202468Class 4
(d)
Figure 4.8 Proﬁle of the shrunken centroids corresponding to λ=4.4(CV optimal in Fig-
ure 4.7). This selects 39 genes. Based on Figure 18.4 of (Hastie et al. 2009). Figure generated by
set, using 6,520 genes, and that RDA (Section 4.2.6) produced 12 errors on the test set, using
all 16,603 genes. The PMTK function cancerHighDimClassifDemo can be used to reproduce
these numbers.
4.3 Inference in jointly Gaussian distributions
Given a joint distribution, p(x1,x2), it is useful to be able to compute marginals p(x1)and
conditionals p(x1|x2). We discuss how to do this below, and then give some applications. These
operations take O(D3)time in the worst case. See Section 20.4.3 for faster methods.

4.3. Inference in jointly Gaussian distributions 111
4.3.1 Statement of the result
Theorem 4.3.1 (Marginals and conditionals of an MVN). Supposex=(x1,x2)is jointly Gaussian
with parameters
μ=(
μ1
μ2)
,Σ=(
Σ11Σ12
Σ21Σ22)
,Λ=Σ−1=(
Λ11Λ12
Λ21Λ22)
(4.67)
Then the marginals are given by
p(x1)=N (x1|μ1,Σ11)
p(x2)=N (x2|μ2,Σ22) (4.68)
and the posterior conditional is given by
p(x1|x2)=N(x1|μ1|2,Σ1|2)
μ1|2=μ1+Σ12Σ−1
22(x2−μ2)
=μ1−Λ−1
11Λ12(x2−μ2)
=Σ1|2(Λ11μ1−Λ12(x2−μ2))
Σ1|2=Σ11−Σ12Σ−1
22Σ21=Λ−1
11(4.69)
Equation 4.69 is of such crucial importance in this book that we have put a box around it, so
you can easily ﬁnd it. For the proof, see Section 4.3.4.
We see that both the marginal and conditional distributions are themselves Gaussian. For the
marginals, we just extract the rows and columns corresponding to x1orx2. For the conditional,
we have to do a bit more work. However, it is not that complicated: the conditional mean is

=μ1−Λ−1
22Σ21=Λ−1
11(4.69)
just a linear function of x2, and the conditional covariance is just a constant matrix that is
independent of x2. We give three different (but equivalent) expressions for the posterior mean,
and two different (but equivalent) expressions for the posterior covariance; each one is useful indifferent circumstances.
4.3.2 Examples
Below we give some examples of these equations in action, which will make them seem moreintuitive.
4.3.2.1 Marginals and conditionals of a 2d Gaussian

Let us consider a 2d example. The covariance matrix is
Σ=(σ
2
1ρσ1σ2
ρσ1σ2σ2
2)
(4.70)
The marginal p(x1)is a 1D Gaussian, obtained by projecting the joint distribution onto the x1
line:
p(x1)=N(x1|μ1,σ2
1) (4.71)

112 Chapter 4. Gaussian models
−5 −4 −3 −2 −1 0 1 2 3 4 5−3−2−10123
x1x2p(x1,x2)
(a)−5 −4 −3 −2 −1 0 1 2 3 4 500.010.020.030.040.050.060.070.08
x1x2p(x1)
(b)−5 −4 −3 −2 −1 0 1 2 3 4 501234567
x1x2p(x1|x2=1)
(c)
Figure 4.9 (a) A joint Gaussian distribution p(x1,x2)with a correlation coefficient of 0.8. We plot the
95% contour and the principal axes. (b) The unconditional marginal p(x1). (c) The conditional p(x1|x2)=
N(x1|0.8,0.36), obtained by slicing (a) at height x2=1. Figure generated by gaussCondition2Ddemo2 .
Suppose we observe X2=x2; the conditional p(x1|x2)is obtained by “slicing” the joint
distribution through the X2=x2line (see Figure 4.9):
p(x1|x2)=N(
x1|μ1+ρσ1σ2
σ2
2(x2−μ2),σ2
1−(ρσ1σ2)2
σ2
2)
(4.72)
Ifσ1=σ2=σ,w eg e t
x1|μ1+ρ(x2−μ2),σ2(1−ρ2))
(4.73)
In Figure 4.9 we show an example where ρ=0.8,σ1=σ2=1,μ=0andx2=1.W e
see that E[x1|x2=1 ]=0 .8, which makes sense, since ρ=0.8means that we believe that if

σ2
σ2
2)
(4.72)
(4.73)
x2increases by 1 (beyond its mean), then x1increases by 0.8. We also see var[x1|x2=1 ]=
1−0.82=0.36. This also makes sense: our uncertainty about x1has gone down, since we
have learned something about x1(indirectly) by observing x2.I fρ=0,w eg e tp(x1|x2)=
N(
x1|μ1,σ2
1)
, sincex2conveys no information about x1if they are uncorrelated (and hence
independent).
4.3.2.2 Interpolating noise-free data
Suppose we want to estimate a 1d function, deﬁned on the interval [0,T], such that yi=f(ti)
forNobserved points ti. We assume for now that the data is noise-free, so we want to

N(
x1|μ1,σ2
1)
interpolate it, that is, ﬁt a function that goes exactly through the data. (See Section 4.4.2.3 for
the noisy data case.) The question is: how does the function behave in between the observed
data points? It is often reasonable to assume that the unknown function is smooth. In Chapter 15,we shall see how to encode priors over functions, and how to update such a prior with observed

values to get a posterior over functions. But in this section, we take a simpler approach, whichis adequate for MAP estimation of functions deﬁned on 1d inputs. We follow the presentationof (Calvetti and Somersalo 2007, p135).
We start by discretizing the problem. First we divide the support of the function into Dequal
subintervals. We then deﬁne
x
j=f(sj),sj=jh, h=T
D,1≤j≤D (4.74)

4.3. Inference in jointly Gaussian distributions 113
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1−5−4−3−2−1012345λ=30
(a)0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1−5−4−3−2−1012345λ=0p1
(b)
Figure 4.10 Interpolating noise-free data using a Gaussian with prior precision λ. (a)λ=3 0. (b)
λ=0.01. See also Figure 4.15. Based on Figure 7.1 of (Calvetti and Somersalo 2007). Figure generated by
gaussInterpDemo .
We can encode our smoothness prior by assuming that xjis an average of its neighbors, xj−1
andxj+1, plus some Gaussian noise:
xj=1
2(xj−1+xj+1)+ϵj,2≤j≤D−2 (4.75)
whereϵ∼N(0,(1/λ)I). The precision term λcontrols how much we think the function will
vary: a large λcorresponds to a belief that the function is very smooth, a small λcorresponds
to a belief that the function is quite “wiggly”. In vector form, the above equation can be written
Lx=ϵ (4.76)
whereLis the(D−2)×Dsecond order ﬁnite difference matrix
L=1
2⎛
⎜⎜⎜⎝−12− 1
−12− 1
...
−12− 1⎞
⎟⎟⎟⎠(4.77)
The corresponding prior has the form
p(x)=N(x|0,(λ

xj=1
L=1
2⎛
⎜⎜⎜⎝−12− 1
−12− 1
...
−12− 1⎞
2LTL)−1)∝exp(
−λ2
2||Lx||2
2)
(4.78)
We will henceforth assume we have scaled Lbyλso we can ignore the λterm, and just write
Λ=LTLfor the precision matrix.
Note that although xisD-dimensional, the precision matrix Λonly has rank D−2. Thus
this is an improper prior, known as an intrinsic Gaussian random ﬁeld (see Section 19.4.4 for

114 Chapter 4. Gaussian models
more information). However, providing we observe N≥2data points, the posterior will be
proper.
Now letx2be theNnoise-free observations of the function, and x1be theD−Nunknown
function values. Without loss of generality, assume that the unknown variables are ordered ﬁrst,
then the known variables. Then we can partition the Lmatrix as follows:
L=[L1,L2],L1∈R(D−2)×(D−N),L2∈R(D−2)×(N)(4.79)
We can also partition the precision matrix of the joint distribution:
Λ=LTL=(Λ11Λ12
Λ21Λ22)
=(LT
1L1LT1L2
LT2L1LT2L2)
(4.80)
Using Equation 4.69, we can write the conditional distribution as follows:
p(x1|x2)=N (μ1|2,Σ1|2) (4.81)
μ1|2=−Λ−1
11Λ12x2=−LT
1L2x2 (4.82)
Σ1|2=Λ−1
11 (4.83)
Note that we can compute the mean by solving the following system of linear equations:
L1μ1|2=−L2x2 (4.84)
This is efficient since L1is tridiagonal. Figure 4.10 gives an illustration of these equations. We
see that the posterior mean μ1|2equals the observed data at the speciﬁed points, and smoothly

Λ21Λ22)
=(LT
1L1LT1L2
(4.80)
μ1|2=−Λ−1
Σ1|2=Λ−1
11 (4.83)
interpolates in between, as desired.
It is also interesting to plot the 95% pointwise marginal credibility intervals, μj±
2√Σ1|2,jj, shown in grey. We see that the variance goes up as we move away from the
data. We also see that the variance goes up as we decrease the precision of the prior, λ. In-
terestingly, λhas no effect on the posterior mean, since it cancels out when multiplying Λ11
andΛ12. By contrast, when we consider noisy data in Section 4.4.2.3, we will see that the prior

precision affects the smoothness of posterior mean estimate.
The marginal credibility intervals do not capture the fact that neighboring locations are
correlated. We can represent that by drawing complete functions (i.e., vectors x)f r o mt h e
posterior, and plotting them. These are shown by the thin lines in Figure 4.10. These are not
quite as smooth as the posterior mean itself. This is because the prior only penalizes ﬁrst-orderdifferences. See Section 4.4.2.3 for further discussion of this point.
4.3.2.3 Data imputation

Suppose we are missing some entries in a design matrix. If the columns are correlated, we canuse the observed entries to predict the missing entries. Figure 4.11 shows a simple example. Wesampled some data from a 20 dimensional Gaussian, and then deliberately “hid” 50% of the datain each row. We then inferred the missing entries given the observed entries, using the true(generating) model. More precisely, for each row i, we compute p(x
hi|xvi,θ),w h e r ehiandvi

are the indices of the hidden and visible entries in case i. From this, we compute the marginal
distribution of each missing variable, p(xhij|xvi,θ). We then plot the mean of this distribution,
ˆxij=E[xj|xvi,θ]; this represents our “best guess” about the true value of that entry, in the

4.3. Inference in jointly Gaussian distributions 115
0 5 10 15 20−10−50510observed
0 5 10 15 20−10−50510imputed
0 5 10 15 20−10−50510truth
Figure 4.11 Illustration of data imputation. Left column: visualization of three rows of the data matrix
with missing entries. Middle column: mean of the posterior predictive, based on partially observed
data in that row, but the true model parameters. Right column: true values. Figure generated bygaussImputationDemo .
sense that it minimizes our expected squared error (see Section 5.7 for details). Figure 4.11 shows
that the estimates are quite close to the truth. (Of course, if j∈vi, the expected value is equal
to the observed value, ˆxij=xij.)
We can use var[
xhij|xvi,θ]
as a measure of conﬁdence in this guess, although this is not
shown. Alternatively, we could draw multiple samples from p(xhi|xvi,θ); this is called multiple
imputation.
In addition to imputing the missing entries, we may be interested in computing the like-

lihood of each partially observed row in the table, p(xvi|θ), which can be computed using
Equation 4.68. This is useful for detecting outliers (atypical observations).
4.3.3 Information form
Supposex∼N(μ,Σ). One can show that E[x]=μis the mean vector, and cov[x]=Σis
the covariance matrix. These are called the moment parameters of the distribution. However,
it is sometimes useful to use the canonical parameters ornatural parameters, deﬁned as
Λ≜Σ−1,ξ≜Σ−1μ (4.85)
We can convert back to the moment parameters using
μ=Λ−1ξ,Σ=Λ−1(4.86)

Using the canonical parameters, we can write the MVN in information form (i.e., in exponential
family form, deﬁned in Section 9.2):
Nc(x|ξ,Λ)=( 2 π)−D/2|Λ|1
2exp[
−1
2(xTΛx+ξTΛ−1ξ−2xTξ)]
(4.87)
where we use the notation Nc()to distinguish from the moment parameterization N().
It is also possible to derive the marginalization and conditioning formulas in information
form. We ﬁnd
p(x2)=N c(x2|ξ2−Λ21Λ−1
11ξ1,Λ22−Λ21Λ−1
11Λ12) (4.88)
p(x1|x2)=N c(x1|ξ1−Λ12x2,Λ11) (4.89)

116 Chapter 4. Gaussian models
Thus we see that marginalization is easier in moment form, and conditioning is easier in
information form.
Another operation that is signiﬁcantly easier in information form is multiplying two Gaussians.
One can show that
Nc(ξf,λf)Nc(ξg,λg)=N c(ξf+ξg,λf+λg) (4.90)
However, in moment form, things are much messier:
N(μf,σ2
f)N(μg,σ2
g)=N(
μfσ2
g+μgσ2
f
σ2g+σ2g,σ2
fσ2
g
σ2g+σ2g)
(4.91)
4.3.4 Proof of the result *
We now prove Theorem 4.3.1. Readers who are intimidated by heavy matrix algebra can safelyskip this section. We ﬁrst derive some results that we will need here and elsewhere in the book.We will return to the proof at the end.
4.3.4.1 Inverse of a partitioned matrix using Schur complements
The key tool we need is a way to invert a partitioned matrix. This can be done using thefollowing result.
Theorem 4.3.2 (Inverse of a partitioned matrix). Consider a general partitioned matrix
M=(EF
GH)
(4.92)
where we assume EandHare invertible. We have
M
−1=((M/H)−1−(M/H)−1FH−1

M=(EF
GH)
(4.92)
M
−H−1G(M/H)−1H−1+H−1G(M/H)−1FH−1)
(4.93)
=(E−1+E−1F(M/E)−1GE−1−E−1F(M/E)−1
−(M/E)−1GE−1(M/E)−1)
(4.94)
where
M/H≜E−FH−1G (4.95)
M/E≜H−GE−1F (4.96)
We say that M/His theSchur complement ofMwrtH. Equation 4.93 is called the partitioned
inverse formula.Proof.If we could block diagonalize M, it would be easier to invert. To zero out the top right
block ofMwe can pre-multiply as follows
(I−FH
−1
0I)(EF
GH)
=(E−FH−1G0
GH)
(4.97)

4.3. Inference in jointly Gaussian distributions 117
Similarly, to zero out the bottom left we can post-multiply as follows
(
E−FH−1G0
GH)(
I0
−H−1GI)
=(
E−FH−1G0
0H)
(4.98)
Putting it all together we get
(
I−FH−1
0I)

X(
EF
GH)

M(
I0
−H−1GI)

Z=(
E−FH−1G0
0H)

W(4.99)
Taking the inverse of both sides yields
Z−1M−1X−1=W−1(4.100)
and hence
M−1=ZW−1X (4.101)
Substituting in the deﬁnitions we get
(EF
GH)−1
=(I0
−H−1GI)((M/H)−10
0H−1)(I−FH−1
0I)
(4.102)
=((M/H)−10
−H−1G(M/H)−1H−1)(I−FH−1
0I)
(4.103)
=((M/H)−1−(M/H)−1FH−1
(4.104)
Alternatively, we could have decomposed the matrix Min terms of EandM/E=(H−
GE−1F), yielding
(EF
GH)−1
(4.105)
4.3.4.2 The matrix inversion lemma
We now derive some useful corollaries of the above result.
Corollary 4.3.1 (Matrix inversion lemma). Consider a general partitioned matrix M=(EF
GH)
,
(E−FH−1G)−1=E−1+E−1F(H−GE−1F)−1GE−1(4.106)

(4.103)
(4.104)
(EF
GH)−1
(4.105)
GH)
,
(E−FH−1G)−1FH−1=E−1F(H−GE−1F)−1(4.107)
|E−FH−1G|=|H−GE−1F||H−1||E| (4.108)

118 Chapter 4. Gaussian models
The ﬁrst two equations are s known as the matrix inversion lemma or theSherman-
Morrison-Woodbury formula. The third equation is known as the matrix determinant
lemma. A typical application in machine learning/ statistics is the following. Let E=Σ
be aN×Ndiagonal matrix, let F=GT=Xof sizeN×D,w h e r eN≫D, and let
H−1=−I. Then we have
(Σ+XXT)−1=Σ−1−Σ−1X(I+XTΣ−1X)−1XTΣ−1(4.109)
The LHS takes O(N3)time to compute, the RHS takes time O(D3)to compute.
Another application concerns computing a rank one update of an inverse matrix. Let
H=−1(a scalar), F=u(a column vector), and G=vT(a row vector). Then we have
(E+uvT)−1=E−1+E−1u(−1−vTE−1u)−1vTE−1(4.110)
=E−1−E−1uvTE−1
1+vTE−1u(4.111)
This is useful when we incrementally add a data vector to a design matrix, and want to update
our sufficient statistics. (One can derive an analogous formula for removing a data vector.)
Proof.To prove Equation 4.106, we simply equate the top left block of Equation 4.93 and Equa-

tion 4.94. To prove Equation 4.107, we simple equate the top right blocks of Equations 4.93 and
4.94. The proof of Equation 4.108 is left as an exercise.
4.3.4.3 Proof of Gaussian conditioning formulas
We can now return to our original goal, which is to derive Equation 4.69. Let us factor the jointp(x
1,x2)asp(x2)p(x1|x2)as follows:
E=e x p{
−1
2(x1−μ1
x2−μ2)T(Σ11Σ12
Σ21Σ22)−1(x1−μ1
x2−μ2)}
(4.112)
Using Equation 4.102 the above exponent becomes
E=e x p{
−1
2(x1−μ1
x2−μ2)T(I0
−Σ−1
22Σ21I)((Σ/Σ22)−10
0Σ−1
22)
(4.113)
×(
I−Σ12Σ−1
22
0I)(x1−μ1
x2−μ2)}
(4.114)
=e x p{
−1
2(x1−μ1−Σ12Σ−1

E=e x p{
−1
2(x1−μ1
x2−μ2)}
(4.112)
E=e x p{
−1
2(x1−μ1
−Σ−1
0Σ−1
22)
(4.113)
×(
I−Σ12Σ−1
22
0I)(x1−μ1
x2−μ2)}
(4.114)
=e x p{
−1
22(x2−μ2))T(Σ/Σ22)−1(4.115)
(x1−μ1−Σ12Σ−1
22(x2−μ2))}
×exp{
−1
2(x2−μ2)TΣ−1
22(x2−μ2)}
(4.116)
This is of the form
exp(quadratic form in x1,x2)×exp(quadratic form in x2) (4.117)

4.4. Linear Gaussian systems 119
Hence we have successfully factorized the joint as
p(x1,x2)=p( x1|x2)p(x2) (4.118)
=N(x1|μ1|2,Σ1|2)N(x2|μ2,Σ22) (4.119)
where the parameters of the conditional distribution can be read off from the above equations
using
22(x2−μ2) (4.120)
Σ1|2=Σ/Σ22=Σ11−Σ12Σ−1
22Σ21 (4.121)
We can also use the fact that |M|=|M/H||H|to check the normalization constants are
correct:
(2π)(d1+d2)/2|Σ|1
2=( 2π)(d1+d2)/2(|Σ/Σ22||Σ22|)1
2 (4.122)
=( 2π)d1/2|Σ/Σ22|1
2(2π)d2/2|Σ22|1
2 (4.123)
whered1=d i m (x1)andd2=d i m (x2).
We leave the proof of the other forms of the result in Equation 4.69 as an exercise.
4.4 Linear Gaussian systems
Suppose we have two variables, xandy.L e tx∈RDxbe a hidden variable, and y∈RDybe
a noisy observation of x. Let us assume we have the following prior and likelihood:
p(x)=N(x|μx,Σx)
p(y|x)=N(y|Ax+b,Σy)(4.124)
whereAis a matrix of size Dy×Dx. This is an example of a linear Gaussian system.W e

2 (4.122)
2 (4.123)
can represent this schematically as x→y, meaning xgenerates y. In this section, we show
how to “invert the arrow”, that is, how to infer xfromy. We state the result below, then give
several examples, and ﬁnally we derive the result. We will see many more applications of theseresults in later chapters.
4.4.1 Statement of the result
Theorem 4.4.1 (Bayes rule for linear Gaussian systems). Given a linear Gaussian system, as in
Equation 4.124, the posterior p(x|y)is given by the following:
p(x|y)=N(x|μx|y,Σx|y)
Σ−1
x|y=Σ−1

Σ−1
x|y=Σ−1
x+ATΣ−1
yA
μx|y=Σx|y[ATΣ−1
y(y−b)+Σ−1
xμx](4.125)

120 Chapter 4. Gaussian models
In addition, the normalization constant p(y)is given by
p(y)=N(y|Aμx+b,Σy+AΣxAT) (4.126)
For the proof, see Section 4.4.3.
4.4.2 Examples
In this section, we give some example applications of the above result.
4.4.2.1 Inferring an unknown scalar from noisy measurements
Suppose we make Nnoisy measurements yiof some underlying quantity x; let us assume the
measurement noise has ﬁxed precision λy=1/σ2, so the likelihood is
p(yi|x)=N (yi|x,λ−1
y) (4.127)
Now let us use a Gaussian prior for the value of the unknown source:
p(x)=N (x|μ0,λ−1
0) (4.128)
We want to compute p(x|y1,...,y N,σ2). We can convert this to a form that lets us apply
Bayes rule for Gaussians by deﬁning y=(y1,...,y N),A=1T
N(an1×Nrow vector of 1’s),
andΣ−1
y=d i a g (λyI). Then we get
p(x|y)=N (x|μN,λ−1
N) (4.129)
λN=λ0+Nλy (4.130)
μN=Nλyy+λ0μ0
λN=Nλy
Nλy+λ0y+λ0
Nλy+λ0μ0 (4.131)
These equations are quite intuitive: the posterior precision λNis the prior precision λ0plusN

y) (4.127)
0) (4.128)
andΣ−1
N) (4.129)
λN=Nλy
units of measurement precision λy. Also, the posterior mean μNis a convex combination of
the MLEyand the prior mean μ0. This makes it clear that the posterior mean is a compromise
between the MLE and the prior. If the prior is weak relative to the signal strength (λ 0is
small relative to λy), we put more weight on the MLE. If the prior is strong relative to the
signal strength (λ 0is large relative to λy), we put more weight on the prior. This is illustrated

in Figure 4.12, which is very similar to the analogous results for the beta-binomial model in
Figure 3.6.
Note that the posterior mean is written in terms of Nλyy, so having Nmeasurements each
of precision λyis like having one measurement with value yand precision Nλy.
We can rewrite the results in terms of the posterior variance, rather than posterior precision,

4.4. Linear Gaussian systems 121
−5 0 500.10.20.30.40.50.6prior variance = 1.00
  
prior
lik
post
−5 0 500.10.20.30.40.50.6prior variance = 5.00
  
prior
lik
post
Figure 4.12 Inference about xgiven a noisy observation y=3. (a) Strong prior N(0,1). The posterior
mean is “shrunk” towards the prior mean, which is 0. (a) Weak prior N(0,5). The posterior mean is
similar to the MLE. Figure generated by gaussInferParamsMean1d .
p(x|D,σ2)=N (x|μN,τ2
N) (4.132)
τ2
N=1
N
σ2+1
τ2
0=σ2τ2
0
Nτ2
0+σ2(4.133)
μN=τ2
N(μ0
τ2
0+Ny
σ2)
=σ2
Nτ2
0+σ2μ0+Nτ2
0
Nτ2
0+σ2y (4.134)
whereτ2
0=1/λ0is the prior variance and τ2
N=1/λNis the posterior variance.
We can also compute the posterior sequentially, by updating after each observation. If
N=1, we can rewrite the posterior after seeing a single observation as follows (where we
deﬁneΣy=σ2,Σ0=τ2
0andΣ1=τ2
1to be the variances of the likelihood, prior and
posterior):
p(x|y)=N (x|μ1,Σ1) (4.135)
Σ1=(1
Σ0+1
Σy)−1
=ΣyΣ0
Σ0+Σy(4.136)
μ1=Σ1(μ0
Σ0+y
Σy)
(4.137)

0
Nτ2
μN=τ2
N(μ0
τ2
0+Ny
σ2)
=σ2
Nτ2
0
Nτ2
whereτ2
0andΣ1=τ2
Σ1=(1
Σ0+1
Σy)−1
=ΣyΣ0
μ1=Σ1(μ0
Σ0+y
Σy)
(4.137)
We can rewrite the posterior mean in 3 different ways:
μ1=Σy
Σy+Σ0μ0+Σ0
Σy+Σ0y (4.138)
=μ0+(y−μ0)Σ0
Σy+Σ0(4.139)
=y−(y−μ0)Σy
Σy+Σ0(4.140)

122 Chapter 4. Gaussian models
The ﬁrst equation is a convex combination of the prior and the data. The second equation is the
prior mean adjusted towards the data. The third equation is the data adjusted towards the priormean; this is called shrinkage. These are all equivalent ways of expressing the tradeoff between
likelihood and prior. If Σ
0is small relative to ΣY, corresponding to a strong prior, the amount
of shrinkage is large (see Figure 4.12(a)), whereas if Σ0is large relative to Σy, corresponding to
a weak prior, the amount of shrinkage is small (see Figure 4.12(b)).
Another way to quantify the amount of shrinkage is in terms of the signal-to-noise ratio,
which is deﬁned as follows:
SNR≜E[
X2]
E[ϵ2]=Σ0+μ2
0
Σy(4.141)
wherex∼N(μ0,Σ0)is the true signal, y=x+ϵis the observed signal, and ϵ∼N(0,Σy)
is the noise term.
4.4.2.2 Inferring an unknown vector from noisy measurements
Now consider Nvector-valued observations, yi∼N(x,Σy), and a Gaussian prior, x∼

SNR≜E[
X2]
0
Σy(4.141)
N(μ0,Σ0). Setting A=I,b=0, and using yfor the effective observation with precision
NΣ−1
y,w eh a v e
p(x|y1,...,yN)=N (x|μN,ΣN) (4.142)
Σ−1
N=Σ−1
0+NΣ−1
y (4.143)
μN=ΣN(Σ−1
y(Ny)+Σ−1
0μ0) (4.144)
See Figure 4.13 for a 2d example. We can think of xas representing the true, but unknown,
location of an object in 2d space, such as a missile or airplane, and the yias being noisy
observations, such as radar “blips”. As we receive more blips, we are better able to localize the

NΣ−1
y,w eh a v e
Σ−1
N=Σ−1
0+NΣ−1
y (4.143)
μN=ΣN(Σ−1
y(Ny)+Σ−1
source. In Section 18.3.1, we will see how to extend this example to track moving objects usingthe famous Kalman ﬁlter algorithm.
Now suppose we have multiple measuring devices, and we want to combine them together;
this is known as sensor fusion. If we have multiple observations with different covariances (cor-
responding to sensors with different reliabilities), the posterior will be an appropriate weightedaverage of the data. Consider the example in Figure 4.14. We use an uninformative prior on x,
namelyp(x)=N(μ

0,Σ0)=N(0,1010I2). We get 2 noisy observations, y1∼N(x,Σy,1)
andy2∼N(x,Σy,2). We then compute p(x|y1,y2).
In Figure 4.14(a), we set Σy,1=Σy,2=0.01I2, so both sensors are equally reliable. In this
case, the posterior mean is half way between the two observations, y1andy2. In Figure 4.14(b),
we setΣy,1=0.05I2andΣy,2=0.01I2, so sensor 2 is more reliable than sensor 1. In this
case, the posterior mean is closer to y2. In Figure 4.14(c), we set
Σy,1=0.01(10 1
11)
,Σy,2=0.01(1111 0)
(4.145)
so sensor 1 is more reliable in the y
2component (vertical direction), and sensor 2 is more

11)
(4.145)
reliable in the y1component (horizontal direction). In this case, the posterior mean uses y1’s
vertical component and y2’s horizontal component.

4.4. Linear Gaussian systems 123
−1 0 1−1−0.500.51data prior
−1 0 1−1−0.500.51post after 10 obs
−1 0 1−1−0.500.51
Figure 4.13 Illustration of Bayesian inference for the mean of a 2d Gaussian. (a) The data is generated
fromyi∼N(x,Σy),w h e r ex=[ 0.5,0.5]TandΣy=0.1[2,1;1,1]). We assume the sensor noise
covariance Σyis known but xis unknown. The black cross represents x. (b) The prior is p(x)=
N(x|0,0.1I2). (c) We show the posterior after 10 data points have been observed. Figure generated by
gaussInferParamsMean2d .
−0.4 −0.2 0 0.2 0.4 0.6 0.8 1 1.2 1.4−1.4−1.2−1−0.8−0.6−0.4−0.200.20.4
(a)−0.6−0.4 −0.2 0 0.2 0.4 0.6 0.8 1 1.2 1.4−1.6−1.4−1.2−1−0.8−0.6−0.4−0.200.20.4
(b)−1 −0.5 0 0.5 1 1.5−1.5−1−0.500.51
(c)
Figure 4.14 We observe y1=( 0,−1)(red cross) and y2=( 1,0)(green cross) and infer E(μ|y1,y2,θ)
(black cross). (a) Equally reliable sensors, so the posterior mean estimate is in between the two circles.

(c)
(b) Sensor 2 is more reliable, so the estimate shifts more towards the green circle. (c) Sensor 1 is morereliable in the vertical direction, Sensor 2 is more reliable in the horizontal direction. The estimate is anappropriate combination of the two measurements. Figure generated by sensorFusion2d .
Note that this technique crucially relies on modeling our uncertainty of each sensor; comput-
ing an unweighted average would give the wrong result. However, we have assumed the sensor
precisions are known. When they are not, we should model out uncertainty about Σ1andΣ2

as well. See Section 4.6.4 for details.
4.4.2.3 Interpolating noisy data
We now revisit the example of Section 4.3.2.2. This time we no longer assume noise-freeobservations. Instead, let us assume that we obtain Nnoisy observations y
i; without loss
of generality, assume these correspond to x1,...,x N. We can model this setup as a linear

124 Chapter 4. Gaussian models
Gaussian system:
y=Ax+ϵ (4.146)
whereϵ∼N(0,Σy),Σy=σ2I,σ2is the observation noise, and Ais aN×Dprojection
matrix that selects out the observed elements. For example, if N=2andD=4we have
A=(
1000
0100)
(4.147)
Using the same improper prior as before, Σx=(LTL)−1, we can easily compute the posterior
mean and variance. In Figure 4.15, we plot the posterior mean, posterior variance, and someposterior samples. Now we see that the prior precision λeffects the posterior mean as well as
the posterior variance. In particular, for a strong prior (large λ), the estimate is very smooth, and
the uncertainty is low. but for a weak prior (small λ), the estimate is wiggly, and the uncertainty
(away from the data) is high.
The posterior mean can also be computed by solving the following optimization problem:
min
x1
2σ2N∑
i=1(xi−yi)2+λ
2D∑
j=1[
(xj−xj−1)2+(xj−xj+1)2]
(4.148)
where we have deﬁned x0=x1andxD+1=xDfor notational simplicity. We recognize this

min
x1
2σ2N∑
2D∑
j=1[
(4.148)
as a discrete approximation to the following problem:
min
f1
2σ2∫
(f(t)−y(t))2dt+λ
2∫
[f′(t)]2dt (4.149)
wheref′(t)is the ﬁrst derivative of f. The ﬁrst term measures ﬁt to the data, and the second
term penalizes functions that are “too wiggly”. This is an example of Tikhonov regularization,
which is a popular approach to functional data analysis. See Chapter 15 for more sophisticated
approaches, which enforce higher order smoothness (so the resulting samples look less “jagged”).
4.4.3 Proof of the result *

min
f1
2σ2∫
2∫
We now derive Equation 4.125. The basic idea is to derive the joint distribution, p(x,y)=
p(x)p(y|x), and then to use the results from Section 4.3.1 for computing p(x|y).
In more detail, we proceed as follows. The log of the joint distribution is as follows (dropping
irrelevant constants):
logp(x,y)=−1
2(x−μx)TΣ−1
x(x−μx)−1
2(y−Ax−b)TΣ−1
y(y−Ax−b)(4.150)
This is clearly a joint Gaussian distribution, since it is the exponential of a quadratic form.
Expanding out the quadratic terms involving xandy, and ignoring linear and constant terms,
we have
Q=−1
2xTΣ−1

x(x−μx)−1
we have
Q=−1
2xTΣ−1
xx−1
2yTΣ−1
yy−1
2(Ax)TΣ−1
y(Ax)+yTΣ−1
yAx (4.151)
=−1
2(x
y)T(Σ−1
x+ATΣ−1
yA−ATΣ−1
y
−Σ−1
yAΣ−1
y)(x
y)
(4.152)
=−1
2(
x
y)T
Σ−1(
x
y)
(4.153)

4.5. Digression: The Wishart distribution * 125
(b)
Figure 4.15 Interpolating noisy data (noise variance σ2=1) using a Gaussian with prior precision λ. (a)
λ=3 0. (b)λ=0.01. See also Figure 4.10. Based on Figure 7.1 of (Calvetti and Somersalo 2007). Figure
generated by gaussInterpNoisyDemo . See also splineBasisDemo .
where the precision matrix of the joint is deﬁned as
Σ−1=(Σ−1
x+ATΣ−1
yA−ATΣ−1
y
−Σ−1
yAΣ−1
y)
≜Λ=(ΛxxΛxy
ΛyxΛyy)
(4.154)
From Equation 4.69, and using the fact that μy=Aμx+b,w eh a v e
p(x|y)=N (μx|y,Σx|y) (4.155)
Σx|y=Λ−1
xx=(Σ−1
x+ATΣ−1
yA)−1(4.156)
μx|y=Σx|y(
Λxxμx−Λxy(y−μy))
(4.157)
=Σx|y(
Σ−1
xμ+ATΣ−1
y(y−b))
(4.158)
4.5 Digression: The Wishart distribution *
TheWishartdistribution is the generalization of the Gamma distribution to positive deﬁnite
matrices. Press (Press 2005, p107) has said “The Wishart distribution ranks next to the (multi-

Σ−1=(Σ−1
x+ATΣ−1
yA−ATΣ−1
y
−Σ−1
yAΣ−1
y)
ΛyxΛyy)
(4.154)
Σx|y=Λ−1
xx=(Σ−1
x+ATΣ−1
(4.157)
=Σx|y(
Σ−1
xμ+ATΣ−1
y(y−b))
(4.158)
variate) normal distribution in order of importance and usefuleness in multivariate statistics”.We will mostly use it to model our uncertainty in covariance matrices, Σ, or their inverses,
Λ=Σ
−1.
The pdf of the Wishart is deﬁned as follows:
Wi(Λ|S,ν)=1
ZWi|Λ|(ν−D−1)/2exp(
−1
2tr(ΛS−1))
(4.159)
Hereνis called the “degrees of freedom” and Sis the “scale matrix”. (We shall get more
intuition for these parameters shortly.) The normalization constant for this distribution (which

126 Chapter 4. Gaussian models
requires integrating over all symmetric pd matrices) is the following formidable expression
ZWi=2νD/2ΓD(ν/2)|S|ν/2(4.160)
whereΓD(a)is themultivariate gamma function:
ΓD(x)=πD(D−1)/4D∏
i=1Γ(x+(1−i)/2) (4.161)
HenceΓ1(a)=Γ (a)and
ΓD(ν0/2) =D∏
i=1Γ(ν0+1−i
2) (4.162)
The normalization constant only exists (and hence the pdf is only well deﬁned) if ν>D−1.
There is a connection between the Wishart distribution and the Gaussian. In particular,
letxi∼N(0,Σ). Then the scatter matrix S=∑N
i=1xixT
ihas a Wishart distribution:
S∼Wi(Σ,1). Hence E[S]=NΣ. More generally, one can show that the mean and mode of
Wi(S,ν)are given by
mean=νS,mode=(ν−D−1)S (4.163)
where the mode only exists if ν>D+1.
IfD=1, the Wishart reduces to the Gamma distribution:
Wi(λ|s−1,ν)=G a (λ|ν
2,s
2) (4.164)
4.5.1 Inverse Wishart distribution
Recall that we showed (Exercise 2.10) that if λ∼Ga(a,b), then that1
λ∼IG(a,b). Similarly,
ifΣ−1∼Wi(S,ν)thenΣ∼IW(S−1,ν+D+1 ),w h e r eIWis theinverse Wishart, the

i=1xixT
2,s
2) (4.164)
multidimensional generalization of the inverse Gamma. It is deﬁned as follows, for ν>D−1
andS≻0:
IW(Σ|S,ν)=1
ZIW|Σ|−(ν+D+1)/2exp(
−1
2tr(S−1Σ−1))
(4.165)
ZIW=|S|−ν/22νD/2ΓD(ν/2) (4.166)
One can show that the distribution has these properties
mean=S−1
ν−D−1,mode=S−1
ν+D+1(4.167)
IfD=1, this reduces to the inverse Gamma:
IW(σ2|S−1,ν)=I G (σ2|ν/2,S/2) (4.168)

4.6. Inferring the parameters of an MVN 127
−4−2 024−202
−5 0 5−505
−10 0 10−505
−5 0 5−4−2024
−5 0 5−505
−2 02−202
−4−2 024−202Wi(dof=3.0, S), E=[9.5, −0.1; −0.1, 1.9], ρ=−0.0
(a)0 5 10 15 2000.020.040.060.08σ2
1
0 1 2 3 400.10.20.30.4σ22−2 −1 0 1 200.20.40.60.8ρ(1,2)
(b)
Figure 4.16 Visualization of the Wishart distribution. Left: Some samples from the Wishart distribution,
Σ∼Wi(S,ν),w h e r eS=[ 3.1653,−0.0262;−0.0262,0.6477]andν=3. Right: Plots of the marginals
(which are Gamma), and the approximate (sample-based) marginal on the correlation coefficient. If ν=3
there is a lot of uncertainty about the value of the correlation coefficient ρ(see the almost uniform
distribution on [−1,1]). The sampled matrices are highly variable, and some are nearly singular. As ν
increases, the sampled matrices are more concentrated on the prior S. Figure generated by wiPlotDemo .
4.5.2 Visualizing the Wishart distribution *

Since the Wishart is a distribution over matrices, it is hard to plot as a density function. However,
we can easily sample from it, and in the 2d case, we can use the eigenvectors of the resultingmatrix to deﬁne an ellipse, as explained in Section 4.1.2. See Figure 4.16 for some examples.
For higher dimensional matrices, we can plot marginals of the distribution. The diagonals of

a Wishart distributed matrix have Gamma distributions, so are easy to plot. It is hard in generalto work out the distribution of the off-diagonal elements, but we can sample matrices fromthe distribution, and then compute the distribution empirically. In particular, we can converteach sampled matrix to a correlation matrix, and thus compute a Monte Carlo approximation(Section 2.7) to the expected correlation coefficients:
E[R
ij]≈1
SS∑
s=1R(Σ(s))ij (4.169)
whereΣ(s)∼Wi(Σ,ν)andR(Σ)converts matrix Σinto a correlation matrix:
Rij=Σij√
ΣiiΣjj(4.170)

E[R
ij]≈1
SS∑
Rij=Σij√
We can then use kernel density estimation (Section 14.7.2) to produce a smooth approximationto the univariate density E[R
ij]for plotting purposes. See Figure 4.16 for some examples.
4.6 Inferring the parameters of an MVN
So far, we have discussed inference in a Gaussian assuming the parameters θ=(μ,Σ)are
known. We now discuss how to infer the parameters themselves. We will assume the data has

128 Chapter 4. Gaussian models
the form xi∼N(μ,Σ)fori=1:N and is fully observed, so we have no missing data (see
Section 11.6.1 for how to estimate parameters of an MVN in the presence of missing values). To
simplify the presentation, we derive the posterior in three parts: ﬁrst we compute p(μ|D,Σ);
then we compute p(Σ|D,μ); ﬁnally we compute the joint p(μ,Σ|D).
4.6.1 Posterior distribution of μ
We have discussed how to compute the MLE for μ; we now discuss how to compute its posterior,
which is useful for modeling our uncertainty about its value.
The likelihood has the form
p(D|μ)=N(x|μ,1
NΣ) (4.171)
For simplicity, we will use a conjugate prior, which in this case is a Gaussian. In particular, ifp(μ)=N(μ|m
0,V0)then we can derive a Gaussian posterior for μbased on the results in
Section 4.4.2.2. We get
p(μ|D,Σ)=N (μ|mN,VN) (4.172)
V−1
N=V−1
0+NΣ−1(4.173)
mN=VN(Σ−1(Nx)+V−1
0m0) (4.174)

V−1
N=V−1
This is exactly the same process as inferring the location of an object based on noisy radar“blips”, except now we are inferring the mean of a distribution based on noisy samples. (To aBayesian, there is no difference between uncertainty about parameters and uncertainty aboutanything else.)
We can model an uninformative prior by setting V
0=∞I. In this case we have p(μ|D,Σ)=
N(x,1
NΣ), so the posterior mean is equal to the MLE. We also see that the posterior variance

N(x,1
goes down as 1/N, which is a standard result from frequentist statistics.
4.6.2 Posterior distribution of Σ*
We now discuss how to compute p(Σ|D,μ). The likelihood has the form
p(D|μ,Σ)∝|Σ|−N
2exp(
−1
2tr(SμΣ−1))
(4.175)
The corresponding conjugate prior is known as the inverse Wishart distribution (Section 4.5.1).Recall that this has the following pdf:
IW(Σ|S
−1
0,ν0)∝|Σ|−(ν0+D+1)/2exp(
−1
2tr(S0Σ−1))
(4.176)
Hereν0>D−1is the degrees of freedom (dof), and S0is a symmetric pd matrix. We see
thatS−1

2exp(
−1
(4.175)
IW(Σ|S
−1
−1
(4.176)
thatS−1
0plays the role of the prior scatter matrix, and N0≜ν0+D+1controls the strength
of the prior, and hence plays a role analogous to the sample size N.

4.6. Inferring the parameters of an MVN 129
0 5 10 15 20 2500.511.5eigenvalueN=100, D=50
  
true, k=10.00
MLE, k=  71
MAP, k=8.62
0 5 10 15 20 2500.511.5eigenvalueN=50, D=50
  
MLE, k=1.7e+17
MAP, k=8.85
0 5 10 15 20 2500.511.5eigenvalueN=25, D=50
  
MLE, k=2.2e+18
MAP, k=21.09
Figure 4.17 Estimating a covariance matrix in D=5 0dimensions using N∈{100,50,25}samples.
We plot the eigenvalues in descending order for the true covariance matrix (solid black), the MLE (dotted
blue) and the MAP estimate (dashed red), using Equation 4.184 with λ=0.9. We also list the condition
number of each matrix in the legend. Based on Figure 1 of (Schaefer and Strimmer 2005). Figure generatedbyshrinkcovDemo .
Multiplying the likelihood and prior we ﬁnd that the posterior is also inverse Wishart:
p(Σ|D,μ)∝|Σ|−N
2exp(
−1
2tr(Σ−1Sμ))
|Σ|−(ν0+D+1)/2
exp(
−1
2tr(Σ−1S0))
(4.177)
=|Σ|−N+(ν0+D+1)
2exp(
−1
2tr[
Σ−1(Sμ+S0)])
(4.178)
=I W (Σ|SN,νN) (4.179)
νN=ν0+N (4.180)
S−1
N=S0+Sμ (4.181)

2exp(
−1
exp(
−1
(4.177)
2exp(
−1
2tr[
(4.178)
S−1
In words, this says that the posterior strength νNis the prior strength ν0plus the number of
observations N, and the posterior scatter matrix SNis the prior scatter matrix S0plus the data
scatter matrix Sμ.
4.6.2.1 MAP estimation
We see from Equation 4.7 that ˆΣmleis a rank min(N,D )matrix. If N<D, this is not
full rank, and hence will be uninvertible. And even if N>D, it may be the case that ˆΣis
ill-conditioned (meaning it is nearly singular).

S−1
To solve these problems, we can use the posterior mode (or mean). One can show (using
techniques analogous to the derivation of the MLE) that the MAP estimate is given by
ˆΣmap=SN
νN+D+1=S0+Sμ
N0+N(4.182)
If we use an improper uniform prior, corresponding to N0=0andS0=0, we recover the
MLE.

130 Chapter 4. Gaussian models
Let us now consider the use of a proper informative prior, which is necessary whenever D/N
is large (say bigger than 0.1). Let μ=x,s oSμ=Sx. Then we can rewrite the MAP estimate
as a convex combination of the prior mode and the MLE. To see this, let Σ0≜S0
N0be the prior
mode. Then the posterior mode can be rewritten as
ˆΣmap=S0+Sx
N0+N=N0
N0+NS0
N0+N
N0+NS
N=λΣ0+(1−λ)ˆΣmle (4.183)
whereλ=N0
N0+N, controls the amount of shrinkage towards the prior.
This begs the question: where do the parameters of the prior come from? It is common to
setλby cross validation. Alternatively, we can use the closed-form formula provided in (Ledoit
and Wolf 2004b,a; Schaefer and Strimmer 2005), which is the optimal frequentist estimate if we

N0+N=N0
N0+NS0
N0+N
N0+NS
whereλ=N0
use squared loss. This is arguably not the most natural loss function for covariance matrices(because it ignores the postive deﬁnite constraint), but it results in a simple estimator, whichis implemented in the PMTK function shrinkcov . We discuss Bayesian ways of estimating λ
later.
As for the prior covariance matrix, S
0, it is common to use the following (data dependent)
prior:S0=d i a g (ˆΣmle). In this case, the MAP estimate is given by
ˆΣmap(i,j)={ˆΣmle(i,j) ifi=j
(1−λ)ˆΣmle(i,j)otherwise(4.184)

later.
Thus we see that the diagonal entries are equal to their ML estimates, and the off diago-nal elements are “shrunk” somewhat towards 0. This technique is therefore called shrinkage
estimation,o rregularized estimation.
The beneﬁts of MAP estimation are illustrated in Figure 4.17. We consider ﬁtting a 50 dimen-
sional Gaussian to N= 100,N=5 0andN=2 5data points. We see that the MAP estimate
is always well-conditioned, unlike the MLE. In particular, we see that the eigenvalue spectrum

of the MAP estimate is much closer to that of the true matrix than the MLE’s. The eigenvectors,however, are unaffected.
The importance of regularizing the estimate of Σwill become apparent in later chapters,
when we consider ﬁtting covariance matrices to high dimensional data.
4.6.2.2 Univariate posterior
In the 1d case, the likelihood has the form
p(D|σ
2)∝(σ2)−N/2exp(
−1
2σ2N∑
i=1(xi−μ)2)
(4.185)
The standard conjugate prior is the inverse Gamma distribution, which is just the scalar versionof the inverse Wishart:
IG(σ

p(D|σ
−1
2σ2N∑
(4.185)
IG(σ
2|a0,b0)∝(σ2)−(a0+1)exp(−b0
σ2) (4.186)

4.6. Inferring the parameters of an MVN 131
0 5 10 1500.050.10.150.20.250.30.35
σ2prior = IW( ν=0.001, S=0.001), true σ2=10.000
  N=2
N=5
N=50
N=100
Figure 4.18 Sequential updating of the posterior for σ2starting from an uninformative prior. The data
was generated from a Gaussian with known mean μ=5and unknown variance σ2=1 0. Figure generated
bygaussSeqUpdateSigma1D .
Multiplying the likelihood and the prior, we see that the posterior is also IG:
p(σ2|D)=I G ( σ2|aN,bN) (4.187)
aN=a0+N/2 (4.188)
bN=b0+1
2N∑
i=1(xi−μ)2(4.189)
See Figure 4.18 for an illustration.
The form of the posterior is not quite as pretty as the multivariate case, because of the
factors of1
2. This arises because IW(σ2|s0,ν0)=I G (σ2|s0
2,ν0
2). Another problem with using
theIG(a0,b0)distribution is that the strength of the prior is encoded in both a0andb0.
To avoid both of these problems, it is common (in the statistics literature) to use an alternative

bN=b0+1
2N∑
2,ν0
parameterization of the IG distribution, known as the (scaled) inverse chi-squared distribution.
This is deﬁned as follows:
χ−2(σ2|ν0,σ2
0)=I G (σ2|ν0
2,ν0σ2
0
2)∝(σ2)−ν0/2−1exp(−ν0σ2
0
2σ2) (4.190)
Hereν0controls the strength of the prior, and σ2
0encodes the value of the prior. With this
prior, the posterior becomes
p(σ2|D,μ)=χ−2(σ2|νN,σ2
N) (4.191)
νN=ν0+N (4.192)
σ2
N=ν0σ2
0+∑N
i=1(xi−μ)2
νN(4.193)
We see that the posterior dof νNis the prior dof ν0plusN, and the posterior sum of squares
νNσ2
Nis the prior sum of squares ν0σ2

2,ν0σ2
0
0
N) (4.191)
σ2
N=ν0σ2
0+∑N
νN(4.193)
νNσ2
0plus the data sum of squares.
We can emulate an uninformative prior, p(σ2)∝σ−2, by setting ν0=0, which makes
intuitive sense (since it corresponds to a zero virtual sample size).

132 Chapter 4. Gaussian models
4.6.3 Posterior distribution of μandΣ*
We now discuss how to compute p(μ,Σ|D). These results are a bit complex, but will prove
useful later on in this book. Feel free to skip this section on a ﬁrst reading.
4.6.3.1 Likelihood
The likelihood is given by
p(D|μ,Σ)=( 2 π)−ND/2|Σ|−N
2exp(
−1
2N∑
i=1(xi−μ)TΣ−1(xi−μ))
(4.194)
Now one can show that
N∑
i=1(xi−μ)TΣ−1(xi−μ)=t r ( Σ−1Sx)+N(x−μ)TΣ−1(x−μ) (4.195)
Hence we can rewrite the likelihood as follows:
2exp(
−N
2(μ−x)TΣ−1(μ−x))
(4.196)
exp(
−N
2tr(Σ−1Sx))
(4.197)
We will use this form below.
4.6.3.2 Prior
The obvious prior to use is the following
p(μ,Σ)=N(μ|m0,V0)IW(Σ|S0,ν0) (4.198)
Unfortunately, this is not conjugate to the likelihood. To see why, note that μandΣappear
together in a non-factorized way in the likelihood; hence they will also be coupled together in
the posterior.
The above prior is sometimes called semi-conjugate orconditionally conjugate, since both

2exp(
−N
(4.196)
exp(
−N
(4.197)
conditionals, p(μ|Σ)andp(Σ|μ), are individually conjugate. To create a full conjugate prior,
we need to use a prior where μandΣare dependent on each other. We will use a joint
distribution of the form
p(μ,Σ)=p(Σ)p(μ|Σ) (4.199)
Looking at the form of the likelihood equation, Equation 4.197, we see that a natural conjugate

4.6. Inferring the parameters of an MVN 133
prior has the form of a Normal-inverse-wishart orNIWdistribution, deﬁned as follows:
NIW(μ,Σ|m0,κ0,ν0,S0)≜ (4.200)
N(μ|m0,1
κ0Σ)×IW(Σ|S0,ν0) (4.201)
=1
ZNIW|Σ|−1
2exp(
−κ0
2(μ−m0)TΣ−1(μ−m0))
(4.202)
×|Σ|−ν0+D+1
2exp(
−1
(4.203)
=1
ZNIW|Σ|−ν0+D+2
2 (4.204)
×exp(
−κ0
2(μ−m0)TΣ−1(μ−m0)−1
(4.205)
ZNIW=2v0D/2ΓD(ν0/2)(2π/κ0)D/2|S0|−ν0/2(4.206)
whereΓD(a)is the multivariate Gamma function.
The parameters of the NIW can be interpreted as follows: m0is our prior mean for μ, and
κ0is how strongly we believe this prior; and S0is (proportional to) our prior mean for Σ, and
ν0is how strongly we believe this prior.3
One can show (Minka 2000f) that the (improper) uninformative prior has the form
lim
k→0N(μ|m0,Σ/k)IW(Σ|S0,k)∝|2πΣ|−1
2|Σ|−(D+1)/2(4.207)
∝|Σ|−(D
2+1)∝NIW(μ,Σ|0,0,0,0I) (4.208)
In practice, it is often better to use a weakly informative data-dependent prior. A common

lim
∝|Σ|−(D
choice (see e.g., (Chipman et al. 2001, p81), (Fraley and Raftery 2007, p6)) is to use S0=
diag(S x)/N, andν0=D+2, to ensure E[Σ]=S0, and to set μ0=xandκ0to some small
number, such as 0.01.
3. Although this prior has four parameters, there are really only three free parameters, since our uncertainty in the
mean is proportional to the variance. In particular, if we believe that the variance is large, then our uncertainty in μ

must be large too. This makes sense intuitively, since if the data has large spread, it may be hard to pin down its mean.
See also Exercise 9.1, where we will see the three free parameters more explicitly. If we want separate “control” over our
conﬁdence in μandΣ, we must use a semi-conjugate prior.

134 Chapter 4. Gaussian models
4.6.3.3 Posterior
The posterior can be shown (Exercise 4.11) to be NIW with updated parameters:
p(μ,Σ|D)=N I W ( μ,Σ|mN,κN,νN,SN) (4.209)
mN=κ0m0+Nx
κN=κ0
κ0+Nm0+N
κ0+Nx (4.210)
κN=κ0+N (4.211)
νN=ν0+N (4.212)
SN=S0+Sx+κ0N
κ0+N(x−m0)(x−m0)T(4.213)
=S0+S+κ0m0mT
0−κNmNmTN(4.214)
where we have deﬁned S≜∑N
i=1xixT
ias the uncentered sum-of-squares matrix (this is easier
to update incrementally than the centered version).
This result is actually quite intuitive: the posterior mean is a convex combination of the prior
mean and the MLE, with “strength” κ0+N; and the posterior scatter matrix SNis the prior
scatter matrix S0plus the empirical scatter matrix Sxplus an extra term due to the uncertainty
in the mean (which creates its own virtual scatter matrix).
4.6.3.4 Posterior mode
The mode of the joint distribution has the following form:
argmaxp(μ,Σ|D)=(mN,SN
νN+D+2) (4.215)
If we setκ0=0, this reduces to
argmaxp(μ,Σ|D)=(x,S0+Sx
ν0+N+D+2) (4.216)

The corresponding estimate ˆΣis almost the same as Equation 4.183, but differs by 1 in the
denominator, because this is the mode of the joint, not the mode of the marginal.
4.6.3.5 Posterior marginals
The posterior marginal for Σis simply
p(Σ|D)=∫
p(μ,Σ|D)dμ=I W (Σ|SN,νN) (4.217)
The mode and mean of this marginal are given by
ˆΣmap=SN
νN+D+1,E[Σ]=SN
νN−D−1(4.218)
One can show that the posterior marginal for μhas a multivariate Student T distribution:
p(μ|D)=∫
p(μ,Σ|D)dΣ=T(μ|mN,1
κN(νN−D+1)SN,νN−D+1)(4.219)

p(Σ|D)=∫
ˆΣmap=SN
p(μ|D)=∫
This follows from the fact that the Student distribution can be represented as a scaled mixture
of Gaussians (see Equation 11.61).

4.6. Inferring the parameters of an MVN 135
Figure 4.19 TheNIχ2(m0,κ0,ν0,σ2
0)distribution. m0is the prior mean and κ0is how strongly we
believe this; σ2
0is the prior variance and ν0is how strongly we believe this. (a) m0=0,κ0=1,ν0=
1,σ2
0=1. Notice that the contour plot (underneath the surface) is shaped like a “squashed egg”. (b) We
increase the strength of our belief in the mean, so it gets narrower: m0=0,κ0=5,ν0=1,σ2
0=1. (c)
We increase the strength of our belief in the variance, so it gets narrower: m0=0,κ0=1,ν0=5,σ2
0=
1. Figure generated by NIXdemo2 .
4.6.3.6 Posterior predictive
The posterior predictive is given by
p(x|D)=p(x,D)
p(D)(4.220)
so it can be easily evaluated in terms of a ratio of marginal likelihoods.
It turns out that this ratio has the form of a multivariate Student-T distribution:
p(x|D)=∫∫
N(x|μ,Σ)NIW(μ,Σ|mN,κN,νN,SN)dμdΣ (4.221)
=T(x|mN,κN+1
κN(νN−D+1)SN,νN−D+1) (4.222)
The Student-T has wider tails than a Gaussian, which takes into account the fact that Σis

0=
p(x|D)=∫∫
unknown. However, this rapidly becomes Gaussian-like.
4.6.3.7 Posterior for scalar data
We now specialise the above results to the case where xiis 1d. These results are widely used
in the statistics literature. As in Section 4.6.2.2, it is conventional not to use the normal inverse

136 Chapter 4. Gaussian models
Wishart, but to use the normal inverse chi-squared orNIXdistribution, deﬁned by
NIχ2(μ,σ2|m0,κ0,ν0,σ2
0)≜N(μ|m0,σ2/κ0)χ−2(σ2|ν0,σ2
0) (4.223)
∝(1
σ2)(ν0+3)/2exp(
−ν0σ2
0+κ0(μ−m0)2
2σ2)
(4.224)
See Figure 4.19 for some plots. Along the μaxis, the distribution is shaped like a Gaussian, and
along the σ2axis, the distribution is shaped like a χ−2; the contours of the joint density have
a “squashed egg” appearance. Interestingly, we see that the contours for μare more peaked
for small values of σ2, which makes sense, since if the data is low variance, we will be able to
estimate its mean more reliably.
One can show that the posterior is given by
p(μ,σ2|D)=NIχ2(μ,σ2|mN,κN,νN,σ2
N) (4.225)
κN(4.226)
κN=κ0+N (4.227)
νN=ν0+N (4.228)
νNσ2
N=ν0σ2
0+N∑
i=1(xi−x)2+Nκ0
κ0+N(m0−x)2(4.229)
The posterior marginal for σ2is just
p(σ2|D)=∫
p(μ,σ2|D)dμ=χ−2(σ2|νN,σ2
N) (4.230)
with the posterior mean given by E[
σ2|D]
=νN
νN−2σ2
N.

N) (4.225)
κN(4.226)
νNσ2
N=ν0σ2
0+N∑
p(σ2|D)=∫
N) (4.230)
σ2|D]
=νN
νN−2σ2
N.
The posterior marginal for μhas a Student T distribution, which follows from the scale
mixture representation of the student:
p(μ|D)=∫
p(μ,σ2|D)dσ2=T(μ|mN,σ2
N/κN,νN) (4.231)
with the posterior mean given by E[μ|D]=mN.
Let us see how these results look if we use the following uninformative prior:
p(μ,σ2)∝p(μ)p(σ2)∝σ−2∝NIχ2(μ,σ2|μ0=0,κ0=0,ν0=−1,σ2
0=0 ) (4.232)
With this prior, the posterior has the form
p(μ,σ2|D)=NIχ2(μ,σ2|mN=x,κN=N,νN=N−1,σ2
N=s2) (4.233)
where
s2≜1
N−1N∑
i=1(xi−x)2=N
N−1ˆσ2
mle (4.234)

p(μ|D)=∫
where
s2≜1
N−1N∑
N−1ˆσ2
is the the sample standard deviation. (In Section 6.4.2, we show that this is an unbiased
estimate of the variance.) Hence the marginal posterior for the mean is given by
p(μ|D)=T(μ|x,s2
N,N−1) (4.235)

4.6. Inferring the parameters of an MVN 137
and the posterior variance of μis
var[μ|D]=νN
νN−2σ2
N=N−1
N−3s2
N→s2
N(4.236)
The square root of this is called the standard error of the mean:
√
var[μ|D]≈s√
N(4.237)
Thus an approximate 95% posterior credible interval for the mean is
I.95(μ|D)=x±2s√
N(4.238)
(Bayesian credible intervals are discussed in more detail in Section 5.2.2; they are contrasted
with frequentist conﬁdence intervals in Section 6.6.1.)
4.6.3.8 Bayesian t-test
Suppose we want to test the hypothesis that μ̸=μ0for some known value μ0(often 0), given
valuesxi∼N(μ,σ2). This is called a two-sided, one-sample t-test. A simple way to perform
such a test is just to check if μ0∈I0.95(μ|D). If it is not, then we can be 95% sure that
μ̸=μ0.4A more common scenario is when we want to test if two paired samples have
the same mean. More precisely, suppose yi∼N(μ1,σ2)andzi∼N(μ2,σ2). We want to
determine if μ=μ1−μ2>0, usingxi=yi−zias our data. We can evaluate this quantity
p(μ>μ0|D)=∫∞

μ0p(μ|D)dμ (4.239)
This is called a one-sided, paired t-test. (For a similar approach to unpaired tests, comparing
the difference in binomial proportions, see Section 5.2.3.)
To calculate the posterior, we must specify a prior. Suppose we use an uninformative prior.
As we showed above, we ﬁnd that the posterior marginal on μhas the form
N,N−1) (4.240)
Now let us deﬁne the following t statistic:
t≜x−μ0
s/√
N(4.241)
where the denominator is the standard error of the mean. We see that
p(μ|D)=1−FN−1(t) (4.242)

t≜x−μ0
s/√
N(4.241)
whereFν(t)is the cdf of the standard Student t distribution T(0,1,ν).
4. A more complex approach is to perform Bayesian model comparison. That is, we compute the Bayes factor (described
in Section 5.3.3) p(D|H0)/p(D|H1),w h e r eH0is the point null hypothesis that μ=μ0, andH1is the alternative
hypothesis that μ̸=μ0. See (Gonen et al. 2005; Rouder et al. 2009) for details.

138 Chapter 4. Gaussian models
4.6.3.9 Connection with frequentist statistics *
If we use an uninformative prior, it turns out that the above Bayesian analysis gives the same
result as derived using frequentist methods. (We discuss frequentist statistics in Chapter 6.)Speciﬁcally, from the above results, we see that
μ−
x√
s/N|D ∼tN−1 (4.243)
This has the same form as the sampling distribution of the MLE:
μ−X√
s/N|μ∼tN−1 (4.244)
ThereasonisthattheStudentdistributionissymmetricinitsﬁrsttwoarguments, so T(x|μ,σ2,ν)=
T(μ|x,σ2,ν); hence statements about the posterior for μhave the same form as statements
about the sampling distribution of x. Consequently, the (one-sided) p-value (deﬁned in Sec-
tion 6.6.2) returned by a frequentist test is the same as p(μ>μ0|D)returned by the Bayesian
method. See bayesTtestDemo for an example.
Despite the superﬁcial similarity, these two results have a different interpretation: in the
Bayesian approach, μis unknown and xis ﬁxed, whereas in the frequentist approach, X

is unknown and μis ﬁxed. More equivalences between frequentist and Bayesian inference
in simple models using uninformative priors can be found in (Box and Tiao 1973). See alsoSection 7.6.3.3.
4.6.4 Sensor fusion with unknown precisions *

In this section, we apply the results in Section 4.6.3 to the problem of sensor fusion in thecase where the precision of each measurement device is unknown. This generalizes the resultsof Section 4.4.2.2, where the measurement model was assumed to be Gaussian with knownprecision. The unknown precision case turns out to give qualitatively different results, yieldinga potentially multi-modal posterior as we will see. Our presentation is based on (Minka 2001e).
Suppose we want to pool data from multiple sources to estimate some quantity μ∈R, but the

reliability of the sources is unknown. Speciﬁcally, suppose we have two different measurementdevices,xandy, with different precisions: x
i|μ∼N(μ,λ−1
x)andyi|μ∼N(μ,λ−1
y).W e
make two independent measurements with each device, which turn out to be
x1=1.1,x2=1.9,y1=2.9,y2=4.1 (4.245)
We will use a non-informative prior for μ,p(μ)∝1, which we can emulate using an inﬁnitely
broad Gaussian, p(μ)=N(μ|m0=0,λ−1
0=∞). If theλxandλyterms were known, then
the posterior would be Gaussian:
p(μ|D,λx,λy)=N (μ|mN,λ−1
N) (4.246)
λN=λ0+Nxλx+Nyλy (4.247)
mN=λxNxx+λyNyy
Nxλx+Nyλy(4.248)

4.6. Inferring the parameters of an MVN 139
whereNx=2is the number of xmeasurements, Ny=2is the number of y measurements,
x=1
Nx∑Nx
i=1xi=1.5andy=1
Ny∑Ny
i=1yi=3.5. This result follows because the posterior
precision is the sum of the measurement precisions, and the posterior mean is a weighted sum
of the prior mean (which is 0) and the data means.
However, the measurement precisions are not known. Initially we will estimate them by
maximum likelihood. The log-likelihood is given by
ℓ(μ,λx,λy)=l o g λx−λx
2∑
i(xi−μ)2+logλy−λy
2∑
i(yi−μ)2(4.249)
The MLE is obtained by solving the following simultaneous equations:
∂ℓ
∂μ=λxNx(x−μ)+λyNy(y−μ)=0 (4.250)
∂ℓ
∂λx=1
λx−1
NxNx∑
i=1(xi−μ)2=0 (4.251)
∂ℓ
∂λy=1
λy−1
NyNy∑
i=1(yi−μ)2=0 (4.252)
This gives
ˆμ=Nxˆλxx+Nyˆλyy
Nxˆλx+Nyˆλy(4.253)
1/ˆλx=1
Nx∑
i(xi−ˆμ)2(4.254)
1/ˆλy=1
Ny∑
i(yi−ˆμ)2(4.255)
We notice that the MLE for μhas the same form as the posterior mean, mN.
Wecansolvetheseequationsbyﬁxedpointiteration. Letusinitializebyestimating λx=1/s2
x

2∑
2∑
∂ℓ
∂ℓ
∂λx=1
λx−1
NxNx∑
∂ℓ
∂λy=1
λy−1
NyNy∑
This gives
1/ˆλx=1
Nx∑
1/ˆλy=1
Ny∑
x
andλy=1/s2y,w h e r es2x=1
Nx∑Nx
i=1(xi−x)2=0.16ands2y=1
Ny∑Ny
i=1(yi−y)2=0.36.
Using this, we get ˆμ=2.1154,s o p(μ|D,ˆλx,ˆλy)=N(μ|2.1154,0.0554).I fw en o wi t e r a t e ,
we converge to ˆλx=1/0.1662,ˆλy=1/4.0509,p(μ|D,ˆλx,ˆλy)=N(μ|1.5788,0.0798).
The plug-in approximation to the posterior is plotted in Figure 4.20(a). This weights each
sensor according to its estimated precision. Since sensor ywas estimated to be much less
reliable than sensor x,w eh a v eE[
μ|D,ˆλx,ˆλy]

x
Nx∑Nx
Ny∑Ny
≈x, so we effectively ignore the ysensor.
Now we will adopt a Bayesian approach and integrate out the unknown precisions, rather
than trying to estimate them. That is, we compute
p(μ|D)∝p(μ)[∫
p(Dx|μ,λx)p(λx|μ)dλx][∫
p(Dy|μ,λy)p(λy|μ)dλy]
(4.256)
We will use uninformative Jeffrey’s priors, p(μ)∝1,p(λx|μ)∝1/λxandp(λy|μ)∝1/λy.

140 Chapter 4. Gaussian models
Since the xandyterms are symmetric, we will just focus on one of them. The key integral is
I=∫
p(Dx|μ,λx)p(λx|μ)dλx∝∫
λ−1
x(Nxλx)Nx/2(4.257)
exp(
−Nx
2λx(x−μ)2−Nx
2s2
xλx)
dλx(4.258)
Exploiting the fact that Nx=2this simpliﬁes to
I=∫
λ−1
xλ1xexp(−λx[(x−μ)2+s2x])dλx (4.259)
We recognize this as proportional to the integral of an unnormalized Gamma density
Ga(λ|a,b)∝λa−1e−λb(4.260)
wherea=1andb=(x−μ)2+s2
x. Hence the integral is proportional to the normalizing
constant of the Gamma distribution, Γ(a)b−a,s ow eg e t
I∝∫
p(Dx|μ,λx)p(λx|μ)dλx∝(
x−μ)2+s2
x)−1(4.261)
and the posterior becomes
p(μ|D)∝1
(x−μ)2+s2x1
(y−μ)2+s2y(4.262)
The exact posterior is plotted in Figure 4.20(b). We see that it has two modes, one near
x=1.5and one near y=3.5. These correspond to the beliefs that the xsensor is more
reliable than the yone, and vice versa. The weight of the ﬁrst mode is larger, since the data
from the xsensor agree more with each other, so it seems slightly more likely that the xsensor

I∝∫
x−μ)2+s2
p(μ|D)∝1
is the reliable one. (They obviously cannot both be reliable, since they disagree on the values
that they are reporting.) However, the Bayesian solution keeps open the possibility that the y
sensor is the more reliable one; from two measurements, we cannot tell, and choosing just thexsensor, as the plug-in approximation does, results in over conﬁdence (a posterior that is too
narrow).
Exercises
Exercise 4.1 Uncorrelated does not imply independent
LetX∼U(−1,1)andY=X2. Clearly Yis dependent on X(in fact,Yis uniquely determined

narrow).
Exercises
byX). However, show that ρ(X,Y)=0. Hint: if X∼U(a,b)thenE[X]=(a+b)/2and
var[X]=(b−a)2/12.
Exercise 4.2 Uncorrelated and Gaussian does not imply independent unless jointlyGaussian
LetX∼N(0,1)andY=WX,w h e r ep(W=−1) =p(W=1 )=0 .5. It is clear that XandYare
not independent, since Yis a function of X.
a. ShowY∼N(0,1).

4.6. Inferring the parameters of an MVN 141
−2 −1 0 1 2 3 4 5 600.10.20.30.40.50.60.70.8
(a)−2 −1 0 1 2 3 4 5 600.511.5
(b)
Figure 4.20 Posterior for μ. (a) Plug-in approximation. (b) Exact posterior. Figure generated by
sensorFusionUnknownPrec .
b. Showcov[X,Y ]=0. ThusXandYare uncorrelated but dependent, even though they are Gaussian.
Hint: use the deﬁnition of covariance
cov[X,Y ]=E[XY]−E[X]E[Y] (4.263)
and therule of iterated expectation
E[XY]=E[E[XY|W]] (4.264)
Exercise 4.3 Correlation coefficient is between -1 and +1
Prove that −1≤ρ(X,Y)≤1
Exercise 4.4 Correlation coefficient for linearly related variables is ±1
Show that, if Y=aX+bfor some parameters a>0andb, thenρ(X,Y)=1. Similarly show that if
a<0, thenρ(X,Y)=−1.
Exercise 4.5 Normalization constant for a multidimensional Gaussian
Prove that the normalization constant for a d-dimensional Gaussian is given by
(2π)d/2|Σ|1
2=∫
exp(−1
2(x−μ)TΣ−1(x−μ))dx (4.265)
Hint: diagonalize Σand use the fact that |Σ|=∏
iλito write the joint pdf as a product of done-

2=∫
exp(−1
dimensional Gaussians in a transformed coordinate system. (You will need the change of variables formula.)
Finally, use the normalization constant for univariate Gaussians.
Exercise 4.6 Bivariate Gaussian
Letx∼N(μ,Σ)wherex∈R2and
Σ=(σ2
1ρσ1σ2
ρσ1σ2σ2
2)
(4.266)
whereρis the correlation coefficient. Show that the pdf is given by
p(x1,x2)=1
2πσ1σ2√
1−ρ2(4.267)
exp(
−1
2(1−ρ2)((x1−μ1)2
σ2
1+(x2−μ2)2
σ2
2−2ρ(x1−μ1)
σ1(x2−μ2)
σ2))
(4.268)

142 Chapter 4. Gaussian models
60 65 70 75 8080100120140160180200220240260280
12
34
56
789
1011
1213
141516
17
181920
2122
23
2425
26
27
28
293031
32333435
36
37
383940
41424344
45
46474849
5051
52
53
5455
5657
58
5960
61 62636465
6667
68
69 70
717273raw
−3 −2 −1 0 1 2 3−3−2−101234
12
34
56
789
1011
1213
141516
17
181920
2122
23
2425
26
27
28
293031
32333435
3637
383940
41424344
45
46474849
5051
52
535455
5657
58
596061 62636465
6667
68
69 70717273standarized
12
34
56
789
1011
1213
141516
17
181920
2122
23
2425
2627
28
293031
32333435
3637
383940
41424344
45
464748 49
5051
52
535455
5657
58
596061 6263 6465
666768
69 70717273whitened
Figure 4.21 (a) Height/weight data for the men. (b) Standardized. (c) Whitened.
Exercise 4.7 Conditioning a bivariate Gaussian
Consider a bivariate Gaussian distribution p(x1,x2)=N(x|μ,Σ)where
Σ=(σ2
1σ12
σ21σ2
2)
=σ1σ2(σ1
σ2ρ
ρσ2
σ1)
(4.269)
where the correlation coefficient is given by
ρ≜σ12
σ1σ2(4.270)

12
34
56
789
1011
1213
141516
17
181920
2122
23
2425
2627
28
293031
32333435
3637
383940
41424344
45
464748 49
5051
52
535455
5657
58
666768
Σ=(σ2
1σ12
σ21σ2
2)
=σ1σ2(σ1
σ2ρ
ρσ2
σ1)
(4.269)
ρ≜σ12
a. What is P(X2|x1)? Simplify your answer by expressing it in terms of ρ,σ2,σ1,μ1,μ2andx1.
b. Assume σ1=σ2=1. What is P(X2|x1)now?
Exercise 4.8 Whitening vs standardizing
a. Load the height/weight data using rawdata = dlmread(’heightWeightData.txt’) . The ﬁrst col-
umn is the class label (1=male, 2=female), the second column is height, the third weight. Extract the

Σ=(σ2
1σ12
σ21σ2
2)
=σ1σ2(σ1
σ2ρ
ρσ2
σ1)
(4.269)
ρ≜σ12
height/weight data corresponding to the males. Fit a 2d Gaussian to the male data, using the empiricalmean and covariance. Plot your Gaussian as an ellipse (use gaussPlot2d ), superimposing on your
scatter plot. It should look like Figure 4.21(a), where have labeled each datapoint by its index. Turn inyour ﬁgure and code.
b.Standardizing the data means ensuring the empirical variance along each dimension is 1. This can be
done by computing
xij−xj
σj,w h e r eσjis the empirical std of dimension j. Standardize the data and

xij−xj
replot. It should look like Figure 4.21(b). (Use axis(’equal’) .) Turn in your ﬁgure and code.
c.Whitening orsphereing the data means ensuring its empirical covariance matrix is proportional to
I, so the data is uncorrelated and of equal variance along each dimension. This can be done by
computing Λ−1
2UTxfor each data vector x,w h e r e Uare the eigenvectors and Λthe eigenvalues of
X. Whiten the data and replot. It should look like Figure 4.21(c). Note that whitening rotates the data,
so people move to counter-intuitive locations in the new coordinate system (see e.g., person 2, who
moves from the right hand side to the left).
Exercise 4.9 Sensor fusion with known variances in 1d

Suppose we have two sensors with known (and different) variances v1andv2, but unknown (and the same)
meanμ. Suppose we observe n1observations y(1)
i∼N(μ,v1)from the ﬁrst sensor and n2observations

4.6. Inferring the parameters of an MVN 143
y(2)
i∼N(μ,v2)from the second sensor. (For example, suppose μis the true temperature outside,
and sensor 1 is a precise (low variance) digital thermosensing device, and sensor 2 is an imprecise (high
variance) mercury thermometer.) Let Drepresent all the data from both sensors. What is the posterior
p(μ|D), assuming a non-informative prior for μ(which we can simulate using a Gaussian with a precision
of 0)? Give an explicit expression for the posterior mean and variance.
Exercise 4.10 Derivation of information form formulae for marginalizing and conditioning
Derive the information form results of Section 4.3.1.Exercise 4.11 Derivation of the NIW posterior
Derive Equation 4.209. Hint: one can show that
N(
x−μ)(x−μ)T+κ0(μ−m0)(μ−m0)T(4.271)
=κN(μ−mN)(μ−mN)T+κ0N
κN(x−m0)(x−m0)T(4.272)
This is a matrix generalization of an operation called completing the square.5
Derive the corresponding result for the normal-Wishart model.Exercise 4.12 BIC for Gaussians

N(
(Source: Jaakkola.)
The Bayesian information criterion (BIC) is a penalized log-likelihood function that can be used for model
selection (see Section 5.3.2.4). It is deﬁned as
BIC=l o gp(D|ˆθ
ML)−d
2log(N) (4.273)
wheredis the number of free parameters in the model and Nis the number of samples. In this question,

ML)−d
we will see how to use this to choose between a full covariance Gaussian and a Gaussian with a diagonalcovariance. Obviously a full covariance Gaussian has higher likelihood, but it may not be “worth” the extraparameters if the improvement over a diagonal covariance matrix is too small. So we use the BIC score tochoose the model.
Following Section 4.1.3, we can write
logp(D|ˆΣ,ˆμ)=−N
2tr(
ˆΣ−1ˆS)
−N
2log(ˆ|Σ|) (4.274)
ˆS=1
NN∑
i=1(xi−x)(xi−x)T(4.275)

2tr(
ˆΣ−1ˆS)
−N
ˆS=1
NN∑
whereˆSis the scatter matrix (empirical covariance), the trace of a matrix is the sum of its diagonals, and
we have used the trace trick.
a. Derive the BIC score for a Gaussian in Ddimensions with full covariance matrix. Simplify your answer
as much as possible, exploiting the form of the MLE. Be sure to specify the number of free parameters
d.
b. Derive the BIC score for a Gaussian in Ddimensions with a diagonal covariance matrix. Be sure to
specify the number of free parameters d. Hint: for the digaonal case, the ML estimate of Σis the same

d.
asˆΣMLexcept the off-diagonal terms are zero:
ˆΣdiag=d i a g (ˆΣML(1,1),...,ˆΣML(D,D)) (4.276)
5. In the scalar case, completing the square means rewriting c2x2+c1x+c0as−a(x−b)2+wwherea=−c2,
b=c1
2c2andw=c2
1
4c2+c0.

144 Chapter 4. Gaussian models
Exercise 4.13 Gaussian posterior credible interval
(Source: DeGroot.)
LetX∼N(μ,σ2=4 )whereμis unknown but has prior μ∼N(μ0,σ2
0=9 ). The posterior after
seeingnsamples is μ∼N(μn,σ2
n). (This is called a credible interval, and is the Bayesian analog of a
conﬁdence interval.) How big does nhave to be to ensure
p(ℓ≤μn≤u|D)≥0.95 (4.277)
where(ℓ,u)is an interval (centered on μn) of width 1 and Dis the data. Hint: recall that 95% of the
probability mass of a Gaussian is within ±1.96σof the mean.
Exercise 4.14 MAP estimation for 1D Gaussians
(Source: Jaakkola.)Consider samples x
1,...,x nfrom a Gaussian random variable with known variance σ2and unknown
meanμ. We further assume a prior distribution (also Gaussian) over the mean, μ∼N(m,s2), with ﬁxed
meanmand ﬁxed variance s2. Thus the only unknown is μ.
a. Calculate the MAP estimate ˆμMAP. You can state the result without proof. Alternatively, with a lot
more work, you can compute derivatives of the log posterior, set to zero and solve.

b. Show that as the number of samples nincrease, the MAP estimate converges to the maximum likelihood
estimate.
c. Suppose nis small and ﬁxed. What does the MAP estimator converge to if we increase the prior
variances2?
d. Suppose nis small and ﬁxed. What does the MAP estimator converge to if we decrease the prior
Exercise 4.15 Sequential (recursive) updating of ˆΣ
(Source: (Duda et al. 2001, Q3.35,3.36).)The unbiased estimates for the covariance of a d-dimensional Gaussian based on nsamples is given by

estimate.
ˆΣ=C
n=1
n−1n∑
i=1(xi−mn)(xi−mn)T(4.278)
It is clear that it takes O(nd2)time to compute Cn. If the data points arrive one at a time, it is more
efficient to incrementally update these estimates than to recompute from scratch.
a. Show that the covariance can be sequentially udpated as follows
Cn+1=n−1
nCn+1
n+1(xn+1−mn)(xn+1−mn)T(4.279)
b. How much time does it take per sequential update? (Use big-O notation.)
c. Show that we can sequentially update the precision matrix using
C−1
n+1=n
n−1[
C−1
n−C−1
n(xn+1−mn)(xn+1−mn)TC−1
n
n2−1
n+(xn+1−mn)TC−1
n(xn+1−mn)]
(4.280)

Cn+1=n−1
nCn+1
C−1
n+1=n
n−1[
C−1
n−C−1
n
n2−1
(4.280)
Hint: notice that the update to Cn+1consists of adding a rank-one matrix, namely uuT,w h e r e
u=xn+1−mn. Use the matrix inversion lemma for rank-one updates (Equation 4.111), which we
repeat here for convenience:
(E+uvT)−1=E−1−E−1uvTE−1
1+vTE−1u(4.281)

4.6. Inferring the parameters of an MVN 145
d. What is the time complexity per update?
Exercise 4.16 Likelihood ratio for Gaussians
Source: Source: Alpaydin p103 ex 4. Consider a binary classiﬁer where the Kclass conditional densities
are MVN p(x|y=j)=N(x|μj,Σj). By Bayes rule, we have
logp(y=1|x)
p(y=0|x)=l o gp(x|y=1 )
p(x|y=0 )+logp(y=1 )
p(y=0 )(4.282)
In other words, the log posterior ratio is the log likelihood ratio plus the log prior ratio. For each of the 4
cases in the table below, derive an expression for the log likelihood ratio logp(x|y=1)
p(x|y=0), simplifying as much
as possible.
Form ofΣj Cov Num parameters
Arbitrary Σj Kd(d+1)/2
Shared Σj=Σ d(d+1)/2
Shared, axis-aligned Σj=ΣwithΣij=0fori̸=jd
Shared, spherical Σj=σ2I 1
Exercise 4.17 LDA/QDA on height/weight data
The function discrimAnalysisHeightWeightDemo ﬁts an LDA and QDA model to the height/weight
data. Compute the misclassiﬁcation rate of both of these models on the training set. Turn in your numbers
and code.

and code.
Exercise 4.18 Naive Bayes with mixed features
Consider a 3 class naive Bayes classiﬁer with one binary feature and one Gaussian feature:
y∼Mu(y|π,1),x1|y=c∼Ber(x1|θc),x2|y=c∼N(x2|μc,σ2
c) (4.283)
Let the parameter vectors be as follows:
π=( 0.5,0.25,0.25),θ=( 0.5,0.5,0.5),μ=(−1,0,1),σ2=( 1,1,1) (4.284)
a. Compute p(y|x1=0,x2=0 )(the result should be a vector of 3 numbers that sums to 1).
b. Compute p(y|x1=0 ).
c. Compute p(y|x2=0 ).
d. Explain any interesting patterns you see in your results. Hint: look at the parameter vector θ.

c) (4.283)
Exercise 4.19 Decision boundary for LDA with semi tied covariances
Consider a generative classiﬁer with class conditional densities of the form N(x|μc,Σc). I nL D A ,w e
assumeΣc=Σ, and in QDA, each Σcis arbitrary. Here we consider the 2 class case in which
Σ1=kΣ0,f o rk>1. That is, the Gaussian ellipsoids have the same “shape”, but the one for class 1
is “wider”. Derive an expression for p(y=1|x,θ), simplifying as much as possible. Give a geometric

interpretation of your result, if possible.Exercise 4.20 Logistic regression vs LDA/QDA
(Source: Jaakkola.) Suppose we train the following binary classiﬁers via maximum likelihood.
a. GaussI: A generative classiﬁer, where the class conditional densities are Gaussian, with both covariance
matrices set to I(identity matrix), i.e., p(x|y=c)=N(x|μ
c,I). We assume p(y)is uniform.
b. GaussX: as for GaussI, but the covariance matrices are unconstrained, i.e., p(x|y=c)=N(x|μc,Σc).

146 Chapter 4. Gaussian models
c. LinLog: A logistic regression model with linear features.
d. QuadLog: A logistic regression model, using linear and quadratic features (i.e., polynomial basis function
expansion of degree 2).
After training we compute the performance of each model Mon the training set as follows:
L(M)=1
nn∑
i=1logp(yi|xi,ˆθ,M) (4.285)
(Note that this is the conditional log-likelihood p(y|x,ˆθ)and not the joint log-likelihood p(y,x|ˆθ).) We
now want to compare the performance of each model. We will write L(M)≤L(M′)if modelMmust
have lower (or equal) log likelihood (on the training set) than M′, for any training set (in other words, Mis
worse than M′, at least as far as training set logprob is concerned). For each of the following model pairs,
state whether L(M)≤L(M′),L(M)≥L(M′), or whether no such statement can be made (i.e., M
might sometimes be better than M′and sometimes worse); also, for each question, brieﬂy (1-2 sentences)
explain why.
a. GaussI, LinLog.
b. GaussX, QuadLog.

c. LinLog, QuadLog.
d. GaussI, QuadLog.
e. Now suppose we measure performance in terms of the average misclassiﬁcation rate on the training
set:
R(M)=1
nn∑
i=1I(yi̸=ˆy(xi)) (4.286)
Is it true in general that L(M)>L(M′)implies that R(M)<R(M′)? Explain why or why not.
Exercise 4.21 Gaussian decision boundaries
(Source: (Duda et al. 2001, Q3.7).) Let p(x|y=j)=N(x|μj,σj)wherej=1,2andμ1=0,σ2
1=
1,μ2=1,σ2
2=1 06. Let the class priors be equal, p(y=1 )=p (y=2 )=0 .5.
a. Find the decision region
R1={x:p(x|μ1,σ1)≥p(x|μ2,σ2)} (4.287)

set:
R(M)=1
nn∑
1=
1,μ2=1,σ2
Sketch the result. Hint: draw the curves and ﬁnd where they intersect. Find bothsolutions of the
equation
p(x|μ1,σ1)=p(x|μ2,σ2) (4.288)
Hint: recall that to solve a quadratic equation ax2+bx+c=0,w eu s e
x=−b±√
b2−4ac
2a(4.289)
b. Now suppose σ2=1(and all other parameters remain the same). What is R1in this case?

4.6. Inferring the parameters of an MVN 147
Exercise 4.22 QDA with 3 classes
Consider a three category classiﬁcation problem. Let the prior probabilites:
P(Y=1 )=P (Y=2 )=P (Y=3 )=1 /3 (4.290)
The class-conditional densities are multivariate normal densities with parameters:
μ1=[ 0,0]T,μ2=[ 1,1]T,μ3=[−1,1]T(4.291)
Σ1=[0.70
00.7]
,Σ2=[0.80.2
0.20.8]
,Σ3=[0.80.2
0.20.8]
(4.292)
Classify the following points:
a.x=[−0.5,0.5]
b.x=[ 0.5,0.5]
Exercise 4.23 Scalar QDA
[Note: you can solve this exercise by hand or using a computer (matlab, R, whatever). In either case, show
your work.] Consider the following training set of heights x(in inches) and gender y(male/female) of some
US college students: x=( 6 7,79,71,68,67,60),y=(m,m,m,f,f,f ).
a. Fit a Bayes classiﬁer to this data, using maximum likelihood estimation, i.e., estimate the parameters of
the class conditional likelihoods
p(x|y=c)=N(x;μc,σc) (4.293)
and the class prior
p(y=c)=πc (4.294)

What are your values of μc,σc,πcforc=m,f? Show your work (so you can get partial credit if you
make an arithmetic error).
b. Compute p(y=m|x,ˆθ),w h e r ex=7 2, andˆθare the MLE parameters. (This is called a plug-in
prediction.)
c. What would be a simple way to extend this technique if you had multiple attributes per person, such
as height and weight? Write down your proposed model as an equation.

5 Bayesian statistics
5.1 Introduction
We have now seen a variety of different probability models, and we have discussed how to
ﬁt them to data, i.e., we have discussed how to compute MAP parameter estimates ˆθ=
argmaxp(θ|D), using a variety of different priors. We have also discussed how to compute
the full posterior p(θ|D), as well as the posterior predictive density, p(x|D), for certain special
cases (and in later chapters, we will discuss algorithms for the general case).
Using the posterior distribution to summarize everything we know about a set of unknown
variables is at the core of Bayesian statistics. In this chapter, we discuss this approach to
statistics in more detail. In Chapter 6, we discuss an alternative approach to statistics known asfrequentist or classical statistics.
5.2 Summarizing posterior distributions
The posterior p(θ|D)summarizes everything we know about the unknown quantities θ. In this

section, we discuss some simple quantities that can be derived from a probability distribution,such as a posterior. These summary statistics are often easier to understand and visualize thanthe full joint.
5.2.1 MAP estimation
We can easily compute a point estimate of an unknown quantity by computing the posterior

mean, median or mode. In Section 5.7, we discuss how to use decision theory to choose betweenthese methods. Typically the posterior mean or median is the most appropriate choice for a real-valued quantity, and the vector of posterior marginals is the best choice for a discrete quantity.However, the posterior mode, aka the MAP estimate, is the most popular choice because itreduces to an optimization problem, for which efficient algorithms often exist. Futhermore, MAPestimation can be interpreted in non-Bayesian terms, by thinking of the log prior as a regularizer(see Section 6.5 for more details).

Although this approach is computationally appealing, it is important to point out that there
are various drawbacks to MAP estimation, which we brieﬂy discuss below. This will providemotivation for the more thoroughly Bayesian approach which we will study later in this chapter(and elsewhere in this book).

150 Chapter 5. Bayesian statistics
−2 −1 0 1 2 3 400.511.522.533.544.5
(a)1 2 3 4 5 6 70.10.20.30.40.50.60.70.80.9
(b)
Figure 5.1 (a) A bimodal distribution in which the mode is very untypical of the distribution. The thin
blue vertical line is the mean, which is arguably a better summary of the distribution, since it is near the
majority of the probability mass. Figure generated by bimodalDemo . (b) A skewed distribution in which
the mode is quite different from the mean. Figure generated by gammaPlotDemo .
5.2.1.1 No measure of uncertainty
The most obvious drawback of MAP estimation, and indeed of any other point estimate such
as the posterior mean or median, is that it does not provide any measure of uncertainty. In
many applications, it is important to know how much one can trust a given estimate. We canderive such conﬁdence measures from the posterior, as we discuss in Section 5.2.2.
5.2.1.2 Plugging in the MAP estimate can result in overﬁtting

In machine learning, we often care more about predictive accuracy than in interpreting theparameters of our models. However, if we don’t model the uncertainty in our parameters, thenour predictive distribution will be overconﬁdent. We saw several examples of this in Chapter 3,and we will see more examples later. Overconﬁdence in predictions is particularly problematicin situations where we may be risk averse; see Section 5.7 for details.
5.2.1.3 The mode is an untypical point

Choosing the mode as a summary of a posterior distribution is often a very poor choice, sincethe mode is usually quite untypical of the distribution, unlike the mean or median. This isillustrated in Figure 5.1(a) for a 1d continuous space. The basic problem is that the mode is apoint of measure zero, whereas the mean and median take the volume of the space into account.Another example is shown in Figure 5.1(b): here the mode is 0, but the mean is non-zero. Suchskewed distributions often arise when inferring variance parameters, especially in hierarchicalmodels. In such cases the MAP estimate (and hence the MLE) is obviously a very bad estimate.
How should we summarize a posterior if the mode is not a good choice? The answer is to
use decision theory, which we discuss in Section 5.7. The basic idea is to specify a loss function,whereL(θ,ˆθ)is the loss you incur if the truth is θand your estimate is ˆθ. If we use 0-1 loss,

L(θ,ˆθ)=I(θ̸=ˆθ), then the optimal estimate is the posterior mode. 0-1 loss means you only
get “points” if you make no errors, otherwise you get nothing: there is no “partial credit” under

5.2. Summarizing posterior distributions 151
0 2 4 6 8 10 1200.10.20.30.40.50.60.70.80.91
pXpYg
Figure 5.2 Example of the transformation of a density under a nonlinear transform. Note how the mode
of the transformed distribution is not the transform of the original mode. Based on Exercise 1.4 of (Bishop
2006b). Figure generated by bayesChangeOfVar .
this loss function! For continuous-valued quantities, we often prefer to use squared error loss,
L(θ,ˆθ)=(θ−ˆθ)2; the corresponding optimal estimator is then the posterior mean, as we show
in Section 5.7. Or we can use a more robust loss function, L(θ,ˆθ)=|θ−ˆθ|, which gives rise to
the posterior median.
5.2.1.4 MAP estimation is not invariant to reparameterization *

A more subtle problem with MAP estimation is that the result we get depends on how we pa-rameterize the probability distribution. Changing from one representation to another equivalentrepresentation changes the result, which is not very desirable, since the units of measurementare arbitrary (e.g., when measuring distance, we can use centimetres or inches).
To understand the problem, suppose we compute the posterior for x. If we deﬁne y=f(x),
the distribution for yis given by Equation 2.87, which we repeat here for convenience:
p
y(y)=px(x)⏐⏐dx
dy⏐⏐ (5.1)
The|dx

p
dy⏐⏐ (5.1)
The|dx
dy|term is called the Jacobian, and it measures the change in size of a unit volume passed
throughf.L e tˆx= argmaxxpx(x)be the MAP estimate for x. In general it is not the case
thatˆy= argmaxypy(y)is given by f(ˆx). For example, let x∼N(6,1)andy=f(x),w h e r e
f(x)=1
1+exp(−x+5)(5.2)
We can derive the distribution of yusing Monte Carlo simulation (see Section 2.7.1). The result
is shown in Figure 5.2. We see that the original Gaussian has become “squashed” by the sigmoid
nonlinearity. In particular, we see that the mode of the transformed distribution is not equal tothe transform of the original mode.

152 Chapter 5. Bayesian statistics
To see how this problem arises in the context of MAP estimation, consider the following
example, due to Michael Jordan. The Bernoulli distribution is typically parameterized by its
meanμ,s op(y=1|μ)=μ,w h e r ey∈{0,1}. Suppose we have a uniform prior on the
unit interval: pμ(μ)=1I(0≤μ≤1). If there is no data, the MAP estimate is just the
mode of the prior, which can be anywhere between 0 and 1. We will now show that differentparameterizations can pick different points in this interval arbitrarily.
First letθ=√
μsoμ=θ2. The new prior is
pθ(θ)=pμ(μ)⏐⏐dμ
dθ⏐⏐=2θ (5.3)
forθ∈[0,1]so the new mode is
ˆθ
MAP=a r gm a x
θ∈[0,1]2θ=1 (5.4)
Now letφ=1−√1−μ. The new prior is
pφ(φ)=pμ(μ)⏐⏐dμ
dφ⏐⏐=2 ( 1−φ) (5.5)
forφ∈[0,1], so the new mode is
ˆφ
φ∈[0,1]2−2φ=0 (5.6)
Thus the MAP estimate depends on the parameterization. The MLE does not suffer from this

ˆθ
ˆφ
since the likelihood is a function, not a probability density. Bayesian inference does not sufferfrom this problem either, since the change of measure is taken into account when integratingover the parameter space.
One solution to the problem is to optimize the following objective function:
ˆθ=a r g m a x
θp(D|θ)p(θ)|I(θ)|−1
2 (5.7)
HereI(θ)is the Fisher information matrix associated with p(x|θ)(see Section 6.2.2). This
estimate is parameterization independent, for reasons explained in (Jermyn 2005; Druilhet andMarin 2007). Unfortunately, optimizing Equation 5.7 is often difficult, which minimizes theappeal of the whole approach.

ˆθ=a r g m a x
2 (5.7)
5.2.2 Credible intervals
In addition to point estimates, we often want a measure of conﬁdence. A standard measure ofconﬁdence in some (scalar) quantity θis the “width” of its posterior distribution. This can be
measured using a 100(1−α)%credible interval, which is a (contiguous) region C=(ℓ,u)
(standing for lower and upper) which contains 1−αof the posterior probability mass, i.e.,
C
α(D)=(ℓ,u):P(ℓ≤θ≤u|D)=1−α (5.8)
There may be many such intervals, so we choose one such that there is (1−α)/2mass in each
tail; this is called a central interval.

5.2. Summarizing posterior distributions 153
0 0.2 0.4 0.6 0.8 100.511.522.533.5
(a)0 0.2 0.4 0.6 0.8 100.511.522.533.5
(b)
Figure 5.3 (a) Central interval and (b) HPD region for a Beta(3,9) posterior. The CI is (0.06, 0.52) and the
HPD is (0.04, 0.48). Based on Figure 3.6 of (Hoff 2009). Figure generated by betaHPD.
If the posterior has a known functional form, we can compute the posterior central interval
usingℓ=F−1(α/2)andu=F−1(1−α/2),w h e r eFis the cdf of the posterior. For example, if
the posterior is Gaussian, p(θ|D)=N(0,1), andα=0.05, then we have ℓ=Φ (α/2) =−1.96,
andu=Φ ( 1−α/2) = 1.96,w h e r e Φdenotes the cdf of the Gaussian. This is illustrated in
Figure 2.3(c). This justiﬁes the common practice of quoting a credible interval in the form of
μ±2σ,w h e r eμrepresents the posterior mean, σrepresents the posterior standard deviation,
and 2 is a good approximation to 1.96.
Of course, the posterior is not always Gaussian. For example, in our coin example, if we

use a uniform prior and we observe N1=4 7heads out of N= 100trials, then the posterior
is a beta distribution, p(θ|D) = Beta(48, 54). We ﬁnd the 95% posterior credible interval is
(0.3749,0.5673)(seebetaCredibleInt for the one line of Matlab code we used to compute
this).
If we don’t know the functional form, but we can draw samples from the posterior, then we
can use a Monte Carlo approximation to the posterior quantiles: we simply sort the Ssamples,

this).
and ﬁnd the one that occurs at location α/Salong the sorted list. As S→∞, this converges
to the true quantile. See mcQuantileDemo for a demo.
People often confuse Bayesian credible intervals with frequentist conﬁdence intervals. How-

this).
ever, they are not the same thing, as we discuss in Section 6.6.1. In general, credible intervals areusually what people want to compute, but conﬁdence intervals are usually what they actuallycompute, because most people are taught frequentist statistics but not Bayesian statistics. Fortu-nately, the mechanics of computing a credible interval is just as easy as computing a conﬁdenceinterval (see e.g., betaCredibleInt for how to do it in Matlab).
5.2.2.1 Highest posterior density regions *

A problem with central intervals is that there might be points outside the CI which have higherprobability density. This is illustrated in Figure 5.3(a), where we see that points outside theleft-most CI boundary have higher density than those just inside the right-most CI boundary.
This motivates an alternative quantity known as the highest posterior density orHPDregion.
This is deﬁned as the (set of) most probable points that in total constitute 100(1−α)% of the

154 Chapter 5. Bayesian statistics
α/2 α/2
(a)pMIN
(b)
Figure 5.4 (a) Central interval and (b) HPD region for a hypothetical multimodal posterior. Based on
Figure 2.2 of (Gelman et al. 2004). Figure generated by postDensityIntervals .
probability mass. More formally, we ﬁnd the threshold p∗on the pdf such that
1−α=∫
θ:p(θ|D)>p∗p(θ|D)dθ (5.9)
and then deﬁne the HPD as
Cα(D)={θ:p(θ|D)≥p∗} (5.10)
In 1d, the HPD region is sometimes called a highest density interval orHDI. For example,
Figure 5.3(b) shows the 95% HDI of a Beta(3,9)distribution, which is (0.04,0.48). We see that
this is narrower than the CI, even though it still contains 95% of the mass; furthermore, every
point inside of it has higher density than every point outside of it.
For a unimodal distribution, the HDI will be the narrowest interval around the mode contain-

ing 95% of the mass. To see this, imagine “water ﬁlling” in reverse, where we lower the leveluntil 95% of the mass is revealed, and only 5% is submerged. This gives a simple algorithm forcomputing HDIs in the 1d case: simply search over points such that the interval contains 95%of the mass and has minimal width. This can be done by 1d numerical optimization if we knowthe inverse CDF of the distribution, or by search over the sorted data points if we have a bag ofsamples (see betaHPD for a demo).

If the posterior is multimodal, the HDI may not even be a connected region: see Figure 5.4(b)
for an example. However, summarizing multimodal posteriors is always difficult.
5.2.3 Inference for a difference in proportions

Sometimes we have multiple parameters, and we are interested in computing the posteriordistribution of some function of these parameters. For example, suppose you are about to buysomething from Amazon.com, and there are two sellers offering it for the same price. Seller 1has 90 positive reviews and 10 negative reviews. Seller 2 has 2 positive reviews and 0 negativereviews. Who should you buy from?
1
1. This example is from www.johndcook .com/blog/2011/09/27/bayesian-amazon . See also lingpipe-blog .c
om/2009/10/13/bayesian-counterpart-to-fisher-exact-test-on-contingency-tables .

5.3. Bayesian model selection 155
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 102468101214
  
p(θ1|data)
p(θ2|data)
(a)−0.4 −0.2 0 0.2 0.4 0.6 0.8 100.511.522.5
δpdf
(b)
Figure 5.5 (a) Exact posteriors p(θi|Di). (b) Monte Carlo approximation to p(δ|D). We use kernel density
estimation to get a smooth plot. The vertical lines enclose the 95% central interval. Figure generated by
amazonSellerDemo ,
On the face of it, you should pick seller 2, but we cannot be very conﬁdent that seller 2 is
better since it has had so few reviews. In this section, we sketch a Bayesian analysis of this
problem. Similar methodology can be used to compare rates or proportions across groups for avariety of other settings.
Letθ
1andθ2be the unknown reliabilities of the two sellers. Since we don’t know much
about them, we’ll endow them both with uniform priors, θi∼Beta(1,1). The posteriors are
p(θ1|D1) = Beta(91, 11)andp(θ2|D2) = Beta(3 ,1).
We want to compute p(θ1>θ2|D). For convenience, let us deﬁne δ=θ1−θ2as the

Letθ
difference in the rates. (Alternatively we might want to work in terms of the log-odds ratio.) Wecan compute the desired quantity using numerical integration:
p(δ>0|D)=∫
1
0∫1
0I(θ1>θ2)Beta(θ1|y1+1,N1−y1+1)
Beta(θ2|y2+1,N2−y2+1)dθ1dθ2 (5.11)
We ﬁnd p(δ>0|D)=0.710, which means you are better off buying from seller 1! See
amazonSellerDemo for the code. (It is also possible to solve the integral analytically (Cook
2005).)
A simpler way to solve the problem is to approximate the posterior p(δ|D)by Monte Carlo

1
0∫1
2005).)
sampling. This is easy, since θ1andθ2are independent in the posterior, and both have beta
distributions, which can be sampled from using standard methods. The distributions p(θi|Di)
are shown in Figure 5.5(a), and a MC approximation to p(δ|D), together with a 95% HPD, is
shown Figure 5.5(b). An MC approximation to p(δ>0|D)is obtained by counting the fraction
of samples where θ1>θ2; this turns out to be 0.718, which is very close to the exact value. (See
amazonSellerDemo for the code.)

5.3 Bayesian model selection
In Figure 1.18, we saw that using too high a degree polynomial results in overﬁtting, and usingtoo low a degree results in underﬁtting. Similarly, in Figure 7.8(a), we saw that using too small

156 Chapter 5. Bayesian statistics
a regularization parameter results in overﬁtting, and too large a value results in underﬁtting. In
general, when faced with a set of models (i.e., families of parametric distributions) of differentcomplexity, how should we choose the best one? This is called the model selection problem.
One approach is to use cross-validation to estimate the generalization error of all the candiate
models, and then to pick the model that seems the best. However, this requires ﬁtting eachmodelKtimes, where Kis the number of CV folds. A more efficient approach is to compute
the posterior over models,
p(m|D)=p(D|m)p(m)
∑
m∈Mp(m,D)(5.12)
From this, we can easily compute the MAP model, ˆm=a r g m a x p(m|D). This is called
Bayesian model selection.
If we use a uniform prior over models, p(m)∝1, this amounts to picking the model which
maximizes
p(D|m)=∫
p(D|θ)p(θ|m)dθ (5.13)
This quantity is called the marginal likelihood, the integrated likelihood, or the evidence for

∑
maximizes
p(D|m)=∫
modelm. The details on how to perform this integral will be discussed in Section 5.3.2. But
ﬁrst we give an intuitive interpretation of what this quantity means.
5.3.1 Bayesian Occam’s razor
One might think that using p(D|m)to select models would always favor the model with the
most parameters. This is true if we use p(D|ˆθm)to select models, where ˆθmis the MLE or
MAP estimate of the parameters for model m, because models with more parameters will ﬁt the

data better, and hence achieve higher likelihood. However, if we integrate out the parameters,rather than maximizing them, we are automatically protected from overﬁtting: models withmore parameters do not necessarily have higher marginal likelihood. This is called the Bayesian
Occam’s razor effect (MacKay 1995b; Murray and Ghahramani 2005), named after the principle
known as Occam’s razor, which says one should pick the simplest model that adequately
explains the data.

One way to understand the Bayesian Occam’s razor is to notice that the marginal likelihood
can be rewritten as follows, based on the chain rule of probability (Equation 2.5):
p(D)=p(y
1)p(y2|y1)p(y3|y1:2)...p(yN|y1:N−1) (5.14)
where we have dropped the conditioning on xfor brevity. This is similar to a leave-one-out

p(D)=p(y
cross-validation estimate (Section 1.4.8) of the likelihood, since we predict each future point givenall the previous ones. (Of course, the order of the data does not matter in the above expression.)If a model is too complex, it will overﬁt the “early” examples and will then predict the remainingones poorly.
Another way to understand the Bayesian Occam’s razor effect is to note that probabilities must
sum to one. Hence∑
D′p(D′|m)=1, where the sum is over all possible data sets. Complex

models, which can predict many things, must spread their probability mass thinly, and hencewill not obtain as large a probability for any given data set as simpler models. This is sometimes

5.3. Bayesian model selection 157
Figure 5.6 A schematic illustration of the Bayesian Occam’s razor. The broad (green) curve corresponds
to a complex model, the narrow (blue) curve to a simple model, and the middle (red) curve is just right.
Based on Figure 3.13 of (Bishop 2006a). See also (Murray and Ghahramani 2005, Figure 2) for a similar plot
produced on real data.
called the conservation of probability mass principle, and is illustrated in Figure 5.6. On the
horizontal axis we plot all possible data sets in order of increasing complexity (measured in
some abstract sense). On the vertical axis we plot the predictions of 3 possible models: a simple
one,M1; a medium one, M2; and a complex one, M3. We also indicate the actually observed
dataD0by a vertical line. Model 1 is too simple and assigns low probability to D0. Model 3
also assigns D0relatively low probability, because it can predict many data sets, and hence it

spreads its probability quite widely and thinly. Model 2 is “just right”: it predicts the observed
data with a reasonable degree of conﬁdence, but does not predict too many other things. Hence
model 2 is the most probable model.
As a concrete example of the Bayesian Occam’s razor, consider the data in Figure 5.7. We plot
polynomials of degrees 1, 2 and 3 ﬁt to N=5data points. It also shows the posterior over
models, where we use a Gaussian prior (see Section 7.6 for details). There is not enough data

to justify a complex model, so the MAP model is d=1. Figure 5.8 shows what happens when
N=3 0. Now it is clear that d=2is the right model (the data was in fact generated from a
quadratic).
As another example, Figure 7.8(c) plots logp(D|λ)vslog(λ), for the polynomial ridge regres-
sion model, where λranges over the same set of values used in the CV experiment. We see
that the maximum evidence occurs at roughly the same point as the minimum of the test MSE,
which also corresponds to the point chosen by CV.

When using the Bayesian approach, we are not restricted to evaluating the evidence at a
ﬁnite grid of values. Instead, we can use numerical optimization to ﬁnd λ∗=a r g m a xλp(D|λ).
This technique is called empirical Bayes ortype II maximum likelihood (see Section 5.6 for
details). An example is shown in Figure 7.8(b): we see that the curve has a similar shape to the
CV estimate, but it can be computed more efficiently.

158 Chapter 5. Bayesian statistics
−2 0 2 4 6 8 10 12−20−10010203040506070d=1, logev=−18.593, EB
(a)−2 0 2 4 6 8 10 12−80−60−40−20020406080d=2, logev=−20.218, EB
(b)
−2 0 2 4 6 8 10 12−200−150−100−50050100150200250300d=3, logev=−21.718, EB
(c)1 2 300.20.40.60.81
MP(M|D)N=5, method=EB
(d)
Figure 5.7 (a-c) We plot polynomials of degrees 1, 2 and 3 ﬁt to N=5data points using empirical
Bayes. The solid green curve is the true function, the dashed red curve is the prediction (dotted blue lines
represent ±σaround the mean). (d) We plot the posterior over models, p(d|D), assuming a uniform prior
p(d)∝1. Based on a ﬁgure by Zoubin Ghahramani. Figure generated by linregEbModelSelVsN .
5.3.2 Computing the marginal likelihood (evidence)
When discussing parameter inference for a ﬁxed model, we often wrote
p(θ|D,m)∝p(θ|m)p(D|θ,m) (5.15)
thus ignoring the normalization constant p(D|m). This is valid since p(D|m)is constant wrt θ.
However, when comparing models, we need to know how to compute the marginal likelihood,

p(D|m). In general, this can be quite hard, since we have to integrate over all possible parameter
values, but when we have a conjugate prior, it is easy to compute, as we now show.
Letp(θ)=q(θ)/Z0be our prior, where q(θ)is an unnormalized distribution, and Z0is
the normalization constant of the prior. Let p(D|θ)=q(D|θ)/Zℓbe the likelihood, where Zℓ
contains any constant factors in the likelihood. Finally let p(θ|D)=q(θ|D)/ZNbe our poste-

5.3. Bayesian model selection 159
−2 0 2 4 6 8 10 12−10010203040506070d=1, logev=−106.110, EB
(a)−2 0 2 4 6 8 10 12−1001020304050607080d=2, logev=−103.025, EB
(b)
−2 0 2 4 6 8 10 12−20020406080100d=3, logev=−107.410, EB
MP(M|D)N=30, method=EB
(d)
Figure 5.8 Same as Figure 5.7 except now N=3 0. Figure generated by linregEbModelSelVsN .
rior, where q(θ|D)=q(D|θ)q(θ)is the unnormalized posterior, and ZNis the normalization
constant of the posterior. We have
p(θ|D)=p(D|θ)p(θ)
p(D)(5.16)
q(θ|D)
ZN=q(D|θ)q(θ)
ZℓZ0p(D)(5.17)
p(D)=ZN
Z0Zℓ(5.18)
So assuming the relevant normalization constants are tractable, we have an easy way to compute
the marginal likelihood. We give some examples below.

160 Chapter 5. Bayesian statistics
5.3.2.1 Beta-binomial model
LetusapplytheaboveresulttotheBeta-binomialmodel. Sinceweknow p(θ|D)=B e t a ( θ|a′,b′),
wherea′=a+N1andb′=b+N0, we know the normalization constant of the posterior is
B(a′,b′). Hence
p(D)(5.19)
=1
p(D)[1
B(a,b)θa−1(1−θ)b−1][(N
N1)
θN1(1−θ)N0]
(5.20)
=(N
N1)1
p(D)1
B(a,b)[
θa+N 1−1(1−θ)b+N 0−1]
(5.21)
So
1
B(a+N1,b+N0)=(N
N1)1
p(D)1
B(a,b)(5.22)
p(D)=(N
N1)B(a+N1,b+N0)
B(a,b)(5.23)
The marginal likelihood for the Beta-Bernoulli model is the same as above, except it is missing
the(N
N1)
term.
5.3.2.2 Dirichlet-multinoulli model
By the same reasoning as the Beta-Bernoulli case, one can show that the marginal likelihood forthe Dirichlet-multinoulli model is given by
p(D)=B(N+α)
B(α)(5.24)
where
B(α)=∏K
k=1Γ(αk)
Γ(∑
kαk)(5.25)
Hence we can rewrite the above result in the following form, which is what is usually presented
in the literature:
p(D)=Γ(∑
kαk)
Γ(N+∑
kαk)∏
kΓ(Nk+αk)
Γ(αk)(5.26)

the(N
N1)
term.
where
B(α)=∏K
k=1Γ(αk)
Γ(∑
p(D)=Γ(∑
kαk)
Γ(N+∑
kαk)∏
kΓ(Nk+αk)
We will see many applications of this equation later.
5.3.2.3 Gaussian-Gaussian-Wishart model
Consider the case of an MVN with a conjugate NIW prior. Let Z0be the normalizer for the
prior,ZNbe normalizer for the posterior, and let Zl=( 2π)ND/2be the normalizer for the

5.3. Bayesian model selection 161
likelihood. Then it is easy to see that
p(D)=ZN
Z0Zl(5.27)
=1
πND/21
2ND/2(
2π
κN)D/2
|SN|−νN/22(ν0+N)D/2ΓD(νN/2)
(
2π
κ0)D/2
|S0|−ν0/22ν0D/2ΓD(ν0/2)(5.28)
=1
πND/2(κ0
κN)D/2|S0|ν0/2
|SN|νN/2ΓD(νN/2)
ΓD(ν0/2)(5.29)
This equation will prove useful later.
5.3.2.4 BIC approximation to log marginal likelihood
In general, computing the integral in Equation 5.13 can be quite difficult. One simple but popular
approximation is known as the Bayesian information criterion orBIC, which has the following
form (Schwarz 1978):
BIC≜logp(D|ˆθ)−dof(ˆθ)
2logN≈logp(D) (5.30)
wheredof(ˆθ)is the number of degrees of freedom in the model, and ˆθis the MLE for the
model.2We see that this has the form of a penalized log likelihood, where the penalty term
depends on the model’s complexity. See Section 8.4.2 for the derivation of the BIC score.
As an example, consider linear regression. As we show in Section 7.3, the MLE is given by ˆw=
(XTX)−1XTyandˆσ2=R S S/N,w h e r eRSS =∑N
i=1(yi−ˆwT

mlexi)2. The corresponding
log likelihood is given by
logp(D|ˆθ)=−N
2log(2πˆσ2)−N
2(5.31)
Hence the BIC score is as follows (dropping constant terms)
BIC=−N
2log(ˆσ2)−D
2log(N) (5.32)
whereDis the number of variables in the model. In the statistics literature, it is common to
use an alternative deﬁnition of BIC, which we call the BIC cost(since we want to minimize it):
BIC-cost ≜−2logp(D|ˆθ)+dof(ˆθ)logN≈−2logp(D) (5.33)
In the context of linear regression, this becomes
BIC-cost=Nlog(ˆσ2)+Dlog(N) (5.34)

2(5.31)
BIC=−N
2. Traditionally the BIC score is deﬁned using the ML estimate ˆθ, so it is independent of the prior. However, for models
such as mixtures of Gaussians, the ML estimate can be poorly behaved, so it is better to evaluate the BIC score using
the MAP estimate, as in (Fraley and Raftery 2007).

162 Chapter 5. Bayesian statistics
The BIC method is very closely related to the minimum description length orMDLprinciple,
which characterizes the score for a model in terms of how well it ﬁts the data, minus how
complex the model is to deﬁne. See (Hansen and Yu 2001) for details.
There is a very similar expression to BIC/ MDL called the Akaike information criterion or
AIC, deﬁned as
AIC(m,D)≜logp(D|ˆθMLE)−dof(m) (5.35)
This is derived from a frequentist framework, and cannot be interpreted as an approximationto the marginal likelihood. Nevertheless, the form of this expression is very similar to BIC. Wesee that the penalty for AIC is less than for BIC. This causes AIC to pick more complex models.However, this can result in better predictive accuracy. See e.g., (Clarke et al. 2009, sec 10.2) forfurther discussion on such information criteria.
5.3.2.5 Effect of the prior

Sometimes it is not clear how to set the prior. When we are performing posterior inference, thedetails of the prior may not matter too much, since the likelihood often overwhelms the prioranyway. But when computing the marginal likelihood, the prior plays a much more importantrole, since we are averaging the likelihood over all possible parameter settings, as weighted bythe prior.
In Figures 5.7 and 5.8, where we demonstrated model selection for linear regression, we used
a prior of the form p(w)=N(0,α

−1I).H e r eαis a tuning parameter that controls how strong
the prior is. This parameter can have a large effect, as we discuss in Section 7.5. Intuitively, ifαis large, the weights are “forced” to be small, so we need to use a complex model with many
small parameters (e.g., a high degree polynomial) to ﬁt the data. Conversely, if αis small, we
will favor simpler models, since each parameter is “allowed” to vary in magnitude by a lot.

If the prior is unknown, the correct Bayesian procedure is to put a prior on the prior. That is,
we should put a prior on the hyper-parameter αas well as the parametrs w. To compute the
marginal likelihood, we should integrate out all unknowns, i.e., we should compute
p(D|m)=∫∫
p(D|w)p(w|α,m)p(α|m)dwdα (5.36)
Of course, this requires specifying the hyper-prior. Fortunately, the higher up we go in theBayesian hierarchy, the less sensitive are the results to the prior settings. So we can usuallymake the hyper-prior uninformative.

p(D|m)=∫∫
A computational shortcut is to optimize αrather than integrating it out. That is, we use
p(D|m)≈∫
p(D|w)p(w|ˆα,m)dw (5.37)
where
ˆα=a r g m a x
αp(D|α,m) = argmax
α∫
p(D|w)p(w|α,m)dw (5.38)
This approach is called empirical Bayes (EB), and is discussed in more detail in Section 5.6. Thisis the method used in Figures 5.7 and 5.8.

5.3. Bayesian model selection 163
Bayes factor BF(1,0) Interpretation
BF <1
100Decisive evidence for M0
BF <1
10Strong evidence for M0
1
10<BF<1
3Moderate evidence for M0
1
3<BF<1 Weak evidence for M0
1<BF<3 Weak evidence for M1
3<BF<10 Moderate evidence for M1
BF >10 Strong evidence for M1
BF >100 Decisive evidence for M1
Table 5.1 Jeffreys’ scale of evidence for interpreting Bayes factors.
5.3.3 Bayes factors
Suppose our prior on models is uniform, p(m)∝1. Then model selection is equivalent to
picking the model with the highest marginal likelihood. Now suppose we just have two models
we are considering, call them the null hypothesis, M0, and the alternative hypothesis, M1.
Deﬁne the Bayes factor as the ratio of marginal likelihoods:
BF1,0≜p(D|M1)
p(D|M0)=p(M1|D)
p(M0|D)/p(M1)
p(M0)(5.39)
(This is like a likelihood ratio, except we integrate out the parameters, which allows us to
compare models of different complexity.) If BF1,0>1then we prefer model 1, otherwise we
prefer model 0.

Of course, it might be that BF1,0is only slightly greater than 1. In that case, we are not
very conﬁdent that model 1 is better. Jeffreys (1961) proposed a scale of evidence for interpretingthe magnitude of a Bayes factor, which is shown in Table 5.1. This is a Bayesian alternative tothe frequentist concept of a p-value.
3Alternatively, we can just convert the Bayes factor to a
posterior over models. If p(M1)=p(M0)=0.5,w eh a v e
p(M0|D)=BF0,1
1+BF0,1=1
BF1,0+1(5.40)
5.3.3.1 Example: Testing if a coin is fair

1+BF0,1=1
Suppose we observe some coin tosses, and want to decide if the data was generated by a faircoin,θ=0.5, or a potentially biased coin, where θcould be any value in [0,1]. Let us denote
the ﬁrst model by M
0and the second model by M1. The marginal likelihood under M0is
simply
p(D|M0)=(1
2)N
(5.41)
3. Ap-value, is deﬁned as the probability (under the null hypothesis) of observing some test statistic f(D)(such as the
chi-squared statistic) that is as large or largerthan that actually observed, i.e., pvalue(D)≜P(f(˜D)≥f(D)|˜D∼

simply
2)N
(5.41)
H0). Note that has almost nothing to do with what we really want to know, which is p(H0|D).

164 Chapter 5. Bayesian statistics
01111122222222223333333333444445−1.8−1.6−1.4−1.2−1−0.8−0.6−0.4log10 p(D|M1)
(a)01111122222222223333333333444445−2.5−2.45−2.4−2.35−2.3−2.25−2.2−2.15−2.1−2.05−2BIC approximation to log10 p(D|M1)
(b)
Figure 5.9 (a) Log marginal likelihood for the coins example. (b) BIC approximation. Figure generated by
coinsModelSelDemo .
whereNis the number of coin tosses. The marginal likelihood under M1, using a Beta prior, is
p(D|M1)=∫
p(D|θ)p(θ)dθ=B(α1+N1,α0+N0)
B(α1,α0)(5.42)
We plotlogp(D|M1)vs the number of heads N1in Figure 5.9(a), assuming N=5and
α1=α0=1. (The shape of the curve is not very sensitive to α1andα0, as long as α0=α1.)
If we observe 2 or 3 heads, the unbiased coin hypothesis M0is more likely than M1, sinceM0
is a simpler model (it has no free parameters) — it would be a suspicious coincidence if the

p(D|M1)=∫
coin were biased but happened to produce almost exactly 50/50 heads/tails. However, as thecounts become more extreme, we favor the biased coin hypothesis. Note that, if we plot the logBayes factor, logBF
1,0, it will have exactly the same shape, since logp(D|M0)is a constant.
See also Exercise 3.18.
In Figure 5.9(b) shows the BIC approximation to logp(D|M1)for our biased coin example

from Section 5.3.3.1. We see that the curve has approximately the same shape as the exact logmarginal likelihood, which is all that matters for model selection purposes, since the absolutescale is irrelevant. In particular, it favors the simpler model unless the data is overwhelminglyin support of the more complex model.
5.3.4 Jeffreys-Lindley paradox *

Problems can arise when we use improper priors (i.e., priors that do not integrate to 1) for modelselection/ hypothesis testing, even though such priors may be acceptable for other purposes. Forexample, consider testing the hypotheses M
0:θ∈Θ0vsM1:θ∈Θ1. To deﬁne the marginal
density on θ, we use the following mixture model
p(θ)=p( θ|M0)p(M0)+p(θ|M1)p(M1) (5.43)

5.4. Priors 165
This is only meaningful if p(θ|M0)andp(θ|M1)are proper (normalized) density functions. In
p(M0|D)=p(M0)p(D|M0)
p(M0)p(D|M0)+p(M1)p(D|M1)(5.44)
=p(M0)∫
Θ0p(D|θ)p(θ|M0)dθ
p(M0)∫
Θ0p(D|θ)p(θ|M0)dθ+p(M1)∫
Θ1p(D|θ)p(θ|M1)dθ(5.45)
Now suppose we use improper priors, p(θ|M0)∝c0andp(θ|M1)∝c1. Then
p(M0|D)=p(M0)c0∫
Θ0p(D|θ)dθ
p(M0)c0∫
Θ0p(D|θ)dθ+p(M1)c1∫
Θ1p(D|θ)dθ(5.46)
=p(M0)c0ℓ0
p(M0)c0ℓ0+p(M1)c1ℓ1(5.47)
whereℓi=∫
Θip(D|θ)dθis the integrated or marginal likelihood for model i. Now let p(M0)=
p(M1)=1
2. Hence
p(M0|D)=c0ℓ0
c0ℓ0+c1ℓ1=ℓ0
ℓ0+(c1/c0)ℓ1(5.48)
Thus we can change the posterior arbitrarily by choosing c1andc0as we please. Note that
using proper, but very vague, priors can cause similar problems. In particular, the Bayes factor
will always favor the simpler model, since the probability of the observed data under a complexmodel with a very diffuse prior will be very small. This is called the Jeffreys-Lindley paradox.

p(M1)=1
2. Hence
Thus it is important to use proper priors when performing model selection. Note, however,
that, ifM
0andM1share the same prior over a subset of the parameters, this part of the prior
can be improper, since the corresponding normalization constant will cancel out.
5.4 Priors
The most controversial aspect of Bayesian statistics is its reliance on priors. Bayesians arguethis is unavoidable, since nobody is a tabula rasa orblank slate: all inference must be done

that, ifM
5.4 Priors
conditional on certain assumptions about the world. Nevertheless, one might be interested inminimizing the impact of one’s prior assumptions. We brieﬂy discuss some ways to do thisbelow.
5.4.1 Uninformative priors
If we don’t have strong beliefs about what θshould be, it is common to use an uninformative
ornon-informative prior, and to “let the data speak for itself”.
The issue of designing uninformative priors is actually somewhat tricky. As an example
of the difficulty, consider a Bernoulli parameter, θ∈[0,1]. One might think that the most

uninformative prior would be the uniform distribution, Beta(1,1). But the posterior mean in
this case is E[θ|D]=N1+1
N1+N 0+2, whereas the MLE isN1
N1+N 0. Hence one could argue that the
prior wasn’t completely uninformative after all.

166 Chapter 5. Bayesian statistics
Clearly by decreasing the magnitude of the pseudo counts, we can lessen the impact of the
prior. By the above argument, the most non-informative prior is
lim
c→0Beta(c,c) = Beta(0 ,0) (5.49)
which is a mixture of two equal point masses at 0 and 1 (see (Zhu and Lu 2004)). This is also
called the Haldane prior. Note that the Haldane prior is an improper prior, meaning it does not
integrate to 1. However, as long as we see at least one head and at least one tail, the posteriorwill be proper.
In Section 5.4.2.1 we will argue that the “right” uninformative prior is in fact Beta(
1
2,1
2).
Clearly the difference in practice between these three priors is very likely negligible. In general,it is advisable to perform some kind of sensitivity analysis, in which one checks how much

1
2,1
2).
one’s conclusions or predictions change in response to change in the modeling assumptions,which includes the choice of prior, but also the choice of likelihood and any kind of data pre-processing. If the conclusions are relatively insensitive to the modeling assumptions, one canhave more conﬁdence in the results.
5.4.2 Jeffreys priors *
Harold Jeffreys4designed a general purpose technique for creating non-informative priors. The
result is known as the Jeffreys prior. The key observation is that if p(φ)is non-informative,

then any re-parameterization of the prior, such as θ=h(φ)for some function h, should also
be non-informative. Now, by the change of variables formula,
pθ(θ)=pφ(φ)⏐⏐dφ
dθ⏐⏐ (5.50)
so the prior will in general change. However, let us pick
p
φ(φ)∝(I(φ))1
2 (5.51)
whereI(φ)is theFisher information:
I(φ)≜−E[(dlogp(X|φ)
dφ)
2]
(5.52)
This is a measure of curvature of the expected negative log likelihood and hence a measure of
stability of the MLE (see Section 6.2.2). Now
dlogp(x|θ)
dθ=dlogp(x|φ)
dφdφ
dθ(5.53)
Squaring and taking expectations over x,w eh a v e
I(θ)=−E[(dlogp(X|θ)
dθ)2]
=I(φ)(dφ
dθ)2
(5.54)
I(θ)1
2=I(φ)1
2⏐⏐dφ
dθ⏐⏐ (5.55)
4. Harold Jeffreys, 1891 – 1989, was an English mathematician, statistician, geophysicist, and astronomer.

5.4. Priors 167
so we ﬁnd the transformed prior is
pθ(θ)=p φ(φ)⏐⏐dφ
dθ⏐⏐∝(I(φ))1
2⏐⏐dφ
dθ⏐⏐=I(θ)1
2 (5.56)
Sopθ(θ)andpφ(φ)are the same.
Some examples will make this clearer.
5.4.2.1 Example: Jeffreys prior for the Bernoulli and multinoulli
SupposeX∼Ber(θ). The log likelihood for a single sample is
logp(X|θ)=Xlogθ+(1−X)log(1−θ) (5.57)
Thescore function is just the gradient of the log-likelihood:
s(θ)≜d
dθlogp(X|θ)=X
θ−1−X
1−θ(5.58)
Theobserved information is the second derivative of the log-likelihood:
J(θ)=−d2
dθ2logp(X|θ)=−s′(θ|X)=X
θ2+1−X
(1−θ)2(5.59)
The Fisher information is the expected information:
I(θ)=E [J(θ|X)|X∼θ]=θ
θ2+1−θ
(1−θ)2=1
θ(1−θ)(5.60)
Hence Jeffreys’ prior is
p(θ)∝θ−1
2(1−θ)−1
2=1√
θ(1−θ)∝Beta(1
2,1
2) (5.61)
Now consider a multinoulli random variable with Kstates. One can show that the Jeffreys’
prior is given by
p(θ)∝Dir(1
2,...,1
2) (5.62)
Note that this is different from the more obvious choices of Dir(1
K,...,1
K)orDir(1,...,1).

J(θ)=−d2
θ2+1−X
θ2+1−θ
(1−θ)2=1
p(θ)∝θ−1
2(1−θ)−1
2=1√
2,1
2) (5.61)
2,...,1
2) (5.62)
K,...,1
5.4.2.2 Example: Jeffreys prior for location and scale parameters
One can show that the Jeffreys prior for a location parameter, such as the Gaussian mean, is
p(μ)∝1. Thus is an example of a translation invariant prior, which satisﬁes the property
that the probability mass assigned to any interval, [A,B]is the same as that assigned to any
other shifted interval of the same width, such as [A−c,B−c]. That is,
∫B−c
A−cp(μ)dμ=(A−c)−(B−c)=(A−B)=∫B
Ap(μ)dμ (5.63)

168 Chapter 5. Bayesian statistics
This can be achieved using p(μ)∝1, which we can approximate by using a Gaussian with
inﬁnite variance, p(μ)=N(μ|0,∞). Note that this is an improper prior, since it does not
integrate to 1. Using improper priors is ﬁne as long as the posterior is proper, which will be the
case provided we have seen N≥1data points, since we can “nail down” the location as soon
as we have seen a single data point.
Similarly, one can show that the Jeffreys prior for a scale parameter, such as the Gaussian
variance, is p(σ2)∝1/σ2. This is an example of a scale invariant prior, which satisﬁes the
property that the probability mass assigned to any interval [A,B]is the same as that assigned
to any other interval [A/c,B/c] which is scaled in size by some constant factor c>0.( F o r
example, if we change units from meters to feet we do not want that to affect our inferences.)This can be achieved by using
p(s)∝1/s (5.64)
To see this, note that
∫
B/c
A/cp(s)ds= [logs]B/c
A/c= log(B/c )−log(A/c) (5.65)

∫
B/c
= log(B )−log(A)=∫B
Ap(s)ds (5.66)
WecanapproximatethisusingadegenerateGammadistribution(Section2.4.4), p(s)=G a (s|0,0).
The prior p(s)∝1/sis also improper, but the posterior is proper as soon as we have seen
N≥2data points (since we need at least two data points to estimate a variance).
5.4.3 Robust priors
In many cases, we are not very conﬁdent in our prior, so we want to make sure it does not have
an undue inﬂuence on the result. This can be done by using robust priors (Insua and Ruggeri

2000), which typically have heavy tails, which avoids forcing things to be too close to the priormean.
Let us consider an example from (Berger 1985, p7). Suppose x∼N(θ,1). We observe that
x=5and we want to estimate θ. The MLE is of course ˆθ=5, which seems reasonable. The
posterior mean under a uniform prior is also
θ=5. But now suppose we know that the prior
median is 0, and the prior quantiles are at -1 and 1, so p(θ≤−1) =p(−1<θ≤0) =p(0<
θ≤1) =p(1<θ)=0.25. Let us also assume the prior is smooth and unimodal.

It is easy to show that a Gaussian prior of the form N(θ|0,2.192)satisﬁes these prior
constraints. But in this case the posterior mean is given by 3.43, which doesn’t seem verysatisfactory.
Now suppose we use as a Cauchy prior T(θ|0,1,1). This also satisﬁes the prior constraints of
our example. But this time we ﬁnd (using numerical method integration: see robustPriorDemo
for the code) that the posterior mean is about 4.6, which seems much more reasonable.
5.4.4 Mixtures of conjugate priors

Robust priors are useful, but can be computationally expensive to use. Conjugate priors simplifythe computation, but are often not robust, and not ﬂexible enough to encode our prior knowl-

5.4. Priors 169
edge. However, it turns out that a mixture of conjugate priors is also conjugate (Exercise 5.1),
and can approximate any kind of prior (Dallal and Hall 1983; Diaconis and Ylvisaker 1985). Thus
such priors provide a good compromise between computational convenience and ﬂexibility.
For example, suppose we are modeling coin tosses, and we think the coin is either fair, or
is biased towards heads. This cannot be represented by a beta distribution. However, we canmodel it using a mixture of two beta distributions. For example, we might use
p(θ)=0.5B e t a (θ|20,20)+0.5B e t a (θ|30,10) (5.67)
Ifθcomes from the ﬁrst distribution, the coin is fair, but if it comes from the second, it is
biased towards heads.
We can represent a mixture by introducing a latent indicator variable z,w h e r ez=kmeans
thatθcomes from mixture component k. The prior has the form
p(θ)=∑
kp(z=k)p(θ|z=k) (5.68)
where each p(θ|z=k)is conjugate, and p(z=k)are called the (prior) mixing weights. One can

p(θ)=∑
show (Exercise 5.1) that the posterior can also be written as a mixture of conjugate distributionsas follows:
p(θ|D)=∑
kp(z=k|D)p(θ|D,z=k) (5.69)
wherep(Z=k|D)are the posterior mixing weights given by
p(Z=k|D)=p(Z=k)p(D|Z=k)∑
k′p(Z=k′)p(D|Z=k′)(5.70)
Here the quantity p(D|Z=k)is the marginal likelihood for mixture component k(see Sec-
tion 5.3.2.1).
5.4.4.1 Example
Suppose we use the mixture prior
p(θ)=0.5Beta(θ|a1,b1)+0.5Beta(θ|a2,b2) (5.71)
wherea1=b1=2 0anda2=b2=1 0. and we observe N1heads and N0tails. The posterior
becomes

p(θ|D)=∑
becomes
p(θ|D)=p(Z=1|D)Beta(θ|a1+N1,b1+N0)+p(Z=2|D)Beta(θ|a2+N1,b2+N0)(5.72)
IfN1=2 0heads and N0=1 0tails, then, using Equation 5.23, the posterior becomes
p(θ|D)=0.346 Beta(θ |40,30)+0.654 Beta(θ |50,20) (5.73)
See Figure 5.10 for an illustration.

170 Chapter 5. Bayesian statistics
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 100.511.522.533.544.55mixture of Beta distributions
  
prior
posterior
Figure 5.10 A mixture of two Beta distributions. Figure generated by mixBetaDemo .
5.4.4.2 Application: Finding conserved regions in DNA and protein sequences
We mentioned that Dirichlet-multinomial models are widely used in biosequence analysis. Let
us give a simple example to illustrate some of the machinery that has developed. Speciﬁcally,consider the sequence logo discussed in Section 2.3.2.1. Now suppose we want to ﬁnd locationswhich represent coding regions of the genome. Such locations often have the same letter acrossall sequences, because of evolutionary pressure. So we need to ﬁnd columns which are “pure”,or nearly so, in the sense that they are mostly all As, mostly all Ts, mostly all Cs, or mostly allGs. One approach is to look for low-entropy columns; these will be ones whose distribution isnearly deterministic (pure).

But suppose we want to associate a conﬁdence measure with our estimates of purity. This
can be useful if we believe adjacent locations are conserved together. In this case, we can letZ
1=1if location tis conserved, and let Zt=0otherwise. We can then add a dependence
between adjacent Ztvariables using a Markov chain; see Chapter 17 for details.
In any case, we need to deﬁne a likelihood model, p(Nt|Zt),w h e r eNtis the vector of
(A,C,G,T) counts for column t. It is natural to make this be a multinomial distribution with
parameter θt. Since each column has a different distribution, we will want to integrate out θt
and thus compute the marginal likelihood
p(Nt|Zt)=∫
p(Nt|θt)p(θt|Zt)dθt (5.74)
But what prior should we use for θt? WhenZt=0we can use a uniform prior, p(θ|Zt=0 )=
Dir(1,1,1,1), but what should we use if Zt=1? After all, if the column is conserved, it could

be a (nearly) pure column of As, Cs, Gs, or Ts. A natural approach is to use a mixture of Dirichletpriors, each one of which is “tilted” towards the appropriate corner of the 4-dimensional simplex,e.g.,
p(θ|Z
t=1 )=1
4Dir(θ|(10,1,1,1))+···+1
4Dir(θ|(1,1,1,10)) (5.75)
Since this is conjugate, we can easily compute p(Nt|Zt). See (Brown et al. 1993) for an

5.5. Hierarchical Bayes 171
application of these ideas to a real bio-sequence problem.
5.5 Hierarchical Bayes
A key requirement for computing the posterior p(θ|D)is the speciﬁcation of a prior p(θ|η),
whereηare the hyper-parameters. What if we don’t know how to set η? In some cases, we can
use uninformative priors, we we discussed above. A more Bayesian approach is to put a prior on
our priors! In terms of graphical models (Chapter 10), we can represent the situation as follows:
η→θ→D (5.76)
This is an example of a hierarchical Bayesian model, also called a multi-level model, since
there are multiple levels of unknown quantities. We give a simple example below, and we willsee many others later in the book.
5.5.1 Example: modeling related cancer rates
Consider the problem of predicting cancer rates in various cities (this example is from (Johnsonand Albert 1999, p24)). In particular, suppose we measure the number of people in variouscities,N

i, and the number of people who died of cancer in these cities, xi. We assume
xi∼Bin(N i,θi), and we want to estimate the cancer rates θi. One approach is to estimate
them all separately, but this will suffer from the sparse data problem (underestimation of therate of cancer due to small N
i). Another approach is to assume all the θiare the same; this is
calledparameter tying. The resulting pooled MLE is just ˆθ=∑
ixi∑
iNi. But the assumption that

ixi∑
all the cities have the same rate is a rather strong one. A compromise approach is to assumethat theθ
iare similar, but that there may be city-speciﬁc variations. This can be modeled by
assuming the θiare drawn from some common distribution, say θi∼Beta(a,b). The full joint
distribution can be written as
p(D,θ,η|N)=p( η)N∏
i=1Bin(xi|Ni,θi)Beta(θi|η) (5.77)
whereη=(a,b).
Note that it is crucial that we infer η=(a,b)from the data; if we just clamp it to a constant,
theθiwill be conditionally independent, and there will be no information ﬂow between them.

By contrast, by treating ηas an unknown (hidden variable), we allow the data-poor cities to
borrow statistical strength from data-rich ones.
Suppose we compute the joint posterior p(η,θ|D). From this we can get the posterior
marginals p(θi|D). In Figure 5.11(a), we plot the posterior means, E[θi|D], as blue bars, as well
as the population level mean, E[a/(a+b)|D], shown as a red line (this represents the average
of theθi’s). We see that the posterior mean is shrunk towards the pooled estimate more strongly

for cities with small sample sizes Ni. For example, city 1 and city 20 both have a 0 observed
cancer incidence rate, but city 20 has a smaller population, so its rate is shrunk more towardsthe population-level estimate (i.e., it is closer to the horizontal red line) than city 1.
Figure 5.11(b) shows the 95% posterior credible intervals for θ
i. We see that city 15, which has
a very large population (53,637 people), has small posterior uncertainty. Consequently this city

172 Chapter 5. Bayesian statistics
0 5 10 15 20 2505number of people with cancer (truncated at 5)
0 5 10 15 20 25010002000pop of city (truncated at 2000)
0 5 10 15 20 250510MLE*1000 (red line=pooled MLE)
0 5 10 15 20 25024posterior mean*1000 (red line=pop mean)
(a)0 1 2 3 4 5 6 7 8
x 10−30246810121416182095% credible interval on theta, *=median
(b)
Figure 5.11 (a) Results of ﬁtting the model using the data from (Johnson and Albert 1999, p24). First
row: Number of cancer incidents xiin 20 cities in Missouri. Second row: population size Ni. The largest
city (number 15) has a population of N15= 53637 andx15=5 4incidents, but we truncate the vertical
axes of the ﬁrst two rows so that the differences between the other cities are visible. Third row: MLE ˆθi.
The red line is the pooled MLE. Fourth row: posterior mean E[θi|D]. The red line is E[a/(a+b)|D],
the population-level mean. (b) Posterior 95% credible intervals on the cancer rates. Figure generated by
cancerRatesEb

has the largest impact on the posterior estimate of η, which in turn will impact the estimate of
the cancer rates for other cities. Cities 10 and 19, which have the highest MLE, also have the
highest posterior uncertainty, reﬂecting the fact that such a high estimate is in conﬂict with theprior (which is estimated from all the other cities).
In the above example, we have one parameter per city, modeling the probability the response
is on. By making the Bernoulli rate parameter be a function of covariates, θ
i=s i g m (wT
ix),w e

i=s i g m (wT
ix),w e
can model multiple correlated logistic regression tasks. This is called multi-task learning, and
will be discussed in more detail in Section 9.5.
5.6 Empirical Bayes
In hierarchical Bayesian models, we need to compute the posterior on multiple levels of latentvariables. For example, in a two-level model, we need to compute
p(η,θ|D)∝p(D|θ)p(θ|η)p(η) (5.78)
In some cases, we can analytically marginalize out θ; this leaves is with the simpler problem of
just computing p(η|D).
As a computational shortcut, we can approximate the posterior on the hyper-parameters with

a point-estimate, p(η|D)≈δ
ˆη(η),w h e r eˆη=a r g m a x p(η|D). Sinceηis typically much
smaller than θin dimensionality, it is less prone to overﬁtting, so we can safely use a uniform
prior onη. Then the estimate becomes
ˆη=a r g m a x p(D|η) = argmax[∫
p(D|θ)p(θ|η)dθ]
(5.79)

5.6. Empirical Bayes 173
where the quantity inside the brackets is the marginal or integrated likelihood, sometimes called
the evidence. This overall approach is called empirical Bayes (EB)o rtype-II maximum
likelihood. In machine learning, it is sometimes called the evidence procedure.
Empirical Bayes violates the principle that the prior should be chosen independently of the
data. However, we can just view it as a computationally cheap approximation to inference in ahierarchical Bayesian model, just as we viewed MAP estimation as an approximation to inferencein the one level model θ→D. In fact, we can construct a hierarchy in which the more integrals
one performs, the “more Bayesian” one becomes:
Method Deﬁnition
Maximum likelihood ˆθ=a r g m a xθp(D|θ)
MAP estimation ˆθ=a r g m a xθp(D|θ)p(θ|η)
ML-II (Empirical Bayes) ˆη=a r g m a xη∫
p(D|θ)p(θ|η)dθ=a r g m a xηp(D|η)
MAP-II ˆη=a r g m a xη∫
p(D|θ)p(θ|η)p(η)dθ=a r g m a xηp(D|η)p(η)
Full Bayes p(θ,η|D)∝p(D|θ)p(θ|η)p(η)

Note that EB can be shown to have good frequentist properties (see e.g., (Carlin and Louis
1996; Efron 2010)), so it is widely used by non-Bayesians. For example, the popular James-Steinestimator, discussed in Section 6.3.3.2, can be derived using EB.
5.6.1 Example: beta-binomial model
Let us return to the cancer rates model. We can analytically integrate out the θi’s, and write
down the marginal likelihood directly, as follows:
p(D|a,b)=∏
i∫
Bin(xi|Ni,θi)Beta(θi|a,b)dθi (5.80)
=∏
iB(a+xi,b+Ni−xi)
B(a,b)(5.81)
Various ways of maximizing this wrt aandbare discussed in (Minka 2000e).
Having estimated aandb, we can plug in the hyper-parameters to compute the posterior

i∫
=∏
p(θi|ˆa,ˆb,D)in the usual way, using conjugate analysis. The net result is that the posterior
mean of each θiis a weighted average of its local MLE and the prior means, which depends on
η=(a,b); but since ηis estimated based on all the data, each θiis inﬂuenced by all the data.
5.6.2 Example: Gaussian-Gaussian model
We now study another example that is analogous to the cancer rates example, except the data isreal-valued. We will use a Gaussian likelihood and a Gaussian prior. This will allow us to writedown the solution analytically.

In particular, suppose we have data from multiple related groups. For example, x
ijcould be
the test score for student iin school j,f o rj=1:D andi=1:N j. We want to estimate
the mean score for each school, θj. However, since the sample size, Nj, may be small for

174 Chapter 5. Bayesian statistics
some schools, we can regularize the problem by using a hierarchical Bayesian model, where we
assumeθjcome from a common prior, N(μ,τ2).
The joint distribution has the following form:
p(θ,D|η,σ2)=D∏
j=1N(θj|μ,τ2)Nj∏
i=1N(xij|θj,σ2) (5.82)
where we assume σ2is known for simplicity. (We relax this assumption in Exercise 24.4.) We
explain how to estimate ηbelow. Once we have estimated η=(μ,τ), we can compute the
posteriors over the θj’s. To do that, it simpliﬁes matters to rewrite the joint distribution in the
following form, exploiting the fact that NjGaussian measurements with values xijand variance
σ2are equivalent to one measurement of value xj≜1
Nj∑Nj
i=1xijwith variance σ2
j≜σ2/Nj.
This yields
p(θ,D|ˆη,σ2)=D∏
j=1N(θj|ˆμ,ˆτ2)N(xj|θj,σ2
j) (5.83)
From this, it follows from the results of Section 4.4.1 that the posteriors are given by
p(θj|D,ˆμ,ˆτ2)=N (θj|ˆBjˆμ+(1−ˆBj)xj,(1−ˆBj)σ2
j) (5.84)
ˆBj≜σ2
j
σ2
j+ˆτ2(5.85)
whereˆμ=xandˆτ2will be deﬁned below.

Nj∑Nj
j≜σ2/Nj.
j) (5.83)
j) (5.84)
ˆBj≜σ2
j
σ2
The quantity 0≤ˆBj≤1controls the degree of shrinkage towards the overall mean, μ.I f
the data is reliable for group j(e.g., because the sample size Njis large), then σ2
jwill be small
relative to τ2; henceˆBjwill be small, and we will put more weight on xjwhen we estimate θj.
However, groups with small sample sizes will get regularized (shrunk towards the overall meanμ) more heavily. We will see an example of this below.
Ifσ
j=σfor all groups j, the posterior mean becomes
ˆθj=ˆBx+(1−ˆB)xj=x+(1−ˆB)(xj−x) (5.86)

Ifσ
This has exactly the same form as the James Stein estimator discussed in Section 6.3.3.2.
5.6.2.1 Example: predicting baseball scores
We now give an example of shrinkage applied to baseball batting averages, from (Efron andMorris 1975). We observe the number of hits for D=1 8players during the ﬁrst T=4 5games.
Call the number of hits b
i. We assume bj∼Bin(T,θ j),w h e r eθjis the “true” batting average
for player j. The goal is to estimate the θj. The MLE is of course ˆθj=xj,w h e r exj=bj/Tis
the empirical batting average. However, we can use an EB approach to do better.

To apply the Gaussian shrinkage approach described above, we require that the likelihood be
Gaussian, xj∼N(θj,σ2)for known σ2. (We drop the isubscript since we assume Nj=1,

5.6. Empirical Bayes 175
0.2 0.25 0.3 0.35 0.400.20.40.60.81MLE (top) and shrinkage estimates (bottom)
(a)1 2 3 4 500.050.10.150.20.250.30.350.4
player numberMSEMSE MLE = 0.0042, MSE shrunk = 0.0013
true
shrunk
MLE
(b)
Figure 5.12 (a) MLE parameters (top) and corresponding shrunken estimates (bottom). (b) We plot the
true parameters (blue), the posterior mean estimate (green), and the MLEs (red) for 5 of the players. Figure
generated by shrinkageDemoBaseball .
sincexjalready represents the average for player j.) However, in this example we have a
binomial likelihood. While this has the right mean, E[xj]=θj, the variance is not constant:
var[xj]=1
T2var[bj]=Tθj(1−θj)
T2(5.87)
So let us apply a variance stabilizing transform5toxjto better match the Gaussian assump-
tion:
yj=f(yj)=√
Tarcsin(2y j−1) (5.88)
Now we have approximately yj∼N(f(θj),1) =N(μj,1). We use Gaussian shrinkage to
estimate the μjusing Equation 5.86 with σ2=1, and we then transform back to get
ˆθj=0.5(sin(ˆμj/√
T)+1) (5.89)

var[xj]=1
T2(5.87)
tion:
The results are shown in Figure 5.12(a-b). In (a), we plot the MLE ˆθjand the posterior mean θj.
We see that all the estimates have shrunk towards the global mean, 0.265. In (b), we plot the
true value θj, the MLE ˆθjand the posterior mean θj. (The “true” values of θjare estimated
from a large number of independent games.) We see that, on average, the shrunken estimate
is much closer to the true parameters than the MLE is. Speciﬁcally, the mean squared error,
deﬁned by MSE =1
N∑D
j=1(θj−θj)2, is over three times smaller using the shrinkage estimates

N∑D
θjthan using the MLEs ˆθj.
5.6.2.2 Estimating the hyper-parameters
In this section, we give an algorithm for estimating η. Suppose initially that σ2
j=σ2is the
same for all groups. In this case, we can derive the EB estimate in closed form, as we now show.
From Equation 4.126, we have
p(xj|μ,τ2,σ2)=∫
N(xj|θj,σ2)N(θj|μ,τ2)dθj=N(xj|μ,τ2+σ2) (5.90)
5. Suppose E[X]=μandvar[X]=σ2(μ).L e t Y=f(X). Then a Taylor series expansions gives Y≈
f(μ)+(X−μ)f′(μ). Hence var[Y]≈f′(μ)2var[X−μ]=f′(μ)2σ2(μ). A variance stabilizing transformation

j=σ2is the
is a function fsuch that f′(μ)2σ2(μ)is independent of μ.

176 Chapter 5. Bayesian statistics
Hence the marginal likelihood is
p(D|μ,τ2,σ2)=D∏
j=1N(xj|μ,τ2+σ2) (5.91)
Thus we can estimate the hyper-parameters using the usual MLEs for a Gaussian. For μ,w e
have
ˆμ=1
DD∑
j=1xj=x (5.92)
which is the overall mean.
For the variance, we can use moment matching (which is equivalent to the MLE for a
Gaussian): we simply equate the model variance to the empirical variance:
ˆτ2+σ2=1
DD∑
j=1(xj−x)2≜s2(5.93)
soˆτ2=s2−σ2. Since we know τ2must be positive, it is common to use the following revised
estimate:
ˆτ2=m a x{0,s2−σ2}=(s2−σ2)+ (5.94)
Hence the shrinkage factor is
ˆB=σ2
σ2+ˆτ2=σ2
σ2+(s2−σ2)+(5.95)
In the case where the σ2
j’s are different, we can no longer derive a solution in closed form.
Exercise 11.13 discusses how to use the EM algorithm to derive an EB estimate, and Exercise 24.4
discusses how to perform full Bayesian inference in this hierarchical model.
5.7 Bayesian decision theory

estimate:
ˆB=σ2
σ2+ˆτ2=σ2
We have seen how probability theory can be used to represent and updates our beliefs aboutthe state of the world. However, ultimately our goal is to convert our beliefs into actions. In thissection, we discuss the optimal way to do this.
We can formalize any given statistical decision problem as a game against nature (as opposed

to a game against other strategic players, which is the topic of game theory, see e.g., (Shohamand Leyton-Brown 2009) for details). In this game, nature picks a state or parameter or label,y∈Y, unknown to us, and then generates an observation, x∈X, which we get to see. We
then have to make a decision, that is, we have to choose an action afrom some action space
A. Finally we incur some loss,L(y,a), which measures how compatible our action ais with
nature’s hidden state y. For example, we might use misclassiﬁcation loss, L(y,a)=I(y̸=a),
or squared loss, L(y,a)=(y−a)

2. We will see some other examples below.

5.7. Bayesian decision theory 177
Our goal is to devise a decision procedure orpolicy,δ:X→A, which speciﬁes the
optimal action for each possible input. By optimal, we mean the action that minimizes the
expected loss:
δ(x) = argmin
a∈AE[L(y,a)] (5.96)
In economics, it is more common to talk of a utility function; this is just negative loss,
U(y,a)=−L(y,a). Thus the above rule becomes
δ(x) = argmax
a∈AE[U(y,a)] (5.97)
This is called the maximum expected utility principle, and is the essence of what we mean
byrational behavior.
Note that there are two different interpretations of what we mean by “expected”. In the
Bayesian version, which we discuss below, we mean the expected value of ygiven the data we
have seen so far. In the frequentist version, which we discuss in Section 6.3, we mean theexpected value of yandxthat we expect to see in the future.
In the Bayesian approach to decision theory, the optimal action, having observed x, is deﬁned
as the action athat minimizes the posterior expected loss:
ρ(a|x)≜E

ρ(a|x)≜E
p(y|x)[L(y,a)] =∑
yL(y,a)p(y|x) (5.98)
(Ifyis continuous (e.g., when we want to estimate a parameter vector), we should replace the
sum with an integral.) Hence the Bayes estimator, also called the Bayes decision rule, is given
by
δ(x)=a r gm i n
a∈Aρ(a|x) (5.99)
5.7.1 Bayes estimators for common loss functions
In this section we show how to construct Bayes estimators for the loss functions most commonlyarising in machine learning.
5.7.1.1 MAP estimate minimizes 0-1 loss
The0-1 lossis deﬁned by

ρ(a|x)≜E
by
L(y,a)=I(y̸=a)={0ifa=y
1ifa̸=y(5.100)
This is commonly used in classiﬁcation problems where yis the true class label and a=ˆyis
the estimate.
For example, in the two class case, we can write the loss matrix as follows:
ˆy=1 ˆy=0
y=101
y=0 10

178 Chapter 5. Bayesian statistics
/g2/g3/g4
/g4/g3/g4/g1/g2/g3/g4/g5/g2/g6/g7/g8
/g9/g5/g6/g7/g8/g2/g1 /g9/g1/g10/g11 /g5/g6/g7/g1/g8/g1/g12/g1/g9/g1/g10/g11
/g13/g14/g15/g14/g16/g17 /g1
/g13/g14/g18/g19/g20/g21
Figure 5.13 For some regions of input space, where the class posteriors are uncertain, we may prefer not
to choose class 1 or 2; instead we may prefer the reject option. Based on Figure 1.26 of (Bishop 2006a).
(In Section 5.7.2, we generalize this loss function so it penalizes the two kinds of errors on
the off-diagonal differently.)
The posterior expected loss is
ρ(a|x)=p(a̸=y|x)=1−p(y|x) (5.101)
Hence the action that minimizes the expected loss is the posterior mode or MAP estimate
y∗(x) = argmax
y∈Yp(y|x) (5.102)
5.7.1.2 Reject option
In classiﬁcation problems where p(y|x)is very uncertain, we may prefer to choose a reject
action, in which we refuse to classify the example as any of the speciﬁed classes, and instead

say “don’t know”. Such ambiguous cases can be handled by e.g., a human expert. See Figure 5.13for an illustration. This is useful in risk averse domains such as medicine and ﬁnance.
We can formalize the reject option as follows. Let choosing a=C+1correspond to
picking the reject action, and choosing a∈{1,...,C}correspond to picking one of the classes.
Suppose we deﬁne the loss function as
L(y=j,a=i)=⎧
⎨
⎩0ifi=jandi,j∈{1,...,C}
λ
r ifi=C+1
λs otherwise(5.103)
whereλris the cost of the reject action, and λsis the cost of a substitution error. In Exercise 5.3,

⎨
λ
r ifi=C+1
you will show that the optimal action is to pick the reject action if the most probable class has
a probability below 1−λr
λs; otherwise you should just pick the most probable class.

5.7. Bayesian decision theory 179
−2 −1 0 1 2012|x|0.2
(a)−2 −1 0 1 2012|x|1.0
(b)−2 −1 0 1 2012|x|2.0
(c)
Figure 5.14 (a-c). Plots of the L(y,a)=|y−a|qvs|y−a|forq=0.2,q=1andq=2. Figure
generated by lossFunctionFig .
5.7.1.3 Posterior mean minimizes ℓ2(quadratic) loss
For continuous parameters, a more appropriate loss function is squared error, ℓ2loss,o r
quadratic loss, deﬁned as
L(y,a)=(y−a)2(5.104)
The posterior expected loss is given by
ρ(a|x)=E[
(y−a)2|x]
=E[
y2|x]
−2aE[y|x]+a2(5.105)
Hence the optimal estimate is the posterior mean:
∂
∂aρ(a|x)=−2E[y|x]+2a=0⇒ˆy=E[y|x]=∫
yp(y|x)dy (5.106)
This is often called the minimum mean squared error estimate or MMSEestimate.
In a linear regression problem, we have
p(y|x,θ)=N(y|xTw,σ2) (5.107)
In this case, the optimal estimate given some training data Dis given by
E[y|x,D]=xTE[w|D] (5.108)
That is, we just plug-in the posterior mean parameter estimate. Note that this is the optimal
thing to do no matter what prior we use for w.

∂
5.7.1.4 Posterior median minimizes ℓ1(absolute) loss
Theℓ2loss penalizes deviations from the truth quadratically, and thus is sensitive to outliers. A
more robust alternative is the absolute or ℓ1loss,L(y,a)=|y−a|(see Figure 5.14). The optimal
estimate is the posterior median, i.e., a value asuch that P(y<a|x)=P(y≥a|x)=0.5.
See Exercise 5.9 for a proof.
5.7.1.5 Supervised learning
Consider a prediction function δ:X→Y, and suppose we have some cost function ℓ(y,y′)

which gives the cost of predicting y′when the truth is y. We can deﬁne the loss incurred by

180 Chapter 5. Bayesian statistics
taking action δ(i.e., using this predictor) when the unknown state of nature is θ(the parameters
of the data generating mechanism) as follows:
L(θ,δ)≜E(x,y)∼p(x,y|θ)[ℓ(y,δ(x)] =∑
x∑
yL(y,δ(x))p(x,y|θ) (5.109)
This is known as the generalization error. Our goal is to minimize the posterior expected loss,
given by
ρ(δ|D)=∫
p(θ|D)L(θ,δ)dθ (5.110)
This should be contrasted with the frequentist risk which is deﬁned in Equation 6.47.
5.7.2 The false positive vs false negative tradeoff
In this section, we focus on binary decision problems, such as hypothesis testing, two-class
classiﬁcation, object/ event detection, etc. There are two types of error we can make: a false
positive(akafalse alarm), which arises when we estimate ˆy=1but the truth is y=0;o ra
false negative (akamissed detection), which arises when we estimate ˆy=0but the truth is
y=1. The 0-1 loss treats these two kinds of errors equivalently. However, we can consider the
following more general loss matrix:
ˆy=1 ˆy=0

ˆy=1 ˆy=0
y=10LFN
y=0LFP0
whereLFNis the cost of a false negative, and LFPis the cost of a false positive. The
posterior expected loss for the two possible actions is given by
ρ(ˆy=0|x)=L FNp(y=1|x) (5.111)
ρ(ˆy=1|x)=L FPp(y=0|x) (5.112)
Hence we should pick class ˆy=1iff
ρ(ˆy=0|x)>ρ(ˆy=1|x) (5.113)
p(y=1|x)
p(y=0|x)>LFP
LFN(5.114)
IfLFN=cLFP, it is easy to show (Exercise 5.10) that we should pick ˆy=1iffp(y=
1|x)/p(y=0|x)>τ,w h e r eτ=c/(1 +c)(see also (Muller et al. 2004)). For example, if a

ˆy=1 ˆy=0
y=10LFN
y=0LFP0
p(y=1|x)
false negative costs twice as much as false positive, so c=2, then we use a decision threshold
of2/3before declaring a positive.
Below we discuss ROC curves, which provide a way to study the FP-FN tradeoff without having
to choose a speciﬁc threshold.
5.7.2.1 ROC curves and all that
Suppose we are solving a binary decision problem, such as classiﬁcation, hypothesis testing,object detection, etc. Also, assume we have a labeled data set, D={(x
i,yi)}.L e tδ(x)=

5.7. Bayesian decision theory 181
Truth
10 Σ
Estimate1 TP FP ˆN+=TP+FP
0 FN TN ˆN−=FN+TN
ΣN+=TP+FN N −=FP+TNN=TP+FP+FN+TN
Table 5.2 Quantities derivable from a confusion matrix. N+is the true number of positives, ˆN+is the
“called” number of positives, N−is the true number of negatives, ˆN−is the “called” number of negatives.
y=1 y=0
ˆy=1TP/N +=TPR=sensitivity=recall FP/N −=FPR=type I
ˆy=0FN/N +=FNR=miss rate=type II TN/N −=TNR=speciﬁty
Table 5.3 Estimating p(ˆy|y)from a confusion matrix. Abbreviations: FNR = false negative rate, FPR =
false positive rate, TNR = true negative rate, TPR = true positive rate.
I(f(x)>τ)be our decision rule, where f(x)is a measure of conﬁdence that y=1(this
should be monotonically related to p(y=1|x), but does not need to be a probability), and τis
some threshold parameter. For each given value of τ, we can apply our decision rule and count
the number of true positives, false positives, true negatives, and false negatives that occur, as

shown in Table 5.2. This table of errors is called a confusion matrix.
From this table, we can compute the true positive rate (TPR), also known as the sensitivity,
recallorhit rate, by using TPR=TP/N +≈p(ˆy=1|y=1 ). We can also compute the
false positive rate (FPR), also called the false alarm rate, or the type I error rate, by using
FPR=FP/N −≈p(ˆy=1|y=0 ). These and other deﬁnitions are summarized in Tables 5.3
and 5.4. We can combine these errors in any way we choose to compute a loss function.

However, rather than than computing the TPR and FPR for a ﬁxed threshold τ, we can run
our detector for a set of thresholds, and then plot the TPR vs FPR as an implicit function ofτ. This is called a receiver operating characteristic orROCcurve. See Figure 5.15(a) for an
example. Any system can achieve the point on the bottom left, (FPR=0,TPR=0 ),b y
settingτ=1and thus classifying everything as negative; similarly any system can achieve the

point on the top right, (FPR=1,TPR=1 ), by setting τ=0and thus classifying everything
as positive. If a system is performing at chance level, then we can achieve any point on thediagonal line TPR=FPRby choosing an appropriate threshold. A system that perfectly
separates the positives from negatives has a threshold that can achieve the top left corner,(FPR=0,TPR=1 ); by varying the threshold such a system will “hug” the left axis and
then the top axis, as shown in Figure 5.15(a).
The quality of a ROC curve is often summarized as a single number using the area under the

curveorAUC. Higher AUC scores are better; the maximum is obviously 1. Another summary
statistic that is used is the equal error rate orEER, also called the cross over rate, deﬁned
as the value which satisﬁes FPR=FNR. SinceFNR=1−TPR, we can compute the
EER by drawing a line from the top left to the bottom right and seeing where it intersects theROC curve (see points A and B in Figure 5.15(a)). Lower EER scores are better; the minimum isobviously 0.

182 Chapter 5. Bayesian statistics
0 101
fprtprA
B
(a)0 101
recallprecisionA B
(b)
Figure 5.15 (a) ROC curves for two hypothetical classiﬁcation systems. A is better than B. We plot the
true positive rate (TPR) vs the false positive rate (FPR) as we vary the threshold τ. We also indicate the
equal error rate (EER) with the red and blue dots, and the area under the curve (AUC) for classiﬁer B. (b)
A precision-recall curve for two hypothetical classiﬁcation systems. A is better than B. Figure generated byPRhand.
y=1 y=0
ˆy=1TP/ˆN+=precision=PPV FP/ˆN+=FDP
ˆy=0 FN/ˆN− TN/ˆN−=NPV
Table 5.4 Estimating p(y|ˆy)from a confusion matrix. Abbreviations: FDP = false discovery probability,
NPV = negative predictive value, PPV = positive predictive value,
5.7.2.2 Precision recall curves
When trying to detect a rare event (such as retrieving a relevant document or ﬁnding a face
in an image), the number of negatives is very large. Hence comparing TPR=TP/N +to

y=1 y=0
FPR=FP/N −is not very informative, since the FPR will be very small. Hence all the
“action” in the ROC curve will occur on the extreme left. In such cases, it is common to plotthe TPR versus the number of false positives, rather than vs the false positive rate.
However, in some cases, the very notion of “negative” is not well-deﬁned. For example, when

detecting objects in images (see Section 1.2.1.3), if the detector works by classifying patches, thenthe number of patches examined — and hence the number of true negatives — is a parameterof the algorithm, not part of the problem deﬁnition. So we would like to use a measure thatonly talks about positives.
Theprecision is deﬁned as TP/ˆN
+=p(y=1|ˆy=1 )and therecallis deﬁned as
TP/N +=p(ˆy=1|y=1 ). Precision measures what fraction of our detections are actually
positive, and recall measures what fraction of the positives we actually detected. If ˆyi∈{0,1}

is the predicted label, and yi∈{0,1}is the true label, we can estimate precision and recall
using
P=∑
iyiˆyi∑
iˆyi,R=∑
iyiˆyi∑
iyi(5.115)
Aprecision recall curve is a plot of precision vs recall as we vary the threshold τ. See
Figure 5.15(b). Hugging the top right is the best one can do.
This curve can be summarized as a single number using the mean precision (averaging over

5.7. Bayesian decision theory 183
Class 1
y=1y=0
ˆy=110 10
ˆy=010 970Class 2
y=1y=0
ˆy=190 10
ˆy=0 10 890Pooled
y=1y=0
ˆy=1100 20
ˆy=020 1860
Table 5.5 Illustration of the difference between macro- and micro-averaging. yis the true label, and ˆy
is the called label. In this example, the macro-averaged precision is [10/(10+10)+90/ (10+90)]/ 2=
(0.5+0.9)/2=0.7. The micro-averaged precision is 100/(100 + 20) ≈0.83. Based on Table 13.7 of
(Manning et al. 2008).
recall values), which approximates the area under the curve. Alternatively, one can quote the
precision for a ﬁxed recall level, such as the precision of the ﬁrst K=1 0entities recalled.
This is called the average precision at K score. This measure is widely used when evaluating
information retrieval systems.
5.7.2.3 F-scores *
For a ﬁxed threshold, one can compute a single precision and recall value. These are oftencombined into a single statistic called the Fs c o r e,o rF1 score, which is the harmonic mean of
precision and recall:
F
1≜2
1/P+1/R=2PR

F
1≜2
R+P(5.116)
Using Equation 5.115, we can write this as
F1=2∑N
i=1yiˆyi∑Ni=1yi+∑Ni=1ˆyi(5.117)
This is a widely used measure in information retrieval systems.
To understand why we use the harmonic mean instead of the arithmetic mean, (P+R)/2,
consider the following scenario. Suppose we recall all entries, so R=1. The precision will be
given by the prevalence, p(y=1 ). Suppose the prevalence is low, say p(y=1 )=1 0−4. The
arithmetic mean of PandRis given by (P+R)/2=( 1 0−4+1)/2≈50%. By contrast, the
harmonic mean of this strategy is only2×10−4×1

F1=2∑N
1+10−4≈0.2%.
In the multi-class case (e.g., for document classiﬁcation problems), there are two ways to
generalize F1scores. The ﬁrst is called macro-averaged F1, and is deﬁned as∑Cc=1F1(c)/C,
whereF1(c)is theF1score obtained on the task of distinguishing class cfrom all the others.
The other is called micro-averaged F1, and is deﬁned as the F1score where we pool all the
counts from each class’s contingency table.
Table 5.5 gives a worked example that illustrates the difference. We see that the precision of

class 1 is 0.5, and of class 2 is 0.9. The macro-averaged precision is therefore 0.7, whereas the
micro-averaged precision is 0.83. The latter is much closer to the precision of class 2 than tothe precision of class 1, since class 2 is ﬁve times larger than class 1. To give equal weight toeach class, use macro-averaging.

184 Chapter 5. Bayesian statistics
5.7.2.4 False discovery rates *
Suppose we are trying to discover a rare phenomenon using some kind of high throughput
measurement device, such as a gene expression micro array, or a radio telescope. We will needto make many binary decisions of the form p(y
i=1|D)>τ,w h e r eD ={xi}N
i=1andNmay
be large. This is called multiple hypothesis testing. Note that the difference from standard
binary classiﬁcation is that we are classifying yibased on all the data, not just based on xi.S o
this is a simultaneous classiﬁcation problem, where we might hope to do better than a series of
individual classiﬁcation problems.
How should we set the threshold τ? A natural approach is to try to minimize the expected
number of false positives. In the Bayesian approach, this can be computed as follows:
FD(τ,D)≜∑
i(1−pi)
pr. errorI(pi>τ)
discovery(5.118)
wherepi≜p(yi=1|D)is your belief that this object exhibits the phenomenon in question.

FD(τ,D)≜∑
We then deﬁne the posterior expected false discovery rate as follows:
FDR(τ,D)≜FD(τ,D)/N(τ,D) (5.119)
whereN(τ,D)=∑
iI(pi>τ)is the number of discovered items. Given a desired FDR
tolerance, say α=0.05, one can then adapt τto achieve this; this is called the direct posterior
probability approach to controlling the FDR (Newton et al. 2004; Muller et al. 2004).
In order to control the FDR it is very helpful to estimate the pi’s jointly (e.g., using a hierar-

chical Bayesian model, as in Section 5.5), rather than independently. This allows the pooling ofstatistical strength, and thus lower FDR. See e.g., (Berry and Hochberg 1999) for more information.
5.7.3 Other topics *
In this section, we brieﬂy mention a few other topics related to Bayesian decision theory. We donot have space to go into detail, but we include pointers to the relevant literature.
5.7.3.1 Contextual bandits
Aone-armed bandit is a colloquial term for a slot machine, found in casinos around the world.

The game is this: you insert some money, pull an arm, and wait for the machine to stop; ifyou’re lucky, you win some money. Now imagine there is a bank of Ksuch machines to choose
from. Which one should you use? This is called a multi-armed bandit, and can be modeled
using Bayesian decision theory: there are Kpossible actions, and each action has an unknown
reward (payoff function) r
k. By maintaining a belief state, p(r1:K|D)=∏
kp(rk|D), one can
devise an optimal policy; this can be compiled into a series of Gittins Indices (Gittins 1989).
This optimally solves the exploration-exploitation tradeoff, which speciﬁes how many times

one should try each action before deciding to go with the winner.
Now consider an extension where each arm, and the player, has an associated feature vector;
call all these features x. This is called a contextual bandit (see e.g., (Sarkar 1991; Scott 2010;
Li et al. 2011)). For example, the “arms” could represent ads or news articles which we wantto show to the user, and the features could represent properties of these ads or articles, such

5.7. Bayesian decision theory 185
as a bag of words, as well as properties of the user, such as demographics. If we assume a
linear model for reward, rk=θT
kx, we can maintain a distribution over the parameters of each
arm,p(θk|D),w h e r eDis a series of tuples of the form (a,x,r), which speciﬁes which arm
was pulled, what its features were, and what the resulting outcome was (e.g., r=1if the user
clicked on the ad, and r=0otherwise). We discuss ways to compute p(θk|D)from linear and
logistic regression models in later chapters.
Given the posterior, we must decide what action to take. One common heuristic, known as
UCB(which stands for “upper conﬁdence bound”) is to take the action which maximizes
k∗=Kargmax
k=1μk+λσk (5.120)
whereμk=E[rk|D],σ2
k= var[r k|D]andλis a tuning parameter that trades off exploration
and exploitation. The intuition is that we should pick actions about which we believe are good
(μkis large), and/ or actions about which we are uncertain (σ kis large).

An even simpler method, known as Thompson sampling, is as follows. At each step, we pick
actionkwith a probability that is equal to its probability of being the optimal action:
pk=∫
I(E[r|a,x,θ]=m a x
a′E[r|a′,x,θ])p(θ|D)dθ (5.121)
We can approximate this by drawing a single sample from the posterior, θt∼p(θ|D), and then
choosing k∗=a r g m a xkE[
r|x,k,θt]
. Despite its simplicity, this has been shown to work quite
well (Chapelle and Li 2011).
5.7.3.2 Utility theory

pk=∫
r|x,k,θt]
Suppose we are a doctor trying to decide whether to operate on a patient or not. We imaginethere are 3 states of nature: the patient has no cancer, the patient has lung cancer, or thepatient has breast cancer. Since the action and state space is discrete, we can represent the lossfunctionL(θ,a)as aloss matrix, such as the following:
Surgery No surgery
No cancer 20 0
Lung cancer 10 50
Breast cancer 10 60
These numbers reﬂects the fact that not performing surgery when the patient has cancer is

very bad (loss of 50 or 60, depending on the type of cancer), since the patient might die; notperforming surgery when the patient does not have cancer incurs no loss (0); performing surgerywhen the patient does not have cancer is wasteful (loss of 20); and performing surgery whenthe patient does have cancer is painful but necessary (10).
It is natural to ask where these numbers come from. Ultimately they represent the personal
preferences or values of a ﬁctitious doctor, and are somewhat arbitrary: just as some people

prefer chocolate ice cream and others prefer vanilla, there is no such thing as the “right” loss/utility function. However, it can be shown (see e.g., (DeGroot 1970)) that any set of consistentpreferences can be converted to a scalar loss/ utility function. Note that utility can be measuredon an arbitrary scale, such as dollars, since it is only relative values that matter.
6
6. People are often squeamish about talking about human lives in monetary terms, but all decision making requires

186 Chapter 5. Bayesian statistics
5.7.3.3 Sequential decision theory
So far, we have concentrated on one-shot decision problems, where we only have to make
one decision and then the game ends. In Setion 10.6, we will generalize this to multi-stage or
sequential decision problems. Such problems frequently arise in many business and engineeringsettings. This is closely related to the problem of reinforcement learning. However, furtherdiscussion of this point is beyond the scope of this book.
Exercises
Exercise 5.1 Proof that a mixture of conjugate priors is indeed conjugate
Derive Equation 5.69.
Exercise 5.2 Optimal threshold on classiﬁcation probability
Consider a case where we have learned a conditional probability distribution P(y|x). Suppose there are
only two classes, and let p0=P(Y=0|x)andp1=P(Y=1|x). Consider the loss matrix below:
predicted true label y
labelˆy 01
0 0λ01
1 λ100
a. Show that the decision ˆythat minimizes the expected loss is equivalent to setting a probability threshold

Exercises
labelˆy 01
0 0λ01
1 λ100
θand predicting ˆy=0ifp1<θandˆy=1ifp1≥θ. What is θas a function of λ01andλ10? (Show
your work.)
b. Show a loss matrix where the threshold is 0.1. (Show your work.)
Exercise 5.3 Reject option in classiﬁers
(Source: (Duda et al. 2001, Q2.13).)
In many classiﬁcation problems one has the option either of assigning xto classjor, if you are too
uncertain, of choosing the reject option. If the cost for rejects is less than the cost of falsely classifying

the object, it may be the optimal action. Let αimean you choose action i,f o ri=1:C+1,w h e r eC
is the number of classes and C+1is the reject action. Let Y=jbe the true (but unknown) state of
nature. Deﬁne the loss function as follows
λ(αi|Y=j)=⎧
⎨
λr ifi=C+1
λs otherwise(5.122)
In otherwords, you incur 0 loss if you correctly classify, you incur λrloss (cost) if you choose the reject
option, and you incur λsloss (cost) if you make a substitution error (misclassiﬁcation).

⎨
λr ifi=C+1
tradeoffs, and one needs to use some kind of “currency” to compare different courses of action. Insurance companies
do this all the time. Ross Schachter, a decision theorist at Stanford University, likes to tell a story of a school board who
rejected a study on absestos removal from schools because it performed a cost-beneﬁt analysis, which was considered

“inhumane” because they put a dollar value on children’s health; the result of rejecting the report was that the absestoswas not removed, which is surely more “inhumane”. In medical domains, one often measures utility in terms of QALY,o r
quality-adjusted life-years, instead of dollars, but it’s the same idea. Of course, even if you do not explicitly specify howmuch you value different people’s lives, your behaviorwill reveal your implicit values/ preferences, and these preferences

can then be converted to a real-valued scale, such as dollars or QALY. Inferring a utility function from behavior is calledinverse reinforcement learning.

5.7. Bayesian decision theory 187
Decision true label y
ˆy 01
predict 0 010
predict 1 100
reject 33
a. Show that the minimum risk is obtained if we decide Y=jifp(Y=j|x)≥p(Y=k|x)for allk
(i.e.,jis the most probable class) andifp(Y=j|x)≥1−λr
λs; otherwise we decide to reject.
b. Describe qualitatively what happens as λr/λsis increased from 0 to 1 (i.e., the relative cost of rejection
increases).
Exercise 5.4 More reject options
In many applications, the classiﬁer is allowed to “reject” a test example rather than classifying it into one
of the classes. Consider, for example, a case in which the cost of a misclassiﬁcation is $10 but the cost ofhaving a human manually make the decison is only $3. We can formulate this as the following loss matrix:
a. Suppose P(y=1|x)is predicted to be 0.2. Which decision minimizes the expected loss?
b. Now suppose P(y=1|x)=0.4. Now which decision minimizes the expected loss?

c. Show that in general, for this loss matrix, but for any posterior distribution, there will be two thresholds
θ
0andθ1such that the optimal decisionn is to predict 0 if p1<θ0, reject if θ0≤p1≤θ1, and
predict 1 if p1>θ1(wherep1=p(y=1|x)). What are these thresholds?
Exercise 5.5 Newsvendor problem
Consider the following classic problem in decision theory/ economics. Suppose you are trying to decidehow much quantity Qof some product (e.g., newspapers) to buy to maximize your proﬁts. The optimal

θ
amount will depend on how much demand Dyou think there is for your product, as well as its cost
to youCand its selling price P. Suppose Dis unknown but has pdf f(D)and cdfF(D). We can
evaluate the expected proﬁt by considering two cases: if D>Q, then we sell all Qitems, and make proﬁt
π=(P−C)Q; but ifD<Q, we only sell Ditems, at proﬁt (P−C)D, but have wasted C(Q−D)
on the unsold items. So the expected proﬁt if we buy quantity Qis
Eπ(Q)=∫
∞
Q(P−C)Qf(D)dD+∫Q
0(P−C)Df(D)−∫Q
0C(Q−D)f(D)dD (5.123)

Eπ(Q)=∫
∞
Simplify this expression, and then take derivatives wrt Qto show that the optimal quantity Q∗(which
maximizes the expected proﬁt) satisﬁes
F(Q∗)=P−C
P(5.124)
Exercise 5.6 Bayes factors and ROC curves
LetB=p(D|H1)/p(D|H0)be the bayes factor in favor of model 1. Suppose we plot two ROC curves,
one computed by thresholding B, and the other computed by thresholding p(H1|D). Will they be the
same or different? Explain why.
Exercise 5.7 Bayes model averaging helps predictive accuracy

F(Q∗)=P−C
P(5.124)
LetΔbe a quantity that we want to predict, let Dbe the observed data and Mbe a ﬁnite set of models.
Suppose our action is to provide a probabilistic prediction p(), and the loss function is L(Δ,p()) =

188 Chapter 5. Bayesian statistics
−logp(Δ). We can either perform Bayes model averaging and predict using
pBMA(Δ) =∑
m∈Mp(Δ|m,D)p(m|D) (5.125)
or we could predict using any single model (a plugin approximation)
pm(Δ) =p(Δ|m,D) (5.126)
Show that, for all models m∈M, the posterior expected loss using BMA is lower, i.e.,
E[
L(Δ,pBMA)]
≤E[L(Δ,pm)] (5.127)
where the expectation over Δis with respect to
p(Δ|D)=∑
m∈Mp(Δ|m,D)p(m|D) (5.128)
Hint: use the non-negativity of the KL divergence.
Exercise 5.8 MLE and model selection for a 2d discrete distribution
Letx∈{0,1}denote the result of a coin toss (x =0for tails,x=1for heads). The coin is potentially
biased, so that heads occurs with probability θ1. Suppose that someone else observes the coin ﬂip and
reports to you the outcome, y. But this person is unreliable and only reports the result correctly with
probability θ2; i.e.,p(y|x,θ2)is given by
y=0y=1
x=0θ21−θ2
x=11−θ2θ2
Assume that θ2is independent of xandθ1.

y=0y=1
x=0θ21−θ2
x=11−θ2θ2
a. Write down the joint probability distribution p(x,y|θ)as a2×2table, in terms of θ=(θ1,θ2).
b. Suppose have the following dataset: x=( 1,1,0,1,1,0,0),y=( 1,0,0,0,1,0,1). What are the
MLEs for θ1andθ2? Justify your answer. Hint: note that the likelihood function factorizes,
p(x,y|θ)=p(y|x,θ2)p(x|θ1) (5.129)
What isp(D|ˆθ,M2)whereM2denotes this 2-parameter model? (You may leave your answer in
fractional form if you wish.)
c. Now consider a model with 4 parameters, θ=(θ0,0,θ0,1,θ1,0,θ1,1), representing p(x,y|θ)=θx,y.

(Only 3 of these parameters are free to vary, since they must sum to one.) What is the MLE of θ? What
isp(D|ˆθ,M4)whereM4denotes this 4-parameter model?
d. Suppose we are not sure which model is correct. We compute the leave-one-out cross validated log
likelihood of the 2-parameter model and the 4-parameter model as follows:
L(m)=n∑
i=1logp(xi,yi|m,ˆθ(D−i)) (5.130)
andˆθ(D−i))denotes the MLE computed on Dexcluding row i. Which model will CV pick and why?
Hint: notice how the table of counts changes when you omit each training case one at a time.

5.7. Bayesian decision theory 189
e. Recall that an alternative to CV is to use the BIC score, deﬁned as
BIC(M,D)≜logp(D|ˆθMLE)−dof(M)
2logN (5.131)
wheredof(M)is the number of free parameters in the model, Compute the BIC scores for both models
(use log base e). Which model does BIC prefer?
Exercise 5.9 Posterior median is optimal estimate under L1 loss
Prove that the posterior median is optimal estimate under L1 loss.
Exercise 5.10 Decision rule for trading off FPs and FNs
IfLFN=cLFP, show that we should pick ˆy=1iffp(y=1|x)/p(y=0|x)>τ,w h e r eτ=c/(1+c)

6 Frequentist statistics
6.1 Introduction
The approach to statistical inference that we described in Chapter 5 is known as Bayesian
statistics. Perhaps surprisingly, this is considered controversial by some people, whereas the ap-plication of Bayes rule to non-statistical problems — such as medical diagnosis (Section 2.2.3.1),spam ﬁltering (Section 3.4.4.1), or airplane tracking (Section 18.2.1) — is not controversial. Thereason for the objection has to do with a misguided distinction between parameters of a statis-tical model and other kinds of unknown quantities.
1
Attempts have been made to devise approaches to statistical inference that avoid treating
parameters like random variables, and which thus avoid the use of priors and Bayes rule. Suchapproaches are known as frequentist statistics, classical statistics ororthodox statistics.

1
Instead of being based on the posterior distribution, they are based on the concept of a samplingdistribution. This is the distribution that an estimator has when applied to multiple data setssampled from the true but unknown distribution; see Section 6.2 for details. It is this notionof variation across repeated trials that forms the basis for modeling uncertainty used by thefrequentist approach.
By contrast, in the Bayesian approach, we only ever condition on the actually observed data;

there is no notion of repeated trials. This allows the Bayesian to compute the probability ofone-off events, as we discussed in Section 2.1. Perhaps more importantly, the Bayesian approachavoids certain paradoxes that plague the frequentist approach (see Section 6.6). Nevertheless, itis important to be familiar with frequentist statistics (especially Section 6.5), since it is widelyused in machine learning.
6.2 Sampling distribution of an estimator

In frequentist statistics, a parameter estimate ˆθis computed by applying an estimator δto
some data D,s oˆθ=δ(D). The parameter is viewed as ﬁxed and the data as random, which
is the exact opposite of the Bayesian approach. The uncertainty in the parameter estimate canbe measured by computing the sampling distribution of the estimator. To understand this
1. Parameters are sometimes considered to represent true (but unknown) physical quantities, which are therefore not

random. However, we have seen that it is perfectly reasonable to use a probability distribution to represent one’s
uncertainty about an unknown constant.

192 Chapter 6. Frequentist statistics
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 105001000150020002500300035004000Boot: true = 0.70, n=10, mle = 0.90, se = 0.001
(a)0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10500100015002000250030003500Boot: true = 0.70, n=100, mle = 0.70, se = 0.000
(b)
Figure 6.1 A bootstrap approximation to the sampling distribution of ˆθfor a Bernoulli distribution. We
useB=1 0,000bootstrap samples. The Ndatacases were generated from Ber(θ=0.7). (a) MLE with
N=1 0. (b) MLE with N= 100. Figure generated by bootstrapDemoBer .
concept, imagine sampling many different data sets D(s)from some true model, p(·|θ∗), i.e., let
D(s)={x(s)
i}N
i=1,w h e r exsi∼p(·|θ∗), andθ∗is the true parameter. Here s=1:Sindexes
the sampled data set, and Nis the size of each such dataset. Now apply the estimator ˆθ(·)
to eachD(s)to get a set of estimates, {ˆθ(D(s))}.A sw el e t S→∞, the distribution induced
onˆθ(·)is the sampling distribution of the estimator. We will discuss various ways to use the

i}N
sampling distribution in later sections. But ﬁrst we sketch two approaches for computing the
sampling distribution itself.
6.2.1 Bootstrap
Thebootstrap is a simple Monte Carlo technique to approximate the sampling distribution. This
is particularly useful in cases where the estimator is a complex function of the true parameters.
The idea is simple. If we knew the true parameters θ∗, we could generate many (say S) fake
datasets, each of size N, from the true distribution, xs
i∼p(·|θ∗),f o rs=1:S,i=1:N.

We could then compute our estimator from each sample, ˆθs=f(xs
1:N)and use the empirical
distribution of the resulting samples as our estimate of the sampling distribution. Since θis
unknown, the idea of the parametric bootstrap is to generate the samples using ˆθ(D)instead.
An alternative, called the non-parametric bootstrap, is to sample the xs
i(with replacement)
from the original data D, and then compute the induced distribution as before. Some methods

for speeding up the bootstrap when applied to massive data sets are discussed in (Kleiner et al.
2011).
Figure 6.1 shows an example where we compute the sampling distribution of the MLE for
a Bernoulli using the parametric bootstrap. (Results using the non-parametric bootstrap areessentially the same.) We see that the sampling distribution is asymmetric, and therefore quitefar from Gaussian, when N=1 0; whenN= 100, the distribution looks more Gaussian, as
theory suggests (see below).

2011).
A natural question is: what is the connection between the parameter estimates ˆθ
s=ˆθ(xs
1:N)
computed by the bootstrap and parameter values sampled from the posterior, θs∼p(·|D)?

6.2. Sampling distribution of an estimator 193
Conceptually they are quite different. But in the common case that that the prior is not very
strong, they can be quite similar. For example, Figure 6.1(c-d) shows an example where wecompute the posterior using a uniform Beta(1,1) prior, and then sample from it. We see thatthe posterior and the sampling distribution are quite similar. So one can think of the bootstrapdistribution as a “poor man’s” posterior; see (Hastie et al. 2001, p235) for details.
However, perhaps surprisingly, bootstrap can be slower than posterior sampling. The reason
is that the bootstrap has to ﬁt the model Stimes, whereas in posterior sampling, we usually
only ﬁt the model once (to ﬁnd a local mode), and then perform local exploration around themode. Such local exploration is usually much faster than ﬁtting a model from scratch.
6.2.2 Large sample theory for the MLE *

In some cases, the sampling distribution for some estimators can be computed analytically. Inparticular, it can be shown that, under certain conditions, as the sample size tends to inﬁnity,the sampling distribution of the MLE becomes Gaussian. Informally, the requirement for thisresult to hold is that each parameter in the model gets to “see” an inﬁnite amount of data, andthat the model be identiﬁable. Unfortunately this excludes many of the models of interest tomachine learning. Nevertheless, let us assume we are in a simple setting where the theoremholds.

The center of the Gaussian will be the MLE ˆθ. But what about the variance of this Gaussian?
Intuitively the variance of the estimator will be (inversely) related to the amount of curvature ofthe likelihood surface at its peak. If the curvature is large, the peak will be “sharp”, and thevariance low; in this case, the estimate is “well determined”. By contrast, if the curvature issmall, the peak will be nearly “ﬂat”, so the variance is high.
Let us now formalize this intuition. Deﬁne the score function as the gradient of the log
likelihood evaluated at some point ˆθ:
s(ˆθ)≜∇logp(D|θ)|
ˆθ(6.1)
Deﬁne the observed information matrix as the gradient of the negative score function, or
equivalently, the Hessian of the NLL:
J(ˆθ(D))≜−∇s(ˆθ)=−∇2
θlogp(D|θ)|ˆθ(6.2)
In 1D, this becomes
J(ˆθ(D)) =−d
dθ2logp(D|θ)|ˆθ(6.3)
This is just a measure of curvature of the log-likelihood function at ˆθ.
Since we are studying the sampling distribution, D=(x1,...,xN)is a set of random

ˆθ(6.1)
variables. The Fisher information matrix is deﬁned to be the expected value of the observed
information matrix:2
IN(ˆθ|θ∗)≜Eθ∗[
J(ˆθ|D)]
(6.4)
2. This is not the usual deﬁnition, but is equivalent to it under standard assumptions. More precisely, the standard
deﬁnition is as follows (we just give the scalar case to simplify notation): I(ˆθ|θ∗)≜varθ∗[
d
dθlogp(X|θ)|ˆθ]
, that
is, the variance of the score function. If ˆθis the MLE, it is easy to see that Eθ∗[
d
=0(since

194 Chapter 6. Frequentist statistics
where Eθ∗[f(D)]≜1
N∑N
i=1f(xi)p(xi|θ∗)is the expected value of the function fwhen
applied to data sampled from θ∗.O f t e nθ∗, representing the “true parameter” that generated
the data, is assumed known, so we just write IN(ˆθ)≜IN(ˆθ|θ∗)for short. Furthermore, it is
easy to see that IN(ˆθ)=NI1(ˆθ), because the log-likelihood for a sample of size Nis justN
times “steeper” than the log-likelihood for a sample of size 1. So we can drop the 1subscript
and just write I(ˆθ)≜I1(ˆθ). This is the notation that is usually used.
Now letˆθ≜ˆθmle(D)be the MLE, where D∼θ∗. It can be shown that
ˆθ→N(θ∗,IN(θ∗)−1) (6.5)
asN→∞(see e.g., (Rice 1995, p265) for a proof). We say that the sampling distribution of the
MLE isasymptotically normal.
What about the variance of the MLE, which can be used as some measure of conﬁdence
in the MLE? Unfortunately, θ∗is unknown, so we can’t evaluate the variance of the sampling

distribution. However, we can approximate the sampling distribution by replacing θ∗withˆθ.
Consequently, the approximate standard errors ofˆθkare given by
ˆsek≜IN(ˆθ)−1
2
kk(6.6)
For example, from Equation 5.60 we know that the Fisher information for a binomial sampling
model is
I(θ)=1
θ(1−θ)(6.7)
So the approximate standard error of the MLE is
ˆse=1√
IN(ˆθ)=1√
NI(ˆθ)=(ˆθ(1−ˆθ)
N)1
2
(6.8)
whereˆθ=1
N∑
iXi. Compare this to Equation 3.27, which is the posterior standard deviation
under a uniform prior.
6.3 Frequentist decision theory

2
kk(6.6)
model is
I(θ)=1
ˆse=1√
IN(ˆθ)=1√
N)1
2
(6.8)
whereˆθ=1
N∑
In frequentist or classical decision theory, there is a loss function and a likelihood, but there is
no prior and hence no posterior or posterior expected loss. Thus there is no automatic way ofderiving an optimal estimator, unlike the Bayesian case. Instead, in the frequentist approach, weare free to choose any estimator or decision procedure δ:X→Awe want.
3
the gradient must be zero at a maximum), so the variance reduces to the expected square of the score function:
I(ˆθ|θ∗)=E θ∗[
(d
dθlogp(X|θ))2]
. It can be shown (e.g., (Rice 1995, p263)) that Eθ∗[
(d

3
(d
(d
=
−Eθ∗[
d2
dθ2logp(X|θ)]
, so now the Fisher information reduces to the expected second derivative of the NLL, which
is a much more intuitive quantity than the variance of the score.
3. In practice, the frequentist approach is usually only applied to one-shot statistical decision problems — such as
classiﬁcation, regression and parameter estimation — since its non-constructive nature makes it difficult to apply to
sequential decision problems, which adapt to data online.

6.3. Frequentist decision theory 195
Having chosen an estimator, we deﬁne its expected loss or riskas follows:
R(θ∗,δ)≜Ep(˜D|θ∗)[
L(θ∗,δ(˜D))]
=∫
L(θ∗,δ(˜D))p(˜D|θ∗)d˜D (6.9)
where˜Dis data sampled from “nature’s distribution”, which is represented by parameter θ∗.I n
other words, the expectation is wrt the sampling distribution of the estimator. Compare this to
the Bayesian posterior expected loss:
ρ(a|D,π)≜Ep(θ|D,π)[L(θ,a)] =∫
ΘL(θ,a)p(θ|D,π)dθ (6.10)
We see that the Bayesian approach averages over θ(which is unknown) and conditions on D
(which is known), whereas the frequentist approach averages over ˜D(thus ignoring the observed
data), and conditions on θ∗(which is unknown).
Not only is the frequentist deﬁnition unnatural, it cannot even be computed, because θ∗is
unknown. Consequently, we cannot compare different estimators in terms of their frequentistrisk. We discuss various solutions to this below.
6.3.1 Bayes risk
How do we choose amongst estimators? We need some way to convert R(θ∗,δ)into a single

measure of quality, R(δ), which does not depend on knowing θ∗. One approach is to put a
prior onθ∗, and then to deﬁne Bayes risk orintegrated risk of an estimator as follows:
RB(δ)≜Ep(θ∗)[R(θ∗,δ)] =∫
R(θ∗,δ)p(θ∗)dθ∗(6.11)
ABayes estimator orBayes decision rule is one which minimizes the expected risk:
δB≜argmin
δRB(δ) (6.12)
Note that the integrated risk is also called the preposterior risk, since it is before we have seen
the data. Minimizing this can be useful for experiment design.

δB≜argmin
We will now prove a very important theorem, that connects the Bayesian and frequentist
approaches to decision theory.
Theorem 6.3.1. A Bayes estimator can be obtained by minimizing the posterior expected loss for
eachx.
Proof.By switching the order of integration, we have
RB(δ)=∫[∑
x∑
yL(y,δ(x))p(x,y|θ∗)]
p(θ∗)dθ∗(6.13)
=∑
x∑
y∫
ΘL(y,δ(x))p(x,y,θ∗)dθ∗(6.14)
=∑
x[∑
yL(y,δ(x))p(y|x)dy]
p(x) (6.15)
=∑
xρ(δ(x)|x)p(x) (6.16)

196 Chapter 6. Frequentist statistics
R(θ,δ2)R(θ,δ1)
θR
Figure 6.2 Risk functions for two decision procedures, δ1andδ2. Sinceδ1has lower worst case risk, it
is the minimax estimator, even though δ2has lower risk for most values of θ. Thus minimax estimators
are overly conservative.
To minimize the overall expectation, we just minimize the term inside for each x, so our decision
rule is to pick
δB(x) = argmin
a∈Aρ(a|x) (6.17)
Hence we see that the picking the optimal action on a case-by-case basis (as in the Bayesian
approach) is optimal on average (as in the frequentist approach). In other words, the Bayesian
approach provides a good way of achieving frequentist goals. In fact, one can go further andprove the following.
Theorem 6.3.2 (Wald, 1950). Every admissable decision rule is a Bayes decision rule with respect
to some, possibly improper, prior distribution.
This theorem shows that the best way to minimize frequentist risk is to be Bayesian! See

(Bernardo and Smith 1994, p448) for further discussion of this point.
6.3.2 Minimax risk
Obviously some frequentists dislike using Bayes risk since it requires the choice of a prior (al-
though this is only in the evaluation of the estimator, not necessarily as part of its construction).An alternative approach is as follows. Deﬁne the maximum risk of an estimator as
R
max(δ)≜max
θ∗R(θ∗,δ) (6.18)
Aminimax rule is one which minimizes the maximum risk:
δMM≜argmin
δRmax(δ) (6.19)

6.3. Frequentist decision theory 197
For example, in Figure 6.2, we see that δ1has lower worst-case risk than δ2, ranging over all
possible values of θ∗, so it is the minimax estimator (see Section 6.3.3.1 for an explanation of
how to compute a risk function for an actual model).
Minimax estimators have a certain appeal. However, computing them can be hard. And
furthermore, they are very pessimistic. In fact, one can show that all minimax estimators
are equivalent to Bayes estimators under a least favorable prior. In most statistical situations
(excluding game theoretic ones), assuming nature is an adversary is not a reasonable assumption.
6.3.3 Admissible estimators
The basic problem with frequentist decision theory is that it relies on knowing the true distri-butionp(·|θ
∗)in order to evaluate the risk. However, It might be the case that some estimators
are worse than others regardless of the value of θ∗. In particular, if R(θ,δ 1)≤R(θ,δ 2)for all

θ∈Θ, then we say that δ1dominates δ2. The domination is said to be strict if the inequality
is strict for some θ. An estimator is said to be admissible if it is not strictly dominated by any
other estimator.
6.3.3.1 Example
Let us give an example, based on (Bernardo and Smith 1994). Consider the problem of estimatingthe mean of a Gaussian. We assume the data is sampled from x
i∼N(θ∗,σ2=1 )and use
quadratic loss, L(θ,ˆθ)=(θ−ˆθ)2. The corresponding risk function is the MSE. Some possible

decision rules or estimators ˆθ(x)=δ(x)are as follows:
•δ1(x)=x, the sample mean
•δ2(x)=˜x, the sample median
•δ3(x)=θ0, a ﬁxed value
•δκ(x), the posterior mean under a N(θ|θ0,σ2/κ)prior:
δκ(x)=N
N+κx+κ
N+κθ0=wx+(1−w)θ0 (6.20)
Forδκ, we consider a weak prior, κ=1, and a stronger prior, κ=5. The prior mean is θ0,
some ﬁxed value. We assume σ2is known. (Thus δ3(x)is the same as δκ(x)with an inﬁnitely
strong prior, κ=∞.)
Let us now derive the risk functions analytically. (We can do this since in this toy example,

δκ(x)=N
N+κx+κ
we know the true parameter θ∗.) In Section 6.4.4, we show that the MSE can be decomposed
into squared bias plus variance:
MSE(ˆθ(·)|θ∗)=v a r[
ˆθ]
+bias2(ˆθ) (6.21)
The sample mean is unbiased, so its risk is
MSE(δ1|θ∗)=v a r[x]=σ2
N(6.22)

198 Chapter 6. Frequentist statistics
−2 −1.5 −1 −0.5 0 0.5 1 1.5 200.050.10.150.20.250.30.350.40.450.5
θ*R(θ*,δ)risk functions for n=5
  
mle
median
fixed
postmean1
postmean5
(a)−2 −1.5 −1 −0.5 0 0.5 1 1.5 200.020.040.060.080.10.120.140.160.18
θ*R(θ*,δ)risk functions for n=20
  
mle
median
fixed
postmean1
postmean5
(b)
Figure 6.3 Risk functions for estimating the mean of a Gaussian using data sampled N(θ∗,σ2=1 ).
The solid dark blue horizontal line is the MLE, the solid light blue curved line is the posterior mean when
κ=5. Left:N=5samples. Right: N=2 0samples. Based on Figure B.1 of (Bernardo and Smith 1994).
Figure generated by riskFnGauss .
The sample median is also unbiased. One can show that the variance is approximately π/(2N),
so
MSE(δ2|θ∗)=π
2N(6.23)
Forδ3(x)=θ0, the variance is zero, so
MSE(δ3|θ∗)=(θ∗−θ0)2(6.24)
Finally, for the posterior mean, we have
MSE(δκ|θ∗)=E[
(wx+(1−w)θ0−θ∗)2]
(6.25)
=E[
(w(x−θ∗)+(1−w)(θ0−θ∗))2]
(6.26)
=w2σ2
N+(1−w)2(θ0−θ∗)2(6.27)
=1
(N+κ)2(
Nσ2+κ2(θ0−θ∗)2)
(6.28)

so
2N(6.23)
(6.25)
=E[
(6.26)
=w2σ2
=1
(N+κ)2(
(6.28)
These functions are plotted in Figure 6.3 for N∈{5,20}. We see that in general, the best
estimator depends on the value of θ∗, which is unknown. If θ∗is very close to θ0, thenδ3
(which just predicts θ0) is best. If θ∗is within some reasonable range around θ0, then the
posterior mean, which combines the prior guess of θ0with the actual data, is best. If θ∗is far
fromθ0, the MLE is best. None of this should be suprising: a small amount of shrinkage (using

=1
(N+κ)2(
(6.28)
the posterior mean with a weak prior) is usually desirable, assuming our prior mean is sensible.
What is more surprising is that the risk of decision rule δ2(sample median) is always higher
than that of δ1(sample mean) for every value of θ∗. Consequently the sample median is an

6.3. Frequentist decision theory 199
inadmissible estimator for this particular problem (where the data is assumed to come from a
Gaussian).
In practice, the sample median is often better than the sample mean, because it is more
robust to outliers. One can show (Minka 2000d) that the median is the Bayes estimator (undersquared loss) if we assume the data comes from a Laplace distribution, which has heavier tailsthan a Gaussian. More generally, we can construct robust estimators by using ﬂexible modelsof our data, such as mixture models or non-parametric density estimators (Section 14.7.2), andthen computing the posterior mean or median.
6.3.3.2 Stein’s paradox *
Suppose we have Niid random variables X
i∼N(θi,1), and we want to estimate the θi. The
obvious estimator is the MLE, which in this case sets ˆθi=xi. It turns out that this is an
inadmissible estimator under quadratic loss, when N≥4.
To show this, it suffices to construct an estimator that is better. The James-Stein estimator is

one such estimator, and is deﬁned as follows:
ˆθi=ˆBx+(1−ˆB)xi=x+(1−ˆB)(xi−x) (6.29)
wherex=1
N∑N
i=1xiand0<B<1 is some tuning constant. This estimate “shrinks” the
θitowards the overall mean. (We derive this estimator using an empirical Bayes approach in
Section 5.6.2.)
It can be shown that this shrinkage estimator has lower frequentist risk (MSE) than the MLE
(sample mean) for N≥4. This is known as Stein’s paradox. The reason it is called a paradox
is illustrated by the following example. Suppose θiis the “true” IQ of student iandXiis his
test score. Why should my estimate of θidepend on the global mean x, and hence on some

other student’s scores? One can create even more paradoxical examples by making the different
dimensions be qualitatively different, e.g., θ1is my IQ, θ2is the average rainfall in Vancouver,
etc.
The solution to the paradox is the following. If your goal is to estimate just θi, you cannot do
better than using xi, but if the goal is to estimate the whole vector θ, and you use squared error
as your loss function, then shrinkage helps. To see this, suppose we want to estimate ||θ||2
2from

etc.
2from
a single sample x∼N(θ,I). A simple estimate is ||x||22, but this will overestimate the result,
since
E[
∥x∥2
2]
=E[∑
ix2i]
=N∑
i=1(
1+θ2
i)
=N+||θ||22(6.30)
Consequently we can reduce our risk by pooling information, even from unrelated sources, and
shrinking towards the overall mean. In Section 5.6.2, we give a Bayesian explanation for this.See also (Efron and Morris 1975).
6.3.3.3 Admissibility is not enough

2from
since
E[
∥x∥2
2]
=E[∑
ix2i]
=N∑
i=1(
1+θ2
i)
It seems clear that we can restrict our search for good estimators to the class of admissibleestimators. But in fact it is easy to construct admissible estimators, as we show in the followingexample.

200 Chapter 6. Frequentist statistics
Theorem 6.3.3. LetX∼N(θ,1), and consider estimating θunder squared loss. Let δ1(x)=θ 0,
a constant independent of the data. This is an admissible estimator.
Proof.Suppose not. Then there is some other estimator δ2with smaller risk, so R(θ∗,δ2)≤
R(θ∗,δ1), where the inequality must be strict for some θ∗. Suppose the true parameter is
θ∗=θ0. ThenR(θ∗,δ1)=0, and
R(θ∗,δ2)=∫
(δ2(x)−θ0)2p(x|θ0)dx (6.31)
Since0≤R(θ∗,δ2)≤R(θ∗,δ1)for allθ∗, andR(θ0,δ1)=0,w eh a v e R(θ0,δ2)=0and
henceδ2(x)=θ 0=δ1(x). Thus the only way δ2can avoid having higher risk than δ1at some
speciﬁc point θ0is by being equal to δ1. Hence there is no other estimator δ2with strictly lower
risk, soδ2is admissible.
6.4 Desirable properties of estimators
Sincefrequentistdecisiontheorydoesnotprovideanautomaticwaytochoosethebestestimator,

we need to come up with other heuristics for choosing amongst them. In this section, we discusssome properties we would like estimators to have. Unfortunately, we will see that we cannotachieve all of these properties at the same time.
6.4.1 Consistent estimators
An estimator is said to be consistent if it eventually recovers the true parameters that generated
the data as the sample size goes to inﬁnity, i.e., ˆθ(D)→θ∗as|D| → ∞ (where the arrow

denotes convergence in probability). Of course, this concept only makes sense if the data actuallycomes from the speciﬁed model with parameters θ
∗, which is not usually the case with real
data. Nevertheless, it can be a useful theoretical property.
It can be shown that the MLE is a consistent estimator. The intuitive reason is that maxi-
mizing likelihood is equivalent to minimizing KL(
p(·|θ∗)||p(·|ˆθ))
,w h e r ep(·|θ∗)is the true
distribution and p(·|ˆθ)is our estimate. We can achieve 0 KL divergence iff ˆθ=θ∗.4
6.4.2 Unbiased estimators

Thebiasof an estimator is deﬁned as
bias(ˆθ(·)) = Ep(D|θ∗)[
ˆθ(D)−θ∗]
(6.32)
whereθ∗is the true parameter value. If the bias is zero, the estimator is called unbiased. This
means the sampling distribution is centered on the true parameter. For example, the MLE for aGaussian mean is unbiased:
bias(ˆμ)=E[
x]−μ=E[
1
NN∑
i=1xi]
−μ=Nμ
N−μ=0 (6.33)
4. If the model is unidentiﬁable, the MLE may select a set of parameters that is different from the true parameters
but for which the induced distribution, p(·|ˆθ), is the same as the exact distribution. Such parameters are said to be
likelihood equivalent.

6.4. Desirable properties of estimators 201
However, the MLE for a Gaussian variance, ˆσ2, is not an unbiased estimator of σ2. In fact, one
can show (Exercise 6.3) that
E[
ˆσ2]
=N−1
Nσ2(6.34)
However, the following estimator
ˆσ2
N−1=N
N−1ˆσ2=1
N−1N∑
i=1(xi−x)2(6.35)
isan unbiased estimator, which we can easily prove as follows:
E[
ˆσ2
N−1]
=E[N
N−1ˆσ2]
=N
N−1N−1
Nσ2=σ2(6.36)
In Matlab, var(X)returnsˆσ2
N−1, whereas var(X,1) returnsˆσ2(the MLE). For large enough
N, the difference will be negligible.
Although the MLE may sometimes be a biased estimator, one can show that asymptotically, it
is always unbiased. (This is necessary for the MLE to be a consistent estimator.)
Although being unbiased sounds like a desirable property, this is not always true. See Sec-
tion 6.4.4 and (Lindley 1972) for discussion of this point.
6.4.3 Minimum variance estimators
It seems intuitively reasonable that we want our estimator to be unbiased (although we shall

give some arguments against this claim below). However, being unbiased is not enough. Forexample, suppose we want to estimate the mean of a Gaussian from D={x
1,...,x N}. The
estimator that just looks at the ﬁrst data point, ˆθ(D)=x 1, is an unbiased estimator, but will
generally be further from θ∗than the empirical mean x(which is also unbiased). So the variance
of an estimator is also important.
A natural question is: how long can the variance go? A famous result, called the Cramer-

Rao lower bound, provides a lower bound on the variance of any unbiased estimator. Moreprecisely,
Theorem 6.4.1 (Cramer-Rao inequality). LetX
1,...,X n∼p(X|θ0)andˆθ=ˆθ(x1,...,x n)be
an unbiased estimator of θ0. Then, under various smoothness assumptions on p(X|θ0),w eh a v e
var[
ˆθ]
≥1
nI(θ0)(6.37)
whereI(θ0)is the Fisher information matrix (see Section 6.2.2).
A proof can be found e.g., in (Rice 1995, p275).
It can be shown that the MLE achieves the Cramer Rao lower bound, and hence has the

var[
ˆθ]
≥1
smallest asymptotic variance of any unbiased estimator. Thus MLE is said to be asymptotically
optimal.

202 Chapter 6. Frequentist statistics
6.4.4 The bias-variance tradeoff
Although using an unbiased estimator seems like a good idea, this is not always the case. To see
why, suppose we use quadratic loss. As we showed above, the corresponding risk is the MSE.We now derive a very useful decomposition of the MSE. (All expectations and variances are wrtthe true distribution p(D|θ
∗), but we drop the explicit conditioning for notational brevity.) Let
ˆθ=ˆθ(D)denote the estimate, and θ=E[
ˆθ]
denote the expected value of the estimate (as we
varyD). Then we have
E[
(ˆθ−θ∗)2]
=E[[
(ˆθ−θ)+(θ−θ∗)]2]
(6.38)
=E[(
ˆθ−θ)2]
+2(θ−θ∗)E[
ˆθ−θ]
+(θ−θ∗)2(6.39)
=E[(
ˆθ−θ)2]
+(θ−θ∗)2(6.40)
=v a r[
ˆθ]
+bias2(ˆθ) (6.41)
In words,
MSE = variance + bias2(6.42)
This is called the bias-variance tradeoff (see e.g., (Geman et al. 1992)). What it means is that
it might be wise to use a biased estimator, so long as it reduces our variance, assuming our goalis to minimize squared error.
6.4.4.1 Example: estimating a Gaussian mean

E[
(ˆθ−θ∗)2]
=E[[
(6.38)
=E[(
ˆθ−θ)2]
ˆθ−θ]
=E[(
ˆθ−θ)2]
=v a r[
ˆθ]
In words,
Let us give an example, based on (Hoff 2009, p79). Suppose we want to estimate the mean of aGaussian from x=(x
1,...,x N). We assume the data is sampled from xi∼N(θ∗=1,σ2).
An obvious estimate is the MLE. This has a bias of 0 and a variance of
var[x|θ∗]=σ2
N(6.43)
But we could also use a MAP estimate. In Section 4.6.1, we show that the MAP estimate undera Gaussian prior of the form N(θ
0,σ2/κ0)is given by
˜x≜N
N+κ0x+κ0
N+κ0θ0=wx+(1−w)θ0 (6.44)
where0≤w≤1controls how much we trust the MLE compared to our prior. (This is also the

N(6.43)
˜x≜N
N+κ0x+κ0
posterior mean, since the mean and mode of a Gaussian are the same.) The bias and varianceare given by
E[˜x]−θ
∗=wθ0+(1−w)θ0−θ∗=( 1−w)(θ0−θ∗) (6.45)
var[˜x]=w2σ2
N(6.46)

6.4. Desirable properties of estimators 203
−1 −0.5 0 0.5 1 1.5 2 2.500.511.5sampling distribution, truth = 1.0, prior = 0.0, n = 5
  
postMean0
postMean1
postMean2
postMean3
(a)0 10 20 30 40 500.50.60.70.80.911.11.21.3
sample sizerelative MSEMSE of postmean / MSE of MLE
  
postMean0
postMean1
postMean2
postMean3
(b)
Figure 6.4 Left: Sampling distribution of the MAP estimate with different prior strengths κ0. (The MLE
corresponds to κ0=0.) Right: MSE relative to that of the MLE versus sample size. Based on Figure 5.6 of
(Hoff 2009). Figure generated by samplingDistGaussShrinkage .
So although the MAP estimate is biased (assuming w<1), it has lower variance.
Let us assume that our prior is slightly misspeciﬁed, so we use θ0=0, whereas the truth is
θ∗=1. In Figure 6.4(a), we see that the sampling distribution of the MAP estimate for κ0>0
is biased away from the truth, but has lower variance (is narrower) than that of the MLE.

In Figure 6.4(b), we plot mse(˜x)/mse(x)vsN. We see that the MAP estimate has lower MSE
than the MLE, especially for small sample size, for κ0∈{1,2}. The case κ0=0corresponds to
the MLE, and the case κ0=3corresponds to a strong prior, which hurts performance because
the prior mean is wrong. It is clearly important to “tune” the strength of the prior, a topic we
discuss later.
6.4.4.2 Example: ridge regression

Another important example of the bias variance tradeoff arises in ridge regression, which wediscuss in Section 7.5. In brief, this corresponds to MAP estimation for linear regression undera Gaussian prior, p(w)=N(w|0,λ
−1I)The zero-mean prior encourages the weights to be
small, which reduces overﬁtting; the precision term, λ, controls the strength of this prior. Setting
λ=0results in the MLE; using λ>0results in a biased estimate. To illustrate the effect on

the variance, consider a simple example. Figure 6.5 on the left plots each individual ﬁtted curve,and on the right plots the average ﬁtted curve. We see that as we increase the strength of theregularizer, the variance decreases, but the bias increases.
6.4.4.3 Bias-variance tradeoff for classiﬁcation

If we use 0-1 loss instead of squared error, the above analysis breaks down, since the frequentistrisk is no longer expressible as squared bias plus variance. In fact, one can show (Exercise 7.2of (Hastie et al. 2009)) that the bias and variance combine multiplicatively. If the estimate is on

204 Chapter 6. Frequentist statistics
0 0.2 0.4 0.6 0.8 1−1.5−1−0.500.511.5ln(λ) = 5
0 0.2 0.4 0.6 0.8 1−1−0.500.51ln(λ) = 5
0 0.2 0.4 0.6 0.8 1−1.5−1−0.500.511.5ln(λ) = −5
0 0.2 0.4 0.6 0.8 1−1−0.500.51ln(λ) = −5
Figure 6.5 Illustration of bias-variance tradeoff for ridge regression. We generate 100 data sets from the
true function, shown in solid green. Left: we plot the regularized ﬁt for 20 different data sets. We use
linear regression with a Gaussian RBF expansion, with 25 centers evenly spread over the [0,1]interval.
Right: we plot the average of the ﬁts, averaged over all 100 datasets. Top row: strongly regularized: we seethat the individual ﬁts are similar to each other (low variance), but the average is far from the truth (highbias). Bottom row: lightly regularized: we see that the individual ﬁts are quite different from each other(high variance), but the average is close to the truth (low bias). Based on (Bishop 2006a) Figure 3.5. Figuregenerated by biasVarModelComplexity3 .

the correct side of the decision boundary, then the bias is negative, and decreasing the variance
will decrease the misclassiﬁcation rate. But if the estimate is on the wrong side of the decisionboundary, then the bias is positive, so it pays to increasethe variance (Friedman 1997a). This

little known fact illustrates that the bias-variance tradeoff is not very useful for classiﬁcation.It is better to focus on expected loss (see below), not directly on bias and variance. We canapproximate the expected loss using cross validatinon, as we discuss in Section 6.5.3.
6.5 Empirical risk minimization
Frequentist decision theory suffers from the fundamental problem that one cannot actuallycompute the risk function, since it relies on knowing the true data distribution. (By contrast,the Bayesian posterior expected loss can always be computed, since it conditions on the thedata rather than conditioning on θ
∗.) However, there is one setting which avoids this problem,

and that is where the task is to predict observable quantities, as opposed to estimating hiddenvariables or parameters. That is, instead of looking at loss functions of the form L(θ,δ(D)),
whereθis the true but unknown parameter, and δ(D)is our estimator, let us look at loss

6.5. Empirical risk minimization 205
functions of the form L(y,δ(x)),w h e r e yis the true but unknown response, and δ(x)is our
prediction given the input x. In this case, the frequentist risk becomes
R(p∗,δ)≜E(x,y)∼p∗[L(y,δ(x)] =∑
x∑
yL(y,δ(x))p∗(x,y) (6.47)
wherep∗represents “nature’s distribution”. Of course, this distribution is unknown, but a simple
approach is to use the empirical distribution, derived from some training data, to approximate
p∗, i.e.,
p∗(x,y)≈pemp(x,y)≜1
NN∑
i=1δxi(x)δyi(y) (6.48)
We then deﬁne the empirical risk as follows:
Remp(D,D)≜R(pemp,δ)=1
NN∑
i=1L(yi,δ(xi)) (6.49)
In the case of 0-1 loss, L(y,δ(x)) =I(y̸=δ(x)), this becomes the misclassiﬁcation rate.I n
the case of squared error loss, L(y,δ(x)) = (y−δ(x))2, this becomes the mean squared error.
We deﬁne the task of empirical risk minimization orERMas ﬁnding a decision procedure
(typically a classiﬁcation rule) to minimize the empirical risk:
δERM(D) = argmin
δRemp(D,δ) (6.50)

NN∑
NN∑
In the unsupervised case, we eliminate all references to y, and replace L(y,δ(x))with
L(x,δ(x)), where, for example, L(x,δ(x)) =||x−δ(x)||2
2, which measures the reconstruc-
tion error. We can deﬁne the decision rule using δ(x)=decode( encode(x)), as in vector
quantization (Section 11.4.2.6) or PCA (section 12.2). Finally, we deﬁne the empirical risk as
Remp(D,δ)=1
NN∑
i=1L(xi,δ(xi)) (6.51)
Of course, we can always trivially minimize this risk by setting δ(x)=x, so it is critical that

NN∑
the encoder-decoder go via some kind of bottleneck.
6.5.1 Regularized risk minimization
Note that the empirical risk is equal to the Bayes risk if our prior about “nature’s distribution” is
that it is exactly equal to the empirical distribution (Minka 2001b):
E[R(p∗,δ)|p∗=pemp]=Remp(D,δ) (6.52)
Therefore minimizing the empirical risk will typically result in overﬁtting. It is therefore oftennecessary to add a complexity penalty to the objective function:
R
′(D,δ)=Remp(D,δ)+λC(δ) (6.53)

206 Chapter 6. Frequentist statistics
whereC(δ)measures the complexity of the prediction function δ(x)andλcontrols the strength
of the complexity penalty. This approach is known as regularized risk minimization (RRM).
Note that if the loss function is negative log likelihood, and the regularizer is a negative log
prior, this is equivalent to MAP estimation.
The two key issues in RRM are: how do we measure complexity, and how do we pick λ.F o r
a linear model, we can deﬁne the complexity of in terms of its degrees of freedom, discussed inSection 7.5.3. For more general models, we can use the VC dimension, discussed in Section 6.5.4.To pickλ, we can use the methods discussed in Section 6.5.2.
6.5.2 Structural risk minimization
The regularized risk minimization principle says that we should ﬁt the model, for a givencomplexity penalty, by using
ˆδ
λ=a r g m i n
δ[Remp(D,δ)+λC(δ)] (6.54)
But how should we pick λ? We cannot using the training set, since this will underestimate the

ˆδ
λ=a r g m i n
true risk, a problem known as optimism of the training error. As an alternative, we can use
the following rule, known as the structural risk minimization principle: (Vapnik 1998):
ˆλ=a r g m i n
λˆR(ˆδλ) (6.55)
whereˆR(δ)is an estimate of the risk. There are two widely used estimates: cross validation
and theoretical upper bounds on the risk. We discuss both of these below.
6.5.3 Estimating the risk using cross validation
We can estimate the risk of some estimator using a validation set. If we don’t have a separatevalidation set, we can use cross validation (CV), as we brieﬂy discussed in Section 1.4.8. More
precisely, CV is deﬁned as follows. Let there be N=|D|data cases in the training set. Denote

the data in the k’th test fold by D
kand all the other data by D−k. (Instratiﬁed CV, these folds
are chosen so the class proportions (if discrete labels are present) are roughly equal in eachfold.) Let Fbe a learning algorithm or ﬁtting function that takes a dataset and a model index
m(this could a discrete index, such as the degree of a polynomial, or a continuous index, such
as the strength of a regularizer) and returns a parameter vector:
ˆθ
m=F(D,m) (6.56)

ˆθ
Finally, let Pbe a prediction function that takes an input and a parameter vector and returns a
prediction:
ˆy=P(x,ˆθ)=f(x,ˆθ) (6.57)
Thus the combined ﬁt-predict cycle is denoted as
fm(x,D)=P(x,F(D,m)) (6.58)

6.5. Empirical risk minimization 207
TheK-fold CV estimate of the risk of fmis deﬁned by
R(m,D,K)≜1
NK∑
k=1∑
i∈DkL(yi,P(xi,F(D−k,m))) (6.59)
Note that we can call the ﬁtting algorithm once per fold. Let fk
m(x)=P(x,F(D−k,m))be
the function that was trained on all the data except for the test data in fold k. Then we can
rewrite the CV estimate as
R(m,D,K)=1
NK∑
k=1∑
i∈DkL(
yi,fk
m(xi))
=1
NN∑
i=1L(
yi,fk(i)
m(xi))
(6.60)
wherek(i)is the fold in which iis used as test data. In other words, we predict yiusing a
model that was trained on data that does not contain xi.
OfK=N, the method is known as leave one out cross validation or LOOCV. In this case,
n the estimated risk becomes
R(m,D,N)=1
NN∑
i=1L(
yi,f−i
m(xi))
(6.61)
wherefi
m(x)=P(x,F(D−i,m)). This requires ﬁtting the model Ntimes, where for f−i
mwe
omit thei’th training case. Fortunately, for some model classes and loss functions (namely linear
models and quadratic loss), we can ﬁt the model once, and analytically “remove” the effect of

NN∑
i=1L(
yi,f−i
m(xi))
(6.61)
wherefi
mwe
thei’th training case. This is known as generalized cross validation or GCV.
6.5.3.1 Example: using CV to pick λfor ridge regression
As a concrete example, consider picking the strength of the ℓ2regularizer in penalized linear
regression. We use the following rule:
ˆλ=a r g m i n
λ∈[λmin,λmax]R(λ,Dtrain,K) (6.62)
where[λmin,λmax]is a ﬁnite range of λvalues that we search over, and R(λ,Dtrain,K)is the
K-fold CV estimate of the risk of using λ, given by
R(λ,Dtrain,K)=1
|Dtrain|K∑
k=1∑
i∈DkL(yi,fk
λ(xi)) (6.63)
wherefk

ˆλ=a r g m i n
k=1∑
wherefk
λ(x)=xTˆwλ(D−k)is the prediction function trained on data excluding fold k, and
ˆwλ(D)=a r gm i n wNLL(w,D)+λ||w||2
2is the MAP estimate. Figure 6.6(b) gives an example
of a CV estimate of the risk vs log(λ), where the loss function is squared error.
When performing classiﬁcation, we usually use 0-1 loss. In this case, we optimize a convex
upper bound on the empirical risk to estimate wλm but we optimize (the CV estimate of) the
risk itself to estimate λ. We can handle the non-smooth 0-1 loss function when estimating λ

because we are using brute-force search over the entire (one-dimensional) space.
When we have more than one or two tuning parameters, this approach becomes infeasible.
In such cases, one can use empirical Bayes, which allows one to optimize large numbers of
hyper-parameters using gradient-based optimizers instead of brute-force search. See Section 5.6for details.

208 Chapter 6. Frequentist statistics
−25 −20 −15 −10 −5 0 502468101214
log lambdamean squared error
  
train mse
test mse
(a)−15 −10 −5 0 502468101214161820
log lambdamse5−fold cross validation, ntrain = 50
(b)
Figure 6.6 (a) Mean squared error for ℓ2penalized degree 14 polynomial regression vs log regularizer.
Same as in Figures 7.8, except now we have N=5 0training points instead of 21. The stars correspond
to the values used to plot the functions in Figure 7.7. (b) CV estimate. The vertical scale is truncated for
clarity. The blue line corresponds to the value chosen by the one standard error rule. Figure generated bylinregPolyVsRegDemo .
6.5.3.2 The one standard error rule
The above procedure estimates the risk, but does not give any measure of uncertainty. A
standard frequentist measure of uncertainty of an estimate is the standard error of the mean,deﬁned by
se=ˆσ
√
N=√
ˆσ2
N(6.64)
whereˆσ2is an estimate of the variance of the loss:
ˆσ2=1
NN∑
i=1(Li−L)2,Li=L(yi,fk(i)
m(xi))L=1
NN∑
i=1Li (6.65)

se=ˆσ
√
N=√
ˆσ2
N(6.64)
ˆσ2=1
NN∑
m(xi))L=1
NN∑
Note that σmeasures the intrinsic variability of Liacross samples, whereas semeasures our
uncertainty about the mean L.
Suppose we apply CV to a set of models and compute the mean and se of their estimated
risks. A common heuristic for picking a model from these noisy estimates is to pick the valuewhich corresponds to the simplest model whose risk is no more than one standard error abovethe risk of the best model; this is called the one-standard error rule (Hastie et al. 2001, p216).

NN∑
For example, in Figure 6.6, we see that this heuristic does not choose the lowest point on thecurve, but one that is slightly to its right, since that corresponds to a more heavily regularizedmodel with essentially the same empirical performance.

6.5. Empirical risk minimization 209
6.5.3.3 CV for model selection in non-probabilistic unsupervised learning
If we are performing unsupervised learning, we must use a loss function such as L(x,δ(x)) =
||x−δ(x)||2, which measures reconstruction error. Here δ(x)is some encode-decode scheme.
However, as we discussed in Section 11.5.2, we cannot use CV to determine the complexity of δ,
since we will always get lower loss with a more complex model, even if evaluated on the test set.
This is because more complex models will compress the data less, and induce less distortion.Consequently, we must either use probabilistic models, or invent other heuristics.
6.5.4 Upper bounding the risk using statistical learning theory *
The principle problem with cross validation is that it is slow, since we have to ﬁt the modelmultiple times. This motivates the desire to compute analytic approximations or bounds tothe generalization error. This is the studied in the ﬁeld of statistical learning theory (SLT).

More precisely, SLT tries to bound the risk R(p
∗,h)for any data distribution p∗and hypothesis
h∈Hin terms of the empirical risk Remp(D,h), the sample size N=|D|, and the size of the
hypothesis space H.
Let us initially consider the case where the hypothesis space is ﬁnite, with size dim(H)=|H|.
In other words, we are selecting a model/ hypothesis from a ﬁnite list, rather than optimizingreal-valued parameters, Then we can prove the following.
Theorem 6.5.1. For any data distribution p
∗, and any dataset Dof sizeNdrawn from p∗, the
probability that our estimate of the error rate will be more than ϵwrong, in the worst case, is upper
bounded as follows:
P(
max

P(
max
h∈H|Remp(D,h)−R(p∗,h)|>ϵ)
≤2dim(H)e−2Nϵ2(6.66)
Proof.To prove this, we need two useful results. First, Hoeffding’s inequality, which states that
ifX1,...,X N∼Ber(θ), then, for any ϵ>0,
P(|x−θ|>ϵ)≤2e−2Nϵ2(6.67)
wherex=1
N∑N
i=1xi. Second, the union bound, which says that if A1,...,A dare a set of
events, then P(∪d
i=1Ai)≤∑d
i=1P(Ai).
Finally, fornotationalbrevity, let R(h)=R(h,p∗)bethetruerisk, and ˆRN(h)=Remp(D,h)
be the empirical risk.
Using these results we have
P(
max
h∈H|ˆRN(h)−R(h)|>ϵ)
=P(⋃
(6.68)
≤∑
h∈HP(

wherex=1
N∑N
i=1Ai)≤∑d
i=1P(Ai).
P(
max
=P(⋃
(6.68)
≤∑
h∈HP(
|ˆRN(h)−R(h)|>ϵ)
(6.69)
≤∑
h∈H2e−2Nϵ2=2d i m ( H)e−2Nϵ2(6.70)

210 Chapter 6. Frequentist statistics
Ths bound tells us that the optimism of the training error increases with dim(H)but de-
creases with N=|D|, as is to be expected.
If the hypothesis space His inﬁnite (e.g., we have real-valued parameters), we cannot use
dim(H)=|H|. Instead, we can use a quantity called the Vapnik-Chervonenkis orVCdimen-
sion of the hypothesis class. See (Vapnik 1998) for details.
Stepping back from all the theory, the key intuition behind statistical learning theory is quite
simple. Suppose we ﬁnd a model with low empirical risk. If the hypothesis space His very
big, relative to the data size, then it is quite likely that we just got “lucky” and were given a

data set that is well-modeled by our chosen function by chance. However, this does not meanthat such a function will have low generalization error. But if the hypothesis class is sufficientlyconstrained in size, and/or the training set is sufficiently large, then we are unlikely to get luckyin this way, so a low empirical risk is evidence of a low true risk.
Note that optimism of the training error does not necessarily increase with model complexity,
but it does increase with the number of different models that are being searched over.

The advantage of statistical learning theory compared to CV is that the bounds on the risk
are quicker to compute than using CV. The disadvantage is that it is hard to compute the VCdimension for many interesting models, and the upper bounds are usually very loose (althoughsee (Kaariainen and Langford 2005)).
One can extend statistical learning theory by taking computational complexity of the learner
into account. This ﬁeld is called computational learning theory orCOLT. Most of this work
focuses on the case where his a binary classiﬁer, and the loss function is 0-1 loss. If we observe
a low empirical risk, and the hypothesis space is suitably “small”, then we can say that ourestimated function is probably approximately correct orPAC. A hypothesis space is said to be

efficiently PAC-learnable if there is a polynomial time algorithm that can identify a function
that is PAC. See (Kearns and Vazirani 1994) for details.
6.5.5 Surrogate loss functions

Minimizing the loss in the ERM/ RRM framework is not always easy. For example, we mightwant to optimize the AUC or F1 scores. Or more simply, we might just want to minimize the 0-1loss, as is common in classiﬁcation. Unfortunately, the 0-1 risk is a very non-smooth objectiveand hence is hard to optimize. One alternative is to use maximum likelihood estimation instead,since log-likelihood is a smooth convex upper bound on the 0-1 risk, as we show below.
To see this, consider binary logistic regression, and let y
i∈{ −1,+1}. Suppose our decision

function computes the log-odds ratio,
f(xi)=l o gp(y=1|xi,w)
p(y=−1|xi,w)=wTxi=ηi (6.71)
Then the corresponding probability distribution on the output label is
p(yi|xi,w) = sigm(y iηi) (6.72)
Let us deﬁne the log-lossas as
Lnll(y,η)=−logp(y|x,w)=l o g ( 1+ e−yη) (6.73)

6.6. Pathologies of frequentist statistics * 211
−2 −1.5 −1 −0.5 0 0.5 1 1.5 200.511.522.53
ηloss
  
0−1
hinge
logloss
Figure 6.7 Illustration of various loss functions for binary classiﬁcation. The horizontal axis is the margin
yη, the vertical axis is the loss. The log loss uses log base 2. Figure generated by hingeLossPlot .
It is clear that minimizing the average log-loss is equivalent to maximizing the likelihood.
Now consider computing the most probable label, which is equivalent to using ˆy=−1if
ηi<0andˆy=+ 1ifηi≥0. The 0-1 loss of our function becomes
L01(y,η)= I(y̸=ˆy)=I(yη <0) (6.74)
Figure 6.7 plots these two loss functions. We see that the NLL is indeed an upper bound on the
0-1 loss.
Log-loss is an example of a surrogate loss function. Another example is the hinge loss:
Lhinge(y,η)=m a x ( 0 ,1−yη) (6.75)

0-1 loss.
See Figure 6.7 for a plot. We see that the function looks like a door hinge, hence its name.This loss function forms the basis of a popular classiﬁcation method known as support vectormachines (SVM), which we will discuss in Section 14.5.
The surrogate is usually chosen to be a convex upper bound, since convex functions are easy
to minimize. See e.g., (Bartlett et al. 2006) for more information.
6.6 Pathologies of frequentist statistics *

I believe that it would be very difficult to persuade an intelligent person that current[frequentist] statistical practice was sensible, but that there would be much less difficultywith an approach via likelihood and Bayes’ theorem. — George Box, 1962.

Frequentist statistics exhibits various forms of weird and undesirable behaviors, known aspathologies. We give a few examples below, in order to caution the reader; these and otherexamples are explained in more detail in (Lindley 1972; Lindley and Phillips 1976; Lindley 1982;Berger 1985; Jaynes 2003; Minka 1999).

212 Chapter 6. Frequentist statistics
6.6.1 Counter-intuitive behavior of conﬁdence intervals
Aconﬁdence interval is an interval derived from the sampling distribution of an estimator
(whereas a Bayesian credible interval is derived from the posterior of a parameter, as we dis-
cussed in Section 5.2.2). More precisely, a frequentist conﬁdence interval for some parameter θ
is deﬁned by the following (rather un-natural) expression:
C′
α(θ)=(ℓ,u):P(ℓ(˜D)≤θ≤u(˜D)|˜D∼θ)=1−α (6.76)
That is, if we sample hypothetical future data ˜Dfromθ, then(ℓ(˜D),u(˜D)), is a conﬁdence
interval if the parameter θlies inside this interval 1−αpercent of the time.
Let us step back for a moment and think about what is going on. In Bayesian statistics,
we condition on what is known — namely the observed data, D— and average over what
is not known, namely the parameter θ. In frequentist statistics, we do exactly the opposite:
we condition on what is unknown — namely the true parameter value θ— and average over

hypothetical future data sets ˜D.
This counter-intuitive deﬁnition of conﬁdence intervals can lead to bizarre results. Consider
the following example from (Berger 1985, p11). Suppose we draw two integers D=(x1,x2)from
p(x|θ)=⎧
⎨
⎩0.5ifx=θ
0.5ifx=θ+1
0otherwise(6.77)
Ifθ=3 9, we would expect the following outcomes each with probability 0.25:
(39,39),(39,40),(40,39),(40,40) (6.78)
Letm=m i n (x1,x2)and deﬁne the following conﬁdence interval:
[ℓ(D),u(D) ]=[m,m] (6.79)
For the above samples this yields
[39,39],[39,39],[39,39],[40,40] (6.80)

p(x|θ)=⎧
⎨
⎩0.5ifx=θ
Hence Equation 6.79 is clearly a 75% CI, since 39is contained in 3/4 of these intervals. However,
ifD=( 3 9,40)thenp(θ=3 9|D)=1.0, so we know that θmust be 39, yet we only have 75%
“conﬁdence” in this fact.
Another, less contrived example, is as follows. Suppose we want to estimate the parameter θ
of a Bernoulli distribution. Let x=1
N∑N
i=1xibe the sample mean. The MLE is ˆθ=x.A n
approximate 95% conﬁdence interval for a Bernoulli parameter is x±1.96√
x(1−x)/N(this is

N∑N
called aWald interval and is based on a Gaussian approximation to the Binomial distribution;
compare to Equation 3.27). Now consider a single trial, where N=1andx1=0. The MLE
is 0, which overﬁts, as we saw in Section 3.3.4.1. But our 95% conﬁdence interval is also (0,0),
which seems even worse. It can be argued that the above ﬂaw is because we approximated
the true sampling distribution with a Gaussian, or because the sample size was to small, or theparameter “too extreme”. However, the Wald interval can behave badly even for large N, and

non-extreme parameters (Brown et al. 2001).

6.6. Pathologies of frequentist statistics * 213
6.6.2 p-values considered harmful
Suppose we want to decide whether to accept or reject some baseline model, which we will
call thenull hypothesis. We need to deﬁne some decision rule. In frequentist statistics, it
is standard to ﬁrst compute a quantity called the p-value, which is deﬁned as the probability
(under the null) of observing some test statistic f(D)(such as the chi-squared statistic) that is
as large or largerthan that actually observed:5
pvalue(D )≜P(f(˜D)≥f(D)|˜D∼H0) (6.81)
This quantity relies on computing a tail area probability of the sampling distribution; we give
an example of how to do this below.
Given the p-value, we deﬁne our decision rule as follows: we reject the null hypothesis iff the
p-value is less than some threshold, such as α=0.05. If we do reject it, we say the difference
between the observed test statistic and the expected test statistic is statistically signiﬁcant at

levelα. This approach is known as null hypothesis signiﬁcance testing,o rNHST.
This procedure guarantees that our expected type I (false positive) error rate is at most α.

This is sometimes interpreted as saying that frequentist hypothesis testing is very conservative,since it is unlikely to accidently reject the null hypothesis. But in fact the opposite is the case:because this method only worries about trying to reject the null, it can never gather evidencein favor of the null, no matter how large the sample size. Because of this, p-values tend tooverstate the evidence against the null, and are thus very “trigger happy”.

In general there can be huge differences between p-values and the quantity that we really
care about, which is the posterior probability of the null hypothesis given the data, p(H
0|D).
In particular, Sellke et al. (2001) show that even if the p-value is as slow as 0.05, the posteriorprobability of H
0is at least 30%, and often much higher. So frequentists often claim to have

0|D).
“signiﬁcant” evidence of an effect that cannot be explained by the null hypothesis, whereasBayesians are usually more conservative in their claims. For example, p-values have been usedto “prove” that ESP (extra-sensory perception) is real (Wagenmakers et al. 2011), even though ESPis clearly very improbable. For this reason, p-values have been banned from certain medicaljournals (Matthews 1998).
Another problem with p-values is that their computation depends on decisions you make

about when to stop collecting data, even if these decisions don’t change the data you actuallyobserved. For example, suppose I toss a coin n=1 2times and observe s=9successes (heads)
andf=3failures (tails), so n=s+f. In this case, nis ﬁxed and s(and hence f) is random.
The relevant sampling model is the binomial
Bin(s|n,θ)=(n
s)
θ
s(1−θ)n−s(6.82)
Let the null hypothesis be that the coin is fair, θ=0.5,w h e r e θis the probability of success
(heads). The one-sided p-value, using test statistic t(s)=s,i s
p1=P(S≥9|H0)=12∑

s)
θ
s=9Bin(s|12,0.5) =12∑
s=9(12
s)
0.512=0.073 (6.83)
5. The reason we cannot just compute the probability of the observed value of the test statistic is that this will have
probability zero under a pdf. The p-value is deﬁned in terms of the cdf, so is always a number between 0 and 1.

214 Chapter 6. Frequentist statistics
The two-sided p-value is
p2=12∑
s=9Bin(s|12,0.5)+3∑
s=0Bin(s|12,0.5) = 0.073+0.073 = 0.146 (6.84)
In either case, the p-value is larger than the magical 5% threshold, so a frequentist would not
reject the null hypothesis.
Now suppose I told you that I actually kept tossing the coin until I observed f=3tails. In
this case, fis ﬁxed and n(and hence s=n−f) is random. The probability model becomes
thenegative binomial distribution, given by
NegBinom(s |f,θ)=(s+f−1
f−1)
θs(1−θ)f(6.85)
wheref=n−s.
Note that the term which depends on θis the same in Equations 6.82 and 6.85, so the
posterior over θwould be the same in both cases. However, these two interpretations of the
same data give different p-values. In particular, under the negative binomial model we get
p3=P(S≥9|H0)=∞∑
s=9(3+s−1
2)
(1/2)s(1/2)3=0.0327 (6.86)

f−1)
s=9(3+s−1
2)
So the p-value is 3%, and suddenly there seems to be signiﬁcant evidence of bias in the coin!Obviously this is ridiculous: the data is the same, so our inferences about the coin should bethe same. After all, I could have chosen the experimental protocol at random. It is the outcomeof the experiment that matters, not the details of how I decided which one to run.
Although this might seem like just a mathematical curiosity, this also has signiﬁcant practical

s=9(3+s−1
2)
implications. In particular, the fact that the stopping rule affects the computation of the p-

value means that frequentists often do not terminate experiments early, even when it is obviouswhat the conclusions are, lest it adversely affect their statistical analysis. If the experiments arecostly or harmful to people, this is obviously a bad idea. Perhaps it is not surprising, then, thatthe US Food and Drug Administration (FDA), which regulates clinical trials of new drugs, hasrecently become supportive of Bayesian methods
6, since Bayesian methods are not affected by
the stopping rule.
6.6.3 The likelihood principle
The fundamental reason for many of these pathologies is that frequentist inference violatesthelikelihood principle, which says that inference should be based on the likelihood of the

observed data, not based on hypothetical future data that you have not observed. Bayes obviouslysatisﬁes the likelihood principle, and consequently does not suffer from these pathologies.
A compelling argument in favor of the likelihood principle was presented in (Birnbaum 1962),
who showed that it followed automatically from two simpler principles. The ﬁrst of these is thesufficiency principle, which says that a sufficient statistic contains all the relevant information
6. See http://yamlb .wordpress .com/2006/06/19/the-us-fda-is-becoming-progressively-more-bayes
ian/.

6.6. Pathologies of frequentist statistics * 215
about an unknown parameter (arguably this is true by deﬁnition). The second principle is
known as weak conditionality, which says that inferences should be based on the events that

happened, not which might have happened. To motivate this, consider an example from (Berger1985). Suppose we need to analyse a substance, and can send it either to a laboratory in NewYork or in California. The two labs seem equally good, so a fair coin is used to decide betweenthem. The coin comes up heads, so the California lab is chosen. When the results come back,should it be taken into account that the coin could have come up tails and thus the New Yorklab could have been used? Most people would argue that the New York lab is irrelevant, sincethe tails event didn’t happen. This is an example of weak conditionality. Given this principle,one can show that all inferences should only be based on what was observed, which is incontrast to standard frequentist procedures. See (Berger and Wolpert 1988) for further details onthe likelihood principle.
6.6.4 Why isn’t everyone a Bayesian?

Given these fundamental ﬂaws of frequentist statistics, and the fact that Bayesian methodsdo not have such ﬂaws, an obvious question to ask is: “Why isn’t everyone a Bayesian?” The(frequentist) statistician Bradley Efron wrote a paper with exactly this title (Efron 1986). His shortpaper is well worth reading for anyone interested in this topic. Below we quote his openingsection:
The title is a reasonable question to ask on at least two counts. First of all, everone usedto be a Bayesian. Laplace wholeheatedly endorsed Bayes’s formulation of the inferenceproblem, and most 19th-century scientists followed suit. This included Gauss, whosestatistical work is usually presented in frequentist terms.
A second and more important point is the cogency of the Bayesian argument. Modern

statisticians, following the lead of Savage and de Finetti, have advanced powerful theoret-ical arguments for preferring Bayesian inference. A byproduct of this work is a disturbingcatalogue of inconsistencies in the frequentist point of view.
Nevertheless, everyone is not a Bayesian. The current era (1986) is the ﬁrst century in
which statistics has been widely used for scientiﬁc reporting, and in fact, 20th-centurystatistics is mainly non-Bayesian. However, Lindley (1975) predicts a change for the 21stcentury.
Time will tell whether Lindley was right....
Exercises
Exercise 6.1 Pessimism of LOOCV

Exercises
(Source: Witten05, p152.). Suppose we have a completely random labeled dataset (i.e., the features xtell us
nothing about the class labels y) withN1examples of class 1, and N2examples of class 2, where N1=N2.
What is the best misclassiﬁcation rate any method can achieve? What is the estimated misclassiﬁcation
rate of the same method using LOOCV?
Exercise 6.2 James Stein estimator for Gaussian means
Consider the 2 stage model Yi|θi∼N(θi,σ2)andθi|μ∼N(m0,τ2
0). Suppose σ2= 500is known and
we observe the following 6 data points, i=1:6:

216 Chapter 6. Frequentist statistics
1505, 1528, 1564, 1498, 1600, 1470
a. Find the ML-II estimates of m0andτ2
0.
b. Find the posterior estimates E[θi|yi,m0,τ0]andvar[θi|yi,m0,τ0]fori=1. (The other terms,
i=2:6, are computed similarly.)
c. Give a 95% credible interval for p(θi|yi,m0,τ0)fori=1. Do you trust this interval (assuming the
Gaussian assumption is reasonable)? i.e. is it likely to be too large or too small, or just right?
d. What do you expect would happen to your estimates if σ2were much smaller (say σ2=1)? You do
not need to compute the numerical answer; just brieﬂy explain what would happen qualitatively, and
why.
Exercise 6.3 ˆσ2
MLEis biased
Show that ˆσ2
MLE=1
N∑N
n=1(xn−ˆμ)2is a biased estimator of σ2, i.e., show
EX1,...,X n∼N(μ,σ)[ˆσ2(X1,...,X n)̸=σ2
Hint: note that X1,...,X Nare independent, and use the fact that the expectation of a product of
independent random variables is the product of the expectations.
Exercise 6.4 Estimation of σ2whenμis known

why.
MLE=1
N∑N
Suppose we sample x1,...,x N∼N(μ,σ2)whereμis aknownconstant. Derive an expression for the
MLE forσ2in this case. Is it unbiased?

7 Linear regression
7.1 Introduction
Linear regression is the “work horse” of statistics and (supervised) machine learning. When
augmented with kernels or other forms of basis function expansion, it can model also non-linear relationships. And when the Gaussian output is replaced with a Bernoulli or multinoullidistribution, it can be used for classiﬁcation, as we will see below. So it pays to study this modelin detail.
7.2 Model speciﬁcation
As we discussed in Section 1.4.5, linear regression is a model of the form
p(y|x,θ)=N(y|wTx,σ2) (7.1)
p(y|x,θ)=N(y|wTφ(x),σ2) (7.2)
This is known as basis function expansion. (Note that the model is still linear in the parameters
w, so it is still called linear regression; the importance of this will become clear below.) A simple
example are polynomial basis functions, where the model has the form
φ(x)=[ 1,x,x2,...,xd] (7.3)

Figure 1.18 illustrates the effect of changing d: increasing the degree dallows us to create
increasingly complex functions.
We can also apply linear regression to more than 1 input. For example, consider modeling
temperature as a function of location. Figure 7.1(a) plots E[y|x]=w0+w1x1+w2x2, and
Figure 7.1(b) plots E[y|x]=w0+w1x1+w2x2+w3x2
1+w4x22.
7.3 Maximum likelihood estimation (least squares)
A common way to esitmate the parameters of a statistical model is to compute the MLE, which
is deﬁned as
ˆθ≜argmax

1+w4x22.
ˆθ≜argmax
θlogp(D|θ) (7.4)

218 Chapter 7. Linear regression
010203040
05101520253015.51616.51717.518
(a)010203040
01020301515.51616.51717.518
(b)
Figure 7.1 Linear regression applied to 2d data. Vertical axis is temperature, horizontal axes are location
within a room. Data was collected by some remote sensing motes at Intel’s lab in Berkeley, CA (data
courtesy of Romain Thibaux). (a) The ﬁtted plane has the form ˆf(x)=w 0+w1x1+w2x2. (b)
Temperature data is ﬁtted with a quadratic of the form ˆf(x)=w 0+w1x1+w2x2+w3x2
1+w4x22.
Produced by surfaceFitDemo .
It is common to assume the training examples are independent and identically distributed,
commonly abbreviated to iid. This means we can write the log-likelihood as follows:
ℓ(θ)≜logp(D|θ)=N∑
i=1logp(yi|xi,θ) (7.5)
Instead of maximizing the log-likelihood, we can equivalently minimize the negative log likeli-
hoodorNLL:
NLL(θ)≜−N∑
i=1logp(yi|xi,θ) (7.6)

1+w4x22.
The NLL formulation is sometimes more convenient, since many optimization software packagesare designed to ﬁnd the minima of functions, rather than maxima.
Now let us apply the method of MLE to the linear regression setting. Inserting the deﬁnition
of the Gaussian into the above, we ﬁnd that the log likelihood is given by
ℓ(θ)= N∑
i=1log[(1
2πσ2)1
2
exp(
−1
2σ2(yi−wTxi)2)]
(7.7)
=−1
2σ2RSS(w)−N
2log(2πσ2) (7.8)
RSS stands for residual sum of squares and is deﬁned by
RSS(w)≜N∑
i=1(yi−wTxi)2(7.9)

ℓ(θ)= N∑
i=1log[(1
2πσ2)1
2
exp(
−1
(7.7)
=−1
RSS(w)≜N∑
The RSS is also called the sum of squared errors, or SSE, and SSE/N is called the mean
squared error orMSE. It can also be written as the square of the ℓ2normof the vector of

7.3. Maximum likelihood estimation (least squares) 219
−4 −3 −2 −1 0 1 2 3 4−3−2−1012345
prediction
truth
(a)Sum of squares error contours for linear regression
w0w1
−1 0 1 2 3−1−0.500.511.522.53
(b)
Figure 7.2 (a) In linear least squares, we try to minimize the sum of squared distances from each training
point (denoted by a red circle) to its approximation (denoted by a blue cross), that is, we minimize the
sum of the lengths of the little vertical blue lines. The red diagonal line represents ˆy(x)=w 0+w1x,
which is the least squares regression line. Note that these residual lines are not perpendicular to the leastsquares line, in contrast to Figure 12.5. Figure generated by residualsDemo . (b) Contours of the RSS error
surface for the same example. The red cross represents the MLE, w=( 1.45,0.93). Figure generated by
contoursSSEdemo .
residual errors:
RSS(w)=||ϵ||2
2=N∑
i=1ϵ2i(7.10)
whereϵi=(yi−wTxi).
We see that the MLE for wis the one that minimizes the RSS, so this method is known

2=N∑
asleast squares. This method is illustrated in Figure 7.2(a). The training data (xi,yi)are
shown as red circles, the estimated values (xi,ˆyi)are shown as blue crosses, and the residuals
ϵi=yi−ˆyiare shown as vertical blue lines. The goal is to ﬁnd the setting of the parameters
(the slope w1and intercept w0) such that the resulting red line minimizes the sum of squared
residuals (the lengths of the vertical blue lines).
In Figure 7.2(b), we plot the NLL surface for our linear regression example. We see that it is a

quadratic “bowl” with a unique minimum, which we now derive. (Importantly, this is true even
if we use basis function expansion, such as polynomials, because the NLL is still linear in the
parameters w, even if it is not linear in the inputs x.)
7.3.1 Derivation of the MLE
First, we rewrite the objective in a form that is more amenable to differentiation:
NLL(w)=1
2(y−Xw)T(y−Xw)=1
2wT(XTX)w−wT(XTy) (7.11)

220 Chapter 7. Linear regression
where
XTX=N∑
i=1xixT
i=N∑
i=1⎛
⎜⎝x2
i,1···xi,1xi,D
...
xi,Dxi,1···x2
i,D⎞
⎟⎠ (7.12)
is thesum of squares matrix and
XTy=N∑
i=1xiyi. (7.13)
Using results from Equation 4.10, we see that the gradient of this is given by
g(w)=[XTXw−XTy]=N∑
i=1xi(wTxi−yi) (7.14)
Equating to zero we get
XTXw=XTy (7.15)
This is known as the normal equation. The corresponding solution ˆwto this linear system of
equations is called the ordinary least squares orOLSsolution, which is given by
ˆwOLS=(XTX)−1XTy (7.16)
7.3.2 Geometric interpretation
This equation has an elegant geometrical intrepretation, as we now explain. We assume N>D,
so we have more examples than features. The columns of Xdeﬁne a linear subspace of
dimensionality Dwhich is embedded in Ndimensions. Let the j’th column be ˜ xj, which is
a vector in RN. (This should not be confused with xi∈RD, which represents the i’th data
case.) Similarly, yis a vector in RN. For example, suppose we have N=3examples in D=2
dimensions:
X=⎛
⎝12
1−2
12⎞

X=⎛
⎝12
1−2
12⎞
⎠,y=⎛⎝8.8957
0.6130
1.7761⎞⎠ (7.17)
These vectors are illustrated in Figure 7.3.
We seek a vector ˆy∈R
Nthat lies in this linear subspace and is as close as possible to y,
i.e., we want to ﬁnd
argmin
ˆy∈span({˜ x1,...,˜ xD})∥y−ˆy∥2. (7.18)
Sinceˆy∈span(X), there exists some weight vector wsuch that
ˆy=w1˜ x1+···+wD˜ xD=Xw (7.19)

7.3. Maximum likelihood estimation (least squares) 221
00.20.40.60.81
−1−0.500.5100.20.40.60.81
y
x1x2ˆy
x1
(0,0,0)
x2x3
Figure 7.3 Graphical interpretation of least squares for N=3examples and D=2features. ˜ x1and
˜ x2are vectors in R3; together they deﬁne a 2D plane. yis also a vector in R3but does not lie on this
2D plane. The orthogonal projection of yonto this plane is denoted ˆy. The red line from ytoˆyis
the residual, whose norm we want to minimize. For visual clarity, all vectors have been converted to unit
norm. Figure generated by leastSquaresProjection .
To minimize the norm of the residual, y−ˆy, we want the residual vector to be orthogonal to
every column of X,s o˜ xT
j(y−ˆy)=0forj=1:D. Hence
˜ xT
j(y−ˆy)=0⇒XT(y−Xw)=0⇒w=(XTX)−1XTy (7.20)
Hence our projected value of yis given by
ˆy=Xˆw=X(XTX)−1XTy (7.21)
This corresponds to an orthogonal projection ofyonto the column space of X. The projection
matrixP≜X(XTX)−1XTis called the hat matrix, since it “puts the hat on y”.
7.3.3 Convexity

˜ xT
When discussing least squares, we noted that the NLL had a bowl shape with a unique minimum.
The technical term for functions like this is convex. Convex functions play a very important
role in machine learning.
Let us deﬁne this concept more precisely. We say a setSisconvexif for any θ,θ′∈S,w e
have
λθ+(1−λ)θ′∈S,∀λ∈[0,1] (7.22)

222 Chapter 7. Linear regression
(a)
 (b)
Figure 7.4 (a) Illustration of a convex set. (b) Illustration of a nonconvex set.
x yλ1 − λ
(a)A B
(b)
Figure 7.5 (a) Illustration of a convex function. We see that the chord joining (x,f(x))to(y,f(y))lies
above the function. (b) A function that is neither convex nor concave. Ais a local minimum, Bis a global
minimum. Figure generated by convexFnHand .
That is, if we draw a line from θtoθ′, all points on the line lie inside the set. See Figure 7.4(a)
for an illustration of a convex set, and Figure 7.4(b) for an illustration of a non-convex set.
Afunctionf(θ)is called convex if its epigraph (the set of points above the function) deﬁnes
a convex set. Equivalently, a function f(θ)is called convex if it is deﬁned on a convex set and
if, for any θ,θ′∈S, and for any 0≤λ≤1,w eh a v e
f(λθ+(1−λ)θ′)≤λf(θ)+(1−λ)f(θ′) (7.23)
See Figure 7.5 for a 1d example. A function is called strictly convex if the inequality is strict. A

functionf(θ)isconcaveif−f(θ)is convex. Examples of scalar convex functions include θ2,
eθ, andθlogθ(forθ>0). Examples of scalar concave functions include log(θ)and√
θ.
Intuitively, a (strictly) convex function has a “bowl shape”, and hence has a unique global
minimum θ∗corresponding to the bottom of the bowl. Hence its second derivative must be
positive everywhere,d
dθf(θ)>0. A twice-continuously differentiable, multivariate function fis
convex iff its Hessian is positive deﬁnite for all θ.1In the machine learning context, the function

θ.
foften corresponds to the NLL.
1. Recall that the Hessian is the matrix of second partial derivatives, deﬁned by Hjk=∂f2(θ)
∂θj∂θk.Also, recall that a
matrixHispositive deﬁnite iffvTHv>0for any non-zero vector v.

7.4. Robust linear regression * 223
0 0.2 0.4 0.6 0.8 1−6−5−4−3−2−101234Linear data with noise and outliers
  
least squares
laplace
(a)−3 −2 −1 0 1 2 3−0.500.511.522.533.544.55
  
L2
L1
huber
(b)
Figure 7.6 (a) Illustration of robust linear regression. Figure generated by linregRobustDemoCombined .
(b) Illustration of ℓ2,ℓ1, and Huber loss functions. Figure generated by huberLossDemo .
Models where the NLL is convex are desirable, since this means we can always ﬁnd the
globally optimal MLE. We will see many examples of this later in the book. However, many
models of interest will not have concave likelihoods. In such cases, we will discuss ways toderive locally optimal parameter estimates.
7.4 Robust linear regression *
It is very common to model the noise in regression models using a Gaussian distributionwith zero mean and constant variance, ϵ
i∼N(0,σ2),w h e r eϵi=yi−wTxi. In this case,

maximizing likelihood is equivalent to minimizing the sum of squared residuals, as we haveseen. However, if we have outliersin our data, this can result in a poor ﬁt, as illustrated in
Figure 7.6(a). (The outliers are the points on the bottom of the ﬁgure.) This is because squarederror penalizes deviations quadratically, so points far from the line have more affect on the ﬁtthan points near to the line.
One way to achieve robustness to outliers is to replace the Gaussian distribution for the
response variable with a distribution that has heavy tails. Such a distribution will assign higher

likelihood to outliers, without having to perturb the straight line to “explain” them.
One possibility is to use the Laplace distribution, introduced in Section 2.4.3. If we use this
as our observation model for regression, we get the following likelihood:
p(y|x,w,b) = Lap(y |w
Tx,b)∝exp(−1
b|y−wTx|) (7.24)
The robustness arises from the use of |y−wTx|instead of (y−wTx)2. For simplicity, we will
assumebis ﬁxed. Let ri≜yi−wTxibe thei’th residual. The NLL has the form
ℓ(w)=∑
i|ri(w)| (7.25)

224 Chapter 7. Linear regression
Likelihood Prior Name Section
Gaussian Uniform Least squares 7.3
Gaussian Gaussian Ridge 7.5Gaussian Laplace Lasso 13.3Laplace Uniform Robust regression 7.4Student Uniform Robust regression Exercise 11.12
Table 7.1 Summary of various likelihoods and priors used for linear regression. The likelihood refers to
the distributional form of p(y|x,w,σ2), and the prior refers to the distributional form of p(w). MAP
estimation with a uniform distribution corresponds to MLE.
Unfortunately, this is a non-linear objective function, which is hard to optimize. Fortunately, wecan convert the NLL to a linear objective, subject to linear constraints, using the following split
variable trick. First we deﬁne
r
i≜r+
i−r−
i (7.26)
and then we impose the linear inequality constraints that r+
i≥0andr−
i≥0. Now the
constrained objective becomes
min
w,r+,r−∑
i(r+
i−r−
i)s.t.r+
i≥0,r−
i≥0,wTxi+r+
i+r−
i=yi (7.27)
This is an example of a linear program withD+2Nunknowns and 3Nconstraints.

r
i≜r+
i−r−
i (7.26)
i≥0andr−
min
w,r+,r−∑
i(r+
i−r−
i)s.t.r+
i≥0,r−
i+r−
Since this is a convex optimization problem, it has a unique solution. To solve an LP, we must
ﬁrst write it in standard form, which as follows:
min
θfTθs.t.Aθ≤b,Aeqθ=beq,l≤θ≤u (7.28)
In our current example, θ=(w,r+,r−),f=[0,1,1],A=[ ],b=[ ],Aeq=[X,I,−I],
beq=y,l=[−∞1,0,0],u=[ ]. This can be solved by any LP solver (see e.g., (Boyd and
Vandenberghe 2004)). See Figure 7.6(a) for an example of the method in action.
An alternative to using NLL under a Laplace likelihood is to minimize the Huber loss function

min
(Huber 1964), deﬁned as follows:
LH(r,δ)={r2/2if|r|≤δ
δ|r|−δ2/2if|r|>δ(7.29)
This is equivalent to ℓ2for errors that are smaller than δ, and is equivalent to ℓ1for larger errors.
See Figure 7.6(b). The advantage of this loss function is that it is everywhere differentiable,using the fact that
d
dr|r|=sign(r)ifr̸=0. We can also check that the function is C1
continuous, since the gradients of the two parts of the function match at r=±δ, namely
d

d
d
drLH(r,δ)|r=δ=δ. Consequently optimizing the Huber loss is much faster than using the
Laplace likelihood, since we can use standard smooth optimization methods (such as quasi-Newton) instead of linear programming.
Figure 7.6(a) gives an illustration of the Huber loss function. The results are qualitatively
similiar to the probabilistic methods. (In fact, it turns out that the Huber method also has aprobabilistic interpretation, although it is rather unnatural (Pontil et al. 1998).)

7.5. Ridge regression 225
0 5 10 15 20−10−505101520ln lambda −20.135
(a)0 5 10 15 20−15−10−505101520ln lambda −8.571
(b)
Figure 7.7 Degree 14 Polynomial ﬁt to N=2 1data points with increasing amounts of ℓ2regularization.
Data was generated from noise with variance σ2=4. The error bars, representing the noise variance σ2,
get wider as the ﬁt gets smoother, since we are ascribing more of the data variation to the noise. Figure
generated by linregPolyVsRegDemo .
7.5 Ridge regression
One problem with ML estimation is that it can result in overﬁtting. In this section, we discuss a
way to ameliorate this problem by using MAP estimation with a Gaussian prior. For simplicity,we assume a Gaussian likelihood, rather than a robust likelihood.
7.5.1 Basic idea

The reason that the MLE can overﬁt is that it is picking the parameter values that are thebest for modeling the training data; but if the data is noisy, such parameters often result incomplex functions. As a simple example, suppose we ﬁt a degree 14 polynomial to N=2 1data
points using least squares. The resulting curve is very “wiggly”, as shown in Figure 7.7(a). Thecorresponding least squares coefficients (excluding w
0) are as follows:
6.560, -36.934, -109.255, 543.452, 1022.561, -3046.224, -3768.013,8524.540, 6607.897, -12640.058, -5530.188, 9479.730, 1774.639, -2821.526

We see that there are many large positive and negative numbers. These balance out exactly
to make the curve “wiggle” in just the right way so that it almost perfectly interpolates the data.But this situation is unstable: if we changed the data a little, the coefficients would change a lot.
We can encourage the parameters to be small, thus resulting in a smoother curve, by using a
zero-mean Gaussian prior:
p(w)=∏
jN(wj|0,τ2) (7.30)
where1/τ2controls the strength of the prior. The corresponding MAP estimation problem
becomes
argmax
wN∑
i=1logN(yi|w0+wTxi,σ2)+D∑
j=1logN(wj|0,τ2) (7.31)

226 Chapter 7. Linear regression
−25 −20 −15 −10 −5 0 50510152025
  
train mse
test mse
(a)−20 −15 −10 −5 0 50.10.20.30.40.50.60.70.80.9
log lambda  
negative log marg. likelihood
CV estimate of MSE
(b)
Figure 7.8 (a) Training error (dotted blue) and test error (solid red) for a degree 14 polynomial ﬁt by
ridge regression, plotted vs log(λ). Data was generated from noise with variance σ2=4(training set
has sizeN=2 1). Note: Models are ordered from complex (small regularizer) on the left to simple (large
regularizer) on the right. The stars correspond to the values used to plot the functions in Figure 7.7. (b)
Estimate of performance using training set. Dotted blue: 5-fold cross-validation estimate of future MSE.Solid black: negative log marginal likelihood, −logp(D|λ). Both curves have been vertically rescaled to
[0,1] to make them comparable. Figure generated by linregPolyVsRegDemo .
It is a simple exercise to show that this is equivalent to minimizing the following:
J(w)=1
NN∑

J(w)=1
NN∑
i=1(yi−(w0+wTxi))2+λ||w||2
2 (7.32)
whereλ≜σ2/τ2and||w||2
2=∑
jw2
j=wTwis the squared two-norm. Here the ﬁrst term is
the MSE/ NLL as usual, and the second term, λ≥0, is a complexity penalty. The corresponding
solution is given by
ˆwridge=(λID+XTX)−1XTy (7.33)
This technique is known as ridge regression,o r penalized least squares. In general, adding
a Gaussian prior to the parameters of a model to encourage them to be small is called ℓ2
regularization orweight decay. Note that the offset term w0is not regularized, since this just

2 (7.32)
2=∑
jw2
affects the height of the function, not its complexity. By penalizing the sum of the magnitudes
of the weights, we ensure the function is simple (since w=0corresponds to a straight line,
which is the simplest possible function, corresponding to a constant.)
We illustrate this idea in Figure 7.7, where we see that increasing λresults in smoother
functions. The resulting coefficients also become smaller. For example, using λ=1 0−3,w e
have

7.5. Ridge regression 227
2.128, 0.807, 16.457, 3.704, -24.948, -10.472, -2.625, 4.360, 13.711,
10.063, 8.716, 3.966, -9.349, -9.232
In Figure 7.8(a), we plot the MSE on the training and test sets vs log(λ). We see that, as we
increaseλ(so the model becomes more constrained), the error on the training set increases.
For the test set, we see the characteristic U-shaped curve, where the model overﬁts and thenunderﬁts. It is common to use cross validation to pick λ, as shown in Figure 7.8(b). In
Section 1.4.8, we will discuss a more probabilistic approach.
We will consider a variety of different priors in this book. Each of these corresponds to a
different form of regularization. This technique is very widely used to prevent overﬁtting.
7.5.2 Numerically stable computation *
Interestingly, ridge regression, which works better statistically, is also easier to ﬁt numerically,since(λI
D+XTX)is much better conditioned (and hence more likely to be invertible) than
XTX, at least for suitable largy λ.

Nevertheless, inverting matrices is still best avoided, for reasons of numerical stability. (Indeed,
if you write w=inv(X’ * X)*X’*y in Matlab, it will give you a warning.) We now describe
a useful trick for ﬁtting ridge regression models (and hence by extension, computing vanillaOLS estimates) that is more numerically robust. We assume the prior has the form p(w)=
N(0,Λ
−1),w h e r eΛis the precision matrix. In the case of ridge regression, Λ=( 1/τ2)I.T o

N(0,Λ
avoid penalizing the w0term, we should center the data ﬁrst, as explained in Exercise 7.5.
First let us augment the original data with some “virtual data” coming from the prior:
˜X=(X/σ√
Λ)
,˜y=(y/σ
0D×1)
(7.34)
whereΛ=√
Λ√
ΛTis aCholesky decomposition ofΛ. We see that ˜Xis(N+D)×D,
where the extra rows represent pseudo-data from the prior.
We now show that the NLL on this expanded data is equivalent to penalized NLL on the
original data:
f(w)=(˜y−˜Xw)T(˜y−˜Xw) (7.35)
=((y/σ
0)
−(X/σ√
Λ)
w)T((y/σ
0)
−(X/σ√
Λ)
w)
(7.36)

˜X=(X/σ√
Λ)
,˜y=(y/σ
0D×1)
(7.34)
whereΛ=√
Λ√
=((y/σ
0)
−(X/σ√
Λ)
w)T((y/σ
0)
−(X/σ√
Λ)
w)
(7.36)
=(1
σ(y−Xw)
−√
Λw)T(1
σ(y−Xw)
−√
Λw)
(7.37)
=1
σ2(y−Xw)T(y−Xw)+(√
Λw)T(√
Λw) (7.38)
=1
σ2(y−Xw)T(y−Xw)+wTΛw (7.39)
Hence the MAP estimate is given by
ˆwridge=(˜XT˜X)−1˜XT˜y (7.40)
as we claimed.

228 Chapter 7. Linear regression
Now let
˜X=QR (7.41)
be theQR decomposition ofX,w h e r eQis orthonormal (meaning QTQ=QQT=I), and
Ris upper triangular. Then
(˜XT˜X)−1=(RTQTQR)−1=(RTR)−1=R−1R−T(7.42)
Hence
ˆwridge=R−1R−TRTQT˜y=R−1Q˜y (7.43)
Note that Ris easy to invert since it is upper triangular. This gives us a way to compute the
ridge estimate while avoiding having to invert (Λ+XTX).
We can use this technique to ﬁnd the MLE, by simply computing the QR decomposition of
the unaugmented matrix X, and using the original y. This is the method of choice for solving
least squares problems. (In fact, it is so sommon that it can be implemented in one line of
Matlab, using the backslash operator: w=X\y.) Note that computing the QR decomposition of
anN×Dmatrix takes O(ND2)time, and is numerically very stable.
IfD≫N, we should ﬁrst perform an SVD decomposition. In particular, let X=USVTbe
the SVD of X,w h e r eVTV=IN,UUT=UTU=IN, andSis a diagonal N×Nmatrix.

Now letZ=UDbe anN×Nmatrix. Then we can rewrite the ridge estimate thus:
ˆwridge=V(ZTZ+λIN)−1ZTy (7.44)
In other words, we can replace the D-dimensional vectors xiwith theN-dimensional vectors
ziand perform our penalized ﬁt as before. We then transform the N-dimensional solution
to theD-dimensional solution by multiplying by V. Geometrically, we are rotating to a new
coordinate system in which all but the ﬁrst Ncoordinates are zero. This does not affect the

solution since the spherical Gaussian prior is rotationally invariant. The overall time is nowO(DN
2)operations.
7.5.3 Connection with PCA *
In this section, we discuss an interesting connection between ridge regression and PCA (Sec-tion 12.2), which gives further insight into why ridge regression works well. Our discussion isbased on (Hastie et al. 2009, p66).
LetX=USV
Tbe the SVD of X. From Equation 7.44, we have
ˆwridge=V(S2+λI)−1SUTy (7.45)
Hence the ridge predictions on the training set are given by
ˆy=Xˆwridge=USVTV(S2+λI)−1SUTy (7.46)
=U˜SUTy=D∑

LetX=USV
j=1uj˜SjjuT
jy (7.47)

7.5. Ridge regression 229
prior meanMAP EstimateML Estimateu1u2
Figure 7.9 Geometry of ridge regression. The likelihood is shown as an ellipse, and the prior is shown
as a circle centered on the origin. Based on Figure 3.15 of (Bishop 2006b). Figure generated by geomRidge
where
˜Sjj≜[S(S2+λI)−1S]jj=σ2
j
σ2
j+λ(7.48)
andσjare the singular values of X. Hence
ˆy=Xˆwridge=D∑
j=1ujσ2
j
σ2
j+λuT
jy (7.49)
In contrast, the least squares prediction is
ˆy=Xˆwls=(USVT)(VS−1UTy)=UUTy=D∑
j=1ujuTjy (7.50)
Ifσ2
jis small compared to λ, then direction ujwill not have much effect on the prediction. In
view of this, we deﬁnethe effective number of degrees of freedom of the model as follows:
dof(λ)=D∑
j=1σ2
j
σ2
j+λ(7.51)
Whenλ=0,dof(λ)=D , and asλ→∞,dof(λ)→0.
Let us try to understand why this behavior is desirable. In Section 7.6, we show that
cov[w|D]=σ2(XTX)−1, if we use a uniform prior for w. Thus the directions in which
we are most uncertain about ware determined by the eigenvectors of this matrix with the

Ifσ2
dof(λ)=D∑
j=1σ2
j
σ2
j+λ(7.51)
smallest eigenvalues, as shown in Figure 4.1. Furthermore, in Section 12.2.3, we show that the
squared singular values σ2
jare equal to the eigenvalues of XTX. Hence small singular values σj
correspond to directions with high posterior variance. It is these directions which ridge shrinksthe most.

230 Chapter 7. Linear regression
This process is illustrated in Figure 7.9. The horizontal w1parameter is not-well determined
by the data (has high posterior variance), but the vertical w2parameter is well-determined.
Hencewmap
2is close to ˆwmle
2, butwmap
1is shifted strongly towards the prior mean, which is 0.
(Compare to Figure 4.14(c), which illustrated sensor fusion with sensors of different reliabilities.)
In this way, ill-determined parameters are reduced in size towards 0. This is called shrinkage.
There is a related, but different, technique called principal components regression. The idea
is this: ﬁrst use PCA to reduce the dimensionality to Kdimensions, and then use these low
dimensional features as input to regression. However, this technique does not work as well asridge in terms of predictive accuracy (Hastie et al. 2001, p70). The reason is that in PC regression,only the ﬁrst K(derived) dimensions are retained, and the remaining D−Kdimensions are

entirely ignored. By contrast, ridge regression uses a “soft” weighting of all the dimensions.
7.5.4 Regularization effects of big data
Regularization is the most common way to avoid overﬁtting. However, another effective approach— which is not always available — is to use lots of data. It should be intuitively obvious thatthe more training data we have, the better we will be able to learn.
2So we expect the test set
error to decrease to some plateau as Nincreases.

This is illustrated in Figure 7.10, where we plot the mean squared error incurred on the test set
achieved by polynomial regression models of different degrees vs N(a plot of error vs training
set size is known as a learning curve). The level of the plateau for the test error consists of
two terms: an irreducible component that all models incur, due to the intrinsic variability ofthe generating process (this is called the noise ﬂoor); and a component that depends on the

discrepancy between the generating process (the “truth”) and the model: this is called structural
error.
In Figure 7.10, the truth is a degree 2 polynomial, and we try ﬁtting polynomials of degrees 1,
2 and 25 to this data. Call the 3 models M
1,M2andM25. We see that the structural error
for models M2andM25is zero, since both are able to capture the true generating process.
However, the structural error for M1is substantial, which is evident from the fact that the
plateau occurs high above the noise ﬂoor.

error.
For any model that is expressive enough to capture the truth (i.e., one with small structural
error), the test error will go to the noise ﬂoor as N→∞. However, it will typically go to

zero faster for simpler models, since there are fewer parameters to estimate. In particular, forﬁnite training sets, there will be some discrepancy between the parameters that we estimateand the best parameters that we could estimate given the particular model class. This is calledapproximation error, and goes to zero as N→∞, but it goes to zero faster for simpler
models. This is illustrated in Figure 7.10. See also Exercise 7.1.
In domains with lots of data, simple methods can work surprisingly well (Halevy et al. 2009).

However, there are still reasons to study more sophisticated learning methods, because therewill always be problems for which we have little data. For example, even in such a data-richdomain as web search, as soon as we want to start personalizing the results, the amount of dataavailable for any given user starts to look small again (relative to the complexity of the problem).
2. This assumes the training data is randomly sampled, and we don’t just get repetitions of the same examples. Having
informatively sampled data can help even more; this is the motivation for an approach known as active learning, where
you get to choose your training data.

7.6. Bayesian linear regression 231
0 20 40 60 80 100 120 140 160 180 2000246810121416182022
size of training setmsetruth=degree 2, model = degree 1
  
train
test
(a)0 20 40 60 80 100 120 140 160 180 2000246810121416182022
size of training setmsetruth=degree 2, model = degree 2
  
train
test
(b)
size of training setmsetruth=degree 2, model = degree 10
  
train
test
(c)0 20 40 60 80 100 120 140 160 180 2000246810121416182022
size of training setmsetruth=degree 2, model = degree 25
  
train
test
(d)
Figure 7.10 MSE on training and test sets vs size of training set, for data generated from a degree 2
polynomial with Gaussian noise of variance σ2=4. We ﬁt polynomial models of varying degree to this
data. (a) Degree 1. (b) Degree 2. (c) Degree 10. (d) Degree 25. Note that for small training set sizes, the test

train
test
  
train
test
(d)
error of the degree 25 polynomial is higher than that of the degree 2 polynomial, due to overﬁtting, butthis difference vanishes once we have enough data. Note also that the degree 1 polynomial is too simpleand has high test error even given large amounts of training data. Figure generated by linregPolyVsN .
In such cases, we may want to learn multiple related models at the same time, which is known

as multi-task learning. This will allow us to “borrow statistical strength” from tasks with lots ofdata and to share it with tasks with little data. We will discuss ways to do later in the book.
7.6 Bayesian linear regression
Although ridge regression is a useful way to compute a point estimate, sometimes we want tocompute the full posterior over wandσ
2. For simplicity, we will initially assume the noise
varianceσ2is known, so we focus on computing p(w|D,σ2). Then in Section 7.6.3 we consider

232 Chapter 7. Linear regression
the general case, where we compute p(w,σ2|D). We assume throughout a Gaussian likelihood
model. Performing Bayesian inference with a robust likelihood is also possible, but requires more
advanced techniques (see Exercise 24.5).
7.6.1 Computing the posterior
In linear regression, the likelihood is given by
p(y|X,w,μ,σ2)=N (y|μ+Xw,σ2IN) (7.52)
∝exp(
−1
2σ2(y−μ1N−Xw)T(y−μ1N−Xw))
(7.53)
whereμis an offset term. If the inputs are centered, so∑
ixij=0for eachj, the mean of the
output is equally likely to be positive or negative. So let us put an improper prior on μof the
formp(μ)∝1, and then integrate it out to get
p(y|X,w,σ2)∝exp(
−1
2σ2||y−y1N−Xw||2
2)
(7.54)
wherey=1
N∑N
i=1yiis the empirical mean of the output. For notational simplicity, we shall
assume the output has been centered, and write yfory−y1N.
The conjugate prior to the above Gaussian likelihood is also a Gaussian, which we will denote

−1
2)
(7.54)
wherey=1
N∑N
byp(w)=N(w|w0,V0). Using Bayes rule for Gaussians, Equation 4.125, the posterior is given
by
p(w|X,y,σ2)∝N(w|w0,V0)N(y|Xw,σ2IN)=N(w|wN,VN) (7.55)
wN=VNV−1
0w0+1
σ2VNXTy (7.56)
V−1
N=V−1
0+1
σ2XTX (7.57)
VN=σ2(σ2V−1
0+XTX)−1(7.58)
Ifw0=0andV0=τ2I, then the posterior mean reduces to the ridge estimate, if we deﬁne
λ=σ2
τ2. This is because the mean and mode of a Gaussian are the same.
To gain insight into the posterior distribution (and not just its mode), let us consider a 1D
example:
y(x,w)=w0+w1x+ϵ (7.59)

by
wN=VNV−1
0w0+1
V−1
N=V−1
0+1
λ=σ2
example:
where the “true” parameters are w0=−0.3andw1=0.5. In Figure 7.11 we plot the prior,
the likelihood, the posterior, and some samples from the posterior predictive. In particular,
the right hand column plots the function y(x,w(s))wherexranges over [−1,1], andw(s)∼
N(w|wN,VN)is a sample from the parameter posterior. Initially, when we sample from the

example:
prior (ﬁrst row), our predictions are “all over the place”, since our prior is uniform. After we seeone data point (second row), our posterior becomes constrained by the corresponding likelihood,and our predictions pass close to the observed data. However, we see that the posterior hasa ridge-like shape, reﬂecting the fact that there are many possible solutions, with different

7.6. Bayesian linear regression 233
 W0  W1 prior/posterior
−1 0 1−101
−1 0 1−101
 x  y data space
 W0  W1 
−1 0 1−101
 W0  W1 
−1 0 1−101
−1 0 1−101
 x  y 
 W0  W1 
−1 0 1−101
 W0  W1 
−1 0 1−101
−1 0 1−101
 x  y 
 W0  W1 
−1 0 1−101
 W0  W1 
−1 0 1−101
−1 0 1−101
 x  y likelihood
Figure 7.11 Sequential Bayesian updating of a linear regression model p(y|x)=N(y|w0x0+w1x1,σ2).
Row 0 represents the prior, row 1 represents the ﬁrst data point (x1,y1), row 2 represents the second
data point (x2,y2), row 3 represents the 20th data point (x20,y20). Left column: likelihood function for
current data point. Middle column: posterior given data so far, p(w|x1:n,y1:n)(so the ﬁrst line is the
prior). Right column: samples from the current prior/posterior predictive distribution. The white cross in

columns 1 and 2 represents the true parameter value; we see that the mode of the posterior rapidly (after20 samples) converges to this point. The blue circles in column 3 are the observed data points. Based onFigure 3.7 of (Bishop 2006a). Figure generated by bayesLinRegDemo2d .
slopes/intercepts. This makes sense since we cannot uniquely infer two parameters from one

observation. After we see two data points (third row), the posterior becomes much narrower,and our predictions all have similar slopes and intercepts. After we observe 20 data points (lastrow), the posterior is essentially a delta function centered on the true value, indicated by a whitecross. (The estimate converges to the truth since the data was generated from this model, andbecause Bayes is a consistent estimator; see Section 6.4.1 for discussion of this point.)
7.6.2 Computing the posterior predictive

It’s tough to make predictions, especially about the future. — Yogi Berra

234 Chapter 7. Linear regression
In machine learning, we often care more about predictions than about interpreting the parame-
ters. Using Equation 4.126, we can easily show that the posterior predictive distribution at a testpointxis also Gaussian:
p(y|x,D,σ
2)=∫
N(y|xTw,σ2)N(w|wN,VN)dw (7.60)
=N(y|wT
Nx,σ2
N(x)) (7.61)
σ2
N(x)=σ2+xTVNx (7.62)
The variance in this prediction, σ2
N(x), depends on two terms: the variance of the observation
noise,σ2, and the variance in the parameters, VN. The latter translates into variance about
observations in a way which depends on how close xis to the training data D. This is illustrated
in Figure 7.12, where we see that the error bars get larger as we move away from the trainingpoints, representing increased uncertainty. This is important for applications such as activelearning, where we want to model what we don’t know as well as what we do. By contrast, theplugin approximation has constant sized error bars, since
p(y|x,D,σ
2)≈∫
N(y|xTw,σ2)δˆw(w)dw=p(y|x,ˆw,σ2) (7.63)

p(y|x,D,σ
2)≈∫
See Figure 7.12(a).
7.6.3 Bayesian inference when σ2is unknown *
In this section, we apply the results in Section 4.6.3 to the problem of computing p(w,σ2|D)
for a linear regression model. This generalizes the results from Section 7.6.1 where we assumedσ
2was known. In the case where we use an uninformative prior, we will see some interesting
connections to frequentist statistics.
7.6.3.1 Conjugate prior
As usual, the likelihood has the form
p(y|X,w,σ2)=N(y|Xw,σ2IN) (7.64)

By analogy to Section 4.6.3, one can show that the natural conjugate prior has the followingform:
p(w,σ
2)=N I G ( w,σ2|w0,V0,a0,b0) (7.65)
≜N(w|w0,σ2V0)IG(σ2|a0,b0) (7.66)
=ba0
0
(2π)D/2|V0|1
2Γ(a0)(σ2)−(a0+(D/2)+1)(7.67)
×exp[
−(w−w0)TV−1
0(w−w0)+2b0
2σ2]
(7.68)

7.6. Bayesian linear regression 235
−8 −6 −4 −2 0 2 4 6 80102030405060plugin approximation (MLE)
  
training data
(a)−8 −6 −4 −2 0 2 4 6 8−1001020304050607080Posterior predictive (known variance)
  
(b)
−8 −6 −4 −2 0 2 4 6 805101520253035404550functions sampled from plugin approximation to posterior
(c)−8 −6 −4 −2 0 2 4 6 8−20020406080100functions sampled from posterior
(d)
Figure 7.12 (a) Plug-in approximation to predictive density (we plug in the MLE of the parameters). (b)
Posterior predictive density, obtained by integrating out the parameters. Black curve is posterior mean,
error bars are 2 standard deviations of the posterior predictive density. (c) 10 samples from the pluginapproximation to posterior predictive. (d) 10 samples from the posterior predictive. Figure generated bylinregPostPredDemo .
With this prior and likelihood, one can show that the posterior has the following form:
p(w,σ2|D) = NIG(w ,σ2|wN,VN,aN,bN) (7.69)
wN=VN(V−1
0w0+XTy) (7.70)
VN=(V−1

wN=VN(V−1
VN=(V−1
0+XTX)−1(7.71)
aN=a0+n/2 (7.72)
bN=b0+1
2(
wT
0V−1
0w0+yTy−wT
NV−1
NwN)
(7.73)
The expressions for wNandVNare similar to the case where σ2is known. The expression for
aNis also intuitive, since it just updates the counts. The expression for bNcan be interpreted

236 Chapter 7. Linear regression
as follows: it is the prior sum of squares, b0, plus the empirical sum of squares, yTy, plus a
term due to the error in the prior on w.
The posterior marginals are as follows:
p(σ2|D)=I G ( aN,bN) (7.74)
p(w|D)=T (wN,bN
aNVN,2aN) (7.75)
We give a worked example of using these equations in Section 7.6.3.3.
By analogy to Section 4.6.3.6, the posterior predictive distribution is a Student T distribution.
In particular, given mnew test inputs ˜X,w eh a v e
p(˜y|˜X,D)=T (˜y|˜XwN,bN
aN(Im+˜XVN˜XT),2aN) (7.76)
The predictive variance has two components: (bN/aN)Imdue to the measurement noise, and
(bN/aN)˜XVN˜XTdue to the uncertainty in w. This latter terms varies depending on how
close the test inputs are to the training data.
It is common to set a0=b0=0, corresponding to an uninformative prior for σ2, and to set
w0=0andV0=g(XTX)−1for any positive value g. This is called Zellner’s g-prior(Zellner

1986). Here gplays a role analogous to 1/λin ridge regression. However, the prior covariance is
proportional to (XTX)−1rather than I. This ensures that the posterior is invariant to scaling
of the inputs (Minka 2000b). See also Exercise 7.10.
We will see below that if we use an uninformative prior, the posterior precision given N
measurements is V−1
N=XTX. Theunit information prior is deﬁned to contain as much
information as one sample (Kass and Wasserman 1995). To create a unit information prior for

linear regression, we need to use V−1
0=1
NXTX, which is equivalent to the g-prior with
g=N.
7.6.3.2 Uninformative prior
An uninformative prior can be obtained by considering the uninformative limit of the conjugateg-prior, which corresponds to setting g=∞. This is equivalent to an improper NIG prior with
w
0=0,V0=∞I,a0=0andb0=0, which gives p(w,σ2)∝σ−(D+2).
Alternatively, we can start with the semi-conjugate prior p(w,σ2)=p(w)p(σ2), and take

0=1
g=N.
w
each term to its uninformative limit individually, which gives p(w,σ2)∝σ−2. This is equivalent
to an improper NIG prior with w0=0,V=∞I,a0=−D/2andb0=0. The corresponding
posterior is given by
p(w,σ2|D)=N I G ( w,σ2|wN,VN,aN,bN) (7.77)
wN=ˆwmle=(XTX)−1XTy (7.78)
VN=(XTX)−1(7.79)
aN=N−D
2(7.80)
bN=s2
2(7.81)
s2≜(y−Xˆwmle)T(y−Xˆwmle (7.82)

7.6. Bayesian linear regression 237
wjE[wj|D]√
var[wj|D]95% CI sig
w0 10.998 3.06027 [4.652, 17.345] *
w1 -0.004 0.00156 [-0.008, -0.001] *w2 -0.054 0.02190 [-0.099, -0.008] *w3 0.068 0.09947 [-0.138, 0.274]w4 -1.294 0.56381 [-2.463, -0.124] *w5 0.232 0.10438 [0.015, 0.448] *w6 -0.357 1.56646 [-3.605, 2.892]w7 -0.237 1.00601 [-2.324, 1.849]w8 0.181 0.23672 [-0.310, 0.672]w9 -1.285 0.86485 [-3.079, 0.508]w10 -0.433 0.73487 [-1.957, 1.091]
Table 7.2 Posterior mean, standard deviation and credible intervals for a linear regression model with an
uninformative prior ﬁt to the caterpillar data. Produced by linregBayesCaterpillar .
The marginal distribution of the weights is given by
p(w|D)=T(w|ˆw,s2
N−DC,N−D) (7.83)
whereC=(XTX)−1andˆwis the MLE. We discuss the implications of these equations below.
7.6.3.3 An example where Bayesian and frequentist inference coincide *

The use of a (semi-conjugate) uninformative prior is interesting because the resulting posteriorturns out to be equivalent to the results from frequentist statistics (see also Section 4.6.3.9). Inparticular, from Equation 7.83 we have
p(w
j|D)=T(wj|ˆwj,Cjjs2
N−D,N−D) (7.84)
This is equivalent to the sampling distribution of the MLE which is given by the following (seee.g., (Rice 1995, p542), (Casella and Berger 2002, p554)):
w
j−ˆwj
sj∼tN−D (7.85)
where
sj=√
s2Cjj
N−D(7.86)

p(w
w
j−ˆwj
where
sj=√
s2Cjj
N−D(7.86)
is the standard error of the estimated parameter. (See Section 6.2 for a discussion of samplingdistributions.) Consequently, the frequentist conﬁdence interval and the Bayesian marginalcredible interval for the parameters are the same in this case.
As a worked example of this, consider the caterpillar dataset from (Marin and Robert 2007).
(The details of what the data mean don’t matter for our present purposes.) We can compute

238 Chapter 7. Linear regression
the posterior mean and standard deviation, and the 95% credible intervals (CI) for the regression
coefficients using Equation 7.84. The results are shown in Table 7.2. It is easy to check that these95% credible intervals are identical to the 95% conﬁdence intervals computed using standardfrequentist methods (see linregBayesCaterpillar for the code).
We can also use these marginal posteriors to compute if the coefficients are “signiﬁcantly”
different from 0. An informal way to do this (without using decision theory) is to check if its 95%CI excludes 0. From Table 7.2, we see that the CIs for coefficients 0, 1, 2, 4, 5 are all signiﬁcantby this measure, so we put a little star by them. It is easy to check that these results are thesame as those produced by standard frequentist software packages which compute p-values atthe 5% level.
Although the correspondence between the Bayesian and frequentist results might seem ap-

pealing to some readers, recall from Section 6.6 that frequentist inference is riddled with patholo-gies. Also, note that the MLE does not even exist when N<D, so standard frequentist inference
theory breaks down in this setting. Bayesian inference theory still works, although it requiresthe use of proper priors. (See (Maruyama and George 2008) for one extension of the g-prior tothe case where D>N.)
7.6.4 EB for linear regression (evidence procedure)

So far, we have assumed the prior is known. In this section, we describe an empirical Bayesprocedure for picking the hyper-parameters. More precisely, we choose η=(α,λ)to maximize
the marignal likelihood, where λ=1/σ
2be the precision of the observation noise and αis
the precision of the prior, p(w)=N(w|0,α−1I). This is known as the evidence procedure
(MacKay 1995b).3See Section 13.7.4 for the algorithmic details.
The evidence procedure provides an alternative to using cross validation. For example, in

Figure 7.13(b), we plot the log marginal likelihood for different values of α, as well as the
maximum value found by the optimizer. We see that, in this example, we get the same resultas 5-CV, shown in Figure 7.13(a). (We kept λ=1/σ
2ﬁxed in both methods, to make them
comparable.)
The principle practical advantage of the evidence procedure over CV will become apparent
in Section 13.7, where we generalize the prior by allowing a different αjfor every feature. This

can be used to perform feature selection, using a technique known as automatic relevancydetermination or ARD. By contrast, it would not be possible to use CV to tune Ddifferent
hyper-parameters.
The evidence procedure is also useful when comparing different kinds of models, since it
provides a good approximation to the evidence:
p(D|m)=∫∫
p(D|w,m)p(w|m,η)p(η|m)dwdη (7.87)
≈max
η∫
p(D|w,m)p(w|m,η)p(η|m)dw (7.88)
It is important to (at least approximately) integrate over ηrather than setting it arbitrarily, for

p(D|m)=∫∫
≈max
η∫
reasons discussed in Section 5.3.2.5. Indeed, this is the method we used to evaluate the marginal
3. Alternatively, we could integrate out λanalytically, as shown in Section 7.6.3, and just optimize α(Buntine and
Weigend 1991). However, it turns out that this is less accurate than optimizing both αandλ(MacKay 1999).

7.6. Bayesian linear regression 239
−25 −20 −15 −10 −5 0 5100101102103104105106107
log lambdamse5−fold cross validation, ntrain = 21
(a)−25 −20 −15 −10 −5 0 5−150−140−130−120−110−100−90−80−70−60−50
log alphalog evidence
(b)
Figure 7.13 (a) Estimate of test MSE produced by 5-fold cross-validation vs log(λ). The smallest value is
indicated by the vertical line. Note the vertical scale is in log units. (c) Log marginal likelihood vs log(α).
The largest value is indicated by the vertical line. Figure generated by linregPolyVsRegDemo .
likelihood for the polynomial regression models in Figures 5.7 and 5.8. For a “more Bayesian”
approach, in which we model our uncertainty about ηrather than computing point estimates,
see Section 21.5.2.
Exercises
Exercise 7.1 Behavior of training set error with increasing sample size
The error on the test will always decrease as we get more training data, since the model will be better

Exercises
estimated. However, as shown in Figure 7.10, for sufficiently complex models, the error on the training setcan increase we we get more training data, until we reach some plateau. Explain why.
Exercise 7.2 Multi-output linear regression
When we have multiple independent outputs in linear regression, the model becomes
p(y|x,W)=M∏
j=1N(yj|wT
jxi,σ2
j) (7.89)
Since the likelihood factorizes across dimensions, so does the MLE. Thus
ˆW=[ˆw1,...,ˆwM] (7.90)
whereˆwj=(XTX)−1Y:,j.

jxi,σ2
j) (7.89)
In this exercise we apply this result to a model with 2 dimensional response vector yi∈R2. Suppose we
have some binary input data, xi∈{0,1}. The training data is as follows:

240 Chapter 7. Linear regression
xy
0(−1,−1)T
0(−1,−2)T
0(−2,−1)T
1(1,1)T
1(1,2)T
1(2,1)T
Let us embed each xiinto 2d using the following basis function:
φ(0) = (1, 0)T,φ(1) = (0, 1)T(7.91)
The model becomes
ˆy=WTφ(x) (7.92)
whereWis a2×2matrix. Compute the MLE for Wfrom the above data.
Exercise 7.3 Centering and ridge regression
Assume that x=0, so the input data has been centered. Show that the optimizer of
J(w,w0)=( y−Xw−w01)T(y−Xw−w01)+λwTw (7.93)
is
ˆw0=y (7.94)
w=(XTX+λI)−1XTy (7.95)
Exercise 7.4 MLE forσ2for linear regression
Show that the MLE for the error variance in linear regression is given by
ˆσ2=1
NN∑
i=1(yi−xT
iˆw)2(7.96)
This is just the empirical variance of the residual errors when we plug in our estimate of ˆw.
Exercise 7.5 MLE for the offset term in linear regression
Linear regression has the form E[y|x]=w 0+wTx. It is common to include a column of 1’s in the
design matrix, so we can solve for the offset term w0term and the other parameters wat the same time

ˆσ2=1
NN∑
i=1(yi−xT
using the normal equations. However, it is also possible to solve for wandw0separately. Show that
ˆw0=1
N∑
iyi−1
N∑
ixTiw=y−xTw (7.97)
Soˆw0models the difference in the average output from the average predicted output. Also, show that
ˆw=(XTcXc)−1XTcyc=[N∑
i=1(xi−x)(xi−x)T]−1[N∑
i=1(yi−y)(xi−x)]
(7.98)
whereXcis the centered input matrix containing xc
i=xi−xalong its rows, and yc=y−yis
the centered output vector. Thus we can ﬁrst compute ˆwon centered data, and then estimate w0using
y−xTˆw.

7.6. Bayesian linear regression 241
Exercise 7.6 MLE for simple linear regression
Simple linear regression refers to the case where the input is scalar, so D=1. Show that the MLE in
this case is given by the following equations, which may be familiar from basic statistics classes:
w1=∑
i(xi−x)(yi−¯y)∑
i(xi−¯x)2=∑
ixiyi−Nxy∑
ix2
i−Nx2≈cov[X,Y ]
var[X](7.99)
w0=¯y−w1¯x≈E[Y]−w1E[X] (7.100)
SeelinregDemo1 for a demo.
Exercise 7.7 Sufficient statistics for online linear regression
(Source: Jaakkola.) Consider ﬁtting the model ˆy=w0+w1xusing least squares. Unfortunately we did
not keep the original data, xi,yi, but we do have the following functions (statistics) of the data:
x(n)=1
nn∑
i=1xi,y(n)=1
nn∑
i=1yi (7.101)
C(n)
xx=1
nn∑
i=1(xi−x)2,C(n)
xy=1
nn∑
i=1(xi−x)(yi−y),C(n)
yy=1
nn∑
i=1(yi−y)2(7.102)
a. What are the minimal set of statistics that we need to estimate w1? (Hint: see Equation 7.99.)
b. What are the minimal set of statistics that we need to estimate w0? (Hint: see Equation 7.97.)

x(n)=1
nn∑
nn∑
C(n)
xx=1
nn∑
xy=1
nn∑
yy=1
nn∑
c. Suppose a new data point, xn+1,yn+1arrives, and we want to update our sufficient statistics without
looking at the old data, which we have not stored. (This is useful for online learning.) Show that we
can this for xas follows.
x(n+1)≜1
n+1n+1∑
i=1xi=1
n+1(
nx(n)+xn+1)
(7.103)
=x(n)+1
n+1(xn+1−x(n)) (7.104)
This has the form: new estimate is old estimate plus correction. We see that the size of the correctiondiminishes over time (i.e., as we get more samples). Derive a similar expression to update
y
d. Show that one can update C(n+1)
xyrecursively using
C(n+1)
xy=1
n+1[
xn+1yn+1+nC(n)

x(n+1)≜1
n+1n+1∑
i=1xi=1
n+1(
(7.103)
=x(n)+1
y
C(n+1)
xy=1
n+1[
xy+nx(n)y(n)−(n+1)x(n+1)y(n+1)]
(7.105)
Derive a similar expression to update Cxx.
e. Implementtheonlinelearningalgorithm, i.e., writeafunctionoftheform [w,ss] = linregUpdateSS(ss,
x, y), where x and y are scalars and ss is a structure containing the sufficient statistics.
f. Plot the coefficients over “time”, using the dataset in linregDemo1 . (Speciﬁcally, use [x,y] =
polyDataMake(’sampling’,’thibaux’). ) Check that they converge to the solution given by the

C(n+1)
xy=1
n+1[
(7.105)
batch (offline) learner (i.e, ordinary least squares). Your result should look like Figure 7.14.
Turn in your derivation, code and plot.
Exercise 7.8 Bayesian linear regression in 1d with known σ2
(Source: Bolstad.) Consider ﬁtting a model of the form
p(y|x,θ)=N(y|w0+w1x,σ2) (7.106)
to the data shown below:

242 Chapter 7. Linear regression
0 5 10 15 20−6−5−4−3−2−10123
timeweightsonline linear regression
  
w0
w1
w0 batch
w1 batch
Figure 7.14 Regression coefficients over time. Produced by Exercise 7.7.
x = [94,96,94,95,104,106,108,113,115,121,131];
y = [0.47, 0.75, 0.83, 0.98, 1.18, 1.29, 1.40, 1.60, 1.75, 1.90, 2.23];
a. Compute an unbiased estimate of σ2using
ˆσ2=1
N−2N∑
i=1(yi−ˆyi)2(7.107)
(The denominator is N−2since we have 2 inputs, namely the offset term and x.) Hereˆyi=ˆw0+ˆw1xi,
andˆw=(ˆw0,ˆw1)is the MLE.
b. Now assume the following prior on w:
p(w)=p(w0)p(w1) (7.108)
Use an (improper) uniform prior on w0and aN(0,1)prior onw1. Show that this can be written as
a Gaussian prior of the form p(w)=N(w|w0,V0). What are w0andV0?
c. Compute the marginal posterior of the slope, p(w1|D,σ2),w h e r eDis the data above, and σ2is the
unbiased estimate computed above. What is E[
w1|D,σ2]
andvar[
w1|D,σ2]
Show your work. (You
can use Matlab if you like.) Hint: the posterior variance is a very small number!

w1|D,σ2]
andvar[
w1|D,σ2]
d. What is a 95% credible interval for w1?
Exercise 7.9 Generative model for linear regression
Linear regression is the problem of estimating E[Y|x]using a linear function of the form w0+wTx.
Typically we assume that the conditional distribution of YgivenXis Gaussian. We can either estimate this
conditional Gaussian directly (a discriminative approach), or we can ﬁt a Gaussian to the joint distributionofX,Yand then derive E[Y|X=x].
In Exercise 7.5 we showed that the discriminative approach leads to these equations

E[Y|x]=w
0+wTx (7.109)
w0=y−xTw (7.110)
w=(XT
cXc)−1XTcyc (7.111)

7.6. Bayesian linear regression 243
whereXc=X−¯Xis the centered input matrix, and ¯X=1nxTreplicates xacross the rows. Similarly,
yc=y−yis the centered output vector, and y=1nyreplicates yacross the rows.
a. By ﬁnding the maximum likelihood estimates of ΣXX,ΣXY,μXandμY, derive the above equations
by ﬁtting a joint Gaussian to X,Yand using the formula for conditioning a Gaussian (see Section 4.3.1).
Show your work.
b. What are the advantages and disadvantages of this approach compared to the standard discriminative
approach?
Exercise 7.10 Bayesian linear regression using the g-prior
Show that when we use the g-prior, p(w,σ2)=N I G ( w,σ2|0,g(XTX)−1,0,0), the posterior has the
following form:
p(w,σ2|D)=N I G ( w,σ2|wN,VN,aN,bN) (7.112)
VN=g
g+1(XTX)−1(7.113)
wN=g
g+1ˆwmle (7.114)
aN=N/2 (7.115)
bN=s2
2+1
2(g+1)ˆwT
mleXTXˆwmle (7.116)
(7.117)

8 Logistic regression
8.1 Introduction
One way to build a probabilistic classiﬁer is to create a joint model of the form p(y,x)and
then to condition on x, thereby deriving p(y|x). This is called the generative approach. An
alternative approach is to ﬁt a model of the form p(y|x)directly. This is called the discrimi-
nativeapproach, and is the approach we adopt in this chapter. In particular, we will assume
discriminative models which are linear in the parameters. This will turn out to signiﬁcantly sim-
plify model ﬁtting, as we will see. In Section 8.6, we compare the generative and discriminativeapproaches, and in later chapters, we will consider non-linear and non-parametric discriminativemodels.
8.2 Model speciﬁcation
As we discussed in Section 1.4.6, logistic regression corresponds to the following binary classiﬁ-cation model:
p(y|x,w)=B e r (y|sigm(w
Tx)) (8.1)
A 1d example is shown in Figure 1.19(b). Logistic regression can easily be extended to higher-

Tx)) (8.1)
dimensional inputs. For example, Figure 8.1 shows plots of p(y=1|x,w) = sigm( wTx)for
2d input and different weight vectors w. If we threshold these probabilities at 0.5, we induce a
linear decision boundary, whose normal (perpendicular) is given by w.
8.3 Model ﬁtting
In this section, we discuss algorithms for estimating the parameters of a logistic regressionmodel.

246 Chapter 8. Logistic regression
−3 −2 −1 0 1 2 3 4 5 6−3−2−1012345
−10
0
10−1001000.51
x2
W = ( −2 , −1 )
x1
−10
0
x2
W = ( −2 , 3 )
x1
−10
0
x2
W = ( 0 , 2 )
x1
−10
0
x2
W = ( 1 , 4 )
x1
−10
0
x2
W = ( 1 , 0 )
x1
−10
0
x2
W = ( 2 , 2 )
x1
−10
0
x2
W = ( 2 , −2 )
x1
−10
0
x2
W = ( 3 , 0 )
x1
−10
0
x2
W = ( 5 , 4 )
x1
−10
0
x2
W = ( 5 , 1 )
x1
w1w2
Figure 8.1 Plots ofsigm(w1x1+w2x2).H e r ew=(w1,w2)deﬁnes the normal to the decision
boundary. Points to the right of this have sigm(wTx)>0.5, and points to the left have sigm(wTx)<
0.5. Based on Figure 39.3 of (MacKay 2003). Figure generated by sigmoidplot2D .
8.3.1 MLE
The negative log-likelihood for logistic regression is given by
NLL(w)=−N∑
i=1log[μI(yi=1)
i×(1−μi)I(yi=0)] (8.2)
=−N∑
i=1[yilogμi+(1−yi)log(1−μi)] (8.3)
This is also called the cross-entropy error function (see Section 2.8.2).

x1
w1w2
8.3.1 MLE
=−N∑
Another way of writing this is as follows. Suppose ˜yi∈{ −1,+1}instead of yi∈{0,1}.W e
havep(y=1 )=1
1+exp(−wTx)andp(y=1 )=1
1+exp(+ wTx). Hence
NLL(w)=N∑
i=1log(1+exp( −˜yiwTxi)) (8.4)
Unlike linear regression, we can no longer write down the MLE in closed form. Instead, we
need to use an optimization algorithm to compute it. For this, we need to derive the gradient
and Hessian.
In the case of logistic regression, one can show (Exercise 8.3) that the gradient and Hessian

8.3. Model ﬁtting 247
0 0.5 1 1.5 2−0.500.511.522.53
(a)0 0.5 1 1.5 2−0.500.511.522.53
(b)
Figure 8.2 Gradient descent on a simple function, starting from (0,0), for 20 steps, using a ﬁxed
learning rate (step size) η. The global minimum is at (1,1). (a)η=0.1. (b)η=0.6. Figure generated by
steepestDescentDemo .
of this are given by the following
g=d
dwf(w)=∑
i(μi−yi)xi=XT(μ−y) (8.5)
H=d
dwg(w)T=∑
i(∇wμi)xT
i=∑
iμi(1−μi)xixTi(8.6)
=XTSX (8.7)
whereS≜diag(μ i(1−μi)). One can also show (Exercise 8.3) that His positive deﬁnite.
Hence the NLL is convex and has a unique global minimum. Below we discuss some methods
for ﬁnding this minimum.
8.3.2 Steepest descent
Perhaps the simplest algorithm for unconstrained optimization is gradient descent, also known
assteepest descent. This can be written as follows:
θk+1=θk−ηkgk (8.8)
whereηkis thestep size orlearning rate. The main issue in gradient descent is: how should

i=∑
we set the step size? This turns out to be quite tricky. If we use a constant learning rate, butmake it too small, convergence will be very slow, but if we make it too large, the method can failto converge at all. This is illustrated in Figure 8.2. where we plot the following (convex) function
f(θ)=0.5(θ
2
1−θ2)2+0.5(θ1−1)2, (8.9)
We arbitrarily decide to start from (0,0). In Figure 8.2(a), we use a ﬁxed step size of η=0.1;w e

2
see that it moves slowly along the valley. In Figure 8.2(b), we use a ﬁxed step size of η=0.6;w e
see that the algorithm starts oscillating up and down the sides of the valley and never convergesto the optimum.

248 Chapter 8. Logistic regression
exact line searching 1
(a) (b)
Figure 8.3 (a) Steepest descent on the same function as Figure 8.2, starting from (0,0), using line search.
Figure generated by steepestDescentDemo . (b) Illustration of the fact that at the end of a line search
(top of picture), the local gradient of the function will be perpendicular to the search direction. Based on
Figure 10.6.1 of (Press et al. 1988).
Let us develop a more stable method for picking the step size, so that the method is guaran-
teed to converge to a local optimum no matter where we start. (This property is called global
convergence, which should not be confused with convergence to the global optimum!) By
Taylor’s theorem, we have
f(θ+ηd)≈f(θ)+ηgTd (8.10)
wheredis our descent direction. So if ηis chosen small enough, then f(θ+ηd)<f(θ), since
the gradient will be negative. But we don’t want to choose the step size ηtoo small, or we will

move very slowly and may not reach the minimum. So let us pick ηto minimize
φ(η)=f(θk+ηdk) (8.11)
This is called line minimization orline search. There are various methods for solving this 1d
optimization problem; see (Nocedal and Wright 2006) for details.
Figure 8.3(a) demonstrates that line search does indeed work for our simple problem. However,
we see that the steepest descent path with exact line searches exhibits a characteristic zig-zag

behavior. To see why, note that an exact line search satisﬁes ηk=a r g m i n η>0φ(η).A
necessary condition for the optimum is φ′(η)=0. By the chain rule, φ′(η)=dTg,w h e r e
g=f′(θ+ηd)is the gradient at the end of the step. So we either have g=0, which means
we have found a stationary point, or g⊥d, which means that exact search stops at a point
where the local gradient is perpendicular to the search direction. Hence consecutive directionswill be orthogonal (see Figure 8.3(b)). This explains the zig-zag behavior.

One simple heuristic to reduce the effect of zig-zagging is to add a momentum term,(θ
k−
θk−1), as follows:
θk+1=θk−ηkgk+μk(θk−θk−1) (8.12)

8.3. Model ﬁtting 249
where0≤μk≤1controls the importance of the momentum term. In the optimization
community, this is known as the heavy ball method (see e.g., (Bertsekas 1999)).
An alternative way to minimize “zig-zagging” is to use the method of conjugate gradients
(see e.g., (Nocedal and Wright 2006, ch 5) or (Golub and van Loan 1996, Sec 10.2)). This is the
method of choice for quadratic objectives of the form f(θ)=θTAθ, which arise when solving
linear systems. However, non-linear CG is less popular.
8.3.3 Newton’s method
Algorithm 8.1: Newton’s method for minimizing a strictly convex function
1Initializeθ0;
2fork=1,2,... until convergence do
3Evaluategk=∇f(θk);
4EvaluateHk=∇2f(θk);
5SolveHkdk=−gkfordk;
6Use line search to ﬁnd stepsize ηkalongdk;
7θk+1=θk+ηkdk;
One can derive faster optimization methods by taking the curvature of the space (i.e., the
Hessian) into account. These are called second order optimization metods. The primary

example is Newton’s algorithm. This is an iterative algorithm which consists of updates of the
form
θk+1=θk−ηkH−1
kgk (8.13)
The full pseudo-code is given in Algorithm 2.
This algorithm can be derived as follows. Consider making a second-order Taylor series
approximation of f(θ)aroundθk:
fquad(θ)=fk+gT
k(θ−θk)+1
2(θ−θk)THk(θ−θk) (8.14)
Let us rewrite this as
fquad(θ)=θTAθ+bTθ+c (8.15)
where
A=1
2Hk,b=gk−Hkθk,c=fk−gT
kθk+1
2θT
kHkθk (8.16)
The minimum of fquadis at
θ=−1
2A−1b=θk−H−1
kgk (8.17)

form
kgk (8.13)
k(θ−θk)+1
where
A=1
kθk+1
2θT
θ=−1
kgk (8.17)
Thus the Newton step dk=−H−1
kgkis what should be added to θkto minimize the second
order approximation of faroundθk. See Figure 8.4(a) for an illustration.

250 Chapter 8. Logistic regression
f(x)
fquad(x)
xkxk+dk
(a)f(x)
fquad(x)
xkxk+dk
(b)
Figure 8.4 Illustration of Newton’s method for minimizing a 1d function. (a) The solid curve is the
functionf(x). The dotted line fquad(x)is its second order approximation at xk. The Newton step dk
is what must be added to xkto get to the minimum of fquad(x). Based on Figure 13.4 of (Vandenberghe
2006). Figure generated by newtonsMethodMinQuad . (b) Illustration of Newton’s method applied to a
nonconvex function. We ﬁt a quadratic around the current point xkand move to its stationary point,
xk+1=xk+dk. Unfortunately, this is a local maximum, not minimum. This means we need to be careful
about the extent of our quadratic approximation. Based on Figure 13.11 of (Vandenberghe 2006). Figure
generated by newtonsMethodNonConvex .
In its simplest form (as listed), Newton’s method requires that Hkbe positive deﬁnite, which
will hold if the function is strictly convex. If not, the objective function is not convex, then

Hkmay not be positive deﬁnite, so dk=−H−1
kgkmay not be a descent direction (see
Figure 8.4(b) for an example). In this case, one simple strategy is to revert to steepest descent,d
k=−gk. TheLevenberg Marquardt algorithm is an adaptive way to blend between Newton
steps and steepest descent steps. This method is widely used when solving nonlinear leastsquares problems. An alternative approach is this: Rather than computing d
k=−H−1
kgk
directly, we can solve the linear system of equations Hkdk=−gkfordkusing conjugate

k=−H−1
kgk
gradient (CG). If Hkis not positive deﬁnite, we can simply truncate the CG iterations as soon
as negative curvature is detected; this is called truncated Newton.
8.3.4 Iteratively reweighted least squares (IRLS)
Let us now apply Newton’s algorithm to ﬁnd the MLE for binary logistic regression. The Newtonupdate at iteration k+1for this model is as follows (using η
k=1, since the Hessian is exact):
wk+1=wk−H−1gk (8.18)
=wk+(XTSkX)−1XT(y−μk) (8.19)
=(XTSkX)−1[
(XTSkX)wk+XT(y−μk)]
(8.20)
=(XTSkX)−1XT[SkXwk+y−μk] (8.21)

(8.20)
=(XTSkX)−1XTSkzk (8.22)
where we have deﬁned the working response as
zk≜Xwk+S−1
k(y−μk) (8.23)

8.3. Model ﬁtting 251
Equation 8.22 is an example of a weighted least squares problem, which is a minimizer of
N∑
i=1Ski(zki−wTxi)2(8.24)
SinceSkis a diagonal matrix, we can rewrite the targets in component form (for each case
i=1:N)a s
zki=wT
kxi+yi−μki
μki(1−μki)(8.25)
This algorithm is known as iteratively reweighted least squares orIRLSfor short, since at
each iteration, we solve a weighted least squares problem, where the weight matrix Skchanges
at each iteration. See Algorithm 10 for some pseudocode.
Algorithm 8.2: Iteratively reweighted least squares (IRLS)
1w=0D;
2w0= log(y/(1−y));
3repeat
4ηi=w0+wTxi;
5μi=s i g m (ηi);
6si=μi(1−μi);
7zi=ηi+yi−μi
si;
8S=d i a g (s1:N);
9w=(XTSX)−1XTSz;
10until converged ;
8.3.5 Quasi-Newton (variable metric) methods
The mother of all second-order optimization algorithm is Newton’s algorithm, which we dis-
cussed in Section 8.3.3. Unfortunately, it may be too expensive to compute Hexplicitly. Quasi-

1w=0D;
3repeat
si;
Newtonmethods iteratively build up an approximation to the Hessian using information gleaned
from the gradient vector at each step. The most common method is called BFGS(named after
its inventors, Broyden, Fletcher, Goldfarb and Shanno), which updates the approximation to theHessianB
k≈Hkas follows:
Bk+1=Bk+ykyT
k
yT
ksk−(Bksk)(Bksk)T
sT
kBksk(8.26)
sk=θk−θk−1 (8.27)
yk=gk−gk−1 (8.28)
This is a rank-two update to the matrix, and ensures that the matrix remains positive deﬁnite

k
yT
sT
(under certain restrictions on the step size). We typically start with a diagonal approximation,B
0=I. Thus BFGS can be thought of as a “diagonal plus low-rank” approximation to the
Hessian.

252 Chapter 8. Logistic regression
Alternatively, BFGS can iteratively update an approximation to the inverse Hessian, Ck≈H−1
k,
Ck+1=(
I−skyT
k
yT
ksk)
Ck(
I−yksT
k
yT
ksk)
+sksTk
yT
ksk(8.29)
Since storing the Hessian takes O(D2)space, for very large problems, one can use limited
memory BFGS,o rL-BFGS,w h e r e HkorH−1
kis approximated by a diagonal plus low rank
matrix. In particular, the product H−1
kgkcan be obtained by performing a sequence of inner
products with skandyk, using only the mmost recent (sk,yk)pairs, and ignoring older
information. The storage requirements are therefore O(mD). Typically m∼20suffices for
good performance. See (Nocedal and Wright 2006, p177) for more information. L-BFGS is
often the method of choice for most unconstrained smooth optimization problems that arise inmachine learning (although see Section 8.5).
8.3.6ℓ2regularization

Just as we prefer ridge regression to linear regression, so we should prefer MAP estimation forlogistic regression to computing the MLE. In fact, regularization is important in the classiﬁcationsetting even if we have lots of data. To see why, suppose the data is linearly separable. Inthis case, the MLE is obtained when ||w|| → ∞, corresponding to an inﬁnitely steep sigmoid
function, I(w
Tx>w0), also known as a linear threshold unit. This assigns the maximal

amount of probability mass to the training data. However, such a solution is very brittle andwill not generalize well.
To prevent this, we can use ℓ
2regularization, just as we did with ridge regression. We note
that the new objective, gradient and Hessian have the following forms:
f′(w) = NLL( w)+λwTw (8.30)
g′(w)=g (w)+λw (8.31)
H′(w)=H( w)+λI (8.32)
It is a simple matter to pass these modiﬁed equations into any gradient-based optimizer.
8.3.7 Multi-class logistic regression

Now we consider multinomial logistic regression , sometimes called a maximum entropy
classiﬁer. This is a model of the form
p(y=c|x,W)=exp(wT
cx)∑C
c′=1exp(wT
c′x)(8.33)
A slight variant, known as a conditional logit model, normalizes over a different set of classes
for each data case; this can be useful for modeling choices that users make between different
sets of items that are offered to them.
Let us now introduce some notation. Let μic=p(yi=c|xi,W)=S(ηi)c,w h e r eηi=

cx)∑C
WTxiis aC×1vector. Also, let yic=I(yi=c)be the one-of-C encoding of yi; thusyiis a
bit vector, in which the c’th bit turns on iff yi=c. Following (Krishnapuram et al. 2005), let us

8.3. Model ﬁtting 253
setwC=0, to ensure identiﬁability, and deﬁne w= vec(W(:,1:C−1))to be aD×(C−1)
column vector.
With this, the log-likelihood can be written as
ℓ(W)=l o gN∏
i=1C∏
c=1μyic
ic=N∑
i=1C∑
c=1yiclogμic (8.34)
=N∑
i=1[(C∑
c=1yicwT
cxi)
−log(C∑
c′xi))]
(8.35)
Deﬁne the NLL as
f(w)=−ℓ(w) (8.36)
We now proceed to compute the gradient and Hessian of this expression. Since wis block-
structured, the notation gets a bit heavy, but the ideas are simple. It helps to deﬁne A⊗B
be thekronecker product of matrices AandB.I fAis anm×nmatrix and Bis ap×q
matrix, then A×Bis themp×nqblock matrix
A⊗B=⎡
⎢⎣a11B···a1nB
.........
a
m1B···amnB⎤
⎥⎦ (8.37)
Returning to the task at hand, one can show (Exercise 8.4) that the gradient is given by
g(W)=∇f(w)=N∑
i=1(μi−yi)⊗xi (8.38)
whereyi=(I(yi=1 ),...,I(yi=C−1))andμi(W)=[p(yi=1|xi,W),...,p(yi=
C−1|xi,W)]are column vectors of length C−1, For example, if we have D=3feature
dimensions and C=3classes, this becomes
g(W)=∑
i⎛
⎜⎜⎜⎜⎜⎜⎝(μ
i1−yi1)xi1
(μi1−yi1)xi2

A⊗B=⎡
.........
a
⎥⎦ (8.37)
g(W)=∑
i⎛
⎜⎜⎜⎜⎜⎜⎝(μ
(μi1−yi1)xi3
(μi2−yi2)xi1
(μi2−yi2)xi2
(μi2−yi2)xi3⎞
⎟⎟⎟⎟⎟⎟⎠(8.39)
In other words, for each class c, the derivative for the weights in the c’th column is
∇
wcf(W)=∑
i(μic−yic)xi (8.40)
This has the same form as in the binary logistic regression case, namely an error term times xi.
(This turns out to be a general property of distributions in the exponential family, as we will see
in Section 9.3.2.)

254 Chapter 8. Logistic regression
One can also show (Exercise 8.4) that the Hessian is the following block structured D(C−
1)×D(C−1)matrix:
H(W)=∇2f(w)=N∑
i=1(diag(μi)−μiμT
i)⊗(xixTi) (8.41)
For example, if we have 3 features and 3 classes, this becomes
H(W)=∑
i(μi1−μ2
i1−μi1μi2
−μi1μi2μi2−μ2i2)
⊗⎛
⎝xi1xi1xi1xi2xi1xi3
xi2xi1xi2xi2xi2xi3
xi3xi1xi3xi2xi3xi3⎞⎠ (8.42)
=∑
i((μi1−μ2
i1)Xi−μi1μi2Xi
−μi1μi2Xi(μi2−μ2i2)Xi)
(8.43)
whereXi=xixTi. In other words, the block c,c′submatrix is given by
Hc,c′(W)=∑
iμic(δc,c′−μi,c′)xixT
i (8.44)
This is also a positive deﬁnite matrix, so there is a unique MLE.
Now consider minimizing
f′(W)≜−logp(D|w)−logp(W) (8.45)
wherep(W)=∏
cN(wc|0,V0). The new objective, its gradient and Hessian are given by
f′(W)=f (W)+1
2∑
cwcV−1
0wc (8.46)
g′(W)=g (W)+V−1
0(∑
cwc) (8.47)
H′(W)=H( W)+IC⊗V−1
0 (8.48)
This can be passed to any gradient-based optimizer to ﬁnd the MAP estimate. Note, however,
that the Hessian has size O((CD)×(CD)), which is Ctimes more row and columns than

i (8.44)
2∑
cwcV−1
0wc (8.46)
0(∑
0 (8.48)
in the binary case, so limited memory BFGS is more appropriate than Newton’s method. SeelogregFit for some Matlab code.
8.4 Bayesian logistic regression
It is natural to want to compute the full posterior over the parameters, p(w|D), for logistic
regression models. This can be useful for any situation where we want to associate conﬁdenceintervals with our predictions (e.g., this is necessary when solving contextual bandit problems,discussed in Section 5.7.3.1).

Unfortunately, unlike the linear regression case, this cannot be done exactly, since there is no
convenient conjugate prior for logistic regression. We discuss one simple approximation below;some other approaches include MCMC (Section 24.3.3.1), variational inference (Section 21.8.1.1),expectation propagation (Kuss and Rasmussen 2005), etc. For notational simplicity, we stick tobinary logistic regression.

8.4. Bayesian logistic regression 255
8.4.1 Laplace approximation
In this section, we discuss how to make a Gaussian approximation to a posterior distribution.
The approximation works as follows. Suppose θ∈RD.L e t
p(θ|D)=1
Ze−E(θ)(8.49)
whereE(θ)is called an energy function, and is equal to the negative log of the unnormal-
ized log posterior, E(θ)=−logp(θ,D), withZ=p(D)being the normalization constant.
Performing a Taylor series expansion around the mode θ∗(i.e., the lowest energy state) we get
E(θ)≈E(θ∗)+(θ−θ∗)Tg+1
2(θ−θ∗)TH(θ−θ∗) (8.50)
wheregis the gradient and His the Hessian of the energy function evaluated at the mode:
g≜∇E(θ)⏐⏐
θ∗,H≜∂2E(θ)
∂θ∂θT|θ∗ (8.51)
Sinceθ∗is the mode, the gradient term is zero. Hence
ˆp(θ|D)≈1
Ze−E(θ∗)exp[
−1
2(θ−θ∗)TH(θ−θ∗)]
(8.52)
=N(θ|θ∗,H−1) (8.53)
Z=p(D)≈∫
ˆp(θ|D)dθ=e−E(θ∗)(2π)D/2|H|−1
2 (8.54)
The last line follows from normalization constant of the multivariate Gaussian.
Equation 8.54 is known as the Laplace approximation to the marginal likelihood. Therefore

g≜∇E(θ)⏐⏐
ˆp(θ|D)≈1
−1
(8.52)
Z=p(D)≈∫
2 (8.54)
Equation 8.52 is sometimes called the the Laplace approximation to the posterior. However,
in the statistics community, the term “Laplace approximation” refers to a more sophisticated

ˆp(θ|D)≈1
−1
(8.52)
Z=p(D)≈∫
2 (8.54)
method (see e.g. (Rue et al. 2009) for details). It may therefore be better to use the term“Gaussian approximation” to refer to Equation 8.52. A Gaussian approximation is often areasonable approximation, since posteriors often become more “Gaussian-like” as the samplesize increases, for reasons analogous to the central limit theorem. (In physics, there is ananalogous technique known as a saddle point approximation.)
8.4.2 Derivation of the BIC

We can use the Gaussian approximation to write the log marginal likelihood as follows, droppingirrelevant constants:
logp(D)≈logp(D|θ
∗)+logp(θ∗)−1
2log|H| (8.55)
The penalization terms which are added to the logp(D|θ∗)are sometimes called the Occam
factor, and are a measure of model complexity. If we have a uniform prior, p(θ)∝1, we can
drop the second term, and replace θ∗with the MLE, ˆθ.

256 Chapter 8. Logistic regression
We now focus on approximating the third term. We have H=∑N
i=1Hi,w h e r eHi=
∇∇logp(Di|θ). Let us approximate each Hib yaﬁ x e dm a t r i x ˆH. Then we have
log|H|=l o g|NˆH|= log(Nd|ˆH|)=DlogN+log|ˆH| (8.56)
whereD=d i m (θ)and we have assumed His full rank. We can drop the log|ˆH|term, since
it is independent of N, and thus will get overwhelmed by the likelihood. Putting all the pieces
together, we recover the BIC score (Section 5.3.2.4):
logp(D)≈logp(D|ˆθ)−D
2logN (8.57)
8.4.3 Gaussian approximation for logistic regression
Now let us apply the Gaussian approximation to logistic regression. We will use a a Gaussian
prior of the form p(w)=N(w|0,V0), just as we did in MAP estimation. The approximate
p(w|D)≈N(w|ˆw,H−1) (8.58)
whereˆw=a r gm i n wE(w),E(w)=−(logp(D|w)+logp(w)), andH=∇2E(w)|ˆw.
As an example, consider the linearly separable 2D data in Figure 8.5(a). There are many

parameter settings that correspond to lines that perfectly separate the training data; we show 4examples. The likelihood surface is shown in Figure 8.5(b), where we see that the likelihood isunbounded as we move up and to the right in parameter space, along a ridge where w
2/w1=
2.35(this is indicated by the diagonal line). The reasons for this is that we can maximize the
likelihood by driving ||w||to inﬁnity (subject to being on this line), since large regression weights

2/w1=
make the sigmoid function very steep, turning it into a step function. Consequently the MLE isnot well deﬁned when the data is linearly separable.
Toregularizetheproblem, letususeavaguesphericalpriorcenteredattheorigin, N(w|0,100I).

2/w1=
Multiplying this spherical prior by the likelihood surface results in a highly skewed posterior,shown in Figure 8.5(c). (The posterior is skewed because the likelihood function “chops off”regions of parameter space (in a “soft” fashion) which disagree with the data.) The MAP estimateis shown by the blue dot. Unlike the MLE, this is not at inﬁnity.
The Gaussian approximation to this posterior is shown in Figure 8.5(d). We see that this is

a symmetric distribution, and therefore not a great approximation. Of course, it gets the modecorrect (by construction), and it at least represents the fact that there is more uncertainty alongthe southwest-northeast direction (which corresponds to uncertainty about the orientation ofseparating lines) than perpendicular to this. Although a crude approximation, this is surelybetter than approximating the posterior by a delta function, which is what MAP estimation does.
8.4.4 Approximating the posterior predictive

Given the posterior, we can compute credible intervals, perform hypothesis tests, etc., just as wedid in Section 7.6.3.3 in the case of linear regression. But in machine learning, interest usuallyfocusses on prediction. The posterior predictive distribution has the form
p(y|x,D)=∫
p(y|x,w)p(w|D)dw (8.59)

8.4. Bayesian logistic regression 257
−10 −5 0 5−8−6−4−202468data
(a)Log−Likelihood
123 4
−8 −6 −4 −2 0 2 4 6 8−8−6−4−202468
(b)
Log−Unnormalised Posterior
(c)Laplace Approximation to Posterior
(d)
Figure 8.5 (a) Two-class data in 2d. (b) Log-likelihood for a logistic regression model. The line is drawn
from the origin in the direction of the MLE (which is at inﬁnity). The numbers correspond to 4 points
in parameter space, corresponding to the lines in (a). (c) Unnormalized log posterior (assuming vaguespherical prior). (d) Laplace approximation to posterior. Based on a ﬁgure by Mark Girolami. Figuregenerated by logregLaplaceGirolamiDemo .
Unfortunately this integral is intractable.
The simplest approximation is the plug-in approximation, which, in the binary case, takes the
form
p(y=1|x,D)≈p(y=1|x,E[w]) (8.60)
where E[w]is the posterior mean. In this context, E[w]is called the Bayes point. Of course,

form
such a plug-in estimate underestimates the uncertainty. We discuss some better approximations
below.

258 Chapter 8. Logistic regression
p(y=1|x, wMAP)
(a)−10 −8 −6 −4 −2 0 2 4 6 8−8−6−4−202468decision boundary for sampled w
(b)
MC approx of p(y=1|x)
(c)numerical approx of p(y=1|x)
(d)
Figure 8.6 Posterior predictive distribution for a logistic regression model in 2d. Top left: contours of
p(y=1|x,ˆwmap). Top right: samples from the posterior predictive distribution. Bottom left: Averaging
over these samples. Bottom right: moderated output (probit approximation). Based on a ﬁgure by Mark
Girolami. Figure generated by logregLaplaceGirolamiDemo .
8.4.4.1 Monte Carlo approximation
A better approach is to use a Monte Carlo approximation, as follows:
p(y=1|x,D)≈1
SS∑
s=1sigm((ws)Tx) (8.61)
wherews∼p(w|D)are samples from the posterior. (This technique can be trivially extended
to the multi-class case.) If we have approximated the posterior using Monte Carlo, we can reuse

SS∑
these samples for prediction. If we made a Gaussian approximation to the posterior, we candraw independent samples from the Gaussian using standard methods.
Figure 8.6(b) shows samples from the posteiror predictive for our 2d example. Figure 8.6(c)

8.4. Bayesian logistic regression 259
460 480 500 520 540 560 580 600 620 64000.10.20.30.40.50.60.70.80.91
(a)−6 −4 −2 0 2 4 600.10.20.30.40.50.60.70.80.91
  
sigmoid
probit
(b)
Figure 8.7 (a) Posterior predictive density for SAT data. The red circle denotes the posterior mean, the
blue cross the posterior median, and the blue lines denote the 5th and 95th percentiles of the predictive
distribution. Figure generated by logregSATdemoBayes . (b) The logistic (sigmoid) function sigm(x) in
solid red, with the rescaled probit function Φ(λx)in dotted blue superimposed. Here λ=√
π/8, which
was chosen so that the derivatives of the two curves match at x=0. Based on Figure 4.9 of (Bishop
2006b). Figure generated by probitPlot . Figure generated by probitRegDemo .
shows the average of these samples. By averaging over multiple predictions, we see that the

π/8, which
uncertainty in the decision boundary “splays out” as we move further from the training data.So although the decision boundary is linear, the posterior predictive density is not linear. Notealso that the posterior mean decision boundary is roughly equally far from both classes; this isthe Bayesian analog of the large margin principle discussed in Section 14.5.2.2.
Figure 8.7(a) shows an example in 1d. The red dots denote the mean of the posterior predictive

evaluated at the training data. The vertical blue lines denote 95% credible intervals for theposterior predictive; the small blue star is the median. We see that, with the Bayesian approach,we are able to model our uncertainty about the probability a student will pass the exam basedon his SAT score, rather than just getting a point estimate.
8.4.4.2 Probit approximation (moderated output) *
If we have a Gaussian approximation to the posterior p(w|D)≈N(w|m
N,VN), we can also

compute a deterministic approximation to the posterior predictive distribution, at least in thebinary case. We proceed as follows:
p(y=1|x,D)≈∫
sigm(w
Tx)p(w|D)dw=∫
sigm(a)N(a|μa,σ2
a)da (8.62)
a≜wTx (8.63)
μa≜E[a]=mT
Nx (8.64)
σ2
a≜var[a]=∫
p(a|D)[a2−E[
a2]
]da (8.65)
=∫
p(w|D)[(wTx)2−(mTNx)2]dw=xTVNx (8.66)

260 Chapter 8. Logistic regression
Thus we see that we need to evaluate the expectation of a sigmoid with respect to a Gaussian.
This can be approximated by exploiting the fact that the sigmoid function is similar to theprobitfunction, which is given by the cdf of the standard normal:
Φ(a)≜∫
a
−∞N(x|0,1)dx (8.67)
Figure 8.7(b) plots the sigmoid and probit functions. We have rescaled the axes so that sigm(a)
has the same slope as Φ(λa)at the origin, where λ2=π/8.
The advantage of using the probit is that one can convolve it with a Gaussian analytically:
∫
Φ(λa)N(a|μ,σ2)da=Φ(a
(λ−2+σ2)1
2)
(8.68)
We now plug in the approximation sigm(a)≈Φ(λa)to both sides of this equation to get
∫
sigm(a)N(a|μ,σ2)da≈sigm(κ(σ2)μ) (8.69)
κ(σ2)≜(1+πσ2/8)−1
2 (8.70)
Applying this to the logistic regression model we get the following expression (ﬁrst suggested in(Spiegelhalter and Lauritzen 1990)):
p(y=1|x,D)≈sigm(κ(σ
2
a)μa) (8.71)
Figure 8.6(d) indicates that this gives very similar results to the Monte Carlo approximation.

∫
(λ−2+σ2)1
2)
(8.68)
∫
2 (8.70)
2
Using Equation 8.71 is sometimes called a moderated output, since it is less extreme than
the plug-in estimate. To see this, note that 0≤κ(σ2)≤1and hence
sigm(κ(σ2)μ)≤sigm(μ)=p(y=1|x,ˆw) (8.72)
where the inequality is strict if μ̸=0.I fμ>0,w eh a v e p(y=1|x,ˆw)>0.5, but the
moderated prediction is always closer to 0.5, so it is less conﬁdent. However, the decisionboundary occurs whenever p(y=1|x,D) = sigm(κ( σ
2)μ)=0.5, which implies μ=ˆwTx=

0. Hence the decision boundary for the moderated approximation is the same as for the plug-inapproximation. So the number of misclassiﬁcations will be the same for the two methods,but the log-likelihood will not. (Note that in the multiclass case, taking into account posteriorcovariance gives different answers than the plug-in approach: see Exercise 3.10.3 of (Rasmussenand Williams 2006).)
8.4.5 Residual analysis (outlier detection) *
It is sometimes useful to detect data cases which are “outliers”. This is called residual analysis

orcase analysis. In a regression setting, this can be performed by computing ri=yi−ˆyi,w h e r e
ˆyi=ˆwTxi. These values should follow a N(0,σ2)distribution, if the modelling assumptions
are correct. This can be assessed by creating a qq-plot, where we plot the Ntheoretical
quantiles of a Gaussian distribution against the Nempirical quantiles of the ri. Points that
deviate from the straightline are potential outliers.

8.5. Online learning and stochastic optimization 261
Classical methods, based on residuals, do not work well for binary data, because they rely
on asymptotic normality of the test statistics. However, adopting a Bayesian approach, we
can just deﬁne outliers to be points which which p(yi|ˆyi)is small, where we typically use
ˆyi=s i g m (ˆwTxi). Note that ˆwwas estimated from all the data. A better method is to exclude
(xi,yi)from the estimate of wwhen predicting yi. That is, we deﬁne outliers to be points
which have low probability under the cross-validated posterior predictive distribution, deﬁnedby
p(y
i|xi,x−i,y−i)=∫
p(yi|xi,w)∏
i′̸=ip(yi′|xi′,w)p(w)dw (8.73)
This can be efficiently approximated by sampling methods (Gelfand 1996). For further discussionof residual analysis in logistic regression models, see e.g.,(Johnson and Albert 1999, Sec 3.4).
8.5 Online learning and stochastic optimization
Traditionally machine learning is performed offline, which means we have a batchof data, and

p(y
we optimize an equation of the following form
f(θ)=1
NN∑
i=1f(θ,zi) (8.74)
wherezi=(xi,yi)in the supervised case, or just xiin the unsupervised case, and f(θ,zi)is
some kind of loss function. For example, we might use
f(θ,zi)=−logp(yi|xi,θ) (8.75)
in which case we are trying to maximize the likelihood. Alternatively, we might use
f(θ,zi)=L(yi,h(xi,θ)) (8.76)
whereh(xi,θ)is a prediction function, and L(y,ˆy)is some other loss function such as squared

f(θ)=1
NN∑
error or the Huber loss. In frequentist decision theory, the average loss is called the risk (seeSection 6.3), so this overall approach is called empirical risk minimization or ERM (see Section 6.5for details).
However, if we have streaming data, we need to perform online learning, so we can update

our estimates as each new data point arrives rather than waiting until “the end” (which maynever occur). And even if we have a batch of data, we might want to treat it like a stream if it istoo large to hold in main memory. Below we discuss learning methods for this kind of scenario.
1
1. A simple implementation trick can be used to speed up batch learning algorithms when applied to data sets that
are too large to hold in memory. First note that the naive implementation makes a pass over the data ﬁle, from the

1
beginning to end, accumulating the sufficient statistics and gradients as it goes; then an update is performed and theprocess repeats. Unfortunately, at the end of each pass, the data from the beginning of the ﬁle will have been evicted
from the cache (since are are assuming it cannot all ﬁt into memory). Rather than going back to the beginning of the
ﬁle and reloading it, we can simply work backwards from the end of the ﬁle, which is already in memory. We then
repeat this forwards-backwards pattern over the data. This simple trick is known as rocking.

262 Chapter 8. Logistic regression
8.5.1 Online learning and regret minimization
Suppose that at each step, “nature” presents a sample zkand the “learner” must respond with
a parameter estimate θk. In the theoretical machine learning community, the objective used in
online learning is the regret, which is the averaged loss incurred relative to the best we could
have gotten in hindsight using a single ﬁxed parameter value:
regretk≜1
kk∑
t=1f(θt,zt)−min
θ∗∈Θ1
kk∑
t=1f(θ∗,zt) (8.77)
For example, imagine we are investing in the stock-market. Let θjbe the amount we invest in
stockj, and letzjbe the return on this stock. Our loss function is f(θ,z)=−θTz. The regret
is how much better (or worse) we did by trading at each step, rather than adopting a “buy and
hold” strategy using an oracle to choose which stocks to buy.
One simple algorithm for online learning is online gradient descent (Zinkevich 2003), which
is as follows: at each step k, update the parameters using
θk+1=p r ojΘ(θk−ηkgk) (8.78)

whereprojV(v) = argminw∈V||w−v||2is the projection of vector vonto space V,gk=
∇f(θk,zk)is the gradient, and ηkis the step size. (The projection step is only needed if
the parameter must be constrained to live in a certain subset of RD. See Section 13.4.3 for
details.) Below we will see how this approach to regret minimization relates to more traditionalobjectives, such as MLE.
There are a variety of other approaches to regret minimization which are beyond the scope
of this book (see e.g., Cesa-Bianchi and Lugosi (2006) for details).
8.5.2 Stochastic optimization and risk minimization

Now suppose that instead of minimizing regret with respect to the past, we want to minimizeexpected loss in the future, as is more common in (frequentist) statistical learning theory. Thatis, we want to minimize
f(θ)=E[f(θ,z)] (8.79)
where the expectation is taken over future data. Optimizing functions where some of thevariables in the objective are random is called stochastic optimization.
2
Suppose we receive an inﬁnite stream of samples from the distribution. One way to optimize

2
stochastic objectives such as Equation 8.79 is to perform the update in Equation 8.78 at eachstep. This is called stochastic gradient descent orSGD(Nemirovski and Yudin 1978). Since we
typically want a single parameter estimate, we can use a running average:
θk=1
kk∑
t=1θt (8.80)
2. Note that in stochastic optimization, the objective is stochastic, and therefore the algorithms will be, too. However,
it is also possible to apply stochastic optimization algorithms to deterministic objectives. Examples include simulated

θk=1
kk∑
annealing (Section 24.6.1) and stochastic gradient descent applied to the empirical risk minimization problem. There are
some interesting theoretical connections between online learning and stochastic optimization (Cesa-Bianchi and Lugosi
2006), but this is beyond the scope of this book.

8.5. Online learning and stochastic optimization 263
This is called Polyak-Ruppert averaging, and can be implemented recursively as follows:
θk=θk−1−1
k(θk−1−θk) (8.81)
See e.g., (Spall 2003; Kushner and Yin 2003) for details.
8.5.2.1 Setting the step size
We now discuss some sufficient conditions on the learning rate to guarantee convergence of
SGD. These are known as the Robbins-Monro conditions:
∞∑
k=1ηk=∞,∞∑
k=1η2
k<∞. (8.82)
The set of values of ηkover time is called the learning rate schedule. Various formulas are
used, such as ηk=1/k, or the following (Bottou 1998; Bach and Moulines 2011):
ηk=(τ0+k)−κ(8.83)
whereτ0≥0slows down early iterations of the algorithm, and κ∈(0.5,1]controls the rate at
which old values of are forgotten.
The need to adjust these tuning parameters is one of the main drawback of stochastic
optimization. One simple heuristic (Bottou 2007) is as follows: store an initial subset of thedata, and try a range of ηvalues on this subset; then choose the one that results in the fastest

decrease in the objective and apply it to all the rest of the data. Note that this may not resultin convergence, but the algorithm can be terminated when the performance improvement on ahold-out set plateaus (this is called early stopping).
8.5.2.2 Per-parameter step sizes
One drawback of SGD is that it uses the same step size for all parameters. We now brieﬂypresent a method known as adagrad (short for adaptive gradient) (Duchi et al. 2010), which is

similar in spirit to a diagonal Hessian approximation. (See also (Schaul et al. 2012) for a similarapproach.) In particular, if θ
i(k)is parameter iat timek, andgi(k)is its gradient, then we
make an update as follows:
θi(k+1)=θi(k)−ηgi(k)
τ0+√
si(k)(8.84)
where the diagonal step size vector is the gradient vector squared, summed over all time steps.This can be recursively updated as follows:
s
i(k)=si(k−1)+gi(k)2(8.85)

τ0+√
s
The result is a per-parameter step size that adapts to the curvature of the loss function. Thismethod was original derived for the regret minimization case, but it can be applied moregenerally.

264 Chapter 8. Logistic regression
8.5.2.3 SGD compared to batch learning
If we don’t have an inﬁnite data stream, we can “simulate” one by sampling data points at
random from our training set. Essentially we are optimizing Equation 8.74 by treating it as anexpectation with respect to the empirical distribution.
Algorithm 8.3: Stochastic gradient descent
1Initializeθ,η;
2repeat
3Randomly permute data;
4fori=1:Ndo
5 g=∇f(θ,zi);
6 θ←projΘ(θ−ηg);
7 Updateη;
8until converged ;
In theory, we should sample with replacement, although in practice it is usually better to
randomly permute the data and sample without replacement, and then to repeat. A single suchpass over the entire data set is called an epoch. See Algorithm 8 for some pseudocode.
In this offline case, it is often better to compute the gradient of a mini-batch ofBdata cases.
IfB=1, this is standard SGD, and if B=N, this is standard steepest descent. Typically
B∼100is used.

7 Updateη;
Although a simple ﬁrst-order method, SGD performs surprisingly well on some problems,

especially ones with large data sets (Bottou 2007). The intuitive reason for this is that one canget a fairly good estimate of the gradient by looking at just a few examples. Carefully evaluatingprecise gradients using large datasets is often a waste of time, since the algorithm will haveto recompute the gradient again anyway at the next step. It is often a better use of computertime to have a noisy estimate and to move rapidly through parameter space. As an extremeexample, suppose we double the training set by duplicating every example. Batch methods willtake twice as long, but online methods will be unaffected, since the direction of the gradienthas not changed (doubling the size of the data changes the magnitude of the gradient, but thatis irrelevant, since the gradient is being scaled by the step size anyway).

In addition to enhanced speed, SGD is often less prone to getting stuck in shallow local
minima, because it adds a certain amount of “noise”. Consequently it is quite popular in themachine learning community for ﬁtting models with non-convex objectives, such as neuralnetworks (Section 16.5) and deep belief networks (Section 28.1).
8.5.3 The LMS algorithm
As an example of SGD, let us consider how to compute the MLE for linear regression in anonline fashion. We derived the batch gradient in Equation 7.14. The online gradient at iterationkis given by
g
k=xi(θT
kxi−yi) (8.86)

8.5. Online learning and stochastic optimization 265
black line = LMS trajectory towards LS soln (red cross)
w0w1
(a)0 5 10 15 20 25 30345678910RSS vs iteration
(b)
Figure 8.8 Illustration of the LMS algorithm. Left: we start from θ=(−0.5,2)and slowly converging
to the least squares solution of ˆθ=( 1.45,0.92)(red cross). Right: plot of objective function over time.
Note that it does not decrease monotonically. Figure generated by LMSdemo.
wherei=i(k)is the training example to use at iteration k. If the data set is streaming, we use
i(k)=k; we shall assume this from now on, for notational simplicity. Equation 8.86 is easy
to interpret: it is the feature vector xkweighted by the difference between what we predicted,
ˆyk=θT
kxk, and the true response, yk; hence the gradient acts like an error signal.
After computing the gradient, we take a step along it as follows:
θk+1=θk−ηk(ˆyk−yk)xk (8.87)

ˆyk=θT
(There is no need for a projection step, since this is an unconstrained optimization problem.)
This algorithm is called the least mean squares orLMSalgorithm, and is also known as the
delta rule,o rt h eWidrow-Hoff rule.
Figure 8.8 shows the results of applying this algorithm to the data shown in Figure 7.2. We
start atθ=(−0.5,2)and converge (in the sense that ||θk−θk−1||2
2drops below a threshold
of10−2) in about 26 iterations.
Note that LMS may require multiple passes through the data to ﬁnd the optimum. By

contrast, the recursive least squares algorithm, which is based on the Kalman ﬁlter and which
uses second-order information, ﬁnds the optimum in a single pass (see Section 18.2.3). See alsoExercise 7.7.
8.5.4 The perceptron algorithm
Now let us consider how to ﬁt a binary logistic regression model in an online manner. Thebatch gradient was given in Equation 8.5. In the online case, the weight update has the simpleform
θ
k=θk−1−ηkgi=θk−1−ηk(μi−yi)xi (8.88)
whereμi=p(yi=1|xi,θk)=E[yi|xi,θk]. We see that this has exactly the same form as the

θ
LMS algorithm. Indeed, this property holds for all generalized linear models (Section 9.3).

266 Chapter 8. Logistic regression
We now consider an approximation to this algorithm. Speciﬁcally, let
ˆyi=a r g m a x
y∈{0,1}p(y|xi,θ) (8.89)
represent the most probable class label. We replace μi=p(y=1|xi,θ) = sigm(θTxi)in the
gradient expression with ˆyi. Thus the approximate gradient becomes
gi≈(ˆyi−yi)xi (8.90)
It will make the algebra simpler if we assume y∈{ −1,+1}rather than y∈{0,1}. In this
case, our prediction becomes
ˆyi=s i g n (θTxi) (8.91)
Then ifˆyiyi=−1, we have made an error, but if ˆyiyi=+ 1, we guessed the right label.
At each step, we update the weight vector by adding on the gradient. The key observation is
that, if we predicted correctly, then ˆyi=yi, so the (approximate) gradient is zero and we do
not change the weight vector. But if xiis misclassiﬁed, we update the weights as follows: If
ˆyi=1butyi=−1, then the negative gradient is −(ˆyi−yi)xi=−2xi; and ifˆyi=−1but
yi=1, then the negative gradient is −(ˆyi−yi)xi=2xi. We can absorb the factor of 2 into

the learning rate ηand just write the update, in the case of a misclassiﬁcation, as
θk=θk−1+ηkyixi (8.92)
Since it is only the sign of the weights that matter, not the magnitude, we will set ηk=1. See
Algorithm 11 for the pseudocode.
One can show that this method, known as the perceptron algorithm (Rosenblatt 1958), will
converge, provided the data is linearly separable, i.e., that there exist parameters θsuch that
predicting with sign(θTx)achieves 0 error on the training set. However, if the data is not

linearly separable, the algorithm will not converge, and even if it does converge, it may take

a long time. There are much better ways to train logistic regression models (such as usingproper SGD, without the gradient approximation, or IRLS, discussed in Section 8.3.4). However,the perceptron algorithm is historically important: it was one of the ﬁrst machine learningalgorithms ever derived (by Frank Rosenblatt in 1957), and was even implemented in analoghardware. In addition, the algorithm can be used to ﬁt models where computing marginalsp(y

i|x,θ)is more expensive than computing the MAP output, argmax yp(y|x,θ); this arises
in some structured-output classiﬁcation problems. See Section 19.7 for details.
8.5.5 A Bayesian view
Another approach to online learning is to adopt a Bayesian view. This is conceptually quitesimple: we just apply Bayes rule recursively:
p(θ|D
1:k)∝p(Dk|θ)p(θ|D1:k−1) (8.93)

p(θ|D
This has the obvious advantage of returning a posterior instead of just a point estimate. It alsoallows for the online adaptation of hyper-parameters, which is important since cross-validationcannot be used in an online setting. Finally, it has the (less obvious) advantage that it can be

8.6. Generative vs discriminative classiﬁers 267
Algorithm 8.4: Perceptron algorithm
1Input: linearly separable data set xi∈RD,yi∈{ −1,+1}fori=1:N;
2Initializeθ0;
3k←0;
4repeat
5k←k+1;
6i←kmodN;
7ifˆyi̸=yithen
8 θk+1←θk+yixi
9else
10 no-op
11until converged ;
quicker than SGD. To see why, note that by modeling the posterior variance of each parameter
in addition to its mean, we effectively associate a different learning rate for each parameter(de Freitas et al. 2000), which is a simple way to model the curvature of the space. Thesevariances can then be adapted using the usual rules of probability theory. By contrast, gettingsecond-order optimization methods to work online is more tricky (see e.g., (Schraudolph et al.2007; Sunehag et al. 2009; Bordes et al. 2009, 2010)).
As a simple example, in Section 18.2.3 we show how to use the Kalman ﬁlter to ﬁt a linear

regression model online. Unlike the LMS algorithm, this converges to the optimal (offline) answerin a single pass over the data. An extension which can learn a robust non-linear regression modelin an online fashion is described in (Ting et al. 2010). For the GLM case, we can use an assumeddensity ﬁlter (Section 18.5.3), where we approximate the posterior by a Gaussian with a diagonalcovariance; the variance terms serve as a per-parameter step-size. See Section 18.5.3.2 for details.Another approach is to use particle ﬁltering (Section 23.5); this was used in (Andrieu et al. 2000)for sequentially learning a kernelized linear/logistic regression model.
8.6 Generative vs discriminative classiﬁers
In Section 4.2.2, we showed that the posterior over class labels induced by Gaussian discrim-inant analysis (GDA) has exactly the same form as logistic regression, namely p(y=1|x)=
sigm(w

sigm(w
Tx). The decision boundary is therefore a linear function of xin both cases. Note,
however, that many generative models can give rise to a logistic regression posterior, e.g., if eachclass-conditional density is Poisson, p(x|y=c)=P o i (x|λ
c). So the assumptions made by GDA
are much stronger than the assumptions made by logistic regression.
A further difference between these models is the way they are trained. When ﬁtting a discrim-
inative model, we usually maximize the conditional log likelihood∑N
i=1logp(yi|xi,θ), whereas
whenﬁttingagenerativemodel, weusuallymaximizethejointloglikelihood,∑Ni=1logp(yi,xi|θ).
It is clear that these can, in general, give different results (see Exercise 4.20).
When the Gaussian assumptions made by GDA are correct, the model will need less training

data than logistic regression to achieve a certain level of performance, but if the Gaussian

268 Chapter 8. Logistic regression
assumptions are incorrect, logistic regression will do better (Ng and Jordan 2002). This is
because discriminative models do not need to model the distribution of the features. This isillustrated in Figure 8.10. We see that the class conditional densities are rather complex; inparticular, p(x|y=1 )is a multimodal distribution, which might be hard to estimate. However,
the class posterior, p(y=c|x), is a simple sigmoidal function, centered on the threshold value
of 0.55. This suggests that, in general, discriminative methods will be more accurate, since their“job” is in some sense easier. However, accuracy is not the only important factor when choosinga method. Below we discuss some other advantages and disadvantages of each approach.
8.6.1 Pros and cons of each approach
•Easy to ﬁt? As we have seen, it is usually very easy to ﬁt generative classiﬁers. For example,

in Sections 3.5.1.1 and 4.2.4, we show that we can ﬁt a naive Bayes model and an LDA modelby simple counting and averaging. By contrast, logistic regression requires solving a convexoptimization problem (see Section 8.3.4 for the details), which is much slower.
•Fit classes separately? In a generative classiﬁer, we estimate the parameters of each class

conditional density independently, so we do not have to retrain the model when we addmore classes. In contrast, in discriminative models, all the parameters interact, so the wholemodel must be retrained if we add a new class. (This is also the case if we train a generativemodel to maximize a discriminative objective Salojarvi et al. (2005).)
•Handle missing features easily? Sometimes some of the inputs (components of x) are not

observed. In a generative classiﬁer, there is a simple method for dealing with this, as wediscuss in Section 8.6.2. However, in a discriminative classiﬁer, there is no principled solutionto this problem, since the model assumes that xis always available to be conditioned on
(although see (Marlin 2008) for some heuristic approaches).
•Can handle unlabeled training data? There is much interest in semi-supervised learning,

which uses unlabeled data to help solve a supervised task. This is fairly easy to do usinggenerative models (see e.g., (Lasserre et al. 2006; Liang et al. 2007)), but is much harder to dowith discriminative models.
•Symmetric in inputs and outputs? We can run a generative model “backwards”, and
infer probable inputs given the output by computing p(x|y). This is not possible with a
discriminative model. The reason is that a generative model deﬁnes a joint distribution on x

andy, and hence treats both inputs and outputs symmetrically.
•Can handle feature preprocessing? A big advantage of discriminative methods is that they
allow us to preprocess the input in arbitrary ways, e.g., we can replace xwithφ(x), which
could be some basis function expansion, as illustrated in Figure 8.9. It is often hard todeﬁne a generative model on such pre-processed data, since the new features are correlatedin complex ways.
•Well-calibrated probabilities? Some generative models, such as naive Bayes, make strong

independence assumptions which are often not valid. This can result in very extreme poste-rior class probabilities (very near 0 or 1). Discriminative models, such as logistic regression,are usually better calibrated in terms of their probability estimates.
We see that there are arguments for and against both kinds of models. It is therefore useful
to have both kinds in your “toolbox”. See Table 8.1 for a summary of the classiﬁcation and

8.6. Generative vs discriminative classiﬁers 269
−1 −0.5 0 0.5 1−1−0.500.51Linear Multinomial Logistic Regression
(a)−1 −0.5 0 0.5 1−1−0.500.51Kernel−RBF Multinomial Logistic Regression
(b)
Figure 8.9 (a) Multinomial logistic regression for 5 classes in the original feature space. (b) After basis
function expansion, using RBF kernels with a bandwidth of 1, and using all the data points as centers.
Figure generated by logregMultinomKernelDemo .
0 0.2 0.4 0.6 0.8 1012345
xclass conditional densitiesp(x|y=1)p(x|y=2)
(a)0 0.2 0.4 0.6 0.8 100.20.40.60.811.2
xp(y=1|x) p(y=2|x)
(b)
Figure 8.10 The class-conditional densities p(x|y=c)(left) may be more complex than the
class posteriors p(y=c|x)(right). Based on Figure 1.27 of (Bishop 2006a). Figure generated by
generativeVsDiscrim .
regression techniques we cover in this book.
8.6.2 Dealing with missing data
Sometimes some of the inputs (components of x) are not observed; this could be due to a

(b)
sensor failure, or a failure to complete an entry in a survey, etc. This is called the missing data
problem (Little. and Rubin 1987). The ability to handle missing data in a principled way is one
of the biggest advantages of generative models.
To formalize our assumptions, we can associate a binary response variable ri∈{0,1},
that speciﬁes whether each value xiis observed or not. The joint model has the form
p(xi,ri|θ,φ)=p(ri|xi,φ)p(xi|θ),w h e r eφare the parameters controlling whether the item

270 Chapter 8. Logistic regression
Model Classif/regr Gen/Discr Param/Non Section
Discriminant analysis Classif Gen Param Sec. 4.2.2, 4.2.4
Naive Bayes classiﬁer Classif Gen Param Sec. 3.5, 3.5.1.2
Tree-augmented Naive Bayes classiﬁer Classif Gen Param Sec. 10.2.1
Linear regression Regr Discrim Param Sec. 1.4.5, 7.3, 7.6,
Logistic regression Classif Discrim Param Sec. 1.4.6, 8.3.4, 8.4.3, 21.8.1.1
Sparse linear/ logistic regression Both Discrim Param Ch. 13
Mixture of experts Both Discrim Param Sec. 11.2.4
Multilayer perceptron (MLP)/ Neural network Both Discrim Param Ch. 16
Conditional random ﬁeld (CRF) Classif Discrim Param Sec. 19.6
Knearest neighbor classiﬁer Classif Gen Non Sec. 1.4.2, 14.7.3
(Inﬁnite) Mixture Discriminant analysis Classif Gen Non Sec. 14.7.3Classiﬁcation and regression trees (CART) Both Discrim Non Sec. 16.2
Boosted model Both Discrim Non Sec. 16.4
Sparse kernelized lin/logreg (SKLR) Both Discrim Non Sec. 14.3.2
Relevance vector machine (RVM) Both Discrim Non Sec. 14.3.2

Support vector machine (SVM) Both Discrim Non Sec. 14.5
Gaussian processes (GP) Both Discrim Non Ch. 15
Smoothing splines Regr Discrim Non Section 15.4.6
Table 8.1 List of various models for classiﬁcation and regression which we discuss in this book. Columns
are as follows: Model name; is the model suitable for classiﬁcation, regression, or both; is the model

generative or discriminative; is the model parametric or non-parametric; list of sections in book whichdiscuss the model. See also http://pmtk3 .googlecode .com/svn/trunk/docs/tutorial/html/tu
tSupervised .htmlfor the PMTK equivalents of these models. Any generative probabilistic model (e.g.,
HMMs, Boltzmann machines, Bayesian networks, etc.) can be turned into a classiﬁer by using it as a classconditional density.
is observed or not. If we assume p(ri|xi,φ)=p(ri|φ), we say the data is missing completely

at random orMCAR. If we assume p(ri|xi,φ)=p(ri|xo
i,φ),w h e r exoiis the observed part of
xi, we say the data is missing at random orMAR. If neither of these assumptions hold, we say
the data is not missing at random orNMAR. In this case, we have to model the missing data
mechanism, since the pattern of missingness is informative about the values of the missing data

and the corresponding parameters. This is the case in most collaborative ﬁltering problems, forexample. See e.g., (Marlin 2008) for further discussion. We will henceforth assume the data isMAR.
When dealing with missing data, it is helpful to distinguish the cases when there is missing-
ness only at test time (so the training data is complete data), from the harder case when there

is missingness also at training time. We will discuss these two cases below. Note that the classlabel is always missing at test time, by deﬁnition; if the class label is also sometimes missing attraining time, the problem is called semi-supervised learning.

8.6. Generative vs discriminative classiﬁers 271
8.6.2.1 Missing data at test time
In a generative classiﬁer, we can handle features that are MAR by marginalizing them out. For
example, if we are missing the value of x1, we can compute
p(y=c|x2:D,θ)∝p(y=c|θ)p(x2:D|y=c,θ) (8.94)
=p(y=c|θ)∑
x1p(x1,x2:D|y=c,θ) (8.95)
If we make the naive Bayes assumption, the marginalization can be performed as follows:
∑
x1p(x1,x2:D|y=c,θ)=[∑
x1p(x1|θ1c)]D∏
j=2p(xj|θjc)=D∏
j=2p(xj|θjc) (8.96)
where we exploited the fact that∑
x1p(x1|y=c,θ)=1. Hence in a naive Bayes classiﬁer, we
can simply ignore missing features at test time. Similarly, in discriminant analysis, no matterwhat regularization method was used to estimate the parameters, we can always analyticallymarginalize out the missing variables (see Section 4.3):
p(x
2:D|y=c,θ)=N(x2:D|μc,2:D,Σc,2:D,2:D) (8.97)
8.6.2.2 Missing data at training time

∑
p(x
Missing data at training time is harder to deal with. In particular, computing the MLE or MAPestimate is no longer a simple optimization problem, for reasons discussed in Section 11.3.2.However, soon we will study are a variety of more sophisticated algorithms (such as EM algo-rithm, in Section 11.4) for ﬁnding approximate ML or MAP estimates in such cases.
8.6.3 Fisher’s linear discriminant analysis (FLDA) *

p(x
Discriminant analysis is a generative approach to classiﬁcation, which requires ﬁtting an MVN tothe features. As we have discussed, this can be problematic in high dimensions. An alternativeapproach is to reduce the dimensionality of the features x∈R
Dand then ﬁt an MVN to the
resulting low-dimensional features z∈RL. The simplest approach is to use a linear projection
matrix,z=Wx,w h e r e Wis aL×Dmatrix. One approach to ﬁnding Wwould be to use

PCA (Section 12.2); the result would be very similar to RDA (Section 4.2.6), since SVD and PCAare essentially equivalent. However, PCA is an unsupervised technique that does not take classlabels into account. Thus the resulting low dimensional features are not necessarily optimalfor classiﬁcation, as illustrated in Figure 8.11. An alternative approach is to ﬁnd the matrixWsuch that the low-dimensional data can be classiﬁed as well as possible using a Gaussian

class-conditional density model. The assumption of Gaussianity is reasonable since we arecomputing linear combinations of (potentially non-Gaussian) features. This approach is calledFisher’s linear discriminant analysis,o rFLDA.
FLDA is an interesting hybrid of discriminative and generative techniques. The drawback of
this technique is that it is restricted to using L≤C−1dimensions, regardless of D, for reasons
that we will explain below. In the two-class case, this means we are seeking a single vector w

onto which we can project the data. Below we derive the optimal win the two-class case. We

272 Chapter 8. Logistic regression
−4 −2 0 2 4 6 801234means
fisher
pca
(a)
−45 −40 −35 −30 −25 −20 −15 −10 −5 005101520fisher
(b)
−8 −6 −4 −2 0 2 40510152025pca
(c)
Figure 8.11 Example of Fisher’s linear discriminant. (a) Two class data in 2D. Dashed green line = ﬁrst
principal basis vector. Dotted red line = Fisher’s linear discriminant vector. Solid black line joins the
class-conditional means. (b) Projection of points onto Fisher’s vector shows good class separation. (c)Projection of points onto PCA vector shows poor class separation. Figure generated by fisherLDAdemo .

8.6. Generative vs discriminative classiﬁers 273
then generalize to the multi-class case, and ﬁnally we give a probabilistic interpretation of this
technique.
8.6.3.1 Derivation of the optimal 1d projection
We now derive this optimal direction w, for the two-class case, following the presentation of
(Bishop 2006b, Sec 4.1.4). Deﬁne the class-conditional means as
μ1=1
N1∑
i:yi=1xi,μ2=1
N2∑
i:yi=2xi (8.98)
Letmk=wTμkbe the projection of each mean onto the line w. Also, let zi=wTxibe the
projection of the data onto the line. The variance of the projected points is proportional to
s2
k=∑
i:yi=k(zi−mk)2(8.99)
The goal is to ﬁnd wsuch that we maximize the distance between the means, m2−m1, while
also ensuring the projected clusters are “tight”:
J(w)=(m2−m1)2
s2
1+s22(8.100)
We can rewrite the right hand side of the above in terms of was follows
J(w)=wTSBw
wTSWw(8.101)
whereSBis the between-class scatter matrix given by
SB=(μ2−μ1)(μ2−μ1)T(8.102)
andSWis the within-class scatter matrix, given by
SW=∑

s2
k=∑
s2
SW=∑
i:yi=1(xi−μ1)(xi−μ1)T+∑
i:yi=2(xi−μ2)(xi−μ2)T(8.103)
wTSBw=wT(μ2−μ1)(μ2−μ1)Tw=(m2−m1)(m2−m1) (8.104)
and
wTSWw=∑
i:yi=1wT(xi−μ1)(xi−μ1)Tw+∑
i:yi=2wT(xi−μ2)(xi−μ2)Tw(8.105)
=∑
i:yi=1(zi−m1)2+∑
i:yi=2(zi−m2)2(8.106)
Equation 8.101 is a ratio of two scalars; we can take its derivative with respect to wand equate
to zero. One can show (Exercise 12.6) that that J(w)is maximized when
SBw=λSWw (8.107)

274 Chapter 8. Logistic regression
−4 −3 −2 −1 0 1 2 3 4−3−2−10123
(a)−4. 5 −4 −3.5 −3 −2. 5 −2 −1. 5 −1 −0.5 0−3.5−3−2.5−2−1.5−1−0.500.5
(b)
Figure 8.12 (a) PCA projection of vowel data to 2d. (b) FLDA projection of vowel data to 2d. We see there
is better class separation in the FLDA case. Based on Figure 4.11 of (Hastie et al. 2009). Figure generated by
fisherDiscrimVowelDemo , by Hannes Bretschneider.
where
λ=wTSBw
wTSWw(8.108)
Equation 8.107 is called a generalized eigenvalue problem. If SWis invertible, we can convert
it to a regular eigenvalue problem:
S−1
WSBw=λw (8.109)
However, in the two class case, there is a simpler solution. In particular, since
SBw=(μ2−μ1)(μ2−μ1)Tw=(μ2−μ1)(m2−m1) (8.110)
then, from Equation 8.109 we have
λw=S−1
W(μ2−μ1)(m2−m1) (8.111)
w∝S−1
W(μ2−μ1) (8.112)
Since we only care about the directionality, and not the scale factor, we can just set
w=S−1
W(μ2−μ1) (8.113)
This is the optimal solution in the two-class case. If SW∝I, meaning the pooled covariance

S−1
λw=S−1
w∝S−1
w=S−1
matrix is isotropic, then wis proportional to the vector that joins the class means. This is an
intuitively reasonable direction to project onto, as shown in Figure 8.11.
8.6.3.2 Extension to higher dimensions and multiple classes
We can extend the above idea to multiple classes, and to higher dimensional subspaces, by
ﬁnding a projection matrixWwhich maps from DtoLso as to maximize
J(W)=|WΣBWT|
|WΣWWT|(8.114)

8.6. Generative vs discriminative classiﬁers 275
where
ΣB≜∑
cNc
N(μc−μ)(μc−μ)T(8.115)
ΣW≜∑
cNc
NΣc (8.116)
Σc≜1
Nc∑
i:yi=c(xi−μc)(xi−μc)T(8.117)
The solution can be shown to be
W=Σ−1
2
WU (8.118)
whereUare theLleading eigenvectors of Σ−1
2
WΣBΣ−1
2
W, assuming ΣWis non-singular. (If it
is singular, we can ﬁrst perform PCA on all the data.)
Figure 8.12 gives an example of this method applied to some D=1 0dimensional speech
data, representing C=1 1different vowel sounds. We see that FLDA gives better class separation
than PCA.
Note that FLDA is restricted to ﬁnding at most a L≤C−1dimensional linear subspace,
no matter how large D, because the rank of the between class covariance matrix ΣBisC−1.
(The -1 term arises because of the μterm, which is a linear function of the μc.) This is a rather
severe restriction which limits the usefulness of FLDA.
8.6.3.3 Probabilistic interpretation of FLDA *
To ﬁnd a valid probabilistic interpretation of FLDA, we follow the approach of (Kumar and Andreo

than PCA.
1998; Zhou et al. 2009). They proposed a model known as heteroscedastic LDA (HLDA), which
works as follows. Let Wbe aD×Dinvertible matrix, and let zi=Wxibe a transformed
version of the data. We now ﬁt full covariance Gaussians to the transformed data, one per class,but with the constraint that only the ﬁrst Lcomponents will be class-speciﬁc; the remaining
H=D−Lcomponents will be shared across classes, and will thus not be discriminative. That
is, we use
p(z
i|θ,yi=c)=N (zi|μc,Σc) (8.119)
μc≜(mc;m0) (8.120)
Σc≜(Sc0
0S0)
(8.121)

is, we use
p(z
Σc≜(Sc0
0S0)
(8.121)
wherem0is the shared Hdimensional mean and S0is the shared H×Hcovariace. The pdf
of the original (untransformed) data is given by
p(xi|yi=c,W,θ)=|W |N(Wxi|μc,Σc) (8.122)
=|W|N(WLxi|mc,Sc)N(WHxi|m0,S0) (8.123)
whereW=(WL
WH)
. For ﬁxed W, it is easy to derive the MLE for θ. One can then optimize
Wusing gradient methods.

276 Chapter 8. Logistic regression
In the special case that the Σcare diagonal, there is a closed-form solution for W(Gales
1999). And in the special case the Σcare all equal, we recover classical LDA (Zhou et al. 2009).
In view of this this result, it should be clear that HLDA will outperform LDA if the class
covariances are not equal within the discriminative subspace (i.e., if the assumption that Σcis
independent of cis a poor assumption). This is easy to demonstrate on synthetic data, and is
also the case on more challenging tasks such as speech recognition (Kumar and Andreo 1998).
Furthermore, we can extend the model by allowing each class to use its own projection matrix;this is known as multiple LDA (Gales 2002).
Exercises
Exercise 8.1 Spam classiﬁcation using logistic regression
Consider the email spam data set discussed on p300 of (Hastie et al. 2009). This consists of 4601 email
messages, from which 57 features have been extracted. These are as follows:

Exercises
• 48 features, in [0,100], giving the percentage of words in a given message which match a given word
on the list. The list contains words such as “business”, “free”, “george”, etc. (The data was collected by
George Forman, so his name occurs quite a lot.)
• 6 features, in [0,100], giving the percentage of characters in the email that match a given character on
the list. The characters are ;([!$#
• Feature 55: The average length of an uninterrupted sequence of capital letters (max is 40.3, mean is 4.9)

• Feature 56: The length of the longest uninterrupted sequence of capital letters (max is 45.0, mean is
52.6)
• Feature 57: The sum of the lengts of uninterrupted sequence of capital letters (max is 25.6, mean is
282.2)
Load the data from spamData.mat , which contains a training set (of size 3065) and a test set (of size
1536).One can imagine performing several kinds of preprocessing to this data. Try each of the following
separately:
a. Standardize the columns so they all have mean 0 and unit variance.

52.6)
282.2)
b. Transform the features using log(x
ij+0.1).
c. Binarize the features using I(xij>0).
For each version of the data, ﬁt a logistic regression model. Use cross validation to choose the strengthof theℓ
2regularizer. Report the mean error rate on the training and test sets. You should get numbers
similar to this:
method train test
stnd 0.082 0.079
log 0.052 0.059
binary 0.065 0.072
(The precise values will depend on what regularization value you choose.) Turn in your code and numericalresults.

ij+0.1).
(See also Exercise 8.2.
Exercise 8.2 Spam classiﬁcation using naive Bayes
We will re-examine the dataset from Exercise 8.1.

8.6. Generative vs discriminative classiﬁers 277
a. Use naiveBayesFit andnaiveBayesPredict on the binarized spam data. What is the training and
test error? (You can try different settings of the pseudocount αif you like (this corresponds to the
Beta(α,α) prior each θjc), although the default of α=1is probably ﬁne.) Turn in your error rates.
b. Modify the code so it can handle real-valued features. Use a Gaussian density for each feature; ﬁt it
with maximum likelihood. What are the training and test error rates on the standardized data and the
log transformed data? Turn in your 4 error rates and code.
Exercise 8.3 Gradient and Hessian of log-likelihood for logistic regression
a. Letσ(a)=1
1+e−abe the sigmoid function. Show that
dσ(a)
da=σ(a)(1−σ(a)) (8.124)
b. Using the previous result and the chain rule of calculus, derive an expression for the gradient of the
log likelihood (Equation 8.5).
c. The Hessian can be written as H=XTSX,w h e r e S≜diag(μ 1(1−μ1),...,μ n(1−μn)). Show

dσ(a)
thatHis positive deﬁnite. (You may assume that 0<μi<1, so the elements of Swill be strictly
positive, and that Xis full rank.)
Exercise 8.4 Gradient and Hessian of log-likelihood for multinomial logistic regression
a. Letμik=S(ηi)k. Prove that the Jacobian of the softmax is
∂μik
∂ηij=μik(δkj−μij) (8.125)
whereδkj=I(k=j).
b. Hence show that
∇wcℓ=∑
i(yic−μic)xi (8.126)
Hint: use the chain rule and the fact that∑
cyic=1.
c. Show that the block submatrix of the Hessian for classes candc′is given by
Hc,c′=−∑
i (8.127)

∂μik
∇wcℓ=∑
cyic=1.
Hc,c′=−∑
i (8.127)
Exercise 8.5 Symmetric version of ℓ2regularized multinomial logistic regression
(Source: Ex 18.3 of (Hastie et al. 2009).)
Multiclass logistic regression has the form
p(y=c|x,W)=exp(wc0+wT
cx)∑C
k=1exp(w k0+wT
kx)(8.128)
whereWis a(D+1)×Cweight matrix. We can arbitrarily deﬁne wc=0for one of the classes, say
c=C, sincep(y=C|x,W)=1−∑C−1
c=1p(y=c|x,w). In this case, the model has the form
cx)
1+∑C−1
k=1exp(wk0+wT
kx)(8.129)

278 Chapter 8. Logistic regression
If we don’t “clamp” one of the vectors to some constant value, the parameters will be unidentiﬁable.
However, suppose we don’t clamp wc=0, so we are using Equation 8.128, but we add ℓ2regularization
by optimizing
N∑
i=1logp(yi|xi,W)−λC∑
c=1||wc||2
2 (8.130)
Show that at the optimum we have∑C
c=1ˆwcj=0forj=1:D. (For the unregularized ˆwc0terms, we
still need to enforce that w0C=0to ensure identiﬁability of the offset.)
Exercise 8.6 Elementary properties of ℓ2regularized logistic regression
(Source: Jaaakkola.). Consider minimizing
J(w)=−ℓ(w,Dtrain)+λ||w||2
2 (8.131)
where
ℓ(w,D)=1
|D|∑
i∈Dlogσ(yixTiw) (8.132)
is the average log-likelihood on data set D,f o ryi∈{ −1,+1}. Answer the following true/ false questions.
a.J(w)has multiple locally optimal solutions: T/F?
b. Letˆw= argmin wJ(w)be a global optimum. ˆwis sparse (has many zero entries): T/F?
c. If the training data is linearly separable, then some weights wjmight become inﬁnite if λ=0: T/F?

2 (8.131)
where
ℓ(w,D)=1
|D|∑
d.ℓ(ˆw,Dtrain)always increases as we increase λ: T/F?
e.ℓ(ˆw,Dtest)always increases as we increase λ: T/F?
Exercise 8.7 Regularizing separate terms in 2d logistic regression
(Source: Jaaakkola.)
a. Consider the data in Figure 8.13, where we ﬁt the model p(y=1|x,w)=σ(w0+w1x1+w2x2).
Suppose we ﬁt the model by maximum likelihood, i.e., we minimize
J(w)=−ℓ(w,Dtrain) (8.133)
whereℓ(w,Dtrain)is the log likelihood on the training set. Sketch a possible decision boundary

corresponding to ˆw. (Copy the ﬁgure ﬁrst (a rough sketch is enough), and then superimpose your
answer on your copy, since you will need multiple versions of this ﬁgure). Is your answer (decision
boundary) unique? How many classiﬁcation errors does your method make on the training set?
b. Now suppose we regularize only the w0parameter, i.e., we minimize
J0(w)=−ℓ(w,Dtrain)+λw2
0 (8.134)
Supposeλis a very large number, so we regularize w0all the way to 0, but all other parameters are

0 (8.134)
unregularized. Sketch a possible decision boundary. How many classiﬁcation errors does your methodmake on the training set? Hint: consider the behavior of simple linear regression, w
0+w1x1+w2x2
whenx1=x2=0.
c. Now suppose we heavily regularize only the w1parameter, i.e., we minimize
J1(w)=−ℓ(w,Dtrain)+λw2
1 (8.135)
Sketch a possible decision boundary. How many classiﬁcation errors does your method make on thetraining set?

8.6. Generative vs discriminative classiﬁers 279
Figure 8.13 Data for logistic regression question.
d. Now suppose we heavily regularize only the w2parameter. Sketch a possible decision boundary. How
many classiﬁcation errors does your method make on the training set?

9Generalized linear models and the
exponential family
9.1 Introduction
We have now encountered a wide variety of probability distributions: the Gaussian, the Bernoulli,
the Student t, the uniform, the gamma, etc. It turns out that most of these are members of a
broader class of distributions known as the exponential family.1In this chapter, we discuss
various properties of this family. This allows us to derive theorems and algorithms with verybroad applicability.
We will see how we can easily use any member of the exponential family as a class-conditional
density in order to make a generative classiﬁer. In addition, we will discuss how to builddiscriminative models, where the response variable has an exponential family distribution, whosemean is a linear function of the inputs; this is known as a generalized linear model, andgeneralizes the idea of logistic regression to other kinds of response variables.
9.2 The exponential family

Before deﬁning the exponential family, we mention several reasons why it is important:
• It can be shown that, under certain regularity conditions, the exponential family is the only
family of distributions with ﬁnite-sized sufficient statistics, meaning that we can compress
the data into a ﬁxed-sized summary without loss of information. This is particularly usefulfor online learning, as we will see later.
• The exponential family is the only family of distributions for which conjugate priors exist,

which simpliﬁes the computation of the posterior (see Section 9.2.5).
• The exponential family can be shown to be the family of distributions that makes the least
set of assumptions subject to some user-chosen constraints (see Section 9.2.6).
• The exponential family is at the core of generalized linear models, as discussed in Section 9.3.
• The exponential family is at the core of variational inference, as discussed in Section 21.2.

1. The exceptions are the Student t, which does not have the right form, and the uniform distribution, which does not
have ﬁxed support independent of the parameter values.

282 Chapter 9. Generalized linear models and the exponential family
9.2.1 Deﬁnition
A pdf or pmf p(x|θ),f o rx=(x1,...,x m)∈Xmandθ∈Θ⊆Rd, is said to be in the
exponential family if it is of the form
p(x|θ)=1
Z(θ)h(x)exp[θTφ(x)] (9.1)
=h(x)exp[θTφ(x)−A(θ)] (9.2)
where
Z(θ)=∫
Xmh(x)exp[θTφ(x)]dx (9.3)
A(θ)=l o g Z(θ) (9.4)
Hereθare called the natural parameters orcanonical parameters, φ(x)∈Rdis called a
vector of sufficient statistics, Z(θ)is called the partition function, A(θ)is called the log
partition function orcumulant function, and h(x)is the a scaling constant, often 1. If
φ(x)=x, we say it is a natural exponential family.
Equation 9.2 can be generalized by writing
p(x|θ)=h(x)exp[η(θ)Tφ(x)−A(η(θ))] (9.5)
whereηis a function that maps the parameters θto the canonical parameters η=η(θ).I f
dim(θ)<dim(η(θ)), it is called a curved exponential family , which means we have more
sufficient statistics than parameters. If η(θ)=θ, the model is said to be in canonical form.

We will assume models are in canonical form unless we state otherwise.
9.2.2 Examples
Let us consider some examples to make things clearer.
9.2.2.1 Bernoulli
The Bernoulli for x∈{0,1}can be written in exponential family form as follows:
Ber(x|μ)=μx(1−μ)1−x=e x p [xlog(μ)+(1− x)log(1−μ) ]=e x p [ φ(x)Tθ](9.6)
whereφ(x)=[ I(x=0 ),I(x=1 ) ]andθ= [log(μ),log(1−μ)]. However, this representation
isover-complete since there is a linear dependendence between the features:
1Tφ(x)=I(x=0 )+ I(x=1 )=1 (9.7)

Consequently θis not uniquely identiﬁable. It is common to require that the representation be
minimal, which means there is a unique θassociated with the distribution. In this case, we
can just deﬁne
Ber(x|μ)=( 1−μ)exp[
xlog(μ
1−μ)]
(9.8)

9.2. The exponential family 283
Now we have φ(x)=x,θ=l o g(
μ
1−μ)
, which is the log-odds ratio, and Z=1/(1−μ).W e
can recover the mean parameter μfrom the canonical parameter using
μ=s i g m (θ)=1
1+e−θ(9.9)
9.2.2.2 Multinoulli
We can represent the multinoulli as a minimal exponential family as follows (where xk=I(x=
k)):
Cat(x|μ)=K∏
k=1μxk
k=e x p[K∑
k=1xklogμk]
(9.10)
=e x p[K−1∑
k=1xklogμk+(
1−K−1∑
k=1xk)
log(1−K−1∑
k=1μk)]
(9.11)
=e x p[K−1∑
k=1xklog(
μk
1−∑K−1
j=1μj)
+log(1−K−1∑
k=1μk)]
(9.12)
=e x p[K−1∑
k=1xklog(μk
μK)
+logμK]
(9.13)
whereμK=1−∑K−1
k=1μk. We can write this in exponential family form as follows:
Cat(x|θ)=e x p ( θTφ(x)−A(θ)) (9.14)
θ= [logμ1
μK,...,logμK−1
μK] (9.15)
φ(x)=[ I(x=1 ),...,I(x=K−1)] (9.16)
We can recover the mean parameters from the canonical parameters using
μk=eθk
1+∑K−1
j=1eθj(9.17)
From this, we ﬁnd
μK=1−∑K−1
j=1eθj
1+∑K−1
j=1eθj=1∑K−1
j=1eθj(9.18)
and hence
A(θ)=l o g(
1+K−1∑
k=1eθk)
(9.19)
If we deﬁne θK=0, we can write μ=S(θ)andA(θ)=l o g∑K
k=1eθk,w h e r eSis the

μK)
+logμK]
(9.13)
θ= [logμ1
μK] (9.15)
μk=eθk
1+∑K−1
μK=1−∑K−1
j=1eθj
1+∑K−1
and hence
A(θ)=l o g(
1+K−1∑
k=1eθk)
(9.19)
softmax function in Equation 4.39.

284 Chapter 9. Generalized linear models and the exponential family
9.2.2.3 Univariate Gaussian
The univariate Gaussian can be written in exponential family form as follows:
N(x|μ,σ2)=1
(2πσ2)1
2exp[−1
2σ2(x−μ)2] (9.20)
=1
(2πσ2)1
2exp[−1
2σ2x2+μ
σ2x−1
2σ2μ2] (9.21)
=1
Z(θ)exp(θTφ(x)) (9.22)
where
θ=(μ/σ2
−1
2σ2)
(9.23)
φ(x)=(x
x2)
(9.24)
Z(μ,σ2)=√
2πσexp[μ2
2σ2] (9.25)
A(θ)=−θ2
1
4θ2−1
2log(−2θ 2)−1
2log(2π) (9.26)
9.2.2.4 Non-examples
Not all distributions of interest belong to the exponential family. For example, the uniform
distribution, X∼Unif(a,b ), does not, since the support of the distribution depends on the
parameters. Also, the Student T distribution (Section 11.4.5) does not belong, since it does nothave the required form.
9.2.3 Log partition function
An important property of the exponential family is that derivatives of the log partition functioncan be used to generate cumulants of the sufficient statistics.
2For this reason, A(θ)is

sometimes called a cumulant function. We will prove this for a 1-parameter distribution;
this can be generalized to a K-parameter distribution in a straightforward way. For the ﬁrst
2. The ﬁrst and second cumulants of a distribution are its mean E[X]and variance var[X], whereas the ﬁrst and
second moments are its mean E[X]andE[
X2]
.

9.2. The exponential family 285
derivative we have
dA
dθ=d
dθ(
log∫
exp(θφ(x))h(x)dx)
(9.27)
=d
dθ∫
exp(θφ(x))h(x)dx∫
exp(θφ(x))h(x)dx(9.28)
=∫
φ(x)exp(θφ(x))h(x)dx
exp(A(θ))(9.29)
=∫
φ(x)exp(θφ(x)−A(θ))h(x)dx (9.30)
=∫
φ(x)p(x)dx=E[φ(x)] (9.31)
For the second derivative we have
d2A
dθ2=∫
φ(x)exp(θφ(x)−A(θ))h(x)(φ(x)−A′(θ))dx (9.32)
=∫
φ(x)p(x)(φ(x)−A′(θ))dx (9.33)
=∫
φ2(x)p(x)dx−A′(θ)∫
φ(x)p(x)dx (9.34)
=E[
φ2(X)]
−E[φ(x)]2= var[φ(x)] (9.35)
where we used the fact that A′(θ)=dA
dθ=E[φ(x)].
In the multivariate case, we have that
∂2A
∂θi∂θj=E[φi(x)φj(x)]−E[φi(x)]E[φj(x)] (9.36)
and hence
∇2A(θ)=c o v[φ(x)] (9.37)
Since the covariance is positive deﬁnite, we see that A(θ)is a convex function (see Section 7.3.3).
9.2.3.1 Example: the Bernoulli distribution
For example, consider the Bernoulli distribution. We have A(θ)=l o g ( 1+ eθ), so the mean is
given by
dA
dθ=eθ
1+eθ=1
1+e−θ=s i g m (θ)=μ (9.38)
The variance is given by
d2A
dθ2=d
dθ(1+e−θ)−1=( 1+e−θ)−2.e−θ(9.39)
=e−θ
1+e−θ1
1+e−θ=1
eθ+11
1+e−θ=( 1−μ)μ (9.40)

286 Chapter 9. Generalized linear models and the exponential family
9.2.4 MLE for the exponential family
The likelihood of an exponential family model has the form
p(D|θ)=[N∏
i=1h(xi)]
g(θ)Nexp(
η(θ)T[N∑
i=1φ(xi)])
(9.41)
We see that the sufficient statistics are Nand
φ(D)=[N∑
i=1φ1(xi),...,N∑
i=1φK(xi)] (9.42)
For example, for the Bernoulli model we have φ=[∑
iI(xi=1 ) ], and for the univariate
Gaussian, we have φ=[∑
ixi,∑
ix2
i]. (We also need to know the sample size, N.)
ThePitman-Koopman-Darmois theorem states that, under certain regularity conditions, the
exponential family is the only family of distributions with ﬁnite sufficient statistics. (Here, ﬁnite
means of a size independent of the size of the data set.)
One of the conditions required in this theorem is that the support of the distribution not be
dependent on the parameter. For a simple example of such a distribution, consider the uniformdistribution
p(x|θ)=U(x|θ)=1
θI(0≤x≤θ) (9.43)
p(D|θ)=θ−NI(0≤max{x i}≤θ) (9.44)

So the sufficient statistics are Nands(D)=m a x ixi. This is ﬁnite in size, but the uni-
form distribution is not in the exponential family because its support set, X, depends on the
parameters.
We now descibe how to compute the MLE for a canonical exponential family model. Given
Niid data points D=(x1,...,x N), the log-likelihood is
logp(D|θ)=θTφ(D)−NA(θ) (9.45)
Since−A(θ)is concave in θ, andθTφ(D)is linear in θ, we see that the log likelihood is

concave, and hence has a unique global maximum. To derive this maximum, we use the factthat the derivative of the log partition function yields the expected value of the sufficient statisticvector (Section 9.2.3):
∇
θlogp(D|θ)=φ(D)−NE[φ(X)] (9.46)
Setting this gradient to zero, we see that at the MLE, the empirical average of the sufficient
statistics must equal the model’s theoretical expected sufficient statistics, i.e., ˆθmust satisfy
E[φ(X)] =1
NN∑
i=1φ(xi) (9.47)

9.2. The exponential family 287
This is called moment matching. For example, in the Bernoulli distribution, we have φ(X)=
I(X=1 ), so the MLE satisﬁes
E[φ(X)] =p(X=1 )=ˆμ=1
NN∑
i=1I(xi=1 ) (9.48)
9.2.5 Bayes for the exponential family *
We have seen that exact Bayesian analysis is considerably simpliﬁed if the prior is conjugate to
the likelihood. Informally this means that the prior p(θ|τ)has the same form as the likelihood
p(D|θ). For this to make sense, we require that the likelihood have ﬁnite sufficient statistics, so
that we can write p(D|θ)=p(s(D)|θ). This suggests that the only family of distributions for
which conjugate priors exist is the exponential family. We will derive the form of the prior andposterior below.
9.2.5.1 Likelihood
The likelihood of the exponential family is given by
p(D|θ)∝g(θ)
Nexp(
η(θ)TsN)
(9.49)
wheresN=∑N
i=1s(xi). In terms of the canonical parameters this becomes
p(D|η)∝exp(NηTs−NA(η)) (9.50)
wheres=1
NsN.
9.2.5.2 Prior
The natural conjugate prior has the form

Nexp(
η(θ)TsN)
(9.49)
wheres=1
NsN.
p(θ|ν0,τ0)∝g(θ)ν0exp(
η(θ)Tτ0)
(9.51)
Let us write τ0=ν0τ0, to separate out the size of the prior pseudo-data, ν0, from the mean of
the sufficient statistics on this pseudo-data, τ0. In canonical form, the prior becomes
p(η|ν0,τ0)∝exp(ν0ηTτ0−ν0A(η)) (9.52)
9.2.5.3 Posterior
The posterior is given by
p(θ|D)=p(θ|νN,τN)=p(θ|ν0+N,τ0+sN) (9.53)
So we see that we just update the hyper-parameters by adding. In canonical form, this becomes
p(η|D)∝exp(
ηT(ν0τ0+Ns)−(ν0+N)A(η)))
(9.54)
=p(η|ν0+N,ν0τ0+Ns
ν0+N) (9.55)

η(θ)Tτ0)
(9.51)
(9.54)
So we see that the posterior hyper-parameters are a convex combination of the prior mean
hyper-parameters and the average of the sufficient statistics.

288 Chapter 9. Generalized linear models and the exponential family
9.2.5.4 Posterior predictive density
Let us derive a generic expression for the predictive density for future observables D′=
(˜x1,...,˜xN′)given past data D=(x1,...,xN)as follows. For notational brevity, we
will combine the sufficient statistics with the size of the data, as follows: ˜τ0=(ν0,τ0),
˜s(D)=(N,s(D)), and˜s(D′)=(N′,s(D′)). So the prior becomes
p(θ|˜τ0)=1
Z(˜τ0)g(θ)ν0exp(η(θ)Tτ0) (9.56)
The likelihood and posterior have a similar form. Hence
p(D′|D)=∫
p(D′|θ)p(θ|D)dθ (9.57)
=⎡
⎣N′∏
i=1h(˜xi)⎤⎦Z(˜τ
0+˜s(D))−1∫
g(θ)ν0+N+N′dθ (9.58)
×exp⎛⎝∑
kηk(θ)(τk+N∑
i=1sk(xi)+N′∑
i=1sk(˜xi)⎞⎠dθ (9.59)
=⎡⎣
N′∏
0+˜s(D)+˜s(D′))
Z(˜τ0+˜s(D))(9.60)
IfN=0, this becomes the marginal likelihood of D′, which reduces to the familiar form of
normalizer of the posterior divided by the normalizer of the prior, multiplied by a constant.
9.2.5.5 Example: Bernoulli distribution

p(D′|D)=∫
=⎡
⎣N′∏
×exp⎛⎝∑
=⎡⎣
N′∏
As a simple example, let us revisit the Beta-Bernoulli model in our new notation.
p(D|θ)=( 1−θ)Nexp(
log(θ
1−θ)∑
ixi)
(9.61)
Hence the conjugate prior is given by
p(θ|ν0,τ0)∝(1−θ)ν0exp(
log(θ
1−θ)τ0)
(9.62)
=θτ0(1−θ)ν0−τ0(9.63)
If we deﬁne α=τ0+1andβ=ν0−τ0+1, we see that this is a beta distribution.
We can derive the posterior as follows, where s=∑
iI(xi=1 )is the sufficient statistic:
p(θ|D)∝θτ0+s(1−θ)ν0−τ0+n−s(9.64)
=θτn(1−θ)νn−τn(9.65)

log(θ
1−θ)∑
ixi)
(9.61)
log(θ
1−θ)τ0)
(9.62)
We can derive the posterior predictive distribution as follows. Assume p(θ)=B e t a ( θ|α,β),
and lets=s(D)be the number of heads in the past data. We can predict the probability of a

9.2. The exponential family 289
given sequence of future heads, D′=(˜x1,...,˜xm), with sufficient statistic s′=∑m
i=1I(˜xi=
1), as follows:
p(D′|D)=∫1
0p(D′|θ|Beta(θ|αn,βn)dθ (9.66)
=Γ(αn+βn)
Γ(αn)Γ(βn)∫1
0θαn+t′−1(1−θ)βn+m−t′−1dθ (9.67)
=Γ(αn+βn)
Γ(αn)Γ(βn)Γ(αn+m)Γ(βn+m)
Γ(αn+m+βn+m)(9.68)
where
αn+m=αn+s′=α+s+s′(9.69)
βn+m=βn+(m−s′)=β+(n−s)+(m−s′) (9.70)
9.2.6 Maximum entropy derivation of the exponential family *
Although the exponential family is convenient, is there any deeper justiﬁcation for its use? It
turns out that there is: it is the distribution that makes the least number of assumptions aboutthe data, subject to a speciﬁc set of user-speciﬁed constraints, as we explain below. In particular,suppose all we know is the expected values of certain features or functions:
∑
xfk(x)p(x)=Fk (9.71)
whereFkare known constants, and fk(x)is an arbitrary function. The principle of maximum
entropyormaxentsays we should pick the distribution with maximum entropy (closest to

∑
uniform), subject to the constraints that the moments of the distribution match the empiricalmoments of the speciﬁed functions.
To maximize entropy subject to the constraints in Equation 9.71, and the constraints that
p(x)≥0and∑
xp(x)=1, we need to use Lagrange multipliers. The Lagrangian is given by
J(p,λ)=−∑
xp(x)logp(x)+λ0(1−∑
xp(x))+∑
kλk(Fk−∑
xp(x)fk(x))(9.72)
We can use the calculus of variations to take derivatives wrt the function p, but we will adopt

J(p,λ)=−∑
xp(x))+∑
kλk(Fk−∑
a simpler approach and treat pas a ﬁxed length vector (since we are assuming xis discrete).
∂J
∂p(x)=−1−logp(x)−λ0−∑
kλkfk(x) (9.73)
Setting∂J
∂p(x)=0yields
p(x)=1
Zexp(−∑
kλkfk(x)) (9.74)

290 Chapter 9. Generalized linear models and the exponential family
xiw
ηi μi θig−1
gΨ
Ψ−1
Figure 9.1 A visualization of the various features of a GLM. Based on Figure 8.3 of (Jordan 2007).
whereZ=e1+λ0. Using the sum to one constraint, we have
1=∑
xp(x)=1
Z∑
xexp(−∑
kλkfk(x)) (9.75)
Hence the normalization constant is given by
Z=∑
xexp(−∑
kλkfk(x)) (9.76)
Thus the maxent distribution p(x)has the form of the exponential family (Section 9.2), also
known as the Gibbs distribution.
9.3 Generalized linear models (GLMs)
Linear and logistic regression are examples of generalized linear models, or GLMs (McCullagh
and Nelder 1989). These are models in which the output density is in the exponential family

1=∑
xp(x)=1
Z∑
xexp(−∑
Z=∑
xexp(−∑
(Section 9.2), and in which the mean parameters are a linear combination of the inputs, passedthrough a possibly nonlinear function, such as the logistic function. We describe GLMs in moredetail below. We focus on scalar outputs for notational simplicity. (This excludes multinomiallogistic regression, but this is just to simplify the presentation.)
9.3.1 Basics
To understand GLMs, let us ﬁrst consider the case of an unconditional dstribution for a scalarresponse variable:
p(y
i|θ,σ2)=e x p[yiθ−A(θ)
σ2+c(yi,σ2)]
(9.77)

p(y
(9.77)
whereσ2is thedispersion parameter (often set to 1), θis the natural parameter, Ais the
partition function, and cis a normalization constant. For example, in the case of logistic
regression, θis the log-odds ratio, θ= log(μ
1−μ),w h e r eμ=E[y]=p(y=1 )is the mean
parameter (see Section 9.2.2.1). To convert from the mean parameter to the natural parameter

9.3. Generalized linear models (GLMs) 291
Distrib. Link g(μ)θ=ψ(μ)μ=ψ−1(θ)=E[y]
N(μ,σ2)identity θ=μμ =θ
Bin(N,μ )logit θ= log(μ
1−μ)μ=s i g m (θ)
Poi(μ)log θ= log(μ)μ=eθ
Table 9.1 Canonical link functions ψand their inverses for some common GLMs.
we can use a function ψ,s oθ=Ψ (μ). This function is uniquely determined by the form of the
exponential family distribution. In fact, this is an invertible mapping, so we have μ=Ψ−1(θ).
Furthermore, we know from Section 9.2.3 that the mean is given by the derivative of the partition
function, so we have μ=Ψ−1(θ)=A′(θ).
Now let us add inputs/ covariates. We ﬁrst deﬁne a linear function of the inputs:
ηi=wTxi (9.78)
We now make the mean of the distribution be some invertible monotonic function of this linearcombination. By convention, this function, known as the mean function, is denoted by g
−1,s o
μi=g−1(ηi)=g−1(wTxi) (9.79)
See Figure 9.1 for a summary of the basic model.
The inverse of the mean function, namely g(), is called the link function.W ea r ef r e et o

−1,s o
choose almost any function we like for g, so long as it is invertible, and so long as g−1has the
appropriate range. For example, in logistic regression, we set μi=g−1(ηi) = sigm(η i).
One particularly simple form of link function is to use g=ψ; this is called the canonical
link function. In this case, θi=ηi=wTxi, so the model becomes
p(yi|xi,w,σ2)=e x p[yiwTxi−A(wTxi)
(9.80)

(9.80)
In Table 9.1, we list some distributions and their canonical link functions. We see that for theBernoulli/ binomial distribution, the canonical link is the logit function, g(μ)=l o g (η/(1−η)),
whose inverse is the logistic function, μ=s i g m (η).
Based on the results in Section 9.2.3, we can show that the mean and variance of the response
variable are as follows:
E[
y|x
i,w,σ2]
=μi=A′(θi) (9.81)
var[
y|xi,w,σ2]
=σ2
i=A′′(θi)σ2(9.82)
To make the notation clearer, let us consider some simple examples.
• For linear regression, we have

E[
y|x
i,w,σ2]
var[
=σ2
logp(yi|xi,w,σ2)=yiμi−μ2
i
2
σ2−1
2(y2
i
σ2+log(2πσ2))
(9.83)
whereyi∈R, andθi=μi=wTxiHereA(θ)=θ2/2,s oE[yi]=μiandvar[yi]=σ2.

292 Chapter 9. Generalized linear models and the exponential family
• For binomial regression, we have
logp(yi|xi,w)=y ilog(πi
1−πi)+Nilog(1−πi)+log(
Ni
yi)
(9.84)
whereyi∈{0,1,...,N i},πi=s i g m (wTxi),θi= log(π i/(1−πi)) =wTxi, andσ2=1.
HereA(θ)=Nilog(1+eθ), soE[yi]=Niπi=μi,var[yi]=Niπi(1−πi).
•F o rpoisson regression,w eh a v e
logp(yi|xi,w)=y ilogμi−μi−log(yi!) (9.85)
whereyi∈{0,1,2,...},μi=e x p (wTxi),θi= log(μi)=wTxi, andσ2=1.H e r e
A(θ)=eθ,s oE[yi]=v a r[yi]=μi. Poisson regression is widely used in bio-statistical
applications, where yimight represent the number of diseases of a given person or place,
or the number of reads at a genomic location in a high-throughput sequencing context (see
e.g., (Kuan et al. 2009)).
9.3.2 ML and MAP estimation
One of the appealing properties of GLMs is that they can be ﬁt using exactly the same methodsthat we used to ﬁt logistic regression. In particular, the log-likelihood has the following form:
ℓ(w)=l o gp(D|w)=1
σ2N∑
i=1ℓi (9.86)
ℓi≜θiyi−A(θi) (9.87)

σ2N∑
We can compute the gradient vector using the chain rule as follows:
dℓi
dwj=dℓi
dθidθi
dμidμi
dηidηi
dwj(9.88)
=(yi−A′(θi))dθi
dμidμi
dηixij (9.89)
=(yi−μi)dθi
dμidμi
dηixij (9.90)
If we use a canonical link, θi=ηi, this simpliﬁes to
∇wℓ(w)=1
σ2[N∑
i=1(yi−μi)xi]
(9.91)

σ2N∑
dℓi
dwj=dℓi
dθidθi
dμidμi
dηidηi
dwj(9.88)
dμidμi
dμidμi
∇wℓ(w)=1
σ2[N∑
(9.91)
which is a sum of the input vectors, weighted by the errors. This can be used inside a (stochastic)gradient descent procedure, discussed in Section 8.5.2. However, for improved efficiency, weshould use a second-order method. If we use a canonical link, the Hessian is given by
H=−1
σ2N∑
i=1dμi
dθixixT
i=−1
σ2XTSX (9.92)

9.4. Probit regression 293
Name Formula
Logistic g−1(η)=s i g m ( η)=eη
1+eη
Probit g−1(η)=Φ (η)
Log-log g−1(η)=e x p (−exp(−η))
Complementary log-log g−1(η)=1−exp(−exp(η))
Table 9.2 Summary of some possible mean functions for binary regression.
whereS=d i a g (dμ1
dθ1,...,dμN
dθN)is a diagonal weighting matrix. This can be used inside the
IRLS algorithm (Section 8.3.4). Speciﬁcally, we have the following Newton update:
wt+1=(XTStX)−1XTStzt (9.93)
zt=θt+S−1
t(y−μt) (9.94)
whereθt=Xwtandμt=g−1(ηt).
If we extend the derivation to handle non-canonical links, we ﬁnd that the Hessian has another
term. However, it turns out that the expected Hessian is the same as in Equation 9.92; using
the expected Hessian (known as the Fisher information matrix) instead of the actual Hessian isknown as the Fisher scoring method.
It is straightforward to modify the above procedure to perform MAP estimation with a Gaus-
sian prior: we just modify the objective, gradient and Hessian, just as we added ℓ
2regularization

to logistic regression in Section 8.3.6.
9.3.3 Bayesian inference
Bayesian inference for GLMs is usually conducted using MCMC (Chapter 24). Possible methodsinclude Metropolis Hastings with an IRLS-based proposal (Gamerman 1997), Gibbs samplingusing adaptive rejection sampling (ARS) for each full-conditional (Dellaportas and Smith 1993),etc. See e.g., (Dey et al. 2000) for futher information. It is also possible to use the Gaussianapproximation (Section 8.4.1) or variational inference (Section 21.8.1.1).

9.4 Probit regression
In (binary) logistic regression, we use a model of the form p(y=1|xi,w) = sigm( wTxi).I n
general, we can write p(y=1|xi,w)=g−1(wTxi), for any function g−1that maps [−∞,∞]
to[0,1]. Several possible mean functions are listed in Table 9.2.
In this section, we focus on the case where g−1(η)=Φ (η),w h e r eΦ(η)is the cdf of the
standard normal. This is known as probit regression. The probit function is very similar to

the logistic function, as shown in Figure 8.7(b). However, this model has some advantages overlogistic regression, as we will see.

294 Chapter 9. Generalized linear models and the exponential family
9.4.1 ML/MAP estimation using gradient-based optimization
We can ﬁnd the MLE for probit regression using standard gradient methods. Let μi=wTxi,
and let˜yi∈{ −1,+1}. Then the gradient of the log-likelihod for a speciﬁc case is given by
gi≜d
dwlogp(˜yi|wTxi)=dμi
dwd
dμilogp(˜yi|wTxi)=xi˜yiφ(μi)
Φ(˜yiμi)(9.95)
whereφis the standard normal pdf, and Φis its cdf. Similarly, the Hessian for a single case is
given by
Hi=d
dw2logp(˜yi|wTxi)=−xi(φ(μi)2
Φ(˜yiμi)2+˜yiμiφ(μi)
Φ(˜yiμi))
xT
i (9.96)
We can modify these expressions to compute the MAP estimate in a straightforward manner. In
particular, if we use the prior p(w)=N(0,V0), the gradient and Hessian of the penalized
log likelihood have the form∑
igi+2V−1
0wand∑
iHi+2V−1
0. These expressions can be
passed to any gradient-based optimizer. See probitRegDemo for a demo.
9.4.2 Latent variable interpretation
We can interpret the probit (and logistic) model as follows. First, let us associate each itemx

Φ(˜yiμi))
xT
i (9.96)
igi+2V−1
0wand∑
iHi+2V−1
iwith two latent utilities, u0iandu1i, corresponding to the possible choices of yi=0and
yi=1. We then assume that the observed choice is whichever action has larger utility. More
precisely, the model is as follows:
u0i≜wT
0xi+δ0i (9.97)
u1i≜wT
1xi+δ1i (9.98)
yi=I(u1i>u10) (9.99)
whereδ’s are error terms, representing all the other factors that might be relevant in decision
making that we have chosen not to (or are unable to) model. This is called a random utility
modelorRUM(McFadden 1974; Train 2009).

u0i≜wT
u1i≜wT
Since it is only the difference in utilities that matters, let us deﬁne zi=u1i−u0i+ϵi,w h e r e
ϵi=δ1i−δ0i. If theδ’s have a Gaussian distribution, then so does ϵi. Thus we can write
zi≜wTxi+ϵi (9.100)
ϵi∼N(0,1) (9.101)
yi=1 = I(zi≥0) (9.102)
Following (Fruhwirth-Schnatter and Fruhwirth 2010), we call this the difference RUM or dRUM
model.
When we marginalize out zi, we recover the probit model:
p(yi=1|xi,w)=∫
I(zi≥0)N(zi|wTxi,1)dzi (9.103)
=p(wTxi+ϵ≥0) =p(ϵ≥−wTxi) (9.104)
=1−Φ(−wTxi)=Φ (wTxi) (9.105)

9.4. Probit regression 295
where we used the symmetry of the Gaussian.3This latent variable interpretation provides an
alternative way to ﬁt the model, as discussed in Section 11.4.6.
Interestingly, if we use a Gumbel distribution for the δ’s, we induce a logistic distibution for
ϵi, and the model reduces to logistic regression. See Section 24.5.1 for further details.
9.4.3 Ordinal probit regression *
One advantage of the latent variable interpretation of probit regression is that it is easy to extend
to the case where the response variable is ordinal, that is, it can take on Cdiscrete values which
can be ordered in some way, such as low, medium and high. This is called ordinal regression.
The basic idea is as follows. We introduce C+1thresholds γjand set
yi=jifγj−1<zi≤γj (9.106)
whereγ0≤···≤γC. For identiﬁability reasons, we set γ0=−∞,γ1=0andγC=∞.F o r
example, if C=2, this reduces to the standard binary probit model, whereby zi<0produces

yi=0andzi≥0produces yi=1.I fC=3, we partition the real line into 3 intervals:
(−∞,0],(0,γ2],(γ2,∞). We can vary the parameter γ2to ensure the right relative amount
of probability mass falls in each interval, so as to match the empirical frequencies of each classlabel.
Finding the MLEs for this model is a bit trickier than for binary probit regression, since
we need to optimize for wandγ, and the latter must obey an ordering constraint. See e.g.,

(Kawakatsu and Largey 2009) for an approach based on EM. It is also possible to derive a simpleGibbs sampling algorithm for this model (see e.g., (Hoff 2009, p216)).
9.4.4 Multinomial probit models *
Now consider the case where the response variable can take on Cunordered categorical values,
yi∈{1,...,C}. Themultinomial probit model is deﬁned as follows:
zic=wTxic+ϵic (9.107)
ϵ∼N(0,R) (9.108)
yi=a r g m a x
czic (9.109)

yi=a r g m a x
See e.g., (Dow and Endersby 2004; Scott 2009; Fruhwirth-Schnatter and Fruhwirth 2010) formore details on the model and its connection to multinomial logistic regression. (By deﬁningw=[w
1,...,wC], andxic=[0,...,0,xi,0,...,0], we can recover the more familiar
formulation zic=xT
iwc.) Since only relative utilities matter, we constrain Rto be a correlation
matrix. If instead of setting yi=a r g m a xczicwe useyic=I(zic>0), we get a model known
asmultivariate probit, which is one way to model Ccorrelated binary outcomes (see e.g.,
(Talhouk et al. 2011)).

3. Note that the assumption that the Gaussian noise term is zero mean and unit variance is made without loss of
generality. To see why, suppose we used some other mean μand variance σ2. Then we could easily rescale wand add
an offset term without changing the likelihood. since P(N(0,1)≥−wTx)=P(N(μ,σ2)≥−(wTx+μ)/σ).

296 Chapter 9. Generalized linear models and the exponential family
9.5 Multi-task learning
Sometimes we want to ﬁt many related classiﬁcation or regression models. It is often reasonable
to assume the input-output mapping is similar across these different models, so we can getbetter performance by ﬁtting all the parameters at the same time. In machine learning, thissetup is often called multi-task learning (Caruana 1998), transfer learning (e.g., (Raina et al.
2005)), or learning to learn (Thrun and Pratt 1997). In statistics, this is usually tackled using
hierarchical Bayesian models (Bakker and Heskes 2003), as we discuss below, although there areother possible methods (see e.g., (Chai 2010)).
9.5.1 Hierarchical Bayes for multi-task learning
Letyijbe the response of the i’ t hi t e mi ng r o u pj ,f o ri=1:N jandj=1:J. For example,
jmight index schools, imight index students within a school, and yijmight be the test score,

as in Section 5.6.2. Or jmight index people, and imight index purchaes, and yijmight be
the identity of the item that was purchased (this is known as discrete choice modeling (Train
2009)). Let xijbe a feature vector associated with yij. The goal is to ﬁt the models p(yj|xj)
for allj.
Although some groups may have lots of data, there is often a long tail, where the majority

for allj.
of groups have little data. Thus we can’t reliably ﬁt each model separately, but we don’t wantto use the same model for all groups. As a compromise, we can ﬁt a separate model foreach group, but encourage the model parameters to be similar across groups. More precisely,suppose E[y
ij|xij]=g(xT
ijβj),w h e r egis the link function for the GLM. Furthermore, suppose
βj∼N(β∗,σ2
jI), and that β∗∼N(μ,σ2
∗I). In this model, groups with small sample
size borrow statistical strength from the groups with larger sample size, because the βj’s are

correlated via the latent common parents β∗(see Section 5.5 for further discussion of this point).
The term σ2
jcontrols how much group jdepends on the common parents and the σ2
∗term
controls the strength of the overall prior.
Suppose, for simplicity, that μ=0, and that σ2
jandσ2
∗are all known (e.g., they could be set
by cross validation). The overall log probability has the form
logp(D|β)+logp(β)=∑
j[
logp(Dj|βj)−||βj−β∗||2
2σ2
j]
−||β∗||2
2σ2∗(9.110)
We can perform MAP estimation of β=(β1:J,β∗)using standard gradient methods. Alter-
natively, we can perform an iterative optimization scheme, alternating between optimizing the
βjand theβ∗; since the likelihood and prior are convex, this is guaranteed to converge to the

jandσ2
j[
2σ2
j]
−||β∗||2
global optimum. Note that once the models are trained, we can discard β∗, and use each model
separately.
9.5.2 Application to personalized email spam ﬁltering
An interesting application of multi-task learning is personalized spam ﬁltering. Suppose we
want to ﬁt one classiﬁer per user, βj. Since most users do not label their email as spam or not,
it will be hard to estimate these models independently. So we will let the βjhave a common
priorβ∗, representing the parameters of a generic user.

9.5. Multi-task learning 297
In this case, we can emulate the behavior of the above model with a simple trick (Daume
2007b; Attenberg et al. 2009; Weinberger et al. 2009): we make two copies of each feature xi,
one concatenated with the user id, and one not. The effect will be to learn a predictor of the
form
E[yi|xi,u]=(β∗,w1,···,wJ)T[xi,I(u=1 )xi,···,I(u=J)xi] (9.111)
whereuis the user id. In other words,
E[yi|xi,u=j]=(βT
∗+wj)Txi (9.112)
Thusβ∗will be estimated from everyone’s email, whereas wjwill just be estimated from user
j’s email.
To see the correspondence with the above hierarchical Bayesian model, deﬁne wj=βj−β∗.
Then the log probability of the original model can be rewritten as
∑
j[
logp(Dj|β∗+wj)−||wj||2
2σ2
j]
−||β∗||2
2σ2∗(9.113)
If we assume σ2
j=σ2
∗, the effect is the same as using the augmented feature trick, with the
same regularizer strength for both wjandβ∗. However, one typically gets better performance
by not requiring that σ2
jbe equal to σ2
∗(Finkel and Manning 2009).

j’s email.
∑
j[
2σ2
j]
−||β∗||2
j=σ2
9.5.3 Application to domain adaptation
Domain adaptation is the problem of training a set of classiﬁers on data drawn from different
distributions, such as email and newswire text. This problem is obviously a special case of
multi-task learning, where the tasks are the same.
(Finkel and Manning 2009) used the above hierarchical Bayesian model to perform domain

adaptation for two NLP tasks, namely named entity recognition and parsing. They report reason-ably large improvements over ﬁtting separate models to each dataset, and small improvementsover the approach of pooling all the data and ﬁtting a single model.
9.5.4 Other kinds of prior
In multi-task learning, it is common to assume that the prior is Gaussian. However, sometimesother priors are more suitable. For example, consider the task of conjoint analysis, which

requires ﬁguring out which features of a product customers like best. This can be modelledusing the same hierarchical Bayesian setup as above, but where we use a sparsity-promotingprior onβ
j, rather than a Gaussian prior. This is called multi-task feature selection. See e.g.,
(Lenk et al. 1996; Argyriou et al. 2008) for some possible approaches.
It is not always reasonable to assume that all tasks are all equally similar. If we pool the

parameters across tasks that are qualitatively different, the performance will be worse than notusing pooling, because the inductive bias of our prior is wrong. Indeed, it has been foundexperimentally that sometimes multi-task learning does worse than solving each task separately(this is called negative transfer).

298 Chapter 9. Generalized linear models and the exponential family
One way around this problem is to use a more ﬂexible prior, such as a mixture of Gaussians.
Such ﬂexible priors can provide robustness against prior mis-speciﬁcation. See e.g., (Xue et al.
2007; Jacob et al. 2008) for details. One can of course combine mixtures with sparsity-promotingpriors (Ji et al. 2009). Many other variants are possible.
9.6 Generalized linear mixed models *
Suppose we generalize the multi-task learning scenario to allow the response to include infor-mation at the group level, x
j, as well as at the item level, xij. Similarly, we can allow the
parameters to vary across groups, βj, or to be tied across groups, α. This gives rise to the
following model:
E[yij|xij,xj]=g(
φ1(xij)Tβj+φ2(xj)Tβ′
j+φ3(xij)Tα+φ4(xj)Tα′)
(9.114)
where the φkare basis functions. This model can be represented pictorially as shown in
Figure 9.2(a). (Such ﬁgures will be explained in Chapter 10.) Note that the number of βj

(9.114)
parameters grows with the number of groups, whereas the size of αis ﬁxed.
Frequentists call the terms βjrandom effects, since they vary randomly across groups, but
they call αaﬁxed effect, since it is viewed as a ﬁxed but unknown constant. A model with
both ﬁxed and random effects is called a mixed model.I f p(y|x)is a GLM, the overall model
is called a generalized linear mixed effects model orGLMM. Such models are widely used in
9.6.1 Example: semi-parametric GLMMs for medical data
Consider the following example from (Wand 2009). Suppose yijis the amount of spinal bone

mineral density (SBMD) for person jat measurement i.L e txijbe the age of person, and let
xjbe their ethnicity, which can be one of: White, Asian, Black, or Hispanic. The primary goal
is to determine if there are signiﬁcant differences in the mean SBMD among the four ethnic
groups, after accounting for age. The data is shown in the light gray lines in Figure 9.2(b). Wesee that there is a nonlinear effect of SBMD vs age, so we will use a semi-parametric model

which combines linear regression with non-parametric regression (Ruppert et al. 2003). We alsosee that there is variation across individuals within each group, so we will use a mixed effectsmodel. Speciﬁcally, we will use φ
1(xij)=1to account for the random effect of each person;
φ2(xij)=0since no other coefficients are person-speciﬁc; φ3(xij)=[bk(xij)],w h e r e bkis
thek’th spline basis functions (see Section 15.4.6.2), to account for the nonlinear effect of age;
andφ4(xj)=[I(xj=w),I(xj=a),I(xj=b),I(xj=h)]to account for the effect of the

different ethnicities. Furthermore, we use a linear link function. The overall model is therefore
E[yij|xij,xj]=β j+αTb(xij)+ϵij (9.115)
+α′
wI(xj=w)+α′
aI(xj=a)+α′
bI(xj=b)+α′
hI(xj=h)(9.116)
whereϵij∼N(0,σ2
y).αcontains the non-parametric part of the model related to age, α′
contains the parametric part of the model related to ethnicity, and βjis a random offset
for person j. We endow all of these regression coefficients with separate Gaussian priors.
We can then perform posterior inference to compute p(α,α′,β,σ2|D)(see Section 9.6.2 for

9.6. Generalized linear mixed models * 299
xjxijyij
Njβj
Jμβσ2
β
α μα
σ2
ασ2
y
(a)
age in yearsspinal bone mineral density (g/cm2)
0.60.81.01.21.4
10 15 20 25Asian BlackHispanic10 15 20 25
0.60.81.01.21.4White
(b)
Figure 9.2 (a) Directed graphical model for generalized linear mixed effects model with Jgroups. (b)
Spinal bone mineral density vs age for four different ethnic groups. Raw data is shown in the light gray
lines. Fitted model shown in black (solid is the posterior predicted mean, dotted is the posterior predictivevariance). From Figure 9 of (Wand 2009). Used with kind permission of Matt Wand

300 Chapter 9. Generalized linear models and the exponential family
computational details). After ﬁtting the model, we can compute the prediction for each group.
See Figure 9.2(b) for the results. We can also perform signiﬁcance testing, by computing p(αg−
αw|D)for each ethnic group grelative to some baseline (say, White), as we did in Section 5.2.3.
9.6.2 Computational issues
The principle problem with GLMMs is that they can be difficult to ﬁt, for two reasons. First,p(y
ij|θ)may not be conjugate to the prior p(θ)whereθ=(α,β). Second, there are two levels
of unknowns in the model, namely the regression coefficients θand the means and variances
of the priors η=(μ,σ).
One approach is to adopt fully Bayesian inference methods, such as variational Bayes (Hall
et al. 2011) or MCMC (Gelman and Hill 2007). We discuss VB in Section 21.5, and MCMC inSection 24.1.
An alternative approach is to use empirical Bayes, which we discuss in general terms in

Section 5.6. In the context of a GLMM, we can use the EM algorithm (Section 11.4), where in theE step we compute p(θ|η,D), and in the M step we optimize η. If the linear regression setting,
the E step can be performed exactly, but in general we need to use approximations. Traditionalmethods use numerical quadrature or Monte Carlo (see e.g., (Breslow and Clayton 1993)). Afaster approach is to use variational EM; see (Braun and McAuliffe 2010) for an application ofvariational EM to a multi-level discrete choice modeling problem.

In frequentist statistics, there is a popular method for ﬁtting GLMMs called generalized
estimating equations orGEE(Hardin and Hilbe 2003). However, we do not recommend this
approach, since it is not as statistically efficient as likelihood-based methods (see Section 6.4.3).In addition, it can only provide estimates of the population parameters α, but not the random
effectsβ
j, which are sometimes of interest in themselves.
9.7 Learning to rank *
In this section, we discuss the learning to rank orLETORproblem. That is, we want to learn a

effectsβ
function that can rank order a set of items (we will be more precise below). The most commonapplication is to information retrieval. Speciﬁcally, suppose we have a query qand a set of
documents d
1,...,dmthat might be relevant to q(e.g., all documents that contain the string q).
We would like to sort these documents in decreasing order of relevance and show the top kto
the user. Similar problems arise in other areas, such as collaborative ﬁltering. (Ranking playersin a game or tournament setting is a slightly different kind of problem; see Section 22.5.5.)

Below we summarize some methods for solving this problem, following the presentation of
(Liu 2009). This material is not based on GLMs, but we include it in this chapter anyway forlack of a better place.
A standard way to measure the relevance of a document dto a query qis to use a probabilistic
language model based on a bag of words model. That is, we deﬁne sim(q,d)≜p(q|d)=∏
n
i=1p(qi|d),w h e r eqiis thei’th word or term, and p(qi|d)is a multinoulli distribution
estimated from document d. In practice, we need to smooth the estimated distribution, for

n
example by using a Dirichlet prior, representing the overall frequency of each word. This can be

9.7. Learning to rank * 301
estimated from all documents in the system. More precisely, we can use
p(t|d)=( 1−λ)TF(t,d)
LEN(d)+λp(t|background) (9.117)
where TF(t,d)is the frequency of term tin document d, LEN(d)is the number of words in d,
and0<λ<1is a smoothing parameter (see e.g., Zhai and Lafferty (2004) for details).
However, there might be many other signals that we can use to measure relevance. For
example, the PageRank of a web document is a measure of its authoritativeness, derived from
the web’s link structure (see Section 17.2.4 for details). We can also compute how often andwhere the query occurs in the document. Below we discuss how to learn how to combine allthese signals.
4
9.7.1 The pointwise approach
Suppose we collect some training data representing the relevance of a set of documents for eachquery. Speciﬁcally, for each query q, suppose that we retrieve mpossibly relevant documents
d
j,f o rj=1:m. For each query document pair, we deﬁne a feature vector, x(q,d).F o r

4
d
example, this might contain the query-document similarity score and the page rank score of thedocument. Furthermore, suppose we have a set of labels y
jrepresenting the degree of relevance
of document djto queryq. Such labels might be binary (e.g., relevant or irrelevant), or they may
represent a degree of relevance (e.g., very relevant, somewhat relevant, irrelevant). Such labelscan be obtained from query logs, by thresholding the number of times a document was clickedon for a given query.

If we have binary relevance labels, we can solve the problem using a standard binary clas-
siﬁcation scheme to estimate, p(y=1|x(q,d)). If we have ordered relevancy labels, we can
use ordinal regression to predict the rating, p(y=r|x(q,d)). In either case, we can then sort
the documents by this scoring metric. This is called the pointwise approach to LETOR, and

is widely used because of its simplicity. However, this method does not take into account thelocation of each document in the list. Thus it penalizes errors at the end of the list just as muchas errors at the beginning, which is often not the desired behavior. In addition, each decisionabout relevance is made very myopically.
9.7.2 The pairwise approach
There is evidence (e.g., (Carterette et al. 2008)) that people are better at judging the relativerelevance of two items rather than absolute relevance. Consequently, the data might tell usthatd
jis more relevant than dkfor a given query, or vice versa. We can model this kind of

data using a binary classiﬁer of the form p(yjk|x(q,dj),x(q,dk)), where we set yjk=1if
rel(dj,q)>rel(dk,q)andyjk=0otherwise.
One way to model such a function is as follows:
p(yjk=1|xj,xk) = sigm(f (xj)−f(xk)) (9.118)
4. Rather surprisingly, Google does not (or at least, did not as of 2008) using such learning methods in its search engine.
Source: Peter Norvig, quoted in http://anand .typepad .com/datawocky/2008/05/are-human-experts-less-p
rone-to-catastrophic-errors-than-machine-learned-models .html.

302 Chapter 9. Generalized linear models and the exponential family
wheref(x)is a scoring function, often taken to be linear, f(x)=wTx. This is a special
kind of neural network known as RankNet (Burges et al. 2005) (see Section 16.5 for a general
discussion of neural networks). We can ﬁnd the MLE of wby maximizing the log likelihood, or
equivalently, by minimizing the cross entropy loss, given by
L=N∑
i=1mi∑
j=1mi∑
k=j+1Lijk (9.119)
−Lijk=I(yijk=1 )l o gp(yijk=1|xij,xik,w)
+I(yijk=0 )l o gp(yijk=0|xij,xik,w) (9.120)
This can be optimized using gradient descent. A variant of RankNet is used by Microsoft’s Bing
search engine.5
9.7.3 The listwise approach
The pairwise approach suffers from the problem that decisions about relevance are made justbased on a pair of items (documents), rather than considering the full context. We now considermethods that look at the entire list of items at the same time.
We can deﬁne a total order on a list by specifying a permutation of its indices, π. To model

our uncertainty about π, we can use the Plackett-Luce distribution, which derives its name
from independent work by (Plackett 1975) and (Luce 1959). This has the following form:
p(π|s)=
m∏
j=1sj∑m
u=jsu(9.121)
wheresj=s(π−1(j))is the score of the document ranked at the j’th position.
To understand Equation 9.121, let us consider a simple example. Suppose π=(A,B,C).
Then we have that p(π)is the probability of Abeing ranked ﬁrst, times the probability of B

p(π|s)=
m∏
j=1sj∑m
being ranked second given that Ais ranked ﬁrst, times the probabilty of Cbeing ranked third
given that AandBare ranked ﬁrst and second. In other words,
p(π|s)=sA
sA+sB+sC×sB
sB+sC×sC
sC(9.122)
To incorporate features, we can deﬁne s(d)=f(x(q,d)), where we often take fto be a
linear function, f(x)=wTx. This is known as the ListNetmodel (Cao et al. 2007). To train
this model, let yibe the relevance scores of the documents for query i. We then minimize the
cross entropy term
−∑
i∑
πp(π|yi)logp(π|si) (9.123)

p(π|s)=sA
sB+sC×sC
sC(9.122)
−∑
i∑
Of course, as stated, this is intractable, since the i’th term needs to sum over mi!permutations.
To make this tractable, we can consider permutations over the top kpositions only:
p(π1:k|s1:m)=k∏
j=1sj∑mu=1su(9.124)
5. Source: http://www .bing.com/community/site_blogs/b/search/archive/2009/06/01/user-needs-f
eatures-and-the-science-behind-bing .aspx.

9.7. Learning to rank * 303
There are only m!/(m−k)!such permutations. If we set k=1, we can evaluate each cross
entropy term (and its derivative) in O(m)time.
In the special case where only one document from the presented list is deemed relevant, say
yi=c, we can instead use multinomial logistic regression:
p(yi=c|x)=exp(sc)∑m
c′=1exp(sc′)(9.125)
This often performs at least as well as ranking methods, at least in the context of collaborative
ﬁltering (Yang et al. 2011).
9.7.4 Loss functions for ranking
There are a variety of ways to measure the performance of a ranking system, which we summa-rize below.
•Mean reciprocal rank (MRR). For a query q, let the rank position of its ﬁrst relevant
document be denoted by r(q). Then we deﬁne the mean reciprocal rank to be1/r(q).
This is a very simple performance measure.
•Mean average precision (MAP). In the case of binary relevance labels, we can deﬁne the
precision at k of some ordering as follows:
P@k(π)≜num. relevant documents in the top kpositions of π
k(9.126)

k(9.126)
We then deﬁne the average precision as follows:
AP(π)≜∑
kP@k(π)·Ik
num. relevant documents(9.127)
whereIkis 1 iff document kis relevant. For example, if we have the relevancy labels
y=( 1,0,1,0,1), then the AP is1
3(1
1+2
3+3
5)≈0.76. Finally, we deﬁne the mean average
precision as the AP averaged over all queries.
•Normalized discounted cumulative gain (NDCG). Suppose the relevance labels have multi-ple levels. We can deﬁne the discounted cumulative gain of the ﬁrst kitems in an ordering
DCG@k(r)=r
1+k∑
i=2ri
log2i(9.128)

AP(π)≜∑
3(1
1+2
3+3
1+k∑
i=2ri
whereriis the relevance of item iand thelog2term is used to discount items later in
the list. Table 9.3 gives a simple numerical example. An alternative deﬁnition, that placesstronger emphasis on retrieving relevant documents, uses
DCG@k(r)=k∑
i=12ri−1
log2(1+i)(9.129)
The trouble with DCG is that it varies in magnitude just because the length of a returnedlist may vary. It is therefore common to normalize this measure by the ideal DCG, which is

304 Chapter 9. Generalized linear models and the exponential family
i12 3 4 5 6
ri32 3 0 1 2
log2i0 1 1.59 2.0 2.32 2.59
ri
log2iN/A 2 1.887 0 0.431 0.772
Table 9.3 Illustration of how to compute NDCG, from http://en .wikipedia .org/wiki/Discounted
_cumulative_gain . The value riis the relevance score of the item in position i. From this, we see
that DCG@6 =3+( 2+1 .887+0+0 .431+0.772) = 8.09. The maximum DCG is obtained using the
ordering with scores 3, 3, 2, 2, 1, 0. Hence the ideal DCG is 8.693, and so the normalized DCG is 8.09 /
8.693 = 0.9306.
the DCG obtained by using the optimal ordering: IDCG@k( r) = argmaxπDCG@k(r). This
can be easily computed by sorting r1:mand then computing DCG@k. Finally, we deﬁne
thenormalized discounted cumulative gain orNDCGas DCG/IDCG. Table 9.3 gives a
simple numerical example. The NDCG can be averaged over queries to give a measure of
•Rank correlation . We can measure the correlation between the ranked list, π, and the

relevance judegment, π∗, using a variety of methods. One approach, known as the (weighted)
Kendall’s τstatistics, is deﬁned in terms of the weighted pairwise inconsistency between the
two lists:
τ(π,π∗)=∑
u<vwuv[1+sgn(π u−πv)sgn(π∗
u−π∗
v)]
2∑
u<vwuv(9.130)
A variety of other measures are commonly used.
These loss functions can be used in different ways. In the Bayesian approach, we ﬁrst ﬁt the

two lists:
τ(π,π∗)=∑
u−π∗
v)]
2∑
model using posterior inference; this depends on the likelihood and prior, but not the loss. Wethen choose our actions at test time to minimize the expected future loss. One way to do this isto sample parameters from the posterior, θ
s∼p(θ|D), and then evaluate, say, the precision@k
for different thresholds, averaging over θs. See (Zhang et al. 2010) for an example of such an
approach.
In the frequentist approach, we try to minimize the empirical loss on the training set. The

approach.
problem is that these loss functions are not differentiable functions of the model parameters.We can either use gradient-free optimization methods, or we can minimize a surrogate lossfunction instead. Cross entropy loss (i.e., negative log likelihood) is an example of a widely usedsurrogate loss function.
Another loss, known as weighted approximate-rank pairwise orWARPloss, proposed in
(Usunier et al. 2009) and extended in (Weston et al. 2010), provides a better approximation tothe precision@k loss. WARP is deﬁned as follows:

WARP(f(x,:),y)≜L(rank(f(x,:),y)) (9.131)
rank(f(x,:),y)=∑
y′̸=yI(f(x,y′)≥f(x,y)) (9.132)
L(k)≜k∑
j=1αj,withα1≥α2≥···≥0 (9.133)

9.7. Learning to rank * 305
Heref(x,:) = [f(x,1),...,f(x,|y|)]is the vector of scores for each possible output label,
or, in IR terms, for each possible document corresponding to input query x. The expression
rank(f(x,:),y)measures the rank of the true label yassigned by this scoring function. Finally,
Ltransforms the integer rank into a real-valued penalty. Using α1=1andαj>1=0would
optimize the proportion of top-ranked correct labels. Setting α1:kto be non-zero values would
optimize the top kin the ranked list, which will induce good performance as measured by
MAP or precision@k. As it stands, WARP loss is still hard to optimize, but it can be further
approximated by Monte Carlo sampling, and then optimized by gradient descent, as describedin (Weston et al. 2010).
Exercises
Exercise 9.1 Conjugate prior for univariate Gaussian in exponential family form
Derive the conjugate prior for μandλ=1/σ2for a univariate Gaussian using the exponential family,

Exercises
by analogy to Section 9.2.5.5. By suitable reparameterization, show that the prior has the form p(μ,λ)=
N(μ|γ,λ(2α−1))Ga(λ|α,β), and thus only has 3 free parameters.
Exercise 9.2 The MVN is in the exponential family
Show that we can write the MVN in exponential family form. Hint: use the information form deﬁned in
Section 4.3.3.

10 Directed graphical models (Bayes nets)
10.1 Introduction
I basically know of two principles for treating complicated systems in simple ways: the
ﬁrst is the principle of modularity and the second is the principle of abstraction. Iam an apologist for computational probability in machine learning because I believe thatprobability theory implements these two principles in deep and intriguing ways — namelythrough factorization and through averaging. Exploiting these two mechanisms as fullyas possible seems to me to be the way forward in machine learning. — Michael Jordan,1997 (quoted in (Frey 1998)).
Suppose we observe multiple correlated variables, such as words in a document, pixels in animage, or genes in a microarray. How can we compactly representthejoint distribution p(x|θ)?
How can we use this distribution to inferone set of variables given another in a reasonable
amount of computation time? And how can we learnthe parameters of this distribution with a

reasonable amount of data? These questions are at the core of probabilistic modeling, inferenceand learning, and form the topic of this chapter.
10.1.1 Chain rule
By thechain rule of probability, we can always represent a joint distribution as follows, using
any ordering of the variables:
p(x1:V)=p(x1)p(x2|x1)p(x3|x2,x1)p(x4|x1,x2,x3)...p(xV|x1:V−1) (10.1)
whereVis the number of variables, the Matlab-like notation 1:Vdenotes the set {1,2,...,V},
and where we have dropped the conditioning on the ﬁxed parameters θfor brevity. The problem
with this expression is that it becomes more and more complicated to represent the conditionaldistributions p(x

t|x1:t−1)astgets large.
For example, suppose all the variables have Kstates. We can represent p(x1)as a table
ofO(K)numbers, representing a discrete distribution (there are actually only K−1free
parameters, due to the sum-to-one constraint, but we write O(K)for simplicity). Similarly, we
can represent p(x2|x1)as a table of O(K2)numbers by writing p(x2=j|x1=i)=Tij;w e
say thatTis astochastic matrix, since it satisﬁes the constraint∑
jTij=1for all rows i,

and0≤Tij≤1for all entries. Similarly, we can represent p(x3|x1,x2)as a 3d table with

308 Chapter 10. Directed graphical models (Bayes nets)
O(K3)numbers. These are called conditional probability tables orCPTs. We see that there
areO(KV)parameters in the model. We would need an awful lot of data to learn so many
One solution is to replace each CPT with a more parsimonius conditional probability distri-
butionorCPD, such as multinomial logistic regression, i.e., p(xt=k|x1:t−1)=S(Wtx1:t−1)k.
The total number of parameters is now only O(K2V2), making this a compact density model
(Neal 1992; Frey 1998). This is adequate if all we want to do is evaluate the probability of a fully
observed vector x1:T. For example, we can use this model to deﬁne a class-conditional density,
p(x|y=c), thus making a generative classiﬁer (Bengio and Bengio 2000). However, this model
is not useful for other kinds of prediction tasks, since each variable depends on all the previousvariables. So we need another approach.
10.1.2 Conditional independence

The key to efficiently representing large joint distributions is to make some assumptions aboutconditional independence (CI). Recall from Section 2.2.4 that XandYare conditionally inde-
pendent given Z, denoted X⊥Y|Z, if and only if (iff) the conditional joint can be written as
a product of conditional marginals, i.e.,
X⊥Y|Z⇐⇒p(X,Y|Z)=p(X|Z)p(Y|Z) (10.2)
Let us see why this might help. Suppose we assume that x
t+1⊥x1:t−1|xt, or in words,
“the future is independent of the past given the present”. This is called the (ﬁrst order) Markov

assumption. Using this assumption, plus the chain rule, we can write the joint distribution asfollows:
p(x
1:V)=p(x1)V∏
t=1p(xt|xt−1) (10.3)
This is called a (ﬁrst-order) Markov chain. They can be characterized by an initial distribution
over states, p(x1=i), plus astate transition matrix p(xt=j|xt−1=i). See Section 17.2 for
more information.
10.1.3 Graphical models
Although the ﬁrst-order Markov assumption is useful for deﬁning distributions on 1d sequences,how can we deﬁne distributions on 2d images, or 3d videos, or, in general, arbitrary collectionsof variables (such as genes belonging to some biological pathway)? This is where graphicalmodels come in.

Agraphical model (GM) is a way to represent a joint distribution by making CI assumptions.
Inparticular, thenodesinthegraphrepresentrandomvariables, andthe(lackof)edgesrepresentCI assumptions. (A better name for these models would in fact be “independence diagrams”,but the term “graphical models” is now entrenched.) There are several kinds of graphical model,depending on whether the graph is directed, undirected, or some combination of directed andundirected. In this chapter, we just study directed graphs. We consider undirected graphs inChapter 19.

10.1. Introduction 309
45231
(a)45231
(b)
Figure 10.1 (a) A simple DAG on 5 nodes, numbered in topological order. Node 1 is the root, nodes 4 and
5 are the leaves. (b) A simple undirected graph, with the following maximal cliques: {1,2,3},{2,3,4},
{3,5}.
10.1.4 Graph terminology
Before we continue, we must deﬁne a few basic terms, most of which are very intuitive.
AgraphG=(V,E)consists of a set of nodesorvertices, V={1,...,V }, and a set
ofedges,E={(s,t):s,t∈V }. We can represent the graph by its adjacency matrix,i n
which we write G(s,t)=1to denote (s,t)∈E, that is, if s→tis an edge in the graph.
IfG(s,t)=1iffG(t,s)=1, we say the graph is undirected, otherwise it is directed.W e
usually assume G(s,s)=0, which means there are no self loops.
Here are some other terms we will commonly use:
•ParentFor a directed graph, the parentsof a node is the set of all nodes that feed into it:
pa(s)≜{t:G(t,s)=1}.
•ChildFor a directed graph, the children of a node is the set of all nodes that feed out of it:

ch(s)≜{t:G(s,t)=1} .
•FamilyFor a directed graph, the familyof a node is the node and its parents, fam(s)=
{s}∪pa(s).
•RootFor a directed graph, a rootis a node with no parents.
•LeafFor a directed graph, a leafis a node with no children.
•Ancestors For a directed graph, the ancestors are the parents, grand-parents, etc of a node.
That is, the ancestors of tis the set of nodes that connect to tvia a trail: anc(t) ≜{s:s;
t}.
•Descendants For a directed graph, the descendants are the children, grand-children, etc of

t}.
a node. That is, the descendants of sis the set of nodes that can be reached via trails from
s:desc(s)≜{t:s;t}.
•Neighbors For any graph, we deﬁne the neighbors of a node as the set of all immediately
connected nodes, nbr(s)≜{t:G(s,t)=1∨G(t,s)=1}. For an undirected graph, we

310 Chapter 10. Directed graphical models (Bayes nets)
writes∼tto indicate that sandtare neighbors (so (s,t)∈Eis an edge in the graph).
•DegreeThedegreeof a node is the number of neighbors. For directed graphs, we speak of
thein-degree andout-degree, which count the number of parents and children.
•Cycle or loop For any graph, we deﬁne a cycleorloopto be a series of nodes such that
we can get back to where we started by following edges, s1−s2···−sn−s1,n≥2. If the
graph is directed, we may speak of a directed cycle. For example, in Figure 10.1(a), there are
no directed cycles, but 1→2→4→3→1is an undirected cycle.
•DAGAdirected acyclic graph orDAGis a directed graph with no directed cycles. See
Figure 10.1(a) for an example.
•Topological ordering For a DAG, a topological ordering ortotal ordering is a numbering
of the nodes such that parents have lower numbers than their children. For example, inFigure 10.1(a), we can use (1,2,3,4,5),o r(1,3,2,5,4), etc.

•Path or trail Apathortrails;tis a series of directed edges leading from stot.
•TreeAn undirected treeis an undirectecd graph with no cycles. A directed tree is a DAG in
which there are no directed cycles. If we allow a node to have multiple parents, we call it apolytree, otherwise we call it a moral directed tree.
•ForestAforestis a set of trees.
•Subgraph A (node-induced) subgraph G
Ais the graph created by using the nodes in Aand
their corresponding edges, GA=(VA,EA).

•CliqueFor an undirected graph, a cliqueis a set of nodes that are all neighbors of each
other. A maximal clique is a clique which cannot be made any larger without losing the
clique property. For example, in Figure 10.1(b), {1,2}is a clique but it is not maximal, since
we can add 3 and still maintain the clique property. In fact, the maximal cliques are asfollows:{1,2,3},{2,3,4},{3,5}.
10.1.5 Directed graphical models
Adirected graphical model orDGMis a GM whose graph is a DAG. These are more commonly

known as Bayesian networks. However, there is nothing inherently “Bayesian” about Bayesian
networks: they are just a way of deﬁning probability distributions. These models are also calledbelief networks. The term “belief” here refers to subjective probability. Once again, there isnothing inherently subjective about the kinds of probability distributions represented by DGMs.Finally, these models are sometimes called causal networks, because the directed arrows are

sometimes interpreted as representing causal relations. However, there is nothing inherentlycausal about DGMs. (See Section 26.6.1 for a discussion of causal DGMs.) For these reasons, weuse the more neutral (but less glamorous) term DGM.
The key property of DAGs is that the nodes can be ordered such that parents come before
children. This is called a topological ordering, and it can be constructed from any DAG. Givensuch an order, we deﬁne the ordered Markov property to be the assumption that a node only

depends on its immediate parents, not on all predecessors in the ordering, i.e.,
x
s⊥xpred(s)\pa(s) |xpa(s) (10.4)
wherepa(s)are the parents of node s, andpred(s)are the predecessors of node sin the
ordering. This is a natural generalization of the ﬁrst-order Markov property to from chains togeneral DAGs.

=p(x1)p(x2|x1)p(x3|x1)p(x4|x2,x3)p(x5|x3) (10.6)
p(x1:V|G)=V∏
t=1p(xt|xpa(t)) (10.7)
where each term p(xt|xpa(t))is a CPD. We have written the distribution as p(x|G)to emphasize
that this equation only holds if the CI assumptions encoded in DAG Gare correct. However,
we will usual drop this explicit conditioning for brevity. If each node has O(F)parents and
Kstates, the number of parameters in the model is O(VKF), which is much less than the
O(KV)needed by a model which makes no CI assumptions.
10.2 Examples
In this section, we show a wide variety of commonly used probabilistic models can be conve-
niently represented as DGMs.
10.2.1 Naive Bayes classiﬁers
In Section 3.5, we introduced the naive Bayes classiﬁer. This assumes the features are condi-tionally independent given the class label. This assumption is illustrated in Figure 10.2(a). Thisallows us to write the joint distirbution as follows:
p(y,x)=p(y)
D∏
j=1p(xj|y) (10.8)

D∏
The naive Bayes assumption is rather naive, since it assumes the features are conditionally
independent. One way to capture correlation between the features is to use a graphical model.In particular, if the model is a tree, the method is known as a tree-augmented naive Bayes

312 Chapter 10. Directed graphical models (Bayes nets)
x1 x2 x3···
(a)x1 x2 x3 x4···
(b)
Figure 10.3 A ﬁrst and second order Markov chain.
x1 x2 xTz1 z2 zT
Figure 10.4 A ﬁrst-order HMM.
classiﬁer orTANmodel (Friedman et al. 1997). This is illustrated in Figure 10.2(b). The reason
to use a tree, as opposed to a generic graph, is two-fold. First, it is easy to ﬁnd the optimal
tree structure using the Chow-Liu algorithm, as explained in Section 26.3. Second, it is easy tohandle missing features in a tree-structured model, as we explain in Section 20.2.
10.2.2 Markov and hidden Markov models
Figure 10.3(a) illustrates a ﬁrst-order Markov chain as a DAG. Of course, the assumption that theimmediate past, x
t−1, captures everything we need to know about the entire history, x1:t−2,i s
a bit strong. We can relax it a little by adding a dependence from xt−2toxtas well; this is
called asecond order Markov chain, and is illustrated in Figure 10.3(b). The corresponding
joint has the following form:

p(x1:T)=p(x1,x2)p(x3|x1,x2)p(x4|x2,x3)...=p(x1,x2)T∏
t=3p(xt|xt−1,xt−2)(10.9)
We can create higher-order Markov models in a similar way. See Section 17.2 for a more detaileddiscussion of Markov models.
Unfortunately, even the second-order Markov assumption may be inadequate if there are long-

range correlations amongst the observations. We can’t keep building ever higher order models,since the number of parameters will blow up. An alternative approach is to assume that thereis an underlying hidden process, that can be modeled by a ﬁrst-order Markov chain, but thatthe data is a noisy observation of this process. The result is known as a hidden Markov model
orHMM, and is illustrated in Figure 10.4. Here z
tis known as a hidden variable at “time” t,

andxtis the observed variable. (We put “time” in quotation marks, since these models can be
applied to any kind of sequence data, such as genomics or language, where trepresents location
rather than time.) The CPD p(zt|zt−1)is thetransition model, and the CPD p(xt|zt)is the
observation model.

10.2. Examples 313
h0h1h2P(v=0|h1,h2)P(v=1|h1,h2)
100 θ0 1−θ0
110 θ0θ1 1−θ0θ1
101 θ0θ2 1−θ0θ2
111 θ0θ1θ2 1−θ0θ1θ2
Table 10.1 Noisy-OR CPD for 2 parents augmented with leak node. We have omitted the tsubscript for
brevity.
The hidden variables often represent quantities of interest, such as the identity of the word
that someone is currently speaking. The observed variables are what we measure, such as the
acoustic waveform. What we would like to do is estimate the hidden state given the data, i.e., tocompute p(z
t|x1:t,θ). This is called state estimation, and is just another form of probabilistic
inference. See Chapter 17 for further details on HMMs.
10.2.3 Medical diagnosis
Consider modeling the relationship between various variables that are measured in an intensivecare unit (ICU), such as the breathing rate of a patient, their blood pressure, etc. The alarm
network in Figure 10.5(a) is one way to represent these (in)dependencies (Beinlich et al. 1989).
This model has 37 variables and 504 parameters.

Since this model was created by hand, by a process called knowledge engineering,i ti s
known as a probabilistic expert system. In Section 10.4, we discuss how to learn the parameters
of DGMs from data, assuming the graph structure is known, and in Chapter 26, we discuss howto learn the graph structure itself.
A different kind of medical diagnosis network, known as the quick medical reference or
QMRnetwork (Shwe et al. 1991), is shown in Figure 10.5(b). This was designed to model infectious

diseases. The QMR model is a bipartite graph structure, with diseases (causes) at the top and
symptoms or ﬁndings at the bottom. All nodes are binary. We can write the distribution asfollows:
p(v,h)=∏
sp(hs)∏
tp(vt|hpa(t)) (10.10)
wherehsrepresent the hidden nodes (diseases), and vtrepresent the visible nodes (symptoms).
The CPD for the root nodes are just Bernoulli distributions, representing the prior probability

p(v,h)=∏
sp(hs)∏
of that disease. Representing the CPDs for the leaves (symptoms) using CPTs would requiretoo many parameters, because the fan-in(number of parents) of many leaf nodes is very
high. A natural alternative is to use logistic regression to model the CPD, p(v
t=1|hpa(t))=
sigm(wT
thpa(t)). (A DGM in which the CPDs are logistic regression distributions is known as a
sigmoid belief net (Neal 1992).) However, since the parameters of this model were created by
hand, an alternative CPD, known as the noisy-OR model, was used.

sigm(wT
The noisy-OR model assumes that if a parent is on, then the child will usually also be on
(since it is an or-gate), but occasionally the “links” from parents to child may fail, independentlyat random. In this case, even if the parent is on, the child may be off. To model this moreprecisely, let θ
st=1−qstbe the probability that the s→tlink fails, so qst=1−θst=p(vt=

10.2. Examples 315
GpGmp(X=a)p(X=b)p(X=o)p(X=ab)
aa 1000
ab 000 1
ao 1000
ba 000 1
bb 0100
bo 0100
oa 1000
ob 0100
oo 001 0
Table 10.2 CPT which encodes a mapping from genotype to phenotype (bloodtype). This is a determin-
istic, but many-to-one, mapping.
1|hs=1,h−s=0 )is the probability that scan activate ton its own (its “causal power”). The
only way for the child to be off is if all the links from all parents that are on fail independently
at random. Thus
p(vt=0|h)=∏
s∈pa(t)θI(hs=1)
st (10.11)
Obviously, p(vt=1|h)=1−p(vt=0|h).
If we observe that vt=1but all its parents are off, then this contradicts the model. Such
a data case would get probability zero under the model, which is problematic, because it ispossible that someone exhibits a symptom but does not have any of the speciﬁed diseases. Tohandle this, we add a dummy leak node h
0, which is always on; this represents “all other
causes”. The parameter q0trepresents the probability that the background leak can cause the

st (10.11)
effect on its own. The modiﬁed CPD becomes p(vt=0|h)=θ0t∏
s∈pa(t)θhs
st. See Table 10.1
for a numerical example.
If we deﬁne wst≜log(θst), we can rewrite the CPD as
p(vt=1|h)=1−exp(
w0t+∑
shswst)
(10.12)
We see that this is similar to a logistic regression model.
Bipartite models with noisy-OR CPDs are called BN2Omodels. It is relatively easy to set the
θstparameters by hand, based on domain expertise. However, it is also possible to learn them

w0t+∑
shswst)
(10.12)
from data (see e.g, (Neal 1992; Meek and Heckerman 1997)). Noisy-OR CPDs have also proveduseful in modeling human causal learning (Griffiths and Tenenbaum 2005), as well as generalbinary classiﬁcation settings (Yuille and Zheng 2009).
10.2.4 Genetic linkage analysis *
Another important (and historically very early) application of DGMs is to the problem of genetic
linkage analysis. We start with a pedigree graph, which is a DAG that representing the

relationship between parents and children, as shown in Figure 10.6(a). We then convert this to aDGM, as we explain below. Finally we perform probabilistic inference in the resulting model.

10.2. Examples 317
In more detail, for each person (or animal) iand location or locus jalong the genome, we
create three nodes: the observed markerXij(which can be a property such as blood type,
or just a fragment of DNA that can be measured), and two hidden alleles,Gm
ijandGp
ij, one
inherited from i’s mother (maternal allele) and the other from i’s father (paternal allele). Together,
the ordered pair Gij=(Gm
ij,Gp
ij)constitutes i’s hidden genotype at locusj.
Obviously we must add Gm
ij→XijandGp
ij→Xijarcs representing the fact that genotypes
cause phenotypes (observed manifestations of genotypes). The CPD p(Xij|Gm
ij,Gp
ij)is called
thepenetrance model. As a very simple example, suppose Xij∈{A,B,O,AB }represents
personi’s observed bloodtype, and Gm
ij,Gp
ij∈{A,B,O}is their genotype. We can repre-
sent the penetrance model using the deterministic CPD shown in Table 10.2. For example, A
dominates O, so if a person has genotype AO or OA, their phenotype will be A.

ij,Gp
ij,Gp
In addition, we add arcs from i’s mother and father into Gij, reﬂecting the Mendelian
inheritance of genetic material from one’s parents. More precisely, let mi=kbei’s mother.
ThenGm
ijcould either be equal to Gm
kjorGp
kj, that is,i’s maternal allele is a copy of one of its
mother’s two alleles. Let Zm
ijbe a hidden variable than speciﬁes the choice. We can model this
using the following CPD, known as the inheritance model:
p(Gm
ij|Gmkj,Gp
kj,Zm
ij)={I(Gm
ij=Gm
kj)ifZm
ij=m
I(Gm
ij=Gp
kj)ifZm
ij=p(10.13)
We can deﬁne p(Gp
ij|Gm

ThenGm
kjorGp
p(Gm
kj,Zm
ij)={I(Gm
ij=Gm
kj)ifZm
ij=m
I(Gm
ij=Gp
kj)ifZm
ij|Gm
kj,Gp
kj,Zp
ij)similarly, where k=piisi’s father. The values of the Zij
are said to specify the phaseof the genotype. The values of Gp
i,j,Gm
i,j,Zp
i,jandZm
i,jconstitute
thehaplotype of person iat locusj.1
Next, we need to specify the prior for the root nodes, p(Gmij)andp(Gp
ij). This is called
thefounder model, and represents the overall prevalence of difference kinds of alleles in the
population. We usually assume independence between the loci for these founder alleles.

ij|Gm
kj,Gp
kj,Zp
i,j,Gm
i,j,Zp
i,jandZm
Finally, we need to specify priors for the switch variables that control the inheritance process.
These variables are spatially correlated, since adjacent sites on the genome are typically inherited
together (recombination events are rare). We can model this by imposing a two-state Markovchain on the Z’s, where the probability of switching state at locus jis given by θ
j=1
2(1−
e−2dj),w h e r edjis the distance between loci jandj+1. This is called the recombination
model.

j=1
2(1−
model.
The resulting DGM is shown in Figure 10.6(b): it is a series of replicated pedigree DAGs,
augmented with switching Zvariables, which are linked using Markov chains. (There is a
related model known as phylogenetic HMM (Siepel and Haussler 2003), which is used to model
evolution amongst phylogenies.)
As a simpliﬁed example of how this model can be used, suppose we only have one locus,
corresponding to blood type. For brevity, we will drop the jindex. Suppose we observe xi=A.

model.
Then there are 3 possible genotypes: Giis(A,A),(A,O)or(O,A). There is ambiguity
because the genotype to phenotype mapping is many-to-one. We want to reverse this mapping.This is known as an inverse problem. Fortunately, we can use the blood types of relatives to
help disambiguate the evidence. Information will “ﬂow” from the other x
i′’s up to their Gi′’s,
then across to i’sGivia the pedigree DAG. Thus we can combine our local evidence p(xi|Gi)

1. Sometimes the observed marker is equal to the unphased genotype, which is the unordered set {Gp
ij,Gm
ij}; however,
the phased or hidden genotype is not directly measurable.

318 Chapter 10. Directed graphical models (Bayes nets)
with an informative prior, p(Gi|x−i), conditioned on the other data, to get a less entropic local
posterior, p(Gi|x)∝p(xi|Gi)p(Gi|x−i).
In practice, the model is used to try to determine where along the genome a given disease-
causing gene is assumed to lie — this is the genetic linkage analysis task. The method works as
follows. First, suppose all the parameters of the model, including the distance between all themarker loci, are known. The only unknown is the location of the disease-causing gene. If thereareLmarker loci, we construct L+1models: in model ℓ, we postulate that the disease gene
comes after marker ℓ,f o r0<ℓ<L+1. We can estimate the Markov switching parameter ˆθ
ℓ,
and hence the distance dℓbetween the disease gene and its nearest known locus. We measure
the quality of that model using its likelihood, p(D|ˆθℓ). We then can then pick the model with
highest likelihood (which is equivalent to the MAP model under a uniform prior).

ℓ,
Note, however, that computing the likelihood requires marginalizing out all the hidden Z
andGvariables. See (Fishelson and Geiger 2002) and the references therein for some exact
methods for this task; these are based on the variable elimination algorithm, which we discussin Section 20.3. Unfortunately, for reasons we explain in Section 20.5, exact methods can becomputationally intractable if the number of individuals and/or loci is large. See (Albers et al.2006) for an approximate method for computing the likelihood; this is based on a form ofvariational inference, which we will discuss in Section 22.4.1.
10.2.5 Directed Gaussian graphical models *

Consider a DGM where all the variables are real-valued, and all the CPDs have the followingform:
p(x
t|xpa(t))=N(xt|μt+wT
txpa(t),σ2
t) (10.14)
This is called a linear Gaussian CPD. As we show below, multiplying all these CPDs together
results in a large joint Gaussian distribution of the form p(x)=N(x|μ,Σ). This is called a
directed GGM, or a Gaussian Bayes net.
We now explain how to derive μandΣfrom the CPD parameters, following (Shachter and
Kenley 1989, App. B). For convenience, we will rewrite the CPDs in the following form:
xt=μt+∑

p(x
t) (10.14)
xt=μt+∑
s∈pa(t)wts(xs−μs)+σtzt (10.15)
wherezt∼N(0,1),σtis the conditional standard deviation of xtgiven its parents, wtsis the
strength of the s→tedge, and μtis the local mean.2
It is easy to see that the global mean is just the concatenation of the local means, μ=
(μ1,...,μ D). We now derive the global covariance, Σ.L e tS≜diag(σ)be a diagonal matrix
containing the standard deviations. We can rewrite Equation 10.15 in matrix-vector form asfollows:
(x−μ)=W(x−μ)+Sz (10.16)
2. If we do not subtract off the parent’s mean (i.e., if we use xt=μt+∑

s∈pa(t)wtsxs+σtzt), the derivation of Σ
is much messier, as can be seen by looking at (Bishop 2006b, p370).

10.3. Inference 319
Now letebe a vector of noise terms:
e≜Sz (10.17)
We can rearrange this to get
e=(I−W)(x−μ) (10.18)
SinceWis lower triangular (because wts=0ift>sin the topological ordering), we have that
I−Wis lower triangular with 1s on the diagonal. Hence
⎛
⎜⎜⎜⎝e
1
e2
...
ed⎞
⎟⎟⎟⎠=⎛
⎜⎜⎜⎜⎜⎝1
−w
211
−w32−w311
......
−wd1−wd2...−wd,d−11⎞
⎟⎟⎟⎟⎟⎠⎛
⎜⎜⎜⎝x
1−μ1
x2−μ2
...
xd−μd⎞
⎟⎟⎟⎠(10.19)
SinceI−Wis always invertible, we can write
x−μ=(I−W)−1e≜Ue=USz (10.20)
where we deﬁned U=(I−W)−1. Thus the regression weights correspond to a Cholesky
decomposition of Σ,a sw en o ws h o w :
Σ=c o v [x]=c o v[x−μ] (10.21)
=c o v [USz]=UScov[z]SUT=US2UT(10.22)
10.3 Inference
We have seen that graphical models provide a compact way to deﬁne joint probability distribu-
tions. Given such a joint distribution, what can we do with it? The main use for such a jointdistribution is to perform probabilistic inference. This refers to the task of estimating unknown

quantities from known quantities. For example, in Section 10.2.2, we introduced HMMs, andsaid that one of the goals is to estimate the hidden states (e.g., words) from the observations(e.g., speech signal). And in Section 10.2.4, we discussed genetic linkage analysis, and said thatone of the goals is to estimate the likelihood of the data under various DAGs, corresponding todifferent hypotheses about the location of the disease-causing gene.

In general, we can pose the inference problem as follows. Suppose we have a set of correlated
random variables with joint distribution p(x
1:V|θ). (In this section, we are assuming the
parameters θof the model are known. We discuss how to learn the parameters in Section 10.4.)
Let us partition this vector into the visible variables xv, which are observed, and the hidden
variables, xh, which are unobserved. Inference refers to computing the posterior distribution
of the unknowns given the knowns:
p(xh|xv,θ)=p(xh,xv|θ)
p(xv|θ)=p(xh,xv|θ)∑
x′
hp(x′
h,xv|θ)(10.23)

x′
hp(x′
Essentially we are conditioning on the data by clamping the visible variables to their observed
values,xv, and then normalizing, to go from p(xh,xv)top(xh|xv). The normalization constant
p(xv|θ)is the likelihood of the data, also called the probability of the evidence.

320 Chapter 10. Directed graphical models (Bayes nets)
Sometimes only some of the hidden variables are of interest to us. So let us partition the
hidden variables into query variables, xq, whose value we wish to know, and the remaining
nuisance variables, xn, which we are not interested in. We can compute what we are interested
in bymarginalizing out the nuisance variables:
p(xq|xv,θ)=∑
xnp(xq,xn|xv,θ) (10.24)
In Section 4.3.1, we saw how to perform all these operations for a multivariate Gaussian in
O(V3)time, where Vis the number of variables. What if we have discrete random variables,
with say Kstates each? If the joint distribution is represented as a multi-dimensional table,
we can always perform these operations exactly, but this will take O(KV)time. In Chapter 20,
we explain how to exploit the factorization encoded by the GM to perform these operations inO(VK
w+1)time, where wis a quantity known as the treewidth of the graph. This measures

how “tree-like” the graph is. If the graph is a tree (or a chain), we have w=1, so for these
models, inference takes time linear in the number of nodes. Unfortunately, for more generalgraphs, exact inference can take time exponential in the number of nodes, as we explain inSection 20.5. We will therefore examine various approximate inference schemes later in thebook.
10.4 Learning
In the graphical models literature, it is common to distinguish between inference and learning.Inference means computing (functions of) p(x

h|xv,θ),w h e r evare the visible nodes, hare the
hidden nodes, and θare the parameters of the model, assumed to be known. Learning usually
means computing a MAP estimate of the parameters given data:
ˆθ=a r g m a x
θN∑
i=1logp(xi,v|θ)+logp(θ) (10.25)
wherexi,vare the visible variables in case i. If we have a uniform prior, p(θ)∝1, this reduces
to the MLE, as usual.
If we adopt a Bayesian view, the parameters are unknown variables and should also be

ˆθ=a r g m a x
θN∑
inferred. Thus to a Bayesian, there is no distinction between inference and learning. In fact, wecan just add the parameters as nodes to the graph, condition on D, and then infer the values
of all the nodes. (We discuss this in more detail below.)
In this view, the main difference between hidden variables and parameters is that the number

of hidden variables grows with the amount of training data (since there is usually a set of hiddenvariables for each observed data case), whereas the number of parameters in usually ﬁxed (atleast in a parametric model). This means that we must integrate out the hidden variables to avoidoverﬁtting, but we may be able to get away with point estimation techniques for parameters,which are fewer in number.
10.4.1 Plate notation

When inferring parameters from data, we often assume the data is iid. We can represent thisassumption explicitly using a graphical model, as shown in Figure 10.7(a). This illustrates the

10.4. Learning 321
θ
X1XN
Nθ
Xi
Figure 10.7 Left: data points xiare conditionally independent given θ. Right: Plate notation. This
represents the same model as the one on the left, except the repeated xinodes are inside a box, known as
a plate; the number in the lower right hand corner, N, speciﬁes the number of repetitions of the Xinode.
assumption that each data case was generated independently but from the same distribution.
Notice that the data cases are only independent conditional on the parameters θ; marginally,
the data cases are dependent. Nevertheless, we can see that, in this example, the order in whichthe data cases arrive makes no difference to our beliefs about θ, since all orderings will have
the same sufficient statistics. Hence we say the data is exchangeable.
To avoid visual clutter, it is common to use a form of syntactic sugar calledplates:w e

simply draw a little box around the repeated variables, with the convention that nodes withinthe box will get repeated when the model is unrolled. We often write the number of copies or
repetitions in the bottom right corner of the box. See Figure 10.7(b) for a simple example. Thecorresponding joint distribution has the form
p(θ,D)=p(θ)[
N∏
i=1p(xi|θ)]
(10.26)
This DGM represents the CI assumptions behind the models we considered in Chapter 5.
A slightly more complex example is shown in Figure 10.8. On the left we show a naive Bayes

N∏
(10.26)
classiﬁer that has been “unrolled” for Dfeatures, but uses a plate to represent repetition over
casesi=1:N. The version on the right shows the same model using nested plate notation.
When a variable is inside two plates, it will have two sub-indices. For example, we write θjc
to represent the parameter for feature jin class-conditional density c. Note that plates can
be nested or crossing. Notational devices for modeling more complex parameter tying patternscan be devised (e.g., (Heckerman et al. 2004)), but these are not widely used. What is not clearfrom the ﬁgure is that θ
jcis used to generate xijiffyi=c, otherwise it is ignored. This is an

example of context speciﬁc independence, since the CI relationship xij⊥θjconly holds if
yi̸=c.

322 Chapter 10. Directed graphical models (Bayes nets)
θc1 ... θcD
CXi1 ... XiDYi
Nπ
(a)DCN
θjcXijYiπ
(b)
Figure 10.8 Naive Bayes classiﬁer as a DGM. (a) With single plates. (b) WIth nested plates.
10.4.2 Learning from complete data
If all the variables are fully observed in each case, so there is no missing data and there are no
hidden variables, we say the data is complete. For a DGM with complete data, the likelihood is
given by
p(D|θ)=N∏
i=1p(xi|θ)=N∏
i=1V∏
t=1p(xit|xi,pa(t),θt)=V∏
t=1p(Dt|θt) (10.27)
whereDtis the data associated with node tand its parents, i.e., the t’th family. This is a
product of terms, one per CPD. We say that the likelihood decomposes according to the graph
Now suppose that the prior factorizes as well:
p(θ)=V∏
t=1p(θt) (10.28)
Then clearly the posterior also factorizes:
p(θ|D)∝p(D|θ)p(θ)=V∏
t=1p(Dt|θt)p(θt) (10.29)
This means we can compute the posterior of each CPD independently. In other words,
factored prior plus factored likelihood implies factored posterior (10.30)

p(θ)=V∏
Let us consider an example, where all CPDs are tabular, thus extending the earlier results of
Secion 3.5.1.2, where discussed Bayesian naive Bayes. We have a separate row (i.e., a separatemultinoulli distribution) for each conditioning case, i.e., for each combination of parent values,
as in Table 10.2. Formally, we can write the t’th CPT as x
t|xpa(t)=c∼Cat(θtc),w h e r e
θtck≜p(xt=k|xpa(t)=c),f o rk=1:Kt,c=1:Ctandt=1:T.H e r eKtis the number

10.4. Learning 323
of states for node t,Ct≜∏
s∈pa(t)Ksis the number of parent combinations, and Tis the
number of nodes. Obviously∑
kθtck=1for each row of each CPT.
Let us put a separate Dirichlet prior on each row of each CPT, i.e., θtc∼Dir(αtc). Then we
can compute the posterior by simply adding the pseudo counts to the empirical counts to get
θtc|D ∼Dir(N tc+αtc),w h e r eN tckis the number of times that node tis in state kwhile its
parents are in state c:
Ntck≜N∑
i=1I(xi,t=k,xi,pa(t)=c) (10.31)
From Equation 2.77, the mean of this distribution is given by the following:
θtck=Ntck+αtck∑
k′(Ntck′+αtck′)(10.32)
For example, consider the DGM in Figure 10.1(a). Suppose the training data consists of the
following 5 cases:
x1x2x3x4x5
0010001111
11010
0110001110
Below we list all the sufficient statistics N
tck, and the posterior mean parameters θickunder
a Dirichlet prior with αick=1(corresponding to add-one smoothing) for the t=4node:
x2x3Ntck=1Ntck=0θtck=1θtck=0
00 00 1/2 1/2
10 10 2/3 1/3
01 01 1/3 2/3

11010
11 21 3/5 2/5
It is easy to show that the MLE has the same form as Equation 10.32, except without the αtck
terms, i.e.,
ˆθtck=Ntck∑
k′Ntck′(10.33)
Of course, the MLE suffers from the zero-count problem discussed in Section 3.3.4.1, so it isimportant to use a prior to regularize the estimation problem.
10.4.3 Learning with missing and/or latent variables

If we have missing data and/or hidden variables, the likelihood no longer factorizes, and indeedit is no longer convex, as we explain in detail in Section 11.3. This means we will usually canonly compute a locally optimal ML or MAP estimate. Bayesian inference of the parameters iseven harder. We discuss suitable approximate inference techniques in later chapters.

324 Chapter 10. Directed graphical models (Bayes nets)
10.5 Conditional independence properties of DGMs
At the heart of any graphical model is a set of conditional indepence (CI) assumptions. We write
xA⊥GxB|xCifAis independent of BgivenCin the graph G, using the semantics to be
deﬁned below. Let I(G)be the set of all such CI statements encoded by the graph.
We say that Gis anI-map(independence map) for p, or that pisMarkovwrtG,i ff
I(G)⊆I(p),w h e r eI(p)is the set of all CI statements that hold for distribution p. In other
words, the graph is an I-map if it does not make any assertions of CI that are not true of thedistribution. This allows us to use the graph as a safe proxy for pwhen reasoning about p’s CI
properties. This is helpful for designing algorithms that work for large classes of distributions,regardless of their speciﬁc numerical parameters θ.
Note that the fully connected graph is an I-map of all distributions, since it makes no CI

assertions at all (since it is not missing any edges). We therefore say Gis aminimal I-map of
pifGis an I-map of p, and if there is no G
′⊆Gwhich is an I-map of p.
It remains to specify how to determine if xA⊥GxB|xC. Deriving these independencies
for undirected graphs is easy (see Section 19.2), but the DAG situation is somewhat complicated,because of the need to respect the orientation of the directed edges. We give the details below.
10.5.1 d-separation and the Bayes Ball algorithm (global Markov properties)

First, we introduce some deﬁnitions. We say an undirected path Pisd-separated b yas e to f
nodesE(containing the evidence) iff at least one of the following conditions hold:
1. P contains a chain, s→m→tors←m←t,w h e r em ∈E
2. P contains a tent or fork, s↙m↘t,w h e r em ∈E
3. P contains a colliderorv-structure, s↘m↙t,w h e r e mis not in Eand nor is any
descendant of m.
Next, we say that a set of nodes Ais d-separated from a different set of nodes Bgiven a
third observed set Eiff each undirected path from every node a∈Ato every node b∈Bis

d-separated by E. Finally, we deﬁne the CI properties of a DAG as follows:
xA⊥GxB|xE⇐⇒A is d-separated from B given E (10.34)
TheBayes ball algorithm (Shachter 1998) is a simple way to see if Ais d-separated from B
givenE, based on the above deﬁnition. The idea is this. We “shade” all nodes in E, indicating
that they are observed. We then place “balls” at each node in A, let them “bounce around”
according to some rules, and then ask if any of the balls reach any of the nodes in B. The three

main rules are shown in Figure 10.9. Notice that balls can travel opposite to edge directions.We see that a ball can pass through a chain, but not if it is shaded in the middle. Similarly, aball can pass through a fork, but not if it is shaded in the middle. However, a ball cannot passthrough a v-structure, unless it is shaded in the middle.
We can justify the 3 rules of Bayes ball as follows. First consider a chain structure X→Y→
Z, which encodes
p(x,y,z)=p(x)p(y|x)p(z|y) (10.35)

10.5. Conditional independence properties of DGMs 325
XYZ
(a)XY
Z
(b)
X
YZ
(c)XYZ
(d)
XY
Z
(e)X
YZ
(f)
Figure 10.9 Bayes ball rules. A shaded node is one we condition on. If there is an arrow hitting a bar, it
means the ball cannot pass through; otherwise the ball can pass through. Based on (Jordan 2007).
When we condition on y,a r exandzindependent? We have
p(x,z|y)=p(x)p(y|x)p(z|y)
p(y)=p(x,y)p(z|y)
p(y)=p(x|y)p(z|y) (10.36)
and therefore x⊥z|y. So observing the middle node of chain breaks it in two (as in a Markov
chain).
Now consider the tent structure X←Y→Z. The joint is
p(x,y,z)=p(y)p(x|y)p(z|y) (10.37)

326 Chapter 10. Directed graphical models (Bayes nets)
xy
(a)xy
(b)y′yxz
(c)
Figure 10.10 (a-b) Bayes ball boundary conditions. (c) Example of why we need boundary conditions. y′
is an observed child of y, rendering y“effectively observed”, so the ball bounces back up on its way from
xtoz.
p(x,z|y)=p(x,y,z)
p(y)=p(y)p(x|y)p(z|y)
p(y)=p(x|y)p(z|y) (10.38)
and therefore x⊥z|y. So observing a root node separates its children (as in a naive Bayes
classiﬁer: see Section 3.5).
Finally consider a v-structure X→Y←Z. The joint is
p(x,y,z)=p(x)p(z)p(y|x,z) (10.39)
p(x,z|y)=p(x)p(z)p(y|x,z)
p(y)(10.40)
sox̸⊥z|y. However, in the unconditional distribution, we have
p(x,z)=p(x)p(z) (10.41)
so we see that xandzare marginally independent. So we see that conditioning on a common
child at the bottom of a v-structure makes its parents become dependent. This important effect

is calledexplaining away, inter-causal reasoning,o r Berkson’s paradox. As an example of
explaining away, suppose we toss two coins, representing the binary numbers 0 and 1, and weobserve the “sum” of their values. A priori, the coins are independent, but once we observetheir sum, they become coupled (e.g., if the sum is 1, and the ﬁrst coin is 0, then we know thesecond coin is 1).
Finally, Bayes Ball also needs the “boundary conditions” shown in Figure 10.10(a-b). To
understand where these rules come from, consider Figure 10.10(c). Suppose Y

′is a noise-free
copy ofY. Then if we observe Y′, we effectively observe Yas well, so the parents XandZ
have to compete to explain this. So if we send a ball down X→Y→Y′, it should “bounce
back” up along Y′→Y→Z. However, if Yand all its children are hidden, the ball does not
bounce back.

10.5. Conditional independence properties of DGMs 327
12
354
67
Figure 10.11 A DGM.
For example, in Figure 10.11, we see that x2⊥x6|x5, since the 2→5→6path is blocked
byx5(which is observed), the 2→4→7→6path is blocked by x7(which is hidden), and
the2→1→3→6path is blocked by x1(which is hidden). However, we also see that
x2̸⊥x6|x5,x7, since now the 2→4→7→6path is no longer blocked by x7(which is
observed). Exercise 10.2 gives you some more practice in determining CI relationships for DGMs.
10.5.2 Other Markov properties of DGMs
From the d-separation criterion, one can conclude that
t⊥nd(t)\pa(t)|pa(t) (10.42)
where the non-descendants of a node nd(t)are all the nodes except for its descendants,
nd(t)=V\{ t∪desc(t)}. Equation 10.42 is called the directed local Markov property.F o r
example, in Figure 10.11, we have nd(3) ={2,4}, andpa(3) = 1,s o3 ⊥2,4|1.
A special case of this property is when we only look at predecessors of a node according to
some topological ordering. We have
t⊥pred(t)\pa(t)|pa(t) (10.43)

which follows since pred(t)⊆nd(t). This is called the ordered Markov property, which
justiﬁes Equation 10.7. For example, in Figure 10.11, if we use the ordering 1,2,...,7. we ﬁnd
pred(3) = {1,2}andpa(3) = 1,s o3 ⊥2|1.
We
have now described three Markov properties for DAGs: the directed global Markov property
G in Equation 10.34, the ordered Markov property O in Equation 10.43, and the directed local
Markov property L in Equation 10.42. It is obvious that G=⇒L=⇒O. What is less

We
obvious, but nevertheless true, is that O=⇒L=⇒G(see e.g., (Koller and Friedman 2009)
for the proof). Hence all these properties are equivalent.
Furthermore, any distribution pthat is Markov wrt Gcan be factorized as in Equation 10.7;
this is called the factorization property F. It is obvious that O=⇒F, but one can show that
the converse also holds (see e.g., (Koller and Friedman 2009) for the proof).
10.5.3 Markov blanket and full conditionals

The set of nodes that renders a node tconditionally independent of all the other nodes in
the graph is called t’sMarkov blanket; we will denote this by mb(t). One can show that the
Markov blanket of a node in a DGM is equal to the parents, the children, and the co-parents,

328 Chapter 10. Directed graphical models (Bayes nets)
i.e., other nodes who are also parents of its children:
mb(t)≜ch(t)∪pa(t)∪copa(t) (10.44)
For example, in Figure 10.11, we have
mb(5) = {6,7}∪{2,3}∪{4}={2,3,4,6,7} (10.45)
where 4 is a co-parent of 5 because they share a common child, namely 7.
To see why the co-parents are in the Markov blanket, note that when we derive p(xt|x−t)=
p(xt,x−t)/p(x−t), all the terms that do not involve xtwill cancel out between numerator and
denominator, so we are left with a product of CPDs which contain xtin theirscope. Hence
p(xt|x−t)∝p(xt|xpa(t))∏
s∈ch(t)p(xs|xpa(s)) (10.46)
For example, in Figure 10.11 we have
p(x5|x−5)∝p(x5|x2,x3)p(x6|x3,x5)p(x7|x4,x5,x6) (10.47)
The resulting expression is called t’sfull conditional, and will prove to be important when we
study Gibbs sampling (Section 24.2).
10.6 Inﬂuence (decision) diagrams *
We can represent multi-stage (Bayesian) decision problems by using a graphical notation known

as adecision diagram or aninﬂuence diagram (Howard and Matheson 1981; Kjaerulff and
Madsen 2008). This extends directed graphical models by adding decision nodes (also called ac-
tion nodes), represented by rectangles, and utility nodes (also called value nodes), represented
by diamonds. The original random variables are called chance nodes, and are represented by
ovals, as usual.
Figure 10.12(a) gives a simple example, illustrating the famous oil wild-catter problem.3In

this problem, you have to decide whether to drill an oil well or not. You have two possibleactions:d=1means drill, d=0means don’t drill. You assume there are 3 states of nature:
o=0means the well is dry, o=1means it is wet (has some oil), and o=2means it is
soaking (has a lot of oil). Suppose your prior beliefs are p(o)=[ 0.5,0.3,0.2]. Finally, you must
specify the utility function U(d,o). Since the states and actions are discrete, we can represent
it as a table (analogous to a CPT in a DGM). Suppose we use the following numbers, in dollars:

o=0o=1o=2
d=0000
d=1-70 50 200
We see that if you don’t drill, you incur no costs, but also make no money. If you drill a dry
well, you lose $70; if you drill a wet well, you gain $50; and if you drill a soaking well, you gain$200. Your prior expected utility if you drill is given by
EU(d=1 )= 2∑
o=0p(o)U(d,o)=0.5·(−70)+0.3·50+0.2·200 = 20 (10.48)
3. This example is originally from (Raiffa 1968). Our presentation is based on some notes by Daphne Koller.

10.6. Inﬂuence (decision) diagrams * 329
Oil
UtilityDrill
(a)Oil
UtilityDrillSound
(b)
Oil
UtilityDrillSoundTest
Cost
(c)
Figure 10.12 (a) Inﬂuence diagram for basic oil wild catter problem. (b) An extension in which we have
an information arc from the Sound chance node to the Drill decision node. (c) An extension in which we
get to decide whether to perform the test or not.
Your expected utility if you don’t drill is 0. So your maximum expected utility is
MEU=m a x{EU(d=0 ),EU(d=1 )}=m a x{0,20}=2 0 (10.49)
and therefore the optimal action is to drill:
d∗=a r gm a x {EU(d=0 ),EU(d=1 )}=1 (10.50)
Now let us consider a slight extension to the model. Suppose you perform a sounding to
estimate the state of the well. The sounding observation can be in one of 3 states: s=0is
a diffuse reﬂection pattern, suggesting no oil; s=1is an open reﬂection pattern, suggesting
some oil; and s=2is a closed reﬂection pattern, indicating lots of oil. Since Sis caused by O,

we add an O→Sarc to our model. In addition, we assume that the outcome of the sounding
test will be available before we decide whether to drill or not; hence we add an information
arcfromStoD. This is illustrated in Figure 10.12(b).
Let us model the reliability of our sensor using the following conditional distribution for
p(s|o):

330 Chapter 10. Directed graphical models (Bayes nets)
s=0s=1s=2
o=00.6 0.3 0.1
o=10.3 0.4 0.3
o=20.1 0.4 0.5
Suppose we do the sounding test and we observe s=0. The posterior over the oil state is
p(o|s=0 )=[ 0 .732,0.219,0.049] (10.51)
Now your posterior expected utility of performing action dis
EU(d|s=0 )=2∑
o=0p(o|s=0 )U(o,d) (10.52)
Ifd=1, this gives
EU(d=1|s=0 )=0 .732×(−70)+0.219×50+0.049×200 =−30.5 (10.53)
However, if d=0, thenEU(d=0|s=0 )=0 , since not drilling incurs no cost. So if we
observes=0, we are better off not drilling, which makes sense.
Now suppose we do the sounding test and we observe s=1. By similar reasoning, one
can show that EU(d=1|s=1 )=3 2 .9, which is higher than EU(d=0|s=1 )=0 .
Similarly, if we observe s=2,w eh a v e EU(d=1|s=2 )=8 7 .5which is much higher
thanEU(d=0|s=2 )=0 .Hence the optimal policy d∗(s)is as follows: if s=0, choose
d∗(0) = 0and get $0; if s=1, choosed∗(1) = 1and get $32.9; and if s=2, choosed∗(2) = 1
and get $87.5.

You can compute your expected proﬁt or maximum expected utility as follows:
MEU=∑
sp(s)EU(d∗(s)|s) (10.54)
This is the expected utility given possible outcomes of the sounding test, assuming you act
optimally given the outcome. The prior marginal on the outcome of the test is
p(s)=∑
op(o)p(s|o)=[ 0.41,0.35,0.24] (10.55)
Hence your maximum expected utility is
MEU=0.41×0+0.35×32.9+0.24×87.5=3 2.2 (10.56)
Now suppose you can choose whether to do the test or not. This can be modelled as shown

MEU=∑
p(s)=∑
in Figure 10.12(c), where we add a new test node T.I fT=1, we do the test, and Scan enter 1
of 3 states, determined by O, exactly as above. If T=0, we don’t do the test, and Senters a
special unknown state. There is also some cost associated with performing the test.
Is it worth doing the test? This depends on how much our MEU changes if we know the
outcome of the test (namely the state of S). If you don’t do the test, we have MEU=2 0

from Equation 10.49. If you do the test, you have MEU=3 2.2from Equation 10.56. So the
improvement in utility if you do the test (and act optimally on its outcome) is $12.2. This is

10.6. Inﬂuence (decision) diagrams * 331
at
zt zt+1xt xt+1
Rt
(a)at
xt xt+1
Rt
(b)
Figure 10.13 (a) A POMDP, shown as an inﬂuence diagram. ztare hidden world states. We implicitly
make the no forgetting assumption, which effectively means that athas arrows coming into it from all
previous observations, x1:t. (b) An MDP, shown as an inﬂuence diagram.
called the value of perfect information (VPI). So we should do the test as long as it costs less
than $12.2.
In terms of graphical models, the VPI of a variable Tcan be determined by computing the
MEU for the base inﬂuence diagram, I, and then computing the MEU for the same inﬂuence
diagram where we add information arcs from Tto the action nodes, and then computing the
difference. In other words,
VPI=MEU(I+T→D)−MEU(I) (10.57)
whereDis the decision node and Tis the variable we are measuring.
It is possible to modify the variable elimination algorithm (Section 20.3) so that it computes

the optimal policy given an inﬂuence diagram. These methods essentially work backwards from
the ﬁnal time-step, computing the optimal decision at each step assuming all following actionsare chosen optimally. See e.g., (Lauritzen and Nilsson 2001; Kjaerulff and Madsen 2008) fordetails.
We could continue to extend the model in various ways. For example, we could imagine a

dynamical system in which we test, observe outcomes, perform actions, move on to the nextoil well, and continue drilling (and polluting) in this way. In fact, many problems in robotics,business, medicine, public policy, etc. can be usefully formulated as inﬂuence diagrams unrolledover time (Raiffa 1968; Lauritzen and Nilsson 2001; Kjaerulff and Madsen 2008).
A generic model of this form is shown in Figure 10.13(a). This is known as a partially
observed Markov decision process orPOMDP(pronounced “pom-d-p”). This is basically a

hidden Markov model (Section 17.3) augmented with action and reward nodes. This can be usedto model the perception-action cycle that all intelligent agents use (see e.g., (Kaelbling et al.
1998) for details).
A special case of a POMDP, in which the states are fully observed, is called a Markov decision
processorMDP, shown in Figure 10.13(b). This is much easier to solve, since we only have
to compute a mapping from observed states to actions. This can be solved using dynamicprogramming (see e.g., (Sutton and Barto 1998) for details).
In the POMDP case, the information arc from x
ttoatis not sufficient to uniquely determine

332 Chapter 10. Directed graphical models (Bayes nets)
(a)AB CE FI DG H
(b)JGA D
H IE FB C
(c)
Figure 10.14 Some DGMs.
the best action, since the state is not fully observed. Instead, we need to choose actions based
on ourbelief state ,p(zt|x1:t,a1:t). Since the belief updating process is deterministic (see
Section 17.4.2), we can compute a belief state MDP . For details on to compute the policies for
such models, see e.g., (Kaelbling et al. 1998; Spaan and Vlassis 2005).
Exercises
Exercise 10.1 Marginalizing a node in a DGM
Consider the DAG Gin Figure 10.14(a). Assume it is a minimal I-map for p(A,B,C,D,E,F,X ).N o w
consider marginalizing out X. Construct a new DAG G′which is a minimal I-map for p(A,B,C,D,E,F ).
Specify (and justify) which extra edges need to be added.
Exercise 10.2 Bayes Ball
(Source: Jordan.)
Here we compute some global independence statements from some directed graphical models. You can

Exercises
use the “Bayes ball” algorithm, the d-separation criterion, or the method of converting to an undirected
graph (all should give the same results).
a. Consider the DAG in Figure 10.14(b). List all variables that are independent of Agiven evidence on B.
b. Consider the DAG in Figure 10.14(c). List all variables that are independent of Agiven evidence on J.
Exercise 10.3 Markov blanket for a DGM
Prove that the full conditional for node iin a DGM is given by
p(Xi|X−i)∝p(Xi|Pa(Xi))∏
Yj∈ch(Xi)p(Yj|Pa(Yj)) (10.58)

wherech(Xi)are the children of XiandPa(Yj)are the parents of Yj.
Exercise 10.4 Hidden variables in DGMs
Consider the DGMs in Figure 11.1 which both deﬁne p(X1:6), where we number empty nodes left to right,
top to bottom. The graph on the left deﬁnes the joint as
p(X1:6)=∑
hp(X1)p(X2)p(X3)p(H=h|X1:3)p(X4|H=h)p(X5|H=h)p(X6|H=h)(10.59)

10.6. Inﬂuence (decision) diagrams * 333
(a)
 (b)
Figure 10.15 (a) Weather BN. (b) Fishing BN.
where we have marginalized over the hidden variable H. The graph on the right deﬁnes the joint as
p(X1:6)=p(X1)p(X2)p(X3)p(X4|X1:3)p(X5|X1:4)p(X6|X1:5) (10.60)
a. (5 points) Assuming all nodes (including H) are binary and all CPDs are tabular, prove that the model
on the left has 17 free parameters.
b. (5 points) Assuming all nodes are binary and all CPDs are tabular, prove that the model on the right
has 59 free parameters.
c. (5 points) Suppose we have a data set D=Xn
1:6forn=1:N, where we observe the Xs but not H,
and we want to estimate the parameters of the CPDs using maximum likelihood. For which model is
this easier? Explain your answer.
Exercise 10.5 B a y e sn e t sf o rar a i n yd a y
(Source: Nando de Freitas.). In this question you must model a problem with 4 binary variables: G=”gray”,
V=”Vancouver”, R=”rain” and S=”sad”. Consider the directed graphical model describing the relation-

ship between these variables shown in Figure 10.15(a).
a. Write down an expression for P(S=1|V=1 )in terms of α,β,γ,δ.
b. Write down an expression for P(S=1|V=0 ). Is this the same or different to P(S=1|V=1 )?
Explain why.
c. Find maximum likelihood estimates of α,β,γusing the following data set, where each row is a training
case. (You may state your answers without proof.)
VGRS
1111
1101
1000(10.61)
Exercise 10.6 Fishing nets

VGRS
1111
1101
(Source: (Duda et al. 2001)..) Consider the Bayes net shown in Figure 10.15(b). Here, the nodes represent
the following variables
X1∈{winter, spring, summer, autumn },X2∈{salmon, sea bass } (10.62)
X3∈{light, medium, dark },X4∈{wide, thin } (10.63)

334 Chapter 10. Directed graphical models (Bayes nets)
X1 X2 X3 X4 X5Z1 Z2 Z3
(a)X1 X2 X4Z1 Z2 Z3
(b)
Figure 10.16 (a) A QMR-style network with some hidden leaves. (b) Removing the barren nodes.
The corresponding conditional probability tables are
p(x1)=(
.25.25.25.25)
,p(x2|x1)=⎛
⎜⎜⎝.9.1
.3.7
.4.6
.8.2⎞
⎟⎟⎠(10.64)
p(x3|x2)=(.33.33.34
.8.1.1)
,p(x4|x2)=(.4.6
.95.05)
(10.65)
Note that in p(x4|x2), the rows represent x2and the columns x4(so each row sums to one and represents
the child of the CPD). Thus p(x4=thin|x2=sea bass)=0 .05,p(x4=thin|x2=salmon)=0 .6, etc.
Answer the following queries. You may use matlab or do it by hand. In either case, show your work.
a. Suppose the ﬁsh was caught on December 20 — the end of autumn and the beginning of winter —
and thus let p(x1)=(.5,0,0,.5)instead of the above prior. (This is called soft evidence, since we
do not know the exact value of X1, but we have a distribution over it.) Suppose the lightness has not

been measured but it is known that the ﬁsh is thin. Classify the ﬁsh as salmon or sea bass.
b. Suppose all we know is that the ﬁsh is thin and medium lightness. What season is it now, most likely?
Usep(x1)=(
Exercise 10.7 Removing leaves in BN20 networks
a. Consider the QMR network, where only some of the symtpoms are observed. For example, in Fig-
ure 10.16(a), X4andX5are hidden. Show that we can safely remove all the hidden leaf nodes without

affecting the posterior over the disease nodes, i.e., prove that we can compute p(z1:3|x1,x2,x4)using
the network in Figure 10.16(b). This is called barren node removal, and can be applied to any DGM.
b. Now suppose we partition the leaves into three groups: on, off and unknown. Clearly we can remove the
unknown leaves, since they are hidden and do not affect their parents. Show that we can analytically
remove the leaves that are in the “off state”, by absorbing their effect into the prior of the parents.(This trick only works for noisy-OR CPDs.)

Exercise 10.8 Handling negative ﬁndings in the QMR network
Consider the QMR network. Let dbe the hidden diseases, f−be the negative ﬁndings (leaf nodes that are
off), andf−be the positive ﬁndings (leaf nodes that are on). We can compute the posterior p(d|f,f+)in
two steps: ﬁrst absorb the negative ﬁndings, p(d|f−)∝p(d)p(f−|d), then absorb the positive ﬁndings,
p(d|f−,f+)∝p(d|f−)p(f+|d). Show that the ﬁrst step can be done in O(|d||f−|)time, where |d|is
the number of dieases and |f−|is the number of negative ﬁndings. For simplicity, you can ignore leak

nodes. (Intuitively, the reason for this is that there is no correlation induced amongst the parents whenthe ﬁnding is off, since there is no explaining away.)

10.6. Inﬂuence (decision) diagrams * 335
Exercise 10.9 Moralization does not introduce new independence statements
Recall that the process of moralizing a DAG means connecting together all “unmarried” parents that share
a common child, and then dropping all the arrows. Let Mbe the moralization of DAG G. Show that
CI(M)⊆CI(G), where CI are the set of conditional independence statements implied by the model.

11 Mixture models and the EM algorithm
11.1 Latent variable models
In Chapter 10 we showed how graphical models can be used to deﬁne high-dimensional joint
probability distributions. The basic idea is to model dependence between two variables byadding an edge between them in the graph. (Technically the graph represents conditionalindependence, but you get the point.)
An alternative approach is to assume that the observed variables are correlated because they
arise from a hidden common “cause”. Model with hidden variables are also known as latent
variable models orLVMs. As we will see in this chapter, such models are harder to ﬁt than

models with no latent variables. However, they can have signiﬁcant advantages, for two mainreasons. First, LVMs often have fewer parameters than models that directly represent correlationin the visible space. This is illustrated in Figure 11.1. If all nodes (including H) are binary and allCPDs are tabular, the model on the left has 17 free parameters, whereas the model on the righthas 59 free parameters.
Second, the hidden variables in an LVM can serve as a bottleneck, which computes a

compressed representation of the data. This forms the basis of unsupervised learning, as wewill see. Figure 11.2 illustrates some generic LVM structures that can be used for this purpose.In general there are Llatent variables, z
i1,...,z IL, andDvisible variables, xi1,...,x iD,
where usually D≫L.I fw eh a v e L>1, there are many latent factors contributing to each
observation, so we have a many-to-many mapping. If L=1, we we only have a single latent

variable; in this case, ziis usually discrete, and we have a one-to-many mapping. We can
also have a many-to-one mapping, representing different competing factors or causes for eachobserved variable; such models form the basis of probabilistic matrix factorization, discussedin Section 27.6.2. Finally, we can have a one-to-one mapping, which can be represented asz
i→xi. By allowing ziand/orxito be vector-valued, this representation can subsume all the
others. Depending on the form of the likelihood p(xi|zi)and the prior p(zi), we can generate

a variety of different models, as summarized in Table 11.1.
11.2 Mixture models
The simplest form of LVM is when zi∈{1,...,K}, representing a discrete latent state. We will
use a discrete prior for this, p(zi)=C a t (π). For the likelihood, we use p(xi|zi=k)=pk(xi),

11.2. Mixture models 339
p(xi|zi) p(zi) Name Section
MVN Discrete Mixture of Gaussians 11.2.1
Prod. Discrete Discrete Mixture of multinomials 11.2.2
Prod. Gaussian Prod. Gaussian Factor analysis/ probabilistic PCA 12.1.5
Prod. Gaussian Prod. Laplace Probabilistic ICA/ sparse coding 12.6
Prod. Discrete Prod. Gaussian Multinomial PCA 27.2.3
Prod. Discrete Dirichlet Latent Dirichlet allocation 27.3
Prod. Noisy-OR Prod. Bernoulli BN20/ QMR 10.2.3
Prod. Bernoulli Prod. Bernoulli Sigmoid belief net 27.7
Table 11.1 Summary of some popular directed latent variable models. Here “Prod” means product, so
“Prod. Discrete” in the likelihood means a factored distribution of the form∏
jCat(xij|zi), and “Prod.
Gaussian”meansafactoreddistributionoftheform∏
jN(xij|zi). “PCA”standsfor“principalcomponents
analysis”. “ICA” stands for “indepedendent components analysis”.
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90.20.30.40.50.60.70.8
(a)
 (b)

(a)
 (b)
Figure 11.3 A mixture of 3 Gaussians in 2d. (a) We show the contours of constant probability for each
component in the mixture. (b) A surface plot of the overall density. Based on Figure 2.23 of (Bishop 2006a).
Figure generated by mixGaussPlotDemo .
11.2.1 Mixtures of Gaussians
The most widely used mixture model is the mixture of Gaussians (MOG), also called a Gaussian
mixture model orGMM. In this model, each base distribution in the mixture is a multivariate
Gaussian with mean μkand covariance matrix Σk. Thus the model has the form

p(xi|θ)=K∑
k=1πkN(xi|μk,Σk) (11.2)
Figure 11.3 shows a mixture of 3 Gaussians in 2D. Each mixture component is represented by a
different set of eliptical contours. Given a sufficiently large number of mixture components, a
GMM can be used to approximate any density deﬁned on RD.

340 Chapter 11. Mixture models and the EM algorithm
11.2.2 Mixture of multinoullis
We can use mixture models to deﬁne density models on many kinds of data. For example,
suppose our data consist of D-dimensional bit vectors. In this case, an appropriate class-
conditional density is a product of Bernoullis:
p(xi|zi=k,θ)=D∏
j=1Ber(xij|μjk)=D∏
j=1μxij
jk(1−μjk)1−xij(11.3)
whereμjkis the probability that bit jturns on in cluster k.
The latent variables do not have to any meaning, we might simply introduce latent variables
in order to make the model more powerful. For example, one can show (Exercise 11.8) that themean and covariance of the mixture distribution are given by
E[x]=∑
kπkμk (11.4)
cov[x]=∑
kπk[Σk+μkμT
k]−E[x]E[x]T(11.5)
whereΣk=d i a g (μjk(1−μjk)). So although the component distributions are factorized,
the joint distribution is not. Thus the mixture distribution can capture correlations between
variables, unlike a single product-of-Bernoullis model.
11.2.3 Using mixture models for clustering

E[x]=∑
cov[x]=∑
There are two main applications of mixture models. The ﬁrst is to use them as a black-box
density model, p(xi). This can be useful for a variety of tasks, such as data compression, outlier
detection, and creating generative classiﬁers, where we model each class-conditional densityp(x|y=c)by a mixture distribution (see Section 14.7.3).
The second, and more common, application of mixture models is to use them for clustering.

We discuss this topic in detail in Chapter 25, but the basic idea is simple. We ﬁrst ﬁt the mixturemodel, and then compute p(z
i=k|xi,θ), which represents the posterior probability that point
ibelongs to cluster k. This is known as the responsibility of cluster kfor point i, and can be
computed using Bayes rule as follows:
rik≜p(zi=k|xi,θ)=p(zi=k|θ)p(xi|zi=k,θ)∑K
k′=1p(zi=k′|θ)p(xi|zi=k′,θ)(11.6)
This procedure is called soft clustering, and is identical to the computations performed when

using a generative classiﬁer. The difference between the two models only arises at training time:
in the mixture case, we never observe zi, whereas with a generative classiﬁer, we do observe yi
(which plays the role of zi).
We can represent the amount of uncertainty in the cluster assignment by using 1−maxkrik.
Assuming this is small, it may be reasonable to compute a hard clustering using the MAP
estimate, given by
z∗
i=a r gm a x
krik=a r gm a x
klogp(xi|zi=k,θ)+logp(zi=k|θ) (11.7)

11.2. Mixture models 341
0 9.5 11.5 13.5 15.5 18.5 20.5−5−4−3−2−1012345
timegenesyeast microarray data
(a)K−Means centroids
(b)
Figure 11.4 (a) Some yeast gene expression data plotted as a time series. (c) Visualizing the 16 cluster
centers produced by K-means. Figure generated by kmeansYeastDemo .
0.12
 0.14
 0.12
 0.06
 0.13
0.07
 0.05
 0.15
 0.07
 0.09
Figure 11.5 We ﬁt a mixture of 10 Bernoullis to the binarized MNIST digit data. We show the MLE for the
corresponding cluster means, μk. The numbers on top of each image represent the mixing weights ˆπk.
No labels were used when training the model. Figure generated by mixBerMnistEM .
Hard clustering using a GMM is illustrated in Figure 1.8, where we cluster some data rep-
resenting the height and weight of people. The colors represent the hard assignments. Note
that the identity of the labels (colors) used is immaterial; we are free to rename all the clusters,
without affecting the partitioning of the data; this is called label switching .

Another example is shown in Figure 11.4. Here the data vectors xi∈R7represent the
expression levels of different genes at 7 different time points. We clustered them using a GMM.
We see that there are several kinds of genes, such as those whose expression level goes up
monotonically over time (in response to a given stimulus), those whose expression level goes
down monotonically, and those with more complex response patterns. We have clustered the
series into K=1 6groups. (See Section 11.5 for details on how to choose K.) For example, we

can represent each cluster by a prototype orcentroid. This is shown in Figure 11.4(b).
As an example of clustering binary data, consider a binarized version of the MNIST handwrit-
ten digit dataset (see Figure 1.5(a)), where we ignore the class labels. We can ﬁt a mixture of

342 Chapter 11. Mixture models and the EM algorithm
Bernoullis to this, using K=1 0, and then visualize the resulting centroids, ˆμk, as shown in
Figure 11.5. We see that the method correctly discovered some of the digit classes, but overall the
results aren’t great: it has created multiple clusters for some digits, and no clusters for others.There are several possible reasons for these “errors”:
• The model is very simple and does not capture the relevant visual characteristics of a digit.
For example, each pixel is treated independently, and there is no notion of shape or a stroke.
• Although we think there should be 10 clusters, some of the digits actually exhibit a fair degree
of visual variety. For example, there are two ways of writing 7’s (with and without the cross
bar). Figure 1.5(a) illustrates some of the range in writing styles. Thus we need K≫10
clusters to adequately model this data. However, if we set Kto be large, there is nothing

in the model or algorithm preventing the extra clusters from being used to create multipleversions of the same digit, and indeed this is what happens. We can use model selectionto prevent too many clusters from being chosen but what looks visually appealing and whatmakes a good density estimator may be quite different.
• The likelihood function is not convex, so we may be stuck in a local optimum, as we explain
in Section 11.3.2.
This example is typical of mixture modeling, and goes to show one must be very cautious

trying to “interpret” any clusters that are discovered by the method. (Adding a little bit ofsupervision, or using informative priors, can help a lot.)
11.2.4 Mixtures of experts
Section 14.7.3 described how to use mixture models in the context of generative classiﬁers. Wecan also use them to create discriminative models for classiﬁcation and regression. For example,consider the data in Figure 11.6(a). It seems like a good model would be three different linearregression functions, each applying to a different part of the input space. We can model this byallowing the mixing weights and the mixture densities to be input-dependent:
p(y
i|xi,zi=k,θ)=N (yi|wT
kxi,σ2
k) (11.8)
p(zi|xi,θ)=C a t ( zi|S(VTxi)) (11.9)
See Figure 11.7(a) for the DGM.

p(y
kxi,σ2
k) (11.8)
This model is called a mixture of experts or MoE (Jordan and Jacobs 1994). The idea is that
each submodel is considered to be an “expert” in a certain region of input space. The functionp(z
i=k|xi,θ)is called a gating function, and decides which expert to use, depending on
the input values. For example, Figure 11.6(b) shows how the three experts have “carved up” the1d input space, Figure 11.6(a) shows the predictions of each expert individually (in this case, theexperts are just linear regression models), and Figure 11.6(c) shows the overall prediction of themodel, obtained using
p(y
i|xi,θ)=∑
kp(zi=k|xi,θ)p(yi|xi,zi=k,θ) (11.10)
We discuss how to ﬁt this model in Section 11.4.3.

11.2. Mixture models 343
−1 −0.5 0 0.5 1−1.5−1−0.500.511.5expert predictions, fixed mixing weights=0
(a)−1 −0.5 0 0.5 100.10.20.30.40.50.60.70.80.91gating functions, fixed mixing weights=0
(b)
−1.5 −1 −0.5 0 0.5 1−2−1.5−1−0.500.511.5predicted mean and var, fixed mixing weights=0
(c)
Figure 11.6 (a) Some data ﬁt with three separate regression lines. (b) Gating functions for three different
“experts”. (c) The conditionally weighted average of the three expert predictions. Figure generated by
mixexpDemo .
yizixi
(a)yiz1
i
z2
ixi
(b)
Figure 11.7 (a) A mixture of experts. (b) A hierarchical mixture of experts.

344 Chapter 11. Mixture models and the EM algorithm
0 0.2 0.4 0.6 0.8 1−0.200.20.40.60.811.2forwards problem
(a)−0.2 0 0.2 0.4 0.6 0.8 1 1.2−0.200.20.40.60.811.2expert predictions
(b)
−0.2 0 0.2 0.4 0.6 0.8 1 1.200.10.20.30.40.50.60.70.80.91prediction
  
mean
mode
(c)
Figure 11.8 (a) Some data from a simple forwards model. (b) Some data from the inverse model, ﬁt
with a mixture of 3 linear regressions. Training points are color coded by their responsibilities. (c) The
predictive mean (red cross) and mode (black square). Based on Figures 5.20 and 5.21 of (Bishop 2006b).Figure generated by mixexpDemoOneToMany .
It should be clear that we can “plug in” any model for the expert. For example, we can use
neural networks (Chapter 16) to represent both the gating functions and the experts. The result
is known as a mixture density network. Such models are slower to train, but can be more
ﬂexible than mixtures of experts. See (Bishop 1994) for details.

It is also possible to make each expert be itself a mixture of experts. This gives rise to a
model known as the hierarchical mixture of experts. See Figure 11.7(b) for the DGM, and
Section 16.2.6 for further details.
11.2.4.1 Application to inverse problems
Mixtures of experts are useful in solving inverse problems. These are problems where we have
to invert a many-to-one mapping. A typical example is in robotics, where the location of theend effector (hand) yis uniquely determined by the joint angles of the motors, x. However,

for any given location y, there are many settings of the joints xthat can produce it. Thus the
inverse mapping x=f
−1(y)is not unique. Another example is kinematic tracking of people
from video (Bo et al. 2008), where the mapping from image appearance to pose is not unique,due to self occlusion, etc.

11.3. Parameter estimation for mixture models 345
θz
z1 zN
···
x1 xN
θx
(a)θz
Nzi
xi
θx
(b)
Figure 11.9 A LVM represented as a DGM. Left: Model is unrolled for Nexamples. Right: same model
using plate notation.
A simpler example, for illustration purposes, is shown in Figure 11.8(a). We see that this
deﬁnes a function, y=f(x), since for every value xalong the horizontal axis, there is a
unique response y. This is sometimes called the forwards model. Now consider the problem
of computing x=f−1(y). The corresponding inverse model is shown in Figure 11.8(b); this is
obtained by simply interchanging the xandyaxes. Now we see that for some values along
the horizontal axis, there are multiple possible outputs, so the inverse is not uniquely deﬁned.
For example, if y=0.8, thenxcould be 0.2 or 0.8. Consequently, the predictive distribution,
p(x|y,θ)is multimodal.
We can ﬁt a mixture of linear experts to this data. Figure 11.8(b) shows the prediction of each

expert, and Figure 11.8(c) shows (a plugin approximation to) the posterior predictive mode andmean. Note that the posterior mean does not yield good predictions. In fact, any model whichis trained to minimize mean squared error — even if the model is a ﬂexible nonlinear model,such as neural network — will work poorly on inverse problems such as this. However, theposterior mode, where the mode is input dependent, provides a reasonable approximation.
11.3 Parameter estimation for mixture models

We have seen how to compute the posterior over the hidden variables given the observedvariables, assuming the parameters are known. In this section, we discuss how to learn theparameters.
In Section 10.4.2, we showed that when we have complete data and a factored prior, the
posterior over the parameters also factorizes, making computation very simple. Unfortunatelythis is no longer true if we have hidden variables and/or missing data. The reason is apparentfrom looking at Figure 11.9. If the z

iwere observed, then by d-separation, we see that θz⊥θx|D,
and hence the posterior will factorize. But since, in an LVM, the ziare hidden, the parameters
are no longer independent, and the posterior does not factorize, making it much harder to

346 Chapter 11. Mixture models and the EM algorithm
−25 −20 −15 −10 −5 0 5 10 15 20 2505101520253035
(a)μ1μ2
−15.5 −10.5 −5.5 −0.5 4.5 9.5 14.5 19.5−15.5−10.5−5.5−0.54.59.514.519.5
(b)
Figure 11.10 Left:N= 200data points sampled from a mixture of 2 Gaussians in 1d, with πk=0.5,
σk=5,μ1=−10andμ2=1 0. Right: Likelihood surface p(D|μ1,μ2), with all other parameters set
to their true values. We see the two symmetric modes, reﬂecting the unidentiﬁability of the parameters.
Figure generated by mixGaussLikSurfaceDemo .
compute. This also complicates the computation of MAP and ML estimates, as we discus below.
11.3.1 Unidentiﬁability
The main problem with computing p(θ|D)for an LVM is that the posterior may have multiple
modes. To see why, consider a GMM. If the ziwere all observed, we would have a unimodal
posterior for the parameters:
p(θ|D) = Dir(π |D)K∏
k=1NIW(μk,Σk|D) (11.11)
Consequently we can easily ﬁnd the globally optimal MAP estimate (and hence globally optimal
MLE).

MLE).
But now suppose the zi’s are hidden. In this case, for each of the possible ways of “ﬁlling in”
thezi’s, we get a different unimodal likelihood. Thus when we marginalize out over the zi’s, we
get a multi-modal posterior for p(θ|D).1These modes correspond to different labelings of the
clusters. This is illustrated in Figure 11.10(b), where we plot the likelihood function, p(D|μ1,μ2),
for a 2D GMM with K=2for the data is shown in Figure 11.10(a). We see two peaks, one

MLE).
corresponding to the case where μ1=−10,μ2=1 0, and the other to the case where μ1=1 0,
μ2=−10. We say the parameters are not identiﬁable, since there is not a unique MLE.
Therefore there cannot be a unique MAP estimate (assuming the prior does not rule out certainlabelings), and hence the posterior must be multimodal. The question of how many modes there
1. Do not confuse multimodality of the parameter posterior, p(θ|D), with the multimodality deﬁned by the model,

p(x|θ). In the latter case, if we have Kclusters, we would expect to only get Kpeaks, although it is theoretically
possible to get more than K, at least if D>1(Carreira-Perpinan and Williams 2003).

11.3. Parameter estimation for mixture models 347
are in the parameter posterior is hard to answer. There are K!possible labelings, but some of
the peaks might get merged. Nevertheless, there can be an exponential number, since ﬁnding
the optimal MLE for a GMM is NP-hard (Aloise et al. 2009; Drineas et al. 2004).
Unidentiﬁability can cause a problem for Bayesian inference. For example, suppose we
draw some samples from the posterior, θ(s)∼p(θ|D), and then average them, to try to
approximate the posterior mean, θ=1
S∑S
s=1θ(s). (This kind of Monte Carlo approach is
explained in more detail in Chapter 24.) If the samples come from different modes, the average
will be meaningless. Note, however, that it is reasonable to average the posterior predictive
distributions, p(x)≈1
S∑S
s=1p(x|θ(s)), since the likelihood function is invariant to which
mode the parameters came from.
A variety of solutions have been proposed to the unidentiﬁability problem. These solutions

S∑S
S∑S
depend on the details of the model and the inference algorithm that is used. For example, see
(Stephens 2000) for an approach to handling unidentiﬁability in mixture models using MCMC.
The approach we will adopt in this chapter is much simpler: we just compute a single

S∑S
local mode, i.e., we perform approximate MAP estimation. (We say “approximate” since ﬁndingthe globally optimal MLE, and hence MAP estimate, is NP-hard, at least for mixture models(Aloise et al. 2009).) This is by far the most common approach, because of its simplicity. Itis also a reasonable approximation, at least if the sample size is sufficiently large. To see why,consider Figure 11.9(a). We see that there are Nlatent variables, each of which gets to “see”

one data point each. However, there are only two latent parameters, each of which gets toseeNdata points. So the posterior uncertainty about the parameters is typically much less
than the posterior uncertainty about the latent variables. This justiﬁes the common strategyof computing p(z
i|xi,ˆθ), but not bothering to compute p(θ|D). In Section 5.6, we will study
hierarchical Bayesian models, which essentially put structure on top of the parameters. In suchmodels, it is important to model p(θ|D), so that the parameters can send information between

themselves. If we used a point estimate, this would not be possible.
11.3.2 Computing a MAP estimate is non-convex
In the previous sections, we have argued, rather heuristically, that the likelihood function hasmultiple modes, and hence that ﬁnding an MAP or ML estimate will be hard. In this section, weshow this result by more algebraic means, which sheds some additional insight into the problem.Our presentation is based in part on (Rennie 2004).
Consider the log-likelihood for an LVM:
logp(D|θ)=∑
ilog[∑
zip(xi,zi|θ)]
(11.12)

ilog[∑
(11.12)
Unfortunately, this objective is hard to maximize. since we cannot push the log inside the sum.This precludes certain algebraic simplications, but does not prove the problem is hard.
Now suppose the joint probability distribution p(z
i,xi|θ)is in the exponential family, which
means it can be written as follows:
p(x,z|θ)=1
Z(θ)exp[θTφ(x,z)] (11.13)

348 Chapter 11. Mixture models and the EM algorithm
whereφ(x,z)are the sufficient statistics, and Z(θ)is the normalization constant (see Sec-
tion 9.2 for more details). It can be shown (Exercise 9.2) that the MVN is in the exponential
family, as are nearly all of the distributions we have encountered so far, including Dirichlet,multinomial, Gamma, Wishart, etc. (The Student distribution is a notable exception.) Further-more, mixtures of exponential families are also in the exponential family, providing the mixingindicator variables are observed (Exercise 11.11).
With this assumption, the complete data log likelihood can be written as follows:
ℓ
c(θ)=∑
ilogp(xi,zi|θ)=θT(∑
iφ(xi,zi))−NZ(θ) (11.14)
The ﬁrst term is clearly linear in θ. One can show that Z(θ)is a convex function (Boyd and
Vandenberghe 2004), so the overall objective is concave (due to the minus sign), and hence hasa unique maximum.
Now consider what happens when we have missing data. The observed data log likelihood
is given by
ℓ(θ)=∑
ilog∑

ℓ
c(θ)=∑
is given by
ℓ(θ)=∑
ilog∑
zip(xi,zi|θ)=∑
ilog[∑
zieθTφ(zi,xi)]
−NlogZ(θ) (11.15)
One can show that the log-sum-exp function is convex (Boyd and Vandenberghe 2004), and weknow that Z(θ)is convex. However, the difference of two convex functions is not, in general,
convex. So the objective is neither convex nor concave, and has local optima.
The disadvantage of non-convex functions is that it is usually hard to ﬁnd their global op-

is given by
ℓ(θ)=∑
ilog∑
ilog[∑
timum. Most optimization algorithms will only ﬁnd a local optimum; which one they ﬁnddepends on where they start. There are some algorithms, such as simulated annealing (Sec-tion 24.6.1) or genetic algorithms, that claim to always ﬁnd the global optimum, but this is onlyunder unrealistic assumptions (e.g., if they are allowed to be cooled “inﬁnitely slowly”, or al-lowed to run “inﬁnitely long”). In practice, we will run a local optimizer, perhaps using multiple
random restarts to increase out chance of ﬁnding a “good” local optimum. Of course, careful

initialization can help a lot, too. We give examples of how to do this on a case-by-case basis.
Note that a convex method for ﬁtting mixtures of Gaussians has been proposed. The idea
is to assign one cluster per data point, and select from amongst them, using a convex ℓ
1-type
penalty, rather than trying to optimize the locations of the cluster centers. See (Lashkari andGolland 2007) for details. This is essentially an unsupervised version of the approach used insparse kernel logistic regression, which we will discuss in Section 14.3.2. Note, however, that theℓ
1penalty, although convex, is not necessarily a good way to promote sparsity, as discussed in
Chapter 13. In fact, as we will see in that Chapter, some of the best sparsity-promoting methodsuse non-convex penalties, and use EM to optimie them! The moral of the story is: do not beafraid of non-convexity.
11.4 The EM algorithm

For many models in machine learning and statistics, computing the ML or MAP parameterestimate is easy provided we observe all the values of all the relevant random variables, i.e., if

11.4. The EM algorithm 349
Model Section
Mix. Gaussians 11.4.2
Mix. experts 11.4.3
Factor analysis 12.1.5
Student T 11.4.5
Probit regression 11.4.6
DGM with hidden variables 11.4.4
MVN with missing data 11.6.1
HMMs 17.5.2
Shrinkage estimates of Gaussian means Exercise 11.13
Table 11.2 Some models discussed in this book for which EM can be easily applied to ﬁnd the ML/ MAP
parameter estimate.
we have complete data. However, if we have missing data and/or latent variables, then computing
the ML/MAP estimate becomes hard.
One approach is to use a generic gradient-based optimizer to ﬁnd a local minimum of the
negative log likelihood orNLL, given by
NLL(θ)=−≜1
Nlogp(D|θ) (11.16)
However, we often have to enforce constraints, such as the fact that covariance matrices must bepositive deﬁnite, mixing weights must sum to one, etc., which can be tricky (see Exercise 11.5). Insuchcases, itis oftenmuchsimpler (butnotalways faster)to useanalgorithm called expectation

maximization,o r EMfor short (Dempster et al. 1977; Meng and van Dyk 1997; McLachlan and
Krishnan 1997). This is a simple iterative algorithm, often with closed-form updates at each step.Furthermore, the algorithm automatically enforce the required constraints.
EM exploits the fact that if the data were fully observed, then the ML/ MAP estimate would be

easy to compute. In particular, EM is an iterative algorithm which alternates between inferringthe missing values given the parameters (E step), and then optimizing the parameters given the“ﬁlled in” data (M step). We give the details below, followed by several examples. We end witha more theoretical discussion, where we put the algorithm in a larger context. See Table 11.2 fora summary of the applications of EM in this book.
11.4.1 Basic idea
Letxibe the visible or observed variables in case i, and let zibe the hidden or missing
variables. The goal is to maximize the log likelihood of the observed data:
ℓ(θ)=N∑
i=1logp(xi|θ)=N∑
i=1log[∑
(11.17)

ℓ(θ)=N∑
i=1log[∑
(11.17)
Unfortunately this is hard to optimize, since the log cannot be pushed inside the sum.

350 Chapter 11. Mixture models and the EM algorithm
EM gets around this problem as follows. Deﬁne the complete data log likelihood to be
ℓc(θ)≜N∑
i=1logp(xi,zi|θ) (11.18)
This cannot be computed, since ziis unknown. So let us deﬁne the expected complete data
log likelihood as follows:
Q(θ,θt−1)=E[
ℓc(θ)⏐⏐D,θt−1]
(11.19)
wheretis the current iteration number. Qis called the auxiliary function. The expectation
is taken wrt the old parameters, θt−1, and the observed data D. The goal of the Es t e pis to
compute Q(θ,θt−1), or rather, the terms inside of it which the MLE depends on; these are
known as the expected sufficient statistics or ESS. In the Ms t e p, we optimize the Q function
wrtθ:
θt=a r gm a x
θQ(θ,θt−1) (11.20)
To perform MAP estimation, we modify the M step as follows:
θt=a r g m a x
θQ(θ,θt−1)+logp(θ) (11.21)
The E step remains unchanged.
In Section 11.4.7 we show that the EM algorithm monotonically increases the log likelihood of

wrtθ:
θt=a r gm a x
θt=a r g m a x
the observed data (plus the log prior, if doing MAP estimation), or it stays the same. So if the
objective ever goes down, there must be a bug in our math or our code. (This is a surprisinglyuseful debugging tool!)
Below we explain how to perform the E and M steps for several simple models, that should
make things clearer.
11.4.2 EM for GMMs

θt=a r g m a x
In this section, we discuss how to ﬁt a mixture of Gaussians using EM. Fitting other kinds ofmixture models requires a straightforward modiﬁcation — see Exercise 11.3. We assume thenumber of mixture components, K, is known (see Section 11.5 for discussion of this point).

11.4. The EM algorithm 351
11.4.2.1 Auxiliary function
The expected complete data log likelihood is given by
Q(θ,θ(t−1))≜E[∑
ilogp(xi,zi|θ)]
(11.22)
=∑
iE[
log[K∏
k=1(πkp(xi|θk))I(zi=k)]]
(11.23)
=∑
i∑
kE[I(zi=k)]log[π kp(xi|θk)] (11.24)
=∑
i∑
kp(zi=k|xi,θt−1)log[π kp(xi|θk)] (11.25)
=∑
i∑
kriklogπk+∑
i∑
kriklogp(xi|θk) (11.26)
whererik≜p(zi=k|xi,θ(t−1))is theresponsibility that cluster ktakes for data point i.
This is computed in the E step, described below.
11.4.2.2 E step
The E step has the following simple form, which is the same for any mixture model:
rik=πkp(xi|θ(t−1)
k)
∑
k′πk′p(xi|θ(t−1)
k′)(11.27)
11.4.2.3 M step
In the M step, we optimize Qwrtπand theθk.F o rπ, we obviously have
πk=1
N∑
irik=rk
N(11.28)
whererk≜∑
irikis the weighted number of points assigned to cluster k.
To derive the M step for the μkandΣkterms, we look at the parts of Qthat depend on μk
andΣk. We see that the result is
ℓ(μk,Σk)=∑
k∑
iriklogp(xi|θk) (11.29)
=−1
2∑
irik[
log|Σk|+(xi−μk)TΣ−1
k(xi−μk)]
(11.30)

k)
∑
πk=1
N∑
irik=rk
N(11.28)
whererk≜∑
k∑
=−1
2∑
irik[
k(xi−μk)]
(11.30)
This is just a weighted version of the standard problem of computing the MLEs of an MVN (see
Section 4.1.3). One can show (Exercise 11.2) that the new parameter estimates are given by
μk=∑
irikxi
rk(11.31)
Σk=∑
irik(xi−μk)(xi−μk)T
rk=∑
irikxixT
i
rk−μkμT
k (11.32)

352 Chapter 11. Mixture models and the EM algorithm
These equations make intuitive sense: the mean of cluster kis just the weighted average of all
points assigned to cluster k, and the covariance is proportional to the weighted empirical scatter
matrix.
After computing the new estimates, we set θt=(πk,μk,Σk)fork=1:K, and go to the
next E step.
11.4.2.4 Example
An example of the algorithm in action is shown in Figure 11.11. We start with μ1=(−1,1),
Σ1=I,μ2=( 1,−1),Σ2=I. We color code points such that blue points come from cluster
1 and red points from cluster 2. More precisely, we set the color to
color(i)=ri1blue+ri2red (11.33)
so ambiguous points appear purple. After 20 iterations, the algorithm has converged on a good
clustering. (The data was standardized, by removing the mean and dividing by the standarddeviation, before processing. This often helps convergence.)
11.4.2.5 K-means algorithm
There is a popular variant of the EM algorithm for GMMs known as the K-means algorithm,

which we now discuss. Consider a GMM in which we make the following assumptions: Σ
k=
σ2IDis ﬁxed, and πk=1/Kis ﬁxed, so only the cluster centers, μk∈RD,h a v et ob e
estimated. Now consider the following delta-function approximation to the posterior computedduring the E step:
p(z
i=k|xi,θ)≈I(k=z∗
i) (11.34)
wherezi∗=a r g m a xkp(zi=k|xi,θ). This is sometimes called hard EM, since we are making
a hard assignment of points to clusters. Since we assumed an equal spherical covariance matrixfor each cluster, the most probable cluster for x

k=
p(z
i) (11.34)
ican be computed by ﬁnding the nearest
prototype:
z∗
i=a r gm i n
k||xi−μk||2
2 (11.35)
Hence in each E step, we must ﬁnd the Euclidean distance between Ndata points and Kcluster
centers, which takes O(NKD)time. However, this can be sped up using various techniques,
such as applying the triangle inequality to avoid some redundant computations (Elkan 2003).
Given the hard cluster assignments, the M step updates each cluster center by computing themean of all points assigned to it:
μ
k=1
Nk∑
i:zi=kxi (11.36)
See Algorithm 5 for the pseudo-code.

11.4. The EM algorithm 353
−2 0 2−202
(a)
 (b)
(c)
 (d)
(e)
 (f)
Figure 11.11 Illustration of the EM for a GMM applied to the Old Faithful data. (a) Initial (random) values
of the parameters. (b) Posterior responsibility of each point computed in the ﬁrst E step. The degree of
redness indicates the degree to which the point belongs to the red cluster, and similarly for blue; this
purple points have a roughly uniform posterior over clusters. (c) We show the updated parameters after
the ﬁrst M step. (d) After 3 iterations. (e) After 5 iterations. (f) After 16 iterations. Based on (Bishop 2006a)
Figure 9.8. Figure generated by mixGaussDemoFaithful .

354 Chapter 11. Mixture models and the EM algorithm
Algorithm 11.1: K-means algorithm
1initialize mk;
2repeat
3Assign each data point to its closest cluster center: zi=a r gm i n k||xi−μk||2
2;
4Update each cluster center by computing the mean of all points assigned to it:
μk=1
Nk∑
i:zi=kxi;
5until converged ;
K=2
50 100 150 200 250 30020
40
60
80
100
120
140
160
180
200
(a)
 (b)
Figure 11.12 An image compressed using vector quantization with a codebook of size K. (a)K=2. (b)
K=4. Figure generated by vqDemo.
11.4.2.6 Vector quantization
Since K-means is not a proper EM algorithm, it is not maximizing likelihood. Instead, it can be
interpreted as a greedy algorithm for approximately minimizing a loss function related to data
compression, as we now explain.
Suppose we want to perform lossy compression of some real-valued vectors, xi∈RD.Av e r y
simple approach to this is to use vector quantization orVQ. The basic idea is to replace each

real-valued vector xi∈RDwith a discrete symbol zi∈{1,...,K}, which is an index into a
codebook ofKprototypes, μk∈RD. Each data vector is encoded by using the index of the
most similar prototype, where similarity is measured in terms of Euclidean distance:
encode(xi) = argmin
k||xi−μk||2(11.37)
We can deﬁne a cost function that measures the quality of a codebook by computing the
reconstruction error ordistortion it induces:
J(μ,z|K,X)≜1
NN∑
i=1||xi−decode(encode(xi))||2=1
NN∑
i=1||xi−μzi||2(11.38)

NN∑
NN∑
where decode (k)=μk. The K-means algorithm can be thought of as a simple iterative scheme
for minimizing this objective.
Of course, we can achieve zero distortion if we assign one prototype to every data vector,
but that takes O(NDC)space, where Nis the number of real-valued data vectors, each of

11.4. The EM algorithm 355
lengthD, andCis the number of bits needed to represent a real-valued scalar (the quantization
accuracy). However, in many data sets, we see similar vectors repeatedly, so rather than storing
them many times, we can store them once and then create pointers to them. Hence we canreduce the space requirement to O(Nlog
2K+KDC): theO(Nlog2K)term arises because
each of the Ndata vectors needs to specify which of the Kcodewords it is using (the pointers);
and theO(KDC)term arises because we have to store each codebook entry, each of which is
aD-dimensional vector. Typically the ﬁrst term dominates the second, so we can approximate
therateof the encoding scheme (number of bits needed per object) as O(log2K), which is
typically much less than O(DC).
One application of VQ is to image compression. Consider the N= 200×320 = 64 ,000pixel
image in Figure 11.12; this is gray-scale, so D=1. If we use one byte to represent each pixel

(a gray-scale intensity of 0 to 255), then C=8, so we need NC= 512,000bits to represent
the image. For the compressed image, we need Nlog2K+KCbits. ForK=4, this is about
128kb, a factor of 4 compression. For K=8, this is about 192kb, a factor of 2.6 compression,

at negligible perceptual loss (see Figure 11.12(b)). Greater compression could be achieved if wemodelled spatial correlation between the pixels, e.g., if we encoded 5x5 blocks (as used by JPEG).This is because the residual errors (differences from the model’s predictions) would be smaller,and would take fewer bits to encode.
11.4.2.7 Initialization and avoiding local minima
Both K-means and EM need to be initialized. It is common to pick Kdata points at random, and

to make these be the initial cluster centers. Or we can pick the centers sequentially so as to tryto “cover” the data. That is, we pick the initial point uniformly at random. Then each subsequentpoint is picked from the remaining points with probability proportional to its squared distanceto the points’s closest cluster center. This is known as farthest point clustering (Gonzales 1985),
ork-means++ (Arthur and Vassilvitskii 2007; Bahmani et al. 2012). Surprisingly, this simple trick

can be shown to guarantee that the distortion is never more than O(logK)worse than optimal
(Arthur and Vassilvitskii 2007).
An heuristic that is commonly used in the speech recognition community is to incrementally

“grow” GMMs: we initially give each cluster a score based on its mixture weight; after eachround of training, we consider splitting the cluster with the highest score into two, with the newcentroids being random perturbations of the original centroid, and the new scores being half ofthe old scores. If a new cluster has too small a score, or too narrow a variance, it is removed.We continue in this way until the desired number of clusters is reached. See (Figueiredo andJain 2002) for a similar incremental approach.
11.4.2.8 MAP estimation
As usual, the MLE may overﬁt. The overﬁtting problem is particularly severe in the case ofGMMs. To understand the problem, suppose for simplicity that Σ
k=σ2

k=σ2
kI, and that K=2.I t
is possible to get an inﬁnite likelihood by assigning one of the centers, say μ2, to a single data
point, say x1, since then the 1st term makes the following contribution to the likelihood:
N(x1|μ2,σ2
2I)=1√
2πσ2
2e0(11.39)

356 Chapter 11. Mixture models and the EM algorithm
xp(x)
(a)10 20 30 40 50 60 70 80 90 10000.10.20.30.40.50.60.70.80.91
dimensionalityfraction of times EM for GMM fails
  
MLE
MAP
(b)
Figure 11.13 (a) Illustration of how singularities can arise in the likelihood function of GMMs. Based on
(Bishop 2006a) Figure 9.7. Figure generated by mixGaussSingularity . (b) Illustration of the beneﬁt of
MAP estimation vs ML estimation when ﬁtting a Gaussian mixture model. We plot the fraction of times
(out of 5 random trials) each method encounters numerical problems vs the dimensionality of the problem,forN=1 0 0samples. Solid red (upper curve): MLE. Dotted black (lower curve): MAP. Figure generated by
mixGaussMLvsMAP .
Hence we can drive this term to inﬁnity by letting σ2→0, as shown in Figure 11.13(a). We will
call this the “collapsing variance problem”.
An easy solution to this is to perform MAP estimation. The new auxiliary function is the
expected complete data log-likelihood plus the log prior:
Q′(θ,θold)=[∑
i∑

i∑
kriklogπik+∑
i∑
kriklogp(xi|θk)]
+logp(π)+∑
klogp(θk)(11.40)
Note that the E step remains unchanged, but the M step needs to be modiﬁed, as we now
explain.
For the prior on the mixture weights, it is natural to use a Dirichlet prior, π∼Dir(α), since
this is conjugate to the categorical distribution. The MAP estimate is given by
πk=rk+αk−1
N+∑
kαk−K(11.41)
If we use a uniform prior, αk=1, this reduces to Equation 11.28.
The prior on the parameters of the class conditional densities, p(θk), depends on the form of
the class conditional densities. We discuss the case of GMMs below, and leave MAP estimationfor mixtures of Bernoullis to Exercise 11.3.

explain.
N+∑
For simplicity, let us consider a conjugate prior of the form
p(μ
k,Σk) = NIW( μk,Σk|m0,κ0,ν0,S0) (11.42)

11.4. The EM algorithm 357
From Section 4.6.3, the MAP estimate is given by
ˆμk=rkxk+κ0m0
rk+κ0(11.43)
(11.44)
xk≜∑
irikxi
rk(11.45)
ˆΣk=S0+Sk+κ0rk
κ0+rk(xk−m0)(xk−m0)T
ν0+rk+D+2(11.46)
Sk≜∑
irik(xi−xk)(xi−xk)T(11.47)
We now illustrate the beneﬁts of using MAP estimation instead of ML estimation in the
context of GMMs. We apply EM to some synthetic data in Ddimensions, using either ML or
MAP estimation. We count the trial as a “failure” if there are numerical issues involving singular
matrices. For each dimensionality, we conduct 5 random trials. The results are illustrated inFigure 11.13(b) using N= 100. We see that as soon as Dbecomes even moderately large, ML
estimation crashes and burns, whereas MAP estimation never encounters numerical problems.
When using MAP estimation, we need to specify the hyper-parameters. Here we mention
some simple heuristics for setting them (Fraley and Raftery 2007, p163). We can set κ
0=0,

0=0,
so that the μkare unregularized, since the numerical problems only arise from Σk. In this
case, the MAP estimates simplify to ˆμk=xkandˆΣk=S0+Sk
ν0+rk+D+2, which is not quite so
scary-looking.
Now we discuss how to set S0. One possibility is to use
S0=1
K1/Ddiag(s2
1,...,s2D) (11.48)
wheresj=( 1/N)∑N
i=1(xij−xj)2is the pooled variance for dimension j. (The reason
for the1
K1/Dterm is that the resulting volume of each ellipsoid is then given by |S0|=
1
K|diag(s2
1,...,s2
D)|.) The parameter ν0controls how strongly we believe this prior. The

S0=1
for the1
1
K|diag(s2
1,...,s2
weakest prior we can use, while still being proper, is to set ν0=D+2, so this is a common
choice.
11.4.3 EM for mixture of experts
We can ﬁt a mixture of experts model using EM in a straightforward manner. The expected
complete data log likelihood is given by
Q(θ,θold)=N∑
i=1K∑
k=1riklog[πikN(yi|wT
kxi,σ2
k)] (11.49)
πi,k≜S(VTxi)k (11.50)
rik∝πold
ikN(yi|xT
iwold
k,(σold
k)2) (11.51)
So the E step is the same as in a standard mixture model, except we have to replace πkwith
πi,kwhen computing rik.

358 Chapter 11. Mixture models and the EM algorithm
In the M step, we need to maximize Q(θ,θold)wrtwk,σ2
kandV. For the regression
parameters for model k, the objective has the form
Q(θk,θold)=N∑
i=1rik{
−1
σ2
k(yi−wT
kxi)}
(11.52)
We recognize this as a weighted least squares problem, which makes intuitive sense: if rikis
small, then data point iwill be downweighted when estimating model k’s parameters. From
Section 8.3.4 we can immediately write down the MLE as
wk=(XTRkX)−1XTRky (11.53)
whereRk=d i a g (r:,k). The MLE for the variance is given by
σ2
k=∑N
i=1rik(yi−wT
kxi)2
∑Ni=1rik(11.54)
We replace the estimate of the unconditional mixing weights πwith the estimate of the gating
parameters, V. The objective has the form
ℓ(V)=∑
i∑
kriklogπi,k (11.55)
We recognize this as equivalent to the log-likelihood for multinomial logistic regression in
Equation 8.34, except we replace the “hard” 1-of-C encoding yiwith the “soft” 1-of-K encoding

σ2
k=∑N
kxi)2
ℓ(V)=∑
i∑
ri. Thus we can estimate Vby ﬁtting a logistic regression model to soft target labels.
11.4.4 EM for DGMs with hidden variables
We can generalize the ideas behind EM for mixtures of experts to compute the MLE or MAPestimate for an arbitrary DGM. We could use gradient-based methods (Binder et al. 1997), but itis much simpler to use EM (Lauritzen 1995): in the E step, we just estimate the hidden variables,and in the M step, we will compute the MLE using these ﬁlled-in values. We give the detailsbelow.

For simplicity of presentation, we will assume all CPDs are tabular. Based on Section 10.4.2,
let us write each CPT as follows:
p(x
it|xi,pa(t),θt)=Kpa(t )∏
c=1Kt∏
k=1θI(xit=i,xi,pa(t )=c)
tck(11.56)
The log-likelihood of the complete data is given by
logp(D|θ)=V∑
t=1Kpa(t )∑
c=1Kt∑
k=1Ntcklogθtck (11.57)
whereNtck=∑N
i=1I(xit=i,xi,pa(t)=c)are the empirical counts. Hence the expected
complete data log-likelihood has the form
E[logp(D|θ)] =∑
t∑
c∑
kNtcklogθtck (11.58)

11.4. The EM algorithm 359
where
Ntck=N∑
i=1E[
I(xit=i,xi,pa(t)=c)]
=∑
ip(xit=k,xi,pa(t)=c|Di) (11.59)
whereDiare all the visible variables in case i.
The quantity p(xit,xi,pa(t)|Di,θ)is known as a family marginal, and can be computed
using any GM inference algorithm. The Ntjkare the expected sufficient statistics, and constitute
the output of the E step.
Given these ESS, the M step has the simple form
k′Ntjk′(11.60)
This can be proved by adding Lagrange multipliers (to enforce the constraint∑
kθtjk=1)
to the expected complete data log likelihood, and then optimizing each parameter vector θtc
separately. We can modify this to perform MAP estimation with a Dirichlet prior by simply
adding pseudo counts to the expected counts.
11.4.5 EM for the Student distribution *

kθtjk=1)
One problem with the Gaussian distribution is that it is sensitive to outliers, since the log-probability only decays quadratically with distance from the center. A more robust alternative isthe Student t distribution, as discussed in Section ??.
Unlike the case of a Gaussian, there is no closed form formula for the MLE of a Student, even
if we have no missing data, so we must resort to iterative optimization methods. The easiestone to use is EM, since it automatically enforces the constraints that νis positive and that Σ

is symmetric positive deﬁnite. In addition, the resulting algorithm turns out to have a simpleintuitive form, as we see below.
At ﬁrst blush, it might not be apparent why EM can be used, since there is no missing data.
The key idea is to introduce an “artiﬁcial” hidden or auxiliary variable in order to simplify thealgorithm. In particular, we will exploit the fact that a Student distribution can be written as aGaussian scale mixture:
T(x
i|μ,Σ,ν)=∫
N(xi|μ,Σ/zi)Ga(zi|ν
2,ν
2)dzi (11.61)
(See Exercise 11.1 for a proof of this in the 1d case.) This can be thought of as an “inﬁnite”mixture of Gaussians, each one with a slightly different covariance matrix.
Treating the z
ias missing data, we can write the complete data log likelihood as

T(x
2,ν
ℓc(θ)=N∑
i=1[logN(xi|μ,Σ/zi)+logGa( zi|ν/2,ν/2)] (11.62)
=N∑
i=1[
−D
2log(2π)−1
2log|Σ|−zi
2δi+ν
2logν
2−logΓ(ν
2) (11.63)
+ν
2(logzi−zi)+(D
2−1)logzi]
(11.64)

360 Chapter 11. Mixture models and the EM algorithm
where we have deﬁned the Mahalanobis distance to be
δi=(xi−μ)TΣ−1(xi−μ) (11.65)
We can partition this into two terms, one involving μandΣ, and the other involving ν.W e
have, dropping irrelevant constants,
ℓc(θ)=L N(μ,Σ)+L G(ν) (11.66)
LN(μ,Σ)≜−1
2Nlog|Σ|−1
2N∑
i=1ziδi (11.67)
LG(ν)≜−NlogΓ(ν/2)+1
2Nνlog(ν/2)+1
2νN∑
i=1(logzi−zi) (11.68)
11.4.5.1 EM with νknown
Let us ﬁrst derive the algorithm with νassumed known, for simplicity. In this case, we can
ignore the LGterm, so we only need to ﬁgure out how to compute E[zi]wrt the old parameters.
From Section 4.6.2.2 we have
p(zi|xi,θ)=G a ( zi|ν+D
2,ν+δi
2) (11.69)
Now ifzi∼Ga(a,b), then E[zi]=a/b. Hence the E step at iteration tis
z(t)
i≜E[
zi|xi,θ(t)]
=ν(t)+D
ν(t)+δ(t)
i(11.70)
The M step is obtained by maximizing E[LN(μ,Σ)]to yield
ˆμ(t+1)=∑
iz(t)ixi∑
iz(t)
i(11.71)
ˆΣ(t+1)=1
N∑
iz(t)
i(xi−ˆμ(t+1))(xi−ˆμ(t+1))T(11.72)
=1
N[∑
iz(t)ixixT
i−(N∑
i=1z(t)
i)
ˆμ(t+1)(ˆμ(t+1))T]
(11.73)

2,ν+δi
2) (11.69)
z(t)
i≜E[
=ν(t)+D
ν(t)+δ(t)
i(11.70)
ˆμ(t+1)=∑
iz(t)ixi∑
iz(t)
i(11.71)
ˆΣ(t+1)=1
N∑
iz(t)
=1
N[∑
i−(N∑
i=1z(t)
i)
(11.73)
These results are quite intuitive: the quantity ziis the precision of measurement i,s oi fi ti s
small, the corresponding data point is down-weighted when estimating the mean and covariance.
This is how the Student achieves robustness to outliers.
11.4.5.2 EM with νunknown
To compute the MLE for the degrees of freedom, we ﬁrst need to compute the expectation ofL
G(ν), which involves ziandlogzi.N o wi fzi∼Ga(a,b), then one can show that
ℓ(t)
i≜E[
logzi|θ(t)]
=Ψ (a)−logb (11.74)

11.4. The EM algorithm 361
−5 −4 −3 −2 −1 0 1 2−7−6−5−4−3−2−1012314 errors using gauss (red=error)
Bankrupt
Solvent
(a)−5 −4 −3 −2 −1 0 1 2−7−6−5−4−3−2−10124 errors using student (red=error)
Bankrupt
Solvent
(b)
Figure 11.14 Mixture modeling on the bankruptcy data set. Left: Gaussian class conditional densities.
Right: Student class conditional densities. Points that belong to class 1 are shown as triangles, points that
belong to class 2 are shown as circles The estimated labels, based on the posterior probability of belongingto each mixture component, are computed. If these are incorrect, the point is colored red, otherwise it iscolored blue. (Training data is in black.) Figure generated by mixStudentBankruptcyDemo .
whereΨ(x)=d
dxlogΓ(x) is the digamma function. Hence, from Equation 11.69, we have
ℓ(t)
i=Ψ (ν(t)+D
2)−log(ν(t)+δ(t)
i
2) (11.75)
= log(z(t)
i)+Ψ(ν(t)+D
2)−log(ν(t)+D
2) (11.76)
Substituting into Equation 11.68, we have
E[LG(ν)] =−NlogΓ(ν/2)+Nν
2log(ν/2)+ν
2∑
i(ℓ(t)
i−z(t)
i) (11.77)

ℓ(t)
i
2) (11.75)
= log(z(t)
2) (11.76)
2∑
i(ℓ(t)
i−z(t)
i) (11.77)
The gradient of this expression is equal to
d
dνE[LG(ν)] =−N
2Ψ(ν/2)+N
2log(ν/2)+N
2+1
2∑
i(ℓ(t)
i−z(t)
i) (11.78)
This has a unique solution in the interval (0,+∞]which can be found using a 1d constrained
optimizer.
Performing a gradient-based optimization in the M step, rather than a closed-form update, is
an example of what is known as the generalized EM algorithm. One can show that EM will still
converge to a local optimum even if we only perform a “partial” improvement to the parameters
in the M step.
11.4.5.3 Mixtures of Student distributions
It is easy to extend the above methods to ﬁt a mixture of Student distributions. See Exercise 11.4for the details.

Let us consider a small example from (Lo 2009, ch3). We have a N=6 6,D=2data
set regarding the bankrupty patterns of certain companies. The ﬁrst feature speciﬁes the ratio

362 Chapter 11. Mixture models and the EM algorithm
of retained earnings (RE) to total assets, and the second feature speciﬁes the ratio of earnings
before interests and taxes (EBIT) to total assets. We ﬁt two models to this data, ignoring theclass labels: a mixture of 2 Gaussians, and a mixture of 2 Students. We then use each ﬁttedmodel to classify the data. We compute the most probable cluster membership and treat thisasˆy
i. We then compare ˆyito the true labels yiand compute an error rate. If this is more
than 50%, we permute the latent labels (i.e., we consider cluster 1 to represent class 2 and viceversa), and then recompute the error rate. Points which are misclassiﬁed are then shown in red.The result is shown in Figure 11.14. We see that the Student model made 4 errors, the Gaussianmodel made 21. This is because the class-conditional densities contain some extreme values,causing the Gaussian to be a poor choice.
11.4.6 EM for probit regression *

In Section 9.4.2, we described the latent variable interpretation of probit regression. Recall thatthis has the form p(y
i=1|zi)=I(zi>0),w h e r ezi∼N(wTxi,1)is latent. We now show
how to ﬁt this model using EM. (Although it is possible to ﬁt probit regression models usinggradient based methods, as shown in Section 9.4.1, this EM-based approach has the advantagethat it generalized to many other kinds of models, as we will see later on.)
The complete data log likelihood has the following form, assuming a N(0,V
0)prior onw:

ℓ(z,w|V0)=l o g p(y|z)+logN(z|Xw,I)+logN(w|0,V0) (11.79)
=∑
ilogp(yi|zi)−1
2(z−Xw)T(z−Xw)−1
2wTV−1
0w+const(11.80)
The posterior in the E step is a truncated Gaussian:
p(zi|yi,xi,w)={N(zi|wTxi,1)I(zi>0)ifyi=1
N(zi|wTxi,1)I(zi<0)ifyi=0(11.81)
In Equation 11.80, we see that wonly depends linearly on z, so we just need to compute
E[zi|yi,xi,w]. Exercise 11.15 asks you to show that the posterior mean is given by
E[zi|w,xi]={
μi+φ(μi)
1−Φ(−μi)=μi+φ(μi)
Φ(μi)ifyi=1
μi−φ(μi)
Φ(−μi)=μi−φ(μi)
1−Φ(μi)ifyi=0(11.82)
whereμi=wTxi.
In the M step, we estimate wusing ridge regression, where μ=E[z]is the output we are

μi+φ(μi)
μi−φ(μi)
trying to predict. Speciﬁcally, we have
ˆw=(V−1
0+XTX)−1XTμ (11.83)
The EM algorithm is simple, but can be much slower than direct gradient methods, as
illustrated in Figure 11.15. This is because the posterior entropy in the E step is quite high, sincewe only observe that zis positive or negative, but are given no information from the likelihood

μi−φ(μi)
ˆw=(V−1
about its magnitude. Using a stronger regularizer can help speed convergence, because itconstrains the range of plausible zvalues. In addition, one can use various speedup tricks, such
as data augmentation (van Dyk and Meng 2001), but we do not discuss that here.

11.4. The EM algorithm 363
0 20 40 60 80 100 120010203040506070
iterpenalized NLLprobit regression with L2 regularizer of 0.100
  
em
minfunc
Figure 11.15 Fitting a probit regression model in 2d using a quasi-Newton method or EM. Figure generated
byprobitRegDemo .
11.4.7 Theoretical basis for EM *
In this section, we show that EM monotonically increases the observed data log likelihood until
it reaches a local maximum (or saddle point, although such points are usually unstable). Ourderivation will also serve as the basis for various generalizations of EM that we will discuss later.
11.4.7.1 Expected complete data log likelihood is a lower bound
Consider an arbitrary distribution q(z
i)over the hidden variables. The observed data log
likelihood can be written as follows:
ℓ(θ)≜N∑
i=1log[∑
=N∑
i=1log[∑
ziq(zi)p(xi,zi|θ)
q(zi)]
(11.84)
Nowlog(u)is a concavefunction, so from Jensen’s inequality (Equation 2.113) we have the
following lower bound:
ℓ(θ)≥∑
i∑
ziqi(zi)logp(xi,zi|θ)
qi(zi)(11.85)

ℓ(θ)≜N∑
i=1log[∑
=N∑
i=1log[∑
q(zi)]
(11.84)
ℓ(θ)≥∑
i∑
Let us denote this lower bound as follows:
Q(θ,q)≜∑
iEqi[logp(xi,zi|θ)]+H(qi) (11.86)
where H(qi)is the entropy of qi.
The above argument holds for any positive distribution q. Which one should we choose?
Intuitively we should pick the qthat yields the tightest lower bound. The lower bound is a sum

364 Chapter 11. Mixture models and the EM algorithm
overiof terms of the following form:
L(θ,qi)=∑
qi(zi)(11.87)
=∑
ziqi(zi)logp(zi|xi,θ)p(xi|θ)
qi(zi)(11.88)
=∑
ziqi(zi)logp(zi|xi,θ)
qi(zi)+∑
ziqi(zi)logp(xi|θ) (11.89)
=−KL(qi(zi)||p(zi|xi,θ))+logp(xi|θ) (11.90)
Thep(xi|θ)term is independent of qi, so we can maximize the lower bound by setting qi(zi)=
p(zi|xi,θ). Of course, θis unknown, so instead we use qt
i(zi)=p(zi|xi,θt),w h e r eθtis our
estimate of the parameters at iteration t. This is the output of the E step.
Plugging this in to the lower bound we get
Q(θ,qt)=∑
iEqt
i[logp(xi,zi|θ)]+H(
qt
i)
(11.91)
We recognize the ﬁrst term as the expected complete data log likelihood. The second term is a
constant wrt θ. So the M step becomes
θt+1=a r gm a x
θQ(θ,θt) = argmax
θ∑
iEqt
i[logp(xi,zi|θ)] (11.92)
as usual.
Now comes the punchline. Since we used qt
i(zi)=p(zi|xi,θt), the KL divergence becomes
zero, soL(θt,qi)=l o gp(xi|θt), and hence
Q(θt,θt)=∑
ilogp(xi|θt)=ℓ(θt) (11.93)

Q(θ,qt)=∑
iEqt
qt
i)
(11.91)
θ∑
iEqt
as usual.
We see that the lower bound is tight after the E step. Since the lower bound “touches” thefunction, maximizing the lower bound will also “push up” on the function itself. That is, theM step is guaranteed to modify the parameters so as to increase the likelihood of the observeddata (unless it is already at a local maximum).
This process is sketched in Figure 11.16. The dashed red curve is the original function (the
observed data log-likelihood). The solid blue curve is the lower bound, evaluated at θ
t; this

t; this
touches the objective function at θt. We then set θt+1to the maximum of the lower bound
(blue curve), and ﬁt a new bound at that point (dotted green curve). The maximum of this newbound becomes θ
t+2, etc. (Compare this to Newton’s method in Figure 8.4(a), which repeatedly
ﬁts and then optimizes a quadratic approximation.)
11.4.7.2 EM monotonically increases the observed data log likelihood
We now prove that EM monotonically increases the observed data log likelihood until it reachesa local optimum. We have
ℓ(θ
t+1)≥Q(θt+1,θt)≥Q(θt,θt)=ℓ(θt) (11.94)

11.4. The EM algorithm 365
  
Q(θ,θt)
Q(θ,θt+1)
l(θ)
θtθt+1θt+2
Figure 11.16 Illustration of EM as a bound optimization algorithm. Based on Figure 9.14 of (Bishop 2006a).
Figure generated by emLogLikelihoodMax .
where the ﬁrst inequality follows since Q(θ,·)is a lower bound on ℓ(θ); the second inequality
follows since, by deﬁnition, Q(θt+1,θt) = max θQ(θ,θt)≥Q(θt,θt); and the ﬁnal equality
follows Equation 11.93.
As a consequence of this result, if you do not observe monotonic increase of the observed
data log likelihood, you must have an error in your math and/or code. (If you are performing
MAP estimation, you must add on the log prior term to the objective.) This is a surprisinglypowerful debugging tool.
11.4.8 Online EM
When dealing with large or streaming datasets, it is important to be able to learn online, aswe discussed in Section 8.5. There are two main approaches to online EM in the literature.
The ﬁrst approach, known as incremental EM (Neal and Hinton 1998), optimizes the lower
boundQ(θ,q

1,...,q N)oneqiat a time; however, this requires storing the expected sufficient
statistics for each data case. The second approach, known as stepwise EM (Sato and Ishii 2000;
Cappe and Mouline 2009; Cappe 2010), is based on stochastic approximation theory, and onlyrequires constant memory use. We explain both approaches in more detail below, following thepresentation of (Liang and Klein Liang and Klein).
11.4.8.1 Batch EM review

Before explaining online EM, we review batch EM in a more abstract setting. Let φ(x,z)be a
vector of sufficient statistics for a single data case. (For example, for a mixture of multinoullis,this would be the count vector a(j), which is the number of cluster jwas used in z, plus the
matrixB(j,v), which is of the number of times the hidden state was jand the observed letter
wasv.) Lets
i=∑
zp(z|xi,θ)φ(xi,z)be the expected sufficient statistics for case i, and
μ=∑N
i=1sibe the sum of the ESS. Given μ, we can derive an ML or MAP estimate of the

i=∑
μ=∑N
parameters in the M step; we will denote this operation by θ(μ). (For example, in the case of
mixtures of multinoullis, we just need to normalize aand each row of B.) With this notation
under our belt, the pseudo code for batch EM is as shown in Algorithm 8.

366 Chapter 11. Mixture models and the EM algorithm
Algorithm 11.2: Batch EM algorithm
1initialize μ;
2repeat
3μnew=0;
4foreach example i=1:Ndo
5 si:=∑
zp(z|xi,θ(μ))φ(xi,z);
6 μnew:=μnew+si;;
7μ:=μnew;
11.4.8.2 Incremental EM
In incremental EM (Neal and Hinton 1998), we keep track of μas well as the si. When we come
to a data case, we swap out the old siand replace it with the new snew
i, as shown in the code
in Algorithm 8. Note that we can exploit the sparsity of snewito speedup the computation of θ,
since most components of μwil not have changed.
Algorithm 11.3: Incremental EM algorithm
1initialize sifori=1:N;
2μ=∑
isi;
3repeat
4foreach example i=1:N in a random order do
5 snewi:=∑
6 μ:=μ+snewi−si;
7 si:=snewi;
This can be viewed as maximizing the lower bound Q(θ,q1,...,q N)by optimizing q1, then
θ, thenq2, thenθ, etc. As such, this method is guaranteed to monotonically converge to a local

2μ=∑
isi;
3repeat
5 snewi:=∑
maximum of the lower bound and to the log likelihood itself.
11.4.8.3 Stepwise EM
In stepwise EM, whenever we compute a new si,w em o v e μtowards it, as shown in Algorithm 7.2
At iteration k, the stepsize has value ηk, which must satisfy the Robbins-Monro conditions in
Equation 8.82. For example, (Liang and Klein Liang and Klein) use ηk=( 2+k)−κfor
0.5<κ≤1. We can get somewhat better behavior by using a minibatch of size mbefore
each update. It is possible to optimize mandκto maximize the training set likelihood, by

2. A detail: As written the update for μdoes not exploit the sparsity of si. We can ﬁx this by storing m=μ∏
j<k(1−ηj)
instead of μ, and then using the sparse update m:=m+ηk∏
j<k(1−ηj)si. This will not affect the results (i.e.,
θ(μ)=θ(m)), since scaling the counts by a global constant has no effect.

5 μ:= (1−ηk)μ+ηksi;
6 k:=k+1
7until converged ;
(Liang and Klein Liang and Klein) compare batch EM, incremental EM, and stepwise EM
on four different unsupervised language modeling tasks. They found that stepwise EM (usingκ≈0.7andm≈1000) was faster than incremental EM, and both were much faster than batch
EM. In terms of accuracy, stepwise EM was usually as good or sometimes even better than batchEM; incremental EM was often worse than either of the other methods.
11.4.9 Other EM variants *
EM is one of the most widely used algorithms in statistics and machine learning. Not surpris-ingly, many variations have been proposed. We brieﬂy mention a few below, some of which wewill use in later chapters. See (McLachlan and Krishnan 1997) for more information.
•Annealed EM In general, EM will only converge to a local maximum. To increase the chance
of ﬁnding the global maximum, we can use a variety of methods. One approach is to use
a method known as deterministic annealing (Rose 1998). The basic idea is to “smooth”

the posterior “landscape” by raising it to a temperature, and then gradually cooling it, all thewhile slowly tracking the global maximum. See Figure 11.17. for a sketch. (A stochastic version

368 Chapter 11. Mixture models and the EM algorithm
true log−likelihood
lower bound
training time
(a)true log−likelihood
(b)
Figure 11.18 Illustration of possible behaviors of variational EM. (a) The lower bound increases at each
iteration, and so does the likelihood. (b) The lower bound increases but the likelihood decreases. In
this case, the algorithm is closing the gap between the approximate and true posterior. This can have aregularizing effect. Based on Figure 6 of (Saul et al. 1996). Figure generated by varEMbound .
of this algorithm is described in Section 24.6.1.) An annealed version of EM is described in
(Ueda and Nakano 1998).
•Variational EM In Section 11.4.7, we showed that the optimal thing to do in the E step is to
makeqibe the exact posterior over the latent variables, qt
i(zi)=p(zi|xi,θt). In this case,

the lower bound on the log likelihood will be tight, so the M step will “push up” on thelog-likelihood itself. However, sometimes it is computationally intractable to perform exactinference in the E step, but we may be able to perform approximate inference. If we canensure that the E step is performing inference based onaal o w e rbound to the likelihood,then the M step can be seen as monotonically increasing this lower bound (see Figure 11.18).This is called variational EM (Neal and Hinton 1998). See Chapter 21 for some variational
inference methods that can be used in the E step.
•Monte Carlo EM Another approach to handling an intractable E step is to use a Monte

Carlo approximation to the expected sufficient statistics. That is, we draw samples from theposterior, z
s
i∼p(zi|xi,θt), and then compute the sufficient statistics for each completed
vector,(xi,zsi), and then average the results. This is called Monte Carlo EM orMCEM(Wei
and Tanner 1990). (If we only draw a single sample, it is called stochastic EM (Celeux and
Diebolt 1985).) One way to draw samples is to use MCMC (see Chapter 24). However, if we
have to wait for MCMC to converge inside each E step, the method becomes very slow. Analternative is to use stochastic approximation, and only perform “brief” sampling in the Estep, followed by a partial parameter update. This is called stochastic approximation EM

(Delyon et al. 1999) and tends to work better than MCEM. Another alternative is to applyMCMC to infer the parameters as well as the latent variables (a fully Bayesian approach), thuseliminating the distinction between E and M steps. See Chapter 24 for details.
•Generalized EM Sometimes we can perform the E step exactly, but we cannot perform the

M step exactly. However, we can still monotonically increase the log likelihood by performinga “partial” M step, in which we merely increase the expected complete data log likelihood,rather than maximizing it. For example, we might follow a few gradient steps. This is called

11.4. The EM algorithm 369
0 2 4 6 8 10 12 14 16 18−42−41.5−41−40.5−40−39.5−39−38.5
iterationsloglikK=5, D=15, N=5000
  
EM (1.080)
OR(1) (1.358)
OR(1.25) (1.141)
OR(2) (1.219)
OR(5) (1.433)
(a)0 2 4 6 8 10 12 14 16 18−42−41−40−39−38−37−36
  
EM (1.315)
OR(1) (1.368)
OR(1.25) (1.381)
OR(2) (1.540)
OR(5) (1.474)
(b)
Figure 11.19 Illustration of adaptive over-relaxed EM applied to a mixture of 5 Gaussians in 15 dimensions.
We show the algorithm applied to two different datasets, randomly sampled from a mixture of 10 Gaussians.
We plot the convergence for different update rates η. Usingη=1gives the same results as regular EM.
The actual running time is printed in the legend. Figure generated by mixGaussOverRelaxedEmDemo .
thegeneralized EM orGEMalgorithm. (This is an unfortunate term, since there are many
ways to generalize EM....)
•ECM(E) algorithm TheECMalgorithm stands for “expectation conditional maximization”,

and refers to optimizing the parameters in the M step sequentially, if they turn out to be
dependent. The ECMEalgorithm, which stands for “ECM either” (Liu and Rubin 1995), is
a variant of ECM in which we maximize the expected complete data log likelihood (the Q

function) as usual, or the observed data log likelihood, during one or more of the conditionalmaximization steps. The latter can be much faster, since it ignores the results of the E step,and directly optimizes the objective of interest. A standard example of this is when ﬁttingthe Student T distribution. For ﬁxed ν, we can update Σas usual, but then to update ν,
we replace the standard update of the form ν
t+1= argmax νQ((μt+1,Σt+1,ν),θt)with
νt+1= argmax νlogp(D|μt+1,Σt+1,ν). See (McLachlan and Krishnan 1997) for more

•Over-relaxed EM Vanilla EM can be quite slow, especially if there is lots of missing data. The
adaptiveoverrelaxed EM algorithm (Salakhutdinov and Roweis 2003) performs an update
of the form θt+1=θt+η(M(θt)−θt),w h e r eηis a step-size parameter, and M(θt)is
the usual update computed during the M step. Obviously this reduces to standard EM ifη=1, but using larger values of ηcan result in faster convergence. See Figure 11.19 for an
illustration. Unfortunately, using too large a value of ηcan cause the algorithm to fail to
converge.
Finally, note that EM is in fact just a special case of a larger class of algorithms known as
bound optimization orMMalgorithms (MM stands for minorize-maximize). See (Hunter and
Lange 2004) for further discussion.

370 Chapter 11. Mixture models and the EM algorithm
11.5 Model selection for latent variable models
When using LVMs, we must specify the number of latent variables, which controls the model
complexity. In particuarl, in the case of mixture models, we must specify K, the number
of clusters. Choosing these parameters is an example of model selection. We discuss someapproaches below.
11.5.1 Model selection for probabilistic models
The optimal Bayesian approach, discussed in Section 5.3, is to pick the model with the largestmarginal likelihood, K
∗=a r g m a xkp(D|K).
There are two problems with this. First, evaluating the marginal likelihood for LVMs is
quite difficult. In practice, simple approximations, such as BIC, can be used (see e.g., (Fraleyand Raftery 2002)). Alternatively, we can use the cross-validated likelihood as a performancemeasure, although this can be slow, since it requires ﬁtting each model Ftimes, where Fis the
number of CV folds.

The second issue is the need to search over a potentially large number of models. The usual
approach is to perform exhaustive search over all candidate values of K. However, sometimes
we can set the model to its maximal size, and then rely on the power of the Bayesian Occam’srazor to “kill off” unwanted components. An example of this will be shown in Section 21.6.1.6,when we discuss variational Bayes.
An alternative approach is to perform stochastic sampling in the space of models. Traditional

approaches, such as (Green 1998, 2003; Lunn et al. 2009), are based on reversible jump MCMC,and use birth moves to propose new centers, and death moves to kill off old centers. However,this can be slow and difficult to implement. A simpler approach is to use a Dirichlet processmixture model, which can be ﬁt using Gibbs sampling, but still allows for an unbounded numberof mixture components; see Section 25.2 for details.
Perhaps surprisingly, these sampling-based methods can be faster than the simple approach

of evaluating the quality of each Kseparately. The reason is that ﬁtting the model for each
Kis often slow. By contrast, the sampling methods can often quickly determine that a certain
value ofKis poor, and thus they need not waste time in that part of the posterior.
11.5.2 Model selection for non-probabilistic methods
What if we are not using a probabilistic model? For example, how do we choose Kfor theK-
means algorithm? Since this does not correspond to a probability model, there is no likelihood,so none of the methods described above can be used.
An obvious proxy for the likelihood is the reconstruction error. Deﬁne the squared recon-
struction error of a data set D, using model complexity K, as follows:
E(D,K)=1
|D|∑
i∈D||xi−ˆxi||2(11.95)
In the case of K-means, the reconstruction is given by ˆxi=μzi,w h e r ezi=a r g m i nk||xi−
μk||2
2, as explained in Section 11.4.2.6.

E(D,K)=1
|D|∑
μk||2
Figure 11.20(a) plots the reconstruction error on the test setfor K-means. We notice that the
error decreases with increasing model complexity! The reason for this behavior is as follows:

11.5. Model selection for latent variable models 371
2 4 6 8 10 12 14 1600.050.10.150.20.25MSE on test vs K for K−means
(a)2 4 6 8 10 12 14 1611951200120512101215122012251230123512401245NLL on test set vs K for GMM
(b)
Figure 11.20 Test set performance vs Kfor data generated from a mixture of 3 Gaussians in 1d (data is
shown in Figure 11.21(a)). (a) MSE on test set for K-means. (b) Negative log likelihood on test set for GMM.
Figure generated by kmeansModelSel1d .
−3 −2 −1 0 1 2 30102030405060Xtrain
(a)−1 0 100.20.40.60.81K=2, mse=0.2023
−2 0 200.20.40.60.81K=3, mse=0.0818
−2 0 200.20.40.60.81K=4, mse=0.0562
−2 0 200.20.40.60.81K=5, mse=0.0368
−2 0 200.20.40.60.81K=6, mse=0.0275
−2 0 200.20.40.60.81K=10, mse=0.0111
(b)
−2 0 200.10.20.30.40.5K=2, nll=1244.7882
−2 0 200.10.20.30.40.5K=3, nll=1198.9738
−2 0 200.10.20.30.40.5K=4, nll=1196.9937
−2 0 200.10.20.30.40.5K=5, nll=1202.5869
−2 0 200.10.20.30.40.5K=6, nll=1199.5574
−2 0 200.10.20.30.40.5K=10, nll=1203.2931
(c)

(b)
(c)
Figure 11.21 Synthetic data generated from a mixture of 3 Gaussians in 1d. (a) Histogram of training data.
(Test data looks essentially the same.) (b) Centroids estimated by K-means for K∈{2,3,4,5,6,10}.
(c) GMM density model estimated by EM for for the same values of K. Figure generated by
kmeansModelSel1d .

372 Chapter 11. Mixture models and the EM algorithm
when we add more and more centroids to K-means, we can “tile” the space more densely, as
shown in Figure 11.21(b). Hence any given test vector is more likely to ﬁnd a close prototype to
accurately represent it as Kincreases, thus decreasing reconstruction error. However, if we use
a probabilistic model, such as the GMM, and plot the negative log-likelihood, we get the usualU-shaped curve on the test set, as shown in Figure 11.20(b).
In supervised learning, we can always use cross validation to select between non-probabilistic
models of different complexity, but this is not the case with unsupervised learning. Althoughthis is not a novel observation (e.g., it is mentioned in passing in (Hastie et al. 2009, p519), oneof the standard references in this ﬁeld), it is perhaps not as widely appreciated as it should be.In fact, it is one of the more compelling arguments in favor of probabilistic models.

Given that cross validation doesn’t work, and supposing one is unwilling to use probabilistic
models (for some bizarre reason...), how can one choose K? The most common approach is to
plot the reconstruction error on the training set versus K, and to try to identify a kneeorkink
in the curve. The idea is that for K<K
∗,w h e r eK∗is the “true” number of clusters, the rate
of decrease in the error function will be high, since we are splitting apart things that shouldnot be grouped together. However, for K>K

∗, we are splitting apart “natural” clusters, which
does not reduce the error by as much.
This kink-ﬁnding process can be automated by use of the gap statistic (Tibshirani et al.
2001). Nevertheless, identifying such kinks can be hard, as shown in Figure 11.20(a), since theloss function usually drops off gradually. A different approach to “kink ﬁnding” is described inSection 12.3.2.1.
11.6 Fitting models with missing data

Suppose we want to ﬁt a joint density model by maximum likelihood, but we have “holes” in ourdata matrix, due to missing data (usually represented by NaNs). More formally, let O
ij=1if
component jof data case iis observed, and let Oij=0otherwise. Let Xv={xij:Oij=1}
be the visible data, and Xh={xij:Oij=0}be the missing or hidden data. Our goal is to
compute
ˆθ=a r g m a x
θp(Xv|θ,O) (11.96)
Under the missing at random assumption (see Section 8.6.2), we have
p(Xv|θ,O)=N∏
i=1p(xiv|θ) (11.97)
wherexivis a vector created from row iand the columns indexed by the set {j:Oij=1}.

ij=1if
compute
ˆθ=a r g m a x
Hence the log-likelihood has the form
logp(Xv|θ)=∑
ilogp(xiv|θ) (11.98)
where
p(xiv|θ)=∑
xihp(xiv,xih|θ) (11.99)

11.6. Fitting models with missing data 373
andxihis the vector of hidden variables for case i(assumed discrete for notational simplicity).
Substituting in, we get
ilog[∑
xihp(xiv,xih|θ)]
(11.100)
Unfortunately, this objective is hard to maximize. since we cannot push the log inside the sum.
However, we can use the EM algorithm to compute a local optimum. We give an example ofthis below.
11.6.1 EM for the MLE of an MVN with missing data
Suppose we want to ﬁt an MVN by maximum likelihood, but we have missing data. We can useEM to ﬁnd a local maximum of the objective, as we explain below.
11.6.1.1 Getting started
To get the algorithm started, we can compute the MLE based on those rows of the data ma-trix that are fully observed. If there are no such rows, we can use some ad-hoc imputationprocedures, and then compute an initial MLE.
11.6.1.2 E step
Once we have θ
t−1, we can compute the expected complete data log likelihood at iteration tas
follows:
Q(θ,θt−1)=E[N∑
i=1logN(xi|μ,Σ)|D,θt−1]
(11.101)
=−N

follows:
(11.101)
=−N
2log|2πΣ|−1
2∑
iE[
(xi−μ)TΣ−1(xi−μ)]
(11.102)
=−N
2tr(Σ−1∑
iE[
(xi−μ)(xi−μ)T]
(11.103)
=−N
2log|Σ|−ND
2tr(Σ−1E[S(μ)]) (11.104)
where
E[S(μ)]≜∑
i(
E[
xixT
i]
+μμT−2μE[xi]T)
(11.105)
(We drop the conditioning of the expectation on Dandθt−1for brevity.) We see that we need
to compute∑
iE[xi]and∑
iE[
xixT
i]
; these are the expected sufficient statistics.
To compute these quantities, we use the results from Section 4.3.1. Speciﬁcally, consider case
i, where components vare observed and components hare unobserved. We have
xih|xiv,θ∼N(mi,Vi) (11.106)
mi≜μh+ΣhvΣ−1

(11.103)
=−N
where
E[S(μ)]≜∑
i(
E[
xixT
i]
(11.105)
iE[
xixT
i]
vv(xiv−μv) (11.107)
Vi≜Σhh−ΣhvΣ−1
vvΣvh (11.108)

374 Chapter 11. Mixture models and the EM algorithm
Hence the expected sufficient statistics are
E[xi]=(E[xih];xiv)=(mi;xiv) (11.109)
where we have assumed (without loss of generality) that the unobserved variables come before
the observed variables in the node ordering.
To compute E[
xixT
i]
, we use the result that cov[x]=E[
xxT]
−E[x]E[
xT]
. Hence
E[
xixT
i]
=E[(xih
xiv)(
xT
ihxT
iv)]
=(E[
xihxT
ih]
E[xih]xT
iv
xivE[xih]TxivxTiv)
(11.110)
E[
xihxT
ih]
=E[xih]E[xih]T+Vi (11.111)
11.6.1.3 M step
By solving ∇Q(θ,θ(t−1))=0, we can show that the M step is equivalent to plugging these
ESS into the usual MLE equations to get
μt=1
N∑
iE[xi] (11.112)
Σt=1
N∑
iE[
xixTi]
−μt(μt)T(11.113)
Thus we see that EM is notequivalent to simply replacing variables by their expectations and
applying the standard MLE formula; that would ignore the posterior variance and would result

=E[(xih
xiv)(
xT
ihxT
iv)]
=(E[
xihxT
ih]
E[xih]xT
iv
(11.110)
E[
xihxT
ih]
μt=1
N∑
Σt=1
N∑
iE[
xixTi]
in an incorrect estimate. Instead we must compute the expectation of the sufficient statistics,and plug that into the usual equation for the MLE. We can easily modify the algorithm toperform MAP estimation, by plugging in the ESS into the equation for the MAP estimate. For animplementation, see gaussMissingFitEm .
11.6.1.4 Example
As an example of this procedure in action, let us reconsider the imputation problem fromSection 4.3.2.3, which had N= 10010-dimensional data cases, with 50% missing data. Let

us ﬁt the parameters using EM. Call the resulting parameters ˆθ. We can use our model for
predictions by computing E[
x
ih|xiv,ˆθ]
. Figure 11.22(a-b) indicates that the results obtained
using the learned parameters are almost as good as with the true parameters. Not surprisingly,performance improves with more data, or as the fraction of missing data is reduced.
11.6.1.5 Extension to the GMM case
It is straightforward to ﬁt a mixture of Gaussians in the presence of partially observed datavectorsx

x
i. We leave the details as an exercise.
Exercises
Exercise 11.1 Student T as inﬁnite mixture of Gaussians
Derive Equation 11.61. For simplicity, assume a one-dimensional distribution.

11.6. Fitting models with missing data 375
−10 0 10−10−50510
truthimputedR2 = 0.260
−20 0 20−1001020
truthimputedR2 = 0.685
truthimputedR2 = 0.399imputation with true params
truthimputedR2 = 0.531
(a)−10 0 10−10−50510
truthimputedR2 = 0.220
truthimputedR2 = 0.609
truthimputedR2 = 0.113imputation with em
truthimputedR2 = 0.532
(b)
Figure 11.22 Illustration of data imputation. (a) Scatter plot of true values vs imputed values us-
ing true parameters. (b) Same as (b), but using parameters estimated with EM. Figure generated by
gaussImputationDemo .
Exercise 11.2 EM for mixtures of Gaussians
Show that the M step for ML estimation of a mixture of Gaussians is given by
μk=∑
irikxi
rk(11.114)
Σk=∑
rk=∑
irikxixT
i−rkμkμT
k
rk(11.115)
Exercise 11.3 EM for mixtures of Bernoullis
• Show that the M step for ML estimation of a mixture of Bernoullis is given by
μkj=∑
irikxij∑
irik(11.116)

μk=∑
irikxi
Σk=∑
rk=∑
irikxixT
i−rkμkμT
k
μkj=∑
irikxij∑
• Show that the M step for MAP estimation of a mixture of Bernoullis with a β(α,β)prior is given by
μkj=(∑
irikxij)+α−1
(∑
irik)+α+β−2(11.117)
Exercise 11.4 EM for mixture of Student distributions
Derive the EM algorithm for ML estimation of a mixture of multivariate Student T distributions.
Exercise 11.5 Gradient descent for ﬁtting GMM
Consider the Gaussian mixture model
p(x|θ)=∑
kπkN(x|μk,Σk) (11.118)
Deﬁne the log likelihood as
ℓ(θ)=N∑
n=1logp(xn|θ) (11.119)

376 Chapter 11. Mixture models and the EM algorithm
N
m lμj σkxnJn Knpq
Figure 11.23 A mixture of Gaussians with two discrete latent indicators. Jnspeciﬁes which mean to use,
andKnspeciﬁes which variance to use.
Deﬁne the posterior responsibility that cluster khas for datapoint nas follows:
rnk≜p(zn=k|xn,θ)=πkN(xn|μk,Σk)∑K
k′=1πk′N(xn|μk′,Σk′)(11.120)
a. Show that the gradient of the log-likelihood wrt μkis
d
dμkℓ(θ)=∑
nrnkΣ−1
k(xn−μk) (11.121)
b. Derive the gradient of the log-likelihood wrt πk. (For now, ignore any constraints on πk.)
c. One way to handle the constraint that∑Kk=1πk=1is to reparameterize using the softmax function:
πk≜ewk
∑Kk
′=1ewk′(11.122)
Herewk∈Rare unconstrained parameters. Show that
d
dwkℓ(θ)=∑
nrnk−πk (11.123)
(There may be a constant factor missing in the above expression...) Hint: use the chain rule and the
fact that
dπj
dwk={πj(1−πj)ifj=k
−πjπkifj̸=k(11.124)
which follows from Exercise 8.4(1).

nrnkΣ−1
πk≜ewk
∑Kk
d
dwkℓ(θ)=∑
fact that
dπj
d. Derive the gradient of the log-likelihood wrt Σk. (For now, ignore any constraints on Σk.)
e. One way to handle the constraint that Σkbe a symmetric positive deﬁnite matrix is to reparame-
terize using a Cholesky decomposition, Σk=RT
kR,w h e r eRis an upper-triangular, but otherwise
unconstrained matrix. Derive the gradient of the log-likelihood wrt Rk.
Exercise 11.6 EM for a ﬁnite scale mixture of Gaussians
(Source: Jaakkola..) Consider the graphical model in Figure 11.23 which deﬁnes the following:

p(xn|θ)=m∑
j=1pj[l∑
k=1qkN(xn|μj,σ2
k)]
(11.125)

11.6. Fitting models with missing data 377
whereθ={p1,...,p m,μ1,...,μ m,q1,...,q l,σ2
1,...,σ2
l}are all the parameters. Here pj≜P(Jn=
j)andqk≜P(Kn=k)are the equivalent of mixture weights. We can think of this as a mixture
ofmnon-Gaussian components, where each component distribution is a scale mixture, p(x|j;θ)=∑l
k=1qkN(x;μj,σ2
k), combining Gaussians with different variances (scales).
We will now derive a generalized EM algorithm for this model. (Recall that in generalized EM, we do a
partial update in the M step, rather than ﬁnding the exact maximum.)
a. Derive an expression for the responsibilities, P(Jn=j,Kn=k|xn,θ), needed for the E step.
b. Write out a full expression for the expected complete log-likelihood
Q(θnew,θold)=EθoldN∑
n=1logP(Jn,Kn,xn|θnew) (11.126)
c. Solving the M-step would require us to jointly optimize the means μ1,...,μ mand the variances
σ2
1,...,σ2
l. It will turn out to be simpler to ﬁrst solve for the μj’s given ﬁxed σ2
j’s, and subsequently
solve for σ2

σ2
1,...,σ2
j’s given the new values of μj’s. For brevity, we will just do the ﬁrst part. Derive an
expression for the maximizing μj’s given ﬁxed σ2
1:l, i.e., solve∂Q
∂μnew=0.
Exercise 11.7 Manual calculation of the M step for a GMM
(Source: de Freitas.) In this question we consider clustering 1D data with a mixture of 2 Gaussians usingthe EM algorithm. You are given the 1-D data points x= [ 11 02 0 ] . Suppose the output of the E
step is the following matrix:
R=⎡
⎣10
0.40.6
01⎤⎦ (11.127)
where entry r

∂μnew=0.
R=⎡
⎣10
0.40.6
i,cis the probability of obervation xibelonging to cluster c(the responsibility of cluster cfor
data point i). You just have to compute the M step. You may state the equations for maximum likelihood
estimates of these quantities (which you should know) without proof; you just have to apply the equations
to this data set. You may leave your answer in fractional form. Show your work.
a. Write down the likelihood function you are trying to optimize.

R=⎡
⎣10
0.40.6
b. After performing the M step for the mixing weights π1,π2, what are the new values?
c. After performing the M step for the means μ1andμ2, what are the new values?
Exercise 11.8 Moments of a mixture of Gaussians
Consider a mixture of KGaussians
p(x)=K∑
k=1πkN(x|μk,Σk) (11.128)
a. Show that
E[x]=∑
kπkμk (11.129)

378 Chapter 11. Mixture models and the EM algorithm
0 2 4 6 8 10 12 14 16 18−2−1012345
Figure 11.24 Some data points in 2d. Circles represent the initial guesses for m1andm2.
b. Show that
cov[x]=∑
k]−E[x]E[x]T(11.130)
Hint: use the fact that cov[x]= E[
xxT]
−E[x]E[x]T.
Exercise 11.9 K-means clustering by hand
In Figure 11.24, we show some data points which lie on the integer grid. (Note that the x-axis has been
compressed; distances should be measured using the actual grid coordinates.) Suppose we apply the K-means algorithm to this data, using K=2and with the centers initialized at the two circled data points.
Draw the ﬁnal clusters obtained after K-means converges (show the approximate location of the new centersand group together all the points assigned to each center). Hint: think about shortest Euclidean distance.
Exercise 11.10 Deriving the K-means cost function
Show that
J
W(z)=1
2K∑
k=1∑
i:zi=k∑
i′:zi′=k(xi−xi′)2=K∑
k=1nk∑
i:zi=k(xi−xk)2(11.131)

Show that
J
W(z)=1
2K∑
k=1∑
i:zi=k∑
k=1nk∑
Hint: note that, for any μ,
∑
i(xi−μ)2=∑
i[(xi−x)−(μ−x)]2(11.132)
=∑
i(xi−x)2+∑
i(x−μ)2−2∑
i(xi−x)(μ−x) (11.133)
=ns2+n(x−μ)2(11.134)
wheres2=1
n∑n
i=1(xi−x)2, since
∑
i(xi−x)(μ−x)=( μ−x)(
(∑
ixi)−nx)
=(μ−x)(nx−nx)=0 (11.135)
Exercise 11.11 Visible mixtures of Gaussians are in the exponential family
Show that the joint distribution p(x,z|θ)for a 1d GMM can be represented in exponential family form.

11.6. Fitting models with missing data 379
2 2.05 2.1 2.15 2.2 2.25 2.3 2.35 2.42.62.833.23.43.63.844.24.44.6
inverse temperaturesurvival timeregression with censored data; red x = censored, green * = predicted
  
EM
OLS
Figure 11.25 Example of censored linear regression. Black circles are observed training points, red crosses
are observed but censored training points. Green stars are predicted values of the censored training points.
We also show the lines ﬁt by least squares (ignoring censoring) and by EM. Based on Figure 5.6 of (Tanner1996). Figure generated by linregCensoredSchmeeHahnDemo , written by Hannes Bretschneider.
Exercise 11.12 EM for robust linear regression with a Student t likelihood
Consider a model of the form
p(y
i|xi,w,σ2,ν)=T(yi|wTxi,σ2,ν) (11.136)
Derive an EM algorithm to compute the MLE for w. You may assume νandσ2are ﬁxed, for simplicity.
Hint: see Section 11.4.5.
Exercise 11.13 EM for EB estimation of Gaussian shrinkage model

p(y
Extend the results of Section 5.6.2.2 to the case where the σ2
jare not equal (but are known). Hint: treat
theθjas hidden variables, and then to integrate them out in the E step, and maximize η=(μ,τ2)in the
M step.Exercise 11.14 EM for censored linear regression
Censored regression refers to the case where one knows the outcome is at least (or at most) a certain
value, but the precise value is unknown. This arises in many different settings. For example, suppose one

is trying to learn a model that can predict how long a program will take to run, for different settings ofits parameters. One may abort certain runs if they seem to be taking too long; the resulting run times aresaid to be right censored. For such runs, all we know is that y
i≥ci,w h e r eciis the censoring time,
that is,yi=m i n (zi,ci),w h e r eziis the true running time and yiis the observed running time. We
can also deﬁne left censored andinterval censored models.3Derive an EM algorithm for ﬁtting a linear

regression model to right-censored data. Hint: use the results from Exercise 11.15. See Figure 11.25 for anexample, based on the data from (Schmee and Hahn 1979). We notice that the EM line is tilted upwardsmore, since the model takes into account the fact that the truncated values are actually higher than theobserved values.
3. There is a closely related model in econometrics called the Tobit model, in which yi=m a x (zi,0), so we only
get to observe positive outcomes. An example of this is when zirepresents “desired investment”, and yiis actual
investment. Probit regression (Section 9.4) is another example.

380 Chapter 11. Mixture models and the EM algorithm
Exercise 11.15 Posterior mean and variance of a truncated Gaussian
Letzi=μi+σϵi,w h e r eϵi∼N(0,1). Sometimes, such as in probit regression or censored regression,
we do not observe zi, but we observe the fact that it is above some threshold, namely we observe the event
E=I(zi≥ci)=I(ϵi≥ci−μi
σ). (See Exercise 11.14 for details on censored regression, and Section 11.4.6
for probit regression.) Show that
E[zi|zi≥ci]=μ i+σH(ci−μi
σ)
(11.137)
and
E[
z2
i|zi≥ci]
=μ2
i+σ2+σ(ci+μi)H(ci−μi
σ)
(11.138)
where we have deﬁned
H(u)≜φ(u)
1−Φ(u)(11.139)
and where φ(u)is the pdf of a standard Gaussian, and Φ(u)is its cdf.
Hint 1: we have p(ϵi|E)=p(ϵi,E)
p(E),w h e r eE is some event of interest.
Hint 2: It can be shown that
d
dwN(w|0,1) =−wN(w|0,1) (11.140)
and hence
∫c
bwN(w|0,1) =N(b|0,1)−N(c|0,1) (11.141)

12 Latent linear models
12.1 Factor analysis
One problem with mixture models is that they only use a single latent variable to generate the
observations. In particular, each observation can only come from one of Kprototypes. One can
think of a mixture model as using Khidden binary variables, representing a one-hot encoding
of the cluster identity. But because these variables are mutually exclusive, the model is stilllimited in its representational power.
An alternative is to use a vector of real-valued latent variables, z
i∈RL. The simplest prior
to use is a Gaussian (we will consider other choices later):
p(zi)=N(zi|μ0,Σ0) (12.1)
If the observations are also continuous, so xi∈RD, we may use a Gaussian for the likelihood.
Just as in linear regression, we will assume the mean is a linear function of the (hidden) inputs,thus yielding
p(x
i|zi,θ)=N(Wzi+μ,Ψ) (12.2)
whereWis aD×Lmatrix, known as the factor loading matrix, and Ψis aD×Dcovariance

p(x
matrix. We take Ψto be diagonal, since the whole point of the model is to “force” zito explain
the correlation, rather than “baking it in” to the observation’s covariance. This overall modelis called factor analysis orFA. The special case in which Ψ=σ
2Iis called probabilistic
principal components analysis orPPCA. The reason for this name will become apparent later.
The generative process, where L=1,D=2andΨis diagonal, is illustrated in Figure 12.1.

We take an isotropic Gaussian “spray can” and slide it along the 1d line deﬁned by wzi+μ.
This induces an ellongated (and hence correlated) Gaussian in 2d.
12.1.1 FA is a low rank parameterization of an MVN
FA can be thought of as a way of specifying a joint density model on xusing a small number
of parameters. To see this, note that from Equation 4.126, the induced marginal distributionp(x
i|θ)is a Gaussian:
p(xi|θ)=∫
N(xi|Wzi+μ,Ψ)N(zi|μ0,Σ0)dzi (12.3)
=N(xi|Wμ0+μ,Ψ+WΣ0WT) (12.4)

2
0. Then we
ﬁnd
cov[x|θ]=˜WT+E[
ϵϵT]
=(WΣ−1
2
0)Σ0(WΣ−1
2
0)T+Ψ=WWT+Ψ (12.5)
We thus see that FA approximates the covariance matrix of the visible vector using a low-rank
decomposition:
C≜cov[x]=WWT+Ψ (12.6)
This only uses O(LD)parameters, which allows a ﬂexible compromise between a full covariance
Gaussian, with O(D2)parameters, and a diagonal covariance, with O(D)parameters. Note that
if we did not restrict Ψto be diagonal, we could trivially set Ψto a full covariance matrix; then
we could set W=0, in which case the latent factors would not be required.
12.1.2 Inference of the latent factors
Although FA can be thought of as just a way to deﬁne a density on x, it is often used because
we hope that the latent factors zwill reveal something interesting about the data. To do this,
we need to compute the posterior over the latent factors. We can use Bayes rule for Gaussians
to give
p(zi|xi,θ)=N(zi|mi,Σi) (12.7)
Σi≜(Σ−1
0+WTΨ−1W)−1(12.8)
mi≜Σi(WTΨ−1(xi−μ)+Σ−1
0μ0) (12.9)

to give
Σi≜(Σ−1
Note that in the FA model, Σiis actually independent of i, so we can denote it by Σ. Computing
this matrix takes O(L3+L2D)time, and computing each mi=E[zi|xi,θ]takesO(L2+LD)
time. The miare sometimes called the latent scores,o rl a t e n t factors.

12.1. Factor analysis 383
−1 −0.5 0 0.5 1−1−0.8−0.6−0.4−0.200.20.40.60.81
RetailDealerEngine
Cylinders
Horsepower
CityMPGHighwayMPGWeight WheelbaseLengthWidth
Component 1Component 2rotation=none
Porsche 911 Honda InsightGMC Yukon XL 2500 SLT
Mercedes−Benz CL600Kia Sorento LX
Mercedes−Benz G500
Saturn Ion1Nissan Pathfinder Armada SE
Figure 12.2 2D projection of 2004 cars data based on factor analysis. The blue text are the names of cars
corresponding to certain chosen points. Figure generated by faBiplotDemo .
Let us give a simple example, based (Shalizi 2009). We consider a dataset of D=1 1variables
andN= 387cases describing various aspects of cars, such as the engine size, the number of
cylinders, the miles per gallon (MPG), the price, etc. We ﬁrst ﬁt a L=2dimensional model. We
can plot the miscores as points in R2, to visualize the data, as shown in Figure 12.2.
To get a better understanding of the “meaning” of the latent factors, we can project unit vectors

corresponding to each of the feature dimensions, e1=( 1,0,...,0),e2=( 0,1,0,...,0), etc.
into the low dimensional space. These are shown as blue lines in Figure 12.2; this is known as
abiplot. We see that the horizontal axis represents price, corresponding to the features labeled

“dealer” and “retail”, with expensive cars on the right. The vertical axis represents fuel efficiency(measured in terms of MPG) versus size: heavy vehicles are less efficient and are higher up,whereas light vehicles are more efficient and are lower down. We can “verify” this interpretationby clicking on some points, and ﬁnding the closest exemplars in the training set, and printingtheir names, as in Figure 12.2. However, in general, interpreting latent variable models is fraughtwith difficulties, as we discuss in Section 12.1.3.

12.1.3 Unidentiﬁability
Just like with mixture models, FA is also unidentiﬁable. To see this, suppose Ris an arbitrary
orthogonal rotation matrix, satisfying RRT=I. Let us deﬁne ˜W=WR; then the likelihood

384 Chapter 12. Latent linear models
function of this modiﬁed matrix is the same as for the unmodiﬁed matrix, since
cov[x]=˜WE[
zzT]˜WT+E[
ϵϵT]
(12.10)
=WRRTWT+Ψ=WWT+Ψ (12.11)
Geometrically, multiplying Wby an orthogonal matrix is like rotating zbefore generating x;
but since zis drawn from an isotropic Gaussian, this makes no difference to the likelihood.
Consequently, we cannot unique identify W, and therefore cannot uniquely identify the latent
factors, either.
To ensure a unique solution, we need to remove L(L−1)/2degrees of freedom, since that
is the number of orthonormal matrices of size L×L.1In total, the FA model has D+LD−
L(L−1)/2free parameters (excluding the mean), where the ﬁrst term arises from Ψ. Obviously
we require this to be less than or equal to D(D+1)/2, which is the number of parameters in
an unconstrained (but symmetric) covariance matrix. This gives us an upper bound on L,a s
follows:
Lmax=⌊D+0.5(1−√
1+8D)⌋ (12.12)

follows:
For example, D=6impliesL≤3. But we usually never choose this upper bound, since it
would result in overﬁtting (see discussion in Section 12.3 on how to choose L).
Unfortunately, even if we set L<L max, we still cannot uniquely identify the parameters,
since the rotational ambiguity still exists. Non-identiﬁability does not affect the predictive per-

follows:
formance of the model. However, it does affect the loading matrix, and hence the interpretationof the latent factors. Since factor analysis is often used to uncover structure in the data, thisproblem needs to be addressed. Here are some commonly used solutions:
•ForcingWto be orthonormal Perhaps the cleanest solution to the identiﬁability problem
is to force Wto be orthonormal, and to order the columns by decreasing variance of the
corresponding latent factors. This is the approach adopted by PCA, which we will discuss in

Section 12.2. The result is not necessarily more interpretable, but at least it is unique.
•ForcingWto be lower triangular One way to achieve identiﬁability, which is popular
in the Bayesian community (e.g., (Lopes and West 2004)), is to ensure that the ﬁrst visiblefeature is only generated by the ﬁrst latent factor, the second visible feature is only generatedby the ﬁrst two latent factors, and so on. For example, if L=3andD=4, the correspond
factor loading matrix is given by
W=⎛
⎜⎜⎝w
1100
w21w220
w31w32w33
w41w42w43⎞
⎟⎟⎠(12.13)
We also require that wjj>0forj=1:L. The total number of parameters in this
constrained matrix is D+DL−L(L−1)/2, which is equal to the number of uniquely

W=⎛
⎜⎜⎝w
1100
w21w220
w31w32w33
identiﬁable parameters. The disadvantage of this method is that the ﬁrst Lvisible variables,
1. To see this, note that there are L−1free parameters in Rin the ﬁrst column (since the column vector must be
normalized to unit length), there are L−2free parameters in the second column (which must be orthogonal to the
ﬁrst), and so on.

12.1. Factor analysis 385
Ψπ
Wkμk
xiqizi
N K
Figure 12.3 Mixture of factor analysers as a DGM.
known as the founder variables, affect the interpretation of the latent factors, and so must
be chosen carefully.
•Sparsity promoting priors on the weights Instead of pre-specifying which entries in W
are zero, we can encourage the entries to be zero, using ℓ1regularization (Zou et al. 2006),
ARD (Bishop 1999; Archambeau and Bach 2008), or spike-and-slab priors (Rattray et al. 2009).
This is called sparse factor analysis. This does not necessarily ensure a unique MAP estimate,but it does encourage interpretable solutions. See Section 13.8.
•Choosing an informative rotation matrix There are a variety of heuristic methods that try
to ﬁnd rotation matrices Rwhich can be used to modify W(and hence the latent factors) so
as to try to increase the interpretability, typically by encouraging them to be (approximately)sparse. One popular method is known as varimax (Kaiser 1958).

•Use of non-Gaussian priors for the latent factors In Section 12.6, we will dicuss how re-
placingp(z
i)with a non-Gaussian distribution can enable us to sometimes uniquely identify
Was well as the latent factors. This technique is known as ICA.
12.1.4 Mixtures of factor analysers
The FA model assumes that the data lives on a low dimensional linear manifold. In reality, mostdata is better modeled by some form of low dimensional curvedmanifold. We can approximate

a curved manifold by a piecewise linear manifold. This suggests the following model: let thek’th linear subspace of dimensionality L
kbe represented by Wk,f o rk=1:K. Suppose we
have a latent indicator qi∈{1,...,K}specifying which subspace we should use to generate
the data. We then sample zifrom a Gaussian prior and pass it through the Wkmatrix (where
k=qi), and add noise. More precisely, the model is as follows:
p(xi|zi,qi=k,θ)=N (xi|μk+Wkzi,Ψ) (12.14)
p(zi|θ)=N (zi|0,I) (12.15)
p(qi|θ)=C a t ( qi|π) (12.16)

386 Chapter 12. Latent linear models
−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−1.5−1−0.500.511.5
(a)−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−1.5−1−0.500.511.5
(b)
Figure 12.4 Mixture of 1d PPCAs ﬁt to a dataset, for K=1,10. Figure generated by
mixPpcaDemoNetlab .
This is called a mixture of factor analysers (MFA) (Hinton et al. 1997). The CI assumptions are
represented in Figure 12.3.
Another way to think about this model is as a low-rank version of a mixture of Gaussians. In
particular, this model needs O(KLD)parameters instead of the O(KD2)parameters needed
for a mixture of full covariance Gaussians. This can reduce overﬁtting. In fact, MFA is a good
generic density model for high-dimensional real-valued data.
12.1.5 EM for factor analysis models

Using the results from Chapter 4, it is straightforward to derive an EM algorithm to ﬁt an FAmodel. With just a little more work, we can ﬁt a mixture of FAs. Below we state the resultswithout proof. The derivation can be found in (Ghahramani and Hinton 1996a); however, derivingthese equations yourself is a useful exercise if you want to become proﬁcient at the math.
To obtain the results for a single factor analyser, just set r
ic=1andc=1in the equations

below. In Section 12.2.5 we will see a further simpliﬁcation of these equations that arises whenﬁtting a PPCA model, where the results will turn out to have a particularly simple and elegantintepretation.
In the E step, we compute the posterior responsibility of cluster cfor data point iusing
r
ic≜p(qi=c|xi,θ)∝πcN(xi|μc,WcWT
c+Ψ) (12.17)
The conditional posterior for ziis given by
p(zi|xi,qi=c,θ)=N (zi|mic,Σic) (12.18)
Σic≜(IL+WT
cΨ−1
cWc)−1(12.19)
mic≜Σic(WT
cΨ−1
c(xi−μc)) (12.20)

r
cΨ−1
cΨ−1
In the M step, it is easiest to estimate μcandWcat the same time, by deﬁning ˜Wc=

12.2. Principal components analysis (PCA) 387
(Wc,μc),˜z=(z,1), Also, deﬁne
bic≜E[˜ z|xi,qi=c]=[mic;1] (12.21)
Cic≜E[
˜z˜zT|xi,qi=c]
=(E[
zzT|xi,qi=c]
E[z|xi,qi=c]
E[z|xi,qi=c]T1)
(12.22)
Then the M step is as follows:
ˆ˜Wc=[∑
iricxibT
ic][∑
iricCic]−1
(12.23)
ˆΨ=1
Ndiag{∑
icric(
xi−ˆ˜Wcbic)
xT
i}
(12.24)
ˆπc=1
NN∑
i=1ric (12.25)
Note that these updates are for “vanilla” EM. A much faster version of this algorithm, based
on ECM, is described in (Zhao and Yu 2008).
12.1.6 Fitting FA models with missing data
In many applications, such as collaborative ﬁltering, we have missing data. One virtue of the
EM approach to ﬁtting an FA/PPCA model is that it is easy to extend to this case. However,overﬁtting can be a problem if there is a lot of missing data. Consequently it is important toperform MAP estimation or to use Bayesian inference. See e.g., (Ilin and Raiko 2010) for details.
12.2 Principal components analysis (PCA)
Consider the FA model where we constrain Ψ=σ2I, andWto be orthonormal. It can

be shown (Tipping and Bishop 1999) that, as σ2→0, this model reduces to classical (non-
probabilistic) principal components analysis (PCA), also known as the Karhunen Loeve
transform. The version where σ2>0is known as probabilistic PCA (PPCA) (Tipping and
Bishop 1999), or sensible PCA (Roweis 1997). (An equivalent result was derived independently,
from a different perspective, in (Moghaddam and Pentland 1995).)
To make sense of this result, we ﬁrst have to learn about classical PCA. We then connect PCA

to the SVD. And ﬁnally we return to discuss PPCA.
12.2.1 Classical PCA: statement of the theorem
Thesynthesis view of classical PCA is summarized in the forllowing theorem.
Theorem 12.2.1. Suppose we want to ﬁnd an orthogonal set of Llinear basis vectors wj∈RD,
and the corresponding scores zi∈RL, such that we minimize the average reconstruction error
J(W,Z)=1
NN∑
i=1||xi−ˆxi||2(12.26)

388 Chapter 12. Latent linear models
−5 0 5−3−2−101234
(a)−4 −3 −2 −1 0 1 2 3−5−4−3−2−101234
(b)
Figure 12.5 An illustration of PCA and PPCA where D=2andL=1. Circles are the original data
points, crosses are the reconstructions. The red star is the data mean. (a) PCA. The points are orthogonally
projected onto the line. Figure generated by pcaDemo2d . (b) PPCA. The projection is no longer orthogonal:
the reconstructions are shrunk towards the data mean (red star). Based on Figure 7.6 of (Nabney 2001).Figure generated by ppcaDemo2d .
whereˆxi=Wzi, subject to the constraint that Wis orthonormal. Equivalently, we can write this
objective as follows:
J(W,Z)=||X−WZT||2
F (12.27)
whereZis anN×Lmatrix with the ziin its rows, and ||A||Fis theFrobenius norm of matrix
A, deﬁned by
||A||F=√m∑
i=1n∑
j=1a2
ij=√
tr(ATA)=||A(:)|| 2 (12.28)
The optimal solution is obtained by setting ˆW=VL, whereVLcontains the Leigenvectors
with largest eigenvalues of the empirical covariance matrix, ˆΣ=1
N∑N
i=1xixT
i. (We assume the

F (12.27)
i=1n∑
j=1a2
ij=√
N∑N
i=1xixT
xihave zero mean, for notational simplicity.) Furthermore, the optimal low-dimensional encoding
of the data is given by ˆzi=WTxi, which is an orthogonal projection of the data onto the column
space spanned by the eigenvectors.
An example of this is shown in Figure 12.5(a) for D=2andL=1. The diagonal line is the
vectorw1; this is called the ﬁrst principal component or principal direction. The data points
xi∈R2are orthogonally projected onto this line to get zi∈R. This is the best 1-dimensional

approximation to the data. (We will discuss Figure 12.5(b) later.)
In general, it is hard to visualize higher dimensional data, but if the data happens to be a
set of images, it is easy to do so. Figure 12.6 shows the ﬁrst three principal vectors, reshaped
as images, as well as the reconstruction of a speciﬁc image using a varying number of basisvectors. (We discuss how to choose Lin Section 11.5.)
Below we will show that the principal directions are the ones along which the data shows

maximal variance. This means that PCA can be “misled” by directions in which the varianceis high merely because of the measurement scale. Figure 12.7(a) shows an example, where thevertical axis (weight) uses a large range than the horizontal axis (height), resulting in a line thatlooks somewhat “unnatural”. It is therefore standard practice to standardize the data ﬁrst, or

12.2. Principal components analysis (PCA) 389
mean
 principal basis 1
principal basis 2
 principal basis 3
(a)
reconstructed with 2 bases
 reconstructed with 10 bases
reconstructed with 100 bases
 reconstructed with 506 bases
(b)
Figure 12.6 (a) The mean and the ﬁrst three PC basis vectors (eigendigits) based on 25 images of the digit
3 (from the MNIST dataset). (b) Reconstruction of an image based on 2, 10, 100 and all the basis vectors.
Figure generated by pcaImageDemo .
55 60 65 70 75 80 8550100150200250300
(a)−4 −3 −2 −1 0 1 2 3 4−4−3−2−1012345
(b)
Figure 12.7 Effect of standardization on PCA applied to the height/ weight dataset. Left: PCA of raw data.
Right: PCA of standardized data. Figure generated by pcaDemoHeightWeight .
equivalently, to work with correlation matrices instead of covariance matrices. The beneﬁts of
this are apparent from Figure 12.7(b).
12.2.2 Proof *
Proof.We usewj∈RDto denote the j’th principal direction, xi∈RDto denote the i’th

(b)
high-dimensional observation, zi∈RLto denote the i’th low-dimensional representation, and
˜ zj∈RNto denote the [z1j,...,z Nj], which is the j’th component of all the low-dimensional
vectors.
Let us start by estimating the best 1d solution, w1∈RD, and the corresponding projected
points˜ z1∈RN. We will ﬁnd the remaining bases w2,w3, etc. later. The reconstruction error

390 Chapter 12. Latent linear models
is given by
J(w1,z1)=1
NN∑
i=1||xi−zi1w1||2=1
NN∑
i=1(xi−zi1w1)T(xi−zi1w1)(12.29)
=1
NN∑
i=1[xT
ixi−2zi1wT
1xi+z2
i1wT
1w1] (12.30)
=1
NN∑
i=1[xTixi−2zi1wT
1xi+z2
i1] (12.31)
sincewT
1w1=1(by the orthonormality assumption). Taking derivatives wrt zi1and equating
to zero gives
∂
∂zi1J(w1,z1)=1
N[−2wT
1xi+2zi1]=0⇒zi1=wT
1xi (12.32)
So the optimal reconstruction weights are obtained by orthogonally projecting the data onto the
ﬁrst principal direction, w1(see Figure 12.5(a)). Plugging back in gives
J(w1)=1
NN∑
i=1[xT
ixi−z2
i1]=c o n s t −1
NN∑
i=1z2
i1 (12.33)
Now the variance of the projected coordinates is given by
var[˜ z1]=E[
˜ z21]
−(E[˜ z1])2=1
NN∑
i=1z2
i1−0 (12.34)
since
E[zi1]=E[
xTiw1]
=E[xi]Tw1=0 (12.35)
because the data has been centered. From this, we see that minimizing the reconstruction error
is equivalent to maximizing the variance of the projected data, i.e.,
argmin
w1J(w1) = argmax
w1var[˜ z1] (12.36)

J(w1)=1
NN∑
i=1[xT
ixi−z2
NN∑
i=1z2
i1 (12.33)
˜ z21]
NN∑
i=1z2
since
E[zi1]=E[
xTiw1]
argmin
This is why it is often said that PCA ﬁnds the directions of maximal variance. This is called the
analysis view of PCA.
The variance of the projected data can be written as
1
NN∑
i=1z2
i1=1
NN∑
i=1wT
1xixT
iw1=wT
1ˆΣw1 (12.37)
whereˆΣ=1
N∑N
i=1∑
ixixT
iis the empirical covariance matrix (or correlation matrix if the
data is standardized).

12.2. Principal components analysis (PCA) 391
We can trivially maximize the variance of the projection (and hence minimize the recon-
struction error) by letting ||w1|| → ∞, so we impose the constraint ||w1||=1and instead
maximize
˜J(w1)=wT
1ˆΣw1+λ1(wT
1w1−1) (12.38)
whereλ1is the Lagrange multiplier. Taking derivatives and equating to zero we have
∂
∂w1˜J(w1)=2 ˆΣw1−2λ1w1=0 (12.39)
ˆΣw1=λ1w1 (12.40)
Hence the direction that maximizes the variance is an eigenvector of the covariance matrix. Left
multiplying by w1(and using wT
1w1=1) we ﬁnd that the variance of the projected data is
wT
1ˆΣw1=λ1 (12.41)
Since we want to maximize the variance, we pick the eigenvector which corresponds to thelargest eigenvalue.
Now let us ﬁnd another direction w
2to further minimize the reconstruction error, subject to
wT
1w2=0andwT
2w2=1. The error is
J(w1,z1,w2,z2)=1
NN∑
i=1||xi−zi1w1−zi2w2||2(12.42)
Optimizing wrt w1andz1gives the same solution as before. Exercise 12.4 asks you to show
that∂J
∂z2=0yieldszi2=wT

wT
wT
NN∑
that∂J
2xi. In other words, the second principal encoding is gotten by
projecting onto the second principal direction. Substituting in yields
J(w2)=1
nN∑
i=1[xT
ixi−wT
1xixTiw1−wT
2xixTiw2]=c o n s t −wT
2ˆΣw2 (12.43)
Dropping the constant term and adding the constraints yields
˜J(w2)=−wT
2ˆΣw2+λ2(wT
2w2−1)+λ12(wT
2w1−0) (12.44)
Exercise 12.4 asks you to show that the solution is given by the eigenvector with the second
largest eigenvalue:
ˆΣw2=λ2w2 (12.45)

that∂J
J(w2)=1
nN∑
i=1[xT
ixi−wT
The proof continues in this way. (Formally one can use induction.)

392 Chapter 12. Latent linear models
12.2.3 Singular value decomposition (SVD)
We have deﬁned the solution to PCA in terms of eigenvectors of the covariance matrix. However,
there is another way to obtain the solution, based on the singular value decomposition,o r
SVD. This basically generalizes the notion of eigenvectors from square matrices to any kind of
matrix.
In particular, any (real) N×DmatrixXcan be decomposed as follows
X
N×D=U
N×NS
N×DVT

D×D(12.46)
whereUis anN×Nmatrix whose columns are orthornormal (so UTU=IN),VisD×D
matrix whose rows and columns are orthonormal (so VTV=VVT=ID), andSis aN×D
matrix containing the r=m i n (N,D)singular values σi≥0on the main diagonal, with 0s
ﬁlling the rest of the matrix. The columns of Uare the left singular vectors, and the columns
ofVare the right singular vectors. See Figure 12.8(a) for an example.
Since there are at most Dsingular values (assuming N>D), the last N−Dcolumns of U

N×NS
N×DVT

are irrelevant, since they will be multiplied by 0. The economy sized SVD,o rthin SVD, avoids
computing these unnecessary elements. Let us denote this decomposition by ˆUˆSˆV.I fN>D,
we have
X
N×D=ˆU
N×DˆS
D×DˆVT

D×D(12.47)
as in Figure 12.8(a). If N<D,w eh a v e
X
N×NˆS
N×NˆVT

N×D(12.48)
Computing the economy-sized SVD takes O(NDmin(N,D ))time (Golub and van Loan 1996,
p254).
The connection between eigenvectors and singular vectors is the following. For an arbitrary

we have
X
N×DˆS
D×DˆVT

X
N×NˆS
N×NˆVT

p254).
real matrix X,i fX=USVT,w eh a v e
XTX=VSTUTUSVT=V(STS)VT=VDVT(12.49)
whereD=S2is a diagonal matrix containing the squares singular values. Hence
(XTX)V=VD (12.50)
so the eigenvectors of XTXare equal to V, the right singular vectors of X, and the eigenvalues
ofXTXare equal to D, the squared singular values. Similarly
XXT=USVTVSTUT=U(SST)UT(12.51)
(XXT)U=U(SST)=UD (12.52)
so the eigenvectors of XXTare equal to U, the left singular vectors of X. Also, the eigenvalues
ofXXTare equal to the squared singular values. We can summarize all this as follows:

U=evec(XXT),V=evec(XTX),S2=eval(XXT)=eval(XTX) (12.53)

12.2. Principal components analysis (PCA) 393
=
0σ1
σD...D
DD N−D D D
N
X= US VT
(a)
≃L
σ1...σLL D
LD
N
X≃ULSLVT
L
(b)
Figure 12.8 (a) SVD decomposition of non-square matrices X=USVT. The shaded parts of S, and all
the off-diagonal terms, are zero. The shaded entries in UandSare not computed in the economy-sized
version, since they are not needed. (b) Truncated SVD approximation of rank L.
Since the eigenvectors are unaffected by linear scaling of a matrix, we see that the right
singular vectors of Xare equal to the eigenvectors of the empirical covariance ˆΣ. Furthermore,
the eigenvalues of ˆΣare a scaled version of the squared singular values. This means we can
perform PCA using just a few lines of code (see pcaPmtk).
However, the connection between PCA and SVD goes deeper. From Equation 12.46, we can
represent a rank rmatrix as follows:
X=σ1⎛
⎝|
u1
|⎞⎠(
−v
T
1−)
+···+σr⎛⎝|
u
r
|⎞⎠(
−v
T
r−)
(12.54)
If the singular values die off quickly as in Figure 12.10, we can produce a rank Lapproximation

X=σ1⎛
⎝|
u1
|⎞⎠(
−v
T
1−)
u
r
|⎞⎠(
−v
T
r−)
(12.54)
to the matrix as follows:
X≈U:,1:LS1:L,1:LVT
:,1:L (12.55)
This is called a truncated SVD (see Figure 12.8(b)). The total number of parameters needed to
represent an N×Dmatrix using a rank Lapproximation is
NL+LD+L=L(N+D+1) (12.56)

394 Chapter 12. Latent linear models
rank 200
(a)
rank 2
(b)
rank 5
(c)
rank 20
(d)
Figure 12.9 Low rank approximations to an image. Top left: The original image is of size 200×320,s o
has rank 200. Subsequent images have ranks 2, 5, and 20. Figure generated by svdImageDemo .
0 10 20 30 40 50 60 70 80 90 10045678910log(σi)
i  
original
randomized
Figure 12.10 First 50 log singular values for the clown image (solid red line), and for a data matrix
obtained by randomly shuffling the pixels (dotted green line). Figure generated by svdImageDemo .

12.2. Principal components analysis (PCA) 395
As an example, consider the 200×320pixel image in Figure 12.9(top left). This has 64,000
numbers in it. We see that a rank 20 approximation, with only (200+320+1) ×20 = 10, 420
numbers is a very good approximation.
One can show that the error in this approximation is given by
||X−XL||F≈σL+1 (12.57)
Furthermore, one can show that the SVD offers the best rank Lapproximation to a matrix (best
in the sense of minimizing the above Frobenius norm).
Let us connect this back to PCA. Let X=USVTbe a truncated SVD of X. We know that
ˆW=V, and that ˆZ=XˆW,s o
ˆZ=USVTV=US (12.58)
Furthermore, the optimal reconstruction is given by ˆX=ZˆWT,s ow eﬁ n d
ˆX=USVT(12.59)
This is precisely the same as a truncated SVD approximation! This is another illustration of the
fact that PCA is the best low rank approximation to the data.
12.2.4 Probabilistic PCA
We are now ready to revisit PPCA. One can show the following remarkable result.

Theorem 12.2.2 ((Tipping and Bishop 1999)). Consider a factor analysis model in which Ψ=σ2I
andWis orthogonal. The observed data log likelihood is given by
logp(X|W,σ2)=−N
2ln|C|−1
2N∑
i=1xT
iC−1xi=−N
2ln|C|+tr(C−1ˆΣ)(12.60)
whereC=WWT+σ2IandS=1
N∑N
i=1xixT
i=( 1/N)XTX. (We are assuming centered
data, for notational simplicity.) The maxima of the log-likelihood are given by
ˆW=V(Λ−σ2I)1
2R (12.61)
whereRis an arbitrary L×Lorthogonal matrix, Vis theD×Lmatrix whose columns are the

2ln|C|−1
2N∑
i=1xT
iC−1xi=−N
N∑N
i=1xixT
2R (12.61)
ﬁrstLeigenvectors of S, andΛis the corresponding diagonal matrix of eigenvalues. Without loss
of generality, we can set R=I. Furthermore, the MLE of the noise variance is given by
ˆσ2=1
D−LD∑
j=L+1λj (12.62)
which is the average variance associated with the discarded dimensions.
Thus, asσ2→0,w eh a v e ˆW→V, as in classical PCA. What about ˆZ? It is easy to see that
the posterior over the latent factors is given by
p(zi|xi,ˆθ)=N (zi|ˆF−1ˆWTxi,σ2ˆF−1) (12.63)
ˆF≜ˆWTˆW+ˆσ2I (12.64)

396 Chapter 12. Latent linear models
(Do not confuse F=WTW+σ2IwithC=WWT+σ2I.) Hence, as σ2→0, we ﬁnd
ˆW→V,ˆF→Iandˆzi→VTxi. Thus the posterior mean is obtained by an orthogonal
projection of the data onto the column space of V, as in classical PCA.
Note, however, that if σ2>, the posterior mean is not an orthogonal projection, since it is
shrunk somewhat towards the prior mean, as illustrated in Figure 12.5(b). This sounds like an
undesirable property, but it means that the reconstructions will be closer to the overall datamean,ˆμ=
x.
12.2.5 EM algorithm for PCA
Although the usual way to ﬁt a PCA model uses eigenvector methods, or the SVD, we can alsouse EM, which will turn out to have some advantages that we discuss below. EM for PCA relieson the probabilistic formulation of PCA. However the algorithm continues to work in the zeronoise limit, σ
2=0, as shown by (Roweis 1997).
Let˜Zbe aL×Nmatrix storing the posterior means (low-dimensional representations)

x.
along its columns. Similarly, let ˜X=XTstore the original data along its columns. From
Equation 12.63, when σ2=0,w eh a v e
˜Z=(WTW)−1WT˜X (12.65)
This constitutes the E step. Notice that this is just an orthogonal projection of the data.
From Equation 12.23, the M step is given by
ˆW=[∑
ixiE[zi]T][∑
iE[zi]E[zi]T]−1
(12.66)
where we exploited the fact that Σ=c o v[zi|xi,θ]=0Iwhenσ2=0. It is worth comparing
this expression to the MLE for multi-output linear regression (Equation 7.89), which has the formW=(∑
iyixT
i)(∑

ˆW=[∑
(12.66)
iyixT
i)(∑
ixixTi)−1. Thus we see that the M step is like linear regression where we
replace the observed inputs by the expected values of the latent variables.
In summary, here is the entire algorithm:
•Es t e p˜Z=(WTW)−1WT˜X
•Ms t e pW =˜X˜ZT(˜Z˜ZT)−1
(Tipping and Bishop 1999) showed that the only stable ﬁxed point of the EM algorithm is the
globally optimal solution. That is, the EM algorithm converges to a solution where Wspans
the same linear subspace as that deﬁned by the ﬁrst Leigenvectors. However, if we want W

to be orthogonal, and to contain the eigenvectors in descending order of eigenvalue, we have
to orthogonalize the resulting matrix (which can be done quite cheaply). Alternatively, we canmodify EM to give the principal basis directly (Ahn and Oh 2003).
This algorithm has a simple physical analogy in the case D=2andL=1(Roweis 1997).
Consider some points in R
2attached by springs to a rigid rod, whose orientation is deﬁned by a
vectorw.L e tzibe the location where the i’th spring attaches to the rod. In the E step, we hold

the rod ﬁxed, and let the attachment points slide around so as to minimize the spring energy(which is proportional to the sum of squared residuals). In the M step, we hold the attachment

12.2. Principal components analysis (PCA) 397
−3 −2 −1 0 1 2 3−2.5−2−1.5−1−0.500.511.522.5E step 1
(a)−3 −2 −1 0 1 2 3−2.5−2−1.5−1−0.500.511.522.5M step 1
(b)
−3 −2 −1 0 1 2 3−2.5−2−1.5−1−0.500.511.522.5E step 2
(c)−3 −2 −1 0 1 2 3−3−2−10123M step 2
(d)
Figure 12.11 Illustration of EM for PCA when D=2andL=1. Green stars are the original data points,
black circles are their reconstructions. The weight vector wis represented by blue line. (a) We start with
a random initial guess of w. The E step is represented by the orthogonal projections. (b) We update the
rodwin the M step, keeping the projections onto the rod (black circles) ﬁxed. (c) Another E step. The
black circles can ’slide’ along the rod, but the rod stays ﬁxed. (d) Another M step. Based on Figure 12.12 of
(Bishop 2006b). Figure generated by pcaEmStepByStep .
points ﬁxed and let the rod rotate so as to minimize the spring energy. See Figure 12.11 for an
illustration.

Apart from this pleasing intuitive interpretation, EM for PCA has the following advantages
over eigenvector methods:
• EM can be faster. In particular, assuming N,D≫L, the dominant cost of EM is the pro-
jection operation in the E step, so the overall time is O(TLND),w h e r eTis the number of

398 Chapter 12. Latent linear models
(a) (b) (c) (d) (e) (f)
Figure 12.12 Illustration of estimating the effective dimensionalities in a mixture of factor analysers using
VBEM. The blank columns have been forced to 0 via the ARD mechanism. The data was generated from
6 clusters with intrinsic dimensionalities of 7,4,3,2,2,1, which the method has successfully estimated.
Source: Figure 4.4 of (Beal 2003). Used with kind permission of Matt Beal.
iterations. (Roweis 1997) showed experimentally that the number of iterations is usually very
small (the mean was 3.6), regardless of NorD. (This results depends on the ratio of eigenval-
ues of the empirical covariance matrix.) This is much faster than the O(min(ND2,DN2))
time required by straightforward eigenvector methods, although more sophisticated eigenvec-tor methods, such as the Lanczos algorithm, have running times comparable to EM.
• EM can be implemented in an online fashion, i.e., we can update our estimate of Was the
data streams in.

• EM can handle missing data in a simple way (see Section 12.1.6).
• EM can be extended to handle mixtures of PPCA/ FA models.• EM can be modiﬁed to variational EM or to variational Bayes EM to ﬁt more complex models.
12.3 Choosing the number of latent dimensions
In Section 11.5, we discussed how to choose the number of components Kin a mixture model.
In this section, we discuss how to choose the number of latent dimensions Lin a FA/PCA model.
12.3.1 Model selection for FA/PPCA

If we use a probabilistic model, we can in principle compute L∗= argmaxLp(L|D). However,
there are two problems with this. First, evaluating the marginal likelihood for LVMs is quite
difficult. In practice, simple approximations, such as BIC or variational lower bounds (seeSection 21.5), can be used (see also (Minka 2000a)). Alternatively, we can use the cross-validatedlikelihood as a performance measure, although this can be slow, since it requires ﬁtting eachmodelFtimes, where Fis the number of CV folds.

approach is to perform exhaustive search over all candidate values of L. However, sometimes
we can set the model to its maximal size, and then use a technique called automatic relevancydetermination (Section 13.7), combined with EM, to automatically prune out irrelevant weights.

12.3. Choosing the number of latent dimensions 399
number of points 
  per cluster 1 7 4 3 2 2
  8        2       1  8      1      2 16 1       4 2 3 2 163322 6 4 1743221 2 8 174322intrinsic dimensionalities
Figure 12.13 We show the estimated number of clusters, and their estimated dimensionalities, as a
function of sample size. The VBEM algorithm found two different solutions when N=8. Note that more
clusters, with larger effective dimensionalities, are discovered as the sample sizes increases. Source: Table
4.1 of (Beal 2003). Used with kind permission of Matt Beal.
This technique will be described in a supervised context in Chapter 13, but can be adapted to
the (M)FA context as shown in (Bishop 1999; Ghahramani and Beal 2000).
Figure 12.12 illustrates this approach applied to a mixture of FAs ﬁt to a small synthetic dataset.
The ﬁgures visualize the weight matrices for each cluster, using Hinton diagrams, where where

the size of the square is proportional to the value of the entry in the matrix.2We see that
many of them are sparse. Figure 12.13 shows that the degree of sparsity depends on the amountof training data, in accord with the Bayesian Occam’s razor. In particular, when the samplesize is small, the method automatically prefers simpler models, but as the sample size getssufficiently large, the method converges on the “correct” solution, which is one with 6 subspacesof dimensionality 1, 2, 2, 3, 4 and 7.
Although the ARD/ EM method is elegant, it still needs to perform search over K. This is

done using “birth” and “death” moves (Ghahramani and Beal 2000). An alternative approach is toperform stochastic sampling in the space of models. Traditional approaches, such as (Lopes andWest 2004), are based on reversible jump MCMC, and also use birth and death moves. However,this can be slow and difficult to implement. More recent approaches use non-parametric priors,combined with Gibbs sampling, see e.g., (Paisley and Carin 2009).
12.3.2 Model selection for PCA

Since PCA is not a probabilistic model, we cannot use any of the methods described above. Anobvious proxy for the likelihood is the reconstruction error:
E(D,L)=1
|D|∑
i∈D||xi−ˆxi||2(12.67)
In the case of PCA, the reconstruction is given by by ˆxi=Wzi+μ,w h e r ezi=WT(xi−μ)
andWandμare estimated from Dtrain.
2. Geoff Hinton is an English professor of computer science at the University of Toronto.

400 Chapter 12. Latent linear models
0 100 200 300 400 5000102030405060rmse
num PCstrain set reconstruction error
(a)0 100 200 300 400 5000102030405060rmse
num PCstest set reconstruction error
(b)
Figure 12.14 Reconstruction error on MNIST vs number of latent dimensions used by PCA. (a) Training
set. (b) Test set. Figure generated by pcaOverfitDemo .
Figure 12.14(a) plots E(Dtrain,L)vsLon the MNIST training data in Figure 12.6. We see that
it drops off quite quickly, indicating that we can capture most of the empirical correlation of the
pixels with a small number of factors, as illustrated qualitatively in Figure 12.6.
Exercise 12.5 asks you to prove that the residual error from only using Lterms is given by the
sum of the discarded eigenvalues:
E(Dtrain,L)=D∑
j=L+1λj (12.68)
Therefore an alternative to plotting the error is to plot the retained eigenvalues, in decreasingorder. This is called a scree plot, because “the plot looks like the side of a mountain, and ’scree’

refers to the debris fallen from a mountain and lying at its base”.
3This will have the same shape
as the residual error plot.
A related quantity is the fraction of variance explained, deﬁned as
F(Dtrain,L)=∑L
j=1λj∑Lmax
j′=1λj′(12.69)
This captures the same information as the scree plot.
Of course, if we use L= rank(X) , we get zero reconstruction error on the training set.
To avoid overﬁtting, it is natural to plot reconstruction error on the test set. This is shown in

Figure 12.14(b). Here we see that the error continues to go down even as the model becomesmore complex! Thus we do not get the usual U-shaped curve that we typically expect to see.
What is going on? The problem is that PCA is not a proper generative model of the data.

It is merely a compression technique. If you give it more latent dimensions, it will be able toapproximate the test data more accurately. By contrast, a probabilistic model enjoys a BayesianOccam’s razor effect (Section 5.3.1), in that it gets “punished” if it wastes probability mass onparts of the space where there is little data. This is illustrated in Figure 12.15, which plots the
3. Quotation from http://janda .org/workshop/factoranalysis/SPSSrun/SPSS08 .htm.

12.3. Choosing the number of latent dimensions 401
0 100 200 300 400 5001.31.41.51.61.71.81.922.12.2x 106negloglik
num PCstrain set negative loglik
(a)0 100 200 300 400 5001.81.922.12.22.32.42.5x 106negloglik
num PCstest set negative loglik
(b)
Figure 12.15 Negative log likelihood on MNIST vs number of latent dimensions used by PPCA. (a) Training
negative log likelihood, computed using PPCA, vs L. Here, on the test set, we see the usual
U-shaped curve.
These results are analogous to those in Section 11.5.2, where we discussed the issue of choosing
Kin the K-means algorithm vs using a GMM.
12.3.2.1 Proﬁle likelihood
Although there is no U-shape, there is sometimes a “regime change” in the plots, from relatively
large errors to relatively small. One way to automate the detection of this is described in (Zhuand Ghodsi 2006). The idea is this. Let λ
kbe some measure of the error incurred by a model of

sizek, such that λ1≥λ2≥···≥λ Lmax. In PCA, these are the eigenvalues, but the method can
also be applied to K-means. Now consider partitioning these values into two groups, dependingon whether k<Lork>L,w h e r eLis some threshold which we will determine. To measure
the quality of L, we will use a simple change-point model, where λ
k∼N(μ1,σ2)ifk≤L,
andλk∼N(μ2,σ2)ifk>L. (It is important that σ2be the same in both models, to prevent
overﬁtting in the case where one regime has less data than the other.) Within each of the tworegimes, we assume the λ

kare iid, which is obviously incorrect, but is adequate for our present
purposes. We can ﬁt this model for each L=1:Lmaxby partitioning the data and computing
the MLEs, using a pooled estimate of the variance:
μ1(L)=∑
k≤Lλk
L,μ2(L)=∑
k>Lλk
N−L(12.70)
σ2(L)=∑
k≤L(λk−μ1(L))2+∑
k>L(λk−μ2(L))2
N(12.71)
We can then evaluate the proﬁle log likelihood
ℓ(L)=L∑
k=1logN(λk|μ1(L),σ2(L))+K∑
k=L+1logN(λk|μ2(L),σ2(L)) (12.72)
Finally, we choose L∗= argmax ℓ(L). This is illustrated in Figure 12.16. On the left, we plot

μ1(L)=∑
k≤Lλk
L,μ2(L)=∑
k>Lλk
σ2(L)=∑
N(12.71)
ℓ(L)=L∑
the scree plot, which has the same shape as in Figure 12.14(a). On the right, we plot the proﬁle

402 Chapter 12. Latent linear models
0 10 20 30 40 5000.511.522.533.54x 105
num PCseigenvaluescree plot
(a)0 10 20 30 40 50−5750−5700−5650−5600−5550−5500−5450
num PCsprofile log likelihood
(b)
Figure 12.16 (a) Scree plot for training set, corresponding to Figure 12.14(a). (b) Proﬁle likelihood. Figure
generated by pcaOverfitDemo .
likelihood. Rather miraculously, we see a fairly well-determined peak.
12.4 PCA for categorical data
In this section, we consider extending the factor analysis model to the case where the observed
data is categorical rather than real-valued. That is, the data has the form yij∈{1,...,C},
wherej=1:Ris the number of observed response variables. We assume each yijis generated
from a latent variable zi∈RL, with a Gaussian prior, which is passed through the softmax
function as follows:
p(zi)=N (0,I) (12.73)
p(yi|zi,θ)=R∏
r=1Cat(yir|S(WT
rzi+w0r)) (12.74)
whereWr∈RL×Mis the factor loading matrix for response j, andw0r∈RMis the offset
term for response r, andθ=(Wr,w0r)R

r=1. (We need an explicit offset term, since clamping
one element of zito 1 can cause problems when computing the posterior covariance.) As in
factor analysis, we have deﬁned the prior mean to be m0=0and the prior covariance V0=I,
since we can capture non-zero mean by changing w0jand non-identity covariance by changing
Wr. We will call this categorical PCA. See Chapter 27 for a discussion of related models.
It is interesting to study what kinds of distributions we can induce on the observed variables

by varying the parameters. For simplicity, we assume there is a single ternary response variable,
soyilives in the 3d probability simplex. Figure 12.17 shows what happens when we vary the
parameters of the prior, m0andV0, which is equivalent to varying the parameters of the
likelihood, W1andw01. We see that this can deﬁne fairly complex distributions over the
simplex. This induced distribution is known as the logistic normal distribution (Aitchison
1982).

1982).
We can ﬁt this model to data using a modiﬁed version of EM. The basic idea is to infer
a Gaussian approximation to the posterior p(zi|yi,θ)in the E step, and then to maximize θ
in the M step. The details for the multiclass case, can be found in (Khan et al. 2010) (see

12.4. PCA for categorical data 403
Figure 12.17 Some examples of the logistic normal distribution deﬁned on the 3d simplex. (a) Diagonal
covariance and non-zero mean. (b) Negative correlation between states 1 and 2. (c) Positive correlation
between states 1 and 2. Source: Figure 1 of (Blei and Lafferty 2007). Used with kind permission of David
Blei.
2 4 6 8 10 12 14 1620
40
60
80
100
120
140
(a)−1.5 −1 −0.5 0 0.5 1 1.5−1.5−1−0.500.511.52
(b)
Figure 12.18 Left: 150 synthetic 16 dimensional bit vectors. Right: the 2d embedding learned by binary
PCA, using variational EM. We have color coded points by the identity of the true “prototype” that generated
them. Figure generated by binaryFaDemoTipping .
also Section 21.8.1.1). The details for the binary case for the the sigmoid link can be found in
Exercise 21.9, and for the probit link in Exercise 21.10.
One application of such a model is to visualize high dimensional categorical data. Fig-

(b)
ure 12.18(a) shows a simple example where we have 150 6-dimensional bit vectors. It is clear that
each sample is just a noisy copy of one of three binary prototypes. We ﬁt a 2d catFA to this
model, yielding approximate MLEs ˆθ. In Figure 12.18(b), we plot E[
zi|xi,ˆθ]
. We see that there
are three distinct clusters, as is to be expected.
In (Khan et al. 2010), we show that this model outperforms ﬁnite mixture models on the task

zi|xi,ˆθ]
of imputing missing entries in design matrices consisting of real and categorical data. This is
useful for analysing social science survey data, which often has missing data and variables of
mixed type.

404 Chapter 12. Latent linear models
yixiziwyWx
N
(a)xiyizs
i zxi
BxWxWy
N
(b)
xiyizs
i zxi zy
i
Bx ByWxWy
N
(c)
Figure 12.19 Gaussian latent factor models for paired data. (a) Supervised PCA. (b) Partial least squares.
(c) Canonical correlation analysis.
12.5 PCA for paired and multi-view data
It is common to have a pair of related datasets, e.g., gene expression and gene copy number, or
movie ratings by users and movie reviews. It is natural to want to combine these together into alow-dimensional embedding. This is an example of data fusion. In some cases, we might want
to predict one element of the pair, say x
i1, from the other one, xi2, via the low-dimensional
“bottleneck”.
Below we discuss various latent Gaussian models for these tasks, following the presentation
of (Virtanen 2010). The models easily generalize from pairs to sets of data, xim,f o rm=1:M.
We focus on the case where xim∈RDm. In this case, the joint distribution is multivariate

Gaussian, so we can easily ﬁt the models using EM, or Gibbs sampling.
We can generalize the models to handle discrete and count data by using the exponential
family as a response distribution instead of the Gaussian, as we explain in Section 27.2.2.However, this will require the use of approximate inference in the E step (or an analogousmodiﬁcation to MCMC).

12.5. PCA for paired and multi-view data 405
12.5.1 Supervised PCA (latent factor regression)
Consider the following model, illustrated in Figure 12.19(a):
p(zi)=N (0,IL) (12.75)
p(yi|zi)=N (wT
yzi+μy,σ2
y) (12.76)
p(xi|zi)=N (Wxzi+μx,σ2
xID) (12.77)
In (Yu et al. 2006), this is called supervised PCA. In (West 2003), this is called Bayesian factor
regression. This model is like PCA, except that the target variable yiis taken into account when
learning the low dimensional embedding. Since the model is jointly Gaussian, we have
yi|xi∼N(xT
iw,σ2
y+wT
yCwy) (12.78)
wherew=Ψ−1WxCwy,Ψ=σ2
xID, andC−1=I+WT
xΨ−1Wx. So although this is a
joint density model of (yi,xi), we can infer the implied conditional distribution.
We now show an interesting connection to Zellner’s g-prior. Suppose p(wy)=N(0,1
gΣ2),
and letX=RVTbe the SVD of X,w h e r e VTV=IandRTR=Σ2=d i a g (σ2
j)contains
the squared singular values. Then one can show (West 2003) that
p(w)=N(0,gV−TΣ−2V−1)=N(0,g(XTX)−1) (12.79)

iw,σ2
y+wT
gΣ2),
So the dependence of the prior for wonXarises from the fact that wis derived indirectly by
a joint model of Xandy.
The above discussion focussed on regression. (Guo 2009) generalizes CCA to the exponential
family, which is more appropriate if xiand/oryiare discrete. Although we can no longer
compute the conditional p(yi|xi,θ)in closed form, the model has a similar interpretation to
the regression case, namely that we are predicting the response via a latent “bottleneck”.
The basic idea of compressing xito predict yican be formulated using information theory.

In particular, we might want to ﬁnd an encoding distribution p(z|x)such that we minimize
I(X;Z)−βI(X;Y) (12.80)
whereβ≥0is some parameter controlling the tradeoff between compression and predictive
accuracy. This is known as the information bottleneck (Tishby et al. 1999). Often Zis taken to
be discrete, as in clustering. However, in the Gaussian case, IB is closely related to CCA (Chechik
et al. 2005).
We can easily generalize CCA to the case where yiis a vector of responses to be predicted, as

in multi-label classiﬁcation. (Ma et al. 2008; Williamson and Ghahramani 2008) used this modelto perform collaborative ﬁltering, where the goal is to predict y
ij∈{1,...,5}, the rating person
igives to movie j, where the “side information” xitakes the form of a list of i’s friends. The

intuition behind this approach is that knowledge of who your friends are, as well as the ratingsof all other users, should help predict which movies you will like. In general, any setting wherethe tasks are correlated could beneﬁt from CCA. Once we adopt a probabilistic view, variousextensions are straightforward. For example, we can easily generalize to the semi-supervisedcase, where we do not observe y
ifor alli(Yu et al. 2006).

406 Chapter 12. Latent linear models
12.5.1.1 Discriminative supervised PCA
One problem with this model is that it puts as much weight on predicting the inputs xias the
outputsyi. This can be partially alleviated by using a weighted objective of the following form
(Rish et al. 2008):
ℓ(θ)=∏
ip(yi|ηiy)αyp(xi|ηix)αx(12.81)
where the αmcontrol the relative importance of the data sources, and ηim=Wmzi.F o r
Gaussian data, we can see that αmjust controls the noise variance:
ℓ(θ)∝∏
iexp(−1
2αx||xT
i−ηix||2)exp(−1
2αy||yT
i−ηiy||2) (12.82)
This interpretation holds more generally for the exponential family. Note, however, that it is hard
to estimate the αmparameters, because changing them changes the normalization constant of
the likelihood. We give an alternative approach to weighting ymore heavily below.
12.5.2 Partial least squares
The technique of partial least squares (PLS) (Gustafsson 2001; Sun et al. 2009) is an asym-

ℓ(θ)∝∏
iexp(−1
2αx||xT
2αy||yT
metric or more “discriminative” form of supervised PCA. The key idea is to allow some of the(co)variance in the input features to be explained by its own subspace, z
x
i, and to let the rest of
the subspace, zsi, be shared between input and output. The model has the form
p(zi)=N (zs
i|0,ILs)N(zxi|0,ILx) (12.83)
p(yi|zi)=N (Wyzsi+μy,σ2IDy) (12.84)
p(xi|zi)=N (Wxzsi+Bxzxi+μx,σ2IDx) (12.85)
See Figure 12.19(b). The corresponding induced distribution on the visible variables has the form
p(vi|θ)=∫
N(vi|Wzi+μ,σ2I)N(zi|0,I)dzi=N(vi|μ,WWT+σ2I) (12.86)

x
p(vi|θ)=∫
wherevi=(xi;yi),μ=(μy;μx)and
W=(Wy0
WxBx)
(12.87)
WWT=(WyWT
yWxWT
x
WxWT
xWxWT
x+BxBT
x)
(12.88)
We should choose Llarge enough so that the shared subspace does not capture covariate-
speciﬁc variation.
This model can be easily generalized to discrete data using the exponential family (Virtanen
2010).

12.6. Independent Component Analysis (ICA) 407
12.5.3 Canonical correlation analysis
Canonical correlation analysis orCCAis like a symmetric unsupervised version of PLS: it
allows each view to have its own “private” subspace, but there is also a shared subspace. If we
have two observed variables, xiandyi, then we have three latent variables, zs
i∈RL0which is
shared,zxi∈RLxandzy
i∈RLywhich are private. We can write the model as follows (Bach
and Jordan 2005):
i|0,ILs)N(zxi|0,ILx)N(zy
i|0,ILy) (12.89)
p(xi|zi)=N (xi|Bxzx
i+Wxzsi+μx,σ2IDx) (12.90)
p(yi|zi)=N (yi|Byzy
i+Wyzs
i+μy,σ2IDy) (12.91)
See Figure 12.19(c). The corresponding observed joint distribution has the form
p(vi|θ)=∫
N(vi|Wzi+μ,σ2I)N(zi|0,I)dzi=N(vi|μ,WWT+σ2ID)(12.92)
where
W=(WxBx0
Wy0B y)
(12.93)
WWT=(WxWT
x+BxBT
xWxWT
y
WyWT
yWyWT
y+ByBTy)
(12.94)
One can compute the MLE for this model using EM. (Bach and Jordan 2005) show that the

i+Wyzs
p(vi|θ)=∫
where
W=(WxBx0
Wy0B y)
(12.93)
WWT=(WxWT
x+BxBT
xWxWT
y
WyWT
yWyWT
y+ByBTy)
(12.94)
resulting MLE is equivalent (up to rotation and scaling) to the classical, non-probabilistic view.However, the advantages of the probabilistic view are many: we can trivially generalize to M>2

p(vi|θ)=∫
where
W=(WxBx0
Wy0B y)
(12.93)
WWT=(WxWT
x+BxBT
xWxWT
y
WyWT
yWyWT
y+ByBTy)
(12.94)
observed variables; we can create mixtures of CCA (Viinikanoja et al. 2010); we can create sparseversions of CCA using ARD (Archambeau and Bach 2008); we can generalize to the exponentialfamily (Klami et al. 2010); we can perform Bayesian inference of the parameters (Wang 2007;Klami and Kaski 2008); we can handle non-parametric sparsity-promoting priors for WandB
(Rai and Daume 2009); and so on.
12.6 Independent Component Analysis (ICA)

Consider the following situation. You are in a crowded room and many people are speaking.Your ears essentially act as two microphones, which are listening to a linear combination of thedifferent speech signals in the room. Your goal is to deconvolve the mixed signals into theirconstituent parts. This is known as the cocktail party problem, and is an example of blind
signal separation (BSS), orblind source separation, where “blind” means we know “nothing”

about the source of the signals. Besides the obvious applications to acoustic signal processing,this problem also arises when analysing EEG and MEG signals, ﬁnancial data, and any otherdataset (not necessarily temporal) where latent sources or factors get mixed together in a linearway.
We can formalize the problem as follows. Let x
t∈RDbe the observed signal at the sensors
at “time” t, andzt∈RLbe the vector of source signals. We assume that
xt=Wzt+ϵt (12.95)

408 Chapter 12. Latent linear models
0 100 200 300 400 500−202
0 100 200 300 400 500−505
0 100 200 300 400 500−202truth
0 100 200 300 400 500−10010
(a)0 100 200 300 400 500−10010
0 100 200 300 400 500−10010observed signals
(b)
0 100 200 300 400 500−202PCA estimate
0 100 200 300 400 500−101
(c)0 100 200 300 400 500−505
0 100 200 300 400 500−202ICA estimate
(d)
Figure 12.20 Illustration of ICA applied to 500 iid samples of a 4d source signal. (a) Latent signals. (b)
Observations. (c) PCA estimate. (d) ICA estimate. Figure generated by icaDemo, written by Aapo Hyvarinen.
whereWis anD×Lmatrix, and ϵt∼N(0,Ψ). In this section, we treat each time point
as an independent observation, i.e., we do not model temporal correlation (so we could replace
thetindex with i, but we stick with tto be consistent with much of the ICA literature). The

(d)
goal is to infer the source signals, p(zt|xt,θ), as illustrated in Figure 12.20. In this context, W
is called the mixing matrix.I f L=D(number of sources = number of sensors), it will be a
square matrix. Often we will assume the noise level, |Ψ|, is zero, for simplicity.
So far, the model is identical to factor analysis (or PCA if there is no noise, except we don’t in
general require orthogonality of W). However, we will use a different prior for p(zt). In PCA,

we assume each source is independent, and has a Gaussian distribution
p(zt)=L∏
j=1N(ztj|0,1) (12.96)
We will now relax this Gaussian assumption and let the source distributions be any non-Gaussian

12.6. Independent Component Analysis (ICA) 409
−3 −2 −1 0 1 2 3−3−2−10123uniform data
(a)−3 −2 −1 0 1 2 3−3−2−10123uniform data after linear mixing
(b)
−3 −2 −1 0 1 2 3−3−2−10123PCA applied to mixed data from uniform source
(c)−3 −2 −1 0 1 2 3−3−2−10123ICA applied to mixed data from uniform source
(d)
Figure 12.21 Illustration of ICA and PCA applied to 100 iid samples of a 2d source signal with a uniform
distribution. (a) Latent signals. (b) Observations. (c) PCA estimate. (d) ICA estimate. Figure generated by
icaDemoUniform , written by Aapo Hyvarinen.
distribution
p(zt)=L∏
j=1pj(ztj) (12.97)
Without loss of generality, we can constrain the variance of the source distributions to be 1,
because any other variance can be modelled by scaling the rows of Wappropriately. The
resulting model is known as independent component analysis orICA.
The reason the Gaussian distribution is disallowed as a source prior in ICA is that it does not

p(zt)=L∏
permit unique recovery of the sources, as illustrated in Figure 12.20(c). This is because the PCAlikelihood is invariant to any orthogonal transformation of the sources z
tand mixing matrix W.
PCA can recover the best linear subspace in which the signals lie, but cannot uniquely recoverthe signals themselves.

410 Chapter 12. Latent linear models
To illustrate this, suppose we have two independent sources with uniform distributions, as
shown in Figure 12.21(a). Now suppose we have the following mixing matrix
W=(
23
21)
(12.98)
Then we observe the data shown in Figure 12.21(b) (assuming no noise). If we apply PCA followedby scaling to this, we get the result in Figure 12.21(c). This corresponds to a whitening of thedata. To uniquely recover the sources, we need to perform an additional rotation. The troubleis, there is no information in the symmetric Gaussian posterior to tell us which angle to rotateby. In a sense, PCA solves “half” of the problem, since it identiﬁes the linear subspace; allthat ICA has to do is then to identify the appropriate rotation. (Hence we see that ICA is notthat different from methods such as varimax, which seek good rotations of the latent factors toenhance interpretability.)
Figure 12.21(d) shows that ICA can recover the source, up to a permutation of the indices and

possible sign change. ICA requires that Wis square and hence invertible. In the non-square
case (e.g., where we have more sources than sensors), we cannot uniquely recover the true signal,but we can compute the posterior p(z
t|xt,ˆW), which represents our beliefs about the source.
In both cases, we need to estimate Was well as the source distributions pj. We discuss how
to do this below.
12.6.1 Maximum likelihood estimation
In this section, we discuss ways to estimate square mixing matrices Wfor the noise-free ICA
model. As usual, we will assume that the observations have been centered; hence we can alsoassumezis zero-mean. In addition, we assume the observations have been whitened, which
can be done with PCA.
If the data is centered and whitened, we have E[
xx
T]
=I. But in the noise free case, we
also have
cov[x]=E[
xxT]
=WE[
zzT]
WT=WWT(12.99)

xx
T]
also have
cov[x]=E[
xxT]
=WE[
zzT]
Hence we see that Wmust be orthogonal. This reduces the number of parameters we have to
estimate from D2toD(D−1)/2. It will also simplify the math and the algorithms.
LetV=W−1; these are often called the recognition weights, as opposed to W, which are
thegenerative weights.4
Sincex=Wz, we have, from Equation 2.89,
px(Wzt)=pz(zt)|det(W−1)|=pz(Vxt)|det(V)| (12.100)
Hence we can write the log-likelihood, assuming Tiid samples, as follows:
1
Tlogp(D|V)=l o g|det(V)|+1
TL∑
j=1T∑
t=1logpj(vT
jxt) (12.101)

1
TL∑
j=1T∑
4. In the literature, it is common to denote the generative weights by Aand the recognition weights by W, but we are
trying to be consistent with the notation used earlier in this chapter.

12.6. Independent Component Analysis (ICA) 411
wherevjis thej’th row of V. Since we are constraining Vto be orthogonal, the ﬁrst term is a
constant, so we can drop it. We can also replace the average over the data with an expectation
operator to get the following objective
NLL(V)=L∑
j=1E[Gj(zj)] (12.102)
wherezj=vT
jxandGj(z)≜−logpj(z). We want to minimize this subject to the constraint
that the rows of Vare orthogonal. We also want them to be unit norm, since this ensures
that the variance of the factors is unity (since, with whitened data, E[
vT
jx]
=||vj||2), which is
necessary to ﬁx the scale of the weights. In otherwords, Vshould be an orthonormal matrix.
It is straightforward to derive a gradient descent algorithm to ﬁt this model; however, it

NLL(V)=L∑
vT
jx]
is rather slow. One can also derive a faster algorithm that follows the natural gradient; seee.g., (MacKay 2003, ch 34) for details. A popular alternative is to use an approximate Newtonmethod, which we discuss in Section 12.6.2. Another approach is to use EM, which we discussin Section 12.6.3.
12.6.2 The FastICA algorithm
We now describe the fast ICA algorithm, based on (Hyvarinen and Oja 2000), which we will
show is an approximate Newton method for ﬁtting ICA models.

For simplicity of presentation, we initially assume there is only one latent factor. In addition,
we initially assume all source distributions are known and are the same, so we can just writeG(z)=−logp(z).L e tg(z)=
d
dzG(z). The constrained objective, and its gradient and
Hessian, are given by
f(v)=E[
G(vTx)]
+λ(1−vTv) (12.103)
∇f(v)=E[
xg(vTx)]
−βv (12.104)
H(v)=E[
xxTg′(vTx)]
−βI (12.105)
whereβ=2λis a Lagrange multiplier. Let us make the approximation
E[
≈E[
xxT]
E[
g′(vTx)]
=E[
g′(vTx)]
(12.106)

d
f(v)=E[
G(vTx)]
∇f(v)=E[
xg(vTx)]
H(v)=E[
E[
≈E[
xxT]
E[
g′(vTx)]
=E[
g′(vTx)]
(12.106)
This makes the Hessian very easy to invert, giving rise to the following Newton update:
v∗≜v−E[
xg(vTx)]
−βv
E[g′(vTx)]−β(12.107)
One can rewrite this in the following way
v∗≜E[
xg(vTx)]
−E[
g′(vTx)]
v (12.108)
(In practice, the expectations can be replaced by Monte Carlo estimates from the training set,which gives an efficient online learning algorithm.) After performing this update, one shouldproject back onto the constraint surface using
v
new≜v∗
||v∗||(12.109)

412 Chapter 12. Latent linear models
−4 −3 −2 −1 0 1 2 3 400.050.10.150.20.250.30.350.40.450.5
(a)−4 −3 −2 −1 0 1 2 3 4−3−2−10123Gaussian
(b)
−10 −5 0 5 10−8−6−4−20246810Laplace
(c)−2 −1 0 1 2−1.5−1−0.500.511.5Uniform
(d)
Figure 12.22 Illustration of Gaussian, sub-Gaussian (uniform) and super-Gaussian (Laplace) distributions
in 1d and 2d. Figure generated by subSuperGaussPlot , written by Kevin Swersky.
One iterates this algorithm until convergence. (Due to the sign ambiguity of v, the values of v
may not converge, but the direction deﬁned by this vector should converge, so one can assess
convergence by monitoring |vTvnew|, which should approach 1.)
Since the objective is not convex, there are multiple local optima. We can use this fact to
learn multiple different weight vectors or features. We can either learn the features sequentially
and then project out the part of vjthat lies in the subspace deﬁned by earlier features, or

we can learn them in parallel, and orthogonalize Vin parallel. This latter approach is usually
preferred, since, unlike PCA, the features are not ordered in any way. So the ﬁrst feature is not“more important” than the second, and hence it is better to treat them symmetrically.

12.6. Independent Component Analysis (ICA) 413
12.6.2.1 Modeling the source densities
So far, we have assumed that G(z)=−logp(z)is known. What kinds of models might be
reasonable as signal priors? We know that using Gaussians (which correspond to quadratic
functions for G) won’t work. So we want some kind of non-Gaussian distribution. In general,
there are several kinds of non-Gaussian distributions, such as the following:
•Super-Gaussian distributions These are distributions which have a big spike at the mean,
and hence (in order to ensure unit variance) have heavy tails. The Laplace distribution is
a classic example. See Figure 12.22. Formally, we say a distribution is super-Gaussian or
leptokurtic (“lepto” coming from the Greek for “thin”) if kurt(z)>0,w h e r e kurt(z)is the
kurtosis of the distribution, deﬁned by
kurt(z)≜μ4
σ4−3 (12.110)
whereσis the standard deviation, and μkis thek’thcentral moment, or moment about
the mean:
μk≜E[
(X−E[X])k]
(12.111)

the mean:
μk≜E[
(12.111)
(Soμ1=μis the mean, and μ2=σ2is the variance.) It is conventional to subtract 3 in the
deﬁnition of kurtosis to make the kurtosis of a Gaussian variable equal to zero.
•Sub-Gaussian distributions Asub-Gaussian orplatykurtic (“platy” coming from the Greek
for “broad”) distribution has negative kurtosis. These are distributions which are much ﬂatterthan a Gaussian. The uniform distribution is a classic example. See Figure 12.22.
•Skewed distributions Another way to “be non-Gaussian” is to be asymmetric. One measure
of this is skewness, deﬁned by
skew(z)≜μ
3

skew(z)≜μ
3
σ3(12.112)
An example of a (right) skewed distribution is the gamma distribution (see Figure 2.9).
When one looks at the empirical distribution of many natural signals, such as images and

skew(z)≜μ
3
speech, when passed through certain linear ﬁlters, they tend to be very super-Gaussian. Thisresult holds both for the kind of linear ﬁlters found in certain parts of the brain, such as thesimple cells found in the primary visual cortex, as well as for the kinds of linear ﬁlters used insignal processing, such as wavelet transforms. One obvious choice for modeling natural signalswith ICA is therefore the Laplace distribution. For mean zero and variance 1, this has a log pdfgiven by
logp(z)=−√
2|z|−log(√
2) (12.113)

Since the Laplace prior is not differentiable at the origin, it is more common to use other,smoother super-Gaussian distributions. One example is the logistic distribution. The corre-sponding log pdf, for the case where the mean is zero and the variance is 1 (so μ=0and
s=
√
3
π), is given by the following:
logp(z)=−2logcosh(π
2√
3z)−log4√
3
π(12.114)

414 Chapter 12. Latent linear models
ψTxt1 xtDzt1 ztDqt1 qtD
Wσ1kμ1k
π
σDkμDk
Figure 12.23 Modeling the source distributions using a mixture of univariate Gaussians (the independent
factor analysis model of (Moulines et al. 1997; Attias 1999)).
Various ways of estimating G(Z)=−logp(z)are discussed in the seminal paper (Pham and
Garrat 1997). However, when ﬁtting ICA by maximum likelihood, it is not critical that the exact
shape of the source distribution be known (although it is important to know whether it is subor super Gaussian). Consequently, it is common to just use G(z)=√
zorG(z)=l o gc o s h ( z)
instead of the more complex expressions above.
12.6.3 Using EM
An alternative to assuming a particular form for G(z), or equivalently for p(z), is to use a
ﬂexible non-parametric density estimator, such as a mixture of (uni-variate) Gaussians:
p(qj=k)=π k (12.115)
p(zj|qj=k)=N (μj,k,σ2
j,k) (12.116)
p(x|z)=N (Wz,Ψ) (12.117)

This approach was proposed in (Moulines et al. 1997; Attias 1999), and the corresponding graph-ical model is shown in Figure 12.23.
It is possible to derive an exact EM algorithm for this model. The key observation is that
it is possible to compute E[z
t|xt,θ]exactly by summing over all KLcombinations of the qt
variables, where Kis the number of mixture components per source. (If this is too expensive,

one can use a variational mean ﬁeld approximation (Attias 1999).) We can then estimate all thesource distributions in parallel by ﬁtting a standard GMM to E[z
t]. When the source GMMs are

12.6. Independent Component Analysis (ICA) 415
known, we can compute the marginals pj(zj)very easily, using
pj(zj)=K∑
k=1πj,kN(zj|μj,k,σ2
j,k) (12.118)
Given the pj’s, we can then use an ICA algorithm to estimate W. Of course, these steps should
be interleaved. The details can be found in (Attias 1999).
12.6.4 Other estimation principles *
It is quite common to estimate the parameters of ICA models using methods that seem different
to maximum likelihood. We will review some of these methods below, because they giveadditional insight into ICA. However, we will also see that these methods in fact are equivalentto maximum likelihood after all. Our presentation is based on (Hyvarinen and Oja 2000).
12.6.4.1 Maximizing non-Gaussianity
An early approach to ICA was to ﬁnd a matrix Vsuch that the distribution z=Vxis as far
from Gaussian as possible. (There is a related approach in statistics called projection pursuit.)

One measure of non-Gaussianity is kurtosis, but this can be sensitive to outliers. Anothermeasure is the negentropy, deﬁned as
negentropy( z)≜H(
N(μ,σ
2))
−H(z) (12.119)
whereμ=E[z]andσ2= var[z]. Since the Gaussian is the maximum entropy distribution,
this measure is always non-negative and becomes large for distributions that are highly non-Gaussian.
We can deﬁne our objective as maximizing
J(V)=∑
jnegentropy( zj)=∑
jH(
N(μj,σ2
j))
−H(zj) (12.120)

N(μ,σ
2))
J(V)=∑
jH(
N(μj,σ2
j))
wherez=Vx.I fw eﬁ x Vto be orthogonal, and if we whiten the data, the covariance of z
will beIindependently of V, so the ﬁrst term is a constant. Hence
J(V)=∑
j−H(zj)+const=∑
jE[logp(zj)]+const (12.121)
which we see is equal (up to a sign change, and irrelevant constants) to the log-likelihood inEquation 12.102.
12.6.4.2 Minimizing mutual information
One measure of dependence of a set of random variables is the multi-information:
I(z)≜KL⎛
⎝p(z)||∏
jp(zj)⎞⎠=∑
jH(zj)−H(z) (12.122)

416 Chapter 12. Latent linear models
We would like to minimize this, since we are trying to ﬁnd independent components. Put
another way, we want the best possible factored approximation to the joint distribution.
Now since z=Vx,w eh a v e
I(z)=∑
jH(zj)−H(Vx) (12.123)
If we constrain Vto be orthogonal, we can drop the last term, since then H(Vx)= H(x)
(since multiplying by Vdoes not change the shape of the distribution), and H(x)is a constant
which is is solely determined by the empirical distribution. Hence we have I(z)=∑
jH(zj).
Minimizing this is equivalent to maximizing the negentropy, which is equivalent to maximumlikelihood.
12.6.4.3 Maximizing mutual information (infomax)
Instead of trying to minimize the mutual information between the components of z,l e tu s
imagine a neural network where xis the input and y
j=φ(vT
jx)+ϵis the noisy output, where
φis some nonlinear scalar function, and ϵ∼N(0,1). It seems reasonable to try to maximize

jH(zj).
j=φ(vT
the information ﬂow through this system, a principle known as infomax. (Bell and Sejnowski
1995). That is, we want to maximize the mutual information between y(the internal neural
representation) and x(the observed input signal). We have I(x;y)=H(y)−H(y|x),w h e r e
the latter term is constant if we assume the noise has constant variance. One can show that wecan approximate the former term as follows
H(y)=L∑
j=1E[
logφ′(vT
jx)]
+log|det(V)| (12.124)

H(y)=L∑
j=1E[
logφ′(vT
jx)]
where, as usual, we can drop the last term if Vis orthogonal. If we deﬁne φ(z)to be a cdf,
thenφ′(z)is its pdf, and the above expression is equivalent to the log likelihood. In particular,
if we use a logistic nonlinearity, φ(z) = sigm( z), then the corresponding pdf is the logistic
distribution, and logφ′(z) = logcosh( z)(ignoring irrelevant constants). Thus we see that
infomax is equivalent to maximum likelihood.
Exercises
Exercise 12.1 Ms t e pf o rF A
For the FA model, show that the MLE in the M step for Wis given by Equation 12.23.

Exercises
Exercise 12.2 MAP estimation for the FA model
Derive the M step for the FA model using conjugate priors for the parameters.
Exercise 12.3 Heuristic for assessing applicability of PCA
(Source: (Press 2005, Q9.8).). Let the empirical covariance matrix Σhave eigenvalues λ1≥λ2≥···≥
λd>0. Explain why the variance of the evalues, σ2=1
d∑d
i=1(λi−λ)2is a good measure of whether
or not PCA would be useful for analysing the data (the higher the value of σ2the more useful PCA).

12.6. Independent Component Analysis (ICA) 417
Exercise 12.4 Deriving the second principal component
a. Let
J(v2,z2)=1
nn∑
i=1(xi−zi1v1−zi2v2)T(xi−zi1v1−zi2v2) (12.125)
Show that∂J
∂z2=0yieldszi2=vT
2xi.
b. Show that the value of v2that minimizes
˜J(v2)=−vT
2Cv2+λ2(vT
2v2−1)+λ12(vT
2v1−0) (12.126)
is given by the eigenvector of Cwith the second largest eigenvalue. Hint: recall that Cv1=λ1v1and
∂xTAx
∂x=(A+AT)x.
Exercise 12.5 Deriving the residual error for PCA
a. Prove that
||xi−K∑
j=1zijvj||2=xT
ixi−K∑
j=1vT
jxixTivj (12.127)
Hint: ﬁrst consider the case K=2. Use the fact that vT
jvj=1andvT
jvk=0fork̸=j. Also,
recallzij=xT
ivj.
b. Now show that
JK≜1
nn∑
i=1(
xT
ixi−K∑
j=1vT
jxixTivj)
=1
nn∑
i=1xTixi−K∑
j=1λj (12.128)
Hint: recall vT
jCvj=λjvT
jvj=λj.
c. IfK=dthere is no truncation, so Jd=0. Use this to show that the error from only using K<d
terms is given by
JK=d∑
j=K+1λj (12.129)
Hint: partition the sum∑d
j=1λjinto∑Kj=1λjand∑dj=K+1λj.
Exercise 12.6 Derivation of Fisher’s linear discriminant

ixi−K∑
j=1vT
ivj.
JK≜1
nn∑
i=1(
xT
ixi−K∑
j=1vT
jxixTivj)
=1
nn∑
jCvj=λjvT
jvj=λj.
JK=d∑
Show that the maximum of J(w)=wTSBw
wTSWwis given by SBw=λSWw
whereλ=wTSBw
wTSWw.Hint: recall that the derivative of a ratio of two scalars is given byd
dxf(x)
g(x)=f′g−fg′
g2,
wheref′=d
dxf(x)andg′=d
dxg(x). Also, recall thatd
dxxTAx=(A+AT)x.
Exercise 12.7 PCA via successive deﬂation
Letv1,v2,...,v kbe the ﬁrst keigenvectors with largest eigenvalues of C=1
nXTX, i.e., the principal
basis vectors. These satisfy
vT
jvk={0ifj̸=k
1ifj=k(12.130)
We will construct a method for ﬁnding the vjsequentially.

418 Chapter 12. Latent linear models
As we showed in class, v1is the ﬁrst principal eigenvector of C, and satisﬁes Cv1=λ1v1. Now deﬁne
˜xias the orthogonal projection of xionto the space orthogonal to v1:
˜xi=P⊥v1xi=(I−v1vT
1)xi (12.131)
Deﬁne˜X=[˜x1;...;˜xn]as thedeﬂated matrix of rankd−1, which is obtained by removing from the d
dimensional data the component that lies in the direction of the ﬁrst principal direction:
˜X=(I−v1vT
1)TX=(I−v1vT
1)X (12.132)
a. Using the facts that XTXv1=nλ1v1(and hence vT
1XTX=nλ1vT
1) andvT
1v1=1, show that
the covariance of the deﬂated matrix is given by
˜C≜1
n˜XT˜X=1
nXTX−λ1v1vT
1 (12.133)
b. Letube the principal eigenvector of ˜C. Explain why u=v2. (You may assume uis unit norm.)
c. Suppose we have a simple method for ﬁnding the leading eigenvector and eigenvalue of a pd matrix,
denoted by [λ,u]=f(C). Write some pseudo code for ﬁnding the ﬁrst Kprincipal basis vectors of
Xthat only uses the special ffunction and simple vector arithmetic, i.e., your code should not use

1) andvT
˜C≜1
n˜XT˜X=1
1 (12.133)
S V Do rt h eeig function. Hint: this should be a simple iterative routine that takes 2–3 lines to write.
The input is C,Kand the function f, the output should be vjandλjforj=1:K. Do not worry
about being syntactically correct.
Exercise 12.8 Latent semantic indexing
(Source: de Freitas.). In this exercise, we study a technique called latent semantic indexing , which applies
SVD to a document by term matrix, to create a low-dimensional embedding of the data that is designed to

capture semantic similarity of words.
The ﬁle lsiDocuments.pdf contains 9 documents on various topics. A list of all the 460 unique
words/terms that occur in these documents is in lsiWords.txt . A document by term matrix is in
lsiMatrix.txt .
a. LetXbe the transpose of lsiMatrix , so each column represents a document. Compute the SVD of X
and make an approximation to it ˆXusing the ﬁrst 2 singular values/ vectors. Plot the low dimensional
representation of the 9 documents in 2D. You should get something like Figure 12.24.

b. Consider ﬁnding documents that are about alien abductions. If If you look at lsiWords.txt , there
are 3 versions of this word, term 23 (“abducted”), term 24 (“abduction”) and term 25 (“abductions”).
Suppose we want to ﬁnd documents containing the word “abducted”. Documents 2 and 3 contain it,but document 1 does not. However, document 1 is clearly related to this topic. Thus LSI should alsoﬁnd document 1. Create a test document qcontaining the one word “abducted”, and project it into

the 2D subspace to make ˆq. Now compute the cosine similarity between ˆqand the low dimensional
representation of all the documents. What are the top 3 closest matches?
Exercise 12.9 Imputation in a FA model
Derive an expression for p(x
h|xv,θ)for a FA model.
Exercise 12.10 Efficiently evaluating the PPCA density
Derive an expression for p(x|ˆW,ˆσ2)for the PPCA model based on plugging in the MLEs and using the
matrix inversion lemma.

12.6. Independent Component Analysis (ICA) 419
−0.45 −0.4 −0.35 −0.3 −0.25 −0.2−0.8−0.6−0.4−0.200.20.40.6
12
34
5678
9
Figure 12.24 Projection of 9 documents into 2 dimensions. Figure generated by lsiCode.
Exercise 12.11 PPCA vs FA
(Source: Exercise 14.15 of (Hastie et al. 2009), due to Hinton.). Generate 200observations from the following
model, where zi∼N(0,I):xi1=zi1,xi2=zi1+0.001zi2,xi3=1 0zi3. Fit a FA and PCA model
with 1 latent factor. Hence show that the corresponding weight vector waligns with the maximal variance
direction (dimension 3) in the PCA case, but with the maximal correlation direction (dimensions 1+2) in the
case of FA.

13 Sparse linear models
13.1 Introduction
We introduced the topic of feature selection in Section 3.5.4, where we discussed methods for
ﬁnding input variables which had high mutual information with the output. The trouble withthis approach is that it is based on a myopic strategy that only looks at one variable at a time.This can fail if there are interaction effects. For example, if y=xor(x
1,x2), then neither x1nor
x2on its own can predict the response, but together they perfectly predict the response. For a
real-world example of this, consider genetic association studies: sometimes two genes on theirown may be harmless, but when present together they cause a recessive disease (Balding 2006).
In this chapter, we focus on selecting sets of variables at a time using a model-based approach.
If the model is a generalized linear model, of the form p(y|x)=p(y|f(w
Tx))for some link
functionf, then we can perform feature selection by encouraging the weight vector wto be

sparse, i.e., to have lots of zeros. This approach turns out to offer signiﬁcant computationaladvantages, as we will see below.
Here are some applications where feature selection/ sparsity is useful:
• In many problems, we have many more dimensions Dthan training cases N. The cor-
responding design matrix is short and fat, rather than tall and skinny. This is called thesmallN, largeDproblem. This is becoming increasingly prevalent as we develop more

high throughput measurement devices, For example, with gene microarrays, it is commonto measure the expression levels of D∼10,000genes, but to only get N∼100such
examples. (It is perhaps a sign of the times that even our data seems to be getting fatter...)We may want to ﬁnd the smallest set of features that can accurately predict the response(e.g., growth rate of the cell) in order to prevent overﬁtting, to reduce the cost of building adiagnostic device, or to help with scientiﬁc insight into the problem.

• In Chapter 14, we will use basis functions centered on the training examples, so φ(x)=
[κ(x,x
1),...,κ(x,xN)],w h e r e κis a kernel function. The resulting design matrix has size
N×N. Feature selection in this context is equivalent to selecting a subset of the training
examples, which can help reduce overﬁtting and computational cost. This is known as asparse kernel machine.
• In signal processing, it is common to represent signals (images, speech, etc.) in terms of

[κ(x,x
wavelet basis functions. To save time and space, it is useful to ﬁnd a sparse representation

422 Chapter 13. Sparse linear models
of the signals, in terms of a small number of such basis functions. This allows us to estimate
signals from a small number of measurements, as well as to compress the signal. SeeSection 13.8.3 for more information.
Note that the topic of feature selection and sparsity is currently one of the most active areas
of machine learning/ statistics. In this chapter, we only have space to give an overview of themain results.
13.2 Bayesian variable selection
A natural way to pose the variable selection problem is as follows. Let γj=1if feature jis
“relevant”, and let γj=0otherwise. Our goal is to compute the posterior over models
p(γ|D)=e−f(γ)
∑
γ′e−f(γ′)(13.1)
wheref(γ)is the cost function:
f(γ)≜−[logp(D|γ)+logp(γ)] (13.2)
For example, suppose we generate N=2 0samples from a D=1 0dimensional linear
regression model, yi∼N(wTxi,σ2), in which K=5elements of ware non-zero. In
particular, we use w=( 0.00,−1.67,0.13,0.00,0.00,1.19,0.00,−0.04,0.33,0.00)andσ2=

∑
1. We enumerate all 210= 1024models and compute p(γ|D)for each one (we give the
equations for this below). We order the models in Gray code order, which ensures consecutive
vectors differ by exactly 1 bit (the reasons for this are computational, and are discussed inSection 13.2.3).
The resulting set of bit patterns is shown in Figure 13.1(a). The cost of each model, f(γ),i s

shown in Figure 13.1(b). We see that this objective function is extremely “bumpy”. The resultsare easier to interpret if we compute the posterior distribution over models, p(γ|D). This is
shown in Figure 13.1(c). The top 8 models are listed below:
model prob members
4 0.447 2,61 0.241 2, 6,452 0.103 2, 6, 9,60 0.091 2, 3, 6,29 0.041 2, 5,68 0.021 2, 6, 7,36 0.015 2, 5, 6,5 0.010 2, 3,
The “true” model is {2,3,6,8,9}. However, the coefficients associated with features 3 and 8
are very small (relative to σ

2). so these variables are harder to detect. Given enough data, the
method will converge on the true model (assuming the data is generated from a linear model),but for ﬁnite data sets, there will usually be considerable posterior uncertainty.
Interpreting the posterior over a large number of models is quite difficult, so we will seek
various summary statistics. A natural one is the posterior mode, or MAP estimate
ˆγ=a r g m a x p(γ|D) = argmin f(γ) (13.3)

13.2. Bayesian variable selection 423
100 200 300 400 500 600 700 800 900 10001
2
3
4
5
6
7
8
9
10
(a)0 200 400 600 800 1000−220−200−180−160−140−120−100−80−60−40log p(model, data)
(b)
0 200 400 600 800 100000.010.020.030.040.050.060.070.080.090.1p(model|data)
(c)1 2 3 4 5 6 7 8 9 1000.10.20.30.40.50.60.70.80.91p(gamma(j)|data
(d)
Figure 13.1 (a) All possible bit vectors of length 10 enumerated in Gray code order. (b) Score function for
all possible models. (c) Posterior over all 1024 models. Vertical scale has been truncated at 0.1 for clarity.
(d) Marginal inclusion probabilities. Figure generated by linregAllsubsetsGraycodeDemo .
However, the mode is often not representative of the full posterior mass (see Section 5.2.1.3). A
better summary is the median model (Barbieri and Berger 2004; Carvahlo and Lawrence 2007),
computed using
ˆγ={j:p(γj=1|D)>0.5} (13.4)
This requires computing the posterior marginal inclusion probabilities ,p(γj=1|D). These

are shown in Figure 13.1(d). We see that the model is conﬁdent that variables 2 and 6 are
included; if we lower the decision threshold to 0.1, we would add 3 and 9 as well. However, if
we wanted to “capture” variable 8, we would incur two false positives (5 and 7). This tradeoff
between false positives and false negatives is discussed in more detail in Section 5.7.2.1.
The above example illustrates the “gold standard” for variable selection: the problem was
sufficiently small (only 10 variables) that we were able to compute the full posterior exactly.

Of course, variable selection is most useful in the cases where the number of dimensions is
large. Since there are 2Dpossible models (bit vectors), it will be impossible to compute the
full posterior in general, and even ﬁnding summaries, such as the MAP estimate or marginal

424 Chapter 13. Sparse linear models
inclusion probabilities, will be intractable. We will therefore spend most of this chapter focussing
on algorithmic speedups. But before we do that, we will explain how we computed p(γ|D)in
the above example.
13.2.1 The spike and slab model
p(γ|D)∝p(γ)p(D|γ) (13.5)
We ﬁrst consider the prior, then the likelihood.
It is common to use the following prior on the bit vector:
p(γ)=D∏
j=1Ber(γj|π0)=π||γ||0
0(1−π0)D−||γ||0(13.6)
whereπ0is the probability a feature is relevant, and ||γ||0=∑D
j=1γjis theℓ0pseudo-norm,
that is, the number of non-zero elements of the vector. For comparison with later models, it is
useful to write the log prior as follows:
logp(γ|π0)=||γ ||0logπ0+(D−||γ||0)log(1−π0) (13.7)
=||γ||0(logπ0−log(1−π0))+const (13.8)
=−λ||γ||0+const (13.9)
whereλ≜log1−π0
π0controls the sparsity of the model.
We can write the likelihood as follows:
p(D|γ)=p(y|X,γ)=∫∫
p(y|X,w,γ)p(w|γ,σ2)p(σ2)dwdσ2(13.10)

For notational simplicity, we have assumed the response is centered, (i.e., y=0), so we can
ignore any offset term μ.
We now discuss the prior p(w|γ,σ2).I fγj=0, featurejis irrelevant, so we expect wj=0.
Ifγj=1, we expect wjto be non-zero. If we standardize the inputs, a reasonable prior is
N(0,σ2σ2
w),w h e r eσ2
wcontrols how big we expect the coefficients associated with the relevant
variables to be (which is scaled by the overall noise level σ2). We can summarize this prior as
follows:

N(0,σ2σ2
follows:
p(wj|σ2,γj)={δ0(wj) ifγj=0
N(wj|0,σ2σ2
w)ifγj=1(13.11)
The ﬁrst term is a “spike” at the origin. As σ2
w→∞, the distribution p(wj|γj=1 )approaches
a uniform distribution, which can be thought of as a “slab” of constant height. Hence this iscalled the spike and slab model (Mitchell and Beauchamp 1988).
We can drop the coefficients w
jfor which wj=0from the model, since they are clamped
to zero under the prior. Hence Equation 13.10 becomes the following (assuming a Gaussianlikelihood):
p(D|γ)=∫∫
N(y|X
γwγ,σ2IN)N(wγ|0Dγ,σ2σ2

p(D|γ)=∫∫
N(y|X
wIDγ)p(σ2)dwγdσ2(13.12)

13.2. Bayesian variable selection 425
whereDγ=||γ||0is the number of non-zero elements in γ. In what follows, we will generalize
this slightly by deﬁning a prior of the form p(w|γ,σ2)=N(wγ|0Dγ,σ2Σγ)for any positive
deﬁnite matrix Σγ.1
Given these priors, we can now compute the marginal likelihood. If the noise variance is
known, we can write down the marginal likelihood (using Equation 13.151) as follows:
p(D|γ,σ2)=∫
N(y|Xγwγ,σ2I)N(wγ|0,σ2Σγ)dwγ=N(y|0,Cγ) (13.13)
Cγ≜σ2XγΣγXT
γ+σ2IN (13.14)
If the noise is unknown, we can put a prior on it and integrate it out. It is common to use
p(σ2)=I G (σ2|aσ,bσ). Some guidelines on setting a,bcan be found in (Kohn et al. 2001). If
we usea=b=0, we recover the Jeffrey’s prior, p(σ2)∝σ−2. When we integrate out the noise,
we get the following more complicated expression for the marginal likelihood (Brown et al. 1998):
p(D|γ)=∫∫
p(y|γ,wγ,σ2)p(wγ|γ,σ2)p(σ2)dwγdσ2(13.15)
∝|XTγXγ+Σ−1
γ|−1
2|Σγ|−1
2(2bσ+S(γ))−(2aσ+N−1)/2(13.16)
whereS(γ)is the RSS:
S(γ)≜yTy−yTXγ(XTγXγ+Σ−1

p(D|γ)=∫∫
γ|−1
2|Σγ|−1
γ)−1XTγy (13.17)
See also Exercise 13.4.
When the marginal likelihood cannot be computed in closed form (e.g., if we are using logistic
regression or a nonlinear model), we can approximate it using BIC, which has the form
logp(D|γ)≈logp(y|X,ˆwγ,ˆσ2)−||γ||0
2logN (13.18)
whereˆwγis the ML or MAP estimate based on Xγ, and||γ||0is the “degrees of freedom” of
the model (Zou et al. 2007). Adding the log prior, the overall objective becomes
logp(γ|D)≈logp(y|X,ˆwγ,ˆσ2)−||γ||0
2logN−λ||γ||0+const (13.19)
We see that there are two complexity penalties: one arising from the BIC approximation to

the marginal likelihood, and the other arising from the prior on p(γ). Obviously these can be
combined into one overall complexity parameter, which we will denote by λ.
13.2.2 From the Bernoulli-Gaussian model to ℓ0regularization
Another model that is sometimes used (e.g., (Kuo and Mallick 1998; Zhou et al. 2009; Soussenet al. 2010)) is the following:
y
i|xi,w,γ,σ2∼N(∑
jγjwjxij,σ2) (13.20)
γj∼Ber(π0) (13.21)
wj∼N(0,σ2
w) (13.22)
1. It is common to use a g-prior of the form Σγ=g(XT
γXγ)−1for reasons explained in Section 7.6.3.1 (see also

y
wj∼N(0,σ2
w) (13.22)
Exercise 13.4). Various approaches have been proposed for setting g, including cross validation, empirical Bayes (Minka
2000b; George and Foster 2000), hierarchical Bayes (Liang et al. 2008), etc.

426 Chapter 13. Sparse linear models
In the signal processing literature (e.g., (Soussen et al. 2010)), this is called the Bernoulli-
Gaussian model, although we could also call it the binary mask model, since we can think of
theγjvariables as “masking out” the weights wj.
Unlike the spike and slab model, we do not integrate out the “irrelevant” coefficients; they
always exist. In addition, the binary mask model has the form γj→y←wj, whereas the spike
and slab model has the form γj→wj→y. In the binary mask model, only the product γjwj
can be identiﬁed from the likelihood.
One interesting aspect of this model is that it can be used to derive an objective function that
is widely used in the (non-Bayesian) subset selection literature. First, note that the joint prior
p(γ,w)∝N(w|0,σ2
wI)π||γ||0
0(1−π0)D−||γ||0(13.23)
Hence the scaled unnormalized negative log posterior has the form
f(γ,w)≜−2σ2logp(γ,w,y|X)=||y −X(γ.∗w)||2
+σ2
σ2w||w||2+λ||γ||0+const (13.24)
where
λ≜2σ2log(1−π0
π0) (13.25)

+σ2
where
Let us split winto two subvectors, w−γandwγ, indexed by the zero and non-zero entries of
γrespectively. Since X(γ.∗w)=Xγwγ, we can just set w−γ=0.
Now consider the case where σ2
w→∞, so we do not regularize the non-zero weights (so
there is no complexity penalty coming from the marginal likelihood or its BIC approximation).In this case, the objective becomes
f(γ,w)=||y−X
γwγ||2
2+λ||γ||0 (13.26)
This is similar to the BIC objective above.
Instead of keeping track of the bit vector γ, we can deﬁne the set of relevant variables to

γwγ||2
be thesupport, or set of non-zero entries, of w. Then we can rewrite the above equation as
follows:
f(w)=||y−Xw||22+λ||w||0 (13.27)
This is called ℓ0regularization. We have converted the discrete optimization problem (over
γ∈{0,1}D) into a continuous one (over w∈RD); however, the ℓ0pseudo-norm makes the
objective very non smooth, so this is still hard to optimize. We will discuss different solutions
to this in the rest of this chapter.
13.2.3 Algorithms
Since there are 2Dmodels, we cannot explore the full posterior, or ﬁnd the globally optimal

follows:
model. Instead we will have to resort to heuristics of one form or another. All of the methodswe will discuss involve searching through the space of models, and evaluating the cost f(γ)at

13.2. Bayesian variable selection 427
{}{1}{ 2}{ 3}{ 4}{1,2}{ 1,3}{ 1,4}{ 2,3}{ 2,4}{ 3,4}{1,2,3}{2,3,4}{1,3,4}{1,2,4}{1,2,3,4}
(a)0 1 2 3 4 5 6 7 80.40.60.811.21.4
subset sizetraining set errorall subsets on prostate cancer
(b)
Figure 13.2 (a) A lattice of subsets of {1,2,3,4}. (b) Residual sum of squares versus subset size, on the
prostate cancer data set. The lower envelope is the best RSS achievable for any set of a given size. Based
on Figure 3.5 of (Hastie et al. 2001). Figure generated by prostateSubsets .
each point. This requires ﬁtting the model (i.e., computing argmaxp(D|w)), or evaluating its
marginal likelihood (i.e., computing∫
p(D|w)p(w)dw) at each step. This is sometimes called
thewrapper method, since we “wrap” our search for the best model (or set of good models)
around a generic model-ﬁtting procedure.
In order to make wrapper methods efficient, it is important that we can quickly evaluate the
score function for some new model, γ′, given the score of a previous model, γ. This can be

done provided we can efficiently update the sufficient statistics needed to compute f(γ). This
is possible provided γ′only differs from γin one bit (corresponding to adding or removing
a single variable), and provided f(γ)only depends on the data via Xγ. In this case, we can
use rank-one matrix updates/ downdates to efficiently compute XT
γ′Xγ′fromXT
γXγ. These
updates are usually applied to the QR decomposition of X. See e.g., (Miller 2002; Schniter et al.
2008) for details.
13.2.3.1 Greedy search

γXγ. These
Suppose we want to ﬁnd the MAP model. If we use the ℓ0-regularized objective in Equation 13.27,
we can exploit properties of least squares to derive various efficient greedy forwards search
methods, some of which we summarize below. For further details, see (Miller 2002; Soussenet al. 2010).
•Single best replacement The simplest method is to use greedy hill climbing, where at each
step, we deﬁne the neighborhood of the current model to be all models than can be reached

by ﬂipping a single bit of γ, i.e., for each variable, if it is currently out of the model, we
consider adding it, and if it is currently in the model, we consider removing it. In (Soussenet al. 2010), they call this the single best replacement (SBR). Since we are expecting a
sparse solution, we can start with the empty set, γ=0. We are essentially moving through
the lattice of subsets, shown in Figure 13.2(a). We continue adding or removing until noimprovement is possible.

•Orthogonal least squares If we set λ=0in Equation 13.27, so there is no complexity
penalty, there will be no reason to perform deletion steps. In this case, the SBR algorithm isequivalent to orthogonal least squares (Chen and Wigger 1995), which in turn is equivalent

428 Chapter 13. Sparse linear models
to greedy forwards selection. In this algorithm, we start with the empty set and add the
best feature at each step. The error will go down monotonically with ||γ||0, as shown in
Figure 13.2(b). We can pick the next best feature j∗to add to the current set γtby solving
j∗=a r gm i n
j̸∈γtmin
w||y−(Xγt∪j)w||2(13.28)
We then update the active set by setting γ(t+1)=γ(t)∪{j∗}. To choose the next feature to
add at step t, we need to solve D−Dtleast squares problems at step t,w h e r eDt=|γt|is
the cardinality of the current active set. Having chosen the best feature to add, we need to
solve an additional least squares problem to compute wt+1).
•Orthogonal matching pursuits Orthogonal least squares is somewhat expensive. A simpli-
ﬁcation is to “freeze” the current weights at their current value, and then to pick the nextfeature to add by solving
j
∗=a r gm i n
j̸∈γtmin
β||y−Xwt−βx:,j||2(13.29)
This inner optimization is easy to solve: we simply set β=xT

j
∗=a r gm i n
j̸∈γtmin
:,jrt/||x:,j||2,w h e r ert=
y−Xwtis the current residual vector. If the columns are unit norm, we have
j∗=a r gm a x xT
:,jrt (13.30)
so we are just looking for the column that is most correlated with the current residual. We
then update the active set, and compute the new least squares estimate wt+1usingXγt+1.
This method is called orthogonal matching pursuits orOMP(Mallat et al. 1994). This only

j
∗=a r gm i n
j̸∈γtmin
requires one least squares calculation per iteration and so is faster than orthogonal leastsquares, but is not quite as accurate (Blumensath and Davies 2007).
•Matching pursuits An even more aggressive approximation is to just greedily add the feature
that is most correlated with the current residual. This is called matching pursuits (Mallat
and Zhang 1993). This is also equivalent to a method known as least squares boosting(Section 16.4.6).

•Backwards selection Backwards selection starts with all variables in the model (the so-
calledsaturated model), and then deletes the worst one at each step. This is equivalent
to performing a greedy search from the top of the lattice downwards. This can give betterresults than a bottom-up search, since the decision about whether to keep a variable ornot is made in the context of all the other variables that might depende on it. However,this method is typically infeasible for large problems, since the saturated model will be tooexpensive to ﬁt.

•FoBaTheforwards-backwards algorithm of (Zhang 2008) is similar to the single best
replacement algorithm presented above, except it uses an OMP-like approximation whenchoosing the next move to make. A similar “dual-pass” algorithm was described in (Moghad-dam et al. 2008).
•Bayesian Matching pursuit The algorithm of (Schniter et al. 2008) is similiar to OMP except

it uses a Bayesian marginal likelihood scoring criterion (under a spike and slab model) insteadof a least squares objective. In addition, it uses a form of beam search to explore multiplepaths through the lattice at once.

13.3.ℓ1regularization: basics 429
13.2.3.2 Stochastic search
If we want to approximate the posterior, rather than just computing a mode (e.g. because we
want to compute marginal inclusion probabilities), one option is to use MCMC. The standardapproach is to use Metropolis Hastings, where the proposal distribution just ﬂips single bits.This enables us to efficiently compute p(γ
′|D)givenp(γ|D). The probability of a state (bit
conﬁguration) is estimated by counting how many times the random walk visits this state. See(O’Hara and Sillanpaa 2009) for a review of such methods, and (Bottolo and Richardson 2010)for a very recent method based on evolutionary MCMC.
However, in a discrete state space, MCMC is needlessly inefficient, since we can compute the
(unnormalized) probability of a state directly using p(γ,D)=e x p ( −f(γ)); thus there is no

need to ever revisit a state. A much more efficient alternative is to use some kind of stochasticsearch algorithm, to generate a set Sof high scoring models, and then to make the following
approximation
p(γ|D)≈e
−f(γ)
∑
γ′∈Se−f(γ′)(13.31)
See (Heaton and Scott 2009) for a review of recent methods of this kind.
13.2.3.3 EM and variational inference *
It is tempting to apply EM to the spike and slab model, which has the form γj→wj→y.W e
can compute p(γj=1|wj)in the E step, and optimize win the M step. However, this will not

p(γ|D)≈e
−f(γ)
∑
work, because when we compute p(γj=1|wj), we are comparing a delta-function, δ0(wj), with
a Gaussian pdf, N(wj|0,σ2
w). We can replace the delta function with a narrow Gaussian, and
then the E step amounts to classifying wjunder the two possible Gaussian models. However,
this is likely to suffer from severe local minima.
An alternative is to apply EM to the Bernoulli-Gaussian model, which has the form γj→y←
wj. In this case, the posterior p(γ|D,w)is intractable to compute because all the bits become
correlated due to explaining away. However, it is possible to derive a mean ﬁeld approximationof the form∏
jq(γj)q(wj)(Huang et al. 2007; Rattray et al. 2009).

13.3ℓ1regularization: basics
When we have many variables, it is computationally difficult to ﬁnd the posterior mode ofp(γ|D). And although greedy algorithms often work well (see e.g., (Zhang 2008) for a theoretical
analysis), they can of course get stuck in local optima.
Part of the problem is due to the fact that the γ
jvariables are discrete, γj∈{0,1}.I n

the optimization community, it is common to relax hard constraints of this form by replacingdiscrete variables with continuous variables. We can do this by replacing the spike-and-slab styleprior, that assigns ﬁnite probability mass to the event that w
j=0, to continuous priors that
“encourage” wj=0by putting a lot of probability density near the origin, such as a zero-mean
Laplace distribution. This was ﬁrst introduced in Section 7.4 in the context of robust linearregression. There we exploited the fact that the Laplace has heavy tails. Here we exploit the fact

430 Chapter 13. Sparse linear models
Figure 13.3 Illustration of ℓ1(left) vsℓ2(right) regularization of a least squares problem. Based on Figure
3.12 of (Hastie et al. 2001).
that it has a spike near μ=0. More precisely, consider a prior of the form
p(w|λ)=D∏
j=1Lap(wj|0,1/λ)∝D∏
j=1e−λ|wj|(13.32)
We will use a uniform prior on the offset term, p(w0)∝1. Let us perform MAP estimation with
this prior. The penalized negative log likelihood has the form
f(w)=−logp(D|w)−logp(w|λ) = NLL( w)+λ||w||1 (13.33)
where||w||1=∑D
j=1|wj|is theℓ1norm ofw. For suitably large λ, the estimate ˆwwill be
sparse, for reasons we explain below. Indeed, this can be thought of as a convex approximation
to the non-convex ℓ0objective
argmin
wNLL(w)+λ||w||0 (13.34)
In the case of linear regression, the ℓ1objective becomes
f(w)=N∑
i=1−1
2σ2(yi−(w0+wTxi))2+λ||w||1 (13.35)
= RSS( w)+λ′||w||1 (13.36)
whereλ′=2λσ2. This method is known as basis pursuit denoising orBPDN(Chen et al. 1998).

argmin
f(w)=N∑
i=1−1
The reason for this term will become clear later. In general, the technique of putting a zero-mean
Laplace prior on the parameters and performing MAP estimation is called ℓ1regularization .
It can be combined with any convex or non-convex NLL term. Many different algorithms have
been devised for solving such problems, some of which we review in Section 13.4.
13.3.1 Why does ℓ1regularization yield sparse solutions?
We now explain why ℓ1regularization results in sparse solutions, whereas ℓ2regularization does

not. We focus on the case of linear regression, although similar arguments hold for logistic
regression and other GLMs.

13.3.ℓ1regularization: basics 431
The objective is the following non-smooth objective function:
min
wRSS(w)+λ||w||1 (13.37)
We can rewrite this as a constrained but smooth objective (a quadratic function with linear
constraints):
min
wRSS(w)s.t.||w||1≤B (13.38)
whereBis an upper bound on the ℓ1-norm of the weights: a small (tight) bound Bcorresponds
to a large penalty λ, and vice versa.2Equation 13.38 is known as lasso, which stands for “least
absolute shrinkage and selection operator” (Tibshirani 1996). We will see why it has this namelater.
Similarly, we can write ridge regression
min
wRSS(w)+λ||w||2
2 (13.39)
or as a bound constrained form:
min
wRSS(w)s.t.||w||22≤B (13.40)
In Figure 13.3, we plot the contours of the RSS objective function, as well as the contours of
theℓ2andℓ1constraint surfaces. From the theory of constrained optimization, we know that
the optimal solution occurs at the point where the lowest level set of the objective function

min
2 (13.39)
min
intersects the constraint surface (assuming the constraint is active). It should be geometricallyclear that as we relax the constraint B,w e“ g r o w ”t h eℓ
1“ball” until it meets the objective; the
corners of the ball are more likely to intersect the ellipse than one of the sides, especially in highdimensions, because the corners “stick out” more. The corners correspond to sparse solutions,which lie on the coordinate axes. By contrast, when we grow the ℓ
2ball, it can intersect the
objective at any point; there are no “corners”, so there is no preference for sparsity.

To see this another away, notice that, with ridge regression, the prior cost of a sparse solution,
such asw=( 1,0), is the same as the cost of a dense solution, such as w=( 1/√
2,1/√
2),
as long as they have the same ℓ2norm:
||(1,0)||2=||(1/√
2,1/√
2||2=1 (13.41)
However, for lasso, setting w=( 1,0)is cheaper than setting w=( 1/√
2,1/√
2), since
||(1,0)||1=1<||(1/√
2,1/√
2||1=√
2 (13.42)
The most rigorous way to see that ℓ1regularization results in sparse solutions is to examine
conditions that hold at the optimum. We do this in Section 13.3.2.
13.3.2 Optimality conditions for lasso

2,1/√
2),
2,1/√
2,1/√
2), since
2,1/√
2||1=√
2 (13.42)
The lasso objective has the form
f(θ) = RSS( θ)+λ||w||1 (13.43)
2. Equation 13.38 is an example of a quadratic program orQP, since we have a quadratic objective subject to linear
inequality constraints. Its Lagrangian is given by Equation 13.37.

432 Chapter 13. Sparse linear models
X0Xc(x − x0)f(x) − f(x0)cʼ
c
Figure 13.4 Illustration of some sub-derivatives of a function at point x0. Based on a ﬁgure at http:
//en.wikipedia .org/wiki/Subderivative . Figure generated by subgradientPlot .
Unfortunately, the ||w||1term is not differentiable whenever wj=0. This is an example of a
non-smooth optimization problem.
To handle non-smooth functions, we need to extend the notion of a derivative. We deﬁne a
subderivative orsubgradient of a (convex) function f:I→Rat a point θ0to be a scalar g
such that
f(θ)−f(θ0)≥g(θ−θ0)∀θ∈I (13.44)
whereIis some interval containing θ0. See Figure 13.4 for an illustration.3We deﬁne the setof
subderivatives as the interval [a,b]whereaandbare the one-sided limits
a= lim
θ→θ−
0f(θ)−f(θ0)
θ−θ0,b= lim
θ→θ+
θ−θ0(13.46)
The set[a,b]of all subderivatives is called the subdifferential of the function fatθ0and
is denoted ∂f(θ)|θ0. For example, in the case of the absolute value function f(θ)=|θ|, the
subderivative is given by

such that
a= lim
θ→θ−
θ→θ+
∂f(θ)=⎧
⎨
⎩{−1}ifθ<0
[−1,1]ifθ=0
{+1}ifθ>0(13.47)
If the function is everywhere differentiable, then ∂f(θ)={df(θ)
dθ}. By analogy to the standard
calculus result, one can show that the point ˆθis a local minimum of fiff0∈∂f(θ)|ˆθ.
3. In general, for a vector valued function, we say that gis a subgradient of fatθ0if for all vectors θ,
f(θ)−f(θ0)≥(θ−θ0)Tg (13.45)
sogis a linear lower bound to the function at θ0.

13.3.ℓ1regularization: basics 433
FN
(a)FN
(b)
Figure 13.5 Left: soft thresholding. The ﬂat region is the interval [−λ,+λ]. Right: hard thresholding.
Let us apply these concepts to the lasso problem. Let us initially ignore the non-smooth
penalty term. One can show (Exercise 13.1) that
∂
∂wjRSS(w)=a jwj−cj (13.48)
aj=2n∑
i=1x2
ij (13.49)
cj=2n∑
i=1xij(yi−wT
−jxi,−j) (13.50)
wherew−jiswwithout component j, and similarly for xi,−j. We see that cjis (proportional
to) the correlation between the j’th feature x:,jand the residual due to the other features,
r−j=y−X:,−jw−j. Hence the magnitude of cjis an indication of how relevant feature jis
for predicting y(relative to the other features and the current parameters).
Adding in the penalty term, we ﬁnd that the subderivative is given by
∂wjf(w)=(ajwj−cj)+λ∂ wj||w||1 (13.51)
=⎧
⎨
⎩{ajwj−cj−λ}ifwj<0
[−cj−λ,−cj+λ]ifwj=0
{ajwj−cj+λ}ifwj>0(13.52)
We can write this in a more compact fashion as follows:
XT(Xw−y)j∈⎧
⎨
⎩{−λ}ifwj<0
[−λ,λ]ifwj=0
{λ}ifwj>0(13.53)

=⎧
⎨
⎨
Depending on the value of cj, the solution to ∂wjf(w)=0can occur at 3 different values
ofwj, as follows:

434 Chapter 13. Sparse linear models
1. Ifcj<−λ, so the feature is strongly negatively correlated with the residual, then the
subgradient is zero at ˆwj=cj+λ
aj<0.
2. Ifcj∈[−λ,λ], so the feature is only weakly correlated with the residual, then the subgradient
is zero at ˆwj=0.
3. Ifcj>λ, so the feature is strongly positively correlated with the residual, then the subgra-
dient is zero at ˆwj=cj−λ
aj>0.
In summary, we have
ˆwj(cj)=⎧
⎨
⎩(cj+λ)/ajifcj<−λ
0ifcj∈[−λ,λ]
(cj−λ)/ajifcj>λ(13.54)
We can write this as follows:
ˆwj=soft(cj
aj;λ
aj) (13.55)
where
soft(a;δ)≜sign(a)(|a|−δ)+(13.56)
andx+= max(x,0)is the positive part of x. This is called soft thresholding. This is
illustrated in Figure 13.5(a), where we plot ˆwjvscj. The dotted line is the line wj=cj/aj
corresponding to the least squares ﬁt. The solid line, which represents the regularized estimate
ˆwj(cj), shifts the dotted line down (or up) by λ, except when −λ≤cj≤λ, in which case it
setswj=0.

aj;λ
where
setswj=0.
By contrast, in Figure 13.5(b), we illustrate hard thresholding. This sets values of wjto
0i f−λ≤cj≤λ, but it does not shrink the values of wjoutside of this interval. The
slope of the soft thresholding line does not coincide with the diagonal, which means that even
large coefficients are shrunk towards zero; consequently lasso is a biased estimator. This isundesirable, since if the likelihood indicates (via c
j) that the coefficient wjshould be large, we

setswj=0.
do not want to shrink it. We will discuss this issue in more detail in Section 13.6.2.
Now we ﬁnally can understand why Tibshirani invented the term “lasso” in (Tibshirani 1996):
it stands for “least absolute selection and shrinkage operator”, since it selects a subset of thevariables, and shrinks all the coefficients by penalizing the absolute values. If λ=0, we get the
OLS solution (of minimal ℓ
1norm). If λ≥λmax,w eg e tˆw=0,w h e r e
λmax=||XTy||∞=m a x
j|yTx:,j| (13.57)

This value is computed using the fact that 0is optimal if (XTy)j∈[−λ,λ]for allj. In general,
the maximum penalty for an ℓ1regularized objective is
λmax=m a x
j|∇jNLL(0)| (13.58)

13.3.ℓ1regularization: basics 435
13.3.3 Comparison of least squares, lasso, ridge and subset selection
We can gain further insight into ℓ1regularization by comparing it to least squares, and ℓ2and
ℓ0regularized least squares. For simplicity, assume all the features of Xare orthonormal, so
XTX=I. In this case, the RSS is given by
RSS(w)=||y −Xw||2=yTy+wTXTXw−2wTXTy (13.59)
= const+∑
kw2
k−2∑
k∑
iwkxikyi (13.60)
so we see this factorizes into a sum of terms, one per dimension. Hence we can write down the
MAP and ML estimates analytically, as follows:
•MLEThe OLS solution is given by
ˆwOLS
k=xT
:ky (13.61)
wherex:kis thek’th column of X. This follows trivially from Equation 13.60. We see
thatˆwOLS
kis just the orthogonal projection of feature konto the response vector (see
Section 7.3.2).
•RidgeOne can show that the ridge estimate is given by
ˆwridge
k=ˆwOLS
k
1+λ(13.62)
•LassoFrom Equation 13.55, and using the fact that ak=2andˆwOLS
k=ck/2,w eh a v e
ˆwlasso
k=s i g n ( ˆ wOLS
k)(
|ˆwOLS
k|−λ
2)
+(13.63)

ˆwOLS
k=xT
thatˆwOLS
ˆwridge
k=ˆwOLS
k
ˆwlasso
k)(
|ˆwOLS
k|−λ
2)
+(13.63)
This corresponds to soft thresholding, shown in Figure 13.5(a).
•Subset selection If we pick the best Kfeatures using subset selection, the parameter
estimate is as follows
ˆwSS
k={
ˆwOLS
kif rank(|wOLS
k|)≤K
0 otherwise(13.64)
where rank refers to the location in the sorted list of weight magnitudes. This corresponds
to hard thresholding, shown in Figure 13.5(b).
Figure 13.6(a) plots the MSE vs λfor lasso for a degree 14 polynomial, and Figure 13.6(b) plots

k)(
|ˆwOLS
k|−λ
2)
+(13.63)
ˆwSS
k={
ˆwOLS
k|)≤K
the MSE vs polynomial order. We see that lasso gives similar results to the subset selectionmethod.
As another example, consider a data set concerning prostate cancer. We have D=8features
andN=6 7training cases; the goal is to predict the log prostate-speciﬁc antigen levels (see

(Hastie et al. 2009, p4) for more biological details). Table 13.1 shows that lasso gives betterprediction accuracy (at least on this particular data set) than least squares, ridge, and bestsubset regression. (In each case, the strength of the regularizer was chosen by cross validation.)Lasso also gives rise to a sparse solution. Of course, for other problems, ridge may give betterpredictive accuracy. In practice, a combination of lasso and ridge, known as the elastic net,often performs best, since it provides a good combination of sparsity and regularization (seeSection 13.5.3).

436 Chapter 13. Sparse linear models
103.249 10 1 0.5 0.1 0.01 0.0001 005101520253035
lambdamsetrain
test
(a)0 2 4 6 8 10 12 14 1605101520253035
degreemseperformance of MLE
  
train
test
(b)
Figure 13.6 (a) MSE vs λfor lasso for a degree 14 polynomial. Note that λdecreases as we move to
the right. Figure generated by linregPolyLassoDemo . (b) MSE versus polynomial degree. Note that the
model order increases as we move to the right. See Figure 1.18 for a plot of some of these polynomial
regression models. Figure generated by linregPolyVsDegree .
Term LS Best Subset Ridge Lasso
Intercept 2.452 2.481 2.479 2.480
lcavol 0.716 0.651 0.656 0.653
lweight 0.293 0.380 0.300 0.297
age -0.143 -0.000 -0.129 -0.119
lbph 0.212 -0.000 0.208 0.200
svi 0.310 -0.000 0.301 0.289
lcp -0.289 -0.000 -0.260 -0.236
gleason -0.021 -0.000 -0.019 0.000
pgg45 0.277 0.178 0.256 0.226
Test Error 0.586 0.572 0.580 0.564
Table 13.1 Results of different methods on the prostate cancer data, which has 8 features and 67 training

cases. Methods are: LS = least squares, Subset = best subset regression, Ridge, Lasso. Rows representthe coefficients; we see that subset regression and lasso give sparse solutions. Bottom row is the meansquared error on the test set (30 cases). Based on Table 3.3. of (Hastie et al. 2009). Figure generated byprostateComparison .
13.3.4 Regularization path
As we increase λ, the solution vector ˆw(λ)will tend to get sparser, although not necessarily
monotonically. We can plot the values ˆwj(λ)vsλfor each feature j; this is known as the
regularization path.

This is illustrated for ridge regression in Figure 13.7(a), where we plot ˆwj(λ)as the regularizer
λdecreases. We see that when λ=∞, all the coefficients are zero. But for any ﬁnite value of
λ, all coefficients are non-zero; furthermore, they increase in magnitude as λis decreased.
In Figure 13.7(b), we plot the analogous result for lasso. As we move to the right, the upper
bound on the ℓ1penalty,B, increases. When B=0, all the coefficients are zero. As we increase

13.3.ℓ1regularization: basics 437
0 5 10 15 20 25 30−0.2−0.100.10.20.30.40.50.6
  
lcavol
lweight
age
lbph
svi
lcp
gleason
pgg45
(a)0 5 10 15 20 25−0.2−0.100.10.20.30.40.50.60.7
  
lcavol
lweight
age
lbph
svi
lcp
gleason
pgg45
(b)
Figure 13.7 (a) Proﬁles of ridge coefficients for the prostate cancer example vs bound on ℓ2norm ofw,
so smallt(largeλ) is on the left. The vertical line is the value chosen by 5-fold CV using the 1SE rule.
Based on Figure 3.8 of (Hastie et al. 2009). Figure generated by ridgePathProstate . (b) Proﬁles of lasso
coefficients for the prostate cancer example vs bound on ℓ1norm ofw, so small t(largeλ) is on the left.
Based on Figure 3.10 of (Hastie et al. 2009). Figure generated by lassoPathProstate .
0 0.5 1 1.5 2−0.2−0.100.10.20.30.40.50.60.7
τ  
lcavol
lweight
age
lbph
svi
lcp
gleason
pgg45
(a)1 2 3 4 5 6 7 8 9−0.2−0.100.10.20.30.40.50.60.7
lars step  
lcavol
lweight
age
lbph
svi
lcp
gleason
pgg45
(b)

τ  
lcavol
lweight
age
lbph
svi
lcp
gleason
pgg45
lars step  
lcavol
lweight
age
lbph
svi
lcp
gleason
pgg45
(b)
Figure 13.8 Illustration of piecewise linearity of regularization path for lasso on the prostate cancer
example. (a) We plot ˆwj(B)vsBfor the critical values of B. (b) We plot vs steps of the LARS algorithm.
Figure generated by lassoPathProstate .
B, the coefficients gradually “turn on”. But for any value between 0 and Bmax=||ˆwOLS||1,
the solution is sparse.4
Remarkably, it can be shown that the solution path is a piecewise linear function of B(Efron

lcavol
lweight
age
lbph
svi
lcp
gleason
pgg45
(b)
et al. 2004). That is, there are a set of critical values of Bwhere the active set of non-zero
coefficients changes. For values of Bbetween these critical values, each non-zero coefficient
increases or decreases in a linear fashion. This is illustrated in Figure 13.8(a). Furthermore,
one can solve for these critical values analytically. This is the basis of the LARSalgorithm

(Efron et al. 2004), which stands for “least angle regression and shrinkage” (see Section 13.4.2for details). Remarkably, LARS can compute the entire regularization path for roughly the same
4. It is common to plot the solution versus the shrinkage factor, deﬁned as s(B)=B/B max, rather than against B.
This merely affects the scale of the horizontal axis, not the shape of the curves.

438 Chapter 13. Sparse linear models
0 1000 2000 3000 4000−101Original (D = 4096, number of nonzeros = 160)
0 1000 2000 3000 4000−101L1 reconstruction (K0 = 1024, lambda = 0.0516, MSE = 0.0027)
0 1000 2000 3000 4000−101Debiased (MSE = 3.26e−005)
0 1000 2000 3000 4000−0.500.5Minimum norm solution (MSE = 0.0292)
Figure 13.9 Example of recovering a sparse signal using lasso. See text for details. Based on Figure 1 of
(Figueiredo et al. 2007). Figure generated by sparseSensingDemo , written by Mario Figueiredo.
computational cost as a single least squares ﬁt (namely O(min(ND2,DN2)).
In Figure 13.8(b), we plot the coefficients computed at each critical value of B. Now the
piecewise linearity is more evident. Below we display the actual coefficient values at each step
along the regularization path (the last line is the least squares solution):
Listing 13.1 Output of lassoPathProstate
00000000
0.4279 0 0 0 0 0 0 0
0.5015 0.0735 0 0 0 0 0 0
0.5610 0.1878 0 0 0.0930 0 0 0
0.5622 0.1890 0 0.0036 0.0963 0 0 0

00000000
0.5797 0.2456 0 0.1435 0.2003 0 0 0.0901
0.5864 0.2572 -0.0321 0.1639 0.2082 0 0 0.1066
0.6994 0.2910 -0.1337 0.2062 0.3003 -0.2565 0 0.2452
0.7164 0.2926 -0.1425 0.2120 0.3096 -0.2890 -0.0209 0.2773
By changing Bfrom 0 to Bmax, we can go from a solution in which all the weights are zero
to a solution in which all weights are non-zero. Unfortunately, not all subset sizes are achievableusing lasso. One can show that, if D>N, the optimal solution can have at most Nvariables in

it, before reaching the complete set corresponding to the OLS solution of minimal ℓ
1norm. In
Section 13.5.3, we will see that by using an ℓ2regularizer as well as an ℓ1regularizer (a method
known as the elastic net), we can achieve sparse solutions which contain more variables thantraining cases. This lets us explore model sizes between NandD.

13.3.ℓ1regularization: basics 439
13.3.5 Model selection
It is tempting to use ℓ1regularization to estimate the set of relevant variables. In some cases,
we can recover the true sparsity pattern of w∗, the parameter vector that generated the data.
A method that can recover the true model in the N→∞limit is called model selection
consistent. The details on which methods enjoy this property, and when, are beyond the scope
of this book; see e.g., (Buhlmann and van de Geer 2011) for details.
Instead of going into a theoretical discussion, we will just show a small example. We ﬁrst
generate a sparse signal w∗of sizeD= 4096, consisting of 160 randomly placed ±1spikes.
Next we generate a random design matrix Xof sizeN×D,w h e r eN= 1024. Finally we
generate a noisy observation y=Xw∗+ϵ,w h e r eϵi∼N(0,0.012). We then estimate wfrom
yandX.
The original w∗is shown in the ﬁrst row of Figure 13.9. The second row is the ℓ1estimate
ˆwL1usingλ=0.1λmax. We see that this has “spikes” in the right places, but they are too

yandX.
small. The third row is the least squares estimate of the coefficients which are estimated to benon-zero based on supp(ˆw
L1). This is called debiasing, and is necessary because lasso shrinks
the relevant coefficients as well as the irrelevant ones. The last row is the least squares estimatefor all the coefficients jointly, ignoring sparsity. We see that the (debiased) sparse estimateis an excellent estimate of the original signal. By contrast, least squares without the sparsityassumption performs very poorly.

Of course, to perform model selection, we have to pick λ. It is common to use cross validation.
However, it is important to note that cross validation is picking a value of λthat results in good
predictive accuracy. This is not usually the same value as the one that is likely to recover the“true” model. To see why, recall that ℓ
1regularization performs selection andshrinkage, that is,
the chosen coefficients are brought closer to 0. In order to prevent relevant coefficients frombeing shrunk in this way, cross validation will tend to pick a value of λthat is not too large. Of

course, this will result in a less sparse model which contains irrelevant variables (false positives).Indeed, it was proved in (Meinshausen and Buhlmann 2006) that the prediction-optimal valueofλdoes not result in model selection consistency. In Section 13.6.2, we will discuss some
adaptive mechanisms for automatically tuning λon a per-dimension basis that does result in
model selection consistency.
A downside of using ℓ
1regularization to select variables is that it can give quite different

results if the data is perturbed slightly. The Bayesian approach, which estimates posteriormarginal inclusion probabilities, p(γ
j=1|D), is much more robust. A frequentist solution to
this is to use bootstrap resampling (see Section 6.2.1), and to rerun the estimator on differentversions of the data. By computing how often each variable is selected across different trials,we can approximate the posterior inclusion probabilities. This method is known as stability
selection (Meinshausen and BÃijhlmann 2010).

We can threshold the stability selection (bootstrap) inclusion probabilities at some level, say
90%, and thus derive a sparse estimator. This is known as bootstrap lasso orbolasso(Bach

2008). It will include a variable if it occurs in at least 90% of sets returned by lasso (for a ﬁxedλ). This process of intersecting the sets is a way of eliminating the false positives that vanillalasso produces. The theoretical results in (Bach 2008) prove that bolasso is model selectionconsistent under a wider range of conditions than vanilla lasso.
As an illustration, we reproduced the experiments in (Bach 2008). In particular, we created

440 Chapter 13. Sparse linear models
lasso on sign inconsistent data
−log( λ)variable index
0 5 10 152
4
6
8
10
12
14
16
(a)
bolasso on sign inconsistent data
128 bootstraps
0 5 10 152
4
6
8
10
12
14
16
(b)0 5 10 1500.51lasso vs bolasso on sign inconsistent data
nbootstraps = [0,2,4,8,16,32,64,128,256]
−log( λ)P(correct support)lasso
bolasso
(c)
Figure 13.10 (a) Probability of selection of each variable (white = large probabilities, black = small proba-
bilities) vs. regularization parameter for Lasso. As we move from left to right, we decrease the amount of
regularization, and therefore select more variables. (b) Same as (a) but for bolasso. (c) Probability of correct
sign estimation vs. regularization parameter. Bolasso (red, dashed) and Lasso (black, plain): The number
of bootstrap replications is in {2,4,8,16,32,64,128,256}. Based on Figures 1-3 of (Bach 2008). Figure
generated by bolassoDemo .

256 datasets of size N= 1000withD=1 6variables, of which 8are relevant. See (Bach 2008)
for more detail on the experimental setup. For dataset n, variable j, and sparsity level k, deﬁne
S(j,k,n)=I(ˆwj(λk,Dn)̸=0 ). Now deﬁne P(j,k)be the average of S(j,k,n)over the 256
datasets. In Figure 13.10(a-b), we plot Pvs−log(λ)for lasso and bolasso. We see that for
bolasso, there is a large range of λwhere the true variables are selected, but this is not the
case for lasso. This is emphasized in Figure 13.10(c), where we plot the empirical probability that

the correct set of variables is recovered, for lasso and for bolasso with an increasing number of
bootstrap samples. Of course, using more samples takes longer. In practice, 32 bootstraps seems
to be a good compromise between speed and accuracy.
With bolasso, there is the usual issue of picking λ. Obviously we could use cross validation,
but plots such as Figure 13.10(b) suggest another heuristic: shuffle the rows to create a large
black block, and then pick λto be in the middle of this region. Of course, operationalizing this

intuition may be tricky, and will require various ad-hoc thresholds (it is reminiscent of the “ﬁnd
the knee in the curve” heuristic discussed in Section 11.5.2 when discussing how to pick Kfor
mixture models). A Bayesian approach provides a more principled method for selecting λ.
13.3.6 Bayesian inference for linear models with Laplace priors
We have been focusing on MAP estimation in sparse linear models. It is also possible to perform
Bayesian inference (see e.g., (Park and Casella 2008; Seeger 2008)). However, the posterior mean

and median, as well as samples from the posterior, are not sparse; only the mode is sparse. This
is another example of the phenomenon discussed in Section 5.2.1, where we said that the MAP
estimate is often untypical of the bulk of the posterior.
Another argument in favor of using the posterior mean comes from Equation 5.108, which
showed that that plugging in the posterior mean, rather than the posterior mode, is the optimal
thing to do if we want to minimize squared prediction error. (Schniter et al. 2008) shows

experimentally, and (Elad and Yavnch 2009) shows theoretically, that using the posterior mean
with a spike-and-slab prior results in better prediction accuracy than using the posterior mode
with a Laplace prior, albeit at slightly higher computational cost.

13.4.ℓ1regularization: algorithms 441
13.4ℓ1regularization: algorithms
In this section, we give a brief review of some algorithms that can be used to solve ℓ1regularized
estimation problems. We focus on the lasso case, where we have a quadratic loss. However,
most of the algorithms can be extended to more general settings, such as logistic regression (see(Yaun et al. 2010) for a comprehensive review of ℓ
1regularized logistic regression). Note that this
area of machine learning is advancing very rapidly, so the methods below may not be state ofthe art by the time you read this chapter. (See (Schmidt et al. 2009; Yaun et al. 2010; Yang et al.2010) for some recent surveys.)
13.4.1 Coordinate descent
Sometimes it is hard to optimize all the variables simultaneously, but it easy to optimize themone by one. In particular, we can solve for the j’th coefficient with all the others held ﬁxed:
w
∗
j=a r g m i n
zf(w+zej)−f(w) (13.65)

w
∗
j=a r g m i n
whereejis thej’th unit vector. We can either cycle through the coordinates in a deterministic
fashion, or we can sample them at random, or we can choose to update the coordinate forwhich the gradient is steepest.
The coordinate descent method is particularly appealing if each one-dimensional optimization
problem can be solved analytically For example, the shooting algorithm (Fu 1998; Wu and Lange
2008) for lasso uses Equation 13.54 to compute the optimal value of w
jgiven all the other

coefficients. See Algorithm 7 for the pseudo code (and LassoShooting for some Matlab code).
See (Yaun et al. 2010) for some extensions of this method to the logistic regression case. The
resulting algorithm was the fastest method in their experimental comparison, which concerneddocument classiﬁcation with large sparse feature vectors (representing bags of words). Othertypes of data (e.g., dense features and/or regression problems) might call for different algorithms.

Algorithm 13.1: Coordinate descent for lasso (aka shooting algorithm)
1Initializew=(XTX+λI)−1XTy;
2repeat
3forj=1,...,Ddo
4 aj=2∑n
i=1x2
ij;
5 cj=2∑n
i=1xij(yi−wTxi+wjxij);
6 wj=soft(cj
aj,λ
aj);
13.4.2 LARS and other homotopy methods
The problem with coordinate descent is that it only updates one variable at a time, so can be
slow to converge. Active set methods update many variables at a time. Unfortunately, they are

442 Chapter 13. Sparse linear models
more complicated, because of the need to identify which variables are constrained to be zero,
and which are free to be updated.
Active set methods typically only add or remove a few variables at a time, so they can take a
long if they are started far from the solution. But they are ideally suited for generating a set ofsolutions for different values of λ, starting with the empty set, i.e., for generating regularization
path. These algorithms exploit the fact that one can quickly compute ˆw(λ
k)fromˆw(λk−1)
ifλk≈λk−1; this is known as warm starting. In fact, even if we only want the solution for
a single value of λ, call it λ∗, it can sometimes be computationally more efficient to compute
a set of solutions, from λmaxdown toλ∗, using warm-starting; this is called a continuation
methodorhomotopy method. This is often much faster than directly “cold-starting” at λ∗; this
is particularly true if λ∗is small.

Perhaps the most well-known example of a homotopy method in machine learning is the
LARSalgorithm, which stands for “least angle regression and shrinkage” (Efron et al. 2004) (a
similar algorithm was independently invented in (Osborne et al. 2000b,a)). This can compute
ˆw(λ)for all possible values of λin an efficient manner.
LARS works as follows. It starts with a large value of λ, such that only the variable that is most
correlated with the response vector yis chosen. Then λis decreased until a second variable

is found which has the same correlation (in terms of magnitude) with the current residual asthe ﬁrst variable, where the residual at step kis deﬁned as r
k=y−X:,Fkwk,w h e r eFkis
the current active set (c.f., Equation 13.50). Remarkably, one can solve for this new value of
λanalytically, by using a geometric argument (hence the term “least angle”). This allows the
algorithm to quickly “jump” to the next point on the regularization path where the active setchanges. This repeats until all the variables are added.

It is necessary to allow variables to be removed from the active set if we want the sequence of
solutions to correspond to the regularization path of lasso. If we disallow variable removal, weget a slightly different algorithm called LAR, which tends to be faster. In particular, LAR costs
the same as a single ordinary least squares ﬁt, namely O(NDmin(N,D )), which is O(ND
2)
ifN>D, andO(N2D)ifD>N. LAR is very similar to greedy forward selection, and a
method known as least squares boosting (see Section 16.4.6).
There have been many attempts to extend the LARS algorithm to compute the full regulariza-

2)
tion path for ℓ1regularized GLMs, such as logistic regression. In general, one cannot analytically
solve for the critical values of λ. Instead, the standard approach is to start at λmax, and then
slowly decrease λ, tracking the solution as we go; this is called a continuation method or
homotopy method. These methods exploit the fact that we can quickly compute ˆw(λk)from
ˆw(λk−1)ifλk≈λk−1; this is known as warm starting. Even if we don’t want the full path,

this method is often much faster than directly “cold-starting” at the desired value of λ(this is
particularly true if λis small).
The method described in (Friedman et al. 2010) combines coordinate descent with this warm-
starting strategy, and computes the full regularization path for any ℓ1regularized GLM. This has
been implemented in the glmnet package, which is bundled with PMTK.
13.4.3 Proximal and gradient projection methods

In this section, we consider some methods that are suitable for very large scale problems, wherehomotopy methods made be too slow. These methods will also be easy to extend to other kinds

13.4.ℓ1regularization: algorithms 443
of regularizers, beyond ℓ1, as we will see later. Our presentation in this section is based on
(Vandenberghe 2011; Yang et al. 2010).
Consider a convex objective of the form
f(θ)=L(θ)+R(θ) (13.66)
whereL(θ)(representing the loss) is convex and differentiable, and R(θ)(representing the
regularizer) is convex but not necessarily differentiable. For example, L(θ) = RSS(θ )and
R(θ)=λ||θ||1corresponds to the BPDN problem. As another example, the lasso problem can
be formulated as follows: L(θ) = RSS( θ)andR(θ)=IC(θ),w h e r eC={θ:||θ||1≤B},
andIC(θ)is the indicator function of a convex set C, deﬁned as
IC(θ)≜{0θ∈C
+∞otherwise(13.67)
In some cases, it is easy to optimize functions of the form in Equation 13.66. For example,
supposeL(θ) = RSS(θ ), and the design matrix is simply X=I. Then the obective becomes
f(θ)=R(θ)+1
2||θ−y|2
2. The minimizer of this is given by proxR(y), which is the proximal
operator for the convex function R, deﬁned by
proxR(y) = argmin
z(
R(z)+1

2||θ−y|2
z(
R(z)+1
2||z−y||2
2)
(13.68)
Intuitively, we are returning a point that minimizes Rbut which is also close (proximal) to y.
In general, we will use this operator inside an iterative optimizer, in which case we want to stay
close to the previous iterate. In this case, we use
proxR(θk) = argmin
z(
R(z)+1
2||z−θk||2
2)
(13.69)
The key issues are: how do we efficiently compute the proximal operator for different regu-
larizersR, and how do we extend this technique to more general loss functions L? We discuss
these issues below.
13.4.3.1 Proximal operators

z(
R(z)+1
2)
(13.69)
IfR(θ)=λ||θ||1, the proximal operator is given by componentwise soft-thresholding:
proxR(θ)=s o f t (θ,λ) (13.70)
as we showed in Section 13.3.2. If R(θ)=λ||θ||0, the proximal operator is given by componen-
twise hard-thresholding:
proxR(θ)=h a r d ( θ,√
2λ) (13.71)
wherehard(u,a) ≜uI(|u|>a).
IfR(θ)=IC(θ), the proximal operator is given by the projection onto the set C:
proxR(θ) = argmin
z∈C||z−θ||22=p r ojC(θ) (13.72)

444 Chapter 13. Sparse linear models
Figure 13.11 Illustration of projected gradient descent. The step along the negative gradient, to θk−gk,
takes us outside the feasible set. If we project that point onto the closest point in the set we get
θk+1= projΘ(θk−gk). We can then derive the implicit update direction using dk=θk+1−θk. Used
with kind permission of Mark Schmidt.
For some convex sets, it is easy to compute the projection operator. For example, to project
onto the rectangular set deﬁned by the box constraints C={θ:ℓj≤θj≤uj}we can use
projC(θ)j=⎧
⎨
⎩ℓjθj≤ℓj
θjℓj≤θj≤uj
ujθj≥uj(13.73)
To project onto the Euclidean ball C={θ:||θ||2≤1}we can use
projC(θ)={θ
||θ||2||θ||2>1
θ||θ||2≤1(13.74)
To project onto the 1-norm ball C={θ:||θ||1≤1}we can use
projC(θ)=s o f t (θ,λ) (13.75)
whereλ=0if||θ||1≤1, and otherwise λis the solution to the equation
D∑
j=1max(|θj|−λ,0) = 1 (13.76)
We can implement the whole procedure in O(D)time, as explained in (Duchi et al. 2008).

⎨
⎩ℓjθj≤ℓj
D∑
We will see an application of these different projection methods in Section 13.5.1.2.
13.4.3.2 Proximal gradient method
We now discuss how to use the proximal operator inside of a gradient descent routine. The
basic idea is to minimize a simple quadratic approximation to the loss function, centered on the

13.4.ℓ1regularization: algorithms 445
θk:
θk+1=a r g m i n
zR(z)+L(θk)+gT
k(z−θk)+1
2tk||z−θk||2
2 (13.77)
wheregk=∇L(θk)is the gradient of the loss, tkis a constant discussed below, and the last
term arises from a simple approximation to the Hessian of the loss of the form ∇2L(θk)≈1
tkI.
Dropping terms that are independent of z, and multiplying by tk, we can rewrite the above
expression in terms of a proximal operator as follows:
z[
tkR(z)+1
2||z−uk||22]
=p r o xtkR(uk) (13.78)
uk=θk−tkgk (13.79)
gk=∇L(θk) (13.80)
IfR(θ)=0, this is equivalent to gradient descent. If R(θ)=IC(θ), the method is equivalent
toprojected gradient descent, sketched in Figure 13.11. If R(θ)=λ||θ ||1, the method is
known as iterative soft thresholding.
There are several ways to pick tk, or equivalently, αk=1/tk. Given that αkIis an approxi-
mation to the Hessian ∇2L, we require that
αk(θk−θk−1)≈gk−gk−1 (13.81)
in the least squares sense. Hence
αk=a r g m i n
α||α(θk−θk−1)−(gk−gk−1)||22=(θk−θk−1)T(gk−gk−1)

αk=a r g m i n
(θk−θk−1)T(θk−θk−1)(13.82)
This is known as the Barzilai-Borwein (BB) orspectral stepsize (Barzilai and Borwein 1988;
Fletcher 2005; Raydan 1997). This stepsize can be used with any gradient method, whether
proximal or not. It does not lead to monotonic decrease of the objective, but it is much fasterthan standard line search techniques. (To ensure convergence, we require that the objectivedecrease “on average”, where the average is computed over a sliding window of size M+1.)

When we combine the BB stepsize with the iterative soft thresholding technique (for R(θ)=
λ||θ||
1), plus a continuation method that gradually reduces λ, we get a fast method for the
BPDN problem known as the SpaRSA algorithm, which stands for “sparse reconstruction byseparable approximation” (Wright et al. 2009). However, we will call it the iterative shrinkage andthresholding algorithm. See Algorithm 12 for some pseudocode, and SpaRSAfor some Matlab

λ||θ||
code. See also Exercise 13.11 for a related approach based on projected gradient descent.
13.4.3.3 Nesterov’s method
A faster version of proximal gradient descent can be obtained by epxanding the quadraticapproximationaroundapointotherthanthemostrecentparametervalue. Inparticular, considerperforming updates of the form
θ
k+1=p r o xtkR(φk−tkgk) (13.83)
gk=∇L(φk) (13.84)
φk=θk+k−1
k+2(θk−θk−1) (13.85)

446 Chapter 13. Sparse linear models
Algorithm 13.2: Iterative Shrinkage-Thresholding Algorithm (ISTA)
1Input:X∈RN×D,y∈RN, parameters λ≥0,M≥1,0<s<1;
2Initializeθ0=0,α=1,r=y,λ0=∞;
3repeat
4λt= max(s ||XTr||∞,λ)// Adapt the regularizer ;
5repeat
6 g=∇L(θ);
7 u=θ−1
αg;
8 θ=s o f t (u,λt
α);
9 Updateαusing BB stepsize in Equation 13.82 ;
10untilf(θ)increased too much within the past Msteps;
11r=y−Xθ// Update residual ;
12untilλt=λ;
σ2
Nyi
xiDwjτjγ
Figure 13.12 Representing lasso using a Gaussian scale mixture prior.
This is known as Nesterov’s method (Nesterov 2004; Tseng 2008). As before, there are a variety
of ways of setting tk; typically one uses line search.
When this method is combined with the iterative soft thresholding technique (for R(θ)=
λ||θ||1), plus a continuation method that gradually reduces λ, we get a fast method for the
BPDN problem known as the fast iterative shrinkage thesholding algorithm orFISTA(Beck
and Teboulle 2009).

13.4.ℓ1regularization: algorithms 447
13.4.4 EM for lasso
In this section, we show how to solve the lasso problem using lasso. At ﬁrst sight, this might
seem odd, since there are no hidden variables. The key insight is that we can represent theLaplace distribution as a Gaussian scale mixture (GSM) (Andrews and Mallows 1974; West 1987)
Lap(w
j|0,1/γ)=γ
2e−γ|wj|=∫
N(wj|0,τ2
j)Ga(τ2
j|1,γ2
2)dτ2
j (13.86)
Thus the Laplace is a GSM where the mixing distibution on the variances is the exponential
distribution, Expon(τ2
j|γ2
2=G a (τ2
j|1,γ2
2). Using this decomposition, we can represent the
lasso model as shown in Figure 13.12. The corresponding joint distribution has the form
p(y,w,τ,σ2|X)=N (y|Xw,σ2IN)N(w|0,Dτ)
IG(σ2|aσ,bσ)⎡
⎣∏
jGa(τ2
j|1,γ2/2)⎤⎦ (13.87)
whereD
τ=d i a g (τ2
j), and where we have assumed for notational simplicity that Xis stan-
dardized and that yis centered (so we can ignore the offset term μ). Expanding out, we
get
p(y,w,τ,σ2|X)∝(
σ2)−N/2exp(
−1
2σ2||y−Xw||2
2)
|Dτ|−1
2
exp(
−1

j|γ2
2=G a (τ2
j|1,γ2
⎣∏
jGa(τ2
whereD
τ=d i a g (τ2
get
−1
2)
|Dτ|−1
2
exp(
−1
2wTDτw)
(σ2)−(aσ+1)
exp(−bσ/σ2)∏
jexp(−γ2
2τ2
j) (13.88)
Below we describe how to apply the EM algorithm to the model in Figure 13.12.5In brief, in
the E step we infer τ2
jandσ2, and in the M step we estimate w. The resulting estimate ˆwis
the same as the lasso estimator. This approach was ﬁrst proposed in (Figueiredo 2003) (see also
(Griffin and Brown 2007; Caron and Doucet 2008; Ding and Harrison 2010) for some extensions).
13.4.4.1 Why EM?

−1
2)
|Dτ|−1
2
exp(
−1
2wTDτw)
jexp(−γ2
2τ2
j) (13.88)
Before going into the details of EM, it is worthwhile asking why we are presenting this approachat all, given that there are a variety of other (often much faster) algorithms that directly solve theℓ
1MAP estimation problem (see linregFitL1Test for an empirical comparison). The reason
is that the latent variable perspective brings several advantages, such as the following:
• It provides an easy way to derive an algorithm to ﬁnd ℓ1-regularized parameter estimates for

a variety of other models, such as robust linear regression (Exercise 11.12) or probit regression
(Exercise 13.9).
5. To ensure the posterior is unimodal, one can follow (Park and Casella 2008) and slightly modify the model by
making the prior variance for the weights depend on the observation noise: p(wj|τ2
j,σ2)=N(wj|0,σ2τ2
j). The EM
algorithm is easy to modify.

448 Chapter 13. Sparse linear models
• It suggests trying other priors on the variances besides Ga(τ2
j|1,γ2/2). We will consider
various extensions below.
• It makes it clear how we can compute the full posterior, p(w|D), rather than just a MAP
estimate. This technique is known as the Bayesian lasso (Park and Casella 2008; Hans 2009).
13.4.4.2 The objective function
From Equation 13.88, the complete data penalized log likelihood is as follows (dropping terms
that do not depend on w)
ℓc(w)=−1
2−1
2wTΛw+const (13.89)
whereΛ=d i a g (1
τ2
j)is the precision matrix for w.
13.4.4.3 The E step
The key is to compute E[
1
τ2
j|wj]
. We can derive this directly (see Exercise 13.8). Alternatively,
we can derive the full posterior, which is given by the following (Park and Casella 2008):
p(1/τ2
j|w,D)=InverseGaussian(√
γ2
w2
j,γ2)
(13.90)
(Note that the inverse Gaussian distribution is also known as the Wald distribution.) Hence
E[
1
τ2
j|wj]
=γ
|wj|(13.91)
LetΛ=d i a g ( E[
1/τ2
1]
,...,E[
1/τ2
D]

2−1
τ2
1
τ2
j|wj]
p(1/τ2
γ2
w2
j,γ2)
(13.90)
E[
1
τ2
j|wj]
=γ
1/τ2
1]
,...,E[
1/τ2
D]
)denote the result of this E step.
We also need to infer σ2. It is easy to show that that the posterior is
p(σ2|D,w)=I G (aσ+(N)/2,bσ+1
2(y−Xˆw)T(y−Xˆw)) = IG(a N,bN)(13.92)
Hence
E[
1/σ2]
=aN
bN≜ω (13.93)
13.4.4.4 The M step
The M step consists of computing
ˆw=a r g m a x
w−1
2ω||y−Xw||22−1
2wTΛw (13.94)
This is just MAP estimation under a Gaussian prior:
ˆw=(σ2Λ+XTX)−1XTy (13.95)

13.5.ℓ1regularization: extensions 449
However, since we expect many wj=0, we will have τ2
j=0for many j, making inverting Λ
numerically unstable. Fortunately, we can use the SVD of X, given by X=UDVT, as follows:
ˆw=ΨV(VTΨV+1
ωD−2)−1D−1UTy (13.96)
where
Ψ=Λ−1=d i a g (1
E[
1/τ2
j]) = diag(|wj|
π′(wj)) (13.97)
13.4.4.5 Caveat
Since the lasso objective is convex, this method should always ﬁnd the global optimum. Unfor-
tunately, this sometimes does not happen, for numerical reasons. In particular, suppose that inthe true solution, w
∗
j̸=0. Further, suppose that we set ˆwj=0in an M step. In the following E
step we infer that τ2
j=0, so then we set ˆwj=0again; thus we can never “undo” our mistake.
Fortunately, in practice, this situation seems to be rare. See (Hunter and Li 2005) for furtherdiscussion.
13.5ℓ1regularization: extensions
In this section, we discuss various extensions of “vanilla” ℓ1regularization.
13.5.1 Group Lasso

∗
In standard ℓ1regularization, we assume that there is a 1:1 correspondence between parameters
and variables, so that if ˆwj=0, we interpret this to mean that variable jis excluded. But
in more complex models, there may be many parameters associated with a given variable. Inparticular, we may have a vector of weights for each input, w
j. Here are some examples:
•Multinomial logistic regression Each feature is associated with Cdifferent weights, one
per class.
•Linear regression with categorical inputs Each scalar input is one-hot encoded into a
vector of length C.

per class.
•Multi-task learning In multi-task learning, we have multiple related prediction problems.
For example, we might have Cseparate regression or binary classiﬁcation problems. Thus
each feature is associated with Cdifferent weights. We may want to use a feature for all of
the tasks or none of the tasks, and thus select weights at the group level (Obozinski et al.2007).
If we use an ℓ
1regularizer of the form ||w||=∑
j∑
c|wjc|, we may end up with with some

j∑
elements of wj,:being zero and some not. To prevent this kind of situation, we partition the
parameter vector into Ggroups. We now minimize the following objective
J(w) = NLL( w)+G∑
g=1λg||wg||2 (13.98)

450 Chapter 13. Sparse linear models
where
||wg||2=√∑
j∈gw2
j (13.99)
is the 2-norm of the group weight vector. If the NLL is least squares, this method is called
group lasso (Yuan and Lin 2006).
We often use a larger penalty for larger groups, by setting λg=λ√
dg,w h e r edgis the
number of elements in group g. For example, if we have groups {1,2}and{3,4,5}, the
objective becomes
J(w) = NLL( w)+λ[√
2√
(w2
1+w2
2|)+√
3√
(w2
3+w2
4+w2
5)]
(13.100)
Note that if we had used the square of the 2-norms, the model would become equivalent toridge regression, since
G∑
g=1||wg||2
2=∑
g∑
j∈gw2
j=||w||22(13.101)
By using the square root, we are penalizing the radius of a ball containing the group’s weight
vector: the only way for the radius to be small is if all elements are small. Thus the square rootresults in group sparsity.
A variant of this technique replaces the 2-norm with the inﬁnity-norm (Turlach et al. 2005;
Zhao et al. 2005):
||w
g||∞=m a x
j∈g|wj| (13.102)

G∑
2=∑
g∑
j∈gw2
||w
g||∞=m a x
It is clear that this will also result in group sparsity.
An illustration of the difference is shown in Figures 13.13 and 13.14. In both cases, we have a
true signal wof sizeD=212= 4096, divided into 64 groups each of size 64. We randomly
choose 8 groups of wand assign them non-zero values. In the ﬁrst example, the values are
drawn from a N(0,1). In the second example, the values are all set to 1. We then pick a random
design matrix Xof sizeN×D,w h e r eN=210= 1024. Finally, we generate y=Xw+ϵ,
whereϵ∼N(0,10−4IN). Given this data, we estimate the support of wusingℓ1or group ℓ1,

and then estimate the non-zero values using least squares. We see that group lasso does a muchbetter job than vanilla lasso, since it respects the known group structure.
6We also see that the
ℓ∞norm has a tendency to make all the elements within a block to have similar magnitude.
This is appropriate in the second example, but not the ﬁrst. (The value of λwas the same in all
examples, and was chosen by hand.)
13.5.1.1 GSM interpretation of group lasso
Group lasso is equivalent to MAP estimation using the following prior
p(w|γ,σ2)∝exp(
−γ
σG∑
g=1||wg||2)
(13.103)

−γ
σG∑
(13.103)
6. The slight non-zero “noise” in the ℓ∞group lasso results is presumably due to numerical errors.

13.5.ℓ1regularization: extensions 451
0 500 1000 1500 2000 2500 3000 3500 4000−202Original (D = 4096, number groups = 64, active groups = 8)
0 500 1000 1500 2000 2500 3000 3500 4000−202Standard L1 (debiased 1, tau = 0.385, MSE = 0.06929)
(a)
0 500 1000 1500 2000 2500 3000 3500 4000−202Block−L2 (debiased 1, tau = 0.385, MSE = 0.000351)
0 500 1000 1500 2000 2500 3000 3500 4000−202Block−Linf (debiased 1, tau = 0.385, MSE = 0.053)
(b)
Figure 13.13 Illustration of group lasso where the original signal is piecewise Gaussian. Top left: original
signal. Bottom left:: vanilla lasso estimate. Top right: group lasso estimate using a ℓ2norm on the blocks.
Bottom right: group lasso estimate using an ℓ∞norm on the blocks. Based on Figures 3-4 of (Wright et al.
2009). Figure generated by groupLassoDemo , based on code by Mario Figueiredo.
Now one can show (Exercise 13.10) that this prior can be written as a GSM, as follows:
wg|σ2,τ2
g∼N(0,σ2τ2
gIdg) (13.104)
τ2
g|γ∼Ga(dg+1
2,γ
2) (13.105)

wg|σ2,τ2
τ2
2,γ
wheredgis the size of group g. So we see that there is one variance term per group, each
of which comes from a Gamma prior, whose shape parameter depends on the group size, and
whose rate parameter is controlled by γ. Figure 13.15 gives an example, where we have 2 groups,
one of size 2 and one of size 3.
This picture also makes it clearer why there should be a grouping effect. Suppose w1,1is
small; then τ2
1will be estimated to be small, which will force w1,2to be small. Converseley,
supposew1,1is large; then τ2

1will be estimated to be large, which will allow w1,2to be become
large as well.

452 Chapter 13. Sparse linear models
0 500 1000 1500 2000 2500 3000 3500 400000.51Original (D = 4096, number groups = 64, active groups = 8)
0 500 1000 1500 2000 2500 3000 3500 400000.51Standard L1 (debiased 1, tau = 0.356, MSE = 0.1206)
(a)
0 500 1000 1500 2000 2500 3000 3500 400000.51Block−L2 (debiased 1, tau = 0.356, MSE = 0.000342)
0 500 1000 1500 2000 2500 3000 3500 400000.51Block−Linf (debiased 1, tau = 0.356, MSE = 0.000425)
(b)
Figure 13.14 Same as Figure 13.13, except the original signal is piecewise constant.
σ2yi
xiw11w12 w21w22w23τ1 τ2γ
Figure 13.15 Graphical model for group lasso with 2 groups, the ﬁrst has size G1=2, the second has
sizeG2=3.

13.5.ℓ1regularization: extensions 453
13.5.1.2 Algorithms for group lasso
There are a variety of algorithms for group lasso. Here we brieﬂy mention two. The ﬁrst
approach is based on proximal gradient descent, discussed in Section 13.4.3. Since the regularizeris separable, R(w)=∑
g||wg||p, the proximal operator decomposes into Gseparate operators
of the form
proxR(b) = argmin
z∈RDg||z−b||2
2+λ||z||p (13.106)
whereb=θkg−tkgkg.I fp=2, one can show (Combettes and Wajs 2005) that this can be
implemented as follows
proxR(b)=b−projλC(b) (13.107)
whereC={z:||z||2≤1}is theℓ2ball. Using Equation 13.74, if ||b||2<λ,w eh a v e
proxR(b)=b−b=0 (13.108)
otherwise we have
proxR(b)=b−λb
||b||2=b||b||2−λ
||b||2(13.109)
We can combine these into a vectorial soft-threshold function as follows (Wright et al. 2009):
proxR(b)=bmax(||b||2−λ,0)
max(||b||2−λ,0)+λ(13.110)
Ifp=∞, we use C={z:||z||1≤1}, which is the ℓ1ball. We can project onto this in O(dg)
time using an algorithm described in (Duchi et al. 2008).

Another approach is to modify the EM algorithm. The method is almost the same as for
vanilla lasso. If we deﬁne τ2
j=τ2
g(j),w h e r eg(j)is the group to which dimension jbelongs,
we can use the same full conditionals for σ2andwas before. The only changes are as follows:
• We must modify the full conditional for the weight precisions, which are estimated based on
a shared set of weights:
1
τ2g|γ,w,σ2,y,X∼InverseGaussian(√
γ2σ2
||wg||2
2,γ2) (13.111)
where||wg||22=∑
j∈gw2
jg. For the E step, we can use
E[1

j=τ2
1
γ2σ2
||wg||2
j∈gw2
E[1
τ2g]
=γσ
||wg||2(13.112)
• We must modify the full conditional for the tuning parameter, which is now only estimated
based on Gvalues of τ2
g:
p(γ2|τ)=G a (aγ+G/2,b γ+1
2G∑
gτ2
g) (13.113)

454 Chapter 13. Sparse linear models
●●
●●
●
●●
●●
●●
●●●
●●
●●●
●●
●●
●●●●
●●●
●
●●●
●
●●●●
●●
●●●
●●
●●
●●●●
●
●
●
●●●
●●●
●●
●
●●
●●
●●●
●
●●●
●●
●●
●●● ●
●●
●
●
●
●●
●●●
●●●
●
●●
●●
●●
●●●
●
●●
●●
●●●●●
●●
●●●
●●●●●●
●
●●●
●●
●●●●
●●●
●●●
●
●●●●
●
●●●
●●
●
● ●●●●
●●
●●
●●●●●
●●●
●
●●
●●
●
●●●
●●
●●●
●
●●●
●●●
●
●
●●
●●
●
●● ●
●●●
●●●
●●
●
●
●●
●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●
●●●●●
●
●
●●
●
●●● ●
●
●
●●
●
●●
●●●
●
●●●●●
●
●●●●
●●
●
●●●
●●●●●●●●
●
●
●●●
●
●
●●
●
●●
●●
●
●●
●
●
●●●
●●●
●
●●
●●
●
●●●●
●
●
●●●
●
●●
●
●●●●
●●
●●
●
●
●
●●●●
●●●
●●
●
●●●●●
●●
●●●●
●
●●●
●●●●●●
●
●
●●
●●●●●●
●●●
●●●
●
0 100 200 300 400−1.5 −1.0 −0.5 0.0 0.5
IndexCGH
(a) (b) (c)
Figure 13.16 (a) Example of the fused lasso. The vertical axis represents array CGH (chromosomal genome
hybridization) intensity, and the horizontal axis represents location along a genome. Source: Figure 1 of

●●
●●
●●
●●●●
●●●●
●●●●●
●
●
●●
●
●●● ●
●
●
●●
●
●●
●●●
●
●●●●●
●
●●●●
●●
●
●●●
●●●●●●●●
●
●
●●●
●
●
●●
●
●●
●●
●
●●
●
●
●●●
●●●
●
●●
●●
●
●●●●
●
●
●●●
●
●●
●
●●●●
●●
●●
●
●
●
●●●●
●●●
●●
●
●●●●●
●●
●●●●
●
●●●
●●●●●●
●
●
●●
●●●●●●
●●●
●●●
●
IndexCGH
(a) (b) (c)
(Hoeﬂing 2010). (b) Noisy image. (c) Fused lasso estimate using 2d lattice prior. Source: Figure 2 of(Hoeﬂing 2010). Used with kind permission of Holger Hoeﬂing.
13.5.2 Fused lasso
In some problem settings (e.g., functional data analysis), we want neighboring coefficients to be

IndexCGH
(a) (b) (c)
similar to each other, in addition to being sparse. An example is given in Figure 13.16(a), wherewe want to ﬁt a signal that is mostly “off”, but in addition has the property that neighboringlocations are typically similar in value. We can model this by using a prior of the form
p(w|σ
2)∝exp⎛
⎝−λ1
σD∑
j=1|wj|−λ2
σD−1∑
j=1|wj+1−wj|⎞⎠ (13.114)
This is known as the fused lasso penalty. In the context of functional data analysis, we often

p(w|σ
2)∝exp⎛
⎝−λ1
σD∑
σD−1∑
useX=I, so there is one coefficient for each location in the signal (see Section 4.4.2.3). In this
case, the overall objective has the form
J(w,λ
1,λ2)=N∑
i=1(yi−wi)2+λ1N∑
i=1|wi|+λ2N−1∑
i=1|wi+1−wi| (13.115)
This is a sparse version of Equation 4.148.
It is possible to generalize this idea beyond chains, and to consider other graph structures,
using a penalty of the form
J(w,λ1,λ2)=∑
s∈V(ys−ws)2+λ1∑
s∈V|ws|+λ2∑
(s,t)∈E|ws−wt| (13.116)
This is called graph-guided fused lasso (see e.g., (Chen et al. 2010)). The graph might come

J(w,λ
1,λ2)=N∑
from some prior knowledge, e.g., from a database of known biological pathways. Another
example is shown in Figure 13.16(b-c), where the graph structure is a 2d lattice.

13.5.ℓ1regularization: extensions 455
13.5.2.1 GSM interpretation of fused lasso
One can show (Kyung et al. 2010) that the fused lasso model is equivalent to the following
hierarchical model
w|σ2,τ,ω∼N(0,σ2Σ(τ,ω)) (13.117)
τ2
j|γ1∼Expon(γ2
1
2),j=1:D (13.118)
ω2
j|γ2∼Expon(γ2
2
2),j=1:D−1 (13.119)
whereΣ=Ω−1, andΩis a tridiagonal precision matrix with
main diagonal ={1
τ2
j+1
ω2
j−1+1
ω2
j} (13.120)
off diagonal ={−1
ω2
j} (13.121)
where we have deﬁned ω−2
0=ω−2
D=0. This is very similar to the model in Section 4.4.2.3,
where we used a chain-structured Gaussian Markov random ﬁeld as the prior, with ﬁxed vari-ance. Here we just let the variance be random. In the case of graph-guided lasso, the structureof the graph is reﬂected in the zero pattern of the Gaussian precision matrix (see Section 19.4.4).
13.5.2.2 Algorithms for fused lasso

τ2
j+1
ω2
j−1+1
ω2
ω2
0=ω−2
It is possible to generalize the EM algorithm to ﬁt the fused lasso model, by exploiting theMarkov structure of the Gaussian prior for efficiency. Direct solvers (which don’t use the latentvariable trick) can also be derived (see e.g., (Hoeﬂing 2010)). However, this model is undeniablymore expensive to ﬁt than the other variants we have considered.
13.5.3 Elastic net (ridge and lasso combined)

Although lasso has proved to be effective as a variable selection technique, it has severalproblems (Zou and Hastie 2005), such as the following:
• If there is a group of variables that are highly correlated (e.g., genes that are in the same
pathway), then the lasso tends to select only one of them, chosen rather arbitrarily. (This

is evident from the LARS algorithm: once one member of the group has been chosen, theremaining members of the group will not be very correlated with the new residual and hencewill not be chosen.) It is usually better to select all the relevant variables in a group. If weknow the grouping structure, we can use group lasso, but often we don’t know the groupingstructure.
• In the D>Ncase, lasso can select at most Nvariables before it saturates.
•I fN>D, but the variables are correlated, it has been empirically observed that the
prediction performance of ridge is better than that of lasso.

456 Chapter 13. Sparse linear models
Zou and Hastie (Zou and Hastie 2005) proposed an approach called the elastic net, which is
a hybrid between lasso and ridge regression, which solves all of these problems. It is apparently
called the “elastic net” because it is “like a stretchable ﬁshing net that retains ’all the big ﬁsh”’(Zou and Hastie 2005).
13.5.3.1 Vanilla version
The vanilla version of the model deﬁnes the following objective function:
J(w,λ
1,λ2)=||y−Xw||2+λ2||w||2
2+λ1||w||1 (13.122)
Notice that this penalty function is strictly convex (assuming λ2>0) so there is a unique global
minimum, even if Xis not full rank.
It can be shown (Zou and Hastie 2005) that any strictly convex penalty on wwill exhibit
agrouping effect, which means that the regression coefficients of highly correlated variables
tend to be equal (up to a change of sign if they are negatively correlated). For example, if two
features are equal, so X:j=X:k, one can show that their estimates are also equal, ˆwj=ˆwk.

By contrast, with lasso, we may have that ˆwj=0andˆwk̸=0or vice versa.
13.5.3.2 Algorithms for vanilla elastic net
It is simple to show (Exercise 13.5) that the elastic net problem can be reduced to a lasso problemon modiﬁed data. In particular, deﬁne
˜X=c(X√
λ2ID)
,˜y=(y
0D×1)
(13.123)
wherec=( 1+λ2)−1
2. Then we solve
˜w=a r gm i n
˜w||˜y−˜X˜w||2+cλ1||˜w||1 (13.124)
and setw=c˜w.
We can use LARS to solve this subproblem; this is known as the LARS-EN algorithm. If we

˜X=c(X√
λ2ID)
,˜y=(y
0D×1)
(13.123)
˜w=a r gm i n
stop the algorithm after mvariables have been included, the cost is O(m3+Dm2). Note that
we can use m=Dif we wish, since ˜Xhas rank D. This is in contrast to lasso, which cannot
select more than Nvariables (before jumping to the OLS solution) if N<D.
When using LARS-EN (or other ℓ1solvers), one typically uses cross-validation to select λ1and
λ2.
13.5.3.3 Improved version

λ2.
Unfortunately it turns out that the “vanilla” elastic net does not produce functions that predictvery accurately, unless it is very close to either pure ridge or pure lasso. Intuitively the reasonis that it performs shrinkage twice: once due to the ℓ
2penalty and again due to the ℓ1penalty.
The solution is simple: undo the ℓ2shrinkage by scaling up the estimates from the vanilla
version. In other words, if w∗is the solution of Equation 13.124, then a better estimate is
ˆw=√
1+λ2˜w (13.125)

13.6. Non-convex regularizers 457
We will call this a corrected estimate.
One can show that the corrected estimates are given by
ˆw=a r gm i n
wwT(XTX+λ2I
1+λ2)
w−2yTXw+λ1||w||1 (13.126)
Now
XTX+λ2I
1+λ2=( 1−ρ)ˆΣ+ρI (13.127)
whereρ=λ2/(1 +λ2). So the the elastic net is like lasso but where we use a version of
ˆΣthat is shrunk towards I. (See Section 4.2.6 for more discussion of regularized estimates of
covariance matrices.)
13.5.3.4 GSM interpretation of elastic net
The implicit prior being used by the elastic net obviously has the form
p(w|σ2)∝exp⎛
⎝−γ1
σD∑
j=1|wj|−γ2
2σ2D∑
j=1w2
j⎞⎠ (13.128)
which is just a product of Gaussian and Laplace distributions.
This can be written as a hierarchical prior as follows (Kyung et al. 2010; Chen et al. 2011):
w
j|σ2,τ2
j∼N(0,σ2(τ−2
j+γ2)−1) (13.129)
τ2
1
2) (13.130)
Clearly if γ2=0, this reduces to the regular lasso.
It is possible to perform MAP estimation in this model using EM, or Bayesian inference using

⎝−γ1
σD∑
2σ2D∑
j=1w2
w
j|σ2,τ2
τ2
1
MCMC (Kyung et al. 2010) or variational Bayes (Chen et al. 2011).
13.6 Non-convex regularizers
Although the Laplace prior results in a convex optimization problem, from a statistical point

w
j|σ2,τ2
τ2
1
of view this prior is not ideal. There are two main problems with it. First, it does not putenough probability mass near 0, so it does not sufficiently suppress noise. Second, it doesnot put enough probability mass on large values, so it causes shrinkage of relevant coefficients,corresponding to “signal”. (This can be seen in Figure 13.5(a): we see that ℓ
1estimates of large
coefficients are signiﬁcantly smaller than their ML estimates, a phenomenon known as bias.)

Both problems can be solved by going to more ﬂexible kinds of priors which have a larger
spike at 0 and heavier tails. Even though we cannot ﬁnd the global optimum anymore, thesenon-convex methods often outperform ℓ
1regularization, both in terms of predictive accuracy
and in detecting relevant variables (Fan and Li 2001; Schniter et al. 2008). We give some examplesbelow.

458 Chapter 13. Sparse linear models
13.6.1 Bridge regression
A natural generalization of ℓ1regularization, known as bridge regression (Frank and Friedman
1993), has the form
ˆw= NLL(w)+λ∑
j|wj|b(13.131)
forb≥0. This corresponds to MAP estimation using a exponential power distribution given
by
ExpPower( w|μ,a,b) ≜b
2aΓ(1+1/b)exp(
−|x−μ|
ab)
(13.132)
Ifb=2, we get the Gaussian distribution (with a=σ√
2), corresonding to ridge regression; if
we setb=1, we get the Laplace distribution, corresponding to lasso; if we set b=0,w eg e t
ℓ0regression, which is equivalent to best subset selection. Unfortunately, the objective is not
convex for b<1, and is not sparsity promoting for b>1. So the ℓ1norm is the tightest convex
approximation to the ℓ0norm.
The effect of changing bis illustrated in Figure 13.17, where we plot the prior for b=2,b=1
andb=0.4; we assume p(w)=p(w1)p(w2). We also plot the posterior after seeing a single
observation, (x,y), which imposes a single linear constraint of the form, y=wTx, with a

certain tolerance controlled by the observation noise (compare to Figure 7.11). We see see that
the mode of the Laplace is on the vertical axis, corresponding to w1=0. By contrast, there are
two modes when using b=0.4, corresponding to two different sparse solutions. When using
the Gaussian, the MAP estimate is not sparse (the mode does not lie on either of the coordinateaxes).
13.6.2 Hierarchical adaptive lasso

Recall that one of the principal problems with lasso is that it results in biased estimates.This is because it needs to use a large value of λto “squash” the irrelevant parameters, but
this then over-penalizes the relevant parameters. It would be better if we could associate adifferent penalty parameter with each parameter. Of course, it is completely infeasible to tuneDparameters by cross validation, but this poses no problem to the Bayesian: we simply make
eachτ
2

eachτ
2
jhave its own private tuning parameter, γj, which are now treated as random variables
coming from the conjugate prior γj∼IG(a,b). The full model is as follows:
γj∼IG(a,b) (13.133)
τ2
j|γj∼Ga(1,γ2
j/2) (13.134)
wj|τ2
j∼N(0,τ2
j) (13.135)
See Figure 13.18(a). This has been called the hierarchical adaptive lasso (HAL) (Lee et al. 2010)
(see also (Lee et al. 2011; Cevher 2009; Armagan et al. 2011)). We can integrate out τ2
j, which

eachτ
2
τ2
wj|τ2
j∼N(0,τ2
j, which
induces a Lap(w j|0,1/γj)distribution on wjas before. The result is that p(wj)is now a
scaled mixture of Laplacians. It turns out that we can ﬁt this model (i.e., compute a local
posterior mode) using EM, as we explain below. The resulting estimate, ˆwHAL, often works

13.6. Non-convex regularizers 459
Figure 13.17 Top: plot of log priorfor three different distributions with unit variance: Gaussian, Laplace
and exponential power. Bottom: plot of log posteriorafter observing a single observation, corresponding
to a single linear constraint. The precision of this observation is shown by the diagonal lines in the top
ﬁgure. In the case of the Gaussian prior, the posterior is unimodal and symmetric. In the case of theLaplace prior, the posterior is unimodal and asymmetric (skewed). In the case of the exponential prior, theposterior is bimodal. Based on Figure 1 of (Seeger 2008). Figure generated by sparsePostPlot , written
by Florian Steinke.
much better than the estimate returned by lasso, ˆwL1, in the sense that it is more likely to
contain zeros in the right places (model selection consistency) and more likely to result in good
predictions (prediction consistency) (Lee et al. 2010). We give an explanation for this behavior inSection 13.6.2.2.
13.6.2.1 EM for HAL

Since the inverse Gamma is conjugate to the Laplace, we ﬁnd that the E step for γ
jis given by
p(γj|wj)=I G (a+1,b+|wj|) (13.136)
The E step for σ2is the same as for vanilla lasso.
The prior for whas the following form:
p(w|γ)=∏
j1
2γjexp(−|w j|/γj) (13.137)
Hence the M step must optimize
ˆw(t+1)=a r g m a x
wlogN(y|Xw,σ2)−∑
j|wj|E[1/γj] (13.138)

460 Chapter 13. Sparse linear models
aσ
bσσ2
yi
xiwjτ2
jγjab
ND
(a)−1 −0.8−0.6−0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1−0.8−0.6−0.4−0.200.20.40.60.81HAL
  
a=1, b=0.01
a=1, b=0.10
a=1, b=1.00
(b)
Figure 13.18 (a) DGM for hierarchical adaptive lasso. (b) Contours of Hierarchical adpative Laplace. Based
on Figure 1 of (Lee et al. 2010). Figure generated by normalGammaPenaltyPlotDemo .
The expectation is given by
E[1/γj]=a+1
b+|w(t)
j|≜s(t)
j (13.139)
Thus the M step becomes a weighted lasso problem:
ˆw(t+1)= argmin
w||y−Xw||2
2+∑
js(t)
j|wj| (13.140)
This is easily solved using standard methods (e.g., LARS). Note that if the coefficient was esti-
mated to be large in the previous iteration (so w(t)
jis large), then the scaling factor s(t)
jwill
be small, so large coefficients are not penalized heavily. Conversely, small coefficients doget
penalized heavily. This is the way that the algorithm adapts the penalization strength of each

j|≜s(t)
j (13.139)
2+∑
js(t)
jwill
coefficient. The result is an estimate that is often much sparser than returned by lasso, but alsoless biased.
N o t et h a ti fw es e ta =b=0, and we only perform 1 iteration of EM, we get a method that
is closely related to the adaptive lasso of (Zou 2006; Zou and Li 2008). This EM algorithm is
also closely related to some iteratively reweighted ℓ
1methods proposed in the signal processing
community (Chartrand and Yin 2008; Candes et al. 2008).
13.6.2.2 Understanding the behavior of HAL

We can get a better understanding of HAL by integrating out γjto get the following marginal
distribution,
p(wj|a,b)=a
2b(|wj|
b+1)−(a+1)
(13.141)

13.6. Non-convex regularizers 461
−10 −8 −6 −4 −2 0 2 4 6 8 10−10−8−6−4−20246810Lasso
wMLEwMAP
(a)−10 −8 −6 −4 −2 0 2 4 6 8 10−10−8−6−4−20246810
wMLEwMAPHAL
  
b = 0.010, a=1
b = 0.100, a=1
b = 1.000, a=1
(b)
Figure 13.19 Thresholding behavior of two penalty functions (negative log priors). (a) Laplace.
(b) Hierarchical adaptive Laplace. Based on Figure 2 of (Lee et al. 2010). Figure generated by
normalGammaThresholdPlotDemo .
This is an instance of the generalized t distribution (McDonald and Newey 1988) (in (Cevher
2009; Armagan et al. 2011), this is called the double Pareto distribution) deﬁned as
GT(w|μ,a,c,q)≜q
2ca1/qB(1/q,a)(
1+|w−μ|q
acq)−(a+1/q)
(13.142)
wherecis the scale parameter (which controls the degree of sparsity), and ais related to the
degrees of freedom. When q=2andc=√
2we recover the standard t distribution; when
a→∞, we recover the exponential power distribution; and when q=1anda=∞we
get the Laplace distribution. In the context of the current model, we see that p(wj|a,b)=

1+|w−μ|q
(13.142)
GT(wj|0,a,b/a, 1).
The resulting penalty term has the form
πλ(wj)≜−logp(wj)=(a+1)log(1+|wj|
b)+const (13.143)
whereλ=(a,b)are the tuning parameters. We plot this penalty in 2d (i.e., we plot πλ(w1)+
πλ(w2)) in Figure 13.18(b) for various values of b. Compared to the diamond-shaped Laplace
penalty, shown in Figure 13.3(a), we see that the HAL penalty looks more like a “star ﬁsh”: it
puts much more density along the “spines”, thus enforcing sparsity more aggressively. Note thatthis penalty is clearly not convex.

We can gain further understanding into the behavior of this penalty function by considering
applying it to the problem of linear regression with an orthogonal design matrix. In this case,

462 Chapter 13. Sparse linear models
p(τ2
j) p(γj)p(wj) Ref
Ga(1,γ2
2)Fixed Lap(0,1/γ)(Andrews and Mallows 1974; West 1987)
Ga(1,γ2
2)I G ( a,b)G T ( 0 ,a,b/a,1)(Lee et al. 2010, 2011; Cevher 2009; Armagan et al. 2011)
Ga(1,γ2
2)G a ( a,b)NEG(a,b) (Griffin and Brown 2007, 2010; Chen et al. 2011)
Ga(δ,γ2
2)Fixed NG (δ,γ) (Griffin and Brown 2007, 2010)
Ga(τ2
j|0,0)-N J (wj) (Figueiredo 2003)
IG(δ
2,δγ2
2)Fixed T(0,δ,γ) (Andrews and Mallows 1974; West 1987)
C+(0,γ)C+(0,b)horseshoe( b)(Carvahlo et al. 2010)
Table 13.2 Some scale mixtures of Gaussians. Abbreviations: C+= half-rectiﬁed Cauchy; Ga= Gamma
(shape and rate parameterization); GT = generalized t; IG= inverse Gamma; NEG = Normal-Exponential-
Gamma; NG = Normal-Gamma; NJ = Normal-Jeffreys. The horseshoe distribution is the name we give
to the distribution induced on wjby the prior described in (Carvahlo et al. 2010); this has no simple

IG(δ
2,δγ2
analytic form. The deﬁnitions of the NEG and NG densities are a bit complicated, but can be found in thereferences. The other distributions are deﬁned in the text.
one can show that the objective becomes
J(w)=1
2||y−Xw||2
2+D∑
j=1πλ(|wj|) (13.144)
=1
2||y−ˆy||2+1
2D∑
j=1(ˆwmle
j−wj)2+D∑
j=1πλ(|wj|) (13.145)
whereˆwmle=XTyis the MLE and ˆy=Xˆwmle. Thus we can compute the MAP estimate
one dimension at a time by solving the following 1d optimization problem:
ˆwj=a r g m i n
wj1
2(ˆwmle
j−wj)2+πλ(wj) (13.146)

J(w)=1
2+D∑
=1
2D∑
j=1(ˆwmle
j−wj)2+D∑
wj1
2(ˆwmle
In Figure 13.19(a) we plot the lasso estimate, ˆwL1, vs the ML estimate, ˆwmle. We see that the
ℓ1estimator has the usual soft-thresholding behavior seen earlier in Figure 13.5(a). However,
this behavior is undesirable since the large magnitude coefficients are also shrunk towards 0,
whereas we would like them to be equal to their unshrunken ML estimates.
In Figure 13.19(b) we plot the HAL estimate, ˆwHAL, vs the ML estimate ˆwmle. We see that

wj1
2(ˆwmle
this approximates the more desirable hard thresholding behavior seen earlier in Figure 13.5(b)much more closely.
13.6.3 Other hierarchical priors
Many other hierarchical sparsity-promoting priors have been proposed; see Table 13.2 for a briefsummary. In some cases, we can analytically derive the form of the marginal prior for w
j.
Generally speaking, this prior is not concave.
A particularly interesting prior is the improper Normal-Jeffreys prior, which has been used

j.
in (Figueiredo 2003). This puts a non-informative Jeffreys prior on the variance, Ga(τ2
j|0,0)∝

13.7. Automatic relevance determination (ARD)/sparse Bayesian learning (SBL) 463
1/τ2
j; the resulting marginal has the form p(wj)=NJ(wj)∝1/|wj|. This gives rise to a
thresholding rule that looks very similar to HAL in Figure 13.19(b), which in turn is very similar
to hard thresholding. However, this prior has no free parameters, which is both a good thing(nothing to tune) and a bad thing (no ability to adapt the level of sparsity).
13.7 Automatic relevance determination (ARD)/sparse Bayesian learning (SBL)
All the methods we have considered so far (except for the spike-and-slab methods in Sec-tion 13.2.1) have used a factorial prior of the form p(w)=∏
jp(wj). We have seen how these
priors can be represented in terms of Gaussian scale mixtures of the form wj∼N(0,τ2
j),w h e r e
τ2
jhas one of the priors listed in Table 13.2. Using these latent variances, we can represent the
model in the form τ2
j→wj→y←X. We can then use EM to perform MAP estimation,
w h e r ei nt h eEs t e pw ei n f e rp( τ2

j),w h e r e
τ2
j|wj), and in the M step we estimate wfromy,Xandτ.
This M step either involves a closed-form weighted ℓ2optimization (in the case of Gaussian
scale mixtures), or a weighted ℓ1optimization (in the case of Laplacian scale mixtures). We also
discussed how to perform Bayesian inference in such models, rather than just computing MAPestimates.
In this section, we discuss an alternative approach based on type II ML estimation (empirical
Bayes), whereby we integrate out wand maximize the marginal likelihood wrt τ. This EB

procedure can be implemented via EM, or via a reweighted ℓ
1scheme, as we will explain below.
Having estimated the variances, we plug them in to compute the posterior mean of the weights,E[w|ˆτ,D]; rather surprisingly (in view of the Gaussian prior), the result is an (approximately)
sparse estimate, for reasons we explain below.
In the context of neural networks, this this method is called called automatic relevance
determination orARD(MacKay 1995b; Neal 1996): see Section 16.5.7.5. In the context of the

linear models we are considering in this chapter, this method is called sparse Bayesian learning
orSBL(Tipping 2001). Combining ARD/SBL with basis function expansion in a linear model
gives rise to a technique called the relevance vector machine (RVM), which we will discuss inSection 14.3.2.
13.7.1 ARD for linear regression

We will explain the procedure in the context of linear regression; ARD for GLMs requires the useof the Laplace (or some other) approximation. case can be It is conventional, when discussingARD / SBL, to denote the weight precisions by α
j=1/τ2
j, and the measurement precision
byβ=1/σ2(do not confuse this with the use of βin statistics to represent the regression
coefficients!). In particular, we will assume the following model:
p(y|x,w,β)=N (y|wTx,1/β) (13.147)
p(w)=N (w|0,A−1) (13.148)

464 Chapter 13. Sparse linear models
whereA=d i a g (α). The marginal likelihood can be computed analytically as follows:
p(y|X,α,β)=∫
N(y|Xw,βIN)N(w|0,A)dw (13.149)
=N(y|0,βIN+XA−1XT) (13.150)
=( 2π)−N/2|Cα|−1
2exp(−1
2yTC−1
αy) (13.151)
where
Cα≜β−1IN+XA−1XT(13.152)
Compare this to the marginal likelihood in Equation 13.13 in the spike and slab model; modulo
theβ=1/σ2factor missing from the second term, the equations are the same, except we have
replaced the binary γj∈{0,1}with continuous αj∈R+. In log form, the objective becomes
ℓ(α,β)≜−1
2logp(y|X,α,β)=l o g|Cα|+yTC−1
αy (13.153)
To regularize the problem, we may put a conjugate prior on each precision, αj∼Ga(a,b)
andβ∼Ga(c,d). The modiﬁed objective becomes
ℓ(α,β)≜−1
2logp(y|X,α,β)+∑
jlogGa(α j|a,b)+logGa( β|c,d) (13.154)
=l o g|Cα|+yTC−1
αy+∑
j(alogαj−bαj)+clogβ−dβ (13.155)
This is useful when performing Bayesian inference for αandβ(Bishop and Tipping 2000).
However, when performing (type II) point estimation, we will use the improper prior a=b=

ℓ(α,β)≜−1
ℓ(α,β)≜−1
αy+∑
c=d=0, which results in maximal sparsity.
Below we describe how to optimize ℓ(α,β)wrt the precision terms αandβ.7This is a
proxy for ﬁnding the most probable model setting of γin the spike and slab model, which in
turn is closely related to ℓ0regularization. In particular, it can be shown (Wipf et al. 2010) that
the objective in Equation 13.153 has many fewer local optima than the ℓ0objective, and hence
is much easier to optimize.
Once we have estimated αandβ, we can compute the posterior over the parameters using

p(w|D,ˆα,ˆβ)=N(μ,Σ) (13.156)
Σ−1=ˆβXTX+A (13.157)
μ=ˆβΣXTy (13.158)
The fact that we compute a posterior over w, while simultaneously encouraging sparsity, is why
the method is called “sparse Bayesian learning”. Nevertheless, since there are many ways to besparse and Bayesian, we will use the “ARD” term instead, even in the linear model context. (Inaddition, SBL is only “being Bayesian” about the values of the coefficients, rather than reﬂectinguncertainty about the set of relevant variables, which is typically of more interest.)

7. An alternative approach to optimizing βis to put a Gamma prior on βand to integrate it out to get a Student
posterior for w(Buntine and Weigend 1991). However, it turns out that this results in a less accurate estimate for
α(MacKay 1999). In addition, working with Gaussians is easier than working with the Student distribution, and the
Gaussian case generalizes more easily to other cases such as logistic regression.

13.7. Automatic relevance determination (ARD)/sparse Bayesian learning (SBL) 465
x
y
(a)Cy
(b)
Figure 13.20 Illustration of why ARD results in sparsity. The vector of inputs xdoes not point towards
the vector of outputs y, so the feature should be removed. (a) For ﬁnite α, the probability density is spread
in directions away from y. (b) When α=∞, the probability density at yis maximized. Based on Figure
8 of (Tipping 2001).
13.7.2 Whence sparsity?
Ifˆαj≈0, we ﬁnd ˆwj≈ˆwmle
j, since the Gaussian prior shrinking wjtowards 0 has zero
precision. However, if we ﬁnd that ˆαj≈∞, then the prior is very conﬁdent that wj=0, and
hence that feature jis “irrelevant”. Hence the posterior mean will have ˆwj≈0. Thus irrelevant
features automatically have their weights “turned off” or “pruned out”.
We now give an intuitive argument, based on (Tipping 2001), about why ML-II should encour-
ageαj→∞for irrelevant features. Consider a 1d linear regression with 2 training examples,

soX=x=(x1,x2), andy=(y1,y2). We can plot xandyas vectors in the plane, as
shown in Figure 13.20. Suppose the feature is irrelevant for predicting the response, so xpoints
in a nearly orthogonal direction to y. Let us see what happens to the marginal likelihood as we
changeα. The marginal likelihood is given by p(y|x,α,β)=N(y|0,C),w h e r e
C=1
βI+1
αxxT(13.159)
Ifαis ﬁnite, the posterior will be elongated along the direction of x, as in Figure 13.20(a).
However, if α=∞, we ﬁnd C=1

C=1
βI+1
βI,s oCis spherical, as in Figure 13.20(b). If |C|is held
constant, the latter assigns higher probability density to the observed response vector y, so this
is the preferred solution. In other words, the marginal likelihood “punishes” solutions where αj
is small but X:,jis irrelevant, since these waste probability mass. It is more parsimonious (from
the point of view of Bayesian Occam’s razor) to eliminate redundant dimensions.
13.7.3 Connection to MAP estimation

ARD seems quite different from the MAP estimation methods we have been considering earlier
in this chapter. In particular, in ARD, we are not integrating out αand optimizing w, but vice

466 Chapter 13. Sparse linear models
versa. Because the parameters wjbecome correlated in the posterior (due to explaining away),
when we estimate αjwe are borrowing information from all the features, not just feature j.
Consequently, the effective prior p(w|ˆα)isnon-factorial, and furthermore it depends on the
dataD(andσ2). However, in (Wipf and Nagarajan 2007), it was shown that ARD can be viewed
as the following MAP estimation problem:
ˆwARD=a r g m i n
wβ||y−Xw||2
2+gARD(w) (13.160)
gARD(w)≜min
α≥0∑
jαjw2
j+log|Cα| (13.161)
The proof, which is based on convex analysis, is a little complicated and hence is omitted.
Furthermore, (Wipf and Nagarajan 2007; Wipf et al. 2010) prove that MAP estimation with
non-factorial priors is strictly better than MAP estimation with any possible factorial prior in

α≥0∑
jαjw2
the following sense: the non-factorial objective always has fewer local minima than factorialobjectives, while still satisfying the property that the global optimum of the non-factorial objec-tive corresponds to the global optimum of the ℓ
0objective — a property that ℓ1regularization,
which has no local minima, does not enjoy.
13.7.4 Algorithms for ARD *
In this section, we review several different algorithms for implementing ARD.
13.7.4.1 EM algorithm

The easiest way to implement SBL/ARD is to use EM. The expected complete data log likelihoodis given by
Q(α,β)=E[
logN(y|Xw,σ
2I)+logN(w|0,A−1)]
(13.162)
=1
2E⎡
⎣Nlogβ−β||y−Xw||2+∑
jlogαj−tr(AwwT)⎤⎦+const(13.163)
=1
2Nlogβ−β
2(
||y−Xμ||2+tr(XTXΣ))
+1
2∑
jlogαj−1
2tr[A(μμT+Σ)]+const (13.164)
whereμandΣare computed in the E step using Equation 13.158.
Suppose we put a Ga(a,b) prior onαjand aGa(c,d)prior onβ. The penalized objective
becomes
Q′(α,β)=Q(α,β)+∑
j(alogαj−bαj)+clogβ−dβ (13.165)
SettingdQ′
dαj=0we get the following M step:
αj=1+2a
E[
w2
j]
+2b=1+2a
m2

Q(α,β)=E[
(13.162)
=1
2E⎡
=1
2Nlogβ−β
2(
+1
2∑
jlogαj−1
becomes
αj=1+2a
E[
w2
j]
+2b=1+2a
m2
j+Σjj+2b(13.166)

13.7. Automatic relevance determination (ARD)/sparse Bayesian learning (SBL) 467
Ifαj=α, anda=b=0, the update becomes
α=D
E[wTw]=D
μTμ+tr(Σ)(13.167)
The update for βis given by
β−1
new=||y−Xμ||2+β−1∑
j(1−αjΣjj)+2d
N+2c(13.168)
(Deriving this is Exercise 13.2.)
13.7.4.2 Fixed-point algorithm
A faster and more direct approach is to directly optimize the objective in Equation 13.155. One
can show (Exercise 13.3) that the equationsdℓ
dαj=0anddℓ
dβ=0lead to the following ﬁxed
point updates:
αj←γj+2a
m2
j+2b(13.169)
β−1←||y−Xμ||2+2d
N−∑
jγj+2c(13.170)
γj≜1−αjΣjj (13.171)
The quantity γjis a measure of how well-determined wjis by the data (MacKay 1992). Hence
γ=∑
jγjis the effective degrees of freedom of the model. See Section 7.5.3 for further
Sinceαandβboth depend on μandΣ(which can be computed using Equation 13.158 or the
Laplace approximation), we need to re-estimate these equations until convergence. (Convergence

αj←γj+2a
m2
N−∑
γ=∑
properties of this algorithm have been studied in (Wipf and Nagarajan 2007).) At convergence,the results are formally identical to those obtained by EM, but since the objective is non-convex,the results can depend on the initial values.
13.7.4.3 Iteratively reweighted ℓ
1algorithm
Another approach to solving the ARD problem is based on the view that it is a MAP estimationproblem. Although the log prior g(w)is rather complex in form, it can be shown to be a
non-decreasing, concave function of |w

j|. This means that it can be solved by an iteratively
reweighted ℓ1problem of the form
wt+1=a r gm i n
wNLL(w)+∑
jλ(t)
j|wj| (13.172)
In (Wipf and Nagarajan 2010), the following procedure for setting the penalty terms is suggested
(based on a convex bound to the penalty function). We initialize with λ(0)
j=1, and then at

468 Chapter 13. Sparse linear models
iterationt+1, compute λ(t+1)
jby iterating the following equation a few times:8
λj←[
X:,j(
σ2I+Xdiag(1/λ j)diag(|w(t+1)
j|))−1
XT)−1X:,j]1
2
(13.173)
We see that the new penalty λjdepends on allthe old weights. This is quite different from the
adaptive lasso method of Section 13.6.2.
To understand this difference, consider the noiseless case where σ2=0, and assume D≫N.
In this case, there are(D
N)
solutions which perfectly reconstruct the data, Xw=y, and which
have sparsity ||w||0=N; these are called basic feasible solutions or BFS. What we want are
solutions that satsify Xw=ybut which are much sparser than this. Suppose the method has
found a BFS. We do not want to increase the penalty on a weight just because it is small (as
in adaptive lasso), since that will just reinforce our current local optimum. Instead, we want toincrease the penalty on a weight if it is small and if we have ||w
(t+1)||<N. The covariance
term(Xdiag(1/λj)diag(|w(t+1)

j|))−1has this effect: if wis a BFS, this matrix will be full rank,
so the penalty will not increase much, but if wis sparser than N, the matrix will not be full
rank, so the penalties associated with zero-valued coefficients will increase, thus reinforcing thissolution (Wipf and Nagarajan 2010).
13.7.5 ARD for logistic regression
Now consider binary logistic regression, p(y|x,w)=B e r ( y|sigm(wTx)), using the same
Gaussian prior, p(w)=N(w|0,A−1). We can no longer use EM to estimate α, since the

Gaussian prior is not conjugate to the logistic likelihood, so the E step cannot be done exactly.One approach is to use a variational approximation to the E step, as discussed in Section 21.8.1.1.A simpler approach is to use a Laplace approximation (see Section 8.4.1) in the E step. We canthen use this approximation inside the same EM procedure as before, except we no longer needto update β. Note, however, that this is not guaranteed to converge.

An alternative is to use the techniques from Section 13.7.4.3. In this case, we can use exact
methods to compute the inner weighted ℓ
1regularized logistic regression problem, and no
approximations are required.
13.8 Sparse coding *
So far, we have been concentrating on sparse priors for supervised learning. In this section, wediscuss how to use them for unsupervised learning.
In Section 12.6, we discussed ICA, which is like PCA except it uses a non-Gaussian prior
for the latent factors z
i. If we make the non-Gaussian prior be sparsity promoting, such as a

Laplace distribution, we will be approximating each observed vector xias a sparse combination
of basis vectors (columns of W); note that the sparsity pattern (controlled by zi) changes from
data case to data case. If we relax the constraint that Wis orthogonal, we get a method called
8. The algorithm in (Wipf and Nagarajan 2007) is equivalent to a single iteration of Equation 13.173. However, since the
equation is cheap to compute (only O(ND||w(t+1)||0)time), it is worth iterating a few times before solving the more
expensive ℓ1problem.

13.8. Sparse coding * 469
Method p(zi) p(W)Worthogonal
PCA Gauss - yes
FA Gauss - no
ICA Non-Gauss - yes
Sparse coding Laplace - no
Sparse PCA Gauss Laplace maybeSparse MF Laplace Laplace no
Table 13.3 Summary of various latent factor models. A dash “-” in the p(W)column means we are
performing ML parameter estimation rather than MAP parameter estimation. Summary of abbreviations:
PCA = principal components analysis; FA = factor analysis; ICA = independent components analysis; MF =matrix factorization.
sparse coding. In this context, we call the factor loading matrix Wadictionary; each column
is referred to as an atom.9In view of the sparse representation, it is common for L>D,i n
which case we call the representation overcomplete.
In sparse coding, the dictionary can be ﬁxed or learned. If it is ﬁxed, it is common to use a
wavelet or DCT basis, since many natural signals can be well approximated by a small number

of such basis functions. However, it is also possible to learn the dictionary, by maximizing thelikelihood
logp(D|W)=N∑
i=1log∫
ziN(xi|Wzi,σ2I)p(zi)dzi (13.174)
We discuss ways to optimize this below, and then we present several interesting applications.
Do not confuse sparse coding with sparse PCA (see e.g., (Witten et al. 2009; Journee et al.
2010)): this puts a sparsity promoting prior on the regression weights W, whereas in sparse
coding, we put a sparsity promoting prior on the latent factors zi. Of course, the two techniques

i=1log∫
can be combined; we call the result sparse matrix factorization, although this term is non-
standard. See Table 13.3 for a summary of our terminology.
13.8.1 Learning a sparse coding dictionary
Since Equation 13.174 is a hard objective to maximize, it is common to make the followingapproximation:
logp(D|W)≈N∑
i=1max
zi[
logN(xi|Wzi,σ2I)+logp(zi)]
(13.175)
Ifp(zi)is Laplace, we can rewrite the NLL as
NLL(W,Z)=N∑
i=11
2||xi−Wzi||2
2+λ||zi||1 (13.176)
9. It is common to denote the dictionary by D, and to denote the latent factors by αi. However, we will stick with the
Wandzinotation.

470 Chapter 13. Sparse linear models
To prevent Wfrom becoming arbitrarily large, it is common to constrain the ℓ2norm of its
columns to be less than or equal to 1. Let us denote this constraint set by
C={W∈RD×Ls.t.wT
jwj≤1} (13.177)
Then we want to solve minW∈C,Z∈RN×LNLL(W,Z). F o raﬁ x e d zi, the optimization over
Wis a simple least squares problem. And for a ﬁxed dictionary W, the optimization problem
overZis identical to the lasso problem, for which many fast algorithms exist. This suggests
an obvious iterative optimization scheme, in which we alternate between optimizing WandZ.
(Mumford 1994) called this kind of approach an analysis-synthesis loop, where estimating the
basisWis the analysis phase, and estimating the coefficients Zis the synthesis phase. In cases
where this is too slow, more sophisticated algorithms can be used, see e.g., (Mairal et al. 2010).
A variety of other models result in an optimization problem that is similar to Equation 13.176.

For example, non-negative matrix factorization orNMF(Paatero and Tapper 1994; Lee and
Seung 2001) requires solving an objective of the form
min
W∈C,Z∈RL×N1
2N∑
i=1||xi−Wzi||2
2s.t.W≥0,zi≥0 (13.178)
(Note that this has no hyper-parameters to tune.) The intuition behind this constraint is that the

min
2N∑
learned dictionary may be more interpretable if it is a positive sum of positive “parts”, ratherthan a sparse sum of atoms that may be positive or negative. Of course, we can combine NMFwith a sparsity promoting prior on the latent factors. This is called non-negative sparse coding
(Hoyer 2004).
Alternatively, we can drop the positivity constraint, but impose a sparsity constraint on both
the factors z
iand the dictionary W. We call this sparse matrix factorization . To ensure strict

convexity, we can use an elastic net type penalty on the weights (Mairal et al. 2010) resulting in
min
W,Z1
2N∑
2+λ||zi||1s.t.||wj||22+γ||wj||1≤1 (13.179)
There are several related objectives one can write down. For example, we can replace the lasso
NLL with group lasso or fused lasso (Witten et al. 2009).
We can also use other sparsity-promoting priors besides the Laplace. For example, (Zhou et al.
2009) propose a model in which the latent factors ziare made sparse using the binary mask

min
W,Z1
2N∑
model of Section 13.2.2. Each bit of the mask can be generated from a Bernoulli distributionwith parameter π, which can be drawn from a beta distribution. Alternatively, we can use a
non-parametric prior, such as the beta process. This allows the model to use dictionaries ofunbounded size, rather than having to specify Lin advance. One can perform Bayesian inference

in this model using e.g., Gibbs sampling or variational Bayes. One ﬁnds that the effective sizeof the dictionary goes down as the noise level goes up, due to the Bayesian Occam’s razor. Thiscan prevent overﬁtting. See (Zhou et al. 2009) for details.
13.8.2 Results of dictionary learning from image patches
One reason that sparse coding has generated so much interest recently is because it explains aninteresting phenomenon in neuroscience. In particular, the dictionary that is learned by applying

13.8. Sparse coding * 471
(a)
 (b)
(c)
 (d)
(e)
 (f)
Figure 13.21 Illustration of the ﬁlters learned by various methods when applied to natural image patches.
(Each patch is ﬁrst centered and normalized to unit norm.) (a) ICA. Figure generated by icaBasisDemo ,
kindly provided by Aapo Hyvarinen. (b) sparse coding. (c) PCA. (d) non-negative matrix factorization. (e)
sparse PCA with low sparsity on weight matrix. (f) sparse PCA with high sparsity on weight matrix. Figure
generated by sparseDictDemo , written by Julien Mairal.

472 Chapter 13. Sparse linear models
sparse coding to patches of natural images consists of basis vectors that look like the ﬁlters that
are found in simple cells in the primary visual cortex of the mammalian brain (Olshausen andField 1996). In particular, the ﬁlters look like bar and edge detectors, as shown in Figure 13.21(b).(In this example, the parameter λwas chosen so that the number of active basis functions
(non-zero components of z
i) is about 10.) Interestingly, using ICA gives visually similar results,
as shown in Figure 13.21(a). By contrast, applying PCA to the same data results in sinusoidalgratings, as shown in Figure 13.21(c); these do not look like cortical cell response patterns.
10It
has therefore been conjectured that parts of the cortex may be performing sparse coding of thesensory input; the resulting latent representation is then further processed by higher levels ofthe brain.
Figure 13.21(d) shows the result of using NMF, and Figure 13.21(e-f) show the results of sparse

10It
PCA, as we increase the sparsity of the basis vectors.
13.8.3 Compressed sensing
Although it is interesting to look at the dictionaries learned by sparse coding, it is not necessarilyvery useful. However, there are some practical applications of sparse coding, which we discussbelow.
Imagine that, instead of observing the data x∈R
D, we observe a low-dimensional projection
of it,y=Rx+ϵwherey∈RM,Ris aM×Dmatrix,M≪D, andϵis a noise term
(usually Gaussian). We assume Ris a known sensing matrix, corresponding to different linear

projections of x. For example, consider an MRI scanner: each beam direction corresponds to a
vector, encoded as a row in R. Figure 13.22 illustrates the modeling assumptions.
Our goal is to infer p(x|y,R). How can we hope to recover all of xif we do not measure
all ofx? The answer is: we can use Bayesian inference with an appropriate prior, that exploits
the fact that natural signals can be expressed as a weighted combination of a small number ofsuitably chosen basis functions. That is, we assume x=Wz,w h e r ezhas a sparse prior, and

Wis suitable dictionary. This is called compressed sensing orcompressive sensing (Candes
et al. 2006; Baruniak 2007; Candes and Wakin 2008; Bruckstein et al. 2009).
For CS to work, it is important to represent the signal in the right basis, otherwise it will
not be sparse. In traditional CS applications, the dictionary is ﬁxed to be a standard form,such as wavelets. However, one can get much better performance by learning a domain-speciﬁcdictionary using sparse coding (Zhou et al. 2009). As for the sensing matrix R, it is often chosen

to be a random matrix, for reasons explained in (Candes and Wakin 2008). However, one canget better performance by adapting the projection matrix to the dictionary (Seeger and Nickish2008; Chang et al. 2009).
13.8.4 Image inpainting and denoising
Suppose we have an image which is corrupted in some way, e.g., by having text or scratchessparsely superimposed on top of it, as in Figure 13.23. We might want to estimate the underlying
10. ThereasonPCAdiscoverssinusoidalgratingpatternsisbecauseitistryingtomodelthecovarianceofthedata, which,

in the case of image patches, is translation invariant. This means cov[I(x,y),I(x′,y′)] =f[
(x−x′)2+(y−y′)2]
for some function f,w h e r eI(x,y)is the image intensity at location (x,y). One can show (Hyvarinen et al. 2009, p125)
that the eigenvectors of a matrix of this kind are always sinusoids of different phases, i.e., PCA discovers a Fourier basis.

13.8. Sparse coding * 473
y Rx Wz λ
Figure 13.22 Schematic DGM for compressed sensing. We observe a low dimensional measurement y
generated by passing xthrough a measurement matrix R, and possibly subject to observation noise with
varianceσ2. We assume that xhas a sparse decomposition in terms of the dictionary Wand the latent
variables z. the parameter λcontrolls the sparsity level.
(a)
 (b)
Figure 13.23 An example of image inpainting using sparse coding. Left: original image. Right: recon-
struction. Source: Figure 13 of (Mairal et al. 2008). Used with kind permission of Julien Mairal.
“clean” image. This is called image inpainting . One can use similar techniques for image
denoising .
We can model this as a special kind of compressed sensing problem. The basic idea is as
follows. We partition the image into overlapping patches, yi, and concatenate them to form y.
We deﬁne Rso that the i’th row selects out patch i. Now deﬁne Vto be the visible (uncorrupted)

components of y, andHto be the hidden components. To perform image inpainting, we just
computep(yH|yV,θ),w h e r eθare the model parameters, which specify the dictionary Wand
the sparsity level λofz. We can either learn a dictionary offline from a database of images, or
we can learn a dictionary just for this image, based on the non-corrupted patches.
Figure 13.23 shows this technique in action. The dictionary (of size 256 atoms) was learned
from7×106undamaged 12×12color patches in the 12 mega-pixel image.

An alternative approach is to use a graphical model (e.g., the ﬁelds of experts model (S.

474 Chapter 13. Sparse linear models
and Black 2009)) which directly encodes correlations between neighboring image patches, rather
than using a latent variable model. Unfortunately such models tend to be computationally moreexpensive.
Exercises
Exercise 13.1 Partial derivative of the RSS
Deﬁne
RSS(w)=||Xw−y||2
2 (13.180)
∂
∂wkRSS(w)=a kwk−ck (13.181)
ak=2n∑
i=1x2ik=2||x:,k||2(13.182)
ck=2n∑
i=1xik(yi−wT
−kxi,−k)=2xT:,krk (13.183)
wherew−k=wwithout component k,xi,−kisxiwithout component k, andrk=y−wT
−kx:,−k
is the residual due to using all the features except feature k. Hint: Partition the weights into those
involving kand those not involving k.
b. Show that if∂
∂wkRSS(w)=0, then
ˆwk=xT
:,krk
||x:,k||2(13.184)
Hence when we sequentially add features, the optimal weight for feature kis computed by computing
orthogonally projecting x:,konto the current residual.
Exercise 13.2 Derivation of M step for EB for linear regression

−kx:,−k
ˆwk=xT
:,krk
Derive Equations 13.166 and 13.168. Hint: the following identity should be useful
ΣXTX=ΣXTX+β−1ΣA−β−1ΣA (13.185)
=Σ(XTXβ+A)β−1−β−1ΣA (13.186)
=(A+βXTX)−1(XTXβ+A)β−1−β−1ΣA (13.187)
=(I−AΣ)β−1(13.188)
Exercise 13.3 Derivation of ﬁxed point updates for EB for linear regression
Derive Equations 13.169 and 13.170. Hint: The easiest way to derive this result is to rewrite logp(D|α,β)
as in Equation 8.54. This is exactly equivalent, since in the case of a Gaussian prior and likelihood, the

posterior is also Gaussian, so the Laplace “approximation” is exact. In this case, we get
logp(D|α,β)=N
2logβ−β
+1
2∑
jlogαj−1
2mTAm+1
2log|Σ|−D
2log(2π) (13.189)
The rest is straightforward algebra.

13.8. Sparse coding * 475
Exercise 13.4 Marginal likelihood for linear regression
Suppose we use a g-prior of the form Σγ=g(XT
γXγ)−1. Show that Equation 13.16 simpliﬁes to
p(D|γ)∝(1+g)−Dγ/2(2bσ+S(γ))−(2aσ+N−1)/2(13.190)
S(γ)=yTy−g
1+gyTXγ(XT
γXγ)−1XTγy (13.191)
Exercise 13.5 Reducing elastic net to lasso
Deﬁne
J1(w)=|y−Xw|2+λ2|w|2+λ1|w|1 (13.192)
and
J2(w)=|˜y−˜X˜w|2+cλ1|w|1 (13.193)
2and
˜X=c(X√
λ2Id)
,˜y=(y
0d×1)
(13.194)
Show
argminJ1(w)=c(argmin J2(w)) (13.195)
i.e.
J1(cw)=J2(w) (13.196)
and hence that one can solve an elastic net problem using a lasso solver on modiﬁed data.
Exercise 13.6 Shrinkage in linear regression
(Source: Jaakkola.) Consider performing linear regression with an orthonormal design matrix, so ||x:,k||2
2=
1for each column (feature) k, andxT
:,kx:,j=0, so we can estimate each parameter wkseparately.
Figure 13.24 plots ˆwkvsck=2yTx:,k, the correlation of feature kwith the response, for 3 different

Show
i.e.
2=
esimation methods: ordinary least squares (OLS), ridge regression with parameter λ2, and lasso with
parameter λ1.
a. Unfortunately we forgot to label the plots. Which method does the solid (1), dotted (2) and dashed (3)
line correspond to? Hint: see Section 13.3.3.
b. What is the value of λ1?
c. What is the value of λ2?
Exercise 13.7 Prior for the Bernoulli rate parameter in the spike and slab model
Consider the model in Section 13.2.1. Suppose we put a prior on the sparsity rates, πj∼Beta(α 1,α2).

Derive an expression for p(γ|α)after integrating out the πj’s. Discuss some advantages and disadvantages
of this approach compared to assuming πj=π0for ﬁxedπ0.

476 Chapter 13. Sparse linear models
−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−1−0.8−0.6−0.4−0.200.20.40.60.81
ckwk1
2
3
Figure 13.24 Plot ofˆwkvs amount of correlation ckfor three different estimators.
Exercise 13.8 Deriving E step for GSM prior
Show that
E[1
τ2
j|wj]
=π′(wj)
|wj|(13.197)
whereπ(wj)=−logp(wj)andp(wj)=intN(wj|0,τ2
j)p(τ2
j)dτ2
j. Hint 1:
1
τ2
jN(wj|0,τ2
j)∝1
τ2
jexp(−w2
j
2τ2
j) (13.198)
=−1
|wj|−2wj
2τ2
jexp(−w2
j
2τ2
j) (13.199)
=−1
|wj|d
d|wj|N(wj|0,τ2
j) (13.200)
Hint 2:
d
d|wj|p(wj)=1
p(wj)d
d|wj|logp(wj) (13.201)
Exercise 13.9 EM for sparse probit regression with Laplace prior
Derive an EM algorithm for ﬁtting a binary probit classiﬁer (Section 9.4) using a Laplace prior on the
weights. (If you get stuck, see (Figueiredo 2003; Ding and Harrison 2010).)
Exercise 13.10 GSM representation of group lasso
Consider the prior τ2
j∼Ga(δ,ρ2/2), ignoring the grouping issue for now. The marginal distribution
induced on the weights by a Gamma mixing distribution is called the normal Gamma distribution and is

13.8. Sparse coding * 477
given by
NG(w j|δ,ρ)=∫
N(wj|0,τ2
j)Ga(τ2
j|δ,ρ2/2)dτ2
j (13.202)
=1
Z|wj|δ−1/2Kδ−1
2(ρ|wj|) (13.203)
1/Z=ρδ+1
2√π2δ−1/2ρ(δ)(13.204)
whereKα(x)is the modiﬁed Bessel function of the second kind (the besselk function in Matlab).
Now suppose we have the following prior on the variances
p(σ2
1:D)=G∏
g=1p(σ21:d
g),p(σ21:d
g)=∏
j∈gGa(τ2
j|δg,ρ2/2) (13.205)
The corresponding marginal for each group of weights has the form
p(wg)∝|ug|δg−dg/2Kδg−dg/2(ρug) (13.206)
where
ug≜√∑
j∈gw2
g,j=||wg||2 (13.207)
Now suppose δg=(dg+1)/2,s oδg−dg/2=1
2. Conveniently, we have K1
2(z)=√π
2zexp(−z). Show
that the resulting MAP estimate is equivalent to group lasso.
Exercise 13.11 Projected gradient descent for ℓ1regularized least squares
Consider the BPDN problem argminθRSS(θ)+λ||θ||1. By using the split variable trick introducted in
Section 7.4 (i.e., by deﬁning θ=[θ+,θ−]), rewrite this as a quadratic program with a simple bound

where
ug≜√∑
j∈gw2
2(z)=√π
constraint. Then sketch how to use projected gradient descent to solve this problem. (If you get stuck,
consult (Figueiredo et al. 2007).)
Exercise 13.12 Subderivative of the hinge loss function
Letf(x)=( 1−x)+be the hinge loss function, where (z)+=m a x ( 0 ,z). What are ∂f(0),∂f(1), and
∂f(2)?
Exercise 13.13 Lower bounds to convex functions
Letfbe a convex function. Explain how to ﬁnd a global affine lower bound to fat an arbitrary point
x∈dom(f).

14 Kernels
14.1 Introduction
So far in this book, we have been assuming that each object that we wish to classify or cluster
or process in anyway can be represented as a ﬁxed-size feature vector, typically of the formx
i∈RD. However, for certain kinds of objects, it is not clear how to best represent them
as ﬁxed-sized feature vectors. For example, how do we represent a text document or proteinsequence, which can be of variable length? or a molecular structure, which has complex 3dgeometry? or an evolutionary tree, which has variable size and shape?
One approach to such problems is to deﬁne a generative model for the data, and use the
inferred latent representation and/or the parameters of the model as features, and then to plugthese features in to standard methods. For example, in Chapter 28, we discuss deep learning,which is essentially an unsupervised way to learn good feature representations.
Another approach is to assume that we have some way of measuring the similarity between

objects, that doesn’t require preprocessing them into feature vector format. For example, whencomparing strings, we can compute the edit distance between them. Let κ(x,x
′)≥0be some
measure of similarity between objects x,x′∈X,w h e r eXis some abstract space; we will call κ
akernel function. Note that the word “kernel” has several meanings; we will discuss a different
interpretation in Section 14.7.1.
In this chapter, we will discuss several kinds of kernel functions. We then describe some

algorithms that can be written purely in terms of kernel function computations. Such methodscan be used when we don’t have access to (or choose not to look at) the “inside” of the objectsxthat we are processing.
14.2 Kernel functions
We deﬁne a kernel function to be a real-valued function of two arguments, κ(x,x′)∈R,f o r
x,x′∈X. Typically the function is symmetric (i.e., κ(x,x′)=κ(x′,x)), and non-negative (i.e.,
κ(x,x′)≥0), so it can be interpreted as a measure of similarity, but this is not required. We

give several examples below.

480 Chapter 14. Kernels
14.2.1 RBF kernels
Thesquared exponential kernel (SE kernel) or Gaussian kernel is deﬁned by
κ(x,x′)=e x p(
−1
2(x−x′)TΣ−1(x−x′))
(14.1)
IfΣis diagonal, this can be written as
κ(x,x′)=e x p⎛
⎝−1
2D∑
j=11
σ2
j(xj−x′
j)2⎞
⎠ (14.2)
We can interpret the σjas deﬁning the characteristic length scale of dimension j.I fσj=∞,
the corresponding dimension is ignored; hence this is known as the ARD kernel.I f Σis
spherical, we get the isotropic kernel
−||x−x′||2
2σ2)
(14.3)
Hereσ2is known as the bandwidth. Equation 14.3 is an example of a a radial basis function
orRBFkernel, since it is only a function of ||x−x′||.
14.2.2 Kernels for comparing documents
When performing document classiﬁcation or retrieval, it is useful to have a way of comparing
two documents, xiandxi′. If we use a bag of words representation, where xijis the number
of times words joccurs in document i, we can use the cosine similarity, which is deﬁned by
κ(xi,xi′)=xT
ixi′
||xi||2||xi′||2(14.4)

2σ2)
(14.3)
ixi′
This quantity measures the cosine of the angle between xiandxi′when interpreted as vectors.
Sincexiis a count vector (and hence non-negative), the cosine similarity is between 0 and 1,
where 0 means the vectors are orthogonal and therefore have no words in common.
Unfortunately, this simple method does not work very well, for two main reasons. First, if xi
has any word in common with xi′, it is deemed similar, even though some popular words, such

ixi′
as “the” or “and” occur in many documents, and are therefore not discriminative. (These are
known as stop words.) Second, if a discriminative word occurs many times in a document, the
similarity is artiﬁcially boosted, even though word usage tends to be bursty, meaning that oncea word is used in a document it is very likely to be used again (see Section 3.5.5).
Fortunately, we can signiﬁcantly improve performance using some simple preprocessing. The

idea is to replace the word count vector with a new feature vector called the TF-IDFrepresenta-
tion, which stands for “term frequency inverse document frequency”. We deﬁne this as follows.First, the term frequency is deﬁned as a log-transform of the count:
tf(x
ij)≜log(1+xij) (14.5)
This reduces the impact of words that occur many times within one document. Second, theinverse document frequency is deﬁned as
idf(j)≜logN
1+∑N
i=1I(xij>0)(14.6)

14.2. Kernel functions 481
whereNis the total number of documents, and the denominator counts how many documents
contain term j. Finally, we deﬁne
tf-idf(xi)≜[tf(xij)×idf(j)]V
j=1 (14.7)
(There are several other ways to deﬁne the tf and idf terms, see (Manning et al. 2008) for details.)
We then use this inside the cosine similarity measure. That is, our new kernel has the form
κ(xi,xi′)=φ(xi)Tφ(xi′)
||φ(xi)||2||φ(xi′)||2(14.8)
whereφ(x)=tf-idf(x). This gives good results for information retrieval (Manning et al. 2008).
A probabilistic interpretation of the tf-idf kernel is given in (Elkan 2005).
14.2.3 Mercer (positive deﬁnite) kernels
Some methods that we will study require that the kernel function satisfy the requirement thattheGram matrix, deﬁned by
K=⎛
⎜⎝κ(x
1,x1)···κ(x1,xN)
...
κ(xN,x1)···κ(xN,xN)⎞
⎟⎠ (14.9)
be positive deﬁnite for any set of inputs {xi}N
i=1. We call such a kernel a Mercer kernel,o r
positive deﬁnite kernel. It can be shown (Schoelkopf and Smola 2002) that the Gaussian kernel

K=⎛
⎜⎝κ(x
...
⎟⎠ (14.9)
is a Mercer kernel as is the cosine similarity kernel (Sahami and Heilman 2006).
The importance of Mercer kernels is the following result, known as Mercer’s theorem. If the
Gram matrix is positive deﬁnite, we can compute an eigenvector decomposition of it as follows
K=UTΛU (14.10)
whereΛis a diagonal matrix of eigenvalues λi>0. Now consider an element of K:
kij=(Λ1
2U:,i)T(Λ1
2U:j) (14.11)
Let us deﬁne φ(xi)=Λ1
2U:i. Then we can write
kij=φ(xi)Tφ(xj) (14.12)

kij=(Λ1
Thus we see that the entries in the kernel matrix can be computed by performing an innerproduct of some feature vectors that are implicitly deﬁned by the eigenvectors U. In general, if
the kernel is Mercer, then there exists a function φmapping x∈XtoR
Dsuch that
κ(x,x′)=φ(x)Tφ(x′) (14.13)
whereφdepends on the eigen functions ofκ(soDis a potentially inﬁnite dimensional space).
For example, consider the (non-stationary) polynomial kernel κ(x,x′)=(γxTx′+r)M,
wherer>0. One can show that the corresponding feature vector φ(x)will contain all terms

Dsuch that
up to degree M. For example, if M=2,γ=r=1andx,x′∈R2,w eh a v e
(1+xTx′)2=( 1 + x1x′
1+x2x′2)2(14.14)
=1 + 2 x1x′1+2x2x′2+(x1x1)2+(x2x′2)2+2x1x′1x2x′2(14.15)

482 Chapter 14. Kernels
This can be written as φ(x)Tφ(x′),w h e r e
φ(x)=[ 1,√
2x1,√
2x2,x2
1,x22,√
2x1x2]T(14.16)
So using this kernel is equivalent to working in a 6 dimensional feature space. In the case of
a Gaussian kernel, the feature map lives in an inﬁnite dimensional space. In such a case, it isclearly infeasible to explicitly represent the feature vectors.
An example of a kernel that is not a Mercer kernel is the so-called sigmoid kernel, deﬁned
by
κ(x,x
′)=t a n h ( γxTx′+r) (14.17)
(Note that this uses the tanhfunction even though it is called a sigmoid kernel.) This kernel
was inspired by the multi-layer perceptron (see Section 16.5), but there is no real reason to useit. (For a true “neural net kernel”, which is positive deﬁnite, see Section 15.4.5.)
In general, establishing that a kernel is a Mercer kernel is difficult, and requires techniques

by
κ(x,x
from functional analysis. However, one can show that it is possible to build up new Mercerkernels from simpler ones using a set of standard rules. For example, if κ
1andκ2are both
Mercer, so is κ(x,x′)=κ1(x,x′)+κ2(x,x′). See e.g., (Schoelkopf and Smola 2002) for details.
14.2.4 Linear kernels
Deriving the feature vector implied by a kernel is in general quite difficult, and only possible ifthe kernel is Mercer. However, deriving a kernel from a feature vector is easy: we just use
κ(x,x

κ(x,x
′)=φ(x)Tφ(x′)=⟨φ(x),φ(x′)⟩ (14.18)
Ifφ(x)=x,w eg e tt h elinear kernel, deﬁned by
κ(x,x′)=xTx′(14.19)
This is useful if the original data is already high dimensional, and if the original features areindividually informative, e.g., a bag of words representation where the vocabulary size is large,or the expression level of many genes. In such a case, the decision boundary is likely to berepresentable as a linear combination of the original features, so it is not necessary to work insome other feature space.

Of course, not all high dimensional problems are linearly separable. For example, images are
high dimensional, but individual pixels are not very informative, so image classiﬁcation typicallyrequires non-linear kernels (see e.g., Section 14.2.7).
14.2.5 Matern kernels
TheMatern kernel, which is commonly used in Gaussian process regression (see Section 15.2),
has the following form
κ(r)=21−ν
Γ(ν)(√
2νr
ℓ)ν
Kν(√
2νr
ℓ)
(14.20)

14.2. Kernel functions 483
wherer=||x−x′||,ν>0,ℓ>0, and Kνis a modiﬁed Bessel function. As ν→∞, this
approaches the SE kernel. If ν=1
2, the kernel simpliﬁes to
κ(r)=e x p (−r/ℓ) (14.21)
IfD=1, and we use this kernel to deﬁne a Gaussian process (see Chapter 15), we get the
Ornstein-Uhlenbeck process, which describes the velocity of a particle undergoing Brownian
motion (the corresponding function is continuous but not differentiable, and hence is very“jagged”).
14.2.6 String kernels
The real power of kernels arises when the inputs are structured objects. As an example, we nowdescribe one way of comparing two variable length strings using a string kernel. We follow the
presentation of (Rasmussen and Williams 2006, p100) and (Hastie et al. 2009, p668).
Consider two strings x, andx
′of lengths D,D’, each deﬁned over the alphabet A.
For example, consider two amino acid sequences, deﬁned over the 20 letter alphabet A=
{A,R,N,D,C,E,Q,G,H,I,L,K,M,F,P,S,T,W,Y,V }.L e txbe the following sequence
of length 110

IPTSALVKETLALLSTHRTLLIANETLRIPVPVHKNHQLCTEEIFQGIGTLESQTVQGGTV
ERLFKNLSLIKKYIDGQKKKCGEERRRVNQFLDY LQEFLGVMNTEWI
and letx′be the following sequence of length 153
PHRRDLCSRSIWLARKIRSDLTALTESYVKHQGLWSELTEAER LQENLQAYRTFHVLLA
RLLEDQQVHFTPTEGDFHQAIHTLLLQVAAFAYQIEELMILLEYKIPRNEADGMLFEKKLWGLKV LQE
LSQWTVRSIHDLRFISSHQTGIP
These strings have the substring LQEin common. We can deﬁne the similarity of two strings
to be the number of substrings they have in common.
More formally and more generally, let us say that sis a substring of xif we can write x=usv
for some (possibly empty) strings u,sandv. Now let φs(x)denote the number of times that

substring sappears in string x. We deﬁne the kernel between two strings xandx′as
κ(x,x′)=∑
s∈A∗wsφs(x)φs(x′) (14.22)
wherews≥0andA∗is the set of all strings (of any length) from the alphabet A(this is known
as the Kleene star operator). This is a Mercer kernel, and be computed in O(|x|+|x′|)time
(for certain settings of the weights {ws}) using suffix trees (Leslie et al. 2003; Vishwanathan and
Smola 2003; Shawe-Taylor and Cristianini 2004).
There are various cases of interest. If we set ws=0for|s|>1we get a bag-of-characters

κ(x,x′)=∑
kernel. This deﬁnes φ(x)to be the number of times each character in Aoccurs in x.I fw e
requiresto be bordered by white-space, we get a bag-of-words kernel, where φ(x)counts how
many times each possible word occurs. Note that this is a very sparse vector, since most words

484 Chapter 14. Kernels
optimal partial 
matching
matching
Figure 14.1 Illustration of a pyramid match kernel computed from two images. Used with kind permission
of Kristen Grauman.
will not be present. If we only consider strings of a ﬁxed length k, we get the k-spectrum
kernel. This has been used to classify proteins into SCOP superfamilies (Leslie et al. 2003). For
example if k=3,w eh a v e φLQE(x)=1andφLQE(x′)=2for the two strings above.
Various extensions are possible. For example, we can allow character mismatches (Leslie et al.
2003). And we can generalize string kernels to compare trees, as described in (Collins and Duffy
2002). This is useful for classifying (or ranking) parse trees, evolutionary trees, etc.
14.2.7 Pyramid match kernels
In computer vision, it is common to create a bag-of-words representation of an image by
computing a feature vector (often using SIFT (Lowe 1999)) from a variety of points in the image,

commonly chosen by an interest point detector. The feature vectors at the chosen places are
then vector-quantized to create a bag of discrete symbols.
One way to compare two variable-sized bags of this kind is to use a pyramid match kernel
(Grauman and Darrell 2007). The basic idea is illustrated in Figure 14.1. Each feature set is
mapped to a multi-resolution histogram. These are then compared using weighted histogram
intersection. It turns out that this provides a good approximation to the similarity measure one

would obtain by performing an optimal bipartite match at the ﬁnest spatial resolution, and then
summing up pairwise similarities between matched points. However, the histogram method is
faster and is more robust to missing and unequal numbers of points. This is a Mercer kernel.

14.2. Kernel functions 485
14.2.8 Kernels derived from probabilistic generative models
Suppose we have a probabilistic generative model of feature vectors, p(x|θ). Then there are
several ways we can use this model to deﬁne kernel functions, and thereby make the model
suitable for discriminative tasks. We sketch two approaches below.
14.2.8.1 Probability product kernels
One approach is to deﬁne a kernel as follows:
κ(xi,xj)=∫
p(x|xi)ρp(x|xj)ρdx (14.23)
whereρ>0, and p(x|xi)is often approximated by p(x|ˆθ(xi)),w h e r e ˆθ(xi)is a parameter
estimate computed using a single data vector. This is called a probability product kernel
(Jebara et al. 2004).
Although it seems strange to ﬁt a model to a single data point, it is important to bear in
mind that the ﬁtted model is only being used to see how similar two objects are. In particular,if we ﬁt the model to x
iand then the model thinks xjis likely, this means that xiandxjare
similar. For example, suppose p(x|θ)=N(μ,σ2I),w h e r eσ2is ﬁxed. If ρ=1, and we use

ˆμ(xi)=xiandˆμ(xj)=xj, we ﬁnd (Jebara et al. 2004, p825) that
κ(xi,xj)=1
(4πσ2)D/2exp(
−1
4σ2||xi−xj||2)
(14.24)
which is (up to a constant factor) the RBF kernel.
It turns out that one can compute Equation 14.23 for a variety of generative models, including
ones with latent variables, such as HMMs. This provides one way to deﬁne kernels on variablelength sequences. Furthermore, this technique works even if the sequences are of real-valuedvectors, unlike the string kernel in Section 14.2.6. See (Jebara et al. 2004) for further details.

−1
(14.24)
14.2.8.2 Fisher kernels
A more efficient way to use generative models to deﬁne kernels is to use a Fisher kernel
(Jaakkola and Haussler 1998) which is deﬁned as follows:
κ(x,x
′)=g(x)TF−1g(x′) (14.25)
wheregis the gradient of the log likelihood, or score vector, evaluated at the MLE ˆθ
g(x)≜∇θlogp(x|θ)⏐⏐ˆθ(14.26)
andFis the Fisher information matrix, which is essentially the Hessian:
F=∇∇logp(x|θ)⏐⏐
ˆθ(14.27)
Note that ˆθis a function of all the data, so the similarity of xandx′is computed in the context

κ(x,x
ˆθ(14.27)
of all the data as well. Also, note that we only have to ﬁt one model.
The intuition behind the Fisher kernel is the following: let g(x)be the direction (in parameter
space) in which xwould like the parameters to move (from ˆθ) so as to maximize its own

486 Chapter 14. Kernels
(a)0 1 2 3 4 5 6−101234567poly10
(b)0 1 2 3 4 5 6−101234567rbf prototypes
(c)
Figure 14.2 (a) xor truth table. (b) Fitting a linear logistic regression classiﬁer using degree 10 polynomial
expansion. (c) Same model, but using an RBF kernel with centroids speciﬁed by the 4 black crosses. Figure
generated by logregXorDemo .
likelihood; call this the directional gradient. Then we say that two vectors xandx′are similar
if their directional gradients are similar wrt the the geometry encoded by the curvature of the
likelihood function (see Section 7.5.3).
Interestingly, it was shown in (Saunders et al. 2003) that the string kernel of Section 14.2.6
is equivalent to the Fisher kernel derived from an L’th order Markov chain (see Section 17.2).
Also, it was shown in (Elkan 2005) that a kernel deﬁned by the inner product of TF-IDF vectors
(Section 14.2.2) is approximately equal to the Fisher kernel for a certain generative model of text

based on the compound Dirichlet multinomial model (Section 3.5.5).
14.3 Using kernels inside GLMs
In this section, we discuss one simple way to use kernels for classiﬁcation and regression. We
will see other approaches later.
14.3.1 Kernel machines
We deﬁne a kernel machine to be a GLM where the input feature vector has the form
φ(x)=[κ(x,μ1),...,κ(x,μK)] (14.28)
whereμk∈Xa r eas e to f Kcentroids .I fκis an RBF kernel, this is called an RBF network .
We discuss ways to choose the μkparameters below. We will call Equation 14.28 a kernelised

feature vector . Note that in this approach, the kernel need not be a Mercer kernel.
We can use the kernelized feature vector for logistic regression by deﬁning p(y|x,θ)=
Ber(wTφ(x)). This provides a simple way to deﬁne a non-linear decision boundary. As an
example, consider the data coming from the exclusive or orxorfunction. This is a binary-
valued function of two binary inputs. Its truth table is shown in Figure 14.2(a). In Figure 14.2(b),
we have show some data labeled by the xor function, but we have jitteredthe points to make

the picture clearer.1We see we cannot separate the data even using a degree 10 polynomial.
1. Jittering is a common visualization trick in statistics, wherein points in a plot/display that would otherwise land on
top of each other are dispersed with uniform additive noise.

14.3. Using kernels inside GLMs 487
0 5 10 15 20−1001020
0 5 10 15 2000.20.40.60.8
2 4 6 8 105
10
15
20
0 5 10 15 2000.010.020.030.04
2 4 6 8 105
10
15
20
0 5 10 15 207.27.47.67.88x 10−3
2 4 6 8 105
10
15
20
Figure 14.3 RBF basis in 1d. Left column: ﬁtted function. Middle column: basis functions evaluated on
a grid. Right column: design matrix. Top to bottom we show different bandwidths: τ=0.1,τ=0.5,
τ=5 0. Figure generated by linregRbfDemo .
However, using an RBF kernel and just 4 prototypes easily solves the problem as shown in
Figure 14.2(c).
We can also use the kernelized feature vector inside a linear regression model by deﬁning
p(y|x,θ)=N(wTφ(x),σ2). For example, Figure 14.3 shows a 1d data set ﬁt with K=1 0
uniformly spaced RBF prototypes, but with the bandwidth ranging from small to large. Small
values lead to very wiggly functions, since the predicted function value will only be non-zero for

pointsxthat are close to one of the prototypes μk. If the bandwidth is very large, the design
matrix reduces to a constant matrix of 1’s, since each point is equally close to every prototype;
hence the corresponding function is just a straight line.
14.3.2 L1VMs, RVMs, and other sparse vector machines
The main issue with kernel machines is: how do we choose the centroids μk? If the input is
low-dimensional Euclidean space, we can uniformly tile the space occupied by the data with

prototypes, as we did in Figure 14.2(c). However, this approach breaks down in higher numbers
of dimensions because of the curse of dimensionality. If μk∈RD, we can try to perform
numerical optimization of these parameters (see e.g., (Haykin 1998)), or we can use MCMC
inference, (see e.g., (Andrieu et al. 2001; Kohn et al. 2001)), but the resulting objective function
/ posterior is highly multimodal. Furthermore, these techniques is hard to extend to structured
input spaces, where kernels are most useful.

Another approach is to ﬁnd clusters in the data and then to assign one prototype per cluster

488 Chapter 14. Kernels
center (many clustering algorithms just need a similarity metric as input). However, the regions
of space that have high density are not necessarily the ones where the prototypes are mostuseful for representing the output, that is, clustering is an unsupervised task that may not yielda representation that is useful for prediction. Furthermore, there is the need to pick the numberof clusters.
A simpler approach is to make each example x
ibe a prototype, so we get
φ(x)=[κ(x,x1),...,κ(x,xN)] (14.29)
Now we see D=N, so we have as many parameters as data points. However, we can use any
of the sparsity-promoting priors for wdiscussed in Chapter 13 to efficiently select a subset of
the training exemplars. We call this a sparse vector machine.
The most natural choice is to use ℓ1regularization (Krishnapuram et al. 2005). (Note that in
the multi-class case, it is necessary to use group lasso, since each exemplar is associated with C

weights, one per class.) We call this L1VM, which stands for “ℓ 1-regularized vector machine”. By
analogy, we deﬁne the use of an ℓ2regularizer to be a L2VMor “ℓ2-regularized vector machine”;
this of course will not be sparse.
We can get even greater sparsity by using ARD/SBL, resulting in a method called the rele-
vance vector machine orRVM(Tipping 2001). One can ﬁt this model using generic ARD/SBL

algorithms, although in practice the most common method is the greedy algorithm in (Tippingand Faul 2003) (this is the algorithm implemented in Mike Tipping’s code, which is bundled withPMTK).
Another very popular approach to creating a sparse kernel machine is to use a support
vector machine orSVM. This will be discussed in detail in Section 14.5. Rather than using a

sparsity-promoting prior, it essentially modiﬁes the likelihood term, which is rather unnaturalfrom a Bayesian point of view. Nevertheless, the effect is similar, as we will see.
In Figure 14.4, we compare L2VM, L1VM, RVM and an SVM using the same RBF kernel on a
binary classiﬁcation problem in 2d. For simplicity, λwas chosen by hand for L2VM and L1VM;
for RVMs, the parameters are estimated using empirical Bayes; and for the SVM, we use CV topickC=1/λ, since SVM performance is very sensitive to this parameter (see Section 14.5.3).

We see that all the methods give similar performance. However, RVM is the sparsest (and hencefastest at test time), then L1VM, and then SVM. RVM is also the fastest to train, since CV for anSVM is slow. (This is despite the fact that the RVM code is in Matlab and the SVM code is inC.) This result is fairly typical.
In Figure 14.5, we compare L2VM, L1VM, RVM and an SVM using an RBF kernel on a 1d
regression problem. Again, we see that predictions are quite similar, but RVM is the sparsest,then L2VM, then SVM. This is further illustrated in Figure 14.6.
14.4 The kernel trick

Rather than deﬁning our feature vector in terms of kernels, φ(x)=[κ(x,x1),...,κ(x,xN)],
we can instead work with the original feature vectors x, but modify the algorithm so that it
replaces all inner products of the form ⟨x,x′⟩with a call to the kernel function, κ(x,x′). This
is called the kernel trick. It turns out that many algorithms can be kernelized in this way. We
give some examples below. Note that we require that the kernel be a Mercer kernel for this trickto work.

14.4. The kernel trick 489
−2 −1 0 1 2 3−3−2−10123logregL2, nerr=174
(a)−2 −1 0 1 2 3−3−2−10123logregL1, nerr=169
(b)
−2 −1 0 1 2 3−3−2−10123RVM, nerr=173
(c)−2 −1 0 1 2 3−3−2−10123SVM, nerr=173
(d)
Figure 14.4 Example of non-linear binary classiﬁcation using an RBF kernel with bandwidth σ=0.3. (a)
L2VM with λ=5. (b) L1VM with λ=1. (c) RVM. (d) SVM with C=1/λchosen by cross validation.
Black circles denote the support vectors. Figure generated by kernelBinaryClassifDemo .
14.4.1 Kernelized nearest neighbor classiﬁcation
Recall that in a 1NN classiﬁer (Section 1.4.2), we just need to compute the Euclidean distance of
a test vector to all the training points, ﬁnd the closest one, and look up its label. This can bekernelized by observing that
||x
i−xi′||2
2=⟨xi,xi⟩+⟨xi′,xi′⟩−2⟨xi,xi′⟩ (14.30)
This allows us to apply the nearest neighbor classiﬁer to structured data objects.
14.4.2 Kernelized K-medoids clustering
K-means clustering (Section 11.4.2.5) uses Euclidean distance to measure dissimilarity, which is

||x
i−xi′||2
not always appropriate for structured objects. We now describe how to develop a kernelized

490 Chapter 14. Kernels
−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−0.4−0.200.20.40.60.811.2linregL2
(a)−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−0.4−0.200.20.40.60.811.2linregL1
(b)
−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−0.4−0.200.20.40.60.811.2RVM
(c)−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−0.4−0.200.20.40.60.811.2SVM
(d)
Figure 14.5 Example of kernel based regression on the noisy sinc function using an RBF kernel with
bandwidth σ=0.3. (a) L2VM with λ=0.5. (b) L1VM with λ=0.5. (c) RVM. (d) SVM regression with
C=1/λchosen by cross validation, and ϵ=0.1(the default for SVMlight). Red circles denote the
retained training exemplars. Figure generated by kernelRegrDemo .
version of the algorithm.
The ﬁrst step is to replace the K-means algorithm with the K-medoids algorothm. This is
similar to K-means, but instead of representing each cluster’s centroid by the mean of all data

vectors assigned to this cluster, we make each centroid be one of the data vectors themselves.Thus we always deal with integer indexes, rather than data objects. We assign objects to theirclosest centroids as before. When we update the centroids, we look at each object that belongsto the cluster, and measure the sum of its distances to all the others in the same cluster; wethen pick the one which has the smallest such sum:
m
k=a r g m i n
i:zi=k∑
i′:zi′=kd(i,i′) (14.31)

14.4. The kernel trick 491
0 20 40 60 80 100−0.06−0.04−0.0200.020.040.06weights for linregL2
(a)0 20 40 60 80 100−0.06−0.04−0.0200.020.040.060.080.10.12weights for linregL1
(b)
0 20 40 60 80 100−0.200.20.40.60.81weights for RVM
(c)0 20 40 60 80 100−1.5−1−0.500.511.5weights for SVM
(d)
Figure 14.6 Coefficient vectors of length N= 100for the models in Figure 14.6. Figure generated by
kernelRegrDemo .
where
d(i,i′)≜||xi−xi′||2
2 (14.32)
This takes O(n2
k)work per cluster, whereas K-means takes O(nkD)to update each cluster. The
pseudo-code is given in Algorithm 5. This method can be modiﬁed to derive a classiﬁer, by
computing the nearest medoid for each class. This is known as nearest medoid classiﬁcation
(Hastie et al. 2009, p671).
This algorithm can be kernelized by using Equation 14.30 to replace the distance computation,
d(i,i′).

492 Chapter 14. Kernels
Algorithm 14.1: K-medoids algorithm
1initialize m1:Kas a random subset of size Kfrom{1,...,N };
2repeat
3zi=a r g m i nkd(i,mk)fori=1:N;
4mk←argmini:zi=k∑
i′:zi′=kd(i,i′)fork=1:K;
14.4.3 Kernelized ridge regression
Applying the kernel trick to distance-based methods was straightforward. It is not so obvious
how to apply it to parametric models such as ridge regression. However, it can be done, as wenow explain. This will serve as a good “warm up” for studying SVMs.
14.4.3.1 The primal problem
Letx∈R
Dbe some feature vector, and Xbe the corresponding N×Ddesign matrix. We
want to minimize
J(w)=(y−Xw)T(y−Xw)+λ||w||2(14.33)
The optimal solution is given by
w=(XTX+λID)−1XTy=(∑
ixixT
i+λID)−1XTy (14.34)
14.4.3.2 The dual problem
Equation 14.34 is not yet in the form of inner products. However, using the matrix inversion
lemma (Equation 4.107) we rewrite the ridge estimate as follows
w=XT(XXT+λIN)−1y (14.35)

Letx∈R
ixixT
which takes O(N3+N2D)time to compute. This can be advantageous if Dis large. Further-
more, we see that we can partially kernelize this, by replacing XXTwith the Gram matrix K.
But what about the leading XTterm?
Let us deﬁne the following dual variables:
α≜(K+λIN)−1y (14.36)
Then we can rewrite the primal variables as follows
w=XTα=N∑
i=1αixi (14.37)
This tells us that the solution vector is just a linear sum of the Ntraining vectors. When we
plug this in at test time to compute the predictive mean, we get
ˆf(x)=wTx=N∑
i=1αixT
ix=N∑
i=1αiκ(x,xi) (14.38)

14.4. The kernel trick 493
−1 0 1−0.500.511.5Eigenvalue=22.558
−1 0 1−0.500.511.5Eigenvalue=20.936
−1 0 1−0.500.511.5Eigenvalue=4.648
−1 0 1−0.500.511.5Eigenvalue=3.988
−1 0 1−0.500.511.5Eigenvalue=3.372
−1 0 1−0.500.511.5Eigenvalue=2.956
−1 0 1−0.500.511.5Eigenvalue=2.760
−1 0 1−0.500.511.5Eigenvalue=2.211
Figure 14.7 Visualization of the ﬁrst 8 kernel principal component basis functions derived from some 2d
data. We use an RBF kernel with σ2=0.1. Figure generated by kpcaScholkopf , written by Bernhard
Scholkopf.
So we have succesfully kernelized ridge regression by changing from primal to dual variables.
This technique can be applied to many other linear models, such as logistic regression.
14.4.3.3 Computational cost
The cost of computing the dual variables αisO(N3), whereas the cost of computing the primal
variables wisO(D3). Hence the kernel method can be useful in high dimensional settings,

even if we only use a linear kernel (c.f., the SVD trick in Equation 7.44). However, predictionusing the dual variables takes O(ND)time, while prediction using the primal variables only
takesO(D)time. We can speedup prediction by making αsparse, as we discuss in Section 14.5.
14.4.4 Kernel PCA
In Section 12.2, we saw how we could compute a low-dimensional linear embedding of somedata using PCA. This required ﬁnding the eigenvectors of the sample covariance matrix S=

494 Chapter 14. Kernels
1
N∑N
i=1xixT
i=( 1/N)XTX. However, we can also compute PCA by ﬁnding the eigenvectors
of the inner product matrix XXT, as we show below. This will allow us to produce a nonlinear
embedding, using the kernel trick, a method known as kernel PCA (Schoelkopf et al. 1998).
First, letUbe an orthogonal matrix containing the eigenvectors of XXTwith corresponding
eigenvalues in Λ. By deﬁnition we have (XXT)U=UΛ. Pre-multiplying by XTgives
(XTX)(XTU)=(XTU)Λ (14.39)
from which we see that the eigenvectors of XTX(and hence of S)a r eV=XTU, with eigen-
values given by Λas before. However, these eigenvectors are not normalized, since ||vj||2=
uTjXXTuj=λjuTjuj=λj. So the normalized eigenvectors are given by Vpca=XTUΛ−1
2.
This is a useful trick for regular PCA if D>N, sinceXTXhas sizeD×D, whereas XXT
has sizeN×N. It will also allow us to use the kernel trick, as we now show.
Now letK=XXTbe the Gram matrix. Recall from Mercer’s theorem that the use of a kernel

2.
implies some underlying feature space, so we are implicitly replacing xiwithφ(xi)=φi.L e t
Φbe the corresponding (notional) design matrix, and Sφ=1
N∑
iφiφT
ibe the corresponding
(notional) covariance matrix in feature space. The eigenvectors are given by Vkpca=ΦTUΛ−1
2,
whereUandΛcontain the eigenvectors and eigenvalues of K. Of course, we can’t actually
compute Vkpca, sinceφiis potentially inﬁnite dimensional. However, we can compute the
projection of a test vector x∗onto the feature space as follows:

N∑
iφiφT
2,
φT∗Vkpca=φT∗ΦUΛ−1
2=kT
∗UΛ−1
2 (14.40)
wherek∗=[κ(x∗,x1),...,κ(x∗,xN)].
There is one ﬁnal detail to worry about. So far, we have assumed the projected data has
zero mean, which is not the case in general. We cannot simply subtract off the mean in
feature space. However, there is a trick we can use. Deﬁne the centered feature vector as
˜φi=φ(xi)−1
N∑N
j=1φ(xj). The Gram matrix of the centered feature vectors is given by
˜Kij=˜φT
i˜φj (14.41)
=φT
iφj−1
NN∑
k=1φTiφk−1
NN∑
k=1φTjφk+1
N2N∑
k=1M∑
l=1φTkφl (14.42)

2=kT
∗UΛ−1
2 (14.40)
N∑N
˜Kij=˜φT
=φT
iφj−1
NN∑
NN∑
N2N∑
k=1M∑
=κ(xi,xj)−1
NN∑
k=1κ(xi,xk)−1
NN∑
k=1κ(xj,xk)+1
N2N∑
k=1M∑
l=1κ(xk,xl)(14.43)
This can be expressed in matrix notation as follows:
˜K=HKH (14.44)
whereH≜I−1
N1N1T
N. is thecentering matrix. We can convert all this algebra into the
pseudocode shown in Algorithm 9.
Whereas linear PCA is limited to using L≤Dcomponents, in kPCA, we can use up to N
components, since the rank of ΦisN×D∗,w h e r eD∗is the (potentially inﬁnite) dimensionality
of embedded feature vectors. Figure 14.7 gives an example of the method applied to some

NN∑
N2N∑
k=1M∑
N1N1T
D=2dimensional data using an RBF kernel. We project points in the unit grid onto the ﬁrst

14.4. The kernel trick 495
Algorithm 14.2: Kernel PCA
1Input:Kof sizeN×N,K∗of sizeN∗×N, num. latent dimensions L;
2O=1N1T
N/N;
3˜K=K−OK−KO+OKO;
4[U,Λ]=eig( ˜K);
5fori=1:Ndo
6vi=ui/√λi
7O∗=1N∗1TN/N;
8˜K∗=K∗−O∗K∗−K∗O∗+O∗K∗O∗;
9Z=˜K∗V(:,1:L)
−0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8−0.8−0.6−0.4−0.200.20.40.6pca
(a)−0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8−0.8−0.6−0.4−0.200.20.40.6kpca
(b)
Figure 14.8 2d visualization of some 2d data. (a) PCA projection. (b) Kernel PCA projection. Figure
generated by kpcaDemo2 , based on code by L.J.P. van der Maaten.
8 components and visualize the corresponding surfaces using a contour plot. We see that the
ﬁrst two component separate the three clusters, and following components split the clusters.
Although the features learned by kPCA can be useful for classiﬁcation (Schoelkopf et al. 1998),

(b)
they are not necessarily so useful for data visualization. For example, Figure 14.8 shows theprojection of the data from Figure 14.7 onto the ﬁrst 2 principal bases computed using PCA andkPCA. Obviously PCA perfectly represents the data. kPCA represents each cluster by a differentline.
Of course, there is no need to project 2d data back into 2d. So let us consider a different

data set. We will use a 12 dimensional data set representing the three known phases of ﬂowin an oil pipeline. (This data, which is widely used to compare data visualization methods, issynthetic, and comes from (Bishop and James 1993).) We project this into 2d using PCA andkPCA (with an RBF kernel). The results are shown in Figure 14.9. If we perform nearest neighborclassiﬁcation in the low-dimensional space, kPCA makes 13 errors and PCA makes 20 (Lawrence

496 Chapter 14. Kernels
−2 −1 0 1 2−32.5−21.5−10.500.5
(a)0 0.05 0.1 0.15 0.20.050.1.150.2.250.3
(b)
Figure 14.9 2d representation of 12 dimensional oil ﬂow data. The different colors/symbols represent the
3 phases of oil ﬂow. (a) PCA. (b) Kernel PCA with Gaussian kernel. Compare to Figure 15.10(b). From Figure
1 of (Lawrence 2005). Used with kind permission of Neil Lawrence.
2005). Nevertheless, the kPCA projection is rather unnatural. In Section 15.5, we will discuss
how to make kernelized versions of probabilistic PCA.
Note that there is a close connection between kernel PCA and a technique known as mul-
tidimensional scaling or MDS. This methods ﬁnds a low-dimensional embedding such thatEuclidean distance in the embedding space approximates the original dissimilarity matrix. Seee.g., (Williams 2002) for details.
14.5 Support vector machines (SVMs)
In Section 14.3.2, we saw one way to derive a sparse kernel machine, namely by using a GLMwith kernel basis functions, plus a sparsity-promoting prior such as ℓ

1or ARD. An alternative
approach is to change the objective function from negative log likelihood to some other lossfunction, as we discussed in Section 6.5.5. In particular, consider the ℓ
2regularized empirical
risk function
J(w,λ)=N∑
i=1L(yi,ˆyi)+λ||w||2(14.45)
whereˆyi=wTxi+w0. (So far this is in the original feature space; we introduce kernels in a
moment.) If Lis quadratic loss, this is equivalent to ridge regression, and if Lis the log-loss
deﬁned in Equation 6.73, this is equivalent to logistic regression.

J(w,λ)=N∑
In the ridge regression case, we know that the solution to this has the form ˆw=(XTX+
λI)−1XTy, and plug-in predictions take the form ˆw0+ˆwTx. As we saw in Section 14.4.3,
we can rewrite these equations in a way that only involves inner products of the form xTx′,
which we can replace by calls to a kernel function, κ(x,x′). This is kernelized, but not sparse.

However, if we replace the quadratic/ log-loss with some other loss function, to be explainedbelow, we can ensure that the solution is sparse, so that predictions only depend on a subsetof the training data, known as support vectors. This combination of the kernel trick plus a
modiﬁed loss function is known as a support vector machine orSVM. This technique was

14.5. Support vector machines (SVMs) 497
−3 −2 −1 0 1 2 3−0.500.511.522.533.544.55
  
L2
ε−insensitive
huber
(a)xy(x)
y−ϵyy+ϵ
ξ∗>0ξ>0
(b)
Figure 14.10 (a) Illustration of ℓ2, Huber and ϵ-insensitive loss functions, where ϵ=1.5. Figure generated
byhuberLossDemo . (b) Illustration of the ϵ-tube used in SVM regression. Points above the tube have
ξi>0andξ∗
i=0. Points below the tube have ξi=0andξ∗
i>0. Points inside the tube have
ξi=ξ∗
i=0. Based on Figure 7.7 of (Bishop 2006a).
originally designed for binary classiﬁcation, but can be extended to regression and multi-class
classiﬁcation as we explain below.
Note that SVMs are very unnatural from a probabilistic point of view. First, they encode

ξi>0andξ∗
ξi=ξ∗
sparsity in the loss function rather than the prior. Second, they encode kernels by using analgorithmic trick, rather than being an explicit part of the model. Finally, SVMs do not result inprobabilistic outputs, which causes various difficulties, especially in the multi-class classiﬁcationsetting (see Section 14.5.2.4 for details).
It is possible to obtain sparse, probabilistic, multi-class kernel-based classiﬁers, which work as

well or better than SVMs, using techniques such as the L1VM or RVM, discussed in Section 14.3.2.However, we include a discussion of SVMs, despite their non-probabilistic nature, for two mainreasons. First, they are very popular and widely used, so all students of machine learning shouldknow about them. Second, they have some computational advantages over probabilistic methodsin the structured output case; see Section 19.7.
14.5.1 SVMs for regression
The problem with kernelized ridge regression is that the solution vector wdepends on all the

training inputs. We now seek a method to produce a sparse estimate.
Vapnik (Vapnik et al. 1997) proposed a variant of the Huber loss function (Section 7.4) called
theepsilon insensitive loss function, deﬁned by
Lϵ(y,ˆy)≜{0if|y−ˆy|<ϵ
|y−ˆy|−ϵotherwise(14.46)
This means that any point lying inside an ϵ-tubearound the prediction is not penalized, as in
Figure 14.10.
The corresponding objective function is usually written in the following form
J=CN∑
i=1Lϵ(yi,ˆyi)+1
2||w||2(14.47)

498 Chapter 14. Kernels
whereˆyi=f(xi)=wTxi+w0andC=1/λis a regularization constant. This objective is
convex and unconstrained, but not differentiable, because of the absolute value function in the
loss term. As in Section 13.4, where we discussed the lasso problem, there are several possiblealgorithms we could use. One popular approach is to formulate the problem as a constrainedoptimization problem. In particular, we introduce slack variables to represent the degree to
which each point lies outside the tube:
y
i≤f(xi)+ϵ+ξ+
i (14.48)
yi≥f(xi)−ϵ−ξ−
i (14.49)
Given this, we can rewrite the objective as follows:
J=CN∑
i=1(ξ+
i+ξ−
i)+1
2||w||2(14.50)
This is a quadratic function of w, and must be minimized subject to the linear constraints
in Equations 14.48-14.49, as well as the positivity constraints ξ+
i≥0andξ−
i≥0. This is a
standard quadratic program in 2N+D+1variables.
One can show (see e.g., (Schoelkopf and Smola 2002)) that the optimal solution has the form
ˆw=∑
iαixi (14.51)

y
i (14.48)
i (14.49)
J=CN∑
i=1(ξ+
i+ξ−
i)+1
i≥0andξ−
ˆw=∑
whereαi≥0. Furthermore, it turns out that the αvector is sparse, because we don’t care
about errors which are smaller than ϵ. Thexifor which αi>0are called the support vectors;
thse are points for which the errors lie on or outside the ϵtube.
Once the model is trained, we can then make predictions using
ˆy(x)=ˆw0+ˆwTx (14.52)
Plugging in the deﬁnition of ˆwwe get
ˆy(x)=ˆw0+∑
iαixT
ix (14.53)
Finally, we can replace xT
ixwithκ(xi,x)to get a kernelized solution:
iαiκ(xi,x) (14.54)
14.5.2 SVMs for classiﬁcation

iαixT
ix (14.53)
We now discuss how to apply SVMs to classiﬁcation. We ﬁrst focus on the binary case, and
then discuss the multi-class case in Section 14.5.2.4.
14.5.2.1 Hinge loss
In Section 6.5.5, we showed that the negative log likelihood of a logistic regression model,
Lnll(y,η)=−logp(y|x,w)=l o g ( 1+ e−yη) (14.55)

14.5. Support vector machines (SVMs) 499
was a convex upper bound on the 0-1 risk of a binary classiﬁer, where η=f(x)=wTx+w0
is the log odds ratio, and we have assumed the labels are y∈{1,−1}rather than {0,1}.I n
this section, we replace the NLL loss with the hinge loss, deﬁned as
Lhinge(y,η)=m a x ( 0 ,1−yη)=( 1 −yη)+ (14.56)
Hereη=f(x)is our “conﬁdence” in choosing label y=1; however, it need not have any
probabilistic semantics. See Figure 6.7 for a plot. We see that the function looks like a door
hinge, hence its name. The overall objective has the form
min
w,w01
2||w||2+CN∑
i=1(1−yif(xi))+ (14.57)
Once again, this is non-differentiable, because of the max term. However, by introducing slackvariables ξ
i, one can show that this is equivalent to solving
min
w,w0,ξ1
i=1ξis.t.ξi≥0,yi(xT
iw+w0)≥1−ξi,i=1:N (14.58)
This is a quadratic program in N+D+1variables, subjet to O(N)constraints. We
can eliminate the primal variables w,w0andξi, and just solve the Ndual variables, which

min
w,w01
min
w,w0,ξ1
correspond to the Lagrange multipliers for the constraints. Standard solvers take O(N3)time.
However, specialized algorithms, which avoid the use of generic QP solvers, have been developed
for this problem, such as the sequential minimal optimization orSMOalgorithm (Platt 1998).
In practice this can take O(N2). However, even this can be too slow if Nis large. In such
settings, it is common to use linear SVMs, which take O(N)time to train (Joachims 2006; Bottou
et al. 2007).
One can show that the solution has the form
ˆw=∑

ˆw=∑
iαixi (14.59)
whereαi=λiyiand where αis sparse (because of the hinge loss). The xifor which αi>0are
called support vectors; these are points which are either incorrectly classiﬁed, or are classiﬁedcorrectly but are on or inside the margin (we disuss margins below). See Figure 14.12(b) for anillustration.
At test time, prediction is done using
ˆy(x)=s g n (f(x)) = sgn(
ˆw
0+ˆwTx)
(14.60)
Using Equation 14.59 and the kernel trick we have
ˆy(x)=s g n(
ˆw0+N∑
i=1αiκ(xi,x))
(14.61)
This takes O(sD) time to compute, where s≤Nis the number of support vectors. This

ˆw
0+ˆwTx)
(14.60)
ˆw0+N∑
(14.61)
depends on the sparsity level, and hence on the regularizer C.

500 Chapter 14. Kernels
Figure 14.11 Illustration of the large margin principle. Left: a separating hyper-plane with large margin.
Right: a separating hyper-plane with small margin.
R1
R0y=0y>0
y<0
wx
r=f(x)
∥w∥
x⊥
−w0
∥w∥
(a)y=0
y=1y=−1
ξ> 1
ξ< 1
ξ=0ξ=0
(b)
Figure 14.12 (a) Illustration of the geometry of a linear decision boundary in 2d. A point xis classiﬁed
as belonging in decision region R1iff(x)>0, otherwise it belongs in decision region R2;h e r ef(x)
is known as a discriminant function . The decision boundary is the set of points such that f(x)=0.
wis a vector which is perpendicular to the decision boundary. The term w0controls the distance of
the decision boundary from the origin. The signed distance of xfrom its orthogonal projection onto the
decision boundary, x⊥, is given by f(x)/||w||. Based on Figure 4.1 of (Bishop 2006a). (b) Illustration of
the soft margin principle. Points with circles around them are support vectors. We also indicate the value

of the corresponding slack variables. Based on Figure 7.3 of (Bishop 2006a).

14.5. Support vector machines (SVMs) 501
14.5.2.2 The large margin principle
In this section, we derive Equation 14.58 form a completely different perspective. Recall that our
goal is to derive a discriminant function f(x)which will be linear in the feature space implied
by the choice of kernel. Consider a point xin this induced space. Referring to Figure 14.12(a),
we see that
x=x⊥+rw
||w||(14.62)
whereris the distance of xfrom the decision boundary whose normal vector is w, andx⊥is
the orthogonal projection of xonto this boundary. Hence
f(x)=wTx+w0=(wTx⊥+w0)+rwTw
||w||(14.63)
Nowf(x⊥)=0so0=wTx⊥+w0. Hencef(x)=rwTw√
wTw, andr=f(x)
||w||.
We would like to make this distance r=f(x)/||w||as large as possible, for reasons illustrated

we see that
x=x⊥+rw
||w||.
in Figure 14.11. In particular, there might be many lines that perfectly separate the training data(especially if we work in a high dimensional feature space), but intuitively, the best one to pickis the one that maximizes the margin, i.e., the perpendicular distance to the closest point. Inaddition, we want to ensure each point is on the correct side of the boundary, hence we wantf(x
i)yi>0. So our objective becomes
max
w,w0N
min
i=1yi(wTxi+w0)
||w||(14.64)
Note that by rescaling the parameters using w→kwandw0→kw0, we do not change the

max
w,w0N
min
distance of any point to the boundary, since the kfactor cancels out when we divide by ||w||.
Therefore let us deﬁne the scale factor such that yifi=1for the point that is closest to the
decision boundary. We therefore want to optimize
min
w,w01
2||w||2s.t.yi(wTxi+w0)≥1,i=1:N (14.65)
(The fact of1
2is added for convenience and doesn’t affect the optimal parameters.) The
constraint says that we want all points to be on the correct side of the decision boundary witha margin of at least 1. For this reason, we say that an SVM is an example of a large margin
classiﬁer.
If the data is not linearly separable (even after using the kernel trick), there will be no feasible
solution in which y
ifi≥1for alli. We therefore introduce slack variables ξi≥0such that
ξi=0if the point is on or inside the correct margin boundary, and ξi=|yi−fi|otherwise. If

0<ξi≤1the point lies inside the margin, but on the correct side of the decision boundary.
Ifξi>1, the point lies on the wrong side of the decision boundary. See Figure 14.12(b).
We replace the hard constraints that yifi≥0with thesoft margin constraints thatyifi≥
1−ξi. The new objective becomes
min
w,w0,ξ1
iw+w0)≥1−ξi (14.66)

502 Chapter 14. Kernels
0.4 0.6 0.8 1 1.2−6−4−20246Correct log−odds
RVM y(x)
SVM y(x)
Figure 14.13 Log-odds vs xfor 3 different methods. Based on Figure 10 of (Tipping 2001). Used with kind
permission of Mike Tipping.
which is the same as Equation 14.58. Since ξi>1means point iis misclassiﬁed, we can
interpret∑
iξias an upper bound on the number of misclassiﬁed points.
The parameter Cis a regularization parameter that controls the number of errors we are
willing to tolerate on the training set. It is common to deﬁne this using C=1/(νN),w h e r e
0<ν≤1controls the fraction of misclassiﬁed points that we allow during the training phase.
This is called a ν-SVM classiﬁer. This is usually set using cross-validation (see Section 14.5.3).
14.5.2.3 Probabilistic output
An SVM classiﬁer produces a hard-labeling, ˆy(x) = sign(f (x)). However, we often want a
measure of conﬁdence in our prediction. One heuristic approach is to interpret f(x)as the
log-odds ratio, logp(y=1|x)

p(y=0|x). We can then convert the output of an SVM to a probability using
p(y=1|x,θ)=σ(af(x)+b) (14.67)
wherea,bcan be estimated by maximum likelihood on a separate validation set. (Using the
training set to estimate aandbleads to severe overﬁtting.) This technique was ﬁrst proposed in
(Platt 2000).
However, the resulting probabilities are not particularly well calibrated, since there is nothing
in the SVM training procedure that justiﬁes interpreting f(x)as a log-odds ratio. To illustrate
this, consider an example from (Tipping 2001). Suppose we have 1d data where p(x|y=0 )=

Unif(0,1)andp(x|y=1 )=U n i f( 0 .5,1.5). Since the class-conditional distributions overlap in
the middle, the log-odds of class 1 over class 0 should be zero in [0.5,1.0], and inﬁnite outside
this region. We sampled 1000 points from the model, and then ﬁt an RVM and an SVM with

a Gaussian kenel of width 0.1. Both models can perfectly capture the decision boundary, andachieve a generalizaton error of 25%, which is Bayes optimal in this problem. The probabilisticoutput from the RVM is a good approximation to the true log-odds, but this is not the case forthe SVM, as shown in Figure 14.13.

14.5. Support vector machines (SVMs) 503
?
R1 R2
R3 C1
NotC1C2
NotC2
(a)?R1
R2R3C1 C3
C2
C3C1
C2
(b)
Figure 14.14 (a) The one-versus-rest approach. The green region is predicted to be both class 1 and class
2. (b) The one-versus-one approach. The label of the green region is ambiguous. Based on Figure 4.2 of
(Bishop 2006a).
14.5.2.4 SVMs for multi-class classiﬁcation
In Section 8.3.7, we saw how we could “upgrade” a binary logistic regression model to the multi-
class case, by replacing the sigmoid function with the softmax, and the Bernoulli distributionwith the multinomial. Upgrading an SVM to the multi-class case is not so easy, since the outputsare not on a calibrated scale and hence are hard to compare to each other.
The obvious approach is to use a one-versus-the-rest approach (also called one-vs-all), in
which we train Cbinary classiﬁers, f
c(x), where the data from class cis treated as positive,

and the data from all the other classes is treated as negative. However, this can result in regionsof input space which are ambiguously labeled, as shown in Figure 14.14(a).
A common alternative is to pick ˆy(x) = argmax
cfc(x). However, this technique may
not work either, since there is no guarantee that the different fcfunctions have comparable
magnitudes. In addition, each binary subproblem is likely to suffer from the class imbalance
problem. To see this, suppose we have 10 equally represented classes. When training f1,w e

will have 10% positive examples and 90% negative examples, which can hurt performance. It ispossible to devise ways to train all Cclassiﬁers simultaneously (Weston and Watkins 1999), but
the resulting method takes O(C
2N2)time, instead of the usual O(CN2)time.
Another approach is to use the one-versus-one or OVO approach, also called all pairs,i n
which we train C(C−1)/2classiﬁers to discriminate all pairs fc,c′. We then classify a point into
the class which has the highest number of votes. However, this can also result in ambiguities,as shown in Figure 14.14(b). Also, it takes O(C
2N2)time to train and O(C2Nsv)to test each

data point, where Nsvis the number of support vectors.2See also (Allwein et al. 2000) for an
approach based on error-correcting output codes.
It is worth remembering that all of these difficulties, and the plethora of heuristics that have
been proposed to ﬁx them, fundamentally arise because SVMs do not model uncertainty usingprobabilities, so their output scores are not comparable across classes.
2. We can reduce the test time by structuring the classes into a DAG (directed acyclic graph), and performing O(C)

pairwise comparisons (Platt et al. 2000). However, the O(C2)factor in the training time is unavoidable.

504 Chapter 14. Kernels
10−2100102104
10−11001010.10.20.30.40.5
C γcv error
(a)10−210−11001011021031040.20.250.30.35
Ccv errorγ = 5.0
(b)
Figure 14.15 (a) A cross validation estimate of the 0-1 error for an SVM classiﬁer with RBF kernel with
different precisions γ=1/(2σ2)and different regularizer λ=1/C, applied to a synthetic data set drawn
from a mixture of 2 Gaussians. (b) A slice through this surface for γ=5The red dotted line is the Bayes
optimal error, computed using Bayes rule applied to the model used to generate the data. Based on Figure
12.6 of (Hastie et al. 2009). Figure generated by svmCgammaDemo .
14.5.3 Choosing C
SVMs for both classiﬁcation and regression require that you specify the kernel function and the
parameter C. Typically Cis chosen by cross-validation. Note, however, that Cinteracts quite
strongly with the kernel parameters. For example, suppose we are using an RBF kernel withprecision γ=
1
2σ2.I fγ=5, corresponding to narrow kernels, we need heavy regularization,

1
and hence small C(soλ=1/Cis big). If γ=1, a larger value of Cshould be used. So we
see thatγandCare tightly coupled. This is illustrated in Figure 14.15, which shows the CV
estimate of the 0-1 risk as a function of Candγ.
The authors of libsvm recommend (Hsu et al. 2009) using CV over a 2d grid with values C∈
{2−5,2−3,...,215}andγ∈{2−15,2−13,...,23}. In addition, it is important to standardize
the data ﬁrst, for a spherical Gaussian kernel to make sense.
To choose Cefficiently, one can develop a path following algorithm in the spirit of lars

(Section 13.3.4). The basic idea is to start with λlarge, so that the margin 1/||w(λ)||is wide,
and hence all points are inside of it and have αi=1. By slowly decreasing λ, a small set of
points will move from inside the margin to outside, and their αivalues will change from 1 to 0,
as they cease to be support vectors. When λis maximal, the function is completely smoothed,
and no support vectors remain. See (Hastie et al. 2004) for the details.
14.5.4 Summary of key points

Summarizing the above discussion, we recognize that SVM classiﬁers involve three key ingre-dients: the kernel trick, sparsity, and the large margin principle. The kernel trick is necessaryto prevent underﬁtting, i.e., to ensure that the feature vector is sufficiently rich that a linearclassiﬁer can separate the data. (Recall from Section 14.2.3 that any Mercer kernel can be viewedas implicitly deﬁning a potentially high dimensional feature vector.) If the original features arealready high dimensional (as in many gene expression and text classiﬁcation problems), it suf-ﬁces to use a linear kernel, κ(x,x

′)=xTx′, which is equivalent to working with the original
features.

14.6. Comparison of discriminative kernel methods 505
Method Opt. wOpt. kernel Sparse Prob. Multiclass Non-Mercer Section
L2VM Convex EB No Yes Yes Yes 14.3.2
L1VM Convex CV Yes Yes Yes Yes 14.3.2RVM Not convex EB Yes Yes Yes Yes 14.3.2SVM Convex CV Yes No Indirectly No 14.5GP N/A EB No Yes Yes No 15
Table 14.1 Comparison of various kernel based classiﬁers. EB = empirical Bayes, CV = cross validation.
See text for details.
The sparsity and large margin principles are necessary to prevent overﬁtting, i.e., to ensure
that we do not use all the basis functions. These two ideas are closely related to each other,and both arise (in this case) from the use of the hinge loss function. However, there are othermethods of achieving sparsity (such as ℓ
1), and also other methods of maximizing the margin
(such as boosting). A deeper discussion of this point takes us outside of the scope of this book.See e.g., (Hastie et al. 2009) for more information.
14.5.5 A probabilistic interpretation of SVMs

In Section 14.3, we saw how to use kernels inside GLMs to derive probabilistic classiﬁers, such asthe L1VM and RVM. And in Section 15.3, we will discuss Gaussian process classiﬁers, which alsouse kernels. However, all of these approaches use a logistic or probit likelihood, as opposed tothe hinge loss used by SVMs. It is natural to wonder if one can interpret the SVM more directlyas a probabilistic model. To do so, we must interpret Cg(m)as a negative log likelihood, where
g(m)=( 1 −m)

+,w h e r em=yf(x)is the margin. Hence p(y=1|f)=e x p ( −Cg(f))
andp(y=−1|f)=e x p ( −Cg(−f)). By summing over both values of y, we require that
exp(−Cg(f))+exp(−Cg (−f))be a constant independent of f. But it turns out this is not
possible for any C>0(Sollich 2002).
However, if we are willing to relax the sum-to-one condition, and work with a pseudo-
likelihood, we canderive a probabilistic interpretation of the hinge loss (Polson and Scott 2011).
In particular, one can show that
exp(−2(1−yixT
iw)+)=∫∞
01√2πλiexp(
−1

iw)+)=∫∞
−1
2(1+λi−yixT
iw)2
λi)
dλi (14.68)
Thus the exponential of the negative hinge loss can be represented as a Gaussian scale mixture.
This allows one to ﬁt an SVM using EM or Gibbs sampling, where λiare the latent variables. This
in turn opens the door to Bayesian methods for setting the hyper-parameters for the prior onw. See (Polson and Scott 2011) for details. (See also (Franc et al. 2011) for a different probabilistic
interpretation of SVMs.)
14.6 Comparison of discriminative kernel methods

−1
iw)2
λi)
We have mentioned several different methods for classiﬁcation and regression based on kernels,which we summarize in Table 14.1. (GP stands for “Gaussian process”, which we discuss inChapter 15.) The columns have the following meaning:

506 Chapter 14. Kernels
• Optimize w: a key question is whether the objective J(w)=−logp(D|w)−logp(w)
is convex or not. L2VM, L1VM and SVMs have convex objectives. RVMs do not. GPs are
Bayesian methods that do not perform parameter estimation.
• Optimize kernel: all the methods require that one “tune” the kernel parameters, such as the
bandwidth of the RBF kernel, as well as the level of regularization. For methods based onGaussians, including L2VM, RVMs and GPs, we can use efficient gradient based optimizers tomaximize the marginal likelihood. For SVMs, and L1VM, we must use cross validation, whichis slower (see Section 14.5.3).
• Sparse: L1VM, RVMs and SVMs are sparse kernel methods, in that they only use a subset of
the training examples. GPs and L2VM are not sparse: they use all the training examples. Theprinciple advantage of sparsity is that prediction at test time is usually faster. In addition,one can sometimes get improved accuracy.

• Probabilistic: All the methods except for SVMs produce probabilistic output of the form
p(y|x). SVMs produce a “conﬁdence” value that can be converted to a probability, but such
probabilities are usually very poorly calibrated (see Section 14.5.2.3).
• Multiclass: All the methods except for SVMs naturally work in the multiclass setting, by using
a multinoulli output instead of Bernoulli. The SVM can be made into a multiclass classiﬁer,but there are various difficulties with this approach, as discussed in Section 14.5.2.4.
• Mercer kernel: SVMs and GPs require that the kernel is positive deﬁnite; the other techniques
do not.

do not.
Apart from these differences, there is the natural question: which method works best? In
a small experiment
3, we found that all of these methods had similar accuracy when averaged
over a range of problems, provided they have the same kernel, and provided the regularizationconstants are chosen appropriately.
Given that the statistical performance is roughly the same, what about the computational
performance? GPs and L2VM are generally the slowest, taking O(N
3)time, since they don’t

do not.
exploit sparsity (although various speedups are possible, see Section 15.6). SVMs also takeO(N
3)time to train (unless we use a linear kernel, in which case we only need O(N)time
(Joachims 2006)). However, the need to use cross validation can make SVMs slower than RVMs.L1VM should be faster than an RVM, since an RVM requires multiple rounds of ℓ
1minimization
(see Section 13.7.4.3). However, in practice it is common to use a greedy method to train RVMs,which is faster than ℓ

1minimization. This is reﬂected in our empirical results.
The conclusion of all this is as follows: if speed matters, use an RVM, but if well-calibrated

probabilistic output matters (e.g., for active learning or control problems), use a GP. The onlycircumstances under which using an SVM seems sensible is the structured output case, wherelikelihood-based methods can be slow. (We attribute the enormous popularity of SVMs notto their superiority, but to ignorance of the alternatives, and also to the lack of high qualitysoftware implementing the alternatives.)
Section 16.7.1 gives a more extensive experimental comparison of supervised learning methods,
including SVMs and various non kernel methods.

3. See http://pmtk3 .googlecode .com/svn/trunk/docs/tutorial/html/tutKernelClassif .html.

14.7. Kernels for building generative models 507
−1.5 −1 −0.5 0 0.5 1 1.500.10.20.30.40.50.60.70.80.9
  
Boxcar
Epanechnikov
Tricube
Gaussian
Figure 14.16 A comparison of some popular smoothing kernels. The boxcar kernel has compact support
but is not smooth. The Epanechnikov kernel has compact support but is not differentiable at its boundary.
The tri-cube has compact support and two continuous derivatives at the boundary of its support. TheGaussian is differentiable, but does not have compact support. Based on Figure 6.2 of (Hastie et al. 2009).Figure generated by smoothingKernelPlot .
14.7 Kernels for building generative models
There is a different kind of kernel known as a smoothing kernel which can be used to create
non-parametric density estimates. This can be used for unsupervised density estimation, p(x),
as well as for creating generative models for classiﬁcation and regression by making models ofthe form p(y,x).
14.7.1 Smoothing kernels

Asmoothing kernel is a function of one argument which satisﬁes the following properties:∫
κ(x)dx=1,∫
xκ(x)dx=0,∫
x2κ(x)dx >0 (14.69)
A simple example is the Gaussian kernel,
κ(x)≜1
(2π)1
2e−x2/2(14.70)
We can control the width of the kernel by introducing a bandwidth parameter h:
κh(x)≜1
hκ(x
h) (14.71)
We can generalize to vector valued inputs by deﬁning an RBF kernel:
κh(x)=κh(||x||) (14.72)
In the case of the Gaussian kernel, this becomes
κh(x)=1
hD(2π)D/2D∏
j=1exp(−1
2h2x2
j) (14.73)

508 Chapter 14. Kernels
Although Gaussian kernels are popular, they have unbounded support. An alternative kernel,
with compact support, is the Epanechnikov kernel, deﬁned by
κ(x)≜3
4(1−x2)I(|x|≤1) (14.74)
This is plotted in Figure 14.16. Compact support can be useful for efficiency reasons, since one
can use fast nearest neighbor methods to evaluate the density.
Unfortunately, the Epanechnikov kernel is not differentiable at the boundary of its support.
An alterative is the tri-cube kernel, deﬁned as follows:
κ(x)≜70
81(1−|x|3)3I(|x|≤1) (14.75)
This has compact support and has two continuous derivatives at the boundary of its support.See Figure 14.16.
Theboxcar kernel is simply the uniform distribution:
κ(x)≜I(|x|≤1) (14.76)
We will use this kernel below.
14.7.2 Kernel density estimation (KDE)
Recall the Gaussian mixture model from Section 11.2.1. This is a parametric density estimator fordata in R
D. However, it requires specifying the number Kand locations μkof the clusters. An

κ(x)≜70
alternative to estimating the μkis to allocate one cluster center per data point, so μi=xi.I n
this case, the model becomes
p(x|D)=1
NN∑
i=1N(x|xi,σ2I) (14.77)
We can generalize the approach by writing
ˆp(x)=1
NN∑
i=1κh(x−xi) (14.78)
This is called a Parzen window density estimator,o r kernel density estimator (KDE), and

p(x|D)=1
NN∑
ˆp(x)=1
NN∑
is a simple non-parametric density model. The advantage over a parametric model is that nomodel ﬁtting is required (except for tuning the bandwidth, usually done by cross-validation). andthere is no need to pick K. The disadvantage is that the model takes a lot of memory to store,
and a lot of time to evaluate. It is also of no use for clustering tasks.
Figure 14.17 illustrates KDE in 1d for two kinds of kernel. On the top, we use a boxcar kernel,
κ(x)=I(−1≤z≤1). The result is equivalent to a histogram estimate of the density, since
we just count how many data points land within an interval of size haroundx

i. On the bottom,
we use a Gaussian kernel, which results in a smoother ﬁt.
The usual way to pick his to minimize an estimate (such as cross validation) of the frequentist
risk (see e.g., (Bowman and Azzalini 1997)). In Section 25.2, we discuss a Bayesian approach tonon-parametric density estimation, based on Dirichlet process mixture models, which allows us

14.7. Kernels for building generative models 509
−5 0 5 1000.050.10.150.20.250.30.35unif, h=1.000
(a)−5 0 5 1000.020.040.060.080.10.120.14unif, h=2.000
(b)
−5 0 5 1000.020.040.060.080.10.120.140.16gauss, h=1.000
(c)−5 0 5 1000.010.020.030.040.050.06gauss, h=2.000
(d)
Figure 14.17 A nonparametric (Parzen) density estimator in 1D estimated from 6 data points, denoted
by x. Top row: uniform kernel. Bottom row: Gaussian kernel. Rows represent increasingly large band-
width parameters. Based on http://en .wikipedia .org/wiki/Kernel_density_estimation . Figure
generated by parzenWindowDemo2 .
to inferh. DP mixtures can also be more efficient than KDE, since they do not need to store
all the data. See also Section 15.2.4 where we discuss an empirical Bayes approach to estimating
kernel parameters in a Gaussian process model for classiﬁcation/ regression.
14.7.3 From KDE to KNN

We can use KDE to deﬁne the class conditional densities in a generative classiﬁer. This turnsout to provide an alternative derivation of the nearest neighbors classiﬁer, which we introducedin Section 1.4.2. To show this, we follow the presentation of (Bishop 2006a, p125). In kdewith a boxcar kernel, we ﬁxed the bandwidth and count how many data points fall within thehyper-cube centered on a datapoint. Suppose that, instead of ﬁxing the bandwidth h, we instead

510 Chapter 14. Kernels
−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−0.4−0.200.20.40.60.811.2Gaussian kernel regression
  
true
data
estimate
Figure 14.18 An example of kernel regression in 1d using a Gaussian kernel. Figure generated by
kernelRegressionDemo , based on code by Yi Cao.
allow the bandwidth or volume to be different for each data point. Speciﬁcally, we will “grow”
a volume around xuntil we encounter Kdata points, regardless of their class label. Let the
resulting volume have size V(x)(this was previously hD), and let there be Nc(x)examples
from class cin this volume. Then we can estimate the class conditional density as follows:
p(x|y=c,D)=Nc(x)
NcV(x)(14.79)
whereNcis the total number of examples in class cin the whole data set. The class prior can
be estimated by
p(y=c|D)=Nc
N(14.80)
Hence the class posterior is given by
p(y=c|x,D)=Nc(x)
NcV(x)Nc
N∑
c′Nc′(x)
Nc′V(x)Nc′
N=Nc(x)∑
c′Nc′(x)=Nc(x)
K(14.81)
where we used the fact that∑
cNc(x)=K, since we choose a total of Kpoints (regardless of

N(14.80)
NcV(x)Nc
N∑
c′Nc′(x)
N=Nc(x)∑
K(14.81)
class) around every point. This is equivalent to Equation 1.2, since Nc(x)=∑
i∈NK(x,D)I(yi=
c).
14.7.4 Kernel regression
In Section 14.7.2, we discussed the use of kernel density estimation or KDE for unsupervisedlearning. We can also use KDE for regression. The goal is to compute the conditional expectation
f(x)=E[y|x]=∫
yp(y|x)dy=∫
yp(x,y)dy
∫
p(x,y)dy(14.82)

14.7. Kernels for building generative models 511
We can use KDE to approximate the joint density p(x,y)as follows:
p(x,y)≈1
NN∑
i=1κh(x−xi)κh(y−yi) (14.83)
Hence
f(x)=1
N∑N
i=1κh(x−xi)∫
yκh(y−yi)dy
1
N∑Ni=1κh(x−xi)∫
κh(y−yi)dy(14.84)
=∑Ni=1κh(x−xi)yi∑Ni=1κh(x−xi)(14.85)
To derive this result, we used two properties of smoothing kernels. First, that they integrate to
one, i.e.,∫
κh(y−yi)dy=1. And second, the fact that∫
yκh(y−yi)dy=yi. This follows by
deﬁningx=y−yiand using the zero mean property of smoothing kernels:
∫
(x+yi)κh(x)dx=∫
xκh(x)dx+yi∫
κh(x)dx=0+yi=yi (14.86)
We can rewrite the above result as follows:
f(x)=N∑
i=1wi(x)yi (14.87)
wi(x)≜κh(x−xi)∑N
i′=1κh(x−xi′)(14.88)
We see that the prediction is just a weighted sum of the outputs at the training points, where
the weights depend on how similar xis to the stored training points. This method is called
kernel regression, kernel smoothing, or the Nadaraya-Watson model. See Figure 14.18 for an
example, where we use a Gaussian kernel.

∫
f(x)=N∑
Note that this method only has one free parameter, namely h. One can show (Bowman and
Azzalini 1997) that for 1d data, if the true density is Gaussian and we are using Gaussian kernels,the optimal bandwidth his given by
h=(4
3N)1/5
ˆσ (14.89)
We can compute a robust approximation to the standard deviation by ﬁrst computing the mean
absolute deviation
MAD=median(|x−median(x)|) (14.90)
and then using
ˆσ=1.4826MAD=1
0.6745MAD (14.91)
The code used to produce Figure 14.18 estimated hxandhyseparately, and then set h=√
hxhy.

512 Chapter 14. Kernels
Although these heuristics seem to work well, their derivation rests on some rather dubious
assumptions (such as Gaussianity of the true density). Furthermore, these heuristics are limited
to tuning just a single parameter. In Section 15.2.4 we discuss an empirical Bayes approach toestimating multiple kernel parameters in a Gaussian process model for classiﬁcation/ regression,which can handle many tuning parameters, and which is based on much more transparentprinciples (maximizing the marginal likelihood).
14.7.5 Locally weighted regression
If we deﬁne κh(x−xi)=κ(x,xi), we can rewrite the prediction made by kernel regression as
follows
ˆf(x∗)=N∑
i=1yiκ(x∗,xi)∑N
i′=1κ(x∗,xi′)(14.92)
Note that κ(x,xi)need not be a smoothing kernel. If it is not, we no longer need the
normalization term, so we can just write
ˆf(x∗)=N∑
i=1yiκ(x∗,xi) (14.93)
This model is essentially ﬁtting a constant function locally. We can improve on this by ﬁtting a
linear regression model for each point x∗by solving
min

follows
ˆf(x∗)=N∑
ˆf(x∗)=N∑
min
β(x∗)N∑
i=1κ(x∗,xi)[yi−β(x∗)Tφ(xi)]2(14.94)
whereφ(x)=[ 1,x]. This is called locally weighted regression. An example of such a method
isLOESS,a k aLOWESS, which stands for “locally-weighted scatterplot smoothing” (Cleveland
and Devlin 1988). See also (Edakunni et al. 2010) for a Bayesian version of this model.
We can compute the paramters β(x∗)for each test case by solving the following weighted
least squares problem:
β(x∗)=(ΦTD(x∗)Φ)−1ΦTD(x∗)y (14.95)
whereΦis anN×(D+1 )design matrix and D=d i a g (κ(x∗,xi)). The corresponding

prediction has the form
ˆf(x∗)=φ(x∗)Tβ(x∗)=(ΦTD(x∗)Φ)−1ΦTD(x∗)y=N∑
i=1wi(x∗)yi (14.96)
The term wi(x∗), which combines the local smoothing kernel with the effect of linear regression,
is called the equivalent kernel. See also Section 15.4.2.
Exercises
Exercise 14.1 Fitting an SVM classiﬁer by hand
(Source: Jaakkola.) Consider a dataset with 2 points in 1d: (x1=0,y1=−1)and(x2=√
2,y2=1 ).
Consider mapping each point to 3d using the feature vector φ(x)=[ 1,√
2x,x2]T. (This is equivalent to

14.7. Kernels for building generative models 513
using a second order polynomial kernel.) The max margin classiﬁer has the form
min||w||2s.t. (14.97)
y1(wTφ(x1)+w0)≥1 (14.98)
y2(wTφ(x2)+w0)≥1 (14.99)
a. Write down a vector that is parallel to the optimal vector w. Hint: recall from Figure 7.8 (12Apr10
version) that wis perpendicular to the decision boundary between the two points in the 3d feature
space.
b. What is the value of the margin that is achieved by this w? Hint: recall that the margin is the distance
from each support vector to the decision boundary. Hint 2: think about the geometry of 2 points in
space, with a line separating one from the other.
c. Solve for w, using the fact the margin is equal to 1/||w||.
d. Solve for w0using your value for wand Equations 14.97 to 14.99. Hint: the points will be on the
decision boundary, so the inequalities will be tight.
e. Write down the form of the discriminant function f(x)=w0+wTφ(x)as an explicit function of x.
Exercise 14.2 Linear separability

(Source: Koller..) Consider ﬁtting an SVM with C>0to a dataset that is linearly separable. Is the resulting
decision boundary guaranteed to separate the classes?

15 Gaussian processes
15.1 Introduction
In supervised learning, we observe some inputs xiand some outputs yi. We assume that
yi=f(xi), for some unknown function f, possibly corrupted by noise. The optimal approach
is to infer a distribution over functions given the data, p(f|X,y), and then to use this to make
predictions given new inputs, i.e., to compute
p(y∗|x∗,X,y)=∫
p(y∗|f,x∗)p(f|X,y)df (15.1)
Up until now, we have focussed on parametric representations for the function f, so that
instead of inferring p(f|D), we infer p(θ|D). In this chapter, we discuss a way to perform
Bayesian inference over functions themselves.
Our approach will be based on Gaussian processes orGPs. A GP deﬁnes a prior over
functions, which can be converted into a posterior over functions once we have seen some data.
Although it might seem difficult to represent a distribution over a function, it turns out that weonly need to be able to deﬁne a distribution over the function’s values at a ﬁnite, but arbitrary,set of points, say x

1,...,xN. A GP assumes that p(f(x1),...,f(xN))is jointly Gaussian, with
some mean μ(x)and covariance Σ(x)given byΣij=κ(xi,xj),w h e r eκis a positive deﬁnite
kernel function (see Section 14.2 information on kernels). The key idea is that if xiandxjare
deemed by the kernel to be similar, then we expect the output of the function at those pointsto be similar, too. See Figure 15.1 for an illustration.
It turns out that, in the regression setting, all these computations can be done in closed form,
inO(N

inO(N
3)time. (We discuss faster approximations in Section 15.6.) In the classiﬁcation setting,
we must use approximations, such as the Gaussian approximation, since the posterior is nolonger exactly Gaussian.
GPs can be thought of as a Bayesian alternative to the kernel methods we discussed in Chap-

inO(N
ter 14, including L1VM, RVM and SVM. Although those methods are sparser and therefore faster,they do not give well-calibrated probabilistic outputs (see Section 15.4.4 for further discussion).Having properly tuned probabilistic output is important in certain applications, such as onlinetracking for vision and robotics (Ko and Fox 2009), reinforcement learning and optimal control(Engel et al. 2005; Deisenroth et al. 2009), global optimization of non-convex functions (Mockuset al. 1996; Lizotte 2008; Brochu et al. 2009), experiment design (Santner et al. 2003), etc.

516 Chapter 15. Gaussian processes
x1 x2 x⋆f1 f2 f⋆y1 y2 y⋆
Figure 15.1 A Gaussian process for 2 training points and 1 testing point, represented as a mixed directed
and undirected graphical model representing p(y,f|x)=N(f|0,K(x))∏
ip(yi|fi). The hidden nodes
fi=f(xi)represent the value of the function at each of the data points. These hidden nodes are fully
interconnected by undirected edges, forming a Gaussian graphical model; the edge strengths represent the
covariance terms Σij=κ(xi,xj). If the test point x∗is similar to the training points x1andx2, then
the predicted output y∗will be similar to y1andy2.
Our presentation is closely based on (Rasmussen and Williams 2006), which should be con-
sulted for futher details. See also (Diggle and Ribeiro 2007), which discusses the related approach
known as kriging, which is widely used in the spatial statistics literature.
15.2 GPs for regression
In this section, we discuss GPs for regression. Let the prior on the regression function be a GP,denoted by

f(x)∼GP(m(x),κ(x,x
′)) (15.2)
wherem(x)is the mean function and κ(x,x′)is the kernel or covariance function, i.e.,
m(x)=E [f(x)] (15.3)
κ(x,x′)=E[
(f(x)−m(x))(f(x′)−m(x′))T]
(15.4)
We obviously require that κ()be a positive deﬁnite kernel. For any ﬁnite set of points, this
process deﬁnes a joint Gaussian:
p(f|X)=N(f|μ,K) (15.5)
whereKij=κ(xi,xj)andμ=(m(x1),...,m(xN)).
Note that it is common to use a mean function of m(x)=0, since the GP is ﬂexible enough

′)) (15.2)
(15.4)
to model the mean arbitrarily well, as we will see below. However, in Section 15.2.6 we willconsider parametric models for the mean function, so the GP just has to model the residualerrors. This semi-parametric approach combines the interpretability of parametric models withthe accuracy of non-parametric models.

15.2. GPs for regression 517
−5 0 5−2−1.5−1−0.500.511.52
(a)−5 0 5−2−1.5−1−0.500.511.522.5
(b)
Figure 15.2 Left: some functions sampled from a GP prior with SE kernel. Right: some samples from a GP
posterior, after conditioning on 5 noise-free observations. The shaded area represents E[f(x)]±2std(f(x).
Based on Figure 2.2 of (Rasmussen and Williams 2006). Figure generated by gprDemoNoiseFree .
15.2.1 Predictions using noise-free observations
Suppose we observe a training set D={(xi,fi),i=1:N},w h e r efi=f(xi)is the noise-free
observation of the function evaluated at xi. Given a test set X∗of sizeN∗×D, we want to
predict the function outputs f∗.
If we ask the GP to predict f(x)for a value of xthat it has already seen, we want the GP to
return the answer f(x)with no uncertainty. In other words, it should act as an interpolator
of the training data. This will only happen if we assume the observations are noiseless. We will
consider the case of noisy observations below.

Now we return to the prediction problem. By deﬁnition of the GP, the joint distribution has
the following form
(f
f∗)
∼N((μ
μ∗)
,(KK ∗
KT
∗K∗∗))
(15.6)
whereK=κ(X,X)isN×N,K∗=κ(X,X∗)isN×N∗, andK∗∗=κ(X∗,X∗)isN∗×N∗.
By the standard rules for conditioning Gaussians (Section 4.3), the posterior has the following
form
p(f∗|X∗,X,f)=N (f∗|μ∗,Σ∗) (15.7)
μ∗=μ(X∗)+KT
∗K−1(f−μ(X)) (15.8)
Σ∗=K∗∗−KT∗K−1K∗ (15.9)
This process is illustrated in Figure 15.2. On the left we show sample samples from the prior,

(f
f∗)
∼N((μ
μ∗)
,(KK ∗
KT
∗K∗∗))
(15.6)
form
p(f|X), where we use a squared exponential kernel, aka Gaussian kernel or RBF kernel. In
1d, this is given by
κ(x,x′)=σ2
fexp(−1
2ℓ2(x−x′)2) (15.10)
Hereℓcontrols the horizontal length scale over which the function varies, and σ2
fcontrols the
vertical variation. (We discuss how to estimate such kernel parameters below.) On the right we

518 Chapter 15. Gaussian processes
show samples from the posterior, p(f∗|X∗,X,f). We see that the model perfectly interpolates
the training data, and that the predictive uncertainty increases as we move further away from
the observed data.
One application of noise-free GP regression is as a computationally cheap proxy for the
behavior of a complex simulator, such as a weather forecasting program. (If the simulator isstochastic, we can deﬁne fto be its mean output; note that there is still no observation noise.)
One can then estimate the effect of changing simulator parameters by examining their effecton the GP’s predictions, rather than having to run the simulator many times, which may beprohibitively slow. This strategy is known as DACE, which stands for design and analysis ofcomputer experiments (Santner et al. 2003).
15.2.2 Predictions using noisy observations
Now let us consider the case where what we observe is a noisy version of the underlyingfunction, y=f(x)+ϵ,w h e r eϵ∼N(0,σ
2

2
y). In this case, the model is not required to
interpolate the data, but it must come “close” to the observed data. The covariance of theobserved noisy responses is
cov[y
p,yq]=κ(xp,xq)+σ2
yδpq (15.11)
whereδpq=I(p=q). In other words
cov[y|X]=K+σ2
yIN≜Ky (15.12)
The second matrix is diagonal because we assumed the noise terms were independently addedto each observation.
The joint density of the observed data and the latent, noise-free function on the test points
is given by
(y
f
∗)
∼N(
0,(KyK∗
KT
∗K∗∗))
(15.13)

cov[y
is given by
(y
f
∗)
∼N(
0,(KyK∗
KT
∗K∗∗))
(15.13)
where we are assuming the mean is zero, for notational simplicity. Hence the posterior predictive
density is
p(f∗|X∗,X,y)=N (f∗|μ∗,Σ∗) (15.14)
μ∗=KT
∗K−1
yy (15.15)
Σ∗=K∗∗−KT∗K−1
yK∗ (15.16)
In the case of a single test input, this simpliﬁes as follows
p(f∗|x∗,X,y)=N (f∗|kT∗K−1
yy,k∗∗−kT∗K−1
yk∗) (15.17)
wherek∗=[κ(x∗,x1),...,κ(x∗,xN)]andk∗∗=κ(x∗,x∗). Another way to write the
posterior mean is as follows:
f∗=kT∗K−1
yy=N∑
i=1αiκ(xi,x∗) (15.18)
whereα=K−1
yy. We will revisit this expression later.

15.2. GPs for regression 519
−8 −6 −4 −2 0 2 4 6 8−3−2−10123
(a)−8 −6 −4 −2 0 2 4 6 8−3−2−10123
(b)
(c)
Figure 15.3 Some 1d GPs with SE kernels but different hyper-parameters ﬁt to 20 noisy observations. The
kernel has the form in Equation 15.19. The hyper-parameters (ℓ,σf,σy)are as follows: (a) (1,1,0.1) (b) (0.3,
0.1.08, 0.00005), (c) (3.0, 1.16, 0.89). Based on Figure 2.5 of (Rasmussen and Williams 2006). Figure generated
bygprDemoChangeHparams , written by Carl Rasmussen.
15.2.3 Effect of the kernel parameters
The predictive performance of GPs depends exclusively on the suitability of the chosen kernel.
Suppose we choose the following squared-exponential (SE) kernel for the noisy observations
κy(xp,xq)=σ2
fexp(−1
2ℓ2(xp−xq)2)+σ2
yδpq (15.19)
Hereℓis the horizontal scale over which the function changes, σ2
fcontrols the vertical scale of
the function, and σ2
yis the noise variance. Figure 15.3 illustrates the effects of changing these

fexp(−1
parameters. We sampled 20 noisy data points from the SE kernel using (ℓ,σf,σy)=( 1,1,0.1),
and then made predictions various parameters, conditional on the data. In Figure 15.3(a), we use(ℓ,σ
f,σy)=( 1,1,0.1), and the result is a good ﬁt. In Figure 15.3(b), we reduce the length scale
toℓ=0.3(the other parameters were optimized by maximum (marginal) likelihood, a technique

we discuss below); now the function looks more “wiggly”. Also, the uncertainty goes up faster,since the effective distance from the training points increases more rapidly. In Figure 15.3(c), weincrease the length scale to ℓ=3; now the function looks smoother.

520 Chapter 15. Gaussian processes
−202
−202−2−1012
input x1 input x2output y
(a)−202
(b)
−202
(c)
Figure 15.4 Some 2d functions sampled from a GP with an SE kernel but different hyper-parameters. The
kernel has the form in Equation 15.20 where (a) M=I, (b)M= diag(1 ,3)−2, (c)M=( 1,−1;−1,1)+
diag(6,6)−2. Based on Figure 5.1 of (Rasmussen and Williams 2006). Figure generated by gprDemoArd ,
written by Carl Rasmussen.
We can extend the SE kernel to multiple dimensions as follows:
fexp(−1
2(xp−xq)TM(xp−xq))+σ2
yδpq (15.20)
We can deﬁne the matrix Min several ways. The simplest is to use an isotropic matrix,
M1=ℓ−2I. See Figure 15.4(a) for an example. We can also endow each dimension with its
own characteristic length scale, M2=d i a g (ℓ)−2. If any of these length scales become large,
the corresponding feature dimension is deemed “irrelevant”, just as in ARD (Section 13.7). In

fexp(−1
Figure 15.4(b), we use M=M2withℓ=( 1,3), so the function changes faster along the x1
direction than the x2direction.
We can also create a matrix of the form M3=ΛΛT+diag(ℓ)−2,w h e r eΛis aD×Kmatrix,
whereK<D. (Rasmussen and Williams 2006, p107) calls this the factor analysis distance
function, by analogy to the fact that factor analysis (Section 12.1) approximates a covariancematrix as a low rank matrix plus a diagonal matrix. The columns of Λcorrespond to relevant

directions in input space. In Figure 15.4(c), we use ℓ= (6;6)andΛ=( 1 ;−1), so the function
changes mostly rapidly in the direction which is perpendicular to (1,1).

15.2. GPs for regression 521
15.2.4 Estimating the kernel parameters
To estimate the kernel parameters, we could use exhaustive search over a discrete grid of values,
with validation loss as an objective, but this can be quite slow. (This is the approach used totune kernels used by SVMs.) Here we consider an empirical Bayes approach, which will allow usto use continuous optimization methods, which are much faster. In particular, we will maximizethe marginal likelihood
1
p(y|X)=∫
p(y|f,X)p(f|X)df (15.21)
Sincep(f|X)=N(f|0,K), andp(y|f)=∏
iN(yi|fi,σ2
y), the marginal likelihood is given by
logp(y|X)=l o g N(y|0,Ky)=−1
2yK−1
yy−1
2log|Ky|−N
2log(2π) (15.22)
The ﬁrst term is a data ﬁt term, the second term is a model complexity term, and the third termis just a constant. To understand the tradeoff between the ﬁrst two terms, consider a SE kernelin 1D, as we vary the length scale ℓand hold σ
2
yﬁxed. Let J(ℓ)=−logp(y|X,ℓ). For short
length scales, the ﬁt will be good, so yTK−1

p(y|X)=∫
2yK−1
yy−1
2
yywill be small. However, the model complexity
will be high: Kwill be almost diagonal (as in Figure 14.3, top right), since most points will not
be considered “near” any others, so the log|Ky|will be large. For long length scales, the ﬁt will
be poor but the model complexity will be low: Kwill be almost all 1’s (as in Figure 14.3, bottom
right), so log|Ky|will be small.
We now discuss how to maximize the marginal likelhiood. Let the kernel parameters (also

called hyper-parameters) be denoted by θ. One can show that
∂
∂θjlogp(y|X)=1
2yTK−1
y∂Ky
∂θjK−1
yy−1
2tr(K−1
y∂Ky
∂θj) (15.23)
=1
2tr(
(ααT−K−1
y)∂Ky
∂θj)
(15.24)
yy. It takes O(N3)time to compute K−1
y, and then O(N2)time per hyper-
parameter to compute the gradient.
The form of∂Ky
∂θjdepends on the form of the kernel, and which parameter we are taking
derivatives with respect to. Often we have constraints on the hyper-parameters, such as σ2
y≥0.
In this case, we can deﬁne θ= log(σ2

∂
2yTK−1
y∂Ky
∂θjK−1
yy−1
2tr(K−1
y∂Ky
=1
2tr(
(ααT−K−1
y)∂Ky
∂θj)
(15.24)
y≥0.
y), and then use the chain rule.
Given an expression for the log marginal likelihood and its derivative, we can estimate the
kernel parameters using any standard gradient-based optimizer. However, since the objective isnot convex, local minima can be a problem, as we illustrate below.
15.2.4.1 Example
ConsiderFigure15.5. WeusetheSEkernelinEquation15.19with σ
2
f=1, andplot logp(y|X,ℓ,σ2
y)
(whereXandyare the 7 data points shown in panels b and c) as we vary ℓandσ2
y. The two

2
y)
y. The two
1. The reason it is called the marginal likelihood, rather than just likelihood, is because we have marginalized out the
latent Gaussian vector f. This moves us up one level of the Bayesian hierarchy, and reduces the chances of overﬁtting
(the number of kernel parameters is usually fairly small compared to a standard parametric model).

522 Chapter 15. Gaussian processes
10010110−1100
characteristic len gthscalenoise standard deviation
(a)−5 0 5−2−1012
input, xoutput, y
(b)
−5 0 5−2−1012
(c)
Figure 15.5 Illustration of local minima in the marginal likelihood surface. (a) We plot the log marginal
likelihood vs σ2
yandℓ, for ﬁxed σ2
f=1, using the 7 data points shown in panels b and c. (b) The function
corresponding to the lower left local minimum, (ℓ,σ2
n)≈(1,0.2). This is quite “wiggly” and has low
noise. (c) The function corresponding to the top right local minimum, (ℓ,σ2
n)≈(10,0.8). This is quite
smooth and has high noise. The data was generated using (ℓ,σ2
n)=( 1,0.1). Source: Figure 5.5 of
(Rasmussen and Williams 2006). Figure generated by gprDemoMarglik , written by Carl Rasmussen.
local optima are indicated by +. The bottom left optimum corresponds to a low-noise, short-

length scale solution (shown in panel b). The top right optimum corresponds to a high-noise,long-length scale solution (shown in panel c). With only 7 data points, there is not enoughevidence to conﬁdently decide which is more reasonable, although the more complex model(panel b) has a marginal likelihood that is about 60% higher than the simpler model (panel c).With more data, the MAP estimate should come to dominate.
Figure 15.5 illustrates some other interesting (and typical) features. The region where σ
2
y≈1

2
y≈1
(top of panel a) corresponds to the case where the noise is very high; in this regime, the marginallikelihood is insensitive to the length scale (indicated by the horizontal contours), since all thedata is explained as noise. The region where ℓ≈0.5(left hand side of panel a) corresponds to
the case where the length scale is very short; in this regime, the marginal likelihood is insensitiveto the noise level, since the data is perfectly interpolated. Neither of these regions would bechosen by a good optimizer.

15.2. GPs for regression 523
log(length−scale)log(magn itude)
z1z2
2.8 3 3.2 3.4−3−2.5−2−1.5−1−0.5
(a)log(length−scale)log(magn itude)
(b)
(c)
Figure 15.6 Three different approximations to the posterior over hyper-parameters: grid-based, Monte
Carlo, and central composite design. Source: Figure 3.2 of (Vanhatalo 2010). Used with kind permission
of Jarno Vanhatalo.
15.2.4.2 Bayesian inference for the hyper-parameters
An alternative to computing a point estimate of the hyper-parameters is to compute their poste-
rior. Letθrepresent all the kernel parameters, as well as σ2
y. If the dimensionality of θis small,
we can compute a discrete grid of possible values, centered on the MAP estimate ˆθ(computed
as above). We can then approximate the posterior over the latent variables using
p(f|D)∝S∑
s=1p(f|D,θs)p(θs|D)δs (15.25)
whereδsdenotes the weight for grid point s.

p(f|D)∝S∑
In higher dimensions, a regular grid suffers from the curse of dimensionality. An obvious
alternative is Monte Carlo, but this can be slow. Another approach is to use a form of quasi-Monte Carlo, whereby we place grid points at the mode, and at a distance ±1sd from the mode
along each dimension, for a total of 2|θ|+1points. This is called a central composite design

p(f|D)∝S∑
(Rue et al. 2009). (This is also used in the unscented Kalman ﬁlter, see Section 18.5.2.) To makethis Gaussian-like approximation more reasonable, we often log-transform the hyper-parameters.See Figure 15.6 for an illustration.

524 Chapter 15. Gaussian processes
15.2.4.3 Multiple kernel learning
A quite different approach to optimizing kernel parameters known as multiple kernel learning.
The idea is to deﬁne the kernel as a weighted sum of base kernels, κ(x,x′)=∑
jwjκj(x,x′),
and then to optimize the weights wjinstead of the kernel parameters themselves. This is
particularly useful if we have different kinds of data which we wish to fuse together. See
e.g., (Rakotomamonjy et al. 2008) for an approach based on risk-minimization and convexoptimization, and (Girolami and Rogers 2005) for an approach based on variational Bayes.
15.2.5 Computational and numerical issues *
The predictive mean is given by f∗=kT
∗K−1
yy. For reasons of numerical stability, it is unwise
to directly invert Ky. A more robust alternative is to compute a Cholesky decomposition,
Ky=LLT. We can then compute the predictive mean and variance, and the log marginal
likelihood, as shown in the pseudo-code in Algorithm 6 (based on (Rasmussen and Williams

∗K−1
2006, p19)). It takes O(N3)time to compute the Cholesky decomposition, and O(N2)time to
solve forα=K−1
yy=L−TL−1y. We can then compute the mean using kT
∗αinO(N)time
and the variance using k∗∗−kT∗L−TL−1k∗inO(N2)time for each test case.
An alternative to Cholesky decomposition is to solve the linear system Kyα=yusing
conjugate gradients (CG). If we terminate this algorithm after kiterations, it takes O(kN2)time.
If we run for k=N, it gives the exact solution in O(N3)time. Another approach is to
approximate the matrix-vector multiplies needed by CG using the fast Gauss transform. (Yang

et al. 2005); however, this doesn’t scale to high-dimensional inputs. See also Section 15.6 for adiscussion of other speedup techniques.
Algorithm 15.1: GP regression
1L=cholesky( K+σ2
yI);
2α=LT\(L\y);
3E[f∗]=kT
∗α;
4v=L\k∗;
5var[f∗]=κ(x∗,x∗)−vTv;
6logp(y|X)=−1
2yTα−∑
ilogLii−N
2log(2π)
15.2.6 Semi-parametric GPs *
Sometimes it is useful to use a linear model for the mean of the process, as follows:
f(x)=βTφ(x)+r(x) (15.26)
wherer(x)∼GP(0,κ(x,x′))models the residuals. This combines a parametric and a non-

yI);
3E[f∗]=kT
∗α;
4v=L\k∗;
2yTα−∑
ilogLii−N
2log(2π)
parametric model, and is known as a semi-parametric model.
If we assume β∼N(b,B), we can integrate these parameters out to get a new GP (O’Hagan
1978):
f(x)∼GP(
φ(x)Tb,κ(x,x′)+φ(x)TBφ(x′))
(15.27)

15.3. GPs meet GLMs 525
logp(yi|fi)∂
∂filogp(yi|fi)∂2
∂f2
ilogp(yi|fi)
logsigm(y ifi)ti−πi −πi(1−πi)
logΦ(y ifi)yiφ(fi)
Φ(yifi)−φ2
i
Φ(yifi)2−yifiφ(fi)
Φ(yifi)
Table 15.1 Likelihood, gradient and Hessian for binary logistic/ probit GP regression. We assume yi∈
{−1,+1}and deﬁne ti=(yi+1)/2∈{0,1}andπi= sigm(f i)for logistic regression, and πi=Φ (fi)
for probit regression. Also, φandΦare the pdf and cdf of N(0,1). From (Rasmussen and Williams 2006,
p43).
Integrating out β, the corresponding predictive distribution for test inputs X∗has the following
form (Rasmussen and Williams 2006, p28):
p(f∗|X∗,X,y)=N (f∗,cov[f∗]) (15.28)
f∗=ΦT
∗β+KT
∗K−1
y(y−Φβ) (15.29)
β=(ΦTK−1
yΦ+B−1)−1(ΦK−1
yy+B−1b) (15.30)
cov[f∗]=K ∗∗−KT∗K−1
yK∗+RT(B−1+ΦK−1
yΦT)−1R (15.31)
R=Φ∗−ΦK−1
yΦ∗ (15.32)
The predictive mean is the output of the linear model plus a correction term due to the GP, and
the predictive covariance is the usual GP covariance plus an extra term due to the uncertaintyinβ.
15.3 GPs meet GLMs

f∗=ΦT
∗β+KT
∗K−1
β=(ΦTK−1
R=Φ∗−ΦK−1
In this section, we extend GPs to the GLM setting, focussing on the classiﬁcation case. As withBayesian logistic regression, the main difficulty is that the Gaussian prior is not conjugate tothe bernoulli/ multinoulli likelihood. There are several approximations one can adopt: Gaussianapproximation (Section 8.4.3), expectation propagation (Kuss and Rasmussen 2005; Nickisch andRasmussen 2008), variational (Girolami and Rogers 2006; Opper and Archambeau 2009), MCMC(Neal 1997; Christensen et al. 2006), etc. Here we focus on the Gaussian approximation, since itis the simplest and fastest.
15.3.1 Binary classiﬁcation

In the binary case, we deﬁne the model as p(yi|xi)=σ(yif(xi)), where, following (Rasmussen
and Williams 2006), we assume yi∈{ −1,+1}, and we let σ(z) = sigm(z )(logistic regression)
orσ(z)=Φ (z)(probit regression). As for GP regression, we assume f∼GP(0,κ).
15.3.1.1 Computing the posterior
Deﬁne the log of the unnormalized posterior as follows:
ℓ(f)=l o gp(y|f)+logp(f|X)=l o g p(y|f)−1
2fTK−1f−1
2log|K|−N
2log2π(15.33)

526 Chapter 15. Gaussian processes
LetJ(f)≜−ℓ(f)be the function we want to minimize. The gradient and Hessian of this are
given by
g=−∇logp(y|f)+K−1f (15.34)
H=−∇∇logp(y|f)+K−1=W+K−1(15.35)
Note that W≜−∇∇logp(y|f)is a diagonal matrix because the data are iid (conditional on
f). Expressions for the gradient and Hessian of the log likelihood for the logit and probit case
are given in Sections 8.3.1 and 9.4.1, and summarized in Table 15.1.
We can use IRLS to ﬁnd the MAP estimate. The update has the form
fnew=f−H−1g=f+(K−1+W)−1(∇logp(y|f)−K−1f) (15.36)
=(K−1+W)−1(Wf+∇logp(y|f)) (15.37)
At convergence, the Gaussian approximation of the posterior takes the following form:
p(f|X,y)≈N(ˆf,(K−1+W)−1) (15.38)
15.3.1.2 Computing the posterior predictive
We now compute the posterior predictive. First we predict the latent function at the test case
x∗. For the mean we have
E[f∗|x∗,X,y]=∫
E[f∗|f,x∗,X,y]p(f|X,y)df (15.39)
=∫
kT
∗K−1fp(f|X,y)df (15.40)
=kT∗K−1E[f|X,y]≈kT∗K−1ˆf (15.41)

=∫
kT
where we used Equation 15.8 to get the mean of f∗given noise-free f.
To compute the predictive variance, we use the rule of iterated variance:
var[f∗]=E [var[f∗|f]]+v ar[ E[f∗|f]] (15.42)
where all probabilities are conditioned on x∗,X,y. From Equation 15.9 we have
E[var[f∗|f]] =E[
k∗∗−kT∗K−1k∗]
=k∗∗−kT∗K−1k∗ (15.43)
From Equation 15.9 we have
var[E[f∗|f] ]=v a r[
k∗K−1f]
=kT∗K−1cov[f]K−1k∗ (15.44)
Combining these we get
var[f∗]=k ∗∗−kT∗(K−1−K−1cov[f]K−1)k∗ (15.45)

k∗K−1f]
From Equation 15.38 we have cov[f]≈(K−1+W)−1. Using the matrix inversion lemma we
get
var[f∗]≈k∗∗−kT∗K−1k∗+kT∗K−1(K−1+W)−1K−1k∗ (15.46)
=k∗∗−kT∗(K+W−1)−1k∗ (15.47)

15.3. GPs meet GLMs 527
So in summary we have
p(f∗|x∗,X,y)=N(E[f∗],var[f∗]) (15.48)
To convert this in to a predictive distribution for binary responses, we use
π∗=p(y∗=1|x∗,X,y)≈∫
σ(f∗)p(f∗|x∗,X,y)df∗ (15.49)
This can be approximated using any of the methods discussed in Section 8.4.4, where we
discussed Bayesian logistic regression. For example, using the probit approximation of Sec-tion 8.4.4.2, we have π
∗≈sigm(κ(v)E[f∗]),w h e r ev = var[f ∗]andκ2(v)=( 1+ πv/8)−1.
15.3.1.3 Computing the marginal likelihood
We need the marginal likelihood in order to optimize the kernel parameters. Using the Laplaceapproximation in Equation 8.54 we have
logp(y|X)≈ℓ(ˆf)−1
2log|H|+const (15.50)
Hence
logp(y|X)≈logp(y|ˆf)−1
2ˆfTK−1ˆf−1
2log|K|−1
2log|K−1+W| (15.51)
Computing the derivatives∂logp(y|X,θ)
∂θjis more complex than in the regression case, since ˆf
andW, as well as K, depend on θ. Details can be found in (Rasmussen and Williams 2006,
p125).
15.3.1.4 Numerically stable computation *

Hence
2log|K|−1
p125).
To implement the above equations in a numerically stable way, it is best to avoid inverting K
orW. (Rasmussen and Williams 2006, p45) suggest deﬁning
B=IN+W1
2KW1
2 (15.52)
which has eigenvalues bounded below by 1 (because of the I) and above by 1+N
4maxijKij
(becausewii=πi(1−π)≤0.25), and hence can be safely inverted.
One can use the matrix inversion lemma to show
(K−1+W)−1=K−KW1
2B−1W1
2K (15.53)
Hence the IRLS update becomes
fnew=(K−1+W)−1(Wf+∇logp(y|f))  
b(15.54)
=K(I−W1
2B−1W1
2K)b (15.55)
=K(b−W1
2LT\(L\(W1
2Kb)))  
a(15.56)

528 Chapter 15. Gaussian processes
whereB=LLTis a Cholesky decomposition of B. The ﬁtting algorithm takes in O(TN3)
time and O(N2)space, where Tis the number of Newton iterations.
At convergence we have a=K−1ˆf, so we can evaluate the log marginal likelihood (Equa-
tion 15.51) using
logp(y|X)=l o gp(y|ˆf)−1
2aTˆf−∑
ilogLii (15.57)
where we exploited the fact that
|B|=|K||K−1+W|=|IN+W1
2KW1
2| (15.58)
We now compute the predictive distribution. Rather than using E[f∗]=kT
∗K−1ˆf, we exploit
the fact that at the mode, ∇ℓ=0,s oˆf=K(∇logp(y|ˆf)). Hence we can rewrite the predictive
mean as follows:2
E[f∗]=kT
∗∇logp(y|ˆf) (15.59)
To compute the predictive variance, we exploit the fact that
(K+W−1)−1=W1
2W−1
2(K+W−1)−1W−1
2W1
2=W1
2B−1W1
2 (15.60)
to get
var[f∗]=k∗∗−kT∗W1
2(LLT)−1W1
2k∗=k∗∗−vTv (15.61)
wherev=L\(W1
2k∗). We can then compute π∗.
The whole algorithm is summarized in Algorithm 16, based on (Rasmussen and Williams 2006,

E[f∗]=kT
2W−1
2W1
2=W1
2B−1W1
2 (15.60)
to get
p46). Fitting takes O(N3)time, and prediction takes O(N2N∗)time, where N∗is the number
of test cases.
15.3.1.5 Example
In Figure 15.7, we show a synthetic binary classiﬁcation problem in 2d. We use an SE kernel. On
the left, we show predictions using hyper-parameters set by hand; we use a short length scale,hence the very sharp turns in the decision boundary. On the right, we show the predictionsusing the learned hyper-parameters; the model favors a more parsimonious explanation of thedata.
15.3.2 Multi-class classiﬁcation

In this section, we consider a model of the form p(yi|xi)=C a t ( yi|S(fi)),w h e r e fi=
(fi1,...,f iC), and we assume f.c∼GP(0,κc). Thus we have one latent function per class,
which are a priori independent, and which may use different kernels. As before, we will usea Gaussian approximation to the posterior. (A similar model, but using the multinomial probitfunction instead of the multinomial logit, is described in (Girolami and Rogers 2006).)
2. We see that training points that are well-predicted by the model, for which ∇ilogp(yi|fi)≈0, do not contribute

strongly to the prediction at test points; this is similar to the behavior of support vectors in an SVM (see Section 14.5).

15.3. GPs meet GLMs 529
Algorithm 15.2: GP binary classiﬁcation using Gaussian approximation
1// First compute MAP estimate using IRLS;
2f=0;
3repeat
4W=−∇∇logp(y|f);
5B=IN+W1
2KW1
2;
6L=cholesky(B);
7b=Wf+∇logp(y|f);
8a=b−W1
2Kb));
9f=Ka;
11logp(y|X)=l o gp(y|f)−1
2aTf−∑
ilogLii;
12// Now perform prediction ;
13E[f∗]=kT
∗∇logp(y|f);
14v=L\(W1
2k∗);
15var[f∗]=k∗∗−vTv;
16p(y∗=1 )=∫
sigm(z)N(z|E[f∗],var[f∗])dz;
SE kernel, l=0.500, σ2=10.000
−4 −3 −2 −1 0 1 2 3 4−4−3−2−101234
0.10.20.30.40.50.60.70.80.9
(a)SE kernel, l=1.280, σ2=14.455
(b)
Figure 15.7 Contours of the posterior predictive probability for the red circle class generated by a GP with
an SE kernel. Thick black line is the decision boundary if we threshold at a probability of 0.5. (a) Manual
parameters, short length scale. (b) Learned parameters, long length scale. Figure generated by gpcDemo2d ,
based on code by Carl Rasmussen.

530 Chapter 15. Gaussian processes
15.3.2.1 Computing the posterior
The unnormalized log posterior is given by
ℓ(f)=−1
2fTK−1f+yTf−N∑
i=1log(C∑
c=1expfic)
−1
2log|K|−CN
2log2π(15.62)
where
f=(f11,...,f N1,f12,...,f N2,···,f1C,...,f NC)T(15.63)
andyis a dummy encoding of the yi’s which has the same layout as f. Also,Kis a block
diagonal matrix containing Kc,w h e r eKc=[κc(xi,xj)]models the correlation of the c’th
latent function.
The gradient and Hessian are given by
∇ℓ=−K−1f+y−π (15.64)
∇∇ℓ=−K−1−W (15.65)
whereW≜diag(π)−ΠΠT,w h e r eΠis aCN×Nmatrix obtained by stacking diag(π :c)
vertically. (Compare these expressions to standard logistic regression in Section 8.3.7.)
We can use IRLS to compute the mode. The Newton step has the form
fnew=(K−1+W)−1(Wf+y−π) (15.66)
Naively implementing this would take O(C3N3)time. However, we can reduce this to O(CN3),
as shown in (Rasmussen and Williams 2006, p52).
15.3.2.2 Computing the posterior predictive

We can compute the posterior predictive in a manner analogous to Section 15.3.1.2. For the
mean of the latent response we have
E[f∗c]=kc(x∗)TK−1
cˆfc=kc(x∗)T(yc−ˆπc) (15.67)
We can put this in vector form by writing
E[f∗]=Q∗T(y−ˆπ) (15.68)
where
Q∗=⎛
⎜⎝k1(x∗)...0
...
0...kC(x∗)⎞
⎟⎠ (15.69)
Using a similar argument to Equation 15.47, we can show that the covariance of the latent
response is given by
cov[f∗]=Σ +QT
∗K−1(K−1+W)−1K−1Q∗ (15.70)
=d i a g ( k(x∗,x∗))−QT∗(K+W−1)−1Q∗ (15.71)

15.3. GPs meet GLMs 531
whereΣis aC×Cdiagonal matrix with Σcc=κc(x∗,x∗)−kT
c(x∗)K−1
ckc(x∗), and
k(x∗,x∗)=[κc(x∗,x∗)].
To compute the posterior predictive for the visible response, we need to use
p(y|x∗,X,y)≈∫
Cat(y|S(f∗))N(f∗|E[f∗],cov[f∗])df∗ (15.72)
We can use any of deterministic approximations to the softmax function discussed in Sec-
tion 21.8.1.1 to compute this. Alternatively, we can just use Monte Carlo.
15.3.2.3 Computing the marginal likelihood
Using arguments similar to the binary case, we can show that
logp(y|X)≈−1
2ˆfTK−1ˆf+yTˆf−N∑
i=1log(C∑
c=1expˆfic)
−1
2log|ICN+W1
2KW1
2|(15.73)
This can be optimized numerically in the usual way.
15.3.2.4 Numerical and computational issues
One can implement model ﬁtting in O(TCN3)time and O(CN2)space, where Tis the
number of Newton iterations, using the techniques described in (Rasmussen and Williams 2006,p50). Prediction takes O(CN
3+CN2N∗)time, where N∗is the number of test cases.
15.3.3 GPs for Poisson regression

i=1log(C∑
−1
2KW1
2|(15.73)
In this section, we illustrate GPs for Poisson regression. An interesting application of this is tospatialdisease mapping. For example, (Vanhatalo et al. 2010) discuss the problem of modeling
the relative risk of heart attack in different regions in Finland. The data consists of the heartattacks in Finland from 1996-2000 aggregated into 20km x 20km lattice cells. The model hasthe following form:
y
i∼Poi(eiri) (15.74)
whereeiis the known expected number of deaths (related to the population of cell iand the

y
overall death rate), and riis therelative risk of celliwhich we want to infer. Since the
data counts are small, we regularize the problem by sharing information with spatial neighbors.Hence we assume f≜log(r)∼GP(0,κ), where we use a Matern kernel with ν=3/2, and a
length scale and magnitude that are estimated from data.
Figure 15.8 gives an example of the kind of output one can obtain from this method, based

on data from 911 locations. On the left we plot the posterior mean relative risk (RR), and on theright, the posterior variance. We see that the RR is higher in Eastern Finland, which is consistentwith other studies. We also see that the variance in the North is higher, since there are fewerpeople living there.

532 Chapter 15. Gaussian processes
0 10 20 300102030405060Posterior mean of the relative risk, FIC
  
0.70.80.911.11.21.31.4
(a)0 10 20 300102030405060Posterior variance of the relative risk, FIC
  
0.0050.010.0150.020.0250.030.035
(b)
Figure 15.8 We show the relative risk of heart disease in Finland using a Poisson GP. Left: posterior mean.
Right: posterior variance. Figure generated by gpSpatialDemoLaplace , written by Jarno Vanhatalo.
15.4 Connection with other methods
There are variety of other methods in statistics and machine learning that are closely related to
GP regression/ classiﬁcation. We give a brief review of some of these below.
15.4.1 Linear models compared to GPs
Consider Bayesian linear regression for D-dimensional features, where the prior on the weights
isp(w)=N(0,Σ). The posterior predictive distribution is given by the following;
p(f∗|x∗,X,y)=N(μ,σ2) (15.75)
μ=1
σ2yxT
∗A−1XTy (15.76)
σ2=xT
∗A−1x∗ (15.77)
whereA=σ−2
yXTX+Σ−1. One can show that we can rewrite the above distribution as

μ=1
σ2yxT
σ2=xT
follows
μ=xT
∗ΣXT(K+σ2
yI)−1y (15.78)
σ2=xT
∗Σx∗−xT
∗ΣXT(K+σ2I)−1XΣx∗ (15.79)
where we have deﬁned K=XΣXT, which is of size N×N. Since the features only ever
appear in the form XΣXT,xT
∗ΣXTorxT
∗Σx∗, we can kernelize the above expression by
deﬁningκ(x,x′)=xTΣx′.
Thus we see that Bayesian linear regression is equivalent to a GP with covariance function
κ(x,x′)=xTΣx′. Note, however, that this is a degenerate covariance function, since it has at
mostDnon-zero eigenvalues. Intuitively this reﬂects the fact that the model can only represent

σ2=xT
∗Σx∗−xT
∗ΣXTorxT
a limited number of functions. This can result in underﬁtting, since the model is not ﬂexible
enough to capture the data. What is perhaps worse, it can result in overconﬁdence, since the

15.4. Connection with other methods 533
model’s prior is so impoverished that its posterior will become too concentrated. So not only is
the model wrong, it think it’s right!
15.4.2 Linear smoothers compared to GPs
Alinear smoother is a regression function which is a linear function of the training outputs:
ˆf(x∗)=∑
iwi(x∗)yi (15.80)
wherewi(x∗)is called the weight function (Silverman 1984). (Do not confuse this with a linear
model, where the output is a linear function of the input vector.)
There are a variety of linear smoothers, such as kernel regression (Section 14.7.4), locally
weighted regression (Section 14.7.5), smoothing splines (Section 15.4.6), and GP regression. To seethat GP regession is a linear smoother, note that the mean of the posterior predictive distributiono faG Pi sg i v e nb y
f(x∗)=kT
∗(K+σ2
yIN)−1y=N∑
i=1yiwi(x∗) (15.81)
wherewi(x∗)=[ (K+σ2
yIN)−1k∗]i.
In kernel regression, we derive the weight function from a smoothing kernel rather than a

f(x∗)=kT
∗(K+σ2
Mercer kernel, so it is clear that the weight function will then have local support. In the case
of a GP, things are not as clear, since the weight function depends on the inverse of K.F o r
certain GP kernel functions, we can analytically derive the form of wi(x); this is known as the
equivalent kernel (Silverman 1984). One can show that∑N
i=1wi(x∗)=1, although we may
havewi(x∗)<0, so we are computing a linear combination but not a convex combination of

theyi’s. More interestingly, wi(x∗)is a local function, even if the original kernel used by the GP
is not local. Futhermore the effective bandwidth of the equivalent kernel of a GP automatically
decreases as the sample size Nincreases, whereas in kernel smoothing, the bandwidth hneeds
to be set by hand to adapt to N. See e.g., (Rasmussen and Williams 2006, Sec 2.6,Sec 7.1) for
details.
15.4.2.1 Degrees of freedom of linear smoothers

details.
It is clear why this method is called “linear”, but why is it called a “smoother”? This is bestexplained in terms of GPs. Consider the prediction on the training set:
f=K(K+σ2
y)−1y (15.82)
Now letKhave the eigendecomposition K=∑N
i=1λiuiuT
i. SinceKis real and symmetric
positive deﬁnite, the eigenvalues λiare real and non-negative, and the eigenvectors uiare
orthonormal. Now let y=∑N
i=1γiui,w h e r eγi=uT
iy. Then we can rewrite the above
equation as follows:
f=N∑
i=1γiλi
λi+σ2yui (15.83)

534 Chapter 15. Gaussian processes
This is the same as Equation 7.47, except we are working with the eigenvectors of the Gram
matrixKinstead of the data matrix X. In any case, the interpretation is similar: ifλi
λi+σ2
y≪1,
then the corresponding basis function uiwill not have much inﬂuence. Consequently the high-
frequency components in yare smoothed out. The effective degrees of freedom of the linear
smoother is deﬁned as
dof≜tr(K(K+σ2
yI)−1)=N∑
i=1λi
λi+σ2y(15.84)
This speciﬁes how “wiggly” the curve is.
15.4.3 SVMs compared to GPs
We saw in Section 14.5.2 that the SVM objective for binary classiﬁcation is given by Equation 14.57
J(w)=1
i=1(1−yifi)+ (15.85)
We also know from Equation 14.59 that the optimal solution has the form w=∑
iαixi,
so||w||2=∑
i,jαiαjxT
ixj. Kernelizing we get ||w||2=αKα. From Equation 14.61, and
absorbing the ˆw0term into one of the kernels, we have f=Kα,s o||w||2=fTK−1f. Hence
the SVM objective can be rewritten as
J(f)=1
2fTf+CN∑
i=1(1−yifi)+ (15.86)

J(w)=1
iαixi,
i,jαiαjxT
J(f)=1
2fTf+CN∑
Compare this to MAP estimation for GP classiﬁer:
J(f)=1
2fTf−N∑
i=1logp(yi|fi) (15.87)
It is tempting to think that we can “convert” an SVM into a GP by ﬁguring out what likelihood
would be equivalent to the hinge loss. However, it turns out there is no such likelihood (Sollich2002), although there is a pseudo-likelihood that matches the SVM (see Section 14.5.5).
From Figure 6.7 we saw that the hinge loss and the logistic loss (as well as the probit loss)

J(f)=1
2fTf+CN∑
J(f)=1
2fTf−N∑
are quite similar to each other. The main difference is that the hinge loss is strictly 0 for errorslarger than 1. This gives rise to a sparse solution. In Section 14.3.2, we discussed other waysto derive sparse kernel machines. We discuss the connection between these methods and GPsbelow.
15.4.4 L1VM and RVMs compared to GPs
Sparse kernel machines are just linear models with basis function expansion of the form φ(x)=
[κ(x,x1),...,κ(x,xN)]. From Section 15.4.1, we know that this is equivalent to a GP with the

following kernel:
κ(x,x′)=D∑
j=11
αjφj(x)φj(x′) (15.88)

15.4. Connection with other methods 535
wherep(w)=N(0,diag(α−1
j)). This kernel function has two interesting properties. First, it
is degenerate, meaning it has at most Nnon-zero eigenvalues, so the joint distribution p(f,f∗)
will be highly constrained. Second, the kernel depends on the training data. This can cause the
model to be overconﬁdent when extrapolating beyond the training data. To see this, considera pointx
∗far outside the convex hull of the data. All the basis functions will have values
close to 0, so the prediction will back off to the mean of the GP. More worryingly, the variancewill back off to the noise variance. By contrast, when using a non-degenerate kernel function,the predictive variance increases as we move away from the training data, as desired. See(Rasmussen and Quiñonero-Candela 2005) for further discussion.
15.4.5 Neural networks compared to GPs

In Section 16.5, we will discuss neural networks, which are a nonlinear generalization of GLMs.In the binary classiﬁcation case, a neural network is deﬁned by a logistic regression modelapplied to a logistic regression model:
p(y|x,θ)=B e r(
y|sigm(
w
Tsigm(Vx)))
(15.89)
It turns out there is an interesting connection between neural networks and Gaussian processes,as ﬁrst pointed out by (Neal 1996).
To explain the connection, we follow the presentation of (Rasmussen and Williams 2006, p91).

y|sigm(
w
(15.89)
Consider a neural network for regression with one hidden layer. This has the form
p(y|x,θ)=N(y|f(x;θ),σ
2) (15.90)
where
f(x)=b+H∑
j=1vjg(x;uj) (15.91)
wherebis the offset of bias term, vjis the output weight from hidden unit jto the response
y,ujare the inputs weights to unit jfrom the input x, andg()is the hidden unit activation
function. This is typically the sigmoid or tanh function, but can be any smooth function.
Let us use the following priors on the weights: where b∼N(0,σ2
b)v∼∏
jN(vj|0,σ2
w),
u∼∏

2) (15.90)
where
f(x)=b+H∑
b)v∼∏
w),
u∼∏
jp(uj)for some unspeciﬁed p(uj). Denoting all the weights by θwe have
Eθ[f(x) ]=0 (15.92)
Eθ[f(x)f(x′)] =σ2
b+∑
jσ2
vEv[g(x;uj)g(x′;uj)] (15.93)
=σ2
b+Hσ2
vEu[g(x;u)g(x′;u)] (15.94)
where the last equality follows since the Hhidden units are iid. If we let σ2
vscale asω2/H
(since more hidden units will increase the input to the ﬁnal node, so we should scale downthe magnitude of the weights), then the last term becomes ω
2Eu[g(x;u)g(x′;u)]. This is a

b)v∼∏
w),
u∼∏
b+∑
jσ2
=σ2
b+Hσ2
sum over Hiid random variables. Assuming that gis bounded, we can apply the central limit
theorem. The result is that as H→∞, we get a Gaussian process.

536 Chapter 15. Gaussian processes
−0.5
−0.50
00.50.5
0.950.95
input, xinput, xʼ
−4 0 4−404
(a)−4 0 4−101
input, xoutput, f(x)σ = 10
σ = 3
σ = 1
(b)
Figure 15.9 (a) Covariance function κNN(x,x′)forσ0=1 0,σ=1 0. (b) Samples from from a GP with
this kernel, using various values of σ. Figure generated by gpnnDemo , written by Chris Williams.
If we use as activation / transfer function g(x;u)=erf(u0+∑D
j=1ujxj), where erf( z)=
2/√π∫z
0e−t2dt, and we choose u∼N(0,Σ), then (Williams 1998) showed that the covariance
kernel has the form
κNN(x,x′)=2
πsin−1(
2˜xTΣ˜x′
√
(1+2˜xTΣ˜x)(1+2(˜x′)TΣ˜x′))
(15.95)
where˜x=( 1,x1,...,x D). This is a true “neural network” kernel, unlike the “sigmoid” kernel
κ(x,x′)=t a n h ( a+bxTx′), which is not positive deﬁnite.
Figure 15.9(a) illustrates this kernel when D=2andΣ= diag(σ2
0,σ2). Figure 15.9(b) shows
some functions sampled from the corresponding GP. These are equivalent to functions which
are superpositions of erf( u0+ux)whereu0anduare random. As σ2increases, the variance

πsin−1(
2˜xTΣ˜x′
√
(15.95)
ofuincreases, so the function varies more quickly. Unlike the RBF kernel, functions sampled
from this kernel do not tend to 0 away from the data, but rather they tend to remain at thesame value they had at the “edge” of the data.
Now suppose we use an RBF network, which is equivalent to a hidden unit activation function
of the form g(x;u) = exp( −|x−u|
2/(2σ2
g)).I fu∼N(0,σ2
uI), one can show that the
coresponding kernel is equivalent to the RBF or SE kernel.
15.4.6 Smoothing splines compared to GPs *

2/(2σ2
Smoothing splines are a widely used non-parametric method for smoothly interpolating data
(Green and Silverman 1994). They are are a special case of GPs, as we will see. They are usuallyused when the input is 1 or 2 dimensional.
15.4.6.1 Univariate splines
The basic idea is to ﬁt a function fby minimizing the discrepancy to the data plus a smoothing
term that penalizes functions that are “too wiggly”. If we penalize the m’th derivative of the

15.4. Connection with other methods 537
function, the objective becomes
J(f)=N∑
i=1(f(xi)−yi)2+λ∫
(dm
dxmf(x))2dx (15.96)
One can show (Green and Silverman 1994) that the solution is a piecewise polynomial where
the polynomials have order 2m−1in the interior bins [xi−1,xi](denotedI), and order m−1
in the two outermost intervals (−∞,x1]and[xN,∞):
f(x)=m−1∑
j=0βjxj+I(x∈I)(N∑
i=1αi(x−xi)2m−1
+)
+I(x̸∈I)(N∑
i=1αi(x−xi)m−1
+)
(15.97)
For example, if m=2, we get the (natural) cubic spline
f(x)=β 0+β1x+I(x∈I)(N∑
i=1αi(x−xi)3
+)
i=1αi(x−xi)+)
(15.98)
which is a series of truncated cubic polynomials, whose left hand sides are located at each of the
Ntraining points. (The fact that the model is linear on the edges prevents it from extrapolating
too wildly beyond the range of the data; if we drop this requirement, we get an “unrestricted”spline.)
We can clearly ﬁt this model using ridge regression: ˆw=(Φ
TΦ+λIN)−1ΦTy, where the
columns of Φare 1,xiand(x−xi)3
+fori=2:N−1and(x−xi)+fori=1ori=N.

+)
(15.98)
However, we can also derive an O(N)time method (Green and Silverman 1994, Sec 2.3.3).
15.4.6.2 Regression splines
In general, we can place the polynomials at a ﬁxed set of Klocations known as knots, denoted
ξk. The result is called a regression spline. This is a parametric model, which uses basis
function expansion of the following form (where we drop the interior/ exterior distinction for
simplicity):
f(x)=β 0+β1x+K∑
k=1αj(x−ξk)3
+ (15.99)

+ (15.99)
Choosing the number and locations of the knots is just like choosing the number and values of
the support vectors in Section 14.3.2. If we impose an ℓ2regularizer on the regression coefficients
αj, the method is known as penalized splines. See Section 9.6.1 for a practical example of
penalized splines.
15.4.6.3 The connection with GPs
One can show (Rasmussen and Williams 2006, p139) that the cubic spline is the MAP estimateof the following function
f(x)=β
0+β1x+r(x) (15.100)

538 Chapter 15. Gaussian processes
wherep(βj)∝1(so that we don’t penalize the zero’th and ﬁrst derivatives of f), andr(x)∼
GP(0,σ2
fκsp(x,x′)),w h e r e
κsp(x,x′)≜∫1
0(x−u)+(x′−u)+du (15.101)
Note that the kernel in Equation 15.101 is rather unnatural, and indeed posterior samples from
the resulting GP are rather unsmooth. However, the posterior mode/mean is smooth. This shows
that regularizers don’t always make good priors.
15.4.6.4 2d input (thin-plate splines)
One can generalize cubic splines to 2d input by deﬁning a regularizer of the following form:
∫∫[(∂2f(x)
∂x2
1)2
+2(∂2f(x)
∂x1∂x2)2
+(∂2f(x)
∂x22)2]
dx1dx2 (15.102)
f(x)=β 0+βT
1x+N∑
i=1αiφi(x) (15.103)
whereφi(x)=η(||x−xi||), andη(z)=z2logz2. This is known as a thin plate spline. This
is equivalent to MAP estimation with a GP whose kernel is deﬁned in (Williams and Fitzgibbon
2006).
15.4.6.5 Higher-dimensional inputs

∂x2
1)2
+2(∂2f(x)
∂x1∂x2)2
+(∂2f(x)
∂x22)2]
1x+N∑
2006).
It is hard to analytically solve for the form of the optimal solution when using higher-orderinputs. However, in the parametric regression spline setting, where we forego the regularizer onf, we have more freedom in deﬁning our basis functions. One way to handle multiple inputs is
to use atensor product basis, deﬁned as the cross product of 1d basis functions. For example,
for 2d input, we can deﬁne
f(x
1,x2)=β 0+∑
mβ1m(x1−ξ1m)++∑
mβ2m(x2−ξ2m)+ (15.104)
+∑
mβ12m(x1−ξ1m)+(x2−ξ2m)+ (15.105)

f(x
+∑
It is clear that for high-dimensional data, we cannot allow higher-order interactions, because
there will be too many parameters to ﬁt. One approach to this problem is to use a searchprocedure to look for useful interaction terms. This is known as MARS, which stands for“multivariate adaptive regression splines”. See Section 16.3.3 for details.
15.4.7 RKHS methods compared to GPs *

f(x
+∑
We can generalize the idea of penalizing derivatives of functions, as used in smoothing splines,to ﬁt functions with a more general notion of smoothness. Recall from Section 14.2.3 that

15.4. Connection with other methods 539
Mercer’s theorem says that any positive deﬁnite kernel function can be represented in terms of
eigenfunctions:
κ(x,x′)=∞∑
i=1λiφi(x)φi(x′) (15.106)
Theφiform an orthormal basis for a function space:
Hk={f:f(x)=∞∑
i=1fiφi(x),∞∑
i=1f2
i/λi<∞} (15.107)
Now deﬁne the inner product between two functions f(x)=∑∞
i=1fiφi(x)andg(x)=∑∞i=1giφi(x)in this space as follows:
⟨f,g⟩H≜∞∑
i=1figi
λi(15.108)
In Exercise 15.1, we show that this deﬁnition implies that
⟨κ(x1,·),κ(x2,·)⟩H=κ(x1,x2) (15.109)
This is called the reproducing property, and the space of functions Hkis called a reproducing
kernel Hilbert space orRKHS.
Now consider an optimization problem of the form
J(f)=1
2σ2yN∑
i=1(yi−f(xi))2+1
2||f||2
H (15.110)
where||f||Jis thenorm of a function:
||f||H=⟨f,f⟩H=∞∑
i=1f2
i
λi(15.111)
The intuition is that functions that are complex wrt the kernel will have large norms, because

i=1figi
J(f)=1
2σ2yN∑
2||f||2
H (15.110)
i=1f2
i
they will need many eigenfunctions to represent them. We want to pick a simple function thatprovides a good ﬁt to the data.
One can show (see e.g., (Schoelkopf and Smola 2002)) that the solution must have the form
f(x)=N∑
i=1αiκ(x,xi) (15.112)
This is known as the representer theorem, and holds for other convex loss functions besides
squared error.
We can solve for the αby substituting in f(x)=∑N
i=1αiκ(x,xi)and using the reproducing
property to get
J(α)=1
2σ2y|y−Kα|2+1
2αTKα (15.113)

540 Chapter 15. Gaussian processes
Minimizing wrt αwe ﬁnd
ˆα=(K+σ2
yI)−1(15.114)
and hence
ˆf(x∗)=∑
iˆαiκ(x∗,xi)=kT
∗(K+σ2
yI)−1y (15.115)
This is identical to Equation 15.18, the posterior mean of a GP predictive distribution. Indeed,
since the mean and mode of a Gaussian are the same, we can see that linear regresson with anRKHS regularizer is equivalent to MAP estimation with a GP. An analogous statement holds forthe GP logistic regression case, which also uses a convex likelihood / loss function.
15.5 GP latent variable model
In Section 14.4.4, we discussed kernel PCA, which applies the kernel trick to regular PCA. Inthis section, we discuss a different way to combine kernels with probabilistic PCA. The resultingmethod is known as the GP-LVM, which stands for “Gaussian process latent variable model”
(Lawrence 2005).
To explain the method, we start with PPCA. Recall from Section 12.2.4 that the PPCA model is
p(z
i)=N (zi|0,I) (15.116)
p(yi|zi,θ)=N (yi|Wz i,σ2I) (15.117)

p(z
We can ﬁt this model by maximum likelihood, by integrating out the ziand maximizing W
(andσ2). The objective is given by
p(Y|W,σ2)=( 2π)−DN/ 2|C|−N/2exp(
−1
2tr(C−1YTY))
(15.118)
where C=WWT+σ2I. As we showed in Theorem 12.2.2, the MLE for this can be computed
in terms of the eigenvectors of YTY.
Now we consider the dual problem, whereby we maximize Zand integrate out W. We will
use a prior of the form p(W)=∏
jN(wj|0,I). The corresponding likelihood becomes
p(Y|Z,σ2)=D∏
d=1N(y:,d|0,ZZT+σ2I) (15.119)
=( 2π)−DN/ 2|Kz|−D/2exp(

−1
(15.118)
−1
2tr(K−1
zYYT))
(15.120)
where Kz=ZZT+σ2I. Based on our discussion of the connection between the eigenvalues
ofYYTand of YTYin Section 14.4.4, it should come as no surprise that we can also solve
the dual problem using eigenvalue methods (see (Lawrence 2005) for the details).
If we use a linear kernel, we recover PCA. But we can also use a more general kernel:
Kz=K+σ2I,w h e r e Kis the Gram matrix for Z. The MLE for ˆZwill no longer be available

15.5. GP latent variable model 541
0 0.05 0.1 0.15 0.20.050.1.150.2.250.3
(a)−0.8−0.6−0.4 −0.2 0 0.2 0.4 0.60.60.40.200.20.4
(b)
Figure 15.10 2d representation of 12 dimensional oil ﬂow data. The different colors/symbols represent
the 3 phases of oil ﬂow. (a) Kernel PCA with Gaussian kernel. (b) GP-LVM with Gaussian kernel. The
shading represents the precision of the posterior, where lighter pixels have higher precision. From Figure 1
of (Lawrence 2005). Used with kind permission of Neil Lawrence.
via eigenvalue methods; instead we must use gradient-based optimization. The objective is given
by
ℓ=−D
2log|Kz|−1
2tr(K−1
zYYT) (15.121)
and the gradient is given by
∂ℓ
∂Zij=∂ℓ
∂Kz∂Kz
∂Zij(15.122)
where
∂ℓ
∂Kz=K−1
zYYTK−1
z−DK−1
z (15.123)
The form of∂Kz
∂Zijwill of course depend on the kernel used. (For example, with a linear kernel,
whereKz=ZZT+σ2I,w eh a v e∂Kz
∂Z=Z.) We can then pass this gradient to any standard
optimizer, such as conjugate gradient descent.

by
ℓ=−D
2tr(K−1
∂ℓ
∂Zij=∂ℓ
∂Kz∂Kz
where
∂ℓ
∂Kz=K−1
zYYTK−1
z−DK−1
z (15.123)
Let us now compare GP-LVM to kernel PCA. In kPCA, we learn a kernelized mapping from
the observed space to the latent space, whereas in GP-LVM, we learn a kernelized mapping from
the latent space to the observed space. Figure 15.10 illustrates the results of applying kPCA and
GP-LVM to visualize the 12 dimensional oil ﬂow data shown in In Figure 14.9(a). We see that the
embedding produced by GP-LVM is far better. If we perform nearest neighbor classiﬁcation in

the latent space, GP-LVM makes 4 errors, while kernel PCA (with the same kernel but separately
optimized hyper-parameters) makes 13 errors, and regular PCA makes 20 errors.
GP-LVM inherits the usual advantages of probabilistic generative models, such as the ability
to handle missing data and data of different types, the ability to use gradient-based methods
(instead of grid search) to tune the kernel parameters, the ability to handle prior information,

542 Chapter 15. Gaussian processes
etc. For a discussion of some other probabilistic methods for (spectral) dimensionality reduction,
see (Lawrence 2012).
15.6 Approximation methods for large datasets
The principal drawback of GPs is that they take O(N3)time to use. This is because of the
need to invert (or compute the Cholesky decomposition of) the N×Nkernel matrix K.A
variety of approximation methods have been devised which take O(M2N)time, where Mis a
user-speciﬁable parameter. For details, see (Quinonero-Candela et al. 2007).
Exercises
Exercise 15.1 Reproducing property
Prove Equation 15.109.

16 Adaptive basis function models
16.1 Introduction
In Chapters 14 and 15, we discussed kernel methods, which provide a powerful way to create non-
linear models for regression and classiﬁcation. The prediction takes the form f(x)=wTφ(x),
where we deﬁne
φ(x)=[κ(x,μ1),...,κ(x,μN)] (16.1)
and where μkare either all the training data or some subset. Models of this form essen-
tially perform a form of template matching, whereby they compare the input xto the stored
prototypes μk.
Although this can work well, it relies on having a good kernel function to measure the

similarity between data vectors. Often coming up with a good kernel function is quite difficult.For example, how do we deﬁne the similarity between two images? Pixel-wise comparison ofintensities (which is what a Gaussian kernel corresponds to) does not work well. Although it ispossible (and indeed common) to hand-engineer kernels for speciﬁc tasks (see e.g., the pyramidmatch kernel in Section 14.2.7), it would be more interesting if we could learn the kernel.
In Section 15.2.4, we discussed a way to learn the parameters of a kernel function, by maxi-

mizing the marginal likelihood. For example, if we use the ARD kernel,
κ(x,x
′)=θ0exp⎛
⎝−1
2D∑
j=1θj(xj−x′
j)2⎞
⎠ (16.2)
we can can estimate the θj, and thus perform a form of nonlinear feature selection. However,
such methods can be computationally expensive. Another approach, known as multiple kernel
learning (see e.g., (Rakotomamonjy et al. 2008)) uses a convex combination of base kernels,κ(x,x
′)=∑
jwjκj(x,x′), and then estimates the mixing weights wj. But this relies on
having good base kernels (and is also computationally expensive).
An alternative approach is to dispense with kernels altogether, and try to learn useful features
φ(x)directly from the input data. That is, we will create what we call an adaptive basis-
function model (ABM), which is a model of the form
f(x)=w0+M∑
m=1wmφm(x) (16.3)

544 Chapter 16. Adaptive basis function models
whereφm(x)is them’th basis function, which is learned from data. This framework covers all
of the models we will discuss in this chapter.
Typically the basis functions are parametric, so we can write φm(x)=φ(x;vm),w h e r evm
are the parameters of the basis function itself. We will use θ=(w0,w1:M,{vm}M
m=1)to
denote the entire parameter set. The resulting model is not linear-in-the-parameters anymore,
so we will only be able to compute a locally optimal MLE or MAP estimate of θ. Nevertheless,
such models often signiﬁcantly outperform linear models, as we will see.
16.2 Classiﬁcation and regression trees (CART)
Classiﬁcation and regression trees orCARTmodels, also called decision trees (not to be
confused with the decision trees used in decision theory) are deﬁned by recursively partitioningthe input space, and deﬁning a local model in each resulting region of input space. This can berepresented by a tree, with one leaf per region, as we explain below.

16.2.1 Basics
To explain the CART approach, consider the tree in Figure 16.1(a). The ﬁrst node asks if x1is
less than some threshold t1. If yes, we then ask if x2is less than some other threshold t2.I f
yes, we are in the bottom left quadrant of space, R1. If no, we ask if x1is less than t3. And
so on. The result of these axis parallel splits is to partition 2d space into 5 regions, as shown
in Figure 16.1(b). We can now associate a mean response with each of these regions, resulting inthe piecewise constant surface shown in Figure 16.1(c).

We can write the model in the following form
f(x)=E[y|x]=M∑
m=1wmI(x∈Rm)=M∑
m=1wmφ(x;vm) (16.4)
whereRmis them’th region, wmis the mean response in this region, and vmencodes the
choice of variable to split on, and the threshold value, on the path from the root to the m’th leaf.
This makes it clear that a CART model is just a an adaptive basis-function model, where thebasis functions deﬁne the regions, and the weights specify the response value in each region.We discuss how to ﬁnd these basis functions below.

We can generalize this to the classiﬁcation setting by storing the distribution over class labels

in each leaf, instead of the mean response. This is illustrated in Figure 16.2. This model canbe used to classify the data in Figure 1.1. For example, we ﬁrst check the color of the object.If it is blue, we follow the left branch and end up in a leaf labeled “4,0”, which means wehave 4 positive examples and 0 negative examples which match this criterion. Hence we predictp(y=1|x)=4/4ifxis blue. If it is red, we then check the shape: if it is an ellipse, we
end up in a leaf labeled “1,1”, so we predict p(y=1|x)=1/2. If it is red but not an ellipse,

we predict p(y=1|x)=0/2; If it is some other colour, we check the size: if less than 10,
we predict p(y=1|x)=4/4, otherwise p(y=1|x)=0/5. These probabilities are just the
empirical fraction of positive examples that satisfy each conjunction of feature values, whichdeﬁnes a path from the root to a leaf.

16.2. Classiﬁcation and regression trees (CART) 545
X1≤t1
X2≤t2
R1X1≤t3
R4 R5X2≤t4
R2 R3
(a)0246810
02468102345678910
(b)
Figure 16.1 A simple regression tree on two inputs. Based on Figure 9.2 of (Hastie et al. 2009). Figure
generated by regtreeSurfaceDemo .
1,1 0,2shape4,0color
size<10
4,0 0,5ellipse otherblueredother
yes no
Figure 16.2 A simple decision tree for the data in Figure 1.1. A leaf labeled as (n1,n0)means that
there are n1positive examples that match this path, and n0negative examples. In this tree, most of
the leaves are “pure”, meaning they only have examples of one class or the other; the only exception is
leaf representing red ellipses, which has a label distribution of (1,1). We could distinguish positive from
negative red ellipses by adding a further test based on size. However, it is not always desirable to constructtrees that perfectly model the training data, due to overﬁtting.
16.2.2 Growing a tree

Finding the optimal partitioning of the data is NP-complete (Hyaﬁl and Rivest 1976), so it is
common to use the greedy procedure shown in Algorithm 6 to compute a locally optimal MLE.This method is used by CART, (Breiman et al. 1984) C4.5(Quinlan 1993), and ID3(Quinlan 1986),
which are three popular implementations of the method. (See dtfitfor a simple Matlab
implementation.)
The split function chooses the best feature, and the best value for that feature, as follows:
(j
∗,t∗) = arg min
j∈{1,...,D }min

(j
t∈Tjcost({xi,yi:xij≤t})+cost( {xi,yi:xij>t}) (16.5)

546 Chapter 16. Adaptive basis function models
Algorithm 16.1: Recursive procedure to grow a classiﬁcation/ regression tree
1function ﬁtTree(node, D, depth) ;
2node.prediction = mean(y i:i∈D) // or class label distribution ;
3(j∗,t∗,DL,DR)=split(D);
4ifnot worthSplitting(depth, cost, DL,DR)then
5return node
6else
7node.test = λx.xj∗<t∗// anonymous function;
8node.left = ﬁtTree(node, DL, depth+1);
9node.right = ﬁtTree(node, DR, depth+1);
10return node;
where the cost function for a given dataset will be deﬁned below. For notational simplicity, we
have assumed all inputs are real-valued or ordinal, so it makes sense to compare a feature xij
to a numeric value t. The set of possible thresholds Tjfor feature jcan be obtained by sorting
the unique values of xij. For example, if feature 1 has the values {4.5,−12,72,−12}, then we
setT1={−12,4.5,72}. In the case of categorical inputs, the most common approach is to
consider splits of the form xij=ckandxij̸=ck, for each possible class label ck. Although

we could allow for multi-way splits (resulting in non-binary trees), this would result in data
fragmentation, meaning too little data might “fall” into each subtree, resulting in overﬁtting.
The function that checks if a node is worth splitting can use several stopping heuristics, such
as the following:
• is the reduction in cost too small? Typically we deﬁne the gain of using a feature to be a
normalized measure of the reduction in cost:
Δ≜cost(D)−(|DL|
|D|cost(DL)+|DR|
|D|cost(DR))
(16.6)
• has the tree exceeded the maximum desired depth?

(16.6)
• is the distribution of the response in either DLorDRsufficiently homogeneous (e.g., all
labels are the same, so the distribution is pure)?
• is the number of examples in either DLorDRtoo small?
All that remains is to specify the cost measure used to evaluate the quality of a proposed
split. This depends on whether our goal is regression or classiﬁcation. We discuss both cases
below.
16.2.2.1 Regression cost
In the regression setting, we deﬁne the cost as follows:
cost(D)=∑
i∈D(yi−y)2(16.7)

16.2. Classiﬁcation and regression trees (CART) 547
wherey=1
|D|∑
i∈Dyiis the mean of the response variable in the speciﬁed set of data.
Alternatively, we can ﬁt a linear regression model for each leaf, using as inputs the features that
were chosen on the path from the root, and then measure the residual error.
16.2.2.2 Classiﬁcation cost
In the classiﬁcation setting, there are several ways to measure the quality of a split. First, weﬁt a multinoulli model to the data in the leaf satisfying the test X
j<tby estimating the
class-conditional probabilities as follows:
ˆπc=1
|D|∑
i∈DI(yi=c) (16.8)
whereDis the data in the leaf. Given this, there are several common error measures for
evaluating a proposed partition:
•Misclassiﬁcation rate. We deﬁne the most probable class label as ˆyc=a r g m a xcˆπc. The
corresponding error rate is then
1
|D|∑
i∈DI(yi̸=ˆy)=1−ˆπˆy (16.9)
•Entropy,o rdeviance:
H(ˆπ)=−C∑
c=1ˆπclogˆπc (16.10)
Note that minimizing the entropy is equivalent to maximizing the information gain (Quinlan

ˆπc=1
|D|∑
1
|D|∑
H(ˆπ)=−C∑
1986) between test Xj<tand the class label Y, deﬁned by
infoGain( Xj<t ,Y)≜H(Y)−H(Y|Xj<t) (16.11)
=(
−∑
cp(y=c)logp(y=c))
(16.12)
+(∑
cp(y=c|Xj<t)logp(c|Xj<t))
(16.13)
sinceˆπcis an MLE for the distribution p(c|Xj<t).1
1. IfXjis categorical, and we use tests of the form Xj=k, then taking expectations over values of Xjgives
the mutual information between XjandY:E[infoGain(X j,Y)] =∑
kp(Xj=k)infoGain(X j=k,Y)=
H(Y)−H(Y|Xj)=I(Y;Xj).

548 Chapter 16. Adaptive basis function models
0 0.2 0.4 0.6 0.8 100.050.10.150.20.250.30.350.40.450.5
  
Error rate
Gini
Entropy
Figure 16.3 Node impurity measures for binary classiﬁcation. The horizontal axis corresponds to p, the
probability of class 1. The entropy measure has been rescaled to pass through (0.5,0.5). Based on Figure
9.3 of (Hastie et al. 2009). Figure generated by giniDemo .
•Gini index
C∑
c=1ˆπc(1−ˆπc)=∑
cˆπc−∑
cˆπ2
c=1−∑
cˆπ2
c (16.14)
This is the expected error rate. To see this, note that ˆπcis the probability a random entry in
the leaf belongs to class c, and(1−ˆπcis the probability it would be misclassiﬁed.
In the two-class case, where p=πm(1), the misclassiﬁcation rate is 1−max(p,1−p), the
entropy is H2(p), and the Gini index is 2p(1−p). These are plotted in Figure 16.3. We see
that the cross-entropy and Gini measures are very similar, and are more sensitive to changes in

C∑
cˆπc−∑
cˆπ2
c=1−∑
cˆπ2
c (16.14)
class probability than is the misclassiﬁcation rate. For example, consider a two-class problemwith 400 cases in each class. Suppose one split created the nodes (300,100) and (100,300), whilethe other created the nodes (200,400) and (200,0). Both splits produce a misclassiﬁcation rate of0.25. However, the latter seems preferable, since one of the nodes is pure, i.e., it only contains
one class. The cross-entropy and Gini measures will favor this latter choice.
16.2.2.3 Example

As an example, consider two of the four features from the 3-class iris dataset, shown in Fig-ure 16.4(a). The resulting tree is shown in Figure 16.5(a), and the decision boundaries are shownin Figure 16.4(b). We see that the tree is quite complex, as are the resulting decision boundaries.In Figure 16.5(b), we show that the CV estimate of the error is much higher than the training seterror, indicating overﬁtting. Below we discuss how to perform a tree-pruning stage to simplifythe tree.

16.2. Classiﬁcation and regression trees (CART) 549
4 4.5 5 5.5 6 6.5 7 7.5 822.533.544.5
Sepal lengthSepal width
  
setosa
versicolor
virginica
(a)4 4.5 5 5.5 6 6.5 7 7.5 822.533.544.5
xyunpruned decision tree
  
setosa
virginica
(b)
Figure 16.4 (a) Iris data. We only show the ﬁrst two features, sepal length and sepal width, and ignore
petal length and petal width. (b) Decision boundaries induced by the decision tree in Figure 16.5(a).
versicolor setosa
setosa virginica
versicolor versicolor
versicolor virginica virginica
virginica versicolor
versicolor virginica
virginica versicolor versicolor virginicaSL < 5.45   
W < 2.8   SL < 6.15   
SW < 3.45   SL < 7.05   
SL < 5.75   SW < 2.4   
SW < 3.1   SL < 6.95   
SW < 2.95   SW < 3.15   
SL < 6.55   
SW < 2.95   SL < 6.65   
SL < 6.45   SW < 2.65   
SW < 2.85   SW < 2.9     SL >= 5.45
  SW >= 2.8   SL >= 6.15
  SW >= 3.45   SL >= 7.05
  SL >= 5.75   SW >= 2.4
  SW >= 3.1   SL >= 6.95
  SW >= 2.95   SW >= 3.15
  SL >= 6.55

SL < 6.55   
  SL >= 6.55
  SW >= 2.95   SL >= 6.65
  SL >= 6.45   SW >= 2.65
  SW >= 2.85   SW >= 2.9
(a)0 5 10 15 200.10.20.30.40.50.60.70.8
Number of terminal nodesCost (misclassification error)
  
Cross−validation
Training set
Min + 1 std. err.
Best choice
(b)
Figure 16.5 (a) Unpruned decision tree for Iris data. (b) Plot of misclassiﬁcation error rate vs depth of
tree. Figure generated by dtreeDemoIris .
16.2.3 Pruning a tree
To prevent overﬁtting, we can stop growing the tree if the decrease in the error is not sufficient

SW >= 2.95   SL >= 6.65
  
(b)
to justify the extra complexity of adding an extra subtree. However, this tends to be too myopic.For example, on the xor data in Figure 14.2(c), it would might never make any splits, since eachfeature on its own has little predictive power.
The standard approach is therefore to grow a “full” tree, and then to perform pruning. This
can be done using a scheme that prunes the branches giving the least increase in the error. See(Breiman et al. 1984) for details.

To determine how far to prune back, we can evaluate the cross-validated error on each such
subtree, and then pick the tree whose CV error is within 1 standard error of the minimum. Thisis illustrated in Figure 16.4(b). The point with the minimum CV error corresponds to the simpletree in Figure 16.6(a).

550 Chapter 16. Adaptive basis function models
versicolor setosa virginica
versicolor setosaSL < 5.45   
SW < 2.8   SL < 6.15   
SW < 3.45     SL >= 5.45
  SW >= 3.45
xypruned decision tree
  
setosa
virginica
(b)
Figure 16.6 Pruned decision tree for Iris data. Figure generated by dtreeDemoIris .
16.2.4 Pros and cons of trees
CART models are popular for several reasons: they are easy to interpret2, they can easily handle
mixed discrete and continuous inputs, they are insensitive to monotone transformations of the
inputs (because the split points are based on ranking the data points), they perform automaticvariable selection, they are relatively robust to outliers, they scale well to large data sets, andthey can be modiﬁed to handle missing inputs.
3
However, CART models also have some disadvantages. The primary one is that they do

3
not predict very accurately compared to other kinds of model. This is in part due to thegreedy nature of the tree construction algorithm. A related problem is that trees are unstable:
small changes to the input data can have large effects on the structure of the tree, due to thehierarchical nature of the tree-growing process, causing errors at the top to affect the rest of thetree. In frequentist terminology, we say that trees are high variance estimators. We discuss asolution to this below.
16.2.5 Random forests

One way to reduce the variance of an estimate is to average together many estimates. Forexample, we can train Mdifferent trees on different subsets of the data, chosen randomly with
2. We can postprocess the tree to derive a series of logical rulessuch as “If x1<5.45then ...” (Quinlan 1990).
3. The standard heuristic for handling missing inputs in decision trees is to look for a series of ”backup” variables,
which can induce a similar partition to the chosen variable at any given split; these can be used in case the chosen
variable is unobserved at test time. These are called surrogate splits. This method ﬁnds highly correlated features,

and can be thought of as learning a local joint model of the input. This has the advantage over a generative modelof not modeling the entire joint distribution of inputs, but it has the disadvantage of being entirely ad hoc. A simpler
approach, applicable to categorical variables, is to code “missing” as a new value, and then to treat the data as fully
observed.

16.2. Classiﬁcation and regression trees (CART) 551
replacement, and then compute the ensemble
f(x)=M∑
m=11
Mfm(x) (16.15)
wherefmis them’th tree. This technique is called bagging(Breiman 1996), which stands for
“bootstrap aggregating”.
Unfortunately, simply re-running the same learning algorithm on different subsets of the data
can result in highly correlated predictors, which limits the amount of variance reduction that is
possible. The technique known as random forests (Breiman 2001a) tries to decorrelate the base
learners by learning trees based on a randomly chosen subset of input variables, as well as arandomly chosen subset of data cases. Such models often have very good predictive accuracy(Caruana and Niculescu-Mizil 2006), and have been widely used in many applications (e.g., forbody pose recognition using Microsoft’s popular kinect sensor (Shotton et al. 2011)).
Bagging is a frequentist concept. It is also possible to adopt a Bayesian approach to learning

trees. Inparticular, (Chipmanetal.1998;Denisonetal.1998;Wuetal.2007)performapproximateinference over the space of trees (structure and parameters) using MCMC. This reduces thevariance of the predictions. We can also perform Bayesian inference over the space of ensemblesof trees, which tends to work much better. This is known as Bayesian adaptive regression
treesorBART(Chipman et al. 2010). Note that the cost of these sampling-based Bayesian

methods is comparable to the sampling-based random forest method. That is, both approachesare farily slow to train, but produce high quality classiﬁers.
Unfortunately, methods that use multiple trees (whether derived from a Bayesian or frequen-
tist standpoint) lose their nice interpretability properties. Fortunately, various post-processingmeasures can be applied, as discussed in Section 16.8.
16.2.6 CART compared to hierarchical mixture of experts *

An interesting alternative to a decision tree is known as the hierarchical mixture of experts.Figure 11.7(b) gives an illustration where we have two levels of experts. This can be thought ofas a probabilistic decision tree of depth 2, since we recursively partition the space, and applya different expert to each partition. Hastie et al. (Hastie et al. 2009, p331) write that “TheHME approach is a promising competitor to CART trees”. Some of the advantages include thefollowing:

• The model can partition the input space using any set of nested linear decision boundaries.
By contrast, standard decision trees are constrained to use axis-parallel splits.
• The model makes predictions by averaging over all experts. By contrast, in a standard
decision tree, predictions are made only based on the model in the corresponding leaf. Since
leaves often contain few training examples, this can result in overﬁtting.
• Fitting an HME involves solving a smooth continuous optimization problem (usually using

EM), which is likely to be less prone to local optima than the standard greedy discreteoptimization methods used to ﬁt decision trees. For similar reasons, it is computationallyeasier to “be Bayesian” about the parameters of an HME (see e.g., (Peng et al. 1996; Bishop

552 Chapter 16. Adaptive basis function models
and Svensén 2003)) than about the structure and parameters of a decision tree (see e.g., (Wu
et al. 2007)).
16.3 Generalized additive models
A simple way to create a nonlinear model with multiple inputs is to use a generalized additive
model(Hastie and Tibshirani 1990), which is a model of the form
f(x)=α+f1(x1)+···+fD(xD) (16.16)
Here each fjcan be modeled by some scatterplot smoother, and f(x)can be mapped to p(y|x)
using a link function, as in a GLM (hence the term generalized additive model).
If we use regression splines (or some other ﬁxed basis function expansion approach) for the
fj, then each fj(xj)can be written as βT
jφj(xj), so the whole model can be written as
f(x)=βTφ(x),w h e r eφ(x)=[ 1,φ1(x1),...,φD(xD)]. However, it is more common to use
smoothing splines (Section 15.4.6) for the fj. In this case, the objective (in the regression setting)
becomes
J(α,f1,...,f D)=N∑
i=1⎛
⎝yi−α−D∑
j=1fj(xij)⎞⎠2
+D∑
j=1λj∫
f′′
j(tj)2dtj (16.17)

becomes
i=1⎛
⎝yi−α−D∑
+D∑
j=1λj∫
f′′
whereλjis the strength of the regularizer for fj.
16.3.1 Backﬁtting
We now discuss how to ﬁt the model using MLE. The constant αis not uniquely identiﬁable,
since we can always add or subtract constants to any of the fjfunctions. The convention is to
assume∑N
i=1fj(xij)=0for allj. In this case, the MLE for αis justˆα=1
N∑Ni=1yi.
To ﬁt the rest of the model, we can center the responses (by subtracting ˆα), and then
iteratively update each fjin turn, using as a target vector the residuals obtained by omitting
termfj:
ˆfj:=smoother( {yi−∑
k̸=jˆfk(xik)}N
i=1) (16.18)

assume∑N
N∑Ni=1yi.
termfj:
We should then ensure the output is zero mean using
ˆfj:=ˆfj−1
NN∑
i=1ˆfj(xij) (16.19)
This is called the backﬁtting algorithm (Hastie and Tibshirani 1990). If Xhas full column rank,
then the above objective is convex (since each smoothing spline is a linear operator, as shown
in Section 15.4.2), so this procedure is guaranteed to converge to the global optimum.
In the GLM case, we need to modify the method somewhat. The basic idea is to replace the

termfj:
NN∑
weighted least squares step of IRLS (see Section 8.3.4) with a weighted backﬁtting algorithm. Inthe logistic regression case, each response has weight s
i=μi(1−μi)associated with it, where
μi=s i g m (ˆα+∑D
j=1ˆfj(xij)).)

16.3. Generalized additive models 553
16.3.2 Computational efficiency
Each call to the smoother takes O(N)time, so the total cost is O(NDT),w h e r eTis the
number of iterations. If we have high-dimensional inputs, ﬁtting a GAM is expensive. One
approach is to combine it with a sparsity penalty, see e.g., the SpAM(sparse additive model)
approach of (Ravikumar et al. 2009). Alternatively, we can use a greedy approach, such asboosting (see Section 16.4.6)
16.3.3 Multivariate adaptive regression splines (MARS)
We can extend GAMs by allowing for interaction effects. In general, we can create an ANOVAdecomposition:
f(x)=β
0+D∑
j=1fj(xj)+∑
j,kfjk(xj,xk)+∑
j,k,lfjkl(xj,xk,xl)+··· (16.20)
Of course, we cannot allow for too many higher-order interactions, because there will be toomany parameters to ﬁt.
It is common to use greedy search to decide which variables to add. The multivariate
adaptive regression splines orMARSalgorithm is one example of this (Hastie et al. 2009,

f(x)=β
0+D∑
Sec9.4). It ﬁts models of the form in Equation 16.20, where it uses a tensor product basis ofregression splines to represent the multidimensional regression functions. For example, for 2dinput, we might use
f(x
mβ1m(x1−t1m)+
+∑
mβ2m(t2m−x2)++∑
mβ12m(x1−t1m)+(t2m−x2)+ (16.21)
To create such a function, we start with a set of candidate basis functions of the form
C={(xj−t)+,(t−xj)+:t∈{x1j,...,x Nj},j=1,...,D} (16.22)

f(x
+∑
These are 1d linear splines where the knots are at all the observed values for that variable. Weconsider splines sloping up in both directions; this is called a reﬂecting pair. See Figure 16.7(a).
LetMrepresent the current set of basis functions. We initialize by using M={1}.W e
consider creating a new basis function pair by multplying an h
m∈Mwith one of the reﬂecting
pairs inC. For example, we might initially get
f(x)=2 5−4(x1−5)++20(5−x1)+ (16.23)
obtained by multiplying h0(x)=1with a reﬂecting pair involving x1with knot t=5. This

pair is added to M. See Figure 16.7(b). At the next step, we might create a model such as
f(x)== 2 −2(x1−5)++3(5−x1)+
−(x2−10)+×(5−x1)+−1.2(10−x2)+×(5−x1)+ (16.24)
obtained by multiplying (5−x1)+fromMby the new reﬂecting pair (x2−10)+and(10−x2)+.
This new function is shown in Figure 16.7(c).

554 Chapter 16. Adaptive basis function models
0 5 10 15 20051015
(a)0 5 10 15 20−40−20020406080100120
(b)05101520
05101520−60−40−20020
(c)
Figure 16.7 (a) Linear spline function with a knot at 5. Solid blue: (x−5)+. Dotted red: (5−x)+. (b) A
MARS model in 1d given by Equation 16.23. (c) A simple MARS model in 2d given by Equation 16.24. Figure
generated by marsDemo .
We proceed in this way until the model becomes very large. (We may impose an upper
bound on the order of interactions.) Then we prune backwards, at each step eliminating the
basis function that causes the smallest increase in the residual error, until the CV error stopsimproving.
The whole procedure is closely related to CART. To see this, suppose we replace the piecewise
linear basis functions by step functions I(x
j>t)andI(xj<t). Multiplying by a pair of

reﬂected step functions is equivalent to splitting a node. Now suppose we impose the constraintthat once a variable is involved in a multiplication by a candidate term, that variable getsreplaced by the interaction, so the original variable is no longer available. This ensures that avariable can not be split more than once, thus guaranteeing that the resulting model can berepresented as a tree. In this case, the MARS growing strategy is the same as the CART growingstrategy.
16.4 Boosting

Boosting (Schapire and Freund 2012) is a greedy algorithm for ﬁtting adaptive basis-function
models of the form in Equation 16.3, where the φmare generated by an algorithm called a weak
learneror abase learner. The algorithm works by applying the weak learner sequentially to
weighted versions of the data, where more weight is given to examples that were misclassiﬁedby earlier rounds.
This weak learner can be any classiﬁcation or regression algorithm, but it is common to use a

CART model. In 1998, the late Leo Breiman called boosting, where the weak learner is a shallowdecision tree, the “best off-the-shelf classiﬁer in the world” (Hastie et al. 2009, p340). Thisis supported by an extensive empirical comparison of 10 different classiﬁers in (Caruana andNiculescu-Mizil 2006), who showed that boosted decision trees were the best both in terms ofmisclassiﬁcation error and in terms of producing well-calibrated probabilities, as judged by ROCcurves. (The second best method was random forests, invented by Breiman; see Section 16.2.5.)By contrast, single decision trees performed very poorly.

Boosting was originally derived in the computational learning theory literature (Schapire 1990;
Freund and Schapire 1996), where the focus is binary classiﬁcation. In these papers, it wasproved that one could boost the performance (on the training set) of any weak learner arbitrarily

16.4. Boosting 555
0 20 40 60 80 100 120 14000.020.040.060.080.10.120.140.16
train
test
Figure 16.8 Performance of adaboost using a decision stump as a weak learner on the data in Figure 16.10.
Training (solid blue) and test (dotted red) error vs number of iterations. Figure generated by boostingDemo ,
written by Richard Stapenhurst.
high, provided the weak learner could always perform slightly better than chance. For example,

train
test
in Figure 16.8, we plot the training and test error for boosted decision stumps on a 2d datasetshown in Figure 16.10. We see that the training set error rapidly goes to near zero. What is moresurprising is that the test set error continues to decline even after the training set error hasreached zero (although the test set error will eventually go up). Thus boosting is very resistantto overﬁtting. (Boosted decision stumps form the basis of a very successful face detector (Violaand Jones 2001), which was used to generate the results in Figure 1.6, and which is used in manydigital cameras.)

In view of its stunning empirical success, statisticians started to become interested in this
method. Breiman (Breiman 1998) showed that boosting can be interpreted as a form of gradient
descent in function space. This view was then extended in (Friedman et al. 2000), who showedhow boosting could be extended to handle a variety of loss functions, including for regression,robust regression, Poisson regression, etc. In this section, we shall present this statistical inter-pretation of boosting, drawing on the reviews in (Buhlmann and Hothorn 2007) and (Hastie et al.2009, ch10), which should be consulted for further details.
16.4.1 Forward stagewise additive modeling
The goal of boosting is to solve the following optimization problem:
min
fN∑
i=1L(yi,f(xi)) (16.25)
andL(y,ˆy)is some loss function, and fis assumed to be an ABM model as in Equation 16.3.
Common choices for the loss function are listed in Table 16.1.
If we use squared error loss, the optimal estimate is given by
f∗(x) = argmin
f(x)=Ey|x[

min
fN∑
(Y−f(x))2]
=E[Y|x] (16.26)

556 Chapter 16. Adaptive basis function models
Name Loss Derivative f∗Algorithm
Squared error1
2(yi−f(xi))2yi−f(xi) E[y|xi] L2Boosting
Absolute error |yi−f(xi)|sgn(yi−f(xi))median(y|xi)Gradient boosting
Exponential loss exp(−˜yif(xi))−˜yiexp(−˜yif(xi))1
2logπi
1−πiAdaBoost
Logloss log(1+e−˜yifi)yi−πi1
2logπi
1−πiLogitBoost
Table 16.1 Some commonly used loss functions, their gradients, their population minimizers f∗, and
some algorithms to minimize the loss. For binary classiﬁcation problems, we assume ˜yi∈{ −1,+1},
yi∈{0,1}andπi= sigm(2f (xi)). For regression problems, we assume yi∈R. Adapted from (Hastie
et al. 2009, p360) and (Buhlmann and Hothorn 2007, p483).
−2 −1.5 −1 −0.5 0 0.5 1 1.5 201234567
y− floss
  
0−1
logloss
exp
Figure 16.9 Illustration of various loss functions for binary classiﬁcation. The horizontal axis is the
marginyη, the vertical axis is the loss. The log loss uses log base 2. Figure generated by hingeLossPlot .

y− floss
  
0−1
logloss
exp
as we showed in Section 5.7.1.3. Of course, this cannot be computed in practice since it requires
knowing the true conditional distribution p(y|x). Hence this is sometimes called the population
minimizer, where the expectation is interpreted in a frequentist sense. Below we will see thatboosting will try to approximate this conditional expectation.
For binary classiﬁcation, the obvious loss is 0-1 loss, but this is not differentiable. Instead

it is common to use logloss, which is a convex upper bound on 0-1 loss, as we showed inSection 6.5.5. In this case, one can show that the optimal estimate is given by
f
∗(x)=1
2logp(˜y=1|x)
p(˜y=−1|x)(16.27)
where˜y∈{ −1,+1}. One can generalize this framework to the multiclass case, but we will not
discuss that here.
An alternative convex upper bound is exponential loss, deﬁned by
L(˜y,f)=e x p (−˜yf) (16.28)

f
∗(x)=1
See Figure 16.9 for a plot. This will have some computational advantages over the logloss,to be discussed below. It turns out that the optimal estimate for this loss is also f
∗(x)=

16.4. Boosting 557
1
p(˜y=−1|x). To see this, we can just set the derivative of the expected loss (for each x)t o
zero:
∂
∂f(x)E[
e−˜yf(x)|x]
=∂
∂f(x)[p(˜y=1|x)e−f(x)+p(˜y=−1|x)ef(x)] (16.29)
=−p(˜y=1|x)e−f(x)+p(˜y=−1|x)ef(x)(16.30)
=0⇒p(˜y=1|x)
p(˜y=1−|x)=e2f(x)(16.31)
So in both cases, we can see that boosting should try to approximate (half) the log-odds ratio.
Since ﬁnding the optimal fis hard, we shall tackle it sequentially. We initialise by deﬁning
f0(x)=a r gm i n
γN∑
i=1L(yi,f(xi;γ)) (16.32)
For example, if we use squared error, we can set f0(x)=y, and if we use log-loss or exponential
loss , we can set f0(x)=1
2logˆπ
1−ˆπ,w h e r eˆπ=1
N∑N
i=1I(yi=1 ). We could also use a more
powerful model for our baseline, such as a GLM.
Then at iteration m, we compute
(βm,γm) = argmin
β,γN∑
i=1L(yi,fm−1(xi)+βφ(xi;γ)) (16.33)
and then we set
fm(x)=fm−1(x)+βmφ(x;γm) (16.34)
The key point is that we do not go back and adjust earlier parameters. This is why the method

γN∑
2logˆπ
N∑N
β,γN∑
is calledforward stagewise additive modeling.
We continue this for a ﬁxed number of iterations M. In factMis the main tuning parameter
of the method. Often we pick it by monitoring the performance on a separate validation set, andthen stopping once performance starts to decrease; this is called early stopping. Alternatively,
we can use model selection criteria such as AIC or BIC (see e.g., (Buhlmann and Hothorn 2007)for details).

In practice, better (test set) performance can be obtained by performing “partial updates” of
the form
f
m(x)=fm−1(x)+νβ mφ(x;γm) (16.35)
Here0<ν≤1is a step-size parameter. In practice it is common to use a small value such as
ν=0.1. This is called shrinkage.
Below we discuss how to solve the suproblem in Equation 16.33. This will depend on the
form of loss function. However, it is independent of the form of weak learner.
16.4.2 L2boosting
Suppose we used squared error loss. Then at step mthe loss has the form
L(yi,fm−1(xi)+βφ(xi;γ)) = (rim−φ(xi;γ))2(16.36)

558 Chapter 16. Adaptive basis function models
(a)
 (b)
 (c)
Figure 16.10 Example of adaboost using a decision stump as a weak learner. The degree of blackness
represents the conﬁdence in the red class. The degree of whiteness represents the conﬁdence in the blue
class. The size of the datapoints represents their weight. Decision boundary is in yellow. (a) After 1
round. (b) After 3 rounds. (c) After 120 rounds. Figure generated by boostingDemo , written by Richard
Stapenhurst.
whererim≜yi−fm−1(xi)is the current residual, and we have set β=1without loss of
generality. Hence we can ﬁnd the new basis function by using the weak learner to predict rm.
This is called L2boosting ,o rleast squares boosting (Buhlmann and Yu 2003). In Section 16.4.6,
we will see that this method, with a suitable choice of weak learner, can be made to give the
same results as LARS, which can be used to perform variable selection (see Section 13.4.2).
16.4.3 AdaBoost

Consider a binary classiﬁcation problem with exponential loss. At step mwe have to minimize
Lm(φ)=N∑
i=1exp[−˜yi(fm−1(xi)+βφ(xi))] =N∑
i=1wi,mexp(−β˜yiφ(xi)) (16.37)
wherewi,m≜exp(−˜yifm−1(xi))is a weight applied to datacase i, and˜yi∈{ −1,+1}.W e
can rewrite this objective as follows:
Lm=e−β∑
˜yi=φ(xi)wi,m+eβ∑
˜yi̸=φ(xi)wi,m (16.38)
=(eβ−e−β)N∑
i=1wi,mI(˜yi̸=φ(xi))+e−βN∑
i=1wi,m (16.39)
Consequently the optimal function to add is
φm=a r g m i n
φwi,mI(˜yi̸=φ(xi)) (16.40)

Lm(φ)=N∑
Lm=e−β∑
φm=a r g m i n
This can be found by applying the weak learner to a weighted version of the dataset, with
weightswi,m. Subsituting φmintoLmand solving for βwe ﬁnd
βm=1
2log1−errm
errm(16.41)

16.4. Boosting 559
where
errm=∑N
i=1wiI(˜yi̸=φm(xi))
∑Ni=1wi,m(16.42)
The overall update becomes
fm(x)=fm−1(x)+βmφm(x) (16.43)
With this, the weights at the next iteration become
wi,m+1=wi,me−βm˜yiφm(xi)(16.44)
=wi,meβm(2I(˜yi̸=φm(xi))−1)(16.45)
=wi,me2βmI(˜yi̸=φm(xi))e−βm(16.46)
where we exploited the fact that −˜yiφm(xi)=−1if˜yi=φm(xi)and−˜yiφm(xi)=+ 1
otherwise. Since e−βmwill cancel out in the normalization step, we can drop it. The result is
the algorithm shown in Algorithm 7, known Adaboost.M1.4
An example of this algorithm in action, using decision stumps as the weak learner, is given in
Figure 16.10. We see that after many iterations, we can “carve out” a complex decision boundary.
What is rather surprising is that AdaBoost is very slow to overﬁt, as is apparent in Figure 16.8.See Section 16.4.8 for a discussion of this point.
Algorithm 16.2: Adaboost.M1, for binary classiﬁcation with exponential loss
1wi=1/N;
2form=1:M do
3Fit a classiﬁer φm(x)to the training set using weights w;
4Compute err m=∑N

1wi=1/N;
i=1wi,mI(˜yi̸=φm(xi))∑N
i=1wi,m);
5Compute αm= log[(1−errm)/errm];
6Setwi←wiexp[αmI(˜yi̸=φm(xi))];
7Returnf(x)=s g n[∑M
m=1αmφm(x)]
;
16.4.4 LogitBoost
The trouble with exponential loss is that it puts a lot of weight on misclassiﬁed examples, as
is apparent from the exponential blowup on the left hand side of Figure 16.9. This makes themethod very sensitive to outliers (mislabeled examples). In addition, e
−˜yfis not the logarithm
of any pmf for binary variables ˜y∈{ −1,+1}; consequently we cannot recover probability
estimates from f(x).

;
4. In (Friedman et al. 2000), this is called discrete AdaBoost, since it assumes that the base classiﬁer φmreturns a
binary class label. If φmreturns a probability instead, a modiﬁed algorithm, known as real AdaBoost, can be used. See
(Friedman et al. 2000) for details.

560 Chapter 16. Adaptive basis function models
A natural alternative is to use logloss instead. This only punishes mistakes linearly, as is clear
from Figure 16.9. Furthermore, it means that we will be able to extract probabilities from the
ﬁnal learned function, using
p(y=1|x)=ef(x)
e−f(x)+ef(x)=1
1+e−2f(x)(16.47)
The goal is to minimze the expected log-loss, given by
Lm(φ)=N∑
i=1log[1+exp(− 2˜yi(fm−1(x)+φ(xi)))] (16.48)
By performing a Newton upate on this objective (similar to IRLS), one can derive the algorithmshown in Algorithm 8. This is known as logitBoost (Friedman et al. 2000). It can be generalized
to the multi-class setting, as explained in (Friedman et al. 2000).
Algorithm 16.3: LogitBoost, for binary classiﬁcation with log-loss
1wi=1/N,πi=1/2;
3Compute the working response zi=y∗
i−πi
πi(1−πi);
4Compute the weights wi=πi(1−πi);
5φm=a r g m i nφ∑N
i=1wi(zi−φ(xi))2;
6Updatef(x)←f(x)+1
2φm(x);
7Compute πi=1/(1+exp(− 2f(xi)));
8Returnf(x)=s g n[∑Mm=1φm(x)]
;

i−πi
πi(1−πi);
2φm(x);
;
16.4.5 Boosting as functional gradient descent
Rather than deriving new versions of boosting for every different loss function, it is possible to
derive a generic version, known as gradient boosting (Friedman 2001; Mason et al. 2000). To
explain this, imagine minimizing
ˆf=a r g m i n
fL(f) (16.49)
wheref=(f(x1),...,f(xN))are the “parameters”. We will solve this stagewise, using gradient
descent. At step m,l e tg mbe the gradient of L(f)evaluated at f=fm−1:
gim=[∂L(yi,f(xi))
∂f(xi)]
f=fm−1(16.50)
Gradients of some common loss functions are given in Table 16.1. We then make the update
fm=fm−1−ρmgm (16.51)

16.4. Boosting 561
whereρmis the step length, chosen by
ρm=a r g m i n
ρL(fm−1−ρgm) (16.52)
This is called functional gradient descent.
In its current form, this is not much use, since it only optimizes fat a ﬁxed set of Npoints,
so we do not learn a function that can generalize. However, we can modify the algorithm by
ﬁtting a weak learner to approximate the negative gradient signal. That is, we use this update
γm=a r g m i n
γN∑
i=1(−gim−φ(xi;γ))2(16.53)
The overall algorithm is summarized in Algorithm 6. (We have omitted the line search step,which is not strictly necessary, as argued in (Buhlmann and Hothorn 2007).)
Algorithm 16.4: Gradient boosting
1Initializef0(x) = argminγ∑N
i=1L(yi,φ(xi;γ));
3Compute the gradient residual using rim=−[
∂L(yi,f(xi))
∂f(xi)]
f(xi)=fm−1(xi);
4Use the weak learner to compute γmwhich minimizes∑Ni=1(rim−φ(xi;γm))2;
5Updatefm(x)=fm−1(x)+νφ(x;γm);
6Returnf(x)=fM(x)
If we apply this algorithm using squared loss, we recover L2Boosting. If we apply this

∂f(xi)]
algorithm to log-loss, we get an algorithm known as BinomialBoost (Buhlmann and Hothorn
2007). The advantage of this over LogitBoost is that it does not need to be able to do weighted
ﬁtting: it just applies any black-box regression model to the gradient vector. Also, it is relativelyeasy to extend to the multi-class case (see (Hastie et al. 2009, p387)). We can also apply thisalgorithm to other loss functions, such as the Huber loss (Section 7.4), which is more robust tooutliers than squared error loss.
16.4.6 Sparse boosting

Suppose we use as our weak learner the following algorithm: search over all possible variablesj=1:D, and pick the one j(m)that best predicts the residual vector:
j(m) = argmin
jN∑
i=1(rim−ˆβjmxij)2(16.54)
ˆβjm=∑N
i=1xijrim∑Ni=1x2
ij(16.55)
φm(x)=ˆβj(m),mxj(m) (16.56)

562 Chapter 16. Adaptive basis function models
This method, which is known as sparse boosting (Buhlmann and Yu 2006), is identical to the
matching pursuit algorithm discussed in Section 13.2.3.1.
It is clear that this will result in a sparse estimate, at least if Mis small. To see this, let us
rewrite the update as follows:
βm:=βm−1+ν(0,...,0,ˆβj(m),m,0,...,0) (16.57)
where the non-zero entry occurs in location j(m). This is known as forward stagewise linear
regression (Hastie et al. 2009, p608), which becomes equivalent to the LAR algorithm discussed
in Section 13.4.2 as ν→0. Increasing the number of steps min boosting is analogous to
decreasing the regularization penalty λ. If we modify boosting to allow some variable deletion
steps (Zhao and Yu 2007), we can make it equivalent to the LARS algorithm, which computes
the full regularization path for the lasso problem. The same algorithm can be used for sparselogistic regression, by simply modifying the residual to be the appropriate negative gradient.

Now consider a weak learner that is similar to the above, except it uses a smoothing spline
instead of linear regression when mapping from x
jto the residual. The result is a sparse
generalized additive model (see Section 16.3). It can obviously be extended to pick pairs ofvariables at a time. The resulting method often works much better than MARS (Buhlmann andYu 2006).
16.4.7 Multivariate adaptive regression trees (MART)

It is quite common to use CART models as weak learners. It is usually advisable to use a shallowtree, so that the variance is low. Even though the bias will be high (since a shallow tree is likelyto be far from the “truth”), this will compensated for in subsequent rounds of boosting.
The height of the tree is an additional tuning parameter (in addition to M, the number of
rounds of boosting, and ν, the shrinkage factor). Suppose we restrict to trees with Jleaves.
IfJ=2, we get a stump, which can only split on a single variable. If J=3, we allow for

two-variable interactions, etc. In general, it is recommended (e.g., in (Hastie et al. 2009, p363)and (Caruana and Niculescu-Mizil 2006)) to use J≈6.
If we combine the gradient boosting algorithm with (shallow) regression trees, we get a model
known as MART, which stands for “multivariate adaptive regression trees”. This actually includes
a slight reﬁnement to the basic gradient boosting algorithm: after ﬁtting a regression tree to theresidual (negative gradient), we re-estimate the parameters at the leaves of the tree to minimizethe loss:
γ
jm=a r g m i n
γ∑
xi∈RjmL(yi,fm−1(xi)+γ) (16.58)
whereRjmis the region for leaf jin them’th tree, and γjmis the corresponding parameter (the

γ
jm=a r g m i n
γ∑
mean response of yfor regression problems, or the most probable class label for classiﬁcation
problems).
16.4.8 Why does boosting work so well?
We have seen that boosting works very well, especially for classiﬁers. There are two mainreasons for this. First, it can be seen as a form of ℓ
1regularization, which is known to help

16.5. Feedforward neural networks (multilayer perceptrons) 563
prevent overﬁtting by eliminating “irrelevant” features. To see this, imagine pre-computing all
possible weak-learners, and deﬁning a feature vector of the form φ(x)=[φ1(x),...,φ K(x)].
We could use ℓ1regularization to select a subset of these. Alternatively we can use boosting,
where at each step, the weak learner creates a new φkon the ﬂy. It is possible to combine
boosting and ℓ1regularization, to get an algorithm known as L1-Adaboost (Duchi and Singer
2009). Essentially this method greedily adds the best features (weak learners) using boosting,and then prunes off irrelevant ones using ℓ
1regularization.
Another explanation has to do with the concept of margin, which we introduced in Sec-
tion 14.5.2.2. (Schapire et al. 1998; Ratsch et al. 2001) proved that AdaBoost maximizes themargin on the training data. (Rosset et al. 2004) generalized this to other loss functions, such aslog-loss.
16.4.9 A Bayesian view

So far, our presentation of boosting has been very frequentist, since it has focussed on greedilyminimizing loss functions. A likelihood interpretation of the algorithm was given in (Neal andMacKay 1998; Meek et al. 2002). The idea is to consider a mixture of experts model of the form
p(y|x,θ)=
M∑
m=1πmp(y|x,γm) (16.59)
where each expert p(y|x,γm)is like a weak learner. We usually ﬁt all Mexperts at once

p(y|x,θ)=
M∑
using EM, but we can imagine a sequential scheme, whereby we only update the parametersfor one expert at a time. In the E step, the posterior responsibilities will reﬂect how well theexisting experts explain a given data point; if this is a poor ﬁt, these data points will havemore inﬂuence on the next expert that is ﬁtted. (This view naturally suggest a way to use aboosting-like algorithm for unsupervised learning: we simply sequentially ﬁt mixture models,instead of mixtures of experts.)
Notice that this is a rather “broken” MLE procedure, since it never goes back to update the

parameters of an old expert. Similarly, if boosting ever wants to change the weight assigned to aweak learner, the only way to do this is to add the weak learner again with a new weight. Thiscan result in unnecessarily large models. By contrast, the BART model (Chipman et al. 2006,2010) uses a Bayesian version of backﬁtting to ﬁt a small sum of weak learners (typically trees).
16.5 Feedforward neural networks (multilayer perceptrons)
Afeedforward neural network,a k a multi-layer perceptron (MLP), is a series of logistic
regression models stacked on top of each other, with the ﬁnal layer being either another logisticregression or a linear regression model, depending on whether we are solving a classiﬁcation orregression problem. For example, if we have two layers, and we are solving a regression problem,the model has the form
p(y|x,θ)=N (y|w
Tz(x),σ2) (16.60)
z(x)=g(Vx)=[g(vT
1x),...,g(vT
Hx)] (16.61)

wheregis a non-linear activation ortransfer function (commonly the logistic function),
z(x)=φ(x,V)is called the hidden layer (a deterministic function of the input), His the

564 Chapter 16. Adaptive basis function models
xn1xnixnD
zn1znjznH
yn1ynkynC
......
vij wjk
Figure 16.11 A neural network with one hidden layer.
number of hidden units, Vis the weight matrix from the inputs to the hidden nodes, and
wis the weight vector from the hidden nodes to the output. It is important that gbe non-
linear, otherwise the whole model collapses into a large linear regression model of the form
y=wT(Vx). One can show that an MLP is a universal approximator, meaning it can model
any suitably smooth function, given enough hidden units, to any desired level of accuracy(Hornik 1991).
To handle binary classiﬁcation, we pass the output through a sigmoid, as in a GLM:
p(y|x,θ)=B e r (y|sigm(w
Tz(x))) (16.62)
We can easily extend the MLP to predict multiple outputs. For example, in the regression case,we have
p(y|x,θ)=N(y|Wφ(x,V),σ
2I) (16.63)
See Figure 16.11 for an illustration. If we add mutual inhibition arcs between the output units,

ensuring that only one of them turns on, we can enforce a sum-to-one constraint, which can beused for multi-class classiﬁcation. The resulting model has the form
p(y|x,θ)=C a t ( y|S(Wz(x)) (16.64)
16.5.1 Convolutional neural networks
The purpose of the hidden units is to learn non-linear combinations of the original inputs; thisis calledfeature extraction orfeature construction. These hidden features are then passed as

input to the ﬁnal GLM. This approach is particularly useful for problems where the original inputfeatures are not very individually informative. For example, each pixel in an image is not veryinformative; it is the combination of pixels that tells us what objects are present. Conversely, fora task such as document classiﬁcation using a bag of words representation, each feature (wordcount) isinformative on its own, so extracting “higher order” features is less important. Not
suprisingly, then, much of the work in neural networks has been motivated by visual pattern

16.5. Feedforward neural networks (multilayer perceptrons) 565
Figure 16.12 The convolutional neural network from (Simard et al. 2003). Source: http://www .codep
roject.com/KB/library/NeuralNetRecognition .aspx. Used with kind permission of Mike O’Neill.
recognition (e.g., (LeCun et al. 1989)), although they have also been applied to other types of
data, including text (e.g., (Collobert and Weston 2008)).
A form of MLP which is particularly well suited to 1d signals like speech or text, or 2d signals
like images, is the convolutional neural network . This is an MLP in which the hidden units
have local receptive ﬁelds (as in the primary visual cortex), and in which the weights are tied
or shared across the image, in order to reduce the number of parameters. Intuitively, the effect
of such spatial parameter tying is that any useful features that are “discovered” in some portion
of the image can be re-used everywhere else without having to be independently learned. The

resulting network then exhibits translation invariance , meaning it can classify patterns no
matter where they occur inside the input image.
Figure 16.12 gives an example of a convolutional network, designed by Simard and colleagues
(Simard et al. 2003), with 5 layers (4 layers of adjustable parameters) designed to classify 29×29
gray-scale images of handwritten digits from the MNIST dataset (see Section 1.2.1.3). In layer 1,
we have 6 feature maps each of which has size 13×13. Each hidden node in one of these

feature maps is computed by convolving the image with a 5×5weight matrix (sometimes called
a kernel), adding a bias, and then passing the result through some form of nonlinearity. There
are therefore 13×13×6 = 1014 neurons in Layer 1, and (5×5+1)×6 = 156weights. (The
"+1" is for the bias.) If we did not share these parameters, there would be 1014×26 = 26,364
weights at the ﬁrst layer. In layer 2, we have 50 feature maps, each of which is obtained by
convolving each feature map in layer 1 with a 5×5weight matrix, adding them up, adding a

bias, and passing through a nonlinearity. There are therefore 5×5×50 = 1250 neurons in
Layer 2,(5×5+1 )×6×50 = 7800 adjustable weights (one kernel for each pair of feature

566 Chapter 16. Adaptive basis function models
maps in layers 1 and 2), and 1250×26 = 32, 500connections. Layer 3 is fully connected to
layer 2, and has 100 neurons and 100×(1250+1) = 125, 100weights. Finally, layer 4 is also
fully connected, and has 10 neurons, and 10×(100 + 1) = 1010 weights. Adding the above
numbers, there are a total of 3,215 neurons, 134,066 adjustable weights, and 184,974 connections.
This model is usually trained using stochastic gradient descent (see Section 16.5.4 for details).
A single pass over the data set is called an epoch. When Mike O’Neill did these experiments in
2006, he found that a single epoch took about 40 minutes (recall that there are 60,000 trainingexamples in MNIST). Since it took about 30 epochs for the error rate to converge, the totaltraining time was about 20 hours.
5Using this technique, he obtained a misclassiﬁcation rate on
the 10,000 test cases of about 1.40%.
To further reduce the error rate, a standard trick is to expand the training set by including

distorted versions of the original data, to encourage the network to be invariant to small changes

that don’t affect the identity of the digit. These can be created by applying a random ﬂow ﬁeldto shift pixels around. See Figure 16.13 for some examples. (If we use online training, such asstochastic gradient descent, we can create these distortions on the ﬂy, rather than having tostore them.) Using this technique, Mike O’Neill obtained a misclassiﬁcation rate on the 10,000test cases of about 0.74%, which is close to the current state of the art.
6
Yann Le Cun and colleagues (LeCun et al. 1998) obtained similar performance using a slightly
more complicated architecture shown in Figure 16.14. This model is known as LeNet5, and

6
historically it came before the model in Figure 16.12. There are two main differences. First,LeNet5 has a subsampling layer between each convolutional layer, which either averages or
computes the max over each small window in the previous layer, in order to reduce the size, andto obtain a small amount of shift invariance. The convolution and sub-sampling combinationwas inspired by Hubel and Wiesel’s model of simple and complex cells in the visual cortex(Hubel and Wiesel 1962), and it continues to be popular in neurally-inspired models of visualobject recognition (Riesenhuber and Poggio 1999). A similar idea ﬁrst appeared in Fukushima’sneocognitron (Fukushima 1975), though no globally supervised training algorithm was available
at that time.

The second difference between LeNet5 and the Simard architecture is that the ﬁnal layer is
actually an RBF network rather than a more standard sigmoidal or softmax layer. This modelgets a test error rate of about 0.95% when trained with no distortions, and 0.8% when trainedwith distortions. Figure 16.15 shows all 82 errors made by the system. Some are genuinelyambiguous, but several are errors that a person would never make. A web-based demo of theLeNet5 can be found at http://yann .lecun.com/exdb/lenet/index .html.
Of course, classifying isolated digits is of limited applicability: in the real world, people usually

write strings of digits or other letters. This requires both segmentation and classiﬁcation. Le Cunand colleagues devised a way to combine convolutional neural networks with a model similarto a conditional random ﬁeld (described in Section 19.6) to solve this problem. The systemwas eventually deployed by the US postal service. (See (LeCun et al. 1998) for a more detailedaccount of the system, which remains one of the best performing systems for this task.)
5. Implementation details: Mike used C++ code and a variety of speedup tricks. He was using standard 2006 era
hardware (an Intel Pentium 4 hyperthreaded processor running at 2.8GHz). See http://www .codeproject .com/KB/
library/NeuralNetRecognition .aspxfor details.
6. A list of various methods, along with their misclassiﬁcation rates on the MNIST test set, is available from http:

//yann.lecun.com/exdb/mnist/ . Error rates within 0.1–0.2% of each other are not statistically signiﬁcantly different.

16.5. Feedforward neural networks (multilayer perceptrons) 567
(a)
 (b)
(c) (d)
(e) (f)
Figure 16.13 Several synthetic warpings of a handwritten digit. Based on Figure 5.14 of (Bishop 2006a).
Figure generated by elasticDistortionsDemo , written by Kevin Swersky.
INPUT
32x32
Convolutions Subsampling ConvolutionsC1: feature maps 
6@28x28
SubsamplingS2: f. maps
6@14x14S4: f. maps 16@5x5
C5: layer
120C3: f. maps 16@10x10
F6: layer
 84
Full connection
Full connectionGaussian connectionsOUTPUT
 10
Figure 16.14 LeNet5, a convolutional neural net for classifying handwritten digits. Source: Figure 2 from
(LeCun et al. 1998) . Used with kind permission of Yann LeCun.

568 Chapter 16. Adaptive basis function models
4−>6
3−>5
8−>2
2−>1
5−>3
4−>8
2−>8
3−>5
6−>5
7−>3
9−>4
8−>0
7−>8
5−>3
8−>7
0−>6
3−>7
2−>7
8−>3
9−>4
8−>2
5−>3
4−>8
3−>9
6−>0
9−>8
4−>9
6−>1
9−>4
9−>1
9−>4
2−>0
6−>1
3−>5
3−>2
9−>5
6−>0
6−>0
6−>0
6−>8
4−>6
7−>3
9−>4
4−>6
2−>7
9−>7
4−>3
9−>4
9−>4
9−>4
8−>7
4−>2
8−>4
3−>5
8−>4
6−>5
8−>5
3−>8
3−>8
9−>8
1−>5
9−>8
6−>3
0−>2
6−>5
9−>5
0−>7
1−>6
4−>9
2−>1
2−>8
8−>5
4−>9
7−>2
7−>2
6−>5
9−>7
6−>1
5−>6
5−>0
4−>9
2−>8
Figure 16.15 These are the 82 errors made by LeNet5 on the 10,000 test cases of MNIST. Below each
image is a label of the form correct-label →estimated-label. Source: Figure 8 of (LeCun et al. 1998).
Used with kind permission of Yann LeCun. (Compare to Figure 28.4(b) which shows the results of a deep
generative model.)
16.5.2 Other kinds of neural networks
Other network topologies are possible besides the ones discussed above. For example, we can
have skip arcs that go directly from the input to the output, skipping the hidden layer; we

can have sparse connections between the layers; etc. However, the MLP always requires that
the weights form a directed acyclic graph. If we allow feedback connections, the model is
known as a recurrent neural network ; this deﬁnes a nonlinear dynamical system, but does
not have a simple probabilistic interpretation. Such RNN models are currently the best approach
for language modeling (i.e., performing word prediction in natural language) (Tomas et al. 2011),
signiﬁcantly outperforming the standard n-gram-based methods discussed in Section 17.2.2.

If we allow symmetric connections between the hidden units, the model is known as a Hop-
ﬁeld network orassociative memory ; its probabilistic counterpart is known as a Boltzmann
machine (see Section 27.7) and can be used for unsupervised learning.
16.5.3 A brief history of the ﬁeld
Neural networks have been the subject of great interest for many decades, due to the desire to
understand the brain, and to build learning machines. It is not possible to review the entire
history here. Instead, we just give a few “edited highlights”.

The ﬁeld is generally viewed as starting with McCulloch and Pitts (McCullich and Pitts 1943),
who devised a simple mathematical model of the neuron in 1943, in which they approximated the

16.5. Feedforwardneuralnetworks(multilayerperceptrons) 569
output as a weighted sum of inputs passed through a threshold function, y=I(∑
iwixi>θ),
for some threshold θ. This is similar to a sigmoidal activation function. Frank Rosenblatt
invented the perceptron learning algorithm in 1957, which is a way to estimate the parameters of
a McCulloch-Pitts neuron (see Section 8.5.4 for details). A very similar model called the adaline
(for adaptive linear element) was invented in 1960 by Widrow and Hoff.
In 1969, Minsky and Papert (Minsky and Papert 1969) published a famous book called “Percep-
trons” in which they showed that such linear models, with no hidden layers, were very limitedin their power, since they cannot classify data that is not linearly separable. This considerablyreduced interest in the ﬁeld.
In 1986, Rumelhart, Hinton and Williams (Rumelhart et al. 1986) discovered the backpropa-

gation algorithm (see Section 16.5.4), which allows one to ﬁt models with hidden layers. (Thebackpropagation algorithm was originally discovered in (Bryson and Ho 1969), and independentlyin (Werbos 1974); however, it was (Rumelhart et al. 1986) that brought the algorithm to people’sattention.) This spawned a decade of intense interest in these models.
In 1987, Sejnowski and Rosenberg (Sejnowski and Rosenberg 1987) created the famous NETtalk

system, that learned a mapping from English words to phonetic symbols which could be fedinto a speech synthesizer. An audio demo of the system as it learns over time can be found athttp://www .cnl.salk. edu/ParallelNetsPronounce/nettalk .mp3. The systems starts by
“babbling” and then gradually learns to pronounce English words. NETtalk learned a distributed
representation (via its hidden layer) of various sounds, and its success spawned a big debate in
psychology between connectionism, based on neural networks, and computationalism, based

on syntactic rules. This debate lives on to some extent in the machine learning community,wheretherearestillargumentsaboutwhetherlearningisbestperformedusinglow-level, “neural-like” representations, or using more structured models.
In 1989, Yann Le Cun and others (LeCun et al. 1989) created the famous LeNet system described
in Section 16.5.1.
In 1992, the support vector machine (see Section 14.5) was invented (Boser et al. 1992). SVMs
provide similar prediction accuracy to neural networks while being considerably easier to train(since they use a convex objective function). This spawned a decade of interest in kernel methodsin general.
7Note, however, that SVMs do not use adaptive basis functions, so they require a fair

amount of human expertise to design the right kernel function.
In 2002, Geoff Hinton invented the contrastive divergence training procedure (Hinton 2002),
which provided a way, for the ﬁrst time, to learn deep networks, by training one layer at a timein an unsupervised fashion (see Section 27.7.2.4 for details). This in turn has spawned renewedinterest in neural networks over the last few years (see Chapter 28).
16.5.4 The backpropagation algorithm

Unlike a GLM, the NLL of an MLP is a non-convex function of its parameters. Nevertheless,we can ﬁnd a locally optimal ML or MAP estimate using standard gradient-based optimizationmethods. Since MLPs have lots of parameters, they are often trained on very large data sets.
7. It became part of the folklore during the 1990s that to get published in the top machine learning conference known as
NIPS, which stands for “neural information processing systems”, it was important to ensure your paper did not contain
the word “neural network”!

570 Chapter16. Adaptivebasisfunctionmodels
−10 −5 0 5 10−1−0.8−0.6−0.4−0.200.20.40.60.81
  
tanh
sigmoid
Figure 16.16 Two possible activation functions. tanhmaps Rto[−1,+1]and is the preferred nonlin-
earity for the hidden nodes. sigmmaps Rto[0,1]and is the preferred nonlinearity for binary nodes at
the output layer. Figure generated by tanhPlot .
Consequently it is common to use ﬁrst-order online methods, such as stochastic gradient descent
(Section 8.5.2), whereas GLMs are usually ﬁt with IRLS, which is a second-order offline method.
We now discuss how to compute the gradient vector of the NLL by applying the chain rule of
calculus. The resulting algorithm is known as backpropagation, for reasons that will become
apparent.
For notational simplicity, we shall assume a model with just one hidden layer. It is helpful
to distinguish the pre- and post-synaptic values of a neuron, that is, before and after we applythe nonlinearity. Let x
nbe then’th input, an=Vxnbe the pre-synaptic hidden layer, and

apparent.
zn=g(an)be the post-synaptic hidden layer, where gis sometransfer function. We typically
useg(a) = sigm( a), but we may also use g(a) = tanh( a): see Figure 16.16 for a comparison.
(When the input to sigmortanhis a vector, we assume it is applied component-wise.)
We now convert this hidden layer to the output layer as follows. Let bn=Wznbe the
pre-synaptic output layer, and ˆyn=h(bn)be the post-synaptic output layer, where his
another nonlinearity, corresponding to the canonical link for the GLM. (We reserve the notationy

n, without the hat, for the output corresponding to the n’th training case.) For a regression
model, we use h(b)=b; for binary classifcation, we use h(b)=[ s i g m ( b1),...,sigm(bc)];f o r
multi-class classiﬁcation, we use h(b)=S(b).
We can write the overall model as follows:
xnV→ang→znW→bnh→ˆyn (16.65)
The parameters of the model are θ=(V,W), the ﬁrst and second layer weight matrices. Offset
or bias terms can be accomodated by clamping an element of xnandznto 1.8

8. In the regression setting, we can easily estimate the variance of the output noise using the empirical variance of the
residual errors, ˆσ2=1
N||ˆy(ˆθ)−y||2, after training is complete. There will be one value of σ2for each output node,
if we are performing multi-target regression, as we usually assume.

16.5. Feedforwardneuralnetworks(multilayerperceptrons) 571
In the regression case, with Koutputs, the NLL is given by the squared error:
J(θ)=−∑
n∑
k(ˆynk(θ)−ynk)2(16.66)
In the classiﬁcation case, with Kclasses, the NLL is given by the cross entropy
J(θ)=−∑
n∑
kynklogˆynk(θ) (16.67)
Our task is to compute ∇θJ. We will derive this for each nseparately; the overall gradient is
obtained by summing over n, although often we just use a mini-batch (see Section 8.5.2).
Let us start by considering the output layer weights. We have
∇wkJn=∂Jn
∂bnk∇wkbnk=∂Jn
∂bnkzn (16.68)
sincebnk=wT
kzn. Assuming his the canonical link function for the output GLM, then
Equation 9.91 tells us that
∂Jn
∂bnk≜δw
nk=(ˆynk−ynk) (16.69)
which is the error signal. So the overall gradient is
∇wkJn=δw
nkzn (16.70)
which is the pre-synaptic input to the output layer, namely zn, times the error signal, namely
δw
nk.
For the input layer weights, we have
∇vjJn=∂Jn
∂anj∇vjanj≜δv
njxn (16.71)
where we exploited the fact that anj=vT

∇wkJn=∂Jn
∂Jn
∂bnk≜δw
∇wkJn=δw
δw
nk.
∇vjJn=∂Jn
jxn. All that remains is to compute the ﬁrst level
error signal δv
nj.W eh a v e
δv
nj=∂Jn
∂anj=K∑
k=1∂Jn
∂bnk∂bnk
∂anj=K∑
k=1δw
nk∂bnk
∂anj(16.72)
Now
bnk=∑
jwkjg(anj) (16.73)
so
∂bnk
∂anj=wkjg′(anj) (16.74)
whereg′(a)=d
dag(a). For tanh units, g′(a)=d
datanh(a)=1−tanh2(a)=sech2(a), and
for sigmoid units, g′(a)=d
daσ(a)=σ(a)(1−σ(a)). Hence
δv
nj=K∑
k=1δw
nkwkjg′(anj) (16.75)

572 Chapter16. Adaptivebasisfunctionmodels
Thus the layer 1 errors can be computed by passing the layer 2 errors back through the Wmatrix;
hencetheterm“backpropagation”. Thekeypropertyisthatwecancomputethegradientslocally:
each node only needs to know about its immediate neighbors. This is supposed to make thealgorithm “neurally plausible”, although this interpretation is somewhat controversial.
Putting it all together, we can compute all the gradients as follows: we ﬁrst perform a
forwards pass to compute a
n,zn,bnandˆyn. We then compute the error for the output layer,
δ(2)
n=ˆyn−yn, which we pass backwards through Wusing Equation 16.75 to compute the
error for the hidden layer, δ(1)n. We then compute the overall gradient as follows:
∇θJ(θ)=∑
n[δvnxn,δwnzn] (16.76)
16.5.5 Identiﬁability
It is easy to see that the parameters of a neural network are not identiﬁable. For example, we

δ(2)
∇θJ(θ)=∑
can change the sign of the weights going into one of the hidden units, so long as we changethe sign of all the weights going out of it; these effects cancel, since tanh is an odd function, sotanh(−a)=−tanh(a). There will be Hsuch sign ﬂip symmetries, leading to 2
Hequivalent
settings of the parameters. Similarly, we can change the identity of the hidden units withoutaffecting the likelihood. There are H!such permutations. The total number of equivalent
parameter settings (with the same likelihood) is therefore H!2
H.

H.
In addition, there may be local minima due to the non-convexity of the NLL. This can
be a more serious problem, although with enough data, these local optima are often quite“shallow”, and simple stochastic optimization methods can avoid them. In addition, it is commonto perform multiple restarts, and to pick the best solution, or to average over the resultingpredictions. (It does not make sense to average the parameters themselves, since they are notidentiﬁable.)
16.5.6 Regularization
As usual, the MLE can overﬁt, especially if the number of nodes is large. A simple way to preventthis is called early stopping, which means stopping the training procedure when the error on

the validation set ﬁrst starts to increase. This method works because we usually initialize fromsmall random weights, so the model is initially simple (since the tanhandsigmfunctions are
nearly linear near the origin). As training progresses, the weights become larger, and the modelbecomes nonlinear. Eventually it will overﬁt.
Another way to prevent overﬁtting, that is more in keeping with the approaches used elsewhere
in this book, is to impose a prior on the parameters, and then use MAP estimation. It is standardto use aN(0,α
−1I)prior (equivalent to ℓ2regularization), where αis the precision (strength)
of the prior. In the neural networks literature, this is called weight decay, since it encourages
small weights, and hence simpler models. The penalized NLL objective becomes
J(θ)=−N∑

J(θ)=−N∑
n=1logp(yn|xn,θ)+α
2[∑
ijv2
ij+∑
jkw2
jk] (16.77)

16.5. Feedforwardneuralnetworks(multilayerperceptrons) 573
(Note that we don’t penalize the bias terms.) The gradient of the modiﬁed objective becomes
∇θJ(θ)=[∑
nδv
nxn+αv,∑
nδwnzn+αw] (16.78)
as in Section 8.3.6.
If the regularization is sufficiently strong, it does not matter if we have too many hidden units
(apart from wasted computation). Hence it is advisable to set Hto be as large as you can afford
(say 10–100), and then to choose an appropriate regularizer. We can set the αparameter by
cross validation or empirical Bayes (see Section 16.5.7.5).
As with ridge regression, it is good practice to standardize the inputs to zero mean and unit
variance, so that the spherical Gaussian prior makes sense.
16.5.6.1 Consistent Gaussian priors *
One can show (MacKay 1992) that using the same regularization parameter for both the ﬁrst and

second layer weights results in the lack of a certain desirable invariance property. In particular,suppose we linearly scale and shift the inputs and/or outputs to a neural network regressionmodel. Then we would like the model to learn to predict the same function, by suitably scalingits internal weights and bias terms. However, the amount of scaling needed by the ﬁrst andsecond layer weights to compensate for a change in the inputs and/or outputs is not the same.Therefore we need to use a different regularization strength for the ﬁrst and second layer.Fortunately, this is easy to do — we just use the following prior:
p(θ)=N(W|0,1
αwI)N(V|0,1
αvI)N(b|0,1
αbI)N(c|0,1

αcI) (16.79)
wherebandcare the bias terms.9
To get a feeling for the effect of these hyper-parameters, we can sample MLP parameters
from this prior and plot the resulting random functions. Figure 16.17 shows some examples.Decreasing α
vallows the ﬁrst layer weights to get bigger, making the sigmoid-like shape of
the functions steeper. Decreasing αballows the ﬁrst layer biases to get bigger, which allows
the center of the sigmoid to shift left and right more. Decreasing αwallows the second layer
weights to get bigger, making the functions more “wiggly” (greater sensitivity to change in theinput, and hence larger dynamic range). And decreasing α
callows the second layer biases to
get bigger, allowing the mean level of the function to move up and down more. (In Chapter 15,we will see an easier way to deﬁne priors over functions.)
16.5.6.2 Weight pruning

Since there are many weights in a neural network, it is often helpful to encourage sparsity.Various ad-hoc methods for doing this, with names such as “optimal brain damage”, weredevised in the 1990s; see e.g., (Bishop 1995) for details.
9. Since we are regularizing the output bias terms, it is helpful, in the case of regression, to normalize the target
responses in the training set to zero mean, to be consistent with the fact that the prior on the output bias has zero
mean.

574 Chapter16. Adaptivebasisfunctionmodels
−1 −0.5 0 0.5 1−10−8−6−4−20246810aw1=0.010, ab1=0.100, aw2=1.000, ab2=1.000
(a)−1 −0.5 0 0.5 1−10−8−6−4−20246810aw1=0.001, ab1=0.100, aw2=1.000, ab2=1.000
(b)
−1 −0.5 0 0.5 1−10−8−6−4−20246810aw1=0.010, ab1=0.010, aw2=1.000, ab2=1.000
(c)−1 −0.5 0 0.5 1−10−8−6−4−20246810aw1=0.010, ab1=0.100, aw2=0.100, ab2=1.000
(d)
−1 −0.5 0 0.5 1−10−8−6−4−20246810aw1=0.010, ab1=0.100, aw2=1.000, ab2=0.100
(e)
Figure 16.17 The effects of changing the hyper-parameters on an MLP. (a) Default parameter values
αv=0.01,αb=0.1,αw=1,αc=1. (b) Decreasing αvby factor of 10. (c) Decreasing αbby
factor of 10. (d) Decreasing αwby factor of 10. (e) Decreasing αcby factor of 10. Figure generated by
mlpPriorsDemo .

16.5. Feedforwardneuralnetworks(multilayerperceptrons) 575
Neural Network
y
h44 h43h42 h41h40
h34 h33h31h30
h24 h23 h21h20
h14 h13h22h32
h12 h11h10
x1x0
(a)−5 0 5−4−3−2−101234
  
Data
Deep Neural Net
(b)
Figure 16.18 (a) A deep but sparse neural network. The connections are pruned using ℓ1regularization.
At each level, nodes numbered 0 are clamped to 1, so their outgoing weights correspond to the offset/bias
terms. (b) Predictions made by the model on the training set. Figure generated by sparseNnetDemo ,
written by Mark Schmidt.
However, we can also use the more principled sparsity-promoting techniques we discussed in
Chapter 13. One approach is to use an ℓ1regularizer. See Figure 16.18 for an example. Another
approach is to use ARD; this is discussed in more detail in Section 16.5.7.5.
16.5.6.3 Soft weight sharing*
Another way to regularize the parameters is to encourage similar weights to share statistical

strength. But how do we know which parameters to group together? We can learn this, by usinga mixture model. That is, we model p(θ)as a mixture of (diagonal) Gaussians. Parameters that
are assigned to the same cluster will share the same mean and variance and thus will havesimilar values (assuming the variance for that cluster is low). This is called soft weight sharing
(Nowlan and Hinton 1992). In practice, this technique is not widely used. See e.g., (Bishop 2006a,p271) if you want to know the details.

16.5.6.4 Semi-supervised embedding *

An interesting way to regularize “deep” feedforward neural networks is to encourage the hiddenlayers to assign similar objects to similar representations. This is useful because it is often easyto obtain “side” information consisting of sets of pairs of similar and dissimilar objects. Forexample, in a video classiﬁcation task, neighboring frames can be deemed similar, but framesthat are distant in time can be deemed dis-similar (Mobahi et al. 2009). Note that this can bedone without collecting any labels.
LetS
ij=1if examples iandjare similar, and Sij=0otherwise. Let f(xi)be some

LetS
embedding of item xi, e.g.,f(xi)=z(xi,θ),w h e r ezis the hidden layer of a neural network.
Now deﬁne a loss function L(f(xi),f(xj),Sij)that depends on the embedding of two objects,

576 Chapter16. Adaptivebasisfunctionmodels
and the observed similarity measure. For example, we might want to force similar objects to
have similar embeddings, and to force the embeddings of dissimilar objects to be a minimaldistance apart:
L(f
i,fj,Sij)={
||fi−fj||2ifSij=1
max(0,m −||fi−fj||2)ifSij=0(16.80)
wheremis some minimal margin. We can now deﬁne an augmented loss function for training
the neural network:
∑
i∈LNLL(f(xi),yi)+λ∑
i,j∈UL(f(xi),f(xj),Sij) (16.81)
whereLis the labeled training set, Uis the unlabeled training set, and λ≥0is some tradeoff
parameter. This is called semi-supervised embedding (Weston et al. 2008).
Such an objective can be easily optimized by stochastic gradient descent. At each itera-
tion, pick a random labeled training example, (xn,yn), and take a gradient step to optimize
NLL(f(xi),yi). Then pick a random pair of similar unlabeled examples xi,xj(these can
sometimes be generated on the ﬂy rather than stored in advance), and make a gradient step tooptimize λL(f(x

i),f(xj),1), Finally, pick a random unlabeled example xk, which with high
probability is dissimilar to xi, and make a gradient step to optimize λL(f(xi),f(xk),0).
Note that this technique is effective because it can leverage massive amounts of data. In
a related approach, (Collobert and Weston 2008) trained a neural network to distinguish validEnglish sentences from invalid ones. This was done by taking all 631 million words from EnglishWikipedia (en .wikipedia .org), and then creating windows of length 11 containing neighboring

words. This constitutes the positive examples. To create negative examples, the middle word ofeach window was replaced by a random English word (this is likely to be an “invalid” sentence— either grammatically and/or semantically — with high probability). This neural network wasthen trained over the course of 1 week, and its latent representation was then used as the inputto a supervised semantic role labeling task, for which very little labeled training data is available.(See also (Ando and Zhang 2005) for related work.)
16.5.7 Bayesian inference *

Although MAP estimation is a succesful way to reduce overﬁtting, there are still some goodreasons to want to adopt a fully Bayesian approach to “ﬁtting” neural networks:
• Integrating out the parameters instead of optimizing them is a much stronger form of regu-
larization than MAP estimation.
• We can use Bayesian model selection to determine things like the hyper-parameter settings
and the number of hidden units. This is likely to be much faster than cross validation,
especially if we have many hyper-parameters (e.g., as in ARD).
• Modelling uncertainty in the parameters will induce uncertainty in our predictive distribu-
tions, which is important for certain problems such as active learning and risk-averse decisionmaking.

16.5. Feedforwardneuralnetworks(multilayerperceptrons) 577
• We can use online inference methods, such as the extended Kalman ﬁlter, to do online
learning (Haykin 2001).
One can adopt a variety of approximate Bayesian inference techniques in this context. In this
section, we discuss the Laplace approximation, ﬁrst suggested in (MacKay 1992, 1995b). One can
also use hybrid Monte Carlo (Neal 1996), or variational Bayes (Hinton and Camp 1993; Barberand Bishop 1998).
16.5.7.1 Parameter posterior for regression
We start by considering regression, following the presentation of (Bishop 2006a, sec 5.7), whichsummarizes the work of (MacKay 1992, 1995b). We will use a prior of the form p(w)=
N(w|0,(1/α)I),w h e r ewrepresents all the weights combined. We will denote the precision
of the noise by β=1/σ
2.
The posterior can be approximated as follows:
p(w|D,α,β)∝exp(−E(w)) (16.82)
E(w)≜βED(w)+αE W(w) (16.83)
ED(w)≜1
2N∑
n=1(yn−f(xn,w))2(16.84)
EW(w)≜1
2wTw (16.85)

2.
ED(w)≜1
2N∑
EW(w)≜1
whereEDis the data error, EWis the prior error, and Eis the overall error (negative log
prior plus log likelihood). Now let us make a second-order Taylor series approximation of E(w)
around its minimum (the MAP estimate)
E(w)≈E(wMP)+1
2(w−wMP)TA(w−wMP) (16.86)
whereAis the Hessian of E:
A=∇∇E(wMP)=βH+αI (16.87)
whereH=∇∇ED(wMP)is the Hessian of the data error. This can be computed exactly
inO(d2)time, where dis the number of parameters, using a variant of backpropagation (see

EW(w)≜1
(Bishop 2006a, sec 5.4) for details). Alternatively, if we use a quasi-Newton method to ﬁndthe mode, we can use its internally computed (low-rank) approximation to H. (Note that
diagonal approximations of Hare usually very inaccurate.) In either case, using this quadratic
approximation, the posterior becomes Gaussian:
p(w|α,β,D)≈N(w|w
MP,A−1) (16.88)

578 Chapter16. Adaptivebasisfunctionmodels
16.5.7.2 Parameter posterior for classiﬁcation
The classiﬁcation case is the same as the regression case, except β=1andEDis a cross-
entropy error of the form
ED(w)≜N∑
n=1[ynlnf(xn,w)+(1−yn)lnf(xn,w)] (16.89)
(16.90)
16.5.7.3 Predictive posterior for regression
The posterior predictive density is given by
p(y|x,D,α,β)=∫
N(y|f(x,w),1/β)N(w|wMP,A−1)dw (16.91)
This is not analytically tractable because of the nonlinearity of f(x,w). Let us therefore
construct a ﬁrst-order Taylor series approximation around the mode:
f(x,w)≈f(x,wMP)+gT(w−wMP) (16.92)
where
g=∇wf(x,w)|w=wMP (16.93)
We now have a linear-Gaussian model with a Gaussian prior on the weights. From Equation 4.126
we have
p(y|x,D,α,β)≈N(y|f(x,wMP),σ2(x)) (16.94)
where the predictive variance depends on the input xas follows:
σ2(x)=β−1+gTA−1g (16.95)
The error bars will be larger in regions of input space where we have little training data. SeeFigure 16.19 for an example.

where
we have
16.5.7.4 Predictive posterior for classiﬁcation
In this section, we discuss how to approximate p(y|x,D)in the case of binary classiﬁcation.
The situation is similar to the case of logistic regression, discussed in Section 8.4.4, except inaddition the posterior predictive mean is a non-linear function of w. Speciﬁcally, we have
μ=E[y|x,w]=s i g m ( a(x,w)),w h e r e a(x,w)is the pre-synaptic output of the ﬁnal layer.
Let us make a linear approximation to this:
a(x,w)≈a
MP(x)+gT(w−wMP) (16.96)

a(x,w)≈a
whereaMP(x)=a(x,wMP)andg=∇xa(x,wMP)can be found by a modiﬁed version of
backpropagation. Clearly
p(a|x,D)≈N(a(x,wMP),g(x)TA−1g(x)) (16.97)

16.5. Feedforwardneuralnetworks(multilayerperceptrons) 579
0 0.2 0.4 0.6 0.8 1−1.5−1−0.500.511.5
InputTarget
  
data
function
network
error bars
(a)0 0.2 0.4 0.6 0.8 1−1.5−1−0.500.511.5
  
Data
Function
Prediction
Samples
(b)
Figure 16.19 The posterior predictive density for an MLP with 3 hidden nodes, trained on 16 data points.
The dashed green line is the true function. (a) Result of using a Laplace approximation, after performing
empirical Bayes to optimize the hyperparameters. The solid red line is the posterior mean prediction,and the dotted blue lines are 1 standard deviation above and below the mean. Figure generated bymlpRegEvidenceDemo . (b) Result of using hybrid Monte Carlo, using the same trained hyperparameters
as in (a). The solid red line is the posterior mean prediction, and the dotted blue lines are samples fromthe posterior predictive. Figure generated by mlpRegHmcDemo , written by Ian Nabney.
Hence the posterior predictive for the output is
p(y=1|x,D)=∫
sigm(a)p(a|x,D)da≈sigm(κ(σ2

a)bTwMP) (16.98)
whereκis deﬁned by Equation 8.70, which we repeat here for convenience:
2 (16.99)
Of course, a simpler (and potentially more accurate) alternative to this is to draw a few samples
from the Gaussian posterior and to approximate the posterior predictive using Monte Carlo.
In either case, the effect of taking uncertainty of the parameters into account, as in Sec-
tion 8.4.4, is to “moderate” the conﬁdence of the output; the decision boundary itself is unaf-
fected, however.
16.5.7.5 ARD for neural networks
Once we have made the Laplace approximation to the posterior, we can optimize the marginallikelihood wrt the hyper-parameters αusing the same ﬁxed-point equations as in Section 13.7.4.2.

Typically we use one hyper-parameter for the weight vector leaving each node, to achieve aneffect similar to group lasso (Section 13.5.1). That is, the prior has the form
p(θ)=
D∏
i=1N(v:,i|0,1
αv,iI)H∏
j=1N(w:,j|0,1
αw,jI) (16.100)
If we ﬁnd αv,i=∞, then input feature iis irrelevant, and its weight vector v:,iis pruned out.
Similarly, if we ﬁnd αw,j=∞, then hidden feature jis irrelevant. This is known as automatic

580 Chapter16. Adaptivebasisfunctionmodels
relevancy determination or ARD, which was discussed in detail in Section 13.7. Applying this to
neural networks gives us an efficient means of variable selection in non-linear models.
The software package NETLABcontains a simple example of ARD applied to a neural network,
called demard. This demo creates some data according to a nonlinear regression function
f(x1,x2,x3)=s i n ( 2 πx1)+ϵ,w h e r ex2is a noisy copy of x1. We see that x2andx3are
irrelevant for predicting the target. However, x2is correlated with x1, which is relevant. Using
ARD, the ﬁnal hyper-parameters are as follows:
α=[ 0.2,21.4,249001.8] (16.101)
This clearly indicates that feature 3 is irrelevant, feature 2 is only weakly relevant, and feature 1is very relevant.
16.6 Ensemble learning
Ensemble learning refers to learning a weighted combination of base models of the form
f(y|x,π)=∑
m∈Mwmfm(y|x) (16.102)
where the wmare tunable parameters. Ensemble learning is sometimes called a committee

method, since each base model fmgets a weighted “vote”.
Clearly ensemble learning is closely related to learning adaptive-basis function models. In
fact, one can argue that a neural net is an ensemble method, where fmrepresents the m’th
hidden unit, and wmare the output layer weights. Also, we can think of boosting as kind of
ensemble learning, where the weights on the base models are determined sequentially. Belowwe describe some other forms of ensemble learning.
16.6.1 Stacking
An obvious way to estimate the weights in Equation 16.102 is to use
ˆw=a r g m i n
wN∑
i=1L(yi,M∑

ˆw=a r g m i n
wN∑
m=1wmfm(x)) (16.103)
However, this will result in overﬁtting, with wmbeing large for the most complex model. A
simple solution to this is to use cross-validation. In particular, we can use the LOOCV estimate
ˆw=a r g m i n
wN∑
m=1wmˆf−i
m(x)) (16.104)
whereˆf−i
m(x)is the predictor obtained by training on data excluding (xi,yi). This is known
asstacking, which stands for “stacked generalization” (Wolpert 1992). This technique is more

ˆw=a r g m i n
wN∑
ˆw=a r g m i n
wN∑
m=1wmˆf−i
whereˆf−i
robust to the case where the “true” model is not in the model class than standard BMA (Clarke2003). This approach was used by the Netﬂix team known as “The Ensemble”, which tied thesubmission of the winning team (BellKor’s Pragmatic Chaos) in terms of accuracy (Sill et al.2009). Stacking has also been used for problems such as image segmentation and labeling.

16.6. Ensemblelearning 581
ClassC1C2C3C4C5C6···C15
0 110000 ···1
1 001111 ···0
...
90 1 1 1 0 0 ···0
Table 16.2 Part of a 15-bit error-correcting output code for a 10-class problem. Each row deﬁnes a
two-class problem. Based on Table 16.1 of (Hastie et al. 2009).
16.6.2 Error-correcting output codes
An interesting form of ensemble learning is known as error-correcting output codes orECOC
(Dietterich and Bakiri 1995), which can be used in the context of multi-class classiﬁcation. The
idea is that we are trying to decode a symbol (namely the class label) which has Cpossible
states. We could use a bit vector of length B=⌈log2C⌉to encode the class label, and train
Bseparate binary classiﬁers to predict each bit. However, by using more bits, and by designing

the codewords to have maximal Hamming distance from each other, we get a method that ismore resistant to individual bit-ﬂipping errors (misclassiﬁcation). For example, in Table 16.2, weuseB=1 5bits to encode a C=1 0class problem. The minimum Hamming distance between
any pair of rows is 7. The decoding rule is
ˆc(x)=m i n
cB∑
b=1|Ccb−ˆpb(x)| (16.105)
whereCcbis theb’th bit of the codeword for class c. (James and Hastie 1998) showed that a

ˆc(x)=m i n
cB∑
random code worked just as well as the optimal code: both methods work by averaging theresults of multiple classiﬁers, thereby reducing variance.
16.6.3 Ensemble learning is not equivalent to Bayes model averaging
In Section 5.3, we discussed Bayesian model selection. An alternative to picking the best model,and then using this to make predictions, is to make a weighted average of the predictions madeby each model, i.e., we compute
p(y|x,D)=∑
m∈Mp(y|x,m,D)p(m|D) (16.106)
This is called Bayes model averaging (BMA), and can sometimes give better performance than

using any single model (Hoeting et al. 1999). Of course, averaging over all models is typicallycomputationally infeasible (analytical integration is obviously not possible in a discrete space,although one can sometimes use dynamic programming to perform the computation exactly,e.g., (Meila and Jaakkola 2006)). A simple approximation is to sample a few models from theposterior. An even simpler approximation (and the one most widely used in practice) is to justuse the MAP model.
It is important to note that BMA is not equivalent to ensemble learning (Minka 2000c). This

latter technique corresponds to enlarging the model space, by deﬁning a single new model

582 Chapter16. Adaptivebasisfunctionmodels
model 1st2nd3rd4th5th6th7th8th9th10th
bst-dt 0.580 0.228 0.160 0.023 0.009 0.000 0.000 0.000 0.000 0.000
rf 0.390 0.525 0.084 0.001 0.000 0.000 0.000 0.000 0.000 0.000
bag-dt 0.030 0.232 0.571 0.150 0.017 0.000 0.000 0.000 0.000 0.000
svm 0.000 0.008 0.148 0.574 0.240 0.029 0.001 0.000 0.000 0.000
ann 0.000 0.007 0.035 0.230 0.606 0.122 0.000 0.000 0.000 0.000
knn 0.000 0.000 0.000 0.009 0.114 0.592 0.245 0.038 0.002 0.000
bst-stmp 0.000 0.000 0.002 0.013 0.014 0.257 0.710 0.004 0.000 0.000
dt 0.000 0.000 0.000 0.000 0.000 0.000 0.004 0.616 0.291 0.089
logreg 0.000 0.000 0.000 0.000 0.000 0.000 0.040 0.312 0.423 0.225
nb 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.030 0.284 0.686
Table 16.3 Fraction of time each method achieved a speciﬁed rank, when sorting by mean performance
across 11 datasets and 8 metrics. Based on Table 4 of (Caruana and Niculescu-Mizil 2006). Used with kind
permission of Alexandru Niculescu-Mizil.

which is a convex combination of base models, as follows:
p(y|x,π)=∑
m∈Mπmp(y|x,m) (16.107)
In principle, we can now perform Bayesian inference to compute p(π|D); we then make pre-
dictions using p(y|x,D)=∫
p(y|x,π)p(π|D)dπ. However, it is much more common to use
point estimation methods for π,a sw es a wa b o v e .
16.7 Experimental comparison
We have described many different methods for classiﬁcation and regression. Which one should

you use? That depends on which inductive bias you think is most appropriate for your domain.Usually this is hard to assess, so it is common to just try several different methods, andsee how they perform empirically. Below we summarize two such comparisons that werecarefully conducted (although the data sets that were used are relatively small). See the websitemlcomp.orgfor a distributed way to perform large scale comparisons of this kind. Of course,

we must always remember the no free lunch theorem (Section 1.4.9), which tells us that there isno universally best learning method.
16.7.1 Low-dimensional features
In 2006, Rich Caruana and Alex Niculescu-Mizil (Caruana and Niculescu-Mizil 2006) conducteda very extensive experimental comparison of 10 different binary classiﬁcation methods, on 11different data sets. The 11 data sets all had 5000 training cases, and had test sets containing∼10,000examples on average. The number of features ranged from 9 to 200, so this is much

lower dimensional than the NIPS 2003 feature selection challenge. 5-fold cross validation wasused to assess average test error. (This is separate from any internal CV a method may need touse for model selection.)

16.7. Experimentalcomparison 583
The methods they compared are as follows (listed in roughly decreasing order of performance,
as assessed by Table 16.3):
• BST-DT: boosted decision trees
• RF: random forest• BAG-DT: bagged decision trees• SVM: support vector machine• ANN: artiﬁcial neural network• KNN: K-nearest neighbors• BST-STMP: boosted stumps• DT: decision tree• LOGREG: logistic regression• NB: naive Bayes
They used 8 different performance measures, which can be divided into three groups. Thresh-
old metrics just require a point estimate as output. These include accuracy, F-score (Sec-
tion 5.7.2.3), etc. Ordering/ ranking metrics measure how well positive cases are ordered beforethe negative cases. These include area under the ROC curve (Section 5.7.2.1), average precision,and the precision/recall break even point. Finally, the probability metrics included cross-entropy(log-loss) and squared error, (y−ˆp)
2. Methods such as SVMs that do not produce calibrated

probabilities were post-processed using Platt’s logistic regression trick (Section 14.5.2.3), or usingisotonic regression. Performance measures were standardized to a 0:1 scale so they could becompared.
Obviously the results vary by dataset and by metric. Therefore just averaging the performance

does not necessarily give reliable conclusions. However, one can perform a bootstrap analysis,which shows how robust the conclusions are to such changes. The results are shown inTable 16.3. We see that most of the time, boosted decision trees are the best method, followedby random forests, bagged decision trees, SVMs and neural networks. However, the followingmethods all did relatively poorly: KNN, stumps, single decision trees, logistic regression andnaive Bayes.
These results are generally consistent with conventional wisdom of practioners in the ﬁeld.

Of course, the conclusions may change if there the features are high dimensional and/ or thereare lots of irrelevant features (as in Section 16.7.2), or if there is lots of noise, etc.
16.7.2 High-dimensional features
In 2003, the NIPS conference ran a competition where the goal was to solve binary classiﬁcationproblems with large numbers of (mostly irrelevant) features, given small training sets. (Thiswas called a “feature selection” challenge, but performance was measured in terms of predictiveaccuracy, not in terms of the ability to select features.) The ﬁve datasets that were used aresummarized in Table 16.4. The term proberefers to artiﬁcal variables that were added to the
problem to make it harder. These have no predictive power, but are correlated with the originalfeatures.
Results of the competition are discussed in (Guyon et al. 2006). The overall winner was an

approach based on Bayesian neural networks (Neal and Zhang 2006). In a follow-up study

584 Chapter16. Adaptivebasisfunctionmodels
Dataset Domain Type D %probes NtrainNvalNtest
Aracene Mass spectrometry Dense 10,000 30 100 100 700
Dexter Text classiﬁcation Sparse 20,000 50 300 300 2000Dorothea Drug discovery Sparse 100,000 50 800 350 800Gisette Digit recognition Dense 5000 30 6000 1000 6500Madelon Artiﬁcial Dense 500 96 2000 600 1800
Table 16.4 Summary of the data used in the NIPS 2003 “feature selection” challenge. For the Dorothea
datasets, the features are binary. For the others, the features are real-valued.
Screened features ARD
Method Avg rank Avg time Avg rank Avg time
HMC MLP 1.5 384 (138) 1.6 600 (186)
Boosted MLP 3.8 9.4 (8.6) 2.2 35.6 (33.5)
Bagged MLP 3.6 3.5 (1.1) 4.0 6.4 (4.4)
Boosted trees 3.4 3.03 (2.5) 4.0 34.1 (32.4)
Random forests 2.7 1.9 (1.7) 3.2 11.2 (9.3)
Table 16.5 Performance of different methods on the NIPS 2003 “feature selection” challenge. (HMC
stands for hybrid Monte Carlo; see Section 24.5.4.) We report the average rank (lower is better) across the

5 datasets. We also report the average training time in minutes (standard error in brackets). The MCMCand bagged MLPs use two hidden layers of 20 and 8 units. The boosted MLPs use one hidden layer with 2or 4 hidden units. The boosted trees used depths between 2 and 9, and shrinkage between 0.001 and 0.1.Each tree was trained on 80% of the data chosen at random at each step (so-called stochastic gradient
boosting). From Table 11.3 of (Hastie et al. 2009).

(Johnson 2009), Bayesian neural nets (MLPs with 2 hidden layers) were compared to several other
methods based on bagging and boosting. Note that all of these methods are quite similar: ineach case, the prediction has the form
ˆf(x
∗)=M∑
m=1wmE[y|x∗,θm] (16.108)
The Bayesian MLP was ﬁt by MCMC (hybrid Monte Carlo), so we set wm=1/Mand setθm
to a draw from the posterior. In bagging, we set wm=1/Mandθmis estimated by ﬁtting
the model to a bootstrap sample from the data. In boosting, we set wm=1and theθmare
estimated sequentially.

ˆf(x
∗)=M∑
To improve computational and statistical performance, some feature selection was performed.
Two methods were considered: simple uni-variate screening using T-tests, and a method basedon MLP+ARD. Results of this follow-up study are shown in Table 16.5. We see that Bayesian MLPsare again the winner. In second place are either random forests or boosted MLPs, dependingon the preprocessing. However, it is not clear how statistically signiﬁcant these differences are,since the test sets are relatively small.

In terms of training time, we see that MCMC is much slower than the other methods. It would
be interesting to see how well deterministic Bayesian inference (e.g., Laplace approximation)would perform. (Obviously it will be much faster, but the question is: how much would one lose

16.8. Interpretingblack-boxmodels 585
0.2 0.4 0.6 0.88 1 01 21 41 61 82 0
x1partial dependence
x20.2 0.4 0.6 0.88 1 01 21 41 61 82 0
x30.2 0.4 0.6 0.8 1.08 1 01 21 41 61 82 0
x40.2 0.4 0.6 0.88 1 01 21 41 61 82 0
x5
x6partial dependence
0.2 0.4 0.6 0.8 1.08 1 01 21 41 61 82 0
x70.2 0.4 0.6 0.88 1 01 21 41 61 82 0
x80.2 0.4 0.6 0.88 1 01 21 41 61 82 0
x90.2 0.4 0.6 0.88 1 01 21 41 61 82 0
x10
Figure 16.20 Partial dependence plots for the 10 predictors in Friedman’s synthetic 5-dimensional re-
gression problem. Source: Figure 4 of (Chipman et al. 2010) . Used with kind permission of Hugh
Chipman.
in statistical performance?)
16.8 Interpreting black-box models
Linear models are popular in part because they are easy to interpet. However, they often are

x10
Chipman.
poor predictors, which makes them a poor proxy for “nature’s mechanism”. Thus any conclusionsabout the importance of particular variables should only be based on models that have goodpredictive accuracy (Breiman 2001b). (Interestingly, many standard statistical tests of “goodnessof ﬁt” do not test the predictive accuracy of a model.)
In this chapter, we studied black-box models, which do have good predictive accuracy.

Unfortunately, they are hard to interpret directly. Fortunately, there are various heuristics we canuse to “probe” such models, in order to assess which input variables are the most important.
As a simple example, consider the following non-linear function, ﬁrst proposed (Friedman
1991) to illustrate the power of MARS:
f(x)=1 0s i n ( πx
1x2)+20(x3−0.5)2+10x4+5x5+ϵ (16.109)
whereϵ∼N(0,1). We see that the output is a complex function of the inputs. By augmenting
thexvector with additional irrelevant random variables, all drawn uniform on [0,1], we can

create a challenging feature selection problem. In the experiments below, we add 5 extra dummyvariables.

586 Chapter16. Adaptivebasisfunctionmodels
1111
1
1 1 1 1 1
2468 1 00.00 0.05 0.10 0.15 0.20 0.25usage2
222
2
2 2 2 2 23333
3
3 33 3 34
444
4
4444 45555
5
5 5 555
Figure 16.21 Average usage of each variable in a BART model ﬁt to data where only the ﬁrst 5 features are
relevant. The different coloured lines correspond to different numbers of trees in the ensemble. Source:
Figure 5 of (Chipman et al. 2010) . Used with kind permission of Hugh Chipman.
One useful way to measure the effect of a set sof variables on the output is to compute a
partial dependence plot (Friedman 2001). This is a plot of f(xs)vsxs,w h e r ef(xs)is deﬁned
as the response to xswith the other predictors averaged out:
f(xs)=1
NN∑
i=1f(xs,xi,−s) (16.110)
Figure 16.20 shows an example where we use sets corresponding to each single variable. The data

f(xs)=1
NN∑
was generated from Equation 16.109, with 5 irrelevant variables added. We then ﬁt a BART model(Section 16.2.5) and computed the partial dependence plots. We see that the predicted responseis invariant for s∈{6,...,10}, indicating that these variables are (marginally) irrelevant. The
response is roughly linear in x
4andx5, and roughly quadratic in x3. (The error bars are obtained
by computing empirical quantiles of f(x,θ)based on posterior samples of θ; alternatively, we
can use bootstrap.)

Another very useful summary computes the relative importance of predictor variables.
This can be thought of as a nonlinear, or even “model free”, way of performing variable selection,although the technique is restricted to ensembles of trees. The basic idea, originally proposedin (Breiman et al. 1984), is to count how often variable jis used as a node in any of the trees.
In particular, let v
j=1
M∑M
m=1I(j∈Tm)be the proportion of all splitting rules that use xj,

j=1
M∑M
whereTmis them’th tree. If we can sample the posterior of trees, p(T1:M|D), we can easily
compute the posterior for vj. Alternatively, we can use bootstrap.
Figure 16.21 gives an example, using BART. We see that the ﬁve relevant variables are chosen
much more than the ﬁve irrelevant variables. As we increase the number Mof trees, all the
variables are more likely to be chosen, reducing the sensitivity of this method, but for small M,
the method is farily diagnostic.

16.8. Interpretingblack-boxmodels 587
Exercises
Exercise 16.1 Nonlinear regression for inverse dynamics
In this question, we ﬁt a model which can predict what torques a robot needs to apply in order to make
its arm reach a desired point in space. The data was collected from a SARCOS robot arm with 7 degrees of
freedom. The input vector x∈R21encodes the desired position, velocity and accelaration of the 7 joints.
The output vector y∈R7encodes the torques that should be applied to the joints to reach that point.
The mapping from xtoyis highly nonlinear.
We have N=4 8,933training points and Ntest=4,449testing points. For simplicity, we following
standard practice and focus on just predicting a scalar output, namely the torque for the ﬁrst joint.
Download the data from http://www .gaussianprocess .org/gpml . Standardize the inputs so they
have zero mean and unit variance on the training set, and center the outputs so they have zero mean

on the training set. Apply the corresponding transformations to the test data. Below we will describevarious models which you should ﬁt to this transformed data. Then make predictions and compute thestandardized mean squared error on the test set as follows:
SMSE=
1
Ntest∑Ntest
i=1(yi−ˆyi)2
σ2(16.111)
whereσ2=1
Ntrain∑Ntrain
i=1(yi−y)2is the variance of the output computed on the training set.
a. The ﬁrst method you should try is standard linear regression. Turn in your numbers and code.
(According to (Rasmussen and Williams 2006, p24), you should be able to achieve a SMSE of 0.075

SMSE=
1
whereσ2=1
using this method.)
b. Now try running K-means clustering (using cross validation to pick K). Then ﬁt an RBF network to
the data, using the μkestimated by K-means. Use CV to estimate the RBF bandwidth. What SMSE do
you get? Turn in your numbers and code. (According to (Rasmussen and Williams 2006, p24), Gaussianprocess regression can get an SMSE of 0.011, so the goal is to get close to that.)
c. Now try ﬁtting a feedforward neural network. Use CV to pick the number of hidden units and the
strength of the ℓ
2regularizer. What SMSE do you get? Turn in your numbers and code.

17 Markov and hidden Markov models
17.1 Introduction
In this chapter, we discuss probabilistic models for sequences of observations, X1,...,X T,o f
arbitrary length T. Such models have applications in computational biology, natural language
processing, time series forecasting, etc. We focus on the case where we the observations occur
at discrete “time steps”, although “time” may also refer to locations within a sequence.
17.2 Markov models
Recall from Section 10.2.2 that the basic idea behind a Markov chain is to assume that Xt
captures all the relevant information for predicting the future (i.e., we assume it is a sufficientstatistic). If we assume discrete time steps, we can write the joint distribution as follows:
p(X
1:T)=p(X1)p(X2|X1)p(X3|X2)...=p(X1)T∏
t=2p(Xt|Xt−1) (17.1)
This is called a Markov chain orMarkov model.
If we assume the transition function p(Xt|Xt−1)is independent of time, then the chain is
calledhomogeneous, stationary,o r time-invariant. This is an example of parameter tying,

p(X
since the same parameter is shared by multiple variables. This assumption allows us to modelan arbitrary number of variables using a ﬁxed number of parameters; such models are calledstochastic processes.
If we assume that the observed variables are discrete, so X
t∈{1,...,K}, this is called a
discrete-state or ﬁnite-state Markov chain. We will make this assumption throughout the rest ofthis section.
17.2.1 Transition matrix
WhenXtis discrete, so Xt∈{1,...,K}, the conditional distribution p(Xt|Xt−1)can be

written as a K×Kmatrix, known as the transition matrix A,w h e r e Aij=p(Xt=
j|Xt−1=i)is the probability of going from state ito statej. Each row of the matrix sums to
one,∑
jAij=1, so this is called a stochastic matrix.

590 Chapter17. MarkovandhiddenMarkovmodels
1 2α
β1−α1−β
(a)1 2 3A12 A23A11 A22 A33
(b)
Figure 17.1 State transition diagrams for some simple Markov chains. Left: a 2-state chain. Right: a
3-state left-to-right chain.
A stationary, ﬁnite-state Markov chain is equivalent to a stochastic automaton. It is common
to visualize such automata by drawing a directed graph, where nodes represent states and arrows
represent legal transitions, i.e., non-zero elements of A. This is known as a state transition
diagram. The weights associated with the arcs are the probabilities. For example, the following2-state chain
A=(1−αα
β1−β)
(17.2)
is illustrated in Figure 17.1(left). The following 3-state chain
A=⎛
⎝A
11A120
0A22A23
001⎞⎠ (17.3)
is illustrated in Figure 17.1(right). This is called a left-to-right transition matrix, and is com-
monly used in speech recognition (Section 17.6.2).
TheA
ijelement of the transition matrix speciﬁes the probability of getting from itojin

A=(1−αα
β1−β)
(17.2)
A=⎛
⎝A
11A120
0A22A23
TheA
one step. The n-step transition matrix A(n)is deﬁned as
Aij(n)≜p(Xt+n=j|Xt=i) (17.4)
which is the probability of getting from itojin exactly nsteps. Obviously A(1) =A. The
Chapman-Kolmogorov equations state that
Aij(m+n)=K∑
k=1Aik(m)Akj(n) (17.5)
In words, the probability of getting from itojinm+nsteps is just the probability of getting
fromitokinmsteps, and then from ktojinnsteps, summed up over all k. We can write
the above as a matrix multiplication
A(m+n)=A(m)A(n) (17.6)
Hence
A(n)=AA(n−1) =AAA(n−2) =···=An(17.7)

Hence
Thus we can simulate multiple steps of a Markov chain by “powering up” the transition matrix.

17.2. Markovmodels 591
SAYS IT’S NOT IN THE CARDS LEGENDARY RECONNAISSANCE BY ROLLIE
DEMOCRACIES UNSUSTAINABLE COULD STRIKE REDLINING VISITS TO PROFIT
BOOKING WAIT HERE AT MADISON SQUARE GARDEN COUNTY COURTHOUSE WHERE HE
HAD BEEN DONE IN THREE ALREADY IN ANY WAY IN WHICH A TEACHER
Table 17.1 Example output from an 4-gram word model, trained using backoff smoothing on the Broadcast
News corpus. The ﬁrst 4 words are speciﬁed by hand, the model generates the 5th word, and then the
results are fed back into the model. Source: http://www .fit.vutbr .cz/~imikolov/rnnlm/gen-4gra
m.txt.
17.2.2 Application: Language modeling
One important application of Markov models is to make statistical language models, which are
probability distributions over sequences of words. We deﬁne the state space to be all the words
in English (or some other language). The marginal probabilities p(Xt=k)are called unigram
statistics. If we use a ﬁrst-order Markov model, then p(Xt=k|Xt−1=j)is called a bigram

m.txt.
model. If we use a second-order Markov model, then p(Xt=k|Xt−1=j,Xt−2=i)is
called atrigram model. And so on. In general these are called n-gram models. For example,
Figure 17.2 shows 1-gram and 2-grams counts for the letters{a,...,z,−} (where - represents
space) estimated from Darwin’s OnTheOriginOfSpecies.
Language models can be used for several things, such as the following:
•Sentence completion A language model can predict the next word given the previous

words in a sentence. This can be used to reduce the amount of typing required, which isparticularly important for disabled users (see e.g., David Mackay’s Dasher system
1), or uses of
mobile devices.
•Data compression Any density model can be used to deﬁne an encoding scheme, by
assigning short codewords to more probable strings. The more accurate the predictive model,the fewer the number of bits it requires to store the data.
•Text classiﬁcation Any density model can be used as a class-conditional density and hence

turned into a (generative) classiﬁer. Note that using a 0-gram class-conditional density (i.e.,only unigram statistics) would be equivalent to a naive Bayes classiﬁer (see Section 3.5).
•Automatic essay writing One can sample from p(x
1:t)to generate artiﬁcial text. This is
one way of assessing the quality of the model. In Table 17.1, we give an example of textgenerated from a 4-gram model, trained on a corpus with 400 million words. ((Tomas et al.2011) describes a much better language model, based on recurrent neural networks, whichgenerates much more semantically plausible text.)
1.http://www .inference .phy.cam.ac.uk/dasher/

592 Chapter17. MarkovandhiddenMarkovmodels
 1   0.16098   _
 2   0.06687   a
 3   0.01414   b
 4   0.02938   c
 5   0.03107   d
 6   0.11055   e
 7   0.02325   f
 8   0.01530   g
 9   0.04174   h
10   0.06233   i
11   0.00060   j
12   0.00309   k
13   0.03515   l
14   0.02107   m
15   0.06007   n
16   0.06066   o
17   0.01594   p
18   0.00077   q
19   0.05265   r
20   0.05761   s
21   0.07566   t
22   0.02149   u
23   0.00993   v
24   0.01341   w
25   0.00208   x
26   0.01381   y
27   0.00039   zUnigrams_abcdefghijklmnopqrstuvwxyz
_
a
b
c
d
e
f
g
h
i
j
k
l
m
n
o
p
q
r
s
t
u
v
w
x
y
zBigrams
Figure 17.2 Unigram and bigram counts from Darwin’s OnTheOriginOfSpecies. The 2D picture on the
right is a Hinton diagram of the joint distribution. The size of the white squares is proportional to the
value of the entry in the corresponding vector/ matrix. Based on (MacKay 2003, p22). Figure generated byngramPlot .
17.2.2.1 MLE for Markov language models

_
a
b
c
d
e
f
g
h
i
j
k
l
m
n
o
p
q
r
s
t
u
v
w
x
y
zBigrams
We now discuss a simple way to estimate the transition matrix from training data. The proba-
bility of any particular sequence of length Tis given by
p(x1:T|θ)=π (x1)A(x1,x2)...A(xT−1,xT) (17.8)
=K∏
j=1(πj)I(x1=j)T∏
t=2K∏
j=1K∏
k=1(Ajk)I(xt=k,xt−1=j)(17.9)
Hence the log-likelihood of a set of sequences D=(x1,...,xN),w h e r exi=(xi1,...,x i,Ti)
is a sequence of length Ti, is given by
logp(D|θ)=N∑
i=1logp(xi|θ)=∑
jN1
jlogπj+∑
j∑
kNjklogAjk (17.10)
where we deﬁne the following counts:
N1
j≜N∑
i=1I(xi1=j),Njk≜N∑

=K∏
t=2K∏
j=1K∏
jN1
jlogπj+∑
j∑
N1
j≜N∑
i=1Ti−1∑
t=1I(xi,t=j,xi,t+1=k) (17.11)

17.2. Markovmodels 593
Hence we can write the MLE as the normalized counts:
ˆπj=N1
j∑
jN1
j,ˆAjk=Njk∑
kNjk(17.12)
These results can be extended in a straightforward way to higher order Markov models.
However, the problem of zero-counts becomes very acute whenever the number of states K,
and/or the order of the chain, n, is large. An n-gram models has O(Kn)parameters. If we have
K∼50,000words in our vocabulary, then a bi-gram model will have about 2.5 billion free
parameters, corresponding to all possible word pairs. It is very unlikely we will see all of these
in our training data. However, we do not want to predict that a particular word string is totallyimpossible just because we happen not to have seen it in our training text — that would be asevere form of overﬁtting.
2
A simple solution to this is to use add-one smoothing, where we simply add one to all the

2
empirical counts before normalizing. The Bayesian justiﬁcation for this is given in Section 3.3.4.1.However add-one smoothing assumes all n-grams are equally likely, which is not very realistic.A more sophisticated Bayesian approach is discussed in Section 17.2.2.2.
An alternative to using smart priors is to gather lots and lots of data. For example, Google
has ﬁt n-gram models (for n=1:5) based on one trillion words extracted from the web. Their
data, which is over 100GB when uncompressed, is publically available.

3An example of their
data, for a set of 4-grams, is shown below.
serve as the incoming 92
serve as the incubator 99
serve as the independent 794
serve as the index 223
serve as the indication 72
serve as the indicator 120
serve as the indicators 45
serve as the indispensable 111
serve as the indispensible 40
serve as the individual 234
...
Although such an approach, based on “brute force and ignorance”, can be successful, it is
rather unsatisfying, since it is clear that this is not how humans learn (see e.g., (Tenenbaum
and Xu 2000)). A more reﬁned Bayesian approach, that needs much less data, is described inSection 17.2.2.2.
17.2.2.2 Empirical Bayes version of deleted interpolation

...
A common heuristic used to ﬁx the sparse data problem is called deleted interpolation (Chen
and Goodman 1996). This deﬁnes the transition matrix as a convex combination of the bigram
2. A famous example of an improbable, but syntactically valid, English word string, due to Noam Chomsky, is “colourless
green ideas sleep furiously”. We would not want our model to predict that this string is impossible. Even ungrammatical

constructs should be allowed by our model with a certain probability, since people frequently violate grammatical rules,
especially in spoken language.
3. See http://googleresearch .blogspot .com/2006/08/all-our-n-gram-are-belong-to-you .htmlfor de-
tails.

594 Chapter17. MarkovandhiddenMarkovmodels
frequencies fjk=Njk/Njand the unigram frequencies fk=Nk/N:
Ajk=( 1−λ)fjk+λfk (17.13)
The term λis usually set by cross validation. There is also a closely related technique called
backoff smoothing; the idea is that if fjkis too small, we “back off” to a more reliable estimate,
namelyfk.
We will now show that the deleted interpolation heuristic is an approximation to the predic-
tions made by a simple hierarchical Bayesian model. Our presentation follows (McKay and Peto
1995). First, let us use an independent Dirichlet prior on each row of the transition matrix:
Aj∼Dir(α0m1,...,α 0mK)=D i r (α0m)=D i r ( α) (17.14)
whereAjis rowjof the transition matrix, mis the prior mean (satisfying∑
kmk=1) and
α0is the prior strength. We will use the same prior for each row: see Figure 17.3.
The posterior is given by Aj∼Dir(α+Nj),w h e r eNj=(Nj1,...,N jK)is the vector
that records the number of times we have transitioned out of state jto each of the other states.

kmk=1) and
From Equation 3.51, the posterior predictive density is
p(Xt+1=k|Xt=j,D)=Ajk=Njk+αmk
Nj+α0=fjkNj+αmk
Nj+α0=( 1−λj)fjk+λjmk(17.15)
whereAjk=E[Ajk|D,α]and
λj=α
Nj+α0(17.16)
This is very similar to Equation 17.13 but not identical. The main difference is that the Bayesian
model uses a context-dependent weight λjto combine mkwith the empirical frequency fjk,
rather than a ﬁxed weight λ. This is like adaptive deleted interpolation. Furthermore, rather

λj=α
than backing off to the empirical marginal frequencies fk, we back off to the model parameter
mk.
The only remaining question is: what values should we use for αandm? Let’s use empirical
Bayes. Since we assume each row of the transition matrix is a priori independent given α, the
marginal likelihood for our Markov model is found by applying Equation 5.24 to each row:
p(D|α)=∏
jB(Nj+α)
B(α)(17.17)
whereNj=(Nj1,...,N jK)are the counts for leaving state jandB(α)is the generalized
beta function.

mk.
p(D|α)=∏
jB(Nj+α)
We can ﬁt this using the methods discussed in (Minka 2000e). However, we can also use the
following approximation (McKay and Peto 1995, p12):
mk∝| {j:Njk>0}| (17.18)
This says that the prior probability of word kis given by the number of different contexts in
which it occurs, rather than the number of times it occurs. To justify the reasonableness of thisresult, Mackay and Peto (McKay and Peto 1995) give the following example.

17.2. Markovmodels 595
Figure 17.3 A Markov chain in which we put a different Dirichlet prior on every row of the transition
matrixA, but the hyperparameters of the Dirichlet are shared.
Imagine, you see, that the language, you see, has, you see, a
frequently occuring couplet ’you see’, you see, in which the second
word of the couplet, see, follows the first word, you, with very high
probability, you see. Then the marginal statistics, you see, are going
to become hugely dominated, you see, by the words you and see, with
equal frequency, you see.
If we use the standard smoothing formula, Equation 17.13, then P(you |novel) and P(see |novel),
for some novel context word not seen before, would turn out to be the same, since the marginal
frequencies of ’you’ and ’see’ are the same (11 times each). However, this seems unreasonable.
’You’ appears in many contexts, so P(you |novel) should be high, but ’see’ only follows ’you’, so

P(see|novel) should be low. If we use the Bayesian formula Equation 17.15, we will get this effect
for free, since we back off to mknotfk, andmkwill be large for ’you’ and small for ’see’ by
Equation 17.18.
Unfortunately, although elegant, this Bayesian model does not beat the state-of-the-art lan-
guage model, known as interpolated Kneser-Ney (Kneser and Ney 1995; Chen and Goodman
1998). However, in (Teh 2006), it was shown how one can build a non-parametric Bayesian

model which outperforms interpolated Kneser-Ney, by using variable-length contexts. In (Wood
et al. 2009), this method was extended to create the “sequence memoizer”, which is currently
(2010) the best-performing language model.4
17.2.2.3 Handling out-of-vocabulary words
While the above smoothing methods handle the case where the counts are small or even zero,
none of them deal with the case where the test set may contain a completely novel word. In

particular, they all assume that the words in the vocabulary (i.e., the state space of Xt)i sﬁ x e d
and known (typically it is the set of unique words in the training data, or in some dictionary).
4. Interestingly, these non-parametric methods are based on posterior inference using MCMC (Section 24.1) and/or
particle ﬁltering (Section 23.5), rather than optimization methods such as EB. Despite this, they are quite efficient.

596 Chapter17. MarkovandhiddenMarkovmodels
1 2 31.0 0.5
0.51.0
(a)12340.9
0.90.5 0.51.0 0.1 0.1
(b)
Figure 17.4 Some Markov chains. (a) A 3-state aperiodic chain. (b) A reducible 4-state chain.
Even if all Ajk’s are non-zero, none of these models will predict a novel word outside of this set,
and hence will assign zero probability to a test sentence with an unfamiliar word. (Unfamiliar
words are bound to occur, because the set of words is an open class. For example, the set ofproper nouns (names of people and places) is unbounded.)
A standard heuristic to solve this problem is to replace all novel words with the special symbol
unk, which stands for “unknown”. A certain amount of probability mass is held aside for thisevent.
AmoreprincipledsolutionwouldbetouseaDirichletprocess, whichcangenerateacountably

inﬁnite state space, as the amount of data increases (see Section 25.2.2). If all novel words are“accepted” as genuine words, then the system has no predictive power, since any misspellingwill be considered a new word. So the novel word has to be seen frequently enough to warrantbeing added to the vocabulary. See e.g., (Friedman and Singer 1999; Griffiths and Tenenbaum2001) for details.
17.2.3 Stationary distribution of a Markov chain *

We have been focussing on Markov models as a way of deﬁning joint probability distributionsover sequences. However, we can also interpret them as stochastic dynamical systems, wherewe “hop” from one state to another at each time step. In this case, we are often interested in thelong term distribution over states, which is known as the stationary distribution of the chain.

In this section, we discuss some of the relevant theory. Later we will consider two importantapplications: Google’s PageRank algorithm for ranking web pages (Section 17.2.4), and the MCMCalgorithm for generating samples from hard-to-normalize probability distributions (Chapter 24).
17.2.3.1 What is a stationary distribution?
LetA
ij=p(Xt=j|Xt−1=i)be the one-step transition matrix, and let πt(j)=p(Xt=j)
be the probability of being in state jat timet. It is conventional in this context to assume that
πis arowvector. If we have an initial distribution over states of π0, then at time 1 we have

LetA
π1(j)=∑
iπ0(i)Aij (17.19)
or, in matrix notation,
π1=π0A (17.20)

17.2. Markovmodels 597
We can imagine iterating these equations. If we ever reach a stage where
π=πA (17.21)
then we say we have reached the stationary distribution (also called the invariant distribution
orequilibrium distribution). Once we enter the stationary distribution, we will never leave.
For example, consider the chain in Figure 17.4(a). To ﬁnd its stationary distribution, we write
(π1π2π3)
=(π1π2π3)⎛
⎝1−A12−A13 A12 A13
A21 1−A21−A23 A23
A31 A32 1−A31−A32⎞⎠(17.22)
so
π
1=π1(1−A12−A12)+π2A21+π3A31 (17.23)
or
π1(A12+A13)=π2A21+π3A31 (17.24)
πi∑
j̸=iAij=∑
j̸=iπjAji (17.25)
In other words, the probability of being in state itimes the net ﬂow out of state imust equal
the probability of being in each other state jtimes the net ﬂow from that state into i. These
are called the global balance equations. We can then solve these equations, subject to the
constraint that∑
jπj=1.
17.2.3.2 Computing the stationary distribution

so
π
or
πi∑
j̸=iAij=∑
jπj=1.
To ﬁnd the stationary distribution, we can just solve the eigenvector equation ATv=v, and
then to set π=vT,w h e r evis an eigenvector with eigenvalue 1. (We can be sure such an
eigenvector exists, since Ais a row-stochastic matrix, so A1=1; also recall that the eigenvalues
ofAandATare the same.) Of course, since eigenvectors are unique only up to constants of
proportionality, we must normalize vat the end to ensure it sums to one.

jπj=1.
Note, however, that the eigenvectors are only guaranteed to be real-valued if the matrix is
positive,Aij>0(and hence Aij<1, due to the sum-to-one constraint). A more general
approach, which can handle chains where some transition probabilities are 0 or 1 (such as
Figure 17.4(a)), is as follows (Resnick 1992, p138). We have Kconstraints from π(I−A)=0K×1
and 1 constraint from π1K×1=0. Since we only have Kunknowns, this is overconstrained.

So let us replace any column (e.g., the last) of I−Awith1, to get a new matrix, call it M.
Next we deﬁne r=[ 0,0,...,1], where the 1 in the last position corresponds to the column of
all 1s inM. We then solve πM=r. For example, for a 3 state chain we have to solve this
linear system:
(
π1π2π3)⎛
⎝1−A11−A121
−A211−A221
−A31−A321⎞⎠=(001)
(17.26)

598 Chapter17. MarkovandhiddenMarkovmodels
For the chain in Figure 17.4(a) we ﬁnd π=[ 0.4,0.4,0.2]. We can easily verify this is correct,
sinceπ=πA. See mcStatDist for some Matlab code.
Unfortunately, not all chains have a stationary distribution. as we explain below.
17.2.3.3 When does a stationary distribution exist? *
Consider the 4-state chain in Figure 17.4(b). If we start in state 4, we will stay there forever, since
4i sa nabsorbing state. Thus π=( 0,0,0,1)is one possible stationary distribution. However,
if we start in 1 or 2, we will oscillate between those two states for ever. So π=( 0.5,0.5,0,0)
is another possible stationary distribution. If we start in state 3, we could end up in either ofthe above stationary distributions.
We see from this example that a necessary condition to have a unique stationary distribution
is that the state transition diagram be a singly connected component, i.e., we can get from anystate to any other state. Such chains are called irreducible.

Now consider the 2-state chain in Figure 17.1(a). This is irreducible provided α,β >0.
Supposeα=β=0.9. It is clear by symmetry that this chain will spend 50% of its time in
each state. Thus π=( 0.5,0.5). But now suppose α=β=1. In this case, the chain will
oscillate between the two states, but the long-term distribution on states depends on where youstart from. If we start in state 1, then on every odd time step (1,3,5,...) we will be in state 1; butif we start in state 2, then on every odd time step we will be in state 2.

This example motivates the following deﬁnition. Let us say that a chain has a limiting
distribution ifπ
j= limn→∞An
ijexists and is independent of i, for allj. If this holds, then
the long-run distribution over states will be independent of the starting state:
P(Xt=j)=∑
iP(X0=i)Aij(t)→πjast→∞ (17.27)
Let us now characterize when a limiting distribution exists. Deﬁne the periodof stateito be
d(i)=gcd{t :Aii(t)>0} (17.28)
where gcd stands for greatest common divisor, i.e., the largest integer that divides all the

P(Xt=j)=∑
members of the set. For example, in Figure 17.4(a), we have d(1) =d(2) =gcd(2,3,4,6,...)=1
andd(3) =gcd(3,5,6,...)=1 .W es a yas t a t e iisaperiodic ifd(i)=1. (A sufficient condition
to ensure this is if state ihas a self-loop, but this is not a necessary condition.) We say a chain
is aperiodic if all its states are aperiodic. One can show the following important result:
Theorem 17.2.1. Every irreducible (singly connected), aperiodic ﬁnite state Markov chain has a
limitingdistribution,whichisequalto π,itsuniquestationarydistribution.

A special case of this result says that every regular ﬁnite state chain has a unique stationary
distribution, where a regularchain is one whose transition matrix satisﬁes An
ij>0for some
integernand alli,j, i.e., it is possible to get from any state to any other state in nsteps.
Consequently, after nsteps, the chain could be in any state, no matter where it started. One
can show that sufficient conditions to ensure regularity are that the chain be irreducible (singly
connected) and that every state have a self-transition.

To handle the case of Markov chains whose state-space is not ﬁnite (e.g, the countable set of
all integers, or all the uncountable set of all reals), we need to generalize some of the earlier

17.2. Markovmodels 599
deﬁnitions. Since the details are rather technical, we just brieﬂy state the main results without
proof. See e.g., (Grimmett and Stirzaker 1992) for details.
For a stationary distribution to exist, we require irreducibility (singly connected) and aperiod-
icity, as before. But we also require that each state is recurrent. (A chain in which all states
are recurrent is called a recurrent chain.) Recurrent means that you will return to that statewith probability 1. As a simple example of a non-recurrent state (i.e., a transient state), consider
Figure 17.4(b): states 3 is transient because one immediately leaves it and either spins aroundstate 4 forever, or oscillates between states 1 and 2 forever. There is no way to return to state 3.
It is clear that any ﬁnite-state irreducible chain is recurrent, since you can always get back to

where you started from. But now consider an example with an inﬁnite state space. Suppose weperform a random walk on the integers, X={...,−2,−1,0,1,2,...}.L e tA
i,i+1=pbe the
probability of moving right, and Ai,i−1=1−pbe the probability of moving left. Suppose we
start atX1=0.I fp>0.5, we will shoot off to +∞; we are not guaranteed to return. Similarly,
ifp<0.5, we will shoot off to −∞. So in both cases, the chain is not recurrent, even though
it is irreducible.

It should be intuitively obvious that we require all states to be recurrent for a stationary
distribution to exist. However, this is not sufficient. To see this, consider the random walkon the integers again, and suppose p=0.5. In this case, we can return to the origin an

inﬁnite number of times, so the chain is recurrent. However, it takes inﬁnitely long to doso. This prohibits it from having a stationary distribution. The intuitive reason is that thedistribution keeps spreading out over a larger and larger set of the integers, and never convergesto a stationary distribution. More formally, we deﬁne a state to be non-null recurrent if the
expected time to return to this state is ﬁnite. A chain in which all states are non-null is called anon-null chain.

For brevity, we we say that a state is ergodicif it is aperiodic, recurrent and non-null, and
we say a chain is ergodic if all its states are ergodic.
We can now state our main theorem:
Theorem 17.2.2. Every irreducible (singly connected), ergodic Markov chain has a limiting distri-
bution,whichisequalto π,itsuniquestationarydistribution.
This generalizes Theorem 17.2.1, since for irreducible ﬁnite-state chains, all states are recurrent
and non-null.
17.2.3.4 Detailed balance

Establishing ergodicity can be difficult. We now give an alternative condition that is easier toverify.
We say that a Markov chain Aistime reversible if there exists a distribution πsuch that
π
iAij=πjAji (17.29)
These are called the detailed balance equations. This says that the ﬂow from itojmust
equal the ﬂow from jtoi, weighted by the appropriate source probabilities.
We have the following important result.
Theorem 17.2.3. If a Markov chain with transition matrix Ais regular and satisﬁes detailed

π
balancewrtdistribution π,thenπisastationarydistributionofthechain.

600 Chapter17. MarkovandhiddenMarkovmodels
X1
X2
X3X4
X5 X6
Figure 17.5 A very small world wide web. Figure generated by pagerankDemo , written by Tim Davis.
Proof.To see this, note that
∑
iπiAij=∑
iπjAji=πj∑
iAji=πj (17.30)
and hence π=Aπ.
Note that this condition is sufficient but not necessary (see Figure 17.4(a) for an example of a
chain with a stationary distribution which does not satisfy detailed balance).
In Section 24.1, we will discuss Markov chain Monte Carlo or MCMC methods. These take
as input a desired distribution πand construct a transition matrix (or in general, a transition
kernel)Awhich satisﬁes detailed balance wrt π. Thus by sampling states from such a chain,
we will eventually enter the stationary distribution, and will visit states with probabilities given
byπ.
17.2.4 Application: Google’s PageRank algorithm for web page ranking *
The results in Section 17.2.3 form the theoretical underpinnings to Google’s PageRank algorithm,

byπ.
which is used for information retrieval on the world-wide web. We sketch the basic idea below;see (Byran and Leise 2006) for a more detailed explanation.
We will treat the web as a giant directed graph, where nodes represent web pages (documents)
and edges represent hyper-links.
5We then perform a process called web crawling. We start at
a few designated root nodes, such as dmoz. org, the home of the Open Directory Project, and
then follows the links, storing all the pages that we encounter, until we run out of time.

Next, all of the words in each web page are entered into a data structure called an inverted
index. That is, for each word, we store a list of the documents where this word occurs. (Inpractice, we store a list of hash codes representing the URLs.) At test time, when a user enters
5. In 2008, Google said it had indexed 1 trillion (1012)unique URLs. If we assume there are about 10 URLs per page
(on average), this means there were about 100 billion unique web pages. Estimates for 2010 are about 121 billion unique
web pages. Source: thenextweb .com/shareables/2011/01/11/infographic-how-big-is-the-internet .

17.2. Markovmodels 601
a query, we can just look up all the documents containing each word, and intersect these
lists (since queries are deﬁned by a conjunction of search terms). We can get a reﬁned searchby storing the location of each word in each document. We can then test if the words in adocument occur in the same order as in the query.
Let us give an example, from http://en .wikipedia .org/wiki/Inverted_index .W e
have 3 documents, T
0= “it is what it is”, T1= “what is it” and T2= “it is a banana”. Then
we can create the following inverted index, where each pair represents a document and wordlocation:
"a": {(2, 2)}
"banana": {(2, 3)}"is": {(0, 1), (0, 4), (1, 1), (2, 1)}"it": {(0, 0), (0, 3), (1, 2), (2, 0)}"what": {(0, 2), (1, 0)}
For example, we see that the word “what” occurs at location 2 (counting from 0) in document
0, and location 0 in document 1. Suppose we search for “what is it”. If we ignore word order,we retrieve the following documents:
{T
0,T1}∩{T0,T1,T2}∩{T0,T1,T2}={T0,T1} (17.31)

{T
If we require that the word order matches, only document T1would be returned. More generally,
we can allow out-of-order matches, but can give “bonus points” to documents whose word ordermatches the query’s word order, or to other features, such as if the words occur in the title ofa document. We can then return the matching documents in decreasing order of their score/relevance. This is called document ranking.
So far, we have described the standard process of information retrieval. But the link structure

of the web provides an additional source of information. The basic idea is that some web pagesare more authoritative than others, so these should be ranked higher (assuming they matchthe query). A web page is an authority if it is linked to by many other pages. But to protectagainst the effect of so-called link farms, which are dummy pages which just link to a given

site to boost its apparent relevance, we will weight each incoming link by the source’s authority.Thus we get the following recursive deﬁnition for the authoritativeness of page j, also called its
PageRank:
π
j=∑
iAijπi (17.32)
whereAijis the probability of following a link from itoj. We recognize Equation 17.32 as the
stationary distribution of a Markov chain.
In the simplest setting, we deﬁne Ai.as a uniform distribution over all states that iis

PageRank:
π
j=∑
connected to. However, to ensure the distribution is unique, we need to make the chain into aregular chain. This can be done by allowing each state ito jump to any other state (including
itself) with some small probability. This effectively makes the transition matrix aperiodic andfully connected (although the adjacency matrix G
ijof the web itself is highly sparse).
We discuss efficient methods for computing the leading eigenvector of this giant matrix below.
But ﬁrst, let us give an example of the PageRank algorithm. Consider the small web in Figure 17.5.

602 Chapter17. MarkovandhiddenMarkovmodels
0 100 200 300 400 5000
50
100
150
200
250
300
350
400
450
500
nz = 2636
(a)0 100 200 300 400 50000.0020.0040.0060.0080.010.0120.0140.0160.0180.02
(b)
Figure 17.6 (a) Web graph of 500 sites rooted at www.harvard .edu. (b) Corresponding page rank vector.
Figure generated by pagerankDemoPmtk , Based on code by Cleve Moler (Moler 2004).
We ﬁnd that the stationary distribution is
π=( 0.3209,0.1706,0.1065,0.1368,0.0643,0.2008) (17.33)
So a random surfer will visit site 1 about 32% of the time. We see that node 1 has a higher
PageRank than nodes 4 or 6, even though they all have the same number of in-links. This isbecause being linked to from an inﬂuential nodehelps increase your PageRank score more thanbeing linked to by a less inﬂuential node.
As a slightly larger example, Figure 17.6(a) shows a web graph, derived from the root of
harvard.edu. Figure 17.6(b) shows the corresponding PageRank vector.
17.2.4.1 Efficiently computing the PageRank vector
LetG

LetG
ij=1iff there is a link from jtoi. Now imagine performing a random walk on
this graph, where at every time step, with probability p=0.85you follow one of the outlinks
uniformly at random, and with probability 1−pyou jump to a random node, again chosen
uniformly at random. If there are no outlinks, you just jump to a random page. (These randomjumps, including self-transitions, ensure the chain is irreducible (singly connected) and regular.Hence we can solve for its unique stationary distribution using eigenvector methods.) Thisdeﬁnes the following transition matrix:
M

M
ij={pGij/cj+δifcj̸=0
1/n ifcj=0(17.34)
wherenis the number of nodes, δ=( 1−p)/nis the probability of jumping from one page
to another without following a link and cj=∑
iGijrepresents the out-degree of page j. (If
n=4·109andp=0.85, then δ=3.75·10−11.) HereMis a stochastic matrix in which
columnssum to one. Note that M=ATin our earlier notation.
We can represent the transition matrix compactly as follows. Deﬁne the diagonal matrix D
with entries
djj={
1/cjifcj̸=0
0ifcj=0(17.35)

17.3. HiddenMarkovmodels 603
Deﬁne the vector zwith components
zj={
δifcj̸=0
1/nifcj=0(17.36)
Then we can rewrite Equation 17.34 as follows:
M=pGD+1zT(17.37)
The matrix Mis not sparse, but it is a rank one modiﬁcation of a sparse matrix. Most of the
elements of Mare equal to the small constant δ. Obviously these do not need to be stored
explicitly.
Our goal is to solve v=Mv,w h e r ev=πT. One efficient method to ﬁnd the leading
eigenvector of a large matrix is known as the power method. This simply consists of repeated
matrix-vector multiplication, followed by normalization:
v∝Mv=pGDv+1zTv (17.38)
It is possible to implement the power method without using any matrix multiplications, by
simply sampling from the transition matrix and counting how often you visit each state. This is
essentially a Monte Carlo approximation to the sum implied by v=Mv. Applying this to the

data in Figure 17.6(a) yields the stationary distribution in Figure 17.6(b). This took 13 iterations toconverge, starting from a uniform distribution. (See also the function pagerankDemo ,b yT i m
Davis, for an animation of the algorithm in action, applied to the small web example.) To handlechanging web structure, we can re-run this algorithm every day or every week, starting voff at
the old distribution (Langville and Meyer 2006).
For details on how to perform this Monte Carlo power method in a parallel distributed

computing environment, see e.g., (Rajaraman and Ullman 2010).
17.2.4.2 Web spam
PageRank is not foolproof. For example, consider the strategy adopted by JC Penney, a depart-ment store in the USA. During the Christmas season of 2010, it planted many links to its homepage on 1000s of irrelevant web pages, thus increasing its ranking on Google’s search engine(Segal 2011). Even though each of these source pages has low PageRank, there were so manyof them that their effect added up. Businesses call this search engine optimization; Google
calls itweb spam . When Google was notiﬁed of this scam (by the NewYorkTimes ), it manually

downweighted JC Penney, since such behavior violates Google’s code of conduct. The resultwas that JC Penney dropped from rank 1 to rank 65, essentially making it disappear from view.Automatically detecting such scams relies on various techniques which are beyond the scope ofthis chapter.
17.3 Hidden Markov models
As we mentioned in Section 10.2.2, a hidden Markov model orHMMconsists of a discrete-time,
discrete-state Markov chain, with hidden states zt∈{1,...,K}, plus an observation model

604 Chapter17. MarkovandhiddenMarkovmodels
−20 −15 −10 −5 0 5 10 15 20−10−505101520
12 3 4567
89
101112
13
14
151617
18
1920
(a)2 4 6 8 10 12 14 16 18 2011.21.41.61.822.22.42.62.83
(b)
Figure 17.7 (a) Some 2d data sampled from a 3 state HMM. Each state emits from a 2d Gaussian. (b) The
hidden state sequence. Based on Figure 13.8 of (Bishop 2006b). Figure generated by hmmLillypadDemo .
p(xt|zt). The corresponding joint distribution has the form
p(z1:T,x1:T)=p( z1:T)p(x1:T|z1:T)=[
p(z1)T∏
t=2p(zt|zt−1)][T∏
t=1p(xt|zt)]
(17.39)
The observations in an HMM can be discrete or continuous. If they are discrete, it is common
for the observation model to be an observation matrix:
p(xt=l|zt=k,θ)=B(k,l) (17.40)
If the observations are continuous, it is common for the observation model to be a conditionalGaussian:
p(x
t|zt=k,θ)=N(xt|μk,Σk) (17.41)

p(z1)T∏
(17.39)
p(x
Figure 17.7 shows an example where we have 3 states, each of which emits a different Gaussian.The resulting model is similar to a Gaussian mixture model, except the cluster membershiphas Markovian dynamics. (Indeed, HMMs are sometimes called Markov switching models
(Fruhwirth-Schnatter 2007).) We see that we tend to get multiple observations in the samelocation, and then a sudden jump to a new cluster.
17.3.1 Applications of HMMs

p(x
HMMs can be used as black-box density models on sequences. They have the advantageover Markov models in that they can represent long-range dependencies between observations,mediated via the latent variables. In particular, note that they do not assume the Markovproperty holds for the observations themselves. Such black-box models are useful for time-series prediction (Fraser 2008). They can also be used to deﬁne class-conditional densitiesinside a generative classiﬁer.

However, it is more common to imbue the hidden states with some desired meaning, and to
then try to estimate the hidden states from the observations, i.e., to compute p(z
t|x1:t)if we are

17.3. HiddenMarkovmodels 605
x x ...x
A G ---C
A- AG- C
AG- AA-
- - AAAC
A G ---Cbat
rat
cat
gnat
goat
2. . . 13
(a)
M
0M End Begin MMMIIIIDDD
1 02 3 4
(b)
Figure 17.8 (a) Some DNA sequences. (b) State transition diagram for a proﬁle HMM. Source: Figure 5.7
of (Durbin et al. 1998). Used with kind permission of Richard Durbin.
in an online scenario, or p(zt|x1:T)if we are in an offline scenario (see Section 17.4.1 for further
discussion of the differences between these two approaches). Below we give some examples of
applications which use HMMs in this way:
•Automatic speech recognition.H e r e xtrepresents features extracted from the speech
signal, and ztrepresents the word that is being spoken. The transition model p(zt|zt−1)
represents the language model, and the observation model p(xt|zt)represents the acoustic
model. See e.g., (Jelinek 1997; Jurafsky and Martin 2008) for details.
•Activity recognition.H e r e xtrepresents features extracted from a video frame, and ztis

the class of activity the person is engaged in (e.g., running, walking, sitting, etc.) See e.g.,
(Szeliski 2010) for details.
•Part of speech tagging.H e r e xtrepresents a word, and ztrepresents its part of speech
(noun, verb, adjective, etc.) See Section 19.6.2.1 for more information on POS tagging and

606 Chapter17. MarkovandhiddenMarkovmodels
related tasks.
•Gene ﬁnding.H e r e xtrepresents the DNA nucleotides (A,C,G,T), and ztrepresents whether
we are inside a gene-coding region or not. See e.g., (Schweikerta et al. 2009) for details.
•Protein sequence alignment.H e r e xtrepresents an amino acid, and ztrepresents whether
this matches the latent consensus sequence at this location. This model is called a proﬁle
HMMand is illustrated in Figure 17.8. The HMM has 3 states, called match, insert and delete.
Ifztis a match state, then xtis equal to the t’th value of the consensus. If ztis an insert
state, then xtis generated from a uniform distribution that is unrelated to the consensus
sequence. If ztis a delete state, then xt=−. In this way, we can generate noisy copies of
the consensus sequence of different lengths. In Figure 17.8(a), the consensus is “AGC”, and

we see various versions of this below. A path through the state transition diagram, shownin Figure 17.8(b), speciﬁes how to align a sequence to the consensus, e.g., for the gnat, themost probable path is D,D,I,I,I,M . This means we delete the A and G parts of the

consensus sequence, we insert 3 A’s, and then we match the ﬁnal C. We can estimate themodel parameters by counting the number of such transitions, and the number of emissionsfrom each kind of state, as shown in Figure 17.8(c). See Section 17.5 for more information ontraining an HMM, and (Durbin et al. 1998) for details on proﬁle HMMs.
Note that for some of these tasks, conditional random ﬁelds, which are essentially discrimi-
native versions of HMMs, may be more suitable; see Chapter 19 for details.
17.4 Inference in HMMs

We now discuss how to infer the hidden state sequence of an HMM, assuming the parametersare known. Exactly the same algorithms apply to other chain-structured graphical models, suchas chain CRFs (see Section 19.6.1). In Chapter 20, we generalize these methods to arbitrarygraphs. And in Section 17.5.2, we show how we can use the output of inference in the contextof parameter estimation.
17.4.1 Types of inference problems for temporal models
There are several different kinds of inferential tasks for an HMM (and SSM in general). Toillustrate the differences, we will consider an example called the occasionally dishonest casino,
from (Durbin et al. 1998). In this model, x
t∈{1,2,...,6}represents which dice face shows
up, andztrepresents the identity of the dice that is being used. Most of the time the casino

uses a fair dice, z=1, but occasionally it switches to a loaded dice, z=2, for a short period.
Ifz=1the observation distribution is a uniform multinoulli over the symbols {1,...,6}.I f
z=2, the observation distribution is skewed towards face 6 (see Figure 17.9). If we sample from
this model, we may observe data such as the following:
Listing 17.1 Example output of casinoDemo
Rolls: 664153216162115234653214356634261655234232315142464156663246
Die: LLLLLLLLLLLLLLFFFFFFLLLLLLLLLLLLLLFFFFFFFFFFFFFFFFFFLLLLLLLL

Here “rolls” refers to the observed symbol and “die” refers to the hidden state (L is loaded and
F is fair). Thus we see that the model generates a sequence of symbols, but the statistics of the

using a loaded die. (a) Filtered estimate of probability of using a loaded dice. (b) Smoothed estimates. (c)
MAP trajectory. Figure generated by casinoDemo .
distribution changes abruptly every now and then. In a typical application, we just see the rolls
and want to infer which dice is being used. But there are different kinds of inference, which wesummarize below.
•Filtering means to compute the belief state p(z
t|x1:t)online, or recursively, as the data
streams in. This is called “ﬁltering” because it reduces the noise more than simply estimating
the hidden state using just the current estimate, p(zt|xt). We will see below that we can
perform ﬁltering by simply applying Bayes rule in a sequential fashion. See Figure 17.10(a) foran example.
•Smoothing means to compute p(z
t|x1:T)offline, given all the evidence. See Figure 17.10(b)

for an example. By conditioning on past and future data, our uncertainty will be signiﬁcantlyreduced. To understand this intuitively, consider a detective trying to ﬁgure out who com-mitted a crime. As he moves through the crime scene, his uncertainty is high until he ﬁndsthe key clue; then he has an “aha” moment, his uncertainty is reduced, and all the previouslyconfusing observations are, in hindsight, easy to explain.

involves computing p(zt−ℓ|x1:t),w h e r eℓ>0is called the lag. This gives better performance
than ﬁltering, but incurs a slight delay. By changing the size of the lag, one can trade off
accuracy vs delay.
•Prediction Instead of predicting the past given the future, as in ﬁxed lag smoothing, we
might want to predict the future given the past, i.e., to compute p(zt+h|x1:t),w h e r eh>0
is called the prediction horizon. For example, suppose h=2; then we have
p(zt+2|x1:t)=∑
zt+1∑
ztp(zt+2|zt+1)p(zt+1|zt)p(zt|x1:t) (17.42)
It is straightforward to perform this computation: we just power up the transition matrix andapply it to the current belief state. The quantity p(z
t+h|x1:t)is a prediction about future
hidden states; it can be converted into a prediction about future observations using
p(xt+h|x1:t)=∑
zt+hp(xt+h|zt+h)p(zt+h|x1:t) (17.43)

zt+1∑
This is the posterior predictive density, and can be used for time-series forecasting (see(Fraser 2008) for details). See Figure 17.11 for a sketch of the relationship between ﬁltering,smoothing, and prediction.
•MAP estimation This means computing argmax
z1:Tp(z1:T|x1:T), which is a most prob-
able state sequence. In the context of HMMs, this is known as Viterbi decoding (see

17.4. InferenceinHMMs 609
Section 17.4.4). Figure 17.10 illustrates the difference between ﬁltering, smoothing and MAP
decoding for the occasionally dishonest casino HMM. We see that the smoothed (offline)estimate is indeed smoother than the ﬁltered (online) estimate. If we threshold the estimatesat 0.5 and compare to the true sequence, we ﬁnd that the ﬁltered method makes 71 errorsout of 300, and the smoothed method makes 49/300; the MAP path makes 60/300 errors. It isnot surprising that smoothing makes fewer errors than Viterbi, since the optimal way to min-imize bit-error rate is to threshold the posterior marginals (see Section 5.7.1.1). Nevertheless,for some applications, we may prefer the Viterbi decoding, as we discuss in Section 17.4.4.
•Posterior samples If there is more than one plausible interpretation of the data, it can be
useful to sample from the posterior, z
1:T∼p(z1:T|x1:T). These sample paths contain much
more information than the sequence of marginals computed by smoothing.

•Probability of the evidence We can compute the probability of the evidence, p(x1:T),
by summing up over all hidden paths, p(x1:T)=∑
z1:Tp(z1:T,x1:T). This can be used to
classify sequences (e.g., if the HMM is used as a class conditional density), for model-basedclustering, for anomaly detection, etc.
17.4.2 The forwards algorithm
We now describe how to recursively compute the ﬁltered marginals, p(zt|x1:t)in an HMM.
The algorithm has two steps. First comes the prediction step, in which we compute the
one-step-ahead predictive density; this acts as the new prior for time t:
p(zt=j|x1:t−1)=∑
ip(zt=j|zt−1=i)p(zt−1=i|x1:t−1) (17.44)
Next comes the update step, in which we absorb the observed data from time tusing Bayes
rule:

rule:
αt(j)≜p(zt=j|x1:t)=p(zt=j|xt,x1:t−1) (17.45)
=1
Ztp(xt|zt=j,x1:t−1)p(zt=j|x1:t−1) (17.46)
where the normalization constant is given by
Zt≜p(xt|x1:t−1)=∑
jp(zt=j|x1:t−1)p(xt|zt=j) (17.47)
This process is known as the predict-update cycle. The distribution p(zt|x1:t)is called the
(ﬁltered)belief state at timet, and is a vector of Knumbers, often denoted by αt. In matrix-
vector notation, we can write the update in the following simple form:
αt∝ψt⊙(ΨTαt−1) (17.48)
whereψt(j)=p(xt|zt=j)is the local evidence at time t,Ψ(i,j)=p(zt=j|zt−1=i)is

=1
the transition matrix, and u⊙vis theHadamard product, representing elementwise vector
multiplication. See Algorithm 6 for the pseudo-code, and hmmFilter for some Matlab code.

610 Chapter17. MarkovandhiddenMarkovmodels
In addition to computing the hidden states, we can use this algorithm to compute the log
probability of the evidence:
logp(x1:T|θ)=T∑
t=1logp(xt|x1:t−1)=T∑
t=1logZt (17.49)
(We need to work in the log domain to avoid numerical underﬂow.)
Algorithm 17.1: Forwards algorithm
1Input: Transition matrices ψ(i,j)=p(zt=j|zt−1=i), local evidence vectors
ψt(j)=p(xt|zt=j), initial state distribution π(j)=p(z1=j);
2[α1,Z1]=normalize( ψ1⊙π);
3fort=2:Tdo
4[αt,Zt]=normalize( ψt⊙(ΨTαt−1));
5Returnα1:Tandlogp(y1:T)=∑
tlogZt;
6Subroutine: [v,Z]=normalize( u):Z=∑
juj;vj=uj/Z;
17.4.3 The forwards-backwards algorithm
In Section 17.4.2, we explained how to compute the ﬁltered marginals p(zt=j|x1:t)using
online inference. We now discuss how to compute the smoothed marginals, p(zt=j|x1:T),
using offline inference.
17.4.3.1 Basic idea
The key decomposition relies on the fact that we can break the chain into two parts, the past
and the future, by conditioning on zt:

tlogZt;
p(zt=j|x1:T)∝p(zt=j,xt+1:T|x1:t)∝p(zt=j|x1:t)p(xt+1:T|zt=j,x1:t)(17.50)
Letαt(j)≜p(zt=j|x1:t)be the ﬁltered belief state as before. Also, deﬁne
βt(j)≜p(xt+1:T|zt=j) (17.51)
as the conditional likelihood of future evidence given that the hidden state at time tisj.
(Note that this is not a probability distribution over states, since it does not need to satisfy∑
jβt(j)=1.) Finally, deﬁne
γt(j)≜p(zt=j|x1:T) (17.52)
as the desired smoothed posterior marginal. From Equation 17.50, we have

γt(j)∝αt(j)βt(j) (17.53)

=∑
jβt(j)ψt(j)ψ(i,j) (17.58)
We can write the resulting equation in matrix-vector form as
βt−1=Ψ(ψt⊙βt) (17.59)
The base case is
βT(i)=p(xT+1:T|zT=i)=p(∅|zT=i)=1 (17.60)
which is the probability of a non-event.
Having computed the forwards and backwards messages, we can combine them to compute
γt(j)∝αt(j)βt(j). The overall algorithm is known as the forwards-backwards algorithm.
The pseudo code is very similar to the forwards case; see hmmFwdBack for an implementation.
We can think of this algorithm as passing “messages” from left to right, and then from right
to left, and then combining them at each node. We will generalize this intuition in Section 20.2,
when we discuss belief propagation.
17.4.3.2 Two-slice smoothed marginals
When we estimate the parameters of the transition matrix using EM (see Section 17.5), we willneed to compute the expected number of transitions from state ito statej:
N
ij=T−1∑
t=1E[I(zt=i,zt+1=j)|x1:T]=T−1∑
t=1p(zt=i,zt+1=j|x1:T) (17.61)

N
ij=T−1∑
The term p(zt=i,zt+1=j|x1:T)is called a (smoothed) two-slice marginal, and can be
computed as follows
ξt,t+1(i,j)≜p(zt=i,zt+1=j|x1:T) (17.62)
∝p(zt|x1:t)p(zt+1|zt,xt+1:T) (17.63)
∝p(zt|x1:t)p(xt+1:T|zt,zt+1)p(zt+1|zt) (17.64)
∝p(zt|x1:t)p(xt+1|zt+1)p(xt+2:T|zt+1)p(zt+1|zt) (17.65)
=αt(i)φt+1(j)βt+1(j)ψ(i,j) (17.66)

612 Chapter17. MarkovandhiddenMarkovmodels
In matrix-vector form, we have
ξt,t+1∝Ψ⊙(αt(φt+1⊙βt+1)T) (17.67)
For another interpretation of these equations, see Section 20.2.4.3.
17.4.3.3 Time and space complexity
It is clear that a straightforward implementation of FB takes O(K2T)time, since we must
perform a K×Kmatrix multiplication at each step. For some applications, such as speech
recognition, Kis very large, so the O(K2)term becomes prohibitive. Fortunately, if the
transition matrix is sparse, we can reduce this substantially. For example, in a left-to-right
transition matrix, the algorithm takes O(TK)time.
In some cases, we can exploit special properties of the state space, even if the transition
matrix is not sparse. In particular, suppose the states represent a discretization of an underlyingcontinuous state-space, and the transition matrix has the form ψ(i,j)∝exp(−σ
2|zi−zj|),
whereziis the continuous vector represented by state i. Then one can implement the forwards-

backwards algorithm in O(TKlogK)time. This is very useful for models with large state
spaces. See Section 22.2.6.1 for details.
In some cases, the bottleneck is memory, not time. The expected sufficient statistics needed
by EM are∑
tξt−1,t(i,j); this takes constant space (independent of T); however, to compute
them, we need O(KT)working space, since we must store αtfort=1,...,Tuntil we do the
backwards pass. It is possible to devise a simple divide-and-conquer algorithm that reduces thespace complexity from O(KT)toO(KlogT)at the cost of increasing the running time from
O(K

by EM are∑
O(K
2T)toO(K2TlogT): see (Binder et al. 1997; Zweig and Padmanabhan 2000) for details.
17.4.4 The Viterbi algorithm
TheViterbialgorithm (Viterbi 1967) can be used to compute the most probable sequence of
states in a chain-structured graphical model, i.e., it can compute
z∗=a r gm a x
z1:Tp(z1:T|x1:T) (17.68)
This is equivalent to computing a shortest path through the trellis diagram in Figure 17.12,

O(K
z∗=a r gm a x
where the nodes are possible states at each time step, and the node and edge weights are logprobabilities. That is, the weight of a path z
1,z2,...,z Tis given by
logπ1(z1)+logφ1(z1)+T∑
t=2[logψ(zt−1,zt)+logφt(zt)] (17.69)
17.4.4.1 MAP vs MPE
Before discussing how the algorithm works, let us make one important remark: the (jointly)most
probablesequenceofstatesisnotnecessarilythesameasthesequenceof(marginally)mostprobablestates. The former is given by Equation 17.68, and is what Viterbi computes, whereas the latter isgiven by the maximizer of the posterior marginals or MPM:
ˆz= (argmax
z1p(z1|x1:T),...,argmax

zTp(zT|x1:T)) (17.70)

suppose we are performing speech recognition and someones says “recognize speech”. This
could be mis-heard as “wreck a nice beach”. Locally it may appear that “beach” is the mostprobable interpretation of that particular window of sound, but when we add the requirementthat the data be explained by a single linguistically plausible path, this interpretation becomesless likely.
On the other hand, the MPM estimates can be more robust (Marroquin et al. 1987). To see
why, note that in Viterbi, when we estimate z
t, we “max out” the other variables:
z∗
t=a r gm a x
ztmax
z1:t−1,zt+1:Tp(z1:t−1,zt,zt+1:T|x1:T) (17.71)
whereas we when we use forwards-backwards, we sum out the other variables:
p(zt|x1:T)=∑
z1:t−1,zt+1:Tp(z1:t−1,zt,zt+1:T|x1:T) (17.72)
This makes the MPM in Equation 17.70 more robust, since we estimate each node averaging overits neighbors, rather than conditioning on a speciﬁc value of its neighbors.
6

z∗
t=a r gm a x
ztmax
6
6. In general, we may want to mix max and sum. For example, consider a joint distribution where we observe

614 Chapter17. MarkovandhiddenMarkovmodels
17.4.4.2 Details of the algorithm
It is tempting to think that we can implement Viterbi by just replacing the sum-operator in
forwards-backwards with a max-operator. The former is called the sum-product, and the
latter the max-product algorithm. If there is a unique mode, running max-product and then
computing using Equation 17.70 will give the same result as using Equation 17.68 (Weiss andFreeman 2001b), but in general, it can lead to incorrect results if there are multiple equallyprobably joint assignments. The reasons is that each node breaks ties independently and hencemay do so in a manner that is inconsistent with its neighbors. The Viterbi algorithm is thereforenot quite as simple as replacing sum with max. In particular, the forwards pass does use max-product, but the backwards pass uses a traceback procedure to recover the most probable path
through the trellis of states. Essentially, once z
tpicks its most probable state, the previous nodes

condition on this event, and therefore they will break ties consistently.
In more detail, deﬁne
δt(j)≜max
z1,...,zt−1p(z1:t−1,zt=j|x1:t) (17.73)
This is the probability of ending up in state jat timet, given that we take the most probable
path. The key insight is that the most probable path to state jat timetmust consist of the
most probable path to some other state iat timet−1, followed by a transition from itoj.
Hence
δt(j)=m a x
iδt−1(i)ψ(i,j)φt(j) (17.74)
We also keep track of the most likely previous state, for each possible state that we end up in:
at(j) = argmax
iδt−1(i)ψ(i,j)φt(j) (17.75)
That is,at(j)tells us the most likely previous state on the most probable path to zt=j.W e
initialize by setting
δ1(j)=πjφ1(j) (17.76)
and we terminate by computing the most probable ﬁnal state z∗
T:
z∗
T=a r gm a x
iδT(i) (17.77)
We can then compute the most probable sequence of states using traceback:
z∗
t=at+1(z∗

Hence
δt(j)=m a x
T:
z∗
T=a r gm a x
z∗
t=at+1(z∗
t+1) (17.78)
As usual, we have to worry about numerical underﬂow. We are free to normalize the δtterms
at each step; this will not affect the maximum. However, unlike the forwards-backwards case,
vand we want to query q;l e tnbe the remaining nuisance variables. We deﬁne the MAP estimate as x∗
q=
argmax xq∑
xnp(xq,xn|xv), where we max over xqand sum over xn. By contrast, we deﬁne the MPEor
most probable explanation as (x∗q,x∗n) = argmax xq,xnp(xq,xn|xv), where we max over both xqandxn. This

z∗
t=at+1(z∗
q=
argmax xq∑
terminology is due to (Pearl 1988), although it is not widely used outside the Bayes net literatire. Obviously MAP=MPE if
n=∅. However, if n̸=∅, then summing out the nuisance variables can give different results than maxing them out.
Summing out nuisance variables is more sensible, but computationally harder, because of the need to combine max and
sum operations (Lerner and Parr 2001).

vector quantized into 7 possible symbols, C1,...,C 7. Each state z1,z2,z3has a different distribution over
these symbols. Based on Figure 15.20 of (Russell and Norvig 2002). (b) Illustration of the Viterbi algorithm
applied to this model, with data sequence C1,C3,C4,C6. The columns represent time, and the rows
represent states. An arrow from state iatt−1to statejattis annotated with two numbers: the ﬁrst
is the probability of the i→jtransition, and the second is the probability of generating observation xt
from state j. The bold lines/ circles represent the most probable sequence of states. Based on Figure 24.27
of (Russell and Norvig 1995).
we can also easily work in the log domain. The key difference is that logmax = maxlog ,
whereaslog∑̸=∑log. Hence we can use
logδt(j)≜max
z1:t−1logp(z1:t−1,zt=j|x1:t) (17.79)
=m a x
ilogδt−1(i)+logψ(i,j)+logφt(j) (17.80)
In the case of Gaussian observation models, this can result in a signiﬁcant (constant factor)

=m a x
speedup, since computing logp(xt|zt)can be much faster than computing p(xt|zt)for a high-
dimensional Gaussian. This is one reason why the Viterbi algorithm is widely used in the E step
of EM (Section 17.5.2) when training large speech recognition systems based on HMMs.
17.4.4.3 Example
Figure 17.13 gives a worked example of the Viterbi algorithm, based on (Russell et al. 1995).
Suppose we observe the discrete sequence of observations x1:4=(C1,C3,C4,C6), representing
codebook entries in a vector-quantized version of a speech signal. The model starts in state

z1. The probability of generating C1inz1is 0.5, so we have δ1(1) = 0.5, andδ1(i)=0for
all other states. Next we can self-transition to z1with probability 0.3, or transition to z2with
proabability 0.7. If we end up in z1, the probability of generating C3is 0.3; if we end up in z2,

616 Chapter17. MarkovandhiddenMarkovmodels
the probability of generating C3is 0.2. Hence we have
δ2(1) = δ1(1)ψ(1,1)φ2(1) = 0.5·0.3·0.3=0.045 (17.81)
δ2(2) = δ1(1)ψ(1,2)φ2(2) = 0.5·0.7·0.2=0.07 (17.82)
Thus state 2 is more probable at t=2; see the second column of Figure 17.13(b). In time step
3, we see that there are two paths into z2,f r o mz1and from z2. The bold arrow indicates that
the latter is more probable. Hence this is the only one we have to remember. The algorithm
continues in this way until we have reached the end of the sequence. One we have reached theend, we can follow the black arrows back to recover the MAP path (which is 1,2,2,3).
17.4.4.4 Time and space complexity
The time complexity of Viterbi is clearly O(K
2T)in general, and the space complexity
isO(KT), both the same as forwards-backwards. If the transition matrix has the form
ψ(i,j)∝exp(−σ2||zi−zj||2),w h e r eziis the continuous vector represented by state i,w e

can implement Viterbi in O(TK)time, instead of O(TKlogK)needed by forwards-backwards.
See Section 22.2.6.1 for details.
17.4.4.5 N-best list
The Viterbi algorithm returns one of the most probable paths. It can be extended to return thetopNpaths (Schwarz and Chow 1990; Nilsson and Goldberger 2001). This is called the N-best

list. Once can then use a discriminative method to rerank the paths based on global featuresderived from the fully observed state sequence (as well as the visible features). This techniqueis widely used in speech recognition. For example, consider the sentence “recognize speech”. Itis possible that the most probable interpretation by the system of this acoustic signal is “wrecka nice speech”, or maybe “wreck a nice beach”. Maybe the correct interpretation is much lowerdown on the list. However, by using a re-ranking system, we may be able to improve the scoreof the correct interpretation based on a more global context.

One problem with the N-best list is that often the top Npaths are very similar to each other,
rather than representing qualitatively different interpretations of the data. Instead we might wantto generate a more diverse set of paths to more accurately represent posterior uncertainty. Oneway to do this is to sample paths from the posterior, as we discuss below. For some other waysto generate diverse MAP estimates, see e.g., (Yadollahpour et al. 2011; Kulesza and Taskar 2011).
17.4.5 Forwards ﬁltering, backwards sampling
It is often useful to sample paths from the posterior:
zs
1:T∼p(z1:T|x1:T) (17.83)
We can do this is as follow: run forwards backwards, to compute the two-slice smoothed posteri-
ors,p(zt−1,t|x1:T); next compute the conditionals p(zt|zt−1,x1:T)by normalizing; sample from
the initial pair of states, z∗
1,2∼p(z1,2|x1:T); ﬁnally, recursively sample z∗
t∼p(zt|z∗
t−1,x1:T).
Note that the above solution requires a forwards-backwards pass, and then an additional

zs
t∼p(zt|z∗
forwards sampling pass. An alternative is to do the forwards pass, and then perform sampling

p(zt+1|x1:t+1)(17.88)
=p(xt+1|zt+1)p(zt+1|zt,x1:t)p(zt|x1:t)
p(zt+1|x1:t+1)(17.89)
=φt+1(j)ψ(i,j)αt(i)
αt+1(j)(17.90)
zs
T∼p(zT=i|x1:T)=αT(i) (17.91)
This algorithm forms the basis of blocked-Gibbs sampling methods for parameter inference,
as we will see below.
17.5 Learning for HMMs
We now discuss how to estimate the parameters θ=(π,A,B),w h e r eπ(i)=p(z1=i)is
the initial state distribution, A(i,j)=p(zt=j|zt−1=i)is the transition matrix, and Bare
the parameters of the class-conditional densities p(xt|zt=j). We ﬁrst consider the case where
z1:Tis observed in the training set, and then the harder case where z1:Tis hidden.
17.5.1 Training with fully observed data
If we observe the hidden state sequences, we can compute the MLEs for Aandπexactly as in
Section 17.2.2.1. If we use a conjugate prior, we can also easily compute the posterior.
The details on how to estimate Bdepend on the form of the observation model. The

situation is identical to ﬁtting a generative classiﬁer. For example, if each state has a multinoullidistribution associated with it, with parameters B
jl=p(Xt=l|zt=j),w h e r el∈{1,...,L}
represents the observed symbol, the MLE is given by
ˆBjl=NX
jl
Nj,NX
jl≜N∑
i=1Ti∑
t=1I(zi,t=j,xi,t=l) (17.92)

618 Chapter17. MarkovandhiddenMarkovmodels
This result is quite intuitive: we simply add up the number of times we are in state jand we
see a symbol l, and divide by the number of times we are in state j.
Similarly, if each state has a Gaussian distribution associated with it, we have (from Sec-
tion 4.2.4) the following MLEs:
ˆμk=xk
Nk,ˆΣk=(xx)T
k−NkˆμkˆμT
k
Nk(17.93)
where the sufficient statistics are given by
xk≜N∑
i=1Ti∑
t=1I(zi,t=k)xi,t (17.94)
(xx)T
k≜N∑
i=1Ti∑
t=1I(zi,t=k)xi,txTi,t(17.95)
Analogous results can be derived for other kinds of distributions. One can also easily extend all
of these results to compute MAP estimates, or even full posteriors over the parameters.
17.5.2 EM for HMMs (the Baum-Welch algorithm)
If theztvariables are not observed, we are in a situation analogous to ﬁtting a mixture model.

ˆμk=xk
k
Nk(17.93)
xk≜N∑
i=1Ti∑
(xx)T
k≜N∑
i=1Ti∑
The most common approach is to use the EM algorithm to ﬁnd the MLE or MAP parameters,although of course one could use other gradient-based methods (see e.g., (Baldi and Chauvin1994)). In this Section, we derive the EM algorithm. When applied to HMMs, this is also knownas theBaum-Welch algorithm (Baum et al. 1970).
17.5.2.1 E step
It is straightforward to show that the expected complete data log likelihood is given by
Q(θ,θ
old)=K∑
k=1E[
N1
k]
logπk+K∑
j=1K∑
k=1E[Njk]logAjk (17.96)
+N∑
i=1Ti∑
t=1K∑

Q(θ,θ
old)=K∑
k=1E[
N1
k]
logπk+K∑
j=1K∑
+N∑
i=1Ti∑
t=1K∑
k=1p(zt=k|xi,θold)logp(xi,t|φk) (17.97)
where the expected counts are given by
E[
N1
k]
=N∑
i=1p(zi1=k|xi,θold) (17.98)
E[Njk]=N∑
i=1Ti∑
t=2p(zi,t−1=j,zi,t=k|xi,θold) (17.99)
E[Nj]=N∑
i=1Ti∑
t=1p(zi,t=j|xi,θold) (17.100)

17.5. LearningforHMMs 619
These expected sufficient statistics can be computed by running the forwards-backwards algo-
rithm on each sequence. In particular, this algorithm computes the following smoothed nodeand edge marginals:
γ
i,t(j)≜p(zt=j|xi,1:Ti,θ) (17.101)
ξi,t(j,k)≜p(zt−1=j,zt=k|xi,1:Ti,θ) (17.102)
17.5.2.2 M step
Based on Section 11.3, we have that the M step for Aandπis to just normalize the expected
counts:
ˆAjk=E[Njk]∑
k′E[Njk′],ˆπk=E[
N1
k]
N(17.103)
This result is quite intuitive: we simply add up the expected number of transitions from jtok,
and divide by the expected number of times we transition from jto anything else.
For a multinoulli observation model, the expected sufficient statistics are
E[Mjl]=N∑
i=1Ti∑
t=1γi,t(j)I(xi,t=l)=N∑
i=1∑
t:xi,t=lγi,t(j) (17.104)
The M step has the form
ˆBjl=E[Mjl]
E[Nj](17.105)
This result is quite intuitive: we simply add up the expected number of times we are in state j
and we see a symbol l, and divide by the expected number of times we are in state j.

E[Mjl]=N∑
i=1Ti∑
i=1∑
For a Gaussian observation model, the expected sufficient statistics are given by
E[xk]=N∑
i=1Ti∑
t=1γi,t(k)xi,t (17.106)
E[
(xx)T
k]
=N∑
i=1Ti∑
t=1γi,t(k)xi,txTi,t(17.107)
The M step becomes
ˆμk=E[xk]
E[Nk],ˆΣk=E[
(xx)T
k]
−E[Nk]ˆμkˆμT
k
E[Nk](17.108)
This can (and should) be regularized in the same way we regularize GMMs.
17.5.2.3 Initialization
As usual with EM, we must take care to ensure that we initialize the parameters carefully, to
minimize the chance of getting stuck in poor local optima. There are several ways to do this,such as

620 Chapter17. MarkovandhiddenMarkovmodels
• Use some fully labeled data to initialize the parameters.
• Initially ignore the Markov dependencies, and estimate the observation parameters using the
standard mixture model estimation methods, such as K-means or EM.
• Randomly initialize the parameters, use multiple restarts, and pick the best solution.
Techniques such as deterministic annealing (Ueda and Nakano 1998; Rao and Rose 2001)
can help mitigate the effect of local minima. Also, just as K-means is often used to initialize
EM for GMMs, so it is common to initialize EM for HMMs using Viterbi training, which
means approximating the posterior over paths with the single most probable path. (This is notnecessarily a good idea, since initially the parameters are often poorly estimated, so the Viterbipath will be fairly arbitrary. A safer option is to start training using forwards-backwards, and toswitch to Viterbi near convergence.)
17.5.3 Bayesian methods for “ﬁtting” HMMs *

EM returns a MAP estimate of the parameters. In this section, we brieﬂy discuss some methodsfor Bayesian parameter estimation in HMMs. (These methods rely on material that we will coverlater in the book.)
One approach is to use variational Bayes EM (VBEM), which we discuss in general terms in

Section 21.6. The details for the HMM case can be found in (MacKay 1997; Beal 2003), butthe basic idea is this: The E step uses forwards-backwards, but where (roughly speaking) weplug in the posterior mean parameters instead of the MAP estimates. The M step updates theparameters of the conjugate posteriors, instead of updating the parameters themselves.
An alternative to VBEM is to use MCMC. A particularly appealing algorithm is block Gibbs
sampling, which we discuss in general terms in Section 24.2.8. The details for the HMM casecan be found in (Fruhwirth-Schnatter 2007), but the basic idea is this: we sample z
1:Tgiven

1:Tgiven
the data and parameters using forwards-ﬁltering, backwards-sampling, and we then sample theparameters from their posteriors, conditional on the sampled latent paths. This is simple toimplement, but one does need to take care of unidentiﬁability (label switching), just as withmixture models (see Section 11.3.1).
17.5.4 Discriminative training
Sometimes HMMs are used as the class conditional density inside a generative classiﬁer. In thiscase,p(x|y=c,θ)can be computed using the forwards algorithm. We can easily maximize the
joint likelihood∏
N
i=1p(xi,yi|θ)by using EM (or some other method) to ﬁt the HMM for each
class-conditional density separately.
However, we might like to ﬁnd the parameters that maximize the conditional likelihood
N∏

N
N∏
i=1p(yi|xi,θ)=∏
ip(yi|θ)p(xi|yi,θ)∑
cp(yi=c|θ)p(xi|c,θ)(17.109)
This is more expensive than maximizing the joint likelihood, since the denominator couples all C
class-conditional HMMs together. Furthermore, EM can no longer be used, and one must resort

17.6. GeneralizationsofHMMs 621
to generic gradient based methods. Nevertheless, discriminative training can result in improved
accuracies. The standard practice in speech recognition is to initially train the generative modelsseparately using EM, and then to ﬁne tune them discriminatively (Jelinek 1997).
17.5.5 Model selection
In HMMs, the two main model selection issues are: how many states, and what topology to usefor the state transition diagram. We discuss both of these issues below.
17.5.5.1 Choosing the number of hidden states
Choosing the number of hidden states Kin an HMM is analogous to the problem of choosing
the number of mixture components. Here are some possible solutions:
• Use grid-search over a range of K’s, using as an objective function cross-validated likelihood,
the BIC score, or a variational lower bound to the log-marginal likelihood.
• Use reversible jump MCMC. See (Fruhwirth-Schnatter 2007) for details. Note that this is very
slow and is not widely used.

• Use variational Bayes to “extinguish” unwanted components, by analogy to the GMM case
discussed in Section 21.6.1.6. See (MacKay 1997; Beal 2003) for details.
• Use an “inﬁnite HMM”, which is based on the hierarchical Dirichlet process. See e.g., (Beal
et al. 2002; Teh et al. 2006) for details.
17.5.5.2 Structure learning
The term structure learning in the context of HMMs refers to learning a sparse transition
matrix. That is, we want to learn the structure of the state transition diagram, not the structure

of the graphical model (which is ﬁxed). A large number of heuristic methods have been proposed.Most alternate between parameter estimation and some kind of heuristic split merge method
(see e.g., (Stolcke and Omohundro 1992)).
Alternatively, one can pose the problem as MAP estimation using a minimum entropy prior,
of the form
p(A
i,:)∝exp(− H(Ai,:)) (17.110)

of the form
p(A
This prior prefers states whose outgoing distribution is nearly deterministic, and hence has lowentropy (Brand 1999). The corresponding M step cannot be solved in closed form, but numericalmethods can be used. The trouble with this is that we might prune out all incoming transitionsto a state, creating isolated “islands” in state-space. The inﬁnite HMM presents an interestingalternative to these methods. See e.g., (Beal et al. 2002; Teh et al. 2006) for details.
17.6 Generalizations of HMMs

Many variants of the basic HMM model have been proposed. We brieﬂy discuss some of thembelow.

622 Chapter17. MarkovandhiddenMarkovmodels
Dt−1 DtDt+1
Qt−1 QtQt−1
Xt−1 XtXt+1
Figure 17.14 Encoding a hidden semi-Markov model as a DGM. Dtare deterministic duration counters.
17.6.1 Variable duration (semi-Markov) HMMs
In a standard HMM, the probability we remain in state ifor exactly dsteps is
p(ti=d)=( 1−Aii)Ad
ii∝exp(dlogAii) (17.111)
whereAiiis the self-loop probability. This is called the geometric distribution. However, this
kind of exponentially decaying function of dis sometimes unrealistic.
To allow for more general durations, one can use a semi-Markov model. It is called semi-
Markov because to predict the next state, it is not sufficient to condition on the past state: we
also need to know how long we’ve been in that state. When the state space is not observeddirectly, the result is called a hidden semi-Markov model (HSMM), a variable duration HMM,
or anexplicit duration HMM.
HSMMs are widely used in many gene ﬁnding programs, since the length distribution of

exons and introns is not geometric (see e.g., (Schweikerta et al. 2009)), and in some chip-Seqdata analysis programs (see e.g., (Kuan et al. 2009)).
HSMMs are useful not only because they can model the waiting time of each state more
accurately, but also because they can model the distribution of a whole batch of observations atonce, instead of assuming all observations are conditionally iid. That is, they can use likelihoodmodels of the form p(x
t:t+l|zt=k,dt=l), which generate lcorrelated observations if the

duration in state kis forltime steps. This is useful for modeling data that is piecewise linear,
or shows other local trends (Ostendorf et al. 1996).
17.6.1.1 HSMM as augmented HMMs
One way to represent a HSMM is to use the graphical model shown in Figure 17.14. (In thisﬁgure, we have assumed the observations are iid within each state, but this is not required,as mentioned above.) The D
t∈{0,1,...,D}node is a state duration counter, where Dis
the maximum duration of any state. When we ﬁrst enter state j, we sample Dtfrom the
duration distribution for that state, Dt∼pj(·). Thereafer, Dtdeterministically counts down

17.6. GeneralizationsofHMMs 623
1 2 3 41−p 1−p 1−pp p p p
(a)0 100 200 300 400 500 60000.0020.0040.0060.0080.010.012
  
n=1
n=2
n=5
(b)
Figure 17.15 (a) A Markov chain with n=4repeated states and self loops. (b) The resulting distribution
over sequence lengths, for p=0.99and various n. Figure generated by hmmSelfLoopDist .
untilDt=0. WhileDt>0, the state ztis not allowed to change. When Dt=0, we make a
stochastic transition to a new state.
More precisely, we deﬁne the CPDs as follows:
p(Dt=d′|Dt−1=d,zt=j)=⎧
⎨
⎩pj(d′)ifd=0
1ifd′=d−1andd≥1
0otherwise(17.112)
p(zt=k|zt−1=j,Dt−1=d)=⎧
⎨
⎩1ifd>0andj=k
Ajkifd=0
0otherwise(17.113)
Note that pj(d)could be represented as a table (a non-parametric approach) or as some kind
of parametric distribution, such as a Gamma distribution. If pj(d)is a geometric distribution,
this emulates a standard HMM.
One can perform inference in this model by deﬁning a mega-variable Yt=(Dt,zt). However,

⎨
⎨
Ajkifd=0
this is rather inefficient, since Dtis deterministic. It is possible to marginalize Dtout, and derive
special purpose inference procedures. See (Guedon 2003; Yu and Kobayashi 2006) for details.
Unfortunately, all these methods take O(TK2D)time, where Tis the sequence length, Kis
the number of states, and Dis the maximum duration of any state.
17.6.1.2 Approximations to semi-Markov models

A more efficient, but less ﬂexible, way to model non-geometric waiting times is to replace eachstate with nnew states, each with the same emission probabilities as the original state. For
example, consider the model in Figure 17.15(a). Obviously the smallest sequence this can generateis of length n=4. Any path of length dthrough the model has probability p
d−n(1−p)n;
multiplying by the number of possible paths we ﬁnd that the total probability of a path of lengthdis
p(d)=(d−1
n−1)
p
d−n(1−p)n(17.114)

624 Chapter17. MarkovandhiddenMarkovmodels
need on the words
phones
sub-
phonesaa n end n iy d dh
nax
iyend
end
end end
Figure 17.16 An example of an HHMM for an ASR system which can recognize 3 words. The top level
represents bigram word probabilities. The middle level represents the phonetic spelling of each word. The
bottom level represents the subphones of each phone. (It is traditional to represent a phone as a 3 stateHMM, representing the beginning, middle and end.) Based on Figure 7.5 of (Jurafsky and Martin 2000).
This is equivalent to the negative binomial distribution. By adjusting nand the self-loop
probabilities pof each state, we can model a wide range of waiting times: see Figure 17.15(b).
LetEbe the number of expansions of each state needed to approximate pj(d). Forwards-
backwards on this model takes O(T(KE)Fin)time, where Finis the average number of
predecessor states, compared to O(TK(Fin+D))for the HSMM. For typical speech recognition

applications, Fin∼3,D∼50,K∼106,T∼105. (Similar ﬁgures apply to problems such
as gene ﬁnding, which also often uses HSMMs.) Since Fin+D≫EFin, the expanded state
method is much faster than an HSMM. See (Johnson 2005) for details.
17.6.2 Hierarchical HMMs
Ahierarchical HMM (HHMM) (Fine et al. 1998) is an extension of the HMM that is designed to
model domains with hierarchical structure. Figure 17.16 gives an example of an HHMM used in

automatic speech recognition. The phone and subphone models can be “called” from differenthigher level contexts. We can always “ﬂatten” an HHMM to a regular HMM, but a factoredrepresentation is often easier to interpret, and allows for more efficient inference and modelﬁtting.
HHMMs have been used in many application domains, e.g., speech recognition (Bilmes 2001),

gene ﬁnding (Hu et al. 2000), plan recognition (Bui et al. 2002), monitoring transportationpatterns (Liao et al. 2007), indoor robot localization (Theocharous et al. 2004), etc. HHMMs areless expressive than stochastic context free grammars (SCFGs), since they only allow hierarchiesof bounded depth, but they support more efficient inference. In particular, inference in SCFGs(using the inside outside algorithm, (Jurafsky and Martin 2008)) takes O(T
3)whereas inference
in an HHMM takes O(T)time (Murphy and Paskin 2001).

We can represent an HHMM as a directed graphical model as shown in Figure 17.17. Qℓ
t
represents the state at time tand level ℓ. A state transition at level ℓis only “allowed” if the

17.6. GeneralizationsofHMMs 625
F1
1 F1
2 F1
3
Q1
1 Q12 Q13
F2
1 F2
2 F2
3
Q2
1 Q22 Q23
F3
1 F3
2 F3
3
Q3
1 Q32 Q33
Y1 Y2 Y3
Figure 17.17 An HHMM represented as a DGM. Qℓ
tis the state at time t,l e v e lℓ;Fℓ
t=1if the HMM at
levelℓhas ﬁnished (entered its exit state), otherwise Fℓ
t=0. Shaded nodes are observed; the remaining
nodes are hidden. We may optionally clamp Fℓ
T=1,w h e r eTis the length of the observation sequence,
to ensure all models have ﬁnished by the end of the sequence. Source: Figure 2 of (Murphy and Paskin
2001).
chain at the level below has “ﬁnished”, as determined by the Fℓ−1
tnode. (The chain below
ﬁnishes when it chooses to enter its end state.) This mechanism ensures that higher level chains
evolve more slowly than lower level chains, i.e., lower levels are nested within higher levels.
A variable duration HMM can be thought of as a special case of an HHMM, where the top

2001).
level is a deterministic counter, and the bottom level is a regular HMM, which can only changestates once the counter has “timed out”. See (Murphy and Paskin 2001) for further details.
17.6.3 Input-output HMMs
It is straightforward to extend an HMM to handle inputs, as shown in Figure 17.18(a). This deﬁnesa conditional density model for sequences of the form
p(y
1:T,z1:T|u1:T,θ) (17.115)
whereutis the input at time t; this is sometimes called a control signal. If the inputs and
outputs are continuous, a typical parameterization would be

p(y
p(zt|xt,zt−1=i,θ)=C a t ( zt|S(Wiut)) (17.116)
p(yt|xt,zt=j,θ)=N (yt|Vjut,Σj) (17.117)
Thus the transition matrix is a logistic regression model whose parameters depend on theprevious state. The observation model is a Gaussian whose parameters depend on the current

626 Chapter17. MarkovandhiddenMarkovmodels
yt−1 ytzt−1 ztut−1 ut
(a)x1x2 xTz1z2 zT
(b) (c)
Figure 17.18 (a) Input-output HMM. (b) First-order auto-regressive HMM. (c) A second-order buried Markov
model. Depending on the value of the hidden variables, the effective graph structure between the com-
ponents of the observed variables (i.e., the non-zero elements of the regression matrix and the precisionmatrix) can change, although this is not shown.
state. The whole model can be thought of as a hidden version of a maximum entropy Markov
model (Section 19.6.1).
Conditional on the inputs u1:Tand the parameters θ, one can apply the standard forwards-
backwards algorithm to estimate the hidden states. It is also straightforward to derive an EMalgorithm to estimate the parameters (see (Bengio and Frasconi 1996) for details).
17.6.4 Auto-regressive and buried HMMs

The standard HMM assumes the observations are conditionally independent given the hiddenstate. In practice this is often not the case. However, it is straightforward to have direct arcs fromx
t−1toxtas well as from zttoxt, as in Figure 17.18(b). This is known as an auto-regressive
HMM,o raregime switching Markov model. For continuous data, the observation modelbecomes
p(x
t|xt−1,zt=j,θ)=N(xt|Wjxt−1+μj,Σj) (17.118)
This is a linear regression model, where the parameters are chosen according to the currenthidden state. We can also consider higher-order extensions, where we condition on the last L

p(x
observations:
p(x
t|xt−L:t−1,zt=j,θ)=N(xt|L∑
ℓ=1Wj,ℓxt−ℓ+μj,Σj) (17.119)
Such models are widely used in econometrics (Hamilton 1990). Similar models can be deﬁnedfor discrete observations.
The AR-HMM essentially combines two Markov chains, one on the hidden variables, to capture
long range dependencies, and one on the observed variables, to capture short range dependen-cies (Berchtold 1999). Since the Xnodes are observed, the connections between them only

17.6. GeneralizationsofHMMs 627
x1 x2 x3z3,1 z3,2 z3,3z2,1 z2,2 z2,3z1,1 z1,2 z1,3
(a)x31 x32 x33z31 z32 z33x21 x22 x23z21 z22 z23x11 x12 x13z11 z12 z13
(b)
Figure 17.19 (a) A factorial HMM with 3 chains. (b) A coupled HMM with 3 chains.
change the computation of the local evidence; inference can still be performed using the stan-
dard forwards-backwards algorithm. Parameter estimation using EM is also straightforward: theE step is unchanged, as is the M step for the transition matrix. If we assume scalar observationsfor notational simplicty, the M step involves minimizing
∑
tE[1
σ2(st)(yt−yT
t−L:t−1w(st))2+logσ2(st)]
(17.120)
Focussing on the wterms, we see that this requires solving Kweighted least squares problems:
J(w1:K)=∑
j∑
tγt(j)
σ2(j)(yt−yT
t−L:t−1wj)2(17.121)
whereγt(j)=p(zt=k|x1:T)is the smoothed posterior marginal. This is a weighted linear

∑
tE[1
(17.120)
J(w1:K)=∑
j∑
tγt(j)
regression problem, where the design matrix has a Toeplitz form. This subproblem can be solvedefficiently using the Levinson-Durbin method (Durbin and Koopman 2001).
Buried Markov models generalize AR-HMMs by allowing the dependency structure between
the observable nodes to change based on the hidden state, as in Figure 17.18(c). Such a modelis called a dynamic Bayesian multi net, since it is a mixture of different networks. In the
linear-Gaussian setting, we can change the structure of the of x
t−1→xtarcs by using sparse
regression matrices, Wj, and we can change the structure of the connections within the
components of xtby using sparse Gaussian graphical models, either directed or undirected. See
(Bilmes 2000) for details.

17.6.5 Factorial HMM
An HMM represents the hidden state using a single discrete random variable zt∈{1,...,K}.
To represent 10 bits of information would require K=210= 1024states. By contrast, consider
adistributed representation of the hidden state, where each zc,t∈{0,1}represents the c’th

628 Chapter17. MarkovandhiddenMarkovmodels
bit of the t’th hidden state. Now we can represent 10 bits using just 10 binary variables, as
illustrated in Figure 17.19(a). This model is called a factorial HMM (Ghahramani and Jordan
1997). The hope is that this kind of model could capture different aspects of a signal, e.g., one
chain would represent speaking style, another the words that are being spoken.
Unfortunately, conditioned on xt, all the hidden variables are correlated (due to explaining
away the common observed child xt). This make exact state estimation intractable. However,
we can derive efficient approximate inference algorithms, as we discuss in Section 21.4.1.
17.6.6 Coupled HMM and the inﬂuence model
If we have multiple related data streams, we can use a coupled HMM (Brand 1996), as illustrated
in Figure 17.19(b). This is a series of HMMs where the state transitions depend on the states ofneighboring chains. That is, we represent the joint conditional distribution as
p(z
t|zt−1)=∏

p(z
t|zt−1)=∏
cp(zct|zt−1) (17.122)
p(zct|zt−1)=p( zct|zc,t−1,zc−1,t−1,zc+1,t−1) (17.123)
This has been used for various tasks, such as audio-visual speech recognition (Neﬁan et al.
2002) and modeling freeway traffic ﬂows (Kwon and Murphy 2000).
The trouble with the above model is that it requires O(CK4)parameters to specify, if there
areCchains with Kstates per chain, because each state depends on its own past plus the
past of its two neighbors. There is a closely related model, known as the inﬂuence model

t|zt−1)=∏
(Asavathiratham 2000), which uses fewer parameters. It models the joint conditional distributionas
p(z
ct|zt−1)=C∑
c′=1αc,c′p(zct|zc′,t−1) (17.124)
where∑
c′αc,c′=1for eachc. That is, we use a convex combination of pairwise transition
matrices. The αc,c′parameter speciﬁes how much inﬂuence chain chas on chain c′. This
model only takes O(C2+CK2)parameters to specify. Furthermore, it allows each chain to

p(z
where∑
be inﬂuenced by all the other chains, not just its nearest neighbors. (Hence the correspondinggraphical model is similar to Figure 17.19(b), except that each node has incoming edges fromall the previous nodes.) This has been used for various tasks, such as modeling conversationalinteractions between people (Basu et al. 2001).
Unfortunately, inference in both of these models takes O(T(K
C)2)time, since all the chains

become fully correlated even if the interaction graph is sparse. Various approximate inferencemethods can be applied, as we discuss later.
17.6.7 Dynamic Bayesian networks (DBNs)
Adynamic Bayesian network is just a way to represent a stochastic process using a directed
graphical model.7Note that the network is not dynamic (the structure and parameters are ﬁxed),
7. The acronym DBNcan stand for either “dynamic Bayesian network” or “deep belief network” (Section 28.1) depending

on the context. Geoff Hinton (who invented the term “deep belief network”) has suggested the acronyms DyBNand
DeeBNto avoid this ambiguity.

17.6. GeneralizationsofHMMs 629
1471917162023212527
6
12
13
10
8
4
132826222418
15
119
5
2SensorValid1
FYdotDiff1
FcloseSlow1Xdot0 Xdot1
InLane0 InLane1LeftClr0 LeftClr1
RightClr0 RightClr1
LatAction0 LatAction1
FwdAction0 FwdAction1
Ydot0 Ydot1
Stopped0 Stopped1BXdot1
EngStatus0 EngStatus1 BcloseFast1
FrontBackStatus0 FrontBackStatus1 BYdotDiff1Fclr1
Bclr1XdotSens1
YdotSens1LeftClrSens1
RightClrSens1
TurnSignal1
FYdotDiffSens1
FclrSens1
BXdotSens1
BclrSens1
BYdotDiffSens1
slice ts lice t+1 evidence
Figure 17.20 The BATnet DBN. The transient nodes are only shown for the second slice, to minimize
clutter. The dotted lines can be ignored. Used with kind permission of Daphne Koller.
rather it is a network representation of a dynamical system. All of the HMM variants we have

FclrSens1
BclrSens1
seen above could be considered to be DBNs. However, we prefer to reserve the term “DBN”for graph structures that are more “irregular” and problem-speciﬁc. An example is shown inFigure 17.20, which is a DBN designed to monitor the state of a simulated autonomous carknown as the “Bayesian Automated Taxi”, or “BATmobile” (Forbes et al. 1995).
Deﬁning DBNs is straightforward: you just need to specify the structure of the ﬁrst time-slice,

the structure between two time-slices, and the form of the CPDs. Learning is also easy. Themain problem is that exact inference can be computationally expensive, because all the hiddenvariables become correlated over time (this is known as entanglement — see e.g., (Koller and
Friedman 2009, Sec. 15.2.4) for details). Thus a sparse graph does not necessarily result intractable exact inference. However, later we will see algorithms that can exploit the graphstructure for efficient approximate inference.
Exercises
Exercise 17.1 Derivation of Qfunction for HMM
Derive Equation 17.97.

Exercises
Exercise 17.2 Two ﬁlter approach to smoothing in HMMs
Assuming that Πt(i)=p(St=i)>0for alliandt, derive a recursive algorithm for updating rt(i)=
p(St=i|xt+1:T). Hint: it should be very similar to the standard forwards algorithm, but using a time-
reversed transition matrix. Then show how to compute the posterior marginals γt(i)=p(St=i|x1:T)

630 Chapter17. MarkovandhiddenMarkovmodels
from the backwards ﬁltered messages rt(i), the forwards ﬁltered messages αt(i), and the stationary
distribution Πt(i).
Exercise 17.3 EM for for HMMs with mixture of Gaussian observations
Consider an HMM where the observation model has the form
p(xt|zt=j,θ)=∑
kwjkN(xt|μjk,Σjk) (17.125)
• Draw the DGM.
• Derive the E step.• Derive the M step.
Exercise 17.4 EM for for HMMs with tied mixtures
In many applications, it is common that the observations are high-dimensional vectors (e.g., in speech
recognition, x
tis often a vector of cepstral coefficients and their derivatives, so xt∈R39), so estimating a
full covariance matrix for KMvalues (where Mis the number of mixture components per hidden state),
as in Exercise 17.3, requires a lot of data. An alternative is to use just MGaussians, rather than MK
Gaussians, and to let the state inﬂuence the mixing weights but not the means and covariances. This is
called asemi-continuous HMM ortied-mixture HMM.

• Draw the corresponding graphical model.

18 State space models
18.1 Introduction
Astate space model orSSMis just like an HMM, except the hidden states are continuous. The
model can be written in the following generic form:
zt=g(ut,zt−1,ϵt) (18.1)
yt=h(zt,ut,δt) (18.2)
whereztis the hidden state, utis an optional input or control signal, ytis the observation, g
is thetransition model, his theobservation model, ϵtis the system noise at time t, andδt
is the observation noise at time t. We assume that all parameters of the model, θ, are known;
if not, they can be included into the hidden state, as we discuss below.
OneoftheprimarygoalsinusingSSMsistorecursivelyestimatethebeliefstate, p(zt|y1:t,u1:t,θ).
(Note: we will often drop the conditioning on uandθfor brevity.) We will discuss algorithms for
this later in this chapter. We will also discuss how to convert our beliefs about the hidden state
into predictions about future observables by computing the posterior predictive p(yt+1|y1:t).

An important special case of an SSM is where all the CPDs are linear-Gaussian. In other
words, we assume
• The transition model is a linear function
zt=Atzt−1+Btut+ϵt (18.3)
• The observation model is a linear function
yt=Ctzt+Dtut+δt (18.4)
• The system noise is Gaussian
ϵt∼N(0,Qt) (18.5)
• The observation noise is Gaussian
δt∼N(0,Rt) (18.6)
This model is called a linear-Gaussian SSM (LG-SSM)o ralinear dynamical system (LDS).
If the parameters θt=(At,Bt,Ct,Dt,Qt,Rt)are independent of time, the model is called
stationary.

632 Chapter18. Statespacemodels
10 12 14 16 18 20 22468101214
  
observed
truth
(a)8 10 12 14 16 18 20 22 2446810121416
  
observed
filtered
(b)10 15 20 25468101214
  
observed
smoothed
(c)
Figure 18.1 Illustration of Kalman ﬁltering and smoothing. (a) Observations (green cirles) are generated
by an object moving to the right (true location denoted by black squares). (b) Filtered estimated is shown
by dotted red line. Red cross is the posterior mean, blue circles are 95% conﬁdence ellipses derived fromthe posterior covariance. For clarity, we only plot the ellipses every other time step. (c) Same as (b), butusing offline Kalman smoothing. Figure generated by kalmanTrackingDemo .
The LG-SSM is important because it supports exact inference, as we will see. In particular,
if the initial belief state is Gaussian, p(z1)=N(μ1|0,Σ1|0), then all subsequent belief states
will also be Gaussian; we will denote them by p(zt|y1:t)=N(μt|t,Σt|t). (The notation μt|τ

denotes E[zt|y1:τ], and similarly for Σt|t; thusμt|0denotes the prior for z1before we have
seen any data. For brevity we will denote the posterior belief states using μt|t=μtand
Σt|t=Σt.) We can compute these quantities efficiently using the celebrated Kalman ﬁlter,
as we show in Section 18.3.1. But before discussing algorithms, we discuss some important
applications.
18.2 Applications of SSMs
SSMs have many applications, some of which we discuss in the sections below. We mostlyfocus on LG-SSMs, for simplicity, although non-linear and/or non-Gaussian SSMs are even morewidely used.
18.2.1 SSMs for object tracking

One of the earliest applications of Kalman ﬁltering was for tracking objects, such as airplanesand missiles, from noisy measurements, such as radar. Here we give a simpliﬁed example toillustrate the key ideas. Consider an object moving in a 2D plane. Let z
1tandz2tbe the
horizontal and vertical locations of the object, and ˙z1tand˙z2tbe the corresponding velocity.
We can represent this as a state vector zt∈R4as follows:
zT
t=(z1tz2t˙z1t˙z2t)
. (18.7)

18.2. ApplicationsofSSMs 633
Let us assume that the object is moving at constant velocity, but is “perturbed” by random
Gaussian noise (e.g., due to the wind). Thus we can model the system dynamics as follows:
zt=Atzt−1+ϵt (18.8)⎛
⎜⎜⎝z1t
z2t
˙z1t
˙z2t⎞
⎟⎟⎠=⎛
⎜⎜⎝10Δ0
010Δ001 0000 1⎞
⎟⎟⎠⎛
⎜⎜⎝z
1,t−1
z2,t−1
˙z1,t−1
˙z2,t−1⎞
⎟⎟⎠+⎛
⎜⎜⎝ϵ1t
ϵ2t
ϵ3t
ϵ4t⎞
⎟⎟⎠(18.9)
whereϵt∼N(0,Q)is the system noise, and Δis thesampling period. This says that the
new location zj,tis the old location zj,t−1plusΔtimes the old velocity ˙zj,t−1, plus random
noise,ϵjt,f o rj=1:2. Also, the new velocity ˙zj,tis the old velocity ˙zj,t−1plus random
noise,ϵjt,f o rj=3:4. This is called a random accelerations model, since the object moves
according to Newton’s laws, but is subject to random changes in velocity.
Now suppose that we can observe the location of the object but not its velocity. Let yt∈R2
represent our observation, which we assume is subject to Gaussian noise. We can model this as
follows:
yt=Ctzt+δt (18.10)
(y1t
y2t)
=(10000100)⎛

follows:
(y1t
y2t)
⎜⎜⎝z
1t
z2t
˙z1t
˙z2t⎞
⎟⎟⎠+⎛
⎜⎜⎝δ1t
δ2t
δ3t
δ4t⎞
⎟⎟⎠(18.11)
whereδt∼N(0,R)is the measurement noise.
Finally, we need to specify our initial (prior) beliefs about the state of the object, p(z1).W e
will assume this is a Gaussian, p(z1)=N(z1|μ1|0,Σ1|0). We can represent prior ignorance by
makingΣ1|0suitably “broad”, e.g., Σ1|0=∞I. We have now fully speciﬁed the model and can
perform sequential Bayesian updating to compute p(zt|y1:t)using an algorithm known as the
Kalman ﬁlter, to be described in Section 18.3.1.

1t
z2t
˙z1t
˙z2t⎞
⎟⎟⎠+⎛
⎜⎜⎝δ1t
δ2t
δ3t
δ4t⎞
Figure 18.1(a) gives an example. The object moves to the right and generates an observation
at each time step (think of “blips” on a radar screen). We observe these blips and ﬁlter out
the noise by using the Kalman ﬁlter. At every step, we have p(zt|y1:t), from which we can
compute p(z1t,z2t|y1:t)by marginalizing out the dimensions corresponding to the velocities.
(This is easy to do since the posterior is Gaussian.) Our “best guess” about the location of theobject is the posterior mean, E[z

t|y1:t], denoted as a red cross in Figure 18.1(b). Our uncertainty
associated with this is represented as an ellipse, which contains 95% of the probability mass. Wesee that our uncertainty goes down over time, as the effects of the initial uncertainty get “washedout”. We also see that the estimated trajectory has “ﬁltered out” some of the noise. To obtainthe much smoother plot in Figure 18.1(c), we need to use the Kalman smoother, which computesp(z

t|y1:T); this depends on “future” as well as “past” data, as discussed in Section 18.3.2.
18.2.2 Robotic SLAM
Consider a robot moving around an unknown 2d world. It needs to learn a map and keeptrack of its location within that map. This problem is known as simultaneous localization and

634 Chapter18. Statespacemodels
L2Y1 Y2 YTY1 Y3
X1 X2 X3
...XTL1
Figure 18.2 Illustration of graphical model underlying SLAM. Liis the ﬁxed location of landmark i,xt
is the location of the robot, and ytis the observation. In this trace, the robot sees landmarks 1 and 2 at
time step 1, then just landmark 2, then just landmark 1, etc. Based on Figure 15.A.3 of (Koller and Friedman
2009).
Robot pose
(a) (b)
Figure 18.3 Illustration of the SLAM problem. (a) A robot starts at the top left and moves clockwise in a
circle back to where it started. We see how the posterior uncertainty about the robot’s location increases
and then decreases as it returns to a familar location, closing the loop. If we performed smoothing, this
new information would propagate backwards in time to disambiguate the entire trajectory. (b) We show the
precision matrix, representing sparse correlations between the landmarks, and between the landmarks and

the robot’s position (pose). This sparse precision matrix can be visualized as a Gaussian graphical model,
as shown. Source: Figure 15.A.3 of (Koller and Friedman 2009) . Used with kind permission of Daphne
Koller.

18.2. ApplicationsofSSMs 635
mapping,o r SLAMfor short, and is widely used in mobile robotics, as well as other applications
such as indoor navigation using cellphones (since GPS does not work inside buildings).
Let us assume we can represent the map as the 2d locations of a ﬁxed set of Klandmarks,
denote them by L1,...,LK(each is a vector in R2). For simplicity, we will assume these are
uniquely identiﬁable. Let xtrepresent the unknown location of the robot at time t. We deﬁne
the state space to be zt=(xt,L1:K); we assume the landmarks are static, so their motion
model is a constant, and they have no system noise. If ytmeasures the distance from xtto
the set of closest landmarks, then the robot can update its estimate of the landmark locations
based on what it sees. Figure 18.2 shows the corresponding graphical model for the case whereK=2, and where on the ﬁrst step it sees landmarks 1 and 2, then just landmark 2, then just
landmark 1, etc.
If we assume the observation model p(y

t|zt,L)is linear-Gaussian, and we use a Gaussian
motion model for p(xt|xt−1,ut), we can use a Kalman ﬁlter to maintain our belief state about
the location of the robot and the location of the landmarks (Smith and Cheeseman 1986; Chosetand Nagatani 2001).
Over time, the uncertainty in the robot’s location will increase, due to wheel slippage etc.,

but when the robot returns to a familiar location, its uncertainty will decrease again. This iscalledclosing the loop, and is illustrated in Figure 18.3(a), where we see the uncertainty ellipses,
representing cov[x
t|y1:t,u1:t], grow and then shrink. (Note that in this section, we assume that
a human is joysticking the robot through the environment, so u1:tis given as input, i.e., we do
not address the decision-theoretic issue of choosing where to explore.)
Since the belief state is Gaussian, we can visualize the posterior covariance matrix Σt. Ac-

tually, it is more interesting to visualize the posterior precision matrix, Λt=Σ−1
t, since that
is fairly sparse, as shown in Figure 18.3(b). The reason for this is that zeros in the precisionmatrix correspond to absent edges in the corresponding undirected Gaussian graphical model(see Section 19.4.4). Initially all the landmarks are uncorrelated (assuming we have a diagonalprior onL), so the GGM is a disconnected graph, and Λ
tis diagonal. However, as the robot

moves about, it will induce correlation between nearby landmarks. Intuitively this is because therobot is estimating its position based on distance to the landmarks, but the landmarks’ locationsare being estimated based on the robot’s position, so they all become inter-dependent. This canbe seen more clearly from the graphical model in Figure 18.2: it is clear that L
1andL2are not
d-separated by y1:t, because there is a path between them via the unknown sequence of x1:t

nodes. As a consequence of the precision matrix becoming denser, exact inference takes O(K3)
time. (This is an example of the entanglement problem for inference in DBNs.) This preventsthe method from being applied to large maps.
There are two main solutions to this problem. The ﬁrst is to notice that the correlation pattern

moves along with the location of the robot (see Figure 18.3(b)). The remaining correlationsbecome weaker over time. Consequently we can dynamically “prune out” weak edges fromthe GGM using a technique called the thin junction tree ﬁlter (Paskin 2003) (junction trees areexplained in Section 20.4).
A second approach is to notice that, conditional on knowing the robot’s path, x
1:t, the
landmark locations are independent. That is, p(L|x1:t,y1:t)=∏K
k=1p(Lk|x1:t,y1:t). This
forms the basis of a method known as FastSLAM, which combines Kalman ﬁltering and particle

1:t, the
ﬁltering, as discussed in Section 23.6.3.
(Thrun et al. 2006) provides a more detailed account of SLAM and mobile robotics.

636 Chapter18. Statespacemodels
xt−1 xtyt−1 ytθt−1 θt
(a)0 5 10 15 20 25−8−6−4−2024
  
w0
w1
w0 batch
w1 batch
(b)
Figure 18.4 (a) A dynamic generalization of linear regression. (b) Illustration of the recursive least squares
algorithm applied to the model p(y|x,θ)=N(y|w0+w1x,σ2). We plot the marginal posterior of w0
andw1vs number of data points. (Error bars represent E[wj|y1:t]±√
var[wj|y1:t].) After seeing all
the data, we converge to the offline ML (least squares) solution, represented by the horizontal lines. Figure
generated by linregOnlineDemoKalman .
18.2.3 Online parameter learning using recursive least squares
We can perform online Bayesian inference for the parameters of various statistical models using
SSMs. In this section, we focus on linear regression; in Section 18.5.3.2, we discuss logisticregression.
The basic idea is to let the hidden state represent the regression parameters, and to let the

(time-varying) observation model represent the current data vector. In more detail, deﬁne theprior to be p(θ)=N(θ|θ
0,Σ0). (If we want to do online ML estimation, we can just set
Σ0=∞I.) Let the hidden state be zt=θ; if we assume the regression parameters do not
change, we can set At=IandQt=0I,s o
p(θt|θt−1)=N(θt|θt−1,0I)=δθt−1(θt) (18.12)
(If we do let the parameters change over time, we get a so-called dynamic linear model
(Harvey 1990; West and Harrison 1997; Petris et al. 2009).) Let Ct=xT

t, andRt=σ2, so the
(non-stationary) observation model has the form
N(yt|Ctzt,Rt)=N(yt|xT
tθt,σ2) (18.13)
Applying the Kalman ﬁlter to this model provides a way to update our posterior beliefs about
the parameters as the data streams in. This is known as the recursive least squares orRLS
We can derive an explicit form for the updates as follows. In Section 18.3.1, we show that the
Kalman update for the posterior mean has the form
μt=Atμt−1+Kt(yt−CtAtμt−1) (18.14)

18.2. ApplicationsofSSMs 637
whereKtis known as the Kalman gain matrix. Based on Equation 18.39, one can show that
Kt=ΣtCT
tR−1
t. In this context, we have Kt=Σtxt/σ2. Hence the update for the
parameters becomes
ˆθt=ˆθt−1+1
σ2Σt|t(yt−xT
tˆθt−1)xt (18.15)
If we approximate1
σ2Σt|t−1withηtI, we recover the least mean squares orLMSalgorithm,
discussed in Section 8.5.3. In LMS, we need to specify how to adapt the update parameter
ηtto ensure convergence to the MLE. Furthermore, the algorithm may take multiple passes
through the data. By contrast, the RLS algorithm automatically performs step-size adaptation,and converges to the optimal posterior in one pass over the data. See Figure 18.4 for an example.
18.2.4 SSM for time series forecasting *

SSMs are very well suited for time-series forecasting, as we explain below. We focus on the caseof scalar (one dimensional) time series, for simplicity. Our presentation is based on (Varian 2011).See also (Aoki 1987; Harvey 1990; West and Harrison 1997; Durbin and Koopman 2001; Petriset al. 2009; Prado and West 2010) for good books on this topic.
At ﬁrst sight, it might not be apparent why SSMs are useful, since the goal in forecasting is

to predict future visible variables, not to estimate hidden states of some system. Indeed, mostclassical methods for time series forecasting are just functions of the form ˆy
t+1=f(y1:t,θ),
where hidden variables play no role (see Section 18.2.4.4). The idea in the state-space approach totime series is to create a generative model of the data in terms of latent processes, which capturedifferent aspects of the signal. We can then integrate out the hidden variables to compute theposterior predictive of the visibles.

Since the model is linear-Gaussian, we can just add these processes together to explain the
observed data. This is called a structural time series model. Below we explain some of the
basic building blocks.
18.2.4.1 Local level model
The simplest latent process is known as the local level model, which has the form
y
t=at+ϵy
t,ϵyt∼N(0,R) (18.16)
at=at−1+ϵa
t,ϵat∼N(0,Q) (18.17)
where the hidden state is just zt=at. This model asserts that the observed data yt∈Ris
equal to some unknown level term at∈R, plus observation noise with variance R. In addition,
the levelatevolves over time subject to system noise with variance Q. See Figure 18.5 for some
examples.

638 Chapter18. Statespacemodels
at−1 at
yt
(a)0 20 40 60 80 100 120 140 160 180 200−8−6−4−20246local level, a=1.000
  
Q=0.0, R=0.1
Q=0.1, R=0.0
Q=0.1, R=0.1
(b)
Figure 18.5 (a) Local level model. (b) Sample output, for a0=1 0. Black solid line: Q=0,R=1
(deterministic system, noisy observations). Red dotted line: Q=0.1,R=0(noisy system, deterministic
observation). Blue dot-dash line: Q=0.1,R=1(noisy system and observations). Figure generated by
ssmTimeSeriesSimple .
at−1 at
bt−1 bt
yt
(a)0 10 20 30 40 50 60 70 80 90 100−1000−800−600−400−2000200local trend, a=10.000, b=1.000
  
Q=0.0, R=100.0
Q=1.0, R=0.0
Q=1.0, R=100.0
(b)
Figure 18.6 (a) Local Trend. (b) Sample output, for a0=1 0,b0=1. Color code as in Figure 18.5. Figure
generated by ssmTimeSeriesSimple .
18.2.4.2 Local linear trend
Many time series exhibit linear trends upwards or downwards, at least locally. We can model
this by letting the level atchange by an amount btat each step as follows:
yt=at+ϵy
t,ϵyt∼N(0,R) (18.18)
at=at−1+bt−1+ϵa

(b)
yt=at+ϵy
t,ϵat∼N(0,Qa) (18.19)
bt=bt−1+ϵbt,ϵbt∼N(0,Qb) (18.20)
See Figure 18.6(a). We can write this in standard form by deﬁning zt=(at,bt)and
A=(11
01)
,C=(10)
,Q=(Qa0
0Qb)
(18.21)
WhenQb=0,w eh a v e bt=b0, which is some constant deﬁning the slope of the line. If in
addition we have Qa=0,w eh a v ea t=at−1+b0t. Unrolling this, we have at=a0+b0t, and

18.2. ApplicationsofSSMs 639
at−1 at
bt−1 bt
ytc1
t−1 c1
tc2
t−1 c2
tc3
t−1 c3
t-1
-1
-1
0 2 4 6 8 10 12 14 16 18 20−30−20−10010203040seasonal model, s=4, a=0.000, b=0.000
  
Q=0.0, R=1.0
Q=1.0, R=1.0
Figure 18.7 (a) Seasonal model. (b) Sample output, for a0=b0=0,c0=( 1,1,1), with a period of 4.
Color code as in Figure 18.5. Figure generated by ssmTimeSeriesSimple .
hence E[yt|y1:t−1]=a0+tb0. This is thus a generalization of the classic constant linear trend
model, an example of which is shown in the black line of Figure 18.6(b).
18.2.4.3 Seasonality
Many time series ﬂuctuate periodically, as illustrated in Figure 18.7(b). This can be modeled by
adding a latent process consisting of a series offset terms, ct, which sum to zero (on average)
over a complete cycle of Ssteps:
ct=−S−1∑
s=1ct−s+ϵc
t,ϵct∼N(0,Qc) (18.22)
See Figure 18.7(a) for the graphical model for the case S=4(we only need 3 seasonal vari-
able because of the sum-to-zero constraint). Writing this in standard LG-SSM form is left to

ct=−S−1∑
Exercise 18.2.
18.2.4.4 ARMA models *
The classical approach to time-series forecasting is based on ARMAmodels. “ARMA” stands for
auto-regressive moving-average, and refers to a model of the form
xt=p∑
i=1αixt−i+q∑
j=1βjwt−j+vt (18.23)
wherevt,wt∼N(0,1)are independent Gaussian noise terms. If q=0,w eh a v eap u r eA R
model, where xt⊥xi|xt−1:t−p,f o ri<t−p. For example, if p=1, we have the AR(1) model

640 Chapter18. Statespacemodels
x1 x2 x3 x4
(a)w1 w3 w2
x1 x2 x3 x4
(b)
w1 w3 w2
x1 x2 x3 x4
(c)
Figure 18.8 (a) An AR(1) model. (b) An MA(1) model represented as a bi-directed graph. (c) An ARMA(1,1)
model. Source: Figure 5.14 of (Choi 2011). Used with kind permission of Myung Choi.
shown in Figure 18.8(a). (The vtnodes are implicit in the Gaussian CPD for xt.) This is just a
ﬁrst-order Markov chain. If p=0, we have a pure MA model, where xt⊥xi,f o ri<t−q.
For example, if q=1, we have the MA(1) model shown in Figure 18.8(b). Here the wtnodes are
hidden common causes, which induces dependencies between adjacent time steps. This models
short-range correlation. If p=q=1, we get the ARMA(1,1) model shown in Figure 18.8(c), which
captures correlation at short and long time scales.
It turns out that ARMA models can be represented as SSMs, as explained in (Aoki 1987; Harvey

1990; West and Harrison 1997; Durbin and Koopman 2001; Petris et al. 2009; Prado and West2010). However, the structural approach to time series is often easier to understand than theARMA approach. In addition, it allows the parameters to evolve over time, which makes themodels more adaptive to non-stationarity.
18.3 Inference in LG-SSM

In this section, we discuss exact inference in LG-SSM models. We ﬁrst consider the online case,which is analogous to the forwards algorithm for HMMs. We then consider the offline case,which is analogous to the forwards-backwards algorithm for HMMs.
18.3.1 The Kalman ﬁltering algorithm
TheKalman ﬁlter is an algorithm for exact Bayesian ﬁltering for linear-Gaussian state space
models. We will represent the marginal posterior at time tby
p(zt|y1:t,u1:t)=N(zt|μt,Σt) (18.24)

Since everything is Gaussian, we can perform the prediction and update steps in closed form,as we explain below. The resulting algorithm is the Gaussian analog of the HMM ﬁlter inSection 17.4.2.

18.3. InferenceinLG-SSM 641
18.3.1.1 Prediction step
The prediction step is straightforward to derive:
p(zt|y1:t−1,u1:t)=∫
N(zt|Atzt−1+Btut,Qt)N(zt−1|μt−1,Σt−1)dzt−1(18.25)
=N(zt|μt|t−1,Σt|t−1) (18.26)
μt|t−1≜Atμt−1+Btut (18.27)
Σt|t−1≜AtΣt−1AT
t+Qt (18.28)
18.3.1.2 Measurement step
The measurement step can be computed using Bayes rule, as follows
p(zt|yt,y1:t−1,u1:t)∝p(yt|zt,ut)p(zt|y1:t−1,u1:t) (18.29)
In Section 18.3.1.6, we show that this is given by
p(zt|y1:t,ut)=N (zt|μt,Σt) (18.30)
μt=μt|t−1+Ktrt (18.31)
Σt=(I−KtCt)Σt|t−1 (18.32)
wherertis theresidual orinnovation, given by the difference between our predicted observa-
tion and the actual observation:
rt≜yt−ˆyt (18.33)
ˆyt≜E[yt|y1:t−1,u1:t]=Ctμt|t−1+Dtut (18.34)
andKtis theKalman gain matrix, given by
Kt≜Σt|t−1CTtS−1
t (18.35)
where
St≜cov[rt|y1:t−1,u1:t] (18.36)
=E[
(Ctzt+δt−ˆyt)(Ctzt+δt−ˆyt)T|y1:t−1,u1:t]
(18.37)
=CtΣt|t−1CTt+Rt (18.38)
whereδt∼N(0,Rt)is an observation noise term which is independent of all other noise

t (18.35)
where
=E[
(18.37)
sources. Note that by using the matrix inversion lemma, the Kalman gain matrix can also be
written as
Kt=Σt|t−1CT(CΣt|t−1CT+R)−1=(Σ−1
t|t−1+CTRC)−1CTR−1(18.39)
We now have all the quantities we need to implement the algorithm; see kalmanFilter for
some Matlab code.
Let us try to make sense of these equations. In particular, consider the equation for the
mean update: μt=μt|t−1+Ktrt. This says that the new mean is the old mean plus a

642 Chapter18. Statespacemodels
correction factor, which is Kttimes the error signal rt. The amount of weight placed on the
error signal depends on the Kalman gain matrix. If Ct=I, thenKt=Σt|t−1S−1
t, which
is the ratio between the covariance of the prior (from the dynamic model) and the covariance
of the measurement error. If we have a strong prior and/or very noisy sensors, |Kt|will be
small, and we will place little weight on the correction term. Conversely, if we have a weak priorand/or high precision sensors, then |K
t|will be large, and we will place a lot of weight on the
correction term.
18.3.1.3 Marginal likelihood
As a byproduct of the algorithm, we can also compute the log-likelihood of the sequence using
logp(y1:T|u1:T)=∑
tlogp(yt|y1:t−1,u1:t) (18.40)
where
p(yt|y1:t−1,u1:t)=N(yt|Ctμt|t−1,St) (18.41)
18.3.1.4 Posterior predictive
The one-step-ahead posterior predictive density for the observations can be computed as follows
p(yt|y1:t−1,u1:t)=∫
N(yt|Czt,R)N(zt|μt|t−1,Σt|t−1)dzt (18.42)

where
=N(yt|Cμt|t−1,CΣt|t−1CT+R) (18.43)
This is useful for time series forecasting.
18.3.1.5 Computational issues
There are two dominant costs in the Kalman ﬁlter: the matrix inversion to compute the Kalmangain matrix, K
t, which takes O(|yt|3)time; and the matrix-matrix multiply to compute Σt,
which takes O(|zt|2)time. In some applications (e.g., robotic mapping), we have |zt|≫|yt|,s o
the latter cost dominates. However, in such cases, we can sometimes use sparse approximations(see (Thrun et al. 2006)).
In cases where |y

t|≫|zt|, we can precompute Kt, since, suprisingly, it does not depend on
the actual observations y1:t(an unusual property that is speciﬁc to linear Gaussian systems).
The iterative equations for updating Σtare called the Ricatti equations, and for time invariant
systems (i.e., where θt=θ), they converge to a ﬁxed point. This steady state solution can then
be used instead of using a time-speciﬁc gain matrix.
In practice, more sophisticated implementations of the Kalman ﬁlter should be used, for rea-

sons of numerical stability. One approach is the information ﬁlter , which recursively updates
the canonical parameters of the Gaussian, Λt=Σ−1
tandηt=Λtμt, instead of the moment
parameters. Another approach is the square root ﬁlter, which works with the Cholesky de-
composition or the UtDtUtdecomposition of Σt. This is much more numerically stable than
directly updating Σt. Further details can be found at http://www .cs.unc.edu/~welch/kal
man/and in various books, such as (Simon 2006).

18.3. InferenceinLG-SSM 643
18.3.1.6 Derivation *
We now derive the Kalman ﬁlter equations. For notational simplicity, we will ignore the input
termsu1:t. From Bayes rule for Gaussians (Equation 4.125), we have that the posterior precision
is given by
Σ−1
t=Σ−1
t|t−1+CT
tR−1
tCt (18.44)
From the matrix inversion lemma (Equation 4.106) we can rewrite this as
Σt=Σt|t−1−Σt|t−1CTt(Rt+CtΣt|t−1CTt)−1CtΣt|t−1 (18.45)
=(I−KtCt)Σt|t−1 (18.46)
From Bayes rule for Gaussians (Equation 4.125), the posterior mean is given by
μt=ΣtCtR−1
tyt+ΣtΣ−1
t|t−1μt|t−1 (18.47)
We will now massage this into the form stated earlier. Applying the second matrix inversion
lemma (Equation 4.107) to the ﬁrst term of Equation 18.47 we have
ΣtCtR−1
tyt=(Σ−1
t|t−1+CT
tR−1
tCt)−1CtR−1
tyt (18.48)
=Σt|t−1CTt(Rt+CtΣt|t−1CTt)−1yt=Ktyt (18.49)
Now applying the matrix inversion lemma (Equation 4.106) to the second term of Equation 18.47
we have
ΣtΣ−1
t|t−1μt|t−1 (18.50)
=(Σ−1
t|t−1+CT
tR−1
tCt)−1Σ−1
t|t−1μt|t−1 (18.51)
=[

tyt+ΣtΣ−1
ΣtCtR−1
tyt=(Σ−1
t|t−1+CT
tR−1
we have
ΣtΣ−1
=(Σ−1
t|t−1+CT
tR−1
tCt)−1Σ−1
=[
Σt|t−1−Σt|t−1CT(Rt+CTtΣt|t−1CTt)CtΣt|t−1]
Σ−1
t|t−1μt|t−1 (18.52)
=(Σt|t−1−KtCTtΣt|t−1)Σ−1
t|t−1μt|t−1 (18.53)
=μt|t−1−KtCTtμt|t−1 (18.54)
Putting the two together we get
μt=μt|t−1+Kt(yt−Ctμt|t−1) (18.55)
18.3.2 The Kalman smoothing algorithm
In Section 18.3.1, we described the Kalman ﬁlter, which sequentially computes p(zt|y1:t)for each
t. This is useful for online inference problems, such as tracking. However, in an offline setting,
we can wait until all the data has arrived, and then compute p(zt|y1:T). By conditioning

Σ−1
on past and future data, our uncertainty will be signiﬁcantly reduced. This is illustrated inFigure 18.1(c), where we see that the posterior covariance ellipsoids are smaller for the smoothedtrajectory than for the ﬁltered trajectory. (The ellipsoids are larger at the beginning and end ofthe trajectory, since states near the boundary do not have as many useful neighbors from whichto borrow information.)

644 Chapter18. Statespacemodels
We now explain how to compute the smoothed estimates, using an algorithm called the
RTS smoother, named after its inventors, Rauch, Tung and Striebel (Rauch et al. 1965). It is
also known as the Kalman smoothing algorithm. The algorithm is analogous to the forwards-
backwards algorithm for HMMs, although there are some small differences which we discussbelow.
18.3.2.1 Algorithm
Kalman ﬁltering can be regarded as message passing on a graph, from left to right. When themessages have reached the end of the graph, we have successfully computed p(z
T|y1:T).N o w
we work backwards, from right to left, sending information from the future back to the past,and them combining the two information sources. The question is: how do we compute thesebackwards equations? We ﬁrst give the equations, then the derivation.
We have
p(z
t|y1:T)=N (μt|T,Σt|T) (18.56)
μt|T=μt|t+Jt(μt+1|T−μt+1|t) (18.57)
Σt|T=Σt|t+Jt(Σt+1|T−Σt+1|t)JT
t (18.58)
Jt≜Σt|tATt+1Σ−1
t+1|t(18.59)

We have
p(z
t (18.58)
whereJtis the backwards Kalman gain matrix. The algorithm can be initialized from μT|T
andΣT|Tfrom the Kalman ﬁlter. Note that this backwards pass does not need access to the
data, that is, it does not need y1:T. This allows us to “throw away” potentially high dimensional
observation vectors, and just keep the ﬁltered belief states, which usually requires less memory.
18.3.2.2 Derivation *
We now derive the Kalman smoother, following the presentation of (Jordan 2007, sec 15.7).
The key idea is to leverage the Markov property, which says that ztis independent of future

data,yt+1:T, as long as zt+1is known. Of course, zt+1is not known, but we have a distribution
over it. So we condition on zt+1and then integrate it out, as follows.
p(zt|y1:T)=∫
p(zt|y1:T,zt+1)p(zt+1|y1:T)dzt+1 (18.60)
=∫
p(zt|y1:t,yt+1:T,zt+1)p(zt+1|y1:T)dzt+1 (18.61)
By induction, assume we have already computed the smoothed distribution for t+1:
p(zt+1|y1:T)=N(zt+1|μt+1|T,Σt+1|T) (18.62)
The question is: how do we perform the integration?
First, we compute the ﬁltered two-slice distribution p(zt,zt+1|y1:t)as follows:

18.3. InferenceinLG-SSM 645
Now we use Gaussian conditioning to compute p(zt|zt+1,y1:t)as follows:
p(zt|zt+1,y1:t)=N (zt|μt|t+Jt(zt+1−μt+1|t),Σt|t−JtΣt+1|tJT
t) (18.64)
We can compute the smoothed distribution for tusing the rules of iterated expectation and
iterated covariance. First, the mean:
μt|T=E[
E[zt|zt+1,y1:T]⏐⏐y1:T]
(18.65)
=E[
E[zt|zt+1,y1:t]⏐⏐y
1:T]
(18.66)
=E[
μt|t+Jt(zt+1−μt+1|t)⏐⏐y
1:T]
(18.67)
=μt|t+Jt(μt+1|T−μt+1|t) (18.68)
Now the covariance:
Σt|T=c o v[
E[zt|zt+1,y1:T]⏐⏐y
1:T]
+E[
cov[zt|zt+1,y1:T]⏐⏐y
1:T]
(18.69)
=c o v [ E[zt|zt+1,y1:t]|y1:T]+E[
cov[zt|zt+1,y1:t]⏐⏐y
1:T]
(18.70)
=c o v[
μt|t+Jt(zt+1−μt+1|t)|y1:T]
+E[
Σt|t−JtΣt+1|tJT
t|y1:T]
(18.71)
=Jtcov[
zt+1−μt+1|t|y1:T]
JTt+Σt|t−JtΣt+1|tJTt(18.72)
=JtΣt+1|TJTt+Σt|t−JtΣt+1|tJTt(18.73)
=Σt|t+Jt(Σt+1|T−Σt+1|t)JTt(18.74)
The algorithm can be initialized from μT|TandΣT|Tfrom the last step of the ﬁltering algo-
rithm.
18.3.2.3 Comparison to the forwards-backwards algorithm for HMMs *

Σt|T=c o v[
1:T]
+E[
1:T]
(18.69)
1:T]
(18.70)
=c o v[
+E[
t|y1:T]
(18.71)
=Jtcov[
rithm.
Note that in both the forwards and backwards passes for LDS, we always worked with normalized
distributions, either conditioned on the past data or conditioned on all the data. Furthermore,the backwards pass depends on the results of the forwards pass. This is different from the usualpresentation of forwards-backwards for HMMs, where the backwards pass can be computedindependently of the forwards pass (see Section 17.4.3).
It turns out that we can rewrite the Kalman smoother in a modiﬁed form which makes it

more similar to forwards-backwards for HMMs. In particular, we have
p(z
t|y1:T)=∫
p(zt|y1:t,zt+1)p(zt+1|y1:T)dzt+1 (18.75)
=∫
p(zt,zt+1|y1:t)p(zt+1|y1:T)
p(zt+1|y1:t)dzt+1 (18.76)
Now
p(zt+1|y1:T)=p(yt+1:T|zt+1,y1:t)p(zt+1|y1:t)
p(yt+1:T|y1:t)(18.77)
so
p(zt+1|y1:T)
p(zt+1|y1:t)=p(zt+1|y1:t)p(yt+1:T|zt+1)
p(zt+1|y1:t)p(yt+1:T|y1:t)∝p(yt+1:T|zt+1) (18.78)

646 Chapter18. Statespacemodels
which is the conditional likelihood of the future data. This backwards message can be computed
independently of the forwards message. However, this approach has several disadvantages: (1)it needs access to the original observation sequence; (2) the backwards message is a likelihood,not a posterior, so it need not to integrate to 1 over z
t– in fact, it may not always be possible
to represent p(yt+1:T|zt+1)as a Gaussian with positive deﬁnite covariance (this problem does
not arise in discrete state-spaces, as used in HMMs); (3) when exact inference is not possible, itmakes more sense to try to approximate the smoothed distribution rather than the backwardslikelihood term (see Section 22.5).
There is yet another variant, known as two-ﬁlter smoothing, whereby we compute p(z
t|y1:t)
in the forwards pass as usual, and the ﬁltered posterior p(zt|yt+1:T)in the backwards pass.
These can then be easily combined to compute p(zt|y1:T). See (Kitagawa 2004; Briers et al.
2010) for details.

t|y1:t)
18.4 Learning for LG-SSM
In this section, we brieﬂy discuss how to estimate the parameters of an LG-SSM. In the controltheory community, this is known as systems identiﬁcation (Ljung 1987).
When using SSMs for time series forecasting, and also in some physical state estimation
problems, the observation matrix Cand the transition matrix Aare both known and ﬁxed, by
deﬁnition of the model. In such cases, all that needs to be learned are the noise covariances Q
andR. (The initial state estimate μ
0is often less important, since it will get “washed away” by

the data after a few time steps. This can be encouraged by setting the initial state covarianceto be large, representing a weak prior.) Although we can estimate QandRoffline, using the
methods described below, it is also possible to derive a recursive procedure to exactly computethe posterior p(z
t,R,Q|y1:t), which has the form of a Normal-inverse-Wishart; see (West and
Harrison 1997; Prado and West 2010) for details.
18.4.1 Identiﬁability and numerical stability
In the more general setting, where the hidden states have no pre-speciﬁed meaning, we need tolearnAandC. However, in this case we can set Q=Iwithout loss of generality, since an

arbitrary noise covariance can be modeled by appropriately modifying A. Also, by analogy with
factor analysis, we can require Rto be diagonal without loss of generality. Doing this reduces
the number of free parameters and improves numerical stability.
Another constraint that is useful to impose is on the eigenvalues of the dynamics matrix A.
To see why this is important, consider the case of no system noise. In this case, the hiddenstate at time tis given by
z
t=Atz1=UΛtU−1z1 (18.79)

z
whereUis the matrix of eigenvectors for A, andΛ=d i a g (λi)contains the eigenvalues. If
anyλi>1, then for large t,ztwill blow up in magnitude. Consequently, to ensure stability, it
is useful to require that all the eigenvalues are less than 1 (Siddiqi et al. 2007). Of course, if allthe eigenvalues are less than 1, then E[z
t]=0for larget, so the state will return to the origin.
Fortunately, when we add noise, the state become non-zero, so the model does not degenerate.

18.5. Approximateonlineinferencefornon-linear,non-GaussianSSMs 647
Below we discuss how to estimate the parameters. However, for simplicity of presentation, we
do not impose any of the constraints mentioned above.
18.4.2 Training with fully observed data
If we observe the hidden state sequences, we can ﬁt the model by computing the MLEs (or even
the full posteriors) for the parameters by solving a multivariate linear regression problem forz
t−1→ztand forzt→yt. That is, we can estimate Aby solving the least squares problem
J(A)=∑2
t=1(zt−Azt−1)2, and similarly for C. We can estimate the system noise covariance
Qfrom the residuals in predicting ztfromzt−1, and estimate the observation noise covariance
Rfrom the residuals in predicting ytfromzt.
18.4.3 EM for LG-SSM
If we only observe the output sequence, we can compute ML or MAP estimates of the parameters

J(A)=∑2
using EM. The method is conceptually quite similar to the Baum-Welch algorithm for HMMs(Section 17.5), except we use Kalman smoothing instead of forwards-backwards in the E step,and use different calculations in the M step. We leave the details to Exercise 18.1.
18.4.4 Subspace methods
EM does not always give satisfactory results, because it is sensitive to the initial parameterestimates. One way to avoid this is to use a different approach known as a subspace method
(Overschee and Moor 1996; Katayama 2005).

To understand this approach, let us initially assume there is no observation noise and no
system noise. In this case, we have z
t=Azt−1andyt=Czt, and hence yt=CAt−1z1.
Consequently all the observations must be generated from a dim(zt)-dimensional linear mani-
fold or subspace. We can identify this subspace using PCA (see the above references for details).Once we have an estimate of the z
t’s, we can ﬁt the model as if it were fully observed. We can
either use these estimates in their own right, or use them to initialize EM.
18.4.5 Bayesian methods for “ﬁtting” LG-SSMs

There are various offline Bayesian alternatives to the EM algorithm, including variational BayesEM (Beal 2003; Barber and Chiappa 2007) and blocked Gibbs sampling (Carter and Kohn 1994;Cappe et al. 2005; Fruhwirth-Schnatter 2007). The Bayesian approach can also be used toperform online learning, as we discussed in Section 18.2.3. Unfortunately, once we add the SSMparameters to the state space, the model is generally no longer linear Gaussian. Consequentlywe must use some of the approximate online inference methods to be discussed below.

18.5 Approximate online inference for non-linear, non-Gaussian SSMs
In Section 18.3.1, we discussed how to perform exact online inference for LG-SSMs. However,many models are non linear. For example, most moving objects do not move in straight lines.And even if they did, if we assume the parameters of the model are unknown and add them

648 Chapter18. Statespacemodels
to the state space, the model becomes nonlinear. Furthermore, non-Gaussian noise is also very
common, e.g., due to outliers, or when inferring parameters for GLMs instead of just linearregression. For these more general models, we need to use approximate inference.
The approximate inference algorithms we discuss below approximate the posterior by a Gaus-
sian. In general, if Y=f(X),w h e r eXhas a Gaussian distribution and fis a non-linear
function, there are two main ways to approximate p(Y)by a Gaussian. The ﬁrst is to use a
ﬁrst-order approximation of f. The second is to use the exact f, but to project f(X)onto the
space of Gaussians by moment matching. We discuss each of these methods in turn. (See alsoSection 23.5, where we discuss particle ﬁltering, which is a stochastic algorithm for approximateonline inference, which uses a non-parametric approximation to the posterior, which is oftenmore accurate but slower to compute.)
18.5.1 Extended Kalman ﬁlter (EKF)

In this section, we focus on non-linear models, but we assume the noise is Gaussian. That is,we consider models of the form
z
t=g(ut,zt−1)+N(0,Qt) (18.80)
yt=h(zt)+N(0,Rt) (18.81)
where the transition model gand the observation model hare nonlinear but differentiable
functions. Furthermore, we focus on the case where we approximate the posterior by a singleGaussian. (The simplest way to handle more general posteriors (e.g., multi-modal, discrete, etc).is to use particle ﬁltering, which we discuss in Section 23.5.)

z
Theextended Kalman ﬁlter orEKFcan be applied to nonlinear Gaussian dynamical systems
of this form. The basic idea is to linearize gandhabout the previous state estimate using
a ﬁrst order Taylor series expansion, and then to apply the standard Kalman ﬁlter equations.(The noise variance in the equations (Q andR) is not changed, i.e., the additional error due to
linearization is not modeled.) Thus we approximate the stationary non-linear dynamical systemwith a non-stationary linear dynamical system.
The intuition behind the approach is shown in Figure 18.9, which shows what happens when

we pass a Gaussian distribution p(x), shown on the bottom right, through a nonlinear function
y=g(x), shown on the top right. The resulting distribution (approximated by Monte Carlo) is
shown in the shaded gray area in the top left corner. The best Gaussian approximation to this,computed from E[g(x)]andvar[g(x)]by Monte Carlo, is shown by the solid black line. The
EKF approximates this Gaussian as follows: it linearizes the gfunction at the current mode, μ,

and then passes the Gaussian distribution p(x)through this linearized function. In this example,
the result is quite a good approximation to the ﬁrst and second moments of p(y), for much less
cost than an MC approximation.
In more detail, the method works as follows. We approximate the measurement model using
p(y
t|zt)≈N(yt|h(μt|t−1)+Ht(yt−μt|t−1),Rt) (18.82)
whereHtis the Jacobian matrix of hevaluated at the prior mode:
Hij≜∂hi(z)
∂zj(18.83)
Ht≜H|z=μt|t−1(18.84)

18.5. Approximateonlineinferencefornon-linear,non-GaussianSSMs 649
p(y)yp(y)
Gaussian of p(y)Mean of p(y)EKF GaussianMean of EKF
xy=g(x)Function g(x)Taylor approx.
Mean μ
g(μ)
xp(x)p(x)Mean μ
Figure 18.9 Nonlinear transformation of a Gaussian random variable. The prior p(x)is shown on the
bottom right. The function y=g(x)is shown on the top right. The transformed distribution p(y)is
shown in the top left. A linear function induces a Gaussian distribution, but a non-linear function induces
a complex distribution. The solid line is the best Gaussian approximation to this; the dotted line is the EKFapproximation to this. Source: Figure 3.4 of (Thrun et al. 2006). Used with kind permission of SebastianThrun.
Similarly, we approximate the system model using
p(zt|zt−1,ut)≈N(zt|g(ut,μt−1)+Gt(zt−1−μt−1),Qt) (18.85)
where
Gij(u)≜∂gi(u,z)
∂zj(18.86)
Gt≜G(ut)|z=μt−1(18.87)
soGis the Jacobian matrix of gevaluated at the prior mode.
Given this, we can then apply the Kalman ﬁlter to compute the posterior as follows:

where
μt|t−1=g(ut,μt−1) (18.88)
Vt|t−1=GtVt−1GT
t+Qt (18.89)
Kt=Vt|t−1HTt(HtVt|t−1HTt+Rt)−1(18.90)
μt=μt|t−1+Kt(yt−h(μt|t−1)) (18.91)
Vt=(I−KtHt)Vt|t−1 (18.92)

650 Chapter18. Statespacemodels
mean covariance sigma 
points Actual (sampling)  Sigma-Point Linearized (EKF) 
()=yf x () iiχϒ=f
transformed 
sigma points S-P 
covariance S-P 
mean 
yf x P P==( )    yAAT
x
true 
mean 
true 
covariance fx()AAT
xP
Figure 18.10 An example of the unscented transform in two dimensions. Source: (Wan and der Merwe
2001). Used with kind permission of Eric Wan.
We see that the only difference from the regular Kalman ﬁlter is that, when we compute the
state prediction, we use g(ut,μt−1)instead of Atμt−1+Btut, and when we compute the
measurement update we use h(μt|t−1)instead of Ctμt|t−1.
It is possible to improve performance by repeatedly re-linearizing the equations around μt
instead of μt|t−1; this is called the iterated EKF , and yields better results, although it is of
course slower.
There are two cases when the EKF works poorly. The ﬁrst is when the prior covariance is
large. In this case, the prior distribution is broad, so we end up sending a lot of probability

mass through different parts of the function that are far from the mean, where the function has
been linearized. The other setting where the EKF works poorly is when the function is highly
nonlinear near the current mean. In Section 18.5.2, we will discuss an algorithm called the UKF
which works better than the EKF in both of these settings.
18.5.2 Unscented Kalman ﬁlter (UKF)
Theunscented Kalman ﬁlter (UKF) is a better version of the EKF (Julier and Uhlmann 1997).
(Apparently it is so-called because it “doesn’t stink”!) The key intuition is this: it is easier

to approximate a Gaussian than to approximate a function. So instead of performing a linear
approximation to the function, and passing a Gaussian through it, instead pass a deterministically
chosen set of points, known as sigma points , through the function, and ﬁt a Gaussian to the
resulting transformed points. This is known as the unscented transform , and is sketched in
Figure 18.10. (We explain this ﬁgure in detail below.)

18.5. Approximateonlineinferencefornon-linear,non-GaussianSSMs 651
The UKF basically uses the unscented transform twice, once to approximate passing through
the system model g, and once to approximate passing through the measurement model h.W e
give the details below. Note that the UKF and EKF both perform O(d3)operations per time step
wheredis the size of the latent state-space. However, the UKF is accurate to at least second
order, whereas the EKF is only a ﬁrst order approximation (although both the EKF and UKF can
be extended to capture higher order terms). Furthermore, the unscented transform does notrequire the analytic evaluation of any derivatives or Jacobians (a so-called derivative free ﬁlter),
making it simpler to implement and more widely applicable.
18.5.2.1 The unscented transform
BeforeexplainingtheUKF,weﬁrstexplaintheunscentedtransform. Assume p(x)=N(x|μ,Σ),
and consider estimating p(y),w h e r ey=f(x)for some nonlinear function f. The unscented

transform does this as follows. First we create a set of 2d+1sigma points x
i, given by
x=(
μ,{μ+(√
(d+λ)Σ):i}d
i=1,{μ−(√
(d+λ)Σ):i}di=1)
(18.93)
whereλ=α2(d+κ)−dis a scaling parameter to be speciﬁed below, and the notation M:i
means the i’th column of matrix M.
These sigma points are propagated through the nonlinear function to yield yi=f(xi), and
the mean and covariance for yis computed as follows:
μy=2d∑
i=0wi
myi (18.94)
Σy=2d∑
i=0wi
c(yi−μy)(yi−μy)T(18.95)
where the w’s are weighting terms, given by
wi
m=λ
d+λ(18.96)
wi
c=λ
d+λ+(1−α2+β) (18.97)
wi
m=wi
c=1

i, given by
x=(
μ,{μ+(√
i=1,{μ−(√
(18.93)
μy=2d∑
i=0wi
Σy=2d∑
i=0wi
wi
m=λ
wi
c=λ
wi
m=wi
c=1
2(d+λ)(18.98)
See Figure 18.10 for an illustration.
In general, the optimal values of α,βandκare problem dependent, but when d=1, they
areα=1,β=0,κ=2. Thus in the 1d case, λ=2, so the 3 sigma points are μ,μ+√
3σ
andμ−√
3σ.
18.5.2.2 The UKF algorithm
The UKF algorithm is simply two applications of the unscented tranform, one to compute
p(zt|y1:t−1,u1:t)and the other to compute p(zt|y1:t,u1:t). We give the details below.

652 Chapter18. Statespacemodels
The ﬁrst step is to approximate the predictive density p(zt|y1:t−1,u1:t)≈N(zt|μt,Σt)by
passing the old belief state N(zt−1|μt−1,Σt−1)through the system model gas follows:
z0
t−1=(
μt−1,{μt−1+γ(√
Σt−1):i}di=1,{μt−1−γ(√
Σt−1):i}di=1)
(18.99)
z∗it=g(ut,z0it−1) (18.100)
μt=2d∑
i=0wi
mz∗it(18.101)
Σt=2d∑
i=0wi
c(z∗it−μt)(z∗it−μt)+Qt (18.102)
whereγ=√
d+λ.
The second step is to approximate the likelihood p(yt|zt)≈N(yt|ˆyt,St)by passing the
priorN(zt|μt,Σt)through the observation model h:
z0t=(
μt,{μt+γ(√
Σt):i}di=1,{μt−γ(√
Σt):i}di=1)
(18.103)
y∗it=h(z0it) (18.104)
ˆyt=2d∑
i=0wi
my∗it(18.105)
St=2d∑
i=0wi
c(y∗it−ˆyt)(y∗it−ˆyt)T+Rt (18.106)
Finally, we use Bayes rule for Gaussians to get the posterior p(zt|y1:t,u1:t)≈N(zt|μt,Σt):
Σz,y
t=2d∑
i=0wi
c(z∗i
t−μt)(y∗it−ˆyt)T(18.107)
Kt=Σz,y
tS−1
t (18.108)
μt=μt+Kt(yt−ˆyt) (18.109)
Σt=Σt−KtStKT
t (18.110)
18.5.3 Assumed density ﬁltering (ADF)
In this section, we discuss inference where we perform an exact update step, but then approx-

z0t=(
(18.103)
ˆyt=2d∑
i=0wi
St=2d∑
i=0wi
Σz,y
t=2d∑
i=0wi
c(z∗i
Kt=Σz,y
tS−1
t (18.108)
t (18.110)
imate the posterior by a distribution of a certain convenient form, such as a Gaussian. Moreprecisely, let the unknowns that we want to infer be denoted by θ
t. Suppose that Qi sas e to f
tractable distributions, e.g., Gaussians with a diagonal covariance matrix, or a product of discretedistributions. Suppose that we have an approximate prior q
t−1(θt−1)≈p(θt−1|y1:t−1),w h e r e
qt−1∈Q. We can update this with the new measurement to get the approximate posterior
ˆp(θt)=1
Ztp(yt|θt)qt|t−1(θt) (18.111)

18.5. Approximateonlineinferencefornon-linear,non-GaussianSSMs 653
qt−1qt|t−1ˆpt
qtqt+1|tˆpt+1
qt+1PredictUpdateProject
PredictUpdate Project
(a)xt−1 xtyt−1 ytst−1 stθt−1 θt
(b)
Figure 18.11 (a) Illustration of the predict-update-project cycle of assumed density ﬁltering. (b) A dynam-
ical logistic regression model. Compare to Figure 18.4(a).
where
Zt=∫
p(yt|θt)qt|t−1(θt)dθt (18.112)
is the normalization constant and
qt|t−1(θt)=∫
p(θt|θt−1)qt−1(θt−1)dθt−1 (18.113)
is the one step ahead predictive distribution. If the prior is from a suitably restricted family, this
one-step update process is usually tractable. However, we often ﬁnd that the resulting posterioris no longer in our tractable family, ˆp(θ
t)̸∈Q. So after updating we seek the best tractable
approximation by computing
q(θt) = argmin
q∈QKL(ˆp(θt)||q(θt)) (18.114)
This minimizes the the Kullback-Leibler divergence (Section 2.8.2) from the approximation q(θt)

to the “exact” posterior ˆp(θt), and can be thought of as projecting ˆponto the space of tractable
distributions. The whole algorithm consists of predict-update-project cycles. This is known as
assumed density ﬁltering orADF(Maybeck 1979). See Figure 18.11(a) for a sketch.
Ifqis in the exponential family, one can show that this KL minimization can be done by
moment matching. We give some examples of this below.
18.5.3.1 Boyen-Koller algorithm for online inference in DBNs

If we are performing inference in a discrete-state dynamic Bayes net (Section 17.6.7), where θtj
is thej’th hidden variable at time t, then the exact posterior p(θt)becomes intractable to
compute because of the entanglement problem. Suppose we use a fully factored approximation
of the form q(θt)=∏D
j=1Cat(θt,j|πt,j),w h e r eπtjk=q(θt,j=k)is the probability variable
jis in state k, andDis the number of variables. In this case, the moment matching operation
becomes
πtjk=ˆp(θt,j=k) (18.115)

654 Chapter18. Statespacemodels
This can be computed by performing a predict-update step using the factored prior, and then
computing the posterior marginals. This is known as the Boyen-Koller algorithm, named after
the authors of (Boyen and Koller 1998), who demonstrated that the error incurred by this seriesof repeated approximations remains bounded (under certain assumptions about the stochasticityof the system).
18.5.3.2 Gaussian approximation for online inference in GLMs
Now suppose q(θ
t)=∏D
j=1N(θt,j|μt,j,τt,j),w h e r eτt,jis the variance. Then the optimal
parameters of the tractable approximation to the posterior are
μt,j=Eˆp[θt,j],τt,j=v a rˆp[θt,j] (18.116)
This method can be used to do online inference for the parameters of many statistical models.

t)=∏D
For example, theTrueSkill system, used in Microsoft’s Xbox to rank players over time, uses thisform of approximation (Herbrich et al. 2007). We can also apply this method to simpler models,such as GLM, which have the advantage that the posterior is log-concave. Below we explain howto do this for binary logistic regression, following the presentation of (Zoeter 2007).
The model has the form
p(y
t|xt,θt)=B e r ( yt|sigm(xT
tθt)) (18.117)
p(θt|θt−1)=N (θt|θt−1,σ2I) (18.118)
whereσ2is some process noise which allows the parameters to change slowly over time. (This
can be set to 0, as in the recursive least squares method (Section 18.2.3), if desired.) We will

p(y
assumeqt−1(θt−1)=∏
jN(θt−1,j|μt−1,j,τt−1,j)is the tractable prior. We can compute the
one-step-ahead predictive density qt|t−1(θt)using the standard linear-Gaussian update. So now
we concentrate on the measurement update step.
Deﬁne the deterministic quantity st=θT
txt, as shown in Figure 18.11(b). If qt|t−1(θt)=∏
jN(θt,j|μt|t−1,j,τt|t−1,j), then we can compute the predictive distribution for stas follows:
qt|t−1(st)=N (st|mt|t−1,vt|t−1) (18.119)
mt|t−1=∑
jxt,jμt|t−1,j (18.120)
vt|t−1=∑
jx2
t,jτt|t−1,j (18.121)
The posterior for stis given by
qt(st)=N (st|mt,vt) (18.122)
mt=∫
st1
Ztp(yt|st)qt|t−1(st)dst (18.123)
vt=∫
s2t1
Ztp(yt|st)qt|t−1(st)dst−m2t(18.124)
Zt=∫
p(yt|st)qt|t−1(st)dst (18.125)

18.6. Hybriddiscrete/continuousSSMs 655
wherep(yt|st)=B e r ( yt|st). These integrals are one dimensional, and so can be computed
using Gaussian quadrature (see (Zoeter 2007) for details). This is the same as one step of the
UKF algorithm.
Having inferred q(st), we need to compute q(θ|st). This can be done as follows. Deﬁne δm
as the change in the mean of standδvas the change in the variance:
mt=mt|t−1+δm,vt=vt|t−1+δv (18.126)
Then one can show that the new factored posterior over the model parameters is given by
q(θt,j)=N (θt,j|μt,j,τt,j) (18.127)
μt,j=μt|t−1,j+ajδm (18.128)
τt,j=τt|t−1,j+a2
jδv (18.129)
aj≜xt,jτt|t−1,j∑
j′x2
t,j′τ2
t|t−1,j(18.130)
Thus we see that the parameters which correspond to inputs with larger magnitude (big |xt,j|)
or larger uncertainty (big τt|t−1,j) get updated most, which makes intuitive sense.
In (Opper 1998) a version of this algorithm is derived using a probit likelihood (see Section 9.4).
Inthiscase, themeasurementupdatecanbedoneinclosedform, withouttheneedfornumerical

j′x2
t,j′τ2
integration. In either case, the algorithm only takes O(D)operations per time step, so it can

j′x2
t,j′τ2
be applied to models with large numbers of parameters. And since it is an online algorithm,it can also handle massive datasets. For example (Zhang et al. 2010) use a version of thisalgorithm to ﬁt a multi-class classiﬁer online to very large datasets. They beat alternative (nonBayesian) online learning algorithms, and sometimes even outperform state of the art batch(offline) learning methods such as SVMs (described in Section 14.5).
18.6 Hybrid discrete/continuous SSMs

Many systems contain both discrete and continuous hidden variables; these are known as hybrid
systems. For example, the discrete variables may indicate whether a measurement sensor isfaulty or not, or which “regime” the system is in. We will see some other examples below.
A special case of a hybrid system is when we combine an HMM and an LG-SSM. This is
called aswitching linear dynamical system (SLDS), a jump Markov linear system (JMLS),
or aswitching state space model (SSSM). More precisely, we have a discrete latent variable,
q

q
t∈{1,...,K}, a continuous latent variable, zt∈RL, an continuous observed response
yt∈RDand an optional continuous observed input or control ut∈RU. We then assume that
the continuous variables have linear Gaussian CPDs, conditional on the discrete states:
p(qt=k|qt−1=j,θ)=A ij (18.131)
p(zt|zt−1,qt=k,ut,θ)=N (zt|Akzt−1+Bkut,Qk) (18.132)
p(yt|zt,qt=k,ut,θ)=N (yt|Ckzt+Dkut,Rk) (18.133)
See Figure 18.12(a) for the DGM representation.

656 Chapter18. Statespacemodels
yt−1 ytzt−1 ztqt−1 qtut−1 ut
(a) (b)
Figure 18.12 A switching linear dynamical system. (a) Squares represent discrete nodes, circles represent
continuous nodes. (b) Illustration of how the number of modes in the belief state grows exponentially over
time. We assume there are two binary states.
18.6.1 Inference
Unfortunately inference (i.e., state estimation) in hybrid models, including the switching LG-
SSM model, is intractable. To see why, suppose qtis binary, but that only the dynamics
Adepend on qt, not the observation matrix. Our initial belief state will be a mixture of
2 Gaussians, corresponding to p(z1|y1,q1=1 )andp(z1|y1,q1=2 ). The one-step-ahead
predictive density will be a mixture of 4 Gaussians p(z2|y1,q1=1,q2=1 ),p(z2|y1,q1=
1,q2=2 ),p(z2|y1,q1=2,q2=1 ), andp(z2|y1,q1=2,q2=2 ), obtained by passing each of

the prior modes through the 2 possible transition models. The belief state at step 2 will also bea mixture of 4 Gaussians, obtained by updating each of the above distributions with y
2.A ts t e p
3, the belief state will be a mixture of 8 Gaussians. And so on. So we see there is an exponentialexplosion in the number of modes (see Figure 18.12(b)).
Various approximate inference methods have been proposed for this model, such as the
• Prune off low probability trajectories in the discrete tree; this is the basis of multiple

2.A ts t e p
hypothesis tracking (Bar-Shalom and Fortmann 1988; Bar-Shalom and Li 1993).
• Use Monte Carlo. Essentially we just sample discrete trajectories, and apply an analytical
ﬁlter to the continuous variables conditional on a trajectory. See Section 23.6 for details.
• Use ADF, where we approximate the exponentially large mixture of Gaussians with a smaller
mixture of Gaussians. See Section 18.6.1.1 for details.
18.6.1.1 A Gaussian sum ﬁlter for switching SSMs
AGaussian sum ﬁlter (Sorenson and Alspach 1971) approximates the belief state at each step
by a mixture of KGaussians. This can be implemented by running KKalman ﬁlters in

algorithm, known as the “second order generalized pseudo Bayes ﬁlter ” (GPB2) (Bar-Shalom
and Fortmann 1988). We assume that the prior belief state bt−1is a mixture of KGaussians,
one per discrete state:
bi
t−1≜p(zt−1,qt−1=i|y1:t−1)=πt−1,iN(zt−1|μt−1,i,Σt−1,i) (18.134)
We then pass this through the Kdifferent linear models to get
bij
t≜p(zt,qt−1=i,qt=j|y1:t)=πtijN(zt|μt,ij,Σt,ij) (18.135)
whereπtij=πt−1,ip(qt=j|qt−1=i). Finally, for each value of j, we collapse the KGaussian
mixtures down to a single mixture to give
bj
t≜p(zt,qt=j|y1:t)=πtjN(zt|μt,j,Σt,j) (18.136)

658 Chapter18. Statespacemodels
See Figure 18.13(a) for a sketch.
The optimal way to approximate a mixture of Gaussians with a single Gaussian is given by
q=a r gm i n qKL(q||p),w h e r ep(z)=∑
kπkN(z|μk,Σk)andq(z)=N(z|μ,Σ). This can
be solved by moment matching, that is,
μ=E[z]=∑
kπkμk (18.137)
Σ=c o v [z]=∑
kπk(
Σk+(μk−μ)(μk−μ)T)
(18.138)
In the graphical model literature, this is called weak marginalization (Lauritzen 1992), since it
preserves the ﬁrst two moments. Applying these equations to our model, we can go from bij
tto
bjtas follows (where we drop the tsubscript for brevity):
πj=∑
iπij (18.139)
πj|i=πij∑
j′πij′(18.140)
μj=∑
iπj|iμij (18.141)
Σj=∑
iπj|i(
Σij+(μij−μj)(μij−μj)T)
(18.142)
This algorithm requires running K2ﬁlters at each step. A cheaper alternative is to represent
the belief state by a single Gaussian, marginalizing over the discrete switch at each step. This
is a straightforward application of ADF. An offline extension to this method, called expectation

tto
πj=∑
πj|i=πij∑
μj=∑
Σj=∑
iπj|i(
(18.142)
correction, is described in (Barber 2006; Mesot and Barber 2009).
Another heuristic approach, known as interactive multiple models orIMM(Bar-Shalom and
Fortmann 1988), can be obtained by ﬁrst collapsing the prior to a single Gaussian (by momentmatching), and then updating it using Kdifferent Kalman ﬁlters, one per value of q
t. See
Figure 18.13(b) for a sketch.
18.6.2 Application: data association and multi-target tracking
Suppose we are tracking Kobjects, such as airplanes, and at time t, we observe K′detection

t. See
events, e.g., “blips” on a radar screen. We can have K′<Kdue to occlusion or missed
detections. We can have K′>Kdue to clutter or false alarms. Or we can have K′=K.I n
any case, we need to ﬁgure out the correspondence between the K′detections ytkand theK
objectsztj. This is called the problem of data association, and it arises in many application
domains.
Figure 18.14 gives an example in which we are tracking K=2objects. At each time step, qt

domains.
is the unknown mapping which speciﬁes which objects caused which observations. It speciﬁesthe “wiring diagram” for time slice t. The standard way to solve this problem is to compute
a weight which measures the “compatibility” between object jand measurement k, typically
based on how close kis to where the model thinks jshould be (the so-called nearest neighbor
data association heuristic). This gives us a K×K
′weight matrix. We can make this into a

18.6. Hybriddiscrete/continuousSSMs 659
zt−1,1
zt−1,2zt,1
zt,2zt+1,1
zt+1,2
yt−1,1
yt−1,2
yt−1,3yt,1yt+1,1
yt+1,2
qt−1 qt qt+1
Figure 18.14 A model for tracking two objects in the presence of data-assocation ambiguity. We observe
3, 1 and 2 detections in the ﬁrst three time steps.
square matrix of size N×N,w h e r eN = max(K,K′), by adding dummy background objects,
which can explain all the false alarms, and adding dummy observations, which can explain all
the missed detections. We can then compute the maximal weight bipartite matching using theHungarian algorithm, which takes O(N
3)time (see e.g., (Burkard et al. 2009)). Conditional
on this, we can perform a Kalman ﬁlter update, where objects that are assigned to dummyobservations do not perform a measurement update.
An extension of this method, to handle a variable and/or unknown number of objects, is
known as multi-target tracking. This requires dealing with a variable-sized state space. There

are many ways to do this, but perhaps the simplest and most robust methods are based onsequential Monte Carlo (e.g., (Ristic et al. 2004)) or MCMC (e.g., (Khan et al. 2006; Oh et al.2009)).
18.6.3 Application: fault diagnosis

Consider the model in Figure 18.15(a). This represents an industrial plant consisting of varioustanks of liquid, interconnected by pipes. In this example, we just have two tanks, for simplicity.We want to estimate the pressure inside each tank, based on a noisy measurement of the ﬂowinto and out of each tank. However, the measurement devices can sometimes fail. Furthermore,pipes can burst or get blocked; we call this a “resistance failure”. This model is widely used asa benchmark in the fault diagnosis community (Mosterman and Biswas 1999).

We can create a probabilistic model of the system as shown in Figure 18.15(b). The square
nodes represent discrete variables, such as measurement failures and resistance failures. Theremaining variables are continuous. A variety of approximate inference algorithms can be appliedto this model. See (Koller and Lerner 2001) for one approach, based on Rao-Blackwellized particleﬁltering (which is explained in Section 23.6).

660 Chapter18. Statespacemodels
(a)RF1
1 RF1
2
R1
1 R1
2
MF1
1 MF1
2
F1
1 M1
1 F1
2 M1
2
P1
1 P1
2
MF12
1 MF12
2
F12
1 M12
1 F12
2 M12
2
R12
1 R12
2
RF12
1 RF12
2
P2
1 P2
2
F2
1 M2
1 F2
2 M2
2
MF2
1 MF2
2
R2
1 R2
2
RF2
1 RF2
2
(b)
Figure 18.15 (a) The two-tank system. The goal is to infer when pipes are blocked or have burst, or
sensors have broken, from (noisy) observations of the ﬂow out of tank 1, F1o, out of tank 2, F2o,o r
between tanks 1 and 2, F12.R1ois a hidden variable representing the resistance of the pipe out of
tank 1,P1is a hidden variable representing the pressure in tank 1, etc. Source: Figure 11 of (Koller and
Lerner 2001) . Used with kind permission of Daphne Koller. (b) Dynamic Bayes net representation of the
two-tank system. Discrete nodes are squares, continuous nodes are circles. Abbreviations: R = resistance,P = pressure, F = ﬂow, M = measurement, RF = resistance failure, MF = measurement failure. Based onFigure 12 of (Koller and Lerner 2001).
18.6.4 Application: econometric forecasting

The switching LG-SSM model is widely used in econometric forecasting, where it is called
aregime switching model. For example, we can combine two linear trend models (see Sec-
tion 18.2.4.2), one in which bt>0reﬂects a growing economy, and one in which bt<0reﬂects
a shrinking economy. See (West and Harrison 1997) for further details.
Exercises
Exercise 18.1 Derivation of EM for LG-SSM
Derive the E and M steps for computing a (locally optimal) MLE for an LG-SSM model. Hint: the results

Exercises
are in (Ghahramani and Hinton 1996b); your task is to derive these results.
Exercise 18.2 Seasonal LG-SSM model in standard form
Write the seasonal model in Figure 18.7(a) as an LG-SSM. Deﬁne the matrices A,C,QandR.

19Undirected graphical models (Markov
random ﬁelds)
19.1 Introduction
In Chapter 10, we discussed directed graphical models (DGMs), commonly known as Bayes nets.
However, for some domains, being forced to choose a direction for the edges, as required bya DGM, is rather awkward. For example, consider modeling an image. We might suppose thatthe intensity values of neighboring pixels are correlated. We can create a DAG model with a 2dlattice topology as shown in Figure 19.1(a). This is known as a causal MRF or aMarkov mesh
(Abend et al. 1965). However, its conditional independence properties are rather unnatural. Inparticular, the Markov blanket (deﬁned in Section 10.5) of the node X
8in the middle is the other
colored nodes (3, 4, 7, 9, 12 and 13) rather than just its 4 nearest neighbors as one might expect.
An alternative is to use an undirected graphical model (UGM), also called a Markov random
ﬁeld(MRF)o r Markov network. These do not require us to specify edge orientations, and are

much more natural for some problems such as image analysis and spatial statistics. For example,an undirected 2d lattice is shown in Figure 19.1(b); now the Markov blanket of each node is justits nearest neighbors, as we show in Section 19.2.
Roughly speaking, the main advantages of UGMs over DGMs are: (1) they are symmetric and

therefore more “natural” for certain domains, such as spatial or relational data; and (2) discrimi-nativel UGMs (aka conditional random ﬁelds, or CRFs), which deﬁne conditional densities of theformp(y|x), work better than discriminative DGMs, for reasons we explain in Section 19.6.1. The

main disadvantages of UGMs compared to DGMs are: (1) the parameters are less interpretableand less modular, for reasons we explain in Section 19.3; and (2) parameter estimation is com-putationally more expensive, for reasons we explain in Section 19.5. See (Domke et al. 2008) foran empirical comparison of the two approaches for an image processing task.
19.2 Conditional independence properties of UGMs
19.2.1 Key properties
UGMs deﬁne CI relationships via simple graph separation as follows: for sets of nodes A,B,
andC,w es a y xA⊥GxB|xCiffCseparates AfromBin the graph G. This means that,

when we remove all the nodes in C, if there are no paths connecting any node in Ato any
node inB, then the CI property holds. This is called the global Markov property for UGMs.
For example, in Figure 19.2(b), we have that {1,2}⊥{6,7}|{3,4,5}.

662 Chapter19. Undirectedgraphicalmodels(Markovrandomﬁelds)
X1 X2 X3 X4 X5
X6 X7 X8 X9 X10
X11 X12 X13 X14 X15
X16 X17 X18 X19 X20
(a)X1 X2 X3 X4 X5
(b)
Figure 19.1 (a) A 2d lattice represented as a DAG. The dotted red node X8is independent of all other
nodes (black) given its Markov blanket, which include its parents (blue), children (green) and co-parents
(orange). (b) The same model represented as a UGM. The red node X8is independent of the other black
nodes given its neighbors (blue nodes).
12
354
67
(a)12
354
67
(b)
Figure 19.2 (a) A DGM. (b) Its moralized version, represented as a UGM.
the graph is called t’sMarkov blanket; we will denote this by mb(t). Formally, the Markov
blanket satisﬁes the following property:
t⊥V\cl(t)|mb(t) (19.1)
wherecl(t)≜mb(t)∪{t}is theclosureof nodet. One can show that, in a UGM, a node’s

12
354
67
(a)12
354
67
(b)
Markov blanket is its set of immediate neighbors. This is called the undirected local Markov
property. For example, in Figure 19.2(b), we have mb(5) = {2,3,4,6}.
From the local Markov property, we can easily see that two nodes are conditionally indepen-
dent given the rest if there is no direct edge between them. This is called the pairwise Markov
property. In symbols, this is written as
s⊥t|V \{s,t}⇐ ⇒G st=0 (19.2)
UsingthethreeMarkovpropertieswehavediscussed, wecanderivethefollowingCIproperties
(amongst others) from the UGM in Figure 19.2(b):

•Pairwise 1⊥7|rest
•Local1⊥rest|2,3

19.2. ConditionalindependencepropertiesofUGMs 663
GLP
p(x)>0
Figure 19.3 Relationship between Markov properties of UGMs.
12
354
(a)12
354
(b)
Figure 19.4 (a) The ancestral graph induced by the DAG in Figure 19.2(a) wrt U={2,4,5}. (b) The
moralized version of (a).
•Global1,2⊥6,7|3,4,5
It is obvious that global Markov implies local Markov which implies pairwise Markov. What is
less obvious, but nevertheless true (assuming p(x)>0for allx, i.e., that pis a positive density),
is that pairwise implies global, and hence that all these Markov properties are the same, as
illustrated in Figure 19.3 (see e.g., (Koller and Friedman 2009, p119) for a proof).1The importance
of this result is that it is usually easier to empirically assess pairwise conditional independence;such pairwise CI statements can be used to construct a graph from which global CI statementscan be extracted.
19.2.2 An undirected alternative to d-separation

We have seen that determinining CI relationships in UGMs is much easier than in DGMs, becausewe do not have to worry about the directionality of the edges. In this section, we show how todetermine CI relationships for a DGM using a UGM.
It is tempting to simply convert the DGM to a UGM by dropping the orientation of the edges,
but this is clearly incorrect, since a v-structure A→B←Chas quite different CI properties
than the corresponding undirected chain A−B−C. The latter graph incorrectly states that

A⊥C|B. To avoid such incorrect CI statements, we can add edges between the “unmarried”
parentsAandC, and then drop the arrows from the edges, forming (in this case) a fully
connected undirected graph. This process is called moralization. Figure 19.2(b) gives a larger
1. The restriction to positive densities arises because deterministic constraints can result in independencies present in
the distribution that are not explicitly represented in the graph. See e.g., (Koller and Friedman 2009, p120) for some

examples. Distributions with non-graphical CI properties are said to be unfaithful to the graph, so I(p)̸=I(G).

664 Chapter19. Undirectedgraphicalmodels(Markovrandomﬁelds)
3UREDELOLVWLF0RGHOV
*UDSKLFDO0RGHOV
'LUHFWHG 8QGLUHFWHG &KRUGDO
Figure 19.5 DGMs and UGMs can perfectly represent different sets of distributions. Some distributions
can be perfectly represented by either DGMs or UGMs; the corresponding graph must be chordal.
example of moralization: we interconnect 2 and 3, since they have a common child 5, and we
interconnect 4, 5 and 6, since they have a common child 7.
Unfortunately, moralization loses some CI information, and therefore we cannot use the
moralized UGM to determine CI properties of the DGM. For example, in Figure 19.2(a), usingd-separation, we see that 4⊥5|2. Adding a moralization arc 4−5would lose this fact (see
Figure 19.2(b)). However, notice that the 4-5 moralization edge, due to the common child 7,is not needed if we do not observe 7 or any of its descendants. This suggests the followingapproach to determining if A⊥B|C. First we form the ancestral graph of DAGGwith respect

toU=A∪B∪C. This means we remove all nodes from Gthat are not in Uor are not
ancestors of U. We then moralize this ancestral graph, and apply the simple graph separation
rules for UGMs. For example, in Figure 19.4(a), we show the ancestral graph for Figure 19.2(a)usingU={2,4,5}. In Figure 19.4(b), we show the moralized version of this graph. It is clear
that we now correctly conclude that 4⊥5|2.
19.2.3 Comparing directed and undirected graphical models

Which model has more “expressive power”, a DGM or a UGM? To formalize this question, recallthat we say that Gis an I-map of a distribution pifI(G)⊆I(p). Now deﬁne Gto be
perfect map ofpifI(G)=I(p), in other words, the graph can represent all (and only) the CI
properties of the distribution. It turns out that DGMs and UGMs are perfect maps for differentsets of distributions (see Figure 19.5). In this sense, neither is more powerful than the other asa representation language.
As an example of some CI relationships that can be perfectly modeled by a DGM but not a

UGM, consider a v-structure A→C←B. This asserts that A⊥B, andA̸⊥B|C.I fw ed r o p
the arrows, we get A−C−B, which asserts A⊥B|CandA̸⊥B, which is incorrect. In fact,
there is no UGM that can precisely represent all and only the two CI statements encoded by a v-structure. In general, CI properties in UGMs are monotonic, in the following sense: if A⊥B|C,
thenA⊥B|(C∪D). But in DGMs, CI properties can be non-monotonic, since conditioning

19.3. ParameterizationofMRFs 665
C
(a) (b) (c)ADB B DA
CB DA
C
Figure 19.6 A UGM and two failed attempts to represent it as a DGM. Source: Figure 3.10 of (Koller and
Friedman 2009). Used with kind permission of Daphne Koller.
on extra variables can eliminate conditional independencies due to explaining away.
As an example of some CI relationships that can be perfectly modeled by a UGM but not a
DGM, consider the 4-cycle shown in Figure 19.6(a). One attempt to model this with a DGM is
shown in Figure 19.6(b). This correctly asserts that A⊥C|B,D. However, it incorrectly asserts
thatB⊥D|A. Figure 19.6(c) is another incorrect DGM: it correctly encodes A⊥C|B,D, but
incorrectly encodes B⊥D. In fact there is no DGM that can precisely represent all and only
the CI statements encoded by this UGM.
Some distributions can be perfectly modeled by either a DGM or a UGM; the resulting graphs
are called decomposable orchordal. Roughly speaking, this means the following: if we collapse

together all the variables in each maximal clique, to make “mega-variables”, the resulting graphwill be a tree. Of course, if the graph is already a tree (which includes chains as a special case),it will be chordal. See Section 20.4.1 for further details.
19.3 Parameterization of MRFs
Although the CI properties of UGM are simpler and more natural than for DGMs, representingthe joint distribution for a UGM is less natural than for a DGM, as we see below.
19.3.1 The Hammersley-Clifford theorem

Since there is no topological ordering associated with an undirected graph, we can’t use the chainrule to represent p(y). So instead of associating CPDs with each node, we associate potential
functionso rfactors with each maximal clique in the graph. We will denote the potentialfunction for clique cbyψ
c(yc|θc). A potential function can be any non-negative function of

its arguments. The joint distribution is then deﬁned to be proportional to the product of cliquepotentials. Rather surprisingly, one can show that any positive distribution whose CI propertiescan be represented by a UGM can be represented in this way. We state this result more formallybelow.

666 Chapter19. Undirectedgraphicalmodels(Markovrandomﬁelds)
Theorem 19.3.1 (Hammersley-Clifford). A positive distribution p(y)>0satisﬁes the CI prop-
erties of an undirected graph Giffpcan be represented as a product of factors, one per maximal
clique,i.e.,
p(y|θ)=1
Z(θ)∏
c∈Cψc(yc|θc) (19.3)
whereCisthesetofallthe(maximal)cliquesof G,andZ(θ)isthepartition function givenby
Z(θ)≜∑
x∏
c∈Cψc(yc|θc) (19.4)
Notethatthepartitionfunctioniswhatensurestheoveralldistributionsumsto1.2
The proof was never published, but can be found in e.g., (Koller and Friedman 2009).
For example, consider the MRF in Figure 10.1(b). If psatisﬁes the CI properties of this graph
then we can write pas follows:
p(y|θ)=1
Z(θ)ψ123(y1,y2,y3)ψ234(y2,y3,y4)ψ35(y3,y5) (19.5)
where
Z=∑
yψ123(y1,y2,y3)ψ234(y2,y3,y4)ψ35(y3,y5) (19.6)
There is a deep connection between UGMs and statistical physics. In particular, there is a
model known as the Gibbs distribution, which can be written as follows:
p(y|θ)=1
Z(θ)exp(−∑
cE(yc|θc)) (19.7)

p(y|θ)=1
where
Z=∑
p(y|θ)=1
whereE(yc)>0is the energy associated with the variables in clique c. We can convert this to
a UGM by deﬁning
ψc(yc|θc)=e x p (−E(yc|θc)) (19.8)
We see that high probability states correspond to low energy conﬁgurations. Models of this formare known as energy based models, and are commonly used in physics and biochemistry, as
well as some branches of machine learning (LeCun et al. 2006).
Note that we are free to restrict the parameterization to the edges of the graph, rather than
the maximal cliques. This is called a pairwise MRF. In Figure 10.1(b), we get
p(y|θ)∝ψ

p(y|θ)∝ψ
12(y1,y2)ψ13(y1,y3)ψ23(y2,y3)ψ24(y2,y4)ψ34(y3,y4)ψ35(y3,y5)(19.9)
∝∏
s∼tψst(ys,yt) (19.10)
This form is widely used due to its simplicity, although it is not as general.
2. The partition function is denoted by Zbecause of the German word zustandssumme, which means “sum over states”.
This reﬂects the fact that a lot of pioneering working in statistical physics was done by Germans.

19.3. ParameterizationofMRFs 667
19.3.2 Representing potential functions
If the variables are discrete, we can represent the potential or energy functions as tables of
(non-negative) numbers, just as we did with CPTs. However, the potentials are not probabilities.Rather, they represent the relative “compatibility” between the different assignments to thepotential. We will see some examples of this below.
A more general approach is to deﬁne the log potentials as a linear function of the parameters:
logψ
c(yc)≜φc(yc)Tθc (19.11)
whereφc(xc)is a feature vector derived from the values of the variables yc. The resulting log
probability has the form
logp(y|θ)=∑
cφc(yc)Tθc−Z(θ) (19.12)
This is also known as a maximum entropy or alog-linear model.
For example, consider a pairwise MRF, where for each edge, we associate a feature vector of
lengthK2as follows:
φst(ys,yt)=[...,I(ys=j,yt=k),...] (19.13)
If we have a weight for each feature, we can convert this into a K×Kpotential function as
follows:

logψ
follows:
ψst(ys=j,yt=k)=e x p ( [ θT
stφst]jk)=e x p (θst(j,k)) (19.14)
So we see that we can easily represent tabular potentials using a log-linear form. But the
log-linear form is more general.
To see why this is useful, suppose we are interested in making a probabilistic model of English
spelling. Since certain letter combinations occur together quite frequently (e.g., “ing”), we willneed higher order factors to capture this. Suppose we limit ourselves to letter trigrams. Atabular potential still has 26

follows:
3=1 7,576parameters in it. However, most of these triples will
never occur.
An alternative approach is to deﬁne indicator functions that look for certain “special” triples,
such as “ing”, “qu-”, etc. Then we can deﬁne the potential on each trigram as follows:
ψ(yt−1,yt,yt+1)=e x p (∑
kθkφk(yt−1,yt,yt+1)) (19.15)
wherekindexes the different features, corresponding to “ing”, “qu-”, etc., and φkis the corre-
sponding binary feature function. By tying the parameters across locations, we can deﬁne the

probability of a word of any length using
p(y|θ)∝exp(∑
t∑
kθkφk(yt−1,yt,yt+1)) (19.16)
This raises the question of where these feature functions come from. In many applications,they are created by hand to reﬂect domain knowledge (we will see examples later), but it is alsopossible to learn them from data, as we discuss in Section 19.5.6.

668 Chapter19. Undirectedgraphicalmodels(Markovrandomﬁelds)
19.4 Examples of MRFs
In this section, we show how several popular probability models can be conveniently expressed
as UGMs.
19.4.1 Ising model
TheIsing model is an example of an MRF that arose from statistical physics.3It was originally
used for modeling the behavior of magnets. In particular, let ys∈{ −1,+1}represent the spin
of an atom, which can either be spin down or up. In some magnets, called ferro-magnets,
neighboring spins tend to line up in the same direction, whereas in other kinds of magnets,calledanti-ferromagnets, the spins “want” to be different from their neighbors.
We can model this as an MRF as follows. We create a graph in the form of a 2D or 3D lattice,
and connect neighboring variables, as in Figure 19.1(b). We then deﬁne the following pairwiseclique potential:
ψ
st(ys,yt)=(ewste−wst
e−wstewst)
(19.17)
Herewstis the coupling strength between nodes sandt. If two nodes are not connected in

ψ
(19.17)
the graph, we set wst=0. We assume that the weight matrix Wis symmetric, so wst=wts.
Often we assume all edges have the same strength, so wst=J(assuming wst̸=0).
If all the weights are positive, J>0, then neighboring spins are likely to be in the same
state; this can be used to model ferromagnets, and is an example of an associative Markov

ψ
(19.17)
network. If the weights are sufficiently strong, the corresponding probability distribution willhave two modes, corresponding to the all +1’s state and the all -1’s state. These are called theground states of the system.
If all of the weights are negative, J<0, then the spins want to be different from their
neighbors; this can be used to model an anti-ferromagnet, and results in a frustrated system,

in which not all the constraints can be satisﬁed at the same time. The corresponding probabilitydistribution will have multiple modes. Interestingly, computing the partition function Z(J)can
be done in polynomial time for associative Markov networks, but is NP-hard in general (Cipra2000).
There is an interesting analogy between Ising models and Gaussian graphical models. First,
assuming y
t∈{ −1,+1}, we can write the unnormalized log probability of an Ising model as
follows:
log ˜p(y)=−∑
s∼tyswstyt=−1
2yTWy (19.18)

assuming y
follows:
(The factor of1
2arises because we sum each edge twice.) If wst=J>0, we get a low energy
(and hence high probability) if neighboring states agree.
Sometimes there is an external ﬁeld, which is an energy term which is added to each spin.
This can be modelled using a local energy term of the form −bTy,w h e r ebis sometimes called
3. Ernst Ising was a German-American physicist, 1900–1998.

19.4. ExamplesofMRFs 669
abias term. The modiﬁed distribution is given by
log ˜p(y)=∑
s∼twstysyt+∑
sbsys=1
2yTWy+bTy (19.19)
whereθ=(W,b).
If we deﬁne μ≜−1
2Σ−1b,Σ−1=−W, andc≜1
2μTΣ−1μ, we can rewrite this in a form
that looks similar to a Gaussian:
˜p(y)∝exp(−1
2(y−μ)TΣ−1(y−μ)+c) (19.20)
One very important difference is that, in the case of Gaussians, the normalization constant,
Z=|2πΣ|, requires the computation of a matrix determinant, which can be computed in
O(D3)time, whereas in the case of the Ising model, the normalization constant requires
summing over all 2Dbit vectors; this is equivalent to computing the matrix permanent, which
is NP-hard in general (Jerrum et al. 2004).
19.4.2 Hopﬁeld networks
AHopﬁeld network (Hopﬁeld 1982) is a fully connected Ising model with a symmetric weight
matrix,W=WT. These weights, plus the bias terms b, can be learned from training data
using (approximate) maximum likelihood, as described in Section 19.5.4

The main application of Hopﬁeld networks is as an associative memory orcontent ad-
dressable memory. The idea is this: suppose we train on a set of fully observed bit vectors,corresponding to patterns we want to memorize. Then, at test time, we present a partial patternto the network. We would like to estimate the missing variables; this is called pattern com-

pletion. See Figure 19.7 for an example. This can be thought of as retrieving an example frommemory based on a piece of the example itself, hence the term “associative memory”.
Since exact inference is intractable in this model, it is standard to use a coordinate descent
algorithm known as iterative conditional modes (ICM), which just sets each node to its most
likely (lowest energy) state, given all its neighbors. The full conditional can be shown to be
p(y
s=1|y−s,θ) = sigm(wT
s,:y−s+bs) (19.21)
Picking the most probable state amounts to using the rule y∗
s=1if∑
twstyt>bsand using

p(y
s=1if∑
y∗
s=0otherwise. (Much better inference algorithms will be discussed later in this book.)
Since inference is deterministic, it is also possible to interpret this model as a recurrent
neural network. (This is quite different from the feedforward neural nets studied in Section 16.5;they are univariate conditional density models of the form p(y|x,θ)which can only be used for
supervised learning.) See Hertz et al. (1991) for further details on Hopﬁeld networks.
ABoltzmann machine generalizes the Hopﬁeld / Ising model by including some hidden

nodes, which makes the model representationally more powerful. Inference in such modelsoften uses Gibbs sampling, which is a stochastic version of ICM (see Section 24.2 for details).
4. ML estimation works much better than the outer product rule proposed in in (Hopﬁeld 1982), because it not only
lowers the energy of the observed patterns, but it also raises the energy of the non-observed patterns, in order to make
the distribution sum to one (Hillar et al. 2012).

670 Chapter19. Undirectedgraphicalmodels(Markovrandomﬁelds)
Figure 19.7 Examples of how an associative memory can reconstruct images. These are binary images
of size50×50pixels. Top: training images. Row 2: partially visible test images. Row 3: estimate after
5 iterations. Bottom: ﬁnal state estimate. Based on Figure 2.1 of Hertz et al. (1991). Figure generated by
hopfieldDemo .
(a)
 (b)
 (c)
Figure 19.8 Visualizing a sample from a 10-state Potts model of size 128×128for different association
strengths: (a) J=1.42, (b)J=1.44, (c)J=1.46. The regions are labeled according to size: blue is
largest, red is smallest. Used with kind permission of Erik Sudderth. See gibbsDemoIsing for Matlab
code to produce a similar plot for the Ising model.
However, we could equally well apply Gibbs to a Hopﬁeld net and ICM to a Boltzmann machine:
the inference algorithm is not part of the model deﬁnition. See Section 27.7 for further details
on Boltzmann machines.

19.4. ExamplesofMRFs 671
xs xtys yt
Figure 19.9 A grid-structured MRF with local evidence nodes.
19.4.3 Potts model
It is easy to generalize the Ising model to multiple discrete states, yt∈{1,2,...,K}.I t i s
common to use a potential function of the following form:
ψst(ys,yt)=⎛
⎝eJ00
0eJ0
00 eJ⎞⎠ (19.22)
This is called the Potts model.
5IfJ>0, then neighboring nodes are encouraged to have the
same label. Some samples from this model are shown in Figure 19.8. We see that for J>1.44,
large clusters occur, for J<1.44, many small clusters occur, and at the critical value of
K=1.44, there is a mix of small and large clusters. This rapid change in behavior as we vary
a parameter of the system is called a phase transition, and has been widely studied in the
physics community. An analogous phenomenon occurs in the Ising model; see (MacKay 2003,
ch 31) for details.
The Potts model can be used as a prior for image segmentation, since it says that neighboring

pixels are likely to have the same discrete label and hence belong to the same segment. We cancombine this prior with a likelihood term as follows:
p(y,x|θ)=p(y|J)∏
tp(xt|yt,θ)=[
1
Z(J)∏
s∼tψ(ys,yt;J)]∏
tp(xt|yt,θ) (19.23)
wherep(xt|yt=k,θ)is the probability of observing pixel xtgiven that the corresponding
segment belongs to class k. This observation model can be modeled using a Gaussian or a
non-parametric density. (Note that we label the hidden nodes ytand the observed nodes xt,t o
be compatible with Section 19.6.)

1
Z(J)∏
The corresponding graphical model is a mix of undirected and directed edges, as shown in
Figure 19.9. The undirected 2d lattice represents the prior p(y); in addition, there are directed
edge from each ytto its corresponding xt, representing the local evidence. Technically speak-
ing, this combination of an undirected and directed graph is called a chain graph. However,
5. Renfrey Potts was an Australian mathematician, 1925–2005.

672 Chapter19. Undirectedgraphicalmodels(Markovrandomﬁelds)
since the xtnodes are observed, they can be “absorbed” into the model, thus leaving behind an
undirected “backbone”.
This model is a 2d analog of an HMM, and could be called a partially observed MRF.A s
in an HMM, the goal is to perform posterior inference, i.e., to compute (some function of)
p(y|x,θ). Unfortunately, the 2d case is provably much harder than the 1d case, and we must
resort to approximate methods, as we discuss in later chapters.
Although the Potts prior is adequate for regularizing supervised learning problems, it is not
sufficiently accurate to perform image segmentation in an unsupervised way, since the segmentsproduced by this model do not accurately represent the kinds of segments one sees in naturalimages (Morris et al. 1996).
6For the unsupervised case, one needs to use more sophisticated
priors, such as the truncated Gaussian process prior of (Sudderth and Jordan 2008).
19.4.4 Gaussian MRFs

An undirected GGM, also called a Gaussian MRF (see e.g., (Rue and Held 2005)), is a pairwise
MRF of the following form:
p(y|θ)∝∏
s∼tψst(ys,yt)∏
tψt(yt) (19.24)
ψst(ys,yt)=e x p ( −1
2ysΛstyt) (19.25)
ψt(yt)=e x p ( −1
2Λtty2
t+ηtyt) (19.26)
(Note that we could easily absorb the node potentials ψtinto the edge potentials, but we have
kept them separate for clarity.) The joint distribution can be written as follows:
p(y|θ)∝exp[ηTy−1
2yTΛy] (19.27)
We recognize this as a multivariate Gaussian written in information form whereΛ=Σ−1and
η=Λμ.

p(y|θ)∝∏
2Λtty2
η=Λμ.
IfΛst=0, then there is no pairwise term connecting sandt, so by the factorization theorem
(Theorem 2.2.1), we conclude that
ys⊥yt|y−(st)⇐⇒Λst=0 (19.28)
The zero entries in Λare called structural zeros, since they represent the absent edges in the
graph. Thus undirected GGMs correspond to sparse precision matrices, a fact which we exploitin Section 26.7.2 to efficiently learn the structure of the graph.
19.4.4.1 Comparing Gaussian DGMs and UGMs *

η=Λμ.
In Section 10.2.5, we saw that directed GGMs correspond to sparse regression matrices, and hencesparse Cholesky factorizations of covariance matrices, whereas undirected GGMs correspond to
6. An inﬂuential paper (Geman and Geman 1984), which introduced the idea of a Gibbs sampler (Section 24.2), proposed
using the Potts model as a prior for image segmentation, but the results in their paper are misleading because they did
not run their Gibbs sampler for long enough. See Figure 24.10 for a vivid illustration of this point.

19.4. ExamplesofMRFs 673
Figure 19.10 A VAR(2) process represented as a dynamic chain graph. Source: (Dahlhaus and Eichler
2000). Used with kind permission of Rainer Dahlhaus and Oxford University Press.
sparse precision matrices. The advantage of the DAG formulation is that we can make the
regression weights W, and hence Σ, be conditional on covariate information (Pourahmadi 2004),
without worrying about positive deﬁnite constraints. The disadavantage of the DAG formulation
is its dependence on the order, although in certain domains, such as time series, there is already
a natural ordering of the variables.
It is actually possible to combine both representations, resulting in a Gaussian chain graph.
For example, consider a a discrete-time, second-order Markov chain in which the states are
continuous, yt∈RD. The transition function can be represented as a (vector-valued) linear-
Gaussian CPD:
p(yt|yt−1,yt−2,θ)=N(yt|A1yt−1+A2yt−2,Σ) (19.29)

This is called vector auto-regressive orVARprocess of order 2. Such models are widely used
in econometrics for time-series forecasting.
The time series aspect is most naturally modeled using a DGM. However, if Σ−1is sparse,
then the correlation amongst the components within a time slice is most naturally modeled
using a UGM. For example, suppose we have
A1=⎛
⎜⎜⎜⎜⎝3
501
500
03
50−1
50
2
51
33
500
000 −1
21
5
001
502
5⎞
⎟⎟⎟⎟⎠,A2=⎛
⎜⎜⎜⎜⎝00−1
500
0 0000
0 0000
001
501
3
0 000 −1
5⎞
⎟⎟⎟⎟⎠(19.30)
and
Σ=⎛
⎜⎜⎜⎜⎝11
21
300
1
21−1
300
1
3−1
310 0
00 01 0

A1=⎛
⎜⎜⎜⎜⎝3
501
500
03
50−1
50
2
51
33
500
000 −1
21
5
001
502
5⎞
⎜⎜⎜⎜⎝00−1
500
0 0000
0 0000
001
501
3
0 000 −1
5⎞
and
Σ=⎛
⎜⎜⎜⎜⎝11
21
300
1
21−1
300
1
3−1
310 0
00 01 0
00 00 1⎞
⎟⎟⎟⎟⎠,Σ−1=⎛
⎜⎜⎜⎜⎝2.13−1.47−1.200
−1.47 2.13 1.20 0
−1.21.21.80 0
00 0 1 0
00 0 0 1⎞
⎟⎟⎟⎟⎠(19.31)

674 Chapter19. Undirectedgraphicalmodels(Markovrandomﬁelds)
x1 x2 x3
(a)
x1 x2 x3w1 w2
(b)
Figure 19.11 (a) A bi-directed graph. (b) The equivalent DAG. Here the wnodes are latent confounders.
Based on Figures 5.12-5.13 of (Choi 2011). Used with kind permission of Myung Choi.
The resulting graphical model is illustrated in Figure 19.10. Zeros in the transition matrices A1
andA2correspond to absent directed arcs from yt−1andyt−2intoyt. Zeros in the precision
matrixΣ−1correspond to absent undirected arcs between nodes in yt.
Sometimes we have a sparse covariance matrix rather than a sparse precision matrix. This can
be represented using a bi-directed graph, where each edge has arrows in both directions, as in
Figure 19.11(a). Here nodes that are not connected are unconditionally independent. For example
in Figure 19.11(a) we see that Y1⊥Y3. In the Gaussian case, this means Σ1,3=Σ3,1=0.( A
graph representing a sparse covariance matrix is called a covariance graph.) By contrast, if

this were an undirected model, we would have that Y1⊥Y3|Y2, andΛ1,3=Λ3,1=0,w h e r e
Λ=Σ−1.
A bidirected graph can be converted to a DAG with latent variables, where each bidirected
edge is replaced with a hidden variable representing a hidden common cause, or confounder,
as illustrated in Figure 19.11(b). The relevant CI properties can then be determined using d-separation.
We can combine bidirected and directed edges to get a directed mixed graphical model.

Λ=Σ−1.
This is useful for representing a variety of models, such as ARMA models (Section 18.2.4.4),structural equation models (Section 26.5.5), etc.
19.4.5 Markov logic networks *

In Section 10.2.2, we saw how we could “unroll” Markov models and HMMs for an arbitrarynumber of time steps in order to model variable-length sequences. Similarly, in Section 19.4.1,we saw how we could expand a lattice UGM to model images of any size. What about morecomplex domains, where we have a variable number of objects and relationships between them?Creating models for such scenarios is often done using ﬁrst-order logic (see e.g., (Russell and

Norvig 2010)). For example, consider the sentences “Smoking causes cancer” and “If two peopleare friends, and one smokes, then so does the other”. We can write these sentences in ﬁrst-order

19.4. ExamplesofMRFs 675
Friends(A,A) Smokes(A) Smokes(B) Friends(B,B)Friends(A,B)
Friends(B,A)Cancer(A) Cancer(B)
Figure 19.12 An example of a ground Markov logic network represented as a pairwise MRF for 2 people.
Based on Figure 2.1 from (Domingos and Lowd 2009). Used with kind permission of Pedro Domingos.
logic as follows:
∀x.Sm(x)=⇒Ca(x) (19.32)
∀x.∀y.Fr(x,y)∧Sm(x)=⇒Sm(y) (19.33)
whereSmandCaare predicates, and Fris a relation.7
Of course, such rules are not always true. Indeed, this brittleness is the main reason why
logical approaches to AI are no longer widely used, at least not in their pure form. There
have been a variety of attempts to combine ﬁrst order logic with probability theory, an areaknown as statistical relational AI orprobabilistic relational modeling (Kersting et al. 2011).
One simple approach is to take logical rules and attach weights (known as certainty factors)t o

them, and then to interpret them as conditional probability distributions. For example, we mightsayp(Ca(x)=1|Sm(x)=1 )=0 .9. Unfortunately, the rule does not say what to predict if
Sm(x)=0. Furthermore, combining CPDs in this way is not guaranteed to deﬁne a consistent
joint distribution, because the resulting graph may not be a DAG.
An alternative approach is to treat these rules as a way of deﬁning potential functions in an
unrolled UGM. The result is known as a Markov logic network (Domingos and Lowd 2009).

To specify the network, we ﬁrst rewrite all the rules in conjunctive normal form (CNF), also
known as clausal form. In this case, we get
¬Sm(x)∨Ca(x) (19.34)
¬Fr(x,y)∨¬Sm(x)∨Sm(y) (19.35)
The ﬁrst clause can be read as “Either xdoes not smoke or he has cancer”, which is logically
equivalent to Equation 19.32. (Note that in a clause, any unbound variable, such as x, is assumed
to be universally quantiﬁed.)
7. A predicate is just a function of one argument, known as an object, that evaluates to true or false, depending on
whether the property holds or not of that object. A (logical) relation is just a function of two or more arguments (objects)

that evaluates to true or false, depending on whether the relationship holds between that set of objects or not.

676 Chapter19. Undirectedgraphicalmodels(Markovrandomﬁelds)
Inference in ﬁrst-order logic is only semi-decidable, so it is common to use a restricted subset.
A common approach (as used in Prolog) is to restrict the language to Horn clauses, which are
clauses that contain at most one positive literal. Essentially this means the model is a series of
if-then rules, where the right hand side of the rules (the “then” part, or consequence) has onlya single term.
Once we have encoded our knowledge base as a set of clauses, we can attach weights to
each one; these weights are the parameter of the model, and they deﬁne the clique potentialsas follows:
ψ
c(xc)=e x p (wcφc(xc)) (19.36)
whereφc(xc)is a logical expression which evaluates clause capplied to the variables xc, and
wcis the weight we attach to this clause. Roughly speaking, the weight of a clause speciﬁes
the probability of a world in which this clause is satsiﬁed relative to a world in which it is notsatisﬁed.

ψ
Now suppose there are two objects (people) in the world, Anna and Bob, which we will denote
byconstant symbols AandB. We can make a ground network from the above clauses by
creating binary random variables S
x,Cx, andFx,yforx,y∈{A,B}, and then “wiring these
up” according to the clauses above. The result is the UGM in Figure 19.12 with 8 binary nodes.Note that we have not encoded the fact that Fris a symmetric relation, so Fr(A,B)and
Fr(B,A)might have different values. Similarly, we have the “degenerate” nodes Fr(A,A)and

Fr(B,B), since we did not enforce x̸=yin Equation 19.33. (If we add such constraints,
then the model compiler, which generates the ground network, could avoid creating redundantnodes.)
In summary, we can think of MLNs as a convenient way of specifying a UGM template, that

can get unrolled to handle data of arbitrary size. There are several other ways to deﬁne relationalprobabilistic models; see e.g., (Koller and Friedman 2009; Kersting et al. 2011) for details. In somecases, there is uncertainty about the number or existence of objects or relations (the so-calledopen universe problem). Section 18.6.2 gives a concrete example in the context of multi-object
tracking. See e.g., (Russell and Norvig 2010; Kersting et al. 2011) and references therein for furtherdetails.
19.5 Learning

In this section, we discuss how to perform ML and MAP parameter estimation for MRFs. We willsee that this is quite computationally expensive. For this reason, it is rare to perform Bayesianinference for the parameters of MRFs (although see (Qi et al. 2005)).
19.5.1 Training maxent models using gradient methods
Consider an MRF in log-linear form:
p(y|θ)=1
Z(θ)exp(∑
cθT
cφc(y))
(19.37)

19.5. Learning 677
wherecindexes the cliques. The scaled log-likelihood is given by
ℓ(θ)≜1
N∑
ilogp(yi|θ)=1
N∑
i[∑
cθT
cφc(yi)−logZ(θ)]
(19.38)
Since MRFs are in the exponential family, we know that this function is convex in θ(see
Section 9.2.3), so it has a unique global maximum which we can ﬁnd using gradient-based
optimizers. In particular, the derivative for the weights of a particular clique, c, is given by
∂ℓ
∂θc=1
N∑
i[
φc(yi)−∂
∂θclogZ(θ)]
(19.39)
Exercise 19.1 asks you to show that the derivative of the log partition function wrt θcis the
expectation of the c’th feature under the model, i.e.,
∂logZ(θ)
∂θc=E[φc(y)|θ]=∑
yφc(y)p(y|θ) (19.40)
Hence the gradient of the log likelihood is
∂ℓ
∂θc=[
1
N∑
iφc(yi)]
−E[φc(y)] (19.41)
In the ﬁrst term, we ﬁx yto its observed values; this is sometimes called the clamped term.
In the second term, yis free; this is sometimes called the unclamped term orcontrastive
term. Note that computing the unclamped term requires inference in the model, and this must

∂logZ(θ)
∂ℓ
∂θc=[
1
N∑
iφc(yi)]
be done once per gradient step. This makes UGM training much slower than DGM training.
The gradient of the log likelihood can be rewritten as the expected feature vector according
to the empirical distribution minus the model’s expectation of the feature vector:
∂ℓ
∂θc=Epemp[φc(y)]−Ep(·|θ)[φc(y)] (19.42)
At the optimum, the gradient will be zero, so the empirical distribution of the features willmatch the model’s predictions:
E
pemp[φc(y)] =Ep(·|θ)[φc(y)] (19.43)
This is called moment matching. This observation motivates a different optimization algorithm

∂ℓ
E
which we discuss in Section 19.5.7.
19.5.2 Training partially observed maxent models
Suppose we have missing data and/or hidden variables in our model. In general, we canrepresent such models as follows:
p(y,h|θ)=1
Z(θ)exp(∑
cθT
cφc(h,y)) (19.44)

678 Chapter19. Undirectedgraphicalmodels(Markovrandomﬁelds)
The log likelihood has the form
ℓ(θ)=1
N∑
ilog(∑
hip(yi,hi|θ))
=1
N∑
ilog(
1
Z(θ)∑
hi˜p(yi,hi|θ))
(19.45)
where
˜p(y,h|θ)≜exp(∑
cθT
cφc(h,y))
(19.46)
is the unnormalized distribution. The term∑
hi˜p(yi,hi|θ)is the same as the partition function
for the whole model, except that yis ﬁxed at yi. Hence the gradient is just the expected features
where we clamp yi, but average over h:
∂
∂θclog(∑
=E[φc(h,yi)|θ] (19.47)
So the overall gradient is given by
∂ℓ
∂θc=1
N∑
i{E[φc(h,yi)|θ]−E[φc(h,y)|θ]} (19.48)
The ﬁrst set of expectations are computed by “clamping” the visible nodes to their observed
values, and the second set are computed by letting the visible nodes be free. In both cases, wemarginalize over h
i.
An alternative approach is to use generalized EM, where we use gradient methods in the M
step. See (Koller and Friedman 2009, p956) for details.
19.5.3 Approximate methods for computing the MLEs of MRFs

∂ℓ
∂θc=1
N∑
i.
When ﬁtting a UGM there is (in general) no closed form solution for the ML or the MAP estimateof the parameters, so we need to use gradient-based optimizers. This gradient requires inference.In models where inference is intractable, learning also becomes intractable. This has motivatedvarious computationally faster alternatives to ML/MAP estimation, which we list in Table 19.1. Wedicsuss some of these alternatives below, and defer others to later sections.
19.5.4 Pseudo likelihood

One alternative to MLE is to maximize the pseudo likelihood (Besag 1975), deﬁned as follows:
ℓPL(θ)≜∑
yD∑
d=1pemp(ylogp(yd|y−d)=1
NN∑
i=1D∑
d=1logp(yid|yi,−d,θ) (19.49)
That is, we optimize the product of the full conditionals, also known as the composite likeli-
hood(Lindsay 1988), Compare this to the objective for maximum likelihood:
ℓML(θ)=∑
y,xpemp(ylogp(y|θ)=N∑
i=1logp(yi|θ) (19.50)

19.5. Learning 679
Method Restriction Exact MLE? Section
Closed form Only Chordal MRF Exact Section 19.5.7.4
IPF Only Tabular / Gaussian MRF Exact Section 19.5.7
Gradient-based optimization Low tree width Exact Section 19.5.1
Max-margin training Only CRFs N/A Section 19.7
Pseudo-likelihood No hidden variables Approximate Section 19.5.4
Stochastic ML - Exact (up to MC error) Section 19.5.5
Contrastive divergence - Approximate Section 27.7.2.4
Minimum probability ﬂow Can integrate out the hiddens Approximate Sohl-Dickstein et al. (2011)
Table 19.1 Some methods that can be used to compute approximate ML/ MAP parameter estimates for
MRFs/ CRFs. Low tree-width means that, in order for the method to be efficient, the graph must “tree-like”;
see Section 20.5 for details.
(a) (b)
Figure 19.13 (a) A small 2d lattice. (b) The representation used by pseudo likelihood. Solid nodes are
observed neighbors. Based on Figure 2.2 of (Carbonetto 2003).

(a) (b)
In the case of Gaussian MRFs, PL is equivalent to ML (Besag 1975), but this is not true in general
(Liang and Jordan 2008).
The PL approach is illustrated in Figure 19.13 for a 2d grid. We learn to predict each node,
given all of its neighbors. This objective is generally fast to compute since each full conditionalp(y
id|yi,−d,θ)only requires summing over the states of a single node, yid, in order to compute

the local normalization constant. The PL approach is similar to ﬁtting each full conditionalseparately (which is the method used to train dependency networks, discussed in Section 26.2.2),except that the parameters are tied between adjacent nodes.
One problem with PL is that it is hard to apply to models with hidden variables (Parise and
Welling 2005). Another more subtle problem is that each node assumes that its neighbors haveknown values. If node t∈nbr(s)is a perfect predictor for node s, thenswill learn to rely

completely on node t, even at the expense of ignoring other potentially useful information, such
as its local evidence.
However, experiments in (Parise and Welling 2005; Hoeﬂing and Tibshirani 2009) suggest that
PL works as well as exact ML for fully observed Ising models, and of course PL is muchfaster.
19.5.5 Stochastic maximum likelihood
Recall that the gradient of the log-likelihood for a fully observed MRF is given by
∇θℓ(θ)=1
N∑
i[φ(yi)−E[φ(y)]] (19.51)

680 Chapter19. Undirectedgraphicalmodels(Markovrandomﬁelds)
The gradient for a partially observed MRF is similar. In both cases, we can approximate the
model expectations using Monte Carlo sampling. We can combine this with stochastic gradientdescent (Section 8.5.2), which takes samples from the empirical distribution. Pseudocode for theresulting method is shown in Algorithm 3.
Algorithm 19.1: Stochastic maximum likelihood for ﬁtting an MRF
1Initialize weights θrandomly;
2k=0,η=1;
3foreachepoch do
4foreachminibatchofsize Bdo
5 foreachsample s=1:Sdo
6 Sampleys,k∼p(y|θk);
7 ˆE(φ(y)) =1
S∑S
s=1φ(ys,k);
8 foreachtrainingcase iinminibatch do
9 gik=φ(yi)−ˆE(φ(y));
10 gk=1
B∑
i∈Bgik;
11 θk+1=θk−ηgk;
12 k=k+1;
13 Decrease step size η;
Typically we use MCMC to generate the samples. Of course, running MCMC to convergence
at each step of the inner loop would be extremely slow. Fortunately, it was shown in (Younes

2k=0,η=1;
S∑S
10 gk=1
B∑
i∈Bgik;
12 k=k+1;
1989) that we can start the MCMC chain at its previous value, and just take a few steps. Inotherwords, we sample y
s,kby initializing the MCMC chain at ys,k−1, and then run for a few
iterations. This is valid since p(y|θk)is likely to be close to p(y|θk−1), since we only changed
the parameters a small amount. We call this algorithm stochastic maximum likelihood or
SML. (There is a closely related algorithm called persistent contrastive divergence which wediscuss in Section 27.7.2.5.)
19.5.6 Feature induction for maxent models *

MRFs require a good set of features. One unsupervised way to learn such features, known asfeature induction, is to start with a base set of features, and then to continually create newfeature combinations out of old ones, greedily adding the best ones to the model. This approachwas ﬁrst proposed in (Pietra et al. 1997; Zhu et al. 1997), and was later extended to the CRF casein (McCallum 2003).
To illustrate the basic idea, we present an example from (Pietra et al. 1997), which described

how to build unconditional probabilistic models to represent English spelling. Initially the modelhas no features, which represents the uniform distribution. The algorithm starts by choosing toadd the feature
φ
1(y)=∑
tI(yt∈{a,...,z}) (19.52)

19.5. Learning 681
which checks if any letter is lower case or not. After the feature is added, the parameters are
(re)-ﬁt by maximum likelihood. For this feature, it turns out that ˆθ1=1.944, which means that
a word with a lowercase letter in any position is about e1.944≈7times more likely than the
same word without a lowercase letter in that position. Some samples from this model, generatedusing (annealed) Gibbs sampling (Section 24.2), are shown below.
8
m, r, xevo, ijjiir, b, to, jz, gsr, wq, vf, x, ga, msmGh, pcp, d, oziVlal,
hzagh, yzop, io, advzmxnv, ijv_bolft, x, emx, kayerf, mlj, rawzyb, jp, ag,ctdnnnbg, wgdw, t, kguv, cy, spxcq, uzflbbf, dxtkkn, cxwx, jpd, ztzh, lv,zhpkvnu, l^, r, qee, nynrx, atze4n, ik, se, w, lrh, hp+, yrqyka’h, zcngotcnx,igcump, zjcjs, lqpWiqu, cefmfhc, o, lb, fdcY, tzby, yopxmvk, by, fz„ t, govyccm,ijyiduwfzo, 6xr, duh, ejv, pk, pjw, l, fl, w
The second feature added by the algorithm checks if two adjacent characters are lower case:
φ2(y)=∑

φ2(y)=∑
s∼tI(ys∈{a,...,z},yt∈{a,...,z}) (19.53)
Now the model has the form
p(y)=1
Zexp(θ1φ1(y)+θ2φ2(y)) (19.54)
Continuing in this way, the algorithm adds features for the strings s>anding>,w h e r e>
represents the end of word, and for various regular expressions such as [0-9], etc. Some
samples from the model with 1000 features, generated using (annealed) Gibbs sampling, are
shown below.
was, reaser, in, there, to, will, „ was, by, homes, thing, be, reloverated,

φ2(y)=∑
p(y)=1
ther, which, conists, at, fores, anditing, with, Mr., proveral, the, „ ***,on’t, prolling, prothere, „ mento, at, yaou, 1, chestraing, for, have, to,intrally, of, qut, ., best, compers, ***, cluseliment, uster, of, is, deveral,this, thise, of, offect, inatever, thifer, constranded, stater, vill, in, thase,in, youse, menttering, and, ., of, in, verate, of, to
This approach of feature learning can be thought of as a form of graphical model structure
learning (Chapter 26), except it is more ﬁne-grained: we add features that are useful, regardless

of the resulting graph structure. However, the resulting graphs can become densely connected,which makes inference (and hence parameter estimation) intractable.
19.5.7 Iterative proportional ﬁtting (IPF) *
Consider a pairwise MRF where the potentials are represented as tables, with one parameter pervariable setting. We can represent this in log-linear form using
ψ
st(ys,yt)=e x p(
θT
st[I(ys=1,yt=1 ),...,I(ys=K,yt=K)])
(19.55)
and similarly for ψt(yt). Thus the feature vectors are just indicator functions.
8. We thank John Lafferty for sharing this example.

682 Chapter19. Undirectedgraphicalmodels(Markovrandomﬁelds)
FromEquation19.43, wehavethat, atthemaximumofthelikelihood, theempiricalexpectation
of the features equals the model’s expectation:
Epemp[I(ys=j,yt=k)] = Ep(·|θ)[I(ys=j,yt=k)] (19.56)
pemp(ys=j,yt=k)=p( ys=j,yt=k|θ) (19.57)
wherepempis the empirical probability:
pemp(ys=j,yt=k)=Nst,jk
N=∑N
n=1I(yns=j,ynt=k)
N(19.58)
For a general graph, the condition that must hold at the optimum is
pemp(yc)=p(yc|θ) (19.59)
For a special family of graphs known as decomposable graphs (deﬁned in Section 20.4.1), one
can show that p(yc|θ)=ψ c(yc). However, even if the graph is not decomposable, we can
imagine trying to enforce this condition. This suggests an iterative coordinate ascent schemewhere at each step we compute
ψ
t+1
c(yc)=ψt
c(yc)×pemp(yc)
p(yc|ψt)(19.60)
where the multiplication is elementwise. This is known as iterative proportional ﬁtting orIPF
(Fienberg 1970; Bishop et al. 1975). See Algorithm 7 for the pseudocode.

ψ
t+1
c(yc)=ψt
Algorithm 19.2: Iterative Proportional Fitting algorithm for tabular MRFs
1Initializeψc=1forc=1:C;
2repeat
3forc=1:Cdo
4 pc=p(yc|ψ);
5 ˆpc=pemp(yc);
6 ψc=ψc∗ˆpc
pc;
7untilconverged ;
19.5.7.1 Example
Let us consider a simple example from http://en .wikipedia .org/wiki/Iterative_propo
rtional_fitting . We have two binary variables, Y1andY2,w h e r eYn1=1if mannis left
handed, and Yn1=0otherwise; similarly, Yn2=1if woman nis left handed, and Yn2=0
otherwise. We can summarize the data using the following 2×2contingency table:
right-handed left-handed Total
male 43 9 52
female 44 4 48

2repeat
pc;
male 43 9 52
Total 87 13 100

19.5. Learning 683
Suppose we want to ﬁt a disconnected graphical model containing nodes Y1andY2but with
no edge between them. That is, we want to ﬁnd vectors ψ1andψ2such that M≜ψ1ψT
2≈C,
whereMare the model’s expected counts, and Care the empirical counts. By moment
matching, we ﬁnd that the row and column sums of the model must exactly match the row
and column sums of the data. One possible solution is to use ψ1=[ 0.5200,0.4800]and
ψ2=[ 8 7,13]. Below we show the model’s predictions, M=ψ1ψT
2.
male 45.24 6.76 52
female 41.76 6.24 48
It is easy to see that this matches the required constraints. See IPFdemo2x2 for some Matlab
code that computes these numbers. This method is easily to generalized to arbitrary graphs.
19.5.7.2 Speed of IPF
IPF is a ﬁxed point algorithm for enforcing the moment matching constraints and is guaranteed

2.
to converge to the global optimum (Bishop et al. 1975). The number of iterations depends on theform of the model. If the graph is decomposable, then IPF converges in a single iteration, but ingeneral, IPF may require many iterations.
It is clear that the dominant cost of IPF is computing the required marginals under the model.
Efficient methods, such as the junction tree algorithm (Section 20.4), can be used, resulting insomething called efficient IPF (Jirousek and Preucil 1995).

Nevertheless, coordinate descent can be slow. An alternative method is to update all the

parameters at once, by simply following the gradient of the likelihood. This gradient approachhas the further signiﬁcant advantage that it works for models in which the clique potentials maynot be fully parameterized, i.e., the features may not consist of all possible indicators for eachclique, but instead can be arbitrary. Although it is possible to adapt IPF to this setting of generalfeatures, resulting in a method known as iterative scaling, in practice the gradient method is
much faster (Malouf 2002; Minka 2003).
19.5.7.3 Generalizations of IPF

We can use IPF to ﬁt Gaussian graphical models: instead of working with empirical counts, wework with empirical means and covariances (Speed and Kiiveri 1986). It is also possible to createa Bayesian IPF algorithm for sampling from the posterior of the model’s parameters (see e.g.,(Dobra and Massam 2010)).
19.5.7.4 IPF for decomposable graphical models
There is a special family of undirected graphical models known as decomposable graphicalmodels. This is formally deﬁned in Section 20.4.1, but the basic idea is that it contains graphswhich are “tree-like”. Such graphs can be represented by UGMs or DGMs without any loss ofinformation.
In the case of decomposable graphical models, IPF converges in one iteration. In fact, the

684 Chapter19. Undirectedgraphicalmodels(Markovrandomﬁelds)
MLE has a closed form solution (Lauritzen 1996). In particular, for tabular potentials we have
ˆψc(yc=k)=∑N
i=1I(yi,c=k)
N(19.61)
and for Gaussian potentials, we have
ˆμc=∑Ni=1yic
N,ˆΣc=∑
i(yic−ˆμc)(xic−ˆμc)T
N(19.62)
By using conjugate priors, we can also easily compute the full posterior over the model pa-
rameters in the decomposable case, just as we did in the DGM case. See (Lauritzen 1996) fordetails.
19.6 Conditional random ﬁelds (CRFs)
Aconditional random ﬁeld orCRF(Lafferty et al. 2001), sometimes a discriminative random
ﬁeld(Kumar and Hebert 2003), is just a version of an MRF where all the clique potentials are
conditioned on input features:
p(y|x,w)=1
Z(x,w)∏
cψc(yc|x,w) (19.63)
A CRF can be thought of as a structured output extension of logistic regression. We will usually
assume a log-linear representation of the potentials:
ψc(yc|x,w)=e x p (wT
cφ(x,yc)) (19.64)

Z(x,w)∏
whereφ(x,yc)is a feature vector derived from the global inputs xand the local set of labels
yc. We will give some examples below which will make this notation clearer.
The advantage of a CRF over an MRF is analogous to the advantage of a discriminative
classiﬁer over a generative classiﬁer (see Section 8.6), namely, we don’t need to “waste resources”modeling things that we always observe. Instead we can focus our attention on modeling whatwe care about, namely the distribution of labels given the data.

Another important advantage of CRFs is that we can make the potentials (or factors) of the
model be data-dependent. For example, in image processing applications, we may “turn off” thelabel smoothing between two neighboring nodes sandtif there is an observed discontinuity in
the image intensity between pixels sandt. Similarly, in natural language processing problems,

we can make the latent labels depend on global properties of the sentence, such as whichlanguage it is written in. It is hard to incorporate global features into generative models.
The disadvantage of CRFs over MRFs is that they require labeled training data, and they
are slower to train, as we explain in Section 19.6.3. This is analogous to the strengths andweaknesses of logistic regression vs naive Bayes, discussed in Section 8.6.
19.6.1 Chain-structured CRFs, MEMMs and the label-bias problem

The most widely used kind of CRF uses a chain-structured graph to model correlation amongstneighboring labels. Such models are useful for a variety of sequence labeling tasks (see Sec-tion 19.6.2).

19.6. Conditionalrandomﬁelds(CRFs) 685
xt−1 xt xt+1yt−1 yt yt+1
(a)xt−1 xt xt+1yt−1ytyt+1xg
(b)xt−1 xt xt+1yt−1ytyt+1xg
(c)
Figure 19.14 Various models for sequential data. (a) A generative directed HMM. (b) A discriminative
directed MEMM. (c) A discriminative undirected CRF.
Traditionally, HMMs (discussed in detail in Chapter 17) have been used for such tasks. These
are joint density models of the form
p(x,y|w)=T∏
t=1p(yt|yt−1,w)p(xt|yt,w) (19.65)
where we have dropped the initial p(y1)term for simplicity. See Figure 19.14(a). If we observe
bothxtandytfor allt, it is very easy to train such models, using techniques described in
Section 17.5.1.
An HMM requires specifying a generative observation model, p(xt|yt,w), which can be
difficult. Furthemore, each xtis required to be local, since it is hard to deﬁne a generative
model for the whole stream of observations, x=x1:T.
An obvious way to make a discriminative version of an HMM is to “reverse the arrows” from

yttoxt, as in Figure 19.14(b). This deﬁnes a directed discriminative model of the form
p(y|x,w)=∏
tp(yt|yt−1,x,w) (19.66)
wherex=(x1:T,xg),xgare global features, and xtare features speciﬁc to node t. (This
partition into local and global is not necessary, but helps when comparing to HMMs.) This is
called amaximum entropy Markov model orMEMM(McCallum et al. 2000; Kakade et al.
2002).
An MEMM is simply a Markov chain in which the state transition probabilities are conditioned

2002).
on the input features. (It is therefore a special case of an input-output HMM, discussed inSection 17.6.3.) This seems like the natural generalization of logistic regression to the structured-output setting, but it suffers from a subtle problem known (rather obscurely) as the label bias
problem (Lafferty et al. 2001). The problem is that local features at time tdo not inﬂuence states
prior to time t. This follows by examining the DAG, which shows that x
tis d-separated from

yt−1(and all earlier time points) by the v-structure at yt, which is a hidden child, thus blocking
the information ﬂow.
To understand what this means in practice, consider the part of speech (POS) tagging task.
Suppose we see the word “banks”; this could be a verb (as in “he banks at BoA”), or a noun (asin “the river banks were overﬂowing”). Locally the POS tag for the word is ambiguous. However,

686 Chapter19. Undirectedgraphicalmodels(Markovrandomﬁelds)
(a)
 (b)
 (c)
 (d)
 (e)
Figure 19.15 Example of handwritten letter recognition. In the word ’brace’, the ’r’ and the ’c’ look very
similar, but can be disambiguated using context. Source: (Taskar et al. 2003) . Used with kind permission
of Ben Taskar.
suppose that later in the sentence, we see the word “ﬁshing”; this gives us enough context to
infer that the sense of “banks” is “river banks”. However, in an MEMM (unlike in an HMM and
CRF), the “ﬁshing” evidence will not ﬂow backwards, so we will not be able to disambiguate
“banks”.
Now consider a chain-structured CRF. This model has the form
Z(x,w)T∏
t=1ψ(yt|x,w)T−1∏
t=1ψ(yt,yt+1|x,w) (19.67)
From the graph in Figure 19.14(c), we see that the label bias problem no longer exists, since yt
does not block the information from xtfrom reaching other yt′nodes.
The label bias problem in MEMMs occurs because directed models are locally normalized ,

“banks”.
Z(x,w)T∏
meaning each CPD sums to 1. By contrast, MRFs and CRFs are globally normalized , which
means that local factors do not need to sum to 1, since the partition function Z, which sums over
all joint conﬁgurations, will ensure the model deﬁnes a valid distribution. However, this solution
comes at a price: we do not get a valid probability distribution over yuntil we have seen
the whole sentence, since only then can we normalize over all conﬁgurations. Consequently,

CRFs are not as useful as DGMs (whether discriminative or generative) for online or real-time
inference. Furthermore, the fact that Zdepends on all the nodes, and hence all their parameters,
makes CRFs much slower to train than DGMs, as we will see in Section 19.6.3.
19.6.2 Applications of CRFs
CRFs have been applied to many interesting problems; we give a representative sample below.
These applications illustrate several useful modeling tricks, and will also provide motivation for
some of the inference techniques we will discuss in Chapter 20.

19.6.2.1 Handwriting recognition
A natural application of CRFs is to classify hand-written digit strings, as illustrated in Figure 19.15.
The key observation is that locally a letter may be ambiguous, but by depending on the (un-
known) labels of one’s neighbors, it is possible to use context to reduce the error rate. Note
that the node potential, ψt(yt|xt), is often taken to be a probabilistic discriminative classiﬁer,

19.6. Conditionalrandomﬁelds(CRFs) 687
its withdrawal from the UAL Airways rose after announcing  British dealADJ N V IN V PRP N IN N N DTBIO O O BIO I
POSNP I B
Begin noun phrase
Within noun phraseNot a noun phraseNounAdjectiveBIONADJVerbPrepositionPossesive pronounDeterminer (e.g., a, an, the)VINPRPDTKEY
Figure 19.16 A CRF for joint POS tagging and NP segmentation. Source: Figure 4.E.1 of (Koller and
such as a neural network or RVM, that is trained on isolated letters, and the edge potentials,
ψst(ys,yt), are often taken to be a language bigram model. Later we will discuss how to train
all the potentials jointly.
19.6.2.2 Noun phrase chunking
One common NLP task is noun phrase chunking, which refers to the task of segmenting a
sentence into its distinct noun phrases (NPs). This is a simple example of a technique known asshallow parsing.
In more detail, we tag each word in the sentence with B (meaning beginning of a new NP), I

(meaning inside a NP), or O (meaning outside an NP). This is called BIOnotation. For example,
in the following sentence, the NPs are marked with brackets:
B I O O OB IO B I I
(British Airways) rose after announcing (its withdrawl) from (the UAI deal)
(We need the B symbol so that we can distinguish II, meaning two words within a single NP,
from BB, meaning two separate NPs.)
A standard approach to this problem would ﬁrst convert the string of words into a string of
POS tags, and then convert the POS tags to a string of BIOs. However, such a pipeline method

can propagate errors. A more robust approach is to build a joint probabilistic model of the
formp(NP
1:T,POS1:T|words1:T). One way to do this is to use the CRF in Figure 19.16. The

formp(NP
connections between adjacent labels encode the probability of transitioning between the B, Iand O states, and can enforce constraints such as the fact that B must preceed I. The featuresare usually hand engineered and include things like: does this word begin with a capital letter, isthis word followed by a full stop, is this word a noun, etc. Typically there are ∼1,000−10,000
features per node.
The number of features has minimal impact on the inference time, since the features are

observed and do not need to be summed over. (There is a small increase in the cost of

688 Chapter19. Undirectedgraphicalmodels(Markovrandomﬁelds)
 Mrs. Green spoke today in New Y ork
()Green chairs the ﬁnance committe eB-PER I-PER OTH OTH OTH B-LOC I-LOC B-PER OTH OTH OTH OTH
KEY
Begin person name
Within person nameBegin location nameB-PERI-PERB-LOCWithin location nameNot an entitiyI-LOCOTH
Figure 19.17 A skip-chain CRF for named entity recognition. Source: Figure 4.E.1 of (Koller and Friedman
2009). Used with kind permission of Daphne Koller.
evaluating potential functions with many features, but this is usually negligible; if not, one can
useℓ1regularization to prune out irrelevant features.) However, the graph structure can have a
dramatic effect on inference time. The model in Figure 19.16 is tractable, since it is essentially a“fat chain”, so we can use the forwards-backwards algorithm (Section 17.4.3) for exact inferenceinO(T|POS|
2|NP|2)time, where|POS|is the number of POS tags, and |NP|is the number

of NP tags. However, the seemingly similar graph in Figure 19.17, to be explained below, iscomputationally intractable.
19.6.2.3 Named entity recognition
A task that is related to NP chunking is named entity extraction. Instead of just segmenting
out noun phrases, we can segment out phrases to do with people and locations. Similartechniques are used to automatically populate your calendar from your email messages; this iscalledinformation extraction.
A simple approach to this is to use a chain-structured CRF, but to expand the state space

from BIO to B-Per, I-Per, B-Loc, I-Loc, and Other. However, sometimes it is ambiguous whethera word is a person, location, or something else. (Proper nouns are particularly difficult to dealwith because they belong to an open class, that is, there is an unbounded number of possible

names, unlike the set of nouns and verbs, which is large but essentially ﬁxed.) We can get betterperformance by considering long-range correlations between words. For example, we might adda link between all occurrences of the same word, and force the word to have the same tag ineach occurence. (The same technique can also be helpful for resolving the identity of pronouns.)This is known as a skip-chain CRF. See Figure 19.17 for an illustration.
We see that the graph structure itself changes depending on the input, which is an additional

advantage of CRFs over generative models. Unfortunately, inference in this model is gener-ally more expensive than in a simple chain with local connections, for reasons explained inSection 20.5.

19.6. Conditionalrandomﬁelds(CRFs) 689
Figure 19.18 Illustration of a simple parse tree based on a context free grammar in Chomsky normal
form. The feature vector φ(x,y)=Ψ (x,y)counts the number of times each production rule was used.
Source: Figure 5.2 of (Altun et al. 2006) . Used with kind permission of Yasemin Altun.
19.6.2.4 Natural language parsing
A generalization of chain-structured models for language is to use probabilistic grammars. In
particular, a probabilistic context free grammar orPCFGis a set of re-write or production
rules of the form σ→σ′σ′′orσ→x,w h e r eσ,σ′,σ′′∈Σare non-terminals (analogous to
parts of speech), and x∈Xare terminals, i.e., words. See Figure 19.18 for an example. Each
such rule has an associated probability. The resulting model deﬁnes a probability distribution
over sequences of words. We can compute the probability of observing a particular sequence
x=x1...xTby summing over all trees that generate it. This can be done in O(T3)time

using the inside-outside algorithm ; see e.g., (Jurafsky and Martin 2008; Manning and Schuetze
1999) for details.
PCFGs are generative models. It is possible to make discriminative versions which encode
the probability of a labeled tree, y, given a sequence of words, x, by using a CRF of the form
p(y|x)∝exp(wTφ(x,y)). For example, we might deﬁne φ(x,y)to count the number of
times each production rule was used (which is analogous to the number of state transitions in
a chain-structured model). See e.g., (Taskar et al. 2004) for details.
19.6.2.5 Hierarchical classiﬁcation

Suppose we are performing multi-class classiﬁcation, where we have a label taxonomy , which
groups the classes into a hierarchy. We can encode the position of ywithin this hierarchy by
deﬁning a binary vector φ(y), where we turn on the bit for component yand for all its children.
This can be combined with input features φ(x)using a tensor product, φ(x,y)=φ(x)⊗φ(y).
See Figure 19.19 for an example.
This method is widely used for text classiﬁcation, where manually constructed taxnomies

(such as the Open Directory Project at www.dmoz.org) are quite common. The beneﬁt is that
information can be shared between the parameters for nearby categories, enabling generalization
across classes.

690 Chapter19. Undirectedgraphicalmodels(Markovrandomﬁelds)
⟨w,Ψ(x,2)⟩=⟨w2,x⟩+⟨w6,x⟩+⟨w9,x⟩
Figure 19.19 Illustration of a simple label taxonomy, and how it can be used to compute a distributed
representation for the label for class 2. In this ﬁgure, φ(x)=x,φ(y=2 )=Λ ( 2 ) ,φ(x,y)is denoted
byΨ(x,2), andwTφ(x,y)is denoted by ⟨w,Ψ(x,2)⟩. Source: Figure 5.1 of (Altun et al. 2006) . Used
with kind permission of Yasemin Altun.
19.6.2.6 Protein side-chain prediction
An interesting analog to the skip-chain model arises in the problem of predicting the structure
of protein side chains. Each residue in the side chain has 4 dihedral angles, which are usually
discretized into 3 values called rotamers. The goal is to predict this discrete sequence of angles,
y, from the discrete sequence of amino acids, x.
We can deﬁne an energy function E(x,y), where we include various pairwise interaction
terms between nearby residues (elements of the yvector). This energy is usually deﬁned as a

weighted sum of individual energy terms, E(x,y|w)=∑D
j=1θjEj(x,y), where the Ejare
energy contribution due to various electrostatic charges, hydrogen bonding potentials, etc, and
ware the parameters of the model. See (Yanover et al. 2007) for details.
Given the model, we can compute the most probable side chain conﬁguration using y∗=
argminE(x,y|w). In general, this problem is NP-hard, depending on the nature of the graph
induced by the Ejterms, due to long-range connections between the variables. Nevertheless,

some special cases can be efficiently handled, using methods discussed in Section 22.6.
19.6.2.7 Stereo vision
Low-level vision problems are problems where the input is an image (or set of images), and
the output is a processed version of the image. In such cases, it is common to use 2d lattice-
structured models; the models are similar to Figure 19.9, except that the features can be global,
and are not generated by the model. We will assume a pairwise CRF.
A classic low-level vision problem is dense stereo reconstruction , where the goal is to

estimate the depth of every pixel given two images taken from slightly different angles. In this
section (based on (Sudderth and Freeman 2008)), we give a sketch of how a simple CRF can be
used to solve this task. See e.g., (Sun et al. 2003) for a more sophisticated model.
By using some standard preprocessing techniques, one can convert depth estimation into a

19.6. Conditionalrandomﬁelds(CRFs) 691
problem of estimating the disparity ysbetween the pixel at location (is,js)in the left image
and the corresponding pixel at location (is+ys,js)in the right image. We typically assume
that corresponding pixels have similar intensity, so we deﬁne a local node potential of the form
ψs(ys|x)∝exp{
−1
2σ2(xL(is,js)−xR(is+ys,js))2}
(19.68)
wherexLis the left image and xRis the right image. This equation can be generalized to model
the intensity of small windows around each location. In highly textured regions, it is usually
possible to ﬁnd the corresponding patch using cross correlation, but in regions of low texture,there will be considerable ambiguity about the correct value of y
s.
We can easily add a Gaussian prior on the edges of the MRF that encodes the assumption
that neighboring disparities ys,ytshould be similar, as follows:
ψst(ys,yt)∝exp(
−1
2γ2(ys−yt)2)
(19.69)
The resulting model is a Gaussian CRF.

s.
−1
(19.69)
However, using Gaussian edge-potentials will oversmooth the estimate, since this prior fails
to account for the occasional large changes in disparity that occur between neighboring pixelswhich are on different sides of an occlusion boundary. One gets much better results using atruncated Gaussian potential of the form
ψ
st(ys,yt)∝exp{
−1
2γ2min(
(ys−yt)2,δ2
0)}
(19.70)
whereγencodes the expected smoothness, and δ0encodes the maximum penalty that will

(19.69)
ψ
−1
2γ2min(
0)}
(19.70)
be imposed if disparities are signiﬁcantly different. This is called a discontinuity preserving
potential; note that such penalties are not convex. The local evidence potential can be maderobust in a similar way, in order to handle outliers due to specularities, occlusions, etc.
Figure 19.20 illustrates the difference between these two forms of prior. On the top left is an

−1
2γ2min(
0)}
(19.70)
image from the standard Middlebury stereo benchmark dataset (Scharstein and Szeliski 2002).On the bottom left is the corresponding true disparity values. The remaining columns representthe estimated disparity after 0, 1 and an “inﬁnite” number of rounds of loopy belief propagation(see Section 22.2), where by “inﬁnite” we mean the results at convergence. The top row showsthe results using a Gaussian edge potential, and the bottom row shows the results using thetruncated potential. The latter is clearly better.

Unfortunately, performing inference with real-valued variables is computationally difficult,
unless the model is jointly Gaussian. Consequently, it is common to discretize the variables.(For example, Figure 19.20(bottom) used 50 states.) The edge potentials still have the form givenin Equation 19.69. The resulting model is called a metric CRF, since the potentials form a
metric.
9Inference in metric CRFs is more efficient than in CRFs where the discrete labels
have no natural ordering, as we explain in Section 22.6.3.3. See Section 22.6.4 for a comparisonof various approximate inference methods applied to low-level CRFs, and see (Blake et al. 2011;Prince 2012) for more details on probabilistic models for computer vision.
9. A function fis said to be a metricif it satisﬁes the following three properties: Reﬂexivity: f(a,b)=0 iffa=b;
Symmetry: f(a,b)=f (b,a); and Triangle inequality: f(a,b)+f(b,c)≥f(a,c).I f fsatisﬁes only the ﬁrst two
properties, it is called a semi-metric.

692 Chapter19. Undirectedgraphicalmodels(Markovrandomﬁelds)
/g2/g3 /g5/g10/g13/g7/g1/g4/g8/g11/g9/g6/g10/g8/g12/g8/g7/g11 
Figure 19.20 Illustration of belief propagation for stereo depth estimation. Left column: image and true
disparities. Remaining columns: initial estimate, estimate after 1 iteration, and estimate at convergence.
Top row: Gaussian edge potentials. Bottom row: robust edge potentials. Source: Figure 4 of (Sudderth and
Freeman 2008). Used with kind permission of Erik Sudderth.
19.6.3 CRF training
We can modify the gradient based optimization of MRFs described in Section 19.5.1 to the CRF
case in a straightforward way. In particular, the scaled log-likelihood becomes
ℓ(w)≜1
N∑
ilogp(yi|xi,w)=1
N∑
i[∑
cwT
cφc(yi,xi)−logZ(w,xi)]
(19.71)
and the gradient becomes
∂ℓ
∂wc=1
N∑
i[
φc(yi,xi)−∂
∂wclogZ(w,xi)]
(19.72)
=1
N∑
i[φc(yi,xi)−E[φc(y,xi)]] (19.73)
Note that we now have to perform inference for every single training case inside each gradient

ℓ(w)≜1
N∑
N∑
i[∑
cwT
(19.71)
∂ℓ
∂wc=1
N∑
i[
(19.72)
=1
N∑
step, which is O(N)times slower than the MRF case. This is because the partition function
depends on the inputs xi.
In most applications of CRFs (and some applications of MRFs), the size of the graph structure
can vary. Hence we need to use parameter tying to ensure we can deﬁne a distribution of
arbitrary size. In the pairwise case, we can write the model as follows:
Z(w,x)exp(
wTφ(y,x))
(19.74)

19.7. StructuralSVMs 693
wherew=[wn,we]are the node and edge parameters, and
φ(y,x)≜[∑
tφt(yt,x),∑
s∼tφst(ys,yt,x)] (19.75)
are the summed node and edge features (these are the sufficient statistics). The gradient
expression is easily modiﬁed to handle this case.
In practice, it is important to use a prior/ regularization to prevent overﬁtting. If we use a
Gaussian prior, the new objective becomes
ℓ′(w)≜1
N∑
ilogp(yi|xi,w)−λ||w||2
2 (19.76)
It is simple to modify the gradient expression.
Alternatively, we can use ℓ1regularization. For example, we could use ℓ1for the edge weights
weto learn a sparse graph structure, and ℓ2for the node weights wn, as in (Schmidt et al.
2008). In other words, the objective becomes
ℓ′(w)≜1
N∑
ilogp(yi|xi,w)−λ1||we||1−λ2||wn||22(19.77)
Unfortunately, the optimization algorithms are more complicated when we use ℓ1(see Sec-
tion 13.4), although the problem is still convex.
To handle large datasets, we can use stochastic gradient descent (SGD), as described in
Section 8.5.2.

ℓ′(w)≜1
N∑
It is possible (and useful) to deﬁne CRFs with hidden variables, for example to allow for an
unknown alignment between the visible features and the hidden labels (see e.g., (Schnitzspan
et al. 2010)). In this case, the objective function is no longer convex. Nevertheless, we can ﬁnda locally optimal ML or MAP parameter estimate using EM and/ or gradient methods.
19.7 Structural SVMs

We have seen that training a CRF requires inference, in order to compute the expected sufficientstatistics needed to evaluate the gradient. For certain models, computing a joint MAP estimateof the states is provably simpler than computing marginals, as we discuss in Section 22.6. In thissection, we discuss a way to train structured output classiﬁers that that leverages the existence offast MAP solvers. (To avoid confusion with MAP estimation of parameters, we will often refer toMAP estimation of states as decoding.) These methods are known as structural support vector

machines orSSVMs(Tsochantaridis et al. 2005). (There is also a very similar class of methods
known as max margin Markov networks orM3nets(Taskar et al. 2003); see Section 19.7.2 for
a discussion of the differences.)
19.7.1 SSVMs: a probabilistic view
In this book, we have mostly concentrated on ﬁtting models using MAP parameter estimation,i.e., by minimizing functions of the form
R
MAP(w)=−logp(w)−N∑
i=1logp(yi|xi,w) (19.78)

694 Chapter19. Undirectedgraphicalmodels(Markovrandomﬁelds)
However, at test time, we pick the label so as to minimize the posterior expected loss (deﬁned
in Section 5.7):
ˆy(x|w) = argmin
ˆy∑
yL(ˆy,y)p(y|x,w) (19.79)
whereL(y∗,ˆy)is the loss we incur when we estimate ˆybut the truth is y∗. It therefore seems
reasonable to take the loss function into account when performing parameter estimation.10So,
following (Yuille and He 2011), let us instead minimized the posterior expected loss on thetraining set:
R
EL(w)≜−logp(w)+N∑
i=1log[∑
yL(yi,y)p(y|xi,w)]
(19.80)
In the special case of 0-1 loss, L(yi,y)=1−δy,yi, this reduces to RMAP.
We will assume that we can write our model in the following form:
p(y|x,w)=exp(wTφ(x,y))
Z(x,w)(19.81)
p(w)=exp(−E(w))
Z(19.82)
whereZ(x,w)=∑
yexp(wTφ(x,y)). Also, let us deﬁne L(yi,y)=e x p ˜L(yi,y). With
this, we can rewrite our objective as follows:
REL(w)=− logp(w)+∑
ilog[∑
yexp˜L(yi,y)exp(wTφ(x,y))
Z(x,w)]
(19.83)
=E(w)+∑
i−logZ(xi,w)+log∑
yexp(
˜L(vyi,y)+wTφ(xi,y))
(19.84)

R
i=1log[∑
(19.80)
Z(19.82)
ilog[∑
Z(x,w)]
(19.83)
=E(w)+∑
yexp(
(19.84)
We will now consider various bounds in order to simplify this objective. First note that for
any function f(y)we have
max
y∈Yf(y)≤log∑
y∈Yexp[f(y)]≤log[
|Y|exp(
max
yf(y))]
=l o g|Y|+max
yf(y)(19.85)
For example, suppose Y={0,1,2}andf(y)=y. Then we have
2 = log[exp(2)] ≤log[exp(0)+exp(1)+exp(2)] ≤log[3×exp(2)] = log(3)+2 (19.86)
We can ignore the log|Y|term, which is independent of y, and treat maxy∈Yf(y)as both a
lower and upper bound. Hence we see that
REL(w)∼E(w)+N∑
i=1[
max
y{
˜L(yi,y)+wTφ(xi,y)}
−max

max
|Y|exp(
max
yf(y))]
i=1[
max
y{
−max
ywTφ(xi,y)]
(19.87)
10. Note that this violates the fundamental Bayesian distinction between inference and decision making. However,
performing these tasks separately will only result in an optimal decision if we can compute the exact posterior. In most
cases, this is intractable, so we need to perform loss-calibrated inference (Lacoste-Julien et al. 2011). In this section,
we just perform loss-calibrated MAP parameter estimation, which is computationally simpler. (See also (Stoyanov et al.2011).)

19.7. StructuralSVMs 695
wherex∼ymeansc1+x≤y+c2for some constants c1,c2. Unfortunately, this objective
is not convex in w. However, we can devise a convex upper bound by exploiting the following
looser lower bound on the log-sum-exp function:
f(y′)≤log∑
yexp[f(y)] (19.88)
for anyy′∈Y. Applying this equation to our earlier example, for f(y)=yandy′=1,w eg e t
1 = log[exp(1)] ≤log[exp(0)+exp(1)+exp(2)] . And applying this bound to RELwe get
REL(w)≤E(w)+N∑
i=1[
max
y{
−wTφ(xi,yi)]
(19.89)
If we setE(w)=−1
2C||w||2
2(corresponding to a spherical Gaussian prior), we get
RSSVM(w)≜1
i=1[
max
y{
(19.90)
This is the same objective as used in the SSVM approach of (Tsochantaridis et al. 2005).
In the special case that Y={−1,+1}L(y∗,y)=1−δy,y∗, andφ(x,y)=1
2yx, this
criterion reduces to the following (by considering the two cases that y=yiandy̸=yi):
RSVM(w)≜1
i=1[
max{0,1−yiwTxi}]
(19.91)

i=1[
max
y{
(19.89)
2C||w||2
i=1[
max
y{
(19.90)
2yx, this
RSVM(w)≜1
i=1[
(19.91)
which is the standard binary SVM objective (see Equation 14.57).
So we see that the SSVM criterion can be seen as optimizing an upper bound on the Bayesian
objective, a result ﬁrst shown in (Yuille and He 2011). This bound will be tight (and hence
the approximation will be a good one) when ||w||is large, since in that case, p(y|x,w)will
concentrate its mass on argmaxyp(y|x,w). Unfortunately, a large ||w||corresponds to a

RSVM(w)≜1
i=1[
(19.91)
model that is likely to overﬁt, so it is unlikely that we will be working in this regime (because wewill tune the strength of the regularizer to avoid this situation). An alternative justiﬁcation for theSVM criterion is that it focusses effort on ﬁtting parameters that affect the decision boundary.This is a better use of computational resources than ﬁtting the full distribution, especially whenthe model is wrong.
19.7.2 SSVMs: a non-probabilistic view

We now present SSVMs in a more traditional (non-probabilistic) way, following (Tsochantaridiset al. 2005). The resulting objective will be the same as the one above. However, this derivationwill set the stage for the algorithms we discuss below.
Letf(x;w) = argmax
y∈YwTφ(x,y)be the prediction function. We can obtain zero loss
on the training set using this predictor if
∀i.max
y∈Y\yiwTφ(xi,y)≤wTφ(xi,yi) (19.92)

696 Chapter19. Undirectedgraphicalmodels(Markovrandomﬁelds)
Each one of these nonlinear inequalities can be equivalently replaced by |Y|−1linear inequal-
ities, resulting in a total of N|Y|−Nlinear constraints of the following form:
∀i.∀y∈Y\yi.wTφ(xi,yi)−wTφ(xi,y)≥0 (19.93)
For brevity, we introduce the notation
δi(y)≜φ(xi,yi)−φ(xi,y) (19.94)
so we can rewrite these constraints as wTδi(y)≥0.
If we can achieve zero loss, there will typically be multiple solution vectors w. We pick the
one that maximizes the margin, deﬁned as
γ≜min
if(x,yi;w)−max
y′∈Y\yf(x,y′;w) (19.95)
Since the margin can be made arbitrarily large by rescaling w, we ﬁx its norm to be 1, resulting
in the optimization problem
max
γ,w:||w||=1s.t.∀i.∀y∈Y\yi.wTδi(y)≥γ (19.96)
Equivalently, we can write
min
w1
2||w||2s.t.∀i.∀y∈Y\yi.wTδi(y)≥1 (19.97)
To allow for the case where zero loss cannot be achieved (equivalent to the data being inseparable
in the case of binary classiﬁcation), we relax the constraints by introducing slack terms ξi, one

γ≜min
max
min
w1
per data case. This yields
min
w,ξ1
i=1ξis.t.∀i.∀y∈Y\yi.wTδi(y)≥1−ξi,ξi≥0 (19.98)
In the case of structured outputs, we don’t want to treat all constraint violations equally. For
example, in a segmentation problem, getting one position wrong should be punished less thangetting many positions wrong. One way to achieve this is to divide the slack variable by the sizeof the loss (this is called slack re-scaling). This yields
min
w,ξ1
i=1ξis.t.∀i.∀y∈Y\yi.wTδi(y)≥1−ξi
L(yi,y),ξi≥0(19.99)

min
w,ξ1
min
w,ξ1
Alternatively, we can deﬁne the margin to be proportional to the loss (this is called margin
re-rescaling). This yields
min
w,ξ1
i=1ξis.t.∀i.∀y∈Y\yi.wTδi(y)≥L(yi,y)−ξi,ξi≥0(19.100)
(In fact, we can write ∀y∈Yinstead of∀y∈Y\yi, since if y=yi, thenwTδi(y)=0and
ξi=0. By using the simpler notation, which doesn’t exclude yi, we add an extra but redundant
constraint.) This latter approach is used in M3nets.

19.7. StructuralSVMs 697
For future reference, note that we can solve for the ξ∗
iterms as follows:
ξ∗
i(w)=m a x{0,max
y(L(yi,y)−wTδi))}=m a x
y(L(yi,y)−wTδi)) (19.101)
Substituting in, and dropping the constraints, we get the following equivalent problem:
min
w1
2||w||2+C∑
imax
y{
L(yi,y)+wTφ(xi,y)}
−wTφ(xi,yi) (19.102)
19.7.2.1 Empirical risk minimization
Let us pause and consider whether the above objective is reasonable. Recall that in the frequen-
tist approach to machine learning (Section 6.5), the goal is to minimize the regularized empiricalrisk, deﬁned by
R(w)+C
NN∑
i=1L(yi,f(xi,w)) (19.103)
whereR(w)is the regularizer, and f(xi,w) = argmaxywTφ(xi,y)=ˆyiis the prediction.
Since this objective is hard to optimize, because the loss is not differentiable, we will constructa convex upper bound instead.
We can show that
R(w)+C
N∑
imax
y(L(yi,y)−wTδi)) (19.104)
is such a convex upper bound. To see this, note that
L(yi,f(xi,w))≤L(yi,f(xi,w))−wTφ(xi,yi)+wTφ(xi,ˆyi) (19.105)
≤max

R(w)+C
NN∑
R(w)+C
N∑
imax
≤max
yL(yi,y)−wTφ(xi,yi)+wTφ(xi,y) (19.106)
Using this bound and R(w)=1
2||w||2yields Equation 19.102.
19.7.2.2 Computational issues
Although the above objectives are simple quadratic programs (QP), they have O(N|Y|)con-
straints. This is intractable, since Yis usually exponentially large. In the case of the margin
rescaling formulation, it is possible to reduce the exponential number of constraints to a poly-nomial number, provided the loss function and the feature vector decompose according to agraphical model. This is the approach used in M3nets (Taskar et al. 2003).

An alternative approach is to work directly with the exponentially sized QP. This allows for
the use of more general loss functions. There are several possible methods to make this feasible.One is to use cutting plane methods. Another is to use stochastic subgradient methods. Wediscuss both of these below.

698 Chapter19. Undirectedgraphicalmodels(Markovrandomﬁelds)
Fi C llh S f h l lh I
Figure 19.21 Illustration of the cutting plane algorithm in 2d. We start with the estimate w=w0=0.
(a) We add the ﬁrst constraint; the shaded region is the new feasible set. The new minimum norm solution
isw1. (b) We add another constraint; the dark shaded region is the new feasible set. (c) We add a third
constraint. Source: Figure 5.3 of (Altun et al. 2006) . Used with kind permission of Yasemin Altun.
19.7.3 Cutting plane methods for ﬁtting SSVMs
In this section, we discuss an efficient algorithm for ﬁtting SSVMs due to (Joachims et al. 2009).
This method can handle general loss functions, and is implemented in the popular SVMstruct
package11. The method is based on the cutting plane method from convex optimization (Kelley
1960).
The basic idea is as follows. We start with an initial guess wand no constraints. At each
iteration, we then do the following: for each example i, we ﬁnd the “most violated” constraint

1960).
involving xiandˆyi. If the loss-augmented margin violation exceeds the current value of ξiby
more than ϵ, we add ˆyito the working set of constraints for this training case, Wi, and then
solve the resulting new QP to ﬁnd the new w,ξ. See Figure 19.21 for a sketch, and Algorithm 11
for the pseudo code. (Since at each step we only add one new constraint, we can warm-start
the QP solver.) We can can easily modify the algorithm to optimize the slack rescaling version
by replacing the expression L(yi,y)−wTδi(ˆyi)withL(yi,y)(1−wTδi(ˆyi)).

The key to the efficiency of this method is that only polynomially many constraints need to
be added, and as soon as they are, the exponential number of other constraints are guaranteed
to also be satisﬁed to within a tolerance of ϵ(see (Tsochantaridis et al. 2005) for the proof).
19.7.3.1 Loss-augmented decoding
The other key to efficiency is the ability to ﬁnd the most violated constraint in line 5 of the
algorithm, i.e., to compute
argmax
y∈YL(yi,y)−wTδi(y) = argmax
y∈YL(yi,y)+wTφ(xi,y) (19.107)
11.http://svmlight .joachims .org/svm_struct .html

19.7. StructuralSVMs 699
Algorithm 19.3: Cutting plane algorithm for SSVMs (margin rescaling, N-slack version)
1InputD={(x1,y1),...,(xN,yn)},C,ϵ;
2Wi=∅,ξi=0fori=1:N;
3repeat
5 ˆyi=a r g m a xˆyi∈YL(yi,y)−wTδi(ˆyi);
6 ifL(yi,y)−wTδi(ˆyi)>ξi+ϵthen
7Wi=Wi∪{ˆyi};
8 (w,ξ) = argminw,ξ≥01
2||w||2
2+C∑N
i=1ξi;
9 s.t.∀i=1:N,∀y′∈Wi:wTδi(ˆyi)≥L(yi,y′)−ξi;
10untilnoWihaschanged ;
11Return (w ,ξ)
We call this process loss-augmented decoding . (In (Joachims et al. 2009), this procedure is
called the separation oracle .) If the loss function has an additive decomposition of the same
form as the features, then we can fold the loss into the weight vector, i.e., we can ﬁnd a new
set of parameters w′such that (w′)Tδi(y)=wTδi(y). We can then use a standard decoding
algorithm, such as Viterbi, on the model p(y|x,w′).
Inthespecialcaseof0-1loss, theoptimumwilleitherbethebestsolution, argmaxywTφ(xi,y),
with a value of of 0−wTδi(ˆy), or it will be the second best solution, i.e.,
˜y=a r g m a x
y̸=ˆywTφ(xi,y) (19.108)

˜y=a r g m a x
which achieves an overall value of 1−wTδi(˜y). For chain structured CRFs, we can use the
Viterbi algorithm to do decoding; the second best path will differ from the best path in a singleposition, which can be obtained by changing the variable whose max marginal is closest to itsdecision boundary to its second best value. We can generalize this (with a bit more work) toﬁnd theN-best list (Schwarz and Chow 1990; Nilsson and Goldberger 2001).
For Hamming loss, L(y
∗,y)=∑
tI(y∗
t̸=yt), and for the F1 score (deﬁned in Section 5.7.2.3),

∗,y)=∑
tI(y∗
we can devise a dynamic programming algorithm to compute Equation 19.107. See (Altun et al.2006) for details. Other models and loss function combinations will require different methods.
19.7.3.2 A linear time algorithm
Although the above algorithm takes polynomial time, we can do better, and devise an algorithmthat runs in lineartime, assuming we use a linear kernel (i.e., we work with the original features
φ(x,y)and do not apply the kernel trick). The basic idea, as explained in (Joachims et al.
2009), is to have a single slack variable, ξ, instead of N, but to use|Y|

Nconstraints, instead of

700 Chapter19. Undirectedgraphicalmodels(Markovrandomﬁelds)
justN|Y|. Speciﬁcally, we optimize the following (assuming the margin rescaling formulation):
min
w,ξ≥01
2||w||2
2+Cξ
s.t.∀(y1,...,yN)∈YN:1
NwTN∑
i=1δi(yi)≥1
NN∑
i=1L(yi,yi)−ξ(19.109)
Compare this to the original version, which was
min
w,ξ≥01
2||w||22+C
Nξs.t.∀i=1:N,∀y∈Y:wTδi(y)≥L(yi,yi)−ξi(19.110)
One can show that any solution w∗of Equation 19.109 is also a solution of Equation 19.110 and
vice versa, with ξ∗=1
Nξ∗
i.
Algorithm 19.4: Cutting plane algorithm for SSVMs (margin rescaling, 1-slack version)
2W=∅;
3repeat
4(w,ξ) = argminw,ξ≥01
2||w||2
2+C∑N
i=1ξ;
5 s.t.∀(y1,...,yN)∈W:1
NwT∑Ni=1δi(yi)≥1
N∑Ni=1L(yi,yi)−ξ;
6fori=1:Ndo
7 ˆyi=a r g m a xˆyi∈YL(yi,ˆyi)+wTφ(xi,ˆyi)
8W=W∪{(ˆy1,...,ˆyN)};
9until1
N∑Ni=1L(yi,ˆyi)−1
NwT∑Ni=1δi(ˆyi)≤ξ+ϵ;
10Return (w ,ξ)
We can optimize Equation 19.109 using the cutting plane algorithm in Algorithm 10. (This

Nξ∗
i.
2W=∅;
3repeat
2||w||2
2+C∑N
i=1ξ;
9until1
is what is implemented in SVMstruct.) The inner QP in line 4 can be solved in O(N)time
using the method of (Joachims 2006). In line 7 we make Ncalls to the loss-augmented decoder.
Finally, it can be shown that the number of iterations is a constant independent on N. Thus
the overall running time is linear.
19.7.4 Online algorithms for ﬁtting SSVMs
Although the cutting plane algorithm can be made to run in time linear in the number of data

points, that can still be slow if we have a large dataset. In such cases, it is preferable to useonline learning. We brieﬂy mention a few possible algorithms below.
19.7.4.1 The structured perceptron algorithm
A very simple algorithm for ﬁtting SSVMs is the structured perceptron algorithm (Collins
2002). This method is an extension of the regular perceptron algorithm of Section 8.5.4. At each

19.7. StructuralSVMs 701
step, we compute ˆy=a r g m a x p(y|x)(e.g., using the Viterbi algorithm) for the current training
samplex.I fˆy=y, we do nothing, otherwise we update the weight vector using
wk+1=wk+φ(y,x)−φ(ˆy,x) (19.111)
To get good performance, it is necessary to average the parameters over the last few updates
(see Section 8.5.2 for details), rather than using the most recent value.
19.7.4.2 Stochastic subgradient descent
The disadvantage of the structured perceptron algorithm is that it implicitly assumes 0-1 loss,and it does not enforce any kind of margin. An alternative approach is to perform stochasticsubgradient descent. A speciﬁc instance of this the Pegasos algorithm (Shalev-Shwartz et al.
2007), which stands for “primal estimated sub-gradient solver for SVM”. Pegasos was designedfor binary SVMs, but can be extended to SSVMS.
Let us start by considering the objective function:
f(w)=
N∑
i=1max
ˆyi[
L(yi,ˆyi)+wTφ(xi,ˆyi)]
−wTφ(xi,yi)+λ||w||2(19.112)

f(w)=
N∑
i=1max
ˆyi[
Lettingˆyibe the argmax of this max. Then the subgradient of this objective function is
g(w)=N∑
i=1φ(xi,ˆyi)−φ(xi,yi)+2λw (19.113)
In stochastic subgradient descent, we approximate this gradient with a single term, i, and then
perform an update:
wk+1=wk−ηkgi(wk)=wk−ηk[φ(xi,ˆyi)−φ(xi,yi)+(2/N)λw] (19.114)
whereηkis the step size parameter, which should satisfy the Robbins-Monro conditions (Sec-
tion 8.5.2.1). (Notice that the perceptron algorithm is just a special case where λ=0and
ηk=1.) To ensure that whas unit norm, we can project it onto the ℓ2ball after each update.
19.7.5 Latent structural SVMs
In many applications of interest, we have latent or hidden variables h. For example, in object

detections problems, we may be told that the image contains an object, so y=1, but we may
not know where it is. The location of the object, or its pose, can be considered a hidden variable.Or in machine translation, we may know the source text x(say English) and the target text y
(say French), but we typically do not know the alignment between the words.
We will extend our model as follows, to get a latent CRF:
p(y,h|x,w)=exp(w
Tφ(x,y,h))
Z(x,w)(19.115)
Z(x,w)=∑
y,hexp(wTφ(x,y,h)) (19.116)

702 Chapter19. Undirectedgraphicalmodels(Markovrandomﬁelds)
In addition, we introduce the loss function L(y∗,y,h); this measures the loss when the “action”
that we take is to predict yusing latent variables h. We could just use L(y∗,y)as before, since
his usually a nuisance variable and not of direct interest. However, hcan sometimes play a
useful role in deﬁning a loss function.12
Given the loss function, we deﬁne our objective as
ilog⎡
⎣∑
y,hexp˜L(yi,y,h)exp(wTφ(x,y,h))
Z(x,w)⎤⎦(19.117)
Using the same loose lower bound as before, we get
R
EL(w)≤E(w)+N∑
i=1max
y,h{
˜L(yi,y,h)+wTφ(xi,y,h)}
−N∑
i=1max
hwTφ(xi,yi,h) (19.118)
If we set E(w)=−1
2C||w||2
2, we get the same objective as is optimized in latent SVMs (Yu
and Joachims 2009).
Unfortunately, this objective is no longer convex. However, it is a difference of convex
functions, and hence can be solved efficiently using the CCCPorconcave-convex procedure

ilog⎡
⎣∑
R
i=1max
y,h{
−N∑
i=1max
2C||w||2
(Yuille and Rangarajan 2003). This is a method for minimizing functions of the form f(w)−
g(w),w h e r efandgare convex. The method alternates between ﬁnding a linear upper bound
uon−g, and then minimizing the convex function f(w)+u(w); see Algorithm 6 for the
pseudocode. CCCP is guaranteed to decrease the objective at every iteration, and to converge to
a local minimum or a saddle point.
Algorithm 19.5: Concave-Convex Procedure (CCCP)
1Sett=0and initialize w0;
2repeat

2repeat
3Find hyperplane vtsuch that−g(w)≤−g(wt)+(w−wt)Tvtfor allw;
4Solvewt+1=a r g m i nwf(w)+wTvt;
5Sett=t+1
6untilconverged ;
When applied to latent SSVMs, CCCP is very similar to (hard) EM. In the “E step”, we compute
12. For example, consider the problem of learning to classify a set of documents as relevant or not to a query. That
is, givenndocuments x1,...,x nfor a single query q, we want to produce a labeling yj∈{ −1,+1}, representing

2repeat
5Sett=t+1
whether document jis relevant to qor not. Suppose our goal is to maximize the precision at k, which is a metric widely
used in ranking (see Section 9.7.4). We will introduce a latent variable for each document hjrepresenting its degree
of relevance. This corresponds to a latent total ordering, that has to be consistent with the observed partial ordering
y. Given this, we can deﬁne the following loss function: L(y,ˆy,ˆh)=m i n {1,n(y)
k}−1
k∑k
j=1I(yhj=1 ),where

k}−1
k∑k
n(y)is the total number of relevant documents. This loss is essentially just 1 minus the precision@k, except we replace
1 withn(y)/kso that the loss will have a minimum of zero. See (Yu and Joachims 2009) for details.

19.7. StructuralSVMs 703
the linear upper bound by setting vt=−C∑N
i=1φ(xi,yi,h∗
i),w h e r e
hi=a r g m a x
hwT
tφ(xi,yi,h) (19.119)
In the “M step”, we estimate wusing techniques for solving fully visible SSVMs. Speciﬁcally, we
minimize
1
i=1max
y,h{
L(yi,y,h)+wTφ(xi,y,h)}
−CN∑
i=1wTφ(xi,yi,h∗
i)(19.120)
Exercises
Exercise 19.1 Derivative of the log partition function
Derive Equation 19.40.
Exercise 19.2 CI properties of Gaussian graphical models
In this question, we study the relationship between sparse matrices and sparse graphs for Gaussian
graphical models. Consider a multivariate Gaussian N(x|μ,Σ)in 3 dimensions. Suppose μ=( 0,0,0)T
throughout.
Recall that for jointly Gaussian random variables, we know that XiandXjare independent iff they are
uncorrelated, ie. Σij=0. (This is not true in general, or even if XiandXjare Gaussian but not jointly
Gaussian.) Also, Xiis conditionally independent of Xjgiven all the other variables iff Σ−1
ij=0.
a. Suppose
Σ=⎛
⎝0.75 0.50 .25

ij=0.
a. Suppose
Σ=⎛
0.51.00.5
0.25 0.50 .75⎞⎠
Are there any marginal independencies amongst X
1,X2andX3? What about conditional indepen-
dencies? Hint: compute Σ−1and expand out xTΣ−1x: which pairwise terms xixjare missing? Draw
an undirected graphical model that captures as many of these independence statements (marginal and
conditional) as possible, but does not make any false independence assertions.
b. Suppose
Σ=⎛
⎝210
121012⎞
⎠
1,X2andX3? Are there any conditional inde-

0.51.00.5
b. Suppose
Σ=⎛
⎝210
121012⎞
⎠
pendencies amongst X1,X2andX3? Draw an undirected graphical model that captures as many of
these independence statements (marginal and conditional) as possible, but does not make any false
independence assertions.
c. Now suppose the distribution on Xcan be represented by the following DAG:
X1→X2→X3
Let the CPDs be as follows:
P(X1)=N(X1;0,1),P(X2|x1)=N(X2;x1,1),P(X3|x2)=N(X3;x2,1) (19.121)
Multiply these 3 CPDs together and complete the square (Bishop p101) to ﬁnd the corresponding joint

X1→X2→X3
distribution N(X1:3|μ,Σ). (You may ﬁnd it easier to solve for Σ−1rather than Σ.)

704 Chapter19. Undirectedgraphicalmodels(Markovrandomﬁelds)
d. For the DAG model in the previous question: Are there any marginal independencies amongst X1,X2
andX3? What about conditional independencies? Draw an undirected graphical model that captures
as many of these independence statements as possible, but does not make any false independence
assertions (either marginal or conditional).
Exercise 19.3 Independencies in Gaussian graphical models
(Source: MacKay.)
a. Consider the DAG X1←X2→X3. Assume that all the CPDs are linear-Gaussian. Which of the
following matrices couldbe the covariance matrix?
A=⎛
⎝931
393139⎞
⎠,B=⎛⎝8−31
−39−3
1−38⎞⎠,C=⎛⎝930
393039⎞
⎠,D=⎛⎝9−30
−31 0−3
0−39⎞⎠(19.122)
b. Which of the above matrices could be inverse covariance matrix?
c. Consider the DAG X1→X2←X3. Assume that all the CPDs are linear-Gaussian. Which of the
above matrices could be the covariance matrix?
d. Which of the above matrices could be the inverse covariance matrix?
e. Let three variables x

A=⎛
⎝931
393139⎞
−39−3
393039⎞
−31 0−3
1,x2,x4have covariance matrix Σ(1:3)and precision matrix Ω(1:3)=Σ−1
(1:3)as
follows
Σ(1:3)=⎛⎝10.50
0.510 .5
00.51⎞⎠,Ω
(1:3)=⎛⎝1.5−10.5
−12−1
0.5−11.5⎞
⎠ (19.123)
Now focus on x
1andx2. Which of the following statements about their covariance matrix Σ(1:2)and
precision matrix Ω(1:2)are true?
A:Σ(1:2)=(10.5
0.51)
,B:Ω(1:2)=(1.5−1
−12)
(19.124)
Exercise 19.4 Cost of training MRFs and CRFs
(Source: Koller.) Consider the process of gradient-ascent training for a log-linear model with kfeatures,

(1:3)as
follows
0.510 .5
00.51⎞⎠,Ω
−12−1
0.5−11.5⎞
⎠ (19.123)
0.51)
−12)
(19.124)
given a data set with Ntraining instances. Assume for simplicity that the cost of computing a single
feature over a single instance in our data set is constant, as is the cost of computing the expected value
of each feature once we compute a marginal over the variables in its scope. Assume that it takes ctime
to compute all the marginals for each data case. Also, assume that we need riterations for the gradient
process to converge.
• Using this notation, what is the time required to train an MRF in big-O notation?

• Using this notation, what is the time required to train a CRF in big-O notation?
Exercise 19.5 Full conditional in an Ising model
Consider an Ising model
p(x1,...,x n|θ)=1
Z(θ)∏
<ij>exp(J ijxixj)n∏
i=1exp(hixi) (19.125)
where<i j>denotes all unique pairs (i.e., all edges), Jij∈Ris the coupling strength (weight) on edge
i−j,hi∈Ris the local evidence (bias term), and θ=(J,h)are all the parameters.

19.7. StructuralSVMs 705
Ifxi∈{0,1}, derive an expression for the full conditional
p(xi=1|x−i,θ)=p(xi=1|xnbi,θ) (19.126)
wherex−iare all nodes except i, andnbiare the neighbors of iin the graph. Hint: you answer should
use the sigmoid/ logistic function σ(z)=1/(1 +e−z). Now suppose xi∈{ −1,+1}. Derive a related
expression for p(xi|x−i,θ)in this case. (This result can be used when applying Gibbs sampling to the
model.)

20 Exact inference for graphical models
20.1 Introduction
In Section 17.4.3, we discussed the forwards-backwards algorithm, which can exactly compute the
posterior marginals p(xt|v,θ)in any chain-structured graphical model, where xare the hidden
variables (assumed discrete) and vare the visible variables. This algorithm can be modiﬁed
to compute the posterior mode and posterior samples. A similar algorithm for linear-Gaussianchains, known as the Kalman smoother, was discussed in Section 18.3.2. Our goal in this chapteris to generalize these exact inference algorithms to arbitrary graphs. The resulting methods applyto both directed and undirected graphical models. We will describe a variety of algorithms, butwe omit their derivations for brevity. See e.g., (Darwiche 2009; Koller and Friedman 2009) for adetailed exposition of exact inference techniques for discrete directed graphical models.
20.2 Belief propagation for trees

In this section, we generalize the forwards-backwards algorithm from chains to trees. Theresulting algorithm is known as belief propagation (BP) (Pearl 1988), or the sum-product
20.2.1 Serial protocol
We initially assume (for notational simplicity) that the model is a pairwise MRF (or CRF), i.e.,
p(x|v)=1
Z(v)∏
s∈Vψs(xs)∏
(s,t)∈Eψs,t(xs,xt) (20.1)
whereψsis the local evidence for node s, andψstis the potential for edge s−t. We will
consider the case of models with higher order cliques (such as directed trees) later on.
One way to implement BP for undirected trees is as follows. Pick an arbitrary node and call it
the root,r. Now orient all edges away from r(intuitively, we can imagine “picking up the graph”
at noderand letting all the edges “dangle” down). This gives us a well-deﬁned notion of parent
and child. Now we send messages up from the leaves to the root (the collect evidence phase)

and then back down from the root (the distribute evidence phase), in a manner analogous to
forwards-backwards on chains.

708 Chapter20. Exactinferenceforgraphicalmodels
s1s2s
u1u2utroot
v−
st
(a)s1s2s
u1u2utrootv+
st
(b)
Figure 20.1 Message passing on a tree. (a) Collect-to-root phase. (b) Distribute-from-root phase.
To explain the process in more detail, consider the example in Figure 20.1. Suppose we want
to compute the belief state at node t. We will initially condition the belief only on evidence that
is at or below tin the graph, i.e., we want to compute bel−
t(xt)≜p(xt|v−
t). We will call this a
“bottom-up belief state”. Suppose, by induction, that we have computed “messages” from t’s two
children, summarizing what they think tshould know about the evidence in their subtrees, i.e.,
we have computed m−
s→t(xt)=p(xt|v−
st),w h e r ev−
stis all the evidence on the downstream
side of the s−tedge (see Figure 20.1(a)), and similarly we have computed mu→t(xt). Then we
can compute the bottom-up belief state at tas follows:
bel−
t)=1
Ztψt(xt)∏
c∈ch(t)m−
c→t(xt) (20.2)

bel−
t)=1
Ztψt(xt)∏
c∈ch(t)m−
whereψt(xt)∝p(xt|vt)is the local evidence for node t, andZtis the local normalization
constant. In words, we multiply all the incoming messages from our children, as well as the
incoming message from our local evidence, and then normalize.
We have explained how to compute the bottom-up belief states from the bottom-up messages.
How do we compute the messages themselves? Consider computing m−
s→t(xt),w h e r esis one
oft’s children. Assume, by recursion, that we have computed bel−
s(xs)=p(xs|v−
s). Then we

s). Then we
can compute the message as follows:
m−
s→t(xt)=∑
xsψst(xs,xt)bel−
s(xs) (20.3)
Essentially we convert beliefs about xsinto beliefs about xtby using the edge potential ψst.
We continue in this way up the tree until we reach the root. Once at the root, we have “seen”
all the evidence in the tree, so we can compute our local belief state at the root using
belr(xr)≜p(xr|v)=p(xt|v−
r)∝ψr(xr)∏
c∈ch(r)m−
c→r(xr) (20.4)
This completes the end of the upwards pass, which is analogous to the forwards pass in an

m−
s→t(xt)=∑
c∈ch(r)m−
HMM. As a “side effect”, we can compute the probability of the evidence by collecting the

20.2. Beliefpropagationfortrees 709
normalization constants:
p(v)=∏
tZt (20.5)
We can now pass messages down from the root. For example, consider node s, with parent t,
as shown in Figure 20.1(b). To compute the belief state for s, we need to combine the bottom-up
belief for stogether with a top-down message from t, which summarizes all the information in
the rest of the graph, m+
t→s(xs)≜p(xt|v+
st),w h e r ev+
stis all the evidence on the upstream
(root) side of the s−tedge, as shown in Figure 20.1(b). We then have
bels(xs)≜p(xs|v)∝bel−
s(xs)∏
t∈pa(s)m+
t→s(xt) (20.6)
How do we compute these downward messages? For example, consider the message from t
tos. Suppose t’s parent is r, andt’s children are sandu, as shown in Figure 20.1(b). We want
to include in m+t→sall the information that thas received, except for the information that s
sent it:
m+t→s(xs)≜p(xs|v+
st)=∑
xtψst(xs,xt)belt(xt)
m−s→t(xt)(20.7)
Rather than dividing out the message sent up to t, we can plug in the equation of beltto get
m+t→s(xs)=∑

s(xs)∏
t∈pa(s)m+
sent it:
st)=∑
xtψst(xs,xt)ψt(xt)∏
c∈ch(t),c̸=sm−c→t(xt)∏
p∈pa(t)m+p→t(xt) (20.8)
In other words, we multiply together all the messages coming into tfrom all nodes except for
the recipient s, combine together, and then pass through the edge potential ψst. In the case of
a chain,tonly has one child sand one parent p, so the above simpliﬁes to
xtψst(xs,xt)ψt(xt)m+p→t(xt) (20.9)
The version of BP in which we use division is called belief updating, and the version in

which we multiply all-but-one of the messages is called sum-product. The belief updating
version is analogous to how we formulated the Kalman smoother in Section 18.3.2: the top-

down messages depend on the bottom-up messages. This means they can be interpreted asconditional posterior probabilities. The sum-product version is analogous to how we formulatedthe backwards algorithm in Section 17.4.3: the top-down messages are completely independentof the bottom-up messages, which means they can only be interpreted as conditional likelihoods.See Section 18.3.2.3 for a more detailed discussion of this subtle difference.
20.2.2 Parallel protocol

So far, we have presented a serial version of the algorithm, in which we send messages upto the root and back. This is the optimal approach for a tree, and is a natural extension offorwards-backwards on chains. However, as a prelude to handling general graphs with loops, wenow consider a parallel version of BP. This gives equivalent results to the serial version but isless efficient when implemented on a serial machine.

710 Chapter20. Exactinferenceforgraphicalmodels
The basic idea is that all nodes receive messages from their neighbors in parallel, they then
updates their belief states, and ﬁnally they send new messages back out to their neighbors.
This process repeats until convergence. This kind of computing architecture is called a systolic
array, due to its resemblance to a beating heart.
More precisely, we initialize all messages to the all 1’s vector. Then, in parallel, each node
absorbs messages from all its neighbors using
bels(xs)∝ψs(xs)∏
t∈nbrsmt→s(xs) (20.10)
Then, in parallel, each node sends messages to each of its neighbors:
ms→t(xt)=∑
xs⎛
⎝ψs(xs)ψst(xs,xt)∏
u∈nbrs\tmu→s(xs)⎞⎠ (20.11)
Them
s→tmessage is computed by multiplying together all incoming messages, except the one
sent by the recipient, and then passing through the ψstpotential.
At iteration Tof the algorithm, bels(xs)represents the posterior belief of xsconditioned on

xs⎛
Them
the evidence that is Tsteps away in the graph. After D(G)steps, where D(G)is thediameter
of the graph (the largest distance between any two pairs of nodes), every node has obtained
information from all the other nodes. Its local belief state is then the correct posterior marginal.Since the diameter of a tree is at most |V|−1, the algorithm converges in a linear number of
steps.
We can actually derive the up-down version of the algorithm by imposing the condition that

steps.
a node can only send a message once it has received messages from all its other neighbors.This means we must start with the leaf nodes, which only have one neighbor. The messagesthen propagate up to the root and back. We can also update the nodes in a random order.The only requirement is that each node get updated D(G)times. This is just enough time for
information to spread throughout the whole tree.
Similar parallel, distributed algorithms for solving linear systems of equations are discussed

in (Bertsekas 1997). In particular, the Gauss-Seidel algorithm is analogous to the serial up-downversion of BP, and the Jacobi algorithm is analogous to the parallel version of BP.
20.2.3 Gaussian BP *
Now consider the case where p(x|v)is jointly Gaussian, so it can be represented as a Gaussian

pairwise MRF, as in Section 19.4.4. We now present the belief propagation algorithm for thisclass of models, follow the presentation of (Bickson 2009) (see also (Malioutov et al. 2006)). Wewill assume the following node and edge potentials:
ψ
t(xt)=e x p (−1
2Attx2
t+btxt) (20.12)
ψst(xs,xt)=e x p (−1
2xsAstxt) (20.13)
so the overall model has the form
p(x|v)∝exp(−1
2xTAx+bTx) (20.14)

20.2. Beliefpropagationfortrees 711
This is the information form of the MVN (see Exercise 9.2), where Ais the precision matrix.
Note that by completing the square, the local evidence can be rewritten as a Gaussian:
ψt(xt)∝N(bt/Att,A−1
tt)≜N(mt,ℓ−1
t) (20.15)
Below we describe how to use BP to compute the posterior node marginals,
p(xt|v)=N(μt,λ−1
t) (20.16)
If the graph is a tree, the method is exact. If the graph is loopy, the posterior means may still
be exact, but the posterior variances are often too small (Weiss and Freeman 1999).
Although the precision matrix Ais often sparse, computing the posterior mean requires
inverting it, since μ=A−1b. BP provides a way to exploit graph structure to perform this
computation in O(D)time instead of O(D3). This is related to various methods from linear
algebra, as discussed in (Bickson 2009).
Since the model is jointly Gaussian, all marginals and all messages will be Gaussian. The

key operations we need are to multiply together two Gaussian factors, and to marginalize out avariable from a joint Gaussian factor.
For multiplication, we can use the fact that the product of two Gaussians is Gaussian:
N(x|μ
1,λ−1
1)×N(x|μ2,λ−1
2)=CN(x|μ,λ−1) (20.17)
λ=λ1+λ2 (20.18)
μ=λ−1(μ1λ1+μ2λ2) (20.19)
where
C=√
λ
λ1λ2exp(1
2(λ1μ2
1(λ−1λ1−1)+λ2μ22(λ−1λ2−1)+2λ−1λ1λ2μ1μ2))
(20.20)
See Exercise 20.2 for the proof.
For marginalization, we have the following result:
∫
exp(−ax2+bx)dx=√
π/aexp(b2/4a) (20.21)

N(x|μ
1,λ−1
where
C=√
λ
λ1λ2exp(1
2(λ1μ2
(20.20)
∫
which follows from the normalization constant of a Gaussian (Exercise 2.11).
We now have all the pieces we need. In particular, let the message ms→t(xt)be a Gaussian
with mean μstand precision λst. From Equation 20.10, the belief at node sis given by the
product of incoming messages times the local evidence (Equation 20.15) and hence
bels(xs)=ψ s(xs)∏
t∈nbr(s)mts(xs)=N(xs|μs,λ−1
s) (20.22)
λs=ℓs+∑
t∈nbr(s)λts (20.23)
μs=λ−1
s⎛
⎝ℓsms+∑
t∈nbr(s)λtsμts⎞⎠ (20.24)

712 Chapter20. Exactinferenceforgraphicalmodels
To compute the messages themselves, we use Equation 20.11, which is given by
ms→t(xt)=∫
xs⎛
⎝ψst(xs,xt)ψs(xs)∏
u∈nbrs\tmu→s(xs)⎞⎠dx
s (20.25)
=∫
xsψst(xs,xt)fs\t(xs)dxs (20.26)
wherefs\t(xs)is the product of the local evidence and all incoming messages excluding the
message from t:
fs\t(xs)≜ψs(xs)∏
u∈nbrs\tmu→s(xs) (20.27)
=N(xs|μs\t,λ−1
s\t) (20.28)
λs\t≜ℓs+∑
u∈nbr(s)\tλus (20.29)
μs\t≜λ−1
s\t⎛⎝ℓ
sms+∑
u∈nbr(s)\tλusμus⎞⎠ (20.30)
Returning to Equation 20.26 we have
m
s→t(xt)=∫
xsexp(−xsAstxt)  
ψst(xs,xt)exp(−λs\t/2(xs−μs\t)2)  
fs\t(xs)dxs (20.31)
=∫
xsexp(
(−λs\tx2
s/2)+(λs\tμs\t−Astxt)xs)
dxs+const (20.32)
∝exp(
(λs\tμs\t−Astxt)2/(2λs\t))
(20.33)
∝N(μst,λ−1
st) (20.34)
λst=A2st/λs\t (20.35)
μst=Astμs\t/λst (20.36)
One can generalize these equations to the case where each node is a vector, and the messages
become small MVNs instead of scalar Gaussians (Alag and Agogino 1996). If we apply the

m
s→t(xt)=∫
=∫
xsexp(
(−λs\tx2
∝exp(
(20.33)
resulting algorithm to a linear dynamical system, we recover the Kalman smoothing algorithmof Section 18.3.2.
To perform message passing in models with non-Gaussian potentials, one can use sampling
methods to approximate the relevant integrals. This is called non-parametric BP (Sudderth
et al. 2003; Isard 2003; Sudderth et al. 2010).
20.2.4 Other BP variants *
In this section, we brieﬂy discuss several variants of the main algorithm.

713
xt xt+1
vt vt+1ψt,t+1
ψt ψt+1ft βt+1
Figure 20.2 Illustration of how to compute the two-slice distribution for an HMM. The ψtandψt+1
terms are the local evidence messages from the visible nodes vt,vt+1to the hidde nodes xt,xt+1
respectively; ftis the forwards message from xt−1andβt+1is the backwards message from xt+2.
20.2.4.1 Max-product algorithm
It is possible to devise a max-product version of the BP algorithm, by replacing the∑operator
with themaxoperator. We can then compute the local MAP marginal of each node. However,
if there are ties, this might not be globally consistent, as discussed in Section 17.4.4. Fortunately,
we can generalize the Viterbi algorithm to trees, where we use max and argmax in the collect-to-root phase, and perform traceback in the distribute-from-root phase. See (Dawid 1992) fordetails.
20.2.4.2 Sampling from a tree

It is possible to draw samples from a tree structured model by generalizing the forwards ﬁltering/ backwards sampling algorithm discussed in Section 17.4.5. See (Dawid 1992) for details.
20.2.4.3 Computing posteriors on sets of variables
In Section 17.4.3.2, we explained how to compute the “two-slice” distribution ξ
t,t+1(i,j)=
p(xt=i,xt+1=j|v)in an HMM, namely by using
ξt,t+1(i,j)=α t(i)ψt+1(j)βt+1(j)ψt,t+1(i,j) (20.37)
Sinceαt(i)∝ψt(i)ft(i),w h e r eft=p(xt|v1:t−1)is the forwards message, we can think of

this as sending messages ftandψtintoxt,βt+1andφt+1intoxt+1, and then combining
them with the Ψmatrix, as shown in Figure 20.2. This is like treating xtandxt+1as a single
“mega node”, and then multiplying all the incoming messages as well as all the local factors(here,ψ
t,t+1).20.2. Beliefpropagationfortrees

714 Chapter20. Exactinferenceforgraphicalmodels
Coherence
Diﬃculty
GradeIntelligence
SAT
Letter
HappyJob
(a)Coherence
Diﬃculty
SAT
Letter
HappyJob
(b)
Figure 20.3 Left: The “student” DGM. Right: the equivalent UGM. We add moralization arcs D-I, G-J and
L-S. Based on Figure 9.8 of (Koller and Friedman 2009).
20.3 The variable elimination algorithm
We have seen how to use BP to compute exact marginals on chains and trees. In this section,
we discuss an algorithm to compute p(xq|xv)for any kind of graph.
We will explain the algorithm by example. Consider the DGM in Figure 20.3(a). This model,
from (Koller and Friedman 2009), is a hypothetical model relating various variables pertaining toa typical student. The corresponding joint has the following form:
P(C,D,I,G,S,L,J,H ) (20.38)
=P(C)P(D|C)P(I)P(G|I,D)P(S|I)P(L|G)P(J|L,S)P(H|G,J) (20.39)

Note that the forms of the CPDs do not matter, since all our calculations will be symbolic.However, for illustration purposes, we will assume all variables are binary.
Before proceeding, we convert our model to undirected form. This is not required, but it

makes for a more uniﬁed presentation, since the resulting method can then be applied to bothDGMs and UGMs (and, as we will see in Section 20.3.1, to a variety of other problems thathave nothing to do with graphical models). Since the computational complexity of inference inDGMs and UGMs is, generally speaking, the same, nothing is lost in this transformation from acomputational point of view.
1
To convert the DGM to a UGM, we simply deﬁne a potential or factor for every CPD, yielding

1
p(C,D,I,G,S,L,J,H ) (20.40)
=ψC(C)ψD(D,C)ψI(I)ψG(G,I,D)ψS(S,I)ψL(L,G)ψJ(J,L,S)ψH(H,G,J)(20.41)
1. There are a few “tricks” one can exploit in the directed case that cannot easily be exploited in the undirected case.
One important example is barren node removal. To explain this, consider a naive Bayes classiﬁer, as in Figure 10.2.
Suppose we want to infer yand we observe x1andx2, but not x3andx4. It is clear that we can safely remove
x3andx4, since∑

x3p(x3|y)=1, and similarly for x4. In general, once we have removed hidden leaves, we can
apply this process recursively. Since potential functions do not necessary sum to one, we cannot use this trick in theundirected case. See (Koller and Friedman 2009) for a variety of other speedup tricks.

20.3. Thevariableeliminationalgorithm 715
Since all the potentials are locally normalized, since they are CPDs, there is no need for a
global normalization constant, so Z=1. The corresponding undirected graph is shown in
Figure 20.3(b). Note that it has more edges than the DAG. In particular, any “unmarried” nodes
that share a child must get “married”, by adding an edge between them; this process is knownasmoralization. Only then can the arrows be dropped. In this example, we added D-I, G-J, and
L-S moralization arcs. The reason this operation is required is to ensure that the CI propertiesof the UGM match those of the DGM, as explained in Section 19.2.2. It also ensures there is aclique that can “store” the CPDs of each family.
Now suppose we want to compute p(J=1 ), the marginal probability that a person will get a
job. Since we have 8 binary variables, we could simply enumerate over all possible assignmentsto all the variables (except for J), adding up the probability of each joint instantiation:
p(J)=∑
L∑

p(J)=∑
L∑
S∑
G∑
H∑
I∑
D∑
Cp(C,D,I,G,S,L,J,H ) (20.42)
However, this would take O(27)time. We can be smarter by pushing sums inside products.
This is the key idea behind the variable elimination algorithm (Zhang and Poole 1996), also
calledbucket elimination (Dechter 1996), or, in the context of genetic pedigree trees, the
peeling algorithm (Cannings et al. 1978). In our example, we get
p(J)=∑
L,S,G,H,I,D,Cp(C,D,I,G,S,L,J,H )
=∑
L,S,G,H,I,D,CψC(C)ψD(D,C)ψI(I)ψG(G,I,D)ψS(S,I)ψL(L,G)
×ψJ(J,L,S)ψH(H,G,J)
=∑
L,SψJ(J,L,S)∑
GψL(L,G)∑
HψH(H,G,J)∑
IψS(S,I)ψI(I)
×∑
DψG(G,I,D)∑
CψC(C)ψD(D,C)
We now evaluate this expression, working right to left as shown in Table 20.1. First we multiplytogether all the terms in the scope of the∑

p(J)=∑
=∑
=∑
GψL(L,G)∑
×∑
Coperator to create the temporary factor
τ′
1(C,D)=ψC(C)ψD(D,C) (20.43)
Then we marginalize out Cto get the new factor
τ1(D)=∑
Cτ′
1(C,D) (20.44)
Next we multiply together all the terms in the scope of the∑
Doperator and then marginalize
out to create
τ′
2(G,I,D)=ψ G(G,I,D)τ1(D) (20.45)
τ2(G,I)=∑
Dτ′
2(G,I,D) (20.46)

716 Chapter20. Exactinferenceforgraphicalmodels
∑
L∑
SψJ(J,L,S)∑
GψL(L,G)∑
HψH(H,G,J )∑
IψS(S,I)ψI(I)∑
  
τ1(D)
∑
L∑
GψL(L,G)∑
DψG(G,I,D)τ1(D)
  
τ2(G,I)
∑
L∑
GψL(L,G)∑
IψS(S,I)ψI(I)τ2(G,I)
  
τ3(G,S)
∑
L∑
GψL(L,G)∑
HψH(H,G,J )
  
τ4(G,J)τ3(G,S)
∑
L∑
GψL(L,G)τ 4(G,J)τ3(G,S)
  
τ5(J,L,S)
∑
L∑
SψJ(J,L,S)τ5(J,L,S)
  
τ6(J,L)
∑
Lτ6(J,L)
 
τ7(J)
Table 20.1 Eliminating variables from Figure 20.3 in the order C,D,I,H,G,S,L to compute P(J).
Ioperator and then marginalize
τ′
3(G,I,S)=ψ S(S,I)ψI(I)τ2(G,I) (20.47)
τ3(G,S)=∑
Iτ′
3(G,I,S) (20.48)
And so on.
The above technique can be used to compute any marginal of interest, such as p(J)o r
p(J,H). To compute a conditional, we can take a ratio of two marginals, where the visible

  
τ6(J,L)
∑
Lτ6(J,L)
 
τ7(J)
τ′
τ3(G,S)=∑
Iτ′
And so on.
variables have been clamped to their known values (and hence don’t need to be summed over).
For example,
p(J=j|I=1,H=0 )=p(J=j,I=1,H=0 )∑
j′p(J=j′,I=1,H=0 )(20.49)
In general, we can write
p(xq|xv)=p(xq,xv)
p(xv)=∑
xhp(xh,xq,xv)∑
xh∑
x′qp(xh,x′q,xv)(20.50)

20.3. Thevariableeliminationalgorithm 717
The normalization constant in the denominator, p(xv), is called the probability of the evi-
dence.
SeevariableElimination for a simple Matlab implementation of this algorithm, which
works for arbitrary graphs, and arbitrary discrete factors. But before you go too crazy, please
read Section 20.3.2, which points out that VE can be exponentially slow in the worst case.
20.3.1 The generalized distributive law *
Abstractly, VE can be thought of as computing the following expression:
p(xq|xv)∝∑
x∏
cψc(xc) (20.51)
It is understood that the visible variables xvare clamped, and not summed over. VE uses
non-serial dynamic programming (Bertele and Brioschi 1972), caching intermediate results to
avoid redundant computation.
However, there are other tasks we might like to solve for any given graphical model. For
example, we might want the MAP estimate:
x∗=a r g m a x
x∏
cψc(xc) (20.52)

x∏
x∗=a r g m a x
x∏
Fortunately, essentially the same algorithm can also be used to solve this task: we just replacesum with max. (We also need a traceback step, which actually recovers the argmax, as opposed
to just the value of max; these details are explained in Section 17.4.4.)
In general, VE can be applied to any commutative semi-ring. This is a set K, together with
two binary operations called “+” and “× ”, which satisfy the following three axioms:
1. The operation “+” is associative and commutative, and there is an additive identity element

called “0” such that k+0=kfor allk∈K.
2. The operation “× ” is associative and commutative, and there is a multiplicative identity
element called “1” such that k×1=kfor allk∈K.
3. Thedistributive law holds, i.e.,
(a×b)+(a×c)=a×(b+c) (20.53)
for all triples (a,b,c)fromK.
Thisframeworkcoversanextremelywiderangeofimportantapplications, includingconstraint
satisfaction problems (Bistarelli et al. 1997; Dechter 2003), the fast Fourier transform (Aji andMcEliece 2000), etc. See Table 20.2 for some examples.
20.3.2 Computational complexity of VE

The running time of VE is clearly exponential in the size of the largest factor, since we have sumover all of the corresponding variables. Some of the factors come from the original model (andare thus unavoidable), but new factors are created in the process of summing out. For example,

718 Chapter20. Exactinferenceforgraphicalmodels
Domain + × Name
[0,∞)( + ,0) (×,1)sum-product
[0,∞)( m a x ,0) (×,1)max-product
(−∞,∞]( m i n ,∞)( + ,0)min-sum
{T,F}(∨,F)(∧,T)Boolean satisﬁability
Table 20.2 Some commutative semirings.
∑
D∑
CψD(D,C)∑
H∑
L∑
IψI(I)ψS(S,I)∑
GψG(G,I,D)ψL(L,)ψH(H,G,J )
  
τ1(I,D,L,J,H )
∑
D∑
CψD(D,C)∑
H∑
L∑
IψI(I)ψS(S,I)τ1(I,D,L,J,H )
  
τ2(D,L,S,J,H )
∑
D∑
CψD(D,C)∑
H∑
L∑
SψJ(J,L,S)τ2(D,L,S,J,H )
  
τ3(D,L,J,H )
∑
D∑
CψD(D,C)∑
H∑
Lτ3(D,L,J,H )
  
τ4(D,J,H )
∑
D∑
CψD(D,C)∑
Hτ4(D,J,H )
  
τ5(D,J)
∑
D∑
CψD(D,C)τ5(D,J)
  
τ6(D,J)
∑
Dτ6(D,J)
  
τ7(J)
Table 20.3 Eliminating variables from Figure 20.3 in the order G,I,S,L,H,C,D .
in Equation 20.47, we created a factor involving G, I and S; but these nodes were not originally
present together in any factor.
The order in which we perform the summation is known as the elimination order. This

∑
D∑
CψD(D,C)∑
H∑
L∑
  
∑
D∑
CψD(D,C)∑
H∑
  
τ4(D,J,H )
∑
D∑
CψD(D,C)∑
  
τ5(D,J)
∑
D∑
  
τ6(D,J)
∑
Dτ6(D,J)
  
τ7(J)
can have a large impact on the size of the intermediate factors that are created. For example,consider the ordering in Table 20.1: the largest created factor (beyond the original ones in themodel) has size 3, corresponding to τ
5(J,L,S). Now consider the ordering in Table 20.3: now
the largest factors are τ1(I,D,L,J,H )andτ2(D,L,S,J,H ), which are much bigger.
We can determine the size of the largest factor graphically, without worrying about the actual

numerical values of the factors. When we eliminate a variable Xt, we connect it to all variables

20.3. Thevariableeliminationalgorithm 719
Coherence
Diﬃculty
SAT
Letter
HappyJob
Diﬃculty
SAT
Letter
HappyJob
(b)Coherence
Diﬃculty
SAT
Letter
HappyJob
(c)
Figure 20.4 Example of the elimination process, in the order C,D,I, etc. When we eliminate I(ﬁgure
c), we add a ﬁll-in edge between GandS, since they are not connected. Based on Figure 9.10 of (Koller
and Friedman 2009).
that share a factor with Xt(to reﬂect the new temporary factor τ′
t). The edges created by this
process are called ﬁll-in edges. For example, Figure 20.4 shows the ﬁll-in edges introduced
when we eliminate in the order C,D,I,... . The ﬁrst two steps do not introduce any ﬁll-ins,
but when we eliminate I, we connect GandS, since they co-occur in Equation 20.48.
LetG(≺)be the (undirected) graph induced by applying variable elimination to Gusing
elimination ordering ≺. The temporary factors generated by VE correspond to maximal cliques

in the graph G(≺). For example, with ordering (C,D,I,H,G,S,L ), the maximal cliques are
{C,D},{D,I,G},{G,L,S,J},{G,J,H},{G,I,S} (20.54)
It is clear that the time complexity of VE is
∏
c∈C(G(≺))K|c|(20.55)
whereCare the cliques that are created, |c|is the size of the clique c, and we assume for
notational simplicity that all the variables have Kstates each.
Let us deﬁne the induced width of a graph given elimination ordering ≺, denoted w(≺),a s

∏
the size of the largest factor (i.e., the largest clique in the induced graph ) minus 1. Then it is
easy to see that the complexity of VE with ordering ≺isO(Kw(≺)+1).
Obviously we would like to minimize the running time, and hence the induced width. Let us
deﬁne the treewidth of a graph as the minimal induced width.
w≜min
≺max
c∈G(≺)|c|−1 (20.56)
Then clearly the best possible running time for VE is O(DKw+1). Unfortunately, one can show
that for arbitrary graphs, ﬁnding an elimination ordering ≺that minimizes w(≺)is NP-hard

w≜min
≺max
(Arnborg et al. 1987). In practice greedy search techniques are used to ﬁnd reasonable orderings(Kjaerulff 1990), although people have tried other heuristic methods for discrete optimization,

720 Chapter20. Exactinferenceforgraphicalmodels
such as genetic algorithms (Larranaga et al. 1997). It is also possible to derive approximate
algorithms with provable performance guarantees (Amir 2010).
In some cases, the optimal elimination ordering is clear. For example, for chains, we should
work forwards or backwards in time. For trees, we should work from the leaves to the root.These orderings do not introduce any ﬁll-in edges, so w=1. Consequently, inference in chains
and trees takes O(VK
2)time. This is one reason why Markov chains and Markov trees are so
widely used.
Unfortunately, for other graphs, the treewidth is large. For example, for an m×n2d lattice,
the treewidth is O(min{m,n}) (Lipton and Tarjan 1979). So VE on a 100×100Ising model
would take O(2100)time.
Of course, just because VE is slow doesn’t mean that there isn’t some smarter algorithm out
there. We discuss this issue in Section 20.5.
20.3.3 A weakness of VE

The main disadvantage of the variable elimination algorithm (apart from its exponential depen-dence on treewidth) is that it is inefficient if we want to compute multiple queries conditionedon the same evidence. For example, consider computing all the marginals in a chain-structuredgraphical model such as an HMM. We can easily compute the ﬁnal marginal p(x
T|v)by elimi-
nating all the nodes x1toxT−1in order. This is equivalent to the forwards algorithm, and takes
O(K2T)time. But now suppose we want to compute p(xT−1|v). We have to run VE again, at

a cost of O(K2T)time. So the total cost to compute all the marginals is O(K2T2). However,
we know that we can solve this problem in O(K2T)using forwards-backwards. The difference
is that FB caches the messages computed on the forwards pass, so it can reuse them later.
The same argument holds for BP on trees. For example, consider the 4-node tree in Fig-
ure 20.5. We can compute p(x1|v)by eliminating x2:4; this is equivalent to sending messages
up tox1(the messages correspond to the τfactors created by VE). Similarly we can compute
p(x2|v),p(x3|v)and then p(x4|v). We see that some of the messages used to compute the
marginal on one node can be re-used to compute the marginals on the other nodes. By storingthe messages for later re-use, we can compute all the marginals in O(DK
2)time. This is what

the up-down (collect-distribute) algorithm on trees does.
The question is: how can we combine the efficiency of BP on trees with the generality of VE?
The answer is given in Section 20.4.
20.4 The junction tree algorithm *
Thejunction tree algorithm orJTAgeneralizes BP from trees to arbitrary graphs. We sketch
the basic idea below; for details, see e.g., (Koller and Friedman 2009).
20.4.1 Creating a junction tree

The basic idea behind the JTA is this. We ﬁrst run the VE algorithm “symbolically”, adding ﬁll-inedges as we go, according to a given elimination ordering. The resulting graph will be a chordal
graph, which means that every undirected cycle X
1−X2···Xk−X1of length k≥4has a

722 Chapter20. Exactinferenceforgraphicalmodels
chord, i.e., an edge connects Xi,Xjfor all non-adjacent nodes i,jin the cycle.2
Having created a chordal graph, we can extract its maximal cliques. In general, ﬁnding max
cliques is computationally hard, but it turns out that it can be done efficiently from this special
kind of graph. Figure 20.7(b) gives an example, where the max cliques are as follows:
{C,D},{G,I,D},{G,S,I},{G,J,S,L},{H,G,J} (20.57)
Note that if the original graphical model was already chordal, the elimination process would notadd any extra ﬁll-in edges (assuming the optimal elimination ordering was used). We call suchmodelsdecomposable, since they break into little pieces deﬁned by the cliques.
It turns out that the cliques of a chordal graph can be arranged into a special kind of
tree known as a junction tree. This enjoys the running intersection property (RIP), which

means that any subset of nodes containing a given variable forms a connected component.Figure 20.7(c) gives an example of such a tree. We see that the node Ioccurs in two adjacent
tree nodes, so they can share information about this variable. A similar situation holds for allthe other variables.
One can show that if a tree that satisﬁes the running intersection property, then applying
BP to this tree (as we explain below) will return the exact values of p(x
c|v)for each node c

in the tree (i.e., clique in the induced graph). From this, we can easily extract the node andedge marginals, p(x
t|v)andp(xs,xt|v)from the original model, by marginalizing the clique
distributions.3
20.4.2 Message passing on a junction tree
Having constructed a junction tree, we can use it for inference. The process is very similarto belief propagation on a tree. As in Section 20.2, there are two versions: the sum-productform, also known as the Shafer-Shenoy algorithm, named after (Shafer and Shenoy 1990); and

the belief updating form (which involves division), also known as the Hugin(named after a
company) or the Lauritzen-Spiegelhalter algorithm (named after (Lauritzen and Spiegelhalter
1988)). See (Lepar and Shenoy 1998) for a detailed comparison of these methods. Below wesketch how the Hugin algorithm works.
We assume the original model has the following form:
p(x)=1
Z∏
c∈C(G)ψc(xc) (20.58)
whereC(G)are the cliques of the original graph. On the other hand, the tree deﬁnes a
distribution of the following form:
p(x)=∏
c∈C(T)ψc(xc)
∏
s∈S(T)ψs(xs)(20.59)

p(x)=1
Z∏
p(x)=∏
∏
2. The largest loop in a chordal graph is length 3. Consequently chordal graphs are sometimes called triangulated.
However, it is not enough for the graph to look like it is made of little triangles. For example, Figure 20.6(a) is not
chordal, even though it is made of little triangles, since it contains the chordless 5-cycle 1-2-3-4-5-1.
3. If we want the joint distribution of some variables that are not in the same clique — a so-called out-of-clique

p(x)=∏
∏
query— we can adapt the technique described in Section 20.2.4.3 as follows: create a mega node containing the query
variables and any other nuisance variables that lie on the path between them, multiply in messages onto the boundary
of the mega node, and then marginalize out the internal nuisance variables. This internal marginalization may require
the use of BP or VE. See (Koller and Friedman 2009) for details.

20.4. Thejunctiontreealgorithm* 723
Grade
Letter
Job
HappyCoherence
SAT
G ,J ,S ,L H ,G ,J G, S, I G, I, D C, D
G, J G, S G, I DIntelligence Difﬁculty
Grade
Letter
Job
SATIntelligence Difﬁculty
(a) (b)
(c)
Figure 20.7 (a) The student graph with ﬁll-in edges added. (b) The maximal cliques. (c) The junction
tree. An edge between nodes sandtis labeled by the intersection of the sets on nodes sandt; this is
called the separating set. From Figure 9.11 of (Koller and Friedman 2009). Used with kind permission of
Daphne Koller.
whereC(T)are the nodes of the junction tree (which are the cliques of the chordal graph), and
S(T)are the separators of the tree. To make these equal, we initialize by deﬁning ψs=1for
all separators and ψc=1for all cliques. Then, for each clique in the original model, c∈C(G),
we ﬁnd a clique in the tree c′∈C(T)which contains it, c′⊇c. We then multiply ψcontoψc′
by computing ψc′=ψc′ψc. After doing this for all the cliques in the original graph, we have
∏
c∈C(T)ψc(xc)=∏

∏
c∈C(G)ψc(xc) (20.60)
As in Section 20.2.1, we now send messages from the leaves to the root and back, as sketched
in Figure 20.1. In the upwards pass, also known as the collect-to-root phase, node isends to
its parent jthe following message:
mi→j(Sij)=∑
Ci\Sijψi(Ci) (20.61)
That is, we marginalize out the variables that node i“knows about” which are irrelevant to j,
and then we send what is left over. Once a node has received messages from all its children, it
updates its belief state using
ψi(Ci)∝ψi(Ci)∏
j∈chimj→i(Sij) (20.62)

724 Chapter20. Exactinferenceforgraphicalmodels
At the root, ψr(Cr)represents p(xCr|v), which is the posterior over the nodes in clique
Crconditioned on all the evidence. Its normalization constant is p(v)/Z0,w h e r eZ0is the
normalization constant for the unconditional prior, p(x).( W eh a v e Z0=1if the original model
was a DGM.)
In the downwards pass, also known as the distribute-from-root phase, node isends to its
childrenjthe following message:
Ci\Sijψi(Ci)
mj→i(Sij)(20.63)
We divide out by what jsent toito avoid double counting the evidence. This requires that we
store the messages from the upwards pass. Once a node has received a top-down message from
its parent, it can compute its ﬁnal belief state using
ψj(Cj)∝ψj(Cj)mi→j(Sij) (20.64)
An equivalent way to present this algorithm is based on storing the messages inside the
separator potentials. So on the way up, sending from itojwe compute the separator potential
ψ∗
ij(Sij)=∑
Ci\Sijψi(Ci) (20.65)
and then update the recipient potential:
ψ∗

ψ∗
ij(Sij)=∑
ψ∗
j(Cj)∝ψj(Cj)ψ∗
ij(Sij)
ψij(Sij)(20.66)
(Recall that we initialize ψij(Sij)=1.) This is sometimes called passing a ﬂow fromitoj.
On the way down, from itoj, we compute the separator potential
ψ∗∗
ij(Sij)=∑
Ci\Sijψ∗
i(Ci) (20.67)
ψ∗∗
j(Cj)∝ψ∗
j(Cj)ψ∗∗
ij(Sij)
ψ∗
ij(Sij)(20.68)
This process is known as junction tree calibration. See Figure 20.1 for an illustration. Its

ψ∗
ij(Sij)=∑
ψ∗
ij(Sij)
ψ∗∗
ij(Sij)=∑
Ci\Sijψ∗
ψ∗∗
j(Cj)∝ψ∗
j(Cj)ψ∗∗
ij(Sij)
ψ∗
correctness follows from the fact that each edge partitions the evidence into two distinct groups,plus the fact that the tree satisﬁes RIP, which ensures that no information is lost by onlyperforming local computations.
20.4.2.1 Example: jtree algorithm on a chain

ψ∗∗
ij(Sij)=∑
Ci\Sijψ∗
ψ∗∗
j(Cj)∝ψ∗
j(Cj)ψ∗∗
ij(Sij)
ψ∗
It is interesting to see what happens if we apply this process to a chain structured graph such asan HMM. A detailed discussion can be found in (Smyth et al. 1997), but the basic idea is this. Thecliques are the edges, and the separators are the nodes, as shown in Figure 20.8. We initializethe potentials as follows: we set ψ
s=1for all the separators, we set ψc(xt−1,xt)=p(xt|xt−1)
for clique c=(Xt−1,Xt), and we set ψc(xt,yt)=p(yt|xt)for clique c=(Xt,Yt).

20.4. Thejunctiontreealgorithm* 725
X1X2
X1X2
X1Y1X2Y2X2X2X3
X3
X3Y3X3X3X4
X4
X4Y4
Figure 20.8 The junction tree derived from an HMM/SSM of length T=4.
Next we send messages from left to right. Consider clique (Xt−1,Xt)with potential
p(Xt|Xt−1). It receives a message from clique (Xt−2,Xt−1)via separator Xt−1of the form∑
xt−2p(Xt−2,Xt−1|v1:t−1)=p(Xt−1|v1:t−1). When combined with the clique potential,
this becomes the two-slice predictive density
p(Xt|Xt−1)p(Xt−1|v1:t−1)=p(Xt−1,Xt|v1:t−1 (20.69)
The clique (Xt−1,Xt)also receives a message from (Xt,Yt)via separator Xtof the form
p(yt|Xt), which corresponds to its local evidence. When combined with the updated clique
potential, this becomes the two-slice ﬁltered posterior
p(Xt−1,Xt|v1:t−1)p(vt|Xt)=p(Xt−1,Xt|v1:t) (20.70)
Thus the messages in the forwards pass are the ﬁltered belief states αt, and the clique potentials
are the two-slice distributions. In the backwards pass, the messages are the update factorsγt
αt,

αt,
whereγt(k)=p(xt=k|v1:T)andαt(k)=p(xt=k|v1:t). By multiplying by this message,
we “swap out” the old αtmessage and “swap in” the new γtmessage. We see that the backwards
pass involves working with posterior beliefs, not conditional likelihoods. See Section 18.3.2.3 for
further discussion of this difference.
20.4.3 Computational complexity of JTA
If all nodes are discrete with Kstates each, it is clear that the JTA takes O(|C|Kw+1)time
and space, where |C|is the number of cliques and wis the treewidth of the graph, i.e., the

size of the largest clique minus 1. Unfortunately, choosing a triangulation so as to minimize thetreewidth is NP-hard, as explained in Section 20.3.2.
The JTA can be modiﬁed to handle the case of Gaussian graphical models. The graph-theoretic
steps remain unchanged. Only the message computation differs. We just need to deﬁne howto multiply, divide, and marginalize Gaussian potential functions. This is most easily done ininformation form. See e.g., (Lauritzen 1992; Murphy 1998; Cemgil 2001) for the details. Thealgorithm takes O(|C|w

3)time and O(|C|w2)space. When applied to a chain structured graph,
the algorithm is equivalent to the Kalman smoother in Section 18.3.2.

726 Chapter20. Exactinferenceforgraphicalmodels
Q1 Qn Q4 Q3 Q2
C1
A1 X Am–2 A2Cm Cm–1 C3 C2
. . .. . .
Figure 20.9 Encoding a 3-SAT problem on nvariables and mclauses as a DGM. The Qsvariables are
binary random variables. The Ctvariables are deterministic functions of the Qs’s, and compute the truth
value of each clause. The Atnodes are a chain of AND gates, to ensure that the CPT for the ﬁnal xnode
has bounded size. The double rings denote nodes with deterministic CPDs. Source: Figure 9.1 of (Koller
and Friedman 2009). Used with kind permission of Daphne Koller.
20.4.4 JTA generalizations *
We have seen how to use the JTA algorithm to compute posterior marginals in a graphical model.

There are several possible generalizations of this algorithm, some of which we mention below.All of these exploit graph decomposition in some form or other. They only differ in terms ofhow they deﬁne/ compute messages and “beliefs”. The key requirement is that the operatorswhich compute messages form a commutative semiring (see Section 20.3.1).
• Computing the MAP estimate. We just replace the sum-product with max-product in the
collect phase, and use traceback in the distribute phase, as in the Viterbi algorithm (Sec-

tion 17.4.4). See (Dawid 1992) for details.
• Computing the N-most probable conﬁgurations (Nilsson 1998).
• Computing posterior samples. The collect pass is the same as usual, but in the distribute
pass, we sample variables given the values higher up in the tree, thus generalizing forwards-
ﬁltering backwards-sampling for HMMs described in Section 17.4.5. See (Dawid 1992) fordetails.
• Solving constraint satisfaction problems (Dechter 2003).
• Solving logical reasoning problems (Amir and McIlraith 2005).
20.5 Computational intractability of exact inference in the worst case
As we saw in Sections 20.3.2 and 20.4.3, VE and JTA take time that is exponential in the treewidth
of a graph. Since the treewidth can be O(number of nodes) in the worst case, this means thesealgorithms can be exponential in the problem size.

Of course, just because VE and JTA are slow doesn’t mean that there isn’t some smarter algo-
rithm out there. Unfortunately, this seems unlikely, since it is easy to show that exact inferenceis NP-hard (Dagum and Luby 1993). The proof is a simple reduction from the satisﬁability prob-

20.5. Computationalintractabilityofexactinferenceintheworstcase 727
Method Restriction Section
Forwards-backwards Chains, D or LG Section 17.4.3
Belief propagation Trees, D or LG Section 20.2
Variable elimination Low treewidth, D or LG, single query Section 20.3
Junction tree algorithm Low treewidth, D or LG Section 20.4
Loopy belief propagation Approximate, D or LG Section 22.2
Convex belief propagation Approximate, D or LG Section 22.4.2
Mean ﬁeld Approximate, C-E Section 21.3
Gibbs sampling Approximate Section 24.2
Table 20.4 Summary of some methods that can be used for inference in graphical models. “D” means
that all the hidden variables must be discrete. “L-G” means that all the factors must be linear-Gaussian.
The term “single query” refers to the restriction that VE only computes one marginal p(xq|xv)at a time.

See Section 20.3.3 for a discussion of this point. “C-E” stands for “conjugate exponential”; this means thatvariational mean ﬁeld only applies to models where the likelihood is in the exponential family, and theprior is conjugate. This includes the D and LG case, but many others as well, as we will see in Section 21.5.
lem. In particular, note that we can encode any 3-SAT problem4as a DGM with deterministic
links, as shown in Figure 20.9. We clamp the ﬁnal node, x, to be on, and we arrange the CPTs

so thatp(x=1 )>0iff there a satisfying assignment. Computing any posterior marginal
requires evaluating the normalization constant p(x=1 ), which represents the probability of the
evidence, so inference in this model implicitly solves the SAT problem.
In fact, exact inference is #P-hard (Roth 1996), which is even harder than NP-hard. (See e.g.,
(Arora and Barak 2009) for deﬁnitions of these terms.) The intuitive reason for this is that to

compute the normalizing constant Z,w eh a v et ocount how many satisfying assignments there
are. By contrast, MAP estimation is provably easier for some model classes (Greig et al. 1989),since, intuitively speaking, it only requires ﬁnding one satisfying assignment, not counting all ofthem.
20.5.1 Approximate inference

Many popular probabilistic models support efficient exact inference, since they are based onchains, trees or low treewidth graphs. But there are many other models for which exactinference is intractable. In fact, even simple two node models of the form θ→xmay not
support exact inference if the prior on θis not conjugate to the likelihood p(x|θ).
5
Therefore we will need to turn to approximate inference methods. See Table 20.4 for a

5
summary of coming attractions. For the most part, these methods do not come with anyguarantee as to their accuracy or running time. Theoretical computer scientists would thereforedescribe them as heuristics rather than approximation algorithms. In fact, one can prove that
4. A 3-SAT problem is a logical expression of the form (Q1∧Q2∧¬Q3)∨(Q1∧¬Q4∧Q5)···, where the Qiare
binary variables, and each clause consists of the conjunction of three variables (or their negation). The goal is to ﬁnd a

satisfying assignment, which is a set of values for the Qivariables such that the expression evaluates to true.
5. For discrete random variables, conjugacy is not a concern, since discrete distributions are always closed underconditioning and marginalization. Consequently, graph-theoretic considerations are of more importance when discussing
inference in models with discrete hidden states.

728 Chapter20. Exactinferenceforgraphicalmodels
it is not possible to construct polynomial time approximation schemes for inference in general
discrete GMs (Dagum and Luby 1993; Roth 1996). Fortunately, we will see that for many of theseheuristic methods often perform well in practice.
Exercises
Exercise 20.1 Variable elimination
Consider the MRF in Figure 10.14(b).
a. Suppose we want to compute the partition function using the elimination ordering ≺=( 1,2,3,4,5,6),
i.e.,
∑
x6∑
x5∑
x4∑
x3∑
x2∑
x1ψ12(x1,x2)ψ13(x1,x3)ψ24(x2,x4)ψ34(x3,x4)ψ45(x4,x5)ψ56(x5,x6)(20.71)
If we use the variable elimination algorithm, we will create new intermediate factors. What is the largest
intermediate factor?
b. Add an edge to the original MRF between every pair of variables that end up in the same factor. (These
are called ﬁll in edges.) Draw the resulting MRF. What is the size of the largest maximal clique in thisgraph?
c. Now consider elimination ordering ≺=( 4,1,2,3,5,6), i.e.,
∑
x6∑
x5∑
x3∑
x2∑
x1∑

x4∑
x3∑
x2∑
∑
x6∑
x5∑
x3∑
x2∑
x1∑
x4ψ12(x1,x2)ψ13(x1,x3)ψ24(x2,x4)ψ34(x3,x4)ψ45(x4,x5)ψ56(x5,x6)(20.72)
If we use the variable elimination algorithm, we will create new intermediate factors. What is the largestintermediate factor?
d. Add an edge to the original MRF between every pair of variables that end up in the same factor. (These
Exercise 20.2 Gaussian times Gaussian is Gaussian

∑
x6∑
x5∑
x3∑
x2∑
x1∑
Prove Equation 20.17. Hint: use completing the square.
Exercise 20.3 Message passing on a tree
Consider the DGM in Figure 20.10 which represents the following ﬁctitious biological model. Each G
i
represents the genotype of a person: Gi=1if they have a healthy gene and Gi=2if they have an
unhealthy gene. G2andG3may inherit the unhealthy gene from their parent G1.Xi∈Ris a continuous
measure of blood pressure, which is low if you are healthy and high if you are unhealthy. We deﬁne the
CPDs as follows
p(G1)=[ 0 .5,0.5] (20.73)

i
p(G2|G1)=(0.90.1
0.10.9)
(20.74)
p(G3|G1)=(0.90.1
0.10.9)
(20.75)
p(Xi|Gi=1 ) = N(Xi|μ=5 0,σ2= 10) (20.76)
p(Xi|Gi=2 ) = N(Xi|μ=6 0,σ2= 10) (20.77)
The meaning of the matrix for p(G2|G1)is thatp(G2=1|G1=1 )=0 .9,p(G2=1|G1=2 )=0 .1,
etc.

20.5. Computationalintractabilityofexactinferenceintheworstcase 729
X1
X2X3G1
G2 G3
Figure 20.10 A simple DAG representing inherited diseases.
a. Suppose you observe X2=5 0, andX1is unobserved. What is the posterior belief on G1, i.e.,
p(G1|X2= 50)?
b. Now suppose you observe X2=5 0amdX3=5 0. What is p(G1|X2,X3)? Explain your answer
intuitively.
c. Now suppose X2=6 0,X3=6 0. What is p(G1|X2,X3)? Explain your answer intuitively.
d. Now suppose X2=5 0,X3=6 0. What is p(G1|X1,X2)? Explain your answer intuitively.
Exercise 20.4 Inference in 2D lattice MRFs
Consider an MRF with a 2D m×nlattice graph structure, so each hidden node, Xij, is connected to its
4 nearest neighbors, as in an Ising model. In addition, each hidden node has its own local evidence, Yij.
Assume all hidden nodes have K>2states. In general, exact inference in such models is intractable,
because the maximum cliques of the corresponding triangulated graph have size O(max{m,n}). Suppose
m≪ni.e., the lattice is short and fat.

a. How can one efficiently perform exact inference (using a deterministic algorithm) in such models? (By
exact inference, I mean computing marginal probabilities P(Xij|⃗ y)exactly, where ⃗ yis all the evidence.)
Give abriefdescription of your method.
b. What is the asymptotic complexity (running time) of your algorithm?
c. Now suppose the lattice is large and square, so m=n, but all hidden states are binary (ie K=2). In
this case, how can one efficiently exactly compute (using a deterministic algorithm) the MAP estimate

argmax xP(x|y),w h e r ex is the joint assignment to all hidden nodes?

21 Variational inference
21.1 Introduction
We have now seen several algorithms for computing (functions of) a posterior distribution. For
discrete graphical models, we can use the junction tree algorithm to perform exact inference,as explained in Section 20.4. However, this takes time exponential in the treewidth of thegraph, rendering exact inference often impractical. For the case of Gaussian graphical models,exact inference is cubic in the treewidth. However, even this can be too slow if we have manyvariables. In addition, the JTA does not work for continuous random variables outside of theGaussian case, nor for mixed discrete-continuous variables, outside of the conditionally Gaussiancase.
For some simple two node graphical models, of the form x→D, we can compute the
exact posterior p(x|D)in closed form, provided the prior p(x)is conjugate to the likelihood,
p(D|x)(which means the likelihood must be in the exponential family). See Chapter 5 for some

examples of this. (Note that in this chapter, xrepresent the unknown variables, whereas in
Chapter 5, we used θto represent the unknowns.)
In more general settings, we must use approximate inference methods. In Section 8.4.1, we
discussed the Gaussian approximation, which is useful for inference in two node models of theformx→D, where the prior is not conjugate. (For example, Section 8.4.3 applied the method
to logistic regression.)
The Gaussian approximation is simple. However, some posteriors are not naturally modelled
using Gaussians. For example, when inferring multinomial parameters, a Dirichlet distribution isa better choice, and when inferring states in a discrete graphical model, a categorical distributionis a better choice.

In this chapter, we will study a more general class of deterministic approximate inference
algorithms based on variational inference (Jordan et al. 1998; Jaakkola and Jordan 2000; Jaakkola
2001; Wainwright and Jordan 2008a). The basic idea is to pick an approximation q(x)to the
distribution from some tractable family, and then to try to make this approximation as closeas possible to the true posterior, p
∗(x)≜p(x|D). This reduces inference to an optimization

problem. By relaxing the constraints and/or approximating the objective, we can trade accuracyfor speed. The bottom line is that variational inference often gives us the speed beneﬁts of MAPestimation but the statistical beneﬁts of the Bayesian approach.

732 Chapter21. Variationalinference
21.2 Variational inference
Supposep∗(x)is our true but intractable distribution and q(x)is some approximation, chosen
from some tractable family, such as a multivariate Gaussian or a factored distribution. We
assumeqhas some free parameters which we want to optimize so as to make q“similar to” p∗.
An obvious cost function to try to minimize is the KL divergence:
KL(p∗||q)=∑
xp∗(x)logp∗(x)
q(x)(21.1)
However, this is hard to compute, since taking expectations wrt p∗is assumed to be intractable.
A natural alternative is the reverse KL divergence:
KL(q||p∗)=∑
xq(x)logq(x)
p∗(x)(21.2)
The main advantage of this objective is that computing expectations wrt qis tractable (by choos-
ing a suitable form for q). We discuss the statistical differences between these two objectives in
Section 21.2.2.
Unfortunately, Equation 21.2 is still not tractable as written, since even evaluating p∗(x)=
p(x|D)pointwise is hard, since it requires evaluating the intractable normalization constant

Z=p(D). However, usually the unnormalized distribution ˜p(x)≜p(x,D)=p∗(x)Zis
tractable to compute. We therefore deﬁne our new objective function as follows:
J(q)≜KL(q||˜p) (21.3)
where we are slightly abusing notation, since ˜pis not a normalized distribution. Plugging in the
deﬁnition of KL, we get
J(q)=∑
˜p(x)(21.4)
=∑
Zp∗(x)(21.5)
=∑
p∗(x)−logZ (21.6)
=KL(q||p∗)−logZ (21.7)
SinceZis a constant, by minimizing J(q), we will force qto become close to p∗.

J(q)=∑
=∑
=∑
Since KL divergence is always non-negative, we see that J(q)is an upper bound on the NLL
(negative log likelihood):
J(q)=KL(q||p∗)−logZ≥−logZ=−logp(D) (21.8)
Alternatively, we can try to maximize the following quantity (in (Koller and Friedman 2009), this
is referred to as the energy functional), which is a lower bound on the log likelihood of the
data:
L(q)≜−J(q)=−KL(q||p∗)+logZ≤logZ=l o gp(D) (21.9)
Since this bound is tight when q=p∗, we see that variational inference is closely related to EM
(see Section 11.4.7).

21.2. Variationalinference 733
21.2.1 Alternative interpretations of the variational objective
There are several equivalent ways of writing this objective that provide different insights. One
formulation is as follows:
J(q)=Eq[logq(x)]+Eq[−log ˜p(x)] =−H(q)+Eq[E(x)] (21.10)
which is the expected energy (recall E(x)=−log ˜p(x)) minus the entropy of the system. In
statistical physics, J(q)is called the variational free energy or theHelmholtz free energy.1
Another formulation of the objective is as follows:
J(q)=E q[logq(x)−logp(x)p(D|x)] (21.11)
=Eq[logq(x)−logp(x)−logp(D|x)] (21.12)
=Eq[−logp(D|x)]+KL(q(x)||p(x)) (21.13)
This is the expected NLL, plus a penalty term that measures how far the approximate posterioris from the exact prior.
We can also interpret the variational objective from the point of view of information theory
(the so-called bits-back argument). See (Hinton and Camp 1993; Honkela and Valpola 2004), fordetails.
21.2.2 Forward or reverse KL? *

Since the KL divergence is not symmetric in its arguments, minimizing KL(q||p)wrtqwill give
different behavior than minimizing KL(p||q). Below we discuss these two different methods.
First, consider the reverse KL, KL(q||p), also known as an I-projection orinformation
projection. By deﬁnition, we have
KL(q||p)=∑
xq(x)lnq(x)
p(x)(21.14)
This is inﬁnite if p(x)=0andq(x)>0. Thus if p(x)=0we must ensure q(x)=0.W es a y
that the reverse KL is zero forcing forq. Henceqwill typically under-estimate the support of
p.

p.
Now consider the forwards KL, also known as an M-projection ormoment projection:
xp(x)lnp(x)
q(x)(21.15)
This is inﬁnite if q(x)=0andp(x)>0.S oi fp(x)>0we must ensure q(x)>0.W es a y
that the forwards KL is zero avoiding forq. Henceqwill typically over-estimate the support of
p.
The difference between these methods is illustrated in Figure 21.1. We see that when the true
distribution is multimodal, using the forwards KL is a bad idea (assuming qis constrained to

p.
p.
be unimodal), since the resulting posterior mode/mean will be in a region of low density, rightbetween the two peaks. In such contexts, the reverse KL is not only more tractable to compute,but also more sensible statistically.

734 Chapter21. Variationalinference
(a) (b) (c)
Figure 21.1 Illustrating forwards vs reverse KL on a bimodal distribution. The blue curves are the contours
of the true distribution p. The red curves are the contours of the unimodal approximation q. (a) Minimizing
forwards KL: qtends to “cover” p. (b-c) Minimizing reverse KL: qlocks on to one of the two modes. Based
on Figure 10.3 of (Bishop 2006b). Figure generated by KLfwdReverseMixGauss .
−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1−0.8−0.6−0.4−0.200.20.40.60.81
(a)−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1−0.8−0.6−0.4−0.200.20.40.60.81
(b)
Figure 21.2 Illustrating forwards vs reverse KL on a symmetric Gaussian. The blue curves are the
contours of the true distribution p. The red curves are the contours of a factorized approximation q. (a)
Minimizing KL(q||p). (b) Minimizing KL(p||q). Based on Figure 10.2 of (Bishop 2006b). Figure generated
byKLpqGauss .
Another example of the difference is shown in Figure 21.2, where the target distribution is

(b)
an elongated 2d Gaussian and the approximating distribution is a product of two 1d Gaussians.
That is,p(x)=N(x|μ,Λ−1),w h e r e
μ=(μ1
μ2)
,Λ=(Λ11Λ12
Λ21Λ22)
(21.16)
In Figure 21.2(a) we show the result of minimizing KL(q||p). In this simple example, one can
show that the solution has the form
q(x)=N (x1|m1,Λ−1
11)N(x2|m2,Λ−1
22) (21.17)
m1=μ1−Λ−1
11Λ12(m2−μ2) (21.18)
m2=μ2−Λ−1
22Λ21(m1−μ1) (21.19)
1. It is called “free” because the variables xare free to vary, rather than being ﬁxed. The variational free energy is a

μ=(μ1
μ2)
Λ21Λ22)
(21.16)
m1=μ1−Λ−1
m2=μ2−Λ−1
function of the distribution q, whereas the regular energy is a function of the state vector x.

21.3. Themeanﬁeldmethod 735
Figure 21.2(a) shows that we have correctly captured the mean, but the approximation is too
compact: its variance is controlled by the direction of smallest variance of p. In fact, it is
often the case (although not always (Turner et al. 2008)) that minimizing KL(q||p),w h e r eqis
factorized, results in an approximation that is overconﬁdent.
In Figure 21.2(b), we show the result of minimizing KL(p||q). As we show in Exercise 21.7,
the optimal solution when minimizing the forward KL wrt a factored approximation is to set q
to be the product of marginals. Thus the solution has the form
q(x)=N(x1|μ1,Λ−1
11)N(x2|μ2,Λ−1
22) (21.20)
Figure 21.2(b) shows that this is too broad, since it is an over-estimate of the support of p.
For the rest of this chapter, and for most of the next, we will focus on minimizing KL(q||p).
In Section 22.5, when we discuss expectation proagation, we will discuss ways to locally optimizeKL(p||q).

One can create a family of divergence measures indexed by a parameter α∈Rby deﬁning
thealpha divergence as follows:
D
α(p||q)≜4
1−α2(
1−∫
p(x)(1+α)/2q(x)(1−α)/2dx)
(21.21)
This measure satisﬁes Dα(p||q)=0iffp=q, but is obviously not symmetric, and hence is
not a metric. KL(p||q)corresponds to the limit α→1, whereas KL(q||p)corresponds to the
limitα→−1. When α=0, we get a symmetric divergence measure that is linearly related to
theHellinger distance, deﬁned by
DH(p||q)≜∫(
p(x)1
2−q(x)1
2)2
dx (21.22)
Note that√

D
α(p||q)≜4
1−α2(
1−∫
(21.21)
p(x)1
2−q(x)1
2)2
dx (21.22)
Note that√
DH(p||q)is a valid distance metric, that is, it is symmetric, non-negative and
satisﬁes the triangle inequality. See (Minka 2005) for details.
21.3 The mean ﬁeld method
One of the most popular forms of variational inference is called the mean ﬁeld approxima-
tion (Opper and Saad 2001). In this approach, we assume the posterior is a fully factorizedapproximation of the form
q(x)=∏
iqi(xi) (21.23)
Our goal is to solve this optimization problem:
min
q1,...,qDKL(q||p) (21.24)
where we optimize over the parameters of each marginal distribution qi. In Section 21.3.1, we

q(x)=∏
min
derive a coordinate descent method, where at each step we make the following update:
logqj(xj)=E−qj[log ˜p(x)]+const (21.25)

736 Chapter21. Variationalinference
Ising model Section 21.3.2
Factorial HMM Section 21.4.1
Univariate Gaussian Section 21.5.1
Linear regression Section 21.5.2Logistic regression Section 21.8.1.1Mixtures of Gaussians Section 21.6.1Latent Dirichlet allocation Section 27.3.6.3
Table 21.1 Some models in this book for which we provide detailed derivations of the mean ﬁeld inference
where˜p(x)=p(x,D)is the unnormalized posterior and the notation E−qj[f(x)]means to
take the expectation over f(x)with respect to all the variables except for xj. For example, if
we have three variables, then
E−q2[f(x)] =∑
x1∑
x3q(x1)q3(x3)f(x1,x2,x3) (21.26)
where sums get replaced by integrals where necessary.
When updating qj, we only need to reason about the variables which share a factor with xj,
i.e., the terms in j’s Markov blanket (see Section 10.5.3); the other terms get absorbed into the

x1∑
constant term. Since we are replacing the neighboring values by their mean value, the methodis known as mean ﬁeld. This is very similar to Gibbs sampling (Section 24.2), except insteadof sending sampled values between neighboring nodes, we send mean values between nodes.This tends to be more efficient, since the mean can be used as a proxy for a large number ofsamples. (On the other hand, mean ﬁeld messages are dense, whereas samples are sparse; thiscan make sampling more scalable to very large models.)

Of course, updating one distribution at a time can be slow, since it is a form of coordinate
descent. Several methods have been proposed to speed up this basic approach, including usingpattern search (Honkela et al. 2003), and techniques based on parameter expansion (Qi andJaakkola 2008). However, we will not consider these methods in this chapter.
It is important to note that the mean ﬁeld method can be used to infer discrete or continuous
latent quantities, using a variety of parametric forms for q

i, as we will see below. This is
in contrast to some of the other variational methods we will encounter later, which are morerestricted in their applicability. Table 21.1 lists some of the examples of mean ﬁeld that we coverin this book.
21.3.1 Derivation of the mean ﬁeld update equations
Recall that the goal of variational inference is to minimize the upper bound J(q)≥−logp(D).
Equivalently, we can try to maximize the lower bound
L(q)≜−J(q)=∑
xq(x)log˜p(x)
q(x)≤logp(D) (21.27)

We will do this one term at a time.

21.3. Themeanﬁeldmethod 737
If we write the objective singling out the terms that involve qj, and regarding all the other
terms as constants, we get
L(qj)=∑
x∏
iqi(xi)[
log ˜p(x)−∑
klogqk(xk)]
(21.28)
=∑
xj∑
x−jqj(xj)∏
i̸=jqi(xi)[
(21.29)
=∑
xjqj(xj)∑
x−j∏
i̸=jqi(xi)log ˜p(x)
−∑
xjqj(xj)∑
x−j∏
i̸=jqi(xi)⎡
⎣∑
k̸=jlogqk(xk)+qj(xj)⎤⎦ (21.30)
=∑
xjqj(xj)logfj(xj)−∑
xjqj(xj)logqj(xj)+const (21.31)
where
logfj(xj)≜∑
x−j∏
i̸=jqi(xi)log ˜p(x)=E−qj[log ˜p(x)] (21.32)
So we average out all the hidden variables except for xj. Thus we can rewrite L(qj)as follows:
L(qj)=−KL(qj||fj) (21.33)
We can maximize Lby minimizing this KL, which we can do by setting qj=fj, as follows:
qj(xj)=1
Zjexp(
E−qj[log ˜p(x)])
(21.34)
We can usually ignore the local normalization constant Zj, since we know qjmust be a
normalized distribution. Hence we usually work with the form
logqj(xj)=E−qj[log ˜p(x)]+const (21.35)
The functional form of the qjdistributions will be determined by the type of variables xj,a s

qj(xj)=1
Zjexp(
(21.34)
well as the form of the model. (This is sometimes called free-form optimization.) If xjis a
discrete random variable, then qjwill be a discrete distribution; if xjis a continuous random
variable, then qjwill be some kind of pdf. We will see examples of this below.
21.3.2 Example: mean ﬁeld for the Ising model
Consider the image denoising example from Section 19.4.1, where xi∈{−1,+1}are the hidden
pixel values of the “clean” image. We have a joint model of the form
p(x,y)=p( x)p(y|x) (21.36)

738 Chapter21. Variationalinference
where the prior has the form
p(x)=1
Z0exp(−E0(x)) (21.37)
E0(x)=−D∑
i=1∑
j∈nbriWijxixj (21.38)
and the likelihood has the form
p(y|x)=∏
ip(yi|xi)=∑
iexp(−Li(xi)) (21.39)
Therefore the posterior has the form
p(x|y)=1
Zexp(−E(x)) (21.40)
E(x)=E 0(x)−∑
iLi(xi) (21.41)
We will now approximate this by a fully factored approximation
q(x)=∏
iq(xi,μi) (21.42)
whereμiis the mean value of node i. To derive the update for the variational parameter μi,w e
ﬁrst write out log ˜p(x)=−E(x), dropping terms that do not involve xi:
log ˜p(x)=xi∑
j∈nbriWijxj+Li(xi)+const (21.43)
This only depends on the states of the neighboring nodes. Now we take expectations of this wrt∏
j̸=iqj(xj)to get
qi(xi)∝exp⎛
⎝xi∑
j∈nbriWijμj+Li(xi)⎞⎠ (21.44)
Thus we replace the states of the neighbors by their average values. Let
m
i=∑
j∈nbriWijμj (21.45)
be the mean ﬁeld inﬂuence on node i. Also, let L+
i≜Li(+1)andL−i≜Li(−1). The
approximate marginal posterior is given by
qi(xi=1 ) =emi+L+
i
emi+L+i+e−mi+L−i=1

⎝xi∑
m
i=∑
i
1+e−2mi+L−i−L+i=s i g m ( 2 ai) (21.46)
ai≜mi+0.5(L+
i−L−i) (21.47)

21.4. Structuredmeanﬁeld* 739
sample 1, meanfieldH
  
−0.8−0.6−0.4−0.200.20.40.60.8
(a)
sample 3, meanfieldH
  
(b)
mean after 15 sweeps of meanfieldH
  
(c)
Figure 21.3 Example of image denoising using mean ﬁeld (with parallel updates and a damping factor
of 0.5). We use an Ising prior with Wij=1and a Gaussian noise model with σ=2. We show
the results after 1, 3 and 15 iterations across the image. Compare to Figure 24.1. Figure generated by
isingImageDenoiseDemo .
Similarly, we have qi(xi=−1) = sigm(−2ai). From this we can compute the new mean for
sitei:
μi=Eqi[xi]=qi(xi=+ 1 )·(+1)+qi(xi=−1)·(−1) (21.48)
=1
1+e−2ai−1
1+e2ai=eai
eai+e−ai−e−ai
e−ai+eai= tanh(ai) (21.49)
Hence the update equation becomes
μi= tanh⎛
⎝∑
j∈nbriWijμj+0.5(L+
i−L−
i)⎞
⎠ (21.50)
See also Exercise 21.6 for an alternative derivation of these equations.
We can turn the above equations in to a ﬁxed point algorithm by writing
μt
i= tanh⎛
⎝∑
j∈nbriWijμt−1
j+0.5(L+
i−L−
i)⎞
⎠ (21.51)

sitei:
=1
1+e−2ai−1
μi= tanh⎛
⎝∑
i−L−
i)⎞
⎠ (21.50)
μt
i= tanh⎛
⎝∑
j+0.5(L+
i−L−
i)⎞
⎠ (21.51)
It is usually better to use damped updates of the form
μt
i=( 1−λ)μt−1
i+λtanh⎛
⎝∑
j+0.5(L+
i−L−
i)⎞
⎠ (21.52)
for0<λ<1. We can update all the nodes in parallel, or update them asychronously.
Figure 21.3 shows the method in action, applied to a 2d Ising model with homogeneous
attractive potentials, Wij=1. We use parallel updates with a damping factor of λ=0.5. (If we
don’t use damping, we tend to get “checkerboard” artefacts.)
21.4 Structured mean ﬁeld *

j+0.5(L+
i−L−
i)⎞
⎠ (21.51)
μt
i+λtanh⎛
⎝∑
j+0.5(L+
i−L−
i)⎞
⎠ (21.52)
Assuming that all the variables are independent in the posterior is a very strong assumption that
can lead to poor results. Sometimes we can exploit tractable substructure in our problem, so

740 Chapter21. Variationalinference
y1y2y3x3,1x3,2x3,3x2,1x2,2x2,3x1,1x1,2x1,3
(a)x3,1x3,2x3,3x2,1x2,2x2,3x1,1x1,2x1,3
(b)x3,1x3,2x3,3x2,1x2,2x2,3x1,1x1,2x1,3
(c)
Figure 21.4 (a) A factorial HMM with 3 chains. (b) A fully factorized approximation. (c) A product-of-
chains approximation. Based on Figure 2 of (Ghahramani and Jordan 1997).
that we can efficiently handle some kinds of dependencies. This is called the structured mean
ﬁeldapproach (Saul and Jordan 1995). The approach is the same as before, except we group sets
of variables together, and we update them simultaneously. (This follows by simply treating all
the variables in the i’th group as a single “mega-variable”, and then repeating the derivation in
Section 21.3.1.) As long as we can perform efficient inference in each qi, the method is tractable
overall. We give an example below. See (Bouchard-Cote and Jordan 2009) for some more recentwork in this area.
21.4.1 Example: factorial HMM

Consider the factorial HMM model (Ghahramani and Jordan 1997) introduced in Section 17.6.5.Suppose there are Mchains, each of length T, and suppose each hidden node has Kstates.
The model is deﬁned as follows
p(x,y)=∏
m∏
tp(xtm|xt−1,m)p(yt|xtm) (21.53)
wherep(xtm=k|xt−1,m=j)=A mjkis an entry in the transition matrix for chain m,
p(x1m=k|x0m)=p(x1m=k)=πmk, is the initial state distribution for chain m, and
p(yt|xt)=N(
yt|M∑
m=1Wmxtm,Σ)
(21.54)
is the observation model, where xtmis a 1-of-K encoding of xtmandWmis aD×K

p(x,y)=∏
m∏
yt|M∑
(21.54)
matrix (assuming yt∈RD). Figure 21.4(a) illustrates the model for the case where M=3.
Even though each chain is a priori independent, they become coupled in the posterior dueto having an observed common child, y
t. The junction tree algorithm applied to this graph
takesO(TMKM+1)time. Below we will derive a structured mean ﬁeld algorithm that takes
O(TMK2I)time, where Iis the number of mean ﬁeld iterations (typically I∼10suffices for
good performance).

21.4. Structuredmeanﬁeld* 741
We can write the exact posterior in the following form:
p(x|y)=1
Zexp(−E(x,y)) (21.55)
E(x,y)=1
2T∑
t=1(
yt−∑
mWmxtm)T
Σ−1(
yt−∑
mWmxtm)
−∑
mxT
1m˜πm−T∑
t=2∑
mxTtm˜Amxt−1,m (21.56)
where˜Am≜logAmand˜πm≜logπm(both interpreted elementwise).
We can approximate the posterior as a product of marginals, as in Figure 21.4(b), but a better
approximation is to use a product of chains, as in Figure 21.4(c). Each chain can be tractably
updated individually, using the forwards-backwards algorithm. More precisely, we assume
q(x|y)=1
ZqM∏
m=1q(x1m|ξ1m)T∏
t=2q(xtm|xt−1,m,ξtm) (21.57)
q(x1m|ξ1m)=K∏
k=1(ξ1mkπmk)x1mk(21.58)
q(xtm|xt−1,m,ξtm)=K∏
k=1⎛
⎝ξtmkK∏
j=1(Amjk)xt−1,m,j⎞⎠xtmk
(21.59)
We see that the ξtmkparameters play the role of an approximate local evidence, averaging out
the effects of the other chains. This is contrast to the exact local evidence, which couples all
the chains together.
We can rewrite the approximate posterior as q(x)=1
Zqexp(−Eq(x)),w h e r e
Eq(x)=−T∑
t=1M∑
m=1xT

q(x|y)=1
ZqM∏
k=1⎛
⎝ξtmkK∏
(21.59)
Eq(x)=−T∑
t=1M∑
m=1xT
tm˜ξtm−M∑
m=1xT1m˜πm−T∑
t=2M∑
m=1xTtm˜Amxt−1,m (21.60)
where˜ξtm=l o gξtm. We see that this has the same temporal factors as the exact posterior,
but the local evidence term is different. The objective function is given by
KL(q||p)=E[E]−E[Eq]−logZq+logZ (21.61)
where the expectations are taken wrt q. One can show (Exercise 21.8) that the update has the
form
ξtm=e x p(
WT
mΣ−1˜ytm−1
2δm)
(21.62)
δm≜diag(WT
mΣ−1Wm) (21.63)
˜ytm≜yt−M∑
ℓ̸=mWℓE[xt,ℓ] (21.64)

742 Chapter21. Variationalinference
Theξtmparameter plays the role of the local evidence, averaging over the neighboring chains.
Having computed this for each chain, we can perform forwards-backwards in parallel, using
these approximate local evidence terms to compute q(xt,m|y1:T)for eachmandt.
The update cost is O(TMK2)for a full “sweep” over all the variational parameters, since we
have to run forwards-backwards Mtimes, for each chain independently. This is the same cost
as a fully factorized approximation, but is much more accurate.
21.5 Variational Bayes
So far we have been concentrating on inferring latent variables ziassuming the parameters θ
of the model are known. Now suppose we want to infer the parameters themselves. If wemake a fully factorized (i.e., mean ﬁeld) approximation, p(θ|D)≈∏
kq(θk), we get a method
known as variational Bayes orVB(Hinton and Camp 1993; MacKay 1995a; Attias 2000; Beal
and Ghahramani 2006; Smidl and Quinn 2005).2We give some examples of VB below, assuming

that there are no latent variables. If we want to infer both latent variables and parameters, andwe make an approximation of the form p(θ,z
1:N|D)≈q(θ)∏
iqi(zi), we get a method known
as variational Bayes EM, which we described in Section 21.6.
21.5.1 Example: VB for a univariate Gaussian
Following (MacKay 2003, p429), let us consider how to apply VB to infer the posterior over theparameters for a 1d Gaussian, p(μ,λ|D),w h e r eλ=1/σ
2is the precision. For convenience, we
will use a conjugate prior of the form
p(μ,λ)=N(μ|μ0,(κ0λ)−1)Ga(λ|a0,b0) (21.65)

However, we will use an approximate factored posterior of the form
q(μ,λ)=qμ(μ)qλ(λ) (21.66)
We do not need to specify the forms for the distributions qμandqλ; the optimal forms will “fall
out” automatically during the derivation (and conveniently, they turn out to be Gaussian andGamma respectively).
You might wonder why we would want to do this, since we know how to compute the

exact posterior for this model (Section 4.6.3.7). There are two reasons. First, it is a usefulpedagogical exercise, since we can compare the quality of our approximation to the exactposterior. Second, it is simple to modify the method to handle a semi-conjugate prior of theformp(μ,λ)=N(μ|μ
0,τ0)Ga(λ|a0,b0), for which exact inference is no longer possible.
2. This method was originally called ensemble learning (MacKay 1995a), since we are using an ensemble of parameters

(a distribution) instead of a point estimate. However, the term “ensemble learning” is also used to describe methods
such as boosting, so we prefer the term VB.

21.5. VariationalBayes 743
21.5.1.1 Target distribution
The unnormalized log posterior has the form
log ˜p(μ,λ)=l o g p(μ,λ,D)=l o gp(D|μ,λ)+logp(μ|λ)+logp(λ) (21.67)
=N
2logλ−λ
2N∑
i=1(xi−μ)2−κ0λ
2(μ−μ0)2
+1
2log(κ0λ)+(a0−1)logλ−b0λ+const (21.68)
21.5.1.2 Updating qμ(μ)
The optimal form for qμ(μ)is obtained by averaging over λ:
logqμ(μ)=E qλ[logp(D|μ,λ)+logp(μ|λ)]+const (21.69)
=−Eqλ[λ]
2{
κ0(μ−μ0)2+N∑
i=1(xi−μ)2}
+const (21.70)
By completing the square one can show that qμ(μ)=N(μ|μN,κ−1
N),w h e r e
μN=κ0μ0+Nx
κ0+N,κN=(κ0+N)Eqλ[λ] (21.71)
At this stage we don’t know what qλ(λ)is, and hence we cannot compute E[λ], but we will
derive this below.
21.5.1.3 Updating qλ(λ)
The optimal form for qλ(λ)is given by
logqλ(λ)=E qμ[logp(D|μ,λ)+logp(μ|λ)+logp(λ)]+const (21.72)
=(a0−1)logλ−b0λ+1
2logλ+N
2logλ
−λ
2Eqμ[
i=1(xi−μ)2]
+const (21.73)
We recognize this as the log of a Gamma distribution, hence qλ(λ)=G a ( λ|aN,bN),w h e r e
aN=a0+N+1
2(21.74)
bN=b0+1
2Eqμ[
(21.75)

744 Chapter21. Variationalinference
21.5.1.4 Computing the expectations
To implement the updates, we have to specify how to compute the various expectations. Since
q(μ)=N(μ|μN,κ−1
N),w eh a v e
Eq(μ)[μ]=μ N (21.76)
Eq(μ)[
μ2]
=1
κN+μ2
N (21.77)
Sinceq(λ)=G a ( λ|aN,bN),w eh a v e
Eq(λ)[λ]=aN
bN(21.78)
We can now give explicit forms for the update equations. For q(μ)we have
κ0+N(21.79)
κN=(κ0+N)aN
bN(21.80)
and forq(λ)we have
aN=a0+N+1
2(21.81)
bN=b0+κ0(E[
μ2]
+μ20−2E[μ]μ0)+1
2N∑
i=1(
x2i+E[
μ2]
−2E[μ]xi)
(21.82)
We see that μNandaNare in fact ﬁxed constants, and only κNandbNneed to be updated
iteratively. (In fact, one can solve for the ﬁxed points of κNandbNanalytically, but we don’t
do this here in order to illustrate the iterative updating scheme.)
21.5.1.5 Illustration
Figure 21.5 gives an example of this method in action. The green contours represent the

bN(21.80)
aN=a0+N+1
2(21.81)
μ2]
2N∑
i=1(
x2i+E[
μ2]
−2E[μ]xi)
(21.82)
exact posterior, which is Gaussian-Gamma. The dotted red contours represent the variationalapproximation over several iterations. We see that the ﬁnal approximation is reasonably close tothe exact solution. However, it is more “compact” than the true distribution. It is often the casethat mean ﬁeld inference underestimates the posterior uncertainty; See Section 21.2.2 for morediscussion of this point.
21.5.1.6 Lower bound *
In VB, we are maximizing L(q), which is a lower bound on the log marginal likelihood:

L(q)≤logp(D)=l o g∫∫
p(D|μ,λ)p(μ,λ)dμdλ (21.83)
It is very useful to compute the lower bound itself, for three reasons. First, it can be used toassess convergence of the algorithm. Second, it can be used to assess the correctness of one’s

21.5. VariationalBayes 745
μλ
  
−1 −0.5 0 0.5 100.20.40.60.811.21.41.61.82
exact
vb
(a)μλ
  
exact
vb
(b)
μλ
  
exact
vb
(c)μλ
  
exact
vb
(d)
Figure 21.5 Factored variational approximation (red) to the Gaussian-Gamma distribution (green). (a)
Initial guess. (b) After updating qμ. (c) After updating qλ. (d) At convergence (after 5 iterations). Based on
10.4 of (Bishop 2006b). Figure generated by unigaussVbDemo .
code: as with EM, if the bound does not increase monotonically, there must be a bug. Third,
the bound can be used as an approximation to the marginal likelihood, which can be used forBayesian model selection.
Unfortunately, computing this lower bound involves a fair amount of tedious algebra. We
work out the details for this example, but for other models, we will just state the results withoutproof, or even omit discussion of the bound altogether, for brevity.

For this model, L(q)can be computed as follows:
L(q)=∫∫
q(μ,λ)logp(D,μ,λ)
q(μ,λ)dμdλ (21.84)
=E[logp(D|μ,λ)]+E[logp(μ|λ)]+ E[logp(λ)]
−E[logq(μ)]−E[logq(λ)] (21.85)
where all expectations are wrt q(μ,λ). We recognize the last two terms as the entropy of a
Gaussian and the entropy of a Gamma distribution, which are given by
H(
N(μN,κ−1
N))
=−1
2logκN+1
2(1+log(2π )) (21.86)
H(Ga(aN,bN)) = logΓ(a N)−(aN−1)ψ(aN)−log(bN)+aN (21.87)

746 Chapter21. Variationalinference
whereψ()is the digamma function.
To compute the other terms, we need the following facts:
E[logx|x∼Ga(a,b)] =ψ(a)−log(b) (21.88)
E[x|x∼Ga(a,b)] =a
b(21.89)
E[
x|x∼N(μ,σ2)]
=μ (21.90)
E[
x2|x∼N(μ,σ2)]
=μ+σ2(21.91)
For the expected log likelihood, one can show that
Eq(μ,λ)[logp(D|μ,λ)] (21.92)
=−N
2log(2π)+N
2Eq(λ)[logλ]−E[λ]q(λ)
2N∑
i=1Eq(μ)[
(xi−μ)2]
=−N
2(ψ(aN)−logbN) (21.93)
−NaN
2bN(
ˆσ2+x2−2μNx+μ2
N+1
κN)
(21.94)
wherexandˆσ2are the empirical mean and variance.
For the expected log prior of λ,w eh a v e
Eq(λ)[logp(λ) ]=( a0−1)E[logλ]−b0E[λ]+a0logb0−logΓ(a 0) (21.95)
=(a0−1)(ψ(aN)−logbN)−b0aN
bN+a0logb0−logΓ(a 0)(21.96)
For the expected log prior of μ, one can show that
Eq(μ,λ)[logp(μ|λ)] =1
2logκ0
2π+1
2E[logλ]q(λ)−1
2Eq(μ,λ)[
(μ−μ0)2κ0λ]
=1
2logκ0
2π+1
2(ψ(aN)−logbN)
−κ0
2aN
bN[1
κN+(μN−μ0)2]
(21.97)
Putting it altogether, one can show that
L(q)=1
2log1
κN+logΓ(a N)−aNlogbN+const (21.98)
This quantity monotonically increases after each VB update.

2logκ0
2π+1
2Eq(μ,λ)[
=1
2logκ0
2π+1
−κ0
2aN
bN[1
(21.97)
L(q)=1
2log1
21.5.2 Example: VB for linear regression
In Section 7.6.4, we discussed an empirical Bayes approach to setting the hyper-parameters for
ridge regression known as the evidence procedure. In particular, we assumed a likelihood ofthe form p(y|X,θ)=N(Xw,λ
−1)and a prior of the form p(w)=N(w|0,α−1I). We then

21.5. VariationalBayes 747
computed a type II estimate of αandλ. The same approach was extended in Section 13.7 to
handle a prior of the form N(w|0,diag(α)−1), which allows one hyper-parameter per feature,
a technique known as automatic relevancy determination.
In this section, we derive a VB algorithm for this model. We follow the presentation of
(Drugowitsch 2008).3Initially we will use the following prior:
p(w,λ,α)=N (w|0,(λα)−1I)Ga(λ|aλ
0,bλ0)Ga(α|aα0,bα0) (21.99)
We choose to use the following factorized approximation to the posterior:
q(w,α,λ)=q(w,λ)q(α) (21.100)
Given these assumptions, one can show (see (Drugowitsch 2008)) that the optimal form for the
posterior is
q(w,α,λ)=N (w|wN,λ−1VN)Ga(λ|aλ
N,bλN)Ga(α|aαN,bαN) (21.101)
where
V−1
N=A+XX(21.102)
wN=VNXTy (21.103)
aλN=aλ0+N
2(21.104)
bλN=bλ0+1
2(||y−Xw||2+wT
NAwN) (21.105)
aαN=aα0+D
2(21.106)
bαN=bα0+1
2(aλ
N
bλNwT
NwN+tr(VN))
(21.107)
A=⟨α⟩I=aαN
bαNI (21.108)

where
V−1
aλN=aλ0+N
2(21.104)
bλN=bλ0+1
aαN=aα0+D
2(21.106)
bαN=bα0+1
2(aλ
N
bλNwT
(21.107)
This method can be extended to the ARD case in a straightforward way, by using the following
priors:
p(w)=N (0,diag(α)−1) (21.109)
p(α)=D∏
j=1Ga(αj|aα
0,bα0) (21.110)
The posterior for wandλis computed as before, except we use A=d i a g (aα
N/bαN
j)instead of
3. Note that Drugowitsch uses a0,b0as the hyper-parameters for p(λ)andc0,d0as the hyper-parameters for p(α),
whereas (Bishop 2006b, Sec 10.3) uses a0,b0as the hyper-parameters for p(α)and treats λas ﬁxed. To (hopefully)
avoid confusion, I use aλ

priors:
p(α)=D∏
N/bαN
0,bλ0as the hyper-parameters for p(λ), andaα0,bα0as the hyper-parameters for p(α).

748 Chapter21. Variationalinference
aα
N/bαNI. The posterior for αhas the form
q(α)=∏
jGa(αj|aα
N,bαN
j) (21.111)
aαN=aα0+1
2(21.112)
bαN
j=bα0+1
2(aλ
N
bλNw2
N,j+(VN)jj)
(21.113)
The algorithm alternates between updating q(w,λ)andq(α). Oncewandλhave been
inferred, the posterior predictive is a Student distribution, as shown in Equation 7.76. Speciﬁcally,
for a single data case, we have
p(y|x,D)=T (y|wT
Nx,bλ
N
aλN(1+xTVNx),2aλ
N) (21.114)
The exact marginal likelihood, which can be used for model selection, is given by
p(D)=∫∫∫
p(y|X,w,λ)p(w|α)p(λ)dwdαdλ (21.115)
We can compute a lower bound on logp(D)as follows:
L(q)=−N
2N∑
i=1(aλ
N
bλN(yi−wT
Nxi)2+xT
iVNxi)
+1
2log|VN|+D
2
−logΓ(aλ0)+aλ0logbλ0−bλ0aλ
N
bλN+logΓ(aλ
N)−aλNlogbλN+aλN
−logΓ(aα0)+aα0logbα0+logΓ(aαN)−aαNlogbαN(21.116)
In the ARD case, the last line becomes
D∑
j=1[
−logΓ(aα0)+aα0logbα0+logΓ(aαN)−aαNlogbαN
j]
(21.117)
Figure 21.6 compare VB and EB on a model selection problem for polynomial regression. We

p(D)=∫∫∫
L(q)=−N
2N∑
i=1(aλ
N
bλN(yi−wT
Nxi)2+xT
iVNxi)
+1
2
N
D∑
j=1[
j]
(21.117)
see that VB gives similar results to EB, but the precise behavior depends on the sample size.WhenN=5, VB’s estimate of the posterior over models is more diffuse than EB’s, since VB
models uncertainty in the hyper-parameters. When N=3 0, the posterior estimate of the hyper-
parameters becomes more well-determined. Indeed, if we compute E[α|D]when we have an
uninformative prior, a
α
0=bα0=0,w eg e t
α=aα
N
bαN=D/2
1
2(aλ
N
bλNwT
NwN+tr(VN))(21.118)

21.6. VariationalBayesEM 749
1 2 300.20.40.60.81
MP(M|D)N=5, method=VB
(a)1 2 300.20.40.60.81
(b)
MP(M|D)N=30, method=VB
(d)
Figure 21.6 We plot the posterior over models (polynomials of degree 1, 2 and 3) assuming a uniform
priorp(m)∝1. We approximate the marginal likelihood using (a,c) VB and (b,d) EB. In (a-b), we use
N=5data points (shown in Figure 5.7). In (c-d), we use N=3 0data points (shown in Figure 5.8). Figure
generated by linregEbModelSelVsN .
Compare this to Equation 13.167 for EB:
ˆα=D
E[wTw]=D
wT
NwN+tr(VN)(21.119)
Modulo the aλ
NandbλNterms, these are the same. In hindsight this is perhaps not that
surprising, since EB is trying to maximize logp(D), and VB is trying to maximize a lower
bound on logp(D).
21.6 Variational Bayes EM
Now consider latent variable models of the form zi→xi←θ. This includes mixtures models,
PCA, HMMs, etc. There are now two kinds of unknowns: parameters, θ, and latent variables, zi.

ˆα=D
E[wTw]=D
wT
As we saw in Section 11.4, it is common to ﬁt such models using EM, where in the E step we
infer the posterior over the latent variables, p(zi|xi,θ), and in the M step, we compute a point
estimate of the parameters, θ. The justiﬁcation for this is two-fold. First, it results in simple
algorithms. Second, the posterior uncertainty in θis usually less than in zi, since the θare
informed by all Ndata cases, whereas ziis only informed by xi; this makes a MAP estimate of

750 Chapter21. Variationalinference
θmore reasonable than a MAP estimate of zi.
However, VB provides a way to be “more Bayesian”, by modeling uncertainty in the parameters
θas well in the latent variables zi, at a computational cost that is essentially the same as EM.
This method is known as variational Bayes EM orVBEM. The basic idea is to use mean ﬁeld,
where the approximate posterior has the form
p(θ,z1:N|D)≈q(θ)q(z)=q(θ)∏
iq(zi) (21.120)
The ﬁrst factorization, between θandz, is a crucial assumption to make the algorithm tractable.
The second factorization follows from the model, since the latent variables are iid conditional
onθ.
In VBEM, we alternate between updating q(zi|D)(the variational E step) and updating q(θ|D)
(the variational M step). We can recover standard EM from VBEM by approximating the param-eter posterior using a delta function, q(θ|D)≈δ
ˆθ(θ).
The variational E step is similar to a standard E step, except instead of plugging in a MAP

onθ.
ˆθ(θ).
estimate of the parameters and computing p(zi|D,ˆθ), we need to average over the parameters.
Roughly speaking, this can be computed by plugging in the posterior mean of the parametersinstead of the MAP estimate, and then computing p(z
i|D,θ)using standard algorithms, such
as forwards-backwards. Unfortunately, things are not quite this simple, but this is the basic idea.The details depend on the form of the model; we give some examples below.
The variational M step is similar to a standard M step, except instead of computing a point

estimate of the parameters, we update the hyper-parameters, using the expected sufficient statis-tics. This process is usually very similar to MAP estimation in regular EM. Again, the details onhow to do this depend on the form of the model.
The principle advantage of VBEM over regular EM is that by marginalizing out the parameters,

we can compute a lower bound on the marginal likelihood, which can be used for modelselection. We will see an example of this in Section 21.6.1.6. VBEM is also “egalitarian”, sinceit treats parameters as “ﬁrst class citizens”, just like any other unknown quantity, whereas EMmakes an artiﬁcial distinction between parameters and latent variables.
21.6.1 Example: VBEM for mixtures of Gaussians *

Let us consider how to “ﬁt” a mixture of Gaussians using VBEM. (We use scare quotes since weare not estimating the model parameters, but inferring a posterior over them.) We will followthe presentation of (Bishop 2006b, Sec 10.2). Unfortunately, the details are rather complicated.Fortunately, as with EM, one gets used to it after a bit of practice. (As usual with math, simplyreading the equations won’t help much, you should really try deriving these results yourself (ortry some of the exercises) if you want to learn this stuff in depth.)
21.6.1.1 The variational posterior

The likelihood function is the usual one for Gaussian mixture models:
p(z,X|θ)=∏
i∏
kπzik
kN(xi|μk,Λ−1
k)zik(21.121)
wherezik=1if data point ibelongs to cluster k, andzik=0otherwise.

21.6. VariationalBayesEM 751
We will assume the following factored conjugate prior
p(θ)=D i r (π|α0)∏
kN(μk|m0,(β0Λk)−1)Wi(Λ k|L0,ν0) (21.122)
whereΛkis the precision matrix for cluster k. The subscript 0 means these are parameters
of the prior; we assume all the prior parameters are the same for all clusters. For the mixing
weights, we usually use a symmetric prior, α0=α01.
The exact posterior p(z,θ|D)is a mixture of KNdistributions, corresponding to all possible
labelings z. We will try to approximate the volume around one of these modes. We will use the
standard VB approximation to the posterior:
p(θ,z1:N|D)≈q(θ)∏
iq(zi) (21.123)
At this stage we have not speciﬁed the forms of the qfunctions; these will be determined by
the form of the likelihood and prior. Below we will show that the optimal form is as follows:
q(z,θ)=q(z|θ)q(θ)=[∏
iCat(zi|ri)]
(21.124)
[
Dir(π|α)∏
kN(μk|mk,(βkΛk)−1)Wi(Λ k|Lk,νk)]
(21.125)

(21.124)
[
Dir(π|α)∏
(21.125)
(The lack of 0 subscript means these are parameters of the posterior, not the prior.) Below wewill derive the update equations for these variational parameters.
21.6.1.2 Derivation of q(z)(variational E step)
The form for q(z)can be obtained by looking at the complete data log joint, ignoring terms that
do not involve z, and taking expectations of what’s left over wrt all the hidden variables except
forz.W eh a v e
logq(z)=E
q(θ)[logp(x,z,θ)]+const (21.126)
=∑
i∑
iziklogρik+const (21.127)
logρik≜Eq(θ)[logπk]+1
2Eq(θ)[log|Λk|]−D
2log(2π)
−1
2Eq(θ)[

logq(z)=E
=∑
i∑
2log(2π)
−1
2Eq(θ)[
(xi−μk)TΛk(xi−μk)]
(21.128)
Using the fact that q(π)=D i r (π),w eh a v e
log˜πk≜E[logπk]=ψ(αk)−ψ(∑
k′αk′) (21.129)

752 Chapter21. Variationalinference
whereψ()is the digamma function. (See Exercise 21.5 for the detailed derivation.) Next, we use
the fact that
q(μk,Λk)=N(μk|mk,(βkΛk)−1)Wi(Λ k|Lk,νk) (21.130)
to get
log˜Λk≜E[log|Λk|]=D∑
j=1ψ(νk+1−j
2)
+Dlog2+log|Λk| (21.131)
Finally, for the expected value of the quadratic form, we get
E[
=Dβ−1
k+νk(xi−mk)TΛk(xi−mk) (21.132)
Putting it altogether, we get that the posterior responsibility of cluster kfor datapoint iis
rik∝˜πk˜Λ1
2
kexp(
−D
2βk−νk
2(xi−mk)TΛk(xi−mk))
(21.133)
Compare this to the expression used in regular EM:
rEM
ik∝ˆπk|ˆΛ|1
2
kexp(
−1
2(xi−ˆμk)TˆΛk(xi−ˆμk))
(21.134)
The signiﬁcance of this difference is discussed further in Section 21.6.1.7.
21.6.1.3 Derivation of q(θ)(variational M step)
Using the mean ﬁeld recipe, we have
logq(θ)=l o g p(π)+∑
klogp(μk,Λk)+∑
iEq(z)[logp(zi|π)]
+∑
k∑
iEq(z)[zik]logN(xi|μk,Λ−1
k)+const (21.135)
We see this factorizes into the form
q(θ)=q(π)∏
kq(μk,Λk) (21.136)
For theπterm, we have
logq(π)=(α0−1)∑
klogπk+∑

(21.133)
rEM
2
kexp(
−1
(21.134)
+∑
k∑
klogπk+∑
k∑
iriklogπk+const (21.137)
Exponentiating, we recognize this as a Dirichlet distribution:
q(π)=D i r ( π|α) (21.138)
αk=α0+Nk (21.139)
Nk=∑
irik (21.140)

21.6. VariationalBayesEM 753
0 20 40 60 80 100−1100−1050−1000−950−900−850−800−750−700−650−600
iterlower bound on log marginal likelihoodvariational Bayes objective for GMM on old faithful data
Figure 21.7 Lower bound vs iterations for the VB algorithm in Figure 21.8. The steep parts of the
curve correspond to places where the algorithm ﬁgures out that it can increase the bound by “killing
off” unnecessary mixture components, as described in Section 21.6.1.6. The plateaus correspond to slowlymoving the clusters around. Figure generated by mixGaussVbDemoFaithful .
For theμkandΛkterms, we have
q(μk,Λk)=N (μk|mk,(βkΛk)−1)Wi(Λ k|Lk,νk) (21.141)
βk=β0+Nk (21.142)
mk=(β0m0+Nkxk)/βk (21.143)
L−1
k=L−1
0+NkSk+β0Nk
β0+Nk(xk−m0)(xk−m0)T(21.144)
νk=ν0+Nk+1 (21.145)
xk=1
Nk∑
irikxi (21.146)
Sk=1
Nk∑
irik(xi−xk)(xi−xk)T(21.147)
This is very similar to the M step for MAP estimation discussed in Section 11.4.2.8, except here
we are computing the parameters of the posterior over θ, rather than MAP estimates of θ.

L−1
k=L−1
xk=1
Nk∑
Sk=1
Nk∑
21.6.1.4 Lower bound on the marginal likelihood
The algorithm is trying to maximize the following lower bound
L=∑
z∫
q(z,θ)logp(x,z,θ)
q(z,θ)dθ≤logp(D) (21.148)
This quantity should increase monotonically with each iteration, as shown in Figure 21.7. Un-
fortunately, deriving the bound is a bit messy, because we need to compute expectations of theunnormalized log posterior as well as entropies of the qdistribution. We leave the details (which
are similar to Section 21.5.1.6) to Exercise 21.4.

754 Chapter21. Variationalinference
21.6.1.5 Posterior predictive distribution
We showed that the approximate posterior has the form
q(θ)=D i r (π|α)∏
kN(μk|mk,(βkΛk)−1)Wi(Λ k|Lk,νk) (21.149)
Consequently the posterior predictive density can be approximated as follows, using the results
from Section 4.6.3.6:
p(x|D)≈∑
z∫
p(x|z,θ)p(z|θ)q(θ)dθ (21.150)
=∑
k∫
πkN(x|μk,Λ−1
k)q(θ)dθ (21.151)
=∑
kαk∑
k′αk′T(x|mk,Mk,νk+1−D) (21.152)
Mk=(νk+1−D)βk
1+βkLk (21.153)
This is just a weighted sum of Student distributions. If instead we used a plug-in approximation,we would get a weighted sum of Gaussian distributions.
21.6.1.6 Model selection using VBEM
The simplest way to select Kwhen using VB is to ﬁt several models, and then to use the
variational lower bound to the log marginal likelihood, L(K)≤logp(D|K), to approximate
p(K|D):
p(K|D)=e
L(K)
∑
K′eL(K′)(21.154)

=∑
k∫
=∑
kαk∑
p(K|D):
p(K|D)=e
L(K)
∑
However, the lower bound needs to be modiﬁed somewhat to take into account the lack ofidentiﬁability of the parameters (Section 11.3.1). In particular, although VB will approximate thevolume occupied by the parameter posterior, it will only do so around one of the local modes.WithKcomponents, there are K!equivalent modes, which differ merely by permuting the
labels. Therefore we should use logp(D|K)≈L(K)+log(K!).
21.6.1.7 Automatic sparsity inducing effects of VBEM

p(K|D):
p(K|D)=e
L(K)
∑
Although VB provides a reasonable approximation to the marginal likelihood (better than BIC(Beal and Ghahramani 2006)), this method still requires ﬁtting multiple models, one for eachvalue ofKbeing considered. A faster alternative is to ﬁt a single model, where Kis set large,
but where α
0is set very small, α0≪1. From Figure 2.14(d), we see that the resulting prior for
the mixing weights πhas “spikes” near the corners of the simplex, encouraging a sparse mixing
weight vector.

but where α
In regular EM, the MAP estimate of the mixing weights will have the form ˆπk∝(αk−1),
whereαk=α0+Nk. Unforuntately, this can be negative if α0=0andNk=0(Figueiredo

21.6. VariationalBayesEM 755
−2 −1.5 −1 −0.5 0 0.5 1 1.5−2.5−2−1.5−1−0.500.511.52
1
2
345
6iter 1
(a)−2 −1.5 −1 −0.5 0 0.5 1 1.5−2.5−2−1.5−1−0.500.511.52
45iter 94
(b)
Figure 21.8 We visualize the posterior mean parameters at various stages of the VBEM algorithm applied
to a mixture of Gaussians model on the Old Faithful data. Shading intensity is proportional to the mixing
weight. We initialize with K-means and use α0=0.001as the Dirichlet hyper-parameter. Based on Figure
10.6 of (Bishop 2006b). Figure generated by mixGaussVbDemoFaithful , based on code by Emtiyaz Khan.
1 2 3 4 5 60102030405060708090iter 1
(a)1 2 3 4 5 6020406080100120140160180iter 94
(b)
Figure 21.9 We visualize the posterior values of αkfor the model in Figure 21.8. We see that unnecessary
components get “killed off”. Figure generated by mixGaussVbDemoFaithful .
and Jain 2002). However, in VBEM, we use
˜πk=exp[Ψ(α k)]
exp[Ψ(∑
k′αk′)](21.155)
Nowexp(Ψ(x))≈x−0.5forx>1.S oi f αk=0, when we compute ˜πk, it’s like we substract

(b)
exp[Ψ(∑
0.5 from the posterior counts. This will hurt small clusters more than large clusters (like a
regressive tax).4The effect is that clusters which have very few (weighted) members become
more and more empty over successive iterations, whereas the popular clusters get more andmore members. This is called the rich get richer phenomenon; we will encounter it again in
Section 25.2, when we discuss Dirichlet process mixture models.
This automatic pruning method is demonstrated in Figure 21.8. We ﬁt a mixture of 6 Gaussians

to the Old Faithful dataset, but the data only really “needs” 2 clusters, so the rest get “killed off”.
4. For more details, see (Liang et al. 2007).

756 Chapter21. Variationalinference
In this example, we used α0=0.001; if we use a larger α0, we do not get a sparsity effect.
In Figure 21.9, we plot q(α|D)at various iterations; we see that the unwanted components
get extinguished. This provides an efficient alternative to performing a discrete search over the
number of clusters.
21.7 Variational message passing and VIBES

We have seen that mean ﬁeld methods, at least of the fully-factorized variety, are all very similar:just compute each node’s full conditional, and average out the neighbors. This is very similarto Gibbs sampling (Section 24.2), except the derivation of the equations is usually a bit morework. Fortunately it is possible to derive a general purpose set of update equations that work forany DGM for which all CPDs are in the exponential family, and for which all parent nodes haveconjugate distributions (Ghahramani and Beal 2001). (See (Wand et al. 2011) for a recent extensionto handle non-conjugate priors.) One can then sweep over the graph, updating nodes one at atime, in a manner similar to Gibbs sampling. This is known as variational message passing or

VMP(Winn and Bishop 2005), and has been implemented in the open-source program VIBES
5.
This is a VB analog to BUGS, which is a popular generic program for Gibbs sampling discussedin Section 24.2.6.
VMP/ mean ﬁeld is best-suited to inference where one or more of the hidden nodes are
continuous (e.g., when performing “Bayesian learning”). For models where all the hidden nodesare discrete, more accurate approximate inference algorithms can be used, as we discuss inChapter 22.
21.8 Local variational bounds *
So far, we have been focusing on mean ﬁeld inference, which is a form of variational inferencebased on minimizing KL(q||˜p),w h e r eqis the approximate posterior, assumed to be factorized,
and˜pis the exact (but unnormalized) posterior. However, there is another kind of variational
inference, where we replace a speciﬁc term in the joint distribution with a simpler function, tosimplify computation of the posterior. Such an approach is sometimes called a local variational

approximation, since we are only modifying one piece of the model, unlike mean ﬁeld, whichis a global approximation. In this section, we study several examples of this method.
21.8.1 Motivating applications
Before we explain how to derive local variational bounds, we give some examples of where thisis useful.
21.8.1.1 Variational logistic regression

Consider the problem of how to approximate the parameter posterior for multiclass logisticregression model under a Gaussian prior. One approach is to use a Gaussian (Laplace) approx-imation, as discussed in Section 8.4.3. However, a variational approach can produce a more
5. Available at http://vibes .sourceforge .net/.

21.8. Localvariationalbounds* 757
accurate approximation to the posterior, since it has tunable parameters. Another advantage is
that the variational approach monotonically optimizes a lower bound on the likelihood of thedata, as we will see.
To see why we need a bound, note that the likelihood can be written as follows:
p(y|X,w)= N∏
i=1exp[
yT
iηi−lse(ηi)]
(21.156)
whereηi=Xiwi=[xT
iw1,...,xTiwM],w h e r eM=C−1(since we set wC=0for
identiﬁability), and where we deﬁne the log-sum-exp orlsefunction as follows:
lse(ηi)≜log(
1+M∑
m=1eηim)
(21.157)
The main problem is that this likelihood is not conjugate to the Gaussian prior. Below we discuss
how to compute “Gaussian-like” lower bounds to this likelihood, which give rise to approximateGaussian posteriors.
21.8.1.2 Multi-task learning

i=1exp[
yT
(21.156)
1+M∑
m=1eηim)
(21.157)
One important application of Bayesian inference for logistic regression is where we have multiplerelated classiﬁers we want to ﬁt. In this case, we want to share information between theparameters for each classiﬁer; this requires that we maintain a posterior distibution over theparameters, so we have a measure of conﬁdence as well as an estimate of the value. We canembed the above variational method inside of a larger hierarchical model in order to performsuch multi-task learning, as described in e.g., (Braun and McAuliffe 2010).

21.8.1.3 Discrete factor analysis
Another situation where variational bounds are useful arises when we ﬁt a factor analysismodel to discrete data. This model is just like multinomial logistic regression, except the inputvariables are hidden factors. We need to perform inference on the hidden variables as well asthe regression weights. For simplicity, we might perform point estimation of the weights, andjust integrate out the hidden variables. We can do this using variational EM, where we use thevariational bound in the E step. See Section 12.4 for details.
21.8.1.4 Correlated topic model

A topic model is a latent variable model for text documents and other forms of discrete data; seeSection 27.3 for details. Often we assume the distribution over topics has a Dirichlet prior, buta more powerful model, known as the correlated topic model, uses a Gaussian prior, which canmodel correlations more easily (see Section 27.4.1 for details). Unfortunately, this also involvesthe lse function. However, we can use our variational bounds in the context of a variational EMalgorithm, as we will see later.

758 Chapter21. Variationalinference
21.8.2 Bohning’s quadratic bound to the log-sum-exp function
All of the above examples require dealing with multiplying a Gaussian prior by a multinomial
likelihood; this is difficult because of the log-sum-exp (lse) term. In this section, we derive a wayto derive a “Gaussian-like” lower bound on this likelihood.
Consider a Taylor series expansion of the lse function around ψ
i∈RM:
lse(ηi) = lse(ψi)+(ηi−ψi)Tg(ψi)+1
2(ηi−ψi)TH(ψi)(ηi−ψi) (21.158)
g(ψi)=e x p [ ψi−lse(ψi)] =S(ψi) (21.159)
H(ψi) = diag(g (ψi))−g(ψi)g(ψi)T(21.160)
wheregandHare the gradient and Hessian of lse, and ψi∈RMis chosen such that equality
holds. An upper bound to lse can be found by replacing the Hessian matrix H(ψi)with a
matrixAisuch that Ai≺H(ψi). (Bohning 1992) showed that this can be achieved if we use
the matrix Ai=1
2[
IM−1
M+11M1T
M]
. (Recall that M+1=Cis the number of classes.)
Note that Aiis independent of ψi; however, we still write it as Ai(rather than dropping the

2[
IM−1
M+11M1T
M]
isubscript), since other bounds that we consider below will have a data-dependent curvature
term. The upper bound on lse therefore becomes
lse(ηi)≤1
2ηT
iAiηi−bTiηi+ci (21.161)
Ai=1
2[
IM−1
M+11M1TM]
(21.162)
bi=Aiψi−g(ψi) (21.163)
ci=1
2ψT
iAiψi−g(ψi)Tψi+lse(ψi) (21.164)
whereψi∈RMis a vector of variational parameters.
We can use the above result to get the following lower bound on the softmax likelihood:
logp(yi=c|xi,w)≥[
yT
iXiw−1
2wTXiAiXiw+bT
iXiw−ci]
c(21.165)
To simplify notation, deﬁne the pseudo-measurement
˜yi≜A−1
i(bi+yi) (21.166)

lse(ηi)≤1
2ηT
Ai=1
2[
IM−1
M+11M1TM]
(21.162)
ci=1
2ψT
yT
iXiw−1
iXiw−ci]
c(21.165)
˜yi≜A−1
Then we can get a “Gaussianized” version of the observation model:
p(yi|xi,w)≥f(xi,ψi)N(˜yi|Xiw,A−1
i) (21.167)
wheref(xi,ψi)is some function that does not depend on w. Given this, it is easy to compute
the posterior q(w)=N(mN,VN), using Bayes rule for Gaussians. Below we will explain how
to update the variational parameters ψi.

21.8. Localvariationalbounds* 759
21.8.2.1 Applying Bohning’s bound to multinomial logistic regression
Let us see how to apply this bound to multinomial logistic regression. From Equation 21.13, we
can deﬁne the goal of variational inference as maximizing
L(q)≜−KL(q(w)||p(w|D))+Eq[N∑
i=1logp(yi|xi,w)]
(21.168)
=−KL(q(w)||p(w|D))+Eq[N∑
i=1yT
(21.169)
=−KL(q(w)||p(w|D))+N∑
i=1yT
iEq[ηi]−N∑
i=1Eq[lse(ηi)] (21.170)
whereq(w)=N(w|mN,VN)is the approximate posterior. The ﬁrst term is just the KL
divergence between two Gaussians, which is given by
−KL(N(m0,V0)||N(mN,VN)) =−1
2[
tr(VNV−1
0)−log|VNV−1
0|
+(mN−m0)TV−1
0(mN−m0)−DM]
(21.171)
whereDMis the dimensionality of the Gaussian, and we assume a prior of the form p(w)=
N(m0,V0), where typically μ0=0DM, andV0is block diagonal. The second term is simply
N∑
i=1yT
iEq[ηi]=N∑
i=1yT
i˜mi (21.172)
where˜mi≜XimN. The ﬁnal term can be lower bounded by taking expectations of our
quadratic upper bound on lse as follows:
−N∑
i=1Eq[lse(ηi)]≥−1
2tr(Ai˜Vi)−1

2[
tr(VNV−1
0|
(21.171)
N∑
i=1yT
i=1yT
−N∑
2˜miAi˜mi+bT
i˜mi−ci (21.173)
where˜Vi≜XiVNXT
i. Putting it altogether, we have
LQJ(q)≥−1
2[
tr(VNV−1
0|+(mN−m0)TV−1
0(mN−m0)]
−1
2DM+N∑
i=1yT
i˜mi−1
i˜mi−ci(21.174)
This lower bound combines Jensen’s inequality (as in mean ﬁeld inference), plus the quadratic
lower bound due to the lse term, so we write it as LQJ.
We will use coordinate ascent to optimize this lower bound. That is, we update the variational

−N∑
LQJ(q)≥−1
2[
tr(VNV−1
0(mN−m0)]
−1
2DM+N∑
i=1yT
i˜mi−1
posterior parameters VNandmN, and then the variational likelihood parameters ψi. We leave

760 Chapter21. Variationalinference
the detailed derivation as an exercise, and just state the results. We have
VN=(
V0+N∑
i=1XT
iAiXi)−1
(21.175)
mN=Vn(
V−1
0m0+N∑
i=1XTi(yi+bi))
(21.176)
ψi=˜mi=XimN (21.177)
We can exploit the fact that Aiis a constant matrix, plus the fact that Xihas block structure,
to simplify the ﬁrst two terms as follows:
VN=(
V0+A⊗N∑
i=1xixTi)−1
(21.178)
mN=Vn(
V−1
0m0+N∑
i=1(yi+bi)⊗xi)
(21.179)
where⊗denotes the kronecker product. See Algorithm 15 for some pseudocode, and http:
//www.cs.ubc.ca/~emtiyaz/software/catLGM .htmlfor some Matlab code.
Algorithm 21.1: Variational inference for multi-class logistic regression using Bohning’s
bound
1Input:yi∈{1,...,C},xi∈RD,i=1:N, priorm0,V0;
2DeﬁneM:=C−1; dummy encode yi∈{0,1}M; deﬁneXi=blockdiag( xT
i);
3Deﬁney:= [y1;...;yN],X:= [X1;...;XN]andA:=1
2[
IM−1
M+11M1T
M]
;
4VN:=(
V−1
0+∑n
i=1XT
iAXi)−1;
5InitializemN:=m0;
6repeat
7ψ:=XmN;
8Ψ:=reshape(m,M,N );
9G:= exp(Ψ−lse(Ψ));
10B:=AΨ−G;
11b:= (B);
12mN:=VN(
V−1
0m0+XT(y+b))
;

bound
i);
2[
IM−1
M+11M1T
M]
;
4VN:=(
V−1
0+∑n
i=1XT
iAXi)−1;
6repeat
7ψ:=XmN;
11b:= (B);
12mN:=VN(
V−1
;
13Compute the lower bound LQJusing Equation 21.174;
14untilconverged ;
15ReturnmNandVN;
21.8.3 Bounds for the sigmoid function
In many models, we just have binary data. In this case, we have yi∈{0,1},M=1and
ηi=wTxiwherew∈RDis a weight vector (not matrix). In this case, the Bohning bound

21.8. Localvariationalbounds* 761
−6 −4 −2 0 2 4 600.10.20.30.40.50.60.70.80.91Bohning bound, χ=−2.5
(a)−6 −4 −2 0 2 4 600.10.20.30.40.50.60.70.80.91JJ bound, χ=2.5
(b)
Figure 21.10 Quadratic lower bounds on the sigmoid (logistic) function. In solid red, we plot sigm(x) vs
x. In dotted blue, we plot the lower bound L(x,ξ)vsxforξ=2.5. (a) Bohning bound. This is tight at
−ξ=2.5. (b) JJ bound. This is tight at ξ=±2.5. Figure generated by sigmoidLowerBounds .
becomes
log(1+eη)≤1
2aη2−bη+c (21.180)
a=1
4(21.181)
b=Aψ−(1+e−ψ)−1(21.182)
c=1
2Aψ2−(1+e−ψ)−1ψ+log(1+ eψ) (21.183)
It is possible to derive an alternative quadratic bound for this case, as shown in (Jaakkola and
Jordan 1996b, 2000). This has the following form
log(1+eη)≤λ(ξ)(η2−ξ2)+1
2(η−ξ)+log(1+eξ) (21.184)
λ(ξ)≜1
4ξtanh(ξ/2) =1
2ξ[
sigm(ξ)−1
2]
(21.185)
We shall refer to this as the JJ bound, after its inventors, (Jaakkola and Jordan 1996b, 2000).
To facilitate comparison with Bohning’s bound, let us rewrite the JJ bound as a quadratic form
as follows

c=1
λ(ξ)≜1
2ξ[
sigm(ξ)−1
2]
(21.185)
as follows
2a(ξ)η2−b(ξ)η+c(ξ) (21.186)
a(ξ)=2λ(ξ) (21.187)
b(ξ)=−1
2(21.188)
c(ξ)=−λ( ξ)ξ2−1
2ξ+log(1+ eξ) (21.189)
The JJ bound has an adaptive curvature term, since adepends on ξ. In addition, it is tight at
two points, as is evident from Figure 21.10(b). By contrast, the Bohning bound is a constantcurvature bound, and is only tight at one point, as is evident from Figure 21.10(a).

762 Chapter21. Variationalinference
If we wish to use the JJ bound for binary logistic regression, we can make some small
modiﬁcations to Algorithm 15. First, we use the new deﬁnitions for ai,biandci. The fact that
aiis not constant when using the JJ bound, unlike when using the Bohning bound, means we
cannot compute VNoutside of the main loop, making the method a constant factor slower.
Next we note that Xi=xT
i, so the updates for the posterior become
V−1
N=V−1
0+2N∑
i=1λ(ξi)xixT
i (21.190)
mN=VN(
V−1
0m0+N∑
i=1(yi−1
2)xi)
(21.191)
Finally, to compute the update for ξi, we isolate the terms in LQJthat depend on ξito get
L(ξ)=N∑
i=1{
lnsigm(ξ i)−ξi/2−λ(ξi)(xTiEq[
wwT]
xi−ξ2
i)}
+const (21.192)
Optimizing this wrt ξigives the equation
0=λ′(ξi)(xTiEq[
wwT]
xi−ξ2
i) (21.193)
Nowλ′(ξi)is monotonic for ξi≥0, and we do not need to consider negative values of ξiby
symmetry of the bound around ξi=0(see Figure 21.10). Hence the only way to make the above
expression 0 is if we have (xT
iE[
wwT]
xi−ξ2

mN=VN(
V−1
0m0+N∑
i=1(yi−1
2)xi)
(21.191)
L(ξ)=N∑
i=1{
wwT]
xi−ξ2
i)}
wwT]
xi−ξ2
iE[
wwT]
xi−ξ2
i)=0. Hence the update becomes
(ξnew
i)2=xT
i(VN+mNmTN)xi (21.194)
Although the JJ bound is tighter than the Bohning bound, sometimes it is not tight enough
in order to estimate the posterior covariance accurately. A more accurate approach, which uses
a piecewise quadratic upper bound to lse, is described in (Marlin et al. 2011). By increasing thenumber of pieces, the bound can be made arbitrarily tight.
21.8.4 Other bounds and approximations to the log-sum-exp function *

iE[
wwT]
xi−ξ2
(ξnew
i)2=xT
There are several other bounds and approximations to the multiclass lse function which wecan use, which we brieﬂy summarize below. Note, however, that all of these require numericaloptimization methods to compute m
NandVN, making them more complicated to implement.
21.8.4.1 Product of sigmoids
The approach in (Bouchard 2007) exploits the fact that
log(K∑
k=1eηk)
≤α+K∑
k=1log(1+eηk−α) (21.195)
It then applies the JJ bound to the term on the right.

21.8. Localvariationalbounds* 763
21.8.4.2 Jensen’s inequality
The approach in (Blei and Lafferty 2006a, 2007) uses Jensen’s inequality as follows:
Eq[lse(ηi)] = Eq[
log(
1+M∑
c=1exp(xT
iwc))]
(21.196)
≤log(
1+M∑
c=1Eq[
exp(xTiwc)])
(21.197)
≤log(
1+M∑
c=1exp(xTimN,c+1
2xTiVN,ccxi))
(21.198)
where the last term follows from the mean of a log-normal distribution, which is eμ+σ2/2.
21.8.4.3 Multivariate delta method
The approach in (Ahmed and Xing 2007; Braun and McAuliffe 2010) uses the multivariate delta
method, which is a way to approximate moments of a function using a Taylor series expansion.
In more detail, let f(w)be the function of interest. Using a second-order approximation around
mwe have
f(w)≈f(m)+(w−m)Tg(w−m)+1
2(w−m)TH(w−m) (21.199)
wheregandHare the gradient and Hessian evaluated at m.I fq(w)=N(w|m,V),w eh a v e
Eq[f(w)]≈f(m)+1
2tr[HV] (21.200)
If we use f(w)=l s e (Xiw),w eg e t
Eq[lse(X iw)]≈lse(Xim)+1
2tr[XiHXT
iV] (21.201)

mwe have
2tr[XiHXT
wheregandHfor the lse function are deﬁned in Equations 21.159 and 21.160.
21.8.5 Variational inference based on upper bounds
So far, we have been concentrating on lower bounds. However, sometimes we need to use an

mwe have
2tr[XiHXT
upper bound. For example, (Saul et al. 1996) derives a mean ﬁeld algorithm for sigmoid beliefnets, which are DGMs in which each CPD is a logistic regression function (Neal 1992). Unlike thecase of Ising models, the resulting MRF is not pairwise, but contains higher order interactions.This makes the standard mean ﬁeld updates intractable. In particular, they turn out to involvecomputing an expression which requires evaluating
E[
log(1+e
−∑
j∈paiwijxj)]
=E[
−logsigm(wT
ixpa(i))]
(21.202)

E[
log(1+e
−∑
=E[
ixpa(i))]
(21.202)
(Notice the minus sign in front.) (Saul et al. 1996) show how to derive an upper bound on thesigmoid function so as to make this update tractable, resulting in a monotonically convergentinference procedure.

764 Chapter21. Variationalinference
Exercises
Exercise 21.1 Laplace approximation to p(μ,logσ|D)for a univariate Gaussian
ComputeaLaplaceapproximationof p(μ,logσ|D)foraGaussian, usinganuninformativeprior p(μ,logσ)∝
1.
Exercise 21.2 Laplace approximation to normal-gamma
Consider estimating μandℓ=l o gσfor a Gaussian using an uniformative normal-Gamma prior. The log
logp(μ,ℓ|D)=−nlogσ−1
2σ2[ns2+n(y−μ)2] (21.203)
a. Show that the ﬁrst derivatives are
∂
∂μlogp(μ,ℓ|D)=n(y−μ)
σ2(21.204)
∂
∂ℓlogp(μ,ℓ|D)=−n +ns2+n(y−μ)2
σ2(21.205)
b. Show that the Hessian matrix is given by
H=(
∂2
∂μ2logp(μ,ℓ|D)∂2
∂μ∂ℓlogp(μ,ℓ|D)
∂2
∂ℓ2logp(μ,ℓ|D)∂2
∂ℓ2logp(μ,ℓ|D))
(21.206)
=(−n
σ2 −2ny−μ
σ2
−2ny−μ
σ2−2
σ2(ns2+n(y−μ)2))
(21.207)
c. Use this to derive a Laplace approximation to the posterior p(μ,ℓ|D).
Exercise 21.3 Variational lower bound for VB for univariate Gaussian
Fill in the details of the derivation in Section 21.5.1.6.
Exercise 21.4 Variational lower bound for VB for GMMs

∂
H=(
∂2
∂2
(21.206)
=(−n
σ2 −2ny−μ
σ2
−2ny−μ
σ2−2
(21.207)
Consider VBEM for GMMs as in Section 21.6.1.4. Show that the lower bound has the following form
L=E[lnp(x|z,μ,Λ)]+ E[lnp(z|π)]+E[lnp(π)]+E[lnp(μ,Λ)]
−E[lnq(z)]−E[lnq(π)]−E[lnq(μ,Λ)] (21.208)

21.8. Localvariationalbounds* 765
where
E[lnp(x|z,μ,Λ)] =1
2∑
kNk{
ln˜Λk−Dβ−1
k−νktr(SkLk)
−νk(xk−mk)TLk(xk−mk)−Dln(2π)}
(21.209)
E[lnp(z|π)] =∑
i∑
krikln˜πk (21.210)
E[lnp(π)] = ln Cdir(α0)+(α0−1)∑
kln˜πk (21.211)
E[lnp(μ,Λ)] =1
2∑
k{
Dln(β0/2π)+ln˜Λk−Dβ0
βk
−β0νk(mk−m0)TLk(mk−m0)
+lnCWi(L0,ν0)+ν0−D−1
2ln˜Λk−1
2νktr(L−1
0Lk)}
(21.212)
E[lnq(z)] =∑
i∑
kriklnrik (21.213)
E[lnq(π)] =∑
k(αk−1)ln˜πk+lnCdir(α) (21.214)
E[lnq(μ,Λ)] =∑
k{1
2ln˜Λk+D
2ln(βk
2π)
−D
2−H(q(Λk))}
(21.215)
where the normalization constant for the Dirichlet and Wishart is given by
Cdir(α)≜Γ(∑
kαk)∏
kΓ(αk)(21.216)
CWi(L,ν)≜|L|−ν/2(
2νD/2ΓD(ν/2))−1
(21.217)
ΓD(α)≜πD(D−1)/4D∏
j=1Γ(α+(1−j)/2) (21.218)
whereΓD(ν)is the multivariate Gamma function. Finally, the entropy of the Wishart is given by
H(Wi(L,ν)) = −lnCWi(L,ν)−ν−D−1
2E[ln|Λ|]+νD
2(21.219)
where E[ln|Λ|]is given in Equation 21.131.
Exercise 21.5 Derivation of E[logπk]under a Dirichlet distribution
Show that
exp(E[logπk]) =exp(Ψ(α k))
exp(Ψ(∑
k′αk′))(21.220)
whereπ∼Dir(α).

kαk)∏
(21.217)
2(21.219)
Show that
exp(Ψ(∑
Exercise 21.6 Alternative derivation of the mean ﬁeld updates for the Ising model
Derive Equation 21.50 by directly optimizing the variational free energy one term at a time.

766 Chapter21. Variationalinference
Exercise 21.7 Forwards vs reverse KL divergence
(Source: Exercise 33.7 of (MacKay 2003).) Consider a factored approximation q(x,y)=q(x)q(y)to a joint
distribution p(x,y). Show that to minimize the forwards KL KL(p||q)we should set q(x)=p(x)and
q(y)=p(y), i.e., the optimal approximation is a product of marginals
Now consider the following joint distribution, where the rows represent yand the columns x.
x
1234
11/8 1/8 0 0
21/8 1/8 0 0
30 0 1/4 0
40 0 0 1/4
Show that the reverse KL KL(q||p)for thisphas three distinct minima. Identify those minima and
evaluate KL(q||p)at each of them. What is the value of KL(q||p)if we setq(x,y)=p(x)p(y)?
Exercise 21.8 Derivation of the structured mean ﬁeld updates for FHMM
Derive the updates in Section 21.4.1.
Exercise 21.9 Variational EM for binary FA with sigmoid link
Consider the binary FA model:
p(xi|zi,θ)=D∏
j=1Ber(xij|sigm(wT
jzi+βj)) =D∏
j=1Ber(xij|sigm(η ij)) (21.221)
ηi=˜W˜zi (21.222)
˜ zi≜(zi;1) (21.223)
˜W≜(W,β) (21.224)

p(zi)=N (0,I) (21.225)
Derive an EM algorithm to ﬁt this model, using the Jaakkola-Jordan bound. Hint: the answer is in (Tipping
1998), but the exercise asks you to derive these equations.
Exercise 21.10 VB for binary FA with probit link
In Section 11.4.6, we showed how to use EM to ﬁt probit regression, using a model of the form p(yi=
1|zi)=I(zi>0),w h e r e zi∼N(wTxi,1)is latent. Now consider the case where the inputs xiare
also unknown, as in binary factor analysis. Show how to ﬁt this model using variational Bayes, making an
approximation to the posterior of the form q(x,z,W)=∏N

i=1q(xi)q(zi)∏Ll=1q(wl). Hint:q(xi)and
q(wi)will be Gaussian, and q(zi)will be a truncated univariate Gaussian.

22 More variational inference
22.1 Introduction
In Chapter 21, we discussed mean ﬁeld inference, which approximates the posterior by a product
of marginal distributions. This allows us to use different parametric forms for each variable,which is particularly useful when performing Bayesian inference for the parameters of statisticalmodels (such as the mean and variance of a Gaussian or GMM, or the regression weights in aGLM), as we saw when we discussed variational Bayes and VB-EM.
In this chapter, we discuss a slightly different kind of variational inference. The basic idea is
to minimize J(q)=KL(q||˜p),w h e r e˜pis the exact but unnormalized posterior as before, but
where we no longer require qto be factorized. In fact, we do not even require qto be a globally
valid joint distribution. Instead, we only require that qis locally consistent, meaning that the
joint distribution of two adjacent nodes agrees with the corresponding marginals (we will deﬁnethis more precisely below).

In addition to this new kind of inference, we will discuss approximate methods for MAP
state estimation in discrete graphical models. It turns out that algorithms for solving the MAPproblem are very similar to some approximate methods for computing marginals, as we will see.
22.2 Loopy belief propagation: algorithmic issues
There is a very simple approximate inference algorithm for discrete (or Gaussian) graphicalmodels known as loopy belief propagation orLBP. The basic idea is extremely simple: we

apply the belief propagation algorithm of Section 20.2 to the graph, even if it has loops (i.e.,even if it is not a tree). This method is simple and efficient, and often works well in practice,outperforming mean ﬁeld (Weiss 2001). In this section, we discuss the algorithm in more detail.In the next section, we analyse this algorithm in terms of variational inference.
22.2.1 A brief history

When applied to loopy graphs, BP is not guaranteed to give correct results, and may not evenconverge. Indeed, Judea Pearl, who invented belief propagation for trees, wrote the followingabout loopy BP in 1988:
When loops are present, the network is no longer singly connected and local propagation

768 Chapter22. Morevariationalinference
schemes will invariably run into trouble ...If we ignore the existence of loops and
permit the nodes to continue communicating with each other as if the network were
singly connected, messages may circulate indeﬁnitely around the loops and the processmay not converge to a stable equilibrium ...Such oscillations do not normally occur in
probabilistic networks ...which tend to bring all messages to some stable equilibrium as
time goes on. However, this asymptotic equilibrium is not coherent, in the sense that itdoes not represent the posterior probabilities of all nodes of the network — (Pearl 1988,p.195)
Despite these reservations, Pearl advocated the use of belief propagation in loopy networks asan approximation scheme (J. Pearl, personal communication) and exercise 4.7in (Pearl 1988)
investigates the quality of the approximation when it is applied to a particular loopy beliefnetwork.

However, the main impetus behind the interest in BP arose when McEliece et al. (1998) showed

that a popular algorithm for error correcting codes known as turbo codes (Berrou et al. 1993)could be viewed as an instance of BP applied to a certain kind of graph. This was an importantobservation since turbo codes have gotten very close to the theoretical lower bound on codingefficiency proved by Shannon. (Another approach, known as low density parity check or LDPCcodes, has achieved comparable performance; it also uses LBP for decoding — see Figure 22.1for an example.) In (Murphy et al. 1999), LBP was experimentally shown to also work well forinference in other kinds of graphical models beyond the error-correcting code context, and sincethen, the method has been widely used in many different applications.
22.2.2 LBP on pairwise models

We now discuss how to apply LBP to an undirected graphical model with pairwise factors (wediscuss the directed case, which can involve higher order factors, in the next section). Themethod is simple: just continually apply Equations 20.11 and 20.10 until convergence. SeeAlgorithm 8 for the pseudocode, and beliefPropagation for some Matlab code. We will
discuss issues such as convergence and accuracy of this method shortly.
Algorithm 22.1: Loopy belief propagation for a pairwise MRF
1Input: node potentials ψs(xs), edge potentials ψst(xs,xt);
2Initialize messages ms→t(xt)=1for all edges s−t;
3Initialize beliefs bels(xs)=1for all nodes s;
4repeat
5Send message on each edge
xs(
ψs(xs)ψst(xs,xt)∏
u∈nbrs\tmu→s(xs))
;
6Update belief of each node bels(xs)∝ψs(xs)∏
t∈nbrsmt→s(xs);
7untilbeliefsdon’tchangesigniﬁcantly ;
8Return marginal beliefs bels(xs);

22.2. Loopybeliefpropagation: algorithmicissues 769
Figure 22.1 (a) A simple factor graph representation of a (2,3) low-density parity check code (factor graphs
are deﬁned in Section 22.2.3.1). Each message bit (hollow round circle) is connected to two parity factors
(solid black squares), and each parity factor is connected to three bits. Each parity factor has the formψ
stu(xs,xt,xu)=I(xs⊗xt⊗xu=1 ),w h e r e⊗is the xor operator. The local evidence factors for
each hidden node are not shown. (b) A larger example of a random LDPC code. We see that this graph is“locally tree-like”, meaning there are no short cycles; rather, each cycle has length ∼logm,w h e r e mis the
number of nodes. This gives us a hint as to why loopy BP works so well on such graphs. (Note, however,that some error correcting code graphs have short loops, so this is not the full explanation.) Source:Figure 2.9 from (Wainwright and Jordan 2008b). Used with kind permission of Martin Wainwright.
22.2.3 LBP on a factor graph

To handle models with higher-order clique potentials (which includes directed models where
some nodes have more than one parent), it is useful to use a representation known as a factorgraph. We explain this representation below, and then describe how to apply LBP to suchmodels.
22.2.3.1 Factor graphs
Afactor graph (Kschischang et al. 2001; Frey 2003) is a graphical representation that uniﬁes

directed and undirected models, and which simpliﬁes certain message passing algorithms. Moreprecisely, a factor graph is an undirected bipartite graph with two kinds of nodes. Round nodesrepresent variables, square nodes represent factors, and there is an edge from each variable toevery factor that mentions it. For example, consider the MRF in Figure 22.2(a). If we assumeone potential per maximal clique, we get the factor graph in Figure 22.2(b), which represents thefunction
f(x
1,x2,x3,x4)=f124(x1,x2,x4)f234(x2,x3,x4) (22.1)

f(x
If we assume one potential per edge. we get the factor graph in Figure 22.2(c), which representsthe function
f(x
1,x2,x3,x4)=f14(x1,x4)f12(x1,x2)f34(x3,x4)f23(x2,x3)f24(x2,x4) (22.2)

770 Chapter22. Morevariationalinference
1
2
34
(a)34 21
(b)34 21
(c)
Figure 22.2 (a) A simple UGM. (b) A factor graph representation assuming one potential per maximal
clique. (c) A factor graph representation assuming one potential per edge.
x4x5x3x1x2
(a)x4p(x4|x3)x3
p(x5|x3)
x5p(x3|x1,x2)x1 x2p(x1) p(x2)
(b)
Figure 22.3 (a) A simple DGM. (b) Its corresponding factor graph. Based on Figure 5 of (Yedidia et al.
2001)..
We can also convert a DGM to a factor graph: just create one factor per CPD, and connect that
factor to all the variables that use that CPD. For example, Figure 22.3 represents the following
factorization:
f(x1,x2,x3,x4,x5)=f1(x1)f2(x2)f123(x1,x2,x3)f34(x3,x4)f35(x3,x5) (22.3)
where we deﬁne f123(x1,x2,x3)=p(x3|x1,x2), etc. If each node has at most one parent (and
hence the graph is a chain or simple tree), then there will be one factor per edge (root nodescan have their prior CPDs absorvbed into their children’s factors). Such models are equivalentto pairwise MRFs.

22.2. Loopybeliefpropagation: algorithmicissues 771
Figure 22.4 Message passing on a bipartite factor graph. Square nodes represent factors, and circles
represent variables. Source: Figure 6 of (Kschischang et al. 2001). Used with kind permission of Brendan
Frey.
22.2.3.2 BP on a factor graph
We now derive a version of BP that sends messages on a factor graph, as proposed in (Kschis-
chang et al. 2001). Speciﬁcally, we now have two kinds of messages: variables to factors
mx→f(x)=∏
h∈nbr(x)\{f}mh→x(x) (22.4)
and factors to variables:
mf→x(x)=∑
yf(x,y)∏
y∈nbr(f)\{x}my→f(y) (22.5)
Herenbr(x)are all the factors that are connected to variable x, andnbr(f)are all the variables
that are connected to factor f. These messages are illustrated in Figure 22.4. At convergence,
we can compute the ﬁnal beliefs as a product of incoming messages:
bel(x)∝∏
f∈nbr(x)mf→x(x) (22.6)
In the following sections, we will focus on LBP for pairwise models, rather than for factor
graphs, but this is just for notational simplicity.

mf→x(x)=∑
yf(x,y)∏
bel(x)∝∏
22.2.4 Convergence
LBP does not always converge, and even when it does, it may converge to the wrong answers.
This raises several questions: how can we predict when convergence will occur? what can we do
to increase the probability of convergence? what can we do to increase the rate of convergence?
We brieﬂy discuss these issues below. We then discuss the issue of accuracy of the results at
convergence.

772 Chapter22. Morevariationalinference
Synchronous Asynchronous No smoothing True01.0
0.90.80.70.60.50.40.30.20.1
0
10 20 30 40 50 60 70 80 90 100
Time (seconds)% of messages converged1.00.90.80.70.60.50.40.30.20.1
0
Time (seconds)P(X10=0 )1.00.90.80.70.60.50.40.30.20.1
0
Time (seconds)P(X115=0 )
1.00.90.80.70.60.50.40.30.20.1
0
Time (seconds)P(X61=0 )1.00.90.80.70.60.50.40.30.20.1
0
Time (seconds)P(X17=0 )1.00.90.80.70.60.50.40.30.20.1
0
Time (seconds)P(X7=0 )0 0.1 0.2 0.3 0.4 0.5
0 0.1 0.2 0.3 0.4 0.5 0 0.1 0.2 0.3 0.4 0.5 0 0.1 0.2 0.3 0.4 0.5(a)
(d)(b)
(e)(c)
(f)
Figure 22.5 Illustration of the behavior of loopy belief propagation on an 11×11Ising grid with
random potentials, wij∼Unif(−C,C ),w h e r eC=1 1. For larger C, inference becomes harder. (a)
Percentage of messasges that have converged vs time for 3 different update schedules: Dotted = damped

0
0
(d)(b)
(e)(c)
(f)
sychronous (few nodes converge), dashed = undamped asychnronous (half the nodes converge), solid =damped asychnronous (all nodes converge). (b-f) Marginal beliefs of certain nodes vs time. Solid straightline = truth, dashed = sychronous, solid = damped asychronous. Source: Figure 11.C.1 of (Koller andFriedman 2009). Used with kind permission of Daphne Koller.
22.2.4.1 When will LBP converge?
The details of the analysis of when LBP will converge are beyond the scope of this chapter, but

we brieﬂy sketch the basic idea. The key analysis tool is the computation tree, which visualizes
the messages that are passed as the algorithm proceeds. Figure 22.6 gives a simple example.In the ﬁrst iteration, node 1 receives messages from nodes 2 and 3. In the second iteration, itreceives one message from node 3 (via node 2), one from node 2 (via node 3), and two messagesfrom node 4 (via nodes 2 and 3). And so on.
The key insight is that Titerations of LBP is equivalent to exact computation in a computation

tree of height T+1. If the strengths of the connections on the edges is sufficiently weak, then
the inﬂuence of the leaves on the root will diminish over time, and convergence will occur. See(Wainwright and Jordan 2008b) and references therein for more information.

22.2. Loopybeliefpropagation: algorithmicissues 773
Figure 22.6 (a) A simple loopy graph. (b) The computation tree, rooted at node 1, after 4 rounds of
message passing. Nodes 2 and 3 occur more often in the tree because they have higher degree than nodes
1 and 2. Source: Figure 8.2 of (Wainwright and Jordan 2008b). Used with kind permission of MartinWainwright.
22.2.4.2 Making LBP converge
Although the theoretical convergence analysis is very interesting, in practice, when faced with a
model where LBP is not converging, what should we do?
One simple way to reduce the chance of oscillation is to use damping. That is, instead of
sending the message Mk
ts, we send a damped message of the form
˜Mk
ts(xs)=λMts(xs)+(1−λ)˜Mk−1
ts(xs) (22.7)
where0≤λ≤1is the damping factor Clearly if λ=1this reduces to the standard scheme,
but forλ<1, this partial updating scheme can help improve convergence. Using a value such
asλ∼0.5is standard practice. The beneﬁts of this approach are shown in Figure 22.5, where

˜Mk
we see that damped updating results in convergence much more often than undamped updating.
It is possible to devise methods, known as double loop algorithms, which are guaranteed to

˜Mk
converge to a local minimum of the same objective that LBP is minimizing (Yuille 2001; Wellingand Teh 2001). Unfortunately, these methods are rather slow and complicated, and the accuracyof the resulting marginals is usually not much greater than with standard LBP. (Indeed, oscillatingmarginals is sometimes a sign that the LBP approximation itself is a poor one.) Consequently,these techniques are not very widely used. In Section 22.4.2, we will see a different convergentversion of BP that is widely used.

22.2.4.3 Increasing the convergence rate: message scheduling
Even if LBP converges, it may take a long time. The standard approach when implementingLBP is to perform synchronous updates, where all nodes absorb messages in parallel, and then
send out messages in parallel. That is, the new messages at iteration k+1are computed in
parallel using
m
k+1=(f1(mk),...,f E(mk)) (22.8)
whereEis the number of edges, and fst(m)is the function that computes the message for

m
edges→tgiven all the old messages. This is analogous to the Jacobi method for solving linear

774 Chapter22. Morevariationalinference
systems of equations. It is well known (Bertsekas 1997) that the Gauss-Seidel method, which
performs asynchronous updates in a ﬁxed round-robin fashion, converges faster when solving
linear systems of equations. We can apply the same idea to LBP, using updates of the form
mk+1
i=fi(
{mk+1
j:j<i},{mk
j:j>i})
(22.9)
where the message for edge iis computed using new messages (iteration k+1) from edges
earlier in the ordering, and using old messages (iteration k) from edges later in the ordering.
This raises the question of what order to update the messages in. One simple idea is to use
a ﬁxed or random order. The beneﬁts of this approach are shown in Figure 22.5, where we see
that (damped) asynchronous updating results in convergence much more often than synchronousupdating.
A smarter approach is to pick a set of spanning trees, and then to perform an up-down
sweep on one tree at a time, keeping all the other messages ﬁxed. This is known as tree

reparameterization (TRP) (Wainwright et al. 2001), which should not be confused with the more
sophisticated tree-reweighted BP (often abbreviated to TRW) to be discussed in Section 22.4.2.1.
However, we can do even better by using an adaptive ordering. The intuition is that we should
focus our computational efforts on those variables that are most uncertain. (Elidan et al. 2006)proposed a technique known as residual belief propagation, in which messages are scheduled

to be sent according to the norm of the difference from their previous value. That is, we deﬁnethe residual of new message m
stat iteration kto be
r(s,t,k)=||logmst−logmk
st||∞=m a x
i|logmst(i)
mk
st(i)| (22.10)
We can store messages in a priority queue, and always send the one with highest residual. When
a message is sent from stot, all of the other messages that depend on mst(i.e., messages of
the form mtuwhereu∈nbr(t)\s) need to be recomputed; their residual is recomputed, and

st||∞=m a x
mk
they are added back to the queue. In (Elidan et al. 2006), they showed (experimentally) that thismethod converges more often, and much faster, than using sychronous updating, asynchronousupdating with a ﬁxed order, and the TRP approach.
A reﬁnement of residual BP was presented in (Sutton and McCallum 2007). In this paper, they

use an upper bound on the residual of a message instead of the actual residual. This meansthat messages are only computed if they are going to be sent; they are not just computed forthe purposes of evaluating the residual. This was observed to be about ﬁve times faster thanresidual BP, although the quality of the ﬁnal results is similar.
22.2.5 Accuracy of LBP

For a graph with a single loop, one can show that the max-product version of LBP will ﬁnd thecorrect MAP estimate, if it converges (Weiss 2000). For more general graphs, one can boundthe error in the approximate marginals computed by LBP, as shown in (Wainwright et al. 2003;Vinyals et al. 2010). Much stronger results are available in the case of Gaussian models (Weissand Freeman 2001a; Johnson et al. 2006; Bickson 2009). In particular, in the Gaussian case, ifthe method converges, the means are exact, although the variances are not (typically the beliefsare over conﬁdent).

22.2. Loopybeliefpropagation: algorithmicissues 775
22.2.6 Other speedup tricks for LBP *
There are several tricks one can use to make BP run faster. We discuss some of them below.
22.2.6.1 Fast message computation for large state spaces
The cost of computing each message in BP (whether in a tree or a loopy graph) is O(Kf),
whereKis the number of states, and fis the size of the largest factor (f =2for pairwise
UGMs). In many vision problems (e.g., image denoising), Kis quite large (say 256), because
it represents the discretization of some underlying continuous space, so O(K2)per message
is too expensive. Fortunately, for certain kinds of pairwise potential functions of the form
ψst(xs,xt)=ψ(xs−xt), one can compute the sum-product messages in O(KlogK)time
using the fast Fourier transform or FFT, as explained in (Felzenszwalb and Huttenlocher 2006).The key insight is that message computation is just convolution:
M
k
st(xt)=∑
xsψ(xs−xt)h(xs) (22.11)
whereh(xs)=ψs(xs)∏
v∈nbr(s)\tMk−1

M
k
st(xt)=∑
vs(xs). If the potential function ψ(z)is a Gaussian-like
potential, we can compute the convolution in O(K)time by sequentially convolving with a
small number of box ﬁlters (Felzenszwalb and Huttenlocher 2006).
For the max-product case, a technique called the distance transform can be used to compute
messages in O(K)time. However, this only works if ψ(z)=e x p (−E(z))and where E(z)
has one the following forms: quadratic, E(z)=z2; truncated linear, E(z)=m i n ( c1|z|,c2);o r

Potts model, E(z)=cI(z̸=0 ). See (Felzenszwalb and Huttenlocher 2006) for details.
22.2.6.2 Multi-scale methods

A method which is speciﬁc to 2d lattice structures, which commonly arise in computer vision,is based on multi-grid techniques. Such methods are widely used in numerical linear algebra,where one of the core problems is the fast solution of linear systems of equations; this isequivalent to MAP estimation in a Gaussian MRF. In the computer vision context, (Felzenszwalband Huttenlocher 2006) suggested using the following heuristic to signiﬁcantly speedup BP:construct a coarse-to-ﬁne grid, compute messages at the coarse level, and use this to initializemessages at the level below; when we reach the bottom level, just a few iterations of standard BPare required, since long-range communication has already been achieved via the initializationprocess.

The beliefs at the coarse level are computed over a small number of large blocks. The local
evidence is computed from the average log-probability each possible block label assigns to allthe pixels in the block. The pairwise potential is based on the discrepancy between labels ofneighboring blocks, taking into account their size. We can then run LBP at the coarse level,and then use this to initialize the messages one level down. Note that the modelis still a
ﬂat grid; however, the initialization process exploits the multi-scale nature of the problem. See
(Felzenszwalb and Huttenlocher 2006) for details.

776 Chapter22. Morevariationalinference
22.2.6.3 Cascades
Another trick for handling high-dimensional state-spaces, that can also be used with exact
inference (e.g., for chain-structured CRFs), is to prune out improbable states based on a com-putationally cheap ﬁltering step. In fact, one can create a hierarchy of models which tradeoffspeed and accuracy. This is called a computational cascade. In the case of chains, one can
guarantee that the cascade will never ﬁlter out the true MAP solution (Weiss et al. 2010).
22.3 Loopy belief propagation: theoretical issues *
We now attempt to understand the LBP algorithm from a variational point of view. Our presen-tation is closely based on an excellent 300-page review article (Wainwright and Jordan 2008a).This paper is sometimes called “the monster” (by its own authors!) in view of its length andtechnical difficulty. This section just sketches some of the main results.
To simplify the presentation, we focus on the special case of pairwise UGMs with discrete

variables and tabular potentials. Many of the results generalize to UGMs with higher-order cliquepotentials (which includes DGMs), but this makes the notation more complex (see (Koller andFriedman 2009) for details of the general case).
22.3.1 UGMs represented in exponential family form
We assume the distribution has the following form:
p(x|θ,G)=1
Z(θ)exp⎧
⎨
⎩∑
s∈Vθs(xs)+∑
(s,t)∈Eθst(xs,xt)⎫
⎬
⎭(22.12)
where graph Ghas nodesVand edgesE. (Henceforth we will drop the explicit conditioning

Z(θ)exp⎧
⎨
⎩∑
⎬
⎭(22.12)
onθandGfor brevity, since we assume both are known and ﬁxed.) We can rewrite this in
exponential family form as follows:
p(x|θ)=1
Z(θ)exp(−E(x)) (22.13)
E(x)≜−θTφ(x) (22.14)
whereθ=({θs;j},{θs,t;j,k})are all the node and edge parameters (the canonical parameters),
andφ(x)=({I(xs=j)},{I(xs=j,xt=k)})are all the node and edge indicator functions
(the sufficient statistics). Note: we use s,t∈Vto index nodes and j,k∈Xto index states.
The mean of the sufficient statistics are known as the mean parameters of the model, and are

p(x|θ)=1
given by
μ=E[φ(x)] = ({p( xs=j)}s,{p(xs=j,xt=k)}s̸=t)=({μs;j}s,{μst;jk}s̸=t)(22.15)
This is a vector of length d=|X||V|+|X|2|E|, containing the node and edge marginals.
It completely characterizes the distribution p(x|θ), so we sometimes treat μas a distribution
itself.
Equation 22.12 is called the standard overcomplete representation. It is called “overcom-
plete” because it ignores the sum-to-one constraints. In some cases, it is convenient to remove

22.3. Loopybeliefpropagation: theoreticalissues* 777
this redundancy. For example, consider an Ising model where Xs∈{0,1}. The model can be
written as
p(x)=1
Z(θ)exp⎧
⎨
⎩∑
s∈Vθsxs+∑
(s,t)∈Eθstxsxt⎫
⎬
⎭(22.16)
Hence we can use the following minimal parameterization
φ(x)=(xs,s∈V;xsxt,(s,t)∈E)∈Rd(22.17)
whered=|V|+|E|. The corresponding mean parameters are μs=p(xs=1 )andμst=
p(xs=1,xt=1 ).
22.3.2 The marginal polytope
The space of allowable μvectors is called the marginal polytope, and is denoted M(G),w h e r e
Gis the structure of the graph deﬁning the UGM. This is deﬁned to be the set of all mean
parameters for the given model that can be generated from a valid probability distribution:
M(G)≜{μ∈Rd:∃ps.t.μ=∑
xφ(x)p(x)for some p(x)≥0,∑
xp(x)=1}(22.18)
For example, consider an Ising model. If we have just two nodes connected as X1−X2,
one can show that we have the following minimal set of constraints: 0≤μ12,0≤μ12≤μ1,
0≤μ12≤μ2, and1+μ12−μ1−μ2≥0. We can write these in matrix-vector form as
⎛
⎜⎜⎝001
10−1
01−1
−1−11⎞

⎛
⎜⎜⎝001
10−1
01−1
−1−11⎞
⎟⎟⎠⎛
⎝μ1
μ2
μ12⎞⎠≥⎛
⎜⎜⎝0
00
−1⎞
⎟⎟⎠(22.19)
These four constraints deﬁne a series of half-planes, whose intersection deﬁnes a polytope,
as shown in Figure 22.7(a).
Since M(G)is obtained by taking a convex combination of the φ(x)vectors, it can also be
written as the convex hull of the feature set:
M(G)=conv{φ
1(x),...,φ d(x)} (22.20)
For example, for a 2 node MRF X1−X2with binary states, we have
M(G)=conv{(0,0,0),(1,0,0),(0,1,0),(1,1,1)} (22.21)

⎛
⎜⎜⎝001
10−1
01−1
−1−11⎞
⎟⎟⎠⎛
⎝μ1
μ2
μ12⎞⎠≥⎛
⎜⎜⎝0
00
−1⎞
These are the four black dots in Figure 22.7(a). We see that the convex hull deﬁnes the same
volume as the intersection of half-spaces.
The marginal polytope will play a crucial role in the approximate inference algorithms we
discuss in the rest of this chapter.

778 Chapter22. Morevariationalinference
(a) (b) (c)
Figure 22.7 (a) Illustration of the marginal polytope for an Ising model with two variables. (b) Cartoon
illustration of the set MF(G), which is a nonconvex inner bound on the marginal polytope M(G).MF(G)
is used by mean ﬁeld. (c) Cartoon illustration of the relationship between M(G)andL(G), which is used
by loopy BP. The set L(G)is always an outer bound on M(G), and the inclusion M(G)⊂L(G)is strict
whenever Ghas loops. Both sets are polytopes, which can be deﬁned as an intersection of half-planes
(deﬁned by facets), or as the convex hull of the vertices. L(G)actually has fewer facets than M(G), despite
the picture. In fact, L(G)hasO(|X||V|+|X|2|E|)facets, where |X|is the number of states per variable,
|V|is the number of variables, and |E|is the number of edges. By contrast, M(G)hasO(|X||V|)facets.
On the other hand, L(G)has more vertices than M(G), despite the picture, since L(G)contains all the

binary vector extreme points μ∈M(G), plus additional fractional extreme points. Source: Figures 3.6,
5.4 and 4.2 of (Wainwright and Jordan 2008a). Used with kind permission of Martin Wainwright.
22.3.3 Exact inference as a variational optimization problem
Recall from Section 21.2 that the goal of variational inference is to ﬁnd the distribution qthat
maximizes the energy functional
L(q)=−KL(q||p)+log Z=Eq[log ˜p(x)]+H(q)≤logZ (22.22)

where˜p(x)=Zp(x)is the unnormalized posterior. If we write log ˜p(x)=θTφ(x), and we
letq=p, then the exact energy functional becomes
max
μ∈M(G)θTμ+H(μ) (22.23)
whereμ=Ep[φ(x)]is a joint distribution over all state conﬁgurations x(so it is valid to write
H(μ)). Since the KL divergence is zero when p=q, we know that
max
μ∈M(G)θTμ+H(μ)=l o gZ(θ) (22.24)
This is a way to cast exact inference as a variational optimization problem.
Equation 22.24 seems easy to optimize: the objective is concave, since it is the sum of a linear

max
max
function and a concave function (see Figure 2.21 to see why entropy is concave); furthermore, we
are maximizing this over a convex set. However, the marginal polytope M(G)has exponentially
many facets. In some cases, there is structure to this polytope that can be exploited by dynamicprogramming (as we saw in Chapter 20), but in general, exact inference takes exponential time.Most of the existing deterministic approximate inference schemes that have been proposed inthe literature can be seen as different approximations to the marginal polytope, as we explainbelow.

22.3. Loopybeliefpropagation: theoreticalissues* 779
22.3.4 Mean ﬁeld as a variational optimization problem
We discussed mean ﬁeld at length in Chapter 21. Let us re-interpret mean ﬁeld inference in
our new more abstract framework. This will help us compare it to other approximate methodswhich we discuss below.
First, letFbe an edge subgraph of the original graph G, and letI(F)⊆Ibe the subset of
sufficient statistics associated with the cliques of F.L e tΩbe the set of canonical parameters
for the full model, and deﬁne the canonical parameter space for the submodel as follows:
Ω(F)≜{θ∈Ω:θ
α=0∀α∈I\I(F)} (22.25)
In other words, we require that the natural parameters associated with the sufficient statisticsαoutside of our chosen class to be zero. For example, in the case of a fully factorized
approximation, F
0, we remove all edges from the graph, giving
Ω(F0)≜{θ∈Ω:θst=0∀(s,t)∈E} (22.26)
In the case of structured mean ﬁeld (Section 21.4), we set θst=0for edges which are not in
our tractable subgraph.

Next, we deﬁne the mean parameter space of the restricted model as follows:
MF(G)≜{μ∈Rd:μ=Eθ[φ(x)]for some θ∈Ω(F)} (22.27)
This is called an inner approximation to the marginal polytope, since MF(G)⊆M(G). See
Figure 22.7(b) for a sketch. Note that MF(G)is a non-convex polytope, which results in multiple
local optima. By contrast, some of the approximations we will consider later will be convex.
We deﬁne the entropy of our approximation H(μ(F))as the entropy of the distribution
μdeﬁned on submodel F. Then we deﬁne the mean ﬁeld energy functional optimization
problem as follows:
max

max
μ∈MF(G)θTμ+H(μ)≤logZ(θ) (22.28)
In the case of the fully factorized mean ﬁeld approximation for pairwise UGMs, we can writethis objective as follows:
max
μ∈Pd∑
s∈V∑
xsθs(xs)μs(xs)+∑
(s,t)∈E∑
xs,xtθst(xs,xt)μs(xs)μt(xt)+∑
s∈VH(μs)(22.29)
whereμs∈P, andPis the probability simplex over X.
Mean ﬁeld involves a concave objective being maximized over a non-convex set. It is typically
optimized using coordinate ascent, since it is easy to optimize a scalar concave function over P

max
max
μ∈Pd∑
s∈V∑
(s,t)∈E∑
for eachμs. For example, for a pairwise UGM we get
μs(xs)∝exp(θs(xs))exp⎛
⎝∑
t∈nbr(s)∑
xtμt(xt)θst(xs,xt)⎞⎠ (22.30)
22.3.5 LBP as a variational optimization problem
In this section, we explain how LBP can be viewed as a variational inference problem.

780 Chapter22. Morevariationalinference
Figure 22.8 (a) Illustration of pairwise UGM on binary nodes, together with a set of pseudo marginals
that are not globally consistent. (b) A slice of the marginal polytope illustrating the set of feasible edge
marginals, assuming the node marginals are clamped at μ1=μ2=μ3=0.5. Source: Figure 4.1 of
(Wainwright and Jordan 2008a). Used with kind permission of Martin Wainwright.
22.3.5.1 An outer approximation to the marginal polytope
If we want to consider all possible probability distributions which are Markov wrt our model, we
need to consider all vectors μ∈M(G). Since the set M(G)is exponentially large, it is usually
infeasible to optimize over. A standard strategy in combinatorial optimization is to relax the
constraints. In this case, instead of requiring probability vector μto live in M(G), we consider
a vectorτthat only satisﬁes the following local consistency constraints:
∑
xsτs(xs)=1 (22.31)
∑
xtτst(xs,xt)=τs(xs) (22.32)

∑
∑
The ﬁrst constraint is called the normalization constraint, and the second is called the marginal-
ization constraint. We then deﬁne the set
L(G)≜{τ≥0:( 2 2.31)holds∀s∈Vand(22.32)holds∀(s,t)∈E} (22.33)
The set L(G)is also a polytope, but it only has O(|V|+|E|)constraints. It is a convex outer
approximation onM(G), as shown in Figure 22.7(c).
We call the terms τs,τst∈L(G)pseudo marginals , since they may not correspond to

∑
∑
marginals of any valid probability distribution. As an example of this, consider Figure 22.8(a).
The picture shows a set of pseudo node and edge marginals, which satisfy the local consistency
requirements. However, they are not globally consistent. To see why, note that τ12implies
p(X1=X2)=0.8,τ23impliesp(X2=X3)=0.8, butτ13impliesp(X1=X3)=0.2, which
is not possible (see (Wainwright and Jordan 2008b, p81) for a formal proof). Indeed, Figure 22.8(b)
shows that L(G)contains points that are not in M(G).

We claim that M(G)⊆L(G), with equality iff Gis a tree. To see this, ﬁrst consider

22.3. Loopybeliefpropagation: theoreticalissues* 781
an element μ∈M(G). Any such vector must satisfy the normalization and marginalization
constraints, hence M(G)⊆L(G).
Now consider the converse. Suppose Tis a tree, and let μ∈L(T). By deﬁnition, this satisﬁes
the normalization and marginalization constraints. However, any tree can be represented in the
form
pμ(x)=∏
s∈Vμs(xs)∏
(s,t)∈Eμst(xs,xt)
μs(xs)μt(xt)(22.34)
Hence satsifying normalization and local consistency is enough to deﬁne a valid distribution forany tree. Hence μ∈M(T)as well.
In contrast, if the graph has loops, we have that M(G)̸=L(G). See Figure 22.8(b) for an
example of this fact.
22.3.5.2 The entropy approximation
From Equation 22.34, we can write the exact entropy of any tree structured distribution μ∈
M(T)as follows:
H(μ)=∑
s∈VHs(μs)−∑
(s,t)∈EIst(μst) (22.35)
Hs(μs)=−∑
xs∈Xsμs(xs)logμs(xs) (22.36)
Ist(μst)=∑
(xs,xt)∈Xs×Xtμst(xs,xt)logμst(xs,xt)
μs(xs)μt(xt)(22.37)

H(μ)=∑
Hs(μs)=−∑
Note that we can rewrite the mutual information term in the form Ist(μst)=Hs(μs)+Ht(μt)−
Hst(μst), and hence we get the following alternative but equivalent expression:
H(μ)=−∑
s∈V(ds−1)Hs(μs)+∑
(s,t)∈EHst(μst) (22.38)
wheredsis the degree (number of neighbors) for node s.
TheBethe1approximation to the entropy is simply the use of Equation 22.35 even when we
don’t have a tree:
HBethe(τ)=∑
s∈VHs(τs)−∑
(s,t)∈EIst(τst) (22.39)
We deﬁne the Bethe free energy as
FBethe(τ)≜−[
θTτ+HBethe(τ)]
(22.40)
We deﬁne the Bethe energy functional as the negative of the Bethe free energy.

H(μ)=−∑
(22.40)
1. Hans Bethe was a German-American physicist, 1906–2005.

782 Chapter22. Morevariationalinference
22.3.5.3 The LBP objective
Combining the outer approximation L(G)with the Bethe approximation to the entropy, we get
the following Bethe variational problem (BVP):
min
τ∈L(G)FBethe(τ)= m a x
τ∈L(G)θTτ+HBethe(τ) (22.41)
The space we are optimizing over is a convex set, but the objective itself is not concave (since
HBetheis not concave). Thus there can be multiple local optima of the BVP.
The value obtained by the BVP is an approximation to logZ(θ). In the case of trees, the
approximation is exact, and in the case of models with attractive potentials, the approximationturns out to be an upper bound (Sudderth et al. 2008).
22.3.5.4 Message passing and Lagrange multipliers
In this subsection, we will show that any ﬁxed point of the LBP algorithm deﬁnes a stationarypoint of the above constrained objective. Let us deﬁne the normalization constraint at C
ss(τ)≜
1−∑
xsτs(xs), and the marginalization constraint as Cts(xs;τ)≜τs(xs)−∑
xtτst(xs,xt)

ss(τ)≜
1−∑
for each edge t→s. We can now write the Lagrangian as
L(τ,λ;θ)≜θTτ+HBethe(τ)+∑
sλssCss(τ)
+∑
s,t[∑
xsλts(xs)Cts(xs;τ)+∑
xtλst(xt)Cst(xt;τ)]
(22.42)
(The constraint that τ≥0is not explicitly enforced, but one can show that it will hold at the
optimum since θ>0.) Some simple algebra then shows that ∇τL=0yields
logτs(xs)=λ ss+θs(xs)+∑
t∈nbr(s)λts(xs) (22.43)
logτst(xs,xt)
˜τs(xs)˜τt(xt)=θst(xs,xt)−λts(xs)−λst(xt) (22.44)
where we have deﬁned ˜τs(xs)≜∑
xtτ(xs,xt). Using the fact that the marginalization con-
straint implies ˜τs(xs)=τs(xs),w eg e t

+∑
s,t[∑
(22.42)
logτst(xs,xt)=λ ss+λtt+θst(xs,xt)+θs(xs)+θt(xt)
+∑
u∈nbr(s)\tλus(xs)+∑
u∈nbr(t)\sλut(xt) (22.45)
To make the connection to message passing, deﬁne Mts(xs)=e x p ( λts(xs)). With this
notation, we can rewrite the above equations (after taking exponents of both sides) as follows:
τs(xs)∝exp(θs(xs))∏
t∈nbr(s)Mts(xs) (22.46)
τst(xs,xt)∝exp(θst(xs,xt)+θs(xs)+θt(xt))
×∏
u∈nbr(s)\tMus(xs)∏
u∈nbr(t)\sMut(xt) (22.47)

22.4. Extensionsofbeliefpropagation* 783
where the λterms are absorbed into the constant of proportionality. We see that this is
equivalent to the usual expression for the node and edge marginals in LBP.
To derive an equation for the messages in terms of other messages (rather than in terms of
λts), we enforce the marginalization condition∑
xtτst(xs,xt)=τs(xs). Then one can show
that
Mts(xs)∝∑
xt⎡
⎣exp{θst(xs,xt)+θt(xt)}∏
u∈nbr(t)\sMut(xt)⎤⎦ (22.48)
We see that this is equivalent to the usual expression for the messages in LBP.
22.3.6 Loopy BP vs mean ﬁeld
It is interesting to compare the naive mean ﬁeld (MF) and LBP approximations. There are several

that
Mts(xs)∝∑
xt⎡
obvious differences. First, LBP is exact for trees whereas MF is not, suggesting LBP will in generalbe more accurate (see (Wainwright et al. 2003) for an analysis). Second, LBP optimizes over nodeand edge marginals, whereas MF only optimizes over node marginals, again suggesting LBP willbe more accurate. Third, in the case that the true edge marginals factorize, so μ
st=μsμt, the
free energy approximations will be the same in both cases.
What is less obvious, but which nevertheless seems to be true, is that the MF objective has

many more local optima than the LBP objective, so optimizing the MF objective seems to beharder. In particular, (Weiss 2001), shows empirically that optimizing MF starting from uniformor random initial conditions often leads to poor results, whereas optimizing BP from uniforminitial messages often leads to good results. Furthermore, initializing MF with the BP marginalsalso leads to good results (although MF tends to be more overconﬁdent than BP), indicating thatthe problem is caused not by the inaccuracy of the MF approximation, but rather by the severenon-convexity of the MF objective, and by the weakness of the standard coordinate descentoptimization method used by MF.
2However, the advantage of MF is that it gives a lower bound

on the partition function, unlike BP, which is useful when using it as a subroutine inside alearning algorithm. Also, MF is easier to extend to other distributions besides discrete andGaussian, as we saw in Chapter 21. Intuitively, this is because MF only works with marginaldistributions, which have a single type, rather than needing to deﬁne pairwise distributions,which may need to have two different types.
22.4 Extensions of belief propagation *
In this section, we discuss various extensions of LBP.
22.4.1 Generalized belief propagation
We can improve the accuracy of loopy BP by clustering together nodes that form a tight loop.This is known as the cluster variational method. The result is a hyper-graph, which is a graph
2. (Honkela et al. 2003) discusses the use of the pattern search algorithm to speedup mean ﬁeld inference in the case

of continuous random variables. It is possible that similar ideas could be adapted to the discrete case, although there
may be no reason to do this, given that LBP already works well in the discrete case.

784 Chapter22. Morevariationalinference
1 2
8 743
95 61245
5 22356
5
547 858
568945 56
Figure 22.9 (a) Kikuchi clusters superimposed on a 3×3lattice graph. (b) Corresponding hyper-graph.
Source: Figure 4.5 of (Wainwright and Jordan 2008b). Used with kind permission of Martin Wainwright.
where there are hyper-edges between sets of vertices instead of between single vertices. Note
that a junction tree (Section 20.4.1) is a kind of hyper-graph. We can represent hyper-graph usinga poset (partially ordered set) diagram, where each node represents a hyper-edge, and there isan arrow e
1→e2ife2⊂e1. See Figure 22.9 for an example.
Lettbe the size of the largest hyper-edge in the hyper-graph. If we allow tto be as large as
the treewidth of the graph, then we can represent the hyper-graph as a tree, and the methodwill be exact, just as LBP is exact on regular trees (with treewidth 1). In this way, we can deﬁnea continuum of approximations, from LBP all the way to exact inference.
Deﬁne L

Deﬁne L
t(G)to be the set of all pseudo-marginals such that normalization and marginaliza-
tion constraints hold on a hyper-graph whose largest hyper-edge is of size t+1. For example,
in Figure 22.9, we impose constraints of the form
∑
x1,x2τ1245(x1,x2,x4,x5)=τ45(x4,x5),∑
x6τ56(x5,x6)=τ5(x5),... (22.49)
Furthermore, we approximate the entropy as follows:
HKikuchi(τ)≜∑
g∈Ec(g)Hg(τg) (22.50)
whereHg(τg)is the entropy of the joint (pseudo) distribution on the vertices in set g, andc(g)
is called the overcounting number of setg. These are related to Mobious numbers in set

∑
theory. Rather than giving a precise deﬁnition, we just give a simple example. For the graph inFigure 22.9, we have
H
Kikuchi(τ)=[H1245+H2356+H4578+H5689]
−[H25+H45+H56+H58]+H5 (22.51)
Putting these two approximations together, we can deﬁne the Kikuchi free energy3as follows:
FKikuchi(τ)≜−[
θTτ+HKikuchi(τ)]
(22.52)
3. Ryoichi Kikuchi is a Japanese physicist.

22.4. Extensionsofbeliefpropagation* 785
Our variational problem becomes
min
τ∈Lt(G)FKikuchi(τ)= m a x
τ∈Lt(G)θTτ+HKikuchi(τ) (22.53)
Just as with the Bethe free energy, this is not a concave objective. There are several possible
algorithms for ﬁnding a local optimum of this objective, including a message passing algorithm
known as generalized belief propagation. However, the details are beyond the scope of this
chapter. See e.g., (Wainwright and Jordan 2008b, Sec 4.2) or (Koller and Friedman 2009, Sec11.3.2) for more information. Suffice it to say that the method gives more accurate results thanLBP, but at increased computational cost (because of the need to handle clusters of nodes). Thiscost, plus the complexity of the approach, have precluded it from widespread use.
22.4.2 Convex belief propagation

The mean ﬁeld energy functional is concave, but it is maximized over a non-convex innerapproximation to the marginal polytope. The Bethe and Kikuchi energy functionals are notconcave, but they are maximized over a convex outer approximation to the marginal polytope.Consequently, for both MF and LBP, the optimization problem has multiple optima, so themethods are sensitive to the initial conditions. Given that the exact formulation (Equation 22.24)a concave objective maximized over a convex set, it is natural to try to come up with anappproximation which involves a concave objective being maximized over a convex set.

We now describe one method, known as convex belief propagation. This involves working
with a set of tractable submodels, F, such as trees or planar graphs. For each model F⊂G,
the entropy is higher, H(μ(F))≥H(μ(G)), since Fhas fewer constraints. Consequently, any
convex combination of such subgraphs will have higher entropy, too:
H(μ(G))≤∑
F∈Fρ(F)H(μ(F))≜H(μ,ρ) (22.54)
whereρ(F)≥0and∑
Fρ(F)=1. Furthermore, H(μ,ρ)is a concave function of μ.W en o w
deﬁne the convex free energy as
FConvex(μ,ρ)≜−[
μTθ+H(μ,ρ)]
(22.55)
We deﬁne the concave energy functional as the negative of the convex free energy. We discusshow to optimize ρbelow.
Having deﬁned an upper bound on the entropy, we now consider a convex outerbound on
the marginal polytope of mean parameters. We want to ensure we can evaluate the entropy ofany vector τin this set, so we restrict it so that the projection of τonto the subgraph Glives
in the projection of MontoF:
L(G;F)≜{τ∈R
d:τ(F)∈M(F)∀F∈F} (22.56)

(22.55)
This is a convex set since each M(F)is a projection of a convex set. Hence we deﬁne our
problem as
min
τ∈L(G;F)FConvex(τ,ρ)= m a x
τ∈L(G;F)τTθ+H(τ,ρ) (22.57)
This is a concave objective being maximized over a convex set, and hence has a unique maxi-mum. We give a speciﬁc example below.

786 Chapter22. Morevariationalinference
b
ef
b
ef
b
ef
b
ef
Figure 22.10 (a) A graph. (b-d) Some of its spanning trees. Source: Figure 7.1 of (Wainwright and Jordan
2008b). Used with kind permission of Martin Wainwright.
22.4.2.1 Tree-reweighted belief propagation
Consider the speciﬁc case where Fis all spanning trees of a graph. For any given tree, the
entropy is given by Equation 22.35. To compute the upper bound, obtained by averaging over
all trees, note that the terms∑
Fρ(F)H(μ(F)s)for single nodes will just be Hs, since node s
appears in every tree, and∑
Fρ(F)=1. But the mutual information term Istreceives weight
ρst=Eρ[I((s,t)∈E(T))], known as the edge appearance probability. Hence we have the
following upper bound on the entropy:
H(μ)≤∑
(s,t)∈EρstIst(μst) (22.58)
The edge appearance probabilities live in a space called the spanning tree polytope . This

H(μ)≤∑
is because they are constrained to arise from a distribution over trees. Figure 22.10 gives anexample of a graph and three of its spanning trees. Suppose each tree has equal weight underρ. The edge foccurs in 1 of the 3 trees, so ρ
f=1/3. The edge eoccurs in 2 of the 3 trees,
soρe=2/3. The edge bappears in all of the trees, so ρb=1. And so on. Ideally we can
ﬁnd a distribution ρ, or equivalently edge probabilities in the spanning tree polytope, that make

the above bound as tight as possible. An algorithm to do this is described in (Wainwright et al.2005). (A simpler approach is to generate spanning trees of Gat random until all edges are
covered, or use all single edges with weight ρ
e=1/E.)
What about the set we are optimizing over? We require μ(T)∈M(T)for each tree T, which
means enforcing normalization and local consistency. Since we have to do this for every tree,we are enforcing normalization and local consistency on every edge. Hence L(G;F)=L(G).
So our ﬁnal optimization problem is as follows:
max

e=1/E.)
max
τ∈L(G)⎧
⎨
⎩τTθ+∑
(s,t)∈E(G)ρstIst(τst)⎫
⎬
⎭(22.59)
which is the same as the LBP objective except for the crucial ρstweights. So long as ρst>0
for all edges (s,t), this problem is strictly concave with a unique maximum.
How can we ﬁnd this global optimum? As for LBP, there are several algorithms, but perhaps the
simplest is a modiﬁcation of belief propagation known as tree reweighted belief propagation ,

22.5. Expectationpropagation 787
also called TRWorTRBPfor short. The message from ttosis now a function of all messages
sent from other neighbors vtot, as before, but now it is also a function of the message sent
fromstot. Speciﬁcally
Mts(xs)∝∑
xtexp(1
ρstθst(xs,xt)+θt(xt))∏
v∈nbr(t)\s[Mvt(xt)]ρvt
[Mst(xt)]1−ρts(22.60)
At convergence, the node and edge pseudo marginals are given by
v∈nbr(s)[Mvs(xs)]ρvs(22.61)
τst(xs,xt)∝ϕst(xs,xt)∏
v∈nbr(s)\t[Mvs(xs)]ρvs
[Mts(xs)]1−ρst∏
[Mst(xt)]1−ρts(22.62)
ϕst(xs,xt)≜exp(1
ρstθst(xs,xt)+θs(xs)+θt(xt))
(22.63)
This algorithm can be derived using a method similar to that described in Section 22.3.5.4.
Ifρst=1for all edges (s,t)∈E, the algorithm reduces to the standard LBP algorithm.
However, the condition ρst=1implies every edge is present in every spanning tree with
probability 1, which is only possible if the original graph is a tree. Hence the method is only
equivalent to standard LBP on trees, when the method is of course exact.

(22.63)
In general, this message passing scheme is not guaranteed to converge to the unique global
optimum. One can devise double-loop methods that are guaranteed to converge (Hazan andShashua 2008), but in practice, using damped updates as in Equation 22.7 is often sufficient toensure convergence.
It is also possible to produce a convex version of the Kikuchi free energy, which one can
optimize with a modiﬁed version of generalized belief propagation. See (Wainwright and Jordan2008b, Sec 7.2.2) for details.

From Equation 22.59, and using the fact that the TRBP entropy approximation is an upper
bound on the true entropy, wee see that the TRBP objective is an upper bound on logZ. Using
the fact that I
st=Hs+Ht−Hst, we can rewrite the upper bound as follows:
logˆZ(θ)≜τTθ+∑
stρstHst(τst)+∑
scsHs(τs)≤logZ(θ) (22.64)
wherecs≜1−∑
tρst.
22.5 Expectation propagation
Expectation propagation (EP) (Minka 2001c) is a form of belief propagation where the mes-

tρst.
sages are approximated. It is a generalization of the assumed density ﬁltering (ADF) algorithm,discussed in Section 18.5.3. In that method, we approximated the posterior at each step usingan assumed functional form, such as a Gaussian. This posterior can be computed using mo-ment matching, which locally optimizes KL(p||q)for a single term. From this, we derived the
message to send to the next time step.

788 Chapter22. Morevariationalinference
ADF works well for sequential Bayesian updating, but the answer it gives depends on the
order in which the data is seen. EP essentially corrects this ﬂaw by making multiple passes over
the data (thus EP is an offline or batch inference algorithm).
22.5.1 EP as a variational inference problem
We now explain how to view EP in terms of variational inference. We follow the presentation of(Wainwright and Jordan 2008b, Sec 4.3), which should be consulted for further details.
Suppose the joint distribution can be written in exponential family form as follows:
p(x|θ,˜θ)∝f
0(x)exp(θTφ(x))dI∏
i=1exp(˜θT
iΦi(x)) (22.65)
where we have partitioned the parameters and the sufficient statistics into a tractable term θof
sizedTanddIintractable terms ˜θi, each of size b.
For example, consider the problem of inferring an unknown vector x, when the observation
model is a mixture of two Gaussians, one centered at xand one centered at 0. (This can be

used to represent outliers, for example.) Minka (who invented EP) calls this the clutter problem.
More formally, we assume an observation model of the form
p(y|x)=( 1−w)N(y|x,I)+wN(y|0,aI) (22.66)
where0<w<1 is the known mixing weight (fraction of outliers), and a>0is the variance
of the background distribution. Assuming a ﬁxed prior of the form p(x)=N(x|0,Σ), we can
write our model in the required form as follows:
p(x|y1:N)∝N(x|0,Σ)N∏
i=1p(yi|x) (22.67)
=e x p(
−1
2xTΣ−1x)
exp(N∑
i=1logp(yi|x))
(22.68)

=e x p(
−1
2xTΣ−1x)
exp(N∑
(22.68)
This matches our canonical form where f0(x)exp(θTφ(x))corresponds to exp(
−1
2xTΣ−1x)
,
usingφ(x)=(x,xxT), and we set Φi(x)=l o gp(yi|x),˜θi=1, anddI=N.
The exact inference problem corresponds to
max
(τ,˜τ)∈M(φ,Φ)τTθ+˜τT˜θ+H((τ,˜τ)) (22.69)
whereM(φ,Φ)is the set of mean parameters realizable by any probability distribution as seen
through the eyes of the sufficient statistics:
M(φ,Φ)={(μ,˜μ)∈RdT×RdIb:(μ,˜μ)=E[(φ(X),Φ1(X),...,ΦdI(X))]}(22.70)

=e x p(
−1
2xTΣ−1x)
exp(N∑
(22.68)
−1
2xTΣ−1x)
,
max
As it stands, it is intractable to perform inference in this distribution. For example, in our
clutter example, the posterior contains 2Nmodes. But suppose we incorporate just one of the
intractable terms, say the i’th one; we will call this the Φi-augmented distribution:
p(x|θ,˜θi)∝f0(x)exp(θTφ(x))exp(˜θTiΦi(x)) (22.71)

22.5. Expectationpropagation 789
In our clutter example, this becomes
p(x|θ,˜θi)=e x p(
−1
2xTΣ−1x)
[wN(yi|0,aI)+(1− w)N(yi|x,I)] (22.72)
Thisistractable to compute, since it is just a mixture of 2 Gaussians.
The key idea behind EP is to work with these the Φi-augmented distributions in an iterative
fashion. First, we approximate the convex set M(φ,Φ)with another, larger convex set:
L(φ,Φ)≜{(τ,˜τ):τ∈M(φ),(τ,˜τi)∈M(φ,Φi)} (22.73)
whereM(φ)={μ∈RdT:μ=E[φ(X)]}andM(φ,Φi)={(μ,˜μi)∈RdT×Rb:
(μ,˜μi)=E[(φ(X),Φi(X))]. Next we approximate the entropy by the following term-by-term
approximation:
Hep(τ,˜τ)≜H(τ)+dI∑
i=1[H(τ,˜τi)−H(τ)] (22.74)
Then the EP problem becomes
max
(τ,˜τ)∈L(φ,Φ)τTθ+˜τT˜θ+Hep(τ,˜τ) (22.75)
22.5.2 Optimizing the EP objective using moment matching
We now discuss how to maximize the EP objective in Equation 22.75. Let us duplicate τdI
times to yield ηi=τ. The augmented set of parameters we need to optimize is now
(τ,(ηi,˜τi)dI
i=1)∈RdT×(RdT×Rb)dI(22.76)

max
subject to the constraints that ηi=τand(ηi,˜τi)∈M(φ;Φi). Let us associate a vector of
Lagrange multipliers λi∈RdTwith the ﬁrst set of constraints. Then the partial Lagrangian
becomes
L(τ;λ)=τTθ+H(τ)+di∑
i=1[
˜τT
i˜θi+H((ηi,˜τi))−H(ηi)+λT
i(τ−ηi)]
(22.77)
By solving∇τL(τ;λ)=0, we can show that the corresponding distribution in M(φ)has
the form
q(x|θ,λ)∝f0(x)exp{(θ+dI∑
i=1λi)Tφ(x)} (22.78)
TheλTiφ(x)terms represents an approximation to the i’th intractable term using the sufficient

becomes
i=1[
˜τT
i(τ−ηi)]
(22.77)
the form
statisticsfromthebasedistribution, aswewillseebelow. Similarly, bysolving ∇(ηi,˜τi)L(τ;λ)=
0, we ﬁnd that the corresponding distribution in M(φ,Φi)has the form
qi(x|θ,˜θi,λ)∝f0(x)exp{(θ+∑
j̸=iλj)Tφ(x)+˜θT
iΦi(x)} (22.79)

790 Chapter22. Morevariationalinference
This corresponds to removing the approximation to the i’th term, λi, from the base distribution,
and adding in the correct i’th term, Φi. Finally,∇λL(τ;λ)=0just enforces the constraints
thatτ=Eq[φ(X)]andηi=Eqi[φ(X)]are equal. In other words, we get the following
moment matching constraints:
∫
q(x|θ,λ)φ(x)dx=∫
qi(x|θ,˜θi,λ)φ(x)dx (22.80)
Thus the overall algorithm is as follows. First we initialize the λi. Then we iterate the following
to convergence: pick a term i; compute qi(corresponding to removing the old approximation
toΦiand adding in the new one); then update the λiterm inqby solving the moment
matching equation Eqi[φ(X)] = Eq[φ(X)]. (Note that this particular optimization scheme is
not guaranteed to converge to a ﬁxed point.)
An equivalent way of stating the algorithm is as follows. Let us assume the true distribution
is given by
p(x|D)=1
Z∏
ifi(x) (22.81)
We approximate each fiby˜fiand set
q(x)=1
Z∏
i˜fi(x) (22.82)
Now we repeat the following until convergence:

is given by
p(x|D)=1
Z∏
q(x)=1
Z∏
1. Choose a factor ˜fito reﬁne.
2. Remove ˜fifrom the posterior by dividing it out:
q−i(x)=q(x)
˜fi(x)(22.83)
This can be implemented by substracting off the natural parameters of ˜fifromq.
3. Compute the new posterior qnew(x)by solving
min
qnew(x)KL(1
Zifi(x)q−i(x)||qnew(x))
(22.84)
This can be done by equating the moments of qnew(x)with those of qi(x)∝q−i(x)fi(x).
The corresponding normalization constant has the form
Zi=∫
q−i(x)fi(x)dx (22.85)
4. Compute the new factor (message) that was implicitly used (so it can be later removed):
˜fi(x)=Ziqnew(x)

min
(22.84)
Zi=∫
q−i(x)(22.86)

22.5. Expectationpropagation 791
After convergence, we can approximate the marginal likelihood using
p(D)≈∫∏
i˜fi(x)dx (22.87)
We will give some examples of this below which will make things clearer.
22.5.3 EP for the clutter problem
Let us return to considering the clutter problem. Our presentation is based on (Bishop 2006b).4
For simplicity, we will assume that the prior is a spherical Gaussian, p(x)=N(0,bI). Also, we
choose to approximate the posterior by a spherical Gaussian, q(x)=N(m,vI).W es e tf0(x)
to be the prior; this can be held ﬁxed. The factor approximations will be “Gaussian like” terms
of the form
˜fi(x)=siN(x|mi,viI) (22.88)
Note, however, that in the EP updates, the variances may be negative! Thus these terms shouldbe interpreted as functions, but not necessarily probability distributions. (If the variance isnegative, it means the that ˜f
icurves upwards instead of downwards.)
First we remove ˜fi(x)fromq(x)by division, which yields q−i(x)=N(m−i,v−iI),w h e r e
v−1
−i=v−1−v−1
i (22.89)

of the form
v−1
i (22.89)
m−i=m+v−iv−1
i(m−mi) (22.90)
The normalization constant is given by
Zi=( 1−w)N(yi|m−i,(v−i+1)I)+wN(yi|0,aI) (22.91)
Next we compute qnew(x)by computing the mean and variance of q−i(x)fi(x)as follows:
m=m−i+ρiv−i
v−i+1(yi−m−i) (22.92)
v=v−i−ρiv2
−i
v−i+1+ρi(1−ρi)v2
−i||yi−mi||2
D(v−i+1)2(22.93)
ρi=1−w
ZiN(yi|0,aI) (22.94)
whereDis the dimensionality of xandρican be interpreted as the probability that yiis not
clutter.
Finally, we compute the new factor ˜fiwhose parameters are given by
v−1
i=v−1−v−1
−i (22.95)

−i
ρi=1−w
clutter.
v−1
i=v−1−v−1
−i (22.95)
mi=m−i+(vi+v−i)v−1
−i(m−m−i) (22.96)
si=Zi
(2πvi)D/2N(mi|m−i,(vi+v−i)I)(22.97)
4. For a handy “crib sheet”, containing many of the standard equations needed for deriving Gaussian EP algorithms, see
http://research .microsoft .com/en-us/um/people/minka/papers/ep/minka-ep-quickref .pdf.

792 Chapter22. Morevariationalinference
At convergence, we can approximate the marginal likelihood as follows:
p(D)≈(2πv)D/2exp(c/2)N∏
i=1si(2πvi)−D/2(22.98)
c≜mTm
v−N∑
i=1mT
imi
vi(22.99)
In (Minka 2001d), it is shown that, at least on this example, EP gives better accuracy per unit
of CPU time than VB and MCMC.
22.5.4 LBP is a special case of EP
We now show that loopy belief propagation is a special case of EP, where the base distribution
contains the node marginals and the “intractable” terms correspond to the edge potentials. Weassume the model has the pairwise form shown in Equation 22.12. If there are mnodes, the
base distribution takes the form
p(x|θ
1,...,θm,0)∝∏
s∈Vexp(θs(xs)) (22.100)
The entropy of this distribution is simply
H(τ1:m)=∑
sH(τs) (22.101)
If we add in the u−vedge, the Φuvaugmented distribution has the form
p(x|θ1:m,θuv)∝[∏
s∈Vexp(θs(xs))]
exp(θuv(xu,xv)) (22.102)
Since this graph is a tree, the exact entropy of this distribution is given by
H(τ1:m,˜τuv)=∑
sH(τs)−I(˜τuv) (22.103)

p(x|θ
H(τ1:m)=∑
whereI(τuv)=H(τu)+H(τv)−H(τuv)is the mutual information. Thus the EP approxi-
mation to the entropy of the full distribution is given by
Hep(τ,˜τ)=H (τ)+∑
(u,v)∈E[H(τ1:m,˜τuv)−H(τ)] (22.104)
=∑
sH(τs)+∑
(u,v)∈E[∑
sH(τs)−I(˜τuv)−∑
sH(τs)]
(22.105)
=∑
sH(τs)−∑
(u,v)∈EI(˜τuv) (22.106)
which is precisely the Bethe approximation to the entropy.

22.5. Expectationpropagation 793
We now show that the convex set that EP is optimizing over, L(φ,Φ)given by Equation 22.73,
is the same as the one that LBP is optimizing over, L(G)given in Equation 22.33. First, let us
consider the set M(φ). This consists of all marginal distributions (τs,s∈V), realizable by
a factored distribution. This is therefore equivalent to the set of all distributions which satisfy
non-negativity τs(xs)≥0and the local normalization constraint∑
xsτ(xs)=1. Now consider
the setM(φ,Φuv)for a single u−vedge. This is equivalent to the marginal polytope M(Guv),
whereGuvis the graph with the single u−vedge added. Since this graph corresponds to a
tree, this set also satisﬁes the marginalization conditions
∑
xvτuv(xu,xv)=τu(xu),∑
xuτuv(xu,xv)=τv(xv) (22.107)
SinceL(φ,Φ)is the union of such sets, as we sweep over all edges in the graph, we recover
the same set as L(G).
We have shown that the Bethe approximation is equivalent to the EP approximation. We now

∑
show how the EP algorithm reduces to LBP. Associated with each intractable term i=(u,v)
will be a pair of Lagrange multipliers, (λuv(xv),λvu(xu)). Recalling that θTφ(x)=[θs(xs)]s,
the base distribution in Equation 22.78 has the form
q(x|θ,λ)∝∏
sexp(θs(xs))∏
(u,v)∈Eexp(λuv(xv)+λvu(xu)) (22.108)
=∏
sexp⎛
⎝θs(xs)+∑
t∈N(s)λts(xs)⎞⎠ (22.109)
Similarly, the augmented distribution in Equation 22.79 has the form
q
uv(x|θ,λ)∝q(x|θ,λ)exp(θuv(xu,xv)−λuv(xv)−λvu(xu)) (22.110)

=∏
sexp⎛
⎝θs(xs)+∑
q
We now need to update τu(xu)andτv(xv)to enforce the moment matching constraints:
(Eq[xs],Eq[xt]) = (E quv[xs],Equv[xt]) (22.111)
It can be shown that this can be done by performing the usual sum-product message passing
step along the u−vedge (in both directions), where the messages are given by Muv(xv)=
exp(λuv(xv)), andMvu(xu)=e x p ( λvu(xu)). Once we have updated q, we can derive the
corresponding messages λuvandλvu.
The above analysis suggests a natural extension, where we make the base distribution be a

tree structure instead of a fully factored distribution. We then add in one edge at a time, absorbits effect, and approximate the resulting distribution by a new tree. This is known as tree EP
(Minka and Qi 2003), and is more accurate than LBP, and sometimes faster. By considering otherkinds of structured base distributions, we can derive algorothms that outperform generalizationbelief propagation (Welling et al. 2005).
22.5.5 Ranking players using TrueSkill

We now present an interesting application of EP to the problem of ranking players who competein games. Microsoft uses this method — known as TrueSkill (Herbrich et al. 2007) — to rank

22.5. Expectationpropagation 795
players who use the Xbox 360 Live online gaming system; this system process over 105games
per day, making this one of the largest application of Bayesian statistics to date.5The same
method can also be applied to other games, such as tennis or chess.6
The basic idea is shown in Figure 22.11(a). We assume each player ihas a latent or true
underlying skill level si∈R. These skill levels can evolve over time according to a simple
dynamical model, p(st
i|st−1
i)=N(sti|st−1
i,γ2). In any given game, we deﬁne the performance
of player ito bepi, which has the conditional distribution p(pi|si)=N(pi|si,β2). We then
deﬁne the performance of a team to be the sum of the performance of its constituent players.
For example, in Figure 22.11(a), we assume team 2 is composed of players 2 and 3, so we deﬁnet
2=p2+p3. Finally, we assume that the outcome of a game depends on the difference in
performance levels of the two teams. For example, in Figure 22.11(a), we assume y1=s i g n (d1),

whered1=t1−t2, and where y1=+ 1means team 1 won, and y1=−1means team 2 won.
Thus the prior probability that team 1 wins is
p(y1=+ 1|s)=∫
p(d1>0|t1,t2)p(t1|s1)p(t2|s2)dt1dt2 (22.112)
wheret1∼N(s1,β2)andt2∼N(s2+s3,β2).7
To simplify the presentation of the algorithm, we will ignore the dynamical model and assume
a common static factored Gaussian prior, N(μ0,σ2
0), on the skills. Also, we will assume that
each team consists of 1 player, so ti=pi, and that there can be no ties. Finally, we will integrate

out the performance variables pi, and assume β2=1, leading to a ﬁnal model of the form
p(s)=∏
iN(si|μ0,σ2) (22.113)
p(dg|s)=N (dg|sig−sjg,1) (22.114)
p(yg|dg)=I(yg=s i g n (dg)) (22.115)
whereigis the ﬁrst player of game g, andjgis the second player. This is represented in
factor graph form in in Figure 22.11(b). We have 3 kinds of factors: the prior factor, fi(si)=
N(si|μ0,σ2
0), the game factor, hg(sig,sjg,dg)=N(dg|sig−sjg,1), and the outcome factor,
kg(dg,yg)=I(yg=s i g n (dg)).

p(s)=∏
Since the likelihood term (yg|dg)is not conjugate to the Gaussian priors, we will have to
perform approximate inference. Thus even when the graph is a tree, we will need to iterate.(If there were an additional game, say between player 1 and player 3, then the graph would nolonger be a tree.) We will represent all messages and marginal beliefs by 1d Gaussians. We willuse the notation μandvfor the mean and variance (the moment parameters), and λ=1/v
andη=λμfor the precision and precision-adjusted mean (the natural parameters).

5. Naive Bayes classiﬁers, which are widely used in spam ﬁlters, are often described as the most common application
of Bayesian methods. However, the parameters of such models are usually ﬁt using non-Bayesian methods, such as
penalized maximum likelihood.
6. Our presentation of this algorithm is based in part on lecture notes by Carl Rasmussen Joaquin Quinonero-Candela,
available at http://mlg .eng.cam .ac.uk/teaching/4f13/1112/lect13 .pdf.

7. Note that this is very similar to probit regression, discussed in Section 9.4, except the inputs are (the differences of)latent 1 dimensional factors. If we assume a logistic noise model instead of a Gaussian noise model, we recover the
Bradley Terry model of ranking.

796 Chapter22. Morevariationalinference
We initialize by assuming that at iteration 0, the initial upward messages from factors hgto
variables siare uniform, i.e.,
m0
hg→sig(sig)=1,λ0h
g→sig=0,η0
hg→sig=0 (22.116)
and similarly m0
hg→sjg(sjg)=1. The messages passing algorithm consists of 6 steps per game,
as illustrated in Figure 22.11(b). We give the details of these steps below.
1. Compute the posterior over the skills variables:
qt(si)=f (si)∏
gmt−1
hg→si(si)=Nc(si|ηt
i,λt
i) (22.117)
λti=λ0+∑
gλt−1
hg→si,ηt
i=η0+∑
gηt−1
hg→si(22.118)
2. Compute the message from the skills variables down to the game factor hg:
mts
ig→hg(sig)=qt(sig)
mt
hg→sig(sig),mt
sjg→hg(sjg)=qt(sjg)
mt
hg→sjg(sjg)(22.119)
where the division is implemented by subtracting the natural parameters as follows:
λt
sig→hg=λts
ig−λth
g→sig,ηt
sig→hg=ηt
sig−ηt
hg→sig(22.120)
and similarly for sjg.
3. Compute the message from the game factor hgdown to the difference variable dg:
mth
g→dg(dg)=∫∫
hg(dg,sig,sjg)mts
ig→hg(sig)mts

λti=λ0+∑
gλt−1
hg→si,ηt
i=η0+∑
gηt−1
mts
mt
mt
λt
ig−λth
g→sig,ηt
sig→hg=ηt
sig−ηt
mth
jg→hg(sjg)dsigdsjg(22.121)
=∫∫
N(dg|sig−sjg,1)N(sig|μts
ig→hg,vt
sig→hg) (22.122)
N(sjg|μts
jg→hg,vt
sjg→hg)dsigdsjg (22.123)
=N(dg|μth
g→dg,vt
hg→dg) (22.124)
vt
hg→dg=1 +vt
sig→hg+vt
sjg→hg(22.125)
μth
g→dg=μts
ig→hg−μts
jg→hg(22.126)
4. Compute the posterior over the difference variables:
qt(dg)∝mth
g→dg(dg)mkg→dg(dg) (22.127)
=N(dg|μth
g→dg,vt
hg→dg)I(yg=s i g n (dg)) (22.128)
≈N(dg|μtg,vt
g) (22.129)

22.5. Expectationpropagation 797
−6 −4 −2 0 2 4 601234567Ψ function
(a)−6 −4 −2 0 2 4 600.10.20.30.40.50.60.70.80.91Λ function
(b)
Figure 22.12 (a)Ψfunction. (b) Λfunction. Based on Figure 2 of (Herbrich et al. 2007). Figure generated
bytrueskillPlot .
(Note that the upward message from the kgfactor is constant.) We can ﬁnd these parameters
by moment matching as follows:
μt
g=ygμth
g→dg+σt
hg→dgΨ(
ygμt
hg→dg
σt
hg→dg)
(22.130)
vt
g=vt
hg→dg[
1−Λ(
ygμth
g→dg
σt
hg→dg)]
(22.131)
Ψ(x) ≜N(x|0,1)
Φ(x)(22.132)
Λ(x) ≜Ψ(x)(Ψ(x)+x) (22.133)
(The derivation of these equations is left as a modiﬁcation to Exercise 11.15.) These functions
are plotted in Figure 22.12. Let us try to understand these equations. Suppose μt
hg→dgis a
large positive number. That means we expect, based on the current estimate of the skills,
thatdgwill be large and positive. Consequently, if we observe yg=+ 1, we will not be
surprised that igis the winner, which is reﬂected in the fact that the update factor for the
mean is small, Ψ(ygμt

hg→dgis a
hg→dg)≈0. Similarly, the update factor for the variance is small,
Λ(ygμth
g→dg)≈0. However, if we observe yg=−1, then the update factor for the mean
and variance becomes quite large.
5. Compute the upward message from the difference variable to the game factor hg:
mt
dg→hg(dg)=qt(dg)
mt
dg→hg(dg)(22.134)
λt
dg→hh=λtg−λth
g→dg,ηt
dg→hh=ηt
g−ηt
hg→dg(22.135)
6. Compute the upward messages from the game factor to the skill variables. Let us assume

798 Chapter22. Morevariationalinference
2
53
4
61
(a)0 1 2 3 4 5 6 7−2−1.5−1−0.500.511.52
(b)
Figure 22.13 (a) A DAG representing a partial ordering of players. (b) Posterior mean plus/minus 1 standard
deviation for the latent skills of each player based on 26 games. Figure generated by trueskillDemo .
thatigis the winner, and jgis the loser. Then we have
mt
hg→sig(sig)=∫∫
hg(dg,sig,sjg)mtd
g→hg(dg)mts
jg→hg(sjg)ddgdsjg(22.136)
=N(sig|μth
g→sig,vt
hg→sig) (22.137)
vt
hg→sig=1 +vt
dg→hg+vt
sjg→hg(22.138)
μth
g→sig=μtd
g→hg+μts
jg→hg(22.139)
And similarly
mth
g→sjg(sjg)=∫∫
ig→hg(sig)ddgdsig(22.140)
=N(sjg|μth
g→sjg,vt
hg→sjg) (22.141)
vt
hg→sjg=1 +vt
dg→hg+vt
sig→hg(22.142)
μth
g→sjg=μtd
g→hg−μts
ig→hg(22.143)
When we compute qt+1(sig)at the next iteration, by combining mt
hg→sig(sig)with the
prior factor, we will see that the posterior mean of siggoes up. Similarly, the posterior mean
ofsjggoes down.

g→sig,vt
vt
dg→hg+vt
μth
g→sig=μtd
g→hg+μts
mth
g→sjg,vt
vt
dg→hg+vt
μth
g→sjg=μtd
g→hg−μts
It is straightforward to combine EP with ADF to perform online inference, which is necessary
for most practical applications.
Let us consider a simple example of this method. We create a partial ordering of 5 players
as shown in Figure 22.13(a). We then sample some game outcomes from this graph, where a

22.6. MAPstateestimation 799
parent always beats a child. We pass this data into (5 iterations of) the EP algorithm and infer the
posterior mean and variance for each player’s skill level. The results are shown in Figure 22.13(b).We see that the method has correctly inferred the rank ordering of the players.
22.5.6 Other applications of EP
The TrueSkill model was developed by researchers at Microsoft. They and others have extendedthe model to a variety of other interesting applications, including personalized ad recommenda-tion (Stern et al. 2009), predicting click-through-rate on ads in the Bing search engine (Graepelet al. 2010), etc. They have also developed a general purpose Bayesian inference toolbox basedon EP called infer.net (Minka et al. 2010).
EP has also been used for a variety of other models, such as Gaussian process classiﬁcation
(Nickisch and Rasmussen 2008). See http://research .microsoft .com/en-us/um/people/
minka/papers/ep/roadmap .htmlfor a list of other EP applications.

22.6 MAP state estimation
In this section, we consider the problem of ﬁnding the most probable conﬁguration of variablesin a discrete-state graphical model, i.e., our goal is to ﬁnd a MAP assignment of the followingform:
x
∗=a r gm a x
x∈Xmp(x|θ) = arg max
x∈Xm∑
i∈Vθi(xi)+∑
f∈Fθf(xf) = arg max
x∈XmθTφ(x)(22.144)
whereθiare the singleton node potentials, and θfare the factor potentials. (In this section, we
follow the notation of (Sontag et al. 2011), which considers the case of general potentials, not justpairwise ones.) Note that the partition function Z(θ)plays no role in MAP estimation.
If the treewidth is low, we can solve this problem with the junction tree algorithm (Sec-

x
∗=a r gm a x
x∈Xm∑
tion 20.4), but in general this problem is intractable. In this section, we discuss various approxi-mations, building on the material from Section 22.3.
22.6.1 Linear programming relaxation
We can rewrite the objective in terms of the variational parameters as follows:
arg max
x∈XmθTφ(x) = arg max
μ∈M(G)θTμ (22.145)
whereφ(x)=[{I(xs=j)},{I(xf=k)})andμis a probability vector in the marginal
polytope. To see why this equation is true, note that we can just set μto be a degenerate
distribution with μ(xs)=I(xs=x∗

arg max
s),w h e r ex∗sis the optimal assigment of node s. So instead
of optimizing over discrete assignments, we now optimize over probability distributions μ.
It seems like we have an easy problem to solve, since the objective in Equation 22.145 is linear
inμ, and the constraint set M(G)is convex. The trouble is, M(G)in general has a number of
facets that is exponential in the number of nodes.
A standard strategy in combinatorial optimization is to relax the constraints. In this case,
instead of requiring probability vector μto live in the marginal polytope M(G), we allow it to

800 Chapter22. Morevariationalinference
live inside a convex outer bound L(G). Having deﬁned this relaxed constraint set, we have
max
x∈XmθTφ(x)= m a x
μ∈M(G)θTμ≤max
τ∈L(G)θTτ (22.146)
If the solution is integral, it is exact; if it is fractional, it is an approximation. This is called
a (ﬁrst order) linear programming relaxtion. The reason it is called ﬁrst-order is that the
constraints that are enforced are those that correspond to consistency on a tree, which is agraph of treewidth 1. It is possible to enforce higher-order consistency, using graphs with largertreewidth (see (Wainwright and Jordan 2008b, sec 8.5) for details).
How should we actually perform the optimization? We can use a generic linear programming
package, but this is often very slow. Fortunately, in the case of graphical models, it is possible todevise specialised distributed message passing algorithms for solving this optimization problem,as we explain below.
22.6.2 Max-product belief propagation

The MAP objective in Equation 22.145, maxμ∈M(G)θTμ, is almost identical to the inference
objective in Equation 22.23, maxμ∈M(G)θTμ+H(μ), apart from the entropy term. One
heuristic way to proceed would be to consider the zero temperature limit of the probability
distribution μ, where the probability distribution has all its mass centered on its mode (see

Section 4.2.2). In such a setting, the entropy term becomes zero. We can then modify themessage passing methods used to solve the inference problem so that they solve the MAPestimation problem instead. In particular, in the zero temperature limit, the sum operatorbecomes the max operator, which results in a method called max-product belief propagation.
In more detail, let
A(θ)≜max
μ∈M(G)θTμ+H(μ) (22.147)
Now consider an inverse temperature βgoing to inﬁnity. We have
lim
β→+∞A(βθ)
β= lim
β→+∞1
βmax
μ∈M(G){
(βθ)Tμ+H(μ)}
(22.148)
=m a x
μ∈M(G){
θTμ+ lim
β→+∞1
βH(μ)}
(22.149)
=m a x
μ∈M(G)θTμ (22.150)

A(θ)≜max
lim
β→+∞A(βθ)
β= lim
β→+∞1
βmax
μ∈M(G){
(22.148)
=m a x
μ∈M(G){
θTμ+ lim
β→+∞1
βH(μ)}
(22.149)
=m a x
It is the concavity of the objective function that allows us to interchange the limandmax
operators (see (Wainwright and Jordan 2008b, p274) for details).
Now consider the Bethe approximation, which has the form maxτ∈L(G)θTτ+HBethe(τ).

lim
β→+∞A(βθ)
β= lim
β→+∞1
βmax
μ∈M(G){
(22.148)
=m a x
μ∈M(G){
θTμ+ lim
β→+∞1
βH(μ)}
(22.149)
=m a x
We showed that loopy BP ﬁnds a local optimum of this objective. In the zero temperature limit,this objective is equivalent to the LP relaxation of the MAP problem. Unfortunately, max-productloopy BP does not solve this LP relaxation unless the graph is a tree (Wainwright and Jordan2008b, p211). The reason is that Bethe energy functional is not concave (except on trees), so weare not licensed to swap the limit and max operators in the above zero-temperature derivation.However, if we use tree-reweighted BP, or TRBP/ TRW, we have a concave objective. In this case,

22.6. MAPstateestimation 801
one can show (Kolmogorov and Wainwright 2005) that the max-product version of TRBP does
solve the above LP relaxation.
A certain scheduling of this algorithm, known as sequential TRBP, TRBP-S,o rTRW-S, can
be shown to always converge (Kolmogorov 2006), and furthermore, it typically does so fasterthan the standard parallel updates. The idea is to pick an arbitrary node ordering X
1,...,X N.
We then consider a set of trees which is a subsequence of this ordering. At each iteration, weperform max-product BP from X
1towardsXNand back along one of these trees. It can be
shown that this monotonically minimizes a lower bound on the energy, and thus is guaranteedto converge to the global optimum of the LP relaxation.
22.6.3 Graphcuts
In this section, we show how to ﬁnd MAP state estimates, or equivalently, minimum energyconﬁgurations, by using the max ﬂow/min cut algorithm for graphs.
8This class of methods is

known as graphcuts and is very widely used, especially in computer vision applications.
We will start by considering the case of MRFs with binary nodes and a restricted class of
potentials; in this case, graphcuts will ﬁnd the exact global optimum. We then consider thecase of multiple states per node, which are assumed to have some underlying ordering; we canapproximately solve this case by solving a series of binary subproblems, as we will see.
22.6.3.1 Graphcuts for the generalized Ising model
Let us start by considering a binary MRF where the edge energies have the following form:
E
uv(xu,xv)={0ifxu=xv

E
λstifxu̸=xv(22.151)
whereλst≥0is the edge cost. This encourages neighboring nodes to have the same value
(since we are trying to minimize energy). Since we are free to add any constant we like to theoverall energy without affecting the MAP state estimate, let us rescale the local energy termssuch that either E
u(1) = 0orEu(0) = 0.
Now let us construct a graph which has the same set of nodes as the MRF, plus two distin-
guished nodes: the source sand the sinkt.I f Eu(1) = 0, we add the edge xu→twith cost

Eu(0). (This ensures that if uis not in partition Xt, meaning uis assigned to state 0, we will
pay a cost of Eu(0)in the cut.) Similarly, If Eu(0) = 0, we add the edge xu→swith cost
Eu(1). Finally, for every pair of variables that are connected in the MRF, we add edges xu→xv
andxv→xu, both with cost λu,v≥0. Figure 22.14 illustrates this construction for an MRF
with 4 nodes, and with the following non-zero energy values:
E1(0) = 7,E 2(1) = 2,E 3(1) = 1,E4(1) = 6 (22.152)
λ1,2=6,λ2,3=6,λ3,4=2,λ1,4=1 (22.153)

Having constructed the graph, we compute a minimal s−tcut. This is a partition of the nodes
into two sets,Xs, which are nodes connected to s, andXt, which are nodes connected to t.W e
8. There are a variety of ways to implement this algorithm, see e.g., (Sedgewick and Wayne 2011). The best take
O(EVlogV)orO(V3)time, where Eis the number of edges and Vis the number of nodes.

802 Chapter22. Morevariationalinference
z1
z4z2
z3
st
1
12
26
6
67
Figure 22.14 Illustration of graphcuts applied to an MRF with 4 nodes. Dashed lines are ones which
contribute to the cost of the cut (for bidirected edges, we only count one of the costs). Here the min cut
has cost 6. Source: Figure 13.5 from (Koller and Friedman 2009). Used with kind permission of DaphneKoller.
pick the partition which minimizes the sum of the cost of the edges between nodes on different
sides of the partition:
cost(Xs,Xt)=∑
xu∈Xs,xv∈Xtcost(xu,sv) (22.154)
In Figure 22.14, we see that the min-cut has cost 6.
Minimizing the cost in this graph is equivalent to minimizing the energy in the MRF. Hence
nodes that are assigned to shave an optimal state of 0, and the nodes that are assigned to t
have an optimal state of 1. In Figure 22.14, we see that the optimal MAP estimate is (1,1,1,0).
22.6.3.2 Graphcuts for binary MRFs with submodular potentials

We now discuss how to extend the graphcuts construction to binary MRFs with more generalkinds of potential functions. In particular, suppose each pairwise energy satisﬁes the followingcondition:
E
uv(1,1)+Euv(0,0)≤Euv(1,0)+Euv(0,1) (22.155)
In other words, the sum of the diagonal energies is less than the sum of the off-diagonal energies.In this case, we say the energies are submodular (Kolmogorov and Zabin 2004).
9An example
of a submodular energy is an Ising model where λuv>0. This is also known as an attractive

E
MRForassociative MRF, since the model “wants” neighboring states to be the same.
9. Submodularity is the discrete analog of convexity. Intuitively, it corresponds to the “law of diminishing returns”, that
is, the extra value of adding one more element to a set is reduced if the set is already large. More formally, we say that
f:2S→Ris submodular if for any A⊂B⊂Sandx∈S,w eh a v ef (A∪{x})−f(A)≥f(B∪{x})−f(B).
If−fis submodular, then fissupermodular.

22.6. MAPstateestimation 803
To apply graphcuts to a binary MRF with submodular potentials, we construct the pairwise
edge weights as follows:
E′
u,v(0,1) =Eu,v(1,0)+Eu,v(0,1)−Eu,v(0,0)−Eu,v(1,1) (22.156)
This is guaranteed to be non-negative by virtue of the submodularity assumption. In addition,
we construct new local edge weights as follows: ﬁrst we initialize E′(u)=E(u), and then for
each edge pair (u,v), we update these values as follows:
E′
u(1) = E′
u(1)+(Eu,v(1,0)−Eu,v(0,0)) (22.157)
E′
v(1) = E′
u(1)+(Eu,v(1,1)−Eu,v(1,0)) (22.158)
We now construct a graph in a similar way to before. Speciﬁcally, if E′
u(1)>E′
u(0),w e
add the edge u→swith cost E′
u(1)−E′
u(0), otherwise we add the edge u→twith cost
E′
u(0)−E′
u(1). Finally for every MRF edge for which E′
u,v(0,1)>0, we add a graphcuts edge
xu−xvwith cost E′
u,v(0,1). (We don’t need to add the edge in both directions.)
One can show (Exercise 22.1) that the min cut in this graph is the same as the minimum

E′
v(1) = E′
u(1)>E′
u(0),w e
u(1)−E′
E′
u(0)−E′
energy conﬁguration. Thus we can use max ﬂow/min cut to ﬁnd the globally optimal MAPestimate (Greig et al. 1989).
22.6.3.3 Graphcuts for nonbinary metric MRFs
We now discuss how to use graphcuts for approximate MAP estimation in MRFs where eachnode can have multiple states (Boykov et al. 2001). However, we require that the pairwise energiesform a metric. We call such a model a metric MRF. For example, suppose the states have a

natural ordering, as commonly arises if they are a discretization of an underlying continuousspace. In this case, we can deﬁne a metric of the form E(x
s,xt)=m i n ( δ,||xs−xt||)or a
semi-metric of the form E(xs,xt)=m i n ( δ,(xs−xt)2), for some constant δ>0. This energy
encourages neighbors to have similar labels, but never “punishes” them by more than δ. Thisδ
term prevents over-smoothing, which we illustrate in Figure 19.20.
One version of graphcuts is the alpha expansion. At each step, it picks one of the available

labels or states and calls it α; then it solves a binary subproblem where each variable can choose
to remain in its current state, or to become state α(see Figure 22.15(d) for an illustration). More
precisely, we deﬁne a new MRF on binary nodes, and we deﬁne the energies of this new model,relative to the current assignment x, as follows:
E
′
u(0) =Eu(xu),E′
u(1) =Eu(α),E′
u,v(0,0) =Eu,v(xu,xv)(22.159)
E′
u,v(0,1) =Eu,v(xu,α),E′
u,v(1,0) =Eu,v(α,xv),E′
u,v(1,1) =Eu,v(α,α) (22.160)
To optimize E′using graph cuts (and thus ﬁgure out the optimal alpha expansion move),

E
′
E′
we require that the energies be submodular. Plugging in the deﬁnition we get the followingconstraint:
E
u,v(xu,xv)+Eu,v(α,α)≤Eu,v(xu,α)+Eu,v(α,xv) (22.161)
For any distance function, Eu,v(α,α)=0 , and the remaining inequality follows from the
triangle inequality. Thus we can apply the alpha expansion move to any metric MRF.

804 Chapter22. Morevariationalinference
(a) initial labeling (b) standard move (c) α-β-swap (d) α-expansion
Figure 22.15 (a) An image with 3 labels. (b) A standard local move (e.g., by iterative conditional modes)
just ﬂips the label of one pixel. (c) An α−βswap allows all nodes that are currently labeled as αto
be relabeled as βif this decreases the energy. (d) An αexpansion allows all nodes that are not currently
labeled as αto be relabeled as αif this decreases the energy. Source: Figure 2 of (Boykov et al. 2001).
Used with kind permission of Ramin Zabih.
At each step of alpha expansion, we ﬁnd the optimal move from amongst an exponentially
large set; thus we reach a strong local optimum , of much lower energy than the local optima
found by standard greedy label ﬂipping methods such as iterative conditional modes. In fact,
one can show that, once the algorithm has converged, the energy of the resulting solution is at
most2ctimes the optimal energy, where
c=m a x
(u,v)∈Emaxα̸=βEuv(α,β)

c=m a x
minα̸=βEuv(α,β)(22.162)
See Exercise 22.3 for the proof. In the case of the Potts model, c=1,s ow eh a v ea2 -
approximation.
Another version of graphcuts is the alpha-beta swap . At each step, two labels are chosen,
call them αandβ. All the nodes currently labeled αcan change to β(and vice versa) if this
reduces the energy (see Figure 22.15(c) for an illustration). The resulting binary subproblem can
be solved exactly, even if the energies are only semi-metric (that is, the triangle inequality need

not hold; see Exercise 22.2). Although the α−βswap version can be applied to a broader class
of models than the α-expansion version, it is theoretically not as powerful. Indeed, in various
low-level vision problems, (Szeliski et al. 2008) show empirically that the expansion version is
usually better than the swap version (see Section 22.6.4).
22.6.4 Experimental comparison of graphcuts and BP
In Section 19.6.2.7, we described lattice-structured CRFs for various low-level vision problems.

(Szeliski et al. 2008) performed an extensive comparison of different approximate optimization
techniques for this class of problems. Some of the results, for the problem of stereo depth
estimation, are shown in Figure 22.16. We see that the graphcut and tree-reweighted max-
product BP (TRW) give the best results, with regular max-product BP being much worse. In terms
of speed, graphcuts is the fastest, with TRW a close second. Other algorithms, such as ICM,

simulated annealing or a standard domain-speciﬁc heuristic known as normalize correlation, are

22.6. MAPstateestimation 805
Max-Product BP
a-Expansion
a-b Swap
TRW
a-b Swap
TRW2
1.9
1.8
1.7
1.6
1.5
1.4
1.3Energy
Running Time (s)100×106
1011021034.2
4.1
4
3.9
3.8
3.7
3.6Energy
Running Time (s)100×105
101102
Figure 22.16 Energy minimization on a CRF for stereo depth estimation. Top row: two input images along
with the ground truth depth values. Bottom row: energy vs time for 4 different optimization algorithms.
Bottom left: results are for the Teddy image (shown in top row). Bottom right: results are for the Tsukuba
image (shown in Figure 22.17(a)). Source: Figure 13.B.1 of (Koller and Friedman 2009). Used with kind
permission of Daphne Koller.
even worse, as shown qualitatively in Figure 22.17.
Since TRW is optimizing the dual of the relaxed LP problem, we can use its value at conver-
gence to evaluate the optimal energy. It turns out that for many of the images in the stereo
benchmark dataset, the ground truth has higher energy (lower probability) than the globally op-

timal estimate (Meltzer et al. 2005). This indicates that we are optimizing the wrong model. This
is not surprising, since the pairwise CRF ignores known long-range constraints. Unfortunately,
if we add these constraints to the model, the graph either becomes too dense (making BP slow),
and/or the potentials become non-submodular (making graphcuts inapplicable).
One way around this is to generate a diverse set of local modes, using repeated applications
of graph cuts, as described in (Yadollahpour et al. 2011). We can then apply a more sophisticated

model, which uses global features, to rerank the solutions.

806 Chapter22. Morevariationalinference
(a) Left image: 384x288, 15 labels (b) Ground truth
(c) Swap algorithm (d) Expansion algorithm
(e) Normalized correlation (f) Simulated annealing
Figure 22.17 An example of stereo depth estimation using an MRF. (a) Left image, of size 384×288
pixels, from the University of Tsukuba. (The corresponding right image is similar, but not shown.) (b)
Ground truth depth map, quantized to 15 levels. (c-f): MAP estimates using different methods: (c) α−β
swap, (d) αexpansion, (e) normalized cross correlation, (f) simulated annealing. Source: Figure 10 of
(Boykov et al. 2001). Used with kind permission of Ramin Zabih.
22.6.5 Dual decomposition
We are interested in computing
p∗=m a x
x∈Xm∑
f∈Fθf(xf) (22.163)
whereFrepresents a set of factors. We will assume that we can tractably optimize each local
factor, but the combination of all of these factors makes the problem intractable. One way to

p∗=m a x
x∈Xm∑
proceed is to optimize each term independently, but then to introduce constraints that force all
the local estimates of the variables’ values to agree with each other. We explain this in more
detail below, following the presentation of (Sontag et al. 2011).

22.6. MAPstateestimation 807
x1 x2
x3 x4θf(x1,x2)
θh(x2,x4)
θk(x3,x4)θg(x1,x3)
(a)x1 x2
x3 x4xf
1 xf2
xg1
xg3xh
2
xh4
xk3 xk4
θk(xk3,xk4)θh(xh2,xh4)θf(xf
1,xf2)
== =
=
=
===
θg(xg
1,xg3)
(b)
Figure 22.18 (a) A pairwise MRF with 4 different edge factors. (b) We have 4 separate variables, plus a
copy of each variable for each factor it participates in. Source: Figure 1.2-1.3 of (Sontag et al. 2011). Used
with kind permission of David Sontag.
22.6.5.1 Basic idea
Let us duplicate the variables xi, once for each factor, and then force them to be equal.
Speciﬁcally, let xf
f={xf
i}i∈fbe the set of variables used by factor f. This construction is
illustrated in Figure 22.18. We can reformulate the objective as follows:
p∗=m a x
x,xf∑
f∈Fθf(xf
f)s.t.xf
i=xi∀f,i∈f (22.164)
Let us now introduce Lagrange multipliers, or dual variables, δfi(k), to enforce these constraints.
The Lagrangian becomes
L(δ,x,xf)=∑
f∈Fθf(xf
f) (22.165)
+∑
f∈F∑
i∈f∑
ˆxiδfi(ˆxi)(
I(xi=ˆxi)−I(xf
i=ˆxi))
(22.166)

f={xf
p∗=m a x
x,xf∑
f∈Fθf(xf
f)s.t.xf
f∈Fθf(xf
+∑
f∈F∑
i∈f∑
i=ˆxi))
(22.166)
This is equivalent to our original problem in the following sense: for any value of δ,w eh a v e
p∗=m a x
x,xfL(δ,x,xf)s.t.xfi=xi∀f,i∈f (22.167)
since if the constraints hold, the last term is zero. We can get an upper bound by dropping the
consistency constraints, and just optimizing the following upper bound:
L(δ)≜max
x,xfL(δ,x,xf) (22.168)
=∑
imax
xi⎛
⎝θi(xi)+∑
f:i∈fδfi(xi)⎞⎠+∑
fmax
xf⎛⎝θ
f(xf)−∑
i∈fδfi(xi)⎞⎠(22.169)
See Figure 22.19 for an illustration.

808 Chapter22. Morevariationalinference
x1δf2(x2)
δf1(x1)
δk4(x4) δk3(x3)δg1(x1)+
−−−−δf1(x1)
δg3(x3)
δg1(x1)−
−δh2(x2)
δh4(x4)−
−
+x3δg3(x3)
δk3(x3)x4+δk4(x4)
δh4(x4)+x2δf2(x2)
δh2(x2)θf(x1,x2)
θh(x2,x4)
x3 x4x4x2x2 x1
x1
x3
Figure 22.19 Illustration of dual decomposition. Source: Figure 1.2 of (Sontag et al. 2011). Used with
kind permission of David Sontag.
This objective is tractable to optimize, since each xfterm is decoupled. Furthermore, we see
thatL(δ)≥p∗, since by relaxing the consistency constraints, we are optimizing over a larger
space. Furthermore, we have the property that
min
δL(δ)=p∗(22.170)
so the upper bound is tight at the optimal value of δ, which enforces the original constraints.
Minimizing this upper bound is known as dual decomposition orLagrangian relaxation
(Komodakis et al. 2011; Sontag et al. 2011; Rush and Collins 2012). Furthemore, it can be shown
thatL(δ)is the dual to the same LP relaxation we saw before. We will discuss several possible

min
optimization algorithms below.
The main advantage of dual decomposition from a practical point of view is that it allows
one to mix and match different kinds of optimization algorithms in a convenient way. Forexample, we can combine a grid structured graph with local submodular factors to performimage segmentation, together with a tree structured model to perform pose estimation (seeExercise 22.4). Analogous methods can be used in natural language processing, where we oftenhave a mix of local and global constraints (see e.g., (Koo et al. 2010; Rush and Collins 2012)).

22.6.5.2 Theoretical guarantees
What can we say about the quality of the solutions obtained in this way? To understand this, letus ﬁrst introduce some more notation:
θδ
i(xi)≜θi(xi)+∑
f:i∈fδfi(xi) (22.171)
θδf(xf)≜θf(xf)−∑
i∈fδfi(xi) (22.172)

22.6. MAPstateestimation 809
This represents a reparameterization of the original problem, in the sense that
∑
iθi(xi)+∑
fθf(xf)=∑
iθδ
i(xi)+∑
fθδf(xf) (22.173)
and hence
L(δ)=∑
imax
xiθδi(xi)+∑
fmax
xfθδf(xf) (22.174)
Now suppose there is a set of dual variables δ∗and an assignment x∗such that the maxi-
mizing assignments to the singleton terms agrees with the assignments to the factor terms, i.e.,
so thatx∗
i∈argmaxxiθδ∗
i(xi)andx∗
f∈argmaxxfθδ∗
f(xf). In this case, we have
L(δ∗)=∑
iθδ∗
i(x∗
i)+∑
fθδ∗
f(x∗f)=∑
iθi(x∗i)+∑
fθf(x∗f) (22.175)
Now since
∑
fθf(x∗f)≤p∗≤L(δ∗) (22.176)
we conclude that L(δ∗)=p∗,s ox∗is the MAP assignment.
So if we can ﬁnd a solution where all the subproblems agree, we can be assured that it is the
global optimum. This happens surprisingly often in practical problems.
22.6.5.3 Subgradient descent
L(δ)is a convex and continuous objective, but it is non-differentiable at points δwhereθδ
i(xi)

L(δ∗)=∑
iθδ∗
i(x∗
i)+∑
fθδ∗
f(x∗f)=∑
Now since
∑
i(xi)
orθδf(xf)have multiple optima. One approach is to use subgradient descent. This updates all
the elements of δat the same time, as follows:
δt+1
fi(xi)=δt
fi(xi)−αtgt
fi(xi) (22.177)
wheregtthe subgradient of L(δ)atδt. If the step sizes αtare set appropriately (see Sec-
tion 8.5.2.1), this method is guaranteed to converge to a global optimum of the dual. (See
(Komodakis et al. 2011) for details.)
One can show that the gradient is given by the following sparse vector. First let xs
i∈
argmaxxiθδt
i(xi)andxf

δt+1
fi(xi)=δt
i∈
f∈argmaxxfθδt
f(xf).N e x tl e t gfi(xi)=0for all elements. Finally,
ifxf
i̸=xs
i(so factor fdisagrees with the local term on how to set variable i), we setgfi(xsi)=
+1andgfi(xf
i)=−1. This has the effect of decreasing θδt
i(xs
i)and increasing θδt
i(xf
i),
bringing them closer to agreement. Similarly, the subgradient update will decrease the value of
θδt
f(xfi,xf\i)and increasing the value of θδt
f(xs
i,xf\i).
To compute the gradient, we need to be able to solve subproblems of the following form:
argmax

ifxf
i̸=xs
i(xs
i(xf
i),
θδt
f(xs
i,xf\i).
argmax
xfθδt
f(xf) = argmax
xf⎡
⎣θf(xf)−∑
i∈fδt
fi(xi)⎤⎦ (22.178)

810 Chapter22. Morevariationalinference
(In (Komodakis et al. 2011), these subproblems are called slaves, whereas L(δ)is called the
master.) Obviously if the scope of factor fis small, this is simple. For example, if each factor is
pairwise, and each variable has Kstates, the cost is just K2. However, there are some kinds of
global factors that also support exact and efficient maximization, including the following:
• Graphical models with low tree width.
• Factors that correspond to bipartite graph matchings (see e.g., (Duchi et al. 2007)). This
is useful for data association problems, where we must match up a sensor reading with
an unknown source. We can ﬁnd the maximal matching using the so-called Hungarianalgorithm in O(|f|
3)time (see e.g., (Padadimitriou and Steiglitz 1982)).
• Supermodular functions. We discuss this case in more detail in Section 22.6.3.2.
• Cardinality constraints. For example, we might have a factor over a large set of binary

variables that enforces that a certain number of bits are turned on; this can be useful in
problems such as image segmentation. In particular, suppose θf(xf)=0if∑
i∈fxi=L
andθf(xf)=−∞otherwise. We can ﬁnd the maximizing assignment in O(|f|log|f|)
time as follows: ﬁrst deﬁne ei=δfi(1)−δfi(0); now sort the ei; ﬁnally set xi=1for the
ﬁrstLvalues, and xi=0for the rest (Tarlow et al. 2010).
• Factors which are constant for all but a small set Sof distinguished values of xf. Then we

i∈fxi=L
can optimize over the factor in O(|S|)time (Rother et al. 2009).
22.6.5.4 Coordinate descent
An alternative to updating the entire δvector at once (albeit sparsely) is to update it using block
coordinate descent. By choosing the size of the blocks, we can trade off convergence speed withease of the local optimization problem.
One approach, which optimizes δ
fi(xi)for alli∈fand allxiat the same time (for a
ﬁxed factor f), is known as max product linear programming (Globerson and Jaakkola 2008).

Algorithmically, this is similar to belief propagation on a factor graph. In particular, we deﬁneδ
f→ias messages sent from factor fto variable i, and we deﬁne δi→fas messages sent from
variableito factorf. These messages can be computed as follows (see (Globerson and Jaakkola
2008) for the derivation):10
δi→f(xi)=θ i(xi)+∑
g̸=fδg→i(xi) (22.179)
δf→i(xi)=−δ i→f(xi)+1
|f|max
xf\i⎡
⎣θf(xf)+∑
j∈fδj→f(xj)⎤⎦ (22.180)
We then set the dual variables δ
fi(xi)to be the messages δf→i(xi).

|f|max
xf\i⎡
⎣θf(xf)+∑
For example, consider a 2×2grid MRF, with the following pairwise factors: θf(x1,x2),
θg(x1,x3),θh(x2,x4), andθk(x3,x4). The outgoing message from factor fto variable 2 is a
10. Note that we denote their δ−f
i(xi)byδi→f(xi).

22.6. MAPstateestimation 811
function of all messages coming into f, andf’s local factor:
δf→2(x2)=−δ2→f(x2)+1
2max
x1[θf(x1,x2)+δ1→f(x1)+δ2→f(x2)] (22.181)
Similarly, the outgoing message from variable 2 to factor fis a function of all the messages
sent into variable 2 from other connected factors (in this example, just factor h) and the local
potential:
δ2→f(x2)=θ2(2)+δh2(x2) (22.182)
The key computational bottleneck is computing the max marginals of each factor, where we
max out all the variables from xfexcept for xi, i.e., we need to be able to compute the following
max marginals efficiently:
max
xf\ih(xf\i,xi),h(xf\i,xi)≜θf(xf)+∑
j∈fδjf(xj) (22.183)
The difference from Equation 22.178 is that we are maxing over all but one of the variables. We

max
can solve this efficiently for low treewidth graphical models using message passing; we can alsosolve this efficiently for factors corresponding to bipartite matchings (Duchi et al. 2007) or tocardinality constraints (Tarlow et al. 2010). However, there are cases where maximizing over allthe variables in a factor’s scope is computationally easier than maximizing over all-but-one (see(Sontag et al. 2011, Sec 1.5.4) for an example); in such cases, we may prefer to use a subgradientmethod.

Coordinate descent is a simple algorithm that is often much faster at minimizing the dual than
gradient descent, especially in the early iterations. It also reduces the objective monotonically,and does not need any step size parameters. Unfortunately, it is not guaranteed to converge tothe global optimum, since L(δ)is convex but not strictly convex (which implies there may be

more than one globally optimizing value). One way to ensure convergence is to replace the maxfunction in the deﬁnition of L(δ)with the soft-max function, which makes the objective strictly
convex (see e.g., (Hazan and Shashua 2010) for details).
22.6.5.5 Recovering the MAP assignment
So far, we have been focussing on ﬁnding the optimal value of δ
∗. But what we really want is
the optimal value of x∗. In general, computing x∗fromδ∗is NP-hard, even if the LP relaxation

is tight and the MAP assignment is unique (Sontag et al. 2011, Theorem 1.4). (The troublesomecases arise when there are fractional assignments with the same optimal value as the MAPestimate.)
However, suppose that each
θδ∗
ihas a unique maximum, x∗
i; in this case, we say that δ∗is
locally decodable tox∗. One can show than in this case, the LP relaxation is unique and its
solution is indeed x∗. If many, but not all, of the nodes are uniquely decodable, we can “clamp”

θδ∗
the uniquely decodable ones to their MAP value, and then use exact inference algorithms to
ﬁgure out the optimal assignment to the remaining variables. Using this method, (Meltzer et al.2005) was able to optimally solve various stereo vision CRF estimation problems, and (Yanoveret al. 2007) was able to optimally solve various protein side-chain structure predicition problems.
Another approach is to use the upper bound provided by the dual in a branch and bound
search procedure (Geoffrion 1974).

812 Chapter22. Morevariationalinference
Exercises
Exercise 22.1 Graphcuts for MAP estimation in binary submodular MRFs
(Source: Ex. 13.14 of (Koller and Friedman 2009).). Show that using the graph construction described in
Section 22.6.3.2, the cost of the cut is equal to the energy of the corresponding assignment, up to anirrelevant constant. (Warning: this exercise involves a lot of algebraic book-keeping.)
Exercise 22.2 Graphcuts for alpha-beta swap
(Source: Ex. 13.15 of (Koller and Friedman 2009).). Show how the optimal alpha-beta swap can be found by
running min-cut on an appropriately constructed graph. More precisely,
a. Deﬁne a set of binary variables t
1,...,t nsuch that ti=0meansx′
i=α,ti=1ifx′i=β, and
x′i=xiis unchanged f xi̸=αandxi̸=β.
b. Deﬁne an energy function over the new variables such that E′(t)=E(x)+const .
c. Show that E′is submodular if Eis a semimetric.
Exercise 22.3 Constant factor optimality for alpha-expansion

(Source: Daphne Koller.). Let Xbe a pairwise metric Markov random ﬁeld over a graph G=(V,E).
Suppose that the variables are nonbinary and that the node potentials are nonnegative. Let Adenote the
set of labels for each X∈X. Though it is not possible to (tractably) ﬁnd the globally optimal assignment
x⋆in general, the α-expansion algorithm provides a method for ﬁnding assignments ˆxthat are locally
optimal with respect to a large set of transformations, i.e., the possible α-expansion moves.

Despite the fact that α-expansion only produces a locally optimal MAP assignment, it is possible to prove
that the energy of this assignment is within a known factor of the energy of the globally optimal solution
x⋆. In fact, this is a special case of a more general principle that applies to a wide variety of algorithms,
including max-product belief propagation and more general move-making algorithms: If one can provethat the solutions obtained by the algorithm are ‘strong local minima’, i.e., local minima with respect to

a large set of potential moves, then it is possible to derive bounds on the (global) suboptimality of thesesolutions, and the quality of the bounds will depend on the nature of the moves considered. (There is aprecise deﬁnition of ‘large set of moves’.)
Consider the following approach to proving the suboptimality bound for α-expansion.
a. Letˆxbe a local minimum with respect to expansion moves. For each α∈A,l e tV
α={s∈V|x⋆
s=
α},i.e., the set of nodes labelled αin the global minimum. Let x′be an assignment that is equal to

α={s∈V|x⋆
s=
x⋆onVαand equal to ˆxelsewhere; this is an α-expansion of ˆx. Verify that E(x⋆)≤E(ˆx)≤E(x′).
b. Building on the previous part, show that E(ˆx)≤2cE(x⋆),wherec=m a x (s,t)∈E(maxα̸=βεst(α,β)
minα̸=βεst(α,β))
andEdenotes the energy of an assignment.
Hint. Think about where x′agrees with ˆxand where it agrees with x⋆.
Exercise 22.4 Dual decomposition for pose segmentation
(Source: Daphne Koller.). Two important problems in computer vision are that of parsing articulated objects
(e.g., the human body), called pose estimation, and segmenting the foreground and the background, called

segmentation. Intuitively, these two problems are linked, in that solving either one would be easier if thesolution to the other were available. We consider solving these problems simultaneously using a jointmodel over human poses and foreground/background labels and then using dual decomposition for MAPinference in this model.
We construct a two-level model, where the high level handles pose estimation and the low level handles
pixel-level background segmentation. Let G=(V,E)be an undirected grid over the pixels. Each node

i∈Vrepresents a pixel. Suppose we have one binary variable x
ifor each pixel, where xi=1means that
pixeliis in the foreground. Denote the full set of these variables by x=(xi).

22.6. MAPstateestimation 813
In addition, suppose we have an undirected tree structure T=(V′,E′)on the parts. For each body
part, we have a discrete set of candidate poses that the part can be in, where each pose is characterized
by parameters specifying its position and orientation. (These candidates are generated by a procedureexternal to the algorithm described here.) Deﬁne y
jkto be a binary variable indicating whether body part
j∈V′is in conﬁguration k. Then the full set of part variables is given by y=(yjk), withj∈V′
andk=1,...,K,w h e r eJis the total number of body parts and Kis the number of candidate poses
for each part. Note that in order to describe a valid conﬁguration, ymust satisfy the constraint that∑K
k=1yjk=1for eachj.
Suppose we have the following energy function on pixels:
E1(x)=∑
i∈V1[xi=1 ]·θi+∑
(i,j)∈E1[xi̸=xj]·θij.
Assume that the θijarises from a metric (e.g., based on differences in pixel intensities), so this can be
viewed as the energy for a pairwise metric MRF with respect to G.

E1(x)=∑
We then have the following energy function for parts:
E2(y)=∑
p∈V′θp(yp)+∑
(p,q)∈E′θpq(yp,yq).
Since each part candidate yjkis assumed to come with a position and orientation, we can compute a
binary mask in the image plane. The mask assigns a value to each pixel, denoted by {wi
jk}i∈V,w h e r e
wi
jk=1if pixelilies on the skeleton and decreases as we move away. We can use this to deﬁne an
energy function relating the parts and the pixels:
E3(x,y)=∑
i∈V∑
j∈V′K∑
k=11[xi=0,yjk=1 ]·wi
jk.

E2(y)=∑
wi
E3(x,y)=∑
i∈V∑
j∈V′K∑
jk.
In other words, this energy term only penalizes the case where a part candidate is active but the pixel
underneath is labeled as background.
Formulate the minimization of E1+E2+E3as an integer program and show how you can use dual
decomposition to solve the dual of this integer program. Your solution should describe the decomposition
into slaves, the method for solving each one, and the update rules for the overall algorithm. Brieﬂy justifyyour design choices, particularly your choice of inference algorithms for the slaves.

23 Monte Carlo inference
23.1 Introduction
So far, we discussed various deterministic algorithms for posterior inference. These meth-
ods enjoy many of the beneﬁts of the Bayesian approach, while still being about as fast asoptimization-based point-estimation methods. The trouble with these methods is that they canbe rather complicated to derive, and they are somewhat limited in their domain of applicabil-ity (e.g., they usually assume conjugate priors and exponential family likelihoods, although see(Wand et al. 2011) for some recent extensions of mean ﬁeld to more complex distributions). Fur-thermore, although they are fast, their accuracy is often limited by the form of the approximationwhich we choose.
In this chapter, we discuss an alternative class of algorithms based on the idea of Monte Carlo
approximation, which we ﬁrst introduced in Section 2.7. The idea is very simple: generate some(unweighted) samples from the posterior, x
s∼p(x|D), and then use these to compute any

quantity of interest, such as a posterior marginal, p(x1|D), or the posterior of the difference of
two quantities, p(x1−x2|D), or the posterior predictive, p(y|D), etc. All of these quantities
can be approximated by E[f|D]≈1
S∑S
s=1f(xs)for some suitable function f.
By generating enough samples, we can achieve any desired level of accuracy we like. The main
issue is: how do we efficiently generate samples from a probability distribution, particularly in

S∑S
high dimensions? In this chapter, we discuss non-iterative methods for generating independentsamples. In the next chapter, we discuss an iterative method known as Markov Chain MonteCarlo, or MCMC for short, which produces dependent samples but which works well in highdimensions. Note that sampling is a large topic. The reader should consult other books, such as(Liu 2001; Robert and Casella 2004), for more information.
23.2 Sampling from standard distributions

We brieﬂy discuss some ways to sample from 1 or 2 dimensional distributions of standard form.These methods are often used as subroutines by more complex methods.
23.2.1 Using the cdf
The simplest method for sampling from a univariate distribution is based on the inverse prob-
ability transform.L e t Fbe a cdf of some distribution we want to sample from, and let F−1

816 Chapter23. MonteCarloinference
0 x0u1 F
Figure 23.1 Sampling using an inverse CDF. Figure generated by sampleCdf .
be its inverse. Then we have the following result.
Theorem 23.2.1. IfU∼U(0,1)isauniformrv,then F−1(U)∼F.
Proof.
Pr(F−1(U)≤x) = Pr( U≤F(x))(applying Fto both sides) (23.1)
=F(x)(because Pr(U≤y)=y (23.2)
where the ﬁrst line follows since Fis a monotonic function, and the second line follows since
Uis uniform on the unit interval.
Hence we can sample from any univariate distribution, for which we can evaluate its inverse
cdf, as follows: generate a random number u∼U(0,1)using apseudo random number
generator (see e.g., (Press et al. 1988) for details). Let urepresent the height up the yaxis. Then
“slide along” the xaxis until you intersect the Fcurve, and then “drop down” and return the
corresponding xvalue. This corresponds to computing x=F−1(u). See Figure 23.1 for an
For example, consider the exponential distribution
Expon(x| λ)≜λe−λxI(x≥0) (23.3)
The cdf is

The cdf is
F(x)=1−e−λxI(x≥0) (23.4)
whose inverse is the quantile function
F−1(p)=−ln(1−p)
λ(23.5)
By the above theorem, if U∼Unif(0,1), we know that F−1(U)∼Expon(λ) . Furthermore,
since1−U∼Unif(0,1)as well, we can sample from the exponential distribution by ﬁrst
sampling from the uniform and then transforming the results using −ln(u)/λ.

23.3. Rejectionsampling 817
23.2.2 Sampling from a Gaussian (Box-Muller method)
We now describe a method to sample from a Gaussian. The idea is we sample uniformly from
a unit radius circle, and then use the change of variables formula to derive samples from aspherical 2d Gaussian. This can be thought of as two samples from a 1d Gaussian.
In more detail, sample z
1,z2∈(−1,1)uniformly, and then discard pairs that do not satisfy
z2
1+z2
2≤1. The result will be points uniformly distributed inside the unit circle, so p(z)=
1
πI(zinside circle) . Now deﬁne
xi=zi(−2lnr2
r2)1
2
(23.6)
fori=1:2,w h e r er2=z2
1+z2
2. Using the multivariate change of variables formula, we have
p(x1,x2)=p(z1,z2)|∂(z1,z2)
∂(x1,x2)|=[1√
2πexp(−1
2x2
1)][1√
2πexp(−1
2x22)]
(23.7)
Hencex1andx2are two independent samples from a univariate Gaussian. This is known as
theBox-Muller method.
To sample from a multivariate Gaussian, we ﬁrst compute the Cholesky decomposition of its

1
r2)1
2
(23.6)
1+z2
2πexp(−1
2x2
1)][1√
2πexp(−1
2x22)]
(23.7)
covariance matrix, Σ=LLT,w h e r eLis lower triangular. Next we sample x∼N(0,I)using
the Box-Muller method. Finally we set y=Lx+μ. This is valid since
cov[y]=Lcov[x]LT=LILT=Σ (23.8)
23.3 Rejection sampling
When the inverse cdf method cannot be used, one simple alternative is to use rejection sam-
pling, which we now explain.
23.3.1 Basic idea
In rejection sampling, we create a proposal distribution q(x)which satisifes Mq(x)≥˜p(x),
for some constant M,w h e r e˜p(x)is an unnormalized version of p(x)(i.e.,p(x)=˜p(x)/Zp

for some possibly unknown constant Zp). The function Mq(x)provides an upper envelope for
˜p. We then sample x∼q(x), which corresponds to picking a random xlocation, and then
we sample u∼U(0,1), which corresponds to picking a random height (y location) under the
envelope. If u>˜p(x)
Mq(x), we reject the sample, otherwise we accept it. See Figure 23.2(a). where
the acceptance region is shown shaded, and the rejection region is the white region between
the shaded zone and the upper envelope.
We now prove that this procedure is correct. Let

S={(x,u):u≤˜p(x)/Mq(x)},S0={(x,u):x≤x0,u≤˜p(x)/Mq(x)} (23.9)

xuMq(x  )
Accept region Reject region
x  ~ q(x)(i)(i)(i)(i)
p(x  )Mq(x  )
(a)0 2 4 6 8 1000.20.40.60.811.21.4
  
target p(x)
comparison function Mq(x)
(b)
Figure 23.2 (a) Schematic illustration of rejection sampling. Source: Figure 2 of (Andrieu et al. 2003).
Used with kind permission of Nando de Freitas. (b) Rejection sampling from a Ga(α=5.7,λ=2 )
distribution (solid blue) using a proposal of the form MGa(k,λ−1)(dotted red), where k=⌊5.7⌋=5.
The curves touch at α−k=0.7. Figure generated by rejectionSamplingDemo .
Then the cdf of the accepted points is given by
P(x≤x0|xaccepted)=P(x≤x0,xaccepted)
P(xaccepted)(23.10)
=∫∫
I((x,u)∈S0)q(x)dudx∫∫
I((x,u)∈S)q(x)dudx=∫x0
−∞˜p(x)dx∫∞
−∞˜p(x)dx(23.11)
which is the cdf of p(x), as desired.
How efficient is this method? Since we generate with probability q(x)and accept with
probability˜p(x)
Mq(x), the probability of acceptance is
p(accept)=∫˜p(x)
Mq(x)q(x)dx=1
M∫
˜p(x)dx (23.12)
Hence we want to choose Mas small as possible while still satisfying Mq(x)≥˜p(x).

=∫∫
M∫
23.3.2 Example
For example, suppose we want to sample from a Gamma distribution:1
Ga(x|α,λ)=1
Γ(α)xα−1λαexp(−λx) (23.13)
One can show that if Xiiid∼Expon(λ), andY=X1+···+Xk, thenY∼Ga(k,λ).F o r
non-integer shape parameters, we cannot use this trick. However, we can use rejection sampling
1. This section is based on notes by Ioana A. Cosma, available at http://users .aims.ac.za/~ioana/cp2 .pdf.

23.3. Rejectionsampling 819
(a)−8 −6 −4 −2 0 2 4 6 800.10.20.30.40.50.60.70.80.91f(x) half−gaussian
(b)−8 −6 −4 −2 0 2 4 6 801002003004005006007008009001000samples from f(x) (by ARS)
(c)
Figure 23.3 (a) Idea behind adaptive rejection sampling. We place piecewise linear upper (and lower)
bounds on the log-concave density. Based on Figure 1 of (Gilks and Wild 1992). Figure generated by
arsEnvelope . (b-c) Using ARS to sample from a half-Gaussian. Figure generated by arsDemo, written by
Daniel Eaton.
using aGa(k,λ−1)distribution as a proposal, where k=⌊α⌋. The ratio has the form
p(x)
q(x)=Ga(x|α,λ)
Ga(x|k,λ−1)=xα−1λαexp(−λx)/Γ(α)
xk−1(λ−1)kexp(−(λ−1)x)/Γ(k)(23.14)
=Γ(k)λα
Γ(α)(λ−1)kxα−kexp(−x) (23.15)
This ratio attains its maximum when x=α−k. Hence
M=Ga(α−k|α,λ)
Ga(α−k|k,λ−1)(23.16)
See Figure 23.2(b) for a plot. (Exercise 23.2 asks you to devise a better proposal distribution
based on the Cauchy distribution.)
23.3.3 Application to Bayesian statistics

p(x)
=Γ(k)λα
Supposewewanttodraw(unweighted)samplesfromtheposterior, p(θ|D)=p(D|θ)p(θ)/p(D).
We can use rejection sampling with ˜p(θ)=p(D|θ)p(θ)as the target distribution, q(θ)=p(θ)
as our proposal, and M=p(D|ˆθ),w h e r eˆθ= argmax p(D|θ)is the MLE; this was ﬁrst
suggested in (Smith and Gelfand 1992). We accept points with probability
˜p(θ)
Mq(θ)=p(D|θ)
p(D|ˆθ)(23.17)

˜p(θ)
Thus samples from the prior that have high likelihood are more likely to be retained in theposterior. Of course, if there is a big mismatch between prior and posterior (which will be thecase if the prior is vague and the likelihood is informative), this procedure is very inefficient. Wediscuss better algorithms later.
23.3.4 Adaptive rejection sampling
We now describe a method that can automatically come up with a tight upper envelope q(x)
to any log concave density p(x). The idea is to upper bound the log density with a piecewise

820 Chapter23. MonteCarloinference
linear function, as illustrated in Figure 23.3(a). We choose the initial locations for the pieces
based on a ﬁxed grid over the support of the distribution. We then evaluate the gradient of thelog density at these locations, and make the lines be tangent at these points.
Since the log of the envelope is piecewise linear, the envelope itself is piecewise exponential:
q(x)=M
iλiexp(−λi(x−xi−1)),xi−1<x≤xi (23.18)
wherexiare the grid points. It is relatively straightforward to sample from this distribution. If
the sample xis rejected, we create a new grid point at x, and thereby reﬁne the envelope. As
the number of grid points is increased, the tightness of the envelope improves, and the rejectionrate goes down. This is known as adaptive rejection sampling (ARS) (Gilks and Wild 1992).
Figure 23.3(b-c) gives an example of the method in action. As with standard rejection sampling,it can be applied to unnormalized distributions.
23.3.5 Rejection sampling in high dimensions

It is clear that we want to make our proposal q(x)as close as possible to the target distribution
p(x), while still being an upper bound. But this is quite hard to achieve, especially in high
dimensions. To see this, consider sampling from p(x)=N(0,σ2
pI)using as a proposal
q(x)=N(0,σ2
qI). Obviously we must have σ2
q≥σ2
pin order to be an upper bound. In D
dimensions, the optimum value is given by M=(σq/σp)D. The acceptance rate is 1/M(since
bothpandqare normalized), which decreases exponentially fast with dimension. For example,

q≥σ2
ifσqexceedsσpby just 1%, then in 1000 dimensions the acceptance ratio will be about 1/20,000.
This is a fundamental weakness of rejection sampling.
In Chapter 24, we will describe MCMC sampling, which is a more efficient way to sample
from high dimensional distributions. Sometimes this uses (adaptive) rejection sampling as asubroutine, which is known as adaptive rejection Metropolis sampling (Gilks et al. 1995).
23.4 Importance sampling
We now describe a Monte Carlo method known as importance sampling for approximating
integrals of the form
I=E[f]=∫
f(x)p(x)dx (23.19)

I=E[f]=∫
23.4.1 Basic idea
The idea is to draw samples xin regions which have high probability, p(x), but also where
|f(x)|is large. The result can be super efficient, meaning it needs less samples than if we
were to sample from the exact distribution p(x). The reason is that the samples are focussed
on the important parts of space. For example, suppose we want to estimate the probability ofarare event. Deﬁne f(x)=I(x∈E), for some set E. Then it is better to sample from a
proposal of the form q(x)∝f(x)p(x)than to sample from p(x)itself.

Importance sampling samples from any proposal, q(x). It then uses these samples to estimate

23.4. Importancesampling 821
the integral as follows:
E[f]=∫
f(x)p(x)
q(x)q(x)dx≈1
SS∑
s=1wsf(xs)=ˆI (23.20)
wherews≜p(xs)
q(xs)are theimportance weights. Note that, unlike rejection sampling, we use all
the samples.
How should we choose the proposal? A natural criterion is to minimize the variance of the
estimateˆI=∑
swsf(xs).N o w
varq(x)[f(x)w(x)] =Eq(x)[
f2(x)w2(x)]
−I2(23.21)
Since the last term is independent of q, we can ignore it. By Jensen’s inequality, we have the
Eq(x)[
≥(Eq(x)[|f(x)w(x)|])2=(∫
|f(x)|p(x)dx)2
(23.22)
The lower bound is obtained when we use the optimal importance distribution:
q∗(x)=|f(x)|p(x)∫
|f(x′)|p(x′)dx′(23.23)
When we don’t have a particular target function f(x)in mind, we often just try to make
q(x)as close as possible to p(x). In general, this is difficult, especially in high dimensions, but
it is possible to adapt the proposal distribution to improve the approximation. This is known as
adaptive importance sampling (Oh and Berger 1992).

(23.22)
23.4.2 Handling unnormalized distributions
It is frequently the case that we can evaluate the unnormalized target distribution, ˜p(x), but not
its normalization constant, Zp. We may also want to use an unnormalized proposal, ˜q(x), with
possibly unknown normlization constant Zq. We can do this as follows. First we evaluate
E[f]=Zq
Zp∫
f(x)˜p(x)
˜q(x)q(x)dx≈Zq
Zp1
SS∑
s=1˜wsf(xs) (23.24)
where˜ws≜˜p(xs)
˜q(xs)is the unnormalized importance weight. We can use the same set of samples

E[f]=Zq
Zp∫
f(x)˜p(x)
Zp1
SS∑
to evaluate the ratio Zp/Zqas follows:
Zp
Zq=1
Zq∫
˜p(x)dx=∫˜p(x)
˜q(x)q(x)dx≈1
SS∑
s=1˜ws (23.25)
Hence
ˆI=1
S∑
s˜wsf(xs)
1
S∑
s˜ws=S∑
s=1wsf(xs) (23.26)

822 Chapter23. MonteCarloinference
where
ws≜˜ws∑
s′˜ws′(23.27)
are the normalized importance weights. The resulting estimate is a ratio of two estimates, and
hence is biased. However, as S→∞, we have that ˆI→I, under weak assumptions (see e.g.,
(Robert and Casella 2004) for details).
23.4.3 Importance sampling for a DGM: likelihood weighting
We now describe a way to use importance sampling to generate samples from a distributionwhich can be represented as a directed graphical model (Chapter 10).
If we have no evidence, we can sample from the unconditional joint distribution of a DGM
p(x)as follows: ﬁrst sample the root nodes, then sample their children, then sample their
children, etc. This is known as ancestral sampling. It works because, in a DAG, we can always
topologically order the nodes so that parents preceed children. (Note that there is no equivalenteasy method for sampling from an unconditional undirected graphical model.)

Now suppose we have some evidence, so some nodes are “clamped” to observed values, and
we want to sample from the posterior p(x|D). If all the variables are discrete, we can use the
following simple procedure: perform ancestral sampling, but as soon as we sample a value thatis inconsistent with an observed value, reject the whole sample and start again. This is knownaslogic sampling (Henrion 1988).
Needless to say, logic sampling is very inefficient, and it cannot be applied when we have

real-valued evidence. However, it can be modiﬁed as follows. Sample unobserved variables asbefore, conditional on their parents. But don’t sample observed variables; instead we just usetheir observed values. This is equivalent to using a proposal of the form
q(x)=∏
t̸∈Ep(xt|xpa(t))∏
t∈Eδx∗
t(xt) (23.28)
whereEis the set of observed nodes, and x∗
tis the observed value for node t. We should
therefore give the overall sample an importance weight as follows:
w(x)=p(x)
q(x)=∏
t̸∈Ep(xt|xpa(t))
p(xt|xpa(t))∏
t∈Ep(xt|xpa(t))
1=∏

q(x)=∏
t∈Eδx∗
w(x)=p(x)
q(x)=∏
1=∏
t∈Ep(xt|xpa(t)) (23.29)
This technique is known as likelihood weighting (Fung and Chang 1989; Shachter and Peot
1989).
23.4.4 Sampling importance resampling (SIR)
We can draw unweighted samples from p(x)by ﬁrst using importance sampling (with proposal
q) to generate a distribution of the form
p(x)≈∑
swsδxs(x) (23.30)

23.5. Particleﬁltering 823
wherewsare the normalized importance weights. We then sample with replacement from
Equation 23.30, where the probability that we pick xsisws. Let this procedure induce a
distribution denoted by ˆp. To see that this is valid, note that
ˆp(x≤x0)=∑
sI(xs≤x0)ws=∑
sI(xs≤x0)˜p(xs)/q(xs)∑
s˜p(xs)/q(xs)(23.31)
→∫
I(x≤x0)˜p(x)
q(x)q(x)dx
∫˜p(x)
q(x)q(x)dx(23.32)
=∫
I(x≤x0)˜p(x)dx∫
˜p(x)dx=∫
I(x≤x0)p(x)dx=p(x≤x0) (23.33)
This is known as sampling importance resampling (SIR) (Rubin 1998). The result is an un-
weighted approximation of the form
p(x)≈1
S′S′∑
s=1δxs(x) (23.34)
Note that we typically take S′≪S.
This algorithm can be used to perform Bayesian inference in low-dimensional settings (Smith
and Gelfand 1992). That is, suppose we want to draw (unweighted) samples from the posterior,
p(θ|D)=p(D|θ)p(θ)/p(D). We can use importance sampling with ˜p(θ)=p(D|θ)p(θ)as
the unnormalized posterior, and q(θ)=p(θ)as our proposal. The normalized weights have the
form
ws=˜p(θs)/q(θs)∑

p(x)≈1
S′S′∑
form
s′˜p(θs′)/q(θs′)=p(D|θs)∑
s′p(D|θs′)(23.35)
We can then use SIR to sample from p(θ|D).
Of course, if there is a big discrepancy between our proposal (the prior) and the target (the
posterior), we will need a huge number of importance samples for this technique to work reliably,since otherwise the variance of the importance weights will be very large, implying that mostsamples carry no useful information. (This issue will come up again in Section 23.5, when wediscuss particle ﬁltering.)
23.5 Particle ﬁltering

Particle ﬁltering (PF) is a Monte Carlo, or simulation based, algorithm for recursive Bayesian
inference. That is, it approximates the predict-update cycle described in Section 18.3.1. It isvery widely used in many areas, including tracking, time-series forecasting, online parameterlearning, etc. We explain the basic algorithm below. For a book-length treatment, see (Doucetet al. 2001); for a good tutorial, see (Arulampalam et al. 2002), or just read on.

824 Chapter23. MonteCarloinference
23.5.1 Sequential importance sampling
The basic idea is to appproximate the belief state (of the entire state trajectory) using a weighted
set of particles:
p(z1:t|y1:t)≈S∑
s=1ˆws
tδzs
1:t(z1:t) (23.36)
whereˆws
tis the normalized weight of sample sat timet. From this representation, we can
easily compute the marginal distribution over the most recent state, p(zt|y1:t), by simply
ignoring the previous parts of the trajectory, z1:t−1. (The fact that PF samples in the space of
entire trajectories has various implications which we will discuss later.)
We update this belief state using importance sampling. If the proposal has the form
q(zs
1:t|y1:t), then the importance weights are given by
ws
t∝p(zs1:t|y1:t)
q(zs1:t|y1:t)(23.37)
which can be normalized as follows:
ˆws
t=ws
t∑
s′ws′
t(23.38)
We can rewrite the numerator recursively as follows:
p(z1:t|y1:t)=p(yt|z1:t,y1:t−1)p(z1:t|y1:t−1)
p(yt|y1:t−1)(23.39)
=p(yt|zt)p(zt|z1:t−1,y1:t−1)p(z1:t−1|y1:t−1)
p(yt|y1:t−1)(23.40)

q(zs
ws
ˆws
t=ws
t∑
s′ws′
t(23.38)
∝p(yt|zt)p(zt|zt−1)p(z1:t−1|y1:t−1) (23.41)
where we have made the usual Markov assumptions. We will restrict attention to proposal
densities of the following form:
q(z1:t|y1:t)=q(zt|z1:t−1,y1:t)q(z1:t−1|y1:t−1) (23.42)
so that we can “grow” the trajectory by adding the new state ztto the end. In this case, the
importance weights simplify to
ws
t∝p(yt|zs
t)p(zst|zs
t−1)p(zs1:t−1|y1:t−1)
q(zs
t|zs
1:t−1,y1:t)q(zs1:t−1|y1:t−1)(23.43)
=ws
t−1p(yt|zs
t−1)
q(zs
t|zs
1:t−1,y1:t)(23.44)

ws
t∝p(yt|zs
q(zs
t|zs
=ws
t−1)
q(zs
t|zs
If we further assume that q(zt|z1:t−1,y1:t)=q(zt|zt−1,yt), then we only need to keep the
most recent part of the trajectory and observation sequence, rather than the whole history, in
order to compute the new sample. In this case, the weight becomes
ws
t∝ws
t−1)
q(zs
t|zs
t−1,yt)(23.45)

23.5. Particleﬁltering 825
Hence we can approximate the posterior ﬁltered density using
p(zt|y1:t)≈S∑
s=1ˆws
tδzs
t(zt) (23.46)
AsS→∞, one can show that this approaches the true posterior (Crisan et al. 1999).
The basic algorithm is now very simple: for each old sample s, propose an extension using
zs
t∼q(zt|zs
t−1,yt), and give this new particle weight ws
tusing Equation 23.45. Unfortunately,
this basic algorithm does not work very well, as we discuss below.
23.5.2 The degeneracy problem
The basic sequential importance sampling algorithm fails after a few steps because most of
the particles will have negligible weight. This is called the degeneracy problem, and occurs
because we are sampling in a high-dimensional space (in fact, the space is growing in size overtime), using a myopic proposal distribution.
We can quantify the degree of degeneracy using the effective sample size, deﬁned by
Seff≜S
1+v ar[w∗s
t](23.47)
wherew∗s
t=p(zs
t|y1:t)/q(zst|zs

Seff≜S
1+v ar[w∗s
t](23.47)
wherew∗s
t=p(zs
t−1,yt)is the “true weight” of particle s. This quantity cannot
be computed exactly, since we don’t know the true posterior, but we can approximate it using
ˆSeff=1∑S
s=1(ws
t)2(23.48)
If the variance of the weights is large, then we are wasting our resources updating particles with
low weight, which do not contribute much to our posterior estimate.
There are two main solutions to the degeneracy problem: adding a resampling step, and using
a good proposal distribution. We discuss both of these in turn.

ˆSeff=1∑S
s=1(ws
23.5.3 The resampling step
The main improvement to the basic SIS algorithm is to monitor the effective sampling size,and whenever it drops below a threshold, to eliminate particles with low weight, and thento create replicates of the surviving particles. (Hence PF is sometimes called survival of the
ﬁttest(Kanazawa et al. 1995).) In particular, we generate a new set {z
s∗
t}S
s=1by sampling with
replacement Stimes from the weighted distribution
s=1ˆws
tδzs
t(zt) (23.49)

s∗
t}S
s=1ˆws
tδzs
where the probability of choosing particle jfor replication is wj
t. (This is sometimes called
rejuvenation.) The result is an iid unweighted sample from the discrete density Equation 23.49,
so we set the new weights to ws
t=1/S. This scheme is illustrated in Figure 23.4.

All these methods take O(S)time. See (Doucet et al. 2001) for details.
The overall particle ﬁltering algorithm is summarized in Algorithm 6. (Note that if an estimate
of the state is required, it should be computed before the resampling step, since this will result
in lower variance.)
Algorithm 23.1: One step of a generic particle ﬁlter
1fors=1:Sdo
2Drawzs
t∼q(zt|zs
t−1,yt);
3Compute weight ws
t∝ws
t)p(zst|zst−1)
q(zs
t|zs
t−1,yt);
4Normalize weights: ws
t=ws
t∑
s′ws′
t;
5Compute ˆSeff=1∑S
s=1(ws
t)2;
6ifˆSeff<Sminthen
7Resample Sindicesπ∼wt;
8z:
t=zπt;
9ws
t=1/S;
Although the resampling step helps with the degeneracy problem, it introduces problems of
its own. In particular, since the particles with high weight will be selected many times, there is
a loss of diversity amongst the population. This is known as sample impoverishment. In the

23.5. Particleﬁltering 827
extreme case of no process noise (e.g., if we have static but unknown parameters as part of the
state space), then all the particles will collapse to a single point within a few iterations.
To mitigate this problem, several solutions have been proposed. (1) Only resample when
necessary, not at every time step. (The original bootstrap ﬁlter (Gordon 1993) resampled at
every step, but this is suboptimal.) (2) After replicating old particles, sample new values usingan MCMC step which leaves the posterior distribution invariant (see e.g., the resample-move
algorithm in (Gilks and Berzuini 2001)). (3) Create a kernel density estimate on top of theparticles,
p(z
t|y1:t)≈S∑
s=1ws
tκ(zt−zs
t) (23.51)
whereκis some smoothing kernel. We then sample from this smoothed distribution. This is
known as a regularized particle ﬁlter (Musso et al. 2001). (4) When performing inference on
static parameters, add some artiﬁcial process noise. (If this is undesirable, other algorithms must

p(z
s=1ws
tκ(zt−zs
t) (23.51)
be used for online parameter estimation, e.g., (Andrieu et al. 2005)).
23.5.4 The proposal distribution
The simplest and most widely used proposal distribution is to sample from the prior:
q(zt|zs
t−1,yt)=p(zt|zst−1) (23.52)
In this case, the weight update simpliﬁes to
ws
t∝ws
t−1p(yt|zst) (23.53)
This can be thought of a “generate and test” approach: we sample values from the dynamic
model, and then evaluate how good they are after we see the data (see Figure 23.4). Thisis the approach used in the condensation algorithm (which stands for “conditional density

q(zt|zs
ws
t∝ws
propagation”) used for visual tracking (Isard and Blake 1998). However, if the likelihood isnarrower than the dynamical prior (meaning the sensor is more informative than the motionmodel, which is often the case), this is a very inefficient approach, since most particles will beassigned very low weight.
It is much better to actually look at the data y
twhen generating a proposal. In fact, the
optimal proposal distribution has the following form:
q(zt|zs
t−1,yt)=p(zt|zst−1,yt)=p(yt|zt)p(zt|zs
t−1)
p(yt|zst−1)(23.54)

q(zt|zs
t−1)
If we use this proposal, the new weight is given by
ws
t∝ws
t−1)=ws
t−1∫
p(yt|z′t)p(z′t|zst−1)dz′t(23.55)
This proposal is optimal since, for any given zs
t−1, the new weight ws
ttakes the same value
regardless of the value drawn for zs
t. Hence, conditional on the old values z.
t−1, the variance of
true weights var[w∗s
t],i sz e r o .

828 Chapter23. MonteCarloinference
In general, it is intractable to sample from p(zt|zs
t−1,yt)and to evaluate the integral needed
to compute the predictive density p(yt|zst−1). However, there are two cases when the optimal
proposal distribution can be used. The ﬁrst setting is when ztis discrete, so the integral becomes
a sum. Of course, if the entire state space is discrete, we can use an HMM ﬁlter instead, but
in some cases, some parts of the state are discrete, and some continuous. The second settingis whenp(z
t|zs
t−1,yt)is Gaussian. This occurs when the dynamics are nonlinear but the
observations are linear. See Exercise 23.3 for the details.
In cases where the model is not linear-Gaussian, we may still compute a Gaussian approxima-
tion top(zt|zst−1,yt)using the unscented transform (Section 18.5.2) and use this as a proposal.
This is known as the unscented particle ﬁlter (van der Merwe et al. 2000). In more general

t|zs
settings, we can use other kinds of data-driven proposals, perhaps based on discriminative
models. Unlike MCMC, we do not need to worry about the proposals being reversible.
23.5.5 Application: robot localization
Consider a mobile robot wandering around an office environment. We will assume that it already
has a map of the world, represented in the form of an occupancy grid, which just speciﬁes

whether each grid cell is empty space or occupied by an something solid like a wall. The goalis for the robot to estimate its location. This can be solved optimally using an HMM ﬁlter, sincewe are assuming the state space is discrete. However, since the number of states, K,i so f t e n
very large, the O(K
2)time complexity per update is prohibitive. We can use a particle ﬁlter as
a sparse approximation to the belief state. This is known as Monte Carlo localization, and is
described in detail in (Thrun et al. 2006).

Figure 23.5 gives an example of the method in action. The robot uses a sonar range ﬁnder,
so it can only sense distance to obstacles. It starts out with a uniform prior, reﬂecting the factthat the owner of the robot may have turned it on in an arbitrary location. (Figuring out whereyou are, starting from a uniform prior, is called global localization.) After the ﬁrst scan, which

indicates two walls on either side, the belief state is shown in (b). The posterior is still fairlybroad, since the robot could be in any location where the walls are fairly close by, such as acorridor or any of the narrow rooms. After moving to location 2, the robot is pretty sure it mustbe in the corridor, as shown in (c). After moving to location 3, the sensor is able to detect theend of the corridor. However, due to symmetry, it is not sure if it is in location I (the truelocation) or location II. (This is an example of perceptual aliasing, which refers to the fact that

different things may look the same.) After moving to locations 4 and 5, it is ﬁnally able to ﬁgureout precisely where it is. The whole process is analogous to someone getting lost in an officebuilding, and wandering the corridors until they see a sign they recognize.
In Section 23.6.3, we discuss how to estimate location and the map at the same time.
23.5.6 Application: visual object tracking
Our next example is concerned with tracking an object (in this case, a remote-controlled heli-copter) in a video sequence. The method uses a simple linear motion model for the centroidof the object, and a color histogram for the likelihood model, using Bhattacharya distance to
compare histograms. The proposal distribution is obtained by sampling from the likelihood. See(Nummiaro et al. 2003) for further details.

23.5. Particleﬁltering 829
3 4Room A
Room BStart 2 15
Room C
I
II
III(a) Path and reference poses
(c) Belief at reference pose 2
(e) Belief at reference pose 4(b) Belief at reference pose 1
(d) Belief at reference pose 3
(f) Belief at reference pose 5
Figure 23.5 Illustration of Monte Carlo localization. Source: Figure 8.7 of (Thrun et al. 2006). Used
with kind permission of Sebastian Thrun.
Figure 23.6 shows some example frames. The system uses S= 250particles, with an effective
sample size of ˆSeff= 134. (a) shows the belief state at frame 1. The system has had to resample
5 times to keep the effective sample size above the threshold of 150; (b) shows the belief state

at frame 251; the red lines show the estimated location of the center of the object over the last250 frames. (c) shows that the system can handle visual clutter, as long as it does not have thesame color as the target object. (d) shows that the system is confused between the grey of thehelicopter and the grey of the building. The posterior is bimodal. The green ellipse, representingthe posterior mean and covariance, is in between the two modes. (e) shows that the probabilitymass has shifted to the wrong mode: the system has lost track. (f) shows the particles spreadout over the gray building; recovery of the object is very unlikely from this state using this

830 Chapter23. MonteCarloinference
(a)
 (b)
(c)
 (d)
(e)
 (f)
Figure 23.6 Example of particle ﬁltering applied to visual object tracking, based on color histograms.
(a-c) succesful tracking: green ellipse is on top of the helicopter. (d-f): tracker gets distracted by gray clutter
in the background. See text for details. Figure generated by pfColorTrackerDemo , written by Sebastien
Paris.
proposal.
We see that the method is able to keep track for a fairly long time, despite the presence
of clutter. However, eventually it loses track of the object. Note that since the algorithm is
stochastic, simply re-running the demo may ﬁx the problem. But in the real world, this is not
an option. The simplest way to improve performance is to use more particles. An alternative
is to perform tracking by detection , by running an object detector over the image every few
frames. See (Forsyth and Ponce 2002; Szeliski 2010; Prince 2012) for details.

23.6. Rao-Blackwellisedparticleﬁltering(RBPF) 831
23.5.7 Application: time series forecasting
In Section 18.2.4, we discussed how to use the Kalman ﬁlter to perform time series forecasting.
This assumes that the model is a linear-Gaussian state-space model. There are many modelswhich are either non-linear and/or non-Gaussian. For example, stochastic volatility models,
which are widely used in ﬁnance, assume that the variance of the system and/or observationnoise changes over time. Particle ﬁltering is widely used in such settings. See e.g., (Doucet et al.2001) and references therein for details.
23.6 Rao-Blackwellised particle ﬁltering (RBPF)
In some models, we can partition the hidden variables into two kinds, qtandzt, such that
we can analytically integrate out ztprovided we know the values of q1:t. This means we only
have sample q1:t, and can represent p(zt|q1:t)parametrically. Thus each particle srepresents
a value for qs
1:tand a distribution of the form p(zt|y1:t,qs1:t). These hybrid particles are are

sometimes called distributional particles orcollapsed particles (Koller and Friedman 2009,
Sec 12.4).
The advantage of this approach is that we reduce the dimensionality of the space in which
we are sampling, which reduces the variance of our estimate. Hence this technique is known
asRao-Blackwellised particle ﬁltering orRBPFfor short, named after Theorem 24.20. The
method is best explained using a speciﬁc example.
23.6.1 RBPF for switching LG-SSMs

The quantity Ls
tkis the predictive density for the new observation ytconditioned on qt=kand
the history qs
1:t−1. In the case of SLDS models, this can be computed using the normalization
constant of the Kalman ﬁlter, Equation 18.41.
We give some pseudo-code in Algorithm 8. (The step marked “KFupdate” refers to the Kalman
ﬁlter update equations in Section 18.3.1.) This is known as a mixture of Kalman ﬁlters.
IfKis small, we can compute the optimal proposal distribution, which is
p(qt=k|y1:t,qs
1:t−1)=ˆpst−1(qt=k|yt) (23.58)
=ˆps
t−1(yt|qt=k)ˆpst−1(qt=k)
ˆpst−1(yt)(23.59)
=Ls
tkp(qt=k|qs
t−1)∑
k′Lstk′p(qt=k′|qs
t−1)(23.60)

832 Chapter23. MonteCarloinference
Algorithm 23.2: One step of RBPF for SLDS using prior as proposal
2k∼p(qt|qs
t−1);
3qs
t:=k;
4(μs
t,Σs
t,Ls
tk)=KFupdate( μs
t−1,Σs
t−1,yt,θk);
5ws
t=ws
t−1Lk
ts;
6Normalize weights: ws
t=ws
t∑
s′ws′
t;
7Compute ˆSeff=1∑S
s=1(ws
t)2;
8ifˆSeff<Sminthen
9Resample Sindicesπ∼wt;
10q:
t=qπt,μ:t=μπt,Σ:
t=Σπt,;
11ws
t=1/S;
where we use the following shorthand:
ˆps
t−1(·)=p(·|y1:t−1,qs1:t−1) (23.61)
We then sample from p(qt|qs
1:t−1,y1:t)and give the resulting particle weight
ws
t∝ws
t−1p(yt|qs
1:t−1,y1:t−1)=ws
t−1∑
k[
Lstkp(qt=k|qs
t−1)]
(23.62)
Since the weights of the particles in Equation 23.62 are independent of the new value that is
actually sampled for qt, we can compute these weights ﬁrst, and use them to decide which
particles to propagate. That is, we choose the ﬁttest particles at time t−1using information
from time t. This is called look-ahead RBPF (de Freitas et al. 2004).
In more detail, the idea is this. We pass each sample in the prior through all Kmodels

ws
t∝ws
t−1∑
k[
t−1)]
(23.62)
to getKposteriors, one per sample. The normalization constants of this process allow us to
compute the optimal weights in Equation 23.62. We then resample Sindices. Finally, for each
old particle sthat is chosen, we sample one new state qs
t=k, and use the corresponding
posterior from the Kpossible alternative that we have already computed. The pseudo-code is
shown in Algorithm 7. This method needs O(KS)storage, but has the advantage that each
particle is chosen using the latest information, yt.

A further improvement can be obtained by exploiting the fact that the state space is discrete.
Hence we can use the resampling method of (Fearnhead 2004) which avoids duplicating particles.
23.6.2 Application: tracking a maneuvering target
One application of SLDS is to track moving objects that have piecewise linear dynamics. Forexample, suppose we want to track an airplane or missile; q
tcan specify if the object is ﬂying
normally or is taking evasive action. This is called maneuvering target tracking.

Figure 23.7 gives an example of an object moving in 2d. The setup is essentially the same
as in Section 18.2.1, except that we add a three-state discrete Markov chain which controls the

23.6. Rao-Blackwellisedparticleﬁltering(RBPF) 833
Algorithm 23.3: One step of look-ahead RBPF for SLDS using optimal proposal
2fork=1:Kdo
3 (μs
tk,Σs
tk,Lk
ts)=KFupdate( μs
t−1,Σs
4ws
t=ws
t−1[∑
kLk
tsp(qt=k|qs
t−1)];
5Normalize weights: ws
t=ws
t∑
s′ws′
t;
6Resample Sindicesπ∼wt;
7fors∈πdo
8Compute optimal proposal p(k|qs
1:t−1,y1:t)=Ls
t−1)∑
k′Ls
t−1);
9Samplek∼p(k|qs
1:t−1,y1:t);
10qs
t=k,μs
t=μs
tk,Σs
t=Σstk;
11ws
t=1/S;
Method misclassiﬁcation rate MSE Time (seconds)
PF 0.440 21.051 6.086
RBPF 0.340 18.168 10.986
Table 23.1 Comparison of PF an RBPF on the maneuvering target problem in Figure 23.7.
input to the system. We deﬁne ut=1and set
B1=( 0,0,0,0)T,B2=(−1.225,−0.35,1.225,0.35)T,B3=( 1.225,0.35,−1.225,−0.35)T
so the system will turn in different directions depending on the discrete state.
Figure 23.7(a) shows the true state of the system from a sample run, starting at (0,0): the

t=k,μs
t=μs
tk,Σs
t=Σstk;
11ws
t=1/S;
colored symbols denote the discrete state, and the location of the symbol denotes the (x,y)
location. The small dots represent noisy observations. Figure 23.7(b) shows the estimate of

the state computed using particle ﬁltering with 500 particles, where the proposal is to samplefrom the prior. The colored symbols denote the MAP estimate of the state, and the location ofthe symbol denotes the MMSE (minimum mean square error) estimate of the location, which isgiven by the posterior mean. Figure 23.7(c) shows the estimate computing using RBPF with 500particles, using the optimal proposal distribution. A more quantitative comparison is shown inTable 23.1. We see that RBPF has slightly better performance, although it is also slightly slower.

Figure 23.8 visualizes the belief state of the system. In (a) we show the distribution over the
discrete states. We see that the particle ﬁlter estimate of the belief state (second column) is notas accurate as the RBPF estimate (third column) in the beginning, although after the ﬁrst fewobservations performance is similar for both methods. In (b), we plot the posterior over the x
locations. For simplicity, we use the PF estimate, which is a set of weighted samples, but wecould also have used the RBPF estimate, which is a set of weighted Gaussians.

834 Chapter23. MonteCarloinference
−90 −80 −70 −60 −50 −40 −30 −20 −10 0−250−200−150−100−50050data
(a)−80 −70 −60 −50 −40 −30 −20 −10 0−250−200−150−100−50050pf, mse 21.051
(b)
−80 −70 −60 −50 −40 −30 −20 −10 0−250−200−150−100−50050rbpf, mse 18.168
(c)
Figure 23.7 (a) A maneuvering target. The colored symbols represent the hidden discrete state. (b)
Particle ﬁlter estimate. (c) RBPF estimate. Figure generated by rbpfManeuverDemo , based on code by
Nando de Freitas.
23.6.3 Application: Fast SLAM
In Section 18.2.2, we introduced the problem of simultaneous localization and mapping or SLAM
for mobile robotics. The main problem with the Kalman ﬁlter implementation is that it is cubicin the number of landmarks. However, by looking at the DGM in Figure 18.2, we see that,conditional on knowing the robot’s path, q
1:t,w h e r eqt∈R2, the landmark locations z∈R2L
are independent. (We assume the landmarks don’t move, so we drop the tsubscript). That is,
p(z|q1:t,y1:t)=∏L

l=1p(zl|q1:t,y1:t). Consequently we can use RBPF, where we sample the
robot’s trajectory, q1:t, and we run Lindependent 2d Kalman ﬁlters inside each particle. This
takesO(L)time per particle. Fortunately, the number of particles needed for good performance
is quite small (this partly depends on the control / exploration policy), so the algorithm is
essentially linear in the number of particles. This technique has the additional advantage that

23.6. Rao-Blackwellisedparticleﬁltering(RBPF) 835
truth
1 2 310
20
30
40
50
60
70
80
90
100
pf, error rate 0.440
1 2 310
20
30
40
50
60
70
80
90
100
rbpf, error rate 0.340
1 2 310
20
30
40
50
60
70
80
90
100
(a)−80−60−40−200
02040608010000.51
x1,tPF
tp(x1,t|y1:t)
(b)
Figure 23.8 Belief states corresponding to Figure 23.7. (a) Discrete state. The system starts in state
2 (red x in Figure 23.7), then moves to state 3 (black * in Figure 23.7), returns brieﬂy to state 2, then
switches to state 1 (blue circle in Figure 23.7), etc. (b) Horizontal location (PF estimate). Figure generated
byrbpfManeuverDemo , based on code by Nando de Freitas.
it is easy to use sampling to handle the data association ambiguity, and that it allows for other
representations of the map, such as occupancy grids. This idea was ﬁrst suggested in (Murphy
2000), and was subsequently extended and made practical in (Thrun et al. 2004), who christened
the technique FastSLAM . See rbpfSlamDemo for a simple demo in a discrete grid world.

Exercises
Exercise 23.1 Sampling from a Cauchy
Show how to use inverse probability transform to sample from a standard Cauchy, T(x|0,1,1).
Exercise 23.2 Rejection sampling from a Gamma using a Cauchy proposal
Show how to use a Cauchy proposal to perform rejection sampling from a Gamma distribution. Derive the
optimal constant M, and plot the density and its upper envelope.
Exercise 23.3 Optimal proposal for particle ﬁltering with linear-Gaussian measurement model
Consider a state-space model of the following form:
zt=ft(zt−1)+N(0,Qt−1) (23.63)
yt=Htzt+N(0,Rt) (23.64)

Derive expressions for p(zt|zt−1,yt)andp(yt|zt−1), which are needed to compute the optimal (minimum
variance) proposal distribution. Hint: use Bayes rule for Gaussians.

24Markov chain Monte Carlo (MCMC)
inference
24.1 Introduction
In Chapter 23, we introduced some simple Monte Carlo methods, including rejection sampling
and importance sampling. The trouble with these methods is that they do not work well in highdimensional spaces. The most popular method for sampling from high-dimensional distributionsisMarkov chain Monte Carlo orMCMC.I nas u r v e yb ySIAMNews
1, MCMC was placed in the
top 10 most important algorithms of the 20th century.
The basic idea behind MCMC is to construct a Markov chain (Section 17.2) on the state space
Xwhose stationary distribution is the target density p∗(x)of interest (this may be a prior or a
posterior). That is, we perform a random walk on the state space, in such a way that the fractionof time we spend in each state xis proportional to p
∗(x). By drawing (correlated!) samples
x0,x1,x2,...,from the chain, we can perform Monte Carlo integration wrt p∗. We give the
details below.

The MCMC algorithm has an interesting history. It was discovered by physicists working

on the atomic bomb at Los Alamos during World War II, and was ﬁrst published in the openliterature in (Metropolis et al. 1953) in a chemistry journal. An extension was published inthe statistics literature in (Hastings 1970), but was largely unnoticed. A special case (Gibbssampling, Section 24.2) was independently invented in 1984 in the context of Ising models andwas published in (Geman and Geman 1984). But it was not until (Gelfand and Smith 1990) thatthe algorithm became well-known to the wider statistical community. Since then it has becomewildly popular in Bayesian statistics, and is becoming increasingly popular in machine learning.
It is worth brieﬂy comparing MCMC to variational inference (Chapter 21). The advantages

of variational inference are (1) for small to medium problems, it is usually faster; (2) it isdeterministic; (3) is it easy to determine when to stop; (4) it often provides a lower bound onthe log likelihood. The advantages of sampling are: (1) it is often easier to implement; (2) itis applicable to a broader range of models, such as models whose size or structure changesdepending on the values of certain variables (e.g., as happens in matching problems), or modelswithout nice conjugate priors; (3) sampling can be faster than variational methods when appliedto really huge models or datasets.
2
1. Source: http://www .siam.org/pdf/news/637 .pdf.
2. The reason is that sampling passes speciﬁc values of variables (or sets of variables), whereas in variational inference,
we pass around distributions. Thus sampling passes sparse messages, whereas variational inference passes dense

2
messages For comparisons of the two approaches, see e.g., (Yoshida and West 2010) and articles in (Bekkerman et al.

838 Chapter24. MarkovchainMonteCarlo(MCMC)inference
24.2 Gibbs sampling
In this section, we present one of the most popular MCMC algorithms, known as Gibbs sam-
pling.3(In physics, this method is known as Glauber dynamics or theheat bath method.) This
is the MCMC analog of coordinate descent.
24.2.1 Basic idea
The idea behind Gibbs sampling is that we sample each variable in turn, conditioned on the
values of all the other variables in the distribution. That is, given a joint sample xsof all the
variables, we generate a new sample xs+1by sampling each component in turn, based on the
most recent values of the other variables. For example, if we have D=3variables, we use
•xs+1
1∼p(x1|xs
2,xs3)
•xs+1
2∼p(x2|xs+11,xs
3)
•xs+1
3∼p(x3|xs+11,xs+12)
This readily generalizes to Dvariables. If xiis a visible variable, we do not sample it, since its
value is already known.
The expression p(xi|x−i)is called the full conditional for variable i. In general, ximay only

•xs+1
1∼p(x1|xs
2,xs3)
•xs+1
3)
•xs+1
depend on some of the other variables. If we represent p(x)as a graphical model, we can infer
the dependencies by looking at i’s Markov blanket, which are its neighbors in the graph. Thus
to sample xi, we only need to know the values of i’s neighbors. In this sense, Gibbs sampling
is a distributed algorithm. However, it is not a parallel algorithm, since the samples must be
generated sequentially.
For reasons that we will explain in Section 24.4.1, it is necessary to discard some of the

initial samples until the Markov chain has burned in, or entered its stationary distribution. We
discuss how to estimate when burnin has occured in Section 24.4.1. In the examples below, wejust discard the initial 25% of the samples, for simplicity.
24.2.2 Example: Gibbs sampling for the Ising model
In Section 21.3.2, we applied mean ﬁeld to an Ising model. Here we apply Gibbs sampling.
Gibbs sampling in pairwise MRF/CRF takes the form
p(xt|x−t,θ)∝∏
s∈nbr(t)ψst(xs,xt) (24.1)

In the case of an Ising model with edge potentials ψ(xs,xt)=e x p ( Jxsxt),w h e r ext∈
2011)
3. Josiah Willard Gibbs, 1839–1903, was an American physicist.

24.2. Gibbssampling 839
samp le 1, Gibb s
  
−1−0.500.51
(a)
sample 5, Gibbs
  
(b)
mean after 15 sweeps of Gibbs
  
(c)
Figure 24.1 Example of image denoising. We use an Ising prior with Wij=J=1and a Gaussian
noise model with σ=2. We use Gibbs sampling (Section 24.2) to perform approximate inference. (a)
Sample from the posterior after one sweep over the image. (b) Sample after 5 sweeps. (c) Posterior mean,
computed by averaging over 15 sweeps. Compare to Figure 21.3 which shows the results of using mean
ﬁeld inference. Figure generated by isingImageDenoiseDemo .
{−1,+1}, the full conditional becomes
p(xt=+ 1|x−t,θ)=∏
s∈nbr(t)ψst(xt=+ 1,xs)
∏
s∈nbr(t)ψ(st=+ 1,xs)+∏
s∈nbr(t)ψ(xt=−1,xs)(24.2)
=exp[J∑
s∈nbr(t)xs]
exp[J∑
s∈nbr(t)xs]+exp[−J∑
s∈nbr(t)xs](24.3)
=exp[Jηt]
exp[Jηt]+exp[−Jηt]=s i g m ( 2 Jηt) (24.4)
whereJis the coupling strength, ηt≜∑
s∈nbr(t)xtandsigm(u)=1/(1+e−u)is the sigmoid
function. It is easy to see that ηt=xt(at−dt),w h e r eatis the number of neighbors that agree

∏
=exp[J∑
exp[J∑
=exp[Jηt]
with (have the same sign as) t, anddtis the number of neighbors who disagree. If this number
is equal, the “forces” on xtcancel out, so the full conditional is uniform.
We can combine an Ising prior with a local evidence term ψt. For example, with a Gaussian
observation model, we have ψt(xt)=N(yt|xt,σ2). The full conditional becomes
p(xt=+ 1|x−t,y,θ)=exp[Jηt]ψt(+1)
exp[Jηt]ψt(+1)+exp[−Jηt]ψt(−1)(24.5)
=s i g m(
2Jηt−logψt(+1)
ψt(−1))
(24.6)
Now the probability of xtentering each state is determined both by compatibility with its

=s i g m(
ψt(−1))
(24.6)
neighbors (the Ising prior) and compatibility with the data (the local likelihood term).
See Figure 24.1 for an example of this algorithm applied to a simple image denoising problem.
The results are similar to mean ﬁeld (Figure 21.3) except that the ﬁnal estimate (based on
averaging the samples) is somewhat “blurrier”, due to the fact that mean ﬁeld tends to be
over-conﬁdent.

840 Chapter24. MarkovchainMonteCarlo(MCMC)inference
24.2.3 Example: Gibbs sampling for inferring the parameters of a GMM
It is straightforward to derive a Gibbs sampling algorithm to “ﬁt” a mixture model, especially
if we use conjugate priors. We will focus on the case of mixture of Gaussians, although theresults are easily extended to other kinds of mixture models. (The derivation, which follows fromthe results of Section 4.6, is much easier than the corresponding variational Bayes algorithm inSection 21.6.1.)
Suppose we use a semi-conjugate prior. Then the full joint distribution is given by
p(x,z,μ,Σ,π)=p( x|z,μ,Σ)p(z|π)p(π)
K∏
k=1p(μk)p(Σk) (24.7)
=(N∏
i=1K∏
k=1(πkN(xi|μk,Σk))I(zi=k))
× (24.8)
Dir(π|α)K∏
k=1N(μk|m0,V0)IW(Σ k|S0,ν0) (24.9)
We use the same prior for each mixture component. The full conditionals are as follows. Forthe discrete indicators, we have
p(z
i=k|xi,μ,Σ,π)∝πkN(xi|μk,Σk) (24.10)
For the mixing weights, we have (using results from Section 3.4)
p(π|z)=D i r ({αk+N∑
i=1I(zi=k)}K

K∏
=(N∏
i=1K∏
× (24.8)
p(z
k=1) (24.11)
For the means, we have (using results from Section 4.6.1)
p(μk|Σk,z,x)=N (μk|mk,Vk) (24.12)
V−1
k=V−1
0+NkΣ−1
k (24.13)
mk=Vk(Σ−1
kNkxk+V−1
0m0) (24.14)
Nk≜N∑
i=1I(zi=k) (24.15)
xk≜∑N
i=1I(zi=k)xi
Nk(24.16)
For the covariances, we have (using results from Section 4.6.2)
p(Σk|μk,z,x)=I W ( Σk|Sk,νk) (24.17)
Sk=S0+N∑
i=1I(zi=k)(xi−μk)(xi−μk)T(24.18)
νk=ν0+Nk (24.19)
SeegaussMissingFitGibbs for some Matlab code. (This code can also sample missing
values for x, if necessary.)

24.2. Gibbssampling 841
24.2.3.1 Label switching
Although it is simple to implement, Gibbs sampling for mixture models has a fundamental
weakness. The problem is that the parameters of the model θ, and the indicator functions z,a r e
unidentiﬁable, since we can arbitrarily permute the hidden labels without affecting the likelihood(see Section 11.3.1). Consequently, we cannot just take a Monte Carlo average of the samples tocompute posterior means, since what one sample considers the parameters for cluster 1 may bewhat another sample considers the parameters for cluster 2. Indeed, if we could average overall modes, we would ﬁnd E[μ
k|D]is the same for all k(assuming a symmetric prior). This is
called the label switching problem.
This problem does not arise in EM or VBEM, which just “lock on” to a single mode. However,
it arises in any method that visits multiple modes. In 1d problems, one can try to prevent thisproblem by introducing constraints on the parameters to ensure identiﬁability, e.g., μ
1<μ2<

1<μ2<
μ3(Richardson and Green 1997). However, this does not always work, since the likelihood might
overwhelm the prior and cause label switching anyway. Furthermore, this technique does notscale to higher dimensions. Another approach is to post-process the samples by searching for aglobal label permutation to apply to each sample that minimizes some loss function (Stephens2000); however, this can be slow.
Perhaps the best solution is simply to “not ask” questions that cannot be uniquely identiﬁed.
For example, instead of asking for the probability that data point ibelongs to cluster k, ask

for the probability that data points iandjbelong to the same cluster. The latter question is
invariant to the labeling. Furthermore, it only refers to observable quantities (are iandjgrouped
together or not), rather than referring to unobservable quantities, such as latent clusters. Thisapproach has the further advantage that it extends to inﬁnite mixture models, discussed inSection 25.2, where Kis unbounded; in such models, the notion of a hidden cluster is not well

deﬁned, but the notion of a partitioning of the data iswell deﬁned
24.2.4 Collapsed Gibbs sampling *
In some cases, we can analytically integrate out some of the unknown quantities, and justsample the rest. This is called a collapsed Gibbs sampler, and it tends to be much more
efficient, since it is sampling in a lower dimensional space.
More precisely, suppose we sample zand integrate out θ. Thus the θparameters do not
participate in the Markov chain; consequently we can draw conditionally independent samplesθ

s∼p(θ|zs,D), which will have much lower variance than samples drawn from the joint state
space (Liu et al. 1994). This process is called Rao-Blackwellisation, named after the following
theorem:
Theorem 24.2.1 (Rao-Blackwell). Letzandθbe dependent random variables, and f(z,θ)be
somescalarfunction. Then
varz,θ[f(z,θ)]≥varz[Eθ[f(z,θ)|z]] (24.20)
This theorem guarantees that the variance of the estimate created by analytically integrating
outθwill always be lower (or rather, will never be higher) than the variance of a direct MC

theorem:
estimate. In collapsed Gibbs, we sample zwithθintegrated out; the above Rao-Blackwell
theorem still applies in this case (Liu et al. 1994).

842 Chapter24. MarkovchainMonteCarlo(MCMC)inference
α
π
z1zizN
x1xixN
θK
β
(a)α
z1zizN
x1xixN
β
(b)
Figure 24.2 (a) A mixture model. (b) After integrating out the parameters.
We will encounter Rao-Blackwellisation again in Section 23.6. Although it can reduce statistical
variance, it is only worth doing if the integrating out can be done quickly, otherwise we will not
be able to produce as many samples per second as the naive method. We give an example ofthis below.
24.2.4.1 Example: collapsed Gibbs for ﬁtting a GMM
Consider a GMM with a fully conjugate prior. In this case we can analytically integrate out themodel parameters μ
k,Σkandπ, and just sample the indicators z. Once we integrate out π,
all thezinodes become inter-dependent. Similarly, once we integrate out θk, all thexinodes
become inter-dependent, as shown in Figure 24.2(b). Nevertheless, we can easily compute thefull conditionals as follows:
p(z
i=k|z−i,x,α,β)∝p(zi=k|z−i,α,β)p(x|zi=k,z−i,α,β) (24.21)
∝p(zi=k|z−i,α)p(xi|x−i,zi=k,z−i,β)

24.2. Gibbssampling 843
Hence
p(zi=k|z−i,α)=p(z1:N|α)
p(z−i|α)=1
Γ(N+α)
1
Γ(N+α−1)×Γ(Nk+α/K)
Γ(Nk,−i+α/K)(24.25)
=Γ(N+α−1)
Γ(N+α)Γ(Nk,−i+1+α/K)
Γ(Nk,−i+α/K)=Nk,−i+α/K
N+α−1(24.26)
whereNk,−i≜∑
n̸=iI(zn=k)=Nk−1, and where we exploited the fact that Γ(x+1)=
xΓ(x).
To obtain the second term in Equation 24.23, which is the posterior predictive distribution for
xigiven all the other data and all the assignments, we use the fact that
p(xi|x−i,z−i,zi=k,β)=p(xi|D−i,k) (24.27)
whereD−i,k={xj:zj=k,j̸=i}is all the data assigned to cluster kexcept for xi.I fw e
use a conjugate prior for θk, we can compute p(xi|D−i,k)in closed form. Furthermore, we can
efficiently update these predictive likelihoods by caching the sufficient statistics for each cluster.
To compute the above expression, we remove xi’s statistics from its current cluster (namely zi),
and then evaluate xiunder each cluster’s posterior predictive. Once we have picked a new
cluster, we add xi’s statistics to this new cluster.

Some pseudo-code for one step of the algorithm is shown in Algorithm 1, based on (Sud-
derth 2006, p94). (We update the nodes in random order to improve the mixing time, assuggested in (Roberts and Sahu 1997).) We can initialize the sample by sequentially samplingfromp(z
i|z1:i−1,x1:i). (See fmGibbs for some Matlab code, by Yee-Whye Teh.) In the case of
GMMs, both the naive sampler and collapsed sampler take O(NKD)time per step.
Algorithm 24.1: Collapsed Gibbs sampler for a mixture model
1foreachi=1:Ninrandomorder do
2Removexi’s sufficient statistics from old cluster zi;

3foreachk=1:Kdo
4 Compute pk(xi)≜p(xi|{xj:zj=k,j̸=i});
5Compute p(zi=k|z−i,D)∝(Nk,−i+α/K)pk(xi);
6Samplezi∼p(zi|·);
7Addxi’s sufficient statistics to new cluster zi
A comparison of this method with the standard Gibbs sampler is shown in Figure 24.3. The
vertical axis is the data log probability at each iteration, computed using
logp(D|z,θ)=N∑
i=1log[πzip(xi|θzi)] (24.28)
To compute this quantity using the collapsed sampler, we have to sample θ=(π,θ1:K)given
the data and the current assignment z.

In Figure 24.3 we see that the collapsed sampler does indeed generally work better than the
vanilla sampler. Occasionally, however, both methods can get stuck in poor local modes. (Note

844 Chapter24. MarkovchainMonteCarlo(MCMC)inference
100101102103−600−550−500−450−400−350
Iterationlog p(x | π, θ)
  
Standard Gibbs Sampler
Rao−Blackwellized Sampler
(a)100101102103−600−550−500−450−400−350
  
(b)
Figure 24.3 Comparison of collapsed (red) and vanilla (blue) Gibbs sampling for a mixture of K=4two-
dimensional Gaussians applied to N= 300data points (shown in Figure 25.7). We plot log probability of
the data vs iteration. (a) 20 different random initializations. (b) logprob averaged over 100 different random
initializations. Solid line is the median, thick dashed in the 0.25 and 0.75 quantiles, and thin dashed arethe 0.05 and 0.95 quintiles. Source: Figure 2.20 of (Sudderth 2006). Used with kind permission of ErikSudderth.
−2 −1 0 1 220304050607080
SESMath Score
(a)5 10 15 20 25 30−50510
Sample SizeSlope
(b)−2 −1 0 1 220304050607080
(c)

(c)
Figure 24.4 (a) Least squares regression lines for math scores vs socio-economic status for 100 schools.
Population mean (pooled estimate) is in bold. (b) Plot of ˆw2j(the slope) vs Nj(sample size) for the 100
schools. The extreme slopes tend to correspond to schools with smaller sample sizes. (c) Predictions fromthe hierarchical model. Population mean is in bold. Based on Figure 11.1 of (Hoff 2009). Figure generatedbymultilevelLinregDemo , written by Emtiyaz Khan.

(c)
that the error bars in Figure 24.3(b) are averaged over starting values, whereas the theorem refers
to MC samples in a single run.)
24.2.5 Gibbs sampling for hierarchical GLMs

Often we have data from multiple related sources. If some sources are more reliable and/ordata-rich than others, it makes sense to model all the data simultaneously, so as to enable theborrowing of statistical strength. One of the most natural way to solve such problems is to usehierarchical Bayesian modeling, also called multi-level modeling. In Section 9.6, we discusseda way to perform approximate inference in such models using variational methods. Here wediscuss how to use Gibbs sampling.
To explain the method, consider the following example. Suppose we have data on students

24.2. Gibbssampling 845
xijyijwj
σ2μwΣw
Nj
J
Figure 24.5 Multi-level model for linear regression.
in different schools. Such data is naturally modeled in a two-level hierarchy: we let yijbe the
response variable we want to predict for student iin school j. This prediction can be based on
school and student speciﬁc covariates, xij. Since the quality of schools varies, we want to use
a separate parameter for each school. So our model becomes
yij=xT
ijwj+ϵij (24.29)
We will illustrate this model below, using a dataset from (Hoff 2009, p197), where xijis the
socio-economic status (SES) of student iin school y, andyijis their math score.
We could ﬁt each wjseparately, but this can give poor results if the sample size of a given
school is small. This is illustrated in Figure 24.4(a), which plots the least squares regression
line estimated separately for each of the J= 100schools. We see that most of the slopes are

yij=xT
positive, but there are a few “errant” cases where the slope is negative. It turns out that the lineswith extreme slopes tend to be in schools with small sample size, as shown in Figure 24.4(b).Thus we may not necessarily trust these ﬁts.
We can get better results if we construct a hierarchical Bayesian model, in which the w
jare
assumed to come from a common prior: wj∼N(μw,Σw). This is illustrated in Figure 24.5. In

jare
this model, the schools with small sample size borrow statistical strength from the schools withlarger sample size, because the w
j’s are correlated via the latent common parents (μw,Σw). (It
is crucial that these hyper-parameters be inferrred from data; if they were ﬁxed constants, thew
jwould be conditionally independent, and there would be no information sharing between
them.)
To complete the model speciﬁcation, we must specify priors for the shared parameters. Fol-
lowing (Hoff 2009, p198), we will use the following semi-conjugate forms, for convenience:
μw∼N(μ0,V0) (24.30)
Σw∼IW(η0,S−1

them.)
0) (24.31)
σ2∼IG(ν0/2,ν0σ2
0/2) (24.32)
Given this, it is simple to show that the full conditionals needed for Gibbs sampling have the

846 Chapter24. MarkovchainMonteCarlo(MCMC)inference
following forms. For the group-speciﬁc weights:
p(wj|Dj,θ)=N (wj|μj,Σj) (24.33)
Σ−1
j=Σ−1+XT
jXj/σ2(24.34)
μj=Σj(Σ−1μ+XTjyj/σ2) (24.35)
For the overall mean:
p(μw|w1:J,Σw)=N (μ|μN,ΣN) (24.36)
Σ−1
N=V−1
0+JΣ−1(24.37)
μN=ΣN(V−1
0μ0+JΣ−1w) (24.38)
wherew=1
J∑
jwj. For the overall covariance:
p(Σw|μw,w1:J)=I W ( ( S0+Sμ)−1,η0+J) (24.39)
Sμ=∑
j(wj−μw)(wj−μw)T(24.40)
For the noise variance:
p(σ2|D,w1:J)=I G ( [ ν0+N]/2,[ν0σ2
0+SSR(w1:J)]/2) (24.41)
SSR(w1:J)=J∑
j=1Nj∑
i=1(yij−wT
jxij)2(24.42)
Applying Gibbs sampling to our hierarchical model, we get the results shown in Figure 24.4(c).
The light gray lines plot the mean of the posterior predictive distribution for each school:
E[yj|xij]=xTijˆwj (24.43)
where
ˆwj=E[wj|D]≈1
SS∑
s=1w(s)
j (24.44)
The dark gray line in the middle plots the prediction using the overall mean parameters, xT
ijˆμw.
We see that the method has regularized the ﬁts quite nicely, without enforcing too much

j=1Nj∑
where
SS∑
s=1w(s)
j (24.44)
ijˆμw.
uniformity. (The amount of shrinkage is controlled by Σw, which in turns depends on the
hyper-parameters; in this example, we used vague values.)
24.2.6 BUGS and JAGS

where
SS∑
s=1w(s)
j (24.44)
ijˆμw.
One reason Gibbs sampling is so popular is that it is possible to design general purpose softwarethat will work for almost any model. This software just needs a model speciﬁcation, usuallyin the form a directed graphical model (speciﬁed in a ﬁle, or created with a graphical userinterface), and a library of methods for sampling from different kinds of full conditionals. (Thiscan often be done using adaptive rejection sampling, described in Section 23.3.4.) An example

24.2. Gibbssampling 847
of such a package is BUGS(Lunn et al. 2000), which stands for “Bayesian updating using Gibbs
Sampling”. BUGS is very widely used in biostatistics and social science. Another more recent,
but very similar, package is JAGS(Plummer 2003), which stands for “Just Another Gibbs Sampler”.
This uses a similar model speciﬁcation language to BUGS.
For example, we can describe the model in Figure 24.5 as follows:
model {
for (i in 1:N) {
for (j in 1:J) {
y[i,j] ~ dnorm(y.hat[i,j], tau.y)y.hat[i,j] <- inprod(W[j, ], X[i, j, ])
}
}tau.y <- pow(sigma.y, -2)sigma.y ~ dunif(0,100)
W[j,] ~ dmnorm(mu, SigmaInv)
}
SigmaInv ~ dwish(S0[,], eta0)mu ~ dmnorm(mu0, V0inv)}
We can then just pass this model to BUGS or JAGS, which will generate samples for us. See
the webpages for details.
Although this approach is appealing, unfortunately it can be much slower than using hand-

model {
}
}
written code, especially for complex models. There has been some work on automaticallyderiving model-speciﬁc optimized inference code (Fischer and Schumann 2003), but fast codestill typically requires human expertise.
24.2.7 The Imputation Posterior (IP) algorithm
TheImputation Posterior or IP algorithm (Tanner and Wong 1987) is a special case of Gibbs
sampling in which we group the variables into two classes: hidden variables zand parameters
θ. This should sound familiar: it is basically an MCMC version of EM, where the E step gets

replaced by the I step, and the M step gets replaced the P step. This is an example of a moregeneral strategy called data augmentation, whereby we introduce auxiliary variables in order
to simplify the posterior computations (here the computation of p(θ|D)). See (Tanner 1996; van
Dyk and Meng 2001) for more information.
24.2.8 Blocking Gibbs sampling
Gibbs sampling can be quite slow, since it only updates one variable at a time (so-called single

site updating). If the variables are highly correlated, it will take a long time to move awayfrom the current state. This is illustrated in Figure 24.6, where we illustrate sampling from a 2dGaussian (see Exercise 24.1 for the details). If the variables are highly correlated, the algorithm

848 Chapter24. MarkovchainMonteCarlo(MCMC)inference
/g1/g2/g1/g3/g1/g4/g5/g4/g3/g2
/g1/g2 /g1/g3 /g1/g4 /g5 /g4 /g3 /g2/g6
/g7
Figure 24.6 Illustration of potentially slow sampling when using Gibbs sampling for a skewed 2D Gaus-
sian. Based on Figure 11.11 of (Bishop 2006b). Figure generated by gibbsGaussDemo .
will move very slowly through the state space. In particular, the size of the moves is controlled
by the variance of the conditional distributions. If this is ℓin thex1direction, and the support
of the distribution is Lalong this dimension, then we need O((L/ℓ)2)steps to obtain an
independent sample.
In some cases we can efficiently sample groups of variables at a time. This is called blocking
Gibbs sampling orblocked Gibbs sampling (Jensen et al. 1995; Wilkinson and Yeung 2002),
and can make much bigger moves through the state space.
24.3 Metropolis Hastings algorithm

Although Gibbs sampling is simple, it is somewhat restricted in the set of models to which itcan be applied. For example, it is not much help in computing p(w|D)for a logistic regression
model, since the corresponding graphical model has no useful Markov structure. In addition,Gibbs sampling can be quite slow, as we mentioned above.
Fortunately, there is a more general algorithm that can be used, known as the Metropolis
Hastings orMHalgorithm, which we describe below.
24.3.1 Basic idea

The basic idea in MH is that at each step, we propose to move from the current state xto a
new state x′with probability q(x′|x),w h e r eqis called the proposal distribution (also called
thekernel). The user is free to use any kind of proposal they want, subject to some conditions
which we explain below. This makes MH quite a ﬂexible method. A commonly used proposal isa symmetric Gaussian distribution centered on the current state, q(x
′|x)=N(x′|x,Σ); this is

called arandom walk Metropolis algorithm. We discuss how to choose Σin Section 24.3.3. If
we use a proposal of the form q(x′|x)=q(x′), where the new state is independent of the old
state, we get a method known as the independence sampler, which is similar to importance
sampling (Section 23.4).
Having proposed a move to x′, we then decide whether to acceptthis proposal or not
according to some formula, which ensures that the fraction of time spent in each state isproportional to p

∗(x). If the proposal is accepted, the new state is x′, otherwise the new state

24.3. MetropolisHastingsalgorithm 849
is the same as the current state, x(i.e., we repeat the sample).
If the proposal is symmetric, so q(x′|x)=q(x|x′), the acceptance probability is given by the
following formula:
r=m i n ( 1,p∗(x′)
p∗(x)) (24.45)
We see that if x′is more probable than x, we deﬁnitely move there (sincep∗(x′)
p∗(x)>1), but if
x′is less probable, we may still move there anyway, depending on the relative probabilities. So
instead of greedily moving to only more probable states, we occasionally allow “downhill” moves
to less probable states. In Section 24.3.6, we prove that this procedure ensures that the fractionof time we spend in each state xis proportional to p
∗(x).
If the proposal is asymmetric, so q(x′|x)̸=q(x|x′), we need the Hastings correction, given
by the following:
r=m i n ( 1 ,α) (24.46)
α=p∗(x′)q(x|x′)
p∗(x)q(x′|x)=p∗(x′)/q(x′|x)
p∗(x)/q(x|x′)(24.47)

∗(x).
This correction is needed to compensate for the fact that the proposal distribution itself (ratherthan just the target distribution) might favor certain states.
An important reason why MH is a useful algorithm is that, when evaluating α, we only need to
know the target density up to a normalization constant. In particular, suppose p
∗(x)=1
Z˜p(x),
where˜p(x)is an unnormalized distribution and Zis the normalization constant. Then
α=(˜p(x′)/Z)q(x|x′)
(˜p(x)/Z)q(x′|x)(24.48)
so theZ’s cancel. Hence we can sample from p∗even ifZis unknown. In particular, all we

∗(x)=1
Z˜p(x),
have to do is evaluate ˜ppointwise, where ˜p(x)=p∗(x)Z.
The overall algorithm is summarized in Algorithm 2.
24.3.2 Gibbs sampling is a special case of MH
It turns out that Gibbs sampling, which we discussed in Section 24.2, is a special case of MH. Inparticular, it is equivalent to using MH with a sequence of proposals of the form
q(x
′|x)=p(x′
i|x−i)I(x′−i=x−i) (24.49)
That is, we move to a new state where xiis sampled from its full conditional, but x−iis left
unchanged.
We now prove that the acceptance rate of each such proposal is 1, so the overall algorithm
also has an acceptance rate of 100%. We have

q(x
′|x)=p(x′
α=p(x′)q(x|x′)
p(x)q(x′|x)=p(x′
i|x′−i)p(x′−i)p(xi|x′−i)
p(xi|x−i)p(x−i)p(x′i|x−i)(24.50)
=p(x′i|x−i)p(x−i)p(xi|x−i)
p(xi|x−i)p(x−i)p(x′i|x−i)=1 (24.51)

850 Chapter24. MarkovchainMonteCarlo(MCMC)inference
Algorithm 24.2: Metropolis Hastings algorithm
1Initializex0;
2fors=0,1,2,...do
3Deﬁnex=xs;
4Samplex′∼q(x′|x);
5Compute acceptance probability
α=˜p(x′)q(x|x′)
˜p(x)q(x′|x)
Compute r=m i n ( 1,α);
6Sampleu∼U(0,1);
7Set new sample to
xs+1={x′ifu<r
xsifu≥r
where we exploited the fact that x′
−i=x−i, and that q(x′|x)=p(x′i|x−i).
The fact that the acceptance rate is 100% does not necessarily mean that Gibbs will converge
rapidly, since it only updates one coordinate at a time (see Section 24.2.8). Fortunately, there are
many other kinds of proposals we can use, as we discuss below.
24.3.3 Proposal distributions
For a given target distribution p∗, a proposal distribution qis valid or admissible if it gives
a non-zero probability of moving to the states that have non-zero probability in the target.Formally, we can write this as
supp(p
∗)⊆∪xsupp(q(·|x)) (24.52)

supp(p
For example, a Gaussian random walk proposal has non-zero probability density on the entirestate space, and hence is a valid proposal for any continuous state space.
Of course, in practice, it is important that the proposal spread its probability mass in just the
right way. Figure 24.7 shows an example where we use MH to sample from a mixture of two1D Gaussians using a random walk proposal, q(x
′|x)=N(x′|x,v). This is a somewhat tricky

supp(p
target distribution, since it consists of two well separated modes. It is very important to set thevariance of the proposal vcorrectly: If the variance is too low, the chain will only explore one
of the modes, as shown in Figure 24.7(a), but if the variance is too large, most of the moveswill be rejected, and the chain will be very sticky, i.e., it will stay in the same state for a long

time. This is evident from the long stretches of repeated values in Figure 24.7(b). If we setthe proposal’s variance just right, we get the trace in Figure 24.7(c), where the samples clearlyexplore the support of the target distribution. We discuss how to tune the proposal below.
One big advantage of Gibbs sampling is that one does not need to choose the proposal

24.3. MetropolisHastingsalgorithm 851
0
200
400
600
800
1000−100−5005010000.10.2
SamplesMH with N(0,1.0002) proposal
Iterations
(a)0
200
400
600
800
1000−100−5005010000.020.040.06
SamplesMH with N(0,500.0002) proposal
(b)
0
200
400
600
800
1000−100−5005010000.010.020.03
SamplesMH with N(0,8.0002) proposal
(c)
Figure 24.7 An example of the Metropolis Hastings algorithm for sampling from a mixture of two 1D
Gaussians (μ =(−20,20),π=( 0.3,0.7),σ= (100,100)), using a Gaussian proposal with variances of
v∈{1,500,8}. (a) When v=1, the chain gets trapped near the starting state and fails to sample from
the mode at μ=−20. (b) When v= 500, the chain is very “sticky”, so its effective sample size is low (as
reﬂected by the rough histogram approximation at the end). (c) Using a variance of v=8is just right and
leads to a good approximation of the true distribution (shown in red). Figure generated by mcmcGmmDemo .
Based on code by Christophe Andrieu and Nando de Freitas.

distribution, and furthermore, the acceptance rate is 100%. Of course, a 100% acceptance can

trivially be achieved by using a proposal with variance 0 (assuming we start at a mode), but thisis obviously not exploring the posterior. So having a high acceptance is not the ultimate goal.We can increase the amount of exploration by increasing the variance of the Gaussian kernel.Often one experiments with different parameters until the acceptance rate is between 25% and40%, which theory suggests is optimal, at least for Gaussian target distributions. These shortinitial runs, used to tune the proposal, are called pilot runs.

852 Chapter24. MarkovchainMonteCarlo(MCMC)inference
−120 −100 −80 −60 −40 −20 000.020.040.060.080.10.120.140.160.180.2
w0w1
(a)−120 −100 −80 −60 −40 −20 0050010001500w0 intercept
(b)0 0.05 0.1 0.15 0.2050010001500w1 slope
(c)
Figure 24.8 (a) Joint posterior of the parameters for 1d logistic regression when applied to some SAT data.
(b) Marginal for the offset w0. (c) Marginal for the slope w1. We see that the marginals do not capture the
fact that the parameters are highly correlated. Figure generated by logregSatMhDemo .
24.3.3.1 Gaussian proposals
If we have a continuous state space, the Hessian Hat a local mode ˆwcan be used to deﬁne
the covariance of a Gaussian proposal distribution. This approach has the advantage that the
Hessian models the local curvature and length scales of each dimension; this approach thereforeavoids some of the slow mixing behavior of Gibbs sampling shown in Figure 24.6.
There are two obvious approaches: (1) an independence proposal, q(w
′|w)=N(w′|ˆw,H−1)

or (2), a random walk proposal, q(w′|w)=N(w′|w,s2H−1),w h e r es2is a scale factor chosen
to facilitate rapid mixing. (Roberts and Rosenthal 2001) prove that, if the posterior is Gaussian,the asymptotically optimal value is to use s
2=2.382/D,w h e r eDis the dimensionality of w;
this results in an acceptance rate of 0.234.
For example, consider MH for binary logistic regression. From Equation 8.7, we have that
the Hessian of the log-likelihood is Hl=XTDX,w h e r eD= diag(μ i(1−μi))andμi=

sigm(ˆwTxi). If we assume a Gaussian prior, p(w)=N(0,V0),w eh a v e H=V−1
0+Hl,s o
the asymptotically optimal Gaussian proposal has the form
q(w′|w)=N(
w,2.382
D(
V−1
0+XTDX)−1)
(24.53)

0+Hl,s o
w,2.382
D(
V−1
(24.53)
See (Gamerman 1997; Rossi et al. 2006; Fruhwirth-Schnatter and Fruhwirth 2010) for furtherdetails. The approach is illustrated in Figure 24.8, where we sample parameters from a 1dlogistic regression model ﬁt to some SAT data. We initialize the chain at the mode, computedusing IRLS, and then use the above random walk Metropolis sampler.
If you cannot afford to compute the mode or its Hessian XDX, an alternative approach,
suggested in (Scott 2009), is to approximate the above proposal as follows:
q(w
′|w)=N(
w,(
V−1
0+6
π2XTX)−1)
(24.54)

24.3. MetropolisHastingsalgorithm 853
24.3.3.2 Mixture proposals
If one doesn’t know what kind of proposal to use, one can try a mixture proposal, which is a
convex combination of base proposals:
q(x′|x)=K∑
k=1wkqk(x′|x) (24.55)
wherewkare the mixing weights. As long as each qkis individually valid, the overall proposal
will also be valid.
24.3.3.3 Data-driven MCMC
The most efficient proposals depend not just on the previous hidden state, but also the visible
data, i.e., they have the form q(x′|x,D). This is called data-driven MCMC (see e.g., (Tu and
Zhu 2002)). To create such proposals, one can sample (x,D)pairs from the forwards model
and then train a discriminative classiﬁer to predict p(x|f(D)),w h e r e f(D)are some features
extracted from the visible data.
Typically xis a high-dimensional vector (e.g., position and orientation of all the limbs of a
person in a visual object detector), so it is hard to predict the entire state vector, p(x|f(D)).

Instead we might train a discriminative detector to predict parts of the state-space, p(xk|fk(D)),
such as the location of just the face of a person. We can then use a proposal of the form
q(x′|x,D)=π0q0(x′|x)+∑
kπkqk(x′
k|fk(D)) (24.56)
whereq0is a standard data-independent proposal (e.g., random walk), and qkupdates the k’th
component of the state space. For added efficiency, the discriminative proposals should suggest
joint changes to multiple variables, but this is often hard to do.

kπkqk(x′
The overall procedure is a form of generate and test: the discriminative proposals q(x′|x)
generate new hypotheses, which are then “tested” by computing the posterior ratiop(x′|D)
p(x|D),t o
see if the new hypothesis is better or worse. By adding an annealing step, one can modifythe algorithm to ﬁnd posterior modes; this is called simulated annealing, and is described in
Section 24.6.1. One advantage of using the mode-seeking version of the algorithm is that we donot need to ensure the proposal distribution is reversible.

p(x|D),t o
24.3.4 Adaptive MCMC
One can change the parameters of the proposal as the algorithm is running to increase efficiency.This is called adaptive MCMC. This allows one to start with a broad covariance (say), allowing
large moves through the space until a mode is found, followed by a narrowing of the covarianceto ensure careful exploration of the region around the mode.
However, one must be careful not to violate the Markov property; thus the parameters of the

proposal should not depend on the entire history of the chain. It turns out that a sufficientcondition to ensure this is that the adaption is “faded out” gradually over time. See e.g., (Andrieuand Thoms 2008) for details.

854 Chapter24. MarkovchainMonteCarlo(MCMC)inference
24.3.5 Initialization and mode hopping
It is necessary to start MCMC in an initial state that has non-zero probability. If the model has
deterministic constraints, ﬁnding such a legal conﬁguration may be a hard problem in itself. Itis therefore common to initialize MCMC methods at a local mode, found using an optimizer.
In some domains (especially with discrete state spaces), it is a more effective use of computa-
tion time to perform multiple restarts of an optimizer, and to average over these modes, ratherthan exploring similar points around a local mode. However, in continuous state spaces, themode contains negligible volume (Section 5.2.1.3), so it is necessary to locally explore aroundeach mode, in order to visit enough posterior probability mass.
24.3.6 Why MH works *
To prove that the MH procedure generates samples from p∗, we have to use a bit of Markov
chain theory, so be sure to read Section 17.2.3 ﬁrst.

The MH algorithm deﬁnes a Markov chain with the following transition matrix:
p(x′|x)={q(x′|x)r(x′|x) ifx′̸=x
q(x|x)+∑
x′̸=xq(x′|x)(1−r(x′|x))otherwise(24.57)
This follows from a case analysis: if you move to x′fromx, you must have proposed it (with
probability q(x′|x)) and it must have been accepted (with probability r(x′|x)); otherwise you
stay in state x, either because that is what you proposed (with probability q(x|x)), or because

q(x|x)+∑
you proposed something else (with probability q(x′|x)) but it was rejected (with probability
1−r(x′|x)).
Let us analyse this Markov chain. Recall from Section 17.2.3.4 that a chain satisﬁes detailed
balanceif
p(x′|x)p∗(x)=p(x|x′)p∗(x′) (24.58)
We also showed that if a chain satisﬁes detailed balance, then p∗is its stationary distribution.
Our goal is to show that the MH algorithm deﬁnes a transition function that satisﬁes detailedbalance and hence that p

balanceif
∗is its stationary distribution. (If Equation 24.58 holds, we say that p∗
is aninvariant distribution wrt the Markov transition kernel q.)
Theorem 24.3.1. IfthetransitionmatrixdeﬁnedbytheMHalgorithm(givenbyEquation24.57)is
ergodicandirreducible,then p∗isitsuniquelimitingdistribution.
Proof.Consider two states xandx′. Either
p∗(x)q(x′|x)<p∗(x′)q(x|x′) (24.59)
or
p∗(x)q(x′|x)>p∗(x′)q(x|x′) (24.60)
We will ignore ties (which occur with probability zero for continuous distributions). Without lossof generality, assume that p
∗(x)q(x′|x)>p∗(x′)q(x|x′). Hence

or
α(x′|x)=p∗(x′)q(x|x′)
p∗(x)q(x′|x)<1 (24.61)

24.3. MetropolisHastingsalgorithm 855
Hence we have r(x′|x)=α(x′|x)andr(x|x′)=1.
N o wt om o v ef r o mx tox′we must ﬁrst propose x′and then accept it. Hence
p(x′|x)=q(x′|x)r(x′|x)=q(x′|x)p∗(x′)q(x|x′)
p∗(x)q(x′|x)=p∗(x′)
p∗(x)q(x|x′) (24.62)
Hence
p∗(x)p(x′|x)=p∗(x′)q(x|x′) (24.63)
The backwards probability is
p(x|x′)=q(x|x′)r(x|x′)=q(x|x′) (24.64)
sincer(x|x′)=1. Inserting this into Equation 24.63 we get
p∗(x)p(x′|x)=p∗(x′)p(x|x′) (24.65)
so detailed balance holds wrt p∗. Hence, from Theorem 17.2.3, p∗is a stationary distribution.
Furthermore, from Theorem 17.2.2, this distribution is unique, since the chain is ergodic and
irreducible.
24.3.7 Reversible jump (trans-dimensional) MCMC *
Suppose we have a set of models with different numbers of parameters, e.g., mixture models inwhich the number of mixture components is unknown. Let the model be denoted by m, and
let its unknowns (e.g., parameters) be denoted by x
m∈Xm(e.g.,Xm=Rnm,w h e r enmis

the dimensionality of model m). Sampling in spaces of differing dimensionality is called trans-
dimensional MCMC (Green 2003). We could sample the model indicator m∈{1,...,M}and
sample all the parameters from the product space∏M
m=1Xm, but this is very inefficient. It is
more parsimonious to sample in the union space X=∪M
m=1{m}×X m, where we only worry
about parameters for the currently active model.
The difficulty with this approach arises when we move between models of different dimen-
sionality. The trouble is that when we compute the MH acceptance ratio, we are comparing

densities deﬁned in different dimensionality spaces, which is meaningless. It is like trying tocompare a sphere with a circle. The solution, proposed by (Green 1998) and known as reversible
jump MCMC orRJMCMC, is to augment the low dimensional space with extra random variables
so that the two spaces have a common measure.
Unfortunately, we do not have space to go into details here. Suffice it to say that the method

can be made to work in theory, although it is a bit tricky in practice. If, however, the continuousparameters can be integrated out (resulting in a method called collapsed RJMCMC), much of thedifficulty goes away, since we are just left with a discrete state space, where there is no needto worry about change of measure. For example, (Denison et al. 2002) includes many examplesof applications of collapsed RJMCMC applied to Bayesian inference fro adaptive basis-functionmodels. They sample basis functions from a ﬁxed set of candidates (e.g., centered on the datapoints), and integrate out the other parameters analytically. This provides a Bayesian alternativeto using RVMs or SVMs.

856 Chapter24. MarkovchainMonteCarlo(MCMC)inference
0 5 10 15 20p(0)(x)
0 5 10 15 20p(1)(x)
0 5 10 15 20p(2)(x)
0 5 10 15 20p(3)(x)
0 5 10 15 20p(10)(x)
0 5 10 15 20p(100)(x)
0 5 10 15 20p(200)(x)Initial Condition X0 = 10
0 5 10 15 20p(400)(x)
(a)0 5 10 15 20p(0)(x)
0 5 10 15 20p(200)(x)Initial Condition X0 = 17
(b)
Figure 24.9 Illustration of convergence to the uniform distribution over {0,1,...,20} using a symmetric
random walk starting from (left) state 10, and (right) state 17. Based on Figures 29.14 and 29.15 of (MacKay
2003). Figure generated by randomWalk0to20Demo .
24.4 Speed and accuracy of MCMC
In this section, we discuss a number of important theoretical and practical issues to do with
MCMC.
24.4.1 The burn-in phase

(b)
MCMC.
We start MCMC from an arbitrary initial state. As we explained in Section 17.2.3, only when thechainhas “forgotten”where itstarted fromwill thesamples becoming fromthe chain’sstationarydistribution. Samples collected before the chain has reached its stationary distribution do notcome from p
∗, and are usually thrown away. The initial period, whose samples will be ignored,
is called the burn-in phase.
For example, consider a uniform distribution on the integers {0,1,...,20}. Suppose we

sample from this using a symmetric random walk. In Figure 24.9, we show two runs of thealgorithm. On the left, we start in state 10; on the right, we start in state 17. Even in this smallproblem it takes over 100 steps until the chain has “forgotten” where it started from.
It is difficult to diagnose when the chain has burned in, an issue we discuss in more detail

below. (This is one of the fundamental weaknesses of MCMC.) As an interesting example of whatcan happen if you start collecting samples too early, consider the Potts model. Figure 24.10(a),shows a sample after 500 iterations of Gibbs sampling. This suggests that the model likes

24.4. SpeedandaccuracyofMCMC 857
(a)
 (b)
Figure 24.10 Illustration of problems caused by poor mixing. (a) One sample from a 5-state Potts model
on a128×128grid with 8 nearest neighbor connectivity and J=2/3(as in (Geman and Geman 1984)),
after 200 iterations. (b) One sample from the same model after 10,000 iterations. Used with kind permission
of Erik Sudderth.
medium-sized regions where the label is the same, implying the model would make a good
prior for image segmentation. Indeed, this was suggested in the original Gibbs sampling paper
(Geman and Geman 1984).
However, it turns out that if you run the chain long enough, you get isolated speckles, as
in Figure 24.10(b). The results depend on the coupling strength, but in general, it is very hard
to ﬁnd a setting which produces nice medium-sized blobs: most parameters result in a few
super-clusters, or lots of small fragments. In fact, there is a rapid phase transition between these

two regimes. This led to a paper called “The Ising/Potts model is not well suited to segmentation
tasks” (Morris et al. 1996). It is possible to create priors more suited to image segmentation
(e.g., (Sudderth and Jordan 2008)), but the main point here is that sampling before reaching
convergence can lead to erroneous conclusions.
24.4.2 Mixing rates of Markov chains *
The amount of time it takes for a Markov chain to converge to the stationary distribution, and

forget its initial state, is called the mixing time . More formally, we say that the mixing time
from state x0is the minimal time such that, for any constant ϵ>0,w eh a v et h a t
τϵ(x0)≜min{t:||δx0(x)Tt−p∗||1≤ϵ} (24.66)
whereδx0(x)is a distribution with all its mass in state x0,Tis the transition matrix of the
chain (which depends on the target p∗and the proposal q), andδx0(x)Ttis the distribution
aftertsteps. The mixing time of the chain is deﬁned as
τϵ≜max
x0τϵ(x0) (24.67)

τϵ≜max
The mixing time is determined by the eigengap γ=λ1−λ2, which is the difference of the

858 Chapter24. MarkovchainMonteCarlo(MCMC)inference
x2
x4
x7x5
x6 x1
x3
Figure 24.11 A Markov chain with low conductance. The dotted arcs represent transitions with very low
probability. Source: Figure 12.6 of (Koller and Friedman 2009). Used with kind permission of Daphne
Koller.
ﬁrst and second eigenvalues of the transition matrix. In particular, one can show that
τϵ≤O(1
γlogn
ϵ) (24.68)
wherenis the number of states. Since computing the transition matrix can be hard to do,
especially for high dimensional and/or continuous state spaces, it is useful to ﬁnd other ways to
estimate the mixing time.
An alternative approach is to examine the geometry of the state space. For example, consider

Koller.
τϵ≤O(1
γlogn
ϵ) (24.68)
the chain in Figure 24.11. We see that the state space consists of two “islands”, each of whichis connected via a narrow “bottleneck”. (If they were completely disconnected, the chain wouldnot be ergodic, and there would no longer be a unique stationary distribution.) We deﬁne theconductance φof a chain as the minimum probability, over all subsets of states, of transitioning
from that set to its complement:
φ≜min
S:0≤p∗(S)≤0.5∑
x∈S,x′∈ScT(x→x′)
p∗(S), (24.69)
τϵ≤O(1
φ2logn
ϵ) (24.70)

φ≜min
τϵ≤O(1
φ2logn
ϵ) (24.70)
Hence chains with low conductance have high mixing time. For example, distributions withwell-separated modes usually have high mixing time. Simple MCMC methods often do not workwell in such cases, and more advanced algorithms, such as parallel tempering, are necessary(see e.g., (Liu 2001)).
24.4.3 Practical convergence diagnostics

φ≜min
τϵ≤O(1
φ2logn
ϵ) (24.70)
Computing the mixing time of a chain is in general quite difficult, since the transition matrixis usually very hard to compute. In practice various heuristics have been proposed to diagnose

24.4. SpeedandaccuracyofMCMC 859
convergence — see (Geyer 1992; Cowles and Carlin 1996; Brooks and Roberts 1998) for a review.
Strictly speaking, these methods do not diagnose convergence, but rather non-convergence. Thatis, the method may claim the chain has converged when in fact it has not. This is a ﬂaw commonto all convergence diagnostics, since diagnosing convergence is computationally intractable ingeneral (Bhatnagar et al. 2010).
One of the simplest approaches to assessing when the method has converged is to run
multiple chains from very different overdispersed starting points, and to plot the samples of
some variables of interest. This is called a trace plot. If the chain has mixed, it should have
“forgotten” where it started from, so the trace plots should converge to the same distribution,and thus overlap with each other.
Figure 24.12 gives an example. We show the traceplot for xwhich was sampled from a

mixture of two 1D Gaussians using four different methods: MH with a symmetric Gaussianproposal of variance σ
2∈{1,8,500}, and Gibbs sampling. We see that σ2=1has not mixed,
which is also evident from Figure 24.7(a), which shows that a single chain never leaves the areawhere it started. The results for the other methods indicate that the chains rapidly converge tothe stationary distribution, no matter where they started. (The sticky nature of the σ
2= 500

2= 500
proposal is very evident. This reduces the computational efficiency, as we discuss below, butnot the statistical validity.)
24.4.3.1 Estimated potential scale reduction (EPSR)
We can assess convergence more quantitatively as follows. The basic idea is to compare thevariance of a quantity within each chain to its variance across chains. More precisely, supposewe collect Ssamples (after burn-in) from each of Cchains of Dvariables, x
isc,i=1:D,
s=1:S,c=1:C.L e tyscbe a scalar quantity of interest derived from x1:D,s,c(e.g.,

ysc=xiscfor some chosen i). Deﬁne the within-sequence mean and overall mean as
y·c≜1
SS∑
s=1ysc,y··≜1
CC∑
c=1y·c (24.71)
Deﬁne the between-sequence and within-sequence variance as
B≜S
C−1C∑
c=1(y·c−y··)2,W≜1
CC∑
c=1[
1
S−1S∑
s=1(ysc−y·c)2]
(24.72)
We can now construct two estimates of the variance of y. The ﬁrst estimate is W: this should
underestimate var[y]if the chains have not ranged over the full posterior. The second estimate
is
ˆV=S−1
SW+1
SB (24.73)
This is an estimate of var[y]that is unbiased under stationarity, but is an overestimate if the

y·c≜1
SS∑
CC∑
B≜S
C−1C∑
CC∑
c=1[
1
S−1S∑
(24.72)
is
ˆV=S−1
SW+1
SB (24.73)
starting points were overdispersed (Gelman and Rubin 1992). From this, we can deﬁne thefollowing convergence diagnostic statistic, known as the estimated potential scale reduction
orEPSR:
ˆR≜√
ˆV
W(24.74)

860 Chapter24. MarkovchainMonteCarlo(MCMC)inference
0 200 400 600 800 1000−1001020304050MH N(0,1.0002), Rhat = 1.493
(a)0 200 400 600 800 1000−60−40−200204060MH N(0,8.0002), Rhat = 1.039
(b)
0 200 400 600 800 1000−50−40−30−20−1001020304050MH N(0,500.0002), Rhat = 1.005
(c)0 200 400 600 800 1000−60−40−200204060gibbs, Rhat = 1.007
(d)
Figure 24.12 Traceplots for MCMC samplers. Each color represents the samples from a different starting
point. (a-c) MH with proposal N(x′|x,σ2)forσ2∈{1,8,500}, corresponding to Figure 24.7. (d) Gibbs
sampling. Figure generated by mcmcGmmDemo .
This quantity, which was ﬁrst proposed in (Gelman and Rubin 1992), measures the degree to
which the posterior variance would decrease if we were to continue sampling in the S→
∞limit. If ˆR≈1for any given quantity, then that estimate is reliable (or at least is not
unreliable). The ˆRvalues for the four samplers in Figure 24.12 are 1.493, 1.039, 1.005 and 1.007.

So this diagnostic has correctly identiﬁed that the sampler using the ﬁrst (σ2=1) proposal is
untrustworthy.
24.4.4 Accuracy of MCMC
The samples produced by MCMC are auto-correlated, and this reduces their information contentrelative to independent or “perfect” samples. We can quantify this as follows.
4Suppose we want
4. This Section is based on (Hoff 2009, Sec 6.6).

24.4. SpeedandaccuracyofMCMC 861
0 5 10 15 20 25 30 35 40 4500.10.20.30.40.50.60.70.80.91MH N(0,1.0002)
(a)0 5 10 15 20 25 30 35 40 4500.10.20.30.40.50.60.70.80.91MH N(0,8.0002)
(b)
0 5 10 15 20 25 30 35 40 45−0.200.20.40.60.811.2MH N(0,500.0002)
(c)0 5 10 15 20 25 30 35 40 45−0.200.20.40.60.811.2gibbs
(d)
Figure 24.13 Autocorrelation functions corresponding to Figure 24.12. Figure generated by mcmcGmmDemo .
to estimate the mean of f(X), for some function f,w h e r eX∼p(). Denote the true mean by
f∗≜E[f(X)] (24.75)
A Monte Carlo estimate is given by
f=1
SS∑
s=1fs (24.76)

862 Chapter24. MarkovchainMonteCarlo(MCMC)inference
wherefs≜f(xs)andxs∼p(x). An MCMC estimate of the variance of this estimate is given
by
VarMCMC[f]=E[
(f−f∗)2]
(24.77)
=E⎡
⎣{
1
SS∑
s=1(fs−f∗)}2⎤⎦ (24.78)
=1
S2E[S∑
s=1(fs−f∗)2]
+1
S2∑
s̸=tE[(fs−f∗)(ft−f∗)] (24.79)
=V a r MC(f)+1
S2∑
s̸=tE[(fs−f∗)(ft−f∗)] (24.80)
where the ﬁrst term is the Monte Carlo estimate of the variance if the samples weren’t correlated,
and the second term depends on the correlation of the samples. We can measure this as follows.Deﬁne the sample-based auto-correlation at lag tof a set of samples f
1,...,f Sas follows:
ρt≜1
S−t∑S−t
s=1(fs−f)(fs+t−f)
1
S−1∑S
s=1(fs−f)2(24.81)
This is called the autocorrelation function (ACF). This is plotted in Figure 24.13 for our four
samplers for the Gaussian mixture model. We see that the ACF of the Gibbs sampler (bottom
right) dies off to 0 much more rapidly than the MH samplers, indicating that each Gibbs sampleis “worth” more than each MH sample.

ρt≜1
S−t∑S−t
1
S−1∑S
A simple method to reduce the autocorrelation is to use thinning, in which we keep every
n’th sample. This does not increase the efficiency of the underlying sampler, but it does savespace, since it avoids storing highly correlated samples.
We can estimate the information content of a set of samples by computing the effective
sample size (ESS)Seff, deﬁned by
Seff≜Var
MC(f)
VarMCMC(f)(24.82)
From Figure 24.12, it is clear that the effective sample size of the Gibbs sampler is higher thanthat of the other samplers (in this example).
24.4.5 How many chains?

Seff≜Var
MC(f)
A natural question to ask is: how many chains should we run? We could either run one longchain to ensure convergence, and then collect samples spaced far apart, or we could run manyshort chains, but that wastes the burnin time. In practice it is common to run a mediumnumber of chains (say 3) of medium length (say 100,000 steps), and to take samples from eachafter discarding the ﬁrst half of the samples. If we initialize at a local mode, we may be able touse all the samples, and not wait for burn-in.

24.5. AuxiliaryvariableMCMC* 863
Model Goal Method Reference
Probit MAP Gradient Section 9.4.1
Probit MAP EM Section 11.4.6
Probit Post EP (Nickisch and Rasmussen 2008)Probit Post Gibbs+ Exercise 24.6Probit Post Gibbs with ARS (Dellaportas and Smith 1993)Probit Post MH using IRLS proposal (Gamerman 1997)
Logit MAP Gradient Section 8.3.4
Logit Post Gibbs+ with Student (Fruhwirth-Schnatter and Fruhwirth 2010)Logit Post Gibbs+ with KS (Holmes and Held 2006)
Table 24.1 Summary of some possible algorithms for estimation and inference for binary classiﬁcation
problems using Gaussian priors. Abbreviations: Aux. = auxiliary variable sampling, ARS = adaptive rejection
sampling, EP = expectation propagation, Gibbs+ = Gibbs sampling with auxiliary variables, IRLS = iterativereweighted least squares, KS = Kolmogorov Smirnov, MAP = maximum a posteriori, MH = MetropolisHastings, Post = posterior.
24.5 Auxiliary variable MCMC *
Sometimes we can dramatically improve the efficiency of sampling by introducing dummy

auxiliary variables, in order to reduce correlation between the original variables. If the originalvariables are denoted by x, and the auxiliary variables by z, we require that∑
zp(x,z)=p(x),
and that p(x,z)is easier to sample from than just p(x). If we meet these two conditions,
we can sample in the enlarged model, and then throw away the sampled zvalues, thereby
recovering samples from p(x). We give some examples below.
24.5.1 Auxiliary variable sampling for logistic regression

In Section 9.4.2, we discussed the latent variable interpretation of probit regression. Recall thatthis had the form
z
i≜wTxi+ϵi (24.83)
ϵi∼N(0,1) (24.84)
yi=1 = I(zi≥0) (24.85)
We exploited this representation in Section 11.4.6, where we used EM to ﬁnd an ML estimate. Itis straightforward to convert this into an auxiliary variable Gibbs sampler (Exercise 24.6), sincep(w|D)is Gaussian and p(z
i|xi,yi,w)is truncated Gaussian, both of which are easy to sample
from.

z
from.
Now let us discuss how to derive an auxiliary variable Gibbs sampler for logistic regression.
Letϵifollow alogistic distribution, with pdf
pLogistic(ϵ)=e−ϵ
(1+e−ϵ)2(24.86)
with mean E[ϵ]=0and variance var[ϵ]=π2/3. The cdf has the form F(ϵ) = sigm(ϵ ), which

864 Chapter24. MarkovchainMonteCarlo(MCMC)inference
is the logistic function. Since yi=1iffwTxi+ϵ>0, we have, by symmetry, that
p(yi=1|xi,w)=∫∞
−wTxif(ϵ)dϵ=∫wTxi
−∞f(ϵ)dϵ=F(wTxi) = sigm(wTxi)(24.87)
as required.
WecanderiveanauxiliaryvariableGibbssamplerbysamplingfrom p(z|w,D)andp(w|z,D).
Unfortunately, sampling directly from p(w|z,D)is not possible. One approach is to deﬁne
ϵi∼N(0,λi),w h e r eλi=( 2ψi)2andψi∼KS, the Kolmogorov Smirnov distribution, and then
to sample w,z,λandψ(Holmes and Held 2006).
A simpler approach is to approximate the logistic distribution by the Student distribution
(Albert and Chib 1993). Speciﬁcally, we will make the approximation ϵi∼T(0,1,ν),w h e r e
ν≈8. We can now use the scale mixture of Gaussians representation of the Student to simplify
inference. In particular, we write
λi∼Ga(ν/2,ν/2) (24.88)
ϵi∼N(0,λ−1
i) (24.89)
zi≜wTxi+ϵi (24.90)
yi=1|zi=I(zi≥0) (24.91)
All of the full conditionals now have a simple form; see Exercise 24.7 for the details.

i) (24.89)
N o t et h a ti fw es e t ν=1, thenzi∼N(wTxi,1), which is equivalent to probit regression (see
Section 9.4). Rather than choosing between probit or logit regression, we can simply estimate
theνparameter. There is no convenient conjugate prior, but we can consider a ﬁnite range of
possible values and evaluate the posterior as follows:
p(ν|λ)∝p(ν)N∏
i=11
Γ(ν/2)(ν/2)ν/2λν/2−1
ie−νλi/2(24.92)
Furthermore, if we deﬁne V0=v0I, we can sample v0as well. For example, suppose we use

i=11
aIG(δ1,δ2)prior forv0. The posterior is given by p(v0|w)=I G (δ1+1
2D,δ2+1
2∑D
j=1w2
j).
This can be interleaved with the other Gibbs sampling steps, and provides an appealing Bayesian
alternative to cross validation for setting the strength of the regularizer.
See Table 24.1 for a summary of various algorithms for ﬁtting probit and logit models. Many
of these methods can also be extended to the multinomial logistic regression case. For details,see (Scott 2009; Fruhwirth-Schnatter and Fruhwirth 2010).
24.5.2 Slice sampling

2D,δ2+1
2∑D
j=1w2
j).
Consider sampling from a univariate, but multimodal, distribution ˜p(x). We can sometimes
improve the ability to make large moves by adding an auxiliary variable u. We deﬁne the joint
distribution as follows:
ˆp(x,u)={1/Zpif0≤u≤˜p(x)
0otherwise(24.93)

24.5. AuxiliaryvariableMCMC* 865
x xx u(i+1)
(i)(i+1)f(x  )(i)
(a)−5 −4 −3 −2 −1 0 1 2 3 4020406080100120140160180
(b)
Figure 24.14 (a) Illustration of the principle behind slice sampling. Given a previous sample xi,w e
sampleui+1uniformly on [0,f(xi)],w h e r efis the target density. We then sample xi+1along the slice
wheref(x)≥ui+1. Source: Figure 15 of (Andrieu et al. 2003) . Used with kind permission of Nando de
Freitas. (b) Slice sampling in action. Figure generated by sliceSamplingDemo1d .
3456
−2.5
−2
−1.5
−101234x 10−11
InterceptSlopePosterior density
(a)
 (b)
Figure 24.15 Binomial regression for 1d data. (a) Grid approximation to posterior. (b) Slice sampling
approximation. Figure generated by sliceSamplingDemo2d .
whereZp=∫
˜p(x)dx. The marginal distribution over xis given by
∫
ˆp(x,u)du=∫˜p(x)
01
Zpdu=˜p(x)
Zp=p(x) (24.94)
so we can sample from p(x)by sampling from ˆp(x,u)and then ignoring u. The full conditionals
have the form
p(u|x)=U[0,˜p(x)](u) (24.95)
p(x|u)=UA(x) (24.96)

3456
−2.5
−2
−1.5
(a)
 (b)
whereZp=∫
∫
01
whereA={x:˜p(x)≥u}is the set of points on or above the chosen height u. This
corresponds to a slice through the distribution, hence the term slice sampling (Neal 2003a).
See Figure 24.14(a).
In practice, it can be difficult to identify the set A. So we can use the following approach:
construct an interval xmin≤x≤xmaxaround the current point xsof some width. We then

866 Chapter24. MarkovchainMonteCarlo(MCMC)inference
test to see if each end point lies within the slice. If it does, we keep extending in that direction
until it lies outside the slice. This is called stepping out. A candidate value x′is then chosen
uniformly from this region. If it lies within the slice, it is kept, so xs+1=x′. Otherwise we
shrink the region such that x′forms one end and such that the region still contains xs. Then
another sample is drawn. We continue in this way until a sample is accepted.
To apply the method to multivariate distributions, we can sample one extra auxiliary variable
for each dimension. The advantage of slice sampling over Gibbs is that it does not needa speciﬁcation of the full-conditionals, just the unnormalized joint. The advantage of slicesampling over MH is that it does not need a user-speciﬁed proposal distribution (although itdoes require a speciﬁcation of the width of the stepping out interval).

Figure 24.14(b) illustrates the algorithm in action on a synthetic 1d problem. Figure 24.15
illustrates its behavior on a slightly harder problem, namely binomial logistic regression. Themodel has the form
y
i∼Bin(ni,logit(β1+β2xi)) (24.97)
We use a vague Gaussian prior for the βj’s. Figure 24.15(a) shows a grid-based approximation
to the posterior, and Figure 24.15(b) shows a sample-based approximation. In this example, thegrid is faster to compute, but for any problem with more than 2 dimensions, the grid approachis infeasible.
24.5.3 Swendsen Wang

y
Consider an Ising model of the following form:
p(x)=1
Z∏
efe(xe) (24.98)
wherexe=(xi,xj)for edgee=(i,j),xi∈{+1,−1}, and the edge factor feis deﬁned by(eJe−J
e−JeJ)
,w h e r eJis the edge strength. Gibbs sampling in such models can be slow when
Jis large in absolute value, because neighboring states can be highly correlated. The Swendsen
Wangalgorithm (Swendsen and Wang 1987) is a auxiliary variable MCMC sampler which mixes
much faster, at least for the case of attractive or ferromagnetic models, with J>0.

p(x)=1
Z∏
e−JeJ)
Suppose we introduce auxiliary binary variables, one per edge.5These are called bond
variables, and will be denoted by z. We then deﬁne an extended model p(x,z)of the form
p(x,z)=1
Z′∏
ege(xe,ze) (24.99)
whereze∈{0,1}, and we deﬁne the new factor as follows: ge(xe,ze=0 )=(e−Je−J
e−Je−J)
,
andge(xe,ze=1 )=(eJ−e−J0
0eJ−e−J)
. It is clear that∑1
ze=0ge(xe,ze)=fe(xe),
5. Our presentation of the method is based on some notes by David Mackay, available from http://www .inference

p(x,z)=1
Z′∏
e−Je−J)
,
0eJ−e−J)
.phy.cam.ac.uk/mackay/itila/swendsen .pdf.

24.5. AuxiliaryvariableMCMC* 867
Figure 24.16 Illustration of the Swendsen Wang algorithm on a 2d grid. Used with kind permission of
Kevin Tang.
and hence that∑
zp(x,z)=p(x). So if we can sample from this extended model, we can just
throw away the zsamples and get valid xsamples from the original distribution.
Fortunately, it is easy to apply Gibbs sampling to this extended model. The full conditional
p(z|x)factorizes over the edges, since the bond variables are conditionally independent given
the node variables. Furthermore, the full conditional p(ze|xe)is simple to compute: if the
nodes on either end of the edge are in the same state ( xi=xj), we set the bond zeto 1 with
probability p=1−e−2J, otherwise we set it to 0. In Figure 24.16 (top right), the bonds that
could be turned on (because their corresponding nodes are in the same state) are represented
by dotted edges. In Figure 24.16 (bottom right), the bonds that are randomly turned on are
represented by solid edges.

To sample p(x|z), we proceed as follows. Find the connected components deﬁned by the
graph induced by the bonds that are turned on. (Note that a connected component may consist
of a singleton node.) Pick one of these components uniformly at random. All the nodes in each
such component must have the same state, since the off-diagonal terms in the ge(xe,ze=1 )
factor are 0. Pick a state ±1uniformly at random, and force all the variables in this component

to adopt this new state. This is illustrated in Figure 24.16 (bottom left), where the green square

868 Chapter24. MarkovchainMonteCarlo(MCMC)inference
denotes the selected connected component, and we choose to force all nodes within in to enter
the white state.
The validity of this algorithm is left as an exercise, as is the extension to handle local evidence
and non-stationary potentials.
It should be intuitively clear that Swendsen Wang makes much larger moves through the state
space than Gibbs sampling. In fact, SW mixes much faster than Gibbs sampling on 2d latticeIsing models for a variety of values of the coupling parameter, provided J>0. More precisely,
let the edge strength be parameterized by J/T,w h e r eT>0is a computational temperature.
For large T, the nodes are roughly independent, so both methods work equally well. However,
asTapproaches a critical temperature T
c, the typical states of the system have very long

correlation lengths, and Gibbs sampling takes a very long time to generate independent samples.As the temperature continues to drop, the typical states are either all on or all off. The frequencywith which Gibbs sampling moves between these two modes is exponentiall small. By contrast,SW mixes rapidly at all temperatures.
Unfortunately, if any of the edge weights are negative, J<0, the system is frustrated, and

there are exponentially many modes, even at low temperature. SW does not work very well inthis setting, since it tries to force many neighboring variables to have the same state. In fact,computation in this regime is provably hard for any algorithm (Jerrum and Sinclair 1993, 1996).
24.5.4 Hybrid/Hamiltonian MCMC *

In this section, we brieﬂy mention a way to perform MCMC sampling for continuous statespaces, for which we can compute the gradient of the (unnormalized) log-posterior. This is thecase in neural network models, for example.
The basic idea is to think of the parameters as a particle in space, and to create auxiliary
variables which represent the “momentum” of this particle. We then update this parameter/momentum pair according to certain rules (see e.g., (Duane et al. 1987; Neal 1993; MacKay 2003;Neal 2010) for details). The resulting method is called hybrid MCMC orHamiltonian MCMC.

The two main parameters that the user must specify are how many leapfrog steps to take
when updating the position/ momentum, and how big to make these steps. Performance canbe quite sensitive to these parameters (although see (Hoffman and Gelman 2011) for a recentway to set them automatically). This method can be combined with stochastic gradient descent(Section 8.5.2) in order to handle large datasets, as explained in (Ahn et al. 2012).
Recently, a more powerful extension of this method has been developed, that exploits second-
order gradient information. See (Girolami et al. 2010) for details.
24.6 Annealing methods

Many distributions are multimodal and hence hard to sample from. However, by analogy to theway metals are heated up and then cooled down in order to make the molecules align, we canimagine using a computational temperature parameter to smooth out a distribution, graduallycooling it to recover the original “bumpy” distribution. We ﬁrst explain this idea in more detailin the context of an algorithm for MAP estimation. We then discuss extensions to the samplingcase.

24.6. Annealingmethods 869
01020304050
02040600246810
xtemp 1.000
y
(a)01020304050
02040600246810x 105
xtemp 0.200
y
(b)
Figure 24.17 An energy surface at different temperatures. Note the different vertical scales. (a) T=1.
(b)T=0.5. Figure generated by saDemoPeaks .
24.6.1 Simulated annealing
Simulated annealing (Kirkpatrick et al. 1983) is a stochastic algorithm that attempts to ﬁnd
the global optimum of a black-box function f(x). It is closely related to the Metropolis-
Hastings algorithm for generating samples from a probability distribution, which we discussed
in Section 24.3. SA can be used for both discrete and continuous optimization.
The method is inspired by statistical physics. The key quantity is the Boltzmann distribution,
which speciﬁes that the probability of being in any particular state xis given by
p(x)∝exp(−f(x)/T) (24.100)
wheref(x)is the “energy” of the system and Tis the computational temperature. As the

temperature approaches 0 (so the system is cooled), the system spends more and more time inits minimum energy (most probable) state.
Figure 24.17 gives an example of a 2d function at different temperatures. At high temperatures,
T≫1, the surface is approximately ﬂat, and hence it is easy to move around (i.e., to avoid

local optima). As the temperature cools, the largest peaks become larger, and the smallest peaksdisappear. By cooling slowly enough, it is possible to “track” the largest peak, and thus ﬁnd theglobal optimum. This is an example of a continuation method.
We can generate an algorithm from this as follows. At each step, sample a new state according
to some proposal distribution x
′∼q(·|xk). For real-valued parameters, this is often simply a
random walk proposal, x′=xk+ϵk,w h e r eϵk∼N(0,Σ). For discrete optimization, other
kinds of local moves must be deﬁned.

Having proposed a new state, we compute
α= exp((f (x)−f(x′))/T) (24.101)
We then accept the new state (i.e., set xk+1=x′) with probability min(1,α), otherwise we stay
in the current state (i.e., set xk+1=xk). This means that if the new state has lower energy (is
more probable), we will deﬁnitely accept it, but it it has higher energy (is less probable), we mightstill accept, depending on the current temperature. Thus the algorithm allows “down-hill” movesin probability space (up-hill in energy space), but less frequently as the temperature drops.

870 Chapter24. MarkovchainMonteCarlo(MCMC)inference
0 200 400 600 800 1000 120000.10.20.30.40.50.60.70.80.91temperature vs iteration
(a)0 200 400 600 800 1000−8.5−8−7.5−7−6.5−6−5.5−5−4.5energy vs iteration
(b)
Figure 24.18 A run of simulated annealing on the energy surface in Figure 24.17. (a) Temperature vs
iteration. (b) Energy vs iteration. Figure generated by saDemoPeaks .
353637383940
22242628300102030405060
x
iter 550, temp 0.064
y
(a)353637383940
2224262830050100150
x
iter 1000, temp 0.007
y
(b)
Figure 24.19 Histogram of samples from the annealed “posterior” at 2 different time points produced by
simulated annealing on the energy surface shown in Figure 24.17. Note that at cold temperatures, most of
the samples are concentrated near the peak at (38,25). Figure generated by saDemoPeaks .
The rate at which the temperature changes over time is called the cooling schedule .I t
has been shown (Kirkpatrick et al. 1983) that if one cools sufficiently slowly, the algorithm will

y
(b)
provably ﬁnd the global optimum. However, it is not clear what “sufficient slowly” means.
In practice it is common to use an exponential cooling schedule of the following form:
Tk=T0Ck,w h e r eT0is the initial temperature (often T0∼1) andCis the cooling rate (often
C∼0.8). See Figure 24.18(a) for a plot of this cooling schedule. Cooling too quickly means one
can get stuck in a local maximum, but cooling too slowly just wastes time. The best cooling

schedule is difficult to determine; this is one of the main drawbacks of simulated annealing.
Figure 24.18(b) shows an example of simulated annealing applied to the function in Figure 24.17
using a random walk proposal. We see that the method stochastically reduces the energy
over time. Figures 24.19 illustrate (a histogram of) samples drawn from the cooled probability
distribution over time. We see that most of the samples are concentrated near the global
maximum. When the algorithm has converged, we just return the largest value found.

24.6. Annealingmethods 871
24.6.2 Annealed importance sampling
We now describe a method known as annealed importance sampling (Neal 2001) that com-
bines ideas from simulated annealing and importance sampling in order to draw independent
samples from difficult (e.g., multimodal) distributions.
Suppose we want to sample from p0(x)∝f0(x), but we cannot do so easily; for example,
this might represent a multimodal posterior. Suppose however that there is an easier distributionwhich we can sample from, call it p
n(x)∝fn(x); for example, this might be the prior. We
can now construct a sequence of intermediate distributions than move slowly from pntop0as
follows:
fj(x)=f0(x)βjfn(x)1−βj(24.102)
where1=β0>β1>···>βn=0,w h e r eβjis an inverse temperature. (Contrast this to the
scheme used by simulated annealing which has the form fj(x)=f0(x)βj; this makes it hard
to sample from pn.) Furthermore, suppose we have a series of Markov chains Tj(x,x′)(fromx

follows:
tox′) which leave each pjinvariant. Given this, we can sample xfromp0by ﬁrst sampling a
sequence z=(zn−1,...,z0)as follows: sample zn−1∼pn; samplezn−2∼Tn−1(zn−1,·); ...;
samplez0∼T1(z1,·). Finally we set x=z0and give it weight
w=fn−1(zn−1)
fn(zn−1)fn−2(zn−2)
fn−1(zn−2)···f1(z1)
f2(z1)f0(z0)
f1(z0)(24.103)
This can be shown to be correct by viewing the algorithm as a form of importance sampling
in an extended state space z=(z0,...,zn−1). Consider the following distribution on this state
space:
p(z)∝f(z)=f0(z0)˜T1(z0,z1)˜T2(z1,z2)···˜Tn−1(zn−2,zn−1) (24.104)

space:
where˜Tjis the reversal of Tj:
˜Tj(z,z′)=Tj(z′,z)pj(z′)/pj(z)=Tj(z′,z)fj(z′)/fj(z) (24.105)
It is clear that∑
z1,...,zn−1f(z)=f0(z0), so we can safely just use the z0part of these
sequences to recover the original ditribution.
Now consider the proposal distribution deﬁned by the algorithm:
q(z)∝g(z)=fn(zn−1)Tn−1(zn−1,zn−2)···T2(z2,z1)T1(z1,z0) (24.106)
One can show that the importance weights w=f(z0,...,zn−1)
g(z0,...,zn−1)are given by Equation 24.103.
24.6.3 Parallel tempering

Another way to combine MCMC and annealing is to run multiple chains in parallel at differenttemperatures, and allow one chain to sample from another chain at a neighboring temperature.In this way, the high temperature chain can make long distance moves through the state space,and have this inﬂuence lower temperature chains. This is known as parallel tempering. See
e.g., (Earl and Deem 2005) for details.

872 Chapter24. MarkovchainMonteCarlo(MCMC)inference
24.7 Approximating the marginal likelihood
The marginal likelihood p(D|M)is a key quantity for Bayesian model selection, and is given by
p(D|M)=∫
p(D|θ,M)p(θ|M)dθ (24.107)
Unfortunately, this integral is often intractable to compute, for example if we have non conjugate
priors, and/or we have hidden variables. In this section, we brieﬂy discuss some ways toapproximate this expression using Monte Carlo. See (Gelman and Meng 1998) for a moreextensive review.
24.7.1 The candidate method
There is a simple method for approximating the marginal likelihood known as the Candidate
method(Chib 1995). This exploits the following identity:
p(D|M)=p(D|θ,M)p(θ|M)
p(θ|D,M)(24.108)
This holds for any value of θ. Once we have picked some value, we can evaluate p(D|θ,M)
andp(θ|M)quite easily. If we have some estimate of the posterior near θ, we can then evaluate
the denominator as well. This posterior is often approximated using MCMC.

The ﬂaw with this method is that it relies on the assumption that p(θ|D,M)has marginalized
over all the modes of the posterior, which in practice is rarely possible. Consequently the methodcan give very inaccurate results in practice (Neal 1998).
24.7.2 Harmonic mean estimate
Newton and Raftery (1994) proposed a simple method for approximating p(D)using the output
of MCMC, as follows:
1/p(D)≈1
SS∑
s=11
p(D|θs)(24.109)
whereθs∼p(θ|D). This expression is the harmonic mean of the likelihood of the data under

1/p(D)≈1
SS∑
s=11
each sample. The theoretical correctness of this expression follows from the following identity:
∫1
p(D|θ)p(θ|D)dθ=∫1
p(D|θ)p(D|θ)p(θ)
p(D)dθ=1
p(D)∫
p(θ|D)dθ=1
p(D)(24.110)
Unfortunately, in practice this method works very poorly. Indeed, Radford Neal called this “theworst Monte Carlo method ever”.
6. The reason it is so bad is that it depends only on samples

1/p(D)≈1
SS∑
s=11
∫1
p(D)dθ=1
p(D)∫
drawn from the posterior. But the posterior is often very insensitive to the prior, whereas themarginal likelihood is not. We only mention this method in order to warn against its use. Wepresent a better method below.
6. Source: radfordneal .wordpress .com/2008/08/17/the-harmonic-mean-of-the-likelihood-worst-mon
te-carlo-method-ever .

24.7. Approximatingthemarginallikelihood 873
24.7.3 Annealed importance sampling
We can use annealed importance sampling (Section 24.6.2) to evaluate a ratio of partition
functions. Notice that Z0=∫
f0(x)dx=∫
f(z)dz, andZn=∫
fn(x)dx=∫
g(z)dz. Hence
Z0
Zn=∫
f(z)dz∫
g(z)dz=∫f(z)
g(z)g(z)dz∫
g(z)dz=Eq[f(z)
g(z)]
≈1
SS∑
s=1ws (24.111)
Iffnis a prior and f0is the posterior, we can estimate Zn=p(D)using the above equation,
provided the prior has a known normalization constant Z0. This is generally considered the
method of choice for evaluating difficult partition functions.
Exercises
Exercise 24.1 Gibbs sampling from a 2D Gaussian
Suppose x∼N(μ,Σ),w h e r eμ=( 1,1)andΣ=( 1,−0.5;−0.5,1). Derive the full condition-
alsp(x1|x2)andp(x2|x1). Implement the algorithm and plot the 1d marginals p(x1)andp(x2)as
histograms. Superimpose a plot of the exact marginals.
Exercise 24.2 Gibbs sampling for a 1D Gaussian mixture model

Exercises
Consider applying Gibbs sampling to a univariate mixture of Gaussians, as in Section 24.2.3. Derive the
expressions for the full conditionals. Hint: if we know zn=j(say), then μjgets “connected” to xn, but
all other values of μi, for alli̸=j, are irrelevant. (This is an example of context-speciﬁc independence,
where the structure of the graph simpliﬁes once we have assigned values to some of the nodes.) Hence,given all the z
nvalues, the posteriors of the μ’s should be independent, so the conditional of μjshould

be independent of μ−j. (Similarly for σj.)
Exercise 24.3 Gibbs sampling from the Potts model
Modify the code in gibbsDemoIsing to draw samples from a Potts prior at different temperatures, as in
Figure 19.8.
Exercise 24.4 Full conditionals for hierarchical model of Gaussian means
Let us reconsider the Gaussian-Gaussian model in Section 5.6.2 for modelling multiple related mean
parameters θj. In this exercise we derive a Gibbs sampler instead of using EB. Suppose, following (Hoff
2009, p134)), that we use the following conjugate priors on the hyper-parameters:
μ∼N(μ0,γ2
0) (24.112)
τ2∼IG(η0/2,η0τ2

μ∼N(μ0,γ2
0/2) (24.113)
0/2) (24.114)

874 Chapter24. MarkovchainMonteCarlo(MCMC)inference
We can set η=(μ0,γ0,η0,τ0,ν0,σ0)to uninformative values. Given this model speciﬁcation, show that
the full conditionals for μ,τ,σand theθjare as follows:
p(μ|θ1:D,τ2)=N (μ|Dθ/τ2+μ0/γ2
0
D/τ2+1/γ2
0,[D/τ2+1/γ2
0]−1) (24.115)
p(θj|μ,τ2,Dj,σ2)=N (θj|Njxj/σ2+1/τ2
Nj/σ2+1/τ2,[Nj/σ2+1/τ2]−1) (24.116)
p(τ2|θ1:D,μ)=I G ( τ2|η0+D
2,η0τ2
0+∑
j(θj−μ)2
2) (24.117)
p(σ2|θ1:D,D)=I G ( σ2|1
2[ν0+D∑
j=1Nj],1
2[ν0σ2
0+D∑
j=1Nj∑
i=1(xij−θj)2]) (24.118)
Exercise 24.5 Gibbs sampling for robust linear regression with a Student t likelihood
Modify the EM algorithm in Exercise 11.12 to perform Gibbs sampling for p(w,σ2,z|D,ν).
Exercise 24.6 Gibbs sampling for probit regression
Modify the EM algorithm in Section 11.4.6 to perform Gibbs sampling for p(w,z|D). Hint: we can
sample from a truncated Gaussian, N(z|μ,σ)I(a≤z≤b)in two steps: ﬁrst sample u∼U(Φ((a−
μ)/σ),Φ((b−μ)/σ)), then set z=μ+σΦ−1(u)(Robert 1995).

j=1Nj],1
2[ν0σ2
0+D∑
j=1Nj∑
Exercise 24.7 Gibbs sampling for logistic regression with the Student approximation
Derive the full conditionals for the joint model deﬁned by Equations 24.88 to 24.91.

25 Clustering
25.1 Introduction
Clustering is the process of grouping similar objects together. There are two kinds of inputs we
might use. In similarity-based clustering, the input to the algorithm is an N×Ndissimilarity
matrixordistance matrix D.I nfeature-based clustering, the input to the algorithm is an
N×Dfeature matrix or design matrix X. Similarity-based clustering has the advantage that it
allows for easy inclusion of domain-speciﬁc similarity or kernel functions (Section 14.2). Feature-
based clustering has the advantage that it is applicable to “raw”, potentially noisy data. We willsee examples of both below.
In addition to the two types of input, there are two possible types of output: ﬂat cluster-
ing, also called partitional clustering, where we partition the objects into disjoint sets; and
hierarchical clustering , where we create a nested tree of partitions. We will discuss both of
these below. Not surprisingly, ﬂat clusterings are usually faster to create (O (ND)for ﬂat vs
O(N

O(N
2logN)for hierarchical), but hierarchical clusterings are often more useful. Furthermore,
most hierarchical clustering algorithms are deterministic and do not require the speciﬁcation ofK, the number of clusters, whereas most ﬂat clustering algorithms are sensitive to the initial
conditions and require some model selection method for K. (We will discuss how to choose K
in more detail below.)
The ﬁnal distinction we will make in this chapter is whether the method is based on a

O(N
probabilistic model or not. One might wonder why we even bother discussing non-probabilisticmethods for clustering. The reason is two-fold: ﬁrst, they are widely used, so readers shouldknow about them; second, they often contain good ideas, which can be used to speed upinference in a probabilistic models.
25.1.1 Measuring (dis)similarity
A dissimilarity matrix Dis a matrix where di,i=0anddi,j≥0is a measure of “distance”
between objects iandj. Subjectively judged dissimilarities are seldom distances in the strict

sense, since the triangle inequality, di,j≤di,k+dj,k, often does not hold. Some algorithms
requireDto be a true distance matrix, but many do not. If we have a similarity matrix S,w e
can convert it to a dissimilarity matrix by applying any monotonically decreasing function, e.g.,D= max(S )−S.
The most common way to deﬁne dissimilarity between objects is in terms of the dissimilarity

876 Chapter25. Clustering
of their attributes:
Δ(xi,xi′)=D∑
j=1Δj(xij,xi′j) (25.1)
Some common attribute dissimilarity functions are as follows:
• Squared (Euclidean) distance:
Δj(xij,xi′j)=(xij−xi′j)2(25.2)
Of course, this only makes sense if attribute jis real-valued.
• Squared distance strongly emphasizes large differences (because differences are squared). A
more robust alternative is to use an ℓ1distance:
Δj(xij,xi′j)=|xij−xi′j| (25.3)
This is also called city block distance, since, in 2D, the distance can be computed by
counting how many rows and columns we have to move horizontally and vertically to get
fromxitoxi′.
•I fxiis a vector (e.g., a time-series of real-valued data), it is common to use the correlation
coefficient (see Section 2.5.1). If the data is standardized, then corr[xi,xi′]=∑
jxijxi′j,
and hence∑
j(xij−xi′j)2=2 ( 1−corr[xi,xi′]). So clustering based on correlation
(similarity) is equivalent to clustering based on squared distance (dissimilarity).

jxijxi′j,
and hence∑
• For ordinal variables, such as {low, medium, high}, it is standard to encode the values as
real-valued numbers, say 1/3,2/3,3/3if there are 3 possible values. One can then apply
any dissimilarity function for quantitative variables, such as squared distance.
• For categorical variables, such as {red, green, blue}, we usually assign a distance of 1 if the
features are different, and a distance of 0 otherwise. Summing up over all the categoricalfeatures gives
Δ(x
i,xi)=D∑
j=1I(xij̸=xi′j) (25.4)
This is called the hamming distance.

Δ(x
i,xi)=D∑
25.1.2 Evaluating the output of clustering methods *
The validation of clustering structures is the most difficult and frustrating part of clusteranalysis. Without a strong effort in this direction, cluster analysis will remain a black artaccessible only to those true believers who have experience and great courage. — Jainand Dubes (Jain and Dubes 1988)

Intuitively, the goal of clustering is to assign points that are similar to the same cluster,
and to ensure that points that are dissimilar are in different clusters. There are several waysof measuring these quantities e.g., see (Jain and Dubes 1988; Kaufman and Rousseeuw 1990).However, these internal criteria may be of limited use. An alternative is to rely on some externalform of data with which to validate the method. For example, suppose we have labels for eachobject, as in Figure 25.1. (Equivalently, we can have a reference clustering; given a clustering, wecan induce a set of labels and vice versa.) Then we can compare the clustering with the labelsusing various metrics which we describe below. We will use some of these metrics later, whenwe compare clustering methods.
25.1.2.1 Purity
LetN
ijbe the number of objects in cluster ithat belong to class j, and letNi=∑C
j=1Nijbe
the total number of objects in cluster i. Deﬁne pij=Nij/Ni; this is the empirical distribution

LetN
j=1Nijbe
over class labels for cluster i. We deﬁne the purityof a cluster as pi≜maxjpij, and the
overall purity of a clustering as
purity≜∑
iNi
Npi (25.5)
For example, in Figure 25.1, we have that the purity is
6
175
6+6
174
6+5
173
5=5+4+3
17=0.71 (25.6)
The purity ranges between 0 (bad) and 1 (good). However, we can trivially achieve a purity of
1 by putting each object into its own cluster, so this measure does not penalize for the number
of clusters.
25.1.2.2 Rand index
LetU={u1,...,u R}andV={v1,...,V C}be two different partitions of the Ndata points,
i.e., two different (ﬂat) clusterings. For example, Umight be the estimated clustering and V
is reference clustering derived from the class labels. Now deﬁne a 2×2contingency table,

878 Chapter25. Clustering
containing the following numbers: TPis the number of pairs that are in the same cluster in
bothUandV(true positives); TNis the number of pairs that are in the different clusters in
bothUandV(true negatives); FNis the number of pairs that are in the different clusters in
Ubut the same cluster in V(false negatives); and FPis the number of pairs that are in the
same cluster in Ubut different clusters in V(false positives). A common summary statistic is
theRand index:
R≜TP+TN
TP+FP+FN+TN(25.7)
This can be interpreted as the fraction of clustering decisions that are correct. Clearly 0≤R≤
1.
For example, consider Figure 25.1, The three clusters contain 6, 6 and 5 points, so the number
of “positives” (i.e., pairs of objects put in the same cluster, regardless of label) is
TP+FP=(6
2)
+(62)
+(52)
=4 0 (25.8)
Of these, the number of true positives is given by
TP=(52)
+(52)
+(32)
+(22)
=2 0 (25.9)
where the last two terms come from cluster 3: there are(32)
pairs labeled Cand(22)
pairs

1.
TP+FP=(6
2)
+(62)
+(52)
=4 0 (25.8)
TP=(52)
+(52)
+(32)
+(22)
=2 0 (25.9)
pairs
labeledA.S oFP=4 0−20 = 20. Similarly, one can show FN=2 4andTN=7 2. So the
Rand index is (20+72) /(20+20+24+72) = 0. 68.
The Rand index only achieves its lower bound of 0 if TP=TN=0, which is a rare event.
One can deﬁne an adjust
 ed Rand index (Hubert and Arabie 1985) as follows:
AR≜index−expected index
max index−expected index(25.10)

TP=(52)
+(52)
+(32)
+(22)
=2 0 (25.9)
pairs
Here the model of randomness is based on using the generalized hyper-geometric distribution,i.e., the two partitions are picked at random subject to having the original number of classesand objects in each, and then the expected value of TP+TNis computed. This model can
be used to compute the statistical signiﬁcance of the Rand index.
The Rand index weights false positives and false negatives equally. Various other summary

statistics for binary decision problems, such as the F-score (Section 5.7.2.2), can also be used.One can compute their frequentist sampling distribution, and hence their statistical signiﬁcance,using methods such as bootstrap.
25.1.2.3 Mutual information
Another way to measure cluster quality is to compute the mutual information between Uand
V(Vaithyanathan and Dom 1999). To do this, let p
UV(i,j)=|ui∩vj|
Nbe the probability that
a randomly chosen object belongs to cluster uiinUandvjinV. Also, let pU(i)=|ui|/N

be the be the probability that a randomly chosen object belongs to cluster uiinU; deﬁne

25.2. Dirichletprocessmixturemodels 879
pV(j)=|vj|/Nsimilarly. Then we have
I(U,V)=R∑
i=1C∑
j=1pUV(i,j)logpUV(i,j)
pU(i)pV(j)(25.11)
This lies between 0 and min{H(U),H(V)}. Unfortunately, the maximum value can be
achieved by using lots of small clusters, which have low entropy. To compensate for this,
we can use the normalized mutual information,
NMI(U,V)≜I(U,V)
(H(U)+H(V))/2(25.12)
This lies between 0 and 1. A version of this that is adjusted for chance (under a particularrandom data model) is described in (Vinh et al. 2009). Another variant, called variation of
information, is described in (Meila 2005).
25.2 Dirichlet process mixture models
The simplest approach to (ﬂat) clustering is to use a ﬁnite mixture model, as we discussed inSection 11.2.3. This is sometimes called model-based clustering, since we deﬁne a probabilistic
model of the data, and optimize a well-deﬁned objective (the likelihood or posterior), as opposedto just using some heuristic algorithm.

The principle problem with ﬁnite mixture models is how to choose the number of components
K. We discussed several techniques in Section 11.5. However, in many cases, there is no well-
deﬁned number of clusters. Even in the simple 2d height-weight data (Figure 1.8), it is not clearif the “correct” value of Kshould be 2, 3, or 4. It would be much better if we did not have to
chooseKat all.
In this section, we discuss inﬁnite mixture models , in which we do not impose any a priori
bound on K. To do this, we will use a non-parametric prior based on the Dirichlet process

(DP). This allows the number of clusters to grow as the amount of data increases. It will alsoprove useful later when we discuss hiearchical clustering.
The topic of non-parametric Bayes is currently very active, and we do not have space to
go into details (see (Hjort et al. 2010) for a recent book on the topic). Instead we just give abrief review of the DP and its application to mixture modeling, based on the presentation in(Sudderth 2006, sec 2.2).
25.2.1 From ﬁnite to inﬁnite mixture models

Consider a ﬁnite mixture model, as shown in Figure 25.2(a). The usual representation is asfollows:
p(x
i|zi=k,θ)=p( xi|θk) (25.13)
p(zi=k|π)=π k (25.14)
p(π|α)=D i r ( π|(α/K)1K) (25.15)
The form of p(θk|λ)is chosen to be conjugate to p(xi|θk). We can write p(xi|θk)asxi∼
F(θzi),w h e r eFis the observation distribution. Similarly, we can write θk∼H(λ),w h e r eH
is the prior.

880 Chapter25. Clustering
(a) (b)
Figure 25.2 Two different representations of a ﬁnite mixture model. Left: traditional representation.
Right: representation where parameters are samples from G, a discrete measure. The picture on the right
illustrates the case where K=4, and we sample 4 Gaussian means θkfrom a Gaussian prior H(.|λ). The
height of the spikes reﬂects the mixing weights πk. This weighted sum of delta functions is G. We then
generate two parameters, θ1andθ2,f r o mG, one per data point. Finally, we generate two data points,
x1andx2,f r o mN(θ1,σ2)andN(θ2,σ2). Source: Figure 2.9 of (Sudderth 2006) . Used with kind
permission of Erik Sudderth.
An equivalent representation for this model is shown in Figure 25.2(b). Here θiis the
parameter used to generate observation xi; these parameters are sampled from distribution G,
which has the form
G(θ)=K∑
k=1πkδθk(θ) (25.16)
whereπ∼Dir(α
K1), andθk∼H. Thus we see that Gis a ﬁnite mixture of delta functions,

G(θ)=K∑
centered on the cluster parameters θk. The probability that θiis equal to θkis exactly πk, the
prior probability for that cluster.
If we sample from this model, we will always (with probability one) get exactly Kclusters,
with data points scattered around the cluster centers. We would like a more ﬂexible model,
that can generate a variable number of clusters. Furthermore, the more data we generate, themore likely we should be to see a new cluster. The way to do this is to replace the discretedistribution Gwith arandom probability measure. Below we will show that the Dirichlet

process, denoted G∼DP(α,H ), is one way to do this.
Before we go into the details, we show some samples from this non-parametric model in
Figure 25.3. We see that it has the desired properties of generating a variable number of clusters,with more clusters as the amount of data increases. The resulting samples look much more likereal data than samples from a ﬁnite mixture model.
Of course, working with an “inﬁnite” model sounds scary. Fortunately, as we show below,

even though this model is potentially inﬁnite, we can perform inference using an amount ofcomputation that is not only tractable, but is often much less than that required to ﬁt a set

25.2. Dirichletprocessmixturemodels 881
(a) (b)
(c) (d)
(e) (f)
Figure 25.3 Some samples from a Dirichlet process mixture model of 2D Gaussians, with concentration
parameter α=1. From left to right, we show N=5 0,N= 500andN= 1000samples. Each row is a
different run. We also show the model parameters as ellipses, which are sampled from a vague NIW base
distribution. Based on Figure 2.25 of (Sudderth 2006). Figure generated by dpmSampleDemo , written by
Yee-Whye Teh.
of ﬁnite mixture models for different K. The intuitive reason is that we can get evidence that
certain values of Kare appropriate (have high posterior support) long before we have been able
to estimate the parameters, so we can focus our computational efforts on models of appropriate
complexity. Thus going to the inﬁnite limit can sometimes be faster. This is especially truewhen we have multiple model selection problems to solve.

882 Chapter25. Clustering
(a) (b) (c)
Figure 25.4 (a) A base measure Hon a 2d space Θ. (b) One possible partition into K=3regions,
where the shading of cell Tkis proportional to E[G(Tk)] =H(Tk). (c) A reﬁned partition into K=5
regions. Source: Figure 2.21 of (Sudderth 2006). Used with kind permission of Erik Sudderth.
25.2.2 The Dirichlet process
Recall from Chapter 15 that a Gaussian process is a distribution over functions of the form
f:X→R. It is deﬁned implicitly by the requirement that p(f(x1),...,f(xN))be jointly
Gaussian, for any set of points xi∈X. The parameters of this Gaussian can be computed using
a mean function μ()and covariance (kernel) function K(). We write f∼GP(μ(),K()). Fur-
thermore, the GP is consistently deﬁned, so that p(f(x1))can be derived from p(f(x1),f(x2)),
etc.
ADirichlet process is a distribution over probability measures G:Θ→R+,w h e r ew e
requireG(θ)≥0and∫
ΘG(θ)dθ=1. The DP is deﬁned implicitly by the requirement that
(G(T1),...,G(TK))has a joint Dirichlet distribution

etc.
Dir(αH(T1),...,αH (TK)) (25.17)
for any ﬁnite partition (T1,...,T K)ofΘ. If this is the case, we write G∼DP(α,H),w h e r e
αis called the concentration parameter andHis called the base measure .1
An example of a DP is shown in Figure 25.4, where the base measure is a 2d Gaussian. The
distribution over all the cells, p(G(T1),...,G(TK)), is Dirichlet, so the marginals in each cell
are beta distributed:
Beta(αH(Ti),α∑
j̸=iH(Tj)) (25.18)
The DP is consistently deﬁned in the sense that if T1andT2form a partition of ˜T1, then

G(T1)+G(T2)andG(˜T1)both follow the same beta distribution.
Recall that if π∼Dir(α), andz|π∼Cat(π), then we can integrate out πto get the
predictive distribution for the Dirichlet-multinoulli model:
z∼Cat(α1/α0,...,α K/α0) (25.19)
1. Unlike a GP, knowing something about G(Tk)does not tell us anything about G(Tk′), beyond the sum-to-one
constraint; we say that the DP is a neutral process . Other stochastic processes can be deﬁned that do not have this
property, but they are not so computationally convenient.

25.2. Dirichletprocessmixturemodels 883
β1
π1
π2
π3
π4
π5β2
β3
β4
β51−β 1
1−β 2
1−β 3
1−β 4
(a)0 10 20 3000.10.20.30.40.5α = 2
0 10 20 3000.10.20.30.4α = 2
0 10 20 3000.10.20.30.4α = 5
0 10 20 3000.050.10.150.2α = 5
(b)
Figure 25.5 Illustration of the stick breaking construction. (a) We have a unit length stick, which we
break at a random point β1; the length of the piece we keep is called π1; we then recursively break off
pieces of the remaining stick, to generate π2,π3,.... Source: Figure 2.22 of (Sudderth 2006). Used with
kind permission of Erik Sudderth. (b) Samples of πkfrom this process for α=2(top row) and α=5
(bottom row). Figure generated by stickBreakingDemo , written by Yee-Whye Teh.
whereα0=∑
kαk. In other words, p(z=k|α)=αk/α0. Also, the updated posterior for π
given one observation is given by
π|z∼Dir(α1+I(z=1 ),...,α K+I(z=K)) (25.20)
The DP generalizes this to arbitrary partitions. If G∼DP(α,H ), thenp(θ∈Ti)=H(Ti)and
the posterior is

whereα0=∑
p(G(T1),...,G(TK)|θ,α,H) = Dir(αH(T1)+I(θ∈T1),...,αH (TK)+I(θ∈TK))(25.21)
This holds for any set of partitions. Hence if we observe multiple samples θi∼G, the new
G|θ1,...,θN,α,H∼DP(
α+N,1
α+N(
αH+N∑
i=1δθi))
(25.22)
Thus we see that the DP effectively deﬁnes a conjugate prior for arbitrary measurable spaces.
The concentration parameter αis like the effective sample size of the base measure H.
25.2.2.1 Stick breaking construction of the DP

α+N,1
α+N(
αH+N∑
i=1δθi))
(25.22)
Our discussion so far has been very abstract. We now give a constructive deﬁnition for the DP,known as the stick-breaking construction.
Letπ={π
k}∞
k=1be an inﬁnite sequence of mixture weights derived from the following
process:
βk∼Beta(1,α) (25.23)
πk=βkk−1∏
l=1(1−βl)=βk(1−k−1∑
l=1πl) (25.24)

884 Chapter25. Clustering
This is often denoted by
π∼GEM(α) (25.25)
where GEM stands for Griffiths, Engen and McCloskey (this term is due to (Ewens 1990)). Some
samples from this process are shown in Figure 25.5. One can show that this process processwill terminate with probability 1, although the number of elements it generates increases withα. Furthermore, the size of the π
kcomponents decreases on average.
Now deﬁne
G(θ)=∞∑
k=1πkδθk(θ) (25.26)
whereπ∼GEM(α)andθk∼H. Then one can show that G∼DP(α,H ).
As a consequence of this construction, we see that samples from a DP are discrete with
probability one. In other words, if you keep sampling it, you will get more and more repetitionsof previously generated values. So if we sample
θi∼G, we will see repeated values; let us
number the unique values θ1,θ2, etc. Data sampled from θiwill therefore cluster around the
θk. This is evident in Figure 25.3, where most data comes from the Gaussians with large πk

values, represented by ellipses with thick borders. This is our ﬁrst indication that the DP mightbe useful for clustering.
25.2.2.2 The Chinese restaurant process (CRP)
Working with inﬁnite dimensional sticks is problematic. However, we can exploit the clusteringproperty to draw samples form a GP, as we now show.
The key result is this: If
θi∼GareNobservations from G∼DP(α,H ), taking on K
distinct values θk, then the predictive distribution of the next observation is given by
p(θN+1=θ|θ1:N,α,H)=1
α+N(

α+N(
αH(θ)+K∑
k=1Nkδθk(θ))
(25.27)
whereNkis the number of previous observations equal to θk. This is called the Polya urn or
Blackwell-MacQueen sampling scheme. This provides a constructive way to sample from a DP.
It is much more convenient to work with discrete variables ziwhich specify which value of
θkto use. That is, we deﬁne θi=θzi. Based on the above expression, we have
p(zN+1=z|z1:N,α)=1
α+N(
αI(z=k∗)+K∑
k=1NkI(z=k))
(25.28)

α+N(
αH(θ)+K∑
(25.27)
α+N(
(25.28)
wherek∗represents a new cluster index that has not yet been used. This is called the Chinese
restaurant process orCRP, based on the seemingly inﬁnite supply of tables at certain Chinese
restaurants. The analogy is as follows: The tables are like clusters, and the customers are likeobservations. When a person enters the restaurant, he may choose to join an existing table withprobability proportional to the number of people already sitting at this table (the N
k); otherwise,

(25.28)
with a probability that diminishes as more people enter the room (due to the 1/(α+N)term),

25.2. Dirichletprocessmixturemodels 885
(a) (b)
Figure 25.6 Two views of a DP mixture model. Left: inﬁnite number of clusters parameters, θk, and
π∼GEM(α). Right:Gis drawn from a DP. Compare to Figure 25.2. Source: Figure 2.24 of (Sudderth
2006). Used with kind permission of Erik Sudderth.
he may choose to sit at a new table k∗. The result is a distribution over partitions of the
integers, which is like a distribution of customers to tables.
The fact that currently occupied tables are more likely to get new customers is sometimes
called the rich get richer phenomenon. Indeed, one can derive an expression for the distri-
bution of cluster sizes induced by this prior process; it is basically a power law. The number
of occupied tables Kalmost surely approaches αlog(N)asN→∞, showing that the model
complexity will indeed grow logarithmically with dataset size. More ﬂexible priors over clustersizes can also be deﬁned, such as the two-parameter Pitman-Yor process.

25.2.3 Applying Dirichlet processes to mixture modeling
The DP is not particularly useful as a model for data directly, since data vectors rarely repeatexactly. However, it is useful as a prior for the parameters of a stochastic data generatingmechanism, such as a mixture model. To create such a model, we follow exactly the same setupas Section 11.2, but we deﬁne G∼DP(α,H ). Equivalently, we can write the model as follows:
π∼GEM(α) (25.29)
z
i∼π (25.30)
θk∼H(λ) (25.31)
xi∼F(θzi) (25.32)
This is illustrated in Figure 25.6. We see that Gis now a random draw of an unbounded number

z
of parameters θkfrom the base distribution H, each with weight πk. Each data point xiis
generated by sampling its own “private” parameter θifromG. As we get more and more data,
it becomes increasingly likely that θiwill be equal to one of the θk’s we have seen before, and
thusxiwill be generated close to an existing datapoint.

886 Chapter25. Clustering
25.2.4 Fitting a DP mixture model
The simplest way to ﬁt a DPMM is to modify the collapsed Gibbs sampler of Section 24.2.4.
From Equation 24.23 we have
p(zi=k|z−i,x,α,λ)∝p(zi=k|z−i,α)p(xi|x−i,zi=k,z−i,λ) (25.33)
By exchangeability, we can assume that ziis the last customer to enter the restaurant. Hence
the ﬁrst term is given by
p(zi|z−i,α)=1
α+N−1(
αI(zi=k∗)+K∑
k=1Nk,−iI(zi=k))
(25.34)
whereKis the number of clusters used by z−i, andk∗is a new cluster. Another way to write
this is as follows:
p(zi=k|z−i,α)={
Nk,−i
α+N−1ifkhas been seen before
α
α+N−1ifkis a new cluster(25.35)
Interestingly, this is equivalent to Equation 24.26, which has the form p(zi=k|z−i,α)=
Nk,−i+α/K
α+N−1,i nt h eK→∞limit (Rasmussen 2000; Neal 2000).
To compute the second term, p(xi|x−i,zi=k,z−i,λ), let us partition the data x−iinto
clusters based on z−i.L e tx−i,c={xj:zj=c,j̸=i}be the data assigned to cluster c.I f
zi=k, thenxiis conditionally independent of all the data points except those assigned to

Nk,−i
α
Nk,−i+α/K
clusterk. Hence we have
p(xi|x−i,z−i,zi=k,λ)=p( xi|x−i,k,λ)=p(xi,x−i,k|λ)
p(x−i,k|λ)(25.36)
where
p(xi,x−i,k|λ)=∫
p(xi|θk)⎡
⎣∏
j̸=i:zj=kp(xj|θk)⎤⎦H(θ
k|λ)dθk (25.37)
is the marginal likelihood of all the data assigned to cluster k, including i, andp(x−i,k|λ)is an
analogous expression excluding i. Thus we see that the term p(xi|x−i,z−i,zi=k,λ)is the
posterior preditive distribution for cluster kevaluated at xi.
Ifzi=k∗, corresponding to a new cluster, we have
p(xi|x−i,z−i,zi=k∗,λ)=p(xi|λ)=∫

where
p(xi|θk)⎡
⎣∏
p(xi|θ)H(θ|λ)dθ (25.38)
which is just the prior predictive distribution for a new cluster evaluated at xi.
See Algorithm 1 for the pseudocode. (This is called “Algorithm 3” in (Neal 2000).) This is very
similar to collapsed Gibbs for ﬁnite mixtures except that we have to consider the case zi=k∗.
An example of this procedure in action is shown in Figure 25.7. The sample clusterings, and
the induced posterior over K, seems reasonable. The method tends to rapidly discover a good

clustering. By contrast, Gibbs sampling (and EM) for a ﬁnite mixture model often gets stuck in

25.3. Affinitypropagation 887
Algorithm 25.1: Collapsed Gibbs sampler for DP mixtures
4 Compute pk(xi)=p(xi|x−i(k));
5 SetNk,−i=d i m (x−i(k));
6 Compute p(zi=k|z−i,D)=Nk,−i
α+N−1;
7Compute p∗(xi)=p(xi|λ);
8Compute p(zi=∗|z−i,D)=α
α+N−1;
9Normalize p(zi|·);
10Samplezi∼p(zi|·);
11Addxi’s sufficient statistics to new cluster zi;
12If any cluster is empty, remove it and decrease K;
poor local optima (not shown). This is because the DPMM is able to create extra redundant
clusters early on, and to use them to escape local optima. Figure 25.8 shows that most of thetime, the DPMM converges more rapidly than a ﬁnite mixture model.
A variety of other ﬁtting methods have been proposed. (Daume 2007a) shows how one can use

α+N−1;
A star search and beam search to quickly ﬁnd an approximate MAP estimate. (Mansinghka et al.2007) discusses how to ﬁt a DPMM online using particle ﬁltering, which is a like a stochasticversion of beam search. This can be more efficient than Gibbs sampling, particularly for largedatasets. (Kurihara et al. 2006) develops a variational approximation that is even faster (see also(Zobay 2009)). Extensions to the case of non-conjugate priors are discussed in (Neal 2000).

Another important issue is how to set the hyper-parameters. For the DP, the value of α
does not have much impact on predictive accuracy, but it does affect the number of clusters.One approach is to put a Ga(a,b) prior forα, and then to from its posterior, p(α|K,N,a,b ),
using auxiliary variable methods (Escobar and West 1995). Alternatively, one can use empiricalBayes (McAuliffe et al. 2006). Similarly, for the base distribution, we can either sample thehyper-parameters λ(Rasmussen 2000) or use empirical Bayes (McAuliffe et al. 2006).

25.3 Affinity propagation
Mixture models, whether ﬁnite or inﬁnite, require access to the raw N×Ddata matrix, and
need to specify a generative model of the data. An alternative approach takes as input an N×N
similarity matrix, and then tries to identify examplars, which will act as cluster centers. TheK-medoids or K-centers algorithm (Section 14.4.2) is one approach, but it can suffer from local
minima. Here we describe an alternative approach called affinity propagation (Frey and Dueck
2007) that works substantially better in practice.

The idea is that each data point must choose another data point as its exemplar or centroid;
some data points will choose themselves as centroids, and this will automatically determine thenumber of clusters. More precisely, let c
i∈{1,...,N}represent the centroid for datapoint i.

888 Chapter25. Clustering
−6 −4 −2 0 2 4−6−4−2024iter# 50
(a)−6 −4 −2 0 2 4−6−4−2024iter# 100
(b)
−6 −4 −2 0 2 4−6−4−2024iter# 200
(c)1 2 3 4 5 600.10.20.30.40.50.60.7
(d)
Figure 25.7 100 data points in 2d are clustered using a DP mixture ﬁt with collapsed Gibbs sampling.
We show samples from the posterior after 50,100, 200 samples. We also show the posterior over K, based
on 200 samples, discarding the ﬁrst 50 as burnin. Figure generated by dpmGauss2dDemo , written by Yee
Whye Teh.
The goal is to maximize the following function
S(c)=N∑
i=1s(i,ci)+N∑
k=1δk(c) (25.39)
The ﬁrst term measures the similarity of each point to its centroid. The second term is a penalty
term that is−∞if some data point ihas chosen kas its exemplar (i.e., ci=k), butkhas not
chosen itself as an exemplar (i.e., we do not have ck=k). More formally,
δk(c)={−∞ifck̸=kbut∃i:ci=k
0otherwise(25.40)
The objective function can be represented as a factor graph. We can either use Nnodes,

25.3. Affinitypropagation 889
  
Dirichlet Process Mixture
Finite Mixture
  
(b)
Figure 25.8 Comparison of collapsed Gibbs samplers for a DP mixture (dark blue) and a ﬁnite mixture
(light red) with K=4applied to N= 300data points (shown in Figure 25.7). Left: logprob vs iteration
for 20 different starting values. Right: median (thick line) and quantiles (dashed lines) over 100 different
starting values. Source: Figure 2.27 of (Sudderth 2006). Used with kind permission of Erik Sudderth.
c1 c2
1
 2
s(1,
) s(2,
)A
cN c32
 N
 3
s(3,
) s(N,
)…
k
…
s(i,
)… …
ci
Figure 25.9 Factor graphs for affinity propagation. Circles are variables, squares are factors. Each cinode
hasNpossible states. From Figure S2 of (Frey and Dueck 2007). Used with kind permission of Brendan
Frey.

c1 c2
1
 2
s(1,
) s(2,
)A
cN c32
 N
 3
s(3,
) s(N,
)…
k
…
s(i,
)… …
ci
Frey.
each with Npossible values, as shown in Figure 25.9, or we can use N2binary nodes (see
(Givoni and Frey 2009) for the details). We will assume the former representation.
We can ﬁnd a strong local maximum of the objective by using max-product loopy belief
propagation (Section 22.2). Referring to the model in Figure 25.9, each variable nodes cisends
a message to each factor node δk. It turns out that this vector of Nnumbers can be reduced

Frey.
to a scalar message, denote ri→k, known as the responsibility. This is a measure of how much
ithinkskwould make a good exemplar, compared to all the other exemplars ihas looked at.
In addition, each factor node δksends a message to each variable node ci. Again this can be
reduced to a scalar message, ai←k, known as the availability. This is a measure of how strongly
kbelieves it should an exemplar for i, based on all the other data points khas looked at.
As usual with loopy BP, the method might oscillate, and convergence is not guaranteed.

/g23/g29/g20/g27/g17/g29/g23/g26/g25/g1/g2/g16 /g23/g29/g20/g27/g17/g29/g23/g26/g25/g1/g2/g15 /g23/g29/g20/g27/g17/g29/g23/g26/g25/g1/g2/g14/g23/g29/g20/g27/g17/g29/g23/g26/g25/g1/g2/g13 /g23/g29/g20/g27/g17/g29/g23/g26/g25/g1/g2/g12 /g23/g29/g20/g27/g17/g29/g23/g26/g25/g1/g2/g11
/g18/g26/g25/g30/g20/g27/g22/g20/g25/g18/g20
/g44/g45/g44/g8/g36/g52/g36/g43/g46/g42/g32/g47/g1 /g36/g52/g36/g43/g46/g42/g32/g47
Figure 25.10 Example of affinity propagation. Each point is colored coded by how much it wants to be
an exemplar (red is the most, green is the least). This can be computed by summing up all the incoming
availability messages and the self-similarity term. The darkness of the i→karrow reﬂects how much
pointiwants to belong to exemplar k. From Figure 1 of (Frey and Dueck 2007). Used with kind permission
of Brendan Frey.
However, by using damping, the method is very reliable in practice. If the graph is densely
connected, message passing takes O(N2)time, but with sparse similarity matrices, it only takes

O(E)time, where Eis the number of edges or non-zero entries in S.
The number of clusters can be controlled by scaling the diagonal terms S(i,i), which reﬂect
how much each data point wants to be an exemplar. Figure 25.10 gives a simple example of some
2d data, where the negative Euclidean distance was used to measured similarity. The S(i,i)
values were set to be the median of all the pairwise similarities. The result is 3 clusters. Many

other results are reported in (Frey and Dueck 2007), who show that the method signiﬁcantly
outperforms K-medoids.
25.4 Spectral clustering
An alternative view of clustering is in terms of graph cuts . The idea is we create a weighted
undirected graph Wfrom the similarity matrix S, typically by using the nearest neighbors of
each point; this ensures the graph is sparse, which speeds computation. If we want to ﬁnd a
partition into Kclusters, say A1,...,A K, one natural criterion is to minimize
cut(A1,...,A K)≜1
2K∑
k=1W(Ak,Ak) (25.41)

25.4. Spectralclustering 891
whereAk=V\Akis the complement of Ak, andW(A,B)≜∑
i∈A,j∈Bwij.F o rK=2this
problem is easy to solve. Unfortunately the optimal solution often just partitions off a single
data point from the rest. To ensure the sets are reasonably large, we can deﬁne the normalized
cutto be
Ncut(A1,...,A K)≜1
2K∑
k=1cut(Ak,Ak)
vol(Ak)(25.42)
where vol (A)≜∑
i∈Adi, anddi=∑N
j=1wijis the weighted degree of node i. This splits
the graph into Kclusters such that nodes within each cluster are similar to each other, but are
different to nodes in other clusters.
We can formulate the Ncut problem in terms of searching for binary vectors ci∈{0,1}N,
wherecik=1if pointibelongs to cluster k, that minimize the objective. Unfortunately this
is NP-hard (Wagner and Wagner 1993). Affinity propagation is one way to solve the problem.
Another is to relax the constraints that cibe binary, and allow them to be real-valued. The

result turns into an eigenvector problem known as spectral clustering (see e.g., (Shi and Malik
2000)). In general, the technique of performing eigenalysis of graphs is called spectral graph
theory(Chung 1997).
Going into the details would take us too far aﬁeld, but below we give a very brief summary,
based on (von Luxburg 2007), since we will encounter some of these ideas later on.
25.4.1 Graph Laplacian
LetWbe a symmetric weight matrix for a graph, where wij=wji≥0.L e tD=d i a g (di)be a

diaogonal matrix containing the weighted degree of each node. We deﬁne the graph Laplacian
L≜D−W (25.43)
This matrix has various important properties. Because each row sums to zero, we have
that1is an eigenvector with eigenvalue 0. Furthermore, the matrix is symmetric and positive
semi-deﬁnite. To see this, note that
fTLf=fTDf−fTWf=∑
idif2
i−∑
i,jfifjwij (25.44)
=1
2⎛
⎝∑
idif2
i−2∑
i,jfifjwij+∑
jdjf2
j⎞⎠=1
2∑
i,jwij(fi−fj)2(25.45)
HencefTLf≥0for allf∈RN. Consequently we see that LhasNnon-negative, real-valued

idif2
i−∑
=1
2⎛
⎝∑
idif2
i−2∑
jdjf2
j⎞⎠=1
2∑
eigenvalues, 0≤λ1≤λ2≤...≤λN.
To get some intuition as to why Lmight be useful for graph-based clustering, we note the
following result.
Theorem 25.4.1. Thesetofeigenvectorsof Lwitheigenvalue0isspannedbytheindicatorvectors
1A1,...,1AK,whereAkaretheKconnectedcomponentsofthegraph.

892 Chapter25. Clustering
Proof.Let us start with the case K=1.I ffis an eigenvector with eigenvalue 0, then
0=∑
ijwij(fi−fj)2. If two nodes are connected, so wij>0, we must have that fi=fj.
Hencefis constant for all vertices which are connected by a path in the graph. Now suppose
K>1. In this case, Lwill be block diagonal. A similar argument to the above shows that we
will have Kindicator functions, which “select out” the connected components.
This suggests the following algorithm. Compute the ﬁrst Keigenvectors ukofL.L e t
U=[u1,...,uK]be anN×Kmatrix with the eigenvectors in its columns. Let yi∈RKbe
thei’th row of U. Since these yiwill be piecewise constant, we can apply K-means clustering
to them to recover the connected components. Now assign point ito cluster kiff rowiofY
was assigned to cluster k.
In reality, we do not expect a graph derived from a real similarity matrix to have isolated
connected components — that would be too easy. But it is reasonable to suppose the graph is

a small “perturbation” from such an ideal. In this case, one can use results from perturbationtheory to show that the eigenvectors of the perturbed Laplacian will be close to these idealindicator functions (Ng et al. 2001).
Note that this approach is related to kernel PCA (Section 14.4.4). In particular, KPCA uses the
largest eigenvectors of W; these are equivalent to the smallest eigenvectors of I−W. This is
similar to the above method, which computes the smallest eigenvectors of L=D−W. See

(Bengio et al. 2004) for details. In practice, spectral clustering gives much better results thanKPCA.
25.4.2 Normalized graph Laplacian
In practice, it is important to normalize the graph Laplacian, to account for the fact that somenodes are more highly connected than others. There are two comon ways to do this. Onemethod, used in e.g., (Shi and Malik 2000; Meila 2001), creates a stochastic matrix where eachrow sums to one:
L
rw≜D−1L=I−D−1W (25.46)

L
The eigenvalues and eigenvectors of LandLrware closely related to each other (see (von
Luxburg 2007) for details). Furthemore, one can show that for Lrw, the eigenspace of 0 is
again spanned by the indicator vectors 1Ak. This suggests the following algorithm: ﬁnd the
smallestKeigenvectors of Lrw,c r e a t eU, cluster the rows of Uusing K-means, then infer the
partitioning of the original points (Shi and Malik 2000). (Note that the eigenvectors/ values ofL
rware equivalent to the generalized eigenvectors/ values of L, which solve Lu=λDU.)

Another method, used in e.g., (Ng et al. 2001), creates a symmetric matrix
Lsym≜D−1
2LD−1
2=I−D−1
2WD−1
2 (25.47)
This time the eigenspace of 0 is spanned by D1
21Ak. This suggest the following algorithm: ﬁnd
the smallest Keigenvectors of Lsym,c r e a t eU, normalize each row to unit norm by creating
tij=uij/√
(∑
ku2
ik), cluster the rows of Tusing K-means, then infer the partitioning of the
original points (Ng et al. 2001).
There is an interesting connection between Ncuts and random walks on a graph (Meila

Lsym≜D−1
2LD−1
2=I−D−1
2WD−1
2 (25.47)
tij=uij/√
(∑
ku2
2001). First note that P=D−1W=I−Lrwis a stochastic matrix, where pij=wij/di

25.5. Hierarchicalclustering 893
−6 −4 −2 0 2 4 6−5−4−3−2−1012345k−means clustering
xy
(a)−6 −4 −2 0 2 4 6−5−4−3−2−1012345spectral clustering
xy
(b)
Figure 25.11 Clustering data consisting of 2 spirals. (a) K-means. (b) Spectral clustering. Figure generated
byspectralClusteringDemo , written by Wei-Lwun Lu.
can be interpreted as the probability of going from itoj. If the graph is connected and
non-bipartite, it possesses a unique stationary distribution π=(π1,...,πN),w h e r eπi=
di/vol(V). Furthermore, one can show that
Ncut(A,A)=p(A|A)+p(A|A) (25.48)
This means that we are looking for a cut such that a random walk rarely makes transitions from
AtoAor vice versa.
25.4.3 Example

Figure 25.11 illustrates the method in action. In Figure 25.11(a), we see that K-means does a poorjob of clustering, since it implicitly assumes each cluster corresponds to a spherical Gaussian.Next we try spectral clustering. We deﬁne a similarity matrix using the Gaussian kernel. Wecompute the ﬁrst two eigenvectors of the Laplacian. From this we can infer the clustering inFigure 25.11(b).
Since the method is based on ﬁnding the smallest Keigenvectors of a sparse matrix, it takes
O(N
3)time. However, a variety of methods can be used to scale it up for large datasets (see

O(N
e.g., (Yan et al. 2009)).
25.5 Hierarchical clustering
Mixture models, whether ﬁnite or inﬁnite, produce a “ﬂat” clustering. Often we want to learn ahierarchical clustering, where clusters can be nested inside each other.
There are two main approaches to hierarchical clustering: bottom-up or agglomerative clus-
tering, and top-down or divisive clustering. Both methods take as input a dissimilarity matrix
between the objects. In the bottom-up approach, the most similar groups are merged at each

894 Chapter25. Clustering
0 1 2 3 4 5 6 7 800.511.522.533.544.55
12
3
45
(a)4 5 1 3 211.522.5
(b)
Figure 25.12 (a) An example of single link clustering using city block distance. Pairs (1,3) and (4,5) are
both distance 1 apart, so get merged ﬁrst. (b) The resulting dendrogram. Based on Figure 7.5 of (Alpaydin
2004). Figure generated by agglomDemo .
(a)0 10 20−3−2−10
0 10 20−2−10
0 10 20−2.5−2−1.5−1−0.5
0 10 20−3−2−101
0 10 20−202
0 10 20−202
0 10 20−4−20
0 10 20−2024
0 10 20−4−202
0 10 20−1012
0 10 200.511.522.5
0 10 200.511.52
0 10 20012
0 10 20−2−10Hierarchical Clustering of Profiles
(b)
Figure 25.13 Hierarchical clustering applied to the yeast gene expression data. (a) The rows are permuted
according to a hierarchical clustering scheme (average link agglomerative clustering), in order to bring
similar rows close together. (b) 16 clusters induced by cutting the average linkage tree at a certain height.
Figure generated by hclustYeastDemo .

0 10 20012
(b)
step. In the top-down approach, groups are split using various different criteria. We give the
Note that agglomerative and divisive clustering are both just heuristics, which do not optimize
any well-deﬁned objective function. Thus it is hard to assess the quality of the clustering they
produce in any formal sense. Furthermore, they will always produce a clustering of the input
data, even if the data has no structure at all (e.g., it is random noise). Later in this section we

will discuss a probabilistic version of hierarchical clustering that solves both these problems.

25.5. Hierarchicalclustering 895
Algorithm 25.2: Agglomerative clustering
1initialize clusters as singletons: fori←1tondoCi←{i};
2initialize set of clusters available for merging: S←{1,...,n};
3repeat
4Pick 2 most similar clusters to merge: (j,k)←argmin j,k∈Sdj,k;
5Create new cluster Cℓ←Cj∪Ck;
6Markjandkas unavailable: S←S\{j,k};
7ifCℓ̸={1,...,n}then
8 Markℓas available, S←S∪{ℓ};
9foreachi∈Sdo
10 Update dissimilarity matrix d(i,ℓ);
11untilnomoreclustersareavailableformerging ;
(a)
 (b)
 (c)
Figure 25.14 Illustration of (a) Single linkage. (b) Complete linkage. (c) Average linkage.
25.5.1 Agglomerative clustering
Agglomerative clustering starts with Ngroups, each initially containing one object, and then at
each step it merges the two most similar groups until there is a single group, containing all the
data. See Algorithm 11 for the pseudocode. Since picking the two most similar clusters to merge
takesO(N2)time, and there are O(N)steps in the algorithm, the total running time is O(N3).

(c)
However, by using a priority queue, this can be reduced to O(N2logN) (see e.g., (Manning
et al. 2008, ch. 17) for details). For large N, a common heuristic is to ﬁrst run K-means, which
takesO(KND)time, and then apply hierarchical clustering to the estimated cluster centers.
The merging process can be represented by a binary tree , called a dendrogram , as shown
in Figure 25.12(b). The initial groups (objects) are at the leaves (at the bottom of the ﬁgure),

and every time two groups are merged, we join them in the tree. The height of the branches
represents the dissimilarity between the groups that are being joined. The root of the tree (which
is at the top) represents a group containing all the data. If we cut the tree at any given height,
we induce a clustering of a given size. For example, if we cut the tree in Figure 25.12(b) at
height 2, we get the clustering {{{4,5},{1,3}},{2}}. We discuss the issue of how to choose
the height/ number of clusters below.

A more complex example is shown in Figure 25.13(a), where we show some gene expression
data. If we cut the tree in Figure 25.13(a) at a certain height, we get the 16 clusters shown in
Figure 25.13(b).
There are actually three variants of agglomerative clustering, depending on how we deﬁne
the dissimilarity between groups of objects. These can give quite different results, as shown in

896 Chapter25. Clustering
0.050.10.150.20.250.3single link
(a)
0.20.40.60.811.21.41.61.82complete link
(b)
0.20.40.60.811.21.41.61.8average link
(c)
Figure 25.15 Hierarchical clustering of yeast gene expression data. (a) Single linkage. (b) Complete linkage.
(c) Average linkage. Figure generated by hclustYeastDemo .

25.5. Hierarchicalclustering 897
Figure 25.15. We give the details below.
25.5.1.1 Single link
Insingle link clustering, also called nearest neighbor clustering, the distance between two
groupsGandHis deﬁned as the distance between the two closest members of each group:
dSL(G,H)= m i n
i∈G,i′∈Hdi,i′ (25.49)
See Figure 25.14(a).
The tree built using single link clustering is a minimum spanning tree of the data, which
is a tree that connects all the objects in a way that minimizes the sum of the edge weights
(distances). To see this, note that when we merge two clusters, we connect together the twoclosest members of the clusters; this adds an edge between the corresponding nodes, and thisis guaranteed to be the “lightest weight” edge joining these two clusters. And once two clustershave been merged, they will never be considered again, so we cannot create cycles. As aconsequence of this, we can actually implement single link clustering in O(N
2)time, whereas
the other variants take O(N3)time.

25.5.1.2 Complete link
Incomplete link clustering, also called furthest neighbor clustering, the distance between
two groups is deﬁned as the distance between the two most distant pairs:
dCL(G,H)= m a x
i∈G,i′∈Hdi,i′ (25.50)
See Figure 25.14(b).
Single linkage only requires that a single pair of objects be close for the two groups to
be considered close together, regardless of the similarity of the other members of the group.Thus clusters can be formed that violate the compactness property, which says that all the

observations within a group should be similar to each other. In particular if we deﬁne thediameter of a group as the largest dissimilarity of its members, d
G=m a x i∈G,i′∈Gdi,i′, then
we can see that single linkage can produce clusters with large diameters. Complete linkagerepresents the opposite extreme: two groups are considered close only if all of the observationsin their union are relatively similar. This will tend to produce clusterings with small diameter,i.e., compact clusters.
25.5.1.3 Average link

In practice, the preferred method is average link clustering, which measures the average
distance between all pairs:
d
avg(G,H)=1
nGnH∑
i∈G∑
i′∈Hdi,i′ (25.51)
wherenGandnHare the number of elements in groups GandH. See Figure 25.14(c).
Average link clustering represents a compromise between single and complete link clustering.
It tends to produce relatively compact clusters that are relatively far apart. However, since it

898 Chapter25. Clustering
involves averaging of the di,i′’s, any change to the measurement scale can change the result. In
contrast, single linkage and complete linkage are invariant to monotonic transformations of di,i′,
since they leave the relative ordering the same.
25.5.2 Divisive clustering
Divisive clustering starts with all the data in a single cluster, and then recursively divides each
cluster into two daughter clusters, in a top-down fashion. Since there are 2N−1−1ways to split
a group of Nitems into 2 groups, it is hard to compute the optimal split, so various heuristics
are used. One approach is pick the cluster with the largest diameter, and split it in two using theK-means or K-medoids algorithm with K=2. This is called the bisecting K-means algorithm
(Steinbach et al. 2000). We can repeat this until we have any desired number of clusters. Thiscan be used as an alternative to regular K-means, but it also induces a hierarchical clustering.

Another method is to build a minimum spanning tree from the dissimilarity graph, and then
to make new clusters by breaking the link corresponding to the largest dissimilarity. (Thisactually gives the same results as single link agglomerative clustering.)
Yet another method, called dissimilarity analysis (Macnaughton-Smith et al. 1964), is as
follows. We start with a single cluster containing all the data, G={1,...,N}. We then
measure the average dissimilarity of i∈Gto all the other i
′∈G:
dG
i=1
nG∑
i′∈Gdi,i′ (25.52)

′∈G:
dG
i=1
nG∑
We remove the most dissimilar object and put it in its own cluster H:
i∗=a r gm a x
i∈GdGi,G=G\{i∗},H={i∗} (25.53)
We now continue to move objects from GtoHuntil some stopping criterion is met. Speciﬁcally,
we pick a point i∗to move that maximizes the average dissimilarity to each i′∈Gbut minimizes
the average dissimilarity to each i′∈H:
dHi=1
nH∑
i′∈Hdi,i′,i∗=a r gm a x
i∈GdGi−dHi(25.54)
We continue to do this until dG
i−dHiis negative. The ﬁnal result is that we have split Ginto
two daughter clusters, GandH. We can then recursively call the algorithm on Gand/orH,o r

i∗=a r gm a x
dHi=1
nH∑
on any other node in the tree. For example, we might choose to split the node Gwhose average
dissimilarity is highest, or whose maximum dissimilarity (i.e., diameter) is highest. We continue
the process until the average dissimilarity within each cluster is below some threshold, and/orall clusters are singletons.
Divisive clustering is less popular than agglomerative clustering, but it has two advantages.
First, it can be faster, since if we only split for a constant number of levels, it takes just O(N)

time. Second, the splitting decisions are made in the context of seeing all the data, whereasbottom-up methods make myopic merge decisions.

25.5. Hierarchicalclustering 899
25.5.3 Choosing the number of clusters
It is difficult to choose the “right” number of clusters, since a hierarchical clustering algorithm
will always create a hierarchy, even if the data is completely random. But, as with choosing K
for K-means, there is the hope that there will be a visible “gap” in the lengths of the links in thedendrogram (represent the dissimilarity between merged groups) between natural clusters andunnatural clusters. Of course, on real data, this gap might be hard to detect. In Section 25.5.4,we will present a Bayesian approach to hierarchical clustering that nicely solves this problem.
25.5.4 Bayesian hierarchical clustering
There are several ways to make probabilistic models which produce results similar to hierarchicalclustering, e.g., (Williams 2000; Neal 2003b; Castro et al. 2004; Lau and Green 2006). Here wepresent one particular approach called Bayesian hierarchical clustering (Heller and Ghahra-

mani 2005). Algorithmically it is very similar to standard bottom-up agglomerative clustering,and takes comparable time, whereas several of the other techniques referenced above are muchslower. However, it uses Bayesian hypothesis tests to decide which clusters to merge (if any),rather than computing the similarity between groups of points in some ad-hoc way. Thesehypothesis tests are closely related to the calculations required to do inference in a Dirichletprocess mixture model, as we will see. Furthermore, the input to the model is a data matrix,not a dissimilarity matrix.
25.5.4.1 The algorithm
LetD={x
1,...,xN}represent all the data, and let Dibe the set of datapoints at the leaves

LetD={x
of the substree Ti. At each step, we compare two trees TiandTjto see if they should be
merged into a new tree. Deﬁne Dijas their merged data, and let Mij=1if they should be
merged, and Mij=0otherwise.
The probability of a merge is given by
rij≜p(Dij|Mij=1 )p(Mij=1 )
p(Dij|Tij)(25.55)
p(Dij|Tij)=p(Dij|Mij=1 )p(Mij=1 )+p(Dij|Mij=0 )p(Mij=0 ) (25.56)
Herep(Mij=1 )is the prior probability of a merge, which can be computed using a bottom-up
algorithm described below. We now turn to the likelihood terms. If Mij=1, the data inDijis
assumed to come from the same model, and hence
p(Dij|Mij=1 )=∫⎡
⎣∏
xn∈Dijp(xn|θ)⎤⎦p(θ|λ)dθ (25.57)
IfM
ij=0, the data inDijis assumed to have been generated by each tree independently, so
p(Dij|Mij=0 )=p(Di|Ti)p(Dj|Tj) (25.58)
These two terms will have already been computed by the bottom-up process. Consequently

⎣∏
IfM
we have all the quantities we need to decide which trees to merge. See Algorithm 9 for thepseudocode, assuming p(M
ij)is uniform. When ﬁnished, we can cut the tree at points where
rij<0.5.

900 Chapter25. Clustering
Algorithm 25.3: Bayesian hierarchical clustering
1InitializeDi={xi},i=1:N;
2Compute p(Di|Ti),i=1:N;
3repeat
4foreachpairofclusters i,jdo
5 Compute p(Dij|Tij)
6Find the pairDiandDjwith highest merge probability rij;
7MergeDk:=Di∪Dj;
8DeleteDi,Dj;
9untilallclustersmerged ;
25.5.4.2 The connection with Dirichlet process mixture models
In this section, we will establish the connection between BHC and DPMMs. This will in turn
give us an algorithm to compute the prior probabilities p(Mij=1 ).
Note that the marginal likelihood of a DPMM, summing over all 2N−1partitions, is given by
p(Dk)=∑
v∈Vp(v)p(Dv) (25.59)
p(v)=αmv∏mv
l=1Γ(nv
l)
Γ(nk+α)
Γ(α)(25.60)
p(Dv)=mv∏
l=1p(Dv
l) (25.61)
whereVis the set of all possible partitions of Dk,p(v)is the probability of partition v,mvis
the number of clusters in partition v,nvlis the number of points in cluster lof partition v,Dv
l
are the points in cluster lof partition v, andnkare the number of points in Dk.

p(Dk)=∑
l=1Γ(nv
l)
Γ(nk+α)
p(Dv)=mv∏
l=1p(Dv
l) (25.61)
l
One can show (Heller and Ghahramani 2005) that p(Dk|Tk)computed by the BHC algorithm
is similar to p(Dk)given above, except for the fact that it only sums over partitions which are
consistent with tree Tk. (The number of tree-consistent partitions is exponential in the number
of data points for balanced binary trees, but this is obviously a subset of all possible partitions.)

l
In this way, we can use the BHC algorithm to compute a lower bound on the marginal likelihoodof the data from a DPMM. Furthermore, we can interpret the algorithm as greedily searchingthrough the exponentially large space of tree-consistent partitions to ﬁnd the best ones of agiven size at each step.
We are now in a position to compute π
k=p(Mk=1 ), for each node kwith children iand
j. This is equal to the probability of cluster Dkcoming from the DPMM, relative to all other

partitions ofDkconsistent with the current tree. This can be computed as follows: initialize
di=αandπi=1for each leaf i; then as we build the tree, for each internal node k, compute
dk=αΓ(nk)+didj, andπk=αΓ(nk)
dk,w h e r ei andjarek’s left and right children.

25.6. Clusteringdatapointsandfeatures 901
Data Set Single Linkage Complete Linkage Average Linkage BHC
Synthetic 0.599±0.033 0.634±0.024 0. 668±0.0400.828±0.025
Newsgroups 0.275±0.001 0.315±0.008 0. 282±0.0020.465±0.016
Spambase 0.598±0.017 0.699±0.017 0. 668±0.0190.728±0.029
Digits 0.224±0.004 0.299±0.006 0. 342±0.0050.393±0.015
Fglass 0.478±0.009 0.476±0.009 0.491±0.009 0.467±0.011
Table 25.1 Purity scores for various hierarchical clustering schemes applied to various data sets. The
synthetic data has N= 200,D =2,C=4and real features. Newsgroups is extracted from the 20
newsgroups dataset (D = 500,N = 800,C =4, binary features). Spambase has N= 100,C =2,D=
57, binary features. Digits is the CEDAR Buffalo digits (N = 200,C =1 0,D=6 4, binarized features).
Fglass is forensic glass dataset (N = 214,C =6,D=9, real features). Source: Table 1 of (Heller and
Ghahramani 2005). Used with kind permission of Katherine Heller.
25.5.4.3 Learning the hyper-parameters

The model has two free-parameters: αandλ,w h e r e λare the hyper-parameters for the prior
on the parameters θ. In (Heller and Ghahramani 2005), they show how one can back-propagate
gradients of the form∂p(Dk|Tk)
∂λthrough the tree, and thus perform an empirical Bayes estimate
of the hyper-parameters.
25.5.4.4 Experimental results
(Heller and Ghahramani 2005) compared BHC with traditional agglomerative clustering algo-

rithms on various data sets in terms of purity scores. The results are shown in Table 25.1. Wesee that BHC did much better than the other methods on all datasets except the forensic glassone.
Figure 25.16 visualizes the tree structure estimated by BHC and agglomerative hierarchical

clustering (AHC) on the newsgroup data (using a beta-Bernoulli model). The BHC tree is clearlysuperior (look at the colors at the leaves, which represent class labels). Figure 25.17 is a zoom-inon the top few nodes of these two trees. BHC splits off clusters concerning sports from clustersconcerning cars and space. AHC keeps sports and cars merged together. Although sports andcars both fall under the same “rec” newsgroup heading (as opposed to space, that comes underthe “sci” newsgroup heading), the BHC clustering still seems more reasonable, and this is borneout by the quantitative purity scores.

BHC has also been applied to gene expression data, with good results (Savage et al. 2009).
25.6 Clustering datapoints and features
So far, we have been concentrating on clustering datapoints. But each datapoint is oftendescribed by multiple features, and we might be interested in clustering them as well. Below wedescribe some methods for doing this.

902 Chapter25. Clustering
4 Newsgroups Average Linkage Clustering
(a)
4 Newsgroups Bayesian Hierarchical Clustering
(b)
Figure 25.16 Hierarchical clustering applied to 800 documents from 4 newsgroups (red is rec.autos, blue
is rec.sport.baseball, green is rec.sport.hockey, and magenta is sci.space). Top: average linkage hierarchical
clustering. Bottom: Bayesian hierarchical clustering. Each of the leaves is labeled with a color, accordingto which newsgroup that document came from. We see that the Bayesian method results in a clusteringthat is more consistent with these labels (which were not used during model ﬁtting). Source: Figure 7 of(Heller and Ghahramani 2005). Used with kind permission of Katherine Heller.

25.6. Clusteringdatapointsandfeatures 903
All Data
Game
Team PlayCarSpace NASA
Baseball  Pitch     HitNHLHockey Round CarDealer DriveSpaceNASA Orbit354 446
205 149 284 162
(a)All Data
Car
Baseball Engine
PitcherBoston BallCarPlayer SpaceQuebecJet Boston
VehicleDealer DriverTeamGame Hockey1 799
2 797
1 796
(b)
Figure 25.17 Zoom-in on the top nodes in the trees of Figure 25.16. (a) Bayesian method. (b) Average
linkage. We show the 3 most probable words per cluster. The number of documents at each cluster is also
given. Source: Figure 5 of (Heller and Ghahramani 2005). Used with kind permission of Katherine Heller.
25.6.1 Biclustering
Clustering the rows and columns is known as biclustering orcoclustering. This is widely used
in bioinformatics, where the rows often represent genes and the columns represent conditions.
It can also be used for collaborative ﬁltering, where the rows represent users and the columnsrepresent movies.

A variety of ad hoc methods for biclustering have been proposed; see (Madeira and Oliveira
2004) for a review. Here we present a simple probabilistic generative model, based on (Kempet al. 2006) (see also (Sheng et al. 2003) for a related approach). The idea is to associate eachrow and each column with a latent indicator, r
i∈{1,...,Kr},cj∈{1,...,Kc}. We then
assume the data are iid across samples and across features within each block:
p(x|r,c,θ)=∏
i∏
jp(xij|ri,cj,θ)=p(xij|θri,cj) (25.62)
whereθa,bare the parameters for row cluster aand column cluster b. Rather than using a ﬁnite

i∏
number of clusters for the rows and columns, we can use a Dirchlet process, as in the inﬁniterelational model which we discuss in Section 27.6.1. We can ﬁt this model using e.g., (collapsed)Gibbs sampling.
The behavior of this model is illustrated in Figure 25.18. The data has the form X(i,j)=1
iff animal ihas feature j,w h e r ei=1:5 0andj=1:8 5. The animals represent whales, bears,

horses, etc. The features represent properties of the habitat (jungle, tree, coastal), or anatomicalproperties (has teeth, quadrapedal), or behavioral properties (swims, eats meat), etc. The model,using a Bernoulli likelihood, was ﬁt to the data. It discovered 12 animal clusters and 33 featureclusters. For example, it discovered a bicluster that represents the fact that mammals tend tohave aquatic features.
25.6.2 Multi-view clustering

The problem with biclustering is that each object (row) can only belong to one cluster. Intuitively,an object can have multiple roles, and can be assigned to different clusters depending on which

904 Chapter25. Clustering
O1 killer whale, blue whale, humpback, seal, walrus, dolphin
O2 antelope, horse, giraffe, zebra, deer
O3 monkey, gorilla, chimp
O4 hippo, elephant, rhino
O5 grizzly bear, polar bear
F1 ﬂippers, strain teeth, swims, arctic, coastal, ocean, water
F2 hooves, long neck, horns
F3 hands, bipedal, jungle, tree
F4 bulbous body shape, slow, inactive
F5 meat teeth, eats meat, hunter, ﬁerce
F6 walks, quadrapedal, groundF1 2 4 35
O1
O2
O3
O4
O56
Figure 25.18 Illustration of biclustering . We show 5 of the 12 animal clusters, and 6 of the 33 feature
clusters. The original data matrix is shown, partitioned according to the discovered clusters. From Figure
3 of (Kemp et al. 2006). Used with kind permission of Charles Kemp.
(a)riv
v=1:∞xij
i=1:Nβcj θjk
k=1:∞
j=1:Dα γ
(b)
Figure 25.19 (a) Illustration of multi-view clustering. Here we have 3 views (column partitions). In the
ﬁrst view, we have 2 clusters (row partitions). In the second view, we have 3 clusters. In the third view,

(a)riv
v=1:∞xij
k=1:∞
j=1:Dα γ
(b)
we have 2 clusters. The number of views and partitions are inferred from data. Rows within each colored
block are assumed to generated iid; however, each column can have a different distributional form, which
is useful for modeling discrete and continuous data. From Figure 1 of (Guan et al. 2010). Used with kind
permission of Jennifer Dy. (b) Corresponding DGM.
subset of features you use. For example, in the animal dataset, we may want to group the
animals on the basis of anatomical features (e.g., mammals are warm blooded, reptiles are not),

or on the basis of behavioral features (e.g., predators vs prey).
We now present a model that can capture this phenomenon. This model was indepen-
dently proposed in (Shafto et al. 2006; Mansinghka et al. 2011), who call it crosscat (for cross-
categorization), and in (Guan et al. 2010; Cui et al. 2010), who call it (non-parametric) multi-clust .
(See also (Rodriguez and Ghosh 2011) for a very similar model.) The idea is that we partition the
columns (features) into Vgroups or views,s ocj∈{1,...,V},w h e r ej∈{1,...,D}indexes

25.6. Clusteringdatapointsandfeatures 905
features. We will use a Dirichlet process prior for p(c), which allows Vto grow automatically.
Then for each partition of the columns (i.e., each view), call it v, we partition the rows, again
using a DP, as illustrated in Figure 25.19(a). Let riv∈{1,...,K(v)}be the cluster to which
thei’th row belongs in view v. Finally, having partitioned the rows and columns, we generate
the data: we assume all the rows and columns within a block are iid. We can deﬁne the model
more precisely as follows:
p(c,r,D)=p( c)p(r|c)p(D|r,c) (25.63)
p(c)=DP( c|α) (25.64)
p(r|c)=V(c)∏
v=1DP(rv|β) (25.65)
p(D|r,c,θ)=V(c)∏
v=1∏
j:cj=v⎡
⎣K(rv)∏
k=1∫∏
i:riv=kp(xij|θjk)p(θjk)dθjk⎤⎦ (25.66)
See Figure 25.19(b) for the DGM.
2
If the data is binary, and we use a Beta(γ,γ)prior forθjk, the likelihood reduces to
p(D|r,c,γ)=V(c)∏
v=1∏
j:cj=vK(rv)∏
k=1Beta(n j,k,v+γ,nj,k,v+γ)
Beta(γ,γ)(25.67)
wherenj,k,v=∑
i:ri,v=kI(xij=1 )counts the number of features which are on in the j’th

v=1∏
j:cj=v⎡
⎣K(rv)∏
k=1∫∏
2
v=1∏
column for view vand for row cluster k. Similarly, nj,k,vcounts how many features are off.
The model is easily extended to other kinds of data, by replacing the beta-Bernoulli with, say,
the Gaussian-Gamma-Gaussian model, as discussed in (Guan et al. 2010; Mansinghka et al. 2011).
Approximate MAP estimation can be done using stochastic search (Shafto et al. 2006), and

approximate inference can be done using variational Bayes (Guan et al. 2010) or Gibbs sampling(Mansinghka et al. 2011). The hyper-parameter γfor the likelihood can usually be set in a non-
informative way, but results are more sensitive to the other two parameters, since αcontrols
the number of column partitions, and βcontrols the number of row partitions. Hence a more
robust technique is to infer the hyper-parameters using MH. This also speeds up convergence(Mansinghka et al. 2011).

Figure 25.20 illustrates the model applied to some binary data containing 22 animals and 106

features. The ﬁgures shows the (approximate) MAP partition. The ﬁrst partition of the columnscontains taxonomic features, such as “has bones”, “is warm-blooded”, “lays eggs”, etc. Thisdivides the animals into birds, reptiles/ amphibians, mammals, and invertebrates. The secondpartition of the columns contains features that are treated as noise, with no apparent structure(except for the single row labeled “frog”). The third partition of the columns contains ecologicalfeatures like “dangerous”, “carnivorous”, “lives in water”, etc. This divides the animals into prey,land predators, sea predators and air predators. Thus each animal (row) can belong to a different

2. The dependence between randcis not shown, since it is not a dependence between the values of rivandcj, but
between the cardinality of vandcj. In other words, the number of row partitions we need to specify (the number of
views, indexed by v) depends on the number of column partitions (clusters) that we have.

906 Chapter25. Clustering
lives in lakes
is an amphibian
is a rodent
is tall
is a fish
is slimy
has horns
has hooves
is a feline
roars
has fins
has webbed feet
eats nuts
is smooth
lives in trees
is large
lives in cold climates 
 
is ferocious
is dangerous
is a carnivore
is a predator
lives in water
flies
is long
eats leaves
eats animals
lives in grass
eats fish
lives in hot climatesLeopard
Alligator
Python
Seal
Dolphin
Frog
Jellyfish
Octopus
Penguin
Finch
Seagull
Owl
Eagle
Dragonfly
Bat
Grasshopper
Ant
Bee
Sheep
Monkey
Iguana
Ostrich
has bones
lays eggs
is warm−blooded
is a mammal
squawks
has a beak
has a tongue
is green
has a spinal cord
is a lizard
has antennae
has flippers
has paws
has a large brain
has a tail
is furry
eats mice
eats rodents
has a snout
is brown
makes loud noises
has teeth
has feet
is smart
travels in groupsLeopard
Sheep
Seal
Dolphin
Monkey
Bat
Alligator
Iguana
Frog
Python
Finch
Ostrich
Seagull
Owl
Penguin
Eagle
Ant
Bee
Jellyfish
Octopus

Eagle
Dragonfly
Bat
Ant
Bee
Sheep
Monkey
Iguana
Ostrich
has bones
lays eggs
is a mammal
squawks
has a beak
is green
is a lizard
has paws
has a tail
is furry
eats mice
has a snout
is brown
has teeth
has feet
is smart
Sheep
Seal
Dolphin
Monkey
Bat
Alligator
Iguana
Frog
Python
Finch
Ostrich
Seagull
Owl
Penguin
Eagle
Ant
Bee
Jellyfish
Octopus
DragonflyA BC
Frog
Figure 25.20 MAP estimate produced by the crosscat system when applied to a binary data matrix of
animals (rows) by features (columns). See text for details. Source: Figure 7 of (Shafto et al. 2006) . Used
with kind permission of Vikash Mansingkha.
cluster depending on what set of features are considered. Uncertainty about the partitions can
be handled by sampling.
It is interesting to compare this model to a standard inﬁnite mixture model. While the

Ant
Bee
Jellyfish
Octopus
Frog
standard model can represent any density on ﬁxed-sized vectors as N→∞, it cannot cope
withD→∞, since it has no way to handle irrelevant, noisy or redundant features. By contrast,
the crosscat/multi-clust system is robust to irrelevant features: it can just partition them off,
and cluster the rows only using the relevant features. Note, however, that it does not need a
separate “background” model, since everything is modelled using the same mechanism. This is

useful, since one’s person’s noise is another person’s signal. (Indeed, this symmetry may explain
why multi-clust outperformed the sparse mixture model approach of (Law et al. 2004) in the
experiments reported in (Guan et al. 2010).)

26 Graphical model structure learning
26.1 Introduction
We have seen how graphical models can be used to express conditional independence assump-
tions between variables. In this chapter, we discuss how to learn the structure of the graphicalmodel itself. That is, we want to compute p(G|D),w h e r eG is the graph structure, represented
as anV×Vadjacency matrix.
As we discussed in Section 1.3.3, there are two main applications of structure learning: knowl-
edge discovery and density estimation. The former just requires a graph topology, whereas thelatter requires a fully speciﬁed model.
The main obstacle in structure learning is that the number of possible graphs is exponential in
the number of nodes: a simple upper bound is O(2
V(V−1)/2). Thus the full posterior p(G|D)
is prohibitively large: even if we could afford to compute it, we could not even store it. So wewill seek appropriate summaries of the posterior. These summary statistics depend on our task.

If our goal is knowledge discovery, we may want to compute posterior edge marginals,
p(G
st=1|D); we can then plot the corresponding graph, where the thickness of each edge
represents our conﬁdence in its presence. By setting a threshold, we can generate a sparsegraph, which can be useful for visualization purposes (see Figure 1.11).
Ifourgoalisdensityestimation, wemaywanttocomputetheMAPgraph, ˆG∈argmax
Gp(G|D).

p(G
Gp(G|D).
In most cases, ﬁnding the globally optimal graph will take exponential time, so we will use dis-crete optimization methods such as heuristic search. However, in the case of trees, we canﬁnd the globally optimal graph structure quite efficiently using exact methods, as we discuss inSection 26.3.
If density estimation is our only goal, it is worth considering whether it would be more

appropriate to learn a latent variable model, which can capture correlation between the visiblevariables via a set of latent common causes (see Chapters 12 and 27). Such models are ofteneasier to learn and, perhaps more importantly, they can be applied (for prediction purposes)much more efficiently, since they do not require performing inference in a learned graph withpotentially high treewidth. The downside with such models is that the latent factors are oftenunidentiﬁable, and hence hard to interpret. Of course, we can combine graphical model structurelearning and latent variable learning, as we will show later in this chapter.

In some cases, we don’t just want to model the observed correlation between variables;
instead, we want to model the causalstructure behind the data, so we can predict the effects
of manipulating variables. This is a much more challenging task, which we brieﬂy discuss in

908 Chapter26. Graphicalmodelstructurelearning
bible
christian
god
jesuscase
factchildren
government
religion computer
science
universitycourse
earth
orbitevidence
human
worldjews law president
rights
statewargun israellaunch
nasa
shuttle
spacelunar
moonmission
solar
Figure 26.1 Part of a relevance network constructed from the 20-news data shown in Figure 1.2. We
show edges whose mutual information is greater than or equal to 20% of the maximum pairwise MI. For
clarity, the graph has been cropped, so we only show a subset of the nodes and edges. Figure generatedbyrelevanceNetworkNewsgroupDemo .
Section 26.6.
26.2 Structure learning for knowledge discovery
Since computing the MAP graph or the exact posterior edge marginals is in general computa-

solar
tionally intractable (Chickering 1996), in this section we discuss some “quick and dirty” methodsfor learning graph structures which can be used to visualize one’s data. The resulting models donot constitute consistent joint probability distributions, so they cannot be used for prediction,and they cannot even be formally evaluated in terms of goodness of ﬁt. Nevertheless, thesemethods are a useful ad hoc tool to have in one’s data visualization toolbox, in view of theirspeed and simplicity.

26.2.1 Relevance networks
Arelevance network is a way of visualizing the pairwise mutual information between multiple
random variables: we simply choose a threshold and draw an edge from node ito nodejif
I(Xi;Xj)is above this threshold. In the Gaussian case, I(Xi;Xj)=−1
2log(1−ρ2
ij),w h e r e
ρijis the correlation coefficient (see Exercise 2.13), so we are essentially visualizing Σ; this is
known as the covariance graph (Section 19.4.4.1).

2log(1−ρ2
ij),w h e r e
This method is quite popular in systems biology (Margolin et al. 2006), where it is used to
visualize the interaction between genes. The trouble with biological examples is that they are
hard for non-biologists to understand. So let us instead illustrate the idea using natural languagetext. Figure 26.1 gives an example, where we visualize the MI between words in the newsgroupdataset from Figure 1.2. The results seem intuitively reasonable.
However, relevance networks suffer from a major problem: the graphs are usually very dense,

since most variables are dependent on most other variables, even after thresholding the MIs.For example, suppose X
1directly inﬂuences X2which directly inﬂuences X3(e.g., these form
components of a signalling cascade, X1−X2−X3). ThenX1has non-zero MI with X3(and
vice versa), so there will be a 1−3edge in the relevance network. Indeed, most pairs will be

26.2. Structurelearningforknowledgediscovery 909
aids
healthbaseball
fans
games hit
league
playersbible
god
jesus
orbitbmw
engine
honda
cancer
disease
patientscar
dealerdrive
driver guninsurance oil card
graphics
videocase children
food christian
mission
religion sciencecomputercourse
data
solar
problem evidencemedicinedisk
filesmemory scsi
spacedisplay
imageserverdoctor
helpdos
windows
formatearth
marsmoon
email
phonepower
msgfact
hockey
team
ftphuman
vitaminwater
number
versionnhl puck season
win
wongovernment
jews president
rightswar
law systemisrael
researchstate launch
shuttle
lunarmac
pc
studiesnasa
satelliteprogramquestion
university software
technologyworld
Figure 26.2 A dependency network constructed from the 20-news data. We show all edges with regres-
sion weight above 0.5 in the Markov blankets estimated by ℓ1penalized logistic regression. Undirected
edges represent cases where a directed edge was found in both directions. From Figure 4.9 of (Schmidt

number
win
rightswar
shuttle
lunarmac
pc
2010). Used with kind permission of Mark Schmidt.
A better approach is to use graphical models, which represent conditional independence,
rather than dependence. In the above example, X1is conditionally independent of X3given
X2, so there will not be a 1−3edge. Consequently graphical models are usually much sparser
than relevance networks, and hence are a more useful way of visualizing interactions between
multiple variables.
26.2.2 Dependency networks

A simple and efficient way to learn a graphical model structure is to independently ﬁt Dsparse
full-conditional distributions p(xt|x−t); this is called a dependency network (Heckerman et al.
2000). The chosen variables constitute the inputs to the node, i.e., its Markov blanket. Wecan then visualize the resulting sparse graph. The advantage over relevance networks is thatredundant variables will not be selected as inputs.
We can use any kind of sparse regression or classiﬁcation method to ﬁt each CPD. (Heckerman

et al. 2000) uses classiﬁcation/ regression trees, (Meinshausen and Buhlmann 2006) use ℓ
1-
regularized linear regression, (Wainwright et al. 2006) use ℓ1-regularized logistic regression (see
depnetFit for some code), (Dobra 2009) uses Bayesian variable selection, etc. (Meinshausen

910 Chapter26. Graphicalmodelstructurelearning
and Buhlmann 2006) discuss theoretical conditions under which ℓ1-regularized linear regression
can recover the true graph structure, assuming the data was generated from a sparse Gaussian
graphical model.
Figure 26.2 shows a dependency network that was learned from the 20-newsgroup data using
ℓ1regularized logistic regression, where the penalty parameter λwas chosen by BIC. Many
of the words present in these estimated Markov blankets represent fairly natural associations(aids:disease, baseball:fans, bible:god, bmw:car, cancer:patients, etc.). However, some of the esti-mated statistical dependencies seem less intuitive, such as baseball:windows and bmw:christian.We can gain more insight if we look not only at the sparsity pattern, but also the values of theregression weights. For example, here are the incoming weights for the ﬁrst 5 words:
•aids: children (0.53), disease (0.84), fact (0.47), health (0.77), president (0.50), research (0.53)

•baseball:christian(-0.98),drive(-0.49), games (0.81), god(-0.46),government (-0.69), hit (0.62),
memory(-1.29), players (1.16), season (0.31), software(-0.68),windows(-1.45)
•bible:car(-0.72),card(-0.88), christian (0.49), fact (0.21), god (1.01), jesus (0.68), orbit (0.83),
program(-0.56), religion (0.24), version (0.49)
•bmw: car (0.60), christian (-11.54), engine (0.69), god(-0.74),government (-1.01),help(-0.50),
windows(-1.43)
•cancer: disease (0.62), medicine (0.58), patients (0.90), research (0.49), studies (0.70)
Words in italic red have negative weights, which represents a dissociative relationship. For
example, the model reﬂects that baseball:windows is an unlikely combination. It turns out thatmost of the weights are negative (1173 negative, 286 positive, 8541 zero) in this model.
In addition to visualizing the data, a dependency network can be used for inference. However,

the only algorithm we can use is Gibbs sampling, where we repeatedly sample the nodes withmissing values from their full conditionals. Unfortunately, a product of full conditionals doesnot, in general, constitute a representation of any valid joint distribution (Heckerman et al.2000), so the output of the Gibbs sampler may not be meaningful. Nevertheless, the method cansometimes give reasonable results if there is not much missing data, and it is a useful methodfor data imputation (Gelman and Raghunathan 2001). In addition, the method can be used asan initialization technique for more complex structure learning methods that we discuss below.

26.3 Learning tree structures
For the rest of this chapter, we focus on learning fully speciﬁed joint probability models, whichcan be used for density estimation, prediction and knowledge discovery.
Since the problem of structure learning for general graphs is NP-hard (Chickering 1996), we
start by considering the special case of trees. Trees are special because we can learn theirstructure efficiently, as we disuscs below, and because, once we have learned the tree, we canuse them for efficient exact inference, as discussed in Section 20.2.

26.3. Learningtreestructures 911
4312
(a)4312
(b)4312
(c)
Figure 26.3 An undirected tree and two equivalent directed trees.
26.3.1 Directed or undirected tree?
Before continuing, we need to discuss the issue of whether we should use directed or undirected
trees. A directed tree, with a single root node r, deﬁnes a joint distribution as follows:
p(x|T)=∏
t∈Vp(xt|xpa(t)) (26.1)
where we deﬁne pa(r)=∅. For example, in Figure 26.3(b-c), we have
p(x1,x2,x3,x4|T)=p( x1)p(x2|x1)p(x3|x2)p(x4|x2) (26.2)
=p(x2)p(x1|x2)p(x3|x2)p(x4|x2) (26.3)
We see that the choice of root does not matter: both of these models are equivalent.
To make the model more symmetric, it is preferable to use an undirected tree. This can be
represented as follows:
p(x|T)=∏
t∈Vp(xt)∏
(s,t)∈Ep(xs,xt)
p(xs)p(xt)(26.4)
wherep(xs,xt)is an edge marginal and p(xt)is a node marginal. For example, in Figure 26.3(a)
we have
p(x1,x2,x3,x4|T)=p(x1)p(x2)p(x3)p(x4)p(x1,x2)p(x2,x3)p(x2,x4)
p(x1)p(x2)p(x2)p(x3)p(x2)p(x4)(26.5)

p(x|T)=∏
t∈Vp(xt)∏
we have
To see the equivalence with the directed representation, let us cancel terms to get
p(x1,x2,x3,x4|T)=p( x1,x2)p(x2,x3)
p(x2)p(x2,x4)
p(x2)(26.6)
=p(x1)p(x2|x1)p(x3|x2)p(x4|x2) (26.7)
=p(x2)p(x1|x2)p(x3|x2)p(x4|x2) (26.8)
wherep(xt|xs)=p(xs,xt)/p(xs).
Thus a tree can be represented as either an undirected or directed graph: the number of

we have
parameters is the same, and hence the complexity of learning is the same. And of course,inference is the same in both representations, too. The undirected representation, which issymmetric, is useful for structure learning, but the directed representation is more convenientfor parameter learning.

912 Chapter26. Graphicalmodelstructurelearning
26.3.2 Chow-Liu algorithm for ﬁnding the ML tree structure
Using Equation 26.4, we can write the log-likelihood for a tree as follows:
logp(D|θ,T)=∑
t∑
kNtklogp(xt=k|θ)
+∑
s,t∑
j,kNstjklogp(xs=j,xt=k|θ)
p(xs=j|θ)p(xt=k|θ)(26.9)
whereNstjkis the number of times node sis in state jand node tis in state k, andNtkis
the number of times node tis in state k. We can rewrite these counts in terms of the empirical
distribution: Nstjk=Npemp(xs=j,xt=k)andNtk=Npemp(xt=k). Setting θto the
MLEs, this becomes
logp(D|θ,T)
N=∑
t∈V∑
kpemp(xt=k)logpemp(xt=k) (26.10)
+∑
(s,t)∈E(T)I(xs,xt|ˆθst) (26.11)
where I(xs,xt|ˆθst)≥0is the mutual information between xsandxtgiven the empirical
distribution:
I(xs,xt|ˆθst)=∑
j∑
kpemp(xs=j,xt=k)logpemp(xs=j,xt=k)
pemp(xs=j)pemp(xt=k)(26.12)
Since the ﬁrst term in Equation 26.11 is independent of the topology T, we can ignore it when
learning structure. Thus the tree topology that maximizes the likelihood can be found by

N=∑
t∈V∑
+∑
j∑
computing the maximum weight spanning tree, where the edge weights are the pairwise mutualinformations, I(y
s,yt|ˆθst). This is called the Chow-Liu algorithm (Chow and Liu 1968).
There are several algorithms for ﬁnding a max spanning tree (MST). The two best known are
Prim’s algorithm and Kruskal’s algorithm. Both can be implemented to run in O(ElogV)time,
whereE=V2is the number of edges and Vis the number of nodes. See e.g., (Sedgewick and
Wayne 2011, 4.3) for details. Thus the overall running time is O(NV2+V2logV), where the

ﬁrst term is the cost of computing the sufficient statistics.
Figure 26.4 gives an example of the method in action, applied to the binary 20 newsgroups
data shown in Figure 1.2. The tree has been arbitrarily rooted at the node representing “email”.The connections that are learned seem intuitively reasonable.
26.3.3 Finding the MAP forest

Since all trees have the same number of parameters, we can safely used the maximum likelihoodscore as a model selection criterion without worrying about overﬁtting. However, sometimes wemay want to ﬁt a forestrather than a single tree, since inference in a forest is much faster than

in a tree (we can run belief propagation in each tree in the forest in parallel). The MLE criterionwill never choose to omit an edge. However, if we use the marginal likelihood or a penalizedlikelihood (such as BIC), the optimal solution may be a forest. Below we give the details for themarginal likelihood case.

26.3. Learningtreestructures 913
aidsbaseball
hit
biblebmw
cancercar
dealer engine hondacard
graphics video
casechildrenchristian
computer coursedata
diseasedisk
drive memory system
display
server
doctordos
scsidriver
earth
godemail
ftp phone
oil
evidencefact
questionfansfiles
format windows
food
msg waterimage
games
jesus religion
power president rights state war
gunhealth
insurance medicinehelp hockey
nhl
humanisraeljewslaunch
lawleague
lunarmac
mars
patients studiesmissionmoon nasanumber
orbit
satellite solar
vitaminpc
software
playersproblemprogram
space
puck
research scienceseason shuttle technology
universityteamversion
worldwin
won
Figure 26.4 The MLE tree on the 20-newsgroup data. From Figure 4.11 of (Schmidt 2010). Used with kind
permission of Mark Schmidt. (A topologically equivalent tree can be produced using chowliuTreeDemo .)
In Section 26.4.2.2, we explain how to compute the marginal likelihood of any DAG using a

lawleague
lunarmac
mars
orbit
vitaminpc
software
space
puck
worldwin
won
Dirichlet prior for the CPTs. The resulting expression can be written as follows:
logp(D|T)=∑
t∈Vlog∫N∏
i=1p(xit|xi,pa(t)|θt)p(θt)dθt=∑
tscore(Nt,pa(t)) (26.13)
whereNt,pa(t)are the counts (sufficient statistics) for node tand its parents, and score is
deﬁned in Equation 26.28.
Now suppose we only allow DAGs with at most one parent. Following (Heckerman et al. 1995,
p227), let us associate a weight with each s→tedge,ws,t≜score(t|s)−score(t|0),w h e r e

t∈Vlog∫N∏
score(t|0)is the score when thas no parents. Note that the weights might be negative (unlike
the MLE case, where edge weights are aways non-negative because they correspond to mutual
information). Then we can rewrite the objective as follows:
tscore(t|pa(t)) =∑
twpa(t),t+∑
tscore(t|0) (26.14)
The last term is the same for all trees T, so we can ignore it. Thus ﬁnding the most probable
tree amounts to ﬁnding a maximal branching in the corresponding weighted directed graph.
This can be found using the algorithm in (Gabow et al. 1984).

914 Chapter26. Graphicalmodelstructurelearning
If the scoring function is prior and likelihood equivalent (these terms are explained in Sec-
tion 26.4.2.3), we have
score(s|t)+score( t|0) =score(t|s)+score( s|0) (26.15)
and hence the weight matrix is symmetric. In this case, the maximal branching is the same
as the maximal weight forest. We can apply a slightly modiﬁed version of the MST algorithmto ﬁnd this (Edwards et al. 2010). To see this, let G=(V,E)be a graph with both positive
and negative edge weights. Now let G
′be a graph obtained by omitting all the negative edges
fromG. This cannot reduce the total weight, so we can ﬁnd the maximum weight forest of G
by ﬁnding the MST for each connected component of G′. We can do this by running Kruskal’s
algorithm directly on G′: there is no need to ﬁnd the connected components explicitly.
26.3.4 Mixtures of trees

A single tree is rather limited in its expressive power. Later in this chapter we discuss ways tolearn more general graphs. However, the resulting graphs can be expensive to do inference in.An interesting alternative is to learn a mixture of trees (Meila and Jordan 2000), where each

mixture component may have a different tree topology. This is like an unsupervised version ofthe TAN classiﬁer discussed in Section 10.2.1. We can ﬁt a mixture of trees by using EM: in theE step, we compute the responsibilities of each cluster for each data point, and in the M step,we use a weighted version of the Chow-Liu algorithm. See (Meila and Jordan 2000) for details.
In fact, it is possible to create an “inﬁnite mixture of trees”, by integrating out over all possible
trees. Remarkably, this can be done in V

3time using the matrix tree theorem. This allows us to
perform exact Bayesian inference of posterior edge marginals etc. However, it is not tractable touse this inﬁnite mixture for inference of hidden nodes. See (Meila and Jaakkola 2006) for details.
26.4 Learning DAG structures
In this section, we discuss how to compute (functions of) p(G|D),w h e r eGis constrained to be
a DAG. This is often called Bayesian network structure learning. In this section, we assume
there is no missing data, and that there are no hidden variables. This is called the complete
data assumption. For simplicity, we will focus on the case where all the variables are categoricaland all the CPDs are tables, although the results generalize to real-valued data and other kindsof CPDs, such as linear-Gaussian CPDs.

Our presentation is based in part on (Heckerman et al. 1995), although we will follow the
notation of Section 10.4.2. In particular, let x
it∈{1,...,K t}be the value of node tin casei,
whereKtis the number of states for node t.L e tθtck≜p(xt=k|xpa(t)=c),f o rk=1:K t,
andc=1:C t,w h e r eCtis the number of parent combinations (possible conditioning cases).
For notational simplicity, we will often assume Kt=K, so all nodes have the same number of
states. We will also let dt=d i m ( p a ( t))be the degree or fan-in of node t, so that Ct=Kdt.
26.4.1 Markov equivalence

In this section, we discuss some fundamental limits to our ability to learn DAG structures fromdata.

26.4. LearningDAGstructures 915
X1
X2X3
X4X5X1
X2X3
X4X5X1
X2X3
X4X5G1 G2 G3
Figure 26.5 Three DAGs. G1andG3are Markov equivalent, G2is not.
Consider the following 3 DGMs: X→Y→Z,X←Y←ZandX←Y→Z. These all
represent the same set of CI statements, namely
X⊥Z|Y, X̸⊥Z (26.16)
We say these graphs are Markov equivalent, since they encode the same set of CI assumptions.
That is, they all belong to the same Markov equivalence class. However, the v-structure
X→Y←ZencodesX⊥ZandX̸⊥Z|Y, which represents the opposite set of CI
assumptions.
One can prove the following theorem.
Theorem 26.4.1 (Verma and Pearl (Verma and Pearl 1990)). TwostructuresareMarkovequivalent
ifftheyhavethesameundirectedskeletonandthesamesetofv-structures.
For example, referring to Figure 26.5, we see that G1̸≡G2, since reversing the 2→4arc
creates a new v-structure. However, G1≡G3, since reversing the 1→5arc does not create a
new v-structure.
We can represent a Markov equivalence class using a single partially directed acyclic graph

(PDAG), also called an essential graph orpattern, in which some edges are directed and some
undirected. The undirected edges represent reversible edges; any combination is possible so
long as no new v-structures are created. The directed edges are called compelled edges, since
changing their orientation would change the v-structures and hence change the equivalenceclass. For example, the PDAG X−Y−Zrepresents{X→Y→Z,X←Y←Z,X←Y→
Z}which encodes X̸⊥ZandX⊥Z|Y. See Figure 26.6.

The signiﬁcance of the above theorem is that, when we learn the DAG structure from data,
we will not be able to uniquely identify all of the edge directions, even given an inﬁnite amountof data. We say that we can learn DAG structure “up to Markov equivalence”. This also cautionsus not to read too much into the meaning of particular edge orientations, since we can oftenchange them without changing the model in any observable way.

916 Chapter26. Graphicalmodelstructurelearning
ZYX
ZYX
ZYX
ZYX
ZYX
ZYX
≡≡
Figure 26.6 PDAG representation of Markov equivalent DAGs.
26.4.2 Exact structural inference
In this section, we discuss how to compute the exact posterior over graphs, p(G|D), ignoring
for now the issue of computational tractability.
26.4.2.1 Deriving the likelihood
Assuming there is no missing data, and that all CPDs are tabular, the likelihood can be written
p(D|G,θ)=N∏
i=1V∏
t=1Cat(xit|xi,pa(t),θt) (26.17)
=N∏
i=1V∏
t=1Ct∏
c=1Cat(xit|θtc)I(xi,pa(t )=c)(26.18)
=N∏
i=1V∏
t=1Ct∏
c=1Kt∏
k=1θI(xi,t=k,xi,pa(t)=c)
tck(26.19)
=V∏
t=1Ct∏
c=1Kt∏
k=1θNtck
tck(26.20)
whereNtckis the number of times node tis in state kand its parents are in state c. (Technically
these counts depend on the graph structure G, but we drop this from the notation.)
26.4.2.2 Deriving the marginal likelihood

i=1V∏
=N∏
i=1V∏
t=1Ct∏
=N∏
i=1V∏
t=1Ct∏
c=1Kt∏
=V∏
t=1Ct∏
c=1Kt∏
k=1θNtck
Of course, choosing the graph with the maximum likelihood will always pick a fully connectedgraph (subject to the acyclicity constraint), since this maximizes the number of parameters. Toavoid such overﬁtting, we will choose the graph with the maximum marginal likelihood, p(D|G);
the magic of the Bayesian Occam’s razor will then penalize overly complex graphs.
To compute the marginal likelihood, we need to specify priors on the parameters. We will
make two standard assumptions. First, we assume global prior parameter independence,
which means
p(θ)=
V∏
t=1p(θt) (26.21)

26.4. LearningDAGstructures 917
Second, we assume local prior parameter independence, which means
p(θt)=Ct∏
c=1p(θtc) (26.22)
for eacht. It turns out that these assumtions imply that the prior for each row of each CPT
must be a Dirichlet (Geiger and Heckerman 1997), that is,
p(θtc)=D i r (θtc|αtc) (26.23)
Given these assumptions, and using the results of Section 5.3.2.2, we can write down the
marginal likelihood of any DAG as follows:
p(D|G)=V∏
t=1Ct∏
c=1∫⎡
⎣∏
i:xi,pa(t )=cCat(xit|θtc)⎤⎦Dir(θ
tc)dθtc (26.24)
=V∏
t=1Ct∏
c=1B(Ntc+αtc)
B(αtc)(26.25)
=V∏
t=1Ct∏
c=1Γ(Ntc)
Γ(Ntc+αtc)Kt∏
k=1Γ(Ntck+αG
tck)
Γ(αG
ijk)(26.26)
=V∏
t=1score(Nt,pa(t)) (26.27)
whereNtc=∑
kNtck,αtc=∑
kαtck,Nt,pa(t)is the vector of counts (sufficient statistics) for
nodetand its parents, and score() is a local scoring function deﬁned by
score(Nt,pa(t))≜Ct∏
B(αtc)(26.28)
We say that the marginal likelihood decomposes or factorizes according to the graph structure.
26.4.2.3 Setting the prior

=V∏
t=1Ct∏
=V∏
t=1Ct∏
c=1Γ(Ntc)
tck)
Γ(αG
=V∏
How should we set the hyper-parameters αtck? It is tempting to use a Jeffreys prior of the form
αtck=1
2(Equation 5.62). However, it turns out that this violates a property called likelihood
equivalence, which is sometimes considered desirable. This property says that if G1andG2are
Markov equivalent (Section 26.4.1), they should have the same marginal likelihood, since they are

αtck=1
essentially equivalent models. Geiger and Heckerman (1997) proved that, for complete graphs,the only prior that satisﬁes likelihood equivalence and parameter independence is the Dirichletprior, where the pseudo counts have the form
α
tck=αp0(xt=k,xpa(t)=c) (26.29)
whereα>0is called the equivalent sample size , andp0is some prior joint probability dis-
tribution. This is called the BDeprior, which stands for Bayesian Dirichlet likelihood equivalent.

918 Chapter26. Graphicalmodelstructurelearning
To derive the hyper-parameters for other graph structures, Geiger and Heckerman (1997)
invoked an additional assumption called parameter modularity, which says that if node Xt
has the same parents in G1andG2, thenp(θt|G1)=p(θt|G2). With this assumption, we
can always derive αtfor a node tin any other graph by marginalizing the pseudo counts in
Equation 26.29.
Typically the prior distribution p0is assumed to be uniform over all possible joint conﬁgura-
tions. In this case, we have
αtck=α
KtCt(26.30)
sincep0(xt=k,xpa(t)=c)=1
KtCt. Thus if we sum the pseudo counts over all Ct×Kt
entries in the CPT, we get a total equivalent sample size of α. This is called the BDeuprior,
where the “u” stands for uniform. This is the most widely used prior for learning Bayes net
structures. For advice on setting the global tuning parameter α, see (Silander et al. 2007).
26.4.2.4 Simple worked example

αtck=α
We now give a very simple worked example from (Neapolitan 2003, p.438). Suppose we havejust 2 binary nodes, and the following 8 data cases:
X
1X2
1112112211211122
Suppose we are interested in two possible graphs: G
1isX1→X2andG2is the disconnected
graph. The empirical counts for node 1 in G1areN1=( 5,3)and for node 2 are
X2=1X2=2
X1=141
X1=212
The BDeu prior for G1isα1=(α/2,α/2),α2|x 1=1=(α/4,α/4)andα2|x 1=2=
(α/4,α/4).F o rG2, the prior for θ1is the same, and for θ2it isα2|x 1=1=(α/2,α/2)
andα2|x 1=2=(α/2,α/2). I fw es e tα =4, and use the BDeu prior, we ﬁnd p(D|G1)=

X
1X2
X2=1X2=2
X1=141
X1=212
7.2150×10−6andp(D|G2)=6.7465×10−6. Hence the posterior probabilites, under a
uniform graph prior, are p(G1|D)=0.51678andp(G2|D)=0.48322.
26.4.2.5 Example: analysis of the college plans dataset
We now consider a more interesting example from (Heckerman et al. 1997). Consider the dataset collected in 1968 by Sewell and Shah which measured 5 variables that might inﬂuence thedecision of high school students about whether to attend college. Speciﬁcally, the variables areas follows:

26.4. LearningDAGstructures 919
Figure 26.7 The two most probable DAGs learned from the Sewell-Shah data. Source: (Heckerman et al.
1997) . Used with kind permission of David Heckerman
•SexMale or female
•SESSocio economic status: low, lower middle, upper middle or high.
•IQIntelligence quotient: discretized into low, lower middle, upper middle or high.
•PEParental encouragment: low or high
•CPCollege plans: yes or no.
These variables were measured for 10,318 Wisconsin high school seniors. There are 2×4×
4×2×= 128possible joint conﬁgurations.
Heckerman et al. computed the exact posterior over all 29,281 possible 5 node DAGs, except
for ones in which SEX and/or SES have parents, and/or CP have children. (The prior probability
of these graphs was set to 0, based on domain knowledge.) They used the BDeu score with
α=5, although they said that the results were robust to any αin the range 3 to 40. The top
two graphs are shown in Figure 26.7. We see that the most probable one has approximately all

of the probability mass, so the posterior is extremely peaked.
It is tempting to interpret this graph in terms of causality (see Section 26.6). In particular,
it seems that socio-economic status, IQ and parental encouragment all causally inﬂuence the
decision about whether to go to college, which makes sense. Also, sex inﬂuences college plans
only indirectly through parental encouragement, which also makes sense. However, the direct

link from socio economic status to IQ seems surprising; this may be due to a hidden common
cause. In Section 26.5.1.4 we will re-examine this dataset allowing for the presence of hidden
variables.
26.4.2.6 The K2 algorithm
Suppose we know a total ordering of the nodes. Then we can compute the distribution over
parents for each node independently, without the risk of introducing any directed cycles: we

920 Chapter26. Graphicalmodelstructurelearning
simply enumerate over all possible subsets of ancestors and compute their marginal likelihoods.1
If we just return the best set of parents for each node, we get the the K2 algorithm (Cooper
and Herskovits 1992).
26.4.2.7 Handling non-tabular CPDs
If all CPDs are linear Gaussian, we can replace the Dirichlet-multinomial model with the normal-
gamma model, and thus derive a different exact expression for the marginal likelihood. See(Geiger and Heckerman 1994) for the details. In fact, we can easily combine discrete nodesand Gaussian nodes, as long as the discrete nodes always have discrete parents; this is called aconditional Gaussian DAG. Again, we can compute the marginal likelihood in closed form. See
(Bottcher and Dethlefsen 2003) for the details.
In the general case (i.e., everything except Gaussians and CPTs), we need to approximate the
marginal likelihood. The simplest approach is to use the BIC approximation, which has the form
∑
tlogp(Dt|ˆθt)−KtCt

∑
2logN (26.31)
26.4.3 Scaling up to larger graphs
The main challenge in computing the posterior over DAGs is that there are so many possiblegraphs. More precisely, (Robinson 1973) showed that the number of DAGs on Dnodes satisﬁes
the following recurrence:
f(D)=D∑
i=1(−1)i+1(D
i)
2i(D−i)f(D−i) (26.32)
forD>2. The base case is f(1) = 1. Solving this recurrence yields the following sequence:
1, 3, 25, 543, 29281, 3781503, etc.2In view of the enormous size of the hypothesis space, we are
generally forced to use approximate methods, some of which we review below.
26.4.3.1 Approximating the mode of the posterior

f(D)=D∑
i)
We can use dynamic programming to ﬁnd the globally optimal MAP DAG (up to Markov equiv-alence) (Koivisto and Sood 2004; Silander and Myllmaki 2006). Unfortunately this method takesV2
Vtime and space, making it intractable beyond about 16 nodes. Indeed, the general problem
of ﬁnding the globally optimal MAP DAG is provably NP-complete (Chickering 1996),
Consequently, we must settle for ﬁnding a locally optimal MAP DAG. The most common

method is greedy hill climbing: at each step, the algorithm proposes small changes to thecurrent graph, such as adding, deleting or reversing a single edge; it then moves to the neigh-boring graph which most increases the posterior. The method stops when it reaches a lo-cal maximum. It is important that the method only proposes local changes to the graph,
1. We can make this method more efficient by using ℓ1-regularization to select the parents (Schmidt et al. 2007). In this

case, we need to approximate the marginal likelhood as we discuss below.
2. A longer list of values can be found at http://www .research .att.com/~njas/sequences/A003024 . Interest-
ingly, the number of DAGs is equal to the number of (0,1) matrices all of whose eigenvalues are positive real numbers(McKay et al. 2004).

26.4. LearningDAGstructures 921
fans leaguebible bmw
cancer
healthcar
dealer
driverengine
honda
insuranceoil
card
graphicscase
children
foodwaterchristian
computer
sciencecourse
dataproblemdisease
medicinepatientsdisk
dos
files
macmemoryprogramspace
displaydoctor
helpversion
windowsdrive
formatscsi
earth
imagemarsmoon
email
phonepowerevidence
fact msg
studiesgod
ftp
vitaminnumbergames
hockey
nhl puckseason
team
winwongovernment
jews law
presidentrights
wargun
hit human
israel
research statelaunch
lunar
pcsoftware
servervideomission
orbit
solarnasa
satelliteshuttle
playersquestion
universitysystemtechnology
world
Figure 26.8 A locally optimal DAG learned from the 20-newsgroup data. From Figure 4.10 of (Schmidt
since this enables the change in marginal likelihood (and hence the posterior) to be computed

ftp
hockey
team
jews law
wargun
hit human
israel
lunar
orbit
solarnasa
world
in constant time (assuming we cache the sufficient statistics). This is because all but oneor two of the terms in Equation 26.25 will cancel out when computing the log Bayes factorδ(G→G
′)=l o gp(G′|D)−logp(G|D).
We can initialize the search from the best tree, which can be found using exact methods

discussed in Section 26.3. For speed, we can restrict the search so it only adds edges which arepart of the Markov blankets estimated from a dependency network (Schmidt 2010). Figure 26.8gives an example of a DAG learned in this way from the 20-newsgroup data.
We can use techniques such as multiple random restarts to increase the chance of ﬁnding a
good local maximum. We can also use more sophisticated local search methods, such as geneticalgorithms or simulated annealing, for structure learning.
26.4.3.2 Approximating other functions of the posterior

If our goal is knowledge discovery, the MAP DAG can be misleading, for reasons we discussed inSection 5.2.1. A better approach is to compute the probability that each edge is present, p(G
st=
1|D), of the probability there is a path from stot. We can do this exactly using dynamic
programming (Koivisto 2006; Parviainen and Koivisto 2011). Unfortunately these methods takeV2
Vtime in the general case, making them intractable for graphs with more than about 16

922 Chapter26. Graphicalmodelstructurelearning
nodes.
An approximate method is to sample DAGs from the posterior, and then to compute the
fraction of times there is an s→tedge or path for each (s,t)pair. The standard way to draw
samples is to use the Metropolis Hastings algorithm (Section 24.3), where we use the same local
proposal as we did in greedy search (Madigan and Raftery 1994).
A faster-mixing method is to use a collapsed MH sampler, as suggested in (Friedman and
Koller 2003). This exploits the fact that, if a total ordering of the nodes is known, we canselect the parents for each node independently, without worrying about cycles, as discussed inSection 26.4.2.6. By summing over all possible choice of parents, we can marginalize out thispart of the problem, and just sample total orders. (Ellis and Wong 2008) also use order-space(collapsed) MCMC, but this time with a parallel tempering MCMC algorithm.
26.5 Learning DAG structure with latent variables

Sometimes the complete data assumption does not hold, either because we have missing data,and/ or because we have hidden variables. In this case, the marginal likelihood is given by
p(D|G)=∫∑
hp(D,h|θ,G)p(θ|G)dθ=∑
h∫
p(D,h|θ,G)p(θ|G)dθ (26.33)
wherehrepresents the hidden or missing data.
In general this is intractable to compute. For example, consider a mixture model, where
we don’t observe the cluster label. In this case, there are KNpossible completions of the

p(D|G)=∫∑
h∫
data (assuming we have Kclusters); we can evaluate the inner integral for each one of these
assignments to h, but we cannot afford to evaluate all of the integrals. (Of course, most of these
integrals will correspond to hypotheses with little posterior support, such as assigning singledata points to isolated clusters, but we don’t know ahead of time the relative weight of theseassignments.)
In this section, we discuss some ways for learning DAG structure when we have latent variables
and/or missing data.

26.5.1 Approximating the marginal likelihood when we have missing data
The simplest approach is to use standard structure learning methods for fully visible DAGs,but to approximate the marginal likelihood. In Section 24.7, we discussed some Monte Carlomethods for approximating the marginal likelihood. However, these are usually too slow to useinside of a search over models. Below we mention some faster deterministic approximations.
26.5.1.1 BIC approximation

A simple approximation is to use the BIC score, which is given by
BIC(G)≜logp(D|ˆθ,G)−logN
2dim(G) (26.34)
wheredim(G)is the number of degrees of freedom in the model and ˆθis the MAP or ML
estimate. However, the BIC score often severely underestimates the true marginal likelihood(Chickering and Heckerman 1997), resulting in it selecting overly simple models.

26.5. LearningDAGstructurewithlatentvariables 923
26.5.1.2 Cheeseman-Stutz approximation
We now present a better method known as the Cheeseman-Stutz approximation (CS) (Cheese-
man and Stutz 1996). We ﬁrst compute a MAP estimate of the parameters ˆθ(e.g., using EM).
Denote the expected sufficient statistics of the data by D=D(ˆθ); in the case of discrete
variables, we just “ﬁll in” the hidden variables with their expectation. We then use the exact
marginal likelihood equation on this ﬁlled-in data:
p(D|G)≈p(D|G)=∫
p(D|θ,G)p(θ|G)dθ (26.35)
However, comparing this to Equation 26.33, we can see that the value will be exponentiallysmaller, since it does not sum over all values of h. To correct for this, we ﬁrst write
logp(D|G)=l o g p(
D|G)+log p(D|G)−logp(D|G) (26.36)
and then we apply a BIC approximation to the last two terms:
logp(D|G)−logp(D|G)≈[
logp(D|ˆθ,G)−N
2dim(ˆθ)]
(26.37)
−[
2dim(ˆθ)]
(26.38)
=l o gp(D|ˆθ,G)−logp(D|ˆθ,G) (26.39)
Putting it altogether we get

2dim(ˆθ)]
(26.37)
−[
2dim(ˆθ)]
(26.38)
logp(D|G)≈logp(D|G)+log p(D|ˆθ,G)−logp(D|ˆθ,G) (26.40)
The ﬁrst term p(D|G)can be computed by plugging in the ﬁlled-in data into the exact marginal
likelihood. The second term p(D|ˆθ,G), which involves an exponential sum (thus matching the
“dimensionality” of the left hand side) can be computed using an inference algorithm. The ﬁnaltermp(
D|ˆθ,G)can be computed by plugging in the ﬁlled-in data into the regular likelihood.
26.5.1.3 Variational Bayes EM

An even more accurate approach is to use the variational Bayes EM algorithm. Recall fromSection 21.6 that the key idea is to make the following factorization assumption:
p(θ,z
1:N|D)≈q(θ)q(z)=q(θ)∏
iq(zi) (26.41)
whereziare the hidden variables in case i. In the E step, we update the q(zi), and in the
M step, we update q(θ). The corresponding variational free energy provides a lower bound on

p(θ,z
the log marginal likelihood. In (Beal and Ghahramani 2006), it is shown that this bound is amuch better approximation to the true log marginal likelihood (as estimated by a slow annealedimportance sampling procedure) than either BIC or CS. In fact, one can prove that the variationalbound will always be more accurate than CS (which in turn is always more accurate than BIC).

924 Chapter26. Graphicalmodelstructurelearning
p(male) = 0.48
SESH
SEX
PE
IQ
CPp(H= 0) = 0.63
p(H= 1) = 0.37
H
0
1
0
1PE
lowlow
highhighp(IQ=high|PE,H)
0.098
0.220.210.49H
01p(SES=high| H)
0.088
0.51
SEX
male
female
male
femaleSES
low
low
high
highp(PE=high|SES,SEX)
0.32
0.166
0.860.81SES
lowlowlowlow
highhighhigh
highPE
low
high
low
high
low
high
low
highIQ
low
low
highhigh
lowlow
high
highp(CP=yes|SES,IQ,PE)
0.011
0.1700.124
0.53
0.093
0.390.24
0.84
Figure 26.9 The most probable DAG with a single binary hidden variable learned from the Sewell-Shah
data. MAP estimates of the CPT entries are shown for some of the nodes. Source: (Heckerman et al. 1997).
Used with kind permission of David Heckerman.
26.5.1.4 Example: college plans revisited
Let us revisit the college plans dataset from Section 26.4.2.5. Recall that if we ignore the

low
high
low
high
low
highIQ
low
low
highhigh
lowlow
high
0.011
0.53
0.093
0.390.24
0.84
possibility of hidden variables there was a direct link from socio economic status to IQ in theMAP DAG. Heckerman et al. decided to see what would happen if they introduced a hiddenvariableH, which they made a parent of both SES and IQ, representing a hidden common cause.
They also considered a variant in which Hpoints to SES, IQ and PE. For both such cases, they

considered dropping none, one, or both of the SES-PE and PE-IQ edges. They varied the numberof states for the hidden node from 2 to 6. Thus they computed the approximate posterior over8×5=4 0different models, using the CS approximation.
The most probable model which they found is shown in Figure 26.9. This is 2·10
10times
more likely than the best model containing no hidden variable. It is also 5·109times more
likely than the second most probable model with a hidden variable. So again the posterior isvery peaked.

10times
These results suggests that there is indeed a hidden common cause underlying both the
socio-economic status of the parents and the IQ of the children. By examining the CPT entries,we see that both SES and IQ are more likely to be high when Htakes on the value 1. They
interpret this to mean that the hidden variable represents “parent quality” (possibly a geneticfactor). Note, however, that the arc between H and SES can be reversed without changing the v-structures in the graph, and thus without affecting the likelihood; this underscores the difficultyin interpreting hidden variables.
Interestingly, the hidden variable model has the same conditional independence assumptions

amongst the visible variables as the most probable visible variable model. So it is not pos-sible to distinguish between these hypotheses by merely looking at the empirical conditionalindependencies in the data (which is the basis of the constraint-based approach to structure
learning (Pearl and Verma 1991; Spirtes et al. 2000)). Instead, by adopting a Bayesian approach,which takes parsimony into account (and not just conditional independence), we can discover

26.5. LearningDAGstructurewithlatentvariables 925
the possible existence of hidden factors. This is the basis of much of scientiﬁc and everday
human reasoning (see e.g. (Griffiths and Tenenbaum 2009) for a discussion).
26.5.2 Structural EM
One way to perform structural inference in the presence of missing data is to use a standardsearch procedure (deterministic or stochastic), and to use the methods from Section 26.5.1 toestimate the marginal likelihood. However, this approach is very efficient, because the marginallikelihood does not decompose when we have missing data, and nor do its approximations.For example, if we use the CS approximation or the VBEM approximation, we have to performinference in every neighboring model, just to evaluate the quality of a single move!
(Friedman 1997b; Thiesson et al. 1998) presents a much more efficient approach called the
structural EM algorithm. The basic idea is this: instead of ﬁtting each candidate neighboring

graph and then ﬁlling in its data, ﬁll in the data once, and use this ﬁlled-in data to evaluatethe score of all the neighbors. Although this might be a bad approximation to the marginallikelihood, it can be a good enough approximation of the difference in marginal likelihoodsbetween different models, which is all we need in order to pick the best neighbor.
More precisely, deﬁne
D(G0,ˆθ0)to be the data ﬁlled in using model G0with MAP parameters
ˆθ0. Now deﬁne a modiﬁed BIC score as follows:
scoreBIC(G,D)≜logp(D|ˆθ,G)−logN
2dim(G)+log p(G)+log p(ˆθ|G) (26.42)
where we have included the log prior for the graph and parameters. One can show (Friedman1997b) that if we pick a graph Gwhich increases the BIC score relative to G
0on the expected
data, it will also increase the score on the actual data, i.e.,

scoreBIC(G,D(G0,ˆθ0))−scoreBIC(G0,D(G0,ˆθ0)≤scoreBIC(G,D)−scoreBIC(G0,D)(26.43)
To convert this into an algorithm, we proceed as follows. First we initialize with some graph
G0and some set of parameters θ0. Then we ﬁll-in the data using the current parameters — in
practice, this means when we ask for the expected counts for any particular family, we perform

inference using our current model. (If we know which counts we will need, we can precomputeall of them, which is much faster.) We then evaluate the BIC score of all of our neighbors usingthe ﬁlled-in data, and we pick the best neighbor. We then reﬁt the model parameters, ﬁll-in thedata again, and repeat. For increased speed, we may choose to only reﬁt the model every fewsteps, since small changes to the structure hopefully won’t invalidate the parameter estimatesand the ﬁlled-in data too much.

One interesting application is to learn a phylogenetic tree structure. Here the observed leaves
are the DNA or protein sequences of currently alive species, and the goal is to infer the topologyof the tree and the values of the missing internal nodes. There are many classical algorithms forthis task (see e.g., (Durbin et al. 1998)), but one that uses SEM is discussed in (Friedman et al.2002).
Another interesting application of this method is to learn sparse mixture models (Barash and

Friedman 2002). The idea is that we have one hidden variable Cspecifying the cluster, and we
have to choose whether to add edges C→X
tfor each possible feature Xt. Thus some features
will be dependent on the cluster id, and some will be independent. (See also (Law et al. 2004)

(Harmeling and Williams 2011). Used with kind permission of Stefan Harmeling.
for a different way to perform this task, using regular EM and a set of bits, one per feature, that
are free to change across data cases.)
26.5.3 Discovering hidden variables
In Section 26.5.1.4, we introduced a hidden variable “by hand”, and then ﬁgured out the localtopology by ﬁtting a series of different models and computing the one with the best marginallikelihood. How can we automate this process?
Figure 11.1 provides one useful intuition: if there is a hidden variable in the “true model”,
then its children are likely to be densely connected. This suggest the following heuristic (Elidanet al. 2000): perform structure learning in the visible domain, and then look for structural

signatures, such as sets of densely connected nodes (near-cliques); introduce a hidden variableand connect it to all nodes in this near-clique; and then let structural EM sort out the details.Unfortunately, this technique does not work too well, since structure learning algorithms arebiased against ﬁtting models with densely connected cliques.
Another useful intuition comes from clustering. In a ﬂat mixture model, also called a latent
class model, the discrete latent variable provides a compressed representation of its children.Thus we want to create hidden variables with high mutual information with their children.
One way to do this is to create a tree-structured hierarchy of latent variables, each of which

only has to explain a small set of children. (Zhang 2004) calls this a hierarchical latent class
model. They propose a greedy local search algorithm to learn such structures, based on addingor deleting hidden nodes, adding or deleting edges, etc. (Note that learning the optimal latent

26.5. LearningDAGstructurewithlatentvariables 927
programearth
launchlunar
marsmission moon
nasaorbit satellite
shuttlesolar
spacetechnology
h1h17
card
computerdatadisk displaydos
drivedriveremail
files formatftp
graphics
helpimage
macmemorynumber
pcphone
problemresearch sciencescsiserver software
system
universityversion
videowindowsh9
h10h11h12
h16h18
h19
h22
h23 h24h25
h26casechildren
course evidence factgovernment
gunhuman israel lawpower president
questionrights statewar
world
h2h3
h4 h20
baseball fans games
hithockey league nhl playerspuck season team win
wonh5
h6 h7aids
cancer disease doctorfoodhealth
medicine msg
patientsstudies
vitaminwater h13h21
bmwcar
dealer
engine hondainsurance
oilh15bible
christian jesusreligion
h8h14
godjews
Figure 26.11 A partially latent tree learned from the 20-newsgroup data. Note that some words can
have multiple meanings, and get connected to different latent variables, representing different “topics”. For

world
h2h3
h4 h20
wonh5
h6 h7aids
bmwcar
dealer
h8h14
godjews
example, the word “win” can refer to a sports context (represented by h5) or the Microsoft Windows context(represented by h25). From Figure 12 of (Choi et al. 2011). Used with kind permission of Jin Choi.
tree is NP-hard (Roch 2006).)
Recently (Harmeling and Williams 2011) proposed a faster greedy algorithm for learning such
models based on agglomerative hierarchical clustering. Rather than go into details, we just give

an example of what this system can learn. Figure 26.10 shows part of a latent forest learnedfrom the 20-newsgroup data. The algorithm imposes the constraint that each latent node hasexactly two children, for speed reasons. Nevertheless, we see interpretable clusters arising. Forexample, Figure 26.10 shows separate clusters concerning medicine, sports and religion. Thisprovides an alternative to LDA and other topic models (Section 4.2.2), with the added advantagethat inference in latent trees is exact and takes time linear in the number of nodes.

An alternative approach is proposed in (Choi et al. 2011), in which the observed data is not
constrained to be at the leaves. This method starts with the Chow-Liu tree on the observeddata, and then adds hidden variables to capture higher-order dependencies between internalnodes. This results in much more compact models, as shown in Figure 26.11. This model alsohas better predictive accuracy than other approaches, such as mixture models, or trees whereall the observed data is forced to be at the leaves. Interestingly, one can show that this methodcan recover the exact latent tree structure, providing the data is generated from a tree. See

928 Chapter26. Graphicalmodelstructurelearning
Figure 26.12 Google’s rephil model. Leaves represent presence or absence of words. Internal nodes
represent clusters of co-occuring words, or “concepts”. All nodes are binary, and all CPDs are noisy-OR.
The model contains 12 million word nodes, 1 million latent cluster nodes, and 350 million edges. Used
with kind permission of Brian Milch.
(Choi et al. 2011) for details. Note, however, that this approach, unlike (Zhang 2004; Harmeling
and Williams 2011), requires that the cardinality of all the variables, hidden and observed, be
the same. Furthermore, if the observed variables are Gaussian, the hidden variables must be
Gaussian also.
26.5.4 Case study: Google’s Rephil
In this section, we describe a huge DGM called Rephil, which was automatically learned from
data.3The model is widely used inside Google for various purposes, including their famous
AdSense system.4
The model structure is shown in Figure 26.12. The leaves are binary nodes, and represent

the presence or absence of words or compounds (such as “New York City”) in a text document
or query. The latent variables are also binary, and represent clusters of co-occuring words. All
CPDs are noisy-OR, since some leaf nodes (representing words) can have many parents. This
means each edge can be augmented with a hidden variable specifying if the link was activated
or not; if the link is not active, then the parent cannot turn the child on. (A very similar model
was proposed independently in (Singliar and Hauskrecht 2006).)

Parameter learning is based on EM, where the hidden activation status of each edge needs
to be inferred (Meek and Heckerman 1997). Structure learning is based on the old neuroscience
3. The original system, called “Phil”, was developed by Georges Harik and Noam Shazeer,. It has been published as US
Patent #8024372, “Method and apparatus for learning a probabilistic generative model for text”, ﬁled in 2004. Rephil is
a more probabilistically sound version of the method, developed by Uri Lerner et al. The summary below is based on

notes by Brian Milch (who also works at Google).
4. AdSense is Google’s system for matching web pages with content-appropriate ads in an automatic way, by extracting
semantic keywords from web pages. These keywords play a role analogous to the words that users type in when
searching; this latter form of information is used by Google’s AdWords system. The details are secret, but (Levy 2011)
gives an overview.

26.5. LearningDAGstructurewithlatentvariables 929
idea that “nodes that ﬁre together should wire together”. To implement this, we run inference
and check for cluster-word and cluster-cluster pairs that frequently turn on together. We thenadd an edge from parent to child if the link can signiﬁcantly increase the probability of thechild. Links that are not activated very often are pruned out. We initialize with one cluster per“document” (corresponding to a set of semantically related phrases). We then merge clusters A
andBifAexplainsB’s top words and vice versa. We can also discard clusters that are used
too rarely.
The model was trained on about 100 billion text snippets or search queries; this takes several

weeks, even on a parallel distributed computing architecture. The resulting model contains 12million word nodes and about 1 million latent cluster nodes. There are about 350 million linksin the model, including many cluster-cluster dependencies. The longest path in the graph haslength 555, so the model is quite deep.
Exact inference in this model is obviously infeasible. However note that most leaves will be

off, since most words do not occur in a given query; such leaves can be analytically removed, asshown in Exercise 10.7. We an also prune out unlikely hidden nodes by following the strongestlinks from the words that are on up to their parents to get a candidate set of concepts. Wethen perform iterative conditional modes to ﬁnd a good set of local maxima. At each step ofICM, each node sets its value to its most probable state given the values of its neighbors in itsMarkov blanket. This continues until it reaches a local maximum. We can repeat this processa few times from random starting conﬁgurations. At Google, this can be made to run in 15milliseconds!
26.5.5 Structural equation models *
Astructural equation model (Bollen 1989) is a special kind of directed mixed graph (Sec-

tion 19.4.4.1), possibly cyclic, in which all CPDs are linear Gaussian, and in which all bidirectededges represent correlated Gaussian noise. Such models are also called path diagrams. SEMs
are widely used, especially in economics and social science. It is common to interpret the edgedirections in terms of causality, where directed cycles are interpreted is in terms of feedback
loops(see e.g., (Pearl 2000, Ch.5)). However, the model is really just a way of specifying a joint
Gaussian, as we show below. There is nothing inherently “causal” about it at all. (We discusscausality in Section 26.6.)
We can deﬁne an SEM as a series of full conditionals as follows:
x
i=μi+∑
j̸=iwijxj+ϵi (26.44)
whereϵ∼N(0,Ψ). We can rewrite the model in matrix form as follows:
x=Wx+μ+ϵ⇒x=(I−W)−1(ϵ+μ) (26.45)
Hence the joint distribution is given by p(x)=N(μ,Σ)where
Σ=(I−W)−1Ψ(I−W)−T(26.46)

x
i=μi+∑
We draw an arc Xi←Xjif|wij|>0.I fWis lower triangular then the graph is acyclic. If,
in addition, Ψis diagonal, then the model is equivalent to a Gaussian DGM, as discussed in
Section 10.2.5; such models are called recursive.I f Ψis not diagonal, then we draw a bidirected

930 Chapter26. Graphicalmodelstructurelearning
Y3Z3Z1 Z1Y1 Y2
Figure 26.13 A cyclic directed mixed graphical model (non-recursive SEM). Note the Z1→Z2→Z3→
Z1feedback loop.
arcXi↔Xjfor each non-zero off-diagonal term. Such edges represent correlation, possibly
due to a hidden common cause.
When using structural equation models, it is common to partition the variables into latent
variables, Zt, and observed or manifest variables Yt. For example, Figure 26.13 illustrates the
⎛
⎜⎜⎜⎜⎜⎜⎝X
1
X2
X3
X4
X5
X6⎞
⎟⎟⎟⎟⎟⎟⎠=⎛
⎜⎜⎜⎜⎜⎜⎝Z
1
Z2
Z3
Y1
Y2
Y3⎞
⎟⎟⎟⎟⎟⎟⎠=⎛
⎜⎜⎜⎜⎜⎜⎝00 w
13000
w210 0 000
0w320 000
w410 0 000
0w520 000
00 w63000⎞
⎟⎟⎟⎟⎟⎟⎠⎛
⎜⎜⎜⎜⎜⎜⎝Z
1
Z2
Z3
Y1
Y2
Y3⎞
⎟⎟⎟⎟⎟⎟⎠+⎛
⎜⎜⎜⎜⎜⎜⎝ϵ
1
ϵ2
ϵ3
ϵ4
ϵ5
ϵ6⎞
⎟⎟⎟⎟⎟⎟⎠, (26.47)
where
Ψ=⎛
⎜⎜⎜⎜⎜⎜⎝Ψ
1100000
0Ψ 220000
00 Ψ 33000
000 Ψ 44Ψ450
000 Ψ 54Ψ550
00000 Ψ 66⎞
⎟⎟⎟⎟⎟⎟⎠(26.48)
The presence of a feedback loop Z
1→Z2→Z3is evident from the fact that Wis not lower
triangular. Also the presence of confounding between Y1andY2is evident in the off-diagonal
terms inΨ.

X2
X3
X4
X5
X6⎞
⎟⎟⎟⎟⎟⎟⎠=⎛
⎜⎜⎜⎜⎜⎜⎝Z
1
Z2
Z3
Y1
Y2
Y3⎞
⎟⎟⎟⎟⎟⎟⎠=⎛
13000
w210 0 000
0w320 000
w410 0 000
0w520 000
00 w63000⎞
⎟⎟⎟⎟⎟⎟⎠⎛
⎜⎜⎜⎜⎜⎜⎝Z
1
Z2
Z3
Y1
Y2
Y3⎞
⎟⎟⎟⎟⎟⎟⎠+⎛
⎜⎜⎜⎜⎜⎜⎝ϵ
1
ϵ2
ϵ3
ϵ4
ϵ5
ϵ6⎞
where
Ψ=⎛
⎜⎜⎜⎜⎜⎜⎝Ψ
1100000
0Ψ 220000
00 Ψ 33000
00000 Ψ 66⎞
terms inΨ.
Often we assume there are multiple observations for each latent variable. To ensure identiﬁa-
bility, we can set the mean of the latent variables Ztto 0, and we can set the regression weights
ofZt→Ytto 1. This essentially deﬁnes the scale of each latent variable. (In addition to the
Z’s, there are the extra hidden variables implied by the presence of the bidirected edges.)
The standard practice in the SEM community, as exempliﬁed by the popular commercial

terms inΨ.
software package called LISREL(available from http://www .ssicentral .com/lisrel/ ), is to

26.6. LearningcausalDAGs 931
build the structure by hand, to estimate the parameters by maximum likelihood, and then to
test if any of the regression weights are signiﬁcantly different from 0, using standard frequentistmethods. However, one can also use Bayesian inference for the parameters (see e.g., (Dunsonet al. 2005)). Structure learning in SEMs is rare, but since recursive SEMs are equivalent toGaussian DAGs, many of the techniques we have been discussing in this section can be applied.
SEMs are closely related to factor analysis (FA) models (Chapter 12). The basic difference is
that in an FA model, the latent Gaussian has a low-rank covariance matrix, and the observednoise has a diagonal covariance (hence no bidirected edges). In an SEM, the covariance of thelatent Gaussian has a sparse Cholesky decomposition (at least if Wis acyclic), and the observed
noise might have a full covariance matrix.
Note that SEMs can be extended in many ways. For example, we can add covariates/ input

variables (possibly noisily observed), we can make some of the observations be discrete (e.g., byusing probit links), and so on.
26.6 Learning causal DAGs
Causal models are models which can predict the effects of interventions to, or manipulations

of, a system. For example, an electronic circuit diagram implicitly provides a compact encodingof what will happen if one removes any given component, or cuts any wire. A causal medicalmodel might predict that if I continue to smoke, I am likely to get lung cancer (and hence ifI cease smoking, I am less likely to get lung cancer). Causal claims are inherently stronger,yet more useful, than purely associative claims, such as “people who smoke often have lung
cancer”.
Causal models are often represented by DAGs (Pearl 2000), although this is somewhat contro-

cancer”.
versial (Dawid 2010). We explain this causal interpretation of DAGs below. We then show howto use a DAG to do causal reasoning. Finally, we brieﬂy discuss how to learn the structure ofcausal DAGs. A more detailed description of this topic can be found in (Pearl 2000) and (Kollerand Friedman 2009, Ch.21).
26.6.1 Causal interpretation of DAGs
In this section, we deﬁne a directed edge A→Bin a DAG to mean that “A directly causes B”,
so if we manipulate A, thenBwill change. This is known as the causal Markov assumption.
(Of course, we have not deﬁned the word “causes”, and we cannot do that by appealing to aDAG, lest we end up with a cyclic deﬁnition; see (Dawid 2010) for further disussion of this point.)
We will also assume that all relevant variables are included in the model, i.e., there are no
unknown confounders, reﬂecting hidden common causes. This is called the causal sufficiency

assumption. (If there are known to be confounders, they should be added to the model, althoughone can sometimes use mixed directed graphs (Section 26.5.5) as a way to avoid having to modelconfounders explicitly.)
Assuming we are willing to make the causal Markov and causal sufficiency assumptions, we
can use DAGs to answer causal questions. The key abstraction is that of a perfect intervention;
this represents the act of setting a variable to some known value, say setting X
itoxi. A real

world example of such a perfect intervention is a gene knockout experiment, in which a geneis “silenced”. We need some notational convention to distinguish this from observing that X
i

932 Chapter26. Graphicalmodelstructurelearning
X XGGdo(X=x)
Figure 26.14 Surgical intervention on X. Based on (Pe’er 2005).
happens to have value xi. We use Pearl’s do calculus notation (as in the verb “to do”) and write
do(Xi=xi)to denote the event that we set Xitoxi. A causal model can be used to make
inferences of the form p(x|do(Xi=xi)), which is different from making inferences of the form
p(x|Xi=xi).
To understand the difference between conditioning on interventions and conditioning on
observations (i.e., the difference between doing and seeing), consider a 2 node DGM S→Y,i n
whichS=1if you smoke and S=0otherwise, and Y=1if you have yellow-stained ﬁngers,
andY=0otherwise. If I observe you have yellow ﬁngers, I am licensed to infer that you are
probably a smoker (since nicotine causes yellow stains):
p(S=1|Y=1 )>p (S=1 ) (26.49)
However, if I intervene and paintyour ﬁngers yellow, I am no longer licensed to infer this, since
I have disrupted the normal causal mechanism. Thus

p(S=1|do(Y=1 ) )=p( S=1 ) (26.50)
One way to model perfect interventions is to use graph surgery: represent the joint distri-
bution by a DGM, and then cut the arcs coming into any nodes that were set by intervention.

See Figure 26.14 for an example. This prevents any information ﬂow from the nodes that wereintervened on from being sent back up to their parents. Having perform this surgery, we canthen perform probabilistic inference in the resulting “mutilated” graph in the usual way to reasonabout the effects of interventions. We state this formally as follows.
Theorem26.6.1 (Manipulationtheorem(Pearl2000;Spirtesetal.2000)). . Tocompute p(X
i|do(Xj))
forsetsofnodes i,j,wecanperformsurgicalinterventiononthe Xjnodesandthenusestandard
probabilisticinferenceinthemutilatedgraph.

i|do(Xj))
We can generalize the notion of a perfect intervention by adding interventions as explicit
action nodes to the graph. The result is like an inﬂuence diagram, except there are no utility
nodes (Lauritzen 2000; Dawid 2002). This has been called the augmented DAG (Pearl 2000). We

26.6. LearningcausalDAGs 933
xy
Figure 26.15 Illustration of Simpson’s paradox. Figure generated by simpsonsParadoxGraph .
can then deﬁne the CPD p(Xi|do(Xi))to be anything we want. We can also allow an action to
affect multiple nodes. This is called a fat hand intervention, a reference to someone trying to
change a single component of some system (e.g., an electronic circuit), but accidently touching
multiple components and thereby causing various side effects (see (Eaton and Murphy 2007) fora way to model this using augmented DAGs).
26.6.2 Using causal DAGs to resolve Simpson’s paradox
In this section, we assume we know the causal DAG. We can then do causal reasoning byapplying d-separation to the mutilated graph. In this section, we give an example of this, andshow how causal reasoning can help resolve a famous paradox, known as Simpon’s paradox.
Simpson’s paradox says that any statistical relationship between two variables can be reversed

by including additional factors in the analysis. For example, suppose some cause C(say, taking
a drug) makes some effect E(say getting better) more likely
P(E|C)>P(E|¬C)
and yet, when we condition on the gender of the patient, we ﬁnd that taking the drug makesthe effect less likely in both females (F ) and males (¬ F):
P(E|C,F)<P(E|¬C,F)
P(E|C,¬F)<P(E|¬C,¬F)
This seems impossible, but by the rules of probability, this is perfectly possible, because theevent space where we condition on (¬C,F)or(¬C,¬F)can be completely different to the

event space when we just condition on ¬C. The table of numbers below shows a concrete
example (from (Pearl 2000, p175)):
Combined Male Female
E¬E Total Rate E¬ETotal Rate E¬ETotal Rate
C20 20 40 50% 18 12 30 60% 2 8 10 20%
¬C16 24 40 40% 7 3 10 70% 9 21 30 30%
Total36 44 80 25 15 40 11 29 40

934 Chapter26. Graphicalmodelstructurelearning
From this table of numbers, we see that
p(E|C)=2 0/40 = 0.5 >p(E|¬C)=1 6/40 = 0.4 (26.51)
p(E|C,F)=2/10 = 0.2 <p(E|¬C,F)=9/30 = 0.3 (26.52)
p(E|C,¬F)=1 8/30 = 0.6 <p(E|¬,¬F)=7/10 = 0.7 (26.53)
A visual representation of the paradox is given in in Figure 26.15. The line which goes up and
to the right shows that the effect (y -axis) increases as the cause (x-axis) increases. However, the
dots represent the data for females, and the crosses represent the data for males. Within eachsubgroup, we see that the effect decreases as we increase the cause.
It is clear that the effect is real, but it is still very counter-intuitive. The reason the paradox
arises is that we are interpreting the statements causally, but we are not using proper causalreasoning when performing our calculations. The statement that the drug Ccauses recovery E
is
P(E|do(C))>P(E|do(¬C)) (26.54)
whereas the data merely tell us
P(E|C)>P(E|¬C) (26.55)
Thisis

is
Thisis
not a contradiction. Observing Cis positive evidence for E, since more males than
females take the drug, and the male recovery rate is higher (regardless of the drug). ThusEquation 26.55 does not imply Equation 26.54.
Nevertheless, we are left with a practical question: should we use the drug or not? It seems
like if we don’t know the patient’s gender, we should use the drug, but as soon as we discoverif they are male or female, we should stop using it. Obviously this conclusion is ridiculous.
To answer the question, we need to make our assumptions more explicit. Suppose reality can

be modeled by the causal DAG in Figure 26.16(a). To compute the causal effect of ConE,w e
need toadjust for (i.e., condition on) the confounding variable F. This is necessary because
there is a backdoor path fromCtoEviaF, so we need to check the C→Erelationship for
each value of Fseparately, to make sure the relationship between CandEis not affected by
any value of F.
Suppose that for each value of F, taking the drug is harmful, that is,
p(E|do(C),F)<p(E|do(¬C),F) (26.56)
p(E|do(C),¬F)<p(E|do(¬C),¬F) (26.57)

Thenw
e can show that taking the drug is harmful overall:
p(E|do(C))<p(E|do(¬C)) (26.58)
The proof is as follows (Pearl 2000, p181). First, from our assumptions in Figure 26.16(a), we seethat drugs have no effect on gender
p(F|do(C)) =p(F|do(¬C)) =p(F) (26.59)
Now using the law of total probability,
p(E|do(C)) =p(E|do(C),F)p(F|do(C))+p(E|do(C),¬F)p(¬F|do(C))(26.60)
=p(E|do(C),F)p(F)+p(E|do(C),¬F)p(¬F) (26.61)

26.6. LearningcausalDAGs 935
C F
ETreatment Gender
Recovery
(a)C F
ETreatment Blood Pressure
Recovery
(b)
Figure26.16 Twodifferent modelsuses toillustrate Simpson’sparadox. (a) Fisgender andis aconfounder
for C and E. (b) F is blood pressure and is caused by C.
Similarly,
p(E|do(¬C)) =p(E|do(¬C),F)p(F)+p(E|do(¬C),¬F)p(¬F) (26.62)
Since every term in Equation 26.61 is less than the corresponding term in Equation 26.62, we
conclude that
p(E|do(C))<p(E|do(¬C)) (26.63)
So if the model in Figure 26.16(a) is correct, we should not administer the drug, since it reducesthe probability of the effect.
Now consider a different version of this example. Suppose we keep the data the same but
interpretFas something that is affected by C, such as blood pressure. See Figure 26.16(b). In
this case, we can no longer assume
p(F|do(C)) =p(F|do(¬C)) =p(F) (26.64)
and the above proof breaks down. So p(E|do(C))−p(E|do(¬C))may
 be positive or negaitve.

In the true model is Figure 26.16(b), then we should not condition on Fwhen assessing the
effect of ConE, since there is no backdoor path in this case, because of the v-structure at
F. That is, conditioning on Fmight block one of the causal pathways. In other words, by
comparing patients with the same post-treatment blood pressure (value of F), we may mask the
effect of one of the two pathways by which the drug operates to bring about recovery.

be positive or negaitve.
Thus we see that different causal assumptions lead to different causal conclusions, and hence
different courses of action. This raises the question on whether we can learn the causal modelfrom data. We discuss this issue below.
26.6.3 Learning causal DAG structures
In this section, we discuss some ways to learn causal DAG structures.

936 Chapter26. Graphicalmodelstructurelearning
26.6.3.1 Learning from observational data
In Section 26.4, we discussed various methods for learning DAG structures from observational
data. It is natural to ask whether these methods can recover the “true” DAG structure that wasused to generate the data. Clearly, even if we have inﬁnite data, an optimal method can onlyidentify the DAG up to Markov equivalence (Section 26.4.1). That is, it can identify the PDAG(partially directed acylic graph), but not the complete DAG structure, because all DAGs which areMarkov equivalent have the same likelihood.
There are several algorithms (e.g., the greedy equivalence search method of (Chickering
2002)) that are consistent estimators of PDAG structure, in the sense that they identify thetrue Markov equivalence class as the sample size goes to inﬁnity, assuming we observe all thevariables. However, we also have to assume that the generating distribution pisfaithfulto

the generating DAG G. This means that all the conditional indepence (CI) properties of pare
exactly captured by the graphical structure, so I(p)=I(G); this means there cannot be any CI
properties in pthat are due to particular settings of the parameters (such as zeros in a regression
matrix) that are not graphically explicit. For this reason, a faithful distribution is also called astabledistribution.
Suppose the assumptions hold and we learn a PDAG. What can we do with it? Instead of

recovering the full graph, we can focus on the causal analog of edge marginals, by computingthe magnitude of the causal effect of one node on another (say A on B). If we know the DAG, wecan do this using techniques described in (Pearl 2000). If the DAG is unknown, we can computea lower bound on the effect as follows (Maathuis et al. 2009): learn an equivalence class (PDAG)from data; enumerate all the DAGs in the equivalence class; apply Pearl’s do-calculus to computethe magnitude of the causal effect of A on B in each DAG; ﬁnally, take the minimum of theseeffects as the lower bound. It is usually computationally infeasible to compute all DAGs in theequivalence class, but fortunately one only needs to be able to identify the local neighborhoodofAandB, which can be esimated more efficiently, as described in (Maathuis et al. 2009). This
technique is called IDA, which is short for “intervention-calculus when the DAG is absent”.

In (Maathuis et al. 2010), this technique was applied to some yeast gene expression data. Gene
knockout data was used to estimate the “ground truth” effect of each 234 single-gene deletionson the remaining 5,361 genes. Then the algorithm was applied to 63 unperturbed (wild-type)samples, and was used to rank order the likely targets of each of the 234 genes. The methodhad a precision of 66% when the recall was set to 10%; while low, this is substantially more thanrival variable-selection methods, such as lasso and elastic net, which were only slightly abovechance.
26.6.3.2 Learning from interventional data
If we want to distinguish between DAGs within the equivalence class, we need to use interven-

tional data, where certain variables have been set, and the consequences have been measured.An example of this is the dataset in Figure 26.17(a), where proteins in a signalling pathwaywere perturbed, and their phosphorylation status was measured using a technique called ﬂowcytometry (Sachs et al. 2005).
It is straightforward to modify the standard Bayesian scoring criteria, such as the marginal
likelihood or BIC score, to handle learning from mixed observational and experimental data: we

26.6. LearningcausalDAGs 937
(a)
B2cAMPf
erkakt
jnkPsitect AKT inh U0126
PMA
p38G06967mek12rafpkc
pip3plcy
pip2
pka
Present
Missing
Int. edge
(b)
Figure 26.17 (a) A design matrix consisting of 5400 data points (rows) measuring the status (using ﬂow
cytometry) of 11 proteins (columns) under different experimental conditions. The data has been discretized
into 3 states: low (black), medium (grey) and high (white). Some proteins were explicitly controlled using
activating or inhibiting chemicals. (b) A directed graphical model representing dependencies between
various proteins (blue circles) and various experimental interventions (pink ovals), which was inferred from
this data. We plot all edges for which p(Gst=1|D)>0.5. Dotted edges are believed to exist in nature
but were not discovered by the algorithm (1 false negative). Solid edges are true positives. The light colored
edges represent the effects of intervention. Source: Figure 6d of (Eaton and Murphy 2007) . This ﬁgure can

be reproduced using the code at http://www .cs.ubc.ca/~murphyk/Software/BDAGL/index .html.

938 Chapter26. Graphicalmodelstructurelearning
just compute the sufficient statistics for a CPD’s parameter by skipping over the cases where that
node was set by intervention (Cooper and Yoo 1999). For example, when using tabular CPDs, wemodify the counts as follows:
N
tck≜∑
i:xitnot setI(xi,t=k,xi,pa(t)=c) (26.65)
The justiﬁcation for this is that in cases where node tis set by force, it is not sampled from its
usual mechanism, so such cases should be ignored when inferring the parameter θt. The mod-
iﬁed scoring criterion can be combined with any of the standard structure learning algorithms.(He and Geng 2009) discusses some methods for choosing which interventions to perform, soas to reduce the posterior uncertainty as quickly as possible (a form of active learning).
The preceeding method assumes the interventions are perfect. In reality, experimenters can

rarely control the state of individual molecules. Instead, they inject various stimulant or inhibitorchemicals which are designed to target speciﬁc molecules, but which may have side effects. Wecan model this quite simply by adding the intervention nodes to the DAG, and then learninga larger augmented DAG structure, with the constraint that there are no edges between theintervention nodes, and no edges from the “regular” nodes back to the intervention nodes.
Figure 26.17(b) shows the augmented DAG that was learned from the interventional ﬂow

cytometry data depicted in Figure 26.17(a). In particular, we plot the median graph, whichincludes all edges for which p(G
ij=1|D)>0.5. These were computed using the exact
algorithm of (Koivisto 2006). It turns out that, in this example, the median model has exactlythe same structure as the optimal MAP model, argmax
Gp(G|D), which was computed using
the algorithm of (Koivisto and Sood 2004; Silander and Myllmaki 2006).
26.7 Learning undirected Gaussian graphical models

Learning the structured of undirected graphical models is easier than learning DAG structurebecause we don’t need to worry about acyclicity. On the other hand, it is harder than learningDAG structure since the likelihood does not decompose (see Section 19.5). This precludes thekind of local search methods (both greedy search and MCMC sampling) we used to learn DAGstructures, because the cost of evaluating each neighboring graph is too high, since we have toreﬁt each model from scratch (there is no way to incrementally update the score of a model).
In this section, we discuss several solutions to this problem, in the context of Gaussian

random ﬁelds or undirected Gaussian graphical models (GGM)s. We consider structure learning
for discrete undirected models in Section 26.8.
26.7.1 MLE for a GGM
Before discussing structure learning, we need to discuss parameter estimation. The task ofcomputing the MLE for a (non-decomposable) GGM is called covariance selection (Dempster
1972).
From Equation 4.19, the log likelihood can be written as
ℓ(Ω)=l o gd e t Ω−tr(SΩ) (26.66)

26.7. LearningundirectedGaussiangraphicalmodels 939
whereΩ=Σ−1is the precision matrix, and S=1
N∑N
i=1(xi−x)(xi−x)Tis the empirical
covariance matrix. (For notational simplicity, we assume we have already estimated ˆμ=x.)
One can show that the gradient of this is given by
∇ℓ(Ω)=Ω−1−S (26.67)
However, we have to enforce the constraints that Ωst=0ifGst=0(structural zeros), and
thatΩis positive deﬁnite. The former constraint is easy to enforce, but the latter is somewhat
challenging (albeit still a convex constraint). One approach is to add a penalty term to the
objective if Ωleaves the positive deﬁnite cone; this is the approach used in ggmFitMinfunc
(see also (Dahl et al. 2008)). Another approach is to use a coordinate descent method, describedin (Hastie et al. 2009, p633), and implemented in ggmFitHtf . Yet another approach is to use
iterative proportional ﬁtting, described in Section 19.5.7. However, IPF requires identifying thecliques of the graph, which is NP-hard in general.

Interestingly, one can show that the MLE must satisfy the following property: Σ
st=Sstif
Gst=1ors=t, i.e., the covariance of a pair that are connected by an edge must match the
empirical covariance. In addition, we have Ωst=0ifGst=0, by deﬁnition of a GGM, i.e.,
the precision of a pair that are not connected must be 0. We say that Σis a positive deﬁnite
matrix completion ofS, since it retains as many of the entries in Sas possible, corresponding
to the edges in the graph, subject to the required sparsity pattern on Σ−1, corresponding to the

st=Sstif
absent edges; the remaining entries in Σare ﬁlled in so as to maximize the likelihood.
Let us consider a worked example from (Hastie et al. 2009, p652). We will use the following
adjacency matrix, representing the cyclic structure, X1−X2−X3−X4−X1, and the following
empirical covariance matrix:
G=⎛
⎜⎜⎝0101
101001011010⎞
⎟⎟⎠,S=⎛
⎜⎜⎝1 0 154
11 02 6
5 2 10 3463 1 0⎞
⎟⎟⎠(26.68)
The MLE is given by
Σ=⎛
⎜⎜⎝10.00 1.001.314.00
1.00 10.00 2.000.87
1.312.00 10.00 3.00
4.000.873.00 10.00⎞
⎟⎟⎠,Ω=⎛
⎜⎜⎝0.12−0.010−0.05
−0.010
.11−0.020
0−0.02 0.11−0.03

G=⎛
⎜⎜⎝0101
⎟⎟⎠,S=⎛
⎜⎜⎝1 0 154
11 02 6
Σ=⎛
⎟⎟⎠,Ω=⎛
−0.010
.11−0.020
−0.050−0.03 0.13⎞
⎟⎟⎠(26.69)
(SeeggmFitDemo for the code to reproduce these numbers.) The constrained elements in Ω,
and the free elements in Σ, both of which correspond to absent edges, have been highlighted.
26.7.2 Graphical lasso
We now discuss one way to learn a sparse GRF structure, which exploits the fact that there is a

Σ=⎛
⎟⎟⎠,Ω=⎛
−0.010
.11−0.020
1:1 correspondence between zeros in the precision matrix and absent edges in the graph. Thissuggests that we can learn a sparse graph structure by using an objective that encourages zerosin the precision matrix. By analogy to lasso (see Section 13.3), one can deﬁne the following ℓ
1
penalized NLL:
J(Ω)=−logdetΩ+tr(SΩ)+λ||Ω||1 (26.70)

940 Chapter26. Graphicalmodelstructurelearning
lambda=36.00, nedges=8
(a)
lambda=27.00, nedges=11
(b)
lambda=7.00, nedges=18
(c)
lambda=0.00, nedges=55
(d)
Figure 26.18 Sparse GGMs learned using graphical lasso applied to the ﬂow cytometry data. (a) λ=3 6.
(b)λ=2 7. (c)λ=7. (d)λ=0. Figure generated by ggmLassoDemo .
where||Ω||1=∑
j,k|ωjk|is the 1-norm of the matrix. This is called the graphical lasso or
Glasso.
Although the objective is convex, it is non-smooth (because of the non-differentiable ℓ1
penalty) and is constrained (because Ωmust be a positive deﬁnite matrix). Several algorithms
have been proposed for optimizing this objective (Yuan and Lin 2007; Banerjee et al. 2008; Duchi
et al. 2008), although arguably the simplest is the one in (Friedman et al. 2008), which uses a
coordinate descent algorithm similar to the shooting algorithm for lasso. See ggmLassoHtf for
an implementation. (See also (Mazumder and Hastie 2012) for a more recent version of this
algorithm.)

As an example, let us apply the method to the ﬂow cytometry dataset from (Sachs et al. 2005).
A discretized version of the data is shown in Figure 26.17(a). Here we use the original continuous
data. However, we are ignoring the fact that the data was sampled under intervention. In
Figure 26.18, we illustrate the graph structures that are learned as we sweep λfrom 0 to a large
value. These represent a range of plausible hypotheses about the connectivity of these proteins.

It is worth comparing this with the DAG that was learned in Figure 26.17(b). The DAG has the
advantage that it can easily model the interventional nature of the data, but the disadvantage
that it cannot model the feedback loops that are known to exist in this biological pathway (see
the discussion in (Schmidt and Murphy 2009)). Note that the fact that we show many UGMs and
only one DAG is incidental: we could easily use BIC to pick the “best” UGM, and conversely, we

26.7. LearningundirectedGaussiangraphicalmodels 941
could easily display several DAG structures, sampled from the posterior.
26.7.3 Bayesian inference for GGM structure *
Although the graphical lasso is reasonably fast, it only gives a point estimate of the structure.
Furthermore, it is not model-selection consistent (Meinshausen 2005), meaning it cannot recoverthe true graph even as N→∞. It would be preferable to integrate out the parameters, and
perform posterior inference in the space of graphs, i.e., to compute p(G|D). We can then extract
summaries of the posterior, such as posterior edge marginals, p(G
ij=1|D), just as we did for
DAGs. In this section, we discuss how to do this.
Note that the situation is analogous to Chapter 13, where we discussed variable selection. In
Section 13.2, we discussed Bayesian variable selection, where we integrated out the regressionweights and computed p(γ|D)and the marginal inclusion probabilities p(γ
j=1|D). Then

in Section 13.3, we discussed methods based on ℓ1regularization. Here we have the same
dichotomy, but we are presenting them in the opposite order.
If the graph is decomposable, and if we use conjugate priors, we can compute the marginal

likelihood in closed form (Dawid and Lauritzen 1993). Furthermore, we can efficiently identifythe decomposable neighbors of a graph (Thomas and Green 2009), i.e., the set of legal edgeadditions and removals. This means that we can perform relatively efficient stochastic localsearch to approximate the posterior (see e.g. (Giudici and Green 1999; Armstrong et al. 2008;Scott and Carvalho 2008)).
However, the restriction to decomposable graphs is rather limiting if one’s goal is knowledge

discovery, since the number of decomposable graphs is much less than the number of generalundirected graphs.
5
A few authors have looked at Bayesian inference for GGM structure in the non-decomposable

5
case (e.g., (Dellaportas et al. 2003; Wong et al. 2003; Jones et al. 2005)), but such methods cannotscale to large models because they use an expensive Monte Carlo approximation to the marginallikelihood (Atay-Kayis and Massam 2005). (Lenkoski and Dobra 2008) suggested using a Laplaceapproxmation. This requires computing the MAP estimate of the parameters for Ωunder a G-
Wishart prior (Roverato 2002). In (Lenkoski and Dobra 2008), they used the iterative proportional
scaling algorithm (Speed and Kiiveri 1986; Hara and Takimura 2008) to ﬁnd the mode. However,this is very slow, since it requires knowing the maximal cliques of the graph, which is NP-hardin general.

In (Moghaddam et al. 2009), a much faster method is proposed. In particular, they modify
the gradient-based methods from Section 26.7.1 to ﬁnd the MAP estimate; these algorithms donot need to know the cliques of the graph. A further speedup is obtained by just using adiagonal Laplace approximation, which is more accurate than BIC, but has essentially the samecost. This, plus the lack of restriction to decomposable graphs, enables fairly fast stochasticsearch methods to be used to approximate p(G|D)and its mode. This approach signiﬁcantly
outperfomed graphical lasso, both in terms of predictive accuracy and structural recovery, for acomparable computational cost.

5. The number of decomposable graphs on Vnodes, for V=2,...,8, is as follows ((Armstrong 2005, p158)): 2; 8; 61;
822; 18,154; 61,7675; 30,888,596. If we divide these numbers by the number of undirected graphs, which is 2V(V−1)/2,
we ﬁnd the ratios are: 1, 1, 0.95, 0.8, 0.55, 0.29, 0.12. So we see that decomposable graphs form a vanishing fraction of
the total hypothesis space.

942 Chapter26. Graphicalmodelstructurelearning
26.7.4 Handling non-Gaussian data using copulas *
The graphical lasso and variants is inhertently limited to data that is jointly Gaussian, which is
a rather severe restriction. Fortunately the method can be generalized to handle non-Gaussian,but still continuous, data in a fairly simple fashion. The basic idea is to estimate a set of D
univariate monotonic transformations f
j, one per variable j, such that the resulting transformed
data is jointly Gaussian. If this is possible, we say the data belongs to the nonparametricNormal distribution, or nonparanormal distribution (Liu et al. 2009). This is equivalent to the
family of Gaussian copulas (Klaassen and Wellner 1997). Details on how to estimate the f
j

j
transformations from the empirical cdf’s of each variable can be found in (Liu et al. 2009). Aftertransforming the data, we can compute the correlation matrix and then apply glasso in the usualway. One can show, under various assumptions, that this is a consistent estimator of the graphstructure, representing the CI assumptions of the original distribution(Liu et al. 2009).
26.8 Learning undirected discrete graphical models
The problem of learning the structure for UGMs with discrete variables is harder than theGaussian case, because computing the partition function Z(θ), which is needed for parameter

estimation, has complexity comparable to computing the permanent of a matrix, which ingeneral is intractable (Jerrum et al. 2004). By contrast, in the Gaussian case, computing Zonly
requires computing a matrix determinant, which is at most O(V
3).
Since stochastic local search is not tractable for general discrete UGMs, below we mention
some possible alternative approaches that have been tried.
26.8.1 Graphical lasso for MRFs/CRFs
It is possible to extend the graphical lasso idea to the discrete MRF and CRF case. However, nowthere is a set of parameters associated with each edge in the graph, so we have to use the graphanalog of group lasso (see Section 13.5.1). For example, consider a pairwise CRF with ternarynodes, and node and edge potentials given by
ψ
t(yt,x)=⎛
⎝vT
t1x
vT
t2x

ψ
t(yt,x)=⎛
⎝vT
t1x
vT
t2x
vT
t3x⎞⎠,ψ
st(ys,yt,x)=⎛⎝w
T
t11xwT
st12xwT
st13x
wT
st21xwT
st22xwT
st23x
wT
st31xwT
st32xwT
st33x⎞⎠ (26.71)
where we assume xbegins with a constant 1 term, to account for the offset. (If xonly contains
1, the CRF reduces to an MRF.) Note that we may choose to set some of the v
tkandwstjk
weights to 0, to ensure identiﬁability, although this can also be taken care of by the prior, as
shown in Exercise 8.5.
To learn sparse structure, we can minimize the following objective:
J=−N∑
i=1[∑
tlogψt(yit,xi,vt)+V∑
s=1V∑
t=s+1logψst(yis,yit,xi,wst)]
+λ1V∑
s=1V∑

wT
st21xwT
st22xwT
st23x
wT
st31xwT
st32xwT
J=−N∑
i=1[∑
s=1V∑
+λ1V∑
s=1V∑
t=s+1||wst||p+λ2V∑
t=1||vt||2
2 (26.72)

26.8. Learningundirecteddiscretegraphicalmodels 943
baseball
games
league
christian
god
jesus
questioncar
dealer drive enginecard
drivergraphics
pcproblem
systemvideo
windowscase
course
evidence
fact
human
law number power
rights
state
worldchildren
president
religion warcomputer
dataemail
program
sciencesoftware
universitymemory
research
spacedisk
filesdisplay
image dos
mac scsiearth
orbitformat
ftphelp
phonejews
fans
hockey
team
versionnhl
season
wingunhealth
insurance
israel
launch moonnasa
shuttle
technology
won
Figure 26.19 An MRF estimated from the 20-newsgroup data using group ℓ1regularization with λ= 256.
Isolated nodes are not plotted. From Figure 5.9 of (Schmidt 2010). Used with kind permission of Mark
Schmidt.
where||wst||pis thep-norm; common choices are p=2orp=∞, as explained in Sec-
tion 13.5.1. This method of CRF structure learning was ﬁrst suggested in (Schmidt et al. 2008).
(The use of ℓ1regularization for learning the structure of binary MRFs was proposed in (Lee

shuttle
won
Schmidt.
et al. 2006).)
Although this objective is convex, it can be costly to evaluate, since we need to perform

inference to compute its gradient, as explained in Section 19.6.3 (this is true also for MRFs). Weshould therefore use an optimizer that does not make too many calls to the objective functionor its gradient, such as the projected quasi-Newton method in (Schmidt et al. 2009). In addition,we can use approximate inference, such as convex belief propagation (Section 22.4.2), to computean approximate objective and gradient more quickly. Another approach is to apply the grouplasso penalty to the pseudo-likelihood discussed in Section 19.5.4. This is much faster, sinceinference is no longer required (Hoeﬂing and Tibshirani 2009). Figure 26.19 shows the result ofapplying this procedure to the 20-newsgroup data, where y

itindicates the presence of word t
in document i, andxi=1(so the model is an MRF).

the models we learn may be intractable to use for inference purposes, which defeats one of thetwo main reasons to learn graph structure in the ﬁrst place (the other reason being “knowledgediscovery”). There have been various attempts to learn graphical models with bounded treewidth(e.g., (Bach and Jordan 2001; Srebro 2001; Elidan and Gould 2008; Shahaf et al. 2009)), also knownasthin junction trees, but the exact problem in general is hard.
An alternative approach is to learn a model with low circuit complexity (Gogate et al.
2010; Poon and Domingos 2011). Such models may have high treewidth, but they exploit context-speciﬁc independence and determinism to enable fast exact inference (see e.g., (Darwiche 2009)).
Exercises
Exercise 26.1 Causal reasoning in the sprinkler network
Consider the causal network in Figure 26.20. Let Trepresent true and Frepresent false.
a. Suppose I perform a perfect intervention and make the grass wet. What is the probability the sprinkler
is on,p(S=T|do(W=T))?

Exercises
b. Suppose I perform a perfect intervention and make the grass dry. What is the probability the sprinkler
is on,p(S=T|do(W=F))?
c. Suppose I perform a perfect intervention and make the clouds “turn on” (e.g., by seeding them). What
is the probability the sprinkler is on, p(S=T|do(C=T))?

27 Latent variable models for discrete data
27.1 Introduction
In this chapter, we are concerned with latent variable models for discrete data, such as bit vectors,
sequences of categorical variables, count vectors, graph structures, relational data, etc. Thesemodels can be used to analyze voting records, text and document collections, low-intensityimages, movie ratings, etc. However, we will mostly focus on text analysis, and this will bereﬂected in our terminology.
Since we will be dealing with so many different kinds of data, we need some precise notation
to keep things clear. When modeling variable-length sequences of categorical variables (i.e.,symbols or tokens), such as words in a document, we will let y
il∈{1,...,V}represent
the identity of the l’th word in document i,w h e r eVis the number of possible words in
the vocabulary. We assume l=1:Li,w h e r eLiis the (known) length of document i, and
i=1:N,w h e r eN is the number of documents.

We will often ignore the word order, resulting in a bag of words. This can be reduced to
a ﬁxed length vector of counts (a histogram). We will use niv∈{0,1,...,L i}to denote the
number of times word voccurs in document i,f o rv=1:V. Note that the N×Vcount
matrixNis often large but sparse, since we typically have many documents, but most words
do not occur in any given document.
In some cases, we might have multiple different bags of words, e.g., bags of text words and

bags of visual words. These correspond to different “channels” or types of features. We willdenote these by y
irl,f o rr=1:R(the number of responses) and l=1:L ir.I fLir=1,i t
means we have a single token (a bag of length 1); in this case, we just write yir∈{1,...,V r}
for brevity. If every channel is just a single token, we write the ﬁxed-size response vector asy
i,1:R; in this case, the N×Rdesign matrix Ywill not be sparse. For example, in social
science surveys, yircould be the response of person ito ther’th multi-choice question.

Out goal is to build joint probability models of p(yi)orp(ni)using latent variables to capture
the correlations. We will then try to interpret the latent variables, which provide a compressedrepresentation of the data. We provide an overview of some approaches in Section 27.2, beforegoing into more detail in later sections.
Towards the end of the chapter, we will consider modeling graphs and relations, which can

also be represented as sparse discrete matrices. For example, we might want to model the graphof which papers mycite which other papers. We will denote these relations by R, reserving the
symbolYfor any categorical data (e.g., text) associated with the nodes.

946 Chapter27. Latentvariablemodelsfordiscretedata
27.2 Distributed state LVMs for discrete data
In this section, we summarize a variety of possible approaches for constructing models of the
formp(yi,1:Li), for bags of tokens; p(y1:R), for vectors of tokens; and p(ni), for vectors of
integer counts.
27.2.1 Mixture models
The simplest approach is to use a ﬁnite mixture model (Chapter 11). This associates a singlediscrete latent variable, q
i∈{1,...,K}, with every document, where Kis the number of
clusters. We will use a discrete prior, qi∼Cat(π). For variable length documents, we can
deﬁnep(yil|qi=k)=bkv,w h e r ebkvis the probability that cluster kgenerates word v. The
value ofqiis known as a topic, and the vector bkis thek’th topic’s word distribution. That is,
the likelihood has the form
p(yi,1:Li|qi=k)=Li∏
l=1Cat(yil|bk) (27.1)
The induced distribution on the visible data is given by
p(yi,1:Li)=∑
kπk[Li∏
l=1Cat(yil|bk)]
(27.2)

kπk[Li∏
(27.2)
The “generative story” which this encodes is as follows: for document i, pick a topic qifrom
π, call itk, and then for each word l=1:L i, pick a word from bk. We will consider more
sophisticated generative models later in this chapter.
If we have a ﬁxed set of categorical observations, we can use a different topic matrix for each
output variable:
p(yi,1:R|qi=k)=R∏
r=1Cat(yil|b(r)
k) (27.3)
This is an unsupervised analog of naive Bayes classiﬁcation.
We can also model count vectors. If the sum Li=∑

(27.2)
k) (27.3)
vnivis known, we can use a
multinomial:
p(ni|Li,qi=k)=M u (ni|Li,bk) (27.4)
If the sum is unknown, we can use a Poisson class-conditional density to give
p(ni|qi=k)=V∏
v=1Poi(niv|λvk) (27.5)
In this case, Li|qi=k∼Poi(∑
vλvk).

27.2. DistributedstateLVMsfordiscretedata 947
27.2.2 Exponential family PCA
Unfortunately, ﬁnite mixture models are very limited in their expressive power. A more ﬂexible
model is to use a vector of real-valued continuous latent variables, similar to the factor analysis(FA) and PCA models in Chapter 12. In PCA, we use a Gaussian prior of the form p(z
i)=
N(μ,Σ),w h e r ezi∈RK, and a Gaussian likelihood of the form p(yi|zi)=N(Wzi,σ2I).
This method can certainly be applied to discrete or count data. Indeed, the method knownaslatent semantic analysis (LSA)o rlatent semantic indexing (LSI) (Deerwester et al. 1990;
Dumais and Landauer 1997) is exactly equivalent to applying PCA to a term by document countmatrix.
A better method for modeling categorical data is to use a multinoulli or multinomial distribu-
tion. We just have to change the likelihood to
p(y
i,1:Li|zi)=Li∏
l=1Cat(yil|S(Wzi)) (27.6)
whereW∈RV×Kis a weight matrix and Sis the softmax function. If we have a ﬁxed number

p(y
of categorical responses, we can use
p(y1:R|zi)=R∏
r=1Cat(yir|S(Wrzi)) (27.7)
whereWr∈RV×Kis the weight matrix for the r’th response variable. This model is called
categorical PCA , and is illustrated in Figure 27.1(a); see Section 12.4 for further discussion. If we
have counts, we can use a multinomial model
p(ni|Li,zi)=M u (ni|Li,S(Wzi)) (27.8)
or a Poisson model
p(ni|zi)=V∏
v=1Poi(niv|exp(wT
v,:zi)) (27.9)
All of these models are examples of exponential family PCA orePCA(Collins et al. 2002;
Mohamed et al. 2008), which is an unsupervised analog of GLMs. The corresponding induceddistribution on the visible variables has the form
p(y

p(y
i,1:Li)=∫[Li∏
l=1p(yil|zi,W)]
N(zi|μ,Σ)dzi (27.10)
Fitting this model is tricky, due to the lack of conjugacy. (Collins et al. 2002) proposed acoordinate ascent method that alternates between estimating the z
iandW. This can be
regarded as a degenerate version of EM, that computes a point estimate of ziin the E step. The

p(y
problem with the degenerate approach is that it is very prone to overﬁtting, since the numberof latent variables is proportional to the number of datacases (Welling et al. 2008). A true EMalgorithm would marginalize out the latent variables z
i. A way to do this for categorical PCA,
using variational EM, is discussed in Section 12.4. For more general models, one can use MCMC(Mohamed et al. 2008).

948 Chapter27. Latentvariablemodelsfordiscretedata
γW1,K,V WR,K,V...yi,1 yi,R ...zi
Nμ Σ
(a)BK,Vni,1...ni,Vπiα
Li
N
(b)
Figure 27.1 Two LVMs for discrete data. Circles are scalar nodes, ellipses are vector nodes, squares are
matrix nodes. (a) Categorical PCA. (b) Multinomial PCA.
27.2.3 LDA and mPCA
In ePCA, the quantity Wzirepresents the natural parameters of the exponential family. Some-
times it is more convenient to use the dual parameters. For example, for the multinomial, the
dual parameter is the probability vector, whereas the natural parameter is the vector of log odds.
If we want to use the dual parameters, we need to constrain the latent variables so they live
in the appropriate parameter space. In the case of categorical data, we will need to ensure thelatent vector lives in S
K, theK-dimensional probability simplex. To avoid confusion with ePCA,
we will denote such a latent vector by πi. In this case, the natural prior for the latent variables

is the Dirichlet, πi∼Dir(α). Typically we set α=α1K.I fw es e t α≪1, we encourage πi
to be sparse, as shown in Figure 2.14.
When we have a count vector whose total sum is known, the likelihood is given by
p(ni|Li,πi)=M u (ni|Li,Bπi) (27.11)
This model is called multinomial PCA ormPCA(Buntine 2002; Buntine and Jakulin 2004,
2006). See Figure 27.1(b). Since we are assuming niv=∑
kbvkπiv, this can be seen as a form
of matrix factorization for the count matrix. Note that we use bv,kto denote the parameter

vector, rather than wv,k, since we impose the constraints that 0≤bv,k≤1and∑
vbv,k=1.
The corresponding marginal distribution has the form
p(ni|Li)=∫
Mu(ni|Li,Bπi)Dir(π i|α)dπi (27.12)
Unfortunately, this integral cannot be computed analytically.
If we have a variable length sequence (of known length), we can use
p(yi,1:Li|πi)=Li∏
l=1Cat(yil|Bπi) (27.13)

27.2. DistributedstateLVMsfordiscretedata 949
This is called latent Dirichlet allocation orLDA(Blei et al. 2003), and will be described in
much greater detail below. LDA can be thought of as a probabilistic extension of LSA, where the
latent quantities πikare non-negative and sum to one. By contrast, in LSA, zikcan be negative
which makes interpetation difficult.
A predecessor to LDA, known as probabilistic latent semantic indexing orPLSI(Hofmann
1999), uses the same model but computes a point estimate of πifor each document (similar to
ePCA), rather than integrating it out. Thus in PLSI, there is no prior for πi.
We can modify LDA to handle a ﬁxed number of different categorical responses as follows:
p(yi,1:R|πi)=R∏
r=1Cat(yil|B(r)πi) (27.14)
This has been called the user rating proﬁle (URP) model (Marlin 2003), and the simplex factor
model(Bhattacharya and Dunson 2011).
27.2.4 GaP model and non-negative matrix factorization

Now consider modeling count vectors where we do not constrain the sum to be observed. Inthis case, the latent variables just need to be non-negative, so we will denote them by z
+
i. This
can be ensured by using a prior of the form
p(z+i)=K∏
k=1Ga(z+
ik|αk,βk) (27.15)
p(ni|z+i)=V∏
v=1Poi(niv|bT
v,:z+
i) (27.16)
This is called the GaP(Gamma-Poisson) model (Canny 2004). See Figure 27.2(a).
In (Buntine and Jakulin 2006), it is shown that the GaP model, when conditioned on a ﬁxed

+
i. This
p(z+i)=K∏
k=1Ga(z+
v,:z+
i) (27.16)
Li, reduces to the mPCA model. This follows since a set of Poisson random variables, when
conditioned on their sum, becomes a multinomial distribution (see e.g., (Ross 1989)).
If we set αk=βk=0in the GaP model, we recover a method known as non-negative
matrix factorization orNMF(Lee and Seung 2001), as shown in (Buntine and Jakulin 2006).
NMF is not a probabilistic generative model, since it does not specify a proper prior for z+i.

Furthermore, the algorithm proposed in (Lee and Seung 2001) is another degenerate EM algo-
rithm, so suffers from overﬁtting. Some procedures to ﬁt the GaP model, which overcome theseproblems, are given in (Buntine and Jakulin 2006).
To encourage z
+
ito be sparse, we can modify the prior to be a spike-and-Gamma type prior
p(z+
ik)=ρkI(z+
ik=0 )+( 1−ρk)Ga(z+
ik|αk,βk) (27.17)
whereρkis the probability of the spike at 0. This is called the conditional Gamma Poisson
model (Buntine and Jakulin 2006). It is simple to modify Gibbs sampling to handle this kind of

+
p(z+
prior, although we will not go into detail here.

950 Chapter27. Latentvariablemodelsfordiscretedata
γBni,1...ni,Vz+
i,1...z+
i,K
Nα1β1
...αKβK
(a)γByilqil
Liπi
Nα
(b)
Figure 27.2 (a) Gaussian-Poisson (GAP) model. (b) Latent Dirichlet allocation (LDA) model.
27.3 Latent Dirichlet allocation (LDA)
In this section, we explain the latent Dirichlet allocation orLDA(Blei et al. 2003) model in
detail.
27.3.1 Basics
In a mixture of multinoullis, every document is assigned to a single topic, qi∈{1,...,K},
drawn from a global distribution π. In LDA, every word is assigned to its own topic, qil∈
{1,...,K}, drawn from a document-speciﬁc distribution πi. Since a document belongs to a
distribution over topics, rather than a single topic, the model is called an admixture mixture
ormixed membership model (Erosheva et al. 2004). This model has many other applications
beyond text analysis, e.g., genetics (Pritchard et al. 2000), health science (Erosheva et al. 2007),
social network analysis (Airoldi et al. 2008), etc.

Adding conjugate priors to the parameters, the full model is as follows:1
πi|α∼Dir(α1K) (27.18)
qil|πi∼Cat(π i) (27.19)
bk|γ∼Dir(γ1V) (27.20)
yil|qil=k,B∼Cat(b k) (27.21)
This is illustrated in Figure 27.2(b). We can marginalize out the qivariables, thereby creating a
1. Our notation is similar to the one we use elsewhere in this book, but is different from that used by most LDA papers.
They typically use wndfor the identity of word nin document d,zndto represent the discrete indicator, θdas the
continuous latent vector for document d, andβkas thek’th topic vector.

27.3. LatentDirichletallocation(LDA) 951
P(word3)P(word1)
01
1
1P(word2)=   observed
document
=   generated
document=   topic
Figure 27.3 Geometric interpretation of LDA. We have K=2topics and V=3words. Each document
(white dots), and each topic (black dots), is a point in the 3d simplex. Source: Figure 5 of (Steyvers and
Griffiths 2007). Used with kind permission of Tom Griffiths.
direct arc from πitoyil, with the following CPD:
p(yil=v|πi)=∑
kp(yil=v|qil=k)p(qil=k)=∑
kπikbkv (27.22)
As we mentioned in the introduction, this is very similar to the multinomial PCA model proposed
in (Buntine 2002), which in turn is closely related to categorical PCA, GaP, NMF, etc.
LDA has an interesting geometric interpretation. Each vector bkdeﬁnes a distribution over
Vwords; each kis known as a topic. Each document vector πideﬁnes a distribution over K

topics. So we model each document as an admixture over topics. Equivalently, we can thinkof LDA as a form of dimensionality reduction (assuming K<V, as is usually the case), where
we project a point in the V-dimensional simplex (a normalized document count vector x
i) onto
theK-dimensional simplex. This is illustrated in Figure 27.3, where we have V=3words and
K=2topics. The observed documents (which live in the 3d simplex) are approximated as
living on a 2d simplex spanned by the 2 topic vectors, each of which lives in the 3d simplex.

i) onto
One advantage of using the simplex as our latent space rather than Euclidean space is that
the simplex can handle ambiguity. This is importance since in natural language, words can oftenhave multiple meanings, a phenomomen known as polysemy. For example, “play” might refer
to a verb (e.g., “to play ball” or “to play the coronet”), or to a noun (e.g., “Shakespeare’s play”).In LDA, we can have multiple topics, each of which can generate the word “play”, as shown inFigure 27.4, reﬂecting this ambiguity.
Given word lin document i, we can compute p(q
il=k|yi,θ), and thus infer its most likely

topic. By looking at the word in isolation, it might be hard to know what sense of the word ismeant, but we can disambiguate this by looking at other words in the document. In particular,givenx
i, we can infer the topic distribution πifor the document; this acts as a prior for
disambiguating qil. This is illustrated in Figure 27.5, where we show three documents from the
TASA corpus.2In the ﬁrst document, there are a variety of music related words, which suggest

2. The TASA corpus is a collection of 37,000 high-school level English documents, comprising over 10 million words,

952 Chapter27. Latentvariablemodelsfordiscretedata
word prob. word prob. word prob.
MUSIC .090 LITERATURE .031 PLAY .136
DANCE .034 POEM .028 BALL .129
SONG .033 POETRY .027 GAME .065
PLAY .030 POET .020 PLAYING .042
SING .026 PLAYS .019 HIT .032
SINGING .026 POEMS .019 PLAYED .031
BAND .026 PLAY .015 BASEBALL .027
PLAYED .023 LITERARY .013 GAMES .025
SANG .022 WRITERS .013 BAT .019
SONGS .021 DRAMA .012 RUN .019
DANCING .020 WROTE .012 THROW .016
PIANO .017 POETS .011 BALLS .015
PLAYING .016 WRITER .011 TENNIS .011
RHYTHM .015 SHAKESPEARE .010 HOME .010
ALBERT .013 WRITTEN .009 CATCH .010
MUSICAL .013 STAGE .009 FIELD .010Topic 166 Topic 82 Topic 77
Figure 27.4 Three topics related to the word play. Source: Figure 9 of (Steyvers and Griffiths 2007).
Used with kind permission of Tom Griffiths.
Document #29795 
Bix beiderbecke, at  age060 fifteen207, sat174on the  slope071of a bluff055 overlooking027the mississippi137 river137.H e

was listening077to music077 coming009from a  passing043riverboat. The  music077had already  captured006his heart157
as well as his  ear119. It was jazz077. Bix beiderbecke had already had  music077 lessons077.H e showed002 promise134on
the piano077, and his  parents035 hoped268he might  consider118becoming a  concert077 pianist077. But bix was
interested268in another  kind050of music077.H e wanted268to play077the cornet. And he wanted268to play077 jazz077...  
Document #1883 There is a  simple

050 reason106why there are so few  periods078of really great  theater082in our whole  western046world.
Too many  things300have to come right at the very same time. The dramatists must have the right  actors082,t h e
actors082must have the right playhouses, the playhouses must have the right  audiences082.W em u s t  remember288that
plays082 exist143to be performed077,n o t merely050to be read254. ( even when you  read254a play082to yourself,  try288to

perform062it, to  put174it on a stage078, as you go along.) as  soon028as a play082has to be  performed082, then some 
kind126of theatrical082...
Document #21359 
Jim296has a  game166 book254. Jim296 reads254the book254. Jim296 sees081a game166for one.  Jim296 plays166the game166.
Jim296 likes081the game166for one. The  game166 book254 helps081 jim296. Don180 comes040into the  house038. Don180and
jim296 read254the game166 book254.T h e  boys020see a game166for two. The two  boys020 play166the game166.T h e

boys020 play166the game166for two. The  boys020like the  game166. Meg282 comes040into the house282. Meg282and
don180and jim296 read254the book254.T h e ys e ea  game166for three.  Meg282and don180and jim296 play166the game166.
They play166...
Figure 27.5 Three documents from the TASA corpus containing different senses of the word play.G r a y e d
out words were ignored by the model, because they correspond to uninteresting stop words (such as “and”,

“the”, etc.) or very low frequency words. Source: Figure 10 of (Steyvers and Griffiths 2007). Used withkind permission of Tom Griffiths.

27.3. LatentDirichletallocation(LDA) 953
πiwill put most of its mass on the music topic (number 77); this in turn makes the music
interpretation of “play” the most likely, as shown by the superscript. The second document
interprets play in the theatrical sense, and the third in the sports sense. Note that is crucialthatπ
ibe a latent variable, so information can ﬂow between the qil’s, thus enabling local
disambiguation to use the full set of words.
27.3.2 Unsupervised discovery of topics
One of the main purposes of LDA is discover topics in a large collection or corpusof docu-

ments (see Figure 27.12 for an example). Unfortunately, since the model is unidentiﬁable, theinterpertation of the topics can be difficult (Chang et al. 2009).. One approach, known as la-beled LDA (Ramage et al. 2009), exploits the existence of tags on documents as a way to ensureidentiﬁability. In particular, it forces the topics to correspond to the tags, and then it learns adistribution over words for each tag. This can make the results easier to interpret.
27.3.3 Quantitatively evaluating LDA as a language model

In order to evaluate LDA quantitatively, we can treat it as a language model , i.e., a probability
distribution over sequences of words. Of course, it is not a very good language model, since itignores word order and just looks at single words (unigrams), but it is interesting to compareLDA to other unigram-based models, such as mixtures of multinoullis, and pLSI. Such simplelanguage models are sometimes useful for information retrieval purposes. The standard way tomeasure the quality of a language model is to use perplexity, which we now deﬁne below.
27.3.3.1 Perplexity
Theperplexity of language model qgiven a stochastic process
3pis deﬁned as
perplexity( p,q)≜2H(p,q)(27.23)
whereH(p,q)is the cross-entropy of the two stochastic processes, deﬁned as
H(p,q)≜lim
N→∞−1
N∑
y1:Np(y1:N)logq(y1:N) (27.24)
The cross entropy (and hence perplexity) is minimized if q=p; in this case, the model can
predict as well as the “true” distribution.

N→∞−1
N∑
We can approximate the stochastic process by using a single long test sequence (composed
of multiple documents and multiple sentences, complete with end-of-sentence markers), callity
∗
1:N. (This approximation becomes more and more accurate as the sequence gets longer,
provided the process is stationary and ergodic (Cover and Thomas 2006).) Deﬁne the empiricaldistribution (an approximation to the stochastic process) as
p
emp(y1:N)=δy∗
1:N(y1:N) (27.25)
collated by a company formerly known as Touchstone Applied Science Associates, but now known as Questar Assessment
Incwww.questarai .com.

∗
p
3. A stochastic process is one which can deﬁne a joint distribution over an arbitrary number of random variables. Wecan think of natural language as a stochastic process, since it can generate an inﬁnite stream of words.

954 Chapter27. Latentvariablemodelsfordiscretedata
In this case, the cross-entropy becomes
H(pemp,q)=−1
Nlogq(y∗
1:N) (27.26)
and the perplexity becomes
perplexity( pemp,q)=2H(pemp,q)=q(y∗
1:N)−1/N=N√N∏
i=11
q(y∗
i|y∗
1:i−1)(27.27)
We see that this is the geometric mean of the inverse predictive probabilities, which is the usual
deﬁnition of perplexity (Jurafsky and Martin 2008, p96).
In the case of unigram models, the cross entropy term is given by
H=−1
NN∑
i=11
LiLi∑
l=1logq(yil) (27.28)
whereNis the number of documents and Liis the number of words in document i. Hence
the perplexity of model qis given by
perplexity( pemp,p)=e x p(
−1
NN∑
i=11
LiLi∑
l=1logq(yil))
(27.29)
Intuitively, perplexity mesures the weighted average branching factor of the model’s predic-
tive distribution. Suppose the model predicts that each symbol (letter, word, whatever) is equallylikely, so p(y
i|y1:i−1)=1/K. Then the perplexity is ((1/K)N)−1/N=K. If some symbols

H=−1
NN∑
i=11
LiLi∑
−1
NN∑
i=11
LiLi∑
(27.29)
are more likely than others, and the model correctly reﬂects this, its perplexity will be lowerthanK. Of course, H(p,p)=H (p)≤H(p,q), so we can never reduce the perplexity below
the entropy of the underlying stochastic process.
27.3.3.2 Perplexity of LDA
The key quantity is p(v), the predictive distribution of the model over possible words. (It is

implicitly conditioned on the training set.) For LDA, this can be approximated by plugging inB(e.g., the posterior mean estimate) and approximately integrating out qusing mean ﬁeld
inference (see (Wallach et al. 2009) for a more accurate way to approximate the predictivelikelihood).
In Figure 27.6, we compare LDA to several other simple unigram models, namely MAP estima-
tion of a multinoulli, MAP estimation of a mixture of multinoullis, and pLSI. (When performingMAP estimation, the same Dirichlet prior on Bwas used as in the LDA model.) The metric

is perplexity, as in Equation 27.29, and the data is a subset of the TREC AP corpus containing16,333 newswire articles with 23,075 unique terms. We see that LDA signiﬁcantly outperformsthese other methods.

27.3. LatentDirichletallocation(LDA) 955
0 20 40 60 80 100 120 140 160 180 2002500300035004000450050005500600065007000
Number of TopicsPerplexity
  
Unigram
Mixtures of Unigrams
LDA
Fold in pLSI
Figure 27.6 Perplexity vs number of topics on the TREC AP corpus for various language models. Based
on Figure 9 of (Blei et al. 2003). Figure generated by bleiLDAperplexityPlot .
γb1...bKy1,1...y1,L1q1,1...q1,L1π1
......... ...
yN,1...yN,LNqN,1...qN,LNπNα
(a)γy1,1...y1,L1q1,1...q1,L1
......yN,1...yN,LNqN,1...qN,LNα
(b)
Figure 27.7 (a) LDA unrolled for Ndocuments. (b) Collapsed LDA, where we integrate out the πiand
thebk.
27.3.4 Fitting using (collapsed) Gibbs sampling
It is straightforward to derive a Gibbs sampling algorithm for LDA. The full conditionals are as
follows:
p(qil=k|·)∝exp[logπik+logbk,xil] (27.30)
p(πi|·) = Dir({αk+∑
lI(zil=k)}) (27.31)
p(bk|·) = Dir({γv+∑
i∑
lI(xil=v,zil=k)}) (27.32)
However, one can get better performance by analytically integrating out the πi’s and the bk’s,

956 Chapter27. Latentvariablemodelsfordiscretedata
both of which have a Dirichlet distribution, and just sampling the discrete qil’s. This approach
was ﬁrst suggested in (Griffiths and Steyvers 2004), and is an example of collapsed Gibbs
sampling. Figure 27.7(b) shows that now all the qilvariables are fully correlated. However, we
can sample them one at a time, as we explain below.
First, we need some notation. Let civk=∑Li
l=1I(qil=k,yil=v)be the number of times
wordvis assigned to topic kin document i.L e tcik=∑
vcivkbe the number of times any
word from document ihas been assigned to topic k.L e tcvk=∑
icivkbe the number of times
wordvhas been assigned to topic kin any document. Let niv=∑
kcivkbe the number of
times word voccurs in document i; this is observed. Let ck=∑
vcvkbe the number of words
assigned to topic k. Finally, let Li=∑
kcikbe the number of words in document i; this is
observed.
We can now derive the marginal prior. By applying Equation 5.24, one can show that
p(q|α)=∏
i∫[Li∏
l=1Cat(qil|πi)]

observed.
p(q|α)=∏
i∫[Li∏
Dir(πi|α1K)dπi (27.33)
=(Γ(Kα)
Γ(α)K)NN∏
i=1∏K
k=1Γ(cik+α)
Γ(Li+Kα)(27.34)
By similar reasoning, one can show
p(y|q,γ)=∏
k∫⎡
⎣∏
il:qil=kCat(yil|bk)⎤⎦Dir(b
k|γ1V)dbk (27.35)
=(Γ(Vβ)
Γ(β)V)KK∏
k=1∏V
v=1Γ(cvk+β)
Γ(ck+Vβ)(27.36)
From the above equations, and using the fact that Γ(x+1)/Γ(x)=x, we can derive the full
conditional for p(qil|q−i,l). Deﬁnec−
ivkto be the same as civkexcept it is compute by summing
over all locations in document iexcept for qil. Also, let yil=v. Then
p(qi,l=k|q−i,l,y,α,γ)∝c−v,k+γ

=(Γ(Kα)
Γ(α)K)NN∏
i=1∏K
k∫⎡
⎣∏
=(Γ(Vβ)
Γ(β)V)KK∏
k=1∏V
c−k+Vγc−i,k+α
Li+Kα(27.37)
We see that a word in a document is assigned to a topic based both on how often that word is
generated by the topic (ﬁrst term), and also on how often that topic is used in that document(second term).
Given Equation 27.37, we can implement the collapsed Gibbs sampler as follows. We randomly
assign a topic to each word, q
il∈{1,...,K}. We can then sample a new topic as follows: for

a given word in the corpus, decrement the relevant counts, based on the topic assigned to thecurrent word; draw a new topic from Equation 27.37, update the count matrices; and repeat.This algorithm can be made efficient since the count matrices are very sparse.
27.3.5 Example
This process is illustrated in Figure 27.8 on a small example with two topics, and ﬁve words.The left part of the ﬁgure illustrates 16 documents that were sampled from the LDA model using

27.3. LatentDirichletallocation(LDA) 957
River Stream Bank Money Loan
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
(a)River Stream Bank Money Loan
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
(b)
Figure 27.8 Illustration of (collapsed) Gibbs sampling applied to a small LDA example. There are N=1 6
documents, each containing a variable number of words drawn from a vocabulary of V=5words, There
are two topics. A white dot means word the word is assigned to topic 1, a black dot means the word is
assigned to topic 2. (a) The initial random assignment of states. (b) A sample from the posterior after 64steps of Gibbs sampling. Source: Figure 7 of (Steyvers and Griffiths 2007). Used with kind permission ofTom Griffiths.
p(money|k=1 )=p(loan| k=1 )=p(bank| k=1 )=1 /3andp(river|k=2 )=p(stream| k=
2) =p(bank|k=2 )=1 /3. For example, we see that the ﬁrst document contains the word
“bank” 4 times (indicated by the four dots in row 1 of the “bank” column), as well as various

other ﬁnancial terms. The right part of the ﬁgure shows the state of the Gibbs sampler after64 iterations. The “correct” topic has been assigned to each token in most cases. For example,in document 1, we see that the word “bank” has been correctly assigned to the ﬁnancial topic,based on the presence of the words “money” and “loan”. The posterior mean estimate of theparameters is given by ˆp(money|k=1 )=0 .32,ˆp(loan|k=1 )=0 .29,ˆp(bank|k=1 )=
0.39,ˆp(river|k=2 )=0 .25,ˆp(stream|k=2 )=0 .4, andˆp(bank|k=2 )=0 .35, w
 hich is

 hich is
impressively accurate, given that there are only 16 training examples.
27.3.6 Fitting using batch variational inference
A faster alternative to MCMC is to use variational EM. (We cannot use exact EM since exactinference of π
iandqiis intractable.) We give the details below.
27.3.6.1 Sequence version
Following (Blei et al. 2003), we will use a fully factorized (mean ﬁeld) approximation of the form
q(πi,qi) = Dir(πi|˜πi)∏
lCat(qil|˜qil) (27.38)
We will follow the usual mean ﬁeld recipe. For q(qil), we use Bayes’ rule, but where we need to
take expectations over the prior:
˜qilk∝byi,l,kexp(E[logπik]) (27.39)
where
E[logπik]=ψk(˜πi.)≜Ψ(˜πik)−Ψ(∑
k′˜πik′) (27.40)

958 Chapter27. Latentvariablemodelsfordiscretedata
whereΨis the digamma function. The update for q(πi)is obtained by adding up the expected
counts:
˜πik=αk+∑
l˜qilk (27.41)
The M step is obtained by adding up the expected counts and normalizing:
ˆbvk∝γv+N∑
i=1Li∑
l=1˜qilkI(yil=v) (27.42)
27.3.6.2 Count version
Note that the E step takes O((∑
iLi)VK)space to store the ˜qilk. It is much more space
efficient to perform inference in the mPCA version of the model, which works with counts; these
only take O(NVK)space, which is a big savings if documents are long. (By contrast, the
collapsed Gibbs sampler must work explicitly with the qilvariables.)
We will focus on approximating p(πi,ci|ni,Li), where we write cias shorthand for ci...W e
will again use a fully factorized (mean ﬁeld) approximation of the form
q(πi,ci)=D i r (πi|˜πi)∏
vMu(civ.|niv,˜civ.) (27.43)
The new E step becomes
˜πik=αk+∑
vniv˜civk (27.44)
˜civk∝bvkexp(E[logπik]) (27.45)
The new M step becomes
ˆbvk∝γv+∑
iniv˜civk (27.46)
27.3.6.3 VB version

˜πik=αk+∑
ˆbvk∝γv+∑
We now modify the algorithm to use VB instead of EM, so that we infer the parameters aswell as the latent variables. There are two advantages to this. First, by setting γ≪1, VB will
encourage Bto be sparse (as in Section 21.6.1.6). Second, we will be able to generalize this to
the online learning setting, as we discuss below.
Our new posterior approximation becomes
q(π
i,ci,B)=D i r ( πi|˜πi)∏
vMu(civ.|niv,˜civ.)∏
kDir(b.k|˜b.k) (27.47)
The update for ˜civkchanges, to the following:
˜civk∝exp(E[logbvk]+E[logπik]) (27.48)

27.3. LatentDirichletallocation(LDA) 959
Algorithm 27.1: Batch VB for LDA
1Input:niv,K,αk,γv;
2Estimate˜bvkusing EM for multinomial mixtures;
3Initialize counts niv;
4whilenotconverged do
5/ /Es t e p;
6svk=0// expected sufficient statistics;
7foreachdocument i=1:Ndo
8 (˜πi,˜ci)=Estep(ni,˜B,α);
9 svk+=niv˜civk;
10/ /Ms t e p;
11foreachtopic k=1:Kdo
12 ˜bvk=γv+svk;
13function(˜πi,˜ci)=Estep(ni,˜B,α);
14Initialize ˜πik=αk;
15repeat
16˜πold
i.=˜πi.,˜πik=αk;
17foreachword v=1:Vdo
18 foreachtopic k=1:Kdo
19 ˜civk=e x p(
ψk(˜bv.)+ψk(˜πold
i.))
;
20 ˜civ.=normalize( ˜civ.);
21 ˜πik+=niv˜civk
22until1
K∑
k|˜πik−˜πold
ik|<thresh;
Also, the M step becomes
˜bvk=γv+∑
i˜civk (27.49)
No normalization is required, since we are just updating the pseudcounts. The overall algorithm
is summarized in Algorithm 22.
27.3.7 Fitting using online variational inference
In the bathc version, the E step clearly takes O(NKVT)time, where Tis the number of

16˜πold
i.))
;
22until1
K∑
˜bvk=γv+∑
mean ﬁeld updates (typically T∼5). This can be slow if we have many documents. This can
be reduced by using stochastic gradient descent (Section 8.5.2) to perform online variationalinference, as we now explain.
We can derive an online version, following (Hoffman et al. 2010). We perform an E step in the
usual way. We then compute the variational parameters for Btreating the expected sufficient
statistics from the single data case as if the whole data set had those statistics. Finally, we make

960 Chapter27. Latentvariablemodelsfordiscretedata
Algorithm 27.2: Online variational Bayes for LDA
1Input:niv,K,αk,γv,τ0,κ;
2Initialize˜bvkrandomly;
3fort=1:∞do
4Set step size ρt=(τ0+t)−κ;
5Pick document i=i(t);;
6(˜πi,˜ci)=Estep(ni,˜B,α);
7˜bnew
vk=γv+Nniv˜civk;
8˜bvk=( 1−ρt)˜bvk+ρt˜bnewvk;
Documents seen (log scale)Perplexity
600650700750800850900
103.5104104.5105105.5106106.5Batch 98KOnline 98K
Online 3.3M
Figure 27.9 Test perplexity vs number of training documents for batch and online VB-LDA. From Figure
1 of (Hoffman et al. 2010). Used with kind permission of David Blei.
a partial update for the variational parameters for B, putting weight ρton the new estimate
and weight 1−ρton the old estimate. The step size ρtdecays over time, as in Equation 8.83.
The overall algorithm is summarized in Algorithm 3. In practice, we should use mini-batches,
as explained in Section 8.5.2.3. In (Hoffman et al. 2010), they used a batch of size 256–4096.

Figure 27.9 plots the perplexity on a test set of size 1000 vs number of analyzed documents (E
steps), where the data is drawn from (English) Wikipedia. The ﬁgure shows that online variationalinference is much faster than offline inference, yet produces similar results.
27.3.8 Determining the number of topics
Choosing K, the number of topics, is a standard model selection problem. Here are some
approaches that have been taken:
• Use annealed importance sampling (Section 24.6.2) to approximate the evidence (Wallach
et al. 2009).

• Cross validation, using the log likelihood on a test set.

27.4. ExtensionsofLDA 961
• Use the variational lower bound as a proxy for logp(D|K).
• Use non-parametric Bayesian methods (Teh et al. 2006).
27.4 Extensions of LDA
Many extensions of LDA have been proposed since the ﬁrst paper came out in 2003. We brieﬂy
discuss a few of these below.
27.4.1 Correlated topic model
One weakness of LDA is that it cannot capture correlation between topics. For example, if adocument has the “business” topic, it is reasonable to expect the “ﬁnance” topic to co-occcur.The source of the problem is the use of a Dirichlet prior for π
i. The problem with the Dirichelt
it that it is characterized by just a mean vector and a strength parameter, but its covariance isﬁxed (Σ
ij=−αiαj), rather than being a free parameter.
One way around this is to replace the Dirichlet prior with the logistic normal distribution, as
in categorical PCA (Section 27.2.2). The model becomes
bk|γ∼Dir(γ1V) (27.50)
zi∼N(μ,Σ) (27.51)
πi|zi=S(zi) (27.52)
qil|πi∼Cat(π i) (27.53)
yil|qil=k,B∼Cat(b k) (27.54)

This is known as the correlated topic model (Blei and Lafferty 2007). This is very similar to
categorical PCA, but slightly different. To see the difference, let us marginalize out the qiland
πi. Then in the CTM we have
yil∼Cat(BS(zi)) (27.55)
whereBis a stochastic matrix. By contrast, in catPCA we have
yil∼Cat(S(Wzi)) (27.56)
whereWis an unconstrained matrix.
Fitting this model is tricky, since the prior for πiis no longer conjugate to the multinomial
likelihood for qil. However, we can use any of the variational methods in Section 21.8.1.1, where

we discussed Bayesian multiclass logistic regression. In the CTM case, things are even hardersince the categorical response variables q
iare hidden, but we can handle this by using an
additional mean ﬁeld approximation. See (Blei and Lafferty 2007) for details.
Having ﬁt the model, one can then convert ˆΣto a sparse precision matrix ˆΣ−1by pruning

low-strength edges, to get a sparse Gaussian graphical model. This allows you to visualize thecorrelation between topics. Figure 27.10 shows the result of applying this procedure to articlesfromSciencemagazine, from 1990-1999. (This corpus contains 16,351 documents, and 5.7M words
(19,088 of them unique), after stop-word and low-frequency removal.) Nodes represent topics,with the top 5 words per topic listed inside. The font size reﬂects the overall prevalence of thetopic in the corpus. Edges represent signiﬁcant elements of the precision matrix.

962 Chapter27. Latentvariablemodelsfordiscretedata
wild type
mutant
mutations
mutants
mutation
plants
plant
gene
genes
arabidopsisp53
cell cycle
activity
cyclin
regulation
amino acids
cdna
sequence
isolated
protein
gene
disease
mutations
families
mutationrna
dna
rna polymerase
cleavage
site
cells
cell
expression
cell lines
bone marrowunited states
women
universities
students
educationscience
scientists
says
research
peopleresearch
funding
support
nih
programsurface
tip
image
sample
device
laser
optical
light
electrons
quantummaterials
organic
polymer
polymers
molecules
volcanic
deposits
magma
eruption
volcanismmantle
crust
upper mantle
meteorites
ratios
earthquake
earthquakes
fault
images
dataancient
found
impact
million years ago
africa
climate
ocean
ice
changes
climate changecells
proteins
researchers
protein
foundpatients
disease
treatment
drugs
clinicalgenetic
population
populations
variationfossil record
birds
fossils
dinosaurs
fossilsequence
sequences
genome
dna
sequencing
bacteria
bacterial

electrons
organic
polymer
polymers
molecules
volcanic
deposits
magma
eruption
crust
ratios
fault
images
found
impact
africa
climate
ocean
ice
changes
proteins
protein
disease
treatment
drugs
birds
fossils
dinosaurs
sequences
genome
dna
bacteria
bacterial
host
resistance
parasitedevelopment
embryos
drosophila
genes
expressionspecies
forest
forests
ecosystemssynapses
ltp
glutamate
synaptic
neuronsneurons
stimulus
motor
visual
cortical
ozone
measurements
stratosphere
concentrationssun
solar wind
earth
planets
planet
co2
carbon
carbon dioxide
methane
waterreceptor
receptors
ligand
ligands
apoptosisproteins
protein
binding
domain
domainsactivated
tyrosine phosphorylation
activation
phosphorylation
kinase
magnetic
magnetic ﬁeld
spin

host
embryos
genes
forest
forests
ltp
glutamate
synaptic
stimulus
motor
visual
cortical
ozone
solar wind
earth
planets
planet
co2
carbon
methane
receptors
ligand
ligands
protein
binding
domain
kinase
magnetic
spin
superconductivity
superconductingphysicists
particles
physics
particle
experimentsurface
liquid
surfaces
ﬂuid
model reaction
reactions
molecule
molecules
transition stateenzyme
enzymes
iron
active site
reduction
pressure
high pressure
pressures
core
inner corebrain
memory
subjects
left
task
computer
problem
computers
problems
stars
astronomers
universe
galaxies
galaxy
virus
hiv
aids
infection
virusesmice
antigen
t cells
antigens
immune response

kinase
magnetic
spin
particles
physics
particle
liquid
surfaces
ﬂuid
reactions
molecule
molecules
enzymes
iron
reduction
pressure
pressures
core
memory
subjects
left
task
computer
problem
computers
problems
stars
universe
galaxies
galaxy
virus
hiv
aids
infection
antigen
t cells
antigens
Figure 27.10 Output of the correlated topic model (with K=5 0topics) when applied to articles from
Science. Nodes represent topics, with the 5 most probable phrases from each topic shown inside. Fontsize reﬂects overall prevalence of the topic. See http://www .cs.cmu.edu/~lemur/science/ for an
interactive version of this model with 100 topics. Source: Figure 2 of (Blei and Lafferty 2007). Used withkind permission of David Blei.
27.4.2 Dynamic topic model

antigen
t cells
antigens
In LDA, the topics (distributions over words) are assumed to be static. In some cases, it makes
sense to allow these distributions to evolve smoothly over time. For example, an article mightuse the topic “neuroscience”, but if it was written in the 1900s, it is more likely to use wordslike “nerve”, whereas if it was written in the 2000s, it is more likely to use words like “calciumreceptor” (this reﬂects the general trend of neuroscience towards molecular biology).

One way to model this is use a dynamic logistic normal model, as illustrated in Figure 27.11.
In particular, we assume the topic distributions evolve according to a Gaussian random walk,and then we map these Gaussian vectors to probabilities via the softmax function:
b
t,k|bt−1,k∼N(bt−1,k,σ21V) (27.57)
πt
i∼Dir(α1K) (27.58)
qt
il|πti∼Cat(πti) (27.59)
yt
il|qt
il=k,Bt∼Cat(S(btk)) (27.60)
This is known as a dynamic topic model (Blei and Lafferty 2006b).

27.4. ExtensionsofLDA 963
bt−1
kyt−1
ilqt−1
ilπt−1
i
N
bt
kyt
ilqt
ilπt
i
N
bt+1
kyt+1
ilqt+1
ilπt+1
i
N
Kα
Figure 27.11 The dynamic topic model.
One can perform approximate infernece in this model using a structured mean ﬁeld method
(Section 21.4), that exploits the Kalman smoothing algorithm (Section 18.3.1) to perform exact
inference on the linear-Gaussian chain between the bt,knodes (see (Blei and Lafferty 2006b) for
details).
Figure 27.12 illustrates a typical output of the system when applied to 100 years of articles
fromScience. On the top, we visualize the top 10 words from a speciﬁc topic (which seems to
be related to neuroscience) after 10 year intervals. On the bottom left, we plot the probabilityof some speciﬁc words belonging to this topic. On the bottom right, we list the titles of somearticles that contained this topic.
One interesting application of this model is to perform temporally-corrected document re-

details).
trieval. That is, suppose we look for documents about the inheritance of disease. Modernarticles will use words like “DNA”, but older articles (before the discovery of DNA) may use otherterms such as “heritable unit”. But both articles are likely to use the same topics. Similar ideascan be used to perform cross-language information retrieval, see e.g., (Cimiano et al. 2009).
27.4.3 LDA-HMM

The LDA model assumes words are exchangeable, which is clearly not true. A simple wayto model sequential dependence between words is to use a hidden Markov model or HMM.The trouble with HMMs is that they can only model short-range dependencies, so they cannotcapture the overall gist of a document. Hence they can generate syntactically correct sentences(see e.g., Table 17.1). but not semantically plausible ones.
It is possible to combine LDA with HMM to create a model called LDA-HMM (Griffiths et al.

964 Chapter27. Latentvariablemodelsfordiscretedata
"Neuroscience" 1881
brain
movement
action
right
eye
hand
left
muscle
nerve
sound  1890
movement
eye
right
hand
brain
left
action
muscle
sound
experiment 1900
brain
eye
movement
right
left
hand
nerve
vision
sound
muscle  1910
movement
brain
sound
nerve
active
muscle
left
eye
right
nervous  1920
movement
sound
muscle
active
nerve
stimulate
ﬁber
reaction
brain
response  1930
stimulate
muscle
sound
movement
response
nerve
frequency
ﬁber
active
brain  1940
record
nerve
stimulate
response
muscle
electrode
active
brain
ﬁber
potential  1950
respons
record
stimulate
nerve
muscle
active
frequency
electrode
potential
study  1960
response
stimulate
record
condition
active
potential
stimulus
nerve
subject
eye  1970
respons
cell
potential
stimul
neuron
active
nerve
eye
record
abstract  1980
cell
neuron
response
active
brain
stimul
muscle
system
nerve
receptor  1990
cell
channel
neuron
ca2
active
brain
receptor
muscle
respons
current  2000
neuron
active
brain
cell
ﬁg

record
nerve
stimulate
response
muscle
electrode
active
brain
ﬁber
respons
record
stimulate
nerve
muscle
active
frequency
electrode
potential
study  1960
response
stimulate
record
condition
active
potential
stimulus
nerve
subject
eye  1970
respons
cell
potential
stimul
neuron
active
nerve
eye
record
cell
neuron
response
active
brain
stimul
muscle
system
nerve
cell
channel
neuron
ca2
active
brain
receptor
muscle
respons
neuron
active
brain
cell
ﬁg
response
channel
receptor
synapse
signal 
1880 1900 1920 1940 1960 1980 2000ca2
neuronnerve1887 Mental Science
1900 Hemianopsia in Migraine
1912 A Defence of the ``New Phrenology''
1921 The Synchronal Flashing of Fireﬂies
1932 Myoesthesis and Imageless Thought
1943 Acetylcholine and the Physiology of the Nervous System
1952 Brain Waves and Unit Discharge in Cerebral Cortex
1963 Errorless Discrimination Learning in the Pigeon
1974 Temporal Summation of Light by a Vertebrate Visual Receptor

brain
cell
ﬁg
response
channel
receptor
synapse
signal 
1983 Hysteresis in the Force-Calcium Relation in Muscle
1993 GABA-Activated Chloride Channels in Secretory Nerve Endings
Figure 27.12 Part of the output of the dynamic topic model when applied to articles from Science.W e
show the top 10 words for the neuroscience topic over time. We also show the probability of three words
within this topic over time, and some articles that contained this topic. Source: Figure 4 of (Blei andLafferty 2006b). Used with kind permission of David Blei.

2004). This model uses the HMM states to model function or syntactic words, such as “and” or
“however”, and uses the LDA to model content or semantic words, which are harder to predict.There is a distinguished HMM state which speciﬁes when the LDA model should be used togenerate the word; the rest of the time, the HMM generates the word.
More formally, for each document i, the model deﬁnes an HMM with states z
il∈{0,...,C}.
In addition, each document has an LDA model associated with it. If zil=0, we generate word

yilfrom the semantic LDA model, with topic speciﬁed by qil; otherwise we generate word yil
from the syntactic HMM model. The DGM is shown in Figure 27.13. The CPDs are as follows:
p(πi)=D i r ( πi|α1K) (27.61)
p(qil=k|πi)=π ik (27.62)
p(zil=c′|zi,l−1=c)=AHMM(c,c′) (27.63)
p(yil=v|qil=k,zil=c)={BLDA(k,v)ifc=0
BHMM(c,v)ifc>0(27.64)
whereBLDAis the usual topic-word matrix, BHMMis the state-word HMM emission matrix
andAHMMis the state-state HMM transition matrix.
Inference in this model can be done with collapsed Gibbs sampling, analytically integrating

out all the continuous quantities. See (Griffiths et al. 2004) for the details.
The results of applying this model (with K= 200LDA topics and C=2 0HMM states) to the
combined Brown and TASA corpora4are shown in Table 27.1. We see that the HMM generally is
4. The Brown corpus consists of 500 documents and 1,137,466 word tokens, with part-of-speech tags for each token.

27.4. ExtensionsofLDA 965
AHMMBHMMBLDA
zi,l−1 zi,l zi,l+1...yi,l−1 yi,l yi,l+1...qi,l−1 qi,l qi,l+1...πiα
N
Figure 27.13 LDA-HMM model.
1.In contrast to this approach, we study here how the overall network activity can control single cell
parameters such as input resistance, as well as time andspace constants, parameters that are crucial for
excitability andspariotemporal (sic) integration .
The integrated architecture in this paper combines feed forward control anderror feedback adaptive
control using neural networks.
2.In other words, for our proof ofconvergence , we require the softassign algorithm toreturn adoubly
stochastic matrix as *sinkhorn theorem guarantees that it will instead of a matrix which is merely close
to being doubly stochastic based on some reasonable metric.
The aim is to construct a portfolio with a maximal expected return for a given risklevel andtime
horizon while simultaneously obeying *institutional or*legally required constraints.

3.The left graph is the standard experiment the right from atraining with #samples .
The graphGis called the *guest graph, andHis called the host graph.
Figure 27.14 Function and content words in the NIPS corpus, as distinguished by the LDA-HMM model.
Graylevel indicates posterior probability of assignment to LDA component, with black being highest. The

boxed word appears as a function word in one sentence, and as a content word in another sentence.Asterisked words had low frequency, and were treated as a single word type by the model. Source: Figure4 of (Griffiths et al. 2004). Used with kind permission of Tom Griffiths.

966 Chapter27. Latentvariablemodelsfordiscretedata
the the the the the a the the the
blood , , of a the , , ,
, and and , of of of a a
of of of to , , a of in
body a in in in in and and game
heart in land and to water in drink ball
and trees to classes picture is story alcohol and
in tree farmers government ﬁlm and is to team
to with for a image matter to bottle to
is on farm state lens are as in play
blood forest farmers government light water story drugs ball
heart trees land state eye matter stories drug game
pressure forests crops federal lens molecules poem alcohol team
body land farm public image liquid characters people *
lungs soil food local mirror particles poetry drinking baseball
oxygen areas people act eyes gas character person players
vessels park farming states glass solid author effects football
arteries wildlife wheat national object substance poems marijuana player
* area farms laws objects temperature life body ﬁeld
breathing rain corn department lenses changes poet use basketball

the in he * be said can time ,
a for it new have made would way ;
his to you other see used will years (
this on they ﬁrst make came could day :
their with i same do went may part )
these at she great know found had number
your by we good get called must kind
her from there small go do place
my as this little take have
some into who old ﬁnd did
Table 27.1 Upper row: Topics extracted by the LDA model when trained on the combined Brown and

TASA corpora. Middle row: topics extracted by LDA part of LDA-HMM model. Bottom row: topics extracted
by HMM part of LDA-HMM model. Each column represents a single topic/class, and words appear in orderof probability in that topic/class. Since some classes give almost all probability to only a few words, a listis terminated when the words account for 90% of the probability mass. Source: Figure 2 of (Griffiths et al.2004). Used with kind permission of Tom Griffiths.

responsible for syntactic words, and the LDA for semantics words. If we did not have the HMM,
the LDA topics would get “polluted” by function words (see top of ﬁgure), which is why suchwords are normally removed during preprocessing.
The model can also help disambiguate when the same word is being used syntactically or
semantically. Figure 27.14 shows some examples when the model was applied to the NIPScorpus.
5We see that the roles of words are distinguished, e.g., “we require the algorithm to

returna matrix” (verb) vs “the maximal expected return” (noun). In principle, a part of speech
tagger could disambiguate these two uses, but note that (1) the LDA-HMM method is fullyunsupervised (no POS tags were used), and (2) sometimes a word can have the same POS tag,but different senses, e.g., “the left graph” (a synactic role) vs “the graph G” (a semantic role).
The topic of probabilistic models for syntax and semantics is a vast one, which we do not

The TASA corpus is an untagged collection of educational materials consisting of 37,651 documents and 12,190,931 word
tokens. Words appearing in fewer than 5 documents were replaced with an asterisk, but punctuation was included. The
combined vocabulary was of size 37,202 unique words.
5. NIPS stands for “Neural Information Processing Systems”. It is one of the top machine learning conferences. The
NIPS corpus volumes 1–12 contains 1713 documents.

27.4. ExtensionsofLDA 967
Byilqil
ci¯qi
wπi
Nα
(a)BA
yilqilciπi
Nα
(b)
Figure 27.15 (a) Supervised LDA. (b) Discriminative LDA.
have space to delve into any more. See e.g., (Jurafsky and Martin 2008) for further information.
27.4.4 Supervised LDA
In this section, we discuss extensions of LDA to handle side information of various kinds beyond
just words.
27.4.4.1 Generative supervised LDA
Suppose we have a variable length sequence of words yil∈{1,...,V}as usual, but we also
have a class label ci∈{1,...,C}. How can we predict cifromyi? There are many possible
approaches, but most are direct mappings from the words to the class. In some cases, suchassentiment analysis , we can get better performance by ﬁrst performing inference, to try

to disambiguate the meaning of words. For example, suppose the goal is to determine if adocument is a favorable review of a movie or not. If we encounter the phrase “Brad Pitt wasexcellent until the middle of the movie”, the word “excellent” may lead us to think the review ispositive, but clearly the overall sentiment is negative.
One way to tackle such problems is to build a joint model of the form p(c
i,yi|θ). (Blei
and McAuliffe 2010) proposes an approach, called supervised LDA, where the class label ciis

generated from the topics as follows:
p(ci|qi) = Ber(sigm(wTqi)) (27.65)
Hereqiis the empirical topic distribution for document i:
qik≜1
LiLi∑
i=1qilk (27.66)
See Figure 27.15(a) for an illustration.

968 Chapter27. Latentvariablemodelsfordiscretedata
Byilqilπixi
NW
(a)Byilqilπixi
NWiμ Σ
(b)Byilqilπiαixi
NW
(c)
Figure 27.16 Discriminative variants of LDA. (a) Mixture of experts aka MR-LDA. The double ring denotes
a node that πia deterministic function of its parents. (b) Mixture of experts with random effects. (c)
DMR-LDA.
We can ﬁt this model using Monte Carlo EM: run the collapsed Gibbs sampler in the E step, to
compute E[qik], and then use this as the input feature to a standard logistic regression package.
27.4.4.2 Discriminative supervised LDA
An alternative approach, known as discriminative LDA (Lacoste-Julien et al. 2009), is shown in
Figure 27.15(b). This is a discriminative model of the form p(yi|ci,θ). The only change from
regular LDA is that the topic prior becomes input dependent, as follows:
p(qil|πi,ci=c,θ)=C a t (Acπ) (27.67)
whereAcis aK×Kstochastic matrix.
So far, we have assumed the “side information” is a single categorical variable ci.O f t e nw e

have high dimensional covariates xi∈RD. For example, consider the task of image tagging.
The idea is that yilrepresent correlated tags or labels, which we want to predict given xi.W e
now discuss several attempts to extend LDA so that it can generate tags given the inputs.
The simplest approach is to use a mixture of experts (Section 11.2.4) with multiple outputs.
This is just like LDA except we replace the Dirichlet prior on πiwith a deterministic function
of the input:
πi=S(Wxi) (27.68)

In (Law et al. 2010), this is called multinomial regression LDA. See Figure 27.16(a). Eliminating
the deterministic πiwe have
p(qil|xi,W)=C a t (S(Wxi)) (27.69)
We can ﬁt this with EM in the usual way. However, (Law et al. 2010) suggest an alternative.
First ﬁt an unsupervised LDA model based only on yi; then treat the inferred πias data, and

27.4. ExtensionsofLDA 969
ﬁt a multinomial logistic regression model mapping xitoπi. Although this is fast, ﬁtting LDA
in an unsupervised fashion does not necessarily result in a discriminative set of latent variables,
as discussed in (Blei and McAuliffe 2010).
There is a more subtle problem with this model. Since πiis a deterministic function of the
inputs, it is effectively observed, rendering the qil(and hence the tags yil) independent. In other
words,
p(yi|xi,θ)=Li∏
l=1p(yil|xi,θ)=Li∏
l=1∑
kp(yil|qil=k,B)p(qil=k|xi,W) (27.70)
This means that if we observe the value of one tag, it will have no inﬂuence on any of theothers. This may explain why the results in (Law et al. 2010) only show negligible improvementover predicting each tag independently.
One way to induce correlations is to make Wa random variable. The resulting model is
shown in Figure 27.16(b). We call this a random effects mixture of experts . We typically
assume a Gaussian prior on W

words,
l=1∑
i.I fxi=1, thenp(qil|xi,wi)=C a t (S(wi)),s ow er e c o v e r
the correlated topic model. It is possible to extend this model by adding Markovian dynamicsto theq
ilvariables. This is called a conditional topic random ﬁeld (Zhu and Xing 2010).
A closely related approach, known as Dirichlet multinomial regression LDA (Mimno and
McCallum 2008), is shown in Figure 27.16(c). This is identical to standard LDA except we makeαa function of the input
α
i=e x p (Wxi) (27.71)

α
whereWis aK×Dmatrix. Eliminating the deterministic αiwe have
πi∼Dir(exp(Wx i)) (27.72)
Unlike (Law et al. 2010), this model allows information to ﬂow between tags via the latent πi.
A variant of this model, where xicorresponds to a bag of discrete labels and πi∼Dir(α⊙
xi), is known as labeled LDA (Ramage et al. 2009). In this case, the labels xiare in 1:1

α
correspondence with the latent topics, which makes the resulting topics much more interpretable.An extension, known as partially labeled LDA (Ramage et al. 2011), allows each label to have
multiple latent sub-topics; this model includes LDA, labeled LDA and a multinomial mixturemodel as special cases.
27.4.4.3 Discriminative categorical PCA

An alternative to using LDA is to expand the categorical PCA model with inputs, as shown inFigure 27.17(a). Since the latent space is now real-valued, we can use simple linear regressionfor the input-hidden mapping. For the hidden-output mapping, we use traditional catPCA:
p(z
i|xi,V)=N (Vxi,Σ) (27.73)
p(yi|zi,W)=∏
lCat(yil|S(Wzi)) (27.74)
This model is essentially a probabilistic neural network with one hidden layer, as shown inFigure 27.17(b), but with exchangeable output (e.g., to handle variable numbers of tags). The

970 Chapter27. Latentvariablemodelsfordiscretedata
yilzixi
NWV
(a)Wyi1... ...yiLizi1...ziKxi1... ...xiD
Nv1 vK
(b)
Figure 27.17 (a) Categorical PCA with inputs and exchangeable outputs. (b) Same as (a), but with the
vector nodes expanded out.
key difference from a neural net is that information can ﬂow between the yil’s via the latent
bottleneck layer zi. This should work better than a conventional neural net when the output
labels are highly correlated, even after conditioning on the features; this problem frequently
arises in multi label classiﬁcation. Note that we could allow a direct xitoyiarc, but this would
require too many parameters if the number of labels is large.6
We can ﬁt this model with a small modiﬁcation of the variational EM algorithm in Section 12.4.
If we use this model for regression, rather than classiﬁcation, we can perform the E step exactly,by modifying the EM algorithm for factor analysis. (Ma et al. 1997) reports that this methodconverges faster than standard backpropagation.

We can also extend the model so that the prior on z
iis a mixture of Gaussians using input-
dependent means. If the output is Gaussian, this corresponds to a mixture of discriminativefactor analysers (Fokoue 2005; Zhou and Liu 2008). If the output is categorical, this would bean (as yet unpublished) model, which we could call “discriminative mixtures of categorical factoranalyzers”.
27.5 LVMs for graph-structured data

Another source of discrete data is when modeling graph or network structures. To see theconnection, recall that any graph on Dnodes can be represented as a D×Dadjacency
matrixG,w h e r e G(i,j)=1iff there is an edge from node ito nodej. Such matrices are
binary, and often very sparse. See Figure 27.19 for an example.
Graphs arise in many application areas, such as modeling social networks, protein-protein
interaction networks, or patterns of disease transmission between people or animals. There areusually two primary goals when analysing such data: ﬁrst, try to discover some “interesting

6. A non-probabilistic version of this idea, using squared loss, was proposed in (Ji et al. 2010). This is similar to a linear
feed-forward neural network with an additional edge from xidirectly to yi.

27.5. LVMsforgraph-structureddata 971
789456123
(a)978235164
(b)
Figure 27.18 (a) A directed graph. (b) The same graph, with the nodes partitioned into 3 groups, making
the block structure more apparent.
1
2
3
4
5
6
7
8
9123456789
(a)
0.1
 0.9
 0.1
0.1
 0.1
 0.9
0.9
 0.1
 0.1
1
6
4
8
2
3
5
9
7164823597
Rz97382
5164η
(b)
Figure 27.19 (a) Adjacency matrix for the graph in Figure 27.18(a). (b) Rows and columns are shown
permuted to show the block structure. We also sketch of how the stochastic block model can generate this
graph. From Figure 1 of (Kemp et al. 2006). Used with kind permission of Charles Kemp.
structure” in the graph, such as clusters or communities; second, try to predict which links
might occur in the future (e.g., who will make friends with whom). Below we summarize some
models that have been proposed for these tasks, some of which are related to LDA. Futher details
on these and other approaches can be found in e.g., (Goldenberg et al. 2009) and the references
therein.

therein.
27.5.1 Stochastic block model
In Figure 27.18(a) we show a directed graph on 9 nodes. There is no apparent structure. However,
if we look more deeply, we see it is possible to partition the nodes into three groups or blocks,
B1={1,4,6},B2={2,3,5,8}, andB3={7,9}, such that most of the connections go from
nodes in B1toB2,o rf r o m B2toB3,o rf r o m B3toB1. This is illustrated in Figure 27.18(b).

972 Chapter27. Latentvariablemodelsfordiscretedata
A
D C B A
DCBAE CD AB F G H
A
B
C
D
E
GF
H
CD
D C B A
DCB
Relational system Sorted matrix
E D C B A
EDCBAB
DA
ECE
GHFBA
CA
DBC
DBA
Figure 27.20 Some examples of graphs generated using the stochastic block model with different kinds
of connectivity patterns between the blocks. The abstract graph (between blocks) represent a ring, a
dominance hierarchy, a common-cause structure, and a common-effect structure. From Figure 4 of (Kemp
et al. 2010). Used with kind permission of Charles Kemp.
The problem is easier to understand if we plot the adjacency matrices. Figure 27.19(a) shows
the matrix for the graph with the nodes in their original ordering. Figure 27.19(b) shows the
matrix for the graph with the nodes in their permtuted ordering. It is clear that there is block
We can make a generative model of block structured graphs as follows. First, for every
node, sample a latent block qi∼Cat(π),w h e r eπkis the probability of choosing block k,f o r

k=1:K. Second, choose the probability of connecting group ato group b, for all pairs of
groups; let us denote this probability by ηa,b. This can come from a beta prior. Finally, generate
each edge Rijusing the following model:
p(Rij=r|qi=a,qj=b,η)=B e r (r|ηa,b) (27.75)
This is called the stochastic block model (Nowicki and Snijders 2001). Figure 27.21(a) illustrates
the model as a DGM, and Figure 27.19(c) illustrates how this model can be used to cluster the
nodes in our example.

Note that this is quite different from a conventional clustering problem. For example, we
see that all the nodes in block 3 are grouped together, even though there are no connections
between them. What they share is the property that they “like to” connect to nodes in block 1,
and to receive connections from nodes in block 2. Figure 27.20 illustrates the power of the model
for generating many different kinds of graph structure. For example, some social networks have

hierarchical structure, which can be modeled by clustering people into different social strata,
whereas others consist of a set of cliques.
Unlike a standard mixture model, it is not possible to ﬁt this model using exact EM, because
all the latent qivariables become correlated. However, one can use variational EM (Airoldi et al.

27.5. LVMsforgraph-structureddata 973
qi
ηa,bRi,j
Iπqj
J
(a)qi→j
ηa,bRi,j
Iπiqi←jπjα
J
(b)
Figure 27.21 (a) Stochastic block model. (b) Mixed membership stochastic block model.
2008), collapsed Gibbs sampling (Kemp et al. 2006), etc. We omit the details (which are similar
to the LDA case).
In (Kemp et al. 2006), they lifted the restriction that the number of blocks Kbe ﬁxed, by
replacing the Dirichlet prior on πby a Dirichlet process (see Section 25.2.2). This is known as
the inﬁnite relational model. See Section 27.6.1 for details.
If we have features associated with each node, we can make a discriminative version of this
model, for example by deﬁning
p(Rij=r|qi=a,qj=b,xi,xj,θ)=B e r (r|wT
a,bf(xi,xj)) (27.76)
wheref(xi,xj)is some way of combining the feature vectors. For example, we could use
concatenation, [xi,xj], or elementwise product xi⊗xjas in supervised LDA. The overall
model is like a relational extension of the mixture of experts model.
27.5.2 Mixed membership stochastic block model

In (Airoldi et al. 2008), they lifted the restriction that each node only belong to one cluster. Thatis, they replaced q
i∈{1,...,K}withπi∈SK. This is known as the mixed membership
stochastic block model, and is similar in spirit to fuzzy clustering orsoft clustering.N o t e
thatπikis not the same as p(zi=k|D); the former represents ontological uncertainty (to
what degree does each object belong to a cluster) wheras the latter represents epistemological
uncertainty (which cluster does an object belong to). If we want to combine epistemological

and ontological uncertainty, we can compute p(πi|D).
In more detail, the generative process is as follows. First, each node picks a distribution over
blocks,πi∼Dir(α). Second, choose the probability of connecting group ato group b, for all
pairs of groups, ηa,b∼β(α,β). Third, for each edge, sample two discrete variables, one for
each direction:
qi→j∼Cat(π i),qi←j∼Cat(π j) (27.77)
Finally, generate each edge Rijusing the following model:
p(Rij=1|qi→j=a,qi←j=b,η)=ηa,b (27.78)

974 Chapter27. Latentvariablemodelsfordiscretedata
(a)   1    2   3
   4   5   6   7
   8   9
   10   11   12
   13
   14   15
   16   17
   18Outcasts
Loyal
Opposition Young
TurksWaverers1 Ambrose
2 Boniface
3 Mark4 Winfrid5 Elias6 Basil7 Simplicius8 Berthold9 John Bosco10 Victor11 Bonaventure12 Amand13 Louis14 Albert15Ramuald16 Peter17 Gregory18 Hugh
(b)
Figure 27.22 (a) Who-likes-whom graph for Sampson’s monks. (b) Mixed membership of each monk in
one of three groups. From Figures 2-3 of (Airoldi et al. 2008). Used with kind permission of Edo Airoldi.
See Figure 27.21(b) for the DGM.
Unlike the regular stochastic block model, each node can play a different role, depending on

2 Boniface
(b)
who it is connecting to. As an illustration of this, we will consider a data set that is widely usedin the social networks analysis literature. The data concerns who-likes-whom amongst of groupof 18 monks. It was collected by hand in 1968 by Sampson (Sampson 1968) over a period ofmonths. (These days, in the era of social media such as Facebook, a social network with only 18people is trivially small, but the methods we are discussing can be made to scale.) Figure 27.22(a)plots the raw data, and Figure 27.22(b) plots E[π]

ifor each monk, where K=3. We see that
most of the monk belong to one of the three clusters, known as the “young turks”, the “outcasts”and the “loyal opposition”. However, some individuals, notably monk 15, belong to two clusters;Sampson called these monks the “waverers”. It is interesting to see that the model can recoverthe same kinds of insights as Sampson derived by hand.
One prevalent problem in social network analysis is missing data. For example, if R
ij=0,
it may be due to the fact that person iandjhave not had an opportunity to interact, or
that data is not available for that interaction, as opposed to the fact that these people don’twant to interact. In other words, absence of evidence is not evidence of absence. We can model
this by modifying the observation model so that with probability ρ, we generate a 0 from the
background model, and we only force the model to explain observed 0s with probability 1−ρ.
In other words, we robustify the observation model to allow for outliers, as follows:
p(R

p(R
ij=r|qi→j=a,qi←j=b,η)=ρδ0(r)+(1−ρ)Ber(r|ηa,b) (27.79)
See (Airoldi et al. 2008) for details.
27.5.3 Relational topic model
In many cases, the nodes in our network have atttributes. For example, if the nodes representacademic papers, and the edges represent citations, then the attributes include the text of thedocument itself. It is therefore desirable to create a model that can explain the text and the linkstructure concurrently. Such a model can predict links given text, or even vice versa.

p(R
Therelational topic model (RTM) (Chang and Blei 2010) is one way to do this. This is a

27.6. LVMsforrelationaldata 975
yilqilπi
¯qiRij
I¯qjqjlyjl
πj
JαB
w
Figure 27.23 DGM for the relational topic model.
simple extension of supervised LDA (Section 27.4.4.1), where the response variable Rij(which
represents whether there is an edge between nodes iandj) is modeled as follows:
p(Rij=1|qi,qj,θ) = sigm(wT(qi⊗qj)+w0) (27.80)
Recall that qiis the empirical topic distribution for document i,qik≜1
Li∑Li
i=1qilk. See
Figure 27.23
Note that it is important that Rijdepend on the actual topics chosen, qiandqj, and not
on the topic distributions, πiandπj, otherwise predictive performance is not as good. The
intuitive reason for this is as follows: if Rijis a child of πiandπj, it will be treated as just
another word, similar to the qil’s andyil’s; but since there are many more words than edges,
the graph structure information will get “washed out”. By making Rija child of qiandqj, the
graph information can inﬂuence the choice of topics more directly.

One can ﬁt this model in a manner similar to SLDA. See (Chang and Blei 2010) for details.
The method does better at predicting missing links than the simpler approach of ﬁrst ﬁtting an
LDA model, and then using the qi’s as inputs to a logistic regression problem. The reason is
analogous to the superiority of partial least squares (Section 12.5.2) to PCA+ linear regression,namely that the RTM learns a latent space that is forced to be predictive of the graph structureand words, whereas LDA might learn a latent space that is not useful for predicting the graph.

27.6 LVMs for relational data
Graphs can be used to represent data which represents the relation amongst variables of acertain type, e.g., friendship relationships between people. But often we have multiple types ofobjects, and multiple types of relations. For example, Figure 27.24 illustrates two relations, onebetween people and people, and one between people and movies.
In general, we deﬁne a k-aryrelationRas a subset of k-tuples of the appropriate types:
R⊆T
1×T2×···×T k (27.81)

976 Chapter27. Latentvariablemodelsfordiscretedata
Figure 27.24 Example of relational data. There are two types of objects, peopleandmovies; one 2-ary
relation,friends:people×people→{0,1}and one 2-ary function, rates:people×movie→R.Ageand
sexare attributes (unary functions) of the peopleclass.
whereTiare sets or types. A binary, pairwise or dyadic relation is a relation deﬁned on pairs
of objects. For example, the seenrelation between people and movies might be represented as
the set of movies that people have seen. We can either represent this explicitly as a set, such as
seen = { (Bob, StarWars), (Bob, TombRaider), (Alice, Jaws)}
or implicitly, using an indicator function for the set:
seen(Bob, StarWars)=1, seen(Bob, TombRaider)=1, seen(Alice, Jaws)=1
A relation between two entities of types T1andT2can be represented as a binary function
R:T1×T2→{0,1}, and hence as a binary matrix. This can also be represented as a bipartite

graph, in which we have nodes of two types. If T1=T2, this becomes a regular directed graph,
as in Section 27.5. However, there are some situations that are not so easily modelled by graphs,
but which can still be modelled by relations. For example, we might have a ternary relation,
R:T1×T1×T2→{0,1}, where, say, R(i,j,k)=1iff protein iinteracts with protein j
when chemical kis present. This can be modelled by a 3d binary matrix. We will give some
examples of this in Section 27.6.1.

Making probabilistic models of relational data is called statistical relational learning (Getoor
and Taskar 2007). One approach is to directly model the relationship between the variables using
graphical models; this is known as probabilistic relational modeling . Another approach is to
use latent variable models, as we discuss below.
27.6.1 Inﬁnite relational model
It is straightforward to extend the stochastic block model to model relational data: we just
associate a latent variable qt

i∈{1,...,K t}with each entity iof each type t. We then deﬁne
the probability of the relation holding between speciﬁc entities by looking up the probability of
the relation holding between entities of that type. For example, if R:T1×T1×T2→{0,1},
we have
p(R(i,j,k)=1|q1
i=a,q1
j=b,q2
k=c,η)=ηa,b,c (27.82)
If we allow the number of clusters Ktfor each type to be unbounded, by using a Dirichlet pro-
cess, the model is called the inﬁnite relational model (IRM) (Kemp et al. 2006). An essentially

27.6. LVMsforrelationaldata 977
/g4/g9/g6/g12/g10/g4/g2/g11/g16
/g5/g10/g16/g6/g2/g16/g6/g16
/g16/g10/g8/g13/g16 /g14/g15/g8/g2/g13/g10/g16/g12/g16/g3/g10/g14/g2/g4/g17/g10/g19/g6/g1/g16/g18/g3/g16/g17/g2/g13/g4/g6/g16
/g3/g10/g14/g11/g14/g8/g10/g4/g2/g11/g1/g7/g18/g13/g4/g17/g10/g14/g13/g16
/g2/g3/g13/g14/g15/g12/g2/g11/g10/g17/g10/g6/g16causes,
result ofaffects,
complicates
disrupts
result of, manifestation ofaffects, process of,
affects,
process of
associated withcauses
manifestation ofaffects
affects,process of,
result ofcomplicates,
manifestation ofinteract with
process of,
result of
affects,
process ofcauses,
manifestation of,associated withcomplicatesaffects,causes
manifestation of,complicatesaffects,
Figure 27.25 Illustration of an ontology learned by IRM applied to the Uniﬁed Medical Language System.
The boxes represent 7 of the 14 concept clusters. Predicates that belong to the same cluster are grouped

affects,
process of
result of
affects,
together, and associated with edges to which they pertain. All links with weight above 0.8 have beenincluded. From Figure 9 of (Kemp et al. 2010). Used with kind permission of Charles Kemp.
identical model, under the name inﬁnite hidden relational model (IHRM), was concurrently
proposed in (Xu et al. 2006). We can ﬁt this model with variational Bayes (Xu et al. 2006, 2007)
or collapsed Gibbs sampling (Kemp et al. 2006). Rather than go into algorithmic detail, we justsketch some interesting applications.

27.6.1.1 Learning ontologies
Anontology refers to an organisation of knowledge. In AI, ontologies are often built by hand
(see e.g., (Russell and Norvig 2010)), but it is interesting to try and learn them from data. In(Kemp et al. 2006), they show how this can be done using the IRM.
The data comes from the Uniﬁed Medical Language System (McCray 2003), which deﬁnes

a semantic network with 135 concepts (such as “disease or syndrome”, “diagnostic procedure”,“animal”), and 49 binary predicates (such as “affects”, “prevents”). We can represent this as aternary relation R:T
1×T1×T2→{0,1},w h e r e T1is the set of concepts and T2is the

set of binary predicates. The result is a 3d cube. We can then apply the IRM to partition thecube into regions of roughly homogoneous response. The system found 14 concept clusters and21 predicate clusters. Some of these are shown in Figure 27.25. The system learns, for example,that biological functions affect organisms (since η
a,b,c≈1wherearepresents the biological
function cluster, brepresents the organism cluster, and crepresents the affects cluster).
27.6.1.2 Clustering based on relations and features

We can also use IRM to cluster objects based on their relations and their features. For example,(Kemp et al. 2006) consider a political dataset (from 1965) consisting of 14 countries, 54 binary

978 Chapter27. Latentvariablemodelsfordiscretedata
joint
membership
of IGOs
joint
of NGOs
negative
behavior
negative
communications
 accusations
 protests
treaties
 conferences
common bloc
economic
aid
 emigration
military
alliance
sends
tourists to
exports
books to
 exports to
Brazil
Netherlands
UK
USA
Burma
Indonesia
Jordan
Egypt
India
Israel
China
Cuba
Poland
USSR
noncommunist
western bloc
constitutional govt
free elections
communist bloc
communists
totalitarian
elitist
high censorship
no free elections
illiteracy
domestic violence
purges
far from US
rainfall
religious books
exports/GNP
govt education $
military personnel
seaborne goods
govt crisis
UN delinquent
neutral bloc
assassinations
govt revolution
num religions
intervening military
some censorship
energy consumed
telephone
defense $
threats
GNP
protests
Catholics
US aid taken
popn. density
land area
railroad length
foreign students
age of country
law NGOs
num languages
aid $ taken
female workers
foreign mail sent

purges
far from US
rainfall
telephone
defense $
threats
GNP
protests
Catholics
land area
law NGOs
aid $ taken
protein in diet
investments
arts NGOs
monarchy
road length
arable
emigrants
unemployed
calories in dieta)
c) d)
e) g) h) i)b)
book
translationsf)
Figure 27.26 Illustration of IRM applied to some political data containing features and pairwise interac-
tions. Top row (a). the partition of the countries into 5 clusters and the features into 5 clusters. Every
second column is labelled with the name of the corresponding feature. Small squares at bottom (a-i): these

aid $ taken
arts NGOs
monarchy
arable
emigrants
c) d)
book
are 8 of the 18 clusters of interaction types. From Figure 6 of (Kemp et al. 2006). Used with kind permission
of Charles Kemp.
predicates representing interaction types between countries (e.g., “sends tourists to”, “economic
aid”), and 90 features (e.g., “communist”, “monarchy”). To create a binary dataset, real-valued
features were thresholded at their mean, and categorical variables were dummy-encoded. The
data has 3 types: T1represents countries, T2represents interactions, and T3represents features.

We have two relations: R1:T1×T1×T2→{0,1}, andR2:T1×T3→{0,1}. (This problem
therefore combines aspects of both the biclustering model and the ontology discovery model.)
When given multiple relations, the IRM treats them as conditionally independent. In this case,
we have
p(R1,R2,q1,q2,q3|θ)=p(R1|q1,q2,θ)p(R2|q1,q3,θ) (27.83)
The results are shown in Figure 27.26. The IRM divides the 90 features into 5 clusters, the
ﬁrst of which contains “noncommunist”, which captures one of the most important aspects of

we have
this Cold-War era dataset. It also clusters the 14 countries into 5 clusters, reﬂecting natural
geo-political groupings (e.g., US and UK, or the Communist Bloc), and the 54 predicates into 18
clusters, reﬂecting similar relationships (e.g., “negative behavior and “accusations”).

27.6. LVMsforrelationaldata 979
27.6.2 Probabilistic matrix factorization for collaborative ﬁltering
As discussed in Section 1.3.4.2, collaborative ﬁltering (CF) requires predicting entries in a matrix
R:T1×T2→R, where for example R(i,j)is the rating that user igave to movie j. Thus
we see that CF is a kind of relational learning problem (and one with particular commercialimportance).
Much of the work in this area makes use of the data that Netﬂix made available in their
competition. In particular, a large 17,770 ×480,189 movie x user ratings matrix is provided. The
full matrix would have ∼8.6×10
9entries, but only 100,480,507 (about 1%) of the entries are

observed, so the matrix is extremely sparse. In addition the data is quite imbalanced, with manyusers rating fewer than 5 movies, and a few users rating over 10,000 movies. The validationset is 1,408,395 (movie,user) pairs. Finally, there is a separate test set with 2,817,131 (movie,user)pairs, for which the ranking is known but withheld from contestants. The performance measureis root mean square error:
RMSE=√
1
NN∑
i=1(X(mi,ui)−ˆX(mi,ui))2 (27.84)
whereX(mi,ui)is the true rating of user uion movie mi, andˆX(mi,ui)is the prediction.

RMSE=√
1
NN∑
The baseline system, known as Cinematch, had an RMSE on the training set of 0.9514, and on
the test set of 0.9525. To qualify for the grand prize, teams needed to reduce the test RMSE by10%, i.e., get a test RMSE of 0.8563 or less. We will discuss some of the basic methods used bytthe winning team below.
Since the ratings are drawn from the set {0,1,2,3,4,5}, it is tempting to use a categorical
observation model. However, this does not capture the fact that the ratings are ordered. Althoughwe could use an ordinal observation model, in practice people use a Gaussian observation modelfor simplicity. One way to make the model better match the data is to pass the model’s predictedmean response through a sigmoid, and then to map the [0,1]interval to [0,5](Salakhutdinov

and Mnih 2008). Alternatively we can make the data a better match to the Gaussian model bytransforming the data using R
ij=√
6−Rij(Aggarwal and Merugu 2007).
We could use the IRM for the CF task, by associating a discrete latent variable for each user
qu
iand for each movie or video qv
j, and then deﬁning
p(Rij=r|qu
i=a,qv
j=b,θ)=N(r|μa,b,σ2) (27.85)
This is just another example of co-clustering. We can also extend the model to generate sideinformation, such as attributes about each user and/or movie. See Figure 27.27 for an illustration.

ij=√
qu
i=a,qv
Another possibility is to replace the discrete latent variables with continuous latent variables
π
u
i∈SKuandπvj∈SKv. However, it has been found (see e.g., (Shan and Banerjee 2010)) that
one obtains much better results by using unconstrained real-valued latent factors for each user
ui∈RKand each movie vj∈RK.7We then use a likelihood of the form
p(Rij=r|ui,vj)=N(r|uT
ivj,σ2) (27.86)
7. Good results with discrete latent variables have been obtained on some datasets that are smaller than Netﬂix, such as

π
u
MovieLens and EachMovie. However, these datasets are much easier to predict, because there is less imbalance between
the number of reviews performed by different users (in Netﬂix, some users have rated more than 10,000 movies, whereas
others have rated less than 5).

980 Chapter27. Latentvariablemodelsfordiscretedata
Figure 27.27 Visualization of a small relational dataset, where we have one relation, likes(user, movie),
and features for movies (here, genre) and users (here, occupation). From Figure 5 of (Xu et al. 2008). Used
with kind permission of Zhao Xu.
Rijuiμu
ΣuT1vj
T2μvΣv
σ2
(a)−1.5 −1.0 −0.5 0.0 0.5 1.0−1.5 −1.0 −0.5 0.0 0.5 1.0 1.5
Factor Vector 1Factor Vector 2Freddy Got Fingered
Freddy vs. JasonHalf Baked
Road Trip
The Sound of MusicSophie’s Choice
Moonstruck
Maid in ManhattanThe Way We Were
Runaway Bride Coyote UglyThe Royal TenenbaumsPunch−Drunk Love
I Heart Huckabees
ArmageddonCitizen Kane
The Waltons: Season 1
StepmomJulien Donkey−Boy
Sister Act The Fast and the FuriousThe Wizard of OzKill Bill: Vol. 1
ScarfaceNatural Born KillersAnnie Hall Belle de JourLost in Translation
The Longest YardBeing John Malkovich
Catwoman
(b)
Figure 27.28 (a) A DGM for probabilistic matrix factorization. (b) Visualization of the ﬁrst two factors in

Catwoman
(b)
the PMF model estimated from the Netﬂix challenge data. Each movie jis plotted at the location speciﬁed
ˆvj. On the left we have low-brow humor and horror movies ( Half Baked ,Freddy vs Jason ), and on the
right we have more serious dramas ( Sophie’s Choice ,Moonstruck ). On the top we have critically acclaimed
independent movies ( Punch-Drunk Love ,I Heart Huckabees ), and on the bottom we have mainstream
Hollywood blockbusters ( Armageddon ,RunwayBride ). TheWizardofOz is right in the middle of these axes.

From Figure 3 of (Koren et al. 2009). Used with kind permission of Yehuda Koren.
This has been called probabilistic matrix factorization (PMF) (Salakhutdinov and Mnih 2008).
See Figure 27.28(a) for the DGM. The intuition behind this method is that each user and each
movie get embedded into the same low-dimensional continuous space (see Figure 27.28(b)). If a
user is close to a movie in that space, they are likely to rate it highly. All of the best entries in
the Netﬂix competition used this approach in one form or another.8

PMF is closely related to the SVD. In particular, if there is no missing data, then computing
the MLE for the ui’s and the vj’s is equivalent to ﬁnding a rank Kapproximation to R.
However, as soon as we have missing data, the problem becomes non-convex, as shown in
8. The winning entry was actually an ensemble of different methods, including PMF, nearest neighbor methods, etc.

27.6. LVMsforrelationaldata 981
0 5 10 15 20 25 30 35 40 45 50 55 600.90.910.920.930.940.950.960.97
EpochsRMSE
PMF
Constrained 
PMFNetflix Baseline Score
SVD
(a)10 100 1000 10000 1000000.8750.880.8850.890.8950.90.9050.9140
60
90
128
180 50
100200
50
100
200
50
100200500 1000
1500
Millions of ParametersRMSEPlain
w/Biases
w/Implicit feedback
w/Temporal dynamics
(b)
Figure 27.29 (a) RMSE on the validation set for different PMF variants vs number of passes through
the data. “SVD” is the unregularized version, λU=λV=0. “PMF1” corresponds to λU=0.01and
λV=0.001, while “PMF2” corresponds to λU=0.001andλV=0.0001. “PMFA1” corresponds to a
version where the mean and diagonal covariance of the Gaussian prior were learned from data. From
Figure 2 of (Salakhutdinov and Mnih 2008). Used with kind permission of Ruslan Salakhutdinov. (b) RMSEon the test set (quiz portion) vs number of parameters for several different models. “Plain” is the baselinePMF with suitably chosen λ

U,λV. “With biases” adds fiandgjoffset terms. “With implicit feedback”
“With temporal dynamics” allows the offset terms to change over time. The Netﬂix baseline system achievesan RMSE of 0.9514, and the grand prize’s required accuracy is 0.8563 (which was obtained on 21 September2009). Figure generated by netflixResultsPlot . From Figure 4 of (Koren et al. 2009). Used with kind
permission of Yehuda Koren.
(Srebro and Jaakkola 2003), and standard SVD methods cannot be applied. (Recall that in the

Netﬂix challenge, only about 1% of the matrix is observed.)
The most straightforward way to ﬁt the PMF model is to minimize the overall NLL:
J(U,V)=−logp(R|U,V,O)=−log⎛
⎝N∏
i=1M∏
j=1[
N(Rij|uT
ivj,σ2)]I(Oij=1)⎞
⎠(27.87)
whereOij=1if userihas seen movie j. Since this is non-convex, we can just ﬁnd a locally
optimal MLE. Since the Netﬂix data is so large (about 100 million observed entries), it is common
to use stochastic gradient descent (Section 8.5.2) for this task. The gradient for uiis given by
dJ
dui=d
dui1
2∑

⎝N∏
i=1M∏
j=1[
N(Rij|uT
⎠(27.87)
dJ
dui=d
dui1
2∑
ijI(Oij=1 ) (Rij−uT
ivj)2=−∑
j:Oij=1eijvj (27.88)
whereeij=Rij−uT
ivjis the error term. By stochastically sampling a single movie jthat user
ihas watched, the update takes the following simple form:
ui=ui+ηeijvj (27.89)
whereηis the learning rate. The update for vjis similar.

982 Chapter27. Latentvariablemodelsfordiscretedata
Of course, just maximizing the likelihood results in overﬁtting, as shown in Figure 27.29(a).
We can regularize this by imposing Gaussian priors:
p(U,V)=∏
iN(ui|μu,Σu)∏
jN(vj|μv,Σv) (27.90)
If we use μu=μv=0,Σu=σ2
UIK, andΣv=σ2
VIK, the new objective becomes
J(U,V)=− logp(R,U,V|O,θ) (27.91)
=∑
i∑
jI(Oij=1 ) (Rij−uT
ivj)2
+λU∑
i||ui||22+λV∑
j||vj||22+const (27.92)
where we have deﬁned λU=σ2/σ2
UandλV=σ2/σ2
V. By varying the regularizers, we can
reduce the effect of overﬁtting, as shown in Figure 27.29(a). We can ﬁnd MAP estimates using
stochastic gradient descent. We can also compute approximate posteriors using variational Bayes(Ilin and Raiko 2010).
If we use diagonal covariances for the priors, we can penalize each latent dimension by a
different amount. Also, if we use non-zero means for the priors, we can account for offset terms.Optimizing the prior parameters (μ
u,Σu,μv,Σv)at the same time as the model parameters

(U,V,σ2)is a way to create an adaptive prior. This avoids the need to search for the optimal
values of λUandλV, and gives even better results, as shown in Figure 27.29(a).
It turns out that much of the variation in the data can be explained by movie-speciﬁc or
user-speciﬁc effects. For example, some movies are popular for all types of users. And someusers give low scores for all types of movies. We can model this by allowing for user and moviespeciﬁc offset or bias terms as follows:
p(R
ij=r|ui,vj,θ)=N(r|uT
ivj+μ+fi+gj,σ2) (27.93)

p(R
whereμis the overall mean, fiis the user bias, gjis the movie bias, and uT
ivjis the
interaction term. This is equivalent to applying PMF just to the residual matrix, and gives much
better results, as shown in Figure 27.29(b). We can estimate the fi,gjandμterms using
stochastic gradient descent, just as we estimated U,Vandθ.
We can also allow the bias terms to evolve over time, to reﬂect the changing preferences of

p(R
ivjis the
users (Koren 2009b). This is important since in the Netﬂix competition, the test data was morerecent than the training data. Figure 27.29(b) shows that allowing for temporal dynamics canhelp a lot.
Often we also have side information of various kinds. In the Netﬂix competition, entrants
knew which movies the user had rated in the test set, even though they did not know thevalues of these ratings. That is, they knew the value of the (dense) Omatrix even on the

test set. If a user chooses to rate a movie, it is likely because they have seen it, which inturns means they thought they would like it. Thus the very act of rating reveals information.Conversely, if a user chooses not rate a movie, it suggests they knew they would not like it.So the data is not missing at random (see e.g., (Marlin and Zemel 2009)). Exploiting this canimprove performance, as shown in Figure 27.29(b). In real problems, information on the test setis not available. However, we often know which movies the user has watched or declined to

27.7. RestrictedBoltzmannmachines(RBMs) 983
watch, even if they did not rate them (this is called implicit feedback), and this can be used as
useful side information.
Another source of side information concerns the content of the movie, such as the movie
genre, the list of the actors, or a synopsis of the plot. This can be denoted by xv, the features
of the video. (In the case where we just have the id of the video, we can treat xvas a|V|-
dimensional bit vector with just one bit turned on.) We may also know features about the user,
which we can denote by xu. In some cases, we only know if the user clicked on the video or
not, that is, we may not have a numerical rating. We can then modify the model as follows:
p(R(u,v)|xu,xv,θ)=B e r (R(u,v)|(Uxu)T(Vxv)) (27.94)
whereUis a|U|×Kmatrix, and Vis a|V|×Kmatrix (we can incorporate an offset term
by appending a 1 to xuandxvin the usual way). A method for computing the approximate

posterior p(U,V|D)in an online fashion, using ADF and EP, was described in (Stern et al.
2009). This was implemented by Microsoft and has been deployed to predict click through rateson all the ads used by Bing.
Unfortunately, ﬁtting this model just from positive binary data can result in an over prediction

of links, since no negative examples are included. Better performance is obtained if one hasaccess to the set of all videos shown to the user, of which at most one was picked; data of thisform is known as an impression log. In this case, we can use a multinomial model instead of

a binary model; in (Yang et al. 2011), this was shown to work much better than a binary model.To understand why, suppose some is presented with a choice of an action movie starring ArnoldSchwarzenegger, an action movie starring Vin Diesel, and a comedy starring Hugh Grant. Ifthe user picks Arnold Schwarzenegger, we learn not only that they like prefer action movies tocomedies, but also that they prefer Schwarzenegger to Diesel. This is more informative than justknowing that they like Schwarzenegger and action movies.

27.7 Restricted Boltzmann machines (RBMs)
So far, all the models we have proposed in this chapter have been representable by directedgraphical models. But some models are better represented using undirected graphs. For example,theBoltzmann machine (Ackley et al. 1985) is a pairwise MRF with hidden nodes hand visible
nodesv, as shown in Figure 27.30(a). The main problem with the Boltzmann machine is that
exact inference is intractable, and even approximate inference, using e.g., Gibbs sampling, canbe slow. However, suppose we restrict the architecture so that the nodes are arranged in layers,and so that there are no connections between nodes within the same layer (see Figure 27.30(b)).Then the model has the form
p(h,v|θ)=1
Z(θ)R∏
r=1K∏
k=1ψrk(vr,hk) (27.95)
whereRis the number of visible (response) variables, Kis the number of hidden variables, and
vplays the role of yearlier in this chapter. This model is known as a restricted Boltzmann
machine (RBM) (Hinton 2002), or a harmonium (Smolensky 1986).

Z(θ)R∏
r=1K∏
An RBM is a special case of a product of experts (PoE) (Hinton 1999), which is so-called
because we are multiplying together a set of “experts” (here, potential functions on each edge)

984 Chapter27. Latentvariablemodelsfordiscretedata
(a)H
V
(b)
Figure 27.30 (a) A general Boltzmann machine, with an arbitrary graph structure. The shaded (visible)
nodes are partitioned into input and output, although the model is actually symmetric and deﬁnes a joint
density on all the nodes. (b) A restricted Boltzmann machine with a bipartite structure. Note the lack of
intra-layer connections.
and then normalizing, whereas in a mixture of experts, we take a convex combination of
normalized distributions. The intuitive reason why PoE models might work better than a mixture
is that each expert can enforce a constraint (if the expert has a value which is ≫1or≪1)
or a “don’t care” condition (if the expert has value 1). By multiplying these experts together
in different ways we can create “sharp” distributions which predict data which satisﬁes the
speciﬁed constraints (Hinton and Teh 2001). For example, consider a distributed model of text.

A given document might have the topics “government”, “maﬁa” and “playboy”. If we “multiply”
the predictions of each topic together, the model may give very high probability to the word
“Berlusconi”9(Salakhutdinov and Hinton 2010). By contrast, adding together experts can only
make the distribution broader (see Figure 14.17).
Typically the hidden nodes in an RBM are binary, so hspeciﬁes which constraints are active.
It is worth comparing this with the directed models we have discussed. In a mixture model, we

have one hidden variable q∈{1,...,K}. We can represent this using a set of Kbits, with the
restriction that exactly one bit is on at a time. This is called a localist encoding , since only
one hidden unit is used to generate the response vector. This is analogous to the hypothetical
notion of grandmother cells in the brain, that are able to recognize only one kind of object.
By contrast, an RBM uses a distributed encoding , where many units are involved in generating
each output. Models that used vector-valued hidden variables, such as π∈SK, as in mPCA/

LDA, orz∈RK, as in ePCA also use distributed encodings.
The main difference between an RBM and directed two-layer models is that the hidden
variables are conditionally independent given the visible variables, so the posterior factorizes:
p(h|v,θ)=∏
kp(hk|v,θ) (27.96)
This makes inference much simpler than in a directed model, since we can estimate each hk
9. Silvio Berlusconi is the current (2011) prime minister of Italy.

27.7. RestrictedBoltzmannmachines(RBMs) 985
Visible Hidden Name Reference
Binary Binary Binary RBM (Hinton 2002)
Gaussian Binary Gaussian RBM (Welling and Sutton 2005)
Categorical Binary Categorical RBM (Salakhutdinov et al. 2007)
Multiple categorical Binary Replicated softmax/ undirected LDA (Salakhutdinov and Hinton 2010)
Gaussian Gaussian Undirected PCA (Marks and Movellan 2001)
Binary Gaussian Undirected binary PCA (Welling and Sutton 2005)
Table 27.2 Summary of different kinds of RBM.
independently and in parallel, as in a feedforward neural network. The disadvantage is that
training undirected models is much harder, as we discuss below.
27.7.1 Varieties of RBMs
In this section, we describe various forms of RBMs, by deﬁning different pairwise potentialfunctions. See Table 27.2 for a summary. All of these are special cases of the exponential
family harmonium (Welling et al. 2004).
27.7.1.1 Binary RBMs

The most common form of RBM has binary hidden nodes and binary visible nodes. The jointdistribution then has the following form:
p(v,h|θ)=1
Z(θ)exp(−E(v,h;θ)) (27.97)
E(v,h;θ)≜−R∑
r=1K∑
k=1vrhkWrk−R∑
r=1vrbr−K∑
k=1hkck (27.98)
=−(vTWh+vTb+hTc) (27.99)
Z(θ)=∑
v∑
hexp(−E(v,h;θ)) (27.100)
whereEis the energy function, Wis aR×Kweight matrix, bare the visible bias terms, care
the hidden bias terms, and θ=(W,b,c)are all the parameters. For notational simplicity, we
will absorb the bias terms into the weight matrix by clamping dummy units v0=1andh0=1

r=1K∑
Z(θ)=∑
v∑
and setting w0,:=candw:,0=b. Note that naively computing Z(θ)takesO(2R2K)time
but we can reduce this to O(min{R2K,K2R})time (Exercise 27.1).
When using a binary RBM, the posterior can be computed as follows:
p(h|v,θ)=K∏
k=1p(hk|v,θ)=∏
kBer(hk|sigm(wT
:,kv)) (27.101)
By symmetry, one can show that we can generate data given the hidden variables as follows:
p(v|h,θ)=∏
rp(vr|h,θ)=∏
rBer(vr|sigm(wT
r,:h)) (27.102)

986 Chapter27. Latentvariablemodelsfordiscretedata
We can write this in matrix-vetor notation as follows:
E[h|vθ]=s i g m ( WTv) (27.103)
E[v|h,θ]=s i g m ( Wh) (27.104)
The weights in Ware called the generative weights, since they are used to generate the
observations, and the weights in WTare called the recognition weights, since they are used
to recognize the input.
From Equation 27.101, we see that we activate hidden node kin proportion to how much the
input vector v“looks like” the weight vector w:,k(up to scaling factors). Thus each hidden node
captures certain features of the input, as encoded in its weight vector, similar to a feedforward
neural network.
27.7.1.2 Categorical RBM
We can extend the binary RBM to categorical visible variables by using a 1-of-C encoding,
whereCis the number of states for each vir. We deﬁne a new energy function as follows
(Salakhutdinov et al. 2007; Salakhutdinov and Hinton 2010):
r=1K∑
k=1C∑
c=1vc
rhkWc
rk−R∑
r=1C∑
c=1vc
rbc
r−K∑
k=1hkck (27.105)

r=1K∑
k=1C∑
c=1vc
rhkWc
rk−R∑
r=1C∑
c=1vc
rbc
r−K∑
The full conditionals are given by
p(vr|h,θ)=C a t (S({bcr+∑
khkWc
rk}Cc=1)) (27.106)
p(hk=1|c,θ) = sigm(c k+∑
r∑
cvc
rWc
rk) (27.107)
27.7.1.3 Gaussian RBM
We can generalize the model to handle real-valued data. In particular, a Gaussian RBM has the
following energy function:
E(v,h|θ)=−R∑
r=1K∑
k=1Wrkhkvr−1
2R∑
r=1(vr−br)2−K∑
k=1akhk (27.108)
The parameters of the model are θ=(wrk,ak,br). (We have assumed the data is standardized,
so we ﬁx the variance to σ2=1.) Compare this to a Gaussian in information form:
Nc(v|η,Λ)∝exp(ηTv−1
2vTΛv) (27.109)

khkWc
r∑
cvc
rWc
r=1K∑
2R∑
whereη=Λμ. We see that we have set Λ=I, andη=∑
khkw:,k. Thus the mean is
given byμ=Λ−1η=∑
khkw:,k. The full conditionals, which are needed for inference and

27.7. RestrictedBoltzmannmachines(RBMs) 987
learning, are given by
p(vr|h,θ)=N (vr|br+∑
kwrkhk,1) (27.110)
p(hk=1|v,θ)=s i g m(
ck+∑
rwrkvr)
(27.111)
We see that each visible unit has a Gaussian distribution whose mean is a function of the
hidden bit vector. More powerful models, which make the (co)variance depend on the hiddenstate, can also be developed (Ranzato and Hinton 2010).
27.7.1.4 RBMs with Gaussian hidden units
If we use Gaussian latent variables and Gaussian visible variables, we get an undirected versionof factor analysis. However, it turns out that it is identical to the standard directed version(Marks and Movellan 2001).
If we use Gaussian latent variables and categorical observed variables, we get an undirected
version of categorical PCA (Section 27.2.2). In (Salakhutdinov et al. 2007), this was applied to theNetﬂix collaborative ﬁltering problem, but was found to be signiﬁcantly inferior to using binarylatent variables, which have more expressive power.
27.7.2 Learning RBMs

In this section, we discuss some ways to compute ML parameter estimates of RBMs, usinggradient-based optimizers. It is common to use stochastic gradient descent, since RBMs oftenhave many parameters and therefore need to be trained on very large datasets. In addition, it isstandard to use ℓ
2regularization, a technique that is often called weight decay in this context.
This requires a very small change to the objective and gradient, as discussed in Section 8.3.6.
27.7.2.1 Deriving the gradient using p(h,v|θ)

To compute the gradient, we can modify the equations from Section 19.5.2, which show how toﬁt a generic latent variable maxent model. In the context of the Boltzmann machine, we haveone feature per edge, so the gradient is given by
∂ℓ
∂wrk=1
NN∑
i=1E[vrhk|vi,θ]−E[vrhk|θ] (27.112)
We can write this in matrix-vector form as follows:
∇wℓ=Epemp(·|θ)[
vhT]
−Ep(·|θ)[
vhT]
(27.113)
wherepemp(v,h|θ)≜p(h|v,θ)pemp(v), andpemp(v)=1
N∑N
i=1δvi(v)is the empirical
distribution. (We can derive a similar expression for the bias terms by setting vr=1or
hk=1.)
The ﬁrst term on the gradient, when vis ﬁxed to a data case, is sometimes called the
clamped phase, and the second term, when vis free, is sometimes called the unclamped

988 Chapter27. Latentvariablemodelsfordiscretedata
phase. When the model expectations match the empirical expectations, the two terms cancel
out, the gradient becomes zero and learning stops. This algorithm was ﬁrst proposed in (Ackleyet al. 1985). The main problem is efficiently computing the expectations. We discuss some waysto do this below.
27.7.2.2 Deriving the gradient using p(v|θ)
We now present an alternative way to derive Equation 27.112, which also applies to other energybased models. First we marginalize out the hidden variables and write the RBM in the formp(v|θ)=
1
Z(θ)exp(−F(v;θ)),w h e r eF (v;θ)is thefree energy:
F(v)≜∑
hE(v,h)=∑
hexp(R∑
r=1K∑
k=1vrhkWrk)
(27.114)
=∑
hK∏
k=1exp(R∑
r=1vrhkWrk)
(27.115)
=K∏
k=1∑
hr∈{0,1}exp(R∑
r=1vrhrWrk)
(27.116)
=K∏
k=1(
1+exp(R∑
r=1vrWrk))
(27.117)
Next we write the (scaled) log-likelihood in the following form:
ℓ(θ)=1
NN∑
i=1logp(vi|θ)=−1
NN∑
i=1F(vi|θ)−logZ(θ) (27.118)
Using the fact that Z(θ)=∑
vexp(−F(v;θ))we have
∇ℓ(θ)=−1
NN∑
i=1∇F(vi)−∇Z
Z(27.119)
=−1

1
F(v)≜∑
hE(v,h)=∑
hexp(R∑
r=1K∑
(27.114)
=∑
hK∏
k=1exp(R∑
(27.115)
=K∏
k=1∑
(27.116)
=K∏
k=1(
1+exp(R∑
(27.117)
ℓ(θ)=1
NN∑
NN∑
∇ℓ(θ)=−1
NN∑
Z(27.119)
=−1
NN∑
i=1∇F(vi)+∑
v∇F(v)exp(−F(v))
Z(27.120)
=−1
NN∑
i=1∇F(vi)+E[∇F(v)] (27.121)
Plugging in the free energy (Equation 27.117), one can show that
∂
∂wrkF(v)=−vrE[hk|v,θ]=−E[vrhk|v,θ] (27.122)
Hence
∂
∂wrkℓ(θ)=1
NN∑
i=1E[vrhk|v,θ]−E[vrhk|θ] (27.123)
which matches Equation 27.112.

27.7.2.3 Approximating the expectations
We can approximate the expectations needed to evaluate the gradient by performing block
Gibbs sampling, using Equations 27.101 and 27.102. In more detail, we can sample from thejoint distribution p(v,h|θ)as follows: initialize the chain at vv
1(e.g. by setting v1=vifor
some data vector), and then sample from h1∼p(h|v1), then from v2∼p(v|h1), then from
h2∼p(h|v2), etc. See Figure 27.31 for an illustration. Note, however, that we have to wait until
the Markov chain reaches equilibrium (i.e., until it has “burned in”) before we can interpret thesamples as coming from the joint distribution of interest, and this might take a long time.
A faster alternative is to use mean ﬁeld, where we make the approximation E[v
rhk]≈
E[vr]E[hk]. However, since p(v,h)is typically multimodal, this is usually a very poor approx-

rhk]≈
imation, since it will average over different modes (see Section 21.2.2). Furthermore, there is amore subtle reason not to use mean ﬁeld: since the gradient has the form E[v
rhk|v]−E[vrhk],
we see that the negative sign in front means that the method will try to make the variationalbound as loose as possible (Salakhutdinov and Hinton 2009). This explains why earlier attemptsto use mean ﬁeld to learn Boltzmann machines (e.g., (Kappen and Rodriguez 1998)) did not work.
27.7.2.4 Contrastive divergence

The problem with using Gibbs sampling to compute the gradient is that it is slow. We nowpresent a faster method known as contrastive divergence orCD(Hinton 2002). CD was

originally derived by approximating an objective function deﬁned as the difference of two KLdivergences, rather than trying to maximize the likelihood itself. However, from an algorithmicpoint of view, it can be thought of as similar to stochastic gradient descent, except it approxi-mates the “unclamped” expectations with “brief” Gibbs sampling where we initialize each Markovchain at the data vectors. That is, we approximate the gradient for one datavector as follows:
∇
wℓ≈E[
vhT|vi]
−Eq[
vhT]
(27.124)

∇
wℓ≈E[
vhT|vi]
−Eq[
vhT]
(27.124)
whereqcorresponds to the distribution generated by Kup-down Gibbs sweeps, started at vi,
as in Figure 27.31. This is known as CD-K . In more detail, the procedure (for K=1)i sa s

990 Chapter27. Latentvariablemodelsfordiscretedata
follows:
hi∼p(h|vi,θ) (27.125)
v′
i∼p(v|hi,θ) (27.126)
h′
i∼p(h|v′
i,θ) (27.127)
We then make the approximation
Eq[
vhT]
≈vi(h′i)T(27.128)
Such samples are sometimes called fantasy data. We can think of v′
ias the model’s best
attempt at reconstructing viafter being coded and then decoded by the model. This is similar
to the way we train auto-encoders, which are models which try to “squeeze” the data through a
restricted parametric “bottleneck” (see Section 28.3.2).
In practice, it is common to use E[h|v′
i]instead of a sampled value h′
iin the ﬁnal upwards
pass, since this reduces the variance. However, it is not valid to use E[h|vi]instead of sampling
hi∼p(h|vi)in the earlier upwards passes, because then each hidden unit would be able to
pass more than 1 bit of information, so it would not act as much of a bottleneck.
The whole procedure is summarized in Algorithm 3. (Note that we follow the positive gradient

since we are maximizing likelihood.) Various tricks can be used to speed this algorithm up, such
as using a momentum term (Section 8.3.2), using mini-batches, averaging the updates, etc. Suchdetails can be found in (Hinton 2010; Swersky et al. 2010).
Algorithm 27.3: CD-1 training for an RBM with binary hidden and visible units
1Initialize weights W∈RR×Krandomly;
2t:= 0;
4t:=t+1;
5foreachminibatchofsize Bdo
6 Set minibatch gradient to zero, g:=0;
7 foreachcase viintheminibatch do

2t:= 0;
4t:=t+1;
8 Compute μi=E[h|vi,W];
9 Samplehi∼p(h|vi,W);
10 Samplev′
i∼p(v|hi,W);
11 Compute μ′
i=E[h|v′
i,W];
12 Compute gradient ∇W=(vi)(μi)T−(v′
i)(μ′i)T;
13 Accumulate g:=g+∇W;
14 Update parameters W:=W+(αt/B)g
27.7.2.5 Persistent CD
In Section 19.5.5, we presented a technique called stochastic maximum likelihood (SML) for
ﬁtting maxent models. This avoids the need to run MCMC to convergence at each iteration,

27.7. RestrictedBoltzmannmachines(RBMs) 991
by exploiting the fact that the parameters are changing slowly, so the Markov chains will not
be pushed too far from equilibrium after each update (Younes 1989). In other words, there aretwo dynamical processes running at different time scales: the states change quickly, and theparameters change slowly. This algorithm was independently rediscovered in (Tieleman 2008),who called it persistent CD. See Algorithm 3 for the pseudocode.
PCD often works better than CD (see e.g., (Tieleman 2008; Marlin et al. 2010; Swersky et al.
2010)), although CD can be faster in the early stages of learning.
Algorithm 27.4: Persistent CD for training an RBM with binary hidden and visible units
1Initialize weights W∈RD×Lrandomly;
2Initialize chains (vs,hs)S
s=1randomly ;
3fort=1,2,...do
4// Mean ﬁeld updates ;
5foreachcase i=1:Ndo
6 μik=s i g m (vT
iw:,k)
7// MCMC updates ;
8foreachsample s=1:Sdo
9 Generate (vs,hs)by brief Gibbs sampling from old (vs,hs)
10// Parameter updates ;
11g=1

iw:,k)
11g=1
N∑N
i=1vi(μi)T−1
S∑Ss=1vs(hs)T;
12W:=W+αtg;
13Decrease αt
27.7.3 Applications of RBMs
The main application of RBMs is as a building block for deep generative models, which we
discuss in Section 28.2. But they can also be used as substitutes for directed two-layer models.They are particularly useful in cases where inference of the hidden states at test time must befast. We give some examples below.
27.7.3.1 Language modeling and document retrieval

11g=1
N∑N
We can use a categorical RBM to deﬁne a generative model for bag-of-words, as an alternativeto LDA. One subtlety is that the partition function in an undirected models depends on howbig the graph is, and therefore on how long the document is. A solution to this was proposedin (Salakhutdinov and Hinton 2010): use a categorical RBM with tied weights, but multiply thehidden activation bias terms c
kby the document length Lto compensate form the fact that the
observed word-count vector vis larger in magnitude:
E(v,h;θ)≜−K∑
k=1C∑
c=1vchkWc

k=1C∑
c=1vchkWc
k−C∑
c=1vcbc
r−LK∑
k=1hkck (27.129)

992 Chapter27. Latentvariablemodelsfordiscretedata
Data set Number of docs K ¯D St. Dev. Avg. Test perplexity per word (in nats)
Train Test LDA-50 LDA-200 R. Soft-50 Unigram
NIPS 1,690 50 13,649 98.0 245.3 3576 3391 3405 4385
20-news 11,314 7,531 2,000 51.8 70.8 1091 1058 953 1335
Reuters 794,414 10,000 10,000 94.6 69.3 1437 1142 988 2208
Figure 27.32 Comparison of RBM (replicated softmax) and LDA on three corpora. Kis the number of
words in the vocabulary, Dis the average document length, and St. Dev. is the standard deviation of the
document length. Source: (Salakhutdinov and Hinton 2010) .
0.02    0.1     0.4     1.6     6.4     25.6    100 102030405060
Recall (%) Precision (%)Replicated 
Softmax 50−D
LDA 50−D
0.001     0.006     0.051     0.4        1.6       6.4       25.6      100 1020304050
LDA 50−D20-newsgroups Reuters
Figure 27.33 Precision-recall curves for RBM (replicated softmax) and LDA on two corpora. From Figure

LDA 50−D
3 of (Salakhutdinov and Hinton 2010). Used with kind permission of Ruslan Salakhutdinov.
wherevc=∑L
l=1I(yil=c). This is like having a single multinomial node (so we have dropped
thersubscript) with Cstates, where Cis the number of words in the vocabulary. This is
called the replicated softmax model (Salakhutdinov and Hinton 2010), and is an undirected
alternative to mPCA/ LDA.
We can compare the modeling power of RBMs vs LDA by measuring the perplexity on a test
set. This can be approximated using annealing importance sampling (Section 24.6.2). The results

are shown in Figure 27.32. We see that the LDA is signiﬁcantly better than a unigram model,
but that an RBM is signiﬁcantly better than LDA.
Another advantage of the LDA is that inference is fast and exact: just a single matrix-vector
multiply followed by a sigmoid nonlinearity, as in Equation 27.107. In addition to being faster,
the RBM is more accurate. This is illustrated in Figure 27.33, which shows precision-recall curves
for RBMs and LDA on two different corpora. These curves were generated as follows: a query

document from the test set is taken, its similarity to all the training documents is computed,
where the similarity is deﬁned as the cosine of the angle between the two topic vectors, and
then the top Mdocuments are returned for varying M. A retrieved document is considered
relevant if it has the same class label as that of the query’s (this is the only place where labels
are used).

27.7. RestrictedBoltzmannmachines(RBMs) 993
27.7.3.2 RBMs for collaborative ﬁltering
RBMs have been applied to the Netﬂix collaborative ﬁltering competition (Salakhutdinov et al.
2007). In fact, an RBM with binary hidden nodes and categorical visible nodes can slightlyoutperform SVD. By combining the two methods, performance can be further improved. (Thewinning entry in the challenge was an ensemble of many different types of model (Koren 2009a).)
Exercises
Exercise 27.1 Partition function for an RBM
Show how to compute Z(θ)for an RBM with Kbinary hidden nodes and Rbinary observed nodes in
O(R2K)time, assuming K<R.

28 Deep learning
28.1 Introduction
Many of the models we have looked at in this book have a simple two-layer architecture of
the form z→yfor unsupervised latent variable models, or x→yfor supervised models.
However, when we look at the brain, we seem many levels of processing. It is believed that eachlevel is learning features or representations at increasing levels of abstraction. For example, thestandard model of the visual cortex (Hubel and Wiesel 1962; Serre et al. 2005; Ranzato et al.
2007) suggests that (roughly speaking) the brain ﬁrst extracts edges, then patches, then surfaces,then objects, etc. (See e.g., (Palmer 1999; Kandel et al. 2000) for more information about howthe brain might perform vision.)
This observation has inspired a recent trend in machine learning known as deep learning
(see e.g., (Bengio 2009), deeplearning .net, and the references therein), which attempts to

replicate this kind of architecture in a computer. (Note the idea can be applied to non-visionproblems as well, such as speech and language.)
In this chapter, we give a brief overview of this new ﬁeld. However, we caution the reader
that the topic of deep learning is currently evolving very quickly, so the material in this chaptermay soon be outdated.
28.2 Deep generative models

Deep models often have millions of parameters. Acquiring enough labeled data to train suchmodels is diffcult, despite crowd sourcing sites such as Mechanical Turk. In simple settings, suchas hand-written character recognition, it is possible to generate lots of labeled data by makingmodiﬁed copies of a small manually labeled training set (see e.g., Figure 16.13), but it seemsunlikely that this approach will scale to complex scenes.
1
To overcome the problem of needing labeled training data, we will focus on unsupervised

1
learning. The most natural way to perform this is to use generative models. In this section, wediscuss three different kinds of deep generative models: directed, undirected, and mixed.
1. There have been some attempts to use computer graphics and video games to generate realistic-looking images of
complex scenes, and then to use this as training data for computer vision systems. However, often graphics programs
cut corners in order to make perceptually appealing images which are not reﬂective of the natural statistics of real-world
images.

996 Chapter28. Deeplearning
(a) (b)
 (c)
Figure 28.1 Some deep multi-layer graphical models. Observed variables are at the bottom. (a) A directed
model. (b) An undirected model (deep Boltzmann machine). (c) A mixed directed-undirected model (deep
belief net).
28.2.1 Deep directed networks
Perhaps the most natural way to build a deep generative model is to construct a deep directed
graphical model, as shown in Figure 28.1(a). The bottom level contains the observed pixels (or
whatever the data is), and the remaining layers are hidden. We have assumed just 3 layers for
notational simplicity. The number and size of layers is usually chosen by hand, although one
can also use non-parametric Bayesian methods (Adams et al. 2010) or boosting (Chen et al. 2010)
to infer the model structure.
We shall call models of this form deep directed networks or DDNs. If all the nodes are
binary, and all CPDs are logistic functions, this is called a sigmoid belief net (Neal 1992). In

this case, the model deﬁnes the following joint distribution:
p(h1,h2,h3,v|θ)=∏
iBer(vi|sigm(hT
1w0i))∏
jBer(h1j|sigm(hT
2w1j)) (28.1)
∏
kBer(h2k|sigm(hT
3w2k))∏
lBer(h3l|w3l) (28.2)
Unfortunately, inference in directed models such as these is intractable because the posterior
on the hidden nodes is correlated due to explaining away. One can use fast mean ﬁeld approxi-
mations (Jaakkola and Jordan 1996a; Saul and Jordan 2000), but these may not be very accurate,

1w0i))∏
∏
3w2k))∏
since they approximate the correlated posterior with a factorial posterior. One can also use
MCMC inference (Neal 1992; Adams et al. 2010), but this can be quite slow because the variables
are highly correlated. Slow inference also results in slow learning.
28.2.2 Deep Boltzmann machines
A natural alternative to a directed model is to construct a deep undirected model. For example,
we can stack a series of RBMs on top of each other, as shown in Figure 28.1(b). This is known

as adeep Boltzmann machine orDBM(Salakhutdinov and Hinton 2009). If we have 3 hidden
layers, the model is deﬁned as follows:
p(h1,h2,h3,v|θ)=1
Z(θ)exp⎛
⎝∑
ijvih1jW1ij+∑
jkh1jh2jW2jk+∑
klh2kh3lW3kl⎞
⎠(28.3)

28.2. Deepgenerativemodels 997
where we are ignoring constant offset or bias terms.
The main advantage over the directed model is that one can perform efficient block (layer-
wise) Gibbs sampling, or block mean ﬁeld, since all the nodes in each layer are conditionally
independent of each other given the layers above and below (Salakhutdinov and Larochelle2010). The main disadvantage is that training undirected models is more difficult, because of thepartition function. However, below we will see a greedy layer-wise strategy for learning deepundirected models.
28.2.3 Deep belief networks
An interesting compromise is to use a model that is partially directed and partially undirected.In particular, suppose we construct a layered model which has directed arrows, except at thetop, where there is an undirected bipartite graph, as shown in Figure 28.1(c). This model isknown as a deep belief network (Hinton et al. 2006) or DBN.
2If we have 3 hidden layers, the
model is deﬁned as follows:

1w1i)∏
jBer(h1j|sigm(hT2w2j) (28.4)
1
Z(θ)exp(∑
klh2kh3lW3kl)
(28.5)
Essentially the top two layers act as an associative memory, and the remaining layers then
generate the output.
The advantage of this peculiar architecture is that we can infer the hidden states in a
fast, bottom-up fashion. To see why, suppose we only have two hidden layers, and thatW
2=WT
1, so the second level weights are tied to the ﬁrst level weights (see Figure 28.2(a)).
This deﬁnes a model of the form p(h1,h2,v|W1). One can show that the distribution
p(h1,v|W1)=∑

1
Z(θ)exp(∑
(28.5)
2=WT
h2p(h1,h2,v|W1)has the form p(h1,v|W1)=1
Z(W 1)exp(vTW1h1),
which is equivalent to an RBM. Since the DBN is equivalent to the RBM as far as p(h1,v|W1)
is concerned, we can infer the posterior p(h1|v,W1)in the DBN exactly as in the RBM. This
posterior is exact, even though it is fully factorized.
Now the only way to get a factored posterior is if the prior p(h1|W1)is acomplementary
prior. This is a prior which, when multiplied by the likelihood p(v|h1), results in a perfectly

factored posterior. Thus we see that the top level RBM in a DBN acts as a complementary priorfor the bottom level directed sigmoidal likelihood function.
If we have multiple hidden levels, and/or if the weights are not tied, the correspondence

between the DBN and the RBM does not hold exactly any more, but we can still use the factoredinference rule as a form of approximate bottom-up inference. Below we show that this is a validvariational lower bound. This bound also suggests a layer-wise training strategy, that we willexplain in more detail later. Note, however, that top-down inference in a DBN is not tractable,so DBNs are usually only used in a feedforward manner.

2. Unforuntately the acronym “DBN” also stands for “dynamic Bayes net” (Section 17.6.7). Geoff Hinton, who invented
deep belief networks, has suggested the acronyms DeeBNsandDyBNsfor these two different meanings. However, this
terminology is non-standard.

998 Chapter28. Deeplearning
(a)
 (b)
 (c)
Figure 28.2 (a) A DBN with two hidden layers and tied weights that is equivalent to an RBM. Source:
Figure 2.2 of (Salakhutdinov 2009). (b) A stack of RBMs trained greedily. (c) The corresponding DBN.
Source: Figure 2.3 of (Salakhutdinov 2009). Used with kind permission of Ruslan Salakhutdinov.
28.2.4 Greedy layer-wise learning of DBNs
The equivalence between DBNs and RBMs suggests the following strategy for learning a DBN.
• Fit an RBM to learn W1using methods described in Section 27.7.2.
• Unroll the RBM into a DBN with 2 hidden layers, as in Figure 28.2(a). Now “freeze” the
directed weights W1and letW2be “untied” so it is no longer forced to be equal to WT
1.
We will now learn a better prior for p(h1|W2)by ﬁtting a second RBM. The input data to
this new RBM is the activation of the hidden units E[h1|v,W1]which can be computed
using a factorial approximation.
• Continue to add more hidden layers until some stopping criterion is satisiﬁed, e.g., you run

1.
out of time or memory, or you start to overﬁt the validation set. Construct the DBN from
these RBMs, as illustrated in Figure 28.2(c).
One can show (Hinton et al. 2006) that this procedure always increases a lower bound the
observed data likelihood. Of course this procedure might result in overﬁtting, but that is a
different matter.
In practice, we want to be able to use any number of hidden units in each level. This means
we will not be able to initialize the weights so that Wℓ=WT
ℓ−1. This voids the theoretical

guarantee. Nevertheless the method works well in practice, as we will see. The method can also
be extended to train DBMs in a greedy way (Salakhutdinov and Larochelle 2010).
After using the greedy layer-wise training strategy, it is standard to “ﬁne tune” the weights,
using a technique called backﬁtting . This works as follows. Perform an upwards sampling pass
to the top. Then perform brief Gibbs sampling in the top level RBM, and perform a CD update
of the RBM parameters. Finally, perform a downwards ancestral sampling pass (which is an

approximate sample from the posterior), and update the logistic CPD parameters using a small
gradient step. This is called the up-down procedure (Hinton et al. 2006). Unfortunately this
procedure is very slow.

28.3. Deepneuralnetworks 999
28.3 Deep neural networks
Given that DBNs are often only used in a feed-forward, or bottom-up, mode, they are effectively
acting like neural networks. In view of this, it is natural to dispense with the generative storyand try to ﬁt deep neural networks directly, as we discuss below. The resulting training methodsare often simpler to implement, and can be faster.
Note, however, that performance with deep neural nets is sometimes not as good as with
probabilistic models (Bengio et al. 2007). One reason for this is that probabilistic models supporttop-down inference as well as bottom-up inference. (DBNs do not support efficient top-downinference, but DBMs do, and this has been shown to help (Salakhutdinov and Larochelle 2010).)Top-down inference is useful when there is a lot of ambiguity about the correct interpretationof the signal.
It is interesting to note that in the mammalian visual cortex, there are many more feedback

connections than there are feedforward connections (see e.g., (Palmer 1999; Kandel et al. 2000)).The role of these feedback connections is not precisely understood, but they presumably providecontextual prior information (e.g., coming from the previous “frame” or retinal glance) whichcan be used to disambiguate the current bottom-up signals (Lee and Mumford 2003).
Of course, we can simulate the effect of top-down inference using a neural network. However
the models we discuss below do not do this.
28.3.1 Deep multi-layer perceptrons

Many decision problems can be reduced to classiﬁcation, e.g., predict which object (if any) ispresent in an image patch, or predict which phoneme is present in a given acoustic featurevector. We can solve such problems by creating a deep feedforward neural network or multi-layer perceptron (MLP), as in Section 16.5, and then ﬁtting the parameters using gradient descent(aka back-propagation).
Unfortunately, this method does not work very well. One problem is that the gradient becomes
weaker the further we move away from the data; this is known as the “vanishing gradient”problem (Bengio and Frasconi 1995). A related problem is that there can be large plateaus inthe error surface, which cause simple ﬁrst-order gadient-based methods to get stuck (Glorot andBengio 2010).

Consequently early attempts to learn deep neural networks proved unsuccesful. Recently there
has been some progress, due to the adoption of GPUs (Ciresan et al. 2010) and second-orderoptimization algorithms (Martens 2010). Nevertheless, such models remain difficult to train.
Below we discuss a way to initialize the parameters using unsupervised learning; this is called

generative pre-training. The advantage of performing unsupervised learning ﬁrst is that themodel is forced to model a high-dimensional response, namely the input feature vector, ratherthan just predicting a scalar response. This acts like a data-induced regularizer, and helpsbackpropagation ﬁnd local minima with good generalization properties (Erhan et al. 2010; Glorotand Bengio 2010).

/g5/g2
/g15/g27/g29/g25/g33/g34/g29/g27/g29/g26/g6/g6
/g4/g4/g5/g5/g6/g19
/g7/g5/g19
/g8/g4/g19
/g9/g3/g19
/g10
/g14/g29/g23/g30/g24/g25/g32/g3/g4/g5/g5/g2
/g6/g5/g4/g19/g3/g19
/g12/g30/g24/g25/g1/g28/g22/g35/g25/g32/g13/g25/g23/g30/g24/g25/g32
/g18/g11/g16/g19/g30/g31
Figure 28.3 Training a deep autoencoder. (a) First we greedily train some RBMs. (b) Then we construct
the auto-encoder by replicating the weights. (c) Finally we ﬁne-tune the weights using back-propagation.
From Figure 1 of (Hinton and Salakhutdinov 2006). Used with kind permission of Ruslan Salakhutdinov.
28.3.2 Deep auto-encoders
Anauto-encoder is a kind of unsupervised neural network that is used for dimensionality
reduction and feature discovery. More precisely, an auto-encoder is a feedforward neural network
that is trained to predict the input itself. To prevent the system from learning the trivial identity
mapping, the hidden layer in the middle is usually constrained to be a narrow bottleneck . The

system can minimize the reconstruction error by ensuring the hidden units capture the most
relevant aspects of the data.
Suppose the system has one hidden layer, so the model has the form v→h→v. Further,
suppose all the functions are linear. In this case, one can show that the weights to the K
hidden units will span the same subspace as the ﬁrst Kprincipal components of the data
(Karhunen and Joutsensalo 1995; Japkowicz et al. 2000). In other words, linear auto-encoders are

equivalent to PCA. However, by using nonlinear activation functions, one can discover nonlinear
representations of the data.
More powerful representations can be learned by using deep auto-encoders . Unfortunately
training such models using back-propagation does not work well, because the gradient signal
becomes too small as it passes back through multiple layers, and the learning algorithm often
gets stuck in poor local minima.
One solution to this problem is to greedily train a series of RBMs and to use these to initialize

an auto-encoder, as illustrated in Figure 28.3. The whole system can then be ﬁne-tuned using
backprop in the usual fashion. This approach, ﬁrst suggested in (Hinton and Salakhutdinov

28.4. Applicationsofdeepnetworks 1001
2000 top-level units
500 units 
500 units 
28 x 28    
pixel 
image10 label units
This could be the 
top level of 
another sensory 
pathway
(a)
 (b)
Figure 28.4 (a) A DBN architecture for classifying MNIST digits. Source: Figure 1 of (Hinton et al. 2006).
Used with kind permission of Geoff Hinton. (b) These are the 125 errors made by the DBN on the 10,000
test cases of MNIST. Above each image is the estimated label. Source: Figure 6 of (Hinton et al. 2006).
Used with kind permission of Geoff Hinton. Compare to Figure 16.15.
2006), works much better than trying to ﬁt the deep auto-encoder directly starting with random
weights.
28.3.3 Stacked denoising auto-encoders
A standard way to train an auto-encoder is to ensure that the hidden layer is narrower than the
visible layer. This prevents the model from learning the identity function. But there are other
ways to prevent this trivial solution, which allow for the use of an over-complete representation.

weights.
One approach is to impose sparsity constraints on the activation of the hidden units (Ranzato
et al. 2006). Another approach is to add noise to the inputs; this is called a denoising auto-
encoder (Vincent et al. 2010). For example, we can corrupt some of the inputs, for example
by setting them to zero, so the model has to learn to predict the missing entries. This can be
shown to be equivalent to a certain approximate form of maximum likelihood training (known
as score matching) applied to an RBM (Vincent 2011).

Of course, we can stack these models on top of each other to learn a deep stacked denoising
auto-encoder, which can be discriminatively ﬁne-tuned just like a feedforward neural network,
if desired.
28.4 Applications of deep networks
In this section, we mention a few applications of the models we have been discussing.
28.4.1 Handwritten digit classiﬁcation using DBNs
Figure 28.4(a) shows a DBN (from (Hinton et al. 2006)) consisting of 3 hidden layers. The visible
layer corresponds to binary images of handwritten digits from the MNIST data set. In addition,

the top RBM is connected to a softmax layer with 10 units, representing the class label.

1002 Chapter28. Deeplearning
(a)Legal/Judicial Leading          
Economic         Indicators       European Community Monetary/Economic  
Accounts/Earnings Interbank Markets
Government Borrowings Disasters and Accidents     Energy Markets
(b)
Figure 28.5 2d visualization of some bag of words data from the Reuters RCV1-v2 corpus. (a) Results of
using LSA. (b) results of using a deep auto-encoder. Source: Figure 4 of (Hinton and Salakhutdinov 2006).
Used with kind permission of Ruslan Salakhutdinov.
The ﬁrst 2 hidden layers were trained in a greedy unsupervised fashion from 50,000 MNIST
digits, using 30 epochs (passes over the data) and stochastic gradient descent, with the CD

heuristic. This process took “a few hours per layer” (Hinton et al. 2006, p1540). Then the toplayer was trained using as input the activations of the lower hidden layer, as well as the classlabels. The corresponding generative model had a test error of about 2.5%. The network weightswere then carefully ﬁne-tuned on all 60,000 training images using the up-down procedure. Thisprocess took “about a week” (Hinton et al. 2006, p1540). The model can be used to classifyby performing a deterministic bottom-up pass, and then computing the free energy for thetop-level RBM for each possible class label. The ﬁnal error on the test set was about 1.25%. Themisclassiﬁed examples are shown in Figure 28.4(b).
This was the best error rate of any method on the permutation-invariant version of MNIST

at that time. (By permutation-invariant, we mean a method that does not exploit the fact thatthe input is an image. Generic methods work just as well on permuted versions of the input(see Figure 1.5), and can therefore be applied to other kinds of datasets.) The only other methodthat comes close is an SVM with a degree 9 polynomial kernel, which has achieved an errorrate of 1.4% (Decoste and Schoelkopf 2002). By way of comparison, 1-nearest neighbor (usingall 60,000 examples) achieves 3.1% (see mnist1NNdemo ). This is not as good, although 1-NN is
much simpler.
3
28.4.2 Data visualization and feature discovery using deep auto-encoders
Deep autoencoders can learn informative features from raw data. Such features are often usedas input to standard supervised learning methods.
To illustrate this, consider ﬁtting a deep auto-encoder with a 2d hidden bottleneck to some

3
3. One can get much improved performance on this task by exploiting the fact that the input is an image. One way to do
this is to create distorted versions of the input, adding small shifts and translations (see Figure 16.13 for some examples).
Applying this trick reduced the SVM error rate to 0.56%. Similar error rates can be achieved using convolutional neural
networks (Section 16.5.1) trained on distorted images ((Simard et al. 2003) got 0.4%). However, the point of DBNs is that
they offer a way to learn such prior knowledge, without it having to be hand-crafted.

28.4. Applicationsofdeepnetworks 1003
/g6/g5/g7/g1 /g6/g5/g8/g1 /g6/g5/g10/g1 /g6/g5/g13/g1 /g7/g5/g12/g1 /g9/g5/g8/g1 /g12/g5/g10/g1 /g7/g8/g5/g13 /g8/g11/g5/g12 /g11/g7/g5/g8 /g7/g6/g6/g1/g7/g6/g8/g6/g9/g6/g10/g6/g11/g6
/g18/g23/g21/g20/g27/g27/g1/g3/g2/g4/g1/g17/g31/g23/g21/g26/g32/g26/g29/g28/g1/g3/g2/g4/g14/g34/g33/g29/g23/g28/g21/g29/g22/g23/g31/g1/g7/g6/g15
/g16/g19/g14/g1/g7/g6/g15
/g16/g19/g14/g1/g11/g6/g15
/g14/g34/g33/g29/g23/g28/g21/g29/g22/g23/g31/g1/g7/g6/g15
/g30/g31/g26/g29/g31/g1/g33/g29/g1/g24/g26/g28/g23 /g33/g34/g28/g26/g28/g25
Figure 28.6 Precision-recall curves for document retrieval in the Reuters RCV1-v2 corpus. Source: Figure
3.9 of (Salakhutdinov 2009). Used with kind permission of Ruslan Salakhutdinov.
text data. The results are shown in Figure 28.5. On the left we show the 2d embedding produced

by LSA (Section 27.2.2), and on the right, the 2d embedding produced by the auto-encoder. It isclear that the low-dimensional representation created by the auto-encoder has captured a lot ofthe meaning of the documents, even though class labels were not used.
4
Note that various other ways of learning low-dimensional continuous embeddings of words
have been proposed. See e.g., (Turian et al. 2010) for details.
28.4.3 Information retrieval using deep auto-encoders (semantic hashing)

4
In view of the sucess of RBMs for information retrieval discussed in Section 27.7.3.1, it is naturalto wonder if deep models can do even better. In fact they can, as is shown in Figure 28.6.
More interestingly, we can use a binary low-dimensional representation in the middle layer

4
of the deep auto-encoder, rather than a continuous representation as we used above. Thisenables very fast retrieval of related documents. For example, if we use a 20-bit code, wecan precompute the binary representation for all the documents, and then create a hash-tablemapping codewords to documents. This approach is known as semantic hashing, since the
binary representation of semantically similar documents will be close in Hamming distance.

For the 402,207 test documents in Reuters RCV1-v2, this results in about 0.4 documents per
entry in the table. At test time, we compute the codeword for the query, and then simply retrievethe relevant documents in constant time by looking up the contents of the relevant address in
memory. To ﬁnd other other related documents, we can compute all the codewords within a
4. Some details. Salakhutdinov and Hinton used the Reuters RCV1-v2 data set, which consists of 804,414 newswire

articles, manually classiﬁed into 103 topics. They represent each document by counting how many times each of the top
2000 most frequent words occurs. They trained a deep auto-encoder with 2000-500-250-125-2 layers on half of the data.
The 2000 visible units use a replicated softmax distribution, the 2 hidden units in the middle layer have a Gaussian
distribution, and the remaining units have the usual Bernoulli-logistic distribution. When ﬁne tuning the auto-encoder,

a cross-entropy loss function (equivalent to maximum likelihood under a multinoulli distribution) was used. See (Hinton
and Salakhutdinov 2006) for further details.

1004 Chapter28. Deeplearning
x1 x2 x3 x4h1
1h12h13 h21h22h23
w1w1w1
w2w2w2
Figure 28.7 A small 1d convolutional RBM with two groups of hidden units, each associated with a ﬁlter
of size 2. h1
1andh21are two different “views” of the data in the ﬁrst window, (x1,x2). The ﬁrst view is
computed using the ﬁlter w1, the second view using ﬁlter w2. Similarly, h12andh22are the views of the
data in the second window, (x2,x3), computed using w1andw2respectively.
Hamming distance of, say, 4. This results in retrieving about 6196×0.4≈2500documents5.
The key point is that the total time is independent of the size of the corpus.
Of course, there are other techniques for fast document retrieval, such as inverted indices.
These rely on the fact that individual words are quite informative, so we can simply intersect all

the documents that contain each word. However, when performing image retrieval, it is clear thatwe do not want to work at the pixel level. Recently (Krizhevsky and Hinton 2010) showed thata deep autoencoder could learn a good semantic hashing function that outperformed previoustechniques (Torralba et al. 2008; Weiss et al. 2008) on the 80 million tiny images dataset. Itis hard to apply inverted indexing techniques to real-valued data (although one could imaginevector quantizing image patches).
28.4.4 Learning audio features using 1d convolutional DBNs

To apply DBNs to time series of unbounded length, it is necessary to use some form of parametertying. One way to do this is to use convolutional DBNs (Lee et al. 2009; Desjardins and Bengio
2008), whichuseconvolutional RBMsas theirbasic unit. These modelsare agenerative versionofconvolutional neural nets discussed in Section 16.5.1. The basic idea is illustrated in Figure 28.7.The hidden activation vector for each group is computed by convolving the input vector withthat group’s ﬁlter (weight vector or matrix). In other words, each node within a hidden groupis a weighted combination of a subset of the inputs. We compute the activaton of all thehidden nodes by “sliding” this weight vector over the input. This allows us to model translation
invariance, since we use the same weights no matter where in the input vector the patternoccurs.
6Each group has its own ﬁlter, corresponding to its own pattern detector.
5. Note that 6196 =∑4
k=0(20
k)

k=0(20
k)
is the number of bit vectors that are up to a Hamming distance of 4 away.
6. It is often said that the goal of deep learnng is to discover invariant features, e.g., a representation of an object
that does not change even as nuisance variables, such as the lighting, do change. However, sometimes these so-called
“nuisance variables” may be the variables of interest. For example if the task is to determine if a photograph was taken
in the morning or the evening, then lighting is one of the more salient features, and object identity may be less relevant.
As always, one task’s “signal” is another task’s “noise”, so it unwise to “throw away” apparently irrelevant information

28.5. Discussion 1005
More formally, for binary 1d signals, we can deﬁne the full conditionals in a convolutional
RBM as follows (Lee et al. 2009):
p(hk
t=1|v)=s i g m ( ( wk⊗v)t+bt) (28.6)
p(vs=1|h) = sigm(∑
k(wk⊗hk)s+cs) (28.7)
wherewkis the weight vector for group k,btandcsare bias terms, and a⊗brepresents the
convolution of vectors aandb.
It is common to add a max pooling layer as well as a convolutional layer, which computes
a local maximum over the ﬁltered response. This allows for a small amount of translation

p(hk
invariance. It also reduces the size of the higher levels, which speeds up computation consider-ably. Deﬁning this for a neural network is simple, but deﬁning this in a way which allows forinformation ﬂow backwards as well as forwards is a bit more involved. The basic idea is similarto a noisy-OR CPD (Section 10.2.3), where we deﬁne a probabilistic relationship between the maxnode and the parts it is maxing over. See (Lee et al. 2009) for details. Note, however, that thetop-down generative process will be difficult, since the max pooling operation throws away somuch information.

(Lee et al. 2009) applies 1d convolutional DBNs of depth 2 to auditory data. When the input
consists of speech signals, the method recovers a representation that is similar to phonemes.When applied to music classiﬁcation and speaker identiﬁcation, their method outperforms tech-niques using standard features such as MFCC. (All features were fed into the same discriminativeclassiﬁer.)
In (Seide et al. 2011), a deep neural net was used in place of a GMM inside a conventional
HMM. The use of DNNs signiﬁcantly improved performance on conversational speech recogni-tion. In an interview, the tech lead of this project said “historically, there have been very fewindividual technologies in speech recognition that have led to improvements of this magnitude”.
7
28.4.5 Learning image features using 2d convolutional DBNs
We can extend a convolutional DBN from 1d to 2d in a straightforward way (Lee et al. 2009), asillustrated in Figure 28.8. The results o fa3l a y e rs y s t e mt r ained on four classes of visual objects

7
(cars, motorbikes, faces and airplanes) from the Caltech 101 dataset are shown in Figure 28.9.We only show the results for layers 2 and 3, because layer 1 learns Gabor-like ﬁlters that arevery similar to those learned by sparse coding, shown in Figure 13.21(b). We see that layer 2 haslearned some generic visual parts that are shared amongst object classes, and layer 3 seems tohave learned ﬁlters that look like grandmother cells, that are speciﬁc to individual object classes,and in some cases, to individual objects.
28.5 Discussion
So far, we have been discussing models inspired by low-level processing in the brain. Thesemodels have produced useful features for simple classiﬁcation tasks. But can this pure bottom-up
too early.

too early.
7. Source: http://research .microsoft .com/en-us/news/features/speechrecognition-082911 .aspx.

1006 Chapter28. Deeplearning
Figure 28.8 A 2d convolutional RBM with max-pooling layers. The input signal is a stack of 2d images
(e.g., color planes). Each input layer is passed through a different set of ﬁlters. Each hidden unit is
obtained by convolving with the appropriate ﬁlter, and then summing over the input planes. The ﬁnal layer
is obtained by computing the local maximum within a small window. Source: Figure 1 of (Chen et al.
2010) . Used with kind permission of Bo Chen.
faces, cars, airplanes, motorbikes
(a)
 (b)
Figure 28.9 Visualization of the ﬁlters learned by a convolutional DBN in layers two and three. Source:
Figure 3 of (Lee et al. 2009). Used with kind permission of Honglak Lee.
approach scale to more challenging problems, such as scene interpretation or natural language
understanding?
To put the problem in perspective, consider the DBN for handwritten digit classiﬁcation in
Figure 28.4(a). This has about 1.6M free parameters ( 28×28×500+500×500+510×2000 =

(a)
 (b)
1,662,000). Although this is a lot, it is tiny compared to the number of neurons in the brain.
As Hinton says,
This is about as many parameters as 0.002 cubic millimetres of mouse cortex, and several
hundred networks of this complexity could ﬁt within a single voxel of a high-resolution
fMRI scan. This suggests that much bigger networks may be required to compete with
human shape recognition abilities. — (Hinton et al. 2006, p1547).

To scale up to more challenging problems, various groups are using GPUs (see e.g., (Raina
et al. 2009)) and/or parallel computing. But perhaps a more efficient approach is to work at a
higher level of abstraction, where inference is done in the space of objects or their parts, rather

28.5. Discussion 1007
than in the space of bits and pixels. That is, we want to bridge the signal-to-symbol divide,
where by “symbol” we mean something atomic, that can be combined with other symbols in a
compositional way.
The question of how to convert low level signals into a more structured/ “semantic” represen-
tation is known as the symbol grounding problem (Harnard 1990). Traditionally such symbols
are associated with words in natural language, but it seems unlikely we can jump directly fromlow-level signals to high-level semantic concepts. Instead, what we need is an intermediate levelof symbolic or atomic parts.
A very simple way to create such parts from real-valued signals, such as images, is to apply
vector quantization. This generates a set of visual words. These can then be modelled using
some of the techniques from Chapter 27 for modeling bags of words. Such models, however, arestill quite “shallow”.
It is possible to deﬁne, and learn, deep models which use discrete latent parts. Here we just

mention a few recent approaches, to give a ﬂavor of the possibilites. (Salakhutdinov et al. 2011)combine RBMs with hierarchical latent Dirichlet allocation methods, trained in an unsupervisedway. (Zhu et al. 2010) use latent and-or graphs, trained in a manner similar to a latent structuralSVM. A similar approach, based on grammars, is described in (Girshick et al. 2011). What isinteresting about these techniques is that they apply data-driven machine learning methodsto rich structured/symbolic “AI-style” models. This seems like a promising future direction formachine learning.

Notation
It is very difficult to come up with a single, consistent notation to cover the wide variety of
data, models and algorithms that we discuss. Furthermore, conventions differ between machinelearning and statistics, and between different books and papers. Nevertheless, we have triedto be as consistent as possible. Below we summarize most of the notation used in this book,although individual sections may introduce new notation. Note also that the same symbol mayhave different meanings depending on the context, although we try to avoid this where possible.
General math notation
Symbol Meaning
⌊x⌋ Floor ofx, i.e., round down to nearest integer
⌈x⌉ Ceiling of x, i.e., round up to nearest integer
x⊗y Convolution of xandy
x⊙y Hadamard (elementwise) product of xandy
a∧b logical AND
a∨b logical OR
¬a logical NOT
I(x) Indicator function, I(x)=1ifxis true, else I(x)=0
∞ Inﬁnity
→ Tends towards, e.g., n→∞
∝ Proportional to, so y=axcan be written as y∝x
|x| Absolute value

∞ Inﬁnity
|S| Size (cardinality) of a set
n! Factorial function
∇ Vector of ﬁrst derivatives
∇2Hessian matrix of second derivatives
≜ Deﬁned as
O(·) Big-O: roughly means order of magnitude
R The real numbers
1:n Range (Matlab convention): 1:n={1,2,...,n}
≈ Approximately equal to
argmaxxf(x)Argmax: the value xthat maximizes f

1010 Notation
B(a,b) Beta function, B(a,b)=Γ(a)Γ(b)
Γ(a+b)
B(α) Multivariate beta function,∏
kΓ(αk)
Γ(∑
kαk) (n
k)
nchoosek, equal to n!/(k!(n−k)!)
δ(x) Dirac delta function, δ(x)=∞ifx=0, elseδ(x)=0
δij Kronecker delta, equals 1 if i=j, otherwise equals 0
δx(y) Kronecker delta, equals 1 if x=y, otherwise equals 0
exp(x) Exponential function ex
Γ(x) Gamma function, Γ(x)=∫∞
0ux−1e−udu
Ψ(x) Digamma function, Ψ(x)=d
dxlogΓ(x)
X A set from which values are drawn (e.g., X=RD)
Linear algebra notation
We use boldface lowercase to denote vectors, such as a, and boldface uppercase to denote
matrices, such as A. Vectors are assumed to be column vectors, unless noted otherwise.
A≻0 Ais a positive deﬁnite matrix
tr(A) Trace of a matrix
det(A) Determinant of matrix A
|A| Determinant of matrix A
A−1I n v e r s eo fam a t r i x
A†Pseudo-inverse of a matrix
ATTranspose of a matrix
aTTranspose of a vector
diag(a) Diagonal matrix made from vector a
diag(A)Diagonal vector extracted from matrix A

IorId Identity matrix of size d×d(ones on diagonal, zeros off)
1or1d Vector of ones (of length d)
0or0d Vector of zeros (of length d)
||x||=||x||2Euclidean or ℓ2norm√∑d
j=1x2
j
||x||1 ℓ1norm∑d
j=1|xj|
A:,j j’th column of matrix
Ai,: transpose of i’th row of matrix (a column vector)
Aij Element(i,j)of matrix A
x⊗y Tensor product of xandy
Probability notation
We denote random and ﬁxed scalars by lower case, random and ﬁxed vectors by bold lower case,

j=1x2
j
j=1|xj|
and random and ﬁxed matrices by bold upper case. Occastionally we use non-bold upper caseto denote scalar random variables. Also, we use p()for both discrete and continuous random

Notation 1011
X⊥YX is independent of Y
X̸⊥YX is not independent of Y
X⊥Y|ZX is conditionally independent of YgivenZ
X̸⊥Y|ZX is not conditionally independent of YgivenZ
X∼pX is distributed according to distribution p
α Parameters of a Beta or Dirichlet distribution
cov[x] Covariance of x
E[X] Expected value of X
Eq[X] Expected value of Xwrt distribution q
H(X)orH(p)Entropy of distribution p(X)
I(X;Y) Mutual information between XandY
KL(p||q) KL divergence from distribution ptoq
ℓ(θ) Log-likelihood function
L(θ,a) Loss function for taking action awhen true state of nature is θ
λ Precision (inverse variance) λ=1/σ2
Λ Precision matrix Λ=Σ−1
mode[X] Most probable value of X
μ Mean of a scalar distribution
μ Mean of a multivariate distribution
p(x) Probability density or mass function
p(x|y) Conditional probability density of xgiveny
Φ cdf of standard normal
φ pdf of standard normal
π multinomial parameter vector, Stationary distribution of Markov chain
ρ Correlation coefficient

sigm(x) Sigmoid (logistic) function,1
1+e−x
σ2Variance
Σ Covariance matrix
var[x] Variance of x
ν Degrees of freedom parameter
Z Normalization constant of a probability distribution
Machine learning/statistics notation
In general, we use upper case letters to denote constants, such as C,D,K,N,S,T, etc. We
use lower case letters as dummy indexes of the appropriate range, such as c=1:Cto index
classes,j=1:D to index input features, k=1:K to index states or clusters, s=1:Sto

1+e−x
index samples, t=1:Tto index time, etc. To index data cases, we use the notation i=1:N,
although the notation n=1:Nis also widely used.
We usexto represent an observed data vector. In a supervised problem, we use yoryto
represent the desired output label. We use zto represent a hidden variable. Sometimes we also
useqto represent a hidden discrete variable.

1012 Notation
C Number of classes
D Dimensionality of data vector (number of features)
R Number of outputs (response variables)
D Training dataD={xi|i=1:N}orD={(xi,yi)|i=1:N}
Dtest Test data
J(θ) Cost function
K Number of states or dimensions of a variable (often latent)
κ(x,y)Kernel function
K Kernel matrix
λ Strength of ℓ2orℓ1regularizer
N Number of data cases
Nc Number of examples of class c,Nc=∑N
n=1I(yn=c)
φ(x) Basis function expansion of feature vector x
Φ Basis function expansion of design matrix X
q() Approximate or proposal distribution
Q(θ,θold)Auxiliary function in EM
S Number of samples
T Length of a sequence
T(D)Test statistic for data
T Transition matrix of Markov chain
θ Parameter vector
θ(s)s’th sample of parameter vector
ˆθ Estimate (usually MLE or MAP) of θ
ˆθML Maximum likelihood estimate of θ
ˆθMAP MAP estimate of θ
θ Estimate (usually posterior mean) of θ
w Vector of regression weights (called βin statistics)
W Matrix of regression weights

xij Component (i.e., feature) jof data case i,f o ri=1:N,j=1:D
xi Training case, i=1:N
X Design matrix of size N×D
x Empirical mean x=1
N∑Ni=1xi
˜x Future test case
x∗ Future test case
y Vector of all training labels y=(y1,...,y N)
zij Latent component jfor casei
Graphical model notation
In graphical models, we index nodes by s,t,u∈V, and states by i,j,k∈X.

Notation 1013
s∼tNodesis connected to node t
bel Belief function
C Cliques of a graph
chjChild of node ji naD A G
descjDescendants of node ji naD A G
G A graph
E Edges of a graph
mbtMarkov blanket of node t
nbdtNeighborhood of node t
patParents of node ti naD A G
predtPredecessors of node tin a DAG wrt some ordering
ψc(xc)Potential function for clique c
S Separators of a graph
θsjkprob. node sis in state kgiven its parents are in states j
V Nodes of a graph

1014 Notation
List of commonly used abbreviations
Abbreviation Meaning
cdf Cumulative distribution function

CPD Conditional probability distributionCPT Conditional probability tableCRF Conditional random ﬁeldDAG Directed acyclic graphicDGM Directed graphical modelEB Empirical BayesEM Expectation maximization algorithmEP Expectation propagationGLM Generalized linear modelGMM Gaussian mixture modelHMM Hidden Markov modeliid Independent and identically distributediff If and only ifKL Kullback Leibler divergenceLDS Linear dynamical systemLHS Left hand side (of an equation)MAP Maximum A Posterior estimateMCMC Markov chain Monte CarloMH Metropolis HastingsMLE Maximum likelihood estimateMPM Maximum of Posterior MarginalsMRF Markov random ﬁeldMSE Mean squared errorNLL Negative log likelihoodOLS Ordinary least squarespd Positive deﬁnite (matrix)pdf Probability density functionpmf Probability mass functionRBPF Rao-Blackwellised particle ﬁlterRHS Right hand side (of an equation)RJMCMC Reversible jump MCMCRSS Residual sum of squaresSLDS Switching linear dynamical systemSSE Sum of squared errorsUGM Undirected graphical modelVB

Monte CarloMH Metropolis HastingsMLE Maximum likelihood estimateMPM Maximum of Posterior MarginalsMRF Markov random ﬁeldMSE Mean squared errorNLL Negative log likelihoodOLS Ordinary least squarespd Positive deﬁnite (matrix)pdf Probability density functionpmf Probability mass functionRBPF Rao-Blackwellised particle ﬁlterRHS Right hand side (of an equation)RJMCMC Reversible jump MCMCRSS Residual sum of squaresSLDS Switching linear dynamical systemSSE Sum of squared errorsUGM Undirected graphical modelVB Variational Bayeswrt With respect to

Abend, K., T. J. Harley, and L. N. Kanal
(1965). Classiﬁcation of Binary Ran-
dom Patterns. IEEETransactionson
InformationTheory 11(4), 538–544.
Ackley, D., G. Hinton, and T. Sejnowski
(1985). A learning algorithm forboltzmann machines. Cognitive
Science 9, 147–169.
Adams, R. P., H. Wallach, and
Z. Ghahramani (2010). Learning thestructure of deep sparse graphicalmodels. InAI/Statistics.
Aggarwal, D. and S. Merugu (2007).
Predictive discrete latent factormodels for large scale dyadic data.
InProc. of the Int’l Conf. on Knowl-
edgeDiscoveryandDataMining.
Ahmed, A. and E. Xing (2007).
On tight approximate inference of
the logistic-normal topic admixturemodel. InAI/Statistics.
Ahn, J.-H. and J.-H. Oh (2003). A Con-
strained EM Algorithm for Princi-
pal Component Analysis. Neural
Computation 15, 57–65.
Ahn, S., A. Korattikara, and M. Welling
(2012). Bayesian Posterior Sam-
pling via Stochastic Gradient Fisher
Scoring. In Intl. Conf. on Machine
Learning.
Airoldi, E., D. Blei, S. Fienberg, and

Learning.
E. Xing (2008). Mixed-membershipstochastic blockmodels. J. of Ma-
chine Learning Research 9, 1981–
2014.
Aitchison, J. (1982). The statistical
analysis of compositional data. J.of
Royal Stat. Soc. Series B 44 (2), 139–
177.Aji, S. M. and R. J. McEliece (2000,
March). The generalized distribu-
tive law. IEEE Trans. Info. The-
ory 46(2), 325–343.
Alag, S. and A. Agogino (1996). In-
ference using message propoga-
tion and topology transformationin vector Gaussian continuous net-works. InUAI.

2014.
Albers, C., M. Leisink, and H. Kap-
pen (2006). The Cluster VariationMethod for Efficient Linkage Anal-ysis on Extended Pedigrees. BMC
Bioinformatics 7.
Albert, J. and S. Chib (1993). Bayesian
analysis of binary and polychoto-
mous response data. J. of the Am.
Stat.Assoc. 88 (422), 669–679.
Allwein, E., R. Schapire, and Y. Singer
(2000). Reducing multiclass to bi-
nary: A unifying approach for mar-
gin classiﬁers. J.ofMachineLearn-
ingResearch , 113–141.
Aloise, D., A. Deshpande, P. Hansen,
and P. Popat (2009). NP-hardness

of Euclidean sum-of-squares clus-
tering.Machine Learning 75, 245–
249.
Alpaydin, E. (2004). Introduction to
machinelearning. MIT Press.
Altun, Y., T. Hofmann, and I. Tsochan-
taridis (2006). Large Margin Meth-
odsf
or Structured and Interde-
pendent Output Variables. InG. Bakir, T. Hofmann, B. Scholkopf,A. Smola, B. Taskar, and S. Vish-wanathan (Eds.), Machine Learning
withStructuredOutputs. MIT Press.
Amir, E. (2010). Approximation Al-
gorithms for Treewidth. Algorith-
mica 56(4), 448.
Amir, E. and S. McIlraith (2005).

249.
odsf
Partition-based logical reason-ing for ﬁrst-order and propo-
sitional theories. Artiﬁcial
Intelligence 162 (1), 49–88.
Ando, R. and T. Zhang (2005). A
framework for learning predictive
structures from multiple tasks and
unlabeled data. J. of Machine
LearningResearch 6, 1817–1853.
Andrews, D. and C. Mallows (1974).
Scale mixtures of Normal distribu-
tions.J. of Royal Stat. Soc. Series
B3 6, 99–102.Andrieu, C., N. de Freitas, and

A. Doucet (2000). SequentialBayesian estimation and model se-lection for dynamic kernel ma-chines. Technical report, Cam-bridge Univ.
Andrieu, C., N. de Freitas, and
A. Doucet (2001). Robust FullBayesian Learning for Radial Ba-
sis Networks. Neural Computa-
tion 13(10), 2359–2407.
Andrieu, C., N. de Freitas, A. Doucet,
and M. Jordan (2003). An introduc-
tion to MCMC for machine learn-ing.MachineLearning 50, 5–43.
Andrieu, C., A. Doucet, and V. Tadic

(2005). Online EM for parameterestimation in nonlinear-non Gaus-sian state-space models. In Proc.
IEEECDC.
Andrieu, C. and J. Thoms (2008). A
tutorial on adaptive MCMC. Statis-
ticalComputing 18, 343–373.
Aoki, M. (1987). Statespacemodelingof
timeseries. Springer.
Archambeau, C. and F. Bach (2008).
Sparse probabilistic projections. InNIPS.
Argyriou, A., T. Evgeniou, and M. Pon-
til (2008). Convex multi-task fea-ture learning. Machine Learn-
ing 73(3), 243–272.
Armagan, A., D. Dunson, and J. Lee

IEEECDC.
(2011). Generalized double paretoshrinkage. Technical report, Duke.
Armstrong, H. (2005). Bayesian esti-
mation of decomposable Gaussian
graphical models. Ph.D. thesis,
UNSW.
Armstrong, H., C. Carter, K. Wong,
and R. Kohn (2008). Bayesian Co-variance Matrix Estimation using aMixture of Decomposable Graphi-
cal Models. Statistics and Comput-
ing, 1573–1375.
Arnborg, S., D. G. Corneil, and
A. Proskurowski (1987). Complex-
ity of ﬁnding embeddings in a k-
tree.SIAM J. on Algebraic and Dis-
creteMethods 8 , 277–284.

UNSW.
Arora, S. and B. Barak (2009). Com-
plexity
Theory: AModernApproach.
Cambridge.
Arthur, D. and S. Vassilvitskii (2007). k-
means++: the advantages of carefulseeding. In Proc. 18th ACM-SIAM
symp. on Discrete algorithms, pp.
1027â˘A¸S1035.

1016 BIBLIOGRAPHY
Arulampalam, M., S. Maskell, N. Gor-
don, and T. Clapp (2002, Febru-
ary). A Tutorial on Particle Fil-ters for Online Nonlinear/Non-Gaussian Bayesian Tracking. IEEE
Trans. on Signal Processing 50 (2),
174–189.
Asavathiratham, C. (2000). The Inﬂu-
enceModel: ATractableRepresenta-
tionfortheDynamicsofNetworkedMarkov Chains. Ph.D. thesis, MIT,
Dept. EECS.
Atay-Kayis, A. and H. Massam (2005).
A Monte Carlo method for comput-ing the marginal likelihood in non-decomposable Gaussian graphical
models.Biometrika 92, 317–335.
Attenberg, J., K. Weinberger, A. Smola,
A. Dasgupta, and M. Zinkevich
(2009). Collaborative spam ﬁlter-ing with the hashing trick. In Virus
Bulletin.
Attias, H. (1999). Independent factor
analysis.Neural Computation 11,
803–851.
Attias, H. (2000). A variational
Bayesian framework for graphicalmodels. InNIPS-12.
Bach, F. (2008). Bolasso: Model Con-
sistent Lasso Estimation through
the Bootstrap. In Intl.Conf.onMa-
chineLearning.
Bach, F. and M. Jordan (2001). Thin

Bulletin.
803–851.
junction trees. In NIPS.
Bach, F. and M. Jordan (2005). A prob-
abilistic interpretation of canonical
correlation analysis. Technical Re-port 688, U. C. Berkeley.
Bach, F. and E. Moulines (2011). Non-
asymptotic analysis of stochasticapproximation algorithms for ma-chine learning. In NIPS.
Bahmani, B., B. Moseley, A. Vattani,
R. Kumar, and S. Vassilvitskii (2012).Scalable k-Means++. In VLDB.
Bakker, B. and T. Heskes (2003). Task
Clustering and Gating for Bayesian
Multitask Learning. J. of Machine

LearningResearch 4, 83–99.
Baldi, P. and Y. Chauvin (1994).
Smooth online learning algorithmsfor hidden Markov models. Neural
Computation 6 , 305–316.
Balding, D. (2006). A tutorial on sta-
tistical methods for population as-
sociation studies. Nature Reviews
Genetics 7 , 81–91.Banerjee, O., L. E. Ghaoui, and
A. d’Aspremont (2008). Model se-lection through sparse maximumlikelihood estimation for multivari-ate gaussian or binary data. J. of
MachineLearningResearch 9, 485–516.
Bar-Shalom, Y. and T. Fortmann

(1988).Tracking and data associa-
tion. Academic Press.
Bar-Shalom, Y. and X. Li (1993). Es-
timation and Tracking: Principles,
Techniques and Software. Artech
House.
Barash, Y. and N. Friedman (2002).
Context-speciﬁc Bayesian cluster-ing for gene expression data. J.
Com
p.Bio. 9, 169–191.
Barber, D. (2006). Expectation Cor-
rection for Smoothed Inference inSwitching Linear Dynamical Sys-
tems.J. of Machine Learning Re-
search 7, 2515–2540.
Barber, D. and C. Bishop (1998).
Ensemble Learning in Bayesian

House.
Com
Neural Networks. In C. Bishop
(Ed.),NeuralNetworksandMachine
Learning, pp. 215–237. Springer.
Barber, D. and S. Chiappa (2007).
Uniﬁed inference for variational
bayesian linear gaussian statespace models. In NIPS.
Barbieri, M. and J. Berger (2004). Op-
timal predictive model selection.
AnnalsofStatistics 32, 870–897.
Bartlett, P., M. Jordan, and J. McAuliffe
(2006). Convexity, Classiﬁcation,and Risk Bounds. J.oftheAm.Stat.
Assoc. 101(473), 138–156.
Baruniak, R. (2007). Compressive sens-

ing.IEEE Signal Processing Maga-
zine.
Barzilai, J. and J. Borwein (1988). Two
point step size gradient methods.IMAJ.ofNumericalAnalysis 8, 141–
148.
Basu, S., T. Choudhury, B. Clarkson,
and A. Pentland (2001). Learn-ing human interactions withthe inﬂuence model. Techni-cal Report 539, MIT Media Lab.ftp://whitechapel.media.mit.edu/pub/tech-reports/TR-539-ABSTRACT.html.
Baum, L. E., T. Petrie, G. Soules, and
N. Weiss (1970). A maximizationtechnique occuring in the statisti-cal analysis of probabalistic func-

zine.
148.
tions in markov chains. TheAnnals
of Mathematical Statistics 41, 164–
171.
Beal, M. (2003). VariationalAlgorithms
forApproximateBayesianInference.Ph.D. thesis, Gatsby Unit.
Beal, M. and Z. Ghahramani (2006).
Variational Bayesian Learning ofDirected Graphical Models with
Hidden Variables. Bayesian Anal-
ysis 1(4).
Beal, M. J., Z. Ghahramani, and C. E.
Rasmussen (2002). The inﬁnite hid-
den Markov model. In NIPS-14.
Beck, A. and M. Teboulle (2009). A fast

171.
ysis 1(4).
iterative shrinkage-thresholding al-gorothm for linear inverse prob-lems.SIAM J. on Imaging Sci-
ences 2(1), 183–202.
Beinlich, I., H. Suermondt, R. Chavez,
and G. Cooper (1989). The ALARMmonitoring system: A case studywith two probabilistic inferencetechniques for belief networks. In
Proc. of the Second European Conf.
onAIinMedicine, pp. 247–256.
Bekkerman, R., M. Bilenko, and
J. Langford (Eds.) (2011). ScalingUp
MachineLearning. Cambridge.
Bell, A. J. and T. J. Sejnowski (1995).
An information maximisation ap-
proach to blind separation and

blind deconvolution. Neural Com-
putation 7 (6), 1129–1159.
Bengio,Y
. (2009). Learning deep ar-
chitectures for AI. Foundationsand
TrendsinMachineLearning 2 (1), 1–
127.
Bengio, Y. and S. Bengio (2000).
Modeling high-dimensional dis-
crete data with multi-layer neuralnetworks. In NIPS.
Bengio, Y., O. Delalleau, N. Roux,
J. Paiement, P. Vincent, andM. Ouimet (2004). Learning eigen-functions links spectral embedding
and kernel PCA. Neural Computa-
tion 16, 2197–2219.
Bengio, Y. and P. Frasconi (1995). Dif-
fusion of context and credit infor-

Bengio,Y
127.
mation in markovian models. J. of
AIResearch 3, 249–270.
Bengio, Y. and P. Frasconi (1996).
Input/output HMMs for sequence
processing. IEEE Trans. on Neural
Networks 7 (5), 1231–1249.

BIBLIOGRAPHY 1017
Bengio, Y., P. Lamblin, D. Popovici,
and H. Larochelle (2007). Greedy
layer-wise training of deep net-works. InNIPS.
Berchtold, A. (1999). The double chain
markov model. Comm. Stat. Theor.
Methods 28, 2569–2589.
Berger, J. (1985). Bayesian salesman-
ship. In P. K. Goel and A. Zellner
(Eds.),Bayesian Inference and De-
cisionTechniqueswithApplications:
Essays in Honor of Bruno deFinetti.
North-Holland.
Berger, J. and R. Wolpert (1988). The
Likelihood Principle. The Instituteof Mathematical Statistics. 2nd edi-tion.
Berkhin, P. (2006). A survey of
clustering datamining techniques.In J. Kogan, C. Nicholas, and
M. Teboulle (Eds.), Grouping Multi-
dimensional Data: Recent Advances
inClustering, pp. 25–71. Springer.
Bernardo, J. and A. Smith (1994).
BayesianTheory. John Wiley.
Berrou, C., A. Glavieux, and P. Thiti-
majashima (1993). Near Shannonlimit error-correcting coding anddecoding: Turbo codes. Proc. IEEE
Intl.Comm.Conf..
Berry, D. and Y. Hochberg (1999).
Bayesian perspectives on multiple

comparisons. J. Statist. Planning
andInference 82, 215–227.
Bertele, U. and F. Brioschi (1972). Non-
serialDynamicProgramming. Aca-
Bertsekas, D. (1997). Parallel and Dis-
tribution Computation: Numerical
Methods. Athena Scientiﬁc.
Bertsekas, D. (1999). Nonlinear Pro-
gramming (Second ed.). Athena
Scientiﬁc.
Bertsekas, D. and J. Tsitsiklis (2008).
Introduction to Probability. Athena
Scientiﬁc. 2nd Edition.
Besag, J. (1975). Statistical analysis of
non-lattice data. TheStatistician24 ,
179–196.

179–196.
Bhatnagar, N., C. Bogdanov, and
E. Mossel (2010). The compu-tational complexity of estimatingconvergence time. Technical re-port, .Bhattacharya, A. and D. B. Dunson
(2011). Simplex factor models formultivariate unordered categorical
data.J. of the Am. Stat. Assoc..T o
appear.
Bickel, P. and E. Levina (2004). Some
theory for Fisher’s linear discrimi-
nant function, "Naive Bayes", andsome alternatives when there aremany more variables than obser-
vations.Bernoulli 10, 989–1010.

179–196.
appear.
Bickson, D. (2009). Gaussian Belief
Propagation: Theory and Applica-
tion. Ph.D. thesis, Hebrew Univer-sity of Jerusalem.
Bilmes, J. (2000). Dynamic Bayesian
multinets. In UAI.
Bilmes, J. A. (2001). Graphical models
andaut
omatic speech recognition.
Technical Report UWEETR-2001-0005, Univ. Washington, Dept. ofElec. Eng.
Binder, J., D. Koller, S. J. Russell, and
K. Kanazawa (1997). Adaptive prob-abilistic networks with hidden vari-ables.Machine Learning 29, 213–
244.
Binder, J., K. Murphy, and S. Russell

andaut
244.
(1997). Space-efficient inference indynamic probabilistic networks. In
Intl.JointConf.onAI.
Birnbaum, A. (1962). On the founda-
tions of statistical infernece. J. of
theAm.Stat.Assoc. 57, 269–326.
Bishop, C. (1999). Bayesian PCA. In
NIPS.
Bishop, C. (2006a). Patternrecognition
andmachinelearning. Springer.
Bishop, C. (2006b). Patternrecognition
Bishop, C. and G. James (1993). Analy-
sis of multiphase ﬂows using dual-
energy densitometry and neural
networks.NuclearInstrumentsand

NIPS.
Methods in Physics Research A327,
580–593.
Bishop, C. and M. Svensén (2003).
Bayesian hierarchical mixtures ofexperts. InUAI.
Bishop, C. and M. Tipping (2000).
Variational relevance vector ma-chines. InUAI.
Bishop, C. M. (1994). Mixture density
networks. Technical Report NCRG4288, Neural Computing ResearchGroup, Department of ComputerScience, Aston University.Bishop, C. M. (1995). Neural Networks
for Pattern Recognition. ClarendonPress.
Bishop, Y., S. Fienberg, and P. Holland
(1975).Discrete Multivariate Analy-
sis: TheoryandPractice. MIT Press.

Bistarelli, S., U. Montanari, and
F. Rossi (1997). Semiring-basedconstraint satisfaction and opti-
mization.J. of the ACM 44 (2), 201–
236.
Blake, A., P. Kohli, and C. Rother (Eds.)
(2011).AdvancesinMarkovRandom
FieldsforVisionandImageProcess-
ing. MIT Press.
Blei, D. and J. Lafferty (2006a). Corre-
lated topic models. In NIPS.
Blei, D. and J. Lafferty (2006b). Dy-
namic topic models. In Intl. Conf.
onMachineLearning, pp. 113–120.
Blei, D. and J. Lafferty (2007). A Corre-
lated Topic Model of "Science". An-

236.
nalsofAppliedStat. 1 (1), 17–35.
Blei, D. and J. McAuliffe (2010, March).
Supervised topic models. Technicalreport, Princeton.
Blei, D., A. Ng, and M. Jordan (2003).
Latent dirichlet allocation. J. of
MachineLearningResearch 3, 993–1022.
Blumensath, T. and M. Davies (2007).
Onthe
difference between Orthog-
onal Matching Pursuit and Orthog-onal Least Squares. Technical re-port, U. Edinburgh.
Bo, L., C. Sminchisescu, A. Kanaujia,
and D. Metaxas (2008). Fast Algo-rithms for Large Scale Conditional3D Prediction. In CVPR.

Onthe
Bohning, D. (1992). Multinomial logis-
tic regression algorithm. Annals of
theInst.ofStatisticalMath. 44, 197–200.
Bollen, K. (1989). Structural Equation
Models with Latent Variables. JohnWiley & Sons.
Bordes, A., L. Bottou, and P. Galli-
nari (2009, July). Sgd-qn: Care-ful quasi-newton stochastic gradi-
ent descent. J.ofMachineLearning
Research 10, 1737–1754.
Bordes, A., L. Bottou, P. Gallinari,
J. Chang, and S. A. Smith (2010). Er-
ratum: SGDQN is Less Careful than
Expected.J. of Machine Learning
Research 11, 2229–2240.

1018 BIBLIOGRAPHY
Boser, B. E., I. M. Guyon, and V. N.
Vapnik (1992). A training algorithm
for optimal margin classiﬁers. In
Proc.oftheWorkshoponComputa-
tionalLearningTheory.
Bottcher, S. G. and C. Dethlefsen
(2003). deal: A package for learning
bayesian networks. J. of Statistical
Software 8 (20).
Bottolo, L. and S. Richardson (2010).
Evolutionary stochastic search.BayesianAnalysis 5 (3), 583–618.
Bottou, L. (1998). Online algorithms
and stochastic approximations. InD. Saad (Ed.), Online Learning and
NeuralNetworks. Cambridge.
Bottou, L. (2007). Learning with large
datasets (nips tutorial).
Bottou, L., O. Chapelle, D. DeCoste,
and J. Weston (Eds.) (2007). Large
ScaleKernelMachines. MIT Press.
Bouchard, G. (2007). Efficient bounds
for the softmax and applications
to approximate inference in hybrid
models. In NIPS 2007 Workshop
onApproximateInferenceinHybrid
Models.
Bouchard-Cote, A. and M. Jordan
(2009). Optimization of structured
mean ﬁeld objectives. In UAI.
Bowman, A.andA.Azzalini(1997). Ap-

Models.
pliedSmoothingTechniquesforData
Analysis. Oxford.
Box, G. and N. Draper (1987). Empir-
ical Model-Building and ResponseSurfaces. Wiley.
Box, G. and G. Tiao (1973). Bayesian
inference in statistical analysis.
Boyd, S. and L. Vandenberghe (2004).
Convexoptimization. Cambridge.
Boyen, X. and D. Koller (1998).
Tractable inference for complexstochastic processes. In UAI.
Boykov, Y., O. Veksler, and R. Zabih
(2001). Fast approximate energyminimization via graph cuts. IEEE

Trans. on Pattern Analysis and Ma-
chineIntelligence 23 (11).
Brand, M. (1996). Coupled hidden
Markov models for modeling inter-
acting processes. Technical Report405, MIT Lab for Perceptual Com-puting.Brand, M. (1999). Structure learning
in conditional probability modelsvia an entropic prior and param-
eter extinction. Neural Computa-
tion 11, 1155–1182.
Braun, M. and J. McAuliffe (2010). Vari-
ational Inference for Large-ScaleModels of Discrete Choice. J.ofthe
Am.Stat.Assoc. 105 (489), 324–335.

Breiman, L. (1996). Bagging predictors.
MachineLearning 24 , 123–140.
Breiman, L. (1998). Arcing classiﬁers.
AnnalsofStatistics 26, 801–849.
Breiman, L. (2001a). Random forests.
MachineLearning 45 (1), 5–32.
Breiman, L. (2001b). Statistical mod-
eling: the two cultures. Statistical
Science 16 (3), 199–231.
Breiman, L., J. Friedman, and R. Ol-
shen(1984).Clas
siﬁcation and re-
gressiontrees. Wadsworth.
Breslow, N. E. and D. G. Clayton (1993).
Approximate inference in general-ized linear mixed models. J. of the

Am.Stat.Assoc. 88 (421), 9–25.
Briers, M., A. Doucet, and S. Maskel
(2010). Smoothing algorithms forstate-space models. Annals of
theInstituteofStatisticalMathemat-
ics 62(1), 61–89.
Brochu, E., M. Cora, and N. de Fre-
itas (2009, November). A tutorialon Bayesian optimization of expen-sive cost functions, with applica-tion to active user modeling andhierarchical reinforcement learn-ing. Technical Report TR-2009-23,Department of Computer Science,University of British Columbia.
Brooks, S. and G. Roberts (1998).

Assessing convergence of MarkovChain Monte Carlo algorithms.
Statistics and Computing 8, 319–
335.
Brown, L., T. Cai, and A. DasGupta
(2001). Interval estimation for a bi-nomial proportion. Statistical Sci-
ence 16(2), 101–133.
Brown, M. P., R. Hughey, A. Krogh, I. S.
Mian, K. Sjölander, and D. Haus-sler (1993). Using dirichlet mixturespriors to derive hidden Markovmodels for protein families. In
Intl. Conf. on Intelligent Systems for
MolecularBiology, pp. 47–55.Brown, P., M. Vannucci, and T. Fearn
(1998). Multivariate Bayesian vari-

335.
able selection and prediction. J. of
the Royal Statistical Society B 60 (3),
627–641.
Bruckstein, A., D. Donoho, and
M. Elad (2009). From sparse so-
lutions of systems of equations tosparse modeling of signals and im-ages.SIAMReview 51 (1), 34–81.
Bryson, A. and Y.-C. Ho (1969). Applied
optimal control: optimization, esti-
mation, andcontrol. Blaisdell Pub-
lishing Company.
Buhlmann, P. and T. Hothorn (2007).
Boosting Algorithms: Regulariza-tion, Prediction and Model Fitting.
StatisticalScience 22 (4), 477–505.
Buhlmann, P. and S. van de

627–641.
Geer (2011). Statistics for High-
Dimensional Data: Methodology,
TheoryandApplications. Springer.
Buhlmann, P. and B. Yu (2003). Boost-
ing with the L2 loss: Regression
and classiﬁcation. J.oftheAm.Stat.
Assoc. 98(462), 324–339.
Buhlmann, P. and B. Yu (2006). Sparse
boosting.J. of Machine Learning
Research 7 , 1001–1024.
Bui, H., S. Venkatesh, and G. West
(2002). Policy Recognition in the
Abstract Hidden Markov Model. J.
ofAIResearch 17, 451–499.
Buntine, W. (2002). Variational Exten-

sions to EM and Multinomial PCA.InIntl.Conf.onMachineLearning.
Buntine, W. and A. Jakulin (2004). Ap-
plying Discrete PCA in Data Analy-sis. InUAI.
Buntine,W
. and A. Jakulin (2006). Dis-
crete Component Analysis. In Sub-
space, Latent Structure and Feature
Selection: Statistical and Optimiza-tionPerspectivesWorkshop.
Buntine, W. and A. Weigend (1991).
Bayesian backpropagation. Com-
plexSystems 5, 603–643.
Burges, C. J., T. Shaked, E. Renshaw,
A. Lazier, M. Deeds, N. Hamilton,

Buntine,W
and G. Hullender (2005). Learningto rank using gradient descent. In
Intl.Conf.onMachineLearning, pp.
89–96.
Burkard, R., M. Dell’Amico, and
S. Martello (2009). Assignment
Problems. SIAM.

BIBLIOGRAPHY 1019
Byran, K. and T. Leise (2006). The
25,000,000,000 Eigenvector: The
Linear Algebra behind Google.SIAMReview 48 (3).
Calvetti, D. and E. Somersalo (2007).
Introduction to Bayesian ScientiﬁcComputing. Springer.
Candes, E., J. Romberg, and T. Tao
(2006). Robust uncertainty prin-ciples: Exact signal reconstructionfrom highly incomplete frequency
information. IEEE. Trans. Inform.
Theory 52 (2), 489–509.
Candes, E. and M. Wakin (2008,
March). An introduction to com-
pressive sampling. IEEESignalPro-
cessingMagazine 21.
Candes, E., M. Wakin, and S. Boyd
(2008). Enhancing sparsity by
reweighted l1 minimization. J. of
FourierAnalysisandApplications 1,
877–905.
Cannings, C., E. A. Thompson, and
M. H. Skolnick (1978). Probabil-ity functions in complex pedigrees.
Advances in Applied Probability 10,
26–61.
Canny, J. (2004). Gap: a factor model
for discrete data. In Proc. An-
nual Intl. ACM SIGIR Conference,pp. 122–129.
Cao, Z., T. Qin, T.-Y. Liu, M.-F. Tsai,

877–905.
26–61.
and H. Li (2007). Learning to rank:From pairwise approach to listwise
approach. In Intl.Conf.onMachine
Learning, pp. 129â ˘A¸S136.
Cappe, O. (2010). Online Expectation
Maximisation. In K. Mengersen,
M. Titterington, and C. Robert(Eds.),Mixtures.
Cappe, O. and E. Mouline (2009, June).
Online EM Algorithm for Latent
Data Models. J. of Royal Stat. Soc.
SeriesB 71 (3), 593–613.
Cappe, O., E. Moulines, and T. Ryden
(2005).Inference in Hidden Markov
Models. Springer.
Carbonetto, P. (2003). Unsupervised

statistical models for general object
recognition. Master’s thesis, Uni-versity of British Columbia.
Carlin, B. P. and T. A. Louis (1996).
BayesandEmpiricalBayesMethods
for Data Analysis. Chapman and
Hall.Caron, F. and A. Doucet (2008).
Sparse Bayesian nonparametric re-
gression. In Intl. Conf. on Machine
Learning.
Carreira-Perpinan, M. and C. Williams
(2003). An isotropic gaussian mix-
ture can have more modes thancomponents. Technical ReportEDI-INF-RR-0185, School of Infor-matics, U. Edinburgh.

Learning.
Carter, C. and R. Kohn (1994). On
Gibbs sampling for state space
models.Biometrika 81 (3), 541–553.
Carterette, B., P. Bennett, D. Chicker-
ing, and S. Dumais (2008). Here
or There: Preference Judgments forRelevance. In Proc.ECIR.
Caruana, R. (1998). A dozen tricks with
multitask learning. In G. Orr and
K.-R. Mueller (Eds.), Neural Net-
work
s: TricksoftheTrade. Springer-
Verlag.
Caruana, R. and A. Niculescu-Mizil
(2006). An empirical comparison
of supervised learning algorithms.
InIntl.Conf.onMachineLearning.

work
Verlag.
Carvahlo, C., N. Polson, and J. Scott
(2010). The horseshoe estimator for
sparse signals. Biometrika 97 (2),
465.
Carvahlo, L. and C. Lawrence (2007).
Centroid estimation in discretehigh-dimensional spaces with ap-
plications in biology. Proc. of
the National Academy of Science,
USA 105(4).
Carvalho, C. M. and M. West (2007).
Dynamic matrix-variate graphical
models.BayesianAnalysis 2 (1), 69–
98.
Casella, G. and R. Berger (2002). Statis-
tical inference. Duxbury. 2nd edi-
tion.
Castro, M., M. Coates, and R. D.

465.
98.
tion.
Nowak (2004). Likelihood based hi-erarchical clustering. IEEETrans.in
SignalProcessing 52 (8), 230.
Celeux, G. and J. Diebolt (1985).
The SEM algorithm: A probabilis-tic teacher derive from the EMalgorithm for the mixture prob-
lem.ComputationalStatisticsQuar-
terly 2, 73–82.
Cemgil, A. T. (2001). A technique for
painless derivation of kalman ﬁlter-
ing recursions. Technical report, U.Nijmegen.Cesa-Bianchi, N. and G. Lugosi (2006).
Prediction, learning, and games.Cambridge University Press.
Cevher, V. (2009). Learning with com-

pressible priors. In NIPS.
Chai, K. M. A. (2010). Multi-tasklearn-
ing with Gaussian processes. Ph.D.thesis, U. Edinburgh.
Chang, H., Y. Weiss, and W. Freeman
(2009). Informative Sensing. Tech-nical report, Hebrew U. Submittedto IEEE Transactions on Info. The-ory.
Chang, J. and D. Blei (2010). Hierar-
chical relational models for docu-ment networks. The Annals of Ap-
pliedStatistics 4 (1), 124–150.
Chang, J., J. Boyd-Graber, S. Gerrish,
C. Wang, and D. Blei (2009). Read-ing tea leaves: How humans inter-pret topic models. In NIPS.

Chapelle, O. and L. Li (2011). An empir-
ical evaluation of Thompson sam-pling. InNIPS.
Chartrand, R. and W. Yin (2008). It-
eratively reweighted algorithms for
compressive sensing. In Intl. Conf.
on Acoustics, Speech and Signal
Proc.
Chechik, G., A. G. N. Tishby, and
Y. Weiss (2005). Information bot-tleneck for gaussian variables. J.
of Machine Learning Research 6,
165â˘A¸S188.
Cheeseman, P
., J. Kelly, M. Self, J. Stutz,
W. Taylor, and D. Freeman (1988).
Autoclass: A Bayesian classiﬁcation

Proc.
system. In Proc. of the Fifth Intl.
WorkshoponMachineLearning.
Cheeseman, P. and J. Stutz (1996).
Bayesian classiﬁcation (autoclass):
Theory and results. In Fayyad,Pratetsky-Shapiro, Smyth, and
Uthurasamy (Eds.), Advances in
KnowledgeDiscoveryandDataMin-
Chen, B., K. Swersky, B. Marlin, and
N. de Freitas (2010). Sparsity priorsand boosting for learning localizeddistributed feature representations.Technical report, UBC.
Chen, B., J.-A. Ting, B. Marlin, and

N. de Freitas (2010). Deep learningof invariant spatio-temporal fea-
tures from video. In NIPSWorkshop
onDeepLearning.

1020 BIBLIOGRAPHY
Chen, M., D. Carlson, A. Zaas,
C. Woods, G. Ginsburg, A. Hero,
J. Lucas, and L. Carin (2011,March). The Bayesian ElasticNet: Classifying Multi-Task Gene-Expression Data. IEEE Trans.
Biomed.Eng. 58 (3), 468–79.
Chen, R. and S. Liu (2000). Mixture
Kalman ﬁlters. J.RoyalStat.Soc.B .
Chen, S. and J. Goodman (1996). An
empirical study of smoothing tech-niques for language modeling. In
Proc.34thACL, pp. 310–318.
Chen, S. and J. Goodman (1998).
An empirical study of smoothing
techniques for language modeling.Technical Report TR-10-98, Dept.Comp. Sci., Harvard.
Chen, S. and J. Wigger (1995, July). Fast
orthogonal least squares algorithmforefficientsubset modelselection.IEEE Trans. Signal Processing 3 (7),
1713–1715.
Chen, S. S., D. L. Donoho, and M. A.
Saunders (1998). Atomic decompo-sition by basis pursuit. SIAM Jour-
nal on Scientiﬁc Computing 20 (1),
33–61.
Chen, X., S. Kim, Q. Lin, J. G.

33–61.
Carbonell, and E. P. Xing (2010).Graph-Structured Multi-task Re-gression and an Efficient Optimiza-tion Method for General FusedLasso. Technical report, CMU.
Chib, S. (1995). Marginal likelihood
from the Gibbs output. J. of the
Am.Stat.Assoc. 90, 1313–1321.
Chickering, D. (1996). Learning
Bayesian networks is NP-Complete.InAI/StatsV.
Chickering, D. and D. Heckerman
(1997). Efficient approximations forthe marginal likelihood of incom-plete data given a Bayesian net-
work.Machine Learning 29, 181–
212.

212.
Chickering, D. M. (2002). Optimal
structure identiﬁcation with greedysearch.Journal of Machine Learn-
ingResearch 3, 507–554.
Chipman, H., E. George, and R. Mc-
Culloch (1998). Bayesian CARTmodel search. J. of the Am. Stat.
Assoc. 93, 935–960.
Culloch (2001). The practical imple-
mentation of Bayesian Model Se-
lection.Model Selection . IMS Lec-
ture Notes.Chipman, H., E. George, and R. Mc-
Culloch (2006). Bayesian Ensemble
Learning. In NIPS.

Culloch (2010). BART: Bayesian ad-
ditive regression trees. Ann. Appl.
Stat. 4(1), 266–298.
Choi, M., V. Tan, A. Anandkumar, and
A. Willsky (2011). Learning latenttree graphical models. J. of Ma-
chineLearningResearch .
Choi, M. J. (2011). Trees and Be-
yond: Exploiting and Improving
Tree-Structured Graphical Models.
Ph.D. thesis, MIT.
Choset, H. and K. Nagatani (2001).
Topological simultaneous localiza-tion and mapping (SLAM): towardex
act localization without explicit
localization. IEEE Trans. Robotics
andAutomation 17 (2).

Chow, C. K. and C. N. Liu (1968).
Approximating discrete probabil-ity distributions with dependence
trees.IEEETrans.onInfo.Theory14,
462–67.
Christensen, O., G. Roberts, and
M. SkÃ˝uld (2006). Robust Markov
chain Monte Carlo methods for
spatial generalized linear mixed
models.J. of Computational and
GraphicalStatistics 15, 1–17.
Chung, F. (1997). Spectral Graph The-
ory. AMS.
Cimiano, P., A. Schultz, S. Sizov,
P. Sorg, and S. Staab (2009). Ex-
plicit versus latent concept modelsfor cross-language information re-

462–67.
ory. AMS.
trieval. InIntl.JointConf.onAI.
Cipra, B. (2000). The Ising Model Is
NP-Complete. SIAMNews 33 (6).
Ciresan, D. C., U. Meier, L. M.
Gambardella, and J. Schmidhuber
(2010). Deep big simple neural netsfor handwritten digit recognition.
Neural Computation 22 (12), 3207–
3220.
Clarke, B. (2003). Bayes model av-
eraging and stacking when model
approximation error cannot be ig-
nored.J. of Machine Learning Re-
search, 683–712.
Clarke, B., E. Fokoue, and H. H. Zhang
(2009).Principles and Theory for
Data Mining and Machine Learn-

3220.
ing. Springer.Cleveland, W. and S. Devlin (1988).
Locally-weighted regression: Anapproach to regression analysis by
local ﬁtting. J. of the Am. Stat. As-
soc. 83(403), 596–610.
Collins, M. (2002). Discrimina-
tive Training Methods for Hidden
Markov Models: Theory and Exper-iments with Perceptron Algorithms.InEMNLP.
Collins, M., S. Dasgupta, and R. E.
Schapire (2002). A generalizationof principal components analysisto the exponential family. In NIPS-
14.
Collins, M. and N. Duffy (2002). Con-

14.
volution kernels for natural lan-guage. InNIPS.
Collobert, R. and J. Weston (2008).
A Uniﬁed Architecture for NaturalLanguage Processing: Deep NeuralNetworks with Multitask Learning.
Combettes, P. and V. Wajs (2005). Sig-
nal recovery by proximal forward-backward splitting. SIAM J. Multi-
scaleModel.Simul. 4 (4), 1168–1200.
Cook, J. (2005). Exact Calculation
of Beta Inequalities. Technical re-
port, M. D. Anderson Cancer Cen-ter, Dept. Biostatistics.
Cooper, G. and E. Herskovits (1992).

A Bayesian method for the induc-tion of probabilistic networks from
data.MachineLearning 9, 309–347.
Coope
r, G. and C. Yoo (1999). Causal
discovery from a mixture of exper-
imental and observational data. InUAI.
Cover, T. and P. Hart (1967). Near-
est neighbor pattern classiﬁcation.
IEEETrans.Inform.Theory 13 (1), 21–
27.
Cover, T. M. and J. A. Thomas
(1991).ElementsofInformationThe-
ory. John Wiley.
(2006).Elements of Information
Theory. John Wiley. 2nd edition.
Cowles, M. and B. Carlin (1996).

Coope
27.
Markov chain monte carlo conver-
gence diagnostics: A comparative
review.J. of the Am. Stat. Assoc. 91,
883–904.

BIBLIOGRAPHY 1021
Crisan, D., P. D. Moral, and T. Lyons
(1999). Discrete ﬁltering using
branching and interacting particle
systems.Markov Processes and Re-
latedFields 5 (3), 293–318.
Cui, Y., X. Z. Fern, and J. G. Dy
(2010). Learning multiple nonre-
dundant clusterings. ACMTransac-
tions on Knowledge Discovery fromData 4(3).
Cukier, K. (2010, February). Data, data
everywhere.
Dagum, P. and M. Luby (1993). Ap-
proximating probabilistic inferencein Bayesian belief networks is NP-
hard.ArtiﬁcialIntelligence 60, 141–
153.
Dahl, J., L. Vandenberghe, and V. Roy-
chowdhury (2008, August). Co-
variance selection for non-chordalgraphs via chordal embedding.
Optimization Methods and Soft-
ware 23(4), 501–502.
Dahlhaus, R. and M. Eichler (2000).
Causality and graphical models fortime series. In P. Green, N. Hjort,
and S. Richardson (Eds.), Highly
structured stochastic systems. Ox-
ford University Press.
Dallal, S. and W. Hall (1983). Approxi-
mating priors by mixtures of natu-
ral conjugate priors. J. of Royal Stat.

153.
Soc.SeriesB 45 , 278–286.
Darwiche, A. (2009). Modeling and
Reasoning with Bayesian Networks.
Daume, H. (2007a). Fast search for
Dirichlet process mixture models.InAI/Statistics.
Daume, H. (2007b). Frustratingly easy
domain adaptation. In Proc.theAs-
soc.forComp.Ling.
Dawid, A. P. (1992). Applications of a
general propagation algorithm forprobabilistic expert systems. Statis-
ticsandComputing 2, 25–36.
Dawid, A. P. (2002). Inﬂuence dia-
grams for causal modelling and in-

ference.Intl. Stat. Review 70, 161–
189. Corrections p437.
Dawid, A. P. (2010). Beware of the DAG!
J. of Machine Learning Research 6,
59–86.Dawid, A. P. and S. L. Lauritzen
(1993). Hyper-markov laws in thestatistical analysis of decompos-
able graphical models. The Annals
ofStatistics 3, 1272–1317.
de Freitas, N., R. Dearden, F. Hut-
ter, R. Morales-Menendez, J. Mutch,
and D. Poole (2004). Diagnosis bya waiter and a mars explorer. Proc.
IEEE 92(3).
de Freitas, N., M. Niranjan, and A. Gee

(2000). Hierarchical Bayesian mod-els for regularisation in sequential
learning.NeuralComputation12 (4),
955–993.
Dechter, R. (1996). Bucket elimination:
a unifying framework for proba-
bilistic inference. In UAI.
Dechter, R. (2003). Constraint Process-
ing. Morgan Kaufmann.
Decoste, D. and B. Schoelkopf (2002).
Tr
aining invariant support vector
machines.Machinelearnng 41, 161–
190.
Deerwester, S., S. Dumais, G. Fur-
nas, T. Landauer, and R. Harshman(1990). Indexing by latent semantic
analysis.J. of the American Society

955–993.
Tr
190.
for Information Science 41 (6), 391–
407.
DeGroot, M. (1970). OptimalStatistical
Decisions. McGraw-Hill.
Deisenroth, M., C. Rasmussen, and
J. Peters (2009). Gaussian Process
Dynamic Programming. Neurocom-
puting 72 (7), 1508–1524.
Dellaportas, P., P. Giudici, and
G. Roberts (2003). Bayesian infer-ence for nondecomposable graphi-
cal gaussian models. Sankhya, Ser.
A6 5, 43–55.
Dellaportas, P. and A. F. M. Smith
(1993). Bayesian Inference for Gen-
eralized Linear and ProportionalHazards Models via Gibbs Sam-pling.J. of the Royal Statisti-

407.
cal Society. Series C (Applied Statis-tics) 42(3), 443–459.
Delyon, B., M. Lavielle, and
E. Moulines (1999). Convergence ofa stochastic approximation version
of the EM algorithm. Annals of
Statistics 27 (1), 94–128.
Dempster, A. (1972). Covariance selec-
tion.Biometrics 28 (1).Dempster, A. P., N. M. Laird, and D. B.
Rubin (1977). Maximum likelihood
from incomplete data via the EM
algorithm.J. of the Royal Statistical
Society,SeriesB 34, 1–38.
Denison, D., C. Holmes, B. Mallick,
and A. Smith (2002). Bayesian

methodsfornonlinearclassiﬁcation
andregression. Wiley.
Denison, D., B. Mallick, and A. Smith
(1998). A Bayesian CART algorithm.
Biometrika 85, 363–377.
Desjardins, G. and Y. Bengio (2008).
Empirical evaluation of convolu-
tional RBMs for vision. TechnicalReport 1327, U. Montreal.
Dey, D., S. Ghosh, and B. Mallick (Eds.)
(2000).Generalized Linear Models:
ABayesianPerspective. Chapman &Hall/CRC Biostatistics Series.
Diaconis, P., S. Holmes, and R. Mont-
gomery (2007). Dynamical Bias inthe Coin Toss. SIAM Review 49 (2),
211–235.
Diaconis, P. and D. Ylvisaker (1985).

211–235.
Quantifying prior opinion. InBayesianStatistics2.
Dietterich, T. G. and G. Bakiri (1995).
Solving multiclass learning prob-
lems via ECOCs. J.ofAIResearch2,
263–286.
Diggle, P. and P. Ribeiro (2007). Model-
basedGeostatistics. Springer.
Ding, Y. and R. Harrison (2010). A
sparse multinomial probit modelfor classiﬁcation. Pattern Analysis
andApplications , 1–9.
Dobra, A. (2009). Dependency net-
works for genome-wide data. Tech-
nicalr
eport, U. Washington.
Dobra, A. and H. Massam (2010). The

263–286.
nicalr
mode oriented stochastic search(MOSS) algorithm for log-linearmodels with conjugate priors. Sta-
tisticalMethodology 7, 240–253.
Domingos, P. and D. Lowd (2009).
MarkovLogic: AnInterfaceLayerforAI. Morgan & Claypool.
Domingos, P. and M. Pazzani (1997).
On the optimality of the simplebayesian classiﬁer under zero-one
loss.Machine Learning 29, 103–
130.

1022 BIBLIOGRAPHY
Domke, J., A. Karapurkar, and Y. Aloi-
monos (2008). Who killed the di-
rected model? In CVPR.
Doucet, A., N.deFreitas, andN.J.Gor-
don (2001). SequentialMonteCarlo
Methods in Practice. Springer Ver-lag.
Doucet, A., N. Gordon, and V. Krish-
namurthy (2001). Particle Filters forState Estimation of Jump MarkovLinear Systems. IEEETrans.onSig-
nalProcessing 49 (3), 613–624.
Dow, J. and J. Endersby (2004). Multi-
nomial probit and multinomiallogit: a comparison of choice mod-
els for voting research. Electoral
Studies 23 (1), 107–122.
Drineas, P., A. Frieze, R. Kannan,
S. Vempala, and V. Vinay (2004).
Clustering large graphs via the sin-gular value decomposition. Ma-
chineLearning 56, 9–33.
Drugowitsch, J. (2008). Bayesian lin-
ear regression. Technical report, U.Rochester.
Druilhet, P. and J.-M. Marin (2007). In-
variant HPD credible sets and MAPestimators. Bayesian Analysis 2 (4),
681–692.
Duane, S., A. Kennedy, B. Pendle-
ton, and D. Roweth (1987). Hy-
brid Monte Carlo. Physics Letters

681–692.
B 195(2), 216–222.
Duchi, J., S. Gould, and D. Koller
(2008). Projected subgradient
methods for learning sparse gaus-sians. InUAI.
Duchi, J., E. Hazan, and Y. Singer
(2010). Adaptive Subgradient Meth-ods for Online Learning andStochastic Optimization. In Proc.
of the Workshop on Computational
LearningTheory.
Duchi, J., S. Shalev-Shwartz, Y. Singer,
and T. Chandra (2008). Efficient
projections onto the L1-ball forlearning in high dimensions. In
Intl.Conf.onMachineLearning.
Duchi, J. and Y. Singer (2009). Boost-

ing with structural sparsity. In Intl.
Conf.onMachineLearning.
Duchi, J., D. Tarlow, G. Elidan, and
D. Koller (2007). Using combi-
natorial optimization within max-product belief propagation. InNIPS.Duda, R. O., P. E. Hart, and D. G. Stork
(2001).PatternClassiﬁcation. Wiley
Interscience. 2nd edition.
Dumais, S. and T. Landauer (1997). A
solution to Plato’s problem: Thelatent semantic analysis theory ofacquisition, induction and repre-
sentation of knowledge. Psycholog-
icalReview 104, 211–240.

Dunson, D., J. Palomo, and K. Bollen
(2005). Bayesian Structural Equa-
tion Modeling. Technical Report2005-5, SAMSI.
Durbin, J. and S. J. Koopman (2001).
Time Series Analysis by State Space
Methods. Oxford University Press.
Durbin, R., S. Eddy, A. Krogh, and
G. Mitchison (1998). Biological Se-
quence Analysis: Probabilistic Mod-els of Proteins and Nucleic Acids.
Cambridge: Cambridge UniversityPress.
Earl, D. and M. Deem (2005). Paral-
leltempering: Theory, applications,
and new perspectives. Phy
 s.Chem.

 s.Chem.
Chem.Phys. 7, 3910.
Eaton, D. and K. Murphy (2007). Exact
Bayesian structure learning from
uncertain interventions. In AI/S-
tatistics.
Edakunni, N., S. Schaal, and S. Vi-
jayakumar (2010). Probabilistic in-cremental locally weighted learn-ing using randomly varying coeffi-cient model. Technical report, USC.
Edwards, D., G. de Abreu, and
R. Labouriau (2010). Selecting high-dimensional mixed graphical mod-els using minimal AIC or BIC
forests.BMCBioinformatics 11 (18).
Efron, B. (1986). Why Isn’t Everyone

a Bayesian? The American Statisti-
cian 40(1).
Efron, B. (2010). Large-Scale Infer-
ence: Empirical Bayes Methods for
Estimation, Testing, and Prediction.
Efron, B., I. Johnstone, T. Hastie, and
R. Tibshirani (2004). Least angle re-
gression.Annals of Statistics 32 (2),
407–499.
Efron, B. and C. Morris (1975). Data
analysis using stein’s estimator andits generalizations. J.oftheAm.Stat.
Assoc. 70(350), 311–319.Elad, M. and I. Yavnch (2009). A plu-
rality of sparse representations is

407–499.
better than the sparsest one alone.
IEEE Trans. on Info. Theory 55 (10),
4701–4714.
Elidan, G. and S. Gould (2008). Learn-
ing Bounded Treewidth BayesianNetworks.J. of Machine Learning
Research, 2699–2731.
Elidan, G., N. Lotner, N. Friedman, and
D. Koller (2000). Discovering hid-
den variables: A structure-basedapproach. In NIPS.
Elidan, G., I. McGraw, and D. Koller
(2006). Residual belief propa-gation: Informed scheduling forasynchronous message passing. InUAI.
Elkan, C. (2003). Using the triangle in-

equality to accelerate k-means. In
Elkan, C. (2005). Deriving TF-IDF as
a Fisher kernel. In Proc.Intl.Symp.
on String Processing and Informa-
tionRetrieval(SPIRE), pp. 296–301.
Elkan, C. (2006). Clustering docu-
ments with an exponential fmaily
approximation of the Dirichletcompoind multinomial model. In
Ellis, B. and W. H. Wong (2008). Learn-
ing causal bayesian network struc-tures from experimental data. J. of
the Am. Stat. Assoc. 103 (482), 778–
789.

789.
Engel, Y., S. Mannor, and R. Meir
(2005). Reinforcement Learningwith Gaussian Processes. In Intl.
Erhan, D., Y. Bengio, A. Courville, P.-A.
Manzagol, P. Vincent, and S. Ben-
gio (2010). Why Does UnsupervisedPre-training Help Deep Learning?
J. of Machine Learning Research 11,
625–660.
Eroshe
va, E., S. Fienberg, and
C. Joutard (2007). Describingdisability through individual-levelmixture models for multivariate bi-nary data.AnnalsofAppliedStatis-
tics.
Erosheva, E., S. Fienberg, and J. Laf-

625–660.
Eroshe
tics.
ferty (2004). Mixed-membershipmodels of scientiﬁc publications.
Proc. of the National Academy of
Science,USA 101, 5220–2227.

BIBLIOGRAPHY 1023
Escobar, M. D. and M. West (1995).
Bayesian density estimation and
inference using mixtures. J. of the
Am.Stat.Assoc. 90 (430), 577–588.
Ewens, W. (1990). Population genet-
ics theory - the past and the fu-ture. In S.Lessard (Ed.), Mathemeti-
cal and Statistica Developments of
Evolutionary Theory, pp. 177–227.
Reidel.
Fan, J. and R. Z. Li (2001). Variable se-
lection via non-concave penalizedlikelihood and its oracle properties.
J. of the Am. Stat. Assoc. 96 (456),
1348–1360.
Fearnhead, P. (2004). Exact bayesian
curve ﬁtting and signal segmen-tation.IEEE Trans. Signal Process-
ing 53, 2160–2166.
Felzenszwalb, P. and D. Huttenlocher
(2006). Efficient belief propagationfor early vision. Intl. J. Computer
Vision 70(1), 41–54.
Ferrucci, D., E. Brown, J. Chu-Carroll,
J. Fan, D. Gondek, A. Kalyanpur,
A. Lally, J. W. Murdock, E. N.amd J. Prager, N. Schlaefter, andC. Welty (2010). Building Wat-son: An Overview of the DeepQAProject.AIMagazine , 59–79.
Fienberg, S. (1970). An iterative pro-

cedure for estimation in contin-gency tables. Annals of Mathemat-
icalStatistics 41 (3), 907â˘A¸S917.
Figueiredo, M. (2003). Adaptive
sparseness for supervised learn-
ing.IEEE Trans. on Pattern Anal-
ysisandMachineIntelligence 25 (9),
1150–1159.
Figueiredo, M., R. Nowak, and
S. Wright (2007). Gradient pro-
jection for sparse reconstruction:application to compressed sensingand other inverse problems. IEEE.
J. on Selected Topics in Signal Pro-cessing.
Figueiredo, M. A. T. and A. K. Jain

(2002). Unsupervised learning of ﬁ-nite mixture models. IEEETrans.on
PatternAnalysisandMachineIntel-
ligence 24 (3), 381–396. Matlab code
at http://www.lx.it.pt/ mtf/mixture-
code.zip.
Fine, S., Y. Singer, and N. Tishby
(1998). The hierarchical HiddenMarkov Model: Analysis and appli-
cations.MachineLearning 32, 41.Finkel, J. and C. Manning (2009). Hier-
archical bayesian domain adapta-
tion. InProc.NAACL, pp. 602–610.
Fischer, B. and J. Schumann (2003).
Autobayes: A system for generatingdata analysis programs from sta-

code.zip.
tistical models. J. Functional Pro-
gramming 13 (3), 483–508.
Fishelson, M. and D. Geiger (2002).
Exact genetic linkage computations
for general pedigrees. BMC Bioin-
formatics 18.
Fletcher, R. (2005). On the Barzilai-
Borwein Method. Applied Opti-
mization 96, 235–256.
Fokoue, E. (2005). Mixtures of factor
analyzers: an extension with co-
variates.J.MultivariateAnalysis 95,
370–384.
Forbes,
J., T. Huang, K. Kanazawa, and
S. Russell (1995). The BATmobile:
Towards a Bayesian automated taxi.
InIntl.JointConf.onAI.
Forsyth, D. and J. Ponce (2002). Com-

370–384.
Forbes,
puter vision: a modern approach.
Prentice Hall.
Fraley, C. and A. Raftery (2002).
Model-based clustering, discrimi-nant analysis, and density estima-
tion.J. of the Am. Stat. Assoc. (97),
611–631.
Fraley, C. and A. Raftery (2007).
Bayesian Regularization for Normal
Mixture Estimation and Model-Based Clustering. J. of Classiﬁca-
tion 24, 155–181.
Franc, V., A. Zien, and B. Schoelkopf
(2011). Support vector machines as
probabilistic models. In Intl. Conf.
onMachineLearning.
Frank, I. and J. Friedman (1993). A

611–631.
statistical view of some chemomet-
rics regression tools. Technomet-
rics 35(2), 109–135.
Fraser, A. (2008). HiddenMarkovMod-
els and Dynamical Systems. SIAMPress.
Freund, Y. and R. R. Schapire (1996).
Experiments with a new boosting
algorithm. In Intl.Conf.onMachine
Learning.
Frey, B. (1998). Graphical Models for
MachineLearningandDigitalCom-
munication. MIT Press.Frey, B. (2003). Extending factor
graphs so as to unify directed andundirected graphical models. InUAI.
Frey, B. and D. Dueck (2007, Febru-

Learning.
ary). Clustering by Passing Mes-sages Between Data Points. Sci-
ence 315, 972â ˘A¸S976.
Friedman, J. (1991). Multivariate
adaptive regression splines. Ann.
Statist. 19, 1–67.
Friedman, J. (1997a). On bias, variance,
0-1 loss and the curse of dimen-sionality.J.DataMiningandKnowl-
edgeDiscovery 1, 55–77.
Friedman, J. (2001). Greedy function
approximation: a gradient boost-
ing machine. AnnalsofStatistics29,
1189–1232.
Friedman, J., T. Hastie, and R. Tibshi-
rani (2000). Additive logistic regres-

sion: a statistical view of boosting.
Annalsofstatistics 28 (2), 337–374.
Friedman, J., T. Hastie, and R. Tib-
shirani (2008). Sparse inverse co-
variance estimation the graphicallasso.Biostatistics 9 (3), 432–441.
rani (2010, Februrary). Regulariza-tion Paths for Generalized LinearModels via Coordinate Descent. J.
ofStatisticalSoftware 33 (1).
Friedman,
N. (1997b). Learning
Bayesian networks in the presenceof missing values and hidden vari-ables. InUAI.

Friedman,
Friedman, N., D. Geiger, and M. Gold-
szmidt (1997). Bayesian network
classiﬁers.Machine Learning J. 29,
131–163.
Friedman, N., D. Geiger, and N. Lot-
ner (2000). Likelihood computation
with value abstraction. In UAI.
Friedman, N. and D. Koller (2003). Be-
ing Bayesian about Network Struc-ture: A Bayesian Approach toStructure Discovery in Bayesian
Networks. Machine Learning 50,
95–126.
Friedman, N., M. Ninion, I. Pe’er, and
T. Pupko (2002). A Structural EM
Algorithm for Phylogenetic Infer-ence.J.Comp.Bio. 9, 331–353.

131–163.
95–126.
Friedman, N. and Y. Singer (1999). Ef-
ﬁcient Bayesian parameter estima-tion in large discrete domains. InNIPS-11.

1024 BIBLIOGRAPHY
Fruhwirth-Schnatter, S. (2007). Fi-
nite Mixture and Markov Switching
Fruhwirth-Schnatter, S. and R. Fruh-
wirth (2010). Data Augmentation
and MCMC for Binary and Multi-nomial Logit Models. In T. Kneib
and G. Tutz (Eds.), Statistical Mod-
ellingandRegressionStructures, pp.
111–132. Springer.
Fu, W. (1998). Penalized regressions:
the bridge verus the lasso. J.Com-
putationalandgraphicalstatistics .
Fukushima, K. (1975). Cognitron: a
self-organizing multilayered neu-
ral network. Biological Cybernet-
ics 20(6), 121–136.
Fung, R. and K. Chang (1989). Weight-
ing and integrating evidence for
stochastic simulation in Bayesiannetworks. In UAI.
Gabow, H., Z. Galil, and T. Spencer
(1984). Efficient implementationof graph algorithms using contrac-
tion. InIEEE Symposium on the
FoundationsofComputerScience.
Gales, M. (2002). Maximum like-
lihood multiple subspace projec-
tions for hidden Markov models.IEEE. Trans. on Speech and AudioProcessing 10 (2), 37–47.

Gales, M. J. F. (1999). Semi-tied covari-
ance matrices for hidden Markovmodels.IEEETrans.onSpeechand
AudioProcessing 7 (3), 272–281.
Gamerman, D. (1997). Efficient sam-
pling from the posterior distribu-tion in generalized linear mixed
models.StatisticsandComputing 7,
57–68.
Geiger, D. and D. Heckerman (1994).
Learning Gaussian networks. In
UAI, Volume 10, pp. 235–243.
Geiger, D. and D. Heckerman (1997).
A characterization of Dirchlet dis-tributions through local and global
independence. Annals of Statis-

57–68.
tics 25, 1344–1368.
Gelfand, A. (1996). Model determina-
tion using sampling-based meth-
ods. In Gilks, Richardson, andSpiegelhalter (Eds.), Markov Chain
Monte Carlo in Practice. Chapman& Hall.
Gelfand, A. and A. Smith (1990).
Sampling-based approaches to cal-
culating marginal densities. J.ofthe
Am.Stat.Assoc. 85, 385–409.Gelman, A., J. Carlin, H. Stern, and
D. Rubin (2004). Bayesian data
analysis. Chapman and Hall. 2nd
edition.
Gelman, A. and J. Hill (2007). Data
analysis using regression and mul-

edition.
tilevel/ hierarchical models. Cam-
bridge.
Gelman, A. and X.-L. Meng (1998).
Simulating normalizing constants:from importance sampling tobridge sampling to path sampling.
StatisicalScience 13, 163–185.
Gelman, A. and T. Raghunathan (2001).
Using conditional distributions formissing-data imputation. Statistical
Science.
Gelman,A
. and D. Rubin (1992). Infer-
ence from iterative simulation us-
ing multiple sequences. Statistical
Science 7, 457–511.
Geman, S., E. Bienenstock, and

bridge.
Science.
Gelman,A
R. Doursat (1992). Neural networksand the bias-variance dilemma.NeuralComputing 4, 1–58.
Geman, S. and D. Geman (1984).
Stochastic relaxation, Gibbs distri-butions, and the Bayesian restora-tion of images. IEEE Trans. on Pat-
tern Analysis and Machine Intelli-gence 6(6).
Geoffrion, A. (1974). Lagrangian
relaxation for integer program-
ming.Mathematical Programming
Study 2, 82–114.
George, E. and D. Foster (2000). Cal-
ibration and empirical bayes vari-able selection. Biometrika 87 (4),
731–747.
Getoor, L. and B. Taskar (Eds.) (2007).

731–747.
IntroductiontoRelationalStatistical
Geyer, C. (1992). Practical markov
chain monte carlo. Statistical Sci-
ence 7, 473–483.
Ghahramani, Z. and M. Beal (2000).
Variational inference for Bayesianmixtures of factor analysers. InNIPS-12.
Ghahramani, Z. and M. Beal (2001).
Propagation algorithms for varia-tional Bayesian learning. In NIPS-
13.
Ghahramani, Z. and G. Hinton (1996a).
The EM algorithm for mixtures offactor analyzers. Technical report,Dept. of Comp. Sci., Uni. Toronto.Ghahramani, Z. and G. Hinton (1996b).

13.
Parameter estimation for linear dy-namical systems. Technical Re-port CRG-TR-96-2, Dept. Comp.Sci., Univ. Toronto.
Ghahramani, Z. and M. Jordan (1997).
Factorial hidden Markov models.MachineLearning 29, 245–273.
Gilks, W. and C. Berzuini (2001).
Following a moving target –Monte Carlo infernece for dynamic
Bayesian models. J. of Royal Stat.
Soc.SeriesB 63, 127–146.
Gilks, W., N. Best, and K. Tan (1995).
Adaptive rejection Metropolis sam-pling.AppliedStatistics44, 455–472.
Gilks, W. and P. Wild (1992). Adaptive

rejection sampling for Gibbs sam-pling.AppliedStatistics41, 337–348.
Girolami, M., B. Calderhead, and
S. Chin (2010). Riemannian Man-
ifold Hamiltonian Monte Carlo. J.
of Royal Stat. Soc. Series B . To ap-
pear.
Girolami, M. and S. Rogers (2005). Hi-
erarchic bayesian models for kernel
learning. In Intl. Conf. on Machine
Learning, pp. 241–248.
Girolami, M. and S. Rogers (2006).
Variational Bayesian multinomial
probit regression with Gaussian
process priors. Neural Comptua-
tion 18(8), 1790 – 1817.
Girshick, R., P. Felzenszwalb, and
D.McAllest
er (2011). Object de-
tection with grammar models. In
NIPS.

pear.
NIPS.
Gittins, J. (1989). Multi-armed Bandit
AllocationIndices. Wiley.
Giudici, P. and P. Green (1999).
Decomposable graphical gaus-sian model determination.Biometrika 86 (4), 785–801.
Givoni, I. E. and B. J. Frey (2009, June).
A binary variable model for affin-
ity propagation. Neural Computa-
tion 21(6), 1589–1600.
Globerson, A. and T. Jaakkola (2008).
Fixing max-product: Convergent
message passing algorithms forMAP LP-relaxations. In NIPS.
Glorot, X. and Y. Bengio (2010, May).

NIPS.
Understanding the difficulty oftraining deep feedforward neuralnetworks. In AI/Statistics, Volume 9,
pp. 249–256.

BIBLIOGRAPHY 1025
Gogate, V., W. A. Webb, and P. Domin-
gos (2010). Learning efficient
Markov networks. In NIPS.
Goldenberg, A., A.X.Zheng, S.E.Fien-
berg, and E. M. Airoldi (2009). ASurvey of Statistical Network Mod-
els.FoundationsandTrendsinMa-
chineLearning , 129–233.
Golub, G. and C. F. van Loan (1996).
Matrix computations. Johns Hop-
kins University Press.
Gonen, M., W. Johnson, Y. Lu, and
P. Westfall (2005, August). The
Bayesian Two-Sample t Test. The
American Statistician 59 (3), 252–
257.
Gonzales, T. (1985). Clustering to
minimize the maximum interclus-
ter distance. Theor. Comp. Sci. 38,
293–306.
Gorder, P. F. (2006, Nov/Dec). Neu-
ral networks show new promise formachine vision. Computing in sci-
ence&engineering 8 (6), 4–8.
Gordon, N. (1993). Novel ap-
proach to nonlinear/non-GaussianBayesian state estimation. IEEPro-
ceedings(F) 140 (2), 107–113.
Graepel, T., J. Quinonero-Candela,
T. Borchert, and R. Herbrich(2010). Web-Scale Bayesian Click-
Through Rate Prediction for Spon-

257.
293–306.
sored Search Advertising in Mi-
crosoftâ˘A´Zs Bing Search Engine. In
Grauman, K. and T. Darrell (2007,
April). The Pyramid Match Kernel:
Efficient Learning with Sets of Fea-
tures.J. of Machine Learning Re-
search 8, 725–760.
Green, P. (1998). Reversible jump
Markov chain Monte Carlo compu-
tation and Bayesian model deter-
mination.Biometrika 82, 711–732.
Green, P. (2003). Tutorial on trans-
dimensional MCMC. In P. Green,
N. Hjort, and S. Richardson (Eds.),

HighlyStructuredStochasticSystems.
OUP.
Green, P. and B. Silverman (1994). Non-
parametric regression and general-
ized linear models. Chapman and
Hall.Greenshtein, E. and J. Park (2009). Ap-
plication of Non Parametric Empir-ical Bayes Estimation to High Di-
mensional Classiﬁcation. J. of Ma-
chine Learning Research 10, 1687–
1704.
Greig, D., B. Porteous, and A. Seheult
(1989). Exact maximum a posteriori
estimation for binary images. J. of
Royal Stat. Soc. Series B 51 (2), 271–
279.
Griffin, J. and P. Brown (2007).

OUP.
1704.
279.
Bayesian adaptive lassos with non-
convex penalization. Technical re-port, U. Kent.
Griffin, J. and P. Brown (2010). In-
ference with normal-gamma priordistributions in regression prob-
lems.Bayesian Analysis 5 (1), 171–
188.
Griffiths, T. . and J. Tenenbaum (2009).
Theory
-Based Causal Induction.
Psychological Review 116 (4), 661–
716.
Griffiths, T. and M. Steyvers (2004).
Finding scientiﬁc topics. Proc. of
USA 101, 5228–5235.
Griffiths, T., M. Steyvers, D. Blei, and

188.
Theory
716.
J. Tenenbaum (2004). Integratingtopics and syntax. In NIPS.
Griffiths, T. and J. Tenenbaum (2001).
Using vocabulary knowledge inbayesian multinomial estimation.InNIPS, pp. 1385–1392.
Griffiths, T. and J. Tenenbaum (2005).
Structure and strength in causal
induction.Cognitive Psychology 51,
334–384.
Grimmett, G. and D. Stirzaker (1992).
Probability and Random Processes.
Oxford.
Guan, Y., J. Dy, D. Niu, and Z. Ghahra-
mani (2010). Variational Inferencefor Nonparametric Multiple Clus-

334–384.
Oxford.
tering. In 1st Intl. Workshop on
Discovering, Summarizing and Us-
ingMultipleClustering(MultiClust).
Guedon, Y. (2003). Estimating hidden
semi-markov chains from discrete
sequences.J.ofComputationaland
GraphicalStatistics 12, 604–639.
Guo, Y. (2009). Supervised exponential
family principal component anal-ysis via convex optimization. InNIPS.Gustafsson, M. (2001). A proba-
bilistic derivation of the partial
least-squares algorithm. Journal of
Chemical Information and Model-
ing 41, 288–294.
Guyon, I., S. Gunn, M. Nikravesh, and

L. Zadeh (Eds.) (2006). Feature Ex-
traction: Foundations and Applica-tions. Springer.
Hacker, J. and P. Pierson (2010).
Winner-Take-All Politics: HowWashington Made the Rich Richer–
and Turned Its Back on the Middle
Class. Simon & Schuster.
Halevy, A., P. Norvig, and F. Pereira
(2009). The unreasonable effective-ness of data. IEEE Intelligent Sys-
tems 24(2), 8–12.
Hall, P., J. T. Ormerod, and M. P. Wand
(2011). Theory of Gaussian Varia-tional Approximation for a Gener-alised Linear Mixed Model. Statis-

ticaSinica 21, 269–389.
Hamilton, J. (1990). Analysis of time
series subject to changes in regime.J.Econometrics 45 , 39–70.
Hans, C. (2009). Bayesian Lasso re-
gression.Biometrika 96 (4), 835–
845.
Hansen, M. and B. Yu (2001). Model
selection and the principle of min-
imum description length. J. of the
Am.Stat.Assoc..
Hara, H. and A. Takimura (2008).
A Localization Approach to Im-
prove Iterative Proportional Scal-ing in Gaussian Graphical Models.
Communications in Statistics - The-

845.
oryandMethod . to appear.
Hardin,
J. and J. Hilbe (2003). Gener-
alizedEstimatingEquations. Chap-man and Hall/CRC.
Harmeling, S. and C. K. I. Williams
(2011). Greedy learning of binarylatent trees. IEEE Trans. on Pat-
tern Analysis and Machine Intelli-gence 33(6), 1087–1097.
Harnard, S. (1990). The symbol
grounding problem. Physica D 42,
335–346.
Harvey, A. C. (1990). Forecasting,Struc-
tural Time Series Models, and theKalman Filter. Cambridge Univer-ity Press.

1026 BIBLIOGRAPHY
Hastie, T., S. Rosset, R. Tibshirani, and
J. Zhu (2004). The entire regular-
ization path for the support vector
machine.J. of Machine Learning
Research 5, 1391–1415.
Hastie, T. and R. Tibshirani (1990).
Generalizedadditivemodels. Chap-
man and Hall.
Hastie, T., R. Tibshirani, and J. Fried-
man (2001). The Elements of Statis-
ticalLearning. Springer.
man (2009). TheElementsofStatisti-
calLearning. Springer. 2nd edition.
Hastings, W. (1970). Monte carlo
sampling methods using markovchains and their applications.
Biometrika 57 (1), 97–109.
Haykin, S. (1998). Neural Networks: A
Comprehensive Foundation. Pren-
tice Hall. 2nd Edition.
Haykin, S. (Ed.) (2001). Kalman Filter-
ingandNeuralNetworks. Wiley.
Hazan, T. and A. Shashua (2008).
Convergent message-passing algo-rithms for inference over generalgraphs with convex free energy. InUAI.
Hazan, T. and A. Shashua (2010).

Norm-product belief propagation:primal-dual message passing forapproximate inference. IEEE Trans.
onInfo.Theory 56 (12), 6294–6316.
He, Y.-B. and Z. Geng (2009). Active
learning of causal networks withintervention experiments and opti-
mal designs. J.ofMachineLearning
Research 10, 2523–2547.
Heaton, M. and J. Scott (2009).
Bayesian computation and the lin-
ear model. Technical report, Duke.
Heckerman, D., D. Chickering,

C. Meek, R. Rounthwaite, andC. Kadie (2000). Dependencynetworks for density estimation,collaborative ﬁltering, and data vi-
sualization. J. of Machine Learning
Research 1, 49–75.
Heckerman, D., D. Geiger, and
M. Chickering (1995). Learning
Bayesian networks: the combina-tion of knowledge and statistical
data.MachineLearning 20 (3), 197–
243.Heckerman, D., C. Meek, and
G. Cooper (1997, February). A
Bayesian approach to causal dis-covery. Technical Report MSR-TR-97-05, Microsoft Research.
Heckerman, D., C. Meek, and D. Koller

(2004). Probabilistic models forrelational data. Technical Re-port MSR-TR-2004-30, MicrosoftResearch.
Heller, K. and Z. Ghahramani (2005).
BayesianHierarchicalClustering. In
Henrion, M. (1988). Propagation of
uncertainty by logic sampling in
Bayes’ networks. In UAI, pp. 149–
164.
Herbrich, R., T. Minka, and T. Graepel
(2007). TrueSkill: A Bayesian skillrating system. In NIPS.
Hertz, J., A. Krogh, and R. G. Palmer
(1991).AnIntroductiontotheTheory
of Neural Comptuation. Addison-
Wesley.
Hillar, C., J. Sohl-Dickstein, and
K. Koepsell (2012, April). Efficientandop

164.
Wesley.
timal binary hopﬁeld asso-
ciative memory storage using min-imum probability ﬂow. Technicalreport.
Hinton, G. (1999). Products of experts.
InProc.9thIntl.Conf.onArtif.Neu-
ralNetworks(ICANN), Volume 1, pp.1–6.
Hinton, G. (2002). Training products of
experts by minimizing contrastive
divergence. NeuralComputation14,
1771–1800.
Hinton, G. (2010). A Practical Guide
to Training Restricted Boltzmann
Machines. Technical report, U.Toronto.
Hinton, G. and D. V. Camp (1993).

Keeping neural networks simple byminimizing the description length
of the weights. In in Proc. of the
6th Ann. ACM Conf. on Computa-
tional Learning Theory, pp. 5–13.
ACM Press.
Hinton, G., S. Osindero, and Y. Teh
(2006). A fast learning algorithm
for deep belief nets. Neural Com-
putation 18, 1527–1554.
Hinton, G.andR.Salakhutdinov(2006,
July). Reducing the dimensionality
of data with neural networks. Sci-
ence 313(5786), 504–507.Hinton, G. E., P. Dayan, and M. Revow
(1997). Modeling the manifolds ofimages of handwritten digits. IEEE

ACM Press.
Trans.onNeuralNetworks8, 65–74.
Hinton, G. E. and Y. Teh (2001).
Discovering multiple constraintsthat are frequently approximatelysatisïˇn˛Aed. InUAI.
Hjort, N., C. Holmes, P. Muller, and
S. Walker (Eds.) (2010). Bayesian
Nonparametrics. Cambridge.
Hoeﬂing, H. (2010). A Path Algorithm
for the Fused Lasso Signal Approx-imator. Technical report, Stanford.
Hoeﬂing, H. and R. Tibshirani
(2009). Estimation of Sparse Bi-naryPairwiseMarkovNetworksus-
ing Pseudo-likelihoods. J. of Ma-
chineLearningResearch 10.
Hoeting, J., D. Madigan, A. Raftery,

and C. Volinsky (1999). Bayesian
model averaging: A tutorial. Statis-
ticalScience 4 (4).
Hoff, P. D. (2009, July). A First
Course in Bayesian Statistical Meth-
ods. Springer.
Hoffman, M., D. Blei, and F. Bach
(2010). Online learning for latent
dirichlet allocation. In NIPS.
Hoffman, M. and A. Gelman (2011).
The no-U-turn sampler: Adaptivelysetting path lengths in Hamilto-nianMonteCarlo. Technicalreport,Columbia U.
Hofmann, T. (1999). Probabilistic la-
tent semantic indexing. Research
and Development in Information
Retrieval, 50–57.

Holmes, C. and L. Held (2006).
Bayesian auxiliary variable models
for binary and multinomial regres-
sion.Bayesian Analysis 1 (1),145–
168.
Honk
ela, A. and H. Valpola (2004).
Variational Learning and Bits-Back
Coding: An Information-TheoreticView to Bayesian Learning. IEEE.
Trans.onNeuralNetworks 15 (4).
Honkela, A., H. Valpola, and
J. Karhunen (2003). Accelerat-ing Cyclic Update Algorithms forParameter Estimation by Pattern
Searches. Neural Processing Let-
ters 17, 191–203.

BIBLIOGRAPHY 1027
Hopﬁeld, J. J. (1982, April). Neu-
ral networks and physical systems
with emergent collective computa-
tional abilities. Proc.oftheNational
Academy of Science, USA 79 (8),
2554â˘A¸S2558.
Hornik, K. (1991). Approximation ca-
pabilities of multilayer feedforwardnetworks. Neural Networks 4 (2),
251â˘A¸S257.
Horvitz, E., J. Apacible, R. Sarin, and
L. Liao (2005). Prediction, Expecta-
tion, and Surprise: Methods, De-signs, and Study of a DeployedTraffic Forecasting Service. In UAI.
Howard, R. and J. Matheson (1981). In-
ﬂuence diagrams. In R. Howard
and J. Matheson (Eds.), Readingson
the Principles and Applications of
DecisionAnalysis,volumeII.S t r a t e -
gic Decisions Group.
Hoyer, P. (2004). Non-negative matrix
factorizaton with sparseness con-
straints.J.ofMachineLearningRe-
search 5, 1457–1469.
Hsu, C.-W., C.-C. Chang, and C.-J. Lin
(2009). A practical guide to sup-
port vector classiﬁcation. Technicalreport, Dept. Comp. Sci., NationalTaiwan University.
Hu, D., L. van der Maaten, Y. Cho,

L. Saul, and S. Lerner (2010). LatentVariable Models for Predicting FileDependencies in Large-Scale Soft-ware Development. In NIPS.
Hu, M., C. Ingram, M.Sirski, C. Pal,
S. Swamy, and C. Patten (2000).A Hierarchical HMM Implementa-tion for Vertebrate Gene Splice SitePrediction. Technical report, Dept.Computer Science, Univ. Waterloo.
Huang, J., Q. Morris, and B. Frey
(2007). Bayesian inference of Mi-croRNA targets from sequence andexpression data. J.Comp.Bio..
Hubel, D. and T. Wiesel (1962). Recep-

tive ﬁelds, binocular itneraction,and functional architecture in thecat’s visual cortex. J.Physiology 160,
106–154.
Huber, P. (1964). Robust estimation
of a location parameter. Annals of
Statistics 53, 73â ˘A¸S101.
Hubert, L. and P. Arabie (1985). Com-
paring partitions. J. of Classiﬁca-
tion 2, 193–218.Hunter, D. and R. Li (2005). Variable
selection using MM algorithms.
AnnalsofStatistics 33, 1617–1642.
Hunter, D. R. and K. Lange (2004). A
Tutorial on MM Algorithms. The
AmericanStatistician 58, 30–37.

106–154.
Hyaﬁl, L. and R. Rivest (1976). Con-
structing Optimal Binary DecisionTrees is NP-complete. Information
ProcessingLetters 5 (1), 15–17.
Hyvarinen, A., J. Hurri, and P. Hoyer
(2009).Natural Image Statistics: a
probabilisticapproachtoearlycom-
putationalvision. Springer.
Hyvarinen,
A. and E. Oja (2000). In-
dependent component analysis: al-
gorithms and applications. Neural
Networks 13, 411–430.
Ilin, A. and T. Raiko (2010). Practi-
cal Approaches to Principal Com-
ponent Analysis in the Presence

of Missing Values. J. of Machine
LearningResearch 11, 1957–2000.
Insua, D. R. and F. Ruggeri (Eds.)
(2000).Robust Bayesian Analysis.
Springer.
Isard, M. (2003). PAMPAS: Real-Valued
Graphical Models for Computer Vi-
sion. InCVPR, Volume 1, pp. 613.
Isard, M. and A. Blake (1998). CON-
DENSATION - conditional densitypropagation for visual tracking.
Intl. J. of Computer Vision 29 (1), 5–
18.
Jaakkola, T. (2001). Tutorial on varia-
tional approximation methods. InM. Opper and D. Saad (Eds.), Ad-

Springer.
18.
vanced mean ﬁeld methods. MIT
Press.
Jaakkola, T. and D. Haussler (1998). Ex-
ploiting generative models in dis-criminative classiﬁers. In NIPS, pp.
487–493.
Jaakkola, T. and M. Jordan (1996a).
Computing upper and lowerbounds on likelihoods in in-tractable networks. In UAI.
Jaakkola, T. and M. Jordan (1996b).
A variational approach to Bayesianlogistic regression problems andtheir extensions. In AI+Statistics.
Jaakkola, T. S. and M. I. Jordan (2000).
Bayesian parameter estimation via
variational methods. Statistics and

Press.
487–493.
Computing 10, 25–37.Jacob, L., F. Bach, and J.-P. Vert (2008).
Clustered Multi-Task Learning: a
Convex Formulation. In NIPS.
Jain, A. and R. Dubes (1988). Algo-
rithmsforClusteringData. PrenticeHall.
James, G. and T. Hastie (1998). The
error coding method and PICTS.
J. of Computational and Graphical
Statistics 7 (3), 377–387.
Japkowicz, N., S. Hanson, and
M. Gluck (2000). Nonlinear autoas-sociation is not equivalent to PCA.
NeuralComputation 12, 531–545.
Jaynes, E. T. (2003). Probability the-

ory: thelogicofscience. Cambridge
university press.
Jebara, T., R. Kondor, and A. Howard
(2004). Probability product kernels.
J. of Machine Learning Research 5,
819–844.
Jeffreys, H. (1961). TheoryofProbability.
Oxford.
Jelinek, F. (1997). Statisticalmethodsfor
speechrecognition. MIT Press.
Jensen, C. S., A. Kong, and U. Kjaerulff
(1995). Blocking-gibbs samplingin very large probabilistic expert
systems.Intl.J.
 Human-Computer
Studies, 647–666.
Jermyn, I. (2005). Invariant bayesian
estimation on manifolds. Annalsof

819–844.
Oxford.
Statistics 33 (2), 583–605.
Jerrum, M. and A. Sinclair (1993).
Polynomial-time approximation al-
gorithms for the Ising model. SIAM
J.onComputing 22, 1087–1116.
Jerrum, M. and A. Sinclair (1996).
The markov chain monte carlomethod: an approach to approxi-mate counting and integration. InD. S. Hochbaum (Ed.), Approxima-
tion Algorithms for NP-hard prob-
lems. PWS Publishing.
Jerrum, M., A. Sinclair, and E. Vigoda
(2004). A polynomial-time approx-
imation algorithm for the perma-nent of a matrix with non-negative

entries.Journal of the ACM , 671–
697.
Ji, S., D. Dunson, and L. Carin
(2009). Multi-task compressivesensing.IEEETrans.SignalProcess-
ing 57(1).

1028 BIBLIOGRAPHY
Ji, S., L. Tang, S. Yu, and J. Ye (2010).
A shared-subspace learning frame-
work for multi-label classiﬁcation.ACMTrans.onKnowledgeDiscovery
from Data 4 (2).
Jirousek, R. and S. Preucil (1995). On
the effective implementation of the
iterative proportional ﬁtting proce-
dure.Computational Statistics &
DataAnalysis 19, 177–189.
Joachims, T. (2006). Training Linear
SVMs in Linear Time. In Proc. of
theInt’lConf.onKnowledgeDiscov-
eryandDataMining.
Joachims, T., T. Finley, and C.-N. Yu
(2009). Cutting-Plane Training of
Structural SVMs. Machine Learn-
ing 77(1), 27–59.
Johnson, J. K., D. M. Malioutov, and
A. S. Willsky (2006). Walk-sum in-
terpretation and analysis of gaus-sian belief propagation. In NIPS,
pp. 579–586.
Johnson, M. (2005). Capacity and
complexity of HMM duration mod-
eling techniques. Signal Processing
Letters 12(5), 407–410.
Johnson, N. (2009). A study of the
NIPS feature selection challenge.
Technical report, Stanford.
Johnson, V. and J. Albert (1999). Ordi-

naldatamodeling. Springer.
Jones, B., A. Dobra, C. Carvalho,
C. Hans, C. Carter, and M. West(2005). Experiments in stochasticcomputation for high-dimensional
graphical models. Statistical Sci-
ence 20, 388–400.
Jordan, M. I. (2007). An introduction to
probabilistic graphical models. In
preparation.
Jordan, M. I. (2011). The era of big
data. InISBA Bulletin, Volume 18,
pp. 1–3.
Jordan, M. I., Z. Ghahramani, T. S.
Jaakkola, and L. K. Saul (1998). Anintroduction to variational meth-ods for graphical models. In M. Jor-

pp. 1–3.
dan (Ed.),Learning in Graphical
Models. MIT Press.
Jordan, M. I. and R. A. Jacobs (1994).
Hierarchical mixtures of expertsand the EM algorithm. Neural
Computation 6, 181–214.Journee, M., Y. Nesterov, P. Richtarik,
and R. Sepulchre (2010). General-
ized power method for sparse prin-
cipal components analysis. J. of
MachineLearningResearch 11, 517–
553.
Julier, S. and J. Uhlmann (1997). A
new extension of the Kalman ﬁl-ter to nonlinear systems. In Proc.
ofAeroSense: The11thIntl.Symp.on

553.
Aerospace/Defence Sensing, Simula-tionandControls.
Jurafsky, D. and J. H. Martin (2000).
Speech and language processing:An Introduction to Natural Lan-
guage Processing, Computational
Linguistics,andSpeechRecognition.
Prentice-Hall.
Jurafsky, D. and J. H. Martin (2008).
Speech and language processing:
An Introduction to Natural Lan-
Prentice-Hall. 2nd edition.
Kaariainen,
M. and J. Langford (2005).
A Comparison of Tight Generaliza-

tion Bounds. In Intl. Conf. on Ma-
Kaelbling, L., M. Littman, and
A. Moore (1996). Reinforcement
learning: A survey. J. of AI Re-
search 4, 237–285.
Kaelbling, L. P., M. Littman, and
A. Cassandra (1998). Planningand acting in partially observable
stochastic domains. ArtiﬁcialIntel-
ligence 101.
Kaiser, H. (1958). The varimax crite-
rion for analytic rotation in factoranalysis.Psychometrika 23 (3).
Kakade, S., Y. W. Teh, and S. Roweis

(2002). An alternate objective func-tion for markovian ﬁelds. In Intl.
Kanazawa, K., D. Koller, and S. Rus-
sell (1995). Stochastic simulation al-
gorithms for dynamic probabilisticnetworks. In UAI.
Kandel, E., J. Schwarts, and T. Jessell
(2000).PrinciplesofNeuralScience.
McGraw-Hill.
Kappen, H. and F. Rodriguez (1998).
Boltzmann machine learning usingmean ﬁeld theory and linear re-sponse correction. In NIPS.Karhunen, J. and J. Joutsensalo

(1995). Generalizations of princi-pal component analysis, optimiza-tion problems, and neural net-
works.Neural Networks 8 (4), 549–
562.
Kass, R. and L. Wasserman (1995). A
reference bayesian test for nested
hypotheses and its relationship to
the schwarz criterio. J. of the Am.
Stat.Assoc. 90 (431), 928–934.
Katayama, T. (2005). SubspaceMethods
for Systems Identiﬁcation. Springer
Verlag.
Kaufman, L. and P. Rousseeuw (1990).
Finding Groups in Data: An Intro-
ductiontoClusterAnalysis. Wiley.

562.
Verlag.
Kawakatsu, H. and A. Largey (2009).
EM algorithms for ordered probit
models with endogenous regres-
sors.TheEconometricsJournal12 (1),
164–186.
Kearns, M. J. and U. V. Vazirani (1994).
An Introduction to Computational
LearningTheory. MIT Press.
Kelley, J. E. (1960). The cutting-plane
method for solving convex pro-
grams.J. of the Soc. for Industrial
andAppliedMath. 8, 703–712.
Kemp, C., J. Tenenbaum, S. Niyogi, and
T. Griffiths (2010). A probabilistic
model of theory formation. Cogni-
tion 114, 165–196.

164–186.
Kemp, C., J. Tenenbaum, T. Y. T. Grif-
ﬁths and, and N. Ueda (2006).Learning systems of concepts withan inﬁnite relational model. InAAAI.
Kersting, K., S. Natarajan, and D. Poole
(2011). Statistical Relational AI:Logic, Probability and Computa-tion. Technical report, UBC.
Khan,M.
E., B. Marlin, G. Bouchard,
and K. P. Murphy (2010). Varia-tional bounds for mixed-data fac-tor analysis. In NIPS.
Khan, Z., T. Balch, and F. Dellaert

Khan,M.
(2006). MCMC Data Associationand Sparse Factorization Updatingfor Real Time Multitarget Trackingwith Merged and Multiple Mea-surements. IEEE Trans. on Pat-
tern Analysis and Machine Intelli-gence 28(12).
Kirkpatrick, S., C. G. Jr., and M. Vecchi
(1983). Optimization by simulatedannealing.Science 220, 671–680.

BIBLIOGRAPHY 1029
Kitagawa, G. (2004). The two-ﬁlter for-
mula for smoothing and an im-
plementation of the Gaussian-sum
smoother.AnnalsoftheInstituteof
Statistical Mathematics 46 (4), 605–
623.
Kjaerulff, U. (1990). Triangulation of
graphs – algorithms giving small
total state space. Technical ReportR-90-09, Dept. of Math. and Comp.Sci., Aalborg Univ., Denmark.
Kjaerulff, U. and A. Madsen (2008).
Bayesian Networks and Inﬂuence
Diagrams: A Guide to ConstructionandAnalysis. Springer.
Klaassen, C. and J. A. Wellner (1997).
Efficient estimation in the bivari-
ate noramal copula model: Nor-mal margins are least favorable.
Bernoulli 3 (1), 55–77.
Klami, A. and S. Kaski (2008). Proba-
bilistic approach to detecting de-
pendencies between data sets.Neurocomputing 72, 39–46.
Klami, A., S. Virtanen, and S. Kaski
(2010). Bayesian exponential fam-ily projections for coupled datasources. InUAI.
Kleiner, A., A. Talwalkar, P. Sarkar, and

M. I. Jordan (2011). A scalable boot-strap for massive data. Technicalreport, UC Berkeley.
Kneser, R. and H. Ney (1995). Im-
proved backing-off for n-gram lan-
guage modeling. In Intl. Conf. on
Acoustics, Speech and Signal Proc.,
Volume 1, pp. 181–184.
Ko, J. and D. Fox (2009). GP-
BayesFilters: Bayesian Filtering Us-ing Gaussian Process Predictionand Observation Models. Au-
tonomousRobotsJournal .
Kohn, R., M. Smith, and D. Chan
(2001). Nonparametric regressionusing linear combinations of basis

functions.Statistical Computing 11,
313–322.
Koivisto, M. (2006). Advances in ex-
act Bayesian structure discovery in
Bayesian networks. In UAI.
Koivisto, M. and K. Sood (2004). Ex-
Bayesian networks. J. of Machine
LearningResearch 5, 549–573.
Koller, D. and N. Friedman (2009).
Probabilistic Graphical Models:
Principles and Techniques . MIT
Press.Koller, D. and U. Lerner (2001). Sam-
pling in Factored Dynamic Sys-
tems. In A. Doucet, N. de Fre-itas, and N. Gordon (Eds.), Sequen-

313–322.
tial Monte Carlo Methods in Prac-tice. Springer.
Kolmogorov, V. (2006, October). Con-
vergent Tree-reweighted MessagePassing for Energy Minimization.
IEEETrans.onPatternAnalysisand
Machine Intelligence 28 (10), 1568–
1583.
Kolmogorov, V. and M. Wainwright
(2005). On optimality properties
of tree-reweighted message pass-ing. InUAI, pp. 316–322.
Kolmogorov, V. and R. Zabin (2004).
What energy functions can be min-imized via graph cuts? IEEETrans.
on Pattern Analysis and Machine
Intelligence 26 (2), 147–159.

1583.
Komodakis, N., N. Paragios, and
G. Tziritas (2011). MRF Energy Mini-
mization and Beyond via Dual De-composition. IEEET
 rans. on Pat-
tern Analysis and Machine Intelli-gence 33(3), 531–552.
Koo, T., A. M. Rush, M. Collins,
T. Jaakkola, and D. Sontag (2010).Dual Decomposition for Parsingwith Non-Projective Head Au-tomata. In Proc. EMNLP, pp.
1288â˘A¸S1298.
Koren, Y. (2009a). The bellkor solution
to the netﬂix grand prize. Techni-cal report, Yahoo! Research.
Koren, Y. (2009b). Collaborative ﬁl-

tering with temporal dynamics. In
Proc.oftheInt’lConf.onKnowledge
DiscoveryandDataMining.
Koren, Y., R. Bell, and C. Volinsky
(2009). Matrix factorization tech-
niques for recommender systems.IEEEComputer 42 (8), 30–37.
Krishnapuram, B., L. Carin,
M. Figueiredo, and A. Hartemink(2005). Learning sparse bayesianclassiﬁers: multi-class formulation,fast algorithms, and generalizationbounds. IEEE Transaction on
Pattern Analysis and Machine
Intelligence .
Krizhevsky, A. and G. Hinton (2010).
Using Very Deep Autoencoders

for Content-Based Image Retrieval.Submitted.Kschischang, F., B. Frey, and H.-A.
Loeliger (2001, February). Factorgraphs and the sum-product algo-
rithm.IEEETransInfo.Theory .
Kuan, P., G. Pan, J. A. Thomson,
R. Stewart, and S. Keles (2009).
A hierarchical semi-Markov modelfor detecting enrichment with ap-plication to ChIP-Seq experiments.Technical report, U. Wisconsin.
Kulesza, A. and B. Taskar (2011). Learn-
ing Determinantal Point Processes.InUAI.
Kumar, N. and A. Andreo (1998). Het-

eroscedastic discriminant analysisand reduced rank HMMs for im-proved speech recognition. Speech
Communication 26, 283–297.
Kumar, S. and M. Hebert (2003). Dis-
criminative random ﬁelds: A dis-criminative framework for contex-tual interaction in classiﬁcation. InIntl.Conf.onComputerVision.
Kuo, L. and B. Mallick (1998). Vari-
able selection for regression mod-
els.SankhyaSeriesB 60, 65–81.
Kurihara, K., M. Welling, and N. Vlas-
sis (2006). Accelerated variational
DP mixture models. In NIPS.
Kushner, H. and G. Yin (2003).

Stochasticapproximationandrecur-
sive algorithms and applications.
Springer.
Kuss and C. Rasmussen (2005). As-
sessing approximate inference forbinary gaussian process classiﬁca-
tion.J. of Machine Learning Re-
search 6, 1679–1704.
Kwon, J. and K. Murphy (2000). Mod-
eling freeway traffic with coupled
HMMs. Technical report, Univ. Cal-ifornia, Berkeley.
Kyung, M., J. Gill, M. Ghosh, and
G. Casella (2010). Penalized Regres-sion, Standard Errors and Bayesian
Lassos.BayesianAnalysis5 (2), 369–
412.
Lacoste-Julien, S., F. Huszar, and
Z. Ghahramani (2011). Approximate

Springer.
412.
inference for the loss-calibratedBay
esian. InAI/Statistics.
Lacoste-Julien, S., F. Sha, and M. I. Jor-
dan (2009). DiscLDA: Discrimina-tive learning for dimensionality re-duction and classiﬁcation. In NIPS.

1030 BIBLIOGRAPHY
Lafferty, J., A. McCallum, and
F. Pereira (2001). Conditional ran-
dom ﬁelds: Probabilistic modelsfor segmenting and labeling se-
quence data. In Intl. Conf. on Ma-
Lange, K., R. Little, and J. Taylor (1989).
Robust statistical modeling usingthe t disribution. J. of the Am. Stat.
Assoc. 84(408), 881–896.
Langville, A. and C. Meyer (2006).
Updating Markov chains with an
eye on Google’s PageRank. SIAM
J. on Matrix Analysis and Applica-tions 27(4), 968–987.
Larranaga, P., C. M. H. Kuijpers,
M. Poza, and R. H. Murga (1997).Decomposing bayesian networks:triangulation of the moral graphwith genetic algorithms. Statistics
andComputing(UK) 7 (1), 19–34.
Lashkari, D. and P. Golland (2007).
Convex clustering with examplar-based models. In NIPS.
Lasserre, J., C. Bishop, and T. Minka
(2006). Principled hybrids of gen-erative and discriminative models.InCVPR.
Lau, J. and P. Green (2006).
Bayesian model-based clustering
procedures. Journal of Computa-
tional and Graphical Statistics 12,

351–357.
Lauritzen, S. (1996). Graphical Models.
OUP.
Lauritzen, S. (2000). Causal infer-
ence from graphical models. InD. R. C. O. E. Barndoff-Nielsenand C. Klueppelberg (Eds.), Com-
plex stochastic systems. Chapmanand Hall.
Lauritzen, S. and D. Nilsson (2001).
Representing and solving decisionproblems with limited information.ManagementScience 47, 1238–1251.
Lauritzen, S. L. (1992, December).
Propagation of probabilities, meansand variances in mixed graphical
association models. J. of the Am.

351–357.
OUP.
Stat.Assoc. 87 (420), 1098–1108.
Lauritzen, S. L. (1995). The EM al-
gorithm for graphical association
models with missing data. Com-
putational Statistics and Data Anal-ysis 19, 191–201.Lauritzen, S. L. and D. J. Spiegelhal-
ter (1988). Local computations withprobabilities on graphical struc-tures and their applications to ex-pert systems. J.R.Stat.Soc.B B (50),
127–224.
Law, E., B. Settles, and T. Mitchell
(2010). Learning to tag from openvocabulary labels. In Proc. Euro-
peanConf.onMachineLearning.

127–224.
Law, M., M. Figueiredo, and A. Jain
(2004). Simultaneous Feature Se-lection and Clustering Using Mix-ture Models. IEEE Trans. on Pat-
tern Analysis and Machine Intelli-gence 26(4).
Lawrence, N. D. (2005). Probabilis-
tic non-linear principal componentanalysis with gaussian process la-
tent variable models. J.ofMachine
LearningResearch 6, 1783–1816.
Lawrence, N. D. (2012). A unifying
probabilistic
perspective for spec-
tral dimensionality reduction: in-sights and new models. J. of Ma-

chine Learning Research 13, 1609–
1638.
Learned-Miller, E. (2004). Hyperspac-
ings and the estimation of infor-mation theoretic quantities. Tech-nical Report 04-104, U. Mass.Amherst Comp. Sci. Dept.
LeCun, Y., B. Boser, J. S. Denker,
D. Henderson, R. E. Howard,W. Hubbard, and L. D. Jackel (1989,Winter). Backpropagation appliedto handwritten zip code recogni-
tion.NeuralComputation 1 (4), 541–
551.
LeCun, Y., L. Bottou, Y. Bengio,
and P. Haffner (1998, November).
Gradient-based learning applied to

1638.
551.
document recognition. Proceedings
oftheIEEE 86 (11), 2278–2324.
LeCun, Y., S. Chopra, R. Hadsell, F.-J.
Huang, and M.-A. Ranzato (2006).
A tutorial on energy-based learn-
ing. In B. et al. (Ed.), Predicting
StructuredOutputs. MIT press.
Ledoit, O. and M. Wolf (2004a). Honey,
I Shrunk the Sample CovarianceMatrix.J. of Portfolio Manage-
ment 31(1).
Ledoit, O. and M. Wolf (2004b). A well-
conditioned estimator for large-
dimensional covariance matrices. J.
of Multivariate Analysis 88 (2), 365–

411.Lee, A., F. Caron, A. Doucet, and
C. Holmes (2010). A hierarchicalbayesian framework for construct-ing sparsity-inducing priors. Tech-nical report, U. Oxford.
Lee, A., F. Caron, A. Doucet, and
C.Holmes(2011). BayesianSparsity-Path-Analysis of Genetic Associ-ation Signal using Generalized tPrior. Technical report, U. Oxford.
Lee, D. and S. Seung (2001). Algo-
rithms for non-negative matrix fac-torization. In NIPS.
Lee, H., R. Grosse, R. Ranganath, and

A. Ng (2009). Convolutional deepbelief networks for scalable un-supervised learning of hierarchical
representations. In Intl. Conf. on
MachineLearning.
Lee, H., Y. Largman, P. Pham, and
A. Ng (2009). Unsupervised feature
learning for audio classiﬁcation us-ing convolutional deep belief net-works. InNIPS.
Lee, S.-I., V. Ganapathi, and D. Koller
(2006). Efficient structure learn-ing of Markov networks using L1-regularization. In NIPS.
Lee, T. S. and D. Mumford (2003). Hi-
erarchical Bayesian inference in the

visual cortex. J.ofOpticalSocietyof
AmericaA 20 (7), 1434–1448.
Lenk, P., W. S. DeSarbo, P. Green,
and M. Young (1996). Hierarchi-
cal Bayes Conjoint Analysis: Re-covery of Partworth Heterogeneityfrom Reduced Experimental De-
signs.MarketingScience 15 (2), 173–
191.
Lenkoski, A. and A. Dobra (2008).
Bayesian structural learning and
estimation in Gaussian graphicalmodels. Technical Report 545, De-partment of Statistics, University ofWashington.
Lepar, V. and P. P. Shenoy (1998).

191.
A Comparison of Lauritzen-Spiegelhalter, Hugin and Shenoy-Shafer Architectures for ComputingMarginals
of Probability Distribu-
tions. In G. Cooper and S. Moral(Eds.),UAI, pp. 328–337. Morgan
Kaufmann.
Lerner, U. and R. Parr (2001). Infer-
ence in hybrid networks: Theoreti-cal limits and practical algorithms.InUAI.

BIBLIOGRAPHY 1031
Leslie, C., E. Eskin, A. Cohen, J. We-
ston, and W. Noble (2003). Mis-
match string kernels for discrimi-native protein classiﬁcation. Bioin-
formatics 1 , 1–10.
Levy, S. (2011). In The Plex: How
Google Thinks, Works, and ShapesOurLives. Simon & Schuster.
Li, L., W. Chu, J. Langford, and
X. Wang (2011). Unbiased offlineevaluation of contextual-bandit-based news article recommenda-tion algorithms. In WSDM.
Liang, F., S. Mukherjee, and M. West
(2007). Understanding the use ofunlabelled data in predictive mod-
elling.Statistical Science 22, 189–
205.
Liang, F., R. Paulo, G. Molina,
M. Clyde, and J. Berger (2008). Mix-
tures of g-priors for Bayesian Vari-
able Selection. J. of the Am. Stat.
Assoc. 103 (481), 410–423.
Liang, P. and M. I. Jordan (2008).
An asymptotic analysis of gen-
erative, discriminative, and pseu-dolikelihood estimators. In In-
ternational Conference on MachineLearning(ICML).
Liang, P. and D. Klein. Online EM
for Unsupervised Models. In Proc.
NAACLConference.

205.
Liao, L., D. J. Patterson, D. Fox, and
H. Kautz (2007). Learning andInferring Transportation Routines.
ArtiﬁcialIntelligence 171 (5), 311–331.
Lindley, D. (1982). Scoring rules and
the inevetability of probability. ISI
Review 50, 1–26.
Lindley, D. V. (1972). BayesianStatistics:
AReview. SIAM.
Lindley, D. V. and L. D. Phillips (1976).
Inference for a Bernoulli Process
(A Bayesian View). The American
Statistician 30 (3), 112–119.
Lindsay, B. (1988). Composite like-
lihood methods. Contemporary
Mathematics 80 (1), 221–239.

Lipton, R. J. and R. E. Tarjan (1979).
A separator theorem for planar
graphs.SIAM Journal of Applied
Math 36, 177–189.
Little., R. J. and D. B. Rubin (1987). Sta-
tistical Analysis with Missing Data.
New York: Wiley and Son.Liu, C. and D. Rubin (1995). ML Esti-
mation of the T distribution usingEM and its extensions, ECM andECME.StatisticaSinica 5, 19–39.
Liu, H., J. Lafferty, and L. Wasserman
(2009). The nonparanormal: Semi-parametric estimation of high di-mensional undirected graphs. J.
of Machine Learning Research 10,2295–2328.

Liu, J. (2001). MonteCarloStrategiesin
ScientiﬁcComputation. Springer.
Liu, J. S., W. H. Wong, and A. Kong
(1994). Covariance structure of thegibbs sampler with applicationsto the comparisons of estima-tors and augmentation schemes.
Biometrika 81 (1), 27–40.
Liu,T
.-Y. (2009). Learning to rank
for information retrieval. Founda-
tionsandTrendsinInformationRe-
trieval 3(3), 225–331.
Lizotte, D. (2008). Practical Bayesian
optimization. Ph.D. thesis, U. Al-
berta.
Ljung, L. (1987). System Identiﬁciation:

Liu,T
berta.
TheoryfortheUser. Prentice Hall.
Lo, C. H. (2009). Statistical methods
forhighthroughputgenomics. Ph.D.thesis, UBC.
Lo, K., F. Hahne, R. Brinkman, R. Ryan,
and R. Gottardo (2009, May). ﬂow-clust: a bioconductor package forautomated gating of ﬂow cytome-
try data.BMC Bioinformatics 10,
145+.
Lopes, H. and M. West (2004).
Bayesian model assessment in fac-
tor analysis. StatisicaSinica 14, 41–
67.
Lowe, D. G. (1999). Object recognition
from local scale-invariant features.InProc. of the International Con-

145+.
67.
ference on Computer Vision ICCV,
Corfu, pp. 1150–1157.
Luce, R. (1959). Individual choice be-
havior: A theoretical analysis.W i -
ley.
Lunn, D., N. Best, and J. Whit-
taker (2009). Generic reversiblejump MCMC using graphical mod-
els.Statistics and Computing 19 (4),
395–408.Lunn, D., A. Thomas, N. Best, and
D. Spiegelhalter (2000). WinBUGS
– a Bayesian modelling framework:concepts, structure, and extensibil-
ity.Statistics and Computing 10,
325–337.
Ma, H., H. Yang, M. Lyu, and I. King

ley.
325–337.
(2008). SoRec: Social recommenda-
tion using probabilistic matrix fac-
torization. In Proc. of17th Conf.on
Information and Knowledge Man-
agement.
Ma, S., C. Ji, and J. Farmer (1997). An
efficient EM-based training algo-rithm for feedforward neural net-works.NeuralNetworks 10 (2), 243–
256.
Maathuis, M., D. Colombo, M. Kalisch,
and P. BÃijhlmann (2010). Pre-dicting causal effects in large-scalesystems from observational data.
NatureMethods 7, 247–248.
Maathuis, M., M. Kalisch, and
P. BÃijhlmann (2009). Estimating

agement.
256.
high-dimensional intervention ef-fects from observational data. An-
nalsofStatistics 37 , 3133–3164.
MacKay, D. (1992). Bayesian interpo-
lation.Neural Computation 4, 415–
447.
MacKay, D. (1995a). Developments in
probabilistic modeling with neuralnetworks — ensemble learning. In
Proc. 3rd Ann. Symp. Neural Net-
works.
MacKay, D. (1995b). Probable net-
works and plausible predictions
— a review of practical Bayesianmethods
for supervised neural net-
works.Network.
MacKay, D. (1997). Ensemble learning

447.
works.
for Hidden Markov Models. Tech-nical report, U. Cambridge.
MacKay, D. (1999). Comparision of
approximate methods for handling
hyperparameters. NeuralComputa-
tion 11(5), 1035–1068.
MacKay, D. (2003). InformationTheory,
Inference,andLearningAlgorithms.
Macnaughton-Smith, P., W. T.
Williams, M. B. Dale, and G. Mock-ett (1964). Dissimilarity analysis:a new technique of hierarchicalsub-division. Nature 202, 1034 –
1035.

1032 BIBLIOGRAPHY
Madeira, S. C. and A. L. Oliveira
(2004). Biclustering algorithms for
biological data analysis: A survey.IEEE/ACM Transactions on Compu-
tational Biology and Bioinformat-
ics 1(1), 24–45.
Madigan, D. and A. Raftery (1994).
Model selection and accountingfor model uncertainty in graphical
modelsusingOccam’swindow. J.of
theAm.Stat.Assoc. 89, 1535–1546.
Madsen, R., D. Kauchak, and C. Elkan
(2005). Modeling word burstiness
using the Dirichlet distribution. In
Mairal, J., F. Bach, J. Ponce, and
G. Sapiro (2010). Online learning
for matrix factorization and sparse
coding.J. of Machine Learning Re-
search 11, 19–60.
Mairal, J., M. Elad, and G. Sapiro
(2008). Sparse representation for
color image restoration. IEEETrans.
onImageProcessing 17 (1), 53–69.
Malioutov, D., J. Johnson, and A. Will-
sky (2006). Walk-sums and beliefpropagation in gaussian graphical
models.J.ofMachineLearningRe-
search 7, 2003–2030.
Mallat, S., G. Davis, and Z. Zhang
(1994, July). Adaptive time-

frequency decompositions. SPIE
Journal of Optical Engineering 33,2183–2919.
Mallat, S. and Z. Zhang (1993). Match-
ing pursuits with time-frequencydictionaries. IEEE Transactions on
SignalProcessing 41 (12), 3397–3415.
Malouf, R. (2002). A comparison of
algorithms for maximum entropyparameter estimation. In Proc.
Sixth Conference on Natural Lan-guage Learning (CoNLL-2002), pp.49–55.
Manning, C., P. Raghavan, and
H. Schuetze (2008). Introduction
toInformationRetrieval. CambridgeUniversity Press.
Manning, C. and H. Schuetze (1999).

Foundations of statistical natural
languageprocessing. MIT Press.
Mansinghka, V., D. Roy, R. Rifkin,
and J. Tenenbaum (2007). AClass:
An online algorithm for generativeclassiﬁcation. In AI/Statistics.Mansinghka, V., P. Shafto, E. Jonas,
C. Petschulat, and J. Tenenbaum(2011). Cross-Categorization: ANonparametric Bayesian Methodfor Modeling Heterogeneous, HighDimensional Data. Technical re-port, MIT.
Margolin, A., I. Nemenman, K. Basso,

C. Wiggins, G. Stolovitzky, and R. F.abd A. Califano (2006). ARACNE:An Algorithm for the Reconstruc-tion of Gene Regulatory Networksin a Mammalian Cellular Context.BMCBionformatics 7 .
Marin, J.-M. and C. Robert (2007).
BayesianCore: apracticalapproach
tocomputationalBayesianstatistics.
Springer.
Marks, T. K. and J. R. Movellan (2001).
Diffusion networks, products of ex-perts, and factor analysis. Techni-cal report, University of CaliforniaSan Diego.
Marlin, B. (2003). Modeling user rat-
ing proﬁles for collaborative ﬁlter-ing. InNIPS.

Springer.
Marlin, B. (2008). Missing
 Data Prob-
lems in Machine Learning. Ph.D.thesis, U. Toronto.
Marlin, B., E. Khan, and K. Murphy
(2011). Piecewise Bounds for Es-timating Bernoulli-Logistic Latent
Gaussian Models. In Intl. Conf. on
Marlin, B. and R. Zemel (2009). Collab-
orative prediction and ranking with
non-random missing data. In Proc.
of the 3rd ACM Conference on Rec-
ommenderSystems.
Marlin, B. M., K. Swersky, B. Chen, and
N. de Freitas (2010). Inductive prin-
ciples for restricted boltzmann ma-chine learning. In AI/Statistics.
Marroquin, J., S. Mitter, and T. Pog-
gio (1987). Probabilistic solutionof ill-posed problems in computa-

tional vision. J. of the Am. Stat. As-
soc. 82(297), 76–89.
Martens, J. (2010). Deep learning via
hessian-free optimization. In Intl.
Maruyama, Y. and E. George (2008). A
g-prior extension for p>n. Tech-
nical report, U. Tokyo.
Mason, L., J. Baxter, P. Bartlett, and
M. Frean (2000). Boosting algo-
rithms as gradient descent. InNIPS, Volume 12, pp. 512–518.Matthews, R. (1998). Bayesian Critique
of Statistics in Health: The GreatHealth Hoax.
Maybeck, P. (1979). Stochastic models,
estimation, and control. AcademicPress.

Mazumder, R. and T. Hastie (2012).
The Graphical Lasso: New Insightsand Alternatives. Technical report.
McAuliffe, J., D. Blei, and M. Jordan
(2006). Nonparametric empiricalbayes for the dirichlet process mix-
ture model. Statistics and Comput-
ing 16(1), 5–14.
McCallum, A. (2003). Efficiently induc-
ing features of conditional random
ﬁelds. InUAI.
McCallum, A., D. Freitag, and
F. Pereira (2000). Maximum En-tropy Markov Models for Informa-tion Extraction and Segmentation.

McCallum, A. and K. Nigam (1998).
A comparison of event models for
naive Bayes text classiﬁcation. In
AAAI/ICML workshop on Learning
forTextCategorization.
McCray, A. (2003). An upper level
ontology for the biomedical do-
main.Comparative and Functional
Genomics 4, 80–84.
McCullagh, P. and J. Nelder (1989).
Generalized linear models. Chap-
man and Hall. 2nd edition.
McCullich, W. and W. Pitts (1943). A
logical calculus of the ideas imma-
nent in nervous activity. Bulletinof
MathematicalBiophysics 5, 115–137.

McDonald, J. and W. Newey (1988).
Partially Adaptive Estimation of Re-
gression Models via the General-ized t Distribution. Econometric
Theory 4(3),428–445.
McEliece,
R. J., D. J. C. MacKay, and
J. F. Cheng (1998). Turbo decod-ing as an instance of Pearl’s ’beliefpropagation’ algorithm. IEEE J. on
SelectedAreasinComm. 16 (2), 140–
152.
McFadden, D. (1974). Conditional logit
analysis of qualitative choice be-havior. In P. Zarembka (Ed.), Fron-
tiers in econometrics, pp. 105–142.Academic Press.

BIBLIOGRAPHY 1033
McGrayne, S. B. (2011). The the-
ory that would not die: how
Bayes’rulecrackedtheenigmacode,hunted down Russian submarines,
and emerged triumphant from two
centuries of controversy. Yale Uni-
versity Press.
McKay, B. D., F. E. Oggier, G. F. Royle,
N. J. A. Sloane, I. M. Wanless, andH. S. Wilf (2004). Acyclic digraphsand eigenvalues of (0,1)-matrices. J.
IntegerSequences 7 (04.3.3).
McKay, D. and L. C. B. Peto (1995).
A hierarchical dirichlet language
model.NaturalLanguageEngineer-
ing 1(3), 289–307.
McLachlan, G. J. and T. Krishnan
(1997).The EM Algorithm and Ex-
tensions. Wiley.
Meek, C. and D. Heckerman (1997).
Structure and parameter learn-
ing for causal independence andcausal interaction models. In UAI,
pp. 366–375.
Meek, C., B. Thiesson, and D. Hecker-
man (2002). Staged mixture mod-elling and boosting. In UAI, San
Francisco, CA, pp. 335–343. Mor-gan Kaufmann.
Meila, M. (2001). A random walks view
of spectral segmentation. In AI/S-
Meila, M. (2005). Comparing cluster-

ings: an axiomatic view. In Intl.
Meila, M. and T. Jaakkola (2006).
Tractable Bayesian learning of tree
belief networks. StatisticsandCom-
puting 16, 77–92.
Meila, M. and M. I. Jordan (2000).
Learning with mixtures of trees. J.
of Machine Learning Research 1,1 –
48.
Meinshausen, N. (2005). A note on the
lasso for gaussian graphical modelselection. Technical report, ETHSeminar fur Statistik.
Meinshausen, N. and P. Buhlmann

48.
(2006). High dimensional graphsand variable selection with thelasso.The Annals of Statistics 34,
1436–1462.
Meinshausen, N. and P. BÃijhlmann
(2010). Stability selection. J.ofRoyal
Stat.Soc.SeriesB 72, 417–473.Meltzer, T., C. Yanover, and Y. Weiss
(2005). Globally optimal solu-tions for energy minimization instereo vision using reweighted be-lief propagation. In ICCV, pp. 428–
435.
Meng, X. L. and D. van Dyk (1997).
The EM algorithm — an old folksong sung to a fast new tune (with
Discussion). J.RoyalStat.Soc.B 59,

435.
511–567.
Mesot, B. and D. Barber (2009). A Sim-
ple Alternative Derivation of the
Expectation Correction Algorithm.
IEEE Signal Processing Letters 16 (1),
121–124.
Metropolis, N., A. Rosenbluth,
M. Rosenbluth, A. Teller, and
E. Teller (1953). Equation of statecalculations by fast computing
machines.J.ofChemicalPhysics 21,
1087–1092.
Metz,
C. (2010). Google behavioral ad
targeter is a Smart Ass. The Regis-
ter.
Miller, A. (2002). Subsetselectioninre-
gression. Chapman and Hall. 2nd
edition.
Mimno, D. and A. McCallum (2008).

121–124.
Metz,
ter.
edition.
Topic models conditioned on ar-bitrary features with dirichlet-multinomial regression. In UAI.
Minka, T. (1999). Pathologies of ortho-
dox statisics. Technical report, MITMedia Lab.
Minka, T. (2000a). Automatical choice
of dimensionality for PCA. Techni-cal report, MIT.
Minka, T. (2000b). Bayesian linear re-
gression. Technical report, MIT.
Minka, T. (2000c). Bayesian model av-
eraging is not model combination.Technical report, MIT Media Lab.
Minka, T. (2000d). Empirical risk min-

imization is an incomplete induc-tive principle. Technical report,MIT.
Minka, T. (2000e). Estimating a Dirich-
let distribution. Technical report,MIT.
Minka, T. (2000f). Inferring a Gaussian
distribution. Technical report, MIT.
Minka, T. (2001a). Bayesian inference
of a uniform distribution. Techni-cal report, MIT.Minka, T. (2001b). Empirical Risk Min-
Minka, T. (2001c). Expectation propa-
gationforapproximateBayesianin-ference. InUAI.

Minka, T. (2001d). A family of algo-
rithmsforapproximateBayesianin-
ference. Ph.D. thesis, MIT.
Minka, T. (2001e). Statistical ap-
proaches to learning and discovery
10-602: Homework assignment 2,question 5. Technical report, CMU.
Minka, T. (2003). A comparison of nu-
merical optimizers for logistic re-gression. Technical report, MSR.
Minka, T. (2005). Divergence measures
and message passing. Technical re-port, MSR Cambridge.
Minka, T. and Y. Qi (2003). Tree-

structured approximations by ex-pectation propagation. In NIPS.
Minka, T., J. Winn, J. Guiver, and
D. Knowles (2010). Infer.NET 2.4.Microsoft Research Cambridge.http://research.microsoft.com/infernet.
Minsky, M. and S. Papert (1969). Per-
ceptrons. MIT Press.
Mitchell, T. (1997). Machine Learning.
Mitchell, T. and J. Beauchamp (1988).
Bayesian Variable Selection in Lin-
ear Regression. J. of the Am. Stat.
Assoc. 83, 1023–1036.
Mobahi, H., R. Collobert, and J. We-

ston (2009). Deep learning from
temporal coherence in video. In
Mockus, J., W. Eddy, A. Mockus,
L. Mockus, and G. Reklaitis (1996).
BayesianHeuristicApproachtoDis-
crete and Global Optimization: Al-gorithms, Visualization, Software,andApplications. Kluwer.
Moghaddam, B., A. Gruber, Y. Weiss,
and S. Avidan (2008). Sparse re-
gression as a sparse eigenvalue
problem. In Information Theory &
ApplicationsWorkshop(ITA’08).
Moghaddam, B., B. Marlin, E. Khan,
andK.
Murphy (2009). Accel-

andK.
erating bayesian structural infer-
ence for non-decomposable gaus-sian graphical models. In NIPS.

1034 BIBLIOGRAPHY
Moghaddam, B. and A. Pentland
(1995). Probabilistic visual learning
for object detection. In Intl. Conf.
onComputerVision.
Mohamed, S., K. Heller, and
Z. Ghahramani (2008). Bayesian
Exponential Family PCA. In NIPS.
Moler, C. (2004). NumericalComputing
withMATLAB. SIAM.
Morris, R. D., X. Descombes, and
J. Zerubia (1996). The Ising/Pottsmodel is not well suited to seg-
mentation tasks. In IEEEDSPWork-
shop.
Mosterman, P. J. and G. Biswas (1999).
Diagnosis of continuous valued
systems in transient operating re-gions.IEEETrans.onSystems,Man,
and Cybernetics, Part A 29 (6), 554–
565.
Moulines, E., J.-F. Cardoso, and E. Gas-
siat (1997). Maximum likelihoodfor blind separation and deconvo-lution of noisy signals using mix-ture models. In Proc. IEEE Int.
Conf. on Acoustics, Speech and Sig-
nal Processing (ICASSP’97), Munich,
Germany, pp. 3617–3620.
Muller, P., G. Parmigiani, C. Robert,
and J. Rousseau (2004). Optimalsample size for multiple testing:the case of gene expression mi-

565.
croarrays. J. of the Am. Stat. As-
soc. 99, 990–1001.
Mumford, D. (1994). Neuronal archi-
tectures for pattern-theoretic prob-
lems. In C. Koch and J. Davis (Eds.),LargeScaleNeuronalTheoriesoftheBrain. MIT Press.
Murphy, K. (2000). Bayesian map
learning in dynamic environments.InNIPS, Volume 12.
Murphy, K. and M. Paskin (2001). Lin-
ear time inference in hierarchicalHMMs. InNIPS.
Murphy, K., Y. Weiss, and M. Jordan
(1999). Loopy belief propagation forapproximate inference: an empiri-cal study. In UAI.

Murphy, K. P. (1998). Filtering and
smoothing in linear dynamical sys-tems using the junction tree algo-rithm. Technical report, U.C. Berke-ley, Dept. Comp. Sci.
Murray, I. and Z. Ghahramani (2005).
A note on the evidence andbayesian occam’s razor. Technicalreport, Gatsby.Musso, C., N. Oudjane, and F. LeGland
(2001). Improving regularized par-ticle ﬁlters. In A. Doucet, J. F. G.de Freitas, and N. Gordon (Eds.),Sequential Monte Carlo Methods inPractice. Springer.
Nabney, I. (2001). NETLAB: algorithms

forpatternrecognition. Springer.
Neal, R. (1992). Connectionist learning
of belief networks. ArtiﬁcialIntelli-
gence 56, 71–113.
Neal, R. (1993). Probabilistic Inference
Using Markov Chain Monte CarloMethods. Technical report, Univ.Toronto.
Neal, R. (1996). Bayesian learning for
neuralnetworks. Springer.
Neal, R. (1997). Monte Carlo Im-
plementation of Gaussian ProcessModels for Bayesian Regressionand Classiﬁcation. Technical Re-port
9702, U. Toronto.
Neal, R. (1998). Erroneous Results

in ’Marginal Likelihood from theGibbs Output’. Technical report, U.Toronto.
Neal, R. (2000). Markov Chain Sam-
pling Methods for Dirichlet Process
Mixture Models. J. of Computa-
tional and Graphical Statistics 9 (2),
249–265.
Neal, R. (2003a). Slice sampling. An-
nalsofStatistics 31 (3), 7–5–767.
Neal, R. (2010). MCMC using Hamil-
tonian Dynamics. In S. Brooks,
A. Gelman, G. Jones, and X.-L.Meng (Eds.), Handbook of Markov
Chain Monte Carlo. Chapman &Hall.
Neal, R. and D. MacKay (1998).

249–265.
Likelihood-based boosting. Techni-cal report, U. Toronto.
Neal, R. and J. Zhang (2006). High
dimensional classiﬁcation Bayesianneural networks and Dirichlet dif-fusion trees. In I. Guyon, S. Gunn,M. Nikravesh, and L. Zadeh (Eds.),FeatureExtraction. Springer.
Neal, R. M. (2001). Annealed impor-
tance sampling. StatisticsandCom-
puting 11, 125–139.
Neal, R. M. (2003b). Density Model-
ing and Clustering using DirichletDiffusion Trees. In J. M. Bernardoet al. (Eds.),BayesianStatistics7, pp.

619–629. Oxford University Press.Neal, R. M. and G. E. Hinton (1998).
A new view of the EM algorithmthat justiﬁes incremental and othervariants. In M. Jordan (Ed.), Learn-
inginGraphicalModels. MIT Press.
Neapolitan, R. (2003). Learning
BayesianNetworks. Prentice Hall.
Neﬁan, A., L. Liang, X. Pi, X. Liu,
and K. Murphy (2002). DynamicBayesian Networks for Audio-
Visual Speech Recognition. J. Ap-
pliedSignalProcessing .
Nemirovski, A. and D. Yudin (1978). On

Cezari’s convergence of the steep-est descent method for approxi-mating saddle points of convex-
concave functions. Soviet Math.
Dokl. 19.
Nesterov, Y. (2004). Introductory Lec-
tures on Convex Optimization. A
basiccourse. Kluwer.
Newton, M., D. Noueiry, D. Sarkar,
and P. Ahlquist (2004). Detecting
differential gene expression witha semiparametric hierarchical mix-ture method. Biostatistics 5, 155–
176.
Newton, M. and A. Raftery (1994). Ap-
proximate Bayesian Inference withthe Weighted Likelihood Bootstrap.
J. of Royal Stat. Soc. Series B 56 (1),
3–48.

Dokl. 19.
176.
3–48.
Ng, A., M. Jordan, and Y. Weiss (2001).
On Spectral Clustering: Analysis
and an algorithm. In NIPS.
Ng, A. Y. and M. I. Jordan (2002). On
discriminative vs. generative classi-ﬁers: A comparison of logistic re-gression and naive bayes. In NIPS-
14.
Nickisch, H. and C. Rasmussen (2008).
Approximations for binary gaus-
sian process classiﬁcation. J.ofMa-
chineLear
ning Research 9, 2035–
2078.
Nilsson, D. (1998). An efficient algo-
rithm for ﬁnding the M most prob-
able conﬁgurations in a probabilis-
tic expert system. Statistics and
Computing 8 , 159–173.

14.
chineLear
2078.
Nilsson, D. and J. Goldberger (2001).
Sequentially ﬁnding the N-Best Listin Hidden Markov Models. In Intl.
JointConf.onAI, pp. 1280–1285.
Nocedal, J. and S. Wright (2006). Nu-
mericalOptimization. Springer.

BIBLIOGRAPHY 1035
Nowicki, K. and T. A. B. Snijders
(2001). Estimation and prediction
for stochastic blockstructures. Jour-
naloftheAmericanStatisticalAsso-ciation 96 (455), 1077–??
Nowlan, S. and G. Hinton (1992). Sim-
plifying neural networks by soft
weight sharing. Neural Computa-
tion 4(4), 473–493.
Nummiaro, K., E. Koller-Meier, and
L. V. Gool (2003). An adaptive
color-based particle ﬁlter. Image
andVisionComputing 21 (1), 99–110.
Obozinski, G., B. Taskar, and M. I. Jor-
dan (2007). Joint covariate selectionfor grouped classiﬁcation. Techni-cal report, UC Berkeley.
Oh, M.-S. and J. Berger (1992). Adap-
tive importance sampling in Monte
Carlo integration. J. of Statistical
Computation and Simulation 41 (3),
143 – 168.
Oh, S., S. Russell, and S. Sastry (2009).
Markov Chain Monte Carlo Data
Association for Multi-Target Track-ing.IEEETrans.onAutomaticCon-
trol 54(3), 481–497.
O’Hagan, A. (1978). Curve ﬁtting and
optimal design for prediction. J. of
RoyalStat.Soc.SeriesB 40, 1–42.

143 – 168.
O’Hara, R. and M. Sillanpaa (2009).
A Review of Bayesian Variable Se-lection Methods: What, How andWhich.Bayesian Analysis 4 (1), 85–
118.
Olshausen, B. A. and D. J. Field (1996).
Emergence of simple cell recep-tive ﬁeld properties by learning asparse code for natural images. Na-
ture 381, 607–609.
Opper, M. (1998). A Bayesian approach
to online learning. In D. Saad
(Ed.),On-line learning in neural
networks. Cambridge.
Opper, M. and C. Archambeau (2009).
The variational Gaussian approxi-mation revisited. Neural Computa-
tion 21(3), 786–792.
Opper, M. and D. Saad (Eds.) (2001).

118.
Advanced mean ﬁeld methods: the-
oryandpractice. MIT Press.
Osborne, M. R., B. Presnell, and B. A.
Turlach (2000a). A new approach to
variable selection in least squares
problems.IMA Journal of Numeri-
calAnalysis 20 (3), 389–403.Osborne, M. R., B. Presnell, and B. A.
Turlach (2000b). On the lasso and
its dual. J. Computational and
graphicalstatistics 9, 319–337.
Ostendorf, M., V. Digalakis, and
O. Kimball (1996). From HMMs tosegment models: a uniﬁed viewof stochastic modeling for speech
recognition. IEEE Trans. on Speech
and Audio Processing 4 (5), 360–
378.

378.
Overschee, P. V. and B. D. Moor
(1996).Subspace Identiﬁcation for
Linear Systems: Theory, Implemen-
tation, Applications . Kluwer Aca-
demic Publishers.
Paatero, P. and U. Tapper (1994). Pos-
itive matrix factorization: A non-
negative factor model with opti-mal utilization of error estimates ofdata values. Environmetrics 5 , 111–
126.
Padadimit
riou, C. and K. Steiglitz
(1982).Combinatorialoptimization:
Algorithms and Complexity. Pren-tice Hall.
Paisley, J. and L. Carin (2009). Non-

378.
126.
Padadimit
parametric factor analysis with
beta process priors. In Intl. Conf.
Palmer, S. (1999). Vision Science: Pho-
tonstoPhenomenology. MIT Press.
Parise, S. and M. Welling (2005).
Learning in Markov Random
Fields: An Empirical Study. In Joint
StatisticalMeeting.
Park, T. and G. Casella (2008). The
Bayesian Lasso. J. of the Am. Stat.
Assoc. 103 (482), 681–686.
Parviainen, P. and M. Koivisto (2011).
Ancestor relations in the presenceof unobserved variables. In Proc.
European Conf. on Machine Learn-ing.

Paskin, M. (2003). Thin junction tree
ﬁlters for simultaneous localizationand mapping. In Intl.JointConf.on
AI.
Pearl, J. (1988). ProbabilisticReasoning
in Intelligent Systems: Networks of
Plausible Inference. Morgan Kauf-
mann.
Pearl, J. (2000). Causality: Models,Rea-
soning and Inference. CambridgeUniv. Press.Pearl, J. and T. Verma (1991). A theory
of inferred causation. In Knowledge
Representation, pp. 441–452.
Pe’er, D. (2005, April). Bayesian net-
work analysis of signaling net-works: a primer. ScienceSTKE 281,
14.

AI.
mann.
14.
Peng, F., R. Jacobs, and M. Tan-
ner (1996). Bayesian Inferencein Mixtures-of-Experts and Hier-archical Mixtures-of-Experts Mod-els With an Application to Speech
Recognition. J. of the Am. Stat. As-
soc. 91(435), 953–960.
Petris, G., S. Petrone, and P. Campag-
noli (2009). Dynamiclinearmodels
withR. Springer.
Pham, D.-T. and P. Garrat (1997). Blind
separation of mixture of inde-
pendent sources through a quasi-maximum likelihood approach.
IEEE Trans. on Signal Process-
ing 45(7), 1712–1725.

14.
Pietra, S. D., V. D. Pietra, and J. Laf-
ferty (1997). Inducing features ofrandom ﬁelds. IEEE Trans. on Pat-
tern Analysis and Machine Intelli-gence 19(4).
Plackett, R. (1975). The analysis of per-
mutations. Applied Stat. 24, 193–
202.
Platt, J. (1998). Using analytic QP
and sparseness to speed training ofsupport vector machines. In NIPS.
Platt, J. (2000). Probabilities for sv ma-
chines. In A. Smola, P. Bartlett,B. Schoelkopf, and D. Schuurmans
(Eds.),Advances in Large Margin
Classiﬁers. MIT Press.
Pla

202.
Pla
tt, J., N. Cristianini, and J. Shawe-
Taylor (2000). Large margin DAGs
for multiclass classiﬁcation. In
NIPS, Volume 12, pp. 547–553.
Plummer, M. (2003). JAGS: A Program
for Analysis of Bayesian Graphi-cal Models Using Gibbs Sampling.
InProc. 3rd Intl. Workshop on Dis-
tributedStatisticalComputing.
Polson, N. and S. Scott (2011). Data
augmentation for support vectormachines. Bayesian Analysis 6 (1),
1–124.
Pontil, M., S. Mukherjee, and F. Girosi
(1998). On the Noise Model of Sup-

Pla
1–124.
port Vector Machine Regression.Technical report, MIT AI Lab.
Poon, H. and P. Domingos (2011). Sum-
product networks: A new deep ar-chitecture. In UAI.

1036 BIBLIOGRAPHY
Pourahmadi, M. (2004). Simultaneous
Modelling of Covariance Matrices:
GLM, Bayesian and Nonparamet-ric Perspectives. Technical report,Northern Illinois University.
Prado, R. and M. West (2010). Time
Series: Modelling, Computation and
Inference. CRC Press.
Press, S. J. (2005). Applied mul-
tivariate analysis, using Bayesianand frequentist methods of infer-
ence. Dover. Second edition.
Press, W., W. Vetterling, S. Teukolosky,
and B. Flannery (1988). Numeri-
cal Recipes in C: The Art of Scien-
tiﬁcComputing (Second ed.). Cam-
Prince, S. (2012). Computer Vision:
Models, Learning and Inference.
Pritchard, J., M. M. Stephens, and
P. Donnelly (2000). Inference ofpopulation structure using multi-locus genotype data. Genetics 155,
945–959.
Qi, Y. and T. Jaakkola (2008). Param-
eter Expanded Variational BayesianMethods. In NIPS.
Qi, Y., M. Szummer, and T. Minka
(2005). Bayesian Conditional Ran-
dom Fields. In 10th Intl. Workshop
onAI/Statistics.

945–959.
Quinlan, J. (1990). Learning logical def-
initions from relations. Machine
Learning 5, 239–266.
Quinlan, J. R. (1986). Induction of de-
cision trees. Machine Learning 1,
81–106.
Quinlan, J. R. (1993). C4.5Programsfor
Machine Learning. Morgan Kauff-
man.
Quinonero-Candela, J., C. Rasmussen,
and C. Williams (2007). Approxi-mation methods for gaussian pro-cess regression. In L. Bottou,O. Chapelle, D. DeCoste, and J. We-
ston (Eds.),Large Scale Kernel Ma-
chines, pp. 203–223. MIT Press.
Rabiner, L. R. (1989). A tutorial on Hid-

81–106.
man.
den Markov Models and selected
applications in speech recognition.
Proc.oftheIEEE 77 (2), 257–286.
Rai, P. and H. Daume (2009). Multi-
label prediction via sparse inﬁnite
CCA. InNIPS.Raiffa, H. (1968). DecisionAnalysis.A d -
dison Wesley.
Raina, R., A. Madhavan, and A. Ng
(2009). Large-scale deep unsuper-vised learning using graphics pro-
cessors. In Intl. Conf. on Machine
Learning.
Raina, R., A. Ng, and D. Koller (2005).
Transfer learning by constructing
informative priors. In NIPS.
Rajaraman, A. and J. Ullman (2010).

Learning.
Mining of massive datasets. Self-published.
Rajaraman, A. and J. Ullman (2011).
Mining of massive datasets. Cam-bridge.
Rakotomamonjy, A., F. Bach, S. Canu,
and Y. Grandvalet (2008). Sim-
pleMKL.J.ofMachineLearningRe-
sear
ch 9, 2491–2521.
Ramage, D., D. Hall, R. Nallapati,
and C. Manning (2009). Labeled
LDA: A supervised topic model forcredit attribution in multi-labeledcorpora. In EMNLP.
Ramage, D., C. Manning, and S. Du-
mais (2011). Partially Labeled TopicModels for Interpretable Text Min-

sear
ing. InProc. of the Int’l Conf. on
ing.
Ramaswamy, S., P. Tamayo, R. Rifkin,
S. Mukherjee, C. Yeang, M. Angelo,C. Ladd, M. Reich, E. Latulippe,J. Mesirov, T. Poggio, W. Gerald,M. Loda, E. Lander, and T. Golub(2001). Multiclass cancer diagno-sis using tumor gene expression
signature. Proc. of the National
AcademyofScience,USA98, 15149–
15154.
Ranzato, M. and G. Hinton (2010).
Modeling pixel means and covari-ances using factored third-orderBoltzmann machines. In CVPR.

ing.
15154.
Ranzato, M., F.-J. Huang, Y.-L.
Boureau, and Y. LeCun (2007). Un-supervised Learning of InvariantFeature Hierarchies with Applica-tions to Object Recognition. InCVPR.
Ranzato, M., C. Poultney, S. Chopra,
and Y.LeCun (2006). Efficientlearning of sparse representationswith an energy-based model. InNIPS.Rao, A. and K. Rose (2001, February).
Deterministically Annealed Designof Hidden Markov Model Speech
Recognizers. IEEETrans.onSpeech
andAudioProc. 9 (2), 111–126.
Rasmussen, C. (2000). The inﬁnite

gaussian mixture model. In NIPS.
Rasmussen, C. E. and J. Quiñonero-
Candela (2005). Healing the rele-
vance vector machine by augmen-
tation. InIntl. Conf. on Machine
Learning, pp. 689–696.
Rasmussen, C. E. and C. K. I. Williams
(2006).Gaussian Processes for Ma-
chineLearning. MIT Press.
Ratsch, G., T. Onoda, and K. Muller
(2001). Soft margins for adaboost.MachineLearning 42, 287–320.
Rattray, M., O. Stegle, K. Sharp, and
J. Winn (2009). Inference algo-
rithms and learning theory forBayesian sparse factor analysis. In

Proc. Intl. Workshop on Statistical-
MechanicalInformatics.
Rauch, H. E., F. Tung, and C. T. Striebel
(1965). Maximum likelihood esti-
mates of linear dynamic systems.
AIAAJournal 3 (8), 1445–1450.
Ravikumar, P., J. Lafferty, H. Liu, and
L. Wasserman (2009). Sparse Ad-ditive Models. J. of Royal Stat. Soc.
SeriesB 71 (5), 1009–1030.
Raydan, M. (1997). The barzilai and
borwein gradient method for the
large scale unconstrained mini-mization problem. SIAMJ.onOpti-
mization 7 (1), 26–33.
Rennie, J. (2004). Why sums are bad.

Technical report, MIT.
Rennie,J.,
L. Shih, J. Teevan, and
D. Karger (2003). Tackling the poorassumptions of naive Bayes text
classiﬁers. In Intl.Conf.onMachine
Learning.
Reshed, D., Y. Reshef, H. Finucane,
S. Grossman, G. McVean, P. Turn-
baugh, E. Lander, M. Mitzen-macher, and P. Sabeti (2011, De-cember). Detecting novel associa-tions in large data sets. Science334,
1518–1524.
Resnick, S. I. (1992). Adventures in
StochasticProcesses. Birkhauser.
Rice, J. (1995). Mathematical statistics
and data analysis. Duxbury. 2ndedition.

BIBLIOGRAPHY 1037
Richardson, S. and P. Green (1997). On
Bayesian Analysis of Mixtures With
an Unknown Number of Compo-
nents.J. of Royal Stat. Soc. Series
B5 9, 731–758.
Riesenhuber, M. and T. Poggio
(1999). Hierarchical models of ob-
ject recognition in cortex. Nature
Neuroscience 2, 1019–1025.
Rish, I., G. Grabarnik, G. Cec-
chi, F. Pereira, and G. Gordon(2008). Closed-form supervised di-mensionality reduction with gener-
alized linear models. In Intl. Conf.
Ristic, B., S. Arulampalam, and N. Gor-
don (2004). BeyondtheKalmanFil-
ter: Particle Filters for Tracking Ap-
plications. Artech House Radar Li-
brary.
Robert, C. (1995). Simulation of trun-
cated normal distributions. Statis-
ticsandcomputing 5, 121–125.
Robert, C. and G. Casella (2004).
Monte Carlo Statisical Methods.Springer. 2nd edition.
Roberts, G. and J. Rosenthal (2001).
Optimal scaling for variousMetropolis-Hastings algorithms.
StatisticalScience 16, 351–367.
Roberts, G. O. and S. K. Sahu
(1997). Updating schemes, corre-

brary.
lation structure, blocking and pa-rameterization for the gibbs sam-
pler.J. of Royal Stat. Soc. Series
B5 9(2), 291–317.
Robinson, R. W. (1973). Counting la-
beled acyclic digraphs. In F. Harary(Ed.),New Directions in the Theory
of Graphs, pp. 239–273. Academic
Press.
Roch, S. (2006). A short proof
that phylogenetic tree reconstru-tion by maximum likelihood ishard.IEEE/ACM Trans. Comp. Bio.
Bioinformatics 31 (1).
Rodriguez, A. and K. Ghosh
(2011). Modeling relationaldata through nested partition

Press.
models.Biometrika . To appear.
Rose, K. (1998, November). Determin-
istic annealing for clustering, com-
pression, classiﬁcation, regression,and related optimization problems.Proc.IEEE 80, 2210–2239.Rosenblatt, F. (1958). The percep-
tron: A probabilistic model forinformation storage and organiza-
tion in the brain. PsychologicalRe-
view 65(6), 386â˘A¸S408.
Ross, S. (1989). Introduction to Proba-
bilityModels. Academic Press.
Rosset, S., J. Zhu, and T. Hastie (2004).

Boosting as a regularized path toa maximum margin classiﬁer. J. of
Machine Learning Research 5, 941–
973.
Rossi, P., G. Allenby, and R. McCulloch
(2006).BayesianStatisticsandMar-
keting. Wiley.
Roth, D. (1996, Apr). On the hardness
of approximate reasoning. Artiﬁcial
Intelligence 82 (1-2),273–302.
Ro
ther, C., P. Kohli, W. Feng, and J. Jia
(2009). Minimizing sparse higherorder energy functions of discretevariables. In CVPR, pp. 1382–1389.
Rouder, J., P. Speckman, D. Sun, and
R. Morey (2009). Bayesian t testsfor accepting and rejecting the null

973.
Ro
hypothesis. PyschonomicBulletin&
Review 16 (2), 225–237.
Roverato, A. (2002). Hyper in-
verse Wishart distribution for non-
decomposable graphs and its ap-plication to Bayesian inference for
Gaussian graphical models. Scand.
J.Statistics 29, 391–411.
Roweis, S. (1997). EM algorithms for
PCA and SPCA. In NIPS.
Rubin, D. (1998). Using the SIR algo-
rithm to simulate posterior distri-
butions. InBayesianStatistics3.
Rue, H. and L. Held (2005). Gaus-
sian Markov Random Fields: The-ory and Applications, Volume 104

ofMonographsonStatisticsandAp-
plied Probability. London: Chap-
man & Hall.
Rue, H., S. Martino, and N. Chopin
(2009). Approximate Bayesian In-ference for Latent Gaussian ModelsUsing Integrated Nested Laplace
Approximations. J.ofRoyalStat.Soc.
SeriesB 71, 319–392.
Rumelhart, D., G. Hinton, and
R. Williams (1986). Learning inter-
nal representations by error propa-gation. In D. Rumelhart, J. McClel-land, and the PDD Research Group
(Eds.),Parallel Distributed Process-
ing: Explorations in the Microstruc-

man & Hall.
tureofCognition. MIT Press.Ruppert, D., M. Wand, and R. Carroll
(2003).Semiparametric Regression.
Rush, A. M. and M. Collins (2012). A
tutorial on Lagrangian relaxation
and dual decomposition for NLP.Technical report, Columbia U.
Russell, S., J. Binder, D. Koller, and
K. Kanazawa (1995). Local learningin probabilistic networks with hid-
den variables. In Intl.JointConf.on
AI.
Russell, S. and P. Norvig (1995). Ar-
tiﬁcial Intelligence: A Modern Ap-
proach. Englewood Cliffs, NJ: Pren-

AI.
tice Hall.
Russell, S. and P. Norvig (2002). Ar-
proach. Prentice Hall. 2nd edition.
Russell, S. and P. Norvig (2010). Ar-
tiﬁcial Intelligence: A Modern Ap-proach. Prentice Hall. 3rd edition.
S. and M. Black (2009, April). Fields
of experts. Intl. J. Computer Vi-
sion 82(2), 205–229.
Sachs, K., O. Perez, D. Pe’er,
D. Lauffenburger, and G. Nolan
(2005). Causal protein-signalingnetworks derived from multipa-rameter single-cell data. Sci-
ence 308.
Sahami,M.
and T. Heilman (2006).

ence 308.
Sahami,M.
A Web-based Kernel Function forMeasuring the Similarity of Short
Text Snippets. In WWWconferenec.
Salakhutdinov, R. (2009). Deep Gen-
erative Models. Ph.D. thesis, U.
Toronto.
Salakhutdinov, R. and G. Hinton
(2009). Deep Boltzmann machines.InAI/Statistics, Volume 5, pp. 448–
455.
(2010). Replicated Softmax: anUndirected Topic Model. In NIPS.
Salakhutdinov, R. and H. Larochelle
(2010). Efficient Learning of DeepBoltzmann Machines. In AI/Statis-
tics.

Toronto.
455.
tics.
Salakhutdinov, R. and A. Mnih (2008).
Probabilistic matrix factorization.InNIPS, Volume 20.

1038 BIBLIOGRAPHY
Salakhutdinov, R. and S. Roweis
(2003). Adaptive overrelaxed bound
optimization methods. In Proceed-
ingsoftheInternationalConference
on Machine Learning, Volume 20,
pp. 664–671.
Salakhutdinov, R., J. Tenenbaum, and
A. Torralba (2011). Learning ToLearn with Compound HD Models.InNIPS.
Salakhutdinov, R. R., A. Mnih, and
G. E. Hinton (2007). Restrictedboltzmann machines for collabo-rative ﬁltering. In Intl. Conf. on
Machine Learning, Volume 24, pp.791–798.
Salojarvi, J., K. Puolamaki, and
S. Klaski (2005). On discriminativejoint density modeling. In Proc.Eu-
ropeanConf.onMachineLearning.
Sampson, F. (1968). A Novitiate in a
Period of Change: An Experimental
and Case Study of Social Relation-ships. Ph.D. thesis, Cornell.
Santner, T., B. Williams, and W. Notz
(2003).The Design and Analysis of
ComputerExperiments. Springer.
Sarkar, J. (1991). One-armed bandit
problems with covariates. The An-
nalsofStatistics 19 (4), 1978–2002.
Sato, M. and S. Ishii (2000). On-line
EM algorithm for the normalized

Gaussian network. Neural Compu-
tation 12, 407–432.
Saul, L., T. Jaakkola, and M. Jordan
(1996). Mean Field Theory for Sig-moid Belief Networks. J. of AI Re-
search 4, 61–76.
Saul, L. and M. Jordan (1995). Exploit-
ing tractable substructures in in-
tractable networks. In NIPS, Vol-
ume 8.
Saul, L. and M. Jordan (2000). Attrac-
tor dynamics in feedforward neural
networks.Neural Computation 12,
1313–1335.
Saunders, C., J. Shawe-Taylor, and
A. Vinokourov (2003). String Ker-
nels, FisherKernelsandFiniteStateAutomata. In NIPS.
Savage, R., K. Heller, Y. Xi, Z. Ghahra-

ume 8.
mani, W. Truman, M. Grant,K. Denby, and D. Wild (2009).R/BHC: fast Bayesian hierarchi-cal clustering for microarray data.
BMCBioinformatics 10 (242).Schaefer, J. and K. Strimmer (2005).
A shrinkage approach to large-
scale covariance matrix estimationand implications for functional ge-
nomics.Statist. Appl. Genet. Mol.
Biol 4(32).
Schapire, R. (1990). The strength of
weak learnability. Machine Learn-
ing 5, 197–227.
Schapire, R. and Y. Freund (2012).
Boosting: Foundations and Algo-
rithms. MIT Press.

Schapire, R., Y. Freund, P. Bartlett, and
W. Lee (1998). Boosting the mar-
gin: a new explanation for the ef-fectiveness of voting methods. An-
nalsofStatistics 5, 1651–1686.
Schar
stein, D. and R. Szeliski (2002). A
taxonomy and evaluation of densetwo-frame stereo correspondence
algorithms. Intl. J. Computer Vi-
sion 47(1), 7–42.
Schaul, T., S. Zhang, and Y. LeCun
(2012). No more pesky learning
rates. Technical report, Courant In-stite of Mathematical Sciences.
Schmee, J. and G. Hahn (1979). A sim-

Schar
ple method for regresssion analy-
sis with censored data. Technomet-
rics 21, 417–432.
Schmidt, M. (2010). Graphical model
structure learning with L1 regular-
ization. Ph.D. thesis, UBC.
Schmidt, M., G. Fung, and R. Rosales
(2009). Optimization methods forℓ−1regularization. Technical re-
port, U. British Columbia.
Schmidt, M. and K. Murphy (2009).
Modeling Discrete InterventionalData using Directed Cyclic Graphi-cal Models. In UAI.
Schmidt, M., K. Murphy, G. Fung, and

R. Rosales (2008). Structure Learn-ing in Random Fields for HeartMotion Abnormality Detection. InCVPR.
Schmidt, M., A. Niculescu-Mizil, and
K. Murphy (2007). Learning Graph-ical Model Structure using L1-Regularization Paths. In AAAI.
Schmidt, M., E. van den Berg,
M. Friedlander, and K. Murphy(2009). Optimizing Costly Func-tions with Simple Constraints: ALimited-Memory Projected Quasi-Newton Algorithm. In AI & Statis-
tics.Schniter, P., L. C. Potter, and J. Ziniel

(2008). Fast Bayesian MatchingPursuit: Model Uncertainty andParameter Estimation for SparseLinear Models. Technical report, U.Ohio. Submitted to IEEE Trans. onSignal Processing.
Schnitzspan, P., S. Roth, and B. Schiele
(2010). Automatic discovery ofmeaningful object parts with latentCRFs. InCVPR.
Schoelkopf, B. and A. Smola (2002).
LearningwithKernels: SupportVec-
tor Machines, Regularization, Opti-mization,andBeyond. MIT Press.
Schoelkopf, B., A. Smola, and K.-R.
Mueller (1998). Nonlinear compo-

nent analysis as a kernel eigen-
value problem. Neural Computa-
tion 10, 1299 – 1319.
Schraudolph, N. N., J. Yu, and S. Gün-
ter (2007). A Stochastic Quasi-
Newton Method for Online ConvexOptimization. In AI/Statistics, pp.
436–443.
Schwarz, G. (1978). Estimating the di-
mension of a model. Annals of
Statistics 6 (2), 461â˘A¸S464.
Schwarz, R. and Y. Chow (1990). The
n-best algorithm: an efficient andexact procedure for ﬁnding the n
most likely hypotheses. In Intl.
nalP
roc.

436–443.
nalP
roc.
Schweikerta, G., A. Zien, G. Zeller,
J. Behr, C. Dieterich, C. Ong,
P. Philips, F. D. Bona, L. Hartmann,
A. Bohlen, N. KrÃijger, S. Son-
nenburg, and G. RÃd’tsch (2009).
mGene: Accurate SVM-based GeneFinding with an Application to Ne-matode Genomes. Genome Re-
search, 19, 2133–2143.
Scott, D. (1979). On optimal
and data-based histograms.
Biometrika 66 (3), 605–610.
Scott, J. G. and C. M. Carvalho (2008).
Feature-inclusion stochastic search
for gaussian graphical models. J.
of Computational and GraphicalStatistics 17 (4), 790–808.

Scott, S. (2009). Data augmenta-
tion, frequentist estimation, andthe bayesian analysis of multino-
mial logit models. StatisticalPapers .
Scott, S. (2010). A modern Bayesian
look at the multi-armed bandit.
Applied Stochastic Models in Busi-
nessandIndustry 26, 639–658.

BIBLIOGRAPHY 1039
Sedgewick, R. and K. Wayne (2011). Al-
gorithms. Addison Wesley.
Seeger, M. (2008). Bayesian Inference
and Optimal Design in the Sparse
Linear Model. J.ofMachineLearn-
ingResearch 9, 759–813.
Seeger, M. and H. Nickish (2008).
Compressed sensing and bayesianexperimental design. In Intl. Conf.
Segal, D. (2011, 12 February). The dirty
little secrets of search. New York
Times.
Seide, F., G. Li, and D. Yu (2011). Con-
versational Speech Transcription
Using Context-Dependent Deep
Neural Networks. In Interspeech.
Sejnowski, T. and C. Rosenberg (1987).
Parallel networks that learn to pro-nounce english text. Complex Sys-
tems 1, 145–168.
Sellke, T., M. J. Bayarri, and J. Berger
(2001). Calibration of p Values
for Testing Precise Null Hypothe-
ses.TheAmericanStatistician 55 (1),
62–71.
Serre, T., L. Wolf, and T. Poggio (2005).
Object recognition with features
inspired by visual cortex. In CVPR,
pp. 994–1000.
Shachter, R. (1998). Bayes-ball: The

62–71.
rational pastime (for determiningirrelevance and requisite informa-tion in belief networks and inﬂu-ence diagrams). In UAI.
Shachter, R. and C. R. Kenley
(1989). Gaussian inﬂuence dia-grams.Managment Science 35 (5),
527–550.
Shachter, R. D. and M. A. Peot (1989).
Simulation approaches to generalprobabilistic inference on beliefnetworks. In UAI, Volume 5.
Shafer, G. R. and P. P. Shenoy (1990).
Probability propagation. Annals of
MathematicsandAI 2, 327–352.
Shafto, P., C. Kemp, V. Mansinghka,

527–550.
M. Gordon, and J. B. Tenenbaum(2006). Learning cross-cutting sys-tems of categories. In CognitiveSci-
enceConference.
Shahaf, D., A. Chechetka, and
C. Guestrin (2009). Learning ThinJunction Trees via Graph Cuts. InAISTATS.Shalev-Shwartz, S., Y. Singer, and
N. Srebro (2007). Pegasos: pri-mal estimated sub-gradient solver
for svm. In Intl. Conf. on Machine
Learning.
Shalizi, C. (2009). Cs 36-350 lecture
10: Principal components: mathe-
matics, example, interpretation.
Shan, H. and A. Banerjee (2010). Resid-

Learning.
ual Bayesian co-clustering for ma-
trix approximation. In SIAM Intl.
Conf.onDataMining.
Shawe-Taylor, J. and N. Cristianini
(2004).Kernel Methods for Pattern
Analysis. Cambridge.
Sheng, Q., Y. Moreau, and B. D. Moor
(2003). BiclusteringMicroarraydata
by Gibbs sampling. Bioinf
 ormat-
ics 19, ii196–ii205.
Shi, J. and J. Malik (2000). Normal-
ized cuts and image segmentation.
MachineIntelligence .
Shoham, Y. and K. Leyton-Brown
(2009).Multiagent Systems: Algo-

 ormat-
rithmic, Game- Theoretic, and Log-ical Foundations. Cambridge Uni-
Shotton, J., A. Fitzgibbon, M. Cook,
T. Sharp, M. Finocchio, R. Moore,A. Kipman, and A. Blake (2011).Real-time human pose recognitionin parts from a single depth image.InCVPR.
Shwe, M., B. Middleton, D. Heck-
erman, M. Henrion, E. Horvitz,H. Lehmann, and G. Cooper (1991).Probabilistic diagnosis using a re-formulation of the internist-1/qmr
knowledge base. Methods. Inf.
Med 30(4), 241–255.
Siddiqi, S., B. Boots, and G. Gordon

(2007). A constraint generation ap-
proach to learning stable linear dy-namical systems. In NIPS.
Siepel, A. and D. Haussler (2003).
Combining phylogenetic and hid-den markov models in biosequence
analysis. In Proc. 7th Intl. Conf.
on Computational Molecular Biol-
ogy(RECOMB).
Silander, T., P. Kontkanen, and P. Myl-
lymÃd’ki (2007). On Sensitivity ofthe MAP Bayesian Network Struc-ture to the Equivalent Sample SizeParameter. In UAI, pp. 360–367.Silander, T. and P. Myllmaki (2006).

A simple approach for ﬁnding theglobally optimal Bayesian networkstructure. In UAI.
Sill, J., G. Takacs, L. Mackey, and
D. Lin (2009). Feature-weighted lin-ear stacking. Technical report, .
Silverman, B. W. (1984). Spline
smoothing: the equivalent variable
kernel method. Annals of Statis-
tics 12(3), 898–916.
Simard, P., D. Steinkraus, and J. Platt
(2003). Best practices for convolu-
tional neural networks applied to
visual document analysis. In Intl.
Conf. on Document Analysis and
Recognition(ICDAR).

Simon, D. (2006). Optimal State Es-
timation: Kalman, H Inﬁnity, and
NonlinearApproaches. Wiley.
Singliar, T. and M. Hauskrecht (2006).
Noisy-OR Component Analysis and
its Application to Link Analysis. J.
ofMachineLearningResearch 7.
Smidl, V. and A. Quinn (2005). The
Variational Bayes Method in SignalProcessing. Springer.
Smith, A. F. M. and A. E. Gelfand
(1992). Bayesian statistics with-out tears: A sampling-resampling
perspective. The American Statisti-
cian 46(2), 84–88.
Smith, R. and P. Cheeseman (1986).

On the representation and estima-tion of spatial uncertainty. Intl. J.
Robo
ticsResearch 5 (4), 56–68.
Smith, V., J. Yu, T. Smulders,
A. Hartemink, and E. Jarvis (2006).
Computational Inference of NeuralInformation Flow Networks. PLOS
Computational Biology 2, 1436–1439.
Smolensky, P. (1986). Information
processing in dynamical systems:foundations of harmony theory.In D. Rumehart and J. McClel-land (Eds.),ParallelDistributedPro-
cessing: Explorations in the Mi-
crostructure of Cognition. Volume 1.

Robo
Smyth, P., D. Heckerman, and M. I. Jor-
dan (1997). Probabilistic indepen-dence networks for hidden Markovprobability models. NeuralCompu-
tation 9(2), 227–269.
Sohl-Dickstein, J., P. Battaglino, and
M. DeWeese (2011). In Intl. Conf.

1040 BIBLIOGRAPHY
Sollich, P. (2002). Bayesian methods
for support vector machines: evi-
dence and predictive class proba-
bilities.Machine Learning 46, 21–
52.
Sontag, D., A. Globerson, and
T. Jaakkola (2011). Introduction to
dual decomposition for inference.In S. Sra, S. Nowozin, and S. J.Wright (Eds.), OptimizationforMa-
Sorenson, H. and D. Alspach (1971).
Recursive Bayesian estimation us-ing Gaussian sums. Automatica 7,
465â˘A¸S 479.
Soussen, C., J. Iier, D. Brie, and
J. Duan (2010). From Bernoulli-Gaussian deconvolution to sparsesignal restoration. Technical report,Centre de Recherche en Automa-tique de Nancy.
Spaan, M. and N. Vlassis (2005).
Perseus: Randomized Point-basedValue Iteration for POMDPs. J.ofAI
Research 24, 195–220.
Spall, J. (2003). IntroductiontoStochas-
tic Search and Optimization: Es-
timation, Simulation, and Control.
Wiley.
Speed, T. (2011, December). A cor-
relation for the 21st century. Sci-
ence 334, 152–1503.
Speed, T. and H. Kiiveri (1986). Gaus-

Wiley.
sian Markov distributions over ﬁ-nite graphs. Annals of Statis-
tics 14(1), 138–150.
Spiegelhalter, D. J. and S. L. Lauritzen
(1990). Sequential updating of con-ditional probabilities on directed
graphical structures. Networks 20.
Spirtes, P., C. Glymour, and
R. Scheines (2000). Causa-
tion, Prediction, and Search. MIT
Press. 2nd edition.
Srebro, N.(2001). MaximumLikelihood
Bounded Tree-Width Markov Net-works. InUAI.
Srebro, N. and T. Jaakkola (2003).
Weighted low-rank approxima-
tions. InIntl. Conf. on Machine
Learning.
Steinbach, M., G. Karypis, and V. Ku-
mar (2000). A comparison of doc-
ument clustering techniques. In

Learning.
KDDWorkshoponTextMining.Stephens, M. (2000). Dealing with
label-switching in mixture mod-els.J.RoyalStatisticalSociety,Series
B6 2, 795–809.
Stern, D., R. Herbrich, and T. Grae-
pel (2009). Matchbox: Large Scale
Bayesian Recommendations. In
Proc. 18th. Intl. World Wide Web
Conference.
Steyvers, M. and T. Griffiths (2007).
Probabilistic topic models. InT. Landauer, D. McNamara, S. Den-nis, and W. Kintsch (Eds.), Latent
SemanticAnalysis: ARoadtoMean-ing. Laurence Erlbaum.
Stigler, S. (1986). The history of statis-

tics. Harvard University press.
Stolc
ke, A. and S. M. Omohundro
(1992). Hidden Markov Model In-duction by Bayesian Model Merg-ing. InNIPS-5.
Stoyanov, V., A. Ropson, and J. Eis-
ner (2011). Empirical risk minimiza-tion of graphical model parametersgiven approximate inference, de-coding, and model structure. InAI/Statistics.
Sudderth, E. (2006). Graphical Models
for Visual Object Recognition and
Tracking. Ph.D. thesis, MIT.
Sudderth, E. and W. Freeman (2008,
March). Signal and Image Process-
ing with Belief Propagation. IEEE
SignalProcessingMagazine .

Sudderth, E., A. Ihler, W. Freeman, and
A. Willsky (2003). NonparametricBelief Propagation. In CVPR.
Sudderth, E., A. Ihler, M. Isard,
W. Freeman, and A. Willsky (2010).Nonparametric Belief Propagation.
Comm.oftheACM 53 (10).
Sudderth, E. and M. Jordan (2008).
Shared Segmentation of Natural
Scenes Using Dependent Pitman-
Yor Processes. In NIPS.
Sudderth, E., M. Wainwright, and
A. Willsky (2008). Loop series andbethe variational bounds for attrac-tive graphical models. In NIPS.
Sun, J., N. Zheng, and H. Shum

(2003). Stereo matching using be-lief propagation. IEEETrans.onPat-
tern Analysis and Machine Intelli-gence 25(7), 787–800.Sun, L., S. Ji, S. Yu, and J. Ye (2009). On
the equivalence between canonicalcorrelation analysis and orthonor-malized partial least squares. In
Sunehag, P., J. Trumpf, S. V. N. Vish-
wanathan, and N. N. Schraudolph
(2009). Variable Metric StochasticApproximation Theory. In AI/Statis-
tics, pp. 560–566.
Sutton, C. and A. McCallum (2007).
Improved Dynamic Schedules forBelief Propagation. In UAI.

Sutton, R. and A. Barto (1998). Rein-
forcmentLearning: AnIntroduction.MIT Press.
Swendsen, R. and J.-S. Wang (1987).
Nonuniversal critical dynamics in
Monte Carlo simulations. Physical
ReviewLetters 58 , 86–88.
Swersky, K., B. Chen, B. Marlin, and
N. de Freitas (2010). A Tuto-
rial on Stochastic ApproximationAlgorithms for Training RestrictedBoltzmann Machines and Deep Be-
lief Nets. In InformationTheoryand
Applications(ITA)Workshop.
Szeliski, R. (2010). Computer Vi-
sion: Algorithms and Applications.
Springer.

Springer.
Szeliski, R., R. Zabih, D. Scharstein,
O. Veksler, V. Kolmogorov, A. Agar-wala, M. Tappen, and C. Rother(2008). A Comparative Studyof Energy Minimization Methodsfor Markov Random Fields withSmoothness-Based
Priors.IEEE
chineIntelligence 30 (6), 1068–1080.
Szepesvari, C. (2010). Algorithms for
Reinforcement Learning . Morgan
Claypool.
Taleb, N. (2007). The Black Swan:
The Impact of the Highly Improba-ble. Random House.
Talhouk, A., K. Murphy, and A. Doucet
(2011). Efficient Bayesian Inference

Claypool.
for Multivariate Probit Models withSparse Inverse Correlation Matri-
ces.J.Comp.Graph.Statist..
Tanner, M. (1996). Tools for statistical
inference. Springer.
Tanner, M. and W. Wong (1987). The
calculation of posterior distribu-tions by data augmentation. J. of
the Am. Stat. Assoc. 82 (398), 528–
540.

BIBLIOGRAPHY 1041
Tarlow, D., I. Givoni, and R. Zemel
(2010). Hop-map: efficient message
passing with high order potentials.InAI/Statistics.
Taskar, B., C. Guestrin, and D. Koller
(2003). Max-margin markov net-works. InNIPS.
Taskar, B., D. Klein, M. Collins,
D. Koller, and C. Manning (2004).Max-margin parsing. In Proc. Em-
pirical Methods in Natural Lan-guageProcessing.
Teh, Y. W. (2006). A hierarchical
Bayesian language model based on
Pitman-Yor processes. In Proc. of
the Assoc. for Computational Lin-
guistics, pp. 985=992.
Teh, Y.-W., M. Jordan, M. Beal, and
D. Blei (2006). Hierarchical Dirich-let processes. J.oftheAm.Stat.As-
soc. 101(476), 1566–1581.
Tenenbaum, J. (1999). A Bayesian
framework for concept learning.Ph.D. thesis, MIT.
Tenenbaum, J. B. and F. Xu (2000).
Word learning as bayesian infer-
ence. InProc.22ndAnnualConf.of
theCognitiveScienceSociety.
Theocharous, G., K. Murphy, and
L. Kaelbling (2004). Representing

hierarchical POMDPs as DBNs formulti-scale robot localization. InIEEEIntl.Conf.onRoboticsandAu-tomation.
Thiesson, B., C. Meek, D. Chickering,
and D. Heckerman (1998). Learningmixtures of DAG models. In UAI.
Thomas, A. and P. Green (2009).
Enumerating the decomposableneighbours of a decomposablegraph under a simple perturbation
scheme.Comp. Statistics and Data
Analysis 53, 1232–1238.
Thrun, S., W. Burgard, and D. Fox
(2006).Probabilistic Robotics. MIT
Press.
Thrun, S., M. Montemerlo, D. Koller,

Press.
B. Wegbreit, J. Nieto, and E. Nebot
(2004). Fastslam: An efficient so-lution to the simultaneous local-ization and mapping problem with
unknown data association. J. of
MachineLearningResearch 2004.
Thrun, S. and L. Pratt (Eds.) (1997).
Learningtolearn. Kluwer.Tibshirani, R. (1996). Regression
shrinkage and selection via thelasso.J. Royal. Statist. Soc B 58 (1),
267–288.
Tibshirani, R., G. Walther, and
T. Hastie (2001). Estimating the
number of clusters in a dataset viathe gap statistic. J.ofRoyalStat.Soc.

267–288.
SeriesB 32 (2), 411–423.
Tieleman, T. (2008). Training re-
stricted Boltzmann machines us-ing approximations to the likeli-
hood gradient. In Proceedings of
the25thinternationalconferenceon
Machine learning, pp. 1064–1071.
ACM New York, NY, USA.
Ting, J., A. D’Souza, S. Vijayakumar,
and S. Schaal (2010). Efficientlearning and feature selection inhigh-dimensional regression. Neu-
ral
Computation 22 (4), 831–886.
Tipping, M. (1998). Probabilistic visual-
ization of high-dimensional binarydata. InNIPS.

ral
Tipping, M. (2001). Sparse bayesian
learning and the relevance vector
Research 1, 211–244.
Tipping, M. and C. Bishop (1999).
Probabilistic principal componentanalysis.J. of Royal Stat. Soc. Series
B2 1(3), 611–622.
Tipping, M. and A. Faul (2003). Fast
marginal likelihood maximisation
for sparse bayesian models. In AI/S-
tats.
Tishby, N., F. Pereira, and W. Biale
(1999). The information bottle-neck method. In The 37th an-
nualAllertonConf.onCommunica-

tats.
tion, Control, and Computing, pp.
368â˘A¸S377.
Tomas, M., D. Anoop, K. Stefan,
B. Lukas, and C. Jan (2011). Empir-
ical evaluation and combination ofadvanced language modeling tech-
niques. InProc. 12th Annual Conf.
of the Intl. Speech Communication
Association(INTERSPEECH).
Torralba, A., R. Fergus, and Y. Weiss
(2008). Small codes and large im-age databases for recognition. InCVPR.
Train, K. (2009). Discrete choice meth-
ods with simulation. CambridgeUniversity Press. Second edition.Tseng, P. (2008). On Accelerated Proxi-

mal Gradient Methods for Convex-Concave Optimization. Technicalreport, U. Washington.
Tsochantaridis, I., T. Joachims, T. Hof-
mann, and Y. Altun (2005, Septem-ber). Large margin methodsfor structured and interdependent
output variables. J. of Machine
LearningResearch 6, 1453–1484.
Tu, Z. and S. Zhu (2002). Image Seg-
mentation by Data-Driven Markov
Chain Monte Carlo. IEEETrans.on
ligence 24 (5), 657–673.
Turian, J., L. Ratinov, and Y. Ben-
gio (2010). Word representations:

a simple and general method forsemi-supervised learning. In Proc.
ACL.
Turlach, B., W. Venables, and S. Wright
(2005). Simultaneous variable se-lection.Technometrics 47 (3), 349–
363.
Turner, R., P. Berkes, M. Sahani, and
D. Mackay (2008). Counterexam-ples to variational free energy com-pactness folk theorems. Technicalreport, U. Cambridge.
Ueda, N. and R. Nakano (1998). Deter-
ministic annealing EM algorithm.
NeuralNetworks 11, 271–282.
Usunier, N., D. Buffoni, and P. Galli-

ACL.
363.
nari (2009). Ranking with ordered
weighted pairwise classiﬁcation.
Vaithyanathan, S. and B. Dom (1999).
Model selection in unsupervisedlearning with applications to doc-
ument clustering. In Intl. Conf. on
van
der Merwe, R., A. Doucet,
N. de Freitas, and E. Wan (2000).
The unscented particle ﬁlter. In
NIPS-13.
van Dyk, D. and X.-L. Meng (2001).
The Art of Data Augmentation.
J. Computational and Graphical
Statistics 10 (1), 1–50.
Vandenberghe, L. (2006). Applied nu-
merical computing: Lecture notes.

van
NIPS-13.
Vandenberghe, L. (2011). Ee236c - op-
timization methods for large-scalesystems.
Vanhatalo, J. (2010). Speeding up the
inference in Gaussian process mod-
els. Ph.D. thesis, Helsinki Univ.
Technology.

1042 BIBLIOGRAPHY
Vanhatalo, J., V. PietilÃd’inen, and
A. Vehtari (2010). Approximate in-
ference for disease mapping withsparse gaussian processes. Statis-
ticsinMedicine 29 (15), 1580–1607.
Vapnik, V. (1998). Statistical Learning
Theory. Wiley.
Vapnik, V., S. Golowich, and A. Smola
(1997). Support vector method forfunction approximation, regressionestimation, and signal processing.InNIPS.
Varian, H. (2011). Structural time series
in R: a Tutorial. Technical report,Google.
Verma, T. and J. Pearl (1990). Equiva-
lence and synthesis of causal mod-els. InUAI.
Viinikanoja, J., A. Klami, and S. Kaski
(2010). VariationalBayesianMixtureof Robust CCA Models. In Proc.Eu-
Vincent, P. (2011). A Connection be-
tween Score Matching and Denois-
ing Autoencoders. Neural Compu-
tation 23(7), 1661–1674.
Vincent, P., H. Larochelle, I. La-
joie, Y. Bengio, and P.-A. Manzagol
(2010). Stacked Denoising Autoen-coders: Learning Useful Represen-tations in a Deep Network with a

Local Denoising Criterion. J.ofMa-
chine Learning Research 11, 3371–
3408.
Vinh, N., J. Epps, and J. Bailey (2009).
Information theoretic measures forclusterings comparison: Is a cor-rection for chance necessary? In
Vinyals, M., J. Cerquides, J. Rodriguez-
Aguilar, and A. Farinelli (2010).
Worst-case bounds on the quality
of max-product ﬁxed-points. InNIPS.
Viola, P. and M. Jones (2001). Rapid
object detection using a boostedcascade of simple classiﬁers. InCVPR.
Virtanen, S. (2010). Bayesian expo-

3408.
nential family projections. Master’sthesis, Aalto University.
Vishwanathan, S. V. N. and A. Smola
(2003). Fast kernels for string andtree matching. In NIPS.Viterbi, A. (1967). Error bounds
for convolutional codes and anasymptotically optimum decoding
algorithm.IEEE Trans. on Informa-
tionTheory 13 (2), 260â˘A¸S269.
von Luxburg, U. (2007). A tutorial on
spectral clustering. Statistics and
Computing 17 (4), 395–416.
Wagenmakers, E.-J., R. Wetzels,
D. Borsboom, and H. van der

Maas (2011). Why PsychologistsMust Change the Way They Ana-lyze Their Data: The Case of Psi.
Journal of Personality and Social
Psychology .
Wagner, D. and F. Wagner (1993). Be-
tween min cut and graph bisec-
tion. InProc. 18th Intl. Symp. on
Math.Found.ofComp.Sci., pp.744–
750.
Wain
wright, M., T. Jaakkola, and
A. Willsky (2001). Tree-based repa-rameterization for approximate es-
timation on loopy graphs. In NIPS-
14.
Wainwright, M., T. Jaakkola, and
A. Willsky (2005). A new class
of upper bounds on the log parti-

750.
Wain
14.
tion function. IEEETrans.Info.The-
ory 51(7), 2313–2335.
Wainwright, M., P. Ravikumar, and
J. Lafferty (2006). Inferring graph-
ical model structure using ℓ−1-
regularized pseudo-likelihood. InNIPS.
Wainwright, M. J., T. S. Jaakkola, and
A. S. Willsky (2003). Tree-basedreparameterization framework foranalysis of sum-product and re-lated algorithms. IEEETrans.onIn-
formationTheory 49 (5), 1120–1146.
Wainwright, M. J. and M. I. Jordan
(2008a). Graphical models, expo-nential families, and variational in-ference.FoundationsandTrendsin
MachineLearning 1–2, 1–305.

(2008b). Graphical models, expo-nential families, and variational in-ference.FoundationsandTrendsin
Wallach, H., I. Murray, R. Salakhutdi-
nov, and D. Mimno (2009). Evalua-tion methods for topic models. In
Wan, E. A. and R. V. der Merwe (2001).
The Unscented Kalman Filter. In
S. Haykin (Ed.), Kalman Filtering
andNeuralNetworks. Wiley.Wand, M. (2009). Semiparametric

regression and graphical models.Aus t .N.Z.J.S t a t .51 (1), 9–41.
Wand, M. P., J. T. Ormerod, S. A.
Padoan, and R. Fruhrwirth (2011).Mean Field Variational Bayes forElaborate Distributions. Bayesian
Analysis 6 (4), 847 – 900.
Wang, C. (2007). Variational Bayesian
Approach to Canonical Correlation
Analysis.IEEETrans.onNeuralNet-
works 18(3), 905–910.
Wasserman, L. (2004). All of statistics.
A concise course in statistical infer-
ence. Springer.
Wei, G. and M. Tanner (1990). A Monte

CarloimplementationoftheEMal-gorithm and the poor man’s data
augmentation algorithms. J. of the
Am.Stat.Assoc. 85 (411), 699–704.
Weinberger, K., A. Dasgupta, J. Atten-
berg, J. Langford, and A. Smola
(2009). Feature hashing for large
scale multitask learning. In Intl.
Weiss, D., B. Sapp, and B. Taskar
(2010). Sidestepping intractable in-
ference with structured ensemblecascades. In NIPS.
Weiss, Y. (2000). Correctness of local
probability propagation in graph-
ical models with loops. Neural

Computation 12 , 1–41.
Weiss,
Y. (2001). Comparing the mean
ﬁeld method and belief propaga-
tion for approximate inference in
MRFs. In Saad and Opper (Eds.),
AdvancedMeanFieldMethods. MIT
Press.
Weiss, Y. and W. T. Freeman (1999).
Correctness of belief propagationin Gaussian graphical models of ar-bitrary topology. In NIPS-12.
Weiss, Y. and W. T. Freeman (2001a).
Correctness of belief propagationin Gaussian graphical models of ar-
bitrary topology. Neural Computa-
tion 13(10), 2173–2200.
Weiss, Y. and W. T. Freeman (2001b).
On the optimality of solutions of

Press.
the max-product belief propaga-tion algorithm in arbitrary graphs.
IEEE Trans. Information Theory,
Special Issue on Codes on GraphsandIterativeAlgorithms 47 (2), 723–
735.
Weiss, Y., A. Torralba, and R. Fergus
(2008). Spectral hashing. In NIPS.

BIBLIOGRAPHY 1043
Welling, M., C. Chemudugunta, and
N. Sutter (2008). Deterministic la-
tent variable models and their pit-
falls. InIntl.Conf.onDataMining.
Welling, M., T. Minka, and Y. W. Teh
(2005). Structured region graphs:
Morphing EP into GBP. In UAI.
Welling, M., M. Rosen-Zvi, and G. Hin-
ton (2004). Exponential family har-moniums with an application toinformation retrieval. In NIPS-14.
Welling, M. and C. Sutton (2005).
Learning in Markov random ﬁeldswith contrastive free energies. In
Tenth International Workshop on
Artiﬁcial Intelligence and Statistics
(AISTATS).
Welling, M. and Y.-W. Teh (2001). Belief
optimization for binary networks:a stable alternative to loopy beliefpropagation. In UAI.
Werbos, P. (1974). Beyond regression:
NewToolsforPredictionandAnaly-
sisintheBehavioralSciences. Ph.D.
thesis, Harvard.
West, M. (1987). On scale mix-
tures of normal distributions.Biometrika 74, 646–648.
West, M. (2003). Bayesian Factor Re-
gression Models in the "Large p,Small n" Paradigm. BayesianStatis-
tics7.

tics7.
West, M. and J. Harrison (1997).
Bayesian forecasting and dynamic
models. Springer.
Weston, J., S. Bengio, and N. Usunier
(2010). Large Scale Image Annota-
tion: Learning to Rank with JointWord-Image Embeddings. In Proc.
Weston, J., F. Ratle, and R. Collobert
(2008). Deep Learning via Semi-
Supervised Embedding. In Intl.
Weston, J. and C. Watkins (1999).
Multi-lcass support vector ma-
chines. InESANN.
Wiering, M. and M. van Otterlo

tics7.
(Eds.) (2012). Reinforcement learn-
ing: State-of-the-art. Springer.
Wilkinson, D. and S. Yeung (2002).
Conditional simulation from highlystructured gaussian systems withapplication to blocking-mcmc forthe bayesian analysis of very large
linear models. Statistics and Com-
puting 12, 287–300.Williams, C. (1998). Computation with
inﬁnite networks. NeuralComputa-
tion 10(5), 1203–1216.
Williams, C. (2000). A MCMC ap-
proach to Hierarchical Mixture
Modelling . In S. A. Solla, T. K.Leen, and K.-R. Müller (Eds.), NIPS.

MIT Press.
Williams, C. (2002). On a Connec-
tion between Kernel PCA and Met-ric Multidimensional Scaling. Ma-
chineLearningJ. 46 (1).
Williams, O. and A. Fitzgibbon (2006).
Gaussian process implicit surfaces.InGaussianprocessesinpractice.
Williamson, S. and Z. Ghahramani
(2008). Pr
obabilistic models for
data combination in recommendersystems. In NIPS Workshop on
LearningfromMultipleSources.
Winn, J. and C. Bishop (2005). Varia-
tional message passing. J. of Ma-
chine Learning Research 6, 661–694.
Wipf, D. and S. Nagarajan (2007). A

(2008). Pr
new view of automatic relevancydetermination. In NIPS.
Wipf, D. and S. Nagarajan (2010, April).
Iterative Reweighted ℓ−1andℓ−2
Methods for Finding Sparse Solu-
tions.J.ofSelectedTopicsinSignal
Processing (Special Issue on Com-
pressiveSensing) 4 (2).
Wipf, D., B. Rao, and S. Nagarajan
(2010). Latent variable bayesianmodels for promoting sparsity.
IEEE Transactions on Information
Theory.
Witten, D., R. Tibshirani, and T. Hastie
(2009). A penalized matrix de-

Theory.
composition, with applications tosparse principal components andcanonical correlation analysis. Bio-
statistics 10 (3), 515–534.
Wolpert, D. (1992). Stacked generaliza-
tion.NeuralNetworks5 (2), 241–259.
Wolpert, D. (1996). The lack of a priori
distinctions between learning algo-
rithms.Neural Computation 8 (7),
1341–1390.
Wong, F., C. Carter, and R. Kohn
(2003). Efficient estimation of
covariance selection models.Biometrika 90 (4), 809–830.Wood, F., C. Archambeau, J. Gasthaus,

L. James, and Y. W. Teh (2009). Astochastic memoizer for sequence
data. InIntl. Conf. on Machine
Learning.
Wright, S., R. Nowak, and
M. Figueiredo (2009). Sparse
reconstruction by separable ap-
proximation. IEEE Trans. on Signal
Processing 57 (7), 2479–2493.
Wu, T. T. and K. Lange (2008). Coordi-
nate descent algorithms for lassopenalized regression. Ann. Appl.
Stat 2(1), 224–244.
Wu, Y., H. Tjelmeland, and M. West
(2007). Bayesian CART: Prior struc-
ture and MCMC computations. J.

Learning.
of Computational and GraphicalStatistics 16 (1), 44–66.
Xu, F. and J. Tenenbaum (2007). Word
learning as Bayesian inference. Psy-
chologicalReview 114 (2).
Xu, Z., V. Tresp, A. Rettinger, and
K. Kersting (2008). Social net-work mining with nonparametric
relational models. In ACM Work-
shoponSocialNetworkMiningand
Analysis(SNA-KDD2008).
Xu, Z., V. Tresp, K. Yu, and H.-P.
Kriegel
(2006). Inﬁnite hidden rela-
tional models. In UAI.
Xu, Z., V. Tresp, S. Yu, K. Yu, and H.-P.
Kriegel (2007). Fast inference in in-

Kriegel
ﬁnite hidden relational models. InWorkshop on Mining and Learning
withGraphs.
Xue, Y., X. Liao, L. Carin, and B. Krish-
napuram (2007). Multi-task learn-
ing for classiﬁcation with dirichlet
process priors. J.ofMachineLearn-
ingResearch 8 , 2007.
Yadollahpour, P., D. Batra, and
G.Shakhnarovich(2011). DiverseM-
best Solutions in MRFs. In NIPS
workshop on Disrete Optimization
inMachineLearning.
Yan, D., L. Huang, and M. I. Jordan
(2009). Fast approximate spectral
clustering. In 15th ACM Conf. on
ing.

ing.
Yang, A., A. Ganesh, S. Sastry, and
Y. Ma (2010, Feb). Fast l1-minimization algorithms and anapplication in robust face recog-nition: A review. Technical Re-port UCB/EECS-2010-13, EECS De-partment, University of California,Berkeley.

1044 BIBLIOGRAPHY
Yang, C., R. Duraiswami, and L. David
(2005). Efficient kernel machines
using the improved fast Gausstransform. In NIPS.
Yang, S., B. Long, A. Smola, H. Zha,
and Z. Zheng (2011). Collaborativecompetitive ﬁltering: learning rec-ommender using context of user
choice. InProc. Annual Intl. ACM
SIGIRConference.
Yanover, C., O. Schueler-Furman, and
Y. Weiss (2007). Minimizing and
Learning Energy Functions for
Side-Chain Prediction. In Recomb.
Yaun, G.-X., K.-W. Chang, C.-J. Hsieh,
and C.-J. Lin (2010). A Comparison
of Optimization Methods and Soft-ware for Large-scale L1-regularized
Linear Classiﬁcation. J. of Machine
LearningResearch 11, 3183–3234.
Yedidia, J., W. T. Freeman, and
Y. Weiss (2001). Understanding be-
lief propagation and its generaliza-
tions. InIntl.JointConf.onAI.
Yoshida, R. and M. West (2010).
Bayesian learning in sparse graphi-
cal factor models via annealed en-tropy.J. of Machine Learning Re-
search 11, 1771–1798.
Younes, L. (1989). Parameter estima-

tion for imperfectly observed Gibb-
sian ﬁelds.Probab.TheoryandRe-
latedFields 82, 625–645.
Yu, C. and T. Joachims (2009). Learn-
ing structural SVMs with latentvariables. In Intl.Conf.onMachine
Learning.
Yu, S., K. Yu, V. Tresp, K. H-P., and
M. Wu (2006). Supervised proba-
bilistic principal component anal-
ysis. InProc. of the Int’l Conf. on
ing.
Yu, S.-Z. and H. Kobayashi (2006).

Learning.
ing.
Practical implementation of an ef-ﬁcient forward-backward algorithmfor an explicit-duration hiddenMarkov model. IEEE Trans. on Sig-
nalProcessing 54 (5), 1947– 1951.
Yuan, M. and Y. Lin (2006). Model
selection and estimation in re-gression with grouped variables.
J. Royal Statistical Society, Series
B6 8(1), 49–67.
Yuan, M. and Y. Lin (2007).
Model selection and estimationin the gaussian graphical model.
Biometrika 94 (1), 19–35.Yuille, A. (2001). CCCP algorithms
to minimze the Bethe and Kikuchi
free energies: convergent alterna-

tives to belief propagation. Neural
Computation 14, 1691–1722.
Yuille, A. and A. Rangarajan (2003).
The concave-convex procedure.NeuralComputation 15, 915.
Yuille, A. and S. Zheng (2009). Com-
positional noisy-logical learning. InIntl.Conf.onMachineLearning.
Yuille, A. L. and X. He (2011). Proba-
bilistic models of vision and max-
margin methods. Frontiers of
Electrical and Electronic Engineer-ing 7(1).
Zellner, A. (1986). On assessing prior
distributions and bayesian regres-sion analysis with g-prior distri-
butions. In Ba
 yesian inference
and decision techniques, Studies

of Bayesian and Econometrics andStatisticsvolume6. North Holland.
Zhai, C. and J. Lafferty (2004). A
study of smoothing methods for
language models applied to infor-mation retrieval. ACMTrans.onIn-
formationSystems 22 (2), 179–214.
Zhang, N. (2004). Hierarchical latnet
class models for cluster analysis. J.
ofMachineLearningResearch , 301–
308.
Zhang, N. and D. Poole (1996). Ex-
ploiting causal independence in
Bayesian network inference. J. of
AIResearch , 301–328.
Zhang, T. (2008). Adaptive Forward-
Backward Greedy Algorithm for

308.
Sparse Learning with Linear Mod-els. InNIPS.
Zhang, X., T. Graepel, and R. Herbrich
(2010). Bayesian Online Learningfor Multi-label and Multi-variatePerformance Measures. In AI/Statis-
tics.
Zhao, J.-H. and P. L. H. Yu (2008,
November). Fast ML Estimation forthe Mixture of Factor Analyzers viaan ECM Algorithm. IEEE.Trans.on
NeuralNetworks 19 (11).
Zhao, P., G. Rocha, and B. Yu (2005).
Grouped and Hierarchical ModelSelection through Composite Abso-lute Penalties. Technical report, UCBerkeley.Zhao, P. and B. Yu (2007). Stagewise

tics.
Lasso.J. of Machine Learning Re-
search 8, 2701–2726.
Zhou, H., D. Karakos, S. Khudanpur,
A. Andreou, and C. Priebe (2009).On Projections of Gaussian Dis-tributions using Maximum Likeli-
hood Criteria. In Proc.oftheWork-
shoponInformationTheoryandits
Applications.
Zhou, M., H. Chen, J. Paisley, L. Ren,
G. Sapiro, and L. Carin (2009).
Non-parametric Bayesian Dictio-nary Learning for Sparse ImageRepresentations. In NIPS.
Zhou, X. and X. Liu (2008). The
EM algorithm for the extended ﬁ-nite mixture of the factor analyz-

ers model.ComputationalStatistics
andDataAnalysis 52, 3939–3953.
Zhu, C. S., N. Y. Wu, and D. Mum-
ford (1997, November). Minimax
entropy principle and its applica-
tion to texture modeling. Neural
Computation 9 (8).
Zhu, J. and E. Xing (2010). Conditional
topic random ﬁelds. In Intl. Conf.
Zhu, L., Y. Chen, A.Yuille, and W. Free-
man (2010). Latent hierarchical
structure learning for object detec-tion. InCVPR.
Zhu, M. and A. Ghodsi (2006). Au-
tomatic dimensionality selectionfrom the scree plot via the use of

proﬁle likelihood. Computational
Statistics & Data Analysis 51, 918–
930.
Zhu, M.andA.Lu(2004). Thecounter-
intuitive non-informative prior for
thebe
rnoulli family. J.StatisticsEd-
ucation.
Zinkevich, M. (2003). Online con-
vex programming and generalized
inﬁnitesimal gradient ascent. In
928â˘A¸S936.
Zobay, O. (2009). Mean ﬁeld inference
for the Dirichlet process mixture
model.Electronic J. of Statistics 3,
507–545.
Zoeter, O. (2007). Bayesian generalized

930.
thebe
ucation.
507–545.
linear models in a terabyte world.InProc. 5th International Sympo-
sium on image and Signal Process-
ingandAnalysis.

BIBLIOGRAPHY 1045
Zou, H. (2006). The adaptive Lasso
and its oracle properties. J. of the
Am.Stat.Assoc., 1418–1429.
Zou, H. and T. Hastie (2005). Regular-
ization and variable selection via
the elastic net. J. of Royal Stat. Soc.
SeriesB 67 (2), 301–320.
Zou, H., T. Hastie, and R. Tibshirani
(2006). Sparse principal compo-nent analysis. J. of Computational
and Graphical Statistics 15 (2), 262–
286.
(2007). On the "Degrees of Free-
dom" of the Lasso. AnnalsofStatis-
tics 35(5), 2173–2192.
Zou, H. and R. Li (2008). One-
step sparse estimates in noncon-cave penalized likelihood models.AnnalsofStatistics36 (4), 1509–1533.
Zweig, G. and M. Padmanabhan
(2000). Exact alpha-beta computa-
tion in logarithmic space with ap-plication to map word graph con-
struction. In Proc.Intl.Conf.Spoken
Lang.

Index to code
agglomDemo, 894
amazonSellerDemo, 155arsDemo, 819arsEnvelope, 819
bayesChangeOfVar, 151
bayesLinRegDemo2d, 233bayesTtestDemo, 138beliefPropagation, 768bernoulliEntropyFig, 57besselk, 477betaBinomPostPredDemo, 79betaCredibleInt, 153betaHPD, 153, 154betaPlotDemo, 43biasVarModelComplexity3, 204bimodalDemo, 150binaryFaDemoTipping, 403binomDistPlot, 35binomialBetaPosteriorDemo, 75bleiLDAperplexityPlot, 955bolassoDemo, 440boostingDemo, 555, 558bootstrapDemoBer, 192
cancerHighDimClassifDemo, 110
cancerRatesEb, 172casinoDemo, 606, 607centralLimitDemo, 52changeOfVarsDemo1d, 53chowliuTreeDemo, 913coinsModelSelDemo, 164contoursSSEdemo, 219convexFnHand, 222curseDimensionality, 18
demard, 580
depnetFit, 909dirichlet3dPlot, 48dirichletHistogramDemo, 48discreteProbDistFig, 28discrimAnalysisDboundariesDemo, 103, 105discrimAnalysisFit, 106discrimAnalysisHeightWeightDemo, 145discrimAnalysisPredict, 106dpmGauss2dDemo, 888dpmSampleDemo, 881dtﬁt, 545dtreeDemoIris, 549, 550
elasticDistortionsDemo, 567

emLogLikelihoodMax, 365
faBiplotDemo, 383
ﬁsherDiscrimVowelDemo, 274ﬁsheririsDemo, 6ﬁsherLDAdemo, 272fmGibbs, 843
gammaPlotDemo, 41, 150

gammaRainfallDemo, 41gampdf, 41gaussCondition2Ddemo2, 112gaussHeightWeight, 102gaussImputationDemo, 115, 375gaussInferParamsMean1d, 121gaussInferParamsMean2d, 123gaussInterpDemo, 113gaussInterpNoisyDemo, 125gaussMissingFitEm, 374gaussMissingFitGibbs, 840gaussPlot2d, 142gaussPlot2Ddemo, 47gaussPlotDemo, 19gaussSeqUpdateSigma1D, 131generativeVsDiscrim, 269geomRidge, 229ggmFitDemo, 939ggmFitHtf, 939ggmFitMinfunc, 939ggmLassoDemo, 13, 940ggmLassoHtf, 940gibbsDemoIsing, 670, 873gibbsGaussDemo, 848giniDemo, 548gpcDemo2d, 529gpnnDemo, 536gprDemoArd, 520gprDemoChangeHparams, 519gprDemoMarglik, 522gprDemoNoiseFree, 517gpSpatialDemoLaplace, 532groupLassoDemo, 451
hclustYeastDemo, 894, 896
hingeLossPlot, 211, 556hmmFilter, 609hmmFwdBack, 611hmmLillypadDemo, 604hmmSelfLoopDist, 623hopﬁeldDemo, 670huberLossDemo, 223, 497
icaBasisDemo, 471

icaDemo, 408icaDemoUniform, 409IPFdemo2x2, 683isingImageDenoiseDemo, 739, 839
kalmanFilter, 641
kalmanTrackingDemo, 632kernelBinaryClassifDemo, 489kernelRegrDemo, 490, 491kernelRegressionDemo, 510KLfwdReverseMixGauss, 734KLpqGauss, 734kmeansHeightWeight, 10kmeansModelSel1d, 371kmeansYeastDemo, 341knnClassifyDemo, 17, 23–25knnVor
onoi, 16
kpcaDemo2, 495kpcaScholkopf, 493
lassoPathProstate, 437, 438
LassoShooting, 441leastSquaresProjection, 221linregAllsubsetsGraycodeDemo, 423linregBayesCaterpillar, 237, 238linregCensoredSchmeeHahnDemo, 379

1048 INDEXES
linregDemo1, 241
linregEbModelSelVsN, 158, 159, 749linregFitL1Test, 447linregOnlineDemoKalman, 636linregPolyLassoDemo, 436linregPolyVsDegree, 9, 20, 436linregPolyVsN, 231linregPolyVsRegDemo, 208, 225, 226, 239linregPostPredDemo, 235linregRbfDemo, 487linregRobustDemoCombined, 223linregWedgeDemo2, 19LMSdemo, 265logregFit, 254logregLaplaceGirolamiDemo, 257, 258logregMultinomKernelDemo, 269logregSATdemo, 21logregSATdemoBayes, 259logregSatMhDemo, 852logregXorDemo, 486logsumexp, 86lossFunctionFig, 179lsiCode, 419
marsDemo, 554

mcAccuracyDemo, 55mcEstimatePi, 54mcmcGmmDemo, 851, 860, 861mcQuantileDemo, 153mcStatDist, 598miMixedDemo, 59mixBerMnistEM, 341mixBetaDemo, 170mixexpDemo, 343mixexpDemoOneToMany, 344mixGaussDemoFaithful, 353mixGaussLikSurfaceDemo, 346mixGaussMLvsMAP, 356mixGaussOverRelaxedEmDemo, 369mixGaussPlotDemo, 339mixGaussSingularity, 356mixGaussVbDemoFaithful, 753, 755mixPpcaDemoNetlab, 386mixStudentBankruptcyDemo, 361mlpPriorsDemo, 574mlpRegEvidenceDemo, 579mlpRegHmcDemo, 579mnist1NNdemo, 25, 1002multilevelLinregDemo, 844mutualInfoAllPairsMixed, 59
naiveBayesBowDemo, 84, 88
naiveBayesFit, 83, 277naiveBayesPredict, 86, 277netﬂixResultsPlot, 981newsgroupsVisualize, 5newtonsMethodMinQuad, 250newtonsMethodNonConvex, 250ngramPlot, 592NIXdemo2, 135normalGammaPenaltyPlotDemo, 460normalGammaThresholdPlotDemo, 461numbersGame, 69–71
pagerankDemo, 600, 603

pagerankDemoPmtk, 602paretoPlot, 44parzenWindowDemo2, 509pcaDemo2d, 388pcaDemo3d, 11pcaDemoHeightWeight, 389pcaEmStepByStep, 397pcaImageDemo, 12, 389pcaOverﬁtDemo, 400–402pcaPmtk, 393pfColorTrackerDemo, 830poissonPlotDemo, 37postDensityIntervals, 154ppcaDemo2d, 388PRhand, 182probitPlot, 259probitRegDemo, 259, 294, 363prostateComparison, 436prostateSubsets, 427
quantileDemo, 33randomWalk0to20Demo, 856
rbpfManeuverDemo, 834, 835rbpfSlamDemo, 835rdaFit, 108regtreeSurfaceDemo, 545rejectionSamplingDemo, 818relevanceNetworkNewsgroupDemo, 908residualsDemo, 219ridgePathProstate, 437riskFnGauss, 198robustDemo, 40robustPriorDemo, 168
saDemoPeaks, 869, 870

sampleCdf, 816samplingDistGaussShrinkage, 203sensorFusion2d, 123sensorFusionUnknownPrec, 141seqlogoDemo, 36shrinkageDemoBaseball, 175shrinkcov, 130shrinkcovDemo, 129shrunkenCentroidsFit, 109shrunkenCentroidsSRBCTdemo, 109, 110shuffledDigitsDemo, 7, 25sigmoidLowerBounds, 761sigmoidPlot, 21sigmoidplot2D, 246simpsonsParadoxGraph, 933sliceSamplingDemo1d, 865sliceSamplingDemo2d, 865smoothingKernelPlot, 507softmaxDemo2, 103SpaRSA, 445sparseDictDemo, 471sparseNnetDemo, 575sparsePostPlot, 459sparseSensingDemo, 438spectralClusteringDemo, 893splineBasisDemo, 125ssmTimeSeriesSimple, 638, 639steepestDescentDemo, 247, 248stickBreakingDemo, 883studentLaplacePdfPlo t,
40

40
subgradientPlot, 432subSuperGaussPlot, 412surfaceFitDemo, 218svdImageDemo, 394svmCgammaDemo, 504
tanhPlot, 570
trueskillDemo, 798

INDEXTOCODE 1049
trueskillPlot, 797
unigaussVbDemo, 745varEMbound, 368
variableElimination, 717visDirichletGui, 48visualizeAlarmNetwork, 314vqDemo, 354
wiPlotDemo, 127

Index to keywords
#P-hard, 727
0-1 loss,177
3-SAT, 727
A star search, 887
absorbing state, 598
accept,848
action, 176action nodes, 328
action space, 176
actions, 176activation, 563
active learning, 230, 234, 938Active set, 441
active set, 442
Activity recognition, 605
Adaboost.M1, 559
adagrad,263
adaline,569
adaptive basis-function model, 543
adaptive importance sampling, 821
adaptive lasso, 460
adaptive MCMC, 853
adaptive rejection Metropolis sampling, 820
adaptive rejection sampling, 820
add-one smoothing, 77, 593
ADF,653, 983
adjacency matrix, 309,970
adjust for, 934
adjusted Rand index, 878
admissible, 197
admixture mixture, 950
AdSense, 928AdWords, 928affinity propagation, 887
agglomerative clustering, 893
agglomerative hierarchical clustering, 927aha,68
AI, 1007AIC,162, 557
Akaike information criterion, 162
alarm network, 313
alignment, 701all pairs,503
alleles,317
alpha divergence, 735
alpha expansion, 803
alpha-beta swap, 804
alternative hypothesis, 163
analysis view, 390
analysis-synthesis, 470

ancestors, 309
ancestral graph, 664
ancestral sampling, 822
and-or graphs, 1007annealed importance sampling, 871, 923
annealing, 853annealing importance sampling, 992ANO
VA, 553
anti-ferromagnets, 668
aperiodic, 598
approximate inference, 727
approximation error, 230
ARD, 238, 463, 520, 580
ARD kernel, 480
area under the curve, 181ARMA,639, 674
array CGH, 454association rules, 15associative, 931
associative Markov network, 668
associative memory, 568,669, 997
associative MRF, 802

VA, 553
assumed density ﬁlter, 267assumed density ﬁltering, 653, 787
asymptotically normal, 194
asymptotically optimal, 201
asynchronous updates, 774
atom,469
atomic bomb, 52attractive MRF, 802
attributes, 2,3
AUC,181
audio-visual speech recognition, 628
augmented DAG, 932
auto-encoder, 1000
auto-encoders, 990auto-regressive HMM, 626
autoclass, 11
autocorrelation function, 862
automatic relevance determination, 463
automatic relevancy determination, 238, 398, 580, 747Automatic speech recognition, 605

atom,469
AUC,181
automatic speech recognition, 624auxiliary function, 350
auxiliary variables, 863, 868
average link clustering, 897
average precision, 303average precision at K, 183
axis aligned, 47
axis parallel splits, 544
back-propagation, 999backdoor path, 934
backﬁtting, 552, 563, 998
background knowledge, 68backoff smoothing, 594
backpropagation, 570, 970
backpropagation algorithm, 569backslash operator, 228
Backwards selection, 428
bago
f words,5,81,945
bag-of-characters, 483bag-of-words, 483bagging,551
bandwidth, 480,507

bago
barren node removal, 334, 714
BART,551, 586
Barzilai-Borwein, 445
base distribution, 338
base learner, 554
base measure, 882
base rate fallacy, 30
basic feasible solution, 468basis function expansion, 20,217
basis functions, 421basis pursuit denoising, 430
batch,261
Baum-Welch, 618
Bayes ball algorithm, 324
Bayes decision rule, 177,195
Bayes estimator, 177,195

INDEXTOKEYWORDS 1051
Bayes factor, 137, 163, 921
Bayes model averaging, 71,581
Bayes point, 257
Bayes risk, 195
Bayes rule, 29, 340
Bayes Theorem, 29
Bayesian, xxvii, 27
Bayesian adaptive regression trees, 551
Bayesian factor regression, 405
Bayesian hierarchical clustering, 899
Bayesian information criterion, 161
Bayesian IPF, 683
Bayesian lasso, 448
Bayesian model selection, 156
Bayesian network structure learning, 914
Bayesian networks, 310
Bayesian Occam’s razor, 156
Bayesian statistics, 149, 191
BDe,917
BDeu,918
beam search, 428, 887belief networks, 310
belief propagation, 611, 707, 767
belief state, 71,332,607,609
belief state MDP, 332
belief updating, 709
bell curve, 20,38
Berkson’s paradox, 326
Bernoulli, 21,34
Bernoulli product model, 88
Bernoulli-Gaussian, 426
Bessel function, 483beta distribution, 42,7 4
beta function, 42beta process, 470beta-binomial, 78
Bethe,781
Bethe energy functional, 781
Bethe free energy, 781
BFGS,251
Bhattacharya distance, 828
bi-directed graph, 674
bias,20,200, 457

BDe,917
BDeu,918
Bethe,781
BFGS,251
bias term, 669
bias-variance
tradeoff,202
BIC,161, 256, 557, 920
biclustering, 903
big data,1
bigram model, 591
binary classiﬁcation, 3,6 5
binary entropy function, 57
binary independence model, 88
binary mask, 426, 470
binary tree, 895
Bing, 302, 799, 983binomial, 34
binomial coefficient, 34
binomial distribution, 74binomial regression, 292BinomialBoost, 561
BIO,687
biosequence analysis, 36, 170
bipartite graph, 313
biplot,383
birth moves, 370bisecting K-means, 898
bits,56

big data,1
BIO,687
bits,56
bits-back, 733black swan paradox, 77,8 4
black-box, 340,585Blackwell-MacQueen, 884
blank slate, 165
blind signal separation, 407
blind source separation, 407
blocked Gibbs sampling, 848
blocking Gibbs sampling, 848
bloodtype, 317BN2O,315
bolasso,439
Boltzmann distribution, 104,869
Boltzmann machine, 568, 669,983
bond variables, 866
Boosting, 554
boosting, 553, 742bootstrap, 192
bootstrap ﬁlter, 827
bootstrap lasso, 439
bootstrap resampling, 439borrow statistical strength, 171, 231, 296, 845
bottleneck, 205, 337,1000

bottleneck layer, 970
boundop
timization, 369
box constraints, 444Box-Muller, 817
boxcar kernel, 508, 508
Boyen-Koller, 654
BP,707
BPDN,430
Bradley Terry, 795branch and bound, 811branching factor, 954
bridge regression, 458
Brownian motion, 483bucket elimination, 715
BUGS, 756, 847
Buried Markov models, 627
burn-in phase, 856
burned in, 838
burstiness, 88
bursty, 480
C4.5,545
calculus of variations, 289
calibration, 724
Candidate method, 872
Canonical correlation analysis, 407
canonical form, 282
canonical link function, 291

boundop
BP,707
BPDN,430
C4.5,545
canonical parameters, 115,282
Cardinality constraints, 810CART,544, 545
Cartesian, 51cascade,776
case analysis, 260
categorical, 2,35
categorical PCA, 402,947, 961
categorical variables, 876Cauchy,40
causal Markov assumption, 931
Causal models, 931
causal MRF, 661
causal networks, 310
causal sufficiency, 931
causality, 919, 929CCA,407
CCCP,702
CD,989
cdf,32,3 8
Censored regression, 379
censored regression, 380

1052 INDEXES
centering matrix, 494
central composite design, 523
central interval, 152
central limit theorem, 38, 51, 255
central moment, 413
central-limit theorem, 55
centroid, 341
centroids, 486
certainty factors, 675
chain graph, 671
chain rule, 29,307
chance nodes, 328
change of variables, 50
channel coding, 56
Chapman-Kolmogorov, 590
characteristic length scale, 480
Cheeseman-Stutz approximation, 923
Chi-squared distribution, 42
chi-squared statistic, 163, 213
children,309, 310
Chinese restaurant process, 884
chip-Seq, 622Cholesky decomposition, 227, 817
Chomsky normal form, 689chordal,665
chordal graph, 720
Chow-Liu algorithm, 312, 912
CI,308
circuit complexity, 944
city block distance, 876
clamped phase, 987
clamped term, 677
clamping, 319
class imbalance, 503
class-conditional density, 30,6 5
classical, 149classical statistics, 191
classiﬁcation, 2,3
Classiﬁcation and regression trees, 544
clausal form, 675
clause, 727click-through rate, 4
clique,310
cliques, 719, 722closing the loop, 635
closure,662

CI,308
cluster variational method, 783
Clustering, 875
clustering, 10, 340
clusters, 487clutter problem, 788
co-cluste
ring, 979
co-occurrence matrix, 5co-parents, 327
coarse-to-ﬁne grid, 775cocktail party problem, 407
coclustering, 903
codebook, 354
collaborative ﬁltering, 14, 300, 387, 903, 979
collapsed Gibbs sampler, 841
collapsed Gibbs sampling, 956
collapsed particles, 831
collect evidence, 707
collect-to-root, 723
collider,324
COLT,210
committee method, 580
commutative semi-ring, 717

co-cluste
ring, 979
COLT,210
commutative semiring, 726compactness, 897compelled edges, 915
complementary prior, 997
complete, 322
complete data, 270, 349
complete data assumption, 914
complete data log likelihood, 348,350
complete link clustering, 897
completing the square, 143
composite likelihood, 678
compressed sensing, 472
compressive sensing, 472
computation tree, 772
computational learning theory, 210
computationalism, 569
concave,222, 286
concave-convex procedure, 702
concentration matrix, 46
concentration parameter, 882

concept, 65concept learning, 65
condensation, 827
conditional entropy, 59
conditional Gamma Poisson, 949conditional Gaussian, 920
conditional independence, 308conditional likelihood, 620conditional logit model, 252
conditional probability, 29
conditional probability distribution, 308
conditional probability tables, 308
conditional random ﬁeld, 684
conditional random ﬁelds, 606, 661conditional topic random ﬁeld, 969
conditionally conjugate, 132
conditionally independent, 31
,82
conditioning, 319

,82
conditioning case, 322
conductance, 858
conﬁdence interval, 212
conﬁdence intervals, 153confounder, 674
confounders, 931
confounding variable, 934
confusion matrix, 181
conjoint analysis, 297
conjugate gradients, 249, 524
conjugate prior, 74
conjugate priors, 281, 287conjunctive normal form, 675
connectionism, 569
consensus sequence, 36,606
conservation of probability mass, 157
consistent, 200
consistent estimator, 233consistent estimators, 70constant symbols, 676

,82
constraint satisfaction problems, 717, 726constraint-based approach, 924
content addressable memory, 669
context free grammar, 689
context speciﬁc independence, 321
context-speciﬁc independence, 944contextual bandit, 184, 254
contingency table, 682continuation method, 442,869
contrastive divergence, 569, 989
contrastive term, 677
control signal, 625, 631converge, 857convex, 58, 221, 247, 285, 677

INDEXTOKEYWORDS 1053
convex belief propagation, 785, 943
convex combination, 76, 130, 338
convex hull, 777
convolutional DBNs, 1004
convolutional neural nets, 1004
convolutional neural network, 565
cooling schedule, 870
corpus,953
correlated topic model, 757, 961
correlation coefficient, 45, 876
correlation matrix, 45
correspondence, 658
cosine similarity, 480
cost-beneﬁt analysis, 186
coupled HMM, 628
covariance, 44
covariance graph, 674, 908
covariance matrix, 45,4 6
covariance selection, 938
covariates, 2
CPD,308
CPTs,308
Cramer-Rao inequality, 201Cramer-Rao lower bound, 201
credible interval, 137,152, 212
CRF, 661, 684
critical temperature, 868
critical value, 671
cross entropy, 57, 571
cross over rate, 181
cross validation, 24,206
cross-entropy, 246, 953
cross-language information retrieval, 963crosscat,904
crowd sourcing, 10, 995
CRP,884
CTR,4
cubic spline, 537
cumulant function, 282,284
cumulants, 284
cumulative distribution function, 32,3 8
curse of dimensionality, 18, 487

CPD,308
CPTs,308
CRP,884
CTR,4
curved exponential family, 282
cutting plane, 698
CV,24
cycle,310
cyclic
permutation property, 99
d-prime,106
d-separated, 324
DACE, 518DAG,310
damped updates, 739
damping, 773
Dasher, 591data association, 658, 810
data augmentation, 362, 847
data compression, 56
data fragmentation, 546
data fusion, 404
data overwhelms the prior, 69
data-driven MCMC, 853
data-driven proposals, 828
DBM,996
DBN,628,997
DCM,89
DCT, 469death moves, 370debiasing, 439
decision, 176decision boundary, 22decision diagram, 328
decision nodes, 328

CV,24
cycle,310
cyclic
DBM,996
DCM,89
decision problem, 176decision procedure, 177
decision rule, 22
decision trees, 544
decoding, 693
decomposable, 665,722, 941
decomposable graphs, 682decomposes, 322,917
DeeBN,628
DeeBNs,997
deep, 929deep auto-encoders, 1000
deep belief network, 997
deep Boltzmann machine, 996
deep directed networks, 996
deep learning, 479, 995
deep networks, 569defender’s fallacy, 61
deﬂated matrix, 418
degeneracy problem, 825
degenerate, 532, 535
degree,310
degrees of freedom, 39,161, 206,229,534
delet
ed interpolation, 593
delta rule, 265

DeeBN,628
delet
dendrogram, 895
denoising auto-encoder, 1001
dense stereo reconstruction, 690
density estimation, 9
dependency network, 909
dependency networks, 679derivative free ﬁlter, 651
descendants, 309
descriptive, 2
design matrix, 3, 875detailed balance, 854
detailed balance equations, 599
determinism, 944deterministic annealing, 367, 620
deviance, 547
DGM,310
diagonal, 46diagonal covariance LDA, 107diagonal LDA, 108
diameter, 710,897
dictionary, 469
digamma, 361, 752, 958digital cameras, 8dimensionality reduction, 11, 1000

DGM,310
Dirac delta function, 39
Dirac measure, 37,68
Dirchlet process, 903direct posterior probability approach, 184
directed,309
directed acyclic graph, 310
directed graphical model, 310
directed local Markov property, 327
directed mixed graph, 929directed mixed graphical model, 674
Dirichlet, 79Dirichlet Compound Multinomial, 89
Dirichlet distribution, 47
Dirichlet multinomial regression LDA, 969
Dirichlet process, 596, 879,882, 973, 976
Dirichlet process mixture models, 508, 755discontinuity preserving, 691

discounted cumulative gain, 303

1054 INDEXES
discrete,35
discrete AdaBoost, 559
discrete choice modeling, 296
discrete random variable, 28
discrete with probability one, 884
discretize, 59, 691
discriminability, 106
discriminant analysis, 101
discriminant function, 500
discriminative, 245
discriminative classiﬁer, 30
discriminative LDA, 968
discriminative random ﬁeld, 684
disease mapping, 531
disease transmission, 970disparity, 691
dispersion parameter, 290
dissimilarity analysis, 898
dissimilarity matrix, 875
distance matrix, 875
distance transform, 775
distorted, 566
distortion, 354
distribute evidence, 707
distribute-from-root, 724
distributed encoding, 984
distributed representation, 569,627
distributional particles, 831
distributive law, 717
divisive clustering, 893
DNA sequences, 36do calculus, 932
Document classiﬁcation, 87
document classiﬁcation, 5
Domain adaptation, 297
domain adaptation, 297dominates, 197
double loop algorithms, 773
double Pareto distribution, 461double sided exponential, 41
dRUM,294
dual decomposition, 808

dRUM,294
dual variables, 492, 499
dummy encoding, 35
dyadic, 976DyBN,628
DyBNs,997
dynamic Bayes net, 653dynamic Bayesian network, 628
dynamic linear model, 636
dynamic programming, 331, 920dynamic topic model, 962
E step,350
e-commer
ce, 11
early stopping, 263,557,572
EB,173
ECM,369, 387
ECME,369
ECOC,581
econometric forecasting, 660
economy sized SVD, 392
edge appearance probability, 786
edges,309
edit distance, 479EER,181
effective sample size, 75,825,862
efficient IPF, 683
efficiently PAC-learnable, 210

DyBNs,997
E step,350
e-commer
ce, 11
EB,173
ECME,369
ECOC,581
edges,309
eigendecomposition, 98eigenfaces, 12eigengap, 857eigenvalue spectrum, 130
EKF,648
elastic net, 438, 456, 936
elimination order, 718
EM, 271,349, 618, 749
email spam ﬁltering, 5
embedding, 575empirical Bayes, 157, 162,173, 300, 746
empirical distribution, 37, 205
empirical measure, 37
empirical risk, 205, 697
empirical risk minimization, 205, 261
end effector, 344energy based models, 666
energy function, 255
energy functional, 732,778
ensemble, 980Ensemble learning, 580
ensemble learning, 742
entanglement, 629

EKF,648
entanglement problem, 635, 653Entropy,547
entropy,56
EP, 983Epanechnikov kernel, 508
ePCA,947
epigraph, 222
epistemological uncertainty, 973
epoch,264,566
epsilon
insensitive loss function, 497
EPSR,859
equal error rate, 181
equilibrium distribution, 597
equivalence class, 915
equivalent kernel, 512,533
equivalent sample size, 76,917
erf,38
ergodic,599
Erlang distribution, 42
ERM,205, 261
error bar, 76
error correcting codes, 768error correction, 56
error function, 38
error signal, 265error-correcting output codes, 503, 581
ESS,862
essential graph, 915

ePCA,947
epsilon
EPSR,859
erf,38
ESS,862
estimated potential scale reduction, 859
estimator, 191
Euclidean distance, 18evidence, 156, 173
evidence procedure, 173,238, 746
evolutionary MCMC, 429exchangeable, 321, 963
exclusive or, 486
expectation correction, 658
expectation maximization, 349
expectation proagation, 735Expectation propagation, 787
expectation propagation, 525expected complete data log likelihood, 350, 351
expected proﬁt, 330
expected sufficient statistics, 350, 359, 619
expected value, 33
explaining away, 326
explicit duration HMM, 622

exploration-exploitation, 184

INDEXTOKEYWORDS 1055
exploratory data analysis, 7
exponential cooling schedule, 870
Exponential distribution, 42
exponential family, 115, 253, 281, 282, 290, 347
exponential family harmonium, 985
exponential family PCA, 947
exponential loss, 556
exponential power distribution, 458
extended Kalman ﬁlter, 648
extension, 67
external ﬁeld, 668
F score,183
F1 score,183, 699
FA,381
face detection, 8
face detector, 555
face recognition, 8
Facebook, 974factor,665
factor analysis, 381, 402, 931, 947
factor analysis distance, 520
factor graph, 769, 769, 771, 888
factor loading matrix, 381
factorial HMM, 628
factorial prior, 463
factors,382
faithful,936
false alarm, 30,180
false alarm rate, 181
false discovery rate, 184
false negative, 180
false positive, 30,180
false positive rate, 181
family,309
family marginal, 359
fan-in,313
fantasy data, 990
farthest point clustering, 355
fast Fourier transform, 717, 775fast Gauss transform, 524fast ICA,411
fast iterative shrinkage thesholding algorithm, 446
FastSLAM, 635, 835

fat hand, 933
fault diagnosis, 659
feature construction, 564
feature extraction, 6,564
feature function, 667
fea
ture induction, 680
feature maps, 565
feature matrix, 875feature selection, 86
feature-based clustering, 875
features,2,3 ,412
feedback loops, 929
feedforward neural network, 563
ferro-magnets, 668
FFT, 775ﬁelds of experts, 473
ﬁll-in edges, 719
Filtering, 607
ﬁltering,87
ﬁnite difference matrix, 113
ﬁnite mixture model, 879ﬁrst-order logic, 674
Fisher information, 166
Fisher information matrix, 152, 193, 293

fea
Fisher kernel, 485
Fisher scoring method, 293Fisher’s linear discriminant analysis, 271
FISTA,446
ﬁt-predict cycle, 206
ﬁxed effect, 298
Fixed lag smoothing, 608
ﬁxed point, 139ﬂat clustering, 875
FLDA,271
ﬂow cytometry, 936folds,24
forest,310,912
forward stagewise additive modeling, 557
forward stagewise linear regression, 562
forwards KL, 733forwards model, 345
forwards selection, 428
forwards-backwards, 644, 688, 707, 720forwards-backwards algorithm, 428,611
founder model, 317
founder variables, 385

FISTA,446
FLDA,271
Fourier basis, 472fraction of variance explained, 400
free energy, 988
free-form optimization, 737
frequent itemset mining, 15
frequentist, 27, 149
frequentist statistics, 191
Frobenius norm, 388
frustrated, 868
frustrated system, 668
full, 46full
conditional, 328,838
function approximation, 3
functional data analysis, 124
functional gradient descent, 561
furthest neighbor clustering, 897
fused lasso, 454
fuzzy clustering, 973
fuzzy set theory, 65
g-prior,236, 425

game against nature, 176game theory, 176Gamma, 623gamma distribution, 41
gamma function, 42GaP,949
gap statistic, 372
gating function, 342
Gauss-Seidel, 710Gaussian, 20,3 8
Gaussian approximation, 255, 731
Gaussian Bayes net, 318
Gaussian copulas, 942
Gaussian graphical models, 725Gaussian kernel, 480,507, 517
Gaussian mixture model, 339
Gaussian MRF, 672
Gaussian process, 483, 505, 509, 512, 882Gaussian processes, 515
Gaussian random ﬁelds, 938
Gaussian RBM, 986
Gaussian scale mixture, 359,447, 505

Gaussian sum ﬁlter, 656
GDA, 101GEE,300
GEM,369
Gene ﬁnding, 606
gene ﬁnding, 622gene knockout experiment, 931

1056 INDEXES
gene microarrays, 421
generalization, 3
generalization error, 23,180
generalization gradient, 66
generalize, 3
generalized additive model, 552
generalized belief propagation, 785
generalized cross validation, 207
generalized eigenvalue, 274
generalized EM, 361,369
generalized estimating equations, 300
generalized linear mixed effects model, 298
generalized linear model, 281, 290
generalized linear models, 281generalized pseudo Bayes ﬁlter, 657
generalized t distribution, 461
generate and test, 853
generative approach, 245generative classiﬁer, 30
generative pre-training, 999
generative weights, 410,986
genetic algorithms, 348, 720, 921genetic linkage analysis, 315, 318
genome, 318genotype, 317
geometric distribution, 622
Gibbs distribution, 290,666
Gibbs sampler, 672Gibbs sampling, 328, 669, 736, 838
Gini index, 548
gist, 963Gittins Indices, 184
Glasso,940
Glauber dynamics, 838
GLM,290, 654
GLMM,298
glmnet, 442global balance equations, 597
global convergence, 248
global localization, 828

GLMM,298
global Markov property, 661
global minimum, 222global prior parameter independence, 916
globally normalized, 686
GM,308
GMM,339
GP-LVM,540
GPs,515
GPUs, 1006gradient boosting, 560
gradient descent, 247, 445
Gram matrix, 481
gr
ammars, 689, 1007
grandmother cells, 984, 1005
graph,309
graph cuts, 890
graph Laplacian, 891
graph surgery, 932
graph-guided fused lasso, 454
graphcuts, 801
graphical lasso, 940
graphical model, 308, 311
graphical models, xxviii, 13, 31, 32, 308, 337, 909Gray code, 422

GM,308
GMM,339
GPs,515
gr
graph,309
greatest common divisor, 598
greedy equivalence search, 936
ground network, 676
ground states, 668
group lasso, 450, 579, 942
grouping effect, 456Gumbel, 295
Hadamard product, 609
Haldane prior, 166
ham, 5
Hamiltonian MCMC, 868
Hammersley-Clifford, 666
hamming distance, 876
handwriting recognition, 7
haplotype, 317
hard clustering, 340
hard EM, 352
hard thresholding, 434, 435
harmonic mean, 183harmonium, 983
Hastings correction, 849
hat matrix, 221
HDI,154
heat bath, 838
heavy ball method, 249
heavy tails, 43,223

ham, 5
HDI,154
Hellinger distance, 735
Helmholtz free energy, 733
Hessian, 193, 852heteroscedastic LDA, 275
heuristics, 727
hidden,10, 349
hidden layer, 563
hidden Markov model, 312,603, 963
hidden nodes, 313
hidden semi-Markov model, 622
hidden units, 564
hidden variable, 312,924
hidden
variables, 319, 914
hierarchical adaptive lasso, 458
hierarchical Bayesian model, 171
hierarchical Bayesian models, 347hierarchical clustering, 875,893
hierarchical Dirichlet process, 621hierarchical HMM, 624
hierarchical latent class model, 926

hidden
hierarchical mixture of experts, 344, 551
high throughput, 184, 421high variance estimators, 550highest density interval, 154
highest posterior density, 153
hill climbing, 920hindsight, 607
hinge loss, 211, 477,499
Hinton diagram, 592Hinton diagrams, 399
histogram, 508
hit rate,181
HMM,312,603
HMM ﬁlter, 640HMMs, 685Hoeffding’s inequality, 209
homogeneous, 589
homotopy, 442
Hopﬁeld network, 568,669
horizon,608
Horn clauses, 676
HPD,153
HSMM,622
Huber loss, 224, 561
Hugin,722
Hungarian algorithm, 659, 810
hybrid MCMC, 868

HPD,153
HSMM,622
Hugin,722
hybrid Monte Carlo, 584hybrid systems, 655

INDEXTOKEYWORDS 1057
hyper-parameters, 74
hypothesis space, 66
I-map,324
I-projection, 733
ICA, 385, 409
ID3,545
IDA,936
identiﬁable, 346
identiﬁable in the limit, 70
iff,68
iid,51,218, 320
ill-conditioned, 106, 129
image classiﬁcation, 7
image compression, 355image denoising, 473
image inpainting, 14,473
image segmentation, 671
image tagging, 968
IMM,658
implicit feedback, 983
importance sampling, 820
importance weights, 821
impression log, 983
improper prior, 166, 168
imputation, 14
Imputation Posterior, 847
in-degree, 310
inclusion probabilities, 423
incremental EM, 365, 366
independence sampler, 848
independent and identically distributed, 51
independent component analysis, 409
indicator function, 17,28, 976
induced width, 719
induction, 66,77
inductive bias, 19, 582
infer.net, 799
inference, 320inﬁnite hidden relational model, 977
inﬁnite HMM, 621inﬁnite mixture models, 841, 879
inﬁnite relational model, 903, 973, 976
inﬂuence diagram, 328, 932
inﬂuence model, 628
infomax, 416

information, 27information arc, 329,331
inf
ormation bottleneck, 405
information extraction, 688
information ﬁlter, 642
information form, 115, 305,672, 711, 725
information gain, 547
Information inequality, 58
information projection, 733
information retrieval, 183, 300, 953information theory, 56
inheritance model, 317
inner approximation, 779
innovation, 641
inside outside, 624inside-outside algorithm, 689
instance-based learning, 17
integrate out, 156integrated likelihood, 156
integrated risk, 195

inf
intensive care unit, 313inter-causal reasoning, 326
interaction effects, 421interactive multiple models, 658interest point detector, 484interpolate, 112
interpolated Kneser-Ney, 595
interpolator, 517
interval censored, 379
interventional data, 936
interventions, 931intrinsic Gaussian random ﬁeld, 113invariant, 8,854
invariant distribution, 597
invariant features, 1004
inverse chi-squared distribution, 131
inverse Gamma, 130inverse gamma, 42
inverse Gaussian, 448
inverse probability transform, 815

inverse problem, 317
inverse problems, 344
inverse reinforcement learning, 186
inverse Wishart, 126, 128
inverted index, 600
inverted indices, 1004IP, 847IPF,682
iris, 6, 548IRLS,251
IRM, 976irreducible, 598
Ising model, 668
isotropic, 46
iterated EKF, 650
iterative conditional modes, 669,804,
 929
iterative proportional ﬁtting, 682, 939
iterative scaling, 683
iterative shrinkage and thresholding algorithm, 445iterative soft thresholding, 445
iteratively reweighted least squares, 251

 929
Jacobi, 710, 773Jacobian, 151, 648, 649Jacobian matrix, 50
JAGS,847
JamBayes, 13James Stein estimator, 174James-Stein estimator, 173, 199JC Penney, 603Jeffreys prior, 166
Jeffreys-Lindley paradox, 165
Jensen’s inequality, 58, 363
Jensen-Shannon divergence, 57
Jeopardy, 4jittered,486
JJ bound, 761
joint distribution, 29,307
joint probability distribution, 44
JTA,720
jump Markov linear system, 655
junction tree, 722
junction tree algorithm, 720, 731
junction trees, 635
K-centers, 887
K-means algorithm, 352
k-means++, 355

JAGS,847
JTA,720
K-medoids algorothm, 490
k-spectrum kernel, 484
K2 algorithm, 920
Kalman ﬁlter, 122, 267, 632, 633, 640, 643
Kalman gain matrix, 637, 641
Kalman smoother, 633, 707

1058 INDEXES
Kalman smoothing, 644, 712, 963
Karhunen Loeve, 387
Karl Popper, 77
KDE,508, 510
Kendall’s τ,304
kernel, 565, 600,848
kernel density estimation, 127, 510kernel density estimator, 508
kernel function, 479, 515
kernel machine, 486
kernel PCA, 494, 540, 892
kernel regression, 511
kernel smoothing, 511
kernel trick, 488
kernelised feature vector, 486
Kikuchi free energy, 784
kinect, 551kinematic tracking, 344
kink,372
KL divergence, 57, 732
Kleene star, 483knee,372
KNN,16
knots,537
knowledge base, 676
knowledge discovery, 2,9
knowledge engineering, 313
Kolmogorov Smirnov, 864kriging,516
kronecker product, 253, 760
Kruskal’s algorithm, 912Kullback-Leibler divergence, 57
kurtosis,413, 415
L-BFGS,252
ℓ
0pseudo-norm, 424
ℓ0regularization, 426
ℓ1loss, 179
ℓ1regularization, 430
L1-Adaboost, 563
L1VM,488, 505
ℓ2loss, 179
ℓ2norm,218
ℓ2regularization, 226
L2boosting, 558
L2VM,488
label, 176label bias, 685
label switching, 341,841
label taxonomy, 689
labeled LDA, 953, 969
lag, 608Lagrange multiplier, 80

ℓ
L2VM,488
Lagrange multipliers, 289Lagrangian, 80, 289
Lagrangian relaxation, 808
Lanczos algorithm, 398language model, 300, 953
language modeling, 81, 568
language models, 591
Laplace, 223, 413, 429Laplace approximation, 255, 468
Laplace distribution, 41
Laplace’s rule of succession, 77
LAR,442, 562
large margin classiﬁer, 501
large margin principle, 259LARS,437,442, 558, 562
lasso,431, 470, 562, 936
latent,11latent class model, 926
latent CRF, 701latent Dirichlet allocation, 949, 950
latent factors, 11

latent semantic analysis, 12, 947
latent semantic indexing, 418,947
latent SVMs, 702
latent variable models, 337
lattice, 668Lauritzen-Spiegelhalter, 722
LBP,767
LDA,104, 927, 949, 950
LDA-HMM, 963
LDPC, 768LDS,631
leaf,309
leak node, 315
leapfrog steps, 868
learning, 320learning curve, 230
learning rate, 247
learning to learn, 296
learning to rank, 300
leastf
avorable prior, 197
least mean squares, 265,637
least squares, 219
least squares boosting, 428, 442, 558
leave one out cross validation, 207

LBP,767
leaf,309
leastf
leave-one out cross validation, 24
leaves, 895left censored, 379
left-to-right, 612left-to-right transition matrix, 590
LeNet5,566
leptokurtic, 413
LETOR,300
level sets, 47Levenberg Marquardt, 250
Levinson-Durbin, 627LG-SSM,631
likelihood, 319likelihood equivalence, 917
likelihood equivalent, 200likelihood principle, 214
likelihood ratio, 67,163
likelihood weighting, 822
limited memory BFGS, 252
limiting distribution, 598
line minimization, 248
line search, 248
linear discriminant analysis, 104

LETOR,300
linear dynamical system, 631
linear Gaussian, 318
linear Gaussian system, 119
linear kernel, 482
linear program, 224
linear programming relaxtion, 800
linear regression, 19
linear smoother, 533
linear threshold unit, 252
linear trend, 660linear-Gaussian CPD, 673linear-Gaussian SSM, 631
linearity of expectation, 49
linearly separable, 22, 252, 266
link farms, 601
link function, 291
LISREL,930
ListNet,302
LMS,265,637

INDEXTOKEYWORDS 1059
local consistency, 780
local evidence, 317,671
local level model, 637
local prior parameter independence, 917
local variational approximation, 756
localist encoding, 984
locally decodable, 811
locally normalized, 686,715
locally weighted regression, 512
LOESS,512
log partition function, 282
log-linear, 667
log-loss,210
log-odds ratio, 283
log-sum-exp, 86,757
logic sampling, 822
logical reasoning problems, 726
logistic,21, 295
logistic distribution, 413, 863
logistic normal, 402, 961
logistic regression, 21, 106
logit,21
logitBoost, 560
long tail, 2, 296
long tails, 43
LOOCV,24, 207
look-ahead RBPF, 832
loop,310
loopy belief propagation, 691, 767, 889
Lorentz,40
loss,176
loss function, 261loss matrix, 185
loss-augmented decoding, 699
loss-calibrated inference, 694
lossy compression, 354low density parity check, 768Low-level vision, 690
LOWESS, 512
LSA,947, 1003
lse,757
LSI,947
LVM,337
M step,350
M-projection, 733
M3nets,693
machine learning, 1
macro-a
veraged F1, 183

logit,21
loop,310
loss,176
lse,757
LSI,947
LVM,337
M step,350
macro-a
Mahalanobis distance, 98mammogram, 29
maneuvering target tracking, 832
manifest, 930
MAP estimate, 4, 178
MAR,270
margin, 563margin re-rescaling), 696
marginal distribution, 29
marginal likelihood, 156, 169
marginal polytope, 777
marginalizing out, 320
marginally independent, 30
marker,317
market basket analysis, 15
Markov,324
Markov assumption, 308
Markov blanket, 327,662, 736, 838
Markov chain, 308,589
Markov Chain Monte Carlo, 815Markov chain Monte Carlo, 52, 600, 837Markov decision process, 331

MAR,270
Markov equivalence, 936Markov equivalent, 915, 917
Markov logic network, 675
Markov mesh, 661
Markov model, 589
Markov models, 32Markov network, 661
Markov random ﬁeld, 661
Markov switching models, 604
MARS, 538, 553, 562
MART,562
master, 810matching pursuit, 562matching pursuits, 428
Matern kernel, 482
MATLAB, xxviiimatrix completion, 14,939
matrix determinant lemma, 118
matrix factorization, 948matrix inversion lemma, 118, 144, 641
matrix permanent, 669matrix tree theorem, 914max ﬂow/min cut, 801

MART,562
max margin Markov networks, 693
max pooling, 1005
max product linear programming, 810
max-product, 614,713
max-pr
oduct belief propagation, 800
maxent,289
maximal branching, 913
maximal clique, 310
maximal information coefficient, 60
maximal weight bipartite matching, 659maximizer of the posterior marginals, 612maximum a posteriori, 4
maximum entropy, 39, 104,289,667
maximum entropy classiﬁer, 252
maximum entropy Markov model, 685
maximum expected utility principle, 177
maximum likelihood estimate, 69

max-pr
maximum risk, 196
maximum weight spanning tree, 912MCAR,270
MCEM,368
MCMC, 52, 596, 600, 815, 837
MDL,162
MDP,331
MDS, 496mean,33
mean absolute deviation, 511
mean average precision, 303
mean ﬁeld, 735, 756, 767, 989
mean ﬁeld energy functional, 779
mean function, 291
mean precision, 182mean reciprocal rank, 303
mean squared error, 205,218
Mechanical Turk, 10, 995median,33
median model, 423
MEMM,685
memory-based learning, 17
Mendelian inheritance, 317
Mercer kernel, 481
Mercer’s theorem, 481, 539

MCEM,368
MDL,162
MDP,331
MEMM,685
message passing, 644, 800metric, 691, 691, 803
metric CRF, 691
metric MRF, 803

1060 INDEXES
Metropolis Hastings, 848, 922
Metropolis-Hastings algorithm, 869
MFCC, 1005MH,848
MI,59
micro-averaged F1, 183
Microsoft, 983mini-batch, 264, 571
minimal, 282
minimal I-map, 324
minimax rule, 196
minimum description length, 162
minimum entropy prior, 621
minimum mean squared error, 179
minimum spanning tree, 897minorize-maximize, 369
misclassiﬁcation loss, 176Misclassiﬁcation rate, 547
misclassiﬁcation rate, 22,205
missed detection, 180
missing, 15missing at random, 270, 372, 982
missing completely at random, 270
missing data, 14, 914, 974missing data problem, 269
mixed directed graphs, 931mixed membership model, 950
mixed membership stochastic block model, 973
mixed model, 298
mixing matrix, 408
mixing time, 857
mixing weights, 169, 338
mixture,72
mixture density network, 344
mixture model, 164, 338
mixture of conjugate priors, 169
mixture of experts, 342, 563, 973, 984
mixture of factor analysers, 386
mixture of Gaussians, 339
mixture of Kalman ﬁlters, 831
mixture of trees, 914

mixture proposal, 853
MLE,69
MLP,563
MM,369
MMSE,179
MNIST,7, 341
Mobious numbers, 784
mode,4
model based clustering, 11
model selection, 10, 24,156
model selection consistent, 439
model-based appr
oach, xxvii
model-based clustering, 879
moderated output, 260
modularity, xxviiiMoE, 342moment matching, 176, 287,653, 658, 677
moment parameters, 115
moment projection, 733
momentum, 248
monks, 974Monte Carlo, 52, 151, 192, 258, 815
Monte Carlo EM, 368
Monte Carlo integration, 53Monte Carlo localization, 828

MLE,69
MLP,563
MM,369
MMSE,179
mode,4
moralization, 663,715
motes, 218motif,36
mPCA,948MPE,614
MPM,612
MRF,661
MSE,218
multi label classiﬁcation, 970multi net, 627
multi-armed bandit, 184
multi-class logistic regression, 104
multi-clust, 904
multi-grid techniques, 775multi-information, 415
multi-label classiﬁcation, 3, 405
multi-layer perceptron, 563, 999
multi-level model, 171
multi-level modeling, 844multi-stage, 186multi-target tracking, 659
multi-task feature selection, 297
multi-task learning, 172, 231,296, 449, 757

MPM,612
MRF,661
MSE,218
multiclass classiﬁcation, 3
multidimensional scaling, 496multinomial, 35
multinomial coefficient, 35
multinomial logistic regression, 104,252
multinomial PCA, 948, 951
multinomial probit, 295
multinomial regression LDA, 968
multinomial resampling, 826
multinoulli distribution, 35
multiple hypothesis testing, 184
multiple hypothesis tracking, 656
multiple imputation, 115
multiplek
ernel learning, 524, 543
multiple LDA, 276
multiple output model, 3
multiple random restarts, 348, 921

multiplek
multiple restarts, 620multivariate adaptive regression splines, 553
multivariate Bernoulli naive Bayes, 82
multivariate delta method, 763
multivariate Gamma function, 133multivariate gamma function, 126
multivariate Gaussian, 46,97, 339
multivariate normal, 46,97
multivariate probit, 295
multivariate Student t, 46
mutual information, 46, 59, 87, 547, 912
mutual inhibition, 564
mutually independent, 62MVN,46,97
N-best list, 616
n-gram, 568n-gram models, 591
Nadaraya-Watson, 511
naive Bayes classiﬁer, 82, 88, 311

naive Bayes classiﬁers, 32named entity extraction, 688
NaN,14
nats,56
natural exponential family, 282
natural gradient, 411natural parameters, 115,282
NDCG,304
nearest centroids classiﬁer, 102
nearest medoid classiﬁcation, 491
nearest neighbor, 16
nearest neighbor clustering, 897
nearest neighbor data association, 658
nearest shrunken centroids, 109

INDEXTOKEYWORDS 1061
negative binomial, 624
negative binomial distribution, 214
negative examples, 65negative log likelihood, 218,349
negative transfer, 297
negentropy, 415
neighbors, 309
neocognitron, 566
nested plate, 321
Nesterov’s method, 446
Netﬂix, 15, 580, 979, 981, 987, 993NETtalk,569
neural network, 302, 969neural networks, 344, 535neutral process, 882
Newton’s algorithm, 249, 251
NHST,213
NIW,133
NIX,136
NLL,218,349
NMAR,270
NMF,470,949
no forgetting, 331
no free lunch theorem, 24, 582
nodes,309
nodes that ﬁre together should wire together, 929
noise ﬂoor, 230
noisy-OR, 313, 928
nominal, 2
non-descendants, 327
non-factorial, 466
non-informative, 165
non-negative matrix factorization, 470,949
non-negative sparse coding, 470
non-null recurrent, 599
non-parametric Bayes, 879
non-parametric bootstrap, 192
non-parametric BP, 712
non-parametric model, 16
non-parametric prior, 879
non-serial dynamic programming, 717
non-smooth, 432
non-terminals, 689nonparanormal, 942
norm of a function, 539
normal,20,3 8

nominal, 2
normal equation, 220
normal Gamma, 476
normal inverse chi-squared, 136
Normal-in v
erse-wishart, 133
normalized cut, 891
normalized discounted cumulative gain, 304
normalized mutual information, 879
not missing at random, 270
noun phrase chunking, 687
NP-complete, 920NP-hard, 726ν-SVM classiﬁer, 502
nuisance variables, 320
null hypothesis, 163,213
null hypothesis signiﬁcance testing, 213
number game, 65
numerical underﬂow, 86
object detection, 8
object localization, 8
observation, 603
observation model, 312,631
observed data log likelihood, 348

observed information, 167observed information matrix, 193
Occam factor, 255
Occam’s razor, 67,156, 399, 400
occasionally dishonest casino, 606
occupancy grid, 828
Octave,xxviii
offline,261
oil wild-catter, 328
OLS,220
OMP,428
one-armed bandit, 184
one-hot encoding, 35
one-of-C encoding, 252one-shot decision problem, 186
one-standard error rule, 208
one-step-ahead predictive density, 609
one-versus-one, 503
one-versus-the-rest, 503
one-vs-all, 503
online EM, 365
online gradient descent, 262

OLS,220
OMP,428
online learning, 75, 241,261
ontological uncertainty, 973
ontology, 977
open class, 596, 688
Open Directory Project, 600, 689open universe, 676
optimal action, 177optimism of the training error, 206
optimiza
tion, 218
ordered Markov property, 310,327
ordinal, 295ordinal regression, 2,295, 301
ordinal variables, 876ordinary least squares, 220
Ornstein-Uhlenbeck process, 483
orthodox statistics, 191
orthogonal least squares, 427
orthogonal matching pursuits, 428
orthogonal projection, 221
out-degree, 310

optimiza
tion, 218
out-of-clique query, 722
outer approximation, 780
outliers, 179, 223
over-complete, 282, 1001
overcomplete, 469
overcounting number, 784
overdispersed, 859
overﬁt,22
overﬁtting, 72overrelaxed EM algorithm, 369
p-value, 138, 163, 163,213
PAC,210
PageRank, 301, 596, 600, 601
paired t-test, 137
pairwise independent, 62pairwise Markov property, 662
pairwise MRF, 666
parallel tempering, 858, 871, 922
parameter, 176parameter expansion, 736parameter modularity, 918
parameter sharing, 107
parameter tying, 107,171,589

overﬁt,22
PAC,210
parametric bootstrap, 192
parametric model, 16,19
parents,309, 310
Pareto distribution, 43
part of speech, 605, 966

1062 INDEXES
Part of speech tagging, 605
partial dependence plot, 586
partial least squares, 406, 975
partially directed acyclic graph, 915
partially labeled LDA, 969
partially observed Markov decision process, 331
partially observed MRF, 672
Particle ﬁltering, 823
particle ﬁltering, 267, 648, 823, 887
partition function, 282,666
partitional clustering, 875
partitioned inverse formula, 116
partitioning, 841partitions of the integers, 885
Parzen window density estimator, 508
passing a ﬂow, 724
path,310
path diagrams, 929
pathologies, 211
pattern,915
pattern completion, 669
pattern recognition, 2
pattern search, 736, 783PCA,12,387, 493, 947
PCFG,689
PDAG, 936pdf,32
pedigree graph, 315
peeling algorithm, 715
Pegasos,701
penalized least squares, 226
penalized log likelihood, 161
penalized splines, 537
penetrance model, 317
perception-action, 331
perceptron, 569perceptron algorithm, 266
perceptual aliasing, 828
perfect intervention, 931
perfect map, 664
period,598
permanent, 942perplexity, 953, 953, 992

PCFG,689
persistent CD, 991
persistent contrastive divergence, 680personalized recommendation, 77personalized spam ﬁltering, 296
perturbation theory, 892phase,317
phase transition, 671, 857
phenotypes, 317phone, 624phonemes, 1005phylogenetic HMM, 317
phylogenetic tree, 925piecewise polynomial, 537
pilo
t runs,851
pipeline,687
Pitman-Koopman-Darmois theorem, 286
Pitman-Yor process, 885
Plackett-Luce, 302
plates,321
platykurtic, 413
PLS,406
PLSI,949
plug-in, 147plug-in approximation, 72
plutocracies, 43pmf,28

pilo
t runs,851
PLS,406
PLSI,949
PMTK,xxviiipoint estimate, 149, 150
pointwise approach, 301
pointwise marginal credibility intervals, 114
pointwise mutual information, 59
Poisson,37
poisson regression, 292
polar, 51policy,177
Polya urn, 89,884
Polyak-Ruppert averaging, 263
polynomial kernel, 481
polynomial regression, 20
polynomial time approximation schemes, 728polysemy, 951
polytree,310
POMDP,331
pooled, 171pooled empirical variance, 108
population minimizer, 556
positive deﬁnite, 125, 222
positive deﬁnite kernel, 481

POMDP,331
positive examples, 65posterior expected loss, 177
posterior mean, 179posterior median, 179posterior mode, 178posterior predictive density, 608posterior predictive distribution, 66, 71, 234
potential function, 665
Potts model, 671, 856
power law, 43
power method, 603
PPCA,381,387
precision, 38,182
precision at k, 303, 702
precision matrix, 46, 100
precision
recall curve, 182
predict-update cycle, 609
predict-update-project, 653
predictive, 2
preferences, 185
preposterior risk, 195
prevalence, 183

precision
Prim’s algorithm, 912primal variables, 492, 499
principal component, 388principal components, 1000principal components analysis, 12,387
principal components regression, 230
principle of insufficient reason, 58
probabilistic decision tree, 551probabilistic expert system, 313
probabilistic inference, 319
probabilistic latent semantic indexing, 949
probabilistic matrix factorization, 337, 980
probabilistic PCA, 387
probabilistic principal components analysis, 381
probabilistic relational modeling, 675,976

probability density function, 32
probability mass function, 28
probability of the evidence, 319,609,717
probability product kernel, 485
probability simplex, 47,7 9
probability theory, xxvii, 1probably approximately correct, 210
probe,583
probit,260, 655
probit regression, 293, 362, 380, 795, 864
product of experts, 983
product rule, 29

INDEXTOKEYWORDS 1063
production rules, 689
proﬁle HMM, 606
proﬁle log likelihood, 401
projected gradient descent, 444, 445
projection, 262projection pursuit, 415
Prolog, 676proposal distribution, 817,848, 869
propose, 848prosecutor’s fallacy, 61
Protein sequence alignment, 606
protein-protein interaction networks, 970prototype, 341
proximal operator, 443
pruning,549
pseudo counts, 75
pseudo likelihood, 678
pseudo marginals, 780
pseudo random number generator, 816
pseudo-likelihood, 943pure,546,548
purity,877
pushing sums inside products, 715
pyramid match kernel, 484
QALY,186
QMR,313
QP,431
qq-plot,260
QR decomposition, 228
quadratic discriminant analysis, 102
quadratic loss, 179
quadratic program, 431, 498, 499
quantile,33
quantize, 59
quartiles, 33
Quasi-Newton, 251
query logs, 301query variables, 320
quick medical reference, 313
radar, 658radial basis function, 480
Rand index, 878
random accelerations model, 633
random effects, 298
random effects mixture of experts, 969
random forests, 551, 554

QALY,186
QMR,313
QP,431
random probability measure, 880
random utility model, 294
random walk Metropolis algorithm, 848
random walk on the integers, 599random walk proposal, 869Rank correlation, 304
rank one update, 118
ranking,87,601,702
RankNe
t,302
Rao-Blackwell, 841Rao-Blackwellisation, 841
Rao-Blackwellised particle ﬁltering, 831
Rao-Blackwellized particle ﬁltering, 659rare event, 182, 820
rate,355
rational behavior, 177
RBF,480
RBF kernel, 517RBF network, 486
RBM,983, 996
RBPF,831
real AdaBoost, 559recall,181, 182

RankNe
t,302
rate,355
RBF,480
RBPF,831
receiver operating characteristic, 181
receptive ﬁelds, 565
recognition weights, 410,986
recombination model, 317
reconstruction error, 354,387
recurrent, 599
recurrent neural network, 568,669
recurrent neural networks, 591recursive, 929
recursive least squares, 265, 636
reﬂecting pair, 553
regime switching, 660
regime switching Markov model, 626
regression, 2
regression spline, 537
regret,262
regular,598
regularization, 227
regularization path, 436, 442, 562
regularized discriminant analysis, 107

regularized estimation, 130
regularized particle ﬁlter, 827
regularized risk minimization, 206
reinforcement learning, 2, 186
reject action, 178
rejection sampling, 817
rejuvenation, 825
relation,975
relational probabilistic models, 676relational topic model, 974
relative entropy, 57
relative importance of predictor variables, 586
rela
tive risk,531
relevance network, 908
relevance vector machine, 463, 488
Rephil,928
replicated softmax model, 992
representer theorem, 539
reproducing kernel Hilbert space, 539

rela
reproducing property, 539
rerank, 616resample-move, 827
residual,641
residual analysis, 260
residual belief propagation, 774
residual error, 19
residual resampling, 826
residual sum of squares, 218
response variable, 2
responsibility, 340,351
restricted Boltzmann machine, 983
reverse KL, 733reversible jump MCMC, 370, 399, 855
reward, 2Ricatti equations, 642
rich get richer, 755,885
ridge regression, 203, 226
right censored, 379
risk,195, 261
risk averse, 4, 178
RJMCMC, 855
RKHS,539
RLS,636
Robbins-Monro, 263, 366, 701

RKHS,539
RLS,636
robust, 179robust priors, 168
robustness, 223
ROC,181
rocking,261

1064 INDEXES
root,309, 895
root mean square error, 979
Rosenblatt, 266rotamers, 690RTS smoother, 644
rule of iterated expectation, 141
rule of total probability, 29
rules,550
RUM,294
running intersection property, 722
RVM,488, 505
saddle point approximation, 255
sample impoverishment, 826
sample standard deviation, 136
samples, 52sampling distribution, 191, 191
sampling importance resampling, 823
sampling period, 633
satisfying assignment, 727saturated model, 428
SBL,463
scalar product, 19
scale invariant prior, 168
scale of evidence, 163scatter plot, 6
SCFGs, 624schedule, 263
Schur complement, 116
scientiﬁc method, 71scope,328
score function, 167,193
score matching, 1001score vector, 485
scores,382
scree plot, 400
screening, 87
search engine optimization, 603
second order, 249
second order Markov chain, 312
second-order Markov model, 591self loops, 309
semantic hashing, 1003
semantic network, 977semantic role labeling, 576semi-conjugate, 132
semi-continuous HMM, 630
semi-Markov model, 622
semi-metric, 691

semi-parametric model, 298,524
semi-supervised, 405semi-supervised embedding, 576
semi-supervised learning, 268, 270
sensible PCA, 387
sensitivity, 29,181
sensitivity analy
sis,166
sensor fusion, 122
sentiment analysis, 967
separating set, 723
separation oracle, 699
sequence logo, 36
sequential, 186sequential minimal optimization, 499
sequential TRBP, 801
SGD,262
Shafer-Shenoy, 722
shallow parsing, 687
shared,103
Sherman-Morrison-Woodbury formula, 118
shooting, 441, 940shrinkage, 122,174,230,557

sis,166
SGD,262
shrinkage estimation, 130
shrinkage factor, 437
side chains, 690side information, 982
SIFT, 484sifting property, 39
sigma points, 650, 651
sigmoid,21, 105
sigmoid belief net, 313,996
sigmoid belief nets, 763sigmoid kernel, 482
signal detection theory, 106signal processing, 421signal-to-noise ratio, 122
signal-to-symbol, 1007
similar, 66, 875similarity-based clustering, 875
simple cells, 413Simple linear regression, 241
simplex factor model, 949
Simpon’s paradox, 933
Simulated annealing, 869

simulated annealing, 262, 348, 853, 921
simulation based, 823
simultaneous localization and mapping, 635
single best replacement, 427
single link clustering, 897
single site updating, 847
singular value decomposition, 392
singular values, 392
SIR, 823size principle, 67
skewness, 413
skip arcs, 568
skip-chain CRF
,688
slack re-scaling, 696
slack variables, 498
SLAM,635, 834
slaves, 810slice sampling, 865
sliding window detector, 8
slippage, 635slot machine, 184smallN, largeD,421
SmartASS, 4SML,680
SMO,499
Smoothing, 607

,688
SMO,499
smoothing kernel, 507, 507
Smoothing splines, 536
social networks, 970soft clustering, 340,973
soft margin constraints, 501
soft thresholding, 434, 435
soft weight sharing, 575
softmax,104, 283
source coding, 56
SpAM,553
spam, 5spanning tree polytope, 786
SpaRSA, 445sparse,15,421, 621, 945, 979
sparse Bayesian learning, 463
sparse boosting, 562
sparse coding, 469
sparse data problem, 77
sparse kernel machine, 421sparse matrix factorization, 469, 470
sparse PCA, 469

INDEXTOKEYWORDS 1065
sparse representation, 421
sparse vector machine, 488
sparsity, 41sparsity-promoting prior, 297spectral,445
spectral clustering, 891
spectral graph theory, 891
speech recognition, 590, 1005sphereing, 142
spherical, 46
spike and slab, 424
spin, 668spline, 298split merge, 621
split variable, 224
square root ﬁlter, 642
squared error, 179
squared exponential kernel, 480,517
squared loss, 176squashing function, 21
SSM,631
SSVMs,693
stability selection, 439
stable,936
stacked denoising auto-encoder, 1001stacking, 580
standard deviation, 34
standard error, 56
standard error of the mean, 137, 208
standard errors, 194
standard model, 995
standard normal, 38
standard overcomplete representation, 776
standardized, 352Standardizing, 142
state, 176state estimation, 313
state space, 28
state space model, 631
state transition diagram, 590, 606
state transition matrix, 308
stationary, 589,631
stationary distribution, 596, 597
statistical learning theory, 209
statistical relational AI, 675

statistical relational learning, 976
statistically signiﬁcant, 213
steepest descent, 247,264
Stein’s paradox, 199
stemming, 81
step size, 247
stepping out, 866
stepwise EM, 365
stick
-breaking construction, 883
sticky,850
stochastic algorithm, 869stochastic approximation, 368stochastic approximation EM, 368
stochastic automaton, 590
stochastic block model, 972
stochastic context free grammars, 624stochastic EM, 368
stochastic gradient boosting, 584
stochastic gradient descent, 262, 570, 868, 981, 987
stochastic matrix, 307,589

stick
stochastic maximum likelihood, 680, 990
stochastic optimization, 262
stochastic process, 953stochastic processes, 589
stochastic search, 429stochastic volatility, 831
stop words, 81,480, 952
stopping rule, 214
stratiﬁed CV, 206
stratiﬁed sampling, 826
streaming data, 261
StreetView, 8strict, 197strictly convex, 222
string kernel, 483
strong local optimum, 804
strong sampling assumption, 67
structural EM, 925
structural equation model, 929
structural equation models, 674structural error, 230

structural risk minimization, 206
structural signatures, 926
structural support vector machines, 693
structural time series, 637
structural zeros, 672
structure learning, 621, 681
structured mean ﬁeld, 740
structured output, 684
structured perceptron algorithm, 700
structured-output classiﬁcation problems, 266Student t, 359Studenttdistribution, 39
sub-Gaussian, 413
subderivative, 432
subdifferential, 432
subgradient, 432, 432
subgraph, 310
subjective, 67
probability, 310
submodular, 802
subsampling, 566

subspace method, 647
sufficiency principle, 214
sufficient statistics, 74, 79, 281, 282, 348
suffix trees, 483sum of squared errors, 218
sum of squares, 220
sum rule, 29
sum-product, 614,709
sum-product algorithm, 707
super efficient, 820
super-Gaussian, 413
supermodular, 802
supervised LDA, 967
supervised learning, 2
supervised PCA, 405
support,426
support vector machine, 488,496, 569
support vector machines, 211support vectors, 496,498, 499
surrogate loss, 304surrogate loss function, 211
surrogate splits, 550

survival of the ﬁttest, 825
suspicious coincidence, 164suspicious coincidences, 67
SVD, 107, 228, 392, 980
SVM, 211, 488,496
SVMstruct, 698, 700
Swendsen Wang, 866
switching linear dynamical system, 655, 831
switching state space model, 655
symbol grounding, 1007
symmetric, 849

1066 INDEXES
synchronous updates, 773
syntactic sugar, 321
synthesis view, 387
systematic resampling, 826
systems biology, 13
systems identiﬁcation, 646
systolic array, 710
t statistic, 137
t-test,137
tabula rasa, 165
tail area probabilities, 33
tail area probability, 213
TAN,312, 914
TASA, 951Taylor series, 255Taylor series expansion, 648Taylor’s theorem, 248temperature, 104
template, 676
template matching, 543
tensor product, 553tensor product basis, 538
terminals, 689test statistic, 163,213
TF-IDF,480
thin junction tree ﬁlter, 635thin junction trees, 944
thin plate spline, 538
thin SVD, 392
thinning, 862
Thompson sampling, 185
tied,103,565, 997
tied-mixture HMM, 630
Tikhonov regularization, 124
time reversible, 599
time-invariant, 589
time-series forecasting, 637, 673Tobit model, 379
Toeplitz, 627tokens,945
topic,946,951
topic model, 757topological ordering, 310, 310
total ordering, 310
trace,99
trace plot, 859
trace trick, 99
traceback, 614,717
tracking, 823tracking by detection, 830

trace,99
tractable substructure, 739
trail,310
training set, 2
tr
ans-dimensional MCMC, 855
transfer function, 563,570
transfer learning, 296
transient, 599
transition matrix, 589, 590
transition model, 312,631
translation invariance, 565,1004
translation invariant, 472translation invariant prior, 167
TRBP,787
TRBP-S,801
tree,310
tree EP,793
tree reparameterization, 774
tree reweighted belief propagation, 786
tree-augmented naive Bayes classiﬁer, 312treewidth, 320, 719, 800
trellis, 614trellis diagram, 612

trail,310
tr
TRBP,787
tree,310
tri-cube kernel, 508
triangle inequality, 352, 875
triangulated, 722
tridiagonal, 114trigram model, 591
true positive rate, 181
TrueSkill, 654, 793
truncated Gaussian, 362
truncated Gaussian potential, 691
truncated Newton, 250
truncated SVD, 393
TRW,787
TRW-S,801
tube,497
tuples, 975turbo codes, 768two-ﬁlter smoothing, 646
two-slice marginal, 611
type I, 213type I error rate, 181
type II maximum likelihood, 157
type-II maximum likelihood, 173
U-shaped curve, 23
UCB,185
UGM,661
UKF,650
unbiased, 200
uncertainty, 27

TRW,787
TRW-S,801
tube,497
UCB,185
UGM,661
UKF,650
unclamped phase, 988
unclamped term, 677
unconditionally independent, 30
under
ﬁts,23
undirected, 309
undirected graphical model, 661
undirected local Markov property, 662
unfaithful, 663
unidentiﬁable, 200, 278, 841Uniﬁed Medical Language System, 977uniform distribution, 32
unigram statistics, 591
unigrams, 953uninformative, 165
union bound, 209
unit information prior, 236
universal approximator, 564
unk,81,596
unknown, 15, 81unrolled, 321
unscented Kalman ﬁlter, 523, 650
unscented particle ﬁlter, 828

under
ﬁts,23
unscented transform, 650
unstable, 550
unsupervised learning, 2,9, 337
up-down, 998
user rating proﬁle, 949
utilities, 294utility function, 177
utility nodes, 328
v-structure, 324, 326
validation set, 23
value nodes, 328
value of perfect information, 331
vanishing gradient, 999
Vapnik-Chervonenkis, 210
VAR,673

INDEXTOKEYWORDS 1067
variable duration HMM, 622
variable elimination, 318, 331, 715
variance, 33
variance stabilizing transform, 175
variation of information, 879
variational Bayes, 742
variational Bayes EM, 620, 750, 923
variational EM, 368
variational free energy, 733
variational inference, 281, 318, 731
variational message passing, 756
varimax,385, 410
VB,742
VBEM,750
VC,210
VC dimension, 206
vector auto-regressive, 673
vector quantization, 354
version space, 67
vertices,309
VIBES,756
views,904
visible, 349visible nodes, 313
visible variables, 319
visual words, 1007
visualizing, 12
Viterbi,612, 701
Viterbi decoding, 608
Viterbi training, 620
VMP,756
Voronoi tessellation, 18
VQ,354
Wald, 448Wald interval, 212
warm starting, 442
WARP,304
Watson, 4wavelet, 469wavelet transforms, 413weak conditionality, 215
weak learner, 554
weak marginalization, 658
web crawling, 600
web spam, 603
weight decay, 226,572, 987
weight function, 533
weight vector, 19
weighted approximate-rank pairwise, 304

views,904
VMP,756
VQ,354
WARP,304
w e i g h t e da v e r a g e ,7 1weighted least squares, 358weighted least squares problem, 251
Whitening,142
w
hitening, 410
Widrow-Hoff rule, 265
Wishart,125
working response, 250
World Health Organization, 60wrapper method, 427
Xbox, 654, 795xor,486
Zellner’s g-prior, 405zero avoiding, 733
zero count problem, 77
zero forcing, 733
zero temperature limit, 800
zig-zag,248Zipf’s law, 43

Information Science and Statistics
Series Editors:
M. Jordan
J. Kleinberg
B. Scho ¨lkopf

data/pdfs/Pattern Recognition and Machine Learning (Information Science and Statistics) .pdf

Information Science and Statistics 
Akaike and Kitagawa: The Practice of Time Series Analysis. 
Bishop:   Pattern Recognition and Machine Learning. 
Cowell, Dawid, Lauritzen, and Spiegelhalter:  Probabilistic Networks and
Expert Systems. 
Doucet, de Freitas, and Gordon:  Sequential Monte Carlo Methods in Practice. 
Fine:  Feedforward Neural Network Methodology. 
Hawkins and Olwell: Cumulative Sum Charts and Charting for Quality Improvement. 
Jensen: Bayesian Networks and Decision Graphs. 
Marchette:  Computer Intrusion Detection and Network Monitoring:
A Statistical Viewpoint. 
Rubinstein and Kroese: The Cross-Entropy Method:  A Unified Approach to 
Combinatorial Optimization, Monte Carlo Simulation, and Machine Learning. 
Studený:  Probabilistic Conditional Independence Structures .
Vapnik:  The Nature of Statistical Learning Theory, Second Edition.  
Wallace:  Statistical and Inductive Inference by Minimum Massage Length.

Christopher M. Bishop F.R.Eng.
Assistant Director
Microsoft Research Ltd
Cambridge CB3 0FB, U.K.
cmbishop@microsoft.com
http://research.microsoft.com/ /H11011cmbishop
Series Editors
Michael Jordan
Department of Computer
Science and Department
of Statistics
University of California,
Berkeley
Berkeley, CA 94720
USAProfessor Jon Kleinberg
Science
Cornell University
Ithaca, NY 14853
USABernhard Scho ¨lkopf
Max Planck Institute for
Biological Cybernetics
Spemannstrasse 38
72076 Tu ¨bingen
Germany
Library of Congress Control Number: 2006922522
ISBN-10: 0-387-31073-8
ISBN-13: 978-0387-31073-2
Printed on acid-free paper.
©2006 Springer Science +Business Media, LLC
All rights reserved. This work may not be translated or copied in whole or in part without the written permission of the publisher
(Springer Science +Business Media, LLC, 233 Spring Street, New York, NY 10013, USA), except for brief excerpts in connection

Germany
with reviews or scholarly analysis. Use in connection with any form of information storage and retrieval, electronic adaptation,
computer software, or by similar or dissimilar methodology now known or hereafter developed is forbidden.
The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such,
is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights.
Printed in Singapore. (KYO)
987654321
springer.com

This book is dedicated to my family:
Jenna, Mark, and Hugh
Total eclipse of the sun, Antalya, Turkey, 29 March 2006.

Preface
Pattern recognition has its origins in engineering, whereas machine learning grew
out of computer science. However, these activities can be viewed as two facets of
the same ﬁeld, and together they have undergone substantial development over the
past ten years. In particular, Bayesian methods have grown from a specialist niche to
become mainstream, while graphical models have emerged as a general framework
for describing and applying probabilistic models. Also, the practical applicability ofBayesian methods has been greatly enhanced through the development of a range of
approximate inference algorithms such as variational Bayes and expectation propa-
gation. Similarly, new models based on kernels have had signiﬁcant impact on bothalgorithms and applications.
This new textbook reﬂects these recent developments while providing a compre-
hensive introduction to the ﬁelds of pattern recognition and machine learning. It isaimed at advanced undergraduates or ﬁrst year PhD students, as well as researchers

and practitioners, and assumes no previous knowledge of pattern recognition or ma-
chine learning concepts. Knowledge of multivariate calculus and basic linear algebrais required, and some familiarity with probabilities would be helpful though not es-
sential as the book includes a self-contained introduction to basic probability theory.
Because this book has broad scope, it is impossible to provide a complete list of
references, and in particular no attempt has been made to provide accurate historical

attribution of ideas. Instead, the aim has been to give references that offer greater
detail than is possible here and that hopefully provide entry points into what, in some
cases, is a very extensive literature. For this reason, the references are often to more
recent textbooks and review articles rather than to original sources.
The book is supported by a great deal of additional material, including lecture
slides as well as the complete set of ﬁgures used in the book, and the reader is

encouraged to visit the book web site for the latest information:
http://research.microsoft.com/ ∼cmbishop/PRML
vii

viii PREFACE
Exercises
The exercises that appear at the end of every chapter form an important com-
ponent of the book. Each exercise has been carefully chosen to reinforce concepts
explained in the text or to develop and generalize them in signiﬁcant ways, and each
is graded according to difﬁculty ranging from (⋆), which denotes a simple exercise
taking a few minutes to complete, through to (⋆⋆⋆ ), which denotes a signiﬁcantly
more complex exercise.
It has been difﬁcult to know to what extent these solutions should be made
widely available. Those engaged in self study will ﬁnd worked solutions very ben-
eﬁcial, whereas many course tutors request that solutions be available only via thepublisher so that the exercises may be used in class. In order to try to meet these
conﬂicting requirements, those exercises that help amplify key points in the text, or
that ﬁll in important details, have solutions that are available as a PDF ﬁle from thebook web site. Such exercises are denoted by

www . Solutions for the remaining
exercises are available to course tutors by contacting the publisher (contact details
are given on the book web site). Readers are strongly encouraged to work throughthe exercises unaided, and to turn to the solutions only as required.
Although this book focuses on concepts and principles, in a taught course the
students should ideally have the opportunity to experiment with some of the keyalgorithms using appropriate data sets. A companion volume (Bishop and Nabney,

2008) will deal with practical aspects of pattern recognition and machine learning,
and will be accompanied by Matlab software implementing most of the algorithmsdiscussed in this book.
Acknowledgements
First of all I would like to express my sincere thanks to Markus Svens ´en who
has provided immense help with preparation of ﬁgures and with the typesetting of
the book in L ATEX. His assistance has been invaluable.
I am very grateful to Microsoft Research for providing a highly stimulating re-

search environment and for giving me the freedom to write this book (the views and
opinions expressed in this book, however, are my own and are therefore not neces-sarily the same as those of Microsoft or its afﬁliates).
Springer has provided excellent support throughout the ﬁnal stages of prepara-
tion of this book, and I would like to thank my commissioning editor John Kimmelfor his support and professionalism, as well as Joseph Piliero for his help in design-

ing the cover and the text format and MaryAnn Brickner for her numerous contribu-
tions during the production phase. The inspiration for the cover design came from adiscussion with Antonio Criminisi.
I also wish to thank Oxford University Press for permission to reproduce ex-
cerpts from an earlier textbook, Neural Networks for Pattern Recognition (Bishop,
1995a). The images of the Mark 1 perceptron and of Frank Rosenblatt are repro-

duced with the permission of Arvin Calspan Advanced Technology Center. I wouldalso like to thank Asela Gunawardana for plotting the spectrogram in Figure 13.1,
and Bernhard Sch ¨olkopf for permission to use his kernel PCA code to plot Fig-
ure 12.17.

PREFACE ix
Many people have helped by proofreading draft material and providing com-
ments and suggestions, including Shivani Agarwal, C ´edric Archambeau, Arik Azran,
Andrew Blake, Hakan Cevikalp, Michael Fourman, Brendan Frey, Zoubin Ghahra-
mani, Thore Graepel, Katherine Heller, Ralf Herbrich, Geoffrey Hinton, Adam Jo-
hansen, Matthew Johnson, Michael Jordan, Eva Kalyvianaki, Anitha Kannan, JuliaLasserre, David Liu, Tom Minka, Ian Nabney, Tonatiuh Pena, Y uan Qi, Sam Roweis,
Balaji Sanjiya, Toby Sharp, Ana Costa e Silva, David Spiegelhalter, Jay Stokes, Tara
Symeonides, Martin Szummer, Marshall Tappen, Ilkay Ulusoy, Chris Williams, JohnWinn, and Andrew Zisserman.
Finally, I would like to thank my wife Jenna who has been hugely supportive
throughout the several years it has taken to write this book.
Chris Bishop
Cambridge
February 2006

Mathematical notation
I have tried to keep the mathematical content of the book to the minimum neces-
sary to achieve a proper understanding of the ﬁeld. However, this minimum level is
nonzero, and it should be emphasized that a good grasp of calculus, linear algebra,
and probability theory is essential for a clear understanding of modern pattern recog-
nition and machine learning techniques. Nevertheless, the emphasis in this book is
on conveying the underlying concepts rather than on mathematical rigour.
I have tried to use a consistent notation throughout the book, although at times
this means departing from some of the conventions used in the corresponding re-
search literature. V ectors are denoted by lower case bold Roman letters such asx, and all vectors are assumed to be column vectors. A superscript Tdenotes the
transpose of a matrix or vector, so that x
Twill be a row vector. Uppercase bold
roman letters, such as M, denote matrices. The notation (w1,...,w M)denotes a

row vector with Melements, while the corresponding column vector is written as
w=(w1,...,w M)T.
The notation [a, b]is used to denote the closed interval from atob, that is the
interval including the values aandbthemselves, while (a, b)denotes the correspond-
ing open interval, that is the interval excluding aandb. Similarly, [a, b)denotes an
interval that includes abut excludes b. For the most part, however, there will be
little need to dwell on such reﬁnements as whether the end points of an interval are
included or not.

TheM×Midentity matrix (also known as the unit matrix) is denoted IM,
which will be abbreviated to Iwhere there is no ambiguity about it dimensionality.
It has elements Iijthat equal 1ifi=jand0ifi̸=j.
A functional is denoted f[y]where y(x)is some function. The concept of a
functional is discussed in Appendix D.
The notation g(x)=O(f(x))denotes that |f(x)/g(x)|is bounded as x→∞ .
For instance if g(x)=3x2+2 , theng(x)=O(x2).
The expectation of a function f(x, y)with respect to a random variable xis de-

noted by Ex[f(x, y)]. In situations where there is no ambiguity as to which variable
is being averaged over, this will be simpliﬁed by omitting the sufﬁx, for instance
xi

xii MATHEMATICAL NOTATION
E[x]. If the distribution of xis conditioned on another variable z, then the corre-
sponding conditional expectation will be written Ex[f(x)|z]. Similarly, the variance
is denoted var[f(x)], and for vector variables the covariance is written cov[x,y].W e
shall also use cov[x]as a shorthand notation for cov[x,x]. The concepts of expecta-
tions and covariances are introduced in Section 1.2.2.
If we have Nvaluesx1,...,xNof aD-dimensional vector x=(x1,...,x D)T,
we can combine the observations into a data matrix Xin which the nthrow of X
corresponds to the row vector xT
n. Thus the n, i element of Xcorresponds to the
ithelement of the nthobservation xn. For the case of one-dimensional variables we
shall denote such a matrix by x, which is a column vector whose nthelement is xn.
Note that x(which has dimensionality N) uses a different typeface to distinguish it
fromx(which has dimensionality D).

Contents
Preface vii
Mathematical notation xiContents xiii
1.1 Example: Polynomial Curve Fitting . . . . ............. 4
1.2 Probability Theory . . ........................ 1 2
1.2.1 Probability densities . .................... 1 7
1.2.2 Expectations and covariances ................ 1 9
1.2.3 Bayesian probabilities .................... 2 1
1.2.4 The Gaussian distribution . . . . . ............. 2 4
1.2.5 Curve ﬁtting re-visited .................... 2 8
1.2.6 Bayesian curve ﬁtting .................... 3 0
1.3 Model Selection . . . ........................ 3 2
1.4 The Curse of Dimensionality . .................... 3 3
1.5 Decision Theory . . . ........................ 3 8
1.5.1 Minimizing the misclassiﬁcation rate . . . ......... 3 9
1.5.2 Minimizing the expected loss . . .............. 4 1
1.5.3 The reject option . . . .................... 4 2
1.5.4 Inference and decision .................... 4 2
1.5.5 Loss functions for regression . . . . ............. 4 6

1.6 Information Theory . . ........................ 4 8
1.6.1 Relative entropy and mutual information . ......... 5 5
Exercises . . . ............................... 5 8
xiii

2 Probability Distributions 67
2.1 Binary V ariables . . . ........................ 6 8
2.1.1 The beta distribution . .................... 7 1
2.2 Multinomial V ariables ........................ 7 4
2.2.1 The Dirichlet distribution . . . . . . ............. 7 6
2.3 The Gaussian Distribution . . .................... 7 8
2.3.1 Conditional Gaussian distributions . ............. 8 5
2.3.2 Marginal Gaussian distributions . . ............. 8 8
2.3.3 Bayes’ theorem for Gaussian variables . . . ......... 9 0
2.3.4 Maximum likelihood for the Gaussian . . . ......... 9 3
2.3.5 Sequential estimation . .................... 9 4
2.3.6 Bayesian inference for the Gaussian ............. 9 7
2.3.7 Student’s t-distribution .................... 1 0 2
2.3.8 Periodic variables . . . .................... 1 0 5
2.3.9 Mixtures of Gaussians .................... 1 1 0
2.4 The Exponential Family . . . .................... 1 1 3
2.4.1 Maximum likelihood and sufﬁcient statistics . . . . . . . . 116

2.4.2 Conjugate priors . . . .................... 1 1 7
2.4.3 Noninformative priors .................... 1 1 7
2.5 Nonparametric Methods . . . .................... 1 2 0
2.5.1 Kernel density estimators . . . . . . ............. 1 2 2
2.5.2 Nearest-neighbour methods . . . . ............. 1 2 4
Exercises . . . ............................... 1 2 7
3 Linear Models for Regression 137
3.1 Linear Basis Function Models .................... 1 3 8
3.1.1 Maximum likelihood and least squares . . . ......... 1 4 0

3.1.2 Geometry of least squares . . . . . ............. 1 4 3
3.1.3 Sequential learning . . .................... 1 4 3
3.1.4 Regularized least squares . . . . . . ............. 1 4 4
3.1.5 Multiple outputs . . . .................... 1 4 6
3.2 The Bias-V ariance Decomposition . . . . . ............. 1 4 7
3.3 Bayesian Linear Regression . .................... 1 5 2
3.3.1 Parameter distribution .................... 1 5 2
3.3.2 Predictive distribution .................... 1 5 6

3.3.3 Equivalent kernel . . . .................... 1 5 9
3.4 Bayesian Model Comparison . .................... 1 6 1
3.5 The Evidence Approximation .................... 1 6 5
3.5.1 Evaluation of the evidence function ............. 1 6 6
3.5.2 Maximizing the evidence function . ............. 1 6 8
3.5.3 Effective number of parameters . . ............. 1 7 0
3.6 Limitations of Fixed Basis Functions . . . ............. 1 7 2
Exercises . . . ............................... 1 7 3

4 Linear Models for Classiﬁcation 179
4.1 Discriminant Functions ........................ 1 8 1
4.1.1 Two classes . . ........................ 1 8 1
4.1.2 Multiple classes ........................ 1 8 2
4.1.3 Least squares for classiﬁcation ................ 1 8 4
4.1.4 Fisher’s linear discriminant . . . . . ............. 1 8 6
4.1.5 Relation to least squares . . . . . . ............. 1 8 9
4.1.6 Fisher’s discriminant for multiple classes . ......... 1 9 1
4.1.7 The perceptron algorithm . . . . . . ............. 1 9 2
4.2 Probabilistic Generative Models . . . ................ 1 9 6
4.2.1 Continuous inputs . . .................... 1 9 8
4.2.2 Maximum likelihood solution . . . ............. 2 0 0
4.2.3 Discrete features . . . .................... 2 0 2
4.2.4 Exponential family . . .................... 2 0 2
4.3 Probabilistic Discriminative Models . . . . ............. 2 0 3
4.3.1 Fixed basis functions . .................... 2 0 4
4.3.2 Logistic regression . . .................... 2 0 5

4.3.3 Iterative reweighted least squares .............. 2 0 7
4.3.4 Multiclass logistic regression . . . .............. 2 0 9
4.3.5 Probit regression . . . .................... 2 1 0
4.3.6 Canonical link functions . . . . . . ............. 2 1 2
4.4 The Laplace Approximation . .................... 2 1 3
4.4.1 Model comparison and BIC . . . .............. 2 1 6
4.5 Bayesian Logistic Regression .................... 2 1 7
4.5.1 Laplace approximation .................... 2 1 7
4.5.2 Predictive distribution .................... 2 1 8

Exercises . . . ............................... 2 2 0
5 Neural Networks 225
5.1 Feed-forward Network Functions . . . . . ............. 2 2 7
5.1.1 Weight-space symmetries . . . . .............. 2 3 1
5.2 Network Training . . . ........................ 2 3 2
5.2.1 Parameter optimization .................... 2 3 6
5.2.2 Local quadratic approximation . . . ............. 2 3 7
5.2.3 Use of gradient information . . . .............. 2 3 9
5.2.4 Gradient descent optimization . . . ............. 2 4 0

5.3 Error Backpropagation ........................ 2 4 1
5.3.1 Evaluation of error-function derivatives . . ......... 2 4 2
5.3.2 A simple example ...................... 2 4 5
5.3.3 Efﬁciency of backpropagation . . . ............. 2 4 6
5.3.4 The Jacobian matrix . .................... 2 4 7
5.4 The Hessian Matrix . . ........................ 2 4 9
5.4.1 Diagonal approximation . . . . . .............. 2 5 0
5.4.2 Outer product approximation . ................ 2 5 1
5.4.3 Inverse Hessian ........................ 2 5 2

5.4.4 Finite difference s....................... 2 5 2
5.4.5 Exact evaluation of the Hessian . . ............. 2 5 3
5.4.6 Fast multiplication by the Hessian . ............. 2 5 4
5.5 Regularization in Neural Networks . . . . ............. 2 5 6
5.5.1 Consistent Gaussian priors . . . . .............. 2 5 7
5.5.2 Early stopping ........................ 2 5 9
5.5.3 Invariances . . ........................ 2 6 1
5.5.4 Tangent propagation . .................... 2 6 3
5.5.5 Training with transformed data . . . ............. 2 6 5
5.5.6 Convolutional networks . . . . . . ............. 2 6 7
5.5.7 Soft weight sharing . . .................... 2 6 9
5.6 Mixture Density Networks . . .................... 2 7 2
5.7 Bayesian Neural Networks . . .................... 2 7 7
5.7.1 Posterior parameter distribution . . ............. 2 7 8
5.7.2 Hyperparameter optimization . . . ............. 2 8 0
5.7.3 Bayesian neural networks for classiﬁcation ......... 2 8 1
Exercises . . . ............................... 2 8 4

6 Kernel Methods 291
6.1 Dual Representations . ........................ 2 9 3
6.2 Constructing Kernels . ........................ 2 9 4
6.3 Radial Basis Function Networks . . . ................ 2 9 9
6.3.1 Nadaraya-Watson model . . . . . . ............. 3 0 1
6.4 Gaussian Processes . . ........................ 3 0 3
6.4.1 Linear regression revisited . . . . . ............. 3 0 4
6.4.2 Gaussian processes for regression . ............. 3 0 6
6.4.3 Learning the hyperparameters . . . ............. 3 1 1

6.4.4 Automatic relevance determination ............. 3 1 2
6.4.5 Gaussian processes for classiﬁcation ............. 3 1 3
6.4.6 Laplace approximation .................... 3 1 5
6.4.7 Connection to neural networks . . . ............. 3 1 9
Exercises . . . ............................... 3 2 0
7 Sparse Kernel Machines 325
7.1 Maximum Margin Classiﬁers .................... 3 2 6
7.1.1 Overlapping class distributions . . .............. 3 3 1
7.1.2 Relation to logistic regression . . .............. 3 3 6

7.1.3 Multiclass SVMs . . . .................... 3 3 8
7.1.4 SVMs for regression . .................... 3 3 9
7.1.5 Computational learning theory . . . ............. 3 4 4
7.2 Relevance V ector Machines . .................... 3 4 5
7.2.1 RVM for regression . . .................... 3 4 5
7.2.2 Analysis of sparsity . . .................... 3 4 9
7.2.3 RVM for classiﬁcation .................... 3 5 3
Exercises . . . ............................... 3 5 7

8 Graphical Models 359
8.1 Bayesian Networks . . ........................ 3 6 0
8.1.1 Example: Polynomial regression . . ............. 3 6 2
8.1.2 Generative models . . .................... 3 6 5
8.1.3 Discrete variables . . . .................... 3 6 6
8.1.4 Linear-Gaussian models . . . . . . ............. 3 7 0
8.2 Conditional Independence . . .................... 3 7 2
8.2.1 Three example graphs .................... 3 7 3
8.2.2 D-separation . ........................ 3 7 8
8.3 Markov Random Fields . . . .................... 3 8 3
8.3.1 Conditional independence properties ............. 3 8 3
8.3.2 Factorization properties . . . . . . ............. 3 8 4
8.3.3 Illustration: Image de-noising . . . ............. 3 8 7
8.3.4 Relation to directed graphs . . . . . ............. 3 9 0
8.4 Inference in Graphical Models .................... 3 9 3
8.4.1 Inference on a chain . .................... 3 9 4
8.4.2 Trees ............................. 3 9 8
8.4.3 Factor graphs . ........................ 3 9 9

8.4.4 The sum-product algorithm . . . . .............. 4 0 2
8.4.5 The max-sum algorithm . . . . . . ............. 4 1 1
8.4.6 Exact inference in general graphs . ............. 4 1 6
8.4.7 Loopy belief propagation . . . . . . ............. 4 1 7
8.4.8 Learning the graph structure . . . .............. 4 1 8
Exercises . . . ............................... 4 1 8
9 Mixture Models and EM 423
9.1 K-means Clustering . ........................ 4 2 4
9.1.1 Image segmentation and compression . . . ......... 4 2 8

9.2 Mixtures of Gaussians ........................ 4 3 0
9.2.1 Maximum likelihood . .................... 4 3 2
9.2.2 EM for Gaussian mixtures . . . . . ............. 4 3 5
9.3 An Alternative View of EM . .................... 4 3 9
9.3.1 Gaussian mixtures revisited . . . .............. 4 4 1
9.3.2 Relation to K-means . .................... 4 4 3
9.3.3 Mixtures of Bernoulli distributions . ............. 4 4 4
9.3.4 EM for Bayesian linear regression . ............. 4 4 8

9.4 The EM Algorithm in General .................... 4 5 0
Exercises . . . ............................... 4 5 5
10 Approximate Inference 461
10.1 V ariational Inference . ........................ 4 6 2
10.1.1 Factorized distributions .................... 4 6 4
10.1.2 Properties of factorized approximations . . ......... 4 6 6
10.1.3 Example: The univariate Gaussian . ............. 4 7 0
10.1.4 Model comparison . . .................... 4 7 3
10.2 Illustration: V ariational Mixture of Gaussians . . . ......... 4 7 4

10.2.1 V ariational distribution .................... 4 7 5
10.2.2 V ariational lower bound . . . . . . ............. 4 8 1
10.2.3 Predictive density . . . .................... 4 8 2
10.2.4 Determining the number of components . . ......... 4 8 3
10.2.5 Induced factorizations .................... 4 8 5
10.3 V ariational Linear Regression .................... 4 8 6
10.3.1 V ariational distribution .................... 4 8 6
10.3.2 Predictive distribution .................... 4 8 8
10.3.3 Lower bound . ........................ 4 8 9
10.4 Exponential Family Distributions . . . . . ............. 4 9 0
10.4.1 V ariational message passing . . . .............. 4 9 1
10.5 Local V ariational Methods . . .................... 4 9 3
10.6 V ariational Logistic Regression . . . ................ 4 9 8
10.6.1 V ariational posterior distribution . . ............. 4 9 8
10.6.2 Optimizing the variational parameters . . . ......... 5 0 0
10.6.3 Inference of hyperparameters ................ 5 0 2

10.7 Expectation Propagation . . . .................... 5 0 5
10.7.1 Example: The clutter problem . . . ............. 5 1 1
10.7.2 Expectation propagation on graphs . ............. 5 1 3
Exercises . . . ............................... 5 1 7
11 Sampling Methods 523
11.1 Basic Sampling Algorithms . .................... 5 2 6
11.1.1 Standard distributions .................... 5 2 6
11.1.2 Rejection sampling . . .................... 5 2 8
11.1.3 Adaptive rejection sampling . . . . ............. 5 3 0

11.1.4 Importance sampling . .................... 5 3 2
11.1.5 Sampling-importance-resampling . ............. 5 3 4
11.1.6 Sampling and the EM algorithm . . ............. 5 3 6
11.2 Markov Chain Monte Carlo . .................... 5 3 7
11.2.1 Markov chains ........................ 5 3 9
11.2.2 The Metropolis-Hastings algorithm ............. 5 4 1
11.3 Gibbs Sampling . . . ........................ 5 4 2
11.4 Slice Sampling . . . . ........................ 5 4 6

11.5 The Hybrid Monte Carlo Algorithm . . . . ............. 5 4 8
11.5.1 Dynamical systems . . .................... 5 4 8
11.5.2 Hybrid Monte Carlo . .................... 5 5 2
11.6 Estimating the Partition Function . . . . . ............. 5 5 4
Exercises . . . ............................... 5 5 6
12 Continuous Latent Variables 559
12.1 Principal Component Analysis .................... 5 6 1
12.1.1 Maximum variance formulation . . ............. 5 6 1
12.1.2 Minimum-error formulation . . . . ............. 5 6 3

12.1.3 Applications of PCA . .................... 5 6 5
12.1.4 PCA for high-dimensional data . . ............. 5 6 9

12.2 Probabilistic PCA . . ........................ 5 7 0
12.2.1 Maximum likelihood PCA . . . . . ............. 5 7 4
12.2.2 EM algorithm for PCA .................... 5 7 7
12.2.3 Bayesian PCA ........................ 5 8 0
12.2.4 Factor analysis ........................ 5 8 3
12.3 Kernel PCA .............................. 5 8 6
12.4 Nonlinear Latent V ariable Models . . . . . ............. 5 9 1
12.4.1 Independent component analysis . . ............. 5 9 1
12.4.2 Autoassociative neural networks . . ............. 5 9 2
12.4.3 Modelling nonlinear manifolds . . . ............. 5 9 5
Exercises . . . ............................... 5 9 9
13 Sequential Data 605
13.1 Markov Models . . . . ........................ 6 0 7
13.2 Hidden Markov Models . . . .................... 6 1 0
13.2.1 Maximum likelihood for the HMM ............. 6 1 5
13.2.2 The forward-backward algorithm . ............. 6 1 8
13.2.3 The sum-product algorithm for the HMM . ......... 6 2 5

13.2.4 Scaling factors ........................ 6 2 7
13.2.5 The Viterbi algorithm . .................... 6 2 9
13.2.6 Extensions of the hidden Markov model . . ......... 6 3 1
13.3 Linear Dynamical Systems . . .................... 6 3 5
13.3.1 Inference in LDS . . . .................... 6 3 8
13.3.2 Learning in LDS . . . .................... 6 4 2
13.3.3 Extensions of LDS . . .................... 6 4 4
13.3.4 Particle ﬁlters . ........................ 6 4 5
Exercises . . . ............................... 6 4 6

14 Combining Models 653
14.1 Bayesian Model Averaging . . .................... 6 5 4
14.2 Committees . . . . . . ........................ 6 5 5
14.3 Boosting ............................... 6 5 7
14.3.1 Minimizing exponential error . . . ............. 6 5 9
14.3.2 Error functions for boosting . . . .............. 6 6 1
14.4 Tree-based Models . . ........................ 6 6 3
14.5 Conditional Mixture Models . .................... 6 6 6
14.5.1 Mixtures of linear regression models ............. 6 6 7

14.5.2 Mixtures of logistic models . . . . ............. 6 7 0
14.5.3 Mixtures of experts . . .................... 6 7 2
Exercises . . . ............................... 6 7 4
Appendix A Data Sets 677Appendix B Probability Distributions 685
Appendix C Properties of Matrices 695

Appendix D Calculus of Variations 703
Appendix E Lagrange Multipliers 707
References 711Index 729

1
The problem of searching for patterns in data is a fundamental one and has a long and
successful history. For instance, the extensive astronomical observations of TychoBrahe in the 16
thcentury allowed Johannes Kepler to discover the empirical laws of
planetary motion, which in turn provided a springboard for the development of clas-
sical mechanics. Similarly, the discovery of regularities in atomic spectra played akey role in the development and veriﬁcation of quantum physics in the early twenti-
eth century. The ﬁeld of pattern recognition is concerned with the automatic discov-
ery of regularities in data through the use of computer algorithms and with the use ofthese regularities to take actions such as classifying the data into different categories.
Consider the example of recognizing handwritten digits, illustrated in Figure 1.1.
Each digit corresponds to a 28×28pixel image and so can be represented by a vector

xcomprising 784 real numbers. The goal is to build a machine that will take such a
vectorxas input and that will produce the identity of the digit 0,...,9as the output.
This is a nontrivial problem due to the wide variability of handwriting. It could be
1

2 1. INTRODUCTION
Figure 1.1 Examples of hand-written dig-
its taken from US zip codes.
tackled using handcrafted rules or heuristics for distinguishing the digits based on
the shapes of the strokes, but in practice such an approach leads to a proliferation of
rules and of exceptions to the rules and so on, and invariably gives poor results.
Far better results can be obtained by adopting a machine learning approach in
which a large set of Ndigits{x1,...,xN}called a training set is used to tune the
parameters of an adaptive model. The categories of the digits in the training setare known in advance, typically by inspecting them individually and hand-labelling
them. We can express the category of a digit using target vector t, which represents
the identity of the corresponding digit. Suitable techniques for representing cate-gories in terms of vectors will be discussed later. Note that there is one such target
vectortfor each digit image x.

The result of running the machine learning algorithm can be expressed as a
function y(x)which takes a new digit image xas input and that generates an output
vectory, encoded in the same way as the target vectors. The precise form of the
function y(x)is determined during the training phase, also known as the learning
phase, on the basis of the training data. Once the model is trained it can then de-
termine the identity of new digit images, which are said to comprise a test set . The
ability to categorize correctly new examples that differ from those used for train-

ing is known as generalization . In practical applications, the variability of the input
vectors will be such that the training data can comprise only a tiny fraction of allpossible input vectors, and so generalization is a central goal in pattern recognition.
For most practical applications, the original input variables are typically prepro-
cessed to transform them into some new space of variables where, it is hoped, the
pattern recognition problem will be easier to solve. For instance, in the digit recogni-

tion problem, the images of the digits are typically translated and scaled so that each
digit is contained within a box of a ﬁxed size. This greatly reduces the variabilitywithin each digit class, because the location and scale of all the digits are now the
same, which makes it much easier for a subsequent pattern recognition algorithm
to distinguish between the different classes. This pre-processing stage is sometimesalso called feature extraction . Note that new test data must be pre-processed using
the same steps as the training data.

Pre-processing might also be performed in order to speed up computation. For
example, if the goal is real-time face detection in a high-resolution video stream,
the computer must handle huge numbers of pixels per second, and presenting thesedirectly to a complex pattern recognition algorithm may be computationally infeasi-
ble. Instead, the aim is to ﬁnd useful features that are fast to compute, and yet that

1. INTRODUCTION 3
also preserve useful discriminatory information enabling faces to be distinguished
from non-faces. These features are then used as the inputs to the pattern recognitionalgorithm. For instance, the average value of the image intensity over a rectangular
subregion can be evaluated extremely efﬁciently (Viola and Jones, 2004), and a set of
such features can prove very effective in fast face detection. Because the number ofsuch features is smaller than the number of pixels, this kind of pre-processing repre-
sents a form of dimensionality reduction. Care must be taken during pre-processing
because often information is discarded, and if this information is important to thesolution of the problem then the overall accuracy of the system can suffer.
Applications in which the training data comprises examples of the input vectors
along with their corresponding target vectors are known as supervised learning prob-
lems. Cases such as the digit recognition example, in which the aim is to assign each

input vector to one of a ﬁnite number of discrete categories, are called classiﬁcation
problems. If the desired output consists of one or more continuous variables, then
the task is called regression . An example of a regression problem would be the pre-
diction of the yield in a chemical manufacturing process in which the inputs consistof the concentrations of reactants, the temperature, and the pressure.
In other pattern recognition problems, the training data consists of a set of input

vectors xwithout any corresponding target values. The goal in such unsupervised
learning problems may be to discover groups of similar examples within the data,
where it is called clustering , or to determine the distribution of data within the input
space, known as density estimation , or to project the data from a high-dimensional
space down to two or three dimensions for the purpose of visualization .
Finally, the technique of reinforcement learning (Sutton and Barto, 1998) is con-

cerned with the problem of ﬁnding suitable actions to take in a given situation inorder to maximize a reward. Here the learning algorithm is not given examples of
optimal outputs, in contrast to supervised learning, but must instead discover them
by a process of trial and error. Typically there is a sequence of states and actions in
which the learning algorithm is interacting with its environment. In many cases, the

current action not only affects the immediate reward but also has an impact on the re-ward at all subsequent time steps. For example, by using appropriate reinforcement
learning techniques a neural network can learn to play the game of backgammon to a
high standard (Tesauro, 1994). Here the network must learn to take a board positionas input, along with the result of a dice throw, and produce a strong move as the
output. This is done by having the network play against a copy of itself for perhaps a

million games. A major challenge is that a game of backgammon can involve dozensof moves, and yet it is only at the end of the game that the reward, in the form of
victory, is achieved. The reward must then be attributed appropriately to all of the
moves that led to it, even though some moves will have been good ones and othersless so. This is an example of a credit assignment problem. A general feature of re-
inforcement learning is the trade-off between exploration , in which the system tries

out new kinds of actions to see how effective they are, and exploitation , in which
the system makes use of actions that are known to yield a high reward. Too strong
a focus on either exploration or exploitation will yield poor results. Reinforcement
learning continues to be an active area of machine learning research. However, a

4 1. INTRODUCTION
Figure 1.2 Plot of a training data set of N=
10points, shown as blue circles,
each comprising an observation
of the input variable xalong with
the corresponding target variable
t. The green curve shows the
function sin(2πx)used to gener-
ate the data. Our goal is to pre-
dict the value of tfor some new
value of x, without knowledge of
the green curve.
xt
0 1−101
detailed treatment lies beyond the scope of this book.
Although each of these tasks needs its own tools and techniques, many of the
key ideas that underpin them are common to all such problems. One of the main
goals of this chapter is to introduce, in a relatively informal way, several of the most
important of these concepts and to illustrate them using simple examples. Later in
the book we shall see these same ideas re-emerge in the context of more sophisti-
cated models that are applicable to real-world pattern recognition applications. This
chapter also provides a self-contained introduction to three important tools that will

be used throughout the book, namely probability theory, decision theory, and infor-
mation theory. Although these might sound like daunting topics, they are in fact
straightforward, and a clear understanding of them is essential if machine learning
techniques are to be used to best effect in practical applications.
1.1. Example: Polynomial Curve Fitting
We begin by introducing a simple regression problem, which we shall use as a run-
ning example throughout this chapter to motivate a number of key concepts. Sup-

pose we observe a real-valued input variable xand we wish to use this observation to
predict the value of a real-valued target variable t. For the present purposes, it is in-
structive to consider an artiﬁcial example using synthetically generated data because
we then know the precise process that generated the data for comparison against any
learned model. The data for this example is generated from the function sin(2πx)
with random noise included in the target values, as described in detail in Appendix A.
Now suppose that we are given a training set comprising Nobservations of x,

written x≡(x1,...,x N)T, together with corresponding observations of the values
oft, denoted t≡(t1,...,t N)T. Figure 1.2 shows a plot of a training set comprising
N=1 0 data points. The input data set xin Figure 1.2 was generated by choos-
ing values of xn, forn=1,...,N , spaced uniformly in range [0,1], and the target
data set twas obtained by ﬁrst computing the corresponding values of the function

1.1. Example: Polynomial Curve Fitting 5
sin(2πx)and then adding a small level of random noise having a Gaussian distri-
bution (the Gaussian distribution is discussed in Section 1.2.4) to each such point inorder to obtain the corresponding value t
n. By generating data in this way, we are
capturing a property of many real data sets, namely that they possess an underlying
regularity, which we wish to learn, but that individual observations are corrupted byrandom noise. This noise might arise from intrinsically stochastic (i.e. random) pro-
cesses such as radioactive decay but more typically is due to there being sources of
variability that are themselves unobserved.
Our goal is to exploit this training set in order to make predictions of the value
ˆtof the target variable for some new value ˆxof the input variable. As we shall see
later, this involves implicitly trying to discover the underlying function sin(2πx).
This is intrinsically a difﬁcult problem as we have to generalize from a ﬁnite data

set. Furthermore the observed data are corrupted with noise, and so for a given ˆx
there is uncertainty as to the appropriate value for ˆt. Probability theory, discussed
in Section 1.2, provides a framework for expressing such uncertainty in a precise
and quantitative manner, and decision theory, discussed in Section 1.5, allows us toexploit this probabilistic representation in order to make predictions that are optimal
according to appropriate criteria.
For the moment, however, we shall proceed rather informally and consider a

simple approach based on curve ﬁtting. In particular, we shall ﬁt the data using a
polynomial function of the form
y(x,w)=w
0+w1x+w2x2+...+wMxM=M∑
j=0wjxj(1.1)
where Mis the order of the polynomial, and xjdenotes xraised to the power of j.
The polynomial coefﬁcients w0,...,w Mare collectively denoted by the vector w.
Note that, although the polynomial function y(x,w)is a nonlinear function of x,i t
is a linear function of the coefﬁcients w. Functions, such as the polynomial, which
are linear in the unknown parameters have important properties and are called linear

y(x,w)=w
models and will be discussed extensively in Chapters 3 and 4.
The values of the coefﬁcients will be determined by ﬁtting the polynomial to the
training data. This can be done by minimizing an error function that measures the
misﬁt between the function y(x,w), for any given value of w, and the training set
data points. One simple choice of error function, which is widely used, is given by
the sum of the squares of the errors between the predictions y(xn,w)for each data

pointxnand the corresponding target values tn, so that we minimize
E(w)=1
2N∑
n=1{y(xn,w)−tn}2(1.2)
where the factor of 1/2is included for later convenience. We shall discuss the mo-
tivation for this choice of error function later in this chapter. For the moment wesimply note that it is a nonnegative quantity that would be zero if, and only if, the

6 1. INTRODUCTION
Figure 1.3 The error function (1.2) corre-
sponds to (one half of) the sum of
the squares of the displacements
(shown by the vertical green bars)
of each data point from the function
y(x,w).t
xy(xn,w)tn
xn
function y(x,w)were to pass exactly through each training data point. The geomet-
rical interpretation of the sum-of-squares error function is illustrated in Figure 1.3.
We can solve the curve ﬁtting problem by choosing the value of wfor which
E(w)is as small as possible. Because the error function is a quadratic function of
the coefﬁcients w, its derivatives with respect to the coefﬁcients will be linear in the
elements of w, and so the minimization of the error function has a unique solution,
denoted by w⋆, which can be found in closed form. The resulting polynomial is Exercise 1.1
given by the function y(x,w⋆).
There remains the problem of choosing the order Mof the polynomial, and as
we shall see this will turn out to be an example of an important concept called model

comparison ormodel selection . In Figure 1.4, we show four examples of the results
of ﬁtting polynomials having orders M=0,1,3,a n d9to the data set shown in
Figure 1.2.
We notice that the constant ( M=0 ) and ﬁrst order ( M=1 ) polynomials
give rather poor ﬁts to the data and consequently rather poor representations of the
function sin(2πx). The third order ( M=3 ) polynomial seems to give the best ﬁt
to the function sin(2πx)of the examples shown in Figure 1.4. When we go to a
much higher order polynomial ( M=9 ), we obtain an excellent ﬁt to the training

data. In fact, the polynomial passes exactly through each data point and E(w⋆)=0 .
However, the ﬁtted curve oscillates wildly and gives a very poor representation of
the function sin(2πx). This latter behaviour is known as over-ﬁtting .
As we have noted earlier, the goal is to achieve good generalization by making
accurate predictions for new data. We can obtain some quantitative insight into the
dependence of the generalization performance on Mby considering a separate test

set comprising 100 data points generated using exactly the same procedure used
to generate the training set points but with new choices for the random noise values
included in the target values. For each choice of M, we can then evaluate the residual
value of E(w⋆)given by (1.2) for the training data, and we can also evaluate E(w⋆)
for the test data set. It is sometimes more convenient to use the root-mean-square

1.1. Example: Polynomial Curve Fitting 7
xtM=0
0 1−101
xtM=1
0 1−101
xtM=3
0 1−101
xtM=9
0 1−101
Figure 1.4 Plots of polynomials having various orders M, shown as red curves, ﬁtted to the data set shown in
(RMS) error deﬁned by
ERMS=√
2E(w⋆)/N (1.3)
in which the division by Nallows us to compare different sizes of data sets on
an equal footing, and the square root ensures that ERMS is measured on the same
scale (and in the same units) as the target variable t. Graphs of the training and
test set RMS errors are shown, for various values of M, in Figure 1.5. The test
set error is a measure of how well we are doing in predicting the values of tfor
new data observations of x. We note from Figure 1.5 that small values of Mgive
relatively large values of the test set error, and this can be attributed to the fact that
the corresponding polynomials are rather inﬂexible and are incapable of capturing
the oscillations in the function sin(2πx). V alues of Min the range 3⩽M⩽8

give small values for the test set error, and these also give reasonable representations
of the generating function sin(2πx), as can be seen, for the case of M=3 , from
Figure 1.4.

8 1. INTRODUCTION
Figure 1.5 Graphs of the root-mean-square
error, deﬁned by (1.3), evaluated
on the training set and on an inde-
pendent test set for various values
ofM.
MERMS
0 3 6 900.51
Training
Test
ForM=9 , the training set error goes to zero, as we might expect because
this polynomial contains 10degrees of freedom corresponding to the 10coefﬁcients
w0,...,w 9, and so can be tuned exactly to the 10data points in the training set.
However, the test set error has become very large and, as we saw in Figure 1.4, the
corresponding function y(x,w⋆)exhibits wild oscillations.
This may seem paradoxical because a polynomial of given order contains all
lower order polynomials as special cases. The M=9 polynomial is therefore capa-
ble of generating results at least as good as the M=3 polynomial. Furthermore, we
might suppose that the best predictor of new data would be the function sin(2πx)
from which the data was generated (and we shall see later that this is indeed the

case). We know that a power series expansion of the function sin(2πx)contains
terms of all orders, so we might expect that results should improve monotonically as
we increase M.
We can gain some insight into the problem by examining the values of the co-
efﬁcients w⋆obtained from polynomials of various order, as shown in Table 1.1.
We see that, as Mincreases, the magnitude of the coefﬁcients typically gets larger.
In particular for the M=9 polynomial, the coefﬁcients have become ﬁnely tuned

to the data by developing large positive and negative values so that the correspond-
Table 1.1 Table of the coefﬁcients w⋆for
polynomials of various order.
Observe how the typical mag-
nitude of the coefﬁcients in-
creases dramatically as the or-
der of the polynomial increases.M=0 M=1 M=6 M=9
w⋆
0 0.19 0.82 0.31 0.35
w⋆
1 -1.27 7.99 232.37
w⋆
2 -25.43 -5321.83
w⋆
3 17.37 48568.31
w⋆
4 -231639.30
w⋆
5 640042.26
w⋆
6 -1061800.52
w⋆
7 1042400.18
w⋆
8 -557682.99
w⋆
9 125201.43

1.1. Example: Polynomial Curve Fitting 9
xtN=1 5
0 1−101
xtN= 100
0 1−101
Figure 1.6 Plots of the solutions obtained by minimizing the sum-of-squares error function using the M=9
polynomial for N=1 5 data points (left plot) and N= 100 data points (right plot). We see that increasing the
size of the data set reduces the over-ﬁtting problem.
ing polynomial function matches each of the data points exactly, but between data
points (particularly near the ends of the range) the function exhibits the large oscilla-
tions observed in Figure 1.4. Intuitively, what is happening is that the more ﬂexible
polynomials with larger values of Mare becoming increasingly tuned to the random
noise on the target values.
It is also interesting to examine the behaviour of a given model as the size of the
data set is varied, as shown in Figure 1.6. We see that, for a given model complexity,
the over-ﬁtting problem become less severe as the size of the data set increases.

Another way to say this is that the larger the data set, the more complex (in other
words more ﬂexible) the model that we can afford to ﬁt to the data. One rough
heuristic that is sometimes advocated is that the number of data points should be
no less than some multiple (say 5 or 10) of the number of adaptive parameters in
the model. However, as we shall see in Chapter 3, the number of parameters is not
necessarily the most appropriate measure of model complexity.
Also, there is something rather unsatisfying about having to limit the number of

parameters in a model according to the size of the available training set. It would
seem more reasonable to choose the complexity of the model according to the com-
plexity of the problem being solved. We shall see that the least squares approach
to ﬁnding the model parameters represents a speciﬁc case of maximum likelihood
(discussed in Section 1.2.5), and that the over-ﬁtting problem can be understood as
a general property of maximum likelihood. By adopting a Bayesian approach, the Section 3.4

over-ﬁtting problem can be avoided. We shall see that there is no difﬁculty from
a Bayesian perspective in employing models for which the number of parameters
greatly exceeds the number of data points. Indeed, in a Bayesian model the effective
number of parameters adapts automatically to the size of the data set.
For the moment, however, it is instructive to continue with the current approach
and to consider how in practice we can apply it to data sets of limited size where we

10 1. INTRODUCTION
xtlnλ=−18
0 1−101
xtlnλ=0
0 1−101
Figure 1.7 Plots of M=9polynomials ﬁtted to the data set shown in Figure 1.2 using the regularized error
function (1.4) for two values of the regularization parameter λcorresponding to lnλ=−18andlnλ=0. The
case of no regularizer, i.e., λ=0, corresponding to lnλ=−∞, is shown at the bottom right of Figure 1.4.
may wish to use relatively complex and ﬂexible models. One technique that is often
used to control the over-ﬁtting phenomenon in such cases is that of regularization ,
which involves adding a penalty term to the error function (1.2) in order to discourage
the coefﬁcients from reaching large values. The simplest such penalty term takes the
form of a sum of squares of all of the coefﬁcients, leading to a modiﬁed error function
of the form
˜E(w)=1
2N∑
n=1{y(xn,w)−tn}2+λ
2∥w∥2(1.4)
where∥w∥2≡wTw=w2
0+w2
1+...+w2
M, and the coefﬁcient λgoverns the rel-
ative importance of the regularization term compared with the sum-of-squares error

of the form
˜E(w)=1
2N∑
0+w2
1+...+w2
term. Note that often the coefﬁcient w0is omitted from the regularizer because its
inclusion causes the results to depend on the choice of origin for the target variable
(Hastie et al. , 2001), or it may be included but with its own regularization coefﬁcient
(we shall discuss this topic in more detail in Section 5.5.1). Again, the error function
in (1.4) can be minimized exactly in closed form. Techniques such as this are known Exercise 1.2
in the statistics literature as shrinkage methods because they reduce the value of the

coefﬁcients. The particular case of a quadratic regularizer is called ridge regres-
sion (Hoerl and Kennard, 1970). In the context of neural networks, this approach is
known as weight decay .
Figure 1.7 shows the results of ﬁtting the polynomial of order M=9 to the
same data set as before but now using the regularized error function given by (1.4).
We see that, for a value of lnλ=−18, the over-ﬁtting has been suppressed and we
now obtain a much closer representation of the underlying function sin(2πx). If,

however, we use too large a value for λthen we again obtain a poor ﬁt, as shown in
Figure 1.7 for lnλ=0 . The corresponding coefﬁcients from the ﬁtted polynomials
are given in Table 1.2, showing that regularization has the desired effect of reducing

1.1. Example: Polynomial Curve Fitting 11
Table 1.2 Table of the coefﬁcients w⋆forM=
9polynomials with various values for
the regularization parameter λ. Note
thatlnλ=−∞ corresponds to a
model with no regularization, i.e., to
the graph at the bottom right in Fig-
ure 1.4. We see that, as the value of
λincreases, the typical magnitude of
the coefﬁcients gets smaller.lnλ=−∞ lnλ=−18 ln λ=0
w⋆
0 0.35 0.35 0.13
w⋆
1 232.37 4.74 -0.05
w⋆
2 -5321.83 -0.77 -0.06
w⋆
3 48568.31 -31.97 -0.05
w⋆
4 -231639.30 -3.89 -0.03
w⋆
5 640042.26 55.28 -0.02
w⋆
6 -1061800.52 41.32 -0.01
w⋆
7 1042400.18 -45.95 -0.00
w⋆
8 -557682.99 -91.53 0.00
w⋆
9 125201.43 72.68 0.01
the magnitude of the coefﬁcients.
The impact of the regularization term on the generalization error can be seen by
plotting the value of the RMS error (1.3) for both training and test sets against lnλ,
as shown in Figure 1.8. We see that in effect λnow controls the effective complexity
of the model and hence determines the degree of over-ﬁtting.

w⋆
w⋆
w⋆
w⋆
w⋆
The issue of model complexity is an important one and will be discussed at
length in Section 1.3. Here we simply note that, if we were trying to solve a practical
application using this approach of minimizing an error function, we would have to
ﬁnd a way to determine a suitable value for the model complexity. The results above
suggest a simple way of achieving this, namely by taking the available data and
partitioning it into a training set, used to determine the coefﬁcients w, and a separate

validation set, also called a hold-out set, used to optimize the model complexity
(either Morλ). In many cases, however, this will prove to be too wasteful of
valuable training data, and we have to seek more sophisticated approaches. Section 1.3
So far our discussion of polynomial curve ﬁtting has appealed largely to in-
tuition. We now seek a more principled approach to solving problems in pattern
recognition by turning to a discussion of probability theory. As well as providing the

foundation for nearly all of the subsequent developments in this book, it will also
Figure 1.8 Graph of the root-mean-square er-
ror (1.3) versus lnλfor the M=9
polynomial.
ERMS
lnλ−35 −30 −25 −2000.51
Training
Test

12 1. INTRODUCTION
give us some important insights into the concepts we have introduced in the con-
text of polynomial curve ﬁtting and will allow us to extend these to more complexsituations.
1.2. Probability Theory
A key concept in the ﬁeld of pattern recognition is that of uncertainty. It arises boththrough noise on measurements, as well as through the ﬁnite size of data sets. Prob-ability theory provides a consistent framework for the quantiﬁcation and manipula-
tion of uncertainty and forms one of the central foundations for pattern recognition.
When combined with decision theory, discussed in Section 1.5, it allows us to makeoptimal predictions given all the information available to us, even though that infor-
mation may be incomplete or ambiguous.
We will introduce the basic concepts of probability theory by considering a sim-
ple example. Imagine we have two boxes, one red and one blue, and in the red box
we have 2 apples and 6 oranges, and in the blue box we have 3 apples and 1 orange.

This is illustrated in Figure 1.9. Now suppose we randomly pick one of the boxesand from that box we randomly select an item of fruit, and having observed which
sort of fruit it is we replace it in the box from which it came. We could imagine
repeating this process many times. Let us suppose that in so doing we pick the red
box 40% of the time and we pick the blue box 60% of the time, and that when we
remove an item of fruit from a box we are equally likely to select any of the piecesof fruit in the box.

In this example, the identity of the box that will be chosen is a random variable,
which we shall denote by B. This random variable can take one of two possible
values, namely r(corresponding to the red box) or b(corresponding to the blue
box). Similarly, the identity of the fruit is also a random variable and will be denoted
byF. It can take either of the values a(for apple) or o(for orange).
To begin with, we shall deﬁne the probability of an event to be the fraction

of times that event occurs out of the total number of trials, in the limit that the total
number of trials goes to inﬁnity. Thus the probability of selecting the red box is 4/10
Figure 1.9 We use a simple example of two
coloured boxes each containing fruit(apples shown in green and or-anges shown in orange) to intro-duce the basic ideas of probability.

1.2. Probability Theory 13
Figure 1.10 We can derive the sum and product rules of probability by
considering two random variables, X, which takes the values {xi}where
i=1,...,M , and Y, which takes the values {yj}where j=1,...,L .
In this illustration we have M=5 andL=3. If we consider a total
number Nof instances of these variables, then we denote the number
of instances where X=xiandY=yjbynij, which is the number of
points in the corresponding cell of the array. The number of points in
column i, corresponding to X=xi, is denoted by ci, and the number of
points in row j, corresponding to Y=yj, is denoted by rj.}}ci
rj yj
xinij
and the probability of selecting the blue box is 6/10. We write these probabilities
asp(B=r)=4/10andp(B=b)=6/10. Note that, by deﬁnition, probabilities
must lie in the interval [0,1]. Also, if the events are mutually exclusive and if they
include all possible outcomes (for instance, in this example the box must be either

rj yj
xinij
red or blue), then we see that the probabilities for those events must sum to one.
We can now ask questions such as: “what is the overall probability that the se-
lection procedure will pick an apple?”, or “given that we have chosen an orange,
what is the probability that the box we chose was the blue one?”. We can answer
questions such as these, and indeed much more complex questions associated with
problems in pattern recognition, once we have equipped ourselves with the two el-

ementary rules of probability, known as the sum rule and the product rule . Having
obtained these rules, we shall then return to our boxes of fruit example.
In order to derive the rules of probability, consider the slightly more general ex-
ample shown in Figure 1.10 involving two random variables XandY(which could
for instance be the Box and Fruit variables considered above). We shall suppose that
Xcan take any of the values xiwhere i=1,...,M , andYcan take the values yj

where j=1,...,L . Consider a total of Ntrials in which we sample both of the
variables XandY, and let the number of such trials in which X=xiandY=yj
benij. Also, let the number of trials in which Xtakes the value xi(irrespective
of the value that Ytakes) be denoted by ci, and similarly let the number of trials in
which Ytakes the value yjbe denoted by rj.
The probability that Xwill take the value xiandYwill take the value yjis
written p(X=xi,Y=yj)and is called the joint probability of X=xiand

Y=yj. It is given by the number of points falling in the cell i,jas a fraction of the
total number of points, and hence
p(X=xi,Y=yj)=nij
N. (1.5)
Here we are implicitly considering the limit N→∞ . Similarly, the probability that
Xtakes the value xiirrespective of the value of Yis written as p(X=xi)and is
given by the fraction of the total number of points that fall in column i, so that
p(X=xi)=ci
N. (1.6)
Because the number of instances in column iin Figure 1.10 is just the sum of the

N. (1.5)
N. (1.6)
number of instances in each cell of that column, we have ci=∑
jnijand therefore,

14 1. INTRODUCTION
from (1.5) and (1.6), we have
p(X=xi)=L∑
j=1p(X=xi,Y=yj) (1.7)
which is the sum rule of probability. Note that p(X=xi)is sometimes called the
marginal probability, because it is obtained by marginalizing, or summing out, the
other variables (in this case Y).
If we consider only those instances for which X=xi, then the fraction of
such instances for which Y=yjis written p(Y=yj|X=xi)and is called the
conditional probability of Y=yjgiven X=xi. It is obtained by ﬁnding the
fraction of those points in column ithat fall in cell i,jand hence is given by
p(Y=yj|X=xi)=nij
ci. (1.8)
From (1.5), (1.6), and (1.8), we can then derive the following relationship
N=nij
ci·ci
N
=p(Y=yj|X=xi)p(X=xi) (1.9)
which is the product rule of probability.
So far we have been quite careful to make a distinction between a random vari-
able, such as the box Bin the fruit example, and the values that the random variable
can take, for example rif the box were the red one. Thus the probability that Btakes

ci. (1.8)
N=nij
ci·ci
N
the value ris denoted p(B=r). Although this helps to avoid ambiguity, it leads
to a rather cumbersome notation, and in many cases there will be no need for such
pedantry. Instead, we may simply write p(B)to denote a distribution over the ran-
dom variable B,o rp(r)to denote the distribution evaluated for the particular value
r, provided that the interpretation is clear from the context.
With this more compact notation, we can write the two fundamental rules of
probability theory in the following form.
The Rules of Probability
sum rule p(X)=∑
Yp(X,Y) (1.10)

product rule p(X,Y)=p(Y|X)p(X). (1.11)
Herep(X,Y)is a joint probability and is verbalized as “the probability of Xand
Y”. Similarly, the quantity p(Y|X)is a conditional probability and is verbalized as
“the probability of YgivenX”, whereas the quantity p(X)is a marginal probability

1.2. Probability Theory 15
and is simply “the probability of X”. These two simple rules form the basis for all
of the probabilistic machinery that we use throughout this book.
From the product rule, together with the symmetry property p(X,Y)=p(Y,X),
we immediately obtain the following relationship between conditional probabilities
p(Y|X)=p(X|Y)p(Y)
p(X)(1.12)
which is called Bayes’ theorem and which plays a central role in pattern recognition
and machine learning. Using the sum rule, the denominator in Bayes’ theorem can
be expressed in terms of the quantities appearing in the numerator
p(X)=∑
Yp(X|Y)p(Y). (1.13)
We can view the denominator in Bayes’ theorem as being the normalization constant
required to ensure that the sum of the conditional probability on the left-hand side of
(1.12) over all values of Yequals one.
In Figure 1.11, we show a simple example involving a joint distribution over two
variables to illustrate the concept of marginal and conditional distributions. Here

p(X)=∑
a ﬁnite sample of N=6 0 data points has been drawn from the joint distribution
and is shown in the top left. In the top right is a histogram of the fractions of data
points having each of the two values of Y. From the deﬁnition of probability, these
fractions would equal the corresponding probabilities p(Y)in the limit N→∞ .W e
can view the histogram as a simple way to model a probability distribution given only
a ﬁnite number of points drawn from that distribution. Modelling distributions from

data lies at the heart of statistical pattern recognition and will be explored in greatdetail in this book. The remaining two plots in Figure 1.11 show the corresponding
histogram estimates of p(X)andp(X|Y=1 ) .
Let us now return to our example involving boxes of fruit. For the moment, we
shall once again be explicit about distinguishing between the random variables and
their instantiations. We have seen that the probabilities of selecting either the red or
the blue boxes are given by
p(B=r)=4 /10 (1.14)

p(B=b)=6 /10 (1.15)
respectively. Note that these satisfy p(B=r)+p(B=b)=1 .
Now suppose that we pick a box at random, and it turns out to be the blue box.
Then the probability of selecting an apple is just the fraction of apples in the blue
box which is 3/4, and so p(F=a|
B=b)=3/4. In fact, we can write out all four
conditional probabilities for the type of fruit, given the selected box
p(F=a|B=r)=1 /4 (1.16)
p(F=o|B=r)=3 /4 (1.17)
p(F=a|B=b)=3 /4 (1.18)
p(F=o|B=b)=1 /4. (1.19)

16 1. INTRODUCTION
p(X,Y)
XY=2
Y=1p(Y)
p(X)
X Xp(X|Y=1 )
Figure 1.11 An illustration of a distribution over two variables, X, which takes 9possible values, and Y, which
takes two possible values. The top left ﬁgure shows a sample of 60points drawn from a joint probability distri-
bution over these variables. The remaining ﬁgures show histogram estimates of the marginal distributions p(X)
andp(Y), as well as the conditional distribution p(X|Y=1 ) corresponding to the bottom row in the top left
ﬁgure.
Again, note that these probabilities are normalized so that
p(F=a|B=r)+p(F=o|B=r)=1 (1.20)
and similarly
p(F=a|B=b)+p(F=o|B=b)=1. (1.21)
We can now use the sum and product rules of probability to evaluate the overall
probability of choosing an apple
p(F=a)= p(F=a|B=r)p(B=r)+p(F=a|B=b)p(B=b)
=1
4×4
10+3
4×6
10=11
20(1.22)
from which it follows, using the sum rule, that p(F=o)=1−11/20 = 9 /20.

1.2. Probability Theory 17
Suppose instead we are told that a piece of fruit has been selected and it is an
orange, and we would like to know which box it came from. This requires thatwe evaluate the probability distribution over boxes conditioned on the identity of
the fruit, whereas the probabilities in (1.16)–(1.19) give the probability distribution
over the fruit conditioned on the identity of the box. We can solve the problem ofreversing the conditional probability by using Bayes’ theorem to give
p(B=r|F=o)=p(F=o|B=r)p(B=r)
p(F=o)=3
4×4
10×20
9=2
3. (1.23)
From the sum rule, it then follows that p(B=b|F=o)=1−2/3=1/3.
We can provide an important interpretation of Bayes’ theorem as follows. If
we had been asked which box had been chosen before being told the identity ofthe selected item of fruit, then the most complete information we have available is
provided by the probability p(B). We call this the prior probability because it is the

p(F=o)=3
4×4
10×20
9=2
3. (1.23)
probability available before we observe the identity of the fruit. Once we are told that
the fruit is an orange, we can then use Bayes’ theorem to compute the probability
p(B|F), which we shall call the posterior probability because it is the probability
obtained after we have observed F. Note that in this example, the prior probability
of selecting the red box was 4/10, so that we were more likely to select the blue box
than the red one. However, once we have observed that the piece of selected fruit is

an orange, we ﬁnd that the posterior probability of the red box is now 2/3, so that
it is now more likely that the box we selected was in fact the red one. This result
accords with our intuition, as the proportion of oranges is much higher in the red box
than it is in the blue box, and so the observation that the fruit was an orange providessigniﬁcant evidence favouring the red box. In fact, the evidence is sufﬁciently strong
that it outweighs the prior and makes it more likely that the red box was chosen

rather than the blue one.
Finally, we note that if the joint distribution of two variables factorizes into the
product of the marginals, so that p(X,Y)=p(X)p(Y), then XandYare said to
beindependent . From the product rule, we see that p(Y|X)=p(Y), and so the
conditional distribution of Ygiven
Xis indeed independent of the value of X.F o r
instance, in our boxes of fruit example, if each box contained the same fraction ofapples and oranges, then p(F|B)=P(F), so that the probability of selecting, say,

an apple is independent of which box is chosen.
1.2.1 Probability densities
As well as considering probabilities deﬁned over discrete sets of events, we
also wish to consider probabilities with respect to continuous variables. We shall
limit ourselves to a relatively informal discussion. If the probability of a real-valued
variable xfalling in the interval (x, x+δx)is given by p(x)δxforδx→0, then
p(x)is called the probability density overx. This is illustrated in Figure 1.12. The

probability that xwill lie in an interval (a, b)is then given by
p(x∈(a, b)) =∫b
ap(x)dx. (1.24)

18 1. INTRODUCTION
Figure 1.12 The concept of probability for
discrete variables can be ex-
tended to that of a probability
density p(x)over a continuous
variable xand is such that the
probability of xlying in the inter-
val(x, x+δx)is given by p(x)δx
forδx→0. The probability
density can be expressed as the
derivative of a cumulative distri-
bution function P(x).
x δxp(x)P(x)
Because probabilities are nonnegative, and because the value of xmust lie some-
where on the real axis, the probability density p(x)must satisfy the two conditions
p(x)⩾0 (1.25)∫∞
−∞p(x)dx=1. (1.26)
Under a nonlinear change of variable, a probability density transforms differently
from a simple function, due to the Jacobian factor. For instance, if we consider
a change of variables x=g(y), then a function f(x)becomes˜f(y)=f(g(y)).
Now consider a probability density px(x)that corresponds to a density py(y)with
respect to the new variable y, where the sufﬁces denote the fact that px(x)andpy(y)

are different densities. Observations falling in the range (x, x+δx)will, for small
values of δx, be transformed into the range (y,y+δy)where px(x)δx≃py(y)δy,
and hence
py(y)= px(x)⏐⏐⏐⏐dx
dy⏐⏐⏐⏐
=px(g(y))|g′(y)|. (1.27)
One consequence of this property is that the concept of the maximum of a probability
density is dependent on the choice of variable. Exercise 1.4
The probability that xlies in the interval (−∞,z)is given by the cumulative
distribution function deﬁned by
P(z)=∫z
−∞p(x)dx (1.28)
which satisﬁes P′(x)=p(x), as shown in Figure 1.12.

and hence
dy⏐⏐⏐⏐
P(z)=∫z
If we have several continuous variables x1,...,x D, denoted collectively by the
vectorx, then we can deﬁne a joint probability density p(x)=p(x1,...,x D)such

1.2. Probability Theory 19
that the probability of xfalling in an inﬁnitesimal volume δxcontaining the point x
is given by p(x)δx. This multivariate probability density must satisfy
p(x)⩾0 (1.29)∫
p(x)dx=1 (1.30)
in which the integral is taken over the whole of xspace. We can also consider joint
probability distributions over a combination of discrete and continuous variables.
Note that if x is a discrete variable, then p(x)is sometimes called a probability
mass function because it can be regarded as a set of ‘probability masses’ concentrated
at the allowed values of x.
The sum and product rules of probability, as well as Bayes’ theorem, apply
equally to the case of probability densities, or to combinations of discrete and con-
tinuous variables. For instance, if xandyare two real variables, then the sum and
product rules take the form
p(x)=∫
p(x, y)dy (1.31)
p(x, y)= p(y|x)p(x). (1.32)
A formal justiﬁcation of the sum and product rules for continuous variables (Feller,

p(x)=∫
1966) requires a branch of mathematics called measure theory and lies outside the
scope of this book. Its validity can be seen informally, however, by dividing eachreal variable into intervals of width ∆and considering the discrete probability dis-
tribution over these intervals. Taking the limit ∆→0then turns sums into integrals
and gives the desired result.
1.2.2 Expectations and covariances
One of the most important operations involving probabilities is that of ﬁnding
weighted averages of functions. The average value of some function f(x)under a

probability distribution p(x)is called the expectation off(x)and will be denoted by
E[f]. For a discrete distribution, it is given by
E[f]=∑
xp(x)f(x) (1.33)
so that the average is weighted by the relative probabilities of the different values
ofx. In the case of continuous variables, expectations are expressed in terms of an
integration with respect to the corresponding probability density
E[f]=∫
p(x)f(x)dx. (1.34)
In either case, if we are given a ﬁnite number Nof points drawn from the probability

E[f]=∑
E[f]=∫
distribution or probability density, then the expectation can be approximated as a

20 1. INTRODUCTION
ﬁnite sum over these points
E[f]≃1
NN∑
n=1f(xn). (1.35)
We shall make extensive use of this result when we discuss sampling methods in
Chapter 11. The approximation in (1.35) becomes exact in the limit N→∞ .
Sometimes we will be considering expectations of functions of several variables,
in which case we can use a subscript to indicate which variable is being averaged
over, so that for instance
Ex[f(x, y)] (1.36)
denotes the average of the function f(x, y)with respect to the distribution of x. Note
that Ex[f(x, y)]will be a function of y.
We can also consider a conditional expectation with respect to a conditional
distribution, so that
Ex[f|y]=∑
xp(x|y)f(x) (1.37)
with an analogous deﬁnition for continuous variables.
The variance off(x)is deﬁned by
var[f]= E[
(f(x)−E[f(x)])2]
(1.38)
and provides a measure of how much variability there is in f(x)around its mean
value E[f(x)]. Expanding out the square, we see that the variance can also be written

Ex[f|y]=∑
var[f]= E[
(1.38)
in terms of the expectations of f(x)andf(x)2Exercise 1.5
var[f]= E[f(x)2]−E[f(x)]2. (1.39)
In particular, we can consider the variance of the variable xitself, which is given by
var[x]= E[x2]−E[x]2. (1.40)
For two random variables xandy, the covariance is deﬁned by
cov[x, y]= Ex,y[{x−E[x]}{y−E[y]}]
= Ex,y[xy]−E[x]E[y] (1.41)
which expresses the extent to which xandyvary together. If xandyare indepen-
dent, then their covariance vanishes. Exercise 1.6
In the case of two vectors of random variables xandy, the covariance is a matrix
cov[x,y]= Ex,y[

{x−E[x]}{yT−E[yT]}]
= Ex,y[xyT]−E[x]E[yT]. (1.42)
If we consider the covariance of the components of a vector xwith each other, then
we use a slightly simpler notation cov[x]≡cov[x,x].

1.2. Probability Theory 21
1.2.3 Bayesian probabilities
So far in this chapter, we have viewed probabilities in terms of the frequencies
of random, repeatable events. We shall refer to this as the classical orfrequentist
interpretation of probability. Now we turn to the more general Bayesian view, in
which probabilities provide a quantiﬁcation of uncertainty.
Consider an uncertain event, for example whether the moon was once in its own
orbit around the sun, or whether the Arctic ice cap will have disappeared by the end
of the century. These are not events that can be repeated numerous times in order
to deﬁne a notion of probability as we did earlier in the context of boxes of fruit.
Nevertheless, we will generally have some idea, for example, of how quickly wethink the polar ice is melting. If we now obtain fresh evidence, for instance from a
new Earth observation satellite gathering novel forms of diagnostic information, we

may revise our opinion on the rate of ice loss. Our assessment of such matters willaffect the actions we take, for instance the extent to which we endeavour to reduce
the emission of greenhouse gasses. In such circumstances, we would like to be able
to quantify our expression of uncertainty and make precise revisions of uncertainty inthe light of new evidence, as well as subsequently to be able to take optimal actions
or decisions as a consequence. This can all be achieved through the elegant, and very

general, Bayesian interpretation of probability.
The use of probability to represent uncertainty, however, is not an ad-hoc choice,
but is inevitable if we are to respect common sense while making rational coherent
inferences. For instance, Cox (1946) showed that if numerical values are used torepresent degrees of belief, then a simple set of axioms encoding common sense
properties of such beliefs leads uniquely to a set of rules for manipulating degrees of

belief that are equivalent to the sum and product rules of probability. This providedthe ﬁrst rigorous proof that probability theory could be regarded as an extension of
Boolean logic to situations involving uncertainty (Jaynes, 2003). Numerous other
authors have proposed different sets of properties or axioms that such measures of
uncertainty should satisfy (Ramsey, 1931; Good, 1950; Savage, 1961; deFinetti,

1970; Lindley, 1982). In each case, the resulting numerical quantities behave pre-cisely according to the rules of probability. It is therefore natural to refer to these
quantities as (Bayesian) probabilities.
In the ﬁeld of pattern recognition, too, it is helpful to have a more general no-
Thomas Bayes
1701–1761
Thomas Bayes was born in Tun-
bridge Wells and was a clergymanas well as an amateur scientist anda mathematician. He studied logicand theology at Edinburgh Univer-sity and was elected Fellow of the

1701–1761
Royal Society in 1742. During the 18
thcentury, is-
sues regarding probability arose in connection withgambling and with the new concept of insurance. One
particularly important problem concerned so-called in-verse probability. A solution was proposed by ThomasBayes in his paper ‘Essay towards solving a problemin the doctrine of chances’, which was published in1764, some three years after his death, in the
Philo-
sophical Transactions of the Royal Society . In fact,
Bayes only formulated his theory for the case of a uni-form prior, and it was Pierre-Simon Laplace who inde-pendently rediscovered the theory in general form andwho demonstrated its broad applicability.

22 1. INTRODUCTION
tion of probability. Consider the example of polynomial curve ﬁtting discussed in
Section 1.1. It seems reasonable to apply the frequentist notion of probability to therandom values of the observed variables t
n. However, we would like to address and
quantify the uncertainty that surrounds the appropriate choice for the model param-
etersw. We shall see that, from a Bayesian perspective, we can use the machinery
of probability theory to describe the uncertainty in model parameters such as w,o r
indeed in the choice of model itself.
Bayes’ theorem now acquires a new signiﬁcance. Recall that in the boxes of fruit
example, the observation of the identity of the fruit provided relevant information
that altered the probability that the chosen box was the red one. In that example,
Bayes’ theorem was used to convert a prior probability into a posterior probability
by incorporating the evidence provided by the observed data. As we shall see in

detail later, we can adopt a similar approach when making inferences about quantitiessuch as the parameters win the polynomial curve ﬁtting example. We capture our
assumptions about w, before observing the data, in the form of a prior probability
distribution p(w). The effect of the observed data D={t
1,...,t N}is expressed
through the conditional probability p(D|w), and we shall see later, in Section 1.2.5,
how this can be represented explicitly. Bayes’ theorem, which takes the form
p(w|D)=p(D|w)p(w)
p(D)(1.43)

then allows us to evaluate the uncertainty in wafter we have observed Din the form
of the posterior probability p(w|D).
The quantity p(D|w)on the right-hand side of Bayes’ theorem is evaluated for
the observed data set Dand can be viewed as a function of the parameter vector
w, in which case it is called the likelihood function . It expresses how probable the
observed data set is for different settings of the parameter vector w. Note that the
likelihood is not a probability distribution over w, and its integral with respect to w
does not (necessarily) equal one.
Given this deﬁnition of likelihood, we can state Bayes’ theorem in words

posterior ∝likelihood ×prior (1.44)
where all of these quantities are viewed as functions of w. The denominator in
(1.43) is the normalization constant, which ensures that the posterior distributionon the left-hand side is a valid probability density and integrates to one. Indeed,
integrating both sides of (1.43) with respect to w, we can express the denominator
in Bayes’ theorem in terms of the prior distribution and the likelihood function
p(D)=∫
p(D|w)p(w)dw. (1.45)
In both the Bayesian and frequentist paradigms, the likelihood function p(D|w)

p(D)=∫
plays a central role. However, the manner in which it is used is fundamentally dif-ferent in the two approaches. In a frequentist setting, wis considered to be a ﬁxed
parameter, whose value is determined by some form of ‘estimator’, and error bars

1.2. Probability Theory 23
on this estimate are obtained by considering the distribution of possible data sets D.
By contrast, from the Bayesian viewpoint there is only a single data set D(namely
the one that is actually observed), and the uncertainty in the parameters is expressed
through a probability distribution over w.
A widely used frequentist estimator is maximum likelihood , in which wis set
to the value that maximizes the likelihood function p(D|w). This corresponds to
choosing the value of wfor which the probability of the observed data set is maxi-
mized. In the machine learning literature, the negative log of the likelihood functionis called an error function . Because the negative logarithm is a monotonically de-
creasing function, maximizing the likelihood is equivalent to minimizing the error.
One approach to determining frequentist error bars is the bootstrap (Efron, 1979;
Hastie et al. , 2001), in which multiple data sets are created as follows. Suppose our

original data set consists of Ndata points X={x
1,...,xN}. We can create a new
data set XBby drawing Npoints at random from X, with replacement, so that some
points in Xmay be replicated in XB, whereas other points in Xmay be absent from
XB. This process can be repeated Ltimes to generate Ldata sets each of size Nand
each obtained by sampling from the original data set X. The statistical accuracy of
parameter estimates can then be evaluated by looking at the variability of predictions

between the different bootstrap data sets.
One advantage of the Bayesian viewpoint is that the inclusion of prior knowl-
edge arises naturally. Suppose, for instance, that a fair-looking coin is tossed three
times and lands heads each time. A classical maximum likelihood estimate of theprobability of landing heads would give 1, implying that all future tosses will land Section 2.1
heads! By contrast, a Bayesian approach with any reasonable prior will lead to a
much less extreme conclusion.

There has been much controversy and debate associated with the relative mer-
its of the frequentist and Bayesian paradigms, which have not been helped by the
fact that there is no unique frequentist, or even Bayesian, viewpoint. For instance,
one common criticism of the Bayesian approach is that the prior distribution is of-
ten selected on the basis of mathematical convenience rather than as a reﬂection ofany prior beliefs. Even the subjective nature of the conclusions through their de-

pendence on the choice of prior is seen by some as a source of difﬁculty. Reducing
the dependence on the prior is one motivation for so-called noninformative priors. Section 2.4.3
However, these lead to difﬁculties when comparing different models, and indeed
Bayesian methods based on poor choices of prior can give poor results with high
conﬁdence. Frequentist evaluation methods offer some protection from such prob-lems, and techniques such as cross-validation remain useful in areas such as model Section 1.3
comparison.

This book places a strong emphasis on the Bayesian viewpoint, reﬂecting the
huge growth in the practical importance of Bayesian methods in the past few years,
while also discussing useful frequentist concepts as required.
Although the Bayesian framework has its origins in the 18
thcentury, the prac-
tical application of Bayesian methods was for a long time severely limited by the
difﬁculties in carrying through the full Bayesian procedure, particularly the need to
marginalize (sum or integrate) over the whole of parameter space, which, as we shall

24 1. INTRODUCTION
see, is required in order to make predictions or to compare different models. The
development of sampling methods, such as Markov chain Monte Carlo (discussed inChapter 11) along with dramatic improvements in the speed and memory capacity
of computers, opened the door to the practical use of Bayesian techniques in an im-
pressive range of problem domains. Monte Carlo methods are very ﬂexible and canbe applied to a wide range of models. However, they are computationally intensive
and have mainly been used for small-scale problems.
More recently, highly efﬁcient deterministic approximation schemes such as
variational Bayes and expectation propagation (discussed in Chapter 10) have been
developed. These offer a complementary alternative to sampling methods and have
allowed Bayesian techniques to be used in large-scale applications (Blei et al. , 2003).
1.2.4 The Gaussian distribution
We shall devote the whole of Chapter 2 to a study of various probability dis-

tributions and their key properties. It is convenient, however, to introduce here one
of the most important probability distributions for continuous variables, called the
normal orGaussian distribution. We shall make extensive use of this distribution in
the remainder of this chapter and indeed throughout much of the book.
For the case of a single real-valued variable x, the Gaussian distribution is de-
ﬁned by
N(
x|µ, σ2)
=1
(2πσ2)1/2exp{
−1
2σ2(x−µ)2}
(1.46)

ﬁned by
N(
x|µ, σ2)
=1
−1
(1.46)
which is governed by two parameters: µ, called the mean , andσ2, called the vari-
ance . The square root of the variance, given by σ, is called the standard deviation ,
and the reciprocal of the variance, written as β=1/σ2, is called the precision .W e
shall see the motivation for these terms shortly. Figure 1.13 shows a plot of the
From the form of (1.46) we see that the Gaussian distribution satisﬁes
N(x|µ, σ2)>0. (1.47)
Also it is straightforward to show that the Gaussian is normalized, so that Exercise 1.7

Pierre-Simon Laplace
1749–1827
It is said that Laplace was seri-
ously lacking in modesty and at onepoint declared himself to be thebest mathematician in France at thetime, a claim that was arguably true.As well as being proliﬁc in mathe-
matics, he also made numerous contributions to as-tronomy, including the nebular hypothesis by which theearth is thought to have formed from the condensa-
tion and cooling of a large rotating disk of gas anddust. In 1812 he published the ﬁrst edition of
Th´eorie
Analytique des Probabilit ´es, in which Laplace states

Th´eorie
that “probability theory is nothing but common sensereduced to calculation”. This work included a discus-sion of the inverse probability calculation (later termedBayes’ theorem by Poincar ´e), which he used to solve
problems in life expectancy, jurisprudence, planetarymasses, triangulation, and error estimation.

1.2. Probability Theory 25
Figure 1.13 Plot of the univariate Gaussian
showing the mean µand the
standard deviation σ.N(x|µ, σ2)
x2σ
µ
∫∞
−∞N(
x|µ, σ2)
dx=1. (1.48)
Thus (1.46) satisﬁes the two requirements for a valid probability density.
We can readily ﬁnd expectations of functions of xunder the Gaussian distribu-
tion. In particular, the average value of xis given by Exercise 1.8
E[x]=∫∞
−∞N(
x|µ, σ2)
xdx=µ. (1.49)
Because the parameter µrepresents the average value of xunder the distribution, it
is referred to as the mean. Similarly, for the second order moment
E[x2]=∫∞
−∞N(
x|µ, σ2)
x2dx=µ2+σ2. (1.50)
From (1.49) and (1.50), it follows that the variance of xis given by
var[x]= E[x2]−E[x]2=σ2(1.51)
and hence σ2is referred to as the variance parameter. The maximum of a distribution
is known as its mode. For a Gaussian, the mode coincides with the mean. Exercise 1.9
We are also interested in the Gaussian distribution deﬁned over a D-dimensional
vectorxof continuous variables, which is given by
N(x|µ,Σ)=1

E[x2]=∫∞
−∞N(
x|µ, σ2)
(2π)D/21
|Σ|1/2exp{
−1
2(x−µ)TΣ−1(x−µ)}
(1.52)
where the D-dimensional vector µis called the mean, the D×DmatrixΣis called
the covariance, and |Σ|denotes the determinant of Σ. We shall make use of the
multivariate Gaussian distribution brieﬂy in this chapter, although its properties will
be studied in detail in Section 2.3.

26 1. INTRODUCTION
Figure 1.14 Illustration of the likelihood function for
a Gaussian distribution, shown by the
red curve. Here the black points de-
note a data set of values {xn}, and
the likelihood function given by (1.53)
corresponds to the product of the blue
values. Maximizing the likelihood in-
volves adjusting the mean and vari-
ance of the Gaussian so as to maxi-
mize this product.
xp(x)
xnN(xn|µ, σ2)
Now suppose that we have a data set of observations x=(x1,...,x N)T, rep-
resenting Nobservations of the scalar variable x. Note that we are using the type-
face xto distinguish this from a single observation of the vector-valued variable
(x1,...,x D)T, which we denote by x. We shall suppose that the observations are
drawn independently from a Gaussian distribution whose mean µand variance σ2
are unknown, and we would like to determine these parameters from the data set.
Data points that are drawn independently from the same distribution are said to be

independent and identically distributed , which is often abbreviated to i.i.d. We have
seen that the joint probability of two independent events is given by the product of
the marginal probabilities for each event separately. Because our data set xis i.i.d.,
we can therefore write the probability of the data set, given µandσ2, in the form
p(x|µ, σ2)=N∏
n=1N(
xn|µ, σ2)
. (1.53)
When viewed as a function of µandσ2, this is the likelihood function for the Gaus-
sian and is interpreted diagrammatically in Figure 1.14.

n=1N(
xn|µ, σ2)
. (1.53)
One common criterion for determining the parameters in a probability distribu-
tion using an observed data set is to ﬁnd the parameter values that maximize the
likelihood function. This might seem like a strange criterion because, from our fore-
going discussion of probability theory, it would seem more natural to maximize the
probability of the parameters given the data, not the probability of the data given the
parameters. In fact, these two criteria are related, as we shall discuss in the context
of curve ﬁtting. Section 1.2.5

For the moment, however, we shall determine values for the unknown parame-
tersµandσ2in the Gaussian by maximizing the likelihood function (1.53). In prac-
tice, it is more convenient to maximize the log of the likelihood function. Because
the logarithm is a monotonically increasing function of its argument, maximization
of the log of a function is equivalent to maximization of the function itself. Taking
the log not only simpliﬁes the subsequent mathematical analysis, but it also helps

numerically because the product of a large number of small probabilities can easily
underﬂow the numerical precision of the computer, and this is resolved by computing
instead the sum of the log probabilities. From (1.46) and (1.53), the log likelihood

1.2. Probability Theory 27
function can be written in the form
lnp(
x|µ, σ2)
=−1
2σ2N∑
n=1(xn−µ)2−N
2lnσ2−N
2ln(2π). (1.54)
Maximizing (1.54) with respect to µ, we obtain the maximum likelihood solution
given by Exercise 1.11
µML=1
NN∑
n=1xn (1.55)
which is the sample mean , i.e., the mean of the observed values {xn}. Similarly,
maximizing (1.54) with respect to σ2, we obtain the maximum likelihood solution
for the variance in the form
σ2
ML=1
NN∑
n=1(xn−µML)2(1.56)
which is the sample variance measured with respect to the sample mean µML. Note
that we are performing a joint maximization of (1.54) with respect to µandσ2,b u t
in the case of the Gaussian distribution the solution for µdecouples from that for σ2
so that we can ﬁrst evaluate (1.55) and then subsequently use this result to evaluate
(1.56).
Later in this chapter, and also in subsequent chapters, we shall highlight the sig-
niﬁcant limitations of the maximum likelihood approach. Here we give an indication

(1.56).
of the problem in the context of our solutions for the maximum likelihood param-
eter settings for the univariate Gaussian distribution. In particular, we shall show
that the maximum likelihood approach systematically underestimates the varianceof the distribution. This is an example of a phenomenon called bias and is related
to the problem of over-ﬁtting encountered in the context of polynomial curve ﬁtting. Section 1.1
We ﬁrst note that the maximum likelihood solutions µ
MLandσ2
MLare functions of

MLandσ2
the data set values x1,...,x N. Consider the expectations of these quantities with
respect to the data set values, which themselves come from a Gaussian distribution
with parameters µandσ2. It is straightforward to show that Exercise 1.12
E[µML]= µ (1.57)
E[σ2
ML]=(N−1
N)
σ2(1.58)
so that on average the maximum likelihood estimate will obtain the correct mean but
will underestimate the true variance by a factor (N−1)/N. The intuition behind
this result is given by Figure 1.15.

MLandσ2
E[σ2
ML]=(N−1
N)
σ2(1.58)
From (1.58) it follows that the following estimate for the variance parameter is
unbiased
˜σ2=N
N−1σ2
ML=1
N−1N∑
n=1(xn−µML)2. (1.59)

28 1. INTRODUCTION
Figure 1.15 Illustration of how bias arises in using max-
imum likelihood to determine the variance
of a Gaussian. The green curve shows
the true Gaussian distribution from which
data is generated, and the three red curves
show the Gaussian distributions obtained
by ﬁtting to three data sets, each consist-
ing of two data points shown in blue, us-
ing the maximum likelihood results (1.55)
and (1.56). Averaged across the three data
sets, the mean is correct, but the variance
is systematically under-estimated because
it is measured relative to the sample mean
and not relative to the true mean.(a)
(b)
(c)
In Section 10.1.3, we shall see how this result arises automatically when we adopt a
Bayesian approach.
Note that the bias of the maximum likelihood solution becomes less signiﬁcant
as the number Nof data points increases, and in the limit N→∞ the maximum
likelihood solution for the variance equals the true variance of the distribution that

(b)
(c)
generated the data. In practice, for anything other than small N, this bias will not
prove to be a serious problem. However, throughout this book we shall be interested
in more complex models with many parameters, for which the bias problems asso-
ciated with maximum likelihood will be much more severe. In fact, as we shall see,
the issue of bias in maximum likelihood lies at the root of the over-ﬁtting problem
that we encountered earlier in the context of polynomial curve ﬁtting.
1.2.5 Curve ﬁtting re-visited

We have seen how the problem of polynomial curve ﬁtting can be expressed in
terms of error minimization. Here we return to the curve ﬁtting example and view it Section 1.1
from a probabilistic perspective, thereby gaining some insights into error functions
and regularization, as well as taking us towards a full Bayesian treatment.
The goal in the curve ﬁtting problem is to be able to make predictions for the
target variable tgiven some new value of the input variable xon the basis of a set of
training data comprising Ninput values x=(x1,...,x N)Tand their corresponding

target values t=(t1,...,t N)T. We can express our uncertainty over the value of
the target variable using a probability distribution. For this purpose, we shall assume
that, given the value of x, the corresponding value of thas a Gaussian distribution
with a mean equal to the value y(x,w)of the polynomial curve given by (1.1). Thus
we have
p(t|x,w,β)=N(
t|y(x,w),β−1)
(1.60)
where, for consistency with the notation in later chapters, we have deﬁned a preci-

we have
(1.60)
sion parameter βcorresponding to the inverse variance of the distribution. This is
illustrated schematically in Figure 1.16.

1.2. Probability Theory 29
Figure 1.16 Schematic illustration of a Gaus-
sian conditional distribution for tgiven xgiven by
(1.60), in which the mean is given by the polyno-
mial function y(x,w), and the precision is given
by the parameter β, which is related to the vari-
ance by β−1=σ2.t
x x02σ y(x0,w)y(x,w)
p(t|x0,w,β)
We now use the training data {x,t}to determine the values of the unknown
parameters wandβby maximum likelihood. If the data are assumed to be drawn
independently from the distribution (1.60), then the likelihood function is given by
p(t|x,w,β)=N∏
n=1N(
tn|y(xn,w),β−1)
. (1.61)
As we did in the case of the simple Gaussian distribution earlier, it is convenient to
maximize the logarithm of the likelihood function. Substituting for the form of the
Gaussian distribution, given by (1.46), we obtain the log likelihood function in the
form
lnp(t|x,w,β)=−β
2N∑
n=1{y(xn,w)−tn}2+N
2lnβ−N
2ln(2π). (1.62)
Consider ﬁrst the determination of the maximum likelihood solution for the polyno-

n=1N(
. (1.61)
form
2N∑
2lnβ−N
mial coefﬁcients, which will be denoted by wML. These are determined by maxi-
mizing (1.62) with respect to w. For this purpose, we can omit the last two terms
on the right-hand side of (1.62) because they do not depend on w. Also, we note
that scaling the log likelihood by a positive constant coefﬁcient does not alter the
location of the maximum with respect to w, and so we can replace the coefﬁcient
β/2with1/2. Finally, instead of maximizing the log likelihood, we can equivalently
minimize the negative log likelihood. We therefore see that maximizing likelihood is

equivalent, so far as determining wis concerned, to minimizing the sum-of-squares
error function deﬁned by (1.2). Thus the sum-of-squares error function has arisen as
a consequence of maximizing likelihood under the assumption of a Gaussian noise
We can also use maximum likelihood to determine the precision parameter βof
the Gaussian conditional distribution. Maximizing (1.62) with respect to βgives
1
βML=1
NN∑
n=1{y(xn,wML)−tn}2. (1.63)

30 1. INTRODUCTION
Again we can ﬁrst determine the parameter vector wMLgoverning the mean and sub-
sequently use this to ﬁnd the precision βMLas was the case for the simple Gaussian
distribution. Section 1.2.4
Having determined the parameters wandβ, we can now make predictions for
new values of x. Because we now have a probabilistic model, these are expressed
in terms of the predictive distribution that gives the probability distribution over t,
rather than simply a point estimate, and is obtained by substituting the maximum
likelihood parameters into (1.60) to give
p(t|x,wML,βML)=N(
t|y(x,wML),β−1
ML)
. (1.64)
Now let us take a step towards a more Bayesian approach and introduce a prior
distribution over the polynomial coefﬁcients w. For simplicity, let us consider a
Gaussian distribution of the form
p(w|α)=N(w|0,α−1I)=(α
2π)(M+1)/2
exp{
−α
2wTw}
(1.65)
where αis the precision of the distribution, and M+1 is the total number of elements

ML)
. (1.64)
exp{
−α
2wTw}
(1.65)
in the vector wfor an Mthorder polynomial. V ariables such as α, which control
the distribution of model parameters, are called hyperparameters . Using Bayes’
theorem, the posterior distribution for wis proportional to the product of the prior
distribution and the likelihood function
p(w|x,t,α,β)∝p(t|x,w,β)p(w|α). (1.66)
We can now determine wby ﬁnding the most probable value of wgiven the data,
in other words by maximizing the posterior distribution. This technique is called

maximum posterior , or simply MAP . Taking the negative logarithm of (1.66) and
combining with (1.62) and (1.65), we ﬁnd that the maximum of the posterior isgiven by the minimum of
β
2N∑
n=1{y(xn,w)−tn}2+α
2wTw. (1.67)
Thus we see that maximizing the posterior distribution is equivalent to minimizing
the regularized sum-of-squares error function encountered earlier in the form (1.4),with a regularization parameter given by λ=α/β .
1.2.6 Bayesian curve ﬁtting
Although we have included a prior distribution p(w|α), we are so far still mak-

β
2N∑
ing a point estimate of wand so this does not yet amount to a Bayesian treatment. In
a fully Bayesian approach, we should consistently apply the sum and product rules
of probability, which requires, as we shall see shortly, that we integrate over all val-ues of w. Such marginalizations lie at the heart of Bayesian methods for pattern
recognition.

1.2. Probability Theory 31
In the curve ﬁtting problem, we are given the training data xand t, along with
a new test point x, and our goal is to predict the value of t. We therefore wish
to evaluate the predictive distribution p(t|x,x,t). Here we shall assume that the
parameters αandβare ﬁxed and known in advance (in later chapters we shall discuss
how such parameters can be inferred from data in a Bayesian setting).
A Bayesian treatment simply corresponds to a consistent application of the sum
and product rules of probability, which allow the predictive distribution to be written
in the form
p(t|x,x,t)=∫
p(t|x,w)p(w|x,t)dw. (1.68)
Herep(t|x,w)is given by (1.60), and we have omitted the dependence on αand
βto simplify the notation. Here p(w|x,t)is the posterior distribution over param-
eters, and can be found by normalizing the right-hand side of (1.66). We shall seein Section 3.3 that, for problems such as the curve-ﬁtting example, this posterior

in the form
distribution is a Gaussian and can be evaluated analytically. Similarly, the integra-
tion in (1.68) can also be performed analytically with the result that the predictive
distribution is given by a Gaussian of the form
p(t|x,x,t)=N(
t|m(x),s2(x))
(1.69)
where the mean and variance are given by
m(x)= βφ(x)TSN∑
n=1φ(xn)tn (1.70)
s2(x)= β−1+φ(x)TSφ(x). (1.71)
Here the matrix Sis given by
S−1=αI+βN∑
n=1φ(xn)φ(x)T(1.72)
whereIis the unit matrix, and we have deﬁned the vector φ(x)with elements
φi(x)=xifori=0,...,M .

(1.69)
We see that the variance, as well as the mean, of the predictive distribution in
(1.69) is dependent on x. The ﬁrst term in (1.71) represents the uncertainty in the
predicted value of tdue to the noise on the target variables and was expressed already
in the maximum likelihood predictive distribution (1.64) through β−1
ML. However, the
second term arises from the uncertainty in the parameters wand is a consequence
of the Bayesian treatment. The predictive distribution for the synthetic sinusoidalregression problem is illustrated in Figure 1.17.

32 1. INTRODUCTION
Figure 1.17 The predictive distribution result-
ing from a Bayesian treatment of
polynomial curve ﬁtting using an
M=9 polynomial, with the ﬁxed
parameters α=5×10−3andβ=
11.1(corresponding to the known
noise variance), in which the red
curve denotes the mean of the
predictive distribution and the red
region corresponds to ±1stan-
dard deviation around the mean.
xt
0 1−101
1.3. Model Selection
In our example of polynomial curve ﬁtting using least squares, we saw that there was
an optimal order of polynomial that gave the best generalization. The order of the
polynomial controls the number of free parameters in the model and thereby governs
the model complexity. With regularized least squares, the regularization coefﬁcient
λalso controls the effective complexity of the model, whereas for more complex
models, such as mixture distributions or neural networks there may be multiple pa-
rameters governing complexity. In a practical application, we need to determine

the values of such parameters, and the principal objective in doing so is usually to
achieve the best predictive performance on new data. Furthermore, as well as ﬁnd-
ing the appropriate values for complexity parameters within a given model, we may
wish to consider a range of different types of model in order to ﬁnd the best one for
our particular application.
We have already seen that, in the maximum likelihood approach, the perfor-
mance on the training set is not a good indicator of predictive performance on un-

seen data due to the problem of over-ﬁtting. If data is plentiful, then one approach is
simply to use some of the available data to train a range of models, or a given model
with a range of values for its complexity parameters, and then to compare them on
independent data, sometimes called a validation set , and select the one having the
best predictive performance. If the model design is iterated many times using a lim-
ited size data set, then some over-ﬁtting to the validation data can occur and so it may

be necessary to keep aside a third test set on which the performance of the selected
model is ﬁnally evaluated.
In many applications, however, the supply of data for training and testing will be
limited, and in order to build good models, we wish to use as much of the available
data as possible for training. However, if the validation set is small, it will give a
relatively noisy estimate of predictive performance. One solution to this dilemma is
to use cross-validation , which is illustrated in Figure 1.18. This allows a proportion

(S−1)/Sof the available data to be used for training while making use of all of the

1.4. The Curse of Dimensionality 33
Figure 1.18 The technique of S-fold cross-validation, illus-
trated here for the case of S=4, involves tak-
ing the available data and partitioning it into S
groups (in the simplest case these are of equal
size). Then S−1of the groups are used to train
a set of models that are then evaluated on the re-
maining group. This procedure is then repeated
for all Spossible choices for the held-out group,
indicated here by the red blocks, and the perfor-
mance scores from the Sruns are then averaged.run 1
run 2
run 3
run 4
data to assess performance. When data is particularly scarce, it may be appropriate
to consider the case S=N, where Nis the total number of data points, which gives
the leave-one-out technique.
One major drawback of cross-validation is that the number of training runs that
must be performed is increased by a factor of S, and this can prove problematic for
models in which the training is itself computationally expensive. A further problem

run 2
run 3
run 4
with techniques such as cross-validation that use separate data to assess performance
is that we might have multiple complexity parameters for a single model (for in-
stance, there might be several regularization parameters). Exploring combinations
of settings for such parameters could, in the worst case, require a number of training
runs that is exponential in the number of parameters. Clearly, we need a better ap-
proach. Ideally, this should rely only on the training data and should allow multiple

hyperparameters and model types to be compared in a single training run. We there-
fore need to ﬁnd a measure of performance which depends only on the training data
and which does not suffer from bias due to over-ﬁtting.
Historically various ‘information criteria’ have been proposed that attempt to
correct for the bias of maximum likelihood by the addition of a penalty term to
compensate for the over-ﬁtting of more complex models. For example, the Akaike

information criterion , or AIC (Akaike, 1974), chooses the model for which the quan-
tity
lnp(D|wML)−M (1.73)
is largest. Here p(D|wML)is the best-ﬁt log likelihood, and Mis the number of
adjustable parameters in the model. A variant of this quantity, called the Bayesian
information criterion ,o r BIC , will be discussed in Section 4.4.1. Such criteria do
not take account of the uncertainty in the model parameters, however, and in practice
they tend to favour overly simple models. We therefore turn in Section 3.4 to a fully

tity
Bayesian approach where we shall see how complexity penalties arise in a natural
and principled way.
1.4. The Curse of Dimensionality
In the polynomial curve ﬁtting example we had just one input variable x. For prac-
tical applications of pattern recognition, however, we will have to deal with spaces

34 1. INTRODUCTION
Figure 1.19 Scatter plot of the oil ﬂow data
for input variables x6andx7,i n
which red denotes the ‘homoge-
nous’ class, green denotes the
‘annular’ class, and blue denotes
the ‘laminar’ class. Our goal is
to classify the new test point de-
noted by ‘ ×’.
x6x7
0 0.25 0.5 0.75 100.511.52
of high dimensionality comprising many input variables. As we now discuss, this
poses some serious challenges and is an important factor inﬂuencing the design of
pattern recognition techniques.
In order to illustrate the problem we consider a synthetically generated data set
representing measurements taken from a pipeline containing a mixture of oil, wa-
ter, and gas (Bishop and James, 1993). These three materials can be present in one
of three different geometrical conﬁgurations known as ‘homogenous’, ‘annular’, and
‘laminar’, and the fractions of the three materials can also vary. Each data point com-
prises a 12-dimensional input vector consisting of measurements taken with gamma

ray densitometers that measure the attenuation of gamma rays passing along nar-
row beams through the pipe. This data set is described in detail in Appendix A.
Figure 1.19 shows 100 points from this data set on a plot showing two of the mea-
surements x6andx7(the remaining ten input values are ignored for the purposes of
this illustration). Each data point is labelled according to which of the three geomet-
rical classes it belongs to, and our goal is to use this data as a training set in order to

be able to classify a new observation (x6,x7), such as the one denoted by the cross
in Figure 1.19. We observe that the cross is surrounded by numerous red points, and
so we might suppose that it belongs to the red class. However, there are also plenty
of green points nearby, so we might think that it could instead belong to the green
class. It seems unlikely that it belongs to the blue class. The intuition here is that the
identity of the cross should be determined more strongly by nearby points from the

training set and less strongly by more distant points. In fact, this intuition turns out
to be reasonable and will be discussed more fully in later chapters.
How can we turn this intuition into a learning algorithm? One very simple ap-
proach would be to divide the input space into regular cells, as indicated in Fig-
ure 1.20. When we are given a test point and we wish to predict its class, we ﬁrst
decide which cell it belongs to, and we then ﬁnd all of the training data points that

1.4. The Curse of Dimensionality 35
Figure 1.20 Illustration of a simple approach
to the solution of a classiﬁcation
problem in which the input space
is divided into cells and any new
test point is assigned to the class
that has a majority number of rep-
resentatives in the same cell as
the test point. As we shall see
shortly, this simplistic approach
has some severe shortcomings.
x6x7
fall in the same cell. The identity of the test point is predicted as being the same
as the class having the largest number of training points in the same cell as the test
point (with ties being broken at random).
There are numerous problems with this naive approach, but one of the most se-
vere becomes apparent when we consider its extension to problems having larger
numbers of input variables, corresponding to input spaces of higher dimensionality.
The origin of the problem is illustrated in Figure 1.21, which shows that, if we divide

a region of a space into regular cells, then the number of such cells grows exponen-
tially with the dimensionality of the space. The problem with an exponentially large
number of cells is that we would need an exponentially large quantity of training data
in order to ensure that the cells are not empty. Clearly, we have no hope of applying
such a technique in a space of more than a few variables, and so we need to ﬁnd a
more sophisticated approach.
We can gain further insight into the problems of high-dimensional spaces by

returning to the example of polynomial curve ﬁtting and considering how we would Section 1.1
Figure 1.21 Illustration of the
curse of dimensionality, showing
how the number of regions of a
regular grid grows exponentially
with the dimensionality Dof the
space. For clarity, only a subset of
the cubical regions are shown for
D=3.
x1
D=1x1x2
D=2x1x2
x3
D=3

36 1. INTRODUCTION
extend this approach to deal with input spaces having several variables. If we have
Dinput variables, then a general polynomial with coefﬁcients up to order 3would
take the form
y(x,w)=w0+D∑
i=1wixi+D∑
i=1D∑
j=1wijxixj+D∑
i=1D∑
j=1D∑
k=1wijkxixjxk.(1.74)
AsDincreases, so the number of independent coefﬁcients (not all of the coefﬁcients
are independent due to interchange symmetries amongst the xvariables) grows pro-
portionally to D3. In practice, to capture complex dependencies in the data, we may
need to use a higher-order polynomial. For a polynomial of order M, the growth in
the number of coefﬁcients is like DM. Although this is now a power law growth, Exercise 1.16
rather than an exponential growth, it still points to the method becoming rapidlyunwieldy and of limited practical utility.
Our geometrical intuitions, formed through a life spent in a space of three di-

mensions, can fail badly when we consider spaces of higher dimensionality. As asimple example, consider a sphere of radius r=1 in a space of Ddimensions, and
ask what is the fraction of the volume of the sphere that lies between radius r=1−ϵ
andr=1 . We can evaluate this fraction by noting that the volume of a sphere of
radius rinDdimensions must scale as r
D, and so we write
VD(r)=KDrD(1.75)
where the constant KDdepends only on D. Thus the required fraction is given by Exercise 1.18
VD(1)−VD(1−ϵ)
VD(1)=1−(1−ϵ)D(1.76)

which is plotted as a function of ϵfor various values of Din Figure 1.22. We see
that, for large D, this fraction tends to 1even for small values of ϵ. Thus, in spaces
of high dimensionality, most of the volume of a sphere is concentrated in a thin shell
near the surface!
As a further example, of direct relevance to pattern recognition, consider the
behaviour of a Gaussian distribution in a high-dimensional space. If we transform
from Cartesian to polar coordinates, and then integrate out the directional variables,we obtain an expression for the density p(r)as a function of radius rfrom the origin. Exercise 1.20

Thusp(r)δris the probability mass inside a thin shell of thickness δrlocated at
radius r. This distribution is plotted, for various values of D, in Figure 1.23, and we
see that for large Dthe probability mass of the Gaussian is concentrated in a thin
shell.
The severe difﬁculty that can arise in spaces of many dimensions is sometimes
called the curse of dimensionality (Bellman, 1961). In this book, we shall make ex-

shell.
tensive use of illustrative examples involving input spaces of one or two dimensions,because this makes it particularly easy to illustrate the techniques graphically. The
reader should be warned, however, that not all intuitions developed in spaces of low
dimensionality will generalize to spaces of many dimensions.

1.4. The Curse of Dimensionality 37
Figure 1.22 Plot of the fraction of the volume of
a sphere lying in the range r=1−ϵ
tor=1 for various values of the
dimensionality D.
ϵvolume fractionD=1D=2D=5D=2 0
0 0.2 0.4 0.6 0.8 100.20.40.60.81
Although the curse of dimensionality certainly raises important issues for pat-
tern recognition applications, it does not prevent us from ﬁnding effective techniques
applicable to high-dimensional spaces. The reasons for this are twofold. First, real
data will often be conﬁned to a region of the space having lower effective dimension-
ality, and in particular the directions over which important variations in the target
variables occur may be so conﬁned. Second, real data will typically exhibit some
smoothness properties (at least locally) so that for the most part small changes in the
input variables will produce small changes in the target variables, and so we can ex-
ploit local interpolation-like techniques to allow us to make predictions of the target

variables for new values of the input variables. Successful pattern recognition tech-
niques exploit one or both of these properties. Consider, for example, an application
in manufacturing in which images are captured of identical planar objects on a con-
veyor belt, in which the goal is to determine their orientation. Each image is a point
Figure 1.23 Plot of the probability density with
respect to radius rof a Gaus-
sian distribution for various values
of the dimensionality D.I n a
high-dimensional space, most of the
probability mass of a Gaussian is lo-

cated within a thin shell at a speciﬁc
radius.D=1
D=2
D=2 0
rp(r)
0 2 4012

38 1. INTRODUCTION
in a high-dimensional space whose dimensionality is determined by the number of
pixels. Because the objects can occur at different positions within the image andin different orientations, there are three degrees of freedom of variability between
images, and a set of images will live on a three dimensional manifold embedded
within the high-dimensional space. Due to the complex relationships between theobject position or orientation and the pixel intensities, this manifold will be highly
nonlinear. If the goal is to learn a model that can take an input image and output the
orientation of the object irrespective of its position, then there is only one degree offreedom of variability within the manifold that is signiﬁcant.
1.5. Decision Theory
We have seen in Section 1.2 how probability theory provides us with a consistentmathematical framework for quantifying and manipulating uncertainty. Here we
turn to a discussion of decision theory that, when combined with probability theory,

allows us to make optimal decisions in situations involving uncertainty such as thoseencountered in pattern recognition.
Suppose we have an input vector xtogether with a corresponding vector tof
target variables, and our goal is to predict tgiven a new value for x. For regression
problems, twill comprise continuous variables, whereas for classiﬁcation problems
twill represent class labels. The joint probability distribution p(x,t)provides a

complete summary of the uncertainty associated with these variables. Determinationofp(x,t)from a set of training data is an example of inference and is typically a
very difﬁcult problem whose solution forms the subject of much of this book. In
a practical application, however, we must often make a speciﬁc prediction for the
value of t, or more generally take a speciﬁc action based on our understanding of the
valuestis likely to take, and this aspect is the subject of decision theory.
Consider, for example, a medical diagnosis problem in which we have taken an

X-ray image of a patient, and we wish to determine whether the patient has cancer
or not. In this case, the input vector xis the set of pixel intensities in the image,
and output variable twill represent the presence of cancer, which we denote by the
classC
1, or the absence of cancer, which we denote by the class C2. We might, for
instance, choose tto be a binary variable such that t=0 corresponds to class C1and
t=1 corresponds to class C2. We shall see later that this choice of label values is
particularly convenient for probabilistic models. The general inference problem then

classC
involves determining the joint distribution p(x,Ck), or equivalently p(x,t), which
gives us the most complete probabilistic description of the situation. Although this
can be a very useful and informative quantity, in the end we must decide either to
give treatment to the patient or not, and we would like this choice to be optimalin some appropriate sense (Duda and Hart, 1973). This is the decision step, and
it is the subject of decision theory to tell us how to make optimal decisions given

the appropriate probabilities. We shall see that the decision stage is generally very
simple, even trivial, once we have solved the inference problem.
Here we give an introduction to the key ideas of decision theory as required for

1.5. Decision Theory 39
the rest of the book. Further background, as well as more detailed accounts, can be
found in Berger (1985) and Bather (2000).
Before giving a more detailed analysis, let us ﬁrst consider informally how we
might expect probabilities to play a role in making decisions. When we obtain the
X-ray image xfor a new patient, our goal is to decide which of the two classes to
assign to the image. We are interested in the probabilities of the two classes given
the image, which are given by p(Ck|x). Using Bayes’ theorem, these probabilities
can be expressed in the form
p(Ck|x)=p(x|Ck)p(Ck)
p(x). (1.77)
Note that any of the quantities appearing in Bayes’ theorem can be obtained from
the joint distribution p(x,Ck)by either marginalizing or conditioning with respect to
the appropriate variables. We can now interpret p(Ck)as the prior probability for the
classCk, andp(Ck|x)as the corresponding posterior probability. Thus p(C1)repre-

sents the probability that a person has cancer, before we take the X-ray measurement.
Similarly, p(C1|x)is the corresponding probability, revised using Bayes’ theorem in
light of the information contained in the X-ray. If our aim is to minimize the chance
of assigning xto the wrong class, then intuitively we would choose the class having
the higher posterior probability. We now show that this intuition is correct, and wealso discuss more general criteria for making decisions.
1.5.1 Minimizing the misclassiﬁcation rate

Suppose that our goal is simply to make as few misclassiﬁcations as possible.
We need a rule that assigns each value of xto one of the available classes. Such a
rule will divide the input space into regions Rkcalled decision regions , one for each
class, such that all points in Rkare assigned to class Ck. The boundaries between
decision regions are called decision boundaries ordecision surfaces . Note that each
decision region need not be contiguous but could comprise some number of disjoint

regions. We shall encounter examples of decision boundaries and decision regions inlater chapters. In order to ﬁnd the optimal decision rule, consider ﬁrst of all the case
of two classes, as in the cancer problem for instance. A mistake occurs when an input
vector belonging to class C
1is assigned to class C2or vice versa. The probability of
this occurring is given by
p(mistake) = p(x∈R1,C2)+p(x∈R2,C1)
=∫
R1p(x,C2)dx+∫
R2p(x,C1)dx. (1.78)
We are free to choose the decision rule that assigns each point xto one of the two

=∫
classes. Clearly to minimize p(mistake) we should arrange that each xis assigned to
whichever class has the smaller value of the integrand in (1.78). Thus, if p(x,C1)>
p(x,C2)for a given value of x, then we should assign that xto class C1. From the
product rule of probability we have p(x,Ck)=p(Ck|x)p(x). Because the factor
p(x)is common to both terms, we can restate this result as saying that the minimum

40 1. INTRODUCTION
R1 R2x0ˆx
p(x,C1)
p(x,C2)
x
Figure 1.24 Schematic illustration of the joint probabilities p(x,Ck)for each of two classes plotted
against x, together with the decision boundary x=bx. Values of x⩾bxare classiﬁed as
classC2and hence belong to decision region R2, whereas points x< bxare classiﬁed
asC1and belong to R1. Errors arise from the blue, green, and red regions, so that for
x< bxthe errors are due to points from class C2being misclassiﬁed as C1(represented by
the sum of the red and green regions), and conversely for points in the region x⩾bxthe
errors are due to points from class C1being misclassiﬁed as C2(represented by the blue
region). As we vary the location bxof the decision boundary, the combined areas of the
blue and green regions remains constant, whereas the size of the red region varies. The
optimal choice for bxis where the curves for p(x,C1)andp(x,C2)cross, corresponding to
bx=x0, because in this case the red region disappears. This is equivalent to the minimum

misclassiﬁcation rate decision rule, which assigns each value of xto the class having the
higher posterior probability p(Ck|x).
probability of making a mistake is obtained if each value of xis assigned to the class
for which the posterior probability p(Ck|x)is largest. This result is illustrated for
two classes, and a single input variable x, in Figure 1.24.
For the more general case of Kclasses, it is slightly easier to maximize the
probability of being correct, which is given by
p(correct) =K∑
k=1p(x∈R k,Ck)
=K∑
k=1∫
Rkp(x,Ck)dx (1.79)

=K∑
k=1∫
which is maximized when the regions Rkare chosen such that each xis assigned
to the class for which p(x,Ck)is largest. Again, using the product rule p(x,Ck)=
p(Ck|x)p(x), and noting that the factor of p(x)is common to all terms, we see
that each xshould be assigned to the class having the largest posterior probability
p(Ck|x).

1.5. Decision Theory 41
Figure 1.25 An example of a loss matrix with ele-
ments Lkjfor the cancer treatment problem. The rows
correspond to the true class, whereas the columns cor-
respond to the assignment of class made by our deci-sion criterion.(cancer normal
cancer 0 1000
normal 10)
1.5.2 Minimizing the expected loss
For many applications, our objective will be more complex than simply mini-
mizing the number of misclassiﬁcations. Let us consider again the medical diagnosisproblem. We note that, if a patient who does not have cancer is incorrectly diagnosed
as having cancer, the consequences may be some patient distress plus the need for
further investigations. Conversely, if a patient with cancer is diagnosed as healthy,
the result may be premature death due to lack of treatment. Thus the consequences
of these two types of mistake can be dramatically different. It would clearly be betterto make fewer mistakes of the second kind, even if this was at the expense of making
more mistakes of the ﬁrst kind.

We can formalize such issues through the introduction of a loss function , also
called a cost function , which is a single, overall measure of loss incurred in taking
any of the available decisions or actions. Our goal is then to minimize the total loss
incurred. Note that some authors consider instead a utility function , whose value
they aim to maximize. These are equivalent concepts if we take the utility to be
simply the negative of the loss, and throughout this text we shall use the loss function

convention. Suppose that, for a new value of x, the true class is C
kand that we assign
xto class Cj(where jmay or may not be equal to k). In so doing, we incur some
level of loss that we denote by Lkj, which we can view as the k,j element of a loss
matrix . For instance, in our cancer example, we might have a loss matrix of the form
shown in Figure 1.25. This particular loss matrix says that there is no loss incurred
if the correct decision is made, there is a loss of 1if a healthy patient is diagnosed as

having cancer, whereas there is a loss of 1000 if a patient having cancer is diagnosed
as healthy.
The optimal solution is the one which minimizes the loss function. However,
the loss function depends on the true class, which is unknown. For a given input
vectorx, our uncertainty in the true class is expressed through the joint probability
distribution p(x,Ck)and so we seek instead to minimize the average loss, where the
average is computed with respect to this distribution, which is given by
E[L]=∑
k∑
j∫

E[L]=∑
k∑
j∫
RjLkjp(x,Ck)dx. (1.80)
Eachxcan be assigned independently to one of the decision regions Rj. Our goal
is to choose the regions Rjin order to minimize the expected loss (1.80), which
implies that for each xwe should minimize∑
kLkjp(x,Ck). As before, we can use
the product rule p(x,Ck)=p(Ck|x)p(x)to eliminate the common factor of p(x).
Thus the decision rule that minimizes the expected loss is the one that assigns each

42 1. INTRODUCTION
Figure 1.26 Illustration of the reject option. Inputs
xsuch that the larger of the two poste-
rior probabilities is less than or equal to
some threshold θwill be rejected.
xp(C1|x) p(C2|x)
0.01.0
θ
reject region
newxto the class jfor which the quantity
∑
kLkjp(Ck|x) (1.81)
is a minimum. This is clearly trivial to do, once we know the posterior class proba-
bilities p(Ck|x).
1.5.3 The reject option
We have seen that classiﬁcation errors arise from the regions of input space
where the largest of the posterior probabilities p(Ck|x)is signiﬁcantly less than unity,
or equivalently where the joint distributions p(x,Ck)have comparable values. These
are the regions where we are relatively uncertain about class membership. In some
applications, it will be appropriate to avoid making decisions on the difﬁcult cases
in anticipation of a lower error rate on those examples for which a classiﬁcation de-
cision is made. This is known as the reject option . For example, in our hypothetical

medical illustration, it may be appropriate to use an automatic system to classify
those X-ray images for which there is little doubt as to the correct class, while leav-
ing a human expert to classify the more ambiguous cases. We can achieve this by
introducing a threshold θand rejecting those inputs xfor which the largest of the
posterior probabilities p(Ck|x)is less than or equal to θ. This is illustrated for the
case of two classes, and a single continuous input variable x, in Figure 1.26. Note

that setting θ=1 will ensure that all examples are rejected, whereas if there are K
classes then setting θ<1/K will ensure that no examples are rejected. Thus the
fraction of examples that get rejected is controlled by the value of θ.
We can easily extend the reject criterion to minimize the expected loss, when
a loss matrix is given, taking account of the loss incurred when a reject decision is
made. Exercise 1.24
1.5.4 Inference and decision

We have broken the classiﬁcation problem down into two separate stages, the
inference stage in which we use training data to learn a model for p(Ck|x), and the

1.5. Decision Theory 43
subsequent decision stage in which we use these posterior probabilities to make op-
timal class assignments. An alternative possibility would be to solve both problemstogether and simply learn a function that maps inputs xdirectly into decisions. Such
a function is called a discriminant function .
In fact, we can identify three distinct approaches to solving decision problems,
all of which have been used in practical applications. These are given, in decreasing
order of complexity, by:
(a) First solve the inference problem of determining the class-conditional densities
p(x|C
k)for each class Ckindividually. Also separately infer the prior class
probabilities p(Ck). Then use Bayes’ theorem in the form
p(x)(1.82)
to ﬁnd the posterior class probabilities p(Ck|x). As usual, the denominator
in Bayes’ theorem can be found in terms of the quantities appearing in the
numerator, because
p(x)=∑
kp(x|Ck)p(Ck). (1.83)

p(x|C
p(x)=∑
Equivalently, we can model the joint distribution p(x,Ck)directly and then
normalize to obtain the posterior probabilities. Having found the posterior
probabilities, we use decision theory to determine class membership for each
new input x. Approaches that explicitly or implicitly model the distribution of
inputs as well as outputs are known as generative models , because by sampling
from them it is possible to generate synthetic data points in the input space.

p(x)=∑
(b) First solve the inference problem of determining the posterior class probabilities
p(Ck|x), and then subsequently use decision theory to assign each new xto
one of the classes. Approaches that model the posterior probabilities directly
are called discriminative models .
(c) Find a function f(x), called a discriminant function, which maps each input x
directly onto a class label. For instance, in the case of two-class problems,
f(·)might be binary valued and such that f=0 represents class C1andf=1

represents class C2. In this case, probabilities play no role.
Let us consider the relative merits of these three alternatives. Approach (a) is the
most demanding because it involves ﬁnding the joint distribution over both xand
Ck. For many applications, xwill have high dimensionality, and consequently we
may need a large training set in order to be able to determine the class-conditionaldensities to reasonable accuracy. Note that the class priors p(C
k)can often be esti-

mated simply from the fractions of the training set data points in each of the classes.
One advantage of approach (a), however, is that it also allows the marginal density
of data p(x)to be determined from (1.83). This can be useful for detecting new data
points that have low probability under the model and for which the predictions may

44 1. INTRODUCTION
p(x|C1)p(x|C2)
xclass densities
0 0.2 0.4 0.6 0.8 100.20.40.60.811.2
Figure 1.27 Example of the class-conditional densities for two classes having a single input variable x(left
plot) together with the corresponding posterior probabilities (right plot). Note that the left-hand mode of the
class-conditional density p(x|C1), shown in blue on the left plot, has no effect on the posterior probabilities. The
vertical green line in the right plot shows the decision boundary in xthat gives the minimum misclassiﬁcation
rate.
be of low accuracy, which is known as outlier detection ornovelty detection (Bishop,
1994; Tarassenko, 1995).
However, if we only wish to make classiﬁcation decisions, then it can be waste-
ful of computational resources, and excessively demanding of data, to ﬁnd the joint
distribution p(x,Ck)when in fact we only really need the posterior probabilities
p(Ck|x), which can be obtained directly through approach (b). Indeed, the class-

rate.
conditional densities may contain a lot of structure that has little effect on the pos-
terior probabilities, as illustrated in Figure 1.27. There has been much interest in
exploring the relative merits of generative and discriminative approaches to machine
learning, and in ﬁnding ways to combine them (Jebara, 2004; Lasserre et al. , 2006).
An even simpler approach is (c) in which we use the training data to ﬁnd a
discriminant function f(x)that maps each xdirectly onto a class label, thereby
combining the inference and decision stages into a single learning problem. In the

example of Figure 1.27, this would correspond to ﬁnding the value of xshown by
the vertical green line, because this is the decision boundary giving the minimum
probability of misclassiﬁcation.
With option (c), however, we no longer have access to the posterior probabilities
p(Ck|x). There are many powerful reasons for wanting to compute the posterior
probabilities, even if we subsequently use them to make decisions. These include:
Minimizing risk. Consider a problem in which the elements of the loss matrix are

subjected to revision from time to time (such as might occur in a ﬁnancial

1.5. Decision Theory 45
application). If we know the posterior probabilities, we can trivially revise the
minimum risk decision criterion by modifying (1.81) appropriately. If we haveonly a discriminant function, then any change to the loss matrix would require
that we return to the training data and solve the classiﬁcation problem afresh.
Reject option. Posterior probabilities allow us to determine a rejection criterion that
will minimize the misclassiﬁcation rate, or more generally the expected loss,for a given fraction of rejected data points.
Compensating for class priors. Consider our medical X-ray problem again, and
suppose that we have collected a large number of X-ray images from the gen-
eral population for use as training data in order to build an automated screeningsystem. Because cancer is rare amongst the general population, we might ﬁnd
that, say, only 1 in every 1,000 examples corresponds to the presence of can-

cer. If we used such a data set to train an adaptive model, we could run intosevere difﬁculties due to the small proportion of the cancer class. For instance,
a classiﬁer that assigned every point to the normal class would already achieve
99.9% accuracy and it would be difﬁcult to avoid this trivial solution. Also,even a large data set will contain very few examples of X-ray images corre-
sponding to cancer, and so the learning algorithm will not be exposed to a

broad range of examples of such images and hence is not likely to generalizewell. A balanced data set in which we have selected equal numbers of exam-
ples from each of the classes would allow us to ﬁnd a more accurate model.
However, we then have to compensate for the effects of our modiﬁcations to
the training data. Suppose we have used such a modiﬁed data set and found
models for the posterior probabilities. From Bayes’ theorem (1.82), we see thatthe posterior probabilities are proportional to the prior probabilities, which we

can interpret as the fractions of points in each class. We can therefore simply
take the posterior probabilities obtained from our artiﬁcially balanced data setand ﬁrst divide by the class fractions in that data set and then multiply by the
class fractions in the population to which we wish to apply the model. Finally,
we need to normalize to ensure that the new posterior probabilities sum to one.Note that this procedure cannot be applied if we have learned a discriminant
function directly instead of determining posterior probabilities.
Combining models. For complex applications, we may wish to break the problem

into a number of smaller subproblems each of which can be tackled by a sep-arate module. For example, in our hypothetical medical diagnosis problem,
we may have information available from, say, blood tests as well as X-ray im-
ages. Rather than combine all of this heterogeneous information into one hugeinput space, it may be more effective to build one system to interpret the X-
ray images and a different one to interpret the blood data. As long as each of
the two models gives posterior probabilities for the classes, we can combinethe outputs systematically using the rules of probability. One simple way to

do this is to assume that, for each class separately, the distributions of inputs
for the X-ray images, denoted by x
I, and the blood data, denoted by xB,a r e

46 1. INTRODUCTION
independent, so that
p(xI,xB|Ck)=p(xI|Ck)p(xB|Ck). (1.84)
This is an example of conditional independence property, because the indepen- Section 8.2
dence holds when the distribution is conditioned on the class Ck. The posterior
probability, given both the X-ray and blood data, is then given by
p(Ck|xI,xB)∝p(xI,xB|Ck)p(Ck)
∝p(xI|Ck)p(xB|Ck)p(Ck)
∝p(Ck|xI)p(Ck|xB)
p(Ck)(1.85)
Thus we need the class prior probabilities p(Ck), which we can easily estimate
from the fractions of data points in each class, and then we need to normalize
the resulting posterior probabilities so they sum to one. The particular condi-tional independence assumption (1.84) is an example of the naive Bayes model . Section 8.2.2
Note that the joint marginal distribution p(x
I,xB)will typically not factorize
under this model. We shall see in later chapters how to construct models forcombining data that do not require the conditional independence assumption
(1.84).
1.5.5 Loss functions for regression

(1.84).
So far, we have discussed decision theory in the context of classiﬁcation prob-
lems. We now turn to the case of regression problems, such as the curve ﬁtting
example discussed earlier. The decision stage consists of choosing a speciﬁc esti- Section 1.1
matey(x)of the value of tfor each input x. Suppose that in doing so, we incur a
lossL(t, y(x)). The average, or expected, loss is then given by
E[L]=∫∫
L(t, y(x))p(x,t)dxdt. (1.86)
A common choice of loss function in regression problems is the squared loss given

E[L]=∫∫
byL(t, y(x)) ={y(x)−t}2. In this case, the expected loss can be written
E[L]=∫∫
{y(x)−t}2p(x,t)dxdt. (1.87)
Our goal is to choose y(x)so as to minimize E[L]. If we assume a completely
ﬂexible function y(x), we can do this formally using the calculus of variations to Appendix D
give
δE[L]
δy(x)=2∫
{y(x)−t}p(x,t)dt=0. (1.88)
Solving for y(x), and using the sum and product rules of probability, we obtain
y(x)=∫
tp(x,t)dt
p(x)=∫
tp(t|x)dt=Et[t|x] (1.89)

1.5. Decision Theory 47
Figure 1.28 The regression function y(x),
which minimizes the expected
squared loss, is given by the
mean of the conditional distri-
bution p(t|x).t
x x0y(x0)y(x)
p(t|x0)
which is the conditional average of tconditioned on xand is known as the regression
function . This result is illustrated in Figure 1.28. It can readily be extended to mul-
tiple target variables represented by the vector t, in which case the optimal solution
is the conditional average y(x)= Et[t|x]. Exercise 1.25
We can also derive this result in a slightly different way, which will also shed
light on the nature of the regression problem. Armed with the knowledge that the
optimal solution is the conditional expectation, we can expand the square term as
follows
{y(x)−t}2={y(x)−E[t|x]+ E[t|x]−t}2
={y(x)−E[t|x]}2+2{y(x)−E[t|x]}{E[t|x]−t}+{E[t|x]−t}2
where, to keep the notation uncluttered, we use E[t|x]to denote Et[t|x]. Substituting
into the loss function and performing the integral over t, we see that the cross-term

follows
vanishes and we obtain an expression for the loss function in the form
E[L]=∫
{y(x)−E[t|x]}2p(x)dx+∫
{E[t|x]−t}2p(x)dx. (1.90)
The function y(x)we seek to determine enters only in the ﬁrst term, which will be
minimized when y(x)is equal to E[t|x], in which case this term will vanish. This
is simply the result that we derived previously and that shows that the optimal least
squares predictor is given by the conditional mean. The second term is the variance

E[L]=∫
of the distribution of t, averaged over x. It represents the intrinsic variability of
the target data and can be regarded as noise. Because it is independent of y(x),i t
represents the irreducible minimum value of the loss function.
As with the classiﬁcation problem, we can either determine the appropriate prob-
abilities and then use these to make optimal decisions, or we can build models that
make decisions directly. Indeed, we can identify three distinct approaches to solving
regression problems given, in order of decreasing complexity, by:

(a) First solve the inference problem of determining the joint density p(x,t). Then
normalize to ﬁnd the conditional density p(t|x), and ﬁnally marginalize to ﬁnd
the conditional mean given by (1.89).

48 1. INTRODUCTION
(b) First solve the inference problem of determining the conditional density p(t|x),
and then subsequently marginalize to ﬁnd the conditional mean given by (1.89).
(c) Find a regression function y(x)directly from the training data.
The relative merits of these three approaches follow the same lines as for classiﬁca-
tion problems above.
The squared loss is not the only possible choice of loss function for regression.
Indeed, there are situations in which squared loss can lead to very poor results and
where we need to develop more sophisticated approaches. An important example
concerns situations in which the conditional distribution p(t|x)is multimodal, as
often arises in the solution of inverse problems. Here we consider brieﬂy one simple Section 5.6
generalization of the squared loss, called the Minkowski loss, whose expectation is
given by
E[Lq]=∫∫
|y(x)−t|qp(x,t)dxdt (1.91)
which reduces to the expected squared loss for q=2 . The function |y−t|qis

given by
E[Lq]=∫∫
plotted against y−tfor various values of qin Figure 1.29. The minimum of E[Lq]
is given by the conditional mean for q=2 , the conditional median for q=1 , and
the conditional mode for q→0. Exercise 1.27
1.6. Information Theory
In this chapter, we have discussed a variety of concepts from probability theory and
decision theory that will form the foundations for much of the subsequent discussion
in this book. We close this chapter by introducing some additional concepts fromthe ﬁeld of information theory, which will also prove useful in our development of

pattern recognition and machine learning techniques. Again, we shall focus only on
the key concepts, and we refer the reader elsewhere for more detailed discussions(Viterbi and Omura, 1979; Cover and Thomas, 1991; MacKay, 2003) .
We begin by considering a discrete random variable xand we ask how much
information is received when we observe a speciﬁc value for this variable. Theamount of information can be viewed as the ‘degree of surprise’ on learning the

value of x. If we are told that a highly improbable event has just occurred, we will
have received more information than if we were told that some very likely eventhas just occurred, and if we knew that the event was certain to happen we would
receive no information. Our measure of information content will therefore depend
on the probability distribution p(x), and we therefore look for a quantity h(x)that
is a monotonic function of the probability p(x)and that expresses the information

content. The form of h(·)can be found by noting that if we have two events x
andythat are unrelated, then the information gain from observing both of them
should be the sum of the information gained from each of them separately, so that
h(x, y)=h(x)+h(y). Two unrelated events will be statistically independent and
sop(x, y)=p(x)p(y). From these two relationships, it is easily shown that h(x)
must be given by the logarithm of p(x)and so we have Exercise 1.28

1.6. Information Theory 49
y−t|y−t|qq=0.3
−2 −1 0 1 2012
y−t|y−t|qq=1
y−t|y−t|qq=2
y−t|y−t|qq=1 0
Figure 1.29 Plots of the quantity Lq=|y−t|qfor various values of q.
h(x)=−log2p(x) (1.92)
where the negative sign ensures that information is positive or zero. Note that low
probability events xcorrespond to high information content. The choice of basis
for the logarithm is arbitrary, and for the moment we shall adopt the convention
prevalent in information theory of using logarithms to the base of 2. In this case, as
we shall see shortly, the units of h(x)are bits (‘binary digits’).
Now suppose that a sender wishes to transmit the value of a random variable to
a receiver. The average amount of information that they transmit in the process is
obtained by taking the expectation of (1.92) with respect to the distribution p(x)and
is given by
H[x]=−∑
xp(x) log2p(x). (1.93)
This important quantity is called the entropy of the random variable x. Note that

is given by
H[x]=−∑
limp→0plnp=0 and so we shall take p(x)l np(x)=0 whenever we encounter a
value for xsuch that p(x)=0 .
So far we have given a rather heuristic motivation for the deﬁnition of informa-

50 1. INTRODUCTION
tion (1.92) and the corresponding entropy (1.93). We now show that these deﬁnitions
indeed possess useful properties. Consider a random variable xhaving 8 possible
states, each of which is equally likely. In order to communicate the value of xto
a receiver, we would need to transmit a message of length 3 bits. Notice that the
entropy of this variable is given by
H[x]=−8×1
8log21
8= 3 bits .
Now consider an example (Cover and Thomas, 1991) of a variable having 8pos-
sible states {a, b, c, d, e, f, g, h }for which the respective probabilities are given by
(1
2,1
4,1
8,1
16,1
64,1
64,1
64,1
64). The entropy in this case is given by
H[x]=−1
2log21
2−1
4log21
4−1
8log21
8−1
16log21
16−4
64log21
64= 2 bits .
We see that the nonuniform distribution has a smaller entropy than the uniform one,
and we shall gain some insight into this shortly when we discuss the interpretation of

8log21
8= 3 bits .
(1
2,1
4,1
8,1
16,1
64,1
64,1
64,1
H[x]=−1
2log21
2−1
4log21
4−1
8log21
8−1
16log21
16−4
64log21
64= 2 bits .
entropy in terms of disorder. For the moment, let us consider how we would transmitthe identity of the variable’s state to a receiver. We could do this, as before, using
a 3-bit number. However, we can take advantage of the nonuniform distribution by
using shorter codes for the more probable events, at the expense of longer codes forthe less probable events, in the hope of getting a shorter average code length. This
can be done by representing the states {a, b, c, d, e, f, g, h }using, for instance, the

following set of code strings: 0, 10, 110, 1110, 111100, 111101, 111110, 111111.The average length of the code that has to be transmitted is then
average code length =1
2×1+1
4×2+1
8×3+1
16×4+4×1
64×6 = 2 bits
which again is the same as the entropy of the random variable. Note that shorter code
strings cannot be used because it must be possible to disambiguate a concatenationof such strings into its component parts. For instance, 11001110 decodes uniquely
into the state sequence c,a,d.

2×1+1
4×2+1
8×3+1
16×4+4×1
This relation between entropy and shortest coding length is a general one. The
noiseless coding theorem (Shannon, 1948) states that the entropy is a lower bound
on the number of bits needed to transmit the state of a random variable.
From now on, we shall switch to the use of natural logarithms in deﬁning en-
tropy, as this will provide a more convenient link with ideas elsewhere in this book.
In this case, the entropy is measured in units of ‘nats’ instead of bits, which differ
simply by a factor of ln2 .

We have introduced the concept of entropy in terms of the average amount of
information needed to specify the state of a random variable. In fact, the concept ofentropy has much earlier origins in physics where it was introduced in the context
of equilibrium thermodynamics and later given a deeper interpretation as a measure
of disorder through developments in statistical mechanics. We can understand thisalternative view of entropy by considering a set of Nidentical objects that are to be

divided amongst a set of bins, such that there are n
iobjects in the ithbin. Consider

1.6. Information Theory 51
the number of different ways of allocating the objects to the bins. There are N
ways to choose the ﬁrst object, (N−1)ways to choose the second object, and
so on, leading to a total of N!ways to allocate all Nobjects to the bins, where N!
(pronounced ‘factorial N’) denotes the product N×(N−1)×···× 2×1. However,
we don’t wish to distinguish between rearrangements of objects within each bin. Inthei
thbin there are ni!ways of reordering the objects, and so the total number of
ways of allocating the Nobjects to the bins is given by
W=N!∏
ini!(1.94)
which is called the multiplicity . The entropy is then deﬁned as the logarithm of the
multiplicity scaled by an appropriate constant
H=1
NlnW=1
NlnN!−1
N∑
ilnni!. (1.95)
We now consider the limit N→∞ , in which the fractions ni/N are held ﬁxed, and
apply Stirling’s approximation
lnN!≃NlnN−N (1.96)
which gives
H=−lim
N→∞∑
i(ni
N)
ln(ni
N)
=−∑
ipilnpi (1.97)
where we have used∑
ini=N. Here pi= lim N→∞(ni/N)is the probability

W=N!∏
H=1
NlnW=1
NlnN!−1
N∑
H=−lim
N→∞∑
i(ni
N)
ln(ni
N)
=−∑
of an object being assigned to the ithbin. In physics terminology, the speciﬁc ar-
rangements of objects in the bins is called a microstate , and the overall distribution
of occupation numbers, expressed through the ratios ni/N, is called a macrostate .
The multiplicity Wis also known as the weight of the macrostate.
We can interpret the bins as the states xiof a discrete random variable X, where
p(X=xi)=pi. The entropy of the random variable Xis then
H[p]=−∑
ip(xi)l np(xi). (1.98)

H[p]=−∑
Distributions p(xi)that are sharply peaked around a few values will have a relatively
low entropy, whereas those that are spread more evenly across many values will
have higher entropy, as illustrated in Figure 1.30. Because 0⩽pi⩽1, the entropy
is nonnegative, and it will equal its minimum value of 0 when one of the pi=
1and all other pj̸=i=0 . The maximum entropy conﬁguration can be found by
maximizing Husing a Lagrange multiplier to enforce the normalization constraint Appendix E
on the probabilities. Thus we maximize
˜H=−∑

˜H=−∑
ip(xi)l np(xi)+λ(∑
ip(xi)−1)
(1.99)

52 1. INTRODUCTIONprobabilitiesH = 1.77
00.250.5
probabilitiesH = 3.09
00.250.5
Figure 1.30 Histograms of two probability distributions over 30bins illustrating the higher value of the entropy
Hfor the broader distribution. The largest entropy would arise from a uniform distribution that would give H=
−ln(1/30) = 3 .40.
from which we ﬁnd that all of the p(xi)are equal and are given by p(xi)=1/M
where Mis the total number of states xi. The corresponding value of the entropy
is then H=l n M. This result can also be derived from Jensen’s inequality (to be
discussed shortly). To verify that the stationary point is indeed a maximum, we can Exercise 1.29
evaluate the second derivative of the entropy, which gives
∂˜H
∂p(xi)∂p(xj)=−Iij1
pi(1.100)
where Iijare the elements of the identity matrix.
We can extend the deﬁnition of entropy to include distributions p(x)over con-
tinuous variables xas follows. First divide xinto bins of width ∆. Then, assuming

∂˜H
pi(1.100)
p(x)is continuous, the mean value theorem (Weisstein, 1999) tells us that, for each
such bin, there must exist a value xisuch that
∫(i+1)∆
i∆p(x)dx=p(xi)∆. (1.101)
We can now quantize the continuous variable xby assigning any value xto the value
xiwhenever xfalls in the ithbin. The probability of observing the value xiis then
p(xi)∆. This gives a discrete distribution for which the entropy takes the form
H∆=−∑
ip(xi)∆ ln( p(xi)∆) = −∑
ip(xi)∆ lnp(xi)−ln∆ (1.102)
ip(xi)∆ = 1 , which follows from (1.101). We now omit

∫(i+1)∆
H∆=−∑
the second term −ln∆ on the right-hand side of (1.102) and then consider the limit

1.6. Information Theory 53
∆→0. The ﬁrst term on the right-hand side of (1.102) will approach the integral of
p(x)l np(x)in this limit so that
lim
∆→0{∑
ip(xi)∆ lnp(xi)}
=−∫
p(x)l np(x)dx (1.103)
where the quantity on the right-hand side is called the differential entropy . We see
that the discrete and continuous forms of the entropy differ by a quantity ln∆ , which
diverges in the limit ∆→0. This reﬂects the fact that to specify a continuous
variable very precisely requires a large number of bits. For a density deﬁned overmultiple continuous variables, denoted collectively by the vector x, the differential
entropy is given by
H[x]=−∫
p(x)l np(x)dx. (1.104)
In the case of discrete distributions, we saw that the maximum entropy con-
ﬁguration corresponded to an equal distribution of probabilities across the possible
states of the variable. Let us now consider the maximum entropy conﬁguration fora continuous variable. In order for this maximum to be well deﬁned, it will be nec-

H[x]=−∫
essary to constrain the ﬁrst and second moments of p(x)as well as preserving the
normalization constraint. We therefore maximize the differential entropy with the
Ludwig Boltzmann
1844–1906
Ludwig Eduard Boltzmann was an
Austrian physicist who created theﬁeld of statistical mechanics. Priorto Boltzmann, the concept of en-tropy was already known fromclassical thermodynamics where it
quantiﬁes the fact that when we take energy from asystem, not all of that energy is typically availableto do useful work. Boltzmann showed that the ther-modynamic entropy S, a macroscopic quantity, could
be related to the statistical properties at the micro-

1844–1906
scopic level. This is expressed through the famous
equation S=klnWin which Wrepresents the
number of possible microstates in a macrostate, andk≃1.38×10
−23(in units of Joules per Kelvin) is
known as Boltzmann’s constant. Boltzmann’s ideaswere disputed by many scientists of they day. One dif-ﬁculty they saw arose from the second law of thermo-dynamics, which states that the entropy of a closed

system tends to increase with time. By contrast, atthe microscopic level the classical Newtonian equa-tions of physics are reversible, and so they found itdifﬁcult to see how the latter could explain the for-mer. They didn’t fully appreciate Boltzmann’s argu-ments, which were statistical in nature and which con-cluded not that entropy could never decrease overtime but simply that with overwhelming probability itwould generally increase. Boltzmann even had a long-running dispute with the editor of the leading Germanphysics journal who refused to let him refer to atomsand molecules as anything other than convenient the-oretical constructs. The continued attacks on his worklead to bouts of depression, and eventually he com-mitted suicide. Shortly after Boltzmann’s death, newexperiments by Perrin on colloidal suspensions veri-ﬁed his theories and conﬁrmed the value of the Boltz-mann constant. The equation S=klnWis carved on
Boltzmann’s tombstone.

54 1. INTRODUCTION
three constraints
∫∞
−∞p(x)dx=1 (1.105)
∫∞
−∞xp(x)dx=µ (1.106)
∫∞
−∞(x−µ)2p(x)dx=σ2. (1.107)
The constrained maximization can be performed using Lagrange multipliers so that Appendix E
we maximize the following functional with respect to p(x)
−∫∞
−∞p(x)l np(x)dx+λ1(∫∞
−∞p(x)dx−1)
+λ2(∫∞
−∞xp(x)dx−µ)
+λ3(∫∞
−∞(x−µ)2p(x)dx−σ2)
.
Using the calculus of variations, we set the derivative of this functional to zero giving Appendix D
p(x)=e x p{
−1+λ1+λ2x+λ3(x−µ)2}
. (1.108)
The Lagrange multipliers can be found by back substitution of this result into the
three constraint equations, leading ﬁnally to the result Exercise 1.34
p(x)=1
−(x−µ)2
2σ2}
(1.109)
and so the distribution that maximizes the differential entropy is the Gaussian. Note
that we did not constrain the distribution to be nonnegative when we maximized the
entropy. However, because the resulting distribution is indeed nonnegative, we seewith hindsight that such a constraint is not necessary.

p(x)=1
−(x−µ)2
2σ2}
(1.109)
If we evaluate the differential entropy of the Gaussian, we obtain Exercise 1.35
H[x]=1
2{
1+l n ( 2 πσ2)}
. (1.110)
Thus we see again that the entropy increases as the distribution becomes broader,
i.e., as σ2increases. This result also shows that the differential entropy, unlike the
discrete entropy, can be negative, because H(x)<0in (1.110) for σ2<1/(2πe).
Suppose we have a joint distribution p(x,y)from which we draw pairs of values
ofxandy. If a value of xis already known, then the additional information needed

H[x]=1
2{
. (1.110)
to specify the corresponding value of yis given by −lnp(y|x). Thus the average
additional information needed to specify ycan be written as
H[y|x]=−∫∫
p(y,x)l np(y|x)dydx (1.111)

1.6. Information Theory 55
which is called the conditional entropy ofygivenx. It is easily seen, using the
product rule, that the conditional entropy satisﬁes the relation Exercise 1.37
H[x,y]=H [y|x]+H [x] (1.112)
where H[x,y]is the differential entropy of p(x,y)andH[x]is the differential en-
tropy of the marginal distribution p(x). Thus the information needed to describe x
andyis given by the sum of the information needed to describe xalone plus the
additional information required to specify ygivenx.
1.6.1 Relative entropy and mutual information
So far in this section, we have introduced a number of concepts from information
theory, including the key notion of entropy. We now start to relate these ideas topattern recognition. Consider some unknown distribution p(x), and suppose that
we have modelled this using an approximating distribution q(x). I fw eu s e q(x)to
construct a coding scheme for the purpose of transmitting values of xto a receiver,

then the average additional amount of information (in nats) required to specify the
value of x(assuming we choose an efﬁcient coding scheme) as a result of using q(x)
instead of the true distribution p(x)is given by
KL(p∥q)= −∫
p(x)l nq(x)dx−(
−∫
p(x)l np(x)dx)
=−∫
p(x)l n{q(x)
p(x)}
dx. (1.113)
This is known as the relative entropy orKullback-Leibler divergence ,o r KL diver-
gence (Kullback and Leibler, 1951), between the distributions p(x)andq(x). Note

−∫
=−∫
p(x)}
that it is not a symmetrical quantity, that is to say KL(p∥q)̸≡KL(q∥p).
We now show that the Kullback-Leibler divergence satisﬁes KL(p∥q)⩾0with
equality if, and only if, p(x)=q(x). To do this we ﬁrst introduce the concept of
convex functions. A function f(x)is said to be convex if it has the property that
every chord lies on or above the function, as shown in Figure 1.31. Any value of x
in the interval from x=atox=bcan be written in the form λa+( 1−λ)bwhere
0⩽λ⩽1. The corresponding point on the chord is given by λf(a)+( 1−λ)f(b),
Claude Shannon
1916–2001

1916–2001
After graduating from Michigan and
MIT, Shannon joined the AT&T BellTelephone laboratories in 1941. Hispaper ‘A Mathematical Theory ofCommunication’ published in the
Bell System Technical Journal in
1948 laid the foundations for modern information the-ory. This paper introduced the word ‘bit’, and his con-

1916–2001
cept that information could be sent as a stream of 1sand 0s paved the way for the communications revo-lution. It is said that von Neumann recommended toShannon that he use the term entropy, not only be-cause of its similarity to the quantity used in physics,but also because “nobody knows what entropy reallyis, so in any discussion you will always have an advan-tage”.

56 1. INTRODUCTION
Figure 1.31 A convex function f(x)is one for which ev-
ery chord (shown in blue) lies on or above
the function (shown in red).
x a b xλchord
xλf(x)
and the corresponding value of the function is f(λa+( 1−λ)b). Convexity then
implies
f(λa+( 1−λ)b)⩽λf(a)+( 1 −λ)f(b). (1.114)
This is equivalent to the requirement that the second derivative of the function be
everywhere positive. Examples of convex functions are xlnx(forx>0) andx2.A Exercise 1.36
function is called strictly convex if the equality is satisﬁed only for λ=0 andλ=1 .
If a function has the opposite property, namely that every chord lies on or below the
function, it is called concave , with a corresponding deﬁnition for strictly concave .I f
a function f(x)is convex, then −f(x)will be concave.
Using the technique of proof by induction, we can show from (1.114) that a Exercise 1.38
convex function f(x)satisﬁes
f(M∑
i=1λixi)
⩽M∑
i=1λif(xi) (1.115)
where λi⩾0and∑
iλi=1 , for any set of points {xi}. The result (1.115) is

f(M∑
i=1λixi)
⩽M∑
known as Jensen’s inequality . If we interpret the λias the probability distribution
over a discrete variable xtaking the values {xi}, then (1.115) can be written
f(E[x])⩽E[f(x)] (1.116)
where E[·]denotes the expectation. For continuous variables, Jensen’s inequality
takes the form
f(∫
xp(x)dx)
⩽∫
f(x)p(x)dx. (1.117)
We can apply Jensen’s inequality in the form (1.117) to the Kullback-Leibler
divergence (1.113) to give
KL(p∥q)=−∫
p(x)}
dx⩾−ln∫
q(x)dx=0 (1.118)

1.6. Information Theory 57
where we have used the fact that −lnxis a convex function, together with the nor-
malization condition∫
q(x)dx=1 . In fact, −lnxis a strictly convex function,
so the equality will hold if, and only if, q(x)=p(x)for allx. Thus we can in-
terpret the Kullback-Leibler divergence as a measure of the dissimilarity of the two
distributions p(x)andq(x).
We see that there is an intimate relationship between data compression and den-
sity estimation (i.e., the problem of modelling an unknown probability distribution)
because the most efﬁcient compression is achieved when we know the true distri-bution. If we use a distribution that is different from the true one, then we must
necessarily have a less efﬁcient coding, and on average the additional information
that must be transmitted is (at least) equal to the Kullback-Leibler divergence be-
tween the two distributions.
Suppose that data is being generated from an unknown distribution p(x)that we

wish to model. We can try to approximate this distribution using some parametric
distribution q(x|θ), governed by a set of adjustable parameters θ, for example a
multivariate Gaussian. One way to determine θis to minimize the Kullback-Leibler
divergence between p(x)andq(x|θ)with respect to θ. We cannot do this directly
because we don’t know p(x). Suppose, however, that we have observed a ﬁnite set
of training points x
n, forn=1,...,N , drawn from p(x). Then the expectation
with respect to p(x)can be approximated by a ﬁnite sum over these points, using
(1.35), so that
KL(p∥q)≃N∑

n=1{−lnq(xn|θ)+l n p(xn)}. (1.119)
The second term on the right-hand side of (1.119) is independent of θ, and the ﬁrst
term is the negative log likelihood function for θunder the distribution q(x|θ)eval-
uated using the training set. Thus we see that minimizing this Kullback-Leiblerdivergence is equivalent to maximizing the likelihood function.
Now consider the joint distribution between two sets of variables xandygiven
byp(x,y). If the sets of variables are independent, then their joint distribution will

factorize into the product of their marginals p(x,y)=p(x)p(y). If the variables are
not independent, we can gain some idea of whether they are ‘close’ to being indepen-
dent by considering the Kullback-Leibler divergence between the joint distributionand the product of the marginals, given by
I[x,y]≡KL(p(x,y)∥p(x)p(y))
=−∫∫
p(x,y)l n(p(x)p(y)
p(x,y))
dxdy (1.120)
which is called the mutual information between the variables xandy. From the

=−∫∫
p(x,y))
properties of the Kullback-Leibler divergence, we see that I(x,y)⩾0with equal-
ity if, and only if, xandyare independent. Using the sum and product rules of
probability, we see that the mutual information is related to the conditional entropy
through Exercise 1.41
I[x,y]=H [x]−H[x|y]=H [y]−H[y|x]. (1.121)

58 1. INTRODUCTION
Thus we can view the mutual information as the reduction in the uncertainty about x
by virtue of being told the value of y(or vice versa). From a Bayesian perspective,
we can view p(x)as the prior distribution for xandp(x|y)as the posterior distribu-
tion after we have observed new data y. The mutual information therefore represents
the reduction in uncertainty about xas a consequence of the new observation y.
Exercises
1.1 (⋆)www Consider the sum-of-squares error function given by (1.2) in which
the function y(x,w)is given by the polynomial (1.1). Show that the coefﬁcients
w={wi}that minimize this error function are given by the solution to the following
set of linear equations
M∑
j=0Aijwj=Ti (1.122)
where
Aij=N∑
n=1(xn)i+j,T i=N∑
n=1(xn)itn. (1.123)
Here a sufﬁx iorjdenotes the index of a component, whereas (x)idenotes xraised
to the power of i.
1.2 (⋆)Write down the set of coupled linear equations, analogous to (1.122), satisﬁed

M∑
where
Aij=N∑
by the coefﬁcients wiwhich minimize the regularized sum-of-squares error function
given by (1.4).
1.3 (⋆⋆)Suppose that we have three coloured boxes r(red), b(blue), and g(green).
Boxrcontains 3 apples, 4 oranges, and 3 limes, box bcontains 1 apple, 1 orange,
and 0 limes, and box gcontains 3 apples, 3 oranges, and 4 limes. If a box is chosen
at random with probabilities p(r)=0 .2,p(b)=0 .2,p(g)=0 .6, and a piece of
fruit is removed from the box (with equal probability of selecting any of the items in

the box), then what is the probability of selecting an apple? If we observe that theselected fruit is in fact an orange, what is the probability that it came from the green
box?
1.4 (⋆⋆)
www Consider a probability density px(x)deﬁned over a continuous vari-
ablex, and suppose that we make a nonlinear change of variable using x=g(y),
so that the density transforms according to (1.27). By differentiating (1.27), show
that the location ˆyof the maximum of the density in yis not in general related to the

box?
1.4 (⋆⋆)
locationˆxof the maximum of the density over xby the simple functional relation
ˆx=g(ˆy)as a consequence of the Jacobian factor. This shows that the maximum
of a probability density (in contrast to a simple function) is dependent on the choice
of variable. V erify that, in the case of a linear transformation, the location of themaximum transforms in the same way as the variable itself.
1.5 (⋆)Using the deﬁnition (1.38) show that var[f(x)]satisﬁes (1.39).

Exercises 59
1.6 (⋆)Show that if two variables xandyare independent, then their covariance is
zero.
1.7 (⋆⋆)www In this exercise, we prove the normalization condition (1.48) for the
univariate Gaussian. To do this consider, the integral
I=∫∞
−∞exp(
−1
2σ2x2)
dx (1.124)
which we can evaluate by ﬁrst writing its square in the form
I2=∫∞
−∞∫∞
−∞exp(
−1
2σ2x2−1
2σ2y2)
dxdy. (1.125)
Now make the transformation from Cartesian coordinates (x, y)to polar coordinates
(r, θ)and then substitute u=r2. Show that, by performing the integrals over θand
u, and then taking the square root of both sides, we obtain
I=(
2πσ2)1/2. (1.126)
Finally, use this result to show that the Gaussian distribution N(x|µ, σ2)is normal-
ized.
1.8 (⋆⋆)www By using a change of variables, verify that the univariate Gaussian
distribution given by (1.46) satisﬁes (1.49). Next, by differentiating both sides of thenormalization condition
∫∞
−∞N(
x|µ, σ2)
dx=1 (1.127)

I=(
ized.
∫∞
−∞N(
x|µ, σ2)
with respect to σ2, verify that the Gaussian satisﬁes (1.50). Finally, show that (1.51)
holds.
1.9 (⋆)www Show that the mode (i.e. the maximum) of the Gaussian distribution
(1.46) is given by µ. Similarly, show that the mode of the multivariate Gaussian
(1.52) is given by µ.
1.10 (⋆)www Suppose that the two variables xandzare statistically independent.
Show that the mean and variance of their sum satisﬁes
E[x+z]= E[x]+ E[z] (1.128)
var[x+z]=v a r [ x]+v a r [ z]. (1.129)

∫∞
−∞N(
x|µ, σ2)
holds.
1.11 (⋆)By setting the derivatives of the log likelihood function (1.54) with respect to µ
andσ2equal to zero, verify the results (1.55) and (1.56).

60 1. INTRODUCTION
1.12 (⋆⋆)www Using the results (1.49) and (1.50), show that
E[xnxm]=µ2+Inmσ2(1.130)
where xnandxmdenote data points sampled from a Gaussian distribution with mean
µand variance σ2, andInm satisﬁes Inm=1 ifn=mandInm=0 otherwise.
Hence prove the results (1.57) and (1.58).
1.13 (⋆)Suppose that the variance of a Gaussian is estimated using the result (1.56) but
with the maximum likelihood estimate µML replaced with the true value µof the
mean. Show that this estimator has the property that its expectation is given by the
true variance σ2.
1.14 (⋆⋆)Show that an arbitrary square matrix with elements wijcan be written in
the form wij=wS
ij+wA
ijwhere wS
ijandwA
ijare symmetric and anti-symmetric
matrices, respectively, satisfying wS
ij=wS
jiandwA
ij=−wA
jifor all iandj.N o w
consider the second order term in a higher order polynomial in Ddimensions, given
by
D∑
i=1D∑
j=1wijxixj. (1.131)
Show that
D∑
i=1D∑
j=1wijxixj=D∑
i=1D∑
j=1wS
ijxixj (1.132)

ij+wA
ijwhere wS
ijandwA
ij=wS
jiandwA
ij=−wA
by
D∑
i=1D∑
Show that
D∑
i=1D∑
i=1D∑
j=1wS
so that the contribution from the anti-symmetric matrix vanishes. We therefore see
that, without loss of generality, the matrix of coefﬁcients wijcan be chosen to be
symmetric, and so not all of the D2elements of this matrix can be chosen indepen-
dently. Show that the number of independent parameters in the matrix wS
ijis given
byD(D+1 )/2.
1.15 (⋆⋆⋆ )www In this exercise and the next, we explore how the number of indepen-
dent parameters in a polynomial grows with the order Mof the polynomial and with
the dimensionality Dof the input space. We start by writing down the Mthorder

ijis given
term for a polynomial in Ddimensions in the form
D∑
i1=1D∑
i2=1···D∑
iM=1wi1i2···iMxi1xi2···xiM. (1.133)
The coefﬁcients wi1i2···iMcomprise DMelements, but the number of independent
parameters is signiﬁcantly fewer due to the many interchange symmetries of the
factor xi1xi2···xiM. Begin by showing that the redundancy in the coefﬁcients can
be removed by rewriting this Mthorder term in the form
D∑
i1=1i1∑
i2=1···iM−1∑
iM=1˜wi1i2···iMxi1xi2···xiM. (1.134)

Exercises 61
Note that the precise relationship between the ˜wcoefﬁcients and wcoefﬁcients need
not be made explicit. Use this result to show that the number of independent param-
etersn(D,M), which appear at order M, satisﬁes the following recursion relation
n(D,M)=D∑
i=1n(i, M−1). (1.135)
Next use proof by induction to show that the following result holds
D∑
i=1(i+M−2)!
(i−1)! (M−1)!=(D+M−1)!
(D−1)!M!(1.136)
which can be done by ﬁrst proving the result for D=1 and arbitrary Mby making
use of the result 0 !=1 , then assuming it is correct for dimension Dand verifying
that it is correct for dimension D+1 . Finally, use the two previous results, together
with proof by induction, to show
n(D,M)=(D+M−1)!
(D−1)!M!. (1.137)
To do this, ﬁrst show that the result is true for M=2 , and any value of D⩾1,
by comparison with the result of Exercise 1.14. Then make use of (1.135), together
with (1.136), to show that, if the result holds at order M−1, then it will also hold at
orderM

orderM
1.16 (⋆⋆⋆ )In Exercise 1.15, we proved the result (1.135) for the number of independent
parameters in the Mthorder term of a D-dimensional polynomial. We now ﬁnd an
expression for the total number N(D,M)of independent parameters in all of the
terms up to and including the M6th order. First show that N(D,M)satisﬁes
N(D,M)=M∑
m=0n(D,m) (1.138)
where n(D,m)is the number of independent parameters in the term of order m.
Now make use of the result (1.137), together with proof by induction, to show that
N(d, M)=(D+M)!

N(D,M)=M∑
D!M!. (1.139)
This can be done by ﬁrst proving that the result holds for M=0 and arbitrary
D⩾1, then assuming that it holds at order M, and hence showing that it holds at
orderM+1 . Finally, make use of Stirling’s approximation in the form
n!≃nne−n(1.140)
for large nto show that, for D≫M, the quantity N(D,M)grows like DM,
and for M≫Dit grows like MD. Consider a cubic ( M=3 ) polynomial in D
dimensions, and evaluate numerically the total number of independent parametersfor (i) D=1 0 and (ii) D= 100 , which correspond to typical small-scale and
medium-scale machine learning applications.

62 1. INTRODUCTION
1.17 (⋆⋆)www The gamma function is deﬁned by
Γ(x)≡∫∞
0ux−1e−udu. (1.141)
Using integration by parts, prove the relation Γ(x+1 )= xΓ(x). Show also that
Γ(1) = 1 and hence that Γ(x+1 )= x!whenxis an integer.
1.18 (⋆⋆)www We can use the result (1.126) to derive an expression for the surface
areaSD, and the volume VD, of a sphere of unit radius in Ddimensions. To do this,
consider the following result, which is obtained by transforming from Cartesian to
polar coordinates
D∏
i=1∫∞
−∞e−x2
idxi=SD∫∞
0e−r2rD−1dr. (1.142)
Using the deﬁnition (1.141) of the Gamma function, together with (1.126), evaluate
both sides of this equation, and hence show that
SD=2πD/2
Γ(D/2). (1.143)
Next, by integrating with respect to radius from 0to1, show that the volume of the
unit sphere in Ddimensions is given by
VD=SD
D. (1.144)
Finally, use the results Γ(1) = 1 andΓ(3/2) =√π/2to show that (1.143) and
(1.144) reduce to the usual expressions for D=2 andD=3 .

D∏
i=1∫∞
−∞e−x2
idxi=SD∫∞
SD=2πD/2
VD=SD
D. (1.144)
1.19 (⋆⋆)Consider a sphere of radius ainD-dimensions together with the concentric
hypercube of side 2a, so that the sphere touches the hypercube at the centres of each
of its sides. By using the results of Exercise 1.18, show that the ratio of the volume
of the sphere to the volume of the cube is given by
volume of sphere
volume of cube=πD/2
D2D−1Γ(D/2). (1.145)
Now make use of Stirling’s formula in the form
Γ(x+1 )≃(2π)1/2e−xxx+1/2(1.146)
which is valid for x≫1, to show that, as D→∞ , the ratio (1.145) goes to zero.

Show also that the ratio of the distance from the centre of the hypercube to one of
the corners, divided by the perpendicular distance to one of the sides, is√
D, which
therefore goes to ∞asD→∞ . From these results we see that, in a space of high
dimensionality, most of the volume of a cube is concentrated in the large number of
corners, which themselves become very long ‘spikes’!

Exercises 63
1.20 (⋆⋆)www In this exercise, we explore the behaviour of the Gaussian distribution
in high-dimensional spaces. Consider a Gaussian distribution in Ddimensions given
by
p(x)=1
(2πσ2)D/2exp(
−∥x∥2
2σ2)
. (1.147)
We wish to ﬁnd the density with respect to radius in polar coordinates in which the
direction variables have been integrated out. To do this, show that the integral of
the probability density over a thin shell of radius rand thickness ϵ, where ϵ≪1,i s
given by p(r)ϵwhere
p(r)=SDrD−1
−r2
2σ2)
(1.148)
where SDis the surface area of a unit sphere in Ddimensions. Show that the function
p(r)has a single stationary point located, for large D,a tˆr≃√
Dσ. By considering
p(ˆr+ϵ)where ϵ≪ˆr, show that for large D,
p(ˆr+ϵ)=p(ˆr)e x p(
−3ϵ2
2σ2)
(1.149)
which shows that ˆris a maximum of the radial probability density and also that p(r)
decays exponentially away from its maximum at ˆrwith length scale σ.W e h a v e

−r2
2σ2)
(1.148)
−3ϵ2
2σ2)
(1.149)
already seen that σ≪ˆrfor large D, and so we see that most of the probability
mass is concentrated in a thin shell at large radius. Finally, show that the probability
density p(x)is larger at the origin than at the radius ˆrby a factor of exp(D/2).
We therefore see that most of the probability mass in a high-dimensional Gaussiandistribution is located at a different radius from the region of high probability density.
This property of distributions in spaces of high dimensionality will have important

consequences when we consider Bayesian inference of model parameters in laterchapters.
1.21 (⋆⋆)Consider two nonnegative numbers aandb, and show that, if a⩽b, then
a⩽(ab)
1/2. Use this result to show that, if the decision regions of a two-class
classiﬁcation problem are chosen to minimize the probability of misclassiﬁcation,this probability will satisfy
p(mistake) ⩽∫
{p(x,C1)p(x,C2)}1/2dx. (1.150)
1.22 (⋆)www Given a loss matrix with elements Lkj, the expected risk is minimized

a⩽(ab)
if, for each x, we choose the class that minimizes (1.81). V erify that, when the
loss matrix is given by Lkj=1−Ikj, where Ikjare the elements of the identity
matrix, this reduces to the criterion of choosing the class having the largest posteriorprobability. What is the interpretation of this form of loss matrix?
1.23 (⋆)Derive the criterion for minimizing the expected loss when there is a general
loss matrix and general prior probabilities for the classes.

64 1. INTRODUCTION
1.24 (⋆⋆)www Consider a classiﬁcation problem in which the loss incurred when
an input vector from class Ckis classiﬁed as belonging to class Cjis given by the
loss matrix Lkj, and for which the loss incurred in selecting the reject option is λ.
Find the decision criterion that will give the minimum expected loss. V erify that this
reduces to the reject criterion discussed in Section 1.5.3 when the loss matrix is givenbyL
kj=1−Ikj. What is the relationship between λand the rejection threshold θ?
1.25 (⋆)www Consider the generalization of the squared loss function (1.87) for a
single target variable tto the case of multiple target variables described by the vector
tgiven by
E[L(t,y(x))] =∫∫
∥y(x)−t∥2p(x,t)dxdt. (1.151)
Using the calculus of variations, show that the function y(x)for which this expected
loss is minimized is given by y(x)= Et[t|x]. Show that this result reduces to (1.89)
for the case of a single target variable t.

tgiven by
1.26 (⋆)By expansion of the square in (1.151), derive a result analogous to (1.90) and
hence show that the function y(x)that minimizes the expected squared loss for the
case of a vector tof target variables is again given by the conditional expectation of
t.
1.27 (⋆⋆)www Consider the expected loss for regression problems under the Lqloss
function given by (1.91). Write down the condition that y(x)must satisfy in order
to minimize E[Lq]. Show that, for q=1 , this solution represents the conditional
median, i.e., the function y(x)such that the probability mass for t<y (x)is the

t.
same as for t⩾y(x). Also show that the minimum expected Lqloss for q→0is
given by the conditional mode, i.e., by the function y(x)equal to the value of tthat
maximizes p(t|x)for each x.
1.28 (⋆)In Section 1.6, we introduced the idea of entropy h(x)as the information gained
on observing the value of a random variable xhaving distribution p(x).W e s a w
that, for independent variables xandyfor which p(x, y)=p(x)p(y), the entropy
functions are additive, so that h(x, y)=h(x)+h(y). In this exercise, we derive the

relation between handpin the form of a function h(p). First show that h(p2)=
2h(p), and hence by induction that h(pn)=nh(p)where nis a positive integer.
Hence show that h(pn/m)=(n/m)h(p)where mis also a positive integer. This
implies that h(px)=xh(p)where xis a positive rational number, and hence by
continuity when it is a positive real number. Finally, show that this implies h(p)
must take the form h(p)∝lnp.
1.29 (⋆)www Consider an M-state discrete random variable x, and use Jensen’s in-
equality in the form (1.115) to show that the entropy of its distribution p(x)satisﬁes

H[x]⩽lnM.
1.30 (⋆⋆) Evaluate the Kullback-Leibler divergence (1.113) between two Gaussians
p(x)=N(x|µ, σ2)andq(x)=N(x|m, s2).

Exercises 65
Table 1.3 The joint distribution p(x, y)for two binary variables
xandyused in Exercise 1.39.y
01
x0 1/3 1/3
1 0 1/3
1.31 (⋆⋆)www Consider two variables xandyhaving joint distribution p(x,y). Show
that the differential entropy of this pair of variables satisﬁes
H[x,y]⩽H[x]+H [y] (1.152)
with equality if, and only if, xandyare statistically independent.
1.32 (⋆)Consider a vector xof continuous variables with distribution p(x)and corre-
sponding entropy H[x]. Suppose that we make a nonsingular linear transformation
ofxto obtain a new variable y=Ax. Show that the corresponding entropy is given
byH[y]=H [x]+l n|A|where|A|denotes the determinant of A.
1.33 (⋆⋆)Suppose that the conditional entropy H[y|x]between two discrete random
variables xandyis zero. Show that, for all values of xsuch that p(x)>0, the
variable ymust be a function of x, in other words for each xthere is only one value
ofysuch that p(y|x)̸=0 .
1.34 (⋆⋆)www Use the calculus of variations to show that the stationary point of the

functional (1.108) is given by (1.108). Then use the constraints (1.105), (1.106),
and (1.107) to eliminate the Lagrange multipliers and hence show that the maximumentropy solution is given by the Gaussian (1.109).
1.35 (⋆)
www Use the results (1.106) and (1.107) to show that the entropy of the
univariate Gaussian (1.109) is given by (1.110).
1.36 (⋆)A strictly convex function is deﬁned as one for which every chord lies above

1.35 (⋆)
the function. Show that this is equivalent to the condition that the second derivativeof the function be positive.
1.37 (⋆)Using the deﬁnition (1.111) together with the product rule of probability, prove
the result (1.112).
1.38 (⋆⋆)
www Using proof by induction, show that the inequality (1.114) for convex
functions implies the result (1.115).
1.39 (⋆⋆⋆ )Consider two binary variables xandyhaving the joint distribution given in
Table 1.3.
Evaluate the following quantities
(a)H[x] (c)H[y|x] (e)H[x, y]
(b)H[y] (d)H[x|y] (f)I[x, y].

1.38 (⋆⋆)
Table 1.3.
Draw a diagram to show the relationship between these various quantities.

66 1. INTRODUCTION
1.40 (⋆)By applying Jensen’s inequality (1.115) with f(x)=l n x, show that the arith-
metic mean of a set of real numbers is never less than their geometrical mean.
1.41 (⋆)www Using the sum and product rules of probability, show that the mutual
information I(x,y)satisﬁes the relation (1.121).

2
Probability
Distributions
In Chapter 1, we emphasized the central role played by probability theory in the
solution of pattern recognition problems. We turn now to an exploration of someparticular examples of probability distributions and their properties. As well as be-
ing of great interest in their own right, these distributions can form building blocks
for more complex models and will be used extensively throughout the book. Thedistributions introduced in this chapter will also serve another important purpose,
namely to provide us with the opportunity to discuss some key statistical concepts,
such as Bayesian inference, in the context of simple models before we encounterthem in more complex situations in later chapters.
One role for the distributions discussed in this chapter is to model the prob-
ability distribution p(x)of a random variable x, given a ﬁnite set x
1,...,xNof
observations. This problem is known as density estimation . For the purposes of

this chapter, we shall assume that the data points are independent and identically
distributed. It should be emphasized that the problem of density estimation is fun-
67

68 2. PROBABILITY DISTRIBUTIONS
damentally ill-posed, because there are inﬁnitely many probability distributions that
could have given rise to the observed ﬁnite data set. Indeed, any distribution p(x)
that is nonzero at each of the data points x1,...,xNis a potential candidate. The
issue of choosing an appropriate distribution relates to the problem of model selec-
tion that has already been encountered in the context of polynomial curve ﬁtting inChapter 1 and that is a central issue in pattern recognition.
We begin by considering the binomial and multinomial distributions for discrete
random variables and the Gaussian distribution for continuous random variables.These are speciﬁc examples of parametric distributions, so-called because they are
governed by a small number of adaptive parameters, such as the mean and variance in
the case of a Gaussian for example. To apply such models to the problem of density
estimation, we need a procedure for determining suitable values for the parameters,

given an observed data set. In a frequentist treatment, we choose speciﬁc valuesfor the parameters by optimizing some criterion, such as the likelihood function. By
contrast, in a Bayesian treatment we introduce prior distributions over the parameters
and then use Bayes’ theorem to compute the corresponding posterior distributiongiven the observed data.
We shall see that an important role is played by conjugate priors, that lead to

posterior distributions having the same functional form as the prior, and that there-fore lead to a greatly simpliﬁed Bayesian analysis. For example, the conjugate prior
for the parameters of the multinomial distribution is called the Dirichlet distribution,
while the conjugate prior for the mean of a Gaussian is another Gaussian. All of thesedistributions are examples of the exponential family of distributions, which possess
a number of important properties, and which will be discussed in some detail.

One limitation of the parametric approach is that it assumes a speciﬁc functional
form for the distribution, which may turn out to be inappropriate for a particular
application. An alternative approach is given by nonparametric density estimation
methods in which the form of the distribution typically depends on the size of the data
set. Such models still contain parameters, but these control the model complexity

rather than the form of the distribution. We end this chapter by considering threenonparametric methods based respectively on histograms, nearest-neighbours, and
kernels.
2.1. Binary Variables
We begin by considering a single binary random variable x∈{0,1}. For example,
xmight describe the outcome of ﬂipping a coin, with x=1 representing ‘heads’,
andx=0 representing ‘tails’. We can imagine that this is a damaged coin so that
the probability of landing heads is not necessarily the same as that of landing tails.

kernels.
The probability of x=1 will be denoted by the parameter µso that
p(x=1|µ)=µ (2.1)

2.1. Binary Variables 69
where 0⩽µ⩽1, from which it follows that p(x=0|µ)=1−µ. The probability
distribution over xcan therefore be written in the form
Bern(x|µ)=µx(1−µ)1−x(2.2)
which is known as the Bernoulli distribution. It is easily veriﬁed that this distribution Exercise 2.1
is normalized and that it has mean and variance given by
E[x]= µ (2.3)
var[x]= µ(1−µ). (2.4)
Now suppose we have a data set D={x1,...,x N}of observed values of x.
We can construct the likelihood function, which is a function of µ, on the assumption
that the observations are drawn independently from p(x|µ), so that
p(D|µ)=N∏
n=1p(xn|µ)=N∏
n=1µxn(1−µ)1−xn. (2.5)
In a frequentist setting, we can estimate a value for µby maximizing the likelihood
function, or equivalently by maximizing the logarithm of the likelihood. In the case
of the Bernoulli distribution, the log likelihood function is given by
lnp(D|µ)=N∑
n=1lnp(xn|µ)=N∑
n=1{xnlnµ+( 1−xn)l n ( 1−µ)}. (2.6)

p(D|µ)=N∏
At this point, it is worth noting that the log likelihood function depends on the N
observations xnonly through their sum∑
nxn. This sum provides an example of a
sufﬁcient statistic for the data under this distribution, and we shall study the impor-
tant role of sufﬁcient statistics in some detail. If we set the derivative of lnp(D|µ) Section 2.4
with respect to µequal to zero, we obtain the maximum likelihood estimator
µML=1
NN∑
n=1xn (2.7)
Jacob Bernoulli
1654–1705
Jacob Bernoulli, also known as

µML=1
NN∑
1654–1705
Jacques or James Bernoulli, was aSwiss mathematician and was theﬁrst of many in the Bernoulli familyto pursue a career in science andmathematics. Although compelled
to study philosophy and theology against his will byhis parents, he travelled extensively after graduatingin order to meet with many of the leading scientists ofhis time, including Boyle and Hooke in England. When

µML=1
NN∑
1654–1705
he returned to Switzerland, he taught mechanics andbecame Professor of Mathematics at Basel in 1687.Unfortunately, rivalry between Jacob and his youngerbrother Johann turned an initially productive collabora-tion into a bitter and public dispute. Jacob’s most sig-niﬁcant contributions to mathematics appeared in
The
Art of Conjecture published in 1713, eight years after
his death, which deals with topics in probability the-ory including what has become known as the Bernoullidistribution.

70 2. PROBABILITY DISTRIBUTIONS
Figure 2.1 Histogram plot of the binomial dis-
tribution (2.9) as a function of mfor
N=1 0 andµ=0.25.
m0 1 2 3 4 5 6 7 8 9 1000.10.20.3
which is also known as the sample mean . If we denote the number of observations
ofx=1 (heads) within this data set by m, then we can write (2.7) in the form
µML=m
N(2.8)
so that the probability of landing heads is given, in this maximum likelihood frame-
work, by the fraction of observations of heads in the data set.
Now suppose we ﬂip a coin, say, 3 times and happen to observe 3 heads. Then
N=m=3 andµML=1 . In this case, the maximum likelihood result would
predict that all future observations should give heads. Common sense tells us that
this is unreasonable, and in fact this is an extreme example of the over-ﬁtting associ-
ated with maximum likelihood. We shall see shortly how to arrive at more sensible
conclusions through the introduction of a prior distribution over µ.

We can also work out the distribution of the number mof observations of x=1 ,
given that the data set has size N. This is called the binomial distribution, and
from (2.5) we see that it is proportional to µm(1−µ)N−m. In order to obtain the
normalization coefﬁcient we note that out of Ncoin ﬂips, we have to add up all
of the possible ways of obtaining mheads, so that the binomial distribution can be
written
Bin(m|N,µ)=(N
m)
µm(1−µ)N−m(2.9)
where (N
m)
≡N!
(N−m)!m!(2.10)

written
m)
where (N
m)
≡N!
is the number of ways of choosing mobjects out of a total of Nidentical objects. Exercise 2.3
Figure 2.1 shows a plot of the binomial distribution for N=1 0 andµ=0.25.
The mean and variance of the binomial distribution can be found by using the
result of Exercise 1.10, which shows that for independent events the mean of the
sum is the sum of the means, and the variance of the sum is the sum of the variances.
Because m=x1+...+xN, and for each observation the mean and variance are

2.1. Binary Variables 71
given by (2.3) and (2.4), respectively, we have
E[m]≡N∑
m=0mBin(m|N,µ)= Nµ (2.11)
var[m]≡N∑
m=0(m−E[m])2Bin(m|N,µ)= Nµ(1−µ). (2.12)
These results can also be proved directly using calculus. Exercise 2.4
2.1.1 The beta distribution
We have seen in (2.8) that the maximum likelihood setting for the parameter µ
in the Bernoulli distribution, and hence in the binomial distribution, is given by the
fraction of the observations in the data set having x=1 . As we have already noted,
this can give severely over-ﬁtted results for small data sets. In order to develop a
Bayesian treatment for this problem, we need to introduce a prior distribution p(µ)
over the parameter µ. Here we consider a form of prior distribution that has a simple
interpretation as well as some useful analytical properties. To motivate this prior,
we note that the likelihood function takes the form of the product of factors of theformµ
x(1−µ)1−x. If we choose a prior to be proportional to powers of µand

(1−µ), then the posterior distribution, which is proportional to the product of the
prior and the likelihood function, will have the same functional form as the prior.This property is called conjugacy and we will see several examples of it later in this
chapter. We therefore choose a prior, called the beta distribution, given by
Beta(µ|a, b)=Γ(a+b)
Γ(a)Γ(b)µa−1(1−µ)b−1(2.13)
where Γ(x)is the gamma function deﬁned by (1.141), and the coefﬁcient in (2.13)

ensures that the beta distribution is normalized, so that Exercise 2.5
∫1
0Beta(µ|a, b)dµ=1. (2.14)
The mean and variance of the beta distribution are given by Exercise 2.6
E[µ]=a
a+b(2.15)
var[µ]=ab
(a+b)2(a+b+1 ). (2.16)
The parameters aandbare often called hyperparameters because they control the
distribution of the parameter µ. Figure 2.2 shows plots of the beta distribution for
various values of the hyperparameters.
The posterior distribution of µis now obtained by multiplying the beta prior

∫1
E[µ]=a
a+b(2.15)
var[µ]=ab
(2.13) by the binomial likelihood function (2.9) and normalizing. Keeping only thefactors that depend on µ, we see that this posterior distribution has the form
p(µ|m, l, a, b )∝µ
m+a−1(1−µ)l+b−1(2.17)

72 2. PROBABILITY DISTRIBUTIONS
µa=0.1
b=0.1
0 0.5 10123
µa=1
b=1
0 0.5 10123
µa=2
b=3
0 0.5 10123
µa=8
b=4
0 0.5 10123
Figure 2.2 Plots of the beta distribution Beta( µ|a, b)given by (2.13) as a function of µfor various values of the
hyperparameters aandb.
where l=N−m, and therefore corresponds to the number of ‘tails’ in the coin
example. We see that (2.17) has the same functional dependence on µas the prior
distribution, reﬂecting the conjugacy properties of the prior with respect to the like-
lihood function. Indeed, it is simply another beta distribution, and its normalization
coefﬁcient can therefore be obtained by comparison with (2.13) to give
p(µ|m, l, a, b )=Γ(m+a+l+b)
Γ(m+a)Γ(l+b)µm+a−1(1−µ)l+b−1. (2.18)
We see that the effect of observing a data set of mobservations of x=1 and
lobservations of x=0 has been to increase the value of abym, and the value of
bbyl, in going from the prior distribution to the posterior distribution. This allows

us to provide a simple interpretation of the hyperparameters aandbin the prior as
aneffective number of observations ofx=1 andx=0 , respectively. Note that
aandbneed not be integers. Furthermore, the posterior distribution can act as the
prior if we subsequently observe additional data. To see this, we can imagine taking
observations one at a time and after each observation updating the current posterior

2.1. Binary Variables 73
µprior
0 0.5 1012
µlikelihood function
0 0.5 1012
µposterior
0 0.5 1012
Figure 2.3 Illustration of one step of sequential Bayesian inference. The prior is given by a beta distribution
with parameters a=2,b=2, and the likelihood function, given by (2.9) with N=m=1, corresponds to a
single observation of x=1, so that the posterior is given by a beta distribution with parameters a=3,b=2.
distribution by multiplying by the likelihood function for the new observation and
then normalizing to obtain the new, revised posterior distribution. At each stage, the
posterior is a beta distribution with some total number of (prior and actual) observed
values for x=1 andx=0 given by the parameters aandb. Incorporation of an
additional observation of x=1 simply corresponds to incrementing the value of a
by1, whereas for an observation of x=0 we increment bby1. Figure 2.3 illustrates
one step in this process.
We see that this sequential approach to learning arises naturally when we adopt

a Bayesian viewpoint. It is independent of the choice of prior and of the likelihood
function and depends only on the assumption of i.i.d. data. Sequential methods make
use of observations one at a time, or in small batches, and then discard them before
the next observations are used. They can be used, for example, in real-time learning
scenarios where a steady stream of data is arriving, and predictions must be made
before all of the data is seen. Because they do not require the whole data set to be
stored or loaded into memory, sequential methods are also useful for large data sets.

Maximum likelihood methods can also be cast into a sequential framework. Section 2.3.5
If our goal is to predict, as best we can, the outcome of the next trial, then we
must evaluate the predictive distribution of x, given the observed data set D. From
the sum and product rules of probability, this takes the form
p(x=1|D)=∫1
0p(x=1|µ)p(µ|D)dµ=∫1
0µp(µ|D)dµ=E[µ|D].(2.19)
Using the result (2.18) for the posterior distribution p(µ|D), together with the result
(2.15) for the mean of the beta distribution, we obtain

p(x=1|D)=m+a
m+a+l+b(2.20)
which has a simple interpretation as the total fraction of observations (both real ob-
servations and ﬁctitious prior observations) that correspond to x=1 . Note that in
the limit of an inﬁnitely large data set m, l→∞ the result (2.20) reduces to the
maximum likelihood result (2.8). As we shall see, it is a very general property that
the Bayesian and maximum likelihood results will agree in the limit of an inﬁnitely

74 2. PROBABILITY DISTRIBUTIONS
large data set. For a ﬁnite data set, the posterior mean for µalways lies between the
prior mean and the maximum likelihood estimate for µcorresponding to the relative
frequencies of events given by (2.7). Exercise 2.7
From Figure 2.2, we see that as the number of observations increases, so the
posterior distribution becomes more sharply peaked. This can also be seen fromthe result (2.16) for the variance of the beta distribution, in which we see that the
variance goes to zero for a→∞ orb→∞ . In fact, we might wonder whether it is
a general property of Bayesian learning that, as we observe more and more data, theuncertainty represented by the posterior distribution will steadily decrease.
To address this, we can take a frequentist view of Bayesian learning and show
that, on average, such a property does indeed hold. Consider a general Bayesian
inference problem for a parameter θfor which we have observed a data set D, de-

scribed by the joint distribution p(θ,D). The following result Exercise 2.8
E
θ[θ]= ED[Eθ[θ|D]] (2.21)
where
Eθ[θ]≡∫
p(θ)θdθ (2.22)
ED[Eθ[θ|D]]≡∫{∫
θp(θ|D)dθ}
p(D)dD (2.23)
says that the posterior mean of θ, averaged over the distribution generating the data,
is equal to the prior mean of θ. Similarly, we can show that
varθ[θ]= ED[varθ[θ|D]] + var D[Eθ[θ|D]]. (2.24)
The term on the left-hand side of (2.24) is the prior variance of θ. On the right-
hand side, the ﬁrst term is the average posterior variance of θ, and the second term

E
where
Eθ[θ]≡∫
measures the variance in the posterior mean of θ. Because this variance is a positive
quantity, this result shows that, on average, the posterior variance of θis smaller than
the prior variance. The reduction in variance is greater if the variance in the posterior
mean is greater. Note, however, that this result only holds on average, and that for a
particular observed data set it is possible for the posterior variance to be larger thanthe prior variance.
2.2. Multinomial Variables

Binary variables can be used to describe quantities that can take one of two possiblevalues. Often, however, we encounter discrete variables that can take on one of K
possible mutually exclusive states. Although there are various alternative ways to
express such variables, we shall see shortly that a particularly convenient represen-tation is the 1-of-Kscheme in which the variable is represented by a K-dimensional
vectorxin which one of the elements x
kequals 1, and all remaining elements equal

2.2. Multinomial Variables 75
0. So, for instance if we have a variable that can take K=6 states and a particular
observation of the variable happens to correspond to the state where x3=1 , thenx
will be represented by
x=( 0,0,1,0,0,0)T. (2.25)
Note that such vectors satisfy∑K
k=1xk=1 . If we denote the probability of xk=1
by the parameter µk, then the distribution of xis given
p(x|µ)=K∏
k=1µxk
k(2.26)
where µ=(µ1,...,µ K)T, and the parameters µkare constrained to satisfy µk⩾0
and∑
kµk=1 , because they represent probabilities. The distribution (2.26) can be
regarded as a generalization of the Bernoulli distribution to more than two outcomes.It is easily seen that the distribution is normalized
∑
xp(x|µ)=K∑
k=1µk=1 (2.27)
and that
E[x|µ]=∑
xp(x|µ)x=(µ1,...,µ M)T=µ. (2.28)
Now consider a data set DofNindependent observations x1,...,xN. The
corresponding likelihood function takes the form
p(D|µ)=N∏
n=1K∏
k=1µxnk
k=K∏
k=1µ(P
nxnk)
k=K∏
k=1µmk
k. (2.29)

and∑
∑
and that
E[x|µ]=∑
p(D|µ)=N∏
n=1K∏
k=1µxnk
k=K∏
k=1µ(P
nxnk)
k=K∏
k=1µmk
k. (2.29)
We see that the likelihood function depends on the Ndata points only through the
Kquantities
mk=∑
nxnk (2.30)
which represent the number of observations of xk=1 . These are called the sufﬁcient
statistics for this distribution. Section 2.4
In order to ﬁnd the maximum likelihood solution for µ, we need to maximize
lnp(D|µ)with respect to µktaking account of the constraint that the µkmust sum
to one. This can be achieved using a Lagrange multiplier λand maximizing Appendix E
K∑
k=1mklnµk+λ(K∑
k=1µk−1)
. (2.31)

mk=∑
K∑
k=1µk−1)
. (2.31)
Setting the derivative of (2.31) with respect to µkto zero, we obtain
µk=−mk/λ. (2.32)

76 2. PROBABILITY DISTRIBUTIONS
We can solve for the Lagrange multiplier λby substituting (2.32) into the constraint∑
kµk=1 to give λ=−N. Thus we obtain the maximum likelihood solution in
the form
µML
k=mk
N(2.33)
which is the fraction of the Nobservations for which xk=1 .
We can consider the joint distribution of the quantities m1,...,m K, conditioned
on the parameters µand on the total number Nof observations. From (2.29) this
Mult( m1,m2,...,m K|µ,N)=(N
m1m2...m K)K∏
k=1µmk
k(2.34)
which is known as the multinomial distribution. The normalization coefﬁcient is the
number of ways of partitioning Nobjects into Kgroups of size m1,...,m Kand is
given by(N
m1m2...m K)
=N!
m1!m2!...m K!. (2.35)
Note that the variables mkare subject to the constraint
K∑
k=1mk=N. (2.36)
2.2.1 The Dirichlet distribution
We now introduce a family of prior distributions for the parameters {µk}of

k=1µmk
k(2.34)
given by(N
=N!
K∑
the multinomial distribution (2.34). By inspection of the form of the multinomialdistribution, we see that the conjugate prior is given by
p(µ|α)∝K∏
k=1µαk−1
k(2.37)
where 0⩽µk⩽1and∑
kµk=1 . Here α1,...,α Kare the parameters of the
distribution, and αdenotes (α1,...,α K)T. Note that, because of the summation
constraint, the distribution over the space of the {µk}is conﬁned to a simplex of
dimensionality K−1, as illustrated for K=3 in Figure 2.4.
The normalized form for this distribution is by Exercise 2.9
Dir(µ|α)=Γ(α0)
Γ(α1)···Γ(αK)K∏
k=1µαk−1

p(µ|α)∝K∏
k=1µαk−1
k(2.37)
k=1µαk−1
k(2.38)
which is called the Dirichlet distribution. Here Γ(x)is the gamma function deﬁned
by (1.141) while
α0=K∑
k=1αk. (2.39)

2.2. Multinomial Variables 77
Figure 2.4 The Dirichlet distribution over three variables µ1,µ2,µ3
is conﬁned to a simplex (a bounded linear manifold) of
the form shown, as a consequence of the constraints
0⩽µk⩽1andP
kµk=1.
µ1µ2
µ3
Plots of the Dirichlet distribution over the simplex, for various settings of the param-
etersαk, are shown in Figure 2.5.
Multiplying the prior (2.38) by the likelihood function (2.34), we obtain the
posterior distribution for the parameters {µk}in the form
p(µ|D,α)∝p(D|µ)p(µ|α)∝K∏
k=1µαk+mk−1
k. (2.40)
We see that the posterior distribution again takes the form of a Dirichlet distribution,
conﬁrming that the Dirichlet is indeed a conjugate prior for the multinomial. This
allows us to determine the normalization coefﬁcient by comparison with (2.38) so
that
p(µ|D,α)=D i r ( µ|α+m)
=Γ(α0+N)
Γ(α1+m1)···Γ(αK+mK)K∏
k(2.41)
where we have denoted m=(m1,...,m K)T. As for the case of the binomial

k. (2.40)
that
=Γ(α0+N)
k(2.41)
distribution with its beta prior, we can interpret the parameters αkof the Dirichlet
prior as an effective number of observations of xk=1 .
Note that two-state quantities can either be represented as binary variables and
Lejeune Dirichlet
1805–1859
Johann Peter Gustav Lejeune
Dirichlet was a modest and re-
served mathematician who made
contributions in number theory, me-
chanics, and astronomy, and who
gave the ﬁrst rigorous analysis of
Fourier series. His family originated from Richelet

1805–1859
in Belgium, and the name Lejeune Dirichlet comesfrom ‘le jeune de Richelet’ (the young person from
Richelet). Dirichlet’s ﬁrst paper, which was published
in 1825, brought him instant fame. It concerned Fer-
mat’s last theorem, which claims that there are no
positive integer solutions to xn+yn=znforn>2.
Dirichlet gave a partial proof for the case n=5, which
was sent to Legendre for review and who in turn com-
pleted the proof. Later, Dirichlet gave a complete proof
forn=1 4 , although a full proof of Fermat’s last theo-

rem for arbitrary nhad to wait until the work of Andrew
Wiles in the closing years of the 20thcentury.

78 2. PROBABILITY DISTRIBUTIONS
Figure 2.5 Plots of the Dirichlet distribution over three vari ables, where the two horizontal axes are coordinates
in the plane of the simplex and the vertical axis corresponds to the value of the density . Here {αk}=0.1on the
left plot, {αk}=1 in the centre plot, and {αk}=1 0 in the right plot.
modelled using the binomial distribution (2.9) or as 1-of-2 variables and modelled
using the multinomial distribution (2.34) with K=2 .
2.3.
 The Gaussian Distribution
The Gaussian, also known as the normal distribution, is a widely used model for the
distribution of continuous variables. In the case of a single variable x, the Gaussian
distribution can be written in the form
N(x|µ,σ2)=1
−1
(2.42)
where µis the mean and σ2is the variance. For a D-dimensional vector x,t h e
multivariate Gaussian distribution takes the form
(2π)D/ 21
−1
(2.43)
where µis aD-dimensional mean vector, Σis aD×Dcovariance matrix, and |Σ|

−1
(2.42)
(2π)D/ 21
−1
(2.43)
denotes the determinant of Σ.
The Gaussian distribution arises in many different contexts and can be motivated
from a variety of different perspectives. For example, we have already seen that for Section 1.6
a single real variable, the distribution that maximizes the entropy is the Gaussian.
This property applies also to the multivariate Gaussian. Exercise 2.14
Another situation in which the Gaussian distribution arises is when we consider
the sum of multiple random variables. The central limit theorem (due to Laplace)

tells us that, subject to certain mild conditions, the sum of a set of random variables,
which is of course itself a random variable, has a distribution that becomes increas-
ingly Gaussian as the number of terms in the sum increases (Walker, 1969). We can

2.3. The Gaussian Distribution 79
N=1
0 0.5 10123N=2
0 0.5 10123N=1 0
0 0.5 10123
Figure 2.6 Histogram plots of the mean of Nuniformly distributed numbers for various values of N.W e
observe that as Nincreases, the distribution tends towards a Gaussian.
illustrate this by considering Nvariables x1,...,x Neach of which has a uniform
distribution over the interval [0,1]and then considering the distribution of the mean
(x1+···+xN)/N. For large N, this distribution tends to a Gaussian, as illustrated
in Figure 2.6. In practice, the convergence to a Gaussian as Nincreases can be
very rapid. One consequence of this result is that the binomial distribution (2.9),
which is a distribution over mdeﬁned by the sum of Nobservations of the random
binary variable x, will tend to a Gaussian as N→∞ (see Figure 2.1 for the case of
N=1 0 ).
The Gaussian distribution has many important analytical properties, and we shall
consider several of these in detail. As a result, this section will be rather more tech-

N=1 0 ).
nically involved than some of the earlier sections, and will require familiarity with
various matrix identities. However, we strongly encourage the reader to become pro- Appendix C
ﬁcient in manipulating Gaussian distributions using the techniques presented here as
this will prove invaluable in understanding the more complex models presented in
later chapters.
We begin by considering the geometrical form of the Gaussian distribution. The
Carl Friedrich Gauss
1777–1855
It is said that when Gauss went

1777–1855
to elementary school at age 7, his
teacher B ¨uttner, trying to keep the
class occupied, asked the pupils to
sum the integers from 1 to 100. To
the teacher’s amazement, Gauss
arrived at the answer in a matter of moments by noting
that the sum can be represented as 50 pairs ( 1 + 100 ,
2+99 , etc.) each of which added to 101, giving the an-
swer 5,050. It is now believed that the problem which
was actually set was of the same form but somewhat
harder in that the sequence had a larger starting value

and a larger increment. Gauss was a German math-ematician and scientist with a reputation for being a
hard-working perfectionist. One of his many contribu-
tions was to show that least squares can be derived
under the assumption of normally distributed errors.
He also created an early formulation of non-Euclidean
geometry (a self-consistent geometrical theory that vi-
olates the axioms of Euclid) but was reluctant to dis-
cuss it openly for fear that his reputation might suffer

if it were seen that he believed in such a geometry.
At one point, Gauss was asked to conduct a geodetic
survey of the state of Hanover, which led to his for-
mulation of the normal distribution, now also known
as the Gaussian. After his death, a study of his di-
aries revealed that he had discovered several impor-
tant mathematical results years or even decades be-
fore they were published by others.

80 2. PROBABILITY DISTRIBUTIONS
functional dependence of the Gaussian on xis through the quadratic form
∆2=(x−µ)TΣ−1(x−µ) (2.44)
which appears in the exponent. The quantity ∆is called the Mahalanobis distance
fromµtoxand reduces to the Euclidean distance when Σis the identity matrix. The
Gaussian distribution will be constant on surfaces in x-space for which this quadratic
form is constant.
First of all, we note that the matrix Σcan be taken to be symmetric, without
loss of generality, because any antisymmetric component would disappear from the
exponent. Now consider the eigenvector equation for the covariance matrix Exercise 2.17
Σui=λiui (2.45)
where i=1,...,D . Because Σis a real, symmetric matrix its eigenvalues will be
real, and its eigenvectors can be chosen to form an orthonormal set, so that Exercise 2.18
uT
iuj=Iij (2.46)
where Iijis thei, jelement of the identity matrix and satisﬁes
Iij={
1,ifi=j
0,otherwise.(2.47)
The covariance matrix Σcan be expressed as an expansion in terms of its eigenvec-

uT
Iij={
1,ifi=j
tors in the form Exercise 2.19
Σ=D∑
i=1λiuiuT
i (2.48)
and similarly the inverse covariance matrix Σ−1can be expressed as
Σ−1=D∑
i=11
λiuiuT
i. (2.49)
Substituting (2.49) into (2.44), the quadratic form becomes
∆2=D∑
i=1y2
i
λi(2.50)
yi=uT
i(x−µ). (2.51)
We can interpret {yi}as a new coordinate system deﬁned by the orthonormal vectors
uithat are shifted and rotated with respect to the original xicoordinates. Forming
the vector y=(y1,...,y D)T,w eh a v e
y=U(x−µ) (2.52)

2.3. The Gaussian Distribution 81
Figure 2.7 The red curve shows the ellip-
tical surface of constant proba-
bility density for a Gaussian in
a two-dimensional space x=
(x1,x2)on which the density
isexp(−1/2)of its value at
x=µ. The major axes of
the ellipse are deﬁned by the
eigenvectors uiof the covari-
ance matrix, with correspond-
ing eigenvalues λi.
x1x2
λ1/2
1λ1/2
2y1y2u1u2
µ
whereUis a matrix whose rows are given by uT
i. From (2.46) it follows that Uis
anorthogonal matrix, i.e., it satisﬁes UUT=I, and hence also UTU=I, where I Appendix C
is the identity matrix.
The quadratic form, and hence the Gaussian density, will be constant on surfaces
for which (2.51) is constant. If all of the eigenvalues λiare positive, then these
surfaces represent ellipsoids, with their centres at µand their axes oriented along ui,
and with scaling factors in the directions of the axes given by λ1/2
i, as illustrated in
Figure 2.7.
For the Gaussian distribution to be well deﬁned, it is necessary for all of the

eigenvalues λiof the covariance matrix to be strictly positive, otherwise the dis-
tribution cannot be properly normalized. A matrix whose eigenvalues are strictly
positive is said to be positive deﬁnite . In Chapter 12, we will encounter Gaussian
distributions for which one or more of the eigenvalues are zero, in which case the
distribution is singular and is conﬁned to a subspace of lower dimensionality. If all
of the eigenvalues are nonnegative, then the covariance matrix is said to be positive
semideﬁnite .

Now consider the form of the Gaussian distribution in the new coordinate system
deﬁned by the yi. In going from the xto theycoordinate system, we have a Jacobian
matrixJwith elements given by
Jij=∂xi
∂yj=Uji (2.53)
where Ujiare the elements of the matrix UT. Using the orthonormality property of
the matrix U, we see that the square of the determinant of the Jacobian matrix is
|J|2=⏐⏐UT⏐⏐2=⏐⏐UT⏐⏐|U|=⏐⏐UTU⏐⏐=|I|=1 (2.54)
and hence |J|=1. Also, the determinant |Σ|of the covariance matrix can be written

82 2. PROBABILITY DISTRIBUTIONS
as the product of its eigenvalues, and hence
|Σ|1/2=D∏
j=1λ1/2
j. (2.55)
Thus in the yjcoordinate system, the Gaussian distribution takes the form
p(y)=p(x)|J|=D∏
j=11
(2πλj)1/2exp{
−y2
j
2λj}
(2.56)
which is the product of Dindependent univariate Gaussian distributions. The eigen-
vectors therefore deﬁne a new set of shifted and rotated coordinates with respectto which the joint probability distribution factorizes into a product of independent
distributions. The integral of the distribution in the ycoordinate system is then
∫
p(y)dy=D∏
j=1∫∞
−∞1
−y2
j
2λj}
dyj=1 (2.57)
where we have used the result (1.48) for the normalization of the univariate Gaussian.
This conﬁrms that the multivariate Gaussian (2.43) is indeed normalized.
We now look at the moments of the Gaussian distribution and thereby provide an
interpretation of the parameters µandΣ. The expectation of xunder the Gaussian
distribution is given by
E[x]=1
(2π)D/21
|Σ|1/2∫
exp{
−1
xdx
=1

∫
p(y)dy=D∏
j=1∫∞
−∞1
−y2
j
2λj}
E[x]=1
(2π)D/21
|Σ|1/2∫
exp{
−1
xdx
=1
(2π)D/21
|Σ|1/2∫
exp{
−1
2zTΣ−1z}
(z+µ)dz (2.58)
where we have changed variables using z=x−µ. We now note that the exponent
is an even function of the components of zand, because the integrals over these are
taken over the range (−∞,∞), the term in zin the factor (z+µ)will vanish by
symmetry. Thus
E[x]=µ (2.59)
and so we refer to µas the mean of the Gaussian distribution.
We now consider second order moments of the Gaussian. In the univariate case,
we considered the second order moment given by E[x2]. For the multivariate Gaus-

−1
2zTΣ−1z}
sian, there are D2second order moments given by E[xixj], which we can group
together to form the matrix E[xxT]. This matrix can be written as
E[xxT]=1
(2π)D/21
|Σ|1/2∫
exp{
−1
xxTdx
=1
(2π)D/21
|Σ|1/2∫
exp{
−1
2zTΣ−1z}
(z+µ)(z+µ)Tdz

2.3. The Gaussian Distribution 83
where again we have changed variables using z=x−µ. Note that the cross-terms
involving µzTandµTzwill again vanish by symmetry. The term µµTis constant
and can be taken outside the integral, which itself is unity because the Gaussian
distribution is normalized. Consider the term involving zzT. Again, we can make
use of the eigenvector expansion of the covariance matrix given by (2.45), togetherwith the completeness of the set of eigenvectors, to write
z=D∑
j=1yjuj (2.60)
where yj=uT
jz, which gives
1
(2π)D/21
|Σ|1/2∫
exp{
−1
2zTΣ−1z}
zzTdz
=1
(2π)D/21
|Σ|1/2D∑
i=1D∑
j=1uiuT
j∫
exp{
−D∑
k=1y2
k
2λk}
yiyjdy
=D∑
i=1uiuT
iλi=Σ (2.61)
where we have made use of the eigenvector equation (2.45), together with the fact
that the integral on the right-hand side of the middle line vanishes by symmetryunless i=j, and in the ﬁnal line we have made use of the results (1.50) and (2.55),
together with (2.48). Thus we have
E[xx
T]=µµT+Σ. (2.62)

z=D∑
1
(2π)D/21
|Σ|1/2∫
exp{
−1
2zTΣ−1z}
zzTdz
=1
(2π)D/21
|Σ|1/2D∑
i=1D∑
j=1uiuT
j∫
exp{
−D∑
k=1y2
k
2λk}
yiyjdy
=D∑
i=1uiuT
E[xx
For single random variables, we subtracted the mean before taking second mo-
ments in order to deﬁne a variance. Similarly, in the multivariate case it is again
convenient to subtract off the mean, giving rise to the covariance of a random vector
xdeﬁned by
cov[x]= E[
. (2.63)
For the speciﬁc case of a Gaussian distribution, we can make use of E[x]=µ,
together with the result (2.62), to give
cov[x]=Σ. (2.64)
Because the parameter matrix Σgoverns the covariance of xunder the Gaussian

xdeﬁned by
cov[x]= E[
. (2.63)
distribution, it is called the covariance matrix.
Although the Gaussian distribution (2.43) is widely used as a density model, it
suffers from some signiﬁcant limitations. Consider the number of free parameters in
the distribution. A general symmetric covariance matrix Σwill have D(D+1 )/2
independent parameters, and there are another Dindependent parameters in µ,g i v - Exercise 2.21
ingD(D+3 )/2parameters in total. For large D, the total number of parameters

84 2. PROBABILITY DISTRIBUTIONS
Figure 2.8 Contours of constant
probability density for a Gaussian
distribution in two dimensions in
which the covariance matrix is (a) of
general form, (b) diagonal, in which
the elliptical contours are aligned
with the coordinate axes, and (c)
proportional to the identity matrix, in
which the contours are concentric
circles.x1x2
(a)x1x2
(b)x1x2
(c)
therefore grows quadratically with D, and the computational task of manipulating
and inverting large matrices can become prohibitive. One way to address this prob-
lem is to use restricted forms of the covariance matrix. If we consider covariance
matrices that are diagonal , so that Σ= diag( σ2
i), we then have a total of 2Dinde-
pendent parameters in the density model. The corresponding contours of constant
density are given by axis-aligned ellipsoids. We could further restrict the covariance
matrix to be proportional to the identity matrix, Σ=σ2I, known as an isotropic co-

variance, giving D+1independent parameters in the model and spherical surfaces
of constant density. The three possibilities of general, diagonal, and isotropic covari-
ance matrices are illustrated in Figure 2.8. Unfortunately, whereas such approaches
limit the number of degrees of freedom in the distribution and make inversion of the
covariance matrix a much faster operation, they also greatly restrict the form of the
probability density and limit its ability to capture interesting correlations in the data.

A further limitation of the Gaussian distribution is that it is intrinsically uni-
modal (i.e., has a single maximum) and so is unable to provide a good approximation
to multimodal distributions. Thus the Gaussian distribution can be both too ﬂexible,
in the sense of having too many parameters, while also being too limited in the range
of distributions that it can adequately represent. We will see later that the introduc-
tion of latent variables, also called hidden variables or unobserved variables, allows

both of these problems to be addressed. In particular, a rich family of multimodal
distributions is obtained by introducing discrete latent variables leading to mixtures
of Gaussians, as discussed in Section 2.3.9. Similarly, the introduction of continuous
latent variables, as described in Chapter 12, leads to models in which the number of
free parameters can be controlled independently of the dimensionality Dof the data
space while still allowing the model to capture the dominant correlations in the data
set. Indeed, these two approaches can be combined and further extended to derive

a very rich set of hierarchical models that can be adapted to a broad range of prac-
tical applications. For instance, the Gaussian version of the Markov random ﬁeld , Section 8.3
which is widely used as a probabilistic model of images, is a Gaussian distribution
over the joint space of pixel intensities but rendered tractable through the imposition
of considerable structure reﬂecting the spatial organization of the pixels. Similarly,

the linear dynamical system , used to model time series data for applications such Section 13.3
as tracking, is also a joint Gaussian distribution over a potentially large number of
observed and latent variables and again is tractable due to the structure imposed on
the distribution. A powerful framework for expressing the form and properties of

2.3. The Gaussian Distribution 85
such complex distributions is that of probabilistic graphical models, which will form
the subject of Chapter 8.
2.3.1 Conditional Gaussian distributions
An important property of the multivariate Gaussian distribution is that if two
sets of variables are jointly Gaussian, then the conditional distribution of one set
conditioned on the other is again Gaussian. Similarly, the marginal distribution of
either set is also Gaussian.
Consider ﬁrst the case of conditional distributions. Suppose xis aD-dimensional
vector with Gaussian distribution N(x|µ,Σ)and that we partition xinto two dis-
joint subsets xaandxb. Without loss of generality, we can take xato form the ﬁrst
Mcomponents of x, withxbcomprising the remaining D−Mcomponents, so that
x=(
xa
xb)
. (2.65)
We also deﬁne corresponding partitions of the mean vector µgiven by
µ=(
µa
µb)
(2.66)
and of the covariance matrix Σgiven by
Σ=(
ΣaaΣab
ΣbaΣbb)
. (2.67)
Note that the symmetry ΣT=Σof the covariance matrix implies that ΣaaandΣbb

x=(
xa
xb)
. (2.65)
µ=(
µa
µb)
(2.66)
Σ=(
ΣaaΣab
ΣbaΣbb)
. (2.67)
are symmetric, while Σba=ΣT
ab.
In many situations, it will be convenient to work with the inverse of the covari-
ance matrix
Λ≡Σ−1(2.68)
which is known as the precision matrix . In fact, we shall see that some properties
of Gaussian distributions are most naturally expressed in terms of the covariance,
whereas others take a simpler form when viewed in terms of the precision. We
therefore also introduce the partitioned form of the precision matrix
Λ=(
ΛaaΛab
ΛbaΛbb)
(2.69)

ab.
Λ=(
ΛaaΛab
ΛbaΛbb)
(2.69)
corresponding to the partitioning (2.65) of the vector x. Because the inverse of a
symmetric matrix is also symmetric, we see that ΛaaandΛbbare symmetric, while Exercise 2.22
ΛT
ab=Λba. It should be stressed at this point that, for instance, Λaais not simply
given by the inverse of Σaa. In fact, we shall shortly examine the relation between
the inverse of a partitioned matrix and the inverses of its partitions.
Let us begin by ﬁnding an expression for the conditional distribution p(xa|xb).

ΛbaΛbb)
(2.69)
ΛT
From the product rule of probability, we see that this conditional distribution can be

86 2. PROBABILITY DISTRIBUTIONS
evaluated from the joint distribution p(x)=p(xa,xb)simply by ﬁxing xbto the
observed value and normalizing the resulting expression to obtain a valid probabilitydistribution over x
a. Instead of performing this normalization explicitly, we can
obtain the solution more efﬁciently by considering the quadratic form in the exponent
of the Gaussian distribution given by (2.44) and then reinstating the normalizationcoefﬁcient at the end of the calculation. If we make use of the partitioning (2.65),
(2.66), and (2.69), we obtain
−1
2(x−µ)TΣ−1(x−µ)=
−1
2(xa−µa)TΛaa(xa−µa)−1
2(xa−µa)TΛab(xb−µb)
−1
2(xb−µb)TΛba(xa−µa)−1
2(xb−µb)TΛbb(xb−µb).(2.70)
We see that as a function of xa, this is again a quadratic form, and hence the cor-
responding conditional distribution p(xa|xb)will be Gaussian. Because this distri-
bution is completely characterized by its mean and its covariance, our goal will beto identify expressions for the mean and covariance of p(x
a|xb)by inspection of
(2.70).

−1
−1
−1
(2.70).
This is an example of a rather common operation associated with Gaussian
distributions, sometimes called ‘completing the square’, in which we are given a
quadratic form deﬁning the exponent terms in a Gaussian distribution, and we need
to determine the corresponding mean and covariance. Such problems can be solvedstraightforwardly by noting that the exponent in a general Gaussian distribution
N(x|µ,Σ)can be written
−1
2(x−µ)TΣ−1(x−µ)=−1
2xTΣ−1x+xTΣ−1µ+c o n s t (2.71)

(2.70).
−1
where ‘const’ denotes terms which are independent of x, and we have made use of
the symmetry of Σ. Thus if we take our general quadratic form and express it in
the form given by the right-hand side of (2.71), then we can immediately equate the
matrix of coefﬁcients entering the second order term in xto the inverse covariance
matrixΣ−1and the coefﬁcient of the linear term in xtoΣ−1µ, from which we can
obtain µ.
Now let us apply this procedure to the conditional Gaussian distribution p(xa|xb)

obtain µ.
for which the quadratic form in the exponent is given by (2.70). We will denote themean and covariance of this distribution by µ
a|bandΣa|b, respectively. Consider
the functional dependence of (2.70) on xain which xbis regarded as a constant. If
we pick out all terms that are second order in xa,w eh a v e
−1
2xT
aΛaaxa (2.72)
from which we can immediately conclude that the covariance (inverse precision) of
p(xa|xb)is given by
Σa|b=Λ−1
aa. (2.73)

2.3. The Gaussian Distribution 87
Now consider all of the terms in (2.70) that are linear in xa
xT
a{Λaaµa−Λab(xb−µb)} (2.74)
where we have used ΛT
ba=Λab. From our discussion of the general form (2.71),
the coefﬁcient of xain this expression must equal Σ−1
a|bµa|band hence
µa|b=Σa|b{Λaaµa−Λab(xb−µb)}
=µa−Λ−1
aaΛab(xb−µb) (2.75)
where we have made use of (2.73).
The results (2.73) and (2.75) are expressed in terms of the partitioned precision
matrix of the original joint distribution p(xa,xb). We can also express these results
in terms of the corresponding partitioned covariance matrix. To do this, we make use
of the following identity for the inverse of a partitioned matrix Exercise 2.24
(
AB
CD)−1
=(
M −MBD−1
−D−1CM D−1+D−1CMBD−1)
(2.76)
M=(A−BD−1C)−1. (2.77)
The quantity M−1is known as the Schur complement of the matrix on the left-hand
side of (2.76) with respect to the submatrix D. Using the deﬁnition
(
ΣaaΣab
ΣbaΣbb)−1
=(
ΛaaΛab
ΛbaΛbb)
(2.78)
and making use of (2.76), we have

(
AB
CD)−1
=(
M −MBD−1
(2.76)
(
ΣaaΣab
ΣbaΣbb)−1
=(
ΛaaΛab
ΛbaΛbb)
(2.78)
Λaa=(Σaa−ΣabΣ−1
bbΣba)−1(2.79)
Λab=−(Σaa−ΣabΣ−1
bbΣba)−1ΣabΣ−1
bb. (2.80)
From these we obtain the following expressions for the mean and covariance of the
conditional distribution p(xa|xb)
µa|b=µa+ΣabΣ−1
bb(xb−µb) (2.81)
Σa|b=Σaa−ΣabΣ−1
bbΣba. (2.82)
Comparing (2.73) and (2.82), we see that the conditional distribution p(xa|xb)takes
a simpler form when expressed in terms of the partitioned precision matrix than
when it is expressed in terms of the partitioned covariance matrix. Note that the

bb. (2.80)
mean of the conditional distribution p(xa|xb), given by (2.81), is a linear function of
xband that the covariance, given by (2.82), is independent of xa. This represents an
example of a linear-Gaussian model. Section 8.1.4

88 2. PROBABILITY DISTRIBUTIONS
2.3.2 Marginal Gaussian distributions
We have seen that if a joint distribution p(xa,xb)is Gaussian, then the condi-
tional distribution p(xa|xb)will again be Gaussian. Now we turn to a discussion of
the marginal distribution given by
p(xa)=∫
p(xa,xb)dxb (2.83)
which, as we shall see, is also Gaussian. Once again, our strategy for evaluating this
distribution efﬁciently will be to focus on the quadratic form in the exponent of the
joint distribution and thereby to identify the mean and covariance of the marginal
distribution p(xa).
The quadratic form for the joint distribution can be expressed, using the par-
titioned precision matrix, in the form (2.70). Because our goal is to integrate out
xb, this is most easily achieved by ﬁrst considering the terms involving xband then
completing the square in order to facilitate integration. Picking out just those terms
that involve xb,w eh a v e
−1
2xT
bΛbbxb+xT
bm=−1
2(xb−Λ−1
bbm)TΛbb(xb−Λ−1
bbm)+1
2mTΛ−1
bbm(2.84)

−1
2xT
bΛbbxb+xT
bm=−1
2(xb−Λ−1
bbm)+1
2mTΛ−1
bbm(2.84)
m=Λbbµb−Λba(xa−µa). (2.85)
We see that the dependence on xbhas been cast into the standard quadratic form of a
Gaussian distribution corresponding to the ﬁrst term on the right-hand side of (2.84),
plus a term that does not depend on xb(but that does depend on xa). Thus, when
we take the exponential of this quadratic form, we see that the integration over xb
required by (2.83) will take the form
∫
exp{
−1
2(xb−Λ−1
bbm)}
dxb. (2.86)
This integration is easily performed by noting that it is the integral over an unnor-

∫
exp{
−1
2(xb−Λ−1
bbm)}
malized Gaussian, and so the result will be the reciprocal of the normalization co-efﬁcient. We know from the form of the normalized Gaussian given by (2.43), that
this coefﬁcient is independent of the mean and depends only on the determinant of
the covariance matrix. Thus, by completing the square with respect to x
b, we can
integrate out xband the only term remaining from the contributions on the left-hand
side of (2.84) that depends on xais the last term on the right-hand side of (2.84) in

b, we can
whichmis given by (2.85). Combining this term with the remaining terms from

2.3. The Gaussian Distribution 89
(2.70) that depend on xa, we obtain
1
2[Λbbµb−Λba(xa−µa)]TΛ−1
bb[Λbbµb−Λba(xa−µa)]
−1
2xT
aΛaaxa+xT
a(Λaaµa+Λabµb)+c o n s t
=−1
2xT
a(Λaa−ΛabΛ−1
bbΛba)xa
+xT
bbΛba)−1µa+c o n s t (2.87)
where ‘const’ denotes quantities independent of xa. Again, by comparison with
(2.71), we see that the covariance of the marginal distribution of p(xa)is given by
Σa=(Λaa−ΛabΛ−1
bbΛba)−1. (2.88)
Similarly, the mean is given by
Σa(Λaa−ΛabΛ−1
bbΛba)µa=µa (2.89)
where we have used (2.88). The covariance in (2.88) is expressed in terms of the
partitioned precision matrix given by (2.69). We can rewrite this in terms of the
corresponding partitioning of the covariance matrix given by (2.67), as we did forthe conditional distribution. These partitioned matrices are related by
(
ΛaaΛab
ΛbaΛbb)−1
=(
ΣaaΣab
ΣbaΣbb)
(2.90)
Making use of (2.76), we then have
(
Λaa−ΛabΛ−1
bbΛba)−1=Σaa. (2.91)
Thus we obtain the intuitively satisfying result that the marginal distribution p(xa)

(
ΛaaΛab
ΛbaΛbb)−1
=(
ΣaaΣab
ΣbaΣbb)
(2.90)
(
has mean and covariance given by
E[xa]= µa (2.92)
cov[xa]= Σaa. (2.93)
We see that for a marginal distribution, the mean and covariance are most simply ex-
pressed in terms of the partitioned covariance matrix, in contrast to the conditional
distribution for which the partitioned precision matrix gives rise to simpler expres-
sions.
Our results for the marginal and conditional distributions of a partitioned Gaus-
sian are summarized below.
Partitioned Gaussians
Given a joint Gaussian distribution N(x|µ,Σ)withΛ≡Σ−1and
x=(
xa
xb)
,µ=(
µa
µb)
(2.94)

90 2. PROBABILITY DISTRIBUTIONS
xaxb=0.7xb
p(xa,xb)
xap(xa)p(xa|xb=0.7)
0 0.5 10510
Figure 2.9 The plot on the left shows the contours of a Gaussian distribution p(xa,xb)over two variables, and
the plot on the right shows the marginal distribution p(xa)(blue curve) and the conditional distribution p(xa|xb)
forxb=0.7(red curve).
Σ=(
ΣaaΣab
ΣbaΣbb)
,Λ=(
ΛaaΛab
ΛbaΛbb)
. (2.95)
Conditional distribution:
p(xa|xb)= N(x|µa|b,Λ−1
aa) (2.96)
µa|b=µa−Λ−1
aaΛab(xb−µb). (2.97)
Marginal distribution:
p(xa)=N(xa|µa,Σaa). (2.98)
We illustrate the idea of conditional and marginal distributions associated with
a multivariate Gaussian using an example involving two variables in Figure 2.9.
2.3.3 Bayes’ theorem for Gaussian variables
In Sections 2.3.1 and 2.3.2, we considered a Gaussian p(x)in which we parti-
tioned the vector xinto two subvectors x=(xa,xb)and then found expressions for
the conditional distribution p(xa|xb)and the marginal distribution p(xa). We noted

that the mean of the conditional distribution p(xa|xb)was a linear function of xb.
Here we shall suppose that we are given a Gaussian marginal distribution p(x)and a
Gaussian conditional distribution p(y|x)in which p(y|x)has a mean that is a linear
function of x, and a covariance which is independent of x. This is an example of

2.3. The Gaussian Distribution 91
alinear Gaussian model (Roweis and Ghahramani, 1999), which we shall study in
greater generality in Section 8.1.4. We wish to ﬁnd the marginal distribution p(y)
and the conditional distribution p(x|y). This is a problem that will arise frequently
in subsequent chapters, and it will prove convenient to derive the general results here.
We shall take the marginal and conditional distributions to be
p(x)= N(
x|µ,Λ−1)
(2.99)
p(y|x)= N(
y|Ax+b,L−1)
(2.100)
where µ,A, andbare parameters governing the means, and ΛandLare precision
matrices. If xhas dimensionality Mandyhas dimensionality D, then the matrix A
has size D×M.
First we ﬁnd an expression for the joint distribution over xandy. To do this, we
deﬁne
z=(
x
y)
(2.101)
and then consider the log of the joint distribution
lnp(z)=l n p(x)+l n p(y|x)
=−1
2(x−µ)TΛ(x−µ)
−1
2(y−Ax−b)TL(y−Ax−b) + const (2.102)
where ‘const’ denotes terms independent of xandy. As before, we see that this is a

(2.100)
deﬁne
z=(
x
y)
(2.101)
=−1
−1
quadratic function of the components of z, and hence p(z)is Gaussian distribution.
To ﬁnd the precision of this Gaussian, we consider the second order terms in (2.102),which can be written as
−1
2xT(Λ+ATLA)x−1
2yTLy+1
2yTLAx+1
2xTATLy
=−1
2(
x
y)T(
Λ+ATLA−ATL
−LA L)(
x
y)
=−1
2zTRz (2.103)
and so the Gaussian distribution over zhas precision (inverse covariance) matrix
given by
R=(
−LA L)
. (2.104)
The covariance matrix is found by taking the inverse of the precision, which can be

−1
2yTLy+1
2yTLAx+1
2xTATLy
=−1
2(
x
y)T(
−LA L)(
x
y)
=−1
given by
R=(
−LA L)
. (2.104)
done using the matrix inversion formula (2.76) to give Exercise 2.29
cov[z]=R−1=(
Λ−1Λ−1AT
AΛ−1L−1+AΛ−1AT)
. (2.105)

92 2. PROBABILITY DISTRIBUTIONS
Similarly, we can ﬁnd the mean of the Gaussian distribution over zby identify-
ing the linear terms in (2.102), which are given by
xTΛµ−xTATLb+yTLb=(
x
y)T(
Λµ−ATLb
Lb)
. (2.106)
Using our earlier result (2.71) obtained by completing the square over the quadratic
form of a multivariate Gaussian, we ﬁnd that the mean of zis given by
E[z]=R−1(
Λµ−ATLb
Lb)
. (2.107)
Making use of (2.105), we then obtain Exercise 2.30
E[z]=(
µ
Aµ+b)
. (2.108)
Next we ﬁnd an expression for the marginal distribution p(y)in which we have
marginalized over x. Recall that the marginal distribution over a subset of the com-
ponents of a Gaussian random vector takes a particularly simple form when ex-pressed in terms of the partitioned covariance matrix. Speciﬁcally, its mean and Section 2.3
covariance are given by (2.92) and (2.93), respectively. Making use of (2.105) and
(2.108) we see that the mean and covariance of the marginal distribution p(y)are
given by
E[y]= Aµ+b (2.109)
cov[y]= L

given by
cov[y]= L
−1+AΛ−1AT. (2.110)
A special case of this result is when A=I, in which case it reduces to the convolu-
tion of two Gaussians, for which we see that the mean of the convolution is the sum
of the mean of the two Gaussians, and the covariance of the convolution is the sumof their covariances.
Finally, we seek an expression for the conditional p(x|y). Recall that the results
for the conditional distribution are most easily expressed in terms of the partitioned
precision matrix, using (2.73) and (2.75). Applying these results to (2.105) and Section 2.3

(2.108) we see that the conditional distribution p(x|y)has mean and covariance
given by
E[x|y]=( Λ+A
TLA)−1{
ATL(y−b)+Λµ}
(2.111)
cov[x|y]=( Λ+ATLA)−1. (2.112)
The evaluation of this conditional can be seen as an example of Bayes’ theorem.
We can interpret the distribution p(x)as a prior distribution over x. If the variable
yis observed, then the conditional distribution p(x|y)represents the corresponding
posterior distribution over x. Having found the marginal and conditional distribu-

given by
TLA)−1{
(2.111)
tions, we effectively expressed the joint distribution p(z)=p(x)p(y|x)in the form
p(x|y)p(y). These results are summarized below.

2.3. The Gaussian Distribution 93
Marginal and Conditional Gaussians
Given a marginal Gaussian distribution for xand a conditional Gaussian distri-
bution for ygivenxin the form
p(x)= N(x|µ,Λ−1) (2.113)
p(y|x)= N(y|Ax+b,L−1) (2.114)
the marginal distribution of yand the conditional distribution of xgivenyare
given by
p(y)= N(y|Aµ+b,L−1+AΛ−1AT) (2.115)
p(x|y)= N(x|Σ{ATL(y−b)+Λµ},Σ) (2.116)
where
Σ=(Λ+ATLA)−1. (2.117)
2.3.4 Maximum likelihood for the Gaussian
Given a data set X=(x1,...,xN)Tin which the observations {xn}are as-
sumed to be drawn independently from a multivariate Gaussian distribution, we canestimate the parameters of the distribution by maximum likelihood. The log likeli-
hood function is given by
lnp(X|µ,Σ)=−ND
2ln(2π)−N
2ln|Σ|−1
2N∑
n=1(xn−µ)TΣ−1(xn−µ).(2.118)
By simple rearrangement, we see that the likelihood function depends on the data set
only through the two quantities
N∑
n=1xn,N∑
n=1xnxT
n. (2.119)
These are known as the sufﬁcient statistics for the Gaussian distribution. Using

2ln(2π)−N
2ln|Σ|−1
2N∑
N∑
n=1xn,N∑
n=1xnxT
n. (2.119)
(C.19), the derivative of the log likelihood with respect to µis given by Appendix C
∂
∂µlnp(X|µ,Σ)=N∑
n=1Σ−1(xn−µ) (2.120)
and setting this derivative to zero, we obtain the solution for the maximum likelihood
estimate of the mean given by
µML=1
NN∑
n=1xn (2.121)

94 2. PROBABILITY DISTRIBUTIONS
which is the mean of the observed set of data points. The maximization of (2.118)
with respect to Σis rather more involved. The simplest approach is to ignore the
symmetry constraint and show that the resulting solution is symmetric as required. Exercise 2.34
Alternative derivations of this result, which impose the symmetry and positive deﬁ-
niteness constraints explicitly, can be found in Magnus and Neudecker (1999). Theresult is as expected and takes the form
Σ
ML=1
NN∑
n=1(xn−µML)(xn−µML)T(2.122)
which involves µMLbecause this is the result of a joint maximization with respect
toµandΣ. Note that the solution (2.121) for µMLdoes not depend on ΣML, and so
we can ﬁrst evaluate µMLand then use this to evaluate ΣML.
If we evaluate the expectations of the maximum likelihood solutions under the
true distribution, we obtain the following results Exercise 2.35
E[µML]= µ (2.123)
E[ΣML]=N−1
NΣ. (2.124)
We see that the expectation of the maximum likelihood estimate for the mean is equal

to the true mean. However, the maximum likelihood estimate for the covariance hasan expectation that is less than the true value, and hence it is biased. We can correct
this bias by deﬁning a different estimator˜Σgiven by
˜Σ=1
N−1N∑
n=1(xn−µML)(xn−µML)T. (2.125)
Clearly from (2.122) and (2.124), the expectation of ˜Σis equal to Σ.
2.3.5 Sequential estimation
Our discussion of the maximum likelihood solution for the parameters of a Gaus-
sian distribution provides a convenient opportunity to give a more general discussion

˜Σ=1
N−1N∑
of the topic of sequential estimation for maximum likelihood. Sequential methods
allow data points to be processed one at a time and then discarded and are important
for on-line applications, and also where large data sets are involved so that batch
processing of all data points at once is infeasible.
Consider the result (2.121) for the maximum likelihood estimator of the mean
µML, which we will denote by µ(N)
MLwhen it is based on Nobservations. If we

2.3. The Gaussian Distribution 95
Figure 2.10 A schematic illustration of two correlated ran-
dom variables zandθ, together with the
regression function f(θ)given by the con-
ditional expectation E[z|θ]. The Robbins-
Monro algorithm provides a general sequen-
tial procedure for ﬁnding the root θ⋆of such
functions. θz
θ⋆f(θ)
dissect out the contribution from the ﬁnal data point xN, we obtain
µ(N)
ML=1
NN∑
n=1xn
=1
NxN+1
NN−1∑
n=1xn
=1
NxN+N−1
Nµ(N−1)
ML
=µ(N−1)
ML +1
N(xN−µ(N−1)
ML). (2.126)
This result has a nice interpretation, as follows. After observing N−1data points
we have estimated µbyµ(N−1)
ML . We now observe data point xN, and we obtain our
revised estimate µ(N)
MLby moving the old estimate a small amount, proportional to
1/N, in the direction of the ‘error signal’ (xN−µ(N−1)
ML). Note that, as Nincreases,
so the contribution from successive data points gets smaller.
The result (2.126) will clearly give the same answer as the batch result (2.121)

ML +1
because the two formulae are equivalent. However, we will not always be able to de-
rive a sequential algorithm by this route, and so we seek a more general formulation
of sequential learning, which leads us to the Robbins-Monro algorithm. Consider a
pair of random variables θandzgoverned by a joint distribution p(z,θ). The con-
ditional expectation of zgivenθdeﬁnes a deterministic function f(θ)that is given
by
f(θ)≡E[z|θ]=∫
zp(z|θ)dz (2.127)

by
and is illustrated schematically in Figure 2.10. Functions deﬁned in this way are
called regression functions .
Our goal is to ﬁnd the root θ⋆at which f(θ⋆)=0 . If we had a large data set
of observations of zandθ, then we could model the regression function directly and
then obtain an estimate of its root. Suppose, however, that we observe values of
zone at a time and we wish to ﬁnd a corresponding sequential estimation scheme
forθ⋆. The following general procedure for solving such problems was given by

96 2. PROBABILITY DISTRIBUTIONS
Robbins and Monro (1951). We shall assume that the conditional variance of zis
ﬁnite so that
E[
(z−f)2|θ]
<∞ (2.128)
and we shall also, without loss of generality, consider the case where f(θ)>0for
θ>θ⋆andf(θ)<0forθ<θ⋆, as is the case in Figure 2.10. The Robbins-Monro
procedure then deﬁnes a sequence of successive estimates of the root θ⋆given by
θ(N)=θ(N−1)+aN−1z(θ(N−1)) (2.129)
where z(θ(N))is an observed value of zwhenθtakes the value θ(N). The coefﬁcients
{aN}represent a sequence of positive numbers that satisfy the conditions
lim
N→∞aN=0 (2.130)
∞∑
N=1aN=∞ (2.131)
∞∑
N=1a2
N<∞. (2.132)
It can then be shown (Robbins and Monro, 1951; Fukunaga, 1990) that the sequence
of estimates given by (2.129) does indeed converge to the root with probability one.
Note that the ﬁrst condition (2.130) ensures that the successive corrections decrease

lim
∞∑
∞∑
N=1a2
in magnitude so that the process can converge to a limiting value. The second con-dition (2.131) is required to ensure that the algorithm does not converge short of the
root, and the third condition (2.132) is needed to ensure that the accumulated noise
has ﬁnite variance and hence does not spoil convergence.
Now let us consider how a general maximum likelihood problem can be solved
sequentially using the Robbins-Monro algorithm. By deﬁnition, the maximum like-lihood solution θ

MLis a stationary point of the log likelihood function and hence
satisﬁes
∂
∂θ{
1
NN∑
n=1lnp(xn|θ)}⏐⏐⏐⏐⏐
θML=0. (2.133)
Exchanging the derivative and the summation, and taking the limit N→∞ we have
lim
N→∞1
NN∑
n=1∂
∂θlnp(xn|θ)= Ex[∂
∂θlnp(x|θ)]
(2.134)
and so we see that ﬁnding the maximum likelihood solution corresponds to ﬁnd-
ing the root of a regression function. We can therefore apply the Robbins-Monroprocedure, which now takes the form
θ
(N)=θ(N−1)+aN−1∂
∂θ(N−1)lnp(xN|θ(N−1)). (2.135)

2.3. The Gaussian Distribution 97
Figure 2.11 In the case of a Gaussian distribution, with θ
corresponding to the mean µ, the regression
function illustrated in Figure 2.10 takes the form
of a straight line, as shown in red. In this
case, the random variable zcorresponds to the
derivative of the log likelihood function and is
given by (x−µML)/σ2, and its expectation that
deﬁnes the regression function is a straight line
given by (µ−µML)/σ2. The root of the regres-
sion function corresponds to the maximum like-
lihood estimator µML.µz
p(z|µ)
µML
As a speciﬁc example, we consider once again the sequential estimation of the
mean of a Gaussian distribution, in which case the parameter θ(N)is the estimate
µ(N)
MLof the mean of the Gaussian, and the random variable zis given by
z=∂
∂µMLlnp(x|µML,σ2)=1
σ2(x−µML). (2.136)
Thus the distribution of zis Gaussian with mean µ−µML, as illustrated in Fig-
ure 2.11. Substituting (2.136) into (2.135), we obtain the univariate form of (2.126),

p(z|µ)
µML
µ(N)
z=∂
provided we choose the coefﬁcients aNto have the form aN=σ2/N. Note that
although we have focussed on the case of a single variable, the same technique,
together with the same restrictions (2.130)–(2.132) on the coefﬁcients aN, apply
equally to the multivariate case (Blum, 1965).
2.3.6 Bayesian inference for the Gaussian
The maximum likelihood framework gave point estimates for the parameters µ
andΣ. Now we develop a Bayesian treatment by introducing prior distributions

over these parameters. Let us begin with a simple example in which we consider a
single Gaussian random variable x. We shall suppose that the variance σ2is known,
and we consider the task of inferring the mean µgiven a set of Nobservations
X={x1,...,x N}. The likelihood function, that is the probability of the observed
data given µ, viewed as a function of µ,i sg i v e nb y
p(X|µ)=N∏
n=1p(xn|µ)=1
(2πσ2)N/2exp{
−1
2σ2N∑
n=1(xn−µ)2}
.(2.137)
Again we emphasize that the likelihood function p(X|µ)is not a probability distri-

p(X|µ)=N∏
−1
2σ2N∑
.(2.137)
bution over µand is not normalized.
We see that the likelihood function takes the form of the exponential of a quad-
ratic form in µ. Thus if we choose a prior p(µ)given by a Gaussian, it will be a

98 2. PROBABILITY DISTRIBUTIONS
conjugate distribution for this likelihood function because the corresponding poste-
rior will be a product of two exponentials of quadratic functions of µand hence will
also be Gaussian. We therefore take our prior distribution to be
p(µ)=N(
µ|µ0,σ2
0)
(2.138)
and the posterior distribution is given by
p(µ|X)∝p(X|µ)p(µ). (2.139)
Simple manipulation involving completing the square in the exponent shows that the Exercise 2.38
posterior distribution is given by
p(µ|X)=N(
µ|µN,σ2
N)
(2.140)
where
µN=σ2
Nσ2
0+σ2µ0+Nσ2
0
Nσ2
0+σ2µML (2.141)
1
σ2
N=1
σ2
0+N
σ2(2.142)
in which µMLis the maximum likelihood solution for µgiven by the sample mean
µML=1
NN∑
n=1xn. (2.143)
It is worth spending a moment studying the form of the posterior mean and
variance. First of all, we note that the mean of the posterior distribution given by
(2.141) is a compromise between the prior mean µ0and the maximum likelihood
solution µML. If the number of observed data points N=0, then (2.141) reduces

µ|µN,σ2
N)
(2.140)
where
µN=σ2
Nσ2
0
Nσ2
1
σ2
N=1
σ2
0+N
σ2(2.142)
µML=1
NN∑
to the prior mean as expected. For N→∞ , the posterior mean is given by the
maximum likelihood solution. Similarly, consider the result (2.142) for the variance
of the posterior distribution. We see that this is most naturally expressed in termsof the inverse variance, which is called the precision. Furthermore, the precisions
are additive, so that the precision of the posterior is given by the precision of the

prior plus one contribution of the data precision from each of the observed datapoints. As we increase the number of observed data points, the precision steadily
increases, corresponding to a posterior distribution with steadily decreasing variance.
With no observed data points, we have the prior variance, whereas if the number of
data points N→∞ , the variance σ
2
Ngoes to zero and the posterior distribution

2
becomes inﬁnitely peaked around the maximum likelihood solution. We thereforesee that the maximum likelihood result of a point estimate for µgiven by (2.143) is
recovered precisely from the Bayesian formalism in the limit of an inﬁnite number
of observations. Note also that for ﬁnite N, if we take the limit σ
2
0→∞ in which the
prior has inﬁnite variance then the posterior mean (2.141) reduces to the maximum
likelihood result, while from (2.142) the posterior variance is given by σ2
N=σ2/N.

2.3. The Gaussian Distribution 99
Figure 2.12 Illustration of Bayesian inference for
the mean µof a Gaussian distri-
bution, in which the variance is as-
sumed to be known. The curves
show the prior distribution over µ
(the curve labelled N=0), which
in this case is itself Gaussian, along
with the posterior distribution given
by (2.140) for increasing numbers N
of data points. The data points are
generated from a Gaussian of mean
0.8and variance 0.1, and the prior is
chosen to have mean 0. In both the
prior and the likelihood function, the
variance is set to the true value.N=0N=1N=2N=1 0
−1 0 105
We illustrate our analysis of Bayesian inference for the mean of a Gaussian
distribution in Figure 2.12. The generalization of this result to the case of a D-
dimensional Gaussian random variable xwith known covariance and unknown mean
is straightforward. Exercise 2.40
We have already seen how the maximum likelihood expression for the mean of

−1 0 105
a Gaussian can be re-cast as a sequential update formula in which the mean after Section 2.3.5
observing Ndata points was expressed in terms of the mean after observing N−1
data points together with the contribution from data point xN. In fact, the Bayesian
paradigm leads very naturally to a sequential view of the inference problem. To see
this in the context of the inference of the mean of a Gaussian, we write the posterior
distribution with the contribution from the ﬁnal data point xNseparated out so that
p(µ|D)∝[
p(µ)N−1∏
n=1p(xn|µ)]

p(µ|D)∝[
p(µ)N−1∏
p(xN|µ). (2.144)
The term in square brackets is (up to a normalization coefﬁcient) just the posterior
distribution after observing N−1data points. We see that this can be viewed as
a prior distribution, which is combined using Bayes’ theorem with the likelihood
function associated with data point xNto arrive at the posterior distribution after
observing Ndata points. This sequential view of Bayesian inference is very general
and applies to any problem in which the observed data are assumed to be independent
and identically distributed.

So far, we have assumed that the variance of the Gaussian distribution over the
data is known and our goal is to infer the mean. Now let us suppose that the mean
is known and we wish to infer the variance. Again, our calculations will be greatly
simpliﬁed if we choose a conjugate form for the prior distribution. It turns out to be
most convenient to work with the precision λ≡1/σ2. The likelihood function for λ
p(X|λ)=N∏
n=1N(xn|µ, λ−1)∝λN/2exp{
−λ
2N∑
. (2.145)

100 2. PROBABILITY DISTRIBUTIONS
λa=0.1
b=0.1
0 1 2012
λa=1
b=1
0 1 2012
λa=4
b=6
0 1 2012
Figure 2.13 Plot of the gamma distribution Gam( λ|a, b)deﬁned by (2.146) for various values of the parameters
aandb.
The corresponding conjugate prior should therefore be proportional to the product
of a power of λand the exponential of a linear function of λ. This corresponds to
thegamma distribution which is deﬁned by
Gam( λ|a, b)=1
Γ(a)baλa−1exp(−bλ). (2.146)
HereΓ(a)is the gamma function that is deﬁned by (1.141) and that ensures that
(2.146) is correctly normalized. The gamma distribution has a ﬁnite integral if a>0, Exercise 2.41
and the distribution itself is ﬁnite if a⩾1. It is plotted, for various values of aand
b, in Figure 2.13. The mean and variance of the gamma distribution are given by Exercise 2.42
E[λ]=a
b(2.147)
var[λ]=a
b2. (2.148)
Consider a prior distribution Gam( λ|a0,b0). If we multiply by the likelihood
function (2.145), then we obtain a posterior distribution
p(λ|X)∝λa0−1λN/2exp{
−b0λ−λ
2N∑

E[λ]=a
b(2.147)
var[λ]=a
−b0λ−λ
2N∑
(2.149)
which we recognize as a gamma distribution of the form Gam( λ|aN,bN)where
aN=a0+N
2(2.150)
bN=b0+1
2N∑
n=1(xn−µ)2=b0+N
2σ2
ML (2.151)
where σ2
MLis the maximum likelihood estimator of the variance. Note that in (2.149)
there is no need to keep track of the normalization constants in the prior and the
likelihood function because, if required, the correct coefﬁcient can be found at the
end using the normalized form (2.146) for the gamma distribution.

2.3. The Gaussian Distribution 101
From (2.150), we see that the effect of observing Ndata points is to increase
the value of the coefﬁcient abyN/2. Thus we can interpret the parameter a0in
the prior in terms of 2a0‘effective’ prior observations. Similarly, from (2.151) we
see that the Ndata points contribute Nσ2
ML/2to the parameter b, where σ2
MLis
the variance, and so we can interpret the parameter b0in the prior as arising from
the2a0‘effective’ prior observations having variance 2b0/(2a0)=b0/a0. Recall
that we made an analogous interpretation for the Dirichlet prior. These distributions Section 2.2
are examples of the exponential family, and we shall see that the interpretation ofa conjugate prior in terms of effective ﬁctitious data points is a general one for the
exponential family of distributions.
Instead of working with the precision, we can consider the variance itself. The
conjugate prior in this case is called the inverse gamma distribution, although we

shall not discuss this further because we will ﬁnd it more convenient to work withthe precision.
Now suppose that both the mean and the precision are unknown. To ﬁnd a
conjugate prior, we consider the dependence of the likelihood function on µandλ
p(X|µ, λ)=
N∏
n=1(λ
2π)1/2
exp{
−λ
2(xn−µ)2}
∝[
λ1/2exp(
−λµ2
2)]N
exp{
λµN∑
n=1xn−λ
2N∑
n=1x2
n}
.(2.152)
We now wish to identify a prior distribution p(µ, λ)that has the same functional
dependence on µandλas the likelihood function and that should therefore take the
form
p(µ, λ)∝[
λ1/2exp(
−λµ2
2)]β

p(X|µ, λ)=
N∏
n=1(λ
2π)1/2
exp{
−λ
2(xn−µ)2}
∝[
λ1/2exp(
−λµ2
2)]N
exp{
λµN∑
n=1xn−λ
2N∑
n=1x2
n}
.(2.152)
form
p(µ, λ)∝[
λ1/2exp(
−λµ2
2)]β
exp{cλµ−dλ}
=e x p{
−βλ
2(µ−c/β)2}
λβ/2exp{
−(
d−c2
2β)
λ}
(2.153)
where c,d, andβare constants. Since we can always write p(µ, λ)=p(µ|λ)p(λ),
we can ﬁnd p(µ|λ)andp(λ)by inspection. In particular, we see that p(µ|λ)is a
Gaussian whose precision is a linear function of λand that p(λ)is a gamma distri-
bution, so that the normalized prior takes the form
p(µ, λ)=N(µ|µ0,(βλ)−1)Gam( λ|a, b) (2.154)
where we have deﬁned new constants given by µ0=c/β,a=1 + β/2,b=
d−c2/2β. The distribution (2.154) is called the normal-gamma orGaussian-gamma

λβ/2exp{
−(
d−c2
2β)
λ}
(2.153)
distribution and is plotted in Figure 2.14. Note that this is not simply the product
of an independent Gaussian prior over µand a gamma prior over λ, because the
precision of µis a linear function of λ. Even if we chose a prior in which µandλ
were independent, the posterior distribution would exhibit a coupling between the
precision of µand the value of λ.

102 2. PROBABILITY DISTRIBUTIONS
Figure 2.14 Contour plot of the normal-gamma
distribution (2.154) for parameter
values µ0=0,β=2,a=5and
b=6.
µλ
−2 0 2012
In the case of the multivariate Gaussian distribution N(
x|µ,Λ−1)
for aD-
dimensional variable x, the conjugate prior distribution for the mean µ, assuming
the precision is known, is again a Gaussian. For known mean and unknown precision
matrixΛ, the conjugate prior is the Wishart distribution given by Exercise 2.45
W(Λ|W,ν)=B|Λ|(ν−D−1)/2exp(
−1
2Tr(W−1Λ))
(2.155)
where νis called the number of degrees of freedom of the distribution, Wis aD×D
scale matrix, and Tr (·)denotes the trace. The normalization constant Bis given by
B(W,ν)=|W|−ν/2(
2νD/2πD(D−1)/4D∏
i=1Γ(ν+1−i
2))−1
. (2.156)
Again, it is also possible to deﬁne a conjugate prior over the covariance matrix itself,
rather than over the precision matrix, which leads to the inverse Wishart distribu-
tion, although we shall not discuss this further. If both the mean and the precision

−1
(2.155)
2))−1
. (2.156)
are unknown, then, following a similar line of reasoning to the univariate case, the
conjugate prior is given by
p(µ,Λ|µ0,β,W,ν)=N(µ|µ0,(βΛ)−1)W(Λ|W,ν) (2.157)
which is known as the normal-Wishart orGaussian-Wishart distribution.
2.3.7 Student’s t-distribution
We have seen that the conjugate prior for the precision of a Gaussian is given
by a gamma distribution. If we have a univariate Gaussian N(x|µ, τ−1)together Section 2.3.6
with a Gamma prior Gam( τ|a, b)and we integrate out the precision, we obtain the

marginal distribution of xin the form Exercise 2.46

2.3. The Gaussian Distribution 103
Figure 2.15 Plot of Student’s t-distribution (2.159)
forµ=0andλ=1for various values
ofν. The limit ν→∞ corresponds
to a Gaussian distribution with mean
µand precision λ.ν→∞
ν=1.0
ν=0.1
−5 0 500.10.20.30.40.5
p(x|µ, a, b)=∫∞
0N(x|µ, τ−1)Gam( τ|a, b)dτ (2.158)
=∫∞
0bae(−bτ)τa−1
Γ(a)(τ
2π)1/2
exp{
−τ
2(x−µ)2}
dτ
=ba
Γ(a)(1
2π)1/2[
b+(x−µ)2
2]−a−1/2
Γ(a+1/2)
where we have made the change of variable z=τ[b+(x−µ)2/2]. By convention
we deﬁne new parameters given by ν=2aandλ=a/b, in terms of which the
distribution p(x|µ, a, b)takes the form
St(x|µ, λ, ν )=Γ(ν/2+1/2)
Γ(ν/2)(λ
πν)1/2[
1+λ(x−µ)2
ν]−ν/2−1/2
(2.159)
which is known as Student’s t-distribution . The parameter λis sometimes called the
precision of the t-distribution, even though it is not in general equal to the inverse
of the variance. The parameter νis called the degrees of freedom , and its effect is
illustrated in Figure 2.15. For the particular case of ν=1, the t-distribution reduces

Γ(ν/2)(λ
πν)1/2[
1+λ(x−µ)2
(2.159)
to the Cauchy distribution, while in the limit ν→∞ the t-distribution St(x|µ, λ, ν )
becomes a Gaussian N(x|µ, λ−1)with mean µand precision λ. Exercise 2.47
From (2.158), we see that Student’s t-distribution is obtained by adding up an
inﬁnite number of Gaussian distributions having the same mean but different preci-
sions. This can be interpreted as an inﬁnite mixture of Gaussians (Gaussian mixtures
will be discussed in detail in Section 2.3.9. The result is a distribution that in gen-

eral has longer ‘tails’ than a Gaussian, as was seen in Figure 2.15. This gives the t-
distribution an important property called robustness , which means that it is much less
sensitive than the Gaussian to the presence of a few data points which are outliers .
The robustness of the t-distribution is illustrated in Figure 2.16, which compares the
maximum likelihood solutions for a Gaussian and a t-distribution. Note that the max-
imum likelihood solution for the t-distribution can be found using the expectation-

maximization (EM) algorithm. Here we see that the effect of a small number of Exercise 12.24

104 2. PROBABILITY DISTRIBUTIONS
(b)−5 0 5 1000.10.20.30.40.5
Figure 2.16 Illustration of the robustness of Student’s t-distribution compared to a Gaussian. (a) Histogram
distribution of 30 data points drawn from a Gaussian distribution, together with the maximum likelihood ﬁt ob-
tained from a t-distribution (red curve) and a Gaussian (green curve, largely hidden by the red curve). Because
the t-distribution contains the Gaussian as a special case it gives almost the same solution as the Gaussian.
(b) The same data set but with three additional outlying data points showing how the Gaussian (green curve) is
strongly distorted by the outliers, whereas the t-distribution (red curve) is relatively unaffected.
outliers is much less signiﬁcant for the t-distribution than for the Gaussian. Outliers
can arise in practical applications either because the process that generates the data
corresponds to a distribution having a heavy tail or simply through mislabelled data.

Robustness is also an important property for regression problems. Unsurprisingly,
the least squares approach to regression does not exhibit robustness, because it cor-
responds to maximum likelihood under a (conditional) Gaussian distribution. By
basing a regression model on a heavy-tailed distribution such as a t-distribution, we
obtain a more robust model.
If we go back to (2.158) and substitute the alternative parameters ν=2a,λ=
a/b, andη=τb/a , we see that the t-distribution can be written in the form
St(x|µ, λ, ν )=∫∞
0N(
x|µ,(ηλ)−1)

0N(
Gam( η|ν/2,ν/2) dη. (2.160)
We can then generalize this to a multivariate Gaussian N(x|µ,Λ)to obtain the cor-
responding multivariate Student’s t-distribution in the form
St(x|µ,Λ,ν)=∫∞
0N(x|µ,(ηΛ)−1)Gam( η|ν/2,ν/2) dη. (2.161)
Using the same technique as for the univariate case, we can evaluate this integral to
give Exercise 2.48

2.3. The Gaussian Distribution 105
St(x|µ,Λ,ν)=Γ(D/2+ν/2)
Γ(ν/2)|Λ|1/2
(πν)D/2[
1+∆2
ν]−D/2−ν/2
(2.162)
where Dis the dimensionality of x, and∆2is the squared Mahalanobis distance
deﬁned by
∆2=(x−µ)TΛ(x−µ). (2.163)
This is the multivariate form of Student’s t-distribution and satisﬁes the following
properties Exercise 2.49
E[x]= µ, ifν>1 (2.164)
cov[x]=ν
(ν−2)Λ−1, ifν>2 (2.165)
mode[x]= µ (2.166)
with corresponding results for the univariate case.
2.3.8 Periodic variables
Although Gaussian distributions are of great practical signiﬁcance, both in their
own right and as building blocks for more complex probabilistic models, there are
situations in which they are inappropriate as density models for continuous vari-
ables. One important case, which arises in practical applications, is that of periodicvariables.
An example of a periodic variable would be the wind direction at a particular

geographical location. We might, for instance, measure values of wind direction on anumber of days and wish to summarize this using a parametric distribution. Another
example is calendar time, where we may be interested in modelling quantities that
are believed to be periodic over 24 hours or over an annual cycle. Such quantities
can conveniently be represented using an angular (polar) coordinate 0⩽θ<2π.
We might be tempted to treat periodic variables by choosing some direction

as the origin and then applying a conventional distribution such as the Gaussian.
Such an approach, however, would give results that were strongly dependent on the
arbitrary choice of origin. Suppose, for instance, that we have two observations atθ
1=1◦andθ2= 359◦, and we model them using a standard univariate Gaussian
distribution. If we choose the origin at 0◦, then the sample mean of this data set
will be 180◦with standard deviation 179◦, whereas if we choose the origin at 180◦,

then the mean will be 0◦and the standard deviation will be 1◦. We clearly need to
develop a special approach for the treatment of periodic variables.
Let us consider the problem of evaluating the mean of a set of observations
D={θ1,...,θ N}of a periodic variable. From now on, we shall assume that θis
measured in radians. We have already seen that the simple average (θ1+···+θN)/N

will be strongly coordinate dependent. To ﬁnd an invariant measure of the mean, wenote that the observations can be viewed as points on the unit circle and can therefore
be described instead by two-dimensional unit vectors x
1,...,xNwhere ∥xn∥=1
forn=1,...,N , as illustrated in Figure 2.17. We can average the vectors {xn}

106 2. PROBABILITY DISTRIBUTIONS
Figure 2.17 Illustration of the representation of val-
uesθnof a periodic variable as two-
dimensional vectors xnliving on the unit
circle. Also shown is the average xof
those vectors.
x1x2
x1x2x3x4
¯x
¯r
¯θ
instead to give
x=1
NN∑
n=1xn (2.167)
and then ﬁnd the corresponding angle θof this average. Clearly, this deﬁnition will
ensure that the location of the mean is independent of the origin of the angular coor-
dinate. Note that xwill typically lie inside the unit circle. The Cartesian coordinates
of the observations are given by xn=( c o s θn,sinθn), and we can write the Carte-
sian coordinates of the sample mean in the form x=(rcosθ,rsinθ). Substituting
into (2.167) and equating the x1andx2components then gives
rcosθ=1
NN∑
n=1cosθn, rsinθ=1
NN∑
n=1sinθn. (2.168)
Taking the ratio, and using the identity tanθ=s i n θ/cosθ, we can solve for θto
give
θ= tan−1{∑
nsinθn∑
ncosθn}
. (2.169)
Shortly, we shall see how this result arises naturally as the maximum likelihood

rcosθ=1
NN∑
NN∑
give
θ= tan−1{∑
nsinθn∑
ncosθn}
. (2.169)
estimator for an appropriately deﬁned distribution over a periodic variable.
We now consider a periodic generalization of the Gaussian called the von Mises
distribution. Here we shall limit our attention to univariate distributions, although
periodic distributions can also be found over hyperspheres of arbitrary dimension.
For an extensive discussion of periodic distributions, see Mardia and Jupp (2000).
By convention, we will consider distributions p(θ)that have period 2π.A n y

probability density p(θ)deﬁned over θmust not only be nonnegative and integrate

2.3. The Gaussian Distribution 107
Figure 2.18 The von Mises distribution can be derived by considering
a two-dimensional Gaussian of the form (2.173), whose
density contours are shown in blue and conditioning on
the unit circle shown in red.
x1x2
p(x)
r=1
to one, but it must also be periodic. Thus p(θ)must satisfy the three conditions
p(θ)⩾0 (2.170)∫2π
0p(θ)dθ=1 (2.171)
p(θ+2π)= p(θ). (2.172)
From (2.172), it follows that p(θ+M2π)=p(θ)for any integer M.
We can easily obtain a Gaussian-like distribution that satisﬁes these three prop-
erties as follows. Consider a Gaussian distribution over two variables x=(x1,x2)
having mean µ=(µ1,µ2)and a covariance matrix Σ=σ2IwhereIis the2×2
identity matrix, so that
2πσ2exp{
−(x1−µ1)2+(x2−µ2)2
2σ2}
. (2.173)
The contours of constant p(x)are circles, as illustrated in Figure 2.18. Now suppose
we consider the value of this distribution along a circle of ﬁxed radius. Then by con-

2πσ2exp{
2σ2}
. (2.173)
struction this distribution will be periodic, although it will not be normalized. We can
determine the form of this distribution by transforming from Cartesian coordinates
(x1,x2)to polar coordinates (r, θ)so that
x1=rcosθ, x 2=rsinθ. (2.174)
We also map the mean µinto polar coordinates by writing
µ1=r0cosθ0,µ 2=r0sinθ0. (2.175)
Next we substitute these transformations into the two-dimensional Gaussian distribu-
tion (2.173), and then condition on the unit circle r=1, noting that we are interested

only in the dependence on θ. Focussing on the exponent in the Gaussian distribution
we have
−1
2σ2{
(rcosθ−r0cosθ0)2+(rsinθ−r0sinθ0)2}
=−1
2σ2{
1+r2
0−2r0cosθcosθ0−2r0sinθsinθ0}
=r0
σ2cos(θ−θ0)+c o n s t (2.176)

108 2. PROBABILITY DISTRIBUTIONS
m=5 ,θ0=π/4
m=1 ,θ0=3π/4
2π0π/4
3π/4
Figure 2.19 The von Mises distribution plotted for two different parameter values, shown as a Cartesian plot
on the left and as the corresponding polar plot on the right.
where ‘const’ denotes terms independent of θ, and we have made use of the following
trigonometrical identities Exercise 2.51
cos2A+s i n2A=1 (2.177)
cosAcosB+s i nAsinB=c o s ( A−B). (2.178)
If we now deﬁne m=r0/σ2, we obtain our ﬁnal expression for the distribution of
p(θ)along the unit circle r=1in the form
p(θ|θ0,m)=1
2πI0(m)exp{mcos(θ−θ0)} (2.179)
which is called the von Mises distribution, or the circular normal . Here the param-
eterθ0corresponds to the mean of the distribution, while m, which is known as
theconcentration parameter, is analogous to the inverse variance (precision) for the
Gaussian. The normalization coefﬁcient in (2.179) is expressed in terms of I0(m),

which is the zeroth-order Bessel function of the ﬁrst kind (Abramowitz and Stegun,
1965) and is deﬁned by
I0(m)=1
2π∫2π
0exp{mcosθ}dθ. (2.180)
For large m, the distribution becomes approximately Gaussian. The von Mises dis- Exercise 2.52
tribution is plotted in Figure 2.19, and the function I0(m)is plotted in Figure 2.20.
Now consider the maximum likelihood estimators for the parameters θ0andm
for the von Mises distribution. The log likelihood function is given by
lnp(D|θ0,m)=−Nln(2π)−NlnI0(m)+mN∑
n=1cos(θn−θ0). (2.181)

2.3. The Gaussian Distribution 109
I0(m)
m0 5 100100020003000
A(m)
m0 5 1000.51
Figure 2.20 Plot of the Bessel function I0(m)deﬁned by (2.180), together with the function A(m)deﬁned by
(2.186).
Setting the derivative with respect to θ0equal to zero gives
N∑
n=1sin(θn−θ0)=0. (2.182)
To solve for θ0, we make use of the trigonometric identity
sin(A−B)=c o s BsinA−cosAsinB (2.183)
from which we obtain Exercise 2.53
θML
0= tan−1{∑
nsinθn∑
ncosθn}
(2.184)
which we recognize as the result (2.169) obtained earlier for the mean of the obser-
vations viewed in a two-dimensional Cartesian space.
Similarly, maximizing (2.181) with respect to m, and making use of I′
0(m)=
I1(m)(Abramowitz and Stegun, 1965), we have
A(m)=1
NN∑
n=1cos(θn−θML
0) (2.185)
where we have substituted for the maximum likelihood solution for θML
0(recalling
that we are performing a joint optimization over θandm), and we have deﬁned
A(m)=I1(m)
I0(m). (2.186)
The function A(m)is plotted in Figure 2.20. Making use of the trigonometric iden-

0(m)=
A(m)=1
NN∑
0) (2.185)
tity (2.178), we can write (2.185) in the form
A(mML)=(
1
NN∑
n=1cosθn)
cosθML
0−(
1
NN∑
n=1sinθn)
sinθML
0. (2.187)

110 2. PROBABILITY DISTRIBUTIONS
Figure 2.21 Plots of the ‘old faith-
ful’ data in which the blue curves
show contours of constant proba-
bility density. On the left is a
single Gaussian distribution which
has been ﬁtted to the data us-
ing maximum likelihood. Note that
this distribution fails to capture the
two clumps in the data and indeed
places much of its probability mass
in the central region between the
clumps where the data are relatively
sparse. On the right the distribution
is given by a linear combination of
two Gaussians which has been ﬁtted
to the data by maximum likelihood
using techniques discussed Chap-
ter 9, and which gives a better rep-
resentation of the data.1 2 3 4 5 6406080100
1 2 3 4 5 6406080100
The right-hand side of (2.187) is easily evaluated, and the function A(m)can be
inverted numerically.
For completeness, we mention brieﬂy some alternative techniques for the con-
struction of periodic distributions. The simplest approach is to use a histogram of

observations in which the angular coordinate is divided into ﬁxed bins. This has the
virtue of simplicity and ﬂexibility but also suffers from signiﬁcant limitations, as we
shall see when we discuss histogram methods in more detail in Section 2.5. Another
approach starts, like the von Mises distribution, from a Gaussian distribution over a
Euclidean space but now marginalizes onto the unit circle rather than conditioning
(Mardia and Jupp, 2000). However, this leads to more complex forms of distribution

and will not be discussed further. Finally, any valid distribution over the real axis
(such as a Gaussian) can be turned into a periodic distribution by mapping succes-
sive intervals of width 2πonto the periodic variable (0,2π), which corresponds to
‘wrapping’ the real axis around unit circle. Again, the resulting distribution is more
complex to handle than the von Mises distribution.
One limitation of the von Mises distribution is that it is unimodal. By forming

mixtures of von Mises distributions, we obtain a ﬂexible framework for modelling
periodic variables that can handle multimodality. For an example of a machine learn-
ing application that makes use of von Mises distributions, see Lawrence et al. (2002),
and for extensions to modelling conditional densities for regression problems, see
Bishop and Nabney (1996).
2.3.9 Mixtures of Gaussians
While the Gaussian distribution has some important analytical properties, it suf-

fers from signiﬁcant limitations when it comes to modelling real data sets. Consider
the example shown in Figure 2.21. This is known as the ‘Old Faithful’ data set,
and comprises 272measurements of the eruption of the Old Faithful geyser at Yel-
lowstone National Park in the USA. Each measurement comprises the duration of Appendix A

2.3. The Gaussian Distribution 111
Figure 2.22 Example of a Gaussian mixture distribution
in one dimension showing three Gaussians
(each scaled by a coefﬁcient) in blue and
their sum in red.
xp(x)
the eruption in minutes (horizontal axis) and the time in minutes to the next erup-
tion (vertical axis). We see that the data set forms two dominant clumps, and that
a simple Gaussian distribution is unable to capture this structure, whereas a linear
superposition of two Gaussians gives a better characterization of the data set.
Such superpositions, formed by taking linear combinations of more basic dis-
tributions such as Gaussians, can be formulated as probabilistic models known as
mixture distributions (McLachlan and Basford, 1988; McLachlan and Peel, 2000).
In Figure 2.22 we see that a linear combination of Gaussians can give rise to very
complex densities. By using a sufﬁcient number of Gaussians, and by adjusting their
means and covariances as well as the coefﬁcients in the linear combination, almost

any continuous density can be approximated to arbitrary accuracy.
We therefore consider a superposition of KGaussian densities of the form
p(x)=K∑
k=1πkN(x|µk,Σk) (2.188)
which is called a mixture of Gaussians . Each Gaussian density N(x|µk,Σk)is
called a component of the mixture and has its own mean µkand covariance Σk.
Contour and surface plots for a Gaussian mixture having 3 components are shown in
Figure 2.23.
In this section we shall consider Gaussian components to illustrate the frame-

p(x)=K∑
work of mixture models. More generally, mixture models can comprise linear com-
binations of other distributions. For instance, in Section 9.3.3 we shall consider
mixtures of Bernoulli distributions as an example of a mixture model for discrete
variables. Section 9.3.3
The parameters πkin (2.188) are called mixing coefﬁcients . If we integrate both
sides of (2.188) with respect to x, and note that both p(x)and the individual Gaussian
components are normalized, we obtain
K∑
k=1πk=1. (2.189)

K∑
Also, the requirement that p(x)⩾0, together with N(x|µk,Σk)⩾0, implies
πk⩾0for all k. Combining this with the condition (2.189) we obtain
0⩽πk⩽1. (2.190)

112 2. PROBABILITY DISTRIBUTIONS
0.50.30.2(a)
0 0.5 100.51(b)
Figure 2.23 Illustration of a mixture of 3 Gaussians in a two-dimensional space. (a) Contours of constant
density for each of the mixture components, in which the 3 components are denoted red, blue and green, and
the values of the mixing coefﬁcients are shown below each component. (b) Contours of the marginal probability
density p(x)of the mixture distribution. (c) A surface plot of the distribution p(x).
We therefore see that the mixing coefﬁcients satisfy the requirements to be probabil-
ities.
From the sum and product rules, the marginal density is given by
p(x)=K∑
k=1p(k)p(x|k) (2.191)
which is equivalent to (2.188) in which we can view πk=p(k)as the prior prob-
ability of picking the kthcomponent, and the density N(x|µk,Σk)=p(x|k)as
the probability of xconditioned on k. As we shall see in later chapters, an impor-
tant role is played by the posterior probabilities p(k|x), which are also known as

ities.
p(x)=K∑
responsibilities . From Bayes’ theorem these are given by
γk(x)≡p(k|x)
=p(k)p(x|k)∑
lp(l)p(x|l)
=πkN(x|µk,Σk)∑
lπlN(x|µl,Σl). (2.192)
We shall discuss the probabilistic interpretation of the mixture distribution in greater
detail in Chapter 9.
The form of the Gaussian mixture distribution is governed by the parameters π,
µandΣ, where we have used the notation π≡{π1,...,π K},µ≡{µ1,...,µK}
andΣ≡{Σ1,...ΣK}. One way to set the values of these parameters is to use

maximum likelihood. From (2.188) the log of the likelihood function is given by
lnp(X|π,µ,Σ)=N∑
n=1ln{K∑
k=1πkN(xn|µk,Σk)}
(2.193)

2.4. The Exponential Family 113
whereX={x1,...,xN}. We immediately see that the situation is now much
more complex than with a single Gaussian, due to the presence of the summationoverkinside the logarithm. As a result, the maximum likelihood solution for the
parameters no longer has a closed-form analytical solution. One approach to maxi-
mizing the likelihood function is to use iterative numerical optimization techniques(Fletcher, 1987; Nocedal and Wright, 1999; Bishop and Nabney, 2008). Alterna-
tively we can employ a powerful framework called expectation maximization , which
will be discussed at length in Chapter 9.
2.4. The Exponential Family
The probability distributions that we have studied so far in this chapter (with theexception of the Gaussian mixture) are speciﬁc examples of a broad class of distri-
butions called the exponential family (Duda and Hart, 1973; Bernardo and Smith,

1994). Members of the exponential family have many important properties in com-mon, and it is illuminating to discuss these properties in some generality.
The exponential family of distributions over x, given parameters η, is deﬁned to
be the set of distributions of the form
p(x|η)=h(x)g(η)e x p
{
ηTu(x)}
(2.194)
wherexmay be scalar or vector, and may be discrete or continuous. Here ηare
called the natural parameters of the distribution, and u(x)is some function of x.
The function g(η)can be interpreted as the coefﬁcient that ensures that the distribu-
tion is normalized and therefore satisﬁes
g(η)∫
h(x)e x p{

{
ηTu(x)}
(2.194)
g(η)∫
h(x)e x p{
ηTu(x)}
dx=1 (2.195)
where the integration is replaced by summation if xis a discrete variable.
We begin by taking some examples of the distributions introduced earlier in
the chapter and showing that they are indeed members of the exponential family.
Consider ﬁrst the Bernoulli distribution
p(x|µ)=B e r n ( x|µ)=µx(1−µ)1−x. (2.196)
Expressing the right-hand side as the exponential of the logarithm, we have
p(x|µ) = exp {xlnµ+( 1−x)l n ( 1−µ)}
=( 1 −µ)e x p{
ln(µ
1−µ)
x}
. (2.197)
Comparison with (2.194) allows us to identify
η=l n(µ
1−µ)
(2.198)

114 2. PROBABILITY DISTRIBUTIONS
which we can solve for µto give µ=σ(η), where
σ(η)=1
1+e x p ( −η)(2.199)
is called the logistic sigmoid function. Thus we can write the Bernoulli distribution
using the standard representation (2.194) in the form
p(x|η)=σ(−η) exp(ηx) (2.200)
where we have used 1−σ(η)=σ(−η), which is easily proved from (2.199). Com-
parison with (2.194) shows that
u(x)= x (2.201)
h(x)=1 (2.202)
g(η)= σ(−η). (2.203)
Next consider the multinomial distribution that, for a single observation x, takes
the form
p(x|µ)=M∏
k=1µxk
k=e x p{M∑
k=1xklnµk}
(2.204)
wherex=(x1,...,x N)T. Again, we can write this in the standard representation
(2.194) so that
p(x|η)=e x p ( ηTx) (2.205)
where ηk=l nµk, and we have deﬁned η=(η1,...,η M)T. Again, comparing with
(2.194) we have
u(x)= x (2.206)
h(x)=1 (2.207)
g(η)=1 . (2.208)
Note that the parameters ηkare not independent because the parameters µkare sub-
ject to the constraint
M∑
k=1µk=1 (2.209)

the form
p(x|µ)=M∏
k=1µxk
k=e x p{M∑
(2.204)
M∑
so that, given any M−1of the parameters µk, the value of the remaining parameter
is ﬁxed. In some circumstances, it will be convenient to remove this constraint by
expressing the distribution in terms of only M−1parameters. This can be achieved
by using the relationship (2.209) to eliminate µMby expressing it in terms of the
remaining {µk}where k=1,...,M −1, thereby leaving M−1parameters. Note
that these remaining parameters are still subject to the constraints
0⩽µk⩽1,M−1∑
k=1µk⩽1. (2.210)

2.4. The Exponential Family 115
Making use of the constraint (2.209), the multinomial distribution in this representa-
tion then becomes
exp{M∑
=e x p{M−1∑
k=1xklnµk+(
1−M−1∑
k=1xk)
ln(
1−M−1∑
k=1µk)}
=e x p{M−1∑
k=1xkln(
µk
1−∑M−1
j=1µj)
+l n(
1−M−1∑
k=1µk)}
.(2.211)
We now identify
ln(
µk
1−∑
jµj)
=ηk (2.212)
which we can solve for µkby ﬁrst summing both sides over kand then rearranging
and back-substituting to give
µk=exp(ηk)
1+∑
jexp(ηj). (2.213)
This is called the softmax function, or the normalized exponential . In this represen-
tation, the multinomial distribution therefore takes the form
p(x|η)=(
1+M−1∑
k=1exp(ηk))−1
exp(ηTx). (2.214)
This is the standard form of the exponential family, with parameter vector η=
(η1,...,η M−1)Tin which
u(x)= x (2.215)
h(x)=1 (2.216)
g(η)=(
1+M−1∑
. (2.217)
Finally, let us consider the Gaussian distribution. For the univariate Gaussian,
we have
p(x|µ, σ2)=1
−1
(2.218)
=1
−1
2σ2x2+µ
σ2x−1
2σ2µ2}
(2.219)

116 2. PROBABILITY DISTRIBUTIONS
which, after some simple rearrangement, can be cast in the standard exponential
family form (2.194) with Exercise 2.57
η=(
µ/σ2
−1/2σ2)
(2.220)
u(x)=(
x
x2)
(2.221)
h(x)=( 2 π)−1/2(2.222)
g(η)=( −2η2)1/2exp(η2
1
4η2)
. (2.223)
2.4.1 Maximum likelihood and sufﬁcient statistics
Let us now consider the problem of estimating the parameter vector ηin the gen-
eral exponential family distribution (2.194) using the technique of maximum likeli-
hood. Taking the gradient of both sides of (2.195) with respect to η,w eh a v e
∇g(η)∫
h(x)e x p{
ηTu(x)}
dx
+g(η)∫
h(x)e x p{
ηTu(x)}
u(x)dx=0. (2.224)
Rearranging, and making use again of (2.195) then gives
−1
g(η)∇g(η)=g(η)∫
h(x)e x p{
ηTu(x)}
u(x)dx=E[u(x)] (2.225)
where we have used (2.194). We therefore obtain the result
−∇lng(η)= E[u(x)]. (2.226)
Note that the covariance of u(x)can be expressed in terms of the second derivatives
ofg(η), and similarly for higher order moments. Thus, provided we can normalize a Exercise 2.58

∇g(η)∫
h(x)e x p{
ηTu(x)}
dx
+g(η)∫
h(x)e x p{
ηTu(x)}
−1
h(x)e x p{
ηTu(x)}
distribution from the exponential family, we can always ﬁnd its moments by simple
differentiation.
Now consider a set of independent identically distributed data denoted by X=
{x1,...,xn}, for which the likelihood function is given by
p(X|η)=(N∏
n=1h(xn))
g(η)Nexp{
ηTN∑
n=1u(xn)}
. (2.227)
Setting the gradient of lnp(X|η)with respect to ηto zero, we get the following
condition to be satisﬁed by the maximum likelihood estimator ηML
−∇lng(ηML)=1
NN∑
n=1u(xn) (2.228)

2.4. The Exponential Family 117
which can in principle be solved to obtain ηML. We see that the solution for the
maximum likelihood estimator depends on the data only through∑
nu(xn), which
is therefore called the sufﬁcient statistic of the distribution (2.194). We do not need
to store the entire data set itself but only the value of the sufﬁcient statistic. For
the Bernoulli distribution, for example, the function u(x)is given just by xand
so we need only keep the sum of the data points {xn}, whereas for the Gaussian
u(x)=(x, x2)T, and so we should keep both the sum of {xn}and the sum of {x2
n}.
If we consider the limit N→∞ , then the right-hand side of (2.228) becomes
E[u(x)], and so by comparing with (2.226) we see that in this limit ηMLwill equal
the true value η.
In fact, this sufﬁciency property holds also for Bayesian inference, although
we shall defer discussion of this until Chapter 8 when we have equipped ourselves

n}.
with the tools of graphical models and can thereby gain a deeper insight into theseimportant concepts.
2.4.2 Conjugate priors
We have already encountered the concept of a conjugate prior several times, for
example in the context of the Bernoulli distribution (for which the conjugate prior
is the beta distribution) or the Gaussian (where the conjugate prior for the mean is

a Gaussian, and the conjugate prior for the precision is the Wishart distribution). Ingeneral, for a given probability distribution p(x|η), we can seek a prior p(η)that is
conjugate to the likelihood function, so that the posterior distribution has the same
functional form as the prior. For any member of the exponential family (2.194), there
exists a conjugate prior that can be written in the form
p(η|χ,ν)=f(χ,ν)g(η)
νexp{
νηTχ}
(2.229)
where f(χ,ν)is a normalization coefﬁcient, and g(η)is the same function as ap-

νexp{
νηTχ}
(2.229)
pears in (2.194). To see that this is indeed conjugate, let us multiply the prior (2.229)
by the likelihood function (2.227) to obtain the posterior distribution, up to a nor-
malization coefﬁcient, in the form
p(η|X,χ,ν)∝g(η)ν+Nexp{
ηT(N∑
n=1u(xn)+νχ)}
. (2.230)
This again takes the same functional form as the prior (2.229), conﬁrming conjugacy.
Furthermore, we see that the parameter νcan be interpreted as a effective number of
pseudo-observations in the prior, each of which has a value for the sufﬁcient statistic
u(x)given by χ.
2.4.3 Noninformative priors
In some applications of probabilistic inference, we may have prior knowledge

ηT(N∑
. (2.230)
that can be conveniently expressed through the prior distribution. For example, if
the prior assigns zero probability to some value of variable, then the posterior dis-tribution will necessarily also assign zero probability to that value, irrespective of

118 2. PROBABILITY DISTRIBUTIONS
any subsequent observations of data. In many cases, however, we may have little
idea of what form the distribution should take. We may then seek a form of priordistribution, called a noninformative prior , which is intended to have as little inﬂu-
ence on the posterior distribution as possible (Jeffries, 1946; Box and Tao, 1973;
Bernardo and Smith, 1994). This is sometimes referred to as ‘letting the data speakfor themselves’.
If we have a distribution p(x|λ)governed by a parameter λ, we might be tempted
to propose a prior distribution p(λ) = const as a suitable prior. If λis a discrete
variable with Kstates, this simply amounts to setting the prior probability of each
state to 1/K. In the case of continuous parameters, however, there are two potential
difﬁculties with this approach. The ﬁrst is that, if the domain of λis unbounded,
this prior distribution cannot be correctly normalized because the integral over λ

diverges. Such priors are called improper . In practice, improper priors can often
be used provided the corresponding posterior distribution is proper , i.e., that it can
be correctly normalized. For instance, if we put a uniform prior distribution over
the mean of a Gaussian, then the posterior distribution for the mean, once we haveobserved at least one data point, will be proper.
A second difﬁculty arises from the transformation behaviour of a probability

density under a nonlinear change of variables, given by (1.27). If a function h(λ)
is constant, and we change variables to λ=η
2, thenˆh(η)=h(η2)will also be
constant. However, if we choose the density pλ(λ)to be constant, then the density
ofηwill be given, from (1.27), by
pη(η)=pλ(λ)⏐⏐⏐⏐dλ
dη⏐⏐⏐⏐=pλ(η2)2η∝η (2.231)
and so the density over ηwill not be constant. This issue does not arise when we use
maximum likelihood, because the likelihood function p(x|λ)is a simple function of

λand so we are free to use any convenient parameterization. If, however, we are to
choose a prior distribution that is constant, we must take care to use an appropriate
representation for the parameters.
Here we consider two simple examples of noninformative priors (Berger, 1985).
First of all, if a density takes the form
p(x|µ)=f(x−µ) (2.232)
then the parameter µis known as a location parameter . This family of densities
exhibits translation invariance because if we shift xby a constant to give ˆx=x+c,
then

then
p(ˆx|ˆµ)=f(ˆx−ˆµ) (2.233)
where we have deﬁned ˆµ=µ+c. Thus the density takes the same form in the
new variable as in the original one, and so the density is independent of the choiceof origin. We would like to choose a prior distribution that reﬂects this translation
invariance property, and so we choose a prior that assigns equal probability mass to

2.4. The Exponential Family 119
an interval A⩽µ⩽Bas to the shifted interval A−c⩽µ⩽B−c. This implies
∫B
Ap(µ)dµ=∫B−c
A−cp(µ)dµ=∫B
Ap(µ−c)dµ (2.234)
and because this must hold for all choices of AandB,w eh a v e
p(µ−c)=p(µ) (2.235)
which implies that p(µ)is constant. An example of a location parameter would be
the mean µof a Gaussian distribution. As we have seen, the conjugate prior distri-
bution for µin this case is a Gaussian p(µ|µ0,σ2
0)=N(µ|µ0,σ2
0), and we obtain a
noninformative prior by taking the limit σ2
0→∞ . Indeed, from (2.141) and (2.142)
we see that this gives a posterior distribution over µin which the contributions from
the prior vanish.
As a second example, consider a density of the form
p(x|σ)=1
σf(x
σ)
(2.236)
where σ>0. Note that this will be a normalized density provided f(x)is correctly
normalized. The parameter σis known as a scale parameter , and the density exhibits Exercise 2.59
scale invariance because if we scale xby a constant to give ˆx=cx, then
p(ˆx|ˆσ)=1
ˆσf(ˆx
ˆσ)
(2.237)

p(x|σ)=1
σf(x
σ)
(2.236)
ˆσf(ˆx
ˆσ)
(2.237)
where we have deﬁned ˆσ=cσ. This transformation corresponds to a change of
scale, for example from meters to kilometers if xis a length, and we would like
to choose a prior distribution that reﬂects this scale invariance. If we consider aninterval A⩽σ⩽B, and a scaled interval A/c⩽σ⩽B/c, then the prior should
assign equal probability mass to these two intervals. Thus we have
∫B
Ap(σ)dσ=∫B/c
A/cp(σ)dσ=∫B
Ap(1
cσ)1
cdσ (2.238)
and because this must hold for choices of AandB,w eh a v e
p(σ)=p(1
cσ)1
c(2.239)

∫B
Ap(1
cσ)1
p(σ)=p(1
cσ)1
c(2.239)
and hence p(σ)∝1/σ. Note that again this is an improper prior because the integral
of the distribution over 0⩽σ⩽∞is divergent. It is sometimes also convenient
to think of the prior distribution for a scale parameter in terms of the density of the
log of the parameter. Using the transformation rule (1.27) for densities we see thatp(lnσ) = const . Thus, for this prior there is the same probability mass in the range
1⩽σ⩽10as in the range 10⩽σ⩽100and in 100⩽σ⩽1000 .

120 2. PROBABILITY DISTRIBUTIONS
An example of a scale parameter would be the standard deviation σof a Gaussian
distribution, after we have taken account of the location parameter µ, because
N(x|µ, σ2)∝σ−1exp{
−(˜x/σ)2}
(2.240)
where˜x=x−µ. As discussed earlier, it is often more convenient to work in terms
of the precision λ=1/σ2rather than σitself. Using the transformation rule for
densities, we see that a distribution p(σ)∝1/σcorresponds to a distribution over λ
of the form p(λ)∝1/λ. We have seen that the conjugate prior for λwas the gamma
distribution Gam( λ|a0,b0)given by (2.146). The noninformative prior is obtained Section 2.3
as the special case a0=b0=0. Again, if we examine the results (2.150) and (2.151)
for the posterior distribution of λ, we see that for a0=b0=0, the posterior depends
only on terms arising from the data and not from the prior.
2.5. Nonparametric Methods

Throughout this chapter, we have focussed on the use of probability distributionshaving speciﬁc functional forms governed by a small number of parameters whosevalues are to be determined from a data set. This is called the parametric approach
to density modelling. An important limitation of this approach is that the chosen
density might be a poor model of the distribution that generates the data, which canresult in poor predictive performance. For instance, if the process that generates the

data is multimodal, then this aspect of the distribution can never be captured by a
Gaussian, which is necessarily unimodal.
In this ﬁnal section, we consider some nonparametric approaches to density es-
timation that make few assumptions about the form of the distribution. Here we shall
focus mainly on simple frequentist methods. The reader should be aware, however,that nonparametric Bayesian methods are attracting increasing interest (Walker et al. ,
1999; Neal, 2000; M ¨uller and Quintana, 2004; Teh et al. , 2006).

Let us start with a discussion of histogram methods for density estimation, which
we have already encountered in the context of marginal and conditional distributions
in Figure 1.11 and in the context of the central limit theorem in Figure 2.6. Here weexplore the properties of histogram density models in more detail, focussing on the
case of a single continuous variable x. Standard histograms simply partition xinto
distinct bins of width ∆
iand then count the number niof observations of xfalling

in bin i. In order to turn this count into a normalized probability density, we simply
divide by the total number Nof observations and by the width ∆iof the bins to
obtain probability values for each bin given by
pi=ni
N∆i(2.241)
for which it is easily seen that∫
p(x)dx=1. This gives a model for the density
p(x)that is constant over the width of each bin, and often the bins are chosen to have
the same width ∆i=∆ .

2.5. Nonparametric Methods 121
Figure 2.24 An illustration of the histogram approach
to density estimation, in which a data set
of50data points is generated from the
distribution shown by the green curve.
Histogram density estimates, based on
(2.241), with a common bin width ∆are
shown for various values of ∆.∆=0 .04
0 0.5 105
∆=0 .08
0 0.5 105
∆=0 .25
0 0.5 105
In Figure 2.24, we show an example of histogram density estimation. Here
the data is drawn from the distribution, corresponding to the green curve, which is
formed from a mixture of two Gaussians. Also shown are three examples of his-
togram density estimates corresponding to three different choices for the bin width
∆. We see that when ∆is very small (top ﬁgure), the resulting density model is very
spiky, with a lot of structure that is not present in the underlying distribution that
generated the data set. Conversely, if ∆is too large (bottom ﬁgure) then the result is

a model that is too smooth and that consequently fails to capture the bimodal prop-
erty of the green curve. The best results are obtained for some intermediate value
of∆(middle ﬁgure). In principle, a histogram density model is also dependent on
the choice of edge location for the bins, though this is typically much less signiﬁcant
than the value of ∆.
Note that the histogram method has the property (unlike the methods to be dis-
cussed shortly) that, once the histogram has been computed, the data set itself can

be discarded, which can be advantageous if the data set is large. Also, the histogram
approach is easily applied if the data points are arriving sequentially.
In practice, the histogram technique can be useful for obtaining a quick visual-
ization of data in one or two dimensions but is unsuited to most density estimation
applications. One obvious problem is that the estimated density has discontinuities
that are due to the bin edges rather than any property of the underlying distribution
that generated the data. Another major limitation of the histogram approach is its

scaling with dimensionality. If we divide each variable in a D-dimensional space
intoMbins, then the total number of bins will be MD. This exponential scaling
withDis an example of the curse of dimensionality. In a space of high dimensional- Section 1.4
ity, the quantity of data needed to provide meaningful estimates of local probability
density would be prohibitive.
The histogram approach to density estimation does, however, teach us two im-
portant lessons. First, to estimate the probability density at a particular location,

we should consider the data points that lie within some local neighbourhood of that
point. Note that the concept of locality requires that we assume some form of dis-
tance measure, and here we have been assuming Euclidean distance. For histograms,

122 2. PROBABILITY DISTRIBUTIONS
this neighbourhood property was deﬁned by the bins, and there is a natural ‘smooth-
ing’ parameter describing the spatial extent of the local region, in this case the binwidth. Second, the value of the smoothing parameter should be neither too large nor
too small in order to obtain good results. This is reminiscent of the choice of model
complexity in polynomial curve ﬁtting discussed in Chapter 1 where the degree M
of the polynomial, or alternatively the value αof the regularization parameter, was
optimal for some intermediate value, neither too large nor too small. Armed with
these insights, we turn now to a discussion of two widely used nonparametric tech-niques for density estimation, kernel estimators and nearest neighbours, which have
better scaling with dimensionality than the simple histogram model.
2.5.1 Kernel density estimators
Let us suppose that observations are being drawn from some unknown probabil-

ity density p(x)in some D-dimensional space, which we shall take to be Euclidean,
and we wish to estimate the value of p(x). From our earlier discussion of locality,
let us consider some small region Rcontaining x. The probability mass associated
with this region is given by
P=∫
Rp(x)dx. (2.242)
Now suppose that we have collected a data set comprising Nobservations drawn
fromp(x). Because each data point has a probability Pof falling within R, the total

P=∫
number Kof points that lie inside Rwill be distributed according to the binomial
distribution Section 2.1
Bin(K|N,P)=N!
K!(N−K)!PK(1−P)1−K. (2.243)
Using (2.11), we see that the mean fraction of points falling inside the region is
E[K/N]=P, and similarly using (2.12) we see that the variance around this mean
isvar[K/N]=P(1−P)/N. For large N, this distribution will be sharply peaked
around the mean and so
K≃NP. (2.244)
If, however, we also assume that the region Ris sufﬁciently small that the probability

density p(x)is roughly constant over the region, then we have
P≃p(x)V (2.245)
where Vis the volume of R. Combining (2.244) and (2.245), we obtain our density
estimate in the form
p(x)=K
NV. (2.246)
Note that the validity of (2.246) depends on two contradictory assumptions, namely
that the region Rbe sufﬁciently small that the density is approximately constant over
the region and yet sufﬁciently large (in relation to the value of that density) that the

p(x)=K
number Kof points falling inside the region is sufﬁcient for the binomial distribution
to be sharply peaked.

2.5. Nonparametric Methods 123
We can exploit the result (2.246) in two different ways. Either we can ﬁx Kand
determine the value of Vfrom the data, which gives rise to the K-nearest-neighbour
technique discussed shortly, or we can ﬁx Vand determine Kfrom the data, giv-
ing rise to the kernel approach. It can be shown that both the K-nearest-neighbour
density estimator and the kernel density estimator converge to the true probabilitydensity in the limit N→∞ provided Vshrinks suitably with N, andKgrows with
N(Duda and Hart, 1973).
We begin by discussing the kernel method in detail, and to start with we take
the region Rto be a small hypercube centred on the point xat which we wish to
determine the probability density. In order to count the number Kof points falling
within this region, it is convenient to deﬁne the following function
k(u)={
1,|ui|⩽1/2,i=1,...,D ,
0,otherwise(2.247)
which represents a unit cube centred on the origin. The function k(u)is an example

k(u)={
of a kernel function , and in this context is also called a Parzen window . From (2.247),
the quantity k((x−xn)/h)will be one if the data point xnlies inside a cube of side
hcentred on x, and zero otherwise. The total number of data points lying inside this
cube will therefore be
K=N∑
n=1k(x−xn
h)
. (2.248)
Substituting this expression into (2.246) then gives the following result for the esti-
mated density at x
p(x)=1
NN∑
n=11
hDk(x−xn
h)
(2.249)
where we have used V=hDfor the volume of a hypercube of side hinDdi-

K=N∑
n=1k(x−xn
h)
. (2.248)
p(x)=1
NN∑
n=11
hDk(x−xn
h)
(2.249)
mensions. Using the symmetry of the function k(u), we can now re-interpret this
equation, not as a single cube centred on xbut as the sum over Ncubes centred on
theNdata points xn.
As it stands, the kernel density estimator (2.249) will suffer from one of the same
problems that the histogram method suffered from, namely the presence of artiﬁcialdiscontinuities, in this case at the boundaries of the cubes. We can obtain a smoother
density model if we choose a smoother kernel function, and a common choice is the
Gaussian, which gives rise to the following kernel density model
p(x)=1
NN∑

p(x)=1
NN∑
n=11
(2πh2)1/2exp{
−∥x−xn∥2
2h2}
(2.250)
where hrepresents the standard deviation of the Gaussian components. Thus our
density model is obtained by placing a Gaussian over each data point and then addingup the contributions over the whole data set, and then dividing by Nso that the den-
sity is correctly normalized. In Figure 2.25, we apply the model (2.250) to the data

124 2. PROBABILITY DISTRIBUTIONS
Figure 2.25 Illustration of the kernel density model
(2.250) applied to the same data set used
to demonstrate the histogram approach in
Figure 2.24. We see that hacts as a
smoothing parameter and that if it is set
too small (top panel), the result is a very
noisy density model, whereas if it is set
too large (bottom panel), then the bimodal
nature of the underlying distribution from
which the data is generated (shown by the
green curve) is washed out. The best den-
sity model is obtained for some intermedi-
ate value of h(middle panel).h=0.005
0 0.5 105
h=0.07
0 0.5 105
h=0.2
0 0.5 105
set used earlier to demonstrate the histogram technique. We see that, as expected,
the parameter hplays the role of a smoothing parameter, and there is a trade-off
between sensitivity to noise at small hand over-smoothing at large h. Again, the
optimization of his a problem in model complexity, analogous to the choice of bin

0 0.5 105
h=0.07
0 0.5 105
h=0.2
0 0.5 105
width in histogram density estimation, or the degree of the polynomial used in curve
ﬁtting.
We can choose any other kernel function k(u)in (2.249) subject to the condi-
tions
k(u)⩾0, (2.251)∫
k(u)du=1 (2.252)
which ensure that the resulting probability distribution is nonnegative everywhere
and integrates to one. The class of density model given by (2.249) is called a kernel
density estimator, or Parzen estimator. It has a great merit that there is no compu-

ﬁtting.
tions
tation involved in the ‘training’ phase because this simply requires storage of the
training set. However, this is also one of its great weaknesses because the computa-
tional cost of evaluating the density grows linearly with the size of the data set.
2.5.2 Nearest-neighbour methods
One of the difﬁculties with the kernel approach to density estimation is that the
parameter hgoverning the kernel width is ﬁxed for all kernels. In regions of high
data density, a large value of hmay lead to over-smoothing and a washing out of

structure that might otherwise be extracted from the data. However, reducing hmay
lead to noisy estimates elsewhere in data space where the density is smaller. Thus
the optimal choice for hmay be dependent on location within the data space. This
issue is addressed by nearest-neighbour methods for density estimation.
We therefore return to our general result (2.246) for local density estimation,
and instead of ﬁxing Vand determining the value of Kfrom the data, we consider
a ﬁxed value of Kand use the data to ﬁnd an appropriate value for V. To do this,

we consider a small sphere centred on the point xat which we wish to estimate the

2.5. Nonparametric Methods 125
Figure 2.26 Illustration of K-nearest-neighbour den-
sity estimation using the same data set
as in Figures 2.25 and 2.24. We see
that the parameter Kgoverns the degree
of smoothing, so that a small value of
Kleads to a very noisy density model
(top panel), whereas a large value (bot-
tom panel) smoothes out the bimodal na-
ture of the true distribution (shown by the
green curve) from which the data set was
generated.K=1
0 0.5 105
K=5
0 0.5 105
K=3 0
0 0.5 105
density p(x), and we allow the radius of the sphere to grow until it contains precisely
Kdata points. The estimate of the density p(x)is then given by (2.246) with Vset to
the volume of the resulting sphere. This technique is known as Knearest neighbours
and is illustrated in Figure 2.26, for various choices of the parameter K, using the
same data set as used in Figure 2.24 and Figure 2.25. We see that the value of K
now governs the degree of smoothing and that again there is an optimum choice for

Kthat is neither too large nor too small. Note that the model produced by Knearest
neighbours is not a true density model because the integral over all space diverges. Exercise 2.61
We close this chapter by showing how the K-nearest-neighbour technique for
density estimation can be extended to the problem of classiﬁcation. To do this, we
apply the K-nearest-neighbour density estimation technique to each class separately
and then make use of Bayes’ theorem. Let us suppose that we have a data set com-

prising Nkpoints in class CkwithNpoints in total, so that∑
kNk=N.I f w e
wish to classify a new point x, we draw a sphere centred on xcontaining precisely
Kpoints irrespective of their class. Suppose this sphere has volume Vand contains
Kkpoints from class Ck. Then (2.246) provides an estimate of the density associated
with each class
p(x|Ck)=Kk
NkV. (2.253)
Similarly, the unconditional density is given by
p(x)=K
NV(2.254)
while the class priors are given by
p(Ck)=Nk
N. (2.255)

p(x)=K
NV(2.254)
p(Ck)=Nk
N. (2.255)
We can now combine (2.253), (2.254), and (2.255) using Bayes’ theorem to obtain
the posterior probability of class membership
p(x)=Kk
K. (2.256)

126 2. PROBABILITY DISTRIBUTIONS
Figure 2.27 (a) In the K-nearest-
neighbour classiﬁer, a new point,
shown by the black diamond, is clas-
siﬁed according to the majority class
membership of the Kclosest train-
ing data points, in this case K=
3. (b) In the nearest-neighbour
(K=1) approach to classiﬁcation,
the resulting decision boundary is
composed of hyperplanes that form
perpendicular bisectors of pairs of
points from different classes.
x1x2
(a)x1x2
(b)
If we wish to minimize the probability of misclassiﬁcation, this is done by assigning
the test point xto the class having the largest posterior probability, corresponding to
the largest value of Kk/K. Thus to classify a new point, we identify the Knearest
points from the training data set and then assign the new point to the class having the
largest number of representatives amongst this set. Ties can be broken at random.
The particular case of K=1 is called the nearest-neighbour rule, because a test

(b)
point is simply assigned to the same class as the nearest point from the training set.
These concepts are illustrated in Figure 2.27.
In Figure 2.28, we show the results of applying the K-nearest-neighbour algo-
rithm to the oil ﬂow data, introduced in Chapter 1, for various values of K.A s
expected, we see that Kcontrols the degree of smoothing, so that small Kproduces
many small regions of each class, whereas large Kleads to fewer larger regions.
x6x7
K=1
0 1 2012
x6x7K=3
0 1 2012
x6x7K=31
0 1 2012

x6x7
K=1
0 1 2012
x6x7K=3
0 1 2012
x6x7K=31
0 1 2012
Figure 2.28 Plot of 200 data points from the oil data set showing values of x6plotted against x7, where the
red, green, and blue points correspond to the ‘laminar’, ‘annular’, and ‘homogeneous’ classes, respectively. Also
shown are the classiﬁcations of the input space given by the K-nearest-neighbour algorithm for various values
ofK.

Exercises 127
An interesting property of the nearest-neighbour ( K=1) classiﬁer is that, in the
limitN→∞ , the error rate is never more than twice the minimum achievable error
rate of an optimal classiﬁer, i.e., one that uses the true class distributions (Cover and
Hart, 1967) .
As discussed so far, both the K-nearest-neighbour method, and the kernel den-
sity estimator, require the entire training data set to be stored, leading to expensive
computation if the data set is large. This effect can be offset, at the expense of some
additional one-off computation, by constructing tree-based search structures to allow(approximate) near neighbours to be found efﬁciently without doing an exhaustive
search of the data set. Nevertheless, these nonparametric methods are still severely
limited. On the other hand, we have seen that simple parametric models are very
restricted in terms of the forms of distribution that they can represent. We therefore

need to ﬁnd density models that are very ﬂexible and yet for which the complexityof the models can be controlled independently of the size of the training set, and we
shall see in subsequent chapters how to achieve this.
Exercises
2.1 (⋆)www Verify that the Bernoulli distribution (2.2) satisﬁes the following prop-
erties
1∑
x=0p(x|µ)=1 (2.257)
E[x]= µ (2.258)
var[x]= µ(1−µ). (2.259)
Show that the entropy H[x]of a Bernoulli distributed random binary variable xis
given by
H[x]=−µlnµ−(1−µ)l n ( 1−µ). (2.260)

Exercises
erties
1∑
given by
2.2 (⋆⋆)The form of the Bernoulli distribution given by (2.2) is not symmetric be-
tween the two values of x. In some situations, it will be more convenient to use an
equivalent formulation for which x∈{ −1,1}, in which case the distribution can be
written
p(x|µ)=(1−µ
2)(1−x)/2(1+µ
2)(1+x)/2
(2.261)
where µ∈[−1,1]. Show that the distribution (2.261) is normalized, and evaluate its
mean, variance, and entropy.
2.3 (⋆⋆)www In this exercise, we prove that the binomial distribution (2.9) is nor-

written
2)(1+x)/2
(2.261)
malized. First use the deﬁnition (2.10) of the number of combinations of midentical
objects chosen from a total of Nto show that
(N
m)
+(N
m−1)
=(N+1
m)
. (2.262)

128 2. PROBABILITY DISTRIBUTIONS
Use this result to prove by induction the following result
(1 +x)N=N∑
m=0(N
m)
xm(2.263)
which is known as the binomial theorem , and which is valid for all real values of x.
Finally, show that the binomial distribution is normalized, so that
N∑
m=0(N
m)
µm(1−µ)N−m=1 (2.264)
which can be done by ﬁrst pulling out a factor (1−µ)Nout of the summation and
then making use of the binomial theorem.
2.4 (⋆⋆)Show that the mean of the binomial distribution is given by (2.11). To do this,
differentiate both sides of the normalization condition (2.264) with respect to µand
then rearrange to obtain an expression for the mean of n. Similarly, by differentiating
(2.264) twice with respect to µand making use of the result (2.11) for the mean of
the binomial distribution prove the result (2.12) for the variance of the binomial.
2.5 (⋆⋆)www In this exercise, we prove that the beta distribution, given by (2.13), is
correctly normalized, so that (2.14) holds. This is equivalent to showing that
∫1

∫1
0µa−1(1−µ)b−1dµ=Γ(a)Γ(b)
Γ(a+b). (2.265)
From the deﬁnition (1.141) of the gamma function, we have
Γ(a)Γ(b)=∫∞
0exp(−x)xa−1dx∫∞
0exp(−y)yb−1dy. (2.266)
Use this expression to prove (2.265) as follows. First bring the integral over yinside
the integrand of the integral over x, next make the change of variable t=y+x
where xis ﬁxed, then interchange the order of the xandtintegrations, and ﬁnally
make the change of variable x=tµwhere tis ﬁxed.

∫1
2.6 (⋆)Make use of the result (2.265) to show that the mean, variance, and mode of the
beta distribution (2.13) are given respectively by
E[µ]=a
a+b(2.267)
var[µ]=ab
(a+b)2(a+b+1 )(2.268)
mode[ µ]=a−1
a+b−2. (2.269)

Exercises 129
2.7 (⋆⋆)Consider a binomial random variable xgiven by (2.9), with prior distribution
forµgiven by the beta distribution (2.13), and suppose we have observed moccur-
rences of x=1andloccurrences of x=0. Show that the posterior mean value of x
lies between the prior mean and the maximum likelihood estimate for µ. To do this,
show that the posterior mean can be written as λtimes the prior mean plus (1−λ)
times the maximum likelihood estimate, where 0⩽λ⩽1. This illustrates the con-
cept of the posterior distribution being a compromise between the prior distribution
and the maximum likelihood solution.
2.8 (⋆)Consider two variables xandywith joint distribution p(x, y). Prove the follow-
ing two results
E[x]= Ey[Ex[x|y]] (2.270)
var[x]= Ey[varx[x|y]] + var y[Ex[x|y]]. (2.271)
Here Ex[x|y]denotes the expectation of xunder the conditional distribution p(x|y),
with a similar notation for the conditional variance.
2.9 (⋆⋆⋆ )www . In this exercise, we prove the normalization of the Dirichlet dis-

tribution (2.38) using induction. We have already shown in Exercise 2.5 that thebeta distribution, which is a special case of the Dirichlet for M=2, is normalized.
We now assume that the Dirichlet distribution is normalized for M−1variables
and prove that it is normalized for Mvariables. To do this, consider the Dirichlet
distribution over Mvariables, and take account of the constraint ∑M
k=1µk=1 by
eliminating µM, so that the Dirichlet is written
pM(µ1,...,µ M−1)=CMM−1∏
k=1µαk−1
k(
1−M−1∑
j=1µj)αM−1
(2.272)

k=1µk=1 by
k=1µαk−1
k(
1−M−1∑
(2.272)
and our goal is to ﬁnd an expression for CM. To do this, integrate over µM−1, taking
care over the limits of integration, and then make a change of variable so that this
integral has limits 0and1. By assuming the correct result for CM−1and making use
of (2.265), derive the expression for CM.
2.10 (⋆⋆)Using the property Γ(x+1 ) = xΓ(x)of the gamma function, derive the
following results for the mean, variance, and covariance of the Dirichlet distribution
given by (2.38)
E[µj]=αj
α0(2.273)
var[µj]=αj(α0−αj)
α2
0(α0+1 )(2.274)
cov[µjµl]= −αjαl
α2
0(α0+1 ),j ̸=l (2.275)
where α0is deﬁned by (2.39).

130 2. PROBABILITY DISTRIBUTIONS
2.11 (⋆)www By expressing the expectation of lnµjunder the Dirichlet distribution
(2.38) as a derivative with respect to αj, show that
E[lnµj]=ψ(αj)−ψ(α0) (2.276)
where α0is given by (2.39) and
ψ(a)≡d
dalnΓ(a) (2.277)
is the digamma function.
2.12 (⋆)The uniform distribution for a continuous variable xis deﬁned by
U(x|a, b)=1
b−a,a ⩽x⩽b. (2.278)
Verify that this distribution is normalized, and ﬁnd expressions for its mean and
variance.
2.13 (⋆⋆)Evaluate the Kullback-Leibler divergence (1.113) between two Gaussians
p(x)=N(x|µ,Σ)andq(x)=N(x|m,L).
2.14 (⋆⋆)www This exercise demonstrates that the multivariate distribution with max-
imum entropy, for a given covariance, is a Gaussian. The entropy of a distributionp(x)is given by
H[x]=−∫
p(x)l np(x)dx. (2.279)
We wish to maximize H[x]over all distributions p(x)subject to the constraints that
p(x)be normalized and that it have a speciﬁc mean and covariance, so that
∫
p(x)dx=1 (2.280)
∫
p(x)xdx=µ (2.281)
∫
p(x)(x−µ)(x−µ)Tdx=Σ. (2.282)

H[x]=−∫
∫
∫
∫
By performing a variational maximization of (2.279) and using Lagrange multipliers
to enforce the constraints (2.280), (2.281), and (2.282), show that the maximum
likelihood distribution is given by the Gaussian (2.43).
2.15 (⋆⋆)Show that the entropy of the multivariate Gaussian N(x|µ,Σ)is given by
H[x]=1
2ln|Σ|+D
2(1 + ln(2 π)) (2.283)
where Dis the dimensionality of x.

Exercises 131
2.16 (⋆⋆⋆ )www Consider two random variables x1andx2having Gaussian distri-
butions with means µ1,µ2and precisions τ1,τ2respectively. Derive an expression
for the differential entropy of the variable x=x1+x2. To do this, ﬁrst ﬁnd the
distribution of xby using the relation
p(x)=∫∞
−∞p(x|x2)p(x2)dx2 (2.284)
and completing the square in the exponent. Then observe that this represents the
convolution of two Gaussian distributions, which itself will be Gaussian, and ﬁnally
make use of the result (1.110) for the entropy of the univariate Gaussian.
2.17 (⋆)www Consider the multivariate Gaussian distribution given by (2.43). By
writing the precision matrix (inverse covariance matrix) Σ−1as the sum of a sym-
metric and an anti-symmetric matrix, show that the anti-symmetric term does not
appear in the exponent of the Gaussian, and hence that the precision matrix may be
taken to be symmetric without loss of generality. Because the inverse of a symmetric

matrix is also symmetric (see Exercise 2.22), it follows that the covariance matrix
may also be chosen to be symmetric without loss of generality.
2.18 (⋆⋆⋆ )Consider a real, symmetric matrix Σwhose eigenvalue equation is given
by (2.45). By taking the complex conjugate of this equation and subtracting theoriginal equation, and then forming the inner product with eigenvector u
i, show that
the eigenvalues λiare real. Similarly, use the symmetry property of Σto show that

two eigenvectors uiandujwill be orthogonal provided λj̸=λi. Finally, show that
without loss of generality, the set of eigenvectors can be chosen to be orthonormal,
so that they satisfy (2.46), even if some of the eigenvalues are zero.
2.19 (⋆⋆)Show that a real, symmetric matrix Σhaving the eigenvector equation (2.45)
can be expressed as an expansion in the eigenvectors, with coefﬁcients given by the
eigenvalues, of the form (2.48). Similarly, show that the inverse matrix Σ−1has a
representation of the form (2.49).

2.20 (⋆⋆)www A positive deﬁnite matrix Σcan be deﬁned as one for which the
quadratic form
aTΣa (2.285)
is positive for any real value of the vector a. Show that a necessary and sufﬁcient
condition for Σto be positive deﬁnite is that all of the eigenvalues λiofΣ, deﬁned
by (2.45), are positive.
2.21 (⋆)Show that a real, symmetric matrix of size D×DhasD(D+1)/2independent
2.22 (⋆)www Show that the inverse of a symmetric matrix is itself symmetric.
2.23 (⋆⋆)By diagonalizing the coordinate system using the eigenvector expansion (2.45),

show that the volume contained within the hyperellipsoid corresponding to a constant

132 2. PROBABILITY DISTRIBUTIONS
Mahalanobis distance ∆is given by
VD|Σ|1/2∆D(2.286)
where VDis the volume of the unit sphere in Ddimensions, and the Mahalanobis
distance is deﬁned by (2.44).
2.24 (⋆⋆)www Prove the identity (2.76) by multiplying both sides by the matrix
(
AB
CD)
(2.287)
and making use of the deﬁnition (2.77).
2.25 (⋆⋆)In Sections 2.3.1 and 2.3.2, we considered the conditional and marginal distri-
butions for a multivariate Gaussian. More generally, we can consider a partitioningof the components of xinto three groups x
a,xb, andxc, with a corresponding par-
titioning of the mean vector µand of the covariance matrix Σin the form
µ=(µa
µb
µc)
, Σ=(ΣaaΣabΣac
ΣbaΣbbΣbc
ΣcaΣcbΣcc)
. (2.288)
By making use of the results of Section 2.3, ﬁnd an expression for the conditional
distribution p(xa|xb)in which xchas been marginalized out.
2.26 (⋆⋆)A very useful result from linear algebra is the Woodbury matrix inversion
formula given by
(A+BCD )−1=A−1−A−1B(C−1+DA−1B)−1DA−1. (2.289)

µ=(µa
µb
µc)
ΣbaΣbbΣbc
. (2.288)
By multiplying both sides by (A+BCD )prove the correctness of this result.
2.27 (⋆)Letxandzbe two independent random vectors, so that p(x,z)=p(x)p(z).
Show that the mean of their sum y=x+zis given by the sum of the means of each
of the variable separately. Similarly, show that the covariance matrix of yis given by
the sum of the covariance matrices of xandz. Conﬁrm that this result agrees with
that of Exercise 1.10.
2.28 (⋆⋆⋆ )www Consider a joint distribution over the variable
z=(
x
y)
(2.290)

z=(
x
y)
(2.290)
whose mean and covariance are given by (2.108) and (2.105) respectively. By mak-
ing use of the results (2.92) and (2.93) show that the marginal distribution p(x)is
given (2.99). Similarly, by making use of the results (2.81) and (2.82) show that theconditional distribution p(y|x)is given by (2.100).

Exercises 133
2.29 (⋆⋆)Using the partitioned matrix inversion formula (2.76), show that the inverse of
the precision matrix (2.104) is given by the covariance matrix (2.105).
2.30 (⋆)By starting from (2.107) and making use of the result (2.105), verify the result
(2.108).
2.31 (⋆⋆)Consider two multidimensional random vectors xandzhaving Gaussian
distributions p(x)=N(x|µx,Σx)andp(z)=N(z|µz,Σz)respectively, together
with their sum y=x+z. Use the results (2.109) and (2.110) to ﬁnd an expression for
the marginal distribution p(y)by considering the linear-Gaussian model comprising
the product of the marginal distribution p(x)and the conditional distribution p(y|x).
2.32 (⋆⋆⋆ )www This exercise and the next provide practice at manipulating the
quadratic forms that arise in linear-Gaussian models, as well as giving an indepen-dent check of results derived in the main text. Consider a joint distribution p(x,y)
deﬁned by the marginal and conditional distributions given by (2.99) and (2.100).

By examining the quadratic form in the exponent of the joint distribution, and usingthe technique of ‘completing the square’ discussed in Section 2.3, ﬁnd expressions
for the mean and covariance of the marginal distribution p(y)in which the variable
xhas been integrated out. To do this, make use of the Woodbury matrix inversion
formula (2.289). Verify that these results agree with (2.109) and (2.110) obtained
using the results of Chapter 2.

2.33 (⋆⋆⋆ )Consider the same joint distribution as in Exercise 2.32, but now use the
technique of completing the square to ﬁnd expressions for the mean and covarianceof the conditional distribution p(x|y). Again, verify that these agree with the corre-
sponding expressions (2.111) and (2.112).
2.34 (⋆⋆)
www To ﬁnd the maximum likelihood solution for the covariance matrix
of a multivariate Gaussian, we need to maximize the log likelihood function (2.118)with respect to Σ, noting that the covariance matrix must be symmetric and positive

2.34 (⋆⋆)
deﬁnite. Here we proceed by ignoring these constraints and doing a straightforward
maximization. Using the results (C.21), (C.26), and (C.28) from Appendix C, showthat the covariance matrix Σthat maximizes the log likelihood function (2.118) is
given by the sample covariance (2.122). We note that the ﬁnal result is necessarily
symmetric and positive deﬁnite (provided the sample covariance is nonsingular).
2.35 (⋆⋆)Use the result (2.59) to prove (2.62). Now, using the results (2.59), and (2.62),
show that
E[x
nxm]=µµT+InmΣ (2.291)

show that
E[x
wherexndenotes a data point sampled from a Gaussian distribution with mean µ
and covariance Σ, andInmdenotes the (n, m)element of the identity matrix. Hence
prove the result (2.124).
2.36 (⋆⋆)www Using an analogous procedure to that used to obtain (2.126), derive
an expression for the sequential estimation of the variance of a univariate Gaussian

134 2. PROBABILITY DISTRIBUTIONS
distribution, by starting with the maximum likelihood expression
σ2
ML=1
NN∑
n=1(xn−µ)2. (2.292)
Verify that substituting the expression for a Gaussian distribution into the Robbins-
Monro sequential estimation formula (2.135) gives a result of the same form, andhence obtain an expression for the corresponding coefﬁcients a
N.
2.37 (⋆⋆)Using an analogous procedure to that used to obtain (2.126), derive an ex-
pression for the sequential estimation of the covariance of a multivariate Gaussiandistribution, by starting with the maximum likelihood expression (2.122). Verify that
substituting the expression for a Gaussian distribution into the Robbins-Monro se-
quential estimation formula (2.135) gives a result of the same form, and hence obtainan expression for the corresponding coefﬁcients a
N.
2.38 (⋆)Use the technique of completing the square for the quadratic form in the expo-
nent to derive the results (2.141) and (2.142).

N.
2.39 (⋆⋆)Starting from the results (2.141) and (2.142) for the posterior distribution
of the mean of a Gaussian random variable, dissect out the contributions from the
ﬁrstN−1data points and hence obtain expressions for the sequential update of
µNandσ2
N. Now derive the same results starting from the posterior distribution
p(µ|x1,...,x N−1)=N(µ|µN−1,σ2
N−1)and multiplying by the likelihood func-
tionp(xN|µ)=N(xN|µ, σ2)and then completing the square and normalizing to
obtain the posterior distribution after Nobservations.
2.40 (⋆⋆)www Consider a D-dimensional Gaussian random variable xwith distribu-

µNandσ2
tionN(x|µ,Σ)in which the covariance Σis known and for which we wish to infer
the mean µfrom a set of observations X={x1,...,xN}. Given a prior distribution
p(µ)=N(µ|µ0,Σ0), ﬁnd the corresponding posterior distribution p(µ|X).
2.41 (⋆)Use the deﬁnition of the gamma function (1.141) to show that the gamma dis-
tribution (2.146) is normalized.
2.42 (⋆⋆)Evaluate the mean, variance, and mode of the gamma distribution (2.146).
2.43 (⋆)The following distribution
p(x|σ2,q)=q
2(2σ2)1/qΓ(1/q)exp(
−|x|q
2σ2)
(2.293)

−|x|q
2σ2)
(2.293)
is a generalization of the univariate Gaussian distribution. Show that this distribution
is normalized so that ∫∞
−∞p(x|σ2,q)dx=1 (2.294)
and that it reduces to the Gaussian when q=2. Consider a regression model in
which the target variable is given by t=y(x,w)+ϵandϵis a random noise

Exercises 135
variable drawn from the distribution (2.293). Show that the log likelihood function
overwandσ2, for an observed data set of input vectors X={x1,...,xN}and
corresponding target variables t=(t1,...,t N)T,i sg i v e nb y
lnp(t|X,w,σ2)=−1
2σ2N∑
n=1|y(xn,w)−tn|q−N
qln(2σ2)+c o n s t (2.295)
where ‘ const ’ denotes terms independent of both wandσ2. Note that, as a function
ofw, this is the Lqerror function considered in Section 1.5.5.
2.44 (⋆⋆)Consider a univariate Gaussian distribution N(x|µ, τ−1)having conjugate
Gaussian-gamma prior given by (2.154), and a data set x={x1,...,x N}of i.i.d.
observations. Show that the posterior distribution is also a Gaussian-gamma distri-
bution of the same functional form as the prior, and write down expressions for the
parameters of this posterior distribution.
2.45 (⋆)Verify that the Wishart distribution deﬁned by (2.155) is indeed a conjugate
prior for the precision matrix of a multivariate Gaussian.

2.46 (⋆)www Verify that evaluating the integral in (2.158) leads to the result (2.159).
2.47 (⋆)www Show that in the limit ν→∞ , the t-distribution (2.159) becomes a
Gaussian. Hint: ignore the normalization coefﬁcient, and simply look at the depen-
dence on x.
2.48 (⋆)By following analogous steps to those used to derive the univariate Student’s
t-distribution (2.159), verify the result (2.162) for the multivariate form of the Stu-
dent’s t-distribution, by marginalizing over the variable ηin (2.161). Using the

dence on x.
deﬁnition (2.161), show by exchanging integration variables that the multivariate
t-distribution is correctly normalized.
2.49 (⋆⋆)By using the deﬁnition (2.161) of the multivariate Student’s t-distribution as a
convolution of a Gaussian with a gamma distribution, verify the properties (2.164),
(2.165), and (2.166) for the multivariate t-distribution deﬁned by (2.162).
2.50 (⋆)Show that in the limit ν→∞ , the multivariate Student’s t-distribution (2.162)
reduces to a Gaussian with mean µand precision Λ.
2.51 (⋆)www The various trigonometric identities used in the discussion of periodic

variables in this chapter can be proven easily from the relation
exp(iA)=c o s A+isinA (2.296)
in which iis the square root of minus one. By considering the identity
exp(iA) exp(−iA)=1 (2.297)
prove the result (2.177). Similarly, using the identity
cos(A−B)=ℜexp{i(A−B)} (2.298)

136 2. PROBABILITY DISTRIBUTIONS
where ℜdenotes the real part, prove (2.178). Finally, by using sin(A−B)=
ℑexp{i(A−B)}, where ℑdenotes the imaginary part, prove the result (2.183).
2.52 (⋆⋆)For large m, the von Mises distribution (2.179) becomes sharply peaked
around the mode θ0. By deﬁning ξ=m1/2(θ−θ0)and making the Taylor ex-
pansion of the cosine function given by
cosα=1−α2
2+O(α4) (2.299)
show that as m→∞ , the von Mises distribution tends to a Gaussian.
2.53 (⋆)Using the trigonometric identity (2.183), show that solution of (2.182) for θ0is
given by (2.184).
2.54 (⋆)By computing ﬁrst and second derivatives of the von Mises distribution (2.179),
and using I0(m)>0form>0, show that the maximum of the distribution occurs
whenθ=θ0and that the minimum occurs when θ=θ0+π(mod2 π).
2.55 (⋆)By making use of the result (2.168), together with (2.184) and the trigonometric
identity (2.178), show that the maximum likelihood solution mMLfor the concentra-

tion of the von Mises distribution satisﬁes A(mML)=rwhere ris the radius of the
mean of the observations viewed as unit vectors in the two-dimensional Euclidean
plane, as illustrated in Figure 2.17.
2.56 (⋆⋆)www Express the beta distribution (2.13), the gamma distribution (2.146),
and the von Mises distribution (2.179) as members of the exponential family (2.194)and thereby identify their natural parameters.
2.57 (⋆)Verify that the multivariate Gaussian distribution can be cast in exponential

family form (2.194) and derive expressions for η,u(x),h(x)andg(η)analogous to
(2.220)–(2.223).
2.58 (⋆)The result (2.226) showed that the negative gradient of lng(η)for the exponen-
tial family is given by the expectation of u(x). By taking the second derivatives of
(2.195), show that
−∇∇ lng(η)= E[u(x)u(x)
T]−E[u(x)]E[u(x)T]=c o v [ u(x)]. (2.300)
2.59 (⋆)By changing variables using y=x/σ, show that the density (2.236) will be
correctly normalized, provided f(x)is correctly normalized.

2.60 (⋆⋆)www Consider a histogram-like density model in which the space xis di-
vided into ﬁxed regions for which the density p(x)takes the constant value hiover
theithregion, and that the volume of region iis denoted ∆i. Suppose we have a set
ofNobservations of xsuch that niof these observations fall in region i. Using a
Lagrange multiplier to enforce the normalization constraint on the density, derive an
expression for the maximum likelihood estimator for the {hi}.

2.61 (⋆)Show that the K-nearest-neighbour density model deﬁnes an improper distribu-
tion whose integral over all space is divergent.

3
Linear
Models for
The focus so far in this book has been on unsupervised learning, including topics
such as density estimation and data clustering. We turn now to a discussion of super-vised learning, starting with regression. The goal of regression is to predict the value
of one or more continuous target variables tgiven the value of a D-dimensional vec-
torxofinput variables. We have already encountered an example of a regression
problem when we considered polynomial curve ﬁtting in Chapter 1. The polynomial
is a speciﬁc example of a broad class of functions called linear regression models,
which share the property of being linear functions of the adjustable parameters, andwhich will form the focus of this chapter. The simplest form of linear regression
models are also linear functions of the input variables. However, we can obtain a

much more useful class of functions by taking linear combinations of a ﬁxed set ofnonlinear functions of the input variables, known as basis functions . Such models
are linear functions of the parameters, which gives them simple analytical properties,
and yet can be nonlinear with respect to the input variables.
137

138 3. LINEAR MODELS FOR REGRESSION
Given a training data set comprising Nobservations {xn}, where n=1,...,N ,
together with corresponding target values {tn}, the goal is to predict the value of t
for a new value of x. In the simplest approach, this can be done by directly con-
structing an appropriate function y(x)whose values for new inputs xconstitute the
predictions for the corresponding values of t. More generally, from a probabilistic
perspective, we aim to model the predictive distribution p(t|x)because this expresses
our uncertainty about the value of tfor each value of x. From this conditional dis-
tribution we can make predictions of t, for any new value of x, in such a way as to
minimize the expected value of a suitably chosen loss function. As discussed in Sec-
tion 1.5.5, a common choice of loss function for real-valued variables is the squared
loss, for which the optimal solution is given by the conditional expectation of t.

Although linear models have signiﬁcant limitations as practical techniques for
pattern recognition, particularly for problems involving input spaces of high dimen-sionality, they have nice analytical properties and form the foundation for more so-
phisticated models to be discussed in later chapters.
3.1. Linear Basis Function Models
The simplest linear model for regression is one that involves a linear combination ofthe input variables
y(x,w)=w
0+w1x1+...+wDxD (3.1)

y(x,w)=w
wherex=(x1,...,x D)T. This is often simply known as linear regression . The key
property of this model is that it is a linear function of the parameters w0,...,w D.I ti s
also, however, a linear function of the input variables xi, and this imposes signiﬁcant
limitations on the model. We therefore extend the class of models by considering
linear combinations of ﬁxed nonlinear functions of the input variables, of the form
y(x,w)=w0+M−1∑
j=1wjφj(x) (3.2)
where φj(x)are known as basis functions . By denoting the maximum value of the

indexjbyM−1, the total number of parameters in this model will be M.
The parameter w0allows for any ﬁxed offset in the data and is sometimes called
abias parameter (not to be confused with ‘bias’ in a statistical sense). It is often
convenient to deﬁne an additional dummy ‘basis function’ φ0(x)=1 so that
y(x,w)=M−1∑
j=0wjφj(x)=wTφ(x) (3.3)
wherew=(w0,...,w M−1)Tandφ=(φ0,...,φ M−1)T. In many practical ap-
plications of pattern recognition, we will apply some form of ﬁxed pre-processing,

3.1. Linear Basis Function Models 139
or feature extraction, to the original data variables. If the original variables com-
prise the vector x, then the features can be expressed in terms of the basis functions
{φj(x)}.
By using nonlinear basis functions, we allow the function y(x,w)to be a non-
linear function of the input vector x. Functions of the form (3.2) are called linear
models, however, because this function is linear in w. It is this linearity in the pa-
rameters that will greatly simplify the analysis of this class of models. However, it
also leads to some signiﬁcant limitations, as we discuss in Section 3.6.
The example of polynomial regression considered in Chapter 1 is a particular
example of this model in which there is a single input variable x, and the basis func-
tions take the form of powers of xso that φj(x)=xj. One limitation of polynomial
basis functions is that they are global functions of the input variable, so that changes

in one region of input space affect all other regions. This can be resolved by dividingthe input space up into regions and ﬁt a different polynomial in each region, leading
tospline functions (Hastie et al. , 2001).
There are many other possible choices for the basis functions, for example
φ
j(x)=e x p{
−(x−µj)2
2s2}
(3.4)
where the µjgovern the locations of the basis functions in input space, and the pa-
rameter sgoverns their spatial scale. These are usually referred to as ‘Gaussian’

φ
j(x)=e x p{
−(x−µj)2
2s2}
(3.4)
basis functions, although it should be noted that they are not required to have a prob-abilistic interpretation, and in particular the normalization coefﬁcient is unimportant
because these basis functions will be multiplied by adaptive parameters w
j.
Another possibility is the sigmoidal basis function of the form
φj(x)=σ(x−µj
s)
(3.5)
where σ(a)is the logistic sigmoid function deﬁned by
σ(a)=1
1+e x p ( −a). (3.6)
Equivalently, we can use the ‘ tanh ’ function because this is related to the logistic

j.
s)
(3.5)
σ(a)=1
sigmoid by tanh(a)=2σ(a)−1, and so a general linear combination of logistic
sigmoid functions is equivalent to a general linear combination of ‘ tanh ’ functions.
These various choices of basis function are illustrated in Figure 3.1.
Yet another possible choice of basis function is the Fourier basis, which leads to
an expansion in sinusoidal functions. Each basis function represents a speciﬁc fre-

quency and has inﬁnite spatial extent. By contrast, basis functions that are localizedto ﬁnite regions of input space necessarily comprise a spectrum of different spatial
frequencies. In many signal processing applications, it is of interest to consider ba-
sis functions that are localized in both space and frequency, leading to a class offunctions known as wavelets . These are also deﬁned to be mutually orthogonal, to
simplify their application. Wavelets are most applicable when the input values live

140 3. LINEAR MODELS FOR REGRESSION
−1 0 100.250.50.751
Figure 3.1 Examples of basis functions, showing polynomials on the left, Gaussians of the form (3.4) in the
centre, and sigmoidal of the form (3.5) on the right.
on a regular lattice, such as the successive time points in a temporal sequence, or the
pixels in an image. Useful texts on wavelets include Ogden (1997), Mallat (1999),
and Vidakovic (1999).
Most of the discussion in this chapter, however, is independent of the particular
choice of basis function set, and so for most of our discussion we shall not specify
the particular form of the basis functions, except for the purposes of numerical il-
lustration. Indeed, much of our discussion will be equally applicable to the situation
in which the vector φ(x)of basis functions is simply the identity φ(x)=x. Fur-
thermore, in order to keep the notation simple, we shall focus on the case of a single

target variable t. However, in Section 3.1.5, we consider brieﬂy the modiﬁcations
needed to deal with multiple target variables.
3.1.1 Maximum likelihood and least squares
In Chapter 1, we ﬁtted polynomial functions to data sets by minimizing a sum-
of-squares error function. We also showed that this error function could be motivated
as the maximum likelihood solution under an assumed Gaussian noise model. Let
us return to this discussion and consider the least squares approach, and its relation

to maximum likelihood, in more detail.
As before, we assume that the target variable tis given by a deterministic func-
tiony(x,w)with additive Gaussian noise so that
t=y(x,w)+ϵ (3.7)
where ϵis a zero mean Gaussian random variable with precision (inverse variance)
β. Thus we can write
p(t|x,w,β)=N(t|y(x,w),β−1). (3.8)
Recall that, if we assume a squared loss function, then the optimal prediction, for a
new value of x, will be given by the conditional mean of the target variable. In the Section 1.5.5

case of a Gaussian conditional distribution of the form (3.8), the conditional mean

3.1. Linear Basis Function Models 141
will be simply
E[t|x]=∫
tp(t|x)dt=y(x,w). (3.9)
Note that the Gaussian noise assumption implies that the conditional distribution of
tgivenxis unimodal, which may be inappropriate for some applications. An ex-
tension to mixtures of conditional Gaussian distributions, which permit multimodalconditional distributions, will be discussed in Section 14.5.1.
Now consider a data set of inputs X={x
1,...,xN}with corresponding target
values t1,...,t N. We group the target variables {tn}into a column vector that we
denote by twhere the typeface is chosen to distinguish it from a single observation
of a multivariate target, which would be denoted t. Making the assumption that
these data points are drawn independently from the distribution (3.8), we obtain thefollowing expression for the likelihood function, which is a function of the adjustable
parameters wandβ, in the form
p(t|X,w,β)=N∏
n=1N(tn|wTφ(xn),β−1) (3.10)

where we have used (3.3). Note that in supervised learning problems such as regres-
sion (and classiﬁcation), we are not seeking to model the distribution of the inputvariables. Thus xwill always appear in the set of conditioning variables, and so
from now on we will drop the explicit xfrom expressions such as p(t|x,w,β)in or-
der to keep the notation uncluttered. Taking the logarithm of the likelihood function,and making use of the standard form (1.46) for the univariate Gaussian, we have
lnp(t|w,β)=
N∑
n=1lnN(tn|wTφ(xn),β−1)

N∑
=N
2lnβ−N
2ln(2π)−βED(w) (3.11)
where the sum-of-squares error function is deﬁned by
ED(w)=1
2N∑
n=1{tn−wTφ(xn)}2. (3.12)
Having written down the likelihood function, we can use maximum likelihood to
determine wandβ. Consider ﬁrst the maximization with respect to w. As observed
already in Section 1.2.5, we see that maximization of the likelihood function under a
conditional Gaussian noise distribution for a linear model is equivalent to minimizing
a sum-of-squares error function given by ED(w). The gradient of the log likelihood
function (3.11) takes the form

ED(w)=1
2N∑
∇lnp(t|w,β)=N∑
n=1{
tn−wTφ(xn)}
φ(xn)T. (3.13)

142 3. LINEAR MODELS FOR REGRESSION
Setting this gradient to zero gives
0=N∑
n=1tnφ(xn)T−wT(N∑
n=1φ(xn)φ(xn)T)
. (3.14)
Solving for wwe obtain
wML=(
ΦTΦ)−1ΦTt (3.15)
which are known as the normal equations for the least squares problem. Here Φis an
N×Mmatrix, called the design matrix , whose elements are given by Φnj=φj(xn),
so that
Φ=⎛
⎜⎜⎝φ0(x1)φ1(x1)···φM−1(x1)
φ0(x2)φ1(x2)···φM−1(x2)
φ0(xN)φ1(xN)···φM−1(xN)⎞
⎟⎟⎠. (3.16)
The quantity
Φ†≡(
ΦTΦ)−1ΦT(3.17)
is known as the Moore-Penrose pseudo-inverse of the matrix Φ(Rao and Mitra,
1971; Golub and Van Loan, 1996). It can be regarded as a generalization of the
notion of matrix inverse to nonsquare matrices. Indeed, if Φis square and invertible,
then using the property (AB)−1=B−1A−1we see that Φ†≡Φ−1.
At this point, we can gain some insight into the role of the bias parameter w0.I f
we make the bias parameter explicit, then the error function (3.12) becomes
ED(w)=1
2N∑
n=1{tn−w0−M−1∑
j=1wjφj(xn)}2. (3.18)

ED(w)=1
2N∑
Setting the derivative with respect to w0equal to zero, and solving for w0, we obtain
w0=t−M−1∑
j=1wjφj (3.19)
t=1
NN∑
n=1tn, φj=1
NN∑
n=1φj(xn). (3.20)
Thus the bias w0compensates for the difference between the averages (over the
training set) of the target values and the weighted sum of the averages of the basis
function values.
We can also maximize the log likelihood function (3.11) with respect to the noise
precision parameter β, giving
1
βML=1
NN∑
n=1{tn−wT
MLφ(xn)}2(3.21)

3.1. Linear Basis Function Models 143
Figure 3.2 Geometrical interpretation of the least-squares
solution, in an N-dimensional space whose axes
are the values of t1,...,t N. The least-squares
regression function is obtained by ﬁnding the or-
thogonal projection of the data vector tonto the
subspace spanned by the basis functions φj(x)
in which each basis function is viewed as a vec-
torϕjof length Nwith elements φj(xn).S
t
y ϕ1ϕ2
and so we see that the inverse of the noise precision is given by the residual variance
of the target values around the regression function.
3.1.2 Geometry of least squares
At this point, it is instructive to consider the geometrical interpretation of the
least-squares solution. To do this we consider an N-dimensional space whose axes
are given by the tn, so that t=(t1,...,t N)Tis a vector in this space. Each basis
function φj(xn), evaluated at the Ndata points, can also be represented as a vector in
the same space, denoted by ϕj, as illustrated in Figure 3.2. Note that ϕjcorresponds

to the jthcolumn of Φ, whereas φ(xn)corresponds to the nthrow of Φ. If the
number Mof basis functions is smaller than the number Nof data points, then the
Mvectors φj(xn)will span a linear subspace Sof dimensionality M. We deﬁne
yto be an N-dimensional vector whose nthelement is given by y(xn,w), where
n=1,...,N . Because yis an arbitrary linear combination of the vectors ϕj, it can
live anywhere in the M-dimensional subspace. The sum-of-squares error (3.12) is

then equal (up to a factor of 1/2) to the squared Euclidean distance between yand
t. Thus the least-squares solution for wcorresponds to that choice of ythat lies in
subspace Sand that is closest to t. Intuitively, from Figure 3.2, we anticipate that
this solution corresponds to the orthogonal projection of tonto the subspace S. This
is indeed the case, as can easily be veriﬁed by noting that the solution for yis given
byΦwML, and then conﬁrming that this takes the form of an orthogonal projection. Exercise 3.2

In practice, a direct solution of the normal equations can lead to numerical difﬁ-
culties when ΦTΦis close to singular. In particular, when two or more of the basis
vectors ϕjare co-linear, or nearly so, the resulting parameter values can have large
magnitudes. Such near degeneracies will not be uncommon when dealing with real
data sets. The resulting numerical difﬁculties can be addressed using the technique
ofsingular value decomposition ,o r SVD (Press et al. , 1992; Bishop and Nabney,
2008). Note that the addition of a regularization term ensures that the matrix is non-

singular, even in the presence of degeneracies.
3.1.3 Sequential learning
Batch techniques, such as the maximum likelihood solution (3.15), which in-
volve processing the entire training set in one go, can be computationally costly for
large data sets. As we have discussed in Chapter 1, if the data set is sufﬁciently large,
it may be worthwhile to use sequential algorithms, also known as on-line algorithms,

144 3. LINEAR MODELS FOR REGRESSION
in which the data points are considered one at a time, and the model parameters up-
dated after each such presentation. Sequential learning is also appropriate for real-time applications in which the data observations are arriving in a continuous stream,
and predictions must be made before all of the data points are seen.
We can obtain a sequential learning algorithm by applying the technique of
stochastic gradient descent , also known as sequential gradient descent , as follows. If
the error function comprises a sum over data points E=∑
nEn, then after presen-
tation of pattern n, the stochastic gradient descent algorithm updates the parameter
vectorwusing
w(τ+1)=w(τ)−η∇En (3.22)
where τdenotes the iteration number, and ηis a learning rate parameter. We shall
discuss the choice of value for ηshortly. The value of wis initialized to some starting
vectorw(0). For the case of the sum-of-squares error function (3.12), this gives
w(τ+1)=w(τ)+η(tn−w(τ)Tφn)φn (3.23)

where φn=φ(xn). This is known as least-mean-squares or the LMS algorithm .
The value of ηneeds to be chosen with care to ensure that the algorithm converges
(Bishop and Nabney, 2008).
3.1.4 Regularized least squares
In Section 1.1, we introduced the idea of adding a regularization term to an
error function in order to control over-ﬁtting, so that the total error function to be
minimized takes the form
ED(w)+λEW(w) (3.24)
where λis the regularization coefﬁcient that controls the relative importance of the

data-dependent error ED(w)and the regularization term EW(w). One of the sim-
plest forms of regularizer is given by the sum-of-squares of the weight vector ele-
ments
EW(w)=1
2wTw. (3.25)
If we also consider the sum-of-squares error function given by
E(w)=1
2N∑
n=1{tn−wTφ(xn)}2(3.26)
then the total error function becomes
1
2N∑
n=1{tn−wTφ(xn)}2+λ
2wTw. (3.27)
This particular choice of regularizer is known in the machine learning literature as

ments
EW(w)=1
E(w)=1
2N∑
1
2N∑
weight decay because in sequential learning algorithms, it encourages weight values
to decay towards zero, unless supported by the data. In statistics, it provides an ex-
ample of a parameter shrinkage method because it shrinks parameter values towards

3.1. Linear Basis Function Models 145
q=0.5 q=1 q=2 q=4
Figure 3.3 Contours of the regularization term in (3.29) for various values of the parameter q.
zero. It has the advantage that the error function remains a quadratic function of
w, and so its exact minimizer can be found in closed form. Speciﬁcally, setting the
gradient of (3.27) with respect to wto zero, and solving for was before, we obtain
w=(
λI+ΦTΦ)−1ΦTt. (3.28)
This represents a simple extension of the least-squares solution (3.15).
A more general regularizer is sometimes used, for which the regularized error
1
2N∑
2M∑
j=1|wj|q(3.29)
where q=2corresponds to the quadratic regularizer (3.27). Figure 3.3 shows con-
tours of the regularization function for different values of q.
The case of q=1 is know as the lasso in the statistics literature (Tibshirani,
1996). It has the property that if λis sufﬁciently large, some of the coefﬁcients
wjare driven to zero, leading to a sparse model in which the corresponding basis

1
2N∑
2M∑
functions play no role. To see this, we ﬁrst note that minimizing (3.29) is equivalent
to minimizing the unregularized sum-of-squares error (3.12) subject to the constraint Exercise 3.5
M∑
j=1|wj|q⩽η (3.30)
for an appropriate value of the parameter η, where the two approaches can be related
using Lagrange multipliers. The origin of the sparsity can be seen from Figure 3.4, Appendix E
which shows that the minimum of the error function, subject to the constraint (3.30).
Asλis increased, so an increasing number of parameters are driven to zero.

M∑
Regularization allows complex models to be trained on data sets of limited size
without severe over-ﬁtting, essentially by limiting the effective model complexity.
However, the problem of determining the optimal model complexity is then shifted
from one of ﬁnding the appropriate number of basis functions to one of determining
a suitable value of the regularization coefﬁcient λ. We shall return to the issue of
model complexity later in this chapter.

146 3. LINEAR MODELS FOR REGRESSION
Figure 3.4 Plot of the contours
of the unregularized error function
(blue) along with the constraint re-
gion (3.30) for the quadratic regular-
izerq=2on the left and the lasso
regularizer q=1 on the right, in
which the optimum value for the pa-
rameter vector wis denoted by w⋆.
The lasso gives a sparse solution in
which w⋆
1=0.
w1w2
w⋆
w1w2
w⋆
For the remainder of this chapter we shall focus on the quadratic regularizer
(3.27) both for its practical importance and its analytical tractability.
3.1.5 Multiple outputs
So far, we have considered the case of a single target variable t. In some applica-
tions, we may wish to predict K>1target variables, which we denote collectively
by the target vector t. This could be done by introducing a different set of basis func-
tions for each component of t, leading to multiple, independent regression problems.
However, a more interesting, and more common, approach is to use the same set of

basis functions to model all of the components of the target vector so that
y(x,w)=WTφ(x) (3.31)
whereyis aK-dimensional column vector, Wis anM×Kmatrix of parameters,
andφ(x)is anM-dimensional column vector with elements φj(x), with φ0(x)=1
as before. Suppose we take the conditional distribution of the target vector to be an
isotropic Gaussian of the form
p(t|x,W,β)=N(t|WTφ(x),β−1I). (3.32)
If we have a set of observations t1,...,tN, we can combine these into a matrix T
of size N×Ksuch that the nthrow is given by tT
n. Similarly, we can combine the

input vectors x1,...,xNinto a matrix X. The log likelihood function is then given
by
lnp(T|X,W,β)=N∑
n=1lnN(tn|WTφ(xn),β−1I)
=NK
2ln(β
2π)
−β
2N∑
n=1tn−WTφ(xn)2.(3.33)

3.2. The Bias-Variance Decomposition 147
As before, we can maximize this function with respect to W, giving
WML=(
ΦTΦ)−1ΦTT. (3.34)
If we examine this result for each target variable tk,w eh a v e
wk=(
ΦTΦ)−1ΦTtk=Φ†tk (3.35)
where tkis anN-dimensional column vector with components tnkforn=1,...N .
Thus the solution to the regression problem decouples between the different target
variables, and we need only compute a single pseudo-inverse matrix Φ†, which is
shared by all of the vectors wk.
The extension to general Gaussian noise distributions having arbitrary covari-
ance matrices is straightforward. Again, this leads to a decoupling into Kinde- Exercise 3.6
pendent regression problems. This result is unsurprising because the parameters W
deﬁne only the mean of the Gaussian noise distribution, and we know from Sec-
tion 2.3.4 that the maximum likelihood solution for the mean of a multivariate Gaus-

sian is independent of the covariance. From now on, we shall therefore consider asingle target variable tfor simplicity.
3.2. The Bias-Variance Decomposition
So far in our discussion of linear models for regression, we have assumed that the
form and number of basis functions are both ﬁxed. As we have seen in Chapter 1,
the use of maximum likelihood, or equivalently least squares, can lead to severe
over-ﬁtting if complex models are trained using data sets of limited size. However,limiting the number of basis functions in order to avoid over-ﬁtting has the side

effect of limiting the ﬂexibility of the model to capture interesting and important
trends in the data. Although the introduction of regularization terms can controlover-ﬁtting for models with many parameters, this raises the question of how to
determine a suitable value for the regularization coefﬁcient λ. Seeking the solution
that minimizes the regularized error function with respect to both the weight vectorwand the regularization coefﬁcient λis clearly not the right approach since this
leads to the unregularized solution with λ=0.

As we have seen in earlier chapters, the phenomenon of over-ﬁtting is really an
unfortunate property of maximum likelihood and does not arise when we marginalize
over parameters in a Bayesian setting. In this chapter, we shall consider the Bayesian
view of model complexity in some depth. Before doing so, however, it is instructive
to consider a frequentist viewpoint of the model complexity issue, known as the bias-
variance trade-off. Although we shall introduce this concept in the context of linear

basis function models, where it is easy to illustrate the ideas using simple examples,
the discussion has more general applicability.
In Section 1.5.5, when we discussed decision theory for regression problems,
we considered various loss functions each of which leads to a corresponding optimal
prediction once we are given the conditional distribution p(t|x). A popular choice is

148 3. LINEAR MODELS FOR REGRESSION
the squared loss function, for which the optimal prediction is given by the conditional
expectation, which we denote by h(x)and which is given by
h(x)= E[t|x]=∫
tp(t|x)dt. (3.36)
At this point, it is worth distinguishing between the squared loss function arising
from decision theory and the sum-of-squares error function that arose in the maxi-
mum likelihood estimation of model parameters. We might use more sophisticatedtechniques than least squares, for example regularization or a fully Bayesian ap-
proach, to determine the conditional distribution p(t|x). These can all be combined
with the squared loss function for the purpose of making predictions.
We showed in Section 1.5.5 that the expected squared loss can be written in the
form
E[L]=∫
{y(x)−h(x)}2p(x)dx+∫
{h(x)−t}2p(x,t)dxdt. (3.37)
Recall that the second term, which is independent of y(x), arises from the intrinsic

form
E[L]=∫
noise on the data and represents the minimum achievable value of the expected loss.The ﬁrst term depends on our choice for the function y(x), and we will seek a so-
lution for y(x)which makes this term a minimum. Because it is nonnegative, the
smallest that we can hope to make this term is zero. If we had an unlimited supply ofdata (and unlimited computational resources), we could in principle ﬁnd the regres-
sion function h(x)to any desired degree of accuracy, and this would represent the
optimal choice for y(x). However, in practice we have a data set Dcontaining only

a ﬁnite number Nof data points, and consequently we do not know the regression
function h(x)exactly.
If we model the h(x)using a parametric function y(x,w)governed by a pa-
rameter vector w, then from a Bayesian perspective the uncertainty in our model is
expressed through a posterior distribution over w. A frequentist treatment, however,
involves making a point estimate of wbased on the data set D, and tries instead
to interpret the uncertainty of this estimate through the following thought experi-
ment. Suppose we had a large number of data sets each of size Nand each drawn

independently from the distribution p(t,x). For any given data set D, we can run
our learning algorithm and obtain a prediction function y(x;D). Different data sets
from the ensemble will give different functions and consequently different values ofthe squared loss. The performance of a particular learning algorithm is then assessed
by taking the average over this ensemble of data sets.
Consider the integrand of the ﬁrst term in (3.37), which for a particular data set
Dtakes the form
{y(x;D)−h(x)}2. (3.38)

Because this quantity will be dependent on the particular data set D, we take its aver-
age over the ensemble of data sets. If we add and subtract the quantity ED[y(x;D)]

3.2. The Bias-Variance Decomposition 149
inside the braces, and then expand, we obtain
{y(x;D)−ED[y(x;D)] + ED[y(x;D)]−h(x)}2
={y(x;D)−ED[y(x;D)]}2+{ED[y(x;D)]−h(x)}2
+2{y(x;D)−ED[y(x;D)]}{ED[y(x;D)]−h(x)}. (3.39)
We now take the expectation of this expression with respect to Dand note that the
ﬁnal term will vanish, giving
ED[
{y(x;D)−h(x)}2]
={ED[y(x;D)]−h(x)}2
 
(bias)2+ED[
{y(x;D)−ED[y(x;D)]}2]
  
variance.(3.40)
We see that the expected squared difference between y(x;D)and the regression
function h(x)can be expressed as the sum of two terms. The ﬁrst term, called the
squared bias, represents the extent to which the average prediction over all data sets
differs from the desired regression function. The second term, called the variance ,
measures the extent to which the solutions for individual data sets vary around their
average, and hence this measures the extent to which the function y(x;D)is sensitive
to the particular choice of data set. We shall provide some intuition to support these

deﬁnitions shortly when we consider a simple example.
So far, we have considered a single input value x. If we substitute this expansion
back into (3.37), we obtain the following decomposition of the expected squared loss
expected loss =(bias)2+variance +noise (3.41)
where
(bias)2=∫
{ED[y(x;D)]−h(x)}2p(x)dx (3.42)
variance =∫
ED[
p(x)dx (3.43)
noise =∫
{h(x)−t}2p(x,t)dxdt (3.44)
and the bias and variance terms now refer to integrated quantities.

where
(bias)2=∫
ED[
noise =∫
Our goal is to minimize the expected loss, which we have decomposed into the
sum of a (squared) bias, a variance, and a constant noise term. As we shall see, there
is a trade-off between bias and variance, with very ﬂexible models having low bias
and high variance, and relatively rigid models having high bias and low variance.The model with the optimal predictive capability is the one that leads to the best
balance between bias and variance. This is illustrated by considering the sinusoidal

data set from Chapter 1. Here we generate 100 data sets, each containing N=2 5 Appendix A
data points, independently from the sinusoidal curve h(x)=s i n ( 2 πx). The data
sets are indexed by l=1,...,L , where L= 100 , and for each data set D
(l)we

150 3. LINEAR MODELS FOR REGRESSION
xtlnλ=2.6
0 1−101
xt
0 1−101
xtlnλ=−0.31
0 1−101
xt
0 1−101
xtlnλ=−2.4
0 1−101
xt
0 1−101
Figure 3.5 Illustration of the dependence of bias and variance on model complexity, governed by a regulariza-
tion parameter λ, using the sinusoidal data set from Chapter 1. There are L= 100 data sets, each having N=2 5
data points, and there are 24Gaussian basis functions in the model so that the total number of parameters is
M=2 5 including the bias parameter. The left column shows the result of ﬁtting the model to the data sets for
various values of lnλ(for clarity, only 20 of the 100 ﬁts are shown). The right column shows the corresponding
average of the 100 ﬁts (red) along with the sinusoidal function from which the data sets were generated (green).

3.2. The Bias-Variance Decomposition 151
Figure 3.6 Plot of squared bias and variance,
together with their sum, correspond-
ing to the results shown in Fig-
ure 3.5. Also shown is the average
test set error for a test data set size
of 1000 points. The minimum value
of(bias)2+ variance occurs around
lnλ=−0.31, which is close to the
value that gives the minimum error
on the test data.
lnλ−3 −2 −1 0 1 200.030.060.090.120.15
(bias)2
variance
(bias)2 + variance
test error
ﬁt a model with 24Gaussian basis functions by minimizing the regularized error
function (3.27) to give a prediction function y(l)(x)as shown in Figure 3.5. The
top row corresponds to a large value of the regularization coefﬁcient λthat gives low
variance (because the red curves in the left plot look similar) but high bias (because
the two curves in the right plot are very different). Conversely on the bottom row, for
which λis small, there is large variance (shown by the high variability between the

red curves in the left plot) but low bias (shown by the good ﬁt between the average
model ﬁt and the original sinusoidal function). Note that the result of averaging many
solutions for the complex model with M=2 5 is a very good ﬁt to the regression
function, which suggests that averaging may be a beneﬁcial procedure. Indeed, a
weighted averaging of multiple solutions lies at the heart of a Bayesian approach,
although the averaging is with respect to the posterior distribution of parameters, not

with respect to multiple data sets.
We can also examine the bias-variance trade-off quantitatively for this example.
The average prediction is estimated from
y(x)=1
LL∑
l=1y(l)(x) (3.45)
and the integrated squared bias and integrated variance are then given by
(bias)2=1
NN∑
n=1{y(xn)−h(xn)}2(3.46)
variance =1
NN∑
n=11
LL∑
l=1{
y(l)(xn)−y(xn)}2(3.47)
where the integral over xweighted by the distribution p(x)is approximated by a
ﬁnite sum over data points drawn from that distribution. These quantities, along

y(x)=1
LL∑
(bias)2=1
NN∑
NN∑
n=11
LL∑
l=1{
with their sum, are plotted as a function of lnλin Figure 3.6. We see that small
values of λallow the model to become ﬁnely tuned to the noise on each individual

152 3. LINEAR MODELS FOR REGRESSION
data set leading to large variance. Conversely, a large value of λpulls the weight
parameters towards zero leading to large bias.
Although the bias-variance decomposition may provide some interesting in-
sights into the model complexity issue from a frequentist perspective, it is of lim-
ited practical value, because the bias-variance decomposition is based on averageswith respect to ensembles of data sets, whereas in practice we have only the single
observed data set. If we had a large number of independent training sets of a given
size, we would be better off combining them into a single large training set, whichof course would reduce the level of over-ﬁtting for a given model complexity.
Given these limitations, we turn in the next section to a Bayesian treatment of
linear basis function models, which not only provides powerful insights into the
issues of over-ﬁtting but which also leads to practical techniques for addressing the
question model complexity.

3.3. Bayesian Linear Regression
In our discussion of maximum likelihood for setting the parameters of a linear re-
gression model, we have seen that the effective model complexity, governed by the
number of basis functions, needs to be controlled according to the size of the data
set. Adding a regularization term to the log likelihood function means the effective
model complexity can then be controlled by the value of the regularization coefﬁ-cient, although the choice of the number and form of the basis functions is of course

still important in determining the overall behaviour of the model.
This leaves the issue of deciding the appropriate model complexity for the par-
ticular problem, which cannot be decided simply by maximizing the likelihood func-
tion, because this always leads to excessively complex models and over-ﬁtting. In-
dependent hold-out data can be used to determine model complexity, as discussedin Section 1.3, but this can be both computationally expensive and wasteful of valu-

able data. We therefore turn to a Bayesian treatment of linear regression, which will
avoid the over-ﬁtting problem of maximum likelihood, and which will also lead toautomatic methods of determining model complexity using the training data alone.
Again, for simplicity we will focus on the case of a single target variable t. Ex-
tension to multiple target variables is straightforward and follows the discussion ofSection 3.1.5.
3.3.1 Parameter distribution
We begin our discussion of the Bayesian treatment of linear regression by in-

troducing a prior probability distribution over the model parameters w. For the mo-
ment, we shall treat the noise precision parameter βas a known constant. First note
that the likelihood function p(t|w)deﬁned by (3.10) is the exponential of a quadratic
function of w. The corresponding conjugate prior is therefore given by a Gaussian
p(w)=N(w|m0,S0) (3.48)
having mean m0and covariance S0.

3.3. Bayesian Linear Regression 153
Next we compute the posterior distribution, which is proportional to the product
of the likelihood function and the prior. Due to the choice of a conjugate Gaus-sian prior distribution, the posterior will also be Gaussian. We can evaluate this
distribution by the usual procedure of completing the square in the exponential, and
then ﬁnding the normalization coefﬁcient using the standard result for a normalizedGaussian. However, we have already done the necessary work in deriving the gen- Exercise 3.7
eral result (2.116), which allows us to write down the posterior distribution directly
in the form
p(w|t)=N(w|m
N,SN) (3.49)
where
mN=SN(
S−1
0m0+βΦTt)
(3.50)
S−1
N=S−1
0+βΦTΦ. (3.51)
Note that because the posterior distribution is Gaussian, its mode coincides with its
mean. Thus the maximum posterior weight vector is simply given by wMAP=mN.
If we consider an inﬁnitely broad prior S0=α−1Iwithα→0, the mean mN

in the form
where
mN=SN(
S−1
0m0+βΦTt)
(3.50)
S−1
N=S−1
of the posterior distribution reduces to the maximum likelihood value wMLgiven
by (3.15). Similarly, if N=0, then the posterior distribution reverts to the prior.
Furthermore, if data points arrive sequentially, then the posterior distribution at any
stage acts as the prior distribution for the subsequent data point, such that the newposterior distribution is again given by (3.49). Exercise 3.8
For the remainder of this chapter, we shall consider a particular form of Gaus-

sian prior in order to simplify the treatment. Speciﬁcally, we consider a zero-meanisotropic Gaussian governed by a single precision parameter αso that
p(w|α)=N(w|0,α
−1I) (3.52)
and the corresponding posterior distribution over wis then given by (3.49) with
mN=βSNΦTt (3.53)
S−1
N=αI+βΦTΦ. (3.54)
The log of the posterior distribution is given by the sum of the log likelihood and
the log of the prior and, as a function of w, takes the form
lnp(w|t)=−β
2N∑
n=1{tn−wTφ(xn)}2−α
2wTw+c o n s t . (3.55)

S−1
2N∑
Maximization of this posterior distribution with respect to wis therefore equiva-
lent to the minimization of the sum-of-squares error function with the addition of a
quadratic regularization term, corresponding to (3.27) with λ=α/β.
We can illustrate Bayesian learning in a linear basis function model, as well as
the sequential update of a posterior distribution, using a simple example involving
straight-line ﬁtting. Consider a single input variable x, a single target variable tand

154 3. LINEAR MODELS FOR REGRESSION
a linear model of the form y(x,w)=w0+w1x. Because this has just two adap-
tive parameters, we can plot the prior and posterior distributions directly in parameterspace. We generate synthetic data from the function f(x,a)=a
0+a1xwith param-
eter values a0=−0.3anda1=0.5by ﬁrst choosing values of xnfrom the uniform
distribution U(x|−1,1), then evaluating f(xn,a), and ﬁnally adding Gaussian noise
with standard deviation of 0.2to obtain the target values tn. Our goal is to recover
the values of a0anda1from such data, and we will explore the dependence on the
size of the data set. We assume here that the noise variance is known and hence weset the precision parameter to its true value β=( 1/0.2)
2=2 5 . Similarly, we ﬁx
the parameter αto2.0. We shall shortly discuss strategies for determining αand
βfrom the training data. Figure 3.7 shows the results of Bayesian learning in this
model as the size of the data set is increased and demonstrates the sequential nature

of Bayesian learning in which the current posterior distribution forms the prior whena new data point is observed. It is worth taking time to study this ﬁgure in detail as
it illustrates several important aspects of Bayesian inference. The ﬁrst row of this
ﬁgure corresponds to the situation before any data points are observed and shows aplot of the prior distribution in wspace together with six samples of the function
y(x,w)in which the values of ware drawn from the prior. In the second row, we

see the situation after observing a single data point. The location (x, t)of the data
point is shown by a blue circle in the right-hand column. In the left-hand column is a
plot of the likelihood function p(t|x,w)for this data point as a function of w. Note
that the likelihood function provides a soft constraint that the line must pass close tothe data point, where close is determined by the noise precision β. For comparison,
the true parameter values a
0=−0.3anda1=0.5used to generate the data set

are shown by a white cross in the plots in the left column of Figure 3.7. When wemultiply this likelihood function by the prior from the top row, and normalize, we
obtain the posterior distribution shown in the middle plot on the second row. Sam-
ples of the regression function y(x,w)obtained by drawing samples of wfrom this
posterior distribution are shown in the right-hand plot. Note that these sample lines

all pass close to the data point. The third row of this ﬁgure shows the effect of ob-serving a second data point, again shown by a blue circle in the plot in the right-hand
column. The corresponding likelihood function for this second data point alone is
shown in the left plot. When we multiply this likelihood function by the posteriordistribution from the second row, we obtain the posterior distribution shown in the
middle plot of the third row. Note that this is exactly the same posterior distribution

as would be obtained by combining the original prior with the likelihood functionfor the two data points. This posterior has now been inﬂuenced by two data points,
and because two points are sufﬁcient to deﬁne a line this already gives a relatively
compact posterior distribution. Samples from this posterior distribution give rise tothe functions shown in red in the third column, and we see that these functions pass
close to both of the data points. The fourth row shows the effect of observing a total

of20data points. The left-hand plot shows the likelihood function for the 20
thdata
point alone, and the middle plot shows the resulting posterior distribution that has
now absorbed information from all 20observations. Note how the posterior is much
sharper than in the third row. In the limit of an inﬁnite number of data points, the

3.3. Bayesian Linear Regression 155
Figure 3.7 Illustration of sequential Bayesian learning for a simple linear model of the form y(x,w)=
w0+w1x. A detailed description of this ﬁgure is given in the text.

156 3. LINEAR MODELS FOR REGRESSION
posterior distribution would become a delta function centred on the true parameter
values, shown by the white cross.
Other forms of prior over the parameters can be considered. For instance, we
can generalize the Gaussian prior to give
p(w|α)=[q
2(α
2)1/q1
Γ(1/q)]M
exp(
−α
2M∑
j=1|wj|q)
(3.56)
in which q=2corresponds to the Gaussian distribution, and only in this case is the
prior conjugate to the likelihood function (3.10). Finding the maximum of the poste-
rior distribution over wcorresponds to minimization of the regularized error function
(3.29). In the case of the Gaussian prior, the mode of the posterior distribution was
equal to the mean, although this will no longer hold if q̸=2.
3.3.2 Predictive distribution
In practice, we are not usually interested in the value of witself but rather in
making predictions of tfor new values of x. This requires that we evaluate the
predictive distribution deﬁned by
p(t|t,α,β)=∫
p(t|w,β)p(w|t,α,β)dw (3.57)

in which tis the vector of target values from the training set, and we have omitted the
corresponding input vectors from the right-hand side of the conditioning statements
to simplify the notation. The conditional distribution p(t|x,w,β)of the target vari-
able is given by (3.8), and the posterior weight distribution is given by (3.49). We
see that (3.57) involves the convolution of two Gaussian distributions, and so making

use of the result (2.115) from Section 8.1.4, we see that the predictive distributiontakes the form Exercise 3.10
p(t|x,t,α,β)=N(t|m
T
Nφ(x),σ2
N(x)) (3.58)
where the variance σ2
N(x)of the predictive distribution is given by
σ2
N(x)=1
β+φ(x)TSNφ(x). (3.59)
The ﬁrst term in (3.59) represents the noise on the data whereas the second term
reﬂects the uncertainty associated with the parameters w. Because the noise process
and the distribution of ware independent Gaussians, their variances are additive.

T
Nφ(x),σ2
σ2
N(x)=1
Note that, as additional data points are observed, the posterior distribution becomesnarrower. As a consequence it can be shown (Qazaz et al. , 1997) that σ
2
N+1(x)⩽
σ2
N(x). In the limit N→∞ , the second term in (3.59) goes to zero, and the variance Exercise 3.11
of the predictive distribution arises solely from the additive noise governed by theparameter β.
As an illustration of the predictive distribution for Bayesian linear regression

2
N+1(x)⩽
σ2
models, let us return to the synthetic sinusoidal data set of Section 1.1. In Figure 3.8,

3.3. Bayesian Linear Regression 157
xt
0 1−101
xt
0 1−101
xt
0 1−101
xt
0 1−101
Figure 3.8 Examples of the predictive distribution (3.58) for a model consisting of 9Gaussian basis functions
of the form (3.4) using the synthetic sinusoidal data set of Section 1.1. See the text for a detailed discussion.
we ﬁt a model comprising a linear combination of Gaussian basis functions to data
sets of various sizes and then look at the corresponding posterior distributions. Here
the green curves correspond to the function sin(2πx)from which the data points
were generated (with the addition of Gaussian noise). Data sets of size N=1,
N=2,N=4, andN=2 5 are shown in the four plots by the blue circles. For
each plot, the red curve shows the mean of the corresponding Gaussian predictive
distribution, and the red shaded region spans one standard deviation either side of
the mean. Note that the predictive uncertainty depends on xand is smallest in the

neighbourhood of the data points. Also note that the level of uncertainty decreases
as more data points are observed.
The plots in Figure 3.8 only show the point-wise predictive variance as a func-
tion of x. In order to gain insight into the covariance between the predictions at
different values of x, we can draw samples from the posterior distribution over w,
and then plot the corresponding functions y(x,w), as shown in Figure 3.9.

158 3. LINEAR MODELS FOR REGRESSION
xt
0 1−101
xt
0 1−101
xt
0 1−101
xt
0 1−101
Figure 3.9 Plots of the function y(x,w)using samples from the posterior distributions over wcorresponding to
the plots in Figure 3.8.
If we used localized basis functions such as Gaussians, then in regions away
from the basis function centres, the contribution from the second term in the predic-
tive variance (3.59) will go to zero, leaving only the noise contribution β−1. Thus,
the model becomes very conﬁdent in its predictions when extrapolating outside the
region occupied by the basis functions, which is generally an undesirable behaviour.
This problem can be avoided by adopting an alternative Bayesian approach to re-
gression known as a Gaussian process. Section 6.4
Note that, if both wandβare treated as unknown, then we can introduce a
conjugate prior distribution p(w,β)that, from the discussion in Section 2.3.6, will
be given by a Gaussian-gamma distribution (Denison et al. , 2002). In this case, the Exercise 3.12

predictive distribution is a Student’s t-distribution. Exercise 3.13

3.3. Bayesian Linear Regression 159
Figure 3.10 The equivalent ker-
nelk(x, x′)for the Gaussian basis
functions in Figure 3.1, shown as
a plot of xversus x′, together with
three slices through this matrix cor-
responding to three different valuesofx. The data set used to generate
this kernel comprised 200 values of
xequally spaced over the interval
(−1,1).
3.3.3 Equivalent kernel
The posterior mean solution (3.53) for the linear basis function model has an in-
teresting interpretation that will set the stage for kernel methods, including Gaussian
processes. If we substitute (3.53) into the expression (3.3), we see that the predictive Chapter 6
mean can be written in the form
y(x,mN)=mT
Nφ(x)=βφ(x)TSNΦTt=N∑
n=1βφ(x)TSNφ(xn)tn (3.60)
whereSNis deﬁned by (3.51). Thus the mean of the predictive distribution at a point
xis given by a linear combination of the training set target variables tn, so that we
can write
y(x,mN)=N∑
n=1k(x,xn)tn (3.61)
where the function
k(x,x′)=βφ(x)TSNφ(x′) (3.62)

can write
is known as the smoother matrix or the equivalent kernel . Regression functions, such
as this, which make predictions by taking linear combinations of the training set
target values are known as linear smoothers . Note that the equivalent kernel depends
on the input values xnfrom the data set because these appear in the deﬁnition of
SN. The equivalent kernel is illustrated for the case of Gaussian basis functions in
Figure 3.10 in which the kernel functions k(x, x′)have been plotted as a function of

x′for three different values of x. We see that they are localized around x, and so the
mean of the predictive distribution at x, given by y(x,mN), is obtained by forming
a weighted combination of the target values in which data points close to xare given
higher weight than points further removed from x. Intuitively, it seems reasonable
that we should weight local evidence more strongly than distant evidence. Note thatthis localization property holds not only for the localized Gaussian basis functions

but also for the nonlocal polynomial and sigmoidal basis functions, as illustrated in
Figure 3.11.

160 3. LINEAR MODELS FOR REGRESSION
Figure 3.11 Examples of equiva-
lent kernels k(x, x′)forx=0
plotted as a function of x′, corre-
sponding (left) to the polynomial ba-
sis functions and (right) to the sig-
moidal basis functions shown in Fig-
ure 3.1. Note that these are local-
ized functions of x′even though the
corresponding basis functions are
nonlocal. −1 0 100.020.04
−1 0 100.020.04
Further insight into the role of the equivalent kernel can be obtained by consid-
ering the covariance between y(x)andy(x′), which is given by
cov[y(x),y(x′) ]=c o v [ φ(x)Tw,wTφ(x′)]
=φ(x)TSNφ(x′)=β−1k(x,x′) (3.63)
where we have made use of (3.49) and (3.62). From the form of the equivalent
kernel, we see that the predictive mean at nearby points will be highly correlated,
whereas for more distant pairs of points the correlation will be smaller.
The predictive distribution shown in Figure 3.8 allows us to visualize the point-
wise uncertainty in the predictions, governed by (3.59). However, by drawing sam-

ples from the posterior distribution over w, and plotting the corresponding model
functions y(x,w)as in Figure 3.9, we are visualizing the joint uncertainty in the
posterior distribution between the yvalues at two (or more) xvalues, as governed by
the equivalent kernel.
The formulation of linear regression in terms of a kernel function suggests an
alternative approach to regression as follows. Instead of introducing a set of basis
functions, which implicitly determines an equivalent kernel, we can instead deﬁne

a localized kernel directly and use this to make predictions for new input vectors x,
given the observed training set. This leads to a practical framework for regression
(and classiﬁcation) called Gaussian processes , which will be discussed in detail in
Section 6.4.
We have seen that the effective kernel deﬁnes the weights by which the training
set target values are combined in order to make a prediction at a new value of x, and
it can be shown that these weights sum to one, in other words
N∑
n=1k(x,xn)=1 (3.64)

N∑
for all values of x. This intuitively pleasing result can easily be proven informally Exercise 3.14
by noting that the summation is equivalent to considering the predictive mean ˆy(x)
for a set of target data in which tn=1 for all n. Provided the basis functions are
linearly independent, that there are more data points than basis functions, and that
one of the basis functions is constant (corresponding to the bias parameter), then it is
clear that we can ﬁt the training data exactly and hence that the predictive mean will

3.4. Bayesian Model Comparison 161
be simplyˆy(x)=1 , from which we obtain (3.64). Note that the kernel function can
be negative as well as positive, so although it satisﬁes a summation constraint, thecorresponding predictions are not necessarily convex combinations of the training
set target variables.
Finally, we note that the equivalent kernel (3.62) satisﬁes an important property
shared by kernel functions in general, namely that it can be expressed in the form an Chapter 6
inner product with respect to a vector ψ(x)of nonlinear functions, so that
k(x,z)=ψ(x)
Tψ(z) (3.65)
where ψ(x)=β1/2S1/2
Nφ(x).
3.4. Bayesian Model Comparison
In Chapter 1, we highlighted the problem of over-ﬁtting as well as the use of cross-
validation as a technique for setting the values of regularization parameters or for
choosing between alternative models. Here we consider the problem of model se-lection from a Bayesian perspective. In this section, our discussion will be very

Nφ(x).
general, and then in Section 3.5 we shall see how these ideas can be applied to the
determination of regularization parameters in linear regression.
As we shall see, the over-ﬁtting associated with maximum likelihood can be
avoided by marginalizing (summing or integrating) over the model parameters in-
stead of making point estimates of their values. Models can then be compared di-rectly on the training data, without the need for a validation set. This allows all

available data to be used for training and avoids the multiple training runs for each
model associated with cross-validation. It also allows multiple complexity parame-ters to be determined simultaneously as part of the training process. For example,
in Chapter 7 we shall introduce the relevance vector machine , which is a Bayesian
model having one complexity parameter for every training data point.
The Bayesian view of model comparison simply involves the use of probabilities

to represent uncertainty in the choice of model, along with a consistent applicationof the sum and product rules of probability. Suppose we wish to compare a set of L
models {M
i}where i=1,...,L . Here a model refers to a probability distribution
over the observed data D. In the case of the polynomial curve-ﬁtting problem, the
distribution is deﬁned over the set of target values t, while the set of input values X
is assumed to be known. Other types of model deﬁne a joint distributions over X

models {M
andt. We shall suppose that the data is generated from one of these models but we Section 1.5.4
are uncertain which one. Our uncertainty is expressed through a prior probability
distribution p(Mi). Given a training set D, we then wish to evaluate the posterior
p(Mi|D)∝p(Mi)p(D|M i). (3.66)
The prior allows us to express a preference for different models. Let us simply
assume that all models are given equal prior probability. The interesting term is

themodel evidence p(D|M i)which expresses the preference shown by the data for

162 3. LINEAR MODELS FOR REGRESSION
different models, and we shall examine this term in more detail shortly. The model
evidence is sometimes also called the marginal likelihood because it can be viewed
as a likelihood function over the space of models, in which the parameters have been
marginalized out. The ratio of model evidences p(D|M i)/p(D|M j)for two models
is known as a Bayes factor (Kass and Raftery, 1995).
Once we know the posterior distribution over models, the predictive distribution
is given, from the sum and product rules, by
p(t|x,D)=L∑
i=1p(t|x,Mi,D)p(Mi|D). (3.67)
This is an example of a mixture distribution in which the overall predictive distribu-
tion is obtained by averaging the predictive distributions p(t|x,Mi,D)of individual
models, weighted by the posterior probabilities p(Mi|D)of those models. For in-
stance, if we have two models that are a-posteriori equally likely and one predicts
a narrow distribution around t=awhile the other predicts a narrow distribution

around t=b, the overall predictive distribution will be a bimodal distribution with
modes at t=aandt=b, not a single model at t=(a+b)/2.
A simple approximation to model averaging is to use the single most probable
model alone to make predictions. This is known as model selection .
For a model governed by a set of parameters w, the model evidence is given,
from the sum and product rules of probability, by
p(D|M i)=∫
p(D|w,Mi)p(w|Mi)dw. (3.68)

p(D|M i)=∫
From a sampling perspective, the marginal likelihood can be viewed as the proba- Chapter 11
bility of generating the data set Dfrom a model whose parameters are sampled at
random from the prior. It is also interesting to note that the evidence is precisely the
normalizing term that appears in the denominator in Bayes’ theorem when evaluatingthe posterior distribution over parameters because
p(w|D,M
i)=p(D|w,Mi)p(w|Mi)
p(D|M i). (3.69)
We can obtain some insight into the model evidence by making a simple approx-

p(w|D,M
imation to the integral over parameters. Consider ﬁrst the case of a model having a
single parameter w. The posterior distribution over parameters is proportional to
p(D|w)p(w), where we omit the dependence on the model Mito keep the notation
uncluttered. If we assume that the posterior distribution is sharply peaked around the
most probable value wMAP, with width ∆wposterior , then we can approximate the in-
tegral by the value of the integrand at its maximum times the width of the peak. If we
further assume that the prior is ﬂat with width ∆wprior so that p(w)=1/∆wprior,
then we have

p(D)=∫
p(D|w)p(w)dw≃p(D|wMAP)∆wposterior
∆wprior(3.70)

3.4. Bayesian Model Comparison 163
Figure 3.12 We can obtain a rough approximation to
the model evidence if we assume that
the posterior distribution over parame-
ters is sharply peaked around its mode
wMAP.∆wposterior
∆wpriorwMAP w
and so taking logs we obtain
lnp(D)≃lnp(D|wMAP)+l n(∆wposterior
∆wprior)
. (3.71)
This approximation is illustrated in Figure 3.12. The ﬁrst term represents the ﬁt to
the data given by the most probable parameter values, and for a ﬂat prior this would
correspond to the log likelihood. The second term penalizes the model according to
its complexity. Because ∆wposterior <∆wprior this term is negative, and it increases
in magnitude as the ratio ∆wposterior /∆wprior gets smaller. Thus, if parameters are
ﬁnely tuned to the data in the posterior distribution, then the penalty term is large.
For a model having a set of Mparameters, we can make a similar approximation
for each parameter in turn. Assuming that all parameters have the same ratio of
∆wposterior /∆wprior, we obtain

lnp(D)≃lnp(D|wMAP)+Mln(∆wposterior
∆wprior)
. (3.72)
Thus, in this very simple approximation, the size of the complexity penalty increases
linearly with the number Mof adaptive parameters in the model. As we increase
the complexity of the model, the ﬁrst term will typically decrease, because a more
complex model is better able to ﬁt the data, whereas the second term will increase
due to the dependence on M. The optimal model complexity, as determined by
the maximum evidence, will be given by a trade-off between these two competing

∆wprior)
. (3.72)
terms. We shall later develop a more reﬁned version of this approximation, based on
a Gaussian approximation to the posterior distribution. Section 4.4.1
We can gain further insight into Bayesian model comparison and understand
how the marginal likelihood can favour models of intermediate complexity by con-
sidering Figure 3.13. Here the horizontal axis is a one-dimensional representation
of the space of possible data sets, so that each point on this axis corresponds to a

speciﬁc data set. We now consider three models M1,M2andM3of successively
increasing complexity. Imagine running these models generatively to produce exam-
ple data sets, and then looking at the distribution of data sets that result. Any given

164 3. LINEAR MODELS FOR REGRESSION
Figure 3.13 Schematic illustration of the
distribution of data sets for
three models of different com-
plexity, in which M1is the
simplest and M3is the most
complex. Note that the dis-
tributions are normalized. In
this example, for the partic-
ular observed data set D0,
the model M2with intermedi-
ate complexity has the largest
evidence.p(D)
DD0M1
M2
M3
model can generate a variety of different data sets since the parameters are governed
by a prior probability distribution, and for any choice of the parameters there may
be random noise on the target variables. To generate a particular data set from a spe-
ciﬁc model, we ﬁrst choose the values of the parameters from their prior distribution
p(w), and then for these parameter values we sample the data from p(D|w). A sim-
ple model (for example, based on a ﬁrst order polynomial) has little variability and
so will generate data sets that are fairly similar to each other. Its distribution p(D)

is therefore conﬁned to a relatively small region of the horizontal axis. By contrast,
a complex model (such as a ninth order polynomial) can generate a great variety of
different data sets, and so its distribution p(D)is spread over a large region of the
space of data sets. Because the distributions p(D|M i)are normalized, we see that
the particular data set D0can have the highest value of the evidence for the model
of intermediate complexity. Essentially, the simpler model cannot ﬁt the data well,

whereas the more complex model spreads its predictive probability over too broad a
range of data sets and so assigns relatively small probability to any one of them.
Implicit in the Bayesian model comparison framework is the assumption that
the true distribution from which the data are generated is contained within the set of
models under consideration. Provided this is so, we can show that Bayesian model
comparison will on average favour the correct model. To see this, consider two

models M1andM2in which the truth corresponds to M1. For a given ﬁnite data
set, it is possible for the Bayes factor to be larger for the incorrect model. However, if
we average the Bayes factor over the distribution of data sets, we obtain the expected
Bayes factor in the form
∫
p(D|M 1)l np(D|M 1)
p(D|M 2)dD (3.73)
where the average has been taken with respect to the true distribution of the data.
This quantity is an example of the Kullback-Leibler divergence and satisﬁes the prop- Section 1.6.1

∫
erty of always being positive unless the two distributions are equal in which case it
is zero. Thus on average the Bayes factor will always favour the correct model.
We have seen that the Bayesian framework avoids the problem of over-ﬁtting
and allows models to be compared on the basis of the training data alone. However,

3.5. The Evidence Approximation 165
a Bayesian approach, like any approach to pattern recognition, needs to make as-
sumptions about the form of the model, and if these are invalid then the results canbe misleading. In particular, we see from Figure 3.12 that the model evidence can
be sensitive to many aspects of the prior, such as the behaviour in the tails. Indeed,
the evidence is not deﬁned if the prior is improper, as can be seen by noting thatan improper prior has an arbitrary scaling factor (in other words, the normalization
coefﬁcient is not deﬁned because the distribution cannot be normalized). If we con-
sider a proper prior and then take a suitable limit in order to obtain an improper prior(for example, a Gaussian prior in which we take the limit of inﬁnite variance) then
the evidence will go to zero, as can be seen from (3.70) and Figure 3.12. It may,
however, be possible to consider the evidence ratio between two models ﬁrst and
then take a limit to obtain a meaningful answer.

In a practical application, therefore, it will be wise to keep aside an independent
test set of data on which to evaluate the overall performance of the ﬁnal system.
3.5. The Evidence Approximation
In a fully Bayesian treatment of the linear basis function model, we would intro-duce prior distributions over the hyperparameters αandβand make predictions by
marginalizing with respect to these hyperparameters as well as with respect to the
parameters w. However, although we can integrate analytically over either wor

over the hyperparameters, the complete marginalization over all of these variablesis analytically intractable. Here we discuss an approximation in which we set the
hyperparameters to speciﬁc values determined by maximizing the marginal likeli-
hood function obtained by ﬁrst integrating over the parameters w. This framework
is known in the statistics literature as empirical Bayes (Bernardo and Smith, 1994;
Gelman et al. , 2004), or type 2 maximum likelihood (Berger, 1985), or generalized
maximum likelihood (Wahba, 1975), and in the machine learning literature is also

called the evidence approximation (Gull, 1989; MacKay, 1992a).
If we introduce hyperpriors over αandβ, the predictive distribution is obtained
by marginalizing over w,αandβso that
p(t|t)=∫∫∫
p(t|w,β)p(w|t,α,β)p(α,β|t)dwdαdβ (3.74)
where p(t|w,β)is given by (3.8) and p(w|t,α,β)is given by (3.49) with mNand
SNdeﬁned by (3.53) and (3.54) respectively. Here we have omitted the dependence
on the input variable xto keep the notation uncluttered. If the posterior distribution
p(α,β|t)is sharply peaked around values ˆαandˆβ, then the predictive distribution is

obtained simply by marginalizing over win which αandβare ﬁxed to the values ˆα
andˆβ, so that
p(t|t)≃p(t|t,ˆα,ˆβ)=∫
p(t|w,ˆβ)p(w|t,ˆα,ˆβ)dw. (3.75)

166 3. LINEAR MODELS FOR REGRESSION
From Bayes’ theorem, the posterior distribution for αandβis given by
p(α,β|t)∝p(t|α,β)p(α,β). (3.76)
If the prior is relatively ﬂat, then in the evidence framework the values of ˆαand
ˆβare obtained by maximizing the marginal likelihood function p(t|α,β). We shall
proceed by evaluating the marginal likelihood for the linear basis function model and
then ﬁnding its maxima. This will allow us to determine values for these hyperpa-rameters from the training data alone, without recourse to cross-validation. Recall
that the ratio α/β is analogous to a regularization parameter.
As an aside it is worth noting that, if we deﬁne conjugate (Gamma) prior distri-
butions over αandβ, then the marginalization over these hyperparameters in (3.74)
can be performed analytically to give a Student’s t-distribution over w(see Sec-
tion 2.3.7). Although the resulting integral over wis no longer analytically tractable,

it might be thought that approximating this integral, for example using the Laplace
approximation discussed (Section 4.4) which is based on a local Gaussian approxi-
mation centred on the mode of the posterior distribution, might provide a practical
alternative to the evidence framework (Buntine and Weigend, 1991). However, the
integrand as a function of wtypically has a strongly skewed mode so that the Laplace
approximation fails to capture the bulk of the probability mass, leading to poorer re-
sults than those obtained by maximizing the evidence (MacKay, 1999).

Returning to the evidence framework, we note that there are two approaches that
we can take to the maximization of the log evidence. We can evaluate the evidence
function analytically and then set its derivative equal to zero to obtain re-estimation
equations for αandβ, which we shall do in Section 3.5.2. Alternatively we use a
technique called the expectation maximization (EM) algorithm, which will be dis-
cussed in Section 9.3.4 where we shall also show that these two approaches converge
to the same solution.

3.5.1 Evaluation of the evidence function
The marginal likelihood function p(t|α,β)is obtained by integrating over the
weight parameters w, so that
p(t|α,β)=∫
p(t|w,β)p(w|α)dw. (3.77)
One way to evaluate this integral is to make use once again of the result (2.115)
for the conditional distribution in a linear-Gaussian model. Here we shall evaluate Exercise 3.16
the integral instead by completing the square in the exponent and making use of the
standard form for the normalization coefﬁcient of a Gaussian.

From (3.11), (3.12), and (3.52), we can write the evidence function in the form Exercise 3.17
p(t|α,β)=(β
2π)N/2(α
2π)M/2∫
exp{−E(w)}dw (3.78)

3.5. The Evidence Approximation 167
where Mis the dimensionality of w, and we have deﬁned
E(w)= βED(w)+αEW(w)
=β
2∥t−Φw∥2+α
2wTw. (3.79)
We recognize (3.79) as being equal, up to a constant of proportionality, to the reg-
ularized sum-of-squares error function (3.27). We now complete the square over w Exercise 3.18
giving
E(w)=E(mN)+1
2(w−mN)TA(w−mN) (3.80)
where we have introduced
A=αI+βΦTΦ (3.81)
together with
E(mN)=β
2∥t−Φm N∥2+α
2mT
NmN. (3.82)
Note that Acorresponds to the matrix of second derivatives of the error function
A=∇∇E(w) (3.83)
and is known as the Hessian matrix . Here we have also deﬁned mNgiven by
mN=βA−1ΦTt. (3.84)
Using (3.54), we see that A=S−1
N, and hence (3.84) is equivalent to the previous
deﬁnition (3.53), and therefore represents the mean of the posterior distribution.
The integral over wcan now be evaluated simply by appealing to the standard
result for the normalization coefﬁcient of a multivariate Gaussian, giving Exercise 3.19
∫
exp{−E(w)}dw
=e x p {−E(mN)}∫
exp{
−1

∫
exp{
−1
2(w−mN)TA(w−mN)}
dw
=e x p {−E(mN)}(2π)M/2|A|−1/2. (3.85)
Using (3.78) we can then write the log of the marginal likelihood in the form
lnp(t|α,β)=M
2lnα+N
2lnβ−E(mN)−1
2ln|A|−N
2ln(2π) (3.86)
which is the required expression for the evidence function.
Returning to the polynomial regression problem, we can plot the model evidence
against the order of the polynomial, as shown in Figure 3.14. Here we have assumed
a prior of the form (1.65) with the parameter αﬁxed at α=5×10−3. The form

exp{
−1
dw
2lnα+N
2ln|A|−N
of this plot is very instructive. Referring back to Figure 1.4, we see that the M=0
polynomial has very poor ﬁt to the data and consequently gives a relatively low value

168 3. LINEAR MODELS FOR REGRESSION
Figure 3.14 Plot of the model evidence versus
the order M, for the polynomial re-
gression model, showing that the
evidence favours the model with
M=3.
M0 2 4 6 8−26−24−22−20−18
for the evidence. Going to the M=1polynomial greatly improves the data ﬁt, and
hence the evidence is signiﬁcantly higher. However, in going to M=2, the data
ﬁt is improved only very marginally, due to the fact that the underlying sinusoidal
function from which the data is generated is an odd function and so has no even terms
in a polynomial expansion. Indeed, Figure 1.5 shows that the residual data error is
reduced only slightly in going from M=1 toM=2. Because this richer model
suffers a greater complexity penalty, the evidence actually falls in going from M=1
toM=2. When we go to M=3 we obtain a signiﬁcant further improvement in
data ﬁt, as seen in Figure 1.4, and so the evidence is increased again, giving the
highest overall evidence for any of the polynomials. Further increases in the value

ofMproduce only small improvements in the ﬁt to the data but suffer increasing
complexity penalty, leading overall to a decrease in the evidence values. Looking
again at Figure 1.5, we see that the generalization error is roughly constant between
M=3 andM=8, and it would be difﬁcult to choose between these models on
the basis of this plot alone. The evidence values, however, show a clear preference
forM=3, since this is the simplest model which gives a good explanation for the
observed data.
3.5.2 Maximizing the evidence function

Let us ﬁrst consider the maximization of p(t|α,β)with respect to α. This can
be done by ﬁrst deﬁning the following eigenvector equation
(
βΦTΦ)
ui=λiui. (3.87)
From (3.81), it then follows that Ahas eigenvalues α+λi. Now consider the deriva-
tive of the term involving ln|A|in (3.86) with respect to α.W eh a v e
d
dαln|A|=d
dαln∏
i(λi+α)=d
dα∑
iln(λi+α)=∑
i1
λi+α.(3.88)
Thus the stationary points of (3.86) with respect to αsatisfy
0=M
2α−1
2mT
NmN−1
2∑
i1
λi+α. (3.89)

3.5. The Evidence Approximation 169
Multiplying through by 2αand rearranging, we obtain
αmT
NmN=M−α∑
i1
λi+α=γ. (3.90)
Since there are Mterms in the sum over i, the quantity γcan be written
γ=∑
iλi
α+λi. (3.91)
The interpretation of the quantity γwill be discussed shortly. From (3.90) we see
that the value of αthat maximizes the marginal likelihood satisﬁes Exercise 3.20
α=γ
mT
NmN. (3.92)
Note that this is an implicit solution for αnot only because γdepends on α, but also
because the mode mNof the posterior distribution itself depends on the choice of
α. We therefore adopt an iterative procedure in which we make an initial choice for
αand use this to ﬁnd mN, which is given by (3.53), and also to evaluate γ, which
is given by (3.91). These values are then used to re-estimate αusing (3.92), and the
process repeated until convergence. Note that because the matrix ΦTΦis ﬁxed, we
can compute its eigenvalues once at the start and then simply multiply these by βto
obtain the λi.

It should be emphasized that the value of αhas been determined purely by look-
ing at the training data. In contrast to maximum likelihood methods, no independent
data set is required in order to optimize the model complexity.
We can similarly maximize the log marginal likelihood (3.86) with respect to β.
To do this, we note that the eigenvalues λideﬁned by (3.87) are proportional to β,
and hence dλi/dβ=λi/βgiving
d
dβln|A|=d
dβ∑
iln(λi+α)=1
β∑
iλi
λi+α=γ
β. (3.93)

d
dβln|A|=d
dβ∑
β∑
iλi
λi+α=γ
β. (3.93)
The stationary point of the marginal likelihood therefore satisﬁes
0=N
2β−1
2N∑
n=1{
tn−mT
Nφ(xn)}2−γ
2β(3.94)
and rearranging we obtain Exercise 3.22
1
β=1
N−γN∑
n=1{
tn−mT
Nφ(xn)}2. (3.95)
Again, this is an implicit solution for βand can be solved by choosing an initial
value for βand then using this to calculate mNandγand then re-estimate βusing
(3.95), repeating until convergence. If both αandβare to be determined from the
data, then their values can be re-estimated together after each update of γ.

170 3. LINEAR MODELS FOR REGRESSION
Figure 3.15 Contours of the likelihood function (red)
and the prior (green) in which the axes in parameter
space have been rotated to align with the eigenvectors
uiof the Hessian. For α=0, the mode of the poste-
rior is given by the maximum likelihood solution wML,
whereas for nonzero αthe mode is at wMAP=mN.I n
the direction w1the eigenvalue λ1, deﬁned by (3.87), is
small compared with αand so the quantity λ1/(λ1+α)
is close to zero, and the corresponding MAP value of
w1is also close to zero. By contrast, in the direction w2
the eigenvalue λ2is large compared with αand so the
quantity λ2/(λ2+α)is close to unity, and the MAP value
ofw2is close to its maximum likelihood value.u1u2
w1w2
wMAPwML
3.5.3 Effective number of parameters
The result (3.92) has an elegant interpretation (MacKay, 1992a), which provides
insight into the Bayesian solution for α. To see this, consider the contours of the like-

w1w2
wMAPwML
lihood function and the prior as illustrated in Figure 3.15. Here we have implicitly
transformed to a rotated set of axes in parameter space aligned with the eigenvec-
torsuideﬁned in (3.87). Contours of the likelihood function are then axis-aligned
ellipses. The eigenvalues λimeasure the curvature of the likelihood function, and
so in Figure 3.15 the eigenvalue λ1is small compared with λ2(because a smaller
curvature corresponds to a greater elongation of the contours of the likelihood func-

tion). Because βΦTΦis a positive deﬁnite matrix, it will have positive eigenvalues,
and so the ratio λi/(λi+α)will lie between 0 and 1. Consequently, the quantity γ
deﬁned by (3.91) will lie in the range 0⩽γ⩽M. For directions in which λi≫α,
the corresponding parameter wiwill be close to its maximum likelihood value, and
the ratio λi/(λi+α)will be close to 1. Such parameters are called well determined
because their values are tightly constrained by the data. Conversely, for directions

in which λi≪α, the corresponding parameters wiwill be close to zero, as will the
ratiosλi/(λi+α). These are directions in which the likelihood function is relatively
insensitive to the parameter value and so the parameter has been set to a small value
by the prior. The quantity γdeﬁned by (3.91) therefore measures the effective total
number of well determined parameters.
We can obtain some insight into the result (3.95) for re-estimating βby com-
paring it with the corresponding maximum likelihood result given by (3.21). Both

of these formulae express the variance (the inverse precision) as an average of the
squared differences between the targets and the model predictions. However, they
differ in that the number of data points Nin the denominator of the maximum like-
lihood result is replaced by N−γin the Bayesian result. We recall from (1.56) that
the maximum likelihood estimate of the variance for a Gaussian distribution over a

3.5. The Evidence Approximation 171
single variable xis given by
σ2
ML=1
NN∑
n=1(xn−µML)2(3.96)
and that this estimate is biased because the maximum likelihood solution µMLfor
the mean has ﬁtted some of the noise on the data. In effect, this has used up onedegree of freedom in the model. The corresponding unbiased estimate is given by
(1.59) and takes the form
σ
2
MAP=1
N−1N∑
n=1(xn−µML)2. (3.97)
We shall see in Section 10.1.3 that this result can be obtained from a Bayesian treat-
ment in which we marginalize over the unknown mean. The factor of N−1in the
denominator of the Bayesian result takes account of the fact that one degree of free-dom has been used in ﬁtting the mean and removes the bias of maximum likelihood.
Now consider the corresponding results for the linear regression model. The mean
of the target distribution is now given by the function w
Tφ(x), which contains M
parameters. However, not all of these parameters are tuned to the data. The effective

number of parameters that are determined by the data is γ, with the remaining M−γ
parameters set to small values by the prior. This is reﬂected in the Bayesian resultfor the variance that has a factor N−γin the denominator, thereby correcting for
the bias of the maximum likelihood result.
We can illustrate the evidence framework for setting hyperparameters using the
sinusoidal synthetic data set from Section 1.1, together with the Gaussian basis func-

tion model comprising 9basis functions, so that the total number of parameters in
the model is given by M=1 0 including the bias. Here, for simplicity of illustra-
tion, we have set βto its true value of 11.1and then used the evidence framework to
determine α, as shown in Figure 3.16.
We can also see how the parameter αcontrols the magnitude of the parameters
{w
i}, by plotting the individual parameters versus the effective number γof param-
eters, as shown in Figure 3.17.
If we consider the limit N≫Min which the number of data points is large in

{w
relation to the number of parameters, then from (3.87) all of the parameters will be
well determined by the data because ΦTΦinvolves an implicit sum over data points,
and so the eigenvalues λiincrease with the size of the data set. In this case, γ=M,
and the re-estimation equations for αandβbecome
α=M
2EW(mN)(3.98)
β=N
2ED(mN)(3.99)
where EWandEDare deﬁned by (3.25) and (3.26), respectively. These results
can be used as an easy-to-compute approximation to the full evidence re-estimation

172 3. LINEAR MODELS FOR REGRESSION
lnα−5 0 5
lnα−5 0 5
Figure 3.16 The left plot shows γ(red curve) and 2αEW(mN)(blue curve) versus lnαfor the sinusoidal
synthetic data set. It is the intersection of these two curves that deﬁnes the optimum value for αgiven by the
evidence procedure. The right plot shows the corresponding graph of log evidence lnp(t|α, β)versus lnα(red
curve) showing that the peak coincides with the crossing point of the curves in the left plot. Also shown is the
test set error (blue curve) showing that the evidence maximum occurs close to the point of best generalization.
formulae, because they do not require evaluation of the eigenvalue spectrum of the
Hessian.
Figure 3.17 Plot of the 10parameters wi
from the Gaussian basis function
model versus the effective num-
ber of parameters γ, in which the
hyperparameter αis varied in the
range 0⩽α⩽∞causing γto
vary in the range 0⩽γ⩽M.
9713625480
γwi
0 2 4 6 8 10−2−1012
3.6. Limitations of Fixed Basis Functions

Hessian.
γwi
Throughout this chapter, we have focussed on models comprising a linear combina-
tion of ﬁxed, nonlinear basis functions. We have seen that the assumption of linearity
in the parameters led to a range of useful properties including closed-form solutions
to the least-squares problem, as well as a tractable Bayesian treatment. Furthermore,
for a suitable choice of basis functions, we can model arbitrary nonlinearities in the

Exercises 173
mapping from input variables to targets. In the next chapter, we shall study an anal-
ogous class of models for classiﬁcation.
It might appear, therefore, that such linear models constitute a general purpose
framework for solving problems in pattern recognition. Unfortunately, there are
some signiﬁcant shortcomings with linear models, which will cause us to turn inlater chapters to more complex models such as support vector machines and neural
networks.
The difﬁculty stems from the assumption that the basis functions φ
j(x)are ﬁxed
before the training data set is observed and is a manifestation of the curse of dimen-
sionality discussed in Section 1.4. As a consequence, the number of basis functions
needs to grow rapidly, often exponentially, with the dimensionality Dof the input
space.
Fortunately, there are two properties of real data sets that we can exploit to help
alleviate this problem. First of all, the data vectors {xn}typically lie close to a non-

space.
linear manifold whose intrinsic dimensionality is smaller than that of the input space
as a result of strong correlations between the input variables. We will see an exampleof this when we consider images of handwritten digits in Chapter 12. If we are using
localized basis functions, we can arrange that they are scattered in input space only
in regions containing data. This approach is used in radial basis function networksand also in support vector and relevance vector machines. Neural network models,

which use adaptive basis functions having sigmoidal nonlinearities, can adapt the
parameters so that the regions of input space over which the basis functions varycorresponds to the data manifold. The second property is that target variables may
have signiﬁcant dependence on only a small number of possible directions within the
data manifold. Neural networks can exploit this property by choosing the directionsin input space to which the basis functions respond.
Exercises

Exercises
3.1 (⋆)www Show that the ‘ tanh ’ function and the logistic sigmoid function (3.6)
are related by
tanh(a)=2σ(2a)−1. (3.100)
Hence show that a general linear combination of logistic sigmoid functions of the
form
y(x,w)=w0+M∑
j=1wjσ(x−µj
s)
(3.101)
is equivalent to a linear combination of ‘ tanh ’ functions of the form
y(x,u)=u0+M∑
j=1ujtanh(x−µj
s)
(3.102)
and ﬁnd expressions to relate the new parameters {u1,...,u M}to the original pa-
rameters {w1,...,w M}.

174 3. LINEAR MODELS FOR REGRESSION
3.2 (⋆⋆)Show that the matrix
Φ(ΦTΦ)−1ΦT(3.103)
takes any vector vand projects it onto the space spanned by the columns of Φ. Use
this result to show that the least-squares solution (3.15) corresponds to an orthogonalprojection of the vector tonto the manifold Sas shown in Figure 3.2.
3.3 (⋆)Consider a data set in which each data point t
nis associated with a weighting
factor rn>0, so that the sum-of-squares error function becomes
ED(w)=1
2N∑
n=1rn{
tn−wTφ(xn)}2. (3.104)
Find an expression for the solution w⋆that minimizes this error function. Give two
alternative interpretations of the weighted sum-of-squares error function in terms of
(i) data dependent noise variance and (ii) replicated data points.
3.4 (⋆)www Consider a linear model of the form
i=1wixi (3.105)
together with a sum-of-squares error function of the form
ED(w)=1
2N∑
n=1{y(xn,w)−tn}2. (3.106)
Now suppose that Gaussian noise ϵiwith zero mean and variance σ2is added in-

ED(w)=1
2N∑
dependently to each of the input variables xi. By making use of E[ϵi]=0 and
E[ϵiϵj]=δijσ2, show that minimizing EDaveraged over the noise distribution is
equivalent to minimizing the sum-of-squares error for noise-free input variables with
the addition of a weight-decay regularization term, in which the bias parameter w0
is omitted from the regularizer.
3.5 (⋆)www Using the technique of Lagrange multipliers, discussed in Appendix E,

show that minimization of the regularized error function (3.29) is equivalent to mini-
mizing the unregularized sum-of-squares error (3.12) subject to the constraint (3.30).
Discuss the relationship between the parameters ηandλ.
3.6 (⋆)www Consider a linear basis function regression model for a multivariate
target variable thaving a Gaussian distribution of the form
p(t|W,Σ)=N(t|y(x,W),Σ) (3.107)
where
y(x,W)=WTφ(x) (3.108)

Exercises 175
together with a training data set comprising input basis vectors φ(xn)and corre-
sponding target vectors tn, with n=1,...,N . Show that the maximum likelihood
solution WMLfor the parameter matrix Whas the property that each column is
given by an expression of the form (3.15), which was the solution for an isotropic
noise distribution. Note that this is independent of the covariance matrix Σ. Show
that the maximum likelihood solution for Σis given by
Σ=1
NN∑
n=1(
tn−WT
MLφ(xn))(
tn−WT
MLφ(xn))T. (3.109)
3.7 (⋆)By using the technique of completing the square, verify the result (3.49) for the
posterior distribution of the parameters win the linear basis function model in which
mNandSNare deﬁned by (3.50) and (3.51) respectively.
3.8 (⋆⋆)www Consider the linear basis function model in Section 3.1, and suppose
that we have already observed Ndata points, so that the posterior distribution over
wis given by (3.49). This posterior can be regarded as the prior for the next obser-

tn−WT
vation. By considering an additional data point (xN+1,tN+1), and by completing
the square in the exponential, show that the resulting posterior distribution is againgiven by (3.49) but with S
Nreplaced by SN+1andmNreplaced by mN+1.
3.9 (⋆⋆)Repeat the previous exercise but instead of completing the square by hand,
make use of the general result for linear-Gaussian models given by (2.116).
3.10 (⋆⋆)www By making use of the result (2.115) to evaluate the integral in (3.57),

verify that the predictive distribution for the Bayesian linear regression model is
given by (3.58) in which the input-dependent variance is given by (3.59).
3.11 (⋆⋆)We have seen that, as the size of a data set increases, the uncertainty associated
with the posterior distribution over model parameters decreases. Make use of the
matrix identity (Appendix C)
(
M+vvT)−1=M−1−(M−1v)(
vTM−1)
1+vTM−1v(3.110)
to show that the uncertainty σ2
N(x)associated with the linear regression function
given by (3.59) satisﬁes
σ2
N+1(x)⩽σ2
N(x). (3.111)

(
vTM−1)
σ2
N+1(x)⩽σ2
3.12 (⋆⋆)We saw in Section 2.3.6 that the conjugate prior for a Gaussian distribution
with unknown mean and unknown precision (inverse variance) is a normal-gamma
distribution. This property also holds for the case of the conditional Gaussian dis-
tribution p(t|x,w,β)of the linear regression model. If we consider the likelihood
function (3.10), then the conjugate prior for wandβis given by
p(w,β)=N(w|m0,β−1S0)Gam( β|a0,b0). (3.112)

176 3. LINEAR MODELS FOR REGRESSION
Show that the corresponding posterior distribution takes the same functional form,
so that
p(w,β|t)=N(w|mN,β−1SN)Gam( β|aN,bN) (3.113)
and ﬁnd expressions for the posterior parameters mN,SN,aN, andbN.
3.13 (⋆⋆)Show that the predictive distribution p(t|x,t)for the model discussed in Ex-
ercise 3.12 is given by a Student’s t-distribution of the form
p(t|x,t)=S t ( t|µ, λ, ν ) (3.114)
and obtain expressions for µ,λandν.
3.14 (⋆⋆)In this exercise, we explore in more detail the properties of the equivalent
kernel deﬁned by (3.62), where SNis deﬁned by (3.54). Suppose that the basis
functions φj(x)are linearly independent and that the number Nof data points is
greater than the number Mof basis functions. Furthermore, let one of the basis
functions be constant, say φ0(x)=1 . By taking suitable linear combinations of
these basis functions, we can construct a new basis set ψj(x)spanning the same
space but that are orthonormal, so that
N∑
n=1ψj(xn)ψk(xn)=Ijk (3.115)

N∑
where Ijkis deﬁned to be 1ifj=kand0otherwise, and we take ψ0(x)=1 . Show
that for α=0, the equivalent kernel can be written as k(x,x′)=ψ(x)Tψ(x′)
where ψ=(ψ1,...,ψ M)T. Use this result to show that the kernel satisﬁes the
summation constraint
N∑
n=1k(x,xn)=1. (3.116)
3.15 (⋆)www Consider a linear basis function model for regression in which the pa-
rameters αandβare set using the evidence framework. Show that the function
E(mN)deﬁned by (3.82) satisﬁes the relation 2E(mN)=N.

N∑
N∑
3.16 (⋆⋆)Derive the result (3.86) for the log evidence function p(t|α,β)of the linear
regression model by making use of (2.115) to evaluate the integral (3.77) directly.
3.17 (⋆)Show that the evidence function for the Bayesian linear regression model can
be written in the form (3.78) in which E(w)is deﬁned by (3.79).
3.18 (⋆⋆)www By completing the square over w, show that the error function (3.79)
in Bayesian linear regression can be written in the form (3.80).

3.19 (⋆⋆)Show that the integration over win the Bayesian linear regression model gives
the result (3.85). Hence show that the log marginal likelihood is given by (3.86).

Exercises 177
3.20 (⋆⋆)www Starting from (3.86) verify all of the steps needed to show that maxi-
mization of the log marginal likelihood function (3.86) with respect to αleads to the
re-estimation equation (3.92).
3.21 (⋆⋆)An alternative way to derive the result (3.92) for the optimal value of αin the
evidence framework is to make use of the identity
d
dαln|A|=Tr(
A−1d
dαA)
. (3.117)
Prove this identity by considering the eigenvalue expansion of a real, symmetric
matrix A, and making use of the standard results for the determinant and trace of
Aexpressed in terms of its eigenvalues (Appendix C). Then make use of (3.117) to
derive (3.92) starting from (3.86).
3.22 (⋆⋆)Starting from (3.86) verify all of the steps needed to show that maximiza-
tion of the log marginal likelihood function (3.86) with respect to βleads to the
re-estimation equation (3.95).
3.23 (⋆⋆)www Show that the marginal probability of the data, in other words the
model evidence, for the model described in Exercise 3.12 is given by
p(t)=1

p(t)=1
(2π)N/2ba0
0
baN
NΓ(aN)
Γ(a0)|SN|1/2
|S0|1/2(3.118)
by ﬁrst marginalizing with respect to wand then with respect to β.
3.24 (⋆⋆)Repeat the previous exercise but now use Bayes’ theorem in the form
p(t)=p(t|w,β)p(w,β)
p(w,β|t)(3.119)
and then substitute for the prior and posterior distributions and the likelihood func-
tion in order to derive the result (3.118).

4
Linear
Models for
Classiﬁcation
In the previous chapter, we explored a class of regression models having particularly
simple analytical and computational properties. We now discuss an analogous classof models for solving classiﬁcation problems. The goal in classiﬁcation is to take an
input vector xand to assign it to one of Kdiscrete classes C
kwhere k=1,...,K .
In the most common scenario, the classes are taken to be disjoint, so that each input isassigned to one and only one class. The input space is thereby divided into decision
regions whose boundaries are called decision boundaries ordecision surfaces .I n
this chapter, we consider linear models for classiﬁcation, by which we mean that thedecision surfaces are linear functions of the input vector xand hence are deﬁned
by(D−1)-dimensional hyperplanes within the D-dimensional input space. Data
sets whose classes can be separated exactly by linear decision surfaces are said to belinearly separable .

For regression problems, the target variable twas simply the vector of real num-
bers whose values we wish to predict. In the case of classiﬁcation, there are various
179

180 4. LINEAR MODELS FOR CLASSIFICATION
ways of using target values to represent class labels. For probabilistic models, the
most convenient, in the case of two-class problems, is the binary representation inwhich there is a single target variable t∈{0,1}such that t=1represents class C
1
andt=0represents class C2. We can interpret the value of tas the probability that
the class is C1, with the values of probability taking only the extreme values of 0 and
1. For K>2classes, it is convenient to use a 1-of- Kcoding scheme in which tis
a vector of length Ksuch that if the class is Cj, then all elements tkoftare zero
except element tj, which takes the value 1. For instance, if we have K=5classes,
then a pattern from class 2would be given the target vector
t=( 0,1,0,0,0)T. (4.1)
Again, we can interpret the value of tkas the probability that the class is Ck.F o r
nonprobabilistic models, alternative choices of target variable representation will
sometimes prove convenient.

In Chapter 1, we identiﬁed three distinct approaches to the classiﬁcation prob-
lem. The simplest involves constructing a discriminant function that directly assigns
each vector xto a speciﬁc class. A more powerful approach, however, models the
conditional probability distribution p(Ck|x)in an inference stage, and then subse-
quently uses this distribution to make optimal decisions. By separating inference
and decision, we gain numerous beneﬁts, as discussed in Section 1.5.4. There are
two different approaches to determining the conditional probabilities p(Ck|x). One

technique is to model them directly, for example by representing them as parametric
models and then optimizing the parameters using a training set. Alternatively, we
can adopt a generative approach in which we model the class-conditional densities
given by p(x|Ck), together with the prior probabilities p(Ck)for the classes, and then
we compute the required posterior probabilities using Bayes’ theorem
p(x). (4.2)
We shall discuss examples of all three approaches in this chapter.

In the linear regression models considered in Chapter 3, the model prediction
y(x,w)was given by a linear function of the parameters w. In the simplest case,
the model is also linear in the input variables and therefore takes the form y(x)=
wTx+w0, so that yis a real number. For classiﬁcation problems, however, we wish
to predict discrete class labels, or more generally posterior probabilities that lie inthe range (0,1). To achieve this, we consider a generalization of this model in which

we transform the linear function of wusing a nonlinear function f(·)so that
y(x)=f(
wTx+w0)
. (4.3)
In the machine learning literature f(·)is known as an activation function , whereas
its inverse is called a link function in the statistics literature. The decision surfaces
correspond to y(x) = constant , so that wTx+w0= constant and hence the deci-
sion surfaces are linear functions of x, even if the function f(·)is nonlinear. For this
reason, the class of models described by (4.3) are called generalized linear models

4.1. Discriminant Functions 181
(McCullagh and Nelder, 1989). Note, however, that in contrast to the models used
for regression, they are no longer linear in the parameters due to the presence of thenonlinear function f(·). This will lead to more complex analytical and computa-
tional properties than for linear regression models. Nevertheless, these models are
still relatively simple compared to the more general nonlinear models that will bestudied in subsequent chapters.
The algorithms discussed in this chapter will be equally applicable if we ﬁrst
make a ﬁxed nonlinear transformation of the input variables using a vector of basisfunctions φ(x)as we did for regression models in Chapter 3. We begin by consider-
ing classiﬁcation directly in the original input space x, while in Section 4.3 we shall
ﬁnd it convenient to switch to a notation involving basis functions for consistency
with later chapters.
4.1. Discriminant Functions

A discriminant is a function that takes an input vector xand assigns it to one of K
classes, denoted Ck. In this chapter, we shall restrict attention to linear discriminants ,
namely those for which the decision surfaces are hyperplanes. To simplify the dis-
cussion, we consider ﬁrst the case of two classes and then investigate the extension
toK>2classes.
4.1.1 Two classes
The simplest representation of a linear discriminant function is obtained by tak-
ing a linear function of the input vector so that
y(x)=wTx+w0 (4.4)

wherewis called a weight vector , andw0is a bias (not to be confused with bias in
the statistical sense). The negative of the bias is sometimes called a threshold .A n
input vector xis assigned to class C1ify(x)⩾0and to class C2otherwise. The cor-
responding decision boundary is therefore deﬁned by the relation y(x)=0 , which
corresponds to a (D−1)-dimensional hyperplane within the D-dimensional input
space. Consider two points xAandxBboth of which lie on the decision surface.
Because y(xA)=y(xB)=0 ,w eh a v e wT(xA−xB)=0 and hence the vector wis

orthogonal to every vector lying within the decision surface, and so wdetermines the
orientation of the decision surface. Similarly, if xis a point on the decision surface,
theny(x)=0 , and so the normal distance from the origin to the decision surface is
given by
wTx
∥w∥=−w0
∥w∥. (4.5)
We therefore see that the bias parameter w0determines the location of the decision
surface. These properties are illustrated for the case of D=2in Figure 4.1.
Furthermore, we note that the value of y(x)gives a signed measure of the per-

given by
wTx
∥w∥=−w0
∥w∥. (4.5)
pendicular distance rof the point xfrom the decision surface. To see this, consider

182 4. LINEAR MODELS FOR CLASSIFICATION
Figure 4.1 Illustration of the geometry of a
linear discriminant function in two dimensions.
The decision surface, shown in red, is perpen-
dicular to w, and its displacement from the
origin is controlled by the bias parameter w0.
Also, the signed orthogonal distance of a gen-
eral point xfrom the decision surface is given
byy(x)/∥w∥.x2
x1wx
y(x)
∥w∥
x⊥
−w0
∥w∥y=0
y<0y>0
R2R1
an arbitrary point xand let x⊥be its orthogonal projection onto the decision surface,
so that
x=x⊥+rw
∥w∥. (4.6)
Multiplying both sides of this result by wTand adding w0, and making use of y(x)=
wTx+w0andy(x⊥)=wTx⊥+w0=0,w eh a v e
r=y(x)
∥w∥. (4.7)
This result is illustrated in Figure 4.1.
As with the linear regression models in Chapter 3, it is sometimes convenient
to use a more compact notation in which we introduce an additional dummy ‘input’
valuex0=1and then deﬁne ˜w=(w0,w)and˜x=(x0,x)so that
y(x)=˜wT˜x. (4.8)
In this case, the decision surfaces are D-dimensional hyperplanes passing through

x=x⊥+rw
∥w∥. (4.6)
r=y(x)
∥w∥. (4.7)
the origin of the D+1-dimensional expanded input space.
4.1.2 Multiple classes
Now consider the extension of linear discriminants to K>2classes. We might
be tempted be to build a K-class discriminant by combining a number of two-class
discriminant functions. However, this leads to some serious difﬁculties (Duda and
Hart, 1973) as we now show.
Consider the use of K−1classiﬁers each of which solves a two-class problem of
separating points in a particular class Ckfrom points not in that class. This is known

as a one-versus-the-rest classiﬁer. The left-hand example in Figure 4.2 shows an

4.1. Discriminant Functions 183
R1
R2
R3?
C1
notC1C2
notC2R1
R2R3
?C1
C2C1C3
C2C3
Figure 4.2 Attempting to construct a Kclass discriminant from a set of two class discriminants leads to am-
biguous regions, shown in green. On the left is an example involving the use of two discriminants designed to
distinguish points in class Ckfrom points not in class Ck. On the right is an example involving three discriminant
functions each of which is used to separate a pair of classes CkandCj.
example involving three classes where this approach leads to regions of input space
that are ambiguously classiﬁed.
An alternative is to introduce K(K−1)/2binary discriminant functions, one
for every possible pair of classes. This is known as a one-versus-one classiﬁer. Each
point is then classiﬁed according to a majority vote amongst the discriminant func-
tions. However, this too runs into the problem of ambiguous regions, as illustrated
in the right-hand diagram of Figure 4.2.

We can avoid these difﬁculties by considering a single K-class discriminant
comprising Klinear functions of the form
yk(x)=wT
kx+wk0 (4.9)
and then assigning a point xto class Ckifyk(x)>yj(x)for all j̸=k. The decision
boundary between class Ckand class Cjis therefore given by yk(x)=yj(x)and
hence corresponds to a (D−1)-dimensional hyperplane deﬁned by
(wk−wj)Tx+(wk0−wj0)=0. (4.10)
This has the same form as the decision boundary for the two-class case discussed in
Section 4.1.1, and so analogous geometrical properties apply.

yk(x)=wT
The decision regions of such a discriminant are always singly connected and
convex. To see this, consider two points xAandxBboth of which lie inside decision
region Rk, as illustrated in Figure 4.3. Any point ˆxthat lies on the line connecting
xAandxBcan be expressed in the form
ˆx=λxA+( 1−λ)xB (4.11)

184 4. LINEAR MODELS FOR CLASSIFICATION
Figure 4.3 Illustration of the decision regions for a mul-
ticlass linear discriminant, with the decision
boundaries shown in red. If two points xA
andxBboth lie inside the same decision re-
gionRk, then any point bxthat lies on the line
connecting these two points must also lie in
Rk, and hence the decision region must be
singly connected and convex.RiRj
Rk
xAxB
ˆx
where 0⩽λ⩽1. From the linearity of the discriminant functions, it follows that
yk(ˆx)=λyk(xA)+( 1 −λ)yk(xB). (4.12)
Because both xAandxBlie inside Rk, it follows that yk(xA)>y j(xA), and
yk(xB)>y j(xB), for all j̸=k, and hence yk(ˆx)>y j(ˆx), and soˆxalso lies
insideRk. Thus Rkis singly connected and convex.
Note that for two classes, we can either employ the formalism discussed here,
based on two discriminant functions y1(x)andy2(x), or else use the simpler but
equivalent formulation described in Section 4.1.1 based on a single discriminant
function y(x).

We now explore three approaches to learning the parameters of linear discrimi-
nant functions, based on least squares, Fisher’s linear discriminant, and the percep-
tron algorithm.
4.1.3 Least squares for classiﬁcation
In Chapter 3, we considered models that were linear functions of the parame-
ters, and we saw that the minimization of a sum-of-squares error function led to a
simple closed-form solution for the parameter values. It is therefore tempting to see

if we can apply the same formalism to classiﬁcation problems. Consider a general
classiﬁcation problem with Kclasses, with a 1-of- Kbinary coding scheme for the
target vector t. One justiﬁcation for using least squares in such a context is that it
approximates the conditional expectation E[t|x]of the target values given the input
vector. For the binary coding scheme, this conditional expectation is given by the
vector of posterior class probabilities. Unfortunately, however, these probabilities

are typically approximated rather poorly, indeed the approximations can have values
outside the range (0,1), due to the limited ﬂexibility of a linear model as we shall
see shortly.
Each class Ckis described by its own linear model so that
yk(x)=wT
kx+wk0 (4.13)
where k=1,...,K . We can conveniently group these together using vector nota-
tion so that
y(x)=˜WT˜x (4.14)

4.1. Discriminant Functions 185
where˜Wis a matrix whose kthcolumn comprises the D+1-dimensional vector
˜wk=(wk0,wT
k)Tand˜xis the corresponding augmented input vector (1,xT)Twith
a dummy input x0=1. This representation was discussed in detail in Section 3.1. A
new input xis then assigned to the class for which the output yk=˜wT
k˜xis largest.
We now determine the parameter matrix ˜Wby minimizing a sum-of-squares
error function, as we did for regression in Chapter 3. Consider a training data set
{xn,tn}where n=1,...,N , and deﬁne a matrix Twhose nthrow is the vector tT
n,
together with a matrix ˜Xwhose nthrow is˜xT
n. The sum-of-squares error function
can then be written as
ED(˜W)=1
2Tr{
(˜X˜W−T)T(˜X˜W−T)}
. (4.15)
Setting the derivative with respect to ˜Wto zero, and rearranging, we then obtain the
solution for ˜Win the form
˜W=(˜XT˜X)−1˜XTT=˜X†T (4.16)
where˜X†is the pseudo-inverse of the matrix ˜X, as discussed in Section 3.1.1. We
then obtain the discriminant function in the form
y(x)=˜WT˜x=TT(
˜X†)T

n,
ED(˜W)=1
2Tr{
. (4.15)
˜X†)T
˜x. (4.17)
An interesting property of least-squares solutions with multiple target variables
is that if every target vector in the training set satisﬁes some linear constraint
aTtn+b=0 (4.18)
for some constants aandb, then the model prediction for any value of xwill satisfy
the same constraint so that Exercise 4.2
aTy(x)+b=0. (4.19)
Thus if we use a 1-of- Kcoding scheme for Kclasses, then the predictions made
by the model will have the property that the elements of y(x)will sum to 1 for any
value of x. However, this summation constraint alone is not sufﬁcient to allow the

model outputs to be interpreted as probabilities because they are not constrained to
lie within the interval (0,1).
The least-squares approach gives an exact closed-form solution for the discrimi-
nant function parameters. However, even as a discriminant function (where we use it
to make decisions directly and dispense with any probabilistic interpretation) it suf-
fers from some severe problems. We have already seen that least-squares solutions Section 2.3.7

lack robustness to outliers, and this applies equally to the classiﬁcation application,as illustrated in Figure 4.4. Here we see that the additional data points in the right-
hand ﬁgure produce a signiﬁcant change in the location of the decision boundary,
even though these point would be correctly classiﬁed by the original decision bound-ary in the left-hand ﬁgure. The sum-of-squares error function penalizes predictions
that are ‘too correct’ in that they lie a long way on the correct side of the decision

186 4. LINEAR MODELS FOR CLASSIFICATION
−4 −2 0 2 4 6 8−8−6−4−2024
Figure 4.4 The left plot shows data from two classes, denoted by red crosses and blue circles, together with
the decision boundary found by least squares (magenta curve) and also by the logistic regression model (green
curve), which is discussed later in Section 4.3.2. The right-hand plot shows the corresponding results obtained
when extra data points are added at the bottom left of the diagram, showing that least squares is highly sensitive
to outliers, unlike logistic regression.
boundary. In Section 7.1.2, we shall consider several alternative error functions for
classiﬁcation and we shall see that they do not suffer from this difﬁculty.
However, problems with least squares can be more severe than simply lack of
robustness, as illustrated in Figure 4.5. This shows a synthetic data set drawn from
three classes in a two-dimensional input space (x1,x2), having the property that lin-

ear decision boundaries can give excellent separation between the classes. Indeed,
the technique of logistic regression, described later in this chapter, gives a satisfac-
tory solution as seen in the right-hand plot. However, the least-squares solution gives
poor results, with only a small region of the input space assigned to the green class.
The failure of least squares should not surprise us when we recall that it cor-
responds to maximum likelihood under the assumption of a Gaussian conditional

distribution, whereas binary target vectors clearly have a distribution that is far from
Gaussian. By adopting more appropriate probabilistic models, we shall obtain clas-
siﬁcation techniques with much better properties than least squares. For the moment,
however, we continue to explore alternative nonprobabilistic methods for setting the
parameters in the linear classiﬁcation models.
4.1.4 Fisher’s linear discriminant
One way to view a linear classiﬁcation model is in terms of dimensionality

reduction. Consider ﬁrst the case of two classes, and suppose we take the D-

4.1. Discriminant Functions 187
Figure 4.5 Example of a synthetic data set comprising three classes, with training data points denoted in red
(×), green (+), and blue (◦). Lines denote the decision boundaries, and the background colours denote the
respective classes of the decision regions. On the left is the result of using a least-squares discriminant. We see
that the region of input space assigned to the green class is too small and so most of the points from this class
are misclassiﬁed. On the right is the result of using logistic regressions as described in Section 4.3.2 showing
correct classiﬁcation of the training data.
dimensional input vector xand project it down to one dimension using
y=wTx. (4.20)
If we place a threshold on yand classify y⩾−w0as class C1, and otherwise class
C2, then we obtain our standard linear classiﬁer discussed in the previous section.
In general, the projection onto one dimension leads to a considerable loss of infor-

mation, and classes that are well separated in the original D-dimensional space may
become strongly overlapping in one dimension. However, by adjusting the com-
ponents of the weight vector w, we can select a projection that maximizes the class
separation. To begin with, consider a two-class problem in which there are N1points
of class C1andN2points of class C2, so that the mean vectors of the two classes are
given by
m1=1
N1∑
n∈C 1xn, m2=1
N2∑
n∈C 2xn. (4.21)

given by
m1=1
N1∑
N2∑
The simplest measure of the separation of the classes, when projected onto w,i st h e
separation of the projected class means. This suggests that we might choose wso as
to maximize
m2−m1=wT(m2−m1) (4.22)
where
mk=wTmk (4.23)

188 4. LINEAR MODELS FOR CLASSIFICATION
−2 2 6−2024
−2 2 6−2024
Figure 4.6 The left plot shows samples from two classes (depicted in red and blue) along with the histograms
resulting from projection onto the line joining the class means. Note that there is considerable class overlap in
the projected space. The right plot shows the corresponding projection based on the Fisher linear discriminant,
showing the greatly improved class separation.
is the mean of the projected data from class Ck. However, this expression can be
made arbitrarily large simply by increasing the magnitude of w. To solve this
problem, we could constrain wto have unit length, so that∑
iw2
i=1. Using
a Lagrange multiplier to perform the constrained maximization, we then ﬁnd that Appendix E
w∝(m2−m1). There is still a problem with this approach, however, as illustrated Exercise 4.4
in Figure 4.6. This shows two classes that are well separated in the original two-

iw2
i=1. Using
dimensional space (x1,x2)but that have considerable overlap when projected onto
the line joining their means. This difﬁculty arises from the strongly nondiagonal
covariances of the class distributions. The idea proposed by Fisher is to maximize
a function that will give a large separation between the projected class means while
also giving a small variance within each class, thereby minimizing the class overlap.
The projection formula (4.20) transforms the set of labelled data points in x

into a labelled set in the one-dimensional space y. The within-class variance of the
transformed data from class Ckis therefore given by
s2
k=∑
n∈Ck(yn−mk)2(4.24)
where yn=wTxn. We can deﬁne the total within-class variance for the whole
data set to be simply s2
1+s2
2. The Fisher criterion is deﬁned to be the ratio of the
between-class variance to the within-class variance and is given by
s2
1+s2
2. (4.25)
We can make the dependence on wexplicit by using (4.20), (4.23), and (4.24) to

s2
k=∑
1+s2
s2
1+s2
2. (4.25)
rewrite the Fisher criterion in the form Exercise 4.5

4.1. Discriminant Functions 189
wTSWw(4.26)
whereSBis the between-class covariance matrix and is given by
SB=(m2−m1)(m2−m1)T(4.27)
andSWis the total within-class covariance matrix, given by
SW=∑
n∈C 1(xn−m1)(xn−m1)T+∑
n∈C 2(xn−m2)(xn−m2)T. (4.28)
Differentiating (4.26) with respect to w, we ﬁnd that J(w)is maximized when
(wTSBw)SWw=(wTSWw)SBw. (4.29)
From (4.27), we see that SBwis always in the direction of (m2−m1). Furthermore,
we do not care about the magnitude of w, only its direction, and so we can drop the
scalar factors (wTSBw)and(wTSWw). Multiplying both sides of (4.29) by S−1
W
we then obtain
w∝S−1
W(m2−m1). (4.30)
Note that if the within-class covariance is isotropic, so that SWis proportional to the
unit matrix, we ﬁnd that wis proportional to the difference of the class means, as
discussed above.
The result (4.30) is known as Fisher’s linear discriminant , although strictly it
is not a discriminant but rather a speciﬁc choice of direction for projection of the

W
w∝S−1
data down to one dimension. However, the projected data can subsequently be used
to construct a discriminant, by choosing a threshold y0so that we classify a new
point as belonging to C1ify(x)⩾y0and classify it as belonging to C2otherwise.
For example, we can model the class-conditional densities p(y|Ck)using Gaussian
distributions and then use the techniques of Section 1.2.4 to ﬁnd the parametersof the Gaussian distributions by maximum likelihood. Having found Gaussian ap-

proximations to the projected classes, the formalism of Section 1.5.1 then gives an
expression for the optimal threshold. Some justiﬁcation for the Gaussian assumptioncomes from the central limit theorem by noting that y=w
Txis the sum of a set of
random variables.
4.1.5 Relation to least squares
The least-squares approach to the determination of a linear discriminant was
based on the goal of making the model predictions as close as possible to a set of
target values. By contrast, the Fisher criterion was derived by requiring maximum

class separation in the output space. It is interesting to see the relationship between
these two approaches. In particular, we shall show that, for the two-class problem,the Fisher criterion can be obtained as a special case of least squares.
So far we have considered 1-of- Kcoding for the target values. If, however, we
adopt a slightly different target coding scheme, then the least-squares solution for

190 4. LINEAR MODELS FOR CLASSIFICATION
the weights becomes equivalent to the Fisher solution (Duda and Hart, 1973). In
particular, we shall take the targets for class C1to beN/N 1, where N1is the number
of patterns in class C1, andNis the total number of patterns. This target value
approximates the reciprocal of the prior probability for class C1. For class C2,w e
shall take the targets to be −N/N 2, where N2is the number of patterns in class C2.
The sum-of-squares error function can be written
E=1
2N∑
n=1(
wTxn+w0−tn)2. (4.31)
Setting the derivatives of Ewith respect to w0andwto zero, we obtain respectively
N∑
n=1(
wTxn+w0−tn)
=0 (4.32)
N∑
n=1(
xn=0. (4.33)
From (4.32), and making use of our choice of target coding scheme for the tn,w e
obtain an expression for the bias in the form
w0=−wTm (4.34)
where we have used
N∑
n=1tn=N1N
N1−N2N
N2=0 (4.35)
and where mis the mean of the total data set and is given by
m=1
NN∑
n=1xn=1
N(N1m1+N2m2). (4.36)

E=1
2N∑
n=1(
N∑
n=1(
=0 (4.32)
N∑
n=1(
N∑
n=1tn=N1N
N1−N2N
m=1
NN∑
n=1xn=1
After some straightforward algebra, and again making use of the choice of tn, the
second equation (4.33) becomes Exercise 4.6
(
SW+N1N2
NSB)
w=N(m1−m2) (4.37)
whereSWis deﬁned by (4.28), SBis deﬁned by (4.27), and we have substituted for
the bias using (4.34). Using (4.27), we note that SBwis always in the direction of
(m2−m1). Thus we can write
w∝S−1
W(m2−m1) (4.38)
where we have ignored irrelevant scale factors. Thus the weight vector coincides

m=1
NN∑
n=1xn=1
(
SW+N1N2
NSB)
w∝S−1
with that found from the Fisher criterion. In addition, we have also found an expres-sion for the bias value w
0given by (4.34). This tells us that a new vector xshould be
classiﬁed as belonging to class C1ify(x)=wT(x−m)>0and class C2otherwise.

4.1. Discriminant Functions 191
4.1.6 Fisher’s discriminant for multiple classes
We now consider the generalization of the Fisher discriminant to K>2classes,
and we shall assume that the dimensionality Dof the input space is greater than the
number Kof classes. Next, we introduce D′>1linear ‘features’ yk=wT
kx, where
k=1,...,D′. These feature values can conveniently be grouped together to form
a vector y. Similarly, the weight vectors {wk}can be considered to be the columns
of a matrix W, so that
y=WTx. (4.39)
Note that again we are not including any bias parameters in the deﬁnition of y. The
generalization of the within-class covariance matrix to the case of Kclasses follows
from (4.28) to give
SW=K∑
k=1Sk (4.40)
where
Sk=∑
n∈Ck(xn−mk)(xn−mk)T(4.41)
mk=1
Nk∑
n∈Ckxn (4.42)
andNkis the number of patterns in class Ck. In order to ﬁnd a generalization of the
between-class covariance matrix, we follow Duda and Hart (1973) and consider ﬁrst
the total covariance matrix
ST=N∑
n=1(xn−m)(xn−m)T(4.43)

SW=K∑
where
Sk=∑
mk=1
Nk∑
ST=N∑
wheremis the mean of the total data set
m=1
NN∑
n=1xn=1
NK∑
k=1Nkmk (4.44)
andN=∑
kNkis the total number of data points. The total covariance matrix can
be decomposed into the sum of the within-class covariance matrix, given by (4.40)
and (4.41), plus an additional matrix SB, which we identify as a measure of the
between-class covariance
ST=SW+SB (4.45)
where
SB=K∑
k=1Nk(mk−m)(mk−m)T. (4.46)

192 4. LINEAR MODELS FOR CLASSIFICATION
These covariance matrices have been deﬁned in the original x-space. We can now
deﬁne similar matrices in the projected D′-dimensional y-space
sW=K∑
k=1∑
n∈Ck(yn−µk)(yn−µk)T(4.47)
and
sB=K∑
k=1Nk(µk−µ)(µk−µ)T(4.48)
where
µk=1
Nk∑
n∈Ckyn, µ=1
NK∑
k=1Nkµk. (4.49)
Again we wish to construct a scalar that is large when the between-class covariance
is large and when the within-class covariance is small. There are now many possible
choices of criterion (Fukunaga, 1990). One example is given by
J(W)=Tr{
s−1
WsB}
. (4.50)
This criterion can then be rewritten as an explicit function of the projection matrix
Win the form
J(w)=Tr{
(WSWWT)−1(WSBWT)}
. (4.51)
Maximization of such criteria is straightforward, though somewhat involved, and is
discussed at length in Fukunaga (1990). The weight values are determined by thoseeigenvectors of S
−1
WSBthat correspond to the D′largest eigenvalues.
There is one important result that is common to all such criteria, which is worth

J(W)=Tr{
s−1
WsB}
. (4.50)
J(w)=Tr{
. (4.51)
−1
emphasizing. We ﬁrst note from (4.46) that SBis composed of the sum of Kma-
trices, each of which is an outer product of two vectors and therefore of rank 1. In
addition, only (K−1)of these matrices are independent as a result of the constraint
(4.44). Thus, SBhas rank at most equal to (K−1)and so there are at most (K−1)
nonzero eigenvalues. This shows that the projection onto the (K−1)-dimensional
subspace spanned by the eigenvectors of SBdoes not alter the value of J(w), and

so we are therefore unable to ﬁnd more than (K−1)linear ‘features’ by this means
(Fukunaga, 1990).
4.1.7 The perceptron algorithm
Another example of a linear discriminant model is the perceptron of Rosenblatt
(1962), which occupies an important place in the history of pattern recognition al-
gorithms. It corresponds to a two-class model in which the input vector xis ﬁrst
transformed using a ﬁxed nonlinear transformation to give a feature vector φ(x),
and this is then used to construct a generalized linear model of the form
y(x)=f(

y(x)=f(
wTφ(x))
(4.52)

4.1. Discriminant Functions 193
where the nonlinear activation function f(·)is given by a step function of the form
f(a)={
+1,a⩾0
−1,a < 0.(4.53)
The vector φ(x)will typically include a bias component φ0(x)=1 . In earlier
discussions of two-class classiﬁcation problems, we have focussed on a target coding
scheme in which t∈{0,1}, which is appropriate in the context of probabilistic
models. For the perceptron, however, it is more convenient to use target values
t=+ 1 for class C1andt=−1for class C2, which matches the choice of activation
function.
The algorithm used to determine the parameters wof the perceptron can most
easily be motivated by error function minimization. A natural choice of error func-
tion would be the total number of misclassiﬁed patterns. However, this does not leadto a simple learning algorithm because the error is a piecewise constant function
ofw, with discontinuities wherever a change in wcauses the decision boundary to

function.
move across one of the data points. Methods based on changing wusing the gradi-
ent of the error function cannot then be applied, because the gradient is zero almost
We therefore consider an alternative error function known as the perceptron cri-
terion . To derive this, we note that we are seeking a weight vector wsuch that
patterns x
nin class C1will have wTφ(xn)>0, whereas patterns xnin class C2
havewTφ(xn)<0. Using the t∈{ −1,+1}target coding scheme it follows that

patterns x
we would like all patterns to satisfy wTφ(xn)tn>0. The perceptron criterion
associates zero error with any pattern that is correctly classiﬁed, whereas for a mis-
classiﬁed pattern xnit tries to minimize the quantity −wTφ(xn)tn. The perceptron
criterion is therefore given by
EP(w)=−∑
n∈MwTφntn (4.54)
Frank Rosenblatt
1928–1969
Rosenblatt’s perceptron played an
important role in the history of ma-chine learning. Initially, Rosenblattsimulated the perceptron on an IBM704 computer at Cornell in 1957,but by the early 1960s he had built

EP(w)=−∑
1928–1969
special-purpose hardware that provided a direct, par-allel implementation of perceptron learning. Many ofhis ideas were encapsulated in “Principles of Neuro-dynamics: Perceptrons and the Theory of Brain Mech-anisms” published in 1962. Rosenblatt’s work wascriticized by Marvin Minksy, whose objections werepublished in the book “Perceptrons”, co-authored withSeymour Papert. This book was widely misinter-

preted at the time as showing that neural networkswere fatally ﬂawed and could only learn solutions forlinearly separable problems. In fact, it only provedsuch limitations in the case of single-layer networkssuch as the perceptron and merely conjectured (in-correctly) that they applied to more general networkmodels. Unfortunately, however, this book contributedto the substantial decline in research funding for neu-ral computing, a situation that was not reversed un-til the mid-1980s. Today, there are many hundreds,if not thousands, of applications of neural networksin widespread use, with examples in areas such ashandwriting recognition and information retrieval be-ing used routinely by millions of people.

194 4. LINEAR MODELS FOR CLASSIFICATION
where Mdenotes the set of all misclassiﬁed patterns. The contribution to the error
associated with a particular misclassiﬁed pattern is a linear function of win regions
ofwspace where the pattern is misclassiﬁed and zero in regions where it is correctly
classiﬁed. The total error function is therefore piecewise linear.
We now apply the stochastic gradient descent algorithm to this error function. Section 3.1.3
The change in the weight vector wis then given by
w(τ+1)=w(τ)−η∇EP(w)=w(τ)+ηφntn (4.55)
where ηis the learning rate parameter and τis an integer that indexes the steps of
the algorithm. Because the perceptron function y(x,w)is unchanged if we multiply
wby a constant, we can set the learning rate parameter ηequal to 1 without of
generality. Note that, as the weight vector evolves during training, the set of patternsthat are misclassiﬁed will change.
The perceptron learning algorithm has a simple interpretation, as follows. We

cycle through the training patterns in turn, and for each pattern x
nwe evaluate the
perceptron function (4.52). If the pattern is correctly classiﬁed, then the weight
vector remains unchanged, whereas if it is incorrectly classiﬁed, then for class C1
we add the vector φ(xn)onto the current estimate of weight vector wwhile for
classC2we subtract the vector φ(xn)fromw. The perceptron learning algorithm is
illustrated in Figure 4.7.
If we consider the effect of a single update in the perceptron learning algorithm,

we see that the contribution to the error from a misclassiﬁed pattern will be reduced
because from (4.55) we have
−w(τ+1)Tφntn=−w(τ)Tφntn−(φntn)Tφntn<−w(τ)Tφntn (4.56)
where we have set η=1, and made use of ∥φntn∥2>0. Of course, this does
not imply that the contribution to the error function from the other misclassiﬁedpatterns will have been reduced. Furthermore, the change in weight vector may have
caused some previously correctly classiﬁed patterns to become misclassiﬁed. Thus

the perceptron learning rule is not guaranteed to reduce the total error function ateach stage.
However, the perceptron convergence theorem states that if there exists an ex-
act solution (in other words, if the training data set is linearly separable), then theperceptron learning algorithm is guaranteed to ﬁnd an exact solution in a ﬁnite num-
ber of steps. Proofs of this theorem can be found for example in Rosenblatt (1962),
Block (1962), Nilsson (1965), Minsky and Papert (1969), Hertz et al. (1991), and

Bishop (1995a). Note, however, that the number of steps required to achieve con-
vergence could still be substantial, and in practice, until convergence is achieved,
we will not be able to distinguish between a nonseparable problem and one that is
simply slow to converge.
Even when the data set is linearly separable, there may be many solutions, and
which one is found will depend on the initialization of the parameters and on the or-

der of presentation of the data points. Furthermore, for data sets that are not linearly
separable, the perceptron learning algorithm will never converge.

4.1. Discriminant Functions 195
−1 −0.5 0 0.5 1−1−0.500.51
Figure 4.7 Illustration of the convergence of the perceptron learning algorithm, showing data points from two
classes (red and blue) in a two-dimensional feature space (φ1,φ2). The top left plot shows the initial parameter
vector wshown as a black arrow together with the corresponding decision boundary (black line), in which the
arrow points towards the decision region which classiﬁed as belonging to the red class. The data point circled
in green is misclassiﬁed and so its feature vector is added to the current weight vector, giving the new decision
boundary shown in the top right plot. The bottom left plot shows the next misclassiﬁed point to be considered,
indicated by the green circle, and its feature vector is again added to the weight vector giving the decision
boundary shown in the bottom right plot for which all data points are correctly classiﬁed.

196 4. LINEAR MODELS FOR CLASSIFICATION
Figure 4.8 Illustration of the Mark 1 perceptron hardware. The photograph on the left shows how the inputs
were obtained using a simple camera system in which an input scene, in this case a printed character, wasilluminated by powerful lights, and an image focussed onto a 20×20array of cadmium sulphide photocells,
giving a primitive 400 pixel image. The perceptron also had a patch board, shown in the middle photograph,which allowed different conﬁgurations of input features to be tried. Often these were wired up at random todemonstrate the ability of the perceptron to learn without the need for precise wiring, in contrast to a moderndigital computer. The photograph on the right shows one of the racks of adaptive weights. Each weight wasimplemented using a rotary variable resistor, also called a potentiometer, driven by an electric motor therebyallowing the value of the weight to be adjusted automatically by the learning algorithm.

Aside from difﬁculties with the learning algorithm, the perceptron does not pro-
vide probabilistic outputs, nor does it generalize readily to K>2classes. The most
important limitation, however, arises from the fact that (in common with all of themodels discussed in this chapter and the previous one) it is based on linear com-
binations of ﬁxed basis functions. More detailed discussions of the limitations of
perceptrons can be found in Minsky and Papert (1969) and Bishop (1995a).
Analogue hardware implementations of the perceptron were built by Rosenblatt,
based on motor-driven variable resistors to implement the adaptive parameters w
j.
These are illustrated in Figure 4.8. The inputs were obtained from a simple camera
system based on an array of photo-sensors, while the basis functions φcould be
chosen in a variety of ways, for example based on simple ﬁxed functions of randomlychosen subsets of pixels from the input image. Typical applications involved learning
to discriminate simple shapes or characters.

j.
At the same time that the perceptron was being developed, a closely related
system called the adaline , which is short for ‘adaptive linear element’, was being
explored by Widrow and co-workers. The functional form of the model was the same
as for the perceptron, but a different approach to training was adopted (Widrow andHoff, 1960; Widrow and Lehr, 1990).
4.2. Probabilistic Generative Models
We turn next to a probabilistic view of classiﬁcation and show how models withlinear decision boundaries arise from simple assumptions about the distribution of

the data. In Section 1.5.4, we discussed the distinction between the discriminative
and the generative approaches to classiﬁcation. Here we shall adopt a generative

4.2. Probabilistic Generative Models 197
Figure 4.9 Plot of the logistic sigmoid function
σ(a)deﬁned by (4.59), shown in
red, together with the scaled pro-
bit function Φ(λa),f o rλ2=π/8,
shown in dashed blue, where Φ(a)
is deﬁned by (4.114). The scal-
ing factor π/8is chosen so that the
derivatives of the two curves are
equal for a=0.
−5 0 500.51
approach in which we model the class-conditional densities p(x|Ck), as well as the
class priors p(Ck), and then use these to compute posterior probabilities p(Ck|x)
through Bayes’ theorem.
Consider ﬁrst of all the case of two classes. The posterior probability for class
C1can be written as
p(C1|x)=p(x|C1)p(C1)
p(x|C1)p(C1)+p(x|C2)p(C2)
=1
1+e x p ( −a)=σ(a) (4.57)
a=l np(x|C1)p(C1)
p(x|C2)p(C2)(4.58)
andσ(a)is the logistic sigmoid function deﬁned by
σ(a)=1
1+e x p ( −a)(4.59)
which is plotted in Figure 4.9. The term ‘sigmoid’ means S-shaped. This type of
function is sometimes also called a ‘squashing function’ because it maps the whole

=1
σ(a)=1
real axis into a ﬁnite interval. The logistic sigmoid has been encountered already
in earlier chapters and plays an important role in many classiﬁcation algorithms. It
satisﬁes the following symmetry property
σ(−a)=1−σ(a) (4.60)
as is easily veriﬁed. The inverse of the logistic sigmoid is given by
a=l n(σ
1−σ)
(4.61)
and is known as the logit function. It represents the log of the ratio of probabilities
ln[p(C1|x)/p(C2|x)]for the two classes, also known as the log odds .

198 4. LINEAR MODELS FOR CLASSIFICATION
Note that in (4.57) we have simply rewritten the posterior probabilities in an
equivalent form, and so the appearance of the logistic sigmoid may seem rather vac-uous. However, it will have signiﬁcance provided a(x)takes a simple functional
form. We shall shortly consider situations in which a(x)is a linear function of x,i n
which case the posterior probability is governed by a generalized linear model.
For the case of K>2classes, we have
p(C
k|x)=p(x|Ck)p(Ck)∑
jp(x|Cj)p(Cj)
=exp(ak)∑
jexp(aj)(4.62)
which is known as the normalized exponential and can be regarded as a multiclass
generalization of the logistic sigmoid. Here the quantities akare deﬁned by
ak=l np(x|Ck)p(Ck). (4.63)
The normalized exponential is also known as the softmax function , as it represents
a smoothed version of the ‘max’ function because, if ak≫ajfor all j̸=k, then
p(Ck|x)≃1, andp(Cj|x)≃0.
We now investigate the consequences of choosing speciﬁc forms for the class-

p(C
=exp(ak)∑
conditional densities, looking ﬁrst at continuous input variables xand then dis-
cussing brieﬂy the case of discrete inputs.
4.2.1 Continuous inputs
Let us assume that the class-conditional densities are Gaussian and then explore
the resulting form for the posterior probabilities. To start with, we shall assume that
all classes share the same covariance matrix. Thus the density for class Ckis given
by
p(x|Ck)=1
(2π)D/21
−1
2(x−µk)TΣ−1(x−µk)}
. (4.64)

by
p(x|Ck)=1
(2π)D/21
−1
. (4.64)
Consider ﬁrst the case of two classes. From (4.57) and (4.58), we have
p(C1|x)=σ(wTx+w0) (4.65)
w=Σ−1(µ1−µ2) (4.66)
w0=−1
2µT
1Σ−1µ1+1
2µT
2Σ−1µ2+l np(C1)
p(C2). (4.67)
We see that the quadratic terms in xfrom the exponents of the Gaussian densities
have cancelled (due to the assumption of common covariance matrices) leading toa linear function of xin the argument of the logistic sigmoid. This result is illus-
trated for the case of a two-dimensional input space xin Figure 4.10. The resulting

4.2. Probabilistic Generative Models 199
Figure 4.10 The left-hand plot shows the class-conditional densities for two classes, denoted red and blue.
On the right is the corresponding posterior probability p(C1|x), which is given by a logistic sigmoid of a linear
function of x. The surface in the right-hand plot is coloured using a proportion of red ink given by p(C1|x)and a
proportion of blue ink given by p(C2|x)=1−p(C1|x).
decision boundaries correspond to surfaces along which the posterior probabilities
p(Ck|x)are constant and so will be given by linear functions of x, and therefore
the decision boundaries are linear in input space. The prior probabilities p(Ck)enter
only through the bias parameter w0so that changes in the priors have the effect of
making parallel shifts of the decision boundary and more generally of the parallel
contours of constant posterior probability.
For the general case of Kclasses we have, from (4.62) and (4.63),
ak(x)=wT
kx+wk0 (4.68)
wk=Σ−1µk (4.69)
wk0=−1
2µT

ak(x)=wT
wk0=−1
2µT
kΣ−1µk+l np(Ck). (4.70)
We see that the ak(x)are again linear functions of xas a consequence of the cancel-
lation of the quadratic terms due to the shared covariances. The resulting decisionboundaries, corresponding to the minimum misclassiﬁcation rate, will occur when
two of the posterior probabilities (the two largest) are equal, and so will be deﬁnedby linear functions of x, and so again we have a generalized linear model.
If we relax the assumption of a shared covariance matrix and allow each class-
conditional density p(x|C

k)to have its own covariance matrix Σk, then the earlier
cancellations will no longer occur, and we will obtain quadratic functions of x,g i v -
ing rise to a quadratic discriminant . The linear and quadratic decision boundaries
are illustrated in Figure 4.11.

200 4. LINEAR MODELS FOR CLASSIFICATION
−2 −1 0 1 2−2.5−2−1.5−1−0.500.511.522.5
Figure 4.11 The left-hand plot shows the class-conditional densities for three classes each having a Gaussian
distribution, coloured red, green, and blue, in which the red and green classes have the same covariance matrix.
The right-hand plot shows the corresponding posterior probabilities, in which the RGB colour vector represents
the posterior probabilities for the respective three classes. The decision boundaries are also shown. Notice that
the boundary between the red and green classes, which have the same covariance matrix, is linear, whereas
those between the other pairs of classes are quadratic.
4.2.2 Maximum likelihood solution
Once we have speciﬁed a parametric functional form for the class-conditional
densities p(x|Ck), we can then determine the values of the parameters, together with
the prior class probabilities p(Ck), using maximum likelihood. This requires a data

set comprising observations of xalong with their corresponding class labels.
Consider ﬁrst the case of two classes, each having a Gaussian class-conditional
density with a shared covariance matrix, and suppose we have a data set {xn,tn}
where n=1,...,N . Here tn=1denotes class C1andtn=0denotes class C2.W e
denote the prior class probability p(C1)=π, so that p(C2)=1−π. For a data point
xnfrom class C1,w eh a v e tn=1and hence
p(xn,C1)=p(C1)p(xn|C1)=πN(xn|µ1,Σ).
Similarly for class C2,w eh a v e tn=0and hence
p(xn,C2)=p(C2)p(xn|C2)=( 1 −π)N(xn|µ2,Σ).

Thus the likelihood function is given by
p(t|π,µ1,µ2,Σ)=N∏
n=1[πN(xn|µ1,Σ)]tn[(1−π)N(xn|µ2,Σ)]1−tn(4.71)
where t=(t1,...,t N)T. As usual, it is convenient to maximize the log of the
likelihood function. Consider ﬁrst the maximization with respect to π. The terms in

4.2. Probabilistic Generative Models 201
the log likelihood function that depend on πare
N∑
n=1{tnlnπ+( 1−tn)l n ( 1−π)}. (4.72)
Setting the derivative with respect to πequal to zero and rearranging, we obtain
π=1
NN∑
n=1tn=N1
N=N1
N1+N2(4.73)
where N1denotes the total number of data points in class C1, andN2denotes the total
number of data points in class C2. Thus the maximum likelihood estimate for πis
simply the fraction of points in class C1as expected. This result is easily generalized
to the multiclass case where again the maximum likelihood estimate of the prior
probability associated with class Ckis given by the fraction of the training set points
assigned to that class. Exercise 4.9
Now consider the maximization with respect to µ1. Again we can pick out of
the log likelihood function those terms that depend on µ1giving
N∑
n=1tnlnN(xn|µ1,Σ)=−1
2N∑
n=1tn(xn−µ1)TΣ−1(xn−µ1)+c o n s t .(4.74)
Setting the derivative with respect to µ1to zero and rearranging, we obtain
µ1=1
N1N∑
n=1tnxn (4.75)

N∑
2N∑
µ1=1
N1N∑
which is simply the mean of all the input vectors xnassigned to class C1.B y a
similar argument, the corresponding result for µ2is given by
µ2=1
N2N∑
n=1(1−tn)xn (4.76)
which again is the mean of all the input vectors xnassigned to class C2.
Finally, consider the maximum likelihood solution for the shared covariance
matrixΣ. Picking out the terms in the log likelihood function that depend on Σ,w e
have
−1
2N∑
n=1tnln|Σ|−1
2N∑
n=1tn(xn−µ1)TΣ−1(xn−µ1)
−1
2N∑
n=1(1−tn)l n|Σ|−1
2N∑
n=1(1−tn)(xn−µ2)TΣ−1(xn−µ2)
=−N
2ln|Σ|−N
2Tr{
Σ−1S}
(4.77)

202 4. LINEAR MODELS FOR CLASSIFICATION
S=N1
NS1+N2
NS2 (4.78)
S1=1
N1∑
n∈C 1(xn−µ1)(xn−µ1)T(4.79)
S2=1
N2∑
n∈C 2(xn−µ2)(xn−µ2)T. (4.80)
Using the standard result for the maximum likelihood solution for a Gaussian distri-
bution, we see that Σ=S, which represents a weighted average of the covariance
matrices associated with each of the two classes separately.
This result is easily extended to the Kclass problem to obtain the corresponding
maximum likelihood solutions for the parameters in which each class-conditional
density is Gaussian with a shared covariance matrix. Note that the approach of ﬁtting Exercise 4.10
Gaussian distributions to the classes is not robust to outliers, because the maximum
likelihood estimation of a Gaussian is not robust. Section 2.3.7
4.2.3 Discrete features
Let us now consider the case of discrete feature values xi. For simplicity, we
begin by looking at binary feature values xi∈{0,1}and discuss the extension to

more general discrete features shortly. If there are Dinputs, then a general distribu-
tion would correspond to a table of 2Dnumbers for each class, containing 2D−1
independent variables (due to the summation constraint). Because this grows expo-
nentially with the number of features, we might seek a more restricted representa-tion. Here we will make the naive Bayes assumption in which the feature values are Section 8.2.2
treated as independent, conditioned on the class C
k. Thus we have class-conditional

distributions of the form
p(x|Ck)=D∏
i=1µxi
ki(1−µki)1−xi(4.81)
which contain Dindependent parameters for each class. Substituting into (4.63) then
gives
ak(x)=D∑
i=1{xilnµki+( 1−xi)l n ( 1−µki)}+l np(Ck) (4.82)
which again are linear functions of the input values xi. For the case of K=2classes,
we can alternatively consider the logistic sigmoid formulation given by (4.57). Anal-
ogous results are obtained for discrete variables each of which can take M> 2
states. Exercise 4.11
4.2.4 Exponential family

i=1µxi
gives
ak(x)=D∑
As we have seen, for both Gaussian distributed and discrete inputs, the posterior
class probabilities are given by generalized linear models with logistic sigmoid ( K=

4.3. Probabilistic Discriminative Models 203
2classes) or softmax ( K⩾2classes) activation functions. These are particular cases
of a more general result obtained by assuming that the class-conditional densitiesp(x|C
k)are members of the exponential family of distributions.
Using the form (2.194) for members of the exponential family, we see that the
distribution of xcan be written in the form
p(x|λk)=h(x)g(λk)e x p{
λT
ku(x)}
. (4.83)
We now restrict attention to the subclass of such distributions for which u(x)=x.
Then we make use of (2.236) to introduce a scaling parameter s, so that we obtain
the restricted set of exponential family class-conditional densities of the form
p(x|λk,s)=1
sh(1
sx)
g(λk)e x p{1
sλT
kx}
. (4.84)
Note that we are allowing each class to have its own parameter vector λkbut we are
assuming that the classes share the same scale parameter s.
For the two-class problem, we substitute this expression for the class-conditional

sh(1
sx)
sλT
kx}
. (4.84)
densities into (4.58) and we see that the posterior class probability is again given bya logistic sigmoid acting on a linear function a(x)which is given by
a(x)=(λ
1−λ2)Tx+l ng(λ1)−lng(λ2)+l n p(C1)−lnp(C2). (4.85)
Similarly, for the K-class problem, we substitute the class-conditional density ex-
pression into (4.63) to give
ak(x)=λT
kx+l ng(λk)+l n p(Ck) (4.86)
and so again is a linear function of x.
4.3. Probabilistic Discriminative Models
For the two-class classiﬁcation problem, we have seen that the posterior probability

a(x)=(λ
ak(x)=λT
of class C1can be written as a logistic sigmoid acting on a linear function of x, for a
wide choice of class-conditional distributions p(x|Ck). Similarly, for the multiclass
case, the posterior probability of class Ckis given by a softmax transformation of a
linear function of x. For speciﬁc choices of the class-conditional densities p(x|Ck),
we have used maximum likelihood to determine the parameters of the densities as
well as the class priors p(Ck)and then used Bayes’ theorem to ﬁnd the posterior class
probabilities.
However, an alternative approach is to use the functional form of the generalized

linear model explicitly and to determine its parameters directly by using maximum
likelihood. We shall see that there is an efﬁcient algorithm ﬁnding such solutions
known as iterative reweighted least squares ,o r IRLS .
The indirect approach to ﬁnding the parameters of a generalized linear model,
by ﬁtting class-conditional densities and class priors separately and then applying

204 4. LINEAR MODELS FOR CLASSIFICATION
x1x2
−1 0 1−101
φ1φ2
Figure 4.12 Illustration of the role of nonlinear basis functions in linear classiﬁcation models. The left plot
shows the original input space (x1,x2)together with data points from two classes labelled red and blue. Two
‘Gaussian’ basis functions φ1(x)andφ2(x)are deﬁned in this space with centres shown by the green crosses
and with contours shown by the green circles. The right-hand plot shows the corresponding feature space
(φ1,φ2)together with the linear decision boundary obtained given by a logistic regression model of the form
discussed in Section 4.3.2. This corresponds to a nonlinear decision boundary in the original input space,
shown by the black curve in the left-hand plot.
Bayes’ theorem, represents an example of generative modelling, because we could
take such a model and generate synthetic data by drawing values of xfrom the
marginal distribution p(x). In the direct approach, we are maximizing a likelihood

function deﬁned through the conditional distribution p(Ck|x), which represents a
form of discriminative training. One advantage of the discriminative approach is
that there will typically be fewer adaptive parameters to be determined, as we shall
see shortly. It may also lead to improved predictive performance, particularly when
the class-conditional density assumptions give a poor approximation to the true dis-
tributions.
4.3.1 Fixed basis functions

So far in this chapter, we have considered classiﬁcation models that work di-
rectly with the original input vector x. However, all of the algorithms are equally
applicable if we ﬁrst make a ﬁxed nonlinear transformation of the inputs using a
vector of basis functions φ(x). The resulting decision boundaries will be linear in
the feature space φ, and these correspond to nonlinear decision boundaries in the
original xspace, as illustrated in Figure 4.12. Classes that are linearly separable

in the feature space φ(x)need not be linearly separable in the original observation
spacex. Note that as in our discussion of linear models for regression, one of the

4.3. Probabilistic Discriminative Models 205
basis functions is typically set to a constant, say φ0(x)=1 , so that the correspond-
ing parameter w0plays the role of a bias. For the remainder of this chapter, we shall
include a ﬁxed basis function transformation φ(x), as this will highlight some useful
similarities to the regression models discussed in Chapter 3.
For many problems of practical interest, there is signiﬁcant overlap between
the class-conditional densities p(x|Ck). This corresponds to posterior probabilities
p(Ck|x), which, for at least some values of x, are not 0 or 1. In such cases, the opti-
mal solution is obtained by modelling the posterior probabilities accurately and thenapplying standard decision theory, as discussed in Chapter 1. Note that nonlinear
transformations φ(x)cannot remove such class overlap. Indeed, they can increase
the level of overlap, or create overlap where none existed in the original observation

space. However, suitable choices of nonlinearity can make the process of modelling
the posterior probabilities easier.
Such ﬁxed basis function models have important limitations, and these will be Section 3.6
resolved in later chapters by allowing the basis functions themselves to adapt to the
data. Notwithstanding these limitations, models with ﬁxed nonlinear basis functionsplay an important role in applications, and a discussion of such models will intro-

duce many of the key concepts needed for an understanding of their more complex
counterparts.
4.3.2 Logistic regression
We begin our treatment of generalized linear models by considering the problem
of two-class classiﬁcation. In our discussion of generative approaches in Section 4.2,
we saw that under rather general assumptions, the posterior probability of class C1
can be written as a logistic sigmoid acting on a linear function of the feature vector
φso that
p(C1|φ)=y(φ)=σ(
wTφ)
(4.87)

φso that
wTφ)
(4.87)
withp(C2|φ)=1−p(C1|φ). Here σ(·)is the logistic sigmoid function deﬁned by
(4.59). In the terminology of statistics, this model is known as logistic regression ,
although it should be emphasized that this is a model for classiﬁcation rather than
regression.
For an M-dimensional feature space φ, this model has Madjustable parameters.
By contrast, if we had ﬁtted Gaussian class conditional densities using maximumlikelihood, we would have used 2Mparameters for the means and M(M+1 )/2

wTφ)
(4.87)
parameters for the (shared) covariance matrix. Together with the class prior p(C
1),
this gives a total of M(M+5)/2+1 parameters, which grows quadratically with M,
in contrast to the linear dependence on Mof the number of parameters in logistic
regression. For large values of M, there is a clear advantage in working with the
logistic regression model directly.
We now use maximum likelihood to determine the parameters of the logistic
regression model. To do this, we shall make use of the derivative of the logistic sig-

1),
moid function, which can conveniently be expressed in terms of the sigmoid functionitself Exercise 4.12
dσ
da=σ(1−σ). (4.88)

206 4. LINEAR MODELS FOR CLASSIFICATION
For a data set {φn,tn}, where tn∈{0,1}andφn=φ(xn), with n=
1,...,N , the likelihood function can be written
p(t|w)=N∏
n=1ytn
n{1−yn}1−tn(4.89)
where t=(t1,...,t N)Tandyn=p(C1|φn). As usual, we can deﬁne an error
function by taking the negative logarithm of the likelihood, which gives the cross-
entropy error function in the form
E(w)=−lnp(t|w)=−N∑
n=1{tnlnyn+( 1−tn)l n ( 1−yn)} (4.90)
where yn=σ(an)andan=wTφn. Taking the gradient of the error function with
respect to w, we obtain Exercise 4.13
∇E(w)=N∑
n=1(yn−tn)φn (4.91)
where we have made use of (4.88). We see that the factor involving the derivative
of the logistic sigmoid has cancelled, leading to a simpliﬁed form for the gradient
of the log likelihood. In particular, the contribution to the gradient from data point
nis given by the ‘error’ yn−tnbetween the target value and the prediction of the
model, times the basis function vector φn. Furthermore, comparison with (3.13)

∇E(w)=N∑
shows that this takes precisely the same form as the gradient of the sum-of-squares
error function for the linear regression model. Section 3.1.1
If desired, we could make use of the result (4.91) to give a sequential algorithm
in which patterns are presented one at a time, in which each of the weight vectors is
updated using (3.22) in which ∇Enis thenthterm in (4.91).
It is worth noting that maximum likelihood can exhibit severe over-ﬁtting for
data sets that are linearly separable. This arises because the maximum likelihood so-

lution occurs when the hyperplane corresponding to σ=0.5, equivalent to wTφ=
0, separates the two classes and the magnitude of wgoes to inﬁnity. In this case, the
logistic sigmoid function becomes inﬁnitely steep in feature space, corresponding to
a Heaviside step function, so that every training point from each class kis assigned
a posterior probability p(Ck|x)=1 . Furthermore, there is typically a continuum Exercise 4.14
of such solutions because any separating hyperplane will give rise to the same pos-

terior probabilities at the training data points, as will be seen later in Figure 10.13.
Maximum likelihood provides no way to favour one such solution over another, and
which solution is found in practice will depend on the choice of optimization algo-rithm and on the parameter initialization. Note that the problem will arise even if
the number of data points is large compared with the number of parameters in the

model, so long as the training data set is linearly separable. The singularity can beavoided by inclusion of a prior and ﬁnding a MAP solution for w, or equivalently by
adding a regularization term to the error function.

4.3. Probabilistic Discriminative Models 207
4.3.3 Iterative reweighted least squares
In the case of the linear regression models discussed in Chapter 3, the maxi-
mum likelihood solution, on the assumption of a Gaussian noise model, leads to a
closed-form solution. This was a consequence of the quadratic dependence of the
log likelihood function on the parameter vector w. For logistic regression, there
is no longer a closed-form solution, due to the nonlinearity of the logistic sigmoid
function. However, the departure from a quadratic form is not substantial. To be
precise, the error function is concave, as we shall see shortly, and hence has a unique
minimum. Furthermore, the error function can be minimized by an efﬁcient iterative
technique based on the Newton-Raphson iterative optimization scheme, which uses a
local quadratic approximation to the log likelihood function. The Newton-Raphson
update, for minimizing a function E(w), takes the form (Fletcher, 1987; Bishop and
Nabney, 2008)

w(new)=w(old)−H−1∇E(w). (4.92)
whereHis the Hessian matrix whose elements comprise the second derivatives of
E(w)with respect to the components of w.
Let us ﬁrst of all apply the Newton-Raphson method to the linear regression
model (3.3) with the sum-of-squares error function (3.12). The gradient and Hessian
of this error function are given by
∇E(w)=N∑
n=1(wTφn−tn)φn=ΦTΦw−ΦTt (4.93)
H=∇∇E(w)=N∑
n=1φnφT
n=ΦTΦ (4.94)
whereΦis theN×Mdesign matrix, whose nthrow is given by φT
n. The Newton- Section 3.1.1

∇E(w)=N∑
n=1φnφT
Raphson update then takes the form
w(new)=w(old)−(ΦTΦ)−1{
ΦTΦw(old)−ΦTt}
=(ΦTΦ)−1ΦTt (4.95)
which we recognize as the standard least-squares solution. Note that the error func-
tion in this case is quadratic and hence the Newton-Raphson formula gives the exact
solution in one step.
Now let us apply the Newton-Raphson update to the cross-entropy error function
(4.90) for the logistic regression model. From (4.91) we see that the gradient and
Hessian of this error function are given by
∇E(w)=N∑

∇E(w)=N∑
n=1(yn−tn)φn=ΦT(y−t) (4.96)
n=1yn(1−yn)φnφT
n=ΦTRΦ (4.97)

208 4. LINEAR MODELS FOR CLASSIFICATION
where we have made use of (4.88). Also, we have introduced the N×Ndiagonal
matrixRwith elements
Rnn=yn(1−yn). (4.98)
We see that the Hessian is no longer constant but depends on wthrough the weight-
ing matrix R, corresponding to the fact that the error function is no longer quadratic.
Using the property 0<yn<1, which follows from the form of the logistic sigmoid
function, we see that uTHu>0for an arbitrary vector u, and so the Hessian matrix
His positive deﬁnite. It follows that the error function is a concave function of w
and hence has a unique minimum. Exercise 4.15
The Newton-Raphson update formula for the logistic regression model then be-
comes
w(new)=w(old)−(ΦTRΦ)−1ΦT(y−t)
=(ΦTRΦ)−1{
ΦTRΦw(old)−ΦT(y−t)}
=(ΦTRΦ)−1ΦTRz (4.99)
where zis anN-dimensional vector with elements
z=Φw(old)−R−1(y−t). (4.100)
We see that the update formula (4.99) takes the form of a set of normal equations for a

comes
weighted least-squares problem. Because the weighing matrix Ris not constant but
depends on the parameter vector w, we must apply the normal equations iteratively,
each time using the new weight vector wto compute a revised weighing matrix
R. For this reason, the algorithm is known as iterative reweighted least squares ,o r
IRLS (Rubin, 1983). As in the weighted least-squares problem, the elements of the
diagonal weighting matrix Rcan be interpreted as variances because the mean and
variance of tin the logistic regression model are given by

E[t]= σ(x)=y (4.101)
var[t]= E[t2]−E[t]2=σ(x)−σ(x)2=y(1−y) (4.102)
where we have used the property t2=tfort∈{0,1}. In fact, we can interpret IRLS
as the solution to a linearized problem in the space of the variable a=wTφ. The
quantity zn, which corresponds to the nthelement of z, can then be given a simple
interpretation as an effective target value in this space obtained by making a local
linear approximation to the logistic sigmoid function around the current operatingpointw
(old)
an(w)≃an(w(old))+dan
dyn⏐⏐⏐⏐
w(old)(tn−yn)
=φT
nw(old)−(yn−tn)

(old)
dyn⏐⏐⏐⏐
=φT
yn(1−yn)=zn. (4.103)

4.3. Probabilistic Discriminative Models 209
4.3.4 Multiclass logistic regression
In our discussion of generative models for multiclass classiﬁcation, we have Section 4.2
seen that for a large class of distributions, the posterior probabilities are given by a
softmax transformation of linear functions of the feature variables, so that
p(Ck|φ)=yk(φ)=exp(ak)∑
jexp(aj)(4.104)
where the ‘activations’ akare given by
ak=wT
kφ. (4.105)
There we used maximum likelihood to determine separately the class-conditional
densities and the class priors and then found the corresponding posterior probabilities
using Bayes’ theorem, thereby implicitly determining the parameters {wk}. Here we
consider the use of maximum likelihood to determine the parameters {wk}of this
model directly. To do this, we will require the derivatives of ykwith respect to all of
the activations aj. These are given by Exercise 4.17
∂yk
∂aj=yk(Ikj−yj) (4.106)
where Ikjare the elements of the identity matrix.

∂yk
Next we write down the likelihood function. This is most easily done using
the 1-of- Kcoding scheme in which the target vector tnfor a feature vector φn
belonging to class Ckis a binary vector with all elements zero except for element k,
which equals one. The likelihood function is then given by
p(T|w1,...,wK)=N∏
n=1K∏
k=1p(Ck|φn)tnk=N∏
n=1K∏
k=1ytnk
nk(4.107)
where ynk=yk(φn), andTis anN×Kmatrix of target variables with elements
tnk. Taking the negative logarithm then gives
E(w1,...,wK)=−lnp(T|w1,...,wK)=−N∑
n=1K∑
k=1tnklnynk (4.108)

n=1K∏
n=1K∏
k=1ytnk
nk(4.107)
n=1K∑
which is known as the cross-entropy error function for the multiclass classiﬁcation
problem.
We now take the gradient of the error function with respect to one of the param-
eter vectors wj. Making use of the result (4.106) for the derivatives of the softmax
function, we obtain Exercise 4.18
∇wjE(w1,...,wK)=N∑
n=1(ynj−tnj)φn (4.109)

210 4. LINEAR MODELS FOR CLASSIFICATION
where we have made use of∑
ktnk=1. Once again, we see the same form arising
for the gradient as was found for the sum-of-squares error function with the linearmodel and the cross-entropy error for the logistic regression model, namely the prod-
uct of the error (y
nj−tnj)times the basis function φn. Again, we could use this
to formulate a sequential algorithm in which patterns are presented one at a time, inwhich each of the weight vectors is updated using (3.22).
We have seen that the derivative of the log likelihood function for a linear regres-
sion model with respect to the parameter vector wfor a data point ntook the form
of the ‘error’ y
n−tntimes the feature vector φn. Similarly, for the combination
of logistic sigmoid activation function and cross-entropy error function (4.90), and
for the softmax activation function with the multiclass cross-entropy error function
(4.108), we again obtain this same simple form. This is an example of a more general

result, as we shall see in Section 4.3.6.
To ﬁnd a batch algorithm, we again appeal to the Newton-Raphson update to
obtain the corresponding IRLS algorithm for the multiclass problem. This requires
evaluation of the Hessian matrix that comprises blocks of size M×Min which
blockj,kis given by
∇wk∇wjE(w1,...,wK)=−N∑
n=1ynk(Ikj−ynj)φnφT
n. (4.110)
As with the two-class problem, the Hessian matrix for the multiclass logistic regres-

n. (4.110)
sion model is positive deﬁnite and so the error function again has a unique minimum. Exercise 4.20
Practical details of IRLS for the multiclass case can be found in Bishop and Nabney(2008).
4.3.5 Probit regression
We have seen that, for a broad range of class-conditional distributions, described
by the exponential family, the resulting posterior class probabilities are given by alogistic (or softmax) transformation acting on a linear function of the feature vari-
ables. However, not all choices of class-conditional density give rise to such a simple

form for the posterior probabilities (for instance, if the class-conditional densities aremodelled using Gaussian mixtures). This suggests that it might be worth exploring
other types of discriminative probabilistic model. For the purposes of this chapter,
however, we shall return to the two-class case, and again remain within the frame-work of generalized linear models so that
p(t=1|a)=f(a) (4.111)
where a=w
Tφ, andf(·)is the activation function.
One way to motivate an alternative choice for the link function is to consider a

where a=w
noisy threshold model, as follows. For each input φn, we evaluate an=wTφnand
then we set the target value according to
{tn=1 ifan⩾θ
tn=0 otherwise .(4.112)

4.3. Probabilistic Discriminative Models 211
Figure 4.13 Schematic example of a probability density p(θ)
shown by the blue curve, given in this example by a mixture
of two Gaussians, along with its cumulative distribution function
f(a), shown by the red curve. Note that the value of the blue
curve at any point, such as that indicated by the vertical green
line, corresponds to the slope of the red curve at the same point.
Conversely, the value of the red curve at this point corresponds
to the area under the blue curve indicated by the shaded green
region. In the stochastic threshold model, the class label takes
the value t=1if the value of a=wTφexceeds a threshold, oth-
erwise it takes the value t=0. This is equivalent to an activation
function given by the cumulative distribution function f(a).
0 1 2 3 400.20.40.60.81
If the value of θis drawn from a probability density p(θ), then the corresponding
activation function will be given by the cumulative distribution function
f(a)=∫a
−∞p(θ)dθ (4.113)

f(a)=∫a
as illustrated in Figure 4.13.
As a speciﬁc example, suppose that the density p(θ)is given by a zero mean,
unit variance Gaussian. The corresponding cumulative distribution function is given
by
Φ(a)=∫a
−∞N(θ|0,1) dθ (4.114)
which is known as the probit function. It has a sigmoidal shape and is compared
with the logistic sigmoid function in Figure 4.9. Note that the use of a more gen-
eral Gaussian distribution does not change the model because this is equivalent to
a re-scaling of the linear coefﬁcients w. Many numerical packages provide for the

by
Φ(a)=∫a
evaluation of a closely related function deﬁned by
erf(a)=2√π∫a
0exp(−θ2/2) dθ (4.115)
and known as the erf function orerror function (not to be confused with the error
function of a machine learning model). It is related to the probit function by Exercise 4.21
Φ(a)=1
2{
1+1√
2erf(a)}
. (4.116)
The generalized linear model based on a probit activation function is known as probit
regression .
We can determine the parameters of this model using maximum likelihood, by a
straightforward extension of the ideas discussed earlier. In practice, the results found

Φ(a)=1
2{
1+1√
2erf(a)}
. (4.116)
using probit regression tend to be similar to those of logistic regression. We shall,

212 4. LINEAR MODELS FOR CLASSIFICATION
however, ﬁnd another use for the probit model when we discuss Bayesian treatments
of logistic regression in Section 4.5.
One issue that can occur in practical applications is that of outliers , which can
arise for instance through errors in measuring the input vector xor through misla-
belling of the target value t. Because such points can lie a long way to the wrong side
of the ideal decision boundary, they can seriously distort the classiﬁer. Note that the
logistic and probit regression models behave differently in this respect because the
tails of the logistic sigmoid decay asymptotically like exp(−x)forx→∞ , whereas
for the probit activation function they decay like exp(−x2), and so the probit model
can be signiﬁcantly more sensitive to outliers.
However, both the logistic and the probit models assume the data is correctly
labelled. The effect of mislabelling is easily incorporated into a probabilistic model

by introducing a probability ϵthat the target value thas been ﬂipped to the wrong
value (Opper and Winther, 2000a), leading to a target value distribution for data point
xof the form
p(t|x)=( 1 −ϵ)σ(x)+ϵ(1−σ(x))
=ϵ+( 1−2ϵ)σ(x) (4.117)
where σ(x)is the activation function with input vector x. Here ϵmay be set in
advance, or it may be treated as a hyperparameter whose value is inferred from the
data.
4.3.6 Canonical link functions
For the linear regression model with a Gaussian noise distribution, the error

data.
function, corresponding to the negative log likelihood, is given by (3.12). If we takethe derivative with respect to the parameter vector wof the contribution to the error
function from a data point n, this takes the form of the ‘error’ y
n−tntimes the
feature vector φn, where yn=wTφn. Similarly, for the combination of the logistic
sigmoid activation function and the cross-entropy error function (4.90), and for the
softmax activation function with the multiclass cross-entropy error function (4.108),

we again obtain this same simple form. We now show that this is a general resultof assuming a conditional distribution for the target variable from the exponential
family, along with a corresponding choice for the activation function known as the
canonical link function .
We again make use of the restricted form (4.84) of exponential family distribu-

tions. Note that here we are applying the assumption of exponential family distribu-tion to the target variable t, in contrast to Section 4.2.4 where we applied it to the
input vector x. We therefore consider conditional distributions of the target variable
of the form
p(t|η,s)=1
sh(t
s)
g(η)e x p{ηt
s}
. (4.118)
Using the same line of argument as led to the derivation of the result (2.226), we see
that the conditional mean of t, which we denote by y, is given by
y≡E[t|η]=−sd
dηlng(η). (4.119)

4.4. The Laplace Approximation 213
Thusyandηmust related, and we denote this relation through η=ψ(y).
Following Nelder and Wedderburn (1972), we deﬁne a generalized linear model
to be one for which yis a nonlinear function of a linear combination of the input (or
feature) variables so that
y=f(wTφ) (4.120)
where f(·)is known as the activation function in the machine learning literature, and
f−1(·)is known as the link function in statistics.
Now consider the log likelihood function for this model, which, as a function of
η,i sg i v e nb y
lnp(t|η,s)=N∑
n=1lnp(tn|η,s)=N∑
n=1{
lng(ηn)+ηntn
s}
+c o n s t (4.121)
where we are assuming that all observations share a common scale parameter (which
corresponds to the noise variance for a Gaussian distribution for instance) and so s
is independent of n. The derivative of the log likelihood with respect to the model
parameters wis then given by
∇wlnp(t|η,s)=N∑
n=1{d
dηnlng(ηn)+tn
s}dηn
dyndyn
dan∇an
=N∑
n=11
s{tn−yn}ψ′(yn)f′(an)φn (4.122)

n=1{
s}
n=1{d
s}dηn
dyndyn
dan∇an
=N∑
n=11
where an=wTφn, and we have used yn=f(an)together with the result (4.119)
for E[t|η]. We now see that there is a considerable simpliﬁcation if we choose a
particular form for the link function f−1(y)given by
f−1(y)=ψ(y) (4.123)
which gives f(ψ(y)) =yand hence f′(ψ)ψ′(y)=1 . Also, because a=f−1(y),
we have a=ψand hence f′(a)ψ′(y)=1 . In this case, the gradient of the error
function reduces to
∇lnE(w)=1
sN∑
n=1{yn−tn}φn. (4.124)
For the Gaussian s=β−1, whereas for the logistic model s=1.
4.4. The Laplace Approximation

∇lnE(w)=1
sN∑
In Section 4.5 we shall discuss the Bayesian treatment of logistic regression. As
we shall see, this is more complex than the Bayesian treatment of linear regression
models, discussed in Sections 3.3 and 3.5. In particular, we cannot integrate exactly

214 4. LINEAR MODELS FOR CLASSIFICATION
over the parameter vector wsince the posterior distribution is no longer Gaussian.
It is therefore necessary to introduce some form of approximation. Later in thebook we shall consider a range of techniques based on analytical approximations Chapter 10
and numerical sampling. Chapter 11
Here we introduce a simple, but widely used, framework called the Laplace ap-
proximation, that aims to ﬁnd a Gaussian approximation to a probability density
deﬁned over a set of continuous variables. Consider ﬁrst the case of a single contin-
uous variable z, and suppose the distribution p(z)is deﬁned by
p(z)=1
Zf(z) (4.125)
where Z=∫
f(z)dzis the normalization coefﬁcient. We shall suppose that the
value of Zis unknown. In the Laplace method the goal is to ﬁnd a Gaussian approx-
imation q(z)which is centred on a mode of the distribution p(z). The ﬁrst step is to
ﬁnd a mode of p(z), in other words a point z0such that p′(z0)=0 , or equivalently
df(z)
dz⏐⏐⏐⏐
z=z0=0. (4.126)

p(z)=1
where Z=∫
df(z)
dz⏐⏐⏐⏐
A Gaussian distribution has the property that its logarithm is a quadratic function
of the variables. We therefore consider a Taylor expansion of lnf(z)centred on the
modez0so that
lnf(z)≃lnf(z0)−1
2A(z−z0)2(4.127)
where
A=−d2
dz2lnf(z)⏐⏐⏐⏐
z=z0. (4.128)
Note that the ﬁrst-order term in the Taylor expansion does not appear since z0is a
local maximum of the distribution. Taking the exponential we obtain
f(z)≃f(z0)e x p{
−A
2(z−z0)2}
. (4.129)
We can then obtain a normalized distribution q(z)by making use of the standard
result for the normalization of a Gaussian, so that
q(z)=(A

where
A=−d2
−A
2(z−z0)2}
. (4.129)
q(z)=(A
2π)1/2
exp{
−A
2(z−z0)2}
. (4.130)
The Laplace approximation is illustrated in Figure 4.14. Note that the Gaussian
approximation will only be well deﬁned if its precision A>0, in other words the
stationary point z0must be a local maximum, so that the second derivative of f(z)
at the point z0is negative.

4.4. The Laplace Approximation 215
−2 −1 0 1 2 3 400.20.40.60.8
−2 −1 0 1 2 3 4010203040
Figure 4.14 Illustration of the Laplace approximation applied to the distribution p(z)∝exp(−z2/2)σ(20z+4 )
where σ(z)is the logistic sigmoid function deﬁned by σ(z)=( 1+ e−z)−1. The left plot shows the normalized
distribution p(z)in yellow, together with the Laplace approximation centred on the mode z0ofp(z)in red. The
right plot shows the negative logarithms of the corresponding curves.
We can extend the Laplace method to approximate a distribution p(z)=f(z)/Z
deﬁned over an M-dimensional space z. At a stationary point z0the gradient ∇f(z)
will vanish. Expanding around this stationary point we have
2(z−z0)TA(z−z0) (4.131)
where the M×MHessian matrix Ais deﬁned by
A=−∇ ∇ lnf(z)|z=z0(4.132)
and∇is the gradient operator. Taking the exponential of both sides we obtain
−1
2(z−z0)TA(z−z0)}
. (4.133)
The distribution q(z)is proportional to f(z)and the appropriate normalization coef-

−1
. (4.133)
ﬁcient can be found by inspection, using the standard result (2.43) for a normalized
multivariate Gaussian, giving
q(z)=|A|1/2
(2π)M/2exp{
−1
=N(z|z0,A−1) (4.134)
where |A|denotes the determinant of A. This Gaussian distribution will be well
deﬁned provided its precision matrix, given by A, is positive deﬁnite, which implies
that the stationary point z0must be a local maximum, not a minimum or a saddle
point.
In order to apply the Laplace approximation we ﬁrst need to ﬁnd the mode z0,

−1
point.
and then evaluate the Hessian matrix at that mode. In practice a mode will typi-
cally be found by running some form of numerical optimization algorithm (Bishop

216 4. LINEAR MODELS FOR CLASSIFICATION
and Nabney, 2008). Many of the distributions encountered in practice will be mul-
timodal and so there will be different Laplace approximations according to which
mode is being considered. Note that the normalization constant Zof the true distri-
bution does not need to be known in order to apply the Laplace method. As a result
of the central limit theorem, the posterior distribution for a model is expected to
become increasingly better approximated by a Gaussian as the number of observed
data points is increased, and so we would expect the Laplace approximation to be
most useful in situations where the number of data points is relatively large.
One major weakness of the Laplace approximation is that, since it is based on a
Gaussian distribution, it is only directly applicable to real variables. In other cases
it may be possible to apply the Laplace approximation to a transformation of the
variable. For instance if 0⩽τ<∞then we can consider a Laplace approximation

oflnτ. The most serious limitation of the Laplace framework, however, is that
it is based purely on the aspects of the true distribution at a speciﬁc value of the
variable, and so can fail to capture important global properties. In Chapter 10 we
shall consider alternative approaches which adopt a more global perspective.
4.4.1 Model comparison and BIC
As well as approximating the distribution p(z)we can also obtain an approxi-
mation to the normalization constant Z. Using the approximation (4.133) we have
Z=∫
f(z)dz
≃f(z0)∫
exp{
−1

Z=∫
f(z)dz
≃f(z0)∫
exp{
−1
dz
=f(z0)(2π)M/2
|A|1/2(4.135)
where we have noted that the integrand is Gaussian and made use of the standard
result (2.43) for a normalized Gaussian distribution. We can use the result (4.135) to
obtain an approximation to the model evidence which, as discussed in Section 3.4,
plays a central role in Bayesian model comparison.
Consider a data set Dand a set of models {Mi}having parameters {θi}.F o r
each model we deﬁne a likelihood function p(D|θi,Mi). If we introduce a prior

exp{
−1
dz
p(θi|Mi)over the parameters, then we are interested in computing the model evi-
dence p(D|M i)for the various models. From now on we omit the conditioning on
Mito keep the notation uncluttered. From Bayes’ theorem the model evidence is
given by
p(D)=∫
p(D|θ)p(θ)dθ. (4.136)
Identifying f(θ)=p(D|θ)p(θ)andZ=p(D), and applying the result (4.135), we
obtain Exercise 4.22
lnp(D)≃lnp(D|θMAP)+lnp(θMAP)+M
2ln(2π)−1
2ln|A|
  
Occam factor(4.137)

4.5. Bayesian Logistic Regression 217
where θMAP is the value of θat the mode of the posterior distribution, and Ais the
Hessian matrix of second derivatives of the negative log posterior
A=−∇∇ lnp(D|θMAP)p(θMAP)=−∇∇ lnp(θMAP|D). (4.138)
The ﬁrst term on the right hand side of (4.137) represents the log likelihood evalu-
ated using the optimized parameters, while the remaining three terms comprise the‘Occam factor’ which penalizes model complexity.
If we assume that the Gaussian prior distribution over parameters is broad, and
that the Hessian has full rank, then we can approximate (4.137) very roughly using Exercise 4.23
lnp(D)≃lnp(D|θ
MAP)−1
2MlnN (4.139)
where Nis the number of data points, Mis the number of parameters in θand
we have omitted additive constants. This is known as the Bayesian Information
Criterion (BIC) or the Schwarz criterion (Schwarz, 1978). Note that, compared to
AIC given by (1.73), this penalizes model complexity more heavily.

MAP)−1
Complexity measures such as AIC and BIC have the virtue of being easy to
evaluate, but can also give misleading results. In particular, the assumption that the
Hessian matrix has full rank is often not valid since many of the parameters are not‘well-determined’. We can use the result (4.137) to obtain a more accurate estimate Section 3.5.3
of the model evidence starting from the Laplace approximation, as we illustrate in
the context of neural networks in Section 5.7.
4.5. Bayesian Logistic Regression

We now turn to a Bayesian treatment of logistic regression. Exact Bayesian infer-ence for logistic regression is intractable. In particular, evaluation of the posterior
distribution would require normalization of the product of a prior distribution and a
likelihood function that itself comprises a product of logistic sigmoid functions, onefor every data point. Evaluation of the predictive distribution is similarly intractable.
Here we consider the application of the Laplace approximation to the problem of

Bayesian logistic regression (Spiegelhalter and Lauritzen, 1990; MacKay, 1992b).
4.5.1 Laplace approximation
Recall from Section 4.4 that the Laplace approximation is obtained by ﬁnding
the mode of the posterior distribution and then ﬁtting a Gaussian centred at that
mode. This requires evaluation of the second derivatives of the log posterior, which
is equivalent to ﬁnding the Hessian matrix.
Because we seek a Gaussian representation for the posterior distribution, it is

natural to begin with a Gaussian prior, which we write in the general form
p(w)=N(w|m0,S0) (4.140)

218 4. LINEAR MODELS FOR CLASSIFICATION
wherem0andS0are ﬁxed hyperparameters. The posterior distribution over wis
given by
p(w|t)∝p(w)p(t|w) (4.141)
where t=(t1,...,t N)T. Taking the log of both sides, and substituting for the prior
distribution using (4.140), and for the likelihood function using (4.89), we obtain
lnp(w|t)= −1
2(w−m0)TS−1
0(w−m0)
+N∑
n=1{tnlnyn+( 1−tn)l n ( 1−yn)}+c o n s t (4.142)
where yn=σ(wTφn). To obtain a Gaussian approximation to the posterior dis-
tribution, we ﬁrst maximize the posterior distribution to give the MAP (maximumposterior) solution w
MAP, which deﬁnes the mean of the Gaussian. The covariance
is then given by the inverse of the matrix of second derivatives of the negative log
likelihood, which takes the form
SN=−∇∇ lnp(w|t)=S−1
0+N∑
n. (4.143)
The Gaussian approximation to the posterior distribution therefore takes the form
q(w)=N(w|wMAP,SN). (4.144)
Having obtained a Gaussian approximation to the posterior distribution, there

0+N∑
n. (4.143)
remains the task of marginalizing with respect to this distribution in order to make
4.5.2 Predictive distribution
The predictive distribution for class C1, given a new feature vector φ(x),i s
obtained by marginalizing with respect to the posterior distribution p(w|t), which is
itself approximated by a Gaussian distribution q(w)so that
p(C1|φ,t)=∫
p(C1|φ,w)p(w|t)dw≃∫
σ(wTφ)q(w)dw (4.145)
with the corresponding probability for class C2given by p(C2|φ,t)=1−p(C1|φ,t).
To evaluate the predictive distribution, we ﬁrst note that the function σ(wTφ)de-

pends on wonly through its projection onto φ. Denoting a=wTφ,w eh a v e
σ(wTφ)=∫
δ(a−wTφ)σ(a)da (4.146)
where δ(·)is the Dirac delta function. From this we obtain
∫
σ(wTφ)q(w)dw=∫
σ(a)p(a)da (4.147)

4.5. Bayesian Logistic Regression 219
where
p(a)=∫
δ(a−wTφ)q(w)dw. (4.148)
We can evaluate p(a)by noting that the delta function imposes a linear constraint
onwand so forms a marginal distribution from the joint distribution q(w)by inte-
grating out all directions orthogonal to φ. Because q(w)is Gaussian, we know from
Section 2.3.2 that the marginal distribution will also be Gaussian. We can evaluatethe mean and covariance of this distribution by taking moments, and interchanging
the order of integration over aandw, so that
µ
a=E[a]=∫
p(a)ada=∫
q(w)wTφdw=wT
MAPφ (4.149)
where we have used the result (4.144) for the variational posterior distribution q(w).
Similarly
σ2
a=v a r [ a]=∫
p(a){
a2−E[a]2}
da
=∫
q(w){
(wTφ)2−(mT
Nφ)2}
dw=φTSNφ. (4.150)
Note that the distribution of atakes the same form as the predictive distribution
(3.58) for the linear regression model, with the noise variance set to zero. Thus our
variational approximation to the predictive distribution becomes
p(C1|t)=∫
σ(a)p(a)da=∫
σ(a)N(a|µa,σ2

µ
a=E[a]=∫
p(a)ada=∫
Similarly
σ2
p(a){
a2−E[a]2}
da
=∫
q(w){
Nφ)2}
p(C1|t)=∫
a)da. (4.151)
This result can also be derived directly by making use of the results for the marginal
of a Gaussian distribution given in Section 2.3.2. Exercise 4.24
The integral over arepresents the convolution of a Gaussian with a logistic sig-
moid, and cannot be evaluated analytically. We can, however, obtain a good approx-imation (Spiegelhalter and Lauritzen, 1990; MacKay, 1992b; Barber and Bishop,
1998a) by making use of the close similarity between the logistic sigmoid function

σ(a)deﬁned by (4.59) and the probit function Φ(a)deﬁned by (4.114). In order to
obtain the best approximation to the logistic function we need to re-scale the hori-
zontal axis, so that we approximate σ(a)byΦ(λa). We can ﬁnd a suitable value of
λby requiring that the two functions have the same slope at the origin, which gives
λ
2=π/8. The similarity of the logistic sigmoid and the probit function, for this Exercise 4.25
choice of λ, is illustrated in Figure 4.9.

λ
The advantage of using a probit function is that its convolution with a Gaussian
can be expressed analytically in terms of another probit function. Speciﬁcally we
can show that Exercise 4.26
∫
Φ(λa)N(a|µ, σ2)da=Φ(µ
(λ−2+σ2)1/2)
. (4.152)

220 4. LINEAR MODELS FOR CLASSIFICATION
We now apply the approximation σ(a)≃Φ(λa)to the probit functions appearing
on both sides of this equation, leading to the following approximation for the convo-lution of a logistic sigmoid with a Gaussian
∫
σ(a)N(a|µ, σ2)da≃σ(
κ(σ2)µ)
(4.153)
κ(σ2) = (1 + πσ2/8)−1/2. (4.154)
Applying this result to (4.151) we obtain the approximate predictive distribution
in the form
p(C1|φ,t)=σ(
κ(σ2
a)µa)
(4.155)
where µaandσ2
aare deﬁned by (4.149) and (4.150), respectively, and κ(σ2
a)is de-
ﬁned by (4.154).
Note that the decision boundary corresponding to p(C1|φ,t)=0.5is given by
µa=0, which is the same as the decision boundary obtained by using the MAP
value for w. Thus if the decision criterion is based on minimizing misclassiﬁca-
tion rate, with equal prior probabilities, then the marginalization over whas no ef-

in the form
κ(σ2
a)µa)
(4.155)
a)is de-
fect. However, for more complex decision criteria it will play an important role.Marginalization of the logistic sigmoid model under a Gaussian approximation to
the posterior distribution will be illustrated in the context of variational inference in
Figure 10.13.
Exercises
4.1 (⋆⋆)Given a set of data points {xn}, we can deﬁne the convex hull to be the set of
all points xgiven by
x=∑
nαnxn (4.156)
where αn⩾0and∑
nαn=1. Consider a second set of points {yn}together with
their corresponding convex hull. By deﬁnition, the two sets of points will be linearly

Exercises
x=∑
separable if there exists a vector ˆwand a scalar w0such thatˆwTxn+w0>0for all
xn, andˆwTyn+w0<0for allyn. Show that if their convex hulls intersect, the two
sets of points cannot be linearly separable, and conversely that if they are linearly
separable, their convex hulls do not intersect.
4.2 (⋆⋆)www Consider the minimization of a sum-of-squares error function (4.15),
and suppose that all of the target vectors in the training set satisfy a linear constraint
aTtn+b=0 (4.157)
wheretncorresponds to the nthrow of the matrix Tin (4.15). Show that as a

consequence of this constraint, the elements of the model prediction y(x)given by
the least-squares solution (4.17) also satisfy this constraint, so that
aTy(x)+b=0. (4.158)

Exercises 221
To do so, assume that one of the basis functions φ0(x)=1 so that the corresponding
parameter w0plays the role of a bias.
4.3 (⋆⋆)Extend the result of Exercise 4.2 to show that if multiple linear constraints
are satisﬁed simultaneously by the target vectors, then the same constraints will also
be satisﬁed by the least-squares prediction of a linear model.
4.4 (⋆)www Show that maximization of the class separation criterion given by (4.23)
with respect to w, using a Lagrange multiplier to enforce the constraint wTw=1,
leads to the result that w∝(m2−m1).
4.5 (⋆)By making use of (4.20), (4.23), and (4.24), show that the Fisher criterion (4.25)
can be written in the form (4.26).
4.6 (⋆)Using the deﬁnitions of the between-class and within-class covariance matrices
given by (4.27) and (4.28), respectively, together with (4.34) and (4.36) and the
choice of target values described in Section 4.1.5, show that the expression (4.33)

that minimizes the sum-of-squares error function can be written in the form (4.37).
4.7 (⋆)www Show that the logistic sigmoid function (4.59) satisﬁes the property
σ(−a)=1−σ(a)and that its inverse is given by σ−1(y)=l n {y/(1−y)}.
4.8 (⋆)Using (4.57) and (4.58), derive the result (4.65) for the posterior class probability
in the two-class generative model with Gaussian densities, and verify the results(4.66) and (4.67) for the parameters wandw
0.
4.9 (⋆)www Consider a generative classiﬁcation model for Kclasses deﬁned by

0.
prior class probabilities p(Ck)=πkand general class-conditional densities p(φ|Ck)
where φis the input feature vector. Suppose we are given a training data set {φn,tn}
where n=1,...,N , andtnis a binary target vector of length Kthat uses the 1-of-
Kcoding scheme, so that it has components tnj=Ijkif pattern nis from class Ck.
Assuming that the data points are drawn independently from this model, show thatthe maximum-likelihood solution for the prior probabilities is given by
π
k=Nk
N(4.159)
where Nkis the number of data points assigned to class Ck.

π
k=Nk
N(4.159)
4.10 (⋆⋆)Consider the classiﬁcation model of Exercise 4.9 and now suppose that the
class-conditional densities are given by Gaussian distributions with a shared covari-
ance matrix, so that
p(φ|Ck)=N(φ|µk,Σ). (4.160)
Show that the maximum likelihood solution for the mean of the Gaussian distribution
for class Ckis given by
µk=1
NkN∑
n=1tnkφn (4.161)

222 4. LINEAR MODELS FOR CLASSIFICATION
which represents the mean of those feature vectors assigned to class Ck. Similarly,
show that the maximum likelihood solution for the shared covariance matrix is givenby
Σ=K∑
k=1Nk
NSk (4.162)
where
Sk=1
NkN∑
n=1tnk(φn−µk)(φn−µk)T. (4.163)
ThusΣis given by a weighted average of the covariances of the data associated with
each class, in which the weighting coefﬁcients are given by the prior probabilities ofthe classes.
4.11 (⋆⋆)Consider a classiﬁcation problem with Kclasses for which the feature vector
φhasMcomponents each of which can take Ldiscrete states. Let the values of the
components be represented by a 1-of- Lbinary coding scheme. Further suppose that,
conditioned on the class C
k, theMcomponents of φare independent, so that the
class-conditional density factorizes with respect to the feature vector components.Show that the quantities a
kgiven by (4.63), which appear in the argument to the

softmax function describing the posterior class probabilities, are linear functions of
the components of φ. Note that this represents an example of the naive Bayes model
which is discussed in Section 8.2.2.
4.12 (⋆)www Verify the relation (4.88) for the derivative of the logistic sigmoid func-
tion deﬁned by (4.59).
4.13 (⋆)www By making use of the result (4.88) for the derivative of the logistic sig-
moid, show that the derivative of the error function (4.90) for the logistic regressionmodel is given by (4.91).

4.14 (⋆)Show that for a linearly separable data set, the maximum likelihood solution
for the logistic regression model is obtained by ﬁnding a vector wwhose decision
boundary w
Tφ(x)=0 separates the classes and then taking the magnitude of wto
inﬁnity.
4.15 (⋆⋆)Show that the Hessian matrix Hfor the logistic regression model, given by
(4.97), is positive deﬁnite. Here Ris a diagonal matrix with elements yn(1−yn),
andynis the output of the logistic regression model for input vector xn. Hence show
that the error function is a concave function of wand that it has a unique minimum.

boundary w
inﬁnity.
4.16 (⋆)Consider a binary classiﬁcation problem in which each observation xnis known
to belong to one of two classes, corresponding to t=0andt=1, and suppose that
the procedure for collecting training data is imperfect, so that training points are
sometimes mislabelled. For every data point xn, instead of having a value tfor the
class label, we have instead a value πnrepresenting the probability that tn=1.
Given a probabilistic model p(t=1|φ), write down the log likelihood function
appropriate to such a data set.

Exercises 223
4.17 (⋆)www Show that the derivatives of the softmax activation function (4.104),
where the akare deﬁned by (4.105), are given by (4.106).
4.18 (⋆)Using the result (4.91) for the derivatives of the softmax activation function,
show that the gradients of the cross-entropy error (4.108) are given by (4.109).
4.19 (⋆)www Write down expressions for the gradient of the log likelihood, as well
as the corresponding Hessian matrix, for the probit regression model deﬁned in Sec-
tion 4.3.5. These are the quantities that would be required to train such a model usingIRLS.
4.20 (⋆⋆)Show that the Hessian matrix for the multiclass logistic regression problem,
deﬁned by (4.110), is positive semideﬁnite. Note that the full Hessian matrix for
this problem is of size MK×MK , where Mis the number of parameters and K
is the number of classes. To prove the positive semideﬁnite property, consider the
product u
THuwhereuis an arbitrary vector of length MK , and then apply Jensen’s
inequality.

product u
4.21 (⋆)Show that the probit function (4.114) and the erf function (4.115) are related by
(4.116).
4.22 (⋆)Using the result (4.135), derive the expression (4.137) for the log model evi-
dence under the Laplace approximation.
4.23 (⋆⋆)www In this exercise, we derive the BIC result (4.139) starting from the
Laplace approximation to the model evidence given by (4.137). Show that if the
prior over parameters is Gaussian of the form p(θ)=N(θ|m,V0), the log model

(4.116).
evidence under the Laplace approximation takes the form
lnp(D)≃lnp(D|θMAP)−1
2(θMAP−m)TV−1
0(θMAP−m)−1
2ln|H|+c o n s t
whereHis the matrix of second derivatives of the log likelihood lnp(D|θ)evaluated
atθMAP. Now assume that the prior is broad so that V−1
0is small and the second
term on the right-hand side above can be neglected. Furthermore, consider the case
of independent, identically distributed data so that His the sum of terms one for each
data point. Show that the log model evidence can then be written approximately in

the form of the BIC expression (4.139).
4.24 (⋆⋆)Use the results from Section 2.3.2 to derive the result (4.151) for the marginal-
ization of the logistic regression model with respect to a Gaussian posterior distribu-
tion over the parameters w.
4.25 (⋆⋆)Suppose we wish to approximate the logistic sigmoid σ(a)deﬁned by (4.59)
by a scaled probit function Φ(λa), where Φ(a)is deﬁned by (4.114). Show that if
λis chosen so that the derivatives of the two functions are equal at a=0, then
λ2=π/8.

224 4. LINEAR MODELS FOR CLASSIFICATION
4.26 (⋆⋆)In this exercise, we prove the relation (4.152) for the convolution of a probit
function with a Gaussian distribution. To do this, show that the derivative of the left-hand side with respect to µis equal to the derivative of the right-hand side, and then
integrate both sides with respect to µand then show that the constant of integration
vanishes. Note that before differentiating the left-hand side, it is convenient ﬁrstto introduce a change of variable given by a=µ+σzso that the integral over a
is replaced by an integral over z. When we differentiate the left-hand side of the
relation (4.152), we will then obtain a Gaussian integral over zthat can be evaluated
analytically.

5
Neural
Networks
In Chapters 3 and 4 we considered models for regression and classiﬁcation that com-
prised linear combinations of ﬁxed basis functions. We saw that such models haveuseful analytical and computational properties but that their practical applicability
was limited by the curse of dimensionality. In order to apply such models to large-
scale problems, it is necessary to adapt the basis functions to the data.
Support vector machines (SVMs), discussed in Chapter 7, address this by ﬁrst
deﬁning basis functions that are centred on the training data points and then selecting
a subset of these during training. One advantage of SVMs is that, although thetraining involves nonlinear optimization, the objective function is convex, and so the
solution of the optimization problem is relatively straightforward. The number of
basis functions in the resulting models is generally much smaller than the number oftraining points, although it is often still relatively large and typically increases with

the size of the training set. The relevance vector machine, discussed in Section 7.2,
also chooses a subset from a ﬁxed set of basis functions and typically results in much
225

226 5. NEURAL NETWORKS
sparser models. Unlike the SVM it also produces probabilistic outputs, although this
is at the expense of a nonconvex optimization during training.
An alternative approach is to ﬁx the number of basis functions in advance but
allow them to be adaptive, in other words to use parametric forms for the basis func-
tions in which the parameter values are adapted during training. The most successfulmodel of this type in the context of pattern recognition is the feed-forward neural
network, also known as the multilayer perceptron , discussed in this chapter. In fact,
‘multilayer perceptron’ is really a misnomer, because the model comprises multi-ple layers of logistic regression models (with continuous nonlinearities) rather than
multiple perceptrons (with discontinuous nonlinearities). For many applications, the
resulting model can be signiﬁcantly more compact, and hence faster to evaluate, than
a support vector machine having the same generalization performance. The price to

be paid for this compactness, as with the relevance vector machine, is that the like-lihood function, which forms the basis for network training, is no longer a convex
function of the model parameters. In practice, however, it is often worth investing
substantial computational resources during the training phase in order to obtain acompact model that is fast at processing new data.
The term ‘neural network’ has its origins in attempts to ﬁnd mathematical rep-

resentations of information processing in biological systems (McCulloch and Pitts,1943; Widrow and Hoff, 1960; Rosenblatt, 1962; Rumelhart et al. , 1986). Indeed,
it has been used very broadly to cover a wide range of different models, many of
which have been the subject of exaggerated claims regarding their biological plau-sibility. From the perspective of practical applications of pattern recognition, how-
ever, biological realism would impose entirely unnecessary constraints. Our focus in

this chapter is therefore on neural networks as efﬁcient models for statistical patternrecognition. In particular, we shall restrict our attention to the speciﬁc class of neu-
ral networks that have proven to be of greatest practical value, namely the multilayer
perceptron.
We begin by considering the functional form of the network model, including
the speciﬁc parameterization of the basis functions, and we then discuss the prob-lem of determining the network parameters within a maximum likelihood frame-

work, which involves the solution of a nonlinear optimization problem. This requires
the evaluation of derivatives of the log likelihood function with respect to the net-work parameters, and we shall see how these can be obtained efﬁciently using the
technique of error backpropagation . We shall also show how the backpropagation
framework can be extended to allow other derivatives to be evaluated, such as theJacobian and Hessian matrices. Next we discuss various approaches to regulariza-

tion of neural network training and the relationships between them. We also consider
some extensions to the neural network model, and in particular we describe a gen-eral framework for modelling conditional probability distributions known as mixture
density networks . Finally, we discuss the use of Bayesian treatments of neural net-
works. Additional background on neural network models can be found in Bishop(1995a).

5.1. Feed-forward Network Functions 227
5.1. Feed-forward Network Functions
The linear models for regression and classiﬁcation discussed in Chapters 3 and 4, re-
spectively, are based on linear combinations of ﬁxed nonlinear basis functions φj(x)
and take the form
y(x,w)=f(M∑
j=1wjφj(x))
(5.1)
where f(·)is a nonlinear activation function in the case of classiﬁcation and is the
identity in the case of regression. Our goal is to extend this model by making the
basis functions φj(x)depend on parameters and then to allow these parameters to
be adjusted, along with the coefﬁcients {wj}, during training. There are, of course,
many ways to construct parametric nonlinear basis functions. Neural networks use
basis functions that follow the same form as (5.1), so that each basis function is itselfa nonlinear function of a linear combination of the inputs, where the coefﬁcients in
the linear combination are adaptive parameters.
This leads to the basic neural network model, which can be described a series

of functional transformations. First we construct Mlinear combinations of the input
variables x
1,...,x Din the form
aj=D∑
i=1w(1)
jixi+w(1)
j0 (5.2)
where j=1,...,M , and the superscript (1)indicates that the corresponding param-
eters are in the ﬁrst ‘layer’ of the network. We shall refer to the parameters w(1)
jias
weights and the parameters w(1)
j0asbiases , following the nomenclature of Chapter 3.
The quantities ajare known as activations . Each of them is then transformed using
a differentiable, nonlinear activation function h(·)to give

aj=D∑
i=1w(1)
jixi+w(1)
j0 (5.2)
jias
zj=h(aj). (5.3)
These quantities correspond to the outputs of the basis functions in (5.1) that, in the
context of neural networks, are called hidden units . The nonlinear functions h(·)are
generally chosen to be sigmoidal functions such as the logistic sigmoid or the ‘ tanh ’
function. Following (5.1), these values are again linearly combined to give output Exercise 5.1
unit activations
ak=M∑
j=1w(2)
kjzj+w(2)
k0(5.4)
where k=1,...,K , andKis the total number of outputs. This transformation cor-

ak=M∑
j=1w(2)
kjzj+w(2)
k0(5.4)
responds to the second layer of the network, and again the w(2)
k0are bias parameters.
Finally, the output unit activations are transformed using an appropriate activationfunction to give a set of network outputs y
k. The choice of activation function is
determined by the nature of the data and the assumed distribution of target variables

228 5. NEURAL NETWORKS
Figure 5.1 Network diagram for the two-
layer neural network corre-
sponding to (5.7). The input,
hidden, and output variables
are represented by nodes, and
the weight parameters are rep-
resented by links between the
nodes, in which the bias pa-
rameters are denoted by links
coming from additional input
and hidden variables x0and
z0. Arrows denote the direc-
tion of information ﬂow through
the network during forward
propagation.x0x1xD
z0z1zM
y1yKw(1)
MDw(2)
KM
w(2)
10hidden units
inputs outputs
and follows the same considerations as for linear models discussed in Chapters 3 and
4. Thus for standard regression problems, the activation function is the identity so
thatyk=ak. Similarly, for multiple binary classiﬁcation problems, each output unit
activation is transformed using a logistic sigmoid function so that
yk=σ(ak) (5.5)
where
σ(a)=1
1+e x p ( −a). (5.6)
Finally, for multiclass problems, a softmax activation function of the form (4.62)

y1yKw(1)
MDw(2)
KM
w(2)
where
σ(a)=1
is used. The choice of output unit activation function is discussed in detail in Sec-
tion 5.2.
We can combine these various stages to give the overall network function that,
for sigmoidal output unit activation functions, takes the form
yk(x,w)=σ(M∑
j=1w(2)
kjh(D∑
i=1w(1)
jixi+w(1)
j0)
+w(2)
k0)
(5.7)
where the set of all weight and bias parameters have been grouped together into a
vectorw. Thus the neural network model is simply a nonlinear function from a set

tion 5.2.
j=1w(2)
kjh(D∑
i=1w(1)
jixi+w(1)
j0)
+w(2)
k0)
(5.7)
of input variables {xi}to a set of output variables {yk}controlled by a vector wof
adjustable parameters.
This function can be represented in the form of a network diagram as shown
in Figure 5.1. The process of evaluating (5.7) can then be interpreted as a forward
propagation of information through the network. It should be emphasized that these
diagrams do not represent probabilistic graphical models of the kind to be consid-
ered in Chapter 8 because the internal nodes represent deterministic variables rather

than stochastic ones. For this reason, we have adopted a slightly different graphical

5.1. Feed-forward Network Functions 229
notation for the two kinds of model. We shall see later how to give a probabilistic
interpretation to a neural network.
As discussed in Section 3.1, the bias parameters in (5.2) can be absorbed into
the set of weight parameters by deﬁning an additional input variable x0whose value
is clamped at x0=1, so that (5.2) takes the form
aj=D∑
i=0w(1)
jixi. (5.8)
We can similarly absorb the second-layer biases into the second-layer weights, so
that the overall network function becomes
j=0w(2)
kjh(D∑
i=0w(1)
jixi))
. (5.9)
As can be seen from Figure 5.1, the neural network model comprises two stages
of processing, each of which resembles the perceptron model of Section 4.1.7, and
for this reason the neural network is also known as the multilayer perceptron ,o r
MLP. A key difference compared to the perceptron, however, is that the neural net-
work uses continuous sigmoidal nonlinearities in the hidden units, whereas the per-

j=0w(2)
kjh(D∑
i=0w(1)
jixi))
. (5.9)
ceptron uses step-function nonlinearities. This means that the neural network func-tion is differentiable with respect to the network parameters, and this property will
play a central role in network training.
If the activation functions of all the hidden units in a network are taken to be
linear, then for any such network we can always ﬁnd an equivalent network without
hidden units. This follows from the fact that the composition of successive linear

transformations is itself a linear transformation. However, if the number of hiddenunits is smaller than either the number of input or output units, then the transforma-
tions that the network can generate are not the most general possible linear trans-
formations from inputs to outputs because information is lost in the dimensionalityreduction at the hidden units. In Section 12.4.2, we show that networks of linear
units give rise to principal component analysis. In general, however, there is little
interest in multilayer networks of linear units.

The network architecture shown in Figure 5.1 is the most commonly used one
in practice. However, it is easily generalized, for instance by considering additional
layers of processing each consisting of a weighted linear combination of the form(5.4) followed by an element-wise transformation using a nonlinear activation func-
tion. Note that there is some confusion in the literature regarding the terminology
for counting the number of layers in such networks. Thus the network in Figure 5.1
may be described as a 3-layer network (which counts the number of layers of units,

and treats the inputs as units) or sometimes as a single-hidden-layer network (whichcounts the number of layers of hidden units). We recommend a terminology in which
Figure 5.1 is called a two-layer network, because it is the number of layers of adap-
tive weights that is important for determining the network properties.
Another generalization of the network architecture is to include skip-layer con-
nections, each of which is associated with a corresponding adaptive parameter. For

230 5. NEURAL NETWORKS
Figure 5.2 Example of a neural network having a
general feed-forward topology. Note that
each hidden and output unit has an
associated bias parameter (omitted for
clarity).
x1x2
z1
z3z2
y1y2
instance, in a two-layer network these would go directly from inputs to outputs. In
principle, a network with sigmoidal hidden units can always mimic skip layer con-
nections (for bounded input values) by using a sufﬁciently small ﬁrst-layer weight
that, over its operating range, the hidden unit is effectively linear, and then com-
pensating with a large weight value from the hidden unit to the output. In practice,
however, it may be advantageous to include skip-layer connections explicitly.
Furthermore, the network can be sparse, with not all possible connections within
a layer being present. We shall see an example of a sparse network architecture when
we consider convolutional neural networks in Section 5.5.6.

Because there is a direct correspondence between a network diagram and its
mathematical function, we can develop more general network mappings by con-
sidering more complex network diagrams. However, these must be restricted to a
feed-forward architecture, in other words to one having no closed directed cycles, to
ensure that the outputs are deterministic functions of the inputs. This is illustrated
with a simple example in Figure 5.2. Each (hidden or output) unit in such a network
computes a function given by
zk=h(∑
jwkjzj)
(5.10)

zk=h(∑
jwkjzj)
(5.10)
where the sum runs over all units that send connections to unit k(and a bias param-
eter is included in the summation). For a given set of values applied to the inputs of
the network, successive application of (5.10) allows the activations of all units in the
network to be evaluated including those of the output units.
The approximation properties of feed-forward networks have been widely stud-
ied (Funahashi, 1989; Cybenko, 1989; Hornik et al. , 1989; Stinchecombe and White,

zk=h(∑
jwkjzj)
(5.10)
1989; Cotter, 1990; Ito, 1991; Hornik, 1991; Kreinovich, 1991; Ripley, 1996) and
found to be very general. Neural networks are therefore said to be universal ap-
proximators . For example, a two-layer network with linear outputs can uniformly
approximate any continuous function on a compact input domain to arbitrary accu-
racy provided the network has a sufﬁciently large number of hidden units. This result
holds for a wide range of hidden unit activation functions, but excluding polynomi-

als. Although such theorems are reassuring, the key problem is how to ﬁnd suitable
parameter values given a set of training data, and in later sections of this chapter we

5.1. Feed-forward Network Functions 231
Figure 5.3 Illustration of the ca-
pability of a multilayer perceptron
to approximate four different func-
tions comprising (a) f(x)=x2, (b)
f(x)=s i n ( x),( c ) , f(x)= |x|,
and (d) f(x)=H(x)where H(x)
is the Heaviside step function. Ineach case, N=5 0 data points,
shown as blue dots, have been sam-pled uniformly in xover the interval
(−1,1)and the corresponding val-
ues of f(x)evaluated. These data
points are then used to train a two-layer network having 3 hidden unitswith ‘tanh’ activation functions and
linear output units. The resultingnetwork functions are shown by thered curves, and the outputs of thethree hidden units are shown by thethree dashed curves. (a) (b)
(c) (d)
will show that there exist effective solutions to this problem based on both maximum
likelihood and Bayesian approaches.
The capability of a two-layer network to model a broad range of functions is
illustrated in Figure 5.3. This ﬁgure also shows how individual hidden units work

(c) (d)
collaboratively to approximate the ﬁnal function. The role of hidden units in a simple
classiﬁcation problem is illustrated in Figure 5.4 using the synthetic classiﬁcationdata set described in Appendix A.
5.1.1 Weight-space symmetries
One property of feed-forward networks, which will play a role when we consider
Bayesian model comparison, is that multiple distinct choices for the weight vector
wcan all give rise to the same mapping function from inputs to outputs (Chen et al. ,
1993). Consider a two-layer network of the form shown in Figure 5.1 with Mhidden

units having ‘ tanh ’ activation functions and full connectivity in both layers. If we
change the sign of all of the weights and the bias feeding into a particular hiddenunit, then, for a given input pattern, the sign of the activation of the hidden unit will
be reversed, because ‘ tanh ’ is an odd function, so that tanh(−a)=−tanh(a). This
transformation can be exactly compensated by changing the sign of all of the weightsleading out of that hidden unit. Thus, by changing the signs of a particular group of

weights (and a bias), the input–output mapping function represented by the network
is unchanged, and so we have found two different weight vectors that give rise tothe same mapping function. For Mhidden units, there will be Msuch ‘sign-ﬂip’

232 5. NEURAL NETWORKS
Figure 5.4 Example of the solution of a simple two-
class classiﬁcation problem involving
synthetic data using a neural network
having two inputs, two hidden units with
‘tanh’ activation functions, and a single
output having a logistic sigmoid activa-
tion function. The dashed blue lines
show the z=0.5contours for each of
the hidden units, and the red line shows
they=0.5decision surface for the net-
work. For comparison, the green line
denotes the optimal decision boundary
computed from the distributions used to
generate the data.
−2 −1 0 1 2−2−10123
symmetries, and thus any given weight vector will be one of a set 2Mequivalent
weight vectors .
Similarly, imagine that we interchange the values of all of the weights (and the
bias) leading both into and out of a particular hidden unit with the corresponding
values of the weights (and bias) associated with a different hidden unit. Again, this
clearly leaves the network input–output mapping function unchanged, but it corre-

sponds to a different choice of weight vector. For Mhidden units, any given weight
vector will belong to a set of M!equivalent weight vectors associated with this inter-
change symmetry, corresponding to the M!different orderings of the hidden units.
The network will therefore have an overall weight-space symmetry factor of M!2M.
For networks with more than two layers of weights, the total level of symmetry will
be given by the product of such factors, one for each layer of hidden units.

It turns out that these factors account for all of the symmetries in weight space
(except for possible accidental symmetries due to speciﬁc choices for the weight val-
ues). Furthermore, the existence of these symmetries is not a particular property of
the ‘tanh ’ function but applies to a wide range of activation functions (K ˙urkov ´a and
Kainen, 1994). In many cases, these symmetries in weight space are of little practi-
cal consequence, although in Section 5.7 we shall encounter a situation in which we

need to take them into account.
5.2. Network Training
So far, we have viewed neural networks as a general class of parametric nonlinear
functions from a vector xof input variables to a vector yof output variables. A
simple approach to the problem of determining the network parameters is to make an
analogy with the discussion of polynomial curve ﬁtting in Section 1.1, and therefore
to minimize a sum-of-squares error function. Given a training set comprising a set

of input vectors {xn}, where n=1,...,N , together with a corresponding set of

5.2. Network Training 233
target vectors {tn}, we minimize the error function
E(w)=1
2N∑
n=1∥y(xn,w)−tn∥2. (5.11)
However, we can provide a much more general view of network training by ﬁrst
giving a probabilistic interpretation to the network outputs. We have already seenmany advantages of using probabilistic predictions in Section 1.5.4. Here it will also
provide us with a clearer motivation both for the choice of output unit nonlinearity
and the choice of error function.
We start by discussing regression problems, and for the moment we consider
a single target variable tthat can take any real value. Following the discussions
in Section 1.2.5 and 3.1, we assume that thas a Gaussian distribution with an x-
dependent mean, which is given by the output of the neural network, so that
p(t|x,w)=N(
(5.12)
where βis the precision (inverse variance) of the Gaussian noise. Of course this
is a somewhat restrictive assumption, and in Section 5.6 we shall see how to extend

(5.12)
this approach to allow for more general conditional distributions. For the conditionaldistribution given by (5.12), it is sufﬁcient to take the output unit activation function
to be the identity, because such a network can approximate any continuous function
fromxtoy. Given a data set of Nindependent, identically distributed observations
X={x
1,...,xN}, along with corresponding target values t={t1,...,t N},w e
can construct the corresponding likelihood function
n=1p(tn|xn,w,β).

X={x
Taking the negative logarithm, we obtain the error function
β
2N∑
n=1{y(xn,w)−tn}2−N
2lnβ+N
2ln(2π) (5.13)
which can be used to learn the parameters wandβ. In Section 5.7, we shall dis-
cuss the Bayesian treatment of neural networks, while here we consider a maximum
likelihood approach. Note that in the neural networks literature, it is usual to con-sider the minimization of an error function rather than the maximization of the (log)
likelihood, and so here we shall follow this convention. Consider ﬁrst the determi-

β
2N∑
2lnβ+N
nation of w. Maximizing the likelihood function is equivalent to minimizing the
sum-of-squares error function given by
E(w)=1
2N∑
n=1{y(xn,w)−tn}2(5.14)

234 5. NEURAL NETWORKS
where we have discarded additive and multiplicative constants. The value of wfound
by minimizing E(w)will be denoted wMLbecause it corresponds to the maximum
likelihood solution. In practice, the nonlinearity of the network function y(xn,w)
causes the error E(w)to be nonconvex, and so in practice local maxima of the
likelihood may be found, corresponding to local minima of the error function, asdiscussed in Section 5.2.1.
Having found w
ML, the value of βcan be found by minimizing the negative log
likelihood to give
1
βML=1
NN∑
n=1{y(xn,wML)−tn}2. (5.15)
Note that this can be evaluated once the iterative optimization required to ﬁnd wML
is completed. If we have multiple target variables, and we assume that they are inde-
pendent conditional on xandwwith shared noise precision β, then the conditional
distribution of the target values is given by
t|y(x,w),β−1I)
. (5.16)
Following the same argument as for a single target variable, we see that the maximum

1
βML=1
NN∑
. (5.16)
likelihood weights are determined by minimizing the sum-of-squares error function(5.11). The noise precision is then given by Exercise 5.2
1
βML=1
NKN∑
n=1∥y(xn,wML)−tn∥2(5.17)
where Kis the number of target variables. The assumption of independence can be
dropped at the expense of a slightly more complex optimization problem. Exercise 5.3
Recall from Section 4.3.6 that there is a natural pairing of the error function
(given by the negative log likelihood) and the output unit activation function. In the

1
βML=1
NKN∑
regression case, we can view the network as having an output activation function that
is the identity, so that yk=ak. The corresponding sum-of-squares error function
has the property
∂E
∂ak=yk−tk (5.18)
which we shall make use of when discussing error backpropagation in Section 5.3.
Now consider the case of binary classiﬁcation in which we have a single target
variable tsuch that t=1 denotes class C1andt=0 denotes class C2. Following

∂E
the discussion of canonical link functions in Section 4.3.6, we consider a network
having a single output whose activation function is a logistic sigmoid
y=σ(a)≡1
1+e x p ( −a)(5.19)
so that 0⩽y(x,w)⩽1. We can interpret y(x,w)as the conditional probability
p(C1|x), with p(C2|x)given by 1−y(x,w). The conditional distribution of targets
given inputs is then a Bernoulli distribution of the form
p(t|x,w)=y(x,w)t{1−y(x,w)}1−t. (5.20)

5.2. Network Training 235
If we consider a training set of independent observations, then the error function,
which is given by the negative log likelihood, is then a cross-entropy error function
of the form
E(w)=−N∑
n=1{tnlnyn+( 1−tn)l n ( 1−yn)} (5.21)
where yndenotes y(xn,w). Note that there is no analogue of the noise precision β
because the target values are assumed to be correctly labelled. However, the model
is easily extended to allow for labelling errors. Simard et al. (2003) found that using Exercise 5.4
the cross-entropy error function instead of the sum-of-squares for a classiﬁcation
problem leads to faster training as well as improved generalization.
If we have Kseparate binary classiﬁcations to perform, then we can use a net-
work having Koutputs each of which has a logistic sigmoid activation function.
Associated with each output is a binary class label tk∈{0,1}, where k=1,...,K .

If we assume that the class labels are independent, given the input vector, then theconditional distribution of the targets is
p(t|x,w)=K∏
k=1yk(x,w)tk[1−yk(x,w)]1−tk. (5.22)
Taking the negative logarithm of the corresponding likelihood function then gives
the following error function Exercise 5.5
E(w)=−N∑
n=1K∑
k=1{tnklnynk+( 1−tnk)l n ( 1−ynk)} (5.23)
where ynkdenotes yk(xn,w). Again, the derivative of the error function with re-

E(w)=−N∑
n=1K∑
spect to the activation for a particular output unit takes the form (5.18) just as in the Exercise 5.6
regression case.
It is interesting to contrast the neural network solution to this problem with the
corresponding approach based on a linear classiﬁcation model of the kind discussed
in Chapter 4. Suppose that we are using a standard two-layer network of the kindshown in Figure 5.1. We see that the weight parameters in the ﬁrst layer of the
network are shared between the various outputs, whereas in the linear model each

classiﬁcation problem is solved independently. The ﬁrst layer of the network canbe viewed as performing a nonlinear feature extraction, and the sharing of features
between the different outputs can save on computation and can also lead to improved
generalization.
Finally, we consider the standard multiclass classiﬁcation problem in which each
input is assigned to one of Kmutually exclusive classes. The binary target variables
t
k∈{0,1}have a 1-of- Kcoding scheme indicating the class, and the network
outputs are interpreted as yk(x,w)=p(tk=1|x), leading to the following error
function
E(w)=−N∑

t
function
E(w)=−N∑
n=1K∑
k=1tknlnyk(xn,w). (5.24)

236 5. NEURAL NETWORKS
Figure 5.5 Geometrical view of the error function E(w)as
a surface sitting over weight space. Point wAis
a local minimum and wBis the global minimum.
At any point wC, the local gradient of the error
surface is given by the vector ∇E.
w1
w2E(w)
wAwBwC
∇E
Following the discussion of Section 4.3.4, we see that the output unit activation
function, which corresponds to the canonical link, is given by the softmax function
yk(x,w)=exp(ak(x,w))∑
jexp(aj(x,w))(5.25)
which satisﬁes 0⩽yk⩽1and∑
kyk=1. Note that the yk(x,w)are unchanged
if a constant is added to all of the ak(x,w), causing the error function to be constant
for some directions in weight space. This degeneracy is removed if an appropriate
regularization term (Section 5.5) is added to the error function.
Once again, the derivative of the error function with respect to the activation for
a particular output unit takes the familiar form (5.18). Exercise 5.7
In summary, there is a natural choice of both output unit activation function

and matching error function, according to the type of problem being solved. For re-
gression we use linear outputs and a sum-of-squares error, for (multiple independent)
binary classiﬁcations we use logistic sigmoid outputs and a cross-entropy error func-
tion, and for multiclass classiﬁcation we use softmax outputs with the corresponding
multiclass cross-entropy error function. For classiﬁcation problems involving two
classes, we can use a single logistic sigmoid output, or alternatively we can use a

network with two outputs having a softmax output activation function.
5.2.1 Parameter optimization
We turn next to the task of ﬁnding a weight vector wwhich minimizes the
chosen function E(w). At this point, it is useful to have a geometrical picture of the
error function, which we can view as a surface sitting over weight space as shown in
Figure 5.5. First note that if we make a small step in weight space from wtow+δw
then the change in the error function is δE≃δwT∇E(w), where the vector ∇E(w)

points in the direction of greatest rate of increase of the error function. Because the
errorE(w)is a smooth continuous function of w, its smallest value will occur at a

5.2. Network Training 237
point in weight space such that the gradient of the error function vanishes, so that
∇E(w)=0 (5.26)
as otherwise we could make a small step in the direction of −∇E(w)and thereby
further reduce the error. Points at which the gradient vanishes are called stationarypoints, and may be further classiﬁed into minima, maxima, and saddle points.
Our goal is to ﬁnd a vector wsuch that E(w)takes its smallest value. How-
ever, the error function typically has a highly nonlinear dependence on the weightsand bias parameters, and so there will be many points in weight space at which the
gradient vanishes (or is numerically very small). Indeed, from the discussion in Sec-
tion 5.1.1 we see that for any point wthat is a local minimum, there will be other
points in weight space that are equivalent minima. For instance, in a two-layer net-
work of the kind shown in Figure 5.1, with Mhidden units, each point in weight
space is a member of a family of M!2
Mequivalent points. Section 5.1.1

Furthermore, there will typically be multiple inequivalent stationary points and
in particular multiple inequivalent minima. A minimum that corresponds to thesmallest value of the error function for any weight vector is said to be a global
minimum . Any other minima corresponding to higher values of the error function
are said to be local minima . For a successful application of neural networks, it may
not be necessary to ﬁnd the global minimum (and in general it will not be known
whether the global minimum has been found) but it may be necessary to compare

several local minima in order to ﬁnd a sufﬁciently good solution.
Because there is clearly no hope of ﬁnding an analytical solution to the equa-
tion∇E(w)=0 we resort to iterative numerical procedures. The optimization of
continuous nonlinear functions is a widely studied problem and there exists an ex-tensive literature on how to solve it efﬁciently. Most techniques involve choosing
some initial value w
(0)for the weight vector and then moving through weight space
in a succession of steps of the form
w(τ+1)=w(τ)+∆w(τ)(5.27)

where τlabels the iteration step. Different algorithms involve different choices for
the weight vector update ∆w(τ). Many algorithms make use of gradient information
and therefore require that, after each update, the value of ∇E(w)is evaluated at
the new weight vector w(τ+1). In order to understand the importance of gradient
information, it is useful to consider a local approximation to the error function based
on a Taylor expansion.
5.2.2 Local quadratic approximation
Insight into the optimization problem, and into the various techniques for solv-

ing it, can be obtained by considering a local quadratic approximation to the errorfunction.
Consider the Taylor expansion of E(w)around some point
ˆwin weight space
E(w)≃E(ˆw)+(w−ˆw)Tb+1
2(w−ˆw)TH(w−ˆw) (5.28)

238 5. NEURAL NETWORKS
where cubic and higher terms have been omitted. Here bis deﬁned to be the gradient
ofEevaluated at ˆw
b≡∇E|w=bw(5.29)
and the Hessian matrix H=∇∇Ehas elements
(H)ij≡∂E
∂wi∂wj⏐⏐⏐⏐
w=bw. (5.30)
From (5.28), the corresponding local approximation to the gradient is given by
∇E≃b+H(w−ˆw). (5.31)
For points wthat are sufﬁciently close to ˆw, these expressions will give reasonable
approximations for the error and its gradient.
Consider the particular case of a local quadratic approximation around a point
w⋆that is a minimum of the error function. In this case there is no linear term,
because ∇E=0atw⋆, and (5.28) becomes
E(w)=E(w⋆)+1
2(w−w⋆)TH(w−w⋆) (5.32)
where the Hessian His evaluated at w⋆. In order to interpret this geometrically,
consider the eigenvalue equation for the Hessian matrix
Hui=λiui (5.33)
where the eigenvectors uiform a complete orthonormal set (Appendix C) so that
uT
iuj=δij. (5.34)
We now expand (w−w⋆)as a linear combination of the eigenvectors in the form
w−w⋆=∑

uT
w−w⋆=∑
iαiui. (5.35)
This can be regarded as a transformation of the coordinate system in which the origin
is translated to the point w⋆, and the axes are rotated to align with the eigenvectors
(through the orthogonal matrix whose columns are the ui), and is discussed in more
detail in Appendix C. Substituting (5.35) into (5.32), and using (5.33) and (5.34),
allows the error function to be written in the form
2∑
iλiα2
i. (5.36)
A matrix His said to be positive deﬁnite if, and only if,
vTHv>0 for allv. (5.37)

5.2. Network Training 239
Figure 5.6 In the neighbourhood of a min-
imum w⋆, the error function
can be approximated by a
quadratic. Contours of con-
stant error are then ellipses
whose axes are aligned with
the eigenvectors uiof the Hes-
sian matrix, with lengths that
are inversely proportional to the
square roots of the correspond-
ing eigenvectors λi.
w1w2
λ−1/2
1λ−1/2
2u1
w⋆u2
Because the eigenvectors {ui}form a complete set, an arbitrary vector vcan be
written in the form
v=∑
iciui. (5.38)
From (5.33) and (5.34), we then have
vTHv=∑
ic2
iλi (5.39)
and so Hwill be positive deﬁnite if, and only if, all of its eigenvalues are positive. Exercise 5.10
In the new coordinate system, whose basis vectors are given by the eigenvectors
{ui}, the contours of constant Eare ellipses centred on the origin, as illustrated Exercise 5.11
in Figure 5.6. For a one-dimensional weight space, a stationary point w⋆will be a
minimum if
∂2E
∂w2⏐⏐⏐⏐
w⋆>0. (5.40)

v=∑
vTHv=∑
ic2
iλi (5.39)
minimum if
∂2E
∂w2⏐⏐⏐⏐
The corresponding result in D-dimensions is that the Hessian matrix, evaluated at
w⋆, should be positive deﬁnite. Exercise 5.12
5.2.3 Use of gradient information
As we shall see in Section 5.3, it is possible to evaluate the gradient of an error
function efﬁciently by means of the backpropagation procedure. The use of this
gradient information can lead to signiﬁcant improvements in the speed with which
the minima of the error function can be located. We can see why this is so, as follows.

In the quadratic approximation to the error function, given in (5.28), the error
surface is speciﬁed by the quantities bandH, which contain a total of W(W+
3)/2independent elements (because the matrix His symmetric), where Wis the Exercise 5.13
dimensionality of w(i.e., the total number of adaptive parameters in the network).
The location of the minimum of this quadratic approximation therefore depends on
O(W2)parameters, and we should not expect to be able to locate the minimum until

we have gathered O(W2)independent pieces of information. If we do not make
use of gradient information, we would expect to have to perform O(W2)function

240 5. NEURAL NETWORKS
evaluations, each of which would require O(W)steps. Thus, the computational
effort needed to ﬁnd the minimum using such an approach would be O(W3).
Now compare this with an algorithm that makes use of the gradient information.
Because each evaluation of ∇Ebrings Witems of information, we might hope to
ﬁnd the minimum of the function in O(W)gradient evaluations. As we shall see,
by using error backpropagation, each such evaluation takes only O(W)steps and so
the minimum can now be found in O(W2)steps. For this reason, the use of gradient
information forms the basis of practical algorithms for training neural networks.
5.2.4 Gradient descent optimization
The simplest approach to using gradient information is to choose the weight
update in (5.27) to comprise a small step in the direction of the negative gradient, sothat
w
(τ+1)=w(τ)−η∇E(w(τ)) (5.41)
where the parameter η>0is known as the learning rate . After each such update, the

w
gradient is re-evaluated for the new weight vector and the process repeated. Note that
the error function is deﬁned with respect to a training set, and so each step requiresthat the entire training set be processed in order to evaluate ∇E. Techniques that
use the whole data set at once are called batch methods. At each step the weight
vector is moved in the direction of the greatest rate of decrease of the error function,and so this approach is known as gradient descent orsteepest descent . Although

such an approach might intuitively seem reasonable, in fact it turns out to be a poor
algorithm, for reasons discussed in Bishop and Nabney (2008).
For batch optimization, there are more efﬁcient methods, such as conjugate gra-
dients and quasi-Newton methods, which are much more robust and much faster
than simple gradient descent (Gill et al. , 1981; Fletcher, 1987; Nocedal and Wright,
1999). Unlike gradient descent, these algorithms have the property that the error

function always decreases at each iteration unless the weight vector has arrived at a
local or global minimum.
In order to ﬁnd a sufﬁciently good minimum, it may be necessary to run a
gradient-based algorithm multiple times, each time using a different randomly cho-
sen starting point, and comparing the resulting performance on an independent vali-dation set.
There is, however, an on-line version of gradient descent that has proved useful
in practice for training neural networks on large data sets (Le Cun et al. , 1989).

Error functions based on maximum likelihood for a set of independent observations
comprise a sum of terms, one for each data point
E(w)=
N∑
n=1En(w). (5.42)
On-line gradient descent, also known as sequential gradient descent orstochastic
gradient descent , makes an update to the weight vector based on one data point at a
time, so that
w(τ+1)=w(τ)−η∇En(w(τ)). (5.43)

5.3. Error Backpropagation 241
This update is repeated by cycling through the data either in sequence or by selecting
points at random with replacement. There are of course intermediate scenarios inwhich the updates are based on batches of data points.
One advantage of on-line methods compared to batch methods is that the former
handle redundancy in the data much more efﬁciently. To see, this consider an ex-treme example in which we take a data set and double its size by duplicating every
data point. Note that this simply multiplies the error function by a factor of 2 and so
is equivalent to using the original error function. Batch methods will require doublethe computational effort to evaluate the batch error function gradient, whereas on-
line methods will be unaffected. Another property of on-line gradient descent is the
possibility of escaping from local minima, since a stationary point with respect to
the error function for the whole data set will generally not be a stationary point for

each data point individually.
Nonlinear optimization algorithms, and their practical application to neural net-
work training, are discussed in detail in Bishop and Nabney (2008).
5.3. Error Backpropagation
Our goal in this section is to ﬁnd an efﬁcient technique for evaluating the gradient
of an error function E(w)for a feed-forward neural network. We shall see that
this can be achieved using a local message passing scheme in which information is
sent alternately forwards and backwards through the network and is known as error
backpropagation , or sometimes simply as backprop .

It should be noted that the term backpropagation is used in the neural com-
puting literature to mean a variety of different things. For instance, the multilayer
perceptron architecture is sometimes called a backpropagation network. The term
backpropagation is also used to describe the training of a multilayer perceptron us-ing gradient descent applied to a sum-of-squares error function. In order to clarify
the terminology, it is useful to consider the nature of the training process more care-

fully. Most training algorithms involve an iterative procedure for minimization of anerror function, with adjustments to the weights being made in a sequence of steps. At
each such step, we can distinguish between two distinct stages. In the ﬁrst stage, the
derivatives of the error function with respect to the weights must be evaluated. Aswe shall see, the important contribution of the backpropagation technique is in pro-
viding a computationally efﬁcient method for evaluating such derivatives. Because

it is at this stage that errors are propagated backwards through the network, we shalluse the term backpropagation speciﬁcally to describe the evaluation of derivatives.
In the second stage, the derivatives are then used to compute the adjustments to be
made to the weights. The simplest such technique, and the one originally consideredby Rumelhart et al. (1986), involves gradient descent. It is important to recognize
that the two stages are distinct. Thus, the ﬁrst stage, namely the propagation of er-

rors backwards through the network in order to evaluate derivatives, can be applied
to many other kinds of network and not just the multilayer perceptron. It can also be
applied to error functions other that just the simple sum-of-squares, and to the eval-

242 5. NEURAL NETWORKS
uation of other derivatives such as the Jacobian and Hessian matrices, as we shall
see later in this chapter. Similarly, the second stage of weight adjustment using thecalculated derivatives can be tackled using a variety of optimization schemes, many
of which are substantially more powerful than simple gradient descent.
5.3.1 Evaluation of error-function derivatives
We now derive the backpropagation algorithm for a general network having ar-
bitrary feed-forward topology, arbitrary differentiable nonlinear activation functions,
and a broad class of error function. The resulting formulae will then be illustrated
using a simple layered network structure having a single layer of sigmoidal hidden
units together with a sum-of-squares error.
Many error functions of practical interest, for instance those deﬁned by maxi-
mum likelihood for a set of i.i.d. data, comprise a sum of terms, one for each data
point in the training set, so that
E(w)=N∑
n=1En(w). (5.44)

E(w)=N∑
Here we shall consider the problem of evaluating ∇En(w)for one such term in the
error function. This may be used directly for sequential optimization, or the results
can be accumulated over the training set in the case of batch methods.
Consider ﬁrst a simple linear model in which the outputs ykare linear combina-
tions of the input variables xiso that
yk=∑
iwkixi (5.45)
together with an error function that, for a particular input pattern n, takes the form
En=1
2∑
k(ynk−tnk)2(5.46)
where ynk=yk(xn,w). The gradient of this error function with respect to a weight
wjiis given by
∂En

yk=∑
En=1
2∑
∂En
∂wji=(ynj−tnj)xni (5.47)
which can be interpreted as a ‘local’ computation involving the product of an ‘error
signal’ ynj−tnjassociated with the output end of the link wjiand the variable xni
associated with the input end of the link. In Section 4.3.2, we saw how a similar
formula arises with the logistic sigmoid activation function together with the crossentropy error function, and similarly for the softmax activation function together

∂En
with its matching cross-entropy error function. We shall now see how this simple
result extends to the more complex setting of multilayer feed-forward networks.
In a general feed-forward network, each unit computes a weighted sum of its
inputs of the form
a
j=∑
iwjizi (5.48)

5.3. Error Backpropagation 243
where ziis the activation of a unit, or input, that sends a connection to unit j, andwji
is the weight associated with that connection. In Section 5.1, we saw that biases can
be included in this sum by introducing an extra unit, or input, with activation ﬁxed
at+1. We therefore do not need to deal with biases explicitly. The sum in (5.48) is
transformed by a nonlinear activation function h(·)to give the activation zjof unit j
in the form
zj=h(aj). (5.49)
Note that one or more of the variables ziin the sum in (5.48) could be an input, and
similarly, the unit jin (5.49) could be an output.
For each pattern in the training set, we shall suppose that we have supplied the
corresponding input vector to the network and calculated the activations of all of
the hidden and output units in the network by successive application of (5.48) and
(5.49). This process is often called forward propagation because it can be regarded
as a forward ﬂow of information through the network.

Now consider the evaluation of the derivative of Enwith respect to a weight
wji. The outputs of the various units will depend on the particular input pattern n.
However, in order to keep the notation uncluttered, we shall omit the subscript n
from the network variables. First we note that Endepends on the weight wjionly
via the summed input ajto unit j. We can therefore apply the chain rule for partial
derivatives to give
∂En
∂wji=∂En
∂aj∂aj
∂wji. (5.50)
We now introduce a useful notation
δj≡∂En
∂aj(5.51)

∂En
∂wji=∂En
∂aj∂aj
δj≡∂En
∂aj(5.51)
where the δ’s are often referred to as errors for reasons we shall see shortly. Using
(5.48), we can write
∂aj
∂wji=zi. (5.52)
Substituting (5.51) and (5.52) into (5.50), we then obtain
∂En
∂wji=δjzi. (5.53)
Equation (5.53) tells us that the required derivative is obtained simply by multiplying
the value of δfor the unit at the output end of the weight by the value of zfor the unit
at the input end of the weight (where z=1in the case of a bias). Note that this takes

δj≡∂En
∂aj(5.51)
∂aj
∂En
the same form as for the simple linear model considered at the start of this section.Thus, in order to evaluate the derivatives, we need only to calculate the value of δ
j
for each hidden and output unit in the network, and then apply (5.53).
As we have seen already, for the output units, we have
δk=yk−tk (5.54)

244 5. NEURAL NETWORKS
Figure 5.7 Illustration of the calculation of δjfor hidden unit jby
backpropagation of the δ’s from those units kto which
unitjsends connections. The blue arrow denotes the
direction of information ﬂow during forward propagation,
and the red arrows indicate the backward propagation
of error information.zi
zjδjδk
δ1wji wkj
provided we are using the canonical link as the output-unit activation function. To
evaluate the δ’s for hidden units, we again make use of the chain rule for partial
derivatives,
δj≡∂En
∂aj=∑
k∂En
∂ak∂ak
∂aj(5.55)
where the sum runs over all units kto which unit jsends connections. The arrange-
ment of units and weights is illustrated in Figure 5.7. Note that the units labelled k
could include other hidden units and/or output units. In writing down (5.55), we are
making use of the fact that variations in ajgive rise to variations in the error func-
tion only through variations in the variables ak. If we now substitute the deﬁnition

δj≡∂En
∂aj=∑
k∂En
∂ak∂ak
∂aj(5.55)
ofδgiven by (5.51) into (5.55), and make use of (5.48) and (5.49), we obtain the
following backpropagation formula
δj=h′(aj)∑
kwkjδk (5.56)
which tells us that the value of δfor a particular hidden unit can be obtained by
propagating the δ’s backwards from units higher up in the network, as illustrated
in Figure 5.7. Note that the summation in (5.56) is taken over the ﬁrst index on
wkj(corresponding to backward propagation of information through the network),
whereas in the forward propagation equation (5.10) it is taken over the second index.

Because we already know the values of the δ’s for the output units, it follows that
by recursively applying (5.56) we can evaluate the δ’s for all of the hidden units in a
feed-forward network, regardless of its topology.
The backpropagation procedure can therefore be summarized as follows.
Error Backpropagation
1. Apply an input vector xnto the network and forward propagate through
the network using (5.48) and (5.49) to ﬁnd the activations of all the hidden
and output units.
2. Evaluate the δkfor all the output units using (5.54).

3. Backpropagate the δ’s using (5.56) to obtain δjfor each hidden unit in the
network.
4. Use (5.53) to evaluate the required derivatives.

5.3. Error Backpropagation 245
For batch methods, the derivative of the total error Ecan then be obtained by
repeating the above steps for each pattern in the training set and then summing overall patterns:
∂E
∂wji=∑
n∂En
∂wji. (5.57)
In the above derivation we have implicitly assumed that each hidden or output unit in
the network has the same activation function h(·). The derivation is easily general-
ized, however, to allow different units to have individual activation functions, simply
by keeping track of which form of h(·)goes with which unit.
5.3.2 A simple example
The above derivation of the backpropagation procedure allowed for general
forms for the error function, the activation functions, and the network topology. In
order to illustrate the application of this algorithm, we shall consider a particularexample. This is chosen both for its simplicity and for its practical importance, be-
cause many applications of neural networks reported in the literature make use of

this type of network. Speciﬁcally, we shall consider a two-layer network of the formillustrated in Figure 5.1, together with a sum-of-squares error, in which the output
units have linear activation functions, so that y
k=ak, while the hidden units have
logistic sigmoid activation functions given by
h(a)≡tanh(a) (5.58)
where
tanh(a)=ea−e−a
ea+e−a. (5.59)
A useful feature of this function is that its derivative can be expressed in a par-
ticularly simple form:
h′(a)=1−h(a)2. (5.60)

where
We also consider a standard sum-of-squares error function, so that for pattern nthe
error is given by
En=1
2K∑
k=1(yk−tk)2(5.61)
where ykis the activation of output unit k, andtkis the corresponding target, for a
particular input pattern xn.
For each pattern in the training set in turn, we ﬁrst perform a forward propagation
using
aj=D∑
i=0w(1)
jixi (5.62)
zj=t a n h ( aj) (5.63)
yk=M∑
j=0w(2)
kjzj. (5.64)

246 5. NEURAL NETWORKS
Next we compute the δ’s for each output unit using
δk=yk−tk. (5.65)
Then we backpropagate these to obtain δs for the hidden units using
δj=( 1−z2
j)K∑
k=1wkjδk. (5.66)
Finally, the derivatives with respect to the ﬁrst-layer and second-layer weights are
given by
∂En
∂w(1)
ji=δjxi,∂En
∂w(2)
kj=δkzj. (5.67)
5.3.3 Efﬁciency of backpropagation
One of the most important aspects of backpropagation is its computational efﬁ-
ciency. To understand this, let us examine how the number of computer operations
required to evaluate the derivatives of the error function scales with the total number
Wof weights and biases in the network. A single evaluation of the error function
(for a given input pattern) would require O(W)operations, for sufﬁciently large W.
This follows from the fact that, except for a network with very sparse connections,the number of weights is typically much greater than the number of units, and so the

bulk of the computational effort in forward propagation is concerned with evaluat-
ing the sums in (5.48), with the evaluation of the activation functions representing asmall overhead. Each term in the sum in (5.48) requires one multiplication and one
addition, leading to an overall computational cost that is O(W).
An alternative approach to backpropagation for computing the derivatives of the
error function is to use ﬁnite differences. This can be done by perturbing each weight

in turn, and approximating the derivatives by the expression
∂E
n
∂wji=En(wji+ϵ)−En(wji)
ϵ+O(ϵ) (5.68)
where ϵ≪1. In a software simulation, the accuracy of the approximation to the
derivatives can be improved by making ϵsmaller, until numerical roundoff problems
arise. The accuracy of the ﬁnite differences method can be improved signiﬁcantly
by using symmetrical central differences of the form
∂En
∂wji=En(wji+ϵ)−En(wji−ϵ)
2ϵ+O(ϵ2). (5.69)
In this case, the O(ϵ)corrections cancel, as can be veriﬁed by Taylor expansion on Exercise 5.14

∂E
n
∂En
the right-hand side of (5.69), and so the residual corrections are O(ϵ2). The number
of computational steps is, however, roughly doubled compared with (5.68).
The main problem with numerical differentiation is that the highly desirable
O(W)scaling has been lost. Each forward propagation requires O(W)steps, and

5.3. Error Backpropagation 247
Figure 5.8 Illustration of a modular pattern
recognition system in which the
Jacobian matrix can be used
to backpropagate error signals
from the outputs through to ear-
lier modules in the system.
xu
wy
zv
there are Wweights in the network each of which must be perturbed individually, so
that the overall scaling is O(W2).
However, numerical differentiation plays an important role in practice, because a
comparison of the derivatives calculated by backpropagation with those obtained us-
ing central differences provides a powerful check on the correctness of any software
implementation of the backpropagation algorithm. When training networks in prac-
tice, derivatives should be evaluated using backpropagation, because this gives the
greatest accuracy and numerical efﬁciency. However, the results should be compared
with numerical differentiation using (5.69) for some test cases in order to check the
correctness of the implementation.
5.3.4 The Jacobian matrix

We have seen how the derivatives of an error function with respect to the weights
can be obtained by the propagation of errors backwards through the network. The
technique of backpropagation can also be applied to the calculation of other deriva-
tives. Here we consider the evaluation of the Jacobian matrix, whose elements are
given by the derivatives of the network outputs with respect to the inputs
Jki≡∂yk
∂xi(5.70)
where each such derivative is evaluated with all other inputs held ﬁxed. Jacobian

Jki≡∂yk
∂xi(5.70)
matrices play a useful role in systems built from a number of distinct modules, as
illustrated in Figure 5.8. Each module can comprise a ﬁxed or adaptive function,
which can be linear or nonlinear, so long as it is differentiable. Suppose we wish
to minimize an error function Ewith respect to the parameter win Figure 5.8. The
derivative of the error function is given by
∂E
∂w=∑
k,j∂E
∂yk∂yk
∂zj∂zj
∂w(5.71)
in which the Jacobian matrix for the red module in Figure 5.8 appears in the middle
term.

∂E
∂w=∑
k,j∂E
∂yk∂yk
∂zj∂zj
∂w(5.71)
term.
Because the Jacobian matrix provides a measure of the local sensitivity of the
outputs to changes in each of the input variables, it also allows any known errors ∆xi

248 5. NEURAL NETWORKS
associated with the inputs to be propagated through the trained network in order to
estimate their contribution ∆ykto the errors at the outputs, through the relation
∆yk≃∑
i∂yk
∂xi∆xi (5.72)
which is valid provided the |∆xi|are small. In general, the network mapping rep-
resented by a trained neural network will be nonlinear, and so the elements of theJacobian matrix will not be constants but will depend on the particular input vector
used. Thus (5.72) is valid only for small perturbations of the inputs, and the Jacobian
itself must be re-evaluated for each new input vector.
The Jacobian matrix can be evaluated using a backpropagation procedure that is
similar to the one derived earlier for evaluating the derivatives of an error function
with respect to the weights. We start by writing the element J
kiin the form
Jki=∂yk
∂xi=∑
j∂yk
∂aj∂aj
∂xi
=∑
jwji∂yk
∂aj(5.73)
where we have made use of (5.48). The sum in (5.73) runs over all units jto which

Jki=∂yk
∂xi=∑
j∂yk
∂aj∂aj
∂xi
=∑
jwji∂yk
∂aj(5.73)
the input unit isends connections (for example, over all units in the ﬁrst hidden
layer in the layered topology considered earlier). We now write down a recursivebackpropagation formula to determine the derivatives ∂y
k/∂aj
∂yk
∂aj=∑
l∂yk
∂al∂al
∂aj
=h′(aj)∑
lwlj∂yk
∂al(5.74)
where the sum runs over all units lto which unit jsends connections (corresponding
to the ﬁrst index of wlj). Again, we have made use of (5.48) and (5.49). This
backpropagation starts at the output units for which the required derivatives can be

k/∂aj
∂yk
∂aj=∑
l∂yk
∂al∂al
∂aj
=h′(aj)∑
lwlj∂yk
∂al(5.74)
found directly from the functional form of the output-unit activation function. For
instance, if we have individual sigmoidal activation functions at each output unit,then
∂y
k
∂aj=δkjσ′(aj) (5.75)
whereas for softmax outputs we have
∂yk
∂aj=δkjyk−ykyj. (5.76)
We can summarize the procedure for evaluating the Jacobian matrix as follows.
Apply the input vector corresponding to the point in input space at which the Ja-
cobian matrix is to be found, and forward propagate in the usual way to obtain the

5.4. The Hessian Matrix 249
activations of all of the hidden and output units in the network. Next, for each row
kof the Jacobian matrix, corresponding to the output unit k, backpropagate using
the recursive relation (5.74), starting with (5.75) or (5.76), for all of the hidden units
in the network. Finally, use (5.73) to do the backpropagation to the inputs. The
Jacobian can also be evaluated using an alternative forward propagation formalism,
which can be derived in an analogous way to the backpropagation approach given
here. Exercise 5.15
Again, the implementation of such algorithms can be checked by using numeri-
cal differentiation in the form
∂yk
∂xi=yk(xi+ϵ)−yk(xi−ϵ)
2ϵ+O(ϵ2) (5.77)
which involves 2Dforward propagations for a network having Dinputs.
5.4. The Hessian Matrix
We have shown how the technique of backpropagation can be used to obtain the ﬁrst
derivatives of an error function with respect to the weights in the network. Back-

∂yk
propagation can also be used to evaluate the second derivatives of the error, given
by
∂2E
∂wji∂wlk. (5.78)
Note that it is sometimes convenient to consider all of the weight and bias parameters
as elements wiof a single vector, denoted w, in which case the second derivatives
form the elements Hijof the Hessian matrixH, where i, j∈{1,...,W }andWis
the total number of weights and biases. The Hessian plays an important role in manyaspects of neural computing, including the following:

by
∂2E
1. Several nonlinear optimization algorithms used for training neural networks
are based on considerations of the second-order properties of the error surface,
which are controlled by the Hessian matrix (Bishop and Nabney, 2008).
2. The Hessian forms the basis of a fast procedure for re-training a feed-forward
network following a small change in the training data (Bishop, 1991).
3. The inverse of the Hessian has been used to identify the least signiﬁcant weights

in a network as part of network ‘pruning’ algorithms (Le Cun et al. , 1990).
4. The Hessian plays a central role in the Laplace approximation for a Bayesian
neural network (see Section 5.7). Its inverse is used to determine the predic-
tive distribution for a trained network, its eigenvalues determine the values of
hyperparameters, and its determinant is used to evaluate the model evidence.
Various approximation schemes have been used to evaluate the Hessian matrix
for a neural network. However, the Hessian can also be calculated exactly using an

extension of the backpropagation technique.

250 5. NEURAL NETWORKS
An important consideration for many applications of the Hessian is the efﬁciency
with which it can be evaluated. If there are Wparameters (weights and biases) in the
network, then the Hessian matrix has dimensions W×Wand so the computational
effort needed to evaluate the Hessian will scale like O(W2)for each pattern in the
data set. As we shall see, there are efﬁcient methods for evaluating the Hessianwhose scaling is indeed O(W
2).
5.4.1 Diagonal approximation
Some of the applications for the Hessian matrix discussed above require the
inverse of the Hessian, rather than the Hessian itself. For this reason, there has
been some interest in using a diagonal approximation to the Hessian, in other wordsone that simply replaces the off-diagonal elements with zeros, because its inverse is
trivial to evaluate. Again, we shall consider an error function that consists of a sum
of terms, one for each pattern in the data set, so that E=∑
nEn. The Hessian can

then be obtained by considering one pattern at a time, and then summing the results
over all patterns. From (5.48), the diagonal elements of the Hessian, for pattern n,
can be written
∂2En
∂w2
ji=∂2En
∂a2
jz2
i. (5.79)
Using (5.48) and (5.49), the second derivatives on the right-hand side of (5.79) can
be found recursively using the chain rule of differential calculus to give a backprop-
agation equation of the form
∂2En
∂a2
j=h′(aj)2∑
k∑
k′wkjwk′j∂2En
∂ak∂ak′+h′′(aj)∑
kwkj∂En
∂ak. (5.80)

∂2En
∂w2
ji=∂2En
∂a2
jz2
i. (5.79)
∂2En
∂a2
k∑
kwkj∂En
If we now neglect off-diagonal elements in the second-derivative terms, we obtain
(Becker and Le Cun, 1989; Le Cun et al. , 1990)
∂2En
∂a2
kw2
kj∂2En
∂a2
k+h′′(aj)∑
kwkj∂En
∂ak. (5.81)
Note that the number of computational steps required to evaluate this approximation
isO(W), where Wis the total number of weight and bias parameters in the network,
compared with O(W2)for the full Hessian.
Ricotti et al. (1988) also used the diagonal approximation to the Hessian, but

kwkj∂En
∂2En
∂a2
kw2
kj∂2En
∂a2
kwkj∂En
they retained all terms in the evaluation of ∂2En/∂a2
jand so obtained exact expres-
sions for the diagonal terms. Note that this no longer has O(W)scaling. The major
problem with diagonal approximations, however, is that in practice the Hessian istypically found to be strongly nondiagonal, and so these approximations, which are
driven mainly be computational convenience, must be treated with care.

5.4. The Hessian Matrix 251
5.4.2 Outer product approximation
When neural networks are applied to regression problems, it is common to use
a sum-of-squares error function of the form
E=1
2N∑
n=1(yn−tn)2(5.82)
where we have considered the case of a single output in order to keep the notation
simple (the extension to several outputs is straightforward). We can then write the Exercise 5.16
Hessian matrix in the form
H=∇∇E=N∑
n=1∇yn∇yn+N∑
n=1(yn−tn)∇∇yn. (5.83)
If the network has been trained on the data set, and its outputs ynhappen to be very
close to the target values tn, then the second term in (5.83) will be small and can
be neglected. More generally, however, it may be appropriate to neglect this term
by the following argument. Recall from Section 1.5.5 that the optimal function thatminimizes a sum-of-squares loss is the conditional average of the target data. The
quantity (y
n−tn)is then a random variable with zero mean. If we assume that its

value is uncorrelated with the value of the second derivative term on the right-handside of (5.83), then the whole term will average to zero in the summation over n. Exercise 5.17
By neglecting the second term in (5.83), we arrive at the Levenberg–Marquardt
approximation or outer product approximation (because the Hessian matrix is built
up from a sum of outer products of vectors), given by
H≃N∑
n=1bnbT
n (5.84)
wherebn=∇yn=∇anbecause the activation function for the output units is

H≃N∑
n=1bnbT
n (5.84)
simply the identity. Evaluation of the outer product approximation for the Hessian
is straightforward as it only involves ﬁrst derivatives of the error function, whichcan be evaluated efﬁciently in O(W)steps using standard backpropagation. The
elements of the matrix can then be found in O(W
2)steps by simple multiplication.
It is important to emphasize that this approximation is only likely to be valid for anetwork that has been trained appropriately, and that for a general network mapping

the second derivative terms on the right-hand side of (5.83) will typically not be
negligible.
In the case of the cross-entropy error function for a network with logistic sigmoid
output-unit activation functions, the corresponding approximation is given by Exercise 5.19
H≃N∑
n=1yn(1−yn)bnbT
n. (5.85)
An analogous result can be obtained for multiclass networks having softmax output-
unit activation functions. Exercise 5.20

252 5. NEURAL NETWORKS
5.4.3 Inverse Hessian
We can use the outer-product approximation to develop a computationally ef-
ﬁcient procedure for approximating the inverse of the Hessian (Hassibi and Stork,
1993). First we write the outer-product approximation in matrix notation as
HN=N∑
n=1bnbT
n (5.86)
wherebn≡∇wanis the contribution to the gradient of the output unit activation
arising from data point n. We now derive a sequential procedure for building up the
Hessian by including data points one at a time. Suppose we have already obtainedthe inverse Hessian using the ﬁrst Ldata points. By separating off the contribution
from data point L+1, we obtain
H
L+1=HL+bL+1bT
L+1. (5.87)
In order to evaluate the inverse of the Hessian, we now consider the matrix identity
(
vTM−1)
1+vTM−1v(5.88)
whereIis the unit matrix, which is simply a special case of the Woodbury identity
(C.7). If we now identify HLwithMandbL+1withv, we obtain
H−1
L+1=H−1
L−H−1
LbL+1bT
L+1H−1
L
1+bT
L+1H−1
LbL+1. (5.89)

H
(
vTM−1)
H−1
L+1=H−1
L−H−1
LbL+1bT
L+1H−1
L
1+bT
L+1H−1
In this way, data points are sequentially absorbed until L+1 = Nand the whole data
set has been processed. This result therefore represents a procedure for evaluating
the inverse of the Hessian using a single pass through the data set. The initial matrix
H0is chosen to be αI, where αis a small quantity, so that the algorithm actually
ﬁnds the inverse of H+αI. The results are not particularly sensitive to the precise
value of α. Extension of this algorithm to networks having more than one output is
straightforward. Exercise 5.21
We note here that the Hessian matrix can sometimes be calculated indirectly as

part of the network training algorithm. In particular, quasi-Newton nonlinear opti-mization algorithms gradually build up an approximation to the inverse of the Hes-
sian during training. Such algorithms are discussed in detail in Bishop and Nabney
(2008).
5.4.4 Finite differences
As in the case of the ﬁrst derivatives of the error function, we can ﬁnd the second
derivatives by using ﬁnite differences, with accuracy limited by numerical precision.
If we perturb each possible pair of weights in turn, we obtain
∂2E
∂wji∂wlk=1
4ϵ2{E(wji+ϵ, wlk+ϵ)−E(wji+ϵ, wlk−ϵ)

(2008).
∂2E
−E(wji−ϵ, wlk+ϵ)+E(wji−ϵ, wlk−ϵ)}+O(ϵ2). (5.90)

5.4. The Hessian Matrix 253
Again, by using a symmetrical central differences formulation, we ensure that the
residual errors are O(ϵ2)rather than O(ϵ). Because there are W2elements in the
Hessian matrix, and because the evaluation of each element requires four forward
propagations each needing O(W)operations (per pattern), we see that this approach
will require O(W3)operations to evaluate the complete Hessian. It therefore has
poor scaling properties, although in practice it is very useful as a check on the soft-
ware implementation of backpropagation methods.
A more efﬁcient version of numerical differentiation can be found by applying
central differences to the ﬁrst derivatives of the error function, which are themselves
calculated using backpropagation. This gives
∂2E
2ϵ{∂E
∂wji(wlk+ϵ)−∂E
∂wji(wlk−ϵ)}
+O(ϵ2). (5.91)
Because there are now only Wweights to be perturbed, and because the gradients
can be evaluated in O(W)steps, we see that this method gives the Hessian in O(W2)

∂2E
2ϵ{∂E
5.4.5 Exact evaluation of the Hessian
So far, we have considered various approximation schemes for evaluating the
Hessian matrix or its inverse. The Hessian can also be evaluated exactly, for a net-work of arbitrary feed-forward topology, using extension of the technique of back-
propagation used to evaluate ﬁrst derivatives, which shares many of its desirable

features including computational efﬁciency (Bishop, 1991; Bishop, 1992). It can beapplied to any differentiable error function that can be expressed as a function of
the network outputs and to networks having arbitrary differentiable activation func-
tions. The number of computational steps needed to evaluate the Hessian scaleslikeO(W
2). Similar algorithms have also been considered by Buntine and Weigend
(1993).
Here we consider the speciﬁc case of a network having two layers of weights,

(1993).
for which the required equations are easily derived. We shall use indices iandi′Exercise 5.22
to denote inputs, indices jandj′to denoted hidden units, and indices kandk′to
denote outputs. We ﬁrst deﬁne
δk=∂En
∂ak,M kk′≡∂2En
∂ak∂ak′(5.92)
where Enis the contribution to the error from data point n. The Hessian matrix for
this network can then be considered in three separate blocks as follows.
1. Both weights in the second layer:
∂2En
∂w(2)
kj∂w(2)
k′j′=zjzj′Mkk′. (5.93)

254 5. NEURAL NETWORKS
2. Both weights in the ﬁrst layer:
∂2En
∂w(1)
ji∂w(1)
j′i′=xixi′h′′(aj′)Ijj′∑
kw(2)
kj′δk
+xixi′h′(aj′)h′(aj)∑
k∑
k′w(2)
k′j′w(2)
kjMkk′. (5.94)
3. One weight in each layer:
∂2En
∂w(1)
ji∂w(2)
kj′=xih′(aj′){
δkIjj′+zj∑
k′w(2)
k′j′Hkk′}
. (5.95)
HereIjj′is thej,j′element of the identity matrix. If one or both of the weights is
a bias term, then the corresponding expressions are obtained simply by setting theappropriate activation(s) to 1. Inclusion of skip-layer connections is straightforward. Exercise 5.23
5.4.6 Fast multiplication by the Hessian
For many applications of the Hessian, the quantity of interest is not the Hessian
matrix Hitself but the product of Hwith some vector v. We have seen that the
evaluation of the Hessian takes O(W2)operations, and it also requires storage that is
O(W2). The vector vTHthat we wish to calculate, however, has only Welements,
so instead of computing the Hessian as an intermediate step, we can instead try to

ﬁnd an efﬁcient approach to evaluating vTHdirectly in a way that requires only
O(W)operations.
To do this, we ﬁrst note that
vTH=vT∇(∇E) (5.96)
where ∇denotes the gradient operator in weight space. We can then write down
the standard forward-propagation and backpropagation equations for the evaluation
of∇Eand apply (5.96) to these equations to give a set of forward-propagation and
backpropagation equations for the evaluation of vTH(Møller, 1993; Pearlmutter,
1994). This corresponds to acting on the original forward-propagation and back-

propagation equations with a differential operator vT∇. Pearlmutter (1994) used the
notation R{·} to denote the operator vT∇, and we shall follow this convention. The
analysis is straightforward and makes use of the usual rules of differential calculus,
together with the result
R{w}=v. (5.97)
The technique is best illustrated with a simple example, and again we choose a
two-layer network of the form shown in Figure 5.1, with linear output units and a
sum-of-squares error function. As before, we consider the contribution to the error

function from one pattern in the data set. The required vector is then obtained as

5.4. The Hessian Matrix 255
usual by summing over the contributions from each of the patterns separately. For
the two-layer network, the forward-propagation equations are given by
aj=∑
iwjixi (5.98)
zj=h(aj) (5.99)
yk=∑
jwkjzj. (5.100)
We now act on these equations using the R{·} operator to obtain a set of forward
propagation equations in the form
R{aj}=∑
ivjixi (5.101)
R{zj}=h′(aj)R{aj} (5.102)
R{yk}=∑
jwkjR{zj}+∑
jvkjzj (5.103)
where vjiis the element of the vector vthat corresponds to the weight wji. Quan-
tities of the form R{zj},R{aj}andR{yk}are to be regarded as new variables
whose values are found using the above equations.
Because we are considering a sum-of-squares error function, we have the fol-
lowing standard backpropagation expressions:
δk=yk−tk (5.104)
kwkjδk. (5.105)
Again, we act on these equations with the R{·} operator to obtain a set of backprop-
agation equations in the form
R{δk}=R{yk} (5.106)
R{δj}=h′′(aj)R{aj}∑
kwkjδk
+h′(aj)∑
kvkjδk+h′(aj)∑
kwkjR{δk}. (5.107)

kwkjδk
+h′(aj)∑
Finally, we have the usual equations for the ﬁrst derivatives of the error
∂E
∂wkj=δkzj (5.108)
∂E
∂wji=δjxi (5.109)

256 5. NEURAL NETWORKS
and acting on these with the R{·} operator, we obtain expressions for the elements
of the vector vTH
R{∂E
∂wkj}
=R{δk}zj+δkR{zj} (5.110)
R{∂E
∂wji}
=xiR{δj}. (5.111)
The implementation of this algorithm involves the introduction of additional
variables R{aj},R{zj}andR{δj}for the hidden units and R{δk}andR{yk}
for the output units. For each input pattern, the values of these quantities can befound using the above results, and the elements of v
THare then given by (5.110)
and (5.111). An elegant aspect of this technique is that the equations for evaluating
vTHmirror closely those for standard forward and backward propagation, and so the
extension of existing software to compute this product is typically straightforward.
If desired, the technique can be used to evaluate the full Hessian matrix by
choosing the vector vto be given successively by a series of unit vectors of the
form(0,0,...,1,...,0)each of which picks out one column of the Hessian. This

leads to a formalism that is analytically equivalent to the backpropagation procedure
of Bishop (1992), as described in Section 5.4.5, though with some loss of efﬁciency
due to redundant calculations.
5.5. Regularization in Neural Networks
The number of input and outputs units in a neural network is generally determined
by the dimensionality of the data set, whereas the number Mof hidden units is a free
parameter that can be adjusted to give the best predictive performance. Note that M

controls the number of parameters (weights and biases) in the network, and so we
might expect that in a maximum likelihood setting there will be an optimum valueofMthat gives the best generalization performance, corresponding to the optimum
balance between under-ﬁtting and over-ﬁtting. Figure 5.9 shows an example of the
effect of different values of Mfor the sinusoidal regression problem.
The generalization error, however, is not a simple function of Mdue to the

presence of local minima in the error function, as illustrated in Figure 5.10. Herewe see the effect of choosing multiple random initializations for the weight vector
for a range of values of M. The overall best validation set performance in this
case occurred for a particular solution having M=8. In practice, one approach to
choosing Mis in fact to plot a graph of the kind shown in Figure 5.10 and then to
choose the speciﬁc solution having the smallest validation set error.
There are, however, other ways to control the complexity of a neural network

model in order to avoid over-ﬁtting. From our discussion of polynomial curve ﬁtting
in Chapter 1, we see that an alternative approach is to choose a relatively large value
forMand then to control complexity by the addition of a regularization term to the
error function. The simplest regularizer is the quadratic, giving a regularized error

5.5. Regularization in Neural Networks 257
M=1
0 1−101 M=3
0 1−101 M=1 0
0 1−101
Figure 5.9 Examples of two-layer networks trained on 10data points drawn from the sinusoidal data set. The
graphs show the result of ﬁtting networks having M=1,3and10hidden units, respectively, by minimizing a
sum-of-squares error function using a scaled conjugate-gradient algorithm.
of the form
˜E(w)=E(w)+λ
2wTw. (5.112)
This regularizer is also known as weight decay and has been discussed at length
in Chapter 3. The effective model complexity is then determined by the choice of
the regularization coefﬁcient λ. As we have seen previously, this regularizer can be
interpreted as the negative logarithm of a zero-mean Gaussian prior distribution over
the weight vector w.
5.5.1 Consistent Gaussian priors
One of the limitations of simple weight decay in the form (5.112) is that is
inconsistent with certain scaling properties of network mappings. To illustrate this,

consider a multilayer perceptron network having two layers of weights and linear
output units, which performs a mapping from a set of input variables {xi}to a set
of output variables {yk}. The activations of the hidden units in the ﬁrst hidden layer
Figure 5.10 Plot of the sum-of-squares test-set
error for the polynomial data set ver-
sus the number of hidden units in the
network, with 30random starts for
each network size, showing the ef-
fect of local minima. For each new
start, the weight vector was initial-
ized by sampling from an isotropic

Gaussian distribution having a mean
of zero and a variance of 10.
0 2 4 6 8 106080100120140160

258 5. NEURAL NETWORKS
zj=h(∑
iwjixi+wj0)
(5.113)
while the activations of the output units are given by
yk=∑
jwkjzj+wk0. (5.114)
Suppose we perform a linear transformation of the input data of the form
xi→˜xi=axi+b. (5.115)
Then we can arrange for the mapping performed by the network to be unchanged
by making a corresponding linear transformation of the weights and biases from theinputs to the units in the hidden layer of the form Exercise 5.24
w
ji→˜wji=1
awji (5.116)
wj0→˜wj0=wj0−b
a∑
iwji. (5.117)
Similarly, a linear transformation of the output variables of the network of the form
yk→˜yk=cyk+d (5.118)
can be achieved by making a transformation of the second-layer weights and biases
using
wkj→˜wkj=cwkj (5.119)
wk0→˜wk0=cwk0+d. (5.120)
If we train one network using the original data and one network using data for which
the input and/or target variables are transformed by one of the above linear transfor-
mations, then consistency requires that we should obtain equivalent networks that

using
differ only by the linear transformation of the weights as given. Any regularizershould be consistent with this property, otherwise it arbitrarily favours one solution
over another, equivalent one. Clearly, simple weight decay (5.112), that treats all
weights and biases on an equal footing, does not satisfy this property.
We therefore look for a regularizer which is invariant under the linear trans-
formations (5.116), (5.117), (5.119) and (5.120). These require that the regularizer

should be invariant to re-scaling of the weights and to shifts of the biases. Such aregularizer is given by
λ
1
2∑
w∈W 1w2+λ2
2∑
w∈W 2w2(5.121)
where W1denotes the set of weights in the ﬁrst layer, W2denotes the set of weights
in the second layer, and biases are excluded from the summations. This regularizer

5.5. Regularization in Neural Networks 259
will remain unchanged under the weight transformations provided the regularization
parameters are re-scaled using λ1→a1/2λ1andλ2→c−1/2λ2.
The regularizer (5.121) corresponds to a prior of the form
p(w|α1,α2)∝exp(
−α1
2∑
w∈W 1w2−α2
2∑
w∈W 2w2)
. (5.122)
Note that priors of this form are improper (they cannot be normalized) because the
bias parameters are unconstrained. The use of improper priors can lead to difﬁculties
in selecting regularization coefﬁcients and in model comparison within the Bayesianframework, because the corresponding evidence is zero. It is therefore common to
include separate priors for the biases (which then break shift invariance) having their
own hyperparameters. We can illustrate the effect of the resulting four hyperpa-rameters by drawing samples from the prior and plotting the corresponding network
functions, as shown in Figure 5.11.
More generally, we can consider priors in which the weights are divided into
any number of groups W
kso that

kso that
p(w)∝exp(
−1
2∑
kαk∥w∥2
k)
(5.123)
where
∥w∥2
k=∑
j∈W kw2
j. (5.124)
As a special case of this prior, if we choose the groups to correspond to the sets
of weights associated with each of the input units, and we optimize the marginallikelihood with respect to the corresponding parameters α
k, we obtain automatic
relevance determination as discussed in Section 7.2.2.
5.5.2 Early stopping
An alternative to regularization as a way of controlling the effective complexity
of a network is the procedure of early stopping . The training of nonlinear network

k=∑
j∈W kw2
j. (5.124)
models corresponds to an iterative reduction of the error function deﬁned with re-
spect to a set of training data. For many of the optimization algorithms used for
network training, such as conjugate gradients, the error is a nonincreasing functionof the iteration index. However, the error measured with respect to independent data,
generally called a validation set, often shows a decrease at ﬁrst, followed by an in-

crease as the network starts to over-ﬁt. Training can therefore be stopped at the pointof smallest error with respect to the validation data set, as indicated in Figure 5.12,
in order to obtain a network having good generalization performance.
The behaviour of the network in this case is sometimes explained qualitatively
in terms of the effective number of degrees of freedom in the network, in which this
number starts out small and then to grows during the training process, corresponding

to a steady increase in the effective complexity of the model. Halting training before

260 5. NEURAL NETWORKS
αw
1=1 ,αb
1=1 ,αw
2=1 ,αb
2=1
−1 −0.5 0 0.5 1−6−4−2024αw
1=1 ,αb
1=1 ,αw
2= 10, αb
2=1
−1 −0.5 0 0.5 1−60−40−2002040
αw
1= 1000, αb
1= 100, αw
2=1 ,αb
2=1
−1 −0.5 0 0.5 1−10−505αw
1= 1000, αb
1= 1000, αw
2=1 ,αb
2=1
−1 −0.5 0 0.5 1−10−505
Figure 5.11 Illustration of the effect of the hyperparameters governing the prior distribution over weights and
biases in a two-layer network having a single input, a single linear output, and 12hidden units having ‘ tanh’
activation functions. The priors are governed by four hyperparameters αb
1,αw
1,αb
2, and αw
2, which represent
the precisions of the Gaussian distributions of the ﬁrst-layer biases, ﬁrst-layer weights, second-layer biases, and
second-layer weights, respectively. We see that the parameter αw
2governs the vertical scale of functions (note
the different vertical axis ranges on the top two diagrams), αw
1governs the horizontal scale of variations in the
function values, and αb

1,αw
1,αb
2, and αw
1governs the horizontal range over which variations occur. The parameter αb
2, whose
effect is not illustrated here, governs the range of vertical offsets of the functions.
a minimum of the training error has been reached then represents a way of limiting
the effective network complexity.
In the case of a quadratic error function, we can verify this insight, and show
that early stopping should exhibit similar behaviour to regularization using a sim-
ple weight-decay term. This can be understood from Figure 5.13, in which the axes

2, whose
in weight space have been rotated to be parallel to the eigenvectors of the Hessian
matrix. If, in the absence of weight decay, the weight vector starts at the origin and
proceeds during training along a path that follows the local negative gradient vec-
tor, then the weight vector will move initially parallel to the w2axis through a point
corresponding roughly to ˜wand then move towards the minimum of the error func-
tionwML. This follows from the shape of the error surface and the widely differing

eigenvalues of the Hessian. Stopping at a point near ˜wis therefore similar to weight
decay. The relationship between early stopping and weight decay can be made quan-
titative, thereby showing that the quantity τη(where τis the iteration index, and η Exercise 5.25
is the learning rate parameter) plays the role of the reciprocal of the regularization

5.5. Regularization in Neural Networks 261
0 10 20 30 40 500.150.20.25
0 10 20 30 40 500.350.40.45
Figure 5.12 An illustration of the behaviour of training set error (left) and validation set error (right) during a
typical training session, as a function of the iteration step, for the sinusoidal data set. The goal of achieving
the best generalization performance suggests that training should be stopped at the point shown by the vertical
dashed lines, corresponding to the minimum of the validation set error.
parameter λ. The effective number of parameters in the network therefore grows
during the course of training.
5.5.3 Invariances
In many applications of pattern recognition, it is known that predictions should
be unchanged, or invariant , under one or more transformations of the input vari-
ables. For example, in the classiﬁcation of objects in two-dimensional images, such
as handwritten digits, a particular object should be assigned the same classiﬁcation

irrespective of its position within the image ( translation invariance ) or of its size
(scale invariance ). Such transformations produce signiﬁcant changes in the raw
data, expressed in terms of the intensities at each of the pixels in the image, and
yet should give rise to the same output from the classiﬁcation system. Similarly
in speech recognition, small levels of nonlinear warping along the time axis, which
preserve temporal ordering, should not change the interpretation of the signal.

If sufﬁciently large numbers of training patterns are available, then an adaptive
model such as a neural network can learn the invariance, at least approximately. This
involves including within the training set a sufﬁciently large number of examples of
the effects of the various transformations. Thus, for translation invariance in an im-
age, the training set should include examples of objects at many different positions.
This approach may be impractical, however, if the number of training examples

is limited, or if there are several invariants (because the number of combinations of
transformations grows exponentially with the number of such transformations). We
therefore seek alternative approaches for encouraging an adaptive model to exhibit
the required invariances. These can broadly be divided into four categories:
1. The training set is augmented using replicas of the training patterns, trans-
formed according to the desired invariances. For instance, in our digit recog-

nition example, we could make multiple copies of each example in which the

262 5. NEURAL NETWORKS
Figure 5.13 A schematic illustration of why
early stopping can give similar
results to weight decay in the
case of a quadratic error func-
tion. The ellipse shows a con-
tour of constant error, and wML
denotes the minimum of the er-
ror function. If the weight vector
starts at the origin and moves ac-
cording to the local negative gra-
dient direction, then it will follow
the path shown by the curve. By
stopping training early, a weight
vector ewis found that is qual-
itatively similar to that obtained
with a simple weight-decay reg-
ularizer and training to the mini-
mum of the regularized error, as
can be seen by comparing with
Figure 3.15.w1w2
˜wwML
digit is shifted to a different position in each image.
2. A regularization term is added to the error function that penalizes changes in
the model output when the input is transformed. This leads to the technique of
tangent propagation , discussed in Section 5.5.4.

˜wwML
3. Invariance is built into the pre-processing by extracting features that are invari-
ant under the required transformations. Any subsequent regression or classi-
ﬁcation system that uses such features as inputs will necessarily also respect
these invariances.
4. The ﬁnal option is to build the invariance properties into the structure of a neu-
ral network (or into the deﬁnition of a kernel function in the case of techniques
such as the relevance vector machine). One way to achieve this is through the

use of local receptive ﬁelds and shared weights, as discussed in the context of
convolutional neural networks in Section 5.5.6.
Approach 1 is often relatively easy to implement and can be used to encourage com-
plex invariances such as those illustrated in Figure 5.14. For sequential training
algorithms, this can be done by transforming each input pattern before it is presented
to the model so that, if the patterns are being recycled, a different transformation

(drawn from an appropriate distribution) is added each time. For batch methods, a
similar effect can be achieved by replicating each data point a number of times and
transforming each copy independently. The use of such augmented data can lead to
signiﬁcant improvements in generalization (Simard et al. , 2003), although it can also
be computationally costly.
Approach 2 leaves the data set unchanged but modiﬁes the error function through
the addition of a regularizer. In Section 5.5.5, we shall show that this approach is
closely related to approach 2.

5.5. Regularization in Neural Networks 263
Figure 5.14 Illustration of the synthetic warping of a handwritten digit. The original image is shown on the
left. On the right, the top row shows three examples of warped digits, with the corresponding displacement
ﬁelds shown on the bottom row. These displacement ﬁelds are generated by sampling random displacements
∆x,∆y∈(0,1)at each pixel and then smoothing by convolution with Gaussians of width 0.01,30and60
respectively.
One advantage of approach 3 is that it can correctly extrapolate well beyond the
range of transformations included in the training set. However, it can be difﬁcult
to ﬁnd hand-crafted features with the required invariances that do not also discard
information that can be useful for discrimination.
5.5.4 Tangent propagation
We can use regularization to encourage models to be invariant to transformations
of the input through the technique of tangent propagation (Simard et al. , 1992).

Consider the effect of a transformation on a particular input vector xn. Provided the
transformation is continuous (such as translation or rotation, but not mirror reﬂection
for instance), then the transformed pattern will sweep out a manifold Mwithin the
D-dimensional input space. This is illustrated in Figure 5.15, for the case of D=
2for simplicity. Suppose the transformation is governed by a single parameter ξ
(which might be rotation angle for instance). Then the subspace Mswept out by xn

Figure 5.15 Illustration of a two-dimensional input space
showing the effect of a continuous transforma-
tion on a particular input vector xn. A one-
dimensional transformation, parameterized by
the continuous variable ξ, applied to xncauses
it to sweep out a one-dimensional manifold M.
Locally, the effect of the transformation can be
approximated by the tangent vector τn.
x1x2
xnτn
ξM

264 5. NEURAL NETWORKS
will be one-dimensional, and will be parameterized by ξ. Let the vector that results
from acting on xnby this transformation be denoted by s(xn,ξ), which is deﬁned
so that s(x,0) =x. Then the tangent to the curve Mis given by the directional
derivative τ=∂s/∂ξ, and the tangent vector at the point xnis given by
τn=∂s(xn,ξ)
∂ξ⏐⏐⏐⏐
ξ=0. (5.125)
Under a transformation of the input vector, the network output vector will, in general,
change. The derivative of output kwith respect to ξis given by
∂yk
∂ξ⏐⏐⏐⏐
ξ=0=D∑
i=1∂yk
∂xi∂xi
∂ξ⏐⏐⏐⏐⏐
ξ=0=D∑
i=1Jkiτi (5.126)
where Jkiis the(k,i)element of the Jacobian matrix J, as discussed in Section 5.3.4.
The result (5.126) can be used to modify the standard error function, so as to encour-age local invariance in the neighbourhood of the data points, by the addition to the
original error function Eof a regularization function Ωto give a total error function
of the form
˜E=E+λΩ (5.127)
where λis a regularization coefﬁcient and
Ω=1
2∑
n∑
k(
∂ynk
∂ξ⏐⏐⏐⏐
ξ=0)2

∂yk
∂ξ⏐⏐⏐⏐
ξ=0=D∑
i=1∂yk
∂xi∂xi
∂ξ⏐⏐⏐⏐⏐
ξ=0=D∑
of the form
Ω=1
2∑
n∑
k(
∂ynk
∂ξ⏐⏐⏐⏐
ξ=0)2
=1
2∑
n∑
k(D∑
i=1Jnkiτni)2
. (5.128)
The regularization function will be zero when the network mapping function is in-
variant under the transformation in the neighbourhood of each pattern vector, and
the value of the parameter λdetermines the balance between ﬁtting the training data
and learning the invariance property.
In a practical implementation, the tangent vector τncan be approximated us-
ing ﬁnite differences, by subtracting the original vector xnfrom the corresponding

2∑
n∑
k(
∂ynk
∂ξ⏐⏐⏐⏐
ξ=0)2
=1
2∑
n∑
k(D∑
. (5.128)
vector after transformation using a small value of ξ, and then dividing by ξ. This is
illustrated in Figure 5.16.
The regularization function depends on the network weights through the Jaco-
bianJ. A backpropagation formalism for computing the derivatives of the regu-
larizer with respect to the network weights is easily obtained by extension of the Exercise 5.26
techniques introduced in Section 5.3.
If the transformation is governed by Lparameters (e.g., L=3 for the case of

translations combined with in-plane rotations in a two-dimensional image), then the
manifold Mwill have dimensionality L, and the corresponding regularizer is given
by the sum of terms of the form (5.128), one for each transformation. If severaltransformations are considered at the same time, and the network mapping is made
invariant to each separately, then it will be (locally) invariant to combinations of the
transformations (Simard et al. , 1992).

5.5. Regularization in Neural Networks 265
Figure 5.16 Illustration showing
(a) the original image xof a hand-
written digit, (b) the tangent vector
τcorresponding to an inﬁnitesimal
clockwise rotation, (c) the result of
adding a small contribution from the
tangent vector to the original image
giving x+ϵτwithϵ=1 5 degrees,
and (d) the true image rotated for
(a)
 (b)
(c)
 (d)
A related technique, called tangent distance , can be used to build invariance
properties into distance-based methods such as nearest-neighbour classiﬁers (Simard
et al. , 1993).
5.5.5 Training with transformed data
We have seen that one way to encourage invariance of a model to a set of trans-
formations is to expand the training set using transformed versions of the original
input patterns. Here we show that this approach is closely related to the technique of
tangent propagation (Bishop, 1995b; Leen, 1995).
As in Section 5.5.4, we shall consider a transformation governed by a single

parameter ξand described by the function s(x,ξ), withs(x,0) =x. We shall
also consider a sum-of-squares error function. The error function for untransformed
inputs can be written (in the inﬁnite data set limit) in the form
E=1
2∫∫
{y(x)−t}2p(t|x)p(x)dxdt (5.129)
as discussed in Section 1.5.5. Here we have considered a network having a single
output, in order to keep the notation uncluttered. If we now consider an inﬁnite
number of copies of each data point, each of which is perturbed by the transformation

266 5. NEURAL NETWORKS
in which the parameter ξis drawn from a distribution p(ξ), then the error function
deﬁned over this expanded data set can be written as
˜E=1
2∫∫∫
{y(s(x,ξ))−t}2p(t|x)p(x)p(ξ)dxdtdξ. (5.130)
We now assume that the distribution p(ξ)has zero mean with small variance, so that
we are only considering small transformations of the original input vectors. We can
then expand the transformation function as a Taylor series in powers of ξto give
s(x,ξ)= s(x,0) +ξ∂
∂ξs(x,ξ)⏐⏐⏐⏐
ξ=0+ξ2
2∂2
∂ξ2s(x,ξ)⏐⏐⏐⏐
ξ=0+O(ξ3)
=x+ξτ+1
2ξ2τ′+O(ξ3)
where τ′denotes the second derivative of s(x,ξ)with respect to ξevaluated at ξ=0.
This allows us to expand the model function to give
y(s(x,ξ)) =y(x)+ξτT∇y(x)+ξ2
2[
(τ′)T∇y(x)+τT∇∇y(x)τ]
+O(ξ3).
Substituting into the mean error function (5.130) and expanding, we then have
˜E=1
2∫∫
{y(x)−t}2p(t|x)p(x)dxdt
+ E[ξ]∫∫
{y(x)−t}τT∇y(x)p(t|x)p(x)dxdt
+ E[ξ2]∫∫[
{y(x)−t}1
2{
(τ′)T∇y(x)+τT∇∇y(x)τ}
+(
τT∇y(x))2]
p(t|x)p(x)dxdt+O(ξ3).

ξ=0+ξ2
2∂2
ξ=0+O(ξ3)
=x+ξτ+1
2[
+O(ξ3).
˜E=1
2∫∫
+ E[ξ]∫∫
+ E[ξ2]∫∫[
{y(x)−t}1
2{
+(
Because the distribution of transformations has zero mean we have E[ξ]=0 . Also,
we shall denote E[ξ2]byλ. Omitting terms of O(ξ3), the average error function then
becomes
˜E=E+λΩ (5.131)
where Eis the original sum-of-squares error, and the regularization term Ωtakes the
form
Ω=∫[
{y(x)−E[t|x]}1
2{
+(
p(x)dx (5.132)
in which we have performed the integration over t.

5.5. Regularization in Neural Networks 267
We can further simplify this regularization term as follows. In Section 1.5.5 we
saw that the function that minimizes the sum-of-squares error is given by the condi-tional average E[t|x]of the target values t. From (5.131) we see that the regularized
error will equal the unregularized sum-of-squares plus terms which are O(ξ), and so
the network function that minimizes the total error will have the form
y(x)= E[t|x]+O(ξ). (5.133)
Thus, to leading order in ξ, the ﬁrst term in the regularizer vanishes and we are left
with
Ω=1
2∫(
τT∇y(x))2p(x)dx (5.134)
which is equivalent to the tangent propagation regularizer (5.128).
If we consider the special case in which the transformation of the inputs simply
consists of the addition of random noise, so that x→x+ξ, then the regularizer
takes the form Exercise 5.27
Ω=1
2∫
∥∇y(x)∥2p(x)dx (5.135)
which is known as Tikhonov regularization (Tikhonov and Arsenin, 1977; Bishop,

with
Ω=1
2∫(
Ω=1
2∫
1995b). Derivatives of this regularizer with respect to the network weights can be
found using an extended backpropagation algorithm (Bishop, 1993). We see that, forsmall noise amplitudes, Tikhonov regularization is related to the addition of random
noise to the inputs, which has been shown to improve generalization in appropriate
circumstances (Sietsma and Dow, 1991).
5.5.6 Convolutional networks
Another approach to creating models that are invariant to certain transformation

of the inputs is to build the invariance properties into the structure of a neural net-
work. This is the basis for the convolutional neural network (Le Cun et al. , 1989;
LeCun et al. , 1998), which has been widely applied to image data.
Consider the speciﬁc task of recognizing handwritten digits. Each input image
comprises a set of pixel intensity values, and the desired output is a posterior proba-

bility distribution over the ten digit classes. We know that the identity of the digit isinvariant under translations and scaling as well as (small) rotations. Furthermore, the
network must also exhibit invariance to more subtle transformations such as elastic
deformations of the kind illustrated in Figure 5.14. One simple approach would be totreat the image as the input to a fully connected network, such as the kind shown in
Figure 5.1. Given a sufﬁciently large training set, such a network could in principle

yield a good solution to this problem and would learn the appropriate invariances by
example.
However, this approach ignores a key property of images, which is that nearby
pixels are more strongly correlated than more distant pixels. Many of the modern
approaches to computer vision exploit this property by extracting local features that
depend only on small subregions of the image. Information from such features canthen be merged in later stages of processing in order to detect higher-order features

268 5. NEURAL NETWORKS
Input image Convolutional layerSub-sampling
layer
Figure 5.17 Diagram illustrating part of a convolutional neural network, showing a layer of convolu-
tional units followed by a layer of subsampling units. Several successive pairs of such
layers may be used.
and ultimately to yield information about the image as whole. Also, local features
that are useful in one region of the image are likely to be useful in other regions of
the image, for instance if the object of interest is translated.
These notions are incorporated into convolutional neural networks through three
mechanisms: (i) local receptive ﬁelds, (ii) weight sharing, and (iii) subsampling. The
structure of a convolutional network is illustrated in Figure 5.17. In the convolutional
layer the units are organized into planes, each of which is called a feature map . Units
in a feature map each take inputs only from a small subregion of the image, and all

of the units in a feature map are constrained to share the same weight values. For
instance, a feature map might consist of 100 units arranged in a 10×10grid, with
each unit taking inputs from a 5×5pixel patch of the image. The whole feature map
therefore has 25adjustable weight parameters plus one adjustable bias parameter.
Input values from a patch are linearly combined using the weights and the bias, and
the result transformed by a sigmoidal nonlinearity using (5.1). If we think of the units

as feature detectors, then all of the units in a feature map detect the same pattern but
at different locations in the input image. Due to the weight sharing, the evaluation
of the activations of these units is equivalent to a convolution of the image pixel
intensities with a ‘kernel’ comprising the weight parameters. If the input image is
shifted, the activations of the feature map will be shifted by the same amount but will
otherwise be unchanged. This provides the basis for the (approximate) invariance of

5.5. Regularization in Neural Networks 269
the network outputs to translations and distortions of the input image. Because we
will typically need to detect multiple features in order to build an effective model,there will generally be multiple feature maps in the convolutional layer, each having
its own set of weight and bias parameters.
The outputs of the convolutional units form the inputs to the subsampling layer
of the network. For each feature map in the convolutional layer, there is a plane of
units in the subsampling layer and each unit takes inputs from a small receptive ﬁeld
in the corresponding feature map of the convolutional layer. These units performsubsampling. For instance, each subsampling unit might take inputs from a 2×2
unit region in the corresponding feature map and would compute the average of
those inputs, multiplied by an adaptive weight with the addition of an adaptive bias
parameter, and then transformed using a sigmoidal nonlinear activation function.

The receptive ﬁelds are chosen to be contiguous and nonoverlapping so that thereare half the number of rows and columns in the subsampling layer compared with
the convolutional layer. In this way, the response of a unit in the subsampling layer
will be relatively insensitive to small shifts of the image in the corresponding regionsof the input space.
In a practical architecture, there may be several pairs of convolutional and sub-

sampling layers. At each stage there is a larger degree of invariance to input trans-formations compared to the previous layer. There may be several feature maps in a
given convolutional layer for each plane of units in the previous subsampling layer,
so that the gradual reduction in spatial resolution is then compensated by an increas-ing number of features. The ﬁnal layer of the network would typically be a fully
connected, fully adaptive layer, with a softmax output nonlinearity in the case of
multiclass classiﬁcation.

The whole network can be trained by error minimization using backpropagation
to evaluate the gradient of the error function. This involves a slight modiﬁcation
of the usual backpropagation algorithm to ensure that the shared-weight constraints
are satisﬁed. Due to the use of local receptive ﬁelds, the number of weights in Exercise 5.28
the network is smaller than if the network were fully connected. Furthermore, thenumber of independent parameters to be learned from the data is much smaller still,
due to the substantial numbers of constraints on the weights.
5.5.7 Soft weight sharing

One way to reduce the effective complexity of a network with a large number
of weights is to constrain weights within certain groups to be equal. This is the
technique of weight sharing that was discussed in Section 5.5.6 as a way of buildingtranslation invariance into networks used for image interpretation. It is only appli-
cable, however, to particular problems in which the form of the constraints can be
speciﬁed in advance. Here we consider a form of soft weight sharing (Nowlan and
Hinton, 1992) in which the hard constraint of equal weights is replaced by a form

of regularization in which groups of weights are encouraged to have similar values.Furthermore, the division of weights into groups, the mean weight value for each
group, and the spread of values within the groups are all determined as part of the
learning process.

270 5. NEURAL NETWORKS
Recall that the simple weight decay regularizer, given in (5.112), can be viewed
as the negative log of a Gaussian prior distribution over the weights. We can encour-age the weight values to form several groups, rather than just one group, by consid-
ering instead a probability distribution that is a mixture of Gaussians. The centres Section 2.3.9
and variances of the Gaussian components, as well as the mixing coefﬁcients, will beconsidered as adjustable parameters to be determined as part of the learning process.
Thus, we have a probability density of the form
p(w)=
∏
ip(wi) (5.136)
where
p(wi)=M∑
j=1πjN(wi|µj,σ2
j) (5.137)
andπjare the mixing coefﬁcients. Taking the negative logarithm then leads to a
regularization function of the form
Ω(w)=−∑
iln(M∑
j))
. (5.138)
The total error function is then given by
˜E(w)=E(w)+λΩ(w) (5.139)
where λis the regularization coefﬁcient. This error is minimized both with respect

p(w)=
∏
where
p(wi)=M∑
j) (5.137)
Ω(w)=−∑
iln(M∑
j))
. (5.138)
to the weights wiand with respect to the parameters {πj,µj,σj}of the mixture
model. If the weights were constant, then the parameters of the mixture model could
be determined by using the EM algorithm discussed in Chapter 9. However, the dis-tribution of weights is itself evolving during the learning process, and so to avoid nu-
merical instability, a joint optimization is performed simultaneously over the weights
and the mixture-model parameters. This can be done using a standard optimizationalgorithm such as conjugate gradients or quasi-Newton methods.

In order to minimize the total error function, it is necessary to be able to evaluate
its derivatives with respect to the various adjustable parameters. To do this it is con-venient to regard the {π
j}asprior probabilities and to introduce the corresponding
posterior probabilities which, following (2.192), are given by Bayes’ theorem in the
form
γj(w)=πjN(w|µj,σ2
j)∑
kπkN(w|µk,σ2
k). (5.140)
The derivatives of the total error function with respect to the weights are then given
by Exercise 5.29
∂˜E
∂wi=∂E
∂wi+λ∑
jγj(wi)(wi−µj)
σ2

form
j)∑
∂˜E
∂wi=∂E
∂wi+λ∑
σ2
j. (5.141)

5.5. Regularization in Neural Networks 271
The effect of the regularization term is therefore to pull each weight towards the
centre of the jthGaussian, with a force proportional to the posterior probability of
that Gaussian for the given weight. This is precisely the kind of effect that we are
seeking.
Derivatives of the error with respect to the centres of the Gaussians are also
easily computed to give Exercise 5.30
∂˜E
∂µj=λ∑
iγj(wi)(µi−wj)
σ2
j(5.142)
which has a simple intuitive interpretation, because it pushes µjtowards an aver-
age of the weight values, weighted by the posterior probabilities that the respective
weight parameters were generated by component j. Similarly, the derivatives with
respect to the variances are given by Exercise 5.31
∂˜E
∂σj=λ∑
iγj(wi)(1
σj−(wi−µj)2
σ3
j)
(5.143)
which drives σjtowards the weighted average of the squared deviations of the weights
around the corresponding centre µj, where the weighting coefﬁcients are again given

∂˜E
∂σj=λ∑
iγj(wi)(1
σ3
j)
(5.143)
by the posterior probability that each weight is generated by component j. Note that
in a practical implementation, new variables ηjdeﬁned by
σ2
j= exp( ηj) (5.144)
are introduced, and the minimization is performed with respect to the ηj. This en-
sures that the parameters σjremain positive. It also has the effect of discouraging
pathological solutions in which one or more of the σjgoes to zero, corresponding
to a Gaussian component collapsing onto one of the weight parameter values. Such

σ2
solutions are discussed in more detail in the context of Gaussian mixture models inSection 9.2.1.
For the derivatives with respect to the mixing coefﬁcients π
j, we need to take
account of the constraints
∑
jπj=1, 0⩽πi⩽1 (5.145)
which follow from the interpretation of the πjas prior probabilities. This can be
done by expressing the mixing coefﬁcients in terms of a set of auxiliary variables
{ηj}using the softmax function given by
πj=exp(ηj)∑M
k=1exp(ηk). (5.146)

∑
The derivatives of the regularized error function with respect to the {ηj}then take
the form Exercise 5.32

272 5. NEURAL NETWORKS
Figure 5.18 The left ﬁgure shows a two-link robot arm,
in which the Cartesian coordinates (x1,x2)of the end ef-
fector are determined uniquely by the two joint angles θ1
andθ2and the (ﬁxed) lengths L1andL2of the arms. This
is know as the forward kinematics of the arm. In prac-
tice, we have to ﬁnd the joint angles that will give rise to a
desired end effector position and, as shown in the right ﬁg-
ure, this inverse kinematics has two solutions correspond-
ing to ‘elbow up’ and ‘elbow down’.L1L2
θ1θ2(x1,x2) (x1,x2)
elbow
downelbow
up
∂˜E
∂ηj=∑
i{πj−γj(wi)}. (5.147)
We see that πjis therefore driven towards the average posterior probability for com-
ponent j.
5.6. Mixture Density Networks
The goal of supervised learning is to model a conditional distribution p(t|x), which
for many simple regression problems is chosen to be Gaussian. However, practical
machine learning problems can often have signiﬁcantly non-Gaussian distributions.

elbow
downelbow
up
∂˜E
∂ηj=∑
ponent j.
These can arise, for example, with inverse problems in which the distribution can be
multimodal, in which case the Gaussian assumption can lead to very poor predic-
tions.
As a simple example of an inverse problem, consider the kinematics of a robot
arm, as illustrated in Figure 5.18. The forward problem involves ﬁnding the end ef- Exercise 5.33
fector position given the joint angles and has a unique solution. However, in practice
we wish to move the end effector of the robot to a speciﬁc position, and to do this we

tions.
must set appropriate joint angles. We therefore need to solve the inverse problem,
which has two solutions as seen in Figure 5.18.
Forward problems often corresponds to causality in a physical system and gen-
erally have a unique solution. For instance, a speciﬁc pattern of symptoms in the
human body may be caused by the presence of a particular disease. In pattern recog-
nition, however, we typically have to solve an inverse problem, such as trying to
predict the presence of a disease given a set of symptoms. If the forward problem

involves a many-to-one mapping, then the inverse problem will have multiple solu-
tions. For instance, several different diseases may result in the same symptoms.
In the robotics example, the kinematics is deﬁned by geometrical equations, and
the multimodality is readily apparent. However, in many machine learning problems
the presence of multimodality, particularly in problems involving spaces of high di-
mensionality, can be less obvious. For tutorial purposes, however, we shall consider

a simple toy problem for which we can easily visualize the multimodality. Data for
this problem is generated by sampling a variable xuniformly over the interval (0,1),
to give a set of values {xn}, and the corresponding target values tnare obtained

5.6. Mixture Density Networks 273
Figure 5.19 On the left is the data
set for a simple ‘forward problem’ in
which the red curve shows the result
of ﬁtting a two-layer neural network
by minimizing the sum-of-squares
error function. The corresponding
inverse problem, shown on the right,
is obtained by exchanging the roles
ofxandt. Here the same net-
work trained again by minimizing the
sum-of-squares error function gives
a very poor ﬁt to the data due to the
multimodality of the data set.0 101
0 101
by computing the function xn+0.3 sin(2 πxn)and then adding uniform noise over
the interval (−0.1,0.1). The inverse problem is then obtained by keeping the same
data points but exchanging the roles of xandt. Figure 5.19 shows the data sets for
the forward and inverse problems, along with the results of ﬁtting two-layer neural
networks having 6hidden units and a single linear output unit by minimizing a sum-
of-squares error function. Least squares corresponds to maximum likelihood under

0 101
a Gaussian assumption. We see that this leads to a very poor model for the highly
non-Gaussian inverse problem.
We therefore seek a general framework for modelling conditional probability
distributions. This can be achieved by using a mixture model for p(t|x)in which
both the mixing coefﬁcients as well as the component densities are ﬂexible functions
of the input vector x, giving rise to the mixture density network . For any given value
ofx, the mixture model provides a general formalism for modelling an arbitrary

conditional density function p(t|x). Provided we consider a sufﬁciently ﬂexible
network, we then have a framework for approximating arbitrary conditional distri-
butions.
Here we shall develop the model explicitly for Gaussian components, so that
p(t|x)=K∑
k=1πk(x)N(
t|µk(x),σ2
k(x))
. (5.148)
This is an example of a heteroscedastic model since the noise variance on the data
is a function of the input vector x. Instead of Gaussians, we can use other distribu-
tions for the components, such as Bernoulli distributions if the target variables are

butions.
p(t|x)=K∑
k(x))
. (5.148)
binary rather than continuous. We have also specialized to the case of isotropic co-
variances for the components, although the mixture density network can readily be
extended to allow for general covariance matrices by representing the covariances
using a Cholesky factorization (Williams, 1996). Even with isotropic components,
the conditional distribution p(t|x)does not assume factorization with respect to the
components of t(in contrast to the standard sum-of-squares regression model) as a
consequence of the mixture distribution.

We now take the various parameters of the mixture model, namely the mixing
coefﬁcients πk(x), the means µk(x), and the variances σ2
k(x), to be governed by

274 5. NEURAL NETWORKS
x1xD
θ1θM
θ
tp(t|x)
Figure 5.20 Themixture density network can represent general conditional probability densities p(t|x)
by considering a parametric mixture model for the distribution of twhose parameters are
determined by the outputs of a neural network that takes xas its input vector.
the outputs of a conventional neural network that takes xas its input. The structure
of this mixture density network is illustrated in Figure 5.20. The mixture density
network is closely related to the mixture of experts discussed in Section 14.5.3. The
principle difference is that in the mixture density network the same function is used
to predict the parameters of all of the component densities as well as the mixing co-
efﬁcients, and so the nonlinear hidden units are shared amongst the input-dependent
functions.
The neural network in Figure 5.20 can, for example, be a two-layer network
having sigmoidal (‘ tanh ’) hidden units. If there are Lcomponents in the mixture

model (5.148), and if thasKcomponents, then the network will have Loutput unit
activations denoted by aπ
kthat determine the mixing coefﬁcients πk(x),Koutputs
denoted by aσ
kthat determine the kernel widths σk(x), andL×Koutputs denoted
byaµ
kjthat determine the components µkj(x)of the kernel centres µk(x). The total
number of network outputs is given by (K+2 )L, as compared with the usual K
outputs for a network, which simply predicts the conditional means of the target

byaµ
The mixing coefﬁcients must satisfy the constraints
K∑
k=1πk(x)=1, 0⩽πk(x)⩽1 (5.149)
which can be achieved using a set of softmax outputs
πk(x)=exp(aπ
k)∑K
l=1exp(aπ
l). (5.150)
Similarly, the variances must satisfy σ2
k(x)⩾0and so can be represented in terms
of the exponentials of the corresponding network activations using
σk(x)=e x p ( aσ
k). (5.151)
Finally, because the means µk(x)have real components, they can be represented

5.6. Mixture Density Networks 275
directly by the network output activations
µkj(x)=aµ
kj. (5.152)
The adaptive parameters of the mixture density network comprise the vector w
of weights and biases in the neural network, that can be set by maximum likelihood,
or equivalently by minimizing an error function deﬁned to be the negative logarithmof the likelihood. For independent data, this error function takes the form
E(w)=−N∑
n=1ln{k∑
k=1πk(xn,w)N(
tn|µk(xn,w),σ2
k(xn,w))}
(5.153)
where we have made the dependencies on wexplicit.
In order to minimize the error function, we need to calculate the derivatives of
the error E(w)with respect to the components of w. These can be evaluated by
using the standard backpropagation procedure, provided we obtain suitable expres-
sions for the derivatives of the error with respect to the output-unit activations. These
represent error signals δfor each pattern and for each output unit, and can be back-

k(xn,w))}
(5.153)
propagated to the hidden units and the error function derivatives evaluated in the
usual way. Because the error function (5.153) is composed of a sum of terms, one
for each training data point, we can consider the derivatives for a particular patternnand then ﬁnd the derivatives of Eby summing over all patterns.
Because we are dealing with mixture distributions, it is convenient to view the
mixing coefﬁcients π
k(x)asx-dependent prior probabilities and to introduce the

corresponding posterior probabilities given by
γk(t|x)=πkNnk∑K
l=1πlNnl(5.154)
where Nnkdenotes N(tn|µk(xn),σ2
k(xn)).
The derivatives with respect to the network output activations governing the mix-
ing coefﬁcients are given by Exercise 5.34
∂En
∂aπ
k=πk−γk. (5.155)
Similarly, the derivatives with respect to the output activations controlling the com-
ponent means are given by Exercise 5.35
∂En
∂aµ
kl=γk{µkl−tl
σ2
k}
. (5.156)
Finally, the derivatives with respect to the output activations controlling the compo-

k(xn)).
∂En
∂aπ
∂En
∂aµ
σ2
k}
. (5.156)
nent variances are given by Exercise 5.36
∂En
∂aσ
k=−γk{∥t−µk∥2
σ3
k−1
σk}
. (5.157)

276 5. NEURAL NETWORKS
Figure 5.21 (a) Plot of the mixing
coefﬁcients πk(x)as a function of
xfor the three kernel functions in a
mixture density network trained on
the data shown in Figure 5.19. The
model has three Gaussian compo-
nents, and uses a two-layer multi-
layer perceptron with ﬁve ‘ tanh’ sig-
moidal units in the hidden layer, and
nine outputs (corresponding to the 3
means and 3 variances of the Gaus-
sian components and the 3 mixing
coefﬁcients). At both small and large
values of x, where the conditional
probability density of the target data
is unimodal, only one of the ker-
nels has a high value for its prior
probability, while at intermediate val-
ues of x, where the conditional den-
sity is trimodal, the three mixing co-
efﬁcients have comparable values.
(b) Plots of the means µk(x)using
the same colour coding as for the
mixing coefﬁcients. (c) Plot of the
contours of the corresponding con-
ditional probability density of the tar-
get data for the same mixture den-

sity network. (d) Plot of the ap-
proximate conditional mode, shown
by the red points, of the conditional
density.0 101
(a)0 101
(b)
(c)0 101
0 101
(d)
We illustrate the use of a mixture density network by returning to the toy ex-
ample of an inverse problem shown in Figure 5.19. Plots of the mixing coefﬁ-
cients πk(x), the means µk(x), and the conditional density contours corresponding
top(t|x), are shown in Figure 5.21. The outputs of the neural network, and hence the

(a)0 101
(b)
(c)0 101
0 101
(d)
parameters in the mixture model, are necessarily continuous single-valued functions
of the input variables. However, we see from Figure 5.21(c) that the model is able to
produce a conditional density that is unimodal for some values of xand trimodal for
other values by modulating the amplitudes of the mixing components πk(x).
Once a mixture density network has been trained, it can predict the conditional
density function of the target data for any given value of the input vector. This

conditional density represents a complete description of the generator of the data, so
far as the problem of predicting the value of the output vector is concerned. From
this density function we can calculate more speciﬁc quantities that may be of interest
in different applications. One of the simplest of these is the mean, corresponding to
the conditional average of the target data, and is given by
E[t|x]=∫
tp(t|x)dt=K∑
k=1πk(x)µk(x) (5.158)

5.7. Bayesian Neural Networks 277
where we have used (5.148). Because a standard network trained by least squares
is approximating the conditional mean, we see that a mixture density network canreproduce the conventional least-squares result as a special case. Of course, as we
have already noted, for a multimodal distribution the conditional mean is of limited
value.
We can similarly evaluate the variance of the density function about the condi-
tional average, to give Exercise 5.37
s
2(x)= E[
∥t−E[t|x]∥2|x]
(5.159)
=K∑
k=1πk(x)⎧
⎨
⎩σ2
k(x)+µk(x)−K∑
l=1πl(x)µl(x)2⎫
⎬
⎭(5.160)
where we have used (5.148) and (5.158). This is more general than the corresponding
least-squares result because the variance is a function of x.
We have seen that for multimodal distributions, the conditional mean can give
a poor representation of the data. For instance, in controlling the simple robot arm
shown in Figure 5.18, we need to pick one of the two possible joint angle settings

s
2(x)= E[
(5.159)
=K∑
k=1πk(x)⎧
⎨
⎩σ2
⎬
⎭(5.160)
in order to achieve the desired end-effector location, whereas the average of the twosolutions is not itself a solution. In such cases, the conditional mode may be of
more value. Because the conditional mode for the mixture density network does not
have a simple analytical solution, this would require numerical iteration. A simplealternative is to take the mean of the most probable component (i.e., the one with the
largest mixing coefﬁcient) at each value of x. This is shown for the toy data set in
Figure 5.21(d).

5.7. Bayesian Neural Networks
So far, our discussion of neural networks has focussed on the use of maximum like-lihood to determine the network parameters (weights and biases). Regularized max-
imum likelihood can be interpreted as a MAP (maximum posterior) approach in
which the regularizer can be viewed as the logarithm of a prior parameter distribu-tion. However, in a Bayesian treatment we need to marginalize over the distribution
of parameters in order to make predictions.
In Section 3.3, we developed a Bayesian solution for a simple linear regression
model under the assumption of Gaussian noise. We saw that the posterior distribu-

tion, which is Gaussian, could be evaluated exactly and that the predictive distribu-
tion could also be found in closed form. In the case of a multilayered network, the
highly nonlinear dependence of the network function on the parameter values means
that an exact Bayesian treatment can no longer be found. In fact, the log of the pos-terior distribution will be nonconvex, corresponding to the multiple local minima in
the error function.
The technique of variational inference, to be discussed in Chapter 10, has been

applied to Bayesian neural networks using a factorized Gaussian approximation

278 5. NEURAL NETWORKS
to the posterior distribution (Hinton and van Camp, 1993) and also using a full-
covariance Gaussian (Barber and Bishop, 1998a; Barber and Bishop, 1998b). Themost complete treatment, however, has been based on the Laplace approximation
(MacKay, 1992c; MacKay, 1992b) and forms the basis for the discussion given here.
We will approximate the posterior distribution by a Gaussian, centred at a mode ofthe true posterior. Furthermore, we shall assume that the covariance of this Gaus-
sian is small so that the network function is approximately linear with respect to the
parameters over the region of parameter space for which the posterior probability issigniﬁcantly nonzero. With these two approximations, we will obtain models that
are analogous to the linear regression and classiﬁcation models discussed in earlier
chapters and so we can exploit the results obtained there. We can then make use of
the evidence framework to provide point estimates for the hyperparameters and to

compare alternative models (for example, networks having different numbers of hid-den units). To start with, we shall discuss the regression case and then later consider
the modiﬁcations needed for solving classiﬁcation tasks.
5.7.1 Posterior parameter distribution
Consider the problem of predicting a single continuous target variable tfrom
a vector xof inputs (the extension to multiple targets is straightforward). We shall
suppose that the conditional distribution p(t|x)is Gaussian, with an x-dependent

mean given by the output of a neural network model y(x,w), and with precision
(inverse variance) β
p(t|x,w,β)=N(t|y(x,w),β−1). (5.161)
Similarly, we shall choose a prior distribution over the weights wthat is Gaussian of
the form
p(w|α)=N(w|0,α−1I). (5.162)
For an i.i.d. data set of Nobservations x1,...,xN, with a corresponding set of target
values D={t1,...,t N}, the likelihood function is given by
p(D|w,β)=N∏
n=1N(tn|y(xn,w),β−1) (5.163)
and so the resulting posterior distribution is then

the form
p(w|D,α,β)∝p(w|α)p(D|w,β). (5.164)
which, as a consequence of the nonlinear dependence of y(x,w)onw, will be non-
Gaussian.
We can ﬁnd a Gaussian approximation to the posterior distribution by using the
Laplace approximation. To do this, we must ﬁrst ﬁnd a (local) maximum of the
posterior, and this must be done using iterative numerical optimization. As usual, itis convenient to maximize the logarithm of the posterior, which can be written in the

5.7. Bayesian Neural Networks 279
form
lnp(w|D)=−α
2wTw−β
2N∑
n=1{y(xn,w)−tn}2+c o n s t (5.165)
which corresponds to a regularized sum-of-squares error function. Assuming for
the moment that αandβare ﬁxed, we can ﬁnd a maximum of the posterior, which
we denote wMAP, by standard nonlinear optimization algorithms such as conjugate
gradients, using error backpropagation to evaluate the required derivatives.
Having found a mode wMAP, we can then build a local Gaussian approximation
by evaluating the matrix of second derivatives of the negative log posterior distribu-
tion. From (5.165), this is given by
A=−∇∇ lnp(w|D,α,β)=αI+βH (5.166)
whereHis the Hessian matrix comprising the second derivatives of the sum-of-
squares error function with respect to the components of w. Algorithms for comput-
ing and approximating the Hessian were discussed in Section 5.4. The correspondingGaussian approximation to the posterior is then given from (4.134) by
q(w|D)=N(w|w
MAP,A−1). (5.167)

Similarly, the predictive distribution is obtained by marginalizing with respect
to this posterior distribution
p(t|x,D)=∫
p(t|x,w)q(w|D)dw. (5.168)
However, even with the Gaussian approximation to the posterior, this integration is
still analytically intractable due to the nonlinearity of the network function y(x,w)
as a function of w. To make progress, we now assume that the posterior distribution
has small variance compared with the characteristic scales of wover which y(x,w)

is varying. This allows us to make a Taylor series expansion of the network function
around wMAP and retain only the linear terms
y(x,w)≃y(x,wMAP)+gT(w−wMAP) (5.169)
g=∇wy(x,w)|w=wMAP. (5.170)
With this approximation, we now have a linear-Gaussian model with a Gaussian
distribution for p(w)and a Gaussian for p(t|w)whose mean is a linear function of
wof the form
p(t|x,w,β)≃N(
t|y(x,wMAP)+gT(w−wMAP),β−1)
. (5.171)

. (5.171)
We can therefore make use of the general result (2.115) for the marginal p(t)to give Exercise 5.38
p(t|x,D,α,β)=N(
t|y(x,wMAP),σ2(x))
(5.172)

280 5. NEURAL NETWORKS
where the input-dependent variance is given by
σ2(x)=β−1+gTA−1g. (5.173)
We see that the predictive distribution p(t|x,D)is a Gaussian whose mean is given
by the network function y(x,wMAP)with the parameter set to their MAP value. The
variance has two terms, the ﬁrst of which arises from the intrinsic noise on the target
variable, whereas the second is an x-dependent term that expresses the uncertainty
in the interpolant due to the uncertainty in the model parameters w. This should
be compared with the corresponding predictive distribution for the linear regression
model, given by (3.58) and (3.59).
5.7.2 Hyperparameter optimization
So far, we have assumed that the hyperparameters αandβare ﬁxed and known.
We can make use of the evidence framework, discussed in Section 3.5, together with
the Gaussian approximation to the posterior obtained using the Laplace approxima-tion, to obtain a practical procedure for choosing the values of such hyperparameters.

The marginal likelihood, or evidence, for the hyperparameters is obtained by
integrating over the network weights
p(D|α,β)=∫
p(D|w,β)p(w|α)dw. (5.174)
This is easily evaluated by making use of the Laplace approximation result (4.135). Exercise 5.39
Taking logarithms then gives
lnp(D|α,β)≃−E(wMAP)−1
2ln|A|+W
2lnα+N
2lnβ−N
2ln(2π)(5.175)
where Wis the total number of parameters in w, and the regularized error function
is deﬁned by
E(wMAP)=β
2N∑
n=1{y(xn,wMAP)−tn}2+α
2wT
MAPwMAP. (5.176)

2ln|A|+W
2lnα+N
2lnβ−N
E(wMAP)=β
2N∑
2wT
We see that this takes the same form as the corresponding result (3.86) for the linear
regression model.
In the evidence framework, we make point estimates for αandβby maximizing
lnp(D|α,β). Consider ﬁrst the maximization with respect to α, which can be done
by analogy with the linear regression case discussed in Section 3.5.2. We ﬁrst deﬁnethe eigenvalue equation
βHu
i=λiui (5.177)

2N∑
2wT
βHu
squares error function, evaluated at w=wMAP. By analogy with (3.92), we obtain
α=γ
wT
MAPwMAP(5.178)

5.7. Bayesian Neural Networks 281
where γrepresents the effective number of parameters and is deﬁned by Section 3.5.3
γ=W∑
i=1λi
α+λi. (5.179)
Note that this result was exact for the linear regression case. For the nonlinear neural
network, however, it ignores the fact that changes in αwill cause changes in the
Hessian H, which in turn will change the eigenvalues. We have therefore implicitly
ignored terms involving the derivatives of λiwith respect to α.
Similarly, from (3.95) we see that maximizing the evidence with respect to β
gives the re-estimation formula
1
β=1
N−γN∑
n=1{y(xn,wMAP)−tn}2. (5.180)
As with the linear model, we need to alternate between re-estimation of the hyper-
parameters αandβand updating of the posterior distribution. The situation with
a neural network model is more complex, however, due to the multimodality of the
posterior distribution. As a consequence, the solution for wMAP found by maximiz-
ing the log posterior will depend on the initialization of w. Solutions that differ only

1
β=1
N−γN∑
as a consequence of the interchange and sign reversal symmetries in the hidden units Section 5.1.1
are identical so far as predictions are concerned, and it is irrelevant which of theequivalent solutions is found. However, there may be inequivalent solutions as well,
and these will generally yield different values for the optimized hyperparameters.
In order to compare different models, for example neural networks having differ-
ent numbers of hidden units, we need to evaluate the model evidence p(D). This can

be approximated by taking (5.175) and substituting the values of αandβobtained
from the iterative optimization of these hyperparameters. A more careful evaluationis obtained by marginalizing over αandβ, again by making a Gaussian approxima-
tion (MacKay, 1992c; Bishop, 1995a). In either case, it is necessary to evaluate the
determinant |A|of the Hessian matrix. This can be problematic in practice because
the determinant, unlike the trace, is sensitive to the small eigenvalues that are often
difﬁcult to determine accurately.
The Laplace approximation is based on a local quadratic expansion around a

mode of the posterior distribution over weights. We have seen in Section 5.1.1 that
any given mode in a two-layer network is a member of a set of M!2
Mequivalent
modes that differ by interchange and sign-change symmetries, where Mis the num-
ber of hidden units. When comparing networks having different numbers of hid-
den units, this can be taken into account by multiplying the evidence by a factor ofM!2
M.
5.7.3 Bayesian neural networks for classiﬁcation
So far, we have used the Laplace approximation to develop a Bayesian treat-
ment of neural network regression models. We now discuss the modiﬁcations to

282 5. NEURAL NETWORKS
this framework that arise when it is applied to classiﬁcation. Here we shall con-
sider a network having a single logistic sigmoid output corresponding to a two-classclassiﬁcation problem. The extension to networks with multiclass softmax outputs
is straightforward. We shall build extensively on the analogous results for linear Exercise 5.40
classiﬁcation models discussed in Section 4.5, and so we encourage the reader tofamiliarize themselves with that material before studying this section.
The log likelihood function for this model is given by
lnp(D|w)=
∑
n=1N{tnlnyn+( 1−tn)l n ( 1−yn)} (5.181)
where tn∈{0,1}are the target values, and yn≡y(xn,w). Note that there is no
hyperparameter β, because the data points are assumed to be correctly labelled. As
before, the prior is taken to be an isotropic Gaussian of the form (5.162).
The ﬁrst stage in applying the Laplace framework to this model is to initialize
the hyperparameter α, and then to determine the parameter vector wby maximizing

lnp(D|w)=
∑
the log posterior distribution. This is equivalent to minimizing the regularized errorfunction
E(w)=−lnp(D|w)+α
2wTw (5.182)
and can be achieved using error backpropagation combined with standard optimiza-
tion algorithms, as discussed in Section 5.3.
Having found a solution wMAP for the weight vector, the next step is to eval-
uate the Hessian matrix Hcomprising the second derivatives of the negative log

likelihood function. This can be done, for instance, using the exact method of Sec-tion 5.4.5, or using the outer product approximation given by (5.85). The second
derivatives of the negative log posterior can again be written in the form (5.166), and
the Gaussian approximation to the posterior is then given by (5.167).
To optimize the hyperparameter α, we again maximize the marginal likelihood,
which is easily shown to take the form Exercise 5.41
lnp(D|α)≃−E(w
MAP)−1
2ln|A|+W
2lnα+c o n s t (5.183)

MAP)−1
2ln|A|+W
where the regularized error function is deﬁned by
E(wMAP)=−N∑
n=1{tnlnyn+( 1−tn)l n ( 1−yn)}+α
2wT
MAPwMAP (5.184)
in which yn≡y(xn,wMAP). Maximizing this evidence function with respect to α
again leads to the re-estimation equation given by (5.178).
The use of the evidence procedure to determine αis illustrated in Figure 5.22
for the synthetic two-dimensional data discussed in Appendix A.
Finally, we need the predictive distribution, which is deﬁned by (5.168). Again,

2ln|A|+W
2wT
this integration is intractable due to the nonlinearity of the network function. The

5.7. Bayesian Neural Networks 283
Figure 5.22 Illustration of the evidence framework
applied to a synthetic two-class data set.
The green curve shows the optimal de-
cision boundary, the black curve shows
the result of ﬁtting a two-layer network
with 8 hidden units by maximum likeli-
hood, and the red curve shows the re-
sult of including a regularizer in which
αis optimized using the evidence pro-
cedure, starting from the initial value
α=0. Note that the evidence proce-
dure greatly reduces the over-ﬁtting of
the network.
simplest approximation is to assume that the posterior distribution is very narrow
and hence make the approximation
p(t|x,D)≃p(t|x,wMAP). (5.185)
We can improve on this, however, by taking account of the variance of the posterior
distribution. In this case, a linear approximation for the network outputs, as was used
in the case of regression, would be inappropriate due to the logistic sigmoid output-

unit activation function that constrains the output to lie in the range (0,1). Instead,
we make a linear approximation for the output unit activation in the form
a(x,w)≃aMAP(x)+bT(w−wMAP) (5.186)
where aMAP(x)=a(x,wMAP), and the vector b≡∇a(x,wMAP)can be found by
backpropagation.
Because we now have a Gaussian approximation for the posterior distribution
overw, and a model for athat is a linear function of w, we can now appeal to the

results of Section 4.5.2. The distribution of output unit activation values, induced by
the distribution over network weights, is given by
p(a|x,D)=∫
δ(
a−aMAP(x)−bT(x)(w−wMAP))
q(w|D)dw (5.187)
where q(w|D)is the Gaussian approximation to the posterior distribution given by
(5.167). From Section 4.5.2, we see that this distribution is Gaussian with mean
aMAP≡a(x,wMAP), and variance
σ2
a(x)=bT(x)A−1b(x). (5.188)
Finally, to obtain the predictive distribution, we must marginalize over ausing
p(t=1|x,D)=∫
σ(a)p(a|x,D)da. (5.189)

284 5. NEURAL NETWORKS
Figure 5.23 An illustration of the Laplace approximation for a Bayesian neural network having 8 hidden units
with ‘tanh’ activation functions and a single logistic-sigmoid output unit. The weight parameters were found using
scaled conjugate gradients, and the hyperparameter αwas optimized using the evidence framework. On the left
is the result of using the simple approximation (5.185) based on a point estimate wMAP of the parameters,
in which the green curve shows the y=0.5decision boundary, and the other contours correspond to output
probabilities of y=0.1,0.3,0.7, and 0.9. On the right is the corresponding result obtained using (5.190). Note
that the effect of marginalization is to spread out the contours and to make the predictions less conﬁdent, so
that at each input point x, the posterior probabilities are shifted towards 0.5, while the y=0.5contour itself is
unaffected.

The convolution of a Gaussian with a logistic sigmoid is intractable. We therefore
apply the approximation (4.153) to (5.189) giving
p(t=1|x,D)=σ(
κ(σ2
a)bTwMAP)
(5.190)
where κ(·)is deﬁned by (4.154). Recall that both σ2
aandbare functions of x.
Figure 5.23 shows an example of this framework applied to the synthetic classi-
ﬁcation data set described in Appendix A.
Exercises
5.1 (⋆⋆)Consider a two-layer network function of the form (5.7) in which the hidden-
unit nonlinear activation functions g(·)are given by logistic sigmoid functions of the
form

κ(σ2
a)bTwMAP)
(5.190)
Exercises
form
σ(a)={1+e x p ( −a)}−1. (5.191)
Show that there exists an equivalent network, which computes exactly the same func-
tion, but with hidden unit activation functions given by tanh(a)where the tanh func-
tion is deﬁned by (5.59). Hint: ﬁrst ﬁnd the relation between σ(a)andtanh(a), and
then show that the parameters of the two networks differ by linear transformations.
5.2 (⋆)www Show that maximizing the likelihood function under the conditional
distribution (5.16) for a multioutput neural network is equivalent to minimizing the

sum-of-squares error function (5.11).

Exercises 285
5.3 (⋆⋆)Consider a regression problem involving multiple target variables in which it
is assumed that the distribution of the targets, conditioned on the input vector x,i sa
Gaussian of the form
p(t|x,w)=N(t|y(x,w),Σ) (5.192)
wherey(x,w)is the output of a neural network with input vector xand weight
vectorw, andΣis the covariance of the assumed Gaussian noise on the targets.
Given a set of independent observations of xandt, write down the error function
that must be minimized in order to ﬁnd the maximum likelihood solution for w,i f
we assume that Σis ﬁxed and known. Now assume that Σis also to be determined
from the data, and write down an expression for the maximum likelihood solutionforΣ. Note that the optimizations of wandΣare now coupled, in contrast to the
case of independent target variables discussed in Section 5.2.
5.4 (⋆⋆)Consider a binary classiﬁcation problem in which the target values are t∈
{0,1}, with a network output y(x,w)that represents p(t=1|x), and suppose that

there is a probability ϵthat the class label on a training data point has been incorrectly
set. Assuming independent and identically distributed data, write down the error
function corresponding to the negative log likelihood. Verify that the error function
(5.21) is obtained when ϵ=0. Note that this error function makes the model robust
to incorrectly labelled data, in contrast to the usual error function.
5.5 (⋆)www Show that maximizing likelihood for a multiclass neural network model
in which the network outputs have the interpretation yk(x,w)=p(tk=1|x)is

equivalent to the minimization of the cross-entropy error function (5.24).
5.6 (⋆)www Show the derivative of the error function (5.21) with respect to the
activation akfor an output unit having a logistic sigmoid activation function satisﬁes
(5.18).
5.7 (⋆)Show the derivative of the error function (5.24) with respect to the activation ak
for output units having a softmax activation function satisﬁes (5.18).
5.8 (⋆)We saw in (4.88) that the derivative of the logistic sigmoid activation function

(5.18).
can be expressed in terms of the function value itself. Derive the corresponding result
for the ‘ tanh ’ activation function deﬁned by (5.59).
5.9 (⋆)www The error function (5.21) for binary classiﬁcation problems was de-
rived for a network having a logistic-sigmoid output activation function, so that
0⩽y(x,w)⩽1, and data having target values t∈{0,1}. Derive the correspond-
ing error function if we consider a network having an output −1⩽y(x,w)⩽1
and target values t=1 for class C1andt=−1for class C2. What would be the

appropriate choice of output unit activation function?
5.10 (⋆)www Consider a Hessian matrix Hwith eigenvector equation (5.33). By
setting the vector vin (5.39) equal to each of the eigenvectors uiin turn, show that
His positive deﬁnite if, and only if, all of its eigenvalues are positive.

286 5. NEURAL NETWORKS
5.11 (⋆⋆)www Consider a quadratic error function deﬁned by (5.32), in which the
Hessian matrix Hhas an eigenvalue equation given by (5.33). Show that the con-
tours of constant error are ellipses whose axes are aligned with the eigenvectors ui,
with lengths that are inversely proportional to the square root of the corresponding
eigenvalues λi.
5.12 (⋆⋆)www By considering the local Taylor expansion (5.32) of an error function
about a stationary point w⋆, show that the necessary and sufﬁcient condition for the
stationary point to be a local minimum of the error function is that the Hessian matrixH, deﬁned by (5.30) with
ˆw=w⋆, be positive deﬁnite.
5.13 (⋆)Show that as a consequence of the symmetry of the Hessian matrix H, the
number of independent elements in the quadratic error function (5.28) is given byW(W+3 )/2.
5.14 (⋆)By making a Taylor expansion, verify that the terms that are O(ϵ)cancel on the
right-hand side of (5.69).

5.15 (⋆⋆)In Section 5.3.4, we derived a procedure for evaluating the Jacobian matrix of a
neural network using a backpropagation procedure. Derive an alternative formalism
for ﬁnding the Jacobian based on forward propagation equations.
5.16 (⋆)The outer product approximation to the Hessian matrix for a neural network
using a sum-of-squares error function is given by (5.84). Extend this result to the
case of multiple outputs.
5.17 (⋆)Consider a squared loss function of the form
E=1
2∫∫

E=1
2∫∫
{y(x,w)−t}2p(x,t)dxdt (5.193)
where y(x,w)is a parametric function such as a neural network. The result (1.89)
shows that the function y(x,w)that minimizes this error is given by the conditional
expectation of tgivenx. Use this result to show that the second derivative of Ewith
respect to two elements wrandwsof the vector w, is given by
∂2E
∂wr∂ws=∫∂y
∂wr∂y
∂wsp(x)dx. (5.194)
Note that, for a ﬁnite sample from p(x), we obtain (5.84).
5.18 (⋆)Consider a two-layer network of the form shown in Figure 5.1 with the addition

∂2E
∂wr∂y
of extra parameters corresponding to skip-layer connections that go directly from
the inputs to the outputs. By extending the discussion of Section 5.3.2, write down
the equations for the derivatives of the error function with respect to these additionalparameters.
5.19 (⋆)
www Derive the expression (5.85) for the outer product approximation to
the Hessian matrix for a network having a single output with a logistic sigmoidoutput-unit activation function and a cross-entropy error function, corresponding to

5.19 (⋆)
the result (5.84) for the sum-of-squares error function.

Exercises 287
5.20 (⋆)Derive an expression for the outer product approximation to the Hessian matrix
for a network having Koutputs with a softmax output-unit activation function and
a cross-entropy error function, corresponding to the result (5.84) for the sum-of-
squares error function.
5.21 (⋆⋆⋆ )Extend the expression (5.86) for the outer product approximation of the Hes-
sian matrix to the case of K>1output units. Hence, derive a recursive expression
analogous to (5.87) for incrementing the number Nof patterns and a similar expres-
sion for incrementing the number Kof outputs. Use these results, together with the
identity (5.88), to ﬁnd sequential update expressions analogous to (5.89) for ﬁnding
the inverse of the Hessian by incrementally including both extra patterns and extraoutputs.
5.22 (⋆⋆)Derive the results (5.93), (5.94), and (5.95) for the elements of the Hessian
matrix of a two-layer feed-forward network by application of the chain rule of cal-
culus.

culus.
5.23 (⋆⋆)Extend the results of Section 5.4.5 for the exact Hessian of a two-layer network
to include skip-layer connections that go directly from inputs to outputs.
5.24 (⋆)Verify that the network function deﬁned by (5.113) and (5.114) is invariant un-
der the transformation (5.115) applied to the inputs, provided the weights and biases
are simultaneously transformed using (5.116) and (5.117). Similarly, show that the
network outputs can be transformed according (5.118) by applying the transforma-tion (5.119) and (5.120) to the second-layer weights and biases.
5.25 (⋆⋆⋆ )

5.25 (⋆⋆⋆ )
www Consider a quadratic error function of the form
E=E0+1
2(w−w⋆)TH(w−w⋆) (5.195)
wherew⋆represents the minimum, and the Hessian matrix His positive deﬁnite and
constant. Suppose the initial weight vector w(0)is chosen to be at the origin and is
updated using simple gradient descent
w(τ)=w(τ−1)−ρ∇E (5.196)
where τdenotes the step number, and ρis the learning rate (which is assumed to be
small). Show that, after τsteps, the components of the weight vector parallel to the
eigenvectors of Hcan be written
w(τ)
j={1−(1−ρηj)τ}w⋆

E=E0+1
w(τ)
j (5.197)
where wj=wTuj, andujandηjare the eigenvectors and eigenvalues, respectively,
ofHso that
Huj=ηjuj. (5.198)
Show that as τ→∞ , this gives w(τ)→w⋆as expected, provided |1−ρηj|<1.
Now suppose that training is halted after a ﬁnite number τof steps. Show that the

288 5. NEURAL NETWORKS
components of the weight vector parallel to the eigenvectors of the Hessian satisfy
w(τ)
j≃w⋆
jwhen ηj≫(ρτ)−1(5.199)
|w(τ)
j|≪|w⋆
j|when ηj≪(ρτ)−1. (5.200)
Compare this result with the discussion in Section 3.5.3 of regularization with simple
weight decay, and hence show that (ρτ)−1is analogous to the regularization param-
eterλ. The above results also show that the effective number of parameters in the
network, as deﬁned by (3.91), grows as the training progresses.
5.26 (⋆⋆)Consider a multilayer perceptron with arbitrary feed-forward topology, which
is to be trained by minimizing the tangent propagation error function (5.127) in
which the regularizing function is given by (5.128). Show that the regularization
termΩcan be written as a sum over patterns of terms of the form
Ωn=1
2∑
k(Gyk)2(5.201)
where Gis a differential operator deﬁned by
G≡∑
iτi∂
∂xi. (5.202)
By acting on the forward propagation equations
zj=h(aj),a j=∑
iwjizi (5.203)

Ωn=1
2∑
G≡∑
iτi∂
with the operator G, show that Ωncan be evaluated by forward propagation using
the following equations:
αj=h′(aj)βj,β j=∑
iwjiαi. (5.204)
where we have deﬁned the new variables
αj≡Gzj,β j≡Gaj. (5.205)
Now show that the derivatives of Ωnwith respect to a weight wrsin the network can
be written in the form
∂Ωn
∂wrs=∑
kαk{φkrzs+δkrαs} (5.206)
δkr≡∂yk
∂ar,φ kr≡Gδkr. (5.207)
Write down the backpropagation equations for δkr, and hence derive a set of back-
propagation equations for the evaluation of the φkr.

Exercises 289
5.27 (⋆⋆)www Consider the framework for training with transformed data in the
special case in which the transformation consists simply of the addition of randomnoisex→x+ξwhere ξhas a Gaussian distribution with zero mean and unit
covariance. By following an argument analogous to that of Section 5.5.5, show that
the resulting regularizer reduces to the Tikhonov form (5.135).
5.28 (⋆)
www Consider a neural network, such as the convolutional network discussed
in Section 5.5.6, in which multiple weights are constrained to have the same value.
Discuss how the standard backpropagation algorithm must be modiﬁed in order to
ensure that such constraints are satisﬁed when evaluating the derivatives of an errorfunction with respect to the adjustable parameters in the network.
5.29 (⋆)
www Verify the result (5.141).
5.30 (⋆)Verify the result (5.142).
5.31 (⋆)Verify the result (5.143).
5.32 (⋆⋆)Show that the derivatives of the mixing coefﬁcients {πk}, deﬁned by (5.146),

5.29 (⋆)
with respect to the auxiliary parameters {ηj}are given by
∂πk
∂ηj=δjkπj−πjπk. (5.208)
Hence, by making use of the constraint∑
kπk=1, derive the result (5.147).
5.33 (⋆)Write down a pair of equations that express the Cartesian coordinates (x1,x2)
for the robot arm shown in Figure 5.18 in terms of the joint angles θ1andθ2and
the lengths L1andL2of the links. Assume the origin of the coordinate system is
given by the attachment point of the lower arm. These equations deﬁne the ‘forward

∂πk
kinematics’ of the robot arm.
5.34 (⋆)www Derive the result (5.155) for the derivative of the error function with
respect to the network output activations controlling the mixing coefﬁcients in the
mixture density network.
5.35 (⋆)Derive the result (5.156) for the derivative of the error function with respect
to the network output activations controlling the component means in the mixture
density network.
5.36 (⋆)Derive the result (5.157) for the derivative of the error function with respect to

the network output activations controlling the component variances in the mixturedensity network.
5.37 (⋆)Verify the results (5.158) and (5.160) for the conditional mean and variance of
the mixture density network model.
5.38 (⋆)Using the general result (2.115), derive the predictive distribution (5.172) for
the Laplace approximation to the Bayesian neural network model.

290 5. NEURAL NETWORKS
5.39 (⋆)www Make use of the Laplace approximation result (4.135) to show that the
evidence function for the hyperparameters αandβin the Bayesian neural network
model can be approximated by (5.175).
5.40 (⋆)www Outline the modiﬁcations needed to the framework for Bayesian neural
networks, discussed in Section 5.7.3, to handle multiclass problems using networks
having softmax output-unit activation functions.
5.41 (⋆⋆)By following analogous steps to those given in Section 5.7.1 for regression
networks, derive the result (5.183) for the marginal likelihood in the case of a net-
work having a cross-entropy error function and logistic-sigmoid output-unit activa-
tion function.

6
Kernel
Methods
In Chapters 3 and 4, we considered linear parametric models for regression and
classiﬁcation in which the form of the mapping y(x,w)from input xto output y
is governed by a vector wof adaptive parameters. During the learning phase, a
set of training data is used either to obtain a point estimate of the parameter vector
or to determine a posterior distribution over this vector. The training data is thendiscarded, and predictions for new inputs are based purely on the learned parameter
vectorw. This approach is also used in nonlinear parametric models such as neural
networks. Chapter 5
However, there is a class of pattern recognition techniques, in which the training
data points, or a subset of them, are kept and used also during the prediction phase.
For instance, the Parzen probability density model comprised a linear combination Section 2.5.1
of ‘kernel’ functions each one centred on one of the training data points. Similarly,

in Section 2.5.2 we introduced a simple technique for classiﬁcation called nearest
neighbours, which involved assigning to each new test vector the same label as the
291

292 6. KERNEL METHODS
closest example from the training set. These are examples of memory-based methods
that involve storing the entire training set in order to make predictions for future datapoints. They typically require a metric to be deﬁned that measures the similarity of
any two vectors in input space, and are generally fast to ‘train’ but slow at making
predictions for test data points.
Many linear parametric models can be re-cast into an equivalent ‘dual represen-
tation’ in which the predictions are also based on linear combinations of a kernel
function evaluated at the training data points. As we shall see, for models which are
based on a ﬁxed nonlinear feature space mapping φ(x), the kernel function is given
by the relation
k(x,x
′)=φ(x)Tφ(x′). (6.1)
From this deﬁnition, we see that the kernel is a symmetric function of its arguments
so that k(x,x′)=k(x′,x). The kernel concept was introduced into the ﬁeld of pat-
tern recognition by Aizerman et al. (1964) in the context of the method of potential

k(x,x
functions, so-called because of an analogy with electrostatics. Although neglected
for many years, it was re-introduced into machine learning in the context of large-margin classiﬁers by Boser et al. (1992) giving rise to the technique of support
vector machines . Since then, there has been considerable interest in this topic, both Chapter 7
in terms of theory and applications. One of the most signiﬁcant developments has
been the extension of kernels to handle symbolic objects, thereby greatly expanding
the range of problems that can be addressed.

The simplest example of a kernel function is obtained by considering the identity
mapping for the feature space in (6.1) so that φ(x)=x, in which case k(x,x
′)=
xTx′. We shall refer to this as the linear kernel.
The concept of a kernel formulated as an inner product in a feature space allows
us to build interesting extensions of many well-known algorithms by making use of
thekernel trick , also known as kernel substitution . The general idea is that, if we have
an algorithm formulated in such a way that the input vector xenters only in the form

′)=
of scalar products, then we can replace that scalar product with some other choice of
kernel. For instance, the technique of kernel substitution can be applied to principalcomponent analysis in order to develop a nonlinear variant of PCA (Sch ¨olkopf et al. , Section 12.3
1998). Other examples of kernel substitution include nearest-neighbour classiﬁers
and the kernel Fisher discriminant (Mika et al. , 1999; Roth and Steinhage, 2000;
Baudat and Anouar, 2000).
There are numerous forms of kernel functions in common use, and we shall en-

counter several examples in this chapter. Many have the property of being a function
only of the difference between the arguments, so that k(x,x
′)=k(x−x′), which
are known as stationary kernels because they are invariant to translations in input
space. A further specialization involves homogeneous kernels, also known as ra-
dial basis functions , which depend only on the magnitude of the distance (typically Section 6.3
Euclidean) between the arguments so that k(x,x′)=k(∥x−x′∥).
For recent textbooks on kernel methods, see Sch ¨olkopf and Smola (2002), Her-

brich (2002), and Shawe-Taylor and Cristianini (2004).

6.1. Dual Representations 293
6.1. Dual Representations
Many linear models for regression and classiﬁcation can be reformulated in terms of
a dual representation in which the kernel function arises naturally. This concept will
play an important role when we consider support vector machines in the next chapter.Here we consider a linear regression model whose parameters are determined by
minimizing a regularized sum-of-squares error function given by
J(w)=1
2N∑
n=1{
wTφ(xn)−tn}2+λ
2wTw (6.2)
where λ⩾0. If we set the gradient of J(w)with respect to wequal to zero, we see
that the solution for wtakes the form of a linear combination of the vectors φ(xn),
with coefﬁcients that are functions of w, of the form
w=−1
λN∑
n=1{
wTφ(xn)−tn}
φ(xn)=N∑
n=1anφ(xn)=ΦTa (6.3)
whereΦis the design matrix, whose nthrow is given by φ(xn)T. Here the vector
a=(a1,...,a N)T, and we have deﬁned
an=−1
λ{
. (6.4)
Instead of working with the parameter vector w, we can now reformulate the least-

2wTw (6.2)
w=−1
λN∑
n=1{
φ(xn)=N∑
an=−1
λ{
. (6.4)
squares algorithm in terms of the parameter vector a, giving rise to a dual represen-
tation . If we substitute w=ΦTaintoJ(w), we obtain
J(a)=1
2aTΦΦTΦΦTa−aTΦΦTt+1
2tTt+λ
2aTΦΦTa (6.5)
where t=(t1,...,t N)T. We now deﬁne the Gram matrix K=ΦΦT, which is an
N×Nsymmetric matrix with elements
Knm=φ(xn)Tφ(xm)=k(xn,xm) (6.6)
where we have introduced the kernel function k(x,x′)deﬁned by (6.1). In terms of
the Gram matrix, the sum-of-squares error function can be written as
J(a)=1
2aTKKa−aTKt+1
2tTt+λ
2aTKa. (6.7)

J(a)=1
2tTt+λ
J(a)=1
2tTt+λ
Setting the gradient of J(a)with respect to ato zero, we obtain the following solu-
tion
a=(K+λIN)−1t. (6.8)

294 6. KERNEL METHODS
If we substitute this back into the linear regression model, we obtain the following
prediction for a new input x
y(x)=wTφ(x)=aTΦφ(x)=k(x)T(K+λIN)−1t (6.9)
where we have deﬁned the vector k(x)with elements kn(x)=k(xn,x). Thus we
see that the dual formulation allows the solution to the least-squares problem to beexpressed entirely in terms of the kernel function k(x,x
′). This is known as a dual
formulation because, by noting that the solution for acan be expressed as a linear
combination of the elements of φ(x), we recover the original formulation in terms of
the parameter vector w. Note that the prediction at xis given by a linear combination Exercise 6.1
of the target values from the training set. In fact, we have already obtained this result,
using a slightly different notation, in Section 3.3.3.
In the dual formulation, we determine the parameter vector aby inverting an
N×Nmatrix, whereas in the original parameter space formulation we had to invert

anM×Mmatrix in order to determine w. Because Nis typically much larger
thanM, the dual formulation does not seem to be particularly useful. However, the
advantage of the dual formulation, as we shall see, is that it is expressed entirely interms of the kernel function k(x,x
′). We can therefore work directly in terms of
kernels and avoid the explicit introduction of the feature vector φ(x), which allows
us implicitly to use feature spaces of high, even inﬁnite, dimensionality.

The existence of a dual representation based on the Gram matrix is a property of
many linear models, including the perceptron. In Section 6.4, we will develop a dual- Exercise 6.2
ity between probabilistic linear models for regression and the technique of Gaussianprocesses. Duality will also play an important role when we discuss support vector
machines in Chapter 7.
6.2. Constructing Kernels

In order to exploit kernel substitution, we need to be able to construct valid kernelfunctions. One approach is to choose a feature space mapping φ(x)and then use
this to ﬁnd the corresponding kernel, as is illustrated in Figure 6.1. Here the kernelfunction is deﬁned for a one-dimensional input space by
k(x, x
′)=φ(x)Tφ(x′)=M∑
i=1φi(x)φi(x′) (6.10)
where φi(x)are the basis functions.
An alternative approach is to construct kernel functions directly. In this case,
we must ensure that the function we choose is a valid kernel, in other words that it

k(x, x
corresponds to a scalar product in some (perhaps inﬁnite dimensional) feature space.
As a simple example, consider a kernel function given by
k(x,z)=(
xTz)2. (6.11)

6.2. Constructing Kernels 295
Figure 6.1 Illustration of the construction of kernel functions starting from a corresponding set of basis func-
tions. In each column the lower plot shows the kernel function k(x, x′)deﬁned by (6.10) plotted as a function of
xforx′=0, while the upper plot shows the corresponding basis functions given by polynomials (left column),
‘Gaussians’ (centre column), and logistic sigmoids (right column).
If we take the particular case of a two-dimensional input space x=(x1,x2)we
can expand out the terms and thereby identify the corresponding nonlinear feature
mapping
k(x,z)=(
xTz)2=(x1z1+x2z2)2
=x2
1z2
1+2x1z1x2z2+x2
2z2
2
=(x2
1,√
2x1x2,x2
2)(z2
1,√
2z1z2,z2
2)T
=φ(x)Tφ(z). (6.12)
We see that the feature mapping takes the form φ(x)=( x2
1,√
2x1x2,x2
2)Tand
therefore comprises all possible second order terms, with a speciﬁc weighting be-
tween them.

mapping
k(x,z)=(
=x2
1z2
2z2
2
=(x2
1,√
2x1x2,x2
2)(z2
1,√
2z1z2,z2
2)T
1,√
2x1x2,x2
2)Tand
More generally, however, we need a simple way to test whether a function con-
stitutes a valid kernel without having to construct the function φ(x)explicitly. A
necessary and sufﬁcient condition for a function k(x,x′)to be a valid kernel (Shawe-
Taylor and Cristianini, 2004) is that the Gram matrix K, whose elements are given by
k(xn,xm), should be positive semideﬁnite for all possible choices of the set {xn}.
Note that a positive semideﬁnite matrix is not the same thing as a matrix whose
elements are nonnegative. Appendix C

One powerful technique for constructing new kernels is to build them out of
simpler kernels as building blocks. This can be done using the following properties:

296 6. KERNEL METHODS
Techniques for Constructing New Kernels.
Given valid kernels k1(x,x′)andk2(x,x′), the following new kernels will also
be valid:
k(x,x′)= ck1(x,x′) (6.13)
k(x,x′)= f(x)k1(x,x′)f(x′) (6.14)
k(x,x′)= q(k1(x,x′)) (6.15)
k(x,x′)=e x p ( k1(x,x′)) (6.16)
k(x,x′)= k1(x,x′)+k2(x,x′) (6.17)
k(x,x′)= k1(x,x′)k2(x,x′) (6.18)
k(x,x′)= k3(φ(x),φ(x′)) (6.19)
k(x,x′)= xTAx′(6.20)
k(x,x′)= ka(xa,x′
a)+kb(xb,x′
b) (6.21)
a)kb(xb,x′
b) (6.22)
where c>0is a constant, f(·)is any function, q(·)is a polynomial with nonneg-
ative coefﬁcients, φ(x)is a function from xtoRM,k3(·,·)is a valid kernel in
RM,Ais a symmetric positive semideﬁnite matrix, xaandxbare variables (not
necessarily disjoint) with x=(xa,xb), andkaandkbare valid kernel functions
over their respective spaces.
Equipped with these properties, we can now embark on the construction of more
complex kernels appropriate to speciﬁc applications. We require that the kernel

b) (6.22)
k(x,x′)be symmetric and positive semideﬁnite and that it expresses the appropriate
form of similarity between xandx′according to the intended application. Here we
consider a few common examples of kernel functions. For a more extensive discus-sion of ‘kernel engineering’, see Shawe-Taylor and Cristianini (2004).
We saw that the simple polynomial kernel k(x,x
′)=(
xTx′)2contains only
terms of degree two. If we consider the slightly generalized kernel k(x,x′)=(

′)=(
xTx′+c)2withc>0, then the corresponding feature mapping φ(x)contains con-
stant and linear terms as well as terms of order two. Similarly, k(x,x′)=(
xTx′)M
contains all monomials of order M. For instance, if xandx′are two images, then
the kernel represents a particular weighted sum of all possible products of Mpixels
in the ﬁrst image with Mpixels in the second image. This can similarly be gener-
alized to include all terms up to degree Mby considering k(x,x′)=(
xTx′+c)M
withc>0. Using the results (6.17) and (6.18) for combining kernels we see that

xTx′)M
xTx′+c)M
these will all be valid kernel functions.
Another commonly used kernel takes the form
k(x,x′) = exp(
−∥x−x′∥2/2σ2)
(6.23)
and is often called a ‘Gaussian’ kernel. Note, however, that in this context it is
not interpreted as a probability density, and hence the normalization coefﬁcient is

6.2. Constructing Kernels 297
omitted. We can see that this is a valid kernel by expanding the square
∥x−x′∥2=xTx+(x′)Tx′−2xTx′(6.24)
to give
k(x,x′)=e x p(
−xTx/2σ2)
exp(
xTx′/σ2)
exp(
−(x′)Tx′/2σ2)
(6.25)
and then making use of (6.14) and (6.16), together with the validity of the linear
kernel k(x,x′)=xTx′. Note that the feature vector that corresponds to the Gaussian
kernel has inﬁnite dimensionality. Exercise 6.11
The Gaussian kernel is not restricted to the use of Euclidean distance. If we use
kernel substitution in (6.24) to replace xTx′with a nonlinear kernel κ(x,x′),w e
obtain
k(x,x′) = exp{
−1
2σ2(κ(x,x)+κ(x′,x′)−2κ(x,x′))}
. (6.26)
An important contribution to arise from the kernel viewpoint has been the exten-
sion to inputs that are symbolic, rather than simply vectors of real numbers. Kernel
functions can be deﬁned over objects as diverse as graphs, sets, strings, and text doc-
uments. Consider, for instance, a ﬁxed set and deﬁne a nonvectorial space consisting

obtain
−1
. (6.26)
of all possible subsets of this set. If A1andA2are two such subsets then one simple
choice of kernel would be
k(A1,A2)=2|A1∩A2|(6.27)
where A1∩A2denotes the intersection of sets A1andA2, and|A|denotes the
number of subsets in A. This is a valid kernel function because it can be shown to
correspond to an inner product in a feature space. Exercise 6.12
One powerful approach to the construction of kernels starts from a probabilistic
generative model (Haussler, 1999), which allows us to apply generative models in a

discriminative setting. Generative models can deal naturally with missing data andin the case of hidden Markov models can handle sequences of varying length. By
contrast, discriminative models generally give better performance on discriminative
tasks than generative models. It is therefore of some interest to combine these twoapproaches (Lasserre et al. , 2006). One way to combine them is to use a generative
model to deﬁne a kernel, and then use this kernel in a discriminative approach.
Given a generative model p(x)we can deﬁne a kernel by
k(x,x
′)=p(x)p(x′). (6.28)

k(x,x
This is clearly a valid kernel function because we can interpret it as an inner product
in the one-dimensional feature space deﬁned by the mapping p(x). It says that two
inputsxandx′are similar if they both have high probabilities. We can use (6.13) and
(6.17) to extend this class of kernels by considering sums over products of different
probability distributions, with positive weighting coefﬁcients p(i), of the form
k(x,x′)=∑
ip(x|i)p(x′|i)p(i). (6.29)

298 6. KERNEL METHODS
This is equivalent, up to an overall multiplicative constant, to a mixture distribution
in which the components factorize, with the index iplaying the role of a ‘latent’
variable. Two inputs xandx′will give a large value for the kernel function, and Section 9.2
hence appear similar, if they have signiﬁcant probability under a range of different
components. Taking the limit of an inﬁnite sum, we can also consider kernels of theform
k(x,x
′)=∫
p(x|z)p(x′|z)p(z)dz (6.30)
wherezis a continuous latent variable.
Now suppose that our data consists of ordered sequences of length Lso that
an observation is given by X={x1,...,xL}. A popular generative model for
sequences is the hidden Markov model, which expresses the distribution p(X)as a Section 13.2
marginalization over a corresponding sequence of hidden states Z={z1,...,zL}.
We can use this approach to deﬁne a kernel function measuring the similarity of twosequences XandX
′by extending the mixture representation (6.29) to give
k(X,X′)=∑

k(X,X′)=∑
Zp(X|Z)p(X′|Z)p(Z) (6.31)
so that both observed sequences are generated by the same hidden sequence Z. This
model can easily be extended to allow sequences of differing length to be compared.
An alternative technique for using generative models to deﬁne kernel functions
is known as the Fisher kernel (Jaakkola and Haussler, 1999). Consider a parametric
generative model p(x|θ)where θdenotes the vector of parameters. The goal is to
ﬁnd a kernel that measures the similarity of two input vectors xandx′induced by the

generative model. Jaakkola and Haussler (1999) consider the gradient with respect
toθ, which deﬁnes a vector in a ‘feature’ space having the same dimensionality as
θ. In particular, they consider the Fisher score
g(θ,x)=∇θlnp(x|θ) (6.32)
from which the Fisher kernel is deﬁned by
k(x,x′)=g(θ,x)TF−1g(θ,x′). (6.33)
HereFis the Fisher information matrix , given by
F=Ex[
g(θ,x)g(θ,x)T]
(6.34)
where the expectation is with respect to xunder the distribution p(x|θ). This can

F=Ex[
(6.34)
be motivated from the perspective of information geometry (Amari, 1998), which
considers the differential geometry of the space of model parameters. Here we sim-
ply note that the presence of the Fisher information matrix causes this kernel to beinvariant under a nonlinear re-parameterization of the density model θ→ψ(θ). Exercise 6.13
In practice, it is often infeasible to evaluate the Fisher information matrix. One
approach is simply to replace the expectation in the deﬁnition of the Fisher informa-tion with the sample average, giving
F≃1
NN∑

F≃1
NN∑
n=1g(θ,xn)g(θ,xn)T. (6.35)

6.3. Radial Basis Function Networks 299
This is the covariance matrix of the Fisher scores, and so the Fisher kernel corre-
sponds to a whitening of these scores. More simply, we can just omit the Fisher Section 12.1.3
information matrix altogether and use the noninvariant kernel
k(x,x′)=g(θ,x)Tg(θ,x′). (6.36)
An application of Fisher kernels to document retrieval is given by Hofmann (2000).
A ﬁnal example of a kernel function is the sigmoidal kernel given by
k(x,x′) = tanh(
axTx′+b)
(6.37)
whose Gram matrix in general is not positive semideﬁnite. This form of kernel
has, however, been used in practice (Vapnik, 1995), possibly because it gives kernel
expansions such as the support vector machine a superﬁcial resemblance to neural
network models. As we shall see, in the limit of an inﬁnite number of basis functions,
a Bayesian neural network with an appropriate prior reduces to a Gaussian process,thereby providing a deeper link between neural networks and kernel methods. Section 6.4.7

(6.37)
6.3. Radial Basis Function Networks
In Chapter 3, we discussed regression models based on linear combinations of ﬁxed
basis functions, although we did not discuss in detail what form those basis functions
might take. One choice that has been widely used is that of radial basis functions ,
which have the property that each basis function depends only on the radial distance(typically Euclidean) from a centre µ
j, so that φj(x)=h(∥x−µj∥).

Historically, radial basis functions were introduced for the purpose of exact func-
tion interpolation (Powell, 1987). Given a set of input vectors {x1,...,xN}along
with corresponding target values {t1,...,t N}, the goal is to ﬁnd a smooth function
f(x)that ﬁts every target value exactly, so that f(xn)=tnforn=1,...,N . This
is achieved by expressing f(x)as a linear combination of radial basis functions, one
centred on every data point
f(x)=N∑
n=1wnh(∥x−xn∥). (6.38)
The values of the coefﬁcients {wn}are found by least squares, and because there

f(x)=N∑
are the same number of coefﬁcients as there are constraints, the result is a function
that ﬁts every target value exactly. In pattern recognition applications, however, the
target values are generally noisy, and exact interpolation is undesirable because this
corresponds to an over-ﬁtted solution.
Expansions in radial basis functions also arise from regularization theory (Pog-
gio and Girosi, 1990; Bishop, 1995a). For a sum-of-squares error function with a

regularizer deﬁned in terms of a differential operator, the optimal solution is givenby an expansion in the Green’s functions of the operator (which are analogous to the
eigenvectors of a discrete matrix), again with one basis function centred on each data

300 6. KERNEL METHODS
point. If the differential operator is isotropic then the Green’s functions depend only
on the radial distance from the corresponding data point. Due to the presence of theregularizer, the solution no longer interpolates the training data exactly.
Another motivation for radial basis functions comes from a consideration of
the interpolation problem when the input (rather than the target) variables are noisy(Webb, 1994; Bishop, 1995a). If the noise on the input variable xis described
by a variable ξhaving a distribution ν(ξ), then the sum-of-squares error function
becomes
E=1
2N∑
n=1∫
{y(xn+ξ)−tn}2ν(ξ)dξ. (6.39)
Using the calculus of variations, we can optimize with respect to the function f(x) Appendix D
to give Exercise 6.17
y(xn)=N∑
n=1tnh(x−xn) (6.40)
where the basis functions are given by
h(x−xn)=ν(x−xn)
N∑
n=1ν(x−xn). (6.41)
We see that there is one basis function centred on every data point. This is known as

becomes
E=1
2N∑
n=1∫
y(xn)=N∑
N∑
theNadaraya-Watson model and will be derived again from a different perspective
in Section 6.3.1. If the noise distribution ν(ξ)is isotropic, so that it is a function
only of ∥ξ∥, then the basis functions will be radial.
Note that the basis functions (6.41) are normalized, so that∑
nh(x−xn)=1
for any value of x. The effect of such normalization is shown in Figure 6.2. Normal-
ization is sometimes used in practice as it avoids having regions of input space where
all of the basis functions take small values, which would necessarily lead to predic-

tions in such regions that are either small or controlled purely by the bias parameter.
Another situation in which expansions in normalized radial basis functions arise
is in the application of kernel density estimation to the problem of regression, as weshall discuss in Section 6.3.1.
Because there is one basis function associated with every data point, the corre-
sponding model can be computationally costly to evaluate when making predictions
for new data points. Models have therefore been proposed (Broomhead and Lowe,

1988; Moody and Darken, 1989; Poggio and Girosi, 1990), which retain the expan-sion in radial basis functions but where the number Mof basis functions is smaller
than the number Nof data points. Typically, the number of basis functions, and the
locations µ
iof their centres, are determined based on the input data {xn}alone. The
basis functions are then kept ﬁxed and the coefﬁcients {wi}are determined by least
squares by solving the usual set of linear equations, as discussed in Section 3.1.1.

6.3. Radial Basis Function Networks 301
−1 −0.5 0 0.5 100.20.40.60.81
Figure 6.2 Plot of a set of Gaussian basis functions on the left, together with the corresponding normalized
basis functions on the right.
One of the simplest ways of choosing basis function centres is to use a randomly
chosen subset of the data points. A more systematic approach is called orthogonal
least squares (Chen et al. , 1991). This is a sequential selection process in which at
each step the next data point to be chosen as a basis function centre corresponds to
the one that gives the greatest reduction in the sum-of-squares error. Values for the
expansion coefﬁcients are determined as part of the algorithm. Clustering algorithms
such as K-means have also been used, which give a set of basis function centres that Section 9.1
no longer coincide with training data points.
6.3.1 Nadaraya-Watson model
In Section 3.3.3, we saw that the prediction of a linear regression model for a

new input xtakes the form of a linear combination of the training set target values
with coefﬁcients given by the ‘equivalent kernel’ (3.62) where the equivalent kernel
satisﬁes the summation constraint (3.64).
We can motivate the kernel regression model (3.61) from a different perspective,
starting with kernel density estimation. Suppose we have a training set {xn,tn}and
we use a Parzen density estimator to model the joint distribution p(x,t), so that Section 2.5.1
p(x,t)=1
NN∑
n=1f(x−xn,t−tn) (6.42)

p(x,t)=1
NN∑
where f(x,t)is the component density function, and there is one such component
centred on each data point. We now ﬁnd an expression for the regression function
y(x), corresponding to the conditional average of the target variable conditioned on

302 6. KERNEL METHODS
the input variable, which is given by
y(x)= E[t|x]=∫∞
−∞tp(t|x)dt
=∫
tp(x,t)dt
∫
p(x,t)dt
=∑
n∫
tf(x−xn,t−tn)dt
∑
m∫
f(x−xm,t−tm)dt. (6.43)
We now assume for simplicity that the component density functions have zero mean
so that ∫∞
−∞f(x,t)tdt=0 (6.44)
for all values of x. Using a simple change of variable, we then obtain
y(x)=∑
ng(x−xn)tn
∑
mg(x−xm)
=∑
nk(x,xn)tn (6.45)
where n, m=1,...,N and the kernel function k(x,xn)is given by
k(x,xn)=g(x−xn)∑
mg(x−xm)(6.46)
and we have deﬁned
g(x)=∫∞
−∞f(x,t)dt. (6.47)
The result (6.45) is known as the Nadaraya-Watson model, or kernel regression
(Nadaraya, 1964; Watson, 1964). For a localized kernel function, it has the prop-
erty of giving more weight to the data points xnthat are close to x. Note that the
kernel (6.46) satisﬁes the summation constraint
N∑
n=1k(x,xn)=1.

6.4. Gaussian Processes 303
Figure 6.3 Illustration of the Nadaraya-Watson kernel
regression model using isotropic Gaussian kernels, for the
sinusoidal data set. The original sine function is shown
by the green curve, the data points are shown in blue,
and each is the centre of an isotropic Gaussian kernel.
The resulting regression function, given by the condi-
tional mean, is shown by the red line, along with the two-
standard-deviation region for the conditional distribution
p(t|x)shown by the red shading. The blue ellipse around
each data point shows one standard deviation contour for
the corresponding kernel. These appear noncircular due
to the different scales on the horizontal and vertical axes.
In fact, this model deﬁnes not only a conditional expectation but also a full
conditional distribution given by
p(t|x)=p(t,x)∫
p(t,x)dt=∑
nf(x−xn,t−tn)
∑
m∫
f(x−xm,t−tm)dt(6.48)
from which other expectations can be evaluated.

∑
m∫
As an illustration we consider the case of a single input variable xin which
f(x, t)is given by a zero-mean isotropic Gaussian over the variable z=(x, t)with
variance σ2. The corresponding conditional distribution (6.48) is given by a Gaus-
sian mixture, and is shown, together with the conditional mean, for the sinusoidal Exercise 6.18
synthetic data set in Figure 6.3.
An obvious extension of this model is to allow for more ﬂexible forms of Gaus-

sian components, for instance having different variance parameters for the input and
target variables. More generally, we could model the joint distribution p(t,x)using
a Gaussian mixture model, trained using techniques discussed in Chapter 9 (Ghahra-
mani and Jordan, 1994), and then ﬁnd the corresponding conditional distribution
p(t|x). In this latter case we no longer have a representation in terms of kernel func-
tions evaluated at the training set data points. However, the number of components

in the mixture model can be smaller than the number of training set points, resulting
in a model that is faster to evaluate for test data points. We have thereby accepted an
increased computational cost during the training phase in order to have a model that
is faster at making predictions.
6.4. Gaussian Processes
In Section 6.1, we introduced kernels by applying the concept of duality to a non-
probabilistic model for regression. Here we extend the role of kernels to probabilis-

304 6. KERNEL METHODS
tic discriminative models, leading to the framework of Gaussian processes. We shall
thereby see how kernels arise naturally in a Bayesian setting.
In Chapter 3, we considered linear regression models of the form y(x,w)=
wTφ(x)in which wis a vector of parameters and φ(x)is a vector of ﬁxed nonlinear
basis functions that depend on the input vector x. We showed that a prior distribution
overwinduced a corresponding prior distribution over functions y(x,w). Given a
training data set, we then evaluated the posterior distribution over wand thereby
obtained the corresponding posterior distribution over regression functions, whichin turn (with the addition of noise) implies a predictive distribution p(t|x)for new
input vectors x.
In the Gaussian process viewpoint, we dispense with the parametric model and
instead deﬁne a prior probability distribution over functions directly. At ﬁrst sight, it

might seem difﬁcult to work with a distribution over the uncountably inﬁnite space offunctions. However, as we shall see, for a ﬁnite training set we only need to consider
the values of the function at the discrete set of input values x
ncorresponding to the
training set and test set data points, and so in practice we can work in a ﬁnite space.
Models equivalent to Gaussian processes have been widely studied in many dif-

ferent ﬁelds. For instance, in the geostatistics literature Gaussian process regression
is known as kriging (Cressie, 1993). Similarly, ARMA (autoregressive moving aver-
age) models, Kalman ﬁlters, and radial basis function networks can all be viewed as
forms of Gaussian process models. Reviews of Gaussian processes from a machine
learning perspective can be found in MacKay (1998), Williams (1999), and MacKay(2003), and a comparison of Gaussian process models with alternative approaches is
given in Rasmussen (1996). See also Rasmussen and Williams (2006) for a recent

textbook on Gaussian processes.
6.4.1 Linear regression revisited
In order to motivate the Gaussian process viewpoint, let us return to the linear
regression example and re-derive the predictive distribution by working in terms
of distributions over functions y(x,w). This will provide a speciﬁc example of a
Gaussian process.
Consider a model deﬁned in terms of a linear combination of Mﬁxed basis
functions given by the elements of the vector φ(x)so that
y(x)=wTφ(x) (6.49)

wherexis the input vector and wis theM-dimensional weight vector. Now consider
a prior distribution over wgiven by an isotropic Gaussian of the form
p(w)=N(w|0,α−1I) (6.50)
governed by the hyperparameter α, which represents the precision (inverse variance)
of the distribution. For any given value of w, the deﬁnition (6.49) deﬁnes a partic-
ular function of x. The probability distribution over wdeﬁned by (6.50) therefore
induces a probability distribution over functions y(x). In practice, we wish to eval-

uate this function at speciﬁc values of x, for example at the training data points

6.4. Gaussian Processes 305
x1,...,xN. We are therefore interested in the joint distribution of the function val-
uesy(x1),...,y (xN), which we denote by the vector ywith elements yn=y(xn)
forn=1,...,N . From (6.49), this vector is given by
y=Φw (6.51)
whereΦis the design matrix with elements Φnk=φk(xn). We can ﬁnd the proba-
bility distribution of yas follows. First of all we note that yis a linear combination of
Gaussian distributed variables given by the elements of wand hence is itself Gaus-
sian. We therefore need only to ﬁnd its mean and covariance, which are given from Exercise 2.31
(6.50) by
E[y]= ΦE[w]=0 (6.52)
cov[ y]= E[
yyT]
=ΦE[
wwT]
ΦT=1
αΦΦT=K (6.53)
whereKis the Gram matrix with elements
Knm=k(xn,xm)=1
αφ(xn)Tφ(xm) (6.54)
andk(x,x′)is the kernel function.
This model provides us with a particular example of a Gaussian process. In gen-
eral, a Gaussian process is deﬁned as a probability distribution over functions y(x)

(6.50) by
cov[ y]= E[
yyT]
=ΦE[
wwT]
ΦT=1
such that the set of values of y(x)evaluated at an arbitrary set of points x1,...,xN
jointly have a Gaussian distribution. In cases where the input vector xis two di-
mensional, this may also be known as a Gaussian random ﬁeld . More generally, a
stochastic process y(x)is speciﬁed by giving the joint probability distribution for
any ﬁnite set of values y(x1),...,y (xN)in a consistent manner.
A key point about Gaussian stochastic processes is that the joint distribution
overNvariables y1,...,y Nis speciﬁed completely by the second-order statistics,

namely the mean and the covariance. In most applications, we will not have anyprior knowledge about the mean of y(x)and so by symmetry we take it to be zero.
This is equivalent to choosing the mean of the prior over weight values p(w|α)to
be zero in the basis function viewpoint. The speciﬁcation of the Gaussian process is
then completed by giving the covariance of y(x)evaluated at any two values of x,
which is given by the kernel function
E[y(x
n)y(xm)] =k(xn,xm). (6.55)

E[y(x
For the speciﬁc case of a Gaussian process deﬁned by the linear regression model
(6.49) with a weight prior (6.50), the kernel function is given by (6.54).
We can also deﬁne the kernel function directly, rather than indirectly through a
choice of basis function. Figure 6.4 shows samples of functions drawn from Gaus-
sian processes for two different choices of kernel function. The ﬁrst of these is a‘Gaussian’ kernel of the form (6.23), and the second is the exponential kernel given
by
k(x, x
′)=e x p( −θ|x−x′|) (6.56)

by
k(x, x
which corresponds to the Ornstein-Uhlenbeck process originally introduced by Uh-
lenbeck and Ornstein (1930) to describe Brownian motion.

306 6. KERNEL METHODS
Figure 6.4 Samples from Gaus-
sian processes for a ‘Gaussian’ ker-
nel (left) and an exponential kernel
(right).
−1 −0.5 0 0.5 1−3−1.501.53
6.4.2 Gaussian processes for regression
In order to apply Gaussian process models to the problem of regression, we need
to take account of the noise on the observed target values, which are given by
tn=yn+ϵn (6.57)
where yn=y(xn), andϵnis a random noise variable whose value is chosen inde-
pendently for each observation n. Here we shall consider noise processes that have
a Gaussian distribution, so that
p(tn|yn)=N(tn|yn,β−1) (6.58)
where βis a hyperparameter representing the precision of the noise. Because the
noise is independent for each data point, the joint distribution of the target values
t=(t1,...,t N)Tconditioned on the values of y=(y1,...,y N)Tis given by an
p(t|y)=N(t|y,β−1IN) (6.59)
whereINdenotes the N×Nunit matrix. From the deﬁnition of a Gaussian process,

the marginal distribution p(y)is given by a Gaussian whose mean is zero and whose
covariance is deﬁned by a Gram matrix Kso that
p(y)=N(y|0,K). (6.60)
The kernel function that determines Kis typically chosen to express the property
that, for points xnandxmthat are similar, the corresponding values y(xn)and
y(xm)will be more strongly correlated than for dissimilar points. Here the notion
of similarity will depend on the application.
In order to ﬁnd the marginal distribution p(t), conditioned on the input values

x1,...,xN, we need to integrate over y. This can be done by making use of the
results from Section 2.3.3 for the linear-Gaussian model. Using (2.115), we see that
the marginal distribution of tis given by
p(t)=∫
p(t|y)p(y)dy=N(t|0,C) (6.61)

6.4. Gaussian Processes 307
where the covariance matrix Chas elements
C(xn,xm)=k(xn,xm)+β−1δnm. (6.62)
This result reﬂects the fact that the two Gaussian sources of randomness, namely
that associated with y(x)and that associated with ϵ, are independent and so their
covariances simply add.
One widely used kernel function for Gaussian process regression is given by the
exponential of a quadratic form, with the addition of constant and linear terms to
give
k(xn,xm)=θ0exp{
−θ1
2∥xn−xm∥2}
+θ2+θ3xT
nxm. (6.63)
Note that the term involving θ3corresponds to a parametric model that is a linear
function of the input variables. Samples from this prior are plotted for various values
of the parameters θ0,...,θ 3in Figure 6.5, and Figure 6.6 shows a set of points sam-
pled from the joint distribution (6.60) along with the corresponding values deﬁned
by (6.61).
So far, we have used the Gaussian process viewpoint to build a model of the
joint distribution over sets of data points. Our goal in regression, however, is to

by (6.61).
make predictions of the target variables for new inputs, given a set of training data.
Let us suppose that tN=(t1,...,t N)T, corresponding to input values x1,...,xN,
comprise the observed training set, and our goal is to predict the target variable tN+1
for a new input vector xN+1. This requires that we evaluate the predictive distri-
bution p(tN+1|tN). Note that this distribution is conditioned also on the variables
x1,...,xNandxN+1. However, to keep the notation simple we will not show these

conditioning variables explicitly.
To ﬁnd the conditional distribution p(tN+1|t), we begin by writing down the
joint distribution p(tN+1), where tN+1denotes the vector (t1,...,t N,tN+1)T.W e
then apply the results from Section 2.3.1 to obtain the required conditional distribu-tion, as illustrated in Figure 6.7.
From (6.61), the joint distribution over t
1,...,t N+1will be given by
p(tN+1)=N(tN+1|0,CN+1) (6.64)
whereCN+1is an(N+1 )×(N+1 ) covariance matrix with elements given by

(6.62). Because this joint distribution is Gaussian, we can apply the results from
Section 2.3.1 to ﬁnd the conditional Gaussian distribution. To do this, we partitionthe covariance matrix as follows
C
N+1=(
CNk
kTc)
(6.65)
whereCNis theN×Ncovariance matrix with elements given by (6.62) for n, m=
1,...,N , the vector khas elements k(xn,xN+1)forn=1,...,N , and the scalar

308 6. KERNEL METHODS
(1.00,4.00,0.00,0.00)
−1 −0.5 0 0.5 1−3−1.501.53(9.00,4.00,0.00,0.00)
−1 −0.5 0 0.5 1−9−4.504.59(1.00,64.00,0.00,0.00)
(1.00,0.25,0.00,0.00)
−1 −0.5 0 0.5 1−3−1.501.53(1.00,4.00,10.00,0.00)
−1 −0.5 0 0.5 1−9−4.504.59(1.00,4.00,0.00,5.00)
−1 −0.5 0 0.5 1−4−2024
Figure 6.5 Samples from a Gaussian process prior deﬁned by the covariance function (6.63). The title above
each plot denotes (θ0,θ1,θ2,θ3).
c=k(xN+1,xN+1)+β−1. Using the results (2.81) and (2.82), we see that the con-
ditional distribution p(tN+1|t)is a Gaussian distribution with mean and covariance
given by
m(xN+1)= kTC−1
Nt (6.66)
σ2(xN+1)= c−kTC−1
Nk. (6.67)
These are the key results that deﬁne Gaussian process regression. Because the vector
kis a function of the test point input value xN+1, we see that the predictive distribu-
tion is a Gaussian whose mean and variance both depend on xN+1. An example of
Gaussian process regression is shown in Figure 6.8.

given by
Nt (6.66)
Nk. (6.67)
The only restriction on the kernel function is that the covariance matrix given by
(6.62) must be positive deﬁnite. If λiis an eigenvalue of K, then the corresponding
eigenvalue of Cwill be λi+β−1. It is therefore sufﬁcient that the kernel matrix
k(xn,xm)be positive semideﬁnite for any pair of points xnandxm, so that λi⩾0,
because any eigenvalue λithat is zero will still give rise to a positive eigenvalue
forCbecause β>0. This is the same restriction on the kernel function discussed
earlier, and so we can again exploit all of the techniques in Section 6.2 to construct

6.4. Gaussian Processes 309
Figure 6.6 Illustration of the sampling of data
points {tn}from a Gaussian process.
The blue curve shows a sample func-
tion from the Gaussian process prior
over functions, and the red points
show the values of ynobtained by
evaluating the function at a set of in-
put values {xn}. The correspond-
ing values of {tn}, shown in green,
are obtained by adding independent
Gaussian noise to each of the {yn}.
xt
−1 0 1−303
suitable kernels.
Note that the mean (6.66) of the predictive distribution can be written, as a func-
tion of xN+1, in the form
m(xN+1)=N∑
n=1ank(xn,xN+1) (6.68)
where anis the nthcomponent of C−1
Nt. Thus, if the kernel function k(xn,xm)
depends only on the distance ∥xn−xm∥, then we obtain an expansion in radial
basis functions.
The results (6.66) and (6.67) deﬁne the predictive distribution for Gaussian pro-
cess regression with an arbitrary kernel function k(xn,xm). In the particular case in

−1 0 1−303
which the kernel function k(x,x′)is deﬁned in terms of a ﬁnite set of basis functions,
we can derive the results obtained previously in Section 3.3.2 for linear regression
starting from the Gaussian process viewpoint. Exercise 6.21
For such models, we can therefore obtain the predictive distribution either by
taking a parameter space viewpoint and using the linear regression result or by taking
a function space viewpoint and using the Gaussian process result.

The central computational operation in using Gaussian processes will involve
the inversion of a matrix of size N×N, for which standard methods require O(N3)
computations. By contrast, in the basis function model we have to invert a matrix
SNof size M×M, which has O(M3)computational complexity. Note that for
both viewpoints, the matrix inversion must be performed once for the given training
set. For each new test point, both methods require a vector-matrix multiply, which
has cost O(N2)in the Gaussian process case and O(M2)for the linear basis func-

tion model. If the number Mof basis functions is smaller than the number Nof
data points, it will be computationally more efﬁcient to work in the basis function

310 6. KERNEL METHODS
Figure 6.7 Illustration of the mechanism of
Gaussian process regression for
the case of one training point and
one test point, in which the red el-
lipses show contours of the joint dis-
tribution p(t1,t2). Here t1is the
training data point, and condition-
i n go nt h ev a l u eo f t1, correspond-
ing to the vertical blue line, we ob-
tainp(t2|t1)shown as a function of
t2by the green curve. t1t2
m(x2)
−1 0 1−101
framework. However, an advantage of a Gaussian processes viewpoint is that we
can consider covariance functions that can only be expressed in terms of an inﬁnite
number of basis functions.
For large training data sets, however, the direct application of Gaussian process
methods can become infeasible, and so a range of approximation schemes have been
developed that have better scaling with training set size than the exact approach
(Gibbs, 1997; Tresp, 2001; Smola and Bartlett, 2001; Williams and Seeger, 2001;

Csat´o and Opper, 2002; Seeger et al. , 2003). Practical issues in the application of
Gaussian processes are discussed in Bishop and Nabney (2008).
We have introduced Gaussian process regression for the case of a single tar-
get variable. The extension of this formalism to multiple target variables, known
as co-kriging (Cressie, 1993), is straightforward. Various other extensions of Gaus- Exercise 6.23
Figure 6.8 Illustration of Gaussian process re-
gression applied to the sinusoidal
data set in Figure A.6 in which the
three right-most data points have

been omitted. The green curve
shows the sinusoidal function from
which the data points, shown in
blue, are obtained by sampling and
addition of Gaussian noise. The
red line shows the mean of the
Gaussian process predictive distri-
bution, and the shaded region cor-
responds to plus and minus two
standard deviations. Notice how
the uncertainty increases in the re-
gion to the right of the data points.0 0.2 0.4 0.6 0.8 1−1−0.500.51

6.4. Gaussian Processes 311
sian process regression have also been considered, for purposes such as modelling
the distribution over low-dimensional manifolds for unsupervised learning (Bishopet al. , 1998a) and the solution of stochastic differential equations (Graepel, 2003).
6.4.3 Learning the hyperparameters
The predictions of a Gaussian process model will depend, in part, on the choice
of covariance function. In practice, rather than ﬁxing the covariance function, we
may prefer to use a parametric family of functions and then infer the parametervalues from the data. These parameters govern such things as the length scale of the
correlations and the precision of the noise and correspond to the hyperparameters in
a standard parametric model.
Techniques for learning the hyperparameters are based on the evaluation of the
likelihood function p(t|θ)where θdenotes the hyperparameters of the Gaussian pro-
cess model. The simplest approach is to make a point estimate of θby maximizing

the log likelihood function. Because θrepresents a set of hyperparameters for the
regression problem, this can be viewed as analogous to the type 2 maximum like-
lihood procedure for linear regression models. Maximization of the log likelihood Section 3.5
can be done using efﬁcient gradient-based optimization algorithms such as conjugate
gradients (Fletcher, 1987; Nocedal and Wright, 1999; Bishop and Nabney, 2008).
The log likelihood function for a Gaussian process regression model is easily
evaluated using the standard form for a multivariate Gaussian distribution, giving
lnp(t|θ)=−1
2ln|CN|−1
2tTC−1
Nt−N
2ln(2π). (6.69)

2ln|CN|−1
2tTC−1
Nt−N
For nonlinear optimization, we also need the gradient of the log likelihood func-
tion with respect to the parameter vector θ. We shall assume that evaluation of the
derivatives of CNis straightforward, as would be the case for the covariance func-
tions considered in this chapter. Making use of the result (C.21) for the derivative ofC
−1
N, together with the result (C.22) for the derivative of ln|CN|, we obtain
∂
∂θilnp(t|θ)=−1
2Tr(
C−1
N∂CN
∂θi)
+1
2tTC−1
N∂CN
∂θiC−1
Nt. (6.70)

Nt−N
−1
∂
2Tr(
C−1
N∂CN
∂θi)
+1
2tTC−1
N∂CN
∂θiC−1
Nt. (6.70)
Because lnp(t|θ)will in general be a nonconvex function, it can have multiple max-
ima.
It is straightforward to introduce a prior over θand to maximize the log poste-
rior using gradient-based methods. In a fully Bayesian treatment, we need to evaluatemarginals over θweighted by the product of the prior p(θ)and the likelihood func-
tionp(t|θ). In general, however, exact marginalization will be intractable, and we
must resort to approximations.

2Tr(
C−1
N∂CN
∂θi)
+1
2tTC−1
N∂CN
∂θiC−1
Nt. (6.70)
ima.
The Gaussian process regression model gives a predictive distribution whose
mean and variance are functions of the input vector x. However, we have assumed
that the contribution to the predictive variance arising from the additive noise, gov-
erned by the parameter β, is a constant. For some problems, known as heteroscedas-
tic, the noise variance itself will also depend on x. To model this, we can extend the

312 6. KERNEL METHODS
Figure 6.9 Samples from the ARD
prior for Gaussian processes, in
which the kernel function is given by
(6.71). The left plot corresponds toη
1=η2=1, and the right plot cor-
responds to η1=1,η2=0.01.
Gaussian process framework by introducing a second Gaussian process to represent
the dependence of βon the input x(Goldberg et al. , 1998). Because βis a variance,
and hence nonnegative, we use the Gaussian process to model lnβ(x).
6.4.4 Automatic relevance determination
In the previous section, we saw how maximum likelihood could be used to de-
termine a value for the correlation length-scale parameter in a Gaussian process.This technique can usefully be extended by incorporating a separate parameter for
each input variable (Rasmussen and Williams, 2006). The result, as we shall see, is
that the optimization of these parameters by maximum likelihood allows the relative
importance of different inputs to be inferred from the data. This represents an exam-

ple in the Gaussian process context of automatic relevance determination ,o r ARD ,
which was originally formulated in the framework of neural networks (MacKay,
1994; Neal, 1996). The mechanism by which appropriate inputs are preferred is
discussed in Section 7.2.2.
Consider a Gaussian process with a two-dimensional input space x=(x
1,x2),
having a kernel function of the form
k(x,x′)=θ0exp{
−1
22∑
i=1ηi(xi−x′
i)2}
. (6.71)
Samples from the resulting prior over functions y(x)are shown for two different

1,x2),
−1
22∑
i)2}
. (6.71)
settings of the precision parameters ηiin Figure 6.9. We see that, as a particu-
lar parameter ηibecomes small, the function becomes relatively insensitive to the
corresponding input variable xi. By adapting these parameters to a data set using
maximum likelihood, it becomes possible to detect input variables that have little
effect on the predictive distribution, because the corresponding values of ηiwill be

i)2}
. (6.71)
small. This can be useful in practice because it allows such inputs to be discarded.ARD is illustrated using a simple synthetic data set having three inputs x
1,x2andx3
(Nabney, 2002) in Figure 6.10. The target variable t, is generated by sampling 100
values of x1from a Gaussian, evaluating the function sin(2πx1), and then adding

6.4. Gaussian Processes 313
Figure 6.10 Illustration of automatic rele-
vance determination in a Gaus-
sian process for a synthetic prob-
lem having three inputs x1,x2,
andx3, for which the curves
show the corresponding values of
the hyperparameters η1(red), η2
(green), and η3(blue) as a func-
tion of the number of iterations
when optimizing the marginal
likelihood. Details are given in
the text. Note the logarithmic
scale on the vertical axis.
0 20 40 60 80 10010−410−2100102
Gaussian noise. Values of x2are given by copying the corresponding values of x1
and adding noise, and values of x3are sampled from an independent Gaussian dis-
tribution. Thus x1is a good predictor of t,x2is a more noisy predictor of t, andx3
has only chance correlations with t. The marginal likelihood for a Gaussian process
with ARD parameters η1,η2,η3is optimized using the scaled conjugate gradients
algorithm. We see from Figure 6.10 that η1converges to a relatively large value, η2

converges to a much smaller value, and η3becomes very small indicating that x3is
irrelevant for predicting t.
The ARD framework is easily incorporated into the exponential-quadratic kernel
(6.63) to give the following form of kernel function, which has been found useful for
applications of Gaussian processes to a range of regression problems
−1
2D∑
i=1ηi(xni−xmi)2}
+θ2+θ3D∑
i=1xnixmi (6.72)
where Dis the dimensionality of the input space.
6.4.5 Gaussian processes for classiﬁcation

−1
2D∑
+θ2+θ3D∑
In a probabilistic approach to classiﬁcation, our goal is to model the posterior
probabilities of the target variable for a new input vector, given a set of training
data. These probabilities must lie in the interval (0,1), whereas a Gaussian process
model makes predictions that lie on the entire real axis. However, we can easily
adapt Gaussian processes to classiﬁcation problems by transforming the output of
the Gaussian process using an appropriate nonlinear activation function.

Consider ﬁrst the two-class problem with a target variable t∈{0,1}. If we de-
ﬁne a Gaussian process over a function a(x)and then transform the function using
a logistic sigmoid y=σ(a), given by (4.59), then we will obtain a non-Gaussian
stochastic process over functions y(x)where y∈(0,1). This is illustrated for the
case of a one-dimensional input space in Figure 6.11 in which the probability distri-

314 6. KERNEL METHODS
−1 −0.5 0 0.5 1−10−50510
−1 −0.5 0 0.5 100.250.50.751
Figure 6.11 The left plot shows a sample from a Gaussian process prior over functions a(x), and the right plot
shows the result of transforming this sample using a logistic sigmoid function.
bution over the target variable tis then given by the Bernoulli distribution
p(t|a)=σ(a)t(1−σ(a))1−t. (6.73)
As usual, we denote the training set inputs by x1,...,xNwith corresponding
observed target variables t=(t1,...,t N)T. We also consider a single test point
xN+1with target value tN+1. Our goal is to determine the predictive distribution
p(tN+1|t), where we have left the conditioning on the input variables implicit. To do
this we introduce a Gaussian process prior over the vector aN+1, which has compo-
nentsa(x1),...,a (xN+1). This in turn deﬁnes a non-Gaussian process over tN+1,
and by conditioning on the training data tNwe obtain the required predictive distri-
bution. The Gaussian process prior for aN+1takes the form

p(aN+1)=N(aN+1|0,CN+1). (6.74)
Unlike the regression case, the covariance matrix no longer includes a noise term
because we assume that all of the training data points are correctly labelled. How-
ever, for numerical reasons it is convenient to introduce a noise-like term governed
by a parameter νthat ensures that the covariance matrix is positive deﬁnite. Thus
the covariance matrix CN+1has elements given by
C(xn,xm)=k(xn,xm)+νδnm (6.75)
where k(xn,xm)is any positive semideﬁnite kernel function of the kind considered

in Section 6.2, and the value of νis typically ﬁxed in advance. We shall assume that
the kernel function k(x,x′)is governed by a vector θof parameters, and we shall
later discuss how θmay be learned from the training data.
For two-class problems, it is sufﬁcient to predict p(tN+1=1|tN)because the
value of p(tN+1=0|tN)is then given by 1−p(tN+1=1|tN). The required

6.4. Gaussian Processes 315
predictive distribution is given by
p(tN+1=1|tN)=∫
p(tN+1=1|aN+1)p(aN+1|tN)daN+1 (6.76)
where p(tN+1=1|aN+1)=σ(aN+1).
This integral is analytically intractable, and so may be approximated using sam-
pling methods (Neal, 1997). Alternatively, we can consider techniques based onan analytical approximation. In Section 4.5.2, we derived the approximate formula
(4.153) for the convolution of a logistic sigmoid with a Gaussian distribution. We
can use this result to evaluate the integral in (6.76) provided we have a Gaussianapproximation to the posterior distribution p(a
N+1|tN). The usual justiﬁcation for a
Gaussian approximation to a posterior distribution is that the true posterior will tend
to a Gaussian as the number of data points increases as a consequence of the centrallimit theorem. In the case of Gaussian processes, the number of variables grows with Section 2.3
the number of data points, and so this argument does not apply directly. However, if

we consider increasing the number of data points falling in a ﬁxed region of xspace,
then the corresponding uncertainty in the function a(x)will decrease, again leading
asymptotically to a Gaussian (Williams and Barber, 1998).
Three different approaches to obtaining a Gaussian approximation have been
considered. One technique is based on variational inference (Gibbs and MacKay, Section 10.1
2000) and makes use of the local variational bound (10.144) on the logistic sigmoid.This allows the product of sigmoid functions to be approximated by a product of
Gaussians thereby allowing the marginalization over a

Nto be performed analyti-
cally. The approach also yields a lower bound on the likelihood function p(tN|θ).
The variational framework for Gaussian process classiﬁcation can also be extended
to multiclass ( K>2) problems by using a Gaussian approximation to the softmax
function (Gibbs, 1997).
A second approach uses expectation propagation (Opper and Winther, 2000b; Section 10.7
Minka, 2001b; Seeger, 2003). Because the true posterior distribution is unimodal, as
we shall see shortly, the expectation propagation approach can give good results.
6.4.6 Laplace approximation

The third approach to Gaussian process classiﬁcation is based on the Laplace
approximation, which we now consider in detail. In order to evaluate the predictive Section 4.4
distribution (6.76), we seek a Gaussian approximation to the posterior distribution
overaN+1, which, using Bayes’ theorem, is given by
p(aN+1|tN)=∫
p(aN+1,aN|tN)daN
=1
p(tN)∫
p(aN+1,aN)p(tN|aN+1,aN)daN
=1
p(tN)∫
p(aN+1|aN)p(aN)p(tN|aN)daN
=∫
p(aN+1|aN)p(aN|tN)daN (6.77)

316 6. KERNEL METHODS
where we have used p(tN|aN+1,aN)=p(tN|aN). The conditional distribution
p(aN+1|aN)is obtained by invoking the results (6.66) and (6.67) for Gaussian pro-
cess regression, to give
p(aN+1|aN)=N(aN+1|kTC−1
NaN,c−kTC−1
Nk). (6.78)
We can therefore evaluate the integral in (6.77) by ﬁnding a Laplace approximation
for the posterior distribution p(aN|tN), and then using the standard result for the
convolution of two Gaussian distributions.
The prior p(aN)is given by a zero-mean Gaussian process with covariance ma-
trixCN, and the data term (assuming independence of the data points) is given by
p(tN|aN)=N∏
n=1σ(an)tn(1−σ(an))1−tn=N∏
n=1eantnσ(−an). (6.79)
We then obtain the Laplace approximation by Taylor expanding the logarithm of
p(aN|tN), which up to an additive normalization constant is given by the quantity
Ψ(aN)=l n p(aN)+l n p(tN|aN)
=−1
2aT
NC−1
NaN−N
2ln(2π)−1
2ln|CN|+tT
NaN
−N∑
n=1ln(1 + ean) + const . (6.80)

=−1
2aT
NC−1
NaN−N
2ln(2π)−1
NaN
−N∑
First we need to ﬁnd the mode of the posterior distribution, and this requires that we
evaluate the gradient of Ψ(aN), which is given by
∇Ψ(aN)=tN−σN−C−1
NaN (6.81)
where σNis a vector with elements σ(an). We cannot simply ﬁnd the mode by
setting this gradient to zero, because σNdepends nonlinearly on aN, and so we
resort to an iterative scheme based on the Newton-Raphson method, which gives rise
to an iterative reweighted least squares (IRLS) algorithm. This requires the second Section 4.3.3

NaN (6.81)
derivatives of Ψ(aN), which we also require for the Laplace approximation anyway,
and which are given by
∇∇Ψ(aN)=−WN−C−1
N(6.82)
whereWNis a diagonal matrix with elements σ(an)(1−σ(an)), and we have used
the result (4.88) for the derivative of the logistic sigmoid function. Note that these
diagonal elements lie in the range (0,1/4), and hence WNis a positive deﬁnite
matrix. Because CN(and hence its inverse) is positive deﬁnite by construction, and

N(6.82)
because the sum of two positive deﬁnite matrices is also positive deﬁnite, we see Exercise 6.24
that the Hessian matrix A=−∇∇Ψ(aN)is positive deﬁnite and so the posterior
distribution p(aN|tN)is log convex and therefore has a single mode that is the global

6.4. Gaussian Processes 317
maximum. The posterior distribution is not Gaussian, however, because the Hessian
is a function of aN.
Using the Newton-Raphson formula (4.92), the iterative update equation for aN
is given by Exercise 6.25
anew
N=CN(I+WNCN)−1{tN−σN+WNaN}. (6.83)
These equations are iterated until they converge to the mode which we denote by
a⋆
N. At the mode, the gradient ∇Ψ(aN)will vanish, and hence a⋆
Nwill satisfy
a⋆
N=CN(tN−σN). (6.84)
Once we have found the mode a⋆
Nof the posterior, we can evaluate the Hessian
matrix given by
H=−∇∇Ψ(aN)=WN+C−1
N(6.85)
where the elements of WNare evaluated using a⋆
N. This deﬁnes our Gaussian ap-
proximation to the posterior distribution p(aN|tN)given by
q(aN)=N(aN|a⋆
N,H−1). (6.86)
We can now combine this with (6.78) and hence evaluate the integral (6.77). Because
this corresponds to a linear-Gaussian model, we can use the general result (2.115) togive Exercise 6.26
E[a
N+1|tN]= kT(tN−σN) (6.87)
var[aN+1|tN]= c−kT(W−1
N+CN)−1k. (6.88)

N(6.85)
E[a
Now that we have a Gaussian distribution for p(aN+1|tN), we can approximate
the integral (6.76) using the result (4.153). As with the Bayesian logistic regression
model of Section 4.5, if we are only interested in the decision boundary correspond-ing to p(t
N+1|tN)=0.5, then we need only consider the mean and we can ignore
the effect of the variance.
We also need to determine the parameters θof the covariance function. One
approach is to maximize the likelihood function given by p(tN|θ)for which we need

expressions for the log likelihood and its gradient. If desired, suitable regularization
terms can also be added, leading to a penalized maximum likelihood solution. Thelikelihood function is deﬁned by
p(t
N|θ)=∫
p(tN|aN)p(aN|θ)daN. (6.89)
This integral is analytically intractable, so again we make use of the Laplace approx-
imation. Using the result (4.135), we obtain the following approximation for the logof the likelihood function
lnp(t
N|θ)=Ψ ( a⋆
N)−1
2ln|WN+C−1
N|+N
2ln(2π) (6.90)

318 6. KERNEL METHODS
where Ψ(a⋆
N)=l n p(a⋆
N|θ)+l n p(tN|a⋆
N). We also need to evaluate the gradient
oflnp(tN|θ)with respect to the parameter vector θ. Note that changes in θwill
cause changes in a⋆
N, leading to additional terms in the gradient. Thus, when we
differentiate (6.90) with respect to θ, we obtain two sets of terms, the ﬁrst arising
from the dependence of the covariance matrix CNonθ, and the rest arising from
dependence of a⋆
Nonθ.
The terms arising from the explicit dependence on θcan be found by using
(6.80) together with the results (C.21) and (C.22), and are given by
∂lnp(tN|θ)
∂θj=1
2a⋆T
NC−1
N∂CN
∂θjC−1
Na⋆
N
−1
2Tr[
(I+CNWN)−1WN∂CN
∂θj]
. (6.91)
To compute the terms arising from the dependence of a⋆
Nonθ, we note that
the Laplace approximation has been constructed such that Ψ(aN)has zero gradient
ataN=a⋆
N, and so Ψ(a⋆
N)gives no contribution to the gradient as a result of its
dependence on a⋆
N. This leaves the following contribution to the derivative with
respect to a component θjofθ

∂θj=1
2a⋆T
NC−1
N∂CN
∂θjC−1
Na⋆
N
−1
2Tr[
∂θj]
. (6.91)
ataN=a⋆
−1
2N∑
n=1∂ln|WN+C−1
N|
∂a⋆n∂a⋆
n
∂θj
=−1
2N∑
n=1[
(I+CNWN)−1CN]
nnσ⋆
n(1−σ⋆
n)(1−2σ⋆
n)∂a⋆
n
∂θj(6.92)
where σ⋆
n=σ(a⋆
n), and again we have used the result (C.22) together with the
deﬁnition of WN. We can evaluate the derivative of a⋆
Nwith respect to θjby differ-
entiating the relation (6.84) with respect to θjto give
∂a⋆
n
∂θj=∂CN
∂θj(tN−σN)−CNWN∂a⋆
n
∂θj. (6.93)
Rearranging then gives
∂a⋆
n
∂θj=(I+WNCN)−1∂CN
∂θj(tN−σN). (6.94)
Combining (6.91), (6.92), and (6.94), we can evaluate the gradient of the log

2N∑
N|
∂a⋆n∂a⋆
n
∂θj
=−1
2N∑
n=1[
nnσ⋆
n(1−σ⋆
n)(1−2σ⋆
n)∂a⋆
n
∂θj(6.92)
where σ⋆
n=σ(a⋆
∂a⋆
n
∂θj=∂CN
n
∂a⋆
n
likelihood function, which can be used with standard nonlinear optimization algo-
rithms in order to determine a value for θ.
We can illustrate the application of the Laplace approximation for Gaussian pro-
cesses using the synthetic two-class data set shown in Figure 6.12. Extension of the Appendix A
Laplace approximation to Gaussian processes involving K> 2classes, using the
softmax activation function, is straightforward (Williams and Barber, 1998).

6.4. Gaussian Processes 319
−2 0 2−202
Figure 6.12 Illustration of the use of a Gaussian process for classiﬁcation, showing the data on the left together
with the optimal decision boundary from the true distribution in green, and the decision boundary from the
Gaussian process classiﬁer in black. On the right is the predicted posterior probability for the blue and red
classes together with the Gaussian process decision boundary.
6.4.7 Connection to neural networks
We have seen that the range of functions which can be represented by a neural
network is governed by the number Mof hidden units, and that, for sufﬁciently
largeM, a two-layer network can approximate any given function with arbitrary
accuracy. In the framework of maximum likelihood, the number of hidden units
needs to be limited (to a level dependent on the size of the training set) in order
to avoid over-ﬁtting. However, from a Bayesian perspective it makes little sense to

limit the number of parameters in the network according to the size of the training
set.
In a Bayesian neural network, the prior distribution over the parameter vector
w, in conjunction with the network function f(x,w), produces a prior distribution
over functions from y(x)whereyis the vector of network outputs. Neal (1996)
has shown that, for a broad class of prior distributions over w, the distribution of
functions generated by a neural network will tend to a Gaussian process in the limit

set.
M→∞ . It should be noted, however, that in this limit the output variables of the
neural network become independent. One of the great merits of neural networks is
that the outputs share the hidden units and so they can ‘borrow statistical strength’
from each other, that is, the weights associated with each hidden unit are inﬂuenced
by all of the output variables not just by one of them. This property is therefore lost
in the Gaussian process limit.

We have seen that a Gaussian process is determined by its covariance (kernel)
function. Williams (1998) has given explicit forms for the covariance in the case of
two speciﬁc choices for the hidden unit activation function (probit and Gaussian).
These kernel functions k(x,x′)are nonstationary, i.e. they cannot be expressed as
a function of the difference x−x′, as a consequence of the Gaussian weight prior
being centred on zero which breaks translation invariance in weight space.

320 6. KERNEL METHODS
By working directly with the covariance function we have implicitly marginal-
ized over the distribution of weights. If the weight prior is governed by hyperpa-rameters, then their values will determine the length scales of the distribution over
functions, as can be understood by studying the examples in Figure 5.11 for the case
of a ﬁnite number of hidden units. Note that we cannot marginalize out the hyperpa-rameters analytically, and must instead resort to techniques of the kind discussed in
Exercises
6.1 (⋆⋆)www Consider the dual formulation of the least squares linear regression
problem given in Section 6.1. Show that the solution for the components anof
the vector acan be expressed as a linear combination of the elements of the vector
φ(xn). Denoting these coefﬁcients by the vector w, show that the dual of the dual
formulation is given by the original representation in terms of the parameter vector
w.

Exercises
w.
6.2 (⋆⋆)In this exercise, we develop a dual formulation of the perceptron learning
algorithm. Using the perceptron learning rule (4.55), show that the learned weightvectorwcan be written as a linear combination of the vectors t
nφ(xn)where tn∈
{−1,+1}. Denote the coefﬁcients of this linear combination by αnand derive a
formulation of the perceptron learning algorithm, and the predictive function for theperceptron, in terms of the α
n. Show that the feature vector φ(x)enters only in the
form of the kernel function k(x,x′)=φ(x)Tφ(x′).

6.3 (⋆)The nearest-neighbour classiﬁer (Section 2.5.2) assigns a new input vector x
to the same class as that of the nearest input vector xnfrom the training set, where
in the simplest case, the distance is deﬁned by the Euclidean metric ∥x−xn∥2.B y
expressing this rule in terms of scalar products and then making use of kernel sub-
stitution, formulate the nearest-neighbour classiﬁer for a general nonlinear kernel.
6.4 (⋆)In Appendix C, we give an example of a matrix that has positive elements but

that has a negative eigenvalue and hence that is not positive deﬁnite. Find an exampleof the converse property, namely a 2×2matrix with positive eigenvalues yet that
has at least one negative element.
6.5 (⋆)
www Verify the results (6.13) and (6.14) for constructing valid kernels.
6.6 (⋆)Verify the results (6.15) and (6.16) for constructing valid kernels.
6.7 (⋆)www Verify the results (6.17) and (6.18) for constructing valid kernels.
6.8 (⋆)Verify the results (6.19) and (6.20) for constructing valid kernels.

6.5 (⋆)
6.9 (⋆)Verify the results (6.21) and (6.22) for constructing valid kernels.
6.10 (⋆)Show that an excellent choice of kernel for learning a function f(x)is given
byk(x,x′)=f(x)f(x′)by showing that a linear learning machine based on this
kernel will always ﬁnd a solution proportional to f(x).

Exercises 321
6.11 (⋆)By making use of the expansion (6.25), and then expanding the middle factor
as a power series, show that the Gaussian kernel (6.23) can be expressed as the innerproduct of an inﬁnite-dimensional feature vector.
6.12 (⋆⋆)
www Consider the space of all possible subsets Aof a given ﬁxed set D.
Show that the kernel function (6.27) corresponds to an inner product in a feature
space of dimensionality 2|D|deﬁned by the mapping φ(A)where Ais a subset of D
and the element φU(A), indexed by the subset U, is given by
φU(A)={
1,ifU⊆A;
0,otherwise.(6.95)
HereU⊆Adenotes that Uis either a subset of Aor is equal to A.
6.13 (⋆)Show that the Fisher kernel, deﬁned by (6.33), remains invariant if we make
a nonlinear transformation of the parameter vector θ→ψ(θ), where the function
ψ(·)is invertible and differentiable.
6.14 (⋆)www Write down the form of the Fisher kernel, deﬁned by (6.33), for the
case of a distribution p(x|µ)=N(x|µ,S)that is Gaussian with mean µand ﬁxed
covariance S.

φU(A)={
1,ifU⊆A;
6.15 (⋆)By considering the determinant of a 2×2Gram matrix, show that a positive-
deﬁnite kernel function k(x, x′)satisﬁes the Cauchy-Schwartz inequality
k(x1,x2)2⩽k(x1,x1)k(x2,x2). (6.96)
6.16 (⋆⋆)Consider a parametric model governed by the parameter vector wtogether
with a data set of input values x1,...,xNand a nonlinear feature mapping φ(x).
Suppose that the dependence of the error function on wtakes the form
J(w)=f(wTφ(x1),...,wTφ(xN)) +g(wTw) (6.97)
where g(·)is a monotonically increasing function. By writing win the form
w=N∑

w=N∑
n=1αnφ(xn)+w⊥ (6.98)
show that the value of wthat minimizes J(w)takes the form of a linear combination
of the basis functions φ(xn)forn=1,...,N .
6.17 (⋆⋆)www Consider the sum-of-squares error function (6.39) for data having
noisy inputs, where ν(ξ)is the distribution of the noise. Use the calculus of vari-
ations to minimize this error function with respect to the function y(x), and hence
show that the optimal solution is given by an expansion of the form (6.40) in which
the basis functions are given by (6.41).

322 6. KERNEL METHODS
6.18 (⋆)Consider a Nadaraya-Watson model with one input variable xand one target
variable thaving Gaussian components with isotropic covariances, so that the co-
variance matrix is given by σ2IwhereIis the unit matrix. Write down expressions
for the conditional density p(t|x)and for the conditional mean E[t|x]and variance
var[t|x], in terms of the kernel function k(x, xn).
6.19 (⋆⋆)Another viewpoint on kernel regression comes from a consideration of re-
gression problems in which the input variables as well as the target variables are
corrupted with additive noise. Suppose each target value tnis generated as usual
by taking a function y(zn)evaluated at a point zn, and adding Gaussian noise. The
value of znis not directly observed, however, but only a noise corrupted version
xn=zn+ξnwhere the random variable ξis governed by some distribution g(ξ).
Consider a set of observations {xn,tn}, where n=1,...,N , together with a cor-

responding sum-of-squares error function deﬁned by averaging over the distribution
of input noise to give
E=1
2N∑
n=1∫
{y(xn−ξn)−tn}2g(ξn)dξn. (6.99)
By minimizing Ewith respect to the function y(z)using the calculus of variations
(Appendix D), show that optimal solution for y(x)is given by a Nadaraya-Watson
kernel regression solution of the form (6.45) with a kernel of the form (6.46).
6.20 (⋆⋆)www Verify the results (6.66) and (6.67).
6.21 (⋆⋆)www Consider a Gaussian process regression model in which the kernel

E=1
2N∑
n=1∫
function is deﬁned in terms of a ﬁxed set of nonlinear basis functions. Show that thepredictive distribution is identical to the result (3.58) obtained in Section 3.3.2 for the
Bayesian linear regression model. To do this, note that both models have Gaussian
predictive distributions, and so it is only necessary to show that the conditional meanand variance are the same. For the mean, make use of the matrix identity (C.6), and
for the variance, make use of the matrix identity (C.7).
6.22 (⋆⋆)Consider a regression problem with Ntraining set input vectors x
1,...,xN

1,...,xN
andLtest set input vectors xN+1,...,xN+L, and suppose we deﬁne a Gaussian
process prior over functions t(x). Derive an expression for the joint predictive dis-
tribution for t(xN+1),...,t (xN+L), given the values of t(x1),...,t (xN). Show the
marginal of this distribution for one of the test observations tjwhere N+1⩽j⩽
N+Lis given by the usual Gaussian process regression result (6.66) and (6.67).
6.23 (⋆⋆)www Consider a Gaussian process regression model in which the target
variable thas dimensionality D. Write down the conditional distribution of tN+1

for a test input vector xN+1, given a training set of input vectors x1,...,xN+1and
corresponding target observations t1,...,tN.
6.24 (⋆)Show that a diagonal matrix Wwhose elements satisfy 0<W ii<1is positive
deﬁnite. Show that the sum of two positive deﬁnite matrices is itself positive deﬁnite.

Exercises 323
6.25 (⋆)www Using the Newton-Raphson formula (4.92), derive the iterative update
formula (6.83) for ﬁnding the mode a⋆
Nof the posterior distribution in the Gaussian
process classiﬁcation model.
6.26 (⋆)Using the result (2.115), derive the expressions (6.87) and (6.88) for the mean
and variance of the posterior distribution p(aN+1|tN)in the Gaussian process clas-
siﬁcation model.
6.27 (⋆⋆⋆ )Derive the result (6.90) for the log likelihood function in the Laplace approx-
imation framework for Gaussian process classiﬁcation. Similarly, derive the results
(6.91), (6.92), and (6.94) for the terms in the gradient of the log likelihood.

7
Sparse Kernel
Machines
In the previous chapter, we explored a variety of learning algorithms based on non-
linear kernels. One of the signiﬁcant limitations of many such algorithms is thatthe kernel function k(x
n,xm)must be evaluated for all possible pairs xnandxm
of training points, which can be computationally infeasible during training and can
lead to excessive computation times when making predictions for new data points.In this chapter we shall look at kernel-based algorithms that have sparse solutions,
so that predictions for new inputs depend only on the kernel function evaluated at a
subset of the training data points.
We begin by looking in some detail at the support vector machine (SVM), which
became popular in some years ago for solving problems in classiﬁcation, regression,
and novelty detection. An important property of support vector machines is that thedetermination of the model parameters corresponds to a convex optimization prob-

lem, and so any local solution is also a global optimum. Because the discussion of
support vector machines makes extensive use of Lagrange multipliers, the reader is
325

326 7. SPARSE KERNEL MACHINES
encouraged to review the key concepts covered in Appendix E. Additional infor-
mation on support vector machines can be found in Vapnik (1995), Burges (1998),Cristianini and Shawe-Taylor (2000), M ¨uller et al. (2001), Sch ¨olkopf and Smola
(2002), and Herbrich (2002).
The SVM is a decision machine and so does not provide posterior probabilities.
We have already discussed some of the beneﬁts of determining probabilities in Sec-
tion 1.5.4. An alternative sparse kernel technique, known as the relevance vector
machine (RVM), is based on a Bayesian formulation and provides posterior proba- Section 7.2
bilistic outputs, as well as having typically much sparser solutions than the SVM.
7.1. Maximum Margin Classiﬁers
We begin our discussion of support vector machines by returning to the two-classclassiﬁcation problem using linear models of the form
y(x)=w
Tφ(x)+b (7.1)
where φ(x)denotes a ﬁxed feature-space transformation, and we have made the

y(x)=w
bias parameter bexplicit. Note that we shall shortly introduce a dual representation
expressed in terms of kernel functions, which avoids having to work explicitly infeature space. The training data set comprises Ninput vectors x
1,...,xN, with
corresponding target values t1,...,t Nwhere tn∈{ −1,1}, and new data points x
are classiﬁed according to the sign of y(x).
We shall assume for the moment that the training data set is linearly separable in
feature space, so that by deﬁnition there exists at least one choice of the parameters

wandbsuch that a function of the form (7.1) satisﬁes y(xn)>0for points having
tn=+ 1 andy(xn)<0for points having tn=−1, so that tny(xn)>0for all
training data points.
There may of course exist many such solutions that separate the classes exactly.
In Section 4.1.7, we described the perceptron algorithm that is guaranteed to ﬁnd
a solution in a ﬁnite number of steps. The solution that it ﬁnds, however, will be
dependent on the (arbitrary) initial values chosen for wandbas well as on the

order in which the data points are presented. If there are multiple solutions all of
which classify the training data set exactly, then we should try to ﬁnd the one that
will give the smallest generalization error. The support vector machine approachesthis problem through the concept of the margin , which is deﬁned to be the smallest
distance between the decision boundary and any of the samples, as illustrated in
Figure 7.1.
In support vector machines the decision boundary is chosen to be the one for

which the margin is maximized. The maximum margin solution can be motivated us-ingcomputational learning theory , also known as statistical learning theory .H o w - Section 7.1.5
ever, a simple insight into the origins of maximum margin has been given by Tong
and Koller (2000) who consider a framework for classiﬁcation based on a hybrid ofgenerative and discriminative approaches. They ﬁrst model the distribution over in-
put vectors xfor each class using a Parzen density estimator with Gaussian kernels

7.1. Maximum Margin Classiﬁers 327
y=1
y=0
y=−1
marginy=1y=0y=−1
Figure 7.1 The margin is deﬁned as the perpendicular distance between the decision boundary and the closest
of the data points, as shown on the left ﬁgure. Maximizing the margin leads to a particular choice of decision
boundary, as shown on the right. The location of this boundary is determined by a subset of the data points,
known as support vectors, which are indicated by the circles.
having a common parameter σ2. Together with the class priors, this deﬁnes an opti-
mal misclassiﬁcation-rate decision boundary. However, instead of using this optimal
boundary, they determine the best hyperplane by minimizing the probability of error
relative to the learned density model. In the limit σ2→0, the optimal hyperplane
is shown to be the one having maximum margin. The intuition behind this result is
that as σ2is reduced, the hyperplane is increasingly dominated by nearby data points

relative to more distant ones. In the limit, the hyperplane becomes independent of
data points that are not support vectors.
We shall see in Figure 10.13 that marginalization with respect to the prior distri-
bution of the parameters in a Bayesian approach for a simple linearly separable data
set leads to a decision boundary that lies in the middle of the region separating the
data points. The large margin solution has similar behaviour.
Recall from Figure 4.1 that the perpendicular distance of a point xfrom a hyper-

plane deﬁned by y(x)=0 where y(x)takes the form (7.1) is given by |y(x)|/∥w∥.
Furthermore, we are only interested in solutions for which all data points are cor-
rectly classiﬁed, so that tny(xn)>0for all n. Thus the distance of a point xnto the
decision surface is given by
tny(xn)
∥w∥=tn(wTφ(xn)+b)
∥w∥. (7.2)
The margin is given by the perpendicular distance to the closest point xnfrom the
data set, and we wish to optimize the parameters wandbin order to maximize this
distance. Thus the maximum margin solution is found by solving
arg max
w,b{1
∥w∥min
n[
tn(
wTφ(xn)+b)]}
(7.3)

tny(xn)
∥w∥. (7.2)
arg max
w,b{1
∥w∥min
n[
tn(
(7.3)
where we have taken the factor 1/∥w∥outside the optimization over nbecause w

328 7. SPARSE KERNEL MACHINES
does not depend on n. Direct solution of this optimization problem would be very
complex, and so we shall convert it into an equivalent problem that is much easierto solve. To do this we note that if we make the rescaling w→κwandb→κb,
then the distance from any point x
nto the decision surface, given by tny(xn)/∥w∥,
is unchanged. We can use this freedom to set
tn(
wTφ(xn)+b)
=1 (7.4)
for the point that is closest to the surface. In this case, all data points will satisfy the
constraints
tn(
⩾1,n =1,...,N. (7.5)
This is known as the canonical representation of the decision hyperplane. In the
case of data points for which the equality holds, the constraints are said to be active ,
whereas for the remainder they are said to be inactive . By deﬁnition, there will
always be at least one active constraint, because there will always be a closest point,
and once the margin has been maximized there will be at least two active constraints.

tn(
The optimization problem then simply requires that we maximize ∥w∥−1, which is
equivalent to minimizing ∥w∥2, and so we have to solve the optimization problem
arg min
w,b1
2∥w∥2(7.6)
subject to the constraints given by (7.5). The factor of 1/2in (7.6) is included for
later convenience. This is an example of a quadratic programming problem in which
we are trying to minimize a quadratic function subject to a set of linear inequalityconstraints. It appears that the bias parameter bhas disappeared from the optimiza-

arg min
w,b1
tion. However, it is determined implicitly via the constraints, because these require
that changes to ∥w∥be compensated by changes to b. We shall see how this works
shortly.
In order to solve this constrained optimization problem, we introduce Lagrange
multipliers a
n⩾0, with one multiplier anfor each of the constraints in (7.5), giving Appendix E
the Lagrangian function
L(w,b ,a)=1
2∥w∥2−N∑
n=1an{
tn(wTφ(xn)+b)−1}
(7.7)
wherea=(a1,...,a N)T. Note the minus sign in front of the Lagrange multiplier
term, because we are minimizing with respect to wandb, and maximizing with

shortly.
2∥w∥2−N∑
n=1an{
(7.7)
respect to a. Setting the derivatives of L(w,b ,a)with respect to wandbequal to
zero, we obtain the following two conditions
w=N∑
n=1antnφ(xn) (7.8)
0=N∑
n=1antn. (7.9)

7.1. Maximum Margin Classiﬁers 329
Eliminating wandbfromL(w,b ,a)using these conditions then gives the dual
representation of the maximum margin problem in which we maximize
˜L(a)=N∑
n=1an−1
2N∑
n=1N∑
m=1anamtntmk(xn,xm) (7.10)
with respect to asubject to the constraints
an⩾0,n =1,...,N, (7.11)
N∑
n=1antn=0. (7.12)
Here the kernel function is deﬁned by k(x,x′)=φ(x)Tφ(x′). Again, this takes the
form of a quadratic programming problem in which we optimize a quadratic function
ofasubject to a set of inequality constraints. We shall discuss techniques for solving
such quadratic programming problems in Section 7.1.1.
The solution to a quadratic programming problem in Mvariables in general has
computational complexity that is O(M3). In going to the dual formulation we have
turned the original optimization problem, which involved minimizing (7.6) over M
variables, into the dual problem (7.10), which has Nvariables. For a ﬁxed set of
basis functions whose number Mis smaller than the number Nof data points, the

move to the dual problem appears disadvantageous. However, it allows the model tobe reformulated using kernels, and so the maximum margin classiﬁer can be applied
efﬁciently to feature spaces whose dimensionality exceeds the number of data points,
including inﬁnite feature spaces. The kernel formulation also makes clear the roleof the constraint that the kernel function k(x,x
′)be positive deﬁnite, because this
ensures that the Lagrangian function ˜L(a)is bounded below, giving rise to a well-
deﬁned optimization problem.

In order to classify new data points using the trained model, we evaluate the sign
ofy(x)deﬁned by (7.1). This can be expressed in terms of the parameters {an}and
the kernel function by substituting for wusing (7.8) to give
y(x)=N∑
n=1antnk(x,xn)+b. (7.13)
Joseph-Louis Lagrange
1736–1813
Although widely considered to be
a French mathematician, Lagrangewas born in Turin in Italy. By the ageof nineteen, he had already madeimportant contributions mathemat-ics and had been appointed as Pro-
fessor at the Royal Artillery School in Turin. For manyyears, Euler worked hard to persuade Lagrange to

y(x)=N∑
1736–1813
move to Berlin, which he eventually did in 1766 wherehe succeeded Euler as Director of Mathematics atthe Berlin Academy. Later he moved to Paris, nar-rowly escaping with his life during the French revo-lution thanks to the personal intervention of Lavoisier(the French chemist who discovered oxygen) who him-self was later executed at the guillotine. Lagrangemade key contributions to the calculus of variationsand the foundations of dynamics.

330 7. SPARSE KERNEL MACHINES
In Appendix E, we show that a constrained optimization of this form satisﬁes the
Karush-Kuhn-Tucker (KKT) conditions, which in this case require that the following
three properties hold
an⩾0 (7.14)
tny(xn)−1⩾0 (7.15)
an{tny(xn)−1}=0. (7.16)
Thus for every data point, either an=0 ortny(xn)=1 . Any data point for
which an=0will not appear in the sum in (7.13) and hence plays no role in making
predictions for new data points. The remaining data points are called support vectors ,
and because they satisfy tny(xn)=1 , they correspond to points that lie on the
maximum margin hyperplanes in feature space, as illustrated in Figure 7.1. This
property is central to the practical applicability of support vector machines. Once
the model is trained, a signiﬁcant proportion of the data points can be discarded andonly the support vectors retained.
Having solved the quadratic programming problem and found a value for a,w e

can then determine the value of the threshold parameter bby noting that any support
vectorx
nsatisﬁes tny(xn)=1 . Using (7.13) this gives
tn(∑
m∈Samtmk(xn,xm)+b)
=1 (7.17)
where Sdenotes the set of indices of the support vectors. Although we can solve
this equation for busing an arbitrarily chosen support vector xn, a numerically more
stable solution is obtained by ﬁrst multiplying through by tn, making use of t2
n=1,
and then averaging these equations over all support vectors and solving for bto give
b=1
NS∑
n∈S(
tn−∑
m∈Samtmk(xn,xm))
(7.18)

vectorx
tn(∑
=1 (7.17)
n=1,
b=1
NS∑
n∈S(
tn−∑
(7.18)
where NSis the total number of support vectors.
For later comparison with alternative models, we can express the maximum-
margin classiﬁer in terms of the minimization of an error function, with a simple
quadratic regularizer, in the form
N∑
n=1E∞(y(xn)tn−1) +λ∥w∥2(7.19)
where E∞(z)is a function that is zero if z⩾0and∞otherwise and ensures that
the constraints (7.5) are satisﬁed. Note that as long as the regularization parameter
satisﬁes λ>0, its precise value plays no role.

tn−∑
(7.18)
N∑
Figure 7.2 shows an example of the classiﬁcation resulting from training a sup-
port vector machine on a simple synthetic data set using a Gaussian kernel of the

7.1. Maximum Margin Classiﬁers 331
Figure 7.2 Example of synthetic data from
two classes in two dimensions
showing contours of constant
y(x)obtained from a support
vector machine having a Gaus-sian kernel function. Also shownare the decision boundary, themargin boundaries, and the sup-port vectors.
form (6.23). Although the data set is not linearly separable in the two-dimensional
data space x, it is linearly separable in the nonlinear feature space deﬁned implicitly
by the nonlinear kernel function. Thus the training data points are perfectly separated
in the original data space.
This example also provides a geometrical insight into the origin of sparsity in
the SVM. The maximum margin hyperplane is deﬁned by the location of the supportvectors. Other data points can be moved around freely (so long as they remain out-
side the margin region) without changing the decision boundary, and so the solution
will be independent of such data points.
7.1.1 Overlapping class distributions

So far, we have assumed that the training data points are linearly separable in the
feature space φ(x). The resulting support vector machine will give exact separation
of the training data in the original input space x, although the corresponding decision
boundary will be nonlinear. In practice, however, the class-conditional distributions
may overlap, in which case exact separation of the training data can lead to poor
We therefore need a way to modify the support vector machine so as to allow

some of the training points to be misclassiﬁed. From (7.19) we see that in the case
of separable classes, we implicitly used an error function that gave inﬁnite error
if a data point was misclassiﬁed and zero error if it was classiﬁed correctly, andthen optimized the model parameters to maximize the margin. We now modify this
approach so that data points are allowed to be on the ‘wrong side’ of the margin
boundary, but with a penalty that increases with the distance from that boundary. Forthe subsequent optimization problem, it is convenient to make this penalty a linear

function of this distance. To do this, we introduce slack variables ,ξ
n⩾0where
n=1,...,N , with one slack variable for each training data point (Bennett, 1992;
Cortes and Vapnik, 1995). These are deﬁned by ξn=0for data points that are on or
inside the correct margin boundary and ξn=|tn−y(xn)|for other points. Thus a
data point that is on the decision boundary y(xn)=0 will have ξn=1, and points

332 7. SPARSE KERNEL MACHINES
Figure 7.3 Illustration of the slack variables ξn⩾0.
Data points with circles around them are
support vectors.
y=1y=0y=−1
ξ> 1
ξ< 1
ξ=0ξ=0
withξn>1will be misclassiﬁed. The exact classiﬁcation constraints (7.5) are then
replaced with
tny(xn)⩾1−ξn,n =1,...,N (7.20)
in which the slack variables are constrained to satisfy ξn⩾0. Data points for which
ξn=0 are correctly classiﬁed and are either on the margin or on the correct side
of the margin. Points for which 0<ξn⩽1lie inside the margin, but on the cor-
rect side of the decision boundary, and those data points for which ξn>1lie on
the wrong side of the decision boundary and are misclassiﬁed, as illustrated in Fig-
ure 7.3. This is sometimes described as relaxing the hard margin constraint to give a
soft margin and allows some of the training set data points to be misclassiﬁed. Note
that while slack variables allow for overlapping class distributions, this framework is

still sensitive to outliers because the penalty for misclassiﬁcation increases linearly
withξ.
Our goal is now to maximize the margin while softly penalizing points that lie
on the wrong side of the margin boundary. We therefore minimize
CN∑
n=1ξn+1
2∥w∥2(7.21)
where the parameter C>0controls the trade-off between the slack variable penalty
and the margin. Because any point that is misclassiﬁed has ξn>1, it follows that∑
nξnis an upper bound on the number of misclassiﬁed points. The parameter Cis

withξ.
CN∑
n=1ξn+1
therefore analogous to (the inverse of) a regularization coefﬁcient because it controls
the trade-off between minimizing training errors and controlling model complexity.
In the limit C→∞ , we will recover the earlier support vector machine for separable
data.
We now wish to minimize (7.21) subject to the constraints (7.20) together with
ξn⩾0. The corresponding Lagrangian is given by
2∥w∥2+CN∑
n=1ξn−N∑
n=1an{tny(xn)−1+ξn}−N∑
n=1µnξn(7.22)

7.1. Maximum Margin Classiﬁers 333
where {an⩾0}and{µn⩾0}are Lagrange multipliers. The corresponding set of
KKT conditions are given by Appendix E
an⩾0 (7.23)
tny(xn)−1+ξn⩾0 (7.24)
an(tny(xn)−1+ξn)=0 (7.25)
µn⩾0 (7.26)
ξn⩾0 (7.27)
µnξn=0 (7.28)
where n=1,...,N .
We now optimize out w,b, and{ξn}making use of the deﬁnition (7.1) of y(x)
to give
∂L
∂w=0⇒w=N∑
n=1antnφ(xn) (7.29)
∂L
∂b=0⇒N∑
n=1antn=0 (7.30)
∂L
∂ξn=0⇒an=C−µn. (7.31)
Using these results to eliminate w,b, and{ξn}from the Lagrangian, we obtain the
dual Lagrangian in the form
˜L(a)=N∑
n=1an−1
2N∑
n=1N∑
m=1anamtntmk(xn,xm) (7.32)
which is identical to the separable case, except that the constraints are somewhat
different. To see what these constraints are, we note that an⩾0is required because
these are Lagrange multipliers. Furthermore, (7.31) together with µn⩾0implies
an⩽C. We therefore have to minimize (7.32) with respect to the dual variables
{an}subject to
0⩽an⩽C (7.33)
N∑
n=1antn=0 (7.34)

˜L(a)=N∑
n=1an−1
2N∑
n=1N∑
N∑
forn=1,...,N , where (7.33) are known as box constraints . This again represents
a quadratic programming problem. If we substitute (7.29) into (7.1), we see that
predictions for new data points are again made by using (7.13).
We can now interpret the resulting solution. As before, a subset of the data
points may have an=0, in which case they do not contribute to the predictive

334 7. SPARSE KERNEL MACHINES
model (7.13). The remaining data points constitute the support vectors. These have
an>0and hence from (7.25) must satisfy
tny(xn)=1−ξn. (7.35)
Ifan<C, then (7.31) implies that µn>0, which from (7.28) requires ξn=0and
hence such points lie on the margin. Points with an=Ccan lie inside the margin
and can either be correctly classiﬁed if ξn⩽1or misclassiﬁed if ξn>1.
To determine the parameter bin (7.1), we note that those support vectors for
which 0<an<C haveξn=0so that tny(xn)=1 and hence will satisfy
tn(∑
=1. (7.36)
Again, a numerically stable solution is obtained by averaging to give
b=1
NM∑
n∈M(
tn−∑
(7.37)
where Mdenotes the set of indices of data points having 0<an<C.
An alternative, equivalent formulation of the support vector machine, known as
theν-SVM , has been proposed by Sch ¨olkopf et al. (2000). This involves maximizing
˜L(a)=−1
2N∑
n=1N∑
m=1anamtntmk(xn,xm) (7.38)
subject to the constraints
0⩽an⩽1/N (7.39)
N∑
n=1antn=0 (7.40)
N∑

tn(∑
=1. (7.36)
b=1
NM∑
n∈M(
tn−∑
(7.37)
˜L(a)=−1
2N∑
n=1N∑
N∑
N∑
n=1an⩾ν. (7.41)
This approach has the advantage that the parameter ν, which replaces C, can be
interpreted as both an upper bound on the fraction of margin errors (points for which
ξn>0and hence which lie on the wrong side of the margin boundary and which may
or may not be misclassiﬁed) and a lower bound on the fraction of support vectors. Anexample of the ν-SVM applied to a synthetic data set is shown in Figure 7.4. Here
Gaussian kernels of the form exp (−γ∥x−x
′∥2)have been used, with γ=0.45.

N∑
Although predictions for new inputs are made using only the support vectors,
the training phase (i.e., the determination of the parameters aandb) makes use of
the whole data set, and so it is important to have efﬁcient algorithms for solving

7.1. Maximum Margin Classiﬁers 335
Figure 7.4 Illustration of the ν-SVM applied
to a nonseparable data set in two
dimensions. The support vectors
are indicated by circles.
−2 0 2−202
the quadratic programming problem. We ﬁrst note that the objective function ˜L(a)
given by (7.10) or (7.32) is quadratic and so any local optimum will also be a global
optimum provided the constraints deﬁne a convex region (which they do as a conse-
quence of being linear). Direct solution of the quadratic programming problem us-
ing traditional techniques is often infeasible due to the demanding computation and
memory requirements, and so more practical approaches need to be found. The tech-
nique of chunking (Vapnik, 1982) exploits the fact that the value of the Lagrangian
is unchanged if we remove the rows and columns of the kernel matrix corresponding
to Lagrange multipliers that have value zero. This allows the full quadratic pro-
gramming problem to be broken down into a series of smaller ones, whose goal is

eventually to identify all of the nonzero Lagrange multipliers and discard the others.
Chunking can be implemented using protected conjugate gradients (Burges, 1998).
Although chunking reduces the size of the matrix in the quadratic function from the
number of data points squared to approximately the number of nonzero Lagrange
multipliers squared, even this may be too big to ﬁt in memory for large-scale appli-
cations. Decomposition methods (Osuna et al. , 1996) also solve a series of smaller

quadratic programming problems but are designed so that each of these is of a ﬁxed
size, and so the technique can be applied to arbitrarily large data sets. However, it
still involves numerical solution of quadratic programming subproblems and these
can be problematic and expensive. One of the most popular approaches to training
support vector machines is called sequential minimal optimization ,o r SMO (Platt,
1999). It takes the concept of chunking to the extreme limit and considers just two

Lagrange multipliers at a time. In this case, the subproblem can be solved analyti-
cally, thereby avoiding numerical quadratic programming altogether. Heuristics are
given for choosing the pair of Lagrange multipliers to be considered at each step.
In practice, SMO is found to have a scaling with the number of data points that is
somewhere between linear and quadratic depending on the particular application.
We have seen that kernel functions correspond to inner products in feature spaces

that can have high, or even inﬁnite, dimensionality. By working directly in terms of
the kernel function, without introducing the feature space explicitly, it might there-
fore seem that support vector machines somehow manage to avoid the curse of di-

336 7. SPARSE KERNEL MACHINES
mensionality. This is not the case, however, because there are constraints amongst Section 1.4
the feature values that restrict the effective dimensionality of feature space. To seethis consider a simple second-order polynomial kernel that we can expand in terms
of its components
k(x,z)=(
1+xTz)2=( 1+ x1z1+x2z2)2
=1 + 2 x1z1+2x2z2+x2
1z2
2z2
2
=( 1 ,√
2x1,√
2x2,x2
1,√
2x1x2,x2
2)(1,√
2z1,√
2z2,z2
1,√
2z1z2,z2
2)T
=φ(x)Tφ(z). (7.42)
This kernel function therefore represents an inner product in a feature space having
six dimensions, in which the mapping from input space to feature space is describedby the vector function φ(x). However, the coefﬁcients weighting these different
features are constrained to have speciﬁc forms. Thus any set of points in the original
two-dimensional space xwould be constrained to lie exactly on a two-dimensional
nonlinear manifold embedded in the six-dimensional feature space.

2)T
We have already highlighted the fact that the support vector machine does not
provide probabilistic outputs but instead makes classiﬁcation decisions for new in-
put vectors. Veropoulos et al. (1999) discuss modiﬁcations to the SVM to allow
the trade-off between false positive and false negative errors to be controlled. How-ever, if we wish to use the SVM as a module in a larger probabilistic system, then
probabilistic predictions of the class label tfor new inputs xare required.

To address this issue, Platt (2000) has proposed ﬁtting a logistic sigmoid to the
outputs of a previously trained support vector machine. Speciﬁcally, the required
conditional probability is assumed to be of the form
p(t=1|x)=σ(Ay(x)+B) (7.43)
where y(x)is deﬁned by (7.1). Values for the parameters AandBare found by
minimizing the cross-entropy error function deﬁned by a training set consisting of
pairs of values y(x
n)andtn. The data used to ﬁt the sigmoid needs to be independent

of that used to train the original SVM in order to avoid severe over-ﬁtting. This two-
stage approach is equivalent to assuming that the output y(x)of the support vector
machine represents the log-odds of xbelonging to class t=1. Because the SVM
training procedure is not speciﬁcally intended to encourage this, the SVM can give
a poor approximation to the posterior probabilities (Tipping, 2001).
7.1.2 Relation to logistic regression
As with the separable case, we can re-cast the SVM for nonseparable distri-

butions in terms of the minimization of a regularized error function. This will alsoallow us to highlight similarities, and differences, compared to the logistic regression
model. Section 4.3.2
We have seen that for data points that are on the correct side of the margin
boundary, and which therefore satisfy y
ntn⩾1,w eh a v e ξn=0, and for the

7.1. Maximum Margin Classiﬁers 337
Figure 7.5 Plot of the ‘hinge’ error function used
in support vector machines, shown
in blue, along with the error function
for logistic regression, rescaled by a
factor of 1/ln(2) so that it passes
through the point (0,1), shown in red.
Also shown are the misclassiﬁcation
error in black and the squared error
in green.
−2 −1 012zE(z)
remaining points we have ξn=1−yntn. Thus the objective function (7.21) can be
written (up to an overall multiplicative constant) in the form
N∑
n=1ESV(yntn)+λ∥w∥2(7.44)
where λ=( 2C)−1, andESV(·)is the hinge error function deﬁned by
ESV(yntn)=[ 1 −yntn]+ (7.45)
where [·]+denotes the positive part. The hinge error function, so-called because
of its shape, is plotted in Figure 7.5. It can be viewed as an approximation to the
misclassiﬁcation error, i.e., the error function that ideally we would like to minimize,
which is also shown in Figure 7.5.
When we considered the logistic regression model in Section 4.3.2, we found it

N∑
convenient to work with target variable t∈{0,1}. For comparison with the support
vector machine, we ﬁrst reformulate maximum likelihood logistic regression using
the target variable t∈{ −1,1}. To do this, we note that p(t=1|y)=σ(y)where
y(x)is given by (7.1), and σ(y)is the logistic sigmoid function deﬁned by (4.59). It
follows that p(t=−1|y)=1−σ(y)=σ(−y), where we have used the properties
of the logistic sigmoid function, and so we can write
p(t|y)=σ(yt). (7.46)

From this we can construct an error function by taking the negative logarithm of the
likelihood function that, with a quadratic regularizer, takes the form Exercise 7.6
N∑
n=1ELR(yntn)+λ∥w∥2. (7.47)
where
ELR(yt)=l n( 1+e x p ( −yt)). (7.48)

338 7. SPARSE KERNEL MACHINES
For comparison with other error functions, we can divide by ln(2) so that the error
function passes through the point (0,1). This rescaled error function is also plotted
in Figure 7.5 and we see that it has a similar form to the support vector error function.
The key difference is that the ﬂat region in ESV(yt)leads to sparse solutions.
Both the logistic error and the hinge loss can be viewed as continuous approx-
imations to the misclassiﬁcation error. Another continuous error function that has
sometimes been used to solve classiﬁcation problems is the squared error, which
is again plotted in Figure 7.5. It has the property, however, of placing increasingemphasis on data points that are correctly classiﬁed but that are a long way from
the decision boundary on the correct side. Such points will be strongly weighted at
the expense of misclassiﬁed points, and so if the objective is to minimize the mis-

classiﬁcation rate, then a monotonically decreasing error function would be a better
choice.
7.1.3 Multiclass SVMs
The support vector machine is fundamentally a two-class classiﬁer. In practice,
however, we often have to tackle problems involving K>2classes. Various meth-
ods have therefore been proposed for combining multiple two-class SVMs in order
to build a multiclass classiﬁer.
One commonly used approach (Vapnik, 1998) is to construct Kseparate SVMs,

choice.
in which the kthmodel yk(x)is trained using the data from class Ckas the positive
examples and the data from the remaining K−1classes as the negative examples.
This is known as the one-versus-the-rest approach. However, in Figure 4.2 we saw
that using the decisions of the individual classiﬁers can lead to inconsistent results
in which an input is assigned to multiple classes simultaneously. This problem issometimes addressed by making predictions for new inputs xusing
y(x)=m a x
kyk(x). (7.49)

y(x)=m a x
Unfortunately, this heuristic approach suffers from the problem that the different
classiﬁers were trained on different tasks, and there is no guarantee that the real-
valued quantities yk(x)for different classiﬁers will have appropriate scales.
Another problem with the one-versus-the-rest approach is that the training sets
are imbalanced. For instance, if we have ten classes each with equal numbers of

y(x)=m a x
training data points, then the individual classiﬁers are trained on data sets comprising90% negative examples and only 10% positive examples, and the symmetry of the
original problem is lost. A variant of the one-versus-the-rest scheme was proposed
by Lee et al. (2001) who modify the target values so that the positive class has target
+1and the negative class has target −1/(K−1).
Weston and Watkins (1999) deﬁne a single objective function for training all
KSVMs simultaneously, based on maximizing the margin from each to remaining

classes. However, this can result in much slower training because, instead of solving
Kseparate optimization problems each over Ndata points with an overall cost of
O(KN
2), a single optimization problem of size (K−1)Nmust be solved giving an
overall cost of O(K2N2).

7.1. Maximum Margin Classiﬁers 339
Another approach is to train K(K−1)/2different 2-class SVMs on all possible
pairs of classes, and then to classify test points according to which class has the high-est number of ‘votes’, an approach that is sometimes called one-versus-one . Again,
we saw in Figure 4.2 that this can lead to ambiguities in the resulting classiﬁcation.
Also, for large Kthis approach requires signiﬁcantly more training time than the
one-versus-the-rest approach. Similarly, to evaluate test points, signiﬁcantly more
computation is required.
The latter problem can be alleviated by organizing the pairwise classiﬁers into
a directed acyclic graph (not to be confused with a probabilistic graphical model)
leading to the DAGSVM (Platt et al. , 2000). For Kclasses, the DAGSVM has a total
ofK(K−1)/2classiﬁers, and to classify a new test point only K−1pairwise
classiﬁers need to be evaluated, with the particular classiﬁers used depending on
which path through the graph is traversed.

A different approach to multiclass classiﬁcation, based on error-correcting out-
put codes, was developed by Dietterich and Bakiri (1995) and applied to support
vector machines by Allwein et al. (2000). This can be viewed as a generalization of
the voting scheme of the one-versus-one approach in which more general partitions
of the classes are used to train the individual classiﬁers. The Kclasses themselves

are represented as particular sets of responses from the two-class classiﬁers chosen,and together with a suitable decoding scheme, this gives robustness to errors and to
ambiguity in the outputs of the individual classiﬁers. Although the application of
SVMs to multiclass classiﬁcation problems remains an open issue, in practice theone-versus-the-rest approach is the most widely used in spite of its ad-hoc formula-
tion and its practical limitations.
There are also single-class support vector machines, which solve an unsuper-

vised learning problem related to probability density estimation. Instead of mod-
elling the density of data, however, these methods aim to ﬁnd a smooth boundary
enclosing a region of high density. The boundary is chosen to represent a quantile of
the density, that is, the probability that a data point drawn from the distribution will
land inside that region is given by a ﬁxed number between 0 and 1 that is speciﬁed inadvance. This is a more restricted problem than estimating the full density but may
be sufﬁcient in speciﬁc applications. Two approaches to this problem using support

vector machines have been proposed. The algorithm of Sch ¨olkopf et al. (2001) tries
to ﬁnd a hyperplane that separates all but a ﬁxed fraction νof the training data from
the origin while at the same time maximizing the distance (margin) of the hyperplane
from the origin, while Tax and Duin (1999) look for the smallest sphere in featurespace that contains all but a fraction νof the data points. For kernels k(x,x
′)that
are functions only of x−x′, the two algorithms are equivalent.
7.1.4 SVMs for regression

′)that
We now extend support vector machines to regression problems while at the
same time preserving the property of sparseness. In simple linear regression, we Section 3.1.4

340 7. SPARSE KERNEL MACHINES
Figure 7.6 Plot of an ϵ-insensitive error function (in
red) in which the error increases lin-
early with distance beyond the insen-
sitive region. Also shown for compar-
ison is the quadratic error function (in
green).
0 zE(z)
−ϵ ϵ
minimize a regularized error function given by
1
2N∑
n=1{yn−tn}2+λ
2∥w∥2. (7.50)
To obtain sparse solutions, the quadratic error function is replaced by an ϵ-insensitive
error function (Vapnik, 1995), which gives zero error if the absolute difference be-
tween the prediction y(x)and the target tis less than ϵwhere ϵ>0. A simple
example of an ϵ-insensitive error function, having a linear cost associated with errors
outside the insensitive region, is given by
Eϵ(y(x)−t)={
0, if|y(x)−t|<ϵ;
|y(x)−t|−ϵ,otherwise(7.51)
and is illustrated in Figure 7.6.
We therefore minimize a regularized error function given by
CN∑
n=1Eϵ(y(xn)−tn)+1
2∥w∥2(7.52)
where y(x)is given by (7.1). By convention the (inverse) regularization parameter,

CN∑
denoted C, appears in front of the error term.
As before, we can re-express the optimization problem by introducing slack
variables. For each data point xn, we now need two slack variables ξn⩾0and
ˆξn⩾0, where ξn>0corresponds to a point for which tn>y(xn)+ϵ, andˆξn>0
corresponds to a point for which tn<y(xn)−ϵ, as illustrated in Figure 7.7.
The condition for a target point to lie inside the ϵ-tube is that yn−ϵ⩽tn⩽
yn+ϵ, where yn=y(xn). Introducing the slack variables allows points to lie outside

the tube provided the slack variables are nonzero, and the corresponding conditions
are
tn⩽y(xn)+ϵ+ξn (7.53)
tn⩾y(xn)−ϵ−ˆξn. (7.54)

7.1. Maximum Margin Classiﬁers 341
Figure 7.7 Illustration of SVM regression, showing
the regression curve together with the ϵ-
insensitive ‘tube’. Also shown are exam-
ples of the slack variables ξand bξ. Points
above the ϵ-tube have ξ>0and bξ=0,
points below the ϵ-tube have ξ=0 and
bξ>0, and points inside the ϵ-tube have
ξ=bξ=0.yy+ϵ
y−ϵy(x)
xˆξ>0ξ>0
The error function for support vector regression can then be written as
CN∑
n=1(ξn+ˆξn)+1
2∥w∥2(7.55)
which must be minimized subject to the constraints ξn⩾0andˆξn⩾0as well as
(7.53) and (7.54). This can be achieved by introducing Lagrange multipliers an⩾0,
ˆan⩾0,µn⩾0, andˆµn⩾0and optimizing the Lagrangian
L=CN∑
2∥w∥2−N∑
n=1(µnξn+ˆµnˆξn)
−N∑
n=1an(ϵ+ξn+yn−tn)−N∑
n=1ˆan(ϵ+ˆξn−yn+tn).(7.56)
We now substitute for y(x)using (7.1) and then set the derivatives of the La-
grangian with respect to w,b,ξn, andˆξnto zero, giving
∂L
∂w=0⇒w=N∑
n=1(an−ˆan)φ(xn) (7.57)
∂L
∂b=0⇒N∑
n=1(an−ˆan)=0 (7.58)
∂L
∂ξn=0⇒an+µn=C (7.59)
∂L
∂ˆξn=0⇒ˆan+ˆµn=C. (7.60)

L=CN∑
2∥w∥2−N∑
−N∑
∂L
∂w=0⇒w=N∑
∂L
∂b=0⇒N∑
∂L
∂L
Using these results to eliminate the corresponding variables from the Lagrangian, we
see that the dual problem involves maximizing Exercise 7.7

342 7. SPARSE KERNEL MACHINES
˜L(a,ˆa)= −1
2N∑
n=1N∑
m=1(an−ˆan)(am−ˆam)k(xn,xm)
−ϵN∑
n=1(an+ˆan)+N∑
n=1(an−ˆan)tn (7.61)
with respect to {an}and{ˆan}, where we have introduced the kernel k(x,x′)=
φ(x)Tφ(x′). Again, this is a constrained maximization, and to ﬁnd the constraints
we note that an⩾0andˆan⩾0are both required because these are Lagrange
multipliers. Also µn⩾0andˆµn⩾0together with (7.59) and (7.60), require
an⩽Candˆan⩽C, and so again we have the box constraints
0⩽an⩽C (7.62)
0⩽ˆan⩽C (7.63)
together with the condition (7.58).
Substituting (7.57) into (7.1), we see that predictions for new inputs can be made
using
y(x)=N∑
n=1(an−ˆan)k(x,xn)+b (7.64)
which is again expressed in terms of the kernel function.
The corresponding Karush-Kuhn-Tucker (KKT) conditions, which state that at
the solution the product of the dual variables and the constraints must vanish, are
given by
an(ϵ+ξn+yn−tn)=0 (7.65)
ˆan(ϵ+ˆξn−yn+tn)=0 (7.66)
(C−an)ξn=0 (7.67)
(C−ˆan)ˆξn=0. (7.68)

using
y(x)=N∑
given by
From these we can obtain several useful results. First of all, we note that a coefﬁcient
ancan only be nonzero if ϵ+ξn+yn−tn=0, which implies that the data point
either lies on the upper boundary of the ϵ-tube ( ξn=0) or lies above the upper
boundary ( ξn>0). Similarly, a nonzero value for ˆanimplies ϵ+ˆξn−yn+tn=0,
and such points must lie either on or below the lower boundary of the ϵ-tube.
Furthermore, the two constraints ϵ+ξn+yn−tn=0andϵ+ˆξn−yn+tn=0

are incompatible, as is easily seen by adding them together and noting that ξnand
ˆξnare nonnegative while ϵis strictly positive, and so for every data point xn, either
anorˆan(or both) must be zero.
The support vectors are those data points that contribute to predictions given by
(7.64), in other words those for which either an̸=0orˆan̸=0. These are points that
lie on the boundary of the ϵ-tube or outside the tube. All points within the tube have

7.1. Maximum Margin Classiﬁers 343
an=ˆan=0. We again have a sparse solution, and the only terms that have to be
evaluated in the predictive model (7.64) are those that involve the support vectors.
The parameter bcan be found by considering a data point for which 0<an<
C, which from (7.67) must have ξn=0, and from (7.65) must therefore satisfy
ϵ+yn−tn=0. Using (7.1) and solving for b, we obtain
b=tn−ϵ−wTφ(xn)
=tn−ϵ−N∑
m=1(am−ˆam)k(xn,xm) (7.69)
where we have used (7.57). We can obtain an analogous result by considering a point
for which 0<ˆan<C. In practice, it is better to average over all such estimates of
b.
As with the classiﬁcation case, there is an alternative formulation of the SVM
for regression in which the parameter governing complexity has a more intuitiveinterpretation (Sch ¨olkopf et al. , 2000). In particular, instead of ﬁxing the width ϵof
the insensitive region, we ﬁx instead a parameter νthat bounds the fraction of points
lying outside the tube. This involves maximizing
2N∑
n=1N∑

b.
2N∑
n=1N∑
+N∑
n=1(an−ˆan)tn (7.70)
0⩽an⩽C/N (7.71)
0⩽ˆan⩽C/N (7.72)
N∑
n=1(an−ˆan)=0 (7.73)
N∑
n=1(an+ˆan)⩽νC. (7.74)
It can be shown that there are at most νNdata points falling outside the insensitive
tube, while at least νNdata points are support vectors and so lie either on the tube
or outside it.
The use of a support vector machine to solve a regression problem is illustrated
using the sinusoidal data set in Figure 7.8. Here the parameters νandChave been Appendix A

+N∑
N∑
N∑
chosen by hand. In practice, their values would typically be determined by cross-validation.

344 7. SPARSE KERNEL MACHINES
Figure 7.8 Illustration of the ν-SVM for re-
synthetic data set using Gaussian
kernels. The predicted regression
curve is shown by the red line, and
theϵ-insensitive tube corresponds
to the shaded region. Also, the
data points are shown in green,
and those with support vectors
are indicated by blue circles.
xt
0 1−101
7.1.5 Computational learning theory
Historically, support vector machines have largely been motivated and analysed
using a theoretical framework known as computational learning theory , also some-
times called statistical learning theory (Anthony and Biggs, 1992; Kearns and Vazi-
rani, 1994; Vapnik, 1995; Vapnik, 1998). This has its origins with Valiant (1984)
who formulated the probably approximately correct , or PAC, learning framework.
The goal of the PAC framework is to understand how large a data set needs to be in
order to give good generalization. It also gives bounds for the computational cost of

learning, although we do not consider these here.
Suppose that a data set Dof size Nis drawn from some joint distribution p(x,t)
wherexis the input variable and trepresents the class label, and that we restrict
attention to ‘noise free’ situations in which the class labels are determined by some
(unknown) deterministic function t=g(x). In PAC learning we say that a function
f(x;D), drawn from a space Fof such functions on the basis of the training set

D, has good generalization if its expected error rate is below some pre-speciﬁed
threshold ϵ, so that
Ex,t[I(f(x;D)̸=t)]<ϵ (7.75)
where I(·)is the indicator function, and the expectation is with respect to the dis-
tribution p(x,t). The quantity on the left-hand side is a random variable, because
it depends on the training set D, and the PAC framework requires that (7.75) holds,
with probability greater than 1−δ, for a data set Ddrawn randomly from p(x,t).
Hereδis another pre-speciﬁed parameter, and the terminology ‘probably approxi-

mately correct’ comes from the requirement that with high probability (greater than
1−δ), the error rate be small (less than ϵ). For a given choice of model space F, and
for given parameters ϵandδ, PAC learning aims to provide bounds on the minimum
sizeNof data set needed to meet this criterion. A key quantity in PAC learning is
theV apnik-Chervonenkis dimension , or VC dimension, which provides a measure of
the complexity of a space of functions, and which allows the PAC framework to be
extended to spaces containing an inﬁnite number of functions.

The bounds derived within the PAC framework are often described as worst-

7.2. Relevance Vector Machines 345
case, because they apply to anychoice for the distribution p(x,t), so long as both
the training and the test examples are drawn (independently) from the same distribu-tion, and for anychoice for the function f(x)so long as it belongs to F. In real-world
applications of machine learning, we deal with distributions that have signiﬁcant reg-
ularity, for example in which large regions of input space carry the same class label.As a consequence of the lack of any assumptions about the form of the distribution,
the PAC bounds are very conservative, in other words they strongly over-estimate
the size of data sets required to achieve a given generalization performance. For thisreason, PAC bounds have found few, if any, practical applications.
One attempt to improve the tightness of the PAC bounds is the PAC-Bayesian
framework (McAllester, 2003), which considers a distribution over the space Fof
functions, somewhat analogous to the prior in a Bayesian treatment. This still con-

siders any possible choice for p(x,t), and so although the bounds are tighter, they
are still very conservative.
7.2. Relevance Vector Machines
Support vector machines have been used in a variety of classiﬁcation and regres-sion applications. Nevertheless, they suffer from a number of limitations, severalof which have been highlighted already in this chapter. In particular, the outputs of
an SVM represent decisions rather than posterior probabilities. Also, the SVM was

originally formulated for two classes, and the extension to K>2classes is prob-
lematic. There is a complexity parameter C,o rν(as well as a parameter ϵin the case
of regression), that must be found using a hold-out method such as cross-validation.
Finally, predictions are expressed as linear combinations of kernel functions that are
centred on training data points and that are required to be positive deﬁnite.
The relevance vector machine or RVM (Tipping, 2001) is a Bayesian sparse ker-

nel technique for regression and classiﬁcation that shares many of the characteristics
of the SVM whilst avoiding its principal limitations. Additionally, it typically leads
to much sparser models resulting in correspondingly faster performance on test datawhilst maintaining comparable generalization error.
In contrast to the SVM we shall ﬁnd it more convenient to introduce the regres-
sion form of the RVM ﬁrst and then consider the extension to classiﬁcation tasks.
7.2.1 RVM for regression

The relevance vector machine for regression is a linear model of the form studied
in Chapter 3 but with a modiﬁed prior that results in sparse solutions. The model
deﬁnes a conditional distribution for a real-valued target variable t, given an input
vectorx, which takes the form
p(t|x,w,β)=N(t|y(x),β−1) (7.76)

346 7. SPARSE KERNEL MACHINES
where β=σ−2is the noise precision (inverse noise variance), and the mean is given
by a linear model of the form
y(x)=M∑
i=1wiφi(x)=wTφ(x) (7.77)
with ﬁxed nonlinear basis functions φi(x), which will typically include a constant
term so that the corresponding weight parameter represents a ‘bias’.
The relevance vector machine is a speciﬁc instance of this model, which is in-
tended to mirror the structure of the support vector machine. In particular, the basis
functions are given by kernels, with one kernel associated with each of the data
points from the training set. The general expression (7.77) then takes the SVM-likeform
y(x)=N∑
n=1wnk(x,xn)+b (7.78)
where bis a bias parameter. The number of parameters in this case is M=N+1,
andy(x)has the same form as the predictive model (7.64) for the SVM, except that
the coefﬁcients anare here denoted wn. It should be emphasized that the subsequent

y(x)=N∑
analysis is valid for arbitrary choices of basis function, and for generality we shallwork with the form (7.77). In contrast to the SVM, there is no restriction to positive-
deﬁnite kernels, nor are the basis functions tied in either number or location to the
Suppose we are given a set of Nobservations of the input vector x, which we
denote collectively by a data matrix Xwhose n
throw isxT
nwithn=1,...,N . The
corresponding target values are given by t=(t1,...,t N)T. Thus, the likelihood
function is given by
n=1p(tn|xn,w,β−1). (7.79)

throw isxT
Next we introduce a prior distribution over the parameter vector wand as in
Chapter 3, we shall consider a zero-mean Gaussian prior. However, the key differ-ence in the RVM is that we introduce a separate hyperparameter α
ifor each of the
weight parameters wiinstead of a single shared hyperparameter. Thus the weight
prior takes the form
p(w|α)=M∏
i=1N(wi|0,α−1
i) (7.80)
where αirepresents the precision of the corresponding parameter wi, andαdenotes
(α1,...,α M)T. We shall see that, when we maximize the evidence with respect
to these hyperparameters, a signiﬁcant proportion of them go to inﬁnity, and the

p(w|α)=M∏
i) (7.80)
corresponding weight parameters have posterior distributions that are concentrated
at zero. The basis functions associated with these parameters therefore play no role

7.2. Relevance Vector Machines 347
in the predictions made by the model and so are effectively pruned out, resulting in
a sparse model.
Using the result (3.49) for linear regression models, we see that the posterior
distribution for the weights is again Gaussian and takes the form
p(w|t,X,α,β)=N(w|m,Σ) (7.81)
where the mean and covariance are given by
m=βΣΦTt (7.82)
Σ=(
A+βΦTΦ)−1(7.83)
whereΦis the N×Mdesign matrix with elements Φni=φi(xn), andA=
diag(αi). Note that in the speciﬁc case of the model (7.78), we have Φ=K, where
Kis the symmetric (N+1 )×(N+1 ) kernel matrix with elements k(xn,xm).
The values of αandβare determined using type-2 maximum likelihood, also
known as the evidence approximation , in which we maximize the marginal likeli- Section 3.5
hood function obtained by integrating out the weight parameters
p(t|X,α,β)=∫
p(t|X,w,β)p(w|α)dw. (7.84)
Because this represents the convolution of two Gaussians, it is readily evaluated to Exercise 7.10
give the log marginal likelihood in the form

lnp(t|X,α,β)=l n N(t|0,C)
=−1
2{
Nln(2π)+l n|C|+tTC−1t}
(7.85)
where t=(t1,...,t N)T, and we have deﬁned the N×NmatrixCgiven by
C=β−1I+ΦA−1ΦT. (7.86)
Our goal is now to maximize (7.85) with respect to the hyperparameters αand
β. This requires only a small modiﬁcation to the results obtained in Section 3.5 for
the evidence approximation in the linear regression model. Again, we can identify
two approaches. In the ﬁrst, we simply set the required derivatives of the marginal

=−1
2{
(7.85)
likelihood to zero and obtain the following re-estimation equations Exercise 7.12
αnew
i=γi
m2
i(7.87)
(βnew)−1=∥t−Φm∥2
N−∑
iγi(7.88)
where miis the ithcomponent of the posterior mean mdeﬁned by (7.82). The
quantity γimeasures how well the corresponding parameter wiis determined by the
data and is deﬁned by Section 3.5.3

348 7. SPARSE KERNEL MACHINES
γi=1−αiΣii (7.89)
in which Σiiis theithdiagonal component of the posterior covariance Σgiven by
(7.83). Learning therefore proceeds by choosing initial values for αandβ, evalu-
ating the mean and covariance of the posterior using (7.82) and (7.83), respectively,and then alternately re-estimating the hyperparameters, using (7.87) and (7.88), and
re-estimating the posterior mean and covariance, using (7.82) and (7.83), until a suit-
able convergence criterion is satisﬁed.
The second approach is to use the EM algorithm, and is discussed in Sec-
tion 9.3.4. These two approaches to ﬁnding the values of the hyperparameters that
maximize the evidence are formally equivalent. Numerically, however, it is found Exercise 9.23
that the direct optimization approach corresponding to (7.87) and (7.88) gives some-
what faster convergence (Tipping, 2001).
As a result of the optimization, we ﬁnd that a proportion of the hyperparameters
{α

{α
i}are driven to large (in principle inﬁnite) values, and so the weight parameters Section 7.2.2
wicorresponding to these hyperparameters have posterior distributions with mean
and variance both zero. Thus those parameters, and the corresponding basis func-tionsφ
i(x), are removed from the model and play no role in making predictions for
new inputs. In the case of models of the form (7.78), the inputs xncorresponding to
the remaining nonzero weights are called relevance vectors , because they are iden-

{α
tiﬁed through the mechanism of automatic relevance determination, and are analo-
gous to the support vectors of an SVM. It is worth emphasizing, however, that thismechanism for achieving sparsity in probabilistic models through automatic rele-
vance determination is quite general and can be applied to any model expressed as
an adaptive linear combination of basis functions.
Having found values α
⋆andβ⋆for the hyperparameters that maximize the

marginal likelihood, we can evaluate the predictive distribution over tfor a new
inputx. Using (7.76) and (7.81), this is given by Exercise 7.14
p(t|x,X,t,α⋆,β⋆)=∫
p(t|x,w,β⋆)p(w|X,t,α⋆,β⋆)dw
=N(
t|mTφ(x),σ2(x))
. (7.90)
Thus the predictive mean is given by (7.76) with wset equal to the posterior mean
m, and the variance of the predictive distribution is given by
σ2(x)=(β⋆)−1+φ(x)TΣφ(x) (7.91)
whereΣis given by (7.83) in which αandβare set to their optimized values α⋆and
β⋆. This is just the familiar result (3.59) obtained in the context of linear regression.

=N(
. (7.90)
Recall that for localized basis functions, the predictive variance for linear regression
models becomes small in regions of input space where there are no basis functions.In the case of an RVM with the basis functions centred on data points, the model will
therefore become increasingly certain of its predictions when extrapolating outside
the domain of the data (Rasmussen and Qui ˜nonero-Candela, 2005), which of course
is undesirable. The predictive distribution in Gaussian process regression does not Section 6.4.2

7.2. Relevance Vector Machines 349
Figure 7.9 Illustration of RVM regression us-
ing the same data set, and the
same Gaussian kernel functions,
as used in Figure 7.8 for the
ν-SVM regression model. The
mean of the predictive distribu-
tion for the RVM is shown by the
red line, and the one standard-
deviation predictive distribution is
shown by the shaded region.
Also, the data points are shown
in green, and the relevance vec-
tors are indicated by blue circles.
Note that there are only 3 rele-
vance vectors compared to 7 sup-
port vectors for the ν-SVM in Fig-
ure 7.8.xt
0 1−101
suffer from this problem. However, the computational cost of making predictions
with a Gaussian processes is typically much higher than with an RVM.
Figure 7.9 shows an example of the RVM applied to the sinusoidal regression
data set. Here the noise precision parameter βis also determined through evidence
maximization. We see that the number of relevance vectors in the RVM is signif-

ure 7.8.xt
0 1−101
icantly smaller than the number of support vectors used by the SVM. For a wide
range of regression and classiﬁcation tasks, the RVM is found to give models that
are typically an order of magnitude more compact than the corresponding support
vector machine, resulting in a signiﬁcant improvement in the speed of processing on
test data. Remarkably, this greater sparsity is achieved with little or no reduction in
generalization error compared with the corresponding SVM.

The principal disadvantage of the RVM compared to the SVM is that training
involves optimizing a nonconvex function, and training times can be longer than for a
comparable SVM. For a model with Mbasis functions, the RVM requires inversion
of a matrix of size M×M, which in general requires O(M3)computation. In the
speciﬁc case of the SVM-like model (7.78), we have M=N+1. As we have noted,
there are techniques for training SVMs whose cost is roughly quadratic in N.O f

course, in the case of the RVM we always have the option of starting with a smaller
number of basis functions than N+1. More signiﬁcantly, in the relevance vector
machine the parameters governing complexity and noise variance are determined
automatically from a single training run, whereas in the support vector machine the
parameters Candϵ(orν) are generally found using cross-validation, which involves
multiple training runs. Furthermore, in the next section we shall derive an alternative

procedure for training the relevance vector machine that improves training speed
signiﬁcantly.
7.2.2 Analysis of sparsity
We have noted earlier that the mechanism of automatic relevance determination
causes a subset of parameters to be driven to zero. We now examine in more detail

350 7. SPARSE KERNEL MACHINES
t1t2
t
C
t1t2
t
Cϕ
Figure 7.10 Illustration of the mechanism for sparsity in a Bayesian linear regression model, showing a training
set vector of target values given by t=(t1,t2)T, indicated by the cross, for a model with one basis vector
ϕ=(φ(x1),φ(x2))T, which is poorly aligned with the target data vector t. On the left we see a model having
only isotropic noise, so that C=β−1I, corresponding to α=∞, with βset to its most probable value. On
the right we see the same model but with a ﬁnite value of α. In each case the red ellipse corresponds to unit
Mahalanobis distance, with |C|taking the same value for both plots, while the dashed green circle shows the
contrition arising from the noise term β−1. We see that any ﬁnite value of αreduces the probability of the
observed data, and so for the most probable solution the basis vector is removed.
the mechanism of sparsity in the context of the relevance vector machine. In the

process, we will arrive at a signiﬁcantly faster procedure for optimizing the hyper-
parameters compared to the direct techniques given above.
Before proceeding with a mathematical analysis, we ﬁrst give some informal
insight into the origin of sparsity in Bayesian linear models. Consider a data set
comprising N=2 observations t1andt2, together with a model having a single
basis function φ(x), with hyperparameter α, along with isotropic noise having pre-

cision β. From (7.85), the marginal likelihood is given by p(t|α,β)=N(t|0,C)in
which the covariance matrix takes the form
C=1
βI+1
αϕϕT(7.92)
where ϕdenotes the N-dimensional vector (φ(x1),φ(x2))T, and similarly t=
(t1,t2)T. Notice that this is just a zero-mean Gaussian process model over twith
covariance C. Given a particular observation for t, our goal is to ﬁnd α⋆andβ⋆by
maximizing the marginal likelihood. We see from Figure 7.10 that, if there is a poor
alignment between the direction of ϕand that of the training data vector t, then the

C=1
βI+1
corresponding hyperparameter αwill be driven to ∞, and the basis vector will be
pruned from the model. This arises because any ﬁnite value for αwill always assign
a lower probability to the data, thereby decreasing the value of the density at t, pro-
vided that βis set to its optimal value. We see that any ﬁnite value for αwould cause
the distribution to be elongated in a direction away from the data, thereby increasing
the probability mass in regions away from the observed data and hence reducing the

value of the density at the target data vector itself. For the more general case of M

7.2. Relevance Vector Machines 351
basis vectors ϕ1,...,ϕMa similar intuition holds, namely that if a particular basis
vector is poorly aligned with the data vector t, then it is likely to be pruned from the
model.
We now investigate the mechanism for sparsity from a more mathematical per-
spective, for a general case involving Mbasis functions. To motivate this analysis
we ﬁrst note that, in the result (7.87) for re-estimating the parameter αi, the terms on
the right-hand side are themselves also functions of αi. These results therefore rep-
resent implicit solutions, and iteration would be required even to determine a singleα
iwith all other αjforj̸=iﬁxed.
This suggests a different approach to solving the optimization problem for the
RVM, in which we make explicit all of the dependence of the marginal likelihood
(7.85) on a particular αiand then determine its stationary points explicitly (Faul and
Tipping, 2002; Tipping and Faul, 2003). To do this, we ﬁrst pull out the contributionfromα

iin the matrix Cdeﬁned by (7.86) to give
C=β−1I+∑
j̸=iα−1
jϕjϕT
j+α−1
iϕiϕT
i
=C−i+α−1
iϕiϕT
i (7.93)
where ϕidenotes the ithcolumn of Φ, in other words the N-dimensional vector with
elements (φi(x1),...,φ i(xN)), in contrast to φn, which denotes the nthrow of Φ.
The matrix C−irepresents the matrix Cwith the contribution from basis function i
removed. Using the matrix identities (C.7) and (C.15), the determinant and inverse
ofCcan then be written
|C|=|C−i||1+α−1
iϕT
iC−1
−iϕi| (7.94)
C−1=C−1
−i−C−1
−iϕiϕT
iC−1
−i
αi+ϕT
iC−1
−iϕi. (7.95)

C=β−1I+∑
j̸=iα−1
jϕjϕT
j+α−1
iϕiϕT
i
=C−i+α−1
iϕiϕT
i (7.93)
iϕT
iC−1
C−1=C−1
−i−C−1
−iϕiϕT
iC−1
−i
αi+ϕT
iC−1
Using these results, we can then write the log marginal likelihood function (7.85) in
the form Exercise 7.15
L(α)=L(α−i)+λ(αi) (7.96)
where L(α−i)is simply the log marginal likelihood with basis function ϕiomitted,
and the quantity λ(αi)is deﬁned by
λ(αi)=1
2[
lnαi−ln(αi+si)+q2
i
αi+si]
(7.97)
and contains all of the dependence on αi. Here we have introduced the two quantities
si=ϕT
iC−1
−iϕi (7.98)
qi=ϕT
iC−1
−it. (7.99)
Heresiis called the sparsity andqiis known as the quality ofϕi, and as we shall

λ(αi)=1
2[
i
αi+si]
(7.97)
si=ϕT
iC−1
qi=ϕT
iC−1
see, a large value of sirelative to the value of qimeans that the basis function ϕi

352 7. SPARSE KERNEL MACHINES
Figure 7.11 Plots of the log
marginal likelihood λ(αi)versus
lnαishowing on the left, the single
maximum at a ﬁnite αiforq2
i=4
andsi=1(so that q2
i>si) and on
the right, the maximum at αi=∞
forq2
i=1 andsi=2 (so that
q2
i<si).
−5 0 5−4−202
is more likely to be pruned from the model. The ‘sparsity’ measures the extent to
which basis function ϕioverlaps with the other basis vectors in the model, and the
‘quality’ represents a measure of the alignment of the basis vector ϕnwith the error
between the training set values t=(t1,...,t N)Tand the vector y−iof predictions
that would result from the model with the vector ϕiexcluded (Tipping and Faul,
2003).
The stationary points of the marginal likelihood with respect to αioccur when
the derivative
dλ(αi)
dαi=α−1
is2
i−(q2
i−si)
2(αi+si)2(7.100)
is equal to zero. There are two possible forms for the solution. Recalling that αi⩾0,
we see that if q2
i<si, thenαi→∞ provides a solution. Conversely, if q2
i>si,w e

2003).
dλ(αi)
dαi=α−1
is2
i−(q2
i−si)
i>si,w e
can solve for αito obtain
αi=s2
i
q2
i−si. (7.101)
These two solutions are illustrated in Figure 7.11. We see that the relative size of
the quality and sparsity terms determines whether a particular basis vector will be
pruned from the model or not. A more complete analysis (Faul and Tipping, 2002),
based on the second derivatives of the marginal likelihood, conﬁrms these solutions
are indeed the unique maxima of λ(αi). Exercise 7.16
Note that this approach has yielded a closed-form solution for αi, for given

αi=s2
i
q2
values of the other hyperparameters. As well as providing insight into the origin of
sparsity in the RVM, this analysis also leads to a practical algorithm for optimizing
the hyperparameters that has signiﬁcant speed advantages. This uses a ﬁxed set
of candidate basis vectors, and then cycles through them in turn to decide whether
each vector should be included in the model or not. The resulting sequential sparse
Bayesian learning algorithm is described below.
Sequential Sparse Bayesian Learning Algorithm

1. If solving a regression problem, initialize β.
2. Initialize using one basis function ϕ1, with hyperparameter α1set using
(7.101), with the remaining hyperparameters αjforj̸=iinitialized to
inﬁnity, so that only ϕ1is included in the model.

7.2. Relevance Vector Machines 353
3. Evaluate Σandm, along with qiandsifor all basis functions.
4. Select a candidate basis function ϕi.
5. Ifq2
i>si, andαi<∞, so that the basis vector ϕiis already included in
the model, then update αiusing (7.101).
6. Ifq2
i>si, andαi=∞, then add ϕito the model, and evaluate hyperpa-
rameter αiusing (7.101).
7. Ifq2
i⩽si, andαi<∞then remove basis function ϕifrom the model,
and set αi=∞.
8. If solving a regression problem, update β.
9. If converged terminate, otherwise go to 3.
Note that if q2
i⩽siandαi=∞, then the basis function ϕiis already excluded
from the model and no action is required.
In practice, it is convenient to evaluate the quantities
Qi=ϕT
iC−1t (7.102)
Si=ϕT
iC−1ϕi. (7.103)
The quality and sparseness variables can then be expressed in the form
qi=αiQi
αi−Si(7.104)
si=αiSi
αi−Si. (7.105)
Note that when αi=∞,w eh a v e qi=Qiandsi=Si. Using (C.7), we can write Exercise 7.17
Qi=βϕT
it−β2ϕT
iΦΣΦTt (7.106)
Si=βϕT
iϕi−β2ϕT
iΦΣΦTϕi (7.107)

Qi=ϕT
Si=ϕT
qi=αiQi
si=αiSi
Qi=βϕT
it−β2ϕT
Si=βϕT
iϕi−β2ϕT
whereΦandΣinvolve only those basis vectors that correspond to ﬁnite hyperpa-
rameters αi. At each stage the required computations therefore scale like O(M3),
where Mis the number of active basis vectors in the model and is typically much
smaller than the number Nof training patterns.
7.2.3 RVM for classiﬁcation
We can extend the relevance vector machine framework to classiﬁcation prob-
lems by applying the ARD prior over weights to a probabilistic linear classiﬁcation

Si=βϕT
iϕi−β2ϕT
model of the kind studied in Chapter 4. To start with, we consider two-class prob-
lems with a binary target variable t∈{0,1}. The model now takes the form of a
linear combination of basis functions transformed by a logistic sigmoid function
y(x,w)=σ(
wTφ(x))
(7.108)

354 7. SPARSE KERNEL MACHINES
where σ(·)is the logistic sigmoid function deﬁned by (4.59). If we introduce a
Gaussian prior over the weight vector w, then we obtain the model that has been
considered already in Chapter 4. The difference here is that in the RVM, this model
uses the ARD prior (7.80) in which there is a separate precision hyperparameter
associated with each weight parameter.
In contrast to the regression model, we can no longer integrate analytically over
the parameter vector w. Here we follow Tipping (2001) and use the Laplace ap-
proximation, which was applied to the closely related problem of Bayesian logistic Section 4.4
regression in Section 4.5.1.
We begin by initializing the hyperparameter vector α. For this given value of
α, we then build a Gaussian approximation to the posterior distribution and thereby
obtain an approximation to the marginal likelihood. Maximization of this approxi-
mate marginal likelihood then leads to a re-estimated value for α, and the process is

repeated until convergence.
Let us consider the Laplace approximation for this model in more detail. For
a ﬁxed value of α, the mode of the posterior distribution over wis obtained by
lnp(w|t,α)=l n {p(t|w)p(w|α)}−lnp(t|α)
=N∑
n=1{tnlnyn+( 1−tn)l n ( 1−yn)}−1
2wTAw+c o n s t (7.109)
whereA=d i a g ( αi). This can be done using iterative reweighted least squares
(IRLS) as discussed in Section 4.3.3. For this, we need the gradient vector and
Hessian matrix of the log posterior distribution, which from (7.109) are given by Exercise 7.18

=N∑
∇lnp(w|t,α)= ΦT(t−y)−Aw (7.110)
∇∇lnp(w|t,α)= −(
ΦTBΦ+A)
(7.111)
whereBis anN×Ndiagonal matrix with elements bn=yn(1−yn), the vector
y=(y1,...,y N)T, andΦis the design matrix with elements Φni=φi(xn). Here
we have used the property (4.88) for the derivative of the logistic sigmoid function.At convergence of the IRLS algorithm, the negative Hessian represents the inverse
covariance matrix for the Gaussian approximation to the posterior distribution.
The mode of the resulting approximation to the posterior distribution, corre-

ΦTBΦ+A)
(7.111)
sponding to the mean of the Gaussian approximation, is obtained setting (7.110) to
zero, giving the mean and covariance of the Laplace approximation in the form
w
⋆=A−1ΦT(t−y) (7.112)
Σ=(
ΦTBΦ+A)−1. (7.113)
We can now use this Laplace approximation to evaluate the marginal likelihood.
Using the general result (4.135) for an integral evaluated using the Laplace approxi-

7.2. Relevance Vector Machines 355
mation, we have
p(t|α)=∫
p(t|w)p(w|α)dw
≃p(t|w⋆)p(w⋆|α)(2π)M/2|Σ|1/2. (7.114)
If we substitute for p(t|w⋆)andp(w⋆|α)and then set the derivative of the marginal
likelihood with respect to αiequal to zero, we obtain Exercise 7.19
−1
2(w⋆
i)2+1
2αi−1
2Σii=0. (7.115)
Deﬁning γi=1−αiΣiiand rearranging then gives
αnew
i=γi
(w⋆
i)2(7.116)
which is identical to the re-estimation formula (7.87) obtained for the regression
RVM.
If we deﬁne
ˆt=Φw⋆+B−1(t−y) (7.117)
we can write the approximate log marginal likelihood in the form
lnp(t|α,β)=−1
2{
Nln(2π)+l n|C|+(ˆt)TC−1ˆt}
(7.118)
where
C=B+ΦAΦT. (7.119)
This takes the same form as (7.85) in the regression case, and so we can apply the
same analysis of sparsity and obtain the same fast learning algorithm in which we
fully optimize a single hyperparameter αiat each step.
Figure 7.12 shows the relevance vector machine applied to a synthetic classiﬁ-

RVM.
If we deﬁne
2{
(7.118)
where
cation data set. We see that the relevance vectors tend not to lie in the region of the Appendix A
decision boundary, in contrast to the support vector machine. This is consistent with
our earlier discussion of sparsity in the RVM, because a basis function φi(x)centred
on a data point near the boundary will have a vector ϕithat is poorly aligned with
the training data vector t.
One of the potential advantages of the relevance vector machine compared with

the SVM is that it makes probabilistic predictions. For example, this allows the RVM
to be used to help construct an emission density in a nonlinear extension of the linear
dynamical system for tracking faces in video sequences (Williams et al. , 2005). Section 13.3
So far, we have considered the RVM for binary classiﬁcation problems. For
K>2classes, we again make use of the probabilistic approach in Section 4.3.4 in
which there are Klinear models of the form
ak=wT
kx (7.120)

356 7. SPARSE KERNEL MACHINES
−2 0 2−202
Figure 7.12 Example of the relevance vector machine applied to a synthetic data set, in which the left-hand plot
shows the decision boundary and the data points, with the relevance vectors indicated by circles. Comparison
with the results shown in Figure 7.4 for the corresponding support vector machine shows that the RVM gives a
much sparser model. The right-hand plot shows the posterior probability given by the RVM output in which the
proportion of red (blue) ink indicates the probability of that point belonging to the red (blue) class.
which are combined using a softmax function to give outputs
yk(x)=exp(ak)∑
jexp(aj). (7.121)
The log likelihood function is then given by
lnp(T|w1,...,wK)=N∏
n=1K∏
k=1ytnk
nk(7.122)
where the target values tnkhave a 1-of- Kcoding for each data point n, andTis a
matrix with elements tnk. Again, the Laplace approximation can be used to optimize
the hyperparameters (Tipping, 2001), in which the model and its Hessian are found

n=1K∏
k=1ytnk
nk(7.122)
using IRLS. This gives a more principled approach to multiclass classiﬁcation than
the pairwise method used in the support vector machine and also provides probabilis-
tic predictions for new data points. The principal disadvantage is that the Hessian
matrix has size MK×MK , where Mis the number of active basis functions, which
gives an additional factor of K3in the computational cost of training compared with
the two-class RVM.
The principal disadvantage of the relevance vector machine is the relatively long

training times compared with the SVM. This is offset, however, by the avoidance of
cross-validation runs to set the model complexity parameters. Furthermore, because
it yields sparser models, the computation time on test points, which is usually the
more important consideration in practice, is typically much less.

Exercises 357
Exercises
7.1 (⋆⋆)www Suppose we have a data set of input vectors {xn}with corresponding
target values tn∈{ −1,1}, and suppose that we model the density of input vec-
tors within each class separately using a Parzen kernel density estimator (see Sec-
tion 2.5.1) with a kernel k(x,x′). Write down the minimum misclassiﬁcation-rate
decision rule assuming the two classes have equal prior probability. Show also that,if the kernel is chosen to be k(x,x
′)=xTx′, then the classiﬁcation rule reduces to
simply assigning a new input vector to the class having the closest mean. Finally,
show that, if the kernel takes the form k(x,x′)=φ(x)Tφ(x′), that the classiﬁcation
is based on the closest mean in the feature space φ(x).
7.2 (⋆)Show that, if the 1on the right-hand side of the constraint (7.5) is replaced by
some arbitrary constant γ>0, the solution for the maximum margin hyperplane is
7.3 (⋆⋆)Show that, irrespective of the dimensionality of the data space, a data set

consisting of just two data points, one from each class, is sufﬁcient to determine the
location of the maximum-margin hyperplane.
7.4 (⋆⋆)www Show that the value ρof the margin for the maximum-margin hyper-
plane is given by
1
ρ2=N∑
n=1an (7.123)
where {an}are given by maximizing (7.10) subject to the constraints (7.11) and
(7.12).
7.5 (⋆⋆)Show that the values of ρand{an}in the previous exercise also satisfy
1
ρ2=2˜L(a) (7.124)
where˜L(a)is deﬁned by (7.10). Similarly, show that
1
ρ2=∥w∥2. (7.125)

1
ρ2=N∑
(7.12).
1
1
7.6 (⋆)Consider the logistic regression model with a target variable t∈{ −1,1}.I f
we deﬁne p(t=1|y)=σ(y)where y(x)is given by (7.1), show that the negative
log likelihood, with the addition of a quadratic regularization term, takes the form(7.47).
7.7 (⋆)Consider the Lagrangian (7.56) for the regression support vector machine. By
setting the derivatives of the Lagrangian with respect to w,b,ξ
n, andˆξnto zero and
then back substituting to eliminate the corresponding variables, show that the dual

Lagrangian is given by (7.61).

358 7. SPARSE KERNEL MACHINES
7.8 (⋆)www For the regression support vector machine considered in Section 7.1.4,
show that all training data points for which ξn>0will have an=C, and similarly
all points for which ˆξn>0will haveˆan=C.
7.9 (⋆)Verify the results (7.82) and (7.83) for the mean and covariance of the posterior
distribution over weights in the regression RVM.
7.10 (⋆⋆)www Derive the result (7.85) for the marginal likelihood function in the
regression RVM, by performing the Gaussian integral over win (7.84) using the
technique of completing the square in the exponential.
7.11 (⋆⋆)Repeat the above exercise, but this time make use of the general result (2.115).
7.12 (⋆⋆)www Show that direct maximization of the log marginal likelihood (7.85) for
the regression relevance vector machine leads to the re-estimation equations (7.87)and (7.88) where γ
iis deﬁned by (7.89).
7.13 (⋆⋆)In the evidence framework for RVM regression, we obtained the re-estimation

formulae (7.87) and (7.88) by maximizing the marginal likelihood given by (7.85).
Extend this approach by inclusion of hyperpriors given by gamma distributions ofthe form (B.26) and obtain the corresponding re-estimation formulae for αandβby
maximizing the corresponding posterior probability p(t,α,β|X)with respect to α
andβ.
7.14 (⋆⋆)Derive the result (7.90) for the predictive distribution in the relevance vector
machine for regression. Show that the predictive variance is given by (7.91).
7.15 (⋆⋆)
www Using the results (7.94) and (7.95), show that the marginal likelihood

andβ.
7.15 (⋆⋆)
(7.85) can be written in the form (7.96), where λ(αn)is deﬁned by (7.97) and the
sparsity and quality factors are deﬁned by (7.98) and (7.99), respectively.
7.16 (⋆)By taking the second derivative of the log marginal likelihood (7.97) for the
regression RVM with respect to the hyperparameter αi, show that the stationary
point given by (7.101) is a maximum of the marginal likelihood.
7.17 (⋆⋆)Using (7.83) and (7.86), together with the matrix identity (C.7), show that

the quantities SnandQndeﬁned by (7.102) and (7.103) can be written in the form
(7.106) and (7.107).
7.18 (⋆)www Show that the gradient vector and Hessian matrix of the log poste-
rior distribution (7.109) for the classiﬁcation relevance vector machine are given by
(7.110) and (7.111).
7.19 (⋆⋆)Verify that maximization of the approximate log marginal likelihood function
(7.114) for the classiﬁcation relevance vector machine leads to the result (7.116) for
re-estimation of the hyperparameters.

8
Graphical
Models
Probabilities play a central role in modern pattern recognition. We have seen in
Chapter 1 that probability theory can be expressed in terms of two simple equationscorresponding to the sum rule and the product rule. All of the probabilistic infer-
ence and learning manipulations discussed in this book, no matter how complex,
amount to repeated application of these two equations. We could therefore proceedto formulate and solve complicated probabilistic models purely by algebraic ma-
nipulation. However, we shall ﬁnd it highly advantageous to augment the analysis
using diagrammatic representations of probability distributions, called probabilistic
graphical models . These offer several useful properties:
1. They provide a simple way to visualize the structure of a probabilistic model
and can be used to design and motivate new models.
2. Insights into the properties of the model, including conditional independence
properties, can be obtained by inspection of the graph.
359

360 8. GRAPHICAL MODELS
3. Complex computations, required to perform inference and learning in sophis-
ticated models, can be expressed in terms of graphical manipulations, in whichunderlying mathematical expressions are carried along implicitly.
A graph comprises nodes (also called vertices ) connected by links (also known
asedges orarcs). In a probabilistic graphical model, each node represents a random
variable (or group of random variables), and the links express probabilistic relation-
ships between these variables. The graph then captures the way in which the jointdistribution over all of the random variables can be decomposed into a product of
factors each depending only on a subset of the variables. We shall begin by dis-
cussing Bayesian networks , also known as directed graphical models , in which the
links of the graphs have a particular directionality indicated by arrows. The other
major class of graphical models are Markov random ﬁelds , also known as undirected

graphical models , in which the links do not carry arrows and have no directional
signiﬁcance. Directed graphs are useful for expressing causal relationships between
random variables, whereas undirected graphs are better suited to expressing soft con-
straints between random variables. For the purposes of solving inference problems,it is often convenient to convert both directed and undirected graphs into a different
representation called a factor graph .

In this chapter, we shall focus on the key aspects of graphical models as needed
for applications in pattern recognition and machine learning. More general treat-
ments of graphical models can be found in the books by Whittaker (1990), Lauritzen
(1996), Jensen (1996), Castillo et al. (1997), Jordan (1999), Cowell et al. (1999),
and Jordan (2007).
8.1. Bayesian Networks
In order to motivate the use of directed graphs to describe probability distributions,consider ﬁrst an arbitrary joint distribution p(a, b, c)over three variables a,b, andc.

Note that at this stage, we do not need to specify anything further about these vari-ables, such as whether they are discrete or continuous. Indeed, one of the powerful
aspects of graphical models is that a speciﬁc graph can make probabilistic statements
for a broad class of distributions. By application of the product rule of probability(1.11), we can write the joint distribution in the form
p(a, b, c)=p(c|a, b)p(a, b). (8.1)
A second application of the product rule, this time to the second term on the right-
hand side of (8.1), gives

p(a, b, c)=p(c|a, b)p(b|a)p(a). (8.2)
Note that this decomposition holds for any choice of the joint distribution. We now
represent the right-hand side of (8.2) in terms of a simple graphical model as follows.First we introduce a node for each of the random variables a,b, andcand associate
each node with the corresponding conditional distribution on the right-hand side of

8.1. Bayesian Networks 361
Figure 8.1 A directed graphical model representing the joint probabil-
ity distribution over three variables a,b, andc, correspond-
ing to the decomposition on the right-hand side of (8.2).a
b
c
(8.2). Then, for each conditional distribution we add directed links (arrows) to the
graph from the nodes corresponding to the variables on which the distribution is
conditioned. Thus for the factor p(c|a, b), there will be links from nodes aandbto
nodec, whereas for the factor p(a)there will be no incoming links. The result is the
graph shown in Figure 8.1. If there is a link going from a node ato a node b, then we
say that node ais the parent of node b, and we say that node bis the child of node a.
Note that we shall not make any formal distinction between a node and the variable
to which it corresponds but will simply use the same symbol to refer to both.
An interesting point to note about (8.2) is that the left-hand side is symmetrical

with respect to the three variables a,b, andc, whereas the right-hand side is not.
Indeed, in making the decomposition in (8.2), we have implicitly chosen a particular
ordering, namely a, b, c , and had we chosen a different ordering we would have
obtained a different decomposition and hence a different graphical representation.
We shall return to this point later.
For the moment let us extend the example of Figure 8.1 by considering the joint

distribution over Kvariables given by p(x1,...,x K). By repeated application of
the product rule of probability, this joint distribution can be written as a product of
conditional distributions, one for each of the variables
p(x1,...,x K)=p(xK|x1,...,x K−1)...p(x2|x1)p(x1). (8.3)
For a given choice of K, we can again represent this as a directed graph having K
nodes, one for each conditional distribution on the right-hand side of (8.3), with each
node having incoming links from all lower numbered nodes. We say that this graph

isfully connected because there is a link between every pair of nodes.
So far, we have worked with completely general joint distributions, so that the
decompositions, and their representations as fully connected graphs, will be applica-
ble to any choice of distribution. As we shall see shortly, it is the absence of links
in the graph that conveys interesting information about the properties of the class of
distributions that the graph represents. Consider the graph shown in Figure 8.2. This

is not a fully connected graph because, for instance, there is no link from x1tox2or
fromx3tox7.
We shall now go from this graph to the corresponding representation of the joint
probability distribution written in terms of the product of a set of conditional dis-
tributions, one for each node in the graph. Each such conditional distribution will
be conditioned only on the parents of the corresponding node in the graph. For in-
stance, x5will be conditioned on x1andx3. The joint distribution of all 7variables

362 8. GRAPHICAL MODELS
Figure 8.2 Example of a directed acyclic graph describing the joint
distribution over variables x1,...,x 7. The corresponding
decomposition of the joint distribution is given by (8.4).x1
x2 x3
x4 x5
x6 x7
is therefore given by
p(x1)p(x2)p(x3)p(x4|x1,x2,x3)p(x5|x1,x3)p(x6|x4)p(x7|x4,x5). (8.4)
The reader should take a moment to study carefully the correspondence between
(8.4) and Figure 8.2.
We can now state in general terms the relationship between a given directed
graph and the corresponding distribution over the variables. The joint distribution
deﬁned by a graph is given by the product, over all of the nodes of the graph, of
a conditional distribution for each node conditioned on the variables corresponding
to the parents of that node in the graph. Thus, for a graph with Knodes, the joint
p(x)=K∏
k=1p(xk|pak) (8.5)
where pakdenotes the set of parents of xk, andx={x1,...,x K}. This key

p(x)=K∏
equation expresses the factorization properties of the joint distribution for a directed
graphical model. Although we have considered each node to correspond to a single
variable, we can equally well associate sets of variables and vector-valued variables
with the nodes of a graph. It is easy to show that the representation on the right-
hand side of (8.5) is always correctly normalized provided the individual conditional
distributions are normalized. Exercise 8.1
The directed graphs that we are considering are subject to an important restric-

tion namely that there must be no directed cycles , in other words there are no closed
paths within the graph such that we can move from node to node along links follow-
ing the direction of the arrows and end up back at the starting node. Such graphs are
also called directed acyclic graphs ,o r DAGs . This is equivalent to the statement that Exercise 8.2
there exists an ordering of the nodes such that there are no links that go from any
node to any lower numbered node.
8.1.1 Example: Polynomial regression

As an illustration of the use of directed graphs to describe probability distri-
butions, we consider the Bayesian polynomial regression model introduced in Sec-

8.1. Bayesian Networks 363
Figure 8.3 Directed graphical model representing the joint
distribution (8.6) corresponding to the Bayesian
polynomial regression model introduced in Sec-
tion 1.2.6.w
t1 tN
tion 1.2.6. The random variables in this model are the vector of polynomial coefﬁ-
cientswand the observed data t=(t1,...,t N)T. In addition, this model contains
the input data x=(x1,...,x N)T, the noise variance σ2, and the hyperparameter α
representing the precision of the Gaussian prior over w, all of which are parameters
of the model rather than random variables. Focussing just on the random variables
for the moment, we see that the joint distribution is given by the product of the prior
p(w)andNconditional distributions p(tn|w)forn=1,...,N so that
p(t,w)=p(w)N∏
n=1p(tn|w). (8.6)
This joint distribution can be represented by a graphical model shown in Figure 8.3.
When we start to deal with more complex models later in the book, we shall ﬁnd

it inconvenient to have to write out multiple nodes of the form t1,...,t Nexplicitly as
in Figure 8.3. We therefore introduce a graphical notation that allows such multiple
nodes to be expressed more compactly, in which we draw a single representative
nodetnand then surround this with a box, called a plate , labelled with Nindicating
that there are Nnodes of this kind. Re-writing the graph of Figure 8.3 in this way,
we obtain the graph shown in Figure 8.4.
We shall sometimes ﬁnd it helpful to make the parameters of a model, as well as

its stochastic variables, explicit. In this case, (8.6) becomes
p(t,w|x,α,σ2)=p(w|α)N∏
n=1p(tn|w,xn,σ2).
Correspondingly, we can make xandαexplicit in the graphical representation. To
do this, we shall adopt the convention that random variables will be denoted by open
circles, and deterministic parameters will be denoted by smaller solid circles. If we
take the graph of Figure 8.4 and include the deterministic parameters, we obtain the
graph shown in Figure 8.5.
When we apply a graphical model to a problem in machine learning or pattern

recognition, we will typically set some of the random variables to speciﬁc observed
Figure 8.4 An alternative, more compact, representation of the graph
shown in Figure 8.3 in which we have introduced a plate
(the box labelled N) that represents Nnodes of which only
a single example tnis shown explicitly.tn
Nw

364 8. GRAPHICAL MODELS
Figure 8.5 This shows the same model as in Figure 8.4 but
with the deterministic parameters shown explicitly
by the smaller solid nodes.
tnxn
Nwα
σ2
values, for example the variables {tn}from the training set in the case of polynomial
curve ﬁtting. In a graphical model, we will denote such observed variables by shad-
ing the corresponding nodes. Thus the graph corresponding to Figure 8.5 in which
the variables {tn}are observed is shown in Figure 8.6. Note that the value of wis
not observed, and so wis an example of a latent variable, also known as a hidden
variable. Such variables play a crucial role in many probabilistic models and will
form the focus of Chapters 9 and 12.
Having observed the values {tn}we can, if desired, evaluate the posterior dis-
tribution of the polynomial coefﬁcients was discussed in Section 1.2.5. For the
moment, we note that this involves a straightforward application of Bayes’ theorem
p(w|T)∝p(w)N∏
n=1p(tn|w) (8.7)

where again we have omitted the deterministic parameters in order to keep the nota-
tion uncluttered.
In general, model parameters such as ware of little direct interest in themselves,
because our ultimate goal is to make predictions for new input values. Suppose we
are given a new input value ˆxand we wish to ﬁnd the corresponding probability dis-
tribution for ˆtconditioned on the observed data. The graphical model that describes
this problem is shown in Figure 8.7, and the corresponding joint distribution of all

of the random variables in this model, conditioned on the deterministic parameters,
is then given by
p(ˆt,t,w|ˆx,x,α,σ2)=[N∏
n=1p(tn|xn,w,σ2)]
p(w|α)p(ˆt|ˆx,w,σ2). (8.8)
Figure 8.6 As in Figure 8.5 but with the nodes {tn}shaded
to indicate that the corresponding random vari-
ables have been set to their observed (training set)
values.
tnxn
Nwα
σ2

8.1. Bayesian Networks 365
Figure 8.7 The polynomial regression model, corresponding
to Figure 8.6, showing also a new input value bx
together with the corresponding model prediction
bt.
tnxn
Nwα
ˆtσ2ˆx
The required predictive distribution for ˆtis then obtained, from the sum rule of
probability, by integrating out the model parameters wso that
p(ˆt|ˆx,x,t,α,σ2)∝∫
p(ˆt,t,w|ˆx,x,α,σ2)dw
where we are implicitly setting the random variables in tto the speciﬁc values ob-
served in the data set. The details of this calculation were discussed in Chapter 3.
8.1.2 Generative models
There are many situations in which we wish to draw samples from a given prob-
ability distribution. Although we shall devote the whole of Chapter 11 to a detailed
discussion of sampling methods, it is instructive to outline here one technique, called
ancestral sampling , which is particularly relevant to graphical models. Consider a
joint distribution p(x1,...,x K)overKvariables that factorizes according to (8.5)

corresponding to a directed acyclic graph. We shall suppose that the variables have
been ordered such that there are no links from any node to any lower numbered node,
in other words each node has a higher number than any of its parents. Our goal is to
draw a sample ˆx1,...,ˆxKfrom the joint distribution.
To do this, we start with the lowest-numbered node and draw a sample from the
distribution p(x1), which we call ˆx1. We then work through each of the nodes in or-
der, so that for node nwe draw a sample from the conditional distribution p(xn|pan)

in which the parent variables have been set to their sampled values. Note that at each
stage, these parent values will always be available because they correspond to lower-
numbered nodes that have already been sampled. Techniques for sampling from
speciﬁc distributions will be discussed in detail in Chapter 11. Once we have sam-
pled from the ﬁnal variable xK, we will have achieved our objective of obtaining a
sample from the joint distribution. To obtain a sample from some marginal distribu-

tion corresponding to a subset of the variables, we simply take the sampled values
for the required nodes and ignore the sampled values for the remaining nodes. For
example, to draw a sample from the distribution p(x2,x4), we simply sample from
the full joint distribution and then retain the values ˆx2,ˆx4and discard the remaining
values {ˆxj̸=2,4}.

366 8. GRAPHICAL MODELS
Figure 8.8 A graphical model representing the process by which
images of objects are created, in which the identity
of an object (a discrete variable) and the position and
orientation of that object (continuous variables) have
independent prior probabilities. The image (a vector
of pixel intensities) has a probability distribution that
is dependent on the identity of the object as well as
on its position and orientation.
ImageObject Orientation Position
For practical applications of probabilistic models, it will typically be the higher-
numbered variables corresponding to terminal nodes of the graph that represent the
observations, with lower-numbered nodes corresponding to latent variables. The
primary role of the latent variables is to allow a complicated distribution over the
observed variables to be represented in terms of a model constructed from simpler
(typically exponential family) conditional distributions.

We can interpret such models as expressing the processes by which the observed
data arose. For instance, consider an object recognition task in which each observed
data point corresponds to an image (comprising a vector of pixel intensities) of one
of the objects. In this case, the latent variables might have an interpretation as the
position and orientation of the object. Given a particular observed image, our goal is
to ﬁnd the posterior distribution over objects, in which we integrate over all possible

positions and orientations. We can represent this problem using a graphical model
of the form show in Figure 8.8.
The graphical model captures the causal process (Pearl, 1988) by which the ob-
served data was generated. For this reason, such models are often called generative
models. By contrast, the polynomial regression model described by Figure 8.5 is
not generative because there is no probability distribution associated with the input

variable x, and so it is not possible to generate synthetic data points from this model.
We could make it generative by introducing a suitable prior distribution p(x), at the
expense of a more complex model.
The hidden variables in a probabilistic model need not, however, have any ex-
plicit physical interpretation but may be introduced simply to allow a more complex
joint distribution to be constructed from simpler components. In either case, the
technique of ancestral sampling applied to a generative model mimics the creation

of the observed data and would therefore give rise to ‘fantasy’ data whose probability
distribution (if the model were a perfect representation of reality) would be the same
as that of the observed data. In practice, producing synthetic observations from a
generative model can prove informative in understanding the form of the probability
distribution represented by that model.
8.1.3 Discrete variables
We have discussed the importance of probability distributions that are members
of the exponential family, and we have seen that this family includes many well- Section 2.4

known distributions as particular cases. Although such distributions are relatively
simple, they form useful building blocks for constructing more complex probability

8.1. Bayesian Networks 367
Figure 8.9 (a) This fully-connected graph describes a general distribu-
tion over two K-state discrete variables having a total of
K2−1parameters. (b) By dropping the link between the
nodes, the number of parameters is reduced to 2(K−1).(a)x1 x2
(b)x1 x2
distributions, and the framework of graphical models is very useful in expressing the
way in which these building blocks are linked together.
Such models have particularly nice properties if we choose the relationship be-
tween each parent-child pair in a directed graph to be conjugate, and we shall ex-
plore several examples of this shortly. Two cases are particularly worthy of note,
namely when the parent and child node each correspond to discrete variables and
when they each correspond to Gaussian variables, because in these two cases the
relationship can be extended hierarchically to construct arbitrarily complex directed
acyclic graphs. We begin by examining the discrete case.

The probability distribution p(x|µ)for a single discrete variable xhaving K
possible states (using the 1-of- Krepresentation) is given by
p(x|µ)=K∏
k=1µxk
k(8.9)
and is governed by the parameters µ=(µ1,...,µ K)T. Due to the constraint∑
kµk=1, only K−1values for µkneed to be speciﬁed in order to deﬁne the
Now suppose that we have two discrete variables, x1andx2, each of which has
Kstates, and we wish to model their joint distribution. We denote the probability of
observing both x1k=1 andx2l=1 by the parameter µkl, where x1kdenotes the

p(x|µ)=K∏
k=1µxk
k(8.9)
kthcomponent of x1, and similarly for x2l. The joint distribution can be written
p(x1,x2|µ)=K∏
k=1K∏
l=1µx1kx2l
kl.
Because the parameters µklare subject to the constraint∑
k∑
lµkl=1, this distri-
bution is governed by K2−1parameters. It is easily seen that the total number of
parameters that must be speciﬁed for an arbitrary joint distribution over Mvariables
isKM−1and therefore grows exponentially with the number Mof variables.
Using the product rule, we can factor the joint distribution p(x1,x2)in the form

k=1K∏
kl.
k∑
p(x2|x1)p(x1), which corresponds to a two-node graph with a link going from the
x1node to the x2node as shown in Figure 8.9(a). The marginal distribution p(x1)
is governed by K−1parameters, as before, Similarly, the conditional distribution
p(x2|x1)requires the speciﬁcation of K−1parameters for each of the Kpossible
values of x1. The total number of parameters that must be speciﬁed in the joint
distribution is therefore (K−1) +K(K−1) =K2−1as before.
Now suppose that the variables x1andx2were independent, corresponding to

the graphical model shown in Figure 8.9(b). Each variable is then described by

368 8. GRAPHICAL MODELS
Figure 8.10 This chain of Mdiscrete nodes, each
having Kstates, requires the speciﬁcation of K−1+
(M−1)K(K−1)parameters, which grows linearly
with the length Mof the chain. In contrast, a fully con-
nected graph of Mnodes would have KM−1param-
eters, which grows exponentially with M.x1 x2 xM
a separate multinomial distribution, and the total number of parameters would be
2(K−1). For a distribution over Mindependent discrete variables, each having K
states, the total number of parameters would be M(K−1), which therefore grows
linearly with the number of variables. From a graphical perspective, we have reduced
the number of parameters by dropping links in the graph, at the expense of having a
restricted class of distributions.
More generally, if we have Mdiscrete variables x1,...,xM, we can model
the joint distribution using a directed graph with one variable corresponding to each
node. The conditional distribution at each node is given by a set of nonnegative pa-

rameters subject to the usual normalization constraint. If the graph is fully connected
then we have a completely general distribution having KM−1parameters, whereas
if there are no links in the graph the joint distribution factorizes into the product of
the marginals, and the total number of parameters is M(K−1). Graphs having in-
termediate levels of connectivity allow for more general distributions than the fully
factorized one while requiring fewer parameters than the general joint distribution.

As an illustration, consider the chain of nodes shown in Figure 8.10. The marginal
distribution p(x1)requires K−1parameters, whereas each of the M−1condi-
tional distributions p(xi|xi−1), fori=2,...,M , requires K(K−1)parameters.
This gives a total parameter count of K−1+(M−1)K(K−1), which is quadratic
inKand which grows linearly (rather than exponentially) with the length Mof the
chain.
An alternative way to reduce the number of independent parameters in a model

chain.
is by sharing parameters (also known as tying of parameters). For instance, in the
chain example of Figure 8.10, we can arrange that all of the conditional distributions
p(xi|xi−1), fori=2,...,M , are governed by the same set of K(K−1)parameters.
Together with the K−1parameters governing the distribution of x1, this gives a total
ofK2−1parameters that must be speciﬁed in order to deﬁne the joint distribution.
We can turn a graph over discrete variables into a Bayesian model by introduc-

ing Dirichlet priors for the parameters. From a graphical point of view, each node
then acquires an additional parent representing the Dirichlet distribution over the pa-
rameters associated with the corresponding discrete node. This is illustrated for the
chain model in Figure 8.11. The corresponding model in which we tie the parame-
ters governing the conditional distributions p(xi|xi−1), fori=2,...,M , is shown
in Figure 8.12.
Another way of controlling the exponential growth in the number of parameters

in models of discrete variables is to use parameterized models for the conditional
distributions instead of complete tables of conditional probability values. To illus-
trate this idea, consider the graph in Figure 8.13 in which all of the nodes represent
binary variables. Each of the parent variables xiis governed by a single parame-

8.1. Bayesian Networks 369
Figure 8.11 An extension of the model of
Figure 8.10 to include Dirich-
let priors over the param-
eters governing the discrete
distributions.
x1 x2 xMµ1 µ2 µM
Figure 8.12 As in Figure 8.11 but with a sin-
gle set of parameters µshared
amongst all of the conditional
distributions p(xi|xi−1).
x1 x2 xMµ1 µ
terµirepresenting the probability p(xi=1 ) , giving Mparameters in total for the
parent nodes. The conditional distribution p(y|x1,...,x M), however, would require
2Mparameters representing the probability p(y=1 ) for each of the 2Mpossible
settings of the parent variables. Thus in general the number of parameters required
to specify this conditional distribution will grow exponentially with M. We can ob-
tain a more parsimonious form for the conditional distribution by using a logistic
sigmoid function acting on a linear combination of the parent variables, giving Section 2.4
p(y=1|x1,...,x M)=σ(
w0+M∑
i=1wixi)
=σ(wTx) (8.10)

w0+M∑
i=1wixi)
where σ(a) = (1+exp( −a))−1is the logistic sigmoid, x=(x0,x1,...,x M)Tis an
(M+1 ) -dimensional vector of parent states augmented with an additional variable
x0whose value is clamped to 1, and w=(w0,w1,...,w M)Tis a vector of M+1
parameters. This is a more restricted form of conditional distribution than the general
case but is now governed by a number of parameters that grows linearly with M.I n
this sense, it is analogous to the choice of a restrictive form of covariance matrix (for

example, a diagonal matrix) in a multivariate Gaussian distribution. The motivation
for the logistic sigmoid representation was discussed in Section 4.2.
Figure 8.13 A graph comprising Mparents x1,...,x Mand a sin-
gle child y, used to illustrate the idea of parameterized
conditional distributions for discrete variables.
yx1 xM

370 8. GRAPHICAL MODELS
8.1.4 Linear-Gaussian models
In the previous section, we saw how to construct joint probability distributions
over a set of discrete variables by expressing the variables as nodes in a directed
acyclic graph. Here we show how a multivariate Gaussian can be expressed as a
directed graph corresponding to a linear-Gaussian model over the component vari-ables. This allows us to impose interesting structure on the distribution, with the
general Gaussian and the diagonal covariance Gaussian representing opposite ex-
tremes. Several widely used techniques are examples of linear-Gaussian models,
such as probabilistic principal component analysis, factor analysis, and linear dy-
namical systems (Roweis and Ghahramani, 1999). We shall make extensive use ofthe results of this section in later chapters when we consider some of these techniques
in detail.
Consider an arbitrary directed acyclic graph over Dvariables in which node i
represents a single continuous random variable x

in detail.
ihaving a Gaussian distribution.
The mean of this distribution is taken to be a linear combination of the states of its
parent nodes paiof node i
p(xi|pai)=N⎛
⎝xi⏐⏐⏐⏐⏐⏐∑
j∈paiwijxj+bi,vi⎞
⎠ (8.11)
where wijandbiare parameters governing the mean, and viis the variance of the
conditional distribution for xi. The log of the joint distribution is then the log of the
product of these conditionals over all nodes in the graph and hence takes the form
lnp(x)=D∑
i=1lnp(xi|pai) (8.12)
=−D∑
i=11
2vi⎛
⎝xi−∑
j∈paiwijxj−bi⎞
⎠2
+c o n s t (8.13)

⎠ (8.11)
lnp(x)=D∑
=−D∑
i=11
2vi⎛
⎝xi−∑
⎠2
wherex=(x1,...,x D)Tand ‘const’ denotes terms independent of x. We see that
this is a quadratic function of the components of x, and hence the joint distribution
p(x)is a multivariate Gaussian.
We can determine the mean and covariance of the joint distribution recursively
as follows. Each variable xihas (conditional on the states of its parents) a Gaussian
distribution of the form (8.11) and so
xi=∑
j∈paiwijxj+bi+√viϵi (8.14)
where ϵiis a zero mean, unit variance Gaussian random variable satisfying E[ϵi]=0

xi=∑
and E[ϵiϵj]=Iij, where Iijis the i, jelement of the identity matrix. Taking the
expectation of (8.14), we have
E[xi]=∑
j∈paiwijE[xj]+bi. (8.15)

8.1. Bayesian Networks 371
Figure 8.14 A directed graph over three Gaussian variables,
with one missing link.x1 x2 x3
Thus we can ﬁnd the components of E[x]=( E[x1],..., E[xD])Tby starting at the
lowest numbered node and working recursively through the graph (here we again
assume that the nodes are numbered such that each node has a higher number than
its parents). Similarly, we can use (8.14) and (8.15) to obtain the i, jelement of the
covariance matrix for p(x)in the form of a recursion relation
cov[xi,xj]= E[(xi−E[xi])(xj−E[xj])]
= E⎡
⎣(xi−E[xi])⎧
⎨
⎩∑
k∈pajwjk(xk−E[xk]) +√vjϵj⎫
⎬
⎭⎤
⎦
=∑
k∈pajwjkcov[xi,xk]+Iijvj (8.16)
and so the covariance can similarly be evaluated recursively starting from the lowest
numbered node.
Let us consider two extreme cases. First of all, suppose that there are no links
in the graph, which therefore comprises Disolated nodes. In this case, there are no
parameters wijand so there are just Dparameters biandDparameters vi. From

= E⎡
⎨
⎩∑
⎬
⎭⎤
⎦
=∑
the recursion relations (8.15) and (8.16), we see that the mean of p(x)is given by
(b1,...,b D)Tand the covariance matrix is diagonal of the form diag(v1,...,v D).
The joint distribution has a total of 2Dparameters and represents a set of Dinde-
pendent univariate Gaussian distributions.
Now consider a fully connected graph in which each node has all lower num-
bered nodes as parents. The matrix wijthen has i−1entries on the ithrow and
hence is a lower triangular matrix (with no entries on the leading diagonal). Then

the total number of parameters wijis obtained by taking the number D2of elements
in aD×Dmatrix, subtracting Dto account for the absence of elements on the lead-
ing diagonal, and then dividing by 2because the matrix has elements only below the
diagonal, giving a total of D(D−1)/2. The total number of independent parameters
{wij}and{vi}in the covariance matrix is therefore D(D+1 )/2corresponding to
a general symmetric covariance matrix. Section 2.3
Graphs having some intermediate level of complexity correspond to joint Gaus-

sian distributions with partially constrained covariance matrices. Consider for ex-
ample the graph shown in Figure 8.14, which has a link missing between variables
x1andx3. Using the recursion relations (8.15) and (8.16), we see that the mean and
covariance of the joint distribution are given by Exercise 8.7
µ=(b1,b2+w21b1,b3+w32b2+w32w21b1)T(8.17)
Σ=(v1 w21v1 w32w21v1
w21v1 v2+w2
21v1 w32(v2+w2
21v1)
w32w21v1w32(v2+w2
21v1)v3+w2
32(v2+w2
21v1))
.(8.18)

372 8. GRAPHICAL MODELS
We can readily extend the linear-Gaussian graphical model to the case in which
the nodes of the graph represent multivariate Gaussian variables. In this case, we canwrite the conditional distribution for node iin the form
p(x
i|pai)=N⎛
j∈paiWijxj+bi,Σi⎞
⎠ (8.19)
where now Wijis a matrix (which is nonsquare if xiandxjhave different dimen-
sionalities). Again it is easy to verify that the joint distribution over all variables is
Gaussian.
Note that we have already encountered a speciﬁc example of the linear-Gaussian
relationship when we saw that the conjugate prior for the mean µof a Gaussian Section 2.3.6
variable xis itself a Gaussian distribution over µ. The joint distribution over xand
µis therefore Gaussian. This corresponds to a simple two-node graph in which
the node representing µis the parent of the node representing x. The mean of the
distribution over µis a parameter controlling a prior, and so it can be viewed as a

Gaussian.
hyperparameter. Because the value of this hyperparameter may itself be unknown,
we can again treat it from a Bayesian perspective by introducing a prior over thehyperparameter, sometimes called a hyperprior , which is again given by a Gaussian
distribution. This type of construction can be extended in principle to any level and is
an illustration of a hierarchical Bayesian model , of which we shall encounter further
examples in later chapters.
8.2. Conditional Independence

An important concept for probability distributions over multiple variables is that ofconditional independence (Dawid, 1980). Consider three variables a,b, andc, and
suppose that the conditional distribution of a,g i v e n bandc, is such that it does not
depend on the value of b, so that
p(a|b, c)=p(a|c). (8.20)
We say that ais conditionally independent of bgivenc. This can be expressed in a
slightly different way if we consider the joint distribution of aandbconditioned on
c, which we can write in the form
p(a, b|c)= p(a|b, c)p(b|c)

=p(a|c)p(b|c). (8.21)
where we have used the product rule of probability together with (8.20). Thus we
see that, conditioned on c, the joint distribution of aandbfactorizes into the prod-
uct of the marginal distribution of aand the marginal distribution of b(again both
conditioned on c). This says that the variables aandbare statistically independent,
givenc
. Note that our deﬁnition of conditional independence will require that (8.20),

8.2. Conditional Independence 373
Figure 8.15 The ﬁrst of three examples of graphs over three variables
a,b, and cused to discuss conditional independence
properties of directed graphical models.c
ab
or equivalently (8.21), must hold for every possible value of c, and not just for some
values. We shall sometimes use a shorthand notation for conditional independence
(Dawid, 1979) in which
a⊥⊥b|c (8.22)
denotes that ais conditionally independent of bgivencand is equivalent to (8.20).
Conditional independence properties play an important role in using probabilis-
tic models for pattern recognition by simplifying both the structure of a model and
the computations needed to perform inference and learning under that model. We
shall see examples of this shortly.
If we are given an expression for the joint distribution over a set of variables in
terms of a product of conditional distributions (i.e., the mathematical representation
underlying a directed graph), then we could in principle test whether any poten-

tial conditional independence property holds by repeated application of the sum and
product rules of probability. In practice, such an approach would be very time con-
suming. An important and elegant feature of graphical models is that conditional
independence properties of the joint distribution can be read directly from the graph
without having to perform any analytical manipulations. The general framework
for achieving this is called d-separation , where the ‘d’ stands for ‘directed’ (Pearl,

1988). Here we shall motivate the concept of d-separation and give a general state-
ment of the d-separation criterion. A formal proof can be found in Lauritzen (1996).
8.2.1 Three example graphs
We begin our discussion of the conditional independence properties of directed
graphs by considering three simple examples each involving graphs having just three
nodes. Together, these will motivate and illustrate the key concepts of d-separation.

The ﬁrst of the three examples is shown in Figure 8.15, and the joint distribution
corresponding to this graph is easily written down using the general result (8.5) to
give
p(a, b, c)=p(a|c)p(b|c)p(c). (8.23)
If none of the variables are observed, then we can investigate whether aandbare
independent by marginalizing both sides of (8.23) with respect to cto give
p(a, b)=∑
cp(a|c)p(b|c)p(c). (8.24)
In general, this does not factorize into the product p(a)p(b), and so
a̸⊥⊥b|∅ (8.25)

374 8. GRAPHICAL MODELS
Figure 8.16 As in Figure 8.15 but where we have conditioned on the
value of variable c.c
ab
where ∅denotes the empty set, and the symbol ̸⊥⊥means that the conditional inde-
pendence property does not hold in general. Of course, it may hold for a particular
distribution by virtue of the speciﬁc numerical values associated with the various
conditional probabilities, but it does not follow in general from the structure of the
graph.
Now suppose we condition on the variable c, as represented by the graph of
Figure 8.16. From (8.23), we can easily write down the conditional distribution of a
andb,g i v e n c, in the form
p(a, b|c)=p(a, b, c)
p(c)
=p(a|c)p(b|c)
and so we obtain the conditional independence property
a⊥⊥b|c.
We can provide a simple graphical interpretation of this result by considering
the path from node ato node bviac. The node cis said to be tail-to-tail with re-
spect to this path because the node is connected to the tails of the two arrows, and

p(c)
a⊥⊥b|c.
the presence of such a path connecting nodes aandbcauses these nodes to be de-
pendent. However, when we condition on node c, as in Figure 8.16, the conditioned
node ‘blocks’ the path from atoband causes aandbto become (conditionally)
independent.
We can similarly consider the graph shown in Figure 8.17. The joint distribution
corresponding to this graph is again obtained from our general formula (8.5) to give
p(a, b, c)=p(a)p(c|a)p(b|c). (8.26)
First of all, suppose that none of the variables are observed. Again, we can test to

see ifaandbare independent by marginalizing over cto give
p(a, b)=p(a)∑
cp(c|a)p(b|c)=p(a)p(b|a).
Figure 8.17 The second of our three examples of 3-node
graphs used to motivate the conditional indepen-
dence framework for directed graphical models.ac b

8.2. Conditional Independence 375
Figure 8.18 As in Figure 8.17 but now conditioning on node c.ac b
which in general does not factorize into p(a)p(b), and so
a̸⊥⊥b|∅ (8.27)
as before.
Now suppose we condition on node c, as shown in Figure 8.18. Using Bayes’
theorem, together with (8.26), we obtain
p(c)
=p(a)p(c|a)p(b|c)
p(c)
and so again we obtain the conditional independence property
a⊥⊥b|c.
As before, we can interpret these results graphically. The node cis said to be
head-to-tail with respect to the path from node ato node b. Such a path connects
nodes aandband renders them dependent. If we now observe c, as in Figure 8.18,
then this observation ‘blocks’ the path from atoband so we obtain the conditional
independence property a⊥⊥b|c.
Finally, we consider the third of our 3-node examples, shown by the graph in
Figure 8.19. As we shall see, this has a more subtle behaviour than the two previous
graphs.

graphs.
The joint distribution can again be written down using our general result (8.5) to
give
p(a, b, c)=p(a)p(b)p(c|a, b). (8.28)
Consider ﬁrst the case where none of the variables are observed. Marginalizing both
sides of (8.28) over cwe obtain
p(a, b)=p(a)p(b)
Figure 8.19 The last of our three examples of 3-node graphs used to
explore conditional independence properties in graphi-
cal models. This graph has rather different properties
from the two previous examples.
cab

376 8. GRAPHICAL MODELS
Figure 8.20 As in Figure 8.19 but conditioning on the value of node
c. In this graph, the act of conditioning induces a depen-
dence between aandb.
cab
and so aandbare independent with no variables observed, in contrast to the two
previous examples. We can write this result as
a⊥⊥b|∅. (8.29)
Now suppose we condition on c, as indicated in Figure 8.20. The conditional distri-
bution of aandbis then given by
p(c)
=p(a)p(b)p(c|a, b)
p(c)
which in general does not factorize into the product p(a)p(b), and so
a̸⊥⊥b|c.
Thus our third example has the opposite behaviour from the ﬁrst two. Graphically,
we say that node cishead-to-head with respect to the path from atobbecause it
connects to the heads of the two arrows. When node cis unobserved, it ‘blocks’
the path, and the variables aandbare independent. However, conditioning on c
‘unblocks’ the path and renders aandbdependent.
There is one more subtlety associated with this third example that we need to

a̸⊥⊥b|c.
consider. First we introduce some more terminology. We say that node yis a de-
scendant of node xif there is a path from xtoyin which each step of the path
follows the directions of the arrows. Then it can be shown that a head-to-head path
will become unblocked if either the node, or any of its descendants , is observed. Exercise 8.10
In summary, a tail-to-tail node or a head-to-tail node leaves a path unblocked
unless it is observed in which case it blocks the path. By contrast, a head-to-head

node blocks a path if it is unobserved, but once the node, and/or at least one of its
descendants, is observed the path becomes unblocked.
It is worth spending a moment to understand further the unusual behaviour of the
graph of Figure 8.20. Consider a particular instance of such a graph corresponding
to a problem with three binary random variables relating to the fuel system on a car,
as shown in Figure 8.21. The variables are called B, representing the state of a

battery that is either charged ( B=1)o rﬂ a t( B=0),Frepresenting the state of
the fuel tank that is either full of fuel ( F=1) or empty ( F=0), and G, which is
the state of an electric fuel gauge and which indicates either full ( G=1) or empty

8.2. Conditional Independence 377
GBF
GBF
GBF
Figure 8.21 An example of a 3-node graph used to illustrate the phenomenon of ‘explaining away’. The three
nodes represent the state of the battery ( B), the state of the fuel tank ( F) and the reading on the electric fuel
gauge ( G). See the text for details.
(G=0). The battery is either charged or ﬂat, and independently the fuel tank is
either full or empty, with prior probabilities
p(B=1 ) = 0 .9
p(F=1 ) = 0 .9.
Given the state of the fuel tank and the battery, the fuel gauge reads full with proba-
bilities given by
p(G=1|B=1,F=1 ) = 0 .8
p(G=1|B=1,F=0 ) = 0 .2
p(G=1|B=0,F=1 ) = 0 .2
p(G=1|B=0,F=0 ) = 0 .1
so this is a rather unreliable fuel gauge! All remaining probabilities are determined
by the requirement that probabilities sum to one, and so we have a complete speciﬁ-
cation of the probabilistic model.
Before we observe any data, the prior probability of the fuel tank being empty
isp(F=0 )=0 .1. Now suppose that we observe the fuel gauge and discover that

it reads empty, i.e., G=0, corresponding to the middle graph in Figure 8.21. We
can use Bayes’ theorem to evaluate the posterior probability of the fuel tank being
empty. First we evaluate the denominator for Bayes’ theorem given by
p(G=0 )=∑
B∈{0,1}∑
F∈{0,1}p(G=0|B,F)p(B)p(F)=0.315 (8.30)
and similarly we evaluate
p(G=0|F=0 )=∑
B∈{0,1}p(G=0|B,F=0 )p(B)=0.81 (8.31)
and using these results we have
p(F=0|G=0 )=p(G=0|F=0 )p(F=0 )
p(G=0 )≃0.257 (8.32)

378 8. GRAPHICAL MODELS
and so p(F=0|G=0 )>p(F=0 ) . Thus observing that the gauge reads empty
makes it more likely that the tank is indeed empty, as we would intuitively expect.Next suppose that we also check the state of the battery and ﬁnd that it is ﬂat, i.e.,
B=0. We have now observed the states of both the fuel gauge and the battery, as
shown by the right-hand graph in Figure 8.21. The posterior probability that the fueltank is empty given the observations of both the fuel gauge and the battery state is
then given by
p(F=0|G=0,B=0 )=p(G=0|B=0,F=0 )p(F=0 )
∑
F∈{0,1}p(G=0|B=0,F)p(F)≃0.111 (8.33)
where the prior probability p(B=0 ) has cancelled between numerator and denom-
inator. Thus the probability that the tank is empty has decreased (from 0.257to
0.111) as a result of the observation of the state of the battery. This accords with our
intuition that ﬁnding out that the battery is ﬂat explains away the observation that the

∑
fuel gauge reads empty. We see that the state of the fuel tank and that of the battery
have indeed become dependent on each other as a result of observing the reading
on the fuel gauge. In fact, this would also be the case if, instead of observing the
fuel gauge directly, we observed the state of some descendant of G. Note that the
probability p(F=0|G=0,B=0 )≃0.111is greater than the prior probability
p(F=0 )=0 .1because the observation that the fuel gauge reads zero still provides
some evidence in favour of an empty fuel tank.
8.2.2 D-separation

We now give a general statement of the d-separation property (Pearl, 1988) for
directed graphs. Consider a general directed graph in which A,B, andCare arbi-
trary nonintersecting sets of nodes (whose union may be smaller than the complete
set of nodes in the graph). We wish to ascertain whether a particular conditional
independence statement A⊥⊥B|Cis implied by a given directed acyclic graph. To
do so, we consider all possible paths from any node in Ato any node in B. Any such
path is said to be blocked if it includes a node such that either

(a)the arrows on the path meet either head-to-tail or tail-to-tail at the node, and the
node is in the set C,o r
(b)the arrows meet head-to-head at the node, and neither the node, nor any of its
descendants, is in the set C.
If all paths are blocked, then Ais said to be d-separated from BbyC, and the joint
distribution over all of the variables in the graph will satisfy A⊥⊥B|C.
The concept of d-separation is illustrated in Figure 8.22. In graph (a), the path
fromatobis not blocked by node fbecause it is a tail-to-tail node for this path

and is not observed, nor is it blocked by node ebecause, although the latter is a
head-to-head node, it has a descendant cbecause is in the conditioning set. Thus
the conditional independence statement a⊥⊥b|cdoes notfollow from this graph.
In graph (b), the path from atobis blocked by node fbecause this is a tail-to-tail
node that is observed, and so the conditional independence property a⊥⊥b|fwill

8.2. Conditional Independence 379
Figure 8.22 Illustration of the con-
cept of d-separation. See the text for
details.f
e ba
c
(a)f
e ba
c
(b)
be satisﬁed by any distribution that factorizes according to this graph. Note that this
path is also blocked by node ebecause eis a head-to-head node and neither it nor its
descendant are in the conditioning set.
For the purposes of d-separation, parameters such as αandσ2in Figure 8.5,
indicated by small ﬁlled circles, behave in the same was as observed nodes. How-
ever, there are no marginal distributions associated with such nodes. Consequently
parameter nodes never themselves have parents and so all paths through these nodes
will always be tail-to-tail and hence blocked. Consequently they play no role in
d-separation.
Another example of conditional independence and d-separation is provided by
the concept of i.i.d. (independent identically distributed) data introduced in Sec-
tion 1.2.4. Consider the problem of ﬁnding the posterior distribution for the mean

of a univariate Gaussian distribution. This can be represented by the directed graph Section 2.3
shown in Figure 8.23 in which the joint distribution is deﬁned by a prior p(µ)to-
gether with a set of conditional distributions p(xn|µ)forn=1,...,N . In practice,
we observe D={x1,...,x N}and our goal is to infer µ. Suppose, for a moment,
that we condition on µand consider the joint distribution of the observations. Using
d-separation, we note that there is a unique path from any xito any other xj̸=iand

that this path is tail-to-tail with respect to the observed node µ. Every such path is
blocked and so the observations D={x1,...,x N}are independent given µ, so that
p(D|µ)=N∏
n=1p(xn|µ). (8.34)
Figure 8.23 (a) Directed graph corre-
sponding to the problem
of inferring the mean µof
a univariate Gaussian dis-
tribution from observations
x1,...,x N. (b) The same
graph drawn using the plate
notation.µ
x1 xN
(a)xnN
Nµ
(b)

380 8. GRAPHICAL MODELS
Figure 8.24 A graphical representation of the ‘naive Bayes’
model for classiﬁcation. Conditioned on the
class label z, the components of the observed
vector x=(x1,...,x D)Tare assumed to be
independent.z
x1 xD
However, if we integrate over µ, the observations are in general no longer indepen-
dent
p(D)=∫∞
0p(D|µ)p(µ)dµ̸=N∏
n=1p(xn). (8.35)
Hereµis a latent variable, because its value is not observed.
Another example of a model representing i.i.d. data is the graph in Figure 8.7
corresponding to Bayesian polynomial regression. Here the stochastic nodes corre-
spond to {tn},wandˆt. We see that the node for wis tail-to-tail with respect to
the path from ˆtto any one of the nodes tnand so we have the following conditional
independence property
ˆt⊥⊥tn|w. (8.36)
Thus, conditioned on the polynomial coefﬁcients w, the predictive distribution for
ˆtis independent of the training data {t1,...,t N}. We can therefore ﬁrst use the

training data to determine the posterior distribution over the coefﬁcients wand then
we can discard the training data and use the posterior distribution for wto make
predictions of ˆtfor new input observations ˆx. Section 3.3
A related graphical structure arises in an approach to classiﬁcation called the
naive Bayes model, in which we use conditional independence assumptions to sim-
plify the model structure. Suppose our observed variable consists of a D-dimensional
vectorx=(x1,...,x D)T, and we wish to assign observed values of xto one of K

classes. Using the 1-of- Kencoding scheme, we can represent these classes by a K-
dimensional binary vector z. We can then deﬁne a generative model by introducing
a multinomial prior p(z|µ)over the class labels, where the kthcomponent µkofµ
is the prior probability of class Ck, together with a conditional distribution p(x|z)
for the observed vector x. The key assumption of the naive Bayes model is that,
conditioned on the class z, the distributions of the input variables x1,...,x Dare in-

dependent. The graphical representation of this model is shown in Figure 8.24. We
see that observation of zblocks the path between xiandxjforj̸=i(because such
paths are tail-to-tail at the node z) and so xiandxjare conditionally independent
givenz. If, however, we marginalize out z(so that zis unobserved) the tail-to-tail
path from xitoxjis no longer blocked. This tells us that in general the marginal
density p(x)will not factorize with respect to the components of x. We encountered

a simple application of the naive Bayes model in the context of fusing data from
different sources for medical diagnosis in Section 1.5.
If we are given a labelled training set, comprising inputs {x1,...,xN}together
with their class labels, then we can ﬁt the naive Bayes model to the training data

8.2. Conditional Independence 381
using maximum likelihood assuming that the data are drawn independently from
the model. The solution is obtained by ﬁtting the model for each class separatelyusing the correspondingly labelled data. As an example, suppose that the probability
density within each class is chosen to be Gaussian. In this case, the naive Bayes
assumption then implies that the covariance matrix for each Gaussian is diagonal,and the contours of constant density within each class will be axis-aligned ellipsoids.
The marginal density, however, is given by a superposition of diagonal Gaussians
(with weighting coefﬁcients given by the class priors) and so will no longer factorizewith respect to its components.
The naive Bayes assumption is helpful when the dimensionality Dof the input
space is high, making density estimation in the full D-dimensional space more chal-
lenging. It is also useful if the input vector contains both discrete and continuous

variables, since each can be represented separately using appropriate models (e.g.,Bernoulli distributions for binary observations or Gaussians for real-valued vari-
ables). The conditional independence assumption of this model is clearly a strong
one that may lead to rather poor representations of the class-conditional densities.Nevertheless, even if this assumption is not precisely satisﬁed, the model may still
give good classiﬁcation performance in practice because the decision boundaries can

be insensitive to some of the details in the class-conditional densities, as illustratedin Figure 1.27.
We have seen that a particular directed graph represents a speciﬁc decomposition
of a joint probability distribution into a product of conditional probabilities. Thegraph also expresses a set of conditional independence statements obtained through
the d-separation criterion, and the d-separation theorem is really an expression of the

equivalence of these two properties. In order to make this clear, it is helpful to thinkof a directed graph as a ﬁlter. Suppose we consider a particular joint probability
distribution p(x)over the variables xcorresponding to the (nonobserved) nodes of
the graph. The ﬁlter will allow this distribution to pass through if, and only if, it can
be expressed in terms of the factorization (8.5) implied by the graph. If we present to
the ﬁlter the set of all possible distributions p(x)over the set of variables x, then the

subset of distributions that are passed by the ﬁlter will be denoted DF,f o r directed
factorization . This is illustrated in Figure 8.25. Alternatively, we can use the graph as
a different kind of ﬁlter by ﬁrst listing all of the conditional independence propertiesobtained by applying the d-separation criterion to the graph, and then allowing a
distribution to pass only if it satisﬁes all of these properties. If we present all possible
distributions p(x)to this second kind of ﬁlter, then the d-separation theorem tells us
that the set of distributions that will be allowed through is precisely the set DF.

It should be emphasized that the conditional independence properties obtained
from d-separation apply to any probabilistic model described by that particular di-rected graph. This will be true, for instance, whether the variables are discrete or
continuous or a combination of these. Again, we see that a particular graph is de-
scribing a whole family of probability distributions.
At one extreme we have a fully connected graph that exhibits no conditional in-
dependence properties at all, and which can represent any possible joint probability

distribution over the given variables. The set DF will contain all possible distribu-

382 8. GRAPHICAL MODELS
p(x) DF
Figure 8.25 We can view a graphical model (in this case a directed graph) as a ﬁlter in which a prob-
ability distribution p(x)is allowed through the ﬁlter if, and only if, it satisﬁes the directed
factorization property (8.5). The set of all possible probability distributions p(x)that pass
through the ﬁlter is denoted DF. We can alternatively use the graph to ﬁlter distributions
according to whether they respect all of the conditional independencies implied by the
d-separation properties of the graph. The d-separation theorem says that it is the same
set of distributions DFthat will be allowed through this second kind of ﬁlter.
tionsp(x). At the other extreme, we have the fully disconnected graph, i.e., one
having no links at all. This corresponds to joint distributions which factorize into the
product of the marginal distributions over the variables comprising the nodes of the
graph.
Note that for any given graph, the set of distributions DF will include any dis-

graph.
tributions that have additional independence properties beyond those described by
the graph. For instance, a fully factorized distribution will always be passed through
the ﬁlter implied by any graph over the corresponding set of variables.
We end our discussion of conditional independence properties by exploring the
concept of a Markov blanket orMarkov boundary . Consider a joint distribution
p(x1,...,xD)represented by a directed graph having Dnodes, and consider the

conditional distribution of a particular node with variables xiconditioned on all of
the remaining variables xj̸=i. Using the factorization property (8.5), we can express
this conditional distribution in the form
p(xi|x{j̸=i})=p(x1,...,xD)∫
p(x1,...,xD)dxi
=∏
kp(xk|pak)
∫∏
kp(xk|pak)dxi
in which the integral is replaced by a summation in the case of discrete variables. We
now observe that any factor p(xk|pak)that does not have any functional dependence
onxican be taken outside the integral over xi, and will therefore cancel between

=∏
∫∏
numerator and denominator. The only factors that remain will be the conditional
distribution p(xi|pai)for node xiitself, together with the conditional distributions
for any nodes xksuch that node xiis in the conditioning set of p(xk|pak), in other
words for which xiis a parent of xk. The conditional p(xi|pai)will depend on the
parents of node xi, whereas the conditionals p(xk|pak)will depend on the children

8.3. Markov Random Fields 383
Figure 8.26 The Markov blanket of a node xicomprises the set
of parents, children and co-parents of the node. It
has the property that the conditional distribution of
xi, conditioned on all the remaining variables in the
graph, is dependent only on the variables in the
Markov blanket.xi
ofxias well as on the co-parents , in other words variables corresponding to parents
of node xkother than node xi. The set of nodes comprising the parents, the children
and the co-parents is called the Markov blanket and is illustrated in Figure 8.26. We
can think of the Markov blanket of a node xias being the minimal set of nodes that
isolates xifrom the rest of the graph. Note that it is not sufﬁcient to include only the
parents and children of node xibecause the phenomenon of explaining away means
that observations of the child nodes will not block paths to the co-parents. We must
therefore observe the co-parent nodes also.
8.3. Markov Random Fields

We have seen that directed graphical models specify a factorization of the joint dis-
tribution over a set of variables into a product of local conditional distributions. They
also deﬁne a set of conditional independence properties that must be satisﬁed by any
distribution that factorizes according to the graph. We turn now to the second ma-
jor class of graphical models that are described by undirected graphs and that again
specify both a factorization and a set of conditional independence relations.

AMarkov random ﬁeld , also known as a Markov network or an undirected
graphical model (Kindermann and Snell, 1980), has a set of nodes each of which
corresponds to a variable or group of variables, as well as a set of links each of
which connects a pair of nodes. The links are undirected, that is they do not carry
arrows. In the case of undirected graphs, it is convenient to begin with a discussion
of conditional independence properties.
8.3.1 Conditional independence properties

In the case of directed graphs, we saw that it was possible to test whether a par- Section 8.2
ticular conditional independence property holds by applying a graphical test called
d-separation. This involved testing whether or not the paths connecting two sets of
nodes were ‘blocked’. The deﬁnition of blocked, however, was somewhat subtle
due to the presence of paths having head-to-head nodes. We might ask whether it
is possible to deﬁne an alternative graphical semantics for probability distributions

such that conditional independence is determined by simple graph separation. This
is indeed the case and corresponds to undirected graphical models. By removing the

384 8. GRAPHICAL MODELS
Figure 8.27 An example of an undirected graph in
which every path from any node in set
Ato any node in set Bpasses through
at least one node in set C. Conse-
quently the conditional independence
property A⊥⊥B|Cholds for any
probability distribution described by this
graph.
AC
B
directionality from the links of the graph, the asymmetry between parent and child
nodes is removed, and so the subtleties associated with head-to-head nodes no longer
arise.
Suppose that in an undirected graph we identify three sets of nodes, denoted A,
B, andC, and that we consider the conditional independence property
A⊥⊥B|C. (8.37)
To test whether this property is satisﬁed by a probability distribution deﬁned by a
graph we consider all possible paths that connect nodes in set Ato nodes in set
B. If all such paths pass through one or more nodes in set C, then all such paths are
‘blocked’ and so the conditional independence property holds. However, if there is at

arise.
least one such path that is not blocked, then the property does not necessarily hold, or
more precisely there will exist at least some distributions corresponding to the graph
that do not satisfy this conditional independence relation. This is illustrated with an
example in Figure 8.27. Note that this is exactly the same as the d-separation crite-
rion except that there is no ‘explaining away’ phenomenon. Testing for conditional
independence in undirected graphs is therefore simpler than in directed graphs.

An alternative way to view the conditional independence test is to imagine re-
moving all nodes in set Cfrom the graph together with any links that connect to
those nodes. We then ask if there exists a path that connects any node in Ato any
node in B. If there are no such paths, then the conditional independence property
must hold.
The Markov blanket for an undirected graph takes a particularly simple form,
because a node will be conditionally independent of all other nodes conditioned only
on the neighbouring nodes, as illustrated in Figure 8.28.
8.3.2 Factorization properties

must hold.
We now seek a factorization rule for undirected graphs that will correspond to
the above conditional independence test. Again, this will involve expressing the joint
distribution p(x)as a product of functions deﬁned over sets of variables that are local
to the graph. We therefore need to decide what is the appropriate notion of locality
in this case.

8.3. Markov Random Fields 385
Figure 8.28 For an undirected graph, the Markov blanket of a node
xiconsists of the set of neighbouring nodes. It has the
property that the conditional distribution of xi, conditioned
on all the remaining variables in the graph, is dependent
only on the variables in the Markov blanket.
If we consider two nodes xiandxjthat are not connected by a link, then these
variables must be conditionally independent given all other nodes in the graph. This
follows from the fact that there is no direct path between the two nodes, and all other
paths pass through nodes that are observed, and hence those paths are blocked. This
conditional independence property can be expressed as
p(xi,xj|x\{i,j})=p(xi|x\{i,j})p(xj|x\{i,j}) (8.38)
wherex\{i,j}denotes the set xof all variables with xiandxjremoved. The factor-
ization of the joint distribution must therefore be such that xiandxjdo not appear
in the same factor in order for the conditional independence property to hold for all

possible distributions belonging to the graph.
This leads us to consider a graphical concept called a clique , which is deﬁned
as a subset of the nodes in a graph such that there exists a link between all pairs of
nodes in the subset. In other words, the set of nodes in a clique is fully connected.
Furthermore, a maximal clique is a clique such that it is not possible to include any
other nodes from the graph in the set without it ceasing to be a clique. These concepts
are illustrated by the undirected graph over four variables shown in Figure 8.29. This

graph has ﬁve cliques of two nodes given by {x1,x2},{x2,x3},{x3,x4},{x4,x2},
and{x1,x3}, as well as two maximal cliques given by {x1,x2,x3}and{x2,x3,x4}.
The set {x1,x2,x3,x4}is not a clique because of the missing link from x1tox4.
We can therefore deﬁne the factors in the decomposition of the joint distribution
to be functions of the variables in the cliques. In fact, we can consider functions
of the maximal cliques, without loss of generality, because other cliques must be
subsets of maximal cliques. Thus, if {x1,x2,x3}is a maximal clique and we deﬁne

an arbitrary function over this clique, then including another factor deﬁned over a
subset of these variables would be redundant.
Let us denote a clique by Cand the set of variables in that clique by xC. Then
Figure 8.29 A four-node undirected graph showing a clique (outlined in
green) and a maximal clique (outlined in blue).x1
x2
x3
x4

386 8. GRAPHICAL MODELS
the joint distribution is written as a product of potential functions ψC(xC)over the
maximal cliques of the graph
p(x)=1
Z∏
CψC(xC). (8.39)
Here the quantity Z, sometimes called the partition function , is a normalization con-
stant and is given by
Z=∑
x∏
CψC(xC) (8.40)
which ensures that the distribution p(x)given by (8.39) is correctly normalized.
By considering only potential functions which satisfy ψC(xC)⩾0we ensure that
p(x)⩾0. In (8.40) we have assumed that xcomprises discrete variables, but the
framework is equally applicable to continuous variables, or a combination of the two,
in which the summation is replaced by the appropriate combination of summation
and integration.
Note that we do not restrict the choice of potential functions to those that have a
speciﬁc probabilistic interpretation as marginal or conditional distributions. This is
in contrast to directed graphs in which each factor represents the conditional distribu-

tion of the corresponding variable, conditioned on the state of its parents. However,
in special cases, for instance where the undirected graph is constructed by startingwith a directed graph, the potential functions may indeed have such an interpretation,
as we shall see shortly.
One consequence of the generality of the potential functions ψ
C(xC)is that
their product will in general not be correctly normalized. We therefore have to in-
troduce an explicit normalization factor given by (8.40). Recall that for directed

graphs, the joint distribution was automatically normalized as a consequence of thenormalization of each of the conditional distributions in the factorization.
The presence of this normalization constant is one of the major limitations of
undirected graphs. If we have a model with Mdiscrete nodes each having Kstates,
then the evaluation of the normalization term involves summing over K
Mstates and
so (in the worst case) is exponential in the size of the model. The partition function

is needed for parameter learning because it will be a function of any parameters thatgovern the potential functions ψ
C(xC). However, for evaluation of local conditional
distributions, the partition function is not needed because a conditional is the ratio
of two marginals, and the partition function cancels between numerator and denom-inator when evaluating this ratio. Similarly, for evaluating local marginal probabil-
ities we can work with the unnormalized joint distribution and then normalize the

marginals explicitly at the end. Provided the marginals only involves a small number
of variables, the evaluation of their normalization coefﬁcient will be feasible.
So far, we have discussed the notion of conditional independence based on sim-
ple graph separation and we have proposed a factorization of the joint distribution
that is intended to correspond to this conditional independence structure. However,

we have not made any formal connection between conditional independence andfactorization for undirected graphs. To do so we need to restrict attention to poten-
tial functions ψ
C(xC)that are strictly positive (i.e., never zero or negative for any

8.3. Markov Random Fields 387
choice of xC). Given this restriction, we can make a precise relationship between
factorization and conditional independence.
To do this we again return to the concept of a graphical model as a ﬁlter, corre-
sponding to Figure 8.25. Consider the set of all possible distributions deﬁned over
a ﬁxed set of variables corresponding to the nodes of a particular undirected graph.We can deﬁne UIto be the set of such distributions that are consistent with the set
of conditional independence statements that can be read from the graph using graph
separation. Similarly, we can deﬁne UFto be the set of such distributions that can
be expressed as a factorization of the form (8.39) with respect to the maximal cliques
of the graph. The Hammersley-Clifford theorem (Clifford, 1990) states that the sets
UIandUFare identical.
Because we are restricted to potential functions which are strictly positive it is
convenient to express them as exponentials, so that
ψ
C(xC) = exp {−E(xC)} (8.41)

ψ
where E(xC)is called an energy function , and the exponential representation is
called the Boltzmann distribution . The joint distribution is deﬁned as the product of
potentials, and so the total energy is obtained by adding the energies of each of the
maximal cliques.
In contrast to the factors in the joint distribution for a directed graph, the po-
tentials in an undirected graph do not have a speciﬁc probabilistic interpretation.

ψ
Although this gives greater ﬂexibility in choosing the potential functions, becausethere is no normalization constraint, it does raise the question of how to motivate a
choice of potential function for a particular application. This can be done by view-
ing the potential function as expressing which conﬁgurations of the local variables
are preferred to others. Global conﬁgurations that have a relatively high probability

are those that ﬁnd a good balance in satisfying the (possibly conﬂicting) inﬂuencesof the clique potentials. We turn now to a speciﬁc example to illustrate the use of
undirected graphs.
8.3.3 Illustration: Image de-noising
We can illustrate the application of undirected graphs using an example of noise
removal from a binary image (Besag, 1974; Geman and Geman, 1984; Besag, 1986).
Although a very simple example, this is typical of more sophisticated applications.

Let the observed noisy image be described by an array of binary pixel values yi∈
{−1,+1}, where the index i=1,...,D runs over all pixels. We shall suppose
that the image is obtained by taking an unknown noise-free image, described by
binary pixel values xi∈{ −1,+1}and randomly ﬂipping the sign of pixels with
some small probability. An example binary image, together with a noise corrupted
image obtained by ﬂipping the sign of the pixels with probability 10%, is shown inFigure 8.30. Given the noisy image, our goal is to recover the original noise-free

image.
Because the noise level is small, we know that there will be a strong correlation
between x
iandyi. We also know that neighbouring pixels xiandxjin an image
are strongly correlated. This prior knowledge can be captured using the Markov

388 8. GRAPHICAL MODELS
Figure 8.30 Illustration of image de-noising using a Markov random ﬁeld. The top row shows the original
binary image on the left and the corrupted image after randomly changing 10% of the pixels on the right. Thebottom row shows the restored images obtained using iterated conditional models (ICM) on the left and usingthe graph-cut algorithm on the right. ICM produces an image where 96% of the pixels agree with the originalimage, whereas the corresponding number for graph-cut is 99%.
random ﬁeld model whose undirected graph is shown in Figure 8.31. This graph has
two types of cliques, each of which contains two variables. The cliques of the form{x
i,yi}have an associated energy function that expresses the correlation between
these variables. We choose a very simple energy function for these cliques of the
form−ηxiyiwhere ηis a positive constant. This has the desired effect of giving a
lower energy (thus encouraging a higher probability) when xiandyihave the same

sign and a higher energy when they have the opposite sign.
The remaining cliques comprise pairs of variables {xi,xj}where iandjare
indices of neighbouring pixels. Again, we want the energy to be lower when the
pixels have the same sign than when they have the opposite sign, and so we choose
an energy given by −βxixjwhere βis a positive constant.
Because a potential function is an arbitrary, nonnegative function over a maximal
clique, we can multiply it by any nonnegative functions of subsets of the clique, or

8.3. Markov Random Fields 389
Figure 8.31 An undirected graphical model representing a
Markov random ﬁeld for image de-noising, in
which xiis a binary variable denoting the state
of pixel iin the unknown noise-free image, and yi
denotes the corresponding value of pixel iin the
observed noisy image.
xiyi
equivalently we can add the corresponding energies. In this example, this allows us
to add an extra term hxifor each pixel iin the noise-free image. Such a term has
the effect of biasing the model towards pixel values that have one particular sign in
preference to the other.
The complete energy function for the model then takes the form
E(x,y)=h∑
ixi−β∑
{i,j}xixj−η∑
ixiyi (8.42)
which deﬁnes a joint distribution over xandygiven by
p(x,y)=1
Zexp{−E(x,y)}. (8.43)
We now ﬁx the elements of yto the observed values given by the pixels of the
noisy image, which implicitly deﬁnes a conditional distribution p(x|y)over noise-
free images. This is an example of the Ising model , which has been widely studied in

E(x,y)=h∑
ixi−β∑
p(x,y)=1
statistical physics. For the purposes of image restoration, we wish to ﬁnd an image x
having a high probability (ideally the maximum probability). To do this we shall use
a simple iterative technique called iterated conditional modes ,o r ICM (Kittler and
F¨oglein, 1984), which is simply an application of coordinate-wise gradient ascent.
The idea is ﬁrst to initialize the variables {xi}, which we do by simply setting xi=
yifor all i. Then we take one node xjat a time and we evaluate the total energy

for the two possible states xj=+ 1 andxj=−1, keeping all other node variables
ﬁxed, and set xjto whichever state has the lower energy. This will either leave
the probability unchanged, if xjis unchanged, or will increase it. Because only
one variable is changed, this is a simple local computation that can be performed Exercise 8.13
efﬁciently. We then repeat the update for another site, and so on, until some suitable
stopping criterion is satisﬁed. The nodes may be updated in a systematic way, for

instance by repeatedly raster scanning through the image, or by choosing nodes at
random.
If we have a sequence of updates in which every site is visited at least once,
and in which no changes to the variables are made, then by deﬁnition the algorithm

390 8. GRAPHICAL MODELS
Figure 8.32 (a) Example of a directed
graph. (b) The equivalent undirected
graph.(a)x1 x2 xN−1 xN
(b)x1 x2 xN−1 xN
will have converged to a local maximum of the probability. This need not, however,
correspond to the global maximum.
For the purposes of this simple illustration, we have ﬁxed the parameters to be
β=1.0,η=2.1andh=0. Note that leaving h=0 simply means that the prior
probabilities of the two states of xiare equal. Starting with the observed noisy image
as the initial conﬁguration, we run ICM until convergence, leading to the de-noised
image shown in the lower left panel of Figure 8.30. Note that if we set β=0,
which effectively removes the links between neighbouring pixels, then the global
most probable solution is given by xi=yifor all i, corresponding to the observed
noisy image. Exercise 8.14
Later we shall discuss a more effective algorithm for ﬁnding high probability so-
lutions called the max-product algorithm, which typically leads to better solutions, Section 8.4

although this is still not guaranteed to ﬁnd the global maximum of the posterior dis-
tribution. However, for certain classes of model, including the one given by (8.42),
there exist efﬁcient algorithms based on graph cuts that are guaranteed to ﬁnd the
global maximum (Greig et al. , 1989; Boykov et al. , 2001; Kolmogorov and Zabih,
2004). The lower right panel of Figure 8.30 shows the result of applying a graph-cut
algorithm to the de-noising problem.
8.3.4 Relation to directed graphs
We have introduced two graphical frameworks for representing probability dis-

tributions, corresponding to directed and undirected graphs, and it is instructive to
discuss the relation between these. Consider ﬁrst the problem of taking a model that
is speciﬁed using a directed graph and trying to convert it to an undirected graph. In
some cases this is straightforward, as in the simple example in Figure 8.32. Here the
joint distribution for the directed graph is given as a product of conditionals in the
form
p(x)=p(x1)p(x2|x1)p(x3|x2)···p(xN|xN−1). (8.44)

form
Now let us convert this to an undirected graph representation, as shown in Fig-
ure 8.32. In the undirected graph, the maximal cliques are simply the pairs of neigh-
bouring nodes, and so from (8.39) we wish to write the joint distribution in the form
p(x)=1
Zψ1,2(x1,x2)ψ2,3(x2,x3)···ψN−1,N(xN−1,xN). (8.45)

8.3. Markov Random Fields 391
Figure 8.33 Example of a simple
directed graph (a) and the corre-
sponding moral graph (b).x1 x3
x4x2
(a)x1 x3
x4x2
(b)
This is easily done by identifying
ψ1,2(x1,x2)= p(x1)p(x2|x1)
ψ2,3(x2,x3)= p(x3|x2)
...
ψN−1,N(xN−1,xN)= p(xN|xN−1)
where we have absorbed the marginal p(x1)for the ﬁrst node into the ﬁrst potential
function. Note that in this case, the partition function Z=1.
Let us consider how to generalize this construction, so that we can convert any
distribution speciﬁed by a factorization over a directed graph into one speciﬁed by a
factorization over an undirected graph. This can be achieved if the clique potentials
of the undirected graph are given by the conditional distributions of the directed
graph. In order for this to be valid, we must ensure that the set of variables that
appears in each of the conditional distributions is a member of at least one clique of
the undirected graph. For nodes on the directed graph having just one parent, this is

achieved simply by replacing the directed link with an undirected link. However, for
nodes in the directed graph having more than one parent, this is not sufﬁcient. These
are nodes that have ‘head-to-head’ paths encountered in our discussion of conditional
independence. Consider a simple directed graph over 4 nodes shown in Figure 8.33.
The joint distribution for the directed graph takes the form
p(x)=p(x1)p(x2)p(x3)p(x4|x1,x2,x3). (8.46)

We see that the factor p(x4|x1,x2,x3)involves the four variables x1,x2,x3, and
x4, and so these must all belong to a single clique if this conditional distribution is
to be absorbed into a clique potential. To ensure this, we add extra links between
all pairs of parents of the node x4. Anachronistically, this process of ‘marrying
the parents’ has become known as moralization , and the resulting undirected graph,
after dropping the arrows, is called the moral graph . It is important to observe that

the moral graph in this example is fully connected and so exhibits no conditional
independence properties, in contrast to the original directed graph.
Thus in general to convert a directed graph into an undirected graph, we ﬁrst add
additional undirected links between all pairs of parents for each node in the graph and

392 8. GRAPHICAL MODELS
then drop the arrows on the original links to give the moral graph. Then we initialize
all of the clique potentials of the moral graph to 1. We then take each conditional
distribution factor in the original directed graph and multiply it into one of the clique
potentials. There will always exist at least one maximal clique that contains all of
the variables in the factor as a result of the moralization step. Note that in all cases
the partition function is given by Z=1.
The process of converting a directed graph into an undirected graph plays an
important role in exact inference techniques such as the junction tree algorithm . Section 8.4
Converting from an undirected to a directed representation is much less common
and in general presents problems due to the normalization constraints.
We saw that in going from a directed to an undirected representation we had to
discard some conditional independence properties from the graph. Of course, we

could always trivially convert any distribution over a directed graph into one over an
undirected graph by simply using a fully connected undirected graph. This would,
however, discard all conditional independence properties and so would be vacuous.
The process of moralization adds the fewest extra links and so retains the maximum
number of independence properties.
We have seen that the procedure for determining the conditional independence
properties is different between directed and undirected graphs. It turns out that the

two types of graph can express different conditional independence properties, and
it is worth exploring this issue in more detail. To do so, we return to the view of
a speciﬁc (directed or undirected) graph as a ﬁlter, so that the set of all possible Section 8.2
distributions over the given variables could be reduced to a subset that respects the
conditional independencies implied by the graph. A graph is said to be a D map
(for ‘dependency map’) of a distribution if every conditional independence statement

satisﬁed by the distribution is reﬂected in the graph. Thus a completely disconnected
graph (no links) will be a trivial D map for any distribution.
Alternatively, we can consider a speciﬁc distribution and ask which graphs have
the appropriate conditional independence properties. If every conditional indepen-
dence statement implied by a graph is satisﬁed by a speciﬁc distribution, then the
graph is said to be an I map (for ‘independence map’) of that distribution. Clearly a
fully connected graph will be a trivial I map for any distribution.

If it is the case that every conditional independence property of the distribution
is reﬂected in the graph, and vice versa, then the graph is said to be a perfect map for
Figure 8.34 Venn diagram illustrating the set of all distributions
P over a given set of variables, together with the
set of distributions D that can be represented as a
perfect map using a directed graph, and the set U
that can be represented as a perfect map using an
undirected graph.
PU D

8.4. Inference in Graphical Models 393
Figure 8.35 A directed graph whose conditional independence
properties cannot be expressed using an undirected
graph over the same three variables.
CAB
that distribution. A perfect map is therefore both an I map and a D map.
Consider the set of distributions such that for each distribution there exists a
directed graph that is a perfect map. This set is distinct from the set of distributions
such that for each distribution there exists an undirected graph that is a perfect map.
In addition there are distributions for which neither directed nor undirected graphs
offer a perfect map. This is illustrated as a Venn diagram in Figure 8.34.
Figure 8.35 shows an example of a directed graph that is a perfect map for
a distribution satisfying the conditional independence properties A⊥⊥B|∅and
A̸⊥⊥B|C. There is no corresponding undirected graph over the same three vari-
ables that is a perfect map.
Conversely, consider the undirected graph over four variables shown in Fig-

ure 8.36. This graph exhibits the properties A̸⊥⊥B|∅,C⊥⊥D|A∪Band
A⊥⊥B|C∪D. There is no directed graph over four variables that implies the same
set of conditional independence properties.
The graphical framework can be extended in a consistent way to graphs that
include both directed and undirected links. These are called chain graphs (Lauritzen
and Wermuth, 1989; Frydenberg, 1990), and contain the directed and undirected
graphs considered so far as special cases. Although such graphs can represent a

broader class of distributions than either directed or undirected alone, there remain
distributions for which even a chain graph cannot provide a perfect map. Chain
graphs are not discussed further in this book.
Figure 8.36 An undirected graph whose conditional independence
properties cannot be expressed in terms of a directed
graph over the same variables.
AC
B
D
8.4. Inference in Graphical Models
We turn now to the problem of inference in graphical models, in which some of

AC
B
D
the nodes in a graph are clamped to observed values, and we wish to compute the
posterior distributions of one or more subsets of other nodes. As we shall see, we
can exploit the graphical structure both to ﬁnd efﬁcient algorithms for inference, and

394 8. GRAPHICAL MODELS
Figure 8.37 A graphical representation of Bayes’ theorem.
See the text for details.x
yx
yx
y
(a) (b) (c)
to make the structure of those algorithms transparent. Speciﬁcally, we shall see that
many algorithms can be expressed in terms of the propagation of local messages
around the graph. In this section, we shall focus primarily on techniques for exact
inference, and in Chapter 10 we shall consider a number of approximate inference
To start with, let us consider the graphical interpretation of Bayes’ theorem.
Suppose we decompose the joint distribution p(x, y)over two variables xandyinto
a product of factors in the form p(x, y)=p(x)p(y|x). This can be represented by
the directed graph shown in Figure 8.37(a). Now suppose we observe the value of
y, as indicated by the shaded node in Figure 8.37(b). We can view the marginal
distribution p(x)as a prior over the latent variable x, and our goal is to infer the

corresponding posterior distribution over x. Using the sum and product rules of
probability we can evaluate
p(y)=∑
x′p(y|x′)p(x′) (8.47)
which can then be used in Bayes’ theorem to calculate
p(x|y)=p(y|x)p(x)
p(y). (8.48)
Thus the joint distribution is now expressed in terms of p(y)andp(x|y). From a
graphical perspective, the joint distribution p(x, y)is now represented by the graph
shown in Figure 8.37(c), in which the direction of the arrow is reversed. This is the

p(y)=∑
simplest example of an inference problem for a graphical model.
8.4.1 Inference on a chain
Now consider a more complex problem involving the chain of nodes of the form
shown in Figure 8.32. This example will lay the foundation for a discussion of exact
inference in more general graphs later in this section.
Speciﬁcally, we shall consider the undirected graph in Figure 8.32(b). We have
already seen that the directed chain can be transformed into an equivalent undirected
chain. Because the directed graph does not have any nodes with more than one

parent, this does not require the addition of any extra links, and the directed and
undirected versions of this graph express exactly the same set of conditional inde-
pendence statements.

8.4. Inference in Graphical Models 395
The joint distribution for this graph takes the form
p(x)=1
Zψ1,2(x1,x2)ψ2,3(x2,x3)···ψN−1,N(xN−1,xN). (8.49)
We shall consider the speciﬁc case in which the Nnodes represent discrete vari-
ables each having Kstates, in which case each potential function ψn−1,n(xn−1,xn)
comprises an K×Ktable, and so the joint distribution has (N−1)K2parameters.
Let us consider the inference problem of ﬁnding the marginal distribution p(xn)
for a speciﬁc node xnthat is part way along the chain. Note that, for the moment,
there are no observed nodes. By deﬁnition, the required marginal is obtained by
summing the joint distribution over all variables except xn, so that
p(xn)=∑
x1···∑
xn−1∑
xn+1···∑
xNp(x). (8.50)
In a naive implementation, we would ﬁrst evaluate the joint distribution and
then perform the summations explicitly. The joint distribution can be represented as
a set of numbers, one for each possible value for x. Because there are Nvariables

p(xn)=∑
x1···∑
xn−1∑
xn+1···∑
each with Kstates, there are KNvalues for xand so evaluation and storage of the
joint distribution, as well as marginalization to obtain p(xn), all involve storage and
computation that scale exponentially with the length Nof the chain.
We can, however, obtain a much more efﬁcient algorithm by exploiting the con-
ditional independence properties of the graphical model. If we substitute the factor-
ized expression (8.49) for the joint distribution into (8.50), then we can rearrange theorder of the summations and the multiplications to allow the required marginal to be

evaluated much more efﬁciently. Consider for instance the summation over x
N. The
potential ψN−1,N(xN−1,xN)is the only one that depends on xN, and so we can
perform the summation∑
xNψN−1,N(xN−1,xN) (8.51)
ﬁrst to give a function of xN−1. We can then use this to perform the summation
overxN−1, which will involve only this new function together with the potential
ψN−2,N−1(xN−2,xN−1), because this is the only other place that xN−1appears.
Similarly, the summation over x1involves only the potential ψ1,2(x1,x2)and so

N. The
can be performed separately to give a function of x2, and so on. Because each
summation effectively removes a variable from the distribution, this can be viewed
as the removal of a node from the graph.
If we group the potentials and summations together in this way, we can express

396 8. GRAPHICAL MODELS
the desired marginal in the form
p(xn)=1
Z⎡
⎣∑
xn−1ψn−1,n(xn−1,xn)···[∑
x2ψ2,3(x2,x3)[∑
x1ψ1,2(x1,x2)]]
···⎤
⎦
  
µα(xn)⎡
⎣∑
xn+1ψn,n+1(xn,xn+1)···[∑
xNψN−1,N(xN−1,xN)]
···⎤
⎦
  
µβ(xn). (8.52)
The reader is encouraged to study this re-ordering carefully as the underlying idea
forms the basis for the later discussion of the general sum-product algorithm. Here
the key concept that we are exploiting is that multiplication is distributive over addi-
tion, so that
ab+ac=a(b+c) (8.53)
in which the left-hand side involves three arithmetic operations whereas the right-
hand side reduces this to two operations.
Let us work out the computational cost of evaluating the required marginal using
this re-ordered expression. We have to perform N−1summations each of which is
overKstates and each of which involves a function of two variables. For instance,
the summation over x1involves only the function ψ1,2(x1,x2), which is a table of

K×Knumbers. We have to sum this table over x1for each value of x2and so this
hasO(K2)cost. The resulting vector of Knumbers is multiplied by the matrix of
numbers ψ2,3(x2,x3)and so is again O(K2). Because there are N−1summations
and multiplications of this kind, the total cost of evaluating the marginal p(xn)is
O(NK2). This is linear in the length of the chain, in contrast to the exponential cost
of a naive approach. We have therefore been able to exploit the many conditional

independence properties of this simple graph in order to obtain an efﬁcient calcula-
tion. If the graph had been fully connected, there would have been no conditional
independence properties, and we would have been forced to work directly with the
full joint distribution.
We now give a powerful interpretation of this calculation in terms of the passing
of local messages around on the graph. From (8.52) we see that the expression for the
marginal p(xn)decomposes into the product of two factors times the normalization
constant
p(xn)=1

constant
p(xn)=1
Zµα(xn)µβ(xn). (8.54)
We shall interpret µα(xn)as a message passed forwards along the chain from node
xn−1to node xn. Similarly, µβ(xn)can be viewed as a message passed backwards

8.4. Inference in Graphical Models 397
Figure 8.38 The marginal distribution
p(xn)for a node xnalong the chain is ob-
tained by multiplying the two messages
µα(xn)andµβ(xn), and then normaliz-
ing. These messages can themselves
be evaluated recursively by passing mes-
sages from both ends of the chain to-
wards node xn.x1 xn−1 xn xn+1 xNµα(xn−1)µα(xn)µβ(xn)µβ(xn+1)
along the chain to node xnfrom node xn+1. Note that each of the messages com-
prises a set of Kvalues, one for each choice of xn, and so the product of two mes-
sages should be interpreted as the point-wise multiplication of the elements of the
two messages to give another set of Kvalues.
The message µα(xn)can be evaluated recursively because
µα(xn)=∑
xn−1ψn−1,n(xn−1,xn)⎡
⎣∑
xn−2···⎤
⎦
=∑
xn−1ψn−1,n(xn−1,xn)µα(xn−1). (8.55)
We therefore ﬁrst evaluate
µα(x2)=∑
x1ψ1,2(x1,x2) (8.56)
and then apply (8.55) repeatedly until we reach the desired node. Note carefully the
structure of the message passing equation. The outgoing message µα(xn)in (8.55)

µα(xn)=∑
⎣∑
xn−2···⎤
⎦
=∑
µα(x2)=∑
is obtained by multiplying the incoming message µα(xn−1)by the local potential
involving the node variable and the outgoing variable and then summing over the
node variable.
Similarly, the message µβ(xn)can be evaluated recursively by starting with
nodexNand using
µβ(xn)=∑
xn+1ψn+1,n(xn+1,xn)⎡
⎣∑
xn+2···⎤
⎦
=∑
xn+1ψn+1,n(xn+1,xn)µβ(xn+1). (8.57)
This recursive message passing is illustrated in Figure 8.38. The normalization con-
stantZis easily evaluated by summing the right-hand side of (8.54) over all states

µβ(xn)=∑
⎣∑
xn+2···⎤
⎦
=∑
ofxn, an operation that requires only O(K)computation.
Graphs of the form shown in Figure 8.38 are called Markov chains , and the
corresponding message passing equations represent an example of the Chapman-
Kolmogorov equations for Markov processes (Papoulis, 1984).

398 8. GRAPHICAL MODELS
Now suppose we wish to evaluate the marginals p(xn)for every node n∈
{1,...,N }in the chain. Simply applying the above procedure separately for each
node will have computational cost that is O(N2M2). However, such an approach
would be very wasteful of computation. For instance, to ﬁnd p(x1)we need to prop-
agate a message µβ(·)from node xNback to node x2. Similarly, to evaluate p(x2)
we need to propagate a messages µβ(·)from node xNback to node x3. This will
involve much duplicated computation because most of the messages will be identical
in the two cases.
Suppose instead we ﬁrst launch a message µβ(xN−1)starting from node xN
and propagate corresponding messages all the way back to node x1, and suppose we
similarly launch a message µα(x2)starting from node x1and propagate the corre-
sponding messages all the way forward to node xN. Provided we store all of the

intermediate messages along the way, then any node can evaluate its marginal sim-ply by applying (8.54). The computational cost is only twice that for ﬁnding the
marginal of a single node, rather than Ntimes as much. Observe that a message
has passed once in each direction across each link in the graph. Note also that thenormalization constant Zneed be evaluated only once, using any convenient node.
If some of the nodes in the graph are observed, then the corresponding variables

are simply clamped to their observed values and there is no summation. To seethis, note that the effect of clamping a variable x
nto an observed value ˆxncan
be expressed by multiplying the joint distribution by (one or more copies of) an
additional function I(xn,ˆxn), which takes the value 1whenxn=ˆxnand the value
0otherwise. One such function can then be absorbed into each of the potentials that
contain xn. Summations over xnthen contain only one term in which xn=ˆxn.

Now suppose we wish to calculate the joint distribution p(xn−1,xn)for two
neighbouring nodes on the chain. This is similar to the evaluation of the marginal
for a single node, except that there are now two variables that are not summed out.
A few moments thought will show that the required joint distribution can be written Exercise 8.15
in the form
p(xn−1,xn)=1
Zµα(xn−1)ψn−1,n(xn−1,xn)µβ(xn). (8.58)
Thus we can obtain the joint distributions over all of the sets of variables in each

in the form
of the potentials directly once we have completed the message passing required to
obtain the marginals.
This is a useful result because in practice we may wish to use parametric forms
for the clique potentials, or equivalently for the conditional distributions if we started
from a directed graph. In order to learn the parameters of these potentials in situa-
tions where not all of the variables are observed, we can employ the EM algorithm , Chapter 9

and it turns out that the local joint distributions of the cliques, conditioned on anyobserved data, is precisely what is needed in the E step. We shall consider some
examples of this in detail in Chapter 13.
8.4.2 Trees
We have seen that exact inference on a graph comprising a chain of nodes can be
performed efﬁciently in time that is linear in the number of nodes, using an algorithm

8.4. Inference in Graphical Models 399
Figure 8.39 Examples of tree-
structured graphs, showing (a) an
undirected tree, (b) a directed tree,
and (c) a directed polytree.
(a) (b) (c)
that can be interpreted in terms of messages passed along the chain. More generally,
inference can be performed efﬁciently using local message passing on a broaderclass of graphs called trees . In particular, we shall shortly generalize the message
passing formalism derived above for chains to give the sum-product algorithm, which
provides an efﬁcient framework for exact inference in tree-structured graphs.
In the case of an undirected graph, a tree is deﬁned as a graph in which there
is one, and only one, path between any pair of nodes. Such graphs therefore do nothave loops. In the case of directed graphs, a tree is deﬁned such that there is a single
node, called the root, which has no parents, and all other nodes have one parent. If

we convert a directed tree into an undirected graph, we see that the moralization stepwill not add any links as all nodes have at most one parent, and as a consequence the
corresponding moralized graph will be an undirected tree. Examples of undirected
and directed trees are shown in Figure 8.39(a) and 8.39(b). Note that a distributionrepresented as a directed tree can easily be converted into one represented by an
undirected tree, and vice versa. Exercise 8.18

If there are nodes in a directed graph that have more than one parent, but there is
still only one path (ignoring the direction of the arrows) between any two nodes, then
the graph is a called a polytree , as illustrated in Figure 8.39(c). Such a graph will
have more than one node with the property of having no parents, and furthermore,the corresponding moralized undirected graph will have loops.
8.4.3 Factor graphs
The sum-product algorithm that we derive in the next section is applicable to

undirected and directed trees and to polytrees. It can be cast in a particularly simple
and general form if we ﬁrst introduce a new graphical construction called a factor
graph (Frey, 1998; Kschischnang et al. , 2001).
Both directed and undirected graphs allow a global function of several vari-
ables to be expressed as a product of factors over subsets of those variables. Factorgraphs make this decomposition explicit by introducing additional nodes for the fac-

tors themselves in addition to the nodes representing the variables. They also allow
us to be more explicit about the details of the factorization, as we shall see.
Let us write the joint distribution over a set of variables in the form of a product
of factors
p(x)=∏
sfs(xs) (8.59)
wherexsdenotes a subset of the variables. For convenience, we shall denote the

400 8. GRAPHICAL MODELS
Figure 8.40 Example of a factor graph, which corresponds
to the factorization (8.60).x1 x2 x3
fa fb fc fd
individual variables by xi, however, as in earlier discussions, these can comprise
groups of variables (such as vectors or matrices). Each factor fsis a function of a
corresponding set of variables xs.
Directed graphs, whose factorization is deﬁned by (8.5), represent special cases
of (8.59) in which the factors fs(xs)are local conditional distributions. Similarly,
undirected graphs, given by (8.39), are a special case in which the factors are po-
tential functions over the maximal cliques (the normalizing coefﬁcient 1/Zcan be
viewed as a factor deﬁned over the empty set of variables).
In a factor graph, there is a node (depicted as usual by a circle) for every variable
in the distribution, as was the case for directed and undirected graphs. There are also
additional nodes (depicted by small squares) for each factor fs(xs)in the joint dis-

tribution. Finally, there are undirected links connecting each factor node to all of the
variables nodes on which that factor depends. Consider, for example, a distribution
that is expressed in terms of the factorization
p(x)=fa(x1,x2)fb(x1,x2)fc(x2,x3)fd(x3). (8.60)
This can be expressed by the factor graph shown in Figure 8.40. Note that there are
two factors fa(x1,x2)andfb(x1,x2)that are deﬁned over the same set of variables.
In an undirected graph, the product of two such factors would simply be lumped

together into the same clique potential. Similarly, fc(x2,x3)andfd(x3)could be
combined into a single potential over x2andx3. The factor graph, however, keeps
such factors explicit and so is able to convey more detailed information about the
underlying factorization.
x1 x2
x3
(a)x1 x2
x3f
(b)x1 x2
x3fa
fb
(c)
Figure 8.41 (a) An undirected graph with a single clique potential ψ(x1,x2,x3). (b) A factor graph with factor

x1 x2
x3
(a)x1 x2
x3f
(b)x1 x2
x3fa
fb
(c)
f(x1,x2,x3)=ψ(x1,x2,x3)representing the same distribution as the undirected graph. (c) A different factor
graph representing the same distribution, whose factors satisfy fa(x1,x2,x3)fb(x1,x2)=ψ(x1,x2,x3).

8.4. Inference in Graphical Models 401
x1 x2
x3
(a)x1 x2
x3f
(b)x1 x2
x3fc
fa fb
(c)
Figure 8.42 (a) A directed graph with the factorization p(x1)p(x2)p(x3|x1,x2). (b) A factor graph representing
the same distribution as the directed graph, whose factor satisﬁes f(x1,x2,x3)=p(x1)p(x2)p(x3|x1,x2). (c)
A different factor graph representing the same distribution with factors fa(x1)=p(x1),fb(x2)=p(x2)and
fc(x1,x2,x3)=p(x3|x1,x2).
Factor graphs are said to be bipartite because they consist of two distinct kinds
of nodes, and all links go between nodes of opposite type. In general, factor graphs
can therefore always be drawn as two rows of nodes (variable nodes at the top and
factor nodes at the bottom) with links between the rows, as shown in the example in
Figure 8.40. In some situations, however, other ways of laying out the graph may
be more intuitive, for example when the factor graph is derived from a directed or
undirected graph, as we shall see.

If we are given a distribution that is expressed in terms of an undirected graph,
then we can readily convert it to a factor graph. To do this, we create variable nodes
corresponding to the nodes in the original undirected graph, and then create addi-
tional factor nodes corresponding to the maximal cliques xs. The factors fs(xs)are
then set equal to the clique potentials. Note that there may be several different factor
graphs that correspond to the same undirected graph. These concepts are illustrated
in Figure 8.41.

Similarly, to convert a directed graph to a factor graph, we simply create variable
nodes in the factor graph corresponding to the nodes of the directed graph, and then
create factor nodes corresponding to the conditional distributions, and then ﬁnally
add the appropriate links. Again, there can be multiple factor graphs all of which
correspond to the same directed graph. The conversion of a directed graph to a
factor graph is illustrated in Figure 8.42.
We have already noted the importance of tree-structured graphs for performing

efﬁcient inference. If we take a directed or undirected tree and convert it into a factor
graph, then the result will again be a tree (in other words, the factor graph will have
no loops, and there will be one and only one path connecting any two nodes). In
the case of a directed polytree, conversion to an undirected graph results in loops
due to the moralization step, whereas conversion to a factor graph again results in a
tree, as illustrated in Figure 8.43. In fact, local cycles in a directed graph due to

links connecting parents of a node can be removed on conversion to a factor graph
by deﬁning the appropriate factor function, as shown in Figure 8.44.
We have seen that multiple different factor graphs can represent the same di-
rected or undirected graph. This allows factor graphs to be more speciﬁc about the

402 8. GRAPHICAL MODELS
(a) (b) (c)
Figure 8.43 (a) A directed polytree. (b) The result of converting the polytree into an undirected graph showing
the creation of loops. (c) The result of converting the polytree into a factor graph, which retains the tree structure.
precise form of the factorization. Figure 8.45 shows an example of a fully connected
undirected graph along with two different factor graphs. In (b), the joint distri-
bution is given by a general form p(x)=f(x1,x2,x3), whereas in (c), it is given
by the more speciﬁc factorization p(x)=fa(x1,x2)fb(x1,x3)fc(x2,x3). It should
be emphasized that the factorization in (c) does not correspond to any conditional
independence properties.
8.4.4 The sum-product algorithm
We shall now make use of the factor graph framework to derive a powerful class
of efﬁcient, exact inference algorithms that are applicable to tree-structured graphs.
Here we shall focus on the problem of evaluating local marginals over nodes or

subsets of nodes, which will lead us to the sum-product algorithm. Later we shall
modify the technique to allow the most probable state to be found, giving rise to the
max-sum algorithm.
Also we shall suppose that all of the variables in the model are discrete, and
so marginalization corresponds to performing sums. The framework, however, is
equally applicable to linear-Gaussian models in which case marginalization involves
integration, and we shall consider an example of this in detail when we discuss linear
dynamical systems. Section 13.3

Figure 8.44 (a) A fragment of a di-
rected graph having a lo-
cal cycle. (b) Conversion
to a fragment of a factor
graph having a tree struc-
ture, in which f(x1,x2,x3)=
p(x1)p(x2|x1)p(x3|x1,x2).x1 x2
x3
(a)x1 x2
x3f(x1,x2,x3)
(b)

8.4. Inference in Graphical Models 403
x1 x2
x3
(a)x1 x2
(b)x1 x2
x3fa
fc fb
(c)
Figure 8.45 (a) A fully connected undirected graph. (b) and (c) Two factor graphs each of which corresponds
to the undirected graph in (a).
There is an algorithm for exact inference on directed graphs without loops known
asbelief propagation (Pearl, 1988; Lauritzen and Spiegelhalter, 1988), and is equiv-
alent to a special case of the sum-product algorithm. Here we shall consider only the
sum-product algorithm because it is simpler to derive and to apply, as well as being
more general.
We shall assume that the original graph is an undirected tree or a directed tree or
polytree, so that the corresponding factor graph has a tree structure. We ﬁrst convert
the original graph into a factor graph so that we can deal with both directed and
undirected models using the same framework. Our goal is to exploit the structure of
the graph to achieve two things: (i) to obtain an efﬁcient, exact inference algorithm

for ﬁnding marginals; (ii) in situations where several marginals are required to allow
computations to be shared efﬁciently.
We begin by considering the problem of ﬁnding the marginal p(x)for partic-
ular variable node x. For the moment, we shall suppose that all of the variables
are hidden. Later we shall see how to modify the algorithm to incorporate evidence
corresponding to observed variables. By deﬁnition, the marginal is obtained by sum-
ming the joint distribution over all variables except xso that
p(x)=∑
x\xp(x) (8.61)

p(x)=∑
wherex\xdenotes the set of variables in xwith variable xomitted. The idea is
to substitute for p(x)using the factor graph expression (8.59) and then interchange
summations and products in order to obtain an efﬁcient algorithm. Consider the
fragment of graph shown in Figure 8.46 in which we see that the tree structure of
the graph allows us to partition the factors in the joint distribution into groups, with
one group associated with each of the factor nodes that is a neighbour of the variable

nodex. We see that the joint distribution can be written as a product of the form
p(x)=∏
s∈ne(x)Fs(x, X s) (8.62)
ne(x)denotes the set of factor nodes that are neighbours of x, andXsdenotes the
set of all variables in the subtree connected to the variable node xvia the factor node

404 8. GRAPHICAL MODELS
Figure 8.46 A fragment of a factor graph illustrating the
evaluation of the marginal p(x).
x fsµfs→x(x)Fs(x,X s)
fs, andFs(x, X s)represents the product of all the factors in the group associated
with factor fs.
Substituting (8.62) into (8.61) and interchanging the sums and products, we ob-
tain
p(x)=∏
s∈ne(x)[∑
XsFs(x, X s)]
=∏
s∈ne(x)µfs→x(x). (8.63)
Here we have introduced a set of functions µfs→x(x), deﬁned by
µfs→x(x)≡∑
XsFs(x, X s) (8.64)
which can be viewed as messages from the factor nodes fsto the variable node x.
We see that the required marginal p(x)is given by the product of all the incoming
messages arriving at node x.
In order to evaluate these messages, we again turn to Figure 8.46 and note that
each factor Fs(x, X s)is described by a factor (sub-)graph and so can itself be fac-
torized. In particular, we can write
Fs(x, X s)=fs(x, x1,...,x M)G1(x1,Xs1)...G M(xM,XsM) (8.65)
where, for convenience, we have denoted the variables associated with factor fx,i n

addition to x,b yx1,...,x M. This factorization is illustrated in Figure 8.47. Note
that the set of variables {x, x1,...,x M}is the set of variables on which the factor
fsdepends, and so it can also be denoted xs, using the notation of (8.59).
Substituting (8.65) into (8.64) we obtain
µfs→x(x)=∑
x1...∑
xMfs(x, x1,...,x M)∏
m∈ne(fs)\x[∑
XxmGm(xm,Xsm)]
=∑
x1...∑
m∈ne(fs)\xµxm→fs(xm) (8.66)

8.4. Inference in Graphical Models 405
Figure 8.47 Illustration of the factorization of the subgraph as-
sociated with factor node fs.
xmxM
xfsµxM→fs(xM)
µfs→x(x)
Gm(xm,Xsm)
where ne(fs)denotes the set of variable nodes that are neighbours of the factor node
fs, andne(fs)\xdenotes the same set but with node xremoved. Here we have
deﬁned the following messages from variable nodes to factor nodes
µxm→fs(xm)≡∑
XsmGm(xm,Xsm). (8.67)
We have therefore introduced two distinct kinds of message, those that go from factor
nodes to variable nodes denoted µf→x(x), and those that go from variable nodes to
factor nodes denoted µx→f(x). In each case, we see that messages passed along a
link are always a function of the variable associated with the variable node that link
connects to.
The result (8.66) says that to evaluate the message sent by a factor node to a vari-
able node along the link connecting them, take the product of the incoming messages

along all other links coming into the factor node, multiply by the factor associated
with that node, and then marginalize over all of the variables associated with the
incoming messages. This is illustrated in Figure 8.47. It is important to note that
a factor node can send a message to a variable node once it has received incoming
messages from all other neighbouring variable nodes.
Finally, we derive an expression for evaluating the messages from variable nodes
to factor nodes, again by making use of the (sub-)graph factorization. From Fig-

ure 8.48, we see that term Gm(xm,Xsm)associated with node xmis given by a
product of terms Fl(xm,Xml)each associated with one of the factor nodes flthat is
linked to node xm(excluding node fs), so that
Gm(xm,Xsm)=∏
l∈ne(xm)\fsFl(xm,Xml) (8.68)
where the product is taken over all neighbours of node xmexcept for node fs. Note
that each of the factors Fl(xm,Xml)represents a subtree of the original graph of
precisely the same kind as introduced in (8.62). Substituting (8.68) into (8.67), we

406 8. GRAPHICAL MODELS
Figure 8.48 Illustration of the evaluation of the message sent by a
variable node to an adjacent factor node.
xm
flfL
fs
Fl(xm,Xml)
then obtain
µxm→fs(xm)=∏
l∈ne(xm)\fs[∑
XmlFl(xm,Xml)]
=∏
l∈ne(xm)\fsµfl→xm(xm) (8.69)
where we have used the deﬁnition (8.64) of the messages passed from factor nodes to
variable nodes. Thus to evaluate the message sent by a variable node to an adjacent
factor node along the connecting link, we simply take the product of the incoming
messages along all of the other links. Note that any variable node that has only
two neighbours performs no computation but simply passes messages through un-
changed. Also, we note that a variable node can send a message to a factor node
once it has received incoming messages from all other neighbouring factor nodes.
Recall that our goal is to calculate the marginal for variable node x, and that this
marginal is given by the product of incoming messages along all of the links arriving

at that node. Each of these messages can be computed recursively in terms of other
messages. In order to start this recursion, we can view the node xas the root of the
tree and begin at the leaf nodes. From the deﬁnition (8.69), we see that if a leaf node
is a variable node, then the message that it sends along its one and only link is given
by
µx→f(x)=1 (8.70)
as illustrated in Figure 8.49(a). Similarly, if the leaf node is a factor node, we see
from (8.66) that the message sent should take the form
µf→x(x)=f(x) (8.71)

by
Figure 8.49 The sum-product algorithm
begins with messages sent
by the leaf nodes, which de-
pend on whether the leaf
node is (a) a variable node,
or (b) a factor node.x fµx→f(x)=1
(a)x fµf→x(x)=f(x)
(b)

8.4. Inference in Graphical Models 407
as illustrated in Figure 8.49(b).
At this point, it is worth pausing to summarize the particular version of the sum-
product algorithm obtained so far for evaluating the marginal p(x). We start by
viewing the variable node xas the root of the factor graph and initiating messages
at the leaves of the graph using (8.70) and (8.71). The message passing steps (8.66)and (8.69) are then applied recursively until messages have been propagated along
every link, and the root node has received messages from all of its neighbours. Each
node can send a message towards the root once it has received messages from allof its other neighbours. Once the root node has received messages from all of its
neighbours, the required marginal can be evaluated using (8.63). We shall illustrate
this process shortly.
To see that each node will always receive enough messages to be able to send out

a message, we can use a simple inductive argument as follows. Clearly, for a graphcomprising a variable root node connected directly to several factor leaf nodes, the
algorithm trivially involves sending messages of the form (8.71) directly from the
leaves to the root. Now imagine building up a general graph by adding nodes one ata time, and suppose that for some particular graph we have a valid algorithm. When
one more (variable or factor) node is added, it can be connected only by a single

link because the overall graph must remain a tree, and so the new node will be a leafnode. It therefore sends a message to the node to which it is linked, which in turn
will therefore receive all the messages it requires in order to send its own message
towards the root, and so again we have a valid algorithm, thereby completing theproof.
Now suppose we wish to ﬁnd the marginals for every variable node in the graph.

This could be done by simply running the above algorithm afresh for each such node.However, this would be very wasteful as many of the required computations would
be repeated. We can obtain a much more efﬁcient procedure by ‘overlaying’ these
multiple message passing algorithms to obtain the general sum-product algorithm
as follows. Arbitrarily pick any (variable or factor) node and designate it as the

root. Propagate messages from the leaves to the root as before. At this point, theroot node will have received messages from all of its neighbours. It can therefore
send out messages to all of its neighbours. These in turn will then have received
messages from all of their neighbours and so can send out messages along the linksgoing away from the root, and so on. In this way, messages are passed outwards
from the root all the way to the leaves. By now, a message will have passed in

both directions across every link in the graph, and every node will have receiveda message from all of its neighbours. Again a simple inductive argument can be
used to verify the validity of this message passing protocol. Because every variable Exercise 8.20
node will have received messages from all of its neighbours, we can readily calculatethe marginal distribution for every variable in the graph. The number of messages
that have to be computed is given by twice the number of links in the graph and

so involves only twice the computation involved in ﬁnding a single marginal. Bycomparison, if we had run the sum-product algorithm separately for each node, the
amount of computation would grow quadratically with the size of the graph. Note
that this algorithm is in fact independent of which node was designated as the root,

408 8. GRAPHICAL MODELS
Figure 8.50 The sum-product algorithm can be viewed
purely in terms of messages sent out by factor
nodes to other factor nodes. In this example,
the outgoing message shown by the blue arrow
is obtained by taking the product of all the in-
coming messages shown by green arrows, mul-
tiplying by the factor fs, and marginalizing over
the variables x1andx2. fsx1
x2x3
and indeed the notion of one node having a special status was introduced only as a
convenient way to explain the message passing protocol.
Next suppose we wish to ﬁnd the marginal distributions p(xs)associated with
the sets of variables belonging to each of the factors. By a similar argument to that
used above, it is easy to see that the marginal associated with a factor is given by the Exercise 8.21
product of messages arriving at the factor node and the local factor at that node
p(xs)=fs(xs)∏
i∈ne(fs)µxi→fs(xi) (8.72)
in complete analogy with the marginals at the variable nodes. If the factors are

parameterized functions and we wish to learn the values of the parameters using
the EM algorithm, then these marginals are precisely the quantities we will need to
calculate in the E step, as we shall see in detail when we discuss the hidden Markov
model in Chapter 13.
The message sent by a variable node to a factor node, as we have seen, is simply
the product of the incoming messages on other links. We can if we wish view the
sum-product algorithm in a slightly different form by eliminating messages from

variable nodes to factor nodes and simply considering messages that are sent out by
factor nodes. This is most easily seen by considering the example in Figure 8.50.
So far, we have rather neglected the issue of normalization. If the factor graph
was derived from a directed graph, then the joint distribution is already correctly nor-
malized, and so the marginals obtained by the sum-product algorithm will similarly
be normalized correctly. However, if we started from an undirected graph, then in

general there will be an unknown normalization coefﬁcient 1/Z. As with the simple
chain example of Figure 8.38, this is easily handled by working with an unnormal-
ized version ˜p(x)of the joint distribution, where p(x)=˜p(x)/Z. We ﬁrst run the
sum-product algorithm to ﬁnd the corresponding unnormalized marginals ˜p(xi). The
coefﬁcient 1/Zis then easily obtained by normalizing any one of these marginals,
and this is computationally efﬁcient because the normalization is done over a single

variable rather than over the entire set of variables as would be required to normalize
˜p(x)directly.
At this point, it may be helpful to consider a simple example to illustrate the
operation of the sum-product algorithm. Figure 8.51 shows a simple 4-node factor

8.4. Inference in Graphical Models 409
Figure 8.51 A simple factor graph used to illustrate the
sum-product algorithm.x1 x2 x3
x4fa fb
fc
graph whose unnormalized joint distribution is given by
˜p(x)=fa(x1,x2)fb(x2,x3)fc(x2,x4). (8.73)
In order to apply the sum-product algorithm to this graph, let us designate node x3
as the root, in which case there are two leaf nodes x1andx4. Starting with the leaf
nodes, we then have the following sequence of six messages
µx1→fa(x1)=1 (8.74)
µfa→x2(x2)=∑
x1fa(x1,x2) (8.75)
µx4→fc(x4)=1 (8.76)
µfc→x2(x2)=∑
x4fc(x2,x4) (8.77)
µx2→fb(x2)= µfa→x2(x2)µfc→x2(x2) (8.78)
µfb→x3(x3)=∑
x2fb(x2,x3)µx2→fb. (8.79)
The direction of ﬂow of these messages is illustrated in Figure 8.52. Once this mes-
sage propagation is complete, we can then propagate messages from the root node
out to the leaf nodes, and these are given by
µx3→fb(x3)=1 (8.80)
µfb→x2(x2)=∑
x3fb(x2,x3) (8.81)
µx2→fa(x2)= µfb→x2(x2)µfc→x2(x2) (8.82)
µfa→x1(x1)=∑
x2fa(x1,x2)µx2→fa(x2) (8.83)

µx2→fc(x2)= µfa→x2(x2)µfb→x2(x2) (8.84)
µfc→x4(x4)=∑
x2fc(x2,x4)µx2→fc(x2). (8.85)

410 8. GRAPHICAL MODELS
x1 x2 x3
x4
(a)x1 x2 x3
x4
(b)
Figure 8.52 Flow of messages for the sum-product algorithm applied to the example graph in Figure 8.51. (a)
From the leaf nodes x1andx4towards the root node x3. (b) From the root node towards the leaf nodes.
One message has now passed in each direction across each link, and we can now
evaluate the marginals. As a simple check, let us verify that the marginal p(x2)is
given by the correct expression. Using (8.63) and substituting for the messages using
the above results, we have
˜p(x2)= µfa→x2(x2)µfb→x2(x2)µfc→x2(x2)
=[∑
x1fa(x1,x2)][∑
x3fb(x2,x3)][∑
x4fc(x2,x4)]
=∑
x1∑
x2∑
x4fa(x1,x2)fb(x2,x3)fc(x2,x4)
=∑
x1∑
x3∑
x4˜p(x) (8.86)
So far, we have assumed that all of the variables in the graph are hidden. In most
practical applications, a subset of the variables will be observed, and we wish to cal-
culate posterior distributions conditioned on these observations. Observed nodes are

=[∑
=∑
x1∑
x2∑
=∑
x1∑
x3∑
easily handled within the sum-product algorithm as follows. Suppose we partition x
into hidden variables hand observed variables v, and that the observed value of v
is denotedˆv. Then we simply multiply the joint distribution p(x)by∏
iI(vi,ˆvi),
where I(v,ˆv)=1 ifv=ˆvandI(v,ˆv)=0 otherwise. This product corresponds
top(h,v=ˆv)and hence is an unnormalized version of p(h|v=ˆv). By run-
ning the sum-product algorithm, we can efﬁciently calculate the posterior marginals
p(hi|v=ˆv)up to a normalization coefﬁcient whose value can be found efﬁciently

using a local computation. Any summations over variables in vthen collapse into a
single term.
We have assumed throughout this section that we are dealing with discrete vari-
ables. However, there is nothing speciﬁc to discrete variables either in the graphical
framework or in the probabilistic construction of the sum-product algorithm. For

8.4. Inference in Graphical Models 411
Table 8.1 Example of a joint distribution over two binary variables for
which the maximum of the joint distribution occurs for dif-
ferent variable values compared to the maxima of the two
marginals.x=0 x=1
y=0 0.3 0.4
y=1 0.3 0.0
continuous variables the summations are simply replaced by integrations. We shall
give an example of the sum-product algorithm applied to a graph of linear-Gaussian
variables when we consider linear dynamical systems. Section 13.3
8.4.5 The max-sum algorithm
The sum-product algorithm allows us to take a joint distribution p(x)expressed
as a factor graph and efﬁciently ﬁnd marginals over the component variables. Two
other common tasks are to ﬁnd a setting of the variables that has the largest prob-
ability and to ﬁnd the value of that probability. These can be addressed through aclosely related algorithm called max-sum , which can be viewed as an application of
dynamic programming in the context of graphical models (Cormen et al. , 2001).

A simple approach to ﬁnding latent variable values having high probability
would be to run the sum-product algorithm to obtain the marginals p(x
i)for ev-
ery variable, and then, for each marginal in turn, to ﬁnd the value x⋆
ithat maximizes
that marginal. However, this would give the set of values that are individually the
most probable. In practice, we typically wish to ﬁnd the set of values that jointly
have the largest probability, in other words the vector xmaxthat maximizes the joint
xmax=a r gm a x

i)for ev-
xp(x) (8.87)
for which the corresponding value of the joint probability will be given by
p(xmax) = max
xp(x). (8.88)
In general, xmaxis not the same as the set of x⋆
ivalues, as we can easily show using
a simple example. Consider the joint distribution p(x, y)over two binary variables
x, y∈{0,1}given in Table 8.1. The joint distribution is maximized by setting x=
1andy=0, corresponding the value 0.4. However, the marginal for p(x), obtained
by summing over both values of y,i sg i v e nb y p(x=0 )=0 .6andp(x=1 )=0 .4,

and similarly the marginal for yis given by p(y=0 )=0 .7andp(y=1 )=0 .3,
and so the marginals are maximized by x=0 andy=0, which corresponds to a
value of 0.3for the joint distribution. In fact, it is not difﬁcult to construct examples
for which the set of individually most probable values has probability zero under the
joint distribution. Exercise 8.27
We therefore seek an efﬁcient algorithm for ﬁnding the value of xthat maxi-

mizes the joint distribution p(x)and that will allow us to obtain the value of the
joint distribution at its maximum. To address the second of these problems, we shall
simply write out the max operator in terms of its components
max
xp(x)=m a x
x1...max
xMp(x) (8.89)

412 8. GRAPHICAL MODELS
where Mis the total number of variables, and then substitute for p(x)using its
expansion in terms of a product of factors. In deriving the sum-product algorithm,we made use of the distributive law (8.53) for multiplication. Here we make use of
the analogous law for the max operator
max(ab, ac)=amax(b, c) (8.90)
which holds if a⩾0(as will always be the case for the factors in a graphical model).
This allows us to exchange products with maximizations.
Consider ﬁrst the simple example of a chain of nodes described by (8.49). The
evaluation of the probability maximum can be written as
max
xp(x)=1
Zmax
x1···max
xN[ψ1,2(x1,x2)···ψN−1,N(xN−1,xN)]
=1
Zmax
x1[
ψ1,2(x1,x2)[
···max
xNψN−1,N(xN−1,xN)]]
.
As with the calculation of marginals, we see that exchanging the max and product
operators results in a much more efﬁcient computation, and one that is easily inter-
preted in terms of messages passed from node xNbackwards along the chain to node
x1.

max
xp(x)=1
Zmax
x1···max
=1
Zmax
x1[
···max
.
x1.
We can readily generalize this result to arbitrary tree-structured factor graphs
by substituting the expression (8.59) for the factor graph expansion into (8.89) andagain exchanging maximizations with products. The structure of this calculation is
identical to that of the sum-product algorithm, and so we can simply translate those
results into the present context. In particular, suppose that we designate a particular
variable node as the ‘root’ of the graph. Then we start a set of messages propagating

x1.
inwards from the leaves of the tree towards the root, with each node sending itsmessage towards the root once it has received all incoming messages from its other
neighbours. The ﬁnal maximization is performed over the product of all messages
arriving at the root node, and gives the maximum value for p(x). This could be called
themax-product algorithm and is identical to the sum-product algorithm except that
summations are replaced by maximizations. Note that at this stage, messages have

been sent from leaves to the root, but not in the other direction.
In practice, products of many small probabilities can lead to numerical under-
ﬂow problems, and so it is convenient to work with the logarithm of the joint distri-
bution. The logarithm is a monotonic function, so that if a>b thenlna>lnb, and
hence the max operator and the logarithm function can be interchanged, so that
ln
(
max
xp(x))
=m a x
xlnp(x). (8.91)
The distributive property is preserved because
max(a+b, a+c)=a+m a x ( b, c). (8.92)

ln
(
max
xp(x))
=m a x
Thus taking the logarithm simply has the effect of replacing the products in the
max-product algorithm with sums, and so we obtain the max-sum algorithm. From

8.4. Inference in Graphical Models 413
the results (8.66) and (8.69) derived earlier for the sum-product algorithm, we can
readily write down the max-sum algorithm in terms of message passing simply byreplacing ‘sum’ with ‘max’ and replacing products with sums of logarithms to give
µ
f→x(x) = max
x1,...,x M⎡
⎣lnf(x, x1,...,x M)+∑
m∈ne(fs)\xµxm→f(xm)⎤
⎦(8.93)
µx→f(x)=∑
l∈ne(x)\fµfl→x(x). (8.94)
The initial messages sent by the leaf nodes are obtained by analogy with (8.70) and
(8.71) and are given by
µx→f(x)=0 (8.95)
µf→x(x)=l n f(x) (8.96)
while at the root node the maximum probability can then be computed, by analogy
with (8.63), using
pmax=m a x
x⎡
⎣∑
s∈ne(x)µfs→x(x)⎤
⎦. (8.97)
So far, we have seen how to ﬁnd the maximum of the joint distribution by prop-
agating messages from the leaves to an arbitrarily chosen root node. The result will
be the same irrespective of which node is chosen as the root. Now we turn to thesecond problem of ﬁnding the conﬁguration of the variables for which the joint dis-

pmax=m a x
x⎡
⎣∑
⎦. (8.97)
tribution attains this maximum value. So far, we have sent messages from the leaves
to the root. The process of evaluating (8.97) will also give the value x
maxfor the
most probable value of the root node variable, deﬁned by
x⎡
⎣∑
⎦. (8.98)
At this point, we might be tempted simply to continue with the message passing al-

maxfor the
x⎡
⎣∑
⎦. (8.98)
gorithm and send messages from the root back out to the leaves, using (8.93) and(8.94), then apply (8.98) to all of the remaining variable nodes. However, because
we are now maximizing rather than summing, it is possible that there may be mul-
tiple conﬁgurations of xall of which give rise to the maximum value for p(x).I n
such cases, this strategy can fail because it is possible for the individual variable
values obtained by maximizing the product of messages at each node to belong todifferent maximizing conﬁgurations, giving an overall conﬁguration that no longer
corresponds to a maximum.

The problem can be resolved by adopting a rather different kind of message
passing from the root node to the leaves. To see how this works, let us return once
again to the simple chain example of Nvariables x
1,...,x Neach having Kstates,

414 8. GRAPHICAL MODELS
Figure 8.53 A lattice, or trellis, diagram show-
ing explicitly the Kpossible states (one per row
of the diagram) for each of the variables xnin the
chain model. In this illustration K=3. The ar-
row shows the direction of message passing in the
max-product algorithm. For every state kof each
variable xn(corresponding to column nof the dia-
gram) the function φ(xn)deﬁnes a unique state at
the previous variable, indicated by the black lines.
The two paths through the lattice correspond to
conﬁgurations that give the global maximum of the
joint probability distribution, and either of these
can be found by tracing back along the black lines
in the opposite direction to the arrow.k=1
k=2
k=3
n−2 n−1 nn +1
corresponding to the graph shown in Figure 8.38. Suppose we take node xNto be
the root node. Then in the ﬁrst phase, we propagate messages from the leaf node x1
to the root node using
µxn→fn,n +1(xn)= µfn−1,n→xn(xn)
µfn−1,n→xn(xn) = max
xn−1[
lnfn−1,n(xn−1,xn)+µxn−1→fn−1,n(xn)]

k=2
k=3
xn−1[
which follow from applying (8.94) and (8.93) to this particular graph. The initial
message sent from the leaf node is simply
µx1→f1,2(x1)=0. (8.99)
The most probable value for xNis then given by
xmax
N=a r gm a x
xN[
µfN−1,N→xN(xN)]
. (8.100)
Now we need to determine the states of the previous variables that correspond to the
same maximizing conﬁguration. This can be done by keeping track of which values
of the variables gave rise to the maximum state of each variable, in other words by
storing quantities given by

xmax
N=a r gm a x
xN[
. (8.100)
φ(xn) = arg max
xn−1[
. (8.101)
To understand better what is happening, it is helpful to represent the chain of vari-
ables in terms of a lattice ortrellis diagram as shown in Figure 8.53. Note that this
is not a probabilistic graphical model because the nodes represent individual states
of variables, while each variable corresponds to a column of such states in the di-
agram. For each state of a given variable, there is a unique state of the previous
variable that maximizes the probability (ties are broken either systematically or at

random), corresponding to the function φ(xn)given by (8.101), and this is indicated

8.4. Inference in Graphical Models 415
by the lines connecting the nodes. Once we know the most probable value of the ﬁ-
nal node xN, we can then simply follow the link back to ﬁnd the most probable state
of node xN−1and so on back to the initial node x1. This corresponds to propagating
a message back down the chain using
xmax
n−1=φ(xmax
n) (8.102)
and is known as back-tracking . Note that there could be several values of xn−1all
of which give the maximum value in (8.101). Provided we chose one of these values
when we do the back-tracking, we are assured of a globally consistent maximizingconﬁguration.
In Figure 8.53, we have indicated two paths, each of which we shall suppose
corresponds to a global maximum of the joint probability distribution. If k=2
andk=3 each represent possible values of x
max
N, then starting from either state
and tracing back along the black lines, which corresponds to iterating (8.102), we
obtain a valid global maximum conﬁguration. Note that if we had run a forward

max
pass of max-sum message passing followed by a backward pass and then applied
(8.98) at each node separately, we could end up selecting some states from one pathand some from the other path, giving an overall conﬁguration that is not a global
maximizer. We see that it is necessary instead to keep track of the maximizing states
during the forward pass using the functions φ(x
n)and then use back-tracking to ﬁnd
a consistent solution.
The extension to a general tree-structured factor graph should now be clear. If

a message is sent from a factor node fto a variable node x, a maximization is
performed over all other variable nodes x1,...,x Mthat are neighbours of that fac-
tor node, using (8.93). When we perform this maximization, we keep a record of
which values of the variables x1,...,x Mgave rise to the maximum. Then in the
back-tracking step, having found xmax, we can then use these stored values to as-
sign consistent maximizing states xmax
1,...,xmax
M. The max-sum algorithm, with

back-tracking, gives an exact maximizing conﬁguration for the variables providedthe factor graph is a tree. An important application of this technique is for ﬁnding
the most probable sequence of hidden states in a hidden Markov model, in which
case it is known as the Viterbi algorithm. Section 13.2
As with the sum-product algorithm, the inclusion of evidence in the form of
observed variables is straightforward. The observed variables are clamped to their
observed values, and the maximization is performed over the remaining hidden vari-

ables. This can be shown formally by including identity functions for the observed
variables into the factor functions, as we did for the sum-product algorithm.
It is interesting to compare max-sum with the iterated conditional modes (ICM)
algorithm described on page 389. Each step in ICM is computationally simpler be-
cause the ‘messages’ that are passed from one node to the next comprise a singlevalue consisting of the new state of the node for which the conditional distribution
is maximized. The max-sum algorithm is more complex because the messages are
functions of node variables xand hence comprise a set of Kvalues for each pos-

sible state of x. Unlike max-sum, however, ICM is not guaranteed to ﬁnd a global
maximum even for tree-structured graphs.

416 8. GRAPHICAL MODELS
8.4.6 Exact inference in general graphs
The sum-product and max-sum algorithms provide efﬁcient and exact solutions
to inference problems in tree-structured graphs. For many practical applications,
however, we have to deal with graphs having loops.
The message passing framework can be generalized to arbitrary graph topolo-
gies, giving an exact inference procedure known as the junction tree algorithm (Lau-
ritzen and Spiegelhalter, 1988; Jordan, 2007). Here we give a brief outline of the
key steps involved. This is not intended to convey a detailed understanding of the
algorithm, but rather to give a ﬂavour of the various stages involved. If the starting
point is a directed graph, it is ﬁrst converted to an undirected graph by moraliza-tion, whereas if starting from an undirected graph this step is not required. Next the
graph is triangulated , which involves ﬁnding chord-less cycles containing four or

more nodes and adding extra links to eliminate such chord-less cycles. For instance,in the graph in Figure 8.36, the cycle A–C–B–D–Ais chord-less a link could be
added between AandBor alternatively between CandD. Note that the joint dis-
tribution for the resulting triangulated graph is still deﬁned by a product of the samepotential functions, but these are now considered to be functions over expanded sets
of variables. Next the triangulated graph is used to construct a new tree-structured

undirected graph called a join tree , whose nodes correspond to the maximal cliques
of the triangulated graph, and whose links connect pairs of cliques that have vari-
ables in common. The selection of which pairs of cliques to connect in this way is
important and is done so as to give a maximal spanning tree deﬁned as follows. Of
all possible trees that link up the cliques, the one that is chosen is one for which the
weight of the tree is largest, where the weight for a link is the number of nodes shared

by the two cliques it connects, and the weight for the tree is the sum of the weightsfor the links. If the tree is condensed, so that any clique that is a subset of another
clique is absorbed into the larger clique, this gives a junction tree . As a consequence
of the triangulation step, the resulting tree satisﬁes the running intersection property ,
which means that if a variable is contained in two cliques, then it must also be con-

tained in every clique on the path that connects them. This ensures that inferenceabout variables will be consistent across the graph. Finally, a two-stage message
passing algorithm, essentially equivalent to the sum-product algorithm, can now be
applied to this junction tree in order to ﬁnd marginals and conditionals. Althoughthe junction tree algorithm sounds complicated, at its heart is the simple idea that
we have used already of exploiting the factorization properties of the distribution to

allow sums and products to be interchanged so that partial summations can be per-formed, thereby avoiding having to work directly with the joint distribution. The
role of the junction tree is to provide a precise and efﬁcient way to organize these
computations. It is worth emphasizing that this is achieved using purely graphicaloperations!
The junction tree is exact for arbitrary graphs and is efﬁcient in the sense that

for a given graph there does not in general exist a computationally cheaper approach.Unfortunately, the algorithm must work with the joint distributions within each node
(each of which corresponds to a clique of the triangulated graph) and so the compu-
tational cost of the algorithm is determined by the number of variables in the largest

8.4. Inference in Graphical Models 417
clique and will grow exponentially with this number in the case of discrete variables.
An important concept is the treewidth of a graph (Bodlaender, 1993), which is de-
ﬁned in terms of the number of variables in the largest clique. In fact, it is deﬁned to
be as one less than the size of the largest clique, to ensure that a tree has a treewidth
of 1. Because there in general there can be multiple different junction trees that canbe constructed from a given starting graph, the treewidth is deﬁned by the junction
tree for which the largest clique has the fewest variables. If the treewidth of the
original graph is high, the junction tree algorithm becomes impractical.
8.4.7 Loopy belief propagation
For many problems of practical interest, it will not be feasible to use exact in-
ference, and so we need to exploit effective approximation methods. An important
class of such approximations, that can broadly be called variational methods, will be

discussed in detail in Chapter 10. Complementing these deterministic approaches isa wide range of sampling methods, also called Monte Carlo methods, that are based
on stochastic numerical sampling from distributions and that will be discussed at
length in Chapter 11.
Here we consider one simple approach to approximate inference in graphs with
loops, which builds directly on the previous discussion of exact inference in trees.

The idea is simply to apply the sum-product algorithm even though there is no guar-antee that it will yield good results. This approach is known as loopy belief propa-
gation (Frey and MacKay, 1998) and is possible because the message passing rules
(8.66) and (8.69) for the sum-product algorithm are purely local. However, because
the graph now has cycles, information can ﬂow many times around the graph. For
some models, the algorithm will converge, whereas for others it will not.
In order to apply this approach, we need to deﬁne a message passing schedule .

Let us assume that one message is passed at a time on any given link and in any
given direction. Each message sent from a node replaces any previous message sentin the same direction across the same link and will itself be a function only of the
most recent messages received by that node at previous steps of the algorithm.
We have seen that a message can only be sent across a link from a node when
all other messages have been received by that node across its other links. Because
there are loops in the graph, this raises the problem of how to initiate the message

passing algorithm. To resolve this, we suppose that an initial message given by theunit function has been passed across every link in each direction. Every node is then
in a position to send a message.
There are now many possible ways to organize the message passing schedule.
For example, the ﬂooding schedule simultaneously passes a message across every
link in both directions at each time step, whereas schedules that pass one message at
a time are called serial schedules .

Following Kschischnang et al. (2001), we will say that a (variable or factor)
nodeahas a message pending on its link to a node bif node ahas received any
message on any of its other links since the last time it send a message to b. Thus,
when a node receives a message on one of its links, this creates pending messages
on all of its other links. Only pending messages need to be transmitted because

418 8. GRAPHICAL MODELS
other messages would simply duplicate the previous message on the same link. For
graphs that have a tree structure, any schedule that sends only pending messageswill eventually terminate once a message has passed in each direction across every
link. At this point, there are no pending messages, and the product of the received Exercise 8.29
messages at every variable give the exact marginal. In graphs having loops, however,the algorithm may never terminate because there might always be pending messages,
although in practice it is generally found to converge within a reasonable time for
most applications. Once the algorithm has converged, or once it has been stoppedif convergence is not observed, the (approximate) local marginals can be computed
using the product of the most recently received incoming messages to each variable
node or factor node on every link.
In some applications, the loopy belief propagation algorithm can give poor re-

sults, whereas in other applications it has proven to be very effective. In particular,state-of-the-art algorithms for decoding certain kinds of error-correcting codes are
equivalent to loopy belief propagation (Gallager, 1963; Berrou et al. , 1993; McEliece
et al. , 1998; MacKay and Neal, 1999; Frey, 1998).
8.4.8 Learning the graph structure
In our discussion of inference in graphical models, we have assumed that the
structure of the graph is known and ﬁxed. However, there is also interest in go-

ing beyond the inference problem and learning the graph structure itself from data(Friedman and Koller, 2003). This requires that we deﬁne a space of possible struc-
tures as well as a measure that can be used to score each structure.
From a Bayesian viewpoint, we would ideally like to compute a posterior dis-
tribution over graph structures and to make predictions by averaging with respect
to this distribution. If we have a prior p(m)over graphs indexed by m, then the

p(m|D)∝p(m)p(D|m) (8.103)
where Dis the observed data set. The model evidence p(D|m)then provides the
score for each model. However, evaluation of the evidence involves marginalization
over the latent variables and presents a challenging computational problem for many
models.
Exploring the space of structures can also be problematic. Because the number
of different graph structures grows exponentially with the number of nodes, it is
often necessary to resort to heuristics to ﬁnd good candidates.
Exercises

models.
Exercises
8.1 (⋆)www By marginalizing out the variables in order, show that the representation
(8.5) for the joint distribution of a directed graph is correctly normalized, provided
each of the conditional distributions is normalized.
8.2 (⋆)www Show that the property of there being no directed cycles in a directed
graph follows from the statement that there exists an ordered numbering of the nodes
such that for each node there are no links going to a lower-numbered node.

Exercises 419
Table 8.2 The joint distribution over three binary variables. abcp(a, b, c)
000 0.192
001 0.144
010 0.048
011 0.216
100 0.192
101 0.064
110 0.048
111 0.096
8.3 (⋆⋆)Consider three binary variables a, b, c ∈{0,1}having the joint distribution
given in Table 8.2. Show by direct evaluation that this distribution has the property
thataandbare marginally dependent, so that p(a, b)̸=p(a)p(b), but that they
become independent when conditioned on c, so that p(a, b|c)=p(a|c)p(b|c)for
bothc=0andc=1.
8.4 (⋆⋆)Evaluate the distributions p(a),p(b|c), andp(c|a)corresponding to the joint
distribution given in Table 8.2. Hence show by direct evaluation that p(a, b, c)=
p(a)p(c|a)p(b|c). Draw the corresponding directed graph.
8.5 (⋆)www Draw a directed probabilistic graphical model corresponding to the
relevance vector machine described by (7.79) and (7.80).
8.6 (⋆)For the model shown in Figure 8.13, we have seen that the number of parameters

required to specify the conditional distribution p(y|x1,...,x M), where xi∈{0,1},
could be reduced from 2MtoM+1by making use of the logistic sigmoid represen-
tation (8.10). An alternative representation (Pearl, 1988) is given by
p(y=1|x1,...,x M)=1−(1−µ0)M∏
i=1(1−µi)xi(8.104)
where the parameters µirepresent the probabilities p(xi=1 ) , andµ0is an additional
parameters satisfying 0⩽µ0⩽1. The conditional distribution (8.104) is known as
thenoisy-OR . Show that this can be interpreted as a ‘soft’ (probabilistic) form of the

logical OR function (i.e., the function that gives y=1whenever at least one of the
xi=1). Discuss the interpretation of µ0.
8.7 (⋆⋆)Using the recursion relations (8.15) and (8.16), show that the mean and covari-
ance of the joint distribution for the graph shown in Figure 8.14 are given by (8.17)
and (8.18), respectively.
8.8 (⋆)www Show that a⊥⊥b, c|dimplies a⊥⊥b|d.
8.9 (⋆)www Using the d-separation criterion, show that the conditional distribution
for a node xin a directed graph, conditioned on all of the nodes in the Markov

blanket, is independent of the remaining variables in the graph.

420 8. GRAPHICAL MODELS
Figure 8.54 Example of a graphical model used to explore the con-
ditional independence properties of the head-to-head
patha–c–bwhen a descendant of c, namely the node
d, is observed.
cab
d
8.10 (⋆)Consider the directed graph shown in Figure 8.54 in which none of the variables
is observed. Show that a⊥⊥b|∅. Suppose we now observe the variable d. Show
that in general a̸⊥⊥b|d.
8.11 (⋆⋆)Consider the example of the car fuel system shown in Figure 8.21, and suppose
that instead of observing the state of the fuel gauge Gdirectly, the gauge is seen by
the driver Dwho reports to us the reading on the gauge. This report is either that the
gauge shows full D=1or that it shows empty D=0. Our driver is a bit unreliable,
as expressed through the following probabilities
p(D=1|G=1 ) = 0 .9 (8.105)
p(D=0|G=0 ) = 0 .9. (8.106)
Suppose that the driver tells us that the fuel gauge shows empty, in other words
that we observe D=0. Evaluate the probability that the tank is empty given only

this observation. Similarly, evaluate the corresponding probability given also the
observation that the battery is ﬂat, and note that this second probability is lower.
Discuss the intuition behind this result, and relate the result to Figure 8.54.
8.12 (⋆)www Show that there are 2M(M−1)/2distinct undirected graphs over a set of
Mdistinct random variables. Draw the 8possibilities for the case of M=3.
8.13 (⋆)Consider the use of iterated conditional modes (ICM) to minimize the energy
function given by (8.42). Write down an expression for the difference in the values

of the energy associated with the two states of a particular variable xj, with all other
variables held ﬁxed, and show that it depends only on quantities that are local to xj
in the graph.
8.14 (⋆)Consider a particular case of the energy function given by (8.42) in which the
coefﬁcients β=h=0. Show that the most probable conﬁguration of the latent
variables is given by xi=yifor all i.
8.15 (⋆⋆)www Show that the joint distribution p(xn−1,xn)for two neighbouring

nodes in the graph shown in Figure 8.38 is given by an expression of the form (8.58).

Exercises 421
8.16 (⋆⋆)Consider the inference problem of evaluating p(xn|xN)for the graph shown
in Figure 8.38, for all nodes n∈{1,...,N −1}. Show that the message passing
algorithm discussed in Section 8.4.1 can be used to solve this efﬁciently, and discuss
which messages are modiﬁed and in what way.
8.17 (⋆⋆)Consider a graph of the form shown in Figure 8.38 having N=5 nodes, in
which nodes x3andx5are observed. Use d-separation to show that x2⊥⊥x5|x3.
Show that if the message passing algorithm of Section 8.4.1 is applied to the evalu-
ation of p(x2|x3,x5), the result will be independent of the value of x5.
8.18 (⋆⋆)www Show that a distribution represented by a directed tree can trivially
be written as an equivalent distribution over the corresponding undirected tree. Alsoshow that a distribution expressed as an undirected tree can, by suitable normaliza-
tion of the clique potentials, be written as a directed tree. Calculate the number of

distinct directed trees that can be constructed from a given undirected tree.
8.19 (⋆⋆)Apply the sum-product algorithm derived in Section 8.4.4 to the chain-of-
nodes model discussed in Section 8.4.1 and show that the results (8.54), (8.55), and
(8.57) are recovered as a special case.
8.20 (⋆)
www Consider the message passing protocol for the sum-product algorithm on

8.20 (⋆)
a tree-structured factor graph in which messages are ﬁrst propagated from the leavesto an arbitrarily chosen root node and then from the root node out to the leaves. Use
proof by induction to show that the messages can be passed in such an order that
at every step, each node that must send a message has received all of the incomingmessages necessary to construct its outgoing messages.
8.21 (⋆⋆)
www Show that the marginal distributions p(xs)over the sets of variables
xsassociated with each of the factors fx(xs)in a factor graph can be found by ﬁrst

8.21 (⋆⋆)
running the sum-product message passing algorithm and then evaluating the requiredmarginals using (8.72).
8.22 (⋆)Consider a tree-structured factor graph, in which a given subset of the variable
nodes form a connected subgraph (i.e., any variable node of the subset is connected
to at least one of the other variable nodes via a single factor node). Show how thesum-product algorithm can be used to compute the marginal distribution over that
subset.
8.23 (⋆⋆)
www In Section 8.4.4, we showed that the marginal distribution p(xi)for a
variable node xiin a factor graph is given by the product of the messages arriving at

subset.
8.23 (⋆⋆)
this node from neighbouring factor nodes in the form (8.63). Show that the marginal
p(xi)can also be written as the product of the incoming message along any one of
the links with the outgoing message along the same link.
8.24 (⋆⋆)Show that the marginal distribution for the variables xsin a factor fs(xs)in
a tree-structured factor graph, after running the sum-product message passing algo-
rithm, can be written as the product of the message arriving at the factor node along
all its links, times the local factor f(xs), in the form (8.72).

422 8. GRAPHICAL MODELS
8.25 (⋆⋆)In (8.86), we veriﬁed that the sum-product algorithm run on the graph in
Figure 8.51 with node x3designated as the root node gives the correct marginal for
x2. Show that the correct marginals are obtained also for x1andx3. Similarly, show
that the use of the result (8.72) after running the sum-product algorithm on this graph
gives the correct joint distribution for x1,x2.
8.26 (⋆)Consider a tree-structured factor graph over discrete variables, and suppose we
wish to evaluate the joint distribution p(xa,xb)associated with two variables xaand
xbthat do not belong to a common factor. Deﬁne a procedure for using the sum-
product algorithm to evaluate this joint distribution in which one of the variables issuccessively clamped to each of its allowed values.
8.27 (⋆⋆)Consider two discrete variables xandyeach having three possible states, for
example x, y∈{0,1,2}. Construct a joint distribution p(x, y)over these variables
having the property that the value

ˆxthat maximizes the marginal p(x), along with
the valueˆythat maximizes the marginal p(y), together have probability zero under
the joint distribution, so that p(ˆx,ˆy)=0 .
8.28 (⋆⋆)www The concept of a pending message in the sum-product algorithm for
a factor graph was deﬁned in Section 8.4.7. Show that if the graph has one or morecycles, there will always be at least one pending message irrespective of how long
the algorithm runs.
8.29 (⋆⋆)
www Show that if the sum-product algorithm is run on a factor graph with a

8.29 (⋆⋆)
tree structure (no loops), then after a ﬁnite number of messages have been sent, therewill be no pending messages.

9
Mixture Models
and EM
If we deﬁne a joint distribution over observed and latent variables, the correspond-
ing distribution of the observed variables alone is obtained by marginalization. Thisallows relatively complex marginal distributions over observed variables to be ex-
pressed in terms of more tractable joint distributions over the expanded space of
observed and latent variables. The introduction of latent variables thereby allowscomplicated distributions to be formed from simpler components. In this chapter,
we shall see that mixture distributions, such as the Gaussian mixture discussed in
Section 2.3.9, can be interpreted in terms of discrete latent variables. Continuouslatent variables will form the subject of Chapter 12.
As well as providing a framework for building more complex probability dis-
tributions, mixture models can also be used to cluster data. We therefore begin ourdiscussion of mixture distributions by considering the problem of ﬁnding clusters

in a set of data points, which we approach ﬁrst using a nonprobabilistic technique
called the K-means algorithm (Lloyd, 1982). Then we introduce the latent variable Section 9.1
423

424 9. MIXTURE MODELS AND EM
view of mixture distributions in which the discrete latent variables can be interpreted
as deﬁning assignments of data points to speciﬁc components of the mixture. A gen- Section 9.2
eral technique for ﬁnding maximum likelihood estimators in latent variable models
is the expectation-maximization (EM) algorithm. We ﬁrst of all use the Gaussian
mixture distribution to motivate the EM algorithm in a fairly informal way, and thenwe give a more careful treatment based on the latent variable viewpoint. We shall Section 9.3
see that the K-means algorithm corresponds to a particular nonprobabilistic limit of
EM applied to mixtures of Gaussians. Finally, we discuss EM in some generality. Section 9.4
Gaussian mixture models are widely used in data mining, pattern recognition,
machine learning, and statistical analysis. In many applications, their parameters are
determined by maximum likelihood, typically using the EM algorithm. However, as

we shall see there are some signiﬁcant limitations to the maximum likelihood ap-
proach, and in Chapter 10 we shall show that an elegant Bayesian treatment can begiven using the framework of variational inference. This requires little additional
computation compared with EM, and it resolves the principal difﬁculties of maxi-
mum likelihood while also allowing the number of components in the mixture to beinferred automatically from the data.
9.1.K-means Clustering
We begin by considering the problem of identifying groups, or clusters, of data points

in a multidimensional space. Suppose we have a data set {x1,...,xN}consisting
ofNobservations of a random D-dimensional Euclidean variable x. Our goal is to
partition the data set into some number Kof clusters, where we shall suppose for
the moment that the value of Kis given. Intuitively, we might think of a cluster as
comprising a group of data points whose inter-point distances are small compared
with the distances to points outside of the cluster. We can formalize this notion byﬁrst introducing a set of D-dimensional vectors µ

k, where k=1,...,K , in which
µkis a prototype associated with the kthcluster. As we shall see shortly, we can
think of the µkas representing the centres of the clusters. Our goal is then to ﬁnd
an assignment of data points to clusters, as well as a set of vectors {µk}, such that
the sum of the squares of the distances of each data point to its closest vector µk,i s
a minimum.
It is convenient at this point to deﬁne some notation to describe the assignment
of data points to clusters. For each data point xn, we introduce a corresponding set

a minimum.
of binary indicator variables rnk∈{0,1}, where k=1,...,K describing which of
theKclusters the data point xnis assigned to, so that if data point xnis assigned to
cluster kthenrnk=1, andrnj=0forj̸=k. This is known as the 1-of-Kcoding
scheme. We can then deﬁne an objective function, sometimes called a distortion
measure , given by
J=N∑
n=1K∑
k=1rnk∥xn−µk∥2(9.1)
which represents the sum of the squares of the distances of each data point to its

9.1.K-means Clustering 425
assigned vector µk. Our goal is to ﬁnd values for the {rnk}and the {µk}so as to
minimize J. We can do this through an iterative procedure in which each iteration
involves two successive steps corresponding to successive optimizations with respect
to thernkand the µk. First we choose some initial values for the µk. Then in the ﬁrst
phase we minimize Jwith respect to the rnk, keeping the µkﬁxed. In the second
phase we minimize Jwith respect to the µk, keeping rnkﬁxed. This two-stage
optimization is then repeated until convergence. We shall see that these two stages
of updating rnkand updating µkcorrespond respectively to the E (expectation) and
M (maximization) steps of the EM algorithm, and to emphasize this we shall use the Section 9.4
terms E step and M step in the context of the K-means algorithm.
Consider ﬁrst the determination of the rnk. Because Jin (9.1) is a linear func-
tion of rnk, this optimization can be performed easily to give a closed form solution.

The terms involving different nare independent and so we can optimize for each
nseparately by choosing rnkto be1for whichever value of kgives the minimum
value of ∥xn−µk∥2. In other words, we simply assign the nthdata point to the
closest cluster centre. More formally, this can be expressed as
rnk={1ifk=a r gm i nj∥xn−µj∥2
0otherwise .(9.2)
Now consider the optimization of the µkwith the rnkheld ﬁxed. The objective
function Jis a quadratic function of µk, and it can be minimized by setting its

derivative with respect to µkto zero giving
2N∑
n=1rnk(xn−µk)=0 (9.3)
which we can easily solve for µkto give
µk=∑
nrnkxn∑
nrnk. (9.4)
The denominator in this expression is equal to the number of points assigned to
cluster k, and so this result has a simple interpretation, namely set µkequal to the
mean of all of the data points xnassigned to cluster k. For this reason, the procedure
is known as the K-means algorithm.
The two phases of re-assigning data points to clusters and re-computing the clus-

2N∑
µk=∑
nrnkxn∑
ter means are repeated in turn until there is no further change in the assignments (or
until some maximum number of iterations is exceeded). Because each phase reduces
the value of the objective function J, convergence of the algorithm is assured. How- Exercise 9.1
ever, it may converge to a local rather than global minimum of J. The convergence
properties of the K-means algorithm were studied by MacQueen (1967).
TheK-means algorithm is illustrated using the Old Faithful data set in Fig- Appendix A

ure 9.1. For the purposes of this example, we have made a linear re-scaling of thedata, known as standardizing , such that each of the variables has zero mean and
unit standard deviation. For this example, we have chosen K=2, and so in this

426 9. MIXTURE MODELS AND EM
(a)
−2 0 2−202 (b)
−2 0 2−202 (c)
−2 0 2−202
(d)
−2 0 2−202 (e)
−2 0 2−202 (f)
−2 0 2−202
(g)
−2 0 2−202 (h)
−2 0 2−202 (i)
−2 0 2−202
Figure 9.1 Illustration of the K-means algorithm using the re-scaled Old Faithful data set. (a) Green points
denote the data set in a two-dimensional Euclidean space. The initial choices for centres µ1andµ2are shown
by the red and blue crosses, respectively. (b) In the initial E step, each data point is assigned either to the red
cluster or to the blue cluster, according to which cluster centre is nearer. This is equivalent to classifying the
points according to which side of the perpendicular bisector of the two cluster centres, shown by the magenta
line, they lie on. (c) In the subsequent M step, each cluster centre is re-computed to be the mean of the points
assigned to the corresponding cluster. (d)–(i) show successive E and M steps through to ﬁnal convergence of
the algorithm.

9.1.K-means Clustering 427
Figure 9.2 Plot of the cost function Jgiven by
(9.1) after each E step (blue points)
and M step (red points) of the K-
means algorithm for the example
shown in Figure 9.1. The algo-
rithm has converged after the third
M step, and the ﬁnal EM cycle pro-
duces no changes in either the as-
signments or the prototype vectors.J
1 2 3 405001000
case, the assignment of each data point to the nearest cluster centre is equivalent to a
classiﬁcation of the data points according to which side they lie of the perpendicular
bisector of the two cluster centres. A plot of the cost function Jgiven by (9.1) for
the Old Faithful example is shown in Figure 9.2.
Note that we have deliberately chosen poor initial values for the cluster centres
so that the algorithm takes several steps before convergence. In practice, a better
initialization procedure would be to choose the cluster centres µkto be equal to a
random subset of Kdata points. It is also worth noting that the K-means algorithm

itself is often used to initialize the parameters in a Gaussian mixture model before
applying the EM algorithm. Section 9.2.2
A direct implementation of the K-means algorithm as discussed here can be
relatively slow, because in each E step it is necessary to compute the Euclidean dis-
tance between every prototype vector and every data point. Various schemes have
been proposed for speeding up the K-means algorithm, some of which are based on
precomputing a data structure such as a tree such that nearby points are in the same

subtree (Ramasubramanian and Paliwal, 1990; Moore, 2000). Other approaches
make use of the triangle inequality for distances, thereby avoiding unnecessary dis-
tance calculations (Hodgson, 1998; Elkan, 2003).
So far, we have considered a batch version of K-means in which the whole data
set is used together to update the prototype vectors. We can also derive an on-line
stochastic algorithm (MacQueen, 1967) by applying the Robbins-Monro procedure Section 2.3.5
to the problem of ﬁnding the roots of the regression function given by the derivatives

ofJin (9.1) with respect to µk. This leads to a sequential update in which, for each Exercise 9.2
data point xnin turn, we update the nearest prototype µkusing
µnew
k=µold
k+ηn(xn−µold
k) (9.5)
where ηnis the learning rate parameter, which is typically made to decrease mono-
tonically as more data points are considered.
TheK-means algorithm is based on the use of squared Euclidean distance as the
measure of dissimilarity between a data point and a prototype vector. Not only does

µnew
k=µold
k) (9.5)
this limit the type of data variables that can be considered (it would be inappropriate
for cases where some or all of the variables represent categorical labels for instance),

428 9. MIXTURE MODELS AND EM
but it can also make the determination of the cluster means nonrobust to outliers. We Section 2.3.7
can generalize the K-means algorithm by introducing a more general dissimilarity
measure V(x,x′)between two vectors xandx′and then minimizing the following
distortion measure
˜J=N∑
n=1K∑
k=1rnkV(xn,µk) (9.6)
which gives the K-medoids algorithm. The E step again involves, for given cluster
prototypes µk, assigning each data point to the cluster for which the dissimilarity to
the corresponding prototype is smallest. The computational cost of this is O(KN),
as is the case for the standard K-means algorithm. For a general choice of dissimi-
larity measure, the M step is potentially more complex than for K-means, and so it
is common to restrict each cluster prototype to be equal to one of the data vectors as-
signed to that cluster, as this allows the algorithm to be implemented for any choice
of dissimilarity measure V(·,·)so long as it can be readily evaluated. Thus the M

step involves, for each cluster k, a discrete search over the Nkpoints assigned to that
cluster, which requires O(N2
k)evaluations of V(·,·).
One notable feature of the K-means algorithm is that at each iteration, every
data point is assigned uniquely to one, and only one, of the clusters. Whereas some
data points will be much closer to a particular centre µkthan to any other centre,

there may be other data points that lie roughly midway between cluster centres. Inthe latter case, it is not clear that the hard assignment to the nearest cluster is the
most appropriate. We shall see in the next section that by adopting a probabilistic
approach, we obtain ‘soft’ assignments of data points to clusters in a way that reﬂectsthe level of uncertainty over the most appropriate assignment. This probabilistic
formulation brings with it numerous beneﬁts.
9.1.1 Image segmentation and compression

As an illustration of the application of the K-means algorithm, we consider
the related problems of image segmentation and image compression. The goal ofsegmentation is to partition an image into regions each of which has a reasonably
homogeneous visual appearance or which corresponds to objects or parts of objects
(Forsyth and Ponce, 2003). Each pixel in an image is a point in a 3-dimensional spacecomprising the intensities of the red, blue, and green channels, and our segmentation

algorithm simply treats each pixel in the image as a separate data point. Note that
strictly this space is not Euclidean because the channel intensities are bounded bythe interval [0,1]. Nevertheless, we can apply the K-means algorithm without difﬁ-
culty. We illustrate the result of running K-means to convergence, for any particular
value of K, by re-drawing the image replacing each pixel vector with the {R, G, B }
intensity triplet given by the centre µ
kto which that pixel has been assigned. Results

for various values of Kare shown in Figure 9.3. We see that for a given value of K,
the algorithm is representing the image using a palette of only Kcolours. It should
be emphasized that this use of K-means is not a particularly sophisticated approach
to image segmentation, not least because it takes no account of the spatial proximityof different pixels. The image segmentation problem is in general extremely difﬁcult

9.1.K-means Clustering 429
K=2
 K=3
 K=1 0
 Original image
Figure 9.3 Two examples of the application of the K-means clustering algorithm to image segmentation show-
ing the initial images together with their K-means segmentations obtained using various values of K. This
also illustrates of the use of vector quantization for data compression, in which smaller values of Kgive higher
compression at the expense of poorer image quality.
and remains the subject of active research and is introduced here simply to illustrate
the behaviour of the K-means algorithm.
We can also use the result of a clustering algorithm to perform data compres-
sion. It is important to distinguish between lossless data compression , in which
the goal is to be able to reconstruct the original data exactly from the compressed
representation, and lossy data compression , in which we accept some errors in the
reconstruction in return for higher levels of compression than can be achieved in the

lossless case. We can apply the K-means algorithm to the problem of lossy data
compression as follows. For each of the Ndata points, we store only the identity
kof the cluster to which it is assigned. We also store the values of the Kclus-
ter centres µk, which typically requires signiﬁcantly less data, provided we choose
K≪N. Each data point is then approximated by its nearest centre µk. New data
points can similarly be compressed by ﬁrst ﬁnding the nearest µkand then storing
the label kinstead of the original data vector. This framework is often called vector

quantization , and the vectors µkare called code-book vectors .

430 9. MIXTURE MODELS AND EM
The image segmentation problem discussed above also provides an illustration
of the use of clustering for data compression. Suppose the original image has N
pixels comprising {R, G, B }values each of which is stored with 8 bits of precision.
Then to transmit the whole image directly would cost 24Nbits. Now suppose we
ﬁrst run K-means on the image data, and then instead of transmitting the original
pixel intensity vectors we transmit the identity of the nearest vector µk. Because
there are Ksuch vectors, this requires log2Kbits per pixel. We must also transmit
theKcode book vectors µk, which requires 24 Kbits, and so the total number of
bits required to transmit the image is 24K+Nlog2K(rounding up to the nearest
integer). The original image shown in Figure 9.3 has 240×180 = 43 ,200pixels
and so requires 24×43,200 = 1 ,036,800bits to transmit directly. By comparison,
the compressed images require 43,248bits (K=2),86,472bits (K=3), and

173,040bits (K=1 0 ), respectively, to transmit. These represent compression ratios
compared to the original image of 4.2%, 8.3%, and 16.7%, respectively. We see that
there is a trade-off between degree of compression and image quality. Note that our
aim in this example is to illustrate the K-means algorithm. If we had been aiming to
produce a good image compressor, then it would be more fruitful to consider small
blocks of adjacent pixels, for instance 5×5, and thereby exploit the correlations that
exist in natural images between nearby pixels.

9.2. Mixtures of Gaussians
In Section 2.3.9 we motivated the Gaussian mixture model as a simple linear super-position of Gaussian components, aimed at providing a richer class of density mod-els than the single Gaussian. We now turn to a formulation of Gaussian mixtures in
terms of discrete latent variables. This will provide us with a deeper insight into this
important distribution, and will also serve to motivate the expectation-maximizationalgorithm.
Recall from (2.188) that the Gaussian mixture distribution can be written as a

linear superposition of Gaussians in the form
p(x)=
K∑
k=1πkN(x|µk,Σk). (9.7)
Let us introduce a K-dimensional binary random variable zhaving a 1-of- Krepre-
sentation in which a particular element zkis equal to 1and all other elements are
equal to 0. The values of zktherefore satisfy zk∈{0,1}and∑
kzk=1, and we
see that there are Kpossible states for the vector zaccording to which element is
nonzero. We shall deﬁne the joint distribution p(x,z)in terms of a marginal dis-

p(x)=
K∑
tribution p(z)and a conditional distribution p(x|z), corresponding to the graphical
model in Figure 9.4. The marginal distribution over zis speciﬁed in terms of the
mixing coefﬁcients πk, such that
p(zk=1 )= πk

9.2. Mixtures of Gaussians 431
Figure 9.4 Graphical representation of a mixture model, in which
the joint distribution is expressed in the form p(x,z)=
p(z)p(x|z).
xz
where the parameters {πk}must satisfy
0⩽πk⩽1 (9.8)
K∑
k=1πk=1 (9.9)
in order to be valid probabilities. Because zuses a 1-of- Krepresentation, we can
also write this distribution in the form
p(z)=K∏
k=1πzk
k. (9.10)
Similarly, the conditional distribution of xgiven a particular value for zis a Gaussian
p(x|zk=1 )= N(x|µk,Σk)
which can also be written in the form
p(x|z)=K∏
k=1N(x|µk,Σk)zk. (9.11)
The joint distribution is given by p(z)p(x|z), and the marginal distribution of xis
then obtained by summing the joint distribution over all possible states of zto give Exercise 9.3
p(x)=∑
zp(z)p(x|z)=K∑
k=1πkN(x|µk,Σk) (9.12)
where we have made use of (9.10) and (9.11). Thus the marginal distribution of xis
a Gaussian mixture of the form (9.7). If we have several observations x1,...,xN,

p(x|z)=K∏
p(x)=∑
then, because we have represented the marginal distribution in the form p(x)=∑
zp(x,z), it follows that for every observed data point xnthere is a corresponding
latent variable zn.
We have therefore found an equivalent formulation of the Gaussian mixture in-
volving an explicit latent variable. It might seem that we have not gained much
by doing so. However, we are now able to work with the joint distribution p(x,z)

432 9. MIXTURE MODELS AND EM
instead of the marginal distribution p(x), and this will lead to signiﬁcant simpliﬁca-
tions, most notably through the introduction of the expectation-maximization (EM)algorithm.
Another quantity that will play an important role is the conditional probability
ofzgivenx. We shall use γ(z
k)to denote p(zk=1|x), whose value can be found
using Bayes’ theorem
γ(zk)≡p(zk=1|x)=p(zk=1 )p(x|zk=1 )
K∑
j=1p(zj=1 )p(x|zj=1 )
=πkN(x|µk,Σk)
K∑
j=1πjN(x|µj,Σj). (9.13)
We shall view πkas the prior probability of zk=1, and the quantity γ(zk)as the
corresponding posterior probability once we have observed x. As we shall see later,
γ(zk)can also be viewed as the responsibility that component ktakes for ‘explain-
ing’ the observation x.
We can use the technique of ancestral sampling to generate random samples Section 8.1.2
distributed according to the Gaussian mixture model. To do this, we ﬁrst generate avalue for z, which we denote
ˆz, from the marginal distribution p(z)and then generate

a value for xfrom the conditional distribution p(x|ˆz). Techniques for sampling from
standard distributions are discussed in Chapter 11. We can depict samples from the
joint distribution p(x,z)by plotting points at the corresponding values of xand
then colouring them according to the value of z, in other words according to which
Gaussian component was responsible for generating them, as shown in Figure 9.5(a).
Similarly samples from the marginal distribution p(x)are obtained by taking the

samples from the joint distribution and ignoring the values of z. These are illustrated
in Figure 9.5(b) by plotting the xvalues without any coloured labels.
We can also use this synthetic data set to illustrate the ‘responsibilities’ by eval-
uating, for every data point, the posterior probability for each component in themixture distribution from which this data set was generated. In particular, we can
represent the value of the responsibilities γ(z
nk)associated with data point xnby

plotting the corresponding point using proportions of red, blue, and green ink givenbyγ(z
nk)fork=1,2,3, respectively, as shown in Figure 9.5(c). So, for instance,
a data point for which γ(zn1)=1 will be coloured red, whereas one for which
γ(zn2)=γ(zn3)=0.5will be coloured with equal proportions of blue and green
ink and so will appear cyan. This should be compared with Figure 9.5(a) in which
the data points were labelled using the true identity of the component from which
they were generated.
9.2.1 Maximum likelihood

Suppose we have a data set of observations {x1,...,xN}, and we wish to model
this data using a mixture of Gaussians. We can represent this data set as an N×D

9.2. Mixtures of Gaussians 433
(a)
0 0.5 100.51(c)
Figure 9.5 Example of 500 points drawn from the mixture of 3 Gaussians shown in Figure 2.23. (a) Samples
from the joint distribution p(z)p(x|z)in which the three states of z, corresponding to the three components of the
mixture, are depicted in red, green, and blue, and (b) the corresponding samples from the marginal distribution
p(x), which is obtained by simply ignoring the values of zand just plotting the xvalues. The data set in (a) is
said to be complete , whereas that in (b) is incomplete . (c) The same samples in which the colours represent the
value of the responsibilities γ(znk)associated with data point xn, obtained by plotting the corresponding point
using proportions of red, blue, and green ink given by γ(znk)fork=1,2,3, respectively
matrix Xin which the nthrow is given by xT
n. Similarly, the corresponding latent
variables will be denoted by an N×Kmatrix Zwith rows zT
n. If we assume that

the data points are drawn independently from the distribution, then we can express
the Gaussian mixture model for this i.i.d. data set using the graphical representation
shown in Figure 9.6. From (9.7) the log of the likelihood function is given by
n=1ln{K∑
. (9.14)
Before discussing how to maximize this function, it is worth emphasizing that
there is a signiﬁcant problem associated with the maximum likelihood framework
applied to Gaussian mixture models, due to the presence of singularities. For sim-

n=1ln{K∑
. (9.14)
plicity, consider a Gaussian mixture whose components have covariance matrices
given by Σk=σ2
kI, where Iis the unit matrix, although the conclusions will hold
for general covariance matrices. Suppose that one of the components of the mixture
model, let us say the jthcomponent, has its mean µjexactly equal to one of the data
Figure 9.6 Graphical representation of a Gaussian mixture model
for a set of Ni.i.d. data points {xn}, with corresponding
latent points {zn}, where n=1,...,N .
xnzn
Nµ Σπ

434 9. MIXTURE MODELS AND EM
Figure 9.7 Illustration of how singularities in the
likelihood function arise with mixtures
of Gaussians. This should be com-
pared with the case of a single Gaus-
sian shown in Figure 1.14 for which no
singularities arise.
xp(x)
points so that µj=xnfor some value of n. This data point will then contribute a
term in the likelihood function of the form
N(xn|xn,σ2
jI)=1
(2π)1/21
σj. (9.15)
If we consider the limit σj→0, then we see that this term goes to inﬁnity and
so the log likelihood function will also go to inﬁnity. Thus the maximization of
the log likelihood function is not a well posed problem because such singularities
will always be present and will occur whenever one of the Gaussian components
‘collapses’ onto a speciﬁc data point. Recall that this problem did not arise in the
case of a single Gaussian distribution. To understand the difference, note that if a
single Gaussian collapses onto a data point it will contribute multiplicative factors

to the likelihood function arising from the other data points and these factors will go
to zero exponentially fast, giving an overall likelihood that goes to zero rather than
inﬁnity. However, once we have (at least) two components in the mixture, one of
the components can have a ﬁnite variance and therefore assign ﬁnite probability to
all of the data points while the other component can shrink onto one speciﬁc data
point and thereby contribute an ever increasing additive value to the log likelihood.

This is illustrated in Figure 9.7. These singularities provide another example of the
severe over-ﬁtting that can occur in a maximum likelihood approach. We shall see
that this difﬁculty does not occur if we adopt a Bayesian approach. For the moment, Section 10.1
however, we simply note that in applying maximum likelihood to Gaussian mixture
models we must take steps to avoid ﬁnding such pathological solutions and instead
seek local maxima of the likelihood function that are well behaved. We can hope to

avoid the singularities by using suitable heuristics, for instance by detecting when a
Gaussian component is collapsing and resetting its mean to a randomly chosen value
while also resetting its covariance to some large value, and then continuing with the
optimization.
A further issue in ﬁnding maximum likelihood solutions arises from the fact
that for any given maximum likelihood solution, a K-component mixture will have
a total of K!equivalent solutions corresponding to the K!ways of assigning K

sets of parameters to Kcomponents. In other words, for any given (nondegenerate)
point in the space of parameter values there will be a further K!−1additional points
all of which give rise to exactly the same distribution. This problem is known as

9.2. Mixtures of Gaussians 435
identiﬁability (Casella and Berger, 2002) and is an important issue when we wish to
interpret the parameter values discovered by a model. Identiﬁability will also arise
when we discuss models having continuous latent variables in Chapter 12. However,
for the purposes of ﬁnding a good density model, it is irrelevant because any of the
equivalent solutions is as good as any other.
Maximizing the log likelihood function (9.14) for a Gaussian mixture model
turns out to be a more complex problem than for the case of a single Gaussian. The
difﬁculty arises from the presence of the summation over kthat appears inside the
logarithm in (9.14), so that the logarithm function no longer acts directly on the
Gaussian. If we set the derivatives of the log likelihood to zero, we will no longer
obtain a closed form solution, as we shall see shortly.
One approach is to apply gradient-based optimization techniques (Fletcher, 1987;

Nocedal and Wright, 1999; Bishop and Nabney, 2008). Although gradient-based
techniques are feasible, and indeed will play an important role when we discuss
mixture density networks in Chapter 5, we now consider an alternative approach
known as the EM algorithm which has broad applicability and which will lay the
foundations for a discussion of variational inference techniques in Chapter 10.
9.2.2 EM for Gaussian mixtures
An elegant and powerful method for ﬁnding maximum likelihood solutions for

models with latent variables is called the expectation-maximization algorithm, or EM
algorithm (Dempster et al. , 1977; McLachlan and Krishnan, 1997). Later we shall
give a general treatment of EM, and we shall also show how EM can be generalized
to obtain the variational inference framework. Initially, we shall motivate the EM Section 10.1
algorithm by giving a relatively informal treatment in the context of the Gaussian
mixture model. We emphasize, however, that EM has broad applicability, and indeed

it will be encountered in the context of a variety of different models in this book.
Let us begin by writing down the conditions that must be satisﬁed at a maximum
of the likelihood function. Setting the derivatives of lnp(X|π,µ,Σ)in (9.14) with
respect to the means µkof the Gaussian components to zero, we obtain
0=−N∑
n=1πkN(xn|µk,Σk)∑
jπjN(xn|µj,Σj)

γ(znk)Σk(xn−µk) (9.16)
where we have made use of the form (2.43) for the Gaussian distribution. Note that

0=−N∑

the posterior probabilities, or responsibilities, given by (9.13) appear naturally on
the right-hand side. Multiplying by Σ−1
k(which we assume to be nonsingular) and
rearranging we obtain
µk=1
NkN∑
n=1γ(znk)xn (9.17)
Nk=N∑
n=1γ(znk). (9.18)

436 9. MIXTURE MODELS AND EM
We can interpret Nkas the effective number of points assigned to cluster k. Note
carefully the form of this solution. We see that the mean µkfor the kthGaussian
component is obtained by taking a weighted mean of all of the points in the data set,
in which the weighting factor for data point xnis given by the posterior probability
γ(znk)that component kwas responsible for generating xn.
If we set the derivative of lnp(X|π,µ,Σ)with respect to Σkto zero, and follow
a similar line of reasoning, making use of the result for the maximum likelihood
solution for the covariance matrix of a single Gaussian, we obtain Section 2.3.4
Σk=1
NkN∑
n=1γ(znk)(xn−µk)(xn−µk)T(9.19)
which has the same form as the corresponding result for a single Gaussian ﬁtted to
the data set, but again with each data point weighted by the corresponding poste-
rior probability and with the denominator given by the effective number of pointsassociated with the corresponding component.

Σk=1
NkN∑
Finally, we maximize lnp(X|π,µ,Σ)with respect to the mixing coefﬁcients
π
k. Here we must take account of the constraint (9.9), which requires the mixing
coefﬁcients to sum to one. This can be achieved using a Lagrange multiplier and Appendix E
maximizing the following quantity
lnp(X|π,µ,Σ)+λ(K∑
k=1πk−1)
(9.20)
0=N∑
n=1N(xn|µk,Σk)∑
jπjN(xn|µj,Σj)+λ (9.21)
where again we see the appearance of the responsibilities. If we now multiply both
sides by πkand sum over kmaking use of the constraint (9.9), we ﬁnd λ=−N.

π
k=1πk−1)
(9.20)
0=N∑
Using this to eliminate λand rearranging we obtain
πk=Nk
N(9.22)
so that the mixing coefﬁcient for the kthcomponent is given by the average respon-
sibility which that component takes for explaining the data points.
It is worth emphasizing that the results (9.17), (9.19), and (9.22) do not con-
stitute a closed-form solution for the parameters of the mixture model because the
responsibilities γ(znk)depend on those parameters in a complex way through (9.13).

πk=Nk
N(9.22)
However, these results do suggest a simple iterative scheme for ﬁnding a solution tothe maximum likelihood problem, which as we shall see turns out to be an instance
of the EM algorithm for the particular case of the Gaussian mixture model. We
ﬁrst choose some initial values for the means, covariances, and mixing coefﬁcients.
Then we alternate between the following two updates that we shall call the E step

9.2. Mixtures of Gaussians 437
(a) −2 0 2−202
(b) −2 0 2−202
(c)L=1
−2 0 2−202
(d)L=2
−2 0 2−202
(e)L=5
−2 0 2−202
(f)L=2 0
−2 0 2−202
Figure 9.8 Illustration of the EM algorithm using the Old Faithful set as used for the illustration of the K-means
algorithm in Figure 9.1. See the text for details.
and the M step, for reasons that will become apparent shortly. In the expectation
step, or E step, we use the current values for the parameters to evaluate the posterior
probabilities, or responsibilities, given by (9.13). We then use these probabilities in
themaximization step, or M step, to re-estimate the means, covariances, and mix-
ing coefﬁcients using the results (9.17), (9.19), and (9.22). Note that in so doing
we ﬁrst evaluate the new means using (9.17) and then use these new values to ﬁnd
the covariances using (9.19), in keeping with the corresponding result for a single
Gaussian distribution. We shall show that each update to the parameters resulting

from an E step followed by an M step is guaranteed to increase the log likelihood
function. In practice, the algorithm is deemed to have converged when the change Section 9.4
in the log likelihood function, or alternatively in the parameters, falls below some
threshold. We illustrate the EM algorithm for a mixture of two Gaussians applied to
the rescaled Old Faithful data set in Figure 9.8. Here a mixture of two Gaussians
is used, with centres initialized using the same values as for the K-means algorithm

in Figure 9.1, and with precision matrices initialized to be proportional to the unit
matrix. Plot (a) shows the data points in green, together with the initial conﬁgura-
tion of the mixture model in which the one standard-deviation contours for the two

438 9. MIXTURE MODELS AND EM
Gaussian components are shown as blue and red circles. Plot (b) shows the result
of the initial E step, in which each data point is depicted using a proportion of blueink equal to the posterior probability of having been generated from the blue com-
ponent, and a corresponding proportion of red ink given by the posterior probability
of having been generated by the red component. Thus, points that have a signiﬁcantprobability for belonging to either cluster appear purple. The situation after the ﬁrst
M step is shown in plot (c), in which the mean of the blue Gaussian has moved to
the mean of the data set, weighted by the probabilities of each data point belongingto the blue cluster, in other words it has moved to the centre of mass of the blue ink.
Similarly, the covariance of the blue Gaussian is set equal to the covariance of the
blue ink. Analogous results hold for the red component. Plots (d), (e), and (f) show

the results after 2, 5, and 20 complete cycles of EM, respectively. In plot (f) the
algorithm is close to convergence.
Note that the EM algorithm takes many more iterations to reach (approximate)
convergence compared with the K-means algorithm, and that each cycle requires
signiﬁcantly more computation. It is therefore common to run the K-means algo-
rithm in order to ﬁnd a suitable initialization for a Gaussian mixture model that is
subsequently adapted using EM. The covariance matrices can conveniently be ini-
tialized to the sample covariances of the clusters found by the K-means algorithm,

and the mixing coefﬁcients can be set to the fractions of data points assigned to the
respective clusters. As with gradient-based approaches for maximizing the log like-
lihood, techniques must be employed to avoid singularities of the likelihood functionin which a Gaussian component collapses onto a particular data point. It should be
emphasized that there will generally be multiple local maxima of the log likelihood

function, and that EM is not guaranteed to ﬁnd the largest of these maxima. Becausethe EM algorithm for Gaussian mixtures plays such an important role, we summarize
it below.
EM for Gaussian Mixtures
Given a Gaussian mixture model, the goal is to maximize the likelihood function
with respect to the parameters (comprising the means and covariances of the
components and the mixing coefﬁcients).
1. Initialize the means µ
k, covariances Σkand mixing coefﬁcients πk, and

it below.
evaluate the initial value of the log likelihood.
2.E step . Evaluate the responsibilities using the current parameter values
γ(znk)=πkN(xn|µk,Σk)
K∑
j=1πjN(xn|µj,Σj). (9.23)

9.3. An Alternative View of EM 439
3.M step . Re-estimate the parameters using the current responsibilities
µnew
k=1
NkN∑
n=1γ(znk)xn (9.24)
Σnew
k=1
NkN∑
n=1γ(znk)(xn−µnew
k)(xn−µnew
k)T(9.25)
πnew
k=Nk
N(9.26)
where
Nk=N∑
n=1γ(znk). (9.27)
4. Evaluate the log likelihood
lnp(X|µ,Σ,π)=N∑
n=1ln{K∑
(9.28)
and check for convergence of either the parameters or the log likelihood. If
the convergence criterion is not satisﬁed return to step 2.
9.3. An Alternative View of EM
In this section, we present a complementary view of the EM algorithm that recog-nizes the key role played by latent variables. We discuss this approach ﬁrst of all
in an abstract setting, and then for illustration we consider once again the case of
Gaussian mixtures.
The goal of the EM algorithm is to ﬁnd maximum likelihood solutions for mod-
els having latent variables. We denote the set of all observed data by X, in which the
n
throw represents xT
n, and similarly we denote the set of all latent variables by Z,

n
with a corresponding row zT
n. The set of all model parameters is denoted by θ, and
so the log likelihood function is given by
lnp(X|θ)=l n{∑
Zp(X,Z|θ)}
. (9.29)
Note that our discussion will apply equally well to continuous latent variables simply
by replacing the sum over Zwith an integral.
A key observation is that the summation over the latent variables appears inside
the logarithm. Even if the joint distribution p(X,Z|θ)belongs to the exponential

440 9. MIXTURE MODELS AND EM
family, the marginal distribution p(X|θ)typically does not as a result of this sum-
mation. The presence of the sum prevents the logarithm from acting directly on thejoint distribution, resulting in complicated expressions for the maximum likelihood
solution.
Now suppose that, for each observation in X, we were told the corresponding
value of the latent variable Z. We shall call {X,Z}thecomplete data set, and we
shall refer to the actual observed data Xasincomplete , as illustrated in Figure 9.5.
The likelihood function for the complete data set simply takes the form lnp(X,Z|θ),
and we shall suppose that maximization of this complete-data log likelihood function
is straightforward.
In practice, however, we are not given the complete data set {X,Z}, but only
the incomplete data X. Our state of knowledge of the values of the latent variables
inZis given only by the posterior distribution p(Z|X,θ). Because we cannot use

the complete-data log likelihood, we consider instead its expected value under the
posterior distribution of the latent variable, which corresponds (as we shall see) to the
E step of the EM algorithm. In the subsequent M step, we maximize this expectation.If the current estimate for the parameters is denoted θ
old, then a pair of successive
E and M steps gives rise to a revised estimate θnew. The algorithm is initialized by
choosing some starting value for the parameters θ0. The use of the expectation may
seem somewhat arbitrary. However, we shall see the motivation for this choice when

we give a deeper treatment of EM in Section 9.4.
In the E step, we use the current parameter values θoldto ﬁnd the posterior
distribution of the latent variables given by p(Z|X,θold). We then use this posterior
distribution to ﬁnd the expectation of the complete-data log likelihood evaluated for
some general parameter value θ. This expectation, denoted Q(θ,θold),i sg i v e nb y
Q(θ,θold)=∑
Zp(Z|X,θold)l np(X,Z|θ). (9.30)
In the M step, we determine the revised parameter estimate θnewby maximizing this

function
θnew=a r gm a x
θQ(θ,θold). (9.31)
Note that in the deﬁnition of Q(θ,θold), the logarithm acts directly on the joint
distribution p(X,Z|θ), and so the corresponding M-step maximization will, by sup-
position, be tractable.
The general EM algorithm is summarized below. It has the property, as we shall
show later, that each cycle of EM will increase the incomplete-data log likelihood(unless it is already at a local maximum). Section 9.4
The General EM Algorithm

function
Given a joint distribution p(X,Z|θ)over observed variables Xand latent vari-
ablesZ, governed by parameters θ, the goal is to maximize the likelihood func-
tionp(X|θ)with respect to θ.
1. Choose an initial setting for the parameters θ
old.

9.3. An Alternative View of EM 441
2.E step Evaluate p(Z|X,θold).
3.M step Evaluate θnewgiven by
θQ(θ,θold) (9.32)
where
Zp(Z|X,θold)l np(X,Z|θ). (9.33)
4. Check for convergence of either the log likelihood or the parameter values.
If the convergence criterion is not satisﬁed, then let
θold←θnew(9.34)
and return to step 2.
The EM algorithm can also be used to ﬁnd MAP (maximum posterior) solutions
for models in which a prior p(θ)is deﬁned over the parameters. In this case the E Exercise 9.4
step remains the same as in the maximum likelihood case, whereas in the M step the
quantity to be maximized is given by Q(θ,θold)+l n p(θ). Suitable choices for the
prior will remove the singularities of the kind illustrated in Figure 9.7.
Here we have considered the use of the EM algorithm to maximize a likelihood
function when there are discrete latent variables. However, it can also be applied

when the unobserved variables correspond to missing values in the data set. Thedistribution of the observed values is obtained by taking the joint distribution of all
the variables and then marginalizing over the missing ones. EM can then be used
to maximize the corresponding likelihood function. We shall show an example ofthe application of this technique in the context of principal component analysis in
Figure 12.11. This will be a valid procedure if the data values are missing at random ,

meaning that the mechanism causing values to be missing does not depend on the
unobserved values. In many situations this will not be the case, for instance if a
sensor fails to return a value whenever the quantity it is measuring exceeds somethreshold.
9.3.1 Gaussian mixtures revisited
We now consider the application of this latent variable view of EM to the spe-
ciﬁc case of a Gaussian mixture model. Recall that our goal is to maximize the log

likelihood function (9.14), which is computed using the observed data set X, and we
saw that this was more difﬁcult than for the case of a single Gaussian distribution
due to the presence of the summation over kthat occurs inside the logarithm. Sup-
pose then that in addition to the observed data set X, we were also given the values
of the corresponding discrete variables Z. Recall that Figure 9.5(a) shows a ‘com-

plete’ data set (i.e., one that includes labels showing which component generatedeach data point) while Figure 9.5(b) shows the corresponding ‘incomplete’ data set.
The graphical model for the complete data is shown in Figure 9.9.

442 9. MIXTURE MODELS AND EM
Figure 9.9 This shows the same graph as in Figure 9.6 except that
we now suppose that the discrete variables znare ob-
served, as well as the data variables xn.
xnzn
Nµ Σπ
Now consider the problem of maximizing the likelihood for the complete data
set{X,Z}. From (9.10) and (9.11), this likelihood function takes the form
p(X,Z|µ,Σ,π)=N∏
n=1K∏
k=1πznk
kN(xn|µk,Σk)znk(9.35)
where znkdenotes the kthcomponent of zn. Taking the logarithm, we obtain
lnp(X,Z|µ,Σ,π)=N∑
n=1K∑
k=1znk{lnπk+l nN(xn|µk,Σk)}. (9.36)
Comparison with the log likelihood function (9.14) for the incomplete data shows
that the summation over kand the logarithm have been interchanged. The loga-
rithm now acts directly on the Gaussian distribution, which itself is a member of
the exponential family. Not surprisingly, this leads to a much simpler solution to
the maximum likelihood problem, as we now show. Consider ﬁrst the maximization
with respect to the means and covariances. Because znis aK-dimensional vec-

tor with all elements equal to 0except for a single element having the value 1, the
complete-data log likelihood function is simply a sum of Kindependent contribu-
tions, one for each mixture component. Thus the maximization with respect to a
mean or a covariance is exactly as for a single Gaussian, except that it involves only
the subset of data points that are ‘assigned’ to that component. For the maximization
with respect to the mixing coefﬁcients, we note that these are coupled for different

values of kby virtue of the summation constraint (9.9). Again, this can be enforced
using a Lagrange multiplier as before, and leads to the result
πk=1
NN∑
n=1znk (9.37)
so that the mixing coefﬁcients are equal to the fractions of data points assigned to
the corresponding components.
Thus we see that the complete-data log likelihood function can be maximized
trivially in closed form. In practice, however, we do not have values for the latent

πk=1
NN∑
variables so, as discussed earlier, we consider the expectation, with respect to the
posterior distribution of the latent variables, of the complete-data log likelihood.

9.3. An Alternative View of EM 443
Using (9.10) and (9.11) together with Bayes’ theorem, we see that this posterior
distribution takes the form
p(Z|X,µ,Σ,π)∝N∏
n=1K∏
k=1[πkN(xn|µk,Σk)]znk. (9.38)
and hence factorizes over nso that under the posterior distribution the {zn}are
independent. This is easily veriﬁed by inspection of the directed graph in Figure 9.6 Exercise 9.5
and making use of the d-separation criterion. The expected value of the indicator Section 8.2
variable znkunder this posterior distribution is then given by
E[znk]=∑
znkznk[πkN(xn|µk,Σk)]znk
∑
znj[
πjN(xn|µj,Σj)]znj
=πkN(xn|µk,Σk)
K∑
j=1πjN(xn|µj,Σj)=γ(znk) (9.39)
which is just the responsibility of component kfor data point xn. The expected value
of the complete-data log likelihood function is therefore given by
EZ[lnp(X,Z|µ,Σ,π)] =N∑
n=1K∑
k=1γ(znk){lnπk+l nN(xn|µk,Σk)}.(9.40)
We can now proceed as follows. First we choose some initial values for the param-
etersµold,Σoldandπold, and use these to evaluate the responsibilities (the E step).

E[znk]=∑
∑
znj[
K∑
n=1K∑
We then keep the responsibilities ﬁxed and maximize (9.40) with respect to µk,Σk
andπk(the M step). This leads to closed form solutions for µnew,Σnewandπnew
given by (9.17), (9.19), and (9.22) as before. This is precisely the EM algorithm for Exercise 9.8
Gaussian mixtures as derived earlier. We shall gain more insight into the role of the
expected complete-data log likelihood function when we give a proof of convergence
of the EM algorithm in Section 9.4.
9.3.2 Relation to K-means

Comparison of the K-means algorithm with the EM algorithm for Gaussian
mixtures shows that there is a close similarity. Whereas the K-means algorithm
performs a hard assignment of data points to clusters, in which each data point is
associated uniquely with one cluster, the EM algorithm makes a softassignment
based on the posterior probabilities. In fact, we can derive the K-means algorithm
as a particular limit of EM for Gaussian mixtures as follows.
Consider a Gaussian mixture model in which the covariance matrices of the

mixture components are given by ϵI, where ϵis a variance parameter that is shared

444 9. MIXTURE MODELS AND EM
by all of the components, and Iis the identity matrix, so that
p(x|µk,Σk)=1
(2πϵ)1/2exp{
−1
2ϵ∥x−µk∥2}
. (9.41)
We now consider the EM algorithm for a mixture of KGaussians of this form in
which we treat ϵas a ﬁxed constant, instead of a parameter to be re-estimated. From
(9.13) the posterior probabilities, or responsibilities, for a particular data point xn,
are given by
γ(znk)=πkexp{−∥xn−µk∥2/2ϵ}∑
jπjexp{
−∥xn−µj∥2/2ϵ}. (9.42)
If we consider the limit ϵ→0, we see that in the denominator the term for which
∥xn−µj∥2is smallest will go to zero most slowly, and hence the responsibilities
γ(znk)for the data point xnall go to zero except for term j, for which the responsi-
bilityγ(znj)will go to unity. Note that this holds independently of the values of the
πkso long as none of the πkis zero. Thus, in this limit, we obtain a hard assignment
of data points to clusters, just as in the K-means algorithm, so that γ(znk)→rnk

where rnkis deﬁned by (9.2). Each data point is thereby assigned to the cluster
having the closest mean.
The EM re-estimation equation for the µk, given by (9.17), then reduces to the
K-means result (9.4). Note that the re-estimation formula for the mixing coefﬁcients
(9.22) simply re-sets the value of πkto be equal to the fraction of data points assigned
to cluster k, although these parameters no longer play an active role in the algorithm.
Finally, in the limit ϵ→0the expected complete-data log likelihood, given by

(9.40), becomes Exercise 9.11
EZ[lnp(X,Z|µ,Σ,π)]→−1
2N∑
n=1K∑
k=1rnk∥xn−µk∥2+c o n s t . (9.43)
Thus we see that in this limit, maximizing the expected complete-data log likelihood
is equivalent to minimizing the distortion measure Jfor the K-means algorithm
given by (9.1).
Note that the K-means algorithm does not estimate the covariances of the clus-
ters but only the cluster means. A hard-assignment version of the Gaussian mixturemodel with general covariance matrices, known as the elliptical K-means algorithm,
has been considered by Sung and Poggio (1994).

n=1K∑
9.3.3 Mixtures of Bernoulli distributions
So far in this chapter, we have focussed on distributions over continuous vari-
ables described by mixtures of Gaussians. As a further example of mixture mod-
elling, and to illustrate the EM algorithm in a different context, we now discuss mix-tures of discrete binary variables described by Bernoulli distributions. This model
is also known as latent class analysis (Lazarsfeld and Henry, 1968; McLachlan and

Peel, 2000). As well as being of practical importance in its own right, our discus-sion of Bernoulli mixtures will also lay the foundation for a consideration of hidden
Markov models over discrete variables. Section 13.2

9.3. An Alternative View of EM 445
Consider a set of Dbinary variables xi, where i=1,...,D , each of which is
governed by a Bernoulli distribution with parameter µi, so that
p(x|µ)=D∏
i=1µxi
i(1−µi)(1−xi)(9.44)
wherex=(x1,...,x D)Tandµ=(µ1,...,µ D)T. We see that the individual
variables xiare independent, given µ. The mean and covariance of this distribution
are easily seen to be
E[x]= µ (9.45)
cov[x]=d i a g {µi(1−µi)}. (9.46)
Now let us consider a ﬁnite mixture of these distributions given by
p(x|µ,π)=K∑
k=1πkp(x|µk) (9.47)
where µ={µ1,...,µK},π={π1,...,π K}, and
p(x|µk)=D∏
i=1µxi
ki(1−µki)(1−xi). (9.48)
The mean and covariance of this mixture distribution are given by Exercise 9.12
E[x]=K∑
k=1πkµk (9.49)
cov[x]=K∑
k=1πk{
Σk+µkµT
k}
−E[x]E[x]T(9.50)
whereΣk=d i a g {µki(1−µki)}. Because the covariance matrix cov[x]is no
longer diagonal, the mixture distribution can capture correlations between the vari-
ables, unlike a single Bernoulli distribution.

i=1µxi
E[x]=K∑
cov[x]=K∑
k=1πk{
Σk+µkµT
k}
If we are given a data set X={x1,...,xN}then the log likelihood function
for this model is given by
lnp(X|µ,π)=N∑
n=1ln{K∑
k=1πkp(xn|µk)}
. (9.51)
Again we see the appearance of the summation inside the logarithm, so that the
maximum likelihood solution no longer has closed form.
We now derive the EM algorithm for maximizing the likelihood function for
the mixture of Bernoulli distributions. To do this, we ﬁrst introduce an explicit latent

446 9. MIXTURE MODELS AND EM
variable zassociated with each instance of x. As in the case of the Gaussian mixture,
z=(z1,...,z K)Tis a binary K-dimensional variable having a single component
equal to 1, with all other components equal to 0. We can then write the conditional
distribution of x, given the latent variable, as
p(x|z,µ)=K∏
k=1p(x|µk)zk(9.52)
while the prior distribution for the latent variables is the same as for the mixture of
Gaussians model, so that
p(z|π)=K∏
k=1πzk
k. (9.53)
If we form the product of p(x|z,µ)andp(z|π)and then marginalize over z, then we
recover (9.47). Exercise 9.14
In order to derive the EM algorithm, we ﬁrst write down the complete-data log
likelihood function, which is given by
lnp(X,Z|µ,π)=N∑
n=1K∑
k=1znk{
lnπk
+D∑
i=1[xnilnµki+( 1−xni)l n ( 1−µki)]}
(9.54)
whereX={xn}andZ={zn}. Next we take the expectation of the complete-data
log likelihood with respect to the posterior distribution of the latent variables to give
EZ[lnp(X,Z|µ,π)] =N∑
n=1K∑
k=1γ(znk){
lnπk
+D∑

n=1K∑
k=1znk{
lnπk
+D∑
(9.54)
n=1K∑
lnπk
+D∑
(9.55)
where γ(znk)= E[znk]is the posterior probability, or responsibility, of component
kgiven data point xn. In the E step, these responsibilities are evaluated using Bayes’
theorem, which takes the form
γ(znk)= E[znk]=∑
znkznk[πkp(xn|µk)]znk
∑
znj[
πjp(xn|µj)]znj
=πkp(xn|µk)
K∑
j=1πjp(xn|µj). (9.56)

9.3. An Alternative View of EM 447
If we consider the sum over nin (9.55), we see that the responsibilities enter
only through two terms, which can be written as
Nk=N∑
n=1γ(znk) (9.57)
xk=1
NkN∑
n=1γ(znk)xn (9.58)
where Nkis the effective number of data points associated with component k. In the
M step, we maximize the expected complete-data log likelihood with respect to the
parameters µkandπ. If we set the derivative of (9.55) with respect to µkequal to
zero and rearrange the terms, we obtain Exercise 9.15
µk=xk. (9.59)
We see that this sets the mean of component kequal to a weighted mean of the
data, with weighting coefﬁcients given by the responsibilities that component ktakes
for data points. For the maximization with respect to πk, we need to introduce a
Lagrange multiplier to enforce the constraint∑
kπk=1. Following analogous
steps to those used for the mixture of Gaussians, we then obtain Exercise 9.16
πk=Nk
N(9.60)
which represents the intuitively reasonable result that the mixing coefﬁcient for com-

πk=Nk
N(9.60)
ponent kis given by the effective fraction of points in the data set explained by that
component.
Note that in contrast to the mixture of Gaussians, there are no singularities in
which the likelihood function goes to inﬁnity. This can be seen by noting that the
likelihood function is bounded above because 0⩽p(xn|µk)⩽1. There exist Exercise 9.17
singularities at which the likelihood function goes to zero, but these will not be

found by EM provided it is not initialized to a pathological starting point, becausethe EM algorithm always increases the value of the likelihood function, until a local
maximum is found. We illustrate the Bernoulli mixture model in Figure 9.10 by Section 9.4
using it to model handwritten digits. Here the digit images have been turned intobinary vectors by setting all elements whose values exceed 0.5to1and setting the
remaining elements to 0. We now ﬁt a data set of N= 600 such digits, comprising
the digits ‘2’, ‘3’, and ‘4’, with a mixture of K=3 Bernoulli distributions by

running 10iterations of the EM algorithm. The mixing coefﬁcients were initialized
toπ
k=1/K, and the parameters µkjwere set to random values chosen uniformly in
the range (0.25,0.75)and then normalized to satisfy the constraint that∑
jµkj=1.
We see that a mixture of 3Bernoulli distributions is able to ﬁnd the three clusters in
the data set corresponding to the different digits.
The conjugate prior for the parameters of a Bernoulli distribution is given by
the beta distribution, and we have seen that a beta prior is equivalent to introducing

448 9. MIXTURE MODELS AND EM
Figure 9.10 Illustration of the Bernoulli mixture model in which the top row shows examples from the digits data
set after converting the pixel values from grey scale to binary using a threshold of 0.5. On the bottom row the ﬁrst
three images show the parameters µkifor each of the three components in the mixture model. As a comparison,
we also ﬁt the same data set using a single multivariate Bernoulli distribution, again using maximum likelihood.
This amounts to simply averaging the counts in each pixel and is shown by the right-most image on the bottomrow.
additional effective observations of x. We can similarly introduce priors into the Section 2.1.1
Bernoulli mixture model, and use EM to maximize the posterior probability distri-butions. Exercise 9.18
It is straightforward to extend the analysis of Bernoulli mixtures to the case of
multinomial binary variables having M>2states by making use of the discrete dis- Exercise 9.19

tribution (2.26). Again, we can introduce Dirichlet priors over the model parameters
9.3.4 EM for Bayesian linear regression
As a third example of the application of EM, we return to the evidence ap-
proximation for Bayesian linear regression. In Section 3.5.2, we obtained the re-estimation equations for the hyperparameters αandβby evaluation of the evidence
and then setting the derivatives of the resulting expression to zero. We now turn to

an alternative approach for ﬁnding αandβbased on the EM algorithm. Recall that
our goal is to maximize the evidence function p(t|α,β)given by (3.77) with respect
toαandβ. Because the parameter vector wis marginalized out, we can regard it as
a latent variable, and hence we can optimize this marginal likelihood function usingEM. In the E step, we compute the posterior distribution of wgiven the current set-
ting of the parameters αandβand then use this to ﬁnd the expected complete-data

log likelihood. In the M step, we maximize this quantity with respect to αandβ.W e
have already derived the posterior distribution of wbecause this is given by (3.49).
The complete-data log likelihood function is then given by
lnp(t,w|α,β)=l n p(t|w,β)+l n p(w|α) (9.61)

9.3. An Alternative View of EM 449
where the likelihood p(t|w,β)and the prior p(w|α)are given by (3.10) and (3.52),
respectively, and y(x,w)is given by (3.3). Taking the expectation with respect to
the posterior distribution of wthen gives
E[lnp(t,w|α,β)] =M
2ln(α
2π)
−α
2E[
wTw]
+N
2ln(β
2π)
−β
2N∑
n=1E[
(tn−wTφn)2]
. (9.62)
Setting the derivatives with respect to αto zero, we obtain the M step re-estimation
equation Exercise 9.20
α=M
E[wTw]=M
mT
NmN+Tr(SN). (9.63)
An analogous result holds for β. Exercise 9.21
Note that this re-estimation equation takes a slightly different form from the
corresponding result (3.92) derived by direct evaluation of the evidence function.However, they each involve computation and inversion (or eigen decomposition) of
anM×Mmatrix and hence will have comparable computational cost per iteration.
These two approaches to determining αshould of course converge to the same
result (assuming they ﬁnd the same local maximum of the evidence function). This

can be veriﬁed by ﬁrst noting that the quantity γis deﬁned by
γ=M−αM∑
i=11
λi+α=M−αTr(SN). (9.64)
At a stationary point of the evidence function, the re-estimation equation (3.92) will
be self-consistently satisﬁed, and hence we can substitute for γto give
αmT
NmN=γ=M−αTr(SN) (9.65)
and solving for αwe obtain (9.63), which is precisely the EM re-estimation equation.
As a ﬁnal example, we consider a closely related model, namely the relevance
vector machine for regression discussed in Section 7.2.1. There we used direct max-

γ=M−αM∑
i=11
αmT
imization of the marginal likelihood to derive re-estimation equations for the hyper-parameters αandβ. Here we consider an alternative approach in which we view the
weight vector was a latent variable and apply the EM algorithm. The E step involves
ﬁnding the posterior distribution over the weights, and this is given by (7.81). In theM step we maximize the expected complete-data log likelihood, which is deﬁned by
E
w[lnp(t|X,w,β)p(w|α)] (9.66)
where the expectation is taken with respect to the posterior distribution computed

E
using the ‘old’ parameter values. To compute the new parameter values we maximize
with respect to αandβto give Exercise 9.22

450 9. MIXTURE MODELS AND EM
αnew
i=1
m2
i+Σii(9.67)
(βnew)−1=∥t−Φm N∥2+β−1∑
iγi
N(9.68)
These re-estimation equations are formally equivalent to those obtained by direct
maxmization. Exercise 9.23
9.4. The EM Algorithm in General
The expectation maximization algorithm, or EM algorithm, is a general technique for
ﬁnding maximum likelihood solutions for probabilistic models having latent vari-
ables (Dempster et al. , 1977; McLachlan and Krishnan, 1997). Here we give a very
general treatment of the EM algorithm and in the process provide a proof that the
EM algorithm derived heuristically in Sections 9.2 and 9.3 for Gaussian mixtures
does indeed maximize the likelihood function (Csisz `ar and Tusn `ady, 1984; Hath-
away, 1986; Neal and Hinton, 1999). Our discussion will also form the basis for the
derivation of the variational inference framework. Section 10.1
Consider a probabilistic model in which we collectively denote all of the ob-

served variables by Xand all of the hidden variables by Z. The joint distribution
p(X,Z|θ)is governed by a set of parameters denoted θ. Our goal is to maximize
the likelihood function that is given by
p(X|θ)=∑
Zp(X,Z|θ). (9.69)
Here we are assuming Zis discrete, although the discussion is identical if Zcom-
prises continuous variables or a combination of discrete and continuous variables,
with summation replaced by integration as appropriate.
We shall suppose that direct optimization of p(X|θ)is difﬁcult, but that opti-

p(X|θ)=∑
mization of the complete-data likelihood function p(X,Z|θ)is signiﬁcantly easier.
Next we introduce a distribution q(Z)deﬁned over the latent variables, and we ob-
serve that, for any choice of q(Z), the following decomposition holds
lnp(X|θ)=L(q,θ)+K L ( q∥p) (9.70)
L(q,θ)=∑
Zq(Z)l n{p(X,Z|θ)
q(Z)}
(9.71)
KL(q∥p)= −∑
Zq(Z)l n{p(Z|X,θ)
q(Z)}
. (9.72)
Note that L(q,θ)is a functional (see Appendix D for a discussion of functionals)
of the distribution q(Z), and a function of the parameters θ. It is worth studying

9.4. The EM Algorithm in General 451
Figure 9.11 Illustration of the decomposition given
by (9.70), which holds for any choice
of distribution q(Z). Because the
Kullback-Leibler divergence satisﬁes
KL(q∥p)⩾0, we see that the quan-
tityL(q,θ)is a lower bound on the log
likelihood function lnp(X|θ).
lnp(X|θ) L(q,θ)KL(q||p)
carefully the forms of the expressions (9.71) and (9.72), and in particular noting that
they differ in sign and also that L(q,θ)contains the joint distribution of XandZ
while KL(q∥p)contains the conditional distribution of ZgivenX. To verify the
decomposition (9.70), we ﬁrst make use of the product rule of probability to give Exercise 9.24
lnp(X,Z|θ)=l n p(Z|X,θ)+l n p(X|θ) (9.73)
which we then substitute into the expression for L(q,θ). This gives rise to two terms,
one of which cancels KL(q∥p)while the other gives the required log likelihood
lnp(X|θ)after noting that q(Z)is a normalized distribution that sums to 1.
From (9.72), we see that KL(q∥p)is the Kullback-Leibler divergence between

q(Z)and the posterior distribution p(Z|X,θ). Recall that the Kullback-Leibler di-
vergence satisﬁes KL(q∥p)⩾0, with equality if, and only if, q(Z)=p(Z|X,θ).I t Section 1.6.1
therefore follows from (9.70) that L(q,θ)⩽lnp(X|θ), in other words that L(q,θ)
is a lower bound on lnp(X|θ). The decomposition (9.70) is illustrated in Fig-
ure 9.11.
The EM algorithm is a two-stage iterative optimization technique for ﬁnding
maximum likelihood solutions. We can use the decomposition (9.70) to deﬁne the

ure 9.11.
EM algorithm and to demonstrate that it does indeed maximize the log likelihood.
Suppose that the current value of the parameter vector is θold. In the E step, the
lower bound L(q,θold)is maximized with respect to q(Z)while holding θoldﬁxed.
The solution to this maximization problem is easily seen by noting that the value
oflnp(X|θold)does not depend on q(Z)and so the largest value of L(q,θold)will
occur when the Kullback-Leibler divergence vanishes, in other words when q(Z)is

equal to the posterior distribution p(Z|X,θold). In this case, the lower bound will
equal the log likelihood, as illustrated in Figure 9.12.
In the subsequent M step, the distribution q(Z)is held ﬁxed and the lower bound
L(q,θ)is maximized with respect to θto give some new value θnew. This will
cause the lower bound Lto increase (unless it is already at a maximum), which will
necessarily cause the corresponding log likelihood function to increase. Because the

distribution qis determined using the old parameter values rather than the new values
and is held ﬁxed during the M step, it will not equal the new posterior distribution
p(Z|X,θnew), and hence there will be a nonzero KL divergence. The increase in the
log likelihood function is therefore greater than the increase in the lower bound, as

452 9. MIXTURE MODELS AND EM
Figure 9.12 Illustration of the E step of
the EM algorithm. The q
distribution is set equal to
the posterior distribution for
the current parameter val-
uesθold, causing the lower
bound to move up to the
same value as the log like-
lihood function, with the KL
divergence vanishing. lnp(X|θold) L(q,θold)KL(q||p)=0
shown in Figure 9.13. If we substitute q(Z)=p(Z|X,θold)into (9.71), we see that,
after the E step, the lower bound takes the form
L(q,θ)=∑
Zp(Z|X,θold)l np(X,Z|θ)−∑
Zp(Z|X,θold)l np(Z|X,θold)
=Q(θ,θold)+c o n s t (9.74)
where the constant is simply the negative entropy of the qdistribution and is there-
fore independent of θ. Thus in the M step, the quantity that is being maximized is the
expectation of the complete-data log likelihood, as we saw earlier in the case of mix-
tures of Gaussians. Note that the variable θover which we are optimizing appears
only inside the logarithm. If the joint distribution p(Z,X|θ)comprises a member of

the exponential family, or a product of such members, then we see that the logarithm
will cancel the exponential and lead to an M step that will be typically much simpler
than the maximization of the corresponding incomplete-data log likelihood function
p(X|θ).
The operation of the EM algorithm can also be viewed in the space of parame-
ters, as illustrated schematically in Figure 9.14. Here the red curve depicts the (in-
Figure 9.13 Illustration of the M step of the EM
algorithm. The distribution q(Z)

p(X|θ).
is held ﬁxed and the lower bound
L(q,θ)is maximized with respect
to the parameter vector θto give
a revised value θnew. Because the
KL divergence is nonnegative, this
causes the log likelihood lnp(X|θ)
to increase by at least as much as
the lower bound does.
lnp(X|θnew) L(q,θnew)KL(q||p)

9.4. The EM Algorithm in General 453
Figure 9.14 The EM algorithm involves alter-
nately computing a lower bound
on the log likelihood for the cur-
rent parameter values and then
maximizing this bound to obtain
the new parameter values. See
the text for a full discussion.
θoldθnewL(q,θ)lnp(X|θ)
complete data) log likelihood function whose value we wish to maximize. We start
with some initial parameter value θold, and in the ﬁrst E step we evaluate the poste-
rior distribution over latent variables, which gives rise to a lower bound L(θ,θ(old))
whose value equals the log likelihood at θ(old), as shown by the blue curve. Note that
the bound makes a tangential contact with the log likelihood at θ(old), so that both
curves have the same gradient. This bound is a convex function having a unique Exercise 9.25
maximum (for mixture components from the exponential family). In the M step, the
bound is maximized giving the value θ(new), which gives a larger value of log likeli-

hood than θ(old). The subsequent E step then constructs a bound that is tangential at
θ(new)as shown by the green curve.
For the particular case of an independent, identically distributed data set, X
will comprise Ndata points {xn}whileZwill comprise Ncorresponding latent
variables {zn}, where n=1,...,N . From the independence assumption, we have
p(X,Z)=∏
np(xn,zn)and, by marginalizing over the {zn}we have p(X)=∏
np(xn). Using the sum and product rules, we see that the posterior probability
that is evaluated in the E step takes the form
p(Z|X,θ)=p(X,Z|θ)∑
Zp(X,Z|θ)=N∏
n=1p(xn,zn|θ)
∑

p(X,Z)=∏
∑
ZN∏
n=1p(xn,zn|θ)=N∏
n=1p(zn|xn,θ)(9.75)
and so the posterior distribution also factorizes with respect to n. In the case of
the Gaussian mixture model this simply says that the responsibility that each of the
mixture components takes for a particular data point xndepends only on the value
ofxnand on the parameters θof the mixture components, not on the values of the
other data points.
We have seen that both the E and the M steps of the EM algorithm are increas-

∑
ZN∏
ing the value of a well-deﬁned bound on the log likelihood function and that the

454 9. MIXTURE MODELS AND EM
complete EM cycle will change the model parameters in such a way as to cause
the log likelihood to increase (unless it is already at a maximum, in which case theparameters remain unchanged).
We can also use the EM algorithm to maximize the posterior distribution p(θ|X)
for models in which we have introduced a prior p(θ)over the parameters. To see this,
we note that as a function of θ,w eh a v e p(θ|X)=p(θ,X)/p(X)and so
lnp(θ|X)=l n p(θ,X)−lnp(X). (9.76)
Making use of the decomposition (9.70), we have
lnp(θ|X)= L(q,θ)+K L ( q∥p)+l n p(θ)−lnp(X)
⩾L(q,θ)+l n p(θ)−lnp(X)
. (9.77)
where lnp(X)is a constant. We can again optimize the right-hand side alternately
with respect to qandθ. The optimization with respect to qgives rise to the same E-
step equations as for the standard EM algorithm, because qonly appears in L(q,θ).
The M-step equations are modiﬁed through the introduction of the prior term lnp(θ),
which typically requires only a small modiﬁcation to the standard maximum likeli-

. (9.77)
hood M-step equations.
The EM algorithm breaks down the potentially difﬁcult problem of maximizing
the likelihood function into two stages, the E step and the M step, each of which will
often prove simpler to implement. Nevertheless, for complex models it may be thecase that either the E step or the M step, or indeed both, remain intractable. This
leads to two possible extensions of the EM algorithm, as follows.
The generalized EM ,o rGEM , algorithm addresses the problem of an intractable

M step. Instead of aiming to maximize L(q,θ)with respect to θ, it seeks instead
to change the parameters in such a way as to increase its value. Again, because
L(q,θ)is a lower bound on the log likelihood function, each complete EM cycle of
the GEM algorithm is guaranteed to increase the value of the log likelihood (unless
the parameters already correspond to a local maximum). One way to exploit the

GEM approach would be to use one of the nonlinear optimization strategies, suchas the conjugate gradients algorithm, during the M step. Another form of GEM
algorithm, known as the expectation conditional maximization , or ECM, algorithm,
involves making several constrained optimizations within each M step (Meng and
Rubin, 1993). For instance, the parameters might be partitioned into groups, and the
M step is broken down into multiple steps each of which involves optimizing one ofthe subset with the remainder held ﬁxed.

We can similarly generalize the E step of the EM algorithm by performing a
partial, rather than complete, optimization of L(q,θ)with respect to q(Z)(Neal and
Hinton, 1999). As we have seen, for any given value of θthere is a unique maximum
ofL(q,θ)with respect to q(Z)that corresponds to the posterior distribution q
θ(Z)=
p(Z|X,θ)and that for this choice of q(Z)the bound L(q,θ)is equal to the log
likelihood function lnp(X|θ). It follows that any algorithm that converges to the
global maximum of L(q,θ)will ﬁnd a value of θthat is also a global maximum
of the log likelihood lnp(X|θ). Provided p(X,Z|θ)is a continuous function of θ

Exercises 455
then, by continuity, any local maximum of L(q,θ)will also be a local maximum of
lnp(X|θ).
Consider the case of Nindependent data points x1,...,xNwith corresponding
latent variables z1,...,zN. The joint distribution p(X,Z|θ)factorizes over the data
points, and this structure can be exploited in an incremental form of EM in whichat each EM cycle only one data point is processed at a time. In the E step, instead
of recomputing the responsibilities for all of the data points, we just re-evaluate the
responsibilities for one data point. It might appear that the subsequent M step wouldrequire computation involving the responsibilities for all of the data points. How-
ever, if the mixture components are members of the exponential family, then the
responsibilities enter only through simple sufﬁcient statistics, and these can be up-
dated efﬁciently. Consider, for instance, the case of a Gaussian mixture, and suppose
we perform an update for data point min which the corresponding old and new

values of the responsibilities are denoted γ
old(zmk)andγnew(zmk). In the M step,
the required sufﬁcient statistics can be updated incrementally. For instance, for the
means the sufﬁcient statistics are deﬁned by (9.17) and (9.18) from which we obtain Exercise 9.26
µnew
k=µold
k+(γnew(zmk)−γold(zmk)
Nnew
k)(
xm−µold
k)
(9.78)
Nnew
k=Nold
k+γnew(zmk)−γold(zmk). (9.79)
The corresponding results for the covariances and the mixing coefﬁcients are analo-
gous.

µnew
k=µold
Nnew
k)(
xm−µold
k)
(9.78)
Nnew
k=Nold
gous.
Thus both the E step and the M step take ﬁxed time that is independent of the
total number of data points. Because the parameters are revised after each data point,
rather than waiting until after the whole data set is processed, this incremental ver-sion can converge faster than the batch version. Each E or M step in this incremental
algorithm is increasing the value of L(q,θ)and, as we have shown above, if the
algorithm converges to a local (or global) maximum of L(q,θ), this will correspond

gous.
to a local (or global) maximum of the log likelihood function lnp(X|θ).
Exercises
9.1 (⋆)www Consider the K-means algorithm discussed in Section 9.1. Show that as
a consequence of there being a ﬁnite number of possible assignments for the set of
discrete indicator variables rnk, and that for each such assignment there is a unique
optimum for the {µk}, theK-means algorithm must converge after a ﬁnite number
of iterations.
9.2 (⋆)Apply the Robbins-Monro sequential estimation procedure described in Sec-

Exercises
tion 2.3.5 to the problem of ﬁnding the roots of the regression function given bythe derivatives of Jin (9.1) with respect to µ
k. Show that this leads to a stochastic
K-means algorithm in which, for each data point xn, the nearest prototype µkis
updated using (9.5).

456 9. MIXTURE MODELS AND EM
9.3 (⋆)www Consider a Gaussian mixture model in which the marginal distribution
p(z)for the latent variable is given by (9.10), and the conditional distribution p(x|z)
for the observed variable is given by (9.11). Show that the marginal distribution
p(x), obtained by summing p(z)p(x|z)over all possible values of z, is a Gaussian
mixture of the form (9.7).
9.4 (⋆)Suppose we wish to use the EM algorithm to maximize the posterior distri-
bution over parameters p(θ|X)for a model containing latent variables, where Xis
the observed data set. Show that the E step remains the same as in the maximum
likelihood case, whereas in the M step the quantity to be maximized is given byQ(θ,θ
old)+l n p(θ)where Q(θ,θold)is deﬁned by (9.30).
9.5 (⋆)Consider the directed graph for a Gaussian mixture model shown in Figure 9.6.
By making use of the d-separation criterion discussed in Section 8.2, show that the

posterior distribution of the latent variables factorizes with respect to the differentdata points so that
p(Z|X,µ,Σ,π)=N∏
n=1p(zn|xn,µ,Σ,π). (9.80)
9.6 (⋆⋆)Consider a special case of a Gaussian mixture model in which the covari-
ance matrices Σkof the components are all constrained to have a common value
Σ. Derive the EM equations for maximizing the likelihood function under such a
model.
9.7 (⋆)www Verify that maximization of the complete-data log likelihood (9.36) for

model.
a Gaussian mixture model leads to the result that the means and covariances of eachcomponent are ﬁtted independently to the corresponding group of data points, and
the mixing coefﬁcients are given by the fractions of points in each group.
9.8 (⋆)
www Show that if we maximize (9.40) with respect to µkwhile keeping the
responsibilities γ(znk)ﬁxed, we obtain the closed form solution given by (9.17).
9.9 (⋆)Show that if we maximize (9.40) with respect to Σkandπkwhile keeping the

9.8 (⋆)
responsibilities γ(znk)ﬁxed, we obtain the closed form solutions given by (9.19)
and (9.22).
9.10 (⋆⋆)Consider a density model given by a mixture distribution
p(x)=K∑
k=1πkp(x|k) (9.81)
and suppose that we partition the vector xinto two parts so that x=(xa,xb).
Show that the conditional density p(xb|xa)is itself a mixture distribution and ﬁnd
expressions for the mixing coefﬁcients and for the component densities.

Exercises 457
9.11 (⋆)In Section 9.3.2, we obtained a relationship between Kmeans and EM for
Gaussian mixtures by considering a mixture model in which all components havecovariance ϵI. Show that in the limit ϵ→0, maximizing the expected complete-
data log likelihood for this model, given by (9.40), is equivalent to minimizing the
distortion measure Jfor the K-means algorithm given by (9.1).
9.12 (⋆)
www Consider a mixture distribution of the form
p(x)=K∑
k=1πkp(x|k) (9.82)
where the elements of xcould be discrete or continuous or a combination of these.
Denote the mean and covariance of p(x|k)byµkandΣk, respectively. Show that
the mean and covariance of the mixture distribution are given by (9.49) and (9.50).
9.13 (⋆⋆)Using the re-estimation equations for the EM algorithm, show that a mix-
ture of Bernoulli distributions, with its parameters set to values corresponding to amaximum of the likelihood function, has the property that
E[x]=1
NN∑
n=1xn≡x. (9.83)

E[x]=1
NN∑
Hence show that if the parameters of this model are initialized such that all compo-
nents have the same mean µk=ˆµfork=1,...,K , then the EM algorithm will
converge after one iteration, for any choice of the initial mixing coefﬁcients, and that
this solution has the property µk=x. Note that this represents a degenerate case of
the mixture model in which all of the components are identical, and in practice wetry to avoid such solutions by using an appropriate initialization.

E[x]=1
NN∑
9.14 (⋆)Consider the joint distribution of latent and observed variables for the Bernoulli
distribution obtained by forming the product of p(x|z,µ)given by (9.52) and p(z|π)
given by (9.53). Show that if we marginalize this joint distribution with respect to z,
then we obtain (9.47).
9.15 (⋆)
www Show that if we maximize the expected complete-data log likelihood
function (9.55) for a mixture of Bernoulli distributions with respect to µk, we obtain
the M step equation (9.59).

9.15 (⋆)
9.16 (⋆)Show that if we maximize the expected complete-data log likelihood function
(9.55) for a mixture of Bernoulli distributions with respect to the mixing coefﬁcients
πk, using a Lagrange multiplier to enforce the summation constraint, we obtain the
M step equation (9.60).
9.17 (⋆)www Show that as a consequence of the constraint 0⩽p(xn|µk)⩽1for
the discrete variable xn, the incomplete-data log likelihood function for a mixture
of Bernoulli distributions is bounded above, and hence that there are no singularities

for which the likelihood goes to inﬁnity.

458 9. MIXTURE MODELS AND EM
9.18 (⋆⋆)Consider a Bernoulli mixture model as discussed in Section 9.3.3, together
with a prior distribution p(µk|ak,bk)over each of the parameter vectors µkgiven
by the beta distribution (2.13), and a Dirichlet prior p(π|α)given by (2.38). Derive
the EM algorithm for maximizing the posterior probability p(µ,π|X).
9.19 (⋆⋆)Consider a D-dimensional variable xeach of whose components iis itself a
multinomial variable of degree Mso that xis a binary vector with components xij
where i=1,...,D andj=1,...,M , subject to the constraint that∑
jxij=1for
alli. Suppose that the distribution of these variables is described by a mixture of the
discrete multinomial distributions considered in Section 2.2 so that
p(x)=K∑
k=1πkp(x|µk) (9.84)
where
i=1M∏
j=1µxij
kij. (9.85)
The parameters µkijrepresent the probabilities p(xij=1|µk)and must satisfy
0⩽µkij⩽1together with the constraint∑
jµkij=1 for all values of kandi.

jxij=1for
p(x)=K∑
where
i=1M∏
j=1µxij
Given an observed data set {xn}, where n=1,...,N , derive the E and M step
equations of the EM algorithm for optimizing the mixing coefﬁcients πkand the
component parameters µkijof this distribution by maximum likelihood.
9.20 (⋆)www Show that maximization of the expected complete-data log likelihood
function (9.62) for the Bayesian linear regression model leads to the M step re-
estimation result (9.63) for α.
9.21 (⋆⋆)Using the evidence framework of Section 3.5, derive the M-step re-estimation

equations for the parameter βin the Bayesian linear regression model, analogous to
the result (9.63) for α.
9.22 (⋆⋆)By maximization of the expected complete-data log likelihood deﬁned by
(9.66), derive the M step equations (9.67) and (9.68) for re-estimating the hyperpa-
rameters of the relevance vector machine for regression.
9.23 (⋆⋆)www In Section 7.2.1 we used direct maximization of the marginal like-
lihood to derive the re-estimation equations (7.87) and (7.88) for ﬁnding values of

the hyperparameters αandβfor the regression RVM. Similarly, in Section 9.3.4
we used the EM algorithm to maximize the same marginal likelihood, giving there-estimation equations (9.67) and (9.68). Show that these two sets of re-estimation
equations are formally equivalent.
9.24 (⋆)Verify the relation (9.70) in which L(q,θ)andKL(q∥p)are deﬁned by (9.71)
and (9.72), respectively.

Exercises 459
9.25 (⋆)www Show that the lower bound L(q,θ)given by (9.71), with q(Z)=
p(Z|X,θ(old)), has the same gradient with respect to θas the log likelihood function
lnp(X|θ)at the point θ=θ(old).
9.26 (⋆)www Consider the incremental form of the EM algorithm for a mixture of
Gaussians, in which the responsibilities are recomputed only for a speciﬁc data pointx
m. Starting from the M-step formulae (9.17) and (9.18), derive the results (9.78)
and (9.79) for updating the component means.
9.27 (⋆⋆)Derive M-step formulae for updating the covariance matrices and mixing
coefﬁcients in a Gaussian mixture model when the responsibilities are updated in-crementally, analogous to the result (9.78) for updating the means.

10
Approximate
Inference
A central task in the application of probabilistic models is the evaluation of the pos-
terior distribution p(Z|X)of the latent variables Zgiven the observed (visible) data
variables X, and the evaluation of expectations computed with respect to this dis-
tribution. The model might also contain some deterministic parameters, which we
will leave implicit for the moment, or it may be a fully Bayesian model in which anyunknown parameters are given prior distributions and are absorbed into the set of
latent variables denoted by the vector Z. For instance, in the EM algorithm we need
to evaluate the expectation of the complete-data log likelihood with respect to theposterior distribution of the latent variables. For many models of practical interest, it
will be infeasible to evaluate the posterior distribution or indeed to compute expec-

tations with respect to this distribution. This could be because the dimensionality ofthe latent space is too high to work with directly or because the posterior distribution
has a highly complex form for which expectations are not analytically tractable. In
the case of continuous variables, the required integrations may not have closed-form
461

462 10. APPROXIMATE INFERENCE
analytical solutions, while the dimensionality of the space and the complexity of the
integrand may prohibit numerical integration. For discrete variables, the marginal-izations involve summing over all possible conﬁgurations of the hidden variables,
and though this is always possible in principle, we often ﬁnd in practice that there
may be exponentially many hidden states so that exact calculation is prohibitivelyexpensive.
In such situations, we need to resort to approximation schemes, and these fall
broadly into two classes, according to whether they rely on stochastic or determin-istic approximations. Stochastic techniques such as Markov chain Monte Carlo, de-
scribed in Chapter 11, have enabled the widespread use of Bayesian methods across
many domains. They generally have the property that given inﬁnite computational
resource, they can generate exact results, and the approximation arises from the use

of a ﬁnite amount of processor time. In practice, sampling methods can be compu-tationally demanding, often limiting their use to small-scale problems. Also, it can
be difﬁcult to know whether a sampling scheme is generating independent samples
from the required distribution.
In this chapter, we introduce a range of deterministic approximation schemes,
some of which scale well to large applications. These are based on analytical ap-

proximations to the posterior distribution, for example by assuming that it factorizesin a particular way or that it has a speciﬁc parametric form such as a Gaussian. As
such, they can never generate exact results, and so their strengths and weaknesses
are complementary to those of sampling methods.
In Section 4.4, we discussed the Laplace approximation, which is based on a
local Gaussian approximation to a mode (i.e., a maximum) of the distribution. Here
we turn to a family of approximation techniques called variational inference orvari-

ational Bayes , which use more global criteria and which have been widely applied.
We conclude with a brief introduction to an alternative variational framework known
asexpectation propagation .
10.1. Variational Inference
Variational methods have their origins in the 18thcentury with the work of Euler,
Lagrange, and others on the calculus of variations . Standard calculus is concerned
with ﬁnding derivatives of functions. We can think of a function as a mapping that
takes the value of a variable as the input and returns the value of the function as theoutput. The derivative of the function then describes how the output value varies

as we make inﬁnitesimal changes to the input value. Similarly, we can deﬁne a
functional as a mapping that takes a function as the input and that returns the value
of the functional as the output. An example would be the entropy H[p], which takes
a probability distribution p(x)as the input and returns the quantity
H[p]=∫
p(x)l np(x)dx (10.1)

10.1. Variational Inference 463
as the output. We can the introduce the concept of a functional derivative , which ex-
presses how the value of the functional changes in response to inﬁnitesimal changesto the input function (Feynman et al. , 1964). The rules for the calculus of variations
mirror those of standard calculus and are discussed in Appendix D. Many problems
can be expressed in terms of an optimization problem in which the quantity beingoptimized is a functional. The solution is obtained by exploring all possible input
functions to ﬁnd the one that maximizes, or minimizes, the functional. Variational
methods have broad applicability and include such areas as ﬁnite element methods(Kapur, 1989) and maximum entropy (Schwarz, 1988).
Although there is nothing intrinsically approximate about variational methods,
they do naturally lend themselves to ﬁnding approximate solutions. This is done
by restricting the range of functions over which the optimization is performed, for

instance by considering only quadratic functions or by considering functions com-posed of a linear combination of ﬁxed basis functions in which only the coefﬁcients
of the linear combination can vary. In the case of applications to probabilistic in-
ference, the restriction may for example take the form of factorization assumptions(Jordan et al. , 1999; Jaakkola, 2001).
Now let us consider in more detail how the concept of variational optimization

can be applied to the inference problem. Suppose we have a fully Bayesian model inwhich all parameters are given prior distributions. The model may also have latent
variables as well as parameters, and we shall denote the set of all latent variables
and parameters by Z. Similarly, we denote the set of all observed variables by X.
For example, we might have a set of Nindependent, identically distributed data,
for which X={x
1,...,xN}andZ={z1,...,zN}. Our probabilistic model
speciﬁes the joint distribution p(X,Z), and our goal is to ﬁnd an approximation for

the posterior distribution p(Z|X)as well as for the model evidence p(X).A si no u r
discussion of EM, we can decompose the log marginal probability using
lnp(X)=L(q) + KL( q∥p) (10.2)
L(q)=∫
q(Z)l n{p(X,Z)
q(Z)}
dZ (10.3)
KL(q∥p)= −∫
q(Z)l n{p(Z|X)
q(Z)}
dZ. (10.4)
This differs from our discussion of EM only in that the parameter vector θno longer
appears, because the parameters are now stochastic variables and are absorbed into
Z. Since in this chapter we will mainly be interested in continuous variables we have
used integrations rather than summations in formulating this decomposition. How-

L(q)=∫
q(Z)}
dZ (10.3)
q(Z)}
dZ. (10.4)
ever, the analysis goes through unchanged if some or all of the variables are discretesimply by replacing the integrations with summations as required. As before, we
can maximize the lower bound L(q)by optimization with respect to the distribution
q(Z), which is equivalent to minimizing the KL divergence. If we allow any possible
choice for q(Z), then the maximum of the lower bound occurs when the KL diver-
gence vanishes, which occurs when q(Z)equals the posterior distribution p(Z|X).

464 10. APPROXIMATE INFERENCE
−2 −1 0 1 2 3 400.20.40.60.81
Figure 10.1 Illustration of the variational approximation for the example considered earlier in Figure 4.14. The
left-hand plot shows the original distribution (yellow) along with the Laplace (red) and variational (green) approx-
imations, and the right-hand plot shows the negative logarithms of the corresponding curves.
However, we shall suppose the model is such that working with the true posterior
distribution is intractable.
We therefore consider instead a restricted family of distributions q(Z)and then
seek the member of this family for which the KL divergence is minimized. Our goal
is to restrict the family sufﬁciently that they comprise only tractable distributions,
while at the same time allowing the family to be sufﬁciently rich and ﬂexible that it
can provide a good approximation to the true posterior distribution. It is important to

emphasize that the restriction is imposed purely to achieve tractability, and that sub-
ject to this requirement we should use as rich a family of approximating distributions
as possible. In particular, there is no ‘over-ﬁtting’ associated with highly ﬂexible dis-
tributions. Using more ﬂexible approximations simply allows us to approach the true
posterior distribution more closely.
One way to restrict the family of approximating distributions is to use a paramet-
ric distribution q(Z|ω)governed by a set of parameters ω. The lower bound L(q)

then becomes a function of ω, and we can exploit standard nonlinear optimization
techniques to determine the optimal values for the parameters. An example of this
approach, in which the variational distribution is a Gaussian and we have optimized
with respect to its mean and variance, is shown in Figure 10.1.
10.1.1 Factorized distributions
Here we consider an alternative way in which to restrict the family of distri-
butions q(Z). Suppose we partition the elements of Zinto disjoint groups that we

denote by Ziwhere i=1,...,M . We then assume that the qdistribution factorizes
with respect to these groups, so that
q(Z)=M∏
i=1qi(Zi). (10.5)

10.1. Variational Inference 465
It should be emphasized that we are making no further assumptions about the distri-
bution. In particular, we place no restriction on the functional forms of the individualfactors q
i(Zi). This factorized form of variational inference corresponds to an ap-
proximation framework developed in physics called mean ﬁeld theory (Parisi, 1988).
Amongst all distributions q(Z)having the form (10.5), we now seek that distri-
bution for which the lower bound L(q)is largest. We therefore wish to make a free
form (variational) optimization of L(q)with respect to all of the distributions qi(Zi),
which we do by optimizing with respect to each of the factors in turn. To achievethis, we ﬁrst substitute (10.5) into (10.3) and then dissect out the dependence on one
of the factors q
j(Zj). Denoting qj(Zj)by simply qjto keep the notation uncluttered,
L(q)=∫∏
iqi{
lnp(X,Z)−∑
ilnqi}
dZ
=∫
qj{∫
lnp(X,Z)∏
i̸=jqidZi}
dZj−∫
qjlnqjdZj+c o n s t
=∫
qjln˜p(X,Zj)dZj−∫

L(q)=∫∏
iqi{
ilnqi}
dZ
=∫
qj{∫
lnp(X,Z)∏
dZj−∫
=∫
qjlnqjdZj+c o n s t (10.6)
where we have deﬁned a new distribution ˜p(X,Zj)by the relation
ln˜p(X,Zj)= Ei̸=j[lnp(X,Z)] + const . (10.7)
Here the notation Ei̸=j[···]denotes an expectation with respect to the qdistributions
over all variables zifori̸=j, so that
Ei̸=j[lnp(X,Z)] =∫
lnp(X,Z)∏
i̸=jqidZi. (10.8)
Now suppose we keep the {qi̸=j}ﬁxed and maximize L(q)in (10.6) with re-
spect to all possible forms for the distribution qj(Zj). This is easily done by rec-
ognizing that (10.6) is a negative Kullback-Leibler divergence between qj(Zj)and

lnp(X,Z)∏
˜p(X,Zj). Thus maximizing (10.6) is equivalent to minimizing the Kullback-Leibler
Leonhard Euler
1707–1783
Euler was a Swiss mathematician
and physicist who worked in St.Petersburg and Berlin and who iswidely considered to be one of thegreatest mathematicians of all time.He is certainly the most proliﬁc, and
his collected works ﬁll 75 volumes. Amongst his manycontributions, he formulated the modern theory of the
function, he developed (together with Lagrange) thecalculus of variations, and he discovered the formulae
iπ=−1, which relates four of the most important

1707–1783
numbers in mathematics. During the last 17 years ofhis life, he was almost totally blind, and yet he pro-duced nearly half of his results during this period.

466 10. APPROXIMATE INFERENCE
divergence, and the minimum occurs when qj(Zj)=˜p(X,Zj). Thus we obtain a
general expression for the optimal solution q⋆
j(Zj)given by
lnq⋆
j(Zj)= Ei̸=j[lnp(X,Z)] + const . (10.9)
It is worth taking a few moments to study the form of this solution as it provides the
basis for applications of variational methods. It says that the log of the optimal so-
lution for factor qjis obtained simply by considering the log of the joint distribution
over all hidden and visible variables and then taking the expectation with respect to
all of the other factors {qi}fori̸=j.
The additive constant in (10.9) is set by normalizing the distribution q⋆
j(Zj).
Thus if we take the exponential of both sides and normalize, we have
q⋆
j(Zj)=exp ( Ei̸=j[lnp(X,Z)])∫
exp ( Ei̸=j[lnp(X,Z)]) dZj.
In practice, we shall ﬁnd it more convenient to work with the form (10.9) and then re-
instate the normalization constant (where required) by inspection. This will become
clear from subsequent examples.

j(Zj).
q⋆
The set of equations given by (10.9) for j=1,...,M represent a set of con-
sistency conditions for the maximum of the lower bound subject to the factorizationconstraint. However, they do not represent an explicit solution because the expres-
sion on the right-hand side of (10.9) for the optimum q
⋆
j(Zj)depends on expectations
computed with respect to the other factors qi(Zi)fori̸=j. We will therefore seek
a consistent solution by ﬁrst initializing all of the factors qi(Zi)appropriately and

⋆
then cycling through the factors and replacing each in turn with a revised estimate
given by the right-hand side of (10.9) evaluated using the current estimates for all of
the other factors. Convergence is guaranteed because bound is convex with respectto each of the factors q
i(Zi)(Boyd and Vandenberghe, 2004).
10.1.2 Properties of factorized approximations
Our approach to variational inference is based on a factorized approximation to
the true posterior distribution. Let us consider for a moment the problem of approx-

imating a general distribution by a factorized distribution. To begin with, we discussthe problem of approximating a Gaussian distribution using a factorized Gaussian,
which will provide useful insight into the types of inaccuracy introduced in using
factorized approximations. Consider a Gaussian distribution p(z)=N(z|µ,Λ
−1)
over two correlated variables z=(z1,z2)in which the mean and precision have
elements
µ=(
µ1
µ2)
, Λ=(
Λ11Λ12
Λ21Λ22)
(10.10)
andΛ21=Λ 12due to the symmetry of the precision matrix. Now suppose we

−1)
elements
µ=(
µ1
µ2)
, Λ=(
Λ11Λ12
Λ21Λ22)
(10.10)
wish to approximate this distribution using a factorized Gaussian of the form q(z)=
q1(z1)q2(z2). We ﬁrst apply the general result (10.9) to ﬁnd an expression for the

10.1. Variational Inference 467
optimal factor q⋆
1(z1). In doing so it is useful to note that on the right-hand side we
only need to retain those terms that have some functional dependence on z1because
all other terms can be absorbed into the normalization constant. Thus we have
lnq⋆
1(z1)= Ez2[lnp(z) ]+c o n s t
= Ez2[
−1
2(z1−µ1)2Λ11−(z1−µ1)Λ12(z2−µ2)]
+c o n s t
=−1
2z2
1Λ11+z1µ1Λ11−z1Λ12(E[z2]−µ2) + const .(10.11)
Next we observe that the right-hand side of this expression is a quadratic function of
z1, and so we can identify q⋆(z1)as a Gaussian distribution. It is worth emphasizing
that we did not assume that q(zi)is Gaussian, but rather we derived this result by
variational optimization of the KL divergence over all possible distributions q(zi).
Note also that we do not need to consider the additive constant in (10.9) explicitly
because it represents the normalization constant that can be found at the end by

inspection if required. Using the technique of completing the square, we can identify Section 2.3.1
the mean and precision of this Gaussian, giving
q⋆(z1)=N(z1|m1,Λ−1
11) (10.12)
where
m1=µ1−Λ−1
11Λ12(E[z2]−µ2). (10.13)
By symmetry, q⋆
2(z2)is also Gaussian and can be written as
q⋆
2(z2)=N(z2|m2,Λ−1
22) (10.14)
in which
m2=µ2−Λ−1
22Λ21(E[z1]−µ1). (10.15)
Note that these solutions are coupled, so that q⋆(z1)depends on expectations com-

where
m1=µ1−Λ−1
q⋆
in which
m2=µ2−Λ−1
puted with respect to q⋆(z2)and vice versa. In general, we address this by treating
the variational solutions as re-estimation equations and cycling through the variablesin turn updating them until some convergence criterion is satisﬁed. We shall see
an example of this shortly. Here, however, we note that the problem is sufﬁciently
simple that a closed form solution can be found. In particular, because E[z
1]=m1
and E[z2]=m2, we see that the two equations are satisﬁed if we take E[z1]=µ1
and E[z2]=µ2, and it is easily shown that this is the only solution provided the dis-

1]=m1
tribution is nonsingular. This result is illustrated in Figure 10.2(a). We see that the Exercise 10.2
mean is correctly captured but that the variance of q(z)is controlled by the direction
of smallest variance of p(z), and that the variance along the orthogonal direction is
signiﬁcantly under-estimated. It is a general result that a factorized variational ap-
proximation tends to give approximations to the posterior distribution that are too
compact.

compact.
By way of comparison, suppose instead that we had been minimizing the reverse
Kullback-Leibler divergence KL(p∥q). As we shall see, this form of KL divergence

468 10. APPROXIMATE INFERENCE
Figure 10.2 Comparison of
the two alternative forms for the
Kullback-Leibler divergence. The
green contours corresponding to
1, 2, and 3 standard deviations for
a correlated Gaussian distribution
p(z)over two variables z1andz2,
and the red contours represent
the corresponding levels for an
approximating distribution q(z)
over the same variables given by
the product of two independent
univariate Gaussian distributions
whose parameters are obtained by
minimization of (a) the Kullback-
Leibler divergence KL(q∥p), and
(b) the reverse Kullback-Leibler
divergence KL(p∥q).z1z2
(a)0 0.5 100.51
z1z2
(b)0 0.5 100.51
is used in an alternative approximate inference framework called expectation prop-
agation . We therefore consider the general problem of minimizing KL(p∥q)when Section 10.7
q(Z)is a factorized approximation of the form (10.5). The KL divergence can then
p(Z)[M∑
i=1lnqi(Zi)]
dZ+c o n s t (10.16)

z1z2
p(Z)[M∑
where the constant term is simply the entropy of p(Z)and so does not depend on
q(Z). We can now optimize with respect to each of the factors qj(Zj), which is
easily done using a Lagrange multiplier to give Exercise 10.3
q⋆
j(Zj)=∫
p(Z)∏
i̸=jdZi=p(Zj). (10.17)
In this case, we ﬁnd that the optimal solution for qj(Zj)is just given by the corre-
sponding marginal distribution of p(Z). Note that this is a closed-form solution and
so does not require iteration.

p(Z)[M∑
q⋆
j(Zj)=∫
p(Z)∏
To apply this result to the illustrative example of a Gaussian distribution p(z)
over a vector zwe can use (2.98), which gives the result shown in Figure 10.2(b).
We see that once again the mean of the approximation is correct, but that it places
signiﬁcant probability mass in regions of variable space that have very low probabil-
ity.
The difference between these two results can be understood by noting that there
is a large positive contribution to the Kullback-Leibler divergence
KL(q∥p)=−∫
q(Z)l n{p(Z)
q(Z)}

ity.
q(Z)}
dZ (10.18)

10.1. Variational Inference 469
(a) (b) (c)
Figure 10.3 Another comparison of the two alternative forms for the Kullback-Leibler divergence. (a) The blue
contours show a bimodal distribution p(Z)given by a mixture of two Gaussians, and the red contours correspond
to the single Gaussian distribution q(Z)that best approximates p(Z)in the sense of minimizing the Kullback-
Leibler divergence KL(p∥q). (b) As in (a) but now the red contours correspond to a Gaussian distribution q(Z)
found by numerical minimization of the Kullback-Leibler divergence KL(q∥p). (c) As in (b) but showing a different
local minimum of the Kullback-Leibler divergence.
from regions of Zspace in which p(Z)is near zero unless q(Z)is also close to
zero. Thus minimizing this form of KL divergence leads to distributions q(Z)that
avoid regions in which p(Z)is small. Conversely, the Kullback-Leibler divergence
KL(p∥q)is minimized by distributions q(Z)that are nonzero in regions where p(Z)
is nonzero.

We can gain further insight into the different behaviour of the two KL diver-
gences if we consider approximating a multimodal distribution by a unimodal one,
as illustrated in Figure 10.3. In practical applications, the true posterior distri-
bution will often be multimodal, with most of the posterior mass concentrated insome number of relatively small regions of parameter space. These multiple modes
may arise through nonidentiﬁability in the latent space or through complex nonlin-

ear dependence on the parameters. Both types of multimodality were encountered in
Chapter 9 in the context of Gaussian mixtures, where they manifested themselves as
multiple maxima in the likelihood function, and a variational treatment based on theminimization of KL(q∥p)will tend to ﬁnd one of these modes. By contrast, if we
were to minimize KL(p∥q), the resulting approximations would average across all

of the modes and, in the context of the mixture model, would lead to poor predictivedistributions (because the average of two good parameter values is typically itself
not a good parameter value). It is possible to make use of KL(p∥q)to deﬁne a useful
inference procedure, but this requires a rather different approach to the one discussedhere, and will be considered in detail when we discuss expectation propagation. Section 10.7
The two forms of Kullback-Leibler divergence are members of the alpha family

470 10. APPROXIMATE INFERENCE
of divergences (Ali and Silvey, 1966; Amari, 1985; Minka, 2005) deﬁned by
Dα(p∥q)=4
1−α2(
1−∫
(10.19)
where −∞<α< ∞is a continuous parameter. The Kullback-Leibler divergence
KL(p∥q)corresponds to the limit α→1, whereas KL(q∥p)corresponds to the limit
α→−1. For all values of αwe have Dα(p∥q)⩾0, with equality if, and only if, Exercise 10.6
p(x)=q(x). Suppose p(x)is a ﬁxed distribution, and we minimize Dα(p∥q)with
respect to some set of distributions q(x). Then for α⩽−1the divergence is zero
forcing , so that any values of xfor which p(x)=0 will have q(x)=0 , and typically
q(x)will under-estimate the support of p(x)and will tend to seek the mode with the
largest mass. Conversely for α⩾1the divergence is zero-avoiding , so that values
ofxfor which p(x)>0will have q(x)>0, and typically q(x)will stretch to cover
all ofp(x), and will over-estimate the support of p(x). When α=0 we obtain a

symmetric divergence that is linearly related to the Hellinger distance given by
DH(p∥q)=∫(
p(x)1/2−q(x)1/2)
dx. (10.20)
The square root of the Hellinger distance is a valid distance metric.
10.1.3 Example: The univariate Gaussian
We now illustrate the factorized variational approximation using a Gaussian dis-
tribution over a single variable x(MacKay, 2003). Our goal is to infer the posterior
distribution for the mean µand precision τ, given a data set D={x1,...,x N}of

observed values of xwhich are assumed to be drawn independently from the Gaus-
sian. The likelihood function is given by
p(D|µ, τ)=(τ
2π)N/2
exp{
−τ
2N∑
. (10.21)
We now introduce conjugate prior distributions for µandτgiven by
p(µ|τ)= N(
µ|µ0,(λ0τ)−1)
(10.22)
p(τ)=G a m ( τ|a0,b0) (10.23)
where Gam( τ|a0,b0)is the gamma distribution deﬁned by (2.146). Together these
distributions constitute a Gaussian-Gamma conjugate prior distribution. Section 2.3.6

2π)N/2
exp{
−τ
2N∑
. (10.21)
p(µ|τ)= N(
(10.22)
For this simple problem the posterior distribution can be found exactly, and again
takes the form of a Gaussian-gamma distribution. However, for tutorial purposes Exercise 2.44
we will consider a factorized variational approximation to the posterior distributiongiven by
q(µ, τ)=q
µ(µ)qτ(τ). (10.24)

10.1. Variational Inference 471
Note that the true posterior distribution does not factorize in this way. The optimum
factors qµ(µ)andqτ(τ)can be obtained from the general result (10.9) as follows.
Forqµ(µ)we have
lnq⋆
µ(µ)= Eτ[lnp(D|µ, τ)+l n p(µ|τ) ]+c o n s t
=−E[τ]
2{
λ0(µ−µ0)2+N∑
+c o n s t .(10.25)
Completing the square over µwe see that qµ(µ)is a Gaussian N(
µ|µN,λ−1
N)
with
mean and precision given by Exercise 10.7
µN=λ0µ0+Nx
λ0+N(10.26)
λN=(λ0+N)E[τ]. (10.27)
Note that for N→∞ this gives the maximum likelihood result in which µN=x
and the precision is inﬁnite.
Similarly, the optimal solution for the factor qτ(τ)is given by
lnq⋆
τ(τ)= Eµ[lnp(D|µ, τ)+l n p(µ|τ) ]+l n p(τ)+c o n s t
=(a0−1) lnτ−b0τ+N
2lnτ
−τ
2Eµ[N∑
n=1(xn−µ)2+λ0(µ−µ0)2]
+c o n s t (10.28)
and hence qτ(τ)is a gamma distribution Gam( τ|aN,bN)with parameters
aN=a0+N
2(10.29)
bN=b0+1
2Eµ[N∑
. (10.30)
Again this exhibits the expected behaviour when N→∞ . Exercise 10.8

lnq⋆
2lnτ
−τ
2Eµ[N∑
aN=a0+N
2(10.29)
bN=b0+1
2Eµ[N∑
. (10.30)
It should be emphasized that we did not assume these speciﬁc functional forms
for the optimal distributions qµ(µ)andqτ(τ). They arose naturally from the structure
of the likelihood function and the corresponding conjugate priors. Section 10.4.1
Thus we have expressions for the optimal distributions qµ(µ)andqτ(τ)each of
which depends on moments evaluated with respect to the other distribution. One ap-
proach to ﬁnding a solution is therefore to make an initial guess for, say, the moment

E[τ]and use this to re-compute the distribution qµ(µ). Given this revised distri-
bution we can then extract the required moments E[µ]and E[µ2], and use these to
recompute the distribution qτ(τ), and so on. Since the space of hidden variables for
this example is only two dimensional, we can illustrate the variational approxima-tion to the posterior distribution by plotting contours of both the true posterior and
the factorized approximation, as illustrated in Figure 10.4.

472 10. APPROXIMATE INFERENCE
µτ(a)
−1 0 1012
µτ(b)
−1 0 1012
µτ(c)
−1 0 1012
µτ(d)
−1 0 1012
Figure 10.4 Illustration of variational inference for the mean µand precision τof a univariate Gaussian distribu-
tion. Contours of the true posterior distribution p(µ, τ|D)are shown in green. (a) Contours of the initial factorized
approximation qµ(µ)qτ(τ)are shown in blue. (b) After re-estimating the factor qµ(µ). (c) After re-estimating the
factor qτ(τ). (d) Contours of the optimal factorized approximation, to which the iterative scheme converges, are
shown in red.
In general, we will need to use an iterative approach such as this in order to
solve for the optimal factorized posterior distribution. For the very simple example
we are considering here, however, we can ﬁnd an explicit solution by solving the
simultaneous equations for the optimal factors qµ(µ)andqτ(τ). Before doing this,
we can simplify these expressions by considering broad, noninformative priors in

which µ0=a0=b0=λ0=0. Although these parameter settings correspond to
improper priors, we see that the posterior distribution is still well deﬁned. Using the
standard result E[τ]=aN/bNfor the mean of a gamma distribution, together with Appendix B
(10.29) and (10.30), we have
1
E[τ]=E[
1
NN∑
n=1(xn−µ)2]
=x2−2xE[µ]+ E[µ2]. (10.31)
Then, using (10.26) and (10.27), we obtain the ﬁrst and second order moments of

10.1. Variational Inference 473
qµ(µ)in the form
E[µ]=x, E[µ2]=x2+1
NE[τ]. (10.32)
We can now substitute these moments into (10.31) and then solve for E[τ]to give Exercise 10.9
1
E[τ]=1
N−1(x2−x2)
=1
N−1N∑
n=1(xn−x)2. (10.33)
We recognize the right-hand side as the familiar unbiased estimator for the variance
of a univariate Gaussian distribution, and so we see that the use of a Bayesian ap-proach has avoided the bias of the maximum likelihood solution. Section 1.2.4
10.1.4 Model comparison
As well as performing inference over the hidden variables Z, we may also
wish to compare a set of candidate models, labelled by the index m, and having
prior probabilities p(m). Our goal is then to approximate the posterior probabilities
p(m|X), where Xis the observed data. This is a slightly more complex situation
than that considered so far because different models may have different structure
and indeed different dimensionality for the hidden variables Z. We cannot there-

fore simply consider a factorized approximation q(Z)q(m), but must instead recog-
nize that the posterior over Zmust be conditioned on m, and so we must consider
q(Z,m)=q(Z|m)q(m). We can readily verify the following decomposition based
on this variational distribution Exercise 10.10
lnp(X)=Lm−∑
m∑
Zq(Z|m)q(m)l n{p(Z,m|X)
q(Z|m)q(m)}
(10.34)
where the Lmis a lower bound on lnp(X)and is given by
Lm=∑
m∑
Zq(Z|m)q(m)l n{p(Z,X,m)
. (10.35)

m∑
(10.34)
Lm=∑
m∑
. (10.35)
Here we are assuming discrete Z, but the same analysis applies to continuous latent
variables provided the summations are replaced with integrations. We can maximize
Lmwith respect to the distribution q(m)using a Lagrange multiplier, with the result Exercise 10.11
q(m)∝p(m)e x p{Lm}. (10.36)
However, if we maximize Lmwith respect to the q(Z|m), we ﬁnd that the solutions
for different mare coupled, as we expect because they are conditioned on m.W e
proceed instead by ﬁrst optimizing each of the q(Z|m)individually by optimization

474 10. APPROXIMATE INFERENCE
of (10.35), and then subsequently determining the q(m)using (10.36). After nor-
malization the resulting values for q(m)can be used for model selection or model
averaging in the usual way.
10.2. Illustration: Variational Mixture of Gaussians
We now return to our discussion of the Gaussian mixture model and apply the vari-
ational inference machinery developed in the previous section. This will provide a
good illustration of the application of variational methods and will also demonstrate
how a Bayesian treatment elegantly resolves many of the difﬁculties associated with
the maximum likelihood approach (Attias, 1999b). The reader is encouraged to workthrough this example in detail as it provides many insights into the practical appli-
cation of variational methods. Many Bayesian models, corresponding to much more
sophisticated distributions, can be solved by straightforward extensions and general-izations of this analysis.

Our starting point is the likelihood function for the Gaussian mixture model, il-
lustrated by the graphical model in Figure 9.6. For each observation x
nwe have
a corresponding latent variable zncomprising a 1-of- Kbinary vector with ele-
ments znkfork=1,...,K . As before we denote the observed data set by X=
{x1,...,xN}, and similarly we denote the latent variables by Z={z1,...,zN}.
From (9.10) we can write down the conditional distribution of Z, given the mixing
coefﬁcients π, in the form
p(Z|π)=N∏
n=1K∏
k=1πznk
k. (10.37)

nwe have
p(Z|π)=N∏
n=1K∏
k=1πznk
k. (10.37)
Similarly, from (9.11), we can write down the conditional distribution of the ob-
served data vectors, given the latent variables and the component parameters
p(X|Z,µ,Λ)=N∏
n=1K∏
k=1N(
xn|µk,Λ−1
k)znk(10.38)
where µ={µk}andΛ={Λk}. Note that we are working in terms of precision
matrices rather than covariance matrices as this somewhat simpliﬁes the mathemat-
ics.
Next we introduce priors over the parameters µ,Λandπ. The analysis is con-
siderably simpliﬁed if we use conjugate prior distributions. We therefore choose a Section 10.4.1

n=1K∏
k=1N(
xn|µk,Λ−1
ics.
Dirichlet distribution over the mixing coefﬁcients π
p(π)=D i r ( π|α0)=C(α0)K∏
k=1πα0−1
k(10.39)
where by symmetry we have chosen the same parameter α0for each of the compo-
nents, and C(α0)is the normalization constant for the Dirichlet distribution deﬁned

10.2. Illustration: Variational Mixture of Gaussians 475
Figure 10.5 Directed acyclic graph representing the Bayesian mix-
ture of Gaussians model, in which the box (plate) de-
notes a set of Ni.i.d. observations. Here µdenotes
{µk}andΛdenotes {Λk}.
xnzn
Nπ
µΛ
by (B.23). As we have seen, the parameter α0can be interpreted as the effective Section 2.2.1
prior number of observations associated with each component of the mixture. If the
value of α0is small, then the posterior distribution will be inﬂuenced primarily by
the data rather than by the prior.
Similarly, we introduce an independent Gaussian-Wishart prior governing the
mean and precision of each Gaussian component, given by
p(µ,Λ)= p(µ|Λ)p(Λ)
=K∏
k=1N(
µk|m0,(β0Λk)−1)
W(Λk|W0,ν0) (10.40)
because this represents the conjugate prior distribution when both the mean and pre-
cision are unknown. Typically we would choose m0=0by symmetry. Section 2.3.6
The resulting model can be represented as a directed graph as shown in Fig-

=K∏
k=1N(
ure 10.5. Note that there is a link from Λtoµsince the variance of the distribution
overµin (10.40) is a function of Λ.
This example provides a nice illustration of the distinction between latent vari-
ables and parameters. Variables such as znthat appear inside the plate are regarded
as latent variables because the number of such variables grows with the size of the
data set. By contrast, variables such as µthat are outside the plate are ﬁxed in
number independently of the size of the data set, and so are regarded as parameters.

From the perspective of graphical models, however, there is really no fundamental
difference between them.
10.2.1 Variational distribution
In order to formulate a variational treatment of this model, we next write down
the joint distribution of all of the random variables, which is given by
p(X,Z,π,µ,Λ)=p(X|Z,µ,Λ)p(Z|π)p(π)p(µ|Λ)p(Λ) (10.41)
in which the various factors are deﬁned above. The reader should take a moment to
verify that this decomposition does indeed correspond to the probabilistic graphical

model shown in Figure 10.5. Note that only the variables X={x1,...,xN}are
observed.

476 10. APPROXIMATE INFERENCE
We now consider a variational distribution which factorizes between the latent
variables and the parameters so that
q(Z,π,µ,Λ)=q(Z)q(π,µ,Λ). (10.42)
It is remarkable that this is the only assumption that we need to make in order to
obtain a tractable practical solution to our Bayesian mixture model. In particular, thefunctional form of the factors q(Z)andq(π,µ,Λ)will be determined automatically
by optimization of the variational distribution. Note that we are omitting the sub-
scripts on the qdistributions, much as we do with the pdistributions in (10.41), and
are relying on the arguments to distinguish the different distributions.
The corresponding sequential update equations for these factors can be easily
derived by making use of the general result (10.9). Let us consider the derivation of
the update equation for the factor q(Z). The log of the optimized factor is given by
lnq
⋆(Z)= Eπ,µ,Λ[lnp(X,Z,π,µ,Λ) ]+c o n s t . (10.43)

lnq
We now make use of the decomposition (10.41). Note that we are only interested in
the functional dependence of the right-hand side on the variable Z. Thus any terms
that do not depend on Zcan be absorbed into the additive normalization constant,
giving
lnq⋆(Z)= Eπ[lnp(Z|π)] + Eµ,Λ[lnp(X|Z,µ,Λ) ]+c o n s t . (10.44)
Substituting for the two conditional distributions on the right-hand side, and again
absorbing any terms that are independent of Zinto the additive constant, we have
lnq⋆(Z)=N∑
n=1K∑
k=1znklnρnk+c o n s t (10.45)

giving
n=1K∑
lnρnk= E[lnπk]+1
2E[ln|Λk|]−D
2ln(2π)
−1
2Eµk,Λk[
(xn−µk)TΛk(xn−µk)]
(10.46)
where Dis the dimensionality of the data variable x. Taking the exponential of both
sides of (10.45) we obtain
q⋆(Z)∝N∏
n=1K∏
k=1ρznk
nk. (10.47)
Requiring that this distribution be normalized, and noting that for each value of n
the quantities znkare binary and sum to 1 over all values of k, we obtain Exercise 10.12
q⋆(Z)=N∏
n=1K∏
k=1rznk
nk(10.48)

10.2. Illustration: Variational Mixture of Gaussians 477
where
rnk=ρnk
K∑
j=1ρnj. (10.49)
We see that the optimal solution for the factor q(Z)takes the same functional form
as the prior p(Z|π). Note that because ρnkis given by the exponential of a real
quantity, the quantities rnkwill be nonnegative and will sum to one, as required.
For the discrete distribution q⋆(Z)we have the standard result
E[znk]=rnk (10.50)
from which we see that the quantities rnkare playing the role of responsibilities.
Note that the optimal solution for q⋆(Z)depends on moments evaluated with respect
to the distributions of other variables, and so again the variational update equations
are coupled and must be solved iteratively.
At this point, we shall ﬁnd it convenient to deﬁne three statistics of the observed
data set evaluated with respect to the responsibilities, given by
Nk=N∑
n=1rnk (10.51)
xk=1
NkN∑
n=1rnkxn (10.52)
Sk=1
NkN∑
n=1rnk(xn−xk)(xn−xk)T. (10.53)

Nk=N∑
xk=1
NkN∑
Sk=1
NkN∑
Note that these are analogous to quantities evaluated in the maximum likelihood EM
algorithm for the Gaussian mixture model.
Now let us consider the factor q(π,µ,Λ)in the variational posterior distribu-
tion. Again using the general result (10.9) we have
lnq⋆(π,µ,Λ)=l n p(π)+K∑
k=1lnp(µk,Λk)+ EZ[lnp(Z|π)]
+K∑
k=1N∑
n=1E[znk]l nN(
xn|µk,Λ−1
k)
+c o n s t . (10.54)
We observe that the right-hand side of this expression decomposes into a sum of
terms involving only πtogether with terms only involving µandΛ, which implies

+K∑
k=1N∑
xn|µk,Λ−1
k)
that the variational posterior q(π,µ,Λ)factorizes to give q(π)q(µ,Λ). Further-
more, the terms involving µandΛthemselves comprise a sum over kof terms
involving µkandΛkleading to the further factorization
q(π,µ,Λ)=q(π)K∏
k=1q(µk,Λk). (10.55)

478 10. APPROXIMATE INFERENCE
Identifying the terms on the right-hand side of (10.54) that depend on π,w eh a v e
lnq⋆(π)=(α0−1)K∑
k=1lnπk+K∑
k=1N∑
n=1rnklnπk+c o n s t (10.56)
where we have used (10.50). Taking the exponential of both sides, we recognize
q⋆(π)as a Dirichlet distribution
q⋆(π)=D i r ( π|α) (10.57)
where αhas components αkgiven by
αk=α0+Nk. (10.58)
Finally, the variational posterior distribution q⋆(µk,Λk)does not factorize into
the product of the marginals, but we can always use the product rule to write it in the
formq⋆(µk,Λk)=q⋆(µk|Λk)q⋆(Λk). The two factors can be found by inspecting
(10.54) and reading off those terms that involve µkandΛk. The result, as expected,
is a Gaussian-Wishart distribution and is given by Exercise 10.13
q⋆(µk,Λk)=N(
µk|mk,(βkΛk)−1)
W(Λk|Wk,νk) (10.59)
βk=β0+Nk (10.60)
mk=1
βk(β0m0+Nkxk) (10.61)
W−1
k=W−1
β0+Nk(xk−m0)(xk−m0)T(10.62)
νk=ν0+Nk. (10.63)
These update equations are analogous to the M-step equations of the EM algorithm

mk=1
W−1
k=W−1
for the maximum likelihood solution of the mixture of Gaussians. We see that the
computations that must be performed in order to update the variational posterior
distribution over the model parameters involve evaluation of the same sums over thedata set, as arose in the maximum likelihood treatment.
In order to perform this variational M step, we need the expectations E[z
nk]=
rnkrepresenting the responsibilities. These are obtained by normalizing the ρnkthat

nk]=
are given by (10.46). We see that this expression involves expectations with respect
to the variational distributions of the parameters, and these are easily evaluated to
give Exercise 10.14
Eµk,Λk[
=Dβ−1
k+νk(xn−mk)TWk(xn−mk)(10.64)
ln˜Λk≡E[ln|Λk|]=D∑
i=1ψ(νk+1−i
2)
+Dln2 + ln |Wk|(10.65)
ln˜πk≡E[lnπk]= ψ(αk)−ψ(ˆα) (10.66)

10.2. Illustration: Variational Mixture of Gaussians 479
where we have introduced deﬁnitions of ˜Λkand˜πk, andψ(·)is the digamma function
deﬁned by (B.25), with ˆα=∑
kαk. The results (10.65) and (10.66) follow from
the standard properties of the Wishart and Dirichlet distributions. Appendix B
If we substitute (10.64), (10.65), and (10.66) into (10.46) and make use of
(10.49), we obtain the following result for the responsibilities
rnk∝˜πk˜Λ1/2
kexp{
−D
2βk−νk
2(xn−mk)TWk(xn−mk)}
. (10.67)
Notice the similarity to the corresponding result for the responsibilities in maximum
likelihood EM, which from (9.13) can be written in the form
rnk∝πk|Λk|1/2exp{
−1
2(xn−µk)TΛk(xn−µk)}
(10.68)
where we have used the precision in place of the covariance to highlight the similarity
to (10.67).
Thus the optimization of the variational posterior distribution involves cycling
between two stages analogous to the E and M steps of the maximum likelihood EM

kexp{
−D
2βk−νk
. (10.67)
−1
(10.68)
algorithm. In the variational equivalent of the E step, we use the current distributions
over the model parameters to evaluate the moments in (10.64), (10.65), and (10.66)and hence evaluate E[z
nk]=rnk. Then in the subsequent variational equivalent
of the M step, we keep these responsibilities ﬁxed and use them to re-compute the
variational distribution over the parameters using (10.57) and (10.59). In each case,we see that the variational posterior distribution has the same functional form as the

corresponding factor in the joint distribution (10.41). This is a general result and is
a consequence of the choice of conjugate distributions. Section 10.4.1
Figure 10.6 shows the results of applying this approach to the rescaled Old Faith-
ful data set for a Gaussian mixture model having K=6 components. We see that
after convergence, there are only two components for which the expected valuesof the mixing coefﬁcients are numerically distinguishable from their prior values.

This effect can be understood qualitatively in terms of the automatic trade-off in a
Bayesian model between ﬁtting the data and the complexity of the model, in which Section 3.4
the complexity penalty arises from components whose parameters are pushed away
from their prior values. Components that take essentially no responsibility for ex-plaining the data points have r
nk≃0and hence Nk≃0. From (10.58), we see
thatαk≃α0and from (10.60)–(10.63) we see that the other parameters revert to

their prior values. In principle such components are ﬁtted slightly to the data points,but for broad priors this effect is too small to be seen numerically. For the varia-
tional Gaussian mixture model the expected values of the mixing coefﬁcients in the
posterior distribution are given by Exercise 10.15
E[π
k]=αk+Nk
Kα0+N. (10.69)
Consider a component for which Nk≃0andαk≃α0. If the prior is broad so that
α0→0, then E[πk]→0and the component plays no role in the model, whereas if

480 10. APPROXIMATE INFERENCE
Figure 10.6 Variational Bayesian
mixture of K=6 Gaussians ap-
plied to the Old Faithful data set, in
which the ellipses denote the one
standard-deviation density contours
for each of the components, and the
density of red ink inside each ellipse
corresponds to the mean value of
the mixing coefﬁcient for each com-
ponent. The number in the top left
of each diagram shows the num-
ber of iterations of variational infer-
ence. Components whose expected
mixing coefﬁcient are numerically in-
distinguishable from zero are not
plotted.0 15
60 120
the prior tightly constrains the mixing coefﬁcients so that α0→∞ , then E[πk]→
1/K.
In Figure 10.6, the prior over the mixing coefﬁcients is a Dirichlet of the form
(10.39). Recall from Figure 2.5 that for α0<1the prior favours solutions in which
some of the mixing coefﬁcients are zero. Figure 10.6 was obtained using α0=1 0−3,
and resulted in two components having nonzero mixing coefﬁcients. If instead we

60 120
1/K.
choose α0=1 we obtain three components with nonzero mixing coefﬁcients, and
forα=1 0 all six components have nonzero mixing coefﬁcients.
As we have seen there is a close similarity between the variational solution for
the Bayesian mixture of Gaussians and the EM algorithm for maximum likelihood.
In fact if we consider the limit N→∞ then the Bayesian treatment converges to the
maximum likelihood EM algorithm. For anything other than very small data sets,

the dominant computational cost of the variational algorithm for Gaussian mixtures
arises from the evaluation of the responsibilities, together with the evaluation and
inversion of the weighted data covariance matrices. These computations mirror pre-
cisely those that arise in the maximum likelihood EM algorithm, and so there is little
computational overhead in using this Bayesian approach as compared to the tradi-
tional maximum likelihood one. There are, however, some substantial advantages.

First of all, the singularities that arise in maximum likelihood when a Gaussian com-
ponent ‘collapses’ onto a speciﬁc data point are absent in the Bayesian treatment.

10.2. Illustration: Variational Mixture of Gaussians 481
Indeed, these singularities are removed if we simply introduce a prior and then use a
MAP estimate instead of maximum likelihood. Furthermore, there is no over-ﬁttingif we choose a large number Kof components in the mixture, as we saw in Fig-
ure 10.6. Finally, the variational treatment opens up the possibility of determining
the optimal number of components in the mixture without resorting to techniquessuch as cross validation. Section 10.2.4
10.2.2 Variational lower bound
We can also straightforwardly evaluate the lower bound (10.3) for this model.
In practice, it is useful to be able to monitor the bound during the re-estimation in
order to test for convergence. It can also provide a valuable check on both the math-ematical expressions for the solutions and their software implementation, because at
each step of the iterative re-estimation procedure the value of this bound should not

decrease. We can take this a stage further to provide a deeper test of the correctnessof both the mathematical derivation of the update equations and of their software im-
plementation by using ﬁnite differences to check that each update does indeed give
a (constrained) maximum of the bound (Svens ´en and Bishop, 2004).
For the variational mixture of Gaussians, the lower bound (10.3) is given by
L=
∑
Z∫∫∫
q(Z,π,µ,Λ)l n{p(X,Z,π,µ,Λ)
q(Z,π,µ,Λ)}
dπdµdΛ
= E[lnp(X,Z,π,µ,Λ)]−E[lnq(Z,π,µ,Λ)]
= E[lnp(X|Z,µ,Λ)] + E[lnp(Z|π)] + E[lnp(π)] + E[lnp(µ,Λ)]

L=
∑
Z∫∫∫
dπdµdΛ
−E[lnq(Z)]−E[lnq(π)]−E[lnq(µ,Λ)] (10.70)
where, to keep the notation uncluttered, we have omitted the ⋆superscript on the
qdistributions, along with the subscripts on the expectation operators because each
expectation is taken with respect to all of the random variables in its argument. The
various terms in the bound are easily evaluated to give the following results Exercise 10.16
E[lnp(X|Z,µ,Λ)] =1
2K∑
k=1Nk{
k−νkTr(SkWk)
−νk(xk−mk)TWk(xk−mk)−Dln(2π)}
(10.71)
E[lnp(Z|π)] =N∑
n=1K∑
k=1rnkln˜πk (10.72)
E[lnp(π)] = ln C(α0)+(α0−1)K∑
k=1ln˜πk (10.73)

482 10. APPROXIMATE INFERENCE
E[lnp(µ,Λ)] =1
2K∑
k=1{
Dln(β0/2π)+l n˜Λk−Dβ0
βk
−β0νk(mk−m0)TWk(mk−m0)}
+KlnB(W0,ν0)
+(ν0−D−1)
2K∑
k=1ln˜Λk−1
2K∑
k=1νkTr(W−1
0Wk) (10.74)
E[lnq(Z)] =N∑
n=1K∑
k=1rnklnrnk (10.75)
E[lnq(π)] =K∑
k=1(αk−1) ln˜πk+l nC(α) (10.76)
E[lnq(µ,Λ)] =K∑
k=1{1
2ln˜Λk+D
2ln(βk
2π)
−D
2−H[q(Λk)]}
(10.77)
where Dis the dimensionality of x,H[q(Λk)]is the entropy of the Wishart distribu-
tion given by (B.82), and the coefﬁcients C(α)andB(W,ν)are deﬁned by (B.23)
and (B.79), respectively. Note that the terms involving expectations of the logs of theqdistributions simply represent the negative entropies of those distributions. Some
simpliﬁcations and combination of terms can be performed when these expressions
are summed to give the lower bound. However, we have kept the expressions sepa-rate for ease of understanding.
Finally, it is worth noting that the lower bound provides an alternative approach

for deriving the variational re-estimation equations obtained in Section 10.2.1. To dothis we use the fact that, since the model has conjugate priors, the functional form of
the factors in the variational posterior distribution is known, namely discrete for Z,
Dirichlet for π, and Gaussian-Wishart for (µ
k,Λk). By taking general parametric
forms for these distributions we can derive the form of the lower bound as a function
of the parameters of the distributions. Maximizing the bound with respect to these

parameters then gives the required re-estimation equations. Exercise 10.18
10.2.3 Predictive density
In applications of the Bayesian mixture of Gaussians model we will often be
interested in the predictive density for a new value ˆxof the observed variable. As-
sociated with this observation will be a corresponding latent variable ˆz, and the pre-
dictive density is then given by
p(ˆx|X)=∑
bz∫∫∫
p(ˆx|ˆz,µ,Λ)p(ˆz|π)p(π,µ,Λ|X)dπdµdΛ (10.78)

10.2. Illustration: Variational Mixture of Gaussians 483
where p(π,µ,Λ|X)is the (unknown) true posterior distribution of the parameters.
Using (10.37) and (10.38) we can ﬁrst perform the summation over ˆzto give
p(ˆx|X)=K∑
k=1∫∫∫
πkN(
ˆx|µk,Λ−1
k)
p(π,µ,Λ|X)dπdµdΛ. (10.79)
Because the remaining integrations are intractable, we approximate the predictive
density by replacing the true posterior distribution p(π,µ,Λ|X)with its variational
approximation q(π)q(µ,Λ)to give
k=1∫∫∫
πkN(ˆx|µk,Λ−1
k)
q(π)q(µk,Λk)dπdµkdΛk(10.80)
where we have made use of the factorization (10.55) and in each term we have im-
plicitly integrated out all variables {µj,Λj}forj̸=kThe remaining integrations
can now be evaluated analytically giving a mixture of Student’s t-distributions Exercise 10.19
p(ˆx|X)=1
ˆαK∑
k=1αkSt(ˆx|mk,Lk,νk+1−D) (10.81)
in which the kthcomponent has mean mk, and the precision is given by
Lk=(νk+1−D)βk
(1 +βk)Wk (10.82)

k=1∫∫∫
k)
p(ˆx|X)=1
ˆαK∑
in which νkis given by (10.63). When the size Nof the data set is large the predictive
distribution (10.81) reduces to a mixture of Gaussians. Exercise 10.20
10.2.4 Determining the number of components
We have seen that the variational lower bound can be used to determine a pos-
terior distribution over the number Kof components in the mixture model. There Section 10.1.4
is, however, one subtlety that needs to be addressed. For any given setting of the

parameters in a Gaussian mixture model (except for speciﬁc degenerate settings),there will exist other parameter settings for which the density over the observed vari-
ables will be identical. These parameter values differ only through a re-labelling of
the components. For instance, consider a mixture of two Gaussians and a single ob-served variable x, in which the parameters have the values π
1=a,π2=b,µ1=c,
µ2=d,σ1=e,σ2=f. Then the parameter values π1=b,π2=a,µ1=d,

µ2=c,σ1=f,σ2=e, in which the two components have been exchanged, will
by symmetry give rise to the same value of p(x). If we have a mixture model com-
prising Kcomponents, then each parameter setting will be a member of a family of
K!equivalent settings. Exercise 10.21
In the context of maximum likelihood, this redundancy is irrelevant because the
parameter optimization algorithm (for example EM) will, depending on the initial-ization of the parameters, ﬁnd one speciﬁc solution, and the other equivalent solu-

tions play no role. In a Bayesian setting, however, we marginalize over all possible

484 10. APPROXIMATE INFERENCE
Figure 10.7 Plot of the variational lower bound
Lversus the number Kof com-
ponents in the Gaussian mixture
model, for the Old Faithful data,
showing a distinct peak at K=
2components. For each value
ofK, the model is trained from
100different random starts, and
the results shown as ‘ +’ symbols
plotted with small random hori-
zontal perturbations so that they
can be distinguished. Note that
some solutions ﬁnd suboptimal
local maxima, but that this hap-
pens infrequently.
Kp(D|K)
1 2 3 4 5 6
parameter values. We have seen in Figure 10.2 that if the true posterior distribution
is multimodal, variational inference based on the minimization of KL(q∥p)will tend
to approximate the distribution in the neighbourhood of one of the modes and ignore
the others. Again, because equivalent modes have equivalent predictive densities,
this is of no concern provided we are considering a model having a speciﬁc number
Kof components. If, however, we wish to compare different values of K, then we

1 2 3 4 5 6
need to take account of this multimodality. A simple approximate solution is to add
a term lnK!onto the lower bound when used for model comparison and averaging. Exercise 10.22
Figure 10.7 shows a plot of the lower bound, including the multimodality fac-
tor, versus the number Kof components for the Old Faithful data set. It is worth
emphasizing once again that maximum likelihood would lead to values of the likeli-
hood function that increase monotonically with K(assuming the singular solutions

have been avoided, and discounting the effects of local maxima) and so cannot be
used to determine an appropriate model complexity. By contrast, Bayesian inference
automatically makes the trade-off between model complexity and ﬁtting the data. Section 3.4
This approach to the determination of Krequires that a range of models having
different Kvalues be trained and compared. An alternative approach to determining
a suitable value for Kis to treat the mixing coefﬁcients πas parameters and make

point estimates of their values by maximizing the lower bound (Corduneanu and
Bishop, 2001) with respect to πinstead of maintaining a probability distribution
over them as in the fully Bayesian approach. This leads to the re-estimation equation Exercise 10.23
πk=1
NN∑
n=1rnk (10.83)
and this maximization is interleaved with the variational updates for the qdistribution
over the remaining parameters. Components that provide insufﬁcient contribution

10.2. Illustration: Variational Mixture of Gaussians 485
to explaining the data will have their mixing coefﬁcients driven to zero during the
optimization, and so they are effectively removed from the model through automatic
relevance determination . This allows us to make a single training run in which we
start with a relatively large initial value of K, and allow surplus components to be
pruned out of the model. The origins of the sparsity when optimizing with respect tohyperparameters is discussed in detail in the context of the relevance vector machine. Section 7.2.2
10.2.5 Induced factorizations
In deriving these variational update equations for the Gaussian mixture model,
we assumed a particular factorization of the variational posterior distribution given
by (10.42). However, the optimal solutions for the various factors exhibit additional
factorizations. In particular, the solution for q⋆(µ,Λ)is given by the product of an
independent distribution q⋆(µk,Λk)over each of the components kof the mixture,

whereas the variational posterior distribution q⋆(Z)over the latent variables, given
by (10.48), factorizes into an independent distribution q⋆(zn)for each observation n
(note that it does not further factorize with respect to kbecause, for each value of n,
theznkare constrained to sum to one over k). These additional factorizations are a
consequence of the interaction between the assumed factorization and the conditionalindependence properties of the true distribution, as characterized by the directed
graph in Figure 10.5.

We shall refer to these additional factorizations as induced factorizations be-
cause they arise from an interaction between the factorization assumed in the varia-
tional posterior distribution and the conditional independence properties of the true
joint distribution. In a numerical implementation of the variational approach it is
important to take account of such additional factorizations. For instance, it would

be very inefﬁcient to maintain a full precision matrix for the Gaussian distributionover a set of variables if the optimal form for that distribution always had a diago-
nal precision matrix (corresponding to a factorization with respect to the individual
variables described by that Gaussian).
Such induced factorizations can easily be detected using a simple graphical test
based on d-separation as follows. We partition the latent variables into three disjoint
groups A,B,Cand then let us suppose that we are assuming a factorization between

Cand the remaining latent variables, so that
q(A,B,C)=q(A,B)q(C). (10.84)
Using the general result (10.9), together with the product rule for probabilities, we
see that the optimal solution for q(A,B)is given by
lnq
⋆(A,B)= EC[lnp(X,A,B,C)] + const
= EC[lnp(A,B|X,C)] + const . (10.85)
We now ask whether this resulting solution will factorize between AandB,i n
other words whether q⋆(A,B)=q⋆(A)q⋆(B). This will happen if, and only if,
lnp(A,B|X,C)=l n p(A|X,C)+l n p(B|X,C), that is, if the conditional inde-
pendence relation
A⊥⊥B|X,C (10.86)

486 10. APPROXIMATE INFERENCE
is satisﬁed. We can test to see if this relation does hold, for any choice of AandB
by making use of the d-separation criterion.
To illustrate this, consider again the Bayesian mixture of Gaussians represented
by the directed graph in Figure 10.5, in which we are assuming a variational fac-
torization given by (10.42). We can see immediately that the variational posteriordistribution over the parameters must factorize between πand the remaining param-
etersµandΛbecause all paths connecting πto either µorΛmust pass through
one of the nodes z
nall of which are in the conditioning set for our conditional inde-
pendence test and all of which are head-to-tail with respect to such paths.
10.3. Variational Linear Regression
As a second illustration of variational inference, we return to the Bayesian linear
regression model of Section 3.3. In the evidence framework, we approximated the
integration over αandβby making point estimates obtained by maximizing the log

marginal likelihood. A fully Bayesian approach would integrate over the hyperpa-
rameters as well as over the parameters. Although exact integration is intractable,
we can use variational methods to ﬁnd a tractable approximation. In order to sim-
plify the discussion, we shall suppose that the noise precision parameter βis known,
and is ﬁxed to its true value, although the framework is easily extended to includethe distribution over β. For the linear regression model, the variational treatment Exercise 10.26

will turn out to be equivalent to the evidence framework. Nevertheless, it provides a
good exercise in the use of variational methods and will also lay the foundation forvariational treatment of Bayesian logistic regression in Section 10.6.
Recall that the likelihood function for w, and the prior over w, are given by
p(t|w)=
N∏
n=1N(tn|wTφn,β−1) (10.87)
p(w|α)= N(w|0,α−1I) (10.88)
where φn=φ(xn). We now introduce a prior distribution over α. From our dis-
cussion in Section 2.3.6, we know that the conjugate prior for the precision of a

p(t|w)=
N∏
Gaussian is given by a gamma distribution, and so we choose
p(α)=G a m ( α|a0,b0) (10.89)
where Gam(·|·,·)is deﬁned by (B.26). Thus the joint distribution of all the variables
is given by
p(t,w,α)=p(t|w)p(w|α)p(α). (10.90)
This can be represented as a directed graphical model as shown in Figure 10.8.
10.3.1 Variational distribution
Our ﬁrst goal is to ﬁnd an approximation to the posterior distribution p(w,α|t).
To do this, we employ the variational framework of Section 10.1, with a variational

10.3. Variational Linear Regression 487
Figure 10.8 Probabilistic graphical model representing the joint dis-
tribution (10.90) for the Bayesian linear regression
model.
tnφn
Nwα
β
posterior distribution given by the factorized expression
q(w,α)=q(w)q(α). (10.91)
We can ﬁnd re-estimation equations for the factors in this distribution by making use
of the general result (10.9). Recall that for each factor, we take the log of the joint
distribution over all variables and then average with respect to those variables not in
that factor. Consider ﬁrst the distribution over α. Keeping only terms that have a
functional dependence on α,w eh a v e
lnq⋆(α)=l n p(α)+ Ew[lnp(w|α) ]+c o n s t
=(a0−1) lnα−b0α+M
2lnα−α
2E[wTw]+c o n s t . (10.92)
We recognize this as the log of a gamma distribution, and so identifying the coefﬁ-
cients of αandlnαwe obtain
q⋆(α)=G a m ( α|aN,bN) (10.93)
where
aN=a0+M
2(10.94)
bN=b0+1
2E[wTw]. (10.95)
Similarly, we can ﬁnd the variational re-estimation equation for the posterior

2lnα−α
where
aN=a0+M
2(10.94)
bN=b0+1
distribution over w. Again, using the general result (10.9), and keeping only those
terms that have a functional dependence on w,w eh a v e
lnq⋆(w)=l n p(t|w)+ Eα[lnp(w|α)] + const (10.96)
=−β
2N∑
n=1{wTφn−tn}2−1
2E[α]wTw+c o n s t (10.97)
=−1
2wT(
E[α]I+βΦTΦ)
w+βwTΦTt+c o n s t . (10.98)
Because this is a quadratic form, the distribution q⋆(w)is Gaussian, and so we can
complete the square in the usual way to identify the mean and covariance, giving
q⋆(w)=N(w|mN,SN) (10.99)

488 10. APPROXIMATE INFERENCE
where
mN=βSNΦTt (10.100)
SN=(
E[α]I+βΦTΦ)−1. (10.101)
Note the close similarity to the posterior distribution (3.52) obtained when αwas
treated as a ﬁxed parameter. The difference is that here αis replaced by its expecta-
tion E[α]under the variational distribution. Indeed, we have chosen to use the same
notation for the covariance matrix SNin both cases.
Using the standard results (B.27), (B.38), and (B.39), we can obtain the required
moments as follows
E[α]= aN/bN (10.102)
E[wwT]= mNmT
N+SN. (10.103)
The evaluation of the variational posterior distribution begins by initializing the pa-
rameters of one of the distributions q(w)orq(α), and then alternately re-estimates
these factors in turn until a suitable convergence criterion is satisﬁed (usually speci-
ﬁed in terms of the lower bound to be discussed shortly).
It is instructive to relate the variational solution to that found using the evidence
framework in Section 3.5. To do this consider the case a0=b0=0, corresponding

to the limit of an inﬁnitely broad prior over α. The mean of the variational posterior
distribution q(α)is then given by
E[α]=aN
bN=M/2
E[wTw]/2=M
mT
NmN+Tr(SN). (10.104)
Comparison with (9.63) shows that in the case of this particularly simple model,
the variational approach gives precisely the same expression as that obtained by
maximizing the evidence function using EM except that the point estimate for α
is replaced by its expected value. Because the distribution q(w)depends on q(α)

E[α]=aN
bN=M/2
mT
only through the expectation E[α], we see that the two approaches will give identical
results for the case of an inﬁnitely broad prior.
10.3.2 Predictive distribution
The predictive distribution over t, given a new input x, is easily evaluated for
this model using the Gaussian variational posterior for the parameters
p(t|x,t)=∫
p(t|x,w)p(w|t)dw
≃∫
p(t|x,w)q(w)dw
=∫
N(t|wTφ(x),β−1)N(w|mN,SN)dw
=N(t|mT
Nφ(x),σ2(x)) (10.105)

10.3. Variational Linear Regression 489
where we have evaluated the integral by making use of the result (2.115) for the
linear-Gaussian model. Here the input-dependent variance is given by
σ2(x)=1
β+φ(x)TSNφ(x). (10.106)
Note that this takes the same form as the result (3.59) obtained with ﬁxed αexcept
that now the expected value E[α]appears in the deﬁnition of SN.
10.3.3 Lower bound
Another quantity of importance is the lower bound Ldeﬁned by
L(q)= E[lnp(w,α,t)]−E[lnq(w,α)]
= Ew[lnp(t|w)] + Ew,α[lnp(w|α)] + Eα[lnp(α)]
−Eα[lnq(w)]w−E[lnq(α)]. (10.107)
Evaluation of the various terms is straightforward, making use of results obtained in Exercise 10.27
previous chapters, and gives
E[lnp(t|w)]w=N
2ln(β
2π)
−β
2tTt+βmT
NΦTt
−β
2Tr[
ΦTΦ(mNmT
N+SN)]
(10.108)
E[lnp(w|α)]w,α=−M
2ln(2π)+M
2(ψ(aN)−lnbN)
−aN
2bN[
mT
NmN+Tr(SN)]
(10.109)
E[lnp(α)]α=a0lnb0+(a0−1) [ψ(aN)−lnbN]
−b0aN
bN−lnΓ(aN) (10.110)
−E[lnq(w)]w=1
2ln|SN|+M
2[1 + ln(2 π)] (10.111)
−E[lnq(α)]α=l n Γ ( aN)−(aN−1)ψ(aN)−lnbN+aN.(10.112)

2ln(β
2π)
−β
2tTt+βmT
NΦTt
−β
2Tr[
ΦTΦ(mNmT
N+SN)]
(10.108)
2ln(2π)+M
−aN
2bN[
mT
(10.109)
−b0aN
2ln|SN|+M
Figure 10.9 shows a plot of the lower bound L(q)versus the degree of a polynomial
model for a synthetic data set generated from a degree three polynomial. Here theprior parameters have been set to a
0=b0=0, corresponding to the noninformative
priorp(α)∝1/α, which is uniform over lnαas discussed in Section 2.3.6. As
we saw in Section 10.1, the quantity Lrepresents lower bound on the log marginal
likelihood p(t|M)for the model. If we assign equal prior probabilities p(M)to the

different values of M, then we can interpret Las an approximation to the poste-
rior model probability p(M|t). Thus the variational framework assigns the highest
probability to the model with M=3. This should be contrasted with the maximum
likelihood result, which assigns ever smaller residual error to models of increasingcomplexity until the residual error is driven to zero, causing maximum likelihood to
favour severely over-ﬁtted models.

490 10. APPROXIMATE INFERENCE
Figure 10.9 Plot of the lower bound Lver-
sus the order Mof the polyno-
mial, for a polynomial model, in
which a set of 10data points is
generated from a polynomial with
M=3 sampled over the inter-
val(−5,5)with additive Gaussian
noise of variance 0.09. The value
of the bound gives the log prob-
ability of the model, and we see
that the value of the bound peaks
atM=3, corresponding to the
true model from which the data
set was generated.
1 3 5 7 9
10.4. Exponential Family Distributions
In Chapter 2, we discussed the important role played by the exponential family of
distributions and their conjugate priors. For many of the models discussed in this
book, the complete-data likelihood is drawn from the exponential family. However,
in general this will not be the case for the marginal likelihood function for the ob-
served data. For example, in a mixture of Gaussians, the joint distribution of obser-
vations xnand corresponding hidden variables znis a member of the exponential

family, whereas the marginal distribution of xnis a mixture of Gaussians and hence
is not.
Up to now we have grouped the variables in the model into observed variables
and hidden variables. We now make a further distinction between latent variables,
denoted Z, and parameters, denoted θ, where parameters are intensive (ﬁxed in num-
ber independent of the size of the data set), whereas latent variables are extensive
(scale in number with the size of the data set). For example, in a Gaussian mixture

is not.
model, the indicator variables zkn(which specify which component kis responsible
for generating data point xn) represent the latent variables, whereas the means µk,
precisions Λkand mixing proportions πkrepresent the parameters.
Consider the case of independent identically distributed data. We denote the
data values by X={xn}, where n=1,...N , with corresponding latent variables
Z={zn}. Now suppose that the joint distribution of observed and latent variables

is a member of the exponential family, parameterized by natural parameters ηso that
p(X,Z|η)=N∏
n=1h(xn,zn)g(η)e x p{
ηTu(xn,zn)}
. (10.113)
We shall also use a conjugate prior for η, which can be written as
p(η|ν0,v0)=f(ν0,χ0)g(η)ν0exp{
νoηTχ0}
. (10.114)
Recall that the conjugate prior distribution can be interpreted as a prior number ν0
of observations all having the value χ0for the uvector. Now consider a variational

10.4. Exponential Family Distributions 491
distribution that factorizes between the latent variables and the parameters, so that
q(Z,η)=q(Z)q(η). Using the general result (10.9), we can solve for the two
factors as follows
lnq⋆(Z)= Eη[lnp(X,Z|η) ]+c o n s t
=N∑
n=1{
lnh(xn,zn)+ E[ηT]u(xn,zn)}
+c o n s t .(10.115)
Thus we see that this decomposes into a sum of independent terms, one for each
value of n, and hence the solution for q⋆(Z)will factorize over nso that q⋆(Z)=∏
nq⋆(zn). This is an example of an induced factorization. Taking the exponential Section 10.2.5
of both sides, we have
q⋆(zn)=h(xn,zn)g(E[η])exp{
E[ηT]u(xn,zn)}
(10.116)
where the normalization coefﬁcient has been re-instated by comparison with the
standard form for the exponential family.
Similarly, for the variational distribution over the parameters, we have
lnq⋆(η)=l n p(η|ν0,χ0)+ EZ[lnp(X,Z|η) ]+c o n s t (10.117)
=ν0lng(η)+ηTχ0+N∑
n=1{
lng(η)+ηTEzn[u(xn,zn)]}
+c o n s t .(10.118)

(10.116)
n=1{
Again, taking the exponential of both sides, and re-instating the normalization coef-
ﬁcient by inspection, we have
q⋆(η)=f(νN,χN)g(η)νNexp{
ηTχN}
(10.119)
νN=ν0+N (10.120)
χN=χ0+N∑
n=1Ezn[u(xn,zn)]. (10.121)
Note that the solutions for q⋆(zn)andq⋆(η)are coupled, and so we solve them iter-
atively in a two-stage procedure. In the variational E step, we evaluate the expected
sufﬁcient statistics E[u(xn,zn)]using the current posterior distribution q(zn)over

ηTχN}
(10.119)
χN=χ0+N∑
the latent variables and use this to compute a revised posterior distribution q(η)over
the parameters. Then in the subsequent variational M step, we use this revised pa-
rameter posterior distribution to ﬁnd the expected natural parameters E[ηT], which
gives rise to a revised variational distribution over the latent variables.
10.4.1 Variational message passing
We have illustrated the application of variational methods by considering a spe-

ciﬁc model, the Bayesian mixture of Gaussians, in some detail. This model can be

492 10. APPROXIMATE INFERENCE
described by the directed graph shown in Figure 10.5. Here we consider more gen-
erally the use of variational methods for models described by directed graphs andderive a number of widely applicable results.
The joint distribution corresponding to a directed graph can be written using the
decomposition
p(x)=∏
ip(xi|pai) (10.122)
wherexidenotes the variable(s) associated with node i, andpaidenotes the parent
set corresponding to node i. Note that ximay be a latent variable or it may belong
to the set of observed variables. Now consider a variational approximation in which
the distribution q(x)is assumed to factorize with respect to the xiso that
q(x)=∏
iqi(xi). (10.123)
Note that for observed nodes, there is no factor q(xi)in the variational distribution.
We now substitute (10.122) into our general result (10.9) to give
lnq⋆
j(xj)= Ei̸=j[∑
ilnp(xi|pai)]
+c o n s t . (10.124)
Any terms on the right-hand side that do not depend on xjcan be absorbed into

q(x)=∏
lnq⋆
the additive constant. In fact, the only terms that do depend on xjare the con-
ditional distribution for xjgiven by p(xj|paj)together with any other conditional
distributions that have xjin the conditioning set. By deﬁnition, these conditional
distributions correspond to the children of node j, and they therefore also depend on
theco-parents of the child nodes, i.e., the other parents of the child nodes besides
nodexjitself. We see that the set of all nodes on which q⋆(xj)depends corresponds

to the Markov blanket of node xj, as illustrated in Figure 8.26. Thus the update
of the factors in the variational posterior distribution represents a local calculationon the graph. This makes possible the construction of general purpose software for
variational inference in which the form of the model does not need to be speciﬁed in
advance (Bishop et al. , 2003).
If we now specialize to the case of a model in which all of the conditional dis-

tributions have a conjugate-exponential structure, then the variational update proce-
dure can be cast in terms of a local message passing algorithm (Winn and Bishop,2005). In particular, the distribution associated with a particular node can be updated
once that node has received messages from all of its parents and all of its children.
This in turn requires that the children have already received messages from their co-parents. The evaluation of the lower bound can also be simpliﬁed because many of

the required quantities are already evaluated as part of the message passing scheme.
This distributed message passing formulation has good scaling properties and is well
suited to large networks.

10.5. Local Variational Methods 493
10.5. Local Variational Methods
The variational framework discussed in Sections 10.1 and 10.2 can be considered a
‘global’ method in the sense that it directly seeks an approximation to the full poste-
rior distribution over all random variables. An alternative ‘local’ approach involves
ﬁnding bounds on functions over individual variables or groups of variables within
a model. For instance, we might seek a bound on a conditional distribution p(y|x),
which is itself just one factor in a much larger probabilistic model speciﬁed by a
directed graph. The purpose of introducing the bound of course is to simplify the
resulting distribution. This local approximation can be applied to multiple variables
in turn until a tractable approximation is obtained, and in Section 10.6.1 we shall
give a practical example of this approach in the context of logistic regression. Here
we focus on developing the bounds themselves.

We have already seen in our discussion of the Kullback-Leibler divergence that
the convexity of the logarithm function played a key role in developing the lower
bound in the global variational approach. We have deﬁned a (strictly) convex func-
tion as one for which every chord lies above the function. Convexity also plays a Section 1.6.1
central role in the local variational framework. Note that our discussion will ap-
ply equally to concave functions with ‘min’ and ‘max’ interchanged and with lower
bounds replaced by upper bounds.

Let us begin by considering a simple example, namely the function f(x)=
exp(−x), which is a convex function of x, and which is shown in the left-hand plot
of Figure 10.10. Our goal is to approximate f(x)by a simpler function, in particular
a linear function of x. From Figure 10.10, we see that this linear function will be a
lower bound on f(x)if it corresponds to a tangent. We can obtain the tangent line
y(x)at a speciﬁc value of x, sayx=ξ, by making a ﬁrst order Taylor expansion
y(x)=f(ξ)+f′(ξ)(x−ξ) (10.125)

so that y(x)⩽f(x)with equality when x=ξ. For our example function f(x)=
Figure 10.10 In the left-hand ﬁg-
ure the red curve shows the function
exp(−x), and the blue line shows
the tangent at x=ξdeﬁned by
(10.125) with ξ=1. This line has
slope λ=f′(ξ)=−exp(−ξ). Note
that any other tangent line, for ex-
ample the ones shown in green, will
have a smaller value of yatx=
ξ. The right-hand ﬁgure shows the
corresponding plot of the function
λξ−g(λ), where g(λ)is given by
(10.131), versus λforξ=1,i n
which the maximum corresponds to
λ=−exp(−ξ)=−1/e.x ξ 0 1.5 300.51
λλξ−g(λ)

λλξ−g(λ)
−1 −0.5 000.20.4

494 10. APPROXIMATE INFERENCE
xy
f(x)
λxxy
f(x)
λx−g(λ)−g(λ)
Figure 10.11 In the left-hand plot the red curve shows a convex function f(x), and the blue line represents the
linear function λx, which is a lower bound on f(x)because f(x)>λ x for all x. For the given value of slope λthe
contact point of the tangent line having the same slope is found by minimizing with respect to xthe discrepancy
(shown by the green dashed lines) given by f(x)−λx. This deﬁnes the dual function g(λ), which corresponds
to the (negative of the) intercept of the tangent line having slope λ.
exp(−x), we therefore obtain the tangent line in the form
y(x)=e x p ( −ξ)−exp(−ξ)(x−ξ) (10.126)
which is a linear function parameterized by ξ. For consistency with subsequent
discussion, let us deﬁne λ=−exp(−ξ)so that
y(x, λ)=λx−λ+λln(−λ). (10.127)
Different values of λcorrespond to different tangent lines, and because all such lines
are lower bounds on the function, we have f(x)⩾y(x, λ). Thus we can write the
function in the form
f(x)=m a x

f(x)=m a x
λ{λx−λ+λln(−λ)}. (10.128)
We have succeeded in approximating the convex function f(x)by a simpler, lin-
ear function y(x, λ). The price we have paid is that we have introduced a variational
parameter λ, and to obtain the tightest bound we must optimize with respect to λ.
We can formulate this approach more generally using the framework of convex
duality (Rockafellar, 1972; Jordan et al. , 1999). Consider the illustration of a convex
function f(x)shown in the left-hand plot in Figure 10.11. In this example, the

function λxis a lower bound on f(x)but it is not the best lower bound that can
be achieved by a linear function having slope λ, because the tightest bound is given
by the tangent line. Let us write the equation of the tangent line, having slope λas
λx−g(λ)where the (negative) intercept g(λ)clearly depends on the slope λof the
tangent. To determine the intercept, we note that the line must be moved vertically by
an amount equal to the smallest vertical distance between the line and the function,
as shown in Figure 10.11. Thus

g(λ)= −min
x{f(x)−λx}
=m a x
x{λx−f(x)}. (10.129)

10.5. Local Variational Methods 495
Now, instead of ﬁxing λand varying x, we can consider a particular xand then
adjust λuntil the tangent plane is tangent at that particular x. Because the yvalue
of the tangent line at a particular xis maximized when that value coincides with its
contact point, we have
f(x) = max
λ{λx−g(λ)}. (10.130)
We see that the functions f(x)andg(λ)play a dual role, and are related through
(10.129) and (10.130).
Let us apply these duality relations to our simple example f(x)=e x p ( −x).
From (10.129) we see that the maximizing value of xis given by ξ=−ln(−λ), and
back-substituting we obtain the conjugate function g(λ)in the form
g(λ)=λ−λln(−λ) (10.131)
as obtained previously. The function λξ−g(λ)is shown, for ξ=1in the right-hand
plot in Figure 10.10. As a check, we can substitute (10.131) into (10.130), which
gives the maximizing value of λ=−exp(−x), and back-substituting then recovers
the original function f(x)=e x p ( −x).

For concave functions, we can follow a similar argument to obtain upper bounds,
in which max’ is replaced with ‘ min’, so that
f(x) = min
λ{λx−g(λ)} (10.132)
g(λ) = min
x{λx−f(x)}. (10.133)
If the function of interest is not convex (or concave), then we cannot directly
apply the method above to obtain a bound. However, we can ﬁrst seek invertibletransformations either of the function or of its argument which change it into a con-
vex form. We then calculate the conjugate function and then transform back to the
original variables.

f(x) = min
g(λ) = min
An important example, which arises frequently in pattern recognition, is the
logistic sigmoid function deﬁned by
σ(x)=1
1+e−x. (10.134)
As it stands this function is neither convex nor concave. However, if we take the
logarithm we obtain a function which is concave, as is easily veriﬁed by ﬁnding the
second derivative. From (10.133) the corresponding conjugate function then takes Exercise 10.30
the form
g(λ)=m i n
x{λx−f(x)}=−λlnλ−(1−λ)l n ( 1−λ) (10.135)
which we recognize as the binary entropy function for a variable whose probability

σ(x)=1
the form
g(λ)=m i n
of having the value 1isλ. Using (10.132), we then obtain an upper bound on the log Appendix B
sigmoid
lnσ(x)⩽λx−g(λ) (10.136)

496 10. APPROXIMATE INFERENCE
λ=0.2
λ=0.7
−6 0 600.51
ξ=2.5
−ξξ −6 0 600.51
Figure 10.12 The left-hand plot shows the logistic sigmoid function σ(x)deﬁned by (10.134) in red, together
with two examples of the exponential upper bound (10.137) shown in blue. The right-hand plot shows the logistic
sigmoid again in red together with the Gaussian lower bound (10.144) shown in blue. Here the parameter
ξ=2.5, and the bound is exact at x=ξandx=−ξ, denoted by the dashed green lines.
and taking the exponential, we obtain an upper bound on the logistic sigmoid itself
of the form
σ(x)⩽exp(λx−g(λ)) (10.137)
which is plotted for two values of λon the left-hand plot in Figure 10.12.
We can also obtain a lower bound on the sigmoid having the functional form of
a Gaussian. To do this, we follow Jaakkola and Jordan (2000) and make transforma-
tions both of the input variable and of the function itself. First we take the log of the
logistic function and then decompose it so that
lnσ(x)= −ln(1 + e−x)=−ln{
e−x/2(ex/2+e−x/2)}

of the form
=x/2−ln(ex/2+e−x/2). (10.138)
We now note that the function f(x)=−ln(ex/2+e−x/2)is a convex function of
the variable x2, as can again be veriﬁed by ﬁnding the second derivative. This leads Exercise 10.31
to a lower bound on f(x), which is a linear function of x2whose conjugate function
is given by
g(λ)=m a x
x2{
λx2−f(√
x2)}
. (10.139)
The stationarity condition leads to
0=λ−dx
dx2d
dxf(x)=λ+1
4xtanh(x
2)
. (10.140)
If we denote this value of x, corresponding to the contact point of the tangent line
for this particular value of λ,b yξ, then we have
λ(ξ)=−1

is given by
g(λ)=m a x
x2{
λx2−f(√
x2)}
. (10.139)
0=λ−dx
dx2d
4xtanh(x
2)
. (10.140)
λ(ξ)=−1
4ξtanh(ξ
2)
=−1
2ξ[
σ(ξ)−1
2]
. (10.141)

10.5. Local Variational Methods 497
Instead of thinking of λas the variational parameter, we can let ξplay this role as
this leads to simpler expressions for the conjugate function, which is then given by
g(λ)=λ(ξ)ξ2−f(ξ)=λ(ξ)ξ2+l n (eξ/2+e−ξ/2). (10.142)
Hence the bound on f(x)can be written as
f(x)⩾λx2−g(λ)=λx2−λξ2−ln(eξ/2+e−ξ/2). (10.143)
The bound on the sigmoid then becomes
σ(x)⩾σ(ξ)e x p{
(x−ξ)/2−λ(ξ)(x2−ξ2)}
(10.144)
where λ(ξ)is deﬁned by (10.141). This bound is illustrated in the right-hand plot of
Figure 10.12. We see that the bound has the form of the exponential of a quadratic
function of x, which will prove useful when we seek Gaussian representations of
posterior distributions deﬁned through logistic sigmoid functions. Section 4.5
The logistic sigmoid arises frequently in probabilistic models over binary vari-
ables because it is the function that transforms a log odds ratio into a posterior prob-

(10.144)
ability. The corresponding transformation for a multiclass distribution is given bythe softmax function. Unfortunately, the lower bound derived here for the logistic Section 4.3
sigmoid does not directly extend to the softmax. Gibbs (1997) proposes a method
for constructing a Gaussian distribution that is conjectured to be a bound (although
no rigorous proof is given), which may be used to apply local variational methods to
multiclass problems.

We shall see an example of the use of local variational bounds in Sections 10.6.1.
For the moment, however, it is instructive to consider in general terms how these
bounds can be used. Suppose we wish to evaluate an integral of the form
I=∫
σ(a)p(a)da (10.145)
where σ(a)is the logistic sigmoid, and p(a)is a Gaussian probability density. Such
integrals arise in Bayesian models when, for instance, we wish to evaluate the pre-
dictive distribution, in which case p(a)represents a posterior parameter distribution.

I=∫
Because the integral is intractable, we employ the variational bound (10.144), which
we write in the form σ(a)⩾f(a, ξ)where ξis a variational parameter. The inte-
gral now becomes the product of two exponential-quadratic functions and so can beintegrated analytically to give a bound on I
I⩾∫
f(a, ξ)p(a)da=F(ξ). (10.146)
We now have the freedom to choose the variational parameter ξ, which we do by
ﬁnding the value ξ⋆that maximizes the function F(ξ). The resulting value F(ξ⋆)
represents the tightest bound within this family of bounds and can be used as an

I⩾∫
approximation to I. This optimized bound, however, will in general not be exact.

498 10. APPROXIMATE INFERENCE
Although the bound σ(a)⩾f(a, ξ)on the logistic sigmoid can be optimized exactly,
the required choice for ξdepends on the value of a, so that the bound is exact for one
value of aonly. Because the quantity F(ξ)is obtained by integrating over all values
ofa, the value of ξ⋆represents a compromise, weighted by the distribution p(a).
10.6. Variational Logistic Regression
We now illustrate the use of local variational methods by returning to the Bayesian
logistic regression model studied in Section 4.5. There we focussed on the use of
the Laplace approximation, while here we consider a variational treatment based on
the approach of Jaakkola and Jordan (2000). Like the Laplace method, this alsoleads to a Gaussian approximation to the posterior distribution. However, the greater
ﬂexibility of the variational approximation leads to improved accuracy compared
to the Laplace method. Furthermore (unlike the Laplace method), the variational

approach is optimizing a well deﬁned objective function given by a rigourous bound
on the model evidence. Logistic regression has also been treated by Dybowski andRoberts (2005) from a Bayesian perspective using Monte Carlo sampling techniques.
10.6.1 Variational posterior distribution
Here we shall make use of a variational approximation based on the local bounds
introduced in Section 10.5. This allows the likelihood function for logistic regres-
sion, which is governed by the logistic sigmoid, to be approximated by the expo-

nential of a quadratic form. It is therefore again convenient to choose a conjugateGaussian prior of the form (4.140). For the moment, we shall treat the hyperparam-
etersm
0andS0as ﬁxed constants. In Section 10.6.3, we shall demonstrate how the
variational formalism can be extended to the case where there are unknown hyper-parameters whose values are to be inferred from the data.
In the variational framework, we seek to maximize a lower bound on the marginal
likelihood. For the Bayesian logistic regression model, the marginal likelihood takesthe form
p(t)=∫

etersm
p(t)=∫
p(t|w)p(w)dw=∫[N∏
n=1p(tn|w)]
p(w)dw. (10.147)
We ﬁrst note that the conditional distribution for tcan be written as
p(t|w)= σ(a)t{1−σ(a)}1−t
=(1
1+e−a)t(
1−1
1+e−a)1−t
=eate−a
1+e−a=eatσ(−a) (10.148)
where a=wTφ. In order to obtain a lower bound on p(t), we make use of the
variational lower bound on the logistic sigmoid function given by (10.144), which

10.6. Variational Logistic Regression 499
we reproduce here for convenience
σ(z)⩾σ(ξ)e x p{
(z−ξ)/2−λ(ξ)(z2−ξ2)}
(10.149)
where
λ(ξ)=1
2ξ[
σ(ξ)−1
2]
. (10.150)
We can therefore write
p(t|w)=eatσ(−a)⩾eatσ(ξ)e x p{
−(a+ξ)/2−λ(ξ)(a2−ξ2)}
.(10.151)
Note that because this bound is applied to each of the terms in the likelihood function
separately, there is a variational parameter ξncorresponding to each training set
observation (φn,tn). Using a=wTφ, and multiplying by the prior distribution, we
obtain the following bound on the joint distribution of tandw
p(t,w)=p(t|w)p(w)⩾h(w,ξ)p(w) (10.152)
where ξdenotes the set {ξn}of variational parameters, and
h(w,ξ)=N∏
n=1σ(ξn)e x p{
wTφntn−(wTφn+ξn)/2
−λ(ξn)([wTφn]2−ξ2
n)}
. (10.153)
Evaluation of the exact posterior distribution would require normalization of the left-
hand side of this inequality. Because this is intractable, we work instead with the
right-hand side. Note that the function on the right-hand side cannot be interpreted

h(w,ξ)=N∏
n)}
. (10.153)
as a probability density because it is not normalized. Once it is normalized to give avariational posterior distribution q(w), however, it no longer represents a bound.
Because the logarithm function is monotonically increasing, the inequality A⩾
Bimplies lnA⩾lnB. This gives a lower bound on the log of the joint distribution
oftandwof the form
ln{p(t|w)p(w)}⩾lnp(w)+N∑
n=1{
lnσ(ξn)+wTφntn
−(wTφn+ξn)/2−λ(ξn)([wTφn]2−ξ2
n)}
. (10.154)
Substituting for the prior p(w), the right-hand side of this inequality becomes, as a

n=1{
n)}
. (10.154)
function of w
−1
0(w−m0)
+N∑
n=1{
wTφn(tn−1/2)−λ(ξn)wT(φnφT
n)w}
+c o n s t . (10.155)

500 10. APPROXIMATE INFERENCE
This is a quadratic function of w, and so we can obtain the corresponding variational
approximation to the posterior distribution by identifying the linear and quadraticterms in w, giving a Gaussian variational posterior of the form
q(w)=N(w|m
N,SN) (10.156)
where
mN=SN(
S−1
0m0+N∑
n=1(tn−1/2)φn)
(10.157)
S−1
N=S−1
0+2N∑
n=1λ(ξn)φnφT
n. (10.158)
As with the Laplace framework, we have again obtained a Gaussian approximation
to the posterior distribution. However, the additional ﬂexibility provided by the vari-ational parameters {ξ
n}leads to improved accuracy in the approximation (Jaakkola
and Jordan, 2000).
Here we have considered a batch learning context in which all of the training
data is available at once. However, Bayesian methods are intrinsically well suited
to sequential learning in which the data points are processed one at a time and then
discarded. The formulation of this variational approach for the sequential case is
straightforward. Exercise 10.32

Note that the bound given by (10.149) applies only to the two-class problem and
so this approach does not directly generalize to classiﬁcation problems with K>2
classes. An alternative bound for the multiclass case has been explored by Gibbs
(1997).
10.6.2 Optimizing the variational parameters
We now have a normalized Gaussian approximation to the posterior distribution,
which we shall use shortly to evaluate the predictive distribution for new data points.
First, however, we need to determine the variational parameters {ξn}by maximizing

(1997).
the lower bound on the marginal likelihood.
To do this, we substitute the inequality (10.152) back into the marginal likeli-
hood to give
lnp(t)=l n∫
p(t|w)p(w)dw⩾ln∫
h(w,ξ)p(w)dw=L(ξ). (10.159)
As with the optimization of the hyperparameter αin the linear regression model of
Section 3.5, there are two approaches to determining the ξn. In the ﬁrst approach, we
recognize that the function L(ξ)is deﬁned by an integration over wand so we can
viewwas a latent variable and invoke the EM algorithm. In the second approach,

we integrate over wanalytically and then perform a direct maximization over ξ. Let
us begin by considering the EM approach.
The EM algorithm starts by choosing some initial values for the parameters
{ξn}, which we denote collectively by ξold. In the E step of the EM algorithm,

10.6. Variational Logistic Regression 501
we then use these parameter values to ﬁnd the posterior distribution over w, which
is given by (10.156). In the M step, we then maximize the expected complete-datalog likelihood which is given by
Q(ξ,ξ
old)= E[lnh(w,ξ)p(w)] (10.160)
where the expectation is taken with respect to the posterior distribution q(w)evalu-
ated using ξold. Noting that p(w)does not depend on ξ, and substituting for h(w,ξ)
we obtain
Q(ξ,ξold)=N∑
n=1{
lnσ(ξn)−ξn/2−λ(ξn)(φT
nE[wwT]φn−ξ2
n)}
+c o n s t
(10.161)
where ‘const’ denotes terms that are independent of ξ. We now set the derivative with
respect to ξnequal to zero. A few lines of algebra, making use of the deﬁnitions of
σ(ξ)andλ(ξ), then gives
0=λ′(ξn)(φT
n). (10.162)
We now note that λ′(ξ)is a monotonic function of ξforξ⩾0, and that we can
restrict attention to nonnegative values of ξwithout loss of generality due to the
symmetry of the bound around ξ=0. Thus λ′(ξ)̸=0, and hence we obtain the

n)}
+c o n s t
(10.161)
following re-estimation equations Exercise 10.33
(ξnew
n)2=φT
nE[wwT]φn=φT
n(
SN+mNmT
N)
φn (10.163)
where we have used (10.156).
Let us summarize the EM algorithm for ﬁnding the variational posterior distri-
bution. We ﬁrst initialize the variational parameters ξold. In the E step, we evaluate
the posterior distribution over wgiven by (10.156), in which the mean and covari-
ance are deﬁned by (10.157) and (10.158). In the M step, we then use this variational

(ξnew
n)2=φT
n(
SN+mNmT
N)
posterior to compute a new value for ξgiven by (10.163). The E and M steps are
repeated until a suitable convergence criterion is satisﬁed, which in practice typicallyrequires only a few iterations.
An alternative approach to obtaining re-estimation equations for ξis to note
that in the integral over win the deﬁnition (10.159) of the lower bound L(ξ), the
integrand has a Gaussian-like form and so the integral can be evaluated analytically.
Having evaluated the integral, we can then differentiate with respect to ξ
n. It turns

n. It turns
out that this gives rise to exactly the same re-estimation equations as does the EMapproach given by (10.163). Exercise 10.34
As we have emphasized already, in the application of variational methods it is
useful to be able to evaluate the lower bound L(ξ)given by (10.159). The integration
overwcan be performed analytically by noting that p(w)is Gaussian and h(w,ξ)
is the exponential of a quadratic function of w. Thus, by completing the square
and making use of the standard result for the normalization coefﬁcient of a Gaussian

distribution, we can obtain a closed form solution which takes the form Exercise 10.35

502 10. APPROXIMATE INFERENCE
0.010.250.750.99
−4 −2 0 2 4−6−4−20246
Figure 10.13 Illustration of the Bayesian approach to logistic regression for a simple linearly separable data
set. The plot on the left shows the predictive distribution obtained using variational inference. We see that
the decision boundary lies roughly mid way between the clusters of data points, and that the contours of the
predictive distribution splay out away from the data reﬂecting the greater uncertainty in the classiﬁcation of such
regions. The plot on the right shows the decision boundaries corresponding to ﬁve samples of the parameter
vector wdrawn from the posterior distribution p(w|t).
L(ξ)=1
2ln|SN|
|S0|−1
2mT
NS−1
NmN+1
2mT
0S−1
0m0
+N∑
n=1{
lnσ(ξn)−1
2ξn−λ(ξn)ξ2
n}
. (10.164)
This variational framework can also be applied to situations in which the data
is arriving sequentially (Jaakkola and Jordan, 2000). In this case we maintain a

L(ξ)=1
2ln|SN|
|S0|−1
2mT
NS−1
NmN+1
2mT
0S−1
0m0
+N∑
n=1{
lnσ(ξn)−1
n}
. (10.164)
Gaussian posterior distribution over w, which is initialized using the prior p(w).A s
each data point arrives, the posterior is updated by making use of the bound (10.151)
and then normalized to give an updated posterior distribution.
The predictive distribution is obtained by marginalizing over the posterior dis-
tribution, and takes the same form as for the Laplace approximation discussed in
Section 4.5.2. Figure 10.13 shows the variational predictive distributions for a syn-
thetic data set. This example provides interesting insights into the concept of ‘large

margin’, which was discussed in Section 7.1 and which has qualitatively similar be-
haviour to the Bayesian solution.
10.6.3 Inference of hyperparameters
So far, we have treated the hyperparameter αin the prior distribution as a known
constant. We now extend the Bayesian logistic regression model to allow the value of
this parameter to be inferred from the data set. This can be achieved by combining
the global and local variational approximations into a single framework, so as to

maintain a lower bound on the marginal likelihood at each stage. Such a combined
approach was adopted by Bishop and Svens ´en (2003) in the context of a Bayesian
treatment of the hierarchical mixture of experts model.

10.6. Variational Logistic Regression 503
Speciﬁcally, we consider once again a simple isotropic Gaussian prior distribu-
tion of the form
p(w|α)=N(w|0,α−1I). (10.165)
Our analysis is readily extended to more general Gaussian priors, for instance if we
wish to associate a different hyperparameter with different subsets of the parame-
terswj. As usual, we consider a conjugate hyperprior over αgiven by a gamma
p(α)=G a m ( α|a0,b0) (10.166)
governed by the constants a0andb0.
The marginal likelihood for this model now takes the form
p(t)=∫∫
p(w,α,t)dwdα (10.167)
where the joint distribution is given by
p(w,α,t)=p(t|w)p(w|α)p(α). (10.168)
We are now faced with an analytically intractable integration over wandα, which
we shall tackle by using both the local and global variational approaches in the samemodel
To begin with, we introduce a variational distribution q(w,α), and then apply
the decomposition (10.2), which in this instance takes the form
lnp(t)=L(q) + KL( q∥p) (10.169)

p(t)=∫∫
where the lower bound L(q)and the Kullback-Leibler divergence KL(q∥p)are de-
ﬁned by
L(q)=∫∫
q(w,α)l n{p(w,α,t)
q(w,α)}
dwdα (10.170)
KL(q∥p)= −∫∫
q(w,α)l n{p(w,α|t))
q(w,α)}
dwdα. (10.171)
At this point, the lower bound L(q)is still intractable due to the form of the
likelihood factor p(t|w). We therefore apply the local variational bound to each of

ﬁned by
L(q)=∫∫
q(w,α)}
q(w,α)}
the logistic sigmoid factors as before. This allows us to use the inequality (10.152)and place a lower bound on L(q), which will therefore also be a lower bound on the
log marginal likelihood
lnp(t)⩾L(q)⩾˜L(q,ξ)
=∫∫
q(w,α)l n{h(w,ξ)p(w|α)p(α)
q(w,α)}
dwdα.(10.172)
Next we assume that the variational distribution factorizes between parameters and
hyperparameters so that
q(w,α)=q(w)q(α). (10.173)

504 10. APPROXIMATE INFERENCE
With this factorization we can appeal to the general result (10.9) to ﬁnd expressions
for the optimal factors. Consider ﬁrst the distribution q(w). Discarding terms that
are independent of w,w eh a v e
lnq(w)= Eα[ln{h(w,ξ)p(w|α)p(α)}]+c o n s t
=l n h(w,ξ)+ Eα[lnp(w|α) ]+c o n s t .
We now substitute for lnh(w,ξ)using (10.153), and for lnp(w|α)using (10.165),
giving
lnq(w)=−E[α]
2wTw+N∑
n=1{
(tn−1/2)wTφn−λ(ξn)wTφnφT
nw}
+c o n s t .
We see that this is a quadratic function of wand so the solution for q(w)will be
Gaussian. Completing the square in the usual way, we obtain
q(w)=N(w|µN,ΣN) (10.174)
Σ−1
NµN=N∑
n=1(tn−1/2)φn (10.175)
Σ−1
N= E[α]I+2N∑
n. (10.176)
Similarly, the optimal solution for the factor q(α)is obtained from
lnq(α)= Ew[lnp(w|α)] + ln p(α)+c o n s t .
Substituting for lnp(w|α)using (10.165), and for lnp(α)using (10.166), we obtain
lnq(α)=M
2lnα−α
2E[
wTw]
+(a0−1) lnα−b0α+c o n s t .

Σ−1
NµN=N∑
Σ−1
lnq(α)=M
2lnα−α
2E[
wTw]
We recognize this as the log of a gamma distribution, and so we obtain
q(α)=G a m ( α|aN,bN)=1
Γ(a0)ab0
0αa0−1e−b0α(10.177)
where
aN=a0+M
2(10.178)
bN=b0+1
2Ew[
wTw]
. (10.179)

10.7. Expectation Propagation 505
We also need to optimize the variational parameters ξn, and this is also done by
maximizing the lower bound ˜L(q,ξ). Omitting terms that are independent of ξ, and
integrating over α,w eh a v e
˜L(q,ξ)=∫
q(w)l nh(w,ξ)dw+c o n s t . (10.180)
Note that this has precisely the same form as (10.159), and so we can again appeal
to our earlier result (10.163), which can be obtained by direct optimization of the
marginal likelihood function, leading to re-estimation equations of the form
(ξnew
n)2=φT
n(
ΣN+µNµT
N)
φn. (10.181)
We have obtained re-estimation equations for the three quantities q(w),q(α),
andξ, and so after making suitable initializations, we can cycle through these quan-
tities, updating each in turn. The required moments are given by Appendix B
E[α]=aN
bN(10.182)
E[
wTw]
=ΣN+µT
NµN. (10.183)
10.7. Expectation Propagation
We conclude this chapter by discussing an alternative form of deterministic approx-

(ξnew
n)2=φT
n(
ΣN+µNµT
N)
E[α]=aN
E[
wTw]
=ΣN+µT
imate inference, known as expectation propagation orEP(Minka, 2001a; Minka,
2001b). As with the variational Bayes methods discussed so far, this too is based
on the minimization of a Kullback-Leibler divergence but now of the reverse form,
which gives the approximation rather different properties.
Consider for a moment the problem of minimizing KL(p∥q)with respect to q(z)
whenp(z)is a ﬁxed distribution and q(z)is a member of the exponential family and
so, from (2.194), can be written in the form
q(z)=h(z)g(η)e x p{
ηTu(z)}
. (10.184)

ηTu(z)}
. (10.184)
As a function of η, the Kullback-Leibler divergence then becomes
KL(p∥q)=−lng(η)−ηTEp(z)[u(z) ]+c o n s t (10.185)
where the constant terms are independent of the natural parameters η. We can mini-
mizeKL(p∥q)within this family of distributions by setting the gradient with respect
toηto zero, giving
−∇lng(η)= Ep(z)[u(z)]. (10.186)
However, we have already seen in (2.226) that the negative gradient of lng(η)is
given by the expectation of u(z)under the distribution q(z). Equating these two
results, we obtain
Eq(z)[u(z)] = Ep(z)[u(z)]. (10.187)

506 10. APPROXIMATE INFERENCE
We see that the optimum solution simply corresponds to matching the expected suf-
ﬁcient statistics. So, for instance, if q(z)is a Gaussian N(z|µ,Σ)then we minimize
the Kullback-Leibler divergence by setting the mean µofq(z)equal to the mean of
the distribution p(z)and the covariance Σequal to the covariance of p(z). This is
sometimes called moment matching . An example of this was seen in Figure 10.3(a).
Now let us exploit this result to obtain a practical algorithm for approximate
inference. For many probabilistic models, the joint distribution of data Dand hidden
variables (including parameters) θcomprises a product of factors in the form
p(D,θ)=∏
ifi(θ). (10.188)
This would arise, for example, in a model for independent, identically distributed
data in which there is one factor fn(θ)=p(xn|θ)for each data point xn, along
with a factor f0(θ)=p(θ)corresponding to the prior. More generally, it would also

p(D,θ)=∏
apply to any model deﬁned by a directed probabilistic graph in which each factor is a
conditional distribution corresponding to one of the nodes, or an undirected graph inwhich each factor is a clique potential. We are interested in evaluating the posterior
distribution p(θ|D)for the purpose of making predictions, as well as the model
evidence p(D)for the purpose of model comparison. From (10.188) the posterior is
given by
p(θ|D)=1
p(D)∏
ifi(θ) (10.189)
and the model evidence is given by
p(D)=∫∏

given by
p(θ|D)=1
p(D)∏
p(D)=∫∏
ifi(θ)dθ. (10.190)
Here we are considering continuous variables, but the following discussion applies
equally to discrete variables with integrals replaced by summations. We shall sup-pose that the marginalization over θ, along with the marginalizations with respect to
the posterior distribution required to make predictions, are intractable so that some
form of approximation is required.
Expectation propagation is based on an approximation to the posterior distribu-
tion which is also given by a product of factors

q(θ)=1
Z∏
i˜fi(θ) (10.191)
in which each factor ˜fi(θ)in the approximation corresponds to one of the factors
fi(θ)in the true posterior (10.189), and the factor 1/Zis the normalizing constant
needed to ensure that the left-hand side of (10.191) integrates to unity. In order to
obtain a practical algorithm, we need to constrain the factors ˜fi(θ)in some way,
and in particular we shall assume that they come from the exponential family. The

q(θ)=1
Z∏
product of the factors will therefore also be from the exponential family and so can

10.7. Expectation Propagation 507
be described by a ﬁnite set of sufﬁcient statistics. For example, if each of the ˜fi(θ)
is a Gaussian, then the overall approximation q(θ)will also be Gaussian.
Ideally we would like to determine the ˜fi(θ)by minimizing the Kullback-Leibler
divergence between the true posterior and the approximation given by
KL(p∥q)=K L(
1
p(D)∏
ifi(θ)1
Z∏
i˜fi(θ))
. (10.192)
Note that this is the reverse form of KL divergence compared with that used in varia-
tional inference. In general, this minimization will be intractable because the KL di-vergence involves averaging with respect to the true distribution. As a rough approx-
imation, we could instead minimize the KL divergences between the corresponding
pairsf
i(θ)and˜fi(θ)of factors. This represents a much simpler problem to solve,
and has the advantage that the algorithm is noniterative. However, because each fac-tor is individually approximated, the product of the factors could well give a poor

pairsf
Expectation propagation makes a much better approximation by optimizing each
factor in turn in the context of all of the remaining factors. It starts by initializing
the factors˜fi(θ), and then cycles through the factors reﬁning them one at a time.
This is similar in spirit to the update of factors in the variational Bayes framework
considered earlier. Suppose we wish to reﬁne factor ˜fj(θ). We ﬁrst remove this
factor from the product to give∏
i̸=j˜fi(θ). Conceptually, we will now determine a
revised form of the factor ˜fj(θ)by ensuring that the product
qnew(θ)∝˜fj(θ)∏
i̸=j˜fi(θ) (10.193)
is as close as possible to
fj(θ)∏
i̸=j˜fi(θ) (10.194)

fj(θ)∏
in which we keep ﬁxed all of the factors ˜fi(θ)fori̸=j. This ensures that the
approximation is most accurate in the regions of high posterior probability as deﬁned
by the remaining factors. We shall see an example of this effect when we apply EP
to the ‘clutter problem’. To achieve this, we ﬁrst remove the factor ˜fj(θ)from the Section 10.7.1
current approximation to the posterior by deﬁning the unnormalized distribution
q\j(θ)=q(θ)
˜fj(θ). (10.195)
Note that we could instead ﬁnd q\j(θ)from the product of factors i̸=j, although

in practice division is usually easier. This is now combined with the factor fj(θ)to
give a distribution
1
Zjfj(θ)q\j(θ) (10.196)

508 10. APPROXIMATE INFERENCE
Figure 10.14 Illustration of the expectation propagation approximation using a Gaussian distribution for the
example considered earlier in Figures 4.14 and 10.1. The left-hand plot shows the original distribution (yellow)
along with the Laplace (red), global variational (green), and EP (blue) approximations, and the right-hand plot
shows the corresponding negative logarithms of the distributions. Note that the EP distribution is broader than
that variational inference, as a consequence of the different form of KL divergence.
where Zjis the normalization constant given by
Zj=∫
fj(θ)q\j(θ)dθ. (10.197)
We now determine a revised factor ˜fj(θ)by minimizing the Kullback-Leibler diver-
gence
KL(fj(θ)q\j(θ)
Zjqnew(θ))
. (10.198)
This is easily solved because the approximating distribution qnew(θ)is from the ex-
ponential family, and so we can appeal to the result (10.187), which tells us that the

Zj=∫
gence
. (10.198)
parameters of qnew(θ)are obtained by matching its expected sufﬁcient statistics to
the corresponding moments of (10.196). We shall assume that this is a tractable oper-
ation. For example, if we choose q(θ)to be a Gaussian distribution N(θ|µ,Σ), then
µis set equal to the mean of the (unnormalized) distribution fj(θ)q\j(θ), andΣis
set to its covariance. More generally, it is straightforward to obtain the required ex-
pectations for any member of the exponential family, provided it can be normalized,

because the expected statistics can be related to the derivatives of the normalization
coefﬁcient, as given by (2.226). The EP approximation is illustrated in Figure 10.14.
From (10.193), we see that the revised factor ˜fj(θ)can be found by taking
qnew(θ)and dividing out the remaining factors so that
˜fj(θ)=Kqnew(θ)
q\j(θ)(10.199)
where we have used (10.195). The coefﬁcient Kis determined by multiplying both

10.7. Expectation Propagation 509
sides of (10.199) by q\i(θ)and integrating to give
K=∫
˜fj(θ)q\j(θ)dθ (10.200)
where we have used the fact that qnew(θ)is normalized. The value of Kcan therefore
be found by matching zeroth-order moments
∫
˜fj(θ)q\j(θ)dθ=∫
fj(θ)q\j(θ)dθ. (10.201)
Combining this with (10.197), we then see that K=Zjand so can be found by
evaluating the integral in (10.197).
In practice, several passes are made through the set of factors, revising each
factor in turn. The posterior distribution p(θ|D)is then approximated using (10.191),
and the model evidence p(D)can be approximated by using (10.190) with the factors
fi(θ)replaced by their approximations ˜fi(θ).
Expectation Propagation
We are given a joint distribution over observed data Dand stochastic variables
θin the form of a product of factors
p(D,θ)=∏
ifi(θ) (10.202)
and we wish to approximate the posterior distribution p(θ|D)by a distribution
of the form
q(θ)=1
Z∏
i˜fi(θ). (10.203)
We also wish to approximate the model evidence p(D).

p(D,θ)=∏
of the form
q(θ)=1
Z∏
1. Initialize all of the approximating factors ˜fi(θ).
2. Initialize the posterior approximation by setting
q(θ)∝∏
i˜fi(θ). (10.204)
3. Until convergence:
(a) Choose a factor ˜fj(θ)to reﬁne.
(b) Remove ˜fj(θ)from the posterior by division
˜fj(θ). (10.205)

510 10. APPROXIMATE INFERENCE
(c) Evaluate the new posterior by setting the sufﬁcient statistics (moments)
ofqnew(θ)equal to those of q\j(θ)fj(θ), including evaluation of the
normalization constant
Zj=∫
q\j(θ)fj(θ)dθ. (10.206)
(d) Evaluate and store the new factor
˜fj(θ)=Zjqnew(θ)
q\j(θ). (10.207)
4. Evaluate the approximation to the model evidence
p(D)≃∫∏
i˜fi(θ)dθ. (10.208)
A special case of EP, known as assumed density ﬁltering (ADF) or moment
matching (Maybeck, 1982; Lauritzen, 1992; Boyen and Koller, 1998; Opper and
Winther, 1999), is obtained by initializing all of the approximating factors except
the ﬁrst to unity and then making one pass through the factors updating each of them
once. Assumed density ﬁltering can be appropriate for on-line learning in which datapoints are arriving in a sequence and we need to learn from each data point and then
discard it before considering the next point. However, in a batch setting we have the

opportunity to re-use the data points many times in order to achieve improved ac-curacy, and it is this idea that is exploited in expectation propagation. Furthermore,
if we apply ADF to batch data, the results will have an undesirable dependence on
the (arbitrary) order in which the data points are considered, which again EP canovercome.
One disadvantage of expectation propagation is that there is no guarantee that
the iterations will converge. However, for approximations q(θ)in the exponential

family, if the iterations do converge, the resulting solution will be a stationary point
of a particular energy function (Minka, 2001a), although each iteration of EP does
not necessarily decrease the value of this energy function. This is in contrast tovariational Bayes, which iteratively maximizes a lower bound on the log marginal
likelihood, in which each iteration is guaranteed not to decrease the bound. It is

possible to optimize the EP cost function directly, in which case it is guaranteedto converge, although the resulting algorithms can be slower and more complex to
implement.
Another difference between variational Bayes and EP arises from the form of
KL divergence that is minimized by the two algorithms, because the former mini-
mizes KL(q∥p)whereas the latter minimizes KL(p∥q). As we saw in Figure 10.3,
for distributions p(θ)which are multimodal, minimizing KL(p∥q)can lead to poor

approximations. In particular, if EP is applied to mixtures the results are not sen-
sible because the approximation tries to capture all of the modes of the posteriordistribution. Conversely, in logistic-type models, EP often out-performs both local
variational methods and the Laplace approximation (Kuss and Rasmussen, 2006).

10.7. Expectation Propagation 511
Figure 10.15 Illustration of the clutter problem
for a data space dimensionality of
D=1. Training data points, de-
noted by the crosses, are drawn
from a mixture of two Gaussians
with components shown in red
and green. The goal is to infer the
mean of the green Gaussian from
θ x −5 0 5 10
10.7.1 Example: The clutter problem
Following Minka (2001b), we illustrate the EP algorithm using a simple exam-
ple in which the goal is to infer the mean θof a multivariate Gaussian distribution
over a variable xgiven a set of observations drawn from that distribution. To make
the problem more interesting, the observations are embedded in background clutter,
which itself is also Gaussian distributed, as illustrated in Figure 10.15. The distribu-
tion of observed values xis therefore a mixture of Gaussians, which we take to be
of the form
p(x|θ)=( 1 −w)N(x|θ,I)+wN(x|0,aI) (10.209)
where wis the proportion of background clutter and is assumed to be known. The

of the form
prior over θis taken to be Gaussian
p(θ)=N(θ|0,bI) (10.210)
and Minka (2001a) chooses the parameter values a=1 0 ,b= 100 andw=0.5.
The joint distribution of Nobservations D={x1,...,xN}andθis given by
p(D,θ)=p(θ)N∏
n=1p(xn|θ) (10.211)
and so the posterior distribution comprises a mixture of 2NGaussians. Thus the
computational cost of solving this problem exactly would grow exponentially with
the size of the data set, and so an exact solution is intractable for moderately large
N.

N.
To apply EP to the clutter problem, we ﬁrst identify the factors f0(θ)=p(θ)
andfn(θ)=p(xn|θ). Next we select an approximating distribution from the expo-
nential family, and for this example it is convenient to choose a spherical Gaussian
q(θ)=N(θ|m,vI). (10.212)

512 10. APPROXIMATE INFERENCE
The factor approximations will therefore take the form of exponential-quadratic
functions of the form
˜fn(θ)=snN(θ|mn,vnI) (10.213)
where n=1,...,N , and we set ˜f0(θ)equal to the prior p(θ). Note that the use of
N(θ|·,·)does not imply that the right-hand side is a well-deﬁned Gaussian density
(in fact, as we shall see, the variance parameter vncan be negative) but is simply a
convenient shorthand notation. The approximations ˜fn(θ), forn=1,...,N , can
be initialized to unity, corresponding to sn=( 2πvn)D/2,vn→∞ andmn=0,
where Dis the dimensionality of xand hence of θ. The initial q(θ), deﬁned by
(10.191), is therefore equal to the prior.
We then iteratively reﬁne the factors by taking one factor fn(θ)at a time and
applying (10.205), (10.206), and (10.207). Note that we do not need to revise the
termf0(θ)because an EP update will leave this term unchanged. Here we state the Exercise 10.37
results and leave the reader to ﬁll in the details.

First we remove the current estimate ˜fn(θ)fromq(θ)by division using (10.205)
to give q\n(θ), which has mean and inverse variance given by Exercise 10.38
m\n=m+v\nv−1
n(m−mn) (10.214)
(v\n)−1=v−1−v−1
n. (10.215)
Next we evaluate the normalization constant Znusing (10.206) to give
Zn=( 1−w)N(xn|m\n,(v\n+1 )I)+wN(xn|0,aI). (10.216)
Similarly, we compute the mean and variance of qnew(θ)by ﬁnding the mean and
variance of q\n(θ)fn(θ)to give Exercise 10.39
m=m\n+ρnv\n
v\n+1(xn−m\n) (10.217)
v=v\n−ρn(v\n)2

v\n+1+ρn(1−ρn)(v\n)2∥xn−m\n∥2
D(v\n+1 )2(10.218)
where the quantity
ρn=1−w
ZnN(xn|0,aI) (10.219)
has a simple interpretation as the probability of the point xnnot being clutter. Then
we use (10.207) to compute the reﬁned factor ˜fn(θ)whose parameters are given by
v−1
n=(vnew)−1−(v\n)−1(10.220)
mn=m\n+(vn+v\n)(v\n)−1(mnew−m\n) (10.221)
sn=Zn
(2πvn)D/2N(mn|m\n,(vn+v\n)I). (10.222)
This reﬁnement process is repeated until a suitable termination criterion is satisﬁed,

ρn=1−w
v−1
sn=Zn
for instance that the maximum change in parameter values resulting from a complete

10.7. Expectation Propagation 513
θ −5 0 5 10 θ −5 0 5 10
Figure 10.16 Examples of the approximation of speciﬁc factors for a one-dimensional version of the clutter
problem, showing fn(θ)in blue, efn(θ)in red, and q\n(θ)in green. Notice that the current form for q\n(θ)controls
the range of θover which efn(θ)will be a good approximation to fn(θ).
pass through all factors is less than some threshold. Finally, we use (10.208) to
evaluate the approximation to the model evidence, given by
p(D)≃(2πvnew)D/2exp(B/2)N∏
n=1{
sn(2πvn)−D/2}
(10.223)
where
B=(mnew)Tmnew
v−N∑
n=1mT
nmn
vn. (10.224)
Examples factor approximations for the clutter problem with a one-dimensional pa-
rameter space θare shown in Figure 10.16. Note that the factor approximations can
have inﬁnite or even negative values for the ‘variance’ parameter vn. This simply
corresponds to approximations that curve upwards instead of downwards and are not
necessarily problematic provided the overall approximate posterior q(θ)has posi-

n=1{
(10.223)
where
v−N∑
n=1mT
nmn
tive variance. Figure 10.17 compares the performance of EP with variational Bayes
(mean ﬁeld theory) and the Laplace approximation on the clutter problem.
10.7.2 Expectation propagation on graphs
So far in our general discussion of EP, we have allowed the factors fi(θ)in the
distribution p(θ)to be functions of all of the components of θ, and similarly for the
approximating factors ˜f(θ)in the approximating distribution q(θ). We now consider

situations in which the factors depend only on subsets of the variables. Such restric-
tions can be conveniently expressed using the framework of probabilistic graphical
models, as discussed in Chapter 8. Here we use a factor graph representation because
this encompasses both directed and undirected graphs.

514 10. APPROXIMATE INFERENCE
epvb laplacePosterior mean
FLOPSError
10410610−5100
epvb
laplaceEvidence
10410610−20410−20210−200
Figure 10.17 Comparison of expectation propagation, variational inference, and the Laplace approximation on
the clutter problem. The left-hand plot shows the error in the predicted posterior mean versus the number of
ﬂoating point operations, and the right-hand plot shows the corresponding results for the model evidence.
We shall focus on the case in which the approximating distribution is fully fac-
torized, and we shall show that in this case expectation propagation reduces to loopy
belief propagation (Minka, 2001a). To start with, we show this in the context of a
simple example, and then we shall explore the general case.
First of all, recall from (10.17) that if we minimize the Kullback-Leibler diver-
gence KL(p∥q)with respect to a factorized distribution q, then the optimal solution
for each factor is simply the corresponding marginal of p.

Now consider the factor graph shown on the left in Figure 10.18, which was
introduced earlier in the context of the sum-product algorithm. The joint distribution Section 8.4.4
is given by
p(x)=fa(x1,x2)fb(x2,x3)fc(x2,x4). (10.225)
We seek an approximation q(x)that has the same factorization, so that
q(x)∝˜fa(x1,x2)˜fb(x2,x3)˜fc(x2,x4). (10.226)
Note that normalization constants have been omitted, and these can be re-instated at
the end by local normalization, as is generally done in belief propagation. Now sup-

is given by
pose we restrict attention to approximations in which the factors themselves factorize
with respect to the individual variables so that
q(x)∝˜fa1(x1)˜fa2(x2)˜fb2(x2)˜fb3(x3)˜fc2(x2)˜fc4(x4) (10.227)
which corresponds to the factor graph shown on the right in Figure 10.18. Because
the individual factors are factorized, the overall distribution q(x)is itself fully fac-
torized.
Now we apply the EP algorithm using the fully factorized approximation. Sup-
pose that we have initialized all of the factors and that we choose to reﬁne factor

10.7. Expectation Propagation 515
x1 x2 x3
x4fa fb
fcx1 x2 x3
x4˜fa1˜fa2˜fb2˜fb3
˜fc2
˜fc4
Figure 10.18 On the left is a simple factor graph from Figure 8.51 and reproduced here for convenience. On
the right is the corresponding factorized approximation.
˜fb(x2,x3)=˜fb2(x2)˜fb3(x3). We ﬁrst remove this factor from the approximating
distribution to give
q\b(x)=˜fa1(x1)˜fa2(x2)˜fc2(x2)˜fc4(x4) (10.228)
and we then multiply this by the exact factor fb(x2,x3)to give
ˆp(x)=q\b(x)fb(x2,x3)=˜fa1(x1)˜fa2(x2)˜fc2(x2)˜fc4(x4)fb(x2,x3).(10.229)
We now ﬁnd qnew(x)by minimizing the Kullback-Leibler divergence KL(ˆp∥qnew).
The result, as noted above, is that qnew(z)comprises the product of factors, one for
each variable xi, in which each factor is given by the corresponding marginal of
ˆp(x). These four marginals are given by
ˆp(x1)∝˜fa1(x1) (10.230)
ˆp(x2)∝˜fa2(x2)˜fc2(x2)∑
x3fb(x2,x3) (10.231)
ˆp(x3)∝∑
x2{
fb(x2,x3)˜fa2(x2)˜fc2(x2)}
(10.232)
ˆp(x4)∝˜fc4(x4) (10.233)

ˆp(x3)∝∑
x2{
(10.232)
andqnew(x)is obtained by multiplying these marginals together. We see that the
only factors in q(x)that change when we update ˜fb(x2,x3)are those that involve
the variables in fbnamely x2andx3. To obtain the reﬁned factor ˜fb(x2,x3)=
˜fb2(x2)˜fb3(x3)we simply divide qnew(x)byq\b(x), which gives
˜fb2(x2)∝∑
x3fb(x2,x3) (10.234)
˜fb3(x3)∝∑
x2{
. (10.235)

516 10. APPROXIMATE INFERENCE
These are precisely the messages obtained using belief propagation in which mes- Section 8.4.4
sages from variable nodes to factor nodes have been folded into the messages from
factor nodes to variable nodes. In particular, ˜fb2(x2)corresponds to the message
µfb→x2(x2)sent by factor node fbto variable node x2and is given by (8.81). Simi-
larly, if we substitute (8.78) into (8.79), we obtain (10.235) in which ˜fa2(x2)corre-
sponds to µfa→x2(x2)and˜fc2(x2)corresponds to µfc→x2(x2), giving the message
˜fb3(x3)which corresponds to µfb→x3(x3).
This result differs slightly from standard belief propagation in that messages are
passed in both directions at the same time. We can easily modify the EP procedureto give the standard form of the sum-product algorithm by updating just one of the
factors at a time, for instance if we reﬁne only˜fb3(x3), then˜fb2(x2)is unchanged
by deﬁnition, while the reﬁned version of ˜fb3(x3)is again given by (10.235). If

we are reﬁning only one term at a time, then we can choose the order in which the
reﬁnements are done as we wish. In particular, for a tree-structured graph we canfollow a two-pass update scheme, corresponding to the standard belief propagation
schedule, which will result in exact inference of the variable and factor marginals.
The initialization of the approximation factors in this case is unimportant.
Now let us consider a general factor graph corresponding to the distribution
p(θ)=∏
ifi(θi) (10.236)

p(θ)=∏
where θirepresents the subset of variables associated with factor fi. We approximate
this using a fully factorized distribution of the form
q(θ)∝∏
i∏
k˜fik(θk) (10.237)
where θkcorresponds to an individual variable node. Suppose that we wish to reﬁne
the particular term ˜fjl(θl)keeping all other terms ﬁxed. We ﬁrst remove the term
˜fj(θj)fromq(θ)to give
q\j(θ)∝∏
i̸=j∏
k˜fik(θk) (10.238)
and then multiply by the exact factor fj(θj). To determine the reﬁned term ˜fjl(θl),

p(θ)=∏
q(θ)∝∏
i∏
q\j(θ)∝∏
i̸=j∏
we need only consider the functional dependence on θl, and so we simply ﬁnd the
corresponding marginal of
q\j(θ)fj(θj). (10.239)
Up to a multiplicative constant, this involves taking the marginal of fj(θj)multiplied
by any terms from q\j(θ)that are functions of any of the variables in θj. Terms that
correspond to other factors ˜fi(θi)fori̸=jwill cancel between numerator and
denominator when we subsequently divide by q\j(θ). We therefore obtain
˜fjl(θl)∝∑
θm̸=l∈θjfj(θj)∏
k∏
m̸=l˜fkm(θm). (10.240)

Exercises 517
We recognize this as the sum-product rule in the form in which messages from vari-
able nodes to factor nodes have been eliminated, as illustrated by the example shown
in Figure 8.50. The quantity ˜fjm(θm)corresponds to the message µfj→θm(θm),
which factor node jsends to variable node m, and the product over kin (10.240)
is over all factors that depend on the variables θmthat have variables (other than
variable θl) in common with factor fj(θj). In other words, to compute the outgoing
message from a factor node, we take the product of all the incoming messages from
other factor nodes, multiply by the local factor, and then marginalize.
Thus, the sum-product algorithm arises as a special case of expectation propa-
gation if we use an approximating distribution that is fully factorized. This suggests
that more ﬂexible approximating distributions, corresponding to partially discon-
nected graphs, could be used to achieve higher accuracy. Another generalization is

to group factors fi(θi)together into sets and to reﬁne all the factors in a set together
at each iteration. Both of these approaches can lead to improvements in accuracy(Minka, 2001b). In general, the problem of choosing the best combination of group-
ing and disconnection is an open research issue.
We have seen that variational message passing and expectation propagation op-
timize two different forms of the Kullback-Leibler divergence. Minka (2005) has

shown that a broad range of message passing algorithms can be derived from a com-
mon framework involving minimization of members of the alpha family of diver-gences, given by (10.19). These include variational message passing, loopy belief
propagation, and expectation propagation, as well as a range of other algorithms,
which we do not have space to discuss here, such as tree-reweighted message pass-
ing(Wainwright et al. , 2005), fractional belief propagation (Wiegerinck and Heskes,
2003), and power EP (Minka, 2004).
Exercises

Exercises
10.1 (⋆)www Verify that the log marginal distribution of the observed data lnp(X)
can be decomposed into two terms in the form (10.2) where L(q)is given by (10.3)
andKL(q∥p)is given by (10.4).
10.2 (⋆)Use the properties E[z1]=m1and E[z2]=m2to solve the simultaneous equa-
tions (10.13) and (10.15), and hence show that, provided the original distributionp(z)is nonsingular, the unique solution for the means of the factors in the approxi-
mation distribution is given by E[z
1]=µ1and E[z2]=µ2.

Exercises
10.3 (⋆⋆)www Consider a factorized variational distribution q(Z)of the form (10.5).
By using the technique of Lagrange multipliers, verify that minimization of theKullback-Leibler divergence KL(p∥q)with respect to one of the factors q
i(Zi),
keeping all other factors ﬁxed, leads to the solution (10.17).
10.4 (⋆⋆)Suppose that p(x)is some ﬁxed distribution and that we wish to approximate
it using a Gaussian distribution q(x)=N(x|µ,Σ). By writing down the form of

i(Zi),
the KL divergence KL(p∥q)for a Gaussian q(x)and then differentiating, show that

518 10. APPROXIMATE INFERENCE
minimization of KL(p∥q)with respect to µandΣleads to the result that µis given
by the expectation of xunder p(x)and that Σis given by the covariance.
10.5 (⋆⋆)www Consider a model in which the set of all hidden stochastic variables, de-
noted collectively by Z, comprises some latent variables ztogether with some model
parameters θ. Suppose we use a variational distribution that factorizes between la-
tent variables and parameters so that q(z,θ)=qz(z)qθ(θ), in which the distribution
qθ(θ)is approximated by a point estimate of the form qθ(θ)=δ(θ−θ0)where θ0
is a vector of free parameters. Show that variational optimization of this factorized
distribution is equivalent to an EM algorithm, in which the E step optimizes qz(z),
and the M step maximizes the expected complete-data log posterior distribution of θ
with respect to θ0.
10.6 (⋆⋆)The alpha family of divergences is deﬁned by (10.19). Show that the Kullback-

Leibler divergence KL(p∥q)corresponds to α→1. This can be done by writing
pϵ= exp( ϵlnp)=1+ ϵlnp+O(ϵ2)and then taking ϵ→0. Similarly show that
KL(q∥p)corresponds to α→−1.
10.7 (⋆⋆)Consider the problem of inferring the mean and precision of a univariate Gaus-
sian using a factorized variational approximation, as considered in Section 10.1.3.Show that the factor q
µ(µ)is a Gaussian of the form N(µ|µN,λ−1
N)with mean and
precision given by (10.26) and (10.27), respectively. Similarly show that the factor
qτ(τ)is a gamma distribution of the form Gam( τ|aN,bN)with parameters given by

(10.29) and (10.30).
10.8 (⋆)Consider the variational posterior distribution for the precision of a univariate
Gaussian whose parameters are given by (10.29) and (10.30). By using the standard
results for the mean and variance of the gamma distribution given by (B.27) and(B.28), show that if we let N→∞ , this variational posterior distribution has a
mean given by the inverse of the maximum likelihood estimator for the variance of
the data, and a variance that goes to zero.

10.9 (⋆⋆)By making use of the standard result E[τ]=a
N/bNfor the mean of a gamma
distribution, together with (10.26), (10.27), (10.29), and (10.30), derive the result
(10.33) for the reciprocal of the expected precision in the factorized variational treat-
ment of a univariate Gaussian.
10.10 (⋆)www Derive the decomposition given by (10.34) that is used to ﬁnd approxi-
mate posterior distributions over models using variational inference.
10.11 (⋆⋆)www By using a Lagrange multiplier to enforce the normalization constraint

on the distribution q(m), show that the maximum of the lower bound (10.35) is given
by (10.36).
10.12 (⋆⋆)Starting from the joint distribution (10.41), and applying the general result
(10.9), show that the optimal variational distribution q⋆(Z)over the latent variables
for the Bayesian mixture of Gaussians is given by (10.48) by verifying the steps
given in the text.

Exercises 519
10.13 (⋆⋆)www Starting from (10.54), derive the result (10.59) for the optimum vari-
ational posterior distribution over µkandΛkin the Bayesian mixture of Gaussians,
and hence verify the expressions for the parameters of this distribution given by
(10.60)–(10.63).
10.14 (⋆⋆)Using the distribution (10.59), verify the result (10.64).
10.15 (⋆)Using the result (B.17), show that the expected value of the mixing coefﬁcients
in the variational mixture of Gaussians is given by (10.69).
10.16 (⋆⋆)www Verify the results (10.71) and (10.72) for the ﬁrst two terms in the
lower bound for the variational Gaussian mixture model given by (10.70).
10.17 (⋆⋆⋆ )Verify the results (10.73)–(10.77) for the remaining terms in the lower bound
for the variational Gaussian mixture model given by (10.70).
10.18 (⋆⋆⋆ )In this exercise, we shall derive the variational re-estimation equations for
the Gaussian mixture model by direct differentiation of the lower bound. To do this

we assume that the variational distribution has the factorization deﬁned by (10.42)and (10.55) with factors given by (10.48), (10.57), and (10.59). Substitute these into
(10.70) and hence obtain the lower bound as a function of the parameters of the varia-
tional distribution. Then, by maximizing the bound with respect to these parameters,derive the re-estimation equations for the factors in the variational distribution, and
show that these are the same as those obtained in Section 10.2.1.

10.19 (⋆⋆)Derive the result (10.81) for the predictive distribution in the variational treat-
ment of the Bayesian mixture of Gaussians model.
10.20 (⋆⋆)
www This exercise explores the variational Bayes solution for the mixture of
Gaussians model when the size Nof the data set is large and shows that it reduces (as
we would expect) to the maximum likelihood solution based on EM derived in Chap-
ter 9. Note that results from Appendix B may be used to help answer this exercise.First show that the posterior distribution q

10.20 (⋆⋆)
⋆(Λk)of the precisions becomes sharply
peaked around the maximum likelihood solution. Do the same for the posterior dis-
tribution of the means q⋆(µk|Λk). Next consider the posterior distribution q⋆(π)
for the mixing coefﬁcients and show that this too becomes sharply peaked around
the maximum likelihood solution. Similarly, show that the responsibilities become
equal to the corresponding maximum likelihood values for large N, by making use
of the following asymptotic result for the digamma function for large x
ψ(x)=l n x+O(1/x). (10.241)

Finally, by making use of (10.80), show that for large N, the predictive distribution
becomes a mixture of Gaussians.
10.21 (⋆)Show that the number of equivalent parameter settings due to interchange sym-
metries in a mixture model with Kcomponents is K!.

520 10. APPROXIMATE INFERENCE
10.22 (⋆⋆)We have seen that each mode of the posterior distribution in a Gaussian mix-
ture model is a member of a family of K!equivalent modes. Suppose that the result
of running the variational inference algorithm is an approximate posterior distribu-
tionqthat is localized in the neighbourhood of one of the modes. We can then
approximate the full posterior distribution as a mixture of K!suchqdistributions,
once centred on each mode and having equal mixing coefﬁcients. Show that if we
assume negligible overlap between the components of the qmixture, the resulting
lower bound differs from that for a single component qdistribution through the ad-
dition of an extra term lnK!.
10.23 (⋆⋆)www Consider a variational Gaussian mixture model in which there is no
prior distribution over mixing coefﬁcients {πk}. Instead, the mixing coefﬁcients are

treated as parameters, whose values are to be found by maximizing the variationallower bound on the log marginal likelihood. Show that maximizing this lower bound
with respect to the mixing coefﬁcients, using a Lagrange multiplier to enforce the
constraint that the mixing coefﬁcients sum to one, leads to the re-estimation result
(10.83). Note that there is no need to consider all of the terms in the lower bound but
only the dependence of the bound on the {π
k}.
10.24 (⋆⋆)www We have seen in Section 10.2 that the singularities arising in the max-

k}.
imum likelihood treatment of Gaussian mixture models do not arise in a Bayesiantreatment. Discuss whether such singularities would arise if the Bayesian model
were solved using maximum posterior (MAP) estimation.
10.25 (⋆⋆)The variational treatment of the Bayesian mixture of Gaussians, discussed in
Section 10.2, made use of a factorized approximation (10.5) to the posterior distribu-tion. As we saw in Figure 10.2, the factorized assumption causes the variance of the
posterior distribution to be under-estimated for certain directions in parameter space.

Discuss qualitatively the effect this will have on the variational approximation to the
model evidence, and how this effect will vary with the number of components in
the mixture. Hence explain whether the variational Gaussian mixture will tend tounder-estimate or over-estimate the optimal number of components.
10.26 (⋆⋆⋆ )Extend the variational treatment of Bayesian linear regression to include
a gamma hyperprior Gam( β|c
0,d0)overβand solve variationally, by assuming a
factorized variational distribution of the form q(w)q(α)q(β). Derive the variational

update equations for the three factors in the variational distribution and also obtain
an expression for the lower bound and for the predictive distribution.
10.27 (⋆⋆)By making use of the formulae given in Appendix B show that the variational
lower bound for the linear basis function regression model, deﬁned by (10.107), can
be written in the form (10.107) with the various terms deﬁned by (10.108)–(10.112).
10.28 (⋆⋆⋆ )Rewrite the model for the Bayesian mixture of Gaussians, introduced in

Section 10.2, as a conjugate model from the exponential family, as discussed in
Section 10.4. Hence use the general results (10.115) and (10.119) to derive the
speciﬁc results (10.48), (10.57), and (10.59).

Exercises 521
10.29 (⋆)www Show that the function f(x)=l n ( x)is concave for 0<x< ∞
by computing its second derivative. Determine the form of the dual function g(λ)
deﬁned by (10.133), and verify that minimization of λx−g(λ)with respect to λ
according to (10.132) indeed recovers the function ln(x).
10.30 (⋆)By evaluating the second derivative, show that the log logistic function f(x)=
−ln(1 + e−x)is concave. Derive the variational upper bound (10.137) directly by
making a second order Taylor expansion of the log logistic function around a point
x=ξ.
10.31 (⋆⋆)By ﬁnding the second derivative with respect to x, show that the function
f(x)=−ln(ex/2+e−x/2)is a concave function of x. Now consider the second
derivatives with respect to the variable x2and hence show that it is a convex function
ofx2. Plot graphs of f(x)against xand against x2. Derive the lower bound (10.144)
on the logistic sigmoid function directly by making a ﬁrst order Taylor series expan-sion of the function f(x)in the variable x

x=ξ.
2centred on the value ξ2.
10.32 (⋆⋆)www Consider the variational treatment of logistic regression with sequen-
tial learning in which data points are arriving one at a time and each must be pro-
cessed and discarded before the next data point arrives. Show that a Gaussian ap-proximation to the posterior distribution can be maintained through the use of the
lower bound (10.151), in which the distribution is initialized using the prior, and as
each data point is absorbed its corresponding variational parameter ξ
nis optimized.

10.33 (⋆)By differentiating the quantity Q(ξ,ξold)deﬁned by (10.161) with respect to
the variational parameter ξnshow that the update equation for ξnfor the Bayesian
logistic regression model is given by (10.163).
10.34 (⋆⋆)In this exercise we derive re-estimation equations for the variational parame-
tersξin the Bayesian logistic regression model of Section 4.5 by direct maximization
of the lower bound given by (10.164). To do this set the derivative of L(ξ)with re-

spect to ξnequal to zero, making use of the result (3.117) for the derivative of the log
of a determinant, together with the expressions (10.157) and (10.158) which deﬁnethe mean and covariance of the variational posterior distribution q(w).
10.35 (⋆⋆)Derive the result (10.164) for the lower bound L(ξ)in the variational logistic
regression model. This is most easily done by substituting the expressions for the
Gaussian prior q(w)=N(w|m
0,S0), together with the lower bound h(w,ξ)on

the likelihood function, into the integral (10.159) which deﬁnes L(ξ). Next gather
together the terms which depend on win the exponential and complete the square
to give a Gaussian integral, which can then be evaluated by invoking the standardresult for the normalization coefﬁcient of a multivariate Gaussian. Finally take the
logarithm to obtain (10.164).
10.36 (⋆⋆)Consider the ADF approximation scheme discussed in Section 10.7, and show
that inclusion of the factor f
j(θ)leads to an update of the model evidence of the
form

form
pj(D)≃pj−1(D)Zj (10.242)

522 10. APPROXIMATE INFERENCE
where Zjis the normalization constant deﬁned by (10.197). By applying this result
recursively, and initializing with p0(D)=1 , derive the result
p(D)≃∏
jZj. (10.243)
10.37 (⋆)www Consider the expectation propagation algorithm from Section 10.7, and
suppose that one of the factors f0(θ)in the deﬁnition (10.188) has the same expo-
nential family functional form as the approximating distribution q(θ). Show that if
the factor˜f0(θ)is initialized to be f0(θ), then an EP update to reﬁne ˜f0(θ)leaves
˜f0(θ)unchanged. This situation typically arises when one of the factors is the prior
p(θ), and so we see that the prior factor can be incorporated once exactly and does
not need to be reﬁned.
10.38 (⋆⋆⋆ )In this exercise and the next, we shall verify the results (10.214)–(10.224)
for the expectation propagation algorithm applied to the clutter problem. Begin byusing the division formula (10.205) to derive the expressions (10.214) and (10.215)

by completing the square inside the exponential to identify the mean and variance.
Also, show that the normalization constant Z
n, deﬁned by (10.206), is given for the
clutter problem by (10.216). This can be done by making use of the general result
(2.115).
10.39 (⋆⋆⋆ )Show that the mean and variance of qnew(θ)for EP applied to the clutter
problem are given by (10.217) and (10.218). To do this, ﬁrst prove the followingresults for the expectations of θandθθ
Tunder qnew(θ)
E[θ]= m\n+v\n∇m\nlnZn (10.244)
E[θTθ]=2 ( v\n)2∇v\nlnZn+2 E[θ]Tm\n−∥m\n∥2(10.245)

(2.115).
and then make use of the result (10.216) for Zn. Next, prove the results (10.220)–
(10.222) by using (10.207) and completing the square in the exponential. Finally,
use (10.208) to derive the result (10.223).

11
Sampling
Methods
For most probabilistic models of practical interest, exact inference is intractable, and
so we have to resort to some form of approximation. In Chapter 10, we discussedinference algorithms based on deterministic approximations, which include methods
such as variational Bayes and expectation propagation. Here we consider approxi-
mate inference methods based on numerical sampling, also known as Monte Carlo
Although for some applications the posterior distribution over unobserved vari-
ables will be of direct interest in itself, for most situations the posterior distributionis required primarily for the purpose of evaluating expectations, for example in order
to make predictions. The fundamental problem that we therefore wish to address in
this chapter involves ﬁnding the expectation of some function f(z)with respect to a
probability distribution p(z). Here, the components of zmight comprise discrete or

continuous variables or some combination of the two. Thus in the case of continuous
523

524 11. SAMPLING METHODS
Figure 11.1 Schematic illustration of a function f(z)
whose expectation is to be evaluated with
respect to a distribution p(z).p(z)f(z)
z
variables, we wish to evaluate the expectation
E[f]=∫
f(z)p(z)dz (11.1)
where the integral is replaced by summation in the case of discrete variables. This
is illustrated schematically for a single continuous variable in Figure 11.1. We shall
suppose that such expectations are too complex to be evaluated exactly using analyt-
ical techniques.
The general idea behind sampling methods is to obtain a set of samples z(l)
(where l=1,...,L ) drawn independently from the distribution p(z). This allows
the expectation (11.1) to be approximated by a ﬁnite sum
ˆf=1
LL∑
l=1f(z(l)). (11.2)
As long as the samples z(l)are drawn from the distribution p(z), then E[ˆf]= E[f]
and so the estimator ˆfhas the correct mean. The variance of the estimator is given
by Exercise 11.1
var[ˆf]=1
LE[
(f−E[f])2]
(11.3)

ˆf=1
LL∑
var[ˆf]=1
LE[
(11.3)
is the variance of the function f(z)under the distribution p(z). It is worth emphasiz-
ing that the accuracy of the estimator therefore does not depend on the dimension-
ality of z, and that, in principle, high accuracy may be achievable with a relatively
small number of samples z(l). In practice, ten or twenty independent samples may
sufﬁce to estimate an expectation to sufﬁcient accuracy.
The problem, however, is that the samples {z(l)}might not be independent, and

var[ˆf]=1
LE[
(11.3)
so the effective sample size might be much smaller than the apparent sample size.
Also, referring back to Figure 11.1, we note that if f(z)is small in regions where
p(z)is large, and vice versa, then the expectation may be dominated by regions
of small probability, implying that relatively large sample sizes will be required to
achieve sufﬁcient accuracy.
For many models, the joint distribution p(z)is conveniently speciﬁed in terms

of a graphical model. In the case of a directed graph with no observed variables, it is

11. SAMPLING METHODS 525
straightforward to sample from the joint distribution (assuming that it is possible to
sample from the conditional distributions at each node) using the following ances-
tral sampling approach, discussed brieﬂy in Section 8.1.2. The joint distribution is
speciﬁed by
p(z)=M∏
i=1p(zi|pai) (11.4)
whereziare the set of variables associated with node i, andpaidenotes the set of
variables associated with the parents of node i. To obtain a sample from the joint
distribution, we make one pass through the set of variables in the order z1,...,zM
sampling from the conditional distributions p(zi|pai). This is always possible be-
cause at each step all of the parent values will have been instantiated. After one pass
through the graph, we will have obtained a sample from the joint distribution.
Now consider the case of a directed graph in which some of the nodes are in-
stantiated with observed values. We can in principle extend the above procedure, at

least in the case of nodes representing discrete variables, to give the following logic
sampling approach (Henrion, 1988), which can be seen as a special case of impor-
tance sampling discussed in Section 11.1.4. At each step, when a sampled value is
obtained for a variable ziwhose value is observed, the sampled value is compared
to the observed value, and if they agree then the sample value is retained and the al-
gorithm proceeds to the next variable in turn. However, if the sampled value and the

observed value disagree, then the whole sample so far is discarded and the algorithmstarts again with the ﬁrst node in the graph. This algorithm samples correctly from
the posterior distribution because it corresponds simply to drawing samples from the
joint distribution of hidden variables and data variables and then discarding those
samples that disagree with the observed data (with the slight saving of not continu-

ing with the sampling from the joint distribution as soon as one contradictory value isobserved). However, the overall probability of accepting a sample from the posterior
decreases rapidly as the number of observed variables increases and as the number
of states that those variables can take increases, and so this approach is rarely usedin practice.
In the case of probability distributions deﬁned by an undirected graph, there is

no one-pass sampling strategy that will sample even from the prior distribution withno observed variables. Instead, computationally more expensive techniques must be
employed, such as Gibbs sampling, which is discussed in Section 11.3.
As well as sampling from conditional distributions, we may also require samples
from a marginal distribution. If we already have a strategy for sampling from a joint
distribution p(u,v), then it is straightforward to obtain samples from the marginal
distribution p(u)simply by ignoring the values for vin each sample.

There are numerous texts dealing with Monte Carlo methods. Those of partic-
ular interest from the statistical inference perspective include Chen et al. (2001),
Gamerman (1997), Gilks et al. (1996), Liu (2001), Neal (1996), and Robert and
Casella (1999). Also there are review articles by Besag et al. (1995), Brooks (1998),
Diaconis and Saloff-Coste (1998), Jerrum and Sinclair (1996), Neal (1993), Tierney(1994), and Andrieu et al. (2003) that provide additional information on sampling

526 11. SAMPLING METHODS
methods for statistical inference.
Diagnostic tests for convergence of Markov chain Monte Carlo algorithms are
summarized in Robert and Casella (1999), and some practical guidance on the use of
sampling methods in the context of machine learning is given in Bishop and Nabney
(2008).
11.1. Basic Sampling Algorithms
In this section, we consider some simple strategies for generating random samplesfrom a given distribution. Because the samples will be generated by a computer
algorithm they will in fact be pseudo-random numbers, that is, they will be deter-
ministically calculated, but must nevertheless pass appropriate tests for randomness.
Generating such numbers raises several subtleties (Press et al. , 1992) that lie outside
the scope of this book. Here we shall assume that an algorithm has been providedthat generates pseudo-random numbers distributed uniformly over (0,1), and indeed
most software environments have such a facility built in.
11.1.1 Standard distributions

We ﬁrst consider how to generate random numbers from simple nonuniform dis-
tributions, assuming that we already have available a source of uniformly distributed
random numbers. Suppose that zis uniformly distributed over the interval (0,1),
and that we transform the values of zusing some function f(·)so that y=f(z).
The distribution of ywill be governed by
p(y)=p(z)⏐⏐⏐⏐dz
dy⏐⏐⏐⏐ (11.5)
where, in this case, p(z)=1 . Our goal is to choose the function f(z)such that the

resulting values of yhave some speciﬁc desired distribution p(y). Integrating (11.5)
we obtain
z=h(y)≡∫y
−∞p(ˆy)dˆy (11.6)
which is the indeﬁnite integral of p(y). Thus, y=h−1(z), and so we have to Exercise 11.2
transform the uniformly distributed random numbers using a function which is the
inverse of the indeﬁnite integral of the desired distribution. This is illustrated in
Figure 11.2.
Consider for example the exponential distribution
p(y)=λexp(−λy) (11.7)

we obtain
z=h(y)≡∫y
where 0⩽y<∞. In this case the lower limit of the integral in (11.6) is 0, and so
h(y)=1−exp(−λy). Thus, if we transform our uniformly distributed variable z
usingy=−λ−1ln(1−z), thenywill have an exponential distribution.

11.1. Basic Sampling Algorithms 527
Figure 11.2 Geometrical interpretation of the trans-
formation method for generating nonuni-
formly distributed random numbers. h(y)
is the indeﬁnite integral of the desired dis-
tribution p(y). If a uniformly distributed
random variable zis transformed using
y=h−1(z), then ywill be distributed ac-
cording to p(y).p(y)h(y)
y01
Another example of a distribution to which the transformation method can be
applied is given by the Cauchy distribution
p(y)=1
π1
1+y2. (11.8)
In this case, the inverse of the indeﬁnite integral can be expressed in terms of the
‘tan’ function. Exercise 11.3
The generalization to multiple variables is straightforward and involves the Ja-
cobian of the change of variables, so that
p(y1,...,y M)=p(z1,...,z M)⏐⏐⏐⏐∂(z1,...,z M)
∂(y1,...,y M)⏐⏐⏐⏐. (11.9)
As a ﬁnal example of the transformation method we consider the Box-Muller
method for generating samples from a Gaussian distribution. First, suppose we gen-

p(y)=1
π1
erate pairs of uniformly distributed random numbers z1,z2∈(−1,1), which we can
do by transforming a variable distributed uniformly over (0,1)usingz→2z−1.
Next we discard each pair unless it satisﬁes z2
1+z2
2⩽1. This leads to a uniform
distribution of points inside the unit circle with p(z1,z2)=1 /π, as illustrated in
Figure 11.3. Then, for each pair z1,z2we evaluate the quantities
Figure 11.3 The Box-Muller method for generating Gaussian dis-
tributed random numbers starts by generating samples

1+z2
from a uniform distribution inside the unit circle.
−1−11
1 z1z2

528 11. SAMPLING METHODS
y1=z1(−2l nz1
r2)1/2
(11.10)
y2=z2(−2l nz2
r2)1/2
(11.11)
where r2=z2
1+z2
2. Then the joint distribution of y1andy2is given by Exercise 11.4
p(y1,y2)= p(z1,z2)⏐⏐⏐⏐∂(z1,z2)
∂(y1,y2)⏐⏐⏐⏐
=[1√
2πexp(−y2
1/2)][1√
2πexp(−y2
2/2)]
(11.12)
and so y1andy2are independent and each has a Gaussian distribution with zero
mean and unit variance.
Ifyhas a Gaussian distribution with zero mean and unit variance, then σy+µ
will have a Gaussian distribution with mean µand variance σ2. To generate vector-
valued variables having a multivariate Gaussian distribution with mean µand co-
variance Σ, we can make use of the Cholesky decomposition , which takes the form
Σ=LLT(Press et al. , 1992). Then, if zis a vector valued random variable whose
components are independent and Gaussian distributed with zero mean and unit vari-
ance, then y=µ+Lzwill have mean µand covariance Σ. Exercise 11.5
Obviously, the transformation technique depends for its success on the ability

to calculate and then invert the indeﬁnite integral of the required distribution. Such
operations will only be feasible for a limited number of simple distributions, and sowe must turn to alternative approaches in search of a more general strategy. Here
we consider two techniques called rejection sampling and importance sampling .A l -
though mainly limited to univariate distributions and thus not directly applicable to
complex problems in many dimensions, they do form important components in more
general strategies.
11.1.2 Rejection sampling

The rejection sampling framework allows us to sample from relatively complex
distributions, subject to certain constraints. We begin by considering univariate dis-tributions and discuss the extension to multiple dimensions subsequently.
Suppose we wish to sample from a distribution p(z)that is not one of the simple,
standard distributions considered so far, and that sampling directly from p(z)is dif-
ﬁcult. Furthermore suppose, as is often the case, that we are easily able to evaluate

p(z)for any given value of z, up to some normalizing constant Z, so that
p(z)=1
Zp˜p(z) (11.13)
where˜p(z)can readily be evaluated, but Zpis unknown.
In order to apply rejection sampling, we need some simpler distribution q(z),
sometimes called a proposal distribution , from which we can readily draw samples.

11.1. Basic Sampling Algorithms 529
Figure 11.4 In the rejection sampling method,
samples are drawn from a sim-
ple distribution q(z)and rejected
if they fall in the grey area be-
tween the unnormalized distribu-
tion ep(z)and the scaled distribu-
tionkq(z). The resulting samples
are distributed according to p(z),
which is the normalized version of
ep(z). z0 zu0kq(z0)kq(z)
˜p(z)
We next introduce a constant kwhose value is chosen such that kq(z)⩾˜p(z)for
all values of z. The function kq(z)is called the comparison function and is illus-
trated for a univariate distribution in Figure 11.4. Each step of the rejection sampler
involves generating two random numbers. First, we generate a number z0from the
distribution q(z). Next, we generate a number u0from the uniform distribution over
[0,kq(z0)]. This pair of random numbers has uniform distribution under the curve
of the function kq(z). Finally, if u0>˜p(z0)then the sample is rejected, otherwise

u0is retained. Thus the pair is rejected if it lies in the grey shaded region in Fig-
ure 11.4. The remaining pairs then have uniform distribution under the curve of ˜p(z),
and hence the corresponding zvalues are distributed according to p(z), as desired. Exercise 11.6
The original values of zare generated from the distribution q(z), and these sam-
ples are then accepted with probability ˜p(z)/kq(z), and so the probability that a
sample will be accepted is given by
p(accept) =∫
{˜p(z)/kq(z)}q(z)dz
=1
k∫
˜p(z)dz. (11.14)

=1
k∫
Thus the fraction of points that are rejected by this method depends on the ratio of
the area under the unnormalized distribution ˜p(z)to the area under the curve kq(z).
We therefore see that the constant kshould be as small as possible subject to the
limitation that kq(z)must be nowhere less than ˜p(z).
As an illustration of the use of rejection sampling, consider the task of sampling
from the gamma distribution
Gam( z|a, b)=baza−1exp(−bz)
Γ(a)(11.15)
which, for a>1, has a bell-shaped form, as shown in Figure 11.5. A suitable

proposal distribution is therefore the Cauchy (11.8) because this too is bell-shaped
and because we can use the transformation method, discussed earlier, to sample from
it. We need to generalize the Cauchy slightly to ensure that it nowhere has a smaller
value than the gamma distribution. This can be achieved by transforming a uniform
random variable yusingz=btany+c, which gives random numbers distributed
according to. Exercise 11.7

530 11. SAMPLING METHODS
Figure 11.5 Plot showing the gamma distribu-
tion given by (11.15) as the green
curve, with a scaled Cauchy pro-
posal distribution shown by the red
curve. Samples from the gamma
distribution can be obtained by
sampling from the Cauchy and
then applying the rejection sam-
pling criterion.
zp(z)
0 10 20 3000.050.10.15
q(z)=k
1+(z−c)2/b2. (11.16)
The minimum reject rate is obtained by setting c=a−1,b2=2a−1and choos-
ing the constant kto be as small as possible while still satisfying the requirement
kq(z)⩾˜p(z). The resulting comparison function is also illustrated in Figure 11.5.
11.1.3 Adaptive rejection sampling
In many instances where we might wish to apply rejection sampling, it proves
difﬁcult to determine a suitable analytic form for the envelope distribution q(z).A n
alternative approach is to construct the envelope function on the ﬂy based on mea-
sured values of the distribution p(z)(Gilks and Wild, 1992). Construction of an

envelope function is particularly straightforward for cases in which p(z)is log con-
cave, in other words when lnp(z)has derivatives that are nonincreasing functions
ofz. The construction of a suitable envelope function is illustrated graphically in
Figure 11.6.
The function lnp(z)and its gradient are evaluated at some initial set of grid
points, and the intersections of the resulting tangent lines are used to construct the
envelope function. Next a sample value is drawn from the envelope distribution.

This is straightforward because the log of the envelope distribution is a succession Exercise 11.9
Figure 11.6 In the case of distributions that are
log concave, an envelope function
for use in rejection sampling can be
constructed using the tangent lines
computed at a set of grid points. If a
sample point is rejected, it is added
to the set of grid points and used to
reﬁne the envelope distribution.
z1 z2 z3 zlnp(z)

11.1. Basic Sampling Algorithms 531
Figure 11.7 Illustrative example of rejection
sampling involving sampling from a
Gaussian distribution p(z)shown by
the green curve, by using rejection
sampling from a proposal distri-
bution q(z)that is also Gaussian
and whose scaled version kq(z)is
shown by the red curve.
zp(z)
−5 0 500.250.5
of linear functions, and hence the envelope distribution itself comprises a piecewise
exponential distribution of the form
q(z)=kiλiexp{−λi(z−zi−1)} zi−1<z⩽zi. (11.17)
Once a sample has been drawn, the usual rejection criterion can be applied. If the
sample is accepted, then it will be a draw from the desired distribution. If, however,
the sample is rejected, then it is incorporated into the set of grid points, a new tangent
line is computed, and the envelope function is thereby reﬁned. As the number of
grid points increases, so the envelope function becomes a better approximation of
the desired distribution p(z)and the probability of rejection decreases.

A variant of the algorithm exists that avoids the evaluation of derivatives (Gilks,
1992). The adaptive rejection sampling framework can also be extended to distri-
butions that are not log concave, simply by following each rejection sampling step
with a Metropolis-Hastings step (to be discussed in Section 11.2.2), giving rise to
adaptive rejection Metropolis sampling (Gilks et al. , 1995).
Clearly for rejection sampling to be of practical value, we require that the com-

parison function be close to the required distribution so that the rate of rejection is
kept to a minimum. Now let us examine what happens when we try to use rejection
sampling in spaces of high dimensionality. Consider, for the sake of illustration,
a somewhat artiﬁcial problem in which we wish to sample from a zero-mean mul-
tivariate Gaussian distribution with covariance σ2
pI, where Iis the unit matrix, by
rejection sampling from a proposal distribution that is itself a zero-mean Gaussian
distribution having covariance σ2

qI. Obviously, we must have σ2
q⩾σ2
pin order that
there exists a ksuch that kq(z)⩾p(z).I nD-dimensions the optimum value of k
is given by k=(σq/σp)D, as illustrated for D=1in Figure 11.7. The acceptance
rate will be the ratio of volumes under p(z)andkq(z), which, because both distribu-
tions are normalized, is just 1/k. Thus the acceptance rate diminishes exponentially
with dimensionality. Even if σqexceeds σpby just one percent, for D=1,000the
acceptance ratio will be approximately 1/20,000. In this illustrative example the

q⩾σ2
comparison function is close to the required distribution. For more practical exam-
ples, where the desired distribution may be multimodal and sharply peaked, it will
be extremely difﬁcult to ﬁnd a good proposal distribution and comparison function.

532 11. SAMPLING METHODS
Figure 11.8 Importance sampling addresses the prob-
lem of evaluating the expectation of a func-
tionf(z)with respect to a distribution p(z)
from which it is difﬁcult to draw samples di-
rectly. Instead, samples {z(l)}are drawn
from a simpler distribution q(z), and the
corresponding terms in the summation are
weighted by the ratios p(z(l))/q(z(l)).p(z)f(z)
zq(z)
Furthermore, the exponential decrease of acceptance rate with dimensionality is a
generic feature of rejection sampling. Although rejection can be a useful technique
in one or two dimensions it is unsuited to problems of high dimensionality. It can,
however, play a role as a subroutine in more sophisticated algorithms for sampling
in high dimensional spaces.
11.1.4 Importance sampling
One of the principal reasons for wishing to sample from complicated probability
distributions is to be able to evaluate expectations of the form (11.1). The technique
ofimportance sampling provides a framework for approximating expectations di-

rectly but does not itself provide a mechanism for drawing samples from distribution
p(z).
The ﬁnite sum approximation to the expectation, given by (11.2), depends on
being able to draw samples from the distribution p(z). Suppose, however, that it is
impractical to sample directly from p(z)but that we can evaluate p(z)easily for any
given value of z. One simplistic strategy for evaluating expectations would be to
discretize z-space into a uniform grid and to evaluate the integrand as a sum of the
form
E[f]≃L∑
l=1p(z(l))f(z(l)). (11.18)

p(z).
form
E[f]≃L∑
An obvious problem with this approach is that the number of terms in the summation
grows exponentially with the dimensionality of z. Furthermore, as we have already
noted, the kinds of probability distributions of interest will often have much of their
mass conﬁned to relatively small regions of zspace and so uniform sampling will be
very inefﬁcient because in high-dimensional problems, only a very small proportion
of the samples will make a signiﬁcant contribution to the sum. We would really like

to choose the sample points to fall in regions where p(z)is large, or ideally where
the product p(z)f(z)is large.
As in the case of rejection sampling, importance sampling is based on the use
of a proposal distribution q(z)from which it is easy to draw samples, as illustrated
in Figure 11.8. We can then express the expectation in the form of a ﬁnite sum over

11.1. Basic Sampling Algorithms 533
samples {z(l)}drawn from q(z)
E[f]=∫
f(z)p(z)dz
=∫
f(z)p(z)
q(z)q(z)dz
≃1
LL∑
l=1p(z(l))
q(z(l))f(z(l)). (11.19)
The quantities rl=p(z(l))/q(z(l))are known as importance weights , and they cor-
rect the bias introduced by sampling from the wrong distribution. Note that, unlike
rejection sampling, all of the samples generated are retained.
It will often be the case that the distribution p(z)can only be evaluated up to a
normalization constant, so that p(z)=˜p(z)/Zpwhere˜p(z)can be evaluated easily,
whereas Zpis unknown. Similarly, we may wish to use an importance sampling
distribution q(z)=˜q(z)/Zq, which has the same property. We then have
E[f]=∫
=Zq
Zp∫
f(z)˜p(z)
˜q(z)q(z)dz
≃Zq
Zp1
LL∑
l=1˜rlf(z(l)). (11.20)
where˜rl=˜p(z(l))/˜q(z(l)). We can use the same sample set to evaluate the ratio
Zp/Zqwith the result
Zp
Zq=1
Zq∫
˜p(z)dz=∫˜p(z)
≃1
LL∑
l=1˜rl (11.21)
and hence
E[f]≃L∑
l=1wlf(z(l)) (11.22)
wl=˜rl∑
m˜rm=˜p(z(l))/q(z(l))∑

E[f]=∫
=Zq
Zp∫
f(z)˜p(z)
≃Zq
Zp1
LL∑
Zp
Zq=1
Zq∫
≃1
LL∑
and hence
E[f]≃L∑
wl=˜rl∑
m˜p(z(m))/q(z(m)). (11.23)
As with rejection sampling, the success of the importance sampling approach
depends crucially on how well the sampling distribution q(z)matches the desired

534 11. SAMPLING METHODS
distribution p(z). If, as is often the case, p(z)f(z)is strongly varying and has a sig-
niﬁcant proportion of its mass concentrated over relatively small regions of zspace,
then the set of importance weights {rl}may be dominated by a few weights hav-
ing large values, with the remaining weights being relatively insigniﬁcant. Thus the
effective sample size can be much smaller than the apparent sample size L. The prob-
lem is even more severe if none of the samples falls in the regions where p(z)f(z)
is large. In that case, the apparent variances of rlandrlf(z(l))may be small even
though the estimate of the expectation may be severely wrong. Hence a major draw-back of the importance sampling method is the potential to produce results that are
arbitrarily in error and with no diagnostic indication. This also highlights a key re-
quirement for the sampling distribution q(z), namely that it should not be small or
zero in regions where p(z)may be signiﬁcant.

For distributions deﬁned in terms of a graphical model, we can apply the impor-
tance sampling technique in various ways. For discrete variables, a simple approach
is called uniform sampling . The joint distribution for a directed graph is deﬁned
by (11.4). Each sample from the joint distribution is obtained by ﬁrst setting thosevariables z
ithat are in the evidence set equal to their observed values. Each of the
remaining variables is then sampled independently from a uniform distribution over

the space of possible instantiations. To determine the corresponding weight associ-ated with a sample z
(l), we note that the sampling distribution ˜q(z)is uniform over
the possible choices for z, and that˜p(z|x)=˜p(z), where xdenotes the subset of
variables that are observed, and the equality follows from the fact that every samplezthat is generated is necessarily consistent with the evidence. Thus the weights r
l
are simply proportional to p(z). Note that the variables can be sampled in any order.

l
This approach can yield poor results if the posterior distribution is far from uniform,as is often the case in practice.
An improvement on this approach is called likelihood weighted sampling (Fung
and Chang, 1990; Shachter and Peot, 1990) and is based on ancestral sampling of
the variables. For each variable in turn, if that variable is in the evidence set, then it
is just set to its instantiated value. If it is not in the evidence set, then it is sampledfrom the conditional distribution p(z

i|pai)in which the conditioning variables are
set to their currently sampled values. The weighting associated with the resulting
sample zis then given by
r(z)=∏
zi̸∈ep(zi|pai)
p(zi|pai)∏
zi∈ep(zi|pai)
1=∏
zi∈ep(zi|pai). (11.24)
This method can be further extended using self-importance sampling (Shachter and
Peot, 1990) in which the importance sampling distribution is continually updated to
reﬂect the current estimated posterior distribution.
11.1.5 Sampling-importance-resampling

r(z)=∏
1=∏
The rejection sampling method discussed in Section 11.1.2 depends in part for
its success on the determination of a suitable value for the constant k. For many
pairs of distributions p(z)andq(z), it will be impractical to determine a suitable

11.1. Basic Sampling Algorithms 535
value for kin that any value that is sufﬁciently large to guarantee a bound on the
desired distribution will lead to impractically small acceptance rates.
As in the case of rejection sampling, the sampling-importance-resampling (SIR)
approach also makes use of a sampling distribution q(z)but avoids having to de-
termine the constant k. There are two stages to the scheme. In the ﬁrst stage,
Lsamples z(1),...,z(L)are drawn from q(z). Then in the second stage, weights
w1,...,w Lare constructed using (11.23). Finally, a second set of Lsamples is
drawn from the discrete distribution (z(1),...,z(L))with probabilities given by the
weights (w1,...,w L).
The resulting Lsamples are only approximately distributed according to p(z),
but the distribution becomes correct in the limit L→∞ . To see this, consider the
univariate case, and note that the cumulative distribution of the resampled values is
given by
p(z⩽a)=∑
l:z(l)⩽awl
=∑
lI(z(l)⩽a)˜p(z(l))/q(z(l))∑
l˜p(z(l))/q(z(l))(11.25)

given by
p(z⩽a)=∑
=∑
where I(.)is the indicator function (which equals 1if its argument is true and 0
otherwise). Taking the limit L→∞ , and assuming suitable regularity of the dis-
tributions, we can replace the sums by integrals weighted according to the original
sampling distribution q(z)
p(z⩽a)=∫
I(z⩽a){˜p(z)/q(z)}q(z)dz
∫
{˜p(z)/q(z)}q(z)dz
=∫
I(z⩽a)˜p(z)dz
∫
˜p(z)dz
=∫
I(z⩽a)p(z)dz (11.26)
which is the cumulative distribution function of p(z). Again, we see that the normal-
ization of p(z)is not required.
For a ﬁnite value of L, and a given initial sample set, the resampled values will

p(z⩽a)=∫
∫
=∫
∫
˜p(z)dz
=∫
only approximately be drawn from the desired distribution. As with rejection sam-
pling, the approximation improves as the sampling distribution q(z)gets closer to
the desired distribution p(z). When q(z)=p(z), the initial samples (z(1),...,z(L))
have the desired distribution, and the weights wn=1/Lso that the resampled values
also have the desired distribution.
If moments with respect to the distribution p(z)are required, then they can be

536 11. SAMPLING METHODS
evaluated directly using the original samples together with the weights, because
E[f(z)] =∫
=∫
f(z)[˜p(z)/q(z)]q(z)dz
∫
[˜p(z)/q(z)]q(z)dz
≃L∑
l=1wlf(zl). (11.27)
11.1.6 Sampling and the EM algorithm
In addition to providing a mechanism for direct implementation of the Bayesian
framework, Monte Carlo methods can also play a role in the frequentist paradigm,for example to ﬁnd maximum likelihood solutions. In particular, sampling methods
can be used to approximate the E step of the EM algorithm for models in which the
E step cannot be performed analytically. Consider a model with hidden variablesZ, visible (observed) variables X, and parameters θ. The function that is optimized
with respect to θin the M step is the expected complete-data log likelihood, given
by
Q(θ,θ
old)=∫
p(Z|X,θold)l np(Z,X|θ)dZ. (11.28)
We can use sampling methods to approximate this integral by a ﬁnite sum over sam-
ples{Z(l)}, which are drawn from the current estimate for the posterior distribution

by
Q(θ,θ
old)=∫
p(Z|X,θold), so that
Q(θ,θold)≃1
LL∑
l=1lnp(Z(l),X|θ). (11.29)
TheQfunction is then optimized in the usual way in the M step. This procedure is
called the Monte Carlo EM algorithm .
It is straightforward to extend this to the problem of ﬁnding the mode of the
posterior distribution over θ(the MAP estimate) when a prior distribution p(θ)has
been deﬁned, simply by adding lnp(θ)to the function Q(θ,θold)before performing
the M step.
A particular instance of the Monte Carlo EM algorithm, called stochastic EM ,

LL∑
the M step.
arises if we consider a ﬁnite mixture model, and draw just one sample at each E step.
Here the latent variable Zcharacterizes which of the Kcomponents of the mixture
is responsible for generating each data point. In the E step, a sample of Zis taken
from the posterior distribution p(Z|X,θold)whereXis the data set. This effectively
makes a hard assignment of each data point to one of the components in the mixture.In the M step, this sampled approximation to the posterior distribution is used to

update the model parameters in the usual way.

11.2. Markov Chain Monte Carlo 537
Now suppose we move from a maximum likelihood approach to a full Bayesian
treatment in which we wish to sample from the posterior distribution over the param-eter vector θ. In principle, we would like to draw samples from the joint posterior
p(θ,Z|X), but we shall suppose that this is computationally difﬁcult. Suppose fur-
ther that it is relatively straightforward to sample from the complete-data parameterposterior p(θ|Z,X). This inspires the data augmentation algorithm, which alter-
nates between two steps known as the I-step (imputation step, analogous to an E
step) and the P-step (posterior step, analogous to an M step).
IP Algorithm
I-step. We wish to sample from p(Z|X)but we cannot do this directly. We
therefore note the relation
p(Z|X)=∫
p(Z|θ,X)p(θ|X)dθ (11.30)
and hence for l=1,...,L we ﬁrst draw a sample θ(l)from the current esti-
mate for p(θ|X), and then use this to draw a sample Z(l)fromp(Z|θ(l),X).
P-step. Given the relation
p(θ|X)=∫
p(θ|Z,X)p(Z|X)dZ (11.31)

p(Z|X)=∫
p(θ|X)=∫
we use the samples {Z(l)}obtained from the I-step to compute a revised
estimate of the posterior distribution over θgiven by
p(θ|X)≃1
LL∑
l=1p(θ|Z(l),X). (11.32)
By assumption, it will be feasible to sample from this approximation in the
I-step.
Note that we are making a (somewhat artiﬁcial) distinction between parameters θ
and hidden variables Z. From now on, we blur this distinction and focus simply on
the problem of drawing samples from a given posterior distribution.
11.2. Markov Chain Monte Carlo

p(θ|X)≃1
LL∑
I-step.
In the previous section, we discussed the rejection sampling and importance sam-pling strategies for evaluating expectations of functions, and we saw that they suffer
from severe limitations particularly in spaces of high dimensionality. We thereforeturn in this section to a very general and powerful framework called Markov chain
Monte Carlo (MCMC), which allows sampling from a large class of distributions,

538 11. SAMPLING METHODS
and which scales well with the dimensionality of the sample space. Markov chain
Monte Carlo methods have their origins in physics (Metropolis and Ulam, 1949),and it was only towards the end of the 1980s that they started to have a signiﬁcant
impact in the ﬁeld of statistics.
As with rejection and importance sampling, we again sample from a proposal
distribution. This time, however, we maintain a record of the current state z
(τ), and
the proposal distribution q(z|z(τ))depends on this current state, and so the sequence
of samples z(1),z(2),...forms a Markov chain. Again, if we write p(z)=˜p(z)/Zp, Section 11.2.1
we will assume that ˜p(z)can readily be evaluated for any given value of z, although
the value of Zpmay be unknown. The proposal distribution itself is chosen to be
sufﬁciently simple that it is straightforward to draw samples from it directly. At
each cycle of the algorithm, we generate a candidate sample z⋆from the proposal

distribution and then accept the sample according to an appropriate criterion.
In the basic Metropolis algorithm (Metropolis et al. , 1953), we assume that the
proposal distribution is symmetric, that is q(zA|zB)=q(zB|zA)for all values of
zAandzB. The candidate sample is then accepted with probability
A(z⋆,z(τ))=m i n(
1,˜p(z⋆)
˜p(z(τ)))
. (11.33)
This can be achieved by choosing a random number uwith uniform distribution over
the unit interval (0,1)and then accepting the sample if A(z⋆,z(τ))>u. Note that

1,˜p(z⋆)
˜p(z(τ)))
. (11.33)
if the step from z(τ)toz⋆causes an increase in the value of p(z), then the candidate
point is certain to be kept.
If the candidate sample is accepted, then z(τ+1)=z⋆, otherwise the candidate
pointz⋆is discarded, z(τ+1)is set to z(τ)and another candidate sample is drawn
from the distribution q(z|z(τ+1)). This is in contrast to rejection sampling, where re-

jected samples are simply discarded. In the Metropolis algorithm when a candidatepoint is rejected, the previous sample is included instead in the ﬁnal list of samples,
leading to multiple copies of samples. Of course, in a practical implementation,
only a single copy of each retained sample would be kept, along with an integer
weighting factor recording how many times that state appears. As we shall see, as
long as q(z
A|zB)is positive for any values of zAandzB(this is a sufﬁcient but
not necessary condition), the distribution of z(τ)tends to p(z)asτ→∞ . It should

long as q(z
be emphasized, however, that the sequence z(1),z(2),... is not a set of independent
samples from p(z)because successive samples are highly correlated. If we wish to
obtain independent samples, then we can discard most of the sequence and just re-
tain every Mthsample. For Msufﬁciently large, the retained samples will for all
practical purposes be independent. Figure 11.9 shows a simple illustrative exam-ple of sampling from a two-dimensional Gaussian distribution using the Metropolis
algorithm in which the proposal distribution is an isotropic Gaussian.

Further insight into the nature of Markov chain Monte Carlo algorithms can be
gleaned by looking at the properties of a speciﬁc example, namely a simple random

11.2. Markov Chain Monte Carlo 539
Figure 11.9 A simple illustration using Metropo-
lis algorithm to sample from a
Gaussian distribution whose one
standard-deviation contour is shown
by the ellipse. The proposal distribu-
tion is an isotropic Gaussian distri-
bution whose standard deviation is
0.2. Steps that are accepted are
shown as green lines, and rejected
steps are shown in red. A total of
150 candidate samples are gener-
ated, of which 43 are rejected.
0 0.5 1 1.5 2 2.5 300.511.522.53
walk. Consider a state space zconsisting of the integers, with probabilities
p(z(τ+1)=z(τ))=0 .5 (11.34)
p(z(τ+1)=z(τ)+1 ) = 0 .25 (11.35)
p(z(τ+1)=z(τ)−1) = 0 .25 (11.36)
where z(τ)denotes the state at step τ. If the initial state is z(1)=0, then by sym-
metry the expected state at time τwill also be zero E[z(τ)]=0 , and similarly it is
easily seen that E[(z(τ))2]=τ/2. Thus after τsteps, the random walk has only trav- Exercise 11.10
elled a distance that on average is proportional to the square root of τ. This square

root dependence is typical of random walk behaviour and shows that random walks
are very inefﬁcient in exploring the state space. As we shall see, a central goal in
designing Markov chain Monte Carlo methods is to avoid random walk behaviour.
11.2.1 Markov chains
Before discussing Markov chain Monte Carlo methods in more detail, it is use-
ful to study some general properties of Markov chains in more detail. In particular,
we ask under what circumstances will a Markov chain converge to the desired dis-

tribution. A ﬁrst-order Markov chain is deﬁned to be a series of random variables
z(1),...,z(M)such that the following conditional independence property holds for
m∈{1,...,M −1}
p(z(m+1)|z(1),...,z(m))=p(z(m+1)|z(m)). (11.37)
This of course can be represented as a directed graph in the form of a chain, an ex-
ample of which is shown in Figure 8.38. We can then specify the Markov chain by
giving the probability distribution for the initial variable p(z(0))together with the

540 11. SAMPLING METHODS
conditional probabilities for subsequent variables in the form of transition probabil-
itiesTm(z(m),z(m+1))≡p(z(m+1)|z(m)). A Markov chain is called homogeneous
if the transition probabilities are the same for all m.
The marginal probability for a particular variable can be expressed in terms of
the marginal probability for the previous variable in the chain in the form
p(z(m+1))=∑
z(m)p(z(m+1)|z(m))p(z(m)). (11.38)
A distribution is said to be invariant, or stationary, with respect to a Markov chain
if each step in the chain leaves that distribution invariant. Thus, for a homogeneousMarkov chain with transition probabilities T(z
′,z), the distribution p⋆(z)is invariant
if
p⋆(z)=∑
z′T(z′,z)p⋆(z′). (11.39)
Note that a given Markov chain may have more than one invariant distribution. For
instance, if the transition probabilities are given by the identity transformation, then
any distribution will be invariant.

if
p⋆(z)=∑
A sufﬁcient (but not necessary) condition for ensuring that the required distribu-
tionp(z)is invariant is to choose the transition probabilities to satisfy the property
ofdetailed balance , deﬁned by
p⋆(z)T(z,z′)=p⋆(z′)T(z′,z) (11.40)
for the particular distribution p⋆(z). It is easily seen that a transition probability
that satisﬁes detailed balance with respect to a particular distribution will leave that
distribution invariant, because
∑
z′p⋆(z′)T(z′,z)=∑
z′p⋆(z)T(z,z′)=p⋆(z)∑
z′p(z′|z)=p⋆(z).(11.41)

∑
A Markov chain that respects detailed balance is said to be reversible .
Our goal is to use Markov chains to sample from a given distribution. We can
achieve this if we set up a Markov chain such that the desired distribution is invariant.
However, we must also require that for m→∞ , the distribution p(z(m))converges
to the required invariant distribution p⋆(z), irrespective of the choice of initial dis-
tribution p(z(0)). This property is called ergodicity , and the invariant distribution

is then called the equilibrium distribution. Clearly, an ergodic Markov chain can
have only one equilibrium distribution. It can be shown that a homogeneous Markovchain will be ergodic, subject only to weak restrictions on the invariant distribution
and the transition probabilities (Neal, 1993).
In practice we often construct the transition probabilities from a set of ‘base’
transitions B
1,...,B K. This can be achieved through a mixture distribution of the
form
T(z′,z)=K∑
k=1αkBk(z′,z) (11.42)

11.2. Markov Chain Monte Carlo 541
for some set of mixing coefﬁcients α1,...,α Ksatisfying αk⩾0and∑
kαk=1.
Alternatively, the base transitions may be combined through successive application,so that
T(z
′,z)=∑
z1...∑
zn−1B1(z′,z1)...B K−1(zK−2,zK−1)BK(zK−1,z).(11.43)
If a distribution is invariant with respect to each of the base transitions, then obvi-
ously it will also be invariant with respect to either of the T(z′,z)given by (11.42)
or (11.43). For the case of the mixture (11.42), if each of the base transitions sat-
isﬁes detailed balance, then the mixture transition Twill also satisfy detailed bal-
ance. This does not hold for the transition probability constructed using (11.43), al-though by symmetrizing the order of application of the base transitions, in the form
B
1,B2,...,B K,BK,...,B 2,B1, detailed balance can be restored. A common ex-
ample of the use of composite transition probabilities is where each base transitionchanges only a subset of the variables.
11.2.2 The Metropolis-Hastings algorithm

B
Earlier we introduced the basic Metropolis algorithm, without actually demon-
strating that it samples from the required distribution. Before giving a proof, we
ﬁrst discuss a generalization, known as the Metropolis-Hastings algorithm (Hast-
ings, 1970), to the case where the proposal distribution is no longer a symmetric
function of its arguments. In particular at step τof the algorithm, in which the cur-
rent state is z(τ), we draw a sample z⋆from the distribution qk(z|z(τ))and then

accept it with probability Ak(z⋆,zτ)where
Ak(z⋆,z(τ))=m i n(
1,˜p(z⋆)qk(z(τ)|z⋆)
˜p(z(τ))qk(z⋆|z(τ)))
. (11.44)
Hereklabels the members of the set of possible transitions being considered. Again,
the evaluation of the acceptance criterion does not require knowledge of the normal-
izing constant Zpin the probability distribution p(z)=˜p(z)/Zp. For a symmetric
proposal distribution the Metropolis-Hastings criterion (11.44) reduces to the stan-
dard Metropolis criterion given by (11.33).

. (11.44)
We can show that p(z)is an invariant distribution of the Markov chain deﬁned
by the Metropolis-Hastings algorithm by showing that detailed balance, deﬁned by
(11.40), is satisﬁed. Using (11.44) we have
p(z)qk(z|z′)Ak(z′,z)=m i n ( p(z)qk(z|z′),p(z′)qk(z′|z))
=m i n ( p(z′)qk(z′|z),p(z)qk(z|z′))
=p(z′)qk(z′|z)Ak(z,z′) (11.45)
The speciﬁc choice of proposal distribution can have a marked effect on the

performance of the algorithm. For continuous state spaces, a common choice is aGaussian centred on the current state, leading to an important trade-off in determin-
ing the variance parameter of this distribution. If the variance is small, then the

542 11. SAMPLING METHODS
Figure 11.10 Schematic illustration of the use of an isotropic
Gaussian proposal distribution (blue circle) to
sample from a correlated multivariate Gaussian
distribution (red ellipse) having very different stan-
dard deviations in different directions, using the
Metropolis-Hastings algorithm. In order to keep
the rejection rate low, the scale ρof the proposal
distribution should be on the order of the smallest
standard deviation σmin, which leads to random
walk behaviour in which the number of steps sep-
arating states that are approximately independent
is of order (σmax/σmin)2where σmaxis the largest
standard deviation.σmax
σminρ
proportion of accepted transitions will be high, but progress through the state space
takes the form of a slow random walk leading to long correlation times. However,
if the variance parameter is large, then the rejection rate will be high because, in the
kind of complex problems we are considering, many of the proposed steps will be

σminρ
to states for which the probability p(z)is low. Consider a multivariate distribution
p(z)having strong correlations between the components of z, as illustrated in Fig-
ure 11.10. The scale ρof the proposal distribution should be as large as possible
without incurring high rejection rates. This suggests that ρshould be of the same
order as the smallest length scale σmin. The system then explores the distribution
along the more extended direction by means of a random walk, and so the number

of steps to arrive at a state that is more or less independent of the original state is
of order (σmax/σmin)2. In fact in two dimensions, the increase in rejection rate as ρ
increases is offset by the larger steps sizes of those transitions that are accepted, and
more generally for a multivariate Gaussian the number of steps required to obtain
independent samples scales like (σmax/σ2)2where σ2is the second-smallest stan-
dard deviation (Neal, 1993). These details aside, it remains the case that if the length

scales over which the distributions vary are very different in different directions, then
the Metropolis Hastings algorithm can have very slow convergence.
11.3. Gibbs Sampling
Gibbs sampling (Geman and Geman, 1984) is a simple and widely applicable Markov
chain Monte Carlo algorithm and can be seen as a special case of the Metropolis-
Hastings algorithm.
Consider the distribution p(z)=p(z1,...,z M)from which we wish to sample,
and suppose that we have chosen some initial state for the Markov chain. Each step

of the Gibbs sampling procedure involves replacing the value of one of the variables
by a value drawn from the distribution of that variable conditioned on the values of
the remaining variables. Thus we replace ziby a value drawn from the distribution
p(zi|z\i), where zidenotes the ithcomponent of z, andz\idenotes z1,...,z Mbut
withziomitted. This procedure is repeated either by cycling through the variables

11.3. Gibbs Sampling 543
in some particular order or by choosing the variable to be updated at each step at
random from some distribution.
For example, suppose we have a distribution p(z1,z2,z3)over three variables,
and at step τof the algorithm we have selected values z(τ)
1,z(τ)
2andz(τ)
3. We ﬁrst
replace z(τ)
1by a new value z(τ+1)
1 obtained by sampling from the conditional distri-
bution
p(z1|z(τ)
2,z(τ)
3). (11.46)
Next we replace z(τ)
2by a value z(τ+1)
2 obtained by sampling from the conditional
p(z2|z(τ+1)
1,z(τ)
3) (11.47)
so that the new value for z1is used straight away in subsequent sampling steps. Then
we update z3with a sample z(τ+1)
3 drawn from
p(z3|z(τ+1)
1,z(τ+1)
2) (11.48)
and so on, cycling through the three variables in turn.
Gibbs Sampling
1. Initialize {zi:i=1,...,M }
2. For τ=1,...,T :
–Sample z(τ+1)
1∼p(z1|z(τ)
2,z(τ)
3,...,z(τ)
M).
2∼p(z2|z(τ+1)
1,z(τ)
M).
...
j∼p(zj|z(τ+1)
1,...,z(τ+1)
j−1,z(τ)
j+1,...,z(τ)
M).
...

1,z(τ)
3) (11.47)
1,z(τ+1)
2) (11.48)
2,z(τ)
M).
1,z(τ)
M).
...
j−1,z(τ)
M).
...
M∼p(zM|z(τ+1)
1,z(τ+1)
2,...,z(τ+1)
M−1).
Josiah Willard Gibbs
1839–1903
Gibbs spent almost his entire life liv-
ing in a house built by his father inNew Haven, Connecticut. In 1863,Gibbs was granted the ﬁrst PhD inengineering in the United States,and in 1871 he was appointed to
the ﬁrst chair of mathematical physics in the UnitedStates at Y ale, a post for which he received no salary

1,z(τ)
M).
...
j−1,z(τ)
M).
...
1,z(τ+1)
M−1).
1839–1903
because at the time he had no publications. He de-veloped the ﬁeld of vector analysis and made contri-butions to crystallography and planetary orbits. His
most famous work, entitled
On the Equilibrium of Het-
erogeneous Substances , laid the foundations for the
science of physical chemistry.

544 11. SAMPLING METHODS
To show that this procedure samples from the required distribution, we ﬁrst of
all note that the distribution p(z)is an invariant of each of the Gibbs sampling steps
individually and hence of the whole Markov chain. This follows from the fact that
when we sample from p(zi|{z\i), the marginal distribution p(z\i)is clearly invariant
because the value of z\iis unchanged. Also, each step by deﬁnition samples from the
correct conditional distribution p(zi|z\i). Because these conditional and marginal
distributions together specify the joint distribution, we see that the joint distribution
is itself invariant.
The second requirement to be satisﬁed in order that the Gibbs sampling proce-
dure samples from the correct distribution is that it be ergodic. A sufﬁcient condition
for ergodicity is that none of the conditional distributions be anywhere zero. If this
is the case, then any point in zspace can be reached from any other point in a ﬁnite

number of steps involving one update of each of the component variables. If thisrequirement is not satisﬁed, so that some of the conditional distributions have zeros,
then ergodicity, if it applies, must be proven explicitly.
The distribution of initial states must also be speciﬁed in order to complete the
algorithm, although samples drawn after many iterations will effectively become
independent of this distribution. Of course, successive samples from the Markov

chain will be highly correlated, and so to obtain samples that are nearly independentit will be necessary to subsample the sequence.
We can obtain the Gibbs sampling procedure as a particular instance of the
Metropolis-Hastings algorithm as follows. Consider a Metropolis-Hastings samplingstep involving the variable z
kin which the remaining variables z\kremain ﬁxed, and
for which the transition probability from ztoz⋆is given by qk(z⋆|z)=p(z⋆
k|z\k).
We note that z⋆
\k=z\kbecause these components are unchanged by the sampling

k|z\k).
step. Also, p(z)=p(zk|z\k)p(z\k). Thus the factor that determines the acceptance
probability in the Metropolis-Hastings (11.44) is given by
A(z⋆,z)=p(z⋆)qk(z|z⋆)
p(z)qk(z⋆|z)=p(z⋆
k|z⋆
\k)p(z⋆
\k)p(zk|z⋆
\k)
p(zk|z\k)p(z\k)p(z⋆
k|z\k)=1 (11.49)
where we have used z⋆
\k=z\k. Thus the Metropolis-Hastings steps are always
accepted.
As with the Metropolis algorithm, we can gain some insight into the behaviour of
Gibbs sampling by investigating its application to a Gaussian distribution. Consider

k|z⋆
\k)p(z⋆
\k)
accepted.
a correlated Gaussian in two variables, as illustrated in Figure 11.11, having con-ditional distributions of width land marginal distributions of width L. The typical
step size is governed by the conditional distributions and will be of order l. Because
the state evolves according to a random walk, the number of steps needed to obtain
independent samples from the distribution will be of order (L/l)
2. Of course if the

Gaussian distribution were uncorrelated, then the Gibbs sampling procedure wouldbe optimally efﬁcient. For this simple problem, we could rotate the coordinate sys-
tem in order to decorrelate the variables. However, in practical applications it will
generally be infeasible to ﬁnd such transformations.
One approach to reducing random walk behaviour in Gibbs sampling is called
over-relaxation (Adler, 1981). In its original form, this applies to problems for which

11.3. Gibbs Sampling 545
Figure 11.11 Illustration of Gibbs sampling by alter-
nate updates of two variables whose
distribution is a correlated Gaussian.
The step size is governed by the stan-
dard deviation of the conditional distri-
bution (green curve), and is O(l), lead-
ing to slow progress in the direction of
elongation of the joint distribution (red
ellipse). The number of steps needed
to obtain an independent sample from
the distribution is O((L/l)2).
z1z2
L
l
the conditional distributions are Gaussian, which represents a more general class of
distributions than the multivariate Gaussian because, for example, the non-Gaussian
distribution p(z,y)∝exp(−z2y2)has Gaussian conditional distributions. At each
step of the Gibbs sampling algorithm, the conditional distribution for a particular
component zihas some mean µiand some variance σ2
i. In the over-relaxation frame-
work, the value of ziis replaced with
z′
i=µi+α(zi−µi)+σi(1−α2
i)1/2ν (11.50)

z1z2
L
l
z′
where νis a Gaussian random variable with zero mean and unit variance, and α
is a parameter such that −1<α< 1.F o r α=0, the method is equivalent to
standard Gibbs sampling, and for α<0the step is biased to the opposite side of the
mean. This step leaves the desired distribution invariant because if zihas mean µi
and variance σ2
i, then so too does z′
i. The effect of over-relaxation is to encourage
directed motion through state space when the variables are highly correlated. The

framework of ordered over-relaxation (Neal, 1999) generalizes this approach to non-
Gaussian distributions.
The practical applicability of Gibbs sampling depends on the ease with which
samples can be drawn from the conditional distributions p(zk|z\k). In the case of
probability distributions speciﬁed using graphical models, the conditional distribu-
tions for individual nodes depend only on the variables in the corresponding Markov
blankets, as illustrated in Figure 11.12. For directed graphs, a wide choice of condi-

tional distributions for the individual nodes conditioned on their parents will lead to
conditional distributions for Gibbs sampling that are log concave. The adaptive re-
jection sampling methods discussed in Section 11.1.3 therefore provide a framework
for Monte Carlo sampling from directed graphs with broad applicability.
If the graph is constructed using distributions from the exponential family, and
if the parent-child relationships preserve conjugacy, then the full conditional distri-
butions arising in Gibbs sampling will have the same functional form as the orig-

546 11. SAMPLING METHODS
Figure 11.12 The Gibbs sampling method requires samples
to be drawn from the conditional distribution of a variable condi-
tioned on the remaining variables. For graphical models, this
conditional distribution is a function only of the states of thenodes in the Markov blanket. For an undirected graph this com-prises the set of neighbours, as shown on the left, while for adirected graph the Markov blanket comprises the parents, thechildren, and the co-parents, as shown on the right.
inal conditional distributions (conditioned on the parents) deﬁning each node, and
so standard sampling techniques can be employed. In general, the full conditional
distributions will be of a complex form that does not permit the use of standard sam-
pling algorithms. However, if these conditionals are log concave, then sampling canbe done efﬁciently using adaptive rejection sampling (assuming the corresponding
variable is a scalar).

If, at each stage of the Gibbs sampling algorithm, instead of drawing a sample
from the corresponding conditional distribution, we make a point estimate of the
variable given by the maximum of the conditional distribution, then we obtain the
iterated conditional modes (ICM) algorithm discussed in Section 8.3.3. Thus ICMcan be seen as a greedy approximation to Gibbs sampling.
Because the basic Gibbs sampling technique considers one variable at a time,

there are strong dependencies between successive samples. At the opposite extreme,if we could draw samples directly from the joint distribution (an operation that we
are supposing is intractable), then successive samples would be independent. We can
hope to improve on the simple Gibbs sampler by adopting an intermediate strategy in
which we sample successively from groups of variables rather than individual vari-
ables. This is achieved in the blocking Gibbs sampling algorithm by choosing blocks

of variables, not necessarily disjoint, and then sampling jointly from the variables in
each block in turn, conditioned on the remaining variables (Jensen et al. , 1995).
11.4. Slice Sampling
We have seen that one of the difﬁculties with the Metropolis algorithm is the sensi-
tivity to step size. If this is too small, the result is slow decorrelation due to random
walk behaviour, whereas if it is too large the result is inefﬁciency due to a high rejec-

tion rate. The technique of slice sampling (Neal, 2003) provides an adaptive step size
that is automatically adjusted to match the characteristics of the distribution. Again
it requires that we are able to evaluate the unnormalized distribution ˜p(z).
Consider ﬁrst the univariate case. Slice sampling involves augmenting zwith
an additional variable uand then drawing samples from the joint (z,u)space. We
shall see another example of this approach when we discuss hybrid Monte Carlo in

Section 11.5. The goal is to sample uniformly from the area under the distribution

11.4. Slice Sampling 547
˜p(z)
z(τ) zu
(a)˜p(z)
z(τ) zu zmin zmax
(b)
Figure 11.13 Illustration of slice sampling. (a) For a given value z(τ), a value of uis chosen uniformly in
the region 0⩽u⩽ep(z(τ)), which then deﬁnes a ‘slice’ through the distribution, shown by the solid horizontal
lines. (b) Because it is infeasible to sample directly from a slice, a new sample of zis drawn from a region
zmin⩽z⩽zmax, which contains the previous value z(τ).
given by
ˆp(z,u)={1/Zpif0⩽u⩽˜p(z)
0 otherwise(11.51)
where Zp=∫˜p(z)dz. The marginal distribution over zis given by
∫
ˆp(z,u)du=∫ep(z)
01
Zpdu=˜p(z)
Zp=p(z) (11.52)
and so we can sample from p(z)by sampling from ˆp(z,u)and then ignoring the u
values. This can be achieved by alternately sampling zandu. Given the value of z
we evaluate ˜p(z)and then sample uuniformly in the range 0⩽u⩽˜p(z), which is
straightforward. Then we ﬁx uand sample zuniformly from the ‘slice’ through the
distribution deﬁned by {z:˜p(z)>u}. This is illustrated in Figure 11.13(a).

∫
01
In practice, it can be difﬁcult to sample directly from a slice through the distribu-
tion and so instead we deﬁne a sampling scheme that leaves the uniform distribution
underˆp(z,u)invariant, which can be achieved by ensuring that detailed balance is
satisﬁed. Suppose the current value of zis denoted z(τ)and that we have obtained
a corresponding sample u. The next value of zis obtained by considering a region
zmin⩽z⩽zmaxthat contains z(τ). It is in the choice of this region that the adap-

tation to the characteristic length scales of the distribution takes place. We want the
region to encompass as much of the slice as possible so as to allow large moves in z
space while having as little as possible of this region lying outside the slice, because
this makes the sampling less efﬁcient.
One approach to the choice of region involves starting with a region containing
z(τ)having some width wand then testing each of the end points to see if they lie

within the slice. If either end point does not, then the region is extended in that
direction by increments of value wuntil the end point lies outside the region. A
candidate value z′is then chosen uniformly from this region, and if it lies within the
slice, then it forms z(τ+1). If it lies outside the slice, then the region is shrunk such
thatz′forms an end point and such that the region still contains z(τ). Then another

548 11. SAMPLING METHODS
candidate point is drawn uniformly from this reduced region and so on, until a value
ofzis found that lies within the slice.
Slice sampling can be applied to multivariate distributions by repeatedly sam-
pling each variable in turn, in the manner of Gibbs sampling. This requires that
we are able to compute, for each component zi, a function that is proportional to
p(zi|z\i).
11.5. The Hybrid Monte Carlo Algorithm
As we have already noted, one of the major limitations of the Metropolis algorithm
is that it can exhibit random walk behaviour whereby the distance traversed through
the state space grows only as the square root of the number of steps. The problem
cannot be resolved simply by taking bigger steps as this leads to a high rejection rate.
In this section, we introduce a more sophisticated class of transitions based on an
analogy with physical systems and that has the property of being able to make large

changes to the system state while keeping the rejection probability small. It is ap-
plicable to distributions over continuous variables for which we can readily evaluate
the gradient of the log probability with respect to the state variables. We will discussthe dynamical systems framework in Section 11.5.1, and then in Section 11.5.2 we
explain how this may be combined with the Metropolis algorithm to yield the pow-

erful hybrid Monte Carlo algorithm. A background in physics is not required as thissection is self-contained and the key results are all derived from ﬁrst principles.
11.5.1 Dynamical systems
The dynamical approach to stochastic sampling has its origins in algorithms for
simulating the behaviour of physical systems evolving under Hamiltonian dynam-
ics. In a Markov chain Monte Carlo simulation, the goal is to sample from a given
probability distribution p(z). The framework of Hamiltonian dynamics is exploited

by casting the probabilistic simulation in the form of a Hamiltonian system. In order
to remain in keeping with the literature in this area, we make use of the relevantdynamical systems terminology where appropriate, which will be deﬁned as we go
along.
The dynamics that we consider corresponds to the evolution of the state variable
z={z
i}under continuous time, which we denote by τ. Classical dynamics is de-
scribed by Newton’s second law of motion in which the acceleration of an object is
proportional to the applied force, corresponding to a second-order differential equa-tion over time. We can decompose a second-order equation into two coupled ﬁrst-

along.
z={z
order equations by introducing intermediate momentum variables r, corresponding
to the rate of change of the state variables z, having components
r
i=dzi
dτ(11.53)
where the zican be regarded as position variables in this dynamics perspective. Thus

11.5. The Hybrid Monte Carlo Algorithm 549
for each position variable there is a corresponding momentum variable, and the joint
space of position and momentum variables is called phase space .
Without loss of generality, we can write the probability distribution p(z)in the
form
p(z)=1
Zpexp (−E(z)) (11.54)
where E(z)is interpreted as the potential energy of the system when in state z. The
system acceleration is the rate of change of momentum and is given by the appliedforce , which itself is the negative gradient of the potential energy
dr
i
dτ=−∂E(z)
∂zi. (11.55)
It is convenient to reformulate this dynamical system using the Hamiltonian
framework. To do this, we ﬁrst deﬁne the kinetic energy by
K(r)=1
2∥r∥2=1
2∑
ir2
i. (11.56)
The total energy of the system is then the sum of its potential and kinetic energies
H(z,r)=E(z)+K(r) (11.57)
where His the Hamiltonian function. Using (11.53), (11.55), (11.56), and (11.57),
we can now express the dynamics of the system in terms of the Hamiltonian equa-

dr
i
dτ=−∂E(z)
K(r)=1
2∥r∥2=1
2∑
ir2
i. (11.56)
tions given by Exercise 11.15
dzi
dτ=∂H
∂ri(11.58)
dri
dτ=−∂H
∂zi. (11.59)
William Hamilton
1805–1865
William Rowan Hamilton was an
Irish mathematician and physicist,and child prodigy, who was ap-pointed Professor of Astronomy atTrinity College, Dublin, in 1827, be-fore he had even graduated. One
of Hamilton’s most important contributions was a newformulation of dynamics, which played a signiﬁcantrole in the later development of quantum mechanics.His other great achievement was the development of
quaternions , which generalize the concept of complex

dri
dτ=−∂H
1805–1865
numbers by introducing three distinct square roots ofminus one, which satisfy i
2=j2=k2=ijk=−1.
It is said that these equations occurred to him whilewalking along the Royal Canal in Dublin with his wife,on 16 October 1843, and he promptly carved theequations into the side of Broome bridge. Although
there is no longer any evidence of the carving, there is
now a stone plaque on the bridge commemorating thediscovery and displaying the quaternion equations.

550 11. SAMPLING METHODS
During the evolution of this dynamical system, the value of the Hamiltonian His
constant, as is easily seen by differentiation
dH
dτ=∑
i{∂H
∂zidzi
dτ+∂H
∂ridri
dτ}
=∑
i{∂H
∂zi∂H
∂ri−∂H
∂ri∂H
∂zi}
=0. (11.60)
A second important property of Hamiltonian dynamical systems, known as Li-
ouville’s Theorem , is that they preserve volume in phase space. In other words, if
we consider a region within the space of variables (z,r), then as this region evolves
under the equations of Hamiltonian dynamics, its shape may change but its volume
will not. This can be seen by noting that the ﬂow ﬁeld (rate of change of location in
phase space) is given by
V=(dz
dτ,dr
dτ)
(11.61)
and that the divergence of this ﬁeld vanishes
divV=∑
i{∂
∂zidzi
dτ+∂
∂ridri
dτ}
=∑
i{
−∂
∂zi∂H
∂ri+∂
∂ri∂H
∂zi}
=0. (11.62)
Now consider the joint distribution over phase space whose total energy is the
Hamiltonian, i.e., the distribution given by
p(z,r)=1
ZHexp(−H(z,r)). (11.63)

V=(dz
dτ,dr
dτ)
(11.61)
divV=∑
i{∂
∂zidzi
dτ+∂
∂ridri
dτ}
=∑
i{
−∂
∂zi∂H
∂ri+∂
∂ri∂H
∂zi}
p(z,r)=1
Using the two results of conservation of volume and conservation of H, it follows
that the Hamiltonian dynamics will leave p(z,r)invariant. This can be seen by
considering a small region of phase space over which His approximately constant.
If we follow the evolution of the Hamiltonian equations for a ﬁnite time, then the
volume of this region will remain unchanged as will the value of Hin this region, and
hence the probability density, which is a function only of H, will also be unchanged.

Although His invariant, the values of zandrwill vary, and so by integrating
the Hamiltonian dynamics over a ﬁnite time duration it becomes possible to make
large changes to zin a systematic way that avoids random walk behaviour.
Evolution under the Hamiltonian dynamics will not, however, sample ergodi-
cally from p(z,r)because the value of His constant. In order to arrive at an ergodic
sampling scheme, we can introduce additional moves in phase space that change

the value of Hwhile also leaving the distribution p(z,r)invariant. The simplest
way to achieve this is to replace the value of rwith one drawn from its distribution
conditioned on z. This can be regarded as a Gibbs sampling step, and hence from

11.5. The Hybrid Monte Carlo Algorithm 551
Section 11.3 we see that this also leaves the desired distribution invariant. Noting
thatzandrare independent in the distribution p(z,r), we see that the conditional
distribution p(r|z)is a Gaussian from which it is straightforward to sample. Exercise 11.16
In a practical application of this approach, we have to address the problem of
performing a numerical integration of the Hamiltonian equations. This will neces-sarily introduce numerical errors and so we should devise a scheme that minimizes
the impact of such errors. In fact, it turns out that integration schemes can be devised
for which Liouville’s theorem still holds exactly. This property will be important inthe hybrid Monte Carlo algorithm, which is discussed in Section 11.5.2. One scheme
for achieving this is called the leapfrog discretization and involves alternately updat-
ing discrete-time approximations
ˆzandˆrto the position and momentum variables
using
ˆri(τ+ϵ/2) =ˆri(τ)−ϵ
2∂E
∂zi(ˆz(τ)) (11.64)

using
2∂E
ˆzi(τ+ϵ)=ˆzi(τ)+ϵˆri(τ+ϵ/2) (11.65)
ˆri(τ+ϵ)=ˆri(τ+ϵ/2)−ϵ
2∂E
∂zi(ˆz(τ+ϵ)). (11.66)
We see that this takes the form of a half-step update of the momentum variables with
step size ϵ/2, followed by a full-step update of the position variables with step size ϵ,
followed by a second half-step update of the momentum variables. If several leapfrog
steps are applied in succession, it can be seen that half-step updates to the momentumvariables can be combined into full-step updates with step size ϵ. The successive

2∂E
updates to position and momentum variables then leapfrog over each other. In order
to advance the dynamics by a time interval τ, we need to take τ/ϵsteps. The error
involved in the discretized approximation to the continuous time dynamics will go to
zero, assuming a smooth function E(z), in the limit ϵ→0. However, for a nonzero
ϵas used in practice, some residual error will remain. We shall see in Section 11.5.2
how the effects of such errors can be eliminated in the hybrid Monte Carlo algorithm.

In summary then, the Hamiltonian dynamical approach involves alternating be-
tween a series of leapfrog updates and a resampling of the momentum variables fromtheir marginal distribution.
Note that the Hamiltonian dynamics method, unlike the basic Metropolis algo-
rithm, is able to make use of information about the gradient of the log probability
distribution as well as about the distribution itself. An analogous situation is familiar

from the domain of function optimization. In most cases where gradient informa-tion is available, it is highly advantageous to make use of it. Informally, this follows
from the fact that in a space of dimension D, the additional computational cost of
evaluating a gradient compared with evaluating the function itself will typically be aﬁxed factor independent of D, whereas the D-dimensional gradient vector conveys
Dpieces of information compared with the one piece of information given by the
function itself.

552 11. SAMPLING METHODS
11.5.2 Hybrid Monte Carlo
As we discussed in the previous section, for a nonzero step size ϵ, the discretiza-
tion of the leapfrog algorithm will introduce errors into the integration of the Hamil-
tonian dynamical equations. Hybrid Monte Carlo (Duane et al. , 1987; Neal, 1996)
combines Hamiltonian dynamics with the Metropolis algorithm and thereby removesany bias associated with the discretization.
Speciﬁcally, the algorithm uses a Markov chain consisting of alternate stochastic
updates of the momentum variable rand Hamiltonian dynamical updates using the
leapfrog algorithm. After each application of the leapfrog algorithm, the resulting
candidate state is accepted or rejected according to the Metropolis criterion basedon the value of the Hamiltonian H. Thus if (z,r)is the initial state and (z
⋆,r⋆)
is the state after the leapfrog integration, then this candidate state is accepted with
probability
min(1 ,exp{H(z,r)−H(z⋆,r⋆)}). (11.67)

⋆,r⋆)
If the leapfrog integration were to simulate the Hamiltonian dynamics perfectly,
then every such candidate step would automatically be accepted because the value
ofHwould be unchanged. Due to numerical errors, the value of Hmay sometimes
decrease, and we would like the Metropolis criterion to remove any bias due to thiseffect and ensure that the resulting samples are indeed drawn from the required dis-
tribution. In order for this to be the case, we need to ensure that the update equations

corresponding to the leapfrog integration satisfy detailed balance (11.40). This iseasily achieved by modifying the leapfrog scheme as follows.
Before the start of each leapfrog integration sequence, we choose at random,
with equal probability, whether to integrate forwards in time (using step size ϵ)o r
backwards in time (using step size −ϵ). We ﬁrst note that the leapfrog integration
scheme (11.64), (11.65), and (11.66) is time-reversible, so that integration for Lsteps

using step size −ϵwill exactly undo the effect of integration for Lsteps using step
sizeϵ. Next we show that the leapfrog integration preserves phase-space volume
exactly. This follows from the fact that each step in the leapfrog scheme updates
either a z
ivariable or an rivariable by an amount that is a function only of the other
variable. As shown in Figure 11.14, this has the effect of shearing a region of phase
space while not altering its volume.
Finally, we use these results to show that detailed balance holds. Consider a

either a z
small region Rof phase space that, under a sequence of Lleapfrog iterations of
step size ϵ, maps to a region R′. Using conservation of volume under the leapfrog
iteration, we see that if Rhas volume δVthen so too will R′. If we choose an initial
point from the distribution (11.63) and then update it using Lleapfrog interactions,
the probability of the transition going from RtoR′is given by
1
ZHexp(−H(R))δV1
2min{1,exp(−H(R)+H(R′))}. (11.68)
where the factor of 1/2arises from the probability of choosing to integrate with a

1
positive step size rather than a negative one. Similarly, the probability of starting in

11.5. The Hybrid Monte Carlo Algorithm 553
ri
zir′
i
z′
i
Figure 11.14 Each step of the leapfrog algorithm (11.64)–(11.66) modiﬁes either a position variable zior a
momentum variable ri. Because the change to one variable is a function only of the other, any region in phase
space will be sheared without change of volume.
region R′and integrating backwards in time to end up in region Ris given by
1
ZHexp(−H(R′))δV1
2min{1,exp(−H(R′)+H(R))}. (11.69)
It is easily seen that the two probabilities (11.68) and (11.69) are equal, and hence
detailed balance holds. Note that this proof ignores any overlap between the regions Exercise 11.17
RandR′but is easily generalized to allow for such overlap.
It is not difﬁcult to construct examples for which the leapfrog algorithm returns
to its starting position after a ﬁnite number of iterations. In such cases, the random
replacement of the momentum values before each leapfrog integration will not be

sufﬁcient to ensure ergodicity because the position variables will never be updated.
Such phenomena are easily avoided by choosing the magnitude of the step size at
random from some small interval, before each leapfrog integration.
We can gain some insight into the behaviour of the hybrid Monte Carlo algo-
rithm by considering its application to a multivariate Gaussian. For convenience,
consider a Gaussian distribution p(z)with independent components, for which the
Hamiltonian is given by
H(z,r)=1
2∑
i1
σ2
iz2
i+1
2∑
ir2

H(z,r)=1
2∑
i1
σ2
iz2
i+1
2∑
ir2
i. (11.70)
Our conclusions will be equally valid for a Gaussian distribution having correlated
components because the hybrid Monte Carlo algorithm exhibits rotational isotropy.
During the leapfrog integration, each pair of phase-space variables zi,rievolves in-
dependently. However, the acceptance or rejection of the candidate point is based
on the value of H, which depends on the values of all of the variables. Thus, a
signiﬁcant integration error in any one of the variables could lead to a high prob-

ir2
i. (11.70)
ability of rejection. In order that the discrete leapfrog integration be a reasonably

554 11. SAMPLING METHODS
good approximation to the true continuous-time dynamics, it is necessary for the
leapfrog integration scale ϵto be smaller than the shortest length-scale over which
the potential is varying signiﬁcantly. This is governed by the smallest value of σi,
which we denote by σmin. Recall that the goal of the leapfrog integration in hybrid
Monte Carlo is to move a substantial distance through phase space to a new statethat is relatively independent of the initial state and still achieve a high probability of
acceptance. In order to achieve this, the leapfrog integration must be continued for a
number of iterations of order σ
max/σmin.
By contrast, consider the behaviour of a simple Metropolis algorithm with an
isotropic Gaussian proposal distribution of variance s2, considered earlier. In order
to avoid high rejection rates, the value of smust be of order σmin. The exploration of
state space then proceeds by a random walk and takes of order (σmax/σmin)2steps

max/σmin.
to arrive at a roughly independent state.
11.6. Estimating the Partition Function
As we have seen, most of the sampling algorithms considered in this chapter re-quire only the functional form of the probability distribution up to a multiplicativeconstant. Thus if we write
p
E(z)=1
ZEexp(−E(z)) (11.71)
then the value of the normalization constant ZE, also known as the partition func-
tion, is not needed in order to draw samples from p(z). However, knowledge of the
value of ZEcan be useful for Bayesian model comparison since it represents the

p
E(z)=1
model evidence (i.e., the probability of the observed data given the model), and so
it is of interest to consider how its value might be obtained. We assume that direct
evaluation by summing, or integrating, the function exp(−E(z))over the state space
ofzis intractable.
For model comparison, it is actually the ratio of the partition functions for two

models that is required. Multiplication of this ratio by the ratio of prior probabilitiesgives the ratio of posterior probabilities, which can then be used for model selection
or model averaging.
One way to estimate a ratio of partition functions is to use importance sampling
from a distribution with energy function G(z)
Z
E
ZG=∑
zexp(−E(z))∑
zexp(−G(z))
=∑
zexp(−E(z)+G(z)) exp( −G(z))∑
= EG(z)[exp(−E+G)]
≃∑
lexp(−E(z(l))+G(z(l))) (11.72)

11.6. Estimating the Partition Function 555
where {z(l)}are samples drawn from the distribution deﬁned by pG(z). If the dis-
tribution pGis one for which the partition function can be evaluated analytically, for
example a Gaussian, then the absolute value of ZEcan be obtained.
This approach will only yield accurate results if the importance sampling distri-
bution pGis closely matched to the distribution pE, so that the ratio pE/pGdoes not
have wide variations. In practice, suitable analytically speciﬁed importance sampling
distributions cannot readily be found for the kinds of complex models considered in
this book.
An alternative approach is therefore to use the samples obtained from a Markov
chain to deﬁne the importance-sampling distribution. If the transition probability for
the Markov chain is given by T(z,z′), and the sample set is given by z(1),...,z(L),
then the sampling distribution can be written as
1
ZGexp (−G(z)) =L∑
l=1T(z(l),z) (11.73)
which can be used directly in (11.72).

this book.
1
Methods for estimating the ratio of two partition functions require for their suc-
cess that the two corresponding distributions be reasonably closely matched. This isespecially problematic if we wish to ﬁnd the absolute value of the partition function
for a complex distribution because it is only for relatively simple distributions that
the partition function can be evaluated directly, and so attempting to estimate the
ratio of partition functions directly is unlikely to be successful. This problem can be

tackled using a technique known as chaining (Neal, 1993; Barber and Bishop, 1997),
which involves introducing a succession of intermediate distributions p
2,...,p M−1
that interpolate between a simple distribution p1(z)for which we can evaluate the
normalization coefﬁcient Z1and the desired complex distribution pM(z). We then
haveZM
Z1=Z2
Z1Z3
Z2···ZM
ZM−1(11.74)
in which the intermediate ratios can be determined using Monte Carlo methods as

haveZM
Z1=Z2
Z1Z3
Z2···ZM
discussed above. One way to construct such a sequence of intermediate systems
is to use an energy function containing a continuous parameter 0⩽α⩽1that
interpolates between the two distributions
Eα(z)=( 1 −α)E1(z)+αEM(z). (11.75)
If the intermediate ratios in (11.74) are to be found using Monte Carlo, it may be
more efﬁcient to use a single Markov chain run than to restart the Markov chain foreach ratio. In this case, the Markov chain is run initially for the system p
1and then
after some suitable number of steps moves on to the next distribution in the sequence.

1and then
Note, however, that the system must remain close to the equilibrium distribution ateach stage.

556 11. SAMPLING METHODS
Exercises
11.1 (⋆)www Show that the ﬁnite sample estimator ˆfdeﬁned by (11.2) has mean
equal to E[f]and variance given by (11.3).
11.2 (⋆)Suppose that zis a random variable with uniform distribution over (0,1)and
that we transform zusingy=h−1(z)where h(y)is given by (11.6). Show that y
has the distribution p(y).
11.3 (⋆)Given a random variable zthat is uniformly distributed over (0,1), ﬁnd a trans-
formation y=f(z)such that yhas a Cauchy distribution given by (11.8).
11.4 (⋆⋆)Suppose that z1andz2are uniformly distributed over the unit circle, as
shown in Figure 11.3, and that we make the change of variables given by (11.10)
and (11.11). Show that (y1,y2)will be distributed according to (11.12).
11.5 (⋆)www Letzbe aD-dimensional random variable having a Gaussian distribu-
tion with zero mean and unit covariance matrix, and suppose that the positive deﬁnite
symmetric matrix Σhas the Cholesky decomposition Σ=LLTwhereLis a lower-

triangular matrix (i.e., one with zeros above the leading diagonal). Show that thevariable y=µ+Lzhas a Gaussian distribution with mean µand covariance Σ.
This provides a technique for generating samples from a general multivariate Gaus-
sian using samples from a univariate Gaussian having zero mean and unit variance.
11.6 (⋆⋆)
www In this exercise, we show more carefully that rejection sampling does
indeed draw samples from the desired distribution p(z). Suppose the proposal dis-

11.6 (⋆⋆)
tribution is q(z)and show that the probability of a sample value zbeing accepted is
given by˜p(z)/kq(z)where˜pis any unnormalized distribution that is proportional to
p(z), and the constant kis set to the smallest value that ensures kq(z)⩾˜p(z)for all
values of z. Note that the probability of drawing a value zis given by the probability
of drawing that value from q(z)times the probability of accepting that value given
that it has been drawn. Make use of this, along with the sum and product rules of

probability, to write down the normalized form for the distribution over z, and show
that it equals p(z).
11.7 (⋆)Suppose that zhas a uniform distribution over the interval [0,1]. Show that the
variable y=btanz+chas a Cauchy distribution given by (11.16).
11.8 (⋆⋆)Determine expressions for the coefﬁcients kiin the envelope distribution
(11.17) for adaptive rejection sampling using the requirements of continuity and nor-malization.
11.9 (⋆⋆)By making use of the technique discussed in Section 11.1.1 for sampling

from a single exponential distribution, devise an algorithm for sampling from the
piecewise exponential distribution deﬁned by (11.17).
11.10 (⋆)Show that the simple random walk over the integers deﬁned by (11.34), (11.35),
and (11.36) has the property that E[(z
(τ))2]= E[(z(τ−1))2]+1/2and hence by
induction that E[(z(τ))2]=τ/2.

Exercises 557
Figure 11.15 A probability distribution over two variables z1
andz2that is uniform over the shaded regions
and that is zero everywhere else.
z1z2
11.11 (⋆⋆)www Show that the Gibbs sampling algorithm, discussed in Section 11.3,
satisﬁes detailed balance as deﬁned by (11.40).
11.12 (⋆)Consider the distribution shown in Figure 11.15. Discuss whether the standard
Gibbs sampling procedure for this distribution is ergodic, and therefore whether it
would sample correctly from this distribution
11.13 (⋆⋆)Consider the simple 3-node graph shown in Figure 11.16 in which the observed
nodexis given by a Gaussian distribution N(x|µ, τ−1)with mean µand precision
τ. Suppose that the marginal distributions over the mean and precision are given
byN(µ|µ0,s0)andGam( τ|a, b), where Gam(·|·,·)denotes a gamma distribution.
Write down expressions for the conditional distributions p(µ|x, τ)andp(τ|x, µ)that
would be required in order to apply Gibbs sampling to the posterior distribution
p(µ, τ|x).

p(µ, τ|x).
11.14 (⋆)Verify that the over-relaxation update (11.50), in which zihas mean µiand
variance σi, and where νhas zero mean and unit variance, gives a value z′
iwith
meanµiand variance σ2
i.
11.15 (⋆)www Using (11.56) and (11.57), show that the Hamiltonian equation (11.58)
is equivalent to (11.53). Similarly, using (11.57) show that (11.59) is equivalent to
(11.55).
11.16 (⋆)By making use of (11.56), (11.57), and (11.63), show that the conditional dis-
tribution p(r|z)is a Gaussian.

p(µ, τ|x).
iwith
i.
(11.55).
Figure 11.16 A graph involving an observed Gaussian variable xwith
prior distributions over its mean µand precision τ.µ τ
x

558 11. SAMPLING METHODS
11.17 (⋆)www Verify that the two probabilities (11.68) and (11.69) are equal, and hence
that detailed balance holds for the hybrid Monte Carlo algorithm.

Appendix AInChapter9,wediscussed probabilistic modelshavingdiscretelatentvariables, such
asthemixtureofGaussians. Wenowexploremodelsinwhichsome,orall,ofthe
latentvariables arecontinuous. Animportant motivation forsuchmodelsisthat
manydatasetshavetheproperty thatthedatapointsalllieclosetoamanifold of
muchlowerdimensionality thanthatoftheoriginaldataspace.Toseewhythis
mightarise,consider anartificialdatasetconstructed bytakingoneoftheoff-line
digits,represented bya64x64pixelgrey-level image, and embedding itinalarger
imageofsize100x100bypaddingwithpixelshavingthevaluezero(corresponding
towhite pixels) inwhichthelocationandorientation ofthedigitisvariedatrandom,
asillustrated inFigure12.1.Eachoftheresulting imagesisrepresented byapointin
the100x100=10,OOO-dimensional dataspace.However, acrossadatasetofsuch
images,thereareonlythreedegreesoffreedomofvariability, corresponding tothe
verticalandhorizontal translations andtherotations. Thedatapointswilltherefore

liveonasubspace ofthedataspacewhoseintrinsic dimensionality isthree.Note
559

560 12.CONTINUOUS LATENT VARIABLES
Figure12.1Asynthetic dataselobtained bytakingoneoftheoff-linedigitimagesandcreatingmulti­
plecopiesineachofwhichthedigithasundergone arandomdisplacement androtation
withinsomelargerimagefield.Theresulting imageseachhave100)(100=10.000
pixels.
thatthemanifold willbenonlinear because. forinstance. ifwetranslate thedigit
pastaparticular pixel,thatpixelvaluewillgofromzero(white)10one(black)and
backtozeroagain.whichisclearlyanonlinear functionofthedigitposition. In
thisexample. !.helranslation androtationparameters arelatentvariables becausewe
observeonlytheimagevectorsandarenottoldwhichvaluesofthetranslation or
rotationvariables wereusedtocreatethem.
Forrealdigitimagedata,therewillbeafunherdegreeoffreedom arisingfrom
scaling. Moreover therewillbemultiple addilional degreesoffreedom associaled
wilhmorecomplex deformations duetothevariability inanindividual's wriling
3Swellaslhedifferences inwritingslylesbetween individuals. evenheless. the

numberofsuchdegreesoffreedom willbesmallcompared tothedimensionality of
Ihedataset.
AppendiX A Another example isprovided bytheoilflowdataset.inwhich(foragivenge-
ometrical configuration ofthegas,WOller,andoilphases)thereareonlytwodegrees
offreedomofvariability corresponding tothefractionofoilinthepipeandthefrac­
tionofwater(thefractionofgasIhenbeingdetermined). Ahhough thedataspace
comprises 12measuremenlS, adatasetofpointswilllieclosetoaIwo-dimensional
manifold embedded withinthisspace.Inthiscase,themanifold comprises scveral

distinctsegments corresponding todifferent flowregimes. eachsuchsegment being
a(noisy)continuous two-dimensional manifold. Ifourgoalisdatacompression. or
densitymodelling, thentherecanbebenefitsinexploiling thismanifold struclUre.
Inpraclice. thedatapointswillnotbeconfined precisely toasmoothlow­
dimensional manifold, andwecaninterpret thedepartures ofdatapointsfromthe
manifold as·noise'. Thisleadsnaturally toagenerative viewofsuchmodelsin
whichwefirstselectapoinlwithinthemanifold according tosomelatentvariable

distribution andthengenerate anobserved datapointby:lddingnoise,drawnfrom
someconditional distribution ofthedatavarillbles giventhelatentvarillbles.
Thcsimplest continuous latentvariable modelassumes Gaussian distributions
forboththclatentandobserved variables andmakesuseofalinear,Gaussian de-
SeCTion8.1..J pendence oftheobserved variables onIheslateofthelatentvariables. Thisleads
toaprobabilislic fonnulation ofthewell-known technique ofprincipal component
analysis(PeA),aswellas10arelatedmodelcalledfactoranalysis.

Section12.1 Inthischapterwwillbeginwilhaslandard, nonprobabilistic treatment ofPeA.
andthcnweshowhowPeAarisesnaturally asthemaximum likelihood solution 10

12.1.Principal C01n[>OM"1 AnalJsis 561
Flgu,e12.2 P'if>cipal compooont a",,~"seeks" $pace
01!owe,dimensionality. kt"(>WIlas!heP<lno>
palsubSpace "nddenoted I:Jythemagenta "1
line.SUChItlettheGrthogonet [jiojectiOh 01
!hedatapoints('eddoIslontotP'Ns~
"""'imizes thevaria,..,.,of!heproja<:ted points
(greendoIs).An"It",nati"" ....finilion01PCA
isbasedonm..mizingthe""",-<>I·squares
of!heprojection errors.ind'cated bythebfi.>e
lines.
S'crio"12.2 apanlcula, fonnoflinear-Gau"ian latem"ariablemodel.Thisprobabilistic refor­
mulation bring~many ad\'imlag~s, su~hastl>l:useI)fEMforparameter eslimalion,
rrinciple<J c~tensioos 10Oli~turc, ofPeAmodel"andBa)'~sian formulat;ons that
allowtbenumberofrrincipal com[>Oncnts tobedetennined aUlOmatically from!be
data.Finally'."cdiSl;us<briefly""'eralgencrali,ation, ofthelatentYariableconcept
thatg<lbe~oodtbelinear-Gaussian assumption including non·Gau"i"n I.tcntyari­
abies.....hichlea'"totbefr.me....orkofindrl"'mJ.m compon.nl anal,-.;., as....ella,

S'di""12.4 modelsha"inganonlinear rclationship bet....eenlatentandoose",e<J ,'lUiable,.
____ 'c2=.~1.Principal Component Analysis
Principal compooem analy,;" orrcA.;satechnique tha!is"'idelyu<edforappli.
cationssuchasdimensionality .-eduction, lossydatacomprc"ion, featuree>tracti"".
anddatav;,ualizatiOll (Jolliffe, 2(02).It;salsokno...."astileKaroan.n·I ..,;""tran,·
f~.
lbcrcan:t....ocommonly useddefinitions ofPeAthatgiyerisetothe>arne
algorithm. PeAcanbedefinedastheunhog<lnal projtttion ofthedataO/1tOalo....er

f~.
dimensionallincar space.kno....nasthepri/lcip.al $uh.•p.aa.sochthatthe\'ariance of
theprojttted datai'ma~imi,e<J (1I",.lIing. 1933).Equi"alemly,;t canbedefinedas
tbelinearprojection thatminimi"'. theaverageprojttlion cost.definedast~mean
squa.-eddistance!letween thedata[>Oint<andtbeirp<ojtttioo, (Pearson, 19(1).The
l"J'"OC"s<ofonhogonal projection i'illustraled inFiguTe12.2.Wecon,ider eachof
thesedefinitions intum.
12,1.1 Mllximllm variance lormulation
Con,ider adalaset<Ifobser"\lations {x,,}where" =1.....S,andx"i,a

Euclidean variable "'ilhdimen,ionality D.Ourgoalistoproject If>/::dataontoa
'paceha"ingdimen,ionality M<D"hileIll3Jli",i,illg the"ariallCeoftheprojttted
data.Forthe!noll..nl.we'hallassumethattbe"alueofMisg;\·en.Latcrinthis

562 12.CONTINUOUS LATENT VARIABLES
chapter,weshallconsider techniques todetermine anappropriate valueofIV!from
thedata.
Tobeginwith,consider theprojection ontoaone-dimensional space(M=1).
Wecandefinethedirection ofthisspaceusingaD-dimensional vectorUl,which
forconvenience (andwithoutlossofgenerality) weshallchoosetobeaunitvector
sothatufUl=1(notethatweareonlyinterested inthedirection definedbyUl,
notinthemagnitude ofUlitself).EachdatapointXnisthenprojected ontoascalar
valueufXn.Themeanoftheprojected dataisufxwherexisthesamplesetmean
givenby
(12.1)
andthevarianceoftheprojected dataisgivenby
whereSisthedatacovariance matrixdefinedby
1N
S= -"(xn-x)(xn-x)TNLJ .
n=l(12.2)
(12.3)
Appendix EWenowmaximize theprojected varianceUfSUlwithrespecttoUl.Clearly,thishas
tobeaconstrained maximization topreventIlulll.....00.Theappropriate constraint
comesfromthenormalization condition ufUl=1.Toenforcethisconstraint,
weintroduce aLagrange multiplier thatweshalldenotebyAI,andthenmakean
unconstrained maximization of
(12.4)

givenby
(12.1)
1N
n=l(12.2)
(12.3)
(12.4)
Bysettingthederivative withrespecttoUlequaltozero,weseethatthisquantity
willhaveastationary pointwhen
(12.5)
whichsaysthatUlmustbeaneigenvector ofS.Ifweleft-multiply byufandmake
useofufUl=1,weseethatthevariance isgivenby
(12.6)
andsothevariance willbeamaximum whenwesetUlequaltotheeigenvector
havingthelargesteigenvalue AI.Thiseigenvector isknownasthefirstprincipal
Wecandefineadditional principal components inanincremental fashionby
choosing eachnewdirection tobethatwhichmaximizes theprojected variance

Exercise 12.1
Section12.2.2
Appendix C12.1.Principal Component Analysis 563
amongst allpossible directions orthogonal tothosealreadyconsidered. Ifwecon­
siderthegeneralcaseofanM-dimensional projection space,theoptimallinearpro­
jectionforwhichthevarianceoftheprojected dataismaximized isnowdefinedby
theMeigenvectors U1,..., UMofthedatacovariance matrixScorresponding tothe
Mlargesteigenvalues >'1,...,AM.Thisiseasilyshownusingproofbyinduction.
Tosummarize, principal component analysis involves evaluating themeanx
andthecovariance matrixSofthedatasetandthenfindingtheMeigenvectors ofS
corresponding totheMlargesteigenvalues. Algorithms forfindingeigenvectors and
eigenvalues, aswellasadditional theorems relatedtoeigenvector decomposition,
canbefoundinGolubandVanLoan(1996). Notethatthecomputational costof
computing thefulleigenvector decomposition foramatrixofsizeDxDisO(D3).
IfweplantoprojectourdataontothefirstMprincipal components, thenweonly
needtofindthefirstMeigenvalues andeigenvectors. Thiscanbedonewithmore

efficient techniques, suchasthepowermethod(GolubandVanLoan,1996),that
scalelikeO(MD2),oralternatively wecanmakeuseoftheEMalgorithm.
12.1.2Minimum-error formulation
Wenowdiscussanalternative formulation ofpeAbasedonprojection error
minimization. Todothis,weintroduce acomplete orthonormal setofD-dimensional
basisvectors{Ui}wherei=1,...,Dthatsatisfy
(12.7)
Because thisbasisiscomplete, eachdatapointcanberepresented exactlybyalinear
combination ofthebasisvectors
D
Xn=Laniui
i=l(12.8)

(12.7)
D
Xn=Laniui
i=l(12.8)
wherethecoefficients aniwillbedifferent fordifferent datapoints.Thissimply
corresponds toarotationofthecoordinate systemtoanewsystemdefinedbythe
{Ui},andtheoriginalDcomponents {Xnl'...,XnD}arereplaced byanequivalent
set{anl'...,anD}. TakingtheinnerproductwithUj,andmakinguseoftheor­
thonormality property, weobtainanj=x;Uj,andsowithoutlossofgenerality we
canwrite
D
Xn=L(X~Ui)Ui·
i=l(12.9)
(12.10)Ourgoal,however, istoapproximate thisdatapointusingarepresentation in­

D
Xn=Laniui
i=l(12.8)
canwrite
D
i=l(12.9)
volvingarestricted numberM<Dofvariables corresponding toaprojection onto
alower-dimensional subspace. TheM-dimensional linearsubspace canberepre­
sented,withoutlossofgenerality, bythefirstMofthebasisvectors, andsowe
approximate eachdatapointXnby
M D
xn=LZniUi+LbiUi
i=l i=M+l

564 12.CONTINUOUS LATENT VARIABLES
wherethe{Zni}dependontheparticular datapoint,whereasthe{bdareconstants
thatarethesameforalldatapoints.Wearefreetochoosethe{Ui},the{Zni},and
the{bdsoastominimize thedistortion introduced bythereduction indimensional­
ity.Asourdistortion measure, weshallusethesquareddistancebetweentheoriginal
datapointXnanditsapproximation Xn,averaged overthedataset,sothatourgoal
istominimize
N
J=~LIlxn -xn112
.
n=l(12.11)
Consider firstofalltheminimization withrespecttothequantities {Zni}.Sub­
stitutingforXn,settingthederivative withrespecttoZnjtozero,andmakinguseof
theorthonormality conditions, weobtain
(12.12)
wherej=1,...,M.Similarly, settingthederivative ofJwithrespecttobitozero,
andagainmakinguseoftheorthonormality relations, gives
b-Tj=XUj (12.13)
wherej=M+1,...,D.Ifwesubstitute forZniandbi,andmakeuseofthegeneral
expansion (12.9),weobtain
D
Xn-Xn=L{(Xn-x)TudUi
i=M+l(12.14)
fromwhichweseethatthedisplacement vectorfromXntoxnliesinthespace

(12.12)
D
orthogonal totheprincipal subspace, becauseitisalinearcombination of{udfor
i=M+1,...,D,asillustrated inFigure12.2.Thisistobeexpected becausethe
projected pointsxnmustliewithintheprincipal subspace, butwecanmovethem
freelywithinthatsubspace, andsotheminimum errorisgivenbytheorthogonal
projection.
Wetherefore obtainanexpression for thedistortion measureJasafunction
purelyofthe{udintheform
1~~(T_T)2 DTJ=NLL XnUi- XUi=LUiSUi.
n=li=M+l i=M+l(12.15)
Thereremainsthetaskofminimizing Jwithrespecttothe{Ui},whichmust

beaconstrained minimization otherwise wewillobtainthevacuousresultUi=O.
Theconstraints arisefromtheorthonormality conditions and,asweshallsee,the
solutionwillbeexpressed intermsoftheeigenvector expansion ofthecovariance
matrix.Beforeconsidering aformalsolution, letustrytoobtainsomeintuitionabout
theresultbyconsidering thecaseofatwo-dimensional dataspaceD=2andaone­
dimensional principal subspace M=1.Wehavetochooseadirection U2soasto

12.1.Principal Component Analysis 565
minimize J=UISU2'subjecttothenormalization constraint uIU2=1.Usinga
Lagrange multiplier A2toenforcetheconstraint, weconsider theminimization of
(12.16)
(12.18)Exercise 12.2
Appendix ASettingthederivative withrespecttoU2tozero,weobtainSU2=A2U2sothatU2
isaneigenvector ofSwitheigenvalue A2.Thusanyeigenvector willdefineasta­
tionarypointofthedistortion measure. TofindthevalueofJattheminimum, we
back-substitute thesolutionforU2intothedistortion measure togiveJ=A2.We
therefore obtaintheminimum valueofJbychoosing U2tobetheeigenvector corre­
sponding tothesmallerofthetwoeigenvalues. Thusweshouldchoosetheprincipal
subspace tobealignedwiththeeigenvector havingthelargereigenvalue. Thisresult
accordswithourintuition that,inordertominimize theaveragesquaredprojection
distance, weshouldchoosetheprincipal component subspace topassthroughthe
meanofthedatapointsandtobealignedwiththedirections ofmaximum variance.
Forthecasewhentheeigenvalues areequal,anychoiceofprincipal direction will

giverisetothesamevalueofJ.
Thegeneralsolutiontotheminimization ofJforarbitrary DandarbitraryM<
Disobtainedbychoosing the{Ui}tobeeigenvectors ofthecovariance matrixgiven
by
SUi=AiUi (12.17)
wherei=1,...,D,andasusualtheeigenvectors {Ui}arechosentobeorthonor­
mal.Thecorresponding valueofthedistortion measureisthengivenby
D
J=LAi
i=M+l
whichissimplythesumoftheeigenvalues ofthoseeigenvectors thatareorthogonal
totheprincipal subspace. Wetherefore obtaintheminimum valueofJbyselecting
theseeigenvectors tobethosehavingtheD-Msmallesteigenvalues, andhence

by
D
J=LAi
i=M+l
theeigenvectors definingtheprincipal subspace arethosecorresponding totheM
largesteigenvalues.
Although wehaveconsidered M<D,thePCAanalysisstillholdsifM=
D,inwhichcasethereisnodimensionality reduction butsimplyarotationofthe
coordinate axestoalignwithprincipal components.
Finally,itisworthnotingthatthereexistsacloselyrelatedlineardimensionality
reduction technique calledcanonical correlation analysis, orCCA(Hotelling, 1936;
BachandJordan,2002).Whereas PCAworkswithasinglerandomvariable, CCA
considers two(ormore)variables andtriestofindacorresponding pairoflinear

subspaces thathavehigh cross-correlation, sothateachcomponent withinoneofthe
subspaces iscorrelated withasinglecomponent fromtheothersubspace. Itssolution
canbeexpressed intermsofageneralized eigenvector problem.
12.1.3Applications ofpeA
Wecanillustrate theuseofPCAfordatacompression byconsidering theoff­
linedigitsdataset.Because eacheigenvector ofthecovariance matrixisavector

566 12.COl\'TINUOliS LATf;I\'T \'ARIAIILES
Figure12.3Themean~'"xaklogwith!heII"'tlou'PCAe;gerrvecl<)rll Ul,...'"lorthe011-.....
cligitsdataset.t<>getl'ler with!hecorrespondi~ ~.
;ntheOIigi",,1D-<limensional space.wecanrepresent thoeigenw:cto<s asimago<of
thosamesilOas,1>0datapoi",,_11,.firstIh'e.ig.n,'occOfS. alongwichtl>ocorre­
sponding .igen,'slue,. are<IIo"'ninFigure12,3,AplO!ofll>ocomplete spect"'m uf
oigo",·alue,. sone<!intodecreasing order.isshowninFigure12.4{ai.Thedi'tortion
measureJaSSQCiated wilhchoo<ing aparticular valueofMisgi.'enbythosum
oftheeig.n",lues fromM+Iupto0andisptO!tedfordifferent ,'aluo<of.\1in
Figure12,4(b).
If"'e<utlslitut. (12,12)and(12.13)into(12.10).wecanwritetheI'CA appro~­
imationtoadata"eel'" x~i"thefonn
M"'-~L{x~",)u,+I:(xl'u,)u, (12.19).-. ._M+l
M-x+L(X~U,-XTU,)U; (12.20)
•,
,to' ,10',, ,
",,
"- ,""", ~ ~; 0", ~ ~" ,., ,.,
FIIIUre12,4(a)PIolat!heeJoI;nv.loo .".,etrum lortheoff·1inedigitsdataset(b)P10t01!hesumatthe

•,
",,
<:liscarded."".Ioos, which"'l'fesoots!he s.um-ol·SQ"",es distortlon Ji<*~ byprojecti<Xl thedataonto
ap<incipalcomponenl slll>spaee '"dimensionalitv M.

11.1.I'rindpall.:"l11pon~nt Anal~·.i. 567
FIIIUr.1:1:.5An",>gi",,1 ~mpIeIromlI>e011·_digilsdata...ttOll"1her withitsPeAre<:onstnxlions
oblair...:! by'e1aio"li!Xl ,Ifj)<incipal ~n1S 10<variousval,,"01,If.As,IIincreason
!tiere<:onst,uctiOfI ~s moreao::urate andwoukl~portee!when.-IfKD~
28x28~."-1.
whereweha"emademoeoftherelation
"x=L(x'",)u;,-,(12.21)
AI''''''''/;'\' A
Seer/on9.1whichfollow.fromthecompletene" ofthe{u,I,Thi.represent. acontpre"ioo
"fthedata>ct.Ilttau>eforeachdatapoimweha,..repla«d theV·dimensiooal
"o<:lorx"Wilhan,I[.din>en,ional "o<:torhavingcomponem, (x~'"_X'",).11Ie
'mailerthe"alueofM.thegreaterthedegreeofcomp.-e",ion. Example. ofPeA
,""on't""tioos ofdatapointsforthedigitsdatasetareshowninFigure12.5
Anolherapplication ofpriocipal compcmenl analy,i.i'todatapre-processing.
Inthi'case,lhegoalisnO!dimensionality redUC1ion butratherthetmn,formmion of
adataselinor<k'tostanda'lli'.e eenainofilSpmpenies. Thiscanbein'portanl in

allowing .ubsequent pallem,""ognition algorithm. 10beappliedsuccessfully 10the
data>ct.Typically. ilisdonewilentheoriginal"ariable. aremea,ured in"arioosdif.
ferentunil'or!la"esignificantly difTerent ,'ariabilil}'. Forinstance intheOldFaithful
datasel.thetimebetv.-een eruption. i.typicany anorderofmagni1Ude greaterthan
lhedUrali""of.nerupt;,,". WhenW'eappliedthe".nlCans algorill"" 10thi<data
set,".-efirstmadeasepar.te linearre-sealing oftheindividual "anable' socbthm

each"ariablehadzeromeanandunit"ariance. llUsisknownasslllNlardiv·.,g the
dota.andthecO\'anance matrixforlhe'lando,di/,ed dalahascomponents
(12,22)
where <1,isthe,'anaoceof:c,.Thisi<knownasthe(",,,el,,,;,,,, matri.'oftheoriginal
dotaandha'thepropeny thaiift""orompooent, X;andx,ofthedataareperfee1ly
correl.ted. thenAi_I.•ndiftheya.-euocorrelated. thenAi_O.
11",,'1""', usingPeAwecanmakeaIt>Of'esubst.mial nonnalizat;oo ofthedata
togi\'Citzeromeanandunitco'·ariance. sothatdifferent "anables becomederorre­

(12,22)
late<lTodothis.wefirst""riletheei8Cn"cclor equation (12,17)intheform
su=UL (12.23)

568 12.CONTINUOUS LATENT VARIABLES
1002 2
9000'BO
80000
00
70 0 080 0
,=~o0cPO0 0 tj~
60O~ 50~OOID-2 -240
2 4 6 -2 0 2 -2 0 2
Figure12.6Illustration oftheeffectsoflinearpre-processing appliedtotheOldFaithfuldataset.Theploton
theleftshowstheoriginaldata.Thecentreplotshowstheresultofstandardizing theindividual variables tozero
meanandunitvariance. Alsoshownaretheprincipal axesofthisnormalized dataset,plottedovertherange
±A~/2.Theplotontherightshowstheresultofwhitening ofthedatatogiveitzeromeanandunitcovariance.
whereLisaDxDdiagonal matrixwithelements Ai,andUisaDxDorthog­
onalmatrixwithcolumns givenbyUi.Thenwedefine,foreachdatapointXn,a
transformed valuegivenby
(12.24)
wherexisthesamplemeandefinedby(12.1).Clearly,theset{Yn}haszeromean,
anditscovariance isgivenbytheidentitymatrixbecause
N
1~LL-1/2UT(Xn-x)(xn-x)TUL-1/2
n=l
L~1/2UTSUL -1/2=L-1/2LL-1/2=I.(12.25)
Appendix A
Appendix AThisoperation isknownaswhitening orsphereing thedataandisillustrated forthe
OldFaithfuldatasetinFigure12.6.

(12.24)
N
n=l
Appendix A
Itisinteresting tocompare PCAwiththeFisherlineardiscriminant whichwas
discussed inSection4.1.4.Bothmethods canbeviewedastechniques forlinear
dimensionality reduction. However, PCAisunsupervised anddependsonlyonthe
values XnwhereasFisherlineardiscriminant alsousesclass-label information. This
difference ishighlighted bytheexample inFigure12.7.
Anothercommon application ofprincipal component analysisistodatavisual­
ization.Hereeachdatapointisprojected ontoatwo-dimensional (M=2)principal

subspace, sothatadatapointXnisplottedatCartesian coordinates givenbyx'J.U1
andx'J.U2,where UlandU2aretheeigenvectors corresponding tothelargestand
secondlargesteigenvalues. Anexample ofsuchaplot,fortheoilflowdataset,is
showninFigure12.8.

12.1.I'<incipal Cl)m..."n~ntAnal}'s;, 569
.,':----;!---;-"
_.S 0 3'~'"~.•••._',.".,-..'~.•''''..,..'~"
':r-'---~+·_ ..-:--'~-J
~,.,
."Acomparison 01pro:ipalcompo­
Mntanalysis ....111Fisha(s linaar
discriminant 101"""",<*man""'"
alityr&duclion. Heretoodatain
twodimansions, belonging totwo
classes sIIOWI1inredandblue.is
tobePfOI"Cled ontoas.ingledi·
mension. PCAc/>xlsas thedirec·
tion01maximum varia""e. sIIOWI1
trythama9""ta Co""'.wt11chleads
tostrongclassoverlap. whereas
!heFisl>efIiMardiSCfOrnillant takes
accoun1 <:Atooclasslabelsand
leadstoaprojection ontotheg<ean
CUM!givingmucht>etlerclass
separationFig"",12.7
Fig"",12.8Visualilatlon 01!heoill'low<latalIetobtained
tryprojoecting the<lataontothelirsttwoprin.
cipalcompone<1ts. The<ed,blue,and9r&en
pointscorre-spond to!he'IamiNI(, 't>omo-
genoous', and'8nnula~ flowoonligurations
",specriveIy.
12.1.4peAforhigh-dimensional data
Insomeapplication. ofpliTlCipal component analysis. thenumberofdatapoints

issmallerthant!>cdimensionality oftroedata'pace.FOI"example. ",emightwantto
applyPeAtoadata<elofafewhundred images,eachof,,'hichrorrespoOOs toa
"eetorina'paceofpoIentially .....mlmilliondimensiOlls (COITesponding tnthfl'e
enlour"aluesforeachofthepi.",lsintroeimage),NOIethatinaD-<limen,ional space
asetofjYpoints.",'hereN<D.definesalinearsubspa::e ",hosedimensi"nality
isat""'stN-1,andSOthereislinlepointinapplying PeAfor,'alue<ofM
thaI"'"greaterthanN-I,Indeed,if"'epelf"",,PeAwewillfindthatatleast

D-N+Ioftheeigen".lues artlero.eorrespnnding tQeigenvectors aloog",hose
direclioos thedata<elhas10mvarianee. Funhem>ore. typicalalgOl"ithm, forfinding
theeigen,'eet"" ofaDxDmatrixha"eacomputatiooal eoslthmscaleslikeO(D~J.
aOOsoforappliealions suchastheimagee,ample. adirec'application ofPeAwill
becomputatiooally infe,,-sibJe.
W.canresoh'ethisproblem asfoIl",",'" Fir;l.letusdefineXtobethe(N"DJ·

570 12.CONTINUOUS LATENT VARIABLES
dimensional centreddatamatrix,whosenthrowisgivenby(xn-X)T.Thecovari­
ancematrix(12.3)canthenbewrittenasS=N-1XTX,andthecorresponding
eigenvector equation becomes
1T-XXUi=AiUi N .
Nowpre-multiply bothsidesbyXtogive
1TNXX(XUi)=Ai(XUi)'
IfwenowdefineVi=XUi,weobtain
1T-XX Vi=AiViN(12.26)
(12.27)
(12.28)
(12.30)whichisaneigenvector equation fortheNxNmatrixN-1XXT.Weseethatthis
hasthesameN-1eigenvalues astheoriginalcovariance matrix(whichitselfhasan
additional D-N+1eigenvalues ofvaluezero).Thuswecansolvetheeigenvector
problem inspacesoflowerdimensionality withcomputational costO(N3)instead
ofO(D3).Inordertodetermine theeigenvectors, wemultiplybothsidesof(12.28)
byXTtogive
(1T) T T NXX(XVi)=Ai(XVi) (12.29)
fromwhichweseethat(XTVi)isaneigenvector ofSwitheigenvalue Ai.Note,
however, thattheseeigenvectors neednotbenormalized. Todetermine theappropri­
atenormalization, were-scaleUiex:XTVibyaconstantsuch that IluiII=1,which,
assuming Vihasbeennormalized tounitlength,gives
1 T

1 T
Ui=(NAi)1/2XVi·
Insummary, toapplythisapproach wefirstevaluateXXTandthenfinditseigen­
vectorsandeigenvalues andthencompute theeigenvectors intheoriginaldataspace
using(12.30).
12.2.Probabilistic peA
Theformulation ofPCAdiscussed intheprevious sectionwasbasedonalinear
projection ofthedataontoasubspace oflowerdimensionality thantheoriginaldata
space.WenowshowthatPCAcanalsobeexpressed asthemaximum likelihood
solutionofaprobabilistic latentvariablemodel.Thisreformulation ofPCA,known
asprobabilistic peA,bringsseveraladvantages compared withconventional PCA:

•Probabilistic PCArepresents aconstrained formoftheGaussian distribution
inwhichthenumberoffreeparameters canberestricted whilestillallowing
themodeltocapturethedominant correlations inadataset.

Section12.2.3
Section8.1.412.2.Probabilistic peA 571
•WecanderiveanEMalgorithm forPCAthatiscomputationally efficientin
situations whereonlyafewleadingeigenvectors arerequired andthatavoids
havingtoevaluatethedatacovariance matrixasanintermediate step.
•Thecombination ofaprobabilistic modelandEMallowsustodealwithmiss­
ingvaluesinthedataset.
•Mixtures ofprobabilistic PCAmodelscanbeformulated inaprincipled way
andtrainedusingtheEMalgorithm.
•Probabilistic PCAformsthebasisforaBayesian treatment ofPCAinwhich
thedimensionality oftheprincipal subspace canbefoundautomatically from
thedata.
•Theexistence ofalikelihood function allowsdirectcomparison withother
probabilistic densitymodels.Bycontrast, conventional PCAwillassignalow
reconstruction costtodatapointsthatareclosetotheprincipal subspace even
iftheyliearbitrarily farfromthetrainingdata.
•Probabilistic PCAcanbeusedtomodelclass-conditional densities andhence
beappliedtoclassification problems.

thedata.
•Theprobabilistic PCAmodelcanberungeneratively toprovidesamplesfrom
thedistribution.
Thisformulation ofPCAasaprobabilistic modelwasproposed independently by
TippingandBishop(1997,1999b)andbyRoweis(1998).Asweshallsee later, itis
closelyrelatedtofactoranalysis(Basilevsky, 1994).
Probabilistic PCAisasimpleexample ofthelinear-Gaussian framework, in
whichallofthemarginal andconditional distributions areGaussian. Wecanformu­
lateprobabilistic PCAbyfirstintroducing anexplicitlatentvariablezcorresponding

totheprincipal-component subspace. NextwedefineaGaussian priordistribution
p(z)overthelatentvariable, togetherwithaGaussian conditional distribution p(xlz)
fortheobserved variablexconditioned onthevalueofthelatentvariable. Specifi­
cally,thepriordistribution overzisgivenbyazero-mean unit-covariance Gaussian
p(z)=N(zIO,I). (12.31)
Similarly, theconditional distribution oftheobserved variablex,conditioned onthe
valueofthelatentvariablez,isagainGaussian, oftheform
p(xlz)=N(xlWz+J-L,a2I) (12.32)

Section8.2.2inwhichthemeanofxisagenerallinearfunction ofzgoverned bytheDxM
matrixWandtheD-dimensional vectorJ-L.Notethatthisfactorizes withrespectto
theelements ofx,inotherwordsthisisanexample ofthenaiveBayesmodel.As
weshallseeshortly,thecolumns ofWspanalinearsubspace withinthedataspace
thatcorresponds totheprincipal subspace. Theotherparameter inthismodelisthe
scalara2governing thevarianceoftheconditional distribution. Notethatthereisno

572 11.CONTINUOUS LAT!::NT VANIM1LI::S
,/.-,,,,,,,
,
Flgu..12.9I\n~I"'tfat"" oIlt>eII"""fative vi&woI1t>ep<ot>abi!;st", PeAmodeIfof" two-dimensiooal <!ala
spaceandaon&-<lirnent.ionallat/l<1t space,AnOb&erved <!alapointxIsgenerated byfirstdrawingavaluei
fof1t>eIat&n1vafiatlle /f(lm ~sprior dist,~t"" P(~)andItlendrawingaval""fofxlromaniSO/fopK:Gaussian
distr~t"" (iijust,al&(l bytheredcir<:ie's)havingmeanwi+"andcoY8r1.once,,'1Thel/f&er\ellips.&$showl!le
density"""toors!of themarg'''''1 dis1r1bulion PIx).
lossofge""raJity inassuming azeromean.unitco\'ariance Gau"ian forthelatent
distributi"n II{Z)becauseamoregcneralGau"i3n di"ributi"n wouldgi"erisetoan
equivalent probabili"ic n>odel.
Wecanviewtheprobabilistic PeAmodelfromageoerati"e \'iew""int in"hich
asampled '-alueoftheob""Yed ,..riableisobIained byfirstchoo,ing a,..Iuefor
thelatent,'ariahle aodthen>ampling theOO",,,'e;j ,-ariable cooditioned onthislao
tent\'alue,Specifically, theV-dimen'ional OO"''''ed '-ariablexisdefinedbyalin·

ea,tran,formati,," ofthe'\/·dimen,i"nal latcnt'-ariable zplu,additi'-eGaussian
'noise',<0that
,,=\VZ+,,+~ (12.33)
w!>erezisanM-di""'nsional Gaussian lalentvariable. and..isaV·dimensi"nal
,ero-mean Gau..ian-distributed noi..,"ariable witbco'-ariance ,,21.Thisgenerative
processisillustrated inFigure12.9.NOIethatthisframe".-orl< isbasedonamapping
fromlatent,pace10dataspace.incontrast 10thenl()l'(:C(""'cnti,,,,"1 "iew"fI'CA
dis.cus"'d alx",e,11Ie",,'e=mapping, fromdataspacetothelatentspace.,,-illhe
OOlained ,honlyusingHaycs·lhwn:m.

SUf!ll'OSC wewish10deten"ine the"aluesofll>oparameters \V.I'and,,'uSIng
maximum likelihuo<l, Towrite""""nlhelikeliltood function, weneedan""pression
fortl>omarginal distributioo p{")oftl>o~,,'ed ...riahle_Thisisexprt __sed.fmn'
thesumaodp,oductrules"fprobability, intheform
(11,34)
E,e,,-ise 12,7ll""auS(: thiscorresponds toalinear·Gau"i,n lT1(llIcLthi<marginal di,tribulion is
againGaussian. atldisgivenby
,,(,,)_N{xllf,C) (IUS)

wheretheDxDcovariance matrixCisdefinedby
C=WWT+0-21.573
(12.36)
Thisresultcanalsobederivedmoredirectlybynotingthatthepredictive distribution
willbeGaussian andthenevaluating itsmeanandcovariance using(12.33). This
gives
IE[x]
cov[x]IE[Wz+JL+E]=JL
IE[(Wz+E)(WZ+E)T]
IE[WZZTWT]+IE[EET]=WWT+0-21(12.37)
(12.38)
wherewehaveusedthefactthatzandEareindependent randomvariables andhence
areuncorrelated.
Intuitively, wecanthinkofthedistribution p(x)asbeingdefinedbytakingan
isotropic Gaussian 'spraycan'andmoving it acrosstheprincipal subspace spraying
Gaussian inkwithdensitydetermined by0-2andweighted bythepriordistribution.
Theaccumulated inkdensitygivesrisetoa'pancake' shapeddistribution represent­
ingthemarginal densityp(x).
Thepredictive distribution p(x)isgoverned bytheparameters JL,W,and0-2•
However, thereisredundancy inthisparameterization corresponding torotations of
thelatentspacecoordinates. Toseethis,consider amatrixW=WRwhereRis

anorthogonal matrix.Usingtheorthogonality propertyRRT=I,weseethatthe
quantityWWTthatappearsinthecovariance matrixCtakestheform
(12.39)
(12.41)
Exercise 12.8andhenceisindependent ofR.ThusthereisawholefamilyofmatricesWallof
whichgiverisetothesamepredictive distribution. Thisinvariance canbeunderstood
intermsofrotations withinthelatentspace.Weshallreturntoadiscussion ofthe
numberofindependent parameters inthismodellater.
Whenweevaluatethepredictive distribution, werequireC-1,whichinvolves

(12.39)
(12.41)
theinversion ofaDxDmatrix.Thecomputation requiredtodothiscanbereduced
bymakinguseofthematrixinversion identity(C.7)togive
C-1=0--11-0--2WM-1WT(12.40)
wheretheM x MmatrixMisdefinedby
M=WTW+0-21.
BecauseweinvertMratherthaninverting Cdirectly,thecostofevaluating C-1is
reducedfromO(D3)toO(M3).
Aswellasthepredictive distribution p(x),wewillalsorequiretheposterior
distributionp(zlx), whichcanagainbewrittendowndirectlyusingtheresult(2.116)
forlinear-Gaussian modelstogive
(12.42)

(12.42)
Notethattheposterior meandependsonx,whereas the posterior covariance isin­
dependent ofx.

574 12.CONTINUOUS LATENT VARIABLES
Figure12.10Theprobabilistic peAmodelforadatasetofNobser­
vationsofxcanbeexpressed asadirectedgraphin
whicheachobservation Xnisassociated withavalue
Znofthelatentvariable.
..-+--w
N
12.2.1Maximum likelihood peA
Wenextconsider thedetermination ofthemodel parameters usingmaximum
likelihood. GivenadatasetX={xn}ofobserved datapoints,theprobabilistic
peAmodelcanbeexpressed asadirectedgraph,asshowninFigure12.10.The
corresponding loglikelihood function isgiven,from(12.35),by
N
Inp(XIJL, W,O'2)=Llnp(xnIW,JL,O'2)
n=l
N
ND N1"" T 1--2-ln(2n)-2lnIe[-2L,..(xn-JL)c-(xn-JL).(12.43)
n=l
Settingthederivative oftheloglikelihood withrespecttoJLequaltozerogivesthe
expected resultJL=xwherexisthedatameandefinedby(12.1).Back-substituting
wecanthenwritetheloglikelihood functionintheform
NInp(XIW, JL,0'2)=-2{DIn(2n)+InIe[+Tr(C-1S)} (12.44)
whereSisthedatacovariance matrixdefinedby(12.3).Becausetheloglikelihood
isaquadratic functionofJL,thissolutionrepresents theuniquemaximum, ascanbe

N
n=l
N
n=l
confirmed bycomputing secondderivatives.
Maximization withrespecttoWand0'2ismorecomplex butnonetheless has
anexactclosed-form solution.ItwasshownbyTippingandBishop(1999b)thatall
ofthestationary pointsoftheloglikelihood functioncanbewrittenas
(12.45)
whereUMisaDxMmatrixwhosecolumns aregivenbyanysubset(ofsizeM)
oftheeigenvectors ofthedatacovariance matrixS,theMxMdiagonal matrix
LMhaselements givenbythecorresponding eigenvalues ..\,andRisanarbitrary
MxMorthogonal matrix.

(12.45)
Furthermore, TippingandBishop(1999b)showedthatthemaximum ofthelike­
lihoodfunction isobtained whentheMeigenvectors arechosentobethosewhose
eigenvalues aretheMlargest(allothersolutions beingsaddlepoints).Asimilarre­
sultwasconjectured independently byRoweis(1998),although noproofwasgiven.

12.2.Probabilistic peA 575
Again,weshallassumethattheeigenvectors havebeenarranged inorderofdecreas­
ingvaluesofthecorresponding eigenvalues, sothattheMprincipal eigenvectors are
Ul,"" UM.Inthiscase,thecolumns ofWdefinetheprincipal subspace ofstan­
dardPCA.Thecorresponding maximum likelihood solutionfor(J'2isthengivenby
1 D
(J'~L=D-MLAi
i=M+l(12.46)
Section12.2.2sothat(J'~Listheaveragevariance associated withthediscarded dimensions.
BecauseRisorthogonal, itcanbeinterpreted asarotationmatrixintheM x M
latentspace.Ifwesubstitute thesolutionforWintotheexpression forC,andmake
useoftheorthogonality propertyRRT=I,weseethatCisindependent ofR.
Thissimplysaysthatthepredictive densityisunchanged byrotations inthelatent
spaceasdiscussed earlier.Fortheparticular caseofR=I,weseethatthecolumns
ofWaretheprincipal component eigenvectors scaledbythevariance parameters
Ai-(J'2.Theinterpretation ofthesescalingfactorsisclearoncewerecognize that
foraconvolution ofindependent Gaussian distributions (in thiscasethelatentspace

distribution andthenoisemodel)thevariances areadditive. Thusthevariance Ai
inthedirection ofaneigenvector Uiiscomposed ofthesumofacontribution Ai­
(J'2fromtheprojection oftheunit-variance latentspacedistribution intodataspace
throughthecorresponding columnofW,plusanisotropic contribution ofvariance
(J'2whichisaddedinalldirections bythenoisemodel.
Itisworthtakingamoment tostudytheformofthecovariance matrixgiven
by(12.36).Consider thevarianceofthepredictive distribution alongsomedirection
specified bytheunitvectorv,wherevTv=1,whichisgivenbyvTCv. First

supposethatvisorthogonal totheprincipal subspace, inotherwordsitisgivenby
somelinearcombination ofthediscarded eigenvectors. ThenvTV=0and hence
vTCv=(J'2.Thusthemodelpredictsanoisevariance orthogonal totheprincipal
subspace, which,from(12.46),isjusttheaverageofthediscarded eigenvalues. Now
supposethatv=UiwhereUiisoneoftheretainedeigenvectors defining theprin­
cipalsubspace. ThenvTCv=(Ai-(J'2)+(J'2=Ai.Inotherwords,thismodel
correctly captures thevarianceofthedataalongtheprincipal axes,andapproximates

thevariance inallremaining directions withasingleaveragevalue(J'2.
Onewaytoconstruct themaximum likelihood densitymodelwouldsimplybe
tofindtheeigenvectors andeigenvalues ofthedatacovariance matrixandthento
evaluateWand (J'2usingtheresultsgivenabove.Inthiscase,wewouldchoose
R=Iforconvenience. However, ifthemaximum likelihood solutionisfoundby
numerical optimization ofthelikelihood function, forinstance usinganalgorithm
suchasconjugate gradients (Fletcher, 1987;Nocedal andWright,1999;Bishopand

Nabney, 2008)orthroughtheEMalgorithm, thentheresulting valueofRises­
sentially arbitrary. Thisimpliesthatthecolumns ofWneednotbeorthogonal. If
anorthogonal basisisrequired, thematrixWcanbepost-processed appropriately
(GolubandVanLoan,1996).Alternatively, theEMalgorithm canbemodified in
suchawayastoyieldorthonormal principal directions, sortedindescending order
ofthecorresponding eigenvalues, directly(AhnandOh,2003).

576 12.CONTINUOUS LATENT VARIABLES
Therotational invariance inlatentspacerepresents aformofstatistical noniden­
tifiability, analogous tothatencountered formixturemodelsinthecaseofdiscrete
latentvariables. Herethereisacontinuum ofparameters allofwhichleadtothe
samepredictive density,incontrasttothediscretenonidentifiability associated with
component re-labelling inthemixturesetting.
Ifweconsider thecaseofM=D,sothatthereisnoreduction ofdimension­
ality,thenUM=UandLM=L.Makinguseoftheorthogonality properties
UUT=IandRRT=I,weseethatthecovariance Cofthemarginal distribution
forxbecomes
(12.47)
andsoweobtainthestandard maximum likelihood solutionfor anunconstrained
Gaussian distribution inwhichthecovariance matrixisgivenbythesamplecovari­
ance.
Conventional PCAisgenerally formulated asaprojection ofpointsfromtheD­
dimensional dataspaceontoanM-dimensional linearsubspace. Probabilistic PCA,
however, ismostnaturally expressed asamapping fromthelatentspaceintothedata

(12.47)
ance.
spacevia(12.33). Forapplications suchasvisualization anddatacompression, we
canreversethismapping usingBayes'theorem. Anypointxindataspacecanthen
besummarized byitsposterior meanandcovariance inlatentspace.From(12.42)
themeanisgivenby
whereMisgivenby(12.41).Thisprojectstoapointindataspacegivenby
WlE[zlx]+J-L.(12.48)
(12.49)
Section3.3.1 Notethatthistakesthesameformastheequations forregularized linearregression
andisaconsequence ofmaximizing thelikelihood function foralinearGaussian
model.Similarly, theposterior covariance isgivenfrom(12.42)by0-2M-1andis

(12.49)
independent ofx.
Ifwetakethelimit0-2----t0,thentheposterior meanreducesto
(12.50)
Exercise 12.11
Exercise 12.12
Section2.3whichrepresents anorthogonal projection ofthedatapointontothelatentspace,
andsowerecoverthestandard PCAmodel.Theposterior covariance inthislimitis
zero,however, andthe density becomes singular. For0-2>0,thelatentprojection
isshiftedtowardstheorigin,relativetotheorthogonal projection.
Finally,wenotethatanimportant rolefortheprobabilistic PCAmodelisin

(12.50)
definingamultivariate Gaussian distribution inwhichthenumberofdegreesoffree­
dom,inotherwordsthenumberofindependent parameters, canbecontrolled whilst
stillallowing themodel to capturethedominant correlations inthedata.Recall
thatageneralGaussian distribution hasD(D+1)/2independent parameters inits
covariance matrix(plusanotherDparameters inits mean). Thusthenumberof
parameters scalesquadratically withDandcanbecomeexcessive inspacesofhigh

12.2.Probabilistic peA 577
dimensionality. Ifwerestrictthecovariance matrixtobediagonal, thenithasonlyD
independent parameters, andsothenumberofparameters nowgrowslinearlywith
dimensionality. However, itnowtreatsthevariables asiftheywereindependent and
hencecannolongerexpressanycorrelations betweenthem.Probabilistic PeApro­
videsanelegantcompromise inwhichtheMmostsignificant correlations canbe
captured whilestillensuring thatthetotalnumberofparameters growsonlylinearly
withD.Wecanseethisbyevaluating thenumberofdegreesoffreedom inthe
PPCAmodelasfollows. Thecovariance matrixCdependsontheparameters W,
whichhassizeDxM,anda2,givingatotalparameter countofDM+1.However,
wehaveseenthatthereissomeredundancy inthisparameterization associated with
rotations ofthecoordinate systeminthelatentspace.Theorthogonal matrixRthat
expresses theserotations hassizeMxM.Inthefirstcolumnofthismatrixthereare
M -1independent parameters, becausethecolumnvectormustbenormalized to

unitlength.InthesecondcolumnthereareM -2independent parameters, because
thecolumnmustbenormalized andalsomustbeorthogonal totheprevious column,
andsoon.Summing thisarithmetic series,weseethatRhasatotalofM(M-1)/2
independent parameters. Thusthenumberofdegreesoffreedom inthecovariance
matrixCisgivenby
DM+1-M(M-1)/2. (12.51)
Exercise 12.14
Section12.2.4
Section9.4Thenumberofindependent parameters inthismodeltherefore onlygrowslinearly
withD,forfixedM.IfwetakeM=D-1,thenwerecoverthestandard result

forafullcovariance Gaussian. Inthiscase,thevariance alongD-1linearlyin­
dependent directions iscontrolled bythecolumnsofW,andthevariancealongthe
remaining direction isgivenbya2.IfM=0,themodelisequivalent totheisotropic
covariance case.
12.2.2EMalgorithm forpeA
Aswehaveseen,theprobabilistic PCAmodelcanbeexpressed intermsofa
marginalization overacontinuous latentspacezinwhichforeachdatapointXn,
thereisacorresponding latentvariable Zn.Wecantherefore makeuseoftheEM

algorithm tofindmaximum likelihood estimates ofthemodelparameters. Thismay
seemratherpointless becausewehavealreadyobtained anexactclosed-form so­
lutionforthemaximum likelihood parameter values.However, inspacesofhigh
dimensionality, theremaybecomputational advantages inusinganiterative EM
procedure ratherthanworkingdirectlywiththesamplecovariance matrix.ThisEM
procedure canalsobeextended tothefactoranalysismodel,forwhichthereisno
closed-form solution. Finally,itallowsmissingdatatobehandledinaprincipled
way.

way.
WecanderivetheEMalgorithm forprobabilistic PCAbyfollowing thegeneral
framework forEM.Thuswewritedownthecomplete-data loglikelihood and take
itsexpectation withrespecttotheposterior distribution ofthelatentdistribution
evaluated using'old'parameter values.Maximization ofthisexpected complete­
dataloglikelihood thenyieldsthe'new'parameter values.Because thedatapoints

578 12.CONTINUOUS LATENT VARIABLES
areassumed independent, thecomplete-data loglikelihood function takestheform
N
Inp(X,ZIJL,W,(J2)=L{lnp(xnlz n) +lnp(zn)}
n=l(12.52)
wherethenthrowofthematrixZisgivenbyZn.Wealreadyknowthattheexact
maximum likelihood solutionforJLisgivenbythesamplemeanxdefinedby(12.1),
anditisconvenient tosubstitute forJLatthisstage.Makinguseoftheexpressions
(12.31)and(12.32)forthelatentandconditional distributions, respectively, andtak­
ingtheexpectation withrespecttotheposterior distribution overthelatentvariables,
weobtain
Notethatthisdepends ontheposterior distribution onlythroughthesufficient statis­
ticsoftheGaussian. ThusintheEstep,weusetheoldparameter valuestoevaluate
M-1WT(Xn-x)
(J2M-1+lE[zn]lE[zn]T(12.54)
(12.55)
Exercise 12.15whichfollowdirectlyfromtheposterior distribution (12.42)together withthestan­
dardresult lE[znz~]=cov[zn]+JE[zn]JE[zn]T. HereMisdefinedby(12.41).
IntheMstep,wemaximize withrespecttoWand (J2,keeping theposterior

weobtain
(12.55)
statistics fixed.Maximization withrespectto(T2isstraightforward. Forthemaxi­
mization withrespecttoWwemakeuseof(C.24),andobtaintheM-stepequations
Wnew
2(Jnew=[t,exn-X)IlIZn]T] [t,Il[ZnZ~]]-'
1N
NDL{llxn-xl12
-2lE[zn]TW~ew(xn -x)
n=l
+Tr(JE[znzJ]W~ew Wnew)}.(12.56)
(12.57)
TheEMalgorithm forprobabilistic PCAproceeds byinitializing theparameters
andthenalternately computing thesufficient statistics ofthelatentspaceposterior
distribution using(12.54)and(12.55)intheEstepandrevisingtheparameter values
using(12.56)and(12.57)intheMstep.

Wnew
1N
n=l
(12.57)
OneofthebenefitsoftheEMalgorithm forPCAiscomputational efficiency
forlarge-scale applications (Roweis, 1998).Unlikeconventional PCAbasedonan

(12.58)12.2.Probabilistic peA 579
eigenvector decomposition ofthesamplecovariance matrix,theEMapproach is
iterativeandsomightappeartobelessattractive. However, eachcycleoftheEM
algorithm canbecomputationally muchmoreefficientthanconventional PCAin
spacesofhighdimensionality. Toseethis,wenotethattheeigendecomposition of
thecovariance matrixrequiresO(D3)computation. Oftenweareinterested only
inthefirstMeigenvectors andtheircorresponding eigenvalues, inwhichcasewe
canusealgorithms thatare0(MD2).However, theevaluation ofthecovariance
matrixitselftakes0(ND2)computations, whereNisthenumberofdatapoints.
Algorithms suchasthesnapshot method(Sirovich, 1987),whichassumethatthe
eigenvectors arelinearcombinations ofthedatavectors,avoiddirectevaluation of
thecovariance matrixbutareO(N3)andhenceunsuitedtolargedatasets.TheEM
algorithm described herealsodoesnotconstruct thecovariance matrixexplicitly.
Instead,themostcomputationally demanding stepsarethoseinvolving sumsover

thedatasetthatare0(NDM).ForlargeD,andM«D,thiscanbeasignificant
savingcompared to0(ND2)andcanoffsettheiterativenatureoftheEMalgorithm.
NotethatthisEMalgorithm canbeimplemented inanon-lineforminwhich
eachD-dimensional datapointisreadinandprocessed andthendiscarded before
thenextdatapointisconsidered. Toseethis,notethatthequantities evaluated in
theEstep(anM-dimensional vectorandanM x M matrix)canbecomputed for
eachdatapointseparately, andintheMstepweneedtoaccumulate sumsoverdata

points,whichwecandoincrementally. Thisapproach canbeadvantageous ifboth
NandDarelarge.
Becausewenowhaveafullyprobabilistic modelforPCA,wecandealwith
missingdata,provided thatitismissingatrandom,bymarginalizing overthedis­
tribution oftheunobserved variables. Againthesemissingvaluescanbetreated
usingtheEMalgorithm. Wegiveanexample oftheuseofthisapproach fordata
visualization inFigure12.11.
AnotherelegantfeatureoftheEMapproach isthatwecantakethelimita2----t0,
corresponding tostandardPCA,andstillobtainavalidEM-likealgorithm (Roweis,

1998).From(12.55),weseethattheonlyquantityweneedtocompute intheEstep
isJE[zn].Furthermore, theMstepissimplifie~ becauseM=WTW.Toemphasize
thesimplicity ofthealgorithm, letusdefineXtobeamatrixofsizeNxDwhose
nthrowisgivenbythevector Xn-xandsimilarly define0tobeamatrixofsize
Dx MwhosenthrowisgivenbythevectorJE[zn].TheEstep(12.54)oftheEM
algorithm forPCAthenbecomes
o=(W~dWold)-lW~dX
andtheMstep(12.56)takestheform
Wnew=XTOT(OOT)-l. (12.59)
Againthesecanbeimplemented inanon-lineform.Theseequations haveasimple

interpretation asfollows.Fromourearlierdiscussion, weseethattheEstepinvolves
anorthogonal projection ofthedatapointsontothecurrentestimatefortheprincipal
subspace. Correspondingly, theMsteprepresents are-estimation oftheprincipal

580 12.CONTlNlJOlJS I"Ht;I'IT Vi\RIARlES
Fig"..12.11Probabilistic PCAvisoo,zsbon 01aportion0I1he""!lowdatasetlo<Ihe!irsl100(lata»einls,The
left..,...nd plotoIIOWSIheI'O"leoo< meanproj9c1ions oIlhfI(latapoimsonlheprincipal subspace. The,;gtrI·hi\nd
plotisobtained byfirslran<lomly omitting30%0I1hevariable.aloo.andlhenus>rlgEM10MndIeI""mi......
values.NoteI!IaIeac/1datapoinl1hen NoSallea.,onemissingmea.u,ement butlhoallhe ploti.""ry..mia,to
lheonaobtained wit"""lmiss....valL>ll$
Ewrrise /2,/7subspace tominimize !hesquaredreoonslruCtioo errorin'oIhichtheproje<:tion, are
C.,N.
Weeangh'ea,implephysical analogy forthisEMalgorithm. whichiseasily
visualized forD=2andM=1.Coo,ider acollectioo nfdatapoint',ntWI)
dimension', aodlettileu""'-dimensiunal principal subspace berepresented bya<ohd
rod.NowatlaCheachdatapointtothenxIviaa,pringoo<:)"ing HooI;:e', law("umJ
energyi,propol1ional 10,liesquareoflilespring". length). Inll1eE'tel',wekeep
thenxIhedandallowtheattachment point'tn,Iideupand<I<,wnll1enxI'"a,to

C.,N.
minimize ll1ee",,'llY,Thiscau",.eachattachment point(independently) 10position
Itselfattheorthogonal pmjeclion ofthec~sponding datapointontothenxI.In
theM'tel'.wekeeptheattachment poiOl'fil<edandthenreleasetilenxIandallowit
tom'>,'e10tileminimum energyposilion. 11IeEandM'tepsarethenrepeated until
a,uitablec""vergence cri.eri"" is..,isfled.a.isilluSlrated inFigure12.12.
12.2.3Bayesian peA
S<JfarinOIlrdi",""ioo ofPeA.wehave",'.nled Ihaltile'"Ine,IIfor,lie
dl,nen,ionalit)" oftileprincipal .ubspace isgi"en,Inpraclice. ".-enlmtcOOosea

suilable ,..I""according 10theapplication. For,isuali,a,ion. wege""",ny choose
.\1=2.whereas forOIherapplication, theapprorrialC choicefor,1/ma)"beless
dea,.Oneappmao:h i.10pi",theeigen"alue 'peclrum forlhedataset.analog,•."
10theexample inFigure12.4fortheoff_linedigitsdalaSCI,andlooktoseeiflite
eige",,,I....nmurally formtwogroupscomprising asetof,mall,'aluesseparated by
a,ign;flcant gapfroma",tofrelativel)" large,'alues,indicating anaturalcholccf<>r
AI,Inpractice. suchagapi,oflen'''''seen

,o-,,o-,,o-,o,",•, ,,<,•,
0 ~,•0 0 ....
-, -, -,
,0,-, 0 -, 0,
,", , ,
-,
Flgu..12.12Synt"'elic <lataillustrating tooEMalgorithm !ofPCAdefinedby(12.58)and(1259).(8)Adata
setXwiththedatapointsshownin1JI'e«l,t"ll"tM' W'i1!1l!>et'IMpMdpal""""""",IS (shownaseigenveclor1
scaledbyIt>esquafll'OOIS04theeigeJ'l\lllluel). (b)Initialconfigurat"'" 01tooprincipalsul>sl>a<:<t definedbyW,
showninmd,tOO"lhfIr withthefK'(Ijeclions 01thelatll<11pointsZinlotoo<lataspace,giItoobyZWT,shownin
cyan,(ojAlter""" Mstep,toolaten!SI'B«lP>asbeenupdate<!wiIhZr>el(lnxed.(d)Me'tt>esuccess....ESlep,
It>e""'-'eo01Zhavubeenup<!atll<:1.~'"Ihogoooal r>rojecliQn$, withWh&k!fixed.(e)Aft....tilese<:o<l<lM
S!flp.<')Afterl!IeMC()<>;lEst"l'
S,uionI.JBe<:au,", th~pm/xlhili>licPeAmoddhasawell·defined likelillood f"flCtion, we
<wIdemploycros,-,-.1idation todelermine the\"aJueofdi"",nsiooa!ity by"'Iecting
tit<:large,tloglikelihood t>I1a'-alidation datasetSuchanopprooch. hov.·~\-er. can

becomecomputationally ro<lly.p3rticularl)' ifweCQnsid<:, •probabilistic miXlUre
ofPeAmodds(Tipping andBishop.1999a)in"hichweseek10<!etermi'" the
appropriate dimen,ionalily ",paraltly fortochcomponenl inlt1emixm""
Gi'-enthaiw.ha,-eaprobabilislic formulalion ofPeA,ils«msnatural10s«k
uBuye,ian approach 10modelseleclion. Todothi,.,,"'enee<!10marginalize 001
themodelparamele" /'.\V.und,,'wilh""peeltoappropriate priordistribution'.
ThisCanbedonebyu,inga,-ariation.l framework to.pproxim'le theallulylic.lly

intractable murginaliUOi;oo, (Bi,hop.1mb).1lIcmarginal likelihood v.lues.given
byttle,'ari.,ionallower bour.d,cunlhenbec<>mpun:d forar.ngeofdifferent'"Tue'
"f;\Iar.dItie'"IuegivingIhtlargestmarginal likelihood ",Iecloo_
l1ereweconsider. simplerapproach introducoo byb.asedontherddmu"p-

582 12.CONTINUOUS LATENT VARIABLES
Figure12.13Probabilistic graphical modelforBayesian peAin
whichthedistribution overtheparameter matrixW
isgoverned byavectoraofhyperparameters.
w
N
proximation, whichisappropriate whenthenumberofdatapointsisrelatively large
andthecorresponding posterior distribution istightlypeaked(Bishop, 1999a).It
involves aspecificchoiceofprioroverWthatallowssurplusdimensions inthe
principal subspace tobeprunedoutofthemodel.Thiscorresponds toanexampleof
automatic relevance determination, orARD,discussed inSection7.2.2.Specifically,
wedefineanindependent Gaussian priorovereachcolumnofW,whichrepresent
thevectorsdefiningtheprincipal subspace. EachsuchGaussian hasanindependent
variance governed byaprecision hyperparameter O:isothat
(12.60)
Section7.2whereWiistheithcolumnofW.Theresulting modelcanberepresented usingthe
directedgraphshowninFigure12.13.
ThevaluesforO:iwillbefounditeratively bymaximizing themarginallikeli­
hoodfunctioninwhichWhasbeenintegrated out.Asaresultofthisoptimization,

(12.60)
someoftheO:imaybedriventoinfinity,withthecorresponding parameters vec­
torWibeingdriventozero(theposterior distribution becomes adeltafunctionat
theorigin)givingasparsesolution. Theeffective dimensionality oftheprincipal
subspace isthendetermined bythenumberoffiniteO:ivalues,andthecorrespond­
ingvectors Wicanbethoughtofas'relevant' formodelling thedatadistribution.
Inthisway,theBayesian approach isautomatically makingthetrade-off between
improving thefittothedata,byusingalargernumberofvectors Wiwiththeircor­

responding eigenvalues Aieachtunedtothedata,andreducing thecomplexity of
themodelbysuppressing someoftheWivectors.Theoriginsofthissparsitywere
discussed earlierinthecontextofrelevance vectormachines.
ThevaluesofO:iarere-estimated duringtrainingbymaximizing thelogmarginal
likelihood givenby
p(Xla, J-L,0'2)=Jp(XIW, J-L,O'2)p(Wla) dW (12.61)
wherethelogofp(XIW, J-L,0'2)isgivenby(12.43).Notethatforsimplicity wealso
treatJ-Land0'2asparameters tobeestimated, ratherthandefiningpriorsoverthese

Section4.4
Section3.5.312.2.Probabilistic peA 583
Becausethisintegration isintractable, wemakeuseoftheLaplaceapproxima­
tion.Ifweassumethattheposterior distribution issharplypeaked,aswilloccurfor
sufficiently largedatasets,thenthere-estimation equations obtained bymaximizing
themarginal likelihood withrespecttoaitakethesimpleform
(12.62)
whichfollowsfrom(3.98),notingthatthedimensionality ofWiisD.Thesere­
estimations areinterleaved withtheEMalgorithm updatesfordetermining Wand
a2•TheE-stepequations areagaingivenby(12.54)and(12.55).Similarly, theM­
stepequation fora2isagaingivenby(12.57). TheonlychangeistotheM-step
equationforW,whichismodified togive
(12.63)
whereA=diag(ai)' ThevalueofI-"isgivenbythesamplemean,asbefore.
IfwechooseM=D-1then,ifallaivaluesarefinite,themodelrepresents
afull-covariance Gaussian, whileifalltheaigotoinfinitythemodelisequivalent
toanisotropic Gaussian, andsothemodelcanencompass allpennissible valuesfor
theeffectivedimensionality oftheprincipal subspace.Itisalsopossibletoconsider

(12.63)
smallervaluesofM,whichwillsaveoncomputational costbutwhichwilllimit
themaximum dimensionality ofthesubspace. Acomparison oftheresultsofthis
algorithm withstandardprobabilistic PCAisshowninFigure12.14.
Bayesian PCAprovides anopportunity toillustrate theGibbssampling algo­
rithmdiscussed inSection11.3.Figure12.15showsanexample ofthesamples
fromthehyperparameters InaiforadatasetinD=4dimensions inwhichthedi­
mensionality ofthelatentspaceisM=3butinwhichthedatasetisgenerated from
aprobabilistic PCAmodelhavingonedirection ofhighvariance, withtheremaining

directions comprising lowvariancenoise.Thisresultshowsclearlythepresence of
threedistinctmodesintheposterior distribution. Ateachstepoftheiteration, oneof
thehyperparameters hasasmallvalueandtheremaining twohavelargevalues,so
thattwoofthethreelatentvariables aresuppressed. DuringthecourseoftheGibbs
sampling, thesolutionmakessharptransitions betweenthethreemodes.
Themodeldescribed hereinvolves aprioronlyoverthematrixW.Afully
Bayesian treatment ofPCA,including priorsover1-",a2,andn,andsolvedus­

ingvariational methods, isdescribed inBishop(1999b). Foradiscussion ofvari­
ousBayesian approaches todetennining theappropriate dimensionality foraPCA
model,seeMinka(2001c).
12.2.4Factoranalysis
Factoranalysisisalinear-Gaussian latentvariablemodelthatiscloselyrelated
toprobabilistic PCA.Itsdefinition differsfromthatofprobabilistic PCAonlyinthat
theconditional distribution oftheobserved variablexgiventhelatentvariablezis

584 12.CONTINUOUS LATENT VARIABLES
•••••• •
••• •••••••••••••••••• ••••••• ••••••
••••••••••••••
Figure12.14'Hinloo'diagrams ofthematrixWinwhicheachelement 01thematrixisdepictedas
asquare(whitelorpositiveandblacklornegative values)whoseareaisproportional
tothemagnitude ofthatelement. Thesynthetic dataselcomprises 300datapointsin
D=10dimensions sampled fromaGaussian distribution havingstandard deviation 1.0
in3directions andstandarddeviation 0.5intheremaining 7directions foradatasetin
D=10dimensions havingAT=3directions withlargervariance thantheremaining 7
directions. Theleft-hand plolshowstheresultIrommaximum likelihood probabilistic PCA,
andtheleft·hand plotshowsthecorresponding resuftfromBayesian peA.Weseehow
theBayesian modelisabletodiscover theappropriate dimensionality bysuppressing the
6surplusdegreesoffreedom.
takentohaveadiagonal ratherthananisotropic covariance sothat
p(xlz)=N(xlWz+1'.\II) (12.64)
whereillisaDxDdiagonal matrix.Notethatthefactoranalysismodel,incommon

withprobabilistic PCA.assumes thattheobserved variables Xl,...,Xoareindepen­
dent.giventhelatentvariable z.Inessence. thefactoranalysis modelisexplaining
theobserved covariance structure ofthedatabyrepresenting theindependent vari­
anceassociated witheachcoordinate inthematrix1J.'andcapturing thecovariance
between variables inthematrixW.Inthefactoranalysis literature. thecolumns
ofW.whichcapturethecorrelations between observed variables. arecalledfaclOr

loadings. andthediagonal elements of1J.'.whichrepresent theindependent noise
variances foreachofthevariables, arecalledllniqllenesses.
Theoriginsoffactoranalysis areasoldasthoseofPCA.anddiscussions of
factoranalysis canbefoundinthebooksbyEveritt(1984).Bartholomew (1987),
andBasilevsky (1994). Linksbetween factoranalysis andPCAwereinvestigated
byLilwley(1953)andAnderson (1963)whoshowed thatatstationary pointsof
thelikelihood function. forafaclOranalysis modelwith1J.'=(121,thecolumnsof
Warescaledeigenvectors ofthesamplecovariance matrix.and(12istheaverage

ofthediscarded eigenvalues. Later.Tipping andBishop(1999b) showedthatthe
maximum oftheloglikelihood function occurswhentheeigenvectors comprising
Warechosentobetheprincipal eigenvectors.
Making useof(2.115). weseethatthemarginal distribution fortheobserved

12.2.l'ru":ohilislk I'CA 585
FllIure12.15 Gillbs.,,,,>p!j"lllo< Bay<lslan
PCAsh<Ming plotsojIno,
versus ~eralionnumberbr
three"values. showing
tr"""tions betw..... tbe
th"'"moOts <A!heposterior
,-"riabl,i'gi,-,nby1'(x)~N(Xlj',C)whe...now
C=WWT+'i'. (12,6~)
Eurr"e 12,19
S"na"12.4Aswithprobabilistic PCA,thi,moMIisim-"ri.rrl to'Olalions in11><0latent'pace.
Histoocally, factoranal)',;s hasbeenlhe,ubjerlofcOl1tro,-ersy wroe"a!tempt<
h",-ebttn"'a<k:toplaceanintc'P"'t"lioo ontheind;vidual faclon(thecOOfdinates
inz_space). whichh3.\pm"enproblematic duetolr.e"""i<lcmifiabilily offactOf
analysisassocimed withMation' inthis'pace.Fromoorperspeoh-e, howe,-er. we
shall.iewfactoranalysisasaformoflalent"ariabledensilymodel.inwhichthe
formoftl>clalent'pacei'ofinterestbutnO!theparticular choiccofcoordinates
usedtodescrit>cil.Ifwewishtoremovethedegeneracy a'sociated withlatent'pace
roIations. ""emu,tcon'ider non-Gaussian latent,-"riabledi'tribution,. gi"irrgrise10
independent component .n.lysi,(lCA)models.

Wecandetenni"e theparameters I'.\V."nd....inthefac!Ofan.ly,i,modelby
muimum likelihood. 11..solutionforI'i'ag"ingivenbythe",,,,pie"'ean.How·
eyC'. "nli~eprobabili,tic l'CA.lllcre i'nolongeraclosed-form maximum likelihood
solutionfor\V.",'hichmu.\ltherdorc befoundi'er.li,'c1)'_ Becausefacloranal)',i.is
alatentvariablemodeLthi'canbedon.usinganEMalgorilhm (R.binandThayer.
1982)!h"tis"nalogou, totheoneused(Ofpml>;lbili.tie PeA.Specihcally. lheE-'lep
eqnJtioo, areg;'-enby
where""eh",'edefi""dE[zoJ=GWT",-'(xn-xl

E[z"z~J _ G+E[zo]E[z,,]T(12,66)
(1267)
(l2,6H)
NOieth"tthi'i'e.pre,<ed inafor'"thaiin,-oh'esinycrsi"n ofmalrices"fSilO,\Ix,If
rathe'lhan DxD(ex"",,,,fortbeDxDdiagooal matrixoJ'"'hosein,-ersei.''ri"ial

586 12.CONTINUOUS LATENT VARIABLES
Exercise 12.22tocompute inO(D)steps),whichisconvenient becauseoftenM«D.Similarly,
theM-step equations taketheform
wnew[~(x"-XllllIZn]"] [~Ill[Znz~I]-'
diag{s-W.'w~~1ll[Zn](Xn _xl"}(12.69)
(12.70)
wherethe'diag'operatorsetsallofthenondiagonal elements ofamatrixtozero.A
Bayesian treatment ofthefactoranalysismodelcanbeobtainedbyastraightforward
application ofthetechniques discussed inthisbook.
Anotherdifference betweenprobabilistic PCAandfactoranalysisconcerns their
Exercise 12.25 different behaviour undertransformations ofthedataset.ForPCAandprobabilis­
ticPCA,ifwerotatethecoordinate systemindataspace,thenweobtainexactly
thesamefittothedatabutwiththeWmatrixtransformed bythecorresponding
rotationmatrix.However, forfactoranalysis, theanalogous property isthatifwe
makeacomponent-wise re-scaling ofthedatavectors,thenthisisabsorbed intoa
corresponding re-scaling oftheelements of\)i.
12.3.KernelpeA
InChapter6,wesawhowthetechnique ofkernelsubstitution allowsustotakean

algorithm expressed intermsofscalarproducts oftheformxTx'andgeneralize
thatalgorithm byreplacing thescalarproducts withanonlinear kernel.Herewe
applythistechnique ofkernelsubstitution toprincipal component analysis, thereby
obtaining anonlinear generalization calledkernelpeA(Scholkopf etal.,1998).
Consider adataset{xn}ofobservations, wheren=1,...,N,inaspaceof
dimensionality D.Inordertokeepthenotationuncluttered, weshallassumethat

wehavealreadysubtracted thesamplemeanfromeachofthevectors Xn,sothatLnXn=O.Thefirststepistoexpressconventional PCAinsuchaformthatthe
datavectors{xn}appearonlyintheformofthescalar products x~Xm.Recallthat
theprincipal components aredefinedbytheeigenvectors Uiofthecovariance matrix
SUi=AiUi (12.71)
wherei=1,...,D.HeretheDxDsample covariance matrixSisdefinedby
(12.72)
andtheeigenvectors arenormalized suchthatuTUi=1.
Nowconsider anonlinear transformation ¢(x)intoanM-dimensional feature

(12.72)
space,sothateachdatapointXnistherebyprojected ontoapoint¢(xn).Wecan

12.3.K~mcil'Co'. 587
...'\
(12.73)Figu'.12.16 SctIematic _,.lion 01kernelPeA.A<UtllHIInlheOflglnal<Utaspacel~'_ plot}..
PfOlEledby'"""'*- tranllklfmalion ~,,}1nIo.fa.tur.space(fIght_ plot).ByI*b'~ PCAinthe
!Hue111**.weoblaO'Ilha pmeiIlaI ~"llS. "'who<:tllha tntIe.........inblUe.,..,..oJano4ed bylha_v,Thagr-.....InIMIuNapamindicMaIha_plOlKlio". onIOlheIirslpoiridl* '""".........11."""'*'_od 110""'*- poOf&Cllu. in....<lfisIonaIOillll111**.HoleIIuIIIngMefM' lanolpox'..110
'...... 1ha,...iIio_poi........00i'........by._1n "apam.
__ptrform)l-.bnlPeAillfnlllKlopICe...-Iudl,mpIiclIly «lIi'Il'S•-'_
prinClpaI ....-..modelillonpnllcbuo~as,1lU>tr*d inFllIft12-16.
Fu"IO..It..lei'"lOS!oUlllt1NlIllt~diu.~lObohalnro mean,
jI)!halL.4>("'.J..O.WedWlrctlll'1l10Itl,~pol'".Ihonly.1llt.1f".If"'"""*
CO\-.ullCC mMfU,n(~.If*'e ,~l""by,
C -.~.L<>(x.j4>(X.)T..,
and,l~",,,,n'"MOl" opan,ioni'«linedby
Cv,=A,v, (12,74)
; =1...,.M.Ourgoalis10soh'"lhiseigen"lIlue problem WilhoUlha"inlllO work

."plici,ly in,hef.lIture'pace.From!hedefinilion ofC.lhe.ill"""""l"'" equalions
lell'U,thaIY,!-ali,fies."sL<bC".){<b(x.lTv,}-)"Y,..,(12.7~)
..........-".-lN1(proo.idcd A,>0)tilt"CC'lorv,isli''nby•Ii_rombllla"on
ofIlltd>(J.....JO<;.-he"-"llmiIllhc(orm,
v,'"'L11,.4>(".)....(12.76)

588 12.CONTINUOUS LATENT VARIABLES
Substituting thisexpansion backintotheeigenvector equation, weobtain
1N N N
NL¢(xn)¢(xn)T Laim¢(Xm)=AiLain¢(Xn),
n=l m=l n=l(12.77)
Thekeystepisnowtoexpressthisintermsofthekernelfunctionk(xn,xm)=
¢(Xn)T¢(xm),whichwedobymultiplying bothsidesby¢(xZ)Ttogive
1N m N
NLk(XI'X n)Laimk(Xn,x m)=AiLaink(XI'X n),
n=l m=l n=l
Thiscanbewritteninmatrixnotationas(12.78)
(12.79)
whereaiisanN-dimensional columnvectorwithelements aniforn=1,...,N.
Wecanfindsolutions foraibysolvingthefollowing eigenvalue problem
(12.80)
Exercise 12.26inwhichwehaveremoved afactorofKfrombothsidesof(12.79). Notethat
thesolutions of(12.79)and(12.80)differonlybyeigenvectors ofKhavingzero
eigenvalues thatdonotaffecttheprincipal components projection.
Thenormalization condition forthecoefficients aiisobtained byrequiring that
theeigenvectors infeaturespacebenormalized. Using(12.76)and(12.80),wehave
N N
1=V;Vi=LLainaim¢(xn)T¢(x m)=a;K~=AiNa;ai' (12.81)
n=lm=l

(12.80)
N N
n=lm=l
Havingsolvedtheeigenvector problem, theresulting principal component pro­
jectionscanthenalsobecastintermsofthekernelfunctionsothat,using(12.76),
theprojection ofapointxontoeigenvector iisgivenby
N N
Yi(X)=¢(x)TVi=Lain¢(x)T ¢(xn)=Laink(X,xn)(12.82)
n=l n=l
andsoagainisexpressed intermsofthekernelfunction.
IntheoriginalD-dimensional xspacethereareDorthogonal eigenvectors and
hencewecanfindatmostDlinearprincipal components. Thedimensionality M
ofthefeaturespace,however, canbemuchlargerthanD(eveninfinite), andthus

N N
n=l n=l
wecanfindanumberofnonlinear principal components thatcanexceedD.Note,
however, thatthenumberofnonzeroeigenvalues cannotexceedthenumberNof
datapoints,because(evenifM>N)thecovariance matrixinfeaturespacehas
rankatmostequaltoN.Thisisreflected inthefactthatkernelPCAinvolvesthe
eigenvector expansion oftheNxNmatrixK.

12.3.KernelPCA 589
Sofarwehaveassumed thattheprojected datasetgivenby¢(xn)haszero
mean,whichingeneralwillnotbethecase.Wecannotsimplycompute andthen
subtractoffthemean,sincewewishtoavoidworkingdirectlyinfeaturespace,and
soagain,weformulate thealgorithm purelyin-!ermsofthekernelfunction. The
projected datapointsaftercentralizing, denoted¢(xn),aregivenby
andthecorresponding elements oftheGrammatrixaregivenby
Knm=¢(xn)T¢(xm)
1N
¢(xn)T¢(x m)-NL¢(xn)T¢(xZ)
Z=l
1N 1N N
- NL¢(XZ)T¢(X m)+N2LL¢(Xj)T¢(xZ)
Z=l j=lZ=l
1N
k(xn,xm) -NLk(xz,xm)
Z=l
1N 1N N
- NLk(xn,xz) +N2LLk(Xj,Xl)'
Z=l j=l1=1
Thiscanbeexpressed inmatrixnotationas(12.83)
(12.84)
(12.85)
Exercise 12.27whereINdenotestheNxNmatrixinwhicheveryelementtakesthevaluel/N.
~ ~
ThuswecanevaluateKusingonlythekernelfunctionandthenuseKtodetermine
theeigenvalues andeigenvectors. Notethatthestandard PCAalgorithm isrecovered
asaspecialcaseifweusealinearkernelk(x,x')=xTx/.Figure12.17showsan
exampleofkernelPCAappliedtoasynthetic dataset(Scholkopf etal.,1998).Here

Z=l j=lZ=l
1N
Z=l
1N 1N N
Z=l j=l1=1
(12.84)
(12.85)
~ ~
a'Gaussian' kerneloftheform
k(x,x')=exp(-llx-x/112/0.1) (12.86)
isappliedtoasynthetic dataset.Thelinescorrespond tocontours alongwhichthe
projection ontothecorresponding principal component, definedby
isconstant.N
¢(X?Vi =Laink(X,xn)
n=l(12.87)

590 12.CONTINUOUS LATENT VAlUABLES_.
(12.88)Figure12.11E"llmple 01kernelPCA,withaGaussian kernelawIiOO10asynthetic <latasatintwo<:Iirnensions,
showing !hefirslflighteigenfunclions along w~hl!>eire9tnvailNls. Theoootours amlinesalongwhich!he
projoc1ion onlot""COffaspMding principal componen1ls co<>stam, NolahawIhefirsltwo~ ....,..rat.
!heth"'"dusters. !he""'"Ill"'"~ spIiI""'*'oIlheeluste,intohaMoS.andt""lolIowing Ihree~ again spI~!heduste,"intohalvesalongdirections orthogonal 10thoprEMouS splils,
Oneobvioo' dls.aJmota~e ofI:emel!'CAIsthafifinvoh'esfindinglheelgen"e<­
torsoftheNxNmalri>:KraW.IhanlheDxDmalri,Sofcor,..emionallinear
!'CA.and!iOInprac1lceforlargedata"'1'appro,lmation< areoftenuS(:d
Finally.""eOOIethati"<tandard linearI'CA,weoftenretainsomeredoce<lnum·
berL<Dofeigenvectors andthenappro,lmale 0datavttl<:>rXnb}'itsprojection
i~0,,1"lheL-dimensional principal subspace, definedby,
i~-L:«",)""
I"kernell'CA. thiswillingencr~1notbeflO'slble, Toseethl',OOIeIhatthemap­

ping4'(x)mapstheD-dimensional xspacei"t"0D-dimensioo.l manijQiII inlhe
M-dimemioo.l femurespace <1>.TlIe:.'ectorxi'koowna<lhef'",.imagr oflhe
c","""ponding poi"l4'(x).However, fheprojec1ioo ofpoinl>infeature<J'3C"""to
thelinearrcA,ub,p""" inthat'pacewilltypically"''' lieOnfhenonlinear D­
dimensional manifold and!iOwillnulha.,.ac"""",pondlng p",.lmo~e indOlOspa<."C,
Technlque< ho.-elherefore bttnproposed forfindingapproximale pre-image< iB""lr
Nat..2(04).

12.4.Nonlinear LatentVariable Models 591
12.4.Nonlinear LatentVariable Models
Exercise 12.28Inthischapter,wehavefocussed onthesimplestclassofmodelshavingcontinuous
latentvariables, namelythosebasedonlinear-Gaussian distributions. Aswellas
havinggreatpractical importance, thesemodelsarerelatively easytoanalyseand
tofittodataandcanalsobeusedascomponents inmorecomplex models. Here
weconsiderbrieflysomegeneralizations ofthisframework tomodelsthatareeither
nonlinear ornon-Gaussian, orboth.
Infact,theissuesofnonlinearity andnon-Gaussianity arerelatedbecausea
generalprobability densitycanbeobtained fromasimplefixedreference density,
suchasaGaussian, bymakinganonlinear changeofvariables. Thisideaformsthe
basisofseveralpractical latentvariablemodelsasweshallseeshortly.
12.4.1Independent component analysis
Webeginbyconsidering modelsinwhichtheobserved variables arerelated
linearlytothelatentvariables, butforwhichthelatentdistribution isnon-Gaussian.

Animportant classofsuchmodels,knownasindependent component analysis, or
leA,ariseswhenweconsider adistribution overthelatentvariables thatfactorizes,
sothat
M
p(z)=IIp(Zj).
j=l(12.89)
Tounderstand theroleofsuchmodels,consider asituation inwhichtwopeople
aretalkingatthesametime,andwerecordtheirvoices using twomicrophones.
Ifweignoreeffectssuchastimedelayandechoes,thenthesignalsreceived by
themicrophones atanypointintimewillbegivenbylinearcombinations ofthe

sothat
M
amplitudes ofthetwovoices.Thecoefficients ofthislinearcombination willbe
constant, andifwecaninfertheirvaluesfromsampledata,thenwecaninvertthe
mixingprocess(assuming it isnonsingular) andtherebyobtaintwocleansignals
eachofwhichcontainsthevoiceofjustoneperson.Thisisanexampleofaproblem
calledblindsourceseparation inwhich'blind'referstothefactthatwearegiven
onlythemixeddata,andneithertheoriginalsourcesnorthemixingcoefficients are
observed (Cardoso, 1998).
Thistypeofproblem issometimes addressed usingthefollowing approach

(MacKay, 2003)inwhichweignorethetemporal natureofthesignalsand treat the
successive samplesasi.i.d.Weconsider agenerative modelinwhichtherearetwo
latentvariables corresponding totheunobserved speechsignalamplitudes, andthere
aretwoobserved variables givenbythesignalvaluesatthemicrophones. Thelatent
variables haveajointdistribution thatfactorizes asabove,andtheobserved variables
aregivenbyalinearcombination ofthelatentvariables. Thereisnoneedtoinclude
anoisedistribution becausethenumberoflatentvariables equalsthenumberofob­

servedvariables, andtherefore themarginal distribution oftheobserved variables
willnotingeneralbesingular, sotheobserved variables aresimplydeterministic
linearcombinations ofthelatentvariables. Givenadatasetofobservations, the

592 12.CONTINUOUS LATENT VARIABLES
Exercise 12.29likelihood function forthismodelisafunctionofthecoefficients inthelinearcom­
bination. Theloglikelihood canbemaximized usinggradient-based optimization
givingrisetoaparticular versionofindependent component analysis.
Thesuccessofthisapproach requiresthatthelatentvariables havenon-Gaussian
distributions. Toseethis,recallthatinprobabilistic PCA(andinfactoranalysis) the
latent-space distribution isgivenbyazero-mean isotropic Gaussian. Themodel
therefore cannotdistinguish between twodifferent choicesforthelatentvariables
wherethesediffersimplybyarotationinlatentspace.Thiscanbeverifieddirectly
bynotingthatthemarginal density(12.35), andhencethelikelihood function, is
unchanged ifwemakethetransformation W-)WRwhereRisanorthogonal
matrixsatisfyingRRT=I,becausethematrixCgivenby(12.36)isitselfinvariant.
Extending themodeltoallowmoregeneralGaussian latentdistributions doesnot
changethisconclusion because, aswehaveseen,suchamodelisequivalent tothe

zero-mean isotropic Gaussian latentvariablemodel.
AnotherwaytoseewhyaGaussian latentvariabledistribution inalinearmodel
isinsufficient tofindindependent components istonotethattheprincipal compo­
nentsrepresent arotationofthecoordinate systemindataspacesuchastodiagonal­
izethecovariance matrix,sothatthedatadistribution inthenewcoordinates isthen
uncorrelated. Although zerocorrelation isanecessary condition forindependence
itisnot,however, sufficient. Inpractice, acommon choiceforthelatent-variable

distribution isgivenby
1p(z)=--,.-----,-
J7fcosh(zj)1(12.90)
whichhasheavytailscompared toaGaussian, reflecting theobservation thatmany
real-world distributions alsoexhibitthisproperty.
TheoriginalICAmodel(BellandSejnowski, 1995)wasbasedontheoptimiza­
tionofanobjective function definedbyinformation maximization. Oneadvantage
ofaprobabilistic latentvariableformulation isthatithelpstomotivate andformu­
lategeneralizations ofbasicICA.Forinstance, independent factoranalysis (Attias,

1999a)considers amodelinwhichthenumberoflatentandobserved variables can
differ,theobserved variables arenoisy,andtheindividual latentvariables haveflex­
ibledistributions modelled bymixtures ofGaussians. Theloglikelihood forthis
modelismaximized usingEM,andthereconstruction ofthelatentvariables isap­
proximated usingavariational approach. Manyothertypesofmodelhavebeen
considered, andthereisnowahugeliterature onICAanditsapplications (Jutten
andHerault, 1991;Comonetat.,1991;Amarietat.,1996;PearlmutterandParra,

1997;Hyvarinen andOja,1997;Hintonetat.,2001;MiskinandMacKay, 2001;
Hojen-Sorensen etat.,2002;Choudrey andRoberts, 2003;Chanetat.,2003;Stone,
2004).
12.4.2Autoassociative neural networks
InChapter5weconsidered neuralnetworks inthecontextofsupervised learn­
ing,wheretheroleofthenetwork istopredicttheoutputvariables givenvalues

12.4.Nonlinear LatentVariable Models 593
Figure12.18Anautoassociative mUltilayer perceptron having
twolayersofweights.Suchanetworkistrainedto
mapinputvectorsontothemselves byminimiza­
tionotasum-ot-squares error.Evenwithnon­
linearunitsinthehiddenlayer,suchanetwork
isequivalent tolinearprincipalcomponent anal­
ysis.Linksrepresenting biasparameters have
beenomittedforclarity.inputs outputs
(12.91)fortheinputvariables. However, neuralnetworks havealsobeenappliedtoun­
supervised learningwheretheyhavebeenusedfordimensionality reduction. This
isachieved byusinganetwork havingthesamenumberofoutputsasinputs,and
optimizing theweightssoastominimize somemeasureofthereconstruction error
betweeninputsandoutputswithrespecttoasetoftrainingdata.
Consider firstamultilayer perceptron oftheformshowninFigure12.18,hav­
ingDinputs,DoutputunitsandMhiddenunits,withM<D.Thetargetsused
totrainthenetworkaresimplytheinputvectorsthemselves, sothatthenetwork
isattempting tomapeachinputvectorontoitself.Suchanetworkissaidtoform

anautoassociative mapping. Sincethenumberofhiddenunitsissmallerthanthe
numberofinputs,aperfectreconstruction ofallinputvectorsisnotingeneralpos­
sible.Wetherefore determine thenetworkparameters wbyminimizing anerror
functionwhichcapturesthedegreeofmismatch betweentheinputvectorsandtheir
reconstructions. Inparticular, weshallchooseasum-of-squares erroroftheform
1N
E(w)="2LIly(xn,w)-xn112
•
n=l
Ifthehiddenunitshavelinearactivations functions, then itcanbeshownthatthe
errorfunctionhasauniqueglobalminimum, andthatatthisminimum thenetwork

1N
•
n=l
performs aprojection ontotheM-dimensional subspace whichisspannedbythefirst
Mprincipal components ofthedata(Bourlard andKamp,1988;BaldiandHornik,
1989).Thus,thevectorsofweightswhichleadintothehiddenunitsinFigure12.18
formabasissetwhichspanstheprincipal subspace. Note,however, thatthesevec­
torsneednotbeorthogonal ornormalized. Thisresultisunsurprising, sinceboth
principal component analysisandtheneuralnetworkareusinglineardimensionality
reduction andareminimizing thesamesum-of-squares errorfunction.

Itmightbethoughtthatthelimitations ofalineardimensionality reduction could
beovercome byusingnonlinear (sigmoidal) activation functions forthehiddenunits
inthenetworkinFigure12.18.However, evenwithnonlinear hiddenunits,themin­
imum error solutionisagaingivenbytheprojection ontotheprincipal component
subspace (Bourlard andKamp,1988).Thereistherefore noadvantage inusingtwo­
layerneuralnetworks toperformdimensionality reduction. Standard techniques for

principal component analysis(basedonsingularvaluedecomposition) areguaran­
teedtogivethecorrectsolutioninfinitetime,andtheyalsogenerate anorderedset
ofeigenvalues withcorresponding orthonormal eigenvectors.

594 12.CONTINUOUS LATENT VARIABLES
Figure12.19Addition ofextrahiddenlay­
ersofnoolinear unitsgivesan
auloassocialive networkwhich
canperformanoolinear dimen­
siooalityreduction.
inputs
x,F,•F,•
outputs
x,
Thesituation isdifferent, however.ifadditional hiddenlayersarepcrmillcd in
thenetwork. Consider thefour-layer autoassociativc networkshowninFigure12.19.
Againtheoutputunitsarelinear,andtheMunitsinthesecondhiddenlayercanalso
belinear.however, thefirstandthirdhiddenlayershavesigmoidal nonlinear activa­
tionfunctions. Thenetworkisagaintrainedbyminimization oftheerrorfunction
(12.91). Wecanviewthisnetworkastwosuccessive functional mappings F]and
F2,asindicated inFigure12.19.Thefirstmapping F]projectstheoriginalD­
dimensional dataontoanAI-dimensional subspaceSdefinedbytheactivations of
theunitsinthesecondhiddenlayer.Becauseofthepresenceofthefirsthiddenlayer
ofnonlinear units.thismapping isverygeneral.andinparticular isnotrestricted to
beinglinear.Similarly. thesecondhalfofthenetworkdefinesanarbitrary functional

mapping fromtheM-dimensional spacebackintotheoriginalD-dimensional input
space.Thishasasimplegeometrical interpretation. asindicated forthecaseD=3
andM=2inFigure12.20.
Suchanetworkeffectively perfonns anonlinear principal component analysis.
X3"F,
•
x,"
Figure12.20Geometrical interpretation ofthemappings performed bythenetworkinFigure12.1gforthecase
of0=3inputsandAI=2unitsinthemiddlehiddenlayer.ThefunctionF,mapsfromanM-dimensional
spaceSintoaD-dimensiooal spaceandtherefore definesthewayinwhichthespaceSisembedded withinthe

X3"F,
•
x,"
originalx-space. SincethemappingF,canber"I()(llinear, theembedding 01Scanbenonplanar, asindicated
inthefigure.Themapping F.thendefinesaprojectiorl ofpointsintheoriginalD-dimensional spaceintothe
M-dimensional subspace S.

12.4.Nonlinear LatentVariable Models 595
Ithastheadvantage ofnotbeinglimitedtolineartransformations, although itcon­
tainsstandard principal component analysisasaspecialcase.However, training
thenetworknowinvolvesanonlinear optimization problem, sincetheerrorfunction
(12.91)isnolongeraquadratic functionofthenetworkparameters. Computation­
allyintensive nonlinear optimization techniques mustbeused,andthereistheriskof
findingasuboptimal localminimum oftheerrorfunction. Also,thedimensionality
ofthesubspace mustbespecified beforetrainingthenetwork.
12.4.3Modelling nonlinear manifolds
Aswehavealreadynoted,manynaturalsourcesofdatacorrespond tolow­
dimensional, possibly noisy,nonlinear manifolds embedded withinthehigherdi­
mensional observed dataspace.Capturing thisproperty explicitly canleadtoim­
proveddensitymodelling compared withmoregeneralmethods. Hereweconsider
brieflyarangeoftechniques thatattempttodothis.
Onewaytomodelthenonlinear structure isthroughacombination oflinear

models,sothatwemakeapiece-wise linearapproximation tothemanifold. Thiscan
beobtained, forinstance, byusingaclustering technique suchasK-meansbasedon
Euclidean distancetopartition thedatasetintolocalgroupswithstandardPCAap­
pliedtoeachgroup.Abetterapproach istousethereconstruction errorforcluster
assignment (Kambhatla andLeen,1997;Hintonetal.,1997)asthen acommon cost
function isbeingoptimized ineachstage.However, theseapproaches stillsuffer
fromlimitations duetotheabsenceofanoveralldensitymodel.Byusingprob­

abilisticPCAitisstraightforward todefineafullyprobabilistic modelsimplyby
considering amixturedistribution inwhichthecomponents areprobabilistic PCA
models(Tipping andBishop,1999a).Suchamodelhasbothdiscretelatentvari­
ables,corresponding tothediscretemixture, aswellascontinuous latentvariables,
andthelikelihood function canbemaximized usingtheEMalgorithm. Afully
Bayesian treatment, basedonvariational inference (BishopandWinn,2000),allows

thenumberofcomponents inthemixture, aswellastheeffective dimensionalities
oftheindividual models,tobeinferredfromthedata.Therearemanyvariantsof
thismodelinwhichparameters suchastheWmatrixorthenoisevariances aretied
acrosscomponents inthemixture, orinwhichtheisotropic noisedistributions are
replacedbydiagonal ones,givingrisetoamixtureoffactoranalysers (Ghahramani
andHinton,1996a;Ghahramani andBeal,2000).Themixtureofprobabilistic PCA
modelscanalsobeextended hierarchically toproduceaninteractive datavisualiza­

tionalgorithm (BishopandTipping, 1998).
Analternative toconsidering amixtureoflinearmodelsistoconsider asingle
nonlinear model.Recallthatconventional PCAfindsalinearsubspace thatpasses
closetothedatainaleast-squares sense.Thisconceptcanbeextended toone­
dimensional nonlinear surfacesintheformofprincipal curves(HastieandStuetzle,
1989).WecandescribeacurveinaD-dimensional dataspaceusingavector-valued
functionf().),whichisavectoreachofwhoseelements isafunctionofthescalar)..
Therearemanypossiblewaystoparameterize thecurve,ofwhichanaturalchoice

isthearclengthalongthecurve.Foranygivenpointxindataspace,wecanfind
thepointonthecurvethatisclosestinEuclidean distance. Wedenotethispointby

596 12.CONTINUOUS LATENT VARIABLES
>..=gf(X)becauseitdepends ontheparticular curvef(>").Foracontinuous data
densityp(x),aprincipal curveisdefinedasoneforwhicheverypointonthecurve
isthemeanofallthosepointsindataspacethatprojecttoit,sothat
JE[Xlgf(X)=>..]=f(>"). (12.92)
Foragivencontinuous density,therecanbemanyprincipal curves.Inpractice, we
areinterested infinitedatasets,andwealsowishtorestrictattention tosmooth
curves.HastieandStuetzle(1989)proposeatwo-stage iterativeprocedure forfind­
ingsuchprincipal curves,somewhat reminiscent oftheEMalgorithm forPCA.The
curveisinitialized usingthefirstprincipal component, andthenthealgorithm alter­
natesbetween adataprojection stepandcurvere-estimation step.Intheprojection
step,eachdatapointisassigned toavalueof>..corresponding totheclosestpoint
onthecurve.Theninthere-estimation step,eachpointonthecurveisgivenby
aweighted averageofthosepointsthatprojecttonearbypointsonthecurve,with
pointsclosestonthecurvegiventhegreatestweight.Inthecasewherethesubspace

isconstrained tobelinear,theprocedure converges tothefirstprincipal component
andisequivalent tothepowermethodforfindingthelargesteigenvector oftheco­
variance matrix.Principal curvescanbegeneralized tomultidimensional manifolds
calledprincipal surfaces although thesehavefoundlimiteduseduetothedifficulty
ofdatasmoothing inhigherdimensions evenfortwo-dimensional manifolds.
PCAisoftenusedtoprojectadatasetontoalower-dimensional space,forex­
ampletwodimensional, forthepurposes ofvisualization. Anotherlineartechnique

withasimilaraimismultidimensional scaling,orMDS(CoxandCox,2000).Itfinds
alow-dimensional projection ofthedatasuchastopreserve, ascloselyaspossible,
thepairwise distances between datapoints,andinvolvesfindingtheeigenvectors of
thedistancematrix.Inthecasewherethedistances areEuclidean, itgivesequivalent
resultstoPCA.TheMDSconceptcanbeextended toawidevarietyofdatatypes
specified intermsofasimilarity matrix,givingnonmetric MDS.
Twoothernonprobabilistic methods fordimensionality reduction anddatavi­
sualization areworthyofmention. Locallylinearembedding, orLLE(Roweisand

Saul,2000)firstcomputes thesetofcoefficients thatbestreconstructs eachdata
pointfromitsneighbours. Thesecoefficients arearranged tobeinvariant torota­
tions,translations, andscalingsofthatdatapointanditsneighbours, andhencethey
characterize thelocalgeometrical properties oftheneighbourhood. LLEthenmaps
thehigh-dimensional datapointsdowntoalowerdimensional spacewhilepreserv­
ingtheseneighbourhood coefficients. Ifthelocalneighbourhood foraparticular

datapointcanbeconsidered linear,thenthetransformation canbeachieved using
acombination oftranslation, rotation, andscaling,suchastopreserve theangles
formedbetween thedatapointsandtheirneighbours. Because theweightsarein­
varianttothesetransformations, weexpectthesameweightvaluestoreconstruct the
datapointsinthelow-dimensional spaceasinthehigh-dimensional dataspace.In
spiteofthenonlinearity, theoptimization forLLEdoesnotexhibitlocalminima.
Inisometric featuremapping, orisomap(Tenenbaum etai.,2000),thegoalis

toprojectthedatatoalower-dimensional spaceusingMDS,butwherethedissim­
ilaritiesaredefinedintermsofthegeodesic distances measured alongthemani-

Chapter5
ChapterJJ12.4.Nonlinear LatentVariable Models 597
fold.Forinstance, iftwopointslieonacircle,thenthegeodesic isthearc-length
distancemeasured aroundthecircumference ofthecirclenotthestraightlinedis­
tancemeasured alongthechordconnecting them.Thealgorithm firstdefinesthe
neighbourhood foreachdatapoint,eitherbyfindingtheKnearestneighbours orby
findingallpointswithinasphereofradius E.Agraphisthenconstructed bylink­
ingallneighbouring pointsandlabelling themwiththeirEuclidean distance. The
geodesic distancebetween anypairofpointsisthenapproximated bythesumof
thearclengthsalongtheshortestpathconnecting them(whichitselfisfoundusing
standardalgorithms). Finally,metricMDSisappliedtothegeodesic distancematrix
tofindthelow-dimensional projection.
Ourfocusinthischapterhasbeenonmodelsforwhichtheobserved vari­
ablesarecontinuous. Wecanalsoconsider modelshavingcontinuous latentvari­
ablestogetherwithdiscreteobserved variables, givingrisetolatenttraitmodels

(Bartholomew, 1987).Inthiscase,themarginalization overthecontinuous latent
variables, evenforalinearrelationship betweenlatentandobserved variables, can­
notbeperformed analytically, andsomoresophisticated techniques arerequired.
Tipping(1999)usesvariational inference inamodelwithatwo-dimensional latent
space,allowing abinarydatasettobevisualized analogously totheuseofPCAto
visualize continuous data.NotethatthismodelisthedualoftheBayesian logistic
regression problemdiscussed inSection4.5.Inthecaseoflogisticregression we

haveNobservations ofthefeaturevector<l>nwhichareparameterized byasingle
parameter vectorw,whereasinthelatentspacevisualization modelthereisasingle
latentspacevariablex(analogous to<1»andNcopiesofthelatentvariable Wn.A
generalization ofprobabilistic latentvariablemodelstogeneralexponential family
distributions isdescribed inCollinsetal.(2002).
Wehavealreadynotedthatanarbitrary distribution canbeformedbytakinga
Gaussian randomvariableandtransforming itthroughasuitablenonlinearity. This
isexploited inagenerallatentvariablemodelcalledadensitynetwork(MacKay,

1995;MacKay andGibbs,1999)inwhichthenonlinear function isgoverned bya
multilayered neuralnetwork.Ifthenetworkhasenoughhiddenunits,itcanapprox­
imateagivennonlinear functiontoanydesiredaccuracy. Thedownside ofhaving
suchaflexiblemodelisthatthemarginalization overthelatentvariables, requiredin
ordertoobtainthelikelihood function, isnolongeranalytically tractable. Instead,
thelikelihood isapproximated usingMonteCarlotechniques bydrawingsamples
fromtheGaussian prior.Themarginalization overthelatentvariables thenbecomes

asimplesumwithonetermforeachsample. However, becausealargenumber
ofsamplepointsmayberequiredinordertogiveanaccuraterepresentation ofthe
marginal, thisprocedure canbecomputationally costly.
Ifweconsidermorerestricted formsforthenonlinear function, andmakeanap­
propriate choiceofthelatentvariabledistribution, thenwecanconstruct alatentvari­
ablemodelthatisbothnonlinear andefficienttotrain.Thegenerative topographic
mapping, orGTM(BishopetaI.,1996;BishopetaI.,1997a;BishopetaI.,1998b)
usesalatentdistribution thatisdefinedbyafiniteregulargridofdeltafunctions over

the(typically two-dimensional) latentspace.Marginalization overthelatentspace
thensimplyinvolvessumming overthecontributions fromeachofthegridlocations.

598 12.CONTINUOUS LATENT VAK1AHU':S
.•.'"
FllIu.e12.21?lotottrleoillkYw<:lataWllisualiz.ed usingPeAontheleftandGTMonItlengr,tFOftileGTM
model.each<latapoinIlsplollfldattilemeanotitsposM'k>< dislribution in..tents;>ace,Tile"","ineanty mlhe
GTM1TlOd8I._lha sepamlion betwoon thegroupsofdatapointstobe.....n""".ckl.arfy,
Ch"l'l~f j
S~etioo/.4Theno"liotar mapping isgi,'enbyalinearregression modelthaIallow,forgeneral
IIO/llinearily whilebeingalinearfuoctionoftileadapli'-e parameler<, NOIethaItilt
usuallimitation oflinearregression modelsarisingfromtheen"",ofdimen,iooalily
does1101ariseintheContr~1oflheGT~Isi""'ethe"\3nifold generall)' ha<t,,'odi"ltn·
sionsirrespecti'-e ofthedimensionality ofthedataspace,Acoo""!",,nce ofIllese
11"0cooicesisthatthelikelihood funclioncanbee~pressed analytically indosed
formandcanbeoptimilC<.! efficiently o,ingtheEMalgorithm_ Theresolting GTM
modelhIsalwo-dimensional nonlinear manifold 10tiledalasel.andbye"alualing

theposterior distrilJ",lion (Werlatentspaceforthedatapoi"",theycanheprojectt<J
backtothelalent'JI'K'"for.'isualilalion purposes, Figure12,21sl"""sacomparison
oftheoildata..,1"isualired wilhlincarPeAandwilhlheIIO/lhnear GT~I,
TIltGTMcanbeseenasaprobabilistic "'rsionofanearliernlOd<lcallMthe'''If
org"nidng ""'p.orSOM(Kohonen. 1982:Kobonen. (995).whichalsorepresents
aIwo-dimensiooal IIO/llincar manifoid asaregulararrayofdisc"'lepoints.The
SOMi'somewhat remin;""'nt oftheK·trlCan, algorithm inthatdatapointsare

a.,igr.ed tonearbyProlOl)'j>C '-eclonthaIarelhensubsc<juenlly updale<!. Initially.
lheproIOI)'jl('S aredistribuled atrandom,andduringthetrainingprocessthey'selr
organize' soastoaPl'ro~imale asmoolhmanifold. Unlike K-mean'.how'e"e..the
SOMisTIOIoptimizing anywell.ddine<! costfunction(Erwin..al..1992)making"
difficult tos."theparameters ofthemodeland10assesscon'-ergence. Therei'also
noguarantee thatthe'",If-<>rganilalion' willtakeplace..thisisdepen""nl 00the

choiceofappropriate paranlttcr "aloC'f,,,anyparticular datasel.
ByOOfItrast, GTMoptimize, theloglikelihood functioo, andtheresolting model
define'aprobabilily den,ityindma,pace,InfaeLilcorre,ponds toacon,m,incd
mi,tureofGaussian, inwhichthecomponent.' ,h.reaCOnlnlOn ".riance.•ndthe
mean,arecon'trained tolieona'mooIhtw-o-diITlCn,iooal n1anifold. Thisproba-

Exercises 599
bilisticfoundation alsomakesitverystraightforward todefinegeneralizations of
GTM(Bishopetal.,1998a)suchasaBayesian treatment, dealingwithmissingval-
Section6.4 ues,aprincipled extension todiscretevariables, theuseofGaussian processes to
definethemanifold, orahierarchical GTMmodel(TinoandNabney,2002).
Becausethemanifold inGTMisdefinedasacontinuous surface,notjustatthe
prototype vectorsasintheSOM,itispossibletocompute themagnification factors
corresponding tothelocalexpansions andcompressions ofthemanifold neededto
fitthedataset(Bishopetal.,1997b)aswellasthedirectional curvatures ofthe
manifold (Tinoetal.,2001).Thesecanbevisualized alongwiththeprojected data
andprovideadditional insightintothemodel.
Exercises
12.1(**)lIB Inthisexercise, weuseproofbyinduction toshowthatthelinear
projection ontoanM-dimensional subspace thatmaximizes thevarianceofthepro­
jecteddataisdefinedbytheMeigenvectors ofthedatacovariance matrixS,given
by(12.3),corresponding totheMlargesteigenvalues. InSection12.1,thisresult

Exercises
wasprovenforthecaseofM=1.Nowsupposetheresultholdsforsomegeneral
valueofMandshowthatitconsequently holdsfordimensionality M+1.Todo
this,firstsetthederivative ofthevariance oftheprojected datawithrespecttoa
vector UM+1definingthenewdirection indataspaceequaltozero.Thisshould
bedonesubjecttotheconstraints thatUM+lbeorthogonal totheexistingvectors
U1,"" UM,andalsothat itbenormalized tounitlength.UseLagrange multipli-
Appendix E erstoenforcetheseconstraints. Thenmakeuseoftheorthonormality properties of

thevectors U1,"" UMtoshowthatthenewvector UM+1isaneigenvector ofS.
Finally,showthatthevariance ismaximized iftheeigenvector ischosentobethe
onecorresponding toeigenvector AM+1wheretheeigenvalues havebeenorderedin
decreasing value.
12.2(**)Showthattheminimum valueofthePCAdistortion measureJgivenby
(12.15)withrespecttotheUi,subjecttotheorthonormality constraints (12.7),is
obtained whentheUiareeigenvectors ofthedatacovariance matrixS.Todothis,
introduce amatrixHofLagrange multipliers, oneforeachconstraint, sothatthe

modified distortion measure, inmatrixnotationreads
]=Tr{UTSU}+Tr{H(I-UTU)} (12.93)
whereUisam~trixofdimensio~ Dx(D-M)whosecolumns aregi:::..enb~Ui.
NowminimizeJwithrespecttoUandshowthatthes~utionsatisfiesSU=UH.
Clearly,onepossiblesolutionisthatthecolumns ofUareeigenvectors ofS,in
whichcaseHisadiagonal matrixcontaining thecorresponding eigenvalues. To
obtainthegeneralsolution, showthatHcanbeassumed tobeasymmetr~ ma~ix,
andbyusingitseigenvect£r expansion showthatthegeneralsolutiontoSU=~UH
givesthesamevalueforJasthespecificsolutioninwhichthecolumns ofUare

600 12.CONTINUOUS LATENT VARIABLES
theeigenvectors ofS.Because thesesolutions areallequivalent, itisconvenient to
choosetheeigenvector solution.
12.3(*)Verifythattheeigenvectors definedby(12.30)arenormalized tounitlength,
assuming thattheeigenvectors Vihaveunitlength.
12.4(*)Imm Suppose wereplacethezero-mean, unit-covariance latentspacedistri­
bution(12.31)intheprobabilistic PCAmodelbyageneralGaussian distribution of
theformN(zlm, ~).Byredefining theparameters ofthemodel,showthatthisleads
toanidentical modelforthemarginal distribution p(x)overtheobserved variables
foranyvalidchoiceofmand~.
12.5(**)LetxbeaD-dimensional randomvariablehavingaGaussian distribution
givenbyN(xIJL, ~),andconsider theM-dimensional randomvariablegivenby
y=Ax+bwhereAisanMxDmatrix.ShowthatyalsohasaGaussian
distribution, andfindexpressions foritsmeanandcovariance. Discusstheformof
thisGaussian distribution forM<D,forM=D,andforM>D.
12.6(*)Imm Drawadirectedprobabilistic graphfortheprobabilistic PCAmodel

described inSection12.2inwhichthecomponents oftheobserved variablexare
shownexplicitly asseparatenodes.Henceverifythattheprobabilistic PCAmodel
hasthesameindependence structure asthenaiveBayesmodeldiscussed inSec­
tion8.2.2.
12.7(**)Bymakinguseoftheresults(2.270)and(2.271)forthemeanandcovariance
ofageneraldistribution, derivetheresult(12.35)forthemarginal distribution p(x)
intheprobabilistic PCAmodel.
12.8(**)ImmBymakinguseoftheresult(2.116),showthattheposterior distribution

p(zlx)fortheprobabilistic PCAmodelisgivenby(12.42).
12.9(*)Verifythatmaximizing theloglikelihood (12.43)fortheprobabilistic PCA
modelwithrespecttotheparameter JLgivestheresult JLML=xwherexisthe
meanofthedatavectors.
12.10(**)Byevaluating thesecondderivatives oftheloglikelihood function(12.43)for
theprobabilistic PCAmodelwithrespecttotheparameter JL,showthatthestationary
pointJLML=xrepresents theuniquemaximum.
12.11 (**)Imm Showthatinthelimit(Y2-.0,theposterior meanfortheprobabilistic

PCAmodelbecomes anorthogonal projection ontotheprincipal subspace, asin
conventional PCA.
12.12(**)For(Y2>0showthattheposterior meanintheprobabilistic PCAmodelis
shiftedtowardstheoriginrelativetotheorthogonal projection.
12.13(**)Showthattheoptimalreconstruction ofadatapointunderprobabilistic PCA,
according totheleastsquaresprojection costofconventional PCA,isgivenby
(12.94)

Exercises 601
12.14(*)Thenumberofindependent parameters inthecovariance matrixfortheproba­
bilisticPCAmodelwithanM-dimensional latentspaceandaD-dimensional data
spaceisgivenby(12.51). VerifythatinthecaseofM=D-1,thenumberof
independent parameters isthesameasinageneralcovariance Gaussian, whereasfor
M=°itisthesameasforaGaussian withanisotropic covariance.
12.15(**)IIiI!I DerivetheM-stepequations (12.56)and(12.57)fortheprobabilistic
PCAmodelbymaximization oftheexpected complete-data loglikelihood function
givenby(12.53).
12.16(***)InFigure12.11,weshowedanapplication ofprobabilistic PCAtoadataset
inwhichsomeofthedatavalues were missingatrandom. DerivetheEMalgorithm
formaximizing thelikelihood functionfortheprobabilistic PCAmodelinthissitu­
ation.Notethatthe{zn},aswellasthemissingdatavaluesthatarecomponents of
thevectors{xn},arenowlatentvariables. Showthatinthespecialcaseinwhichall
ofthedatavaluesareobserved, thisreducestotheEMalgorithm forprobabilistic
PCAderivedinSection12.2.2.

12.17(**)IIiI!I LetWbeaDxMmatrixwhosecolumns definealinearsubspace
ofdimensionality Membedded withinadataspaceofdimensionality D,andletJ1
beaD-dimensional vector.Givenadataset{xn}wheren=1,...,N,wecan
approximate thedatapointsusingalinearmapping fromasetofM-dimensional
vectors{zn},sothatXnisapproximated byWZn+J1.Theassociated sum-of­
squaresreconstruction costisgivenby
N
J=LIlxn-J1-Wzn112.
n=l(12.95)
Firstshowthatminimizing JwithrespecttoJ1leadstoananalogous expression with

N
XnandZnreplacedbyzero-mean variables Xn-xandZn-Z,respectively, wherex
andZdenotesamplemeans.Thenshowthatminimizing JwithrespecttoZn,where
Wiskeptfixed,givesrisetothePCAEstep(12.58),andthatminimizing Jwith
respecttoW,where{zn}iskeptfixed,givesrisetothePCAMstep(12.59).
12.18(*)Deriveanexpression forthenumberofindependent parameters inthefactor
analysismodeldescribed inSection12.2.4.
12.19(**)IIiI!I Showthatthefactoranalysismodeldescribed inSection12.2.4is
invariant underrotations ofthelatentspacecoordinates.

12.20(**)Byconsidering secondderivatives, showthattheonlystationary pointof
theloglikelihood functionforthefactoranalysismodeldiscussed inSection12.2.4
withrespecttotheparameter J1isgivenbythesamplemeandefinedby(12.1).
Furthermore, showthatthisstationary pointisamaximum.
12.21(**)Derivetheformulae (12.66)and(12.67)fortheEstepoftheEMalgorithm
forfactoranalysis. NotethatfromtheresultofExercise 12.20,theparameter J1can
bereplacedbythesamplemeanx.

602 12.CONTINUOUS LATENT VARIABLES
12.22(**)Writedownanexpression fortheexpected complete-data loglikelihood func­
tionforthefactoranalysismodel,andhencederivethecorresponding Mstepequa­
tions(12.69)and(12.70).
12.23(*)III!I Drawadirectedprobabilistic graphical modelrepresenting adiscrete
mixtureofprobabilistic PCAmodelsinwhicheachPCAmodelhasitsownvalues
ofW,JL,and0-2•Nowdrawamodified graphinwhichtheseparameter valuesare
sharedbetweenthecomponents ofthemixture.
12.24(***)WesawinSection2.3.7thatStudent's t-distribution canbeviewedasan
infinitemixtureofGaussians inwhichwemarginalize withrespecttoacontinu­
ouslatentvariable. Byexploiting thisrepresentation, formulate anEMalgorithm
formaximizing theloglikelihood functionforamultivariate Student's t-distribution
givenanobserved setofdatapoints,andderivetheformsoftheEandMstepequa­
tions.
12.25(**)III!IConsider alinear-Gaussian latent-variable modelhavingalatentspace
distribution p(z)=N(xIO,I)andaconditional distribution fortheobserved vari­

tions.
ablep(xlz)=N(xlWz+IL,<p)where<Pisanarbitrary symmetric, positive­
definitenoisecovariance matrix.Nowsupposethatwemakeanonsingular linear
transformation ofthedatavariables x---tAx,whereAisaDxDmatrix.If
JLML'WMLand<PMLrepresent themaximum likelihood solutioncorresponding to
theoriginaluntransformed data,showthatAJLML'AWML,andA<PMLA Twillrep­
resentthecorresponding maximum likelihood solutionforthetransformed dataset.
Finally,showthattheformofthemodelispreserved intwocases:(i)Aisadiagonal
matrixand<Pisadiagonal matrix.Thiscorresponds tothecaseoffactoranalysis.

Thetransformed <Premainsdiagonal, andhencefactoranalysisiscovariant under
component-wise re-scaling ofthedatavariables; (ii)Aisorthogonal and<Pispro­
portional totheunitmatrixsothat<P=0-21.Thiscorresponds toprobabilistic PCA.
Thetransformed <Pmatrixremainsproportional totheunitmatrix,andhenceproba­
bilisticPCAiscovariant underarotationoftheaxesofdataspace,asisthecasefor
\12.26(**)Showthatanyvectoraithatsatisfies(12.80)willalsosatisfy(12.79).Also,

showthatforanysolutionof(12.80)havingeigenvalue A,wecanaddanymultiple
ofaneigenvector ofKhavingzeroeigenvalue, andobtainasolutionto(12.79)
thatalsohaseigenvalue A.Finally,showthatsuchmodifications donotaffectthe
principal-component projection givenby(12.82).
12.27(**)Showthattheconventional linearPCAalgorithm isrecovered asaspecialcase
ofkernelPCAifwechoosethelinearkernelfunctiongivenbyk(x,x')=xTx'.
12.28(**)III!I Usethetransformation property (1.27)ofaprobability densityunder
achangeofvariabletoshowthatanydensityp(y)canbeobtained fromafixed

densityq(x)thatiseverywhere nonzerobymakinganonlinear changeofvariable
y=f(x)inwhichf(x)isamonotonic function sothat0:::;j'(x)<00.Write
downthedifferential equation satisfiedbyf(x)anddrawadiagramillustrating the
transformation ofthedensity.

Exercises 603
12.29(**)EmSupposethattwovariables ZlandZ2areindependent sothatp(zl' Z2)=
P(Zl)P(Z2)' Showthatthecovariance matrixbetween thesevariables isdiagonal.
Thisshowsthatindependence isasufficient condition fortwovariables tobeun­
correlated. Nowconsider twovariables YlandY2inwhich-1:0;;Yl:0;;1and
Y2=yg.Writedowntheconditional distribution p(Y2IYl) andobservethatthisis
dependent onYbshowing thatthetwovariables arenotindependent. Nowshow
thatthecovariance matrixbetweenthesetwovariables isagaindiagonal. Todothis,
usetherelationP(Yl,Y2)=P(YI)p(Y2IYl) toshowthattheoff-diagonal termsare
zero.Thiscounter-example showsthatzerocorrelation isnotasufficient condition
forindependence.

13
Sequential
Data
So far in this book, we have focussed primarily on sets of data points that were as-
sumed to be independent and identically distributed (i.i.d.). This assumption allowedus to express the likelihood function as the product over all data points of the prob-
ability distribution evaluated at each data point. For many applications, however,
the i.i.d. assumption will be a poor one. Here we consider a particularly importantclass of such data sets, namely those that describe sequential data. These often arise
through measurement of time series, for example the rainfall measurements on suc-
cessive days at a particular location, or the daily values of a currency exchange rate,or the acoustic features at successive time frames used for speech recognition. An
example involving speech data is shown in Figure 13.1. Sequential data can also

arise in contexts other than time series, for example the sequence of nucleotide basepairs along a strand of DNA or the sequence of characters in an English sentence.
For convenience, we shall sometimes refer to ‘past’ and ‘future’ observations in a
sequence. However, the models explored in this chapter are equally applicable to all
605

606 13. SEQUENTIAL DATA
Figure 13.1 Example of a spectro-
gram of the spoken words “Bayes’ theo-
rem” showing a plot of the intensity of the
spectral coefﬁcients versus time index.
forms of sequential data, not just temporal sequences.
It is useful to distinguish between stationary and nonstationary sequential dis-
tributions. In the stationary case, the data evolves in time, but the distribution from
which it is generated remains the same. For the more complex nonstationary situa-
tion, the generative distribution itself is evolving with time. Here we shall focus onthe stationary case.
For many applications, such as ﬁnancial forecasting, we wish to be able to pre-
dict the next value in a time series given observations of the previous values. In-tuitively, we expect that recent observations are likely to be more informative than
more historical observations in predicting future values. The example in Figure 13.1

shows that successive observations of the speech spectrum are indeed highly cor-related. Furthermore, it would be impractical to consider a general dependence of
future observations on all previous observations because the complexity of such a
model would grow without limit as the number of observations increases. This leads
us to consider Markov models in which we assume that future predictions are inde-

13.1. Markov Models 607
Figure 13.2 The simplest approach to
modelling a sequence of ob-
servations is to treat them
as independent, correspond-
ing to a graph without links.x1 x2 x3 x4
pendent of all but the most recent observations.
Although such models are tractable, they are also severely limited. We can ob-
tain a more general framework, while still retaining tractability, by the introduction
of latent variables, leading to state space models . As in Chapters 9 and 12, we shall
see that complex models can thereby be constructed from simpler components (in
particular, from distributions belonging to the exponential family) and can be read-
ily characterized using the framework of probabilistic graphical models. Here we
focus on the two most important examples of state space models, namely the hid-
den Markov model , in which the latent variables are discrete, and linear dynamical
systems , in which the latent variables are Gaussian. Both models are described by di-

rected graphs having a tree structure (no loops) for which inference can be performed
efﬁciently using the sum-product algorithm.
13.1. Markov Models
The easiest way to treat sequential data would be simply to ignore the sequential
aspects and treat the observations as i.i.d., corresponding to the graph in Figure 13.2.
Such an approach, however, would fail to exploit the sequential patterns in the data,
such as correlations between observations that are close in the sequence. Suppose,

for instance, that we observe a binary variable denoting whether on a particular day
it rained or not. Given a time series of recent observations of this variable, we wish
to predict whether it will rain on the next day. If we treat the data as i.i.d., then the
only information we can glean from the data is the relative frequency of rainy days.
However, we know in practice that the weather often exhibits trends that may last for
several days. Observing whether or not it rains today is therefore of signiﬁcant help

in predicting if it will rain tomorrow.
To express such effects in a probabilistic model, we need to relax the i.i.d. as-
sumption, and one of the simplest ways to do this is to consider a Markov model .
First of all we note that, without loss of generality, we can use the product rule to
express the joint distribution for a sequence of observations in the form
p(x1,...,xN)=N∏
n=1p(xn|x1,...,xn−1). (13.1)
If we now assume that each of the conditional distributions on the right-hand side
is independent of all previous observations except the most recent, we obtain the

ﬁrst-order Markov chain , which is depicted as a graphical model in Figure 13.3. The

608 13. SEQUENTIAL DATA
Figure 13.3 A ﬁrst-order Markov chain of ob-
servations {xn}in which the dis-
tribution p(xn|xn−1)of a particu-
lar observation xnis conditioned
on the value of the previous ob-
servation xn−1.x1 x2 x3 x4
joint distribution for a sequence of Nobservations under this model is given by
p(x1,...,xN)=p(x1)N∏
n=2p(xn|xn−1). (13.2)
From the d-separation property, we see that the conditional distribution for observa- Section 8.2
tionxn, given all of the observations up to time n,i sg i v e nb y
p(xn|x1,...,xn−1)=p(xn|xn−1) (13.3)
which is easily veriﬁed by direct evaluation starting from (13.2) and using the prod-
uct rule of probability. Thus if we use such a model to predict the next observation Exercise 13.1
in a sequence, the distribution of predictions will depend only on the value of the im-
mediately preceding observation and will be independent of all earlier observations.
In most applications of such models, the conditional distributions p(xn|xn−1)

that deﬁne the model will be constrained to be equal, corresponding to the assump-
tion of a stationary time series. The model is then known as a homogeneous Markov
chain. For instance, if the conditional distributions depend on adjustable parameters
(whose values might be inferred from a set of training data), then all of the condi-
tional distributions in the chain will share the same values of those parameters.
Although this is more general than the independence model, it is still very re-

strictive. For many sequential observations, we anticipate that the trends in the data
over several successive observations will provide important information in predict-
ing the next value. One way to allow earlier observations to have an inﬂuence is to
move to higher-order Markov chains. If we allow the predictions to depend also on
the previous-but-one value, we obtain a second-order Markov chain, represented by
the graph in Figure 13.4. The joint distribution is now given by
p(x1,...,xN)=p(x1)p(x2|x1)N∏

n=3p(xn|xn−1,xn−2). (13.4)
Again, using d-separation or by direct evaluation, we see that the conditional distri-
bution of xngivenxn−1andxn−2is independent of all observations x1,...xn−3.
Figure 13.4 A second-order Markov chain, in
which the conditional distribution
of a particular observation xn
depends on the values of the two
previous observations xn−1and
xn−2.x1 x2 x3 x4

13.1. Markov Models 609
Figure 13.5 We can represent sequen-
tial data using a Markov chain of latent
variables, with each observation condi-
tioned on the state of the corresponding
latent variable. This important graphical
structure forms the foundation both for the
hidden Markov model and for linear dy-
namical systems.zn−1 zn zn+1
xn−1 xn xn+1z1 z2
x1 x2
Each observation is now inﬂuenced by two previous observations. We can similarly
consider extensions to an Mthorder Markov chain in which the conditional distri-
bution for a particular variable depends on the previous Mvariables. However, we
have paid a price for this increased ﬂexibility because the number of parameters in
the model is now much larger. Suppose the observations are discrete variables hav-
ingKstates. Then the conditional distribution p(xn|xn−1)in a ﬁrst-order Markov
chain will be speciﬁed by a set of K−1parameters for each of the Kstates of xn−1
giving a total of K(K−1)parameters. Now suppose we extend the model to an

Mthorder Markov chain, so that the joint distribution is built up from conditionals
p(xn|xn−M,...,xn−1). If the variables are discrete, and if the conditional distri-
butions are represented by general conditional probability tables, then the number
of parameters in such a model will have KM−1(K−1)parameters. Because this
grows exponentially with M, it will often render this approach impractical for larger
values of M.
For continuous variables, we can use linear-Gaussian conditional distributions

in which each node has a Gaussian distribution whose mean is a linear function
of its parents. This is known as an autoregressive orARmodel (Box et al. , 1994;
Thiesson et al. , 2004). An alternative approach is to use a parametric model for
p(xn|xn−M,...,xn−1)such as a neural network. This technique is sometimes
called a tapped delay line because it corresponds to storing (delaying) the previous
Mvalues of the observed variable in order to predict the next value. The number

of parameters can then be much smaller than in a completely general model (for ex-
ample it may grow linearly with M), although this is achieved at the expense of a
restricted family of conditional distributions.
Suppose we wish to build a model for sequences that is not limited by the
Markov assumption to any order and yet that can be speciﬁed using a limited number
of free parameters. We can achieve this by introducing additional latent variables to
permit a rich class of models to be constructed out of simple components, as we did

with mixture distributions in Chapter 9 and with continuous latent variable models in
Chapter 12. For each observation xn, we introduce a corresponding latent variable
zn(which may be of different type or dimensionality to the observed variable). We
now assume that it is the latent variables that form a Markov chain, giving rise to the
graphical structure known as a state space model , which is shown in Figure 13.5. It
satisﬁes the key conditional independence property that zn−1andzn+1are indepen-
dent given zn, so that
zn+1⊥⊥zn−1|zn. (13.5)

610 13. SEQUENTIAL DATA
The joint distribution for this model is given by
p(x1,...,xN,z1,...,zN)=p(z1)[N∏
n=2p(zn|zn−1)]N∏
n=1p(xn|zn). (13.6)
Using the d-separation criterion, we see that there is always a path connecting any
two observed variables xnandxmvia the latent variables, and that this path is never
blocked. Thus the predictive distribution p(xn+1|x1,...,xn)for observation xn+1
given all previous observations does not exhibit any conditional independence prop-
erties, and so our predictions for xn+1depends on all previous observations. The
observed variables, however, do not satisfy the Markov property at any order. We
shall discuss how to evaluate the predictive distribution in later sections of this chap-ter.
There are two important models for sequential data that are described by this
graph. If the latent variables are discrete, then we obtain the hidden Markov model ,
orHMM (Elliott et al. , 1995). Note that the observed variables in an HMM may Section 13.2

be discrete or continuous, and a variety of different conditional distributions can beused to model them. If both the latent and the observed variables are Gaussian (with
a linear-Gaussian dependence of the conditional distributions on their parents), then
we obtain the linear dynamical system . Section 13.3
13.2. Hidden Markov Models
The hidden Markov model can be viewed as a speciﬁc instance of the state space

model of Figure 13.5 in which the latent variables are discrete. However, if weexamine a single time slice of the model, we see that it corresponds to a mixture
distribution, with component densities given by p(x|z). It can therefore also be
interpreted as an extension of a mixture model in which the choice of mixture com-ponent for each observation is not selected independently but depends on the choice
of component for the previous observation. The HMM is widely used in speech

recognition (Jelinek, 1997; Rabiner and Juang, 1993), natural language modelling(Manning and Sch ¨utze, 1999), on-line handwriting recognition (Nag et al. , 1986),
and for the analysis of biological sequences such as proteins and DNA (Krogh et al. ,
1994; Durbin et al. , 1998; Baldi and Brunak, 2001).
As in the case of a standard mixture model, the latent variables are the discrete
multinomial variables z
ndescribing which component of the mixture is responsible

for generating the corresponding observation xn. Again, it is convenient to use a
1-of-Kcoding scheme, as used for mixture models in Chapter 9. We now allow the
probability distribution of znto depend on the state of the previous latent variable
zn−1through a conditional distribution p(zn|zn−1). Because the latent variables are
K-dimensional binary variables, this conditional distribution corresponds to a table
of numbers that we denote by A, the elements of which are known as transition

probabilities . They are given by Ajk≡p(znk=1|zn−1,j=1 ) , and because they
are probabilities, they satisfy 0⩽Ajk⩽1with∑
kAjk=1, so that the matrix A

13.2. Hidden Markov Models 611
Figure 13.6 Transition diagram showing a model whose la-
tent variables have three possible states corre-
sponding to the three boxes. The black lines
denote the elements of the transition matrix
Ajk.A12
A23
A31A21
A32
A13A11A22
A33k=1k=2
k=3
hasK(K−1)independent parameters. We can then write the conditional distribution
explicitly in the form
p(zn|zn−1,A)=K∏
k=1K∏
j=1Azn−1,jznk
jk. (13.7)
The initial latent node z1is special in that it does not have a parent node, and so
it has a marginal distribution p(z1)represented by a vector of probabilities πwith
elements πk≡p(z1k=1 ) , so that
p(z1|π)=K∏
k=1πz1k
k(13.8)
where∑
kπk=1.
The transition matrix is sometimes illustrated diagrammatically by drawing the
states as nodes in a state transition diagram as shown in Figure 13.6 for the case of
K=3. Note that this does not represent a probabilistic graphical model, because
the nodes are not separate variables but rather states of a single variable, and so we

k=1πz1k
k(13.8)
where∑
kπk=1.
have shown the states as boxes rather than circles.
It is sometimes useful to take a state transition diagram, of the kind shown in
Figure 13.6, and unfold it over time. This gives an alternative representation of the
transitions between latent states, known as a lattice ortrellis diagram, and which is Section 8.4.5
shown for the case of the hidden Markov model in Figure 13.7.
The speciﬁcation of the probabilistic model is completed by deﬁning the con-

ditional distributions of the observed variables p(xn|zn,φ), where φis a set of pa-
rameters governing the distribution. These are known as emission probabilities , and
might for example be given by Gaussians of the form (9.11) if the elements of xare
continuous variables, or by conditional probability tables if xis discrete. Because
xnis observed, the distribution p(xn|zn,φ)consists, for a given value of φ,o fa
vector of Knumbers corresponding to the Kpossible states of the binary vector zn.

612 13. SEQUENTIAL DATA
Figure 13.7 If we unfold the state transition dia-
gram of Figure 13.6 over time, we obtain a lattice,
or trellis, representation of the latent states. Each
column of this diagram corresponds to one of the
latent variables zn.k=1
k=2
k=3
n−2 n−1 nn +1A11 A11 A11
A33 A33 A33
We can represent the emission probabilities in the form
p(xn|zn,φ)=K∏
k=1p(xn|φk)znk. (13.9)
We shall focuss attention on homogeneous models for which all of the condi-
tional distributions governing the latent variables share the same parameters A, and
similarly all of the emission distributions share the same parameters φ(the extension
to more general cases is straightforward). Note that a mixture model for an i.i.d. data
set corresponds to the special case in which the parameters Ajkare the same for all
values of j, so that the conditional distribution p(zn|zn−1)is independent of zn−1.
This corresponds to deleting the horizontal links in the graphical model shown in
Figure 13.5.

The joint probability distribution over both latent and observed variables is then
given by
p(X,Z|θ)=p(z1|π)[N∏
n=2p(zn|zn−1,A)]N∏
m=1p(xm|zm,φ) (13.10)
whereX={x1,...,xN},Z={z1,...,zN}, andθ={π,A,φ}denotes the set
of parameters governing the model. Most of our discussion of the hidden Markov
model will be independent of the particular choice of the emission probabilities.
Indeed, the model is tractable for a wide range of emission distributions including
discrete tables, Gaussians, and mixtures of Gaussians. It is also possible to exploit

given by
discriminative models such as neural networks. These can be used to model the Exercise 13.4
emission density p(x|z)directly, or to provide a representation for p(z|x)that can
be converted into the required emission density p(x|z)using Bayes’ theorem (Bishop
et al. , 2004).
We can gain a better understanding of the hidden Markov model by considering
it from a generative point of view. Recall that to generate samples from a mixture of

13.2. Hidden Markov Models 613
k=1
k=2k=3
Figure 13.8 Illustration of sampling from a hidden Markov model having a 3-state latent variable zand a
Gaussian emission model p(x|z)where xis 2-dimensional. (a) Contours of constant probability density for the
emission distributions corresponding to each of the three states of the latent variable. (b) A sample of 50 points
drawn from the hidden Markov model, colour coded according to the component that generated them and with
lines connecting the successive observations. Here the transition matrix was ﬁxed so that in any state there is a
5% probability of making a transition to each of the other states, and consequently a 90% probability of remaining
in the same state.
Gaussians, we ﬁrst chose one of the components at random with probability given by
the mixing coefﬁcients πkand then generate a sample vector xfrom the correspond-
ing Gaussian component. This process is repeated Ntimes to generate a data set of

Nindependent samples. In the case of the hidden Markov model, this procedure is
modiﬁed as follows. We ﬁrst choose the initial latent variable z1with probabilities
governed by the parameters πkand then sample the corresponding observation x1.
Now we choose the state of the variable z2according to the transition probabilities
p(z2|z1)using the already instantiated value of z1. Thus suppose that the sample for
z1corresponds to state j. Then we choose the state kofz2with probabilities Ajk

fork=1,...,K . Once we know z2we can draw a sample for x2and also sample
the next latent variable z3and so on. This is an example of ancestral sampling for
a directed graphical model. If, for instance, we have a model in which the diago- Section 8.1.2
nal transition elements Akkare much larger than the off-diagonal elements, then a
typical data sequence will have long runs of points generated from a single compo-
nent, with infrequent transitions from one component to another. The generation of

samples from a hidden Markov model is illustrated in Figure 13.8.
There are many variants of the standard HMM model, obtained for instance by
imposing constraints on the form of the transition matrix A(Rabiner, 1989). Here we
mention one of particular practical importance called the left-to-right HMM, which
is obtained by setting the elements AjkofAto zero if k<j , as illustrated in the

614 13. SEQUENTIAL DATA
Figure 13.9 Example of the state transition diagram for a 3-state
left-to-right hidden Markov model. Note that once a
state has been vacated, it cannot later be re-entered.
k=1 k=2 k=3A11 A22 A33
A12 A23
A13
state transition diagram for a 3-state HMM in Figure 13.9. Typically for such models
the initial state probabilities for p(z1)are modiﬁed so that p(z11)=1 andp(z1j)=0
forj̸=1, in other words every sequence is constrained to start in state j=1. The
transition matrix may be further constrained to ensure that large changes in the state
index do not occur, so that Ajk=0ifk>j +∆. This type of model is illustrated
using a lattice diagram in Figure 13.10.
Many applications of hidden Markov models, for example speech recognition,
or on-line character recognition, make use of left-to-right architectures. As an illus-
tration of the left-to-right hidden Markov model, we consider an example involving
handwritten digits. This uses on-line data, meaning that each digit is represented

by the trajectory of the pen as a function of time in the form of a sequence of pen
coordinates, in contrast to the off-line digits data, discussed in Appendix A, which
comprises static two-dimensional pixellated images of the ink. Examples of the on-
line digits are shown in Figure 13.11. Here we train a hidden Markov model on a
subset of data comprising 45 examples of the digit ‘2’. There are K=1 6 states,
each of which can generate a line segment of ﬁxed length having one of 16 possible
angles, and so the emission distribution is simply a 16×16table of probabilities

associated with the allowed angle values for each state index value. Transition prob-
abilities are all set to zero except for those that keep the state index kthe same or
that increment it by 1, and the model parameters are optimized using 25 iterations of
EM. We can gain some insight into the resulting model by running it generatively, as
shown in Figure 13.11.
Figure 13.10 Lattice diagram for a 3-state left-
to-right HMM in which the state index kis allowed
to increase by at most 1 at each transition. k=1
k=2
k=3

k=2
k=3
A33 A33 A33

13.2. Hidden Markov Models 615
Figure 13.11 Top row: examples of on-line handwritten
digits. Bottom row: synthetic digits sam-
pled generatively from a left-to-right hid-
den Markov model that has been trainedon a data set of 45 handwritten digits.
One of the most powerful properties of hidden Markov models is their ability to
exhibit some degree of invariance to local warping (compression and stretching) of
the time axis. To understand this, consider the way in which the digit ‘2’ is written
in the on-line handwritten digits example. A typical digit comprises two distinct
sections joined at a cusp. The ﬁrst part of the digit, which starts at the top left, has asweeping arc down to the cusp or loop at the bottom left, followed by a second more-
or-less straight sweep ending at the bottom right. Natural variations in writing style
will cause the relative sizes of the two sections to vary, and hence the location of thecusp or loop within the temporal sequence will vary. From a generative perspective

such variations can be accommodated by the hidden Markov model through changes
in the number of transitions to the same state versus the number of transitions to thesuccessive state. Note, however, that if a digit ‘2’ is written in the reverse order, that
is, starting at the bottom right and ending at the top left, then even though the pen tip
coordinates may be identical to an example from the training set, the probability ofthe observations under the model will be extremely small. In the speech recognition

context, warping of the time axis is associated with natural variations in the speed of
speech, and again the hidden Markov model can accommodate such a distortion andnot penalize it too heavily.
13.2.1 Maximum likelihood for the HMM
If we have observed a data set X={x1,...,xN}, we can determine the param-
eters of an HMM using maximum likelihood. The likelihood function is obtained
from the joint distribution (13.10) by marginalizing over the latent variables
p(X|θ)=∑
Zp(X,Z|θ). (13.11)
Because the joint distribution p(X,Z|θ)does not factorize over n(in contrast to the

p(X|θ)=∑
mixture distribution considered in Chapter 9), we cannot simply treat each of the
summations over znindependently. Nor can we perform the summations explicitly
because there are Nvariables to be summed over, each of which has Kstates, re-
sulting in a total of KNterms. Thus the number of terms in the summation grows

616 13. SEQUENTIAL DATA
exponentially with the length of the chain. In fact, the summation in (13.11) cor-
responds to summing over exponentially many paths through the lattice diagram inFigure 13.7.
We have already encountered a similar difﬁculty when we considered the infer-
ence problem for the simple chain of variables in Figure 8.32. There we were ableto make use of the conditional independence properties of the graph to re-order the
summations in order to obtain an algorithm whose cost scales linearly, instead of
exponentially, with the length of the chain. We shall apply a similar technique to thehidden Markov model.
A further difﬁculty with the expression (13.11) for the likelihood function is that,
because it corresponds to a generalization of a mixture distribution, it represents a
summation over the emission models for different settings of the latent variables.

Direct maximization of the likelihood function will therefore lead to complex ex-pressions with no closed-form solutions, as was the case for simple mixture models Section 9.2
(recall that a mixture model for i.i.d. data is a special case of the HMM).
We therefore turn to the expectation maximization algorithm to ﬁnd an efﬁcient
framework for maximizing the likelihood function in hidden Markov models. The
EM algorithm starts with some initial selection for the model parameters, which we
denote by θ
old. In the E step, we take these parameter values and ﬁnd the posterior

denote by θ
distribution of the latent variables p(Z|X,θold). We then use this posterior distri-
bution to evaluate the expectation of the logarithm of the complete-data likelihood
function, as a function of the parameters θ, to give the function Q(θ,θold)deﬁned
by
Zp(Z|X,θold)l np(X,Z|θ). (13.12)
At this point, it is convenient to introduce some notation. We shall use γ(zn)to
denote the marginal posterior distribution of a latent variable zn, andξ(zn−1,zn)to
denote the joint posterior distribution of two successive latent variables, so that
γ(zn)= p(zn|X,θold) (13.13)
ξ(zn−1,zn)= p(zn−1,zn|X,θold). (13.14)

by
For each value of n, we can store γ(zn)using a set of Knonnegative numbers
that sum to unity, and similarly we can store ξ(zn−1,zn)using a K×Kmatrix of
nonnegative numbers that again sum to unity. We shall also use γ(znk)to denote the
conditional probability of znk=1, with a similar use of notation for ξ(zn−1,j,znk)
and for other probabilistic variables introduced later. Because the expectation of abinary random variable is just the probability that it takes the value 1,w eh a v e
γ(z
nk)= E[znk]=∑
zγ(z)znk (13.15)
ξ(zn−1,j,znk)= E[zn−1,jznk]=∑
zγ(z)zn−1,jznk. (13.16)

γ(z
If we substitute the joint distribution p(X,Z|θ)given by (13.10) into (13.12),

13.2. Hidden Markov Models 617
and make use of the deﬁnitions of γandξ, we obtain
Q(θ,θold)=K∑
k=1γ(z1k)l nπk+N∑
n=2K∑
j=1K∑
k=1ξ(zn−1,j,znk)l nAjk
+N∑
n=1K∑
k=1γ(znk)l np(xn|φk). (13.17)
The goal of the E step will be to evaluate the quantities γ(zn)andξ(zn−1,zn)efﬁ-
ciently, and we shall discuss this in detail shortly.
In the M step, we maximize Q(θ,θold)with respect to the parameters θ=
{π,A,φ}in which we treat γ(zn)andξ(zn−1,zn)as constant. Maximization with
respect to πandAis easily achieved using appropriate Lagrange multipliers with
the results Exercise 13.5
πk=γ(z1k)
K∑
j=1γ(z1j)(13.18)
Ajk=N∑
n=2ξ(zn−1,j,znk)
K∑
l=1N∑
n=2ξ(zn−1,j,znl). (13.19)
The EM algorithm must be initialized by choosing starting values for πandA, which
should of course respect the summation constraints associated with their probabilis-
tic interpretation. Note that any elements of πorAthat are set to zero initially will
remain zero in subsequent EM updates. A typical initialization procedure would Exercise 13.6

πk=γ(z1k)
K∑
Ajk=N∑
K∑
l=1N∑
involve selecting random starting values for these parameters subject to the summa-
tion and non-negativity constraints. Note that no particular modiﬁcation to the EMresults are required for the case of left-to-right models beyond choosing initial values
for the elements A
jkin which the appropriate elements are set to zero, because these
will remain zero throughout.
To maximize Q(θ,θold)with respect to φk, we notice that only the ﬁnal term
in (13.17) depends on φk, and furthermore this term has exactly the same form as

the data-dependent term in the corresponding function for a standard mixture dis-
tribution for i.i.d. data, as can be seen by comparison with (9.40) for the case of a
Gaussian mixture. Here the quantities γ(znk)are playing the role of the responsibil-
ities. If the parameters φkare independent for the different components, then this
term decouples into a sum of terms one for each value of k, each of which can be
maximized independently. We are then simply maximizing the weighted log likeli-hood function for the emission density p(x|φ
k)with weights γ(znk). Here we shall

suppose that this maximization can be done efﬁciently. For instance, in the case of

618 13. SEQUENTIAL DATA
Gaussian emission densities we have p(x|φk)=N(x|µk,Σk), and maximization
of the function Q(θ,θold)then gives
µk=N∑
n=1γ(znk)xn
N∑
n=1γ(znk)(13.20)
Σk=N∑
n=1γ(znk)(xn−µk)(xn−µk)T
N∑
n=1γ(znk). (13.21)
For the case of discrete multinomial observed variables, the conditional distribution
of the observations takes the form
p(x|z)=D∏
i=1K∏
k=1µxizk
ik(13.22)
and the corresponding M-step equations are given by Exercise 13.8
µik=N∑
n=1γ(znk)xni
N∑
n=1γ(znk). (13.23)
An analogous result holds for Bernoulli observed variables.
The EM algorithm requires initial values for the parameters of the emission dis-
tribution. One way to set these is ﬁrst to treat the data initially as i.i.d. and ﬁt the
emission density by maximum likelihood, and then use the resulting values to ini-tialize the parameters for EM.
13.2.2 The forward-backward algorithm
Next we seek an efﬁcient procedure for evaluating the quantities γ(znk)and
ξ(zn−1,j,znk), corresponding to the E step of the EM algorithm. The graph for the

hidden Markov model, shown in Figure 13.5, is a tree, and so we know that the
posterior distribution of the latent variables can be obtained efﬁciently using a two-stage message passing algorithm. In the particular context of the hidden Markov Section 8.4
model, this is known as the forward-backward algorithm (Rabiner, 1989), or the
Baum-Welch algorithm (Baum, 1972). There are in fact several variants of the basic
algorithm, all of which lead to the exact marginals, according to the precise form of

13.2. Hidden Markov Models 619
the messages that are propagated along the chain (Jordan, 2007). We shall focus on
the most widely used of these, known as the alpha-beta algorithm.
As well as being of great practical importance in its own right, the forward-
backward algorithm provides us with a nice illustration of many of the concepts
introduced in earlier chapters. We shall therefore begin in this section with a ‘con-ventional’ derivation of the forward-backward equations, making use of the sum
and product rules of probability, and exploiting conditional independence properties
which we shall obtain from the corresponding graphical model using d-separation.Then in Section 13.2.3, we shall see how the forward-backward algorithm can be
obtained very simply as a speciﬁc example of the sum-product algorithm introduced
in Section 8.4.4.
It is worth emphasizing that evaluation of the posterior distributions of the latent
variables is independent of the form of the emission density p(x|z)or indeed of

whether the observed variables are continuous or discrete. All we require is the
values of the quantities p(x
n|zn)for each value of znfor every n. Also, in this
section and the next we shall omit the explicit dependence on the model parametersθ
oldbecause these ﬁxed throughout.
We therefore begin by writing down the following conditional independence
properties (Jordan, 2007)
p(X|zn)= p(x1,...,xn|zn)
p(xn+1,...,xN|zn) (13.24)
p(x1,...,xn−1|xn,zn)= p(x1,...,xn−1|zn) (13.25)

p(x1,...,xn−1|zn−1,zn)= p(x1,...,xn−1|zn−1) (13.26)
p(xn+1,...,xN|zn,zn+1)= p(xn+1,...,xN|zn+1) (13.27)
p(xn+2,...,xN|zn+1,xn+1)= p(xn+2,...,xN|zn+1) (13.28)
p(X|zn−1,zn)= p(x1,...,xn−1|zn−1)
p(xn|zn)p(xn+1,...,xN|zn)(13.29)
p(xN+1|X,zN+1)= p(xN+1|zN+1) (13.30)
p(zN+1|zN,X)= p(zN+1|zN) (13.31)
whereX={x1,...,xN}. These relations are most easily proved using d-separation.
For instance in the ﬁrst of these results, we note that every path from any one of the
nodesx1,...,xn−1to the node xnpasses through the node zn, which is observed.

Because all such paths are head-to-tail, it follows that the conditional independence
property must hold. The reader should take a few moments to verify each of these
properties in turn, as an exercise in the application of d-separation. These relations
can also be proved directly, though with signiﬁcantly greater effort, from the joint
distribution for the hidden Markov model using the sum and product rules of proba-bility. Exercise 13.10
Let us begin by evaluating γ(z
nk). Recall that for a discrete multinomial ran-

dom variable the expected value of one of its components is just the probability ofthat component having the value 1. Thus we are interested in ﬁnding the posterior
distribution p(z
n|x1,...,xN)ofzngiven the observed data set x1,...,xN. This

620 13. SEQUENTIAL DATA
represents a vector of length Kwhose entries correspond to the expected values of
znk. Using Bayes’ theorem, we have
γ(zn)=p(zn|X)=p(X|zn)p(zn)
p(X). (13.32)
Note that the denominator p(X)is implicitly conditioned on the parameters θold
of the HMM and hence represents the likelihood function. Using the conditional
independence property (13.24), together with the product rule of probability, we
obtain
γ(zn)=p(x1,...,xn,zn)p(xn+1,...,xN|zn)
p(X)=α(zn)β(zn)
p(X)(13.33)
α(zn)≡p(x1,...,xn,zn) (13.34)
β(zn)≡p(xn+1,...,xN|zn). (13.35)
The quantity α(zn)represents the joint probability of observing all of the given
data up to time nand the value of zn, whereas β(zn)represents the conditional
probability of all future data from time n+1up toNgiven the value of zn. Again,
α(zn)andβ(zn)each represent set of Knumbers, one for each of the possible
settings of the 1-of- Kcoded binary vector zn. We shall use the notation α(znk)to

denote the value of α(zn)whenznk=1, with an analogous interpretation of β(znk).
We now derive recursion relations that allow α(zn)andβ(zn)to be evaluated
efﬁciently. Again, we shall make use of conditional independence properties, inparticular (13.25) and (13.26), together with the sum and product rules, allowing us
to express α(z
n)in terms of α(zn−1)as follows
α(zn)= p(x1,...,xn,zn)
=p(x1,...,xn|zn)p(zn)
=p(xn|zn)p(x1,...,xn−1|zn)p(zn)
=p(xn|zn)p(x1,...,xn−1,zn)
=p(xn|zn)∑
zn−1p(x1,...,xn−1,zn−1,zn)

zn−1p(x1,...,xn−1,zn|zn−1)p(zn−1)
zn−1p(x1,...,xn−1|zn−1)p(zn|zn−1)p(zn−1)
zn−1p(x1,...,xn−1,zn−1)p(zn|zn−1)
Making use of the deﬁnition (13.34) for α(zn), we then obtain
α(zn)=p(xn|zn)∑
zn−1α(zn−1)p(zn|zn−1). (13.36)

13.2. Hidden Markov Models 621
Figure 13.12 Illustration of the forward recursion (13.36) for
evaluation of the αvariables. In this fragment
of the lattice, we see that the quantity α(zn1)
is obtained by taking the elements α(zn−1,j)of
α(zn−1)at step n−1and summing them up with
weights given by Aj1, corresponding to the val-
ues of p(zn|zn−1), and then multiplying by the
data contribution p(xn|zn1).k=1
k=2
k=3
n−1 nα(zn−1,1)
α(zn−1,2)
α(zn−1,3)α(zn,1)
A11
A21
A31p(xn|zn,1)
It is worth taking a moment to study this recursion relation in some detail. Note
that there are Kterms in the summation, and the right-hand side has to be evaluated
for each of the Kvalues of znso each step of the αrecursion has computational
cost that scaled like O(K2). The forward recursion equation for α(zn)is illustrated
using a lattice diagram in Figure 13.12.
In order to start this recursion, we need an initial condition that is given by
α(z1)=p(x1,z1)=p(z1)p(x1|z1)=K∏
k=1{πkp(x1|φk)}z1k(13.37)

which tells us that α(z1k), fork=1,...,K , takes the value πkp(x1|φk). Starting
at the ﬁrst node of the chain, we can then work along the chain and evaluate α(zn)
for every latent node. Because each step of the recursion involves multiplying by a
K×Kmatrix, the overall cost of evaluating these quantities for the whole chain is
ofO(K2N).
We can similarly ﬁnd a recursion relation for the quantities β(zn)by making
use of the conditional independence properties (13.27) and (13.28) giving
β(zn)= p(xn+1,...,xN|zn)
=∑

ofO(K2N).
=∑
zn+1p(xn+1,...,xN,zn+1|zn)
=∑
zn+1p(xn+1,...,xN|zn,zn+1)p(zn+1|zn)
=∑
zn+1p(xn+1,...,xN|zn+1)p(zn+1|zn)
=∑
zn+1p(xn+2,...,xN|zn+1)p(xn+1|zn+1)p(zn+1|zn).

622 13. SEQUENTIAL DATA
Figure 13.13 Illustration of the backward recursion
(13.38) for evaluation of the βvariables. In
this fragment of the lattice, we see that the
quantity β(zn1)is obtained by taking the
components β(zn+1,k)ofβ(zn+1)at step
n+1and summing them up with weights
given by the products of A1k, correspond-
ing to the values of p(zn+1|zn)and the cor-
responding values of the emission density
p(xn|zn+1,k).k=1
k=2
k=3
nn +1β(zn,1) β(zn+1,1)
β(zn+1,2)
β(zn+1,3)A11
A12
A13p(xn|zn+1,1)
p(xn|zn+1,2)
p(xn|zn+1,3)
Making use of the deﬁnition (13.35) for β(zn), we then obtain
β(zn)=∑
zn+1β(zn+1)p(xn+1|zn+1)p(zn+1|zn). (13.38)
Note that in this case we have a backward message passing algorithm that evaluates
β(zn)in terms of β(zn+1). At each step, we absorb the effect of observation xn+1
through the emission probability p(xn+1|zn+1), multiply by the transition matrix
p(zn+1|zn), and then marginalize out zn+1. This is illustrated in Figure 13.13.

β(zn+1,2)
A12
β(zn)=∑
Again we need a starting condition for the recursion, namely a value for β(zN).
This can be obtained by setting n=Nin (13.33) and replacing α(zN)with its
deﬁnition (13.34) to give
p(zN|X)=p(X,zN)β(zN)
p(X)(13.39)
which we see will be correct provided we take β(zN)=1 for all settings of zN.
In the M step equations, the quantity p(X)will cancel out, as can be seen, for
instance, in the M-step equation for µkgiven by (13.20), which takes the form
µk=n∑
n∑
n=1γ(znk)=n∑
n=1α(znk)β(znk)xn
n∑

µk=n∑
n∑
n∑
n=1α(znk)β(znk). (13.40)
However, the quantity p(X)represents the likelihood function whose value we typ-
ically wish to monitor during the EM optimization, and so it is useful to be able to
evaluate it. If we sum both sides of (13.33) over zn, and use the fact that the left-hand
side is a normalized distribution, we obtain
p(X)=∑
znα(zn)β(zn). (13.41)

13.2. Hidden Markov Models 623
Thus we can evaluate the likelihood function by computing this sum, for any conve-
nient choice of n. For instance, if we only want to evaluate the likelihood function,
then we can do this by running the αrecursion from the start to the end of the chain,
and then use this result for n=N, making use of the fact that β(zN)is a vector of
1s. In this case no βrecursion is required, and we simply have
p(X)=∑
zNα(zN). (13.42)
Let us take a moment to interpret this result for p(X). Recall that to compute the
likelihood we should take the joint distribution p(X,Z)and sum over all possible
values of Z. Each such value represents a particular choice of hidden state for every
time step, in other words every term in the summation is a path through the lattice
diagram, and recall that there are exponentially many such paths. By expressingthe likelihood function in the form (13.42), we have reduced the computational cost

from being exponential in the length of the chain to being linear by swapping the
order of the summation and multiplications, so that at each time step nwe sum
the contributions from all paths passing through each of the states z
nkto give the
intermediate quantities α(zn).
Next we consider the evaluation of the quantities ξ(zn−1,zn), which correspond
to the values of the conditional probabilities p(zn−1,zn|X)for each of the K×K
settings for (zn−1,zn). Using the deﬁnition of ξ(zn−1,zn), and applying Bayes’

theorem, we have
ξ(zn−1,zn)=p(zn−1,zn|X)
=p(X|zn−1,zn)p(zn−1,zn)
p(X)
=p(x1,...,xn−1|zn−1)p(xn|zn)p(xn+1,...,xN|zn)p(zn|zn−1)p(zn−1)
p(X)
=α(zn−1)p(xn|zn)p(zn|zn−1)β(zn)
p(X)(13.43)
where we have made use of the conditional independence property (13.29) together
with the deﬁnitions of α(zn)andβ(zn)given by (13.34) and (13.35). Thus we can
calculate the ξ(zn−1,zn)directly by using the results of the αandβrecursions.
Let us summarize the steps required to train a hidden Markov model using

p(X)
p(X)
the EM algorithm. We ﬁrst make an initial selection of the parameters θoldwhere
θ≡(π,A,φ). TheAandπparameters are often initialized either uniformly or
randomly from a uniform distribution (respecting their non-negativity and summa-
tion constraints). Initialization of the parameters φwill depend on the form of the
distribution. For instance in the case of Gaussians, the parameters µkmight be ini-
tialized by applying the K-means algorithm to the data, and Σkmight be initialized

to the covariance matrix of the corresponding Kmeans cluster. Then we run both
the forward αrecursion and the backward βrecursion and use the results to evaluate
γ(zn)andξ(zn−1,zn). At this stage, we can also evaluate the likelihood function.

624 13. SEQUENTIAL DATA
This completes the E step, and we use the results to ﬁnd a revised set of parameters
θnewusing the M-step equations from Section 13.2.1. We then continue to alternate
between E and M steps until some convergence criterion is satisﬁed, for instance
when the change in the likelihood function is below some threshold.
Note that in these recursion relations the observations enter through conditional
distributions of the form p(xn|zn). The recursions are therefore independent of
the type or dimensionality of the observed variables or the form of this conditional
distribution, so long as its value can be computed for each of the Kpossible states
ofzn. Since the observed variables {xn}are ﬁxed, the quantities p(xn|zn)can be
pre-computed as functions of znat the start of the EM algorithm, and remain ﬁxed
We have seen in earlier chapters that the maximum likelihood approach is most

effective when the number of data points is large in relation to the number of parame-ters. Here we note that a hidden Markov model can be trained effectively, using max-
imum likelihood, provided the training sequence is sufﬁciently long. Alternatively,
we can make use of multiple shorter sequences, which requires a straightforwardmodiﬁcation of the hidden Markov model EM algorithm. In the case of left-to-right Exercise 13.12
models, this is particularly important because, in a given observation sequence, a

given state transition corresponding to a nondiagonal element of Awill seen at most
once.
Another quantity of interest is the predictive distribution, in which the observed
data isX={x
1,...,xN}and we wish to predict xN+1, which would be important
for real-time applications such as ﬁnancial forecasting. Again we make use of the
sum and product rules together with the conditional independence properties (13.29)
and (13.31) giving
p(xN+1|X)=∑
zN+1p(xN+1,zN+1|X)
=∑
zN+1p(xN+1|zN+1)p(zN+1|X)
=∑
zN+1p(xN+1|zN+1)∑
zNp(zN+1,zN|X)
=∑
zNp(zN+1|zN)p(zN|X)
=∑
zNp(zN+1|zN)p(zN,X)
p(X)
=1
p(X)∑
zNp(zN+1|zN)α(zN)(13.44)

=∑
=∑
=∑
=∑
p(X)
=1
p(X)∑
which can be evaluated by ﬁrst running a forward αrecursion and then computing
the ﬁnal summations over zNandzN+1. The result of the ﬁrst summation over zN
can be stored and used once the value of xN+1is observed in order to run the α
recursion forward to the next step in order to predict the subsequent value xN+2.

13.2. Hidden Markov Models 625
Figure 13.14 A fragment of the fac-
tor graph representation for the hidden
Markov model.χψ n
g1 gn−1 gnz1 zn−1 zn
x1 xn−1 xn
Note that in (13.44), the inﬂuence of all data from x1toxNis summarized in the K
values of α(zN). Thus the predictive distribution can be carried forward indeﬁnitely
using a ﬁxed amount of storage, as may be required for real-time applications.
Here we have discussed the estimation of the parameters of an HMM using max-
imum likelihood. This framework is easily extended to regularized maximum likeli-
hood by introducing priors over the model parameters π,Aandφwhose values are
then estimated by maximizing their posterior probability. This can again be done us-
ing the EM algorithm in which the E step is the same as discussed above, and the M
step involves adding the log of the prior distribution p(θ)to the function Q(θ,θold)
before maximization and represents a straightforward application of the techniques

developed at various points in this book. Furthermore, we can use variational meth-
ods to give a fully Bayesian treatment of the HMM in which we marginalize over the Section 10.1
parameter distributions (MacKay, 1997). As with maximum likelihood, this leads to
a two-pass forward-backward recursion to compute posterior probabilities.
13.2.3 The sum-product algorithm for the HMM
The directed graph that represents the hidden Markov model, shown in Fig-

ure 13.5, is a tree and so we can solve the problem of ﬁnding local marginals for the
hidden variables using the sum-product algorithm. Not surprisingly, this turns out to Section 8.4.4
be equivalent to the forward-backward algorithm considered in the previous section,
and so the sum-product algorithm therefore provides us with a simple way to derive
the alpha-beta recursion formulae.
We begin by transforming the directed graph of Figure 13.5 into a factor graph,
of which a representative fragment is shown in Figure 13.14. This form of the fac-

tor graph shows all variables, both latent and observed, explicitly. However, for
the purpose of solving the inference problem, we shall always be conditioning on
the variables x1,...,xN, and so we can simplify the factor graph by absorbing the
emission probabilities into the transition probability factors. This leads to the sim-
pliﬁed factor graph representation in Figure 13.15, in which the factors are given
by
h(z1)= p(z1)p(x1|z1) (13.45)
fn(zn−1,zn)= p(zn|zn−1)p(xn|zn). (13.46)

626 13. SEQUENTIAL DATA
Figure 13.15 A simpliﬁed form of fac-
tor graph to describe the hidden Markov
model.h fn
z1 zn−1 zn
To derive the alpha-beta algorithm, we denote the ﬁnal hidden variable zNas
the root node, and ﬁrst pass messages from the leaf node hto the root. From the
general results (8.66) and (8.69) for message propagation, we see that the messages
which are propagated in the hidden Markov model take the form
µzn−1→fn(zn−1)= µfn−1→zn−1(zn−1) (13.47)
µfn→zn(zn)=∑
zn−1fn(zn−1,zn)µzn−1→fn(zn−1) (13.48)
These equations represent the propagation of messages forward along the chain and
are equivalent to the alpha recursions derived in the previous section, as we shall
now show. Note that because the variable nodes znhave only two neighbours, they
perform no computation.
We can eliminate µzn−1→fn(zn−1)from (13.48) using (13.47) to give a recur-
sion for the f→zmessages of the form
zn−1fn(zn−1,zn)µfn−1→zn−1(zn−1). (13.49)
If we now recall the deﬁnition (13.46), and if we deﬁne

α(zn)=µfn→zn(zn) (13.50)
then we obtain the alpha recursion given by (13.36). We also need to verify that
the quantities α(zn)are themselves equivalent to those deﬁned previously. This
is easily done by using the initial condition (8.71) and noting that α(z1)is given
byh(z1)=p(z1)p(x1|z1)which is identical to (13.37). Because the initial αis
the same, and because they are iteratively computed using the same equation, all
subsequent αquantities must be the same.

Next we consider the messages that are propagated from the root node back to
the leaf node. These take the form
µfn+1→fn(zn)=∑
zn+1fn+1(zn,zn+1)µfn+2→fn+1(zn+1) (13.51)
where, as before, we have eliminated the messages of the type z→fsince the
variable nodes perform no computation. Using the deﬁnition (13.46) to substitute
forfn+1(zn,zn+1), and deﬁning
β(zn)=µfn+1→zn(zn) (13.52)

13.2. Hidden Markov Models 627
we obtain the beta recursion given by (13.38). Again, we can verify that the beta
variables themselves are equivalent by noting that (8.70) implies that the initial mes-sage send by the root variable node is µ
zN→fN(zN)=1 , which is identical to the
initialization of β(zN)given in Section 13.2.2.
The sum-product algorithm also speciﬁes how to evaluate the marginals once all
the messages have been evaluated. In particular, the result (8.63) shows that the local
marginal at the node znis given by the product of the incoming messages. Because
we have conditioned on the variables X={x1,...,xN}, we are computing the
joint distribution
p(zn,X)=µfn→zn(zn)µfn+1→zn(zn)=α(zn)β(zn). (13.53)
Dividing both sides by p(X), we then obtain
γ(zn)=p(zn,X)
p(X)(13.54)
in agreement with (13.33). The result (13.43) can similarly be derived from (8.72). Exercise 13.11
13.2.4 Scaling factors
There is an important issue that must be addressed before we can make use of the

forward backward algorithm in practice. From the recursion relation (13.36), we note
that at each step the new value α(zn)is obtained from the previous value α(zn−1)
by multiplying by quantities p(zn|zn−1)andp(xn|zn). Because these probabilities
are often signiﬁcantly less than unity, as we work our way forward along the chain,
the values of α(zn)can go to zero exponentially quickly. For moderate lengths of
chain (say 100 or so), the calculation of the α(zn)will soon exceed the dynamic
range of the computer, even if double precision ﬂoating point is used.

In the case of i.i.d. data, we implicitly circumvented this problem with the eval-
uation of likelihood functions by taking logarithms. Unfortunately, this will not help
here because we are forming sums of products of small numbers (we are in fact im-
plicitly summing over all possible paths through the lattice diagram of Figure 13.7).
We therefore work with re-scaled versions of α(zn)andβ(zn)whose values remain

of order unity. As we shall see, the corresponding scaling factors cancel out whenwe use these re-scaled quantities in the EM algorithm.
In (13.34), we deﬁned α(z
n)=p(x1,...,xn,zn)representing the joint distri-
bution of all the observations up to xnand the latent variable zn. Now we deﬁne a
normalized version of αgiven by
ˆα(zn)=p(zn|x1,...,xn)=α(zn)
p(x1,...,xn)(13.55)
which we expect to be well behaved numerically because it is a probability distribu-

tion over Kvariables for any value of n. In order to relate the scaled and original al-
pha variables, we introduce scaling factors deﬁned by conditional distributions overthe observed variables
c
n=p(xn|x1,...,xn−1). (13.56)

628 13. SEQUENTIAL DATA
From the product rule, we then have
p(x1,...,xn)=n∏
m=1cm (13.57)
and so
α(zn)=p(zn|x1,...,xn)p(x1,...,xn)=(n∏
m=1cm)
ˆα(zn). (13.58)
We can then turn the recursion equation (13.36) for αinto one for ˆαgiven by
cnˆα(zn)=p(xn|zn)∑
zn−1ˆα(zn−1)p(zn|zn−1). (13.59)
Note that at each stage of the forward message passing phase, used to evaluate ˆα(zn),
we have to evaluate and store cn, which is easily done because it is the coefﬁcient
that normalizes the right-hand side of (13.59) to give ˆα(zn).
We can similarly deﬁne re-scaled variables ˆβ(zn)using
β(zn)=(N∏
m=n+1cm)
ˆβ(zn) (13.60)
which will again remain within machine precision because, from (13.35), the quan-
titiesˆβ(zn)are simply the ratio of two conditional probabilities
ˆβ(zn)=p(xn+1,...,xN|zn)
p(xn+1,...,xN|x1,...,xn). (13.61)
The recursion result (13.38) for βthen gives the following recursion for the re-scaled
variables
cn+1ˆβ(zn)=∑
zn+1ˆβ(zn+1)p(xn+1|zn+1)p(zn+1|zn). (13.62)

β(zn)=(N∏
m=n+1cm)
variables
In applying this recursion relation, we make use of the scaling factors cnthat were
previously computed in the αphase.
From (13.57), we see that the likelihood function can be found using
p(X)=N∏
n=1cn. (13.63)
Similarly, using (13.33) and (13.43), together with (13.63), we see that the required
marginals are given by Exercise 13.15
γ(zn)=ˆα(zn)ˆβ(zn) (13.64)
ξ(zn−1,zn)= cnˆα(zn−1)p(xn|zn)p(zn|z−1)ˆβ(zn). (13.65)

13.2. Hidden Markov Models 629
Finally, we note that there is an alternative formulation of the forward-backward
algorithm (Jordan, 2007) in which the backward pass is deﬁned by a recursion based
the quantities γ(zn)=ˆα(zn)ˆβ(zn)instead of using ˆβ(zn). This α–γrecursion
requires that the forward pass be completed ﬁrst so that all the quantities ˆα(zn)
are available for the backward pass, whereas the forward and backward passes of
theα–βalgorithm can be done independently. Although these two algorithms have
comparable computational cost, the α–βversion is the most commonly encountered
one in the case of hidden Markov models, whereas for linear dynamical systems a Section 13.3
recursion analogous to the α–γform is more usual.
13.2.5 The Viterbi algorithm
In many applications of hidden Markov models, the latent variables have some
meaningful interpretation, and so it is often of interest to ﬁnd the most probablesequence of hidden states for a given observation sequence. For instance in speech

recognition, we might wish to ﬁnd the most probable phoneme sequence for a given
series of acoustic observations. Because the graph for the hidden Markov model isa directed tree, this problem can be solved exactly using the max-sum algorithm.
We recall from our discussion in Section 8.4.5 that the problem of ﬁnding the most
probable sequence of latent states is not the same as that of ﬁnding the set of statesthat are individually the most probable. The latter problem can be solved by ﬁrst

running the forward-backward (sum-product) algorithm to ﬁnd the latent variable
marginals γ(z
n)and then maximizing each of these individually (Duda et al. , 2001).
However, the set of such states will not, in general, correspond to the most probable
sequence of states. In fact, this set of states might even represent a sequence havingzero probability, if it so happens that two successive states, which in isolation are
individually the most probable, are such that the transition matrix element connecting
them is zero.

In practice, we are usually interested in ﬁnding the most probable sequence of
states, and this can be solved efﬁciently using the max-sum algorithm, which in the
context of hidden Markov models is known as the Viterbi algorithm (Viterbi, 1967).
Note that the max-sum algorithm works with log probabilities and so there is no
need to use re-scaled variables as was done with the forward-backward algorithm.
Figure 13.16 shows a fragment of the hidden Markov model expanded as latticediagram. As we have already noted, the number of possible paths through the lattice

grows exponentially with the length of the chain. The Viterbi algorithm searches this
space of paths efﬁciently to ﬁnd the most probable path with a computational costthat grows only linearly with the length of the chain.
As with the sum-product algorithm, we ﬁrst represent the hidden Markov model
as a factor graph, as shown in Figure 13.15. Again, we treat the variable node z
N
as the root, and pass messages to the root starting with the leaf nodes. Using the

N
results (8.93) and (8.94), we see that the messages passed in the max-sum algorithmare given by
µ
zn→fn+1(zn)= µfn→zn(zn) (13.66)
µfn+1→zn+1(zn+1) = max
zn{
lnfn+1(zn,zn+1)+µzn→fn+1(zn)}
.(13.67)

630 13. SEQUENTIAL DATA
Figure 13.16 A fragment of the HMM lattice
showing two possible paths. The Viterbi algorithm
efﬁciently determines the most probable path from
amongst the exponentially many possibilities. For
any given path, the corresponding probability is
given by the product of the elements of the tran-
sition matrix Ajk, corresponding to the probabil-
itiesp(zn+1|zn)for each segment of the path,
along with the emission densities p(xn|k)asso-
ciated with each node on the path.k=1
k=2
k=3
If we eliminate µzn→fn+1(zn)between these two equations, and make use of (13.46),
we obtain a recursion for the f→zmessages of the form
ω(zn+1)=l n p(xn+1|zn+1) + max
zn{lnp(x+1|zn)+ω(zn)} (13.68)
where we have introduced the notation ω(zn)≡µfn→zn(zn).
From (8.95) and (8.96), these messages are initialized using
ω(z1)=l n p(z1)+l n p(x1|z1). (13.69)
where we have used (13.45). Note that to keep the notation uncluttered, we omit

k=2
k=3
the dependence on the model parameters θthat are held ﬁxed when ﬁnding the most
probable sequence.
The Viterbi algorithm can also be derived directly from the deﬁnition (13.6) of
the joint distribution by taking the logarithm and then exchanging maximizations
and summations. It is easily seen that the quantities ω(zn)have the probabilistic Exercise 13.16
interpretation
ω(zn)= m a x
z1,...,zn−1p(x1,...,xn,z1,...,zn). (13.70)
Once we have completed the ﬁnal maximization over zN, we will obtain the

ω(zn)= m a x
value of the joint distribution p(X,Z)corresponding to the most probable path. We
also wish to ﬁnd the sequence of latent variable values that corresponds to this path.
To do this, we simply make use of the back-tracking procedure discussed in Sec-
tion 8.4.5. Speciﬁcally, we note that the maximization over znmust be performed
for each of the Kpossible values of zn+1. Suppose we keep a record of the values
ofznthat correspond to the maxima for each value of the Kvalues of zn+1. Let us

denote this function by ψ(kn)where k∈{1,...,K }. Once we have passed mes-
sages to the end of the chain and found the most probable state of zN, we can then
use this function to backtrack along the chain by applying it recursively
kmax
n=ψ(kmax
n+1). (13.71)

13.2. Hidden Markov Models 631
Intuitively, we can understand the Viterbi algorithm as follows. Naively, we
could consider explicitly all of the exponentially many paths through the lattice,evaluate the probability for each, and then select the path having the highest proba-
bility. However, we notice that we can make a dramatic saving in computational cost
as follows. Suppose that for each path we evaluate its probability by summing upproducts of transition and emission probabilities as we work our way forward along
each path through the lattice. Consider a particular time step nand a particular state
kat that time step. There will be many possible paths converging on the correspond-
ing node in the lattice diagram. However, we need only retain that particular path
that so far has the highest probability. Because there are Kstates at time step n,w e
need to keep track of Ksuch paths. At time step n+1, there will be K
2possible
paths to consider, comprising Kpossible paths leading out of each of the Kcurrent

2possible
states, but again we need only retain Kof these corresponding to the best path for
each state at time n+1. When we reach the ﬁnal time step Nwe will discover which
state corresponds to the overall most probable path. Because there is a unique path
coming into that state we can trace the path back to step N−1to see what state it
occupied at that time, and so on back through the lattice to the state n=1.
13.2.6 Extensions of the hidden Markov model

The basic hidden Markov model, along with the standard training algorithm
based on maximum likelihood, has been extended in numerous ways to meet therequirements of particular applications. Here we discuss a few of the more important
examples.
We see from the digits example in Figure 13.11 that hidden Markov models can
be quite poor generative models for the data, because many of the synthetic digits
look quite unrepresentative of the training data. If the goal is sequence classiﬁca-

examples.
tion, there can be signiﬁcant beneﬁt in determining the parameters of hidden Markovmodels using discriminative rather than maximum likelihood techniques. Suppose
we have a training set of Robservation sequences X
r, where r=1,...,R , each of
which is labelled according to its class m, where m=1,...,M . For each class, we
have a separate hidden Markov model with its own parameters θm, and we treat the
problem of determining the parameter values as a standard classiﬁcation problem in
which we optimize the cross-entropy
R∑

R∑
r=1lnp(mr|Xr). (13.72)
Using Bayes’ theorem this can be expressed in terms of the sequence probabilities
associated with the hidden Markov models
R∑
r=1ln{
p(Xr|θr)p(mr)∑M
l=1p(Xr|θl)p(lr)}
(13.73)
where p(m)is the prior probability of class m. Optimization of this cost function
is more complex than for maximum likelihood (Kapadia, 1998), and in particular

632 13. SEQUENTIAL DATA
Figure 13.17 Section of an autoregressive hidden
Markov model, in which the distribution
of the observation xndepends on a
subset of the previous observations as
well as on the hidden state zn. In this
example, the distribution of xndepends
on the two previous observations xn−1
andxn−2.zn−1 zn zn+1
xn−1 xn xn+1
requires that every training sequence be evaluated under each of the models in or-
der to compute the denominator in (13.73). Hidden Markov models, coupled with
discriminative training methods, are widely used in speech recognition (Kapadia,
1998).
A signiﬁcant weakness of the hidden Markov model is the way in which it rep-
resents the distribution of times for which the system remains in a given state. To see
the problem, note that the probability that a sequence sampled from a given hidden
Markov model will spend precisely Tsteps in state kand then make a transition to a
different state is given by
p(T)=(Akk)T(1−Akk)∝exp (−TlnAkk) (13.74)

1998).
and so is an exponentially decaying function of T. For many applications, this will
be a very unrealistic model of state duration. The problem can be resolved by mod-
elling state duration directly in which the diagonal coefﬁcients Akkare all set to zero,
and each state kis explicitly associated with a probability distribution p(T|k)of pos-
sible duration times. From a generative point of view, when a state kis entered, a
valueTrepresenting the number of time steps that the system will remain in state k

is then drawn from p(T|k). The model then emits Tvalues of the observed variable
xt, which are generally assumed to be independent so that the corresponding emis-
sion density is simply∏T
t=1p(xt|k). This approach requires some straightforward
modiﬁcations to the EM optimization procedure (Rabiner, 1989).
Another limitation of the standard HMM is that it is poor at capturing long-
range correlations between the observed variables (i.e., between variables that are

separated by many time steps) because these must be mediated via the ﬁrst-order
Markov chain of hidden states. Longer-range effects could in principle be included
by adding extra links to the graphical model of Figure 13.5. One way to address this
is to generalize the HMM to give the autoregressive hidden Markov model (Ephraim
et al. , 1989), an example of which is shown in Figure 13.17. For discrete observa-
tions, this corresponds to expanded tables of conditional probabilities for the emis-

sion distributions. In the case of a Gaussian emission density, we can use the linear-
Gaussian framework in which the conditional distribution for xngiven the values
of the previous observations, and the value of zn, is a Gaussian whose mean is a
linear combination of the values of the conditioning variables. Clearly the number
of additional links in the graph must be limited to avoid an excessive the number of
free parameters. In the example shown in Figure 13.17, each observation depends on

13.2. Hidden Markov Models 633
Figure 13.18 Example of an input-output hidden
Markov model. In this case, both the
emission probabilities and the transition
probabilities depend on the values of a
sequence of observations u1,...,uN.
zn−1 zn zn+1
xn−1 xn xn+1un−1 un un+1
the two preceding observed variables as well as on the hidden state. Although this
graph looks messy, we can again appeal to d-separation to see that in fact it still has
a simple probabilistic structure. In particular, if we imagine conditioning on znwe
see that, as with the standard HMM, the values of zn−1andzn+1are independent,
corresponding to the conditional independence property (13.5). This is easily veri-
ﬁed by noting that every path from node zn−1to node zn+1passes through at least
one observed node that is head-to-tail with respect to that path. As a consequence,
we can again use a forward-backward recursion in the E step of the EM algorithm to
determine the posterior distributions of the latent variables in a computational time

that is linear in the length of the chain. Similarly, the M step involves only a minor
modiﬁcation of the standard M-step equations. In the case of Gaussian emission
densities this involves estimating the parameters using the standard linear regression
equations, discussed in Chapter 3.
We have seen that the autoregressive HMM appears as a natural extension of the
standard HMM when viewed as a graphical model. In fact the probabilistic graphical

modelling viewpoint motivates a plethora of different graphical structures based on
the HMM. Another example is the input-output hidden Markov model (Bengio and
Frasconi, 1995), in which we have a sequence of observed variables u1,...,uN,i n
addition to the output variables x1,...,xN, whose values inﬂuence either the dis-
tribution of latent variables or output variables, or both. An example is shown in
Figure 13.18. This extends the HMM framework to the domain of supervised learn-
ing for sequential data. It is again easy to show, through the use of the d-separation

criterion, that the Markov property (13.5) for the chain of latent variables still holds.
To verify this, simply note that there is only one path from node zn−1to node zn+1
and this is head-to-tail with respect to the observed node zn. This conditional inde-
pendence property again allows the formulation of a computationally efﬁcient learn-
ing algorithm. In particular, we can determine the parameters θof the model by
maximizing the likelihood function L(θ)=p(X|U,θ)whereUis a matrix whose
rows are given by uT

n. As a consequence of the conditional independence property
(13.5) this likelihood function can be maximized efﬁciently using an EM algorithm
in which the E step involves forward and backward recursions. Exercise 13.18
Another variant of the HMM worthy of mention is the factorial hidden Markov
model (Ghahramani and Jordan, 1997), in which there are multiple independent

634 13. SEQUENTIAL DATA
Figure 13.19 A factorial hidden Markov model com-
prising two Markov chains of latent vari-
ables. For continuous observed variables
x, one possible choice of emission model
is a linear-Gaussian density in which the
mean of the Gaussian is a linear combi-
nation of the states of the corresponding
latent variables.z(1)
n−1 z(1)
nz(1)
n+1z(2)
n−1 z(2)
nz(2)
n+1
Markov chains of latent variables, and the distribution of the observed variable at
a given time step is conditional on the states of all of the corresponding latent vari-
ables at that same time step. Figure 13.19 shows the corresponding graphical model.
The motivation for considering factorial HMM can be seen by noting that in order to
represent, say, 10 bits of information at a given time step, a standard HMM would
needK=210= 1024 latent states, whereas a factorial HMM could make use of 10
binary latent chains. The primary disadvantage of factorial HMMs, however, lies in

the additional complexity of training them. The M step for the factorial HMM model
is straightforward. However, observation of the xvariables introduces dependencies
between the latent chains, leading to difﬁculties with the E step. This can be seen
by noting that in Figure 13.19, the variables z(1)
nandz(2)
nare connected by a path
which is head-to-head at node xnand hence they are not d-separated. The exact E
step for this model does notcorrespond to running forward and backward recursions

nandz(2)
along the MMarkov chains independently. This is conﬁrmed by noting that the key
conditional independence property (13.5) is not satisﬁed for the individual Markov
chains in the factorial HMM model, as is shown using d-separation in Figure 13.20.
Now suppose that there are Mchains of hidden nodes and for simplicity suppose
that all latent variables have the same number Kof states. Then one approach would
be to note that there are KMcombinations of latent variables at a given time step

Figure 13.20 Example of a path, highlighted in green,
which is head-to-head at the observed
nodes xn−1andxn+1, and head-to-tail
at the unobserved nodes z(2)
n−1,z(2)
nand
z(2)
n+1. Thus the path is not blocked and
so the conditional independence property
(13.5) does not hold for the individual la-
tent chains of the factorial HMM model.
As a consequence, there is no efﬁcient
exact E step for this model.z(1)
n−1 z(1)
nz(1)
n+1z(2)
n−1 z(2)
nz(2)
n+1

13.3. Linear Dynamical Systems 635
and so we can transform the model into an equivalent standard HMM having a single
chain of latent variables each of which has KMlatent states. We can then run the
standard forward-backward recursions in the E step. This has computational com-
plexity O(NK2M)that is exponential in the number Mof latent chains and so will
be intractable for anything other than small values of M. One solution would be
to use sampling methods (discussed in Chapter 11). As an elegant deterministic al-
ternative, Ghahramani and Jordan (1997) exploited variational inference techniques Section 10.1
to obtain a tractable algorithm for approximate inference. This can be done usinga simple variational posterior distribution that is fully factorized with respect to the
latent variables, or alternatively by using a more powerful approach in which the
variational distribution is described by independent Markov chains corresponding to

the chains of latent variables in the original model. In the latter case, the variational
inference algorithms involves running independent forward and backward recursionsalong each chain, which is computationally efﬁcient and yet is also able to capture
correlations between variables within the same chain.
Clearly, there are many possible probabilistic structures that can be constructed
according to the needs of particular applications. Graphical models provide a general
technique for motivating, describing, and analysing such structures, and variational

methods provide a powerful framework for performing inference in those models forwhich exact solution is intractable.
13.3. Linear Dynamical Systems
In order to motivate the concept of linear dynamical systems, let us consider thefollowing simple problem, which often arises in practical settings. Suppose we wish
to measure the value of an unknown quantity zusing a noisy sensor that returns a
observation xrepresenting the value of zplus zero-mean Gaussian noise. Given a
single measurement, our best guess for zis to assume that z=x. However, we

can improve our estimate for zby taking lots of measurements and averaging them,
because the random noise terms will tend to cancel each other. Now let’s make thesituation more complicated by assuming that we wish to measure a quantity zthat
is changing over time. We can take regular measurements of xso that at some point
in time we have obtained x
1,...,xNand we wish to ﬁnd the corresponding values
z1,...,xN. If we simply average the measurements, the error due to random noise

will be reduced, but unfortunately we will just obtain a single averaged estimate, inwhich we have averaged over the changing value of z, thereby introducing a new
source of error.
Intuitively, we could imagine doing a bit better as follows. To estimate the value
ofz
N, we take only the most recent few measurements, say xN−L,...,xNand just
average these. If zis changing slowly, and the random noise level in the sensor is

ofz
high, it would make sense to choose a relatively long window of observations toaverage. Conversely, if the signal is changing quickly, and the noise levels are small,
we might be better just to use x
Ndirectly as our estimate of zN. Perhaps we could
do even better if we take a weighted average, in which more recent measurements

636 13. SEQUENTIAL DATA
make a greater contribution than less recent ones.
Although this sort of intuitive argument seems plausible, it does not tell us how
to form a weighted average, and any sort of hand-crafted weighing is hardly likely
to be optimal. Fortunately, we can address problems such as this much more sys-
tematically by deﬁning a probabilistic model that captures the time evolution andmeasurement processes and then applying the inference and learning methods devel-
oped in earlier chapters. Here we shall focus on a widely used model known as a
linear dynamical system .
As we have seen, the HMM corresponds to the state space model shown in
Figure 13.5 in which the latent variables are discrete but with arbitrary emission
probability distributions. This graph of course describes a much broader class of
probability distributions, all of which factorize according to (13.6). We now consider

extensions to other distributions for the latent variables. In particular, we considercontinuous latent variables in which the summations of the sum-product algorithm
become integrals. The general form of the inference algorithms will, however, be
the same as for the hidden Markov model. It is interesting to note that, historically,hidden Markov models and linear dynamical systems were developed independently.
Once they are both expressed as graphical models, however, the deep relationship
between them immediately becomes apparent.

One key requirement is that we retain an efﬁcient algorithm for inference which
is linear in the length of the chain. This requires that, for instance, when we take
a quantity
ˆα(zn−1), representing the posterior probability of zngiven observations
x1,...,xn, and multiply by the transition probability p(zn|zn−1)and the emission
probability p(xn|zn)and then marginalize over zn−1, we obtain a distribution over
znthat is of the same functional form as that over ˆα(zn−1). That is to say, the
distribution must not become more complex at each stage, but must only change in

a quantity
its parameter values. Not surprisingly, the only distributions that have this property
of being closed under multiplication are those belonging to the exponential family.
Here we consider the most important example from a practical perspective,
which is the Gaussian. In particular, we consider a linear-Gaussian state space modelso that the latent variables {z
n}, as well as the observed variables {xn}, are multi-
variate Gaussian distributions whose means are linear functions of the states of their

parents in the graph. We have seen that a directed graph of linear-Gaussian unitsis equivalent to a joint Gaussian distribution over all of the variables. Furthermore,
marginals such as
ˆα(zn)are also Gaussian, so that the functional form of the mes-
sages is preserved and we will obtain an efﬁcient inference algorithm. By contrast,suppose that the emission densities p(x
n|zn)comprise a mixture of KGaussians
each of which has a mean that is linear in zn. Then even if ˆα(z1)is Gaussian, the

quantityˆα(z2)will be a mixture of KGaussians, ˆα(z3)will be a mixture of K2
Gaussians, and so on, and exact inference will not be of practical value.
We have seen that the hidden Markov model can be viewed as an extension of
the mixture models of Chapter 9 to allow for sequential correlations in the data.In a similar way, we can view the linear dynamical system as a generalization of the
continuous latent variable models of Chapter 12 such as probabilistic PCA and factor
analysis. Each pair of nodes {z

n,xn}represents a linear-Gaussian latent variable

13.3. Linear Dynamical Systems 637
model for that particular observation. However, the latent variables {zn}are no
longer treated as independent but now form a Markov chain.
Because the model is represented by a tree-structured directed graph, inference
problems can be solved efﬁciently using the sum-product algorithm. The forward re-
cursions, analogous to the αmessages of the hidden Markov model, are known as the
Kalman ﬁlter equations (Kalman, 1960; Zarchan and Musoff, 2005), and the back-
ward recursions, analogous to the βmessages, are known as the Kalman smoother
equations, or the Rauch-Tung-Striebel (RTS) equations (Rauch et al. , 1965). The
Kalman ﬁlter is widely used in many real-time tracking applications.
Because the linear dynamical system is a linear-Gaussian model, the joint distri-
bution over all variables, as well as all marginals and conditionals, will be Gaussian.
It follows that the sequence of individually most probable latent variable values is

the same as the most probable latent sequence. There is thus no need to consider the Exercise 13.19
analogue of the Viterbi algorithm for the linear dynamical system.
Because the model has linear-Gaussian conditional distributions, we can write
the transition and emission distributions in the general form
p(zn|zn−1)= N(zn|Azn−1,Γ) (13.75)
p(xn|zn)= N(xn|Czn,Σ). (13.76)
The initial latent variable also has a Gaussian distribution which we write as
p(z1)=N(z1|µ0,V0). (13.77)

Note that in order to simplify the notation, we have omitted additive constant terms
from the means of the Gaussians. In fact, it is straightforward to include them ifdesired. Traditionally, these distributions are more commonly expressed in an equiv- Exercise 13.24
alent form in terms of noisy linear equations given by
z
n=Azn−1+wn (13.78)
xn=Czn+vn (13.79)
z1=µ0+u (13.80)
where the noise terms have the distributions
w∼N (w|0,Γ) (13.81)
v∼N (v|0,Σ) (13.82)
u∼N (u|0,V0). (13.83)

z
The parameters of the model, denoted by θ={A,Γ,C,Σ,µ0,V0}, can be
determined using maximum likelihood through the EM algorithm. In the E step, weneed to solve the inference problem of determining the local posterior marginals for
the latent variables, which can be solved efﬁciently using the sum-product algorithm,
as we discuss in the next section.

638 13. SEQUENTIAL DATA
13.3.1 Inference in LDS
We now turn to the problem of ﬁnding the marginal distributions for the latent
variables conditional on the observation sequence. For given parameter settings, we
also wish to make predictions of the next latent state znand of the next observation
xnconditioned on the observed data x1,...,xn−1for use in real-time applications.
These inference problems can be solved efﬁciently using the sum-product algorithm,
which in the context of the linear dynamical system gives rise to the Kalman ﬁlter
and Kalman smoother equations.
It is worth emphasizing that because the linear dynamical system is a linear-
Gaussian model, the joint distribution over all latent and observed variables is simplya Gaussian, and so in principle we could solve inference problems by using the
standard results derived in previous chapters for the marginals and conditionals of a

multivariate Gaussian. The role of the sum-product algorithm is to provide a moreefﬁcient way to perform such computations.
Linear dynamical systems have the identical factorization, given by (13.6), to
hidden Markov models, and are again described by the factor graphs in Figures 13.14and 13.15. Inference algorithms therefore take precisely the same form except that
summations over latent variables are replaced by integrations. We begin by consid-
ering the forward equations in which we treat z
Nas the root node, and propagate

messages from the leaf node h(z1)to the root. From (13.77), the initial message will
be Gaussian, and because each of the factors is Gaussian, all subsequent messages
will also be Gaussian. By convention, we shall propagate messages that are nor-malized marginal distributions corresponding to p(z
n|x1,...,xn), which we denote
by
ˆα(zn)=N(zn|µn,Vn). (13.84)
This is precisely analogous to the propagation of scaled variables ˆα(zn)given by
(13.59) in the discrete case of the hidden Markov model, and so the recursion equa-
tion now takes the form
cnˆα(zn)=p(xn|zn)∫
ˆα(zn−1)p(zn|zn−1)dzn−1. (13.85)

by
Substituting for the conditionals p(zn|zn−1)andp(xn|zn), using (13.75) and (13.76),
respectively, and making use of (13.84), we see that (13.85) becomes
cnN(zn|µn,Vn)=N(xn|Czn,Σ)∫
N(zn|Azn−1,Γ)N(zn−1|µn−1,Vn−1)dzn−1. (13.86)
Here we are supposing that µn−1andVn−1are known, and by evaluating the inte-
gral in (13.86), we wish to determine values for µnandVn. The integral is easily
evaluated by making use of the result (2.115), from which it follows that
∫
N(zn|Azn−1,Γ)N(zn−1|µn−1,Vn−1)dzn−1
=N(zn|Aµn−1,Pn−1) (13.87)

13.3. Linear Dynamical Systems 639
Pn−1=AVn−1AT+Γ. (13.88)
We can now combine this result with the ﬁrst factor on the right-hand side of (13.86)
by making use of (2.115) and (2.116) to give
µn=Aµn−1+Kn(xn−CAµn−1) (13.89)
Vn=(I−KnC)Pn−1 (13.90)
cn=N(xn|CAµn−1,CPn−1CT+Σ). (13.91)
Here we have made use of the matrix inverse identities (C.5) and (C.7) and also
deﬁned the Kalman gain matrix
Kn=Pn−1CT(
CPn−1CT+Σ)−1. (13.92)
Thus, given the values of µn−1andVn−1, together with the new observation xn,
we can evaluate the Gaussian marginal for znhaving mean µnand covariance Vn,
as well as the normalization coefﬁcient cn.
The initial conditions for these recursion equations are obtained from
c1ˆα(z1)=p(z1)p(x1|z1). (13.93)
Because p(z1)is given by (13.77), and p(x1|z1)is given by (13.76), we can again
make use of (2.115) to calculate c1and (2.116) to calculate µ1andV1giving
µ1=µ0+K1(x1−Cµ0) (13.94)
V1=(I−K1C)V0 (13.95)
c1=N(x1|Cµ0,CV0CT+Σ) (13.96)
where
K1=V0CT(
CV0CT+Σ)−1. (13.97)

where
K1=V0CT(
Similarly, the likelihood function for the linear dynamical system is given by (13.63)
in which the factors cnare found using the Kalman ﬁltering equations.
We can interpret the steps involved in going from the posterior marginal over
zn−1to the posterior marginal over znas follows. In (13.89), we can view the
quantity Aµn−1as the prediction of the mean over znobtained by simply taking the
mean over zn−1and projecting it forward one step using the transition probability

where
K1=V0CT(
matrix A. This predicted mean would give a predicted observation for xngiven by
CAz n−1obtained by applying the emission probability matrix Cto the predicted
hidden state mean. We can view the update equation (13.89) for the mean of the
hidden variable distribution as taking the predicted mean Aµn−1and then adding
a correction that is proportional to the error xn−CAz n−1between the predicted
observation and the actual observation. The coefﬁcient of this correction is given by

the Kalman gain matrix. Thus we can view the Kalman ﬁlter as a process of makingsuccessive predictions and then correcting these predictions in the light of the new
observations. This is illustrated graphically in Figure 13.21.

640 13. SEQUENTIAL DATA
zn−1 zn zn
Figure 13.21 The linear dynamical system can be viewed as a sequence of steps in which increasing un-
certainty in the state variable due to diffusion is compensated by the arrival of new data. In the left-hand plot,
the blue curve shows the distribution p(zn−1|x1,...,xn−1), which incorporates all the data up to step n−1.
The diffusion arising from the nonzero variance of the transition probability p(zn|zn−1)gives the distribution
p(zn|x1,...,xn−1), shown in red in the centre plot. Note that this is broader and shifted relative to the blue curve
(which is shown dashed in the centre plot for comparison). The next data observation xncontributes through the
emission density p(xn|zn), which is shown as a function of znin green on the right-hand plot. Note that this is not
a density with respect to znand so is not normalized to one. Inclusion of this new data point leads to a revised

distribution p(zn|x1,...,xn)for the state density shown in blue. We see that observation of the data has shifted
and narrowed the distribution compared to p(zn|x1,...,xn−1)(which is shown in dashed in the right-hand plot
for comparison).
If we consider a situation in which the measurement noise is small compared
to the rate at which the latent variable is evolving, then we ﬁnd that the posterior
distribution for zndepends only on the current measurement xn, in accordance with Exercise 13.27

the intuition from our simple example at the start of the section. Similarly, if the
latent variable is evolving slowly relative to the observation noise level, we ﬁnd that
the posterior mean for znis obtained by averaging all of the measurements obtained
up to that time. Exercise 13.28
One of the most important applications of the Kalman ﬁlter is to tracking, and
this is illustrated using a simple example of an object moving in two dimensions in
Figure 13.22.

So far, we have solved the inference problem of ﬁnding the posterior marginal
for a node zngiven observations from x1up toxn. Next we turn to the problem of
ﬁnding the marginal for a node zngiven all observations x1toxN. For temporal
data, this corresponds to the inclusion of future as well as past observations. Al-
though this cannot be used for real-time prediction, it plays a key role in learning the
parameters of the model. By analogy with the hidden Markov model, this problem

can be solved by propagating messages from node xNback to node x1and com-
bining this information with that obtained during the forward message passing stage
used to compute the ˆα(zn).
In the LDS literature, it is usual to formulate this backward recursion in terms
ofγ(zn)=ˆα(zn)ˆβ(zn)rather than in terms of ˆβ(zn). Because γ(zn)must also be
Gaussian, we write it in the form
γ(zn)=ˆα(zn)ˆβ(zn)=N(zn|ˆµn,ˆVn). (13.98)
To derive the required recursion, we start from the backward recursion (13.62) for

13.3. Linear Dynamical Systems 641
Figure 13.22 An illustration of a linear dy-
namical system being used to
track a moving object. The blue
points indicate the true positionsof the object in a two-dimensionalspace at successive time steps,the green points denote noisymeasurements of the positions,and the red crosses indicate themeans of the inferred posteriordistributions of the positions ob-tained by running the Kalman ﬁl-tering equations. The covari-ances of the inferred positionsare indicated by the red ellipses,which correspond to contourshaving one standard deviation.
ˆβ(zn), which, for continuous latent variables, can be written in the form
cn+1ˆβ(zn)=∫
ˆβ(zn+1)p(xn+1|zn+1)p(zn+1|zn)dzn+1. (13.99)
We now multiply both sides of (13.99) by ˆα(zn)and substitute for p(xn+1|zn+1)
andp(zn+1|zn)using (13.75) and (13.76). Then we make use of (13.89), (13.90)
and (13.91), together with (13.98), and after some manipulation we obtain Exercise 13.29
ˆµn=µn+Jn(ˆµn+1−AµN)
(13.100)
ˆVn=Vn+Jn(
ˆVn+1−Pn)
JT
n (13.101)

(13.100)
ˆVn+1−Pn)
JT
n (13.101)
Jn=VnAT(Pn)−1(13.102)
and we have made use of AVn=PnJT
n. Note that these recursions require that the
forward pass be completed ﬁrst so that the quantities µnandVnwill be available
for the backward pass.
For the EM algorithm, we also require the pairwise posterior marginals, which
can be obtained from (13.65) in the form
ξ(zn−1,zn)=(cn)−1ˆα(zn−1)p(xn|zn)p(zn|z−1)ˆβ(zn)
=N(zn−1|µn−1,Vn−1)N(zn|Azn−1,Γ)N(xn|Czn,Σ)N(zn|ˆµn,ˆVn)
cnˆα(zn).
(13.103)
Substituting for ˆα(zn)using (13.84) and rearranging, we see that ξ(zn−1,zn)is a

cnˆα(zn).
(13.103)
Gaussian with mean given with components γ(zn−1)andγ(zn), and a covariance
between znandzn−1given by Exercise 13.31
cov[zn,zn−1]=Jn−1ˆVn. (13.104)

642 13. SEQUENTIAL DATA
13.3.2 Learning in LDS
So far, we have considered the inference problem for linear dynamical systems,
assuming that the model parameters θ={A,Γ,C,Σ,µ0,V0}are known. Next, we
consider the determination of these parameters using maximum likelihood (Ghahra-
mani and Hinton, 1996b). Because the model has latent variables, this can be ad-dressed using the EM algorithm, which was discussed in general terms in Chapter 9.
We can derive the EM algorithm for the linear dynamical system as follows. Let
us denote the estimated parameter values at some particular cycle of the algorithm
byθ
old. For these parameter values, we can run the inference algorithm to determine
the posterior distribution of the latent variables p(Z|X,θold), or more precisely those
local posterior marginals that are required in the M step. In particular, we shall
require the following expectations
E[zn]=ˆµn (13.105)
E[
znzT
n−1]
=Jn−1ˆVn+ˆµnˆµT
n−1 (13.106)
E[
znzT
n]
=ˆVn+ˆµnˆµT
n (13.107)
where we have used (13.104).

byθ
E[
znzT
n−1]
E[
znzT
n]
n (13.107)
Now we consider the complete-data log likelihood function, which is obtained
by taking the logarithm of (13.6) and is therefore given by
lnp(X,Z|θ)=l n p(z1|µ0,V0)+N∑
n=2lnp(zn|zn−1,A,Γ)
+N∑
n=1lnp(xn|zn,C,Σ) (13.108)
in which we have made the dependence on the parameters explicit. We now take the
expectation of the complete-data log likelihood with respect to the posterior distri-
bution p(Z|X,θold)which deﬁnes the function
Q(θ,θold)= EZ|θold[lnp(X,Z|θ)]. (13.109)

n (13.107)
+N∑
In the M step, this function is maximized with respect to the components of θ.
Consider ﬁrst the parameters µ0andV0. If we substitute for p(z1|µ0,V0)in
(13.108) using (13.77), and then take the expectation with respect to Z, we obtain
Q(θ,θold)=−1
2ln|V0|− EZ|θold[1
2(z1−µ0)TV−1
0(z1−µ0)]
+c o n s t
where all terms not dependent on µ0orV0have been absorbed into the additive
constant. Maximization with respect to µ0andV0is easily performed by making

0(z1−µ0)]
+c o n s t
use of the maximum likelihood solution for a Gaussian distribution discussed in
Section 2.3.4, giving Exercise 13.32

13.3. Linear Dynamical Systems 643
µnew
0= E[z1] (13.110)
Vnew
0= E[z1zT
1]−E[z1]E[zT
1]. (13.111)
Similarly, to optimize AandΓ, we substitute for p(zn|zn−1,A,Γ)in (13.108)
using (13.75) giving
Q(θ,θold)=−N−1
2ln|Γ|
−EZ|θold[
1
2N∑
n=2(zn−Azn−1)TΓ−1(zn−Azn−1)]
+c o n s t (13.112)
in which the constant comprises terms that are independent of AandΓ. Maximizing
with respect to these parameters then gives Exercise 13.33
Anew=(N∑
n=2E[
znzT
n−1])(N∑
n=2E[
zn−1zT
n−1])−1
(13.113)
Γnew=1
N−1N∑
n=2{
E[
znzT
n]
−AnewE[
zn−1zT
n]
−E[
znzT
n−1]
Anew+AnewE[
zn−1zT
n−1]
(Anew)T}
.(13.114)
Note that Anewmust be evaluated ﬁrst, and the result can then be used to determine
Γnew.
Finally, in order to determine the new values of CandΣ, we substitute for
p(xn|zn,C,Σ)in (13.108) using (13.76) giving
Q(θ,θold)= −N
2ln|Σ|
−EZ|θold[
1
2N∑
n=1(xn−Czn)TΣ−1(xn−Czn)]
+c o n s t .
Maximizing with respect to CandΣthen gives Exercise 13.34
Cnew=(N∑
n=1xnE[
zT
n])(N∑
n=1E[
znzT
n])−1
(13.115)
Σnew=1
NN∑
n=1{
xnxT
n−CnewE[zn]xT
n
−xnE[
zT

zn−1zT
n]
−E[
znzT
n−1]
zn−1zT
n−1]
(Anew)T}
.(13.114)
Γnew.
2ln|Σ|
−EZ|θold[
1
2N∑
+c o n s t .
Cnew=(N∑
n=1xnE[
zT
n])(N∑
n=1E[
znzT
n])−1
(13.115)
Σnew=1
NN∑
n=1{
xnxT
n
−xnE[
zT
n]
Cnew+CnewE[
znzT
n]
Cnew}
. (13.116)

644 13. SEQUENTIAL DATA
We have approached parameter learning in the linear dynamical system using
maximum likelihood. Inclusion of priors to give a MAP estimate is straightforward,and a fully Bayesian treatment can be found by applying the analytical approxima-
tion techniques discussed in Chapter 10, though a detailed treatment is precluded
here due to lack of space.
13.3.3 Extensions of LDS
As with the hidden Markov model, there is considerable interest in extending
the basic linear dynamical system in order to increase its capabilities. Although the
assumption of a linear-Gaussian model leads to efﬁcient algorithms for inference
and learning, it also implies that the marginal distribution of the observed variablesis simply a Gaussian, which represents a signiﬁcant limitation. One simple extension
of the linear dynamical system is to use a Gaussian mixture as the initial distribution
forz
1. If this mixture has Kcomponents, then the forward recursion equations

forz
(13.85) will lead to a mixture of KGaussians over each hidden variable zn, and so
the model is again tractable.
For many applications, the Gaussian emission density is a poor approximation.
If instead we try to use a mixture of KGaussians as the emission density, then the
posteriorˆα(z1)will also be a mixture of KGaussians. However, from (13.85) the
posteriorˆα(z2)will comprise a mixture of K2Gaussians, and so on, with ˆα(zn)
being given by a mixture of KnGaussians. Thus the number of components grows

exponentially with the length of the chain, and so this model is impractical.
More generally, introducing transition or emission models that depart from the
linear-Gaussian (or other exponential family) model leads to an intractable infer-
ence problem. We can make deterministic approximations such as assumed den-sity ﬁltering or expectation propagation, or we can make use of sampling methods, Chapter 10
as discussed in Section 13.3.4. One widely used approach is to make a Gaussian

approximation by linearizing around the mean of the predicted distribution, whichgives rise to the extended Kalman ﬁlter (Zarchan and Musoff, 2005).
As with hidden Markov models, we can develop interesting extensions of the ba-
sic linear dynamical system by expanding its graphical representation. For example,theswitching state space model (Ghahramani and Hinton, 1998) can be viewed as
a combination of the hidden Markov model with a set of linear dynamical systems.

The model has multiple Markov chains of continuous linear-Gaussian latent vari-ables, each of which is analogous to the latent chain of the linear dynamical system
discussed earlier, together with a Markov chain of discrete variables of the form used
in a hidden Markov model. The output at each time step is determined by stochas-tically choosing one of the continuous latent chains, using the state of the discrete
latent variable as a switch, and then emitting an observation from the corresponding

conditional output distribution. Exact inference in this model is intractable, but vari-
ational methods lead to an efﬁcient inference scheme involving forward-backward
recursions along each of the continuous and discrete Markov chains independently.Note that, if we consider multiple chains of discrete latent variables, and use one as
the switch to select from the remainder, we obtain an analogous model having only
discrete latent variables known as the switching hidden Markov model .

13.3. Linear Dynamical Systems 645
13.3.4 Particle ﬁlters
For dynamical systems which do not have a linear-Gaussian, for example, if
they use a non-Gaussian emission density, we can turn to sampling methods in order Chapter 11
to ﬁnd a tractable inference algorithm. In particular, we can apply the sampling-
importance-resampling formalism of Section 11.1.5 to obtain a sequential MonteCarlo algorithm known as the particle ﬁlter.
Consider the class of distributions represented by the graphical model in Fig-
ure 13.5, and suppose we are given the observed values X
n=(x1,...,xn)and
we wish to draw Lsamples from the posterior distribution p(zn|Xn). Using Bayes’
E[f(zn)] =∫
f(zn)p(zn|Xn)dzn
=∫
f(zn)p(zn|xn,Xn−1)dzn
=∫
f(zn)p(xn|zn)p(zn|Xn−1)dzn
∫
p(xn|zn)p(zn|Xn−1)dzn
≃L∑
l=1w(l)
nf(z(l)
n) (13.117)
where {z(l)
n}is a set of samples drawn from p(zn|Xn−1)and we have made use of
the conditional independence property p(xn|zn,Xn−1)=p(xn|zn), which follows

=∫
=∫
∫
≃L∑
l=1w(l)
nf(z(l)
from the graph in Figure 13.5. The sampling weights {w(l)
n}are deﬁned by
w(l)
n=p(xn|z(l)
n)∑L
m=1p(xn|z(m)
n)(13.118)
where the same samples are used in the numerator as in the denominator. Thus the
posterior distribution p(zn|xn)is represented by the set of samples {z(l)
n}together
with the corresponding weights {w(l)
n}. Note that these weights satisfy 0⩽w(l)
n1
and∑
lw(l)
n=1.
Because we wish to ﬁnd a sequential sampling scheme, we shall suppose that
a set of samples and weights have been obtained at time step n, and that we have

w(l)
n)∑L
n1
and∑
lw(l)
n=1.
subsequently observed the value of xn+1, and we wish to ﬁnd the weights and sam-
ples at time step n+1. We ﬁrst sample from the distribution p(zn+1|Xn). This is

646 13. SEQUENTIAL DATA
straightforward since, again using Bayes’ theorem
p(zn+1|Xn)=∫
p(zn+1|zn,Xn)p(zn|Xn)dzn
=∫
p(zn+1|zn)p(zn|Xn)dzn
=∫
p(zn+1|zn)p(zn|xn,Xn−1)dzn
=∫
p(zn+1|zn)p(xn|zn)p(zn|Xn−1)dzn
∫
=∑
lw(l)
np(zn+1|z(l)
n) (13.119)
where we have made use of the conditional independence properties
p(zn+1|zn,Xn)= p(zn+1|zn) (13.120)
p(xn|zn,Xn−1)= p(xn|zn) (13.121)
which follow from the application of the d-separation criterion to the graph in Fig-
ure 13.5. The distribution given by (13.119) is a mixture distribution, and samplescan be drawn by choosing a component lwith probability given by the mixing coef-
ﬁcients w
(l)and then drawing a sample from the corresponding component.
In summary, we can view each step of the particle ﬁlter algorithm as comprising
two stages. At time step n, we have a sample representation of the posterior dis-
tribution p(zn|Xn)expressed as samples {z(l)
n}with corresponding weights {w(l)
n}.

ﬁcients w
n}.
This can be viewed as a mixture representation of the form (13.119). To obtain the
corresponding representation for the next time step, we ﬁrst draw Lsamples from
the mixture distribution (13.119), and then for each sample we use the new obser-
vationxn+1to evaluate the corresponding weights w(l)
n+1∝p(xn+1|z(l)
n+1). This is
illustrated, for the case of a single variable z, in Figure 13.23.
The particle ﬁltering, or sequential Monte Carlo, approach has appeared in the

n}.
literature under various names including the bootstrap ﬁlter (Gordon et al. , 1993),
survival of the ﬁttest (Kanazawa et al. , 1995), and the condensation algorithm (Isard
and Blake, 1998).
Exercises
13.1 (⋆)www Use the technique of d-separation, discussed in Section 8.2, to verify
that the Markov model shown in Figure 13.3 having Nnodes in total satisﬁes the
conditional independence properties (13.3) for n=2,...,N . Similarly, show that
a model described by the graph in Figure 13.4 in which there are Nnodes in total

Exercises 647
p(zn|Xn)
p(zn+1|Xn)
p(xn+1|zn+1)
p(zn+1|Xn+1)z
Figure 13.23 Schematic illustration of the operation of the particle ﬁlter for a one-dimensional latent
space. At time step n, the posterior p(zn|xn)is represented as a mixture distribution,
shown schematically as circles whose sizes are proportional to the weights w(l)
n.As e to f
Lsamples is then drawn from this distribution and the new weights w(l)
n+1evaluated using
p(xn+1|z(l)
n+1).
satisﬁes the conditional independence properties
p(xn|x1,...,xn−1)=p(xn|xn−1,xn−2) (13.122)
forn=3,...,N .
13.2 (⋆⋆)Consider the joint probability distribution (13.2) corresponding to the directed
graph of Figure 13.3. Using the sum and product rules of probability, verify that
this joint distribution satisﬁes the conditional independence property (13.3) for n=
2,...,N . Similarly, show that the second-order Markov model described by the
joint distribution (13.4) satisﬁes the conditional independence property
p(xn|x1,...,xn−1)=p(xn|xn−1,xn−2) (13.123)

13.3 (⋆)By using d-separation, show that the distribution p(x1,...,xN)of the observed
data for the state space model represented by the directed graph in Figure 13.5 does
not satisfy any conditional independence properties and hence does not exhibit the
Markov property at any ﬁnite order.
13.4 (⋆⋆)www Consider a hidden Markov model in which the emission densities are
represented by a parametric model p(x|z,w), such as a linear regression model or
a neural network, in which wis a vector of adaptive parameters. Describe how the

parameters wcan be learned from data using maximum likelihood.

648 13. SEQUENTIAL DATA
13.5 (⋆⋆)Verify the M-step equations (13.18) and (13.19) for the initial state probabili-
ties and transition probability parameters of the hidden Markov model by maximiza-tion of the expected complete-data log likelihood function (13.17), using appropriate
Lagrange multipliers to enforce the summation constraints on the components of π
andA.
13.6 (⋆)Show that if any elements of the parameters πorAfor a hidden Markov
model are initially set to zero, then those elements will remain zero in all subsequent
updates of the EM algorithm.
13.7 (⋆)Consider a hidden Markov model with Gaussian emission densities. Show that
maximization of the function Q(θ,θ
old)with respect to the mean and covariance
parameters of the Gaussians gives rise to the M-step equations (13.20) and (13.21).
13.8 (⋆⋆)www For a hidden Markov model having discrete observations governed by
a multinomial distribution, show that the conditional distribution of the observations

given the hidden variables is given by (13.22) and the corresponding M step equa-tions are given by (13.23). Write down the analogous equations for the conditional
distribution and the M step equations for the case of a hidden Markov with multiple
binary output variables each of which is governed by a Bernoulli conditional dis-tribution. Hint: refer to Sections 2.1 and 2.2 for a discussion of the corresponding
maximum likelihood solutions for i.i.d. data if required.
13.9 (⋆⋆)
www Use the d-separation criterion to verify that the conditional indepen-

13.9 (⋆⋆)
dence properties (13.24)–(13.31) are satisﬁed by the joint distribution for the hiddenMarkov model deﬁned by (13.6).
13.10 (⋆⋆⋆ )By applying the sum and product rules of probability, verify that the condi-
tional independence properties (13.24)–(13.31) are satisﬁed by the joint distribution
for the hidden Markov model deﬁned by (13.6).
13.11 (⋆⋆)Starting from the expression (8.72) for the marginal distribution over the vari-
ables of a factor in a factor graph, together with the results for the messages in the
sum-product algorithm obtained in Section 13.2.3, derive the result (13.43) for the

joint posterior distribution over two successive latent variables in a hidden Markovmodel.
13.12 (⋆⋆)Suppose we wish to train a hidden Markov model by maximum likelihood
using data that comprises Rindependent sequences of observations, which we de-
note by X
(r)where r=1,...,R . Show that in the E step of the EM algorithm,
we simply evaluate posterior probabilities for the latent variables by running the α
andβrecursions independently for each of the sequences. Also show that in the

note by X
M step, the initial probability and transition probability parameters are re-estimated

Exercises 649
using modiﬁed forms of (13.18 ) and (13.19) given by
πk=R∑
r=1γ(z(r)
1k)
R∑
r=1K∑
j=1γ(z(r)
1j)(13.124)
Ajk=R∑
r=1N∑
n=2ξ(z(r)
n−1,j,z(r)
n,k)
R∑
r=1K∑
l=1N∑
n=2ξ(z(r)
n,l)(13.125)
where, for notational convenience, we have assumed that the sequences are of the
same length (the generalization to sequences of different lengths is straightforward).
Similarly, show that the M-step equation for re-estimation of the means of Gaussianemission models is given by
µ
k=R∑
r=1N∑
n=1γ(z(r)
nk)x(r)
n
R∑
r=1N∑
n=1γ(z(r)
nk). (13.126)
Note that the M-step equations for other emission model parameters and distributions
take an analogous form.
13.13 (⋆⋆)www Use the deﬁnition (8.64) of the messages passed from a factor node
to a variable node in a factor graph, together with the expression (13.6) for the jointdistribution in a hidden Markov model, to show that the deﬁnition (13.50) of the
alpha message is the same as the deﬁnition (13.34).

µ
k=R∑
r=1N∑
n=1γ(z(r)
nk)x(r)
n
R∑
r=1N∑
n=1γ(z(r)
13.14 (⋆⋆)Use the deﬁnition (8.67) of the messages passed from a factor node to a
variable node in a factor graph, together with the expression (13.6) for the joint
distribution in a hidden Markov model, to show that the deﬁnition (13.52) of the
beta message is the same as the deﬁnition (13.35).
13.15 (⋆⋆)Use the expressions (13.33) and (13.43) for the marginals in a hidden Markov
model to derive the corresponding results (13.64) and (13.65) expressed in terms of
re-scaled variables.

13.16 (⋆⋆⋆ )In this exercise, we derive the forward message passing equation for the
Viterbi algorithm directly from the expression (13.6) for the joint distribution. Thisinvolves maximizing over all of the hidden variables z
1,...,zN. By taking the log-
arithm and then exchanging maximizations and summations, derive the recursion

650 13. SEQUENTIAL DATA
(13.68) where the quantities ω(zn)are deﬁned by (13.70). Show that the initial
condition for this recursion is given by (13.69).
13.17 (⋆)www Show that the directed graph for the input-output hidden Markov model,
given in Figure 13.18, can be expressed as a tree-structured factor graph of the form
shown in Figure 13.15 and write down expressions for the initial factor h(z1)and
for the general factor fn(zn−1,zn)where 2⩽n⩽N.
13.18 (⋆⋆⋆ )Using the result of Exercise 13.17, derive the recursion equations, includ-
ing the initial conditions, for the forward-backward algorithm for the input-outputhidden Markov model shown in Figure 13.18.
13.19 (⋆)
www The Kalman ﬁlter and smoother equations allow the posterior distribu-
tions over individual latent variables, conditioned on all of the observed variables,
to be found efﬁciently for linear dynamical systems. Show that the sequence oflatent variable values obtained by maximizing each of these posterior distributions

13.19 (⋆)
individually is the same as the most probable sequence of latent values. To do this,
simply note that the joint distribution of all latent and observed variables in a lineardynamical system is Gaussian, and hence all conditionals and marginals will also be
Gaussian, and then make use of the result (2.98).
13.20 (⋆⋆)
www Use the result (2.115) to prove (13.87).
13.21 (⋆⋆)Use the results (2.115) and (2.116), together with the matrix identities (C.5)
and (C.7), to derive the results (13.89), (13.90), and (13.91), where the Kalman gain

13.20 (⋆⋆)
matrixKnis deﬁned by (13.92).
13.22 (⋆⋆)www Using (13.93), together with the deﬁnitions (13.76) and (13.77) and
the result (2.115), derive (13.96).
13.23 (⋆⋆)Using (13.93), together with the deﬁnitions (13.76) and (13.77) and the result
(2.116), derive (13.94), (13.95) and (13.97).
13.24 (⋆⋆)www Consider a generalization of (13.75) and (13.76) in which we include
constant terms aandcin the Gaussian means, so that
p(zn|zn−1)=N(zn|Azn−1+a,Γ) (13.127)
p(xn|zn)=N(xn|Czn+c,Σ). (13.128)

Show that this extension can be re-case in the framework discussed in this chapter by
deﬁning a state vector zwith an additional component ﬁxed at unity, and then aug-
menting the matrices AandCusing extra columns corresponding to the parameters
aandc.
13.25 (⋆⋆)In this exercise, we show that when the Kalman ﬁlter equations are applied
to independent observations, they reduce to the results given in Section 2.3 for the

aandc.
maximum likelihood solution for a single Gaussian distribution. Consider the prob-lem of ﬁnding the mean µof a single Gaussian random variable x, in which we are
given a set of independent observations {x
1,...,x N}. To model this we can use

Exercises 651
a linear dynamical system governed by (13.75) and (13.76), with latent variables
{z1,...,z N}in which Cbecomes the identity matrix and where the transition prob-
ability A=0because the observations are independent. Let the parameters m0
andV0of the initial state be denoted by µ0andσ2
0, respectively, and suppose that
Σbecomes σ2. Write down the corresponding Kalman ﬁlter equations starting from
the general results (13.89) and (13.90), together with (13.94) and (13.95). Show that
these are equivalent to the results (2.141) and (2.142) obtained directly by consider-
ing independent data.
13.26 (⋆⋆⋆ )Consider a special case of the linear dynamical system of Section 13.3 that is
equivalent to probabilistic PCA, so that the transition matrix A=0, the covariance
Γ=I, and the noise covariance Σ=σ2I. By making use of the matrix inversion
identity (C.7) show that, if the emission density matrix Cis denoted W, then the
posterior distribution over the hidden states deﬁned by (13.89) and (13.90) reduces

to the result (12.42) for probabilistic PCA.
13.27 (⋆)www Consider a linear dynamical system of the form discussed in Sec-
tion 13.3 in which the amplitude of the observation noise goes to zero, so that Σ=0.
Show that the posterior distribution for znhas mean xnand zero variance. This
accords with our intuition that if there is no noise, we should just use the current
observation xnto estimate the state variable znand ignore all previous observations.
13.28 (⋆⋆⋆ )Consider a special case of the linear dynamical system of Section 13.3 in

which the state variable znis constrained to be equal to the previous state variable,
which corresponds to A=IandΓ=0. For simplicity, assume also that V0→∞
so that the initial conditions for zare unimportant, and the predictions are determined
purely by the data. Use proof by induction to show that the posterior mean for state
znis determined by the average of x1,...,xn. This corresponds to the intuitive
result that if the state variable is constant, our best estimate is obtained by averaging
the observations.

13.29 (⋆⋆⋆ )Starting from the backwards recursion equation (13.99), derive the RTS
smoothing equations (13.100) and (13.101) for the Gaussian linear dynamical sys-
tem.
13.30 (⋆⋆)Starting from the result (13.65) for the pairwise posterior marginal in a state
space model, derive the speciﬁc form (13.103) for the case of the Gaussian linear
dynamical system.
13.31 (⋆⋆)Starting from the result (13.103) and by substituting for ˆα(zn)using (13.84),
verify the result (13.104) for the covariance between znandzn−1.

tem.
13.32 (⋆⋆)www Verify the results (13.110) and (13.111) for the M-step equations for
µ0andV0in the linear dynamical system.
13.33 (⋆⋆)Verify the results (13.113) and (13.114) for the M-step equations for AandΓ
in the linear dynamical system.

652 13. SEQUENTIAL DATA
13.34 (⋆⋆)Verify the results (13.115) and (13.116) for the M-step equations for CandΣ

14
Combining
Models
In earlier chapters, we have explored a range of different models for solving classiﬁ-
cation and regression problems. It is often found that improved performance can beobtained by combining multiple models together in some way, instead of just using
a single model in isolation. For instance, we might train Ldifferent models and then
make predictions using the average of the predictions made by each model. Suchcombinations of models are sometimes called committees . In Section 14.2, we dis-
cuss ways to apply the committee concept in practice, and we also give some insight
into why it can sometimes be an effective procedure.
One important variant of the committee method, known as boosting , involves
training multiple models in sequence in which the error function used to train a par-
ticular model depends on the performance of the previous models. This can producesubstantial improvements in performance compared to the use of a single model and
is discussed in Section 14.3.

Instead of averaging the predictions of a set of models, an alternative form of
653

654 14. COMBINING MODELS
model combination is to select one of the models to make the prediction, in which
the choice of model is a function of the input variables. Thus different models be-come responsible for making predictions in different regions of input space. One
widely used framework of this kind is known as a decision tree in which the selec-
tion process can be described as a sequence of binary selections corresponding tothe traversal of a tree structure and is discussed in Section 14.4. In this case, the
individual models are generally chosen to be very simple, and the overall ﬂexibility
of the model arises from the input-dependent selection process. Decision trees canbe applied to both classiﬁcation and regression problems.
One limitation of decision trees is that the division of input space is based on
hard splits in which only one model is responsible for making predictions for any
given value of the input variables. The decision process can be softened by moving

to a probabilistic framework for combining models, as discussed in Section 14.5. Forexample, if we have a set of Kmodels for a conditional distribution p(t|x,k)where
xis the input variable, tis the target variable, and k=1,...,K indexes the model,
then we can form a probabilistic mixture of the form
p(t|x)=
K∑
k=1πk(x)p(t|x,k) (14.1)
in which πk(x)=p(k|x)represent the input-dependent mixing coefﬁcients. Such
models can be viewed as mixture distributions in which the component densities, as

p(t|x)=
K∑
well as the mixing coefﬁcients, are conditioned on the input variables and are known
asmixtures of experts . They are closely related to the mixture density network model
discussed in Section 5.6.
14.1. Bayesian Model Averaging
It is important to distinguish between model combination methods and Bayesian
model averaging, as the two are often confused. To understand the difference, con-
sider the example of density estimation using a mixture of Gaussians in which several Section 9.2

Gaussian components are combined probabilistically. The model contains a binary
latent variable zthat indicates which component of the mixture is responsible for
generating the corresponding data point. Thus the model is speciﬁed in terms of a
p(x,z) (14.2)
and the corresponding density over the observed variable xis obtained by marginal-
izing over the latent variable
p(x)=∑
zp(x,z). (14.3)

14.2. Committees 655
In the case of our Gaussian mixture example, this leads to a distribution of the form
p(x)=K∑
k=1πkN(x|µk,Σk) (14.4)
with the usual interpretation of the symbols. This is an example of model combi-
nation. For independent, identically distributed data, we can use (14.3) to write themarginal probability of a data set X={x
1,...,xN}in the form
p(X)=N∏
n=1p(xn)=N∏
n=1[∑
znp(xn,zn)]
. (14.5)
Thus we see that each observed data point xnhas a corresponding latent variable zn.
Now suppose we have several different models indexed by h=1,...,H with
prior probabilities p(h). For instance one model might be a mixture of Gaussians and
another model might be a mixture of Cauchy distributions. The marginal distributionover the data set is given by
p(X)=H∑
h=1p(X|h)p(h). (14.6)
This is an example of Bayesian model averaging. The interpretation of this summa-
tion over his that just one model is responsible for generating the whole data set,

p(X)=H∑
and the probability distribution over hsimply reﬂects our uncertainty as to which
model that is. As the size of the data set increases, this uncertainty reduces, and
the posterior probabilities p(h|X)become increasingly focussed on just one of the
models.
This highlights the key difference between Bayesian model averaging and model
combination, because in Bayesian model averaging the whole data set is generated

models.
by a single model. By contrast, when we combine multiple models, as in (14.5), wesee that different data points within the data set can potentially be generated from
different values of the latent variable zand hence by different components.
Although we have considered the marginal probability p(X), the same consid-
erations apply for the predictive density p(x|X)or for conditional distributions such
asp(t|x,X,T). Exercise 14.1
14.2. Committees

The simplest way to construct a committee is to average the predictions of a set of
individual models. Such a procedure can be motivated from a frequentist perspective
by considering the trade-off between bias and variance, which decomposes the er- Section 3.2
ror due to a model into the bias component that arises from differences between themodel and the true function to be predicted, and the variance component that repre-
sents the sensitivity of the model to the individual data points. Recall from Figure 3.5

656 14. COMBINING MODELS
that when we trained multiple polynomials using the sinusoidal data, and then aver-
aged the resulting functions, the contribution arising from the variance term tended tocancel, leading to improved predictions. When we averaged a set of low-bias mod-
els (corresponding to higher order polynomials), we obtained accurate predictions
for the underlying sinusoidal function from which the data were generated.
In practice, of course, we have only a single data set, and so we have to ﬁnd
a way to introduce variability between the different models within the committee.
One approach is to use bootstrap data sets, discussed in Section 1.2.3. Consider a
regression problem in which we are trying to predict the value of a single continuous
variable, and suppose we generate Mbootstrap data sets and then use each to train
a separate copy y
m(x)of a predictive model where m=1,...,M . The committee
prediction is given by
yCOM(x)=1
MM∑
m=1ym(x). (14.7)

yCOM(x)=1
MM∑
This procedure is known as bootstrap aggregation or bagging (Breiman, 1996).
Suppose the true regression function that we are trying to predict is given by
h(x), so that the output of each of the models can be written as the true value plus
an error in the form
ym(x)=h(x)+ϵm(x). (14.8)
The average sum-of-squares error then takes the form
Ex[
{ym(x)−h(x)}2]
=Ex[
ϵm(x)2]
(14.9)
where Ex[·]denotes a frequentist expectation with respect to the distribution of the
input vector x. The average error made by the models acting individually is therefore
EAV=1
MM∑

Ex[
=Ex[
ϵm(x)2]
(14.9)
EAV=1
MM∑
m=1Ex[
ϵm(x)2]
. (14.10)
Similarly, the expected error from the committee (14.7) is given by
ECOM = Ex⎡
⎣{
1
MM∑
m=1ym(x)−h(x)}2⎤
⎦
= Ex⎡
⎣{
1
MM∑
m=1ϵm(x)}2⎤
⎦ (14.11)
If we assume that the errors have zero mean and are uncorrelated, so that
Ex[ϵm(x) ]=0 (14.12)
Ex[ϵm(x)ϵl(x) ]=0 ,m ̸=l (14.13)

14.3. Boosting 657
then we obtain Exercise 14.2
ECOM=1
MEAV. (14.14)
This apparently dramatic result suggests that the average error of a model can be
reduced by a factor of Msimply by averaging Mversions of the model. Unfortu-
nately, it depends on the key assumption that the errors due to the individual models
are uncorrelated. In practice, the errors are typically highly correlated, and the reduc-tion in overall error is generally small. It can, however, be shown that the expected
committee error will not exceed the expected error of the constituent models, so that
E
COM⩽EAV. In order to achieve more signiﬁcant improvements, we turn to a more Exercise 14.3
sophisticated technique for building committees, known as boosting.
14.3. Boosting
Boosting is a powerful technique for combining multiple ‘base’ classiﬁers to producea form of committee whose performance can be signiﬁcantly better than that of anyof the base classiﬁers. Here we describe the most widely used form of boosting

E
algorithm called AdaBoost , short for ‘adaptive boosting’, developed by Freund and
Schapire (1996). Boosting can give good results even if the base classiﬁers have a
performance that is only slightly better than random, and hence sometimes the base
classiﬁers are known as weak learners . Originally designed for solving classiﬁcation
problems, boosting can also be extended to regression (Friedman, 2001).
The principal difference between boosting and the committee methods such as

bagging discussed above, is that the base classiﬁers are trained in sequence, andeach base classiﬁer is trained using a weighted form of the data set in which the
weighting coefﬁcient associated with each data point depends on the performance
of the previous classiﬁers. In particular, points that are misclassiﬁed by one of thebase classiﬁers are given greater weight when used to train the next classiﬁer in
the sequence. Once all the classiﬁers have been trained, their predictions are then

combined through a weighted majority voting scheme, as illustrated schematicallyin Figure 14.1.
Consider a two-class classiﬁcation problem, in which the training data comprises
input vectors x
1,...,xNalong with corresponding binary target variables t1,...,t N
where tn∈{ −1,1}. Each data point is given an associated weighting parameter
wn, which is initially set 1/Nfor all data points. We shall suppose that we have
a procedure available for training a base classiﬁer using weighted data to give a

function y(x)∈{ −1,1}. At each stage of the algorithm, AdaBoost trains a new
classiﬁer using a data set in which the weighting coefﬁcients are adjusted accordingto the performance of the previously trained classiﬁer so as to give greater weight
to the misclassiﬁed data points. Finally, when the desired number of base classiﬁers
have been trained, they are combined to form a committee using coefﬁcients thatgive different weight to different base classiﬁers. The precise form of the AdaBoost
algorithm is given below.

658 14. COMBINING MODELS
Figure 14.1 Schematic illustration of the
boosting framework. Each
base classiﬁer ym(x)is trained
on a weighted form of the train-
ing set (blue arrows) in which
the weights w(m)
ndepend on
the performance of the pre-
vious base classiﬁer ym−1(x)
(green arrows). Once all base
classiﬁers have been trained,
they are combined to give
the ﬁnal classiﬁer YM(x)(red
arrows).{w(1)
n}{ w(2)
n}{ w(M)
n}
y1(x) y2(x) yM(x)
YM(x)=s i g n(M∑
mαmym(x))
AdaBoost
1. Initialize the data weighting coefﬁcients {wn}by setting w(1)
n=1/Nfor
n=1,...,N .
2. For m=1,...,M :
(a) Fit a classiﬁer ym(x)to the training data by minimizing the weighted
error function
Jm=N∑
n=1w(m)
nI(ym(xn)̸=tn) (14.15)
where I(ym(xn)̸=tn)is the indicator function and equals 1when
ym(xn)̸=tnand0otherwise.
(b) Evaluate the quantities
ϵm=N∑
n=1w(m)
nI(ym(xn)̸=tn)
N∑
n=1w(m)
n(14.16)
and then use these to evaluate
αm=l n{1−ϵm
ϵm}
. (14.17)
(c) Update the data weighting coefﬁcients
w(m+1)
n =w(m)
nexp{αmI(ym(xn)̸=tn)} (14.18)

14.3. Boosting 659
3. Make predictions using the ﬁnal model, which is given by
YM(x) = sign(M∑
m=1αmym(x))
. (14.19)
We see that the ﬁrst base classiﬁer y1(x)is trained using weighting coefﬁ-
cients w(1)
nthat are all equal, which therefore corresponds to the usual procedure
for training a single classiﬁer. From (14.18), we see that in subsequent iterations
the weighting coefﬁcients w(m)
nare increased for data points that are misclassiﬁed
and decreased for data points that are correctly classiﬁed. Successive classiﬁers are
therefore forced to place greater emphasis on points that have been misclassiﬁed by
previous classiﬁers, and data points that continue to be misclassiﬁed by successive
classiﬁers receive ever greater weight. The quantities ϵmrepresent weighted mea-
sures of the error rates of each of the base classiﬁers on the data set. We therefore
see that the weighting coefﬁcients αmdeﬁned by (14.17) give greater weight to the
more accurate classiﬁers when computing the overall output given by (14.19).

The AdaBoost algorithm is illustrated in Figure 14.2, using a subset of 30data
points taken from the toy classiﬁcation data set shown in Figure A.7. Here each base
learners consists of a threshold on one of the input variables. This simple classiﬁercorresponds to a form of decision tree known as a ‘decision stumps’, i.e., a deci- Section 14.4
sion tree with a single node. Thus each base learner classiﬁes an input according to

whether one of the input features exceeds some threshold and therefore simply parti-tions the space into two regions separated by a linear decision surface that is parallel
to one of the axes.
14.3.1 Minimizing exponential error
Boosting was originally motivated using statistical learning theory, leading to
upper bounds on the generalization error. However, these bounds turn out to be too
loose to have practical value, and the actual performance of boosting is much better

than the bounds alone would suggest. Friedman et al. (2000) gave a different and
very simple interpretation of boosting in terms of the sequential minimization of an
exponential error function.
Consider the exponential error function deﬁned by
E=N∑
n=1exp{−tnfm(xn)} (14.20)
where fm(x)is a classiﬁer deﬁned in terms of a linear combination of base classiﬁers
yl(x)of the form
fm(x)=1
2m∑
l=1αlyl(x) (14.21)
andtn∈{ −1,1}are the training set target values. Our goal is to minimize Ewith

E=N∑
fm(x)=1
2m∑
respect to both the weighting coefﬁcients αland the parameters of the base classiﬁers
yl(x).

660 14. COMBINING MODELS
m=1
−1 0 1 2−202m=2
−1 0 1 2−202m=3
−1 0 1 2−202
m=6
−1 0 1 2−202m=1 0
−1 0 1 2−202m= 150
−1 0 1 2−202
Figure 14.2 Illustration of boosting in which the base learners consist of simple thresholds applied to one or
other of the axes. Each ﬁgure shows the number mof base learners trained so far, along with the decision
boundary of the most recent base learner (dashed black line) and the combined decision boundary of the en-
semble (solid green line). Each data point is depicted by a circle whose radius indicates the weight assigned to
that data point when training the most recently added base learner. Thus, for instance, we see that points that
are misclassiﬁed by the m=1base learner are given greater weight when training the m=2base learner.
Instead of doing a global error function minimization, however, we shall sup-
pose that the base classiﬁers y1(x),...,y m−1(x)are ﬁxed, as are their coefﬁcients
α1,...,α m−1, and so we are minimizing only with respect to αmandym(x). Sep-

arating off the contribution from base classiﬁer ym(x), we can then write the error
E=N∑
n=1exp{
−tnfm−1(xn)−1
2tnαmym(xn)}
=N∑
n=1w(m)
nexp{
−1
(14.22)
where the coefﬁcients w(m)
n=e x p {−tnfm−1(xn)}can be viewed as constants
because we are optimizing only αmandym(x). If we denote by Tmthe set of
data points that are correctly classiﬁed by ym(x), and if we denote the remaining
misclassiﬁed points by Mm, then we can in turn rewrite the error function in the

14.3. Boosting 661
form
E=e−αm/2∑
n∈Tmw(m)
n+eαm/2∑
n∈M mw(m)
n
=(eαm/2−e−αm/2)N∑
n=1w(m)
nI(ym(xn)̸=tn)+e−αm/2N∑
n=1w(m)
n.
(14.23)
When we minimize this with respect to ym(x), we see that the second term is con-
stant, and so this is equivalent to minimizing (14.15) because the overall multiplica-tive factor in front of the summation does not affect the location of the minimum.
Similarly, minimizing with respect to α
m, we obtain (14.17) in which ϵmis deﬁned
by (14.16). Exercise 14.6
From (14.22) we see that, having found αmandym(x), the weights on the data
points are updated using
w(m+1)
n =w(m)
nexp{
−1
. (14.24)
Making use of the fact that
tnym(xn)=1−2I(ym(xn)̸=tn) (14.25)
we see that the weights w(m)
nare updated at the next iteration using
w(m+1)
n =w(m)
nexp(−αm/2) exp {αmI(ym(xn)̸=tn)}. (14.26)
Because the term exp(−αm/2)is independent of n, we see that it weights all data
points by the same factor and so can be discarded. Thus we obtain (14.18).

w(m+1)
n =w(m)
nexp{
−1
. (14.24)
w(m+1)
n =w(m)
Finally, once all the base classiﬁers are trained, new data points are classiﬁed by
evaluating the sign of the combined function deﬁned according to (14.21). Becausethe factor of 1/2does not affect the sign it can be omitted, giving (14.19).
14.3.2 Error functions for boosting
The exponential error function that is minimized by the AdaBoost algorithm
differs from those considered in previous chapters. To gain some insight into the
nature of the exponential error function, we ﬁrst consider the expected error given
by

by
Ex,t[exp{−ty(x)}]=∑
t∫
exp{−ty(x)}p(t|x)p(x)dx. (14.27)
If we perform a variational minimization with respect to all possible functions y(x),
we obtain Exercise 14.7
y(x)=1
2ln{p(t=1|x)
p(t=−1|x)}
(14.28)

662 14. COMBINING MODELS
Figure 14.3 Plot of the exponential (green) and
rescaled cross-entropy (red) error
functions along with the hinge er-
ror (blue) used in support vector
machines, and the misclassiﬁcation
error (black). Note that for large
negative values of z=ty(x), the
cross-entropy gives a linearly in-
creasing penalty, whereas the expo-
nential loss gives an exponentially in-
creasing penalty.
which is half the log-odds. Thus the AdaBoost algorithm is seeking the best approx-
imation to the log odds ratio, within the space of functions represented by the linear
combination of base classiﬁers, subject to the constrained minimization resulting
from the sequential optimization strategy. This result motivates the use of the sign
function in (14.19) to arrive at the ﬁnal classiﬁcation decision.
We have already seen that the minimizer y(x)of the cross-entropy error (4.90)
for two-class classiﬁcation is given by the posterior class probability. In the case

of a target variable t∈{ −1,1}, we have seen that the error function is given by Section 7.1.2
ln(1 + exp( −yt)). This is compared with the exponential error function in Fig-
ure 14.3, where we have divided the cross-entropy error by a constant factor ln(2)
so that it passes through the point (0,1)for ease of comparison. We see that both
can be seen as continuous approximations to the ideal misclassiﬁcation error func-
tion. An advantage of the exponential error is that its sequential minimization leads

to the simple AdaBoost scheme. One drawback, however, is that it penalizes large
negative values of ty(x)much more strongly than cross-entropy. In particular, we
see that for large negative values of ty, the cross-entropy grows linearly with |ty|,
whereas the exponential error function grows exponentially with |ty|. Thus the ex-
ponential error function will be much less robust to outliers or misclassiﬁed data
points. Another important difference between cross-entropy and the exponential er-

ror function is that the latter cannot be interpreted as the log likelihood function of
any well-deﬁned probabilistic model. Furthermore, the exponential error does not Exercise 14.8
generalize to classiﬁcation problems having K>2classes, again in contrast to the
cross-entropy for a probabilistic model, which is easily generalized to give (4.108). Section 4.3.4
The interpretation of boosting as the sequential optimization of an additive model

under an exponential error (Friedman et al. , 2000) opens the door to a wide range
of boosting-like algorithms, including multiclass extensions, by altering the choice
of error function. It also motivates the extension to regression problems (Friedman,
2001). If we consider a sum-of-squares error function for regression, then sequential
minimization of an additive model of the form (14.21) simply involves ﬁtting each
new base classiﬁer to the residual errors tn−fm−1(xn)from the previous model. As Exercise 14.9

we have noted, however, the sum-of-squares error is not robust to outliers, and this

14.4. Tree-based Models 663
Figure 14.4 Comparison of the squared error
(green) with the absolute error (red)
showing how the latter places much
less emphasis on large errors and
hence is more robust to outliers and
mislabelled data points.
0 zE(z)
−11
can be addressed by basing the boosting algorithm on the absolute deviation |y−t|
instead. These two error functions are compared in Figure 14.4.
14.4. Tree-based Models
There are various simple, but widely used, models that work by partitioning the
input space into cuboid regions, whose edges are aligned with the axes, and then
assigning a simple model (for example, a constant) to each region. They can be
viewed as a model combination method in which only one model is responsible
for making predictions at any given point in input space. The process of selecting
a speciﬁc model, given a new input x, can be described by a sequential decision
making process corresponding to the traversal of a binary tree (one that splits into

two branches at each node). Here we focus on a particular tree-based framework
called classiﬁcation and regression trees ,o r CART (Breiman et al. , 1984), although
there are many other variants going by such names as ID3 and C4.5 (Quinlan, 1986;
Quinlan, 1993).
Figure 14.5 shows an illustration of a recursive binary partitioning of the input
space, along with the corresponding tree structure. In this example, the ﬁrst step
Figure 14.5 Illustration of a two-dimensional in-
put space that has been partitioned

into ﬁve regions using axis-aligned
boundaries.
AB
CDE
θ1 θ4θ2θ3
x1x2

664 14. COMBINING MODELS
Figure 14.6 Binary tree corresponding to the par-
titioning of input space shown in Fig-
ure 14.5.x1>θ1
x2>θ3
x1⩽θ4x2⩽θ2
ABCDE
divides the whole of the input space into two regions according to whether x1⩽θ1
orx1>θ1where θ1is a parameter of the model. This creates two subregions, each
of which can then be subdivided independently. For instance, the region x1⩽θ1
is further subdivided according to whether x2⩽θ2orx2>θ2, giving rise to the
regions denoted A and B. The recursive subdivision can be described by the traversal
of the binary tree shown in Figure 14.6. For any new input x, we determine which
region it falls into by starting at the top of the tree at the root node and following
a path down to a speciﬁc leaf node according to the decision criteria at each node.
Note that such decision trees are not probabilistic graphical models.
Within each region, there is a separate model to predict the target variable. For

instance, in regression we might simply predict a constant over each region, or in
classiﬁcation we might assign each region to a speciﬁc class. A key property of tree-
based models, which makes them popular in ﬁelds such as medical diagnosis, for
example, is that they are readily interpretable by humans because they correspond
to a sequence of binary decisions applied to the individual input variables. For in-
stance, to predict a patient’s disease, we might ﬁrst ask “is their temperature greater

than some threshold?”. If the answer is yes, then we might next ask “is their blood
pressure less than some threshold?”. Each leaf of the tree is then associated with a
speciﬁc diagnosis.
In order to learn such a model from a training set, we have to determine the
structure of the tree, including which input variable is chosen at each node to form
the split criterion as well as the value of the threshold parameter θifor the split. We
also have to determine the values of the predictive variable within each region.

Consider ﬁrst a regression problem in which the goal is to predict a single target
variable tfrom a D-dimensional vector x=(x1,...,x D)Tof input variables. The
training data consists of input vectors {x1,...,xN}along with the corresponding
continuous labels {t1,...,t N}. If the partitioning of the input space is given, and we
minimize the sum-of-squares error function, then the optimal value of the predictive
variable within any given region is just given by the average of the values of tnfor
those data points that fall in that region. Exercise 14.10

Now consider how to determine the structure of the decision tree. Even for a
ﬁxed number of nodes in the tree, the problem of determining the optimal structure
(including choice of input variable for each split as well as the corresponding thresh-

14.4. Tree-based Models 665
olds) to minimize the sum-of-squares error is usually computationally infeasible due
to the combinatorially large number of possible solutions. Instead, a greedy opti-mization is generally done by starting with a single root node, corresponding to the
whole input space, and then growing the tree by adding nodes one at a time. At each
step there will be some number of candidate regions in input space that can be split,corresponding to the addition of a pair of leaf nodes to the existing tree. For each
of these, there is a choice of which of the Dinput variables to split, as well as the
value of the threshold. The joint optimization of the choice of region to split, and thechoice of input variable and threshold, can be done efﬁciently by exhaustive search
noting that, for a given choice of split variable and threshold, the optimal choice of
predictive variable is given by the local average of the data, as noted earlier. This

is repeated for all possible choices of variable to be split, and the one that gives the
smallest residual sum-of-squares error is retained.
Given a greedy strategy for growing the tree, there remains the issue of when
to stop adding nodes. A simple approach would be to stop when the reduction in
residual error falls below some threshold. However, it is found empirically that oftennone of the available splits produces a signiﬁcant reduction in error, and yet after
several more splits a substantial error reduction is found. For this reason, it is com-

mon practice to grow a large tree, using a stopping criterion based on the numberof data points associated with the leaf nodes, and then prune back the resulting tree.
The pruning is based on a criterion that balances residual error against a measure of
model complexity. If we denote the starting tree for pruning by T
0, then we deﬁne
T⊂T0to be a subtree of T0if it can be obtained by pruning nodes from T0(in
other words, by collapsing internal nodes by combining the corresponding regions).

Suppose the leaf nodes are indexed by τ=1,...,|T|, with leaf node τrepresenting
a region Rτof input space having Nτdata points, and |T|denoting the total number
of leaf nodes. The optimal prediction for region Rτis then given by
yτ=1
Nτ∑
xn∈Rτtn (14.29)
and the corresponding contribution to the residual sum-of-squares is then
Qτ(T)=∑
xn∈Rτ{tn−yτ}2. (14.30)
The pruning criterion is then given by
C(T)=|T|∑
τ=1Qτ(T)+λ|T| (14.31)
The regularization parameter λdetermines the trade-off between the overall residual

yτ=1
Nτ∑
Qτ(T)=∑
C(T)=|T|∑
sum-of-squares error and the complexity of the model as measured by the number
|T|of leaf nodes, and its value is chosen by cross-validation.
For classiﬁcation problems, the process of growing and pruning the tree is sim-
ilar, except that the sum-of-squares error is replaced by a more appropriate measure

666 14. COMBINING MODELS
of performance. If we deﬁne pτkto be the proportion of data points in region Rτ
assigned to class k, where k=1,...,K , then two commonly used choices are the
cross-entropy
Qτ(T)=K∑
k=1pτklnpτk (14.32)
and the Gini index
Qτ(T)=K∑
k=1pτk(1−pτk). (14.33)
These both vanish for pτk=0andpτk=1and have a maximum at pτk=0.5. They
encourage the formation of regions in which a high proportion of the data points are
assigned to one class. The cross entropy and the Gini index are better measures than
the misclassiﬁcation rate for growing the tree because they are more sensitive to thenode probabilities. Also, unlike misclassiﬁcation rate, they are differentiable and Exercise 14.11
hence better suited to gradient based optimization methods. For subsequent pruning
of the tree, the misclassiﬁcation rate is generally used.
The human interpretability of a tree model such as CART is often seen as its
major strength. However, in practice it is found that the particular tree structure that

is learned is very sensitive to the details of the data set, so that a small change to thetraining data can result in a very different set of splits (Hastie et al. , 2001).
There are other problems with tree-based methods of the kind considered in
this section. One is that the splits are aligned with the axes of the feature space,
which may be very suboptimal. For instance, to separate two classes whose optimal

decision boundary runs at 45 degrees to the axes would need a large number ofaxis-parallel splits of the input space as compared to a single non-axis-aligned split.
Furthermore, the splits in a decision tree are hard, so that each region of input space
is associated with one, and only one, leaf node model. The last issue is particularlyproblematic in regression where we are typically aiming to model smooth functions,
and yet the tree model produces piecewise-constant predictions with discontinuities

at the split boundaries.
14.5. Conditional Mixture Models
We have seen that standard decision trees are restricted by hard, axis-aligned splits of
the input space. These constraints can be relaxed, at the expense of interpretability,
by allowing soft, probabilistic splits that can be functions of all of the input variables,
not just one of them at a time. If we also give the leaf models a probabilistic inter-
pretation, we arrive at a fully probabilistic tree-based model called the hierarchical

mixture of experts , which we consider in Section 14.5.3.
An alternative way to motivate the hierarchical mixture of experts model is to
start with a standard probabilistic mixtures of unconditional density models such asGaussians and replace the component densities with conditional distributions. Here Chapter 9
we consider mixtures of linear regression models (Section 14.5.1) and mixtures of

14.5. Conditional Mixture Models 667
logistic regression models (Section 14.5.2). In the simplest case, the mixing coefﬁ-
cients are independent of the input variables. If we make a further generalization toallow the mixing coefﬁcients also to depend on the inputs then we obtain a mixture
of experts model. Finally, if we allow each component in the mixture model to be
itself a mixture of experts model, then we obtain a hierarchical mixture of experts.
14.5.1 Mixtures of linear regression models
One of the many advantages of giving a probabilistic interpretation to the lin-
ear regression model is that it can then be used as a component in more complex
probabilistic models. This can be done, for instance, by viewing the conditional
distribution representing the linear regression model as a node in a directed prob-abilistic graph. Here we consider a simple example corresponding to a mixture of
linear regression models, which represents a straightforward extension of the Gaus-

sian mixture model discussed in Section 9.2 to the case of conditional Gaussiandistributions.
We therefore consider Klinear regression models, each governed by its own
weight parameter w
k. In many applications, it will be appropriate to use a common
noise variance, governed by a precision parameter β, for all Kcomponents, and this
is the case we consider here. We will once again restrict attention to a single targetvariable t, though the extension to multiple outputs is straightforward. If we denote Exercise 14.12

the mixing coefﬁcients by π
k, then the mixture distribution can be written
p(t|θ)=K∑
k=1πkN(t|wT
kφ,β−1) (14.34)
where θdenotes the set of all adaptive parameters in the model, namely W={wk},
π={πk}, andβ. The log likelihood function for this model, given a data set of
observations {φn,tn}, then takes the form
lnp(t|θ)=N∑
n=1ln(K∑
k=1πkN(tn|wT
kφn,β−1))
(14.35)
where t=(t1,...,t N)Tdenotes the vector of target variables.
In order to maximize this likelihood function, we can once again appeal to the
EM algorithm, which will turn out to be a simple extension of the EM algorithm for

p(t|θ)=K∑
n=1ln(K∑
kφn,β−1))
(14.35)
unconditional Gaussian mixtures of Section 9.2. We can therefore build on our expe-rience with the unconditional mixture and introduce a set Z={z
n}of binary latent
variables where znk∈{0,1}in which, for each data point n, all of the elements
k=1,...,K are zero except for a single value of 1indicating which component
of the mixture was responsible for generating that data point. The joint distribution
over latent and observed variables can be represented by the graphical model shown
in Figure 14.7.

The complete-data log likelihood function then takes the form Exercise 14.13
lnp(t,Z|θ)=N∑
n=1K∑
k=1znkln{
πkN(tn|wT
kφn,β−1)}
. (14.36)

668 14. COMBINING MODELS
Figure 14.7 Probabilistic directed graph representing a mixture of
linear regression models, deﬁned by (14.35).zn
tnφn
N Wβπ
The EM algorithm begins by ﬁrst choosing an initial value θoldfor the model param-
eters. In the E step, these parameter values are then used to evaluate the posterior
probabilities, or responsibilities, of each component kfor every data point ngiven
by
γnk=E[znk]=p(k|φn,θold)=πkN(tn|wT
kφn,β−1)∑
jπjN(tn|wT
jφn,β−1). (14.37)
The responsibilities are then used to determine the expectation, with respect to the
posterior distribution p(Z|t,θold), of the complete-data log likelihood, which takes
the form
Q(θ,θold)= EZ[lnp(t,Z|θ)] =N∑
n=1K∑
k=1γnk{
lnπk+l nN(tn|wT
kφn,β−1)}
.
In the M step, we maximize the function Q(θ,θold)with respect to θ, keeping the
γnkﬁxed. For the optimization with respect to the mixing coefﬁcients πkwe need
to take account of the constraint∑
kπk=1, which can be done with the aid of a

the form
n=1K∑
k=1γnk{
kφn,β−1)}
.
Lagrange multiplier, leading to an M-step re-estimation equation for πkin the form Exercise 14.14
πk=1
NN∑
n=1γnk. (14.38)
Note that this has exactly the same form as the corresponding result for a simple
mixture of unconditional Gaussians given by (9.22).
Next consider the maximization with respect to the parameter vector wkof the
kthlinear regression model. Substituting for the Gaussian distribution, we see that
the function Q(θ,θold), as a function of the parameter vector wk, takes the form

πk=1
NN∑
n=1γnk{
−β
2(
tn−wT
kφn)2}
+c o n s t (14.39)
where the constant term includes the contributions from other weight vectors wjfor
j̸=k. Note that the quantity we are maximizing is similar to the (negative of the)
standard sum-of-squares error (3.12) for a single linear regression model, but with
the inclusion of the responsibilities γnk. This represents a weighted least squares

14.5. Conditional Mixture Models 669
problem, in which the term corresponding to the nthdata point carries a weighting
coefﬁcient given by βγnk, which could be interpreted as an effective precision for
each data point. We see that each component linear regression model in the mixture,
governed by its own parameter vector wk, is ﬁtted separately to the whole data set in
the M step, but with each data point nweighted by the responsibility γnkthat model
ktakes for that data point. Setting the derivative of (14.39) with respect to wkequal
0=N∑
n=1γnk(
tn−wT
kφn)
φn (14.40)
which we can write in matrix notation as
0=ΦTRk(t−Φwk) (14.41)
whereRk= diag( γnk)is a diagonal matrix of size N×N. Solving for wk,w e
obtain
wk=(
ΦTRkΦ)−1ΦTRkt. (14.42)
This represents a set of modiﬁed normal equations corresponding to the weighted
least squares problem, of the same form as (4.99) found in the context of logisticregression. Note that after each E step, the matrix R
kwill change and so we will

0=N∑
n=1γnk(
tn−wT
kφn)
φn (14.40)
obtain
wk=(
have to solve the normal equations afresh in the subsequent M step.
Finally, we maximize Q(θ,θold)with respect to β. Keeping only terms that
depend on β, the function Q(θ,θold)can be written
n=1K∑
k=1γnk{1
2lnβ−β
2(
tn−wT
kφn)2}
. (14.43)
Setting the derivative with respect to βequal to zero, and rearranging, we obtain the
M-step equation for βin the form
1
β=1
NN∑
n=1K∑
k=1γnk(
tn−wT
kφn)2. (14.44)
In Figure 14.8, we illustrate this EM algorithm using the simple example of

n=1K∑
k=1γnk{1
2lnβ−β
2(
tn−wT
kφn)2}
. (14.43)
1
β=1
NN∑
n=1K∑
k=1γnk(
tn−wT
ﬁtting a mixture of two straight lines to a data set having one input variable xand
one target variable t. The predictive density (14.34) is plotted in Figure 14.9 using
the converged parameter values obtained from the EM algorithm, corresponding to
the right-hand plot in Figure 14.8. Also shown in this ﬁgure is the result of ﬁtting
a single linear regression model, which gives a unimodal predictive density. We see
that the mixture model gives a much better representation of the data distribution,

and this is reﬂected in the higher likelihood value. However, the mixture modelalso assigns signiﬁcant probability mass to regions where there is no data because its
predictive distribution is bimodal for all values of x. This problem can be resolved by
extending the model to allow the mixture coefﬁcients themselves to be functions ofx, leading to models such as the mixture density networks discussed in Section 5.6,
and hierarchical mixture of experts discussed in Section 14.5.3.

670 14. COMBINING MODELS
−1 −0.5 0 0.5 1−1.5−1−0.500.511.5
Figure 14.8 Example of a synthetic data set, shown by the green points, having one input variable xand one
target variable t, together with a mixture of two linear regression models whose mean functions y(x,wk), where
k∈{1,2}, are shown by the blue and red lines. The upper three plots show the initial conﬁguration (left), the
result of running 30 iterations of EM (centre), and the result after 50 iterations of EM (right). Here βwas initialized
to the reciprocal of the true variance of the set of target values. The lower three plots show the corresponding
responsibilities plotted as a vertical line for each data point in which the length of the blue segment gives the
posterior probability of the blue line for that data point (and similarly for the red segment).
14.5.2 Mixtures of logistic models

Because the logistic regression model deﬁnes a conditional distribution for the
target variable, given the input vector, it is straightforward to use it as the component
distribution in a mixture model, thereby giving rise to a richer family of conditional
distributions compared to a single logistic regression model. This example involves
a straightforward combination of ideas encountered in earlier sections of the book
and will help consolidate these for the reader.

The conditional distribution of the target variable, for a probabilistic mixture of
Klogistic regression models, is given by
p(t|φ,θ)=K∑
k=1πkyt
k[1−yk]1−t(14.45)
where φis the feature vector, yk=σ(
wT
kφ)
is the output of component k, andθ
denotes the adjustable parameters namely {πk}and{wk}.
Now suppose we are given a data set {φn,tn}. The corresponding likelihood

14.5. Conditional Mixture Models 671
Figure 14.9 The left plot shows the predictive conditional density corresponding to the converged solution in
Figure 14.8. This gives a log likelihood value of −3.0. A vertical slice through one of these plots at a particular
value of xrepresents the corresponding conditional distribution p(t|x), which we see is bimodal. The plot on the
right shows the predictive density for a single linear regression model ﬁtted to the same data set using maximum
likelihood. This model has a smaller log likelihood of −27.6.
function is then given by
p(t|θ)=N∏
n=1(K∑
k=1πkytn
nk[1−ynk]1−tn)
(14.46)
where ynk=σ(wT
kφn)and t=(t1,...,t N)T. We can maximize this likelihood
function iteratively by making use of the EM algorithm. This involves introducing
latent variables znkthat correspond to a 1-of- Kcoded binary indicator variable for
each data point n. The complete-data likelihood function is then given by
p(t,Z|θ)=N∏
n=1K∏
k=1{
πkytn
nk[1−ynk]1−tn}znk(14.47)

p(t|θ)=N∏
n=1(K∑
k=1πkytn
(14.46)
n=1K∏
k=1{
πkytn
whereZis the matrix of latent variables with elements znk. We initialize the EM
algorithm by choosing an initial value θoldfor the model parameters. In the E step,
we then use these parameter values to evaluate the posterior probabilities of the com-ponents kfor each data point n, which are given by
γ
nk=E[znk]=p(k|φn,θold)=πkytn
nk[1−ynk]1−tn
∑
jπjytn
nj[1−ynj]1−tn. (14.48)
These responsibilities are then used to ﬁnd the expected complete-data log likelihood
as a function of θ, given by
Q(θ,θold)= EZ[lnp(t,Z|θ)]

γ
∑
jπjytn
=N∑
n=1K∑
k=1γnk{lnπk+tnlnynk+( 1−tn)l n( 1−ynk)}.(14.49)

672 14. COMBINING MODELS
The M step involves maximization of this function with respect to θ, keeping θold,
and hence γnk, ﬁxed. Maximization with respect to πkcan be done in the usual way,
with a Lagrange multiplier to enforce the summation constraint∑
kπk=1, giving
the familiar result
πk=1
NN∑
n=1γnk. (14.50)
To determine the {wk}, we note that the Q(θ,θold)function comprises a sum
over terms indexed by keach of which depends only on one of the vectors wk,s o
that the different vectors are decoupled in the M step of the EM algorithm. In other
words, the different components interact only via the responsibilities, which are ﬁxedduring the M step. Note that the M step does not have a closed-form solution and
must be solved iteratively using, for instance, the iterative reweighted least squares
(IRLS) algorithm. The gradient and the Hessian for the vector w
kare given by Section 4.3.3
∇kQ=N∑
n=1γnk(tn−ynk)φn (14.51)
Hk=−∇k∇kQ=N∑
n=1γnkynk(1−ynk)φnφT
n (14.52)

∇kQ=N∑
n (14.52)
where ∇kdenotes the gradient with respect to wk. For ﬁxed γnk, these are indepen-
dent of {wj}forj̸=kand so we can solve for each wkseparately using the IRLS
algorithm. Thus the M-step equations for component kcorrespond simply to ﬁtting Section 4.3.3
a single logistic regression model to a weighted data set in which data point ncarries
a weight γnk. Figure 14.10 shows an example of the mixture of logistic regression

∇kQ=N∑
n (14.52)
models applied to a simple classiﬁcation problem. The extension of this model to amixture of softmax models for more than two classes is straightforward. Exercise 14.16
14.5.3 Mixtures of experts
In Section 14.5.1, we considered a mixture of linear regression models, and in
Section 14.5.2 we discussed the analogous mixture of linear classiﬁers. Although
these simple mixtures extend the ﬂexibility of linear models to include more com-

plex (e.g., multimodal) predictive distributions, they are still very limited. We canfurther increase the capability of such models by allowing the mixing coefﬁcients
themselves to be functions of the input variable, so that
p(t|x)=K∑
k=1πk(x)pk(t|x). (14.53)
This is known as a mixture of experts model (Jacobs et al. , 1991) in which the mix-
ing coefﬁcients πk(x)are known as gating functions and the individual component
densities pk(t|x)are called experts . The notion behind the terminology is that differ-

p(t|x)=K∑
ent components can model the distribution in different regions of input space (they

14.5. Conditional Mixture Models 673
Figure 14.10 Illustration of a mixture of logistic regression models. The left plot shows data points drawn
from two classes denoted red and blue, in which the background colour (which varies from pure red to pure blue)
denotes the true probability of the class label. The centre plot shows the result of ﬁtting a single logistic regressionmodel using maximum likelihood, in which the background colour denotes the corresponding probability of theclass label. Because the colour is a near-uniform purple, we see that the model assigns a probability of around0.5to each of the classes over most of input space. The right plot shows the result of ﬁtting a mixture of two
logistic regression models, which now gives much higher probability to the correct labels for many of the points
in the blue class.
are ‘experts’ at making predictions in their own regions), and the gating functions
determine which components are dominant in which region.

The gating functions πk(x)must satisfy the usual constraints for mixing co-
efﬁcients, namely 0⩽πk(x)⩽1and∑
kπk(x)=1 . They can therefore be
represented, for example, by linear softmax models of the form (4.104) and (4.105).
If the experts are also linear (regression or classiﬁcation) models, then the wholemodel can be ﬁtted efﬁciently using the EM algorithm, with iterative reweighted
least squares being employed in the M step (Jordan and Jacobs, 1994).
Such a model still has signiﬁcant limitations due to the use of linear models
for the gating and expert functions. A much more ﬂexible model is obtained byusing a multilevel gating function to give the hierarchical mixture of experts ,o r

HME model (Jordan and Jacobs, 1994). To understand the structure of this model,
imagine a mixture distribution in which each component in the mixture is itself amixture distribution. For simple unconditional mixtures, this hierarchical mixture is
trivially equivalent to a single ﬂat mixture distribution. However, when the mixing Exercise 14.17
coefﬁcients are input dependent, this hierarchical model becomes nontrivial. The
HME model can also be viewed as a probabilistic version of decision trees discussed

in Section 14.4 and can again be trained efﬁciently by maximum likelihood using an
EM algorithm with IRLS in the M step. A Bayesian treatment of the HME has been Section 4.3.3
given by Bishop and Svens ´en (2003) based on variational inference.
We shall not discuss the HME in detail here. However, it is worth pointing out
the close connection with the mixture density network discussed in Section 5.6. The

principal advantage of the mixtures of experts model is that it can be optimized byEM in which the M step for each mixture component and gating model involves
a convex optimization (although the overall optimization is nonconvex). By con-
trast, the advantage of the mixture density network approach is that the component

674 14. COMBINING MODELS
densities and the mixing coefﬁcients share the hidden units of the neural network.
Furthermore, in the mixture density network, the splits of the input space are furtherrelaxed compared to the hierarchical mixture of experts in that they are not only soft,
and not constrained to be axis aligned, but they can also be nonlinear.
Exercises
14.1 (⋆⋆)www Consider a set models of the form p(t|x,zh,θh,h)in which xis the
input vector, tis the target vector, hindexes the different models, zhis a latent vari-
able for model h, andθhis the set of parameters for model h. Suppose the models
have prior probabilities p(h)and that we are given a training set X={x1,...,xN}
andT={t1,...,tN}. Write down the formulae needed to evaluate the predic-
tive distribution p(t|x,X,T)in which the latent variables and the model index are
marginalized out. Use these formulae to highlight the difference between Bayesian
averaging of different models and the use of latent variables within a single model.

14.2 (⋆)The expected sum-of-squares error EAVfor a simple committee model can
be deﬁned by (14.10), and the expected error of the committee itself is given by
(14.11). Assuming that the individual errors satisfy (14.12) and (14.13), derive the
result (14.14).
14.3 (⋆)www By making use of Jensen’s inequality (1.115), for the special case of
the convex function f(x)=x2, show that the average expected sum-of-squares
errorEAVof the members of a simple committee model, given by (14.10), and the

expected error ECOM of the committee itself, given by (14.11), satisfy
ECOM⩽EAV. (14.54)
14.4 (⋆⋆)By making use of Jensen’s in equality (1.115), show that the result (14.54)
derived in the previous exercise hods for any error function E(y), not just sum-of-
squares, provided it is a convex function of y.
14.5 (⋆⋆)www Consider a committee in which we allow unequal weighting of the
constituent models, so that
yCOM(x)=M∑
m=1αmym(x). (14.55)
In order to ensure that the predictions yCOM(x)remain within sensible limits, sup-

pose that we require that they be bounded at each value of xby the minimum and
maximum values given by any of the members of the committee, so that
ymin(x)⩽yCOM(x)⩽ymax(x). (14.56)
Show that a necessary and sufﬁcient condition for this constraint is that the coefﬁ-
cients αmsatisfy
αm⩾0,M∑
m=1αm=1. (14.57)

Exercises 675
14.6 (⋆)www By differentiating the error function (14.23) with respect to αm, show
that the parameters αmin the AdaBoost algorithm are updated using (14.17) in
which ϵmis deﬁned by (14.16).
14.7 (⋆)By making a variational minimization of the expected exponential error function
given by (14.27) with respect to all possible functions y(x), show that the minimizing
function is given by (14.28).
14.8 (⋆)Show that the exponential error function (14.20), which is minimized by the
AdaBoost algorithm, does not correspond to the log likelihood of any well-behaved
probabilistic model. This can be done by showing that the corresponding conditionaldistribution p(t|x)cannot be correctly normalized.
14.9 (⋆)
www Show that the sequential minimization of the sum-of-squares error func-
tion for an additive model of the form (14.21) in the style of boosting simply involvesﬁtting each new base classiﬁer to the residual errors t
n−fm−1(xn)from the previous
model.

14.9 (⋆)
model.
14.10 (⋆)Verify that if we minimize the sum-of-squares error between a set of training
values {tn}and a single predictive value t, then the optimal solution for tis given
by the mean of the {tn}.
14.11 (⋆⋆)Consider a data set comprising 400data points from class C1and400data
points from class C2. Suppose that a tree model A splits these into (300,100) at
the ﬁrst leaf node and (100,300) at the second leaf node, where (n, m)denotes that
npoints are assigned to C1andmpoints are assigned to C2. Similarly, suppose

that a second tree model B splits them into (200,400) and(200,0). Evaluate the
misclassiﬁcation rates for the two trees and hence show that they are equal. Similarly,
evaluate the cross-entropy (14.32) and Gini index (14.33) for the two trees and show
that they are both lower for tree B than for tree A.
14.12 (⋆⋆)Extend the results of Section 14.5.1 for a mixture of linear regression models
to the case of multiple target values described by a vector t. To do this, make use of
the results of Section 3.1.5.

14.13 (⋆)www Verify that the complete-data log likelihood function for the mixture of
linear regression models is given by (14.36).
14.14 (⋆)Use the technique of Lagrange multipliers (Appendix E) to show that the M-step
re-estimation equation for the mixing coefﬁcients in the mixture of linear regressionmodels trained by maximum likelihood EM is given by (14.38).
14.15 (⋆)
www We have already noted that if we use a squared loss function in a regres-

14.15 (⋆)
sion problem, the corresponding optimal prediction of the target variable for a newinput vector is given by the conditional mean of the predictive distribution. Show
that the conditional mean for the mixture of linear regression models discussed in
Section 14.5.1 is given by a linear combination of the means of each component dis-tribution. Note that if the conditional distribution of the target data is multimodal,
the conditional mean can give poor predictions.

676 14. COMBINING MODELS
14.16 (⋆⋆⋆ )Extend the logistic regression mixture model of Section 14.5.2 to a mixture
of softmax classiﬁers representing C⩾2classes. Write down the EM algorithm for
determining the parameters of this model through maximum likelihood.
14.17 (⋆⋆)www Consider a mixture model for a conditional distribution p(t|x)of the
form
p(t|x)=K∑
k=1πkψk(t|x) (14.58)
in which each mixture component ψk(t|x)is itself a mixture model. Show that this
two-level hierarchical mixture is equivalent to a conventional single-level mixturemodel. Now suppose that the mixing coefﬁcients in both levels of such a hierar-
chical model are arbitrary functions of x. Again, show that this hierarchical model
is again equivalent to a single-level model with x-dependent mixing coefﬁcients.
Finally, consider the case in which the mixing coefﬁcients at both levels of the hi-
erarchical mixture are constrained to be linear classiﬁcation (logistic or softmax)

models. Show that the hierarchical mixture cannot in general be represented by asingle-level mixture having linear classiﬁcation models for the mixing coefﬁcients.
Hint: to do this it is sufﬁcient to construct a single counter-example, so consider a
mixture of two components in which one of those components is itself a mixture oftwo components, with mixing coefﬁcients given by linear-logistic models. Show that
this cannot be represented by a single-level mixture of 3 components having mixing

coefﬁcients determined by a linear-softmax model.

Appendix A. Data Sets
In this appendix, we give a brief introduction to the data sets used to illustrate some
of the algorithms described in this book. Detailed information on ﬁle formats for
these data sets, as well as the data ﬁles themselves, can be obtained from the book
web site:
Handwritten Digits
The digits data used in this book is taken from the MNIST data set (LeCun et al. ,
1998), which itself was constructed by modifying a subset of the much larger data
set produced by NIST (the National Institute of Standards and Technology). It com-
prises a training set of 60,000examples and a test set of 10,000examples. Some
of the data was collected from Census Bureau employees and the rest was collected
from high-school children, and care was taken to ensure that the test examples werewritten by different individuals to the training examples.
The original NIST data had binary (black or white) pixels. To create MNIST,

these images were size normalized to ﬁt in a 20×20pixel box while preserving their
aspect ratio. As a consequence of the anti-aliasing used to change the resolution of
the images, the resulting MNIST digits are grey scale. These images were then
centred in a 28×28box. Examples of the MNIST digits are shown in Figure A.1.
Error rates for classifying the digits range from 12% for a simple linear classi-
ﬁer, through 0.56% for a carefully designed support vector machine, to 0.4% for a
convolutional neural network (LeCun et al. , 1998).
677

678 A. DATA SETS
Figure A.1 One hundred examples of the
MNIST digits chosen at ran-
dom from the training set.
Oil Flow
This is a synthetic data set that arose out of a project aimed at measuring nonin-
vasively the proportions of oil, water, and gas in North Sea oil transfer pipelines
(Bishop and James, 1993). It is based on the principle of dual-energy gamma densit-
ometry . The ideas is that if a narrow beam of gamma rays is passed through the pipe,
the attenuation in the intensity of the beam provides information about the density of
material along its path. Thus, for instance, the beam will be attenuated more stronglyby oil than by gas.
A single attenuation measurement alone is not sufﬁcient because there are two
degrees of freedom corresponding to the fraction of oil and the fraction of water (thefraction of gas is redundant because the three fractions must add to one). To address
this, two gamma beams of different energies (in other words different frequencies or

wavelengths) are passed through the pipe along the same path, and the attenuation ofeach is measured. Because the absorbtion properties of different materials vary dif-
ferently as a function of energy, measurement of the attenuations at the two energies
provides two independent pieces of information. Given the known absorbtion prop-erties of oil, water, and gas at the two energies, it is then a simple matter to calculate
the average fractions of oil and water (and hence of gas) measured along the path of
the gamma beams.

There is a further complication, however, associated with the motion of the ma-
terials along the pipe. If the ﬂow velocity is small, then the oil ﬂoats on top of the
water with the gas sitting above the oil. This is known as a laminar orstratiﬁed

A. DATA SETS 679
Figure A.2 The three geometrical conﬁgurations of the oil,
water, and gas phases used to generate the oil-
ﬂow data set. For each conﬁguration, the pro-
portions of the three phases can vary.
MixGasWaterOil
HomogeneousStratiﬁed Annular
ﬂow conﬁguration and is illustrated in Figure A.2. As the ﬂow velocity is increased,
more complex geometrical conﬁgurations of the oil, water, and gas can arise. For the
purposes of this data set, two speciﬁc idealizations are considered. In the annular
conﬁguration the oil, water, and gas form concentric cylinders with the water around
the outside and the gas in the centre, whereas in the homogeneous conﬁguration the
oil, water and gas are assumed to be intimately mixed as might occur at high ﬂow
velocities under turbulent conditions. These conﬁgurations are also illustrated in
Figure A.2.
We have seen that a single dual-energy beam gives the oil and water fractions
measured along the path length, whereas we are interested in the volume fractions of

oil and water. This can be addressed by using multiple dual-energy gamma densit-
ometers whose beams pass through different regions of the pipe. For this particular
data set, there are six such beams, and their spatial arrangement is shown in Fig-
ure A.3. A single observation is therefore represented by a 12-dimensional vector
comprising the fractions of oil and water measured along the paths of each of the
beams. We are, however, interested in obtaining the overall volume fractions of the

three phases in the pipe. This is much like the classical problem of tomographic re-
construction, used in medical imaging for example, in which a two-dimensional dis-
Figure A.3 Cross section of the pipe showing the arrangement of the
six beam lines, each of which comprises a single dual-
energy gamma densitometer. Note that the vertical beams
are asymmetrically arranged relative to the central axis
(shown by the dotted line).

680 A. DATA SETS
tribution is to be reconstructed from an number of one-dimensional averages. Here
there are far fewer line measurements than in a typical tomography application. Onthe other hand the range of geometrical conﬁgurations is much more limited, and so
the conﬁguration, as well as the phase fractions, can be predicted with reasonable
accuracy from the densitometer data.
For safety reasons, the intensity of the gamma beams is kept relatively weak and
so to obtain an accurate measurement of the attenuation, the measured beam intensity
is integrated over a speciﬁc time interval. For a ﬁnite integration time, there arerandom ﬂuctuations in the measured intensity due to the fact that the gamma beams
comprise discrete packets of energy called photons. In practice, the integration time
is chosen as a compromise between reducing the noise level (which requires a long
integration time) and detecting temporal variations in the ﬂow (which requires a short

integration time). The oil ﬂow data set is generated using realistic known values forthe absorption properties of oil, water, and gas at the two gamma energies used, and
with a speciﬁc choice of integration time ( 10seconds) chosen as characteristic of a
typical practical setup.
Each point in the data set is generated independently using the following steps:
1. Choose one of the three phase conﬁgurations at random with equal probability.2. Choose three random numbers f
1,f2andf3from the uniform distribution over

(0,1)and deﬁne
foil=f1
f1+f2+f3,f water=f2
f1+f2+f3. (A.1)
This treats the three phases on an equal footing and ensures that the volume
fractions add to one.
3. For each of the six beam lines, calculate the effective path lengths through oil
and water for the given phase conﬁguration.
4. Perturb the path lengths using the Poisson distribution based on the known
beam intensities and integration time to allow for the effect of photon statistics.
Each point in the data set comprises the 12path length measurements, together

foil=f1
with the fractions of oil and water and a binary label describing the phase conﬁgu-ration. The data set is divided into training, validation, and test sets, each of which
comprises 1,000independent data points. Details of the data format are available
from the book web site.
In Bishop and James (1993), statistical machine learning techniques were used
to predict the volume fractions and also the geometrical conﬁguration of the phases

shown in Figure A.2, from the 12-dimensional vector of measurements. The 12-
dimensional observation vectors can also be used to test data visualization algo-
rithms.
This data set has a rich and interesting structure, as follows. For any given
conﬁguration there are two degrees of freedom corresponding to the fractions of

A. DATA SETS 681
oil and water, and so for inﬁnite integration time the data will locally live on a two-
dimensional manifold. For a ﬁnite integration time, the individual data points will beperturbed away from the manifold by the photon noise. In the homogeneous phase
conﬁguration, the path lengths in oil and water are linearly related to the fractions of
oil and water, and so the data points lie close to a linear manifold. For the annularconﬁguration, the relationship between phase fraction and path length is nonlinear
and so the manifold will be nonlinear. In the case of the laminar conﬁguration the
situation is even more complex because small variations in the phase fractions cancause one of the horizontal phase boundaries to move across one of the horizontal
beam lines leading to a discontinuous jump in the 12-dimensional observation space.
In this way, the two-dimensional nonlinear manifold for the laminar conﬁguration is

broken into six distinct segments. Note also that some of the manifolds for different
phase conﬁgurations meet at speciﬁc points, for example if the pipe is ﬁlled entirelywith oil, it corresponds to speciﬁc instances of the laminar, annular, and homoge-
neous conﬁgurations.
Old Faithful
Old Faithful, shown in Figure A.4, is a hydrothermal geyser in Yellowstone National
Park in the state of Wyoming, U.S.A., and is a popular tourist attraction. Its name
stems from the supposed regularity of its eruptions.
The data set comprises 272observations, each of which represents a single erup-

tion and contains two variables corresponding to the duration in minutes of the erup-
tion, and the time until the next eruption, also in minutes. Figure A.5 shows a plot of
the time to the next eruption versus the duration of the eruptions. It can be seen thatthe time to the next eruption varies considerably, although knowledge of the duration
of the current eruption allows it to be predicted more accurately. Note that there exist
several other data sets relating to the eruptions of Old Faithful.

Figure A.4 The Old Faithful geyser
in Y ellowstone National
Park. c⃝Bruce T. Gourley
www.brucegourley.com.

682 A. DATA SETS
Figure A.5 Plot of the time to the next eruption
in minutes (vertical axis) versus the
duration of the eruption in minutes
(horizontal axis) for the Old Faithful
data set.
1 2 3 4 5 6405060708090100
Synthetic Data
Throughout the book, we use two simple synthetic data sets to illustrate many of the
algorithms. The ﬁrst of these is a regression problem, based on the sinusoidal func-
tion, shown in Figure A.6. The input values {xn}are generated uniformly in range
(0,1), and the corresponding target values {tn}are obtained by ﬁrst computing the
corresponding values of the function sin(2πx), and then adding random noise with
a Gaussian distribution having standard deviation 0.3. Various forms of this data set,
having different numbers of data points, are used in the book.
The second data set is a classiﬁcation problem having two classes, with equal
prior probabilities, and is shown in Figure A.7. The blue class is generated from a

single Gaussian while the red class comes from a mixture of two Gaussians. Be-
cause we know the class priors and the class-conditional densities, it is straightfor-
ward to evaluate and plot the true posterior probabilities as well as the minimum
misclassiﬁcation-rate decision boundary, as shown in Figure A.7.

A. DATA SETS 683
xt
0 1−101
xt
0 1−101
Figure A.6 The left-hand plot shows the synthetic regression data set along with the underlying sinusoidal
function from which the data points were generated. The right-hand plot shows the true conditional distribution
p(t|x)from which the labels are generated, in which the green curve denotes the mean, and the shaded region
spans one standard deviation on each side of the mean.
−2 0 2−202
Figure A.7 The left plot shows the synthetic classiﬁcation data set with data from the two classes shown in
red and blue. On the right is a plot of the true posterior probabilities, shown on a colour scale going from pure
red denoting probability of the red class is 1to pure blue denoting probability of the red class is 0. Because
these probabilities are known, the optimal decision boundary for minimizing the misclassiﬁcation rate (which
corresponds to the contour along which the posterior probabilities for each class equal 0.5) can be evaluated

and is shown by the green curve. This decision boundary is also plotted on the left-hand ﬁgure.

Appendix B. Probability Distributions
In this appendix, we summarize the main properties of some of the most widely used
probability distributions, and for each distribution we list some key statistics such as
the expectation E[x], the variance (or covariance), the mode, and the entropy H[x].
All of these distributions are members of the exponential family and are widely used
as building blocks for more sophisticated probabilistic models.
Bernoulli
This is the distribution for a single binary variable x∈{0,1}representing, for
example, the result of ﬂipping a coin. It is governed by a single continuous parameter
µ∈[0,1]that represents the probability of x=1.
Bern(x|µ)= µx(1−µ)1−x(B.1)
E[x]= µ (B.2)
var[x]= µ(1−µ) (B.3)
mode[ x]={
1ifµ⩾0.5,
0otherwise(B.4)
H[x]= −µlnµ−(1−µ)l n ( 1−µ). (B.5)
The Bernoulli is a special case of the binomial distribution for the case of a single
observation. Its conjugate prior for µis the beta distribution.
685

686 B. PROBABILITY DISTRIBUTIONS
Beta
This is a distribution over a continuous variable µ∈[0,1], which is often used to
represent the probability for some binary event. It is governed by two parameters a
andbthat are constrained by a>0andb>0to ensure that the distribution can be
normalized.
Γ(a)Γ(b)µa−1(1−µ)b−1(B.6)
E[µ]=a
a+b(B.7)
var[µ]=ab
(a+b)2(a+b+1 )(B.8)
a+b−2. (B.9)
The beta is the conjugate prior for the Bernoulli distribution, for which aandbcan
be interpreted as the effective prior number of observations of x=1 andx=0,
respectively. Its density is ﬁnite if a⩾1andb⩾1, otherwise there is a singularity
atµ=0and/or µ=1.F o ra=b=1, it reduces to a uniform distribution. The beta
distribution is a special case of the K-state Dirichlet distribution for K=2.
Binomial
The binomial distribution gives the probability of observing moccurrences of x=1
in a set of Nsamples from a Bernoulli distribution, where the probability of observ-
ingx=1isµ∈[0,1].
m)

Binomial
m)
µm(1−µ)N−m(B.10)
E[m]= Nµ (B.11)
var[m]= Nµ(1−µ) (B.12)
mode[ m]= ⌊(N+1 )µ⌋ (B.13)
where ⌊(N+1 )µ⌋denotes the largest integer that is less than or equal to (N+1 )µ,
and the quantity(N
m)
=N!
m!(N−m)!(B.14)
denotes the number of ways of choosing mobjects out of a total of Nidentical
objects. Here m!, pronounced ‘factorial m’, denotes the product m×(m−1)×
...,×2×1. The particular case of the binomial distribution for N=1is known as
the Bernoulli distribution, and for large Nthe binomial distribution is approximately

m)
=N!
Gaussian. The conjugate prior for µis the beta distribution.

B. PROBABILITY DISTRIBUTIONS 687
Dirichlet
The Dirichlet is a multivariate distribution over Krandom variables 0⩽µk⩽1,
where k=1,...,K , subject to the constraints
0⩽µk⩽1,K∑
k=1µk=1. (B.15)
Denoting µ=(µ1,...,µ K)Tandα=(α1,...,α K)T,w eh a v e
Dir(µ|α)= C(α)K∏
k=1µαk−1
k(B.16)
E[µk]=αk
ˆα(B.17)
var[µk]=αk(ˆα−αk)
ˆα2(ˆα+1 )(B.18)
cov[µjµk]= −αjαk
ˆα2(ˆα+1 )(B.19)
mode[ µk]=αk−1
ˆα−K(B.20)
E[lnµk]= ψ(αk)−ψ(ˆα) (B.21)
H[µ]= −K∑
k=1(αk−1){ψ(αk)−ψ(ˆα)}−lnC(α) (B.22)
where
C(α)=Γ(ˆα)
Γ(α1)···Γ(αK)(B.23)
and
ˆα=K∑
k=1αk. (B.24)
Here
ψ(a)≡d
daln Γ(a) (B.25)
is known as the digamma function (Abramowitz and Stegun, 1965). The parameters
αkare subject to the constraint αk>0in order to ensure that the distribution can be
The Dirichlet forms the conjugate prior for the multinomial distribution and rep-
resents a generalization of the beta distribution. In this case, the parameters αkcan
be interpreted as effective numbers of observations of the corresponding values of

and
ˆα=K∑
Here
ψ(a)≡d
theK-dimensional binary observation vector x. As with the beta distribution, the
Dirichlet has ﬁnite density everywhere provided αk⩾1for all k.

688 B. PROBABILITY DISTRIBUTIONS
Gamma
The Gamma is a probability distribution over a positive random variable τ>0
governed by parameters aandbthat are subject to the constraints a>0andb>0
to ensure that the distribution can be normalized.
Gam( τ|a, b)=1
Γ(a)baτa−1e−bτ(B.26)
E[τ]=a
b(B.27)
var[τ]=a
b2(B.28)
mode[ τ]=a−1
bforα⩾1 (B.29)
E[lnτ]= ψ(a)−lnb (B.30)
H[τ]=l n Γ ( a)−(a−1)ψ(a)−lnb+a (B.31)
where ψ(·)is the digamma function deﬁned by (B.25). The gamma distribution is
the conjugate prior for the precision (inverse variance) of a univariate Gaussian. For
a⩾1the density is everywhere ﬁnite, and the special case of a=1is known as the
exponential distribution.
Gaussian
The Gaussian is the most widely used distribution for continuous variables. It is also
known as the normal distribution. In the case of a single variable x∈(−∞,∞)it is
governed by two parameters, the mean µ∈(−∞,∞)and the variance σ2>0.
N(x|µ, σ2)=1
−1
(B.32)
E[x]= µ (B.33)
var[x]= σ2(B.34)
mode[ x]= µ (B.35)
H[x]=1

Gaussian
−1
(B.32)
H[x]=1
2lnσ2+1
2(1 + ln(2 π)). (B.36)
The inverse of the variance τ=1/σ2is called the precision, and the square root
of the variance σis called the standard deviation. The conjugate prior for µis the
Gaussian, and the conjugate prior for τis the gamma distribution. If both µandτ
are unknown, their joint conjugate prior is the Gaussian-gamma distribution.
For a D-dimensional vector x, the Gaussian is governed by a D-dimensional
mean vector µand a D×Dcovariance matrix Σthat must be symmetric and

B. PROBABILITY DISTRIBUTIONS 689
positive-deﬁnite.
(2π)D/21
−1
(B.37)
E[x]= µ (B.38)
cov[x]= Σ (B.39)
mode[x]= µ (B.40)
H[x]=1
2ln|Σ|+D
2(1 + ln(2 π)). (B.41)
The inverse of the covariance matrix Λ=Σ−1is the precision matrix, which is also
symmetric and positive deﬁnite. Averages of random variables tend to a Gaussian, bythe central limit theorem, and the sum of two Gaussian variables is again Gaussian.
The Gaussian is the distribution that maximizes the entropy for a given variance
(or covariance). Any linear transformation of a Gaussian random variable is againGaussian. The marginal distribution of a multivariate Gaussian with respect to a
subset of the variables is itself Gaussian, and similarly the conditional distribution is
also Gaussian. The conjugate prior for µis the Gaussian, the conjugate prior for Λ
is the Wishart, and the conjugate prior for (µ,Λ)is the Gaussian-Wishart.
If we have a marginal Gaussian distribution for xand a conditional Gaussian

distribution for ygivenxin the form
p(x)= N(x|µ,Λ
−1) (B.42)
p(y|x)= N(y|Ax+b,L−1) (B.43)
then the marginal distribution of y, and the conditional distribution of xgiveny, are
given by
p(y)= N(y|Aµ+b,L−1+AΛ−1AT) (B.44)
p(x|y)= N(x|Σ{ATL(y−b)+Λµ},Σ) (B.45)
where
Σ=(Λ+ATLA)−1. (B.46)
If we have a joint Gaussian distribution N(x|µ,Σ)withΛ≡Σ−1and we
deﬁne the following partitions
x=(
xa
xb)
,µ=(
µa
µb)
(B.47)
Σ=(
ΣaaΣab
ΣbaΣbb)
,Λ=(
ΛaaΛab
ΛbaΛbb)
(B.48)
then the conditional distribution p(xa|xb)is given by

−1) (B.42)
given by
where
x=(
xa
xb)
,µ=(
µa
µb)
(B.47)
Σ=(
ΣaaΣab
ΣbaΣbb)
,Λ=(
ΛaaΛab
ΛbaΛbb)
(B.48)
aa) (B.49)
aaΛab(xb−µb) (B.50)

690 B. PROBABILITY DISTRIBUTIONS
and the marginal distribution p(xa)is given by
p(xa)=N(xa|µa,Σaa). (B.51)
Gaussian-Gamma
This is the conjugate prior distribution for a univariate Gaussian N(x|µ, λ−1)in
which the mean µand the precision λare both unknown and is also called the
normal-gamma distribution. It comprises the product of a Gaussian distribution for
µ, whose precision is proportional to λ, and a gamma distribution over λ.
p(µ, λ|µ0,β,a ,b )=N(
µ|µo,(βλ)−1)
Gam( λ|a, b). (B.52)
Gaussian-Wishart
This is the conjugate prior distribution for a multivariate Gaussian N(x|µ,Λ)in
which both the mean µand the precision Λare unknown, and is also called the
normal-Wishart distribution. It comprises the product of a Gaussian distribution for
µ, whose precision is proportional to Λ, and a Wishart distribution over Λ.
p(µ,Λ|µ0,β,W,ν)=N(
µ|µ0,(βΛ)−1)
W(Λ|W,ν). (B.53)
For the particular case of a scalar x, this is equivalent to the Gaussian-gamma distri-
bution.
Multinomial

bution.
If we generalize the Bernoulli distribution to an K-dimensional binary variable x
with components xk∈{0,1}such that∑
kxk=1, then we obtain the following
discrete distribution
p(x)=K∏
k=1µxk
k(B.54)
E[xk]= µk (B.55)
var[xk]= µk(1−µk) (B.56)
cov[xjxk]= Ijkµk (B.57)
H[x]= −M∑
k=1µklnµk (B.58)

B. PROBABILITY DISTRIBUTIONS 691
where Ijkis thej,kelement of the identity matrix. Because p(xk=1 )= µk, the
parameters must satisfy 0⩽µk⩽1and∑
kµk=1.
The multinomial distribution is a multivariate generalization of the binomial and
gives the distribution over counts mkfor aK-state discrete variable to be in state k
given a total number of observations N.
m1m2...m M)M∏
k=1µm k
k(B.59)
E[mk]= Nµk (B.60)
var[mk]= Nµk(1−µk) (B.61)
cov[mjmk]= −Nµjµk (B.62)
where µ=(µ1,...,µ K)T, and the quantity
(N
=N!
m1!...m K!(B.63)
gives the number of ways of taking Nidentical objects and assigning mkof them to
binkfork=1,...,K . The value of µkgives the probability of the random variable
taking state k, and so these parameters are subject to the constraints 0⩽µk⩽1
and∑
kµk=1. The conjugate prior distribution for the parameters {µk}is the
Dirichlet.
Normal

k=1µm k
k(B.59)
(N
=N!
and∑
Normal
The normal distribution is simply another name for the Gaussian. In this book, weuse the term Gaussian throughout, although we retain the conventional use of the
symbol Nto denote this distribution. For consistency, we shall refer to the normal-
gamma distribution as the Gaussian-gamma distribution, and similarly the normal-Wishart is called the Gaussian-Wishart.
Student’s t
This distribution was published by William Gosset in 1908, but his employer, Gui-

Normal
ness Breweries, required him to publish under a pseudonym, so he chose ‘Student’.In the univariate form, Student’s t-distribution is obtained by placing a conjugate
gamma prior over the precision of a univariate Gaussian distribution and then inte-
grating out the precision variable. It can therefore be viewed as an inﬁnite mixture

692 B. PROBABILITY DISTRIBUTIONS
of Gaussians having the same mean but different variances.
Γ(ν/2)(λ
πν)1/2[
1+λ(x−µ)2
(B.64)
E[x]= µforν>1 (B.65)
var[x]=1
λν
ν−2forν>2 (B.66)
mode[ x]= µ. (B.67)
Hereν>0is called the number of degrees of freedom of the distribution. The
particular case of ν=1is called the Cauchy distribution.
For aD-dimensional variable x, Student’s t-distribution corresponds to marginal-
izing the precision matrix of a multivariate Gaussian with respect to a conjugate
Wishart prior and takes the form
St(x|µ,Λ,ν)=Γ(ν/2+D/2)
(νπ)D/2[
1+∆2
ν]−ν/2−D/2
(B.68)
E[x]= µforν>1 (B.69)
cov[x]=ν
ν−2Λ−1forν>2 (B.70)
mode[x]= µ (B.71)
where ∆2is the squared Mahalanobis distance deﬁned by
∆2=(x−µ)TΛ(x−µ). (B.72)
In the limit ν→∞ , the t-distribution reduces to a Gaussian with mean µand pre-
cisionΛ. Student’s t-distribution provides a generalization of the Gaussian whose
maximum likelihood parameter values are robust to outliers.
Uniform

(νπ)D/2[
1+∆2
(B.68)
cov[x]=ν
Uniform
This is a simple distribution for a continuous variable xdeﬁned over a ﬁnite interval
x∈[a, b]where b>a .
b−a(B.73)
E[x]=(b+a)
2(B.74)
var[x]=(b−a)2
12(B.75)
H[x]=l n ( b−a). (B.76)
Ifxhas distribution U(x|0,1), thena+(b−a)xwill have distribution U(x|a, b).

B. PROBABILITY DISTRIBUTIONS 693
Von Mises
The von Mises distribution, also known as the circular normal or the circular Gaus-
sian, is a univariate Gaussian-like periodic distribution for a variable θ∈[0,2π).
2πI0(m)exp{mcos(θ−θ0)} (B.77)
where I0(m)is the zeroth-order Bessel function of the ﬁrst kind. The distribution
has period 2πso that p(θ+2π)=p(θ)for all θ. Care must be taken in interpret-
ing this distribution because simple expectations will be dependent on the (arbitrary)
choice of origin for the variable θ. The parameter θ0is analogous to the mean of a
univariate Gaussian, and the parameter m>0, known as the concentration param-
eter, is analogous to the precision (inverse variance). For large m, the von Mises
distribution is approximately a Gaussian centred on θ0.
Wishart
The Wishart distribution is the conjugate prior for the precision matrix of a multi-
variate Gaussian.
W(Λ|W,ν)=B(W,ν)|Λ|(ν−D−1)/2exp(
−1
(B.78)
where
B(W,ν)≡|W|−ν/2(
2))−1
(B.79)

Wishart
−1
(B.78)
where
2))−1
(B.79)
E[Λ]= νW (B.80)
E[ln|Λ|]=D∑
i=1ψ(ν+1−i
2)
+Dln 2 + ln |W| (B.81)
H[Λ]= −lnB(W,ν)−(ν−D−1)
2(B.82)
whereWis aD×Dsymmetric, positive deﬁnite matrix, and ψ(·)is the digamma
function deﬁned by (B.25). The parameter νis called the number of degrees of
freedom of the distribution and is restricted to ν>D −1to ensure that the Gamma
function in the normalization factor is well-deﬁned. In one dimension, the Wishart
reduces to the gamma distribution Gam( λ|a, b)given by (B.26) with parameters
a=ν/2andb=1/2W.

Appendix C. Properties of Matrices
In this appendix, we gather together some useful properties and identities involving
matrices and determinants. This is not intended to be an introductory tutorial, and
it is assumed that the reader is already familiar with basic linear algebra. For some
results, we indicate how to prove them, whereas in more complex cases we leave
the interested reader to refer to standard textbooks on the subject. In all cases, we
assume that inverses exist and that matrix dimensions are such that the formulaeare correctly deﬁned. A comprehensive discussion of linear algebra can be found in
Golub and Van Loan (1996), and an extensive collection of matrix properties is given
by L ¨utkepohl (1996). Matrix derivatives are discussed in Magnus and Neudecker
(1999).
Basic Matrix Identities
A matrix Ahas elements Aijwhere iindexes the rows, and jindexes the columns.
We use INto denote the N×Nidentity matrix (also called the unit matrix), and

(1999).
where there is no ambiguity over dimensionality we simply use I. The transpose
matrixAThas elements (AT)ij=Aji. From the deﬁnition of transpose, we have
(AB)T=BTAT(C.1)
which can be veriﬁed by writing out the indices. The inverse of A, denoted A−1,
satisﬁes
AA−1=A−1A=I. (C.2)
Because ABB−1A−1=I,w eh a v e
(AB)−1=B−1A−1. (C.3)
Also we have (
AT)−1=(
A−1)T(C.4)
695

696 C. PROPERTIES OF MATRICES
which is easily proven by taking the transpose of (C.2) and applying (C.1).
A useful identity involving matrix inverses is the following
(P−1+BTR−1B)−1BTR−1=PBT(BPBT+R)−1. (C.5)
which is easily veriﬁed by right multiplying both sides by (BPBT+R). Suppose
thatPhas dimensionality N×NwhileRhas dimensionality M×M, so that Bis
M×N. Then if M≪N, it will be much cheaper to evaluate the right-hand side of
(C.5) than the left-hand side. A special case that sometimes arises is
(I+AB)−1A=A(I+BA)−1. (C.6)
Another useful identity involving inverses is the following:
(A+BD−1C)−1=A−1−A−1B(D+CA−1B)−1CA−1(C.7)
which is known as the Woodbury identity and which can be veriﬁed by multiplying
both sides by (A+BD−1C). This is useful, for instance, when Ais large and
diagonal, and hence easy to invert, while Bhas many rows but few columns (and
conversely for C) so that the right-hand side is much cheaper to evaluate than the
left-hand side.

A set of vectors {a1,...,aN}is said to be linearly independent if the relation∑
nαnan=0 holds only if all αn=0. This implies that none of the vectors
can be expressed as a linear combination of the remainder. The rank of a matrix is
the maximum number of linearly independent rows (or equivalently the maximum
number of linearly independent columns).
Traces and Determinants
Trace and determinant apply to square matrices. The trace Tr (A)of a matrix A
is deﬁned as the sum of the elements on the leading diagonal. By writing out the
indices, we see that

Tr(AB)=Tr(BA). (C.8)
By applying this formula multiple times to the product of three matrices, we see that
Tr(ABC )=Tr(CAB )=Tr(BCA ) (C.9)
which is known as the cyclic property of the trace operator and which clearly extends
to the product of any number of matrices. The determinant |A|of anN×Nmatrix
Ais deﬁned by
|A|=∑
(±1)A1i1A2i2···ANi N (C.10)
in which the sum is taken over all products consisting of precisely one element from
each row and one element from each column, with a coefﬁcient +1or−1according

C. PROPERTIES OF MATRICES 697
to whether the permutation i1i2...iNis even or odd, respectively. Note that |I|=1.
Thus, for a 2×2matrix, the determinant takes the form
|A|=⏐⏐⏐⏐a11a12
a21a22⏐⏐⏐⏐
=a11a22−a12a21. (C.11)
The determinant of a product of two matrices is given by
|AB|=|A||B| (C.12)
as can be shown from (C.10). Also, the determinant of an inverse matrix is given by
⏐⏐A−1⏐⏐
=1
|A|(C.13)
which can be shown by taking the determinant of (C.2) and applying (C.12).
IfAandBare matrices of size N×M, then
⏐⏐IN+ABT⏐⏐=⏐⏐
IM+ATB⏐⏐. (C.14)
A useful special case is ⏐⏐IN+abT⏐⏐=1+aTb (C.15)
whereaandbareN-dimensional column vectors.
Matrix Derivatives
Sometimes we need to consider derivatives of vectors and matrices with respect to
scalars. The derivative of a vector awith respect to a scalar xis itself a vector whose
components are given by(∂a
∂x)
i=∂ai
∂x(C.16)
with an analogous deﬁnition for the derivative of a matrix. Derivatives with respect
to vectors and matrices can also be deﬁned, for instance
(∂x
∂a)
i=∂x

∂x)
i=∂ai
∂x(C.16)
(∂x
∂a)
i=∂x
∂ai(C.17)
and similarly(∂a
∂b)
ij=∂ai
∂bj. (C.18)
The following is easily proven by writing out the components
∂
∂x(
xTa)
=∂
∂x(
aTx)
=a. (C.19)

698 C. PROPERTIES OF MATRICES
Similarly
∂
∂x(AB)=∂A
∂xB+A∂B
∂x. (C.20)
The derivative of the inverse of a matrix can be expressed as
∂
∂x(
A−1)
=−A−1∂A
∂xA−1(C.21)
as can be shown by differentiating the equation A−1A=Iusing (C.20) and then
right multiplying by A−1. Also
∂
∂xln|A|=Tr(
A−1∂A
∂x)
(C.22)
which we shall prove later. If we choose xto be one of the elements of A,w eh a v e
∂
∂AijTr(AB)=Bji (C.23)
as can be seen by writing out the matrices using index notation. We can write this
result more compactly in the form
∂
∂ATr(AB)=BT. (C.24)
With this notation, we have the following properties
∂
∂ATr(
ATB)
=B (C.25)
∂
∂ATr(A)= I (C.26)
∂
∂ATr(ABAT)= A(B+BT) (C.27)
which can again be proven by writing out the matrix indices. We also have
∂
∂Aln|A|=(
A−1)T(C.28)
which follows from (C.22) and (C.26).
Eigenvector Equation
For a square matrix Aof size M×M, the eigenvector equation is deﬁned by
Aui=λiui (C.29)

C. PROPERTIES OF MATRICES 699
fori=1,...,M , where uiis an eigenvector andλiis the corresponding eigenvalue .
This can be viewed as a set of Msimultaneous homogeneous linear equations, and
the condition for a solution is that
|A−λiI|=0 (C.30)
which is known as the characteristic equation . Because this is a polynomial of order
Minλi, it must have Msolutions (though these need not all be distinct). The rank
ofAis equal to the number of nonzero eigenvalues.
Of particular interest are symmetric matrices, which arise as covariance ma-
trices, kernel matrices, and Hessians. Symmetric matrices have the property that
Aij=Aji, or equivalently AT=A. The inverse of a symmetric matrix is also sym-
metric, as can be seen by taking the transpose of A−1A=Iand using AA−1=I
together with the symmetry of I.
In general, the eigenvalues of a matrix are complex numbers, but for symmetric
matrices the eigenvalues λiare real. This can be seen by ﬁrst left multiplying (C.29)
by(u⋆
i)T, where ⋆denotes the complex conjugate, to give

by(u⋆
(u⋆
i)TAui=λi(u⋆
i)Tui. (C.31)
Next we take the complex conjugate of (C.29) and left multiply by uT
ito give
uT
iAu⋆
i=λ⋆
iuT
iu⋆i. (C.32)
where we have used A⋆=Abecause we consider only real matrices A. Taking
the transpose of the second of these equations, and using AT=A, we see that the
left-hand sides of the two equations are equal, and hence that λ⋆
i=λiand so λi
must be real.
The eigenvectors uiof a real symmetric matrix can be chosen to be orthonormal
(i.e., orthogonal and of unit length) so that
uT
iuj=Iij (C.33)

ito give
uT
iAu⋆
i=λ⋆
iuT
uT
where Iijare the elements of the identity matrix I. To show this, we ﬁrst left multiply
(C.29) by uT
jto give
uT
jAui=λiuT
jui (C.34)
and hence, by exchange of indices, we have
uT
iAuj=λjuT
iuj. (C.35)
We now take the transpose of the second equation and make use of the symmetry
property AT=A, and then subtract the two equations to give
(λi−λj)uT
iuj=0. (C.36)
Hence, for λi̸=λj,w eh a v e uT
iuj=0, and hence uiandujare orthogonal. If the

uT
jto give
uT
jAui=λiuT
jui (C.34)
uT
iAuj=λjuT
(λi−λj)uT
two eigenvalues are equal, then any linear combination αui+βujis also an eigen-
vector with the same eigenvalue, so we can select one linear combination arbitrarily,

700 C. PROPERTIES OF MATRICES
and then choose the second to be orthogonal to the ﬁrst (it can be shown that the de-
generate eigenvectors are never linearly dependent). Hence the eigenvectors can bechosen to be orthogonal, and by normalizing can be set to unit length. Because there
areMeigenvalues, the corresponding Morthogonal eigenvectors form a complete
set and so any M-dimensional vector can be expressed as a linear combination of
the eigenvectors.
We can take the eigenvectors u
ito be the columns of an M×Mmatrix U,
which from orthonormality satisﬁes
UTU=I. (C.37)
Such a matrix is said to be orthogonal . Interestingly, the rows of this matrix are also
orthogonal, so that UUT=I. To show this, note that (C.37) implies UTUU−1=
U−1=UTand soUU−1=UUT=I. Using (C.12), it also follows that |U|=1.
The eigenvector equation (C.29) can be expressed in terms of Uin the form
AU=UΛ (C.38)
whereΛis anM×Mdiagonal matrix whose diagonal elements are given by the

If we consider a column vector xthat is transformed by an orthogonal matrix U
to give a new vector
˜x=Ux (C.39)
then the length of the vector is preserved because
˜xT˜x=xTUTUx=xTx (C.40)
and similarly the angle between any two such vectors is preserved because
˜xT˜y=xTUTUy=xTy. (C.41)
Thus, multiplication by Ucan be interpreted as a rigid rotation of the coordinate
system.
From (C.38), it follows that
UTAU=Λ (C.42)
and because Λis a diagonal matrix, we say that the matrix Aisdiagonalized by the

system.
matrixU. If we left multiply by Uand right multiply by UT, we obtain
A=UΛUT(C.43)
Taking the inverse of this equation, and using (C.3) together with U−1=UT,w e
have
A−1=UΛ−1UT. (C.44)

C. PROPERTIES OF MATRICES 701
These last two equations can also be written in the form
A=M∑
i=1λiuiuT
i (C.45)
A−1=M∑
i=11
λiuiuT
i. (C.46)
If we take the determinant of (C.43), and use (C.12), we obtain
|A|=M∏
i=1λi. (C.47)
Similarly, taking the trace of (C.43), and using the cyclic property (C.8) of the trace
operator together with UTU=I,w eh a v e
Tr(A)=M∑
i=1λi. (C.48)
We leave it as an exercise for the reader to verify (C.22) by making use of the results
(C.33), (C.45), (C.46), and (C.47).
A matrix Ais said to be positive deﬁnite , denoted by A≻0,i fwTAw>0for
all values of the vector w. Equivalently, a positive deﬁnite matrix has λi>0for all
of its eigenvalues (as can be seen by setting wto each of the eigenvectors in turn,
and by noting that an arbitrary vector can be expanded as a linear combination of the
eigenvectors). Note that positive deﬁnite is not the same as all the elements beingpositive. For example, the matrix
(
1234)
(C.49)

(
1234)
(C.49)
has eigenvalues λ1≃5.37andλ2≃−0.37. A matrix is said to be positive semidef-
inite ifwTAw⩾0holds for all values of w, which is denoted A⪰0, and is
equivalent to λi⩾0.

Appendix D. Calculus of Variations
We can think of a function y(x)as being an operator that, for any input value x,
returns an output value y. In the same way, we can deﬁne a functional F[y]to be
an operator that takes a function y(x)and returns an output value F. An example of
a functional is the length of a curve drawn in a two-dimensional plane in which the
path of the curve is deﬁned in terms of a function. In the context of machine learning,
a widely used functional is the entropy H[x]for a continuous variable xbecause, for
any choice of probability density function p(x), it returns a scalar value representing
the entropy of xunder that density. Thus the entropy of p(x)could equally well have
been written as H[p].
A common problem in conventional calculus is to ﬁnd a value of xthat max-
imizes (or minimizes) a function y(x). Similarly, in the calculus of variations we
seek a function y(x)that maximizes (or minimizes) a functional F[y]. That is, of all

possible functions y(x), we wish to ﬁnd the particular function for which the func-
tional F[y]is a maximum (or minimum). The calculus of variations can be used, for
instance, to show that the shortest path between two points is a straight line or thatthe maximum entropy distribution is a Gaussian.
If we weren’t familiar with the rules of ordinary calculus, we could evaluate a
conventional derivative dy/dxby making a small change ϵto the variable xand
then expanding in powers of ϵ, so that
y
(x+ϵ)=y(x)+dy
dxϵ+O(ϵ2) (D.1)

y
and ﬁnally taking the limit ϵ→0. Similarly, for a function of several variables
y(x1,...,x D), the corresponding partial derivatives are deﬁned by
y(x1+ϵ1,...,x D+ϵD)=y(x1,...,x D)+D∑
i=1∂y
∂xiϵi+O(ϵ2). (D.2)
The analogous deﬁnition of a functional derivative arises when we consider how
much a functional F[y]changes when we make a small change ϵη(x)to the function
703

704 D. CALCULUS OF V ARIATIONS
Figure D.1 A functional derivative can be deﬁned by
considering how the value of a functional
F[y]changes when the function y(x)is
changed to y(x)+ϵη(x)where η(x)is an
arbitrary function of x.
y(x)
y(x)+ϵη(x)
x
y(x), where η(x)is an arbitrary function of x, as illustrated in Figure D.1. We denote
the functional derivative of E[f]with respect to f(x)byδF/δf (x), and deﬁne it by
the following relation:
F[y(x)+ϵη(x)] =F[y(x)] +ϵ∫δF
δy(x)η(x)dx+O(ϵ2). (D.3)
This can be seen as a natural extension of (D.2) in which F[y]now depends on a
continuous set of variables, namely the values of yat all points x. Requiring that the
functional be stationary with respect to small variations in the function y(x)gives
∫δE
δy(x)η(x)dx=0. (D.4)
Because this must hold for an arbitrary choice of η(x), it follows that the functional
derivative must vanish. To see this, imagine choosing a perturbation η(x)that is zero
everywhere except in the neighbourhood of a point ˆx, in which case the functional

∫δE
derivative must be zero at x=ˆx. However, because this must be true for every
choice ofˆx, the functional derivative must vanish for all values of x.
Consider a functional that is deﬁned by an integral over a function G(y,y′,x)
that depends on both y(x)and its derivative y′(x)as well as having a direct depen-
dence on x
F[y]=∫
G(y(x),y′(x),x)dx (D.5)
where the value of y(x)is assumed to be ﬁxed at the boundary of the region of
integration (which might be at inﬁnity). If we now consider variations in the function
y(x), we obtain
F[y(x)+ϵη(x)] =F[y(x)] +ϵ∫{∂G

dence on x
F[y]=∫
∂yη(x)+∂G
∂y′η′(x)}
dx+O(ϵ2).(D.6)
We now have to cast this in the form (D.3). To do so, we integrate the second term by
parts and make use of the fact that η(x)must vanish at the boundary of the integral
(because y(x)is ﬁxed at the boundary). This gives
∂y−d
dx(∂G
∂y′)}
η(x)dx+O(ϵ2)(D.7)

D. CALCULUS OF V ARIATIONS 705
from which we can read off the functional derivative by comparison with (D.3).
Requiring that the functional derivative vanishes then gives
∂G
∂y−d
dx(∂G
∂y′)
=0 (D.8)
which are known as the Euler-Lagrange equations. For example, if
G=y(x)2+(y′(x))2(D.9)
then the Euler-Lagrange equations take the form
y(x)−d2y
dx2=0. (D.10)
This second order differential equation can be solved for y(x)by making use of the
boundary conditions on y(x).
Often, we consider functionals deﬁned by integrals whose integrands take the
formG(y,x)and that do not depend on the derivatives of y(x). In this case, station-
arity simply requires that ∂G/∂y (x)=0 for all values of x.
If we are optimizing a functional with respect to a probability distribution, then
we need to maintain the normalization constraint on the probabilities. This is often
most conveniently done using a Lagrange multiplier, which then allows an uncon- Appendix E
strained optimization to be performed.

The extension of the above results to a multidimensional variable xis straight-
forward. For a more comprehensive discussion of the calculus of variations, seeSagan (1969).

Appendix E. Lagrange Multipliers
Lagrange multipliers , also sometimes called undetermined multipliers , are used to
ﬁnd the stationary points of a function of several variables subject to one or more
constraints.
Consider the problem of ﬁnding the maximum of a function f(x1,x2)subject to
a constraint relating x1andx2, which we write in the form
g(x1,x2)=0. (E.1)
One approach would be to solve the constraint equation (E.1) and thus express x2as
a function of x1in the form x2=h(x1). This can then be substituted into f(x1,x2)
to give a function of x1alone of the form f(x1,h(x1)). The maximum with respect
tox1could then be found by differentiation in the usual way, to give the stationary
valuex⋆
1, with the corresponding value of x2given by x⋆
2=h(x⋆
1).
One problem with this approach is that it may be difﬁcult to ﬁnd an analytic
solution of the constraint equation that allows x2to be expressed as an explicit func-
tion of x1. Also, this approach treats x1andx2differently and so spoils the natural

valuex⋆
2=h(x⋆
1).
symmetry between these variables.
A more elegant, and often simpler, approach is based on the introduction of a
parameter λcalled a Lagrange multiplier. We shall motivate this technique from
a geometrical perspective. Consider a D-dimensional variable xwith components
x1,...,x D. The constraint equation g(x)=0 then represents a (D−1)-dimensional
surface in x-space as indicated in Figure E.1.
We ﬁrst note that at any point on the constraint surface the gradient ∇g(x)of

the constraint function will be orthogonal to the surface. To see this, consider a point
xthat lies on the constraint surface, and consider a nearby point x+ϵthat also lies
on the surface. If we make a Taylor expansion around x,w eh a v e
g(x+ϵ)≃g(x)+ϵT∇g(x). (E.2)
Because both xandx+ϵlie on the constraint surface, we have g(x)=g(x+ϵ)and
hence ϵT∇g(x)≃0. In the limit ∥ϵ∥→0we have ϵT∇g(x)=0 , and because ϵis
707

708 E. LAGRANGE MULTIPLIERS
Figure E.1 A geometrical picture of the technique of La-
grange multipliers in which we seek to maximize a
function f(x), subject to the constraint g(x)=0 .
IfxisDdimensional, the constraint g(x)=0 cor-
responds to a subspace of dimensionality D−1,
indicated by the red curve. The problem can
be solved by optimizing the Lagrangian function
L(x,λ)=f(x)+λg(x).∇f(x)
∇g(x)xA
g(x)=0
then parallel to the constraint surface g(x)=0 , we see that the vector ∇gis normal
to the surface.
Next we seek a point x⋆on the constraint surface such that f(x)is maximized.
Such a point must have the property that the vector ∇f(x)is also orthogonal to the
constraint surface, as illustrated in Figure E.1, because otherwise we could increase
the value of f(x)by moving a short distance along the constraint surface. Thus ∇f
and∇gare parallel (or anti-parallel) vectors, and so there must exist a parameter λ
such that
∇f+λ∇g=0 (E.3)
where λ̸=0is known as a Lagrange multiplier . Note that λcan have either sign.

such that
At this point, it is convenient to introduce the Lagrangian function deﬁned by
L(x,λ)≡f(x)+λg(x). (E.4)
The constrained stationarity condition (E.3) is obtained by setting ∇xL=0. Fur-
thermore, the condition ∂L/∂λ =0leads to the constraint equation g(x)=0 .
Thus to ﬁnd the maximum of a function f(x)subject to the constraint g(x)=0 ,
we deﬁne the Lagrangian function given by (E.4) and we then ﬁnd the stationary
point of L(x,λ)with respect to both xandλ. For a D-dimensional vector x, this
givesD+1equations that determine both the stationary point x⋆and the value of λ.

If we are only interested in x⋆, then we can eliminate λfrom the stationarity equa-
tions without needing to ﬁnd its value (hence the term ‘undetermined multiplier’).
As a simple example, suppose we wish to ﬁnd the stationary point of the function
f(x1,x2)=1−x2
1−x2
2subject to the constraint g(x1,x2)=x1+x2−1=0 ,a s
illustrated in Figure E.2. The corresponding Lagrangian function is given by
L(x,λ)=1−x2
1−x2
2+λ(x1+x2−1). (E.5)
The conditions for this Lagrangian to be stationary with respect to x1,x2, andλgive

1−x2
1−x2
the following coupled equations:
−2x1+λ=0 (E.6)
−2x2+λ=0 (E.7)
x1+x2−1=0 . (E.8)

E. LAGRANGE MULTIPLIERS 709
Figure E.2 A simple example of the use of Lagrange multipli-
ers in which the aim is to maximize f(x1,x2)=
1−x2
1−x2
2subject to the constraint g(x1,x2)=0
where g(x1,x2)=x1+x2−1. The circles show
contours of the function f(x1,x2), and the diagonal
line shows the constraint surface g(x1,x2)=0 .
g(x1,x2)=0x1x2
(x⋆
1,x⋆
2)
Solution of these equations then gives the stationary point as (x⋆
1,x⋆
2)=(1
2,1
2), and
the corresponding value for the Lagrange multiplier is λ=1.
So far, we have considered the problem of maximizing a function subject to an
equality constraint of the form g(x)=0 . We now consider the problem of maxi-
mizing f(x)subject to an inequality constraint of the form g(x)⩾0, as illustrated
in Figure E.3.
There are now two kinds of solution possible, according to whether the con-
strained stationary point lies in the region where g(x)>0, in which case the con-
straint is inactive , or whether it lies on the boundary g(x)=0 , in which case the

constraint is said to be active . In the former case, the function g(x)plays no role
and so the stationary condition is simply ∇f(x)=0 . This again corresponds to
a stationary point of the Lagrange function (E.4) but this time with λ=0. The
latter case, where the solution lies on the boundary, is analogous to the equality con-
straint discussed previously and corresponds to a stationary point of the Lagrange
function (E.4) with λ̸=0. Now, however, the sign of the Lagrange multiplier is

crucial, because the function f(x)will only be at a maximum if its gradient is ori-
ented away from the region g(x)>0, as illustrated in Figure E.3. We therefore have
∇f(x)=−λ∇g(x)for some value of λ>0.
For either of these two cases, the product λg(x)=0 . Thus the solution to the
Figure E.3 Illustration of the problem of maximizing
f(x)subject to the inequality constraint
g(x)⩾0.∇f(x)
∇g(x)xA
xB
g(x)=0g(x)>0

710 E. LAGRANGE MULTIPLIERS
problem of maximizing f(x)subject to g(x)⩾0is obtained by optimizing the
Lagrange function (E.4) with respect to xandλsubject to the conditions
g(x)⩾0 (E.9)
λ⩾0 (E.10)
λg(x)=0 (E.11)
These are known as the Karush-Kuhn-Tucker (KKT) conditions (Karush, 1939; Kuhn
and Tucker, 1951).
Note that if we wish to minimize (rather than maximize) the function f(x)sub-
ject to an inequality constraint g(x)⩾0, then we minimize the Lagrangian function
L(x,λ)=f(x)−λg(x)with respect to x, again subject to λ⩾0.
Finally, it is straightforward to extend the technique of Lagrange multipliers to
the case of multiple equality and inequality constraints. Suppose we wish to maxi-
mizef(x)subject to gj(x)=0 forj=1,...,J , andhk(x)⩾0fork=1,...,K .
We then introduce Lagrange multipliers {λj}and{µk}, and then optimize the La-
grangian function given by
L(x,{λj},{µk})=f(x)+J∑
j=1λjgj(x)+K∑
k=1µkhk(x) (E.12)
subject to µk⩾0andµkhk(x)=0 fork=1,...,K . Extensions to constrained

functional derivatives are similarly straightforward. For a more detailed discussion Appendix D
of the technique of Lagrange multipliers, see Nocedal and Wright (1999).

REFERENCES 711
References
Abramowitz, M. and I. A. Stegun (1965). Handbook
of Mathematical Functions . Dover.
Adler, S. L. (1981). Over-relaxation method for the
Monte Carlo evaluation of the partition func-tion for multiquadratic actions. Physical Review
D23, 2901–2904.
Ahn, J. H. and J. H. Oh (2003). A constrained EM
algorithm for principal component analysis. Neu-
ral Computation 15(1), 57–65.
Aizerman, M. A., E. M. Braverman, and L. I. Rozo-
noer (1964). The probability problem of pattern
recognition learning and the method of potential
functions. Automation and Remote Control 25,
1175–1190.
Akaike, H. (1974). A new look at statistical model
identiﬁcation. IEEE Transactions on Automatic
Control 19, 716–723.
Ali, S. M. and S. D. Silvey (1966). A general class
of coefﬁcients of divergence of one distributionfrom another. Journal of the Royal Statistical So-
ciety, B 28(1), 131–142.
Allwein, E. L., R. E. Schapire, and Y . Singer (2000).
Reducing multiclass to binary: a unifying ap-

proach for margin classiﬁers. Journal of Machine
Learning Research 1, 113–141.
Amari, S. (1985). Differential-Geometrical Methods
in Statistics . Springer.Amari, S., A. Cichocki, and H. H. Yang (1996). A
new learning algorithm for blind signal separa-
tion. In D. S. Touretzky, M. C. Mozer, and M. E.
Hasselmo (Eds.), Advances in Neural Informa-
tion Processing Systems , V olume 8, pp. 757–763.
MIT Press.
Amari, S. I. (1998). Natural gradient works efﬁ-
ciently in learning. Neural Computation 10,
251–276.

MIT Press.
251–276.
Anderson, J. A. and E. Rosenfeld (Eds.) (1988).
Neurocomputing: F oundations of Research . MIT
Press.
Anderson, T. W. (1963). Asymptotic theory for prin-
cipal component analysis. Annals of Mathemati-
cal Statistics 34, 122–148.
Andrieu, C., N. de Freitas, A. Doucet, and M. I. Jor-
dan (2003). An introduction to MCMC for ma-chine learning. Machine Learning 50, 5–43.
Anthony, M. and N. Biggs (1992). An Introduction
to Computational Learning Theory . Cambridge

251–276.
Press.
Attias, H. (1999a). Independent factor analysis. Neu-
ral Computation 11(4), 803–851.
Attias, H. (1999b). Inferring parameters and struc-
ture of latent variable models by variational
Bayes. In K. B. Laskey and H. Prade (Eds.),

712 REFERENCES
Uncertainty in Artiﬁcial Intelligence: Proceed-
ings of the Fifth Conference , pp. 21–30. Morgan
Kaufmann.
Bach, F. R. and M. I. Jordan (2002). Kernel inde-
pendent component analysis. Journal of Machine
Learning Research 3, 1–48.
Bakir, G. H., J. Weston, and B. Sch ¨olkopf (2004).
Learning to ﬁnd pre-images. In S. Thrun, L. K.
Saul, and B. Sch ¨olkopf (Eds.), Advances in Neu-
ral Information Processing Systems , V olume 16,
pp. 449–456. MIT Press.
Baldi, P. and S. Brunak (2001). Bioinformatics: The
Machine Learning Approach (Second ed.). MIT
Press.
Baldi, P. and K. Hornik (1989). Neural networks
and principal component analysis: learning fromexamples without local minima. Neural Net-
works 2(1), 53–58.
Barber, D. and C. M. Bishop (1997). Bayesian
model comparison by Monte Carlo chaining. In
M. Mozer, M. Jordan, and T. Petsche (Eds.), Ad-
vances in Neural Information Processing Sys-
tems, V olume 9, pp. 333–339. MIT Press.
Barber, D. and C. M. Bishop (1998a). Ensemble

Press.
learning for multi-layer networks. In M. I. Jor-
dan, K. J. Kearns, and S. A. Solla (Eds.), Ad-
vances in Neural Information Processing Sys-tems, V olume 10, pp. 395–401.
Barber, D. and C. M. Bishop (1998b). Ensemble
learning in Bayesian neural networks. In C. M.Bishop (Ed.), Generalization in Neural Networks
and Machine Learning , pp. 215–237. Springer.
Bartholomew, D. J. (1987). Latent V ariable Models
and Factor Analysis . Charles Grifﬁn.
Basilevsky, A. (1994). Statistical Factor Analysis
and Related Methods: Theory and Applications .

Wiley.
Bather, J. (2000). Decision Theory: An Introduction
to Dynamic Programming and Sequential Deci-
sions . Wiley.
Baudat, G. and F. Anouar (2000). Generalized dis-
criminant analysis using a kernel approach. Neu-
ral Computation 12(10), 2385–2404.Baum, L. E. (1972). An inequality and associated
maximization technique in statistical estimationof probabilistic functions of Markov processes.
Inequalities 3, 1–8.
Becker, S. and Y . Le Cun (1989). Improving the con-
vergence of back-propagation learning with sec-

ond order methods. In D. Touretzky, G. E. Hin-
ton, and T. J. Sejnowski (Eds.), Proceedings of
the 1988 Connectionist Models Summer School ,
pp. 29–37. Morgan Kaufmann.
Bell, A. J. and T. J. Sejnowski (1995). An infor-
mation maximization approach to blind separa-
tion and blind deconvolution. Neural Computa-
tion 7(6), 1129–1159.
Bellman, R. (1961). Adaptive Control Processes: A
Guided Tour . Princeton University Press.
Bengio, Y . and P. Frasconi (1995). An input output

HMM architecture. In G. Tesauro, D. S. Touret-
zky, and T. K. Leen (Eds.), Advances in Neural
Information Processing Systems
, V olume 7, pp.
427–434. MIT Press.
Bennett, K. P. (1992). Robust linear programming
discrimination of two linearly separable sets. Op-
timization Methods and Software 1, 23–34.
Berger, J. O. (1985). Statistical Decision Theory and
Bayesian Analysis (Second ed.). Springer.
Bernardo, J. M. and A. F. M. Smith (1994). Bayesian
Theory . Wiley.
Berrou, C., A. Glavieux, and P. Thitimajshima

(1993). Near Shannon limit error-correcting cod-ing and decoding: Turbo-codes (1). In Proceed-
ings ICC’93 , pp. 1064–1070.
Besag, J. (1974). On spatio-temporal models and
Markov ﬁelds. In Transactions of the 7th Prague
Conference on Information Theory, Statistical
Decision Functions and Random Processes , pp.
47–75. Academia.
Besag, J. (1986). On the statistical analysis of dirty
pictures. Journal of the Royal Statistical Soci-
ety B-48 , 259–302.
Besag, J., P. J. Green, D. Hidgon, and K. Megersen
(1995). Bayesian computation and stochastic

systems. Statistical Science 10(1), 3–66.

REFERENCES 713
Bishop, C. M. (1991). A fast procedure for retraining
the multilayer perceptron. International Journal
of Neural Systems 2(3), 229–236.
Bishop, C. M. (1992). Exact calculation of the Hes-
sian matrix for the multilayer perceptron. Neural
Computation 4(4), 494–501.
Bishop, C. M. (1993). Curvature-driven smoothing:
a learning algorithm for feedforward networks.
IEEE Transactions on Neural Networks 4(5),
882–884.
Bishop, C. M. (1994). Novelty detection and neu-
ral network validation. IEE Proceedings: Vision,
Image and Signal Processing 141(4), 217–222.
Special issue on applications of neural networks.
Bishop, C. M. (1995a). Neural Networks for Pattern
Recognition . Oxford University Press.
Bishop, C. M. (1995b). Training with noise is equiv-
alent to Tikhonov regularization. Neural Compu-
tation 7(1), 108–116.
Bishop, C. M. (1999a). Bayesian PCA. In M. S.
Kearns, S. A. Solla, and D. A. Cohn (Eds.), Ad-
tems, V olume 11, pp. 382–388. MIT Press.

Bishop, C. M. (1999b). Variational principal
components. In Proceedings Ninth Interna-
tional Conference on Artiﬁcial Neural Networks,ICANN’99 , V olume 1, pp. 509–514. IEE.
Bishop, C. M. and G. D. James (1993). Analysis of
multiphase ﬂows using dual-energy gamma den-
sitometry and neural networks. Nuclear Instru-
ments and Methods in Physics Research A327 ,
580–593.
Bishop, C. M. and I. T. Nabney (1996). Modelling
conditional probability distributions for periodic
variables. Neural Computation 8(5), 1123–1133.

580–593.
Bishop, C. M. and I. T. Nabney (2008). Pattern
Recognition and Machine Learning: A MatlabCompanion . Springer. In preparation.
Bishop, C. M., D. Spiegelhalter, and J. Winn
(2003). VIBES: A variational inference enginefor Bayesian networks. In S. Becker, S. Thrun,
and K. Obermeyer (Eds.), Advances in NeuralInformation Processing Systems , V olume 15, pp.
793–800. MIT Press.
Bishop, C. M. and M. Svens ´en (2003). Bayesian hi-
erarchical mixtures of experts. In U. Kjaerulff
and C. Meek (Eds.), Proceedings Nineteenth

Conference on Uncertainty in Artiﬁcial Intelli-
gence , pp. 57–64. Morgan Kaufmann.
Bishop, C. M., M. Svens ´en, and G. E. Hinton
(2004). Distinguishing text from graphics in on-
line handwritten ink. In F. Kimura and H. Fu-
jisawa (Eds.), Proceedings Ninth International
Workshop on Frontiers in Handwriting Recogni-
tion, IWFHR-9 , Tokyo, Japan, pp. 142–147.
Bishop, C. M., M. Svens ´en, and C. K. I. Williams
(1996). EM optimization of latent variable den-
sity models. In D. S. Touretzky, M. C. Mozer,
and M. E. Hasselmo (Eds.), Advances in Neural

Information Processing Systems , V olume 8, pp.
465–471. MIT Press.
(1997a). GTM: a principled alternative to the
Self-Organizing Map. In M. C. Mozer, M. I. Jor-
dan, and T. Petche (Eds.), Advances in Neural
Information Processing Systems , V olume 9, pp.
354–360. MIT Press.
(1997b). Magniﬁcation factors for the GTM al-
gorithm. In Proceedings IEE Fifth International
Conference on Artiﬁcial Neural Networks, Cam-

bridge, U.K. , pp. 64–69. Institute of Electrical
Engineers.
(1998a). Developments of the Generative To-
pographic Mapping. Neurocomputing 21, 203–
224.
(1998b). GTM: the Generative TopographicMapping. Neural Computation 10(1), 215–234.
Bishop, C. M. and M. E. Tipping (1998). A hier-
archical latent variable model for data visualiza-tion. IEEE Transactions on Pattern Analysis and
Machine Intelligence 20(3), 281–293.

714 REFERENCES
Bishop, C. M. and J. Winn (2000). Non-linear
Bayesian image modelling. In Proceedings Sixth
European Conference on Computer Vision,
Dublin , V olume 1, pp. 3–17. Springer.
Blei, D. M., M. I. Jordan, and A. Y . Ng (2003). Hi-
erarchical Bayesian models for applications in
information retrieval. In J. M. B. et al. (Ed.),
Bayesian Statistics, 7 , pp. 25–43. Oxford Uni-
Block, H. D. (1962). The perceptron: a model
for brain functioning. Reviews of Modern
Physics 34(1), 123–135. Reprinted in Anderson
and Rosenfeld (1988).
Blum, J. A. (1965). Multidimensional stochastic ap-
proximation methods. Annals of Mathematical
Statistics 25, 737–744.
Bodlaender, H. (1993). A tourist guide through
treewidth. Acta Cybernetica 11, 1–21.
Boser, B. E., I. M. Guyon, and V . N. Vapnik (1992).
A training algorithm for optimal margin classi-
ﬁers. In D. Haussler (Ed.), Proceedings Fifth An-
nual Workshop on Computational Learning The-ory (COLT) , pp. 144–152. ACM.

Bourlard, H. and Y . Kamp (1988). Auto-association
by multilayer perceptrons and singular value de-
composition. Biological Cybernetics 59, 291–
294.
Box, G. E. P., G. M. Jenkins, and G. C. Reinsel
(1994). Time Series Analysis . Prentice Hall.
Box, G. E. P. and G. C. Tao (1973). Bayesian Infer-
ence in Statistical Analysis . Wiley.
Boyd, S. and L. Vandenberghe (2004). Convex Opti-
mization . Cambridge University Press.
Boyen, X. and D. Koller (1998). Tractable inference
for complex stochastic processes. In G. F. Cooper

294.
and S. Moral (Eds.), Proceedings 14th Annual
Conference on Uncertainty in Artiﬁcial Intelli-gence (UAI) , pp. 33–42. Morgan Kaufmann.
Boykov, Y ., O. Veksler, and R. Zabih (2001). Fast
approximate energy minimization via graph cuts.IEEE Transactions on Pattern Analysis and Ma-
chine Intelligence 23(11), 1222–1239.Breiman, L. (1996). Bagging predictors. Machine
Learning 26, 123–140.
Breiman, L., J. H. Friedman, R. A. Olshen, and
P. J. Stone (1984). Classiﬁcation and Regression
Trees . Wadsworth.
Brooks, S. P. (1998). Markov chain Monte

Carlo method and its application. The Statisti-
cian 47(1), 69–100.
Broomhead, D. S. and D. Lowe (1988). Multivari-
able functional interpolation and adaptive net-
works. Complex Systems 2, 321–355.
Buntine, W. and A. Weigend (1991). Bayesian back-
propagation. Complex Systems 5, 603–643.
Buntine, W. L. and A. S. Weigend (1993). Com-
puting second derivatives in feed-forward net-
works: a review. IEEE Transactions on Neural
Networks 5(3), 480–488.
Burges, C. J. C. (1998). A tutorial on support vec-

tor machines for pattern recognition. Knowledge
Discovery and Data Mining 2(2), 121–167.
Cardoso, J.-F. (1998). Blind signal separation: statis-
tical principles. Proceedings of the IEEE 9(10),
2009–2025.
Casella, G. and R. L. Berger (2002). Statistical In-
ference (Second ed.). Duxbury.
Castillo, E., J. M. Guti ´errez, and A. S. Hadi (1997).
Expert Systems and Probabilistic Network Mod-
els. Springer.
Chan, K., T. Lee, and T. J. Sejnowski (2003). Vari-
ational Bayesian learning of ICA with missing

data. Neural Computation 15(8), 1991–2011.
Chen, A. M., H. Lu, and R. Hecht-Nielsen (1993).
On the geometry of feedforward neural network
error surfaces. Neural Computation 5(6), 910–
927.
Chen, M. H., Q. M. Shao, and J. G. Ibrahim (Eds.)
(2001). Monte Carlo Methods for Bayesian Com-
putation . Springer.
Chen, S., C. F. N. Cowan, and P. M. Grant (1991).
Orthogonal least squares learning algorithm forradial basis function networks. IEEE Transac-
tions on Neural Networks 2(2), 302–309.

REFERENCES 715
Choudrey, R. A. and S. J. Roberts (2003). Variational
mixture of Bayesian independent component an-alyzers. Neural Computation 15(1), 213–252.
Clifford, P. (1990). Markov random ﬁelds in statis-
tics. In G. R. Grimmett and D. J. A. Welsh (Eds.),
Disorder in Physical Systems. A V olume in Hon-
our of John M. Hammersley , pp. 19–32. Oxford
Collins, M., S. Dasgupta, and R. E. Schapire (2002).
A generalization of principal component analy-
sis to the exponential family. In T. G. Dietterich,
S. Becker, and Z. Ghahramani (Eds.), Advances
in Neural Information Processing Systems , V ol-
ume 14, pp. 617–624. MIT Press.
Comon, P., C. Jutten, and J. Herault (1991). Blind
source separation, 2: problems statement. Signal
Processing 24(1), 11–20.
Corduneanu, A. and C. M. Bishop (2001). Vari-
ational Bayesian model selection for mixturedistributions. In T. Richardson and T. Jaakkola
(Eds.), Proceedings Eighth International Confer-
ence on Artiﬁcial Intelligence and Statistics , pp.

27–34. Morgan Kaufmann.
Cormen, T. H., C. E. Leiserson, R. L. Rivest, and
C. Stein (2001). Introduction to Algorithms (Sec-
ond ed.). MIT Press.
Cortes, C. and V . N. Vapnik (1995). Support vector
networks. Machine Learning 20, 273–297.
Cotter, N. E. (1990). The Stone-Weierstrass theo-
rem and its application to neural networks. IEEE
Transactions on Neural Networks 1(4), 290–295.
Cover, T. and P. Hart (1967). Nearest neighbor pat-
tern classiﬁcation. IEEE Transactions on Infor-
mation Theory IT-11 , 21–27.

Cover, T. M. and J. A. Thomas (1991). Elements of
Information Theory . Wiley.
Cowell, R. G., A. P. Dawid, S. L. Lauritzen, and D. J.
Spiegelhalter (1999). Probabilistic Networks and
Expert Systems . Springer.
Cox, R. T. (1946). Probability, frequency and
reasonable expectation. American Journal of
Physics 14(1), 1–13.Cox, T. F. and M. A. A. Cox (2000). Multidimen-
sional Scaling (Second ed.). Chapman and Hall.
Cressie, N. (1993). Statistics for Spatial Data . Wiley.

Cristianini, N. and J. Shawe-Taylor (2000). Support
vector machines and other kernel-based learning
methods . Cambridge University Press.
Csat´o, L. and M. Opper (2002). Sparse on-line Gaus-
sian processes. Neural Computation 14(3), 641–
668.
Csisz `ar, I. and G. Tusn `ady (1984). Information ge-
ometry and alternating minimization procedures.
Statistics and Decisions 1(1), 205–237.
Cybenko, G. (1989). Approximation by superposi-
tions of a sigmoidal function. Mathematics of
Control, Signals and Systems 2, 304–314.

668.
Dawid, A. P. (1979). Conditional independence in
statistical theory (with discussion). Journal of the
Royal Statistical Society, Series B 4, 1–31.
Dawid, A. P. (1980). Conditional independence for
statistical operations. Annals of Statistics 8, 598–
617.
deFinetti, B. (1970). Theory of Probability . Wiley
and Sons.
Dempster, A. P., N. M. Laird, and D. B. Rubin
(1977). Maximum likelihood from incompletedata via the EM algorithm. Journal of the Royal
Statistical Society, B 39(1), 1–38.
Denison, D. G. T., C. C. Holmes, B. K. Mallick,

617.
and Sons.
and A. F. M. Smith (2002). Bayesian Methods for
Nonlinear Classiﬁcation and Regression . Wiley.
Diaconis, P. and L. Saloff-Coste (1998). What do we
know about the Metropolis algorithm? Journal
of Computer and System Sciences 57, 20–36.
Dietterich, T. G. and G. Bakiri (1995). Solving
multiclass learning problems via error-correctingoutput codes. Journal of Artiﬁcial Intelligence
Research 2, 263–286.
Duane, S., A. D. Kennedy, B. J. Pendleton, and
D. Roweth (1987). Hybrid Monte Carlo. Physics
Letters B 195(2), 216–222.

Duda, R. O. and P. E. Hart (1973). Pattern Classiﬁ-
cation and Scene Analysis . Wiley.

716 REFERENCES
Duda, R. O., P. E. Hart, and D. G. Stork (2001). Pat-
tern Classiﬁcation (Second ed.). Wiley.
Durbin, R., S. Eddy, A. Krogh, and G. Mitchi-
son (1998). Biological Sequence Analysis . Cam-
Dybowski, R. and S. Roberts (2005). An anthology
of probabilistic models for medical informatics.
In D. Husmeier, R. Dybowski, and S. Roberts
(Eds.), Probabilistic Modeling in Bioinformatics
and Medical Informatics , pp. 297–349. Springer.
Efron, B. (1979). Bootstrap methods: another look
at the jackknife. Annals of Statistics 7, 1–26.
Elkan, C. (2003). Using the triangle inequality to ac-
celerate k-means. In Proceedings of the Twelfth
International Conference on Machine Learning ,
pp. 147–153. AAAI.
Elliott, R. J., L. Aggoun, and J. B. Moore (1995).
Hidden Markov Models: Estimation and Con-
trol. Springer.
Ephraim, Y ., D. Malah, and B. H. Juang (1989).
On the application of hidden Markov models forenhancing noisy speech. IEEE Transactions on

Acoustics, Speech and Signal Processing 37(12),
1846–1856.
Erwin, E., K. Obermayer, and K. Schulten (1992).
Self-organizing maps: ordering, convergence
properties and energy functions. Biological Cy-
bernetics 67, 47–55.
Everitt, B. S. (1984). An Introduction to Latent V ari-
able Models . Chapman and Hall.
Faul, A. C. and M. E. Tipping (2002). Analysis of
sparse Bayesian learning. In T. G. Dietterich,
ume 14, pp. 383–389. MIT Press.

Feller, W. (1966). An Introduction to Probability
Theory and its Applications (Second ed.), V ol-
ume 2. Wiley.
Feynman, R. P., R. B. Leighton, and M. Sands
(1964). The Feynman Lectures of Physics , V ol-
ume Two. Addison-Wesley. Chapter 19.Fletcher, R. (1987). Practical Methods of Optimiza-
tion(Second ed.). Wiley.
Forsyth, D. A. and J. Ponce (2003). Computer Vi-
sion: A Modern Approach . Prentice Hall.
Freund, Y . and R. E. Schapire (1996). Experiments
with a new boosting algorithm. In L. Saitta (Ed.),

Thirteenth International Conference on Machine
Learning , pp. 148–156. Morgan Kaufmann.
Frey, B. J. (1998). Graphical Models for Ma-
chine Learning and Digital Communication .
MIT Press.
Frey, B. J. and D. J. C. MacKay (1998). A revolu-
tion: Belief propagation in graphs with cycles. In
M. I. Jordan, M. J. Kearns, and S. A. Solla (Eds.),Advances in Neural Information Processing Sys-
tems, V olume 10. MIT Press.
Friedman, J. H. (2001). Greedy function approxi-
mation: a gradient boosting machine. Annals of

MIT Press.
Statistics 29(5), 1189–1232.
Friedman, J. H., T. Hastie, and R. Tibshirani (2000).
Additive logistic regression: a statistical view of
boosting. Annals of Statistics 28, 337–407.
Friedman, N. and D. Koller (2003). Being Bayesian
about network structure: A Bayesian approach
to structure discovery in Bayesian networks. Ma-
chine Learning 50, 95–126.
Frydenberg, M. (1990). The chain graph Markov
property. Scandinavian Journal of Statistics 17,
333–353.
Fukunaga, K. (1990). Introduction to Statistical Pat-

333–353.
tern Recognition (Second ed.). Academic Press.
Funahashi, K. (1989). On the approximate realiza-
tion of continuous mappings by neural networks.
Neural Networks 2(3), 183–192.
Fung, R. and K. C. Chang (1990). Weighting and
integrating evidence for stochastic simulation in
Bayesian networks. In P. P. Bonissone, M. Hen-rion, L. N. Kanal, and J. F. Lemmer (Eds.), Un-
certainty in Artiﬁcial Intelligence , V olume 5, pp.
208–219. Elsevier.
Gallager, R. G. (1963). Low-Density Parity-Check
Codes . MIT Press.

REFERENCES 717
Gamerman, D. (1997). Markov Chain Monte Carlo:
Stochastic Simulation for Bayesian Inference .
Chapman and Hall.
Gelman, A., J. B. Carlin, H. S. Stern, and D. B. Ru-
bin (2004). Bayesian Data Analysis (Second ed.).
Geman, S. and D. Geman (1984). Stochastic re-
laxation, Gibbs distributions, and the Bayesian
restoration of images. IEEE Transactions on Pat-
tern Analysis and Machine Intelligence 6(1),
721–741.
Ghahramani, Z. and M. J. Beal (2000). Variational
inference for Bayesian mixtures of factor ana-
lyzers. In S. A. Solla, T. K. Leen, and K. R.
M¨uller (Eds.), Advances in Neural Information
Processing Systems , V olume 12, pp. 449–455.
MIT Press.
Ghahramani, Z. and G. E. Hinton (1996a). The
EM algorithm for mixtures of factor analyzers.
Technical Report CRG-TR-96-1, University of
Toronto.
Ghahramani, Z. and G. E. Hinton (1996b). Param-
eter estimation for linear dynamical systems.Technical Report CRG-TR-96-2, University of
Toronto.

MIT Press.
Toronto.
Toronto.
Ghahramani, Z. and G. E. Hinton (1998). Variational
learning for switching state-space models. Neu-
ral Computation 12(4), 963–996.
Ghahramani, Z. and M. I. Jordan (1994). Super-
vised learning from incomplete data via an EMappproach. In J. D. Cowan, G. T. Tesauro, and
J. Alspector (Eds.), Advances in Neural Informa-
tion Processing Systems , V olume 6, pp. 120–127.
Ghahramani, Z. and M. I. Jordan (1997). Factorial
hidden Markov models. Machine Learning 29,
245–275.

Toronto.
245–275.
Gibbs, M. N. (1997). Bayesian Gaussian processes
for regression and classiﬁcation . Phd thesis, Uni-
versity of Cambridge.
Gibbs, M. N. and D. J. C. MacKay (2000). Varia-
tional Gaussian process classiﬁers. IEEE Trans-
actions on Neural Networks 11, 1458–1464.Gilks, W. R. (1992). Derivative-free adaptive
rejection sampling for Gibbs sampling. InJ. Bernardo, J. Berger, A. P. Dawid, and A. F. M.
Smith (Eds.), Bayesian Statistics , V olume 4. Ox-
Gilks, W. R., N. G. Best, and K. K. C. Tan (1995).

Adaptive rejection Metropolis sampling. Applied
Statistics 44, 455–472.
Gilks, W. R., S. Richardson, and D. J. Spiegelhal-
ter (Eds.) (1996). Markov Chain Monte Carlo in
Practice . Chapman and Hall.
Gilks, W. R. and P. Wild (1992). Adaptive rejection
sampling for Gibbs sampling. Applied Statis-
tics 41, 337–348.
Gill, P. E., W. Murray, and M. H. Wright (1981).
Practical Optimization . Academic Press.
Goldberg, P. W., C. K. I. Williams, and C. M.
Bishop (1998). Regression with input-dependent
noise: A Gaussian process treatment. In Ad-

tems, V olume 10, pp. 493–499. MIT Press.
Golub, G. H. and C. F. Van Loan (1996).
Matrix
Computations (Third ed.). John Hopkins Univer-
Good, I. (1950). Probability and the Weighing of Ev-
idence . Hafners.
Gordon, N. J., D. J. Salmond, and A. F. M. Smith
(1993). Novel approach to nonlinear/non-
Gaussian Bayesian state estimation. IEE
Proceedings-F 140(2), 107–113.
Graepel, T. (2003). Solving noisy linear operator
equations by Gaussian processes: Application

Matrix
to ordinary and partial differential equations. InProceedings of the Twentieth International Con-
ference on Machine Learning , pp. 234–241.
Greig, D., B. Porteous, and A. Seheult (1989). Ex-
act maximum a-posteriori estimation for binary
images. Journal of the Royal Statistical Society,
Series B 51(2), 271–279.
Gull, S. F. (1989). Developments in maximum en-
tropy data analysis. In J. Skilling (Ed.), Maxi-
mum Entropy and Bayesian Methods , pp. 53–71.
Kluwer.

718 REFERENCES
Hassibi, B. and D. G. Stork (1993). Second order
derivatives for network pruning: optimal brainsurgeon. In S. J. Hanson, J. D. Cowan, and
C. L. Giles (Eds.), Advances in Neural Informa-
tion Processing Systems , V olume 5, pp. 164–171.
Hastie, T. and W. Stuetzle (1989). Principal curves.
Journal of the American Statistical Associa-tion 84(106), 502–516.
Hastie, T., R. Tibshirani, and J. Friedman (2001).
The Elements of Statistical Learning . Springer.
Hastings, W. K. (1970). Monte Carlo sampling
methods using Markov chains and their applica-
tions. Biometrika 57, 97–109.
Hathaway, R. J. (1986). Another interpretation of the
EM algorithm for mixture distributions. Statistics
and Probability Letters 4, 53–56.
Haussler, D. (1999). Convolution kernels on discrete
structures. Technical Report UCSC-CRL-99-10,University of California, Santa Cruz, Computer
Science Department.
Henrion, M. (1988). Propagation of uncertainty by

logic sampling in Bayes’ networks. In J. F. Lem-mer and L. N. Kanal (Eds.), Uncertainty in Arti-
ﬁcial Intelligence , V olume 2, pp. 149–164. North
Holland.
Herbrich, R. (2002). Learning Kernel Classiﬁers .
MIT Press.
Hertz, J., A. Krogh, and R. G. Palmer (1991). In-
troduction to the Theory of Neural Computation .
Addison Wesley.
Hinton, G. E., P. Dayan, and M. Revow (1997).
Modelling the manifolds of images of handwrit-ten digits. IEEE Transactions on Neural Net-
works 8(1), 65–74.
Hinton, G. E. and D. van Camp (1993). Keeping

Holland.
MIT Press.
neural networks simple by minimizing the de-scription length of the weights. In Proceedings of
the Sixth Annual Conference on Computational
Learning Theory , pp. 5–13. ACM.
Hinton, G. E., M. Welling, Y . W. Teh, and S. Osin-
dero (2001). A new view of ICA. In Proceedingsof the International Conference on Independent
Component Analysis and Blind Signal Separa-tion, V olume 3.
Hodgson, M. E. (1998). Reducing computational re-
quirements of the minimum-distance classiﬁer.
Remote Sensing of Environments 25, 117–128.
Hoerl, A. E. and R. Kennard (1970). Ridge regres-

sion: biased estimation for nonorthogonal prob-
lems. Technometrics 12, 55–67.
Hofmann, T. (2000). Learning the similarity of doc-
uments: an information-geometric approach to
document retrieval and classiﬁcation. In S. A.Solla, T. K. Leen, and K. R. M ¨uller (Eds.), Ad-
tems, V olume 12, pp. 914–920. MIT Press.
Hojen-Sorensen, P. A., O. Winther, and L. K. Hansen
(2002). Mean ﬁeld approaches to independent
component analysis. Neural Computation 14(4),
889–918.

889–918.
Hornik, K. (1991). Approximation capabilities of
multilayer feedforward networks. Neural Net-
works 4(2), 251–257.
Hornik, K., M. Stinchcombe, and H. White (1989).
Multilayer feedforward networks are universalapproximators. Neural Networks 2
(5), 359–366.
Hotelling, H. (1933). Analysis of a complex of statis-
tical variables into principal components. Jour-
nal of Educational Psychology 24, 417–441.
Hotelling, H. (1936). Relations between two sets of
variables. Biometrika 28, 321–377.

889–918.
Hyv¨arinen, A. and E. Oja (1997). A fast ﬁxed-point
algorithm for independent component analysis.
Neural Computation 9(7), 1483–1492.
Isard, M. and A. Blake (1998). CONDENSATION
– conditional density propagation for visualtracking. International Journal of Computer Vi-
sion 29(1), 5–18.
Ito, Y . (1991). Representation of functions by su-
perpositions of a step or sigmoid function andtheir applications to neural network theory. Neu-
ral Networks 4(3), 385–394.

REFERENCES 719
Jaakkola, T. and M. I. Jordan (2000). Bayesian
parameter estimation via variational methods.Statistics and Computing 10, 25–37.
Jaakkola, T. S. (2001). Tutorial on variational ap-
proximation methods. In M. Opper and D. Saad
(Eds.), Advances in Mean Field Methods , pp.
129–159. MIT Press.
Jaakkola, T. S. and D. Haussler (1999). Exploiting
generative models in discriminative classiﬁers. In
M. S. Kearns, S. A. Solla, and D. A. Cohn (Eds.),
Advances in Neural Information Processing Sys-
tems, V olume 11. MIT Press.
Jacobs, R. A., M. I. Jordan, S. J. Nowlan, and G. E.
Hinton (1991). Adaptive mixtures of local ex-
perts. Neural Computation 3(1), 79–87.
Jaynes, E. T. (2003). Probability Theory: The Logic
of Science . Cambridge University Press.
Jebara, T. (2004). Machine Learning: Discrimina-
tive and Generative . Kluwer.
Jeffries, H. (1946). An invariant form for the prior
probability in estimation problems. Pro. Roy.
Soc. AA 186, 453–461.
Jelinek, F. (1997). Statistical Methods for Speech

Recognition . MIT Press.
Jensen, C., A. Kong, and U. Kjaerulff (1995). Block-
ing gibbs sampling in very large probabilistic
expert systems. International Journal of Human
Computer Studies. Special Issue on Real-World
Applications of Uncertain Reasoning. 42, 647–
666.
Jensen, F. V . (1996). An Introduction to Bayesian
Networks . UCL Press.
Jerrum, M. and A. Sinclair (1996). The Markov
chain Monte Carlo method: an approach to ap-proximate counting and integration. In D. S.

666.
Hochbaum (Ed.), Approximation Algorithms for
NP-Hard Problems . PWS Publishing.
Jolliffe, I. T. (2002). Principal Component Analysis
(Second ed.). Springer.
Jordan, M. I. (1999). Learning in Graphical Models .
MIT Press.Jordan, M. I. (2007). An Introduction to Probabilis-
tic Graphical Models . In preparation.
Jordan, M. I., Z. Ghahramani, T. S. Jaakkola, and
L. K. Saul (1999). An introduction to variational
methods for graphical models. In M. I. Jordan
(Ed.), Learning in Graphical Models , pp. 105–
162. MIT Press.

Jordan, M. I. and R. A. Jacobs (1994). Hierarchical
mixtures of experts and the EM algorithm. Neu-
ral Computation 6(2), 181–214.
Jutten, C. and J. Herault (1991). Blind separation of
sources, 1: An adaptive algorithm based on neu-romimetic architecture. Signal Processing 24(1),
1–10.
Kalman, R. E. (1960). A new approach to linear ﬁl-
tering and prediction problems. Transactions of
the American Society for Mechanical Engineer-
ing, Series D, Journal of Basic Engineering 82,
35–45.
Kambhatla, N. and T. K. Leen (1997). Dimension

1–10.
35–45.
reduction by local principal component analysis.
Neural Computation 9(7), 1493–1516.
Kanazawa, K., D. Koller, and S. Russel (1995).
Stochastic simulation algorithms for dynamicprobabilistic networks. In Uncertainty in Artiﬁ-
cial Intelligence , V olume 11. Morgan Kaufmann.
Kapadia, S. (1998). Discriminative Training of Hid-
den Markov Models . Phd thesis, University of
Cambridge, U.K.
Kapur, J. (1989). Maximum entropy methods in sci-
ence and engineering . Wiley.
Karush, W. (1939). Minima of functions of several

variables with inequalities as side constraints.
Master’s thesis, Department of Mathematics,University of Chicago.
Kass, R. E. and A. E. Raftery (1995). Bayes fac-
tors. Journal of the American Statistical Associ-
ation 90, 377–395.
Kearns, M. J. and U. V . Vazirani (1994). An Intro-
duction to Computational Learning Theory .M I T
Press.

720 REFERENCES
Kindermann, R. and J. L. Snell (1980). Markov Ran-
dom Fields and Their Applications . American
Mathematical Society.
Kittler, J. and J. F ¨oglein (1984). Contextual classiﬁ-
cation of multispectral pixel data. Image and Vi-
sion Computing 2, 13–29.
Kohonen, T. (1982). Self-organized formation of
topologically correct feature maps. Biological
Cybernetics 43, 59–69.
Kohonen, T. (1995). Self-Organizing Maps .
Springer.
Kolmogorov, V . and R. Zabih (2004). What en-
ergy functions can be minimized via graph cuts?IEEE Transactions on Pattern Analysis and Ma-
chine Intelligence 26(2), 147–159.
Kreinovich, V . Y . (1991). Arbitrary nonlinearity is
sufﬁcient to represent all functions by neural net-works: a theorem. Neural Networks 4(3), 381–
383.
Krogh, A., M. Brown, I. S. Mian, K. Sj ¨olander, and
D. Haussler (1994). Hidden Markov models incomputational biology: Applications to protein
modelling. Journal of Molecular Biology 235,
1501–1531.
Kschischnang, F. R., B. J. Frey, and H. A. Loeliger

383.
(2001). Factor graphs and the sum-product algo-
rithm. IEEE Transactions on Information The-
ory47(2), 498–519.
Kuhn, H. W. and A. W. Tucker (1951). Nonlinear
programming. In Proceedings of the 2nd Berke-
ley Symposium on Mathematical Statistics and
Probabilities , pp. 481–492. University of Cali-
fornia Press.
Kullback, S. and R. A. Leibler (1951). On infor-
mation and sufﬁciency. Annals of Mathematical
Statistics 22(1), 79–86.
K˙urkov ´a, V . and P. C. Kainen (1994). Functionally
equivalent feed-forward neural networks. Neural
Computation 6(3), 543–558.

Kuss, M. and C. Rasmussen (2006). Assessing ap-
proximations for Gaussian process classiﬁcation.InAdvances in Neural Information Processing
Systems , Number 18. MIT Press. in press.
Lasserre, J., C. M. Bishop, and T. Minka (2006).
Principled hybrids of generative and discrimina-tive models. In Proceedings 2006 IEEE Confer-
ence on Computer Vision and Pattern Recogni-
tion, New York .
Lauritzen, S. and N. Wermuth (1989). Graphical
models for association between variables, someof which are qualitative some quantitative. An-

nals of Statistics 17, 31–57.
Lauritzen, S. L. (1992). Propagation of probabilities,
means and variances in mixed graphical associa-
tion models. Journal of the American Statistical
Association 87, 1098–1108.
Lauritzen, S. L. (1996).
Graphical Models . Oxford
Lauritzen, S. L. and D. J. Spiegelhalter (1988). Lo-
cal computations with probabailities on graphical
structures and their application to expert systems.Journal of the Royal Statistical Society 50, 157–
224.
Lawley, D. N. (1953). A modiﬁed method of esti-

224.
mation in factor analysis and some large sam-
ple results. In Uppsala Symposium on Psycho-
logical Factor Analysis , Number 3 in Nordisk
Psykologi Monograph Series, pp. 35–42. Upp-
sala: Almqvist and Wiksell.
Lawrence, N. D., A. I. T. Rowstron, C. M. Bishop,
and M. J. Taylor (2002). Optimising synchro-nisation times for mobile devices. In T. G. Di-
etterich, S. Becker, and Z. Ghahramani (Eds.),
tems, V olume 14, pp. 1401–1408. MIT Press.

Lazarsfeld, P. F. and N. W. Henry (1968). Latent
Structure Analysis . Houghton Mifﬂin.
Le Cun, Y ., B. Boser, J. S. Denker, D. Henderson,
R. E. Howard, W. Hubbard, and L. D. Jackel
(1989). Backpropagation applied to handwrittenzip code recognition. Neural Computation 1(4),
541–551.
Le Cun, Y ., J. S. Denker, and S. A. Solla (1990).
Optimal brain damage. In D. S. Touretzky (Ed.),

REFERENCES 721
tems, V olume 2, pp. 598–605. Morgan Kauf-
mann.
LeCun, Y ., L. Bottou, Y . Bengio, and P. Haffner
(1998). Gradient-based learning applied to doc-
ument recognition. Proceedings of the IEEE 86,
2278–2324.
Lee, Y ., Y . Lin, and G. Wahba (2001). Multicategory
support vector machines. Technical Report 1040,
Department of Statistics, University of Madison,Wisconsin.
Leen, T. K. (1995). From data distributions to regu-
larization in invariant learning. Neural Computa-
tion 7, 974–981.
Lindley, D. V . (1982). Scoring rules and the in-
evitability of probability. International Statisti-
cal Review 50, 1–26.
Liu, J. S. (Ed.) (2001). Monte Carlo Strategies in
Scientiﬁc Computing . Springer.
Lloyd, S. P. (1982). Least squares quantization in
PCM. IEEE Transactions on Information The-
ory28(2), 129–137.
L¨utkepohl, H. (1996). Handbook of Matrices . Wiley.
MacKay, D. J. C. (1992a). Bayesian interpolation.
Neural Computation 4(3), 415–447.

MacKay, D. J. C. (1992b). The evidence framework
applied to classiﬁcation networks. Neural Com-
putation 4(5), 720–736.
MacKay, D. J. C. (1992c). A practical Bayesian
framework for back-propagation networks. Neu-
ral Computation 4(3), 448–472.
MacKay, D. J. C. (1994). Bayesian methods for
backprop networks. In E. Domany, J. L. van
Hemmen, and K. Schulten (Eds.), Models of
Neural Networks, III , Chapter 6, pp. 211–254.
Springer.
MacKay, D. J. C. (1995). Bayesian neural networks
and density networks. Nuclear Instruments and

Springer.
Methods in Physics Research, A 354(1), 73–80.
MacKay, D. J. C. (1997). Ensemble learning for hid-
den Markov models. Unpublished manuscript,Department of Physics, University of Cam-
bridge.
MacKay, D. J. C. (1998). Introduction to Gaus-
sian processes. In C. M. Bishop (Ed.), Neural
Networks and Machine Learning , pp. 133–166.
Springer.
MacKay, D. J. C. (1999). Comparison of approx-
imate methods for handling hyperparameters.
Neural Computation 11(5), 1035–1068.
MacKay, D. J. C. (2003). Information Theory, Infer-

bridge.
Springer.
ence and Learning Algorithms . Cambridge Uni-
MacKay, D. J. C. and M. N. Gibbs (1999). Den-
sity networks. In J. W. Kay and D. M. Tittering-ton (Eds.), Statistics and Neural Networks: Ad-
vances at the Interface , Chapter 5, pp. 129–145.
MacKay, D. J. C. and R. M. Neal (1999). Good error-
correcting codes based on very sparse matrices.
IEEE Transactions on Information Theory
45,
399–431.
MacQueen, J. (1967). Some methods for classiﬁca-
tion and analysis of multivariate observations. In

45,
399–431.
L. M. LeCam and J. Neyman (Eds.), Proceed-
ings of the Fifth Berkeley Symposium on Mathe-matical Statistics and Probability , V olume I, pp.
281–297. University of California Press.
Magnus, J. R. and H. Neudecker (1999). Matrix Dif-
ferential Calculus with Applications in Statisticsand Econometrics . Wiley.
Mallat, S. (1999). A Wavelet Tour of Signal Process-
ing(Second ed.). Academic Press.
Manning, C. D. and H. Sch ¨utze (1999). F oundations
of Statistical Natural Language Processing . MIT
Press.

Press.
Mardia, K. V . and P. E. Jupp (2000). Directional
Statistics . Wiley.
Maybeck, P. S. (1982). Stochastic models, estima-
tion and control . Academic Press.
McAllester, D. A. (2003). PAC-Bayesian stochastic
model selection. Machine Learning 51(1), 5–21.

722 REFERENCES
McCullagh, P. and J. A. Nelder (1989). Generalized
Linear Models (Second ed.). Chapman and Hall.
McCulloch, W. S. and W. Pitts (1943). A logical
calculus of the ideas immanent in nervous ac-
tivity. Bulletin of Mathematical Biophysics 5,
115–133. Reprinted in Anderson and Rosenfeld
(1988).
McEliece, R. J., D. J. C. MacKay, and J. F. Cheng
(1998). Turbo decoding as an instance of Pearl’s
‘Belief Ppropagation’ algorithm. IEEE Journal
on Selected Areas in Communications 16, 140–
152.
McLachlan, G. J. and K. E. Basford (1988). Mixture
Models: Inference and Applications to Cluster-
ing. Marcel Dekker.
McLachlan, G. J. and T. Krishnan (1997). The EM
Algorithm and its Extensions . Wiley.
McLachlan, G. J. and D. Peel (2000). Finite Mixture
Models . Wiley.
Meng, X. L. and D. B. Rubin (1993). Maximum like-
lihood estimation via the ECM algorithm: a gen-
eral framework. Biometrika 80, 267–278.
Metropolis, N., A. W. Rosenbluth, M. N. Rosen-

152.
bluth, A. H. Teller, and E. Teller (1953). Equa-tion of state calculations by fast computing
machines. Journal of Chemical Physics 21(6),
Metropolis, N. and S. Ulam (1949). The Monte
Carlo method. Journal of the American Statisti-
cal Association 44(247), 335–341.
Mika, S., G. R ¨atsch, J. Weston, and B. Sch ¨olkopf
(1999). Fisher discriminant analysis with ker-
nels. In Y . H. Hu, J. Larsen, E. Wilson, andS. Douglas (Eds.), Neural Networks for Signal
Processing IX , pp. 41–48. IEEE.

Minka, T. (2001a). Expectation propagation for ap-
proximate Bayesian inference. In J. Breese and
D. Koller (Eds.), Proceedings of the Seventeenth
gence , pp. 362–369. Morgan Kaufmann.
Minka, T. (2001b). A family of approximate al-
gorithms for Bayesian inference . Ph. D. thesis,
MIT.Minka, T. (2004). Power EP. Technical Report
MSR-TR-2004-149, Microsoft Research Cam-bridge.
Minka, T. (2005). Divergence measures and mes-
sage passing. Technical Report MSR-TR-2005-

173, Microsoft Research Cambridge.
Minka, T. P. (2001c). Automatic choice of dimen-
sionality for PCA. In T. K. Leen, T. G. Diet-
terich, and V . Tresp (Eds.), Advances in Neural
Information Processing Systems , V olume 13, pp.
598–604. MIT Press.
Minsky, M. L. and S. A. Papert (1969). Perceptrons .
MIT Press. Expanded edition 1990.
Miskin, J. W. and D. J. C. MacKay (2001). Ensem-
ble learning for blind source separation. In S. J.
Roberts and R. M. Everson (Eds.), Independent
Component Analysis: Principles and Practice .

Møller, M. (1993). Efﬁcient Training of Feed-
Forward Neural Networks. Ph. D. thesis, Aarhus
University, Denmark.
Moody, J. and C. J. Darken (1989). Fast learning in
networks of locally-tuned processing units. Neu-
ral Computation 1(2), 281–294.
Moore, A. W. (2000). The anchors hierarch: us-
ing the triangle inequality to survive high dimen-
sional data. In Proceedings of the Twelfth Con-
ference on Uncertainty in Artiﬁcial Intelligence ,
pp. 397–405.
M¨uller, K. R., S. Mika, G. R ¨atsch, K. Tsuda, and

B. Sch ¨olkopf (2001). An introduction to kernel-
based learning algorithms. IEEE Transactions on
Neural Networks 12(2), 181–202.
M¨uller, P. and F. A. Quintana (2004). Nonparametric
Bayesian data analysis. Statistical Science 19(1),
95–110.
Nabney, I. T. (2002). Netlab: Algorithms for Pattern
Recognition . Springer.
Nadaraya, ´E. A. (1964). On estimating regression.
Theory of Probability and its Applications 9(1),
141–142.

REFERENCES 723
Nag, R., K. Wong, and F. Fallside (1986). Script
recognition using hidden markov models. InICASSP86 , pp. 2071–2074. IEEE.
Neal, R. M. (1993). Probabilistic inference using
Markov chain Monte Carlo methods. Technical
Report CRG-TR-93-1, Department of ComputerScience, University of Toronto, Canada.
Neal, R. M. (1996). Bayesian Learning for Neural
Networks . Springer. Lecture Notes in Statistics
118.
Neal, R. M. (1997). Monte Carlo implementation of
Gaussian process models for Bayesian regressionand classiﬁcation. Technical Report 9702, De-
partment of Computer Statistics, University of
Toronto.
Neal, R. M. (1999). Suppressing random walks in
Markov chain Monte Carlo using ordered over-
relaxation. In M. I. Jordan (Ed.), Learning in
Graphical Models , pp. 205–228. MIT Press.
Neal, R. M. (2000). Markov chain sampling for
Dirichlet process mixture models. Journal of
Computational and Graphical Statistics 9, 249–
265.
Neal, R. M. (2003). Slice sampling. Annals of Statis-
tics 31, 705–767.

Toronto.
265.
Neal, R. M. and G. E. Hinton (1999). A new view of
the EM algorithm that justiﬁes incremental and
other variants. In M. I. Jordan (Ed.), Learning in
Graphical Models , pp. 355–368. MIT Press.
Nelder, J. A. and R. W. M. Wedderburn (1972). Gen-
eralized linear models. Journal of the Royal Sta-
tistical Society, A 135, 370–384.
Nilsson, N. J. (1965). Learning Machines . McGraw-
Hill. Reprinted as The Mathematical F ounda-
tions of Learning Machines , Morgan Kaufmann,
(1990).
Nocedal, J. and S. J. Wright (1999). Numerical Op-
timization . Springer.

(1990).
Nowlan, S. J. and G. E. Hinton (1992). Simplifying
neural networks by soft weight sharing. Neural
Computation 4(4), 473–493.Ogden, R. T. (1997). Essential Wavelets for Statisti-
cal Applications and Data Analysis . Birkh ¨auser.
Opper, M. and O. Winther (1999). A Bayesian ap-
proach to on-line learning. In D. Saad (Ed.), On-
Line Learning in Neural Networks , pp. 363–378.
Opper, M. and O. Winther (2000a). Gaussian
processes and SVM: mean ﬁeld theory and

leave-one-out. In A. J. Smola, P. L. Bartlett,
B. Sch ¨olkopf, and D. Shuurmans (Eds.), Ad-
vances in Large Margin Classiﬁers , pp. 311–326.
MIT Press.
Opper, M. and O. Winther (2000b). Gaussian
processes for classiﬁcation. Neural Computa-
tion 12(11), 2655–2684.
Osuna, E., R. Freund, and F. Girosi (1996). Support
vector machines: training and applications. A.I.
Memo AIM-1602, MIT.
Papoulis, A. (1984). Probability, Random V ariables,
and Stochastic Processes (Second ed.). McGraw-
Hill.

MIT Press.
Hill.
Parisi, G. (1988). Statistical Field Theory . Addison-
Wesley.
Pearl, J. (1988). Probabilistic Reasoning in Intelli-
gent Systems . Morgan Kaufmann.
Pearlmutter, B. A. (1994). Fast exact multiplication
by the Hessian. Neural Computation 6(1), 147–
160.
Pearlmutter, B. A. and L. C. Parra (1997). Maximum
likelihood source separation: a context-sensitivegeneralization of ICA. In M. C. Mozer, M. I. Jor-
dan, and T. Petsche (Eds.), Advances in Neural
613–619. MIT Press.

Wesley.
160.
Pearson, K. (1901). On lines and planes of closest ﬁt
to systems of points in space. The London, Edin-
burgh and Dublin Philosophical Magazine and
Journal of Science, Sixth Series 2, 559–572.
Platt, J. C. (1999). Fast training of support vector
machines using sequential minimal optimization.
In B. Sch ¨olkopf, C. J. C. Burges, and A. J. Smola
(Eds.), Advances in Kernel Methods – Support
V ector Learning , pp. 185–208. MIT Press.

724 REFERENCES
Platt, J. C. (2000). Probabilities for SV machines.
In A. J. Smola, P. L. Bartlett, B. Sch ¨olkopf, and
D. Shuurmans (Eds.), Advances in Large Margin
Classiﬁers , pp. 61–73. MIT Press.
Platt, J. C., N. Cristianini, and J. Shawe-Taylor
(2000). Large margin DAGs for multiclass clas-
siﬁcation. In S. A. Solla, T. K. Leen, and K. R.M¨uller (Eds.), Advances in Neural Information
Processing Systems , V olume 12, pp. 547–553.
MIT Press.
Poggio, T. and F. Girosi (1990). Networks for ap-
proximation and learning. Proceedings of the
IEEE 78(9), 1481–1497.
Powell, M. J. D. (1987). Radial basis functions for
multivariable interpolation: a review. In J. C.
Mason and M. G. Cox (Eds.), Algorithms for
Approximation , pp. 143–167. Oxford University
Press.
Press, W. H., S. A. Teukolsky, W. T. Vetterling, and
B. P. Flannery (1992). Numerical Recipes in C:
The Art of Scientiﬁc Computing (Second ed.).
Qazaz, C. S., C. K. I. Williams, and C. M. Bishop

Press.
(1997). An upper bound on the Bayesian error
bars for generalized linear regression. In S. W.
Ellacott, J. C. Mason, and I. J. Anderson (Eds.),Mathematics of Neural Networks: Models, Algo-
rithms and Applications , pp. 295–299. Kluwer.
Quinlan, J. R. (1986). Induction of decision trees.
Machine Learning 1(1), 81–106.
Quinlan, J. R. (1993). C4.5: Programs for Machine
Learning . Morgan Kaufmann.
Rabiner, L. and B. H. Juang (1993). Fundamentals
of Speech Recognition . Prentice Hall.
Rabiner, L. R. (1989). A tutorial on hidden Markov

models and selected applications in speechrecognition. Proceedings of the IEEE 77(2),
257–285.
Ramasubramanian, V . and K. K. Paliwal (1990). A
generalized optimization of the k-dtree for fast
nearest-neighbour search. In Proceedings F ourth
IEEE Region 10 International Conference (TEN-
CON’89) , pp. 565–568.Ramsey, F. (1931). Truth and probability. In
R. Braithwaite (Ed.), The F oundations of Math-
ematics and other Logical Essays . Humanities
Press.
Rao, C. R. and S. K. Mitra (1971). Generalized In-

257–285.
Press.
verse of Matrices and Its Applications . Wiley.
Rasmussen, C. E. (1996). Evaluation of Gaussian
Processes and Other Methods for Non-Linear
Regression . Ph. D. thesis, University of Toronto.
Rasmussen, C. E. and J. Qui ˜nonero-Candela (2005).
Healing the relevance vector machine by aug-
mentation. In L. D. Raedt and S. Wrobel (Eds.),
Proceedings of the 22nd International Confer-ence on Machine Learning , pp. 689–696.
Rasmussen, C. E. and C. K. I. Williams (2006).
Gaussian Processes for Machine Learning . MIT

Press.
Rauch, H. E., F. Tung, and C. T. Striebel (1965).
Maximum likelihood estimates of linear dynam-
ical systems. AIAA Journal 3, 1445–1450.
Ricotti, L. P., S. Ragazzini, and G. Martinelli (1988).
Learning of word stress in a sub-optimal second
order backpropagation neural network. In Pro-
ceedings of the IEEE International Conferenceon Neural Networks , V olume 1, pp. 355–361.
IEEE.
Ripley, B. D. (1996). Pattern Recognition and Neu-
ral Networks . Cambridge University Press.
Robbins, H. and S. Monro (1951). A stochastic

IEEE.
approximation method. Annals of Mathematical
Statistics 22, 400–407.
Robert, C. P. and G. Casella (1999). Monte Carlo
Statistical Methods . Springer.
Rockafellar, R. (1972). Convex Analysis . Princeton
Rosenblatt, F. (1962). Principles of Neurodynam-
ics: Perceptrons and the Theory of Brain Mech-
anisms . Spartan.
Roth, V . and V . Steinhage (2000). Nonlinear discrim-
inant analysis using kernel functions. In S. A.

REFERENCES 725
Solla, T. K. Leen, and K. R. M ¨uller (Eds.), Ad-
vances in Neural Information Processing Sys-tems, V olume 12. MIT Press.
Roweis, S. (1998). EM algorithms for PCA and
SPCA. In M. I. Jordan, M. J. Kearns, and S. A.
Solla (Eds.), Advances in Neural Information
Processing Systems , V olume 10, pp. 626–632.
MIT Press.
Roweis, S. and Z. Ghahramani (1999). A unifying
review of linear Gaussian models. Neural Com-
putation 11(2), 305–345.
Roweis, S. and L. Saul (2000, December). Nonlinear
dimensionality reduction by locally linear em-
bedding. Science 290, 2323–2326.
Rubin, D. B. (1983). Iteratively reweighted least
squares. In Encyclopedia of Statistical Sciences ,
V olume 4, pp. 272–275. Wiley.
Rubin, D. B. and D. T. Thayer (1982). EM al-
gorithms for ML factor analysis. Psychome-
trika 47(1), 69–76.
Rumelhart, D. E., G. E. Hinton, and R. J. Williams
(1986). Learning internal representations by er-ror propagation. In D. E. Rumelhart, J. L. Mc-
Clelland, and the PDP Research Group (Eds.),

Parallel Distributed Processing: Explorationsin the Microstructure of Cognition , V olume 1:
Foundations, pp. 318–362. MIT Press. Reprinted
in Anderson and Rosenfeld (1988).
Rumelhart, D. E., J. L. McClelland, and the PDP Re-
search Group (Eds.) (1986). Parallel Distributed
Processing: Explorations in the Microstruc-
ture of Cognition , V olume 1: Foundations. MIT
Press.
Sagan, H. (1969). Introduction to the Calculus of
V ariations . Dover.
Savage, L. J. (1961). The subjective basis of sta-

Press.
tistical practice. Technical report, Department of
Statistics, University of Michigan, Ann Arbor.
Sch¨olkopf, B., J. Platt, J. Shawe-Taylor, A. Smola,
and R. C. Williamson (2001). Estimating the sup-port of a high-dimensional distribution. Neural
Computation 13(7), 1433–1471.Sch¨olkopf, B., A. Smola, and K.-R. M ¨uller (1998).
Nonlinear component analysis as a kerneleigenvalue problem. Neural Computation 10(5),
1299–1319.
Sch¨olkopf, B., A. Smola, R. C. Williamson, and P. L.

Bartlett (2000). New support vector algorithms.Neural Computation 12(5), 1207–1245.
Sch¨olkopf, B. and A. J. Smola (2002). Learning with
Kernels . MIT Press.
Schwarz, G. (1978). Estimating the dimension of a
model. Annals of Statistics 6, 461–464.
Schwarz, H. R. (1988). Finite element methods . Aca-
Seeger, M. (2003). Bayesian Gaussian Process Mod-
els: PAC-Bayesian Generalization Error Bounds
and Sparse Approximations . Ph. D. thesis, Uni-
versity of Edinburg.
Seeger, M., C. K. I. Williams, and N. Lawrence

(2003). Fast forward selection to speed up sparse
Gaussian processes. In C. M. Bishop and B. Frey
(Eds.), Proceedings Ninth International Work-
shop on Artiﬁcial Intelligence and Statistics, Key
West, Florida .
Shachter, R. D. and M. Peot (1990). Simulation ap-
proaches to general probabilistic inference on be-lief networks. In P. P. Bonissone, M. Henrion,
L. N. Kanal, and J. F. Lemmer (Eds.), Uncer-
tainty in Artiﬁcial Intelligence , V olume 5. Else-
vier.
Shannon, C. E. (1948). A mathematical theory of
communication. The Bell System Technical Jour-

vier.
nal27(3), 379–423 and 623–656.
Shawe-Taylor, J. and N. Cristianini (2004). Kernel
Methods for Pattern Analysis . Cambridge Uni-
Sietsma, J. and R. J. F. Dow (1991). Creating artiﬁ-
cial neural networks that generalize. Neural Net-
works 4(1), 67–79.
Simard, P., Y . Le Cun, and J. Denker (1993). Efﬁ-
cient pattern recognition using a new transforma-
tion distance. In S. J. Hanson, J. D. Cowan, and

726 REFERENCES
tion Processing Systems , V olume 5, pp. 50–58.
Simard, P., B. Victorri, Y . Le Cun, and J. Denker
(1992). Tangent prop – a formalism for specify-
ing selected invariances in an adaptive network.
In J. E. Moody, S. J. Hanson, and R. P. Lippmann(Eds.), Advances in Neural Information Process-
ing Systems , V olume 4, pp. 895–903. Morgan
Kaufmann.
Simard, P. Y ., D. Steinkraus, and J. Platt (2003).
Best practice for convolutional neural networks
applied to visual document analysis. In Pro-
ceedings International Conference on Document
Analysis and Recognition (ICDAR) , pp. 958–
962. IEEE Computer Society.
Sirovich, L. (1987). Turbulence and the dynamics
of coherent structures. Quarterly Applied Math-
ematics 45(3), 561–590.
Smola, A. J. and P. Bartlett (2001). Sparse greedy
Gaussian process regression. In T. K. Leen, T. G.
Dietterich, and V . Tresp (Eds.), Advances in Neu-
ral Information Processing Systems , V olume 13,

pp. 619–625. MIT Press.
Spiegelhalter, D. and S. Lauritzen (1990). Sequential
updating of conditional probabilities on directed
graphical structures. Networks 20, 579–605.
Stinchecombe, M. and H. White (1989). Universal
approximation using feed-forward networks with
non-sigmoid hidden layer activation functions. In
International Joint Conference on Neural Net-
works , V olume 1, pp. 613–618. IEEE.
Stone, J. V . (2004). Independent Component Analy-
sis: A Tutorial Introduction . MIT Press.

Sung, K. K. and T. Poggio (1994). Example-based
learning for view-based human face detection.
A.I. Memo 1521, MIT.
Sutton, R. S. and A. G. Barto (1998). Reinforcement
Learning: An Introduction . MIT Press.
Svens ´en, M. and C. M. Bishop (2004). Ro-
bust Bayesian mixture modelling. Neurocomput-
ing 64, 235–252.Tarassenko, L. (1995). Novelty detection for the
identiﬁcation of masses in mamograms. In Pro-
ceedings F ourth IEE International Conference
on Artiﬁcial Neural Networks , V olume 4, pp.
442–447. IEE.

Tax, D. and R. Duin (1999). Data domain descrip-
tion by support vectors. In M. Verleysen (Ed.),Proceedings European Symposium on Artiﬁcial
Neural Networks, ESANN , pp. 251–256. D. Facto
Press.
Teh, Y . W., M. I. Jordan, M. J. Beal, and D. M. Blei
(2006). Hierarchical Dirichlet processes. Journal
of the Americal Statistical Association . to appear.
Tenenbaum, J. B., V . de Silva, and J. C. Langford
(2000, December). A global framework for non-linear dimensionality reduction. Science 290,
2319–2323.

Press.
Tesauro, G. (1994). TD-Gammon, a self-teaching
backgammon program, achieves master-level
play. Neural Computation 6(2), 215–219.
Thiesson, B., D. M. Chickering, D. Heckerman, and
C. Meek (2004). ARMA time-series modelling
with graphical models. In M. Chickering and
J. Halpern (Eds.), Proceedings of the Twentieth
gence, Banff, Canada , pp. 552–560. AUAI Press.
Tibshirani, R. (1996). Regression shrinkage and se-
lection via the lasso. Journal of the Royal Statis-

tical Society, B 58, 267–288.
Tierney, L. (1994). Markov chains for exploring pos-
terior distributions. Annals of Statistics 22(4),
1701–1762.
Tikhonov, A. N. and V . Y . Arsenin (1977). Solutions
of Ill-Posed Problems . V . H. Winston.
Tino, P. and I. T. Nabney (2002). Hierarchical
GTM: constructing localized non-linear projec-
tion manifolds in a principled way. IEEE Trans-
actions on Pattern Analysis and Machine Intelli-gence 24(5), 639–656.
Tino, P., I. T. Nabney, and Y . Sun (2001). Us-
ing directional curvatures to visualize folding

patterns of the GTM projection manifolds. In

REFERENCES 727
G. Dorffner, H. Bischof, and K. Hornik (Eds.),
Artiﬁcial Neural Networks – ICANN 2001 , pp.
421–428. Springer.
Tipping, M. E. (1999). Probabilistic visualisation of
high-dimensional binary data. In M. S. Kearns,S. A. Solla, and D. A. Cohn (Eds.), Advances
ume 11, pp. 592–598. MIT Press.
Tipping, M. E. (2001). Sparse Bayesian learning and
the relevance vector machine. Journal of Ma-
chine Learning Research 1, 211–244.
Tipping, M. E. and C. M. Bishop (1997). Probabilis-
tic principal component analysis. Technical Re-port NCRG/97/010, Neural Computing Research
Group, Aston University.
Tipping, M. E. and C. M. Bishop (1999a). Mixtures
of probabilistic principal component analyzers.
Neural Computation 11(2), 443–482.
Tipping, M. E. and C. M. Bishop (1999b). Prob-
abilistic principal component analysis. Journal
of the Royal Statistical Society, Series B 21(3),
611–622.
Tipping, M. E. and A. Faul (2003). Fast marginal

611–622.
likelihood maximization for sparse Bayesian
models. In C. M. Bishop and B. Frey (Eds.),Proceedings Ninth International Workshop on
Artiﬁcial Intelligence and Statistics, Key West,
Florida .
Tong, S. and D. Koller (2000). Restricted Bayes op-
timal classiﬁers. In Proceedings 17th National
Conference on Artiﬁcial Intelligence , pp. 658–
664. AAAI.
Tresp, V . (2001). Scaling kernel-based systems to
large data sets. Data Mining and Knowledge Dis-
covery 5(3), 197–211.
Uhlenbeck, G. E. and L. S. Ornstein (1930). On the

Florida .
664. AAAI.
theory of Brownian motion. Phys. Rev. 36, 823–
841.
Valiant, L. G. (1984). A theory of the learnable.
Communications of the Association for Comput-
ing Machinery 27, 1134–1142.Vapnik, V . N. (1982). Estimation of dependences
based on empirical data . Springer.
Vapnik, V . N. (1995). The nature of statistical learn-
ing theory . Springer.
Vapnik, V . N. (1998). Statistical learning theory .W i -
ley.
Veropoulos, K., C. Campbell, and N. Cristianini
(1999). Controlling the sensitivity of supportvector machines. In Proceedings of the Interna-

841.
ley.
tional Joint Conference on Artiﬁcial Intelligence
(IJCAI99), Workshop ML3 , pp. 55–60.
Vidakovic, B. (1999). Statistical Modelling by
Wavelets . Wiley.
Viola, P. and M. Jones (2004). Robust real-time face
detection. International Journal of Computer Vi-
sion 57(2), 137–154.
Viterbi, A. J. (1967). Error bounds for convolu-
tional codes and an asymptotically optimum de-
coding algorithm. IEEE Transactions on Infor-
mation Theory IT-13 , 260–267.
Viterbi, A. J. and J. K. Omura (1979). Principles of

Digital Communication and Coding . McGraw-
Hill.
Wahba, G. (1975). A comparison of GCV and GML
for choosing the smoothing parameter in the gen-
eralized spline smoothing problem. Numerical
Mathematics 24, 383–393.
Wainwright, M. J., T. S. Jaakkola, and A. S. Willsky
(2005). A new class of upper bounds on the log
partition function. IEEE Transactions on Infor-
mation Theory 51, 2313–2335.
Walker, A. M. (1969). On the asymptotic behaviour
of posterior distributions. Journal of the Royal

Hill.
Statistical Society, B 31(1), 80–88.
Walker, S. G., P. Damien, P. W. Laud, and A. F. M.
Smith (1999). Bayesian nonparametric inference
for random distributions and related functions(with discussion). Journal of the Royal Statisti-
cal Society, B 61(3), 485–527.
Watson, G. S. (1964). Smooth regression analysis.
Sankhy ¯a: The Indian Journal of Statistics. Series
A26, 359–372.

728 REFERENCES
Webb, A. R. (1994). Functional approximation by
feed-forward networks: a least-squares approachto generalisation. IEEE Transactions on Neural
Networks 5(3), 363–371.
Weisstein, E. W. (1999). CRC Concise Encyclopedia
of Mathematics . Chapman and Hall, and CRC.
Weston, J. and C. Watkins (1999). Multi-class sup-
port vector machines. In M. Verlysen (Ed.), Pro-
ceedings ESANN’99, Brussels . D-Facto Publica-
tions.
Whittaker, J. (1990). Graphical Models in Applied
Multivariate Statistics . Wiley.
Widrow, B. and M. E. Hoff (1960). Adaptive
switching circuits. In IRE WESCON Convention
Record , V olume 4, pp. 96–104. Reprinted in An-
derson and Rosenfeld (1988).
Widrow, B. and M. A. Lehr (1990). 30 years of adap-
tive neural networks: perceptron, madeline, and
backpropagation. Proceedings of the IEEE 78(9),
1415–1442.
Wiegerinck, W. and T. Heskes (2003). Fractional
belief propagation. In S. Becker, S. Thrun, andK. Obermayer (Eds.), Advances in Neural Infor-

mation Processing Systems , V olume 15, pp. 455–
462. MIT Press.
Williams, C. K. I. (1998). Computation with inﬁ-
nite neural networks. Neural Computation 10(5),
1203–1216.
Williams, C. K. I. (1999). Prediction with Gaussian
processes: from linear regression to linear pre-diction and beyond. In M. I. Jordan (Ed.), Learn-
ing in Graphical Models , pp. 599–621. MIT
Press.
Williams, C. K. I. and D. Barber (1998). Bayesian
classiﬁcation with Gaussian processes. IEEE
Transactions on Pattern Analysis and Machine

Press.
Intelligence 20, 1342–1351.
Williams, C. K. I. and M. Seeger (2001). Using the
Nystrom method to speed up kernel machines. In
T. K. Leen, T. G. Dietterich, and V . Tresp (Eds.),Advances in Neural Information Processing Sys-
tems, V olume 13, pp. 682–688. MIT Press.Williams, O., A. Blake, and R. Cipolla (2005).
Sparse Bayesian learning for efﬁcient visualtracking. IEEE Transactions on Pattern Analysis
and Machine Intelligence 27(8), 1292–1304.
Williams, P. M. (1996). Using neural networks to

model conditional multivariate densities. Neural
Computation 8(4), 843–854.
Winn, J. and C. M. Bishop (2005). Variational mes-
sage passing. Journal of Machine Learning Re-
search 6, 661–694.
Zarchan, P. and H. Musoff (2005). Fundamentals of
Kalman Filtering: A Practical Approach (Sec-
ond ed.). AIAA.

INDEX 729
Index
Page numbers in bold indicate the primary source of information for the corresponding topic.
1-of-Kcoding scheme, 424
acceptance criterion, 538, 541, 544
activation function, 180, 213, 227
active constraint, 328, 709
AdaBoost, 657, 658
adaline, 196adaptive rejection sampling, 530ADF, seeassumed density ﬁltering
AIC, seeAkaike information criterion
Akaike information criterion, 33, 217
αfamily of divergences, 469
αrecursion, 620
ancestral sampling, 365, 525, 613
annular ﬂow, 679AR model, seeautoregressive model
arc, 360ARD, seeautomatic relevance determination
ARMA, seeautoregressive moving average
assumed density ﬁltering, 510autoassociative networks, 592automatic relevance determination, 259, 312, 349,
485, 582
autoregressive hidden Markov model, 632autoregressive model, 609autoregressive moving average, 304
back-tracking, 415, 630backgammon, 3
backpropagation, 241bagging, 656basis function, 138, 172, 204, 227
batch training, 240Baum-Welch algorithm, 618
Bayes’ theorem, 15

485, 582
Bayes, Thomas, 21Bayesian analysis, vii, 9, 21
hierarchical, 372model averaging, 654
Bayesian information criterion, 33, 216
Bayesian model comparison, 161, 473, 483
Bayesian network, 360Bayesian probability, 21belief propagation, 403Bernoulli distribution, 69, 113, 685
mixture model, 444
Bernoulli, Jacob, 69
beta distribution, 71, 686
beta recursion, 621between-class covariance, 189bias, 27, 149
bias parameter, 138, 181, 227, 346
bias-variance trade-off, 147BIC, seeBayesian information criterion

binary entropy, 495binomial distribution, 70, 686

730 INDEX
biological sequence, 610
bipartite graph, 401bits, 49
blind source separation, 591
blocked path, 374, 378, 384
Boltzmann distribution, 387Boltzmann, Ludwig Eduard, 53Boolean logic, 21boosting, 657bootstrap, 23, 656
bootstrap ﬁlter, 646box constraints, 333, 342
Box-Muller method, 527
C4.5, 663
calculus of variations, 462canonical correlation analysis, 565
canonical link function, 212
CART, seeclassiﬁcation and regression trees
Cauchy distribution, 527, 529, 692
causality, 366CCA, seecanonical correlation analysis
central differences, 246central limit theorem, 78chain graph, 393chaining, 555Chapman-Kolmogorov equations, 397child node, 361Cholesky decomposition, 528chunking, 335circular normal, seevon Mises distribution
classical probability, 21
classiﬁcation, 3
classiﬁcation and regression trees, 663clique, 385clustering, 3clutter problem, 511co-parents, 383, 492
code-book vectors, 429combining models, 45, 653

committee, 655complete data set, 440completing the square, 86computational learning theory, 326, 344concave function, 56concentration parameter, 108, 693
condensation algorithm, 646
conditional entropy, 55
conditional expectation, 20
conditional independence, 46, 372, 383
conditional mixture model, seemixture model
conditional probability, 14
conjugate prior, 68, 98, 117, 490
convex duality, 494
convex function, 55, 493
convolutional neural network, 267
correlation matrix, 567
cost function, 41
covariance, 20

between-class, 189
within-class, 189
covariance matrix
diagonal, 84
isotropic, 84
partitioned, 85, 307
positive deﬁnite, 308
Cox’s axioms, 21
credit assignment, 3
cross-entropy error function, 206, 209, 235, 631,
666
cross-validation, 32, 161
cumulative distribution function, 18
curse of dimensionality, 33,3 6
curve ﬁtting, 4
D map, seedependency map
d-separation, 373, 378, 443
DAG, seedirected acyclic graph
DAGSVM, 339
data augmentation, 537
data compression, 429
decision boundary, 39, 179
decision region,
39, 179
decision surface, seedecision boundary
decision theory, 38
decision tree, 654, 663, 673
decomposition methods, 335
degrees of freedom, 559

666
39, 179
degrees-of-freedom parameter, 102, 693
density estimation, 3, 67

INDEX 731
density network, 597
dependency map, 392descendant node, 376
design matrix, 142, 347
differential entropy, 53digamma function, 687directed acyclic graph, 362directed cycle, 362directed factorization, 381Dirichlet distribution, 76, 687
Dirichlet, Lejeune, 77discriminant function, 43, 180, 181
discriminative model, 43, 203
distortion measure, 424distributive law of multiplication, 396DNA, 610document retrieval, 299dual representation, 293, 329
dual-energy gamma densitometry, 678
dynamic programming, 411dynamical system, 548
E step, seeexpectation step
early stopping, 259ECM, seeexpectation conditional maximization
edge, 360effective number of observations, 72, 101
effective number of parameters, 9, 170, 281
elliptical K-means, 444
EM, seeexpectation maximization
emission probability, 611empirical Bayes, seeevidence approximation
energy function, 387
entropy, 49
conditional, 55differential, 53relative, 55
EP, seeexpectation propagation
ϵ-tube, 341
ϵ-insensitive error function, 340

equality constraint, 709equivalent kernel, 159, 301
erf function, 211error backpropagation, seebackpropagation
error function, 5,2 3error-correcting output codes, 339
Euler, Leonhard, 465Euler-Lagrange equations, 705
evidence approximation, 165, 347, 581
evidence function, 161expectation, 19expectation conditional maximization, 454expectation maximization, 113, 423, 440
Gaussian mixture, 435generalized, 454sampling methods, 536
expectation propagation, 315, 468, 505

expectation step, 437explaining away, 378exploitation, 3exploration, 3exponential distribution, 526, 688
exponential family, 68, 113, 202, 490
extensive variables, 490
face detection, 2
face tracking, 355factor analysis, 583
mixture model, 595
factor graph, 360, 399, 625
factor loading, 584factorial hidden Markov model, 633factorized distribution, 464, 476
feature extraction, 2feature map, 268feature space, 292, 586
Fisher information matrix, 298
Fisher kernel, 298

Fisher’s linear discriminant, 186ﬂooding schedule, 417forward kinematics, 272forward problem, 272forward propagation, 228, 243
forward-backward algorithm, 618fractional belief propagation, 517frequentist probability, 21fuel system, 376function interpolation, 299functional, 462, 703
derivative, 463

732 INDEX
gamma densitometry, 678
gamma distribution, 529, 688
gamma function, 71gating function, 672Gauss, Carl Friedrich, 79
Gaussian, 24, 78, 688
conditional, 85,9 3
marginal, 88,9 3
maximum likelihood, 93
mixture, 110, 270, 273, 430
sequential estimation, 94sufﬁcient statistics, 93wrapped, 110
Gaussian kernel, 296
Gaussian process, 160, 303
Gaussian random ﬁeld, 305Gaussian-gamma distribution, 101, 690
Gaussian-Wishart distribution, 102, 475, 478, 690
GEM, seeexpectation maximization, generalized
generalization, 2generalized linear model, 180, 213
generalized maximum likelihood, seeevidence ap-
proximation
generative model, 43, 196, 297, 365, 572, 631
generative topographic mapping, 597
directional curvature, 599magniﬁcation factor, 599
geodesic distance, 596Gibbs sampling, 542
blocking, 546
Gibbs, Josiah Willard, 543Gini index, 666global minimum, 237
gradient descent, 240
Gram matrix, 293graph-cut algorithm, 390graphical model, 359
bipartite, 401
directed, 360factorization, 362, 384fully connected, 361

inference, 393
tree, 398treewidth, 417triangulated, 416undirected, 360
Green’s function, 299GTM, seegenerative topographic mapping
Hamilton, William Rowan, 549
Hamiltonian dynamics, 548Hamiltonian function, 549Hammersley-Clifford theorem, 387handwriting recognition, 1, 610, 614handwritten digit, 565, 614, 677
head-to-head path, 376head-to-tail path, 375Heaviside step function, 206Hellinger distance, 470Hessian matrix, 167, 215, 217, 238, 249
diagonal approximation, 250exact evaluation, 253
fast multiplication, 254

ﬁnite differences, 252inverse, 252outer product approximation, 251
heteroscedastic, 273, 311
hidden Markov model, 297, 610
autoregressive, 632factorial, 633forward-backward algorithm, 618input-output, 633left-to-right, 613maximum likelihood, 615scaling factor, 627sum-product algorithm, 625switching, 644
variational inference, 625
hidden unit, 227hidden variable, 84, 364, 430, 559

hierarchical Bayesian model, 372hierarchical mixture of experts, 673hinge error function, 337Hinton diagram, 584histogram density estimation, 120HME, seehierarchical mixture of experts
hold-out set, 11homogeneous ﬂow, 679homogeneous kernel, 292homogeneous Markov chain, 540, 608

INDEX 733
Hooke’s law, 580
hybrid Monte Carlo, 548hyperparameter, 71, 280, 311, 346, 372, 502
hyperprior, 372
I map, seeindependence map
i.i.d., seeindependent identically distributed
ICA, seeindependent component analysis
ICM, seeiterated conditional modes
ID3, 663identiﬁability, 435image de-noising, 387importance sampling, 525, 532
importance weights, 533improper prior, 118, 259, 472
imputation step, 537imputation-posterior algorithm, 537
inactive constraint, 328, 709
incomplete data set, 440independence map, 392independent component analysis, 591independent factor analysis, 592independent identically distributed, 26, 379
independent variables, 17independent, identically distributed, 605induced factorization, 485inequality constraint, 709inference, 38, 42
information criterion, 33information geometry, 298information theory, 48input-output hidden Markov model, 633
intensive variables, 490

intrinsic dimensionality, 559invariance, 261inverse gamma distribution, 101inverse kinematics, 272inverse problem, 272inverse Wishart distribution, 102IP algorithm, seeimputation-posterior algorithm
IRLS, seeiterative reweighted least squares
Ising model, 389isomap, 596isometric feature map, 596iterated conditional modes, 389, 415iterative reweighted least squares, 207, 210, 316,
354, 672
Jacobian matrix, 247, 264
Jensen’s inequality, 56
join tree, 416
junction tree algorithm, 392, 416
Knearest neighbours, 125
K-means clustering algorithm, 424, 443

354, 672
K-medoids algorithm, 428
Kalman ﬁlter, 304, 637
extended, 644
Kalman gain matrix, 639Kalman smoother, 637
Karhunen-Lo `eve transform, 561
Karush-Kuhn-Tucker conditions, 330, 333, 342,
710
kernel density estimator, 122, 326
kernel function, 123, 292, 294
Fisher, 298
Gaussian, 296
homogeneous, 292
nonvectorial inputs, 297
stationary, 292
kernel PCA, 586
kernel regression, 300, 302
kernel substitution, 292
kernel trick, 292
kinetic energy, 549KKT, seeKarush-Kuhn-Tucker conditions
KL divergence, seeKullback-Leibler divergence
kriging, seeGaussian process
Kullback-Leibler divergence, 55, 451, 468, 505
Lagrange multiplier, 707
Lagrange, Joseph-Louis, 329

710
Lagrangian, 328, 332, 341, 708
laminar ﬂow, 678
Laplace approximation, 213, 217, 278, 315, 354
Laplace, Pierre-Simon, 24
large margin, seemargin
lasso, 145latent class analysis, 444
latent trait model, 597
latent variable, 84, 364, 430, 559

734 INDEX
lattice diagram, 414, 611, 621, 629
LDS, seelinear dynamical system
leapfrog discretization, 551
learning, 2
learning rate parameter, 240least-mean-squares algorithm, 144
leave-one-out, 33
likelihood function, 22likelihood weighted sampling, 534
linear discriminant, 181
Fisher, 186
linear dynamical system, 84, 635
inference, 638
linear independence, 696
linear regression, 138
EM, 448mixture model, 667
variational, 486
linear smoother, 159linear-Gaussian model, 87, 370
linearly separable, 179
link, 360link function, 180, 213
Liouville’s Theorem, 550
LLE, seelocally linear embedding
LMS algorithm, seeleast-mean-squares algorithm
local minimum, 237
local receptive ﬁeld, 268
locally linear embedding, 596
location parameter, 118log odds, 197
logic sampling, 525
logistic regression, 205, 336
Bayesian, 217, 498
mixture model, 670
multiclass, 209
logistic sigmoid, 114, 139, 197, 205, 220, 227, 495
logit function, 197
loopy belief propagation, 417loss function, 41
loss matrix, 41

lossless data compression, 429lossy data compression, 429
lower bound, 484
M step, seemaximization stepmachine learning, vii
macrostate, 51Mahalanobis distance, 80manifold, 38, 590, 595, 681
MAP, seemaximum posterior
margin, 326, 327, 502
error, 334soft, 332
marginal likelihood, 162, 165
marginal probability, 14
Markov blanket, 382, 384, 545
Markov boundary, seeMarkov blanket
Markov chain, 397, 539
ﬁrst order, 607
homogeneous, 540, 608
second order, 608
Markov chain Monte Carlo, 537
Markov model, 607
homogeneous, 612

Markov network, seeMarkov random ﬁeld
Markov random ﬁeld, 84, 360, 383
max-sum algorithm, 411, 629
maximal clique, 385
maximal spanning tree, 416maximization step, 437maximum likelihood, 9, 23, 26, 116
Gaussian mixture, 432
singularities, 480type 2, see
evidence approximation
maximum margin, seemargin
maximum posterior, 30, 441
MCMC, seeMarkov chain Monte Carlo
MDN, seemixture density network
MDS, seemultidimensional scaling
mean, 24
mean ﬁeld theory, 465
mean value theorem, 52measure theory, 19memory-based methods, 292
message passing, 396

mean, 24
pending message, 417schedule, 417variational, 491
Metropolis algorithm, 538
Metropolis-Hastings algorithm, 541

INDEX 735
microstate, 51
minimum risk, 44Minkowski loss, 48
missing at random, 441, 579
missing data, 579mixing coefﬁcient, 111mixture component, 111mixture density network, 272, 673
mixture distribution, seemixture model
mixture model, 162, 423
conditional, 273, 666
linear regression, 667logistic regression, 670symmetries, 483
mixture of experts, 672mixture of Gaussians, 110, 270, 273, 430
MLP, seemultilayer perceptron
MNIST data, 677
model comparison, 6, 32, 161, 473, 483
model evidence, 161model selection, 162moment matching, 506, 510
momentum variable, 548Monte Carlo EM algorithm, 536Monte Carlo sampling, 24, 523
Moore-Penrose pseudo-inverse, seepseudo-inverse
moralization, 391, 401
MRF, seeMarkov random ﬁeld
multidimensional scaling, 596multilayer perceptron, 226, 229
multimodality, 272multinomial distribution, 76, 114, 690
multiplicity, 51
mutual information, 55, 57
Nadaraya-Watson, seekernel regression
naive Bayes model, 46, 380

nats, 50natural language modelling, 610natural parameters, 113nearest-neighbour methods, 124neural network, 225
convolutional, 267regularization, 256relation to Gaussian process, 319Newton-Raphson, 207, 317
node, 360
noiseless coding theorem, 50nonidentiﬁability, 585
noninformative prior, 23, 117
nonparametric methods, 68, 120
normal distribution, seeGaussian
normal equations, 142normal-gamma distribution, 101, 691
normal-Wishart distribution, 102, 475, 478, 691
normalized exponential, seesoftmax function

node, 360
novelty detection, 44
ν-SVM, 334
object recognition, 366
observed variable, 364Occam factor, 217
oil ﬂow data, 34, 560, 568, 678
Old Faithful data, 110, 479, 484, 681
on-line learning, seesequential learning
one-versus-one classiﬁer, 183, 339
one-versus-the-rest classiﬁer, 182, 338
ordered over-relaxation, 545
Ornstein-Uhlenbeck process, 305orthogonal least squares, 301
outlier, 44, 185, 212
outliers, 103
over-ﬁtting, 6
, 147, 434, 464
over-relaxation, 544
PAC learning, seeprobably approximately correct

ν-SVM, 334
PAC-Bayesian framework, 345
parameter shrinkage, 144
parent node, 361particle ﬁlter, 645
partition function, 386, 554
Parzen estimator, seekernel density estimator
Parzen window, 123
pattern recognition, vii
PCA, seeprincipal component analysis
pending message, 417
perceptron, 192
convergence theorem, 194
hardware, 196
perceptron criterion, 193perfect map, 392

736 INDEX
periodic variable, 105
phase space, 549photon noise, 680plate, 363
polynomial curve ﬁtting, 4, 362
polytree, 399position variable, 548positive deﬁnite covariance, 81positive deﬁnite matrix, 701
positive semideﬁnite covariance, 81
positive semideﬁnite matrix, 701posterior probability, 17posterior step, 537
potential energy, 549
potential function, 386power EP, 517power method, 563
precision matrix, 85
precision parameter, 24predictive distribution, 30, 156
preprocessing, 2principal component analysis, 561, 572, 593
Bayesian, 580
EM algorithm, 577Gibbs sampling, 583mixture distribution, 595
physical analogy, 580
principal curve, 595principal subspace, 561principal surface, 596prior, 17
conjugate, 68, 98, 117, 490
consistent, 257improper, 118, 259, 472
noninformative, 23, 117
probabilistic graphical model, seegraphical model
probabilistic PCA, 570probability, 12
Bayesian, 21
classical, 21
density, 17frequentist, 21mass function, 19prior, 45
product rule, 13, 14, 359sum rule, 13, 14, 359
theory, 12

theory, 12
probably approximately correct, 344
probit function, 211, 219
probit regression, 210product rule of probability, 13, 14, 359
proposal distribution, 528, 532, 538
protected conjugate gradients, 335
protein sequence, 610
pseudo-inverse, 142, 185
pseudo-random numbers, 526
quadratic discriminant, 199
quality parameter, 351
radial basis function, 292, 299
Rauch-Tung-Striebel equations, 637
regression, 3regression function, 47,9 5
regularization, 10
Tikhonov, 267
regularized least squares, 144
reinforcement learning, 3

reject option, 42,4 5
rejection sampling, 528
relative entropy, 55relevance vector, 348
relevance vector machine, 161, 345
responsibility, 112, 432, 477
ridge regression, 10
RMS error, seeroot-mean-square error
Robbins-Monro algorithm, 95robot arm, 272
robustness, 103, 185
root node, 399root-mean-square error, 6
Rosenblatt, Frank, 193
rotation invariance, 573, 585
RTS equations, seeRauch-Tung-Striebel equations
running intersection property, 416RVM, seerelevance vector machine
sample mean, 27
sample variance, 27

sampling-importance-resampling, 534
scale invariance, 119, 261

INDEX 737
scale parameter, 119
scaling factor, 627Schwarz criterion, seeBayesian information crite-
rion
self-organizing map, 598sequential data, 605sequential estimation, 94sequential gradient descent, 144, 240sequential learning, 73, 143
sequential minimal optimization, 335serial message passing schedule, 417
Shannon, Claude, 55
shared parameters, 368shrinkage, 10Shur complement, 87sigmoid, seelogistic sigmoid
simplex, 76single-class support vector machine, 339singular value decomposition, 143sinusoidal data, 682SIR, seesampling-importance-resampling
skip-layer connection, 229slack variable, 331slice sampling, 546SMO, seesequential minimal optimization
smoother matrix, 159smoothing parameter, 122soft margin, 332
soft weight sharing, 269
softmax function, 115, 198, 236, 274, 356, 497
SOM, seeself-organizing map
sparsity, 145, 347, 349, 582
sparsity parameter, 351spectrogram, 606speech recognition, 605, 610
sphereing, 568spline functions, 139standard deviation, 24standardizing, 425, 567
state space model, 609

switching, 644
stationary kernel, 292statistical bias, seebias
statistical independence, seeindependent variablesstatistical learning theory, seecomputational learn-
ing theory, 326, 344
steepest descent, 240
Stirling’s approximation, 51
stochastic, 5
stochastic EM, 536
stochastic gradient descent, 144, 240
stochastic process, 305
stratiﬁed ﬂow, 678
Student’s t-distribution, 102, 483, 691
subsampling, 268
sufﬁcient statistics, 69, 75, 116
sum rule of probability, 13, 14, 359
sum-of-squares error, 5, 29, 184, 232, 662
sum-product algorithm, 399, 402

for hidden Markov model, 625
supervised learning, 3
support vector, 330
support vector machine, 225
for regression, 339
multiclass, 338
survival of the ﬁttest, 646
SVD, seesingular value decomposition
SVM, seesupport vector machine
switching hidden Markov model, 644switching state space model, 644
synthetic data sets, 682
tail-to-tail path, 374
tangent distance, 265
tangent propagation, 262, 263
tapped delay line, 609
target vector, 2
test set, 2, 32
threshold parameter, 181
tied parameters, 368
Tikhonov regularization, 267

time warping, 615
tomography, 679
training, 2
transition probability,
540, 610
translation invariance, 118, 261
tree-reweighted message passing, 517
treewidth, 417

738 INDEX
trellis diagram, seelattice diagram
triangulated graph, 416type 2 maximum likelihood, seeevidence approxi-
mation
undetermined multiplier, seeLagrange multiplier
undirected graph, seeMarkov random ﬁeld
uniform distribution, 692
uniform sampling, 534uniquenesses, 584
unobserved variable, seelatent variable
unsupervised learning, 3utility function, 41
validation set, 11, 32
Vapnik-Chervonenkis dimension, 344
variance, 20, 24, 149
variational inference, 315, 462, 635
for Gaussian mixture, 474
local, 493
VC dimension, seeVapnik-Chervonenkis dimen-
sion
vector quantization, 429vertex, seenode
visualization, 3
Viterbi algorithm, 415, 629
von Mises distribution, 108, 693
wavelets, 139
weak learner, 657weight decay, 10, 144, 257
weight parameter, 227
weight sharing, 268
soft, 269
weight vector, 181
weight-space symmetry, 231, 281
weighted least squares, 668
well-determined parameters, 170whitening, 299, 568
Wishart distribution, 102, 693

local, 493
sion
soft, 269
within-class covariance, 189Woodbury identity, 696
wrapped distribution, 110
Yellowstone National Park, 110, 681

Understanding Machine Learning:
From Theory to Algorithms
c⃝2014 by Shai Shalev-Shwartz and Shai Ben-David
Published 2014 by Cambridge University Press.
This copy is for personal use only. Not for distribution.
Do not post. Please link to:
http://www.cs.huji.ac.il/ ~shais/UnderstandingMachineLearning
Please note: This copy is almost, but not entirely, identical to the printed version
of the book. In particular, page numbers are not identical (but section numbers are the
same).

data/pdfs/Understanding Machine Learning - From Theory to Algorithms.pdf

Understanding Machine LearningMachine learning is one of the fastest growing areas of computer science,with far-reaching applications. The aim of this textbook is to introducemachine learning, and the algorithmic paradigms it offers, in a princi-pled way. The book provides an extensive theoretical account of thefundamental ideas underlying machine learning and the mathematicalderivations that transform these principles into practical algorithms. Fol-lowing a presentation of the basics of the ﬁeld, the book covers a widearray of central topics that have not been addressed by previous text-books. These include a discussion of the computational complexity oflearning and the concepts of convexityand stability; important algorith-mic paradigms including stochastic gradient descent, neural networks,and structured output learning; and emerging theoretical concepts such asthe PAC-Bayes approach and compression-based bounds.D e s i g n e df o ran advanced undergraduate or beginning graduate course, the text makesthe

covers a widearray of central topics that have not been addressed by previous text-books. These include a discussion of the computational complexity oflearning and the concepts of convexityand stability; important algorith-mic paradigms including stochastic gradient descent, neural networks,and structured output learning; and emerging theoretical concepts such asthe PAC-Bayes approach and compression-based bounds.D e s i g n e df o ran advanced undergraduate or beginning graduate course, the text makesthe fundamentals and algorithms of machine learning accessible to stu-dents and nonexpert readers in statistics, computer science, mathematics,and engineering.Shai Shalev-Shwartz is an Associate Professor at the School of ComputerScience and Engineering at The Hebrew University, Israel.Shai Ben-David is a Professor in the School of Computer Science at theUniversity of Waterloo, Canada.

UNDERSTANDINGMACHINE LEARNINGFrom Theory toAlgorithmsShai Shalev-ShwartzThe Hebrew University, JerusalemShai Ben-DavidUniversity of Waterloo, Canada

32 Avenue of the Americas, New York, NY 10013-2473, USACambridge University Press is part of the University of Cambridge.It furthers the University’s mission bydisseminating knowledge in the pursuit ofeducation, learning and research at the highest international levels of excellence.www.cambridge.orgInformation on this title:www.cambridge.org/9781107057135c⃝Shai Shalev-Shwartz and Shai Ben-David 2014This publication is in copyright. Subject to statutory exceptionand to the provisions of relevant collective licensing agreements,no reproduction of any part may take place without the writtenpermission of Cambridge University Press.First published 2014Printed in the United States of AmericaAc a t a l o gr e c o r df o rt h i sp u b l i c a t i o ni sa v a i l a b l ef r o mt h eB r i t i s hL i b r a r yLibrary of Congress Cataloging in Publication DataISBN 978-1-107-05713-5 HardbackCambridge University Press has no responsibility for the persistence or accuracy ofURLs for external or third-party Internet Web

licensing agreements,no reproduction of any part may take place without the writtenpermission of Cambridge University Press.First published 2014Printed in the United States of AmericaAc a t a l o gr e c o r df o rt h i sp u b l i c a t i o ni sa v a i l a b l ef r o mt h eB r i t i s hL i b r a r yLibrary of Congress Cataloging in Publication DataISBN 978-1-107-05713-5 HardbackCambridge University Press has no responsibility for the persistence or accuracy ofURLs for external or third-party Internet Web sites referred to in this publication,and does not guarantee that any content on such Web sites is, or will remain,accurate or appropriate.

vii
Preface
The term machine learning refers to the automated detection of meaningful
patterns in data. In the past couple of decades it has become a common tool in
almost any task that requires information extraction from large data sets. We are
surrounded by a machine learning based technology: search engines learn how
to bring us the best results (while placing proﬁtable ads), anti-spam software
learns to ﬁlter our email messages, and credit card transactions are secured by
a software that learns how to detect frauds. Digital cameras learn to detect
faces and intelligent personal assistance applications on smart-phones learn to
recognize voice commands. Cars are equipped with accident prevention systems
that are built using machine learning algorithms. Machine learning is also widely
used in scientiﬁc applications such as bioinformatics, medicine, and astronomy.
One common feature of all of these applications is that, in contrast to more

traditional uses of computers, in these cases, due to the complexity of the patterns
that need to be detected, a human programmer cannot provide an explicit, ﬁne-
detailed speciﬁcation of how such tasks should be executed. Taking example from
intelligent beings, many of our skills are acquired or reﬁned through learning from
our experience (rather than following explicit instructions given to us). Machine
learning tools are concerned with endowing programs with the ability to “learn”
and adapt.

and adapt.
The ﬁrst goal of this book is to provide a rigorous, yet easy to follow, intro-
duction to the main concepts underlying machine learning: What is learning?
How can a machine learn? How do we quantify the resources needed to learn a
given concept? Is learning always possible? Can we know if the learning process
succeeded or failed?
The second goal of this book is to present several key machine learning algo-
rithms. We chose to present algorithms that on one hand are successfully used

and adapt.
in practice and on the other hand give a wide spectrum of diﬀerent learning
techniques. Additionally, we pay speciﬁc attention to algorithms appropriate for
large scale learning (a.k.a. “Big Data”), since in recent years, our world has be-
come increasingly “digitized” and the amount of data available for learning is
dramatically increasing. As a result, in many applications data is plentiful and
computation time is the main bottleneck. We therefore explicitly quantify both

the amount of data and the amount of computation time needed to learn a given
concept.
The book is divided into four parts. The ﬁrst part aims at giving an initial
rigorous answer to the fundamental questions of learning. We describe a gen-
eralization of Valiant’s Probably Approximately Correct (PAC) learning model,
which is a ﬁrst solid answer to the question “what is learning?”. We describe
the Empirical Risk Minimization (ERM), Structural Risk Minimization (SRM),

concept.
and Minimum Description Length (MDL) learning rules, which shows “how can
a machine learn”. We quantify the amount of data needed for learning using
the ERM, SRM, and MDL rules and show how learning might fail by deriving

viii
a “no-free-lunch” theorem. We also discuss how much computation time is re-
quired for learning. In the second part of the book we describe various learning
algorithms. For some of the algorithms, we ﬁrst present a more general learning
principle, and then show how the algorithm follows the principle. While the ﬁrst
two parts of the book focus on the PAC model, the third part extends the scope
by presenting a wider variety of learning models. Finally, the last part of the
book is devoted to advanced theory.
We made an attempt to keep the book as self-contained as possible. However,
the reader is assumed to be comfortable with basic notions of probability, linear
algebra, analysis, and algorithms. The ﬁrst three parts of the book are intended
for ﬁrst year graduate students in computer science, engineering, mathematics, or
statistics. It can also be accessible to undergraduate students with the adequate
background. The more advanced chapters can be used by researchers intending

to gather a deeper theoretical understanding.
The book is based on Introduction to Machine Learning courses taught by Shai
Shalev-Shwartz at the Hebrew University and by Shai Ben-David at the Univer-
sity of Waterloo. The ﬁrst draft of the book grew out of the lecture notes for
the course that was taught at the Hebrew University by Shai Shalev-Shwartz
during 2010–2013. We greatly appreciate the help of Ohad Shamir, who served
as a TA for the course in 2010, and of Alon Gonen, who served as a TA for the

course in 2011–2013. Ohad and Alon prepared few lecture notes and many of
the exercises. Alon, to whom we are indebted for his help throughout the entire
making of the book, has also prepared a solution manual.
We are deeply grateful for the most valuable work of Dana Rubinstein. Dana
has scientiﬁcally proofread and edited the manuscript, transforming it from
lecture-based chapters into ﬂuent and coherent text.
Special thanks to Amit Daniely, who helped us with a careful read of the

advanced part of the book and also wrote the advanced chapter on multiclass
learnability. We are also grateful for the members of a book reading club in
Jerusalem that have carefully read and constructively criticized every line of
the manuscript. The members of the reading club are: Maya Alroy, Yossi Arje-
vani, Aharon Birnbaum, Alon Cohen, Alon Gonen, Roi Livni, Ofer Meshi, Dan
Rosenbaum, Dana Rubinstein, Shahar Somin, Alon Vinnikov, and Yoav Wald.
We would also like to thank Gal Elidan, Amir Globerson, Nika Haghtalab, Shie

Mannor, Amnon Shashua, Nati Srebro, and Ruth Urner for helpful discussions.
Shai Shalev-Shwartz, Jerusalem, Israel
Shai Ben-David, Waterloo, Canada

Contents
Preface page vii
1 Introduction 19
1.1 What Is Learning? 19
1.2 When Do We Need Machine Learning? 21
1.3 Types of Learning 22
1.4 Relations to Other Fields 24
1.5 How to Read This Book 25
1.5.1 Possible Course Plans Based on This Book 26
1.6 Notation 27
Part I Foundations 31
2 A Gentle Start 33
2.1 A Formal Model – The Statistical Learning Framework 33
2.2 Empirical Risk Minimization 35
2.2.1 Something May Go Wrong – Overﬁtting 35
2.3 Empirical Risk Minimization with Inductive Bias 36
2.3.1 Finite Hypothesis Classes 37
2.4 Exercises 41
3 A Formal Learning Model 43
3.1 PAC Learning 43
3.2 A More General Learning Model 44
3.2.1 Releasing the Realizability Assumption – Agnostic PAC
Learning 45
3.2.2 The Scope of Learning Problems Modeled 47
3.3 Summary 49
3.4 Bibliographic Remarks 50
3.5 Exercises 50
4 Learning via Uniform Convergence 54
4.1 Uniform Convergence Is Suﬃcient for Learnability 54
4.2 Finite Classes Are Agnostic PAC Learnable 55

Understanding Machine Learning, c⃝2014 by Shai Shalev-Shwartz and Shai Ben-David
Personal use only. Not for distribution. Do not post.
Please link to http://www.cs.huji.ac.il/ ~shais/UnderstandingMachineLearning

x Contents
4.3 Summary 58
4.4 Bibliographic Remarks 58
4.5 Exercises 58
5 The Bias-Complexity Tradeoﬀ 60
5.1 The No-Free-Lunch Theorem 61
5.1.1 No-Free-Lunch and Prior Knowledge 63
5.2 Error Decomposition 64
5.3 Summary 65
5.4 Bibliographic Remarks 66
5.5 Exercises 66
6 The VC-Dimension 67
6.1 Inﬁnite-Size Classes Can Be Learnable 67
6.2 The VC-Dimension 68
6.3 Examples 70
6.3.1 Threshold Functions 70
6.3.2 Intervals 71
6.3.3 Axis Aligned Rectangles 71
6.3.4 Finite Classes 72
6.3.5 VC-Dimension and the Number of Parameters 72
6.4 The Fundamental Theorem of PAC learning 72
6.5 Proof of Theorem 6.7 73
6.5.1 Sauer’s Lemma and the Growth Function 73
6.5.2 Uniform Convergence for Classes of Small Eﬀective Size 75
6.6 Summary 78
6.7 Bibliographic remarks 78
6.8 Exercises 78
7 Nonuniform Learnability 83
7.1 Nonuniform Learnability 83
7.1.1 Characterizing Nonuniform Learnability 84
7.2 Structural Risk Minimization 85
7.3 Minimum Description Length and Occam’s Razor 89
7.3.1 Occam’s Razor 91

7.4 Other Notions of Learnability – Consistency 92
7.5 Discussing the Diﬀerent Notions of Learnability 93
7.5.1 The No-Free-Lunch Theorem Revisited 95
7.6 Summary 96
7.7 Bibliographic Remarks 97
7.8 Exercises 97
8 The Runtime of Learning 100
8.1 Computational Complexity of Learning 101

Contents xi
8.1.1 Formal Deﬁnition* 102
8.2 Implementing the ERM Rule 103
8.2.1 Finite Classes 104
8.2.2 Axis Aligned Rectangles 105
8.2.3 Boolean Conjunctions 106
8.2.4 Learning 3-Term DNF 107
8.3 Eﬃciently Learnable, but Not by a Proper ERM 107
8.4 Hardness of Learning* 108
8.5 Summary 110
8.6 Bibliographic Remarks 110
8.7 Exercises 110
Part II From Theory to Algorithms 115
9 Linear Predictors 117
9.1 Halfspaces 118
9.1.1 Linear Programming for the Class of Halfspaces 119
9.1.2 Perceptron for Halfspaces 120
9.1.3 The VC Dimension of Halfspaces 122
9.2 Linear Regression 123
9.2.1 Least Squares 124
9.2.2 Linear Regression for Polynomial Regression Tasks 125
9.3 Logistic Regression 126
9.4 Summary 128
9.5 Bibliographic Remarks 128
9.6 Exercises 128
10 Boosting 130
10.1 Weak Learnability 131
10.1.1 Eﬃcient Implementation of ERM for Decision Stumps 133
10.2 AdaBoost 134
10.3 Linear Combinations of Base Hypotheses 137
10.3.1 The VC-Dimension of L(B,T) 139
10.4 AdaBoost for Face Recognition 140
10.5 Summary 141

10.6 Bibliographic Remarks 141
10.7 Exercises 142
11 Model Selection and Validation 144
11.1 Model Selection Using SRM 145
11.2 Validation 146
11.2.1 Hold Out Set 146
11.2.2 Validation for Model Selection 147
11.2.3 The Model-Selection Curve 148

xii Contents
11.2.4k-Fold Cross Validation 149
11.2.5 Train-Validation-Test Split 150
11.3 What to Do If Learning Fails 151
11.4 Summary 154
11.5 Exercises 154
12 Convex Learning Problems 156
12.1 Convexity, Lipschitzness, and Smoothness 156
12.1.1 Convexity 156
12.1.2 Lipschitzness 160
12.1.3 Smoothness 162
12.2 Convex Learning Problems 163
12.2.1 Learnability of Convex Learning Problems 164
12.2.2 Convex-Lipschitz/Smooth-Bounded Learning Problems 166
12.3 Surrogate Loss Functions 167
12.4 Summary 168
12.5 Bibliographic Remarks 169
12.6 Exercises 169
13 Regularization and Stability 171
13.1 Regularized Loss Minimization 171
13.1.1 Ridge Regression 172
13.2 Stable Rules Do Not Overﬁt 173
13.3 Tikhonov Regularization as a Stabilizer 174
13.3.1 Lipschitz Loss 176
13.3.2 Smooth and Nonnegative Loss 177
13.4 Controlling the Fitting-Stability Tradeoﬀ 178
13.5 Summary 180
13.6 Bibliographic Remarks 180
13.7 Exercises 181
14 Stochastic Gradient Descent 184
14.1 Gradient Descent 185

14.1.1 Analysis of GD for Convex-Lipschitz Functions 186
14.2 Subgradients 188
14.2.1 Calculating Subgradients 189
14.2.2 Subgradients of Lipschitz Functions 190
14.2.3 Subgradient Descent 190
14.3 Stochastic Gradient Descent (SGD) 191
14.3.1 Analysis of SGD for Convex-Lipschitz-Bounded Functions 191
14.4 Variants 193
14.4.1 Adding a Projection Step 193
14.4.2 Variable Step Size 194
14.4.3 Other Averaging Techniques 195

Contents xiii
14.4.4 Strongly Convex Functions* 195
14.5 Learning with SGD 196
14.5.1 SGD for Risk Minimization 196
14.5.2 Analyzing SGD for Convex-Smooth Learning Problems 198
14.5.3 SGD for Regularized Loss Minimization 199
14.6 Summary 200
14.7 Bibliographic Remarks 200
14.8 Exercises 201
15 Support Vector Machines 202
15.1 Margin and Hard-SVM 202
15.1.1 The Homogenous Case 205
15.1.2 The Sample Complexity of Hard-SVM 205
15.2 Soft-SVM and Norm Regularization 206
15.2.1 The Sample Complexity of Soft-SVM 208
15.2.2 Margin and Norm-Based Bounds versus Dimension 208
15.2.3 The Ramp Loss* 209
15.3 Optimality Conditions and “Support Vectors”* 210
15.4 Duality* 211
15.5 Implementing Soft-SVM Using SGD 212
15.6 Summary 213
15.7 Bibliographic Remarks 213
15.8 Exercises 214
16 Kernel Methods 215
16.1 Embeddings into Feature Spaces 215
16.2 The Kernel Trick 217
16.2.1 Kernels as a Way to Express Prior Knowledge 221
16.2.2 Characterizing Kernel Functions* 222
16.3 Implementing Soft-SVM with Kernels 222

16.4 Summary 224
16.5 Bibliographic Remarks 225
16.6 Exercises 225
17 Multiclass, Ranking, and Complex Prediction Problems 227
17.1 One-versus-All and All-Pairs 227
17.2 Linear Multiclass Predictors 230
17.2.1 How to Construct Ψ 230
17.2.2 Cost-Sensitive Classiﬁcation 232
17.2.3 ERM 232
17.2.4 Generalized Hinge Loss 233
17.2.5 Multiclass SVM and SGD 234
17.3 Structured Output Prediction 236
17.4 Ranking 238

xiv Contents
17.4.1 Linear Predictors for Ranking 240
17.5 Bipartite Ranking and Multivariate Performance Measures 243
17.5.1 Linear Predictors for Bipartite Ranking 245
17.6 Summary 247
17.7 Bibliographic Remarks 247
17.8 Exercises 248
18 Decision Trees 250
18.1 Sample Complexity 251
18.2 Decision Tree Algorithms 252
18.2.1 Implementations of the Gain Measure 253
18.2.2 Pruning 254
18.2.3 Threshold-Based Splitting Rules for Real-Valued Features 255
18.3 Random Forests 255
18.4 Summary 256
18.5 Bibliographic Remarks 256
18.6 Exercises 256
19 Nearest Neighbor 258
19.1kNearest Neighbors 258
19.2 Analysis 259
19.2.1 A Generalization Bound for the 1-NN Rule 260
19.2.2 The “Curse of Dimensionality” 263
19.3 Eﬃcient Implementation* 264
19.4 Summary 264
19.5 Bibliographic Remarks 264
19.6 Exercises 265
20 Neural Networks 268
20.1 Feedforward Neural Networks 269
20.2 Learning Neural Networks 270
20.3 The Expressive Power of Neural Networks 271
20.3.1 Geometric Intuition 273

20.4 The Sample Complexity of Neural Networks 274
20.5 The Runtime of Learning Neural Networks 276
20.6 SGD and Backpropagation 277
20.7 Summary 281
20.8 Bibliographic Remarks 281
20.9 Exercises 282
Part III Additional Learning Models 285
21 Online Learning 287
21.1 Online Classiﬁcation in the Realizable Case 288

Contents xv
21.1.1 Online Learnability 290
21.2 Online Classiﬁcation in the Unrealizable Case 294
21.2.1 Weighted-Majority 295
21.3 Online Convex Optimization 300
21.4 The Online Perceptron Algorithm 301
21.5 Summary 304
21.6 Bibliographic Remarks 305
21.7 Exercises 305
22 Clustering 307
22.1 Linkage-Based Clustering Algorithms 310
22.2k-Means and Other Cost Minimization Clusterings 311
22.2.1 The k-Means Algorithm 313
22.3 Spectral Clustering 315
22.3.1 Graph Cut 315
22.3.2 Graph Laplacian and Relaxed Graph Cuts 315
22.3.3 Unnormalized Spectral Clustering 317
22.4 Information Bottleneck* 317
22.5 A High Level View of Clustering 318
22.6 Summary 320
22.7 Bibliographic Remarks 320
22.8 Exercises 320
23 Dimensionality Reduction 323
23.1 Principal Component Analysis (PCA) 324
23.1.1 A More Eﬃcient Solution for the Case d≫m 326
23.1.2 Implementation and Demonstration 326
23.2 Random Projections 329
23.3 Compressed Sensing 330
23.3.1 Proofs* 333
23.4 PCA or Compressed Sensing? 338
23.5 Summary 338

23.6 Bibliographic Remarks 339
23.7 Exercises 339
24 Generative Models 342
24.1 Maximum Likelihood Estimator 343
24.1.1 Maximum Likelihood Estimation for Continuous Ran-
dom Variables 344
24.1.2 Maximum Likelihood and Empirical Risk Minimization 345
24.1.3 Generalization Analysis 345
24.2 Naive Bayes 347
24.3 Linear Discriminant Analysis 347
24.4 Latent Variables and the EM Algorithm 348

xvi Contents
24.4.1 EM as an Alternate Maximization Algorithm 350
24.4.2 EM for Mixture of Gaussians (Soft k-Means) 352
24.5 Bayesian Reasoning 353
24.6 Summary 355
24.7 Bibliographic Remarks 355
24.8 Exercises 356
25 Feature Selection and Generation 357
25.1 Feature Selection 358
25.1.1 Filters 359
25.1.2 Greedy Selection Approaches 360
25.1.3 Sparsity-Inducing Norms 363
25.2 Feature Manipulation and Normalization 365
25.2.1 Examples of Feature Transformations 367
25.3 Feature Learning 368
25.3.1 Dictionary Learning Using Auto-Encoders 368
25.4 Summary 370
25.5 Bibliographic Remarks 371
25.6 Exercises 371
Part IV Advanced Theory 373
26 Rademacher Complexities 375
26.1 The Rademacher Complexity 375
26.1.1 Rademacher Calculus 379
26.2 Rademacher Complexity of Linear Classes 382
26.3 Generalization Bounds for SVM 383
26.4 Generalization Bounds for Predictors with Low ℓ1Norm 386
26.5 Bibliographic Remarks 386
27 Covering Numbers 388
27.1 Covering 388
27.1.1 Properties 388

27.2 From Covering to Rademacher Complexity via Chaining 389
27.3 Bibliographic Remarks 391
28 Proof of the Fundamental Theorem of Learning Theory 392
28.1 The Upper Bound for the Agnostic Case 392
28.2 The Lower Bound for the Agnostic Case 393
28.2.1 Showing That m(ϵ,δ)≥0.5 log(1/(4δ))/ϵ2393
28.2.2 Showing That m(ϵ,1/8)≥8d/ϵ2395
28.3 The Upper Bound for the Realizable Case 398
28.3.1 From ϵ-Nets to PAC Learnability 401

Contents xvii
29 Multiclass Learnability 402
29.1 The Natarajan Dimension 402
29.2 The Multiclass Fundamental Theorem 403
29.2.1 On the Proof of Theorem 29.3 403
29.3 Calculating the Natarajan Dimension 404
29.3.1 One-versus-All Based Classes 404
29.3.2 General Multiclass-to-Binary Reductions 405
29.3.3 Linear Multiclass Predictors 405
29.4 On Good and Bad ERMs 406
29.5 Bibliographic Remarks 408
29.6 Exercises 409
30 Compression Bounds 410
30.1 Compression Bounds 410
30.2 Examples 412
30.2.1 Axis Aligned Rectangles 412
30.2.2 Halfspaces 412
30.2.3 Separating Polynomials 413
30.2.4 Separation with Margin 414
30.3 Bibliographic Remarks 414
31 PAC-Bayes 415
31.1 PAC-Bayes Bounds 415
31.2 Bibliographic Remarks 417
31.3 Exercises 417
Appendix A Technical Lemmas 419
Appendix B Measure Concentration 422
Appendix C Linear Algebra 430
Notes 435
References 437
Index 447

1 Introduction
The subject of this book is automated learning, or, as we will more often call
it, Machine Learning (ML). That is, we wish to program computers so that
they can “learn” from input available to them. Roughly speaking, learning is
the process of converting experience into expertise or knowledge. The input to
a learning algorithm is training data, representing experience, and the output
is some expertise, which usually takes the form of another computer program
that can perform some task. Seeking a formal-mathematical understanding of
this concept, we’ll have to be more explicit about what we mean by each of the
involved terms: What is the training data our programs will access? How can
the process of learning be automated? How can we evaluate the success of such
a process (namely, the quality of the output of a learning program)?
1.1 What Is Learning?
Let us begin by considering a couple of examples from naturally occurring ani-

mal learning. Some of the most fundamental issues in ML arise already in that
context, which we are all familiar with.
Bait Shyness – Rats Learning to Avoid Poisonous Baits: When rats encounter
food items with novel look or smell, they will ﬁrst eat very small amounts, and
subsequent feeding will depend on the ﬂavor of the food and its physiological
eﬀect. If the food produces an ill eﬀect, the novel food will often be associated
with the illness, and subsequently, the rats will not eat it. Clearly, there is a

learning mechanism in play here – the animal used past experience with some
food to acquire expertise in detecting the safety of this food. If past experience
with the food was negatively labeled, the animal predicts that it will also have
a negative eﬀect when encountered in the future.
Inspired by the preceding example of successful learning, let us demonstrate a
typical machine learning task. Suppose we would like to program a machine that
learns how to ﬁlter spam e-mails. A naive solution would be seemingly similar

to the way rats learn how to avoid poisonous baits. The machine will simply
memorize all previous e-mails that had been labeled as spam e-mails by the
human user. When a new e-mail arrives, the machine will search for it in the set

20 Introduction
of previous spam e-mails. If it matches one of them, it will be trashed. Otherwise,
it will be moved to the user’s inbox folder.
While the preceding “learning by memorization” approach is sometimes use-
ful, it lacks an important aspect of learning systems – the ability to label unseen
e-mail messages. A successful learner should be able to progress from individual
examples to broader generalization . This is also referred to as inductive reasoning
orinductive inference . In the bait shyness example presented previously, after
the rats encounter an example of a certain type of food, they apply their attitude
toward it on new, unseen examples of food of similar smell and taste. To achieve
generalization in the spam ﬁltering task, the learner can scan the previously seen
e-mails, and extract a set of words whose appearance in an e-mail message is
indicative of spam. Then, when a new e-mail arrives, the machine can check

whether one of the suspicious words appears in it, and predict its label accord-
ingly. Such a system would potentially be able correctly to predict the label of
unseen e-mails.
However, inductive reasoning might lead us to false conclusions. To illustrate
this, let us consider again an example from animal learning.
Pigeon Superstition: In an experiment performed by the psychologist B. F. Skinner,
he placed a bunch of hungry pigeons in a cage. An automatic mechanism had

been attached to the cage, delivering food to the pigeons at regular intervals
with no reference whatsoever to the birds’ behavior. The hungry pigeons went
around the cage, and when food was ﬁrst delivered, it found each pigeon engaged
in some activity (pecking, turning the head, etc.). The arrival of food reinforced
each bird’s speciﬁc action, and consequently, each bird tended to spend some
more time doing that very same action. That, in turn, increased the chance that

the next random food delivery would ﬁnd each bird engaged in that activity
again. What results is a chain of events that reinforces the pigeons’ association
of the delivery of the food with whatever chance actions they had been perform-
ing when it was ﬁrst delivered. They subsequently continue to perform these
same actions diligently.1
What distinguishes learning mechanisms that result in superstition from useful
learning? This question is crucial to the development of automated learners.

While human learners can rely on common sense to ﬁlter out random meaningless
learning conclusions, once we export the task of learning to a machine, we must
provide well deﬁned crisp principles that will protect the program from reaching
senseless or useless conclusions. The development of such principles is a central
goal of the theory of machine learning.
What, then, made the rats’ learning more successful than that of the pigeons?
As a ﬁrst step toward answering this question, let us have a closer look at the

bait shyness phenomenon in rats.
Bait Shyness revisited – rats fail to acquire conditioning between food and
electric shock or between sound and nausea: The bait shyness mechanism in
1See: http://psychclassics.yorku.ca/Skinner/Pigeon

rats turns out to be more complex than what one may expect. In experiments
carried out by Garcia (Garcia & Koelling 1996), it was demonstrated that if the
unpleasant stimulus that follows food consumption is replaced by, say, electrical
shock (rather than nausea), then no conditioning occurs. Even after repeated
trials in which the consumption of some food is followed by the administration of
unpleasant electrical shock, the rats do not tend to avoid that food. Similar failure
of conditioning occurs when the characteristic of the food that implies nausea
(such as taste or smell) is replaced by a vocal signal. The rats seem to have
some “built in” prior knowledge telling them that, while temporal correlation
between food and nausea can be causal, it is unlikely that there would be a
causal relationship between food consumption and electrical shocks or between
sounds and nausea.
We conclude that one distinguishing feature between the bait shyness learning

and the pigeon superstition is the incorporation of prior knowledge that biases
the learning mechanism. This is also referred to as inductive bias . The pigeons in
the experiment are willing to adopt anyexplanation for the occurrence of food.
However, the rats “know” that food cannot cause an electric shock and that the
co-occurrence of noise with some food is not likely to aﬀect the nutritional value
of that food. The rats’ learning process is biased toward detecting some kind of

patterns while ignoring other temporal correlations between events.
It turns out that the incorporation of prior knowledge, biasing the learning
process, is inevitable for the success of learning algorithms (this is formally stated
and proved as the “No-Free-Lunch theorem” in Chapter 5). The development of
tools for expressing domain expertise, translating it into a learning bias, and
quantifying the eﬀect of such a bias on the success of learning is a central theme

of the theory of machine learning. Roughly speaking, the stronger the prior
knowledge (or prior assumptions) that one starts the learning process with, the
easier it is to learn from further examples. However, the stronger these prior
assumptions are, the less ﬂexible the learning is – it is bound, a priori, by the
commitment to these assumptions. We shall discuss these issues explicitly in
Chapter 5.
1.2 When Do We Need Machine Learning?
When do we need machine learning rather than directly program our computers

Chapter 5.
to carry out the task at hand? Two aspects of a given problem may call for the
use of programs that learn and improve on the basis of their “experience”: the
problem’s complexity and the need for adaptivity.
Tasks That Are Too Complex to Program.
•Tasks Performed by Animals/Humans: There are numerous tasks that
we human beings perform routinely, yet our introspection concern-
ing how we do them is not suﬃciently elaborate to extract a well

22 Introduction
deﬁned program. Examples of such tasks include driving, speech
recognition, and image understanding. In all of these tasks, state
of the art machine learning programs, programs that “learn from
their experience,” achieve quite satisfactory results, once exposed
to suﬃciently many training examples.
•Tasks beyond Human Capabilities: Another wide family of tasks that
beneﬁt from machine learning techniques are related to the analy-
sis of very large and complex data sets: astronomical data, turning
medical archives into medical knowledge, weather prediction, anal-
ysis of genomic data, Web search engines, and electronic commerce.
With more and more available digitally recorded data, it becomes
obvious that there are treasures of meaningful information buried
in data archives that are way too large and too complex for humans
to make sense of. Learning to detect meaningful patterns in large
and complex data sets is a promising domain in which the combi-

nation of programs that learn with the almost unlimited memory
capacity and ever increasing processing speed of computers opens
up new horizons.
Adaptivity. One limiting feature of programmed tools is their rigidity – once
the program has been written down and installed, it stays unchanged.
However, many tasks change over time or from one user to another.
Machine learning tools – programs whose behavior adapts to their input
data – oﬀer a solution to such issues; they are, by nature, adaptive

to changes in the environment they interact with. Typical successful
applications of machine learning to such problems include programs that
decode handwritten text, where a ﬁxed program can adapt to variations
between the handwriting of diﬀerent users; spam detection programs,
adapting automatically to changes in the nature of spam e-mails; and
speech recognition programs.
1.3 Types of Learning
Learning is, of course, a very wide domain. Consequently, the ﬁeld of machine

learning has branched into several subﬁelds dealing with diﬀerent types of learn-
ing tasks. We give a rough taxonomy of learning paradigms, aiming to provide
some perspective of where the content of this book sits within the wide ﬁeld of
machine learning.
We describe four parameters along which learning paradigms can be classiﬁed.
Supervised versus Unsupervised Since learning involves an interaction be-
tween the learner and the environment, one can divide learning tasks

according to the nature of that interaction. The ﬁrst distinction to note
is the diﬀerence between supervised and unsupervised learning. As an

1.3 Types of Learning 23
illustrative example, consider the task of learning to detect spam e-mail
versus the task of anomaly detection. For the spam detection task, we
consider a setting in which the learner receives training e-mails for which
the label spam/not-spam is provided. On the basis of such training the
learner should ﬁgure out a rule for labeling a newly arriving e-mail mes-
sage. In contrast, for the task of anomaly detection, all the learner gets
as training is a large body of e-mail messages (with no labels) and the
learner’s task is to detect “unusual” messages.
More abstractly, viewing learning as a process of “using experience
to gain expertise,” supervised learning describes a scenario in which the
“experience,” a training example, contains signiﬁcant information (say,
thespam/not-spam labels) that is missing in the unseen “test examples”
to which the learned expertise is to be applied. In this setting, the ac-
quired expertise is aimed to predict that missing information for the test

data. In such cases, we can think of the environment as a teacher that
“supervises” the learner by providing the extra information (labels). In
unsupervised learning, however, there is no distinction between training
and test data. The learner processes input data with the goal of coming
up with some summary, or compressed version of that data. Clustering
a data set into subsets of similar objets is a typical example of such a
task.
There is also an intermediate learning setting in which, while the

task.
training examples contain more information than the test examples, the
learner is required to predict even more information for the test exam-
ples. For example, one may try to learn a value function that describes for
each setting of a chess board the degree by which White’s position is bet-
ter than the Black’s. Yet, the only information available to the learner at
training time is positions that occurred throughout actual chess games,
labeled by who eventually won that game. Such learning frameworks are

mainly investigated under the title of reinforcement learning .
Active versus Passive Learners Learning paradigms can vary by the role
played by the learner. We distinguish between “active” and “passive”
learners. An active learner interacts with the environment at training
time, say, by posing queries or performing experiments, while a passive
learner only observes the information provided by the environment (or
the teacher) without inﬂuencing or directing it. Note that the learner of a

spam ﬁlter is usually passive – waiting for users to mark the e-mails com-
ing to them. In an active setting, one could imagine asking users to label
speciﬁc e-mails chosen by the learner, or even composed by the learner, to
enhance its understanding of what
spam is.
Helpfulness of the Teacher When one thinks about human learning, of a
baby at home or a student at school, the process often involves a helpful
teacher, who is trying to feed the learner with the information most use-

24 Introduction
ful for achieving the learning goal. In contrast, when a scientist learns
about nature, the environment, playing the role of the teacher, can be
best thought of as passive – apples drop, stars shine, and the rain falls
without regard to the needs of the learner. We model such learning sce-
narios by postulating that the training data (or the learner’s experience)
is generated by some random process. This is the basic building block in
the branch of “statistical learning.” Finally, learning also occurs when
the learner’s input is generated by an adversarial “teacher.” This may be
the case in the spam ﬁltering example (if the spammer makes an eﬀort
to mislead the spam ﬁltering designer) or in learning to detect fraud.
One also uses an adversarial teacher model as a worst-case scenario,
when no milder setup can be safely assumed. If you can learn against an
adversarial teacher, you are guaranteed to succeed interacting any odd
teacher.

teacher.
Online versus Batch Learning Protocol The last parameter we mention is
the distinction between situations in which the learner has to respond
online, throughout the learning process, and settings in which the learner
has to engage the acquired expertise only after having a chance to process
large amounts of data. For example, a stockbroker has to make daily
decisions, based on the experience collected so far. He may become an
expert over time, but might have made costly mistakes in the process. In

teacher.
contrast, in many data mining settings, the learner – the data miner –
has large amounts of training data to play with before having to output
conclusions.
In this book we shall discuss only a subset of the possible learning paradigms.
Our main focus is on supervised statistical batch learning with a passive learner
(for example, trying to learn how to generate patients’ prognoses, based on large
archives of records of patients that were independently collected and are already

labeled by the fate of the recorded patients). We shall also brieﬂy discuss online
learning and batch unsupervised learning (in particular, clustering).
1.4 Relations to Other Fields
As an interdisciplinary ﬁeld, machine learning shares common threads with the
mathematical ﬁelds of statistics, information theory, game theory, and optimiza-
tion. It is naturally a subﬁeld of computer science, as our goal is to program
machines so that they will learn. In a sense, machine learning can be viewed as

a branch of AI (Artiﬁcial Intelligence), since, after all, the ability to turn expe-
rience into expertise or to detect meaningful patterns in complex sensory data
is a cornerstone of human (and animal) intelligence. However, one should note
that, in contrast with traditional AI, machine learning is not trying to build
automated imitation of intelligent behavior, but rather to use the strengths and

special abilities of computers to complement human intelligence, often perform-
ing tasks that fall way beyond human capabilities. For example, the ability to
scan and process huge databases allows machine learning programs to detect
patterns that are outside the scope of human perception.
The component of experience, or training, in machine learning often refers
to data that is randomly generated. The task of the learner is to process such
randomly generated examples toward drawing conclusions that hold for the en-
vironment from which these examples are picked. This description of machine
learning highlights its close relationship with statistics. Indeed there is a lot in
common between the two disciplines, in terms of both the goals and techniques
used. There are, however, a few signiﬁcant diﬀerences of emphasis; if a doctor
comes up with the hypothesis that there is a correlation between smoking and

heart disease, it is the statistician’s role to view samples of patients and check
the validity of that hypothesis (this is the common statistical task of hypothe-
sis testing). In contrast, machine learning aims to use the data gathered from
samples of patients to come up with a description of the causes of heart disease.
The hope is that automated techniques may be able to ﬁgure out meaningful
patterns (or hypotheses) that may have been missed by the human observer.
In contrast with traditional statistics, in machine learning in general, and

in this book in particular, algorithmic considerations play a major role. Ma-
chine learning is about the execution of learning by computers; hence algorith-
mic issues are pivotal. We develop algorithms to perform the learning tasks and
are concerned with their computational eﬃciency. Another diﬀerence is that
while statistics is often interested in asymptotic behavior (like the convergence
of sample-based statistical estimates as the sample sizes grow to inﬁnity), the
theory of machine learning focuses on ﬁnite sample bounds. Namely, given the

size of available samples, machine learning theory aims to ﬁgure out the degree
of accuracy that a learner can expect on the basis of such samples.
There are further diﬀerences between these two disciplines, of which we shall
mention only one more here. While in statistics it is common to work under the
assumption of certain presubscribed data models (such as assuming the normal-
ity of data-generating distributions, or the linearity of functional dependencies),
in machine learning the emphasis is on working under a “distribution-free” set-

ting, where the learner assumes as little as possible about the nature of the
data distribution and allows the learning algorithm to ﬁgure out which models
best approximate the data-generating process. A precise discussion of this issue
requires some technical preliminaries, and we will come back to it later in the
book, and in particular in Chapter 5.
1.5 How to Read This Book
The ﬁrst part of the book provides the basic theoretical principles that underlie
machine learning (ML). In a sense, this is the foundation upon which the rest

26 Introduction
of the book is built. This part could serve as a basis for a minicourse on the
theoretical foundations of ML.
The second part of the book introduces the most commonly used algorithmic
approaches to supervised machine learning. A subset of these chapters may also
be used for introducing machine learning in a general AI course to computer
science, Math, or engineering students.
The third part of the book extends the scope of discussion from statistical
classiﬁcation to other learning models. It covers online learning, unsupervised
learning, dimensionality reduction, generative models, and feature learning.
The fourth part of the book, Advanced Theory, is geared toward readers who
have interest in research and provides the more technical mathematical tech-
niques that serve to analyze and drive forward the ﬁeld of theoretical machine
learning.
The Appendixes provide some technical tools used in the book. In particular,
we list basic results from measure concentration and linear algebra.

learning.
A few sections are marked by an asterisk, which means they are addressed to
more advanced students. Each chapter is concluded with a list of exercises. A
solution manual is provided in the course Web site.
1.5.1 Possible Course Plans Based on This Book
A 14 Week Introduction Course for Graduate Students:
1. Chapters 2–4.
2. Chapter 9 (without the VC calculation).
3. Chapters 5–6 (without proofs).
4. Chapter 10.
5. Chapters 7, 11 (without proofs).
6. Chapters 12, 13 (with some of the easier proofs).
7. Chapter 14 (with some of the easier proofs).

8. Chapter 15.
9. Chapter 16.
10. Chapter 18.
11. Chapter 22.
12. Chapter 23 (without proofs for compressed sensing).
13. Chapter 24.
14. Chapter 25.
A 14 Week Advanced Course for Graduate Students:
1. Chapters 26, 27.
2. (continued)
3. Chapters 6, 28.
4. Chapter 7.
5. Chapter 31.

6. Chapter 30.
7. Chapters 12, 13.
8. Chapter 14.
9. Chapter 8.
10. Chapter 17.
11. Chapter 29.
12. Chapter 19.
13. Chapter 20.
14. Chapter 21.
1.6 Notation
Most of the notation we use throughout the book is either standard or deﬁned
on the spot. In this section we describe our main conventions and provide a
table summarizing our notation (Table 1.1). The reader is encouraged to skip
this section and return to it if during the reading of the book some notation is
unclear.
We denote scalars and abstract objects with lowercase letters (e.g. xandλ).
Often, we would like to emphasize that some object is a vector and then we
use boldface letters (e.g. xandλ). Theith element of a vector xis denoted
byxi. We use uppercase letters to denote matrices, sets, and sequences. The
meaning should be clear from the context. As we will see momentarily, the input
of a learning algorithm is a sequence of training examples. We denote by zan
abstract example and by S=z1,...,zma sequence of mexamples. Historically,

Sis often referred to as a training set; however, we will always assume that Sis
asequence rather than a set. A sequence of mvectors is denoted by x1,...,xm.
Theith element of xtis denoted by xt,i.
Throughout the book, we make use of basic notions from probability. We
denote byDa distribution over some set,2for example, Z. We use the notation
z∼D to denote that zis sampled according to D. Given a random variable
f:Z→R, its expected value is denoted by Ez∼D[f(z)]. We sometimes use the
shorthand E[f] when the dependence on zis clear from the context. For f:Z→

{true,false}we also use Pz∼D[f(z)] to denoteD({z:f(z) =true}). In the
next chapter we will also introduce the notation Dmto denote the probability
overZminduced by sampling ( z1,...,zm) where each point ziis sampled from
Dindependently of the other points.
In general, we have made an eﬀort to avoid asymptotic notation. However, we
occasionally use it to clarify the main results. In particular, given f:R→R+
andg:R→R+we writef=O(g) if there exist x0,α∈R+such that for all

x>x 0we havef(x)≤αg(x). We write f=o(g) if for every α>0 there exists
2To be mathematically precise, Dshould be deﬁned over some σ-algebra of subsets of Z.
The user who is not familiar with measure theory can skip the few footnotes and remarks
regarding more formal measurability deﬁnitions and assumptions.

28 Introduction
Table 1.1 Summary of notation
symbol meaning
R the set of real numbers
Rdthe set ofd-dimensional vectors over R
R+ the set of non-negative real numbers
N the set of natural numbers
O,o, Θ,ω,Ω,˜O asymptotic notation (see text)
1[Boolean expression] indicator function (equals 1 if expression is true and 0 o.w.)
[a]+= max{0,a}
[n] the set {1,...,n}(forn∈N)
x,v,w (column) vectors
xi,vi,wi theith element of a vector
⟨x,v⟩ =∑d
i=1xivi(inner product)
∥x∥2or∥x∥ =√
⟨x,x⟩(theℓ2norm of x)
∥x∥1 =∑d
i=1|xi|(theℓ1norm of x)
∥x∥∞ = maxi|xi|(theℓ∞norm of x)
∥x∥0 the number of nonzero elements of x
A∈Rd,kad×kmatrix over R
A⊤the transpose of A
Ai,j the (i,j) element of A
x x⊤thed×dmatrixAs.t.Ai,j=xixj(where x∈Rd)
x1,...,xm a sequence of mvectors
xi,j thejth element of the ith vector in the sequence
w(1),...,w(T)the values of a vector wduring an iterative algorithm
w(t)
i theith element of the vector w(t)
X instances domain (a set)
Y labels domain (a set)
Z examples domain (a set)
H hypothesis class (a set)

w(t)
ℓ:H×Z→R+ loss function
D a distribution over some set (usually over Zor overX)
D(A) the probability of a set A⊆Zaccording toD
z∼D samplingzaccording toD
S=z1,...,zm a sequence of mexamples
S∼DmsamplingS=z1,...,zmi.i.d. according to D
P,E probability and expectation of a random variable
Pz∼D[f(z)] =D({z:f(z) =true}) forf:Z→{true,false}
Ez∼D[f(z)] expectation of the random variable f:Z→R
N(µ,C) Gaussian distribution with expectation µand covariance C
f′(x) the derivative of a function f:R→Ratx

f′′(x) the second derivative of a function f:R→Ratx
∂f(w)
∂withe partial derivative of a function f:Rd→Ratww.r.t.wi
∇f(w) the gradient of a function f:Rd→Ratw
∂f(w) the diﬀerential set of a function f:Rd→Ratw
minx∈Cf(x) = min{f(x) :x∈C}(minimal value of foverC)
maxx∈Cf(x) = max{f(x) :x∈C}(maximal value of foverC)
argminx∈Cf(x) the set{x∈C:f(x) = minz∈Cf(z)}
argmaxx∈Cf(x) the set{x∈C:f(x) = maxz∈Cf(z)}
log the natural logarithm

1.6 Notation 29
x0such that for all x > x 0we havef(x)≤αg(x). We write f= Ω(g) if there
existx0,α∈R+such that for all x > x 0we havef(x)≥αg(x). The notation
f=ω(g) is deﬁned analogously. The notation f= Θ(g) means that f=O(g)
andg=O(f). Finally, the notation f=˜O(g) means that there exists k∈N
such thatf(x) =O(g(x) logk(g(x))).
The inner product between vectors xandwis denoted by⟨x,w⟩. Whenever we
do not specify the vector space we assume that it is the d-dimensional Euclidean
space and then⟨x,w⟩=∑d
i=1xiwi. The Euclidean (or ℓ2) norm of a vector wis
∥w∥2=√
⟨w,w⟩. We omit the subscript from the ℓ2norm when it is clear from
the context. We also use other ℓpnorms,∥w∥p= (∑
i|wi|p)1/p, and in particular
∥w∥1=∑
i|wi|and∥w∥∞= maxi|wi|.
We use the notation min x∈Cf(x) to denote the minimum value of the set
{f(x) :x∈C}. To be mathematically more precise, we should use inf x∈Cf(x)
whenever the minimum is not achievable. However, in the context of this book

∥w∥2=√
∥w∥1=∑
the distinction between inﬁmum and minimum is often of little interest. Hence,
to simplify the presentation, we sometimes use the min notation even when inf
is more adequate. An analogous remark applies to max versus sup.

2 A Gentle Start
Let us begin our mathematical analysis by showing how successful learning can be
achieved in a relatively simpliﬁed setting. Imagine you have just arrived in some
small Paciﬁc island. You soon ﬁnd out that papayas are a signiﬁcant ingredient
in the local diet. However, you have never before tasted papayas. You have to
learn how to predict whether a papaya you see in the market is tasty or not.
First, you need to decide which features of a papaya your prediction should be
based on. On the basis of your previous experience with other fruits, you decide
to use two features: the papaya’s color, ranging from dark green, through orange
and red to dark brown, and the papaya’s softness, ranging from rock hard to
mushy. Your input for ﬁguring out your prediction rule is a sample of papayas
that you have examined for color and softness and then tasted and found out
whether they were tasty or not. Let us analyze this task as a demonstration of
the considerations involved in learning problems.

Our ﬁrst step is to describe a formal model aimed to capture such learning
tasks.
2.1 A Formal Model – The Statistical Learning Framework
•The learner’s input: In the basic statistical learning setting, the learner has
access to the following:
– Domain set: An arbitrary set, X. This is the set of objects that we
may wish to label. For example, in the papaya learning problem men-
tioned before, the domain set will be the set of all papayas. Usually,
these domain points will be represented by a vector of features (like

tasks.
the papaya’s color and softness). We also refer to domain points as
instances and toXas instance space.
– Label set: For our current discussion, we will restrict the label set to
be a two-element set, usually {0,1}or{−1,+1}. LetYdenote our
set of possible labels. For our papayas example, let Ybe{0,1}, where
1 represents being tasty and 0 stands for being not-tasty.
– Training data: S= ((x1,y1)...(xm,ym)) is a ﬁnite sequence of pairs in
X×Y : that is, a sequence of labeled domain points. This is the input

that the learner has access to (like a set of papayas that have been

34 A Gentle Start
tasted and their color, softness, and tastiness). Such labeled examples
are often called training examples . We sometimes also refer to Sas a
training set .1
•The learner’s output: The learner is requested to output a prediction rule ,
h:X→Y . This function is also called a predictor , ahypothesis , or a clas-
siﬁer . The predictor can be used to predict the label of new domain points.
In our papayas example, it is a rule that our learner will employ to predict
whether future papayas he examines in the farmers’ market are going to
be tasty or not. We use the notation A(S) to denote the hypothesis that a
learning algorithm, A, returns upon receiving the training sequence S.
•A simple data-generation model We now explain how the training data is
generated. First, we assume that the instances (the papayas we encounter)
are generated by some probability distribution (in this case, representing
the environment). Let us denote that probability distribution over Xby

D. It is important to note that we do not assume that the learner knows
anything about this distribution. For the type of learning tasks we discuss,
this could be any arbitrary probability distribution. As to the labels, in the
current discussion we assume that there is some “correct” labeling function,
f:X→Y , and thatyi=f(xi) for alli. This assumption will be relaxed in
the next chapter. The labeling function is unknown to the learner. In fact,

this is just what the learner is trying to ﬁgure out. In summary, each pair
in the training data Sis generated by ﬁrst sampling a point xiaccording
toDand then labeling it by f.
•Measures of success: We deﬁne the error of a classiﬁer to be the probability
that it does not predict the correct label on a random data point generated
by the aforementioned underlying distribution. That is, the error of his
the probability to draw a random instance x, according to the distribution
D, such that h(x) does not equal f(x).

Formally, given a domain subset,2A⊂X, the probability distribution,
D, assigns a number, D(A), which determines how likely it is to observe a
pointx∈A. In many cases, we refer to Aas an event and express it using
a functionπ:X→{ 0,1}, namely,A={x∈X :π(x) = 1}. In that case,
we also use the notation Px∼D[π(x)] to expressD(A).
We deﬁne the error of a prediction rule, h:X→Y , to be
LD,f(h)def=P
x∼D[h(x)̸=f(x)]def=D({x:h(x)̸=f(x)}). (2.1)
That is, the error of such his the probability of randomly choosing an
examplexfor whichh(x)̸=f(x). The subscript ( D,f) indicates that the

error is measured with respect to the probability distribution Dand the
1Despite the “set” notation, Sis a sequence. In particular, the same example may appear
twice inSand some algorithms can take into account the order of examples in S.
2Strictly speaking, we should be more careful and require that Ais a member of some
σ-algebra of subsets of X, over whichDis deﬁned. We will formally deﬁne our
measurability assumptions in the next chapter.

correct labeling function f. We omit this subscript when it is clear from
the context. L(D,f)(h) has several synonymous names such as the general-
ization error , the risk, or the true error ofh, and we will use these names
interchangeably throughout the book. We use the letter Lfor the error,
since we view this error as the lossof the learner. We will later also discuss
other possible formulations of such loss.
•A note about the information available to the learner The learner is
blind to the underlying distribution Dover the world and to the labeling
functionf. In our papayas example, we have just arrived in a new island
and we have no clue as to how papayas are distributed and how to predict
their tastiness. The only way the learner can interact with the environment
is through observing the training set.
In the next section we describe a simple learning paradigm for the preceding
setup and analyze its performance.
2.2 Empirical Risk Minimization

As mentioned earlier, a learning algorithm receives as input a training set S,
sampled from an unknown distribution Dand labeled by some target function
f, and should output a predictor hS:X→Y (the subscript Semphasizes the
fact that the output predictor depends on S). The goal of the algorithm is to
ﬁndhSthat minimizes the error with respect to the unknown Dandf.
Since the learner does not know what Dandfare, the true error is not directly
available to the learner. A useful notion of error that can be calculated by the

learner is the training error – the error the classiﬁer incurs over the training
sample:
LS(h)def=|{i∈[m] :h(xi)̸=yi}|
m, (2.2)
where [m] ={1,...,m}.
The terms empirical error and empirical risk are often used interchangeably
for this error.
Since the training sample is the snapshot of the world that is available to the
learner, it makes sense to search for a solution that works well on that data.
This learning paradigm – coming up with a predictor hthat minimizes LS(h) –
is called Empirical Risk Minimization or ERM for short.
2.2.1 Something May Go Wrong – Overﬁtting

sample:
m, (2.2)
Although the ERM rule seems very natural, without being careful, this approach
may fail miserably.
To demonstrate such a failure, let us go back to the problem of learning to

36 A Gentle Start
predict the taste of a papaya on the basis of its softness and color. Consider a
sample as depicted in the following:
Assume that the probability distribution Dis such that instances are distributed
uniformly within the gray square and the labeling function, f, determines the
label to be 1 if the instance is within the inner blue square, and 0 otherwise. The
area of the gray square in the picture is 2 and the area of the blue square is 1.
Consider the following predictor:
hS(x) ={
yiif∃i∈[m] s.t. xi=x
0 otherwise .(2.3)
While this predictor might seem rather artiﬁcial, in Exercise 1 we show a natural
representation of it using polynomials. Clearly, no matter what the sample is,
LS(hS) = 0, and therefore this predictor may be chosen by an ERM algorithm (it
is one of the empirical-minimum-cost hypotheses; no classiﬁer can have smaller
error). On the other hand, the true error of any classiﬁer that predicts the label

hS(x) ={
1 only on a ﬁnite number of instances is, in this case, 1 /2. Thus,LD(hS) = 1/2.
We have found a predictor whose performance on the training set is excellent,
yet its performance on the true “world” is very poor. This phenomenon is called
overﬁtting . Intuitively, overﬁtting occurs when our hypothesis ﬁts the training
data “too well” (perhaps like the everyday experience that a person who provides
a perfect detailed explanation for each of his single actions may raise suspicion).
2.3 Empirical Risk Minimization with Inductive Bias

We have just demonstrated that the ERM rule might lead to overﬁtting. Rather
than giving up on the ERM paradigm, we will look for ways to rectify it. We will
search for conditions under which there is a guarantee that ERM does not overﬁt,
namely, conditions under which when the ERM predictor has good performance
with respect to the training data, it is also highly likely to perform well over the
underlying data distribution.
A common solution is to apply the ERM learning rule over a restricted search

space. Formally, the learner should choose in advance (before seeing the data) a
set of predictors. This set is called a hypothesis class and is denoted by H. Each
h∈H is a function mapping from XtoY. For a given class H, and a training
sample,S, the ERMHlearner uses the ERM rule to choose a predictor h∈H,

2.3 Empirical Risk Minimization with Inductive Bias 37
with the lowest possible error over S. Formally,
ERMH(S)∈argmin
h∈HLS(h),
where argmin stands for the set of hypotheses in Hthat achieve the minimum
value ofLS(h) overH. By restricting the learner to choosing a predictor from
H, we bias it toward a particular set of predictors. Such restrictions are often
called an inductive bias . Since the choice of such a restriction is determined
before the learner sees the training data, it should ideally be based on some
prior knowledge about the problem to be learned. For example, for the papaya
taste prediction problem we may choose the class Hto be the set of predictors
that are determined by axis aligned rectangles (in the space determined by the
color and softness coordinates). We will later show that ERM Hover this class is
guaranteed not to overﬁt. On the other hand, the example of overﬁtting that we
have seen previously, demonstrates that choosing Hto be a class of predictors

that includes all functions that assign the value 1 to a ﬁnite set of domain points
does not suﬃce to guarantee that ERM Hwill not overﬁt.
A fundamental question in learning theory is, over which hypothesis classes
ERMHlearning will not result in overﬁtting. We will study this question later
in the book.
Intuitively, choosing a more restricted hypothesis class better protects us
against overﬁtting but at the same time might cause us a stronger inductive
bias. We will get back to this fundamental tradeoﬀ later.
2.3.1 Finite Hypothesis Classes

The simplest type of restriction on a class is imposing an upper bound on its size
(that is, the number of predictors hinH). In this section, we show that if His
a ﬁnite class then ERM Hwill not overﬁt, provided it is based on a suﬃciently
large training sample (this size requirement will depend on the size of H).
Limiting the learner to prediction rules within some ﬁnite hypothesis class may
be considered as a reasonably mild restriction. For example, Hcan be the set of
all predictors that can be implemented by a C++ program written in at most

109bits of code. In our papayas example, we mentioned previously the class of
axis aligned rectangles. While this is an inﬁnite class, if we discretize the repre-
sentation of real numbers, say, by using a 64 bits ﬂoating-point representation,
the hypothesis class becomes a ﬁnite class.
Let us now analyze the performance of the ERM Hlearning rule assuming that
His a ﬁnite class. For a training sample, S, labeled according to some f:X→Y ,
lethSdenote a result of applying ERM HtoS, namely,
hS∈argmin
h∈HLS(h). (2.4)

hS∈argmin
In this chapter, we make the following simplifying assumption (which will be
relaxed in the next chapter).

38 A Gentle Start
definition 2.1 (The Realizability Assumption) There exists h⋆∈ H s.t.
L(D,f)(h⋆) = 0. Note that this assumption implies that with probability 1 over
random samples, S, where the instances of Sare sampled according to Dand
are labeled by f, we haveLS(h⋆) = 0.
The realizability assumption implies that for every ERM hypothesis we have
that3LS(hS) = 0. However, we are interested in the true risk ofhS,L(D,f)(hS),
rather than its empirical risk.
Clearly, any guarantee on the error with respect to the underlying distribution,
D, for an algorithm that has access only to a sample Sshould depend on the
relationship between DandS. The common assumption in statistical machine
learning is that the training sample Sis generated by sampling points from the
distributionDindependently of each other. Formally,
•The i.i.d. assumption: The examples in the training set are independently
and identically distributed (i.i.d.) according to the distribution D. That is,

everyxiinSis freshly sampled according to Dand then labeled according
to the labeling function, f. We denote this assumption by S∼Dmwhere
mis the size of S, andDmdenotes the probability over m-tuples induced
by applyingDto pick each element of the tuple independently of the other
members of the tuple.
Intuitively, the training set Sis a window through which the learner
gets partial information about the distribution Dover the world and the
labeling function, f. The larger the sample gets, the more likely it is to

reﬂect more accurately the distribution and labeling used to generate it.
SinceL(D,f)(hS) depends on the training set, S, and that training set is picked
by a random process, there is randomness in the choice of the predictor hS
and, consequently, in the risk L(D,f)(hS). Formally, we say that it is a random
variable. It is not realistic to expect that with full certainty Swill suﬃce to
direct the learner toward a good classiﬁer (from the point of view of D), as
there is always some probability that the sampled training data happens to

be very nonrepresentative of the underlying D. If we go back to the papaya
tasting example, there is always some (small) chance that all the papayas we
have happened to taste were not tasty, in spite of the fact that, say, 70% of the
papayas in our island are tasty. In such a case, ERM H(S) may be the constant
function that labels every papaya as “not tasty” (and has 70% error on the true
distribution of papapyas in the island). We will therefore address the probability

to sample a training set for which L(D,f)(hS) is not too large. Usually, we denote
the probability of getting a nonrepresentative sample by δ, and call (1−δ) the
conﬁdence parameter of our prediction.
On top of that, since we cannot guarantee perfect label prediction, we intro-
duce another parameter for the quality of prediction, the accuracy parameter ,
3Mathematically speaking, this holds with probability 1. To simplify the presentation, we
sometimes omit the “with probability 1” speciﬁer.

2.3 Empirical Risk Minimization with Inductive Bias 39
commonly denoted by ϵ. We interpret the event L(D,f)(hS)>ϵas a failure of the
learner, while if L(D,f)(hS)≤ϵwe view the output of the algorithm as an approx-
imately correct predictor. Therefore (ﬁxing some labeling function f:X→Y ),
we are interested in upper bounding the probability to sample m-tuple of in-
stances that will lead to failure of the learner. Formally, let S|x= (x1,...,xm)
be the instances of the training set. We would like to upper bound
Dm({S|x:L(D,f)(hS)>ϵ}).
LetHBbe the set of “bad” hypotheses, that is,
HB={h∈H:L(D,f)(h)>ϵ}.
In addition, let
M={S|x:∃h∈HB,LS(h) = 0}
be the set of misleading samples: Namely, for every S|x∈M, there is a “bad”
hypothesis, h∈HB, that looks like a “good” hypothesis on S|x. Now, recall that
we would like to bound the probability of the event L(D,f)(hS)> ϵ. But, since
the realizability assumption implies that LS(hS) = 0, it follows that the event

L(D,f)(hS)> ϵcan only happen if for some h∈HBwe haveLS(h) = 0. In
other words, this event will only happen if our sample is in the set of misleading
samples,M. Formally, we have shown that
{S|x:L(D,f)(hS)>ϵ}⊆M .
Note that we can rewrite Mas
M=⋃
h∈HB{S|x:LS(h) = 0}. (2.5)
Hence,
Dm({S|x:L(D,f)(hS)>ϵ})≤ Dm(M) =Dm(∪h∈HB{S|x:LS(h) = 0}).
(2.6)
Next, we upper bound the right-hand side of the preceding equation using the
union bound – a basic property of probabilities.
lemma 2.2 (Union Bound) For any two sets A,B and a distribution Dwe
have
D(A∪B)≤D(A) +D(B).

M=⋃
Hence,
(2.6)
have
Applying the union bound to the right-hand side of Equation (2.6) yields
Dm({S|x:L(D,f)(hS)>ϵ})≤∑
h∈HBDm({S|x:LS(h) = 0}). (2.7)
Next, let us bound each summand of the right-hand side of the preceding in-
equality. Fix some “bad” hypothesis h∈HB. The event LS(h) = 0 is equivalent

40 A Gentle Start
to the event∀i,h(xi) =f(xi). Since the examples in the training set are sampled
i.i.d. we get that
Dm({S|x:LS(h) = 0}) =Dm({S|x:∀i,h(xi) =f(xi)})
=m∏
i=1D({xi:h(xi) =f(xi)}). (2.8)
For each individual sampling of an element of the training set we have
D({xi:h(xi) =yi}) = 1−L(D,f)(h)≤1−ϵ,
where the last inequality follows from the fact that h∈HB. Combining the
previous equation with Equation (2.8) and using the inequality 1 −ϵ≤e−ϵwe
obtain that for every h∈HB,
Dm({S|x:LS(h) = 0})≤(1−ϵ)m≤e−ϵm. (2.9)
Combining this equation with Equation (2.7) we conclude that
Dm({S|x:L(D,f)(hS)>ϵ})≤ |HB|e−ϵm≤ |H|e−ϵm.
A graphical illustration which explains how we used the union bound is given in
Figure 2.1.
Figure 2.1 Each point in the large circle represents a possible m-tuple of instances.
Each colored oval represents the set of “misleading” m-tuple of instances for some
“bad” predictor h∈HB. The ERM can potentially overﬁt whenever it gets a
misleading training set S. That is, for some h∈HBwe haveLS(h) = 0.

Equation (2.9) guarantees that for each individual bad hypothesis, h∈HB, at most
(1−ϵ)m-fraction of the training sets would be misleading. In particular, the larger m
is, the smaller each of these colored ovals becomes. The union bound formalizes the
fact that the area representing the training sets that are misleading with respect to
someh∈HB(that is, the training sets in M) is at most the sum of the areas of the
colored ovals. Therefore, it is bounded by |HB|times the maximum size of a colored

oval. Any sample Soutside the colored ovals cannot cause the ERM rule to overﬁt.
corollary 2.3 LetHbe a ﬁnite hypothesis class. Let δ∈(0,1)andϵ >0

and letmbe an integer that satisﬁes
m≥log(|H|/δ)
ϵ.
Then, for any labeling function, f, and for any distribution, D, for which the
realizability assumption holds (that is, for some h∈H ,L(D,f)(h) = 0 ), with
probability of at least 1−δover the choice of an i.i.d. sample Sof sizem, we
have that for every ERM hypothesis, hS, it holds that
L(D,f)(hS)≤ϵ.
The preceeding corollary tells us that for a suﬃciently large m, the ERMHrule
over a ﬁnite hypothesis class will be probably (with conﬁdence 1 −δ)approximately
(up to an error of ϵ) correct. In the next chapter we formally deﬁne the model
of Probably Approximately Correct (PAC) learning.
2.4 Exercises
1.Overﬁtting of polynomial matching: We have shown that the predictor
deﬁned in Equation (2.3) leads to overﬁtting. While this predictor seems to
be very unnatural, the goal of this exercise is to show that it can be described
as a thresholded polynomial. That is, show that given a training set S=
{(xi,f(xi))}m

i=1⊆(Rd×{0,1})m, there exists a polynomial pSsuch that
hS(x) = 1 if and only if pS(x)≥0, wherehSis as deﬁned in Equation (2.3).
It follows that learning the class of all thresholded polynomials using the ERM
rule may lead to overﬁtting.
2. LetHbe a class of binary classiﬁers over a domain X. LetDbe an unknown
distribution overX, and letfbe the target hypothesis in H. Fix someh∈H.
Show that the expected value of LS(h) over the choice of S|xequalsL(D,f)(h),
namely,
E
S|x∼Dm[LS(h)] =L(D,f)(h).

namely,
E
3.Axis aligned rectangles: An axis aligned rectangle classiﬁer in the plane
is a classiﬁer that assigns the value 1 to a point if and only if it is inside a
certain rectangle. Formally, given real numbers a1≤b1,a2≤b2, deﬁne the
classiﬁerh(a1,b1,a2,b2)by
h(a1,b1,a2,b2)(x1,x2) ={
1 ifa1≤x1≤b1anda2≤x2≤b2
0 otherwise. (2.10)
The class of all axis aligned rectangles in the plane is deﬁned as
H2
rec={h(a1,b1,a2,b2):a1≤b1,anda2≤b2}.
Note that this is an inﬁnite size hypothesis class. Throughout this exercise we

H2
rely on the realizability assumption.

42 A Gentle Start
1. LetAbe the algorithm that returns the smallest rectangle enclosing all
positive examples in the training set. Show that Ais an ERM.
2. Show that if Areceives a training set of size ≥4 log(4/δ)
ϵthen, with proba-
bility of at least 1 −δit returns a hypothesis with error of at most ϵ.
Hint: Fix some distribution DoverX, letR∗=R(a∗
1,b∗
1,a∗
2,b∗
2) be the rect-
angle that generates the labels, and let fbe the corresponding hypothesis.
Leta1≥a∗
1be a number such that the probability mass (with respect
toD) of the rectangle R1=R(a∗
1,a1,a∗
2,b∗
2) is exactly ϵ/4. Similarly, let
b1,a2,b2be numbers such that the probability masses of the rectangles
R2=R(b1,b∗
1,a∗
2,b∗
2),R3=R(a∗
1,b∗
1,a∗
2,a2),R4=R(a∗
1,b∗
1,b2,b∗
2) are all
exactlyϵ/4. LetR(S) be the rectangle returned by A. See illustration in
Figure 2.2.
+
+++
+--R∗
R(S)
R1
Figure 2.2 Axis aligned rectangles.
•Show thatR(S)⊆R∗.
•Show that if Scontains (positive) examples in all of the rectangles

1,a1,a∗
2,b∗
1,a∗
2,b∗
1,b∗
1,a∗
1,b∗
1,b2,b∗
2) are all
+
+++
+--R∗
R(S)
R1
R1,R2,R3,R4, then the hypothesis returned by Ahas error of at
mostϵ.
•For eachi∈{1,..., 4}, upper bound the probability that Sdoes not
contain an example from Ri.
•Use the union bound to conclude the argument.
3. Repeat the previous question for the class of axis aligned rectangles in Rd.
4. Show that the runtime of applying the algorithm Amentioned earlier is
polynomial in d,1/ϵ,and in log(1 /δ).

3 A Formal Learning Model
In this chapter we deﬁne our main formal learning model – the PAC learning
model and its extensions. We will consider other notions of learnability in Chap-
ter 7.
3.1 PAC Learning
In the previous chapter we have shown that for a ﬁnite hypothesis class, if the
ERM rule with respect to that class is applied on a suﬃciently large training
sample (whose size is independent of the underlying distribution or labeling
function) then the output hypothesis will be probably approximately correct.
More generally, we now deﬁne Probably Approximately Correct (PAC) learning.
definition 3.1 (PAC Learnability) A hypothesis class His PAC learnable
if there exist a function mH: (0,1)2→Nand a learning algorithm with the
following property: For every ϵ,δ∈(0,1), for every distribution DoverX, and
for every labeling function f:X →{ 0,1}, if the realizable assumption holds
with respect toH,D,f, then when running the learning algorithm on m≥

mH(ϵ,δ) i.i.d. examples generated by Dand labeled by f, the algorithm returns
a hypothesis hsuch that, with probability of at least 1 −δ(over the choice of
the examples), L(D,f)(h)≤ϵ.
The deﬁnition of Probably Approximately Correct learnability contains two
approximation parameters. The accuracy parameter ϵdetermines how far the
output classiﬁer can be from the optimal one (this corresponds to the “approx-
imately correct”), and a conﬁdence parameter δindicating how likely the clas-
siﬁer is to meet that accuracy requirement (corresponds to the “probably” part

of “PAC”). Under the data access model that we are investigating, these ap-
proximations are inevitable. Since the training set is randomly generated, there
may always be a small chance that it will happen to be noninformative (for ex-
ample, there is always some chance that the training set will contain only one
domain point, sampled over and over again). Furthermore, even when we are
lucky enough to get a training sample that does faithfully represent D, because

it is just a ﬁnite sample, there may always be some ﬁne details of Dthat it fails

44 A Formal Learning Model
to reﬂect. Our accuracy parameter, ϵ, allows “forgiving” the learner’s classiﬁer
for making minor errors.
Sample Complexity
The function mH: (0,1)2→Ndetermines the sample complexity of learningH:
that is, how many examples are required to guarantee a probably approximately
correct solution. The sample complexity is a function of the accuracy ( ϵ) and
conﬁdence ( δ) parameters. It also depends on properties of the hypothesis class
H– for example, for a ﬁnite class we showed that the sample complexity depends
on log the size of H.
Note that ifHis PAC learnable, there are many functions mHthat satisfy the
requirements given in the deﬁnition of PAC learnability. Therefore, to be precise,
we will deﬁne the sample complexity of learning Hto be the “minimal function,”
in the sense that for any ϵ,δ,mH(ϵ,δ) is the minimal integer that satisﬁes the
requirements of PAC learning with accuracy ϵand conﬁdence δ.
Let us now recall the conclusion of the analysis of ﬁnite hypothesis classes

from the previous chapter. It can be rephrased as stating:
corollary 3.2 Every ﬁnite hypothesis class is PAC learnable with sample
complexity
mH(ϵ,δ)≤⌈log(|H|/δ)
ϵ⌉
.
There are inﬁnite classes that are learnable as well (see, for example, Exer-
cise 3). Later on we will show that what determines the PAC learnability of
a class is not its ﬁniteness but rather a combinatorial measure called the VC
dimension .
3.2 A More General Learning Model
The model we have just described can be readily generalized, so that it can be

ϵ⌉
.
made relevant to a wider scope of learning tasks. We consider generalizations in
two aspects:
Removing the Realizability Assumption
We have required that the learning algorithm succeeds on a pair of data distri-
butionDand labeling function fprovided that the realizability assumption is
met. For practical learning tasks, this assumption may be too strong (can we
really guarantee that there is a rectangle in the color-hardness space that fully
determines which papayas are tasty?). In the next subsection, we will describe

theagnostic PAC model in which this realizability assumption is waived.

3.2 A More General Learning Model 45
Learning Problems beyond Binary Classiﬁcation
The learning task that we have been discussing so far has to do with predicting a
binary label to a given example (like being tasty or not). However, many learning
tasks take a diﬀerent form. For example, one may wish to predict a real valued
number (say, the temperature at 9:00 p.m. tomorrow) or a label picked from
a ﬁnite set of labels (like the topic of the main story in tomorrow’s paper). It
turns out that our analysis of learning can be readily extended to such and many
other scenarios by allowing a variety of loss functions. We shall discuss that in
Section 3.2.2 later.
3.2.1 Releasing the Realizability Assumption – Agnostic PAC Learning
A More Realistic Model for the Data-Generating Distribution
Recall that the realizability assumption requires that there exists h⋆∈H such
thatPx∼D[h⋆(x) =f(x)] = 1. In many practical problems this assumption does

not hold. Furthermore, it is maybe more realistic not to assume that the labels
are fully determined by the features we measure on input elements (in the case of
the papayas, it is plausible that two papayas of the same color and softness will
have diﬀerent taste). In the following, we relax the realizability assumption by
replacing the “target labeling function” with a more ﬂexible notion, a data-labels
generating distribution.
Formally, from now on, let Dbe a probability distribution over X×Y , where,

as before,Xis our domain set and Yis a set of labels (usually we will consider
Y={0,1}). That is,Dis ajoint distribution over domain points and labels. One
can view such a distribution as being composed of two parts: a distribution Dx
over unlabeled domain points (sometimes called the marginal distribution ) and
aconditional probability over labels for each domain point, D((x,y)|x). In the
papaya example, Dxdetermines the probability of encountering a papaya whose

color and hardness fall in some color-hardness values domain, and the conditional
probability is the probability that a papaya with color and hardness represented
byxis tasty. Indeed, such modeling allows for two papayas that share the same
color and hardness to belong to diﬀerent taste categories.
The empirical and the True Error Revised
For a probability distribution, D, overX×Y , one can measure how likely his
to make an error when labeled points are randomly drawn according to D. We
redeﬁne the true error (or risk) of a prediction rule hto be

LD(h)def= P
(x,y)∼D[h(x)̸=y]def=D({(x,y) :h(x)̸=y}). (3.1)
We would like to ﬁnd a predictor, h, for which that error will be minimized.
However, the learner does not know the data generating D. What the learner
does have access to is the training data, S. The deﬁnition of the empirical risk

46 A Formal Learning Model
remains the same as before, namely,
m.
GivenS, a learner can compute LS(h) for any function h:X→{0,1}. Note
thatLS(h) =LD(uniform over S)(h).
The Goal
We wish to ﬁnd some hypothesis, h:X →Y , that (probably approximately)
minimizes the true risk, LD(h).
The Bayes Optimal Predictor.
Given any probability distribution DoverX×{ 0,1}, the best label predicting
function fromXto{0,1}will be
fD(x) ={
1 ifP[y= 1|x]≥1/2
It is easy to verify (see Exercise 7) that for every probability distribution D,
the Bayes optimal predictor fDis optimal, in the sense that no other classiﬁer,
g:X→{ 0,1}has a lower error. That is, for every classiﬁer g,LD(fD)≤LD(g).
Unfortunately, since we do not know D, we cannot utilize this optimal predictor
fD. What the learner does have access to is the training sample. We can now
present the formal deﬁnition of agnostic PAC learnability, which is a natural

fD(x) ={
extension of the deﬁnition of PAC learnability to the more realistic, nonrealizable,
learning setup we have just discussed.
Clearly, we cannot hope that the learning algorithm will ﬁnd a hypothesis
whose error is smaller than the minimal possible error, that of the Bayes predic-
tor.
Furthermore, as we shall prove later, once we make no prior assumptions
about the data-generating distribution, no algorithm can be guaranteed to ﬁnd
a predictor that is as good as the Bayes optimal one. Instead, we require that

tor.
the learning algorithm will ﬁnd a predictor whose error is not much larger than
the best possible error of a predictor in some given benchmark hypothesis class.
Of course, the strength of such a requirement depends on the choice of that
hypothesis class.
definition 3.3 (Agnostic PAC Learnability) A hypothesis class His agnostic
PAC learnable if there exist a function mH: (0,1)2→Nand a learning algorithm
with the following property: For every ϵ,δ∈(0,1) and for every distribution D
overX×Y , when running the learning algorithm on m≥mH(ϵ,δ) i.i.d. examples

generated byD, the algorithm returns a hypothesis hsuch that, with probability
of at least 1−δ(over the choice of the mtraining examples),
LD(h)≤min
h′∈HLD(h′) +ϵ.

3.2 A More General Learning Model 47
Clearly, if the realizability assumption holds, agnostic PAC learning provides
the same guarantee as PAC learning. In that sense, agnostic PAC learning gener-
alizes the deﬁnition of PAC learning. When the realizability assumption does not
hold, no learner can guarantee an arbitrarily small error. Nevertheless, under the
deﬁnition of agnostic PAC learning, a learner can still declare success if its error
is not much larger than the best error achievable by a predictor from the class H.
This is in contrast to PAC learning, in which the learner is required to achieve
a small error in absolute terms and not relative to the best error achievable by
the hypothesis class.
3.2.2 The Scope of Learning Problems Modeled
We next extend our model so that it can be applied to a wide variety of learning
tasks. Let us consider some examples of diﬀerent learning tasks.
•Multiclass Classiﬁcation Our classiﬁcation does not have to be binary.

Take, for example, the task of document classiﬁcation: We wish to design a
program that will be able to classify given documents according to topics
(e.g., news, sports, biology, medicine). A learning algorithm for such a task
will have access to examples of correctly classiﬁed documents and, on the
basis of these examples, should output a program that can take as input a
new document and output a topic classiﬁcation for that document. Here,
thedomain set is the set of all potential documents. Once again, we would

usually represent documents by a set of features that could include counts
of diﬀerent key words in the document, as well as other possibly relevant
features like the size of the document or its origin. The label set in this task
will be the set of possible document topics (so Ywill be some large ﬁnite
set). Once we determine our domain and label sets, the other components
of our framework look exactly the same as in the papaya tasting example;
Our training sample will be a ﬁnite sequence of (feature vector ,label) pairs,

the learner’s output will be a function from the domain set to the label set,
and, ﬁnally, for our measure of success, we can use the probability, over
(document, topic) pairs, of the event that our predictor suggests a wrong
label.
•Regression In this task, one wishes to ﬁnd some simple pattern in the data –
a functional relationship between the XandYcomponents of the data. For
example, one wishes to ﬁnd a linear function that best predicts a baby’s
birth weight on the basis of ultrasound measures of his head circumference,

label.
abdominal circumference, and femur length. Here, our domain set Xis some
subset of R3(the three ultrasound measurements), and the set of “labels,”
Y, is the the set of real numbers (the weight in grams). In this context,
it is more adequate to call Ythetarget set. Our training data as well as
the learner’s output are as before (a ﬁnite sequence of ( x,y) pairs, and
a function fromXtoYrespectively). However, our measure of success is

48 A Formal Learning Model
diﬀerent. We may evaluate the quality of a hypothesis function, h:X→Y ,
by the expected square diﬀerence between the true labels and their predicted
values, namely,
LD(h)def= E
(x,y)∼D(h(x)−y)2. (3.2)
To accommodate a wide range of learning tasks we generalize our formalism
of the measure of success as follows:
Generalized Loss Functions
Given any setH(that plays the role of our hypotheses, or models) and some
domainZletℓbe any function from H×Zto the set of nonnegative real numbers,
ℓ:H×Z→R+.
We call such functions loss functions .
Note that for prediction problems, we have that Z=X×Y . However, our
notion of the loss function is generalized beyond prediction tasks, and therefore
it allowsZto be any domain of examples (for instance, in unsupervised learning
tasks such as the one described in Chapter 22, Zis not a product of an instance
domain and a label domain).
We now deﬁne the risk function to be the expected loss of a classiﬁer, h∈H,

ℓ:H×Z→R+.
with respect to a probability distribution DoverZ, namely,
LD(h)def=E
z∼D[ℓ(h,z)]. (3.3)
That is, we consider the expectation of the loss of hover objects zpicked ran-
domly according to D. Similarly, we deﬁne the empirical risk to be the expected
loss over a given sample S= (z1,...,zm)∈Zm, namely,
LS(h)def=1
mm∑
i=1ℓ(h,zi). (3.4)
The loss functions used in the preceding examples of classiﬁcation and regres-
sion tasks are as follows:
•0–1 loss: Here, our random variable zranges over the set of pairs X×Y and
the loss function is
ℓ0−1(h,(x,y))def={

mm∑
0 ifh(x) =y
1 ifh(x)̸=y
This loss function is used in binary or multiclass classiﬁcation problems.
One should note that, for a random variable, α, taking the values {0,1},
Eα∼D[α] =Pα∼D[α= 1]. Consequently, for this loss function, the deﬁni-
tions ofLD(h) given in Equation (3.3) and Equation (3.1) coincide.
•Square Loss: Here, our random variable zranges over the set of pairs X×Y
and the loss function is
ℓsq(h,(x,y))def= (h(x)−y)2.

This loss function is used in regression problems.
We will later see more examples of useful instantiations of loss functions.
To summarize, we formally deﬁne agnostic PAC learnability for general loss
definition 3.4 (Agnostic PAC Learnability for General Loss Functions) A
hypothesis class His agnostic PAC learnable with respect to a set Zand a
loss function ℓ:H×Z→R+, if there exist a function mH: (0,1)2→N
and a learning algorithm with the following property: For every ϵ,δ∈(0,1)
and for every distribution DoverZ, when running the learning algorithm on
m≥mH(ϵ,δ) i.i.d. examples generated by D, the algorithm returns h∈ H
such that, with probability of at least 1 −δ(over the choice of the mtraining
examples),
LD(h)≤min
h′∈HLD(h′) +ϵ,
whereLD(h) =Ez∼D[ℓ(h,z)].
Remark 3.1 (A Note About Measurability*) In the aforementioned deﬁnition,
for everyh∈H, we view the function ℓ(h,·) :Z→R+as a random variable and
deﬁneLD(h) to be the expected value of this random variable. For that, we need

LD(h)≤min
to require that the function ℓ(h,·) is measurable. Formally, we assume that there
is aσ-algebra of subsets of Z, over which the probability Dis deﬁned, and that
the preimage of every initial segment in R+is in thisσ-algebra. In the speciﬁc
case of binary classiﬁcation with the 0 −1 loss, theσ-algebra is overX×{ 0,1}
and our assumption on ℓis equivalent to the assumption that for every h, the
set{(x,h(x)) :x∈X} is in theσ-algebra.
Remark 3.2 (Proper versus Representation-Independent Learning*) In the pre-

ceding deﬁnition, we required that the algorithm will return a hypothesis from
H. In some situations, His a subset of a set H′, and the loss function can be
naturally extended to be a function from H′×Zto the reals. In this case, we
may allow the algorithm to return a hypothesis h′∈H′, as long as it satisﬁes
the requirement LD(h′)≤minh∈HLD(h) +ϵ. Allowing the algorithm to output
a hypothesis from H′is called representation independent learning, while proper

learning occurs when the algorithm must output a hypothesis from H. Represen-
tation independent learning is sometimes called “improper learning,” although
there is nothing improper in representation independent learning.
3.3 Summary
In this chapter we deﬁned our main formal learning model – PAC learning. The
basic model relies on the realizability assumption, while the agnostic variant does

50 A Formal Learning Model
not impose any restrictions on the underlying distribution over the examples. We
also generalized the PAC model to arbitrary loss functions. We will sometimes
refer to the most general model simply as PAC learning, omitting the “agnostic”
preﬁx and letting the reader infer what the underlying loss function is from the
context. When we would like to emphasize that we are dealing with the original
PAC setting we mention that the realizability assumption holds. In Chapter 7
we will discuss other notions of learnability.
3.4 Bibliographic Remarks
Our most general deﬁnition of agnostic PAC learning with general loss func-
tions follows the works of Vladimir Vapnik and Alexey Chervonenkis (Vapnik &
Chervonenkis 1971). In particular, we follow Vapnik’s general setting of learning
(Vapnik 1982, Vapnik 1992, Vapnik 1995, Vapnik 1998).
PAC learning was introduced by Valiant (1984). Valiant was named the winner
of the 2010 Turing Award for the introduction of the PAC model. Valiant’s

deﬁnition requires that the sample complexity will be polynomial in 1 /ϵand
in 1/δ, as well as in the representation size of hypotheses in the class (see also
Kearns & Vazirani (1994)). As we will see in Chapter 6, if a problem is at all PAC
learnable then the sample complexity depends polynomially on 1 /ϵand log(1/δ).
Valiant’s deﬁnition also requires that the runtime of the learning algorithm will
be polynomial in these quantities. In contrast, we chose to distinguish between

the statistical aspect of learning and the computational aspect of learning. We
will elaborate on the computational aspect later on in Chapter 8, where we
introduce the full PAC learning model of Valiant. For expository reasons, we
use the term PAC learning even when we ignore the runtime aspect of learning.
Finally, the formalization of agnostic PAC learning is due to Haussler (1992).
3.5 Exercises
1.Monotonicity of Sample Complexity: LetHbe a hypothesis class for a

binary classiﬁcation task. Suppose that His PAC learnable and its sample
complexity is given by mH(·,·). Show that mHis monotonically nonincreasing
in each of its parameters. That is, show that given δ∈(0,1), and given 0 <
ϵ1≤ϵ2<1, we have that mH(ϵ1,δ)≥mH(ϵ2,δ). Similarly, show that given
ϵ∈(0,1), and given 0 <δ1≤δ2<1, we have that mH(ϵ,δ1)≥mH(ϵ,δ2).
2. LetXbe a discrete domain, and let HSingleton ={hz:z∈X}∪{h−}, where
for eachz∈X,hzis the function deﬁned by hz(x) = 1 ifx=zandhz(x) = 0

ifx̸=z.h−is simply the all-negative hypothesis, namely, ∀x∈X,h−(x) = 0.
The realizability assumption here implies that the true hypothesis flabels
negatively all examples in the domain, perhaps except one.

3.5 Exercises 51
1. Describe an algorithm that implements the ERM rule for learning HSingleton
in the realizable setup.
2. Show thatHSingleton is PAC learnable. Provide an upper bound on the
sample complexity.
3. LetX=R2,Y={0,1}, and letHbe the class of concentric circles in the
plane, that is,H={hr:r∈R+}, wherehr(x) =1[∥x∥≤r]. Prove thatHis
PAC learnable (assume realizability), and its sample complexity is bounded
by
mH(ϵ,δ)≤⌈log(1/δ)
ϵ⌉
.
4. In this question, we study the hypothesis class of Boolean conjunctions deﬁned
as follows. The instance space is X={0,1}dand the label set is Y={0,1}. A
literal over the variables x1,...,xdis a simple Boolean function that takes the
formf(x) =xi, for somei∈[d], orf(x) = 1−xifor somei∈[d]. We use the
notation ¯xias a shorthand for 1 −xi. A conjunction is any product of literals.
In Boolean logic, the product is denoted using the ∧sign. For example, the
functionh(x) =x1·(1−x2) is written as x1∧¯x2.

ϵ⌉
.
We consider the hypothesis class of all conjunctions of literals over the d
variables. The empty conjunction is interpreted as the all-positive hypothesis
(namely, the function that returns h(x) = 1 for all x). The conjunction x1∧¯x1
(and similarly any conjunction involving a literal and its negation) is allowed
and interpreted as the all-negative hypothesis (namely, the conjunction that
returnsh(x) = 0 for all x). We assume realizability: Namely, we assume

that there exists a Boolean conjunction that generates the labels. Thus, each
example ( x,y)∈X×Y consists of an assignment to the dBoolean variables
x1,...,xd, and its truth value (0 for false and 1 for true).
For instance, let d= 3 and suppose that the true conjunction is x1∧¯x2.
Then, the training set Smight contain the following instances:
((1,1,1),0),((1,0,1),1),((0,1,0),0)((1,0,0),1).
Prove that the hypothesis class of all conjunctions over dvariables is

PAC learnable and bound its sample complexity. Propose an algorithm that
implements the ERM rule, whose runtime is polynomial in d·m.
5. LetXbe a domain and let D1,D2,...,Dmbe a sequence of distributions
overX. LetHbe a ﬁnite class of binary classiﬁers over Xand letf∈H.
Suppose we are getting a sample Sofmexamples, such that the instances are
independent but are not identically distributed; the ith instance is sampled
fromDiand thenyiis set to be f(xi). Let ¯Dmdenote the average, that is,
¯Dm= (D1+···+Dm)/m.
Fix an accuracy parameter ϵ∈(0,1). Show that

P[
∃h∈H s.t.L(¯Dm,f)(h)>ϵandL(S,f)(h) = 0]
≤|H|e−ϵm.

52 A Formal Learning Model
Hint: Use the geometric-arithmetic mean inequality.
6. LetHbe a hypothesis class of binary classiﬁers. Show that if His agnostic
PAC learnable, then His PAC learnable as well. Furthermore, if Ais a suc-
cessful agnostic PAC learner for H, thenAis also a successful PAC learner
forH.
7.(*) The Bayes optimal predictor: Show that for every probability distri-
butionD, the Bayes optimal predictor fDis optimal, in the sense that for
every classiﬁer gfromXto{0,1},LD(fD)≤LD(g).
8.(*)We say that a learning algorithm Ais better than Bwith respect to some
probability distribution, D, if
LD(A(S))≤LD(B(S))
for all samples S∈(X×{ 0,1})m. We say that a learning algorithm Ais better
thanB, if it is better than Bwith respect to all probability distributions D
overX×{ 0,1}.
1. A probabilistic label predictor is a function that assigns to every domain
pointxa probability value, h(x)∈[0,1], that determines the probability of
predicting the label 1. That is, given such an hand an input, x, the label for

xis predicted by tossing a coin with bias h(x) toward Heads and predicting
1 iﬀ the coin comes up Heads. Formally, we deﬁne a probabilistic label
predictor as a function, h:X→ [0,1]. The loss of such hon an example
(x,y) is deﬁned to be |h(x)−y|, which is exactly the probability that the
prediction of hwill not be equal to y. Note that if his deterministic, that
is, returns values in {0,1}, then|h(x)−y|=1[h(x)̸=y].
Prove that for every data-generating distribution DoverX×{ 0,1}, the
Bayes optimal predictor has the smallest risk (w.r.t. the loss function

ℓ(h,(x,y)) =|h(x)−y|, among all possible label predictors, including prob-
abilistic ones).
2. LetXbe a domain and {0,1}be a set of labels. Prove that for every
distributionDoverX×{ 0,1}, there exist a learning algorithm ADthat is
better than any other learning algorithm with respect to D.
3. Prove that for every learning algorithm Athere exist a probability distri-
bution,D, and a learning algorithm Bsuch thatAis not better than B
w.r.t.D.
9. Consider a variant of the PAC model in which there are two example ora-

w.r.t.D.
cles: one that generates positive examples and one that generates negative
examples, both according to the underlying distribution DonX. Formally,
given a target function f:X → { 0,1}, letD+be the distribution over
X+={x∈X :f(x) = 1}deﬁned byD+(A) =D(A)/D(X+), for every
A⊂X+. Similarly,D−is the distribution over X−induced byD.
The deﬁnition of PAC learnability in the two-oracle model is the same as the
standard deﬁnition of PAC learnability except that here the learner has access
tom+
H(ϵ,δ) i.i.d. examples from D+andm−(ϵ,δ) i.i.d. examples from D−. The

tom+
learner’s goal is to output hs.t. with probability at least 1 −δ(over the choice

3.5 Exercises 53
of the two training sets, and possibly over the nondeterministic decisions made
by the learning algorithm), both L(D+,f)(h)≤ϵandL(D−,f)(h)≤ϵ.
1.(*)Show that ifHis PAC learnable (in the standard one-oracle model),
thenHis PAC learnable in the two-oracle model.
2.(**) Deﬁneh+to be the always-plus hypothesis and h−to be the always-
minus hypothesis. Assume that h+,h−∈H. Show that ifHis PAC learn-
able in the two-oracle model, then His PAC learnable in the standard
one-oracle model.

4 Learning via Uniform Convergence
The ﬁrst formal learning model that we have discussed was the PAC model.
In Chapter 2 we have shown that under the realizability assumption, any ﬁnite
hypothesis class is PAC learnable. In this chapter we will develop a general tool,
uniform convergence , and apply it to show that any ﬁnite class is learnable in
the agnostic PAC model with general loss functions, as long as the range loss
function is bounded.
4.1 Uniform Convergence Is Suﬃcient for Learnability
The idea behind the learning condition discussed in this chapter is very simple.
Recall that, given a hypothesis class, H, the ERM learning paradigm works
as follows: Upon receiving a training sample, S, the learner evaluates the risk
(or error) of each hinHon the given sample and outputs a member of Hthat
minimizes this empirical risk. The hope is that an hthat minimizes the empirical
risk with respect to Sis a risk minimizer (or has risk close to the minimum) with

respect to the true data probability distribution as well. For that, it suﬃces to
ensure that the empirical risks of all members of Hare good approximations of
their true risk. Put another way, we need that uniformly over all hypotheses in
the hypothesis class, the empirical risk will be close to the true risk, as formalized
in the following.
definition 4.1 (ϵ-representative sample) A training set Sis calledϵ-representative
(w.r.t. domain Z, hypothesis class H, loss function ℓ, and distribution D) if
∀h∈H,|LS(h)−LD(h)|≤ϵ.

The next simple lemma states that whenever the sample is ( ϵ/2)-representative,
the ERM learning rule is guaranteed to return a good hypothesis.
lemma 4.2 Assume that a training set Sisϵ
2-representative (w.r.t. domain
Z, hypothesis class H, loss function ℓ, and distribution D). Then, any output of
ERMH(S), namely, any hS∈argminh∈HLS(h), satisﬁes
LD(hS)≤min
h∈HLD(h) +ϵ.

Proof For everyh∈H,
LD(hS)≤LS(hS) +ϵ
2≤LS(h) +ϵ
2≤LD(h) +ϵ
2+ϵ
2=LD(h) +ϵ,
where the ﬁrst and third inequalities are due to the assumption that Sisϵ
2-
representative (Deﬁnition 4.1) and the second inequality holds since hSis an
ERM predictor.
The preceding lemma implies that to ensure that the ERM rule is an agnostic
PAC learner, it suﬃces to show that with probability of at least 1 −δover the
random choice of a training set, it will be an ϵ-representative training set. The
uniform convergence condition formalizes this requirement.
definition 4.3 (Uniform Convergence) We say that a hypothesis class Hhas
theuniform convergence property (w.r.t. a domain Zand a loss function ℓ) if
there exists a function mUC
H: (0,1)2→Nsuch that for every ϵ,δ∈(0,1) and
for every probability distribution DoverZ, ifSis a sample of m≥mUC
H(ϵ,δ)
examples drawn i.i.d. according to D, then, with probability of at least 1 −δ,S
isϵ-representative.

H(ϵ,δ)
Similar to the deﬁnition of sample complexity for PAC learning, the function
mUC
Hmeasures the (minimal) sample complexity of obtaining the uniform con-
vergence property, namely, how many examples we need to ensure that with
probability of at least 1 −δthe sample would be ϵ-representative.
The term uniform here refers to having a ﬁxed sample size that works for all
members ofHand over all possible probability distributions over the domain.
The following corollary follows directly from Lemma 4.2 and the deﬁnition of
uniform convergence.

mUC
corollary 4.4 If a classHhas the uniform convergence property with a
functionmUC
Hthen the class is agnostically PAC learnable with the sample com-
plexitymH(ϵ,δ)≤mUC
H(ϵ/2,δ). Furthermore, in that case, the ERMHparadigm
is a successful agnostic PAC learner for H.
4.2 Finite Classes Are Agnostic PAC Learnable
In view of Corollary 4.4, the claim that every ﬁnite hypothesis class is agnostic
PAC learnable will follow once we establish that uniform convergence holds for
a ﬁnite hypothesis class.

To show that uniform convergence holds we follow a two step argument, similar
to the derivation in Chapter 2. The ﬁrst step applies the union bound while the
second step employs a measure concentration inequality. We now explain these
two steps in detail.
Fix someϵ,δ. We need to ﬁnd a sample size mthat guarantees that for any
D, with probability of at least 1 −δof the choice of S= (z1,...,zm) sampled

56 Learning via Uniform Convergence
i.i.d. fromDwe have that for all h∈H,|LS(h)−LD(h)|≤ϵ. That is,
Dm({S:∀h∈H,|LS(h)−LD(h)|≤ϵ})≥1−δ.
Equivalently, we need to show that
Dm({S:∃h∈H,|LS(h)−LD(h)|>ϵ})<δ.
Writing
{S:∃h∈H,|LS(h)−LD(h)|>ϵ}=∪h∈H{S:|LS(h)−LD(h)|>ϵ},
and applying the union bound (Lemma 2.2) we obtain
Dm({S:∃h∈H,|LS(h)−LD(h)|>ϵ})≤∑
h∈HDm({S:|LS(h)−LD(h)|>ϵ}).
(4.1)
Our second step will be to argue that each summand of the right-hand side
of this inequality is small enough (for a suﬃciently large m). That is, we will
show that for any ﬁxed hypothesis, h, (which is chosen in advance prior to the
sampling of the training set), the gap between the true and empirical risks,
|LS(h)−LD(h)|, is likely to be small.
Recall that LD(h) =Ez∼D[ℓ(h,z)] and that LS(h) =1
m∑m
i=1ℓ(h,zi). Since
eachziis sampled i.i.d. from D, the expected value of the random variable
ℓ(h,zi) isLD(h). By the linearity of expectation, it follows that LD(h) is also

m∑m
the expected value of LS(h). Hence, the quantity |LD(h)−LS(h)|is the deviation
of the random variable LS(h) from its expectation. We therefore need to show
that the measure of LS(h) isconcentrated around its expected value.
A basic statistical fact, the law of large numbers , states that when mgoes to
inﬁnity, empirical averages converge to their true expectation. This is true for
LS(h), since it is the empirical average of mi.i.d random variables. However, since

the law of large numbers is only an asymptotic result, it provides no information
about the gap between the empirically estimated error and its true value for any
given, ﬁnite, sample size.
Instead, we will use a measure concentration inequality due to Hoeﬀding, which
quantiﬁes the gap between empirical averages and their expected value.
lemma 4.5 (Hoeﬀding’s Inequality) Letθ1,...,θmbe a sequence of i.i.d. ran-
dom variables and assume that for all i,E[θi] =µandP[a≤θi≤b] = 1 . Then,
for anyϵ>0
P[⏐⏐⏐⏐⏐1
mm∑
i=1θi−µ⏐⏐⏐⏐⏐>ϵ]
≤2 exp(
−2mϵ2/(b−a)2)
.

for anyϵ>0
P[⏐⏐⏐⏐⏐1
mm∑
≤2 exp(
.
The proof can be found in Appendix B.
Getting back to our problem, let θibe the random variable ℓ(h,zi). Sinceh
is ﬁxed and z1,...,zmare sampled i.i.d., it follows that θ1,...,θmare also i.i.d.
random variables. Furthermore, LS(h) =1
m∑m
i=1θiandLD(h) =µ. Let us

4.2 Finite Classes Are Agnostic PAC Learnable 57
further assume that the range of ℓis [0,1] and therefore θi∈[0,1]. We therefore
obtain that
Dm({S:|LS(h)−LD(h)|>ϵ}) =P[⏐⏐⏐⏐⏐1
mm∑
≤2 exp(
−2mϵ2)
.
(4.2)
Combining this with Equation (4.1) yields
h∈H2 exp(
−2mϵ2)
= 2|H|exp(
−2mϵ2)
.
Finally, if we choose
m≥log(2|H|/δ)
2ϵ2
then
Dm({S:∃h∈H,|LS(h)−LD(h)|>ϵ})≤δ.
corollary 4.6 LetHbe a ﬁnite hypothesis class, let Zbe a domain, and let
ℓ:H×Z→[0,1]be a loss function. Then, Henjoys the uniform convergence
property with sample complexity
mUC
H(ϵ,δ)≤⌈log(2|H|/δ)
2ϵ2⌉
.
Furthermore, the class is agnostically PAC learnable using the ERM algorithm
with sample complexity
mH(ϵ,δ)≤mUC
H(ϵ/2,δ)≤⌈2 log(2|H|/δ)
ϵ2⌉
.
Remark 4.1 (The “Discretization Trick”) While the preceding corollary only
applies to ﬁnite hypothesis classes, there is a simple trick that allows us to get
a very good estimate of the practical sample complexity of inﬁnite hypothesis

mUC
2ϵ2⌉
.
ϵ2⌉
.
classes. Consider a hypothesis class that is parameterized by dparameters. For
example, letX=R,Y={±1}, and the hypothesis class, H, be all functions
of the form hθ(x) = sign(x−θ). That is, each hypothesis is parameterized by
one parameter, θ∈R, and the hypothesis outputs 1 for all instances larger than
θand outputs−1 for instances smaller than θ. This is a hypothesis class of an
inﬁnite size. However, if we are going to learn this hypothesis class in practice,

using a computer, we will probably maintain real numbers using ﬂoating point
representation, say, of 64 bits. It follows that in practice, our hypothesis class
is parameterized by the set of scalars that can be represented using a 64 bits
ﬂoating point number. There are at most 264such numbers; hence the actual
size of our hypothesis class is at most 264. More generally, if our hypothesis class
is parameterized by dnumbers, in practice we learn a hypothesis class of size at
most 264d. Applying Corollary 4.6 we obtain that the sample complexity of such

58 Learning via Uniform Convergence
classes is bounded by128d+2 log(2/δ)
ϵ2 . This upper bound on the sample complex-
ity has the deﬁciency of being dependent on the speciﬁc representation of real
numbers used by our machine. In Chapter 6 we will introduce a rigorous way
to analyze the sample complexity of inﬁnite size hypothesis classes. Neverthe-
less, the discretization trick can be used to get a rough estimate of the sample
complexity in many practical situations.
4.3 Summary
If the uniform convergence property holds for a hypothesis class Hthen in most
cases the empirical risks of hypotheses in Hwill faithfully represent their true
risks. Uniform convergence suﬃces for agnostic PAC learnability using the ERM
rule. We have shown that ﬁnite hypothesis classes enjoy the uniform convergence
property and are hence agnostic PAC learnable.
4.4 Bibliographic Remarks
Classes of functions for which the uniform convergence property holds are also

called Glivenko-Cantelli classes, named after Valery Ivanovich Glivenko and
Francesco Paolo Cantelli, who proved the ﬁrst uniform convergence result in
the 1930s. See (Dudley, Gine & Zinn 1991). The relation between uniform con-
vergence and learnability was thoroughly studied by Vapnik – see (Vapnik 1992,
Vapnik 1995, Vapnik 1998). In fact, as we will see later in Chapter 6, the funda-
mental theorem of learning theory states that in binary classiﬁcation problems,

uniform convergence is not only a suﬃcient condition for learnability but is also
a necessary condition. This is not the case for more general learning problems
(see (Shalev-Shwartz, Shamir, Srebro & Sridharan 2010)).
4.5 Exercises
1. In this exercise, we show that the ( ϵ,δ) requirement on the convergence of
errors in our deﬁnitions of PAC learning, is, in fact, quite close to a sim-
pler looking requirement about averages (or expectations). Prove that the
following two statements are equivalent (for any learning algorithm A, any

probability distribution D, and any loss function whose range is [0 ,1]):
1. For every ϵ,δ> 0, there exists m(ϵ,δ) such that∀m≥m(ϵ,δ)
P
S∼Dm[LD(A(S))>ϵ]<δ
2.
lim
m→∞E
S∼Dm[LD(A(S))] = 0

4.5 Exercises 59
(where ES∼Dmdenotes the expectation over samples Sof sizem).
2.Bounded loss functions: In Corollary 4.6 we assumed that the range of the
loss function is [0 ,1]. Prove that if the range of the loss function is [ a,b] then
the sample complexity satisﬁes
H(ϵ/2,δ)≤⌈2 log(2|H|/δ)(b−a)2
ϵ2⌉
.

5 The Bias-Complexity Tradeoﬀ
In Chapter 2 we saw that unless one is careful, the training data can mislead the
learner, and result in overﬁtting. To overcome this problem, we restricted the
search space to some hypothesis class H. Such a hypothesis class can be viewed
as reﬂecting some prior knowledge that the learner has about the task – a belief
that one of the members of the class His a low-error model for the task. For
example, in our papayas taste problem, on the basis of our previous experience
with other fruits, we may assume that some rectangle in the color-hardness plane
predicts (at least approximately) the papaya’s tastiness.
Is such prior knowledge really necessary for the success of learning? Maybe
there exists some kind of universal learner, that is, a learner who has no prior
knowledge about a certain task and is ready to be challenged by any task? Let
us elaborate on this point. A speciﬁc learning task is deﬁned by an unknown

distributionDoverX×Y , where the goal of the learner is to ﬁnd a predictor
h:X→Y , whose risk, LD(h), is small enough. The question is therefore whether
there exist a learning algorithm Aand a training set size m, such that for every
distributionD, ifAreceivesmi.i.d. examples from D, there is a high chance it
outputs a predictor hthat has a low risk.
The ﬁrst part of this chapter addresses this question formally. The No-Free-
Lunch theorem states that no such universal learner exists. To be more precise,

the theorem states that for binary classiﬁcation prediction tasks, for every learner
there exists a distribution on which it fails. We say that the learner fails if, upon
receiving i.i.d. examples from that distribution, its output hypothesis is likely
to have a large risk, say, ≥0.3, whereas for the same distribution, there exists
another learner that will output a hypothesis with a small risk. In other words,
the theorem states that no learner can succeed on all learnable tasks – every

learner has tasks on which it fails while other learners succeed.
Therefore, when approaching a particular learning problem, deﬁned by some
distributionD, we should have some prior knowledge on D. One type of such prior
knowledge is that Dcomes from some speciﬁc parametric family of distributions.
We will study learning under such assumptions later on in Chapter 24. Another
type of prior knowledge on D, which we assumed when deﬁning the PAC learning
model, is that there exists hin some predeﬁned hypothesis class H, such that

LD(h) = 0. A softer type of prior knowledge on Dis assuming that min h∈HLD(h)
is small. In a sense, this weaker assumption on Dis a prerequisite for using the

agnostic PAC model, in which we require that the risk of the output hypothesis
will not be much larger than min h∈HLD(h).
In the second part of this chapter we study the beneﬁts and pitfalls of using
a hypothesis class as a means of formalizing prior knowledge. We decompose
the error of an ERM algorithm over a class Hinto two components. The ﬁrst
component reﬂects the quality of our prior knowledge, measured by the minimal
risk of a hypothesis in our hypothesis class, min h∈HLD(h). This component is
also called the approximation error , or the biasof the algorithm toward choosing
a hypothesis from H. The second component is the error due to overﬁtting,
which depends on the size or the complexity of the class Hand is called the
estimation error . These two terms imply a tradeoﬀ between choosing a more
complexH(which can decrease the bias but increases the risk of overﬁtting)
or a less complex H(which might increase the bias but decreases the potential
overﬁtting).

5.1 The No-Free-Lunch Theorem
In this part we prove that there is no universal learner. We do this by showing
that no learner can succeed on all learning tasks, as formalized in the following
theorem:
theorem 5.1 (No-Free-Lunch) LetAbe any learning algorithm for the task
of binary classiﬁcation with respect to the 0−1loss over a domain X. Letm
be any number smaller than |X|/2, representing a training set size. Then, there
exists a distribution DoverX×{ 0,1}such that:
1. There exists a function f:X→{ 0,1}withLD(f) = 0 .

theorem:
2. With probability of at least 1/7over the choice of S∼ Dmwe have that
LD(A(S))≥1/8.
This theorem states that for every learner, there exists a task on which it fails,
even though that task can be successfully learned by another learner. Indeed, a
trivial successful learner in this case would be an ERM learner with the hypoth-
esis classH={f}, or more generally, ERM with respect to any ﬁnite hypothesis
class that contains fand whose size satisﬁes the equation m≥8 log(7|H|/6) (see
Corollary 2.3).

Proof LetCbe a subset ofXof size 2m. The intuition of the proof is that
any learning algorithm that observes only half of the instances in Chas no
information on what should be the labels of the rest of the instances in C.
Therefore, there exists a “reality,” that is, some target function f, that would
contradict the labels that A(S) predicts on the unobserved instances in C.
Note that there are T= 22mpossible functions from Cto{0,1}. Denote these

functions by f1,...,fT. For each such function, let Dibe a distribution over

62 The Bias-Complexity Tradeoﬀ
C×{0,1}deﬁned by
Di({(x,y)}) ={
1/|C|ify=fi(x)
0 otherwise .
That is, the probability to choose a pair ( x,y) is 1/|C|if the label yis indeed
the true label according to fi, and the probability is 0 if y̸=fi(x). Clearly,
LDi(fi) = 0.
We will show that for every algorithm, A, that receives a training set of m
examples from C×{0,1}and returns a function A(S) :C→{0,1}, it holds that
max
i∈[T]E
S∼Dm
i[LDi(A(S))]≥1/4. (5.1)
Clearly, this means that for every algorithm, A′, that receives a training set of m
examples fromX×{ 0,1}there exist a function f:X→{ 0,1}and a distribution
DoverX×{ 0,1}, such that LD(f) = 0 and
E
S∼Dm[LD(A′(S))]≥1/4. (5.2)
It is easy to verify that the preceding suﬃces for showing that P[LD(A′(S))≥
1/8]≥1/7, which is what we need to prove (see Exercise 1).
We now turn to proving that Equation (5.1) holds. There are k= (2m)m
possible sequences of mexamples from C. Denote these sequences by S1,...,Sk.
Also, ifSj= (x1,...,xm) we denote by Si

E
jthe sequence containing the instances
inSjlabeled by the function fi, namely,Si
j= ((x1,fi(x1)),..., (xm,fi(xm))). If
the distribution is Dithen the possible training sets Acan receive are Si
1,...,Si
k,
and all these training sets have the same probability of being sampled. Therefore,
E
S∼Dm
i[LDi(A(S))] =1
kk∑
j=1LDi(A(Si
j)). (5.3)
Using the facts that “maximum” is larger than “average” and that “average” is
larger than “minimum,” we have
max
i∈[T]1
kk∑
j))≥1
TT∑
i=11
kk∑
j))
=1
kk∑
j=11
TT∑
i=1LDi(A(Si
j))
≥min
j∈[k]1
TT∑

1,...,Si
k,
E
S∼Dm
kk∑
j)). (5.3)
max
i∈[T]1
kk∑
j))≥1
TT∑
i=11
kk∑
j))
=1
kk∑
j=11
TT∑
j))
≥min
j∈[k]1
TT∑
j)). (5.4)
Next, ﬁx some j∈[k]. DenoteSj= (x1,...,xm) and letv1,...,vpbe the
examples in Cthat do not appear in Sj. Clearly,p≥m. Therefore, for every

5.1 The No-Free-Lunch Theorem 63
functionh:C→{0,1}and everyiwe have
LDi(h) =1
2m∑
x∈C1[h(x)̸=fi(x)]
≥1
2mp∑
r=11[h(vr)̸=fi(vr)]
≥1
2pp∑
r=11[h(vr)̸=fi(vr)]. (5.5)
Hence,
1
TT∑
j))≥1
TT∑
i=11
2pp∑
r=11[A(Si
j)(vr)̸=fi(vr)]
=1
2pp∑
r=11
TT∑
i=11[A(Si
≥1
2·min
r∈[p]1
TT∑
i=11[A(Si
j)(vr)̸=fi(vr)]. (5.6)
Next, ﬁx some r∈[p]. We can partition all the functions in f1,...,fTintoT/2
disjoint pairs, where for a pair ( fi,fi′) we have that for every c∈C,fi(c)̸=fi′(c)
if and only if c=vr. Since for such a pair we must have Si
j=Si′
j, it follows that
1[A(Si
j)(vr)̸=fi(vr)]+1[A(Si′
j)(vr)̸=fi′(vr)]= 1,
which yields
1
TT∑
i=11[A(Si
j)(vr)̸=fi(vr)]=1
2.
Combining this with Equation (5.6), Equation (5.4), and Equation (5.3), we
obtain that Equation (5.1) holds, which concludes our proof.
5.1.1 No-Free-Lunch and Prior Knowledge
How does the No-Free-Lunch result relate to the need for prior knowledge? Let us
consider an ERM predictor over the hypothesis class Hof all the functions ffrom

j=Si′
1[A(Si
1
TT∑
i=11[A(Si
2.
Xto{0,1}. This class represents lack of prior knowledge: Every possible function
from the domain to the label set is considered a good candidate. According to the
No-Free-Lunch theorem, any algorithm that chooses its output from hypotheses
inH, and in particular the ERM predictor, will fail on some learning task.
Therefore, this class is not PAC learnable, as formalized in the following corollary:
corollary 5.2 LetXbe an inﬁnite domain set and let Hbe the set of all
functions fromXto{0,1}. Then,His not PAC learnable.

64 The Bias-Complexity Tradeoﬀ
Proof Assume, by way of contradiction, that the class is learnable. Choose
someϵ <1/8 andδ < 1/7. By the deﬁnition of PAC learnability, there must
be some learning algorithm Aand an integer m=m(ϵ,δ), such that for any
data-generating distribution over X×{ 0,1}, if for some function f:X→{ 0,1},
LD(f) = 0, then with probability greater than 1 −δwhenAis applied to
samplesSof sizem, generated i.i.d. by D,LD(A(S))≤ϵ. However, applying
the No-Free-Lunch theorem, since |X|>2m, for every learning algorithm (and
in particular for the algorithm A), there exists a distribution Dsuch that with
probability greater than 1 /7> δ,LD(A(S))>1/8> ϵ, which leads to the
desired contradiction.
How can we prevent such failures? We can escape the hazards foreseen by the
No-Free-Lunch theorem by using our prior knowledge about a speciﬁc learning
task, to avoid the distributions that will cause us to fail when learning that task.
Such prior knowledge can be expressed by restricting our hypothesis class.

But how should we choose a good hypothesis class? On the one hand, we want
to believe that this class includes the hypothesis that has no error at all (in the
PAC setting), or at least that the smallest error achievable by a hypothesis from
this class is indeed rather small (in the agnostic setting). On the other hand,
we have just seen that we cannot simply choose the richest class – the class of
all functions over the given domain. This tradeoﬀ is discussed in the following
section.
5.2 Error Decomposition

section.
To answer this question we decompose the error of an ERM Hpredictor into two
components as follows. Let hSbe an ERMHhypothesis. Then, we can write
LD(hS) =ϵapp+ϵestwhere :ϵapp= min
h∈HLD(h), ϵ est=LD(hS)−ϵapp.(5.7)
•The Approximation Error – the minimum risk achievable by a predictor
in the hypothesis class. This term measures how much risk we have because
we restrict ourselves to a speciﬁc class, namely, how much inductive bias we
have. The approximation error does not depend on the sample size and is
determined by the hypothesis class chosen. Enlarging the hypothesis class

can decrease the approximation error.
Under the realizability assumption, the approximation error is zero. In
the agnostic case, however, the approximation error can be large.1
1In fact, it always includes the error of the Bayes optimal predictor (see Chapter 3), the
minimal yet inevitable error, because of the possible nondeterminism of the world in this
model. Sometimes in the literature the term approximation error refers not to

minh∈HLD(h), but rather to the excess error over that of the Bayes optimal predictor,
namely, min h∈HLD(h)−ϵBayes .

•The Estimation Error – the diﬀerence between the approximation error
and the error achieved by the ERM predictor. The estimation error results
because the empirical risk (i.e., training error) is only an estimate of the
true risk, and so the predictor minimizing the empirical risk is only an
estimate of the predictor minimizing the true risk.
The quality of this estimation depends on the training set size and
on the size, or complexity, of the hypothesis class. As we have shown, for
a ﬁnite hypothesis class, ϵestincreases (logarithmically) with |H|and de-
creases with m. We can think of the size of Has a measure of its complexity.
In future chapters we will deﬁne other complexity measures of hypothesis
classes.
Since our goal is to minimize the total risk, we face a tradeoﬀ, called the bias-
complexity tradeoﬀ . On one hand, choosing Hto be a very rich class decreases the
approximation error but at the same time might increase the estimation error,

classes.
as a richHmight lead to overﬁtting . On the other hand, choosing Hto be a
very small set reduces the estimation error but might increase the approximation
error or, in other words, might lead to underﬁtting . Of course, a great choice for
His the class that contains only one classiﬁer – the Bayes optimal classiﬁer. But
the Bayes optimal classiﬁer depends on the underlying distribution D, which we
do not know (indeed, learning would have been unnecessary had we known D).

Learning theory studies how rich we can make Hwhile still maintaining rea-
sonable estimation error. In many cases, empirical research focuses on designing
good hypothesis classes for a certain domain. Here, “good” means classes for
which the approximation error would not be excessively high. The idea is that
although we are not experts and do not know how to construct the optimal clas-
siﬁer, we still have some prior knowledge of the speciﬁc problem at hand, which
enables us to design hypothesis classes for which both the approximation error

and the estimation error are not too large. Getting back to our papayas example,
we do not know how exactly the color and hardness of a papaya predict its taste,
but we do know that papaya is a fruit and on the basis of previous experience
with other fruit we conjecture that a rectangle in the color-hardness space may
be a good predictor.
5.3 Summary
The No-Free-Lunch theorem states that there is no universal learner. Every
learner has to be speciﬁed to some task, and use some prior knowledge about

that task, in order to succeed. So far we have modeled our prior knowledge by
restricting our output hypothesis to be a member of a chosen hypothesis class.
When choosing this hypothesis class, we face a tradeoﬀ, between a larger, or
more complex, class that is more likely to have a small approximation error,
and a more restricted class that would guarantee that the estimation error will

66 The Bias-Complexity Tradeoﬀ
be small. In the next chapter we will study in more detail the behavior of the
estimation error. In Chapter 7 we will discuss alternative ways to express prior
knowledge.
5.4 Bibliographic Remarks
(Wolpert & Macready 1997) proved several no-free-lunch theorems for optimiza-
tion, but these are rather diﬀerent from the theorem we prove here. The theorem
we prove here is closely related to lower bounds in VC theory, as we will study
in the next chapter.
5.5 Exercises
1. Prove that Equation (5.2) suﬃces for showing that P[LD(A(S))≥1/8]≥1/7.
Hint: Letθbe a random variable that receives values in [0 ,1] and whose
expectation satisﬁes E[θ]≥1/4. Use Lemma B.1 to show that P[θ≥1/8]≥
1/7.
2. Assume you are asked to design a learning algorithm to predict whether pa-
tients are going to suﬀer a heart attack. Relevant patient features the al-
gorithm may have access to include blood pressure (BP), body-mass index
(BMI), age (A), level of physical activity (P), and income (I).

1/7.
You have to choose between two algorithms; the ﬁrst picks an axis aligned
rectangle in the two dimensional space spanned by the features BP and BMI
and the other picks an axis aligned rectangle in the ﬁve dimensional space
spanned by all the preceding features.
1. Explain the pros and cons of each choice.
2. Explain how the number of available labeled training samples will aﬀect
your choice.
3. Prove that if |X|≥kmfor a positive integer k≥2, then we can replace

the lower bound of 1 /4 in the No-Free-Lunch theorem withk−1
2k=1
2−1
2k.
Namely, let Abe a learning algorithm for the task of binary classiﬁcation. Let
mbe any number smaller than |X|/k, representing a training set size. Then,
there exists a distribution DoverX×{ 0,1}such that:
•There exists a function f:X→{ 0,1}withLD(f) = 0.
•ES∼Dm[LD(A(S))]≥1
2−1
2k.

6 The VC-Dimension
In the previous chapter, we decomposed the error of the ERM Hrule into ap-
proximation error and estimation error. The approximation error depends on
the ﬁt of our prior knowledge (as reﬂected by the choice of the hypothesis class
H) to the underlying unknown distribution. In contrast, the deﬁnition of PAC
learnability requires that the estimation error would be bounded uniformly over
all distributions.
Our current goal is to ﬁgure out which classes Hare PAC learnable, and to
characterize exactly the sample complexity of learning a given hypothesis class.
So far we have seen that ﬁnite classes are learnable, but that the class of all
functions (over an inﬁnite size domain) is not. What makes one class learnable
and the other unlearnable? Can inﬁnite-size classes be learnable, and, if so, what
determines their sample complexity?
We begin the chapter by showing that inﬁnite classes can indeed be learn-
able, and thus, ﬁniteness of the hypothesis class is not a necessary condition for

learnability. We then present a remarkably crisp characterization of the family
of learnable classes in the setup of binary valued classiﬁcation with the zero-one
loss. This characterization was ﬁrst discovered by Vladimir Vapnik and Alexey
Chervonenkis in 1970 and relies on a combinatorial notion called the Vapnik-
Chervonenkis dimension (VC-dimension). We formally deﬁne the VC-dimension,
provide several examples, and then state the fundamental theorem of statistical
learning theory, which integrates the concepts of learnability, VC-dimension, the

ERM rule, and uniform convergence.
6.1 Inﬁnite-Size Classes Can Be Learnable
In Chapter 4 we saw that ﬁnite classes are learnable, and in fact the sample
complexity of a hypothesis class is upper bounded by the log of its size. To show
that the size of the hypothesis class is not the right characterization of its sample
complexity, we ﬁrst present a simple example of an inﬁnite-size hypothesis class
that is learnable.
Example 6.1 LetHbe the set of threshold functions over the real line, namely,

H={ha:a∈R}, whereha:R→{0,1}is a function such that ha(x) =1[x<a].
To remind the reader, 1[x<a]is 1 ifx<a and 0 otherwise. Clearly, His of inﬁnite

68 The VC-Dimension
size. Nevertheless, the following lemma shows that His learnable in the PAC
model using the ERM algorithm.
Lemma 6.1 Let Hbe the class of thresholds as deﬁned earlier. Then, His
PAC learnable, using the ERM rule, with sample complexity of mH(ϵ,δ)≤
⌈log(2/δ)/ϵ⌉.
Proof Leta⋆be a threshold such that the hypothesis h⋆(x) =1[x<a⋆]achieves
LD(h⋆) = 0. LetDxbe the marginal distribution over the domain Xand let
a0<a⋆<a1be such that
P
x∼Dx[x∈(a0,a⋆)] = P
x∼Dx[x∈(a⋆,a1)] =ϵ.
a⋆ a0 a1ϵmass ϵmass
(IfDx(−∞,a⋆)≤ϵwe seta0=−∞and similarly for a1). Given a training set
S, letb0= max{x: (x,1)∈S}andb1= min{x: (x,0)∈S}(if no example in S
is positive we set b0=−∞ and if no example in Sis negative we set b1=∞).
LetbSbe a threshold corresponding to an ERM hypothesis, hS, which implies
thatbS∈(b0,b1). Therefore, a suﬃcient condition for LD(hS)≤ϵis that both
b0≥a0andb1≤a1. In other words,
P
S∼Dm[LD(hS)>ϵ]≤P
S∼Dm[b0<a0∨b1>a1],
and using the union bound we can bound the preceding by
P
S∼Dm[b0<a0] +P

P
P
S∼Dm[b1>a1]. (6.1)
The eventb0<a0happens if and only if all examples in Sare not in the interval
(a0,a∗), whose probability mass is deﬁned to be ϵ, namely,
P
S∼Dm[b0<a0] =P
S∼Dm[∀(x,y)∈S, x̸∈(a0,a⋆)] = (1−ϵ)m≤e−ϵm.
Since we assume m > log(2/δ)/ϵit follows that the equation is at most δ/2.
In the same way it is easy to see that PS∼Dm[b1> a1]≤δ/2. Combining with
Equation (6.1) we conclude our proof.
6.2 The VC-Dimension
We see, therefore, that while ﬁniteness of His a suﬃcient condition for learn-

P
ability, it is not a necessary condition. As we will show, a property called the
VC-dimension of a hypothesis class gives the correct characterization of its learn-
ability. To motivate the deﬁnition of the VC-dimension, let us recall the No-Free-
Lunch theorem (Theorem 5.1) and its proof. There, we have shown that without

6.2 The VC-Dimension 69
restricting the hypothesis class, for any learning algorithm, an adversary can
construct a distribution for which the learning algorithm will perform poorly,
while there is another learning algorithm that will succeed on the same distri-
bution. To do so, the adversary used a ﬁnite set C⊂X and considered a family
of distributions that are concentrated on elements of C. Each distribution was
derived from a “true” target function from Cto{0,1}. To make any algorithm
fail, the adversary used the power of choosing a target function from the set of
allpossible functions from Cto{0,1}.
When considering PAC learnability of a hypothesis class H, the adversary
is restricted to constructing distributions for which some hypothesis h∈ H
achieves a zero risk. Since we are considering distributions that are concentrated
on elements of C, we should study how Hbehaves on C, which leads to the
following deﬁnition.
definition 6.2 (Restriction of HtoC) LetHbe a class of functions from X

to{0,1}and letC={c1,...,cm}⊂X . The restriction of HtoCis the set of
functions from Cto{0,1}that can be derived from H. That is,
HC={(h(c1),...,h (cm)) :h∈H},
where we represent each function from Cto{0,1}as a vector in{0,1}|C|.
If the restriction of HtoCis the set of all functions from Cto{0,1}, then
we say thatHshatters the setC. Formally:
definition 6.3 (Shattering) A hypothesis class Hshatters a ﬁnite set C⊂X
if the restriction of HtoCis the set of all functions from Cto{0,1}. That is,
|HC|= 2|C|.
Example 6.2 LetHbe the class of threshold functions over R. Take a set

C={c1}. Now, if we take a=c1+ 1, then we have ha(c1) = 1, and if we take
a=c1−1, then we have ha(c1) = 0. Therefore, HCis the set of all functions
fromCto{0,1}, andHshattersC. Now take a set C={c1,c2}, wherec1≤c2.
Noh∈H can account for the labeling (0 ,1), because any threshold that assigns
the label 0 to c1must assign the label 0 to c2as well. Therefore not all functions
fromCto{0,1}are included inHC; henceCis not shattered by H.
Getting back to the construction of an adversarial distribution as in the proof

of the No-Free-Lunch theorem (Theorem 5.1), we see that whenever some set C
is shattered byH, the adversary is not restricted by H, as they can construct
a distribution over Cbased on anytarget function from Cto{0,1}, while still
maintaining the realizability assumption. This immediately yields:
corollary 6.4 LetHbe a hypothesis class of functions from Xto{0,1}. Let
mbe a training set size. Assume that there exists a set C⊂X of size 2mthat is
shattered byH. Then, for any learning algorithm, A, there exist a distribution D

overX×{ 0,1}and a predictor h∈H such thatLD(h) = 0 but with probability
of at least 1/7over the choice of S∼Dmwe have that LD(A(S))≥1/8.

70 The VC-Dimension
Corollary 6.4 tells us that if Hshatters some set Cof size 2mthen we cannot
learnHusingmexamples. Intuitively, if a set Cis shattered byH, and we
receive a sample containing half the instances of C, the labels of these instances
give us no information about the labels of the rest of the instances in C– every
possible labeling of the rest of the instances can be explained by some hypothesis
inH. Philosophically,
If someone can explain every phenomenon, his explanations are worthless.
This leads us directly to the deﬁnition of the VC dimension.
definition 6.5 (VC-dimension) The VC-dimension of a hypothesis class H,
denoted VCdim(H), is the maximal size of a set C⊂X that can be shattered
byH. IfHcan shatter sets of arbitrarily large size we say that Hhas inﬁnite
VC-dimension.
A direct consequence of Corollary 6.4 is therefore:
theorem 6.6 LetHbe a class of inﬁnite VC-dimension. Then, His not PAC
learnable.
Proof SinceHhas an inﬁnite VC-dimension, for any training set size m, there

exists a shattered set of size 2 m, and the claim follows by Corollary 6.4.
We shall see later in this chapter that the converse is also true: A ﬁnite VC-
dimension guarantees learnability. Hence, the VC-dimension characterizes PAC
learnability. But before delving into more theory, we ﬁrst show several examples.
6.3 Examples
In this section we calculate the VC-dimension of several hypothesis classes. To
show that VCdim( H) =dwe need to show that
1. There exists a set Cof sizedthat is shattered by H.

2. Every set Cof sized+ 1 is not shattered by H.
6.3.1 Threshold Functions
LetHbe the class of threshold functions over R. Recall Example 6.2, where
we have shown that for an arbitrary set C={c1},HshattersC; therefore
VCdim(H)≥1. We have also shown that for an arbitrary set C={c1,c2}where
c1≤c2,Hdoes not shatter C. We therefore conclude that VCdim( H) = 1.

6.3 Examples 71
6.3.2 Intervals
LetHbe the class of intervals over R, namely,H={ha,b:a,b∈R,a < b},
whereha,b:R→{0,1}is a function such that ha,b(x) =1[x∈(a,b)]. Take the set
C={1,2}. Then,HshattersC(make sure you understand why) and therefore
VCdim(H)≥2. Now take an arbitrary set C={c1,c2,c3}and assume without
loss of generality that c1≤c2≤c3. Then, the labeling (1 ,0,1) cannot be obtained
by an interval and therefore Hdoes not shatter C. We therefore conclude that
VCdim(H) = 2.
6.3.3 Axis Aligned Rectangles
LetHbe the class of axis aligned rectangles, formally:
H={h(a1,a2,b1,b2):a1≤a2andb1≤b2}
where
h(a1,a2,b1,b2)(x1,x2) ={
1 ifa1≤x1≤a2andb1≤x2≤b2
0 otherwise(6.2)
We shall show in the following that VCdim( H) = 4. To prove this we need
to ﬁnd a set of 4 points that are shattered by H, and show that no set of 5
points can be shattered by H. Finding a set of 4 points that are shattered is
easy (see Figure 6.1). Now, consider any set C⊂R2of 5 points. In C, take a

where
leftmost point (whose ﬁrst coordinate is the smallest in C), a rightmost point
(ﬁrst coordinate is the largest), a lowest point (second coordinate is the smallest),
and a highest point (second coordinate is the largest). Without loss of generality,
denoteC={c1,...,c 5}and letc5be the point that was not selected. Now,
deﬁne the labeling (1 ,1,1,1,0). It is impossible to obtain this labeling by an
axis aligned rectangle. Indeed, such a rectangle must contain c1,...,c 4; but in

this case the rectangle contains c5as well, because its coordinates are within
the intervals deﬁned by the selected points. So, Cis not shattered by H, and
therefore VCdim( H) = 4.
c1
c2
c3c4c5
Figure 6.1 Left: 4 points that are shattered by axis aligned rectangles. Right: Any axis
aligned rectangle cannot label c5by 0 and the rest of the points by 1.

72 The VC-Dimension
6.3.4 Finite Classes
LetHbe a ﬁnite class. Then, clearly, for any set Cwe have|HC|≤|H| and thusC
cannot be shattered if |H|<2|C|. This implies that VCdim( H)≤log2(|H|). This
shows that the PAC learnability of ﬁnite classes follows from the more general
statement of PAC learnability of classes with ﬁnite VC-dimension, which we shall
see in the next section. Note, however, that the VC-dimension of a ﬁnite class
Hcan be signiﬁcantly smaller than log2(|H|). For example, let X={1,...,k},
for some integer k, and consider the class of threshold functions (as deﬁned in
Example 6.2). Then, |H|=kbut VCdim(H) = 1. Since kcan be arbitrarily
large, the gap between log2(|H|) and VCdim(H) can be arbitrarily large.
6.3.5 VC-Dimension and the Number of Parameters
In the previous examples, the VC-dimension happened to equal the number of
parameters deﬁning the hypothesis class. While this is often the case, it is not
always true. Consider, for example, the domain X=R, and the hypothesis class

H={hθ:θ∈R}wherehθ:X→{ 0,1}is deﬁned by hθ(x) =⌈0.5 sin(θx)⌉. It
is possible to prove that VCdim( H) =∞, namely, for everyd, one can ﬁnd d
points that are shattered by H(see Exercise 8).
6.4 The Fundamental Theorem of PAC learning
We have already shown that a class of inﬁnite VC-dimension is not learnable. The
converse statement is also true, leading to the fundamental theorem of statistical
learning theory:
theorem 6.7 (The Fundamental Theorem of Statistical Learning) LetHbe a

hypothesis class of functions from a domain Xto{0,1}and let the loss function
be the 0−1loss. Then, the following are equivalent:
1.Hhas the uniform convergence property.
2. Any ERM rule is a successful agnostic PAC learner for H.
3.His agnostic PAC learnable.
4.His PAC learnable.
5. Any ERM rule is a successful PAC learner for H.
6.Hhas a ﬁnite VC-dimension.
The proof of the theorem is given in the next section.
Not only does the VC-dimension characterize PAC learnability; it even deter-
mines the sample complexity.

theorem 6.8 (The Fundamental Theorem of Statistical Learning – Quantita-
tive Version) LetHbe a hypothesis class of functions from a domain Xto{0,1}
and let the loss function be the 0−1loss. Assume that VCdim(H) =d <∞.
Then, there are absolute constants C1,C2such that:

1.Hhas the uniform convergence property with sample complexity
C1d+ log(1/δ)
ϵ2≤mUC
H(ϵ,δ)≤C2d+ log(1/δ)
ϵ2
2.His agnostic PAC learnable with sample complexity
ϵ2≤mH(ϵ,δ)≤C2d+ log(1/δ)
ϵ2
3.His PAC learnable with sample complexity
ϵ≤mH(ϵ,δ)≤C2dlog(1/ϵ) + log(1/δ)
ϵ
The proof of this theorem is given in Chapter 28.
Remark 6.3 We stated the fundamental theorem for binary classiﬁcation tasks.
A similar result holds for some other learning problems such as regression with
the absolute loss or the squared loss. However, the theorem does not hold for
all learning tasks. In particular, learnability is sometimes possible even though
the uniform convergence property does not hold (we will see an example in
Chapter 13, Exercise 2). Furthermore, in some situations, the ERM rule fails
but learnability is possible with other learning rules.
6.5 Proof of Theorem 6.7
We have already seen that 1 →2 in Chapter 4. The implications 2 →3 and

3→4 are trivial and so is 2 →5. The implications 4 →6 and 5→6 follow from
the No-Free-Lunch theorem. The diﬃcult part is to show that 6 →1. The proof
is based on two main claims:
•If VCdim(H) =d, then even though Hmight be inﬁnite, when restricting it
to a ﬁnite set C⊂X, its “eﬀective” size, |HC|, is onlyO(|C|d). That is,
the size ofHCgrows polynomially rather than exponentially with |C|. This
claim is often referred to as Sauer’s lemma , but it has also been stated and
proved independently by Shelah and by Perles. The formal statement is

given in Section 6.5.1 later.
•In Section 4 we have shown that ﬁnite hypothesis classes enjoy the uniform
convergence property. In Section 6.5.2 later we generalize this result and
show that uniform convergence holds whenever the hypothesis class has a
“small eﬀective size.” By “small eﬀective size” we mean classes for which
|HC|grows polynomially with |C|.
6.5.1 Sauer’s Lemma and the Growth Function
We deﬁned the notion of shattering , by considering the restriction of Hto a ﬁnite

set of instances. The growth function measures the maximal “eﬀective” size of
Hon a set of mexamples. Formally:

74 The VC-Dimension
definition 6.9 (Growth Function) Let Hbe a hypothesis class. Then the
growth function ofH, denotedτH:N→N, is deﬁned as
τH(m) = max
C⊂X:|C|=m⏐⏐HC⏐⏐.
In words,τH(m) is the number of diﬀerent functions from a set Cof sizemto
{0,1}that can be obtained by restricting HtoC.
Obviously, if VCdim( H) =dthen for any m≤dwe haveτH(m) = 2m. In
such cases,Hinduces all possible functions from Cto{0,1}. The following beau-
tiful lemma, proposed independently by Sauer, Shelah, and Perles, shows that
whenmbecomes larger than the VC-dimension, the growth function increases
polynomially rather than exponentially with m.
lemma 6.10 (Sauer-Shelah-Perles) LetHbe a hypothesis class with VCdim(H)≤
d <∞. Then, for all m,τH(m)≤∑d
i=0(m
i)
. In particular, if m > d + 1 then
τH(m)≤(em/d )d.
Proof of Sauer’s Lemma *
To prove the lemma it suﬃces to prove the following stronger claim: For any
C={c1,...,cm}we have
∀H,|HC| ≤ |{B⊆C:HshattersB}|. (6.3)

i=0(m
i)
The reason why Equation (6.3) is suﬃcient to prove the lemma is that if VCdim( H)≤
dthen no set whose size is larger than dis shattered byHand therefore
|{B⊆C:HshattersB}| ≤d∑
i=0(m
i)
.
Whenm > d + 1 the right-hand side of the preceding is at most ( em/d )d(see
Lemma A.5 in Appendix A).
We are left with proving Equation (6.3) and we do it using an inductive argu-
ment. Form= 1, no matter what His, either both sides of Equation (6.3) equal
1 or both sides equal 2 (the empty set is always considered to be shattered by

i=0(m
i)
.
H). Assume Equation (6.3) holds for sets of size k < m and let us prove it for
sets of size m. FixHandC={c1,...,cm}. DenoteC′={c2,...,cm}and in
addition, deﬁne the following two sets:
Y0={(y2,...,ym) : (0,y2,...,ym)∈HC∨(1,y2,...,ym)∈HC},
and
Y1={(y2,...,ym) : (0,y2,...,ym)∈HC∧(1,y2,...,ym)∈HC}.
It is easy to verify that |HC|=|Y0|+|Y1|. Additionally, since Y0=HC′, using
the induction assumption (applied on HandC′) we have that
|Y0|=|HC′|≤|{B⊆C′:HshattersB}|=|{B⊆C:c1̸∈B∧HshattersB}|.

6.5 Proof of Theorem 6.7 75
Next, deﬁneH′⊆H to be
H′={h∈H:∃h′∈H s.t. (1−h′(c1),h′(c2),...,h′(cm))
= (h(c1),h(c2),...,h (cm)},
namely,H′contains pairs of hypotheses that agree on C′and diﬀer on c1. Using
this deﬁnition, it is clear that if H′shatters a set B⊆C′then it also shatters
the setB∪{c1}and vice versa. Combining this with the fact that Y1=H′
C′and
using the inductive assumption (now applied on H′andC′) we obtain that
|Y1|=|H′
C′|≤|{B⊆C′:H′shattersB}|=|{B⊆C′:H′shattersB∪{c1}}|
=|{B⊆C:c1∈B∧H′shattersB}|≤|{B⊆C:c1∈B∧HshattersB}|.
Overall, we have shown that
|HC|=|Y0|+|Y1|
≤ |{B⊆C:c1̸∈B∧HshattersB}|+|{B⊆C:c1∈B∧HshattersB}|
=|{B⊆C:HshattersB}|,
which concludes our proof.
6.5.2 Uniform Convergence for Classes of Small Eﬀective Size
In this section we prove that if Hhas small eﬀective size then it enjoys the
uniform convergence property. Formally,
theorem 6.11 LetHbe a class and let τHbe its growth function. Then, for
everyDand everyδ∈(0,1), with probability of at least 1−δover the choice of
S∼Dmwe have

|LD(h)−LS(h)|≤4 +√
log(τH(2m))
δ√
2m.
Before proving the theorem, let us ﬁrst conclude the proof of Theorem 6.7.
Proof of Theorem 6.7 It suﬃces to prove that if the VC-dimension is ﬁnite then
the uniform convergence property holds. We will prove that
mUC
H(ϵ,δ)≤416d
(δϵ)2log(16d
(δϵ)2)
+16dlog(2e/d)
(δϵ)2.
From Sauer’s lemma we have that for m > d ,τH(2m)≤(2em/d )d. Combining
this with Theorem 6.11 we obtain that with probability of at least 1 −δ,
|LS(h)−LD(h)|≤4 +√
dlog(2em/d )
δ√
2m.
For simplicity assume that√
dlog(2em/d )≥4; hence,

δ√
2m.
mUC
(δϵ)2)
(δϵ)2.
δ√
2m.
|LS(h)−LD(h)|≤1
δ√
2dlog(2em/d )
m.

76 The VC-Dimension
To ensure that the preceding is at most ϵwe need that
m≥2dlog(m)
(δϵ)2+2dlog(2e/d)
(δϵ)2.
Standard algebraic manipulations (see Lemma A.2 in Appendix A) show that a
suﬃcient condition for the preceding to hold is that
m≥42d
(δϵ)2log(2d
(δϵ)2)
+4dlog(2e/d)
(δϵ)2.
Remark 6.4 The upper bound on mUC
Hwe derived in the proof Theorem 6.7
is not the tightest possible. A tighter analysis that yields the bounds given in
Theorem 6.8 can be found in Chapter 28.
Proof of Theorem 6.11 *
We will start by showing that
E
S∼Dm[
sup
h∈H|LD(h)−LS(h)|]
≤4 +√
log(τH(2m))√
2m. (6.4)
Since the random variable suph∈H|LD(h)−LS(h)|is nonnegative, the proof of
the theorem follows directly from the preceding using Markov’s inequality (see
Section B.1).
To bound the left-hand side of Equation (6.4) we ﬁrst note that for every
h∈H, we can rewrite LD(h) =ES′∼Dm[LS′(h)], whereS′=z′
1,...,z′
mis an
additional i.i.d. sample. Therefore,
E
S∼Dm[
sup
=E
S∼Dm[
sup
h∈H⏐⏐⏐E
S′∼DmLS′(h)−LS(h)⏐⏐⏐]
.

E
S∼Dm[
sup
≤4 +√
2m. (6.4)
1,...,z′
mis an
E
S∼Dm[
sup
=E
S∼Dm[
sup
h∈H⏐⏐⏐E
.
A generalization of the triangle inequality yields
⏐⏐⏐E
S′∼Dm[LS′(h)−LS(h)]⏐⏐⏐≤E
S′∼Dm|LS′(h)−LS(h)|,
and the fact that supermum of expectation is smaller than expectation of supre-
mum yields
sup
h∈HE
S′∼Dm|LS′(h)−LS(h)|≤ E
S′∼Dmsup
h∈H|LS′(h)−LS(h)|.
Formally, the previous two inequalities follow from Jensen’s inequality. Combin-
ing all we obtain
E
S∼Dm[
sup
≤E
S,S′∼Dm[
sup
h∈H|LS′(h)−LS(h)|]
=E
S,S′∼Dm[
sup
h∈H1
m⏐⏐⏐⏐⏐m∑
i=1(ℓ(h,z′
i)−ℓ(h,zi))⏐⏐⏐⏐⏐]
.
(6.5)

6.5 Proof of Theorem 6.7 77
The expectation on the right-hand side is over a choice of two i.i.d. samples
S=z1,...,zmandS′=z′
1,...,z′
m. Since all of these 2 mvectors are chosen
i.i.d., nothing will change if we replace the name of the random vector ziwith the
name of the random vector z′
i. If we do it, instead of the term ( ℓ(h,z′
i)−ℓ(h,zi))
in Equation (6.5) we will have the term −(ℓ(h,z′
i)−ℓ(h,zi)). It follows that for
everyσ∈{± 1}mwe have that Equation (6.5) equals
E
S,S′∼Dm[
sup
h∈H1
m⏐⏐⏐⏐⏐m∑
i=1σi(ℓ(h,z′
Since this holds for every σ∈{± 1}m, it also holds if we sample each component
ofσuniformly at random from the uniform distribution over {±1}, denotedU±.
Hence, Equation (6.5) also equals
E
σ∼Um
±E
S,S′∼Dm[
sup
h∈H1
m⏐⏐⏐⏐⏐m∑
,
and by the linearity of expectation it also equals
E
S,S′∼DmE
σ∼Um
±[
sup
h∈H1
m⏐⏐⏐⏐⏐m∑
.
Next, ﬁxSandS′, and letCbe the instances appearing in SandS′. Then, we

E
S,S′∼Dm[
sup
h∈H1
m⏐⏐⏐⏐⏐m∑
E
σ∼Um
±E
S,S′∼Dm[
sup
h∈H1
m⏐⏐⏐⏐⏐m∑
,
E
S,S′∼DmE
σ∼Um
±[
sup
h∈H1
m⏐⏐⏐⏐⏐m∑
.
can take the supremum only over h∈HC. Therefore,
E
σ∼Um
±[
sup
h∈H1
m⏐⏐⏐⏐⏐m∑
=E
σ∼Um
±[
max
h∈HC1
m⏐⏐⏐⏐⏐m∑
.
Fix someh∈HCand denote θh=1
m∑m
i)−ℓ(h,zi)). Since E[θh] = 0
andθhis an average of independent variables, each of which takes values in
[−1,1], we have by Hoeﬀding’s inequality that for every ρ>0,
P[|θh|>ρ]≤2 exp(
−2mρ2)
.
Applying the union bound over h∈HC, we obtain that for any ρ>0,
P[
max
h∈HC|θh|>ρ]
≤2|HC|exp(
−2mρ2)
.

E
σ∼Um
±[
sup
h∈H1
m⏐⏐⏐⏐⏐m∑
=E
σ∼Um
±[
max
h∈HC1
m⏐⏐⏐⏐⏐m∑
.
m∑m
−2mρ2)
.
P[
max
−2mρ2)
.
Finally, Lemma A.4 in Appendix A tells us that the preceding implies
E[
max
h∈HC|θh|]
≤4 +√
log(|HC|)√
2m.
Combining all with the deﬁnition of τH, we have shown that
E
S∼Dm[
sup
≤4 +√
2m.

78 The VC-Dimension
6.6 Summary
The fundamental theorem of learning theory characterizes PAC learnability of
classes of binary classiﬁers using VC-dimension. The VC-dimension of a class
is a combinatorial property that denotes the maximal sample size that can be
shattered by the class. The fundamental theorem states that a class is PAC learn-
able if and only if its VC-dimension is ﬁnite and speciﬁes the sample complexity
required for PAC learning. The theorem also shows that if a problem is at all
learnable, then uniform convergence holds and therefore the problem is learnable
using the ERM rule.
6.7 Bibliographic remarks
The deﬁnition of VC-dimension and its relation to learnability and to uniform
convergence is due to the seminal work of Vapnik & Chervonenkis (1971). The
relation to the deﬁnition of PAC learnability is due to Blumer, Ehrenfeucht,
Haussler & Warmuth (1989).
Several generalizations of the VC-dimension have been proposed. For exam-

ple, the fat-shattering dimension characterizes learnability of some regression
problems (Kearns, Schapire & Sellie 1994, Alon, Ben-David, Cesa-Bianchi &
Haussler 1997, Bartlett, Long & Williamson 1994, Anthony & Bartlet 1999), and
the Natarajan dimension characterizes learnability of some multiclass learning
problems (Natarajan 1989). However, in general, there is no equivalence between
learnability and uniform convergence. See (Shalev-Shwartz, Shamir, Srebro &
Sridharan 2010, Daniely, Sabato, Ben-David & Shalev-Shwartz 2011).

Sauer’s lemma has been proved by Sauer in response to a problem of Erdos
(Sauer 1972). Shelah (with Perles) proved it as a useful lemma for Shelah’s theory
of stable models (Shelah 1972). Gil Kalai tells1us that at some later time, Benjy
Weiss asked Perles about such a result in the context of ergodic theory, and
Perles, who forgot that he had proved it once, proved it again. Vapnik and
Chervonenkis proved the lemma in the context of statistical learning theory.
6.8 Exercises
1. Show the following monotonicity property of VC-dimension: For every two

hypothesis classes if H′⊆H then VCdim(H′)≤VCdim(H).
2. Given some ﬁnite domain set, X, and a number k≤|X| , ﬁgure out the VC-
dimension of each of the following classes (and prove your claims):
1.HX
=k={h∈{0,1}X:|{x:h(x) = 1}|=k}. That is, the set of all functions
that assign the value 1 to exactly kelements ofX.
1http://gilkalai.wordpress.com/2008/09/28/
extremal-combinatorics-iii-some-basic-theorems

6.8 Exercises 79
2.Hat−most−k={h∈{0,1}X:|{x:h(x) = 1}|≤kor|{x:h(x) = 0}|≤k}.
3. LetXbe the Boolean hypercube {0,1}n. For a setI⊆{1,2,...,n}we deﬁne
aparity function hIas follows. On a binary vector x= (x1,x2,...,xn)∈
{0,1}n,
hI(x) =(∑
i∈Ixi)
mod 2.
(That is,hIcomputes parity of bits in I.) What is the VC-dimension of the
class of all such parity functions, Hn-parity ={hI:I⊆{1,2,...,n}}?
4. We proved Sauer’s lemma by proving that for every class Hof ﬁnite VC-
dimensiond, and every subset Aof the domain,
|HA|≤|{B⊆A:HshattersB}|≤d∑
i=0(|A|
i)
.
Show that there are cases in which the previous two inequalities are strict
(namely, the≤can be replaced by <) and cases in which they can be replaced
by equalities. Demonstrate all four combinations of = and <.
5.VC-dimension of axis aligned rectangles in Rd:LetHd
recbe the class of
axis aligned rectangles in Rd. We have already seen that VCdim( H2
rec) = 4.
Prove that in general, VCdim( Hd
rec) = 2d.
6.VC-dimension of Boolean conjunctions: LetHd

i=0(|A|
i)
.
rec) = 4.
rec) = 2d.
conbe the class of Boolean
conjunctions over the variables x1,...,xd(d≥2). We already know that this
class is ﬁnite and thus (agnostic) PAC learnable. In this question we calculate
VCdim(Hd
con).
1. Show that|Hd
con|≤3d+ 1.
2. Conclude that VCdim( H)≤dlog 3.
3. Show thatHd
conshatters the set of unit vectors {ei:i≤d}.
4.(**) Show that VCdim( Hd
con)≤d.
Hint: Assume by contradiction that there exists a set C={c1,...,cd+1}
that is shattered by Hd
con. Leth1,...,hd+1be hypotheses in Hd
conthat
satisfy
∀i,j∈[d+ 1], hi(cj) ={
0i=j

VCdim(Hd
con).
con)≤d.
conthat
satisfy
0i=j
1 otherwise
For eachi∈[d+ 1],hi(or more accurately, the conjunction that corre-
sponds tohi) contains some literal ℓiwhich is false on ciand true on cj
for eachj̸=i. Use the Pigeonhole principle to show that there must be a
pairi < j≤d+ 1 such that ℓiandℓjuse the same xkand use that fact
to derive a contradiction to the requirements from the conjunctions hi,hj.
5. Consider the class Hd
mcon of monotone Boolean conjunctions over {0,1}d.
Monotonicity here means that the conjunctions do not contain negations.

80 The VC-Dimension
As inHd
con, the empty conjunction is interpreted as the all-positive hy-
pothesis. We augment Hd
mcon with the all-negative hypothesis h−. Show
that VCdim(Hd
mcon) =d.
7. We have shown that for a ﬁnite hypothesis class H, VCdim(H)≤⌊log(|H|)⌋.
However, this is just an upper bound. The VC-dimension of a class can be
much lower than that:
1. Find an example of a class Hof functions over the real interval X= [0,1]
such thatHis inﬁnite while VCdim( H) = 1.
2. Give an example of a ﬁnite hypothesis class Hover the domainX= [0,1],
where VCdim(H) =⌊log2(|H|)⌋.
8.(*)It is often the case that the VC-dimension of a hypothesis class equals (or
can be bounded above by) the number of parameters one needs to set in order
to deﬁne each hypothesis in the class. For instance, if His the class of axis
aligned rectangles in Rd, then VCdim(H) = 2d, which is equal to the number
of parameters used to deﬁne a rectangle in Rd. Here is an example that shows

that this is not always the case. We will see that a hypothesis class might
be very complex and even not learnable, although it has a small number of
Consider the domain X=R, and the hypothesis class
H={x↦→⌈sin(θx)⌉:θ∈R}
(here, we take⌈−1⌉= 0). Prove that VCdim( H) =∞.
Hint: There is more than one way to prove the required result. One option
is by applying the following lemma: If 0 .x1x2x3..., is the binary expansion of
x∈(0,1), then for any natural number m,⌈sin(2mπx)⌉= (1−xm), provided
that∃k≥ms.t.xk= 1.

9. LetHbe the class of signed intervals, that is,
H={ha,b,s:a≤b,s∈{− 1,1}}where
ha,b,s(x) ={
s ifx∈[a,b]
−sifx /∈[a,b]
Calculate VCdim( H).
10. LetHbe a class of functions from Xto{0,1}.
1. Prove that if VCdim( H)≥d, for anyd, then for some probability distri-
butionDoverX×{ 0,1}, for every sample size, m,
E
S∼Dm[LD(A(S))]≥min
h∈HLD(h) +d−m
2d
Hint: Use Exercise 3 in Chapter 5.
2. Prove that for every Hthat is PAC learnable, VCdim( H)<∞. (Note that
this is the implication 3 →6 in Theorem 6.7.)
11.VC of union: LetH1,...,Hrbe hypothesis classes over some ﬁxed domain

E
2d
setX. Letd= maxiVCdim(Hi) and assume for simplicity that d≥3.

6.8 Exercises 81
1. Prove that
VCdim (∪r
i=1Hi)≤4dlog(2d) + 2 log(r).
Hint: Take a set of kexamples and assume that they are shattered by
the union class. Therefore, the union class can produce all 2kpossible
labelings on these examples. Use Sauer’s lemma to show that the union
class cannot produce more than rkdlabelings. Therefore, 2k< rkd. Now
use Lemma A.2.
2.(*)Prove that for r= 2 it holds that
VCdim (H1∪H 2)≤2d+ 1.
12.Dudley classes: In this question we discuss an algebraic framework for
deﬁning concept classes over Rnand show a connection between the VC
dimension of such classes and their algebraic properties. Given a function
f:Rn→Rwe deﬁne the corresponding function, POS (f)(x) =1[f(x)>0]. For
a classFof real valued functions we deﬁne a corresponding class of functions
POS (F) ={POS (f) :f∈F} . We say that a family, F, of real valued func-
tions is linearly closed if for allf,g∈F andr∈R, (f+rg)∈F (where
addition and scalar multiplication of functions are deﬁned point wise, namely,

for allx∈Rn, (f+rg)(x) =f(x) +rg(x)). Note that if a family of functions
is linearly closed then we can view it as a vector space over the reals. For a
functiong:Rn→Rand a family of functions F, letF+gdef={f+g:f∈F} .
Hypothesis classes that have a representation as POS (F+g) for some vector
space of functions Fand some function gare called Dudley classes .
1. Show that for every g:Rn→Rand every vector space of functions Fas
deﬁned earlier, VCdim( POS (F+g)) = VCdim( POS (F)).

2.(**) For every linearly closed family of real valued functions F, the VC-
dimension of the corresponding class POS (F) equals the linear dimension
ofF(as a vector space). Hint: Letf1,...,fdbe a basis for the vector space
F. Consider the mapping x↦→(f1(x),...,fd(x)) (from RntoRd). Note
that this mapping induces a matching between functions over Rnof the
form POS (f) and homogeneous linear halfspaces in Rd(the VC-dimension
of the class of homogeneous linear halfspaces is analyzed in Chapter 9).

3. Show that each of the following classes can be represented as a Dudley
class:
1. The class HSnof halfspaces over Rn(see Chapter 9).
2. The class HHSnof all homogeneous halfspaces over Rn(see Chapter 9).
3. The class Bdof all functions deﬁned by (open) balls in Rd. Use the
Dudley representation to ﬁgure out the VC-dimension of this class.
4. LetPd
ndenote the class of functions deﬁned by polynomial inequalities
of degree≤d, namely,
Pd
n={hp:pis a polynomial of degree ≤din the variables x1,...,xn},

82 The VC-Dimension
where, for x= (x1....,xn),hp(x) =1[p(x)≥0](the degree of a multi-
variable polynomial is the maximal sum of variable exponents over all
of its terms. For example, the degree of p(x) = 3x3
1x2
2+ 4x3x2
7is 5).
1. Use the Dudley representation to ﬁgure out the VC-dimension of the
classPd
1– the class of all d-degree polynomials over R.
2. Prove that the class of all polynomial classiﬁers over Rhas inﬁnite
3. Use the Dudley representation to ﬁgure out the VC-dimension of the
classPd
n(as a function of dandn).

7 Nonuniform Learnability
The notions of PAC learnability discussed so far in the book allow the sample
sizes to depend on the accuracy and conﬁdence parameters, but they are uniform
with respect to the labeling rule and the underlying data distribution. Conse-
quently, classes that are learnable in that respect are limited (they must have
a ﬁnite VC-dimension, as stated by Theorem 6.7). In this chapter we consider
more relaxed, weaker notions of learnability. We discuss the usefulness of such
notions and provide characterization of the concept classes that are learnable
using these deﬁnitions.
We begin this discussion by deﬁning a notion of “nonuniform learnability” that
allows the sample size to depend on the hypothesis to which the learner is com-
pared. We then provide a characterization of nonuniform learnability and show
that nonuniform learnability is a strict relaxation of agnostic PAC learnability.
We also show that a suﬃcient condition for nonuniform learnability is that His

a countable union of hypothesis classes, each of which enjoys the uniform con-
vergence property. These results will be proved in Section 7.2 by introducing a
new learning paradigm, which is called Structural Risk Minimization (SRM). In
Section 7.3 we specify the SRM paradigm for countable hypothesis classes, which
yields the Minimum Description Length (MDL) paradigm. The MDL paradigm
gives a formal justiﬁcation to a philosophical principle of induction called Oc-

cam’s razor. Next, in Section 7.4 we introduce consistency as an even weaker
notion of learnability. Finally, we discuss the signiﬁcance and usefulness of the
diﬀerent notions of learnability.
7.1 Nonuniform Learnability
“Nonuniform learnability” allows the sample size to be nonuniform with respect
to the diﬀerent hypotheses with which the learner is competing. We say that a
hypothesis his (ϵ,δ)-competitive with another hypothesis h′if, with probability
higher than (1−δ),
LD(h)≤LD(h′) +ϵ.

In PAC learnability, this notion of “competitiveness” is not very useful, as we
are looking for a hypothesis with an absolute low risk (in the realizable case) or

84 Nonuniform Learnability
with a low risk compared to the minimal risk achieved by hypotheses in our class
(in the agnostic case). Therefore, the sample size depends only on the accuracy
and conﬁdence parameters. In nonuniform learnability, however, we allow the
sample size to be of the form mH(ϵ,δ,h ); namely, it depends also on the hwith
which we are competing. Formally,
definition 7.1 A hypothesis class Hisnonuniformly learnable if there exist a
learning algorithm, A, and a function mNUL
H: (0,1)2×H→ Nsuch that, for every
ϵ,δ∈(0,1) and for every h∈H, ifm≥mNUL
H(ϵ,δ,h ) then for every distribution
D, with probability of at least 1 −δover the choice of S∼Dm, it holds that
LD(A(S))≤LD(h) +ϵ.
At this point it might be useful to recall the deﬁnition of agnostic PAC learn-
ability (Deﬁnition 3.3):
A hypothesis class His agnostically PAC learnable if there exist a learning algo-
rithm,A, and a function mH: (0,1)2→Nsuch that, for every ϵ,δ∈(0,1)and

for every distribution D, ifm≥mH(ϵ,δ), then with probability of at least 1−δ
over the choice of S∼Dmit holds that
LD(A(S))≤min
Note that this implies that for every h∈H
In both types of learnability, we require that the output hypothesis will be
(ϵ,δ)-competitive with every other hypothesis in the class. But the diﬀerence
between these two notions of learnability is the question of whether the sample
sizemmay depend on the hypothesis hto which the error of A(S) is compared.

Note that that nonuniform learnability is a relaxation of agnostic PAC learn-
ability. That is, if a class is agnostic PAC learnable then it is also nonuniformly
7.1.1 Characterizing Nonuniform Learnability
Our goal now is to characterize nonuniform learnability. In the previous chapter
we have found a crisp characterization of PAC learnable classes, by showing
that a class of binary classiﬁers is agnostic PAC learnable if and only if its VC-
dimension is ﬁnite. In the following theorem we ﬁnd a diﬀerent characterization

for nonuniform learnable classes for the task of binary classiﬁcation.
theorem 7.2 A hypothesis class Hof binary classiﬁers is nonuniformly learn-
able if and only if it is a countable union of agnostic PAC learnable hypothesis
classes.
The proof of Theorem 7.2 relies on the following result of independent interest:

theorem 7.3 LetHbe a hypothesis class that can be written as a countable
union of hypothesis classes, H=⋃
n∈NHn, where eachHnenjoys the uniform
convergence property. Then, His nonuniformly learnable.
Recall that in Chapter 4 we have shown that uniform convergence is suﬃcient
for agnostic PAC learnability. Theorem 7.3 generalizes this result to nonuni-
form learnability. The proof of this theorem will be given in the next section by
introducing a new learning paradigm. We now turn to proving Theorem 7.2.
Proof of Theorem 7.2 First assume that H=⋃
n∈NHnwhere eachHnis ag-
nostic PAC learnable. Using the fundamental theorem of statistical learning, it
follows that each Hnhas the uniform convergence property. Therefore, using
Theorem 7.3 we obtain that His nonuniform learnable.
For the other direction, assume that His nonuniform learnable using some
algorithmA. For every n∈N, letHn={h∈H :mNUL
H(1/8,1/7,h)≤n}.
Clearly,H=∪n∈NHn. In addition, using the deﬁnition of mNUL

Hwe know that
for any distribution Dthat satisﬁes the realizability assumption with respect to
Hn, with probability of at least 6 /7 overS∼Dnwe have that LD(A(S))≤1/8.
Using the fundamental theorem of statistical learning, this implies that the VC-
dimension ofHnmust be ﬁnite, and therefore Hnis agnostic PAC learnable.
The following example shows that nonuniform learnability is a strict relax-
ation of agnostic PAC learnability; namely, there are hypothesis classes that are
nonuniform learnable but are not agnostic PAC learnable.

Example 7.1 Consider a binary classiﬁcation problem with the instance domain
beingX=R. For every n∈NletHnbe the class of polynomial classiﬁers of
degreen; namely,Hnis the set of all classiﬁers of the form h(x) = sign(p(x))
wherep:R→Ris a polynomial of degree n. LetH=⋃
n∈NHn. Therefore,His
the class of all polynomial classiﬁers over R. It is easy to verify that VCdim( H) =
∞while VCdim(Hn) =n+ 1 (see Exercise 12). Hence, His not PAC learnable,
while on the basis of Theorem 7.3, His nonuniformly learnable.
7.2 Structural Risk Minimization

So far, we have encoded our prior knowledge by specifying a hypothesis class
H, which we believe includes a good predictor for the learning task at hand.
Yet another way to express our prior knowledge is by specifying preferences over
hypotheses within H. In the Structural Risk Minimization (SRM) paradigm,
we do so by ﬁrst assuming that Hcan be written as H=⋃
n∈NHnand then
specifying a weight function, w:N→[0,1], which assigns a weight to each
hypothesis class, Hn, such that a higher weight reﬂects a stronger preference

for the hypothesis class. In this section we discuss how to learn with such prior
knowledge. In the next section we describe a couple of important weighting
schemes, including Minimum Description Length.

86 Nonuniform Learnability
Concretely, letHbe a hypothesis class that can be written as H=⋃
n∈NHn.
For example,Hmay be the class of all polynomial classiﬁers where each Hnis
the class of polynomial classiﬁers of degree n(see Example 7.1). Assume that for
eachn, the classHnenjoys the uniform convergence property (see Deﬁnition 4.3
in Chapter 4) with a sample complexity function mUC
Hn(ϵ,δ). Let us also deﬁne
the function ϵn:N×(0,1)→(0,1) by
ϵn(m,δ) = min{ϵ∈(0,1) :mUC
Hn(ϵ,δ)≤m}. (7.1)
In words, we have a ﬁxed sample size m, and we are interested in the lowest
possible upper bound on the gap between empirical and true risks achievable by
using a sample of mexamples.
From the deﬁnitions of uniform convergence and ϵn, it follows that for every
mandδ, with probability of at least 1 −δover the choice of S∼Dmwe have
that
∀h∈Hn,|LD(h)−LS(h)|≤ϵn(m,δ). (7.2)
Letw:N→[0,1] be a function such that∑∞
n=1w(n)≤1. We refer to was
aweight function over the hypothesis classes H1,H2,.... Such a weight function

that
can reﬂect the importance that the learner attributes to each hypothesis class,
or some measure of the complexity of diﬀerent hypothesis classes. If His a ﬁnite
union ofNhypothesis classes, one can simply assign the same weight of 1 /Nto
all hypothesis classes. This equal weighting corresponds to no a priori preference
to any hypothesis class. Of course, if one believes (as prior knowledge) that a
certain hypothesis class is more likely to contain the correct target function,
then it should be assigned a larger weight, reﬂecting this prior knowledge. When

His a (countable) inﬁnite union of hypothesis classes, a uniform weighting is
not possible but many other weighting schemes may work. For example, one can
choosew(n) =6
π2n2orw(n) = 2−n. Later in this chapter we will provide another
convenient way to deﬁne weighting functions using description languages.
The SRM rule follows a “bound minimization” approach. This means that
the goal of the paradigm is to ﬁnd a hypothesis that minimizes a certain upper
bound on the true risk. The bound that the SRM rule wishes to minimize is

given in the following theorem.
theorem 7.4 Letw:N→[0,1]be a function such that∑∞
n=1w(n)≤1. Let
Hbe a hypothesis class that can be written as H=⋃
n∈NHn, where for each n,
Hnsatisﬁes the uniform convergence property with a sample complexity function
mUC
Hn. Letϵnbe as deﬁned in Equation (7.1). Then, for every δ∈(0,1)and
distributionD, with probability of at least 1−δover the choice of S∼Dm, the
following bound holds (simultaneously) for every n∈Nandh∈Hn.
|LD(h)−LS(h)|≤ϵn(m,w(n)·δ).
Therefore, for every δ∈(0,1)and distributionD, with probability of at least

7.2 Structural Risk Minimization 87
1−δit holds that
∀h∈H, LD(h)≤LS(h) + min
n:h∈Hnϵn(m,w(n)·δ). (7.3)
Proof For eachndeﬁneδn=w(n)δ. Applying the assumption that uniform
convergence holds for all nwith the rate given in Equation (7.2), we obtain that
if we ﬁxnin advance, then with probability of at least 1 −δnover the choice of
S∼Dm,
∀h∈Hn,|LD(h)−LS(h)|≤ϵn(m,δn).
Applying the union bound over n= 1,2,..., we obtain that with probability of
at least 1−∑
nδn= 1−δ∑
nw(n)≥1−δ, the preceding holds for all n, which
concludes our proof.
Denote
n(h) = min{n:h∈Hn}, (7.4)
and then Equation (7.3) implies that
LD(h)≤LS(h) +ϵn(h)(m,w(n(h))·δ).
The SRM paradigm searches for hthat minimizes this bound, as formalized
in the following pseudocode:
Structural Risk Minimization (SRM)
prior knowledge:
H=⋃
nHnwhereHnhas uniform convergence with mUC
Hn
w:N→[0,1] where∑
nw(n)≤1
deﬁne:ϵnas in Equation (7.1) ; n(h) as in Equation (7.4)
input: training set S∼Dm, conﬁdence δ
output:h∈argminh∈H[
LS(h) +ϵn(h)(m,w(n(h))·δ)]

Denote
H=⋃
Hn
nw(n)≤1
Unlike the ERM paradigm discussed in previous chapters, we no longer just care
about the empirical risk, LS(h), but we are willing to trade some of our bias
toward low empirical risk with a bias toward classes for which ϵn(h)(m,w(n(h))·δ)
is smaller, for the sake of a smaller estimation error.
Next we show that the SRM paradigm can be used for nonuniform learning
of every class, which is a countable union of uniformly converging hypothesis
classes.
theorem 7.5 LetHbe a hypothesis class such that H=⋃
n∈NHn, where

classes.
eachHnhas the uniform convergence property with sample complexity mUC
Hn. Let
w:N→[0,1]be such that w(n) =6
n2π2. Then,His nonuniformly learnable
using the SRM rule with rate
mNUL
H(ϵ,δ,h )≤mUC
Hn(h)(
ϵ/2,6δ
(πn(h))2)
.

88 Nonuniform Learnability
Proof LetAbe the SRM algorithm with respect to the weighting function w.
For everyh∈H,ϵ, andδ, letm≥mUC
Hn(h)(ϵ,w(n(h))δ). Using the fact that∑
nw(n) = 1, we can apply Theorem 7.4 to get that, with probability of at least
1−δover the choice of S∼Dm, we have that for every h′∈H,
LD(h′)≤LS(h′) +ϵn(h′)(m,w(n(h′))δ).
The preceding holds in particular for the hypothesis A(S) returned by the SRM
rule. By the deﬁnition of SRM we obtain that
h′[
LS(h′) +ϵn(h′)(m,w(n(h′))δ)]
≤LS(h) +ϵn(h)(m,w(n(h))δ).
Finally, ifm≥mUC
Hn(h)(ϵ/2,w(n(h))δ) then clearly ϵn(h)(m,w(n(h))δ)≤ϵ/2. In
addition, from the uniform convergence property of each Hnwe have that with
probability of more than 1 −δ,
LS(h)≤LD(h) +ϵ/2.
Combining all the preceding we obtain that LD(A(S))≤LD(h) +ϵ, which con-
cludes our proof.
Note that the previous theorem also proves Theorem 7.3.
Remark 7.2 (No-Free-Lunch for Nonuniform Learnability) We have shown that

h′[
any countable union of classes of ﬁnite VC-dimension is nonuniformly learnable.
It turns out that, for any inﬁnite domain set, X, the class of all binary valued
functions overXis not a countable union of classes of ﬁnite VC-dimension. We
leave the proof of this claim as a (nontrivial) exercise (see Exercise 5). It follows
that, in some sense, the no free lunch theorem holds for nonuniform learning
as well: namely, whenever the domain is not ﬁnite, there exists no nonuniform

learner with respect to the class of all deterministic binary classiﬁers (although
for each such classiﬁer there exists a trivial algorithm that learns it – ERM with
respect to the hypothesis class that contains only this classiﬁer).
It is interesting to compare the nonuniform learnability result given in The-
orem 7.5 to the task of agnostic PAC learning any speciﬁc Hnseparately. The
prior knowledge, or bias, of a nonuniform learner for His weaker – it is searching

for a model throughout the entire class H, rather than being focused on one spe-
ciﬁcHn. The cost of this weakening of prior knowledge is the increase in sample
complexity needed to compete with any speciﬁc h∈Hn. For a concrete evalua-
tion of this gap, consider the task of binary classiﬁcation with the zero-one loss.
Assume that for all n, VCdim(Hn) =n. SincemUC
Hn(ϵ,δ) =Cn+log(1/δ)
ϵ2 (where
Cis the contant appearing in Theorem 6.8), a straightforward calculation shows
that
mNUL
H(ϵ,δ,h )−mUC
Hn(ϵ/2,δ)≤4C2 log(2n)
ϵ2.

ϵ2 (where
that
mNUL
ϵ2.
That is, the cost of relaxing the learner’s prior knowledge from a speciﬁc Hn
that contains the target hto a countable union of classes depends on the log of

the index of the ﬁrst class in which hresides. That cost increases with the index
of the class, which can be interpreted as reﬂecting the value of knowing a good
priority order on the hypotheses in H.
7.3 Minimum Description Length and Occam’s Razor
LetHbe a countable hypothesis class. Then, we can write Has a countable
union of singleton classes, namely, H=⋃
n∈N{hn}. By Hoeﬀding’s inequality
(Lemma 4.5), each singleton class has the uniform convergence property with
ratemUC(ϵ,δ) =log(2/δ)
2ϵ2. Therefore, the function ϵngiven in Equation (7.1)
becomesϵn(m,δ) =√
log(2/δ)
2mand the SRM rule becomes
argmin
hn∈H[
LS(h) +√
−log(w(n)) + log(2/δ)
2m]
.
Equivalently, we can think of was a function from Hto [0,1], and then the SRM
rule becomes
argmin
h∈H[
LS(h) +√
−log(w(h)) + log(2/δ)
2m]
.
It follows that in this case, the prior knowledge is solely determined by the weight
we assign to each hypothesis. We assign higher weights to hypotheses that we

log(2/δ)
argmin
hn∈H[
LS(h) +√
2m]
.
argmin
h∈H[
LS(h) +√
2m]
.
believe are more likely to be the correct one, and in the learning algorithm we
prefer hypotheses that have higher weights.
In this section we discuss a particular convenient way to deﬁne a weight func-
tion overH, which is derived from the length of descriptions given to hypotheses.
Having a hypothesis class, one can wonder about how we describe, or represent,
each hypothesis in the class. We naturally ﬁx some description language. This
can be English, or a programming language, or some set of mathematical formu-

las. In any of these languages, a description consists of ﬁnite strings of symbols
(or characters) drawn from some ﬁxed alphabet. We shall now formalize these
notions.
LetHbe the hypothesis class we wish to describe. Fix some ﬁnite set Σ
of symbols (or “characters”), which we call the alphabet. For concreteness, we
let Σ ={0,1}. A string is a ﬁnite sequence of symbols from Σ; for example,
σ= (0,1,1,1,0) is a string of length 5. We denote by |σ|the length of a string.
The set of all ﬁnite length strings is denoted Σ∗. A description language for H

notions.
is a function d:H→ Σ∗, mapping each member hofHto a stringd(h).d(h) is
called “the description of h,” and its length is denoted by |h|.
We shall require that description languages be preﬁx-free ; namely, for every
distincth,h′,d(h) is not a preﬁx of d(h′). That is, we do not allow that any
stringd(h) is exactly the ﬁrst |h|symbols of any longer string d(h′). Preﬁx-free
collections of strings enjoy the following combinatorial property:

90 Nonuniform Learnability
lemma 7.6 (Kraft Inequality) IfS⊆{ 0,1}∗is a preﬁx-free set of strings, then
∑
σ∈S1
2|σ|≤1.
Proof Deﬁne a probability distribution over the members of Sas follows: Re-
peatedly toss an unbiased coin, with faces labeled 0 and 1, until the sequence
of outcomes is a member of S; at that point, stop. For each σ∈S, letP(σ)
be the probability that this process generates the string σ. Note that since Sis
preﬁx-free, for every σ∈S, if the coin toss outcomes follow the bits of σthen
we will stop only once the sequence of outcomes equals σ. We therefore get that,
for everyσ∈S,P(σ) =1
2|σ|. Since probabilities add up to at most 1, our proof
is concluded.
In light of Kraft’s inequality, any preﬁx-free description language of a hypoth-
esis class,H, gives rise to a weighting function wover that hypothesis class – we
will simply set w(h) =1
2|h|. This observation immediately yields the following:
theorem 7.7 LetHbe a hypothesis class and let d:H→{ 0,1}∗be a preﬁx-

free description language for H. Then, for every sample size, m, every conﬁdence
parameter, δ >0, and every probability distribution, D, with probability greater
than 1−δover the choice of S∼Dmwe have that,
∀h∈H, LD(h)≤LS(h) +√
|h|+ ln(2/δ)
2m,
where|h|is the length of d(h).
Proof Choosew(h) = 1/2|h|, apply Theorem 7.4 with ϵn(m,δ) =√
ln(2/δ)
2m, and
note that ln(2|h|) =|h|ln(2)<|h|.
As was the case with Theorem 7.4, this result suggests a learning paradigm
forH– given a training set, S, search for a hypothesis h∈H that minimizes

2m,
ln(2/δ)
2m, and
the bound, LS(h) +√
|h|+ln(2/δ)
2m. In particular, it suggests trading oﬀ empirical
risk for saving description length. This yields the Minimum Description Length
learning paradigm.
Minimum Description Length (MDL)
His a countable hypothesis class
His described by a preﬁx-free language over {0,1}
For everyh∈H,|h|is the length of the representation of h
input: A training set S∼Dm, conﬁdence δ
LS(h) +√
2m]
Example 7.3 LetHbe the class of all predictors that can be implemented using

LS(h) +√
2m]
some programming language, say, C++. Let us represent each program using the

7.3 Minimum Description Length and Occam’s Razor 91
binary string obtained by running the gzip command on the program (this yields
a preﬁx-free description language over the alphabet {0,1}). Then,|h|is simply
the length (in bits) of the output of gzip when running on the C++ program
corresponding to h.
7.3.1 Occam’s Razor
Theorem 7.7 suggests that, having two hypotheses sharing the same empirical
risk, the true risk of the one that has shorter description can be bounded by a
lower value. Thus, this result can be viewed as conveying a philosophical message:
A short explanation (that is, a hypothesis that has a short length) tends to be more valid
than a long explanation.
This is a well known principle, called Occam’s razor, after William of Ockham,
a 14th-century English logician, who is believed to have been the ﬁrst to phrase
it explicitly. Here, we provide one possible justiﬁcation to this principle. The
inequality of Theorem 7.7 shows that the more complex a hypothesis his (in the

sense of having a longer description), the larger the sample size it has to ﬁt to
guarantee that it has a small true risk, LD(h).
At a second glance, our Occam razor claim might seem somewhat problematic.
In the context in which the Occam razor principle is usually invoked in science,
the language according to which complexity is measured is a natural language,
whereas here we may consider any arbitrary abstract description language. As-
sume that we have two hypotheses such that |h′|is much smaller than |h|. By

the preceding result, if both have the same error on a given training set, S, then
the true error of hmay be much higher than the true error of h′, so one should
preferh′overh. However, we could have chosen a diﬀerent description language,
say, one that assigns a string of length 3 to hand a string of length 100000 to h′.
Suddenly it looks as if one should prefer hoverh′. But these are the same hand
h′for which we argued two sentences ago that h′should be preferable. Where is
the catch here?
Indeed, there is no inherent generalizability diﬀerence between hypotheses.

The crucial aspect here is the dependency order between the initial choice of
language (or, preference over hypotheses) and the training set. As we know from
the basic Hoeﬀding’s bound (Equation (4.2)), if we commit to any hypothesis be-
foreseeing the data, then we are guaranteed a rather small estimation error term
LD(h)≤LS(h) +√
ln(2/δ)
2m. Choosing a description language (or, equivalently,
some weighting of hypotheses) is a weak form of committing to a hypothesis.

ln(2/δ)
Rather than committing to a single hypothesis, we spread out our commitment
among many. As long as it is done independently of the training sample, our gen-
eralization bound holds. Just as the choice of a single hypothesis to be evaluated
by a sample can be arbitrary, so is the choice of description language.

92 Nonuniform Learnability
7.4 Other Notions of Learnability – Consistency
The notion of learnability can be further relaxed by allowing the needed sample
sizes to depend not only on ϵ, δ, andhbut also on the underlying data-generating
probability distribution D(that is used to generate the training sample and to
determine the risk). This type of performance guarantee is captured by the notion
ofconsistency1of a learning rule.
definition 7.8 (Consistency) Let Zbe a domain set, let Pbe a set of
probability distributions over Z, and letHbe a hypothesis class. A learn-
ing ruleAisconsistent with respect toHandPif there exists a function
mCON
H: (0,1)2×H×P→ Nsuch that, for every ϵ,δ∈(0,1), everyh∈H, and
everyD∈P , ifm≥mNUL
H(ϵ,δ,h,D) then with probability of at least 1 −δover
the choice of S∼Dmit holds that
IfPis the set of all distributions,2we say that Aisuniversally consistent with
respect toH.
The notion of consistency is, of course, a relaxation of our previous notion

mCON
of nonuniform learnability. Clearly if an algorithm nonuniformly learns a class
Hit is also universally consistent for that class. The relaxation is strict in the
sense that there are consistent learning rules that are not successful nonuniform
learners. For example, the algorithm Memorize deﬁned in Example 7.4 later is
universally consistent for the class of all binary classiﬁers over N. However, as
we have argued before, this class is not nonuniformly learnable.

Example 7.4 Consider the classiﬁcation prediction algorithm Memorize deﬁned
as follows. The algorithm memorizes the training examples, and, given a test
pointx, it predicts the majority label among all labeled instances of xthat exist
in the training sample (and some ﬁxed default label if no instance of xappears
in the training set). It is possible to show (see Exercise 6) that the Memorize
algorithm is universally consistent for every countable domain Xand a ﬁnite
label setY(w.r.t. the zero-one loss).

Intuitively, it is not obvious that the Memorize algorithm should be viewed as a
learner , since it lacks the aspect of generalization, namely, of using observed data
to predict the labels of unseen examples. The fact that Memorize is a consistent
algorithm for the class of all functions over any countable domain set therefore
raises doubt about the usefulness of consistency guarantees. Furthermore, the
sharp-eyed reader may notice that the “bad learner” we introduced in Chapter 2,

1In the literature, consistency is often deﬁned using the notion of either convergence in
probability (corresponding to weak consistency) or almost sure convergence (corresponding
to strong consistency).
2Formally, we assume that Zis endowed with some sigma algebra of subsets Ω, and by “all
distributions” we mean all probability distributions that have Ω contained in their
associated family of measurable subsets.

which led to overﬁtting, is in fact the Memorize algorithm. In the next section
we discuss the signiﬁcance of the diﬀerent notions of learnability and revisit the
No-Free-Lunch theorem in light of the diﬀerent deﬁnitions of learnability.
7.5 Discussing the Diﬀerent Notions of Learnability
We have given three deﬁnitions of learnability and we now discuss their useful-
ness. As is usually the case, the usefulness of a mathematical deﬁnition depends
on what we need it for. We therefore list several possible goals that we aim to
achieve by deﬁning learnability and discuss the usefulness of the diﬀerent deﬁni-
tions in light of these goals.
What Is the Risk of the Learned Hypothesis?
The ﬁrst possible goal of deriving performance guarantees on a learning algo-
rithm is bounding the risk of the output predictor. Here, both PAC learning
and nonuniform learning give us an upper bound on the true risk of the learned

hypothesis based on its empirical risk. Consistency guarantees do not provide
such a bound. However, it is always possible to estimate the risk of the output
predictor using a validation set (as will be described in Chapter 11).
How Many Examples Are Required to Be as Good as the Best Hypothesis
inH?
When approaching a learning problem, a natural question is how many exam-
ples we need to collect in order to learn it. Here, PAC learning gives a crisp
answer. However, for both nonuniform learning and consistency, we do not know

inH?
in advance how many examples are required to learn H. In nonuniform learning
this number depends on the best hypothesis in H, and in consistency it also
depends on the underlying distribution. In this sense, PAC learning is the only
useful deﬁnition of learnability. On the ﬂip side, one should keep in mind that
even if the estimation error of the predictor we learn is small, its risk may still
be large ifHhas a large approximation error. So, for the question “How many
examples are required to be as good as the Bayes optimal predictor?” even PAC

guarantees do not provide us with a crisp answer. This reﬂects the fact that the
usefulness of PAC learning relies on the quality of our prior knowledge.
PAC guarantees also help us to understand what we should do next if our
learning algorithm returns a hypothesis with a large risk, since we can bound
the part of the error that stems from estimation error and therefore know how
much of the error is attributed to approximation error. If the approximation error
is large, we know that we should use a diﬀerent hypothesis class. Similarly, if a

nonuniform algorithm fails, we can consider a diﬀerent weighting function over
(subsets of) hypotheses. However, when a consistent algorithm fails, we have
no idea whether this is because of the estimation error or the approximation
error. Furthermore, even if we are sure we have a problem with the estimation

94 Nonuniform Learnability
error term, we do not know how many more examples are needed to make the
estimation error small.
How to Learn? How to Express Prior Knowledge?
Maybe the most useful aspect of the theory of learning is in providing an answer
to the question of “how to learn.” The deﬁnition of PAC learning yields the
limitation of learning (via the No-Free-Lunch theorem) and the necessity of prior
knowledge. It gives us a crisp way to encode prior knowledge by choosing a
hypothesis class, and once this choice is made, we have a generic learning rule –
ERM. The deﬁnition of nonuniform learnability also yields a crisp way to encode
prior knowledge by specifying weights over (subsets of) hypotheses of H. Once
this choice is made, we again have a generic learning rule – SRM. The SRM rule
is also advantageous in model selection tasks, where prior knowledge is partial.
We elaborate on model selection in Chapter 11 and here we give a brief example.

Consider the problem of ﬁtting a one dimensional polynomial to data; namely,
our goal is to learn a function, h:R→R, and as prior knowledge we consider
the hypothesis class of polynomials. However, we might be uncertain regarding
which degree dwould give the best results for our data set: A small degree might
not ﬁt the data well (i.e., it will have a large approximation error), whereas a
high degree might lead to overﬁtting (i.e., it will have a large estimation error).

In the following we depict the result of ﬁtting a polynomial of degrees 2, 3, and
10 to the same training set.
degree 2 degree 3 degree 10
It is easy to see that the empirical risk decreases as we enlarge the degree.
Therefore, if we choose Hto be the class of all polynomials up to degree 10 then
the ERM rule with respect to this class would output a 10 degree polynomial
and would overﬁt. On the other hand, if we choose too small a hypothesis class,
say, polynomials up to degree 2, then the ERM would suﬀer from underﬁtting

(i.e., a large approximation error). In contrast, we can use the SRM rule on the
set of all polynomials, while ordering subsets of Haccording to their degree, and
this will yield a 3rd degree polynomial since the combination of its empirical
risk and the bound on its estimation error is the smallest. In other words, the
SRM rule enables us to select the right model on the basis of the data itself. The
price we pay for this ﬂexibility (besides a slight increase of the estimation error
relative to PAC learning w.r.t. the optimal degree) is that we do not know in

7.5 Discussing the Diﬀerent Notions of Learnability 95
advance how many examples are needed to compete with the best hypothesis in
H.
Unlike the notions of PAC learnability and nonuniform learnability, the deﬁni-
tion of consistency does not yield a natural learning paradigm or a way to encode
prior knowledge. In fact, in many cases there is no need for prior knowledge at
all. For example, we saw that even the Memorize algorithm, which intuitively
should not be called a learning algorithm, is a consistent algorithm for any class
deﬁned over a countable domain and a ﬁnite label set. This hints that consistency
is a very weak requirement.
Which Learning Algorithm Should We Prefer?
One may argue that even though consistency is a weak requirement, it is desirable
that a learning algorithm will be consistent with respect to the set of all functions
fromXtoY, which gives us a guarantee that for enough training examples, we
will always be as good as the Bayes optimal predictor. Therefore, if we have

two algorithms, where one is consistent and the other one is not consistent, we
should prefer the consistent algorithm. However, this argument is problematic for
two reasons. First, maybe it is the case that for most “natural” distributions we
will observe in practice that the sample complexity of the consistent algorithm
will be so large so that in every practical situation we will not obtain enough
examples to enjoy this guarantee. Second, it is not very hard to make any PAC

or nonuniform learner consistent with respect to the class of all functions from
XtoY. Concretely, consider a countable domain, X, a ﬁnite label set Y, and
a hypothesis class, H, of functions from XtoY. We can make any nonuniform
learner forHbe consistent with respect to the class of allclassiﬁers fromXtoY
using the following simple trick: Upon receiving a training set, we will ﬁrst run
the nonuniform learner over the training set, and then we will obtain a bound

on the true risk of the learned predictor. If this bound is small enough we are
done. Otherwise, we revert to the Memorize algorithm. This simple modiﬁcation
makes the algorithm consistent with respect to all functions from XtoY. Since
it is easy to make any algorithm consistent, it may not be wise to prefer one
algorithm over the other just because of consistency considerations.
7.5.1 The No-Free-Lunch Theorem Revisited
Recall that the No-Free-Lunch theorem (Theorem 5.1 from Chapter 5) implies

that no algorithm can learn the class of all classiﬁers over an inﬁnite domain.
In contrast, in this chapter we saw that the Memorize algorithm is consistent
with respect to the class of all classiﬁers over a countable inﬁnite domain. To
understand why these two statements do not contradict each other, let us ﬁrst
recall the formal statement of the No-Free-Lunch theorem.
LetXbe a countable inﬁnite domain and let Y={±1}. The No-Free-Lunch
theorem implies the following: For any algorithm, A, and a training set size, m,

there exist a distribution over Xand a function h⋆:X →Y , such that if A

96 Nonuniform Learnability
will get a sample of mi.i.d. training examples, labeled by h⋆, thenAis likely to
return a classiﬁer with a larger error.
The consistency of Memorize implies the following: For every distribution over
Xand a labeling function h⋆:X→Y , there exists a training set size m(that
depends on the distribution and on h⋆) such that if Memorize receives at least
mexamples it is likely to return a classiﬁer with a small error.
We see that in the No-Free-Lunch theorem, we ﬁrst ﬁx the training set size,
and then ﬁnd a distribution and a labeling function that are bad for this training
set size. In contrast, in consistency guarantees, we ﬁrst ﬁx the distribution and
the labeling function, and only then do we ﬁnd a training set size that suﬃces
for learning this particular distribution and labeling function.
7.6 Summary
We introduced nonuniform learnability as a relaxation of PAC learnability and
consistency as a relaxation of nonuniform learnability. This means that even

classes of inﬁnite VC-dimension can be learnable, in some weaker sense of learn-
ability. We discussed the usefulness of the diﬀerent deﬁnitions of learnability.
For hypothesis classes that are countable, we can apply the Minimum Descrip-
tion Length scheme, where hypotheses with shorter descriptions are preferred,
following the principle of Occam’s razor. An interesting example is the hypothe-
sis class of all predictors we can implement in C++ (or any other programming
language), which we can learn (nonuniformly) using the MDL scheme.

Arguably, the class of all predictors we can implement in C++ is a powerful
class of functions and probably contains all that we can hope to learn in prac-
tice. The ability to learn this class is impressive, and, seemingly, this chapter
should have been the last chapter of this book. This is not the case, because of
the computational aspect of learning: that is, the runtime needed to apply the
learning rule. For example, to implement the MDL paradigm with respect to
all C++ programs, we need to perform an exhaustive search over all C++ pro-

grams, which will take forever. Even the implementation of the ERM paradigm
with respect to all C++ programs of description length at most 1000 bits re-
quires an exhaustive search over 21000hypotheses. While the sample complexity
of learning this class is just1000+log(2/δ)
ϵ2 , the runtime is≥21000. This is a huge
number – much larger than the number of atoms in the visible universe. In the
next chapter we formally deﬁne the computational complexity of learning. In the

second part of this book we will study hypothesis classes for which the ERM or
SRM schemes can be implemented eﬃciently.

7.7 Bibliographic Remarks
Our deﬁnition of nonuniform learnability is related to the deﬁnition of an Occam-
algorithm in Blumer, Ehrenfeucht, Haussler & Warmuth (1987). The concept of
SRM is due to (Vapnik & Chervonenkis 1974, Vapnik 1995). The concept of MDL
is due to (Rissanen 1978, Rissanen 1983). The relation between SRM and MDL
is discussed in Vapnik (1995). These notions are also closely related to the notion
ofregularization (e.g. Tikhonov (1943)). We will elaborate on regularization in
the second part of this book.
The notion of consistency of estimators dates back to Fisher (1922). Our pre-
sentation of consistency follows Steinwart & Christmann (2008), who also derived
several no-free-lunch theorems.
7.8 Exercises
1. Prove that for any ﬁnite class H, and any description language d:H →
{0,1}∗, the VC-dimension of His at most 2 sup{|d(h)|:h∈H} – the maxi-
mum description length of a predictor in H. Furthermore, if dis a preﬁx-free

description then VCdim( H)≤sup{|d(h)|:h∈H} .
2. LetH={hn:n∈N}be an inﬁnite countable hypothesis class for binary
classiﬁcation. Show that it is impossible to assign weights to the hypotheses
inHsuch that
• H could be learnt nonuniformly using these weights. That is, the weighting
functionw:H→ [0,1] should satisfy the condition∑
h∈Hw(h)≤1.
•The weights would be monotonically nondecreasing. That is, if i<j , then
w(hi)≤w(hj).
3.•Consider a hypothesis class H=⋃∞
n=1Hn, where for every n∈N,Hnis
ﬁnite. Find a weighting function w:H→ [0,1] such that∑
h∈Hw(h)≤

h∈Hw(h)≤
1 and so that for all h∈H,w(h) is determined by n(h) = min{n:h∈
Hn}and by|Hn(h)|.
•(*)Deﬁne such a function wwhen for all nHnis countable (possibly
inﬁnite).
4. LetHbe some hypothesis class. For any h∈H, let|h|denote the description
length ofh, according to some ﬁxed description language. Consider the MDL
learning paradigm in which the algorithm returns:
hS∈arg min
h∈H[
LS(h) +√
2m]
,
whereSis a sample of size m. For anyB > 0, letHB={h∈H :|h|≤B},
and deﬁne
h∗
B= arg min
h∈HBLD(h).

98 Nonuniform Learnability
Prove a bound on LD(hS)−LD(h∗
B) in terms of B, the conﬁdence parameter
δ, and the size of the training set m.
•Note: Such bounds are known as oracle inequalities in the literature: We
wish to estimate how good we are compared to a reference classiﬁer (or
“oracle”)h∗
B.
5. In this question we wish to show a No-Free-Lunch result for nonuniform learn-
ability: namely, that, over any inﬁnite domain, the class of allfunctions is not
learnable even under the relaxed nonuniform variation of learning.
Recall that an algorithm, A,nonuniformly learns a hypothesis class Hif
there exists a function mNUL
H: (0,1)2×H→ Nsuch that, for every ϵ,δ∈(0,1)
and for every h∈H, ifm≥mNUL
H(ϵ,δ,h ) then for every distribution D, with
probability of at least 1 −δover the choice of S∼Dm, it holds that
If such an algorithm exists then we say that Hisnonuniformly learnable .
1. LetAbe a nonuniform learner for a class H. For eachn∈NdeﬁneHA
n=

n=
{h∈H:mNUL(0.1,0.1,h)≤n}. Prove that each such class Hnhas a ﬁnite
2. Prove that if a class His nonuniformly learnable then there are classes Hn
so thatH=⋃
n∈NHnand, for every n∈N, VCdim(Hn) is ﬁnite.
3. LetHbe a class that shatters an inﬁnite set. Then, for every sequence
of classes (Hn:n∈N) such thatH=⋃
n∈NHn, there exists some nfor
which VCdim(Hn) =∞.
Hint: Given a class Hthat shatters some inﬁnite set K, and a sequence of
classes (Hn:n∈N), each having a ﬁnite VC-dimension, start by deﬁning
subsetsKn⊆Ksuch that, for all n,|Kn|>VCdim(Hn)and for any

so thatH=⋃
n̸=m,Kn∩Km=∅. Now, pick for each such Kna functionfn:Kn→
{0,1}so that noh∈Hnagrees with fnon the domain Kn. Finally, deﬁne
f:X→{0,1}by combining these fn’s and prove that f∈(
H\⋃
n∈NHn)
.
4. Construct a class H1of functions from the unit interval [0 ,1] to{0,1}that
is nonuniformly learnable but not PAC learnable.
5. Construct a class H2of functions from the unit interval [0 ,1] to{0,1}that
is not nonuniformly learnable.
6. In this question we wish to show that the algorithm Memorize is a consistent

H\⋃
n∈NHn)
.
learner for every class of (binary-valued) functions over any countable domain.
LetXbe a countable domain and let Dbe a probability distribution over X.
1. Let{xi:i∈N}be an enumeration of the elements of Xso that for all
i≤j,D({xi})≤D({xj}). Prove that
lim
n→∞∑
i≥nD({xi}) = 0.
2. Given any ϵ>0 prove that there exists ϵD>0 such that
D({x∈X:D({x})<ϵD})<ϵ.

7.8 Exercises 99
3. Prove that for every η>0, ifnis such thatD({xi})<ηfor alli>n , then
for everym∈N,
P
S∼Dm[∃xi: (D({xi})>ηandxi/∈S)]≤ne−ηm.
4. Conclude that if Xis countable then for every probability distribution D
overXthere exists a function mD: (0,1)×(0,1)→Nsuch that for every
ϵ,δ> 0 ifm>mD(ϵ,δ) then
P
S∼Dm[D({x:x /∈S})>ϵ]<δ.
5. Prove that Memorize is a consistent learner for every class of (binary-
valued) functions over any countable domain.

8 The Runtime of Learning
So far in the book we have studied the statistical perspective of learning, namely,
how many samples are needed for learning. In other words, we focused on the
amount of information learning requires. However, when considering automated
learning, computational resources also play a major role in determining the com-
plexity of a task: that is, how much computation is involved in carrying out a
learning task. Once a suﬃcient training sample is available to the learner, there
is some computation to be done to extract a hypothesis or ﬁgure out the label of
a given test instance. These computational resources are crucial in any practical
application of machine learning. We refer to these two types of resources as the
sample complexity and the computational complexity . In this chapter, we turn
our attention to the computational complexity of learning.
The computational complexity of learning should be viewed in the wider con-

text of the computational complexity of general algorithmic tasks. This area has
been extensively investigated; see, for example, (Sipser 2006). The introductory
comments that follow summarize the basic ideas of that general theory that are
most relevant to our discussion.
The actual runtime (in seconds) of an algorithm depends on the speciﬁc ma-
chine the algorithm is being implemented on (e.g., what the clock rate of the
machine’s CPU is). To avoid dependence on the speciﬁc machine, it is common

to analyze the runtime of algorithms in an asymptotic sense. For example, we
say that the computational complexity of the merge-sort algorithm, which sorts
a list ofnitems, isO(nlog(n)). This implies that we can implement the algo-
rithm on any machine that satisﬁes the requirements of some accepted abstract
model of computation, and the actual runtime in seconds will satisfy the follow-
ing: there exist constants candn0, which can depend on the actual machine,

such that, for any value of n>n 0, the runtime in seconds of sorting any nitems
will be at most cnlog(n). It is common to use the term feasible oreﬃciently
computable for tasks that can be performed by an algorithm whose running time
isO(p(n)) for some polynomial function p. One should note that this type of
analysis depends on deﬁning what is the input size nof any instance to which
the algorithm is expected to be applied. For “purely algorithmic” tasks, as dis-
cussed in the common computational complexity literature, this input size is

clearly deﬁned; the algorithm gets an input instance, say, a list to be sorted, or
an arithmetic operation to be calculated, which has a well deﬁned size (say, the

number of bits in its representation). For machine learning tasks, the notion of
an input size is not so clear. An algorithm aims to detect some pattern in a data
set and can only access random samples of that data.
We start the chapter by discussing this issue and deﬁne the computational
complexity of learning. For advanced students, we also provide a detailed formal
deﬁnition. We then move on to consider the computational complexity of im-
plementing the ERM rule. We ﬁrst give several examples of hypothesis classes
where the ERM rule can be eﬃciently implemented, and then consider some
cases where, although the class is indeed eﬃciently learnable, ERM implemen-
tation is computationally hard. It follows that hardness of implementing ERM
does not imply hardness of learning. Finally, we brieﬂy discuss how one can show
hardness of a given learning task, namely, that no learning algorithm can solve
it eﬃciently.
8.1 Computational Complexity of Learning

Recall that a learning algorithm has access to a domain of examples, Z, a hy-
pothesis class,H, a loss function, ℓ, and a training set of examples from Zthat
are sampled i.i.d. according to an unknown distribution D. Given parameters
ϵ, δ, the algorithm should output a hypothesis hsuch that with probability of
at least 1−δ,
LD(h)≤min
As mentioned before, the actual runtime of an algorithm in seconds depends on
the speciﬁc machine. To allow machine independent analysis, we use the standard

LD(h)≤min
approach in computational complexity theory. First, we rely on a notion of an
abstract machine, such as a Turing machine (or a Turing machine over the reals
(Blum, Shub & Smale 1989)). Second, we analyze the runtime in an asymptotic
sense, while ignoring constant factors, thus the speciﬁc machine is not important
as long as it implements the abstract machine. Usually, the asymptote is with
respect to the size of the input to the algorithm. For example, for the merge-sort

algorithm mentioned before, we analyze the runtime as a function of the number
of items that need to be sorted.
In the context of learning algorithms, there is no clear notion of “input size.”
One might deﬁne the input size to be the size of the training set the algorithm
receives, but that would be rather pointless. If we give the algorithm a very
large number of examples, much larger than the sample complexity of the learn-
ing problem, the algorithm can simply ignore the extra examples. Therefore, a

larger training set does not make the learning problem more diﬃcult, and, con-
sequently, the runtime available for a learning algorithm should not increase as
we increase the size of the training set. Just the same, we can still analyze the
runtime as a function of natural parameters of the problem such as the target
accuracy, the conﬁdence of achieving that accuracy, the dimensionality of the

102 The Runtime of Learning
domain set, or some measures of the complexity of the hypothesis class with
which the algorithm’s output is compared.
To illustrate this, consider a learning algorithm for the task of learning axis
aligned rectangles. A speciﬁc problem of learning axis aligned rectangles is de-
rived by specifying ϵ,δ, and the dimension of the instance space. We can deﬁne a
sequence of problems of the type “rectangles learning” by ﬁxing ϵ, δand varying
the dimension to be d= 2,3,4,.... We can also deﬁne another sequence of “rect-
angles learning” problems by ﬁxing d, δ and varying the target accuracy to be
ϵ=1
2,1
3,.... One can of course choose other sequences of such problems. Once
a sequence of the problems is ﬁxed, one can analyze the asymptotic runtime as
a function of variables of that sequence.
Before we introduce the formal deﬁnition, there is one more subtlety we need
to tackle. On the basis of the preceding, a learning algorithm can “cheat,” by

ϵ=1
2,1
transferring the computational burden to the output hypothesis. For example,
the algorithm can simply deﬁne the output hypothesis to be the function that
stores the training set in its memory, and whenever it gets a test example x
it calculates the ERM hypothesis on the training set and applies it on x. Note
that in this case, our algorithm has a ﬁxed output (namely, the function that
we have just described) and can run in constant time. However, learning is still

hard – the hardness is now in implementing the output classiﬁer to obtain a
label prediction. To prevent this “cheating,” we shall require that the output of
a learning algorithm must be applied to predict the label of a new example in
time that does not exceed the runtime of training (that is, computing the output
classiﬁer from the input training sample). In the next subsection the advanced
reader may ﬁnd a formal deﬁnition of the computational complexity of learning.
8.1.1 Formal Deﬁnition*

The deﬁnition that follows relies on a notion of an underlying abstract machine,
which is usually either a Turing machine or a Turing machine over the reals. We
will measure the computational complexity of an algorithm using the number of
“operations” it needs to perform, where we assume that for any machine that
implements the underlying abstract machine there exists a constant csuch that
any such “operation” can be performed on the machine using cseconds.

definition 8.1 (The Computational Complexity of a Learning Algorithm)
We deﬁne the complexity of learning in two steps. First we consider the compu-
tational complexity of a ﬁxed learning problem (determined by a triplet ( Z,H,ℓ)
– a domain set, a benchmark hypothesis class, and a loss function). Then, in the
second step we consider the rate of change of that complexity along a sequence
of such tasks.
1. Given a function f: (0,1)2→N, a learning task ( Z,H,ℓ), and a learning

algorithm,A, we say thatAsolves the learning task in time O(f) if there
exists some constant number c, such that for every probability distribution D

overZ, and input ϵ, δ∈(0,1), whenAhas access to samples generated i.i.d.
byD,
• A terminates after performing at most cf(ϵ, δ) operations
•The output ofA, denotedhA, can be applied to predict the label of a new
example while performing at most cf(ϵ, δ) operations
•The output ofAis probably approximately correct; namely, with proba-
bility of at least 1 −δ(over the random samples Areceives),LD(hA)≤
minh′∈HLD(h′) +ϵ
2. Consider a sequence of learning problems, ( Zn,Hn,ℓn)∞
n=1, where problem n
is deﬁned by a domain Zn, a hypothesis class Hn, and a loss function ℓn.
LetAbe a learning algorithm designed for solving learning problems of
this form. Given a function g:N×(0,1)2→N, we say that the runtime of
Awith respect to the preceding sequence is O(g), if for all n,Asolves the
problem (Zn,Hn,ℓn) in timeO(fn), wherefn: (0,1)2→Nis deﬁned by
fn(ϵ,δ) =g(n,ϵ,δ ).
We say thatAis an eﬃcient algorithm with respect to a sequence ( Zn,Hn,ℓn)

if its runtime is O(p(n,1/ϵ,1/δ)) for some polynomial p.
From this deﬁnition we see that the question whether a general learning prob-
lem can be solved eﬃciently depends on how it can be broken into a sequence
of speciﬁc learning problems. For example, consider the problem of learning a
ﬁnite hypothesis class. As we showed in previous chapters, the ERM rule over
His guaranteed to ( ϵ,δ)-learnHif the number of training examples is order of
mH(ϵ,δ) = log(|H|/δ)/ϵ2. Assuming that the evaluation of a hypothesis on an

example takes a constant time, it is possible to implement the ERM rule in time
O(|H|mH(ϵ,δ)) by performing an exhaustive search over Hwith a training set
of sizemH(ϵ,δ). For any ﬁxed ﬁnite H, the exhaustive search algorithm runs
in polynomial time. Furthermore, if we deﬁne a sequence of problems in which
|Hn|=n, then the exhaustive search is still considered to be eﬃcient. However, if
we deﬁne a sequence of problems for which |Hn|= 2n, then the sample complex-
ity is still polynomial in nbut the computational complexity of the exhaustive

search algorithm grows exponentially with n(thus, rendered ineﬃcient).
8.2 Implementing the ERM Rule
Given a hypothesis class H, the ERMHrule is maybe the most natural learning
paradigm. Furthermore, for binary classiﬁcation problems we saw that if learning
is at all possible, it is possible with the ERM rule. In this section we discuss the
computational complexity of implementing the ERM rule for several hypothesis
classes.
Given a hypothesis class, H, a domain set Z, and a loss function ℓ, the corre-
sponding ERM Hrule can be deﬁned as follows:

104 The Runtime of Learning
On a ﬁnite input sample S∈Zmoutput some h∈Hthat minimizes the empirical loss,
LS(h) =1
|S|∑
z∈Sℓ(h,z).
This section studies the runtime of implementing the ERM rule for several
examples of learning tasks.
8.2.1 Finite Classes
Limiting the hypothesis class to be a ﬁnite class may be considered as a reason-
ably mild restriction. For example, Hcan be the set of all predictors that can be
implemented by a C++ program written in at most 10000 bits of code. Other ex-
amples of useful ﬁnite classes are any hypothesis class that can be parameterized
by a ﬁnite number of parameters, where we are satisﬁed with a representation
of each of the parameters using a ﬁnite number of bits, for example, the class of
axis aligned rectangles in the Euclidean space, Rd, when the parameters deﬁning
any given rectangle are speciﬁed up to some limited precision.
As we have shown in previous chapters, the sample complexity of learning a
ﬁnite class is upper bounded by mH(ϵ,δ) =clog(c|H|/δ)/ϵc, wherec= 1 in

the realizable case and c= 2 in the nonrealizable case. Therefore, the sample
complexity has a mild dependence on the size of H. In the example of C++
programs mentioned before, the number of hypotheses is 210,000but the sample
complexity is only c(10,000 + log(c/δ))/ϵc.
A straightforward approach for implementing the ERM rule over a ﬁnite hy-
pothesis class is to perform an exhaustive search. That is, for each h∈H we
calculate the empirical risk, LS(h), and return a hypothesis that minimizes
the empirical risk. Assuming that the evaluation of ℓ(h,z) on a single exam-

ple takes a constant amount of time, k, the runtime of this exhaustive search
becomesk|H|m, wheremis the size of the training set. If we let mto be the
upper bound on the sample complexity mentioned, then the runtime becomes
k|H|clog(c|H|/δ)/ϵc.
The linear dependence of the runtime on the size of Hmakes this approach
ineﬃcient (and unrealistic) for large classes. Formally, if we deﬁne a sequence of
problems (Zn,Hn,ℓn)∞
n=1such that log(|Hn|) =n, then the exhaustive search

approach yields an exponential runtime. In the example of C++ programs, if Hn
is the set of functions that can be implemented by a C++ program written in
at mostnbits of code, then the runtime grows exponentially with n, implying
that the exhaustive search approach is unrealistic for practical use. In fact, this
problem is one of the reasons we are dealing with other hypothesis classes, like
classes of linear predictors, which we will encounter in the next chapter, and not
just focusing on ﬁnite classes.

It is important to realize that the ineﬃciency of one algorithmic approach
(such as the exhaustive search) does not yet imply that no eﬃcient ERM imple-
mentation exists. Indeed, we will show examples in which the ERM rule can be
implemented eﬃciently.

8.2 Implementing the ERM Rule 105
8.2.2 Axis Aligned Rectangles
LetHnbe the class of axis aligned rectangles in Rn, namely,
Hn={h(a1,...,an,b1,...,bn):∀i,ai≤bi}
where
h(a1,...,an,b1,...,bn)(x,y) ={
1 if∀i, xi∈[ai,bi]
0 otherwise(8.1)
Eﬃciently Learnable in the Realizable Case
Consider implementing the ERM rule in the realizable case. That is, we are given
a training set S= (x1,y1),..., (xm,ym) of examples, such that there exists an
axis aligned rectangle, h∈Hn, for which h(xi) =yifor alli. Our goal is to ﬁnd
such an axis aligned rectangle with a zero training error, namely, a rectangle
that is consistent with all the labels in S.
We show later that this can be done in time O(nm). Indeed, for each i∈[n],
setai= min{xi: (x,1)∈S}andbi= max{xi: (x,1)∈S}. In words, we take
aito be the minimal value of the i’th coordinate of a positive example in Sand
bito be the maximal value of the i’th coordinate of a positive example in S.
It is easy to verify that the resulting rectangle has zero training error and that

the runtime of ﬁnding each aiandbiisO(m). Hence, the total runtime of this
procedure is O(nm).
Not Eﬃciently Learnable in the Agnostic Case
In the agnostic case, we do not assume that some hypothesis hperfectly predicts
the labels of all the examples in the training set. Our goal is therefore to ﬁnd
hthat minimizes the number of examples for which yi̸=h(xi). It turns out
that for many common hypothesis classes, including the classes of axis aligned

rectangles we consider here, solving the ERM problem in the agnostic setting is
NP-hard (and, in most cases, it is even NP-hard to ﬁnd some h∈H whose error
is no more than some constant c>1 times that of the empirical risk minimizer
inH). That is, unless P = NP, there is no algorithm whose running time is
polynomial in mandnthat is guaranteed to ﬁnd an ERM hypothesis for these
problems (Ben-David, Eiron & Long 2003).
On the other hand, it is worthwhile noticing that, if we ﬁx one speciﬁc hypoth-

esis class, say, axis aligned rectangles in some ﬁxed dimension, n, then there exist
eﬃcient learning algorithms for this class. In other words, there are successful
agnostic PAC learners that run in time polynomial in 1 /ϵand 1/δ(but their
dependence on the dimension nis not polynomial).
To see this, recall the implementation of the ERM rule we presented for the
realizable case, from which it follows that an axis aligned rectangle is determined

by at most 2 nexamples. Therefore, given a training set of size m, we can per-
form an exhaustive search over all subsets of the training set of size at most 2 n
examples and construct a rectangle from each such subset. Then, we can pick

106 The Runtime of Learning
the rectangle with the minimal training error. This procedure is guaranteed to
ﬁnd an ERM hypothesis, and the runtime of the procedure is mO(n). It follows
that ifnis ﬁxed, the runtime is polynomial in the sample size. This does not
contradict the aforementioned hardness result, since there we argued that unless
P=NP one cannot have an algorithm whose dependence on the dimension nis
polynomial as well.
8.2.3 Boolean Conjunctions
A Boolean conjunction is a mapping from X={0,1}ntoY={0,1}that can be
expressed as a proposition formula of the form xi1∧...∧xik∧¬xj1∧...∧¬xjr,
for some indices i1,...,ik,j1,...,jr∈[n]. The function that such a proposition
formula deﬁnes is
h(x) ={
1 ifxi1=···=xik= 1 andxj1=···=xjr= 0
LetHn
Cbe the class of all Boolean conjunctions over {0,1}n. The size ofHn
Cis
at most 3n+ 1 (since in a conjunction formula, each element of xeither appears,
or appears with a negation sign, or does not appear at all, and we also have the

h(x) ={
LetHn
Cis
all negative formula). Hence, the sample complexity of learning Hn
Cusing the
ERM rule is at most nlog(3/δ)/ϵ.
Next, we show that it is possible to solve the ERM problem for Hn
Cin time
polynomial in nandm. The idea is to deﬁne an ERM conjunction by including
in the hypothesis conjunction all the literals that do not contradict any positively
labeled example. Let v1,...,vm+be all the positively labeled instances in the
input sample S. We deﬁne, by induction on i≤m+, a sequence of hypotheses

Cusing the
Cin time
(or conjunctions). Let h0be the conjunction of all possible literals. That is,
h0=x1∧¬x1∧x2∧...∧xn∧¬xn. Note that h0assigns the label 0 to all the
elements ofX. We obtain hi+1by deleting from the conjunction hiall the literals
that are not satisﬁed by vi+1. The algorithm outputs the hypothesis hm+. Note
thathm+labels positively all the positively labeled examples in S. Furthermore,
for everyi≤m+,hiis the most restrictive conjunction that labels v1,...,vi
positively. Now, since we consider learning in the realizable setup, there exists

a conjunction hypothesis, f∈Hn
C, that is consistent with all the examples in
S. Sincehm+is the most restrictive conjunction that labels positively all the
positively labeled members of S, any instance labeled 0 by fis also labeled 0 by
hm+. It follows that hm+has zero training error (w.r.t. S), and is therefore a
legal ERM hypothesis. Note that the running time of this algorithm is O(mn).

As in the case of axis aligned rectangles, unless P = NP, there is no algorithm
whose running time is polynomial in mandnthat guaranteed to ﬁnd an ERM
hypothesis for the class of Boolean conjunctions in the unrealizable case.
8.2.4 Learning 3-Term DNF
We next show that a slight generalization of the class of Boolean conjunctions
leads to intractability of solving the ERM problem even in the realizable case.
Consider the class of 3-term disjunctive normal form formulae (3-term DNF).
The instance space is X={0,1}nand each hypothesis is represented by the
Boolean formula of the form h(x) =A1(x)∨A2(x)∨A3(x), where each Ai(x) is
a Boolean conjunction (as deﬁned in the previous section). The output of h(x) is
1 if eitherA1(x) orA2(x) orA3(x) outputs the label 1. If all three conjunctions
output the label 0 then h(x) = 0.
LetHn
3DNF be the hypothesis class of all such 3-term DNF formulae. The size
ofHn

LetHn
ofHn
3DNF is at most 33n. Hence, the sample complexity of learning Hn
3DNF using
the ERM rule is at most 3 nlog(3/δ)/ϵ.
However, from the computational perspective, this learning problem is hard.
It has been shown (see (Pitt & Valiant 1988, Kearns et al. 1994)) that unless
RP = NP, there is no polynomial time algorithm that properly learns a sequence
of 3-term DNF learning problems in which the dimension of the n’th problem is
n. By “properly” we mean that the algorithm should output a hypothesis that is

ofHn
3DNF using
a 3-term DNF formula. In particular, since ERM Hn
3DNFoutputs a 3-term DNF
formula it is a proper learner and therefore it is hard to implement it. The proof
uses a reduction of the graph 3-coloring problem to the problem of PAC learning
3-term DNF. The detailed technique is given in Exercise 3. See also (Kearns &
Vazirani 1994, Section 1.4).
8.3 Eﬃciently Learnable, but Not by a Proper ERM
In the previous section we saw that it is impossible to implement the ERM rule
eﬃciently for the class Hn

3DNF of 3-DNF formulae. In this section we show that it
is possible to learn this class eﬃciently, but using ERM with respect to a larger
class.
Representation Independent Learning Is Not Hard
Next we show that it is possible to learn 3-term DNF formulae eﬃciently. There
is no contradiction to the hardness result mentioned in the previous section as we
now allow “representation independent” learning. That is, we allow the learning
algorithm to output a hypothesis that is not a 3-term DNF formula. The ba-

class.
sic idea is to replace the original hypothesis class of 3-term DNF formula with
a larger hypothesis class so that the new class is easily learnable. The learning

108 The Runtime of Learning
algorithm might return a hypothesis that does not belong to the original hypoth-
esis class; hence the name “representation independent” learning. We emphasize
that in most situations, returning a hypothesis with good predictive ability is
what we are really interested in doing.
We start by noting that because ∨distributes over∧, each 3-term DNF formula
can be rewritten as
A1∨A2∨A3=⋀
u∈A1,v∈A2,w∈A3(u∨v∨w)
Next, let us deﬁne: ψ:{0,1}n→{0,1}(2n)3such that for each triplet of literals
u,v,w there is a variable in the range of ψindicating if u∨v∨wis true or false.
So, for each 3-DNF formula over {0,1}nthere is a conjunction over {0,1}(2n)3,
with the same truth table. Since we assume that the data is realizable, we can
solve the ERM problem with respect to the class of conjunctions over {0,1}(2n)3.
Furthermore, the sample complexity of learning the class of conjunctions in the
higher dimensional space is at most n3log(1/δ)/ϵ. Thus, the overall runtime of

this approach is polynomial in n.
Intuitively, the idea is as follows. We started with a hypothesis class for which
learning is hard. We switched to another representation where the hypothesis
class is larger than the original class but has more structure, which allows for a
more eﬃcient ERM search. In the new representation, solving the ERM problem
is easy.
3-term-DNF formulae over {0,1}n
conjunctions over{0,1}(2n)3
8.4 Hardness of Learning*
We have just demonstrated that the computational hardness of implementing

is easy.
ERMHdoes not imply that such a class His not learnable. How can we prove
that a learning problem is computationally hard?
One approach is to rely on cryptographic assumptions. In some sense, cryp-
tography is the opposite of learning. In learning we try to uncover some rule
underlying the examples we see, whereas in cryptography, the goal is to make
sure that nobody will be able to discover some secret, in spite of having access

8.4 Hardness of Learning* 109
to some partial information about it. On that high level intuitive sense, results
about the cryptographic security of some system translate into results about
the unlearnability of some corresponding task. Regrettably, currently one has no
way of proving that a cryptographic protocol is not breakable. Even the common
assumption of P ̸= NP does not suﬃce for that (although it can be shown to
be necessary for most common cryptographic scenarios). The common approach
for proving that cryptographic protocols are secure is to start with some cryp-
tographic assumptions . The more these are used as a basis for cryptography, the
stronger is our belief that they really hold (or, at least, that algorithms that will
refute them are hard to come by).
We now brieﬂy describe the basic idea of how to deduce hardness of learnabil-
ity from cryptographic assumptions. Many cryptographic systems rely on the
assumption that there exists a one way function. Roughly speaking, a one way

function is a function f:{0,1}n→{0,1}n(more formally, it is a sequence of
functions, one for each dimension n) that is easy to compute but is hard to in-
vert. More formally, fcan be computed in time poly( n) but for any randomized
polynomial time algorithm A, and for every polynomial p(·),
P[f(A(f(x))) =f(x)]<1
p(n),
where the probability is taken over a random choice of xaccording to the uniform
distribution over {0,1}nand the randomness of A.

p(n),
A one way function, f, is called trapdoor one way function if, for some poly-
nomial function p, for everynthere exists a bit-string sn(called a secret key) of
length≤p(n), such that there is a polynomial time algorithm that, for every n
and every x∈{0,1}n, on input ( f(x),sn) outputs x. In other words, although
fis hard to invert, once one has access to its secret key, inverting fbecomes
feasible. Such functions are parameterized by their secret key.
Now, letFnbe a family of trapdoor functions over {0,1}nthat can be calcu-

lated by some polynomial time algorithm. That is, we ﬁx an algorithm that given
a secret key (representing one function in Fn) and an input vector, it calculates
the value of the function corresponding to the secret key on the input vector in
polynomial time. Consider the task of learning the class of the corresponding
inverses,Hn
F={f−1:f∈Fn}. Since each function in this class can be inverted
by some secret key snof size polynomial in n, the classHn
Fcan be parameter-
ized by these keys and its size is at most 2p(n). Its sample complexity is therefore

polynomial in n. We claim that there can be no eﬃcient learner for this class. If
there were such a learner, L, then by sampling uniformly at random a polynomial
number of strings in {0,1}n, and computing fover them, we could generate a
labeled training sample of pairs ( f(x),x), which should suﬃce for our learner to
ﬁgure out an ( ϵ,δ) approximation of f−1(w.r.t. the uniform distribution over
the range of f), which would violate the one way property of f.
A more detailed treatment, as well as a concrete example, can be found in

(Kearns & Vazirani 1994, Chapter 6). Using reductions, they also show that

110 The Runtime of Learning
the class of functions that can be calculated by small Boolean circuits is not
eﬃciently learnable, even in the realizable case.
8.5 Summary
The runtime of learning algorithms is asymptotically analyzed as a function of
diﬀerent parameters of the learning problem, such as the size of the hypothe-
sis class, our measure of accuracy, our measure of conﬁdence, or the size of the
domain set. We have demonstrated cases in which the ERM rule can be imple-
mented eﬃciently. For example, we derived eﬃcient algorithms for solving the
ERM problem for the class of Boolean conjunctions and the class of axis aligned
rectangles, under the realizability assumption. However, implementing ERM for
these classes in the agnostic case is NP-hard. Recall that from the statistical
perspective, there is no diﬀerence between the realizable and agnostic cases (i.e.,
a class is learnable in both cases if and only if it has a ﬁnite VC-dimension).

In contrast, as we saw, from the computational perspective the diﬀerence is im-
mense. We have also shown another example, the class of 3-term DNF, where
implementing ERM is hard even in the realizable case, yet the class is eﬃciently
learnable by another algorithm.
Hardness of implementing the ERM rule for several natural hypothesis classes
has motivated the development of alternative learning methods, which we will
discuss in the next part of this book.
8.6 Bibliographic Remarks

Valiant (1984) introduced the eﬃcient PAC learning model in which the runtime
of the algorithm is required to be polynomial in 1 /ϵ, 1/δ, and the representation
size of hypotheses in the class. A detailed discussion and thorough bibliographic
notes are given in Kearns & Vazirani (1994).
8.7 Exercises
1. LetHbe the class of intervals on the line (formally equivalent to axis aligned
rectangles in dimension n= 1). Propose an implementation of the ERM H
learning rule (in the agnostic case) that given a training set of size m, runs

in timeO(m2).
Hint: Use dynamic programming.
2. LetH1,H2,...be a sequence of hypothesis classes for binary classiﬁcation.
Assume that there is a learning algorithm that implements the ERM rule in
the realizable case such that the output hypothesis of the algorithm for each
classHnonly depends on O(n) examples out of the training set. Furthermore,

8.7 Exercises 111
assume that such a hypothesis can be calculated given these O(n) examples
in timeO(n), and that the empirical risk of each such hypothesis can be
evaluated in time O(mn). For example, if Hnis the class of axis aligned
rectangles in Rn, we saw that it is possible to ﬁnd an ERM hypothesis in the
realizable case that is deﬁned by at most 2 nexamples. Prove that in such
cases, it is possible to ﬁnd an ERM hypothesis for Hnin the unrealizable case
in timeO(mnmO(n)).
3. In this exercise, we present several classes for which ﬁnding an ERM classi-
ﬁer is computationally hard. First, we introduce the class of n-dimensional
halfspaces, HSn, for a domainX=Rn. This is the class of all functions of
the formhw,b(x) = sign(⟨w,x⟩+b) where w,x∈Rn,⟨w,x⟩is their inner
product, and b∈R. See a detailed description in Chapter 9.
1. Show that ERM Hover the classH=HSnof linear predictors is compu-
tationally hard. More precisely, we consider the sequence of problems in

which the dimension ngrows linearly and the number of examples mis set
to be some constant times n.
Hint: You can prove the hardness by a reduction from the following prob-
lem:
Max FS: Given a system of linear inequalities, Ax>bwithA∈Rm×nandb∈
Rm(that is, a system of mlinear inequalities in nvariables, x= (x1,...,xn)),
ﬁnd a subsystem containing as many inequalities as possible that has a solution
(such a subsystem is called feasible ).
It has been shown (Sankaran 1993) that the problem Max FS is NP-hard.

lem:
Show that any algorithm that ﬁnds an ERM HSnhypothesis for any training
sampleS∈(Rn×{+1,−1})mcan be used to solve the Max FS problem of
sizem,n.Hint: Deﬁne a mapping that transforms linear inequalities in n
variables into labeled points in Rn, and a mapping that transforms vectors
inRnto halfspaces, such that a vector wsatisﬁes an inequality qif and
only if the labeled point that corresponds to qis classiﬁed correctly by the
halfspace corresponding to w. Conclude that the problem of empirical risk

minimization for halfspaces in also NP-hard (that is, if it can be solved in
time polynomial in the sample size, m, and the Euclidean dimension, n,
then every problem in the class NP can be solved in polynomial time).
2. LetX=Rnand letHn
kbe the class of all intersections of k-many linear
halfspaces in Rn. In this exercise, we wish to show that ERM Hn
kis com-
putationally hard for every k≥3. Precisely, we consider a sequence of
problems where k≥3 is a constant and ngrows linearly. The training set

kis com-
size,m, also grows linearly with n.
Towards this goal, consider the k-coloring problem for graphs, deﬁned as
follows:
Given a graph G= (V,E), and a number k, determine whether there exists a
functionf:V→{1...k}so that for every ( u,v)∈E,f(u)̸=f(v).
Thek-coloring problem is known to be NP-hard for every k≥3 (Karp
1972).

112 The Runtime of Learning
We wish to reduce the k-coloring problem to ERMHn
k: that is, to prove
that if there is an algorithm that solves the ERMHn
kproblem in time
polynomial in k,n, and the sample size m, then there is a polynomial time
algorithm for the graph k-coloring problem.
Given a graph G= (V,E), let{v1...vn}be the vertices in V. Construct
a sampleS(G)∈(Rn×{± 1})m, wherem=|V|+|E|, as follows:
•For everyvi∈V, construct an instance eiwith a negative label.
•For every edge ( vi,vj)∈E, construct an instance ( ei+ej)/2 with a
positive label.
1. Prove that if there exists some h∈Hn
kthat has zero error over S(G)
thenGisk-colorable.
Hint: Leth=⋂k
j=1hjbe an ERM classiﬁer in Hn
koverS. Deﬁne a
coloring ofVby settingf(vi) to be the minimal jsuch thathj(ei) =−1.
Use the fact that halfspaces are convex sets to show that it cannot be
true that two vertices that are connected by an edge have the same
color.
2. Prove that if Gisk-colorable then there exists some h∈Hn
kthat has
zero error over S(G).

color.
kthat has
Hint: Given a coloring fof the vertices of G, we should come up with k
hyperplanes, h1...hkwhose intersection is a perfect classiﬁer for S(G).
Letb= 0.6 for all of these hyperplanes and, for t≤klet thei’th weight
of thet’th hyperplane, wt,i, be−1 iff(vi) =tand 0 otherwise.
3. Based on the above, prove that for any k≥3, the ERMHn
kproblem is
NP-hard.
4. In this exercise we show that hardness of solving the ERM problem is equiv-
alent to hardness of proper PAC learning. Recall that by “properness” of the

NP-hard.
algorithm we mean that it must output a hypothesis from the hypothesis
class. To formalize this statement, we ﬁrst need the following deﬁnition.
definition 8.2 The complexity class Randomized Polynomial (RP) time
is the class of all decision problems (that is, problems in which on any instance
one has to ﬁnd out whether the answer is YES or NO) for which there exists a
probabilistic algorithm (namely, the algorithm is allowed to ﬂip random coins
while it is running) with these properties:

•On any input instance the algorithm runs in polynomial time in the input
size.
•If the correct answer is NO, the algorithm must return NO.
•If the correct answer is YES, the algorithm returns YES with probability
a≥1/2 and returns NO with probability 1 −a.1
Clearly the class RP contains the class P. It is also known that RP is
contained in the class NP. It is not known whether any equality holds among
these three complexity classes, but it is widely believed that NP is strictly

size.
1The constant 1/2 in the deﬁnition can be replaced by any constant in (0 ,1).

8.7 Exercises 113
larger than RP. In particular, it is believed that NP-hard problems cannot be
solved by a randomized polynomial time algorithm.
•Show that if a class Hisproperly PAC learnable by a polynomial time
algorithm, then the ERM Hproblem is in the class RP. In particular, this
implies that whenever the ERM Hproblem is NP-hard (for example, the
class of intersections of halfspaces discussed in the previous exercise),
then, unless NP = RP, there exists no polynomial time proper PAC
learning algorithm for H.
Hint: Assume you have an algorithm Athat properly PAC learns a
classHin time polynomial in some class parameter nas well as in 1 /ϵ
and 1/δ. Your goal is to use that algorithm as a subroutine to contract
an algorithm Bfor solving the ERM Hproblem in random polynomial
time. Given a training set, S∈(X×{± 1}m), and some h∈H whose
error onSis zero, apply the PAC learning algorithm to the uniform
distribution over Sand run it so that with probability ≥0.3 it ﬁnds a

functionh∈H that has error less than ϵ= 1/|S|(with respect to that
uniform distribution). Show that the algorithm just described satisﬁes
the requirements for being a RP solver for ERM H.

9 Linear Predictors
In this chapter we will study the family of linear predictors, one of the most
useful families of hypothesis classes. Many learning algorithms that are being
widely used in practice rely on linear predictors, ﬁrst and foremost because of
the ability to learn them eﬃciently in many cases. In addition, linear predictors
are intuitive, are easy to interpret, and ﬁt the data reasonably well in many
natural learning problems.
We will introduce several hypothesis classes belonging to this family – halfspaces,
linear regression predictors, and logistic regression predictors – and present rele-
vant learning algorithms: linear programming and the Perceptron algorithm for
the class of halfspaces and the Least Squares algorithm for linear regression.
This chapter is focused on learning linear predictors using the ERM approach;
however, in later chapters we will see alternative paradigms for learning these
hypothesis classes.
First, we deﬁne the class of aﬃne functions as
Ld={hw,b:w∈Rd,b∈R},
where

where
hw,b(x) =⟨w,x⟩+b=(d∑
i=1wixi)
+b.
It will be convenient also to use the notation
Ld={x↦→⟨w,x⟩+b:w∈Rd,b∈R},
which reads as follows: Ldis a set of functions, where each function is parame-
terized by w∈Rdandb∈R, and each such function takes as input a vector x
and returns as output the scalar ⟨w,x⟩+b.
The diﬀerent hypothesis classes of linear predictors are compositions of a func-
tionφ:R→Y onLd. For example, in binary classiﬁcation, we can choose φto
be the sign function, and for regression problems, where Y=R,φis simply the

i=1wixi)
+b.
identity function.
It may be more convenient to incorporate b, called the bias, into was an
extra coordinate and add an extra coordinate with a value of 1 to all x∈X;
namely, let w′= (b,w1,w2,...wd)∈Rd+1and let x′= (1,x1,x2,...,xd)∈

118 Linear Predictors
Rd+1. Therefore,
hw,b(x) =⟨w,x⟩+b=⟨w′,x′⟩.
It follows that each aﬃne function in Rdcan be rewritten as a homogenous linear
function in Rd+1applied over the transformation that appends the constant 1
to each input vector. Therefore, whenever it simpliﬁes the presentation, we will
omit the bias term and refer to Ldas the class of homogenous linear functions
of the form hw(x) =⟨w,x⟩.
Throughout the book we often use the general term “linear functions” for both
aﬃne functions and (homogenous) linear functions.
9.1 Halfspaces
The ﬁrst hypothesis class we consider is the class of halfspaces, designed for
binary classiﬁcation problems, namely, X=RdandY={−1,+1}. The class of
halfspaces is deﬁned as follows:
HSd= sign◦Ld={x↦→sign(hw,b(x)) :hw,b∈Ld}.
In other words, each halfspace hypothesis in HSdis parameterized by w∈
Rdandb∈Rand upon receiving a vector xthe hypothesis returns the label
sign(⟨w,x⟩+b).
To illustrate this hypothesis class geometrically, it is instructive to consider

the cased= 2. Each hypothesis forms a hyperplane that is perpendicular to the
vector wand intersects the vertical axis at the point (0 ,−b/w2). The instances
that are “above” the hyperplane, that is, share an acute angle with w, are labeled
positively. Instances that are “below” the hyperplane, that is, share an obtuse
angle with w, are labeled negatively.
w
−+−+
In Section 9.1.3 we will show that VCdim( HSd) =d+ 1. It follows that we
can learn halfspaces using the ERM paradigm, as long as the sample size is
Ω(
d+log(1/δ)
ϵ)

w
−+−+
Ω(
ϵ)
. Therefore, we now discuss how to implement an ERM procedure
for halfspaces.
We introduce below two solutions to ﬁnding an ERM halfspace in the realiz-
able case. In the context of halfspaces, the realizable case is often referred to as
the “separable” case, since it is possible to separate with a hyperplane all the
positive examples from all the negative examples. Implementing the ERM rule

9.1 Halfspaces 119
in the nonseparable case (i.e., the agnostic case) is known to be computationally
hard (Ben-David & Simon 2001). There are several approaches to learning non-
separable data. The most popular one is to use surrogate loss functions , namely,
to learn a halfspace that does not necessarily minimize the empirical risk with
the 0−1 loss, but rather with respect to a diﬀferent loss function. For example,
in Section 9.3 we will describe the logistic regression approach, which can be
implemented eﬃciently even in the nonseparable case. We will study surrogate
loss functions in more detail later on in Chapter 12.
9.1.1 Linear Programming for the Class of Halfspaces
Linear programs (LP) are problems that can be expressed as maximizing a linear
function subject to linear inequalities. That is,
max
w∈Rd⟨u,w⟩
subject to Aw≥v
where w∈Rdis the vector of variables we wish to determine, Ais anm×
dmatrix, and v∈Rm,u∈Rdare vectors. Linear programs can be solved

max
w∈Rd⟨u,w⟩
eﬃciently,1and furthermore, there are publicly available implementations of LP
solvers.
We will show that the ERM problem for halfspaces in the realizable case can
be expressed as a linear program. For simplicity, we assume the homogenous
case. LetS={(xi,yi)}m
i=1be a training set of size m. Since we assume the
realizable case, an ERM predictor should have zero errors on the training set.
That is, we are looking for some vector w∈Rdfor which
sign(⟨w,xi⟩) =yi,∀i= 1,...,m.
Equivalently, we are looking for some vector wfor which

solvers.
yi⟨w,xi⟩>0,∀i= 1,...,m.
Letw∗be a vector that satisﬁes this condition (it must exist since we assume
realizability). Deﬁne γ= mini(yi⟨w∗,xi⟩) and let ¯w=w∗
γ. Therefore, for all i
we have
yi⟨¯w,xi⟩=1
γyi⟨w∗,xi⟩≥1.
We have thus shown that there exists a vector that satisﬁes
yi⟨w,xi⟩≥1,∀i= 1,...,m. (9.1)
And clearly, such a vector is an ERM predictor.
To ﬁnd a vector that satisﬁes Equation (9.1) we can rely on an LP solver as
follows. Set Ato be them×dmatrix whose rows are the instances multiplied

we have
1Namely, in time polynomial in m,d, and in the representation size of real numbers.

120 Linear Predictors
byyi. That is,Ai,j=yixi,j, wherexi,jis thej’th element of the vector xi. Let
vbe the vector (1 ,..., 1)∈Rm. Then, Equation (9.1) can be rewritten as
Aw≥v.
The LP form requires a maximization objective, yet all the wthat satisfy the
constraints are equal candidates as output hypotheses. Thus, we set a “dummy”
objective, u= (0,..., 0)∈Rd.
9.1.2 Perceptron for Halfspaces
A diﬀerent implementation of the ERM rule is the Perceptron algorithm of
Rosenblatt (Rosenblatt 1958). The Perceptron is an iterative algorithm that
constructs a sequence of vectors w(1),w(2),.... Initially, w(1)is set to be the
all-zeros vector. At iteration t, the Perceptron ﬁnds an example ithat is mis-
labeled by w(t), namely, an example for which sign( ⟨w(t),xi⟩)̸=yi. Then, the
Perceptron updates w(t)by adding to it the instance xiscaled by the label yi.
That is, w(t+1)=w(t)+yixi. Recall that our goal is to have yi⟨w,xi⟩>0 for
alliand note that
yi⟨w(t+1),xi⟩=yi⟨w(t)+yixi,xi⟩=yi⟨w(t),xi⟩+∥xi∥2.

Hence, the update of the Perceptron guides the solution to be “more correct” on
thei’th example.
Batch Perceptron
input: A training set ( x1,y1),..., (xm,ym)
initialize: w(1)= (0,..., 0)
fort= 1,2,...
if(∃is.t.yi⟨w(t),xi⟩≤0) then
w(t+1)=w(t)+yixi
else
output w(t)
The following theorem guarantees that in the realizable case, the algorithm
stops with all sample points correctly classiﬁed.
theorem 9.1 Assume that (x1,y1),..., (xm,ym)is separable, let B= min{∥w∥:
∀i∈[m], yi⟨w,xi⟩≥ 1}, and letR= maxi∥xi∥. Then, the Perceptron al-

else
gorithm stops after at most (RB)2iterations, and when it stops it holds that
∀i∈[m], yi⟨w(t),xi⟩>0.
Proof By the deﬁnition of the stopping condition, if the Perceptron stops it
must have separated all the examples. We will show that if the Perceptron runs
forTiterations, then we must have T≤(RB)2, which implies the Perceptron
must stop after at most ( RB)2iterations.
Letw⋆be a vector that achieves the minimum in the deﬁnition of B. That is,

9.1 Halfspaces 121
yi⟨w⋆,xi⟩≥1 for alli, and among all vectors that satisfy these constraints, w⋆
is of minimal norm.
The idea of the proof is to show that after performing Titerations, the cosine
of the angle between w⋆andw(T+1)is at least√
T
RB. That is, we will show that
⟨w⋆,w(T+1)⟩
∥w⋆∥∥w(T+1)∥≥√
T
RB. (9.2)
By the Cauchy-Schwartz inequality, the left-hand side of Equation (9.2) is at
most 1. Therefore, Equation (9.2) would imply that
1≥√
T
RB⇒T≤(RB)2,
which will conclude our proof.
To show that Equation (9.2) holds, we ﬁrst show that ⟨w⋆,w(T+1)⟩ ≥T.
Indeed, at the ﬁrst iteration, w(1)= (0,..., 0) and therefore ⟨w⋆,w(1)⟩= 0,
while on iteration t, if we update using example ( xi,yi) we have that
⟨w⋆,w(t+1)⟩−⟨w⋆,w(t)⟩=⟨w⋆,w(t+1)−w(t)⟩
=⟨w⋆,yixi⟩=yi⟨w⋆,xi⟩
≥1.
Therefore, after performing Titerations, we get:
⟨w⋆,w(T+1)⟩=T∑
t=1(
⟨w⋆,w(t+1)⟩−⟨w⋆,w(t)⟩)
≥T, (9.3)
Next, we upper bound ∥w(T+1)∥. For each iteration twe have that
∥w(t+1)∥2=∥w(t)+yixi∥2
=∥w(t)∥2+ 2yi⟨w(t),xi⟩+y2
i∥xi∥2
≤∥w(t)∥2+R2(9.4)

≥1.
t=1(
≥T, (9.3)
i∥xi∥2
where the last inequality is due to the fact that example iis necessarily such
thatyi⟨w(t),xi⟩≤0, and the norm of xiis at mostR. Now, since∥w(1)∥2= 0,
if we use Equation (9.4) recursively for Titerations, we obtain that
∥w(T+1)∥2≤TR2⇒ ∥ w(T+1)∥≤√
TR. (9.5)
Combining Equation (9.3) with Equation (9.5), and using the fact that ∥w⋆∥=
B, we obtain that
⟨w(T+1),w⋆⟩
∥w⋆∥∥w(T+1)∥≥T
B√
TR=√
T
BR.
We have thus shown that Equation (9.2) holds, and this concludes our proof.

122 Linear Predictors
Remark 9.1 The Perceptron is simple to implement and is guaranteed to con-
verge. However, the convergence rate depends on the parameter B, which in
some situations might be exponentially large in d. In such cases, it would be
better to implement the ERM problem by solving a linear program, as described
in the previous section. Nevertheless, for many natural data sets, the size of B
is not too large, and the Perceptron converges quite fast.
9.1.3 The VC Dimension of Halfspaces
To compute the VC dimension of halfspaces, we start with the homogenous case.
theorem 9.2 The VC dimension of the class of homogenous halfspaces in Rd
isd.
Proof First, consider the set of vectors e1,...,ed, where for every ithe vector
eiis the all zeros vector except 1 in the i’th coordinate. This set is shattered
by the class of homogenous halfspaces. Indeed, for every labeling y1,...,yd, set
w= (y1,...,yd), and then⟨w,ei⟩=yifor alli.
Next, let x1,...,xd+1be a set of d+ 1 vectors in Rd. Then, there must exist

isd.
real numbers a1,...,ad+1, not all of them are zero, such that∑d+1
i=1aixi=0.
LetI={i:ai>0}andJ={j:aj<0}. EitherIorJis nonempty. Let us
ﬁrst assume that both of them are nonempty. Then,
∑
i∈Iaixi=∑
j∈J|aj|xj.
Now, suppose that x1,...,xd+1are shattered by the class of homogenous classes.
Then, there must exist a vector wsuch that⟨w,xi⟩>0 for alli∈Iwhile
⟨w,xj⟩<0 for every j∈J. It follows that
0<∑
i∈Iai⟨xi,w⟩=⟨∑
i∈Iaixi,w⟩
=⟨∑
j∈J|aj|xj,w⟩
=∑
j∈J|aj|⟨xj,w⟩<0,
which leads to a contradiction. Finally, if J(respectively, I) is empty then the

∑
i∈Iaixi=∑
0<∑
=⟨∑
=∑
right-most (respectively, left-most) inequality should be replaced by an equality,
which still leads to a contradiction.
theorem 9.3 The VC dimension of the class of nonhomogenous halfspaces in
Rdisd+ 1.
Proof First, as in the proof of Theorem 9.2, it is easy to verify that the set
of vectors 0,e1,...,edis shattered by the class of nonhomogenous halfspaces.
Second, suppose that the vectors x1,...,xd+2are shattered by the class of non-
homogenous halfspaces. But, using the reduction we have shown in the beginning

Rdisd+ 1.
of this chapter, it follows that there are d+ 2 vectors in Rd+1that are shattered
by the class of homogenous halfspaces. But this contradicts Theorem 9.2.

rrrr
rr rr
rrr
Figure 9.1 Linear regression for d= 1. For instance, the x-axis may denote the age of
the baby, and the y-axis her weight.
9.2 Linear Regression
Linear regression is a common statistical tool for modeling the relationship be-
tween some “explanatory” variables and some real valued outcome. Cast as a
learning problem, the domain set Xis a subset of Rd, for somed, and the la-
bel setYis the set of real numbers. We would like to learn a linear function
h:Rd→Rthat best approximates the relationship between our variables (say,
for example, predicting the weight of a baby as a function of her age and weight
at birth). Figure 9.1 shows an example of a linear regression predictor for d= 1.
The hypothesis class of linear regression predictors is simply the set of linear
functions,
Hreg=Ld={x↦→⟨w,x⟩+b:w∈Rd, b∈R}.
Next we need to deﬁne a loss function for regression. While in classiﬁcation the
deﬁnition of the loss is straightforward, as ℓ(h,(x,y)) simply indicates whether

h(x) correctly predicts yor not, in regression, if the baby’s weight is 3 kg, both
the predictions 3.00001 kg and 4 kg are “wrong,” but we would clearly prefer
the former over the latter. We therefore need to deﬁne how much we shall be
“penalized” for the discrepancy between h(x) andy. One common way is to use
the squared-loss function, namely,
ℓ(h,(x,y)) = (h(x)−y)2.
For this loss function, the empirical risk function is called the Mean Squared
Error, namely,
LS(h) =1
mm∑
i=1(h(xi)−yi)2.

124 Linear Predictors
In the next subsection, we will see how to implement the ERM rule for linear
regression with respect to the squared loss. Of course, there are a variety of other
loss functions that one can use, for example, the absolute value loss function,
ℓ(h,(x,y)) =|h(x)−y|. The ERM rule for the absolute value loss function can
be implemented using linear programming (see Exercise 1.)
Note that since linear regression is not a binary prediction task, we cannot an-
alyze its sample complexity using the VC-dimension. One possible analysis of the
sample complexity of linear regression is by relying on the “discretization trick”
(see Remark 4.1 in Chapter 4); namely, if we are happy with a representation of
each element of the vector wand the bias busing a ﬁnite number of bits (say
a 64 bits ﬂoating point representation), then the hypothesis class becomes ﬁnite
and its size is at most 264(d+1). We can now rely on sample complexity bounds

for ﬁnite hypothesis classes as described in Chapter 4. Note, however, that to
apply the sample complexity bounds from Chapter 4 we also need that the loss
function will be bounded. Later in the book we will describe more rigorous means
to analyze the sample complexity of regression problems.
9.2.1 Least Squares
Least squares is the algorithm that solves the ERM problem for the hypoth-
esis class of linear regression predictors with respect to the squared loss. The

ERM problem with respect to this class, given a training set S, and using the
homogenous version of Ld, is to ﬁnd
argmin
wLS(hw) = argmin
w1
mm∑
i=1(⟨w,xi⟩−yi)2.
To solve the problem we calculate the gradient of the objective function and
compare it to zero. That is, we need to solve
2
mm∑
i=1(⟨w,xi⟩−yi)xi= 0.
We can rewrite the problem as the problem Aw=bwhere
A=(m∑
i=1xix⊤
i)
and b=m∑
i=1yixi. (9.6)

9.2 Linear Regression 125
Or, in matrix form:
A=
......
x1...xm
......

......
x1...xm
......
⊤
, (9.7)
b=
......
x1...xm
......

y1
...
ym
. (9.8)
IfAis invertible then the solution to the ERM problem is
w=A−1b.
The case in which Ais not invertible requires a few standard tools from linear
algebra, which are available in Appendix C. It can be easily shown that if the
training instances do not span the entire space of RdthenAis not invertible.
Nevertheless, we can always ﬁnd a solution to the system Aw=bbecause b
is in the range of A. Indeed, since Ais symmetric we can write it using its
eigenvalue decomposition as A=VDV⊤, whereDis a diagonal matrix and V
is an orthonormal matrix (that is, V⊤Vis the identity d×dmatrix). Deﬁne
D+to be the diagonal matrix such that D+
i,i= 0 ifDi,i= 0 and otherwise
D+
i,i= 1/Di,i. Now, deﬁne
A+=VD+V⊤and ˆw=A+b.
Letvidenote the i’th column of V. Then, we have
Aˆw=AA+b=VDV⊤VD+V⊤b=VDD+V⊤b=∑
i:Di,i̸=0viv⊤
ib.

D+
ib.
That is,Aˆwis the projection of bonto the span of those vectors vifor which
Di,i̸= 0. Since the linear span of x1,...,xmis the same as the linear span of
those vi, and bis in the linear span of the xi, we obtain that Aˆw=b, which
concludes our argument.
9.2.2 Linear Regression for Polynomial Regression Tasks
Some learning tasks call for nonlinear predictors, such as polynomial predictors.
Take, for instance, a one dimensional polynomial function of degree n, that is,
p(x) =a0+a1x+a2x2+···+anxn

ib.
where (a0,...,an) is a vector of coeﬃcients of size n+ 1. In the following we
depict a training set that is better ﬁtted using a 3rd degree polynomial predictor
than using a linear predictor.

126 Linear Predictors
We will focus here on the class of one dimensional, n-degree, polynomial re-
gression predictors, namely,
Hn
poly={x↦→p(x)},
wherepis a one dimensional polynomial of degree n, parameterized by a vector
of coeﬃcients ( a0,...,an). Note thatX=R, since this is a one dimensional
polynomial, andY=R, as this is a regression problem.
One way to learn this class is by reduction to the problem of linear regression,
which we have already shown how to solve. To translate a polynomial regression
problem to a linear regression problem, we deﬁne the mapping ψ:R→Rn+1
such thatψ(x) = (1,x,x2,...,xn). Then we have that
p(ψ(x)) =a0+a1x+a2x2+···+anxn=⟨a,ψ(x)⟩
and we can ﬁnd the optimal vector of coeﬃcients aby using the Least Squares
algorithm as shown earlier.
9.3 Logistic Regression
In logistic regression we learn a family of functions hfromRdto the interval [0 ,1].
However, logistic regression is used for classiﬁcation tasks: We can interpret h(x)

as the probability that the label of xis 1. The hypothesis class associated with
logistic regression is the composition of a sigmoid function φsig:R→[0,1] over
the class of linear functions Ld. In particular, the sigmoid function used in logistic
regression is the logistic function , deﬁned as
φsig(z) =1
1 + exp(−z). (9.9)
The name “sigmoid” means “S-shaped,” referring to the plot of this function,
shown in the ﬁgure:

9.3 Logistic Regression 127
The hypothesis class is therefore (where for simplicity we are using homogenous
linear functions):
Hsig=φsig◦Ld={x↦→φsig(⟨w,x⟩) :w∈Rd}.
Note that when⟨w,x⟩is very large then φsig(⟨w,x⟩) is close to 1, whereas if
⟨w,x⟩is very small then φsig(⟨w,x⟩) is close to 0. Recall that the prediction of the
halfspace corresponding to a vector wis sign(⟨w,x⟩). Therefore, the predictions
of the halfspace hypothesis and the logistic hypothesis are very similar whenever
|⟨w,x⟩|is large. However, when |⟨w,x⟩|is close to 0 we have that φsig(⟨w,x⟩)≈
1
2. Intuitively, the logistic hypothesis is not sure about the value of the label so it
guesses that the label is sign( ⟨w,x⟩) with probability slightly larger than 50%.
In contrast, the halfspace hypothesis always outputs a deterministic prediction
of either 1 or−1, even if|⟨w,x⟩|is very close to 0.
Next, we need to specify a loss function. That is, we should deﬁne how bad it
is to predict some hw(x)∈[0,1] given that the true label is y∈{± 1}. Clearly,

1
we would like that hw(x) would be large if y= 1 and that 1−hw(x) (i.e., the
probability of predicting −1) would be large if y=−1. Note that
1−hw(x) = 1−1
1 + exp(−⟨w,x⟩)=exp(−⟨w,x⟩)
1 + exp(−⟨w,x⟩)=1
1 + exp(⟨w,x⟩).
Therefore, any reasonable loss function would increase monotonically with1
1+exp(y⟨w,x⟩),
or equivalently, would increase monotonically with 1 + exp( −y⟨w,x⟩). The lo-
gistic loss function used in logistic regression penalizes hwbased on the log of
1 + exp(−y⟨w,x⟩) (recall that log is a monotonic function). That is,

ℓ(hw,(x,y)) = log (1 + exp( −y⟨w,x⟩)).
Therefore, given a training set S= (x1,y1),..., (xm,ym), the ERM problem
associated with logistic regression is
argmin
w∈Rd1
mm∑
i=1log (1 + exp(−yi⟨w,xi⟩)). (9.10)
The advantage of the logistic loss function is that it is a convex function with
respect to w; hence the ERM problem can be solved eﬃciently using standard
methods. We will study how to learn with convex functions, and in particular
specify a simple algorithm for minimizing convex functions, in later chapters.

argmin
w∈Rd1
mm∑
The ERM problem associated with logistic regression (Equation (9.10)) is iden-
tical to the problem of ﬁnding a Maximum Likelihood Estimator, a well-known
statistical approach for ﬁnding the parameters that maximize the joint probabil-
ity of a given data set assuming a speciﬁc parametric probability function. We
will study the Maximum Likelihood approach in Chapter 24.

128 Linear Predictors
9.4 Summary
The family of linear predictors is one of the most useful families of hypothesis
classes, and many learning algorithms that are being widely used in practice
rely on linear predictors. We have shown eﬃcient algorithms for learning linear
predictors with respect to the zero-one loss in the separable case and with respect
to the squared and logistic losses in the unrealizable case. In later chapters we
will present the properties of the loss function that enable eﬃcient learning.
Naturally, linear predictors are eﬀective whenever we assume, as prior knowl-
edge, that some linear predictor attains low risk with respect to the underlying
distribution. In the next chapter we show how to construct nonlinear predictors
by composing linear predictors on top of simple classes. This will enable us to
employ linear predictors for a variety of prior knowledge assumptions.
9.5 Bibliographic Remarks
The Perceptron algorithm dates back to Rosenblatt (1958). The proof of its

convergence rate is due to (Agmon 1954, Novikoﬀ 1962). Least Squares regression
goes back to Gauss (1795), Legendre (1805), and Adrain (1808).
9.6 Exercises
1. Show how to cast the ERM problem of linear regression with respect to the
absolute value loss function, ℓ(h,(x,y)) =|h(x)−y|, as a linear program;
namely, show how to write the problem
min
wm∑
i=1|⟨w,xi⟩−yi|
as a linear program.
Hint: Start with proving that for any c∈R,
|c|= min
a≥0as.t.c≤aandc≥−a.

min
wm∑
|c|= min
2. Show that the matrix Adeﬁned in Equation (9.6) is invertible if and only if
x1,...,xmspanRd.
3. Show that Theorem 9.1 is tight in the following sense: For any positive integer
m, there exist a vector w∗∈Rd(for some appropriate d) and a sequence of
examples{(x1,y1),..., (xm,ym)}such that the following hold:
•R= maxi∥xi∥≤1.
• ∥w∗∥2=m, and for all i≤m,yi⟨xi,w∗⟩≥1. Note that, using the notation
in Theorem 9.1, we therefore get
B= min{∥w∥:∀i∈[m], yi⟨w,xi⟩≥1}≤√m.

9.6 Exercises 129
Thus, (BR)2≤m.
•When running the Perceptron on this sequence of examples it makes m
updates before converging.
Hint: Choosed=mand for every ichoose xi=ei.
4.(*)Given any number m, ﬁnd an example of a sequence of labeled examples
((x1,y1),..., (xm,ym))∈(R3×{− 1,+1})mon which the upper bound of
Theorem 9.1 equals mand the perceptron algorithm is bound to make m
mistakes.
Hint: Set each xito be a third dimensional vector of the form ( a,b,yi), where
a2+b2=R2−1. Let w∗be the vector (0 ,0,1). Now, go over the proof of
the Perceptron’s upper bound (Theorem 9.1), see where we used inequalities
(≤) rather than equalities (=), and ﬁgure out scenarios where the inequality
actually holds with equality.
5. Suppose we modify the Perceptron algorithm as follows: In the update step,
instead of performing w(t+1)=w(t)+yixiwhenever we make a mistake, we
perform w(t+1)=w(t)+ηyixifor someη >0. Prove that the modiﬁed Per-
ceptron will perform the same number of iterations as the vanilla Perceptron

and will converge to a vector that points to the same direction as the output
of the vanilla Perceptron.
6. In this problem, we will get bounds on the VC-dimension of the class of
(closed) balls in Rd, that is,
Bd={Bv,r:v∈Rd,r> 0},
where
Bv,r(x) ={1 if∥x−v∥≤r
0 otherwise.
1. Consider the mapping φ:Rd→Rd+1deﬁned by φ(x) = (x,∥x∥2). Show
that if x1,...,xmare shattered byBdthenφ(x1),...,φ (xm) are shattered
by the class of halfspaces in Rd+1(in this question we assume that sign(0) =
1). What does this tell us about VCdim( Bd)?

where
2.(*)Find a set of d+ 1 points in Rdthat is shattered by Bd. Conclude that
d+ 1≤VCdim(Bd)≤d+ 2.

10 Boosting
Boosting is an algorithmic paradigm that grew out of a theoretical question and
became a very practical machine learning tool. The boosting approach uses a
generalization of linear predictors to address two major issues that have been
raised earlier in the book. The ﬁrst is the bias-complexity tradeoﬀ. We have
seen (in Chapter 5) that the error of an ERM learner can be decomposed into
a sum of approximation error and estimation error . The more expressive the
hypothesis class the learner is searching over, the smaller the approximation
error is, but the larger the estimation error becomes. A learner is thus faced with
the problem of picking a good tradeoﬀ between these two considerations. The
boosting paradigm allows the learner to have smooth control over this tradeoﬀ.
The learning starts with a basic class (that might have a large approximation
error), and as it progresses the class that the predictor may belong to grows
richer.

richer.
The second issue that boosting addresses is the computational complexity of
learning. As seen in Chapter 8, for many interesting concept classes the task
of ﬁnding an ERM hypothesis may be computationally infeasible. A boosting
algorithm ampliﬁes the accuracy of weak learners . Intuitively, one can think of
a weak learner as an algorithm that uses a simple “rule of thumb” to output a
hypothesis that comes from an easy-to-learn hypothesis class and performs just
slightly better than a random guess. When a weak learner can be implemented

eﬃciently, boosting provides a tool for aggregating such weak hypotheses to
approximate gradually good predictors for larger, and harder to learn, classes.
In this chapter we will describe and analyze a practically useful boosting algo-
rithm, AdaBoost (a shorthand for Adaptive Boosting). The AdaBoost algorithm
outputs a hypothesis that is a linear combination of simple hypotheses. In other
words, AdaBoost relies on the family of hypothesis classes obtained by composing
a linear predictor on top of simple classes. We will show that AdaBoost enables

us to control the tradeoﬀ between the approximation and estimation errors by
varying a single parameter.
AdaBoost demonstrates a general theme, that will recur later in the book, of
expanding the expressiveness of linear predictors by composing them on top of
other functions. This will be elaborated in Section 10.3.
AdaBoost stemmed from the theoretical question of whether an eﬃcient weak
learner can be “boosted” into an eﬃcient strong learner. This question was raised

by Kearns and Valiant in 1988 and solved in 1990 by Robert Schapire, then
a graduate student at MIT. However, the proposed mechanism was not very
practical. In 1995, Robert Schapire and Yoav Freund proposed the AdaBoost
algorithm, which was the ﬁrst truly practical implementation of boosting. This
simple and elegant algorithm became hugely popular, and Freund and Schapire’s
work has been recognized by numerous awards.
Furthermore, boosting is a great example for the practical impact of learning
theory. While boosting originated as a purely theoretical problem, it has led to
popular and widely used algorithms. Indeed, as we shall demonstrate later in
this chapter, AdaBoost has been successfully used for learning to detect faces in
images.
10.1 Weak Learnability
Recall the deﬁnition of PAC learning given in Chapter 3: A hypothesis class,
H, is PAC learnable if there exist mH: (0,1)2→Nand a learning algorithm

images.
with the following property: For every ϵ,δ∈(0,1), for every distribution Dover
X, and for every labeling function f:X →{± 1}, if the realizable assumption
holds with respect to H,D,f, then when running the learning algorithm on
m≥mH(ϵ,δ) i.i.d. examples generated by Dand labeled by f, the algorithm
returns a hypothesis hsuch that, with probability of at least 1 −δ,L(D,f)(h)≤ϵ.
Furthermore, the fundamental theorem of learning theory (Theorem 6.8 in

Chapter 6) characterizes the family of learnable classes and states that every PAC
learnable class can be learned using any ERM algorithm. However, the deﬁnition
of PAC learning and the fundamental theorem of learning theory ignores the
computational aspect of learning. Indeed, as we have shown in Chapter 8, there
are cases in which implementing the ERM rule is computationally hard (even in
the realizable case).
However, perhaps we can trade computational hardness with the requirement
for accuracy. Given a distribution Dand a target labeling function f, maybe there

exists an eﬃciently computable learning algorithm whose error is just slightly
better than a random guess? This motivates the following deﬁnition.
definition 10.1 (γ-Weak-Learnability)
•A learning algorithm, A, is aγ-weak-learner for a class Hif there exists a func-
tionmH: (0,1)→Nsuch that for every δ∈(0,1), for every distribution
DoverX, and for every labeling function f:X→{± 1}, if the realizable
assumption holds with respect to H,D,f, then when running the learning

algorithm on m≥mH(δ) i.i.d. examples generated by Dand labeled by f,
the algorithm returns a hypothesis hsuch that, with probability of at least
1−δ,L(D,f)(h)≤1/2−γ.
•A hypothesis class Hisγ-weak-learnable if there exists a γ-weak-learner for
that class.

132 Boosting
This deﬁnition is almost identical to the deﬁnition of PAC learning, which
here we will call strong learning , with one crucial diﬀerence: Strong learnability
implies the ability to ﬁnd an arbitrarily good classiﬁer (with error rate at most
ϵfor an arbitrarily small ϵ>0). In weak learnability, however, we only need to
output a hypothesis whose error rate is at most 1 /2−γ, namely, whose error
rate is slightly better than what a random labeling would give us. The hope is
that it may be easier to come up with eﬃcient weak learners than with eﬃcient
(full) PAC learners.
The fundamental theorem of learning (Theorem 6.8) states that if a hypothesis
classHhas a VC dimension d, then the sample complexity of PAC learning H
satisﬁesmH(ϵ,δ)≥C1d+log(1/δ)
ϵ, whereC1is a constant. Applying this with
ϵ= 1/2−γwe immediately obtain that if d=∞thenHis notγ-weak-learnable.
This implies that from the statistical perspective (i.e., if we ignore computational

complexity), weak learnability is also characterized by the VC dimension of H
and therefore is just as hard as PAC (strong) learning. However, when we do
consider computational complexity, the potential advantage of weak learning is
that maybe there is an algorithm that satisﬁes the requirements of weak learning
and can be implemented eﬃciently.
One possible approach is to take a “simple” hypothesis class, denoted B, and
to apply ERM with respect to Bas the weak learning algorithm. For this to

work, we need that Bwill satisfy two requirements:
•ERMBis eﬃciently implementable.
•For every sample that is labeled by some hypothesis from H, any ERM B
hypothesis will have an error of at most 1 /2−γ.
Then, the immediate question is whether we can boost an eﬃcient weak learner
into an eﬃcient strong learner. In the next section we will show that this is
indeed possible, but before that, let us show an example in which eﬃcient weak
learnability of a class His possible using a base hypothesis class B.

Example 10.1 (Weak Learning of 3-Piece Classiﬁers Using Decision Stumps)
LetX=Rand letHbe the class of 3-piece classiﬁers, namely, H={hθ1,θ2,b:
θ1,θ2∈R,θ1<θ2,b∈{± 1}}, where for every x,
hθ1,θ2,b(x) ={
+bifx<θ 1orx>θ 2
−bifθ1≤x≤θ2
An example hypothesis (for b= 1) is illustrated as follows:
θ1 θ2+ +−
LetBbe the class of Decision Stumps, that is, B={x↦→sign(x−θ)·b:θ∈
R,b∈{± 1}}. In the following we show that ERM Bis aγ-weak learner for H,
forγ= 1/12.

10.1 Weak Learnability 133
To see that, we ﬁrst show that for every distribution that is consistent with
H, there exists a decision stump with LD(h)≤1/3. Indeed, just note that
every classiﬁer in Hconsists of three regions (two unbounded rays and a center
interval) with alternate labels. For any pair of such regions, there exists a decision
stump that agrees with the labeling of these two components. Note that for every
distributionDoverRand every partitioning of the line into three such regions,
one of these regions must have D-weight of at most 1 /3. Leth∈H be a zero
error hypothesis. A decision stump that disagrees with honly on such a region
has an error of at most 1 /3.
Finally, since the VC-dimension of decision stumps is 2, if the sample size is
greater than Ω(log(1 /δ)/ϵ2), then with probability of at least 1 −δ, the ERM B
rule returns a hypothesis with an error of at most 1 /3 +ϵ. Settingϵ= 1/12 we
obtain that the error of ERM Bis at most 1 /3 + 1/12 = 1/2−1/12.

We see that ERM Bis aγ-weak learner for H. We next show how to implement
the ERM rule eﬃciently for decision stumps.
10.1.1 Eﬃcient Implementation of ERM for Decision Stumps
LetX=Rdand consider the base hypothesis class of decision stumps over Rd,
namely,
HDS={x↦→sign(θ−xi)·b:θ∈R,i∈[d],b∈{± 1}}.
For simplicity, assume that b= 1; that is, we focus on all the hypotheses in
HDSof the form sign( θ−xi). LetS= ((x1,y1),..., (xm,ym)) be a training set.
We will show how to implement an ERM rule, namely, how to ﬁnd a decision

namely,
stump that minimizes LS(h). Furthermore, since in the next section we will
show that AdaBoost requires ﬁnding a hypothesis with a small risk relative to
some distribution over S, we will show here how to minimize such risk functions.
Concretely, let Dbe a probability vector in Rm(that is, all elements of Dare
nonnegative and∑
iDi= 1). The weak learner we describe later receives Dand
Sand outputs a decision stump h:X→Y that minimizes the risk w.r.t. D,
LD(h) =m∑
i=1Di1[h(xi)̸=yi].
Note that if D= (1/m,..., 1/m) thenLD(h) =LS(h).

LD(h) =m∑
Recall that each decision stump is parameterized by an index j∈[d] and a
thresholdθ. Therefore, minimizing LD(h) amounts to solving the problem
min
j∈[d]min
θ∈R
∑
i:yi=1Di1[xi,j>θ]+∑
i:yi=−1Di1[xi,j≤θ]
. (10.1)
Fixj∈[d] and let us sort the examples so that x1,j≤x2,j≤...≤xm,j. Deﬁne
Θj={xi,j+xi+1,j
2:i∈[m−1]}∪{(x1,j−1),(xm,j+ 1)}. Note that for any θ∈R
there exists θ′∈Θjthat yields the same predictions for the sample Sas the

134 Boosting
thresholdθ. Therefore, instead of minimizing over θ∈Rwe can minimize over
θ∈Θj.
This already gives us an eﬃcient procedure: Choose j∈[d] andθ∈Θjthat
minimize the objective value of Equation (10.1). For every jandθ∈Θjwe
have to calculate a sum over mexamples; therefore the runtime of this approach
would beO(dm2). We next show a simple trick that enables us to minimize the
objective in time O(dm).
The observation is as follows. Suppose we have calculated the objective for
θ∈(xi−1,j,xi,j). LetF(θ) be the value of the objective. Then, when we consider
θ′∈(xi,j,xi+1,j) we have that
F(θ′) =F(θ)−Di1[yi=1]+Di1[yi=−1]=F(θ)−yiDi.
Therefore, we can calculate the objective at θ′in a constant time, given the
objective at the previous threshold, θ. It follows that after a preprocessing step
in which we sort the examples with respect to each coordinate, the minimization
problem can be performed in time O(dm). This yields the following pseudocode.
ERM for Decision Stumps
input:

input:
training set S= (x1,y1),..., (xm,ym)
distribution vector D
goal: Findj⋆,θ⋆that solve Equation (10.1)
initialize: F⋆=∞
forj= 1,...,d
sortSusing thej’th coordinate, and denote
x1,j≤x2,j≤···≤xm,j≤xm+1,jdef=xm,j+ 1
F=∑
i:yi=1Di
ifF <F⋆
F⋆=F,θ⋆=x1,j−1,j⋆=j
fori= 1,...,m
F=F−yiDi
ifF <F⋆andxi,j̸=xi+1,j
F⋆=F,θ⋆=1
2(xi,j+xi+1,j),j⋆=j
outputj⋆,θ⋆
10.2 AdaBoost
AdaBoost (short for Adaptive Boosting) is an algorithm that has access to a
weak learner and ﬁnds a hypothesis with a low empirical risk. The AdaBoost

input:
F=∑
i:yi=1Di
ifF <F⋆
F=F−yiDi
F⋆=F,θ⋆=1
algorithm receives as input a training set of examples S= (x1,y1),..., (xm,ym),
where for each i,yi=f(xi) for some labeling function f. The boosting process
proceeds in a sequence of consecutive rounds. At round t, the booster ﬁrst deﬁnes

10.2 AdaBoost 135
a distribution over the examples in S, denoted D(t). That is, D(t)∈Rm
+and∑m
i=1D(t)
i= 1. Then, the booster passes the distribution D(t)and the sample S
to the weak learner. (That way, the weak learner can construct i.i.d. examples
according to D(t)andf.) The weak learner is assumed to return a “weak”
hypothesis, ht, whose error,
ϵtdef=LD(t)(ht)def=m∑
i=1D(t)
i1[ht(xi)̸=yi],
is at most1
2−γ(of course, there is a probability of at most δthat the weak learner
fails). Then, AdaBoost assigns a weight for htas follows: wt=1
2log(
1
ϵt−1)
.
That is, the weight of htis inversely proportional to the error of ht. At the end
of the round, AdaBoost updates the distribution so that examples on which ht
errs will get a higher probability mass while examples on which htis correct will
get a lower probability mass. Intuitively, this will force the weak learner to focus
on the problematic examples in the next round. The output of the AdaBoost

2log(
1
ϵt−1)
.
algorithm is a “strong” classiﬁer that is based on a weighted sum of all the weak
hypotheses. The pseudocode of AdaBoost is presented in the following.
AdaBoost
input:
weak learner WL
number of rounds T
initialize D(1)= (1
m,...,1
m).
fort= 1,...,T :
invoke weak learner ht= WL( D(t),S)
computeϵt=∑m
i=1D(t)
i1[yi̸=ht(xi)]
letwt=1
2log(
1
ϵt−1)
updateD(t+1)
i =D(t)
iexp(−wtyiht(xi))
∑m
j=1D(t)
jexp(−wtyjht(xj))for alli= 1,...,m
output the hypothesis hs(x) = sign(∑T
t=1wtht(x))
.

AdaBoost
input:
m,...,1
m).
i=1D(t)
letwt=1
2log(
1
ϵt−1)
i =D(t)
∑m
j=1D(t)
.
The following theorem shows that the training error of the output hypothesis
decreases exponentially fast with the number of boosting rounds.
theorem 10.2 LetSbe a training set and assume that at each iteration of
AdaBoost, the weak learner returns a hypothesis for which ϵt≤1/2−γ. Then,
the training error of the output hypothesis of AdaBoost is at most
LS(hs) =1
mm∑
i=11[hs(xi)̸=yi]≤exp(−2γ2T).
Proof For eacht, denoteft=∑
p≤twphp. Therefore, the output of AdaBoost

136 Boosting
isfT. In addition, denote
Zt=1
mm∑
i=1e−yift(xi).
Note that for any hypothesis we have that 1[h(x)̸=y]≤e−yh(x). Therefore, LS(fT)≤
ZT, so it suﬃces to show that ZT≤e−2γ2T. To upper bound ZTwe rewrite it
as
ZT=ZT
Z0=ZT
ZT−1·ZT−1
ZT−2···Z2
Z1·Z1
Z0, (10.2)
where we used the fact that Z0= 1 because f0≡0. Therefore, it suﬃces to show
that for every round t,
Zt+1
Zt≤e−2γ2. (10.3)
To do so, we ﬁrst note that using a simple inductive argument, for all tandi,
D(t+1)
i =e−yift(xi)
∑m
j=1e−yjft(xj).
Hence,
Zt+1
Zt=∑m
i=1e−yift+1(xi)
m∑
j=1e−yjft(xj)
=∑m
i=1e−yift(xi)e−yiwt+1ht+1(xi)
m∑
=m∑
i=1D(t+1)
ie−yiwt+1ht+1(xi)
=e−wt+1∑
i:yiht+1(xi)=1D(t+1)
i +ewt+1∑
i:yiht+1(xi)=−1D(t+1)
i
=e−wt+1(1−ϵt+1) +ewt+1ϵt+1
=1√
1/ϵt+1−1(1−ϵt+1) +√
1/ϵt+1−1ϵt+1
=√ϵt+1
1−ϵt+1(1−ϵt+1) +√
1−ϵt+1
ϵt+1ϵt+1
= 2√
ϵt+1(1−ϵt+1).
By our assumption, ϵt+1≤1
2−γ. Since the function g(a) =a(1−a) is mono-
tonically increasing in [0 ,1/2], we obtain that
2√
ϵt+1(1−ϵt+1)≤2√(1
2−γ)(1
2+γ)
=√
1−4γ2.

Finally, using the inequality 1 −a≤e−awe have that√
1−4γ2≤e−4γ2/2=
e−2γ2. This shows that Equation (10.3) holds and thus concludes our proof.
Each iteration of AdaBoost involves O(m) operations as well as a single call to
the weak learner. Therefore, if the weak learner can be implemented eﬃciently
(as happens in the case of ERM with respect to decision stumps) then the total
training process will be eﬃcient.
Remark 10.2 Theorem 10.2 assumes that at each iteration of AdaBoost, the
weak learner returns a hypothesis with weighted sample error of at most 1 /2−γ.
According to the deﬁnition of a weak learner, it can fail with probability δ. Using
the union bound, the probability that the weak learner will not fail at all of the
iterations is at least 1 −δT. As we show in Exercise 1, the dependence of the
sample complexity on δcan always be logarithmic in 1 /δ, and therefore invoking
the weak learner with a very small δis not problematic. We can therefore assume

thatδTis also small. Furthermore, since the weak learner is only applied with
distributions over the training set, in many cases we can implement the weak
learner so that it will have a zero probability of failure (i.e., δ= 0). This is the
case, for example, in the weak learner that ﬁnds the minimum value of LD(h)
for decision stumps, as described in the previous section.
Theorem 10.2 tells us that the empirical risk of the hypothesis constructed by
AdaBoost goes to zero as Tgrows. However, what we really care about is the

true risk of the output hypothesis. To argue about the true risk, we note that the
output of AdaBoost is in fact a composition of a halfspace over the predictions
of theTweak hypotheses constructed by the weak learner. In the next section
we show that if the weak hypotheses come from a base hypothesis class of low
VC-dimension, then the estimation error of AdaBoost will be small; namely, the
true risk of the output of AdaBoost would not be very far from its empirical risk.
10.3 Linear Combinations of Base Hypotheses

As mentioned previously, a popular approach for constructing a weak learner
is to apply the ERM rule with respect to a base hypothesis class (e.g., ERM
over decision stumps). We have also seen that boosting outputs a composition
of a halfspace over the predictions of the weak hypotheses. Therefore, given a
base hypothesis class B(e.g., decision stumps), the output of AdaBoost will be
a member of the following class:
L(B,T) ={
x↦→sign(T∑
:w∈RT,∀t, ht∈B}
. (10.4)
That is, each h∈L(B,T) is parameterized by Tbase hypotheses from Band

L(B,T) ={
. (10.4)
by a vector w∈RT. The prediction of such an hon an instance xis ob-
tained by ﬁrst applying the Tbase hypotheses to construct the vector ψ(x) =

138 Boosting
(h1(x),...,hT(x))∈RT, and then applying the (homogenous) halfspace deﬁned
bywonψ(x).
In this section we analyze the estimation error of L(B,T) by bounding the
VC-dimension of L(B,T) in terms of the VC-dimension of BandT. We will
show that, up to logarithmic factors, the VC-dimension of L(B,T) is bounded
byTtimes the VC-dimension of B. It follows that the estimation error of Ad-
aBoost grows linearly with T. On the other hand, the empirical risk of AdaBoost
decreases with T. In fact, as we demonstrate later, Tcan be used to decrease
the approximation error of L(B,T). Therefore, the parameter Tof AdaBoost
enables us to control the bias-complexity tradeoﬀ.
To demonstrate how the expressive power of L(B,T) increases with T, consider
the simple example, in which X=Rand the base class is Decision Stumps,
HDS1={x↦→sign(x−θ)·b:θ∈R,b∈{± 1}}.
Note that in this one dimensional case, HDS1is in fact equivalent to (nonho-
mogenous) halfspaces on R.

Now, letHbe the rather complex class (compared to halfspaces on the line)
of piece-wise constant functions. Let grbe a piece-wise constant function with at
mostrpieces; that is, there exist thresholds −∞=θ0<θ1<θ2<···<θr=∞
such that
gr(x) =r∑
i=1αi1[x∈(θi−1,θi]]∀i, αi∈{± 1}.
Denote byGrthe class of all such piece-wise constant classiﬁers with at most r
pieces.
In the following we show that GT⊆L(HDS1,T); namely, the class of halfspaces
overTdecision stumps yields all the piece-wise constant classiﬁers with at most
Tpieces.

such that
gr(x) =r∑
pieces.
Tpieces.
Indeed, without loss of generality consider any g∈GTwithαt= (−1)t. This
implies that if xis in the interval ( θt−1,θt], theng(x) = (−1)t. For example:
Now, the function
h(x) = sign(T∑
t=1wtsign(x−θt−1))
, (10.5)
wherew1= 0.5 and fort >1,wt= (−1)t, is inL(HDS1,T) and is equal to g
(see Exercise 2).

10.3 Linear Combinations of Base Hypotheses 139
From this example we obtain that L(HDS1,T) can shatter any set of T+ 1
instances in R; hence the VC-dimension of L(HDS1,T) is at least T+1. Therefore,
Tis a parameter that can control the bias-complexity tradeoﬀ: Enlarging T
yields a more expressive hypothesis class but on the other hand might increase
the estimation error. In the next subsection we formally upper bound the VC-
dimension of L(B,T) for any base class B.
10.3.1 The VC-Dimension of L(B,T)
The following lemma tells us that the VC-dimension of L(B,T) is upper bounded
by˜O(VCdim(B)T) (the ˜Onotation ignores constants and logarithmic factors).
lemma 10.3 LetBbe a base class and let L(B,T)be as deﬁned in Equa-
tion (10.4). Assume that both TandVCdim(B)are at least 3. Then,
VCdim(L(B,T))≤T(VCdim(B) + 1) (3 log( T(VCdim(B) + 1)) + 2) .
Proof Denoted= VCdim(B). LetC={x1,...,xm}be a set that is shat-
tered byL(B,T). Each labeling of Cbyh∈L(B,T) is obtained by ﬁrst choos-

ingh1,...,hT∈Band then applying a halfspace hypothesis over the vector
(h1(x),...,hT(x)). By Sauer’s lemma, there are at most ( em/d )ddiﬀerent di-
chotomies (i.e., labelings) induced by BoverC. Therefore, we need to choose
Thypotheses, out of at most ( em/d )ddiﬀerent hypotheses. There are at most
(em/d )dTways to do it. Next, for each such choice, we apply a linear predictor,
which yields at most ( em/T )Tdichotomies. Therefore, the overall number of
dichotomies we can construct is upper bounded by
(em/d )dT(em/T )T≤m(d+1)T,

where we used the assumption that both dandTare at least 3. Since we assume
thatCis shattered, we must have that the preceding is at least 2m, which yields
2m≤m(d+1)T.
Therefore,
m≤log(m)(d+ 1)T
log(2).
Lemma A.1 in Chapter A tells us that a necessary condition for the above to
hold is that
m≤2(d+ 1)T
log(2)log(d+ 1)T
log(2)≤(d+ 1)T(3 log((d+ 1)T) + 2),
In Exercise 4 we show that for some base classes, B, it also holds that VCdim( L(B,T))≥
Ω(VCdim(B)T).

140 Boosting
A B
C D
Figure 10.1 The four types of functions, g, used by the base hypotheses for face
recognition. The value of gfor typeAorBis the diﬀerence between the sum of the
pixels within two rectangular regions. These regions have the same size and shape and
are horizontally or vertically adjacent. For type C, the value of gis the sum within
two outside rectangles subtracted from the sum in a center rectangle. For type D, we
compute the diﬀerence between diagonal pairs of rectangles.
10.4 AdaBoost for Face Recognition
We now turn to a base hypothesis that has been proposed by Viola and Jones for
the task of face recognition. In this task, the instance space is images, represented
as matrices of gray level values of pixels. To be concrete, let us take images of
size 24×24 pixels, and therefore our instance space is the set of real valued
matrices of size 24 ×24. The goal is to learn a classiﬁer, h:X →{± 1}, that
given an image as input, should output whether the image is of a human face or
not.

not.
Each hypothesis in the base class is of the form h(x) =f(g(x)), wherefis a
decision stump hypothesis and g:R24,24→Ris a function that maps an image
to a scalar. Each function gis parameterized by
•An axis aligned rectangle R. Since each image is of size 24 ×24, there are at
most 244axis aligned rectangles.
•A type,t∈{A,B,C,D}. Each type corresponds to a mask, as depicted in
Figure 10.1.
To calculate gwe stretch the mask tto ﬁt the rectangle Rand then calculate

not.
the sum of the pixels (that is, sum of their gray level values) that lie within the
red rectangles and subtract it from the sum of pixels in the blue rectangles.
Since the number of such functions gis at most 244·4, we can implement a
weak learner for the base hypothesis class by ﬁrst calculating all the possible
outputs ofgon each image, and then apply the weak learner of decision stumps
described in the previous subsection. It is possible to perform the ﬁrst step very

Figure 5:The ﬁrst andsecond features selected byAdaBoost. The twofeatures areshowninthetoprow
andthen overlayed onatypical training faceinthebottom row.Theﬁrstfeature measures thedifference in
intensity between theregion oftheeyesandaregion across theupper cheeks. Thefeature capitalizes onthe
observ ation thattheeyeregion isoften darkerthan thecheeks. Thesecond feature compares theintensities
intheeyeregions totheintensity across thebridge ofthenose.
directly increases computation time.
4The Attentional Cascade
This section describes analgorithm forconstructing acascade ofclassiﬁers which achie vesincreased detec-
tion performance while radically reducing computation time. The keyinsight isthatsmaller ,andtherefore
more efﬁcient, boosted classiﬁers canbeconstructed which reject manyofthenegativesub-windo wswhile
detecting almost allpositi veinstances. Simpler classiﬁers areused toreject themajority ofsub-windo ws
before more comple xclassiﬁers arecalled upon toachie velowfalsepositi verates.

Stages inthecascade areconstructed bytraining classiﬁers using AdaBoost. Starting with atwo-feature
strong classiﬁer ,aneffectivefaceﬁlter canbeobtained byadjusting thestrong classiﬁer threshold tomin-
imize false negatives.The initial AdaBoost threshold,
 ,isdesigned toyield alowerror rateon
thetraining data. Alowerthreshold yields higher detection rates andhigher false positi verates. Based on
performance measured using avalidation training set,thetwo-feature classiﬁer canbeadjusted todetect

100% ofthefaces with afalse positi verateof40%. SeeFigure 5foradescription ofthetwofeatures used
inthisclassiﬁer .
The detection performance ofthetwo-feature classiﬁer isfarfrom acceptable asanobject detection
system. Nevertheless theclassiﬁer cansigniﬁcantly reduce thenumber sub-windo wsthatneed further pro-
cessing with veryfewoperations:
1.Evaluate therectangle features (requires between 6and9array references perfeature).
2.Compute theweak classiﬁer foreach feature (requires onethreshold operation perfeature).
11

11
Figure 10.2 The ﬁrst and second features selected by AdaBoost, as implemented by
Viola and Jones. The two features are shown in the top row and then overlaid on a
typical training face in the bottom row. The ﬁrst feature measures the diﬀerence in
intensity between the region of the eyes and a region across the upper cheeks. The
feature capitalizes on the observation that the eye region is often darker than the
cheeks. The second feature compares the intensities in the eye regions to the intensity
across the bridge of the nose.

eﬃciently by a preprocessing step in which we calculate the integral image of
each image in the training set. See Exercise 5 for details.
In Figure 10.2 we depict the ﬁrst two features selected by AdaBoost when
running it with the base features proposed by Viola and Jones.
10.5 Summary
Boosting is a method for amplifying the accuracy of weak learners. In this chapter
we described the AdaBoost algorithm. We have shown that after Titerations of
AdaBoost, it returns a hypothesis from the class L(B,T), obtained by composing

a linear classiﬁer on Thypotheses from a base class B. We have demonstrated
how the parameter Tcontrols the tradeoﬀ between approximation and estimation
errors. In the next chapter we will study how to tune parameters such as T, based
on the data.
10.6 Bibliographic Remarks
As mentioned before, boosting stemmed from the theoretical question of whether
an eﬃcient weak learner can be “boosted” into an eﬃcient strong learner (Kearns
& Valiant 1988) and solved by Schapire (1990). The AdaBoost algorithm has
been proposed in Freund & Schapire (1995).

Boosting can be viewed from many perspectives. In the purely theoretical
context, AdaBoost can be interpreted as a negative result: If strong learning of
a hypothesis class is computationally hard, so is weak learning of this class. This
negative result can be useful for showing hardness of agnostic PAC learning of
a classBbased on hardness of PAC learning of some other class H, as long as

142 Boosting
His weakly learnable using B. For example, Klivans & Sherstov (2006) have
shown that PAC learning of the class of intersection of halfspaces is hard (even
in the realizable case). This hardness result can be used to show that agnostic
PAC learning of a single halfspace is also computationally hard (Shalev-Shwartz,
Shamir & Sridharan 2010). The idea is to show that an agnostic PAC learner
for a single halfspace can yield a weak learner for the class of intersection of
halfspaces, and since such a weak learner can be boosted, we will obtain a strong
learner for the class of intersection of halfspaces.
AdaBoost also shows an equivalence between the existence of a weak learner
and separability of the data using a linear classiﬁer over the predictions of base
hypotheses. This result is closely related to von Neumann’s minimax theorem
(von Neumann 1928), a fundamental result in game theory.
AdaBoost is also related to the concept of margin, which we will study later on

in Chapter 15. It can also be viewed as a forward greedy selection algorithm, a
topic that will be presented in Chapter 25. A recent book by Schapire & Freund
(2012) covers boosting from all points of view, and gives easy access to the wealth
of research that this ﬁeld has produced.
10.7 Exercises
1.Boosting the Conﬁdence: LetAbe an algorithm that guarantees the fol-
lowing: There exist some constant δ0∈(0,1) and a function mH: (0,1)→N
such that for every ϵ∈(0,1), ifm≥mH(ϵ) then for every distribution Dit

holds that with probability of at least 1 −δ0,LD(A(S))≤minh∈HLD(h) +ϵ.
Suggest a procedure that relies on Aand learnsHin the usual agnostic
PAC learning model and has a sample complexity of
mH(ϵ,δ)≤kmH(ϵ) +⌈2 log(4k/δ)
ϵ2⌉
,
where
k=⌈log(δ)/log(δ0)⌉.
Hint: Divide the data into k+ 1 chunks, where each of the ﬁrst kchunks
is of sizemH(ϵ) examples. Train the ﬁrst kchunks using A. Argue that the
probability that for all of these chunks we have LD(A(S))>minh∈HLD(h)+ϵ
is at mostδk

ϵ2⌉
,
where
0≤δ/2. Finally, use the last chunk to choose from the khypotheses
thatAgenerated from the kchunks (by relying on Corollary 4.6).
2. Prove that the function hgiven in Equation (10.5) equals the piece-wise con-
stant function deﬁned according to the same thresholds as h.
3. We have informally argued that the AdaBoost algorithm uses the weighting
mechanism to “force” the weak learner to focus on the problematic examples
in the next iteration. In this question we will ﬁnd some rigorous justiﬁcation
for this argument.

10.7 Exercises 143
Show that the error of htw.r.t. the distribution D(t+1)is exactly 1/2. That
is, show that for every t∈[T]
m∑
i=1D(t+1)
i1[yi̸=ht(xi)]= 1/2.
4. In this exercise we discuss the VC-dimension of classes of the form L(B,T).
We proved an upper bound of O(dTlog(dT)), whered= VCdim(B). Here we
wish to prove an almost matching lower bound. However, that will not be the
case for all classes B.
1. Note that for every class Band every number T≥1, VCdim(B)≤
VCdim(L(B,T)). Find a class Bfor which VCdim( B) = VCdim( L(B,T))
for everyT≥1.
Hint: TakeXto be a ﬁnite set.
2. LetBdbe the class of decision stumps over Rd. Prove that log( d)≤
VCdim(Bd)≤5 + 2 log(d).
Hints:
•For the upper bound, rely on Exercise 11.
•For the lower bound, assume d= 2k. LetAbe ak×dmatrix whose
columns are all the dbinary vectors in {±1}k. The rows of Aform
a set ofkvectors inRd. Show that this set is shattered by decision
stumps over Rd.
3. LetT≥1 be any integer. Prove that VCdim( L(Bd,T))≥0.5Tlog(d).
Hint: Construct a set ofT

Hints:
2kinstances by taking the rows of the matrix A
from the previous question, and the rows of the matrices 2 A,3A,4A,...,T
2A.
Show that the resulting set is shattered by L(Bd,T).
5.Eﬃciently Calculating the Viola and Jones Features Using an Inte-
gral Image: LetAbe a 24×24 matrix representing an image. The integral
image ofA, denoted by I(A), is the matrix Bsuch thatBi,j=∑
i′≤i,j′≤jAi,j.
•Show thatI(A) can be calculated from Ain time linear in the size of A.
•Show how every Viola and Jones feature can be calculated from I(A) in a

2A.
constant amount of time (that is, the runtime does not depend on the
size of the rectangle deﬁning the feature).

11 Model Selection and Validation
In the previous chapter we have described the AdaBoost algorithm and have
shown how the parameter Tof AdaBoost controls the bias-complexity trade-
oﬀ. But, how do we set Tin practice? More generally, when approaching some
practical problem, we usually can think of several algorithms that may yield a
good solution, each of which might have several parameters. How can we choose
the best algorithm for the particular problem at hand? And how do we set the
algorithm’s parameters? This task is often called model selection .
To illustrate the model selection task, consider the problem of learning a one
dimensional regression function, h:R→R. Suppose that we obtain a training
set as depicted in the ﬁgure.
We can consider ﬁtting a polynomial to the data, as described in Chapter 9.
However, we might be uncertain regarding which degree dwould give the best
results for our data set: A small degree may not ﬁt the data well (i.e., it will

have a large approximation error), whereas a high degree may lead to overﬁtting
(i.e., it will have a large estimation error). In the following we depict the result
of ﬁtting a polynomial of degrees 2, 3, and 10. It is easy to see that the empirical
risk decreases as we enlarge the degree. However, looking at the graphs, our
intuition tells us that setting the degree to 3 may be better than setting it to 10.
It follows that the empirical risk alone is not enough for model selection.

In this chapter we will present two approaches for model selection. The ﬁrst
approach is based on the Structural Risk Minimization (SRM) paradigm we
have described and analyzed in Chapter 7.2. SRM is particularly useful when
a learning algorithm depends on a parameter that controls the bias-complexity
tradeoﬀ (such as the degree of the ﬁtted polynomial in the preceding example
or the parameter Tin AdaBoost). The second approach relies on the concept
ofvalidation . The basic idea is to partition the training set into two sets. One
is used for training each of the candidate models, and the second is used for
deciding which of them yields the best results.
In model selection tasks, we try to ﬁnd the right balance between approxi-
mation and estimation errors. More generally, if our learning algorithm fails to
ﬁnd a predictor with a small risk, it is important to understand whether we
suﬀer from overﬁtting or underﬁtting. In Section 11.3 we discuss how this can
be achieved.

11.1 Model Selection Using SRM
The SRM paradigm has been described and analyzed in Section 7.2. Here we
show how SRM can be used for tuning the tradeoﬀ between bias and complexity
without deciding on a speciﬁc hypothesis class in advance. Consider a countable
sequence of hypothesis classes H1,H2,H3,.... For example, in the problem of
polynomial regression mentioned, we can take Hdto be the set of polynomials
of degree at most d. Another example is taking Hdto be the class L(B,d) used
by AdaBoost, as described in the previous chapter.

We assume that for every d, the classHdenjoys the uniform convergence
property (see Deﬁnition 4.3 in Chapter 4) with a sample complexity function of
the form
mUC
Hd(ϵ,δ)≤g(d) log(1/δ)
ϵ2, (11.1)
whereg:N→Ris some monotonically increasing function. For example, in the
case of binary classiﬁcation problems, we can take g(d) to be the VC-dimension
of the classHdmultiplied by a universal constant (the one appearing in the
fundamental theorem of learning; see Theorem 6.8). For the classes L(B,d) used

the form
mUC
ϵ2, (11.1)
by AdaBoost, the function gwill simply grow with d.
Recall that the SRM rule follows a “bound minimization” approach, where in
our case the bound is as follows: With probability of at least 1 −δ, for every
d∈Nandh∈Hd,
g(d)(log(1/δ) + 2 log(d) + log(π2/6))
m. (11.2)
This bound, which follows directly from Theorem 7.4, shows that for every dand
everyh∈Hd, the true risk is bounded by two terms – the empirical risk, LS(h),

146 Model Selection and Validation
and a complexity term that depends on d. The SRM rule will search for dand
h∈Hdthat minimize the right-hand side of Equation (11.2).
Getting back to the example of polynomial regression described earlier, even
though the empirical risk of the 10th degree polynomial is smaller than that of
the 3rd degree polynomial, we would still prefer the 3rd degree polynomial since
its complexity (as reﬂected by the value of the function g(d)) is much smaller.
While the SRM approach can be useful in some situations, in many practical
cases the upper bound given in Equation (11.2) is pessimistic. In the next section
we present a more practical approach.
11.2 Validation
We would often like to get a better estimation of the true risk of the output pre-
dictor of a learning algorithm. So far we have derived bounds on the estimation
error of a hypothesis class, which tell us that for allhypotheses in the class, the

true risk is not very far from the empirical risk. However, these bounds might be
loose and pessimistic, as they hold for all hypotheses and all possible data dis-
tributions. A more accurate estimation of the true risk can be obtained by using
some of the training data as a validation set, over which one can evalutate the
success of the algorithm’s output predictor. This procedure is called validation .
Naturally, a better estimation of the true risk is useful for model selection, as
we will describe in Section 11.2.2.
11.2.1 Hold Out Set

The simplest way to estimate the true error of a predictor his by sampling an ad-
ditional set of examples, independent of the training set, and using the empirical
error on this validation set as our estimator. Formally, let V= (x1,y1),..., (xmv,ymv)
be a set of fresh mvexamples that are sampled according to D(independently of
themexamples of the training set S). Using Hoeﬀding’s inequality ( Lemma 4.5)
we have the following:
theorem 11.1 Lethbe some predictor and assume that the loss function is in

[0,1]. Then, for every δ∈(0,1), with probability of at least 1−δover the choice
of a validation set Vof sizemvwe have
|LV(h)−LD(h)| ≤√
log(2/δ)
2mv.
The bound in Theorem 11.1 does not depend on the algorithm or the training
set used to construct hand is tighter than the usual bounds that we have seen so
far. The reason for the tightness of this bound is that it is in terms of an estimate
on a fresh validation set that is independent of the way hwas generated. To

log(2/δ)
2mv.
illustrate this point, suppose that hwas obtained by applying an ERM predictor

11.2 Validation 147
with respect to a hypothesis class of VC-dimension d, over a training set of m
examples. Then, from the fundamental theorem of learning (Theorem 6.8) we
obtain the bound
Cd+ log(1/δ)
m,
whereCis the constant appearing in Theorem 6.8. In contrast, from Theo-
rem 11.1 we obtain the bound
LD(h)≤LV(h) +√
log(2/δ)
2mv.
Therefore, taking mvto be order of m, we obtain an estimate that is more
accurate by a factor that depends on the VC-dimension. On the other hand, the
price we pay for using such an estimate is that it requires an additional sample
on top of the sample used for training the learner.
Sampling a training set and then sampling an independent validation set is
equivalent to randomly partitioning our random set of examples into two parts,
using one part for training and the other one for validation. For this reason, the
validation set is often referred to as a hold out set.
11.2.2 Validation for Model Selection

Validation can be naturally used for model selection as follows. We ﬁrst train
diﬀerent algorithms (or the same algorithm with diﬀerent parameters) on the
given training set. Let H={h1,...,hr}be the set of all output predictors of the
diﬀerent algorithms. For example, in the case of training polynomial regressors,
we would have each hrbe the output of polynomial regression of degree r. Now,
to choose a single predictor from Hwe sample a fresh validation set and choose
the predictor that minimizes the error over the validation set. In other words,

we apply ERM Hover the validation set.
This process is very similar to learning a ﬁnite hypothesis class. The only
diﬀerence is that His not ﬁxed ahead of time but rather depends on the train-
ing set. However, since the validation set is independent of the training set we
get that it is also independent of Hand therefore the same technique we used
to derive bounds for ﬁnite hypothesis classes holds here as well. In particular,
combining Theorem 11.1 with the union bound we obtain:

theorem 11.2 LetH={h1,...,hr}be an arbitrary set of predictors and
assume that the loss function is in [0,1]. Assume that a validation set Vof size
mvis sampled independent of H. Then, with probability of at least 1−δover the
choice ofVwe have
∀h∈H,|LD(h)−LV(h)| ≤√
log(2|H|/δ)
2mv.

148 Model Selection and Validation
This theorem tells us that the error on the validation set approximates the
true error as long as His not too large. However, if we try too many methods
(resulting in|H|that is large relative to the size of the validation set) then we’re
in danger of overﬁtting.
To illustrate how validation is useful for model selection, consider again the
example of ﬁtting a one dimensional polynomial as described in the beginning
of this chapter. In the following we depict the same training set, with ERM
polynomials of degree 2, 3, and 10, but this time we also depict an additional
validation set (marked as red, unﬁlled circles). The polynomial of degree 10 has
minimal training error, yet the polynomial of degree 3 has the minimal validation
error, and hence it will be chosen as the best model.
11.2.3 The Model-Selection Curve
The model selection curve shows the training error and validation error as a func-
tion of the complexity of the model considered. For example, for the polynomial

ﬁtting problem mentioned previously, the curve will look like:

11.2 Validation 149
2 4 6 8 1000.10.20.30.4
derrortrain
As can be shown, the training error is monotonically decreasing as we increase
the polynomial degree (which is the complexity of the model in our case). On
the other hand, the validation error ﬁrst decreases but then starts to increase,
which indicates that we are starting to suﬀer from overﬁtting.
Plotting such curves can help us understand whether we are searching the
correct regime of our parameter space. Often, there may be more than a single
parameter to tune, and the possible number of values each parameter can take
might be quite large. For example, in Chapter 13 we describe the concept of
regularization , in which the parameter of the learning algorithm is a real number.
In such cases, we start with a rough grid of values for the parameter(s) and plot
the corresponding model-selection curve. On the basis of the curve we will zoom
in to the correct regime and employ a ﬁner grid to search over. It is important to

verify that we are in the relevant regime. For example, in the polynomial ﬁtting
problem described, if we start searching degrees from the set of values {1,10,20}
and do not employ a ﬁner grid based on the resulting curve, we will end up with
a rather poor model.
11.2.4k-Fold Cross Validation
The validation procedure described so far assumes that data is plentiful and that
we have the ability to sample a fresh validation set. But in some applications,
data is scarce and we do not want to “waste” data on validation. The k-fold

cross validation technique is designed to give an accurate estimate of the true
error without wasting too much data.
Ink-fold cross validation the original training set is partitioned into ksubsets
(folds) of size m/k (for simplicity, assume that m/k is an integer). For each fold,
the algorithm is trained on the union of the other folds and then the error of its
output is estimated using the fold. Finally, the average of all these errors is the

150 Model Selection and Validation
estimate of the true error. The special case k=m, wheremis the number of
examples, is called leave-one-out (LOO).
k-Fold cross validation is often used for model selection (or parameter tuning),
and once the best parameter is chosen, the algorithm is retrained using this
parameter on the entire training set. A pseudocode of k-fold cross validation
for model selection is given in the following. The procedure receives as input a
training set, S, a set of possible parameter values, Θ, an integer, k, representing
the number of folds, and a learning algorithm, A, which receives as input a
training set as well as a parameter θ∈Θ. It outputs the best parameter as well
as the hypothesis trained by this parameter on the entire training set.
k-Fold Cross Validation for Model Selection
input:
set of parameter values Θ
learning algorithm A
integerk
partitionSintoS1,S2,...,Sk
foreachθ∈Θ
fori= 1...k
hi,θ=A(S\Si;θ)
error(θ) =1
k∑k
i=1LSi(hi,θ)
output

input:
integerk
k∑k
output
θ⋆= argminθ[error(θ)]
hθ⋆=A(S;θ⋆)
The cross validation method often works very well in practice. However, it
might sometime fail, as the artiﬁcial example given in Exercise 1 shows. Rig-
orously understanding the exact behavior of cross validation is still an open
problem. Rogers and Wagner (Rogers & Wagner 1978) have shown that for k
local rules (e.g., kNearest Neighbor; see Chapter 19) the cross validation proce-
dure gives a very good estimate of the true error. Other papers show that cross

output
validation works for stable algorithms (we will study stability and its relation to
learnability in Chapter 13).
11.2.5 Train-Validation-Test Split
In most practical applications, we split the available examples into three sets.
The ﬁrst set is used for training our algorithm and the second is used as a
validation set for model selection. After we select the best model, we test the
performance of the output predictor on the third set, which is often called the

“test set.” The number obtained is used as an estimator of the true error of the
learned predictor.

11.3 What to Do If Learning Fails
Consider the following scenario: You were given a learning task and have ap-
proached it with a choice of a hypothesis class, a learning algorithm, and param-
eters. You used a validation set to tune the parameters and tested the learned
predictor on a test set. The test results, unfortunately, turn out to be unsatis-
factory. What went wrong then, and what should you do next?
There are many elements that can be “ﬁxed.” The main approaches are listed
in the following:
•Get a larger sample
•Change the hypothesis class by:
–Enlarging it
–Reducing it
–Completely changing it
–Changing the parameters you consider
•Change the feature representation of the data
•Change the optimization algorithm used to apply your learning rule
In order to ﬁnd the best remedy, it is essential ﬁrst to understand the cause
of the bad performance. Recall that in Chapter 5 we decomposed the true er-

ror of the learned predictor into approximation error and estimation error. The
approximation error is deﬁned to be LD(h⋆) for some h⋆∈argminh∈HLD(h),
while the estimation error is deﬁned to be LD(hS)−LD(h⋆), wherehSis the
learned predictor (which is based on the training set S).
The approximation error of the class does not depend on the sample size or
on the algorithm being used. It only depends on the distribution Dand on the
hypothesis classH. Therefore, if the approximation error is large, it will not help

us to enlarge the training set size, and it also does not make sense to reduce the
hypothesis class. What can be beneﬁcial in this case is to enlarge the hypothesis
class or completely change it (if we have some alternative prior knowledge in
the form of a diﬀerent hypothesis class). We can also consider applying the
same hypothesis class but on a diﬀerent feature representation of the data (see
Chapter 25).
The estimation error of the class does depend on the sample size. Therefore, if
we have a large estimation error we can make an eﬀort to obtain more training

examples. We can also consider reducing the hypothesis class. However, it doesn’t
make sense to enlarge the hypothesis class in that case.
Error Decomposition Using Validation
We see that understanding whether our problem is due to approximation error
or estimation error is very useful for ﬁnding the best remedy. In the previous
section we saw how to estimate LD(hS) using the empirical risk on a validation
set. However, it is more diﬃcult to estimate the approximation error of the class.

152 Model Selection and Validation
Instead, we give a diﬀerent error decomposition, one that can be estimated from
the train and validation sets.
LD(hS) = (LD(hS)−LV(hS)) + (LV(hS)−LS(hS)) +LS(hS).
The ﬁrst term, ( LD(hS)−LV(hS)), can be bounded quite tightly using Theo-
rem 11.1. Intuitively, when the second term, ( LV(hS)−LS(hS)), is large we say
that our algorithm suﬀers from “overﬁtting” while when the empirical risk term,
LS(hS), is large we say that our algorithm suﬀers from “underﬁtting.” Note that
these two terms are not necessarily good estimates of the estimation and ap-
proximation errors. To illustrate this, consider the case in which His a class of
VC-dimension d, andDis a distribution such that the approximation error of H
with respect toDis 1/4. As long as the size of our training set is smaller than
dwe will have LS(hS) = 0 for every ERM hypothesis. Therefore, the training
risk,LS(hS), and the approximation error, LD(h⋆), can be signiﬁcantly diﬀerent.

Nevertheless, as we show later, the values of LS(hS) and (LV(hS)−LS(hS)) still
provide us useful information.
Consider ﬁrst the case in which LS(hS) is large. We can write
LS(hS) = (LS(hS)−LS(h⋆)) + (LS(h⋆)−LD(h⋆)) +LD(h⋆).
WhenhSis an ERMHhypothesis we have that LS(hS)−LS(h⋆)≤0. In addition,
sinceh⋆does not depend on S, the term (LS(h⋆)−LD(h⋆)) can be bounded quite
tightly (as in Theorem 11.1). The last term is the approximation error. It follows
that ifLS(hS) is large then so is the approximation error, and the remedy to the

failure of our algorithm should be tailored accordingly (as discussed previously).
Remark 11.1 It is possible that the approximation error of our class is small,
yet the value of LS(hS) is large. For example, maybe we had a bug in our ERM
implementation, and the algorithm returns a hypothesis hSthat is not an ERM.
It may also be the case that ﬁnding an ERM hypothesis is computationally hard,
and our algorithm applies some heuristic trying to ﬁnd an approximate ERM. In
some cases, it is hard to know how good hSis relative to an ERM hypothesis. But,

sometimes it is possible at least to know whether there are better hypotheses.
For example, in the next chapter we will study convex learning problems in
which there are optimality conditions that can be checked to verify whether
our optimization algorithm converged to an ERM solution. In other cases, the
solution may depend on randomness in initializing the algorithm, so we can try
diﬀerent randomly selected initial points to see whether better solutions pop out.
Next consider the case in which LS(hS) is small. As we argued before, this

does not necessarily imply that the approximation error is small. Indeed, consider
two scenarios, in both of which we are trying to learn a hypothesis class of
VC-dimension dusing the ERM learning rule. In the ﬁrst scenario, we have a
training set of m<d examples and the approximation error of the class is high.
In the second scenario, we have a training set of m > 2dexamples and the

11.3 What to Do If Learning Fails 153
merror
train errorvalidation error
merror
Figure 11.1 Examples of learning curves. Left: This learning curve corresponds to the
scenario in which the number of examples is always smaller than the VC dimension of
the class. Right: This learning curve corresponds to the scenario in which the
approximation error is zero and the number of examples is larger than the VC
dimension of the class.
approximation error of the class is zero. In both cases LS(hS) = 0. How can we
distinguish between the two cases?
One possible way to distinguish between the two cases is by plotting learning
curves . To produce a learning curve we train the algorithm on preﬁxes of the
data of increasing sizes. For example, we can ﬁrst train the algorithm on the
ﬁrst 10% of the examples, then on 20% of them, and so on. For each preﬁx we
calculate the training error (on the preﬁx the algorithm is being trained on)

and the validation error (on a predeﬁned validation set). Such learning curves
can help us distinguish between the two aforementioned scenarios. In the ﬁrst
scenario we expect the validation error to be approximately 1 /2 for all preﬁxes,
as we didn’t really learn anything. In the second scenario the validation error
will start as a constant but then should start decreasing (it must start decreasing
once the training set size is larger than the VC-dimension). An illustration of
the two cases is given in Figure 11.1.

In general, as long as the approximation error is greater than zero we expect
the training error to grow with the sample size, as a larger amount of data points
makes it harder to provide an explanation for all of them. On the other hand,
the validation error tends to decrease with the increase in sample size. If the
VC-dimension is ﬁnite, when the sample size goes to inﬁnity, the validation and
train errors converge to the approximation error. Therefore, by extrapolating
the training and validation curves we can try to guess the value of the approx-

imation error, or at least to get a rough estimate on an interval in which the
approximation error resides.
Getting back to the problem of ﬁnding the best remedy for the failure of
our algorithm, if we observe that LS(hS) is small while the validation error is
large, then in any case we know that the size of our training set is not suﬃcient
for learning the class H. We can then plot a learning curve. If we see that the

154 Model Selection and Validation
validation error is starting to decrease then the best solution is to increase the
number of examples (if we can aﬀord to enlarge the data). Another reasonable
solution is to decrease the complexity of the hypothesis class. On the other hand,
if we see that the validation error is kept around 1 /2 then we have no evidence
that the approximation error of His good. It may be the case that increasing
the training set size will not help us at all. Obtaining more data can still help
us, as at some point we can see whether the validation error starts to decrease
or whether the training error starts to increase. But, if more data is expensive,
it may be better ﬁrst to try to reduce the complexity of the hypothesis class.
To summarize the discussion, the following steps should be applied:
1. If learning involves parameter tuning, plot the model-selection curve to make
sure that you tuned the parameters appropriately (see Section 11.2.3).

2. If the training error is excessively large consider enlarging the hypothesis class,
completely change it, or change the feature representation of the data.
3. If the training error is small, plot learning curves and try to deduce from them
whether the problem is estimation error or approximation error.
4. If the approximation error seems to be small enough, try to obtain more data.
If this is not possible, consider reducing the complexity of the hypothesis class.
5. If the approximation error seems to be large as well, try to change the hy-

pothesis class or the feature representation of the data completely.
11.4 Summary
Model selection is the task of selecting an appropriate model for the learning
task based on the data itself. We have shown how this can be done using the
SRM learning paradigm or using the more practical approach of validation. If
our learning algorithm fails, a decomposition of the algorithm’s error should be
performed using learning curves, so as to ﬁnd the best remedy.
11.5 Exercises
1.Failure of k-fold cross validation Consider a case in that the label is

chosen at random according to P[y= 1] = P[y= 0] = 1/2. Consider a
learning algorithm that outputs the constant predictor h(x) = 1 if the parity
of the labels on the training set is 1 and otherwise the algorithm outputs the
constant predictor h(x) = 0. Prove that the diﬀerence between the leave-one-
out estimate and the true error in such a case is always 1 /2.
2. LetH1,...,Hkbekhypothesis classes. Suppose you are given mi.i.d. training
examples and you would like to learn the class H=∪k
i=1Hi. Consider two
alternative approaches:

•LearnHon themexamples using the ERM rule

11.5 Exercises 155
•Divide themexamples into a training set of size (1 −α)mand a validation
set of sizeαm, for someα∈(0,1). Then, apply the approach of model
selection using validation. That is, ﬁrst train each class Hion the (1−
α)mtraining examples using the ERM rule with respect to Hi, and let
ˆh1,..., ˆhkbe the resulting hypotheses. Second, apply the ERM rule with
respect to the ﬁnite class {ˆh1,..., ˆhk}on theαmvalidation examples.
Describe scenarios in which the ﬁrst method is better than the second and
vice versa.

12 Convex Learning Problems
In this chapter we introduce convex learning problems . Convex learning comprises
an important family of learning problems, mainly because most of what we can
learn eﬃciently falls into it. We have already encountered linear regression with
the squared loss and logistic regression, which are convex problems, and indeed
they can be learned eﬃciently. We have also seen nonconvex problems, such as
halfspaces with the 0-1 loss, which is known to be computationally hard to learn
in the unrealizable case.
In general, a convex learning problem is a problem whose hypothesis class is a
convex set, and whose loss function is a convex function for each example. We be-
gin the chapter with some required deﬁnitions of convexity. Besides convexity, we
will deﬁne Lipschitzness and smoothness, which are additional properties of the
loss function that facilitate successful learning. We next turn to deﬁning convex
learning problems and demonstrate the necessity for further constraints such as

Boundedness and Lipschitzness or Smoothness. We deﬁne these more restricted
families of learning problems and claim that Convex-Smooth/Lipschitz-Bounded
problems are learnable. These claims will be proven in the next two chapters, in
which we will present two learning paradigms that successfully learn all problems
that are either convex-Lipschitz-bounded or convex-smooth-bounded.
Finally, in Section 12.3, we show how one can handle some nonconvex problems

by minimizing “surrogate” loss functions that are convex (instead of the original
nonconvex loss function). Surrogate convex loss functions give rise to eﬃcient
solutions but might increase the risk of the learned predictor.
12.1 Convexity, Lipschitzness, and Smoothness
12.1.1 Convexity
definition 12.1 (Convex Set) A set Cin a vector space is convex if for any
two vectors u,vinC, the line segment between uandvis contained in C. That
is, for anyα∈[0,1] we have that αu+ (1−α)v∈C.
Examples of convex and nonconvex sets in R2are given in the following. For

the nonconvex sets, we depict two points in the set such that the line between
the two points is not contained in the set.

12.1 Convexity, Lipschitzness, and Smoothness 157
non-convex convex
Givenα∈[0,1], the combination, αu+ (1−α)vof the points u,vis called a
convex combination .
definition 12.2 (Convex Function) Let Cbe a convex set. A function f:
C→Ris convex if for every u,v∈Candα∈[0,1],
f(αu+ (1−α)v)≤αf(u) + (1−α)f(v).
In words,fis convex if for any u,v, the graph of fbetween uandvlies below
the line segment joining f(u) andf(v). An illustration of a convex function,
f:R→R, is depicted in the following.
f(u)f(v)
u
αu+ (1−α)vvαf(u) + (1−α)f(v)
f(αu+ (1−α)v)
The epigraph of a function fis the set
epigraph(f) ={(x,β) :f(x)≤β}. (12.1)
It is easy to verify that a function fis convex if and only if its epigraph is a
convex set. An illustration of a nonconvex function f:R→R, along with its
epigraph, is given in the following.

158 Convex Learning Problems
xf(x)
An important property of convex functions is that every local minimum of the
function is also a global minimum. Formally, let B(u,r) ={v:∥v−u∥≤r}be
a ball of radius rcentered around u. We say that f(u) is a local minimum of f
atuif there exists some r>0 such that for all v∈B(u,r) we havef(v)≥f(u).
It follows that for any v(not necessarily in B), there is a small enough α > 0
such that u+α(v−u)∈B(u,r) and therefore
f(u)≤f(u+α(v−u)). (12.2)
Iffis convex, we also have that
f(u+α(v−u)) =f(αv+ (1−α)u)≤(1−α)f(u) +αf(v). (12.3)
Combining these two equations and rearranging terms, we conclude that f(u)≤
f(v). Since this holds for every v, it follows that f(u) is also a global minimum
off.
Another important property of convex functions is that for every wwe can
construct a tangent to fatwthat lies below feverywhere. If fis diﬀerentiable,
this tangent is the linear function l(u) =f(w) +⟨∇f(w),u−w⟩, where∇f(w)

off.
is the gradient of fatw, namely, the vector of partial derivatives of f,∇f(w) =(
∂f(w)
∂w1,...,∂f(w)
∂wd)
. That is, for convex diﬀerentiable functions,
∀u, f(u)≥f(w) +⟨∇f(w),u−w⟩. (12.4)
In Chapter 14 we will generalize this inequality to nondiﬀerentiable functions.
An illustration of Equation (12.4) is given in the following.

12.1 Convexity, Lipschitzness, and Smoothness 159
f(w)f(u)
w uf(w) +⟨u−w,∇f(w)⟩
Iffis a scalar diﬀerentiable function, there is an easy way to check if it is
convex.
lemma 12.3 Letf:R→Rbe a scalar twice diﬀerential function, and let
f′,f′′be its ﬁrst and second derivatives, respectively. Then, the following are
equivalent:
1.fis convex
2.f′is monotonically nondecreasing
3.f′′is nonnegative
Example 12.1
•The scalar function f(x) =x2is convex. To see this, note that f′(x) = 2x
andf′′(x) = 2>0.
•The scalar function f(x) = log(1+exp( x)) is convex. To see this, observe that
f′(x) =exp(x)
1+exp(x)=1
exp(−x)+1. This is a monotonically increasing function
since the exponent function is a monotonically increasing function.
The following claim shows that the composition of a convex scalar function
with a linear function yields a convex vector-valued function.
claim 12.4 Assume that f:Rd→Rcan be written as f(w) =g(⟨w,x⟩+y),
for some x∈Rd, y∈R, andg:R→R. Then, convexity of gimplies the
convexity of f.

Proof Letw1,w2∈Rdandα∈[0,1]. We have
f(αw1+ (1−α)w2) =g(⟨αw1+ (1−α)w2,x⟩+y)
=g(α⟨w1,x⟩+ (1−α)⟨w2,x⟩+y)
=g(α(⟨w1,x⟩+y) + (1−α)(⟨w2,x⟩+y))
≤αg(⟨w1,x⟩+y) + (1−α)g(⟨w2,x⟩+y),
where the last inequality follows from the convexity of g.
Example 12.2

160 Convex Learning Problems
•Given some x∈Rdandy∈R, letf:Rd→Rbe deﬁned as f(w) =
(⟨w,x⟩−y)2. Then,fis a composition of the function g(a) =a2onto a
linear function, and hence fis a convex function.
•Given some x∈Rdandy∈{± 1}, letf:Rd→Rbe deﬁned as f(w) =
log(1 + exp(−y⟨w,x⟩)). Then,fis a composition of the function g(a) =
log(1 + exp( a)) onto a linear function, and hence fis a convex function.
Finally, the following lemma shows that the maximum of convex functions is
convex and that a weighted sum of convex functions, with nonnegative weights,
is also convex.
claim 12.5 Fori= 1,...,r , letfi:Rd→Rbe a convex function. The
following functions from RdtoRare also convex.
•g(x) = maxi∈[r]fi(x)
•g(x) =∑r
i=1wifi(x), where for all i,wi≥0.
Proof The ﬁrst claim follows by
g(αu+ (1−α)v) = max
ifi(αu+ (1−α)v)
≤max
i[αfi(u) + (1−α)fi(v)]
≤αmax
ifi(u) + (1−α) max
ifi(v)
=αg(u) + (1−α)g(v).
For the second claim
g(αu+ (1−α)v) =∑
iwifi(αu+ (1−α)v)
≤∑
iwi[αfi(u) + (1−α)fi(v)]
=α∑
iwifi(u) + (1−α)∑
iwifi(v)

•g(x) =∑r
≤max
≤αmax
ifi(v)
≤∑
=α∑
iwifi(v)
Example 12.3 The function g(x) =|x|is convex. To see this, note that g(x) =
max{x,−x}and that both the function f1(x) =xandf2(x) =−xare convex.
12.1.2 Lipschitzness
The deﬁnition of Lipschitzness below is with respect to the Euclidean norm over
Rd. However, it is possible to deﬁne Lipschitzness with respect to any norm.
definition 12.6 (Lipschitzness) Let C⊂Rd. A function f:Rd→Rkis
ρ-Lipschitz over Cif for every w1,w2∈Cwe have that∥f(w1)−f(w2)∥≤
ρ∥w1−w2∥.

12.1 Convexity, Lipschitzness, and Smoothness 161
Intuitively, a Lipschitz function cannot change too fast. Note that if f:R→R
is diﬀerentiable, then by the mean value theorem we have
f(w1)−f(w2) =f′(u)(w1−w2),
whereuis some point between w1andw2. It follows that if the derivative of f
is everywhere bounded (in absolute value) by ρ, then the function is ρ-Lipschitz.
Example 12.4
•The function f(x) =|x|is 1-Lipschitz over R. This follows from the triangle
inequality: For every x1,x2,
|x1|−|x2|=|x1−x2+x2|−|x2|≤|x1−x2|+|x2|−|x2|=|x1−x2|.
Since this holds for both x1,x2andx2,x1, we obtain that ||x1|−|x2||≤
|x1−x2|.
•The function f(x) = log(1+exp( x)) is 1-Lipschitz over R. To see this, observe
that
|f′(x)|=⏐⏐⏐⏐exp(x)
1 + exp(x)⏐⏐⏐⏐=⏐⏐⏐⏐1
exp(−x) + 1⏐⏐⏐⏐≤1.
•The function f(x) =x2is notρ-Lipschitz over Rfor anyρ. To see this, take
x1= 0 andx2= 1 +ρ, then
f(x2)−f(x1) = (1 +ρ)2>ρ(1 +ρ) =ρ|x2−x1|.
However, this function is ρ-Lipschitz over the set C={x:|x|≤ρ/2}.
Indeed, for any x1,x2∈Cwe have
|x2
1−x2

|x1−x2|.
that
|x2
1−x2
2|=|x1+x2||x1−x2|≤2(ρ/2)|x1−x2|=ρ|x1−x2|.
•The linear function f:Rd→Rdeﬁned by f(w) =⟨v,w⟩+bwhere v∈Rd
is∥v∥-Lipschitz. Indeed, using Cauchy-Schwartz inequality,
|f(w1)−f(w2)|=|⟨v,w1−w2⟩|≤∥ v∥∥w1−w2∥.
The following claim shows that composition of Lipschitz functions preserves
Lipschitzness.
claim 12.7 Letf(x) =g1(g2(x)), whereg1isρ1-Lipschitz and g2isρ2-
Lipschitz. Then, fis(ρ1ρ2)-Lipschitz. In particular, if g2is the linear function,
g2(x) =⟨v,x⟩+b, for some v∈Rd,b∈R, thenfis(ρ1∥v∥)-Lipschitz.
Proof
|f(w1)−f(w2)|=|g1(g2(w1))−g1(g2(w2))|

Proof
≤ρ1∥g2(w1)−g2(w2)∥
≤ρ1ρ2∥w1−w2∥.

162 Convex Learning Problems
12.1.3 Smoothness
The deﬁnition of a smooth function relies on the notion of gradient . Recall that
the gradient of a diﬀerentiable function f:Rd→Ratw, denoted∇f(w), is the
vector of partial derivatives of f, namely,∇f(w) =(
∂f(w)
∂wd)
.
definition 12.8 (Smoothness) A diﬀerentiable function f:Rd→Risβ-
smooth if its gradient is β-Lipschitz; namely, for all v,wwe have∥∇f(v)−
∇f(w)∥≤β∥v−w∥.
It is possible to show that smoothness implies that for all v,wwe have
f(v)≤f(w) +⟨∇f(w),v−w⟩+β
2∥v−w∥2. (12.5)
Recall that convexity of fimplies that f(v)≥f(w)+⟨∇f(w),v−w⟩. Therefore,
when a function is both convex and smooth, we have both upper and lower
bounds on the diﬀerence between the function and its ﬁrst order approximation.
Setting v=w−1
β∇f(w) in the right-hand side of Equation (12.5) and rear-
ranging terms, we obtain
1
2β∥∇f(w)∥2≤f(w)−f(v).
If we further assume that f(v)≥0 for all vwe conclude that smoothness implies
the following:
∥∇f(w)∥2≤2βf(w). (12.6)

1
A function that satisﬁes this property is also called a self-bounded function.
Example 12.5
•The function f(x) =x2is 2-smooth. This follows directly from the fact that
f′(x) = 2x. Note that for this particular function Equation (12.5) and
Equation (12.6) hold with equality.
•The function f(x) = log(1 + exp( x)) is (1/4)-smooth. Indeed, since f′(x) =
1
1+exp(−x)we have that
|f′′(x)|=exp(−x)
(1 + exp(−x))2=1
(1 + exp(−x))(1 + exp(x))≤1/4.
Hence,f′is (1/4)-Lipschitz. Since this function is nonnegative, Equa-

1
tion (12.6) holds as well.
The following claim shows that a composition of a smooth scalar function over
a linear function preserves smoothness.
claim 12.9 Letf(w) =g(⟨w,x⟩+b), whereg:R→Ris aβ-smooth function,
x∈Rd, andb∈R. Then,fis(β∥x∥2)-smooth.

Proof By the chain rule we have that ∇f(w) =g′(⟨w,x⟩+b)x, whereg′is the
derivative of g. Using the smoothness of gand the Cauchy-Schwartz inequality
we therefore obtain
f(v) =g(⟨v,x⟩+b)
≤g(⟨w,x⟩+b) +g′(⟨w,x⟩+b)⟨v−w,x⟩+β
2(⟨v−w,x⟩)2
2(∥v−w∥∥x∥)2
=f(w) +⟨∇f(w),v−w⟩+β∥x∥2
2∥v−w∥2.
Example 12.6
•For any x∈Rdandy∈R, letf(w) = (⟨w,x⟩−y)2. Then,fis (2∥x∥2)-
smooth.
•For any x∈Rdandy∈{± 1}, letf(w) = log(1 + exp(−y⟨w,x⟩)). Then,fis
(∥x∥2/4)-smooth.
12.2 Convex Learning Problems
Recall that in our general deﬁnition of learning (Deﬁnition 3.4 in Chapter 3), we
have a hypothesis class H, a set of examples Z, and a loss function ℓ:H×Z→
R+. So far in the book we have mainly thought of Zas being the product of an
instance space and a target space, Z=X×Y , andHbeing a set of functions from
XtoY. However,Hcan be an arbitrary set. Indeed, throughout this chapter,
we consider hypothesis classes Hthat are subsets of the Euclidean space Rd.

That is, every hypothesis is some real-valued vector. We shall, therefore, denote
a hypothesis inHbyw. Now we can ﬁnally deﬁne convex learning problems:
definition 12.10 (Convex Learning Problem) A learning problem, ( H,Z,ℓ),
is called convex if the hypothesis class His a convex set and for all z∈Z, the
loss function, ℓ(·,z), is a convex function (where, for any z,ℓ(·,z) denotes the
functionf:H→Rdeﬁned byf(w) =ℓ(w,z)).
Example 12.7 (Linear Regression with the Squared Loss) Recall that linear

regression is a tool for modeling the relationship between some “explanatory”
variables and some real valued outcome (see Chapter 9). The domain set X
is a subset of Rd, for some d, and the label set Yis the set of real numbers.
We would like to learn a linear function h:Rd→Rthat best approximates
the relationship between our variables. In Chapter 9 we deﬁned the hypothesis
class as the set of homogenous linear functions, H={x↦→⟨w,x⟩:w∈Rd},
and used the squared loss function, ℓ(h,(x,y)) = (h(x)−y)2. However, we can

equivalently model the learning problem as a convex learning problem as follows.

164 Convex Learning Problems
Each linear function is parameterized by a vector w∈Rd. Hence, we can deﬁne
Hto be the set of all such parameters, namely, H=Rd. The set of examples is
Z=X×Y =Rd×R=Rd+1, and the loss function is ℓ(w,(x,y)) = (⟨w,x⟩−y)2.
Clearly, the setHis a convex set. The loss function is also convex with respect
to its ﬁrst argument (see Example 12.2).
lemma 12.11 Ifℓis a convex loss function and the class His convex, then the
ERMHproblem, of minimizing the empirical loss over H, is a convex optimiza-
tion problem (that is, a problem of minimizing a convex function over a convex
set).
Proof Recall that the ERM Hproblem is deﬁned by
ERMH(S) = argmin
w∈HLS(w).
Since, for a sample S=z1,...,zm, for every w,LS(w) =1
m∑m
i=1ℓ(w,zi),
Claim 12.5 implies that LS(w) is a convex function. Therefore, the ERM rule
is a problem of minimizing a convex function subject to the constraint that the
solution should be in a convex set.
Under mild conditions, such problems can be solved eﬃciently using generic

set).
w∈HLS(w).
m∑m
optimization algorithms. In particular, in Chapter 14 we will present a very
simple algorithm for minimizing convex functions.
12.2.1 Learnability of Convex Learning Problems
We have argued that for many cases, implementing the ERM rule for convex
learning problems can be done eﬃciently. But is convexity a suﬃcient condition
for the learnability of a problem?
To make the quesion more speciﬁc: In VC theory, we saw that halfspaces in
d-dimension are learnable (perhaps ineﬃciently). We also argued in Chapter 9

using the “discretization trick” that if the problem is of dparameters, it is
learnable with a sample complexity being a function of d. That is, for a constant
d, the problem should be learnable. So, maybe all convex learning problems over
Rd, are learnable?
Example 12.8 later shows that the answer is negative, even when dis low. Not
all convex learning problems over Rdare learnable. There is no contradiction
to VC theory since VC theory only deals with binary classiﬁcation while here
we consider a wide family of problems. There is also no contradiction to the

“discretization trick” as there we assumed that the loss function is bounded and
also assumed that a representation of each parameter using a ﬁnite number of
bits suﬃces. As we will show later, under some additional restricting conditions
that hold in many practical scenarios, convex problems are learnable.
Example 12.8 (Nonlearnability of Linear Regression Even If d= 1) LetH=R,
and the loss be the squared loss: ℓ(w,(x,y)) = (wx−y)2(we’re referring to the

12.2 Convex Learning Problems 165
homogenous case). Let Abe any deterministic algorithm.1Assume, by way of
contradiction, that Ais a successful PAC learner for this problem. That is, there
exists a function m(·,·), such that for every distribution Dand for every ϵ,δif
Areceives a training set of size m≥m(ϵ,δ), it should output, with probability
of at least 1−δ, a hypothesis ˆ w=A(S), such that LD( ˆw)−minwLD(w)≤ϵ.
Chooseϵ= 1/100,δ= 1/2, letm≥m(ϵ,δ), and setµ=log(100/99)
2m. We will
deﬁne two distributions, and will show that Ais likely to fail on at least one
of them. The ﬁrst distribution, D1, is supported on two examples, z1= (1,0)
andz2= (µ,−1), where the probability mass of the ﬁrst example is µwhile the
probability mass of the second example is 1 −µ. The second distribution, D2, is
supported entirely on z2.
Observe that for both distributions, the probability that all examples of the
training set will be of the second type is at least 99%. This is trivially true for

2m. We will
D2, whereas forD1, the probability of this event is
(1−µ)m≥e−2µm= 0.99.
Since we assume that Ais a deterministic algorithm, upon receiving a training
set ofmexamples, each of which is ( µ,−1), the algorithm will output some ˆ w.
Now, if ˆw<−1/(2µ), we will set the distribution to be D1. Hence,
LD1( ˆw)≥µ( ˆw)2≥1/(4µ).
However,
min
wLD1(w)≤LD1(0) = (1−µ).
It follows that
LD1( ˆw)−min
wLD1(w)≥1
4µ−(1−µ)>ϵ.
Therefore, such algorithm Afails onD1. On the other hand, if ˆ w≥−1/(2µ)

However,
min
wLD1(w)≥1
then we’ll set the distribution to be D2. Then we have that LD2( ˆw)≥1/4 while
minwLD2(w) = 0, soAfails onD2. In summary, we have shown that for every
Athere exists a distribution on which Afails, which implies that the problem is
not PAC learnable.
A possible solution to this problem is to add another constraint on the hypoth-
esis class. In addition to the convexity requirement, we require that Hwill be
bounded ; namely, we assume that for some predeﬁned scalar B, every hypothesis
w∈H satisﬁes∥w∥≤B.

Boundedness and convexity alone are still not suﬃcient for ensuring that the
problem is learnable, as the following example demonstrates.
Example 12.9 As in Example 12.8, consider a regression problem with the
squared loss. However, this time let H={w:|w|≤1}⊂Rbe a bounded
1Namely, given Sthe output of Ais determined. This requirement is for the sake of
simplicity. A slightly more involved argument will show that nondeterministic algorithms
will also fail to learn the problem.

166 Convex Learning Problems
hypothesis class. It is easy to verify that His convex. The argument will be
the same as in Example 12.8, except that now the two distributions, D1,D2will
be supported on z1= (1/µ,0) andz2= (1,−1). If the algorithm Areturns
ˆw <−1/2 upon receiving mexamples of the second type, then we will set the
distribution to be D1and have that
wLD1(w)≥µ( ˆw/µ)2−LD1(0)≥1/(4µ)−(1−µ)>ϵ.
Similarly, if ˆ w≥−1/2 we will set the distribution to be D2and have that
LD2( ˆw)−min
wLD2(w)≥(−1/2 + 1)2−0>ϵ.
This example shows that we need additional assumptions on the learning
problem, and this time the solution is in Lipschitzness or smoothness of the
loss function. This motivates a deﬁnition of two families of learning problems,
convex-Lipschitz-bounded and convex-smooth-bounded, which are deﬁned later.
12.2.2 Convex-Lipschitz/Smooth-Bounded Learning Problems
definition 12.12 (Convex-Lipschitz-Bounded Learning Problem) A learning

problem, (H,Z,ℓ), is called Convex-Lipschitz-Bounded, with parameters ρ,B if
the following holds:
•The hypothesis class His a convex set and for all w∈H we have∥w∥≤B.
•For allz∈Z, the loss function, ℓ(·,z), is a convex and ρ-Lipschitz function.
Example 12.10 LetX={x∈Rd:∥x∥≤ρ}andY=R. LetH={w∈Rd:
∥w∥≤B}and let the loss function be ℓ(w,(x,y)) =|⟨w,x⟩−y|. This corre-
sponds to a regression problem with the absolute-value loss, where we assume
that the instances are in a ball of radius ρand we restrict the hypotheses to be

homogenous linear functions deﬁned by a vector wwhose norm is bounded by
B. Then, the resulting problem is Convex-Lipschitz-Bounded with parameters
ρ,B.
definition 12.13 (Convex-Smooth-Bounded Learning Problem) A learning
problem, (H,Z,ℓ), is called Convex-Smooth-Bounded, with parameters β,B if
•For allz∈Z, the loss function, ℓ(·,z), is a convex, nonnegative, and β-smooth
function.
Note that we also required that the loss function is nonnegative. This is needed

ρ,B.
function.
to ensure that the loss function is self-bounded, as described in the previous
section.

Example 12.11 LetX={x∈Rd:∥x∥≤β/2}andY=R. LetH={w∈
Rd:∥w∥≤B}and let the loss function be ℓ(w,(x,y)) = (⟨w,x⟩−y)2. This
corresponds to a regression problem with the squared loss, where we assume that
the instances are in a ball of radius β/2 and we restrict the hypotheses to be
homogenous linear functions deﬁned by a vector wwhose norm is bounded by B.
Then, the resulting problem is Convex-Smooth-Bounded with parameters β,B.
We claim that these two families of learning problems are learnable. That is,
the properties of convexity, boundedness, and Lipschitzness or smoothness of the
loss function are suﬃcient for learnability. We will prove this claim in the next
chapters by introducing algorithms that learn these problems successfully.
12.3 Surrogate Loss Functions
As mentioned, and as we will see in the next chapters, convex problems can
be learned eﬀﬁciently. However, in many cases, the natural loss function is not
convex and, in particular, implementing the ERM rule is hard.

As an example, consider the problem of learning the hypothesis class of half-
spaces with respect to the 0 −1 loss. That is,
ℓ0−1(w,(x,y)) =1[y̸=sign(⟨w,x⟩)]=1[y⟨w,x⟩≤0].
This loss function is not convex with respect to wand indeed, when trying to
minimize the empirical risk with respect to this loss function we might encounter
local minima (see Exercise 1). Furthermore, as discussed in Chapter 8, solving
the ERM problem with respect to the 0 −1 loss in the unrealizable case is known
to be NP-hard.

To circumvent the hardness result, one popular approach is to upper bound
the nonconvex loss function by a convex surrogate loss function. As its name
indicates, the requirements from a convex surrogate loss are as follows:
1. It should be convex.
2. It should upper bound the original loss.
For example, in the context of learning halfspaces, we can deﬁne the so-called
hinge loss as a convex surrogate for the 0 −1 loss, as follows:
ℓhinge(w,(x,y))def= max{0,1−y⟨w,x⟩}.

Clearly, for all wand all ( x,y),ℓ0−1(w,(x,y))≤ℓhinge(w,(x,y)). In addition,
the convexity of the hinge loss follows directly from Claim 12.5. Hence, the hinge
loss satisﬁes the requirements of a convex surrogate loss function for the zero-one
loss. An illustration of the functions ℓ0−1andℓhingeis given in the following.

168 Convex Learning Problems
y⟨w,x⟩ℓhinge
ℓ0−1
11
Once we have deﬁned the surrogate convex loss, we can learn the problem with
respect to it. The generalization requirement from a hinge loss learner will have
the form
Lhinge
D(A(S))≤min
w∈HLhinge
D(w) +ϵ,
whereLhinge
D(w) =E(x,y)∼D[ℓhinge(w,(x,y))]. Using the surrogate property, we
can lower bound the left-hand side by L0−1
D(A(S)), which yields
L0−1
w∈HLhinge
D(w) +ϵ.
We can further rewrite the upper bound as follows:
L0−1
w∈HL0−1
D(w) +(
min
w∈HLhinge
D(w)−min
w∈HL0−1
D(w))
+ϵ.
That is, the 0−1 error of the learned predictor is upper bounded by three terms:
•Approximation error : This is the term min w∈HL0−1
D(w), which measures how
well the hypothesis class performs on the distribution. We already elabo-
rated on this error term in Chapter 5.
•Estimation error : This is the error that results from the fact that we only
receive a training set and do not observe the distribution D. We already

L0−1
w∈HL0−1
D(w) +(
min
w∈HLhinge
D(w)−min
w∈HL0−1
D(w))
+ϵ.
elaborated on this error term in Chapter 5.
•Optimization error : This is the term(
minw∈HLhinge
D(w)−minw∈HL0−1
D(w))
that measures the diﬀerence between the approximation error with respect
to the surrogate loss and the approximation error with respect to the orig-
inal loss. The optimization error is a result of our inability to minimize the
training loss with respect to the original loss. The size of this error depends
on the speciﬁc distribution of the data and on the speciﬁc surrogate loss
we are using.
12.4 Summary

D(w))
We introduced two families of learning problems: convex-Lipschitz-bounded and
convex-smooth-bounded. In the next two chapters we will describe two generic

learning algorithms for these families. We also introduced the notion of convex
surrogate loss function, which enables us also to utilize the convex machinery for
nonconvex problems.
12.5 Bibliographic Remarks
There are several excellent books on convex analysis and optimization (Boyd &
Vandenberghe 2004, Borwein & Lewis 2006, Bertsekas 1999, Hiriart-Urruty &
Lemar´ echal 1996). Regarding learning problems, the family of convex-Lipschitz-
bounded problems was ﬁrst studied by Zinkevich (2003) in the context of online
learning and by Shalev-Shwartz, Shamir, Sridharan & Srebro (2009) in the con-
text of PAC learning.
12.6 Exercises
1. Construct an example showing that the 0 −1 loss function may suﬀer from
local minima; namely, construct a training sample S∈(X×{± 1})m(say, for
X=R2), for which there exist a vector wand someϵ>0 such that
1. For any w′such that∥w−w′∥≤ϵwe haveLS(w)≤LS(w′) (where the
loss here is the 0−1 loss). This means that wis a local minimum of LS.

2. There exists some w∗such thatLS(w∗)< LS(w). This means that wis
not a global minimum of LS.
2. Consider the learning problem of logistic regression: Let H=X={x∈
Rd:∥x∥ ≤B}, for some scalar B > 0, letY={±1}, and let the loss
functionℓbe deﬁned as ℓ(w,(x,y)) = log(1 + exp( −y⟨w,x⟩)). Show that
the resulting learning problem is both convex-Lipschitz-bounded and convex-
smooth-bounded. Specify the parameters of Lipschitzness and smoothness.
3. Consider the problem of learning halfspaces with the hinge loss. We limit our

domain to the Euclidean ball with radius R. That is,X={x:∥x∥2≤R}.
The label set isY={±1}and the loss function ℓis deﬁned by ℓ(w,(x,y)) =
max{0,1−y⟨w,x⟩}. We already know that the loss function is convex. Show
that it isR-Lipschitz.
4.(*) Convex-Lipschitz-Boundedness Is Not Suﬃcient for Computa-
tional Eﬃciency: In the next chapter we show that from the statistical
perspective, all convex-Lipschitz-bounded problems are learnable (in the ag-
nostic PAC model). However, our main motivation to learn such problems

resulted from the computational perspective – convex optimization is often
eﬃciently solvable. Yet the goal of this exercise is to show that convexity
alone is not suﬃcient for eﬃciency. We show that even for the case d= 1,
there is a convex-Lipschitz-bounded problem which cannot be learned by any
computable learner.
Let the hypothesis class be H= [0,1] and let the example domain, Z, be

170 Convex Learning Problems
the set of all Turing machines. Deﬁne the loss function as follows. For every
Turing machine T∈Z, letℓ(0,T) = 1 ifThalts on the input 0 and ℓ(0,T) = 0
ifTdoesn’t halt on the input 0. Similarly, let ℓ(1,T) = 0 ifThalts on the
input 0 and ℓ(1,T) = 1 ifTdoesn’t halt on the input 0. Finally, for h∈(0,1),
letℓ(h,T) =hℓ(0,T) + (1−h)ℓ(1,T).
1. Show that the resulting learning problem is convex-Lipschitz-bounded.
2. Show that no computable algorithm can learn the problem.

13 Regularization and Stability
In the previous chapter we introduced the families of convex-Lipschitz-bounded
and convex-smooth-bounded learning problems. In this section we show that all
learning problems in these two families are learnable. For some learning problems
of this type it is possible to show that uniform convergence holds; hence they
are learnable using the ERM rule. However, this is not true for all learning
problems of this type. Yet, we will introduce another learning rule and will show
that it learns all convex-Lipschitz-bounded and convex-smooth-bounded learning
problems.
The new learning paradigm we introduce in this chapter is called Regularized
Loss Minimization , or RLM for short. In RLM we minimize the sum of the em-
pirical risk and a regularization function. Intuitively, the regularization function
measures the complexity of hypotheses. Indeed, one interpretation of the reg-
ularization function is the structural risk minimization paradigm we discussed

problems.
in Chapter 7. Another view of regularization is as a stabilizer of the learning
algorithm. An algorithm is considered stable if a slight change of its input does
not change its output much. We will formally deﬁne the notion of stability (what
we mean by “slight change of input” and by “does not change much the out-
put”) and prove its close relation to learnability. Finally, we will show that using
the squared ℓ2norm as a regularization function stabilizes all convex-Lipschitz or

convex-smooth learning problems. Hence, RLM can be used as a general learning
rule for these families of learning problems.
13.1 Regularized Loss Minimization
Regularized Loss Minimization (RLM) is a learning rule in which we jointly min-
imize the empirical risk and a regularization function. Formally, a regularization
function is a mapping R:Rd→R, and the regularized loss minimization rule
outputs a hypothesis in
argmin
w(LS(w) +R(w)). (13.1)
Regularized loss minimization shares similarities with minimum description length

argmin
algorithms and structural risk minimization (see Chapter 7). Intuitively, the
“complexity” of hypotheses is measured by the value of the regularization func-

172 Regularization and Stability
tion, and the algorithm balances between low empirical risk and “simpler,” or
“less complex,” hypotheses.
There are many possible regularization functions one can use, reﬂecting some
prior belief about the problem (similarly to the description language in Minimum
Description Length). Throughout this section we will focus on one of the most
simple regularization functions: R(w) =λ∥w∥2, whereλ>0 is a scalar and the
norm is the ℓ2norm,∥w∥=√∑d
i=1w2
i. This yields the learning rule:
A(S) = argmin
w(
LS(w) +λ∥w∥2)
. (13.2)
This type of regularization function is often called Tikhonov regularization.
As mentioned before, one interpretation of Equation (13.2) is using structural
risk minimization, where the norm of wis a measure of its “complexity.” Recall
that in the previous chapter we introduced the notion of bounded hypothesis
classes. Therefore, we can deﬁne a sequence of hypothesis classes, H1⊂H 2⊂
H3..., whereHi={w:∥w∥2≤i}. If the sample complexity of each Hidepends

w(
. (13.2)
onithen the RLM rule is similar to the SRM rule for this sequence of nested
classes.
A diﬀerent interpretation of regularization is as a stabilizer. In the next section
we deﬁne the notion of stability and prove that stable learning rules do not
overﬁt. But ﬁrst, let us demonstrate the RLM rule for linear regression with the
squared loss.
13.1.1 Ridge Regression
Applying the RLM rule with Tikhonov regularization to linear regression with
the squared loss, we obtain the following learning rule:
argmin
w∈Rd(
λ∥w∥2
2+1
mm∑

classes.
argmin
w∈Rd(
λ∥w∥2
2+1
mm∑
i=11
2(⟨w,xi⟩−yi)2)
. (13.3)
Performing linear regression using Equation (13.3) is called ridge regression .
To solve Equation (13.3) we compare the gradient of the objective to zero and
obtain the set of linear equations
(2λmI +A)w=b,
whereIis the identity matrix and A,bare as deﬁned in Equation (9.6), namely,
A=(m∑
i=1xix⊤
i)
and b=m∑
i=1yixi. (13.4)
SinceAis a positive semideﬁnite matrix, the matrix 2 λmI +Ahas all its eigen-
values bounded below by 2 λm. Hence, this matrix is invertible and the solution
to ridge regression becomes
w= (2λmI +A)−1b. (13.5)

In the next section we formally show how regularization stabilizes the algo-
rithm and prevents overﬁtting. In particular, the analysis presented in the next
sections (particularly, Corollary 13.11) will yield:
theorem 13.1 LetDbe a distribution over X× [−1,1], whereX={x∈
Rd:∥x∥≤ 1}. LetH={w∈Rd:∥w∥≤B}. For anyϵ∈(0,1), letm≥
150B2/ϵ2. Then, applying the ridge regression algorithm with parameter λ=
ϵ/(3B2)satisﬁes
E
S∼Dm[LD(A(S))]≤min
w∈HLD(w) +ϵ.
Remark 13.1 The preceding theorem tells us how many examples are needed
to guarantee that the expected value of the risk of the learned predictor will be
bounded by the approximation error of the class plus ϵ. In the usual deﬁnition
of agnostic PAC learning we require that the risk of the learned predictor will
be bounded with probability of at least 1 −δ. In Exercise 1 we show how an
algorithm with a bounded expected risk can be used to construct an agnostic
PAC learner.
13.2 Stable Rules Do Not Overﬁt

Intuitively, a learning algorithm is stable if a small change of the input to the
algorithm does not change the output of the algorithm much. Of course, there
are many ways to deﬁne what we mean by “a small change of the input” and
what we mean by “does not change the output much”. In this section we deﬁne
a speciﬁc notion of stability and prove that under this deﬁnition, stable rules do
not overﬁt.
LetAbe a learning algorithm, let S= (z1,...,zm) be a training set of m

examples, and let A(S) denote the output of A. The algorithm Asuﬀers from
overﬁtting if the diﬀerence between the true risk of its output, LD(A(S)), and the
empirical risk of its output, LS(A(S)), is large. As mentioned in Remark 13.1,
throughout this chapter we focus on the expectation (with respect to the choice
ofS) of this quantity, namely, ES[LD(A(S))−LS(A(S))].
We next deﬁne the notion of stability. Given the training set Sand an ad-

ditional example z′, letS(i)be the training set obtained by replacing the i’th
example of Swithz′; namely,S(i)= (z1,...,zi−1, z′, zi+1,...,zm). In our deﬁ-
nition of stability, “a small change of the input” means that we feed AwithS(i)
instead of with S. That is, we only replace one training example. We measure
the eﬀect of this small change of the input on the output of A, by comparing
the loss of the hypothesis A(S) onzito the loss of the hypothesis A(S(i)) onzi.
Intuitively, a good learning algorithm will have ℓ(A(S(i)),zi)−ℓ(A(S),zi)≥0,

since in the ﬁrst term the learning algorithm does not observe the example zi
while in the second term ziis indeed observed. If the preceding diﬀerence is very
large we suspect that the learning algorithm might overﬁt. This is because the

174 Regularization and Stability
learning algorithm drastically changes its prediction on ziif it observes it in the
training set. This is formalized in the following theorem.
theorem 13.2 LetDbe a distribution. Let S= (z1,...,zm)be an i.i.d. se-
quence of examples and let z′be another i.i.d. example. Let U(m)be the uniform
distribution over [m]. Then, for any learning algorithm,
E
S∼Dm[LD(A(S))−LS(A(S))] = E
(S,z′)∼Dm+1,i∼U(m)[ℓ(A(S(i),zi))−ℓ(A(S),zi)].
(13.6)
Proof SinceSandz′are both drawn i.i.d. from D, we have that for every i,
E
S[LD(A(S))] = E
S,z′[ℓ(A(S),z′)] =E
S,z′[ℓ(A(S(i)),zi)].
On the other hand, we can write
E
S[LS(A(S))] =E
S,i[ℓ(A(S),zi)].
Combining the two equations we conclude our proof.
When the right-hand side of Equation (13.6) is small, we say that Ais astable
algorithm – changing a single example in the training set does not lead to a
signiﬁcant change. Formally,
definition 13.3 (On-Average-Replace-One-Stable) Let ϵ:N→Rbe a mono-

(13.6)
E
E
tonically decreasing function. We say that a learning algorithm Ais on-average-
replace-one-stable with rate ϵ(m) if for every distribution D
E
(S,z′)∼Dm+1,i∼U(m)[ℓ(A(S(i),zi))−ℓ(A(S),zi)]≤ϵ(m).
Theorem 13.2 tells us that a learning algorithm does not overﬁt if and only
if it is on-average-replace-one-stable. Of course, a learning algorithm that does
not overﬁt is not necessarily a good learning algorithm – take, for example, an
algorithmAthat always outputs the same hypothesis. A useful algorithm should

E
ﬁnd a hypothesis that on one hand ﬁts the training set (i.e., has a low empirical
risk) and on the other hand does not overﬁt. Or, in light of Theorem 13.2, the
algorithm should both ﬁt the training set and at the same time be stable. As we
shall see, the parameter λof the RLM rule balances between ﬁtting the training
set and being stable.
13.3 Tikhonov Regularization as a Stabilizer
In the previous section we saw that stable rules do not overﬁt. In this section we

show that applying the RLM rule with Tikhonov regularization, λ∥w∥2, leads to
a stable algorithm. We will assume that the loss function is convex and that it
is either Lipschitz or smooth.
The main property of the Tikhonov regularization that we rely on is that it
makes the objective of RLM strongly convex , as deﬁned in the following.

13.3 Tikhonov Regularization as a Stabilizer 175
definition 13.4 (Strongly Convex Functions) A function fisλ-strongly con-
vex if for all w,uandα∈(0,1) we have
f(αw+ (1−α)u)≤αf(w) + (1−α)f(u)−λ
2α(1−α)∥w−u∥2.
Clearly, every convex function is 0-strongly convex. An illustration of strong
convexity is given in the following ﬁgure.
f(w)f(u)
w
αw+ (1−α)uu≥λ
2α(1−α)∥u−w∥2
The following lemma implies that the objective of RLM is (2 λ)-strongly con-
vex. In addition, it underscores an important property of strong convexity.
lemma 13.5
1. The function f(w) =λ∥w∥2is2λ-strongly convex.
2. Iffisλ-strongly convex and gis convex, then f+gisλ-strongly convex.
3. Iffisλ-strongly convex and uis a minimizer of f, then, for any w,
f(w)−f(u)≥λ
2∥w−u∥2.
Proof The ﬁrst two points follow directly from the deﬁnition. To prove the last
point, we divide the deﬁnition of strong convexity by αand rearrange terms to
get that
f(u+α(w−u))−f(u)
α≤f(w)−f(u)−λ
2(1−α)∥w−u∥2.

lemma 13.5
2∥w−u∥2.
get that
Taking the limit α→0 we obtain that the right-hand side converges to f(w)−
f(u)−λ
2∥w−u∥2. On the other hand, the left-hand side becomes the derivative
of the function g(α) =f(u+α(w−u)) atα= 0. Since uis a minimizer of f,
it follows that α= 0 is a minimizer of g, and therefore the left-hand side of the
preceding goes to zero in the limit α→0, which concludes our proof.
We now turn to prove that RLM is stable. Let S= (z1,...,zm) be a training

get that
f(u)−λ
set, letz′be an additional example, and let S(i)= (z1,...,zi−1,z′,zi+1,...,zm).
LetAbe the RLM rule, namely,
w(
.

176 Regularization and Stability
DenotefS(w) =LS(w) +λ∥w∥2, and based on Lemma 13.5 we know that fSis
(2λ)-strongly convex. Relying on part 3 of the lemma, it follows that for any v,
fS(v)−fS(A(S))≥λ∥v−A(S)∥2. (13.7)
On the other hand, for any vandu, and for all i, we have
fS(v)−fS(u) =LS(v) +λ∥v∥2−(LS(u) +λ∥u∥2) (13.8)
=LS(i)(v) +λ∥v∥2−(LS(i)(u) +λ∥u∥2)
+ℓ(v,zi)−ℓ(u,zi)
m+ℓ(u,z′)−ℓ(v,z′)
m.
In particular, choosing v=A(S(i)),u=A(S), and using the fact that vmini-
mizesLS(i)(w) +λ∥w∥2, we obtain that
fS(A(S(i)))−fS(A(S))≤ℓ(A(S(i)),zi)−ℓ(A(S),zi)
m+ℓ(A(S),z′)−ℓ(A(S(i)),z′)
m.
(13.9)
Combining this with Equation (13.7) we obtain that
λ∥A(S(i))−A(S)∥2≤ℓ(A(S(i)),zi)−ℓ(A(S),zi)
m.
(13.10)
The two subsections that follow continue the stability analysis for either Lip-
schitz or smooth loss functions. For both families of loss functions we show that
RLM is stable and therefore it does not overﬁt.
13.3.1 Lipschitz Loss

m.
(13.9)
m.
(13.10)
If the loss function, ℓ(·,zi), isρ-Lipschitz, then by the deﬁnition of Lipschitzness,
ℓ(A(S(i)),zi)−ℓ(A(S),zi)≤ρ∥A(S(i))−A(S)∥. (13.11)
ℓ(A(S),z′)−ℓ(A(S(i)),z′)≤ρ∥A(S(i))−A(S)∥.
Plugging these inequalities into Equation (13.10) we obtain
λ∥A(S(i))−A(S)∥2≤2ρ∥A(S(i))−A(S)∥
m,
∥A(S(i))−A(S)∥≤2ρ
λm.
Plugging the preceding back into Equation (13.11) we conclude that
ℓ(A(S(i)),zi)−ℓ(A(S),zi)≤2ρ2
λm.
Since this holds for any S,z′,iwe immediately obtain:

13.3 Tikhonov Regularization as a Stabilizer 177
corollary 13.6 Assume that the loss function is convex and ρ-Lipschitz.
Then, the RLM rule with the regularizer λ∥w∥2is on-average-replace-one-stable
with rate2ρ2
λm. It follows (using Theorem 13.2) that
E
S∼Dm[LD(A(S))−LS(A(S))]≤2ρ2
λm.
13.3.2 Smooth and Nonnegative Loss
If the loss is β-smooth and nonnegative then it is also self-bounded (see Sec-
tion 12.1):
∥∇f(w)∥2≤2βf(w). (13.12)
We further assume that λ≥2β
m, or, in other words, that β≤λm/2. By the
smoothness assumption we have that
ℓ(A(S(i)),zi)−ℓ(A(S),zi)≤⟨∇ℓ(A(S),zi),A(S(i))−A(S)⟩+β
2∥A(S(i))−A(S)∥2.
(13.13)
Using the Cauchy-Schwartz inequality and Equation (12.6) we further obtain
that
ℓ(A(S(i)),zi)−ℓ(A(S),zi)
≤∥∇ℓ(A(S),zi)∥∥A(S(i))−A(S)∥+β
2∥A(S(i))−A(S)∥2
≤√
2βℓ(A(S),zi)∥A(S(i))−A(S)∥+β
(13.14)
By a symmetric argument it holds that,
ℓ(A(S),z′)−ℓ(A(S(i)),z′)
≤√
2βℓ(A(S(i)),z′)∥A(S(i))−A(S)∥+β

(13.13)
that
≤√
(13.14)
≤√
Plugging these inequalities into Equation (13.10) and rearranging terms we ob-
tain that
∥A(S(i))−A(S)∥≤√2β
(λm−β)(√
ℓ(A(S),zi) +√
ℓ(A(S(i)),z′))
.
Combining the preceding with the assumption β≤λm/2 yields
∥A(S(i))−A(S)∥≤√8β
λm(√
.

178 Regularization and Stability
Combining the preceding with Equation (13.14) and again using the assumption
β≤λm/2 yield
≤√
≤(4β
λm+8β2
(λm)2)(√
ℓ(A(S(i)),z′))2
≤8β
λm(√
≤24β
λm(
ℓ(A(S),zi) +ℓ(A(S(i)),z′))
,
where in the last step we used the inequality ( a+b)2≤3(a2+b2). Taking expecta-
tion with respect to S, z′, iand noting that E[ℓ(A(S),zi)] =E[ℓ(A(S(i)), z′)] =
E[LS(A(S))], we conclude that:
corollary 13.7 Assume that the loss function is β-smooth and nonnegative.
Then, the RLM rule with the regularizer λ∥w∥2, whereλ≥2β
m, satisﬁes
E[
ℓ(A(S(i)),zi)−ℓ(A(S),zi)]
≤48β
λmE[LS(A(S))].
Note that if for all zwe haveℓ(0, z)≤C, for some scalar C > 0, then for
everyS,
LS(A(S))≤LS(A(S)) +λ∥A(S)∥2≤LS(0) +λ∥0∥2=LS(0)≤C.
Hence, Corollary 13.7 also implies that
E[
≤48βC
λm.
13.4 Controlling the Fitting-Stability Tradeoﬀ
We can rewrite the expected risk of a learning algorithm as

E[
≤48β
everyS,
E[
≤48βC
λm.
E
S[LD(A(S))] =E
S[LS(A(S))] +E
S[LD(A(S))−LS(A(S))]. (13.15)
The ﬁrst term reﬂects how well A(S) ﬁts the training set while the second term
reﬂects the diﬀerence between the true and empirical risks of A(S). As we have
shown in Theorem 13.2, the second term is equivalent to the stability of A. Since
our goal is to minimize the risk of the algorithm, we need that the sum of both
terms will be small.
In the previous section we have bounded the stability term. We have shown

E
that the stability term decreases as the regularization parameter, λ, increases.
On the other hand, the empirical risk increases with λ. We therefore face a

13.4 Controlling the Fitting-Stability Tradeoﬀ 179
tradeoﬀ between ﬁtting and overﬁtting. This tradeoﬀ is quite similar to the bias-
complexity tradeoﬀ we discussed previously in the book.
We now derive bounds on the empirical risk term for the RLM rule. Recall
that the RLM rule is deﬁned as A(S) = argminw(
. Fix some
arbitrary vector w∗. We have
LS(A(S))≤LS(A(S)) +λ∥A(S)∥2≤LS(w∗) +λ∥w∗∥2.
Taking expectation of both sides with respect to Sand noting that ES[LS(w∗)] =
LD(w∗), we obtain that
E
S[LS(A(S))]≤LD(w∗) +λ∥w∗∥2. (13.16)
Plugging this into Equation (13.15) we obtain
E
S[LD(A(S))]≤LD(w∗) +λ∥w∗∥2+E
S[LD(A(S))−LS(A(S))].
Combining the preceding with Corollary 13.6 we conclude:
corollary 13.8 Assume that the loss function is convex and ρ-Lipschitz.
Then, the RLM rule with the regularization function λ∥w∥2satisﬁes
∀w∗,E
S[LD(A(S))]≤LD(w∗) +λ∥w∗∥2+2ρ2
λm.
This bound is often called an oracle inequality – if we think of w∗as a hy-

E
E
∀w∗,E
λm.
pothesis with low risk, the bound tells us how many examples are needed so that
A(S) will be almost as good as w∗, had we known the norm of w∗. In practice,
however, we usually do not know the norm of w∗. We therefore usually tune λ
on the basis of a validation set, as described in Chapter 11.
We can also easily derive a PAC-like guarantee1from Corollary 13.8 for convex-
Lipschitz-bounded learning problems:
corollary 13.9 Let(H,Z,ℓ)be a convex-Lipschitz-bounded learning problem
with parameters ρ,B. For any training set size m, letλ=√
2ρ2

2ρ2
B2m. Then, the
RLM rule with the regularization function λ∥w∥2satisﬁes
E
S[LD(A(S))]≤min
w∈HLD(w) +ρB√
8
m.
In particular, for every ϵ > 0, ifm≥8ρ2B2
ϵ2then for every distribution D,
ES[LD(A(S))]≤minw∈HLD(w) +ϵ.
The preceding corollary holds for Lipschitz loss functions. If instead the loss
function is smooth and nonnegative, then we can combine Equation (13.16) with
Corollary 13.7 to get:
1Again, the bound below is on the expected risk, but using Exercise 1 it can be used to
derive an agnostic PAC learning guarantee.

180 Regularization and Stability
corollary 13.10 Assume that the loss function is convex, β-smooth, and
nonnegative. Then, the RLM rule with the regularization function λ∥w∥2, for
λ≥2β
m, satisﬁes the following for all w∗:
E
S[LD(A(S))]≤(
1 +48β
λm)
E
S[LS(A(S))]≤(
1 +48β
λm)(
LD(w∗) +λ∥w∗∥2)
.
For example, if we choose λ=48β
mwe obtain from the preceding that the
expected true risk of A(S) is at most twice the expected empirical risk of A(S).
Furthermore, for this value of λ, the expected empirical risk of A(S) is at most
LD(w∗) +48β
m∥w∗∥2.
We can also derive a learnability guarantee for convex-smooth-bounded learn-
ing problems based on Corollary 13.10.
corollary 13.11 Let(H,Z,ℓ)be a convex-smooth-bounded learning problem
with parameters β,B. Assume in addition that ℓ(0,z)≤1for allz∈Z. For any
ϵ∈(0,1)letm≥150βB2
ϵ2and setλ=ϵ/(3B2). Then, for every distribution D,
E
w∈HLD(w) +ϵ .
13.5 Summary
We introduced stability and showed that if an algorithm is stable then it does not

m∥w∗∥2.
E
overﬁt. Furthermore, for convex-Lipschitz-bounded or convex-smooth-bounded
problems, the RLM rule with Tikhonov regularization leads to a stable learning
algorithm. We discussed how the regularization parameter, λ, controls the trade-
oﬀ between ﬁtting and overﬁtting. Finally, we have shown that all learning prob-
lems that are from the families of convex-Lipschitz-bounded and convex-smooth-
bounded problems are learnable using the RLM rule. The RLM paradigm is the

basis for many popular learning algorithms, including ridge regression (which we
discussed in this chapter) and support vector machines (which will be discussed
in Chapter 15).
In the next chapter we will present Stochastic Gradient Descent, which gives us
a very practical alternative way to learn convex-Lipschitz-bounded and convex-
smooth-bounded problems and can also be used for eﬃciently implementing the
RLM rule.
13.6 Bibliographic Remarks
Stability is widely used in many mathematical contexts. For example, the neces-

RLM rule.
sity of stability for so-called inverse problems to be well posed was ﬁrst recognized
by Hadamard (1902). The idea of regularization and its relation to stability be-
came widely known through the works of Tikhonov (1943) and Phillips (1962).

In the context of modern learning theory, the use of stability can be traced back
at least to the work of Rogers & Wagner (1978), which noted that the sensitiv-
ity of a learning algorithm with regard to small changes in the sample controls
the variance of the leave-one-out estimate. The authors used this observation to
obtain generalization bounds for the k-nearest neighbor algorithm (see Chap-
ter 19). These results were later extended to other “local” learning algorithms
(see Devroye, Gy¨ orﬁ & Lugosi (1996) and references therein). In addition, practi-
cal methods have been developed to introduce stability into learning algorithms,
in particular the Bagging technique introduced by (Breiman 1996).
Over the last decade, stability was studied as a generic condition for learnabil-
ity. See (Kearns & Ron 1999, Bousquet & Elisseeﬀ 2002, Kutin & Niyogi 2002,
Rakhlin, Mukherjee & Poggio 2005, Mukherjee, Niyogi, Poggio & Rifkin 2006).

Our presentation follows the work of Shalev-Shwartz, Shamir, Srebro & Sridha-
ran (2010), who showed that stability is suﬃcient and necessary for learning.
They have also shown that all convex-Lipschitz-bounded learning problems are
learnable using RLM, even though for some convex-Lipschitz-bounded learning
problems uniform convergence does not hold in a strong sense.
13.7 Exercises
1.From Bounded Expected Risk to Agnostic PAC Learning: LetAbe
an algorithm that guarantees the following: If m≥mH(ϵ) then for every
distributionDit holds that
E

E
•Show that for every δ∈(0,1), ifm≥mH(ϵδ) then with probability of at
least 1−δit holds that LD(A(S))≤minh∈HLD(h) +ϵ.
Hint: Observe that the random variable LD(A(S))−minh∈HLD(h) is
nonnegative and rely on Markov’s inequality.
•For everyδ∈(0,1) let
mH(ϵ,δ) =mH(ϵ/2)⌈log2(1/δ)⌉+⌈log(4/δ) + log(⌈log2(1/δ)⌉)
ϵ2⌉
.
Suggest a procedure that agnostic PAC learns the problem with sample
complexity of mH(ϵ,δ), assuming that the loss function is bounded by
1.
Hint: Letk=⌈log2(1/δ)⌉. Divide the data into k+1 chunks, where each

ϵ2⌉
.
1.
of the ﬁrstkchunks is of size mH(ϵ/2) examples. Train the ﬁrst kchunks
usingA. On the basis of the previous question argue that the probability
that for all of these chunks we have LD(A(S))>minh∈HLD(h) +ϵis
at most 2−k≤δ/2. Finally, use the last chunk as a validation set.
2.Learnability without Uniform Convergence: LetBbe the unit ball of

182 Regularization and Stability
Rd, letH=B, letZ=B×{ 0,1}d, and letℓ:Z×H→ Rbe deﬁned as
follows:
ℓ(w,(x,α)) =d∑
i=1αi(xi−wi)2.
This problem corresponds to an unsupervised learning task, meaning that we
do not try to predict the label of x. Instead, what we try to do is to ﬁnd the
“center of mass” of the distribution over B. However, there is a twist, modeled
by the vectors α. Each example is a pair ( x,α), where xis the instance xand
αindicates which features of xare “active” and which are “turned oﬀ.” A
hypothesis is a vector wrepresenting the center of mass of the distribution,
and the loss function is the squared Euclidean distance between xandw, but
only with respect to the “active” elements of x.
•Show that this problem is learnable using the RLM rule with a sample
complexity that does not depend on d.
•Consider a distribution DoverZas follows: xis ﬁxed to be some x0, and
each element of αis sampled to be either 1 or 0 with equal probability.

Show that the rate of uniform convergence of this problem grows with
d.
Hint: Letmbe a training set size. Show that if d≫2m, then there is
a high probability of sampling a set of examples such that there exists
somej∈[d] for which αj= 1 for all the examples in the training set.
Show that such a sample cannot be ϵ-representative. Conclude that the
sample complexity of uniform convergence must grow with log( d).
•Conclude that if we take dto inﬁnity we obtain a problem that is learnable
but for which the uniform convergence property does not hold. Compare

d.
to the fundamental theorem of statistical learning.
3.Stability and Asymptotic ERM Are Suﬃcient for Learnability:
We say that a learning rule Ais an AERM (Asymptotic Empirical Risk
Minimizer) with rateϵ(m) if for every distribution Dit holds that
E
S∼Dm[
LS(A(S))−min
h∈HLS(h)]
≤ϵ(m).
We say that a learning rule Alearns a classHwith rateϵ(m) if for every
E
S∼Dm[
LD(A(S))−min
h∈HLD(h)]
≤ϵ(m).
Prove the following:
theorem 13.12 If a learning algorithm Ais on-average-replace-one-stable

E
S∼Dm[
h∈HLS(h)]
≤ϵ(m).
E
S∼Dm[
h∈HLD(h)]
≤ϵ(m).
with rateϵ1(m)and is an AERM with rate ϵ2(m), then it learnsHwith rate
ϵ1(m) +ϵ2(m).

13.7 Exercises 183
4.Strong Convexity with Respect to General Norms:
Throughout the section we used the ℓ2norm. In this exercise we generalize
some of the results to general norms. Let ∥·∥be some arbitrary norm, and let f
be a strongly convex function with respect to this norm (see Deﬁnition 13.4).
1. Show that items 2–3 of Lemma 13.5 hold for every norm.
2.(*)Give an example of a norm for which item 1 of Lemma 13.5 does not
hold.
3. LetR(w) be a function that is (2 λ)-strongly convex with respect to some
norm∥·∥. LetAbe an RLM rule with respect to R, namely,
w(LS(w) +R(w)).
Assume that for every z, the loss function ℓ(·,z) isρ-Lipschitz with respect
to the same norm, namely,
∀z,∀w,v, ℓ(w,z)−ℓ(v,z)≤ρ∥w−v∥.
Prove that Ais on-average-replace-one-stable with rate2ρ2
λm.
4.(*)Letq∈(1,2) and consider the ℓq-norm
∥w∥q=(d∑
i=1|wi|q)1/q
.
It can be shown (see, for example, Shalev-Shwartz (2007)) that the function
R(w) =1
2(q−1)∥w∥2
q
is 1-strongly convex with respect to ∥w∥q. Show that if q=log(d)

λm.
∥w∥q=(d∑
.
R(w) =1
q
log(d)−1then
R(w) is(
1
3 log(d))
-strongly convex with respect to the ℓ1norm over Rd.

14 Stochastic Gradient Descent
Recall that the goal of learning is to minimize the risk function, LD(h) =
Ez∼D[ℓ(h,z)]. We cannot directly minimize the risk function since it depends
on the unknown distribution D. So far in the book, we have discussed learning
methods that depend on the empirical risk. That is, we ﬁrst sample a training
setSand deﬁne the empirical risk function LS(h). Then, the learner picks a
hypothesis based on the value of LS(h). For example, the ERM rule tells us to
pick the hypothesis that minimizes LS(h) over the hypothesis class, H. Or, in the
previous chapter, we discussed regularized risk minimization, in which we pick a
hypothesis that jointly minimizes LS(h) and a regularization function over h.
In this chapter we describe and analyze a rather diﬀerent learning approach,
which is called Stochastic Gradient Descent (SGD). As in Chapter 12 we will
focus on the important family of convex learning problems, and following the

notation in that chapter, we will refer to hypotheses as vectors wthat come from
a convex hypothesis class, H. In SGD, we try to minimize the risk function LD(w)
directly using a gradient descent procedure. Gradient descent is an iterative
optimization procedure in which at each step we improve the solution by taking
a step along the negative of the gradient of the function to be minimized at
the current point. Of course, in our case, we are minimizing the risk function,
and since we do not know Dwe also do not know the gradient of LD(w). SGD

circumvents this problem by allowing the optimization procedure to take a step
along a random direction, as long as the expected value of the direction is the
negative of the gradient. And, as we shall see, ﬁnding a random direction whose
expected value corresponds to the gradient is rather simple even though we do
not know the underlying distribution D.
The advantage of SGD, in the context of convex learning problems, over the
regularized risk minimization learning rule is that SGD is an eﬃcient algorithm

that can be implemented in a few lines of code, yet still enjoys the same sample
complexity as the regularized risk minimization rule. The simplicity of SGD also
allows us to use it in situations when it is not possible to apply methods that
are based on the empirical risk, but this is beyond the scope of this book.
We start this chapter with the basic gradient descent algorithm and analyze its
convergence rate for convex-Lipschitz functions. Next, we introduce the notion of

subgradient and show that gradient descent can be applied for nondiﬀerentiable
functions as well. The core of this chapter is Section 14.3, in which we describe

the Stochastic Gradient Descent algorithm, along with several useful variants.
We show that SGD enjoys an expected convergence rate similar to the rate
of gradient descent. Finally, we turn to the applicability of SGD to learning
problems.
14.1 Gradient Descent
Before we describe the stochastic gradient descent method, we would like to
describe the standard gradient descent approach for minimizing a diﬀerentiable
convex function f(w).
The gradient of a diﬀerentiable function f:Rd→Ratw, denoted∇f(w),
is the vector of partial derivatives of f, namely,∇f(w) =(
∂f(w)
∂w[1],...,∂f(w)
∂w[d])
.
Gradient descent is an iterative algorithm. We start with an initial value of w
(say,w(1)=0). Then, at each iteration, we take a step in the direction of the
negative of the gradient at the current point. That is, the update step is
w(t+1)=w(t)−η∇f(w(t)), (14.1)
whereη > 0 is a parameter to be discussed later. Intuitively, since the gradi-

∂f(w)
∂w[d])
.
ent points in the direction of the greatest rate of increase of faround w(t),
the algorithm makes a small step in the opposite direction, thus decreasing the
value of the function. Eventually, after Titerations, the algorithm outputs the
averaged vector, ¯w=1
T∑T
t=1w(t). The output could also be the last vector,
w(T), or the best performing vector, argmint∈[T]f(w(t)), but taking the average
turns out to be rather useful, especially when we generalize gradient descent to

T∑T
nondiﬀerentiable functions and to the stochastic case.
Another way to motivate gradient descent is by relying on Taylor approxima-
tion. The gradient of fatwyields the ﬁrst order Taylor approximation of f
around wbyf(u)≈f(w) +⟨u−w,∇f(w)⟩. Whenfis convex, this approxi-
mation lower bounds f, that is,
f(u)≥f(w) +⟨u−w,∇f(w)⟩.
Therefore, for wclose to w(t)we have that f(w)≈f(w(t))+⟨w−w(t),∇f(w(t))⟩.
Hence we can minimize the approximation of f(w). However, the approximation

might become loose for w, which is far away from w(t). Therefore, we would like
to minimize jointly the distance between wandw(t)and the approximation of
faround w(t). If the parameter ηcontrols the tradeoﬀ between the two terms,
we obtain the update rule
w(t+1)= argmin
w1
2∥w−w(t)∥2+η(
f(w(t)) +⟨w−w(t),∇f(w(t))⟩)
.
Solving the preceding by taking the derivative with respect to wand comparing
it to zero yields the same update rule as in Equation (14.1).

186 Stochastic Gradient Descent
Figure 14.1 An illustration of the gradient descent algorithm. The function to be
minimized is 1 .25(x1+ 6)2+ (x2−8)2.
14.1.1 Analysis of GD for Convex-Lipschitz Functions
To analyze the convergence rate of the GD algorithm, we limit ourselves to
the case of convex-Lipschitz functions (as we have seen, many problems lend
themselves easily to this setting). Let w⋆be any vector and let Bbe an upper
bound on∥w⋆∥. It is convenient to think of w⋆as the minimizer of f(w), but
the analysis that follows holds for every w⋆.
We would like to obtain an upper bound on the suboptimality of our solution
with respect to w⋆, namely,f(¯w)−f(w⋆), where ¯w=1
T∑T
t=1w(t). From the
deﬁnition of ¯w, and using Jensen’s inequality, we have that
f(¯w)−f(w⋆) =f(
1
TT∑
t=1w(t))
−f(w⋆)
≤1
TT∑
t=1(
f(w(t)))
−f(w⋆)
=1
TT∑
t=1(
f(w(t))−f(w⋆))
. (14.2)
For everyt, because of the convexity of f, we have that
f(w(t))−f(w⋆)≤ ⟨w(t)−w⋆,∇f(w(t))⟩. (14.3)
Combining the preceding we obtain
f(¯w)−f(w⋆)≤1
TT∑

T∑T
1
TT∑
t=1w(t))
−f(w⋆)
≤1
TT∑
t=1(
f(w(t)))
−f(w⋆)
=1
TT∑
t=1(
. (14.2)
TT∑
t=1⟨w(t)−w⋆,∇f(w(t))⟩.
To bound the right-hand side we rely on the following lemma:

14.1 Gradient Descent 187
lemma 14.1 Letv1,...,vTbe an arbitrary sequence of vectors. Any algorithm
with an initialization w(1)= 0and an update rule of the form
w(t+1)=w(t)−ηvt (14.4)
satisﬁes
T∑
t=1⟨w(t)−w⋆,vt⟩≤∥w⋆∥2
2η+η
2T∑
t=1∥vt∥2. (14.5)
In particular, for every B,ρ> 0, if for alltwe have that∥vt∥≤ρand if we set
η=√
B2
ρ2T, then for every w⋆with∥w⋆∥≤Bwe have
1
TT∑
t=1⟨w(t)−w⋆,vt⟩≤Bρ√
T.
Proof Using algebraic manipulations (completing the square), we obtain:
⟨w(t)−w⋆,vt⟩=1
η⟨w(t)−w⋆,ηvt⟩
=1
2η(−∥w(t)−w⋆−ηvt∥2+∥w(t)−w⋆∥2+η2∥vt∥2)
=1
2η(−∥w(t+1)−w⋆∥2+∥w(t)−w⋆∥2) +η
2∥vt∥2,
where the last equality follows from the deﬁnition of the update rule. Summing
the equality over t, we have
T∑
t=1⟨w(t)−w⋆,vt⟩=1
2ηT∑
t=1(
−∥w(t+1)−w⋆∥2+∥w(t)−w⋆∥2)
+η
2T∑
t=1∥vt∥2.
(14.6)
The ﬁrst sum on the right-hand side is a telescopic sum that collapses to
∥w(1)−w⋆∥2−∥w(T+1)−w⋆∥2.
Plugging this in Equation (14.6), we have
T∑
2η(∥w(1)−w⋆∥2−∥w(T+1)−w⋆∥2) +η
2T∑
t=1∥vt∥2
≤1
2η∥w(1)−w⋆∥2+η
2T∑
t=1∥vt∥2
=1
2η∥w⋆∥2+η

=1
2∥vt∥2,
T∑
2ηT∑
t=1(
+η
2T∑
t=1∥vt∥2.
(14.6)
T∑
2T∑
t=1∥vt∥2
≤1
2T∑
t=1∥vt∥2
=1
2η∥w⋆∥2+η
2T∑
t=1∥vt∥2,
where the last equality is due to the deﬁnition w(1)= 0. This proves the ﬁrst
part of the lemma (Equation (14.5)). The second part follows by upper bounding
∥w⋆∥byB,∥vt∥byρ, dividing by T, and plugging in the value of η.

188 Stochastic Gradient Descent
Lemma 14.1 applies to the GD algorithm with vt=∇f(w(t)). As we will
show later in Lemma 14.7, if fisρ-Lipschitz, then∥∇f(w(t))∥≤ρ. We therefore
satisfy the lemma’s conditions and achieve the following corollary:
corollary 14.2 Letfbe a convex, ρ-Lipschitz function, and let w⋆∈argmin{w:∥w∥≤B}f(w).
If we run the GD algorithm on fforTsteps withη=√
B2
ρ2T, then the output
vector ¯wsatisﬁes
f(¯w)−f(w⋆)≤Bρ√
T.
Furthermore, for every ϵ>0, to achieve f(¯w)−f(w⋆)≤ϵ, it suﬃces to run the
GD algorithm for a number of iterations that satisﬁes
T≥B2ρ2
ϵ2.
14.2 Subgradients
The GD algorithm requires that the function fbe diﬀerentiable. We now gener-
alize the discussion beyond diﬀerentiable functions. We will show that the GD
algorithm can be applied to nondiﬀerentiable functions by using a so-called sub-
gradient of f(w) atw(t), instead of the gradient.
To motivate the deﬁnition of subgradients, recall that for a convex function f,

T≥B2ρ2
ϵ2.
the gradient at wdeﬁnes the slope of a tangent that lies below f, that is,
∀u, f(u)≥f(w) +⟨u−w,∇f(w)⟩. (14.7)
An illustration is given on the left-hand side of Figure 14.2.
The existence of a tangent that lies below fis an important property of convex
functions, which is in fact an alternative characterization of convexity.
lemma 14.3 LetSbe an open convex set. A function f:S→Ris convex iﬀ
for every w∈Sthere exists vsuch that
∀u∈S, f (u)≥f(w) +⟨u−w,v⟩. (14.8)
The proof of this lemma can be found in many convex analysis textbooks (e.g.,

(Borwein & Lewis 2006)). The preceding inequality leads us to the deﬁnition of
subgradients.
definition 14.4 (Subgradients) A vector vthat satisﬁes Equation (14.8) is
called a subgradient offatw. The set of subgradients of fatwis called the
diﬀerential set and denoted ∂f(w).
An illustration of subgradients is given on the right-hand side of Figure 14.2.
For scalar functions, a subgradient of a convex function fatwis a slope of a
line that touches fatwand is not above felsewhere.

14.2 Subgradients 189
f(w)f(u)
Figure 14.2 Left: The right-hand side of Equation (14.7) is the tangent of fatw. For
a convex function, the tangent lower bounds f. Right: Illustration of several
subgradients of a nondiﬀerentiable convex function.
14.2.1 Calculating Subgradients
How do we construct subgradients of a given convex function? If a function is
diﬀerentiable at a point w, then the diﬀerential set is trivial, as the following
claim shows.
claim 14.5 Iffis diﬀerentiable at wthen∂f(w)contains a single element –
the gradient of fatw,∇f(w).
Example 14.1 (The Diﬀerential Set of the Absolute Function) Consider the
absolute value function f(x) =|x|. Using Claim 14.5, we can easily construct
the diﬀerential set for the diﬀerentiable parts of f, and the only point that
requires special attention is x0= 0. At that point, it is easy to verify that the
subdiﬀerential is the set of all numbers between −1 and 1. Hence:
∂f(x) =

{1} ifx>0
{−1}ifx<0
[−1,1] ifx= 0

∂f(x) =

{−1}ifx<0
For many practical uses, we do not need to calculate the whole set of subgra-
dients at a given point, as one member of this set would suﬃce. The following
claim shows how to construct a sub-gradient for pointwise maximum functions.
claim 14.6 Letg(w) = maxi∈[r]gi(w)forrconvex diﬀerentiable functions
g1,...,gr. Given some w, letj∈argmaxigi(w). Then∇gj(w)∈∂g(w).
Proof Sincegjis convex we have that for all u
gj(u)≥gj(w) +⟨u−w,∇gj(w)⟩.
Sinceg(w) =gj(w) andg(u)≥gj(u) we obtain that
g(u)≥g(w) +⟨u−w,∇gj(w)⟩,

190 Stochastic Gradient Descent
Example 14.2 (A Subgradient of the Hinge Loss) Recall the hinge loss function
from Section 12.3, f(w) = max{0,1−y⟨w,x⟩}for some vector xand scalary.
To calculate a subgradient of the hinge loss at some wwe rely on the preceding
claim and obtain that the vector vdeﬁned in the following is a subgradient of
the hinge loss at w:
v={
0 if 1−y⟨w,x⟩≤0
−yxif 1−y⟨w,x⟩>0
14.2.2 Subgradients of Lipschitz Functions
Recall that a function f:A→Risρ-Lipschitz if for all u,v∈A
|f(u)−f(v)|≤ρ∥u−v∥.
The following lemma gives an equivalent deﬁnition using norms of subgradients.
lemma 14.7 LetAbe a convex open set and let f:A→Rbe a convex function.
Then,fisρ-Lipschitz over Aiﬀ for all w∈Aandv∈∂f(w)we have that
∥v∥≤ρ.
Proof Assume that for all v∈∂f(w) we have that∥v∥≤ρ. Since v∈∂f(w)
we have
f(w)−f(u)≤⟨v,w−u⟩.
Bounding the right-hand side using Cauchy-Schwartz inequality we obtain
f(w)−f(u)≤ ⟨v,w−u⟩≤∥v∥∥w−u∥≤ρ∥w−u∥.
An analogous argument can show that f(u)−f(w)≤ρ∥w−u∥. Hencefis
ρ-Lipschitz.

∥v∥≤ρ.
we have
Now assume that fisρ-Lipschitz. Choose some w∈A,v∈∂f(w). SinceA
is open, there exists ϵ >0 such that u=w+ϵv/∥v∥belongs toA. Therefore,
⟨u−w,v⟩=ϵ∥v∥and∥u−w∥=ϵ. From the deﬁnition of the subgradient,
f(u)−f(w)≥⟨v,u−w⟩=ϵ∥v∥.
On the other hand, from the Lipschitzness of fwe have
ρϵ=ρ∥u−w∥≥f(u)−f(w).
Combining the two inequalities we conclude that ∥v∥≤ρ.
14.2.3 Subgradient Descent
The gradient descent algorithm can be generalized to nondiﬀerentiable functions

by using a subgradient of f(w) atw(t), instead of the gradient. The analysis of
the convergence rate remains unchanged: Simply note that Equation (14.3) is
true for subgradients as well.

Figure 14.3 An illustration of the gradient descent algorithm (left) and the stochastic
gradient descent algorithm (right). The function to be minimized is
1.25(x+ 6)2+ (y−8)2. For the stochastic case, the black line depicts the averaged
value of w.
14.3 Stochastic Gradient Descent (SGD)
In stochastic gradient descent we do not require the update direction to be based
exactly on the gradient. Instead, we allow the direction to be a random vector
and only require that its expected value at each iteration will equal the gradient
direction. Or, more generally, we require that the expected value of the random
vector will be a subgradient of the function at the current vector.
Stochastic Gradient Descent (SGD) for minimizing
f(w)
parameters: Scalarη>0, integerT >0
initialize: w(1)=0
fort= 1,2,...,T
choose vtat random from a distribution such that E[vt|w(t)]∈∂f(w(t))
update w(t+1)=w(t)−ηvt
output ¯w=1
T∑T
t=1w(t)

f(w)
T∑T
t=1w(t)
An illustration of stochastic gradient descent versus gradient descent is given
in Figure 14.3. As we will see in Section 14.5, in the context of learning problems,
it is easy to ﬁnd a random vector whose expectation is a subgradient of the risk
function.
14.3.1 Analysis of SGD for Convex-Lipschitz-Bounded Functions
Recall the bound we achieved for the GD algorithm in Corollary 14.2. For the
stochastic case, in which only the expectation of vtis in∂f(w(t)), we cannot
directly apply Equation (14.3). However, since the expected value of vtis a

192 Stochastic Gradient Descent
subgradient of fatw(t), we can still derive a similar bound on the expected
output of stochastic gradient descent. This is formalized in the following theorem.
theorem 14.8 LetB,ρ> 0. Letfbe a convex function and let w⋆∈argminw:∥w∥≤Bf(w).
Assume that SGD is run for Titerations with η=√
B2
ρ2T. Assume also that for
allt,∥vt∥≤ρwith probability 1. Then,
E[f(¯w)]−f(w⋆)≤Bρ√
T.
Therefore, for any ϵ>0, to achieve E[f(¯w)]−f(w⋆)≤ϵ, it suﬃces to run the
SGD algorithm for a number of iterations that satisﬁes
T≥B2ρ2
ϵ2.
Proof Let us introduce the notation v1:tto denote the sequence v1,...,vt.
Taking expectation of Equation (14.2), we obtain
E
v1:T[f(¯w)−f(w⋆)]≤E
v1:T[
1
TT∑
t=1(f(w(t))−f(w⋆))]
.
Since Lemma 14.1 holds for any sequence v1,v2,...vT, it applies to SGD as well.
By taking expectation of the bound in the lemma we have
E
v1:T[
1
TT∑
t=1⟨w(t)−w⋆,vt⟩]
≤Bρ√
T. (14.9)
It is left to show that
E
v1:T[
1
TT∑
≤E
v1:T[
1
TT∑
, (14.10)

T≥B2ρ2
ϵ2.
E
v1:T[
1
TT∑
.
E
v1:T[
1
TT∑
≤Bρ√
T. (14.9)
E
v1:T[
1
TT∑
≤E
v1:T[
1
TT∑
, (14.10)
which we will hereby prove.
Using the linearity of the expectation we have
E
v1:T[
1
TT∑
=1
TT∑
t=1E
v1:T[⟨w(t)−w⋆,vt⟩].
Next, we recall the law of total expectation : For every two random variables α,β,
and a function g,Eα[g(α)] =EβEα[g(α)|β].Settingα=v1:tandβ=v1:t−1we
get that
E
v1:T[⟨w(t)−w⋆,vt⟩] =E
v1:t[⟨w(t)−w⋆,vt⟩]
=E
v1:t−1E
v1:t[⟨w(t)−w⋆,vt⟩|v1:t−1].
Once we know v1:t−1, the value of w(t)is not random any more and therefore
E
v1:t−1E
v1:t[⟨w(t)−w⋆,vt⟩|v1:t−1] =E
v1:t−1⟨w(t)−w⋆,E
vt[vt|v1:t−1]⟩.

Since w(t)only depends on v1:t−1and SGD requires that Evt[vt|w(t)]∈∂f(w(t))
we obtain that Evt[vt|v1:t−1]∈∂f(w(t)). Thus,
E
vt[vt|v1:t−1]⟩ ≥ E
v1:t−1[f(w(t))−f(w⋆)].
E
v1:T[⟨w(t)−w⋆,vt⟩]≥E
v1:t−1[f(w(t))−f(w⋆)]
=E
v1:T[f(w(t))−f(w⋆)].
Summing over t, dividing by T, and using the linearity of expectation, we get
that Equation (14.10) holds, which concludes our proof.
14.4 Variants
In this section we describe several variants of Stochastic Gradient Descent.
14.4.1 Adding a Projection Step
In the previous analyses of the GD and SGD algorithms, we required that the
norm of w⋆will be at most B, which is equivalent to requiring that w⋆is in the
setH={w:∥w∥≤B}. In terms of learning, this means restricting ourselves to
aB-bounded hypothesis class. Yet any step we take in the opposite direction of
the gradient (or its expected direction) might result in stepping out of this bound,

and there is even no guarantee that ¯wsatisﬁes it. We show in the following how
to overcome this problem while maintaining the same convergence rate.
The basic idea is to add a projection step ; namely, we will now have a two-step
update rule, where we ﬁrst subtract a subgradient from the current value of w
and then project the resulting vector onto H. Formally,
1..w(t+1
2)=w(t)−ηvt
2..w(t+1)= argminw∈H∥w−w(t+1
2)∥
The projection step replaces the current value of wby the vector inHclosest
to it.

1..w(t+1
2)∥
to it.
Clearly, the projection step guarantees that w(t)∈H for allt. SinceHis
convex this also implies that ¯w∈H as required. We next show that the analysis
of SGD with projections remains the same. This is based on the following lemma.
lemma 14.9 (Projection Lemma) LetHbe a closed convex set and let vbe the
projection of wontoH, namely,
v= argmin
x∈H∥x−w∥2.

194 Stochastic Gradient Descent
Then, for every u∈H,
∥w−u∥2−∥v−u∥2≥0.
Proof By the convexity of H, for everyα∈(0,1) we have that v+α(u−v)∈H.
Therefore, from the optimality of vwe obtain
∥v−w∥2≤∥v+α(u−v)−w∥2
=∥v−w∥2+ 2α⟨v−w,u−v⟩+α2∥u−v∥2.
Rearranging, we obtain
2⟨v−w,u−v⟩≥−α∥u−v∥2.
Taking the limit α→0 we get that
⟨v−w,u−v⟩≥0.
∥w−u∥2=∥w−v+v−u∥2
=∥w−v∥2+∥v−u∥2+ 2⟨v−w,u−v⟩
≥∥v−u∥2.
Equipped with the preceding lemma, we can easily adapt the analysis of SGD
to the case in which we add projection steps on a closed and convex set. Simply
note that for every t,
∥w(t+1)−w⋆∥2−∥w(t)−w⋆∥2
=∥w(t+1)−w⋆∥2−∥w(t+1
2)−w⋆∥2+∥w(t+1
2)−w⋆∥2−∥w(t)−w⋆∥2
≤∥w(t+1
2)−w⋆∥2−∥w(t)−w⋆∥2.
Therefore, Lemma 14.1 holds when we add projection steps and hence the rest
of the analysis follows directly.
14.4.2 Variable Step Size
Another variant of SGD is decreasing the step size as a function of t. That is,
rather than updating with a constant η, we useηt. For instance, we can set
ηt=B
ρ√

≤∥w(t+1
ηt=B
ρ√
tand achieve a bound similar to Theorem 14.8. The idea is that when
we are closer to the minimum of the function, we take our steps more carefully,
so as not to “overshoot” the minimum.

14.4 Variants 195
14.4.3 Other Averaging Techniques
We have set the output vector to be ¯w=1
T∑T
t=1w(t). There are alternative
approaches such as outputting w(t)for some random t∈[t], or outputting the
average of w(t)over the last αTiterations, for some α∈(0,1). One can also take
a weighted average of the last few iterates. These more sophisticated averaging
schemes can improve the convergence speed in some situations, such as in the
case of strongly convex functions deﬁned in the following.
14.4.4 Strongly Convex Functions*
In this section we show a variant of SGD that enjoys a faster convergence rate for
problems in which the objective function is strongly convex (see Deﬁnition 13.4
of strong convexity in the previous chapter). We rely on the following claim,
which generalizes Lemma 13.5.
claim 14.10 Iffisλ-strongly convex then for every w,uandv∈∂f(w)we
have
⟨w−u,v⟩ ≥f(w)−f(u) +λ
2∥w−u∥2.
The proof is similar to the proof of Lemma 13.5 and is left as an exercise.

have
2∥w−u∥2.
SGD for minimizing a λ-strongly convex function
Goal: Solve min w∈Hf(w)
parameter: T
fort= 1,...,T
Choose a random vector vts.t.E[vt|w(t)]∈∂f(w(t))
Setηt= 1/(λt)
Setw(t+1
2)=w(t)−ηtvt
Setw(t+1)= arg min w∈H∥w−w(t+1
2)∥2
output: ¯w=1
T∑T
t=1w(t)
theorem 14.11 Assume that fisλ-strongly convex and that E[∥vt∥2]≤ρ2.
Letw⋆∈argminw∈Hf(w)be an optimal solution. Then,
E[f(¯w)]−f(w⋆)≤ρ2
2λT(1 + log(T)).
Proof Let∇(t)=E[vt|w(t)]. Sincefis strongly convex and ∇(t)is in the
subgradient set of fatw(t)we have that

Setw(t+1
2)∥2
T∑T
t=1w(t)
⟨w(t)−w⋆,∇(t)⟩ ≥f(w(t))−f(w⋆) +λ
2∥w(t)−w⋆∥2. (14.11)
Next, we show that
⟨w(t)−w⋆,∇(t)⟩ ≤E[∥w(t)−w⋆∥2−∥w(t+1)−w⋆∥2]
2ηt+ηt
2ρ2.(14.12)

196 Stochastic Gradient Descent
Since w(t+1)is the projection of w(t+1
2)ontoH, and w⋆∈H we have that
∥w(t+1
2)−w⋆∥2≥∥w(t+1)−w⋆∥2. Therefore,
∥w(t)−w⋆∥2−∥w(t+1)−w⋆∥2≥∥w(t)−w⋆∥2−∥w(t+1
2)−w⋆∥2
= 2ηt⟨w(t)−w⋆,vt⟩−η2
t∥vt∥2.
Taking expectation of both sides, rearranging, and using the assumption E[∥vt∥2]≤
ρ2yield Equation (14.12). Comparing Equation (14.11) and Equation (14.12) and
summing over twe obtain
T∑
t=1(E[f(w(t))]−f(w⋆))
≤E[T∑
t=1(∥w(t)−w⋆∥2−∥w(t+1)−w⋆∥2
2ηt−λ
2∥w(t)−w⋆∥2)]
+ρ2
2T∑
t=1ηt.
Next, we use the deﬁnition ηt= 1/(λt) and note that the ﬁrst sum on the
right-hand side of the equation collapses to −λT∥w(T+1)−w⋆∥2≤0. Thus,
T∑
t=1(E[f(w(t))]−f(w⋆))≤ρ2
2λT∑
t=11
t≤ρ2
2λ(1 + log(T)).
The theorem follows from the preceding by dividing by Tand using Jensen’s
Remark 14.3 Rakhlin, Shamir & Sridharan (2012) derived a convergence rate
in which the log( T) term is eliminated for a variant of the algorithm in which
we output the average of the last T/2 iterates, ¯w=2
T∑T
t=T/2+1w(t). Shamir &

T∑
2λT∑
t=11
t≤ρ2
T∑T
Zhang (2013) have shown that Theorem 14.11 holds even if we output ¯w=w(T).
14.5 Learning with SGD
We have so far introduced and analyzed the SGD algorithm for general convex
functions. Now we shall consider its applicability to learning tasks.
14.5.1 SGD for Risk Minimization
Recall that in learning we face the problem of minimizing the risk function
LD(w) =E
z∼D[ℓ(w,z)].
We have seen the method of empirical risk minimization, where we minimize the
empirical risk, LS(w), as an estimate to minimizing LD(w). SGD allows us to

LD(w) =E
take a diﬀerent approach and minimize LD(w) directly. Since we do not know
D, we cannot simply calculate ∇LD(w(t)) and minimize it with the GD method.
With SGD, however, all we need is to ﬁnd an unbiased estimate of the gradient of

14.5 Learning with SGD 197
LD(w), that is, a random vector whose conditional expected value is ∇LD(w(t)).
We shall now see how such an estimate can be easily constructed.
For simplicity, let us ﬁrst consider the case of diﬀerentiable loss functions.
Hence the risk function LDis also diﬀerentiable. The construction of the random
vector vtwill be as follows: First, sample z∼D. Then, deﬁne vtto be the
gradient of the function ℓ(w,z) with respect to w, at the point w(t). Then, by
the linearity of the gradient we have
E[vt|w(t)] =E
z∼D[∇ℓ(w(t),z)] =∇E
z∼D[ℓ(w(t),z)] =∇LD(w(t)). (14.13)
The gradient of the loss function ℓ(w,z) atw(t)is therefore an unbiased estimate
of the gradient of the risk function LD(w(t)) and is easily constructed by sampling
a single fresh example z∼D at each iteration t.
The same argument holds for nondiﬀerentiable loss functions. We simply let
vtbe a subgradient of ℓ(w,z) atw(t). Then, for every uwe have
ℓ(u,z)−ℓ(w(t),z)≥⟨u−w(t),vt⟩.

Taking expectation on both sides with respect to z∼D and conditioned on the
value of w(t)we obtain
LD(u)−LD(w(t)) =E[ℓ(u,z)−ℓ(w(t),z)|w(t)]
≥E[⟨u−w(t),vt⟩|w(t)]
=⟨u−w(t),E[vt|w(t)]⟩.
It follows that E[vt|w(t)] is a subgradient of LD(w) atw(t).
To summarize, the stochastic gradient descent framework for minimizing the
risk is as follows.
LD(w)
samplez∼D
pickvt∈∂ℓ(w(t),z)
T∑T

LD(w)
samplez∼D
T∑T
t=1w(t)
We shall now use our analysis of SGD to obtain a sample complexity anal-
ysis for learning convex-Lipschitz-bounded problems. Theorem 14.8 yields the
corollary 14.12 Consider a convex-Lipschitz-bounded learning problem with
parameters ρ,B. Then, for every ϵ>0, if we run the SGD method for minimizing

198 Stochastic Gradient Descent
LD(w)with a number of iterations (i.e., number of examples)
T≥B2ρ2
ϵ2
and withη=√
B2
ρ2T, then the output of SGD satisﬁes
E[LD(¯w)]≤min
It is interesting to note that the required sample complexity is of the same order
of magnitude as the sample complexity guarantee we derived for regularized loss
minimization. In fact, the sample complexity of SGD is even better than what
we have derived for regularized loss minimization by a factor of 8.
14.5.2 Analyzing SGD for Convex-Smooth Learning Problems
In the previous chapter we saw that the regularized loss minimization rule also
learns the class of convex-smooth-bounded learning problems. We now show that
the SGD algorithm can be also used for such problems.
theorem 14.13 Assume that for all z, the loss function ℓ(·,z)is convex,β-
smooth, and nonnegative. Then, if we run the SGD algorithm for minimizing
LD(w)we have that for every w⋆,
E[LD(¯w)]≤1
1−ηβ(
LD(w⋆) +∥w⋆∥2
2ηT)
.

1−ηβ(
2ηT)
.
Proof Recall that if a function is β-smooth and nonnegative then it is self-
bounded:
∥∇f(w)∥2≤2βf(w).
To analyze SGD for convex-smooth problems, let us deﬁne z1,...,zTthe random
samples of the SGD algorithm, let ft(·) =ℓ(·,zt), and note that vt=∇ft(w(t)).
For allt,ftis a convex function and therefore ft(w(t))−ft(w⋆)≤⟨vt,w(t)−w⋆⟩.
Summing over tand using Lemma 14.1 we obtain
T∑
t=1(ft(w(t))−ft(w⋆))≤T∑
t=1⟨vt,w(t)−w⋆⟩≤∥w⋆∥2
2η+η
2T∑
t=1∥vt∥2.
Combining the preceding with the self-boundedness of ftyields
T∑

bounded:
T∑
2η+η
2T∑
t=1∥vt∥2.
T∑
t=1(ft(w(t))−ft(w⋆))≤∥w⋆∥2
2η+ηβT∑
t=1ft(w(t)).
Dividing by Tand rearranging, we obtain
1
TT∑
t=1ft(w(t))≤1
1−ηβ(
1
TT∑
t=1ft(w⋆) +∥w⋆∥2
2ηT)
.
Next, we take expectation of the two sides of the preceding equation with respect

14.5 Learning with SGD 199
toz1,...,zT. Clearly, E[ft(w⋆)] =LD(w⋆). In addition, using the same argument
as in the proof of Theorem 14.8 we have that
E[
1
TT∑
t=1ft(w(t))]
=E[
1
TT∑
t=1LD(w(t))]
≥E[LD(¯w)].
Combining all we conclude our proof.
As a direct corollary we obtain:
corollary 14.14 Consider a convex-smooth-bounded learning problem with
parameters β,B. Assume in addition that ℓ(0,z)≤1for allz∈Z. For every
ϵ>0, setη=1
β(1+3/ϵ). Then, running SGD with T≥12B2β/ϵ2yields
14.5.3 SGD for Regularized Loss Minimization
We have shown that SGD enjoys the same worst-case sample complexity bound
as regularized loss minimization. However, on some distributions, regularized loss
minimization may yield a better solution. Therefore, in some cases we may want
to solve the optimization problem associated with regularized loss minimization,
namely,1
min
w(λ
2∥w∥2+LS(w))
. (14.14)
Since we are dealing with convex learning problems in which the loss function is

namely,1
min
w(λ
. (14.14)
convex, the preceding problem is also a convex optimization problem that can
be solved using SGD as well, as we shall see in this section.
Deﬁnef(w) =λ
2∥w∥2+LS(w). Note that fis aλ-strongly convex function;
therefore, we can apply the SGD variant given in Section 14.4.4 (with H=Rd).
To apply this algorithm, we only need to ﬁnd a way to construct an unbiased
estimate of a subgradient of fatw(t). This is easily done by noting that if
we pickzuniformly at random from S, and choose vt∈∂ℓ(w(t),z) then the

expected value of λw(t)+vtis a subgradient of fatw(t).
To analyze the resulting algorithm, we ﬁrst rewrite the update rule (assuming
1We divided λby 2 for convenience.

200 Stochastic Gradient Descent
thatH=Rdand therefore the projection step does not matter) as follows
w(t+1)=w(t)−1
λt(
λw(t)+vt)
=(
1−1
t)
w(t)−1
λtvt
=t−1
tw(t)−1
λtvt
=t−1
t(t−2
t−1w(t−1)−1
λ(t−1)vt−1)
−1
λtvt
=−1
λtt∑
i=1vi. (14.15)
If we assume that the loss function is ρ-Lipschitz, it follows that for all twe have
∥vt∥≤ρand therefore∥λw(t)∥≤ρ, which yields
∥λw(t)+vt∥≤2ρ.
Theorem 14.11 therefore tells us that after performing Titerations we have that
E[f(¯w)]−f(w⋆)≤4ρ2
λT(1 + log(T)).
14.6 Summary
We have introduced the Gradient Descent and Stochastic Gradient Descent algo-
rithms, along with several of their variants. We have analyzed their convergence
rate and calculated the number of iterations that would guarantee an expected
objective of at most ϵplus the optimal objective. Most importantly, we have
shown that by using SGD we can directly minimize the risk function. We do
so by sampling a point i.i.d from Dand using a subgradient of the loss of the

current hypothesis w(t)at this point as an unbiased estimate of the gradient (or
a subgradient) of the risk function. This implies that a bound on the number of
iterations also yields a sample complexity bound. Finally, we have also shown
how to apply the SGD method to the problem of regularized risk minimization.
In future chapters we show how this yields extremely simple solvers to some
optimization problems associated with regularized risk minimization.
14.7 Bibliographic Remarks

SGD dates back to Robbins & Monro (1951). It is especially eﬀective in large
scale machine learning problems. See, for example, (Murata 1998, Le Cun 2004,
Zhang 2004, Bottou & Bousquet 2008, Shalev-Shwartz, Singer & Srebro 2007,
Shalev-Shwartz & Srebro 2008). In the optimization community it was studied

in the context of stochastic optimization . See, for example, (Nemirovski & Yudin
1978, Nesterov & Nesterov 2004, Nesterov 2005, Nemirovski, Juditsky, Lan &
Shapiro 2009, Shapiro, Dentcheva & Ruszczy´ nski 2009).
The bound we have derived for strongly convex function is due to Hazan,
Agarwal & Kale (2007). As mentioned previously, improved bounds have been
obtained in Rakhlin et al. (2012).
14.8 Exercises
1. Prove Claim 14.10. Hint: Extend the proof of Lemma 13.5.
2. Prove Corollary 14.14.
3.Perceptron as a subgradient descent algorithm: LetS= ((x1, y1),..., (xm, ym))∈
(Rd×{± 1})m. Assume that there exists w∈Rdsuch that for every i∈[m]
we haveyi⟨w,xi⟩≥1, and let w⋆be a vector that has the minimal norm
among all vectors that satisfy the preceding requirement. Let R= maxi∥xi∥.
Deﬁne a function
f(w) = max
i∈[m](1−yi⟨w,xi⟩).
•Show that min w:∥w∥≤∥w⋆∥f(w) = 0 and show that any wfor whichf(w)<
1 separates the examples in S.
•Show how to calculate a subgradient of f.

f(w) = max
•Describe and analyze the subgradient descent algorithm for this case. Com-
pare the algorithm and the analysis to the Batch Perceptron algorithm
given in Section 9.1.2.
4.Variable step size (*): Prove an analog of Theorem 14.8 for SGD with a
variable step size, ηt=B
ρ√
t.

15 Support Vector Machines
In this chapter and the next we discuss a very useful machine learning tool: the
support vector machine paradigm (SVM) for learning linear predictors in high
dimensional feature spaces. The high dimensionality of the feature space raises
both sample complexity and computational complexity challenges.
The SVM algorithmic paradigm tackles the sample complexity challenge by
searching for “large margin” separators. Roughly speaking, a halfspace separates
a training set with a large margin if all the examples are not only on the correct
side of the separating hyperplane but also far away from it. Restricting the
algorithm to output a large margin separator can yield a small sample complexity
even if the dimensionality of the feature space is high (and even inﬁnite). We
introduce the concept of margin and relate it to the regularized loss minimization
paradigm as well as to the convergence rate of the Perceptron algorithm.

In the next chapter we will tackle the computational complexity challenge
using the idea of kernels .
15.1 Margin and Hard-SVM
LetS= (x1,y1),..., (xm,ym) be a training set of examples, where each xi∈Rd
andyi∈{± 1}. We say that this training set is linearly separable, if there exists
a halfspace, ( w,b), such that yi= sign(⟨w,xi⟩+b) for alli. Alternatively, this
condition can be rewritten as
∀i∈[m], yi(⟨w,xi⟩+b)>0.
All halfspaces ( w,b) that satisfy this condition are ERM hypotheses (their 0-1

error is zero, which is the minimum possible error). For any separable training
sample, there are many ERM halfspaces. Which one of them should the learner
pick?
Consider, for example, the training set described in the picture that follows.

15.1 Margin and Hard-SVM 203
xx
While both the dashed-black and solid-green hyperplanes separate the four ex-
amples, our intuition would probably lead us to prefer the black hyperplane over
the green one. One way to formalize this intuition is using the concept of margin .
The margin of a hyperplane with respect to a training set is deﬁned to be the
minimal distance between a point in the training set and the hyperplane. If a
hyperplane has a large margin, then it will still separate the training set even if
we slightly perturb each instance.
We will see later on that the true error of a halfspace can be bounded in terms
of the margin it has over the training sample (the larger the margin, the smaller
the error), regardless of the Euclidean dimension in which this halfspace resides.
Hard-SVM is the learning rule in which we return an ERM hyperplane that
separates the training set with the largest possible margin. To deﬁne Hard-SVM
formally, we ﬁrst express the distance between a point xto a hyperplane using

the parameters deﬁning the halfspace.
claim 15.1 The distance between a point xand the hyperplane deﬁned by
(w, b)where∥w∥= 1is|⟨w,x⟩+b|.
Proof The distance between a point xand the hyperplane is deﬁned as
min{∥x−v∥:⟨w,v⟩+b= 0}.
Taking v=x−(⟨w,x⟩+b)wwe have that
⟨w,v⟩+b=⟨w,x⟩−(⟨w,x⟩+b)∥w∥2+b= 0,
and
∥x−v∥=|⟨w,x⟩+b|∥w∥=|⟨w,x⟩+b|.
Hence, the distance is at most |⟨w,x⟩+b|. Next, take any other point uon the
hyperplane, thus ⟨w,u⟩+b= 0. We have
∥x−u∥2=∥x−v+v−u∥2
=∥x−v∥2+∥v−u∥2+ 2⟨x−v,v−u⟩
≥∥x−v∥2+ 2⟨x−v,v−u⟩

and
=∥x−v∥2+ 2(⟨w,x⟩+b)⟨w,v−u⟩
=∥x−v∥2,
where the last equality is because ⟨w,v⟩=⟨w,u⟩=−b. Hence, the distance

204 Support Vector Machines
between xanduis at least the distance between xandv, which concludes our
proof.
On the basis of the preceding claim, the closest point in the training set to the
separating hyperplane is min i∈[m]|⟨w,xi⟩+b|. Therefore, the Hard-SVM rule is
argmax
(w,b):∥w∥=1min
i∈[m]|⟨w,xi⟩+b|s.t.∀i, yi(⟨w,xi⟩+b)>0.
Whenever there is a solution to the preceding problem (i.e., we are in the sepa-
rable case), we can write an equivalent problem as follows (see Exercise 1):
argmax
i∈[m]yi(⟨w,xi⟩+b). (15.1)
Next, we give another equivalent formulation of the Hard-SVM rule as a quadratic
optimization problem.1
Hard-SVM
input: (x1,y1),..., (xm,ym)
solve:
(w0,b0) = argmin
(w,b)∥w∥2s.t.∀i, yi(⟨w,xi⟩+b)≥1 (15.2)
output: ˆw=w0
∥w0∥,ˆb=b0
∥w0∥
The lemma that follows shows that the output of hard-SVM is indeed the
separating hyperplane with the largest margin. Intuitively, hard-SVM searches
forwof minimal norm among all the vectors that separate the data and for

argmax
Hard-SVM
solve:
∥w0∥
which|⟨w,xi⟩+b|≥1 for alli. In other words, we enforce the margin to be 1,
but now the units in which we measure the margin scale with the norm of w.
Therefore, ﬁnding the largest margin halfspace boils down to ﬁnding wwhose
norm is minimal. Formally:
lemma 15.2 The output of Hard-SVM is a solution of Equation (15.1).
Proof Let (w⋆,b⋆) be a solution of Equation (15.1) and deﬁne the margin
achieved by ( w⋆,b⋆) to beγ⋆= mini∈[m]yi(⟨w⋆,xi⟩+b⋆). Therefore, for all
iwe have
yi(⟨w⋆,xi⟩+b⋆)≥γ⋆
or equivalently
yi(⟨w⋆

iwe have
yi(⟨w⋆
γ⋆,xi⟩+b⋆
γ⋆)≥1.
Hence, the pair (w⋆
γ⋆,b⋆
γ⋆) satisﬁes the conditions of the quadratic optimization
1A quadratic optimization problem is an optimization problem in which the objective is a
convex quadratic function and the constraints are linear inequalities.

15.1 Margin and Hard-SVM 205
problem given in Equation (15.2). Therefore, ∥w0∥≤∥w⋆
γ⋆∥=1
γ⋆. It follows that
for alli,
yi(⟨ˆw,xi⟩+ˆb) =1
∥w0∥yi(⟨w0,xi⟩+b0)≥1
∥w0∥≥γ⋆.
Since∥ˆw∥= 1 we obtain that ( ˆw,ˆb) is an optimal solution of Equation (15.1).
15.1.1 The Homogenous Case
It is often more convenient to consider homogenous halfspaces, namely, halfspaces
that pass through the origin and are thus deﬁned by sign( ⟨w,x⟩), where the bias
termbis set to be zero. Hard-SVM for homogenous halfspaces amounts to solving
min
w∥w∥2s.t.∀i, yi⟨w,xi⟩≥1. (15.3)
As we discussed in Chapter 9, we can reduce the problem of learning nonhomogenous
halfspaces to the problem of learning homogenous halfspaces by adding one more
feature to each instance of xi, thus increasing the dimension to d+ 1.
Note, however, that the optimization problem given in Equation (15.2) does
not regularize the bias term b, while if we learn a homogenous halfspace in Rd+1
using Equation (15.3) then we regularize the bias term (i.e., the d+1 component

min
of the weight vector) as well. However, regularizing busually does not make a
signiﬁcant diﬀerence to the sample complexity.
15.1.2 The Sample Complexity of Hard-SVM
Recall that the VC-dimension of halfspaces in Rdisd+ 1. It follows that the
sample complexity of learning halfspaces grows with the dimensionality of the
problem. Furthermore, the fundamental theorem of learning tells us that if the
number of examples is signiﬁcantly smaller than d/ϵthen no algorithm can learn

anϵ-accurate halfspace. This is problematic when dis very large.
To overcome this problem, we will make an additional assumption on the
underlying data distribution. In particular, we will deﬁne a “separability with
marginγ” assumption and will show that if the data is separable with margin
γthen the sample complexity is bounded from above by a function of 1 /γ2. It
follows that even if the dimensionality is very large (or even inﬁnite), as long as
the data adheres to the separability with margin assumption we can still have a

small sample complexity. There is no contradiction to the lower bound given in
the fundamental theorem of learning because we are now making an additional
assumption on the underlying data distribution.
Before we formally deﬁne the separability with margin assumption, there is a
scaling issue we need to resolve. Suppose that a training set S= (x1,y1),..., (xm,ym)
is separable with a margin γ, namely, the maximal objective value of Equa-
tion (15.1) is at least γ. Then, for any positive scalar α > 0, the training set

206 Support Vector Machines
S′= (αx1,y1),..., (αxm,ym) is separable with a margin of αγ. That is, a sim-
ple scaling of the data can make it separable with an arbitrarily large margin. It
follows that in order to give a meaningful deﬁnition of margin we must take into
account the scale of the examples as well. One way to formalize this is using the
deﬁnition that follows.
definition 15.3 LetDbe a distribution over Rd×{± 1}. We say thatDis
separable with a ( γ,ρ)-margin if there exists ( w⋆,b⋆) such that∥w⋆∥= 1 and
such that with probability 1 over the choice of ( x,y)∼Dwe have that y(⟨w⋆,x⟩+
b⋆)≥γand∥x∥≤ρ. Similarly, we say that Dis separable with a ( γ,ρ)-margin
using a homogenous halfspace if the preceding holds with a halfspace of the form
(w⋆,0).
In the advanced part of the book (Chapter 26), we will prove that the sample
complexity of Hard-SVM depends on ( ρ/γ)2and is independent of the dimension
d. In particular, Theorem 26.13 in Section 26.3 states the following:

(w⋆,0).
theorem 15.4 LetDbe a distribution over Rd×{± 1}that satisﬁes the (γ,ρ)-
separability with margin assumption using a homogenous halfspace. Then, with
probability of at least 1−δover the choice of a training set of size m, the 0-1
error of the output of Hard-SVM is at most
√
4 (ρ/γ)2
m+√
2 log(2/δ)
m.
Remark 15.1 (Margin and the Perceptron) In Section 9.1.2 we have described
and analyzed the Perceptron algorithm for ﬁnding an ERM hypothesis with
respect to the class of halfspaces. In particular, in Theorem 9.1 we upper bounded

√
4 (ρ/γ)2
m+√
2 log(2/δ)
m.
the number of updates the Perceptron might make on a given training set. It
can be shown (see Exercise 2) that the upper bound is exactly ( ρ/γ)2, whereρ
is the radius of examples and γis the margin.
15.2 Soft-SVM and Norm Regularization
The Hard-SVM formulation assumes that the training set is linearly separable,
which is a rather strong assumption. Soft-SVM can be viewed as a relaxation of
the Hard-SVM rule that can be applied even if the training set is not linearly
separable.
The optimization problem in Equation (15.2) enforces the hard constraints

yi(⟨w,xi⟩+b)≥1 for alli. A natural relaxation is to allow the constraint to be
violated for some of the examples in the training set. This can be modeled by
introducing nonnegative slack variables, ξ1,...,ξm, and replacing each constraint
yi(⟨w,xi⟩+b)≥1 by the constraint yi(⟨w,xi⟩+b)≥1−ξi. That is,ξimeasures
by how much the constraint yi(⟨w,xi⟩+b)≥1 is being violated. Soft-SVM jointly
minimizes the norm of w(corresponding to the margin) and the average of ξi

(corresponding to the violations of the constraints). The tradeoﬀ between the two

15.2 Soft-SVM and Norm Regularization 207
terms is controlled by a parameter λ. This leads to the Soft-SVM optimization
problem:
Soft-SVM
parameter: λ>0
solve:
min
w,b,ξ(
λ∥w∥2+1
mm∑
i=1ξi)
s.t.∀i, yi(⟨w,xi⟩+b)≥1−ξiandξi≥0(15.4)
output: w,b
We can rewrite Equation (15.4) as a regularized loss minimization problem.
Recall the deﬁnition of the hinge loss:
ℓhinge((w,b),(x,y)) = max{0,1−y(⟨w,x⟩+b)}.
Given ( w,b) and a training set S, the averaged hinge loss on Sis denoted by
Lhinge
S((w,b)). Now, consider the regularized loss minimization problem:
min
w,b(
λ∥w∥2+Lhinge
S((w,b)))
. (15.5)
claim 15.5 Equation (15.4) and Equation (15.5) are equivalent.
Proof Fix some w,band consider the minimization over ξin Equation (15.4).
Fix somei. Sinceξimust be nonnegative, the best assignment to ξiwould be 0
ifyi(⟨w,xi⟩+b)≥1 and would be 1 −yi(⟨w,xi⟩+b) otherwise. In other words,
ξi=ℓhinge((w,b),(xi,yi)) for alli, and the claim follows.

Lhinge
min
w,b(
S((w,b)))
. (15.5)
We therefore see that Soft-SVM falls into the paradigm of regularized loss
minimization that we studied in the previous chapter. A Soft-SVM algorithm,
that is, a solution for Equation (15.5), has a bias toward low norm separators.
The objective function that we aim to minimize in Equation (15.5) penalizes not
only for training errors but also for large norm.
It is often more convenient to consider Soft-SVM for learning a homogenous
halfspace, where the bias term bis set to be zero, which yields the following
optimization problem:
min
w(
S(w))

min
w(
S(w))
, (15.6)
where
Lhinge
S(w) =1
mm∑
i=1max{0,1−y⟨w,xi⟩}.

208 Support Vector Machines
15.2.1 The Sample Complexity of Soft-SVM
We now analyze the sample complexity of Soft-SVM for the case of homogenous
halfspaces (namely, the output of Equation (15.6)). In Corollary 13.8 we derived
a generalization bound for the regularized loss minimization framework assuming
that the loss function is convex and Lipschitz. We have already shown that the
hinge loss is convex so it is only left to analyze the Lipschitzness of the hinge
loss.
claim 15.6 Letf(w) = max{0,1−y⟨w,x⟩}. Then,fis∥x∥-Lipschitz.
Proof It is easy to verify that any subgradient of fatwis of the form αxwhere
|α|≤1. The claim now follows from Lemma 14.7.
Corollary 13.8 therefore yields the following:
corollary 15.7 LetDbe a distribution over X×{ 0,1}, whereX={x:
∥x∥≤ρ}. Consider running Soft-SVM (Equation (15.6)) on a training set S∼
Dmand letA(S)be the solution of Soft-SVM. Then, for every u,
E
S∼Dm[Lhinge
D(A(S))]≤Lhinge
D(u) +λ∥u∥2+2ρ2
λm.
Furthermore, since the hinge loss upper bounds the 0−1loss we also have

E
λm.
E
S∼Dm[L0−1
λm.
Last, for every B > 0, if we setλ=√
2ρ2
B2mthen
E
S∼Dm[L0−1
D(A(S))]≤E
D(A(S))]≤ min
w:∥w∥≤BLhinge
D(w) +√
8ρ2B2
m.
We therefore see that we can control the sample complexity of learning a half-
space as a function of the norm of that halfspace, independently of the Euclidean
dimension of the space over which the halfspace is deﬁned. This becomes highly
signiﬁcant when we learn via embeddings into high dimensional feature spaces,

E
S∼Dm[L0−1
λm.
2ρ2
B2mthen
E
S∼Dm[L0−1
D(w) +√
8ρ2B2
m.
as we will consider in the next chapter.
Remark 15.2 The condition that Xwill contain vectors with a bounded norm
follows from the requirement that the loss function will be Lipschitz. This is
not just a technicality. As we discussed before, separation with large margin
is meaningless without imposing a restriction on the scale of the instances. In-
deed, without a constraint on the scale, we can always enlarge the margin by
multiplying all instances by a large scalar.
15.2.2 Margin and Norm-Based Bounds versus Dimension

The bounds we have derived for Hard-SVM and Soft-SVM do not depend on the
dimension of the instance space. Instead, the bounds depend on the norm of the

15.2 Soft-SVM and Norm Regularization 209
examples,ρ, the norm of the halfspace B(or equivalently the margin parameter
γ) and, in the nonseparable case, the bounds also depend on the minimum hinge
loss of all halfspaces of norm ≤B. In contrast, the VC-dimension of the class of
homogenous halfspaces is d, which implies that the error of an ERM hypothesis
decreases as√
d/m does. We now give an example in which ρ2B2≪d; hence
the bound given in Corollary 15.7 is much better than the VC bound.
Consider the problem of learning to classify a short text document according
to its topic, say, whether the document is about sports or not. We ﬁrst need to
represent documents as vectors. One simple yet eﬀective way is to use a bag-
of-words representation. That is, we deﬁne a dictionary of words and set the
dimensiondto be the number of words in the dictionary. Given a document,
we represent it as a vector x∈{0,1}d, wherexi= 1 if the i’th word in the

dictionary appears in the document and xi= 0 otherwise. Therefore, for this
problem, the value of ρ2will be the maximal number of distinct words in a given
document.
A halfspace for this problem assigns weights to words. It is natural to assume
that by assigning positive and negative weights to a few dozen words we will
be able to determine whether a given document is about sports or not with
reasonable accuracy. Therefore, for this problem, the value of B2can be set to
be less than 100. Overall, it is reasonable to say that the value of B2ρ2is smaller

document.
than 10,000.
On the other hand, a typical size of a dictionary is much larger than 10,000.
For example, there are more than 100,000 distinct words in English. We have
therefore shown a problem in which there can be an order of magnitude diﬀerence
between learning a halfspace with the SVM rule and learning a halfspace using
the vanilla ERM rule.
Of course, it is possible to construct problems in which the SVM bound will
be worse than the VC bound. When we use SVM, we in fact introduce another

form of inductive bias – we prefer large margin halfspaces. While this induc-
tive bias can signiﬁcantly decrease our estimation error, it can also enlarge the
approximation error.
15.2.3 The Ramp Loss*
The margin-based bounds we have derived in Corollary 15.7 rely on the fact that
we minimize the hinge loss. As we have shown in the previous subsection, the
term√
ρ2B2/mcan be much smaller than the corresponding term in the VC
bound,√
d/m. However, the approximation error in Corollary 15.7 is measured

term√
bound,√
with respect to the hinge loss while the approximation error in VC bounds is
measured with respect to the 0 −1 loss. Since the hinge loss upper bounds the
0−1 loss, the approximation error with respect to the 0 −1 loss will never exceed
that of the hinge loss.
It is not possible to derive bounds that involve the estimation error term√
ρ2B2/mfor the 0−1 loss. This follows from the fact that the 0 −1 loss is scale

210 Support Vector Machines
insensitive, and therefore there is no meaning to the norm of wor its margin
when we measure error with the 0 −1 loss. However, it is possible to deﬁne a loss
function that on one hand it is scale sensitive and thus enjoys the estimation
error√
ρ2B2/mwhile on the other hand it is more similar to the 0 −1 loss. One
option is the ramp loss , deﬁned as
ℓramp(w,(x,y)) = min{1,ℓhinge(w,(x,y))}= min{1,max{0,1−y⟨w,x⟩}}.
The ramp loss penalizes mistakes in the same way as the 0 −1 loss and does not
penalize examples that are separated with margin. The diﬀerence between the
ramp loss and the 0 −1 loss is only with respect to examples that are correctly
classiﬁed but not with a signiﬁcant margin. Generalization bounds for the ramp
loss are given in the advanced part of this book (see Appendix 26.3).
y⟨w,x⟩ℓrampℓhinge
ℓ0−1
11
The reason SVM relies on the hinge loss and not on the ramp loss is that
the hinge loss is convex and, therefore, from the computational point of view,

ℓ0−1
11
minimizing the hinge loss can be performed eﬃciently. In contrast, the problem
of minimizing the ramp loss is computationally intractable.
15.3 Optimality Conditions and “Support Vectors”*
The name “Support Vector Machine” stems from the fact that the solution of
hard-SVM, w0, is supported by (i.e., is in the linear span of) the examples that
are exactly at distance 1 /∥w0∥from the separating hyperplane. These vectors are
therefore called support vectors . To see this, we rely on Fritz John optimality
conditions .

theorem 15.8 Letw0be as deﬁned in Equation (15.3) and let I={i:
|⟨w0,xi⟩|= 1}. Then, there exist coeﬃcients α1,...,αmsuch that
w0=∑
i∈Iαixi.
The examples{xi:i∈I}are called support vectors .
The proof of this theorem follows by applying the following lemma to Equa-
tion (15.3).

lemma 15.9 (Fritz John) Suppose that
w⋆∈argmin
wf(w)s.t.∀i∈[m], gi(w)≤0,
wheref,g1,...,gmare diﬀerentiable. Then, there exists α∈Rmsuch that
∇f(w⋆) +∑
i∈Iαi∇gi(w⋆) =0, whereI={i:gi(w⋆) = 0}.
15.4 Duality*
Historically, many of the properties of SVM have been obtained by considering
thedual of Equation (15.3). Our presentation of SVM does not rely on duality.
For completeness, we present in the following how to derive the dual of Equa-
We start by rewriting the problem in an equivalent form as follows. Consider
the function
g(w) = max
α∈Rm:α≥0m∑
i=1αi(1−yi⟨w,xi⟩) ={
0 if∀i,yi⟨w,xi⟩≥1
∞ otherwise.
We can therefore rewrite Equation (15.3) as
min
w(
∥w∥2+g(w))
. (15.7)
Rearranging the preceding we obtain that Equation (15.3) can be rewritten as
the problem
min
wmax
α∈Rm:α≥0(
1
2∥w∥2+m∑
i=1αi(1−yi⟨w,xi⟩))
. (15.8)
Now suppose that we ﬂip the order of min and max in the above equation. This
can only decrease the objective value (see Exercise 4), and we have
min
wmax
α∈Rm:α≥0(
1

g(w) = max
min
w(
. (15.7)
min
wmax
α∈Rm:α≥0(
1
2∥w∥2+m∑
. (15.8)
min
wmax
α∈Rm:α≥0(
1
2∥w∥2+m∑
≥ max
α∈Rm:α≥0min
w(
1
2∥w∥2+m∑
.
The preceding inequality is called weak duality . It turns out that in our case,
strong duality also holds; namely, the inequality holds with equality. Therefore,
thedual problem is
max
w(
1
2∥w∥2+m∑
. (15.9)
We can simplify the dual problem by noting that once αis ﬁxed, the optimization

212 Support Vector Machines
problem with respect to wis unconstrained and the objective is diﬀerentiable;
thus, at the optimum, the gradient equals zero:
w−m∑
i=1αiyixi= 0⇒w=m∑
i=1αiyixi.
This shows us that the solution must be in the linear span of the examples, a
fact we will use later to derive SVM with kernels. Plugging the preceding into
Equation (15.9) we obtain that the dual problem can be rewritten as
max
α∈Rm:α≥0
1
2m∑
i=1αiyixi2
+m∑
i=1αi
1−yi⟨∑
jαjyjxj,xi⟩

.(15.10)
Rearranging yields the dual problem
max
α∈Rm:α≥0
m∑
i=1αi−1
2m∑
i=1m∑
j=1αiαjyiyj⟨xj,xi⟩
. (15.11)
Note that the dual problem only involves inner products between instances and
does not require direct access to speciﬁc elements within an instance. This prop-
erty is important when implementing SVM with kernels, as we will discuss in
the next chapter.
15.5 Implementing Soft-SVM Using SGD
In this section we describe a very simple algorithm for solving the optimization
problem of Soft-SVM, namely,
min
w(
λ
2∥w∥2+1
mm∑

max
α∈Rm:α≥0
m∑
i=1αi−1
2m∑
i=1m∑
. (15.11)
min
w(
λ
2∥w∥2+1
mm∑
i=1max{0,1−y⟨w,xi⟩})
. (15.12)
We rely on the SGD framework for solving regularized loss minimization prob-
lems, as described in Section 14.5.3.
Recall that, on the basis of Equation (14.15), we can rewrite the update rule
of SGD as
w(t+1)=−1
λtt∑
j=1vj,
where vjis a subgradient of the loss function at w(j)on the random example
chosen at iteration j. For the hinge loss, given an example ( x,y), we can choose vj
to be 0ify⟨w(j),x⟩≥1 and vj=−yxotherwise (see Example 14.2). Denoting
θ(t)=−∑
j<tvjwe obtain the following procedure.

SGD for Solving Soft-SVM
goal: Solve Equation (15.12)
initialize:θ(1)=0
Letw(t)=1
λtθ(t)
Chooseiuniformly at random from [ m]
If (yi⟨w(t),xi⟩<1)
Setθ(t+1)=θ(t)+yixi
Else
Setθ(t+1)=θ(t)
T∑T
t=1w(t)
15.6 Summary
SVM is an algorithm for learning halfspaces with a certain type of prior knowl-
edge, namely, preference for large margin. Hard-SVM seeks the halfspace that
separates the data perfectly with the largest margin, whereas soft-SVM does
not assume separability of the data and allows the constraints to be violated to
some extent. The sample complexity for both types of SVM is diﬀerent from the
sample complexity of straightforward halfspace learning, as it does not depend
on the dimension of the domain but rather on parameters such as the maximal
norms of xandw.
The importance of dimension-independent sample complexity will be realized
in the next chapter, where we will discuss the embedding of the given domain

into some high dimensional feature space as means for enriching our hypothesis
class. Such a procedure raises computational and sample complexity problems.
The latter is solved by using SVM, whereas the former can be solved by using
SVM with kernels, as we will see in the next chapter.
15.7 Bibliographic Remarks
SVMs have been introduced in (Cortes & Vapnik 1995, Boser, Guyon & Vapnik
1992). There are many good books on the theoretical and practical aspects of

SVMs. For example, (Vapnik 1995, Cristianini & Shawe-Taylor 2000, Sch¨ olkopf
& Smola 2002, Hsu, Chang & Lin 2003, Steinwart & Christmann 2008). Using
SGD for solving soft-SVM has been proposed in Shalev-Shwartz et al. (2007).

214 Support Vector Machines
15.8 Exercises
1. Show that the hard-SVM rule, namely,
argmax
i∈[m]|⟨w,xi⟩+b|s.t.∀i, yi(⟨w,xi⟩+b)>0,
is equivalent to the following formulation:
argmax
i∈[m]yi(⟨w,xi⟩+b). (15.13)
Hint: DeﬁneG={(w,b) :∀i,yi(⟨w,xi⟩+b)>0}.
1. Show that
argmax
i∈[m]yi(⟨w,xi⟩+b)∈G
2. Show that∀(w,b)∈G,
min
i∈[m]yi(⟨w,xi⟩+b) = min
i∈[m]|⟨w,xi⟩+b|
2.Margin and the Perceptron Consider a training set that is linearly sep-
arable with a margin γand such that all the instances are within a ball of
radiusρ. Prove that the maximal number of updates the Batch Perceptron
algorithm given in Section 9.1.2 will make when running on this training set
is (ρ/γ)2.
3.Hard versus soft SVM: Prove or refute the following claim:
There exists λ > 0such that for every sample Sofm > 1examples, which
is separable by the class of homogenous halfspaces, the hard-SVM and the
soft-SVM (with parameter λ) learning rules return exactly the same weight
vector.

is (ρ/γ)2.
vector.
4.Weak duality: Prove that for any function fof two vector variables x∈
X,y∈Y, it holds that
min
x∈Xmax
y∈Yf(x,y)≥max
y∈Ymin
x∈Xf(x,y).

16 Kernel Methods
In the previous chapter we described the SVM paradigm for learning halfspaces
in high dimensional feature spaces. This enables us to enrich the expressive
power of halfspaces by ﬁrst mapping the data into a high dimensional feature
space, and then learning a linear predictor in that space. This is similar to the
AdaBoost algorithm, which learns a composition of a halfspace over base hy-
potheses. While this approach greatly extends the expressiveness of halfspace
predictors, it raises both sample complexity and computational complexity chal-
lenges. In the previous chapter we tackled the sample complexity issue using
the concept of margin. In this chapter we tackle the computational complexity
challenge using the method of kernels .
We start the chapter by describing the idea of embedding the data into a high
dimensional feature space. We then introduce the idea of kernels. A kernel is a
type of a similarity measure between instances. The special property of kernel

similarities is that they can be viewed as inner products in some Hilbert space
(or Euclidean space of some high dimension) to which the instance space is vir-
tually embedded. We introduce the “kernel trick” that enables computationally
eﬃcient implementation of learning, without explicitly handling the high dimen-
sional representation of the domain instances. Kernel based learning algorithms,
and in particular kernel-SVM, are very useful and popular machine learning

tools. Their success may be attributed both to being ﬂexible for accommodating
domain speciﬁc prior knowledge and to having a well developed set of eﬃcient
implementation algorithms.
16.1 Embeddings into Feature Spaces
The expressive power of halfspaces is rather restricted – for example, the follow-
ing training set is not separable by a halfspace.
Let the domain be the real line; consider the domain points {−10,−9,−8,..., 0,
1,..., 9,10}where the labels are +1 for all xsuch that|x|>2 and−1 otherwise.

To make the class of halfspaces more expressive, we can ﬁrst map the original
instance space into another space (possibly of a higher dimension) and then
learn a halfspace in that space. For example, consider the example mentioned
previously. Instead of learning a halfspace in the original representation let us

216 Kernel Methods
ﬁrst deﬁne a mapping ψ:R→R2as follows:
ψ(x) = (x,x2).
We use the term feature space to denote the range of ψ. After applying ψthe
data can be easily explained using the halfspace h(x) = sign(⟨w,ψ(x)⟩−b),
where w= (0,1) andb= 5.
The basic paradigm is as follows:
1. Given some domain set Xand a learning task, choose a mapping ψ:X→F ,
for some feature spaceF, that will usually be Rnfor somen(however, the
range of such a mapping can be any Hilbert space , including such spaces of
inﬁnite dimension, as we will show later).
2. Given a sequence of labeled examples, S= (x1,y1),..., (xm,ym), create the
image sequence ˆS= (ψ(x1),y1),..., (ψ(xm),ym).
3. Train a linear predictor hover ˆS.
4. Predict the label of a test point, x, to beh(ψ(x)).
Note that, for every probability distribution DoverX×Y , we can readily
deﬁne its image probability distribution DψoverF×Y by setting, for every
subsetA⊆F×Y ,Dψ(A) =D(ψ−1(A)).1It follows that for every predictor h

over the feature space, LDψ(h) =LD(h◦ψ), whereh◦ψis the composition of h
ontoψ.
The success of this learning paradigm depends on choosing a good ψfor a
given learning task: that is, a ψthat will make the image of the data distribution
(close to being) linearly separable in the feature space, thus making the resulting
algorithm a good learner for a given task. Picking such an embedding requires
prior knowledge about that task. However, often some generic mappings that

ontoψ.
enable us to enrich the class of halfspaces and extend its expressiveness are used.
One notable example is polynomial mappings, which are a generalization of the
ψwe have seen in the previous example.
Recall that the prediction of a standard halfspace classiﬁer on an instance x
is based on the linear mapping x↦→⟨w,x⟩. We can generalize linear mappings
to a polynomial mapping, x↦→p(x), wherepis a multivariate polynomial of
degreek. For simplicity, consider ﬁrst the case in which xis 1 dimensional.
In that case, p(x) =∑k

j=0wjxj, where w∈Rk+1is the vector of coeﬃcients
of the polynomial we need to learn. We can rewrite p(x) =⟨w,ψ(x)⟩where
ψ:R→Rk+1is the mapping x↦→(1, x, x2, x3,...,xk). It follows that
learning akdegree polynomial over Rcan be done by learning a linear mapping
in the (k+ 1) dimensional feature space.
More generally, a degree kmultivariate polynomial from RntoRcan be writ-
ten as
p(x) =∑
J∈[n]r:r≤kwJr∏
i=1xJi. (16.1)
1This is deﬁned for every Asuch thatψ−1(A) is measurable with respect to D.

As before, we can rewrite p(x) =⟨w,ψ(x)⟩where now ψ:Rn→Rdis such
that for every J∈[n]r,r≤k, the coordinate of ψ(x) associated with Jis the
monomial∏r
i=1xJi.
Naturally, polynomial-based classiﬁers yield much richer hypothesis classes
than halfspaces. We have seen at the beginning of this chapter an example in
which the training set, in its original domain ( X=R), cannot be separable
by a halfspace, but after the embedding x↦→(x, x2) it is perfectly separable.
So, while the classiﬁer is always linear in the feature space, it can have highly
nonlinear behavior on the original space from which instances were sampled.
In general, we can choose any feature mapping ψthat maps the original in-
stances into some Hilbert space .2The Euclidean space Rdis a Hilbert space for
any ﬁnited. But there are also inﬁnite dimensional Hilbert spaces (as we shall
see later on in this chapter).
The bottom line of this discussion is that we can enrich the class of halfspaces

by ﬁrst applying a nonlinear mapping, ψ, that maps the instance space into some
feature space, and then learning a halfspace in that feature space. However, if
the range of ψis a high dimensional space we face two problems. First, the VC-
dimension of halfspaces in Rnisn+ 1, and therefore, if the range of ψis very
large, we need many more samples in order to learn a halfspace in the range
ofψ. Second, from the computational point of view, performing calculations in

the high dimensional space might be too costly. In fact, even the representation
of the vector win the feature space can be unrealistic. The ﬁrst issue can be
tackled using the paradigm of large margin (or low norm predictors), as we
already discussed in the previous chapter in the context of the SVM algorithm.
In the following section we address the computational issue.
16.2 The Kernel Trick
We have seen that embedding the input space into some high dimensional feature
space makes halfspace learning more expressive. However, the computational

complexity of such learning may still pose a serious hurdle – computing linear
separators over very high dimensional data may be computationally expensive.
The common solution to this concern is kernel based learning. The term “kernels”
is used in this context to describe inner products in the feature space. Given
an embedding ψof some domain space Xinto some Hilbert space, we deﬁne
the kernel function K(x,x′) =⟨ψ(x),ψ(x′)⟩. One can think of Kas specifying
similarity between instances and of the embedding ψas mapping the domain set

2A Hilbert space is a vector space with an inner product, which is also complete. A space is
complete if all Cauchy sequences in the space converge.
In our case, the norm ∥w∥is deﬁned by the inner product√
⟨w,w⟩. The reason we require
the range of ψto be in a Hilbert space is that projections in a Hilbert space are well
deﬁned. In particular, if Mis a linear subspace of a Hilbert space, then every xin the
Hilbert space can be written as a sum x=u+vwhere u∈Mand⟨v,w⟩= 0 for all

w∈M. We use this fact in the proof of the representer theorem given in the next section.

218 Kernel Methods
Xinto a space where these similarities are realized as inner products. It turns
out that many learning algorithms for halfspaces can be carried out just on the
basis of the values of the kernel function over pairs of domain points. The main
advantage of such algorithms is that they implement linear separators in high
dimensional feature spaces without having to specify points in that space or
expressing the embedding ψexplicitly. The remainder of this section is devoted
to constructing such algorithms.
In the previous chapter we saw that regularizing the norm of wyields a small
sample complexity even if the dimensionality of the feature space is high. Inter-
estingly, as we show later, regularizing the norm of wis also helpful in overcoming
the computational problem. To do so, ﬁrst note that all versions of the SVM op-
timization problem we have derived in the previous chapter are instances of the
following general problem:
min
w(f(⟨w,ψ(x1)⟩,...,⟨w,ψ(xm)⟩) +R(∥w∥)), (16.2)

min
wheref:Rm→Ris an arbitrary function and R:R+→Ris a monotoni-
cally nondecreasing function. For example, Soft-SVM for homogenous halfspaces
(Equation (15.6)) can be derived from Equation (16.2) by letting R(a) =λa2and
f(a1,...,am) =1
m∑
imax{0,1−yiai}. Similarly, Hard-SVM for nonhomogenous
halfspaces (Equation (15.2)) can be derived from Equation (16.2) by letting
R(a) =a2and letting f(a1,...,am) be 0 if there exists bsuch thatyi(ai+b)≥1
for alli, andf(a1,...,am) =∞otherwise.

m∑
The following theorem shows that there exists an optimal solution of Equa-
tion (16.2) that lies in the span of {ψ(x1),...,ψ (xm)}.
theorem 16.1 (Representer Theorem) Assume that ψis a mapping from Xto
a Hilbert space. Then, there exists a vector α∈Rmsuch that w=∑m
i=1αiψ(xi)
is an optimal solution of Equation (16.2).
Proof Letw⋆be an optimal solution of Equation (16.2). Because w⋆is an
element of a Hilbert space, we can rewrite w⋆as
w⋆=m∑
i=1αiψ(xi) +u,
where⟨u,ψ(xi)⟩= 0 for alli. Setw=w⋆−u. Clearly,∥w⋆∥2=∥w∥2+∥u∥2,

w⋆=m∑
thus∥w∥≤∥w⋆∥. SinceRis nondecreasing we obtain that R(∥w∥)≤R(∥w⋆∥).
Additionally, for all iwe have that
⟨w,ψ(xi)⟩=⟨w⋆−u,ψ(xi)⟩=⟨w⋆,ψ(xi)⟩,
hence
f(⟨w,ψ(x1)⟩,...,⟨w,ψ(xm)⟩) =f(⟨w⋆,ψ(x1)⟩,...,⟨w⋆,ψ(xm)⟩).
We have shown that the objective of Equation (16.2) at wcannot be larger
than the objective at w⋆and therefore wis also an optimal solution. Since
w=∑m
i=1αiψ(xi) we conclude our proof.

16.2 The Kernel Trick 219
On the basis of the representer theorem we can optimize Equation (16.2) with
respect to the coeﬃcients αinstead of the coeﬃcients was follows. Writing
w=∑m
j=1αjψ(xj) we have that for all i
⟨w,ψ(xi)⟩=⟨∑
jαjψ(xj),ψ(xi)⟩
=m∑
j=1αj⟨ψ(xj),ψ(xi)⟩.
∥w∥2=⟨∑
jαjψ(xj),∑
jαjψ(xj)⟩
=m∑
i,j=1αiαj⟨ψ(xi),ψ(xj)⟩.
LetK(x,x′) =⟨ψ(x),ψ(x′)⟩be a function that implements the kernel function
with respect to the embedding ψ. Instead of solving Equation (16.2) we can solve
the equivalent problem
min
α∈Rmf
m∑
j=1αjK(xj,x1),...,m∑
j=1αjK(xj,xm)

+R
√m∑
i,j=1αiαjK(xj,xi)
. (16.3)
To solve the optimization problem given in Equation (16.3), we do not need any
direct access to elements in the feature space. The only thing we should know is
how to calculate inner products in the feature space, or equivalently, to calculate
the kernel function. In fact, to solve Equation (16.3) we solely need to know the
value of the m×mmatrixGs.t.Gi,j=K(xi,xj), which is often called the
Gram matrix.

min
α∈Rmf
m∑

+R
√m∑
. (16.3)
In particular, specifying the preceding to the Soft-SVM problem given in Equa-
tion (15.6), we can rewrite the problem as
min
α∈Rm(
λαTGα+1
mm∑
i=1max{
0,1−yi(Gα)i})
, (16.4)
where (Gα)iis thei’th element of the vector obtained by multiplying the Gram
matrixGby the vector α. Note that Equation (16.4) can be written as quadratic
programming and hence can be solved eﬃciently. In the next section we describe
an even simpler algorithm for solving Soft-SVM with kernels.

min
α∈Rm(
λαTGα+1
mm∑
i=1max{
, (16.4)
Once we learn the coeﬃcients αwe can calculate the prediction on a new
instance by
⟨w,ψ(x)⟩=m∑
j=1αj⟨ψ(xj),ψ(x)⟩=m∑
j=1αjK(xj,x).
The advantage of working with kernels rather than directly optimizing win
the feature space is that in some situations the dimension of the feature space

220 Kernel Methods
is extremely large while implementing the kernel function is very simple. A few
examples are given in the following.
Example 16.1 (Polynomial Kernels) The kdegree polynomial kernel is deﬁned
to be
K(x,x′) = (1 +⟨x,x′⟩)k.
Now we will show that this is indeed a kernel function. That is, we will show
that there exists a mapping ψfrom the original space to some higher dimensional
space for which K(x,x′) =⟨ψ(x),ψ(x′)⟩. For simplicity, denote x0=x′
0= 1.
Then, we have
K(x,x′) = (1 +⟨x,x′⟩)k= (1 +⟨x,x′⟩)····· (1 +⟨x,x′⟩)
=
n∑
j=0xjx′
j
·····
n∑
j=0xjx′
j

=∑
J∈{0,1,...,n}kk∏
i=1xJix′
Ji
=∑
i=1xJik∏
i=1x′
Ji.
Now, if we deﬁne ψ:Rn→R(n+1)ksuch that for J∈{0,1,...,n}kthere is an
element ofψ(x) that equals∏k
i=1xJi, we obtain that
K(x,x′) =⟨ψ(x),ψ(x′)⟩.
Sinceψcontains all the monomials up to degree k, a halfspace over the range
ofψcorresponds to a polynomial predictor of degree kover the original space.

0= 1.
=
n∑
j=0xjx′
j
·····
n∑
j=0xjx′
j

=∑
i=1xJix′
Ji
=∑
i=1xJik∏
i=1x′
Ji.
Hence, learning a halfspace with a kdegree polynomial kernel enables us to learn
polynomial predictors of degree kover the original space.
Note that here the complexity of implementing KisO(n) while the dimension
of the feature space is on the order of nk.
Example 16.2 (Gaussian Kernel) Let the original instance space be Rand
consider the mapping ψwhere for each nonnegative integer n≥0 there exists
an element ψ(x)nthat equals1√
n!e−x2
2xn. Then,
⟨ψ(x), ψ(x′)⟩=∞∑
n=0(1√
n!e−x2
2xn)(1√
n!e−(x′)2
2(x′)n)
=e−x2+(x′)2
2∞∑
n=0((xx′)n
n!)

n!e−x2
2xn. Then,
n=0(1√
n!e−x2
2xn)(1√
n!e−(x′)2
2(x′)n)
2∞∑
n!)
=e−∥x−x′∥2
2.
Here the feature space is of inﬁnite dimension while evaluating the kernel is very

16.2 The Kernel Trick 221
simple. More generally, given a scalar σ > 0, the Gaussian kernel is deﬁned to
be
K(x,x′) =e−∥x−x′∥2
2σ.
Intuitively, the Gaussian kernel sets the inner product in the feature space
between x,x′to be close to zero if the instances are far away from each other
(in the original domain) and close to 1 if they are close. σis a parameter that
controls the scale determining what we mean by “close.” It is easy to verify that
Kimplements an inner product in a space in which for any nand any monomial
of orderkthere exists an element of ψ(x) that equals1√
n!e−∥x∥2
2∏n
i=1xJi.
Hence, we can learn any polynomial predictor over the original space by using a
Gaussian kernel.
Recall that the VC-dimension of the class of all polynomial predictors is inﬁ-
nite (see Exercise 12). There is no contradiction, because the sample complexity
required to learn with Gaussian kernels depends on the margin in the feature
space, which will be large if we are lucky, but can in general be arbitrarily small.

n!e−∥x∥2
2∏n
i=1xJi.
The Gaussian kernel is also called the RBF kernel, for “Radial Basis Func-
tions.”
16.2.1 Kernels as a Way to Express Prior Knowledge
As we discussed previously, a feature mapping, ψ, may be viewed as expanding
the class of linear classiﬁers to a richer class (corresponding to linear classiﬁers
over the feature space). However, as discussed in the book so far, the suitability
of any hypothesis class to a given learning task depends on the nature of that

tions.”
task. One can therefore think of an embedding ψas a way to express and utilize
prior knowledge about the problem at hand. For example, if we believe that
positive examples can be distinguished by some ellipse, we can deﬁne ψto be all
the monomials up to order 2, or use a degree 2 polynomial kernel.
As a more realistic example, consider the task of learning to ﬁnd a sequence of
characters (“signature”) in a ﬁle that indicates whether it contains a virus or not.
Formally, letXdbe the set of all strings of length at most dover some alphabet

set Σ. The hypothesis class that one wishes to learn is H={hv:v∈Xd}, where,
for a string x∈Xd,hv(x) is 1 iﬀvis a substring of x(andhv(x) =−1 otherwise).
Let us show how using an appropriate embedding this class can be realized by
linear classiﬁers over the resulting feature space. Consider a mapping ψto a space
Rswheres=|Xd|, so that each coordinate of ψ(x) corresponds to some string v
and indicates whether vis a substring of x(that is, for every x∈Xd,ψ(x) is a
vector in{0,1}|Xd|). Note that the dimension of this feature space is exponential

ind. It is not hard to see that every member of the class Hcan be realized by
composing a linear classiﬁer over ψ(x), and, moreover, by such a halfspace whose
norm is 1 and that attains a margin of 1 (see Exercise 1). Furthermore, for every
x∈X,∥ψ(x)∥=O(d). So, overall, it is learnable using SVM with a sample

222 Kernel Methods
complexity that is polynomial in d. However, the dimension of the feature space
is exponential in dso a direct implementation of SVM over the feature space is
problematic. Luckily, it is easy to calculate the inner product in the feature space
(i.e., the kernel function) without explicitly mapping instances into the feature
space. Indeed, K(x,x′) is simply the number of common substrings of xandx′,
which can be easily calculated in time polynomial in d.
This example also demonstrates how feature mapping enables us to use halfspaces
for nonvectorial domains.
16.2.2 Characterizing Kernel Functions*
As we have discussed in the previous section, we can think of the speciﬁcation of
the kernel matrix as a way to express prior knowledge. Consider a given similarity
function of the form K:X×X→ R. Is it a valid kernel function? That is, does
it represent an inner product between ψ(x) andψ(x′) for some feature mapping
ψ? The following lemma gives a suﬃcient and necessary condition.

lemma 16.2 A symmetric function K:X×X → Rimplements an inner
product in some Hilbert space if and only if it is positive semideﬁnite; namely,
for all x1,...,xm, the Gram matrix, Gi,j=K(xi,xj), is a positive semideﬁnite
matrix.
Proof It is trivial to see that if Kimplements an inner product in some Hilbert
space then the Gram matrix is positive semideﬁnite. For the other direction,
deﬁne the space of functions over XasRX={f:X → R}. For each x∈X
letψ(x) be the function x↦→K(·,x). Deﬁne a vector space by taking all linear

matrix.
combinations of elements of the form K(·,x). Deﬁne an inner product on this
vector space to be
⟨∑
iαiK(·,xi),∑
jβjK(·,x′
j)⟩
=∑
i,jαiβjK(xi,x′
j).
This is a valid inner product since it is symmetric (because Kis symmetric), it is
linear (immediate), and it is positive deﬁnite (it is easy to see that K(x,x)≥0
with equality only for ψ(x) being the zero function). Clearly,
⟨ψ(x),ψ(x′)⟩=⟨K(·,x),K(·,x′)⟩=K(x,x′),
16.3 Implementing Soft-SVM with Kernels
Next, we turn to solving Soft-SVM with kernels. While we could have designed

⟨∑
jβjK(·,x′
j)⟩
=∑
j).
an algorithm for solving Equation (16.4), there is an even simpler approach that

16.3 Implementing Soft-SVM with Kernels 223
directly tackles the Soft-SVM optimization problem in the feature space,
min
w(
λ
2∥w∥2+1
mm∑
i=1max{0,1−y⟨w,ψ(xi)⟩})
, (16.5)
while only using kernel evaluations. The basic observation is that the vector w(t)
maintained by the SGD procedure we have described in Section 15.5 is always in
the linear span of {ψ(x1),...,ψ (xm)}. Therefore, rather than maintaining w(t)
we can maintain the corresponding coeﬃcients α.
Formally, let Kbe the kernel function, namely, for all x,x′,K(x,x′) =
⟨ψ(x),ψ(x′)⟩. We shall maintain two vectors in Rm, corresponding to two vectors
θ(t)andw(t)deﬁned in the SGD procedure of Section 15.5. That is, β(t)will be
a vector such that
θ(t)=m∑
j=1β(t)
jψ(xj) (16.6)
andα(t)be such that
w(t)=m∑
j=1α(t)
jψ(xj). (16.7)
The vectorsβandαare updated according to the following procedure.
SGD for Solving Soft-SVM with Kernels
Goal: Solve Equation (16.5)
Initialize:β(1)=0
Letα(t)=1
λtβ(t)

θ(t)=m∑
j=1β(t)
w(t)=m∑
j=1α(t)
Letα(t)=1
λtβ(t)
For allj̸=isetβ(t+1)
j =β(t)
j
If (yi∑m
j=1α(t)
jK(xj,xi)<1)
Setβ(t+1)
i =β(t)
i+yi
Else
Setβ(t+1)
i =β(t)
i
Output: ¯w=∑m
j=1¯αjψ(xj) where ¯α=1
T∑T
t=1α(t)
The following lemma shows that the preceding implementation is equivalent
to running the SGD procedure described in Section 15.5 on the feature space.
lemma 16.3 Letˆwbe the output of the SGD procedure described in Sec-
tion 15.5, when applied on the feature space, and let ¯w=∑m
j=1¯αjψ(xj)be
the output of applying SGD with kernels. Then ¯w=ˆw.

j =β(t)
j
If (yi∑m
j=1α(t)
Setβ(t+1)
i =β(t)
i+yi
Else
Setβ(t+1)
i =β(t)
i
T∑T
t=1α(t)
Proof We will show that for every tEquation (16.6) holds, where θ(t)is the
result of running the SGD procedure described in Section 15.5 in the feature

224 Kernel Methods
space. By the deﬁnition of α(t)=1
λtβ(t)andw(t)=1
λtθ(t), this claim implies
that Equation (16.7) also holds, and the proof of our lemma will follow. To prove
that Equation (16.6) holds we use a simple inductive argument. For t= 1 the
claim trivially holds. Assume it holds for t≥1. Then,
yi⣨
w(t),ψ(xi)⟩
=yi⟨∑
jα(t)
jψ(xj),ψ(xi)⟩
=yim∑
j=1α(t)
jK(xj,xi).
Hence, the condition in the two algorithms is equivalent and if we update θwe
have
θ(t+1)=θ(t)+yiψ(xi) =m∑
j=1β(t)
jψ(xj) +yiψ(xi) =m∑
j=1β(t+1)
jψ(xj),
16.4 Summary
Mappings from the given domain to some higher dimensional space, on which a
halfspace predictor is used, can be highly powerful. We beneﬁt from a rich and
complex hypothesis class, yet need to solve the problems of high sample and
computational complexities. In Chapter 10, we discussed the AdaBoost algo-
rithm, which faces these challenges by using a weak learner: Even though we’re

have
j=1β(t)
j=1β(t+1)
jψ(xj),
in a very high dimensional space, we have an “oracle” that bestows on us a
single good coordinate to work with on each iteration. In this chapter we intro-
duced a diﬀerent approach, the kernel trick. The idea is that in order to ﬁnd a
halfspace predictor in the high dimensional space, we do not need to know the
representation of instances in that space, but rather the values of inner products
between the mapped instances. Calculating inner products between instances in

the high dimensional space without using their representation in that space is
done using kernel functions. We have also shown how the SGD algorithm can be
implemented using kernels.
The ideas of feature mapping and the kernel trick allow us to use the framework
of halfspaces and linear predictors for nonvectorial data. We demonstrated how
kernels can be used to learn predictors over the domain of strings.
We presented the applicability of the kernel trick in SVM. However, the kernel

trick can be applied in many other algorithms. A few examples are given as
exercises.
This chapter ends the series of chapters on linear predictors and convex prob-
lems. The next two chapters deal with completely diﬀerent types of hypothesis
classes.

16.5 Bibliographic Remarks
In the context of SVM, the kernel-trick has been introduced in Boser et al. (1992).
See also Aizerman, Braverman & Rozonoer (1964). The observation that the
kernel-trick can be applied whenever an algorithm only relies on inner products
was ﬁrst stated by Sch¨ olkopf, Smola & M¨ uller (1998). The proof of the representer
theorem is given in (Sch¨ olkopf, Herbrich, Smola & Williamson 2000, Sch¨ olkopf,
Herbrich & Smola 2001). The conditions stated in Lemma 16.2 are simpliﬁcation
of conditions due to Mercer. Many useful kernel functions have been introduced
in the literature for various applications. We refer the reader to Sch¨ olkopf &
Smola (2002).
16.6 Exercises
1. Consider the task of ﬁnding a sequence of characters in a ﬁle, as described
in Section 16.2.1. Show that every member of the class Hcan be realized by
composing a linear classiﬁer over ψ(x), whose norm is 1 and that attains a
margin of 1.

2.Kernelized Perceptron: Show how to run the Perceptron algorithm while
only accessing the instances via the kernel function. Hint: The derivation is
similar to the derivation of implementing SGD with kernels.
3.Kernel Ridge Regression: The ridge regression problem, with a feature
mappingψ, is the problem of ﬁnding a vector wthat minimizes the function
f(w) =λ∥w∥2+1
2mm∑
i=1(⟨w,ψ(xi)⟩−yi)2, (16.8)
and then returning the predictor
h(x) =⟨w,x⟩.
Show how to implement the ridge regression algorithm with kernels.

2mm∑
Hint: The representer theorem tells us that there exists a vector α∈Rm
such that∑m
i=1αiψ(xi) is a minimizer of Equation (16.8).
1. LetGbe the Gram matrix with regard to SandK. That is, Gij=
K(xi,xj). Deﬁneg:Rm→Rby
g(α) =λ·αTGα+1
2mm∑
i=1(⟨α,G·,i⟩−yi)2, (16.9)
whereG·,iis thei’th column of G. Show that if α∗minimizes Equa-
tion (16.9) then w∗=∑m
i=1α∗
iψ(xi) is a minimizer of f.
2. Find a closed form expression for α∗.
4. LetNbe any positive integer. For every x,x′∈{1,...,N}deﬁne
K(x,x′) = min{x,x′}.

226 Kernel Methods
Prove that Kis a valid kernel; namely, ﬁnd a mapping ψ:{1,...,N}→H
whereHis some Hilbert space, such that
∀x,x′∈{1,...,N}, K(x,x′) =⟨ψ(x),ψ(x′)⟩.
5. A supermarket manager would like to learn which of his customers have babies
on the basis of their shopping carts. Speciﬁcally, he sampled i.i.d. customers,
where for customer i, letxi⊂ {1,...,d}denote the subset of items the
customer bought, and let yi∈ {± 1}be the label indicating whether this
customer has a baby. As prior knowledge, the manager knows that there are
kitems such that the label is determined to be 1 iﬀ the customer bought
at least one of these kitems. Of course, the identity of these kitems is not
known (otherwise, there was nothing to learn). In addition, according to the
store regulation, each customer can buy at most sitems. Help the manager to
design a learning algorithm such that both its time complexity and its sample
complexity are polynomial in s,k, and 1/ϵ.

6. LetXbe an instance set and let ψbe a feature mapping of Xinto some
Hilbert feature space V. LetK:X×X → Rbe a kernel function that
implements inner products in the feature space V.
Consider the binary classiﬁcation algorithm that predicts the label of
an unseen instance according to the class with the closest average. Formally,
given a training sequence S= (x1,y1),..., (xm,ym), for every y∈{± 1}we
deﬁne
cy=1
my∑
i:yi=yψ(xi).
wheremy=|{i:yi=y}|. We assume that m+andm−are nonzero. Then,

deﬁne
cy=1
my∑
the algorithm outputs the following decision rule:
h(x) ={
1∥ψ(x)−c+∥≤∥ψ(x)−c−∥
1. Let w=c+−c−and letb=1
2(∥c−∥2−∥c+∥2). Show that
h(x) = sign(⟨w,ψ(x)⟩+b).
2. Show how to express h(x) on the basis of the kernel function, and without
accessing individual entries of ψ(x) orw.

17 Multiclass, Ranking, and Complex
Prediction Problems
Multiclass categorization is the problem of classifying instances into one of several
possible target classes. That is, we are aiming at learning a predictor h:X→Y ,
whereYis a ﬁnite set of categories. Applications include, for example, catego-
rizing documents according to topic ( Xis the set of documents and Yis the set
of possible topics) or determining which object appears in a given image ( Xis
the set of images and Yis the set of possible objects).
The centrality of the multiclass learning problem has spurred the development
of various approaches for tackling the task. Perhaps the most straightforward
approach is a reduction from multiclass classiﬁcation to binary classiﬁcation. In
Section 17.1 we discuss the most common two reductions as well as the main
drawback of the reduction approach.
We then turn to describe a family of linear predictors for multiclass problems.
Relying on the RLM and SGD frameworks from previous chapters, we describe

several practical algorithms for multiclass prediction.
In Section 17.3 we show how to use the multiclass machinery for complex pre-
diction problems in which Ycan be extremely large but has some structure on
it. This task is often called structured output learning . In particular, we demon-
strate this approach for the task of recognizing handwritten words, in which Y
is the set of all possible strings of some bounded length (hence, the size of Yis
exponential in the maximal length of a word).

Finally, in Section 17.4 and Section 17.5 we discuss ranking problems in which
the learner should order a set of instances according to their “relevance.” A typ-
ical application is ordering results of a search engine according to their relevance
to the query. We describe several performance measures that are adequate for
assessing the performance of ranking predictors and describe how to learn linear
predictors for ranking problems eﬃciently.
17.1 One-versus-All and All-Pairs

The simplest approach to tackle multiclass prediction problems is by reduction
to binary classiﬁcation. Recall that in multiclass prediction we would like to learn
a functionh:X→Y . Without loss of generality let us denote Y={1,...,k}.
In the One-versus-All method (a.k.a. One-versus-Rest) we train kbinary clas-

228 Multiclass, Ranking, and Complex Prediction Problems
siﬁers, each of which discriminates between one class and the rest of the classes.
That is, given a training set S= (x1,y1),..., (xm,ym), where every yiis inY, we
constructkbinary training sets, S1,...,Sk, whereSi= (x1,(−1)1[y1̸=i]),..., (xm,(−1)1[ym̸=i]).
In words,Siis the set of instances labeled 1 if their label in Swasi, and−1
otherwise. For every i∈[k] we train a binary predictor hi:X→{± 1}based on
Si, hoping that hi(x) should equal 1 if and only if xbelongs to class i. Then,
givenh1,...,hk, we construct a multiclass predictor using the rule
h(x)∈argmax
i∈[k]hi(x). (17.1)
When more than one binary hypothesis predicts “1” we should somehow decide
which class to predict (e.g., we can arbitrarily decide to break ties by taking the
minimal index in argmaxihi(x)). A better approach can be applied whenever
eachhihides additional information, which can be interpreted as the conﬁdence

in the prediction y=i. For example, this is the case in halfspaces, where the
actual prediction is sign( ⟨w,x⟩), but we can interpret ⟨w,x⟩as the conﬁdence
in the prediction. In such cases, we can apply the multiclass rule given in Equa-
tion (17.1) on the real valued predictions. A pseudocode of the One-versus-All
approach is given in the following.
One-versus-All
input:
training set S= (x1, y1),..., (xm, ym)
algorithm for binary classiﬁcation A
foreachi∈Y
letSi= (x1,(−1)1[y1̸=i]),..., (xm,(−1)1[ym̸=i])
lethi=A(Si)
output:

input:
output:
the multiclass hypothesis deﬁned by h(x)∈argmaxi∈Yhi(x)
Another popular reduction is the All-Pairs approach, in which all pairs of
classes are compared to each other. Formally, given a training set S= (x1,y1),..., (xm,ym),
where every yiis in [k], for every 1≤i<j≤kwe construct a binary training
sequence,Si,j, containing all examples from Swhose label is either iorj. For
each such an example, we set the binary label in Si,jto be +1 if the multiclass
label inSisiand−1 if the multiclass label in Sisj. Next, we train a binary

classiﬁcation algorithm based on every Si,jto gethi,j. Finally, we construct
a multiclass classiﬁer by predicting the class that had the highest number of
“wins.” A pseudocode of the All-Pairs approach is given in the following.

17.1 One-versus-All and All-Pairs 229
All-Pairs
input:
foreachi,j∈Ys.t.i<j
initializeSi,jto be the empty sequence
fort= 1,...,m
Ifyt=iadd (xt,1) toSi,j
Ifyt=jadd (xt,−1) toSi,j
lethi,j=A(Si,j)
output:
the multiclass hypothesis deﬁned by
h(x)∈argmaxi∈Y(∑
j∈Ysign(j−i)hi,j(x))
Although reduction methods such as the One-versus-All and All-Pairs are
simple and easy to construct from existing algorithms, their simplicity has a
price. The binary learner is not aware of the fact that we are going to use its
output hypotheses for constructing a multiclass predictor, and this might lead
to suboptimal results, as illustrated in the following example.
Example 17.1 Consider a multiclass categorization problem in which the in-
stance space isX=R2and the label set is Y={1,2,3}. Suppose that instances
of the diﬀerent classes are located in nonintersecting balls as depicted in the fol-
lowing.
1 2 3

lowing.
1 2 3
Suppose that the probability masses of classes 1 ,2,3 are 40%,20%,and 40%,
respectively. Consider the application of One-versus-All to this problem, and as-
sume that the binary classiﬁcation algorithm used by One-versus-All is ERM
with respect to the hypothesis class of halfspaces. Observe that for the prob-
lem of discriminating between class 2 and the rest of the classes, the optimal
halfspace would be the all negative classiﬁer. Therefore, the multiclass predic-
tor constructed by One-versus-All might err on all the examples from class 2

(this will be the case if the tie in the deﬁnition of h(x) is broken by the nu-
merical value of the class label). In contrast, if we choose hi(x) =⟨wi,x⟩,
where w1=(
−1√
2,1√
2)
,w2= (0,1), and w3=(
1√
2,1√
2)
, then the classi-
ﬁer deﬁned by h(x) = argmaxihi(x) perfectly predicts all the examples. We see

230 Multiclass, Ranking, and Complex Prediction Problems
that even though the approximation error of the class of predictors of the form
h(x) = argmaxi⟨wi,x⟩is zero, the One-versus-All approach might fail to ﬁnd a
good predictor from this class.
17.2 Linear Multiclass Predictors
In light of the inadequacy of reduction methods, in this section we study a more
direct approach for learning multiclass predictors. We describe the family of
linear multiclass predictors. To motivate the construction of this family, recall
that a linear predictor for binary classiﬁcation (i.e., a halfspace) takes the form
h(x) = sign(⟨w,x⟩).
An equivalent way to express the prediction is as follows:
h(x) = argmax
y∈{±1}⟨w,yx⟩,
whereyxis the vector obtained by multiplying each element of xbyy.
This representation leads to a natural generalization of halfspaces to multiclass
problems as follows. Let Ψ : X×Y→ Rdbe a class-sensitive feature mapping .
That is, Ψ takes as input a pair ( x,y) and maps it into a ddimensional feature

vector. Intuitively, we can think of the elements of Ψ( x,y) as score functions that
assess how well the label yﬁts the instance x. We will elaborate on Ψ later on.
Given Ψ and a vector w∈Rd, we can deﬁne a multiclass predictor, h:X→Y ,
y∈Y⟨w,Ψ(x,y)⟩.
That is, the prediction of hfor the input xis the label that achieves the highest
weighted score, where weighting is according to the vector w.
LetWbe some set of vectors in Rd, for example, W={w∈Rd:∥w∥≤B},

for some scalar B > 0. Each pair (Ψ ,W) deﬁnes a hypothesis class of multiclass
predictors:
HΨ,W={x↦→argmax
y∈Y⟨w,Ψ(x,y)⟩:w∈W}.
Of course, the immediate question, which we discuss in the sequel, is how to
construct a good Ψ. Note that if Y={±1}and we set Ψ( x,y) =yxand
W=Rd, thenHΨ,Wbecomes the hypothesis class of homogeneous halfspace
predictors for binary classiﬁcation.
17.2.1 How to Construct Ψ
As mentioned before, we can think of the elements of Ψ( x,y) as score functions

that assess how well the label yﬁts the instance x. Naturally, designing a good Ψ
is similar to the problem of designing a good feature mapping (as we discussed in

17.2 Linear Multiclass Predictors 231
Chapter 16 and as we will discuss in more detail in Chapter 25). Two examples
of useful constructions are given in the following.
The Multivector Construction:
LetY={1,...,k}and letX=Rn. We deﬁne Ψ :X×Y→ Rd, whered=nk,
as follows
Ψ(x, y) = [ 0,..., 0
∈R(y−1)n, x1,...,xn
∈Rn,0,..., 0
∈R(k−y)n]. (17.2)
That is, Ψ( x,y) is composed of kvectors, each of which is of dimension n, where
we set all the vectors to be the all zeros vector except the y’th vector, which is
set to be x. It follows that we can think of w∈Rnkas being composed of k
weight vectors in Rn, that is, w= [w1;...;wk], hence the name multivec-
tor construction . By the construction we have that ⟨w,Ψ(x,y)⟩=⟨wy,x⟩, and
therefore the multiclass prediction becomes
y∈Y⟨wy,x⟩.
A geometric illustration of the multiclass prediction over X=R2is given in the
following.
w1w2
w3 w4
TF-IDF:
The previous deﬁnition of Ψ( x, y) does not incorporate any prior knowledge

w1w2
w3 w4
TF-IDF:
about the problem. We next describe an example of a feature function Ψ that
does incorporate prior knowledge. Let Xbe a set of text documents and Ybe a
set of possible topics. Let dbe a size of a dictionary of words. For each word in the
dictionary, whose corresponding index is j, letTF(j,x) be the number of times
the word corresponding to jappears in the document x. This quantity is called
Term-Frequency. Additionally, let DF(j,y) be the number of times the word
corresponding to jappears in documents in our training set that are not about

topicy. This quantity is called Document-Frequency and measures whether word
jis frequent in other topics. Now, deﬁne Ψ : X×Y→ Rdto be such that
Ψj(x,y) =TF(j,x) log(
m
DF(j,y))
,
wheremis the total number of documents in our training set. The preced-
ing quantity is called term-frequency-inverse-document-frequency or TF-IDF for

232 Multiclass, Ranking, and Complex Prediction Problems
short. Intuitively, Ψ j(x, y) should be large if the word corresponding to jap-
pears a lot in the document xbut does not appear at all in documents that are
not on topic y. If this is the case, we tend to believe that the document xis on
topicy. Note that unlike the multivector construction described previously, in
the current construction the dimension of Ψ does not depend on the number of
topics (i.e., the size of Y).
17.2.2 Cost-Sensitive Classiﬁcation
So far we used the zero-one loss as our performance measure of the quality of
h(x). That is, the loss of a hypothesis hon an example ( x,y) is 1 ifh(x)̸=yand
0 otherwise. In some situations it makes more sense to penalize diﬀerent levels
of loss for diﬀerent mistakes. For example, in object recognition tasks, it is less
severe to predict that an image of a tiger contains a cat than predicting that
the image contains a whale. This can be modeled by specifying a loss function,

∆ :Y×Y → R+, where for every pair of labels, y′,y, the loss of predicting
the labely′when the correct label is yis deﬁned to be ∆( y′,y). We assume
that ∆(y,y) = 0. Note that the zero-one loss can be easily modeled by setting
∆(y′,y) =1[y′̸=y].
17.2.3 ERM
We have deﬁned the hypothesis class HΨ,Wand speciﬁed a loss function ∆. To
learn the class with respect to the loss function, we can apply the ERM rule with
respect to this class. That is, we search for a multiclass hypothesis h∈H Ψ,W,

17.2.3 ERM
parameterized by a vector w, that minimizes the empirical risk with respect to
∆,
LS(h) =1
mm∑
i=1∆(h(xi),yi).
We now show that when W=Rdand we are in the realizable case, then it is
possible to solve the ERM problem eﬃciently using linear programming. Indeed,
in the realizable case, we need to ﬁnd a vector w∈Rdthat satisﬁes
∀i∈[m], yi= argmax
y∈Y⟨w,Ψ(xi,y)⟩.
Equivalently, we need that wwill satisfy the following set of linear inequalities
∀i∈[m],∀y∈Y\{yi},⟨w,Ψ(xi,yi)⟩>⟨w,Ψ(xi,y)⟩.

∆,
LS(h) =1
mm∑
Finding wthat satisﬁes the preceding set of linear equations amounts to solving
a linear program.
As in the case of binary classiﬁcation, it is also possible to use a generalization
of the Perceptron algorithm for solving the ERM problem. See Exercise 2.
In the nonrealizable case, solving the ERM problem is in general computa-
tionally hard. We tackle this diﬃculty using the method of convex surrogate

17.2 Linear Multiclass Predictors 233
loss functions (see Section 12.3). In particular, we generalize the hinge loss to
17.2.4 Generalized Hinge Loss
Recall that in binary classiﬁcation, the hinge loss is deﬁned to be max {0,1−
y⟨w,x⟩}. We now generalize the hinge loss to multiclass predictors of the form
hw(x) = argmax
y′∈Y⟨w,Ψ(x,y′)⟩.
Recall that a surrogate convex loss should upper bound the original nonconvex
loss, which in our case is ∆( hw(x),y). To derive an upper bound on ∆( hw(x),y)
we ﬁrst note that the deﬁnition of hw(x) implies that
⟨w,Ψ(x,y)⟩≤⟨w,Ψ(x,hw(x))⟩.
∆(hw(x),y)≤∆(hw(x),y) +⟨w,Ψ(x,hw(x))−Ψ(x,y)⟩.
Sincehw(x)∈Ywe can upper bound the right-hand side of the preceding by
max
y′∈Y(∆(y′,y) +⟨w,Ψ(x,y′)−Ψ(x,y)⟩)def=ℓ(w,(x,y)). (17.3)
We use the term “generalized hinge loss” to denote the preceding expression. As
we have shown, ℓ(w,(x,y))≥∆(hw(x),y). Furthermore, equality holds when-
ever the score of the correct label is larger than the score of any other label, y′,

max
by at least ∆( y′,y), namely,
∀y′∈Y\{y},⟨w,Ψ(x,y)⟩≥⟨w,Ψ(x,y′)⟩+ ∆(y′,y).
It is also immediate to see that ℓ(w,(x,y)) is a convex function with respect to w
since it is a maximum over linear functions of w(see Claim 12.5 in Chapter 12),
and thatℓ(w,(x,y)) isρ-Lipschitz with ρ= maxy′∈Y∥Ψ(x,y′)−Ψ(x,y)∥.
Remark 17.2 We use the name “generalized hinge loss” since in the binary
case, whenY={±1}, if we set Ψ( x,y) =yx
2, then the generalized hinge loss
becomes the vanilla hinge loss for binary classiﬁcation,

ℓ(w,(x,y)) = max{0,1−y⟨w,x⟩}.
Geometric Intuition:
The feature function Ψ : X×Y → Rdmaps each xinto|Y|vectors in Rd.
The value of ℓ(w,(x,y)) will be zero if there exists a direction wsuch that
when projecting the |Y|vectors onto this direction we obtain that each vector is
represented by the scalar ⟨w,Ψ(x,y)⟩, and we can rank the diﬀerent points on
the basis of these scalars so that
•The point corresponding to the correct yis top-ranked

234 Multiclass, Ranking, and Complex Prediction Problems
•For eachy′̸=y, the diﬀerence between ⟨w,Ψ(x,y)⟩and⟨w,Ψ(x,y′)⟩is larger
than the loss of predicting y′instead ofy. The diﬀerence⟨w,Ψ(x,y)⟩−
⟨w,Ψ(x,y′)⟩is also referred to as the “margin” (see Section 15.1).
This is illustrated in the following ﬁgure:
w
Ψ(x,y)
Ψ(x,y′)Ψ(x,y′′)
≥∆(y,y
′)≥
∆(y,y′′)
17.2.5 Multiclass SVM and SGD
Once we have deﬁned the generalized hinge loss, we obtain a convex-Lipschitz
learning problem and we can apply our general techniques for solving such prob-
lems. In particular, the RLM technique we have studied in Chapter 13 yields the
multiclass SVM rule:
Multiclass SVM
parameters:
regularization parameter λ>0
loss function ∆ :Y×Y→ R+
class-sensitive feature mapping Ψ : X×Y→ Rd
solve:
min
w∈Rd(
λ∥w∥2+1
mm∑
i=1max
y′∈Y(∆(y′,yi) +⟨w,Ψ(xi,y′)−Ψ(xi,yi)⟩))
output the predictor hw(x) = argmaxy∈Y⟨w,Ψ(x,y)⟩
We can solve the optimization problem associated with multiclass SVM us-

solve:
min
w∈Rd(
λ∥w∥2+1
mm∑
i=1max
ing generic convex optimization algorithms (or using the method described in
Section 15.5). Let us analyze the risk of the resulting hypothesis. The analysis
seamlessly follows from our general analysis for convex-Lipschitz problems given
in Chapter 13. In particular, applying Corollary 13.8 and using the fact that the
generalized hinge loss upper bounds the ∆ loss, we immediately obtain an analog
of Corollary 15.7:
corollary 17.1 LetDbe a distribution over X×Y , let Ψ :X×Y→ Rd,
and assume that for all x∈X andy∈Y we have∥Ψ(x,y)∥≤ρ/2. LetB > 0.

17.2 Linear Multiclass Predictors 235
Consider running Multiclass SVM with λ=√
2ρ2
B2mon a training set S∼Dm
and lethwbe the output of Multiclass SVM. Then,
E
S∼Dm[L∆
D(hw)]≤E
S∼Dm[Lg−hinge
D (w)]≤ min
u:∥u∥≤BLg−hinge
D (u) +√
8ρ2B2
m,
whereL∆
D(h) =E(x,y)∼D[∆(h(x),y)]andLg−hinge
D (w) =E(x,y)∼D[ℓ(w,(x,y))]
withℓbeing the generalized hinge-loss as deﬁned in Equation (17.3).
We can also apply the SGD learning framework for minimizing Lg−hinge
D (w) as
described in Chapter 14. Recall Claim 14.6, which dealt with subgradients of max
functions. In light of this claim, in order to ﬁnd a subgradient of the generalized
hinge loss all we need to do is to ﬁnd y∈Ythat achieves the maximum in the
deﬁnition of the generalized hinge loss. This yields the following algorithm:
SGD for Multiclass Learning
Scalarη>0, integerT >0
initialize: w(1)=0∈Rd
sample ( x, y)∼D
ﬁnd ˆy∈argmaxy′∈Y(
∆(y′,y) +⟨w(t),Ψ(x,y′)−Ψ(x,y)⟩)

setvt= Ψ(x,ˆy)−Ψ(x,y)
T∑T
t=1w(t)
Our general analysis of SGD given in Corollary 14.12 immediately implies:
corollary 17.2 LetDbe a distribution over X×Y , let Ψ :X×Y→ Rd,
Then, for every ϵ>0, if we run SGD for multiclass learning with a number of
iterations (i.e., number of examples)
T≥B2ρ2
ϵ2
B2
E
S∼Dm[L∆
D(h¯w)]≤E
D (¯w)]≤ min
D (u) +ϵ.

T∑T
t=1w(t)
T≥B2ρ2
ϵ2
B2
E
S∼Dm[L∆
D(h¯w)]≤E
D (u) +ϵ.
Remark 17.3 It is interesting to note that the risk bounds given in Corol-
lary 17.1 and Corollary 17.2 do not depend explicitly on the size of the label
setY, a fact we will rely on in the next section. However, the bounds may de-
pend implicitly on the size of Yvia the norm of Ψ( x,y) and the fact that the
bounds are meaningful only when there exists some vector u,∥u∥≤B, for which
Lg−hinge
D (u) is not excessively large.

236 Multiclass, Ranking, and Complex Prediction Problems
17.3 Structured Output Prediction
Structured output prediction problems are multiclass problems in which Yis
very large but is endowed with a predeﬁned structure. The structure plays a
key role in constructing eﬃcient algorithms. To motivate structured learning
problems, consider the problem of optical character recognition (OCR). Suppose
we receive an image of some handwritten word and would like to predict which
word is written in the image. To simplify the setting, suppose we know how to
segment the image into a sequence of images, each of which contains a patch of
the image corresponding to a single letter. Therefore, Xis the set of sequences
of images andYis the set of sequences of letters. Note that the size of Ygrows
exponentially with the maximal length of a word. An example of an image x
corresponding to the label y= “workable” is given in the following.
To tackle structure prediction we can rely on the family of linear predictors

described in the previous section. In particular, we need to deﬁne a reasonable
loss function for the problem, ∆, as well as a good class-sensitive feature mapping,
Ψ. By “good” we mean a feature mapping that will lead to a low approximation
error for the class of linear predictors with respect to Ψ and ∆. Once we do this,
we can rely, for example, on the SGD learning algorithm deﬁned in the previous
section.
However, the huge size of Yposes several challenges:
1. To apply the multiclass prediction we need to solve a maximization problem

section.
overY. How can we predict eﬃciently when Yis so large?
2. How do we train weﬃciently? In particular, to apply the SGD rule we again
need to solve a maximization problem over Y.
3. How can we avoid overﬁtting?
In the previous section we have already shown that the sample complexity of
learning a linear multiclass predictor does not depend explicitly on the number
of classes. We just need to make sure that the norm of the range of Ψ is not too
large. This will take care of the overﬁtting problem. To tackle the computational

challenges we rely on the structure of the problem, and deﬁne the functions Ψ and
∆ so that calculating the maximization problems in the deﬁnition of hwand in
the SGD algorithm can be performed eﬃciently. In the following we demonstrate
one way to achieve these goals for the OCR task mentioned previously.
To simplify the presentation, let us assume that all the words in Yare of length
rand that the number of diﬀerent letters in our alphabet is q. Letyandy′be two

17.3 Structured Output Prediction 237
words (i.e., sequences of letters) in Y. We deﬁne the function ∆( y′,y) to be the
average number of letters that are diﬀerent in y′andy, namely,1
r∑r
i=11[yi̸=y′
i].
Next, let us deﬁne a class-sensitive feature mapping Ψ( x,y). It will be conve-
nient to think about xas a matrix of size n×r, wherenis the number of pixels
in each image, and ris the number of images in the sequence. The j’th column
ofxcorresponds to the j’th image in the sequence (encoded as a vector of gray
level values of pixels). The dimension of the range of Ψ is set to be d=nq+q2.
The ﬁrstnqfeature functions are “type 1” features and take the form:
Ψi,j,1(x,y) =1
rr∑
t=1xi,t1[yt=j].
That is, we sum the value of the i’th pixel only over the images for which y
assigns the letter j. The triple index ( i,j,1) indicates that we are dealing with
feature (i,j) of type 1. Intuitively, such features can capture pixels in the image
whose gray level values are indicative of a certain letter. The second type of

rr∑
features take the form
Ψi,j,2(x,y) =1
rr∑
t=21[yt=i]1[yt−1=j].
That is, we sum the number of times the letter ifollows the letter j. Intuitively,
these features can capture rules like “It is likely to see the pair ‘qu’ in a word”
or “It is unlikely to see the pair ‘rz’ in a word.” Of course, some of these features
will not be very useful, so the goal of the learning process is to assign weights to
features by learning the vector w, so that the weighted score will give us a good
prediction via

rr∑
It is left to show how to solve the optimization problem in the deﬁnition
ofhw(x) eﬃciently, as well as how to solve the optimization problem in the
deﬁnition of ˆ yin the SGD algorithm. We can do this by applying a dynamic
programming procedure. We describe the procedure for solving the maximization
in the deﬁnition of hwand leave as an exercise the maximization problem in the
deﬁnition of ˆ yin the SGD algorithm.
To derive the dynamic programming procedure, let us ﬁrst observe that we
can write
Ψ(x,y) =r∑

can write
Ψ(x,y) =r∑
t=1φ(x,yt,yt−1),
for an appropriate φ:X× [q]×[q]∪{0}→Rd, and for simplicity we assume
thaty0is always equal to 0. Indeed, each feature function Ψ i,j,1can be written
in terms of
φi,j,1(x,yt,yt−1) =xi,t1[yt=j],

238 Multiclass, Ranking, and Complex Prediction Problems
while the feature function Ψ i,j,2can be written in terms of
φi,j,2(x,yt,yt−1) =1[yt=i]1[yt−1=j].
Therefore, the prediction can be written as
y∈Yr∑
t=1⟨w,φ(x,yt,yt−1)⟩. (17.4)
In the following we derive a dynamic programming procedure that solves every
problem of the form given in Equation (17.4). The procedure will maintain a
matrixM∈Rq,rsuch that
Ms,τ= max
(y1,...,yτ):yτ=sτ∑
t=1⟨w,φ(x,yt,yt−1)⟩.
Clearly, the maximum of ⟨w,Ψ(x,y)⟩equals max sMs,r. Furthermore, we can
calculateMin a recursive manner:
Ms,τ= max
s′(Ms′,τ−1+⟨w,φ(x,s,s′)⟩). (17.5)
This yields the following procedure:
Dynamic Programming for Calculating hw(x)as Given
in Equation (17.4)
input: a matrix x∈Rn,rand a vector w
initialize:
foreachs∈[q]
Ms,1=⟨w,φ(x,s,−1)⟩
forτ= 2,...,r
setMs,τas in Equation (17.5)
setIs,τto be thes′that maximizes Equation (17.5)
setyt= argmaxsMs,r
forτ=r, r−1,..., 2
setyτ−1=Iyτ,τ
output: y = (y1,...,yr)
17.4 Ranking

Ms,τ= max
Ranking is the problem of ordering a set of instances according to their “rele-
vance.” A typical application is ordering results of a search engine according to
their relevance to the query. Another example is a system that monitors elec-
tronic transactions and should alert for possible fraudulent transactions. Such a
system should order transactions according to how suspicious they are.
Formally, letX∗=⋃∞
n=1Xnbe the set of all sequences of instances from

17.4 Ranking 239
Xof arbitrary length. A ranking hypothesis, h, is a function that receives a
sequence of instances ¯x= (x1,...,xr)∈X∗, and returns a permutation of [ r].
It is more convenient to let the output of hbe a vector y∈Rr, where by
sorting the elements of ywe obtain the permutation over [ r]. We denote by
π(y) the permutation over [ r] induced by y. For example, for r= 5, the vector
y= (2,1,6,−1,0.5) induces the permutation π(y) = (4,3,5,1,2). That is,
if we sort yin an ascending order, then we obtain the vector ( −1,0.5,1,2,6).
Now,π(y)iis the position of yiin the sorted vector ( −1,0.5,1,2,6). This
notation reﬂects that the top-ranked instances are those that achieve the highest
values inπ(y).
In the notation of our PAC learning model, the examples domain is Z=⋃∞
r=1(Xr×Rr), and the hypothesis class, H, is some set of ranking hypotheses.
We next turn to describe loss functions for ranking. There are many possible ways

to deﬁne such loss functions, and here we list a few examples. In all the examples
we deﬁneℓ(h,(¯x,y)) = ∆(h(¯x),y), for some function ∆ :⋃∞
r=1(Rr×Rr)→R+.
•0–1 Ranking loss: ∆(y′,y) is zero if yandy′induce exactly the same
ranking and ∆( y′,y) = 1 otherwise. That is, ∆( y′,y) =1[π(y′)̸=π(y)]. Such
a loss function is almost never used in practice as it does not distinguish
between the case in which π(y′) is almost equal to π(y) and the case in
whichπ(y′) is completely diﬀerent from π(y).

•Kendall-Tau Loss: We count the number of pairs ( i,j) that are in diﬀerent
order in the two permutations. This can be written as
∆(y′,y) =2
r(r−1)r−1∑
i=1r∑
j=i+11[sign(y′
i−y′
j)̸=sign(yi−yj)].
This loss function is more useful than the 0–1 loss as it reﬂects the level of
similarity between the two rankings.
•Normalized Discounted Cumulative Gain (NDCG): This measure em-
phasizes the correctness at the top of the list by using a monotonically
nondecreasing discount function D:N→R+. We ﬁrst deﬁne a discounted

∆(y′,y) =2
i=1r∑
i−y′
cumulative gain measure:
G(y′,y) =r∑
i=1D(π(y′)i)yi.
In words, if we interpret yias a score of the “true relevance” of item i, then
we take a weighted sum of the relevance of the elements, while the weight
ofyiis determined on the basis of the position of iinπ(y′). Assuming that
all elements of yare nonnegative, it is easy to verify that 0 ≤G(y′,y)≤
G(y,y). We can therefore deﬁne a normalized discounted cumulative gain
by the ratio G(y′,y)/G(y,y), and the corresponding loss function would
be
∆(y′,y) = 1−G(y′,y)
G(y,y)=1
G(y,y)r∑
i=1(D(π(y)i)−D(π(y′)i))yi.

240 Multiclass, Ranking, and Complex Prediction Problems
We can easily see that ∆( y′,y)∈[0,1] and that ∆( y′,y) = 0 whenever
π(y′) =π(y).
A typical way to deﬁne the discount function is by
D(i) ={
1
log2(r−i+2)ifi∈{r−k+ 1,...,r}
wherek<r is a parameter. This means that we care more about elements
that are ranked higher, and we completely ignore elements that are not at
the top-kranked elements. The NDCG measure is often used to evaluate
the performance of search engines since in such applications it makes sense
completely to ignore elements that are not at the top of the ranking.
Once we have a hypothesis class and a ranking loss function, we can learn a
ranking function using the ERM rule. However, from the computational point of
view, the resulting optimization problem might be hard to solve. We next discuss
how to learn linear predictors for ranking.
17.4.1 Linear Predictors for Ranking
A natural way to deﬁne a ranking function is by projecting the instances onto

some vector wand then outputting the resulting scalars as our representation
of the ranking function. That is, assuming that X ⊂Rd, for every w∈Rdwe
deﬁne a ranking function
hw((x1,...,xr)) = (⟨w,x1⟩,...,⟨w,xr⟩). (17.6)
As we discussed in Chapter 16, we can also apply a feature mapping that maps
instances into some feature space and then takes the inner products with win the
feature space. For simplicity, we focus on the simpler form as in Equation (17.6).
Given some W⊂Rd, we can now deﬁne the hypothesis class HW={hw:

w∈W}. Once we have deﬁned this hypothesis class, and have chosen a ranking
loss function, we can apply the ERM rule as follows: Given a training set, S=
(¯x1,y1),..., (¯xm,ym), where each ( ¯xi,yi) is in (X×R)ri, for someri∈N, we
should search w∈Wthat minimizes the empirical loss,∑m
i=1∆(hw(¯xi),yi).
As in the case of binary classiﬁcation, for many loss functions this problem is
computationally hard, and we therefore turn to describe convex surrogate loss
functions. We describe the surrogates for the Kendall tau loss and for the NDCG
loss.

loss.
A Hinge Loss for the Kendall Tau Loss Function:
We can think of the Kendall tau loss as an average of 0 −1 losses for each pair.
In particular, for every ( i,j) we can rewrite
1[sign(y′
i−y′
j)̸=sign(yi−yj)]=1[sign(yi−yj)(y′
i−y′
j)≤0].

17.4 Ranking 241
In our case, y′
i−y′
j=⟨w,xi−xj⟩. It follows that we can use the hinge loss upper
bound as follows:
1[sign(yi−yj)(y′
i−y′
j)≤0]≤max{0,1−sign (yi−yj)⟨w,xi−xj⟩}.
Taking the average over the pairs we obtain the following surrogate convex loss
for the Kendall tau loss function:
∆(hw(¯x),y)≤2
i=1r∑
j=i+1max{0,1−sign(yi−yj)⟨w,xi−xj⟩}.
The right-hand side is convex with respect to wand upper bounds the Kendall
tau loss. It is also a ρ-Lipschitz function with parameter ρ≤maxi,j∥xi−xj∥.
A Hinge Loss for the NDCG Loss Function:
The NDCG loss function depends on the predicted ranking vector y′∈Rrvia
the permutation it induces. To derive a surrogate loss function we ﬁrst make
the following observation. Let Vbe the set of all permutations of [ r] encoded as
vectors; namely, each v∈Vis a vector in [ r]rsuch that for all i̸=jwe have
vi̸=vj. Then (see Exercise 4),
π(y′) = argmax
v∈Vr∑
i=1viy′
i. (17.7)
Let us denote Ψ( ¯x,v) =∑r
i=1vixi; it follows that
π(hw(¯x)) = argmax
v∈Vr∑
i=1vi⟨w,xi⟩

v∈Vr∑
i=1viy′
i. (17.7)
v∈Vr∑
= argmax
v∈V⟨
w,r∑
i=1vixi⟩
= argmax
v∈V⟨w,Ψ(¯x,v)⟩.
On the basis of this observation, we can use the generalized hinge loss for cost-
sensitive multiclass classiﬁcation as a surrogate loss function for the NDCG loss
∆(hw(¯x),y)≤∆(hw(¯x),y) +⟨w,Ψ(¯x,π(hw(¯x)))⟩−⟨w,Ψ(¯x,π(y))⟩
≤max
v∈V[∆(v,y) +⟨w,Ψ(¯x,v)⟩−⟨w,Ψ(¯x,π(y))⟩]
= max
v∈V[
∆(v,y) +r∑
i=1(vi−π(y)i)⟨w,xi⟩]
. (17.8)
The right-hand side is a convex function with respect to w.

v∈Vr∑
= argmax
v∈V⟨
w,r∑
i=1vixi⟩
= argmax
≤max
= max
v∈V[
∆(v,y) +r∑
. (17.8)
We can now solve the learning problem using SGD as described in Section 17.2.5.
The main computational bottleneck is calculating a subgradient of the loss func-
tion, which is equivalent to ﬁnding vthat achieves the maximum in Equa-
tion (17.8) (see Claim 14.6). Using the deﬁnition of the NDCG loss, this is

242 Multiclass, Ranking, and Complex Prediction Problems
equivalent to solving the problem
argmin
v∈Vr∑
i=1(αivi+βiD(vi)),
whereαi=−⟨w,xi⟩andβi=yi/G(y,y). We can think of this problem a little
bit diﬀerently by deﬁning a matrix A∈Rr,rwhere
Ai,j=jαi+D(j)βi.
Now, let us think about each jas a “worker,” each ias a “task,” and Ai,jas
the cost of assigning task ito workerj. With this view, the problem of ﬁnding
vbecomes the problem of ﬁnding an assignment of the tasks to workers of
minimal cost. This problem is called “the assignment problem” and can be solved
eﬃciently. One particular algorithm is the “Hungarian method” (Kuhn 1955).
Another way to solve the assignment problem is using linear programming. To
do so, let us ﬁrst write the assignment problem as
argmin
B∈Rr,r
+r∑
i,j=1Ai,jBi,j (17.9)
s.t.∀i∈[r],r∑
j=1Bi,j= 1
∀j∈[r],r∑
i=1Bi,j= 1
∀i,j, Bi,j∈{0,1}
A matrixBthat satisﬁes the constraints in the preceding optimization problem
is called a permutation matrix. This is because the constraints guarantee that

argmin
B∈Rr,r
+r∑
j=1Bi,j= 1
∀j∈[r],r∑
i=1Bi,j= 1
there is at most a single entry of each row that equals 1 and a single entry of each
column that equals 1. Therefore, the matrix Bcorresponds to the permutation
v∈Vdeﬁned byvi=jfor the single index jthat satisﬁes Bi,j= 1.
The preceding optimization is still not a linear program because of the com-
binatorial constraint Bi,j∈{0,1}. However, as it turns out, this constraint is
redundant – if we solve the optimization problem while simply omitting the
combinatorial constraint, then we are still guaranteed that there is an optimal

solution that will satisfy this constraint. This is formalized later.
Denote⟨A,B⟩=∑
i,jAi,jBi,j. Then, Equation (17.9) is the problem of mini-
mizing⟨A,B⟩such thatBis a permutation matrix.
A matrixB∈Rr,ris called doubly stochastic if all elements of Bare non-
negative, the sum of each row of Bis 1, and the sum of each column of Bis 1.
Therefore, solving Equation (17.9) without the constraints Bi,j∈{0,1}is the
problem
argmin
B∈Rr,r⟨A,B⟩s.t.Bis a doubly stochastic matrix . (17.10)

The following claim states that every doubly stochastic matrix is a convex
combination of permutation matrices.
claim 17.3 ((Birkhoﬀ 1946, Von Neumann 1953)) The set of doubly stochastic
matrices in Rr,ris the convex hull of the set of permutation matrices in Rr,r.
On the basis of the claim, we easily obtain the following:
lemma 17.4 There exists an optimal solution of Equation (17.10) that is also
an optimal solution of Equation (17.9).
Proof LetBbe a solution of Equation (17.10). Then, by Claim 17.3, we can
writeB=∑
iγiCi, where each Ciis a permutation matrix, each γi>0, and∑
iγi= 1. Since all the Ciare also doubly stochastic, we clearly have that
⟨A, B⟩≤⟨A, Ci⟩for everyi. We claim that there is some ifor which⟨A, B⟩=
⟨A, Ci⟩. This must be true since otherwise, if for every i⟨A, B⟩<⟨A, Ci⟩, we
would have that
⟨A,B⟩=⟨
A,∑
iγiCi⟩
=∑
iγi⟨A,Ci⟩>∑
iγi⟨A,B⟩=⟨A,B⟩,
which cannot hold. We have thus shown that some permutation matrix, Ci,

writeB=∑
⟨A,B⟩=⟨
A,∑
iγiCi⟩
=∑
satisﬁes⟨A,B⟩=⟨A,Ci⟩. But, since for every other permutation matrix Cwe
have⟨A,B⟩≤⟨A,C⟩we conclude that Ciis an optimal solution of both Equa-
tion (17.9) and Equation (17.10).
17.5 Bipartite Ranking and Multivariate Performance Measures
In the previous section we described the problem of ranking. We used a vector
y∈Rrfor representing an order over the elements x1,...,xr. If all elements in y
are diﬀerent from each other, then yspeciﬁes a full order over [ r]. However, if two

elements of yattain the same value, yi=yjfori̸=j, then ycan only specify a
partial order over [ r]. In such a case, we say that xiandxjare of equal relevance
according to y. In the extreme case, y∈{± 1}r, which means that each xiis
either relevant or nonrelevant. This setting is often called “bipartite ranking.” For
example, in the fraud detection application mentioned in the previous section,
each transaction is labeled as either fraudulent ( yi= 1) or benign ( yi=−1).

Seemingly, we can solve the bipartite ranking problem by learning a binary
classiﬁer, applying it on each instance, and putting the positive ones at the top
of the ranked list. However, this may lead to poor results as the goal of a binary
learner is usually to minimize the zero-one loss (or some surrogate of it), while the
goal of a ranker might be signiﬁcantly diﬀerent. To illustrate this, consider again
the problem of fraud detection. Usually, most of the transactions are benign (say

99.9%). Therefore, a binary classiﬁer that predicts “benign” on all transactions
will have a zero-one error of 0 .1%. While this is a very small number, the resulting
predictor is meaningless for the fraud detection application. The crux of the

244 Multiclass, Ranking, and Complex Prediction Problems
problem stems from the inadequacy of the zero-one loss for what we are really
interested in. A more adequate performance measure should take into account
the predictions over the entire set of instances. For example, in the previous
section we have deﬁned the NDCG loss, which emphasizes the correctness of the
top-ranked items. In this section we describe additional loss functions that are
speciﬁcally adequate for bipartite ranking problems.
As in the previous section, we are given a sequence of instances, ¯x= (x1,...,xr),
and we predict a ranking vector y′∈Rr. The feedback vector is y∈{± 1}r. We
deﬁne a loss that depends on y′andyand depends on a threshold θ∈R. This
threshold transforms the vector y′∈Rrinto the vector (sign( y′
i−θ),..., sign(y′
r−
θ))∈{± 1}r. Usually, the value of θis set to be 0. However, as we will see, we
sometimes set θwhile taking into account additional constraints on the problem.

r−
The loss functions we deﬁne in the following depend on the following 4 num-
bers:
True positives: a=|{i:yi= +1∧sign(y′
i−θ) = +1}|
False positives: b=|{i:yi=−1∧sign(y′
i−θ) = +1}|
False negatives: c=|{i:yi= +1∧sign(y′
i−θ) =−1}|
True negatives: d=|{i:yi=−1∧sign(y′
i−θ) =−1}|(17.11)
Therecall (a.k.a. sensitivity ) of a prediction vector is the fraction of true
positives y′“catches,” namely,a
a+c. The precision is the fraction of correct
predictions among the positive labels we predict, namely,a
a+b. The speciﬁcity

bers:
i−θ) = +1}|
i−θ) = +1}|
i−θ) =−1}|
is the fraction of true negatives that our predictor “catches,” namely,d
d+b.
Note that as we decrease θthe recall increases (attaining the value 1 when
θ=−∞). On the other hand, the precision and the speciﬁcity usually decrease
as we decrease θ. Therefore, there is a tradeoﬀ between precision and recall, and
we can control it by changing θ. The loss functions deﬁned in the following use
various techniques for combining both the precision and recall.
•Averaging sensitivity and speciﬁcity : This measure is the average of the
sensitivity and speciﬁcity, namely,1
2(
a
a+c+d

d+b.
2(
a
a+c+d
d+b)
. This is also the accuracy
on positive examples averaged with the accuracy on negative examples.
Here, we set θ= 0 and the corresponding loss function is ∆( y′,y) =
1−1
2(
a
a+c+d
d+b)
.
•F1-score : TheF1score is the harmonic mean of the precision and recall:
2
1
Precision+1
Recall. Its maximal value (of 1) is obtained when both precision
and recall are 1, and its minimal value (of 0) is obtained whenever one of
them is 0 (even if the other one is 1). The F1score can be written using

2(
a
a+c+d
d+b)
1−1
2(
a
a+c+d
d+b)
.
2
1
the numbers a, b, c as follows;F1=2a
2a+b+c. Again, we set θ= 0, and the
loss function becomes ∆( y′,y) = 1−F1.
•Fβ-score : It is like F1score, but we attach β2times more importance to
recall than to precision, that is,1+β2
1
Precision+β21
Recall. It can also be written as

17.5 Bipartite Ranking and Multivariate Performance Measures 245
Fβ=(1+β2)a
(1+β2)a+b+β2c. Again, we set θ= 0, and the loss function becomes
∆(y′,y) = 1−Fβ.
•Recall atk: We measure the recall while the prediction must contain at most
kpositive labels. That is, we should set θso thata+b≤k. This is conve-
nient, for example, in the application of a fraud detection system, where a
bank employee can only handle a small number of suspicious transactions.
•Precision at k: We measure the precision while the prediction must contain
at leastkpositive labels. That is, we should set θso thata+b≥k.
The measures deﬁned previously are often referred to as multivariate perfor-
mance measures . Note that these measures are highly diﬀerent from the average
zero-one loss, which in the preceding notation equalsb+d
a+b+c+d. In the aforemen-
tioned example of fraud detection, when 99 .9% of the examples are negatively
labeled, the zero-one loss of predicting that all the examples are negatives is

0.1%. In contrast, the recall of such prediction is 0 and hence the F1score is also
0, which means that the corresponding loss will be 1.
17.5.1 Linear Predictors for Bipartite Ranking
We next describe how to train linear predictors for bipartite ranking. As in the
previous section, a linear predictor for ranking is deﬁned to be
hw(¯x) = (⟨w,x1⟩,...,⟨w,xr⟩).
The corresponding loss function is one of the multivariate performance measures
described before. The loss function depends on y′=hw(¯x) via the binary vector
it induces, which we denote by

b(y′) = (sign(y′
1−θ),..., sign(y′
r−θ))∈ {± 1}r. (17.12)
As in the previous section, to facilitate an eﬃcient algorithm we derive a convex
surrogate loss function on ∆. The derivation is similar to the derivation of the
generalized hinge loss for the NDCG ranking loss, as described in the previous
section.
Our ﬁrst observation is that for all the values of θdeﬁned before, there is some
V⊆{± 1}rsuch that b(y′) can be rewritten as
b(y′) = argmax
v∈Vr∑
i=1viy′
i. (17.13)
This is clearly true for the case θ= 0 if we choose V={±1}r. The two measures

section.
v∈Vr∑
i=1viy′
i. (17.13)
for whichθis not taken to be 0 are precision at kand recall at k. For precision
atkwe can take Vto be the set V≥k, containing all vectors in {±1}rwhose
number of ones is at least k. For recall at k, we can take Vto beV≤k, which is
deﬁned analogously. See Exercise 5.

246 Multiclass, Ranking, and Complex Prediction Problems
Once we have deﬁned bas in Equation (17.13), we can easily derive a convex
surrogate loss as follows. Assuming that y∈V, we have that
∆(hw(¯x),y) = ∆( b(hw(¯x)),y)
≤∆(b(hw(¯x)),y) +r∑
i=1(bi(hw(¯x))−yi)⟨w,xi⟩
≤max
v∈V[
∆(v,y) +r∑
i=1(vi−yi)⟨w,xi⟩]
. (17.14)
tion (17.14) (see Claim 14.6).
In the following we describe how to ﬁnd this maximizer eﬃciently for any
performance measure that can be written as a function of the numbers a, b, c, d
given in Equation (17.11), and for which the set Vcontains all elements in {±1}r
for which the values of a,bsatisfy some constraints. For example, for “recall at
k” the setVis all vectors for which a+b≤k.

The idea is as follows. For any a,b∈[r], let
¯Ya,b={v:|{i:vi= 1∧yi= 1}|=a∧|{i:vi= 1∧yi=−1}|=b}.
Any vector v∈Vfalls into ¯Ya,bfor somea,b∈[r]. Furthermore, if ¯Ya,b∩V
is not empty for some a,b∈[r] then ¯Ya,b∩V=¯Ya,b. Therefore, we can search
within each ¯Ya,bthat has a nonempty intersection with Vseparately, and then
take the optimal value. The key observation is that once we are searching only
within ¯Ya,b, the value of ∆ is ﬁxed so we only need to maximize the expression
max
v∈¯Ya,br∑
i=1vi⟨w,xi⟩.

max
v∈¯Ya,br∑
Suppose the examples are sorted so that ⟨w,x1⟩≥···≥⟨ w,xr⟩. Then, it is
easy to verify that we would like to set vito be positive for the smallest indices
i. Doing this, with the constraint on a,b, amounts to setting vi= 1 for the a
top ranked positive examples and for the btop-ranked negative examples. This
yields the following procedure.

Solving Equation (17.14)
input:
(x1,...,xr),(y1,...,yr),w,V,∆
∆ is a function of a, b, c, d
Vcontains all vectors for which f(a,b) = 1 for some function f
initialize :
P=|{i:yi= 1}|,N=|{i:yi=−1}|
µ= (⟨w,x1⟩,...,⟨w,xr⟩),α⋆=−∞
sort examples so that µ1≥µ2≥···≥µr
leti1,...,iPbe the (sorted) indices of the positive examples
letj1,...,jNbe the (sorted) indices of the negative examples
fora= 0,1,...,P
c=P−a
forb= 0,1,...,N such thatf(a,b) = 1
d=N−b
calculate ∆ using a, b, c, d
setv1,...,vrs.t.vi1=···=via=vj1=···=vjb= 1
and the rest of the elements of vequal−1
setα= ∆ +∑r
i=1viµi
ifα≥α⋆
α⋆=α,v⋆=v
output v⋆
17.6 Summary
Many real world supervised learning problems can be cast as learning a multiclass
predictor. We started the chapter by introducing reductions of multiclass learning
to binary learning. We then described and analyzed the family of linear predictors
for multiclass learning. We have shown how this family can be used even if the

d=N−b
setα= ∆ +∑r
i=1viµi
ifα≥α⋆
α⋆=α,v⋆=v
output v⋆
number of classes is extremely large, as long as we have an adequate structure
on the problem. Finally, we have described ranking problems. In Chapter 29 we
study the sample complexity of multiclass learning in more detail.
17.7 Bibliographic Remarks
The One-versus-All and All-Pairs approach reductions have been uniﬁed un-
der the framework of Error Correction Output Codes (ECOC) (Dietterich &
Bakiri 1995, Allwein, Schapire & Singer 2000). There are also other types of re-

ductions such as tree-based classiﬁers (see, for example, Beygelzimer, Langford
& Ravikumar (2007)). The limitations of reduction techniques have been studied

248 Multiclass, Ranking, and Complex Prediction Problems
in (Daniely et al. 2011, Daniely, Sabato & Shwartz 2012). See also Chapter 29,
in which we analyze the sample complexity of multiclass learning.
Direct approaches to multiclass learning with linear predictors have been stud-
ied in (Vapnik 1998, Weston & Watkins 1999, Crammer & Singer 2001). In par-
ticular, the multivector construction is due to Crammer & Singer (2001).
Collins (2000) has shown how to apply the Perceptron algorithm for structured
output problems. See also Collins (2002). A related approach is discriminative
learning of conditional random ﬁelds; see Laﬀerty, McCallum & Pereira (2001).
Structured output SVM has been studied in (Weston, Chapelle, Vapnik, Elisseeﬀ
& Sch¨ olkopf 2002, Taskar, Guestrin & Koller 2003, Tsochantaridis, Hofmann,
Joachims & Altun 2004).
The dynamic procedure we have presented for calculating the prediction hw(x)
in the structured output section is similar to the forward-backward variables

calculated by the Viterbi procedure in HMMs (see, for instance, (Rabiner &
Juang 1986)). More generally, solving the maximization problem in structured
output is closely related to the problem of inference in graphical models (see, for
example, Koller & Friedman (2009)).
Chapelle, Le & Smola (2007) proposed to learn a ranking function with respect
to the NDCG loss using ideas from structured output learning. They also ob-
served that the maximization problem in the deﬁnition of the generalized hinge

loss is equivalent to the assignment problem.
Agarwal & Roth (2005) analyzed the sample complexity of bipartite ranking.
Joachims (2005) studied the applicability of structured output SVM to bipartite
ranking with multivariate performance measures.
17.8 Exercises
1. Consider a set Sof examples in Rn×[k] for which there exist vectors µ1,...,µk
such that every example ( x,y)∈Sfalls within a ball centered at µywhose
radius isr≥1. Assume also that for every i̸=j,∥µi−µj∥≥ 4r. Con-

sider concatenating each instance by the constant 1 and then applying the
multivector construction, namely,
Ψ(x,y) = [ 0,..., 0
∈R(y−1)(n+1), x1,...,xn,1
∈Rn+1,0,..., 0
∈R(k−y)(n+1)].
Show that there exists a vector w∈Rk(n+1)such thatℓ(w,(x,y)) = 0 for
every ( x,y)∈S.
Hint: Observe that for every example ( x,y)∈Swe can write x=µy+vfor
some∥v∥≤r. Now, take w= [w1,...,wk], where wi= [µi,−∥µi∥2/2].
2.Multiclass Perceptron: Consider the following algorithm:

17.8 Exercises 249
Multiclass Batch Perceptron
Input:
A training set ( x1,y1),..., (xm,ym)
A class-sensitive feature mapping Ψ : X×Y→ Rd
Initialize: w(1)= (0,..., 0)∈Rd
Fort= 1,2,...
If(∃iandy̸=yis.t.⟨w(t),Ψ(xi,yi)⟩≤⟨w(t),Ψ(xi,y)⟩) then
w(t+1)=w(t)+ Ψ(xi,yi)−Ψ(xi,y)
else
theorem 17.5 Assume that there exists w⋆such that for all iand for all
y̸=yiit holds that⟨w⋆,Ψ(xi,yi)⟩≥⟨w⋆,Ψ(xi,y)⟩+1. LetR= maxi,y∥Ψ(xi,yi)−
Ψ(xi,y)∥. Then, the multiclass Perceptron algorithm stops after at most (R∥w⋆∥)2
iterations, and when it stops it holds that ∀i∈[m], yi= argmaxy⟨w(t),Ψ(xi,y)⟩.
3. Generalize the dynamic programming procedure given in Section 17.3 for solv-
ing the maximization problem given in the deﬁnition of ˆhin the SGD proce-
dure for multiclass prediction. You can assume that ∆( y′,y) =∑r
t=1δ(y′
t,yt)
for some arbitrary function δ.
4. Prove that Equation (17.7) holds.
5. Show that the two deﬁnitions of πas deﬁned in Equation (17.12) and Equa-

t=1δ(y′
t,yt)
tion (17.13) are indeed equivalent for all the multivariate performance mea-
sures.

18 Decision Trees
A decision tree is a predictor, h:X→Y , that predicts the label associated with
an instance xby traveling from a root node of a tree to a leaf. For simplicity
we focus on the binary classiﬁcation setting, namely, Y={0,1}, but decision
trees can be applied for other prediction problems as well. At each node on the
root-to-leaf path, the successor child is chosen on the basis of a splitting of the
input space. Usually, the splitting is based on one of the features of xor on a
predeﬁned set of splitting rules. A leaf contains a speciﬁc label. An example of
a decision tree for the papayas example (described in Chapter 2) is given in the
Color?
not-tastyother
Softness?
tastygives slightly to palm pressurepale green to pale yellow
To check if a given papaya is tasty or not, the decision tree ﬁrst examines
the color of the Papaya. If this color is not in the range pale green to pale
yellow, then the tree immediately predicts that the papaya is not tasty without

Color?
Softness?
additional tests. Otherwise, the tree turns to examine the softness of the papaya.
If the softness level of the papaya is such that it gives slightly to palm pressure,
the decision tree predicts that the papaya is tasty. Otherwise, the prediction is
“not-tasty.” The preceding example underscores one of the main advantages of
decision trees – the resulting classiﬁer is very simple to understand and interpret.

18.1 Sample Complexity
A popular splitting rule at internal nodes of the tree is based on thresholding the
value of a single feature. That is, we move to the right or left child of the node on
the basis of 1[xi<θ], wherei∈[d] is the index of the relevant feature and θ∈R
is the threshold. In such cases, we can think of a decision tree as a splitting of
the instance space, X=Rd, into cells, where each leaf of the tree corresponds
to one cell. It follows that a tree with kleaves can shatter a set of kinstances.
Hence, if we allow decision trees of arbitrary size, we obtain a hypothesis class
of inﬁnite VC dimension. Such an approach can easily lead to overﬁtting.
To avoid overﬁtting, we can rely on the minimum description length (MDL)
principle described in Chapter 7, and aim at learning a decision tree that on one
hand ﬁts the data well while on the other hand is not too large.
For simplicity, we will assume that X={0,1}d. In other words, each instance

is a vector of dbits. In that case, thresholding the value of a single feature
corresponds to a splitting rule of the form 1[xi=1]for somei= [d]. For instance,
we can model the “papaya decision tree” earlier by assuming that a papaya is
parameterized by a two-dimensional bit vector x∈{0,1}2, where the bit x1
represents whether the color is pale green to pale yellow or not, and the bit x2
represents whether the softness is gives slightly to palm pressure or not. With
this representation, the node Color? can be replaced with 1[x1=1], and the node

Softness? can be replaced with 1[x2=1]. While this is a big simpliﬁcation, the
algorithms and analysis we provide in the following can be extended to more
general cases.
With the aforementioned simplifying assumption, the hypothesis class becomes
ﬁnite, but is still very large. In particular, any classiﬁer from {0,1}dto{0,1}
can be represented by a decision tree with 2dleaves and depth of d+ 1 (see
Exercise 1). Therefore, the VC dimension of the class is 2d, which means that

the number of examples we need to PAC learn the hypothesis class grows with
2d. Unlessdis very small, this is a huge number of examples.
To overcome this obstacle, we rely on the MDL scheme described in Chapter 7.
The underlying prior knowledge is that we should prefer smaller trees over larger
trees. To formalize this intuition, we ﬁrst need to deﬁne a description language
for decision trees, which is preﬁx free and requires fewer bits for smaller decision
trees. Here is one possible way: A tree with nnodes will be described in n+ 1

blocks, each of size log2(d+ 3) bits. The ﬁrst nblocks encode the nodes of the
tree, in a depth-ﬁrst order (preorder), and the last block marks the end of the
code. Each block indicates whether the current node is:
•An internal node of the form 1[xi=1]for somei∈[d]
•A leaf whose value is 1
•A leaf whose value is 0
•End of the code

252 Decision Trees
Overall, there are d+ 3 options, hence we need log2(d+ 3) bits to describe each
block.
Assuming each internal node has two children,1it is not hard to show that
this is a preﬁx-free encoding of the tree, and that the description length of a tree
withnnodes is (n+ 1) log2(d+ 3).
By Theorem 7.7 we have that with probability of at least 1 −δover a sample
of sizem, for everynand every decision tree h∈H withnnodes it holds that
(n+ 1) log2(d+ 3) + log(2 /δ)
2m. (18.1)
This bound performs a tradeoﬀ: on the one hand, we expect larger, more complex
decision trees to have a smaller training risk, LS(h), but the respective value of
nwill be larger. On the other hand, smaller decision trees will have a smaller
value ofn, butLS(h) might be larger. Our hope (or prior knowledge) is that we
can ﬁnd a decision tree with both low empirical risk, LS(h), and a number of
nodesnnot too high. Our bound indicates that such a tree will have low true
risk,LD(h).
18.2 Decision Tree Algorithms

The bound on LD(h) given in Equation (18.1) suggests a learning rule for decision
trees – search for a tree that minimizes the right-hand side of Equation (18.1).
Unfortunately, it turns out that solving this problem is computationally hard.2
Consequently, practical decision tree learning algorithms are based on heuristics
such as a greedy approach, where the tree is constructed gradually, and locally
optimal decisions are made at the construction of each node. Such algorithms
cannot guarantee to return the globally optimal decision tree but tend to work

reasonably well in practice.
A general framework for growing a decision tree is as follows. We start with
a tree with a single leaf (the root) and assign this leaf a label according to a
majority vote among all labels over the training set. We now perform a series of
iterations. On each iteration, we examine the eﬀect of splitting a single leaf. We
deﬁne some “gain” measure that quantiﬁes the improvement due to this split.
Then, among all possible splits, we either choose the one that maximizes the

gain and perform it, or choose not to split the leaf at all.
In the following we provide a possible implementation. It is based on a popular
decision tree algorithm known as “ID3” (short for “Iterative Dichotomizer 3”).
We describe the algorithm for the case of binary features, namely, X={0,1}d,
1We may assume this without loss of generality, because if a decision node has only one
child, we can replace the node by its child without aﬀecting the predictions of the decision
tree.

tree.
2More precisely, if NP ̸=P then no algorithm can solve Equation (18.1) in time polynomial
inn,d, andm.

18.2 Decision Tree Algorithms 253
and therefore all splitting rules are of the form 1[xi=1]for some feature i∈[d].
We discuss the case of real valued features in Section 18.2.3.
The algorithm works by recursive calls, with the initial call being ID3(S,[d]),
and returns a decision tree. In the pseudocode that follows, we use a call to a
procedure Gain (S,i), which receives a training set Sand an index iand evaluates
the gain of a split of the tree according to the ith feature. We describe several
gain measures in Section 18.2.1.
ID3(S,A)
Input: training set S, feature subset A⊆[d]
ifall examples in Sare labeled by 1, return a leaf 1
ifall examples in Sare labeled by 0, return a leaf 0
ifA=∅, return a leaf whose value = majority of labels in S
else :
Letj= argmaxi∈AGain (S,i)
ifall examples in Shave the same label
Return a leaf whose value = majority of labels in S
else
LetT1be the tree returned by ID3({(x,y)∈S:xj= 1},A\{j}).
LetT2be the tree returned by ID3({(x,y)∈S:xj= 0},A\{j}).
Return the tree:
xj= 1?
T2 T1

ID3(S,A)
else :
else
xj= 1?
T2 T1
18.2.1 Implementations of the Gain Measure
Diﬀerent algorithms use diﬀerent implementations of Gain (S,i). Here we present
three. We use the notation PS[F] to denote the probability that an event holds
with respect to the uniform distribution over S.
Train Error: The simplest deﬁnition of gain is the decrease in training error.
Formally, let C(a) = min{a,1−a}. Note that the training error before splitting on
featureiisC(PS[y= 1]), since we took a majority vote among labels. Similarly,
the error after splitting on feature iis
P

P
S[xi= 1]C(P
S[y= 1|xi= 1]) + P
S[xi= 0]C(P
S[y= 1|xi= 0]).
Therefore, we can deﬁne Gain to be the diﬀerence between the two, namely,
Gain (S,i) :=C(P
S[y= 1])
−(
P
S[y= 1|xi= 0]))
.

254 Decision Trees
Information Gain: Another popular gain measure that is used in the ID3
and C4.5 algorithms of Quinlan (1993) is the information gain. The information
gain is the diﬀerence between the entropy of the label before and after the split,
and is achieved by replacing the function Cin the previous expression by the
entropy function,
C(a) =−alog(a)−(1−a) log(1−a).
Gini Index: Yet another deﬁnition of a gain, which is used by the CART
algorithm of Breiman, Friedman, Olshen & Stone (1984), is the Gini index,
C(a) = 2a(1−a).
Both the information gain and the Gini index are smooth and concave upper
bounds of the train error. These properties can be advantageous in some situa-
tions (see, for example, Kearns & Mansour (1996)).
18.2.2 Pruning
The ID3 algorithm described previously still suﬀers from a big problem: The
returned tree will usually be very large. Such trees may have low empirical risk,
but their true risk will tend to be high – both according to our theoretical

analysis, and in practice. One solution is to limit the number of iterations of ID3,
leading to a tree with a bounded number of nodes. Another common solution is
toprune the tree after it is built, hoping to reduce it to a much smaller tree,
but still with a similar empirical error. Theoretically, according to the bound in
Equation (18.1), if we can make nmuch smaller without increasing LS(h) by
much, we are likely to get a decision tree with a smaller true risk.
Usually, the pruning is performed by a bottom-up walk on the tree. Each node

might be replaced with one of its subtrees or with a leaf, based on some bound
or estimate of LD(h) (for example, the bound in Equation (18.1)). A pseudocode
of a common template is given in the following.
Generic Tree Pruning Procedure
input:
functionf(T,m) (bound/estimate for the generalization error
of a decision tree T, based on a sample of size m),
treeT.
foreach nodejin a bottom-up walk on T(from leaves to root):
ﬁndT′which minimizes f(T′,m), whereT′is any of the following:
the current tree after replacing node jwith a leaf 1.

input:
treeT.
the current tree after replacing node jwith a leaf 0.
the current tree after replacing node jwith its left subtree.
the current tree after replacing node jwith its right subtree.
the current tree.
letT:=T′.

18.2.3 Threshold-Based Splitting Rules for Real-Valued Features
In the previous section we have described an algorithm for growing a decision
tree assuming that the features are binary and the splitting rules are of the
form 1[xi=1]. We now extend this result to the case of real-valued features and
threshold-based splitting rules, namely, 1[xi<θ]. Such splitting rules yield decision
stumps, and we have studied them in Chapter 10.
The basic idea is to reduce the problem to the case of binary features as
follows. Let x1,...,xmbe the instances of the training set. For each real-valued
featurei, sort the instances so that x1,i≤···≤xm,i. Deﬁne a set of thresholds
θ0,i,...,θm+1,isuch thatθj,i∈(xj,i,xj+1,i) (where we use the convention x0,i=
−∞ andxm+1,i=∞). Finally, for each iandjwe deﬁne the binary feature
1[xi<θj,i]. Once we have constructed these binary features, we can run the ID3
procedure described in the previous section. It is easy to verify that for any

decision tree with threshold-based splitting rules over the original real-valued
features there exists a decision tree over the constructed binary features with
the same training error and the same number of nodes.
If the original number of real-valued features is dand the number of examples
ism, then the number of constructed binary features becomes dm. Calculating
theGain of each feature might therefore take O(dm2) operations. However, using
a more clever implementation, the runtime can be reduced to O(dmlog(m)). The

idea is similar to the implementation of ERM for decision stumps as described
in Section 10.1.1.
18.3 Random Forests
As mentioned before, the class of decision trees of arbitrary size has inﬁnite VC
dimension. We therefore restricted the size of the decision tree. Another way
to reduce the danger of overﬁtting is by constructing an ensemble of trees. In
particular, in the following we describe the method of random forests , introduced
by Breiman (2001).
A random forest is a classiﬁer consisting of a collection of decision trees, where

each tree is constructed by applying an algorithm Aon the training set Sand
an additional random vector, θ, whereθis sampled i.i.d. from some distribution.
The prediction of the random forest is obtained by a majority vote over the
predictions of the individual trees.
To specify a particular random forest, we need to deﬁne the algorithm Aand
the distribution over θ. There are many ways to do this and here we describe one
particular option. We generate θas follows. First, we take a random subsample

fromSwith replacements; namely, we sample a new training set S′of sizem′
using the uniform distribution over S. Second, we construct a sequence I1,I2,...,
where each Itis a subset of [ d] of sizek, which is generated by sampling uniformly
at random elements from [ d]. All these random variables form the vector θ. Then,

256 Decision Trees
the algorithm Agrows a decision tree (e.g., using the ID3 algorithm) based on
the sample S′, where at each splitting stage of the algorithm, the algorithm is
restricted to choosing a feature that maximizes Gain from the set It. Intuitively,
ifkis small, this restriction may prevent overﬁtting.
18.4 Summary
Decision trees are very intuitive predictors. Typically, if a human programmer
creates a predictor it will look like a decision tree. We have shown that the VC
dimension of decision trees with kleaves iskand proposed the MDL paradigm
for learning decision trees. The main problem with decision trees is that they
are computationally hard to learn; therefore we described several heuristic pro-
cedures for training them.
18.5 Bibliographic Remarks
Many algorithms for learning decision trees (such as ID3 and C4.5) have been
derived by Quinlan (1986). The CART algorithm is due to Breiman et al. (1984).
Random forests were introduced by Breiman (2001). For additional reading we

refer the reader to (Hastie, Tibshirani & Friedman 2001, Rokach 2007).
The proof of the hardness of training decision trees is given in Hyaﬁl & Rivest
(1976).
18.6 Exercises
1. 1. Show that any binary classiﬁer h:{0,1}d↦→{0,1}can be implemented
as a decision tree of height at most d+ 1, with internal nodes of the form
(xi= 0?) for some i∈{1,...,d}.
2. Conclude that the VC dimension of the class of decision trees over the
domain{0,1}dis 2d.
2.(Suboptimality of ID3)
Consider the following training set, where X={0,1}3andY={0,1}:
((1,1,1),1)
((1,0,0),1)
((1,1,0),0)

(1976).
((0,0,1),0)
Suppose we wish to use this training set in order to build a decision tree of
depth 2 (i.e., for each input we are allowed to ask two questions of the form
(xi= 0?) before deciding on the label).

18.6 Exercises 257
1. Suppose we run the ID3 algorithm up to depth 2 (namely, we pick the root
node and its children according to the algorithm, but instead of keeping
on with the recursion, we stop and pick leaves according to the majority
label in each subtree). Assume that the subroutine used to measure the
quality of each feature is based on the entropy function (so we measure the
information gain ), and that if two features get the same score, one of them
is picked arbitrarily. Show that the training error of the resulting decision
tree is at least 1 /4.
2. Find a decision tree of depth 2 that attains zero training error.

19 Nearest Neighbor
Nearest Neighbor algorithms are among the simplest of all machine learning
algorithms. The idea is to memorize the training set and then to predict the
label of any new instance on the basis of the labels of its closest neighbors in
the training set. The rationale behind such a method is based on the assumption
that the features that are used to describe the domain points are relevant to
their labelings in a way that makes close-by points likely to have the same label.
Furthermore, in some situations, even when the training set is immense, ﬁnding
a nearest neighbor can be done extremely fast (for example, when the training
set is the entire Web and distances are based on links).
Note that, in contrast with the algorithmic paradigms that we have discussed
so far, like ERM, SRM, MDL, or RLM, that are determined by some hypothesis
class,H, the Nearest Neighbor method ﬁgures out a label on any test point
without searching for a predictor within some predeﬁned class of functions.

In this chapter we describe Nearest Neighbor methods for classiﬁcation and
regression problems. We analyze their performance for the simple case of binary
classiﬁcation and discuss the eﬃciency of implementing these methods.
19.1kNearest Neighbors
Throughout the entire chapter we assume that our instance domain, X, is en-
dowed with a metric function ρ. That is,ρ:X×X→ Ris a function that returns
the distance between any two elements of X. For example, ifX=Rdthenρcan
be the Euclidean distance, ρ(x,x′) =∥x−x′∥=√∑d
i=1(xi−x′
i)2.

i=1(xi−x′
i)2.
LetS= (x1,y1),..., (xm,ym) be a sequence of training examples. For each
x∈X, letπ1(x),...,πm(x) be a reordering of {1,...,m}according to their
distance to x,ρ(x,xi). That is, for all i<m ,
ρ(x,xπi(x))≤ρ(x,xπi+1(x)).
For a number k, thek-NN rule for binary classiﬁcation is deﬁned as follows:

Figure 19.1 An illustration of the decision boundaries of the 1-NN rule. The points
depicted are the sample points, and the predicted label of any new point will be the
label of the sample point in the center of the cell it belongs to. These cells are called a
Voronoi Tessellation of the space.
k-NN
input: a training sample S= (x1,y1),..., (xm,ym)
output: for every point x∈X,
return the majority label among {yπi(x):i≤k}
Whenk= 1, we have the 1-NN rule:
hS(x) =yπ1(x).
A geometric illustration of the 1-NN rule is given in Figure 19.1.
For regression problems, namely, Y=R, one can deﬁne the prediction to be
the average target of the knearest neighbors. That is, hS(x) =1
k∑k
i=1yπi(x).
More generally, for some function φ: (X×Y )k→Y, thek-NN rule with respect
toφis:
hS(x) =φ(
(xπ1(x),yπ1(x)),..., (xπk(x),yπk(x)))
. (19.1)
It is easy to verify that we can cast the prediction by majority of labels (for
classiﬁcation) or by the averaged target (for regression) as in Equation (19.1) by

k∑k
toφis:
hS(x) =φ(
. (19.1)
an appropriate choice of φ. The generality can lead to other rules; for example, if
Y=R, we can take a weighted average of the targets according to the distance
fromx:
hS(x) =k∑
i=1ρ(x,xπi(x))
∑k
j=1ρ(x,xπj(x))yπi(x).
19.2 Analysis
Since the NN rules are such natural learning methods, their generalization prop-
erties have been extensively studied. Most previous results are asymptotic con-
sistency results, analyzing the performance of NN rules when the sample size, m,

260 Nearest Neighbor
goes to inﬁnity, and the rate of convergence depends on the underlying distribu-
tion. As we have argued in Section 7.4, this type of analysis is not satisfactory.
One would like to learn from ﬁnite training samples and to understand the gen-
eralization performance as a function of the size of such ﬁnite training sets and
clear prior assumptions on the data distribution. We therefore provide a ﬁnite-
sample analysis of the 1-NN rule, showing how the error decreases as a function
ofmand how it depends on properties of the distribution. We will also explain
how the analysis can be generalized to k-NN rules for arbitrary values of k. In
particular, the analysis speciﬁes the number of examples required to achieve a
true error of 2 LD(h⋆) +ϵ, whereh⋆is the Bayes optimal hypothesis, assuming
that the labeling rule is “well behaved” (in a sense we will deﬁne later).
19.2.1 A Generalization Bound for the 1-NN Rule
We now analyze the true error of the 1-NN rule for binary classiﬁcation with

the 0-1 loss, namely, Y={0,1}andℓ(h,(x,y)) = 1[h(x)̸=y]. We also assume
throughout the analysis that X= [0,1]dandρis the Euclidean distance.
We start by introducing some notation. Let Dbe a distribution over X×Y .
LetDXdenote the induced marginal distribution over Xand letη:Rd→Rbe
the conditional probability1over the labels, that is,
η(x) =P[y= 1|x].
Recall that the Bayes optimal rule (that is, the hypothesis that minimizes LD(h)
over all functions) is
h⋆(x) =1[η(x)>1/2].
We assume that the conditional probability function ηisc-Lipschitz for some

c>0: Namely, for all x,x′∈X,|η(x)−η(x′)|≤c∥x−x′∥. In other words, this
assumption means that if two vectors are close to each other then their labels
are likely to be the same.
The following lemma applies the Lipschitzness of the conditional probability
function to upper bound the true error of the 1-NN rule as a function of the
expected distance between each test instance and its nearest neighbor in the
training set.
lemma 19.1 LetX= [0,1]d,Y={0,1}, andDbe a distribution over X×Y

for which the conditional probability function, η, is ac-Lipschitz function. Let
S= (x1,y1),..., (xm,ym)be an i.i.d. sample and let hSbe its corresponding
1-NN hypothesis. Let h⋆be the Bayes optimal rule for η. Then,
E
S∼Dm[LD(hS)]≤2LD(h⋆) +c E
S∼Dm,x∼D[∥x−xπ1(x)∥].
1Formally, P[y= 1|x] = limδ→0D({(x′,1):x′∈B(x,δ)})
D({(x′,y):x′∈B(x,δ),y∈Y}), whereB(x,δ) is a ball of radius δ
centered around x.

19.2 Analysis 261
Proof SinceLD(hS) =E(x,y)∼D[1[hS(x)̸=y]], we obtain that ES[LD(hS)] is the
probability to sample a training set Sand an additional example ( x,y), such
that the label of π1(x) is diﬀerent from y. In other words, we can ﬁrst sample
munlabeled examples, Sx= (x1,...,xm), according toDX, and an additional
unlabeled example, x∼DX, then ﬁnd π1(x) to be the nearest neighbor of xin
Sx, and ﬁnally sample y∼η(x) andyπ1(x)∼η(π1(x)). It follows that
E
S[LD(hS)] = E
Sx∼Dm
X,x∼DX,y∼η(x),y′∼η(π1(x))[1[y̸=y′]]
= E
Sx∼Dm
X,x∼DX[
P
y∼η(x),y′∼η(π1(x))[y̸=y′]]
. (19.2)
We next upper bound Py∼η(x),y′∼η(x′)[y̸=y′] for any two domain points x,x′:
P
y∼η(x),y′∼η(x′)[y̸=y′] =η(x′)(1−η(x)) + (1−η(x′))η(x)
= (η(x)−η(x) +η(x′))(1−η(x))
+ (1−η(x) +η(x)−η(x′))η(x)
= 2η(x)(1−η(x)) + (η(x)−η(x′))(2η(x)−1).
Using|2η(x)−1|≤1 and the assumption that ηisc-Lipschitz, we obtain that
the probability is at most:
P
y∼η(x),y′∼η(x′)[y̸=y′]≤2η(x)(1−η(x)) +c∥x−x′∥.
Plugging this into Equation (19.2) we conclude that
E
S[LD(hS)]≤E

= E
Sx∼Dm
X,x∼DX[
P
. (19.2)
P
P
E
x[2η(x)(1−η(x))] +cE
S,x[∥x−xπ1(x)∥].
Finally, the error of the Bayes optimal classiﬁer is
LD(h⋆) =E
x[min{η(x),1−η(x)}]≥E
x[η(x)(1−η(x))].
Combining the preceding two inequalities concludes our proof.
The next step is to bound the expected distance between a random xand its
closest element in S. We ﬁrst need the following general probability lemma. The
lemma bounds the probability weight of subsets that are not hit by a random
sample, as a function of the size of that sample.

E
LD(h⋆) =E
lemma 19.2 LetC1,...,Crbe a collection of subsets of some domain set, X.
LetSbe a sequence of mpoints sampled i.i.d. according to some probability
distribution,DoverX. Then,
E
S∼Dm
∑
i:Ci∩S=∅P[Ci]
≤r
me.

262 Nearest Neighbor
Proof From the linearity of expectation, we can rewrite:
E
S
∑
=r∑
i=1P[Ci]E
S[
1[Ci∩S=∅]]
.
Next, for each iwe have
E
S[
=P
S[Ci∩S=∅] = (1−P[Ci])m≤e−P[Ci]m.
Combining the preceding two equations we get
E
S
∑
≤r∑
i=1P[Ci]e−P[Ci]m≤rmax
iP[Ci]e−P[Ci]m.
Finally, by a standard calculus, max aae−ma≤1
meand this concludes the proof.
Equipped with the preceding lemmas we are now ready to state and prove the
main result of this section – an upper bound on the expected error of the 1-NN
learning rule.
theorem 19.3 LetX= [0,1]d,Y={0,1}, andDbe a distribution over X×Y
hSdenote the result of applying the 1-NN rule to a sample S∼Dm. Then,
E
S∼Dm[LD(hS)]≤2LD(h⋆) + 4c√
dm−1
d+1.
Proof Fix someϵ= 1/T, for some integer T, letr=Tdand letC1,...,Crbe the
cover of the setXusing boxes of length ϵ: Namely, for every ( α1,...,αd)∈[T]d,

E
dm−1
d+1.
there exists a set Ciof the form{x:∀j,xj∈[(αj−1)/T,αj/T]}. An illustration
ford= 2,T= 5 and the set corresponding to α= (2,4) is given in the following.
11
For each x,x′in the same box we have ∥x−x′∥≤√
dϵ. Otherwise,∥x−x′∥≤√
d.
E
x,S[∥x−xπ1(x)∥]≤E
S
P
⋃
i:Ci∩S=∅Ci
√
d+P
⋃
i:Ci∩S̸=∅Ci
ϵ√
d
,
and by combining Lemma 19.2 with the trivial bound P[⋃
i:Ci∩S̸=∅Ci]≤1 we
get that
E
x,S[∥x−xπ1(x)∥]≤√
d(r
me+ϵ)
.

19.2 Analysis 263
Since the number of boxes is r= (1/ϵ)dwe get that
E
S,x[∥x−xπ1(x)∥]≤√
d(
2dϵ−d
me+ϵ)
.
Combining the preceding with Lemma 19.1 we obtain that
E
S[LD(hS)]≤2LD(h⋆) +c√
d(
2dϵ−d
me+ϵ)
.
Finally, setting ϵ= 2m−1/(d+1)and noting that
2dϵ−d
me+ϵ=2d2−dmd/(d+1)
me+ 2m−1/(d+1)
=m−1/(d+1)(1/e+ 2)≤4m−1/(d+1)
we conclude our proof.
The theorem implies that if we ﬁrst ﬁx the data-generating distribution and
then letmgo to inﬁnity, then the error of the 1-NN rule converges to twice the
Bayes error. The analysis can be generalized to larger values of k, showing that
the expected error of the k-NN rule converges to (1 +√
8/k) times the error of
the Bayes classiﬁer. This is formalized in Theorem 19.5, whose proof is left as a
guided exercise.
19.2.2 The “Curse of Dimensionality”
The upper bound given in Theorem 19.3 grows with c(the Lipschitz coeﬃcient
ofη) and withd, the Euclidean dimension of the domain set X. In fact, it is easy

to see that a necessary condition for the last term in Theorem 19.3 to be smaller
thanϵis thatm≥(4c√
d/ϵ)d+1. That is, the size of the training set should
increase exponentially with the dimension. The following theorem tells us that
this is not just an artifact of our upper bound, but, for some distributions, this
amount of examples is indeed necessary for learning with the NN rule.
theorem 19.4 For anyc > 1, and every learning rule, L, there exists a
distribution over [0,1]d×{0,1}, such thatη(x)isc-Lipschitz, the Bayes error of

the distribution is 0, but for sample sizes m≤(c+ 1)d/2, the true error of the
ruleLis greater than 1/4.
Proof Fix any values of candd. LetGd
cbe the grid on [0 ,1]dwith distance of
1/cbetween points on the grid. That is, each point on the grid is of the form
(a1/c,...,ad/c) whereaiis in{0,...,c−1,c}. Note that, since any two distinct
points on this grid are at least 1 /capart, any function η:GD
C→[0,1] is a
c-Lipschitz function. It follows that the set of all c-Lipschitz functions over Gd
c

c
contains the set of allbinary valued functions over that domain. We can therefore
invoke the No-Free-Lunch result (Theorem 5.1) to obtain a lower bound on the
needed sample sizes for learning that class. The number of points on the grid is
(c+ 1)d; hence, ifm< (c+ 1)d/2, Theorem 5.1 implies the lower bound we are
after.

264 Nearest Neighbor
The exponential dependence on the dimension is known as the curse of di-
mensionality . As we saw, the 1-NN rule might fail if the number of examples is
smaller than Ω(( c+1)d). Therefore, while the 1-NN rule does not restrict itself to
a predeﬁned set of hypotheses, it still relies on some prior knowledge – its success
depends on the assumption that the dimension and the Lipschitz constant of the
underlying distribution, η, are not too high.
19.3 Eﬃcient Implementation*
Nearest Neighbor is a learning-by-memorization type of rule. It requires the
entire training data set to be stored, and at test time, we need to scan the entire
data set in order to ﬁnd the neighbors. The time of applying the NN rule is
therefore Θ( dm). This leads to expensive computation at test time.
Whendis small, several results from the ﬁeld of computational geometry have
proposed data structures that enable to apply the NN rule in time o(dO(1)log(m)).

However, the space required by these data structures is roughly mO(d), which
makes these methods impractical for larger values of d.
To overcome this problem, it was suggested to improve the search method by
allowing an approximate search. Formally, an r-approximate search procedure is
guaranteed to retrieve a point within distance of at most rtimes the distance
to the nearest neighbor. Three popular approximate algorithms for NN are the
kd-tree, balltrees, and locality-sensitive hashing (LSH). We refer the reader, for

example, to (Shakhnarovich, Darrell & Indyk 2006).
19.4 Summary
Thek-NN rule is a very simple learning algorithm that relies on the assumption
that “things that look alike must be alike.” We formalized this intuition using
the Lipschitzness of the conditional probability. We have shown that with a suf-
ﬁciently large training set, the risk of the 1-NN is upper bounded by twice the
risk of the Bayes optimal rule. We have also derived a lower bound that shows
the “curse of dimensionality” – the required sample size might increase expo-

nentially with the dimension. As a result, NN is usually performed in practice
after a dimensionality reduction preprocessing step. We discuss dimensionality
reduction techniques later on in Chapter 23.
19.5 Bibliographic Remarks
Cover & Hart (1967) gave the ﬁrst analysis of 1-NN, showing that its risk con-
verges to twice the Bayes optimal error under mild conditions. Following a lemma
due to Stone (1977), Devroye & Gy¨ orﬁ (1985) have shown that the k-NN rule

is consistent (with respect to the hypothesis class of all functions from Rdto
{0,1}). A good presentation of the analysis is given in the book of Devroye et al.
(1996). Here, we give a ﬁnite sample guarantee that explicitly underscores the
prior assumption on the distribution. See Section 7.4 for a discussion on con-
sistency results. Finally, Gottlieb, Kontorovich & Krauthgamer (2010) derived
another ﬁnite sample bound for NN that is more similar to VC bounds.
19.6 Exercises
In this exercise we will prove the following theorem for the k-NN rule.
theorem 19.5 LetX= [0,1]d,Y={0,1}, andDbe a distribution over X×Y
for which the conditional probability function, η, is ac-Lipschitz function. Let hS
denote the result of applying the k-NN rule to a sample S∼Dm, wherek≥10.
Leth⋆be the Bayes optimal hypothesis. Then,
E
S[LD(hS)]≤(
1 +√
8
k)
LD(h⋆) +(
6c√
d+k)
m−1/(d+1).
1. Prove the following lemma.
lemma 19.6 LetC1,...,Crbe a collection of subsets of some domain set,

E
1 +√
8
k)
LD(h⋆) +(
6c√
d+k)
X. LetSbe a sequence of mpoints sampled i.i.d. according to some probability
distribution,DoverX. Then, for every k≥2,
E
S∼Dm
∑
i:|Ci∩S|<kP[Ci]
≤2rk
m.
Hints :
•Show that
E
S
∑
=r∑
i=1P[Ci]P
S[|Ci∩S|<k].
•Fix someiand suppose that k<P[Ci]m/2. Use Chernoﬀ’s bound to show
that
P
S[|Ci∩S|<k]≤P
S[|Ci∩S|<P[Ci]m/2]≤e−P[Ci]m/8.
•Use the inequality max aae−ma≤1
meto show that for such iwe have
P[Ci]P
S[|Ci∩S|<k]≤P[Ci]e−P[Ci]m/8≤8
me.

E
S∼Dm
∑
≤2rk
m.
Hints :
•Show that
E
S
∑
=r∑
i=1P[Ci]P
that
P
P[Ci]P
me.
•Conclude the proof by using the fact that for the case k≥P[Ci]m/2 we
clearly have:
P[Ci]P
S[|Ci∩S|<k]≤P[Ci]≤2k
m.

266 Nearest Neighbor
2. We use the notation y∼pas a shorthand for “ yis a Bernoulli random variable
with expected value p.” Prove the following lemma:
lemma 19.7 Letk≥10and letZ1,...,Zkbe independent Bernoulli random
variables with P[Zi= 1] =pi. Denotep=1
k∑
ipiandp′=1
k∑k
i=1Zi. Show
that
E
Z1,...,ZkP
y∼p[y̸=1[p′>1/2]]≤(
1 +√
8
k)
P
y∼p[y̸=1[p>1/2]].
Hints :
W.l.o.g. assume that p≤1/2. Then, Py∼p[y̸=1[p>1/2]] =p. Lety′=1[p′>1/2].
•Show that
E
y∼p[y̸=y′]−p=P
Z1,...,Zk[p′>1/2](1−2p).
•Use Chernoﬀ’s bound (Lemma B.3) to show that
P[p′>1/2]≤e−kph(1
2p−1),
where
h(a) = (1 +a) log(1 +a)−a.
•To conclude the proof of the lemma, you can rely on the following inequality
(without proving it): For every p∈[0,1/2] andk≥10:
(1−2p)e−kp+k
2(log(2p)+1)≤√
8
kp.
3. Fix some p,p′∈[0,1] andy′∈{0,1}. Show that
P
y∼p[y̸=y′]≤P
y∼p′[y̸=y′] +|p−p′|.
4. Conclude the proof of the theorem according to the following steps:
•As in the proof of Theorem 19.3, six some ϵ>0 and letC1,...,Crbe the

2p−1),
where
8
kp.
P
cover of the setXusing boxes of length ϵ. For each x,x′in the same
box we have∥x−x′∥≤√
dϵ. Otherwise,∥x−x′∥≤2√
d. Show that
E
S
∑

+ max
iP
S,(x,y)[
hS(x)̸=y|∀j∈[k],∥x−xπj(x)∥≤ϵ√
d]
.(19.3)
•Bound the ﬁrst summand using Lemma 19.6.
•To bound the second summand, let us ﬁx S|xandxsuch that all the k
neighbors of xinS|xare at distance of at most ϵ√
dfrom x. W.l.o.g
assume that the kNN are x1,...,xk. Denotepi=η(xi) and letp=
1
k∑
ipi. Use Exercise 3 to show that
E
y1,...,yjP
y∼η(x)[hS(x)̸=y]≤E

E
S
∑

+ max
iP
S,(x,y)[
d]
.(19.3)
1
k∑
E
y∼p[hS(x)̸=y] +|p−η(x)|.

19.6 Exercises 267
W.l.o.g. assume that p≤1/2. Now use Lemma 19.7 to show that
P
y∼p[hS(x)̸=y]≤(
1 +√
8
k)
P
y∼p[1[p>1/2]̸=y].
•Show that
P
y∼p[1[p>1/2]̸=y] =p= min{p,1−p}≤min{η(x),1−η(x)}+|p−η(x)|.
•Combine all the preceding to obtain that the second summand in Equa-
tion (19.3) is bounded by
(
1 +√
8
k)
LD(h⋆) + 3cϵ√
d.
•User= (2/ϵ)dto obtain that:
E
1 +√
8
k)
d+2(2/ϵ)dk
m.
Setϵ= 2m−1/(d+1)and use
6cm−1/(d+1)√
d+2k
em−1/(d+1)≤(
6c√
d+k)
m−1/(d+1)
to conclude the proof.

20 Neural Networks
An artiﬁcial neural network is a model of computation inspired by the structure
of neural networks in the brain. In simpliﬁed models of the brain, it consists of
a large number of basic computing devices (neurons) that are connected to each
other in a complex communication network, through which the brain is able to
carry out highly complex computations. Artiﬁcial neural networks are formal
computation constructs that are modeled after this computation paradigm.
Learning with neural networks was proposed in the mid-20th century. It yields
an eﬀective learning paradigm and has recently been shown to achieve cutting-
edge performance on several learning tasks.
A neural network can be described as a directed graph whose nodes correspond
to neurons and edges correspond to links between them. Each neuron receives
as input a weighted sum of the outputs of the neurons connected to its incoming
edges. We focus on feedforward networks in which the underlying graph does not
contain cycles.

In the context of learning, we can deﬁne a hypothesis class consisting of neural
network predictors, where all the hypotheses share the underlying graph struc-
ture of the network and diﬀer in the weights over edges. As we will show in
Section 20.3, every predictor over nvariables that can be implemented in time
T(n) can also be expressed as a neural network predictor of size O(T(n)2), where
the size of the network is the number of nodes in it. It follows that the family
of hypothesis classes of neural networks of polynomial size can suﬃce for all

practical learning tasks, in which our goal is to learn predictors which can be
implemented eﬃciently. Furthermore, in Section 20.4 we will show that the sam-
ple complexity of learning such hypothesis classes is also bounded in terms of the
size of the network. Hence, it seems that this is the ultimate learning paradigm
we would want to adapt, in the sense that it both has a polynomial sample com-
plexity and has the minimal approximation error among all hypothesis classes
consisting of eﬃciently implementable predictors.

The caveat is that the problem of training such hypothesis classes of neural net-
work predictors is computationally hard. This will be formalized in Section 20.5.
A widely used heuristic for training neural networks relies on the SGD frame-
work we studied in Chapter 14. There, we have shown that SGD is a successful
learner if the loss function is convex. In neural networks, the loss function is
highly nonconvex. Nevertheless, we can still implement the SGD algorithm and

hope it will ﬁnd a reasonable solution (as happens to be the case in several
practical tasks). In Section 20.6 we describe how to implement SGD for neural
networks. In particular, the most complicated operation is the calculation of the
gradient of the loss function with respect to the parameters of the network. We
present the backpropagation algorithm that eﬃciently calculates the gradient.
20.1 Feedforward Neural Networks
The idea behind neural networks is that many neurons can be joined together
by communication links to carry out complex computations. It is common to
describe the structure of a neural network as a graph whose nodes are the neurons
and each (directed) edge in the graph links the output of some neuron to the
input of another neuron. We will restrict our attention to feedforward network
structures in which the underlying graph does not contain cycles.
A feedforward neural network is described by a directed acyclic graph, G=

(V,E), and a weight function over the edges, w:E→R. Nodes of the graph
correspond to neurons. Each single neuron is modeled as a simple scalar func-
tion,σ:R→R. We will focus on three possible functions for σ: the sign
function,σ(a) = sign(a), the threshold function, σ(a) =1[a>0], and the sig-
moid function, σ(a) = 1/(1 + exp(−a)), which is a smooth approximation to the
threshold function. We call σthe “activation” function of the neuron. Each edge
in the graph links the output of some neuron to the input of another neuron.

The input of a neuron is obtained by taking a weighted sum of the outputs of
all the neurons connected to it, where the weighting is according to w.
To simplify the description of the calculation performed by the network, we
further assume that the network is organized in layers . That is, the set of nodes
can be decomposed into a union of (nonempty) disjoint subsets, V=·∪T
t=0Vt,
such that every edge in Econnects some node in Vt−1to some node in Vt, for
somet∈[T]. The bottom layer, V0, is called the input layer. It contains n+ 1

t=0Vt,
neurons, where nis the dimensionality of the input space. For every i∈[n], the
output of neuron iinV0is simplyxi. The last neuron in V0is the “constant”
neuron, which always outputs 1. We denote by vt,itheith neuron of the tth layer
and byot,i(x) the output of vt,iwhen the network is fed with the input vector x.
Therefore, for i∈[n] we haveo0,i(x) =xiand fori=n+ 1 we have o0,i(x) = 1.
We now proceed with the calculation in a layer by layer manner. Suppose we
have calculated the outputs of the neurons at layer t. Then, we can calculate

the outputs of the neurons at layer t+ 1 as follows. Fix some vt+1,j∈Vt+1.
Letat+1,j(x) denote the input to vt+1,jwhen the network is fed with the input
vector x. Then,
at+1,j(x) =∑
r: (vt,r,vt+1,j)∈Ew((vt,r,vt+1,j))ot,r(x),

270 Neural Networks
and
ot+1,j(x) =σ(at+1,j(x)).
That is, the input to vt+1,jis a weighted sum of the outputs of the neurons in Vt
that are connected to vt+1,j, where weighting is according to w, and the output
ofvt+1,jis simply the application of the activation function σon its input.
LayersV1,...,VT−1are often called hidden layers . The top layer, VT, is called
the output layer. In simple prediction problems the output layer contains a single
neuron whose output is the output of the network.
We refer to Tas the number of layers in the network (excluding V0), or the
“depth” of the network. The size of the network is |V|. The “width” of the
network is max t|Vt|. An illustration of a layered feedforward neural network of
depth 2, size 10, and width 5, is given in the following. Note that there is a
neuron in the hidden layer that has no incoming edges. This neuron will output
the constant σ(0).
v0,1x1
v0,2x2
v0,3x3
v0,4 constantv1,1
v1,2
v1,3
v1,4
v1,5v2,1 OutputHidden
layer
(V1)Input
layer
(V0)Output
layer

v0,1x1
v0,2x2
v0,3x3
v1,2
v1,3
v1,4
layer
(V1)Input
layer
layer
(V2)
20.2 Learning Neural Networks
Once we have speciﬁed a neural network by ( V,E,σ,w ), we obtain a function
hV,E,σ,w :R|V0|−1→R|VT|. Any set of such functions can serve as a hypothesis
class for learning. Usually, we deﬁne a hypothesis class of neural network predic-
tors by ﬁxing the graph ( V,E) as well as the activation function σand letting
the hypothesis class be all functions of the form hV,E,σ,w for somew:E→R.
The triplet ( V,E,σ ) is often called the architecture of the network. We denote
the hypothesis class by

HV,E,σ ={hV,E,σ,w :wis a mapping from EtoR}. (20.1)

That is, the parameters specifying a hypothesis in the hypothesis class are the
weights over the edges of the network.
We can now study the approximation error, estimation error, and optimization
error of such hypothesis classes. In Section 20.3 we study the approximation
error ofHV,E,σ by studying what type of functions hypotheses in HV,E,σ can
implement, in terms of the size of the underlying graph. In Section 20.4 we
study the estimation error of HV,E,σ , for the case of binary classiﬁcation (i.e.,
VT= 1 andσis the sign function), by analyzing its VC dimension. Finally, in
Section 20.5 we show that it is computationally hard to learn the class HV,E,σ ,
even if the underlying graph is small, and in Section 20.6 we present the most
commonly used heuristic for training HV,E,σ .
20.3 The Expressive Power of Neural Networks
In this section we study the expressive power of neural networks, namely, what

type of functions can be implemented using a neural network. More concretely,
we will ﬁx some architecture, V,E,σ , and will study what functions hypotheses
inHV,E,σ can implement, as a function of the size of V.
We start the discussion with studying which type of Boolean functions (i.e.,
functions from{±1}nto{±1}) can be implemented by HV,E,sign. Observe that
for every computer in which real numbers are stored using bbits, whenever we
calculate a function f:Rn→Ron such a computer we in fact calculate a

functiong:{±1}nb→{± 1}b. Therefore, studying which Boolean functions can
be implemented by HV,E,signcan tell us which functions can be implemented on
a computer that stores real numbers using bbits.
We begin with a simple claim, showing that without restricting the size of the
network, every Boolean function can be implemented using a neural network of
depth 2.
claim 20.1 For everyn, there exists a graph (V,E)of depth 2, such that
HV,E,signcontains all functions from {±1}nto{±1}.

depth 2.
Proof We construct a graph with |V0|=n+ 1,|V1|= 2n+ 1,and|V2|= 1. Let
Ebe all possible edges between adjacent layers. Now, let f:{±1}n→{± 1}
be some Boolean function. We need to show that we can adjust the weights so
that the network will implement f. Let u1,...,ukbe all vectors in {±1}non
whichfoutputs 1. Observe that for every iand every x∈{± 1}n, ifx̸=ui
then⟨x,ui⟩≤n−2 and if x=uithen⟨x,ui⟩=n. It follows that the function
gi(x) = sign(⟨x,ui⟩−n+ 1) equals 1 if and only if x=ui. It follows that we can

adapt the weights between V0andV1so that for every i∈[k], the neuron v1,i
implements the function gi(x). Next, we observe that f(x) is the disjunction of

272 Neural Networks
the functions gi(x), and therefore can be written as
f(x) = sign(k∑
i=1gi(x) +k−1)
,
The preceding claim shows that neural networks can implement any Boolean
function. However, this is a very weak property, as the size of the resulting
network might be exponentially large. In the construction given at the proof of
Claim 20.1, the number of nodes in the hidden layer is exponentially large. This
is not an artifact of our proof, as stated in the following theorem.
theorem 20.2 For everyn, lets(n)be the minimal integer such that there
exists a graph (V,E)with|V|=s(n)such that the hypothesis class HV,E,sign
contains all the functions from {0,1}nto{0,1}. Then,s(n)is exponential in n.
Similar results hold for HV,E,σ whereσis the sigmoid function.
Proof Suppose that for some ( V,E) we have thatHV,E,signcontains all functions
from{0,1}nto{0,1}. It follows that it can shatter the set of m= 2nvectors in

{0,1}nand hence the VC dimension of HV,E,signis 2n. On the other hand, the
VC dimension of HV,E,signis bounded by O(|E|log(|E|))≤O(|V|3), as we will
show in the next section. This implies that |V|≥Ω(2n/3), which concludes our
proof for the case of networks with the sign activation function. The proof for
the sigmoid case is analogous.
Remark 20.1 It is possible to derive a similar theorem for HV,E,σ for anyσ, as
long as we restrict the weights so that it is possible to express every weight using

a number of bits which is bounded by a universal constant. We can even con-
sider hypothesis classes where diﬀerent neurons can employ diﬀerent activation
functions, as long as the number of allowed activation functions is also ﬁnite.
Which functions can we express using a network of polynomial size? The pre-
ceding claim tells us that it is impossible to express all Boolean functions using
a network of polynomial size. On the positive side, in the following we show

that all Boolean functions that can be calculated in time O(T(n)) can also be
expressed by a network of size O(T(n)2).
theorem 20.3 LetT:N→Nand for every n, letFnbe the set of functions
that can be implemented using a Turing machine using runtime of at most T(n).
Then, there exist constants b,c∈R+such that for every n, there is a graph
(Vn,En)of size at most cT(n)2+bsuch thatHVn,En,signcontainsFn.
The proof of this theorem relies on the relation between the time complexity

of programs and their circuit complexity (see, for example, Sipser (2006)). In a
nutshell, a Boolean circuit is a type of network in which the individual neurons

20.3 The Expressive Power of Neural Networks 273
implement conjunctions, disjunctions, and negation of their inputs. Circuit com-
plexity measures the size of Boolean circuits required to calculate functions. The
relation between time complexity and circuit complexity can be seen intuitively
as follows. We can model each step of the execution of a computer program as a
simple operation on its memory state. Therefore, the neurons at each layer of the
network will reﬂect the memory state of the computer at the corresponding time,
and the translation to the next layer of the network involves a simple calculation
that can be carried out by the network. To relate Boolean circuits to networks
with the sign activation function, we need to show that we can implement the
operations of conjunction, disjunction, and negation, using the sign activation
function. Clearly, we can implement the negation operator using the sign activa-
tion function. The following lemma shows that the sign activation function can

also implement conjunctions and disjunctions of its inputs.
lemma 20.4 Suppose that a neuron v, that implements the sign activation
function, has kincoming edges, connecting it to neurons whose outputs are in
{±1}. Then, by adding one more edge, linking a “constant” neuron to v, and
by adjusting the weights on the edges to v, the output of vcan implement the
conjunction or the disjunction of its inputs.
Proof Simply observe that if f:{±1}k→ {± 1}is the conjunction func-
tion,f(x) =∧ixi, then it can be written as f(x) = sign(
1−k+∑k
i=1xi)

1−k+∑k
i=1xi)
.
Similarly, the disjunction function, f(x) =∨ixi, can be written as f(x) =
sign(
k−1 +∑k
i=1xi)
.
So far we have discussed Boolean functions. In Exercise 1 we show that neural
networks are universal approximators . That is, for every ﬁxed precision param-
eter,ϵ>0, and every Lipschitz function f: [−1,1]n→[−1,1], it is possible to
construct a network such that for every input x∈[−1,1]n, the network outputs
a number between f(x)−ϵandf(x) +ϵ. However, as in the case of Boolean

1−k+∑k
i=1xi)
.
sign(
k−1 +∑k
i=1xi)
.
functions, the size of the network here again cannot be polynomial in n. This is
formalized in the following theorem, whose proof is a direct corollary of Theo-
rem 20.2 and is left as an exercise.
theorem 20.5 Fix someϵ∈(0,1). For every n, lets(n)be the minimal integer
such that there exists a graph (V,E)with|V|=s(n)such that the hypothesis class
HV,E,σ , withσbeing the sigmoid function, can approximate, to within precision
ofϵ, every 1-Lipschitz function f: [−1,1]n→[−1,1]. Thens(n)is exponential
inn.

inn.
20.3.1 Geometric Intuition
We next provide several geometric illustrations of functions f:R2→ {± 1}
and show how to express them using a neural network with the sign activation
function.

274 Neural Networks
Let us start with a depth 2 network, namely, a network with a single hidden
layer. Each neuron in the hidden layer implements a halfspace predictor. Then,
the single neuron at the output layer applies a halfspace on top of the binary
outputs of the neurons in the hidden layer. As we have shown before, a halfspace
can implement the conjunction function. Therefore, such networks contain all
hypotheses which are an intersection of k−1 halfspaces, where kis the number
of neurons in the hidden layer; namely, they can express all convex polytopes
withk−1 faces. An example of an intersection of 5 halfspaces is given in the
We have shown that a neuron in layer V2can implement a function that
indicates whether xis in some convex polytope. By adding one more layer, and
letting the neuron in the output layer implement the disjunction of its inputs,
we get a network that computes the union of polytopes. An illustration of such
a function is given in the following.

20.4 The Sample Complexity of Neural Networks
Next we discuss the sample complexity of learning the class HV,E,σ . Recall that
the fundamental theorem of learning tells us that the sample complexity of learn-
ing a hypothesis class of binary classiﬁers depends on its VC dimension. There-
fore, we focus on calculating the VC dimension of hypothesis classes of the form
HV,E,σ , where the output layer of the graph contains a single neuron.

We start with the sign activation function, namely, with HV,E,sign. What is
the VC dimension of this class? Intuitively, since we learn |E|parameters, the
VC dimension should be order of |E|. This is indeed the case, as formalized by
the following theorem.
theorem 20.6 The VC dimension of HV,E,signisO(|E|log(|E|)).

20.4 The Sample Complexity of Neural Networks 275
Proof To simplify the notation throughout the proof, let us denote the hy-
pothesis class by H. Recall the deﬁnition of the growth function, τH(m), from
Section 6.5.1. This function measures max C⊂X:|C|=m|HC|, whereHCis the re-
striction ofHto functions from Cto{0,1}. We can naturally extend the deﬁ-
nition for a set of functions from Xto some ﬁnite set Y, by lettingHCbe the
restriction ofHto functions from CtoY, and keeping the deﬁnition of τH(m)
intact.
Our neural network is deﬁned by a layered graph. Let V0,...,VTbe the layers
of the graph. Fix some t∈[T]. By assigning diﬀerent weights on the edges
betweenVt−1andVt, we obtain diﬀerent functions from R|Vt−1|→{± 1}|Vt|. Let
H(t)be the class of all possible such mappings from R|Vt−1|→{± 1}|Vt|. Then,
Hcan be written as a composition, H=H(T)◦...◦H(1). In Exercise 4 we show
that the growth function of a composition of hypothesis classes is bounded by

intact.
the products of the growth functions of the individual classes. Therefore,
τH(m)≤T∏
t=1τH(t)(m).
In addition, each H(t)can be written as a product of function classes, H(t)=
H(t,1)×···×H(t,|Vt|), where eachH(t,j)is all functions from layer t−1 to{±1}
that thejth neuron of layer tcan implement. In Exercise 3 we bound product
classes, and this yields
τH(t)(m)≤|Vt|∏
i=1τH(t,i)(m).
Letdt,ibe the number of edges that are headed to the ith neuron of layer t.
Since the neuron is a homogenous halfspace hypothesis and the VC dimension

τH(m)≤T∏
of homogenous halfspaces is the dimension of their input, we have by Sauer’s
lemma that
τH(t,i)(m)≤(
em
dt,i)dt,i
≤(em)dt,i.
Overall, we obtained that
τH(m)≤(em)∑
t,idt,i= (em)|E|.
Now, assume that there are mshattered points. Then, we must have τH(m) =
2m, from which we obtain
2m≤(em)|E|⇒m≤|E|log(em)/log(2).
The claim follows by Lemma A.2.
Next, we consider HV,E,σ , whereσis the sigmoid function. Surprisingly, it
turns out that the VC dimension of HV,E,σ is lower bounded by Ω( |E|2) (see

lemma that
em
dt,i)dt,i
Exercise 5.) That is, the VC dimension is the number of tunable parameters
squared. It is also possible to upper bound the VC dimension by O(|V|2|E|2),
but the proof is beyond the scope of this book. In any case, since in practice

276 Neural Networks
we only consider networks in which the weights have a short representation as
ﬂoating point numbers with O(1) bits, by using the discretization trick we easily
obtain that such networks have a VC dimension of O(|E|), even if we use the
sigmoid activation function.
20.5 The Runtime of Learning Neural Networks
In the previous sections we have shown that the class of neural networks with an
underlying graph of polynomial size can express all functions that can be imple-
mented eﬃciently, and that the sample complexity has a favorable dependence
on the size of the network. In this section we turn to the analysis of the time
complexity of training neural networks.
We ﬁrst show that it is NP hard to implement the ERM rule with respect to
HV,E,signeven for networks with a single hidden layer that contain just 4 neurons
in the hidden layer.
theorem 20.7 Letk≥3. For every n, let (V,E)be a layered graph with n
input nodes, k+ 1nodes at the (single) hidden layer, where one of them is the

constant neuron, and a single output node. Then, it is NP hard to implement the
ERM rule with respect to HV,E,sign.
The proof relies on a reduction from the k-coloring problem and is left as
Exercise 6.
One way around the preceding hardness result could be that for the purpose
of learning, it may suﬃce to ﬁnd a predictor h∈H with low empirical error,
not necessarily an exact ERM. However, it turns out that even the task of ﬁnd-
ing weights that result in close-to-minimal empirical error is computationally
infeasible (see (Bartlett & Ben-David 2002)).

One may also wonder whether it may be possible to change the architecture
of the network so as to circumvent the hardness result. That is, maybe ERM
with respect to the original network structure is computationally hard but ERM
with respect to some other, larger, network may be implemented eﬃciently (see
Chapter 8 for examples of such cases). Another possibility is to use other acti-
vation functions (such as sigmoids, or any other type of eﬃciently computable
activation functions). There is a strong indication that all of such approaches

are doomed to fail. Indeed, under some cryptographic assumption, the problem
of learning intersections of halfspaces is known to be hard even in the repre-
sentation independent model of learning (see Klivans & Sherstov (2006)). This
implies that, under the same cryptographic assumption, any hypothesis class
which contains intersections of halfspaces cannot be learned eﬃciently.

practical tasks).
20.6 SGD and Backpropagation
The problem of ﬁnding a hypothesis in HV,E,σ with a low risk amounts to the
problem of tuning the weights over the edges. In this section we show how to
apply a heuristic search for good weights using the SGD algorithm. Throughout
this section we assume that σis the sigmoid function, σ(a) = 1/(1 +e−a), but
the derivation holds for any diﬀerentiable scalar function.
SinceEis a ﬁnite set, we can think of the weight function as a vector w∈R|E|.
Suppose the network has ninput neurons and koutput neurons, and denote by
hw:Rn→Rkthe function calculated by the network if the weight function is
deﬁned by w. Let us denote by ∆( hw(x),y) the loss of predicting hw(x) when
the target is y∈Y. For concreteness, we will take ∆ to be the squared loss,
∆(hw(x),y) =1
2∥hw(x)−y∥2; however, similar derivation can be obtained for

every diﬀerentiable function. Finally, given a distribution Dover the examples
domain, Rn×Rk, letLD(w) be the risk of the network, namely,
LD(w) = E
(x,y)∼D[∆(hw(x),y)].
Recall the SGD algorithm for minimizing the risk function LD(w). We repeat
the pseudocode from Chapter 14 with a few modiﬁcations, which are relevant
to the neural network application because of the nonconvexity of the objective
function. First, while in Chapter 14 we initialized wto be the zero vector, here

LD(w) = E
we initialize wto be a randomly chosen vector with values close to zero. This
is because an initialization with the zero vector will lead all hidden neurons to
have the same weights (if the network is a full layered network). In addition,
the hope is that if we repeat the SGD procedure several times, where each time
we initialize the process with a new random vector, one of the runs will lead
to a good local minimum. Second, while a ﬁxed step size, η, is guaranteed to

be good enough for convex problems, here we utilize a variable step size, ηt, as
deﬁned in Section 14.4.2. Because of the nonconvexity of the loss function, the
choice of the sequence ηtis more signiﬁcant, and it is tuned in practice by a trial
and error manner. Third, we output the best performing vector on a validation
set. In addition, it is sometimes helpful to add regularization on the weights,
with parameter λ. That is, we try to minimize LD(w) +λ
2∥w∥2. Finally, the

gradient does not have a closed form solution. Instead, it is implemented using
the backpropagation algorithm, which will be described in the sequel.

278 Neural Networks
SGD for Neural Networks
number of iterations τ
step size sequence η1,η2,...,ητ
input:
layered graph ( V,E)
diﬀerentiable activation function σ:R→R
choose w(1)∈R|E|at random
(from a distribution s.t. w(1)is close enough to 0)
fori= 1,2,...,τ
sample ( x,y)∼D
calculate gradient vi=backpropagation (x,y,w,(V,E),σ)
update w(i+1)=w(i)−ηi(vi+λw(i))
output:
¯wis the best performing w(i)on a validation set
Backpropagation
input:
example ( x,y), weight vector w, layered graph ( V,E),
activation function σ:R→R
denote layers of the graph V0,...,VTwhereVt={vt,1,...,vt,kt}
deﬁneWt,i,jas the weight of ( vt,j,vt+1,i)
(where we set Wt,i,j= 0 if (vt,j,vt+1,i)/∈E)
forward:
seto0=x
fori= 1,...,kt
setat,i=∑kt−1
j=1Wt−1,i,jot−1,j
setot,i=σ(at,i)
backward:
setδT=oT−y
fort=T−1,T−2,..., 1
δt,i=∑kt+1
j=1Wt,j,iδt+1,jσ′(at+1,j)
output:
foreach edge ( vt−1,j,vt,i)∈E
set the partial derivative to δt,iσ′(at,i)ot−1,j

20.6 SGD and Backpropagation 279
Explaining How Backpropagation Calculates the Gradient:
We next explain how the backpropagation algorithm calculates the gradient of
the loss function on an example ( x,y) with respect to the vector w. Let us ﬁrst
recall a few deﬁnitions from vector calculus. Each element of the gradient is
the partial derivative with respect to the variable in wcorresponding to one of
the edges of the network. Recall the deﬁnition of a partial derivative. Given a
functionf:Rn→R, the partial derivative with respect to the ith variable at w
is obtained by ﬁxing the values of w1,...,wi−1,wi+1,wn, which yields the scalar
functiong:R→Rdeﬁned by g(a) =f((w1,...,wi−1,wi+a,wi+1,...,wn)),
and then taking the derivative of gat 0. For a function with multiple outputs,
f:Rn→Rm, the Jacobian offatw∈Rn, denotedJw(f), is them×nmatrix
whosei,jelement is the partial derivative of fi:Rn→Rw.r.t. itsjth variable
atw. Note that if m= 1 then the Jacobian matrix is the gradient of the function

(represented as a row vector). Two examples of Jacobian calculations, which we
will later use, are as follows.
•Letf(w) =AwforA∈Rm,n. ThenJw(f) =A.
•For everyn, we use the notation σto denote the function from RntoRn
which applies the sigmoid function element-wise. That is, α=σ(θ) means
that for every iwe haveαi=σ(θi) =1
1+exp(−θi). It is easy to verify
thatJθ(σ) is a diagonal matrix whose ( i,i) entry isσ′(θi), whereσ′is
the derivative function of the (scalar) sigmoid function, namely, σ′(θi) =
1
(1+exp(θi))(1+exp(−θi)). We also use the notation diag( σ′(θ)) to denote this

1
matrix.
The chain rule for taking the derivative of a composition of functions can be
written in terms of the Jacobian as follows. Given two functions f:Rn→Rm
andg:Rk→Rn, we have that the Jacobian of the composition function,
(f◦g) :Rk→Rm, atw, is
Jw(f◦g) =Jg(w)(f)Jw(g).
For example, for g(w) =Aw, whereA∈Rn,k, we have that
Jw(σ◦g) = diag(σ′(Aw))A.
To describe the backpropagation algorithm, let us ﬁrst decompose Vinto the
layers of the graph, V=·∪T
t=0Vt. For every t, let us write Vt={vt,1,...,vt,kt},

matrix.
wherekt=|Vt|. In addition, for every tdenoteWt∈Rkt+1,kta matrix which
gives a weight to every potential edge between VtandVt+1. If the edge exists in
Ethen we set Wt,i,jto be the weight, according to w, of the edge ( vt,j,vt+1,i).
Otherwise, we add a “phantom” edge and set its weight to be zero, Wt,i,j= 0.
Since when calculating the partial derivative with respect to the weight of some
edge we ﬁx all other weights, these additional “phantom” edges have no eﬀect

on the partial derivative with respect to existing edges. It follows that we can
assume, without loss of generality, that all edges exist, that is, E=∪t(Vt×Vt+1).

280 Neural Networks
Next, we discuss how to calculate the partial derivatives with respect to the
edges from Vt−1toVt, namely, with respect to the elements in Wt−1. Since we
ﬁx all other weights of the network, it follows that the outputs of all the neurons
inVt−1are ﬁxed numbers which do not depend on the weights in Wt−1. Denote
the corresponding vector by ot−1. In addition, let us denote by ℓt:Rkt→Rthe
loss function of the subnetwork deﬁned by layers Vt,...,VTas a function of the
outputs of the neurons in Vt. The input to the neurons of Vtcan be written as
at=Wt−1ot−1and the output of the neurons of Vtisot=σ(at). That is, for
everyjwe haveot,j=σ(at,j). We obtain that the loss, as a function of Wt−1,
can be written as
gt(Wt−1) =ℓt(ot) =ℓt(σ(at)) =ℓt(σ(Wt−1ot−1)).
It would be convenient to rewrite this as follows. Let wt−1∈Rkt−1ktbe the
column vector obtained by concatenating the rows of Wt−1and then taking the
transpose of the resulting long vector. Deﬁne by Ot−1thekt×(kt−1kt) matrix
Ot−1=
o⊤

Ot−1=
o⊤
t−1 0··· 0
0o⊤
t−1··· 0
0 0···o⊤
t−1
. (20.2)
Then,Wt−1ot−1=Ot−1wt−1, so we can also write
gt(wt−1) =ℓt(σ(Ot−1wt−1)).
Therefore, applying the chain rule, we obtain that
Jwt−1(gt) =Jσ(Ot−1wt−1)(ℓt) diag(σ′(Ot−1wt−1))Ot−1.
Using our notation we have ot=σ(Ot−1wt−1) and at=Ot−1wt−1, which yields
Jwt−1(gt) =Jot(ℓt) diag(σ′(at))Ot−1.
Let us also denote δt=Jot(ℓt). Then, we can further rewrite the preceding as
Jwt−1(gt) =(
δt,1σ′(at,1)o⊤
t−1, ... , δ t,ktσ′(at,kt)o⊤
t−1)
. (20.3)

o⊤
t−1 0··· 0
0o⊤
t−1··· 0
0 0···o⊤
t−1
t−1)
. (20.3)
It is left to calculate the vector δt=Jot(ℓt) for every t. This is the gradient
ofℓtatot. We calculate this in a recursive manner. First observe that for the
last layer we have that ℓT(u) = ∆( u,y), where ∆ is the loss function. Since we
assume that ∆( u,y) =1
2∥u−y∥2we obtain that Ju(ℓT) = (u−y). In particular,
δT=JoT(ℓT) = (oT−y). Next, note that
ℓt(u) =ℓt+1(σ(Wtu)).
Therefore, by the chain rule,
Ju(ℓt) =Jσ(Wtu)(ℓt+1)diag(σ′(Wtu))Wt.

In particular,
δt=Jot(ℓt) =Jσ(Wtot)(ℓt+1)diag(σ′(Wtot))Wt
=Jot+1(ℓt+1)diag(σ′(at+1))Wt
=δt+1diag(σ′(at+1))Wt.
In summary, we can ﬁrst calculate the vectors {at,ot}from the bottom of
the network to its top. Then, we calculate the vectors {δt}from the top of
the network back to its bottom. Once we have all of these vectors, the partial
derivatives are easily obtained using Equation (20.3). We have thus shown that
the pseudocode of backpropagation indeed calculates the gradient.
20.7 Summary
Neural networks over graphs of size s(n) can be used to describe hypothesis
classes of all predictors that can be implemented in runtime of O(√
s(n)). We
have also shown that their sample complexity depends polynomially on s(n)
(speciﬁcally, it depends on the number of edges in the network). Therefore, classes
of neural network hypotheses seem to be an excellent choice. Regrettably, the
problem of training the network on the basis of training data is computationally

s(n)). We
hard. We have presented the SGD framework as a heuristic approach for training
neural networks and described the backpropagation algorithm which eﬃciently
calculates the gradient of the loss function with respect to the weights over the
edges.
20.8 Bibliographic Remarks
Neural networks were extensively studied in the 1980s and early 1990s, but with
mixed empirical success. In recent years, a combination of algorithmic advance-
ments, as well as increasing computational power and data size, has led to a

edges.
breakthrough in the eﬀectiveness of neural networks. In particular, “deep net-
works” (i.e., networks of more than 2 layers) have shown very impressive practical
performance on a variety of domains. A few examples include convolutional net-
works (Lecun & Bengio 1995), restricted Boltzmann machines (Hinton, Osindero
& Teh 2006), auto-encoders (Ranzato, Huang, Boureau & Lecun 2007, Bengio &
LeCun 2007, Collobert & Weston 2008, Lee, Grosse, Ranganath & Ng 2009, Le,

Ranzato, Monga, Devin, Corrado, Chen, Dean & Ng 2012), and sum-product
networks (Livni, Shalev-Shwartz & Shamir 2013, Poon & Domingos 2011). See
also (Bengio 2009) and the references therein.
The expressive power of neural networks and the relation to circuit complexity
have been extensively studied in (Parberry 1994). For the analysis of the sample
complexity of neural networks we refer the reader to (Anthony & Bartlet 1999).
Our proof technique of Theorem 20.6 is due to Kakade and Tewari lecture notes.

282 Neural Networks
Klivans & Sherstov (2006) have shown that for any c>0, intersections of nc
halfspaces over{±1}nare not eﬃciently PAC learnable, even if we allow repre-
sentation independent learning. This hardness result relies on the cryptographic
assumption that there is no polynomial time solution to the unique-shortest-
vector problem. As we have argued, this implies that there cannot be an eﬃcient
algorithm for training neural networks, even if we allow larger networks or other
activation functions that can be implemented eﬃciently.
The backpropagation algorithm has been introduced in Rumelhart, Hinton &
Williams (1986).
20.9 Exercises
1.Neural Networks are universal approximators: Letf: [−1,1]n→
[−1,1] be aρ-Lipschitz function. Fix some ϵ >0. Construct a neural net-
workN: [−1,1]n→[−1,1], with the sigmoid activation function, such that
for every x∈[−1,1]nit holds that|f(x)−N(x)|≤ϵ.
Hint: Similarly to the proof of Theorem 19.3, partition [ −1,1]ninto small

boxes. Use the Lipschitzness of fto show that it is approximately constant
at each box. Finally, show that a neural network can ﬁrst decide which box
the input vector belongs to, and then predict the averaged value of fat that
box.
2. Prove Theorem 20.5.
Hint: For everyf:{−1,1}n→ {− 1,1}construct a 1-Lipschitz function
g: [−1,1]n→[−1,1] such that if you can approximate gthen you can express
f.
3.Growth function of product: Fori= 1,2, letFibe a set of functions from

box.
f.
XtoYi. DeﬁneH=F1×F 2to be the Cartesian product class. That is, for
everyf1∈F1andf2∈F2, there exists h∈Hsuch thath(x) = (f1(x),f2(x)).
Prove that τH(m)≤τF1(m)τF2(m).
4.Growth function of composition: LetF1be a set of functions from X
toZand letF2be a set of functions from ZtoY. LetH=F2◦F1be the
composition class. That is, for every f1∈F1andf2∈F2, there exists h∈H
such thath(x) =f2(f1(x)). Prove that τH(m)≤τF2(m)τF1(m).
5.VC of sigmoidal networks: In this exercise we show that there is a graph

(V,E) such that the VC dimension of the class of neural networks over these
graphs with the sigmoid activation function is Ω( |E|2). Note that for every ϵ>
0, the sigmoid activation function can approximate the threshold activation
function, 1[∑
ixi], up to accuracy ϵ. To simplify the presentation, throughout
the exercise we assume that we can exactly implement the activation function
1[∑
ixi>0]using a sigmoid activation function.
Fix somen.
1. Construct a network, N1, withO(n) weights, which implements a function

1[∑
Fix somen.
fromRto{0,1}nand satisﬁes the following property. For every x∈{0,1}n,

20.9 Exercises 283
if we feed the network with the real number 0 .x1x2...xn, then the output
of the network will be x.
Hint: Denoteα= 0.x1x2...xnand observe that 10kα−0.5 is at least 0 .5
ifxk= 1 and is at most −0.3 ifxk=−1.
2. Construct a network, N2, withO(n) weights, which implements a function
from [n] to{0,1}nsuch thatN2(i) =eifor alli. That is, upon receiving
the inputi, the network outputs the vector of all zeros except 1 at the i’th
neuron.
3. Letα1,...,αnbenreal numbers such that every αiis of the form 0 .a(i)
1a(i)
2...a(i)
n,
witha(i)
j∈{0,1}. Construct a network, N3, withO(n) weights, which im-
plements a function from [ n] toR, and satisﬁes N2(i) =αifor everyi∈[n].
4. Combine N1,N3to obtain a network that receives i∈[n] and output a(i).
5. Construct a network N4that receives ( i,j)∈[n]×[n] and outputs a(i)
j.
Hint: Observe that the AND function over {0,1}2can be calculated using
O(1) weights.
6. Conclude that there is a graph with O(n) weights such that the VC di-

1a(i)
2...a(i)
n,
witha(i)
j.
mension of the resulting hypothesis class is n2.
6. Prove Theorem 20.7.
Hint: The proof is similar to the hardness of learning intersections of halfs-
paces – see Exercise 32 in Chapter 8.

21 Online Learning
In this chapter we describe a diﬀerent model of learning, which is called online
learning. Previously, we studied the PAC learning model, in which the learner
ﬁrst receives a batch of training examples, uses the training set to learn a hy-
pothesis, and only when learning is completed uses the learned hypothesis for
predicting the label of new examples. In our papayas learning problem, this
means that we should ﬁrst buy a bunch of papayas and taste them all. Then, we
use all of this information to learn a prediction rule that determines the taste
of new papayas. In contrast, in online learning there is no separation between a
training phase and a prediction phase. Instead, each time we buy a papaya, it
is ﬁrst considered a testexample since we should predict whether it is going to
taste good. Then, after taking a bite from the papaya, we know the true label,
and the same papaya can be used as a training example that can help us improve
our prediction mechanism for future papayas.

Concretely, online learning takes place in a sequence of consecutive rounds.
On each online round, the learner ﬁrst receives an instance (the learner buys
a papaya and knows its shape and color, which form the instance). Then, the
learner is required to predict a label (is the papaya tasty?). At the end of the
round, the learner obtains the correct label (he tastes the papaya and then knows
whether it is tasty or not). Finally, the learner uses this information to improve
his future predictions.
To analyze online learning, we follow a similar route to our study of PAC

learning. We start with online binary classiﬁcation problems. We consider both
the realizable case, in which we assume, as prior knowledge, that all the labels are
generated by some hypothesis from a given hypothesis class, and the unrealizable
case, which corresponds to the agnostic PAC learning model. In particular, we
present an important algorithm called Weighted-Majority . Next, we study online
learning problems in which the loss function is convex. Finally, we present the

Perceptron algorithm as an example of the use of surrogate convex loss functions
in the online learning model.

288 Online Learning
21.1 Online Classiﬁcation in the Realizable Case
Online learning is performed in a sequence of consecutive rounds, where at round
tthe learner is given an instance, xt, taken from an instance domain X, and is
required to provide its label. We denote the predicted label by pt. After predicting
the label, the correct label, yt∈{0,1}, is revealed to the learner. The learner’s
goal is to make as few prediction mistakes as possible during this process. The
learner tries to deduce information from previous rounds so as to improve its
predictions on future rounds.
Clearly, learning is hopeless if there is no correlation between past and present
rounds. Previously in the book, we studied the PAC model in which we assume
that past and present examples are sampled i.i.d. from the same distribution
source. In the online learning model we make no statistical assumptions regard-
ing the origin of the sequence of examples. The sequence is allowed to be deter-

ministic, stochastic, or even adversarially adaptive to the learner’s own behavior
(as in the case of spam e-mail ﬁltering). Naturally, an adversary can make the
number of prediction mistakes of our online learning algorithm arbitrarily large.
For example, the adversary can present the same instance on each online round,
wait for the learner’s prediction, and provide the opposite label as the correct
label.
To make nontrivial statements we must further restrict the problem. The real-

label.
izability assumption is one possible natural restriction. In the realizable case, we
assume that all the labels are generated by some hypothesis, h⋆:X→Y . Fur-
thermore,h⋆is taken from a hypothesis class H, which is known to the learner.
This is analogous to the PAC learning model we studied in Chapter 3. With this
restriction on the sequence, the learner should make as few mistakes as possible,
assuming that both h⋆and the sequence of instances can be chosen by an ad-

versary. For an online learning algorithm, A, we denote by MA(H) the maximal
number of mistakes Amight make on a sequence of examples which is labeled by
someh⋆∈H. We emphasize again that both h⋆and the sequence of instances
can be chosen by an adversary. A bound on MA(H) is called a mistake-bound and
we will study how to design algorithms for which MA(H) is minimal. Formally:
definition 21.1 (Mistake Bounds, Online Learnability) Let Hbe a hypoth-
esis class and let Abe an online learning algorithm. Given any sequence S=

(x1,h⋆(y1)),..., (xT,h⋆(yT)), whereTis any integer and h⋆∈H, letMA(S) be
the number of mistakes Amakes on the sequence S. We denote by MA(H) the
supremum of MA(S) over all sequences of the above form. A bound of the form
MA(H)≤B <∞is called a mistake bound . We say that a hypothesis class His
online learnable if there exists an algorithm Afor whichMA(H)≤B <∞.
Our goal is to study which hypothesis classes are learnable in the online model,
and in particular to ﬁnd good learning algorithms for a given hypothesis class.

Remark 21.1 Throughout this section and the next, we ignore the computa-

21.1 Online Classiﬁcation in the Realizable Case 289
tional aspect of learning, and do not restrict the algorithms to be eﬃcient. In
Section 21.3 and Section 21.4 we study eﬃcient online learning algorithms.
To simplify the presentation, we start with the case of a ﬁnite hypothesis class,
namely,|H|<∞.
In PAC learning, we identiﬁed ERM as a good learning algorithm, in the sense
that ifHis learnable then it is learnable by the rule ERM H. A natural learning
rule for online learning is to use (at any online round) any ERM hypothesis,
namely, any hypothesis which is consistent with all past examples.
Consistent
input: A ﬁnite hypothesis class H
initialize :V1=H
receive xt
choose any h∈Vt
predictpt=h(xt)
receive true label yt=h⋆(xt)
updateVt+1={h∈Vt:h(xt) =yt}
The Consistent algorithm maintains a set, Vt, of all the hypotheses which
are consistent with ( x1,y1),..., (xt−1,yt−1). This set is often called the version
space. It then picks any hypothesis from Vtand predicts according to this hy-

receive xt
pothesis.
Obviously, whenever Consistent makes a prediction mistake, at least one
hypothesis is removed from Vt. Therefore, after making Mmistakes we have
|Vt|≤|H|−M. SinceVtis always nonempty (by the realizability assumption it
containsh⋆) we have 1≤|Vt|≤|H|−M. Rearranging, we obtain the following:
corollary 21.2 LetHbe a ﬁnite hypothesis class. The Consistent algorithm
enjoys the mistake bound MConsistent (H)≤|H|− 1.
It is rather easy to construct a hypothesis class and a sequence of examples on

pothesis.
which Consistent will indeed make |H|− 1 mistakes (see Exercise 1). Therefore,
we present a better algorithm in which we choose h∈Vtin a smarter way. We
shall see that this algorithm is guaranteed to make exponentially fewer mistakes.
Halving
receive xt
predictpt= argmaxr∈{0,1}|{h∈Vt:h(xt) =r}|
(in case of a tie predict pt= 1)

290 Online Learning
theorem 21.3 LetHbe a ﬁnite hypothesis class. The Halving algorithm
enjoys the mistake bound MHalving (H)≤log2(|H|).
Proof We simply note that whenever the algorithm errs we have |Vt+1|≤|Vt|/2,
(hence the name Halving). Therefore, if Mis the total number of mistakes, we
have
1≤|VT+1|≤|H| 2−M.
Rearranging this inequality we conclude our proof.
Of course, Halving ’s mistake bound is much better than Consistent ’s mistake
bound. We already see that online learning is diﬀerent from PAC learning—while
in PAC, any ERM hypothesis is good, in online learning choosing an arbitrary
ERM hypothesis is far from being optimal.
21.1.1 Online Learnability
We next take a more general approach, and aim at characterizing online learn-
ability. In particular, we target the following question: What is the optimal online
learning algorithm for a given hypothesis class H?
We present a dimension of hypothesis classes that characterizes the best achiev-

able mistake bound. This measure was proposed by Nick Littlestone and we
therefore refer to it as Ldim( H).
To motivate the deﬁnition of Ldim it is convenient to view the online learning
process as a game between two players: the learner versus the environment. On
roundtof the game, the environment picks an instance xt, the learner predicts a
labelpt∈{0,1}, and ﬁnally the environment outputs the true label, yt∈{0,1}.
Suppose that the environment wants to make the learner err on the ﬁrst Trounds
of the game. Then, it must output yt= 1−pt, and the only question is how it

should choose the instances xtin such a way that ensures that for some h⋆∈H
we haveyt=h⋆(xt) for allt∈[T].
A strategy for an adversarial environment can be formally described as a
binary tree, as follows. Each node of the tree is associated with an instance from
X. Initially, the environment presents to the learner the instance associated with
the root of the tree. Then, if the learner predicts pt= 1 the environment will
declare that this is a wrong prediction (i.e., yt= 0) and will traverse to the right

child of the current node. If the learner predicts pt= 0 then the environment
will setyt= 1 and will traverse to the left child. This process will continue and
at each round, the environment will present the instance associated with the
current node.
Formally, consider a complete binary tree of depth T(we deﬁne the depth of
the tree as the number of edges in a path from the root to a leaf). We have
2T+1−1 nodes in such a tree, and we attach an instance to each node. Let

v1,...,v2T+1−1be these instances. We start from the root of the tree, and set
x1=v1. At round t, we set xt=vitwhereitis the current node. At the end of

21.1 Online Classiﬁcation in the Realizable Case 291
v1
v2 v3h1h2h3h4
v10 0 1 1
v20 1∗ ∗
v3∗ ∗ 0 1
Figure 21.1 An illustration of a shattered tree of depth 2. The dashed path
corresponds to the sequence of examples (( v1,1),(v3,0)). The tree is shattered by
H={h1,h2,h3,h4}, where the predictions of each hypothesis in Hon the instances
v1,v2,v3is given in the table (the ’*’ mark means that hj(vi) can be either 1 or 0).
roundt, we go to the left child of itifyt= 0 or to the right child if yt= 1. That
is,it+1= 2it+yt. Unraveling the recursion we obtain it= 2t−1+∑t−1
j=1yj2t−1−j.
The preceding strategy for the environment succeeds only if for every ( y1,...,yT)
there exists h∈Hsuch thatyt=h(xt) for allt∈[T]. This leads to the following
deﬁnition.
definition 21.4 (HShattered Tree) A shattered tree of depth dis a sequence
of instances v1,...,v2d−1inXsuch that for every labeling ( y1,...,yd)∈{0,1}d
there exists h∈H such that for all t∈[d] we haveh(vit) =ytwhereit=
2t−1+∑t−1

2t−1+∑t−1
An illustration of a shattered tree of depth 2 is given in Figure 21.1.
definition 21.5 (Littlestone’s Dimension (Ldim)) Ldim( H) is the maximal
integerTsuch that there exists a shattered tree of depth T, which is shattered
byH.
The deﬁnition of Ldim and the discussion above immediately imply the fol-
lowing:
lemma 21.6 No algorithm can have a mistake bound strictly smaller than
Ldim(H); namely, for every algorithm, A, we haveMA(H)≥Ldim(H).
Proof LetT= Ldim(H) and let v1,...,v2T−1be a sequence that satisﬁes the

byH.
lowing:
requirements in the deﬁnition of Ldim. If the environment sets xt=vitand
yt= 1−ptfor allt∈[T], then the learner makes Tmistakes while the deﬁnition
of Ldim implies that there exists a hypothesis h∈H such thatyt=h(xt) for all
t.
Let us now give several examples.
Example 21.2 LetHbe a ﬁnite hypothesis class. Clearly, any tree that is shat-
tered byHhas depth of at most log2(|H|). Therefore, Ldim( H)≤log2(|H|).
Another way to conclude this inequality is by combining Lemma 21.6 with The-
orem 21.3.
Example 21.3 LetX={1,...,d}andH={h1,...,hd}wherehj(x) = 1 iﬀ

292 Online Learning
x=j. Then, it is easy to show that Ldim( H) = 1 while|H|=dcan be arbitrarily
large. Therefore, this example shows that Ldim( H) can be signiﬁcantly smaller
than log2(|H|).
Example 21.4 LetX= [0,1] andH={x↦→1[x<a]:a∈[0,1]}; namely,His
the class of thresholds on the interval [0 ,1]. Then, Ldim(H) =∞. To see this,
consider the tree
1/2
1/4
1/8 3/83/4
5/8 7/8
This tree is shattered by H. And, because of the density of the reals, this tree
can be made arbitrarily deep.
Lemma 21.6 states that Ldim( H) lower bounds the mistake bound of any
algorithm. Interestingly, there is a standard algorithm whose mistake bound
matches this lower bound. The algorithm is similar to the Halving algorithm.
Recall that the prediction of Halving is made according to a majority vote of
the hypotheses which are consistent with previous examples. We denoted this
set byVt. Put another way, Halving partitionsVtinto two sets: V+
t={h∈Vt:
h(xt) = 1}andV−
t={h∈Vt:h(xt) = 0}. It then predicts according to the

t={h∈Vt:
larger of the two groups. The rationale behind this prediction is that whenever
Halving makes a mistake it ends up with |Vt+1|≤0.5|Vt|.
The optimal algorithm we present in the following uses the same idea, but
instead of predicting according to the larger class, it predicts according to the
class with larger Ldim.
Standard Optimal Algorithm (SOA)
input: A hypothesis class H
receive xt
forr∈{0,1}letV(r)
t={h∈Vt:h(xt) =r}
predictpt= argmaxr∈{0,1}Ldim(V(r)
t)
receive true label yt

receive xt
t)
The following lemma formally establishes the optimality of the preceding al-
gorithm.

21.1 Online Classiﬁcation in the Realizable Case 293
lemma 21.7 SOA enjoys the mistake bound MSOA(H)≤Ldim(H).
Proof It suﬃces to prove that whenever the algorithm makes a prediction mis-
take we have Ldim( Vt+1)≤Ldim(Vt)−1. We prove this claim by assuming the
contrary, that is, Ldim( Vt+1) = Ldim(Vt). If this holds true, then the deﬁnition
ofptimplies that Ldim( V(r)
t) = Ldim(Vt) for bothr= 1 andr= 0. But, then
we can construct a shaterred tree of depth Ldim( Vt) + 1 for the class Vt, which
leads to the desired contradiction.
Combining Lemma 21.7 and Lemma 21.6 we obtain:
corollary 21.8 LetHbe any hypothesis class. Then, the standard optimal
algorithm enjoys the mistake bound MSOA(H) = Ldim(H)and no other algorithm
can haveMA(H)<Ldim(H).
Comparison to VC Dimension
In the PAC learning model, learnability is characterized by the VC dimension of
the classH. Recall that the VC dimension of a class His the maximal number
dsuch that there are instances x1,...,xdthat are shattered by H. That is, for

any sequence of labels ( y1,...,yd)∈{0,1}dthere exists a hypothesis h∈H
that gives exactly this sequence of labels. The following theorem relates the VC
dimension to the Littlestone dimension.
theorem 21.9 For any classH,VCdim(H)≤Ldim(H), and there are classes
for which strict inequality holds. Furthermore, the gap can be arbitrarily larger.
Proof We ﬁrst prove that VCdim( H)≤Ldim(H). Suppose VCdim( H) =dand
letx1,...,xdbe a shattered set. We now construct a complete binary tree of

instances v1,...,v2d−1, where all nodes at depth iare set to be xi– see the
following illustration:
x1
x2
x3 x3x2
x3 x3
Now, the deﬁnition of a shattered set clearly implies that we got a valid shattered
tree of depth d, and we conclude that VCdim( H)≤Ldim(H). To show that the
gap can be arbitrarily large simply note that the class given in Example 21.4 has
VC dimension of 1 whereas its Littlestone dimension is inﬁnite.

294 Online Learning
21.2 Online Classiﬁcation in the Unrealizable Case
In the previous section we studied online learnability in the realizable case. We
now consider the unrealizable case. Similarly to the agnostic PAC model, we
no longer assume that all labels are generated by some h⋆∈H, but we require
the learner to be competitive with the best ﬁxed predictor from H. This is
captured by the regret of the algorithm, which measures how “sorry” the learner
is, in retrospect, not to have followed the predictions of some hypothesis h∈H.
Formally, the regret of an algorithm Arelative tohwhen running on a sequence
ofTexamples is deﬁned as
RegretA(h,T) = sup
(x1,y1),...,(xT,yT)[T∑
t=1|pt−yt|−T∑
t=1|h(xt)−yt|]
, (21.1)
and the regret of the algorithm relative to a hypothesis class His
RegretA(H,T) = sup
h∈HRegretA(h,T). (21.2)
We restate the learner’s goal as having the lowest possible regret relative to H.
An interesting question is whether we can derive an algorithm with low regret,

, (21.1)
meaning that RegretA(H,T) grows sublinearly with the number of rounds, T,
which implies that the diﬀerence between the error rate of the learner and the
best hypothesis in Htends to zero as Tgoes to inﬁnity.
We ﬁrst show that this is an impossible mission—no algorithm can obtain a
sublinear regret bound even if |H|= 2. Indeed, consider H={h0,h1}, whereh0
is the function that always returns 0 and h1is the function that always returns
1. An adversary can make the number of mistakes of any online algorithm be

equal toT, by simply waiting for the learner’s prediction and then providing
the opposite label as the true label. In contrast, for any sequence of true labels,
y1,...,yT, letbbe the majority of labels in y1,...,yT, then the number of
mistakes of hbis at most T/2. Therefore, the regret of any online algorithm
might be at least T−T/2 =T/2, which is not sublinear in T. This impossibility
result is attributed to Cover (Cover 1965).
To sidestep Cover’s impossibility result, we must further restrict the power

of the adversarial environment. We do so by allowing the learner to randomize
his predictions. Of course, this by itself does not circumvent Cover’s impossibil-
ity result, since in deriving this result we assumed nothing about the learner’s
strategy. To make the randomization meaningful, we force the adversarial envir-
onment to decide on ytwithout knowing the random coins ﬂipped by the learner
on roundt. The adversary can still know the learner’s forecasting strategy and

even the random coin ﬂips of previous rounds, but it does not know the actual
value of the random coin ﬂips used by the learner on round t. With this (mild)
change of game, we analyze the expected number of mistakes of the algorithm,
where the expectation is with respect to the learner’s own randomization. That
is, if the learner outputs ˆ ytwhere P[ˆyt= 1] =pt, then the expected loss he pays

21.2 Online Classiﬁcation in the Unrealizable Case 295
on roundtis
P[ˆyt̸=yt] =|pt−yt|.
Put another way, instead of having the predictions of the learner being in {0,1}
we allow them to be in [0 ,1], and interpret pt∈[0,1] as the probability to predict
the label 1 on round t.
With this assumption it is possible to derive a low regret algorithm. In partic-
ular, we will prove the following theorem.
theorem 21.10 For every hypothesis class H, there exists an algorithm for
online classiﬁcation, whose predictions come from [0,1], that enjoys the regret
bound
∀h∈H,T∑
t=1|h(xt)−yt|≤√
2 min{log(|H|),Ldim(H) log(eT)}T.
Furthermore, no algorithm can achieve an expected regret bound smaller than
Ω(√
Ldim(H)T)
.
We will provide a constructive proof of the upper bound part of the preceding
theorem. The proof of the lower bound part can be found in (Ben-David, Pal, &
Shalev-Shwartz 2009).
The proof of Theorem 21.10 relies on the Weighted-Majority algorithm for

bound
∀h∈H,T∑
Ω(√
Ldim(H)T)
.
learning with expert advice. This algorithm is important by itself and we dedicate
the next subsection to it.
21.2.1 Weighted-Majority
Weighted-majority is an algorithm for the problem of prediction with expert ad-
vice. In this online learning problem, on round tthe learner has to choose the
advice ofdgiven experts. We also allow the learner to randomize his choice by
deﬁning a distribution over the dexperts, that is, picking a vector w(t)∈[0,1]d,
with∑
iw(t)
i= 1, and choosing the ith expert with probability w(t)

with∑
iw(t)
i. After the
learner chooses an expert, it receives a vector of costs, vt∈[0,1]d, wherevt,i
is the cost of following the advice of the ith expert. If the learner’s predic-
tions are randomized, then its loss is deﬁned to be the averaged cost, namely,∑
iw(t)
ivt,i=⟨w(t),vt⟩. The algorithm assumes that the number of rounds Tis
given. In Exercise 4 we show how to get rid of this dependence using the doubling
trick.

296 Online Learning
Weighted-Majority
input: number of experts, d; number of rounds, T
parameter: η=√
2 log(d)/T
initialize :˜w(1)= (1,..., 1)
setw(t)=˜w(t)/ZtwhereZt=∑
i˜w(t)
i
choose expert iat random according to P[i] =w(t)
i
receive costs of all experts vt∈[0,1]d
pay cost⟨w(t),vt⟩
update rule∀i,˜w(t+1)
i = ˜w(t)
ie−ηvt,i
The following theorem is key for analyzing the regret bound of Weighted-
Majority.
theorem 21.11 Assuming that T > 2 log(d), the Weighted-Majority algo-
rithm enjoys the bound
T∑
t=1⟨w(t),vt⟩−min
i∈[d]T∑
t=1vt,i≤√
2 log(d)T.
Proof We have:
logZt+1
Zt= log∑
i˜w(t)
i
Zte−ηvt,i= log∑
iw(t)
ie−ηvt,i.
Using the inequality e−a≤1−a+a2/2, which holds for all a∈(0,1), and the
fact that∑
iw(t)
i= 1, we obtain
logZt+1
Zt≤log∑
iw(t)
i(
1−ηvt,i+η2v2
t,i/2)
= log(
1−∑
iw(t)
i(
ηvt,i−η2v2
t,i/2)

def=b)
.
Next, note that b∈(0,1). Therefore, taking log of the two sides of the inequality
1−b≤e−bwe obtain the inequality log(1 −b)≤−b, which holds for all b≤1,
and obtain
logZt+1
Zt≤−∑
iw(t)

T∑
i∈[d]T∑
t=1vt,i≤√
2 log(d)T.
logZt+1
Zt= log∑
i˜w(t)
i
iw(t)
ie−ηvt,i.
fact that∑
iw(t)
logZt+1
Zt≤log∑
iw(t)
i(
t,i/2)
= log(
1−∑
iw(t)
i(
t,i/2)

def=b)
.
and obtain
logZt+1
Zt≤−∑
iw(t)
i(
t,i/2)
=−η⟨w(t),vt⟩+η2∑
iw(t)
iv2
t,i/2
≤−η⟨w(t),vt⟩+η2/2.

21.2 Online Classiﬁcation in the Unrealizable Case 297
Summing this inequality over twe get
log(ZT+1)−log(Z1) =T∑
t=1logZt+1
Zt≤−ηT∑
t=1⟨w(t),vt⟩+Tη2
2. (21.3)
Next, we lower bound ZT+1. For eachi, we can rewrite ˜ w(T+1)
i =e−η∑
tvt,iand
we get that
logZT+1= log(∑
ie−η∑
tvt,i)
≥log(
max
ie−η∑
tvt,i)
=−ηmin
i∑
tvt,i.
Combining the preceding with Equation (21.3) and using the fact that log( Z1) =
log(d) we get that
−ηmin
i∑
tvt,i−log(d)≤ −ηT∑
2,
which can be rearranged as follows:
T∑
i∑
tvt,i≤log(d)
η+ηT
2.
Plugging the value of ηinto the equation concludes our proof.
Proof of Theorem 21.10
Equipped with the Weighted-Majority algorithm and Theorem 21.11, we are
ready to prove Theorem 21.10. We start with the simpler case, in which His
a ﬁnite class, and let us write H={h1,...,hd}. In this case, we can refer to
each hypothesis, hi, as an expert, whose advice is to predict hi(xt), and whose
cost isvt,i=|hi(xt)−yt|. The prediction of the algorithm will therefore be
pt=∑
iw(t)

i∑
η+ηT
2.
pt=∑
iw(t)
ihi(xt)∈[0,1], and the loss is
|pt−yt|=⏐⏐⏐⏐⏐d∑
i=1w(t)
ihi(xt)−yt⏐⏐⏐⏐⏐=⏐⏐⏐⏐⏐d∑
i=1w(t)
i(hi(xt)−yt)⏐⏐⏐⏐⏐.
Now, ifyt= 1, then for all i,hi(xt)−yt≤0. Therefore, the above equals to∑
iw(t)
i|hi(xt)−yt|. Ifyt= 0 then for all i,hi(xt)−yt≥0, and the above also
equals∑
iw(t)
i|hi(xt)−yt|. All in all, we have shown that
|pt−yt|=d∑
i=1w(t)
i|hi(xt)−yt|=⟨w(t),vt⟩.
Furthermore, for each i,∑
tvt,iis exactly the number of mistakes hypothesis hi
makes. Applying Theorem 21.11 we obtain

298 Online Learning
corollary 21.12 LetHbe a ﬁnite hypothesis class. There exists an algorithm
for online classiﬁcation, whose predictions come from [0,1], that enjoys the regret
bound
T∑
t=1|pt−yt|−min
h∈HT∑
t=1|h(xt)−yt| ≤√
2 log(|H|)T.
Next, we consider the case of a general hypothesis class. Previously, we con-
structed an expert for each individual hypothesis. However, if His inﬁnite this
leads to a vacuous bound. The main idea is to construct a set of experts in a
more sophisticated way. The challenge is how to deﬁne a set of experts that, on
one hand, is not excessively large and, on the other hand, contains experts that
give accurate predictions.
We construct the set of experts so that for each hypothesis h∈H and every
sequence of instances, x1,x2,...,xT, there exists at least one expert in the set
which behaves exactly as hon these instances. For each L≤Ldim(H) and each
sequence 1≤i1<i2<···<iL≤Twe deﬁne an expert. The expert simulates

the game between SOA (presented in the previous section) and the environment
on the sequence of instances x1,x2,...,xTassuming that SOA makes a mistake
precisely in rounds i1,i2,...,iL. The expert is deﬁned by the following algorithm.
Expert(i1,i2,...,iL)
input A hypothesis class H; Indicesi1<i2<···<iL
receive xt
deﬁne ˜yt= argmaxrLdim(
V(r)
t)
(in case of a tie set ˜ yt= 0)
ift∈{i1,i2,...,iL}
predict ˆyt= 1−˜yt
else
predict ˆyt= ˜yt
updateVt+1=V(ˆyt)
t

receive xt
V(r)
t)
else
t
Note that each such expert can give us predictions at every round twhile only
observing the instances x1,...,xt. Our generic online learning algorithm is now
an application of the Weighted-Majority algorithm with these experts.
To analyze the algorithm we ﬁrst note that the number of experts is
d=Ldim(H)∑
L=0(T
L)
. (21.4)
It can be shown that when T≥Ldim(H)+2, the right-hand side of the equation
is bounded by ( eT/Ldim(H))Ldim(H)(the proof can be found in Lemma A.5).

21.2 Online Classiﬁcation in the Unrealizable Case 299
Theorem 21.11 tells us that the expected number of mistakes of Weighted-Majority
is at most the number of mistakes of the best expert plus√
2 log(d)T. We will
next show that the number of mistakes of the best expert is at most the number
of mistakes of the best hypothesis in H. The following key lemma shows that,
on any sequence of instances, for each hypothesis h∈H there exists an expert
with the same behavior.
lemma 21.13 LetHbe any hypothesis class with Ldim(H)<∞. Letx1,x2,...,xT
be any sequence of instances. For any h∈H, there exists L≤Ldim(H)and in-
dices 1≤i1< i2<···< iL≤Tsuch that when running Expert(i1,i2,...,iL)
on the sequence x1,x2,...,xT, the expert predicts h(xt)on each online round
t= 1,2,...,T .
Proof Fixh∈Hand the sequence x1,x2,...,xT. We must construct Land the
indicesi1,i2,...,iL. Consider running SOA on the input ( x1,h(x1)), (x2,h(x2)),
..., (xT,h(xT)). SOA makes at most Ldim( H) mistakes on such input. We deﬁne

Lto be the number of mistakes made by SOA and we deﬁne {i1,i2,...,iL}to
be the set of rounds in which SOA made the mistakes.
Now, consider the Expert( i1,i2,...,iL) running on the sequence x1,x2,...,xT.
By construction, the set Vtmaintained by Expert( i1,i2,...,iL) equals the set Vt
maintained by SOA when running on the sequence ( x1,h(x1)),..., (xT,h(xT)).
The predictions of SOA diﬀer from the predictions of hif and only if the round is
in{i1,i2,...,iL}. Since Expert( i1,i2,...,iL) predicts exactly like SOA if tis not

in{i1,i2,...,iL}and the opposite of SOAs’ predictions if tis in{i1,i2,...,iL},
we conclude that the predictions of the expert are always the same as the pre-
dictions ofh.
The previous lemma holds in particular for the hypothesis in Hthat makes the
least number of mistakes on the sequence of examples, and we therefore obtain
corollary 21.14 Let(x1,y1),(x2,y2),..., (xT,yT)be a sequence of examples
and letHbe a hypothesis class with Ldim(H)<∞. There exists L≤Ldim(H)
and indices 1≤i1< i2<···< iL≤T, such that Expert(i1,i2,...,iL)makes

at most as many mistakes as the best h∈H does, namely,
min
h∈HT∑
t=1|h(xt)−yt|
mistakes on the sequence of examples.
Together with Theorem 21.11, the upper bound part of Theorem 21.10 is
proven.

300 Online Learning
21.3 Online Convex Optimization
In Chapter 12 we studied convex learning problems and showed learnability
results for these problems in the agnostic PAC learning framework. In this section
we show that similar learnability results hold for convex problems in the online
learning framework. In particular, we consider the following problem.
Online Convex Optimization
deﬁnitions:
hypothesis classH; domainZ; loss function ℓ:H×Z→R
His convex
∀z∈Z,ℓ(·,z) is a convex function
learner predicts a vector w(t)∈H
environment responds with zt∈Z
learner suﬀers loss ℓ(w(t),zt)
As in the online classiﬁcation problem, we analyze the regret of the algorithm.
Recall that the regret of an online algorithm with respect to a competing hy-
pothesis, which here will be some vector w⋆∈H, is deﬁned as
RegretA(w⋆,T) =T∑
t=1ℓ(w(t),zt)−T∑
t=1ℓ(w⋆,zt). (21.5)
As before, the regret of the algorithm relative to a set of competing vectors, H,
w⋆∈HRegretA(w⋆,T).

In Chapter 14 we have shown that Stochastic Gradient Descent solves convex
learning problems in the agnostic PAC model. We now show that a very similar
algorithm, Online Gradient Descent, solves online convex learning problems.
Online Gradient Descent
parameter: η>0
predict w(t)
receiveztand letft(·) =ℓ(·,zt)
choose vt∈∂ft(w(t))
update:
1.w(t+1
2.w(t+1)= argminw∈H∥w−w(t+1
2)∥

theorem 21.15 The Online Gradient Descent algorithm enjoys the following
regret bound for every w⋆∈H,
RegretA(w⋆,T)≤∥w⋆∥2
2η+η
2T∑
t=1∥vt∥2.
If we further assume that ftisρ-Lipschitz for all t, then setting η= 1/√
Tyields
RegretA(w⋆,T)≤1
2(∥w⋆∥2+ρ2)√
T.
If we further assume that HisB-bounded and we set η=B
ρ√
Tthen
RegretA(H,T)≤Bρ√
T.
Proof The analysis is similar to the analysis of Stochastic Gradient Descent
with projections. Using the projection lemma, the deﬁnition of w(t+1
2), and the
deﬁnition of subgradients, we have that for every t,
≤∥w(t+1
=∥w(t)−ηvt−w⋆∥2−∥w(t)−w⋆∥2
=−2η⟨w(t)−w⋆,vt⟩+η2∥vt∥2
≤−2η(ft(w(t))−ft(w⋆)) +η2∥vt∥2.
Summing over tand observing that the left-hand side is a telescopic sum we
∥w(T+1)−w⋆∥2−∥w(1)−w⋆∥2≤−2ηT∑
t=1(ft(w(t))−ft(w⋆)) +η2T∑
t=1∥vt∥2.
Rearranging the inequality and using the fact that w(1)=0, we get that
T∑

2), and the
≤∥w(t+1
t=1∥vt∥2.
T∑
t=1(ft(w(t))−ft(w⋆))≤∥w(1)−w⋆∥2−∥w(T+1)−w⋆∥2
2η+η
2T∑
t=1∥vt∥2
≤∥w⋆∥2
2η+η
2T∑
t=1∥vt∥2.
This proves the ﬁrst bound in the theorem. The second bound follows from the
assumption that ftisρ-Lipschitz, which implies that ∥vt∥≤ρ.
21.4 The Online Perceptron Algorithm
The Perceptron is a classic online learning algorithm for binary classiﬁcation with
the hypothesis class of homogenous halfspaces, namely, H={x↦→sign(⟨w,x⟩) :

302 Online Learning
w∈Rd}. In Section 9.1.2 we have presented the batch version of the Perceptron,
which aims to solve the ERM problem with respect to H. We now present an
online version of the Perceptron algorithm.
LetX=Rd,Y={−1,1}. On round t, the learner receives a vector xt∈Rd.
The learner maintains a weight vector w(t)∈Rdand predicts pt= sign(⟨w(t),xt⟩).
Then, it receives yt∈Yand pays 1 if pt̸=ytand 0 otherwise.
The goal of the learner is to make as few prediction mistakes as possible. In
Section 21.1 we characterized the optimal algorithm and showed that the best
achievable mistake bound depends on the Littlestone dimension of the class.
We show later that if d≥2 then Ldim(H) =∞, which implies that we have
no hope of making few prediction mistakes. Indeed, consider the tree for which
v1= (1
2,1,0,..., 0),v2= (1
4,1,0,..., 0),v3= (3
4,1,0,..., 0), etc. Because of
the density of the reals, this tree is shattered by the subset of Hwhich contains

v1= (1
all hypotheses that are parametrized by wof the form w= (−1,a,0,..., 0), for
a∈[0,1]. We conclude that indeed Ldim( H) =∞.
To sidestep this impossibility result, the Perceptron algorithm relies on the
technique of surrogate convex losses (see Section 12.3). This is also closely related
to the notion of margin we studied in Chapter 15.
A weight vector wmakes a mistake on an example ( x,y) whenever the sign of
⟨w,x⟩does not equal y. Therefore, we can write the 0 −1 loss function as follows
ℓ(w,(x,y)) =1[y⟨w,x⟩≤0].

On rounds on which the algorithm makes a prediction mistake, we shall use the
hinge-loss as a surrogate convex loss function
ft(w) = max{0,1−yt⟨w,xt⟩}.
The hinge-loss satisﬁes the two conditions:
•ftis a convex function
•For all w,ft(w)≥ℓ(w,(xt,yt)). In particular, this holds for w(t).
On rounds on which the algorithm is correct, we shall deﬁne ft(w) = 0. Clearly,
ftis convex in this case as well. Furthermore, ft(w(t)) =ℓ(w(t),(xt,yt)) = 0.
Remark 21.5 In Section 12.3 we used the same surrogate loss function for all the

examples. In the online model, we allow the surrogate to depend on the speciﬁc
round. It can even depend on w(t). Our ability to use a round speciﬁc surrogate
stems from the worst-case type of analysis we employ in online learning.
Let us now run the Online Gradient Descent algorithm on the sequence of
functions,f1,...,fT, with the hypothesis class being all vectors in Rd(hence,
the projection step is vacuous). Recall that the algorithm initializes w(1)=0
and its update rule is
w(t+1)=w(t)−ηvt
for some vt∈∂ft(w(t)). In our case, if yt⟨w(t),xt⟩>0 thenftis the zero

21.4 The Online Perceptron Algorithm 303
function and we can take vt=0. Otherwise, it is easy to verify that vt=−ytxt
is in∂ft(w(t)). We therefore obtain the update rule
w(t+1)={
w(t)ifyt⟨w(t),xt⟩>0
w(t)+ηytxtotherwise
Denote byMthe set of rounds in which sign( ⟨w(t),xt⟩)̸=yt. Note that on
roundt, the prediction of the Perceptron can be rewritten as
pt= sign(⟨w(t),xt⟩) = sign(
η∑
i∈M:i<tyi⟨xi,xt⟩)
.
This form implies that the predictions of the Perceptron algorithm and the set
Mdo not depend on the actual value of ηas long asη >0. We have therefore
obtained the Perceptron algorithm:
Perceptron
initialize :w1=0
receive xt
predictpt= sign(⟨w(t),xt⟩)
ifyt⟨w(t),xt⟩≤0
w(t+1)=w(t)+ytxt
else
w(t+1)=w(t)
To analyze the Perceptron, we rely on the analysis of Online Gradient De-
scent given in the previous section. In our case, the subgradient of ftwe use
in the Perceptron is vt=−1[yt⟨w(t),xt⟩≤0]ytxt. Indeed, the Perceptron’s update
isw(t+1)=w(t)−vt, and as discussed before this is equivalent to w(t+1)=

receive xt
else
w(t)−ηvtfor everyη>0. Therefore, Theorem 21.15 tells us that
T∑
t=1ft(w(t))−T∑
t=1ft(w⋆)≤1
2η∥w⋆∥2
2+η
2T∑
t=1∥vt∥2
2.
Sinceft(w(t)) is a surrogate for the 0 −1 loss we know that∑T
t=1ft(w(t))≥|M| .
DenoteR= maxt∥xt∥; then we obtain
|M|−T∑
2η∥w⋆∥2
2+η
2|M|R2
Settingη=∥w⋆∥
R√
|M|and rearranging, we obtain
|M|−R∥w⋆∥√
|M|−T∑
t=1ft(w⋆)≤0. (21.6)
This inequality implies

304 Online Learning
theorem 21.16 Suppose that the Perceptron algorithm runs on a sequence
(x1,y1),..., (xT,yT)and letR= maxt∥xt∥. LetMbe the rounds on which the
Perceptron errs and let ft(w) =1[t∈M][1−yt⟨w,xt⟩]+. Then, for every w⋆
|M| ≤∑
tft(w⋆) +R∥w⋆∥√∑
tft(w⋆) +R2∥w⋆∥2.
In particular, if there exists w⋆such thatyt⟨w⋆,xt⟩≥1for alltthen
|M|≤R2∥w⋆∥2.
Proof The theorem follows from Equation (21.6) and the following claim: Given
x,b,c∈R+, the inequality x−b√x−c≤0 implies that x≤c+b2+b√c. The
last claim can be easily derived by analyzing the roots of the convex parabola
Q(y) =y2−by−c.
The last assumption of Theorem 21.16 is called separability with large margin
(see Chapter 15). That is, there exists w⋆that not only satisﬁes that the point
xtlies on the correct side of the halfspace, it also guarantees that xtis not too
close to the decision boundary. More speciﬁcally, the distance from xtto the
decision boundary is at least γ= 1/∥w⋆∥and the bound becomes ( R/γ)2.

When the separability assumption does not hold, the bound involves the term
[1−yt⟨w⋆,xt⟩]+which measures how much the separability with margin require-
ment is violated.
As a last remark we note that there can be cases in which there exists some
w⋆that makes zero errors on the sequence but the Perceptron will make many
errors. Indeed, this is a direct consequence of the fact that Ldim( H) =∞. The
way we sidestep this impossibility result is by assuming more on the sequence of

examples – the bound in Theorem 21.16 will be meaningful only if the cumulative
surrogate loss,∑
tft(w⋆) is not excessively large.
21.5 Summary
In this chapter we have studied the online learning model. Many of the results
we derived for the PAC learning model have an analog in the online model. First,
we have shown that a combinatorial dimension, the Littlestone dimension, char-
acterizes online learnability. To show this, we introduced the SOA algorithm (for

the realizable case) and the Weighted-Majority algorithm (for the unrealizable
case). We have also studied online convex optimization and have shown that
online gradient descent is a successful online learner whenever the loss function
is convex and Lipschitz. Finally, we presented the online Perceptron algorithm
as a combination of online gradient descent and the concept of surrogate convex
loss functions.

21.6 Bibliographic Remarks
The Standard Optimal Algorithm was derived by the seminal work of Lit-
tlestone (1988). A generalization to the nonrealizable case, as well as other
variants like margin-based Littlestone’s dimension, were derived in (Ben-David
et al. 2009). Characterizations of online learnability beyond classiﬁcation have
been obtained in (Abernethy, Bartlett, Rakhlin & Tewari 2008, Rakhlin, Srid-
haran & Tewari 2010, Daniely et al. 2011). The Weighted-Majority algorithm is
due to (Littlestone & Warmuth 1994) and (Vovk 1990).
The term “online convex programming” was introduced by Zinkevich (2003)
but this setting was introduced some years earlier by Gordon (1999). The Per-
ceptron dates back to Rosenblatt (Rosenblatt 1958). An analysis for the re-
alizable case (with margin assumptions) appears in (Agmon 1954, Minsky &
Papert 1969). Freund and Schapire (Freund & Schapire 1999) presented an anal-

ysis for the unrealizable case with a squared-hinge-loss based on a reduction to
the realizable case. A direct analysis for the unrealizable case with the hinge-loss
was given by Gentile (Gentile 2003).
For additional information we refer the reader to Cesa-Bianchi & Lugosi (2006)
and Shalev-Shwartz (2011).
21.7 Exercises
1. Find a hypothesis class Hand a sequence of examples on which Consistent
makes|H|− 1 mistakes.
2. Find a hypothesis class Hand a sequence of examples on which the mistake

bound of the Halving algorithm is tight.
3. Letd≥2,X={1,...,d}and letH={hj:j∈[d]}, wherehj(x) =1[x=j].
CalculateMHalving (H) (i.e., derive lower and upper bounds on MHalving (H),
and prove that they are equal).
4.The Doubling Trick:
In Theorem 21.15, the parameter ηdepends on the time horizon T. In this
exercise we show how to get rid of this dependence by a simple trick.
Consider an algorithm that enjoys a regret bound of the form α√
T, but
its parameters require the knowledge of T. The doubling trick, described in

T, but
the following, enables us to convert such an algorithm into an algorithm that
does not need to know the time horizon. The idea is to divide the time into
periods of increasing size and run the original algorithm on each period.
The Doubling Trick
input: algorithmAwhose parameters depend on the time horizon
form= 0,1,2,...
runAon the 2mroundst= 2m,..., 2m+1−1

306 Online Learning
Show that if the regret of Aon each period of 2mrounds is at most α√
2m,
then the total regret is at most
√
2√
2−1α√
T.
5.Online-to-batch Conversions: In this exercise we demonstrate how a suc-
cessful online learning algorithm can be used to derive a successful PAC
learner as well.
Consider a PAC learning problem for binary classiﬁcation parameterized
by an instance domain, X, and a hypothesis class, H. Suppose that there exists
an online learning algorithm, A, which enjoys a mistake bound MA(H)<∞.
Consider running this algorithm on a sequence of Texamples which are sam-
pled i.i.d. from a distribution Dover the instance space X, and are labeled by
someh⋆∈H. Suppose that for every round t, the prediction of the algorithm
is based on a hypothesis ht:X→{ 0,1}. Show that
E[LD(hr)]≤MA(H)
T,
where the expectation is over the random choice of the instances as well as a
random choice of raccording to the uniform distribution over [ T].

T,
Hint: Use similar arguments to the ones appearing in the proof of Theo-
rem 14.8.

22 Clustering
Clustering is one of the most widely used techniques for exploratory data anal-
ysis. Across all disciplines, from social sciences to biology to computer science,
people try to get a ﬁrst intuition about their data by identifying meaningful
groups among the data points. For example, computational biologists cluster
genes on the basis of similarities in their expression in diﬀerent experiments; re-
tailers cluster customers, on the basis of their customer proﬁles, for the purpose
of targeted marketing; and astronomers cluster stars on the basis of their spacial
proximity.
The ﬁrst point that one should clarify is, naturally, what is clustering? In-
tuitively, clustering is the task of grouping a set of objects such that similar
objects end up in the same group and dissimilar objects are separated into dif-
ferent groups. Clearly, this description is quite imprecise and possibly ambiguous.
Quite surprisingly, it is not at all clear how to come up with a more rigorous

There are several sources for this diﬃculty. One basic problem is that the
two objectives mentioned in the earlier statement may in many cases contradict
each other. Mathematically speaking, similarity (or proximity) is not a transi-
tive relation, while cluster sharing is an equivalence relation and, in particular,
it is a transitive relation. More concretely, it may be the case that there is a
long sequence of objects, x1,...,xmsuch that each xiis very similar to its two

neighbors,xi−1andxi+1, butx1andxmare very dissimilar. If we wish to make
sure that whenever two elements are similar they share the same cluster, then
we must put all of the elements of the sequence in the same cluster. However,
in that case, we end up with dissimilar elements ( x1andxm) sharing a cluster,
thus violating the second requirement.
To illustrate this point further, suppose that we would like to cluster the points
in the following picture into two clusters.

A clustering algorithm that emphasizes not separating close-by points (e.g., the
Single Linkage algorithm that will be described in Section 22.1) will cluster this
input by separating it horizontally according to the two lines:

308 Clustering
In contrast, a clustering method that emphasizes not having far-away points
share the same cluster (e.g., the 2-means algorithm that will be described in
Section 22.1) will cluster the same input by dividing it vertically into the right-
hand half and the left-hand half:
Another basic problem is the lack of “ground truth” for clustering, which is a
common problem in unsupervised learning . So far in the book, we have mainly
dealt with supervised learning (e.g., the problem of learning a classiﬁer from
labeled training data). The goal of supervised learning is clear – we wish to
learn a classiﬁer which will predict the labels of future examples as accurately
as possible. Furthermore, a supervised learner can estimate the success, or the
risk, of its hypotheses using the labeled training data by computing the empirical
loss. In contrast, clustering is an unsupervised learning problem; namely, there
are no labels that we try to predict. Instead, we wish to organize the data in

some meaningful way. As a result, there is no clear success evaluation procedure
for clustering. In fact, even on the basis of full knowledge of the underlying data
distribution, it is not clear what is the “correct” clustering for that data or how
to evaluate a proposed clustering.
Consider, for example, the following set of points in R2:
and suppose we are required to cluster them into two clusters. We have two
highly justiﬁable solutions:

Clustering 309
This phenomenon is not just artiﬁcial but occurs in real applications. A given
set of objects can be clustered in various diﬀerent meaningful ways. This may
be due to having diﬀerent implicit notions of distance (or similarity) between
objects, for example, clustering recordings of speech by the accent of the speaker
versus clustering them by content, clustering movie reviews by movie topic versus
clustering them by the review sentiment, clustering paintings by topic versus
clustering them by style, and so on.
To summarize, there may be several very diﬀerent conceivable clustering so-
lutions for a given data set. As a result, there is a wide variety of clustering
algorithms that, on some input data, will output very diﬀerent clusterings.
A Clustering Model:
Clustering tasks can vary in terms of both the type of input they have and the
type of outcome they are expected to compute. For concreteness, we shall focus
on the following common setup:

Input — a set of elements, X, and a distance function over it. That is, a function
d:X×X→ R+that is symmetric, satisﬁes d(x,x) = 0 for all x∈X
and often also satisﬁes the triangle inequality. Alternatively, the function
could be a similarity function s:X×X → [0,1] that is symmetric
and satisﬁes s(x,x) = 1 for all x∈X. Additionally, some clustering
algorithms also require an input parameter k(determining the number
of required clusters).
Output — a partition of the domain set Xinto subsets. That is, C= (C1,...Ck)
where⋃k

where⋃k
i=1Ci=Xand for alli̸=j,Ci∩Cj=∅. In some situations the
clustering is “soft,” namely, the partition of Xinto the diﬀerent clusters
is probabilistic where the output is a function assigning to each domain
point,x∈X, a vector ( p1(x),...,pk(x)), wherepi(x) =P[x∈Ci] is
the probability that xbelongs to cluster Ci. Another possible output is
a clustering dendrogram (from Greek dendron = tree, gramma = draw-
ing), which is a hierarchical tree of domain subsets, having the singleton
sets in its leaves, and the full domain as its root. We shall discuss this

formulation in more detail in the following.

310 Clustering
In the following we survey some of the most popular clustering methods. In
the last section of this chapter we return to the high level discussion of what is
clustering.
22.1 Linkage-Based Clustering Algorithms
Linkage-based clustering is probably the simplest and most straightforward paradigm
of clustering. These algorithms proceed in a sequence of rounds. They start from
the trivial clustering that has each data point as a single-point cluster. Then,
repeatedly, these algorithms merge the “closest” clusters of the previous cluster-
ing. Consequently, the number of clusters decreases with each such round. If kept
going, such algorithms would eventually result in the trivial clustering in which
all of the domain points share one large cluster. Two parameters, then, need to
be determined to deﬁne such an algorithm clearly. First, we have to decide how
to measure (or deﬁne) the distance between clusters, and, second, we have to

determine when to stop merging. Recall that the input to a clustering algorithm
is a between-points distance function, d. There are many ways of extending dto
a measure of distance between domain subsets (or clusters). The most common
ways are
1. Single Linkage clustering, in which the between-clusters distance is deﬁned
by the minimum distance between members of the two clusters, namely,
D(A,B)def= min{d(x,y) :x∈A, y∈B}
2. Average Linkage clustering, in which the distance between two clusters is

ways are
deﬁned to be the average distance between a point in one of the clusters and
a point in the other, namely,
D(A,B)def=1
|A||B|∑
x∈A,y∈Bd(x,y)
3. Max Linkage clustering, in which the distance between two clusters is deﬁned
as the maximum distance between their elements, namely,
D(A,B)def= max{d(x,y) :x∈A, y∈B}.
The linkage-based clustering algorithms are agglomerative in the sense that they
start from data that is completely fragmented and keep building larger and

|A||B|∑
larger clusters as they proceed. Without employing a stopping rule, the outcome
of such an algorithm can be described by a clustering dendrogram : that is, a tree
of domain subsets, having the singleton sets in its leaves, and the full domain as
its root. For example, if the input is the elements X={a,b,c,d,e}⊂R2with
the Euclidean distance as depicted on the left, then the resulting dendrogram is
the one depicted on the right:

bcdea
{a}{b}{c}{d}{e}{b,c}{d,e}{b,c,d,e}{a,b,c,d,e}
The single linkage algorithm is closely related to Kruskal’s algorithm for ﬁnding
a minimal spanning tree on a weighted graph. Indeed, consider the full graph
whose vertices are elements of Xand the weight of an edge ( x,y) is the distance
d(x,y). Each merge of two clusters performed by the single linkage algorithm
corresponds to a choice of an edge in the aforementioned graph. It is also possible
to show that the set of edges the single linkage algorithm chooses along its run
forms a minimal spanning tree.
If one wishes to turn a dendrogram into a partition of the space (a clustering),
one needs to employ a stopping criterion . Common stopping criteria include
•Fixed number of clusters – ﬁx some parameter, k, and stop merging clusters
as soon as the number of clusters is k.
•Distance upper bound – ﬁx some r∈R+. Stop merging as soon as all the

between-clusters distances are larger than r. We can also set rto be
αmax{d(x,y) :x,y∈ X} for someα < 1. In that case the stopping
criterion is called “scaled distance upper bound.”
22.2k-Means and Other Cost Minimization Clusterings
Another popular approach to clustering starts by deﬁning a cost function over a
parameterized set of possible clusterings and the goal of the clustering algorithm
is to ﬁnd a partitioning (clustering) of minimal cost. Under this paradigm, the
clustering task is turned into an optimization problem. The objective function

is a function from pairs of an input, ( X,d), and a proposed clustering solution
C= (C1,...,Ck), to positive real numbers. Given such an objective function,
which we denote by G, the goalof a clustering algorithm is deﬁned as ﬁnding, for
a given input (X,d), a clustering Cso thatG((X,d),C) is minimized. In order
to reach that goal, one has to apply some appropriate search algorithm.
As it turns out, most of the resulting optimization problems are NP-hard, and

some are even NP-hard to approximate. Consequently, when people talk about,
say,k-means clustering, they often refer to some particular common approxima-
tion algorithm rather than the cost function or the corresponding exact solution
of the minimization problem.
Many common objective functions require the number of clusters, k, as a

312 Clustering
parameter. In practice, it is often up to the user of the clustering algorithm to
choose the parameter kthat is most suitable for the given clustering problem.
In the following we describe some of the most common objective functions.
•Thek-means objective function is one of the most popular clustering
objectives. In k-means the data is partitioned into disjoint sets C1,...,Ck
where each Ciis represented by a centroid µi. It is assumed that the input
setXis embedded in some larger metric space ( X′,d) (so thatX ⊆X′)
and centroids are members of X′. Thek-means objective function measures
the squared distance between each point in Xto the centroid of its cluster.
The centroid of Ciis deﬁned to be
µi(Ci) = argmin
µ∈X′∑
x∈Cid(x,µ)2.
Then, thek-means objective is
Gk−means ((X,d),(C1,...,Ck)) =k∑
i=1∑
x∈Cid(x,µi(Ci))2.
This can also be rewritten as
Gk−means ((X,d),(C1,...,Ck)) = min
µ1,...µk∈X′k∑
i=1∑
x∈Cid(x,µi)2.(22.1)
Thek-means objective function is relevant, for example, in digital com-

µ∈X′∑
i=1∑
i=1∑
munication tasks, where the members of Xmay be viewed as a collection
of signals that have to be transmitted. While Xmay be a very large set
of real valued vectors, digital transmission allows transmitting of only a
ﬁnite number of bits for each signal. One way to achieve good transmis-
sion under such constraints is to represent each member of Xby a “close”
member of some ﬁnite set µ1,...µk, and replace the transmission of any
x∈X by transmitting the index of the closest µi. Thek-means objective

can be viewed as a measure of the distortion created by such a transmission
representation scheme.
•Thek-medoids objective function is similar to the k-means objective,
except that it requires the cluster centroids to be members of the input
set. The objective function is deﬁned by
GK−medoid ((X,d),(C1,...,Ck)) = min
µ1,...µk∈Xk∑
i=1∑
x∈Cid(x,µi)2.
•Thek-median objective function is quite similar to the k-medoids objec-
tive, except that the “distortion” between a data point and the centroid

i=1∑
of its cluster is measured by distance, rather than by the square of the
distance:
GK−median ((X,d),(C1,...,Ck)) = min
i=1∑
x∈Cid(x,µi).

22.2k-Means and Other Cost Minimization Clusterings 313
An example where such an objective makes sense is the facility location
problem. Consider the task of locating kﬁre stations in a city. One can
model houses as data points and aim to place the stations so as to minimize
the average distance between a house and its closest ﬁre station.
The previous examples can all be viewed as center-based objectives. The so-
lution to such a clustering problem is determined by a set of cluster centers,
and the clustering assigns each instance to the center closest to it. More gener-
ally, center-based objective is determined by choosing some monotonic function
f:R+→R+and then deﬁning
Gf((X,d),(C1,...Ck)) = min
i=1∑
x∈Cif(d(x,µi)),
whereX′is eitherXor some superset of X.
Some objective functions are not center based. For example, the sum of in-
cluster distances (SOD)
GSOD((X,d),(C1,...Ck)) =k∑
i=1∑
x,y∈Cid(x,y)
and the MinCut objective that we shall discuss in Section 22.3 are not center-
based objectives.

i=1∑
i=1∑
22.2.1 The k-Means Algorithm
Thek-means objective function is quite popular in practical applications of clus-
tering. However, it turns out that ﬁnding the optimal k-means solution is of-
ten computationally infeasible (the problem is NP-hard, and even NP-hard to
approximate to within some constant). As an alternative, the following simple
iterative algorithm is often used, so often that, in many cases, the term k-means
Clustering refers to the outcome of this algorithm rather than to the cluster-

ing that minimizes the k-means objective cost. We describe the algorithm with
respect to the Euclidean distance function d(x,y) =∥x−y∥.
k-Means
input:X⊂Rn; Number of clusters k
initialize: Randomly choose initial centroids µ1,...,µk
repeat until convergence
∀i∈[k] setCi={x∈X:i= argminj∥x−µj∥}
(break ties in some arbitrary manner)
∀i∈[k] updateµi=1
|Ci|∑
x∈Cix
lemma 22.1 Each iteration of the k-means algorithm does not increase the
k-means objective function (as given in Equation (22.1)).

314 Clustering
Proof To simplify the notation, let us use the shorthand G(C1,...,Ck) for the
k-means objective, namely,
G(C1,...,Ck) = min
µ1,...,µk∈Rnk∑
i=1∑
x∈Ci∥x−µi∥2. (22.2)
It is convenient to deﬁne µ(Ci) =1
|Ci|∑
x∈Cixand note that µ(Ci) = argminµ∈Rn∑
x∈Ci∥x−
µ∥2. Therefore, we can rewrite the k-means objective as
G(C1,...,Ck) =k∑
i=1∑
x∈Ci∥x−µ(Ci)∥2. (22.3)
Consider the update at iteration tof thek-means algorithm. Let C(t−1)
1,...,C(t−1)
k
be the previous partition, let µ(t−1)
i =µ(C(t−1)
i), and letC(t)
1,...,C(t)
kbe the
new partition assigned at iteration t. Using the deﬁnition of the objective as
given in Equation (22.2) we clearly have that
G(C(t)
k)≤k∑
i=1∑
x∈C(t)
i∥x−µ(t−1)
i∥2. (22.4)
In addition, the deﬁnition of the new partition ( C(t)
k) implies that it
minimizes the expression∑k
i=1∑
x∈Ci∥x−µ(t−1)
i∥2over all possible partitions
(C1,...,Ck). Hence,
k∑
i=1∑
x∈C(t)
i∥2≤k∑
i=1∑
x∈C(t−1)
i∥2. (22.5)

kbe the
G(C(t)
k)≤k∑
i=1∑
x∈C(t)
i=1∑
k∑
i=1∑
x∈C(t)
i∥2≤k∑
i=1∑
x∈C(t−1)
Using Equation (22.3) we have that the right-hand side of Equation (22.5) equals
G(C(t−1)
k). Combining this with Equation (22.4) and Equation (22.5),
we obtain that G(C(t)
k)≤G(C(t−1)
k), which concludes our
proof.
While the preceding lemma tells us that the k-means objective is monotonically
nonincreasing, there is no guarantee on the number of iterations the k-means al-
gorithm needs in order to reach convergence. Furthermore, there is no nontrivial

G(C(t−1)
proof.
lower bound on the gap between the value of the k-means objective of the al-
gorithm’s output and the minimum possible value of that objective function. In
fact,k-means might converge to a point which is not even a local minimum (see
Exercise 2). To improve the results of k-means it is often recommended to repeat
the procedure several times with diﬀerent randomly chosen initial centroids (e.g.,
we can choose the initial centroids to be random points from the data).

22.3 Spectral Clustering
Often, a convenient way to represent the relationships between points in a data
setX={x1,...,xm}is by a similarity graph ; each vertex represents a data
pointxi, and every two vertices are connected by an edge whose weight is their
similarity,Wi,j=s(xi,xj), whereW∈Rm,m. For example, we can set Wi,j=
exp(−d(xi,xj)2/σ2), whered(·,·) is a distance function and σis a parameter.
The clustering problem can now be formulated as follows: We want to ﬁnd a
partition of the graph such that the edges between diﬀerent groups have low
weights and the edges within a group have high weights.
In the clustering objectives described previously, the focus was on one side
of our intuitive deﬁnition of clustering – making sure that points in the same
cluster are similar. We now present objectives that focus on the other requirement
– points separated into diﬀerent clusters should be nonsimilar.
22.3.1 Graph Cut

Given a graph represented by a similarity matrix W, the simplest and most
direct way to construct a partition of the graph is to solve the mincut problem,
which chooses a partition C1,...,Ckthat minimizes the objective
cut(C1,...,Ck) =k∑
i=1∑
r∈Ci,s/∈CiWr,s.
Fork= 2, the mincut problem can be solved eﬃciently. However, in practice it
often does not lead to satisfactory partitions. The problem is that in many cases,
the solution of mincut simply separates one individual vertex from the rest of the

i=1∑
graph. Of course, this is not what we want to achieve in clustering, as clusters
should be reasonably large groups of points.
Several solutions to this problem have been suggested. The simplest solution
is to normalize the cut and deﬁne the normalized mincut objective as follows:
RatioCut(C1,...,Ck) =k∑
i=11
|Ci|∑
The preceding objective assumes smaller values if the clusters are not too small.
Unfortunately, introducing this balancing makes the problem computationally

i=11
|Ci|∑
hard to solve. Spectral clustering is a way to relax the problem of minimizing
RatioCut.
22.3.2 Graph Laplacian and Relaxed Graph Cuts
The main mathematical object for spectral clustering is the graph Laplacian
matrix. There are several diﬀerent deﬁnitions of graph Laplacian in the literature,
and in the following we describe one particular deﬁnition.

316 Clustering
definition 22.2 (Unnormalized Graph Laplacian) The unnormalized graph
Laplacian is them×mmatrixL=D−WwhereDis a diagonal matrix with
Di,i=∑m
j=1Wi,j. The matrix Dis called the degree matrix .
The following lemma underscores the relation between RatioCut and the Lapla-
cian matrix.
lemma 22.3 LetC1,...,Ckbe a clustering and let H∈Rm,kbe the matrix
such that
Hi,j=1√
|Cj|1[i∈Cj].
Then, the columns of Hare orthonormal to each other and
RatioCut (C1,...,Ck) = trace(H⊤LH).
Proof Leth1,...,hkbe the columns of H. The fact that these vectors are
orthonormal is immediate from the deﬁnition. Next, by standard algebraic ma-
nipulations, it can be shown that trace( H⊤LH) =∑k
i=1h⊤
iLhiand that for
any vector vwe have
v⊤Lv=1
2(∑
rDr,rv2
r−2∑
r,svrvsWr,s+∑
sDs,sv2
s)
=1
2∑
r,sWr,s(vr−vs)2.
Applying this with v=hiand noting that ( hi,r−hi,s)2is nonzero only if
r∈Ci,s /∈Cior the other way around, we obtain that
h⊤
iLhi=1
|Ci|∑

i=1h⊤
v⊤Lv=1
2(∑
rDr,rv2
r−2∑
sDs,sv2
s)
=1
2∑
h⊤
iLhi=1
|Ci|∑
Therefore, to minimize RatioCut we can search for a matrix Hwhose columns
are orthonormal and such that each Hi,jis either 0 or 1 /√
|Cj|. Unfortunately,
this is an integer programming problem which we cannot solve eﬃciently. Instead,
we relax the latter requirement and simply search an orthonormal matrix H∈
Rm,kthat minimizes trace( H⊤LH). As we will see in the next chapter about
PCA (particularly, the proof of Theorem 23.2), the solution to this problem is

h⊤
iLhi=1
|Ci|∑
to setUto be the matrix whose columns are the eigenvectors corresponding to
thekminimal eigenvalues of L. The resulting algorithm is called Unnormalized
Spectral Clustering.

22.3.3 Unnormalized Spectral Clustering
Unnormalized Spectral Clustering
Input:W∈Rm,m; Number of clusters k
Initialize: Compute the unnormalized graph Laplacian L
LetU∈Rm,kbe the matrix whose columns are the eigenvectors of L
corresponding to the ksmallest eigenvalues
Let v 1,...,vmbe the rows of U
Cluster the points v1,...,vmusingk-means
Output: ClustersC1,...,CKof thek-means algorithm
The spectral clustering algorithm starts with ﬁnding the matrix Hof thek
eigenvectors corresponding to the smallest eigenvalues of the graph Laplacian
matrix. It then represents points according to the rows of H. It is due to the
properties of the graph Laplacians that this change of representation is useful.
In many situations, this change of representation enables the simple k-means
algorithm to detect the clusters seamlessly. Intuitively, if His as deﬁned in
Lemma 22.3 then each point in the new representation is an indicator vector

whose value is nonzero only on the element corresponding to the cluster it belongs
to.
22.4 Information Bottleneck*
The information bottleneck method is a clustering technique introduced by
Tishby, Pereira, and Bialek. It relies on notions from information theory . To
illustrate the method, consider the problem of clustering text documents where
each document is represented as a bag-of-words; namely, each document is a
vector x={0,1}n, wherenis the size of the dictionary and xi= 1 iﬀ the word

to.
corresponding to index iappears in the document. Given a set of mdocuments,
we can interpret the bag-of-words representation of the mdocuments as a joint
probability over a random variable x, indicating the identity of a document (thus
taking values in [ m]), and a random variable y, indicating the identity of a word
in the dictionary (thus taking values in [ n]).
With this interpretation, the information bottleneck refers to the identity of
a clustering as another random variable, denoted C, that takes values in [ k]

(wherekwill be set by the method as well). Once we have formulated x,y,C
as random variables, we can use tools from information theory to express a
clustering objective. In particular, the information bottleneck objective is
min
p(C|x)I(x;C)−βI(C;y),
whereI(·;·) is the mutual information between two random variables,1βis a
1That is, given a probability function, pover the pairs ( x,C),

318 Clustering
parameter, and the minimization is over all possible probabilistic assignments of
points to clusters. Intuitively, we would like to achieve two contradictory goals.
On one hand, we would like the mutual information between the identity of
the document and the identity of the cluster to be as small as possible. This
reﬂects the fact that we would like a strong compression of the original data. On
the other hand, we would like high mutual information between the clustering
variable and the identity of the words, which reﬂects the goal that the “relevant”
information about the document (as reﬂected by the words that appear in the
document) is retained. This generalizes the classical notion of minimal suﬃcient
statistics2used in parametric statistics to arbitrary distributions.
Solving the optimization problem associated with the information bottleneck
principle is hard in the general case. Some of the proposed methods are similar
to the EM principle, which we will discuss in Chapter 24.

22.5 A High Level View of Clustering
So far, we have mainly listed various useful clustering tools. However, some fun-
damental questions remain unaddressed. First and foremost, what is clustering?
What is it that distinguishes a clustering algorithm from any arbitrary function
that takes an input space and outputs a partition of that space? Are there any
basic properties of clustering that are independent of any speciﬁc algorithm or
task?
One method for addressing such questions is via an axiomatic approach. There

task?
have been several attempts to provide an axiomatic deﬁnition of clustering. Let
us demonstrate this approach by presenting the attempt made by Kleinberg
(2003).
Consider a clustering function, F, that takes as input any ﬁnite domain X
with a dissimilarity function dover its pairs and returns a partition of X.
Consider the following three properties of such a function:
Scale Invariance (SI) For any domain set X, dissimilarity function d, and
anyα > 0, the following should hold: F(X,d) =F(X,αd) (where
(αd)(x,y)def=αd(x,y)).

(2003).
Richness (Ri) For any ﬁniteXand every partition C= (C1,...Ck) ofX(into
nonempty subsets) there exists some dissimilarity function doverXsuch
thatF(X,d) =C.
I(x;C) =∑
a∑
bp(a,b) log(p(a,b)
p(a)p(b))
, where the sum is over all values xcan take and all
valuesCcan take.
2A suﬃcient statistic is a function of the data which has the property of suﬃciency with
respect to a statistical model and its associated unknown parameter, meaning that “no
other statistic which can be calculated from the same sample provides any additional

I(x;C) =∑
a∑
p(a)p(b))
information as to the value of the parameter.” For example, if we assume that a variable is
distributed normally with a unit variance and an unknown expectation, then the average
function is a suﬃcient statistic.

22.5 A High Level View of Clustering 319
Consistency (Co) Ifdandd′are dissimilarity functions over X, such that
for everyx,y∈X, ifx,ybelong to the same cluster in F(X,d) then
d′(x,y)≤d(x,y) and ifx,ybelong to diﬀerent clusters in F(X,d) then
d′(x,y)≥d(x,y), thenF(X,d) =F(X,d′).
A moment of reﬂection reveals that the Scale Invariance is a very natural
requirement – it would be odd to have the result of a clustering function depend
on the units used to measure between-point distances. The Richness requirement
basically states that the outcome of the clustering function is fully controlled by
the function d, which is also a very intuitive feature. The third requirement,
Consistency, is the only requirement that refers to the basic (informal) deﬁnition
of clustering – we wish that similar points will be clustered together and that
dissimilar points will be separated to diﬀerent clusters, and therefore, if points
that already share a cluster become more similar, and points that are already

separated become even less similar to each other, the clustering function should
have even stronger “support” of its previous clustering decisions.
However, Kleinberg (2003) has shown the following “impossibility” result:
theorem 22.4 There exists no function, F, that satisﬁes all the three proper-
ties: Scale Invariance, Richness, and Consistency.
Proof Assume, by way of contradiction, that some Fdoes satisfy all three
properties. Pick some domain set Xwith at least three points. By Richness,

there must be some d1such thatF(X,d1) ={{x}:x∈X} and there also exists
somed2such thatF(X,d2)̸=F(X,d1).
Letα∈R+be such that for every x,y∈X,αd2(x,y)≥d1(x,y). Letd3=
αd2. ConsiderF(X,d3). By the Scale Invariance property of F, we should have
F(X,d3) =F(X,d2). On the other hand, since all distinct x,y∈X reside in
diﬀerent clusters w.r.t. F(X,d1), andd3(x,y)≥d1(x,y), the Consistency of F
implies that F(X,d3) =F(X,d1). This is a contradiction, since we chose d1,d2
so thatF(X,d2)̸=F(X,d1).

It is important to note that there is no single “bad property” among the three
properties. For every pair of the the three axioms, there exist natural clustering
functions that satisfy the two properties in that pair (one can even construct such
examples just by varying the stopping criteria for the Single Linkage clustering
function). On the other hand, Kleinberg shows that any clustering algorithm
that minimizes any center-based objective function inevitably fails the consis-

tency property (yet, the k-sum-of-in-cluster-distances minimization clustering
does satisfy Consistency).
The Kleinberg impossibility result can be easily circumvented by varying the
properties. For example, if one wishes to discuss clustering functions that have
a ﬁxed number-of-clusters parameter, then it is natural to replace Richness by
k-Richness (namely, the requirement that every partition of the domain into k
subsets is attainable by the clustering function). k-Richness, Scale Invariance

and Consistency all hold for the k-means clustering and are therefore consistent.

320 Clustering
Alternatively, one can relax the Consistency property. For example, say that two
clusterings C= (C1,...Ck) andC′= (C′
1,...C′
l) are compatible if for every
clustersCi∈CandC′
j∈C′, eitherCi⊆C′
jorC′
j⊆CiorCi∩C′
j=∅(it is
worthwhile noting that for every dendrogram, every two clusterings that are ob-
tained by trimming that dendrogram are compatible). “Reﬁnement Consistency”
is the requirement that, under the assumptions of the Consistency property, the
new clustering F(X,d′) is compatible with the old clustering F(X,d). Many
common clustering functions satisfy this requirement as well as Scale Invariance
and Richness. Furthermore, one can come up with many other, diﬀerent, prop-
erties of clustering functions that sound intuitive and desirable and are satisﬁed
by some common clustering functions.
There are many ways to interpret these results. We suggest to view it as indi-
cating that there is no “ideal” clustering function. Every clustering function will

inevitably have some “undesirable” properties. The choice of a clustering func-
tion for any given task must therefore take into account the speciﬁc properties
of that task. There is no generic clustering solution, just as there is no clas-
siﬁcation algorithm that will learn every learnable task (as the No-Free-Lunch
theorem shows). Clustering, just like classiﬁcation prediction, must take into
account some prior knowledge about the speciﬁc task at hand.
Clustering is an unsupervised learning problem, in which we wish to partition

a set of points into “meaningful” subsets. We presented several clustering ap-
proaches including linkage-based algorithms, the k-means family, spectral clus-
tering, and the information bottleneck. We discussed the diﬃculty of formalizing
the intuitive meaning of clustering.
22.7 Bibliographic Remarks
Thek-means algorithm is sometimes named Lloyd’s algorithm, after Stuart
Lloyd, who proposed the method in 1957. For a more complete overview of
spectral clustering we refer the reader to the excellent tutorial by Von Luxburg

(2007). The information bottleneck method was introduced by Tishby, Pereira
& Bialek (1999). For an additional discussion on the axiomatic approach see
Ackerman & Ben-David (2008).
22.8 Exercises
1.Suboptimality of k-Means: For every parameter t >1, show that there
exists an instance of the k-means problem for which the k-means algorithm

22.8 Exercises 321
(might) ﬁnd a solution whose k-means objective is at least t·OPT, where
OPT is the minimum k-means objective.
2.k-Means Might Not Necessarily Converge to a Local Minimum:
Show that the k-means algorithm might converge to a point which is not
a local minimum. Hint: Suppose that k= 2 and the sample points are
{1,2,3,4} ⊂Rsuppose we initialize the k-means with the centers {2,4};
and suppose we break ties in the deﬁnition of Ciby assigning ito be the
smallest value in argminj∥x−µj∥.
3. Given a metric space ( X,d), where|X|<∞, andk∈N, we would like to ﬁnd
a partition ofXintoC1,...,Ckwhich minimizes the expression
Gk−diam((X,d),(C1,...,Ck)) = max
j∈[d]diam(Cj),
where diam( Cj) = maxx,x′∈Cjd(x,x′) (we use the convention diam( Cj) = 0
if|Cj|<2).
Similarly to the k-means objective, it is NP-hard to minimize the k-
diam objective. Fortunately, we have a very simple approximation algorithm:
Initially, we pick some x∈X and setµ1=x. Then, the algorithm iteratively
sets
∀j∈{2,...,k}, µj= argmax
x∈Xmin

sets
x∈Xmin
i∈[j−1]d(x,µi).
Finally, we set
∀i∈[k], Ci={x∈X:i= argmin
j∈[k]d(x,µj)}.
Prove that the algorithm described is a 2-approximation algorithm. That
is, if we denote its output by ˆC1,..., ˆCk, and denote the optimal solution by
C∗
1,...,C∗
k, then,
Gk−diam((X,d),(ˆC1,..., ˆCk))≤2·Gk−diam((X,d),(C∗
1,...,C∗
k)).
Hint: Consider the point µk+1(in other words, the next center we would have
chosen, if we wanted k+ 1 clusters). Let r= minj∈[k]d(µj,µk+1). Prove the
following inequalities
Gk−diam((X,d),(ˆC1,..., ˆCk))≤2r
Gk−diam((X,d),(C∗
1,...,C∗
k))≥r.

C∗
1,...,C∗
k, then,
1,...,C∗
k)).
1,...,C∗
k))≥r.
4. Recall that a clustering function, F, is called Center-Based Clustering if, for
some monotonic function f:R+→R+, on every given input ( X,d),F(X,d)
is a clustering that minimizes the objective
i=1∑

322 Clustering
Prove that for every k>1 the k-diam clustering function deﬁned in the
previous exercise is not a center-based clustering function.
Hint: Given a clustering input ( X,d), with|X|>2, consider the eﬀect of
adding many close-by points to some (but not all) of the members of X, on
either the k-diam clustering or any given center-based clustering.
5. Recall that we discussed three clustering “properties”: Scale Invariance, Rich-
ness, and Consistency. Consider the Single Linkage clustering algorithm.
1. Find which of the three properties is satisﬁed by Single Linkage with the
Fixed Number of Clusters (any ﬁxed nonzero number) stopping rule.
2. Find which of the three properties is satisﬁed by Single Linkage with the
Distance Upper Bound (any ﬁxed nonzero upper bound) stopping rule.
3. Show that for any pair of these properties there exists a stopping criterion
for Single Linkage clustering, under which these two axioms are satisﬁed.
6. Given some number k, letk-Richness be the following requirement:

For any ﬁniteXand every partition C= (C1,...Ck)ofX(into nonempty subsets)
there exists some dissimilarity function doverXsuch thatF(X,d) =C.
Prove that, for every number k, there exists a clustering function that
satisﬁes the three properties: Scale Invariance, k-Richness, and Consistency.

23 Dimensionality Reduction
Dimensionality reduction is the process of taking data in a high dimensional
space and mapping it into a new space whose dimensionality is much smaller.
This process is closely related to the concept of (lossy) compression in infor-
mation theory. There are several reasons to reduce the dimensionality of the
data. First, high dimensional data impose computational challenges. Moreover,
in some situations high dimensionality might lead to poor generalization abili-
ties of the learning algorithm (for example, in Nearest Neighbor classiﬁers the
sample complexity increases exponentially with the dimension—see Chapter 19).
Finally, dimensionality reduction can be used for interpretability of the data, for
ﬁnding meaningful structure of the data, and for illustration purposes.
In this chapter we describe popular methods for dimensionality reduction. In
those methods, the reduction is performed by applying a linear transformation

to the original data. That is, if the original data is in Rdand we want to embed
it into Rn(n < d ) then we would like to ﬁnd a matrix W∈Rn,dthat induces
the mapping x↦→Wx. A natural criterion for choosing Wis in a way that will
enable a reasonable recovery of the original x. It is not hard to show that in
general, exact recovery of xfromWxis impossible (see Exercise 1).
The ﬁrst method we describe is called Principal Component Analysis (PCA).
In PCA, both the compression and the recovery are performed by linear transfor-

mations and the method ﬁnds the linear transformations for which the diﬀerences
between the recovered vectors and the original vectors are minimal in the least
squared sense.
Next, we describe dimensionality reduction using random matrices W. We
derive an important lemma, often called the “Johnson-Lindenstrauss lemma,”
which analyzes the distortion caused by such a random dimensionality reduction
Last, we show how one can reduce the dimension of all sparse vectors using
again a random matrix. This process is known as Compressed Sensing. In this

case, the recovery process is nonlinear but can still be implemented eﬃciently
using linear programming.
We conclude by underscoring the underlying “prior assumptions” behind PCA
and compressed sensing, which can help us understand the merits and pitfalls of
the two methods.

324 Dimensionality Reduction
23.1 Principal Component Analysis (PCA)
Letx1,...,xmbemvectors in Rd. We would like to reduce the dimensional-
ity of these vectors using a linear transformation. A matrix W∈Rn,d, where
n<d , induces a mapping x↦→Wx, whereWx∈Rnis the lower dimensionality
representation of x. Then, a second matrix U∈Rd,ncan be used to (approxi-
mately) recover each original vector xfrom its compressed version. That is, for
a compressed vector y=Wx, where yis in the low dimensional space Rn, we
can construct ˜x=Uy, so that ˜xis the recovered version of xand resides in the
original high dimensional space Rd.
In PCA, we ﬁnd the compression matrix Wand the recovering matrix Uso
that the total squared distance between the original and recovered vectors is
minimal; namely, we aim at solving the problem
argmin
W∈Rn,d,U∈Rd,nm∑
i=1∥xi−UWxi∥2
2. (23.1)
To solve this problem we ﬁrst show that the optimal solution takes a speciﬁc
form.
lemma 23.1 Let(U,W )be a solution to Equation (23.1). Then the columns of

argmin
2. (23.1)
form.
Uare orthonormal (namely, U⊤Uis the identity matrix of Rn) andW=U⊤.
Proof Fix anyU,W and consider the mapping x↦→UWx. The range of this
mapping,R={UWx:x∈Rd}, is anndimensional linear subspace of Rd. Let
V∈Rd,nbe a matrix whose columns form an orthonormal basis of this subspace,
namely, the range of VisRandV⊤V=I. Therefore, each vector in Rcan be
written asVywhere y∈Rn. For every x∈Rdandy∈Rnwe have
∥x−Vy∥2
2=∥x∥2+y⊤V⊤Vy−2y⊤V⊤x=∥x∥2+∥y∥2−2y⊤(V⊤x),
where we used the fact that V⊤Vis the identity matrix of Rn. Minimizing the

∥x−Vy∥2
preceding expression with respect to yby comparing the gradient with respect
toyto zero gives that y=V⊤x. Therefore, for each xwe have that
VV⊤x= argmin
˜x∈R∥x−˜x∥2
2.
In particular this holds for x1,...,xmand therefore we can replace U,W by
V,V⊤and by that do not increase the objective
m∑
2≥m∑
i=1∥xi−VV⊤xi∥2
2.
Since this holds for every U,W the proof of the lemma follows.
On the basis of the preceding lemma, we can rewrite the optimization problem
given in Equation (23.1) as follows:
argmin
U∈Rd,n:U⊤U=Im∑
i=1∥xi−UU⊤xi∥2
2. (23.2)

23.1 Principal Component Analysis (PCA) 325
We further simplify the optimization problem by using the following elementary
algebraic manipulations. For every x∈Rdand a matrix U∈Rd,nsuch that
U⊤U=Iwe have
∥x−UU⊤x∥2=∥x∥2−2x⊤UU⊤x+x⊤UU⊤UU⊤x
=∥x∥2−x⊤UU⊤x
=∥x∥2−trace(U⊤xx⊤U), (23.3)
where the trace of a matrix is the sum of its diagonal entries. Since the trace is
a linear operator, this allows us to rewrite Equation (23.2) as follows:
argmax
U∈Rd,n:U⊤U=Itrace(
U⊤m∑
i=1xix⊤
iU)
. (23.4)
LetA=∑m
i=1xix⊤
i. The matrix Ais symmetric and therefore it can be
written using its spectral decomposition as A=VDV⊤, whereDis diagonal and
V⊤V=VV⊤=I. Here, the elements on the diagonal of Dare the eigenvalues of
Aand the columns of Vare the corresponding eigenvectors. We assume without
loss of generality that D1,1≥D2,2≥···≥Dd,d. SinceAis positive semideﬁnite
it also holds that Dd,d≥0. We claim that the solution to Equation (23.4) is
the matrix Uwhose columns are the neigenvectors of Acorresponding to the
largestneigenvalues.

theorem 23.2 Letx1,...,xmbe arbitrary vectors in Rd, letA=∑m
i=1xix⊤
i,
and let u1,...,unbeneigenvectors of the matrix Acorresponding to the largest
neigenvalues of A. Then, the solution to the PCA optimization problem given
in Equation (23.1) is to set Uto be the matrix whose columns are u1,...,un
and to setW=U⊤.
Proof LetVDV⊤be the spectral decomposition of A. Fix some matrix U∈Rd,n
with orthonormal columns and let B=V⊤U. Then, VB=VV⊤U=U. It
follows that
U⊤AU=B⊤V⊤VDV⊤VB=B⊤DB,
and therefore
trace(U⊤AU) =d∑
j=1Dj,jn∑
i=1B2
j,i.

i=1xix⊤
i,
j=1Dj,jn∑
i=1B2
j,i.
Note thatB⊤B=U⊤VV⊤U=U⊤U=I. Therefore, the columns of Bare
also orthonormal, which implies that∑d
j=1∑n
i=1B2
j,i=n. In addition, let ˜B∈
Rd,dbe a matrix such that its ﬁrst ncolumns are the columns of Band in
addition ˜B⊤˜B=I. Then, for every jwe have∑d
i=1˜B2
j,i= 1, which implies that∑n
i=1B2
j,i≤1. It follows that:
trace(U⊤AU)≤ max
β∈[0,1]d:∥β∥1≤nd∑
j=1Dj,jβj.

326 Dimensionality Reduction
It is not hard to verify (see Exercise 2) that the right-hand side equals to∑n
j=1Dj,j. We have therefore shown that for every matrix U∈Rd,nwith or-
thonormal columns it holds that trace( U⊤AU)≤∑n
j=1Dj,j. On the other hand,
if we setUto be the matrix whose columns are the nleading eigenvectors of A
we obtain that trace( U⊤AU) =∑n
j=1Dj,j, and this concludes our proof.
Remark 23.1 The proof of Theorem 23.2 also tells us that the value of the
objective of Equation (23.4) is∑n
i=1Di,i. Combining this with Equation (23.3)
and noting that∑m
i=1∥xi∥2= trace(A) =∑d
i=1Di,iwe obtain that the optimal
objective value of Equation (23.1) is∑d
i=n+1Di,i.
Remark 23.2 It is a common practice to “center” the examples before applying
PCA. That is, we ﬁrst calculate µ=1
m∑m
i=1xiand then apply PCA on the
vectors ( x1−µ),..., (xm−µ). This is also related to the interpretation of PCA
as variance maximization (see Exercise 4).
23.1.1 A More Eﬃcient Solution for the Case d≫m

m∑m
In some situations the original dimensionality of the data is much larger than
the number of examples m. The computational complexity of calculating the
PCA solution as described previously is O(d3) (for calculating eigenvalues of A)
plusO(md2) (for constructing the matrix A). We now show a simple trick that
enables us to calculate the PCA solution more eﬃciently when d≫m.
Recall that the matrix Ais deﬁned to be∑m
i=1xix⊤
i. It is convenient to rewrite
A=X⊤XwhereX∈Rm,dis a matrix whose ith row is x⊤
i. Consider the

i=1xix⊤
matrixB=XX⊤. That is,B∈Rm,mis the matrix whose i,jelement equals
⟨xi,xj⟩. Suppose that uis an eigenvector of B: That is,Bu=λufor some
λ∈R. Multiplying the equality by X⊤and using the deﬁnition of Bwe obtain
X⊤XX⊤u=λX⊤u. But, using the deﬁnition of A, we get that A(X⊤u) =
λ(X⊤u). Thus,X⊤u
∥X⊤u∥is an eigenvector of Awith eigenvalue of λ.
We can therefore calculate the PCA solution by calculating the eigenvalues of
Binstead ofA. The complexity is O(m3) (for calculating eigenvalues of B) and
m2d(for constructing the matrix B).

Remark 23.3 The previous discussion also implies that to calculate the PCA
solution we only need to know how to calculate inner products between vectors.
This enables us to calculate PCA implicitly even when dis very large (or even
inﬁnite) using kernels, which yields the kernel PCA algorithm.
23.1.2 Implementation and Demonstration
A pseudocode of PCA is given in the following.

23.1 Principal Component Analysis (PCA) 327
−1.5 −1 −0.5 0 0.5 1 1.5−1.5−1−0.500.511.5
Figure 23.1 A set of vectors in R2(blue x’s) and their reconstruction after
dimensionality reduction to R1using PCA (red circles).
PCA
input
A matrix of mexamplesX∈Rm,d
number of components n
if(m>d )
A=X⊤X
Letu1,...,unbe the eigenvectors of Awith largest eigenvalues
else
B=XX⊤
Letv1,...,vnbe the eigenvectors of Bwith largest eigenvalues
fori= 1,...,n setui=1
∥X⊤vi∥X⊤vi
output :u1,...,un
To illustrate how PCA works, let us generate vectors in R2that approximately
reside on a line, namely, on a one dimensional subspace of R2. For example,
suppose that each example is of the form ( x,x+y) wherexis chosen uniformly
at random from [−1,1] andyis sampled from a Gaussian distribution with mean
0 and standard deviation of 0 .1. Suppose we apply PCA on this data. Then, the
eigenvector corresponding to the largest eigenvalue will be close to the vector
(1/√
2,1/√
2). When projecting a point ( x,x+y) on this principal component

(1/√
2,1/√
we will obtain the scalar2x+y√
2. The reconstruction of the original vector will be
((x+y/2),(x+y/2)). In Figure 23.1 we depict the original versus reconstructed
data.
Next, we demonstrate the eﬀectiveness of PCA on a data set of faces. We
extracted images of faces from the Yale data set (Georghiades, Belhumeur &
Kriegman 2001). Each image contains 50 ×50 = 2500 pixels; therefore the original
dimensionality is very high.

328 Dimensionality Reduction
x xxxxxxoo o oooo
***
****+++++++
Figure 23.2 Images of faces extracted from the Yale data set. Top-Left: the original
images in R50x50. Top-Right: the images after dimensionality reduction to R10and
reconstruction. Middle row: an enlarged version of one of the images before and after
PCA. Bottom: The images after dimensionality reduction to R2. The diﬀerent marks
indicate diﬀerent individuals.
Some images of faces are depicted on the top-left side of Figure 23.2. Using
PCA, we reduced the dimensionality to R10and reconstructed back to the orig-
inal dimension, which is 502. The resulting reconstructed images are depicted
on the top-right side of Figure 23.2. Finally, on the bottom of Figure 23.2 we
depict a 2 dimensional representation of the images. As can be seen, even from a
2 dimensional representation of the images we can still roughly separate diﬀerent
individuals.

23.2 Random Projections
In this section we show that reducing the dimension by using a random linear
transformation leads to a simple compression scheme with a surprisingly low
distortion. The transformation x↦→Wx, whenWis a random matrix, is often
referred to as a random projection. In particular, we provide a variant of a famous
lemma due to Johnson and Lindenstrauss, showing that random projections do
not distort Euclidean distances too much.
Letx1,x2be two vectors in Rd. A matrix Wdoes not distort too much the
distance between x1andx2if the ratio
∥Wx1−Wx2∥
∥x1−x2∥
is close to 1. In other words, the distances between x1andx2before and after
the transformation are almost the same. To show that ∥Wx1−Wx2∥is not too
far away from∥x1−x2∥it suﬃces to show that Wdoes not distort the norm of
the diﬀerence vector x=x1−x2. Therefore, from now on we focus on the ratio
∥Wx∥
∥x∥.
We start with analyzing the distortion caused by applying a random projection
to a single vector.

∥Wx1−Wx2∥
∥x1−x2∥
∥Wx∥
∥x∥.
lemma 23.3 Fix some x∈Rd. LetW∈Rn,dbe a random matrix such that
eachWi,jis an independent normal random variable. Then, for every ϵ∈(0,3)
we have
P[⏐⏐⏐⏐⏐∥(1/√n)Wx∥2
∥x∥2−1⏐⏐⏐⏐⏐>ϵ]
≤2e−ϵ2n/6.
Proof Without loss of generality we can assume that ∥x∥2= 1. Therefore, an
equivalent inequality is
P[
(1−ϵ)n≤∥Wx∥2≤(1 +ϵ)n]
≥1−2e−ϵ2n/6.
Letwibe theith row ofW. The random variable ⟨wi,x⟩is a weighted sum of
dindependent normal random variables and therefore it is normally distributed
with zero mean and variance∑
jx2
j=∥x∥2= 1. Therefore, the random vari-
able∥Wx∥2=∑n

we have
P[
jx2
i=1(⟨wi,x⟩)2has aχ2
ndistribution. The claim now follows
directly from a measure concentration property of χ2random variables stated in
Lemma B.12 given in Section B.7.
The Johnson-Lindenstrauss lemma follows from this using a simple union
bound argument.
lemma 23.4 (Johnson-Lindenstrauss Lemma) LetQbe a ﬁnite set of vectors
inRd. Letδ∈(0,1)andnbe an integer such that
ϵ=√
6 log(2|Q|/δ)
n≤3.

330 Dimensionality Reduction
Then, with probability of at least 1−δover a choice of a random matrix W∈Rn,d
such that each element of Wis distributed normally with zero mean and variance
of1/nwe have
sup
x∈Q⏐⏐⏐⏐∥Wx∥2
∥x∥2−1⏐⏐⏐⏐<ϵ.
Proof Combining Lemma 23.3 and the union bound we have that for every
ϵ∈(0,3):
P[
sup
∥x∥2−1⏐⏐⏐⏐>ϵ]
≤2|Q|e−ϵ2n/6.
Letδdenote the right-hand side of the inequality; thus we obtain that
ϵ=√
n.
Interestingly, the bound given in Lemma 23.4 does not depend on the original
dimension of x. In fact, the bound holds even if xis in an inﬁnite dimensional
Hilbert space.
23.3 Compressed Sensing
Compressed sensing is a dimensionality reduction technique which utilizes a prior
assumption that the original vector is sparse in some basis. To motivate com-
pressed sensing, consider a vector x∈Rdthat has at most snonzero elements.
That is,
∥x∥0def=|{i:xi̸= 0}|≤s.
Clearly, we can compress xby representing it using s(index,value) pairs. Fur-

That is,
thermore, this compression is lossless – we can reconstruct xexactly from the s
(index,value) pairs. Now, lets take one step forward and assume that x=Uα,
whereαis a sparse vector, ∥α∥0≤s, andUis a ﬁxed orthonormal matrix. That
is,xhas a sparse representation in another basis. It turns out that many nat-
ural vectors are (at least approximately) sparse in some representation. In fact,
this assumption underlies many modern compression schemes. For example, the
JPEG-2000 format for image compression relies on the fact that natural images

are approximately sparse in a wavelet basis.
Can we still compress xinto roughly snumbers? Well, one simple way to do
this is to multiply xbyU⊤, which yields the sparse vector α, and then represent
αby itss(index,value) pairs. However, this requires us ﬁrst to “sense” x, to
store it, and then to multiply it by U⊤. This raises a very natural question: Why
go to so much eﬀort to acquire all the data when most of what we get will be
thrown away? Cannot we just directly measure the part that will not end up
being thrown away?

23.3 Compressed Sensing 331
Compressed sensing is a technique that simultaneously acquires and com-
presses the data. The key result is that a random linear transformation can
compress xwithout losing information. The number of measurements needed is
order ofslog(d). That is, we roughly acquire only the important information
about the signal. As we will see later, the price we pay is a slower reconstruction
phase. In some situations, it makes sense to save time in compression even at
the price of a slower reconstruction. For example, a security camera should sense
and compress a large amount of images while most of the time we do not need to
decode the compressed data at all. Furthermore, in many practical applications,
compression by a linear transformation is advantageous because it can be per-
formed eﬃciently in hardware. For example, a team led by Baraniuk and Kelly
has proposed a camera architecture that employs a digital micromirror array to

perform optical calculations of a linear transformation of an image. In this case,
obtaining each compressed measurement is as easy as obtaining a single raw
measurement. Another important application of compressed sensing is medical
imaging, in which requiring fewer measurements translates to less radiation for
the patient.
Informally, the main premise of compressed sensing is the following three “sur-
prising” results:
1. It is possible to reconstruct any sparse signal fully if it was compressed by

x↦→Wx, whereWis a matrix which satisﬁes a condition called the Re-
stricted Isoperimetric Property (RIP). A matrix that satisﬁes this property is
guaranteed to have a low distortion of the norm of any sparse representable
vector.
2. The reconstruction can be calculated in polynomial time by solving a linear
program.
3. A random n×dmatrix is likely to satisfy the RIP condition provided that n
is greater than an order of slog(d).
Formally,
definition 23.5 (RIP) A matrix W∈Rn,dis (ϵ,s)-RIP if for all x̸= 0 s.t.

vector.
program.
Formally,
∥x∥0≤swe have
⏐⏐⏐⏐∥Wx∥2
2
∥x∥2
2−1⏐⏐⏐⏐≤ϵ.
The ﬁrst theorem establishes that RIP matrices yield a lossless compression
scheme for sparse vectors. It also provides a (noneﬃcient) reconstruction scheme.
theorem 23.6 Letϵ<1and letWbe a(ϵ,2s)-RIP matrix. Let xbe a vector
s.t.∥x∥0≤s, lety=Wxbe the compression of x, and let
˜x∈argmin
v:Wv=y∥v∥0
be a reconstructed vector. Then, ˜x=x.

332 Dimensionality Reduction
Proof We assume, by way of contradiction, that ˜x̸=x. Since xsatisﬁes the
constraints in the optimization problem that deﬁnes ˜xwe clearly have that
∥˜x∥0≤∥x∥0≤s. Therefore,∥x−˜x∥0≤2sand we can apply the RIP in-
equality on the vector x−˜x. But, since W(x−˜x) =0we get that|0−1|≤ϵ,
which leads to a contradiction.
The reconstruction scheme given in Theorem 23.6 seems to be noneﬃcient
because we need to minimize a combinatorial objective (the sparsity of v). Quite
surprisingly, it turns out that we can replace the combinatorial objective, ∥v∥0,
with a convex objective, ∥v∥1, which leads to a linear programming problem that
can be solved eﬃciently. This is stated formally in the following theorem.
theorem 23.7 Assume that the conditions of Theorem 23.6 holds and that
ϵ<1
1+√
2. Then,
x= argmin
v:Wv=y∥v∥0= argmin
v:Wv=y∥v∥1.
In fact, we will prove a stronger result, which holds even if xis not a sparse
vector.
theorem 23.8 Letϵ<1
1+√
2and letWbe a (ϵ,2s)-RIP matrix. Let xbe an

ϵ<1
1+√
2. Then,
x= argmin
vector.
1+√
arbitrary vector and denote
xs∈argmin
v:∥v∥0≤s∥x−v∥1.
That is, xsis the vector which equals xon theslargest elements of xand equals
0elsewhere. Let y=Wxbe the compression of xand let
x⋆∈argmin
v:Wv=y∥v∥1
be the reconstructed vector. Then,
∥x⋆−x∥2≤21 +ρ
1−ρs−1/2∥x−xs∥1,
whereρ=√
2ϵ/(1−ϵ).
Note that in the special case that x=xswe get an exact recovery, x⋆=x, so
Theorem 23.7 is a special case of Theorem 23.8. The proof of Theorem 23.8 is
given in Section 23.3.1.
Finally, the third result tells us that random matrices with n≥Ω(slog(d)) are

x⋆∈argmin
whereρ=√
2ϵ/(1−ϵ).
likely to be RIP. In fact, the theorem shows that multiplying a random matrix
by an orthonormal matrix also provides an RIP matrix. This is important for
compressing signals of the form x=Uαwhere xis not sparse but αis sparse.
In that case, if Wis a random matrix and we compress using y=Wxthen this
is the same as compressing αbyy= (WU)αand sinceWU is also RIP we can
reconstructα(and thus also x) from y.

23.3 Compressed Sensing 333
theorem 23.9 LetUbe an arbitrary ﬁxed d×dorthonormal matrix, let ϵ,δ
be scalars in (0,1), letsbe an integer in [d], and letnbe an integer that satisﬁes
n≥100slog(40d/(δϵ))
ϵ2.
LetW∈Rn,dbe a matrix s.t. each element of Wis distributed normally with
zero mean and variance of 1/n. Then, with proabability of at least 1−δover the
choice ofW, the matrix WU is(ϵ,s)-RIP.
23.3.1 Proofs*
Proof of Theorem 23.8
We follow a proof due to Cand` es (2008).
Leth=x⋆−x. Given a vector vand a set of indices Iwe denote by vIthe
vector whose ith element is viifi∈Iand 0 otherwise.
The ﬁrst trick we use is to partition the set of indices [ d] ={1,...,d}into
disjoint sets of size s. That is, we will write [ d] =T0·∪T1·∪T2...Td/s−1where
for alli,|Ti|=s, and we assume for simplicity that d/sis an integer. We deﬁne
the partition as follows. In T0we put the sindices corresponding to the slargest
elements in absolute values of x(ties are broken arbitrarily). Let Tc
0= [d]\T0.

0= [d]\T0.
Next,T1will be thesindices corresponding to the slargest elements in absolute
value of hTc
0. LetT0,1=T0∪T1andTc
0,1= [d]\T0,1. Next,T2will correspond to
theslargest elements in absolute value of hTc
0,1. And, we will construct T3,T4,...
in the same way.
To prove the theorem we ﬁrst need the following lemma, which shows that
RIP also implies approximate orthogonality.
lemma 23.10 LetWbe an (ϵ,2s)-RIP matrix. Then, for any two disjoint sets
I,J, both of size at most s, and for any vector uwe have that⟨WuI,WuJ⟩≤
ϵ∥uI∥2∥uJ∥2.
Proof W.l.o.g. assume∥uI∥2=∥uJ∥2= 1.

⟨WuI,WuJ⟩=∥WuI+WuJ∥2
2−∥WuI−WuJ∥2
2
4.
But, since|J∪I|≤2swe get from the RIP condition that ∥WuI+WuJ∥2
2≤
(1+ϵ)(∥uI∥2
2+∥uJ∥2
2) = 2(1+ϵ) and that−∥WuI−WuJ∥2
2≤−(1−ϵ)(∥uI∥2
2+
∥uJ∥2
2) =−2(1−ϵ), which concludes our proof.
We are now ready to prove the theorem. Clearly,
∥h∥2=∥hT0,1+hTc
0,1∥2≤∥hT0,1∥2+∥hTc
0,1∥2. (23.5)
To prove the theorem we will show the following two claims:
Claim 1:.∥hTc
0,1∥2≤∥hT0∥2+ 2s−1/2∥x−xs∥1.
Claim 2:.∥hT0,1∥2≤2ρ
1−ρs−1/2∥x−xs∥1.

334 Dimensionality Reduction
Combining these two claims with Equation (23.5) we get that
∥h∥2≤∥hT0,1∥2+∥hTc
0,1∥2≤2∥hT0,1∥2+ 2s−1/2∥x−xs∥1
≤2(
2ρ
1−ρ+ 1)
s−1/2∥x−xs∥1
= 21 +ρ
and this will conclude our proof.
Proving Claim 1:
To prove this claim we do not use the RIP condition at all but only use the fact
thatx⋆minimizes the ℓ1norm. Take j >1. For each i∈Tjandi′∈Tj−1we
have that|hi|≤|hi′|. Therefore,∥hTj∥∞≤∥hTj−1∥1/s. Thus,
∥hTj∥2≤s1/2∥hTj∥∞≤s−1/2∥hTj−1∥1.
Summing this over j= 2,3,...and using the triangle inequality we obtain that
∥hTc
0,1∥2≤∑
j≥2∥hTj∥2≤s−1/2∥hTc
0∥1 (23.6)
Next, we show that ∥hTc
0∥1cannot be large. Indeed, from the deﬁnition of x⋆
we have that∥x∥1≥∥x⋆∥1=∥x+h∥1. Thus, using the triangle inequality we
∥x∥1≥∥x+h∥1=∑
i∈T0|xi+hi|+∑
i∈Tc
0|xi+hi|≥∥xT0∥1−∥hT0∥1+∥hTc
0∥1−∥xTc
0∥1
(23.7)
and since∥xTc
0∥1=∥x−xs∥1=∥x∥1−∥xT0∥1we get that
∥hTc
0∥1≤∥hT0∥1+ 2∥xTc
0∥1. (23.8)
Combining this with Equation (23.6) we get that
∥hTc
0,1∥2≤s−1/2(
∥hT0∥1+ 2∥xTc
0∥1)

∥hTc
0,1∥2≤∑
0∥1 (23.6)
i∈Tc
0∥1−∥xTc
0∥1
(23.7)
∥hTc
∥hTc
0∥1)
≤∥hT0∥2+ 2s−1/2∥xTc
0∥1,
which concludes the proof of claim 1.
Proving Claim 2:
For the second claim we use the RIP condition to get that
(1−ϵ)∥hT0,1∥2
2≤∥WhT0,1∥2
2. (23.9)
SinceWhT0,1=Wh−∑
j≥2WhTj=−∑
j≥2WhTjwe have that
∥WhT0,1∥2
2=−∑
j≥2⟨WhT0,1,WhTj⟩=−∑
j≥2⟨WhT0+WhT1,WhTj⟩.
From the RIP condition on inner products we obtain that for all i∈{1,2}and
j≥2 we have
|⟨WhTi,WhTj⟩|≤ϵ∥hTi∥2∥hTj∥2.

23.3 Compressed Sensing 335
Since∥hT0∥2+∥hT1∥2≤√
2∥hT0,1∥2we therefore get that
∥WhT0,1∥2
2≤√
2ϵ∥hT0,1∥2∑
j≥2∥hTj∥2.
Combining this with Equation (23.6) and Equation (23.9) we obtain
2≤√
2ϵ∥hT0,1∥2s−1/2∥hTc
0∥1.
Rearranging the inequality gives
∥hT0,1∥2≤√
2ϵ
1−ϵs−1/2∥hTc
0∥1.
Finally, using Equation (23.8) we get that
∥hT0,1∥2≤ρs−1/2(∥hT0∥1+ 2∥xTc
0∥1)≤ρ∥hT0∥2+ 2ρs−1/2∥xTc
0∥1,
but since∥hT0∥2≤∥hT0,1∥2this implies
∥hT0,1∥2≤2ρ
1−ρs−1/2∥xTc
0∥1,
which concludes the proof of the second claim.
Proof of Theorem 23.9
To prove the theorem we follow an approach due to (Baraniuk, Davenport, De-
Vore & Wakin 2008). The idea is to combine the Johnson-Lindenstrauss (JL)
lemma with a simple covering argument.
We start with a covering property of the unit ball.
lemma 23.11 Letϵ∈(0,1). There exists a ﬁnite set Q⊂Rdof size|Q|≤(3
ϵ)d
such that
sup
x:∥x∥≤1min
v∈Q∥x−v∥ ≤ϵ.
Proof Letkbe an integer and let
Q′={x∈Rd:∀j∈[d],∃i∈{−k,−k+ 1,...,k}s.t. xj=i
k}.

0∥1,
ϵ)d
such that
sup
k}.
Clearly,|Q′|= (2k+ 1)d. We shall set Q=Q′∩B2(1), whereB2(1) is the unit
ℓ2ball of Rd. Since the points in Q′are distributed evenly on the unit ℓ∞ball,
the size ofQis the size of Q′times the ratio between the volumes of the unit ℓ2
andℓ∞balls. The volume of the ℓ∞ball is 2dand the volume of B2(1) is
πd/2
Γ(1 +d/2).
For simplicity, assume that dis even and therefore
Γ(1 +d/2) = (d/2)!≥(
d/2
e)d/2
,

336 Dimensionality Reduction
where in the last inequality we used Stirling’s approximation. Overall we obtained
that
|Q|≤(2k+ 1)d(π/e)d/2(d/2)−d/22−d. (23.10)
Now lets specify k. For each x∈B2(1) let v∈Qbe the vector whose ith element
is sign(xi)⌊|xi|k⌋/k. Then, for each element we have that |xi−vi|≤1/kand
thus
∥x−v∥≤√
d
k.
To ensure that the right-hand side will be at most ϵwe shall set k=⌈√
d/ϵ⌉.
Plugging this value into Equation (23.10) we conclude that
|Q|≤(3√
d/(2ϵ))d(π/e)d/2(d/2)−d/2=(
3
ϵ√
π
2e)d
≤(3
ϵ)d.
Letxbe a vector that can be written as x=UαwithUbeing some orthonor-
mal matrix and∥α∥0≤s. Combining the earlier covering property and the JL
lemma (Lemma 23.4) enables us to show that a random Wwill not distort any
suchx.
lemma 23.12 LetUbe an orthonormal d×dmatrix and let I⊂[d]be a set
of indices of size |I|=s. LetSbe the span of{Ui:i∈I}, whereUiis theith
column ofU. Letδ∈(0,1),ϵ∈(0,1), andn∈Nsuch that
n≥24log(2/δ) +slog(12/ϵ)
ϵ2.

|Q|≤(3√
3
ϵ√
π
2e)d
≤(3
ϵ)d.
suchx.
ϵ2.
such that each element of Wis independently distributed according to N(0,1/n),
we have
sup
x∈S⏐⏐⏐⏐∥Wx∥
∥x∥−1⏐⏐⏐⏐<ϵ.
Proof It suﬃces to prove the lemma for all x∈Swith∥x∥= 1. We can write
x=UIαwhereα∈Rs,∥α∥2= 1, andUIis the matrix whose columns are
{Ui:i∈I}. Using Lemma 23.11 we know that there exists a set Qof size
|Q|≤(12/ϵ)ssuch that
sup
α:∥α∥=1min
v∈Q∥α−v∥≤(ϵ/4).
But sinceUis orthogonal we also have that
sup
v∈Q∥UIα−UIv∥≤(ϵ/4).

we have
sup
sup
sup
Applying Lemma 23.4 on the set {UIv:v∈Q}we obtain that for nsatisfying

23.3 Compressed Sensing 337
the condition given in the lemma, the following holds with probability of at least
1−δ:
sup
v∈Q⏐⏐⏐⏐∥WUIv∥2
∥UIv∥2−1⏐⏐⏐⏐≤ϵ/2,
This also implies that
sup
v∈Q⏐⏐⏐⏐∥WUIv∥
∥UIv∥−1⏐⏐⏐⏐≤ϵ/2.
Letabe the smallest number such that
∀x∈S,∥Wx∥
∥x∥≤1 +a.
Clearlya<∞. Our goal is to show that a≤ϵ. This follows from the fact that
for any x∈Sof unit norm there exists v∈Qsuch that∥x−UIv∥≤ϵ/4 and
therefore
∥Wx∥≤∥WUIv∥+∥W(x−UIv)∥≤1 +ϵ/2 + (1 +a)ϵ/4.
Thus,
∀x∈S,∥Wx∥
∥x∥≤1 + (ϵ/2 + (1 +a)ϵ/4).
But the deﬁnition of aimplies that
a≤ϵ/2 + (1 +a)ϵ/4⇒a≤ϵ/2 +ϵ/4
1−ϵ/4≤ϵ.
This proves that for all x∈Swe have∥Wx∥
∥x∥−1≤ϵ. The other side follows from
this as well since
∥Wx∥≥∥WUIv∥−∥W(x−UIv)∥≥1−ϵ/2−(1 +ϵ)ϵ/4≥1−ϵ.
The preceding lemma tells us that for x∈Sof unit norm we have
(1−ϵ)≤∥Wx∥≤(1 +ϵ),
which implies that
(1−2ϵ)≤∥Wx∥2≤(1 + 3ϵ).
The proof of Theorem 23.9 follows from this by a union bound over all choices
ofI.

338 Dimensionality Reduction
23.4 PCA or Compressed Sensing?
Suppose we would like to apply a dimensionality reduction technique to a given
set of examples. Which method should we use, PCA or compressed sensing? In
this section we tackle this question, by underscoring the underlying assumptions
behind the two methods.
It is helpful ﬁrst to understand when each of the methods can guarantee per-
fect recovery. PCA guarantees perfect recovery whenever the set of examples is
contained in an ndimensional subspace of Rd. Compressed sensing guarantees
perfect recovery whenever the set of examples is sparse (in some basis). On the
basis of these observations, we can describe cases in which PCA will be better
than compressed sensing and vice versa.
As a ﬁrst example, suppose that the examples are the vectors of the standard
basis of Rd, namely, e1,...,ed, where each eiis the all zeros vector except 1 in the
ith coordinate. In this case, the examples are 1-sparse. Hence, compressed sensing

will yield a perfect recovery whenever n≥Ω(log(d)). On the other hand, PCA
will lead to poor performance, since the data is far from being in an ndimensional
subspace, as long as n < d . Indeed, it is easy ro verify that in such a case, the
averaged recovery error of PCA (i.e., the objective of Equation (23.1) divided by
m) will be (d−n)/d, which is larger than 1 /2 whenever n≤d/2.
We next show a case where PCA is better than compressed sensing. Consider
mexamples that are exactly on an ndimensional subspace. Clearly, in such a

case, PCA will lead to perfect recovery. As to compressed sensing, note that
the examples are n-sparse in any orthonormal basis whose ﬁrst nvectors span
the subspace. Therefore, compressed sensing would also work if we will reduce
the dimension to Ω( nlog(d)). However, with exactly ndimensions, compressed
sensing might fail. PCA has also better resilience to certain types of noise. See
(Chang, Weiss & Freeman 2009) for a discussion.
23.5 Summary
We introduced two methods for dimensionality reduction using linear transfor-

mations: PCA and random projections. We have shown that PCA is optimal in
the sense of averaged squared reconstruction error, if we restrict the reconstruc-
tion procedure to be linear as well. However, if we allow nonlinear reconstruction,
PCA is not necessarily the optimal procedure. In particular, for sparse data, ran-
dom projections can signiﬁcantly outperform PCA. This fact is at the heart of
the compressed sensing method.

23.6 Bibliographic Remarks
PCA is equivalent to best subspace approximation using singular value decom-
position (SVD). The SVD method is described in Appendix C. SVD dates back
to Eugenio Beltrami (1873) and Camille Jordan (1874). It has been rediscovered
many times. In the statistical literature, it was introduced by Pearson (1901). Be-
sides PCA and SVD, there are additional names that refer to the same idea and
are being used in diﬀerent scientiﬁc communities. A few examples are the Eckart-
Young theorem (after Carl Eckart and Gale Young who analyzed the method in
1936), the Schmidt-Mirsky theorem, factor analysis, and the Hotelling transform.
Compressed sensing was introduced in Donoho (2006) and in (Candes & Tao
2005). See also Candes (2006).
23.7 Exercises
1. In this exercise we show that in the general case, exact recovery of a linear
compression scheme is impossible.
1. letA∈Rn,dbe an arbitrary compression matrix where n≤d−1. Show

that there exists u,v∈Rn,u̸=vsuch thatAu=Av.
2. Conclude that exact recovery of a linear compression scheme is impossible.
2. Letα∈Rdsuch thatα1≥α2≥···≥αd≥0. Show that
max
j=1αjβj=n∑
j=1αj.
Hint: Take every vector β∈[0,1]dsuch that∥β∥1≤n. Letibe the minimal
index for which βi<1. Ifi=n+ 1 we are done. Otherwise, show that we can
increaseβi, while possibly decreasing βjfor somej >i , and obtain a better
solution. This will imply that the optimal solution is to set βi= 1 fori≤n
andβi= 0 fori>n .

max
j=1αj.
3.Kernel PCA: In this exercise we show how PCA can be used for construct-
ing nonlinear dimensionality reduction on the basis of the kernel trick (see
Chapter 16).
LetXbe some instance space and let S={x1,...,xm}be a set of points
inX. Consider a feature mapping ψ:X→V, whereVis some Hilbert space
(possibly of inﬁnite dimension). Let K:X×X be a kernel function, that is,
k(x,x′) =⟨ψ(x),ψ(x′)⟩. Kernel PCA is the process of mapping the elements
inSintoVusingψ, and then applying PCA over {ψ(x1),...,ψ (xm)}into

Rn. The output of this process is the set of reduced elements.
Show how this process can be done in polynomial time in terms of m
andn, assuming that each evaluation of K(·,·) can be calculated in a con-
stant time. In particular, if your implementation requires multiplication of
two matrices AandB, verify that their product can be computed. Similarly,

340 Dimensionality Reduction
if an eigenvalue decomposition of some matrix Cis required, verify that this
decomposition can be computed.
4.An Interpretation of PCA as Variance Maximization:
Letx1,...,xmbemvectors in Rd, and let xbe a random vector distributed
according to the uniform distribution over x1,...,xm. Assume that E[x] =0.
1. Consider the problem of ﬁnding a unit vector, w∈Rd, such that the
random variable ⟨w,x⟩has maximal variance. That is, we would like to
solve the problem
argmax
w:∥w∥=1Var[⟨w,x⟩] = argmax
w:∥w∥=11
mm∑
i=1(⟨w,xi⟩)2.
Show that the solution of the problem is to set wto be the ﬁrst principle
vector of x1,...,xm.
2. Let w1be the ﬁrst principal component as in the previous question. Now,
suppose we would like to ﬁnd a second unit vector, w2∈Rd, that maxi-
mizes the variance of ⟨w2,x⟩, but is also uncorrelated to ⟨w1,x⟩. That is,
we would like to solve:
argmax
w:∥w∥=1,E[(⟨w1,x⟩)(⟨w,x⟩)]=0Var[⟨w,x⟩].
Show that the solution to this problem is to set wto be the second principal

w:∥w∥=11
mm∑
argmax
component of x1,...,xm.
Hint: Note that
E[(⟨w1,x⟩)(⟨w,x⟩)] =w⊤
1E[xx⊤]w=mw⊤
1Aw,
whereA=∑
ixix⊤
i. Since wis an eigenvector of Awe have that the
constraint E[(⟨w1,x⟩)(⟨w,x⟩)] = 0 is equivalent to the constraint
⟨w1,w⟩= 0.
5.The Relation between SVD and PCA: Use the SVD theorem (Corol-
lary C.6) for providing an alternative proof of Theorem 23.2.
6.Random Projections Preserve Inner Products: The Johnson-Lindenstrauss
lemma tells us that a random projection preserves distances between a ﬁnite

1Aw,
whereA=∑
ixix⊤
⟨w1,w⟩= 0.
set of vectors. In this exercise you need to prove that if the set of vectors are
within the unit ball, then not only are the distances between any two vectors
preserved, but the inner product is also preserved.
LetQbe a ﬁnite set of vectors in Rdand assume that for every x∈Qwe
have∥x∥≤1.
1. Letδ∈(0,1) and n be an integer such that
ϵ=√
6 log(|Q|2/δ)
n≤3.
Prove that with probability of at least 1 −δover a choice of a random

23.7 Exercises 341
matrixW∈Rn,d, where each element of Wis independently distributed
according toN(0,1/n), we have
|⟨Wu,Wv⟩−⟨u,v⟩|≤ϵ
for every u,v∈Q.
Hint: Use JL to bound both∥W(u+v)∥
∥u+v∥and∥W(u−v)∥
∥u−v∥.
2.(*)Letx1,...,xmbe a set of vectors in Rdof norm at most 1, and assume
that these vectors are linearly separable with margin of γ. Assume that
d≫1/γ2. Show that there exists a constant c>0 such that if we randomly
project these vectors into Rn, forn=c/γ2, then with probability of at least
99% it holds that the projected vectors are linearly separable with margin
γ/2.

24 Generative Models
We started this book with a distribution free learning framework; namely, we
did not impose any assumptions on the underlying distribution over the data.
Furthermore, we followed a discriminative approach in which our goal is not to
learn the underlying distribution but rather to learn an accurate predictor. In
this chapter we describe a generative approach, in which it is assumed that the
underlying distribution over the data has a speciﬁc parametric form and our goal
is to estimate the parameters of the model. This task is called parametric density
estimation .
The discriminative approach has the advantage of directly optimizing the
quantity of interest (the prediction accuracy) instead of learning the underly-
ing distribution. This was phrased as follows by Vladimir Vapnik in his principle
for solving problems using a restricted amount of information:
When solving a given problem, try to avoid a more general problem as an intermediate
step.

step.
Of course, if we succeed in learning the underlying distribution accurately,
we are considered to be “experts” in the sense that we can predict by using
the Bayes optimal classiﬁer. The problem is that it is usually more diﬃcult to
learn the underlying distribution than to learn an accurate predictor. However,
in some situations, it is reasonable to adopt the generative learning approach.
For example, sometimes it is easier (computationally) to estimate the parameters

step.
of the model than to learn a discriminative predictor. Additionally, in some cases
we do not have a speciﬁc task at hand but rather would like to model the data
either for making predictions at a later time without having to retrain a predictor
or for the sake of interpretability of the data.
We start with a popular statistical method for estimating the parameters of
the data, which is called the maximum likelihood principle. Next, we describe two
generative assumptions which greatly simplify the learning process. We also de-

scribe the EM algorithm for calculating the maximum likelihood in the presence
of latent variables. We conclude with a brief description of Bayesian reasoning.

24.1 Maximum Likelihood Estimator
Let us start with a simple example. A drug company developed a new drug to
treat some deadly disease. We would like to estimate the probability of survival
when using the drug. To do so, the drug company sampled a training set of m
people and gave them the drug. Let S= (x1,...,xm) denote the training set,
where for each i,xi= 1 if theith person survived and xi= 0 otherwise. We can
model the underlying distribution using a single parameter, θ∈[0,1], indicating
the probability of survival.
We now would like to estimate the parameter θon the basis of the training
setS. A natural idea is to use the average number of 1’s in Sas an estimator.
That is,
ˆθ=1
mm∑
i=1xi. (24.1)
Clearly, ES[ˆθ] =θ. That is, ˆθis an unbiased estimator ofθ. Furthermore, since ˆθis
the average of mi.i.d. binary random variables we can use Hoeﬀding’s inequality
to get that with probability of at least 1 −δover the choice of Swe have that
|ˆθ−θ| ≤√
log(2/δ)
2m. (24.2)

That is,
ˆθ=1
mm∑
|ˆθ−θ| ≤√
log(2/δ)
2m. (24.2)
Another interpretation of ˆθis as the Maximum Likelihood Estimator , as we
formally explain now. We ﬁrst write the probability of generating the sample S:
P[S= (x1,...,xm)] =m∏
i=1θxi(1−θ)1−xi=θ∑
ixi(1−θ)∑
i(1−xi).
We deﬁne the log likelihood ofS, given the parameter θ, as the log of the preceding
L(S;θ) = log ( P[S= (x1,...,xm)]) = log(θ)∑
ixi+ log(1−θ)∑
i(1−xi).
The maximum likelihood estimator is the parameter that maximizes the likeli-
hood
ˆθ∈argmax
θL(S;θ). (24.3)

log(2/δ)
2m. (24.2)
ixi(1−θ)∑
i(1−xi).
i(1−xi).
hood
ˆθ∈argmax
Next, we show that in our case, Equation (24.1) is a maximum likelihood esti-
mator. To see this, we take the derivative of L(S;θ) with respect to θand equate
it to zero:∑
ixi
θ−∑
i(1−xi)
1−θ= 0.
Solving the equation for θwe obtain the estimator given in Equation (24.1).

344 Generative Models
24.1.1 Maximum Likelihood Estimation for Continuous Random Variables
LetXbe a continuous random variable. Then, for most x∈Rwe have P[X=
x] = 0 and therefore the deﬁnition of likelihood as given before is trivialized. To
overcome this technical problem we deﬁne the likelihood as log of the density of
the probability of Xatx. That is, given an i.i.d. training set S= (x1,...,xm)
sampled according to a density distribution Pθwe deﬁne the likelihood of Sgiven
θas
L(S;θ) = log(m∏
i=1Pθ(xi))
=m∑
i=1log(Pθ(xi)).
As before, the maximum likelihood estimator is a maximizer of L(S;θ) with
respect toθ.
As an example, consider a Gaussian random variable, for which the density
function of Xis parameterized by θ= (µ,σ) and is deﬁned as follows:
Pθ(x) =1
σ√
2πexp(
−(x−µ)2
2σ2)
.
We can rewrite the likelihood as
L(S;θ) =−1
2σ2m∑
i=1(xi−µ)2−mlog(σ√
2π).
To ﬁnd a parameter θ= (µ,σ) that optimizes this we take the derivative of the

θas
=m∑
Pθ(x) =1
σ√
2πexp(
−(x−µ)2
2σ2)
.
L(S;θ) =−1
2σ2m∑
2π).
likelihood w.r.t. µand w.r.t.σand compare it to 0. We obtain the following two
d
dµL(S;θ) =1
σ2m∑
i=1(xi−µ) = 0
d
dσL(S;θ) =1
σ3m∑
i=1(xi−µ)2−m
σ= 0
Solving the preceding equations we obtain the maximum likelihood estimates:
ˆµ=1
mm∑
i=1xi and ˆσ=√1
mm∑
i=1(xi−ˆµ)2
Note that the maximum likelihood estimate is not always an unbiased estimator.
For example, while ˆ µis unbiased, it is possible to show that the estimate ˆ σof
the variance is biased (Exercise 1).
Simplifying Notation

d
σ2m∑
d
σ3m∑
σ= 0
ˆµ=1
mm∑
mm∑
To simplify our notation, we use P[X=x] in this chapter to describe both the
probability that X=x(for discrete random variables) and the density of the
distribution at x(for continuous variables).

24.1 Maximum Likelihood Estimator 345
24.1.2 Maximum Likelihood and Empirical Risk Minimization
The maximum likelihood estimator shares some similarity with the Empirical
Risk Minimization (ERM) principle, which we studied extensively in previous
chapters. Recall that in the ERM principle we have a hypothesis class Hand
we use the training set for choosing a hypothesis h∈H that minimizes the
empirical risk. We now show that the maximum likelihood estimator is an ERM
for a particular loss function.
Given a parameter θand an observation x, we deﬁne the loss of θonxas
ℓ(θ,x) =−log(Pθ[x]). (24.4)
That is,ℓ(θ,x) is the negation of the log-likelihood of the observation x, assuming
the data is distributed according to Pθ. This loss function is often referred to as
the log-loss. On the basis of this deﬁnition it is immediate that the maximum
likelihood principle is equivalent to minimizing the empirical risk with respect
to the loss function given in Equation (24.4). That is,
argmin
θm∑
i=1(−log(Pθ[xi])) = argmax
θm∑

argmin
θm∑
θm∑
i=1log(Pθ[xi]).
Assuming that the data is distributed according to a distribution P(not neces-
sarily of the parametric form we employ), the true risk of a parameter θbecomes
E
x[ℓ(θ,x)] =−∑
xP[x] log(Pθ[x])
=∑
xP[x] log(P[x]
Pθ[x])

DRE[P||Pθ]+∑
xP[x] log(1
P[x])

H(P),(24.5)
whereDREis called the relative entropy , andHis called the entropy func-
tion. The relative entropy is a divergence measure between two probabilities.
For discrete variables, it is always nonnegative and is equal to 0 only if the two

E
=∑
Pθ[x])

P[x])

distributions are the same. It follows that the true risk is minimal when Pθ=P.
The expression given in Equation (24.5) underscores how our generative as-
sumption aﬀects our density estimation, even in the limit of inﬁnite data. It
shows that if the underlying distribution is indeed of a parametric form, then by
choosing the correct parameter we can make the risk be the entropy of the distri-
bution. However, if the distribution is not of the assumed parametric form, even

the best parameter leads to an inferior model and the suboptimality is measured
by the relative entropy divergence.
24.1.3 Generalization Analysis
How good is the maximum likelihood estimator when we learn from a ﬁnite
training set?

346 Generative Models
To answer this question we need to deﬁne how we assess the quality of an approxi-
mated solution of the density estimation problem. Unlike discriminative learning,
where there is a clear notion of “loss,” in generative learning there are various
ways to deﬁne the loss of a model. On the basis of the previous subsection, one
natural candidate is the expected log-loss as given in Equation (24.5).
In some situations, it is easy to prove that the maximum likelihood principle
guarantees low true risk as well. For example, consider the problem of estimating
the mean of a Gaussian variable of unit variance. We saw previously that the
maximum likelihood estimator is the average: ˆ µ=1
m∑
ixi. Letµ⋆be the optimal
parameter. Then,
E
x∼N(µ⋆,1)[ℓ(ˆµ,x)−ℓ(µ⋆,x)] = E
x∼N(µ⋆,1)log(Pµ⋆[x]
Pˆµ[x])
= E
x∼N(µ⋆,1)(
−1
2(x−µ⋆)2+1
2(x−ˆµ)2)
=ˆµ2
2−(µ⋆)2
2+ (µ⋆−ˆµ)E
x∼N(µ⋆,1)[x]
=ˆµ2
2−(µ⋆)2
2+ (µ⋆−ˆµ)µ⋆
=1
2(ˆµ−µ⋆)2. (24.6)
Next, we note that ˆ µis the average of mGaussian variables and therefore it is

m∑
E
Pˆµ[x])
= E
−1
2(x−ˆµ)2)
=ˆµ2
2−(µ⋆)2
=ˆµ2
2−(µ⋆)2
=1
also distributed normally with mean µ⋆and variance σ⋆/m. From this fact we
can derive bounds of the form: with probability of at least 1 −δwe have that
|ˆµ−µ⋆|≤ϵwhereϵdepends on σ⋆/mand onδ.
In some situations, the maximum likelihood estimator clearly overﬁts. For
example, consider a Bernoulli random variable Xand letP[X= 1] =θ⋆. As
we saw previously, using Hoeﬀding’s inequality we can easily derive a guarantee
on|θ⋆−ˆθ|that holds with high probability (see Equation (24.2)). However, if
our goal is to obtain a small value of the expected log-loss function as deﬁned in

Equation (24.5) we might fail. For example, assume that θ⋆is nonzero but very
small. Then, the probability that no element of a sample of size mwill be 1 is
(1−θ⋆)m, which is greater than e−2θ⋆m. It follows that whenever m≤log(2)
2θ⋆,
the probability that the sample is all zeros is at least 50%, and in that case, the
maximum likelihood rule will set ˆθ= 0. But the true risk of the estimate ˆθ= 0
is
E
x∼θ⋆[ℓ(ˆθ,x)] =θ⋆ℓ(ˆθ,1) + (1−θ⋆)ℓ(ˆθ,0)
=θ⋆log(1/ˆθ) + (1−θ⋆) log(1/(1−ˆθ))
=θ⋆log(1/0) =∞.

2θ⋆,
is
E
This simple example shows that we should be careful in applying the maximum
likelihood principle.
To overcome overﬁtting, we can use the variety of tools we encountered pre-

viously in the book. A simple regularization technique is outlined in Exercise
2.
24.2 Naive Bayes
The Naive Bayes classiﬁer is a classical demonstration of how generative as-
sumptions and parameter estimations simplify the learning process. Consider
the problem of predicting a label y∈{0,1}on the basis of a vector of features
x= (x1,...,xd), where we assume that each xiis in{0,1}. Recall that the Bayes
optimal classiﬁer is
hBayes(x) = argmax
y∈{0,1}P[Y=y|X=x].
To describe the probability function P[Y=y|X=x] we need 2dparameters,
each of which corresponds to P[Y= 1|X=x] for a certain value of x∈{0,1}d.
This implies that the number of examples we need grows exponentially with the
number of features.
In the Naive Bayes approach we make the (rather naive) generative assumption
that given the label, the features are independent of each other. That is,
P[X=x|Y=y] =d∏
i=1P[Xi=xi|Y=y].
With this assumption and using Bayes’ rule, the Bayes optimal classiﬁer can be
further simpliﬁed:

y∈{0,1}P[Y=y|X=x]
= argmax
y∈{0,1}P[Y=y]P[X=x|Y=y]
= argmax
y∈{0,1}P[Y=y]d∏
i=1P[Xi=xi|Y=y]. (24.7)
That is, now the number of parameters we need to estimate is only 2 d+ 1.
Here, the generative assumption we made reduced signiﬁcantly the number of
parameters we need to learn.
When we also estimate the parameters using the maximum likelihood princi-
ple, the resulting classiﬁer is called the Naive Bayes classiﬁer.
24.3 Linear Discriminant Analysis
Linear discriminant analysis (LDA) is another demonstration of how generative

= argmax
= argmax
assumptions simplify the learning process. As in the Naive Bayes classiﬁer we
consider again the problem of predicting a label y∈{0,1}on the basis of a

348 Generative Models
vector of features x= (x1,...,xd). But now the generative assumption is as
follows. First, we assume that P[Y= 1] =P[Y= 0] = 1/2. Second, we assume
that the conditional probability of XgivenYis a Gaussian distribution. Finally,
the covariance matrix of the Gaussian distribution is the same for both values
of the label. Formally, let µ0,µ1∈Rdand let Σ be a covariance matrix. Then,
the density distribution is given by
P[X=x|Y=y] =1
(2π)d/2|Σ|1/2exp(
−1
2(x−µy)TΣ−1(x−µy))
.
As we have shown in the previous section, using Bayes’ rule we can write
y∈{0,1}P[Y=y]P[X=x|Y=y].
This means that we will predict hBayes(x) = 1 iﬀ
log(P[Y= 1]P[X=x|Y= 1]
P[Y= 0]P[X=x|Y= 0])
>0.
This ratio is often called the log-likelihood ratio .
In our case, the log-likelihood ratio becomes
1
2(x−µ0)TΣ−1(x−µ0)−1
2(x−µ1)TΣ−1(x−µ1)
We can rewrite this as ⟨w,x⟩+bwhere
w= (µ1−µ0)TΣ−1andb=1
2(
µT
0Σ−1µ0−µT
1Σ−1µ1)
. (24.8)
As a result of the preceding derivation we obtain that under the aforemen-

>0.
1
2(
µT
0Σ−1µ0−µT
1Σ−1µ1)
. (24.8)
tioned generative assumptions, the Bayes optimal classiﬁer is a linear classiﬁer.
Additionally, one may train the classiﬁer by estimating the parameter µ0,µ1
and Σ from the data, using, for example, the maximum likelihood estimator.
With those estimators at hand, the values of wandbcan be calculated as in
Equation (24.8).
24.4 Latent Variables and the EM Algorithm
In generative models we assume that the data is generated by sampling from
a speciﬁc parametric distribution over our instance space X. Sometimes, it is

convenient to express this distribution using latent random variables. A natural
example is a mixture of kGaussian distributions. That is, X=Rdand we
assume that each xis generated as follows. First, we choose a random number in
{1,...,k}. LetYbe a random variable corresponding to this choice, and denote
P[Y=y] =cy. Second, we choose xon the basis of the value of Yaccording to
a Gaussian distribution
(2π)d/2|Σy|1/2exp(
−1
2(x−µy)TΣ−1
y(x−µy))
.(24.9)

24.4 Latent Variables and the EM Algorithm 349
Therefore, the density of Xcan be written as:
P[X=x] =k∑
y=1P[Y=y]P[X=x|Y=y]
=k∑
y=1cy1
−1
y(x−µy))
.
Note thatYis a hidden variable that we do not observe in our data. Neverthe-
less, we introduce Ysince it helps us describe a simple parametric form of the
probability of X.
More generally, let θbe the parameters of the joint distribution of XandY
(e.g., in the preceding example, θconsists ofcy,µy, and Σy, for ally= 1,...,k ).
Then, the log-likelihood of an observation xcan be written as
log (Pθ[X=x]) = log(k∑
y=1Pθ[X=x,Y=y])
.
Given an i.i.d. sample, S= (x1,...,xm), we would like to ﬁnd θthat maxi-
mizes the log-likelihood of S,
L(θ) = logm∏
i=1Pθ[X=xi]
=m∑
i=1logPθ[X=xi]
=m∑
i=1log(k∑
y=1Pθ[X=xi,Y=y])
.
The maximum-likelihood estimator is therefore the solution of the maximization
problem
argmax
θL(θ) = argmax
θm∑
i=1log(k∑
.
In many situations, the summation inside the log makes the preceding opti-

.
=m∑
=m∑
i=1log(k∑
.
problem
argmax
θm∑
i=1log(k∑
.
mization problem computationally hard. The Expectation-Maximization (EM)
algorithm, due to Dempster, Laird, and Rubin, is an iterative procedure for
searching a (local) maximum of L(θ). While EM is not guaranteed to ﬁnd the
global maximum, it often works reasonably well in practice.
EM is designed for those cases in which, had we known the values of the latent
variablesY, then the maximum likelihood optimization problem would have been
tractable. More precisely, deﬁne the following function over m×kmatrices and

the set of parameters θ:
F(Q,θ) =m∑
i=1k∑
y=1Qi,ylog (Pθ[X=xi,Y=y]).

350 Generative Models
If each row of Qdeﬁnes a probability over the ith latent variable given X=xi,
then we can interpret F(Q,θ) as the expected log-likelihood of a training set
(x1,y1),..., (xm,ym), where the expectation is with respect to the choice of
eachyion the basis of the ith row ofQ. In the deﬁnition of F, the summation is
outside the log, and we assume that this makes the optimization problem with
respect toθtractable:
assumption 24.1 For any matrix Q∈[0,1]m,k, such that each row of Qsums
to 1, the optimization problem
argmax
θF(Q,θ)
is tractable.
The intuitive idea of EM is that we have a “chicken and egg” problem. On one
hand, had we known Q, then by our assumption, the optimization problem of
ﬁnding the best θis tractable. On the other hand, had we known the parameters
θwe could have set Qi,yto be the probability of Y=ygiven that X=xi.
The EM algorithm therefore alternates between ﬁnding θgivenQand ﬁnding Q
givenθ. Formally, EM ﬁnds a sequence of solutions ( Q(1),θ(1)),(Q(2),θ(2)),...

argmax
θF(Q,θ)
where at iteration t, we construct ( Q(t+1),θ(t+1)) by performing two steps.
•Expectation Step: Set
Q(t+1)
i,y =Pθ(t)[Y=y|X=xi]. (24.10)
This step is called the Expectation step, because it yields a new probabil-
ity over the latent variables, which deﬁnes a new expected log-likelihood
function over θ.
•Maximization Step: Set θ(t+1)to be the maximizer of the expected log-
likelihood, where the expectation is according to Q(t+1):
θ(t+1)= argmax
θF(Q(t+1),θ). (24.11)

Q(t+1)
By our assumption, it is possible to solve this optimization problem eﬃ-
ciently.
The initial values of θ(1)andQ(1)are usually chosen at random and the
procedure terminates after the improvement in the likelihood value stops being
signiﬁcant.
24.4.1 EM as an Alternate Maximization Algorithm
To analyze the EM algorithm, we ﬁrst view it as an alternate maximization
algorithm. Deﬁne the following objective function
G(Q,θ) =F(Q,θ)−m∑
i=1k∑
y=1Qi,ylog(Qi,y).

24.4 Latent Variables and the EM Algorithm 351
The second term is the sum of the entropies of the rows of Q. Let
Q={
Q∈[0,1]m,k:∀i,k∑
y=1Qi,y= 1}
be the set of matrices whose rows deﬁne probabilities over [ k]. The following
lemma shows that EM performs alternate maximization iterations for maximiz-
ingG.
lemma 24.2 The EM procedure can be rewritten as:
Q(t+1)= argmax
Q∈QG(Q,θ(t))
θG(Q(t+1),θ).
Furthermore, G(Q(t+1),θ(t)) =L(θ(t)).
Proof GivenQ(t+1)we clearly have that
argmax
θG(Q(t+1),θ) = argmax
θF(Q(t+1),θ).
Therefore, we only need to show that for any θ, the solution of argmaxQ∈QG(Q,θ)
is to setQi,y=Pθ[Y=y|X=xi]. Indeed, by Jensen’s inequality, for any Q∈Q
we have that
G(Q,θ) =m∑
i=1(k∑
y=1Qi,ylog(Pθ[X=xi,Y=y]
Qi,y))
≤m∑
i=1(
log(k∑
y=1Qi,yPθ[X=xi,Y=y]
Qi,y))
=m∑
i=1log(k∑
=m∑
i=1log (Pθ[X=xi]) =L(θ),

352 Generative Models
while forQi,y=Pθ[Y=y|X=xi] we have
G(Q,θ) =m∑
i=1(k∑
y=1Pθ[Y=y|X=xi] log(Pθ[X=xi,Y=y]
Pθ[Y=y|X=xi]))
=m∑
i=1k∑
y=1Pθ[Y=y|X=xi] log (Pθ[X=xi])
=m∑
i=1log (Pθ[X=xi])k∑
y=1Pθ[Y=y|X=xi]
=m∑
i=1log (Pθ[X=xi]) =L(θ).
This shows that setting Qi,y=Pθ[Y=y|X=xi] maximizes G(Q,θ) overQ∈Q
and shows that G(Q(t+1),θ(t)) =L(θ(t)).
The preceding lemma immediately implies:
theorem 24.3 The EM procedure never decreases the log-likelihood; namely,
for allt,
L(θ(t+1))≥L(θ(t)).
Proof By the lemma we have
L(θ(t+1)) =G(Q(t+2),θ(t+1))≥G(Q(t+1),θ(t)) =L(θ(t)).
24.4.2 EM for Mixture of Gaussians (Soft k-Means)
Consider the case of a mixture of kGaussians in which θis a triplet ( c,{µ1,...,µk},{Σ1,..., Σk})
wherePθ[Y=y] =cyandPθ[X=x|Y=y] is as given in Equation (24.9). For
simplicity, we assume that Σ 1= Σ 2=···= Σk=I, whereIis the identity
matrix. Specifying the EM algorithm for this case we obtain the following:
•Expectation step: For each i∈[m] andy∈[k] we have that
Pθ(t)[Y=y|X=xi] =1

ZiPθ(t)[Y=y]Pθ(t)[X=xi|Y=y]
=1
Zic(t)
yexp(
−1
2∥xi−µ(t)
y∥2)
, (24.12)
whereZiis a normalization factor which ensures that∑
yPθ(t)[Y=y|X=
xi] sums to 1.
•Maximization step: We need to set θt+1to be a maximizer of Equation (24.11),

which in our case amounts to maximizing the following expression w.r.t. c
andµ:
m∑
i=1k∑
y=1Pθ(t)[Y=y|X=xi](
log(cy)−1
2∥xi−µy∥2)
. (24.13)
Comparing the derivative of Equation (24.13) w.r.t. µyto zero and rear-
ranging terms we obtain:
µy=∑m
i=1Pθ(t)[Y=y|X=xi]xi∑m
i=1Pθ(t)[Y=y|X=xi].
That is,µyis a weighted average of the xiwhere the weights are according
to the probabilities calculated in the E step. To ﬁnd the optimal cwe need
to be more careful since we must ensure that cis a probability vector. In
Exercise 3 we show that the solution is:
cy=∑m
i=1Pθ(t)[Y=y|X=xi]
∑k
y′=1∑m
i=1Pθ(t)[Y=y′|X=xi]. (24.14)
It is interesting to compare the preceding algorithm to the k-means algorithm
described in Chapter 22. In the k-means algorithm, we ﬁrst assign each example
to a cluster according to the distance ∥xi−µy∥. Then, we update each center
µyaccording to the average of the examples assigned to this cluster. In the EM

cy=∑m
∑k
y′=1∑m
approach, however, we determine the probability that each example belongs to
each cluster. Then, we update the centers on the basis of a weighted sum over
the entire sample. For this reason, the EM approach for k-means is sometimes
called “soft k-means.”
24.5 Bayesian Reasoning
The maximum likelihood estimator follows a frequentist approach. This means
that we refer to the parameter θas a ﬁxed parameter and the only problem is
that we do not know its value. A diﬀerent approach to parameter estimation

is called Bayesian reasoning. In the Bayesian approach, our uncertainty about
θis also modeled using probability theory. That is, we think of θas a random
variable as well and refer to the distribution P[θ] as a prior distribution . As its
name indicates, the prior distribution should be deﬁned by the learner prior to
observing the data.
As an example, let us consider again the drug company which developed a
new drug. On the basis of past experience, the statisticians at the drug company

believe that whenever a drug has reached the level of clinic experiments on
people, it is likely to be eﬀective. They model this prior belief by deﬁning a
density distribution on θsuch that
P[θ] ={
0.8 ifθ>0.5
0.2 ifθ≤0.5(24.15)

354 Generative Models
As before, given a speciﬁc value of θ, it is assumed that the conditional proba-
bility,P[X=x|θ], is known. In the drug company example, Xtakes values in
{0,1}andP[X=x|θ] =θx(1−θ)1−x.
Once the prior distribution over θand the conditional distribution over X
givenθare deﬁned, we again have complete knowledge of the distribution over
X. This is because we can write the probability over Xas a marginal probability
P[X=x] =∑
θP[X=x,θ] =∑
θP[θ]P[X=x|θ],
where the last equality follows from the deﬁnition of conditional probability. If
θis continuous we replace P[θ] with the density function and the sum becomes
an integral:
P[X=x] =∫
θP[θ]P[X=x|θ]dθ.
Seemingly, once we know P[X=x], a training set S= (x1,...,xm) tells us
nothing as we are already experts who know the distribution over a new point
X. However, the Bayesian view introduces dependency between SandX. This is
because we now refer to θas a random variable. A new point Xand the previous

P[X=x] =∫
points inSare independent only conditioned on θ. This is diﬀerent from the
frequentist philosophy in which θis a parameter that we might not know, but
since it is just a parameter of the distribution, a new point Xand previous points
Sare always independent.
In the Bayesian framework, since XandSare not independent anymore, what
we would like to calculate is the probability of XgivenS, which by the chain
rule can be written as follows:
P[X=x|S] =∑
θP[X=x|θ,S]P[θ|S] =∑
θP[X=x|θ]P[θ|S].

The second inequality follows from the assumption that XandSare independent
when we condition on θ. Using Bayes’ rule we have
P[θ|S] =P[S|θ]P[θ]
P[S],
and together with the assumption that points are independent conditioned on θ,
we can write
P[S]=1
P[S]m∏
i=1P[X=xi|θ]P[θ].
We therefore obtain the following expression for Bayesian prediction:
P[X=x|S] =1
P[S]∑
θP[X=x|θ]m∏
i=1P[X=xi|θ]P[θ]. (24.16)
Getting back to our drug company example, we can rewrite P[X=x|S] as
P[S]∫
θx+∑
ixi(1−θ)1−x+∑

P[S],
P[S]=1
P[S]m∏
P[S]∑
P[S]∫
θx+∑
i(1−xi)P[θ]dθ.

It is interesting to note that when P[θ] is uniform we obtain that
P[X=x|S]∝∫
θx+∑
i(1−xi)dθ.
Solving the preceding integral (using integration by parts) we obtain
P[X= 1|S] =(∑
ixi) + 1
m+ 2.
Recall that the prediction according to the maximum likelihood principle in this
case isP[X= 1|ˆθ] =∑
ixi
m. The Bayesian prediction with uniform prior is rather
similar to the maximum likelihood prediction, except it adds “pseudoexamples”
to the training set, thus biasing the prediction toward the uniform prior.
Maximum A Posteriori
In many situations, it is diﬃcult to ﬁnd a closed form solution to the integral
given in Equation (24.16). Several numerical methods can be used to approxi-
mate this integral. Another popular solution is to ﬁnd a single θwhich maximizes
P[θ|S]. The value of θwhich maximizes P[θ|S] is called the Maximum A Poste-
riori estimator. Once this value is found, we can calculate the probability that
X=xgiven the maximum a posteriori estimator and independently on S.

24.6 Summary
In the generative approach to machine learning we aim at modeling the distri-
bution over the data. In particular, in parametric density estimation we further
assume that the underlying distribution over the data has a speciﬁc paramet-
ric form and our goal is to estimate the parameters of the model. We have
described several principles for parameter estimation, including maximum like-
lihood, Bayesian estimation, and maximum a posteriori . We have also described

several speciﬁc algorithms for implementing the maximum likelihood under dif-
ferent assumptions on the underlying data distribution, in particular, Naive
Bayes, LDA, and EM.
24.7 Bibliographic Remarks
The maximum likelihood principle was studied by Ronald Fisher in the beginning
of the 20th century. Bayesian statistics follow Bayes’ rule, which is named after
the 18th century English mathematician Thomas Bayes.
There are many excellent books on the generative and Bayesian approaches

to machine learning. See, for example, (Bishop 2006, Koller & Friedman 2009,
MacKay 2003, Murphy 2012, Barber 2012).

356 Generative Models
24.8 Exercises
1. Prove that the maximum likelihood estimator of the variance of a Gaussian
variable is biased.
2. Regularization for Maximum Likelihood: Consider the following regularized
loss minimization:
1
mm∑
i=1log(1/Pθ[xi]) +1
m(log(1/θ) + log(1/(1−θ))).
•Show that the preceding objective is equivalent to the usual empirical error
had we added two pseudoexamples to the training set. Conclude that
the regularized maximum likelihood estimator would be
ˆθ=1
m+ 2(
1 +m∑
i=1xi)
.
•Derive a high probability bound on |ˆθ−θ⋆|.Hint: Rewrite this as|ˆθ−E[ˆθ]+
E[ˆθ]−θ⋆|and then use the triangle inequality and Hoeﬀding inequality.
•Use this to bound the true risk. Hint: Use the fact that now ˆθ≥1
m+2to
relate|ˆθ−θ⋆|to the relative entropy.
3.•Consider a general optimization problem of the form:
max
ck∑
y=1νylog(cy) s.t.cy>0,∑
ycy= 1,
whereν∈Rk
+is a vector of nonnegative weights. Verify that the M step
of softk-means involves solving such an optimization problem.
•Letc⋆=1∑

1 +m∑
i=1xi)
.
m+2to
max
ck∑
ycy= 1,
whereν∈Rk
•Letc⋆=1∑
yνyν. Show that c⋆is a probability vector.
•Show that the optimization problem is equivalent to the problem:
min
cDRE(c⋆||c) s.t.cy>0,∑
ycy= 1.
•Using properties of the relative entropy, conclude that c⋆is the solution to
the optimization problem.

25 Feature Selection and Generation
In the beginning of the book, we discussed the abstract model of learning, in
which the prior knowledge utilized by the learner is fully encoded by the choice
of the hypothesis class. However, there is another modeling choice, which we
have so far ignored: How do we represent the instance space X? For example, in
the papayas learning problem, we proposed the hypothesis class of rectangles in
the softness-color two dimensional plane. That is, our ﬁrst modeling choice was
to represent a papaya as a two dimensional point corresponding to its softness
and color. Only after that did we choose the hypothesis class of rectangles as a
class of mappings from the plane into the label set. The transformation from the
real world object “papaya” into the scalar representing its softness or its color
is called a feature function or a feature for short; namely, any measurement of
the real world object can be regarded as a feature. If Xis a subset of a vector

space, each x∈X is sometimes referred to as a feature vector . It is important to
understand that the way we encode real world objects as an instance space Xis
by itself prior knowledge about the problem.
Furthermore, even when we already have an instance space Xwhich is rep-
resented as a subset of a vector space, we might still want to change it into a
diﬀerent representation and apply a hypothesis class on top of it. That is, we
may deﬁne a hypothesis class on Xby composing some class Hon top of a

feature function which maps Xinto some other vector space X′. We have al-
ready encountered examples of such compositions – in Chapter 15 we saw that
kernel-based SVM learns a composition of the class of halfspaces over a feature
mappingψthat maps each original instance in Xinto some Hilbert space. And,
indeed, the choice of ψis another form of prior knowledge we impose on the
problem.
In this chapter we study several methods for constructing a good feature set.

problem.
We start with the problem of feature selection , in which we have a large pool
of features and our goal is to select a small number of features that will be
used by our predictor. Next, we discuss feature manipulations and normalization .
These include simple transformations that we apply on our original features. Such
transformations may decrease the sample complexity of our learning algorithm,
its bias, or its computational complexity. Last, we discuss several approaches for

feature learning . In these methods, we try to automate the process of feature
construction.

358 Feature Selection and Generation
We emphasize that while there are some common techniques for feature learn-
ing one may want to try, the No-Free-Lunch theorem implies that there is no ulti-
mate feature learner. Any feature learning algorithm might fail on some problem.
In other words, the success of each feature learner relies (sometimes implicitly)
on some form of prior assumption on the data distribution. Furthermore, the
relative quality of features highly depends on the learning algorithm we are later
going to apply using these features. This is illustrated in the following example.
Example 25.1 Consider a regression problem in which X=R2,Y=R, and
the loss function is the squared loss. Suppose that the underlying distribution
is such that an example ( x,y) is generated as follows: First, we sample x1from
the uniform distribution over [ −1,1]. Then, we deterministically set y=x12.
Finally, the second feature is set to be x2=y+z, wherezis sampled from the

uniform distribution over [ −0.01,0.01]. Suppose we would like to choose a single
feature. Intuitively, the ﬁrst feature should be preferred over the second feature
as the target can be perfectly predicted based on the ﬁrst feature alone, while it
cannot be perfectly predicted based on the second feature. Indeed, choosing the
ﬁrst feature would be the right choice if we are later going to apply polynomial
regression of degree at least 2. However, if the learner is going to be a linear

regressor, then we should prefer the second feature over the ﬁrst one, since the
optimal linear predictor based on the ﬁrst feature will have a larger risk than
the optimal linear predictor based on the second feature.
25.1 Feature Selection
Throughout this section we assume that X=Rd. That is, each instance is repre-
sented as a vector of dfeatures. Our goal is to learn a predictor that only relies
onk≪dfeatures. Predictors that use only a small subset of features require a

smaller memory footprint and can be applied faster. Furthermore, in applications
such as medical diagnostics, obtaining each possible “feature” (e.g., test result)
can be costly; therefore, a predictor that uses only a small number of features
is desirable even at the cost of a small degradation in performance, relative to
a predictor that uses more features. Finally, constraining the hypothesis class to
use a small subset of features can reduce its estimation error and thus prevent
overﬁtting.

Ideally, we could have tried all subsets of kout ofdfeatures and choose the
subset which leads to the best performing predictor. However, such an exhaustive
search is usually computationally intractable. In the following we describe three
computationally feasible approaches for feature selection. While these methods
cannot guarantee ﬁnding the optimal subset, they often work reasonably well in
practice. Some of the methods come with formal guarantees on the quality of the

selected subsets under certain assumptions. We do not discuss these guarantees
here.

25.1 Feature Selection 359
25.1.1 Filters
Maybe the simplest approach for feature selection is the ﬁlter method, in which
we assess individual features, independently of other features, according to some
quality measure. We can then select the kfeatures that achieve the highest score
(alternatively, decide also on the number of features to select according to the
value of their scores).
Many quality measures for features have been proposed in the literature.
Maybe the most straightforward approach is to set the score of a feature ac-
cording to the error rate of a predictor that is trained solely by that feature.
To illustrate this, consider a linear regression problem with the squared loss.
Letv= (x1,j,...,xm,j)∈Rmbe a vector designating the values of the jth
feature on a training set of mexamples and let y= (y1,...,ym)∈Rmbe the
values of the target on the same mexamples. The empirical squared loss of an
ERM linear predictor that uses only the jth feature would be
min
a,b∈R1
m∥av+b−y∥2,

min
a,b∈R1
where the meaning of adding a scalar bto a vector vis addingbto all coordinates
ofv. To solve this problem, let ¯ v=1
m∑m
i=1vibe the averaged value of the
feature and let ¯ y=1
m∑m
i=1yibe the averaged value of the target. Clearly (see
Exercise 1),
min
a,b∈R1
m∥av+b−y∥2= min
a,b∈R1
m∥a(v−¯v) +b−(y−¯y)∥2. (25.1)
Taking the derivative of the right-hand side objective with respect to band
comparing it to zero we obtain that b= 0. Similarly, solving for a(once we know
thatb= 0) yields a=⟨v−¯v,y−¯y⟩/∥v−¯v∥2. Plugging this value back into the

m∑m
m∑m
min
a,b∈R1
a,b∈R1
objective we obtain the value
∥y−¯y∥2−(⟨v−¯v,y−¯y⟩)2
∥v−¯v∥2.
Ranking the features according to the minimal loss they achieve is equivalent
to ranking them according to the absolute value of the following score (where
now a higher score yields a better feature):
⟨v−¯v,y−¯y⟩
∥v−¯v∥∥y−¯y∥=1
m⟨v−¯v,y−¯y⟩√
1
m∥v−¯v∥2√
1
m∥y−¯y∥2. (25.2)
The preceding expression is known as Pearson’s correlation coeﬃcient . The nu-
merator is the empirical estimate of the covariance of thejth feature and the

∥v−¯v∥2.
1
m∥v−¯v∥2√
1
target value, E[(v−Ev)(y−Ey)], while the denominator is the squared root of
the empirical estimate for the variance of thejth feature, E[(v−Ev)2], times
the variance of the target. Pearson’s coeﬃcient ranges from −1 to 1, where if
the Pearson’s coeﬃcient is either 1 or −1, there is a linear mapping from vtoy
with zero empirical risk.

360 Feature Selection and Generation
If Pearson’s coeﬃcient equals zero it means that the optimal linear function
from vtoyis the all-zeros function, which means that valone is useless for
predicting y. However, this does not mean that vis a bad feature, as it might
be the case that together with other features vcan perfectly predict y. Indeed,
consider a simple example in which the target is generated by the function y=
x1+ 2x2. Assume also that x1is generated from the uniform distribution over
{±1}, andx2=−1
2x1+1
2z, wherezis also generated i.i.d. from the uniform
distribution over {±1}. Then, E[x1] =E[x2] =E[y] = 0, and we also have
E[yx1] =E[x2
1] + 2E[x2x1] =E[x2
1]−E[x2
1] +E[zx1] = 0.
Therefore, for a large enough training set, the ﬁrst feature is likely to have a
Pearson’s correlation coeﬃcient that is close to zero, and hence it will most
probably not be selected. However, no function can predict the target value well
without knowing the ﬁrst feature.

2x1+1
1]−E[x2
There are many other score functions that can be used by a ﬁlter method.
Notable examples are estimators of the mutual information or the area under
the receiver operating characteristic (ROC) curve. All of these score functions
suﬀer from similar problems to the one illustrated previously. We refer the reader
to Guyon & Elisseeﬀ (2003).
25.1.2 Greedy Selection Approaches
Greedy selection is another popular approach for feature selection. Unlike ﬁlter
methods, greedy selection approaches are coupled with the underlying learning

algorithm. The simplest instance of greedy selection is forward greedy selection.
We start with an empty set of features, and then we gradually add one feature
at a time to the set of selected features. Given that our current set of selected
features isI, we go over all i /∈I, and apply the learning algorithm on the set
of features I∪{i}. Each such application yields a diﬀerent predictor, and we
choose to add the feature that yields the predictor with the smallest risk (on
the training set or on a validation set). This process continues until we either

selectkfeatures, where kis a predeﬁned budget of allowed features, or achieve
an accurate enough predictor.
Example 25.2 (Orthogonal Matching Pursuit) To illustrate the forward
greedy selection approach, we specify it to the problem of linear regression with
the squared loss. Let X∈Rm,dbe a matrix whose rows are the mtraining
instances. Let y∈Rmbe the vector of the mlabels. For every i∈[d], letXi
be theith column of X. Given a set I⊂[d] we denote by XIthe matrix whose
columns are{Xi:i∈I}.

The forward greedy selection method starts with I0=∅. At iteration t, we
look for the feature index jt, which is in
argmin
jmin
w∈Rt∥XIt−1∪{j}w−y∥2.

25.1 Feature Selection 361
Then, we update It=It−1∪{jt}.
We now describe a more eﬃcient implementation of the forward greedy selec-
tion approach for linear regression which is called Orthogonal Matching Pursuit
(OMP) . The idea is to keep an orthogonal basis of the features aggregated so
far. LetVtbe a matrix whose columns form an orthonormal basis of the columns
ofXIt.
Clearly,
min
w∥XItw−y∥2= min
θ∈Rt∥Vtθ−y∥2.
We will maintain a vector θtwhich minimizes the right-hand side of the equation.
Initially, we set I0=∅,V0=∅, andθ1to be the empty vector. At round t, for
everyj, we decompose Xj=vj+ujwhere vj=Vt−1V⊤
t−1Xjis the projection
ofXjonto the subspace spanned by Vt−1andujis the part of Xjorthogonal to
Vt−1(see Appendix C). Then,
min
θ,α∥Vt−1θ+αuj−y∥2
= min
θ,α[
∥Vt−1θ−y∥2+α2∥uj∥2+ 2α⟨uj,Vt−1θ−y⟩]
= min
θ,α[
∥Vt−1θ−y∥2+α2∥uj∥2+ 2α⟨uj,−y⟩]
= min
θ[
∥Vt−1θ−y∥2]
+ min
α[
α2∥uj∥2−2α⟨uj,y⟩]
=[
∥Vt−1θt−1−y∥2]
+ min
α[
=∥Vt−1θt−1−y∥2−(⟨uj,y⟩)2
∥uj∥2.
It follows that we should select the feature

min
= min
θ,α[
= min
θ,α[
= min
θ[
+ min
α[
=[
+ min
α[
∥uj∥2.
jt= argmax
j(⟨uj,y⟩)2
∥uj∥2.
The rest of the update is to set
Vt=[
Vt−1,ujt
∥ujt∥2]
,θt=[
θt−1;⟨ujt,y⟩
∥ujt∥2]
.
The OMP procedure maintains an orthonormal basis of the selected features,
where in the preceding description, the orthonormalization property is obtained
by a procedure similar to Gram-Schmidt orthonormalization. In practice, the
Gram-Schmidt procedure is often numerically unstable. In the pseudocode that
follows we use SVD (see Section C.4) at the end of each round to obtain an
orthonormal basis in a numerically stable manner.

362 Feature Selection and Generation
Orthogonal Matching Pursuit (OMP)
input:
data matrix X∈Rm,d, labels vector y∈Rm,
budget of features T
initialize: I1=∅
use SVD to ﬁnd an orthonormal basis V∈Rm,t−1ofXIt
(fort= 1 setVto be the all zeros matrix)
foreachj∈[d]\Itletuj=Xj−VV⊤Xj
letjt= argmaxj/∈It:∥uj∥>0(⟨uj,y⟩)2
∥uj∥2
updateIt+1=It∪{jt}
outputIT+1
More Eﬃcient Greedy Selection Criteria
LetR(w) be the empirical risk of a vector w. At each round of the forward
greedy selection method, and for every possible j, we should minimize R(w)
over the vectors wwhose support is It−1∪{j}. This might be time consuming.
A simpler approach is to choose jtthat minimizes
argmin
jmin
η∈RR(wt−1+ηej),
where ejis the all zeros vector except 1 in the jth element. That is, we keep
the weights of the previously chosen coordinates intact and only optimize over
the new variable. Therefore, for each jwe need to solve an optimization problem
over a single variable, which is a much easier task than optimizing over t.

argmin
jmin
An even simpler approach is to upper bound R(w) using a “simple” function
and then choose the feature which leads to the largest decrease in this upper
bound. For example, if Ris aβ-smooth function (see Equation (12.5) in Chap-
ter 12), then
R(w+ηej)≤R(w) +η∂R(w)
∂wj+βη2/2.
Minimizing the right-hand side over ηyieldsη=−∂R(w)
∂wj·1
βand plugging this
value into the above yields
R(w+ηej)≤R(w)−1
2β(∂R(w)
∂wj)2
.
This value is minimized if the partial derivative of R(w) with respect to wjis

∂wj·1
2β(∂R(w)
∂wj)2
.
maximal. We can therefore choose jtto be the index of the largest coordinate of
the gradient of R(w) atw.
Remark 25.3 (AdaBoost as a Forward Greedy Selection Procedure) It is pos-
sible to interpret the AdaBoost algorithm from Chapter 10 as a forward greedy

25.1 Feature Selection 363
selection procedure with respect to the function
R(w) = log
m∑
i=1exp
−yid∑
j=1wjhj(xi)

. (25.3)
See Exercise 3.
Backward Elimination
Another popular greedy selection approach is backward elimination . Here, we
start with the full set of features, and then we gradually remove one feature at a
time from the set of features. Given that our current set of selected features is I,
we go over all i∈I, and apply the learning algorithm on the set of features I\{i}.
Each such application yields a diﬀerent predictor, and we choose to remove the
featureifor which the predictor obtained from I\{i}has the smallest risk (on
the training set or on a validation set).
Naturally, there are many possible variants of the backward elimination idea.
It is also possible to combine forward and backward greedy steps.
25.1.3 Sparsity-Inducing Norms
The problem of minimizing the empirical risk subject to a budget of kfeatures
min
wLS(w) s.t.∥w∥0≤k,
where1
∥w∥0=|{i:wi̸= 0}|.

min
where1
In other words, we want wto be sparse, which implies that we only need to
measure the features corresponding to nonzero elements of w.
Solving this optimization problem is computationally hard (Natarajan 1995,
Davis, Mallat & Avellaneda 1997). A possible relaxation is to replace the non-
convex function∥w∥0with theℓ1norm,∥w∥1=∑d
i=1|wi|, and to solve the
problem
min
wLS(w) s.t.∥w∥1≤k1, (25.4)
wherek1is a parameter. Since the ℓ1norm is a convex function, this problem
can be solved eﬃciently as long as the loss function is convex. A related problem

problem
min
is minimizing the sum of LS(w) plus anℓ1norm regularization term,
min
w(LS(w) +λ∥w∥1), (25.5)
whereλis a regularization parameter. Since for any k1there exists a λsuch that
1The function∥·∥0is often referred to as the ℓ0norm. Despite the use of the “norm”
notation,∥·∥0is not really a norm; for example, it does not satisfy the positive
homogeneity property of norms, ∥aw∥0̸=|a|∥w∥0.

364 Feature Selection and Generation
Equation (25.4) and Equation (25.5) lead to the same solution, the two problems
are in some sense equivalent.
Theℓ1regularization often induces sparse solutions. To illustrate this, let us
start with the simple optimization problem
min
w∈R(1
2w2−xw+λ|w|)
. (25.6)
It is easy to verify (see Exercise 2) that the solution to this problem is the “soft
thresholding” operator
w= sign(x) [|x|−λ]+, (25.7)
where [a]+def= max{a,0}. That is, as long as the absolute value of xis smaller
thanλ, the optimal solution will be zero.
Next, consider a one dimensional regression problem with respect to the squared
loss:
argmin
w∈Rm(
1
2mm∑
i=1(xiw−yi)2+λ|w|)
.
We can rewrite the problem as
argmin
w∈Rm(
1
2(
1
m∑
ix2
i)
w2−(
1
mm∑
i=1xiyi)
w+λ|w|)
.
For simplicity let us assume that1
m∑
ix2
i= 1, and denote⟨x,y⟩=∑m
i=1xiyi;
then the optimal solution is
w= sign(⟨x,y⟩) [|⟨x,y⟩|/m−λ]+.
That is, the solution will be zero unless the correlation between the feature x

loss:
argmin
w∈Rm(
1
2mm∑
.
argmin
w∈Rm(
1
2(
1
m∑
ix2
i)
w2−(
1
mm∑
i=1xiyi)
w+λ|w|)
.
m∑
ix2
i=1xiyi;
and the labels vector yis larger than λ.
Remark 25.4 Unlike theℓ1norm, theℓ2norm does not induce sparse solutions.
Indeed, consider the problem above with an ℓ2regularization, namely,
argmin
w∈Rm(
1
2mm∑
i=1(xiw−yi)2+λw2)
.
Then, the optimal solution is
w=⟨x,y⟩/m
∥x∥2/m+ 2λ.
This solution will be nonzero even if the correlation between xandyis very small.
In contrast, as we have shown before, when using ℓ1regularization, wwill be
nonzero only if the correlation between xandyis larger than the regularization
parameterλ.

Addingℓ1regularization to a linear regression problem with the squared loss
yields the LASSO algorithm, deﬁned as
argmin
w(1
2m∥Xw−y∥2+λ∥w∥1)
. (25.8)
Under some assumptions on the distribution and the regularization parameter
λ, the LASSO will ﬁnd sparse solutions (see, for example, (Zhao & Yu 2006)
and the references therein). Another advantage of the ℓ1norm is that a vector
with lowℓ1norm can be “sparsiﬁed” (see, for example, (Shalev-Shwartz, Zhang
& Srebro 2010) and the references therein).
25.2 Feature Manipulation and Normalization
Feature manipulations or normalization include simple transformations that we
apply on each of our original features. Such transformations may decrease the
approximation or estimation errors of our hypothesis class or can yield a faster
algorithm. Similarly to the problem of feature selection, here again there are no
absolute “good” and “bad” transformations, but rather each transformation that

we apply should be related to the learning algorithm we are going to apply on
the resulting feature vector as well as to our prior assumptions on the problem.
To motivate normalization , consider a linear regression problem with the
squared loss. Let X∈Rm,dbe a matrix whose rows are the instance vectors
and let y∈Rmbe a vector of target values. Recall that ridge regression returns
the vector
argmin
w[1
m∥Xw−y∥2+λ∥w∥2]
= (2λmI +X⊤X)−1X⊤y.
Suppose that d= 2 and the underlying data distribution is as follows. First we

the vector
argmin
w[1
sampleyuniformly at random from {±1}. Then, we set x1to bey+0.5α, where
αis sampled uniformly at random from {±1}, and we set x2to be 0.0001y. Note
that the optimal weight vector is w⋆= [0; 10000], and LD(w⋆) = 0. However,
the objective of ridge regression at w⋆isλ108. In contrast, the objective of ridge
regression at w= [1; 0] is likely to be close to 0 .25 +λ. It follows that whenever
λ >0.25
108−1≈0.25×10−8, the objective of ridge regression is smaller at the
suboptimal solution w= [1; 0]. Since λtypically should be at least 1 /m(see

λ >0.25
the analysis in Chapter 13), it follows that in the aforementioned example, if the
number of examples is smaller than 108then we are likely to output a suboptimal
solution.
The crux of the preceding example is that the two features have completely
diﬀerent scales. Feature normalization can overcome this problem. There are
many ways to perform feature normalization, and one of the simplest approaches
is simply to make sure that each feature receives values between −1 and 1. In

solution.
the preceding example, if we divide each feature by the maximal value it attains

366 Feature Selection and Generation
we will obtain that x1=y+0.5α
1.5andx2=y. Then, for λ≤10−3the solution of
ridge regression is quite close to w⋆.
Moreover, the generalization bounds we have derived in Chapter 13 for reg-
ularized loss minimization depend on the norm of the optimal vector w⋆and
on the maximal norm of the instance vectors.2Therefore, in the aforementioned
example, before we normalize the features we have that ∥w⋆∥2= 108, while af-
ter we normalize the features we have that ∥w⋆∥2= 1. The maximal norm of
the instance vector remains roughly the same; hence the normalization greatly
improves the estimation error.
Feature normalization can also improve the runtime of the learning algorithm.
For example, in Section 14.5.3 we have shown how to use the Stochastic Gradient
Descent (SGD) optimization algorithm for solving the regularized loss minimiza-
tion problem. The number of iterations required by SGD to converge also depends

on the norm of w⋆and on the maximal norm of ∥x∥. Therefore, as before, using
normalization can greatly decrease the runtime of SGD.
Next, we demonstrate in the following how a simple transformation on features,
such as clipping , can sometime decrease the approximation error of our hypoth-
esis class. Consider again linear regression with the squared loss. Let a >1 be
a large number, suppose that the target yis chosen uniformly at random from
{±1}, and then the single feature xis set to be ywith probability (1 −1/a)

and set to be aywith probability 1 /a. That is, most of the time our feature is
bounded but with a very small probability it gets a very high value. Then, for
anyw, the expected squared loss of wis
LD(w) =E1
2(wx−y)2
=(
1−1
a)1
2(wy−y)2+1
a1
2(awy−y)2.
Solving for wwe obtain that w⋆=2a−1
a2+a−1, which goes to zero as agoes to inﬁn-
ity. Therefore, the objective at w⋆goes to 0.5 asagoes to inﬁnity. For example,
fora= 100 we will obtain LD(w⋆)≥0.48. Next, suppose we apply a “clipping”
transformation; that is, we use the transformation x↦→sign(x) min{1,|x|}. Then,

LD(w) =E1
2(wx−y)2
=(
1−1
a)1
a1
following this transformation, w⋆becomes 1 and LD(w⋆) = 0. This simple ex-
ample shows that a simple transformation can have a signiﬁcant inﬂuence on the
Of course, it is not hard to think of examples in which the same feature trans-
formation actually hurts performance and increases the approximation error.
This is not surprising, as we have already argued that feature transformations
2More precisely, the bounds we derived in Chapter 13 for regularized loss minimization

depend on∥w⋆∥2and on either the Lipschitzness or the smoothness of the loss function.
For linear predictors and loss functions of the form ℓ(w,(x,y)) =φ(⟨w,x⟩,y), whereφis
convex and either 1-Lipschitz or 1-smooth with respect to its ﬁrst argument, we have that
ℓis either∥x∥-Lipschitz or∥x∥2-smooth. For example, for the squared loss,
φ(a,y) =1
2(a−y)2, andℓ(w,(x,y)) =1
2(⟨w,x⟩−y)2is∥x∥2-smooth with respect to its
ﬁrst argument.

25.2 Feature Manipulation and Normalization 367
should rely on our prior assumptions on the problem. In the aforementioned ex-
ample, a prior assumption that may lead us to use the “clipping” transformation
is that features that get values larger than a predeﬁned threshold value give us no
additional useful information, and therefore we can clip them to the predeﬁned
threshold.
25.2.1 Examples of Feature Transformations
We now list several common techniques for feature transformations. Usually, it
is helpful to combine some of these transformations (e.g., centering + scaling).
In the following, we denote by f= (f1,...,fm)∈Rmthe value of the feature f
over themtraining examples. Also, we denote by ¯f=1
m∑m
i=1fithe empirical
mean of the feature over all examples.
Centering:
This transformation makes the feature have zero mean, by setting fi←fi−¯f.
Unit Range:
This transformation makes the range of each feature be [0 ,1]. Formally, let
fmax= maxifiandfmin= minifi. Then, we set fi←fi−fmin
fmax−fmin. Similarly,

m∑m
we can make the range of each feature be [ −1,1] by the transformation fi←
2fi−fmin
fmax−fmin−1. Of course, it is easy to make the range [0 ,b] or [−b,b], wherebis
a user-speciﬁed parameter.
Standardization:
This transformation makes all features have a zero mean and unit variance.
Formally, let ν=1
m∑m
i=1(fi−¯f)2be the empirical variance of the feature.
Then, we set fi←fi−¯f√ν.
Clipping:
This transformation clips high or low values of the feature. For example, fi←
sign(fi) max{b,|fi|}, wherebis a user-speciﬁed parameter.
Sigmoidal Transformation:

2fi−fmin
m∑m
Clipping:
As its name indicates, this transformation applies a sigmoid function on the
feature. For example, fi←1
1+exp(bfi), wherebis a user-speciﬁed parameter.
This transformation can be thought of as a “soft” version of clipping: It has a
small eﬀect on values close to zero and behaves similarly to clipping on values
far away from zero.

368 Feature Selection and Generation
Logarithmic Transformation:
The transformation is fi←log(b+fi), wherebis a user-speciﬁed parameter. This
is widely used when the feature is a “counting” feature. For example, suppose
that the feature represents the number of appearances of a certain word in a
text document. Then, the diﬀerence between zero occurrences of the word and
a single occurrence is much more important than the diﬀerence between 1000
occurrences and 1001 occurrences.
Remark 25.5 In the aforementioned transformations, each feature is trans-
formed on the basis of the values it obtains on the training set, independently
of other features’ values. In some situations we would like to set the parameter
of the transformation on the basis of other features as well. A notable example
is a transformation in which one applies a scaling to the features so that the
empirical average of some norm of the instances becomes 1.
25.3 Feature Learning

So far we have discussed feature selection and manipulations. In these cases, we
start with a predeﬁned vector space Rd, representing our features. Then, we select
a subset of features (feature selection) or transform individual features (feature
transformation). In this section we describe feature learning , in which we start
with some instance space, X, and would like to learn a function, ψ:X →Rd,
which maps instances in Xinto a representation as d-dimensional feature vectors.

The idea of feature learning is to automate the process of ﬁnding a good rep-
resentation of the input space. As mentioned before, the No-Free-Lunch theorem
tells us that we must incorporate some prior knowledge on the data distribution
in order to build a good feature representation. In this section we present a few
feature learning approaches and demonstrate conditions on the underlying data
distribution in which these methods can be useful.

Throughout the book we have already seen several useful feature construc-
tions. For example, in the context of polynomial regression, we have mapped the
original instances into the vector space of all their monomials (see Section 9.2.2
in Chapter 9). After performing this mapping, we trained a linear predictor on
top of the constructed features. Automation of this process would be to learn
a transformation ψ:X →Rd, such that the composition of the class of linear
predictors on top of ψyields a good hypothesis class for the task at hand.

In the following we describe a technique of feature construction called dictio-
nary learning .
25.3.1 Dictionary Learning Using Auto-Encoders
The motivation of dictionary learning stems from a commonly used represen-
tation of documents as a “bag-of-words”: Given a dictionary of words D=
{w1,...,wk}, where each wiis a string representing a word in the dictionary,

25.3 Feature Learning 369
and given a document, ( p1,...,pd), where each piis a word in the document,
we represent the document as a vector x∈{0,1}k, wherexiis 1 ifwi=pjfor
somej∈[d], andxi= 0 otherwise. It was empirically observed in many text
processing tasks that linear predictors are quite powerful when applied on this
representation. Intuitively, we can think of each word as a feature that measures
some aspect of the document. Given labeled examples (e.g., topics of the doc-
uments), a learning algorithm searches for a linear predictor that weights these
features so that a right combination of appearances of words is indicative of the
label.
While in text processing there is a natural meaning to words and to the dic-
tionary, in other applications we do not have such an intuitive representation
of an instance. For example, consider the computer vision application of object
recognition. Here, the instance is an image and the goal is to recognize which

label.
object appears in the image. Applying a linear predictor on the pixel-based rep-
resentation of the image does not yield a good classiﬁer. What we would like
to have is a mapping ψthat would take the pixel-based representation of the
image and would output a bag of “visual words,” representing the content of the
image. For example, a “visual word” can be “there is an eye in the image.” If
we had such representation, we could have applied a linear predictor on top of

this representation to train a classiﬁer for, say, face recognition. Our question is,
therefore, how can we learn a dictionary of “visual words” such that a bag-of-
words representation of an image would be helpful for predicting which object
appears in the image?
A ﬁrst naive approach for dictionary learning relies on a clustering algorithm
(see Chapter 22). Suppose that we learn a function c:X →{ 1,...,k}, where
c(x) is the cluster to which xbelongs. Then, we can think of the clusters as

“words,” and of instances as “documents,” where a document xis mapped to
the vectorψ(x)∈{0,1}k, whereψ(x)iis 1 if and only if xbelongs to the ith
cluster. Now, it is straightforward to see that applying a linear predictor on ψ(x)
is equivalent to assigning the same target value to all instances that belong to
the same cluster. Furthermore, if the clustering is based on distances from a
class center (e.g., k-means), then a linear predictor on ψ(x) yields a piece-wise
constant predictor on x.

Both thek-means and PCA approaches can be regarded as special cases of a
more general approach for dictionary learning which is called auto-encoders . In an
auto-encoder we learn a pair of functions: an “encoder” function, ψ:Rd→Rk,
and a “decoder” function, φ:Rk→Rd. The goal of the learning process is to
ﬁnd a pair of functions such that the reconstruction error,∑
i∥xi−φ(ψ(xi))∥2,
is small. Of course, we can trivially set k=dand bothψ,φ to be the identity

mapping, which yields a perfect reconstruction. We therefore must restrict ψand
φin some way. In PCA, we constrain k <d and further restrict ψandφto be
linear functions. In k-means,kis not restricted to be smaller than d, but now
ψandφrely onkcentroids,µ1,...,µk, andψ(x) returns an indicator vector

370 Feature Selection and Generation
in{0,1}kthat indicates the closest centroid to x, whileφtakes as input an
indicator vector and returns the centroid representing this vector.
An important property of the k-means construction, which is key in allowing
kto be larger than d, is thatψmaps instances into sparse vectors. In fact, in
k-means only a single coordinate of ψ(x) is nonzero. An immediate extension of
thek-means construction is therefore to restrict the range of ψto be vectors with
at mostsnonzero elements, where sis a small integer. In particular, let ψandφ
be functions that depend on µ1,...,µk. The function ψmaps an instance vector
xto a vector ψ(x)∈Rk, whereψ(x) should have at most snonzero elements.
The function φ(v) is deﬁned to be∑k
i=1viµi. As before, our goal is to have a
small reconstruction error, and therefore we can deﬁne
ψ(x) = argmin
v∥x−φ(v)∥2s.t.∥v∥0≤s,
where∥v∥0=|{j:vj̸= 0}|. Note that when s= 1 and we further restrict ∥v∥1=

1 then we obtain the k-means encoding function; that is, ψ(x) is the indicator
vector of the centroid closest to x. For larger values of s, the optimization problem
in the preceding deﬁnition of ψbecomes computationally diﬃcult. Therefore, in
practice, we sometime use ℓ1regularization instead of the sparsity constraint and
deﬁneψto be
v[
∥x−φ(v)∥2+λ∥v∥1]
,
whereλ > 0 is a regularization parameter. Anyway, the dictionary learning
problem is now to ﬁnd the vectors µ1,...,µksuch that the reconstruction er-
ror,∑m

v[
,
ror,∑m
i=1∥xi−φ(ψ(x))∥2, is as small as possible. Even if ψis deﬁned using
theℓ1regularization, this is still a computationally hard problem (similar to
thek-means problem). However, several heuristic search algorithms may give
reasonably good solutions. These algorithms are beyond the scope of this book.
25.4 Summary
Many machine learning algorithms take the feature representation of instances
for granted. Yet the choice of representation requires careful attention. We dis-
cussed approaches for feature selection, introducing ﬁlters, greedy selection al-

gorithms, and sparsity-inducing norms. Next we presented several examples for
feature transformations and demonstrated their usefulness. Last, we discussed
feature learning, and in particular dictionary learning. We have shown that fea-
ture selection, manipulation, and learning all depend on some prior knowledge

25.5 Bibliographic Remarks
Guyon & Elisseeﬀ (2003) surveyed several feature selection procedures, including
many types of ﬁlters.
Forward greedy selection procedures for minimizing a convex objective sub-
ject to a polyhedron constraint date back to the Frank-Wolfe algorithm (Frank
& Wolfe 1956). The relation to boosting has been studied by several authors,
including, (Warmuth, Liao & Ratsch 2006, Warmuth, Glocer & Vishwanathan
2008, Shalev-Shwartz & Singer 2008). Matching pursuit has been studied in the
signal processing community (Mallat & Zhang 1993). Several papers analyzed
greedy selection methods under various conditions. See, for example, Shalev-
Shwartz, Zhang & Srebro (2010) and the references therein.
The use of the ℓ1-norm as a surrogate for sparsity has a long history (e.g. Tib-
shirani (1996) and the references therein), and much work has been done on un-
derstanding the relationship between the ℓ1-norm and sparsity. It is also closely

related to compressed sensing (see Chapter 23). The ability to sparsify low ℓ1
norm predictors dates back to Maurey (Pisier 1980-1981). In Section 26.4 we
also show that low ℓ1norm can be used to bound the estimation error of our
predictor.
Feature learning and dictionary learning have been extensively studied recently
in the context of deep neural networks. See, for example, (Lecun & Bengio 1995,
Hinton et al. 2006, Ranzato et al. 2007, Collobert & Weston 2008, Lee et al.
2009, Le et al. 2012, Bengio 2009) and the references therein.
25.6 Exercises

1. Prove the equality given in Equation (25.1). Hint: Leta∗,b∗be minimizers of
the left-hand side. Find a,bsuch that the objective value of the right-hand
side is smaller than that of the left-hand side. Do the same for the other
direction.
2. Show that Equation (25.7) is the solution of Equation (25.6).
3.AdaBoost as a Forward Greedy Selection Algorithm: Recall the Ad-
aBoost algorithm from Chapter 10. In this section we give another interpre-
tation of AdaBoost as a forward greedy selection algorithm.

•Given a set of minstances x1,...,xm, and a hypothesis class Hof ﬁnite
VC dimension, show that there exist dandh1,...,hdsuch that for every
h∈H there exists i∈[d] withhi(xj) =h(xj) for every j∈[m].
•LetR(w) be as deﬁned in Equation (25.3). Given some w, deﬁnefwto be
fw(·) =d∑
i=1wihi(·).

372 Feature Selection and Generation
LetDbe the distribution over [ m] deﬁned by
Di=exp(−yifw(xi))
Z,
whereZis a normalization factor that ensures that Dis a probability
vector. Show that
∂R(w)
wj=−m∑
i=1Diyihj(xi).
Furthermore, denoting ϵj=∑m
i=1Di1[hj(xi)̸=yi], show that
∂R(w)
wj= 2ϵj−1.
Conclude that if ϵj≤1/2−γthen⏐⏐⏐∂R(w)
wj⏐⏐⏐≥γ/2.
•Show that the update of AdaBoost guarantees R(w(t+1))−R(w(t))≤
log(√
1−4γ2).Hint: Use the proof of Theorem 10.2.

26 Rademacher Complexities
In Chapter 4 we have shown that uniform convergence is a suﬃcient condition
for learnability. In this chapter we study the Rademacher complexity, which
measures the rate of uniform convergence. We will provide generalization bounds
based on this measure.
26.1 The Rademacher Complexity
Recall the deﬁnition of an ϵ-representative sample from Chapter 4, repeated here
for convenience.
definition 26.1 (ϵ-Representative Sample) A training set Sis calledϵ-representative
sup
h∈H|LD(h)−LS(h)|≤ϵ.
We have shown that if Sis anϵ/2 representative sample then the ERM rule
isϵ-consistent, namely, LD(ERMH(S))≤minh∈HLD(h) +ϵ.
To simplify our notation, let us denote
Fdef=ℓ◦Hdef={z↦→ℓ(h,z) :h∈H},
and givenf∈F, we deﬁne
LD(f) =E
z∼D[f(z)], LS(f) =1
mm∑
i=1f(zi).
We deﬁne the representativeness ofSwith respect toFas the largest gap be-
tween the true error of a function fand its empirical error, namely,
RepD(F,S)def= sup
f∈F(

sup
LD(f) =E
mm∑
i=1f(zi).
f∈F(
LD(f)−LS(f))
. (26.1)
Now, suppose we would like to estimate the representativeness of Susing the
sampleSonly. One simple idea is to split Sinto two disjoint sets, S=S1∪S2;
refer toS1as a validation set and to S2as a training set. We can then estimate
the representativeness of Sby
sup
f∈F(
LS1(f)−LS2(f))
. (26.2)

376 Rademacher Complexities
This can be written more compactly by deﬁning σ= (σ1,...,σm)∈{± 1}mto
be a vector such that S1={zi:σi= 1}andS2={zi:σi=−1}. Then, if we
further assume that |S1|=|S2|then Equation (26.2) can be rewritten as
2
msup
f∈Fm∑
i=1σif(zi). (26.3)
The Rademacher complexity measure captures this idea by considering the ex-
pectation of the above with respect to a random choice of σ. Formally, letF◦S
be the set of all possible evaluations a function f∈F can achieve on a sample
S, namely,
F◦S={(f(z1),...,f (zm)) :f∈F}.
Let the variables in σbe distributed i.i.d. according to P[σi= 1] = P[σi=−1] =
1
2. Then, the Rademacher complexity of Fwith respect to Sis deﬁned as follows:
R(F◦S)def=1
mE
σ∼{±1}m[
sup
f∈Fm∑
i=1σif(zi)]
. (26.4)
More generally, given a set of vectors, A⊂Rm, we deﬁne
R(A)def=1
mE
σ[
sup
a∈Am∑
i=1σiai]
. (26.5)
The following lemma bounds the expected value of the representativeness of
Sby twice the expected Rademacher complexity.
lemma 26.2
E
S∼Dm[ RepD(F,S)]≤2E
S∼DmR(F◦S).

1
mE
σ∼{±1}m[
sup
f∈Fm∑
. (26.4)
R(A)def=1
mE
σ[
sup
a∈Am∑
i=1σiai]
. (26.5)
lemma 26.2
E
Proof LetS′={z′
1,...,z′
m}be another i.i.d. sample. Clearly, for all f∈F,
LD(f) =ES′[LS′(f)]. Therefore, for every f∈F we have
LD(f)−LS(f) = E
S′[LS′(f)]−LS(f) = E
S′[LS′(f)−LS(f)].
Taking supremum over f∈Fof both sides, and using the fact that the supremum
of expectation is smaller than expectation of the supremum we obtain
sup
f∈F(
= sup
f∈FE
S′[LS′(f)−LS(f)]
≤E
S′[
sup
f∈F(
LS′(f)−LS(f))]
.
Taking expectation over Son both sides we obtain
E
S[
sup
f∈F(
LD(f)−LS(f))]
≤E
S,S′[
sup
f∈F(

1,...,z′
sup
f∈F(
= sup
f∈FE
≤E
S′[
sup
f∈F(
.
E
S[
sup
f∈F(
≤E
S,S′[
sup
f∈F(
=1
mE
S,S′[
sup
f∈Fm∑
i=1(f(z′
i)−f(zi))]
.(26.6)

26.1 The Rademacher Complexity 377
Next, we note that for each j,zjandz′
jare i.i.d. variables. Therefore, we can
replace them without aﬀecting the expectation:
E
S,S′
sup
f∈F
(f(z′
j)−f(zj)) +∑
i̸=j(f(z′
i)−f(zi))

=
E
S,S′
sup
f∈F
(f(zj)−f(z′
j)) +∑
i̸=j(f(z′

.(26.7)
Letσjbe a random variable such that P[σj= 1] = P[σj=−1] = 1/2. From
Equation (26.7) we obtain that
E
S,S′,σj
sup
f∈F
σj(f(z′
i̸=j(f(z′


=1
2(l.h.s. of Equation (26.7)) +1
2(r.h.s. of Equation (26.7))
=E
S,S′
sup
f∈F
(f(z′
i̸=j(f(z′

.(26.8)
Repeating this for all jwe obtain that
E
S,S′[
sup
f∈Fm∑
i=1(f(z′
=E
S,S′,σ[
sup
f∈Fm∑
i=1σi(f(z′
.(26.9)
Finally,
sup
f∈F∑
iσi(f(z′
i)−f(zi))≤sup
f∈F∑
iσif(z′
i) + sup
f∈F∑
i−σif(zi)
and since the probability of σis the same as the probability of −σ, the right-hand
side of Equation (26.9) can be bounded by
E
S,S′,σ[
sup
f∈F∑
iσif(z′
i) + sup
f∈F∑
iσif(zi)]
=mE
S′[R(F◦S′)] +mE
S[R(F◦S)] = 2mE

=E
S,S′
sup
f∈F
(f(z′
i̸=j(f(z′

.(26.8)
E
S,S′[
sup
f∈Fm∑
i=1(f(z′
=E
S,S′,σ[
sup
f∈Fm∑
.(26.9)
Finally,
sup
f∈F∑
iσi(f(z′
f∈F∑
iσif(z′
i) + sup
f∈F∑
i−σif(zi)
E
S,S′,σ[
sup
f∈F∑
iσif(z′
i) + sup
f∈F∑
iσif(zi)]
=mE
S[R(F◦S)].
The lemma immediately yields that, in expectation, the ERM rule ﬁnds a
hypothesis which is close to the optimal hypothesis in H.
theorem 26.3 We have
E
S∼Dm[LD(ERMH(S))−LS(ERMH(S))]≤2E
S∼DmR(ℓ◦H◦S).
Furthermore, for any h⋆∈H
E
S∼Dm[LD(ERMH(S))−LD(h⋆)]≤2E

378 Rademacher Complexities
Furthermore, if h⋆= argminhLD(h)then for each δ∈(0,1)with probability of
at least 1−δover the choice of Swe have
LD(ERMH(S))−LD(h⋆)≤2ES′∼DmR(ℓ◦H◦S′)
δ.
Proof The ﬁrst inequality follows directly from Lemma 26.2. The second in-
equality follows because for any ﬁxed h⋆,
LD(h⋆) =E
S[LS(h⋆)]≥E
S[LS(ERMH(S))].
The third inequality follows from the previous inequality by relying on Markov’s
inequality (note that the random variable LD(ERMH(S))−LD(h⋆) is nonnega-
tive).
Next, we derive bounds similar to the bounds in Theorem 26.3 with a better
dependence on the conﬁdence parameter δ. To do so, we ﬁrst introduce the
following bounded diﬀerences concentration inequality.
lemma 26.4 (McDiarmid’s Inequality) LetVbe some set and let f:Vm→R
be a function of mvariables such that for some c>0, for alli∈[m]and for all
x1,...,xm,x′
i∈Vwe have
|f(x1,...,xm)−f(x1,...,xi−1,x′
i,xi+1,...,xm)|≤c.
LetX1,...,Xmbemindependent random variables taking values in V. Then,

tive).
i∈Vwe have
with probability of at least 1−δwe have
|f(X1,...,Xm)−E[f(X1,...,Xm)]|≤c√
ln(2
δ)
m/2.
On the basis of the McDiarmid inequality we can derive generalization bounds
with a better dependence on the conﬁdence parameter.
theorem 26.5 Assume that for all zandh∈H we have that|ℓ(h,z)|≤c.
Then,
1. With probability of at least 1−δ, for allh∈H,
LD(h)−LS(h)≤2E
S′∼DmR(ℓ◦H◦S′) +c√
2 ln(2/δ)
m.
In particular, this holds for h= ERMH(S).
2. With probability of at least 1−δ, for allh∈H,
LD(h)−LS(h)≤2R(ℓ◦H◦S) + 4c√
2 ln(4/δ)
m.

ln(2
δ)
m/2.
Then,
2 ln(2/δ)
m.
2 ln(4/δ)
m.
3. For any h⋆, with probability of at least 1−δ,
LD(ERMH(S))−LD(h⋆)≤2R(ℓ◦H◦S) + 5c√
2 ln (8/δ)
m.

26.1 The Rademacher Complexity 379
Proof First note that the random variable RepD(F,S) = suph∈H(LD(h)−LS(h))
satisﬁes the bounded diﬀerences condition of Lemma 26.4 with a constant 2 c/m.
Combining the bounds in Lemma 26.4 with Lemma 26.2 we obtain that with
probability of at least 1 −δ,
RepD(F,S)≤ERepD(F,S) +c√
2 ln(2/δ)
m≤2E
S′R(ℓ◦H◦S′) +c√
2 ln(2/δ)
m.
The ﬁrst inequality of the theorem follows from the deﬁnition of RepD(F,S).
For the second inequality we note that the random variable R(ℓ◦H◦S) also
Therefore, the second inequality follows from the ﬁrst inequality, Lemma 26.4,
and the union bound. Finally, for the last inequality, denote hS= ERMH(S)
and note that
LD(hS)−LD(h⋆)
=LD(hS)−LS(hS) +LS(hS)−LS(h⋆) +LS(h⋆)−LD(h⋆)
≤(LD(hS)−LS(hS)) + (LS(h⋆)−LD(h⋆)). (26.10)
The ﬁrst summand on the right-hand side is bounded by the second inequality of
the theorem. For the second summand, we use the fact that h⋆does not depend

onS; hence by using Hoeﬀding’s inequality we obtain that with probaility of at
least 1−δ/2,
LS(h⋆)−LD(h⋆)≤c√
ln(4/δ)
2m. (26.11)
Combining this with the union bound we conclude our proof.
The preceding theorem tells us that if the quantity R(ℓ◦H◦S) is small then it
is possible to learn the class Husing the ERM rule. It is important to emphasize
that the last two bounds given in the theorem depend on the speciﬁc training
setS. That is, we use Sboth for learning a hypothesis from Has well as for

ln(4/δ)
estimating the quality of it. This type of bound is called a data-dependent bound .
26.1.1 Rademacher Calculus
Let us now discuss some properties of the Rademacher complexity measure.
These properties will help us in deriving some simple bounds on R(ℓ◦H◦S) for
speciﬁc cases of interest.
The following lemma is immediate from the deﬁnition.
lemma 26.6 For anyA⊂Rm, scalarc∈R, and vector a0∈Rm, we have
R({ca+a0:a∈A})≤|c|R(A).
The following lemma tells us that the convex hull of Ahas the same complexity
asA.

380 Rademacher Complexities
lemma 26.7 LetAbe a subset of Rmand letA′={∑N
j=1αja(j):N∈
N,∀j,a(j)∈A,αj≥0,∥α∥1= 1}. Then,R(A′) =R(A).
Proof The main idea follows from the fact that for any vector vwe have
sup
α≥0:∥α∥1=1N∑
j=1αjvj= max
jvj.
mR(A′) =E
σsup
α≥0:∥α∥1=1sup
a(1),...,a(N)m∑
i=1σiN∑
j=1αja(j)
i
=E
σsup
j=1αjsup
a(j)m∑
i=1σia(j)
i
=E
σsup
a∈Am∑
i=1σiai
=mR(A),
and we conclude our proof.
The next lemma, due to Massart, states that the Rademacher complexity of
a ﬁnite set grows logarithmically with the size of the set.
lemma 26.8 (Massart lemma) LetA={a1,...,aN}be a ﬁnite set of vectors
inRm. Deﬁne ¯a=1
N∑N
i=1ai. Then,
R(A)≤max
a∈A∥a−¯a∥√
2 log(N)
m.
Proof Based on Lemma 26.6, we can assume without loss of generality that
¯a=0. Letλ>0 and letA′={λa1,...,λ aN}. We upper bound the Rademacher
complexity as follows:
mR(A′) =E
σ[
max
a∈A′⟨σ,a⟩]
=E
σ[
log(
max
a∈A′e⟨σ,a⟩)]
≤E
σ[
log(∑
≤log(
E
σ[∑
a∈A′e⟨σ,a⟩])
// Jensen’s inequality
= log(∑
a∈A′m∏
i=1E
σi[eσiai])
,

N∑N
R(A)≤max
2 log(N)
m.
mR(A′) =E
σ[
max
=E
σ[
log(
max
≤E
σ[
log(∑
≤log(
E
σ[∑
= log(∑
a∈A′m∏
i=1E
,
where the last equality occurs because the Rademacher variables are indepen-
dent. Next, using Lemma A.6 we have that for all ai∈R,
E
σieσiai=exp(ai) + exp(−ai)
2≤exp(a2
i/2),

26.1 The Rademacher Complexity 381
mR(A′)≤log(∑
a∈A′m∏
i=1exp(a2
i
2))
= log(∑
a∈A′exp(
∥a∥2/2))
≤log(
|A′|max
a∈A′exp(
∥a∥2/2))
= log(|A′|) + max
a∈A′(∥a∥2/2).
SinceR(A) =1
λR(A′) we obtain from the equation that
R(A)≤log(|A|) +λ2maxa∈A(∥a∥2/2)
λm.
Settingλ=√
2 log(|A|)/maxa∈A∥a∥2and rearranging terms we conclude our
proof.
The following lemma shows that composing Awith a Lipschitz function does
not blow up the Rademacher complexity. The proof is due to Kakade and Tewari.
lemma 26.9 (Contraction lemma) For eachi∈[m], letφi:R→Rbe aρ-
Lipschitz function, namely for all α,β∈Rwe have|φi(α)−φi(β)|≤ρ|α−β|.
Fora∈Rmletφ(a)denote the vector (φ1(a1),...,φm(ym)). Letφ◦A={φ(a) :
a∈A}. Then,
R(φ◦A)≤ρR(A).
Proof For simplicity, we prove the lemma for the case ρ= 1. The case ρ̸=
1 will follow by deﬁning φ′=1
ρφand then using Lemma 26.6. Let Ai=
{(a1,...,ai−1,φi(ai),ai+1,...,am) :a∈A}. Clearly, it suﬃces to prove that
for any setAand alliwe haveR(Ai)≤R(A). Without loss of generality we will

prove the latter claim for i= 1 and to simplify notation we omit the subscript
fromφ1. We have
mR(A1) =E
σ[
sup
a∈A1m∑
i=1σiai]
=E
σ[
sup
a∈Aσ1φ(a1) +m∑
i=2σiai]
=1
2E
σ2,...,σm[
sup
a∈A(
φ(a1) +m∑
i=2σiai)
+ sup
a∈A(
−φ(a1) +m∑
i=2σiai)]
=1
2E
sup
a,a′∈A(
φ(a1)−φ(a′
1) +m∑
i=2σiai+m∑
i=2σia′
i)]
≤1
2E
sup
a,a′∈A(
|a1−a′
1|+m∑
i=2σia′
i)]
, (26.12)
where in the last inequality we used the assumption that φis Lipschitz. Next,
we note that the absolute value on |a1−a′
1|in the preceding expression can

382 Rademacher Complexities
be omitted since both aanda′are from the same set Aand the rest of the
expression in the supremum is not aﬀected by replacing aanda′. Therefore,
mR(A1)≤1
2E
sup
a,a′∈A(
a1−a′
1+m∑
i=2σia′
i)]
. (26.13)
But, using the same equalities as in Equation (26.12), it is easy to see that the
right-hand side of Equation (26.13) exactly equals mR(A), which concludes our
proof.
26.2 Rademacher Complexity of Linear Classes
In this section we analyze the Rademacher complexity of linear classes. To sim-
plify the derivation we ﬁrst deﬁne the following two classes:
H1={x↦→⟨w,x⟩:∥w∥1≤1},H2={x↦→⟨w,x⟩:∥w∥2≤1}.(26.14)
The following lemma bounds the Rademacher complexity of H2. We allow
thexito be vectors in any Hilbert space (even inﬁnite dimensional), and the
bound does not depend on the dimensionality of the Hilbert space. This property
becomes useful when analyzing kernel methods.
lemma 26.10 LetS= (x1,...,xm)be vectors in a Hilbert space. Deﬁne: H2◦

S={(⟨w,x1⟩,...,⟨w,xm⟩) :∥w∥2≤1}. Then,
R(H2◦S)≤maxi∥xi∥2√m.
Proof Using Cauchy-Schwartz inequality we know that for any vectors w,vwe
have⟨w,v⟩≤∥w∥∥v∥. Therefore,
mR(H2◦S) =E
σ[
sup
a∈H2◦Sm∑
i=1σiai]
(26.15)
=E
σ[
sup
w:∥w∥≤1m∑
i=1σi⟨w,xi⟩]
=E
σ[
sup
w:∥w∥≤1⟨w,m∑
i=1σixi⟩]
≤E
σ[
∥m∑
i=1σixi∥2]
.
Next, using Jensen’s inequality we have that
E
σ[m∑
i=1σixi
2]
=E
σ

m∑
i=1σixi2
2
1/2
≤
E
σ
m∑
2

1/2
.(26.16)

Finally, since the variables σ1,...,σmare independent we have
E
σ[
∥m∑
i=1σixi∥2
2]
=E
σ
∑
i,jσiσj⟨xi,xj⟩

=∑
i̸=j⟨xi,xj⟩E
σ[σiσj] +m∑
i=1⟨xi,xi⟩E
σ[
σ2
i]
=m∑
i=1∥xi∥2
2≤mmax
i∥xi∥2
2.
Combining this with Equation (26.15) and Equation (26.16) we conclude our
proof.
Next we bound the Rademacher complexity of H1◦S.
lemma 26.11 LetS= (x1,...,xm)be vectors in Rn. Then,
R(H1◦S)≤max
i∥xi∥∞√
2 log(2n)
m.
Proof Using Holder’s inequality we know that for any vectors w,vwe have
⟨w,v⟩≤∥w∥1∥v∥∞. Therefore,
mR(H1◦S) =E
σ[
sup
a∈H1◦Sm∑
i=1σiai]
=E
σ[
sup
w:∥w∥1≤1m∑
=E
σ[
sup
w:∥w∥1≤1⟨w,m∑
i=1σixi⟩]
≤E
σ[
∥m∑
i=1σixi∥∞]
. (26.17)
For eachj∈[n], letvj= (x1,j,...,xm,j)∈Rm. Note that∥vj∥2≤√mmaxi∥xi∥∞.
LetV={v1,...,vn,−v1,...,−vn}. The right-hand side of Equation (26.17) is
mR(V). Using Massart lemma (Lemma 26.8) we have that
R(V)≤max
i∥xi∥∞√
2 log(2n)/m,
26.3 Generalization Bounds for SVM

σ[
sup
a∈H1◦Sm∑
i=1σiai]
=E
σ[
sup
=E
σ[
sup
i=1σixi⟩]
≤E
σ[
∥m∑
. (26.17)
R(V)≤max
i∥xi∥∞√
In this section we use Rademacher complexity to derive generalization bounds
for generalized linear predictors with Euclidean norm constraint. We will show
how this leads to generalization bounds for hard-SVM and soft-SVM.

384 Rademacher Complexities
We shall consider the following general constraint-based formulation. Let H=
{w:∥w∥2≤B}be our hypothesis class, and let Z=X×Y be the examples
domain. Assume that the loss function ℓ:H×Z→Ris of the form
ℓ(w,(x,y)) =φ(⟨w,x⟩,y), (26.18)
whereφ:R×Y→ Ris such that for all y∈Y, the scalar function a↦→φ(a,y)
isρ-Lipschitz. For example, the hinge-loss function, ℓ(w,(x,y)) = max{0,1−
y⟨w,x⟩}, can be written as in Equation (26.18) using φ(a,y) = max{0,1−
ya}, and note that φis 1-Lipschitz for all y∈{± 1}. Another example is the
absolute loss function, ℓ(w,(x,y)) =|⟨w,x⟩−y|, which can be written as in
Equation (26.18) using φ(a,y) =|a−y|, which is also 1-Lipschitz for all y∈R.
The following theorem bounds the generalization error of all predictors in H
using their empirical error.
theorem 26.12 Suppose thatDis a distribution over X×Y such that with
probability 1we have that∥x∥2≤R. LetH={w:∥w∥2≤B}and let
ℓ:H×Z→Rbe a loss function of the form given in Equation (26.18)

such that for all y∈Y,a↦→φ(a,y)is aρ-Lipschitz function and such that
maxa∈[−BR,BR ]|φ(a,y)|≤c. Then, for any δ∈(0,1), with probability of at least
1−δover the choice of an i.i.d. sample of size m,
∀w∈H, LD(w)≤LS(w) +2ρBR√m+c√
2 ln(2/δ)
m.
Proof LetF={(x,y)↦→φ(⟨w,x⟩,y) :w∈ H} . We will show that with
probability 1, R(F◦S)≤ρBR/√mand then the theorem will follow from
Theorem 26.5. Indeed, the set F◦Scan be written as
F◦S={(φ(⟨w,x1⟩,y1),...,φ (⟨w,xm⟩,ym)) :w∈H},
and the bound on R(F◦S) follows directly by combining Lemma 26.9, Lemma 26.10,

2 ln(2/δ)
m.
and the assumption that ∥x∥2≤Rwith probability 1.
We next derive a generalization bound for hard-SVM based on the previous
theorem. For simplicity, we do not allow a bias term and consider the hard-SVM
problem:
argmin
w∥w∥2s.t.∀i, yi⟨w,xi⟩≥1 (26.19)
theorem 26.13 Consider a distribution DoverX×{± 1}such that there exists
some vector w⋆withP(x,y)∼D[y⟨w⋆,x⟩≥1] = 1 and such that∥x∥2≤Rwith
probability 1. Let wSbe the output of Equation (26.19). Then, with probability
of at least 1−δover the choice of S∼Dm, we have that
P

problem:
argmin
P
(x,y)∼D[y̸=sign(⟨wS,x⟩)]≤2R∥w⋆∥√m+ (1 +R∥w⋆∥)√
2 ln(2/δ)
m.

26.3 Generalization Bounds for SVM 385
Proof Throughout the proof, let the loss function be the ramp loss (see Sec-
tion 15.2.3). Note that the range of the ramp loss is [0 ,1] and that it is a
1-Lipschitz function. Since the ramp loss upper bounds the zero-one loss, we
have that
P
(x,y)∼D[y̸= sign(⟨wS,x⟩)]≤LD(wS).
LetB=∥w⋆∥2and consider the set H={w:∥w∥2≤B}. By the deﬁnition of
hard-SVM and our assumption on the distribution, we have that wS∈H with
probability 1 and that LS(wS) = 0. Therefore, using Theorem 26.12 we have
that
LD(wS)≤LS(wS) +2BR√m+√
2 ln(2/δ)
m.
Remark 26.1 Theorem 26.13 implies that the sample complexity of hard-SVM
grows likeR2∥w⋆∥2
ϵ2. Using a more delicate analysis and the separability assump-
tion, it is possible to improve the bound to an order ofR2∥w⋆∥2
ϵ.
The bound in the preceding theorem depends on ∥w⋆∥, which is unknown.
In the following we derive a bound that depends on the norm of the output of
SVM; hence it can be calculated from the training set itself. The proof is similar

that
2 ln(2/δ)
m.
ϵ.
to the derivation of bounds for structure risk minimization (SRM).
theorem 26.14 Assume that the conditions of Theorem 26.13 hold. Then,
with probability of at least 1−δover the choice of S∼Dm, we have that
P
(x,y)∼D[y̸=sign(⟨wS,x⟩)]≤4R∥wS∥√m+√
ln(4 log2(∥wS∥)
δ)
m.
Proof For any integer i, letBi= 2i,Hi={w:∥w∥≤Bi}, and letδi=δ
2i2.
Fixi, then using Theorem 26.12 we have that with probability of at least 1 −δi
∀w∈Hi, LD(w)≤LS(w) +2BiR√m+√
2 ln(2/δi)
m
Applying the union bound and using∑∞
i=1δi≤δwe obtain that with probability

P
δ)
m.
2i2.
2 ln(2/δi)
m
of at least 1−δthis holds for all i. Therefore, for all w, if we leti=⌈log2(∥w∥)⌉
thenw∈Hi,Bi≤2∥w∥, and2
δi=(2i)2
δ≤(4 log2(∥w∥))2
δ. Therefore,
LD(w)≤LS(w) +2BiR√m+√
2 ln(2/δi)
m
≤LS(w) +4∥w∥R√m+√
4(ln(4 log2(∥w∥)) + ln(1/δ))
m.
In particular, it holds for wS, which concludes our proof.

386 Rademacher Complexities
Remark 26.2 Note that all the bounds we have derived do not depend on the
dimension of w. This property is utilized when learning SVM with kernels, where
the dimension of wcan be extremely large.
26.4 Generalization Bounds for Predictors with Low ℓ1Norm
In the previous section we derived generalization bounds for linear predictors
with anℓ2-norm constraint. In this section we consider the following general ℓ1-
norm constraint formulation. Let H={w:∥w∥1≤B}be our hypothesis class,
and letZ=X×Y be the examples domain. Assume that the loss function,
ℓ:H×Z→R, is of the same form as in Equation (26.18), with φ:R×Y→ R
beingρ-Lipschitz w.r.t. its ﬁrst argument. The following theorem bounds the
generalization error of all predictors in Husing their empirical error.
theorem 26.15 Suppose thatDis a distribution over X×Y such that with
probability 1we have that∥x∥∞≤R. LetH={w∈Rd:∥w∥1≤B}and
letℓ:H×Z→Rbe a loss function of the form given in Equation (26.18)

such that for all y∈Y,a↦→φ(a,y)is anρ-Lipschitz function and such that
∀w∈H, LD(w)≤LS(w) + 2ρBR√
2 log(2d)
m+c√
2 ln(2/δ)
m.
Proof The proof is identical to the proof of Theorem 26.12, while relying on
Lemma 26.11 instead of relying on Lemma 26.10.
It is interesting to compare the two bounds given in Theorem 26.12 and The-
orem 26.15. Apart from the extra log( d) factor that appears in Theorem 26.15,

2 log(2d)
m+c√
2 ln(2/δ)
m.
both bounds look similar. However, the parameters B,R have diﬀerent meanings
in the two bounds. In Theorem 26.12, the parameter Bimposes an ℓ2constraint
onwand the parameter Rcaptures a low ℓ2-norm assumption on the instances.
In contrast, in Theorem 26.15 the parameter Bimposes an ℓ1constraint on w
(which is stronger than an ℓ2constraint) while the parameter Rcaptures a low
ℓ∞-norm assumption on the instance (which is weaker than a low ℓ2-norm as-
sumption). Therefore, the choice of the constraint should depend on our prior

knowledge of the set of instances and on prior assumptions on good predictors.
26.5 Bibliographic Remarks
The use of Rademacher complexity for bounding the uniform convergence is
due to (Koltchinskii & Panchenko 2000, Bartlett & Mendelson 2001, Bartlett
& Mendelson 2002). For additional reading see, for example, (Bousquet 2002,
Boucheron, Bousquet & Lugosi 2005, Bartlett, Bousquet & Mendelson 2005).

26.5 Bibliographic Remarks 387
Our proof of the concentration lemma is due to Kakade and Tewari lecture
notes. Kakade, Sridharan & Tewari (2008) gave a uniﬁed framework for deriving
bounds on the Rademacher complexity of linear classes with respect to diﬀerent
assumptions on the norms.

27 Covering Numbers
In this chapter we describe another way to measure the complexity of sets, which
is called covering numbers.
27.1 Covering
definition 27.1 (Covering) Let A⊂Rmbe a set of vectors. We say that A
isr-covered by a set A′, with respect to the Euclidean metric, if for all a∈A
there exists a′∈A′with∥a−a′∥≤r. We deﬁne by N(r,A) the cardinality of
the smallest A′thatr-coversA.
Example 27.1 (Subspace) Suppose that A⊂Rm, letc= max a∈A∥a∥, and as-
sume thatAlies in ad-dimensional subspace of Rm. Then,N(r,A)≤(2c√
d/r)d.
To see this, let v1,...,vdbe an orthonormal basis of the subspace. Then, any
a∈Acan be written as a=∑d
i=1αiviwith∥α∥∞≤∥α∥2=∥a∥2≤c. Let
ϵ∈Rand consider the set
A′={d∑
i=1α′
ivi:∀i,α′
i∈{−c,−c+ϵ,−c+ 2ϵ,...,c}}
.
Given a∈As.t.a=∑d
i=1αiviwith∥α∥∞≤c, there exists a′∈A′such that
∥a−a′∥2=∥∑
i(α′
i−αi)vi∥2≤ϵ2∑
i∥vi∥2≤ϵ2d.
Chooseϵ=r/√
d; then∥a−a′∥≤rand therefore A′is anr-cover ofA. Hence,
N(r,A)≤|A′|=(2c
ϵ)d
=(
2c√
d
r)d
.
27.1.1 Properties

d/r)d.
A′={d∑
i=1α′
ivi:∀i,α′
.
i(α′
ϵ)d
=(
2c√
d
r)d
.
lemma 27.2 For anyA⊂Rm, scalarc>0, and vector a0∈Rm, we have
∀r>0, N(r,{ca+a0:a∈A})≤N(cr,A).

Next, we derive a contraction principle.
lemma 27.3 For eachi∈[m], letφi:R→Rbe aρ-Lipschitz function;
namely, for all α,β∈Rwe have|φi(α)−φi(β)|≤ρ|α−β|. For a∈Rmlet
φ(a)denote the vector (φ1(a1),...,φm(am)). Letφ◦A={φ(a) :a∈A}. Then,
N(ρr,φ◦A)≤N(r,A).
Proof DeﬁneB=φ◦A. LetA′be anr-cover ofAand deﬁne B′=φ◦A′.
Then, for all a∈Athere exists a′∈A′with∥a−a′∥≤r. So,
∥φ(a)−φ(a′)∥2=∑
i(φi(ai)−φi(a′
i))2≤ρ2∑
i(ai−a′
i)2≤(ρr)2.
Hence,B′is an (ρr)-cover ofB.
27.2 From Covering to Rademacher Complexity via Chaining
The following lemma bounds the Rademacher complexity of Abased on the
covering numbers N(r,A). This technique is called Chaining and is attributed
to Dudley.
lemma 27.4 Letc= min ¯amaxa∈A∥a−¯a∥. Then, for any integer M > 0,
R(A)≤c2−M
√m+6c
mM∑
k=12−k√
log(N(c2−k,A)).
Proof Let¯abe a minimizer of the objective function given in the deﬁnition
ofc. On the basis of Lemma 26.6, we can analyze the Rademacher complexity
assuming that ¯a=0.

to Dudley.
R(A)≤c2−M
√m+6c
mM∑
k=12−k√
Consider the set B0={0}and note that it is a c-cover ofA. LetB1,...,BM
be sets such that each Bkcorresponds to a minimal ( c2−k)-cover ofA. Let
a∗= argmaxa∈A⟨σ,a⟩(where if there is more than one maximizer, choose one
in an arbitrary way, and if a maximizer does not exist, choose a∗such that
⟨σ,a∗⟩is close enough to the supremum). Note that a∗is a function of σ. For
eachk, letb(k)be the nearest neighbor of a∗inBk(hence b(k)is also a function
ofσ). Using the triangle inequality,

∥b(k)−b(k−1)∥≤∥b(k)−a∗∥+∥a∗−b(k−1)∥≤c(2−k+ 2−(k−1)) = 3c2−k.
For eachkdeﬁne the set
ˆBk={(a−a′) :a∈Bk,a′∈Bk−1,∥a−a′∥≤3c2−k}.

390 Covering Numbers
We can now write
R(A) =1
mE⟨σ,a∗⟩
=1
mE[
⟨σ,a∗−b(M)⟩+M∑
k=1⟨σ,b(k)−b(k−1)⟩]
≤1
mE[
∥σ∥∥a∗−b(M)∥]
+M∑
k=11
mE[
sup
a∈ˆBk⟨σ,a⟩]
.
Since∥σ∥=√mand∥a∗−b(M)∥ ≤c2−M, the ﬁrst summand is at most
c√m2−M. Additionally, by Massart lemma,
1
mEsup
a∈ˆBk⟨σ,a⟩≤3c2−k√
2 log(N(c2−k,A)2)
m= 6c2−k√
log(N(c2−k,A))
m.
R(A)≤c2−M
√m+6c
mM∑
k=12−k√
As a corollary we obtain the following:
lemma 27.5 Assume that there are α,β > 0such that for any k≥1we have
√
log(N(c2−k,A))≤α+βk.
Then,
R(A)≤6c
m(α+ 2β).
Proof The bound follows from Lemma 27.4 by taking M→∞ and noting that∑∞
k=12−k= 1 and∑∞
k=1k2−k= 2.
Example 27.2 Consider a set Awhich lies in a ddimensional subspace of Rm
and such that c= max a∈A∥a∥. We have shown that N(r,A)≤(
2c√
d
r)d
. There-
fore, for any k,
√
log(N(c2−k,A))≤√
dlog(
2k+1√
d)
≤√
dlog(2√
d) +√
kd
≤√
dlog(2√
d) +√
dk.
Hence Lemma 27.5 yields
R(A)≤6c
m(√
dlog(2√
d) + 2√
d)
=O(
c√
dlog(d)
m)
.

27.3 Bibliographic Remarks
The chaining technique is due to Dudley (1987). For an extensive study of cover-
ing numbers as well as other complexity measures that can be used to bound the
rate of uniform convergence we refer the reader to (Anthony & Bartlet 1999).

28 Proof of the Fundamental Theorem
of Learning Theory
In this chapter we prove Theorem 6.8 from Chapter 6. We remind the reader
the conditions of the theorem, which will hold throughout this chapter: His a
hypothesis class of functions from a domain Xto{0,1}, the loss function is the
0−1 loss, and VCdim( H) =d<∞.
We shall prove the upper bound for both the realizable and agnostic cases
and shall prove the lower bound for the agnostic case. The lower bound for the
realizable case is left as an exercise.
28.1 The Upper Bound for the Agnostic Case
For the upper bound we need to prove that there exists Csuch thatHis agnostic
PAC learnable with sample complexity
mH(ϵ,δ)≤Cd+ ln(1/δ)
ϵ2.
We will prove the slightly looser bound:
mH(ϵ,δ)≤Cdlog(d/ϵ) + ln(1/δ)
ϵ2. (28.1)
The tighter bound in the theorem statement requires a more involved proof, in
which a more careful analysis of the Rademacher complexity using a technique
called “chaining” should be used. This is beyond the scope of this book.

ϵ2.
ϵ2. (28.1)
To prove Equation (28.1), it suﬃces to show that applying the ERM with a
sample size
m≥432d
ϵ2·log(64d
ϵ2)
+8
ϵ2·(8dlog(e/d) + 2 log(4/δ))
yields anϵ,δ-learner forH. We prove this result on the basis of Theorem 26.5.
Let (x1,y1),..., (xm,ym) be a classiﬁcation training set. Recall that the Sauer-
Shelah lemma tells us that if VCdim( H) =dthen
|{(h(x1),...,h (xm)) :h∈H}| ≤(em
d)d
.
DenoteA={(1[h(x1)̸=y1],...,1[h(xm)̸=ym]) :h∈H} . This clearly implies that
|A| ≤(em
d)d
.

m≥432d
ϵ2)
+8
d)d
.
|A| ≤(em
d)d
.

Combining this with Lemma 26.8 we obtain the following bound on the Rademacher
complexity:
R(A)≤√
2dlog(em/d )
m.
Using Theorem 26.5 we obtain that with probability of at least 1 −δ, for every
h∈H we have that
LD(h)−LS(h)≤√
8dlog(em/d )
m+√
2 log(2/δ)
m.
Repeating the previous argument for minus the zero-one loss and applying the
union bound we obtain that with probability of at least 1 −δ, for everyh∈H
it holds that
|LD(h)−LS(h)|≤√
m+√
2 log(4/δ)
m
≤2√
8dlog(em/d ) + 2 log(4/δ)
m.
To ensure that this is smaller than ϵwe need
m≥4
ϵ2·(8dlog(m) + 8dlog(e/d) + 2 log(4/δ)).
Using Lemma A.2, a suﬃcient condition for the inequality to hold is that
m≥432d
ϵ2)
+8
ϵ2·(8dlog(e/d) + 2 log(4/δ)).
28.2 The Lower Bound for the Agnostic Case
Here, we prove that there exists Csuch thatHis agnostic PAC learnable with
sample complexity
mH(ϵ,δ)≥Cd+ ln(1/δ)
ϵ2.
We will prove the lower bound in two parts. First, we will show that m(ϵ,δ)≥

m+√
2 log(4/δ)
m
≤2√
m.
m≥4
m≥432d
ϵ2)
+8
ϵ2.
0.5 log(1/(4δ))/ϵ2, and second we will show that for every δ≤1/8 we have that
m(ϵ,δ)≥8d/ϵ2. These two bounds will conclude the proof.
28.2.1 Showing That m(ϵ,δ)≥0.5 log(1/(4δ))/ϵ2
We ﬁrst show that for any ϵ<1/√
2 and anyδ∈(0,1), we have that m(ϵ,δ)≥
0.5 log(1/(4δ))/ϵ2. To do so, we show that for m≤0.5 log(1/(4δ))/ϵ2,His not
Choose one example that is shattered by H. That is, let cbe an example such

394 Proof of the Fundamental Theorem of Learning Theory
that there are h+,h−∈H for whichh+(c) = 1 and h−(c) =−1. Deﬁne two
distributions,D+andD−, such that for b∈{± 1}we have
Db({(x,y)}) ={1+ybϵ
2ifx=c
That is, all the distribution mass is concentrated on two examples ( c,1) and
(c,−1), where the probability of ( c,b) is1+bϵ
2and the probability of ( c,−b) is
1−bϵ
2.
LetAbe an arbitrary algorithm. Any training set sampled from Dbhas the
formS= (c,y1),..., (c,ym). Therefore, it is fully characterized by the vector
y= (y1,...,ym)∈{± 1}m. Upon receiving a training set S, the algorithm A
returns a hypothesis h:X→{± 1}. Since the error of Aw.r.t.Dbonly depends
onh(c), we can think of Aas a mapping from {±1}minto{±1}. Therefore,
we denote by A(y) the value in{±1}corresponding to the prediction of h(c),
wherehis the hypothesis that Aoutputs upon receiving the training set S=
(c,y1),..., (c,ym).
Note that for any hypothesis hwe have
LDb(h) =1−h(c)bϵ
2.

2.
In particular, the Bayes optimal hypothesis is hband
LDb(A(y))−LDb(hb) =1−A(y)bϵ
2−1−ϵ
2={
ϵifA(y)̸=b
FixA. Forb∈{± 1}, letYb={y∈{0,1}m:A(y)̸=b}. The distribution Db
induces a probability Pbover{±1}m. Hence,
P[LDb(A(y))−LDb(hb) =ϵ] =Db(Yb) =∑
yPb[y]1[A(y)̸=b].
DenoteN+={y:|{i:yi= 1}|≥m/2}andN−={±1}m\N+. Note that for
anyy∈N+we haveP+[y]≥P−[y] and for any y∈N−we haveP−[y]≥P+[y].

28.2 The Lower Bound for the Agnostic Case 395
max
b∈{±1}P[LDb(A(y))−LDb(hb) =ϵ]
= max
b∈{±1}∑
yPb[y]1[A(y)̸=b]
≥1
2∑
yP+[y]1[A(y)̸=+]+1
2∑
yP−[y]1[A(y)̸=−]
=1
2∑
y∈N+(P+[y]1[A(y)̸=+]+P−[y]1[A(y)̸=−]) +1
2∑
y∈N−(P+[y]1[A(y)̸=+]+P−[y]1[A(y)̸=−])
≥1
2∑
y∈N+(P−[y]1[A(y)̸=+]+P−[y]1[A(y)̸=−]) +1
2∑
y∈N−(P+[y]1[A(y)̸=+]+P+[y]1[A(y)̸=−])
=1
2∑
y∈N+P−[y] +1
2∑
y∈N−P+[y].
Next note that∑
y∈N+P−[y] =∑
y∈N−P+[y], and both values are the prob-
ability that a Binomial ( m,(1−ϵ)/2) random variable will have value greater
thanm/2. Using Lemma B.11, this probability is lower bounded by
1
2(
1−√
1−exp(−mϵ2/(1−ϵ2)))
≥1
2(
1−√
1−exp(−2mϵ2))
,
where we used the assumption that ϵ2≤1/2. It follows that if m≤0.5 log(1/(4δ))/ϵ2
then there exists bsuch that
P[LDb(A(y))−LDb(hb) =ϵ]
≥1
2(
1−√
1−√
4δ)
≥δ,
where the last inequality follows by standard algebraic manipulations. This con-
28.2.2 Showing That m(ϵ,1/8)≥8d/ϵ2
We shall now prove that for every ϵ<1/(8√
2) we have that m(ϵ,δ)≥8d
ϵ2.

1
2(
1−√
≥1
2(
1−√
,
≥1
2(
1−√
1−√
4δ)
≥δ,
ϵ2.
Letρ= 8ϵand note that ρ∈(0,1/√
2). We will construct a family of distri-
butions as follows. First, let C={c1,...,cd}be a set of dinstances which are
shattered byH. Second, for each vector ( b1,...,bd)∈{± 1}d, deﬁne a distribu-
tionDbsuch that
Db({(x,y)}) ={
1
d·1+ybiρ
2if∃i:x=ci
That is, to sample an example according to Db, we ﬁrst sample an element ci∈C
uniformly at random, and then set the label to be biwith probability (1 + ρ)/2
or−biwith probability (1 −ρ)/2.

ϵ2.
1
d·1+ybiρ
It is easy to verify that the Bayes optimal predictor for Dbis the hypothesis

396 Proof of the Fundamental Theorem of Learning Theory
h∈H such thath(ci) =bifor alli∈[d], and its error is1−ρ
2. In addition, for
any other function f:X→{± 1}, it is easy to verify that
LDb(f) =1 +ρ
2·|{i∈[d] :f(ci)̸=bi}|
d+1−ρ
2·|{i∈[d] :f(ci) =bi}|
d.
LDb(f)−min
h∈HLDb(h) =ρ·|{i∈[d] :f(ci)̸=bi}|
d. (28.2)
Next, ﬁx some learning algorithm A. As in the proof of the No-Free-Lunch
theorem, we have that
max
Db:b∈{±1}dE
S∼Dm
b[
LDb(A(S))−min
h∈HLDb(h)]
(28.3)
≥ E
Db:b∼U({±1}d)E
S∼Dm
b[
(28.4)
= E
S∼Dm
b[
ρ·|{i∈[d] :A(S)(ci)̸=bi|
d]
(28.5)
=ρ
dd∑
i=1E
S∼Dm
b1[A(S)(ci)̸=bi], (28.6)
where the ﬁrst equality follows from Equation (28.2). In addition, using the
deﬁnition ofDb, to sample S∼Dbwe can ﬁrst sample ( j1,...,jm)∼U([d])m, set
xr=cji, and ﬁnally sample yrsuch that P[yr=bji] = (1 +ρ)/2. Let us simplify
the notation and use y∼bto denote sampling according to P[y=b] = (1+ρ)/2.
Therefore, the right-hand side of Equation (28.6) equals
ρ
dd∑
i=1E

(28.4)
= E
S∼Dm
b[
d]
(28.5)
=ρ
dd∑
i=1E
S∼Dm
ρ
dd∑
i=1E
j∼U([d])mE
b∼U({±1}d)E
∀r,yr∼bjr1[A(S)(ci)̸=bi]. (28.7)
We now proceed in two steps. First, we show that among all learning algorithms,
A, the one which minimizes Equation (28.7) (and hence also Equation (28.4))
is the Maximum-Likelihood learning rule, denoted AML. Formally, for each i,
AML(S)(ci) is the majority vote among the set {yr:r∈[m],xr=ci}. Second,
we lower bound Equation (28.7) for AML.
lemma 28.1 Among all algorithms, Equation (28.4) is minimized for Abeing

ρ
dd∑
i=1E
the Maximum-Likelihood algorithm, AML, deﬁned as
∀i, AML(S)(ci) =sign(∑
r:xr=ciyr)
.
Proof Fix somej∈[d]m. Note that given jandy∈{± 1}m, the training set
Sis fully determined. Therefore, we can write A(j,y) instead of A(S). Let us
also ﬁxi∈[d]. Denoteb¬ithe sequence ( b1,...,bi−1,bi+1,...,bm). Also, for any

28.2 The Lower Bound for the Agnostic Case 397
y∈{± 1}m, letyIdenote the elements of ycorresponding to indices for which
jr=iand lety¬Ibe the rest of the elements of y. We have
E
∀r,yr∼bjr1[A(S)(ci)̸=bi]
=1
2∑
bi∈{±1}E
b¬i∼U({±1}d−1)∑
yP[y|b¬i,bi]1[A(j,y)(ci)̸=bi]
= E
y¬IP[y¬I|b¬i]1
2∑
yI
∑
bi∈{±1}P[yI|bi]1[A(j,y)(ci)̸=bi]
.
The sum within the parentheses is minimized when A(j,y)(ci) is the maximizer
ofP[yI|bi] overbi∈{± 1}, which is exactly the Maximum-Likelihood rule. Re-
peating the same argument for all iwe conclude our proof.
Fixi. For everyj, letni(j) ={|t:jt=i|}be the number of instances in which
the instance is ci. For the Maximum-Likelihood rule, we have that the quantity
E
∀r,yr∼bjr1[AML(S)(ci)̸=bi]
is exactly the probability that a binomial ( ni(j),(1−ρ)/2) random variable will
be larger than ni(j)/2. Using Lemma B.11, and the assumption ρ2≤1/2, we
have that
P[B≥ni(j)/2]≥1
2(
1−√
1−e−2ni(j)ρ2)
.
We have thus shown that
ρ
dd∑
i=1E

E
have that
2(
1−√
.
ρ
dd∑
i=1E
≥ρ
2dd∑
i=1E
j∼U([d])m(
1−√
1−e−2ρ2ni(j))
≥ρ
2dd∑
i=1E
1−√
2ρ2ni(j))
,
where in the last inequality we used the inequality 1 −e−a≤a.
Since the square root function is concave, we can apply Jensen’s inequality to
obtain that the above is lower bounded by
≥ρ
2dd∑
i=1(
1−√
2ρ2E
j∼U([d])mni(j))
=ρ
2dd∑
i=1(
1−√
2ρ2m/d)
=ρ
2(
1−√
2ρ2m/d)
.

398 Proof of the Fundamental Theorem of Learning Theory
As long asm<d
8ρ2, this term would be larger than ρ/4.
In summary, we have shown that if m <d
8ρ2then for any algorithm there
exists a distribution such that
E
S∼Dm[
h∈HLD(h)]
≥ρ/4.
Finally, Let ∆ =1
ρ(LD(A(S))−minh∈HLD(h)) and note that ∆ ∈[0,1] (see
Equation (28.5)). Therefore, using Lemma B.1, we get that
P[LD(A(S))−min
h∈HLD(h)>ϵ] =P[
∆>ϵ
ρ]
≥E[∆]−ϵ
ρ
≥1
4−ϵ
ρ.
Choosingρ= 8ϵwe conclude that if m<d
512ϵ2, then with probability of at least
1/8 we will have LD(A(S))−minh∈HLD(h)≥ϵ.
28.3 The Upper Bound for the Realizable Case
Here we prove that there exists Csuch thatHis PAC learnable with sample
mH(ϵ,δ)≤Cdln(1/ϵ) + ln(1/δ)
ϵ.
We do so by showing that for m≥Cdln(1/ϵ)+ln(1/δ)
ϵ,His learnable using the
ERM rule. We prove this claim based on the notion of ϵ-nets.
definition 28.2 (ϵ-net) LetXbe a domain. S⊂X is anϵ-net forH⊂ 2X
with respect to a distribution DoverXif
∀h∈H:D(h)≥ϵ⇒h∩S̸=∅.

ϵ.
theorem 28.3 LetH⊂ 2Xwith VCdim(H) =d. Fixϵ∈(0,1),δ∈(0,1/4)
and let
m≥8
ϵ(
2dlog(16e
ϵ)
+ log(2
δ))
.
Then, with probability of at least 1−δover a choice of S∼Dmwe have that S
is anϵ-net forH.
Proof Let
B={S⊂X :|S|=m,∃h∈H,D(h)≥ϵ,h∩S=∅}
be the set of sets which are not ϵ-nets. We need to bound P[S∈B]. Deﬁne
B′={(S,T)⊂X :|S|=|T|=m,∃h∈H,D(h)≥ϵ,h∩S=∅,|T∩h|>ϵm
2}.

28.3 The Upper Bound for the Realizable Case 399
Claim 1
P[S∈B]≤2P[(S,T)∈B′].
Proof of Claim 1 : SinceSandTare chosen independently we can write
P[(S,T)∈B′] = E
(S,T)∼D2m[
1[(S,T)∈B′]]
=E
S∼Dm[
E
T∼Dm[
1[(S,T)∈B′]]]
.
Note that (S,T)∈B′impliesS∈Band therefore 1[(S,T)∈B′]=1[(S,T)∈B′]1[S∈B],
P[(S,T)∈B′] =E
S∼DmE
T∼Dm1[(S,T)∈B′]1[S∈B]
=E
S∼Dm1[S∈B]E
T∼Dm1[(S,T)∈B′].
Fix someS. Then, either 1[S∈B]= 0 orS∈Band then∃hSsuch thatD(hS)≥ϵ
and|hS∩S|= 0. It follows that a suﬃcient condition for ( S,T)∈B′is that
|T∩hS|>ϵm
2. Therefore, whenever S∈Bwe have
E
T∼Dm1[(S,T)∈B′]≥P
T∼Dm[|T∩hS|>ϵm
2].
But, since we now assume S∈Bwe know thatD(hS) =ρ≥ϵ. Therefore,
|T∩hS|is a binomial random variable with parameters ρ(probability of success
for a single try) and m(number of tries). Chernoﬀ’s inequality implies
P[|T∩hS|≤ρm
2]≤e−2
mρ(mρ−mρ/2)2
=e−mρ/2≤e−mϵ/2≤e−dlog(1/δ)/2=δd/2≤1/2.
Thus,
P[|T∩hS|>ϵm
2] = 1−P[|T∩hS|≤ϵm
2]≥1−P[|T∩hS|≤ρm
2]≥1/2.
Combining all the preceding we conclude the proof of Claim 1 .

|T∩hS|>ϵm
E
2].
2]≤e−2
Thus,
2]≥1/2.
Claim 2 (Symmetrization):
P[(S,T)∈B′]≤e−ϵm/4τH(2m).
Proof of Claim 2 : To simplify notation, let α=mϵ/2 and for a sequence A=
(x1,...,x 2m) letA0= (x1,...,xm). Using the deﬁnition of B′we get that
P[A∈B′] = E
A∼D2mmax
h∈H1[D(h)≥ϵ]1[|h∩A0|=0]1[|h∩A|≥α]
≤E
A∼D2mmax
h∈H1[|h∩A0|=0]1[|h∩A|≥α].
Now, let us deﬁne by HAthe eﬀective number of diﬀerent hypotheses on A,
namely,HA={h∩A:h∈H} . It follows that
P[A∈B′]≤E
A∼D2mmax
h∈HA1[|h∩A0|=0]1[|h∩A|≥α]
≤E
A∼D2m∑
h∈HA1[|h∩A0|=0]1[|h∩A|≥α].

P[A∈B′] = E
A∼D2mmax
≤E
A∼D2mmax
P[A∈B′]≤E
A∼D2mmax
≤E
A∼D2m∑
LetJ={j⊂[2m] :|j|=m}. For any j∈JandA= (x1,...,x 2m) deﬁne
Aj= (xj1,...,xjm). Since the elements of Aare chosen i.i.d., we have that
for any j∈Jand any function f(A,A 0) it holds that EA∼D2m[f(A,A 0)] =

400 Proof of the Fundamental Theorem of Learning Theory
EA∼D2m[f(A,A j)]. Since this holds for any jit also holds for the expectation of
jchosen at random from J. In particular, it holds for the function f(A,A 0) =∑
h∈HA1[|h∩A0|=0]1[|h∩A|≥α]. We therefore obtain that
P[A∈B′]≤E
A∼D2mE
j∼J∑
h∈HA1[|h∩Aj|=0]1[|h∩A|≥α]
=E
A∼D2m∑
h∈HA1[|h∩A|≥α]E
j∼J1[|h∩Aj|=0].
Now, ﬁx some As.t.|h∩A|≥α. Then, Ej1[|h∩Aj|=0]is the probability that
when choosing mballs from a bag with at least αred balls, we will never choose
a red ball. This probability is at most
(1−α/(2m))m= (1−ϵ/4)m≤e−ϵm/4.
We therefore get that
P[A∈B′]≤E
A∼D2m∑
h∈HAe−ϵm/4≤e−ϵm/4E
A∼D2m|HA|.
Using the deﬁnition of the growth function we conclude the proof of Claim 2.
Completing the Proof: By Sauer’s lemma we know that τH(2m)≤(2em/d )d.
Combining this with the two claims we obtain that
P[S∈B]≤2(2em/d )de−ϵm/4.
We would like the right-hand side of the inequality to be at most δ; that is,
2(2em/d )de−ϵm/4≤δ.
Rearranging, we obtain the requirement
m≥4

P[A∈B′]≤E
A∼D2m∑
m≥4
ϵ(dlog(2em/d ) + log(2/δ)) =4d
ϵlog(m) +4
ϵ(dlog(2e/d) + log(2/δ).
Using Lemma A.2, a suﬃcient condition for the preceding to hold is that
m≥16d
ϵlog(8d
ϵ)
+8
A suﬃcient condition for this is that
m≥16d
ϵlog(8d
ϵ)
+16
ϵ(dlog(2e/d) +1
2log(2/δ)
=16d
ϵ(
log(8d2e
dϵ))
+8
ϵlog(2/δ)
=8
ϵ(
2dlog(16e
ϵ)
+ log(2
δ))
.
and this concludes our proof.

28.3 The Upper Bound for the Realizable Case 401
28.3.1 From ϵ-Nets to PAC Learnability
theorem 28.4 LetHbe a hypothesis class over Xwith VCdim(H) =d. Let
Dbe a distribution over Xand letc∈H be a target hypothesis. Fix ϵ,δ∈(0,1)
and letmbe as deﬁned in Theorem 28.3. Then, with probability of at least 1−δ
over a choice of mi.i.d. instances from Xwith labels according to cwe have that
any ERM hypothesis has a true error of at most ϵ.
Proof Deﬁne the classHc={cah:h∈H} , wherecah= (h\c)∪(c\h). It is
easy to verify that if some A⊂X is shattered byHthen it is also shattered by Hc
and vice versa. Hence, VCdim( H) = VCdim(Hc). Therefore, using Theorem 28.3
we know that with probability of at least 1 −δ, the sample Sis anϵ-net forHc.
Note thatLD(h) =D(hac). Therefore, for any h∈H withLD(h)≥ϵwe have
that|(hac)∩S|>0, which implies that hcannot be an ERM hypothesis, which

29 Multiclass Learnability
In Chapter 17 we have introduced the problem of multiclass categorization, in
which the goal is to learn a predictor h:X→ [k]. In this chapter we address PAC
learnability of multiclass predictors with respect to the 0-1 loss. As in Chapter 6,
the main goal of this chapter is to:
•Characterize which classes of multiclass hypotheses are learnable in the (mul-
ticlass) PAC model.
•Quantify the sample complexity of such hypothesis classes.
In view of the fundamental theorem of learning theory (Theorem 6.8), it is natu-
ral to seek a generalization of the VC dimension to multiclass hypothesis classes.
In Section 29.1 we show such a generalization, called the Natarajan dimension ,
and state a generalization of the fundamental theorem based on the Natarajan
dimension. Then, we demonstrate how to calculate the Natarajan dimension of
several important hypothesis classes.
Recall that the main message of the fundamental theorem of learning theory

is that a hypothesis class of binary classiﬁers is learnable (with respect to the
0-1 loss) if and only if it has the uniform convergence property, and then it
is learnable by any ERM learner. In Chapter 13, Exercise 2, we have shown
that this equivalence breaks down for a certain convex learning problem. The
last section of this chapter is devoted to showing that the equivalence between
learnability and uniform convergence breaks down even in multiclass problems

with the 0-1 loss, which are very similar to binary classiﬁcation. Indeed, we
construct a hypothesis class which is learnable by a speciﬁc ERM learner, but
for which other ERM learners might fail and the uniform convergence property
does not hold.
29.1 The Natarajan Dimension
In this section we deﬁne the Natarajan dimension, which is a generalization of
the VC dimension to classes of multiclass predictors. Throughout this section,
letHbe a hypothesis class of multiclass predictors; namely, each h∈H is a
function fromXto [k].

To deﬁne the Natarajan dimension, we ﬁrst generalize the deﬁnition of shat-
tering.
definition 29.1 (Shattering (Multiclass Version)) We say that a set C⊂X
is shattered byHif there exist two functions f0,f1:C→[k] such that
•For everyx∈C,f0(x)̸=f1(x).
•For everyB⊂C, there exists a function h∈H such that
∀x∈B,h(x) =f0(x) and∀x∈C\B,h(x) =f1(x).
definition 29.2 (Natarajan Dimension) The Natarajan dimension of H, de-
noted Ndim(H), is the maximal size of a shattered set C⊂X.
It is not hard to see that in the case that there are exactly two classes,
Ndim(H) = VCdim(H). Therefore, the Natarajan dimension generalizes the VC
dimension. We next show that the Natarajan dimension allows us to general-
ize the fundamental theorem of statistical learning from binary classiﬁcation to
29.2 The Multiclass Fundamental Theorem
theorem 29.3 (The Multiclass Fundamental Theorem) There exist absolute

constantsC1,C2>0such that the following holds. For every hypothesis class H
of functions from Xto[k], such that the Natarajan dimension of Hisd, we have
ϵ2≤mUC
H(ϵ,δ)≤C2dlog (k) + log(1/δ)
ϵ2.
ϵ2≤mH(ϵ,δ)≤C2dlog (k) + log(1/δ)
ϵ2.
3.His PAC learnable (assuming realizability) with sample complexity
ϵ≤mH(ϵ,δ)≤C2dlog(kd
ϵ)
+ log(1/δ)
ϵ.

ϵ2≤mUC
ϵ2.
ϵ2.
ϵ)
+ log(1/δ)
ϵ.
29.2.1 On the Proof of Theorem 29.3
The lower bounds in Theorem 29.3 can be deduced by a reduction from the
binary fundamental theorem (see Exercise 5).
The upper bounds in Theorem 29.3 can be proved along the same lines of the
proof of the fundamental theorem for binary classiﬁcation, given in Chapter 28
(see Exercise 4). The sole ingredient of that proof that should be modiﬁed in a
nonstraightforward manner is Sauer’s lemma. It applies only to binary classes

ϵ)
+ log(1/δ)
ϵ.
and therefore must be replaced. An appropriate substitute is Natarajan’s lemma:

404 Multiclass Learnability
lemma 29.4 (Natarajan) |H|≤|X|Ndim(H)·k2Ndim(H).
The proof of Natarajan’s lemma shares the same spirit of the proof of Sauer’s
lemma and is left as an exercise (see Exercise 3).
29.3 Calculating the Natarajan Dimension
In this section we show how to calculate (or estimate) the Natarajan dimen-
sion of several popular classes, some of which were studied in Chapter 17. As
these calculations indicate, the Natarajan dimension is often proportional to the
number of parameters required to deﬁne a hypothesis.
29.3.1 One-versus-All Based Classes
In Chapter 17 we have seen two reductions of multiclass categorization to bi-
nary classiﬁcation: One-versus-All and All-Pairs. In this section we calculate the
Natarajan dimension of the One-versus-All method.
Recall that in One-versus-All we train, for each label, a binary classiﬁer that
distinguishes between that label and the rest of the labels. This naturally sug-
gests considering multiclass hypothesis classes of the following form. Let Hbin⊂

{0,1}Xbe a binary hypothesis class. For every ¯h= (h1,...,hk)∈(Hbin)kdeﬁne
T(¯h) :X→ [k] by
T(¯h)(x) = argmax
i∈[k]hi(x).
If there are two labels that maximize hi(x), we choose the smaller one. Also, let
HOvA,k
bin={T(¯h) :¯h∈(Hbin)k}.
What “should” be the Natarajan dimension of HOvA,k
bin? Intuitively, to specify a
hypothesis inHbinwe needd= VCdim(Hbin) parameters. To specify a hypothe-
sis inHOvA,k
bin, we need to specify khypotheses inHbin. Therefore, kdparameters
should suﬃce. The following lemma establishes this intuition.

HOvA,k
lemma 29.5 Ifd= VCdim(Hbin)then
Ndim(HOvA,k
bin)≤3kdlog (kd).
Proof LetC⊂X be a shattered set. By the deﬁnition of shattering (for mul-
ticlass hypotheses)⏐⏐⏐(
HOvA,k
bin)
C⏐⏐⏐≥2|C|.
On the other hand, each hypothesis in HOvA,k
binis determined by using khypothe-
ses fromHbin. Therefore,
⏐⏐⏐(
HOvA,k
bin)
C⏐⏐⏐≤ |(Hbin)C|k.

29.3 Calculating the Natarajan Dimension 405
By Sauer’s lemma, |(Hbin)C|≤|C|d. We conclude that
2|C|≤⏐⏐⏐(
HOvA,k
bin)
C⏐⏐⏐≤ |C|dk.
The proof follows by taking the logarithm and applying Lemma A.1.
How tight is Lemma 29.5? It is not hard to see that for some classes, Ndim( HOvA,k
bin)
can be much smaller than dk(see Exercise 1). However there are several natural
binary classes,Hbin(e.g., halfspaces), for which Ndim( HOvA,k
bin) = Ω(dk) (see
Exercise 6).
29.3.2 General Multiclass-to-Binary Reductions
The same reasoning used to establish Lemma 29.5 can be used to upper bound
the Natarajan dimension of more general multiclass-to-binary reductions. These
reductions train several binary classiﬁers on the data. Then, given a new in-
stance, they predict its label by using some rule that takes into account the
labels predicted by the binary classiﬁers. These reductions include One-versus-
All and All-Pairs.
Suppose that such a method trains lbinary classiﬁers from a binary class Hbin,

andr:{0,1}l→[k] is the rule that determines the (multiclass) label according
to the predictions of the binary classiﬁers. The hypothesis class corresponding
to this method can be deﬁned as follows. For every ¯h= (h1,...,hl)∈(Hbin)l
deﬁneR(¯h) :X→ [k] by
R(¯h)(x) =r(h1(x),...,hl(x)).
Finally, let
Hr
bin={R(¯h) :¯h∈(Hbin)l}.
Similarly to Lemma 29.5 it can be proven that:
lemma 29.6 Ifd= VCdim(Hbin)then
Ndim(Hr
bin)≤3ldlog (ld).
The proof is left as Exercise 2.
29.3.3 Linear Multiclass Predictors

Hr
Ndim(Hr
Next, we consider the class of linear multiclass predictors (see Section 17.2). Let
Ψ :X×[k]→Rdbe some class-sensitive feature mapping and let
HΨ={
x↦→argmax
i∈[k]⟨w,Ψ(x,i)⟩:w∈Rd}
. (29.1)
Each hypothesis in HΨis determined by dparameters, namely, a vector w∈
Rd. Therefore, we would expect that the Natarajan dimension would be upper
bounded by d. Indeed:

406 Multiclass Learnability
theorem 29.7 Ndim(HΨ)≤d.
Proof LetC⊂ X be a shattered set, and let f0,f1:C→[k] be the two
functions that witness the shattering. We need to show that |C|≤d. For every
x∈Cletρ(x) = Ψ(x,f0(x))−Ψ(x,f1(x)). We claim that the set ρ(C)def=
{ρ(x) :x∈C}consists of|C|elements (i.e., ρis one to one) and is shattered
by the binary hypothesis class of homogeneous linear separators on Rd,
H={x↦→sign(⟨w,x⟩) :w∈Rd}.
Since VCdim(H) =d, it will follow that |C|=|ρ(C)|≤d, as required.
To establish our claim it is enough to show that |Hρ(C)|= 2|C|. Indeed, given
a subsetB⊂C, by the deﬁnition of shattering, there exists hB∈H Ψfor which
∀x∈B,hB(x) =f0(x) and∀x∈C\B,hB(x) =f1(x).
LetwB∈Rdbe a vector that deﬁnes hB. We have that, for every x∈B,
⟨w,Ψ(x,f0(x))⟩>⟨w,Ψ(x,f1(x))⟩ ⇒ ⟨ w,ρ(x)⟩>0.
Similarly, for every x∈C\B,
⟨w,ρ(x)⟩<0.
It follows that the hypothesis gB∈H deﬁned by the same w∈Rdlabel the
points inρ(B) by 1 and the points in ρ(C\B) by 0. Since this holds for every

B⊆Cwe obtain that|C|=|ρ(C)|and|Hρ(C)|= 2|C|, which concludes our
proof.
The theorem is tight in the sense that there are mappings Ψ for which Ndim( HΨ) =
Ω(d). For example, this is true for the multivector construction (see Section 17.2
and the Bibliographic Remarks at the end of this chapter). We therefore con-
clude:
corollary 29.8 LetX=Rnand let Ψ :X×[k]→Rnkbe the class sensitive
feature mapping for the multi-vector construction:
∈R(k−y)n].

proof.
clude:
LetHΨbe as deﬁned in Equation (29.1). Then, the Natarajan dimension of HΨ
satisﬁes
(k−1)(n−1)≤Ndim(HΨ)≤kn.
29.4 On Good and Bad ERMs
In this section we present an example of a hypothesis class with the property
that not all ERMs for the class are equally successful. Furthermore, if we allow
an inﬁnite number of labels, we will also obtain an example of a class that is

29.4 On Good and Bad ERMs 407
learnable by some ERM, but other ERMs will fail to learn it. Clearly, this also
implies that the class is learnable but it does not have the uniform convergence
property. For simplicity, we consider only the realizable case.
The class we consider is deﬁned as follows. The instance space Xwill be any
ﬁnite or countable set. Let Pf(X) be the collection of all ﬁnite and coﬁnite
subsets ofX(that is, for each A∈Pf(X), eitherAorX\Amust be ﬁnite).
Instead of [ k], the label set is Y=Pf(X)∪{∗} , where∗is some special label.
For everyA∈Pf(X) deﬁnehA:X→Y by
hA(x) ={
A x∈A
∗x /∈A
Finally, the hypothesis class we take is
H={hA:A∈Pf(X)}.
LetAbe some ERM algorithm for H. Assume thatAoperates on a sample
labeled by hA∈H. SincehAis the only hypothesis inHthat might return
the labelA, ifAobserves the label A, it “knows” that the learned hypothesis
ishA, and, as an ERM, must return it (note that in this case the error of the

hA(x) ={
A x∈A
∗x /∈A
returned hypothesis is 0). Therefore, to specify an ERM, we should only specify
the hypothesis it returns upon receiving a sample of the form
S={(x1,∗),..., (xm,∗)}.
We consider two ERMs: The ﬁrst, Agood, is deﬁned by
Agood(S) =h∅;
that is, it outputs the hypothesis which predicts ‘*’ for every x∈X. The second
ERM,Abad, is deﬁned by
Abad(S) =h{x1,...xm}c.
The following claim shows that the sample complexity of Abadis about|X|-times
larger than the sample complexity of Agood. This establishes a gap between

diﬀerent ERMs. If Xis inﬁnite, we even obtain a learnable class that is not
learnable by every ERM.
claim 29.9
1. Letϵ,δ > 0,Da distribution over XandhA∈H. LetSbe an i.i.d. sample
consisting of m≥1
ϵlog(1
δ)
examples, sampled according to Dand labeled by
hA. Then, with probability of at least 1−δ, the hypothesis returned by Agood
will have an error of at most ϵ.
2. There exists a constant a > 0such that for every 0< ϵ < a there exists a
distributionDoverXandhA∈H such that the following holds. The hypoth-

claim 29.9
ϵlog(1
δ)
esis returned byAbadupon receiving a sample of size m≤|X|−1
6ϵ, sampled
according toDand labeled by hA, will have error≥ϵwith probability≥e−1
6.

408 Multiclass Learnability
Proof LetDbe a distribution over Xand suppose that the correct labeling
ishA. For any sample, Agoodreturns either h∅orhA. If it returns hAthen its
true error is zero. Thus, it returns a hypothesis with error ≥ϵonly if all the m
examples in the sample are from X\Awhile the error of h∅,LD(h∅) =PD[A],
is≥ϵ. Assumem≥1
ϵlog(1
δ); then the probability of the latter event is no more
than (1−ϵ)m≤e−ϵm≤δ. This establishes item 1.
Next we prove item 2. We restrict the proof to the case that |X|=d <∞.
The proof for inﬁnite Xis similar. Suppose that X={x0,...,xd−1}.
Leta >0 be small enough such that 1 −2ϵ≥e−4ϵfor everyϵ < a and ﬁx
someϵ < a . Deﬁne a distribution on Xby setting P[x0] = 1−2ϵand for all
1≤i≤d−1,P[xi] =2ϵ
d−1. Suppose that the correct hypothesis is h∅and let the
sample size be m. Clearly, the hypothesis returned by Abadwill err on all the
examples fromXwhich are not in the sample. By Chernoﬀ’s bound, if m≤d−1
6ϵ,
then with probability ≥e−1

6ϵ,
6, the sample will include no more thand−1
2examples
fromX. Thus the returned hypothesis will have error ≥ϵ.
The conclusion of the example presented is that in multiclass classiﬁcation,
the sample complexity of diﬀerent ERMs may diﬀer. Are there “good” ERMs
forevery hypothesis class? The following conjecture asserts that the answer is
yes.
conjecture 29.10 The realizable sample complexity of every hypothesis class
H⊂ [k]Xis
mH(ϵ,δ) =˜O(Ndim(H)
ϵ)
.
We emphasize that the ˜Onotation may hide only poly-log factors of ϵ,δ, and
Ndim(H), but no factor ofk.

yes.
H⊂ [k]Xis
ϵ)
.
29.5 Bibliographic Remarks
The Natarajan dimension is due to Natarajan (1989). That paper also established
the Natarajan lemma and the generalization of the fundamental theorem. Gen-
eralizations and sharper versions of the Natarajan lemma are studied in Haussler
& Long (1995). Ben-David, Cesa-Bianchi, Haussler & Long (1995) deﬁned a large
family of notions of dimensions, all of which generalize the VC dimension and
may be used to estimate the sample complexity of multiclass classiﬁcation.

The calculation of the Natarajan dimension, presented here, together with
calculation of other classes, can be found in Daniely et al. (2012). The example
of good and bad ERMs, as well as conjecture 29.10, are from Daniely et al.
(2011).

29.6 Exercises
1. Letd,k> 0. Show that there exists a binary hypothesis Hbinof VC dimension
dsuch that Ndim(HOvA,k
bin) =d.
2. Prove Lemma 29.6.
3. Prove Natarajan’s lemma.
Hint: Fix somex0∈X. Fori,j∈[k], denote byHijall the functions f:
X\{x0}→ [k] that can be extended to a function in Hboth by deﬁning
f(x0) =iand by deﬁning f(x0) =j. Show that|H|≤|HX\{x0}|+∑
i̸=j|Hij|
and use induction.
4. Adapt the proof of the binary fundamental theorem and Natarajan’s lemma
to prove that, for some universal constant C > 0 and for every hypothesis
class of Natarajan dimension d, the agnostic sample complexity of His
mH(ϵ,δ)≤Cdlog(kd
ϵ)
+ log(1/δ)
ϵ2.
5. Prove that, for some universal constant C > 0 and for every hypothesis class
of Natarajan dimension d, the agnostic sample complexity of His
mH(ϵ,δ)≥Cd+ log(1/δ)
ϵ2.
Hint: Deduce it from the binary fundamental theorem.
6. LetHbe the binary hypothesis class of (nonhomogenous) halfspaces in Rd.

ϵ)
+ log(1/δ)
ϵ2.
ϵ2.
The goal of this exercise is to prove that Ndim( HOvA,k)≥(d−1)·(k−1).
1. LetHdiscrete be the class of all functions f: [k−1]×[d−1]→{0,1}for
which there exists some i0such that, for every j∈[d−1]
∀i<i 0,f(i,j) = 1 while∀i>i 0,f(i,j) = 0.
Show that Ndim( HOvA,k
discrete) = (d−1)·(k−1).
2. Show thatHdiscrete can be realized by H. That is, show that there exists
a mapping ψ: [k−1]×[d−1]→Rdsuch that
Hdiscrete⊂{h◦ψ:h∈H}.
Hint: You can take ψ(i,j) to be the vector whose jth coordinate is 1, whose
last coordinate is iand the rest are zeros.

3. Conclude that Ndim( HOvA,k)≥(d−1)·(k−1).

30 Compression Bounds
Throughout the book, we have tried to characterize the notion of learnability
using diﬀerent approaches. At ﬁrst we have shown that the uniform conver-
gence property of a hypothesis class guarantees successful learning. Later on we
introduced the notion of stability and have shown that stable algorithms are
guaranteed to be good learners. Yet there are other properties which may be
suﬃcient for learning, and in this chapter and its sequel we will introduce two
approaches to this issue: compression bounds and the PAC-Bayes approach.
In this chapter we study compression bounds. Roughly speaking, we shall see
that if a learning algorithm can express the output hypothesis using a small sub-
set of the training set, then the error of the hypothesis on the rest of the examples
estimates its true error. In other words, an algorithm that can “compress” its
output is a good learner.
30.1 Compression Bounds
To motivate the results, let us ﬁrst consider the following learning protocol.

First, we sample a sequence of kexamples denoted T. On the basis of these
examples, we construct a hypothesis denoted hT. Now we would like to estimate
the performance of hTso we sample a fresh sequence of m−kexamples, denoted
V, and calculate the error of hTonV. SinceVandTare independent, we
immediately get the following from Bernstein’s inequality (see Lemma B.10).
lemma 30.1 Assume that the range of the loss function is [0,1]. Then,
P[
LD(hT)−LV(hT)≥√
2LV(hT) log(1/δ)
|V|+4 log(1/δ)
|V|]
≤δ.
To derive this bound, all we needed was independence between TandV.

P[
|V|]
≤δ.
Therefore, we can redeﬁne the protocol as follows. First, we agree on a sequence
ofkindicesI= (i1,...,ik)∈[m]k. Then, we sample a sequence of mexamples
S= (z1,...,zm). Now, deﬁne T=SI= (zi1,...,zik) and deﬁne Vto be the
rest of the examples in S. Note that this protocol is equivalent to the protocol
we deﬁned before – hence Lemma 30.1 still holds.
Applying a union bound over the choice of the sequence of indices we obtain

30.1 Compression Bounds 411
theorem 30.2 Letkbe an integer and let B:Zk→H be a mapping from
sequences of kexamples to the hypothesis class. Let m≥2kbe a training set
size and let A:Zm→H be a learning rule that receives a training sequence S
of sizemand returns a hypothesis such that A(S) =B(zi1,...,zik)for some
(i1,...,ik)∈[m]k. LetV={zj:j /∈(i1,...,ik)}be the set of examples which
were not selected for deﬁning A(S). Then, with probability of at least 1−δover
the choice of Swe have
LD(A(S))≤LV(A(S)) +√
LV(A(S))4klog(m/δ)
m+8klog(m/δ)
m.
Proof For anyI∈[m]klethI=B(zi1,...,zik). Letn=m−k. Combining
Lemma 30.1 with the union bound we have
P[
∃I∈[m]ks.t. LD(hI)−LV(hI)≥√
2LV(hI) log(1/δ)
n+4 log(1/δ)
n]
≤∑
I∈[m]kP[
LD(hI)−LV(hI)≥√
n]
≤mkδ.
Denoteδ′=mkδ. Using the assumption k≤m/2, which implies that n=
m−k≥m/2, the above implies that with probability of at least 1 −δ′we have
that
LV(A(S))4klog(m/δ′)
m+8klog(m/δ′)
m,

m.
P[
n]
≤∑
I∈[m]kP[
n]
≤mkδ.
that
m,
corollary 30.3 Assuming the conditions of Theorem 30.2, and further as-
suming that LV(A(S)) = 0 , then, with probability of at least 1−δover the choice
ofSwe have
LD(A(S))≤8klog(m/δ)
m.
These results motivate the following deﬁnition:
definition 30.4 (Compression Scheme) LetHbe a hypothesis class of
functions fromXtoYand letkbe an integer. We say that Hhas a compression
scheme of size kif the following holds:
For allmthere exists A:Zm→[m]kandB:Zk→H such that for all h∈H,

ofSwe have
m.
if we feed any training set of the form ( x1,h(x1)),..., (xm,h(xm)) intoAand
then feed (xi1,h(xi1)),..., (xik,h(xik)) intoB, where (i1,...,ik) is the output
ofA, then the output of B, denotedh′, satisﬁesLS(h′) = 0.
It is possible to generalize the deﬁnition for unrealizable sequences as follows.

412 Compression Bounds
definition 30.5 (Compression Scheme for Unrealizable Sequences)
LetHbe a hypothesis class of functions from XtoYand letkbe an integer.
We say thatHhas a compression scheme of size kif the following holds:
if we feed any training set of the form ( x1,y1),..., (xm,ym) intoAand then
feed (xi1,yi1),..., (xik,yik) intoB, where (i1,...,ik) is the output of A, then
the output of B, denotedh′, satisﬁesLS(h′)≤LS(h).
The following lemma shows that the existence of a compression scheme for
the realizable case also implies the existence of a compression scheme for the
unrealizable case.
lemma 30.6 LetHbe a hypothesis class for binary classiﬁcation, and assume
it has a compression scheme of size kin the realizable case. Then, it has a
compression scheme of size kfor the unrealizable case as well.
Proof Consider the following scheme: First, ﬁnd an ERM hypothesis and denote

it byh. Then, discard all the examples on which herrs. Now, apply the realizable
compression scheme on the examples that have not been removed. The output of
the realizable compression scheme, denoted h′, must be correct on the examples
that have not been removed. Since herrs on the removed examples it follows
that the error of h′cannot be larger than the error of h; henceh′is also an ERM
hypothesis.
30.2 Examples
In the examples that follows, we present compression schemes for several hy-

pothesis classes for binary classiﬁcation. In light of Lemma 30.6 we focus on the
realizable case. Therefore, to show that a certain hypothesis class has a com-
pression scheme, it is necessary to show that there exist A,B, andkfor which
LS(h′) = 0.
30.2.1 Axis Aligned Rectangles
Note that this is an uncountable inﬁnite class. We show that there is a simple
compression scheme. Consider the algorithm Athat works as follows: For each
dimension, choose the two positive examples with extremal values at this dimen-

LS(h′) = 0.
sion. Deﬁne Bto be the function that returns the minimal enclosing rectangle.
Then, fork= 2d, we have that in the realizable case, LS(B(A(S))) = 0.
30.2.2 Halfspaces
LetX=Rdand consider the class of homogenous halfspaces, {x↦→sign(⟨w,x⟩) :
w∈Rd}.

30.2 Examples 413
A Compression Scheme:
W.l.o.g. assume all labels are positive (otherwise, replace xibyyixi). The com-
pression scheme we propose is as follows. First, Aﬁnds the vector wwhich is
in the convex hull of {x1,...,xm}and has minimal norm. Then, it represents it
as a convex combination of dpoints in the sample (it will be shown later that
this is always possible). The output of Aare thesedpoints. The algorithm B
receives these dpoints and set wto be the point in their convex hull of minimal
norm.
Next we prove that this indeed is a compression sceme. Since the data is
linearly separable, the convex hull of {x1,...,xm}does not contain the origin.
Consider the point win this convex hull closest to the origin. (This is a unique
point which is the Euclidean projection of the origin onto this convex hull.) We
claim that wseparates the data.1To see this, assume by contradiction that
⟨w,xi⟩≤0 for some i. Take w′= (1−α)w+αxiforα=∥w∥2
∥xi∥2+∥w∥2∈(0,1).
Then w′is also in the convex hull and

norm.
∥w′∥2= (1−α)2∥w∥2+α2∥xi∥2+ 2α(1−α)⟨w,xi⟩
≤(1−α)2∥w∥2+α2∥xi∥2
=∥xi∥4∥w∥2+∥xi∥2∥w∥4
(∥w∥2+∥xi∥2)2
=∥xi∥2∥w∥2
∥w∥2+∥xi∥2
=∥w∥2·1
∥w∥2/∥xi∥2+ 1
<∥w∥2,
We have thus shown that wis also an ERM. Finally, since wis in the convex
hull of the examples, we can apply Caratheodory’s theorem to obtain that wis
also in the convex hull of a subset of d+ 1 points of the polygon. Furthermore,
the minimality of wimplies that wmust be on a face of the polygon and this

=∥w∥2·1
<∥w∥2,
implies it can be represented as a convex combination of dpoints.
It remains to show that wis also the projection onto the polygon deﬁned by the
dpoints. But this must be true: On one hand, the smaller polygon is a subset of
the larger one; hence the projection onto the smaller cannot be smaller in norm.
On the other hand, witself is a valid solution. The uniqueness of projection
30.2.3 Separating Polynomials
LetX=Rdand consider the class x↦→sign(p(x)) wherepis a degree rpolyno-
mial.

mial.
1It can be shown that wis the direction of the max-margin solution.

414 Compression Bounds
Note thatp(x) can be rewritten as ⟨w,ψ(x)⟩where the elements of ψ(x) are all
the monomials of xup to degree r. Therefore, the problem of constructing a com-
pression scheme for p(x) reduces to the problem of constructing a compression
scheme for halfspaces in Rd′whered′=O(dr).
30.2.4 Separation with Margin
Suppose that a training set is separated with margin γ. The Perceptron algorithm
guarantees to make at most 1 /γ2updates before converging to a solution that
makes no mistakes on the entire training set. Hence, we have a compression
scheme of size k≤1/γ2.
30.3 Bibliographic Remarks
Compression schemes and their relation to learning were introduced by Little-
stone & Warmuth (1986). As we have shown, if a class has a compression scheme
then it is learnable. For binary classiﬁcation problems, it follows from the funda-
mental theorem of learning that the class has a ﬁnite VC dimension. The other
direction, namely, whether every hypothesis class of ﬁnite VC dimension has a

compression scheme of ﬁnite size, is an open problem posed by Manfred War-
muth and is still open (see also (Floyd 1989, Floyd & Warmuth 1995, Ben-David
& Litman 1998, Livni & Simon 2013).

31 PAC-Bayes
The Minimum Description Length (MDL) and Occam’s razor principles allow a
potentially very large hypothesis class but deﬁne a hierarchy over hypotheses and
prefer to choose hypotheses that appear higher in the hierarchy. In this chapter
we describe the PAC-Bayesian approach that further generalizes this idea. In
the PAC-Bayesian approach, one expresses the prior knowledge by deﬁning prior
distribution over the hypothesis class.
31.1 PAC-Bayes Bounds
As in the MDL paradigm, we deﬁne a hierarchy over hypotheses in our class H.
Now, the hierarchy takes the form of a prior distribution over H. That is, we
assign a probability (or density if His continuous) P(h)≥0 for eachh∈H
and refer to P(h) as the prior score of h. Following the Bayesian reasoning
approach, the output of the learning algorithm is not necessarily a single hy-
pothesis. Instead, the learning process deﬁnes a posterior probability over H,
which we denote by Q. In the context of a supervised learning problem, where

Hcontains functions from XtoY, one can think of Qas deﬁning a randomized
prediction rule as follows. Whenever we get a new instance x, we randomly pick
a hypothesis h∈H according to Qand predict h(x). We deﬁne the loss of Qon
an example zto be
ℓ(Q,z)def=E
h∼Q[ℓ(h,z)].
By the linearity of expectation, the generalization loss and training loss of Qcan
be written as
LD(Q)def=E
h∼Q[LD(h)] andLS(Q)def=E
h∼Q[LS(h)].
The following theorem tells us that the diﬀerence between the generalization

loss and the empirical loss of a posterior Qis bounded by an expression that
depends on the Kullback-Leibler divergence between Qand the prior distribu-
tionP. The Kullback-Leibler is a natural measure of the distance between two
distributions. The theorem suggests that if we would like to minimize the gen-
eralization loss of Q, we should jointly minimize both the empirical loss of Q
and the Kullback-Leibler distance between Qand the prior distribution. We will

416 PAC-Bayes
later show how in some cases this idea leads to the regularized risk minimization
principle.
theorem 31.1 LetDbe an arbitrary distribution over an example domain Z.
LetHbe a hypothesis class and let ℓ:H×Z→[0,1]be a loss function. Let Pbe
a prior distribution over Hand letδ∈(0,1). Then, with probability of at least
1−δover the choice of an i.i.d. training set S={z1,...,zm}sampled according
toD, for all distributions QoverH(even such that depend on S), we have
LD(Q)≤LS(Q) +√
D(Q||P) + lnm/δ
2(m−1),
where
D(Q||P)def=E
h∼Q[ln(Q(h)/P(h))]
is the Kullback-Leibler divergence.
Proof For any function f(S), using Markov’s inequality:
P
S[f(S)≥ϵ] =P
S[ef(S)≥eϵ]≤ES[ef(S)]
eϵ. (31.1)
Let ∆(h) =LD(h)−LS(h). We will apply Equation (31.1) with the function
f(S) = sup
Q(
2(m−1)E
h∼Q(∆(h))2−D(Q||P))
.
We now turn to bound ES[ef(S)]. The main trick is to upper bound f(S) by
using an expression that does not depend on Qbut rather depends on the prior

2(m−1),
where
P
eϵ. (31.1)
f(S) = sup
Q(
2(m−1)E
.
probability P. To do so, ﬁx some Sand note that from the deﬁnition of D(Q||P)
we get that for all Q,
2(m−1)E
h∼Q(∆(h))2−D(Q||P) =E
h∼Q[ln(e2(m−1)∆(h)2P(h)/Q(h))]
≤lnE
h∼Q[e2(m−1)∆(h)2P(h)/Q(h)]
= lnE
h∼P[e2(m−1)∆(h)2], (31.2)
where the inequality follows from Jensen’s inequality and the concavity of the
log function. Therefore,
E
S[ef(S)]≤E
SE
h∼P[e2(m−1)∆(h)2]. (31.3)
The advantage of the expression on the right-hand side stems from the fact that
we can switch the order of expectations (because Pis a prior that does not

2(m−1)E
≤lnE
= lnE
E
SE
depend onS), which yields
E
h∼PE
S[e2(m−1)∆(h)2]. (31.4)

Next, we claim that for all hwe have ES[e2(m−1)∆(h)2]≤m. To do so, recall that
Hoeﬀding’s inequality tells us that
P
S[∆(h)≥ϵ]≤e−2mϵ2.
This implies that ES[e2(m−1)∆(h)2]≤m(see Exercise 1). Combining this with
Equation (31.4) and plugging into Equation (31.1) we get
P
S[f(S)≥ϵ]≤m
eϵ. (31.5)
Denote the right-hand side of the above δ, thusϵ= ln(m/δ), and we therefore
obtain that with probability of at least 1 −δwe have that for all Q
2(m−1)E
h∼Q(∆(h))2−D(Q||P)≤ϵ= ln(m/δ).
Rearranging the inequality and using Jensen’s inequality again (the function x2
is convex) we conclude that
(
E
h∼Q∆(h))2
≤E
h∼Q(∆(h))2≤ln(m/δ) +D(Q||P)
2(m−1). (31.6)
Remark 31.1 (Regularization) The PAC-Bayes bound leads to the following
learning rule:
Given a prior P, return a posterior Qthat minimizes the function
LS(Q) +√
2(m−1). (31.7)
This rule is similar to the regularized risk minimization principle. That is, we
jointly minimize the empirical loss of Qon the sample and the Kullback-Leibler

(
E
h∼Q∆(h))2
≤E
LS(Q) +√
“distance” between QandP.
31.2 Bibliographic Remarks
PAC-Bayes bounds were ﬁrst introduced by McAllester (1998). See also (McAllester
1999, McAllester 2003, Seeger 2003, Langford & Shawe-Taylor 2003, Langford
2006).
31.3 Exercises
1. LetXbe a random variable that satisﬁes P[X≥ϵ]≤e−2mϵ2. Prove that
E[e2(m−1)X2]≤m.

418 PAC-Bayes
2.•Suppose thatHis a ﬁnite hypothesis class, set the prior to be uniform over
H, and set the posterior to be Q(hS) = 1 for some hSandQ(h) = 0 for
all otherh∈H. Show that
LD(hS)≤LS(h) +√
ln(|H|) + ln(m/δ)
2(m−1).
Compare to the bounds we derived using uniform convergence.
•Derive a bound similar to the Occam bound given in Chapter 7 using the
PAC-Bayes bound

Appendix A Technical Lemmas
lemma A.1 Leta>0. Then:x≥2alog(a)⇒x≥alog(x). It follows that a
necessary condition for the inequality x<a log(x)to hold is that x<2alog(a).
Proof First note that for a∈(0,√e] the inequality x≥alog(x) holds uncon-
ditionally and therefore the claim is trivial. From now on, assume that a>√e.
Consider the function f(x) =x−alog(x). The derivative is f′(x) = 1−a/x.
Thus, forx>a the derivative is positive and the function increases. In addition,
f(2alog(a)) = 2alog(a)−alog(2alog(a))
= 2alog(a)−alog(a)−alog(2 log(a))
=alog(a)−alog(2 log(a)).
Sincea−2 log(a)>0 for alla>0, the proof follows.
lemma A.2 Leta≥1andb>0. Then:x≥4alog(2a)+2b⇒x≥alog(x)+b.
Proof It suﬃces to prove that x≥4alog(2a) + 2bimplies that both x≥
2alog(x) andx≥2b. Since we assume a≥1 we clearly have that x≥2b.
In addition, since b >0 we have that x≥4alog(2a) which using Lemma A.1
implies that x≥2alog(x). This concludes our proof.
lemma A.3 LetXbe a random variable and x′∈Rbe a scalar and assume

that there exists a>0such that for all t≥0we have P[|X−x′|>t]≤2e−t2/a2.
Then, E[|X−x′|]≤4a.
Proof For alli= 0,1,2,...denoteti=ai. Sincetiis monotonically increasing
we have that E[|X−x′|] is at most∑∞
i=1tiP[|X−x′|> ti−1]. Combining this
with the assumption in the lemma we get that E[|X−x′|]≤2a∑∞
i=1ie−(i−1)2.
The proof now follows from the inequalities
∞∑
i=1ie−(i−1)2≤5∑
i=1ie−(i−1)2+∫∞
5xe−(x−1)2dx< 1.8 + 10−7<2.
lemma A.4 LetXbe a random variable and x′∈Rbe a scalar and assume

∞∑
that there exists a >0andb≥esuch that for all t≥0we have P[|X−x′|>
t]≤2be−t2/a2. Then, E[|X−x′|]≤a(2 +√
log(b)).

420 Technical Lemmas
Proof For alli= 0,1,2,...denoteti=a(i+√
log(b)). Sincetiis monotonically
increasing we have that
E[|X−x′|]≤a√
log(b) +∞∑
i=1tiP[|X−x′|>ti−1].
Using the assumption in the lemma we have
∞∑
i=1tiP[|X−x′|>ti−1]≤2ab∞∑
i=1(i+√
log(b))e−(i−1+√
log(b))2
≤2ab∫∞
1+√
log(b)xe−(x−1)2dx
= 2ab∫∞
√
log(b)(y+ 1)e−y2dy
≤4ab∫∞
√
log(b)ye−y2dy
= 2ab[
−e−y2]∞
√
log(b)
= 2ab/b = 2a.
Combining the preceding inequalities we conclude our proof.
lemma A.5 Letm,d be two positive integers such that d≤m−2. Then,
d∑
k=0(m
k)
≤(em
d)d
.
Proof We prove the claim by induction. For d= 1 the left-hand side equals
1 +mwhile the right-hand side equals em; hence the claim is true. Assume that
the claim holds for dand let us prove it for d+ 1. By the induction assumption
we have
d+1∑
k=0(m
k)
≤(em
d)d
+(m
d+ 1)
=(em
d)d(
1 +(d
em)dm(m−1)(m−2)···(m−d)
(d+ 1)d!)
≤(em
d)d(
1 +(d
e)d(m−d)
(d+ 1)d!)
.

Technical Lemmas 421
Using Stirling’s approximation we further have that
≤(em
d)d(
1 +(d
e)d(m−d)
(d+ 1)√
2πd(d/e)d)
=(em
d)d(
1 +m−d√
2πd(d+ 1))
=(em
d)d
·d+ 1 + (m−d)/√
2πd
d+ 1
≤(em
d)d
·d+ 1 + (m−d)/2
d+ 1
=(em
d)d
·d/2 + 1 +m/2
d+ 1
≤(em
d)d
·m
d+ 1,
where in the last inequality we used the assumption that d≤m−2. On the
other hand,
(em
d+ 1)d+1
=(em
d)d
·em
d+ 1·(d
d+ 1)d
=(em
d)d
·em
d+ 1·1
(1 + 1/d)d
≥(em
d)d
·em
d+ 1·1
e
=(em
d)d
·m
d+ 1,
which proves our inductive argument.
lemma A.6 For alla∈Rwe have
ea+e−a
2≤ea2/2.
Proof Observe that
ea=∞∑
n=0an
n!.
ea+e−a
2=∞∑
n=0a2n
(2n)!,
and
ea2/2=∞∑
n=0a2n
2nn!.
Observing that (2 n)!≥2nn! for everyn≥0 we conclude our proof.

Appendix B Measure Concentration
LetZ1,...,Zmbe an i.i.d. sequence of random variables and let µbe their mean.
The strong law of large numbers states that when mtends to inﬁnity, the em-
pirical average,1
m∑m
i=1Zi, converges to the expected value µ, with probability
1. Measure concentration inequalities quantify the deviation of the empirical
average from the expectation when mis ﬁnite.
B.1 Markov’s Inequality
We start with an inequality which is called Markov’s inequality. Let Zbe a
nonnegative random variable. The expectation of Zcan be written as follows:
E[Z] =∫∞
x=0P[Z≥x]dx. (B.1)
Since P[Z≥x] is monotonically nonincreasing we obtain
∀a≥0,E[Z]≥∫a
x=0P[Z≥x]dx≥∫a
x=0P[Z≥a]dx=aP[Z≥a].(B.2)
Rearranging the inequality yields Markov’s inequality:
∀a≥0,P[Z≥a]≤E[Z]
a. (B.3)
For random variables that take value in [0 ,1], we can derive from Markov’s
inequality the following.
lemma B.1 LetZbe a random variable that takes values in [0,1]. Assume that
E[Z] =µ. Then, for any a∈(0,1),
P[Z > 1−a]≥µ−(1−a)
a.

E[Z] =∫∞
a. (B.3)
a.
This also implies that for every a∈(0,1),
P[Z >a ]≥µ−a
1−a≥µ−a.
Proof LetY= 1−Z. ThenYis a nonnegative random variable with E[Y] =
1−E[Z] = 1−µ. Applying Markov’s inequality on Ywe obtain
P[Z≤1−a] =P[1−Z≥a] =P[Y≥a]≤E[Y]
a=1−µ
a.

B.2 Chebyshev’s Inequality 423
P[Z > 1−a]≥1−1−µ
a=a+µ−1
a.
B.2 Chebyshev’s Inequality
Applying Markov’s inequality on the random variable ( Z−E[Z])2we obtain
Chebyshev’s inequality:
∀a>0,P[|Z−E[Z]|≥a] =P[(Z−E[Z])2≥a2]≤Var[Z]
a2, (B.4)
where Var[Z] =E[(Z−E[Z])2] is the variance of Z.
Consider the random variable1
m∑m
i=1Zi. SinceZ1,...,Zmare i.i.d. it is easy
to verify that
Var[
1
mm∑
i=1Zi]
=Var[Z1]
m.
Applying Chebyshev’s inequality, we obtain the following:
lemma B.2 LetZ1,...,Zmbe a sequence of i.i.d. random variables and assume
thatE[Z1] =µandVar[Z1]≤1. Then, for any δ∈(0,1), with probability of at
least 1−δwe have
⏐⏐⏐⏐⏐1
mm∑
i=1Zi−µ⏐⏐⏐⏐⏐≤√
1
δm.
Proof Applying Chebyshev’s inequality we obtain that for all a>0
P[⏐⏐⏐⏐⏐1
mm∑
i=1Zi−µ⏐⏐⏐⏐⏐>a]
≤Var[Z1]
ma2≤1
ma2.
The proof follows by denoting the right-hand side δand solving for a.
The deviation between the empirical average and the mean given previously
decreases polynomially with m. It is possible to obtain a signiﬁcantly faster

⏐⏐⏐⏐⏐1
mm∑
1
δm.
P[⏐⏐⏐⏐⏐1
mm∑
≤Var[Z1]
ma2≤1
ma2.
decrease. In the sections that follow we derive bounds that decrease exponentially
fast.
B.3 Chernoﬀ’s Bounds
LetZ1,...,Zmbe independent Bernoulli variables where for every i,P[Zi= 1] =
piandP[Zi= 0] = 1−pi. Letp=∑m
i=1piand letZ=∑m
i=1Zi. Using the

424 Measure Concentration
monotonicity of the exponent function and Markov’s inequality, we have that for
everyt>0
P[Z > (1 +δ)p] =P[etZ>et(1+δ)p]≤E[etZ]
e(1+δ)tp. (B.5)
Next,
E[etZ] =E[et∑
iZi] =E[∏
ietZi]
=∏
iE[etZi] by independence
=∏
i(
piet+ (1−pi)e0)
=∏
i(
1 +pi(et−1))
≤∏
iepi(et−1)using 1 +x≤ex
=e∑
ipi(et−1)
=e(et−1)p.
Combining the above with Equation (B.5) and choosing t= log(1+δ) we obtain
lemma B.3 LetZ1,...,Zmbe independent Bernoulli variables where for every
i,P[Zi= 1] =piandP[Zi= 0] = 1−pi. Letp=∑m
i=1Zi.
Then, for any δ>0,
P[Z > (1 +δ)p]≤e−h(δ)p,
where
h(δ) = (1 +δ) log(1 +δ)−δ.
Using the inequality h(a)≥a2/(2 + 2a/3) we obtain
lemma B.4 Using the notation of Lemma B.3 we also have
P[Z > (1 +δ)p]≤e−pδ2
2+2δ/3.
For the other direction, we apply similar calculations:
P[Z < (1−δ)p] =P[−Z >−(1−δ)p] =P[e−tZ>e−t(1−δ)p]≤E[e−tZ]
e−(1−δ)tp,(B.6)

B.4 Hoeﬀding’s Inequality 425
and,
E[e−tZ] =E[e−t∑
iZi] =E[∏
ie−tZi]
=∏
iE[e−tZi] by independence
=∏
i(
1 +pi(e−t−1))
≤∏
iepi(e−t−1)using 1 +x≤ex
=e(e−t−1)p.
Settingt=−log(1−δ) yields
P[Z < (1−δ)p]≤e−δp
e(1−δ) log(1−δ)p=e−ph(−δ).
It is easy to verify that h(−δ)≥h(δ) and hence
lemma B.5 Using the notation of Lemma B.3 we also have
P[Z < (1−δ)p]≤e−ph(−δ)≤e−ph(δ)≤e−pδ2
2+2δ/3.
B.4 Hoeﬀding’s Inequality
lemma B.6 (Hoeﬀding’s inequality) LetZ1,...,Zmbe a sequence of i.i.d.
random variables and let ¯Z=1
m∑m
i=1Zi. Assume that E[¯Z] =µandP[a≤
Zi≤b] = 1 for everyi. Then, for any ϵ>0
P[⏐⏐⏐⏐⏐1
mm∑
i=1Zi−µ⏐⏐⏐⏐⏐>ϵ]
≤2 exp(
.
Proof DenoteXi=Zi−E[Zi] and ¯X=1
m∑
iXi. Using the monotonicity of
the exponent function and Markov’s inequality, we have that for every λ > 0
andϵ>0,
P[¯X≥ϵ] =P[eλ¯X≥eλϵ]≤e−λϵE[eλ¯X].
Using the independence assumption we also have
E[eλ¯X] =E[∏
ieλXi/m]
=∏
iE[eλXi/m].
By Hoeﬀding’s lemma (Lemma B.7 later), for every iwe have
E[eλXi/m]≤eλ2(b−a)2
8m2.

426 Measure Concentration
P[¯X≥ϵ]≤e−λϵ∏
ieλ2(b−a)2
8m2=e−λϵ+λ2(b−a)2
8m.
Settingλ= 4mϵ/(b−a)2we obtain
P[¯X≥ϵ]≤e−2mϵ2
(b−a)2.
Applying the same arguments on the variable −¯Xwe obtain that P[¯X≤−ϵ]≤
e−2mϵ2
(b−a)2. The theorem follows by applying the union bound on the two cases.
lemma B.7 (Hoeﬀding’s lemma) LetXbe a random variable that takes values
in the interval [a,b]and such that E[X] = 0 . Then, for every λ>0,
E[eλX]≤eλ2(b−a)2
8.
Proof Sincef(x) =eλxis a convex function, we have that for every α∈(0,1),
andx∈[a,b],
f(x)≤αf(a) + (1−α)f(b).
Settingα=b−x
b−a∈[0,1] yields
eλx≤b−x
b−aeλa+x−a
b−aeλb.
Taking the expectation, we obtain that
E[eλX]≤b−E[X]
b−aeλa+E[x]−a
b−aeλb=b
b−aeλa−a
b−aeλb,
where we used the fact that E[X] = 0. Denote h=λ(b−a),p=−a
b−a, and
L(h) =−hp+ log(1−p+peh). Then, the expression on the right-hand side of
the above can be rewritten as eL(h). Therefore, to conclude our proof it suﬃces
to show that L(h)≤h2
8. This follows from Taylor’s theorem using the facts:

eλx≤b−x
b−aeλb.
b−aeλb=b
b−aeλa−a
b−aeλb,
b−a, and
L(0) =L′(0) = 0 and L′′(h)≤1/4 for allh.
B.5 Bennet’s and Bernstein’s Inequalities
Bennet’s and Bernsein’s inequalities are similar to Chernoﬀ’s bounds, but they
hold for any sequence of independent random variables. We state the inequalities
without proof, which can be found, for example, in Cesa-Bianchi & Lugosi (2006).
lemma B.8 (Bennet’s inequality) LetZ1,...,Zmbe independent random vari-
ables with zero mean, and assume that Zi≤1with probability 1. Let
σ2≥1
mm∑
i=1E[Z2
i].

B.5 Bennet’s and Bernstein’s Inequalities 427
Then for all ϵ>0,
P[m∑
i=1Zi>ϵ]
≤e−mσ2h(ϵ
mσ2).
where
By using the inequality h(a)≥a2/(2 + 2a/3) it is possible to derive the
lemma B.9 (Bernstein’s inequality) LetZ1,...,Zmbe i.i.d. random variables
with a zero mean. If for all i, P(|Zi|<M ) = 1 , then for all t>0:
P[m∑
i=1Zi>t]
≤exp(
−t2/2∑EZ2
j+Mt/3)
.
B.5.1 Application
Bernstein’s inequality can be used to interpolate between the rate 1 /ϵwe derived
for PAC learning in the realizable case (in Chapter 2) and the rate 1 /ϵ2we derived
for the unrealizable case (in Chapter 4).
lemma B.10 Letℓ:H×Z→[0,1]be a loss function. Let Dbe an arbitrary
distribution over Z. Fix some h. Then, for any δ∈(0,1)we have
1.P
S∼Dm[
LS(h)≥LD(h) +√
2LD(h) log(1/δ)
3m+2 log(1/δ)
m]
≤δ
2.P
S∼Dm[
LD(h)≥LS(h) +√
2LS(h) log(1/δ)
m+4 log(1/δ)
m]
≤δ
Proof Deﬁne random variables α1,...,αms.t.αi=ℓ(h,zi)−LD(h). Note that
E[αi] = 0 and that
E[α2
i] =E[ℓ(h,zi)2]−2LD(h)E[ℓ(h,zi)] +LD(h)2
=E[ℓ(h,zi)2]−LD(h)2

1.P
S∼Dm[
m]
≤δ
2.P
S∼Dm[
m]
≤δ
E[α2
≤E[ℓ(h,zi)2]
≤E[ℓ(h,zi)] =LD(h),
where in the last inequality we used the fact that ℓ(h,zi)∈[0,1] and thus
ℓ(h,zi)2≤ℓ(h,zi). Applying Bernsein’s inequality over the αi’s yields
P[m∑
i=1αi>t]
≤exp(
−t2/2∑Eα2
j+t/3)
≤exp(
−t2/2
mLD(h) +t/3)
def=δ.

428 Measure Concentration
Solving for tyields
t2/2
mLD(h) +t/3= log(1/δ)
⇒t2/2−log(1/δ)
3t−log(1/δ)mLD(h) = 0
⇒t=log(1/δ)
3+√
log2(1/δ)
32+ 2 log(1/δ)mLD(h)
≤2log(1/δ)
3+√
2 log(1/δ)mLD(h)
Since1
m∑
iαi=LS(h)−LD(h), it follows that with probability of at least 1 −δ,
LS(h)−LD(h)≤2log(1/δ)
3m+√
2 log(1/δ)LD(h)
m,
which proves the ﬁrst inequality. The second part of the lemma follows in a
similar way.
B.6 Slud’s Inequality
LetXbe a (m,p) binomial variable. That is, X=∑m
i=1Zi, where each Ziis 1
with probability pand 0 with probability 1 −p. Assume that p= (1−ϵ)/2. Slud’s
inequality (Slud 1977) tells us that P[X≥m/2] is lower bounded by the proba-
bility that a normal variable will be greater than or equal to√
mϵ2/(1−ϵ2). The
following lemma follows by standard tail bounds for the normal distribution.
lemma B.11 LetXbe a(m,p)binomial variable and assume that p= (1−ϵ)/2.
Then,
P[X≥m/2]≥1
2(
1−√
.
B.7 Concentration of χ2Variables

Then,
2(
1−√
.
LetX1,...,Xkbekindependent normally distributed random variables. That
is, for alli,Xi∼N(0,1). The distribution of the random variable X2
iis called
χ2(chi square) and the distribution of the random variable Z=X2
1+···+X2
k
is calledχ2
k(chi square with kdegrees of freedom). Clearly, E[X2
i] = 1 and
E[Z] =k. The following lemma states that X2
kis concentrated around its mean.
lemma B.12 LetZ∼χ2
k. Then, for all ϵ>0we have
P[Z≤(1−ϵ)k]≤e−ϵ2k/6,
and for all ϵ∈(0,3)we have
P[Z≥(1 +ϵ)k]≤e−ϵ2k/6.

B.7 Concentration of χ2Variables 429
Finally, for all ϵ∈(0,3),
P[(1−ϵ)k≤Z≤(1 +ϵ)k]≥1−2e−ϵ2k/6.
Proof Let us write Z=∑k
i=1X2
iwhereXi∼N(0,1). To prove both bounds
we use Chernoﬀ’s bounding method. For the ﬁrst inequality, we ﬁrst bound
E[e−λX2
1], whereλ>0 will be speciﬁed later. Since e−a≤1−a+a2
2for alla≥0
E[e−λX2
1]≤1−λE[X2
1] +λ2
2E[X4
1].
Using the well known equalities, E[X2
1] = 1 and E[X4
1] = 3, and the fact that
1−a≤e−awe obtain that
E[e−λX2
1]≤1−λ+3
2λ2≤e−λ+3
2λ2.
Now, applying Chernoﬀ’s bounding method we get that
P[−Z≥−(1−ϵ)k] =P[
e−λZ≥e−(1−ϵ)kλ]
≤e(1−ϵ)kλE[
e−λZ]
=e(1−ϵ)kλ(
E[
e−λX2
1])k
≤e(1−ϵ)kλe−λk+3
2λ2k
=e−ϵkλ+3
2kλ2.
Chooseλ=ϵ/3 we obtain the ﬁrst inequality stated in the lemma.
For the second inequality, we use a known closed form expression for the
moment generating function of a χ2
kdistributed random variable:
∀λ<1
2,E[
eλZ2]
= (1−2λ)−k/2. (B.7)
On the basis of the equation and using Chernoﬀ’s bounding method we have
P[Z≥(1 +ϵ)k)] =P[
eλZ≥e(1+ϵ)kλ]
≤e−(1+ϵ)kλE[
eλZ]

e−λZ]
E[
e−λX2
1])k
2λ2k
=e−ϵkλ+3
2kλ2.
∀λ<1
2,E[
eλZ2]
eλZ]
=e−(1+ϵ)kλ(1−2λ)−k/2
≤e−(1+ϵ)kλekλ=e−ϵkλ,
where the last inequality occurs because (1 −a)≤e−a. Settingλ=ϵ/6 (which
is in (0,1/2) by our assumption) we obtain the second inequality stated in the
lemma.
Finally, the last inequality follows from the ﬁrst two inequalities and the union
bound.

Appendix C Linear Algebra
C.1 Basic Deﬁnitions
In this chapter we only deal with linear algebra over ﬁnite dimensional Euclidean
spaces. We refer to vectors as column vectors.
Given twoddimensional vectors u,v∈Rd, their inner product is
⟨u,v⟩=d∑
i=1uivi.
The Euclidean norm (a.k.a. the ℓ2norm) is∥u∥=√
⟨u,u⟩. We also use the ℓ1
norm,∥u∥1=∑d
i=1|ui|and theℓ∞norm∥u∥∞= maxi|ui|.
A subspace of Rdis a subset of Rdwhich is closed under addition and scalar
multiplication. The span of a set of vectors u1,...,ukis the subspace containing
all vectors of the form
k∑
i=1αiui
where for all i,αi∈R.
A set of vectors U={u1,...,uk}is independent if for every i,uiis not in the
span of u1,...,ui−1,ui+1,...,uk. We say that Uspans a subspace VifVis the
span of the vectors in U. We say that Uis abasis ofVif it is both independent
and spansV. The dimension of Vis the size of a basis of V(and it can be veriﬁed
that all bases of Vhave the same size). We say that Uis an orthogonal set if for

k∑
i=1αiui
alli̸=j,⟨ui,uj⟩= 0. We say that Uis an orthonormal set if it is orthogonal
and if for every i,∥ui∥= 1.
Given a matrix A∈Rn,d, the range of Ais the span of its columns and the
null space of Ais the subspace of all vectors that satisfy Au=0. The rank of A
is the dimension of its range.
The transpose of a matrix A, denotedA⊤, is the matrix whose ( i,j) entry
equals the ( j,i) entry ofA. We say that Ais symmetric if A=A⊤.

C.2 Eigenvalues and Eigenvectors 431
C.2 Eigenvalues and Eigenvectors
LetA∈Rd,dbe a matrix. A non-zero vector uis an eigenvector of Awith a
corresponding eigenvalue λif
Au=λu.
theorem C.1 (Spectral Decomposition) IfA∈Rd,dis a symmetric matrix of
rankk, then there exists an orthonormal basis of Rd,u1,...,ud, such that each
uiis an eigenvector of A. Furthermore, Acan be written as A=∑d
i=1λiuiu⊤
i,
where each λiis the eigenvalue corresponding to the eigenvector ui. This can
be written equivalently as A=UDU⊤, where the columns of Uare the vectors
u1,...,ud, andDis a diagonal matrix with Di,i=λiand fori̸=j,Di,j=
0. Finally, the number of λiwhich are nonzero is the rank of the matrix, the
eigenvectors which correspond to the nonzero eigenvalues span the range of A,
and the eigenvectors which correspond to zero eigenvalues span the null space of
A.
C.3 Positive deﬁnite matrices
A symmetric matrix A∈Rd,dis positive deﬁnite if all its eigenvalues are positive.

A.
Ais positive semideﬁnite if all its eigenvalues are nonnegative.
theorem C.2 LetA∈Rd,dbe a symmetric matrix. Then, the following are
equivalent deﬁnitions of positive semideﬁniteness of A:
•All the eigenvalues of Aare nonnegative.
•For every vector u,⟨u,Au⟩≥0.
•There exists a matrix Bsuch thatA=BB⊤.
C.4 Singular Value Decomposition (SVD)
LetA∈Rm,nbe a matrix of rank r. Whenm̸=n, the eigenvalue decomposition
given in Theorem C.1 cannot be applied. We will describe another decomposition

ofA, which is called Singular Value Decomposition, or SVD for short.
Unit vectors v∈Rnandu∈Rmare called right and left singular vectors of
Awith corresponding singular value σ>0 if
Av=σuandA⊤u=σv.
We ﬁrst show that if we can ﬁnd rorthonormal singular vectors with positive
singular values, then we can decompose A=UDV⊤, with the columns of Uand
Vcontaining the left and right singular vectors, and Dbeing a diagonal r×r
matrix with the singular values on its diagonal.

432 Linear Algebra
lemma C.3 LetA∈Rm,nbe a matrix of rank r. Assume that v1,...,vris an
orthonormal set of right singular vectors of A,u1,...,uris an orthonormal set
of corresponding left singular vectors of A, andσ1,...,σrare the corresponding
singular values. Then,
A=r∑
i=1σiuiv⊤
i.
It follows that if Uis a matrix whose columns are the ui’s,Vis a matrix whose
columns are the vi’s, andDis a diagonal matrix with Di,i=σi, then
A=UDV⊤.
Proof Any right singular vector of Amust be in the range of A⊤(otherwise,
the singular value will have to be zero). Therefore, v1,...,vris an orthonormal
basis of the range of A. Let us complete it to an orthonormal basis of Rnby
adding the vectors vr+1,...,vn. DeﬁneB=∑r
i=1σiuiv⊤
i. It suﬃces to prove
that for all i,Avi=Bvi. Clearly, if i > r thenAvi= 0 andBvi= 0 as well.
Fori≤rwe have
Bvi=r∑
j=1σjujv⊤
jvi=σiui=Avi,
where the last equality follows from the deﬁnition.
The next lemma relates the singular values of Ato the eigenvalues of A⊤A
andAA⊤.

i=1σiuiv⊤
Bvi=r∑
j=1σjujv⊤
andAA⊤.
lemma C.4 v,uare right and left singular vectors of Awith singular value σ
iﬀvis an eigenvector of A⊤Awith corresponding eigenvalue σ2andu=σ−1Av
is an eigenvector of AA⊤with corresponding eigenvalue σ2.
Proof Suppose that σis a singular value of Awithv∈Rnbeing the corre-
sponding right singular vector. Then,
A⊤Av=σA⊤u=σ2v.
AA⊤u=σAv=σ2u.
For the other direction, if λ̸= 0 is an eigenvalue of A⊤A, with vbeing the
corresponding eigenvector, then λ>0 becauseA⊤Ais positive semideﬁnite. Let
σ=√
λ,u=σ−1Av. Then,
σu=√
λAv√
λ=Av,
and

σ=√
σu=√
λAv√
λ=Av,
and
A⊤u=1
σA⊤Av=λ
σv=σv.

C.4 Singular Value Decomposition (SVD) 433
Finally, we show that if Ahas rankrthen it has rorthonormal singular
vectors.
lemma C.5 LetA∈Rm,nwith rankr. Deﬁne the following vectors:
v1= argmax
v∈Rn:∥v∥=1∥Av∥
v2= argmax
v∈Rn:∥v∥=1
⟨v,v1⟩=0∥Av∥
...
vr= argmax
∀i<r,⟨v,vi⟩=0∥Av∥
Then, v1,...,vris an orthonormal set of right singular vectors of A.
Proof First note that since the rank of Aisr, the range of Ais a subspace of
dimensionr, and therefore it is easy to verify that for all i= 1,...,r ,∥Avi∥>0.
LetW∈Rn,nbe an orthonormal matrix obtained by the eigenvalue decompo-
sition ofA⊤A, namely,A⊤A=WDW⊤, withDbeing a diagonal matrix with
D1,1≥D2,2≥···≥ 0. We will show that v1,...,vrare eigenvectors of A⊤A
that correspond to nonzero eigenvalues, and, hence, using Lemma C.4 it follows
that these are also right singular vectors of A. The proof is by induction. For the
basis of the induction, note that any unit vector vcan be written as v=Wx,
forx=W⊤v, and note that∥x∥= 1. Therefore,

∥Av∥2=∥AWx∥2=∥WDW⊤Wx∥2=∥WDx∥2=∥Dx∥2=n∑
i=1D2
i,ixi2.
max
v:∥v∥=1∥Av∥2= max
x:∥x∥=1n∑
i=1D2
i,ixi2.
The solution of the right-hand side is to set x= (1,0,..., 0), which implies that
v1is the ﬁrst eigenvector of A⊤A. Since∥Av1∥>0 it follows that D1,1>0 as
required. For the induction step, assume that the claim holds for some 1 ≤t≤
r−1. Then, any vwhich is orthogonal to v1,...,vtcan be written as v=Wx
with all the ﬁrst telements of xbeing zero. It follows that
max
v:∥v∥=1,∀i≤t,v⊤vi=0∥Av∥2= max
x:∥x∥=1n∑
i=t+1D2
i,ixi2.

i=1D2
i,ixi2.
max
x:∥x∥=1n∑
i=1D2
i,ixi2.
max
x:∥x∥=1n∑
i=t+1D2
i,ixi2.
The solution of the right-hand side is the all zeros vector except xt+1= 1. This
implies that vt+1is the (t+ 1)th column of W. Finally, since∥Avt+1∥>0 it
follows that Dt+1,t+1>0 as required. This concludes our proof.

434 Linear Algebra
corollary C.6 (The SVD theorem) LetA∈Rm,nwith rankr. ThenA=
UDV⊤whereDis anr×rmatrix with nonzero singular values of Aand the
columns of U,V are orthonormal left and right singular vectors of A. Further-
more, for all i,D2
i,iis an eigenvalue of A⊤A, theith column of Vis the cor-
responding eigenvector of A⊤Aand theith column of Uis the corresponding
eigenvector of AA⊤.

Notes

Abernethy, J., Bartlett, P. L., Rakhlin, A. & Tewari, A. (2008), Optimal strategies and
minimax lower bounds for online convex games, in‘Proceedings of the Nineteenth
Annual Conference on Computational Learning Theory’.
Ackerman, M. & Ben-David, S. (2008), Measures of clustering quality: A working set
of axioms for clustering, in‘Proceedings of Neural Information Processing Systems
(NIPS)’, pp. 121–128.
Agarwal, S. & Roth, D. (2005), Learnability of bipartite ranking functions, in‘Pro-
ceedings of the 18th Annual Conference on Learning Theory’, pp. 16–31.
Agmon, S. (1954), ‘The relaxation method for linear inequalities’, Canadian Journal
of Mathematics 6(3), 382–392.
Aizerman, M. A., Braverman, E. M. & Rozonoer, L. I. (1964), ‘Theoretical foundations
of the potential function method in pattern recognition learning’, Automation and
Remote Control 25, 821–837.
Allwein, E. L., Schapire, R. & Singer, Y. (2000), ‘Reducing multiclass to binary: A uni-

fying approach for margin classiﬁers’, Journal of Machine Learning Research 1, 113–
141.
Alon, N., Ben-David, S., Cesa-Bianchi, N. & Haussler, D. (1997), ‘Scale-sensitive dimen-
sions, uniform convergence, and learnability’, Journal of the ACM 44(4), 615–631.
Anthony, M. & Bartlet, P. (1999), Neural Network Learning: Theoretical Foundations ,
Baraniuk, R., Davenport, M., DeVore, R. & Wakin, M. (2008), ‘A simple proof of
the restricted isometry property for random matrices’, Constructive Approximation
28(3), 253–263.

141.
Barber, D. (2012), Bayesian reasoning and machine learning , Cambridge University
Press.
Bartlett, P., Bousquet, O. & Mendelson, S. (2005), ‘Local rademacher complexities’,
Annals of Statistics 33(4), 1497–1537.
Bartlett, P. L. & Ben-David, S. (2002), ‘Hardness results for neural network approxi-
mation problems’, Theor. Comput. Sci. 284(1), 53–66.
Bartlett, P. L., Long, P. M. & Williamson, R. C. (1994), Fat-shattering and the learn-
ability of real-valued functions, in‘Proceedings of the seventh annual conference on

Press.
Computational learning theory’, ACM, pp. 299–310.
Bartlett, P. L. & Mendelson, S. (2001), Rademacher and Gaussian complexities: Risk
bounds and structural results, in‘14th Annual Conference on Computational Learn-
ing Theory, COLT 2001’, Vol. 2111, Springer, Berlin, pp. 224–240.

438 References
Bartlett, P. L. & Mendelson, S. (2002), ‘Rademacher and Gaussian complexities: Risk
bounds and structural results’, Journal of Machine Learning Research 3, 463–482.
Ben-David, S., Cesa-Bianchi, N., Haussler, D. & Long, P. (1995), ‘Characterizations
of learnability for classes of {0,...,n}-valued functions’, Journal of Computer and
System Sciences 50, 74–86.
Ben-David, S., Eiron, N. & Long, P. (2003), ‘On the diﬃculty of approximately maxi-
mizing agreements’, Journal of Computer and System Sciences 66(3), 496–514.
Ben-David, S. & Litman, A. (1998), ‘Combinatorial variability of vapnik-chervonenkis
classes with applications to sample compression schemes’, Discrete Applied Mathe-
matics 86(1), 3–25.
Ben-David, S., Pal, D., & Shalev-Shwartz, S. (2009), Agnostic online learning, in‘Con-
ference on Learning Theory (COLT)’.
Ben-David, S. & Simon, H. (2001), ‘Eﬃcient learning of linear perceptrons’, Advances
in Neural Information Processing Systems pp. 189–195.

Bengio, Y. (2009), ‘Learning deep architectures for AI’, Foundations and Trends in
Machine Learning 2(1), 1–127.
Bengio, Y. & LeCun, Y. (2007), ‘Scaling learning algorithms towards ai’, Large-Scale
Kernel Machines 34.
Bertsekas, D. (1999), Nonlinear Programming , Athena Scientiﬁc.
Beygelzimer, A., Langford, J. & Ravikumar, P. (2007), ‘Multiclass classiﬁcation with
ﬁlter trees’, Preprint, June .
Birkhoﬀ, G. (1946), ‘Three observations on linear algebra’, Revi. Univ. Nac. Tucuman,
ser A 5, 147–151.

Bishop, C. M. (2006), Pattern recognition and machine learning , Vol. 1, springer New
York.
Blum, L., Shub, M. & Smale, S. (1989), ‘On a theory of computation and complexity
over the real numbers: Np-completeness, recursive functions and universal machines’,
Am. Math. Soc 21(1), 1–46.
Blumer, A., Ehrenfeucht, A., Haussler, D. & Warmuth, M. K. (1987), ‘Occam’s razor’,
Information Processing Letters 24(6), 377–380.
Blumer, A., Ehrenfeucht, A., Haussler, D. & Warmuth, M. K. (1989), ‘Learnability

York.
and the Vapnik-Chervonenkis dimension’, Journal of the Association for Computing
Machinery 36(4), 929–965.
Borwein, J. & Lewis, A. (2006), Convex Analysis and Nonlinear Optimization , Springer.
Boser, B. E., Guyon, I. M. & Vapnik, V. N. (1992), A training algorithm for optimal
margin classiﬁers, in‘Conference on Learning Theory (COLT)’, pp. 144–152.
Bottou, L. & Bousquet, O. (2008), The tradeoﬀs of large scale learning, in‘NIPS’,
pp. 161–168.

Boucheron, S., Bousquet, O. & Lugosi, G. (2005), ‘Theory of classiﬁcation: a survey of
recent advances’, ESAIM: Probability and Statistics 9, 323–375.
Bousquet, O. (2002), Concentration Inequalities and Empirical Processes Theory Ap-
plied to the Analysis of Learning Algorithms, PhD thesis, Ecole Polytechnique.
Bousquet, O. & Elisseeﬀ, A. (2002), ‘Stability and generalization’, Journal of Machine
Learning Research 2, 499–526.
Boyd, S. & Vandenberghe, L. (2004), Convex Optimization , Cambridge University
Press.

References 439
Breiman, L. (1996), Bias, variance, and arcing classiﬁers, Technical Report 460, Statis-
tics Department, University of California at Berkeley.
Breiman, L. (2001), ‘Random forests’, Machine learning 45(1), 5–32.
Breiman, L., Friedman, J. H., Olshen, R. A. & Stone, C. J. (1984), Classiﬁcation and
Regression Trees , Wadsworth & Brooks.
Cand` es, E. (2008), ‘The restricted isometry property and its implications for com-
pressed sensing’, Comptes Rendus Mathematique 346(9), 589–592.
Candes, E. J. (2006), Compressive sampling, in‘Proc. of the Int. Congress of Math.,
Madrid, Spain’.
Candes, E. & Tao, T. (2005), ‘Decoding by linear programming’, IEEE Trans. on
Information Theory 51, 4203–4215.
Cesa-Bianchi, N. & Lugosi, G. (2006), Prediction, learning, and games , Cambridge
Chang, H. S., Weiss, Y. & Freeman, W. T. (2009), ‘Informative sensing’, arXiv preprint
arXiv:0901.4275 .
Chapelle, O., Le, Q. & Smola, A. (2007), Large margin optimization of ranking mea-

sures, in‘NIPS Workshop: Machine Learning for Web Search’.
Collins, M. (2000), Discriminative reranking for natural language parsing, in‘Machine
Learning’.
Collins, M. (2002), Discriminative training methods for hidden Markov models: Theory
and experiments with perceptron algorithms, in‘Conference on Empirical Methods
in Natural Language Processing’.
Collobert, R. & Weston, J. (2008), A uniﬁed architecture for natural language process-
ing: deep neural networks with multitask learning, in‘International Conference on

Machine Learning (ICML)’.
Cortes, C. & Vapnik, V. (1995), ‘Support-vector networks’, Machine Learning
20(3), 273–297.
Cover, T. (1965), ‘Behavior of sequential predictors of binary sequences’, Trans. 4th
Prague Conf. Information Theory Statistical Decision Functions, Random Processes
pp. 263–272.
Cover, T. & Hart, P. (1967), ‘Nearest neighbor pattern classiﬁcation’, Information
Theory, IEEE Transactions on 13(1), 21–27.
Crammer, K. & Singer, Y. (2001), ‘On the algorithmic implementation of multiclass

kernel-based vector machines’, Journal of Machine Learning Research 2, 265–292.
Cristianini, N. & Shawe-Taylor, J. (2000), An Introduction to Support Vector Machines ,
Daniely, A., Sabato, S., Ben-David, S. & Shalev-Shwartz, S. (2011), Multiclass learn-
ability and the erm principle, in‘Conference on Learning Theory (COLT)’.
Daniely, A., Sabato, S. & Shwartz, S. S. (2012), Multiclass learning approaches: A
theoretical comparison with implications, in‘NIPS’.

Davis, G., Mallat, S. & Avellaneda, M. (1997), ‘Greedy adaptive approximation’, Jour-
nal of Constructive Approximation 13, 57–98.
Devroye, L. & Gy¨ orﬁ, L. (1985), Nonparametric Density Estimation: The L B1 S View ,
Wiley.
Devroye, L., Gy¨ orﬁ, L. & Lugosi, G. (1996), A Probabilistic Theory of Pattern Recog-
nition , Springer.

440 References
Dietterich, T. G. & Bakiri, G. (1995), ‘Solving multiclass learning problems via error-
correcting output codes’, Journal of Artiﬁcial Intelligence Research 2, 263–286.
Donoho, D. L. (2006), ‘Compressed sensing’, Information Theory, IEEE Transactions
on52(4), 1289–1306.
Dudley, R., Gine, E. & Zinn, J. (1991), ‘Uniform and universal glivenko-cantelli classes’,
Journal of Theoretical Probability 4(3), 485–510.
Dudley, R. M. (1987), ‘Universal Donsker classes and metric entropy’, Annals of Prob-
ability 15(4), 1306–1326.
Fisher, R. A. (1922), ‘On the mathematical foundations of theoretical statistics’, Philo-
sophical Transactions of the Royal Society of London. Series A, Containing Papers
of a Mathematical or Physical Character 222, 309–368.
Floyd, S. (1989), Space-bounded learning and the Vapnik-Chervonenkis dimension, in
‘Conference on Learning Theory (COLT)’, pp. 349–364.
Floyd, S. & Warmuth, M. (1995), ‘Sample compression, learnability, and the Vapnik-

Chervonenkis dimension’, Machine Learning 21(3), 269–304.
Frank, M. & Wolfe, P. (1956), ‘An algorithm for quadratic programming’, Naval Res.
Logist. Quart. 3, 95–110.
Freund, Y. & Schapire, R. (1995), A decision-theoretic generalization of on-line learning
and an application to boosting, in‘European Conference on Computational Learning
Theory (EuroCOLT)’, Springer-Verlag, pp. 23–37.
Freund, Y. & Schapire, R. E. (1999), ‘Large margin classiﬁcation using the perceptron
algorithm’, Machine Learning 37(3), 277–296.

Garcia, J. & Koelling, R. (1996), ‘Relation of cue to consequence in avoidance learning’,
Foundations of animal behavior: classic papers with commentaries 4, 374.
Gentile, C. (2003), ‘The robustness of the p-norm algorithms’, Machine Learning
53(3), 265–299.
Georghiades, A., Belhumeur, P. & Kriegman, D. (2001), ‘From few to many: Illumina-
tion cone models for face recognition under variable lighting and pose’, IEEE Trans.
Pattern Anal. Mach. Intelligence 23(6), 643–660.
Gordon, G. (1999), Regret bounds for prediction problems, in‘Conference on Learning

Theory (COLT)’.
Gottlieb, L.-A., Kontorovich, L. & Krauthgamer, R. (2010), Eﬃcient classiﬁcation for
metric data, in‘23rd Conference on Learning Theory’, pp. 433–440.
Guyon, I. & Elisseeﬀ, A. (2003), ‘An introduction to variable and feature selection’,
Journal of Machine Learning Research, Special Issue on Variable and Feature Selec-
tion3, 1157–1182.
Hadamard, J. (1902), ‘Sur les probl` emes aux d´ eriv´ ees partielles et leur signiﬁcation
physique’, Princeton University Bulletin 13, 49–52.

Hastie, T., Tibshirani, R. & Friedman, J. (2001), The Elements of Statistical Learning ,
Springer.
Haussler, D. (1992), ‘Decision theoretic generalizations of the PAC model for neural
net and other learning applications’, Information and Computation 100(1), 78–150.
Haussler, D. & Long, P. M. (1995), ‘A generalization of sauer’s lemma’, Journal of
Combinatorial Theory, Series A 71(2), 219–240.
Hazan, E., Agarwal, A. & Kale, S. (2007), ‘Logarithmic regret algorithms for online

Springer.
convex optimization’, Machine Learning 69(2–3), 169–192.

References 441
Hinton, G. E., Osindero, S. & Teh, Y.-W. (2006), ‘A fast learning algorithm for deep
belief nets’, Neural Computation 18(7), 1527–1554.
Hiriart-Urruty, J.-B. & Lemar´ echal, C. (1996), Convex Analysis and Minimization Al-
gorithms: Part 1: Fundamentals , Vol. 1, Springer.
Hsu, C.-W., Chang, C.-C. & Lin, C.-J. (2003), ‘A practical guide to support vector
classiﬁcation’.
Hyaﬁl, L. & Rivest, R. L. (1976), ‘Constructing optimal binary decision trees is NP-
complete’, Information Processing Letters 5(1), 15–17.
Joachims, T. (2005), A support vector method for multivariate performance measures,
in‘Proceedings of the International Conference on Machine Learning (ICML)’.
Kakade, S., Sridharan, K. & Tewari, A. (2008), On the complexity of linear prediction:
Risk bounds, margin bounds, and regularization, in‘NIPS’.
Karp, R. M. (1972), Reducibility among combinatorial problems , Springer.
Kearns, M. J., Schapire, R. E. & Sellie, L. M. (1994), ‘Toward eﬃcient agnostic learn-

ing’, Machine Learning 17, 115–141.
Kearns, M. & Mansour, Y. (1996), On the boosting ability of top-down decision tree
learning algorithms, in‘ACM Symposium on the Theory of Computing (STOC)’.
Kearns, M. & Ron, D. (1999), ‘Algorithmic stability and sanity-check bounds for leave-
one-out cross-validation’, Neural Computation 11(6), 1427–1453.
Kearns, M. & Valiant, L. G. (1988), Learning Boolean formulae or ﬁnite automata is
as hard as factoring, Technical Report TR-14-88, Harvard University Aiken Compu-
tation Laboratory.

Kearns, M. & Vazirani, U. (1994), An Introduction to Computational Learning Theory ,
MIT Press.
Kleinberg, J. (2003), ‘An impossibility theorem for clustering’, Advances in Neural
Information Processing Systems pp. 463–470.
Klivans, A. R. & Sherstov, A. A. (2006), Cryptographic hardness for learning intersec-
tions of halfspaces, in‘FOCS’.
Koller, D. & Friedman, N. (2009), Probabilistic Graphical Models: Principles and Tech-
niques , MIT Press.
Koltchinskii, V. & Panchenko, D. (2000), Rademacher processes and bounding the risk

MIT Press.
of function learning, in‘High Dimensional Probability II’, Springer, pp. 443–457.
Kuhn, H. W. (1955), ‘The hungarian method for the assignment problem’, Naval re-
search logistics quarterly 2(1-2), 83–97.
Kutin, S. & Niyogi, P. (2002), Almost-everywhere algorithmic stability and general-
ization error, in‘Proceedings of the 18th Conference in Uncertainty in Artiﬁcial
Intelligence’, pp. 275–282.
Laﬀerty, J., McCallum, A. & Pereira, F. (2001), Conditional random ﬁelds: Probabilistic
models for segmenting and labeling sequence data, in‘International Conference on

Machine Learning’, pp. 282–289.
Langford, J. (2006), ‘Tutorial on practical prediction theory for classiﬁcation’, Journal
of machine learning research 6(1), 273.
Langford, J. & Shawe-Taylor, J. (2003), PAC-Bayes & margins, in‘NIPS’, pp. 423–430.
Le Cun, L. (2004), Large scale online learning., in‘Advances in Neural Information
Processing Systems 16: Proceedings of the 2003 Conference’, Vol. 16, MIT Press,
p. 217.

442 References
Le, Q. V., Ranzato, M.-A., Monga, R., Devin, M., Corrado, G., Chen, K., Dean, J. &
Ng, A. Y. (2012), Building high-level features using large scale unsupervised learning,
in‘International Conference on Machine Learning (ICML)’.
Lecun, Y. & Bengio, Y. (1995), Convolutional Networks for Images, Speech and Time
Series , The MIT Press, pp. 255–258.
Lee, H., Grosse, R., Ranganath, R. & Ng, A. (2009), Convolutional deep belief networks
for scalable unsupervised learning of hierarchical representations, in‘International
Conference on Machine Learning (ICML)’.
Littlestone, N. (1988), ‘Learning quickly when irrelevant attributes abound: A new
linear-threshold algorithm’, Machine Learning 2, 285–318.
Littlestone, N. & Warmuth, M. (1986), Relating data compression and learnability.
Unpublished manuscript.
Littlestone, N. & Warmuth, M. K. (1994), ‘The weighted majority algorithm’, Infor-
mation and Computation 108, 212–261.

Livni, R., Shalev-Shwartz, S. & Shamir, O. (2013), ‘A provably eﬃcient algorithm for
training deep networks’, arXiv preprint arXiv:1304.7045 .
Livni, R. & Simon, P. (2013), Honest compressions and their application to compression
schemes, in‘Conference on Learning Theory (COLT)’.
MacKay, D. J. (2003), Information theory, inference and learning algorithms ,
Cambridge university press.
Mallat, S. & Zhang, Z. (1993), ‘Matching pursuits with time-frequency dictionaries’,
IEEE Transactions on Signal Processing 41, 3397–3415.

McAllester, D. A. (1998), Some PAC-Bayesian theorems, in‘Conference on Learning
McAllester, D. A. (1999), PAC-Bayesian model averaging, in‘Conference on Learning
Theory (COLT)’, pp. 164–170.
McAllester, D. A. (2003), Simpliﬁed PAC-Bayesian margin bounds., in‘Conference on
Learning Theory (COLT)’, pp. 203–215.
Minsky, M. & Papert, S. (1969), Perceptrons: An Introduction to Computational Ge-
ometry , The MIT Press.
Mukherjee, S., Niyogi, P., Poggio, T. & Rifkin, R. (2006), ‘Learning theory: stability is

suﬃcient for generalization and necessary and suﬃcient for consistency of empirical
risk minimization’, Advances in Computational Mathematics 25(1-3), 161–193.
Murata, N. (1998), ‘A statistical study of on-line learning’, Online Learning and Neural
Networks. Cambridge University Press, Cambridge, UK .
Murphy, K. P. (2012), Machine learning: a probabilistic perspective , The MIT Press.
Natarajan, B. (1995), ‘Sparse approximate solutions to linear systems’, SIAM J. Com-
puting 25(2), 227–234.
Natarajan, B. K. (1989), ‘On learning sets and functions’, Mach. Learn. 4, 67–97.

Nemirovski, A., Juditsky, A., Lan, G. & Shapiro, A. (2009), ‘Robust stochastic ap-
proximation approach to stochastic programming’, SIAM Journal on Optimization
19(4), 1574–1609.
Nemirovski, A. & Yudin, D. (1978), Problem complexity and method eﬃciency in opti-
mization , Nauka Publishers, Moscow.
Nesterov, Y. (2005), Primal-dual subgradient methods for convex problems, Technical
report, Center for Operations Research and Econometrics (CORE), Catholic Univer-
sity of Louvain (UCL).

References 443
Nesterov, Y. & Nesterov, I. (2004), Introductory lectures on convex optimization: A
basic course , Vol. 87, Springer Netherlands.
Novikoﬀ, A. B. J. (1962), On convergence proofs on perceptrons, in‘Proceedings of the
Symposium on the Mathematical Theory of Automata’, Vol. XII, pp. 615–622.
Parberry, I. (1994), Circuit complexity and neural networks , The MIT press.
Pearson, K. (1901), ‘On lines and planes of closest ﬁt to systems of points in space’,
The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science
2(11), 559–572.
Phillips, D. L. (1962), ‘A technique for the numerical solution of certain integral equa-
tions of the ﬁrst kind’, Journal of the ACM 9(1), 84–97.
Pisier, G. (1980-1981), ‘Remarques sur un r´ esultat non publi´ e de B. maurey’.
Pitt, L. & Valiant, L. (1988), ‘Computational limitations on learning from examples’,
Journal of the Association for Computing Machinery 35(4), 965–984.

Poon, H. & Domingos, P. (2011), Sum-product networks: A new deep architecture, in
‘Conference on Uncertainty in Artiﬁcial Intelligence (UAI)’.
Quinlan, J. R. (1986), ‘Induction of decision trees’, Machine Learning 1, 81–106.
Quinlan, J. R. (1993), C4.5: Programs for Machine Learning , Morgan Kaufmann.
Rabiner, L. & Juang, B. (1986), ‘An introduction to hidden markov models’, IEEE
ASSP Magazine 3(1), 4–16.
Rakhlin, A., Shamir, O. & Sridharan, K. (2012), Making gradient descent optimal for

strongly convex stochastic optimization, in‘International Conference on Machine
Learning (ICML)’.
Rakhlin, A., Sridharan, K. & Tewari, A. (2010), Online learning: Random averages,
combinatorial parameters, and learnability, in‘NIPS’.
Rakhlin, S., Mukherjee, S. & Poggio, T. (2005), ‘Stability results in learning theory’,
Analysis and Applications 3(4), 397–419.
Ranzato, M., Huang, F., Boureau, Y. & Lecun, Y. (2007), Unsupervised learning of
invariant feature hierarchies with applications to object recognition, in‘Computer

Vision and Pattern Recognition, 2007. CVPR’07. IEEE Conference on’, IEEE, pp. 1–
8.
Rissanen, J. (1978), ‘Modeling by shortest data description’, Automatica 14, 465–471.
Rissanen, J. (1983), ‘A universal prior for integers and estimation by minimum descrip-
tion length’, The Annals of Statistics 11(2), 416–431.
Robbins, H. & Monro, S. (1951), ‘A stochastic approximation method’, The Annals of
Mathematical Statistics pp. 400–407.
Rogers, W. & Wagner, T. (1978), ‘A ﬁnite sample distribution-free performance bound

8.
for local discrimination rules’, The Annals of Statistics 6(3), 506–514.
Rokach, L. (2007), Data mining with decision trees: theory and applications , Vol. 69,
World scientiﬁc.
Rosenblatt, F. (1958), ‘The perceptron: A probabilistic model for information storage
and organization in the brain’, Psychological Review 65, 386–407. (Reprinted in
Neurocomputing (MIT Press, 1988).).
Rumelhart, D. E., Hinton, G. E. & Williams, R. J. (1986), Learning internal represen-
tations by error propagation, inD. E. Rumelhart & J. L. McClelland, eds, ‘Paral-

lel Distributed Processing – Explorations in the Microstructure of Cognition’, MIT
Press, chapter 8, pp. 318–362.

444 References
Sankaran, J. K. (1993), ‘A note on resolving infeasibility in linear programs by con-
straint relaxation’, Operations Research Letters 13(1), 19–20.
Sauer, N. (1972), ‘On the density of families of sets’, Journal of Combinatorial Theory
Series A 13, 145–147.
Schapire, R. (1990), ‘The strength of weak learnability’, Machine Learning 5(2), 197–
227.
Schapire, R. E. & Freund, Y. (2012), Boosting: Foundations and Algorithms , MIT press.
Sch¨ olkopf, B., Herbrich, R. & Smola, A. (2001), A generalized representer theorem, in
‘Computational learning theory’, pp. 416–426.
Sch¨ olkopf, B., Herbrich, R., Smola, A. & Williamson, R. (2000), A generalized repre-
senter theorem, in‘NeuroCOLT’.
Sch¨ olkopf, B. & Smola, A. J. (2002), Learning with Kernels: Support Vector Machines,
Regularization, Optimization and Beyond , MIT Press.
Sch¨ olkopf, B., Smola, A. & M¨ uller, K.-R. (1998), ‘Nonlinear component analysis as a
kernel eigenvalue problem’, Neural computation 10(5), 1299–1319.

Seeger, M. (2003), ‘Pac-bayesian generalisation error bounds for gaussian process clas-
siﬁcation’, The Journal of Machine Learning Research 3, 233–269.
Shakhnarovich, G., Darrell, T. & Indyk, P. (2006), Nearest-neighbor methods in learning
and vision: theory and practice , MIT Press.
Shalev-Shwartz, S. (2007), Online Learning: Theory, Algorithms, and Applications,
PhD thesis, The Hebrew University.
Shalev-Shwartz, S. (2011), ‘Online learning and online convex optimization’, Founda-
tions and Trends R⃝in Machine Learning 4(2), 107–194.

Shalev-Shwartz, S., Shamir, O., Srebro, N. & Sridharan, K. (2010), ‘Learnability,
stability and uniform convergence’, The Journal of Machine Learning Research
9999 , 2635–2670.
Shalev-Shwartz, S., Shamir, O. & Sridharan, K. (2010), Learning kernel-based halfs-
paces with the zero-one loss, in‘Conference on Learning Theory (COLT)’.
Shalev-Shwartz, S., Shamir, O., Sridharan, K. & Srebro, N. (2009), Stochastic convex
optimization, in‘Conference on Learning Theory (COLT)’.
Shalev-Shwartz, S. & Singer, Y. (2008), On the equivalence of weak learnability and

linear separability: New relaxations and eﬃcient boosting algorithms, in‘Proceedings
of the Nineteenth Annual Conference on Computational Learning Theory’.
Shalev-Shwartz, S., Singer, Y. & Srebro, N. (2007), Pegasos: Primal Estimated sub-
GrAdient SOlver for SVM, in‘International Conference on Machine Learning’,
pp. 807–814.
Shalev-Shwartz, S. & Srebro, N. (2008), SVM optimization: Inverse dependence on
training set size, in‘International Conference on Machine Learning’, pp. 928–935.

Shalev-Shwartz, S., Zhang, T. & Srebro, N. (2010), ‘Trading accuracy for sparsity
in optimization problems with sparsity constraints’, Siam Journal on Optimization
20, 2807–2832.
Shamir, O. & Zhang, T. (2013), Stochastic gradient descent for non-smooth optimiza-
tion: Convergence results and optimal averaging schemes, in‘International Confer-
ence on Machine Learning (ICML)’.
Shapiro, A., Dentcheva, D. & Ruszczy´ nski, A. (2009), Lectures on stochastic program-

ming: modeling and theory , Vol. 9, Society for Industrial and Applied Mathematics.

References 445
Shelah, S. (1972), ‘A combinatorial problem; stability and order for models and theories
in inﬁnitary languages’, Pac. J. Math 4, 247–261.
Sipser, M. (2006), Introduction to the Theory of Computation , Thomson Course Tech-
nology.
Slud, E. V. (1977), ‘Distribution inequalities for the binomial law’, The Annals of
Probability 5(3), 404–412.
Steinwart, I. & Christmann, A. (2008), Support vector machines , Springerverlag New
York.
Stone, C. (1977), ‘Consistent nonparametric regression’, The annals of statistics
5(4), 595–620.
Taskar, B., Guestrin, C. & Koller, D. (2003), Max-margin markov networks, in‘NIPS’.
Tibshirani, R. (1996), ‘Regression shrinkage and selection via the lasso’, J. Royal.
Statist. Soc B. 58(1), 267–288.
Tikhonov, A. N. (1943), ‘On the stability of inverse problems’, Dolk. Akad. Nauk SSSR
39(5), 195–198.
Tishby, N., Pereira, F. & Bialek, W. (1999), The information bottleneck method, in
‘The 37’th Allerton Conference on Communication, Control, and Computing’.

Tsochantaridis, I., Hofmann, T., Joachims, T. & Altun, Y. (2004), Support vector
machine learning for interdependent and structured output spaces, in‘Proceedings
of the Twenty-First International Conference on Machine Learning’.
Valiant, L. G. (1984), ‘A theory of the learnable’, Communications of the ACM
27(11), 1134–1142.
Vapnik, V. (1992), Principles of risk minimization for learning theory, inJ. E. Moody,
S. J. Hanson & R. P. Lippmann, eds, ‘Advances in Neural Information Processing
Systems 4’, Morgan Kaufmann, pp. 831–838.

Vapnik, V. (1995), The Nature of Statistical Learning Theory , Springer.
Vapnik, V. N. (1982), Estimation of Dependences Based on Empirical Data , Springer-
Verlag.
Vapnik, V. N. (1998), Statistical Learning Theory , Wiley.
Vapnik, V. N. & Chervonenkis, A. Y. (1971), ‘On the uniform convergence of relative
frequencies of events to their probabilities’, Theory of Probability and its applications
XVI (2), 264–280.
Vapnik, V. N. & Chervonenkis, A. Y. (1974), Theory of pattern recognition , Nauka,
Moscow. (In Russian).

Verlag.
Von Luxburg, U. (2007), ‘A tutorial on spectral clustering’, Statistics and computing
17(4), 395–416.
von Neumann, J. (1928), ‘Zur theorie der gesellschaftsspiele (on the theory of parlor
games)’, Math. Ann. 100, 295—320.
Von Neumann, J. (1953), ‘A certain zero-sum two-person game equivalent to the opti-
mal assignment problem’, Contributions to the Theory of Games 2, 5–12.
Vovk, V. G. (1990), Aggregating strategies, in‘Conference on Learning Theory
(COLT)’, pp. 371–383.
Warmuth, M., Glocer, K. & Vishwanathan, S. (2008), Entropy regularized lpboost, in

‘Algorithmic Learning Theory (ALT)’.
Warmuth, M., Liao, J. & Ratsch, G. (2006), Totally corrective boosting algorithms
that maximize the margin, in‘Proceedings of the 23rd international conference on
Machine learning’.

446 References
Weston, J., Chapelle, O., Vapnik, V., Elisseeﬀ, A. & Sch¨ olkopf, B. (2002), Kernel depen-
dency estimation, in‘Advances in neural information processing systems’, pp. 873–
880.
Weston, J. & Watkins, C. (1999), Support vector machines for multi-class pattern
recognition, in‘Proceedings of the Seventh European Symposium on Artiﬁcial Neural
Networks’.
Wolpert, D. H. & Macready, W. G. (1997), ‘No free lunch theorems for optimization’,
Evolutionary Computation, IEEE Transactions on 1(1), 67–82.
Zhang, T. (2004), Solving large scale linear prediction problems using stochastic gradi-
ent descent algorithms, in‘Proceedings of the Twenty-First International Conference
on Machine Learning’.
Zhao, P. & Yu, B. (2006), ‘On model selection consistency of Lasso’, Journal of Machine
Learning Research 7, 2541–2567.
Zinkevich, M. (2003), Online convex programming and generalized inﬁnitesimal gradi-
ent ascent, in‘International Conference on Machine Learning’.

Index
3-term DNF, 107
F1-score, 244
ℓ1norm, 183, 332, 363, 386
accuracy, 38, 43
activation function, 269
AdaBoost, 130, 134, 362
all-pairs, 228, 404
approximation error, 61, 64
auto-encoders, 368
backpropagation, 278
backward elimination, 363
bag-of-words, 209
base hypothesis, 137
Bayes optimal, 46, 52, 260
Bayes rule, 354
Bayesian reasoning, 353
Bennet’s inequality, 426
Bernstein’s inequality, 426
bias, 37, 61, 64
bias-complexity tradeoﬀ, 65
boolean conjunctions, 51, 79, 106
boosting, 130
boosting the conﬁdence, 142
boundedness, 165
C4.5, 254
CART, 254
chaining, 389
Chebyshev’s inequality, 423
Chernoﬀ bounds, 423
class-sensitive feature mapping, 230
classiﬁer, 34
clustering, 307
spectral, 315
compressed sensing, 330
compression bounds, 410
compression scheme, 411
computational complexity, 100
conﬁdence, 38, 43
consistency, 92
Consistent, 289
contraction lemma, 381
convex, 156
function, 157set, 156
strongly convex, 174, 195
convex-Lipschitz-bounded learning, 166
convex-smooth-bounded learning, 166

C4.5, 254
CART, 254
covering numbers, 388
curse of dimensionality, 263
decision stumps, 132, 133
decision trees, 250
dendrogram, 309, 310
dictionary learning, 368
diﬀerential set, 188
dimensionality reduction, 323
discretization trick, 57
discriminative, 342
distribution free, 342
domain, 33
domain of examples, 48
doubly stochastic matrix, 242
duality, 211
strong duality, 211
weak duality, 211
Dudley classes, 81
eﬃcient computable, 100
EM, 348
empirical error, 35
empirical risk, 35, 48
Empirical Risk Minimization, seeERM
entropy, 345

domain, 33
EM, 348
relative entropy, 345
epigraph, 157
ERM, 35
error decomposition, 64, 168
estimation error, 61, 64
Expectation-Maximization, seeEM
face recognition, seeViola-Jones
feasible, 100
feature, 33
feature learning, 368
feature normalization, 365
feature selection, 357, 358
feature space, 215
feature transformations, 367
ﬁlters, 359

ERM, 35

448 Index
forward greedy selection, 360
frequentist, 353
gain, 253
GD, seegradient descent
generalization error, 35
generative models, 342
Gini index, 254
Glivenko-Cantelli, 58
gradient, 158
gradient descent, 185
Gram matrix, 219
growth function, 73
halfspace, 118
homogenous, 118, 205
non-separable, 119
separable, 118
Halving, 289
hidden layers, 270
Hilbert space, 217
Hoeﬀding’s inequality, 56, 425
hold out, 146
hypothesis, 34
hypothesis class, 36
i.i.d., 38
ID3, 252
improper, seerepresentation independent
inductive bias, seebias
information bottleneck, 317
information gain, 254
instance, 33
instance space, 33
integral image, 143
Johnson-Lindenstrauss lemma, 329
k-means, 311, 313
soft k-means, 352
k-median, 312
k-medoids, 312
Kendall tau, 239
kernel PCA, 326
kernels, 215
Gaussian kernel, 220
kernel trick, 217
polynomial kernel, 220
RBF kernel, 220
label, 33
Lasso, 365, 386
generalization bounds, 386
latent variables, 348
LDA, 347
Ldim, 290, 291
learning curves, 153
least squares, 124
likelihood ratio, 348

label, 33
LDA, 347
linear discriminant analysis, seeLDA
linear predictor, 117homogenous, 118
linear programming, 119
linear regression, 122
linkage, 310
Lipschitzness, 160, 176, 191
sub-gradient, 190
Littlestone dimension, seeLdim
local minimum, 158
logistic regression, 126
loss, 35
loss function, 48
0-1 loss, 48, 167
absolute value loss, 124, 128, 166
convex loss, 163
generalized hinge-loss, 233
hinge loss, 167
Lipschitz loss, 166
log-loss, 345
logistic loss, 127
ramp loss, 209
smooth loss, 166
square loss, 48

loss, 35
surrogate loss, 167, 302
margin, 203
Markov’s inequality, 422
Massart lemma, 380
max linkage, 310
maximum a-posteriori, 355
maximum likelihood, 343
McDiarmid’s inequality, 378
MDL, 89, 90, 251
measure concentration, 55, 422
Minimum Description Length, seeMDL
mistake bound, 288
mixture of Gaussians, 348
model selection, 144, 147
multiclass, 47, 227, 402
cost-sensitive, 232
linear predictors, 230, 405
multi-vector, 231, 406
Perceptron, 248
reductions, 227, 405
SGD, 235
SVM, 234
multivariate performance measures, 243

SGD, 235
SVM, 234
Naive Bayes, 347
Natarajan dimension, 402
NDCG, 239
Nearest Neighbor, 258
k-NN, 258
neural networks, 268
feedforward networks, 269
layered networks, 269
SGD, 277
no-free-lunch, 61
non-uniform learning, 84

Index 449
Normalized Discounted Cumulative Gain,
seeNDCG
Occam’s razor, 91
OMP, 360
one-vs-all, 227
one-vs-rest, seeone-vs-all
one-vs.-all, 404
online convex optimization, 300
online gradient descent, 300
online learning, 287
optimization error, 168
oracle inequality, 179
orthogonal matching pursuit, seeOMP
overﬁtting, 35, 65, 152
PAC, 43
agnostic PAC, 45, 46
agnostic PAC for general loss, 49
PAC-Bayes, 415
parametric density estimation, 342
PCA, 324
Pearson’s correlation coeﬃcient, 359
Perceptron, 120
kernelized Perceptron, 225
multiclass, 248
online, 301
permutation matrix, 242
polynomial regression, 125
precision, 244
predictor, 34
preﬁx free language, 89
Principal Component Analysis, seePCA
prior knowledge, 63
Probably Approximately Correct, seePAC
projection, 193
projection lemma, 193
proper, 49
pruning, 254
Rademacher complexity, 375
random forests, 255
random projections, 329
ranking, 238
bipartite, 243
realizability, 37
recall, 244
regression, 47, 122, 172
regularization, 171
Tikhonov, 172, 174

proper, 49
regularized loss minimization, seeRLM
representation independent, 49, 107
representative sample, 54, 375
representer theorem, 218
ridge regression, 172
kernel ridge regression, 225
RIP, 331
risk, 35, 45, 48
RLM, 171, 199sample complexity, 44
Sauer’s lemma, 73
self-boundedness, 162
sensitivity, 244
SGD, 190
shattering, 69, 403
single linkage, 310
Singular Value Decomposition, seeSVD
Slud’s inequality, 428
smoothness, 162, 177, 198
SOA, 292
sparsity-inducing norms, 363
speciﬁcity, 244
spectral clustering, 315

RIP, 331
SGD, 190
SOA, 292
SRM, 85, 145
stability, 173
Stochastic Gradient Descent, seeSGD
strong learning, 132
Structural Risk Minimization, seeSRM
structured output prediction, 236
sub-gradient, 188
Support Vector Machines, seeSVM
SVD, 431
SVM, 202, 383
generalization bounds, 208, 383
hard-SVM, 203, 204
homogenous, 205
soft-SVM, 206
support vectors, 210
target set, 47
term-frequency, 231
TF-IDF, 231
training error, 35
training set, 33
true error, 35, 45
underﬁtting, 65, 152
uniform convergence, 54, 55
union bound, 39

SVD, 431
unsupervised learning, 308
validation, 144, 146
cross validation, 149
train-validation-test split, 150
Vapnik-Chervonenkis dimension, seeVC
dimension
VC dimension, 67, 70
version space, 289
Viola-Jones, 139
weak learning, 130, 131
Weighted-Majority, 295

Preface This book provides a working guide to the Open Source Computer Vision Library (OpenCV) and also provides a general background to the field of computer vision sufficient to use OpenCV effectively. Purpose Computer vision is a rapidly growing field, partly as a result of both cheaper and more capable cameras, partly because of affordable processing power, and partly because vision algorithms are starting to mature. OpenCV itself has played a role in the growth of computer vision by enabling thousands of people to do more productive work in vision. With its focus on real-time vision, OpenCV helps students and professionals efficiently implement projects and jump-start research by providing them with a computer vision and machine learning infrastructure that was previously available only in a few mature research labs. The purpose of this text is to: • Better document OpenCV—detail what function calling conventions really mean and how to use them correctly. • Rapidly give the reader an intuitive

data/pdfs/Learning OpenCV-Computer Vision in C++ with the OpenCV Library.pdf

thousands of people to do more productive work in vision. With its focus on real-time vision, OpenCV helps students and professionals efficiently implement projects and jump-start research by providing them with a computer vision and machine learning infrastructure that was previously available only in a few mature research labs. The purpose of this text is to: • Better document OpenCV—detail what function calling conventions really mean and how to use them correctly. • Rapidly give the reader an intuitive understanding of how the vision algorithms work. • Give the reader some sense of what algorithm to use and when to use it. • Give the reader a boost in implementing computer vision and machine learning algorithms by providing many working coded examples to start from. • Provide intuitions about how to fix some of the more advanced routines when something goes wrong. Simply put, this is the text the authors wished we had in school and the coding reference book we wished we had at work. This book documents a

understanding of how the vision algorithms work. • Give the reader some sense of what algorithm to use and when to use it. • Give the reader a boost in implementing computer vision and machine learning algorithms by providing many working coded examples to start from. • Provide intuitions about how to fix some of the more advanced routines when something goes wrong. Simply put, this is the text the authors wished we had in school and the coding reference book we wished we had at work. This book documents a tool kit, OpenCV, that allows the reader to do interesting and fun things rapidly in computer vision. It gives an intuitive understanding as to how the algorithms work, which serves to guide the reader in designing and debugging vision applications and also to make the formal descriptions of computer vision and machine learning algorithms in other texts easier to comprehend and remember. After all, it is easier to understand complex algorithms and their associated math when you start with an intuitive

a tool kit, OpenCV, that allows the reader to do interesting and fun things rapidly in computer vision. It gives an intuitive understanding as to how the algorithms work, which serves to guide the reader in designing and debugging vision applications and also to make the formal descriptions of computer vision and machine learning algorithms in other texts easier to comprehend and remember. After all, it is easier to understand complex algorithms and their associated math when you start with an intuitive grasp of how those algorithms work. Who This Book Is For This book contains descriptions, working coded examples, and explanations of the computer vision tools contained in the OpenCV library. As such, it should be helpful to many different kinds of users.

Professionals For those practicing professionals who need to rapidly implement computer vision systems, the sample code provides a quick framework with which to start. Our descriptions of the intuitions behind the algorithms can quickly teach or remind the reader how they work. Students As we said, this is the text we wish had back in school. The intuitive explanations, detailed documentation, and sample code will allow you to boot up faster in computer vision, work on more interesting class projects, and ultimately contribute new research to the field. Teachers Computer vision is a fast-moving field. We’ve found it effective to have the students rapidly cover an accessible text while the instructor fills in formal exposition where needed and supplements with current papers or guest lectures from experts. The students can meanwhile start class projects earlier and attempt more ambitious tasks. Hobbyists Computer vision is fun, here’s how to hack it. We have a strong focus on giving readers enough intuition,

ultimately contribute new research to the field. Teachers Computer vision is a fast-moving field. We’ve found it effective to have the students rapidly cover an accessible text while the instructor fills in formal exposition where needed and supplements with current papers or guest lectures from experts. The students can meanwhile start class projects earlier and attempt more ambitious tasks. Hobbyists Computer vision is fun, here’s how to hack it. We have a strong focus on giving readers enough intuition, documentation, and working code to enable rapid implementation of real-time vision applications. What This Book Is Not This book is not a formal text. We do go into mathematical detail at various points,1 but it is all in the service of developing deeper intuitions behind the algorithms or to clarify the implications of any assumptions built into those algorithms. We have not attempted a formal mathematical exposition here and might even incur some wrath along the way from those who do write formal

documentation, and working code to enable rapid implementation of real-time vision applications. What This Book Is Not This book is not a formal text. We do go into mathematical detail at various points,1 but it is all in the service of developing deeper intuitions behind the algorithms or to clarify the implications of any assumptions built into those algorithms. We have not attempted a formal mathematical exposition here and might even incur some wrath along the way from those who do write formal expositions. This book is not for theoreticians because it has more of an “applied” nature. The book will certainly be of general help, but is not aimed at any of the specialized niches in computer vision (e.g., medical imaging or remote sensing analysis). That said, it is the belief of the authors that having read the explanations here first, a student will not only learn the theory better but remember it longer. Therefore, this book would make a good adjunct text to a theoretical course and would be a great text

This book is not for theoreticians because it has more of an “applied” nature. The book will certainly be of general help, but is not aimed at any of the specialized niches in computer vision (e.g., medical imaging or remote sensing analysis). That said, it is the belief of the authors that having read the explanations here first, a student will not only learn the theory better but remember it longer. Therefore, this book would make a good adjunct text to a theoretical course and would be a great text for an introductory or project-centric course. About the Programs in This Book All the program examples in this book are based on OpenCV version 2.5. The code should definitely work under Linux or Windows and probably under OS-X, too. Source code for the examples in the book can be fetched from this book’s website (http://www.oreilly.com/catalog/9780596516130). OpenCV can be loaded from its source forge site (http://sourceforge.net/projects/opencvlibrary). OpenCV is under ongoing development, with official

for an introductory or project-centric course. About the Programs in This Book All the program examples in this book are based on OpenCV version 2.5. The code should definitely work under Linux or Windows and probably under OS-X, too. Source code for the examples in the book can be fetched from this book’s website (http://www.oreilly.com/catalog/9780596516130). OpenCV can be loaded from its source forge site (http://sourceforge.net/projects/opencvlibrary). OpenCV is under ongoing development, with official releases occurring once or twice a year. To keep up to date with the developments of the library, and for pointers to where to get the very latest updates and versions, you can visit OpenCV.org, the library’s official website. Prerequisites For the most part, readers need only know how to program in C and perhaps some C++. Many of the math sections are optional and are labeled as such. The mathematics involves simple algebra and basic matrix                                                              1

releases occurring once or twice a year. To keep up to date with the developments of the library, and for pointers to where to get the very latest updates and versions, you can visit OpenCV.org, the library’s official website. Prerequisites For the most part, readers need only know how to program in C and perhaps some C++. Many of the math sections are optional and are labeled as such. The mathematics involves simple algebra and basic matrix                                                              1 Always with a warning to more casual users that they may skip such sections.

algebra, and it assumes some familiarity with solution methods to least-squares optimization problems as well as some basic knowledge of Gaussian distributions, Bayes’ law, and derivatives of simple functions. The math is in support of developing intuition for the algorithms. The reader may skip the math and the algorithm descriptions, using only the function definitions and code examples to get vision applications up and running. How This Book Is Best Used This text need not be read in order. It can serve as a kind of user manual: look up the function when you need it; read the function’s description if you want the gist of how it works “under the hood”. The intent of this book is more tutorial, however. It gives you a basic understanding of computer vision along with details of how and when to use selected algorithms. This book was written to allow its use as an adjunct or as a primary textbook for an undergraduate or graduate course in computer vision. The basic strategy with this method is for students to

a kind of user manual: look up the function when you need it; read the function’s description if you want the gist of how it works “under the hood”. The intent of this book is more tutorial, however. It gives you a basic understanding of computer vision along with details of how and when to use selected algorithms. This book was written to allow its use as an adjunct or as a primary textbook for an undergraduate or graduate course in computer vision. The basic strategy with this method is for students to read the book for a rapid overview and then supplement that reading with more formal sections in other textbooks and with papers in the field. There are exercises at the end of each chapter to help test the student’s knowledge and to develop further intuitions. You could approach this text in any of the following ways. Grab Bag Go through Chapter 1–Chapter 3 in the first sitting, then just hit the appropriate chapters or sections as you need them. This book does not have to be read in sequence, except for

read the book for a rapid overview and then supplement that reading with more formal sections in other textbooks and with papers in the field. There are exercises at the end of each chapter to help test the student’s knowledge and to develop further intuitions. You could approach this text in any of the following ways. Grab Bag Go through Chapter 1–Chapter 3 in the first sitting, then just hit the appropriate chapters or sections as you need them. This book does not have to be read in sequence, except for Chapter 11 and Chapter 12 (Calibration and Stereo). Good Progress Read just two chapters a week until you’ve covered Chapter 1–Chapter 12 in six weeks (Chapter 13 is a special case, as discussed shortly). Start on projects and dive into details on selected areas in the field, using additional texts and papers as appropriate. The Sprint Just cruise through the book as fast as your comprehension allows, covering Chapter 1–Chapter 12. Then get started on projects and go into details on selected areas in the

Chapter 11 and Chapter 12 (Calibration and Stereo). Good Progress Read just two chapters a week until you’ve covered Chapter 1–Chapter 12 in six weeks (Chapter 13 is a special case, as discussed shortly). Start on projects and dive into details on selected areas in the field, using additional texts and papers as appropriate. The Sprint Just cruise through the book as fast as your comprehension allows, covering Chapter 1–Chapter 12. Then get started on projects and go into details on selected areas in the field using additional texts and papers. This is probably the choice for professionals, but it might also suit a more advanced computer vision course. Chapter 13 is a long chapter that gives a general background to machine learning in addition to details behind the machine learning algorithms implemented in OpenCV and how to use them. Of course, machine learning is integral to object recognition and a big part of computer vision, but it’s a field worthy of its own book. Professionals should find this text a

using additional texts and papers. This is probably the choice for professionals, but it might also suit a more advanced computer vision course. Chapter 13 is a long chapter that gives a general background to machine learning in addition to details behind the machine learning algorithms implemented in OpenCV and how to use them. Of course, machine learning is integral to object recognition and a big part of computer vision, but it’s a field worthy of its own book. Professionals should find this text a suitable launching point for further explorations of the literature—or for just getting down to business with the code in that part of the library. This chapter should probably be considered optional for a typical computer vision class. This is how the authors like to teach computer vision: Sprint through the course content at a level where the students get the gist of how things work; then get students started on meaningful class projects while the instructor supplies depth and formal rigor in selected areas

launching point for further explorations of the literature—or for just getting down to business with the code in that part of the library. This chapter should probably be considered optional for a typical computer vision class. This is how the authors like to teach computer vision: Sprint through the course content at a level where the students get the gist of how things work; then get students started on meaningful class projects while the instructor supplies depth and formal rigor in selected areas by drawing from other texts or papers in the field. This same method works for quarter, semester, or two-term classes. Students can get quickly up and running with a general understanding of their vision task and working code to match. As they begin more challenging and time-consuming projects, the instructor helps them develop and debug complex systems. For longer courses, the projects themselves can become instructional in terms of project management. Build up working systems first; refine them with more

by drawing from other texts or papers in the field. This same method works for quarter, semester, or two-term classes. Students can get quickly up and running with a general understanding of their vision task and working code to match. As they begin more challenging and time-consuming projects, the instructor helps them develop and debug complex systems. For longer courses, the projects themselves can become instructional in terms of project management. Build up working systems first; refine them with more knowledge, detail, and research later. The goal in such courses is for each project to aim at being worthy of a conference publication and with a few project papers being published subsequent to further (postcourse) work.

Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, file extensions, path names, directories, and Unix utilities. Constant width Indicates commands, options, switches, variables, attributes, keys, functions, types, classes, namespaces, methods, modules, properties, parameters, values, objects, events, event handlers, XMLtags, HTMLtags, the contents of files, or the output from commands. Constant width bold Shows commands or other text that could be typed literally by the user. Also used for emphasis in code samples. Constant width italic Shows text that should be replaced with user-supplied values. […] Indicates a reference to the bibliography. The standard bibliographic form we adopt in this book is the use of the last name of the first author of a paper, followed by a two digit representation of the year of publication. Thus the paper “Self-supervised monocular road detection in desert terrain,” authored

Shows commands or other text that could be typed literally by the user. Also used for emphasis in code samples. Constant width italic Shows text that should be replaced with user-supplied values. […] Indicates a reference to the bibliography. The standard bibliographic form we adopt in this book is the use of the last name of the first author of a paper, followed by a two digit representation of the year of publication. Thus the paper “Self-supervised monocular road detection in desert terrain,” authored by  “H. Dahlkamp, A. Kaehler, D. Stavens, S. Thrun, and G. Bradski” in 2006, would be cited as: “[Dahlkamp06]”. This icon signifies a tip, suggestion, or general note. This icon indicates a warning or caution. Using Code Examples OpenCV is free for commercial or research use, and we have the same policy on the code examples in the book. Use them at will for homework, for research, or for commercial products. We would very much appreciate referencing this book when you do, but it is not required. Other than

by  “H. Dahlkamp, A. Kaehler, D. Stavens, S. Thrun, and G. Bradski” in 2006, would be cited as: “[Dahlkamp06]”. This icon signifies a tip, suggestion, or general note. This icon indicates a warning or caution. Using Code Examples OpenCV is free for commercial or research use, and we have the same policy on the code examples in the book. Use them at will for homework, for research, or for commercial products. We would very much appreciate referencing this book when you do, but it is not required. Other than how it helped with your homework projects (which is best kept a secret), we would like to hear how you are using computer vision for academic research, teaching courses, and in commercial products when you do use OpenCV to help you. Again, not required, but you are always invited to drop us a line. Safari® Books Online When you see a Safari® Books Online icon on the cover of your favorite technology book, that means the book is available online through the O’Reilly Network Safari Bookshelf. Safari offers a

how it helped with your homework projects (which is best kept a secret), we would like to hear how you are using computer vision for academic research, teaching courses, and in commercial products when you do use OpenCV to help you. Again, not required, but you are always invited to drop us a line. Safari® Books Online When you see a Safari® Books Online icon on the cover of your favorite technology book, that means the book is available online through the O’Reilly Network Safari Bookshelf. Safari offers a solution that’s better than e-books. It’s virtual library that lets you easily search thousands of top tech books, cut and paste code samples, download chapters, and find quick answers when you need the most accurate, current information. Try it for free at http://safari.oreilly.com. We’d Like to Hear from You Please address comments and questions concerning this book to the publisher:

O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for this book, where we list examples and any plans for future editions. You can access this information at: http://www.oreilly.com/catalog/9780596516130/ You can also send messages electronically. To be put on the mailing list or request a catalog, send an email to: info@oreilly.com To comment on the book, send an email to: bookquestions@oreilly.com For more information about our books, conferences, Resource Centers, and the O’Reilly Network, see our website at: http://www.oreilly.com Acknowledgments A long-term open source effort sees many people come and go, each contributing in different ways. The list of contributors to this library is far too long to list here, but see the …/opencv/docs/HTML/Contributors/doc_contributors.html file that ships with OpenCV. Thanks for Help on OpenCV Intel is where the library was

email to: bookquestions@oreilly.com For more information about our books, conferences, Resource Centers, and the O’Reilly Network, see our website at: http://www.oreilly.com Acknowledgments A long-term open source effort sees many people come and go, each contributing in different ways. The list of contributors to this library is far too long to list here, but see the …/opencv/docs/HTML/Contributors/doc_contributors.html file that ships with OpenCV. Thanks for Help on OpenCV Intel is where the library was born and deserves great thanks for supporting this project the whole way through. Open source needs a champion and enough development support in the beginning to achieve critical mass. Intel gave it both. There are not many other companies where one could have started and maintained such a project through good times and bad. Along the way, OpenCV helped give rise to—and now takes (optional) advantage of—Intel’s Integrated Performance Primitives, which are hand-tuned assembly language routines in vision,

born and deserves great thanks for supporting this project the whole way through. Open source needs a champion and enough development support in the beginning to achieve critical mass. Intel gave it both. There are not many other companies where one could have started and maintained such a project through good times and bad. Along the way, OpenCV helped give rise to—and now takes (optional) advantage of—Intel’s Integrated Performance Primitives, which are hand-tuned assembly language routines in vision, signal processing, speech, linear algebra, and more. Thus the lives of a great commercial product and an open source product are intertwined. Mark Holler, a research manager at Intel, allowed OpenCV to get started by knowingly turning a blind eye to the inordinate amount of time being spent on an unofficial project back in the library’s earliest days. As divine reward, he now grows wine up in Napa’s Mt. Veeder area. Stuart Taylor in the Performance Libraries group at Intel enabled OpenCV by letting us

signal processing, speech, linear algebra, and more. Thus the lives of a great commercial product and an open source product are intertwined. Mark Holler, a research manager at Intel, allowed OpenCV to get started by knowingly turning a blind eye to the inordinate amount of time being spent on an unofficial project back in the library’s earliest days. As divine reward, he now grows wine up in Napa’s Mt. Veeder area. Stuart Taylor in the Performance Libraries group at Intel enabled OpenCV by letting us “borrow” part of his Russian software team. Richard Wirt was key to its continued growth and survival. As the first author took on management responsibility at Intel, lab director Bob Liang let OpenCV thrive; when Justin Rattner became CTO, we were able to put OpenCV on a more firm foundation under Software Technology Lab—supported by software guru Shinn-Horng Lee and indirectly under his manager, Paul Wiley. Omid Moghadam helped advertise OpenCV in the early days. Mohammad Haghighat and Bill Butera were great

part of his Russian software team. Richard Wirt was key to its continued growth and survival. As the first author took on management responsibility at Intel, lab director Bob Liang let OpenCV thrive; when Justin Rattner became CTO, we were able to put OpenCV on a more firm foundation under Software Technology Lab—supported by software guru Shinn-Horng Lee and indirectly under his manager, Paul Wiley. Omid Moghadam helped advertise OpenCV in the early days. Mohammad Haghighat and Bill Butera were great as technical sounding boards. Nuriel Amir, Denver Dash, John Mark Agosta, and Marzia Polito were of key assistance in launching the machine learning library. Rainer Lienhart, Jean-Yves Bouguet, Radek Grzeszczuk, and Ara Nefian were able technical contributors to OpenCV and great colleagues along the way; the first is now a professor, the second is now making use of OpenCV in some well-known Google projects, and the others are staffing research labs and start-ups. There were many other technical contributors too

sounding boards. Nuriel Amir, Denver Dash, John Mark Agosta, and Marzia Polito were of key assistance in launching the machine learning library. Rainer Lienhart, Jean-Yves Bouguet, Radek Grzeszczuk, and Ara Nefian were able technical contributors to OpenCV and great colleagues along the way; the first is now a professor, the second is now making use of OpenCV in some well-known Google projects, and the others are staffing research labs and start-ups. There were many other technical contributors too numerous to name.

On the software side, some individuals stand out for special mention, especially on the Russian software team. Chief among these is the Russian lead programmer Vadim Pisarevsky, who developed large parts of the library and also managed and nurtured the library through the lean times when boom had turned to bust; he, if anyone, is the true hero of the library. His technical insights have also been of great help during the writing of this book. Giving him managerial support and protection in the lean years was Valery Kuriakin, a man of great talent and intellect. Victor Eruhimov was there in the beginning and stayed through most of it. We thank Boris Chudinovich for all of the contour components. Finally, very special thanks go to Willow Garage [WG], not only for its steady financial backing to OpenCV’s future development but also for supporting one author (and providing the other with snacks and beverages) during the final period of writing this book. Thanks for Help on the Book While preparing this book, we

Valery Kuriakin, a man of great talent and intellect. Victor Eruhimov was there in the beginning and stayed through most of it. We thank Boris Chudinovich for all of the contour components. Finally, very special thanks go to Willow Garage [WG], not only for its steady financial backing to OpenCV’s future development but also for supporting one author (and providing the other with snacks and beverages) during the final period of writing this book. Thanks for Help on the Book While preparing this book, we had several key people contributing advice, reviews, and suggestions. Thanks to John Markoff, Technology Reporter at the New York Times for encouragement, key contacts, and general writing advice born of years in the trenches. To our reviewers, a special thanks go to Evgeniy Bart, physics postdoc at CalTech, who made many helpful comments on every chapter; Kjerstin Williams at Applied Minds, who did detailed proofs and verification until the end; John Hsu at Willow Garage, who went through all the example

had several key people contributing advice, reviews, and suggestions. Thanks to John Markoff, Technology Reporter at the New York Times for encouragement, key contacts, and general writing advice born of years in the trenches. To our reviewers, a special thanks go to Evgeniy Bart, physics postdoc at CalTech, who made many helpful comments on every chapter; Kjerstin Williams at Applied Minds, who did detailed proofs and verification until the end; John Hsu at Willow Garage, who went through all the example code; and Vadim Pisarevsky, who read each chapter in detail, proofed the function calls and the code, and also provided several coding examples. There were many other partial reviewers. Jean-Yves Bouguet at Google was of great help in discussions on the calibration and stereo chapters. Professor Andrew Ng at Stanford University provided useful early critiques of the machine learning chapter. There were numerous other reviewers for various chapters—our thanks to all of them. Of course, any errors result from

and Vadim Pisarevsky, who read each chapter in detail, proofed the function calls and the code, and also provided several coding examples. There were many other partial reviewers. Jean-Yves Bouguet at Google was of great help in discussions on the calibration and stereo chapters. Professor Andrew Ng at Stanford University provided useful early critiques of the machine learning chapter. There were numerous other reviewers for various chapters—our thanks to all of them. Of course, any errors result from our own ignorance or misunderstanding, not from the advice we received. Finally, many thanks go to our editor, Michael Loukides, for his early support, numerous edits, and continued enthusiasm over the long haul.  Contributors: Vadim Pisarevsky Reviewers: Kari Pulli Kjerstin Williams Evgeniy Bart Professor Andrew Ng GPU Appendix: Khanh Yun-Ta Anatoly Baksheev Editors etc: Michael Loukides Rachel B. Steely Rachel Roumeliotis

Adrian Adds… Coming from a background in theoretical physics, the arc that brought me through supercomputer design and numerical computing on to machine learning and computer vision has been a long one. Along the way, many individuals stand out as key contributors. I have had many wonderful teachers, some formal instructors and others informal guides. I should single out Professor David Dorfan of UC Santa Cruz and Hartmut Sadrozinski of SLAC for their encouragement in the beginning, and Norman Christ for teaching me the fine art of computing with the simple edict that “if you cannot make the computer do it, you don’t know what you are talking about”. Special thanks go to James Guzzo, who let me spend time on this sort of thing at Intel—even though it was miles from what I was supposed to be doing—and who encouraged my participation in the Grand Challenge during those years. Finally, I want to thank Danny Hillis for creating the kind of place where all of this technology can make the leap to wizardry and for

me the fine art of computing with the simple edict that “if you cannot make the computer do it, you don’t know what you are talking about”. Special thanks go to James Guzzo, who let me spend time on this sort of thing at Intel—even though it was miles from what I was supposed to be doing—and who encouraged my participation in the Grand Challenge during those years. Finally, I want to thank Danny Hillis for creating the kind of place where all of this technology can make the leap to wizardry and for encouraging my work on the book while at Applied Minds. Such unique institutions are rare indeed in the world. I also would like to thank Stanford University for the extraordinary amount of support I have received from them over the years. From my work on the Grand Challenge team with Sebastian Thrun to the STAIR Robot with Andrew Ng, the Stanford AI Lab was always generous with office space, financial support, and most importantly ideas, enlightening conversation, and (when needed) simple instruction on so many

my work on the book while at Applied Minds. Such unique institutions are rare indeed in the world. I also would like to thank Stanford University for the extraordinary amount of support I have received from them over the years. From my work on the Grand Challenge team with Sebastian Thrun to the STAIR Robot with Andrew Ng, the Stanford AI Lab was always generous with office space, financial support, and most importantly ideas, enlightening conversation, and (when needed) simple instruction on so many aspects of vision, robotics, and machine learning. I have a deep gratitude to these people, who have contributed so significantly to my own growth and learning. No acknowledgment or thanks would be meaningful without a special thanks to my family, who never once faltered in their encouragement of this project or in their willingness to accompany me on trips up and down the state to work with Gary on this book. My thanks and my love go to them. Gary Adds… With three young kids at home, my wife Sonya put in more

of vision, robotics, and machine learning. I have a deep gratitude to these people, who have contributed so significantly to my own growth and learning. No acknowledgment or thanks would be meaningful without a special thanks to my family, who never once faltered in their encouragement of this project or in their willingness to accompany me on trips up and down the state to work with Gary on this book. My thanks and my love go to them. Gary Adds… With three young kids at home, my wife Sonya put in more work to enable this book than I did. Deep thanks and love—even OpenCV gives her recognition, as you can see in the face detection section example image. Further back, my technical beginnings started with the physics department at the University of Oregon followed by undergraduate years at UC Berkeley. For graduate school, I’d like to thank my advisor Steve Grossberg and Gail Carpenter at the Center for Adaptive Systems, Boston University, where I first cut my academic teeth. Though they focus on mathematical

to enable this book than I did. Deep thanks and love—even OpenCV gives her recognition, as you can see in the face detection section example image. Further back, my technical beginnings started with the physics department at the University of Oregon followed by undergraduate years at UC Berkeley. For graduate school, I’d like to thank my advisor Steve Grossberg and Gail Carpenter at the Center for Adaptive Systems, Boston University, where I first cut my academic teeth. Though they focus on mathematical modeling of the brain and I have ended up firmly on the engineering side of AI, I think the perspectives I developed there have made all the difference. Some of my former colleagues in graduate school are still close friends and gave advice, support, and even some editing of the book: thanks to Frank Guenther, Andrew Worth, Steve Lehar, Dan Cruthirds, Allen Gove, and Krishna Govindarajan. I specially thank Stanford University, where I’m currently a consulting professor in the AI and Robotics lab. Having close

of the brain and I have ended up firmly on the engineering side of AI, I think the perspectives I developed there have made all the difference. Some of my former colleagues in graduate school are still close friends and gave advice, support, and even some editing of the book: thanks to Frank Guenther, Andrew Worth, Steve Lehar, Dan Cruthirds, Allen Gove, and Krishna Govindarajan. I specially thank Stanford University, where I’m currently a consulting professor in the AI and Robotics lab. Having close contact with the best minds in the world definitely rubs off, and working with Sebastian Thrun and Mike Montemerlo to apply OpenCV on Stanley (the robot that won the $2M DARPA Grand Challenge) and with Andrew Ng on STAIR (one of the most advanced personal robots) was more technological fun than a person has a right to have. It’s a department that is currently hitting on all cylinders and simply a great environment to be in. In addition to Sebastian Thrun and Andrew Ng there, I thank Daphne Koller for setting

with the best minds in the world definitely rubs off, and working with Sebastian Thrun and Mike Montemerlo to apply OpenCV on Stanley (the robot that won the $2M DARPA Grand Challenge) and with Andrew Ng on STAIR (one of the most advanced personal robots) was more technological fun than a person has a right to have. It’s a department that is currently hitting on all cylinders and simply a great environment to be in. In addition to Sebastian Thrun and Andrew Ng there, I thank Daphne Koller for setting high scientific standards, and also for letting me hire away some key interns and students, as well as Kunle Olukotun and Christos Kozyrakis for many discussions and joint work. I also thank Oussama Khatib, whose work on control and manipulation has inspired my current interests in visually guided robotic manipulation. Horst Haussecker at Intel Research was a great colleague to have, and his own experience in writing a book helped inspire my effort. Finally, thanks once again to Willow Garage for allowing me to

scientific standards, and also for letting me hire away some key interns and students, as well as Kunle Olukotun and Christos Kozyrakis for many discussions and joint work. I also thank Oussama Khatib, whose work on control and manipulation has inspired my current interests in visually guided robotic manipulation. Horst Haussecker at Intel Research was a great colleague to have, and his own experience in writing a book helped inspire my effort. Finally, thanks once again to Willow Garage for allowing me to pursue my lifelong robotic dreams in a great environment featuring world-class talent while also supporting my time on this book and supporting OpenCV itself.

1 Overview What Is OpenCV? OpenCV [OpenCV] is an open source (see http://opensource.org) computer vision library available from http://opencv.org. The library is written in C and C++1 and runs under Linux, Windows, Mac OS X, iOS, and Android. Interfaces are available for Python, Java, Ruby, Matlab, and other languages. OpenCV was designed for computational efficiency with a strong focus on real-time applications: optimizations were made at all levels, from algorithms to multicore and CPU instructions. For example, OpenCV supports optimizations for SSE, MMX, AVX, NEON, OpenMP, and TBB. If you desire further optimization on Intel architectures [Intel] for basic image processing, you can buy Intel’s Integrated Performance Primitives (IPP) libraries [IPP], which consist of low-level optimized routines in many different algorithmic areas. OpenCV automatically uses the appropriate instructions from IPP at runtime. The GPU module also provides CUDA-accelerated versions of many routines (for Nvidia GPUs) and

For example, OpenCV supports optimizations for SSE, MMX, AVX, NEON, OpenMP, and TBB. If you desire further optimization on Intel architectures [Intel] for basic image processing, you can buy Intel’s Integrated Performance Primitives (IPP) libraries [IPP], which consist of low-level optimized routines in many different algorithmic areas. OpenCV automatically uses the appropriate instructions from IPP at runtime. The GPU module also provides CUDA-accelerated versions of many routines (for Nvidia GPUs) and OpenCL-optimized ones (for generic GPUs). One of OpenCV’s goals is to provide a simple-to-use computer vision infrastructure that helps people build fairly sophisticated vision applications quickly. The OpenCV library contains over 500 functions that span many areas, including factory product inspection, medical imaging, security, user interface, camera calibration, stereo vision, and robotics. Because computer vision and machine learning often go hand-in-hand, OpenCV also contains a full, general-purpose

OpenCL-optimized ones (for generic GPUs). One of OpenCV’s goals is to provide a simple-to-use computer vision infrastructure that helps people build fairly sophisticated vision applications quickly. The OpenCV library contains over 500 functions that span many areas, including factory product inspection, medical imaging, security, user interface, camera calibration, stereo vision, and robotics. Because computer vision and machine learning often go hand-in-hand, OpenCV also contains a full, general-purpose Machine Learning Library (MLL). This sub-library is focused on statistical pattern recognition and clustering. The MLL is highly useful for the vision tasks that are at the core of OpenCV’s mission, but it is general enough to be used for any machine learning problem. Who Uses OpenCV? Most computer scientists and practical programmers are aware of some facet of the role that computer vision plays. But few people are aware of all the ways in which computer vision is used. For example, most people are

Machine Learning Library (MLL). This sub-library is focused on statistical pattern recognition and clustering. The MLL is highly useful for the vision tasks that are at the core of OpenCV’s mission, but it is general enough to be used for any machine learning problem. Who Uses OpenCV? Most computer scientists and practical programmers are aware of some facet of the role that computer vision plays. But few people are aware of all the ways in which computer vision is used. For example, most people are somewhat aware of its use in surveillance, and many also know that it is increasingly being used for images and video on the Web. A few have seen some use of computer vision in game interfaces. Yet few people realize that most aerial and street-map images (such as in Google’s Street View) make heavy use of camera calibration and image stitching techniques. Some are aware of niche applications in safety monitoring, unmanned aerial vehicles, or biomedical analysis. But few are aware how pervasive machine vision has

aware of its use in surveillance, and many also know that it is increasingly being used for images and video on the Web. A few have seen some use of computer vision in game interfaces. Yet few people realize that most aerial and street-map images (such as in Google’s Street View) make heavy use of camera calibration and image stitching techniques. Some are aware of niche applications in safety monitoring, unmanned aerial vehicles, or biomedical analysis. But few are aware how pervasive machine vision has become in manufacturing: virtually everything that is mass-produced has been automatically inspected at some point using computer vision.                                                              1 The legacy C interface is still supported, and will remain so for the foreseeable future.

The BSD [BSD] open source license for OpenCV has been structured such that you can build a commercial product using all or part of OpenCV. You are under no obligation to open-source your product or to return improvements to the public domain, though we hope you will. In part because of these liberal licensing terms, there is a large user community that includes people from major companies (Google, IBM, Intel, Microsoft, Nvidia, SONY, and Siemens, to name only a few) and research centers (such as Stanford, MIT, CMU, Cambridge, Georgia Tech and INRIA). OpenCV is also present on the web for users at http://opencv.org, a website that hosts documentation, developer information, and other community resources including links to compiled binaries for various platforms. For vision developers, code, development notes and links to GitHub are at http://code.opencv.org. User questions are answered at http://answers.opencv.org/questions/ but there is still the original Yahoo groups user forum at

(such as Stanford, MIT, CMU, Cambridge, Georgia Tech and INRIA). OpenCV is also present on the web for users at http://opencv.org, a website that hosts documentation, developer information, and other community resources including links to compiled binaries for various platforms. For vision developers, code, development notes and links to GitHub are at http://code.opencv.org. User questions are answered at http://answers.opencv.org/questions/ but there is still the original Yahoo groups user forum at http://groups.yahoo.com/group/OpenCV; it has almost 50,000 members. OpenCV is popular around the world, with large user communities in China, Japan, Russia, Europe, and Israel. OpenCV has a Facebook page at https://www.facebook.com/opencvlibrary.  Since its alpha release in January 1999, OpenCV has been used in many applications, products, and research efforts. These applications include stitching images together in satellite and web maps, image scan alignment, medical image noise reduction, object analysis,

it has almost 50,000 members. OpenCV is popular around the world, with large user communities in China, Japan, Russia, Europe, and Israel. OpenCV has a Facebook page at https://www.facebook.com/opencvlibrary.  Since its alpha release in January 1999, OpenCV has been used in many applications, products, and research efforts. These applications include stitching images together in satellite and web maps, image scan alignment, medical image noise reduction, object analysis, security and intrusion detection systems, automatic monitoring and safety systems, manufacturing inspection systems, camera calibration, military applications, and unmanned aerial, ground, and underwater vehicles. It has even been used in sound and music recognition, where vision recognition techniques are applied to sound spectrogram images. OpenCV was a key part of the vision system in the robot from Stanford, “Stanley”, which won the $2M DARPA Grand Challenge desert robot race [Thrun06], and continues to play an important part in other

systems, automatic monitoring and safety systems, manufacturing inspection systems, camera calibration, military applications, and unmanned aerial, ground, and underwater vehicles. It has even been used in sound and music recognition, where vision recognition techniques are applied to sound spectrogram images. OpenCV was a key part of the vision system in the robot from Stanford, “Stanley”, which won the $2M DARPA Grand Challenge desert robot race [Thrun06], and continues to play an important part in other many robotics challenges. What Is Computer Vision? Computer vision2 is the transformation of data from 2D/3D stills or videos into either a decision or a new representation. All such transformations are done for achieving some particular goal. The input data may include some contextual information such as “the camera is mounted in a car” or “laser range finder indicates an object is 1 meter away”. The decision might be “there is a person in this scene” or “there are 14 tumor cells on this slide”. A new

many robotics challenges. What Is Computer Vision? Computer vision2 is the transformation of data from 2D/3D stills or videos into either a decision or a new representation. All such transformations are done for achieving some particular goal. The input data may include some contextual information such as “the camera is mounted in a car” or “laser range finder indicates an object is 1 meter away”. The decision might be “there is a person in this scene” or “there are 14 tumor cells on this slide”. A new representation might mean turning a color image into a grayscale image or removing camera motion from an image sequence. Because we are such visual creatures, it is easy to be fooled into thinking that computer vision tasks are easy. How hard can it be to find, say, a car when you are staring at it in an image? Your initial intuitions can be quite misleading. The human brain divides the vision signal into many channels that stream different pieces of information into your brain. Your brain has an attention

representation might mean turning a color image into a grayscale image or removing camera motion from an image sequence. Because we are such visual creatures, it is easy to be fooled into thinking that computer vision tasks are easy. How hard can it be to find, say, a car when you are staring at it in an image? Your initial intuitions can be quite misleading. The human brain divides the vision signal into many channels that stream different pieces of information into your brain. Your brain has an attention system that identifies, in a task-dependent way, important parts of an image to examine while suppressing examination of other areas. There is massive feedback in the visual stream that is, as yet, little understood. There are widespread associative inputs from muscle control sensors and all of the other senses that allow the brain to draw on cross-associations made from years of living in the world. The feedback loops in the brain go back to all stages of processing including the hardware sensors

system that identifies, in a task-dependent way, important parts of an image to examine while suppressing examination of other areas. There is massive feedback in the visual stream that is, as yet, little understood. There are widespread associative inputs from muscle control sensors and all of the other senses that allow the brain to draw on cross-associations made from years of living in the world. The feedback loops in the brain go back to all stages of processing including the hardware sensors themselves (the eyes), which mechanically control lighting via the iris and tune the reception on the surface of the retina. In a machine vision system, however, a computer receives a grid of numbers from the camera or from disk, and, in most cases, that’s it. For the most part, there’s no built-in pattern recognition, no automatic control of focus and aperture, no cross-associations with years of experience. For the most part, vision systems are still fairly naïve. Figure 1-1 shows a picture of an automobile. In

(the eyes), which mechanically control lighting via the iris and tune the reception on the surface of the retina. In a machine vision system, however, a computer receives a grid of numbers from the camera or from disk, and, in most cases, that’s it. For the most part, there’s no built-in pattern recognition, no automatic control of focus and aperture, no cross-associations with years of experience. For the most part, vision systems are still fairly naïve. Figure 1-1 shows a picture of an automobile. In that picture we see a side mirror on the driver’s side of the car. What the computer “sees” is just a grid of numbers. Any given number within that grid has a rather large noise component and so by itself gives us little information, but this grid of numbers is all the computer “sees”. Our task then becomes to turn this noisy grid of numbers into the perception: “side mirror”. Figure 1-2 gives some more insight into why computer vision is so hard.                                                              2

picture we see a side mirror on the driver’s side of the car. What the computer “sees” is just a grid of numbers. Any given number within that grid has a rather large noise component and so by itself gives us little information, but this grid of numbers is all the computer “sees”. Our task then becomes to turn this noisy grid of numbers into the perception: “side mirror”. Figure 1-2 gives some more insight into why computer vision is so hard.                                                              2 Computer vision is a vast field. This book will give you a basic grounding in the field, but we also recommend texts by Szeliski [Szeliski2011] for a good overview of practical computer vision algorithms, and Hartley [Hartley06] for how 3D vision really works.

Figure 1-1. To a computer, the car’s side mirror is just a grid of numbers In fact, the problem, as we have posed it thus far, is worse than hard; it is formally impossible to solve. Given a two-dimensional (2D) view of a 3D world, there is no unique way to reconstruct the 3D signal. Formally, such an ill-posed problem has no unique or definitive solution. The same 2D image could represent any of an infinite combination of 3D scenes, even if the data were perfect. However, as already mentioned, the data is corrupted by noise and distortions. Such corruption stems from variations in the world (weather, lighting, reflections, movements), imperfections in the lens and mechanical setup, finite integration time on the sensor (motion blur), electrical noise and compression artifacts after image capture. Given these daunting challenges, how can we make any progress?  Figure 1-2: The ill-posed nature of vision: the 2D appearance of objects can change radically with viewpoints In the design of a practical system,

is corrupted by noise and distortions. Such corruption stems from variations in the world (weather, lighting, reflections, movements), imperfections in the lens and mechanical setup, finite integration time on the sensor (motion blur), electrical noise and compression artifacts after image capture. Given these daunting challenges, how can we make any progress?  Figure 1-2: The ill-posed nature of vision: the 2D appearance of objects can change radically with viewpoints In the design of a practical system, additional contextual knowledge can often be used to work around the limitations imposed on us by visual sensors. Consider the example of a mobile robot that must find and pick up staplers in a building. The robot might use the facts that a desk is an object found inside offices and that staplers are mostly found on desks. This gives an implicit size reference; staplers must be able to fit on desks. It also helps to eliminate falsely “recognizing” staplers in impossible places (e.g., on the ceiling or a

additional contextual knowledge can often be used to work around the limitations imposed on us by visual sensors. Consider the example of a mobile robot that must find and pick up staplers in a building. The robot might use the facts that a desk is an object found inside offices and that staplers are mostly found on desks. This gives an implicit size reference; staplers must be able to fit on desks. It also helps to eliminate falsely “recognizing” staplers in impossible places (e.g., on the ceiling or a window). The robot can safely ignore a 200-foot advertising blimp shaped like a stapler because the blimp

lacks the prerequisite wood-grained background of a desk. In contrast, with tasks such as image retrieval, all stapler images in a database may be of real staplers and so large sizes and other unusual configurations may have been implicitly precluded by the assumptions of those who took the photographs. That is, the photographer perhaps took pictures only of real, normal-sized staplers. Also, when taking pictures, people tend to center objects and put them in characteristic orientations. Thus, there is often quite a bit of unintentional implicit information within photos taken by people. Contextual information can also be modeled explicitly with machine learning techniques. Hidden variables such as size, orientation to gravity, and so on can then be correlated with their values in a labeled training set. Alternatively, one may attempt to measure hidden bias variables by using additional sensors. The use of a laser range finder to measure depth allows us to accurately infer the size of an object. The next

often quite a bit of unintentional implicit information within photos taken by people. Contextual information can also be modeled explicitly with machine learning techniques. Hidden variables such as size, orientation to gravity, and so on can then be correlated with their values in a labeled training set. Alternatively, one may attempt to measure hidden bias variables by using additional sensors. The use of a laser range finder to measure depth allows us to accurately infer the size of an object. The next problem facing computer vision is noise. We typically deal with noise by using statistical methods. For example, it may be impossible to detect an edge in an image merely by comparing a point to its immediate neighbors. But if we look at the statistics over a local region, edge detection becomes much easier. A real edge should appear as a string of such immediate neighbor responses over a local region, each of whose orientation is consistent with its neighbors. It is also possible to compensate for noise

problem facing computer vision is noise. We typically deal with noise by using statistical methods. For example, it may be impossible to detect an edge in an image merely by comparing a point to its immediate neighbors. But if we look at the statistics over a local region, edge detection becomes much easier. A real edge should appear as a string of such immediate neighbor responses over a local region, each of whose orientation is consistent with its neighbors. It is also possible to compensate for noise by taking statistics over time. Still, other techniques account for noise or distortions by building explicit models learned directly from the available data. For example, because lens distortions are well understood, one need only learn the parameters for a simple polynomial model in order to describe—and thus correct almost completely—such distortions. The actions or decisions that computer vision attempts to make based on camera data are performed in the context of a specific purpose or task. We may want

taking statistics over time. Still, other techniques account for noise or distortions by building explicit models learned directly from the available data. For example, because lens distortions are well understood, one need only learn the parameters for a simple polynomial model in order to describe—and thus correct almost completely—such distortions. The actions or decisions that computer vision attempts to make based on camera data are performed in the context of a specific purpose or task. We may want to remove noise or damage from an image so that our security system will issue an alert if someone tries to climb a fence or because we need a monitoring system that counts how many people cross through an area in an amusement park. Vision software for robots that wander through office buildings will employ different strategies than vision software for stationary security cameras because the two systems have significantly different contexts and objectives. As a general rule: the more constrained a computer

to remove noise or damage from an image so that our security system will issue an alert if someone tries to climb a fence or because we need a monitoring system that counts how many people cross through an area in an amusement park. Vision software for robots that wander through office buildings will employ different strategies than vision software for stationary security cameras because the two systems have significantly different contexts and objectives. As a general rule: the more constrained a computer vision context is, the more we can rely on those constraints to simplify the problem and the more reliable our final solution will be. OpenCV is aimed at providing the basic tools needed to solve computer vision problems. In some cases, high-level functionalities in the library will be sufficient to solve the more complex problems in computer vision. Even when this is not the case, the basic components in the library are complete enough to enable creation of a complete solution of your own to almost any

vision context is, the more we can rely on those constraints to simplify the problem and the more reliable our final solution will be. OpenCV is aimed at providing the basic tools needed to solve computer vision problems. In some cases, high-level functionalities in the library will be sufficient to solve the more complex problems in computer vision. Even when this is not the case, the basic components in the library are complete enough to enable creation of a complete solution of your own to almost any computer vision problem. In the latter case, there are some tried-and-true methods of using the library; all of them start with solving the problem using as many available library components as possible. Typically, after you’ve developed this first-draft solution, you can see where the solution has weaknesses and then fix those weaknesses using your own code and cleverness (better known as “solve the problem you actually have, not the one you imagine”). You can then use your draft solution as a benchmark to

vision problem. In the latter case, there are some tried-and-true methods of using the library; all of them start with solving the problem using as many available library components as possible. Typically, after you’ve developed this first-draft solution, you can see where the solution has weaknesses and then fix those weaknesses using your own code and cleverness (better known as “solve the problem you actually have, not the one you imagine”). You can then use your draft solution as a benchmark to assess the improvements you have made. From that point, whatever weaknesses remain can be tackled by exploiting the context of the larger system in which your problem solution is embedded, or by setting out to improve some component of the system with your own novel contributions. The Origin of OpenCV OpenCV grew out of an Intel Research initiative to advance CPU-intensive applications. Toward this end, Intel launched many projects including real-time ray tracing and 3D display walls. One of the authors (Gary)

the improvements you have made. From that point, whatever weaknesses remain can be tackled by exploiting the context of the larger system in which your problem solution is embedded, or by setting out to improve some component of the system with your own novel contributions. The Origin of OpenCV OpenCV grew out of an Intel Research initiative to advance CPU-intensive applications. Toward this end, Intel launched many projects including real-time ray tracing and 3D display walls. One of the authors (Gary) working for Intel at that time was visiting universities and noticed that some top university groups, such as the MIT Media Lab, had well-developed and internally open computer vision infrastructures—code that was passed from student to student and that gave each new student a valuable head start in developing his or her own vision application. Instead of reinventing the basic functions from scratch, a new student could begin by building on top of what came before.

Thus, OpenCV was conceived as a way to make computer vision infrastructure universally available. With the aid of Intel’s Performance Library Team,3 OpenCV started with a core of implemented code and algorithmic specifications being sent to members of Intel’s Russian library team. This is the “where” of OpenCV: it started in Intel’s research lab with collaboration from the Software Performance Libraries group together with implementation and optimization expertise in Russia. Chief among the Russian team members was Vadim Pisarevsky, who managed, coded, and optimized much of OpenCV and who is still at the center of much of the OpenCV effort. Along with him, Victor Eruhimov helped develop the early infrastructure, and Valery Kuriakin managed the Russian lab and greatly supported the effort. There were several goals for OpenCV at the outset: • Advance vision research by providing not only open but also optimized code for basic vision infrastructure. No more reinventing the wheel. • Disseminate vision knowledge

was Vadim Pisarevsky, who managed, coded, and optimized much of OpenCV and who is still at the center of much of the OpenCV effort. Along with him, Victor Eruhimov helped develop the early infrastructure, and Valery Kuriakin managed the Russian lab and greatly supported the effort. There were several goals for OpenCV at the outset: • Advance vision research by providing not only open but also optimized code for basic vision infrastructure. No more reinventing the wheel. • Disseminate vision knowledge by providing a common infrastructure that developers could build on, so that code would be more readily readable and transferable. • Advance vision-based commercial applications by making portable, performance-optimized code available for free—with a license that did not require commercial applications to be open or free themselves. Those goals constitute the “why” of OpenCV. Enabling computer vision applications would increase the need for fast processors. Driving upgrades to faster processors would generate

providing a common infrastructure that developers could build on, so that code would be more readily readable and transferable. • Advance vision-based commercial applications by making portable, performance-optimized code available for free—with a license that did not require commercial applications to be open or free themselves. Those goals constitute the “why” of OpenCV. Enabling computer vision applications would increase the need for fast processors. Driving upgrades to faster processors would generate more income for Intel than selling some extra software. Perhaps that is why this open and free code arose from a hardware vendor rather than a software company. Sometimes, there is more room to be innovative at software within a hardware company. In any open source effort, it is important to reach a critical mass at which the project becomes self-sustaining. There have now been around seven million downloads of OpenCV, and this number is growing by hundreds of thousands every month4. The user group now

more income for Intel than selling some extra software. Perhaps that is why this open and free code arose from a hardware vendor rather than a software company. Sometimes, there is more room to be innovative at software within a hardware company. In any open source effort, it is important to reach a critical mass at which the project becomes self-sustaining. There have now been around seven million downloads of OpenCV, and this number is growing by hundreds of thousands every month4. The user group now approaches 50,000 members. OpenCV receives many user contributions, and central development has long since moved outside of Intel.5 OpenCV’s past timeline is shown in Figure 1-3. Along the way, OpenCV was affected by the dot-com boom and bust and also by numerous changes of management and direction. During these fluctuations, there were times when OpenCV had no one at Intel working on it at all. However, with the advent of multicore processors and the many new applications of computer vision, OpenCV’s value

50,000 members. OpenCV receives many user contributions, and central development has long since moved outside of Intel.5 OpenCV’s past timeline is shown in Figure 1-3. Along the way, OpenCV was affected by the dot-com boom and bust and also by numerous changes of management and direction. During these fluctuations, there were times when OpenCV had no one at Intel working on it at all. However, with the advent of multicore processors and the many new applications of computer vision, OpenCV’s value began to rise. Similarly, rapid growth in the field of robotics has driven much use and development of the library. After becoming an open source library, OpenCV spent several years under active development at Willow Garage and Itseez, and now is supported by the OpenCV foundation at http//opencv.org. Today, OpenCV is actively being developed by the OpenCV.org foundation, Google supports on order of 15 interns a year in the Google Summer of Code program6, and Intel is back actively supporting development. For more

rise. Similarly, rapid growth in the field of robotics has driven much use and development of the library. After becoming an open source library, OpenCV spent several years under active development at Willow Garage and Itseez, and now is supported by the OpenCV foundation at http//opencv.org. Today, OpenCV is actively being developed by the OpenCV.org foundation, Google supports on order of 15 interns a year in the Google Summer of Code program6, and Intel is back actively supporting development. For more information on the future of OpenCV, see Chapter 14.                                                               3 Shinn Lee was of key help as was Stewart Taylor. 4 It is noteworthy, that at the time of the publication of “Learning OpenCV” in 2006, this rate was 26,000 per month. Seven years later, the download rate has grown to over 160,000 downloads per month. 5 As of this writing, Itseez (http://itseez.com/) is the primary maintainer of OpenCV 6 Google Summer of Code

supporting development. For more information on the future of OpenCV, see Chapter 14.                                                               3 Shinn Lee was of key help as was Stewart Taylor. 4 It is noteworthy, that at the time of the publication of “Learning OpenCV” in 2006, this rate was 26,000 per month. Seven years later, the download rate has grown to over 160,000 downloads per month. 5 As of this writing, Itseez (http://itseez.com/) is the primary maintainer of OpenCV 6 Google Summer of Code https://developers.google.com/open-source/soc/

Figure 1-3: OpenCV timeline Who Owns OpenCV? Although Intel started OpenCV, the library is and always was intended to promote commercial and research use. It is therefore open and free, and the code itself may be used or embedded (in whole or in part) in other applications, whether commercial or research. It does not force your application code to be open or free. It does not require that you return improvements back to the library—but we hope that you will.  Downloading and Installing OpenCV The main OpenCV site is at http://opencv.org, from which you can download the complete source code for the latest release, as well as many recent releases. The downloads themselves are found at the downloads page: http://opencv.org/downloads.html. However, if you want the very most up-to-date version it is always found on GitHub at https://github.com/Itseez/opencv, where the active development branch is stored. The computer vision developer’s site (with links to the above) is at http://code.opencv.org/. Installation In

site is at http://opencv.org, from which you can download the complete source code for the latest release, as well as many recent releases. The downloads themselves are found at the downloads page: http://opencv.org/downloads.html. However, if you want the very most up-to-date version it is always found on GitHub at https://github.com/Itseez/opencv, where the active development branch is stored. The computer vision developer’s site (with links to the above) is at http://code.opencv.org/. Installation In modern times, OpenCV uses Git as its development version control system, and CMake to build7. In many cases, you will not need to worry about building, as compiled libraries exist for supported environments.  However, as you become a more advanced user, you will inevitably want to be able to recompile the libraries with specific options tailored to your application and environment. On the tutorial pages at http://docs.opencv.org/doc/tutorials/tutorials.html under “introduction to OpenCV”, there are

In modern times, OpenCV uses Git as its development version control system, and CMake to build7. In many cases, you will not need to worry about building, as compiled libraries exist for supported environments.  However, as you become a more advanced user, you will inevitably want to be able to recompile the libraries with specific options tailored to your application and environment. On the tutorial pages at http://docs.opencv.org/doc/tutorials/tutorials.html under “introduction to OpenCV”, there are descriptions of how to set up OpenCV to work with a number of combinations of operating systems and development tools.                                                              7 In olden times, OpenCV developers used Subversion for version control and automake to build. Those days, however, are long gone.

Windows At the page: http://opencv.org/downloads.html, you will see a link to download the latest version of OpenCV for Windows.  This link will download an executable file which you can run, and which will install OpenCV, register DirectShow filters, and perform various post-installation procedures. You are now almost ready to start using OpenCV.8  The one additional detail is that you will want to add is an OPENCV_DIR environment variable to make it easier to tell your compiler where to find the OpenCV binaries. You can set this by going to a command prompt and typing9: setx -m OPENCV_DIR D:\OpenCV\Build\x86\vc10 If you built the library to link statically, this is all you will need.  If you built the library to link dynamically, then you will also need to tell your system where to find the library binary. To do this, simply add %OPENCV_DIR%\bin to your library path. (For example, in Windows 7, right-click on your Computer icon, select Properties, and then click on Advanced System Settings. Finally select

You can set this by going to a command prompt and typing9: setx -m OPENCV_DIR D:\OpenCV\Build\x86\vc10 If you built the library to link statically, this is all you will need.  If you built the library to link dynamically, then you will also need to tell your system where to find the library binary. To do this, simply add %OPENCV_DIR%\bin to your library path. (For example, in Windows 7, right-click on your Computer icon, select Properties, and then click on Advanced System Settings. Finally select Environment Variables and add the OpenCV binary path to the Path variable.) To add the commercial IPP performance optimizations to Windows, obtain and install IPP from the Intel site (http://www.intel.com/software/products/ipp/index.htm); use version 5.1 or later. Make sure the appropriate binary folder (e.g., c:/program files/intel/ipp/5.1/ia64/bin) is in the system path. IPP should now be automatically detected by OpenCV and loaded at runtime (more on this in Chapter 3). Linux Prebuilt binaries for Linux are not

Variables and add the OpenCV binary path to the Path variable.) To add the commercial IPP performance optimizations to Windows, obtain and install IPP from the Intel site (http://www.intel.com/software/products/ipp/index.htm); use version 5.1 or later. Make sure the appropriate binary folder (e.g., c:/program files/intel/ipp/5.1/ia64/bin) is in the system path. IPP should now be automatically detected by OpenCV and loaded at runtime (more on this in Chapter 3). Linux Prebuilt binaries for Linux are not included with the Linux version of OpenCV owing to the large variety of versions of GCC and GLIBC in different distributions (SuSE, Debian, Ubuntu, etc.). In many cases however, your distribution will include OpenCV. If your distribution doesn’t offer OpenCV, you will have to build it from sources. As with the Windows installation, you can start at the http://opencv.org/downloads.html page, but in this case the link will send you to Sourceforge10, where you can select the tarball for the current OpenCV source

with the Linux version of OpenCV owing to the large variety of versions of GCC and GLIBC in different distributions (SuSE, Debian, Ubuntu, etc.). In many cases however, your distribution will include OpenCV. If your distribution doesn’t offer OpenCV, you will have to build it from sources. As with the Windows installation, you can start at the http://opencv.org/downloads.html page, but in this case the link will send you to Sourceforge10, where you can select the tarball for the current OpenCV source code bundle. To build the libraries and demos, you’ll need GTK+ 2.x or higher, including headers. You’ll also need pkgconfig, libpng, libjpeg, libtiff, and libjasper with development files (i.e., the versions with -dev at the end of their package names). You’ll need Python 2.6 or later with headers installed (developer package). You will also need libavcodec and the other libav* libraries (including headers) from ffmpeg 1.0 or later . Download ffmpeg from http://ffmpeg.mplayerhq.hu/download.html.11 The ffmpeg

bundle. To build the libraries and demos, you’ll need GTK+ 2.x or higher, including headers. You’ll also need pkgconfig, libpng, libjpeg, libtiff, and libjasper with development files (i.e., the versions with -dev at the end of their package names). You’ll need Python 2.6 or later with headers installed (developer package). You will also need libavcodec and the other libav* libraries (including headers) from ffmpeg 1.0 or later . Download ffmpeg from http://ffmpeg.mplayerhq.hu/download.html.11 The ffmpeg program has a lesser general public license (LGPL). To use it with non-GPL software (such as OpenCV), build and use a shared ffmpg library: $> ./configure --enable-shared $> make $> sudo make install You will end up with: /usr/local/lib/libavcodec.so.*, /usr/local/lib/libavformat.so.*, /usr/local/lib/libavutil.so.*, and include files under various /usr/local/include/libav*. To build OpenCV once it is downloaded:12                                                              8 It is important to know that,

program has a lesser general public license (LGPL). To use it with non-GPL software (such as OpenCV), build and use a shared ffmpg library: $> ./configure --enable-shared $> make $> sudo make install You will end up with: /usr/local/lib/libavcodec.so.*, /usr/local/lib/libavformat.so.*, /usr/local/lib/libavutil.so.*, and include files under various /usr/local/include/libav*. To build OpenCV once it is downloaded:12                                                              8 It is important to know that, although the Windows distribution contains binary libraries for release builds, it does not contain the debug builds of these libraries. It is therefore likely that, before developing with OpenCV, you will want to open the solution file and build these libraries for yourself. 9 Of course, the exact path will vary depending on your installation, for example if you are installing on an ia64 machine, then the path will not include “x86”, but rather “ia64”. 10 OpenCV has all of its many builds and versions

although the Windows distribution contains binary libraries for release builds, it does not contain the debug builds of these libraries. It is therefore likely that, before developing with OpenCV, you will want to open the solution file and build these libraries for yourself. 9 Of course, the exact path will vary depending on your installation, for example if you are installing on an ia64 machine, then the path will not include “x86”, but rather “ia64”. 10 OpenCV has all of its many builds and versions available from Sourceforge. See links at http://opencv.org/downloads.html  11 You can check out ffmpeg by: svn checkout svn://svn.mplayerhq.hu/ffmpeg/trunk ffmpeg.

$> mkdir build && cd build $> cmake .. && make $> sudo make install $> sudo ldconfig After installation is complete, the default installation path is /usr/local/lib/ and /usr/local/include/opencv2/. Hence you need to add /usr/local/lib/ to /etc/ld.so.conf (and run ldconfig afterwards) or add it to the LD_LIBRARY_PATH environment variable; then you are done. To actually build the library, you will need to go unpack the .tgz file and go into the created source directory, and do the following: mkdir release cd release cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local .. make sudo make install The first and second commands create a new subdirectory and move you into it. The third command tells CMake how to configure your build. The example options we give are probably the right ones to get you started, but other options allow you to enable various options, determine what examples are built, python support, CUDA GPU support, etc. The last two commands actually build the library and install the

release cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local .. make sudo make install The first and second commands create a new subdirectory and move you into it. The third command tells CMake how to configure your build. The example options we give are probably the right ones to get you started, but other options allow you to enable various options, determine what examples are built, python support, CUDA GPU support, etc. The last two commands actually build the library and install the results into the proper places. To add the commercial IPP performance optimizations to Linux, install IPP as described previously. Let’s assume it was installed in /opt/intel/ipp/5.1/ia32/. Add <your install_path>/bin/ and <your install_path>/bin/linux32 LD_LIBRARY_PATH in your initialization script (.bashrc or similar): LD_LIBRARY_PATH=/opt/intel/ipp/5.1/ia32/bin:/opt/intel/ipp/5.1 /ia32/bin/linux32:$LD_LIBRARY_PATH export LD_LIBRARY_PATH Alternatively, you can add <your install_path>/bin and <your

the results into the proper places. To add the commercial IPP performance optimizations to Linux, install IPP as described previously. Let’s assume it was installed in /opt/intel/ipp/5.1/ia32/. Add <your install_path>/bin/ and <your install_path>/bin/linux32 LD_LIBRARY_PATH in your initialization script (.bashrc or similar): LD_LIBRARY_PATH=/opt/intel/ipp/5.1/ia32/bin:/opt/intel/ipp/5.1 /ia32/bin/linux32:$LD_LIBRARY_PATH export LD_LIBRARY_PATH Alternatively, you can add <your install_path>/bin and <your install_path>/bin/linux32, one per line, to /etc/ld.so.conf and then run ldconfig as root (or use sudo). That’s it. Now OpenCV should be able to locate IPP shared libraries and make use of them on Linux. See …/opencv/INSTALL for more details. MacOS X As of this writing, full functionality on MacOS X is a priority but there are still some limitations (e.g., writing AVIs); these limitations are described in …/opencv/INSTALL. The requirements and building instructions are similar to the Linux case, with the

install_path>/bin/linux32, one per line, to /etc/ld.so.conf and then run ldconfig as root (or use sudo). That’s it. Now OpenCV should be able to locate IPP shared libraries and make use of them on Linux. See …/opencv/INSTALL for more details. MacOS X As of this writing, full functionality on MacOS X is a priority but there are still some limitations (e.g., writing AVIs); these limitations are described in …/opencv/INSTALL. The requirements and building instructions are similar to the Linux case, with the following exceptions: • By default, Carbon is used instead of GTK+. • By default, QuickTime is used instead of ffmpeg. • pkg-config is optional (it is used explicitly only in the samples/c/build_all.sh script). • RPM and ldconfig are not supported by default. Use configure+make+sudo make install to build and install OpenCV, update DYLD_LIBRARY_PATH (unless ./configure --prefix=/usr is used). For full functionality, you should install libpng, libtiff, libjpeg and libjasper from darwinports and/or fink and

following exceptions: • By default, Carbon is used instead of GTK+. • By default, QuickTime is used instead of ffmpeg. • pkg-config is optional (it is used explicitly only in the samples/c/build_all.sh script). • RPM and ldconfig are not supported by default. Use configure+make+sudo make install to build and install OpenCV, update DYLD_LIBRARY_PATH (unless ./configure --prefix=/usr is used). For full functionality, you should install libpng, libtiff, libjpeg and libjasper from darwinports and/or fink and make them available to ./configure (see ./configure --help). Then: sudo port selfupdate sudo port install opencv                                                                                                                                                                                      12 To build OpenCV using Red Hat Package Managers (RPMs), use rpmbuild –ta OpenCV-x.y.z.tar.gz (for RPM 4.x or later), or rpm –ta OpenCV-x.y.z.tar.gz (for earlier versions of RPM), where OpenCV-x.y.z.tar.gz should be

make them available to ./configure (see ./configure --help). Then: sudo port selfupdate sudo port install opencv                                                                                                                                                                                      12 To build OpenCV using Red Hat Package Managers (RPMs), use rpmbuild –ta OpenCV-x.y.z.tar.gz (for RPM 4.x or later), or rpm –ta OpenCV-x.y.z.tar.gz (for earlier versions of RPM), where OpenCV-x.y.z.tar.gz should be put in /usr/src/redhat/SOURCES/ or a similar directory. Then install OpenCV using rpm -i OpenCV-x.y.z.*.rpm.

Notes on Building with CMake The modern OpenCV library relies on CMake in its build system. This has the advantage of making the platform much easier to work with in a cross-platform environment.  Whether you are using the command line version of CMake on Linux (or on Windows using a command line environment such as Cygwin), or using a visual interface to CMake such as cmake-gui, you can build OpenCV in just about any environment in the same way. If you are not already familiar with CMake, the essential concept behind CMake is to allow the developers to specify all of the information needed for compilation in a platform independent way (files called CMakeLists.txt files), which CMake then converts to the platform dependent files used in your environment (e.g., makefiles on Unix and projects or workspaces in the Windows visual environment). When you use CMake, you can supply additional options which tell CMake how to proceed in the generation of the build files.  For example, it is CMake which you tell if you

behind CMake is to allow the developers to specify all of the information needed for compilation in a platform independent way (files called CMakeLists.txt files), which CMake then converts to the platform dependent files used in your environment (e.g., makefiles on Unix and projects or workspaces in the Windows visual environment). When you use CMake, you can supply additional options which tell CMake how to proceed in the generation of the build files.  For example, it is CMake which you tell if you want a debug or release library or if you do or do not want to include a feature like the Intel Performance Primitives (IPP).   An example CMake command line invocation, which we encountered earlier, might be: cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local .. The last argument ‘..’ is actually a directory specification, which tells CMake where the root of the source directory to build is. In this case it is set to ‘..’ (the directory above) because it is conventional in OpenCV to actually

a debug or release library or if you do or do not want to include a feature like the Intel Performance Primitives (IPP).   An example CMake command line invocation, which we encountered earlier, might be: cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local .. The last argument ‘..’ is actually a directory specification, which tells CMake where the root of the source directory to build is. In this case it is set to ‘..’ (the directory above) because it is conventional in OpenCV to actually do the build in a subdirectory of the OpenCV root.  In this case, the –D option is used to set environment variables which CMake will use to determine what to put in your build files. These environment variables might be essentially enumerated values (such as RELEASE or DEBUG for CMAKE_BUILD_TYPE), or they might be strings (such as ‘/usr/local’ for CMAKE_INSTALL_PREFIX).  Below is a partial list13 containing many of the common options that you are likely to want. Table 1-1: Basic CMake options you will

do the build in a subdirectory of the OpenCV root.  In this case, the –D option is used to set environment variables which CMake will use to determine what to put in your build files. These environment variables might be essentially enumerated values (such as RELEASE or DEBUG for CMAKE_BUILD_TYPE), or they might be strings (such as ‘/usr/local’ for CMAKE_INSTALL_PREFIX).  Below is a partial list13 containing many of the common options that you are likely to want. Table 1-1: Basic CMake options you will probably need. Option   Definition Accepted Values CMAKE_BUILD_TYPE Controls release vs. debug build RELEASE or DEBUG CMAKE_INSTALL_PREFIX  Where to put installed library Path, e.g. /usr/local CMAKE_VERBOSE Lots of extra information from CMake ON or OFF  Table 1-2: Options which introduce features into the library. All of these can be either ON or OFF. Option  Definition Default WITH_1394 Use libdc1394 ON WITH_CUDA CUDA support, requires toolkit OFF WITH_EIGEN2 Use Eigen library for linear algebra ON

will probably need. Option   Definition Accepted Values CMAKE_BUILD_TYPE Controls release vs. debug build RELEASE or DEBUG CMAKE_INSTALL_PREFIX  Where to put installed library Path, e.g. /usr/local CMAKE_VERBOSE Lots of extra information from CMake ON or OFF  Table 1-2: Options which introduce features into the library. All of these can be either ON or OFF. Option  Definition Default WITH_1394 Use libdc1394 ON WITH_CUDA CUDA support, requires toolkit OFF WITH_EIGEN2 Use Eigen library for linear algebra ON WITH_FFMPEG Use ffmpeg for video I/O ON                                                              13 You are probably wondering: “why not a complete list?” The reason is simply that the available options fluctuate with the passage of time. The ones we list here are some of the more important ones however which are likely to stay intact into the foreseeable future.

WITH_OPENGL enable wrappers for OpenGL objects in the core module and the function: cv::setOpenGLDrawCallback()   OFF WITH_GSTREAMER Include Gstreamer support ON WITH_GTK Include GTK support ON WITH_IPP Intel Performance Primitives14 OFF WITH_JASPER JPEG 2000 support for imread() ON WITH_JPEG JPEG support for imread() ON WITH_MIKTEX PDF documentation on Windows OFF WITH_OPEN_EXR EXR support for imread() ON WITH_OPENCL OpenCL support (similar to CUDA) OFF WITH_OPENNI Support for Kinect cameras OFF WITH_PNG PNG support for loading ON WITH_QT Qt based Highgui functions OFF WITH_QUICKTIME Use Quicktime for video I/O (instead of QTKit – Mac only) OFF WITH_TBB Intel Thread Building Blocks OFF WITH_TIFF TIFF support for imread() ON WITH_UNICAP Use Unicap library, provides I/O support for cameras using this standard. OFF WITH_V4L Video for Linux support ON WITH_VIDEOINPUT Use alternate library for video I/O (Windows only) OFF WITH_XIMEA Support for Ximea cameras OFF WITH_XINE Enable Xine multimedia library for video

support for loading ON WITH_QT Qt based Highgui functions OFF WITH_QUICKTIME Use Quicktime for video I/O (instead of QTKit – Mac only) OFF WITH_TBB Intel Thread Building Blocks OFF WITH_TIFF TIFF support for imread() ON WITH_UNICAP Use Unicap library, provides I/O support for cameras using this standard. OFF WITH_V4L Video for Linux support ON WITH_VIDEOINPUT Use alternate library for video I/O (Windows only) OFF WITH_XIMEA Support for Ximea cameras OFF WITH_XINE Enable Xine multimedia library for video OFF ENABLE_SOLUTION_FOLDERS Categorize binaries inside directories in Solution Explorer OFF                                                              14 Associated with the WITH_IPP option, there is also the IPP_PATH option. The IPP_PATH can be set to any normal path name and indicates where the IPP libraries should be found. However, it should not be necessary if you have these libraries in the “usual” place for your platform.

Table 1.3: Options passed by CMake to the compiler. Option  Definition Default ENABLE_SSE Enable Streaming SIMD (SSE) instructions ON ENABLE_SSE2  Enable Intel SSE 2 instructions ON ENABLE_SSE3  Enable Intel SSE 3 instructions OFF ENABLE_SSSE3 Enable Supplemental SSE 3 instructions OFF ENABLE_SSE41 Enable Intel SSE 4.1 instructions OFF ENABLE_SSE42 Enable Intel SSE 4.1 instructions OFF USE_O3 Enable high optimization (i.e. –o3) ON USE_OMIT_FRAME_POINTER Enable omit frame pointer (i.e. –fomit-frame-pointer) ON  Table 1-4: ‘Build’ options control exactly what gets created at compile time. All of these can be either ON or OFF. In most all cases the default is OFF. Option Definition Default BUILD_DOCS Generate OpenCV documentation OFF BUILD_EXAMPLES Generate example code OFF BUILD_JAVA_SUPPORT Create Java OpenCV libraries OFF BUILD_PYTHON_SUPPORT Deprecated (use BUILD_NEW_PYTHON_SUPPORT) OFF BUILD_NEW_PYTHON_SUPPORT Create Python OpenCV libraries ON BUILD_PACKAGE Create zip archive with OpenCV sources OFF

ON  Table 1-4: ‘Build’ options control exactly what gets created at compile time. All of these can be either ON or OFF. In most all cases the default is OFF. Option Definition Default BUILD_DOCS Generate OpenCV documentation OFF BUILD_EXAMPLES Generate example code OFF BUILD_JAVA_SUPPORT Create Java OpenCV libraries OFF BUILD_PYTHON_SUPPORT Deprecated (use BUILD_NEW_PYTHON_SUPPORT) OFF BUILD_NEW_PYTHON_SUPPORT Create Python OpenCV libraries ON BUILD_PACKAGE Create zip archive with OpenCV sources OFF BUILD_SHARED_LIBS Build OpenCV as dynamic library (default is static) ON BUILD_TESTS Build test programs for each OpenCV module ON BUILD_PERF_TESTS Build available function performance tests OFF

Table 1-5: 'Install' options determine what compiled executables get placed in your binaries area. INSTALL_C_EXAMPLES  Install the C and C++ examples in the “usual” place for binaries OFF INSTALL_PYTHON_EXAMPLES  Install the Python examples in the “usual” place for binaries OFF Not listed in these tables are additional variables which can be set to indicate the locations of various libraries (e.g., libjasper, etc.) in such case as they are not in their default locations. For more information on these more obscure options, a visit to the online documentation at http://opencv.org is recommended. Getting the Latest OpenCV via Git OpenCV is under active development, and bugs are often fixed rapidly when reports contain accurate descriptions and code that demonstrates the bug. However, official major OpenCV releases only occur two to four times a year. If you are seriously developing a project or product, you will probably want code fixes and updates as soon as they become available. To do this, you will need to

obscure options, a visit to the online documentation at http://opencv.org is recommended. Getting the Latest OpenCV via Git OpenCV is under active development, and bugs are often fixed rapidly when reports contain accurate descriptions and code that demonstrates the bug. However, official major OpenCV releases only occur two to four times a year. If you are seriously developing a project or product, you will probably want code fixes and updates as soon as they become available. To do this, you will need to access OpenCV’s Git repository on Github. This isn’t the place for a tutorial in Git usage. If you’ve worked with other open source projects then you’re probably familiar with it already. If you haven’t, check out Version Control with Git by Jon Loeliger (O’Reilly). A command-line Git client is available for Linux, OS X, and most UNIX-like systems. For Windows users, we recommend TortoiseGit (http://code.google.com/p/tortoisegit/). On Windows, if you want the latest OpenCV from the Git repository then

to access OpenCV’s Git repository on Github. This isn’t the place for a tutorial in Git usage. If you’ve worked with other open source projects then you’re probably familiar with it already. If you haven’t, check out Version Control with Git by Jon Loeliger (O’Reilly). A command-line Git client is available for Linux, OS X, and most UNIX-like systems. For Windows users, we recommend TortoiseGit (http://code.google.com/p/tortoisegit/). On Windows, if you want the latest OpenCV from the Git repository then you’ll need to clone the OpenCV repository on https://github.com/Itseez/opencv.git. On Linux and Mac, you can just use the following command: git clone https://github.com/Itseez/opencv.git  More OpenCV Documentation The primary documentation for OpenCV is the HTML documentation available at: http//opencv.org. In addition to this, there are extensive tutorials on many subjects at http://docs.opencv.org/doc/tutorials/tutorials.html, and an OpenCV Wiki (currently located at

OpenCV from the Git repository then you’ll need to clone the OpenCV repository on https://github.com/Itseez/opencv.git. On Linux and Mac, you can just use the following command: git clone https://github.com/Itseez/opencv.git  More OpenCV Documentation The primary documentation for OpenCV is the HTML documentation available at: http//opencv.org. In addition to this, there are extensive tutorials on many subjects at http://docs.opencv.org/doc/tutorials/tutorials.html, and an OpenCV Wiki (currently located at http://code.opencv.org/projects/opencv/wiki). Online Documentation and the Wiki As briefly mentioned earlier, there is extensive documentation as well as a wiki available at http://opencv.org. The documentation there is divided into several major components:  • Reference (http://docs.opencv.org/): This section contains the functions, their arguments, and some information on how to use them. • Tutorials (http://docs.opencv.org/doc/tutorials/tutorials.html): There is a large collection of tutorials, these

at http://code.opencv.org/projects/opencv/wiki). Online Documentation and the Wiki As briefly mentioned earlier, there is extensive documentation as well as a wiki available at http://opencv.org. The documentation there is divided into several major components:  • Reference (http://docs.opencv.org/): This section contains the functions, their arguments, and some information on how to use them. • Tutorials (http://docs.opencv.org/doc/tutorials/tutorials.html): There is a large collection of tutorials, these tell you how to accomplish various things.  There are tutorials for basic subjects, like how to install OpenCV or create OpenCV projects on various platforms, and more advanced topics like background subtraction of object detection. • Quick Start (http://opencv.org/quickstart.html): This is really a tightly curated subset of the tutorials, containing just ones that help you get up and running on specific platforms.

• Cheat Sheet (http://docs.opencv.org/trunk/opencv_cheatsheet.pdf): This is actually a single .pdf file which contains a truly excellent compressed reference to almost the entire library. Thank Vadim Pisarevsky for this excellent reference as you pin these two beautiful pages to your cubicle wall. • Wiki (http://code.opencv.org/projects/opencv/wiki): The wiki contains everything you could possible want and more. This is where the roadmap can be found, as well as news, open issues, bugs tracking, and countless deeper topics like how to become a contributor to OpenCV. • Q&A (http://answers.opencv.org/questions): This is a vast archive of literally thousands of questions people have asked, and answered.  You can go there to ask questions of the OpenCV community, or to help others by answering their questions. All of these are accessible under the “Documentation” button on the OpenCV.org homepage.  Of all of those great resources, one warrants a little more discussion here, which is the Reference. The reference

deeper topics like how to become a contributor to OpenCV. • Q&A (http://answers.opencv.org/questions): This is a vast archive of literally thousands of questions people have asked, and answered.  You can go there to ask questions of the OpenCV community, or to help others by answering their questions. All of these are accessible under the “Documentation” button on the OpenCV.org homepage.  Of all of those great resources, one warrants a little more discussion here, which is the Reference. The reference is divided into several sections, each of which pertains to what is called a module in the library. The exact module list has evolved over time, but the modules are the primary organizational structure in the library.  Every function in the library is part of one module. Here are the current modules: • core: The “core” is the section of the library which contains all of the basic object types and their basic operations. • imgproc: The image processing module contains basic transformations on images, including

divided into several sections, each of which pertains to what is called a module in the library. The exact module list has evolved over time, but the modules are the primary organizational structure in the library.  Every function in the library is part of one module. Here are the current modules: • core: The “core” is the section of the library which contains all of the basic object types and their basic operations. • imgproc: The image processing module contains basic transformations on images, including filters and similar convolutional operators. • highgui: This HighGUI module contains user interface functions which can be used to display images or take simple user input. It can be thought of as a very light weight window UI toolkit. • video: The video library contains the functions you need to read and write video streams. • calib3d: This module contains implementations of algorithms you will need to calibrate single cameras as well as stereo or multi-camera arrays. • features2d: The features2d module

filters and similar convolutional operators. • highgui: This HighGUI module contains user interface functions which can be used to display images or take simple user input. It can be thought of as a very light weight window UI toolkit. • video: The video library contains the functions you need to read and write video streams. • calib3d: This module contains implementations of algorithms you will need to calibrate single cameras as well as stereo or multi-camera arrays. • features2d: The features2d module contains algorithms for detecting, describing, and matching keypoint features. • objdetect: This module contains algorithms for detecting specific objects, such as faces or pedestrians. You can train the detectors to detect other objects as well. • ml: The Machine Learning Library is actually an entire library in itself, and contains a wide array of machine learning algorithms implemented in such as way as to work with the natural data structures of OpenCV. • flann: FLANN stands for “Fast Library for

contains algorithms for detecting, describing, and matching keypoint features. • objdetect: This module contains algorithms for detecting specific objects, such as faces or pedestrians. You can train the detectors to detect other objects as well. • ml: The Machine Learning Library is actually an entire library in itself, and contains a wide array of machine learning algorithms implemented in such as way as to work with the natural data structures of OpenCV. • flann: FLANN stands for “Fast Library for Approximate Nearest Neighbors”. This library contains methods you will not likely use directly, but which are used by other functions in other modules for doing nearest neighbor searches in large data sets. • gpu: The GPU library contains implementations of most of the rest of the library functions optimized for operation on CUDA GPUs. There are also some functions which are only implemented for GPU operation. Some of these provide excellent results but require computational resources sufficiently high that

Nearest Neighbors”. This library contains methods you will not likely use directly, but which are used by other functions in other modules for doing nearest neighbor searches in large data sets. • gpu: The GPU library contains implementations of most of the rest of the library functions optimized for operation on CUDA GPUs. There are also some functions which are only implemented for GPU operation. Some of these provide excellent results but require computational resources sufficiently high that implementation on non-GPU hardware would provide little utility. • photo: This is a relatively new module which contains tools useful for computational photography. • stitching: This entire module implements a sophisticated image stitching pipeline. This is new functionality in the library but, like the ‘photo’ module is a place where future growth is expected. • nonfree: OpenCV contains some implementations of algorithms which are patented or are otherwise burdened by some usage restrictions (e.g., the SIFT

on non-GPU hardware would provide little utility. • photo: This is a relatively new module which contains tools useful for computational photography. • stitching: This entire module implements a sophisticated image stitching pipeline. This is new functionality in the library but, like the ‘photo’ module is a place where future growth is expected. • nonfree: OpenCV contains some implementations of algorithms which are patented or are otherwise burdened by some usage restrictions (e.g., the SIFT algorithm). Those algorithms are segregated off to their own module, so that you will know that you will need to do some kind of special work in order to use them in a commercial product.

• contrib: This module contains new things that have yet to be blessed into the whole of the library. • legacy: This module contains old things that have yet to be banished from the library altogether. • ocl: The OCL module is a newer module, which could be considered analogous to the GPU module, except that it relies on OpenCL, a Khronos standard for open parallel computing. Though less featured than the GPU module at this time, the OCL module aims to provide implementations which can run on any GPU or other device supporting OpenCL. (This is in contrast to the GPU module which explicitly makes use of the Nvidia CUDA toolkit and so will only work on Nvidia GPU devices.)  Despite the ever-increasing quality of this online documentation, one task which is not within their scope is to provide a proper understanding of the algorithms implemented or of the exact meaning of the parameters these algorithms require. This book aims to provide this information, as well as a more in depth understanding of all of the

device supporting OpenCL. (This is in contrast to the GPU module which explicitly makes use of the Nvidia CUDA toolkit and so will only work on Nvidia GPU devices.)  Despite the ever-increasing quality of this online documentation, one task which is not within their scope is to provide a proper understanding of the algorithms implemented or of the exact meaning of the parameters these algorithms require. This book aims to provide this information, as well as a more in depth understanding of all of the basic building blocks of the library. Exercises 1. Download and install the latest release of OpenCV. Compile it in debug and release mode. 2. Download and build the latest trunk version of OpenCV using Git. 3. Describe at least three ambiguous aspects of converting 3D inputs into a 2D representation. How would you overcome these ambiguities? 4. What shapes can a rectangle take when you look at in with perspective distortion (that is, when you look at it in the real world)? 5. Describe how you might start

basic building blocks of the library. Exercises 1. Download and install the latest release of OpenCV. Compile it in debug and release mode. 2. Download and build the latest trunk version of OpenCV using Git. 3. Describe at least three ambiguous aspects of converting 3D inputs into a 2D representation. How would you overcome these ambiguities? 4. What shapes can a rectangle take when you look at in with perspective distortion (that is, when you look at it in the real world)? 5. Describe how you might start processing the image in Figure 1-1 to identify the mirror on the car? 6. How might you tell the difference between a edges in an image created by: a) A shadow? b) Paint on a surface? c) Two sides of a brick? d) The side of an object and the background?

2 Introduction to OpenCV 2.x Include files After installing the OpenCV library and setting up our programming environment, our next task is to make something interesting happen with code. In order to do this, we’ll have to discuss header files. Fortunately, the headers reflect the new, modular structure of OpenCV introduced in Chapter 1. The main header file of interest is …/include/opencv2/opencv.hpp. This header file just calls the header files for each OpenCV module: #include "opencv2/core/core_c.h" Old C data structures and arithmetic routines. #include "opencv2/core/core.hpp" New C++ data structures and arithmetic routines. #include "opencv2/flann/miniflann.hpp" Approximate nearest neighbor matching functions. (Mostly for internal use) #include "opencv2/imgproc/imgproc_c.h" Old C image processing functions. #include "opencv2/imgproc/imgproc.hpp" New C++ image processing functions.  #include "opencv2/video/photo.hpp"  Algorithms specific to handling and restoring photographs.  #include

Old C data structures and arithmetic routines. #include "opencv2/core/core.hpp" New C++ data structures and arithmetic routines. #include "opencv2/flann/miniflann.hpp" Approximate nearest neighbor matching functions. (Mostly for internal use) #include "opencv2/imgproc/imgproc_c.h" Old C image processing functions. #include "opencv2/imgproc/imgproc.hpp" New C++ image processing functions.  #include "opencv2/video/photo.hpp"  Algorithms specific to handling and restoring photographs.  #include "opencv2/video/video.hpp" Video tracking and background segmentation routines. #include "opencv2/features2d/features2d.hpp" Two-dimensional feature tracking support. #include "opencv2/objdetect/objdetect.hpp" Cascade face detector; latent SVM; HoG; planar patch detector.

#include "opencv2/calib3d/calib3d.hpp" Calibration and stereo. #include "opencv2/ml/ml.hpp" Machine learning: clustering, pattern recognition. #include "opencv2/highgui/highgui_c.h" Old C image display, sliders, mouse interaction, I/O. #include "opencv2/highgui/highgui.hpp" New C++ image display, sliders, buttons, mouse, I/O. #include "opencv2/contrib/contrib.hpp"  User-contributed code: flesh detection, fuzzy mean-shift tracking, spin images, self-similar features.  You may use the include file opencv.hpp to include any and every possible OpenCV function but, since it includes everything, it will cause compile time to be slower.  If you are only using, say, image processing functions, compile time will be faster if you only include opencv2/imgproc/imgproc.hpp. These include files are located on disk under the …/modules directory. For example, imgproc.hpp is located at …/modules/imgproc/include/opencv2/imgproc/imgproc.hpp. Similarly, the sources for the functions themselves are located under their

opencv.hpp to include any and every possible OpenCV function but, since it includes everything, it will cause compile time to be slower.  If you are only using, say, image processing functions, compile time will be faster if you only include opencv2/imgproc/imgproc.hpp. These include files are located on disk under the …/modules directory. For example, imgproc.hpp is located at …/modules/imgproc/include/opencv2/imgproc/imgproc.hpp. Similarly, the sources for the functions themselves are located under their corresponding src directory. For example, cv::Canny() in the imgproc module is located in …/modules/improc/src/canny.cpp.  With the above include files, we can start our first C++ OpenCV program. Legacy code such as the older blob tracking, hmm face detection, condensation tracker, and eigen objects can be included using opencv2/legacy/legacy.hpp, which is located in …/modules/legacy/include/opencv2/legacy/legacy.hpp. First Program—Display a Picture OpenCV provides utilities for reading from a wide array

corresponding src directory. For example, cv::Canny() in the imgproc module is located in …/modules/improc/src/canny.cpp.  With the above include files, we can start our first C++ OpenCV program. Legacy code such as the older blob tracking, hmm face detection, condensation tracker, and eigen objects can be included using opencv2/legacy/legacy.hpp, which is located in …/modules/legacy/include/opencv2/legacy/legacy.hpp. First Program—Display a Picture OpenCV provides utilities for reading from a wide array of image file types, as well as from video and cameras. These utilities are part of a toolkit called HighGUI, which is included in the OpenCV package. On the http://opencv.org site, you can go to the tutorial pages off of the documentation links at http://docs.opencv.org/doc/tutorials/tutorials.html to see tutorials on various aspects of using OpenCV. In the tutorial section, the “introduction to OpenCV” tutorial explains how to set up OpenCV for various combinations of operating systems and development

of image file types, as well as from video and cameras. These utilities are part of a toolkit called HighGUI, which is included in the OpenCV package. On the http://opencv.org site, you can go to the tutorial pages off of the documentation links at http://docs.opencv.org/doc/tutorials/tutorials.html to see tutorials on various aspects of using OpenCV. In the tutorial section, the “introduction to OpenCV” tutorial explains how to set up OpenCV for various combinations of operating systems and development tools. We will use an example from the “highgui module” to create a simple program that opens an image and displays it on the screen (Example 2-1). Example 2-1: A simple OpenCV program that loads an image from disk and displays it on the screen #include <opencv2/opencv.hpp> //Include file for every supported OpenCV function  int main( int argc, char** argv ) {   cv::Mat img = cv::imread(argv[1],-1);   if( img.empty() ) return -1;   cv::namedWindow( "Example1", cv::WINDOW_AUTOSIZE );   cv::imshow( "Example1",

We will use an example from the “highgui module” to create a simple program that opens an image and displays it on the screen (Example 2-1). Example 2-1: A simple OpenCV program that loads an image from disk and displays it on the screen #include <opencv2/opencv.hpp> //Include file for every supported OpenCV function  int main( int argc, char** argv ) {   cv::Mat img = cv::imread(argv[1],-1);   if( img.empty() ) return -1;   cv::namedWindow( "Example1", cv::WINDOW_AUTOSIZE );   cv::imshow( "Example1", img );   cv::waitKey( 0 );   cv::destroyWindow( "Example1" ); } Note that OpenCV functions live within a namespace called cv.  To call OpenCV functions, you must explicitly tell the compiler that you are talking about the cv namespace by prepending cv:: to each function

call. To get out of this bookkeeping chore, we can employ the using namespace cv; directive as shown in Example 2-21. This tells the compiler to assume that functions might belong to that namespace. Note also the difference in include files between Example 2-1 and Example 2-2; in the former, we used the general include opencv.hpp, whereas in the latter, we used only the necessary include file to improve compile time. Example 2-2: Same as Example 2-1 but employing the “using namespace” directive. For faster compile, we use only the needed header file, not the generic opencv.hpp. #include "opencv2/highgui/highgui.hpp"  using namespace cv;  int main( int argc, char** argv ) {   Mat img = imread( argv[1], -1 );   if( img.empty() ) return -1;   namedWindow( "Example2", WINDOW_AUTOSIZE );   imshow( "Example2", img );   waitKey( 0 );   destroyWindow( "Example2" ); } When compiled and run from the command line with a single argument, Example 2-1 loads an image into memory and displays it on the screen. It then waits

compile, we use only the needed header file, not the generic opencv.hpp. #include "opencv2/highgui/highgui.hpp"  using namespace cv;  int main( int argc, char** argv ) {   Mat img = imread( argv[1], -1 );   if( img.empty() ) return -1;   namedWindow( "Example2", WINDOW_AUTOSIZE );   imshow( "Example2", img );   waitKey( 0 );   destroyWindow( "Example2" ); } When compiled and run from the command line with a single argument, Example 2-1 loads an image into memory and displays it on the screen. It then waits until the user presses a key, at which time it closes the window and exits. Let’s go through the program line by line and take a moment to understand what each command is doing. cv::Mat img = cv::imread( argv[1], -1 ); This line loads the image.2 The function cv::imread() is a high-level routine that determines the file format to be loaded based on the file name; it also automatically allocates the memory needed for the image data structure. Note that cv::imread() can read a wide variety of image formats,

until the user presses a key, at which time it closes the window and exits. Let’s go through the program line by line and take a moment to understand what each command is doing. cv::Mat img = cv::imread( argv[1], -1 ); This line loads the image.2 The function cv::imread() is a high-level routine that determines the file format to be loaded based on the file name; it also automatically allocates the memory needed for the image data structure. Note that cv::imread() can read a wide variety of image formats, including BMP, DIB, JPEG, JPE, PNG, PBM, PGM, PPM, SR, RAS, and TIFF. A cv::Mat structure is returned. This structure is the OpenCV construct with which you will deal the most. OpenCV uses this structure to handle all kinds of images: single-channel, multichannel, integer-valued, floating-point-valued, and so on.  The line immediately following if( img.empty() ) return -1; checks to see if an image was in fact read. Another high-level function, cv::namedWindow(), opens a window on the screen that can

including BMP, DIB, JPEG, JPE, PNG, PBM, PGM, PPM, SR, RAS, and TIFF. A cv::Mat structure is returned. This structure is the OpenCV construct with which you will deal the most. OpenCV uses this structure to handle all kinds of images: single-channel, multichannel, integer-valued, floating-point-valued, and so on.  The line immediately following if( img.empty() ) return -1; checks to see if an image was in fact read. Another high-level function, cv::namedWindow(), opens a window on the screen that can contain and display an image. cv::namedWindow( "Example2", cv::WINDOW_AUTOSIZE ); This function, provided by the HighGUI library, also assigns a name to the window (in this case, "Example2"). Future HighGUI calls that interact with this window will refer to it by this name. The second argument to cv::namedWindow() defines window properties. It may be set either to 0 (the default value) or to cv::WINDOW_AUTOSIZE. In the former case, the size of the window will be the same regardless of the image size, and the

and display an image. cv::namedWindow( "Example2", cv::WINDOW_AUTOSIZE ); This function, provided by the HighGUI library, also assigns a name to the window (in this case, "Example2"). Future HighGUI calls that interact with this window will refer to it by this name. The second argument to cv::namedWindow() defines window properties. It may be set either to 0 (the default value) or to cv::WINDOW_AUTOSIZE. In the former case, the size of the window will be the same regardless of the image size, and the image will be scaled to fit within the window. In the latter case, the window will expand or contract automatically when an image is loaded so as to accommodate the image’s true size but may be resized by the user.                                                              1 Of course, once you do this, you risk conflicting names with other potential namespaces. If the function foo() exists, say, in the cv and std namespaces, you must specify which function you are talking about using either cv::foo() or

image will be scaled to fit within the window. In the latter case, the window will expand or contract automatically when an image is loaded so as to accommodate the image’s true size but may be resized by the user.                                                              1 Of course, once you do this, you risk conflicting names with other potential namespaces. If the function foo() exists, say, in the cv and std namespaces, you must specify which function you are talking about using either cv::foo() or std::foo() as you intend. In this book, other than in our specific example of Example 2-2, we will use the explicit form cv:: for objects in the OpenCV namespace, as this is generally considered to be better programming style.   2 A proper program would check for the existence of argv[1] and, in its absence, deliver an instructional error message for the user. We will abbreviate such necessities in this book and assume that the reader is cultured enough to understand the importance of error-handling code.

cv::imshow( "Example2", img ); Whenever we have an image in a cv::Mat structure, we can display it in an existing window with cv::imshow().3 On the call to cv::imshow(), the window will be redrawn with the appropriate image in it, and the window will resize itself as appropriate if it was created using the cv::WINDOW_AUTOSIZE flag. cv::waitKey( 0 ); The cv::waitKey() function asks the program to stop and wait for a keystroke. If a positive argument is given, the program will wait for that number of milliseconds and then continue even if nothing is pressed. If the argument is set to 0 or to a negative number, the program will wait indefinitely for a key-press. This function has another very important role: It handles any windowing events, such as creating windows and drawing their content. So it must be used after cv::imshow() in order to display that image. With cv::Mat, images are automatically deallocated when they go out of scope, similar to the STL-style container classes. This automatic deallocation is

and then continue even if nothing is pressed. If the argument is set to 0 or to a negative number, the program will wait indefinitely for a key-press. This function has another very important role: It handles any windowing events, such as creating windows and drawing their content. So it must be used after cv::imshow() in order to display that image. With cv::Mat, images are automatically deallocated when they go out of scope, similar to the STL-style container classes. This automatic deallocation is controlled by an internal reference counter. For the most part, this means we no longer need to worry about the allocation and deallocation of images, which relieves the programmer from much of the tedious bookkeeping that the OpenCV 1.0 IplImage imposed. cv::destroyWindow( "Example2" ); Finally, we can destroy the window itself. The function cv::destroyWindow() will close the window and deallocate any associated memory usage. For short programs, we will skip this step. For longer, complex programs, the

is controlled by an internal reference counter. For the most part, this means we no longer need to worry about the allocation and deallocation of images, which relieves the programmer from much of the tedious bookkeeping that the OpenCV 1.0 IplImage imposed. cv::destroyWindow( "Example2" ); Finally, we can destroy the window itself. The function cv::destroyWindow() will close the window and deallocate any associated memory usage. For short programs, we will skip this step. For longer, complex programs, the programmer should make sure to tidy up the windows before they go out of scope to avoid memory leaks. Our next task will be to construct a very simple—almost as simple as this one—program to read in and display a video file. After that, we will start to tinker a little more with the actual images. Second Program—Video Playing a video with OpenCV is almost as easy as displaying a single picture. The only new issue we face is that we need some kind of loop to read each frame in sequence; we may also need

the programmer should make sure to tidy up the windows before they go out of scope to avoid memory leaks. Our next task will be to construct a very simple—almost as simple as this one—program to read in and display a video file. After that, we will start to tinker a little more with the actual images. Second Program—Video Playing a video with OpenCV is almost as easy as displaying a single picture. The only new issue we face is that we need some kind of loop to read each frame in sequence; we may also need some way to get out of that loop if the movie is too boring. See Example 2-3. Example 2-3: A simple OpenCV program for playing a video file from disk. In this example we only use specific module headers, rather than just opencv.hpp. This speeds up compilation, and so is sometimes preferable. #include "opencv2/highgui/highgui.hpp" #include "opencv2/imgproc/imgproc.hpp"  int main( int argc, char** argv ) {    cv::namedWindow( "Example3", cv::WINDOW_AUTOSIZE );   cv::VideoCapture cap;   cap.open(

also need some way to get out of that loop if the movie is too boring. See Example 2-3. Example 2-3: A simple OpenCV program for playing a video file from disk. In this example we only use specific module headers, rather than just opencv.hpp. This speeds up compilation, and so is sometimes preferable. #include "opencv2/highgui/highgui.hpp" #include "opencv2/imgproc/imgproc.hpp"  int main( int argc, char** argv ) {    cv::namedWindow( "Example3", cv::WINDOW_AUTOSIZE );   cv::VideoCapture cap;   cap.open( string(argv[1]) );   cv::Mat frame;   while( 1 ) {    cap >> frame;   if( !frame.data ) break;             // Ran out of film     cv::imshow( "Example3", frame );     if( cv::waitKey(33) >= 0 ) break;   }   return 0; }                                                               3 In the case where there is no window in existence at the time you call imshow(), one will be created for you with the name you specified in the imshow() call. This window can still be destroyed as usual with destroyWindow().

Here we begin the function main() with the usual creation of a named window (in this case, named “Example3”). The video capture object cv::VideoCapture is then instantiated. This object can open and close video files of as many types as ffmpeg supports.   cap.open(string(argv[1])); cv::Mat frame; The capture object is given a string containing the path and filename of the video to be opened. Once opened, the capture object will contain all of the information about the video file being read, including state information. When created in this way, the cv::VideoCapture object is initialized to the beginning of the video. In the program, cv::Mat frame instantiates a data object to hold video frames. cap >> frame;  if( !frame.data ) break; cv::imshow( "Example3", frame ); Once inside of the while() loop, the video file is read frame by frame from the capture object stream. The program checks to see if data was actually read from the video file (if(!frame.data)) and quits if not. If a video frame is successfully

information. When created in this way, the cv::VideoCapture object is initialized to the beginning of the video. In the program, cv::Mat frame instantiates a data object to hold video frames. cap >> frame;  if( !frame.data ) break; cv::imshow( "Example3", frame ); Once inside of the while() loop, the video file is read frame by frame from the capture object stream. The program checks to see if data was actually read from the video file (if(!frame.data)) and quits if not. If a video frame is successfully read in, it is displayed using cv::imshow(). if( cv::waitKey(33) >= 0 ) break; Once we have displayed the frame, we then wait for 33 ms4. If the user hits a key during that time then we will exit the read loop. Otherwise, 33 ms will pass and we will just execute the loop again. On exit, all the allocated data is automatically released when they go out of scope. Moving Around Now it’s time to tinker around, enhance our toy programs, and explore a little more of the available functionality. The first thing we

in, it is displayed using cv::imshow(). if( cv::waitKey(33) >= 0 ) break; Once we have displayed the frame, we then wait for 33 ms4. If the user hits a key during that time then we will exit the read loop. Otherwise, 33 ms will pass and we will just execute the loop again. On exit, all the allocated data is automatically released when they go out of scope. Moving Around Now it’s time to tinker around, enhance our toy programs, and explore a little more of the available functionality. The first thing we might notice about the video player of Example 2-3 is that it has no way to move around quickly within the video. Our next task will be to add a slider trackbar, which will give us this ability.  For more control, we will also allow the user to single step the video by pressing the ‘s’ key, to go into run mode by pressing the ‘r’ key,  and whenever the user jumps to a new location in the video with the slider bar, we pause there in single step mode.  The HighGUI toolkit provides a number of simple instruments

notice about the video player of Example 2-3 is that it has no way to move around quickly within the video. Our next task will be to add a slider trackbar, which will give us this ability.  For more control, we will also allow the user to single step the video by pressing the ‘s’ key, to go into run mode by pressing the ‘r’ key,  and whenever the user jumps to a new location in the video with the slider bar, we pause there in single step mode.  The HighGUI toolkit provides a number of simple instruments for working with images and video beyond the simple display functions we have just demonstrated. One especially useful mechanism is the trackbar, which enables us to jump easily from one part of a video to another. To create a trackbar, we call cv::createTrackbar() and indicate which window we would like the trackbar to appear in. In order to obtain the desired functionality, we need a callback that will perform the relocation. Example 2-4 gives the details. Example 2-4: Program to add a trackbar slider to

for working with images and video beyond the simple display functions we have just demonstrated. One especially useful mechanism is the trackbar, which enables us to jump easily from one part of a video to another. To create a trackbar, we call cv::createTrackbar() and indicate which window we would like the trackbar to appear in. In order to obtain the desired functionality, we need a callback that will perform the relocation. Example 2-4 gives the details. Example 2-4: Program to add a trackbar slider to the basic viewer window for moving around within the video file #include "opencv2/highgui/highgui.hpp" #include "opencv2/imgproc/imgproc.hpp" #include <iostream> #include <fstream>  using namespace std;  int g_slider_position = 0; int g_run = 1, g_dontset = 0; //start out in single step mode cv::VideoCapture g_cap;                                                              4 You can wait any amount of time you like. In this case, we are simply assuming that it is correct to play the video at 30 frames

to the basic viewer window for moving around within the video file #include "opencv2/highgui/highgui.hpp" #include "opencv2/imgproc/imgproc.hpp" #include <iostream> #include <fstream>  using namespace std;  int g_slider_position = 0; int g_run = 1, g_dontset = 0; //start out in single step mode cv::VideoCapture g_cap;                                                              4 You can wait any amount of time you like. In this case, we are simply assuming that it is correct to play the video at 30 frames per second and allow user input to interrupt between each frame (thus we pause for input 33 ms between each frame). In practice, it is better to check the cv::VideoCapture structure in order to determine the actual frame rate of the video (more on this in Chapter 4).

void onTrackbarSlide( int pos, void *) {   g_cap.set( cv::CAP_PROP_POS_FRAMES, pos );   if( !g_dontset )     g_run = 1;   g_dontset = 0; }  int main( int argc, char** argv ) {   cv::namedWindow( "Example2_4", cv::WINDOW_AUTOSIZE );   g_cap.open( string(argv[1]) );   int frames = (int) g_cap.get(cv::CAP_PROP_FRAME_COUNT);   int tmpw   = (int) g_cap.get(cv::CAP_PROP_FRAME_WIDTH);   int tmph   = (int) g_cap.get(cv::CAP_PROP_FRAME_HEIGHT);   cout << "Video has " << frames << " frames of dimensions("         << tmpw << ", " << tmph << ")." << endl;   cv::createTrackbar("Position", "Example2_4", &g_slider_position, frames,                  onTrackbarSlide);   cv::Mat frame;   while(1) {     if( g_run != 0 ) {       g_cap >> frame; if(!frame.data) break;       int current_pos = (int)g_cap.get(cv::CAP_PROP_POS_FRAMES);       g_dontset = 1;       cv::setTrackbarPos("Position", "Example2_4", current_pos);       cv::imshow( "Example2_4", frame );       g_run-=1;     }     char c = (char) cv::waitKey(10);     if(c ==

<< tmpw << ", " << tmph << ")." << endl;   cv::createTrackbar("Position", "Example2_4", &g_slider_position, frames,                  onTrackbarSlide);   cv::Mat frame;   while(1) {     if( g_run != 0 ) {       g_cap >> frame; if(!frame.data) break;       int current_pos = (int)g_cap.get(cv::CAP_PROP_POS_FRAMES);       g_dontset = 1;       cv::setTrackbarPos("Position", "Example2_4", current_pos);       cv::imshow( "Example2_4", frame );       g_run-=1;     }     char c = (char) cv::waitKey(10);     if(c == 's') // single step       {g_run = 1; cout << "Single step, run = " << g_run << endl;}     if(c == 'r') // run mode       {g_run = -1; cout << "Run mode, run = " << g_run <<endl;}     if( c == 27 )       break;   }   return(0); }  In essence, the strategy is to add a global variable to represent the trackbar position and then add a callback that updates this variable and relocates the read position in the video. One call creates the trackbar and attaches the callback, and we are off and running.5 Let’s

== 's') // single step       {g_run = 1; cout << "Single step, run = " << g_run << endl;}     if(c == 'r') // run mode       {g_run = -1; cout << "Run mode, run = " << g_run <<endl;}     if( c == 27 )       break;   }   return(0); }  In essence, the strategy is to add a global variable to represent the trackbar position and then add a callback that updates this variable and relocates the read position in the video. One call creates the trackbar and attaches the callback, and we are off and running.5 Let’s look at the details starting with the global variables. int g_slider_position = 0; int g_run = 1, g_dontset = 0;      // start out in single step mode VideoCapture g_cap; First we define a global variable, g_slider_position, to keep the trackbar slider position state. The callback will need access to the capture object g_cap, so we promote that to a global variable as well. Because we are considerate developers and like our code to be readable and easy to understand, we adopt the convention of adding a

look at the details starting with the global variables. int g_slider_position = 0; int g_run = 1, g_dontset = 0;      // start out in single step mode VideoCapture g_cap; First we define a global variable, g_slider_position, to keep the trackbar slider position state. The callback will need access to the capture object g_cap, so we promote that to a global variable as well. Because we are considerate developers and like our code to be readable and easy to understand, we adopt the convention of adding a leading g_ to any global variable. We also instantiate another global variable, g_run, which displays new frames as long it is different from zero. A positive number tells how many frames are displayed before stopping; a negative number means the system runs in continuous video mode.  To avoid confusion, when the user clicks on the trackbar to jump to a new location in the video, we’ll leave the video paused there in the single step state by setting g_run = 1. This, however, brings up a subtle problem: as the

g_ to any global variable. We also instantiate another global variable, g_run, which displays new frames as long it is different from zero. A positive number tells how many frames are displayed before stopping; a negative number means the system runs in continuous video mode.  To avoid confusion, when the user clicks on the trackbar to jump to a new location in the video, we’ll leave the video paused there in the single step state by setting g_run = 1. This, however, brings up a subtle problem: as the video advances, we’d like the slider trackbar position in the display window to advance according to our location in the video. We do this by having the main program call the trackbar callback                                                              5 Note that some AVI and mpeg encodings do not allow you to move backward in the video.

function to update the slider position each time we get a new video frame. However, we don’t want these programmatic calls to the trackbar callback to put us into single step mode. To avoid this, we introduce a final global variable, g_dontset to allow us to update trackbar position without triggering single state mode.   void onTrackbarSlide(int pos, void *) {   g_cap.set(cv::CAP_PROP_POS_FRAMES, pos);   if(!g_dontset)     g_run = 1;   g_dontset = 0; } Now we define a callback routine to be used when the user pokes the trackbar. This routine will be passed a 32-bit integer, pos, which will be the new trackbar position. Inside this callback, we use the new requested position in cv::g_cap.set() to actually advance the video playback to the new position.  The if statement just sets the program to go into single step mode after the next new frame comes in, but only if the callback was triggered by a user click, not if the callback was called from the main function (which sets g_dontset). The call to

when the user pokes the trackbar. This routine will be passed a 32-bit integer, pos, which will be the new trackbar position. Inside this callback, we use the new requested position in cv::g_cap.set() to actually advance the video playback to the new position.  The if statement just sets the program to go into single step mode after the next new frame comes in, but only if the callback was triggered by a user click, not if the callback was called from the main function (which sets g_dontset). The call to cv::g_cap.set() is one we will see often in the future, along with its counterpart cv::g_cap.get(). These routines allow us to configure (or query in the latter case) various properties of the cv::VideoCapture object. In this case, we pass the argument cv::CAP_PROP_POS_FRAMES, which indicates that we would like to set the read position in units of frames.6     int frames = (int) g_cap.get(cv::CAP_PROP_FRAME_COUNT);   int tmpw   = (int) g_cap.get(cv::CAP_PROP_FRAME_WIDTH);   int tmph   = (int)

The call to cv::g_cap.set() is one we will see often in the future, along with its counterpart cv::g_cap.get(). These routines allow us to configure (or query in the latter case) various properties of the cv::VideoCapture object. In this case, we pass the argument cv::CAP_PROP_POS_FRAMES, which indicates that we would like to set the read position in units of frames.6     int frames = (int) g_cap.get(cv::CAP_PROP_FRAME_COUNT);   int tmpw   = (int) g_cap.get(cv::CAP_PROP_FRAME_WIDTH);   int tmph   = (int) g_cap.get(cv::CAP_PROP_FRAME_HEIGHT);   cout << "Video has " << frames << " frames of dimensions("         << tmpw << ", " << tmph << ")." << endl; The core of the main program is the same as in Example 2-3, so we’ll focus on what we’ve added. The first difference after opening the video is that we use cv::g_cap.get() to determine the number of frames in the video and the width and height of the video images. These numbers are printed out. We’ll need the number of frames in the video to calibrate the slider

cout << "Video has " << frames << " frames of dimensions("         << tmpw << ", " << tmph << ")." << endl; The core of the main program is the same as in Example 2-3, so we’ll focus on what we’ve added. The first difference after opening the video is that we use cv::g_cap.get() to determine the number of frames in the video and the width and height of the video images. These numbers are printed out. We’ll need the number of frames in the video to calibrate the slider (in the next step).   createTrackbar("Position", "Example2_4", &g_slider_position, frames,                  onTrackbarSlide); Next we create the trackbar itself. The function cv::createTrackbar() allows us to give the trackbar a label7 (in this case, Position) and to specify a window to put the trackbar in. We then provide a variable that will be bound to the trackbar, the maximum value of the trackbar (the number of frames in the video), and a callback (or NULL if we don’t want one) for when the slider is moved.      if( g_run != 0 ) {

"Example2_4", &g_slider_position, frames,                  onTrackbarSlide); Next we create the trackbar itself. The function cv::createTrackbar() allows us to give the trackbar a label7 (in this case, Position) and to specify a window to put the trackbar in. We then provide a variable that will be bound to the trackbar, the maximum value of the trackbar (the number of frames in the video), and a callback (or NULL if we don’t want one) for when the slider is moved.      if( g_run != 0 ) {       g_cap >> frame; if(!frame.data) break;       int current_pos = (int)g_cap.get(cv::CAP_PROP_POS_FRAMES);       g_dontset = 1;       cv::setTrackbarPos("Position", "Example2_4", current_pos);       cv::imshow( "Example2_4", frame );       g_run-=1;     }                                                              6 Because HighGUI is highly civilized, when a new video position is requested, it will automatically handle such issues as the possibility that the frame we have requested is not a key-frame; it will start at

if(!frame.data) break;       int current_pos = (int)g_cap.get(cv::CAP_PROP_POS_FRAMES);       g_dontset = 1;       cv::setTrackbarPos("Position", "Example2_4", current_pos);       cv::imshow( "Example2_4", frame );       g_run-=1;     }                                                              6 Because HighGUI is highly civilized, when a new video position is requested, it will automatically handle such issues as the possibility that the frame we have requested is not a key-frame; it will start at the previous key-frame and fast forward up to the requested frame without us having to fuss with such details. 7 Because HighGUI is a lightweight, easy-to-use toolkit, cv::createTrackbar() does not distinguish between the name of the trackbar and the label that actually appears on the screen next to the trackbar. You may already have noticed that cv::namedWindow() likewise does not distinguish between the name of the window and the label that appears on the window in the GUI.

In the while loop, in addition to reading and displaying the video frame, we also get our current position in the video, set the g_dontset so that the next trackbar callback will not put us into single step mode, and then invoke the trackbar callback to update the position of the slider displayed to the user. The global g_run is decremented, which has the effect of either keeping us in single step mode or of letting the video run depending on its prior state set by user interaction via keyboard, as we’ll see next.     char c = (char) cv::waitKey(10);     if(c == 's') // single step       {g_run = 1; cout << "Single step, run = " << g_run << endl;}     if(c == 'r') // run mode       {g_run = -1; cout << "Run mode, run = " << g_run <<endl;}     if( c == 27 )       break; At the bottom of the while loop, we look for keyboard input from the user.  If ‘s’ has been pressed, we go into single step mode (g_run is set to 1 which allows reading of a single frame above). If ‘r’ is pressed, we go into continuous video

see next.     char c = (char) cv::waitKey(10);     if(c == 's') // single step       {g_run = 1; cout << "Single step, run = " << g_run << endl;}     if(c == 'r') // run mode       {g_run = -1; cout << "Run mode, run = " << g_run <<endl;}     if( c == 27 )       break; At the bottom of the while loop, we look for keyboard input from the user.  If ‘s’ has been pressed, we go into single step mode (g_run is set to 1 which allows reading of a single frame above). If ‘r’ is pressed, we go into continuous video mode (g_run is set to -1 and further decrementing leaves it negative for any conceivable sized video). Finally, if ESC is pressed, the program will terminate. Note again, for short programs, we’ve omitted the step of cleaning up the window storage using cv::destroyWindow(). A Simple Transformation Great! Now you can use OpenCV to create your own video player, which will not be much different from countless video players out there already. But we are interested in computer vision, and we want to do some of

mode (g_run is set to -1 and further decrementing leaves it negative for any conceivable sized video). Finally, if ESC is pressed, the program will terminate. Note again, for short programs, we’ve omitted the step of cleaning up the window storage using cv::destroyWindow(). A Simple Transformation Great! Now you can use OpenCV to create your own video player, which will not be much different from countless video players out there already. But we are interested in computer vision, and we want to do some of that. Many basic vision tasks involve the application of filters to a video stream. We will modify the program we already have to do a simple operation on every frame of the video as it plays. One particularly simple operation is the smoothing of an image, which effectively reduces the information content of the image by convolving it with a Gaussian or other similar kernel function. OpenCV makes such convolutions exceptionally easy to do. We can start by creating a new window called "Example4-out", where

that. Many basic vision tasks involve the application of filters to a video stream. We will modify the program we already have to do a simple operation on every frame of the video as it plays. One particularly simple operation is the smoothing of an image, which effectively reduces the information content of the image by convolving it with a Gaussian or other similar kernel function. OpenCV makes such convolutions exceptionally easy to do. We can start by creating a new window called "Example4-out", where we can display the results of the processing. Then, after we have called cv::showImage() to display the newly captured frame in the input window, we can compute and display the smoothed image in the output window. See Example 2-5. Example 2-5: Loading and then smoothing an image before it is displayed on the screen #include <opencv2/opencv.hpp>  void example2_5( cv::Mat & image ) {   // Create some windows to show the input   // and output images in.   //   cv::namedWindow( "Example2_5-in",

where we can display the results of the processing. Then, after we have called cv::showImage() to display the newly captured frame in the input window, we can compute and display the smoothed image in the output window. See Example 2-5. Example 2-5: Loading and then smoothing an image before it is displayed on the screen #include <opencv2/opencv.hpp>  void example2_5( cv::Mat & image ) {   // Create some windows to show the input   // and output images in.   //   cv::namedWindow( "Example2_5-in", cv::WINDOW_AUTOSIZE );   cv::namedWindow( "Example2_5-out", cv::WINDOW_AUTOSIZE );       // Create a window to show our input image   //   cv::imshow( "Example2_5-in", image );        // Create an image to hold the smoothed output   cv::Mat out;        // Do the smoothing   //  Could use GaussianBlur(), blur(), medianBlur() or bilateralFilter().   cv::GaussianBlur( image, out, cv::Size(5,5), 3, 3);   cv::GaussianBlur(   out, out, cv::Size(5,5), 3, 3);        // Show the smoothed image in the output window   //

);   cv::namedWindow( "Example2_5-out", cv::WINDOW_AUTOSIZE );       // Create a window to show our input image   //   cv::imshow( "Example2_5-in", image );        // Create an image to hold the smoothed output   cv::Mat out;        // Do the smoothing   //  Could use GaussianBlur(), blur(), medianBlur() or bilateralFilter().   cv::GaussianBlur( image, out, cv::Size(5,5), 3, 3);   cv::GaussianBlur(   out, out, cv::Size(5,5), 3, 3);        // Show the smoothed image in the output window   //   cv::imshow( "Example2_5-out", out );

// Wait for the user to hit a key, windows will self destruct   //   cv::waitKey( 0 ); } The first call to cv::showImage() is no different than in our previous example. In the next call, we allocate another image structure. Next, the C++ object cv::Mat makes life simpler for us; we just instantiate an output matrix “out” and it will automatically resize/reallocate and deallocate itself as necessary as it is used. To make this point clear, we use it in two consecutive calls to cv::GaussianBlur(). In the first call, the input images is blurred by a 5 × 5 Gaussian convolution filter and written to out. The size of the Gaussian kernel should always be given as odd numbers since the Gaussian kernel (specified here by cv::Size(5,5)) is computed at the center pixel in that area. In the next call to cv::GaussianBlur(), out is used as both the input and output since temporary storage is assigned for us in this case.  The resulting double-blurred image is displayed and the routine then waits for any user keyboard

first call, the input images is blurred by a 5 × 5 Gaussian convolution filter and written to out. The size of the Gaussian kernel should always be given as odd numbers since the Gaussian kernel (specified here by cv::Size(5,5)) is computed at the center pixel in that area. In the next call to cv::GaussianBlur(), out is used as both the input and output since temporary storage is assigned for us in this case.  The resulting double-blurred image is displayed and the routine then waits for any user keyboard input before terminating and cleaning up allocated data as it goes out of scope. A Not-So-Simple Transformation That was pretty good, and we are learning to do more interesting things. In Example 2-5, we used Gaussian blurring for no particular purpose. We will now use a function that uses Gaussian blurring to downsample it by a factor of 2 [Rosenfeld80]. If we downsample the image several times, we form a scale space, or image pyramid that is commonly used in computer vision to handle the changing scales

input before terminating and cleaning up allocated data as it goes out of scope. A Not-So-Simple Transformation That was pretty good, and we are learning to do more interesting things. In Example 2-5, we used Gaussian blurring for no particular purpose. We will now use a function that uses Gaussian blurring to downsample it by a factor of 2 [Rosenfeld80]. If we downsample the image several times, we form a scale space, or image pyramid that is commonly used in computer vision to handle the changing scales in which a scene or object is observed.  For those who know some signal processing and the Nyquist-Shannon Sampling Theorem [Shannon49], when you downsample a signal (in this case, create an image where we are sampling every other pixel), it is equivalent to convolving with a series of delta functions (think of these as “spikes”). Such sampling introduces high frequencies into the resulting signal (image). To avoid this, we want to first run a high-pass filter over the signal first to band limit its

scales in which a scene or object is observed.  For those who know some signal processing and the Nyquist-Shannon Sampling Theorem [Shannon49], when you downsample a signal (in this case, create an image where we are sampling every other pixel), it is equivalent to convolving with a series of delta functions (think of these as “spikes”). Such sampling introduces high frequencies into the resulting signal (image). To avoid this, we want to first run a high-pass filter over the signal first to band limit its frequencies so that they are all below the sampling frequency.  In OpenCV, this Gaussian blurring and downsampling is accomplished by the function cv::pyrDown(). We use this very useful function in Example 2-6. Example 2-6: Using cv::pyrDown() to create a new image that is half the width and height of the input image #include <opencv2/opencv.hpp>  int main( int argc, char** argv ) {   cv::Mat img = cv::imread( argv[1] ),img2;   cv::namedWindow( "Example1", cv::WINDOW_AUTOSIZE );   cv::namedWindow(

its frequencies so that they are all below the sampling frequency.  In OpenCV, this Gaussian blurring and downsampling is accomplished by the function cv::pyrDown(). We use this very useful function in Example 2-6. Example 2-6: Using cv::pyrDown() to create a new image that is half the width and height of the input image #include <opencv2/opencv.hpp>  int main( int argc, char** argv ) {   cv::Mat img = cv::imread( argv[1] ),img2;   cv::namedWindow( "Example1", cv::WINDOW_AUTOSIZE );   cv::namedWindow( "Example2", cv::WINDOW_AUTOSIZE );   cv::imshow( "Example1", img );   cv::pyrDown( img, img2);   cv::imshow( "Example2", img2 );   cv::waitKey(0);   return 0; }; Let’s now look at a similar but slightly more involved example involving the Canny edge detector [Canny86] cv::Canny() (see Example 2-7). In this case, the edge detector generates an image that is the full size of the input image but needs only a single-channel image to write to and so we convert to a gray scale, single-channel image first using

"Example2", cv::WINDOW_AUTOSIZE );   cv::imshow( "Example1", img );   cv::pyrDown( img, img2);   cv::imshow( "Example2", img2 );   cv::waitKey(0);   return 0; }; Let’s now look at a similar but slightly more involved example involving the Canny edge detector [Canny86] cv::Canny() (see Example 2-7). In this case, the edge detector generates an image that is the full size of the input image but needs only a single-channel image to write to and so we convert to a gray scale, single-channel image first using cv::cvtColor() with the flag to convert blue, green, red images to gray scale: cv::BGR2GRAY. Example 2-7: The Canny edge detector writes its output to a single-channel (grayscale) image #include <opencv2/opencv.hpp>  int main( int argc, char** argv ) {

cv::Mat img_rgb = cv::imread( argv[1] );  cv::Mat img_gry, img_cny;  cv::cvtColor( img_rgb, img_gry, cv::BGR2GRAY);  cv::namedWindow( "Example Gray", cv::WINDOW_AUTOSIZE );  cv::namedWindow( "Example Canny", cv::WINDOW_AUTOSIZE );  cv::imshow( "Example Gray", img_gry );  cv::Canny( img_gry, img_cny, 10, 100, 3, true );  cv::imshow( "Example Canny", img_cny );  cv::waitKey(0); } This allows us to string together various operators quite easily. For example, if we wanted to shrink the image twice and then look for lines that were present in the twice-reduced image, we could proceed as in Example 2-8. Example 2-8: Combining the pyramid down operator (twice) and the Canny subroutine in a simple image pipeline cv::cvtColor( img_rgb, img_gry, cv::BGR2GRAY ); cv::pyrDown( img_gry, img_pyr ); cv::pyrDown( img_pyr, img_pyr2 ); cv::Canny( img_pyr2, img_cny, 10, 100, 3, true ); // do whatever with 'img_cny' // ... In Example 2-9, we show a simple way to read and write pixel values from Example 2-8. Example 2-9: Getting

for lines that were present in the twice-reduced image, we could proceed as in Example 2-8. Example 2-8: Combining the pyramid down operator (twice) and the Canny subroutine in a simple image pipeline cv::cvtColor( img_rgb, img_gry, cv::BGR2GRAY ); cv::pyrDown( img_gry, img_pyr ); cv::pyrDown( img_pyr, img_pyr2 ); cv::Canny( img_pyr2, img_cny, 10, 100, 3, true ); // do whatever with 'img_cny' // ... In Example 2-9, we show a simple way to read and write pixel values from Example 2-8. Example 2-9: Getting and setting pixels in Example 2-8   int x = 16, y = 32;   cv::Vec3b intensity = img_rgb.at< cv::Vec3b >(y, x);   uchar blue  = intensity.val[0]; // We could write img_rgb.at< cv::Vec3b >(x,y)[0]   uchar green = intensity.val[1];   uchar red   = intensity.val[2];   std::cout << "At (x,y) = (" << x << ", " << y <<             "): (blue, green, red) = (" <<             (unsigned int)blue <<             ", " << (unsigned int)green << ", " <<             (unsigned int)red << ")" << std::endl;    std::cout <<

and setting pixels in Example 2-8   int x = 16, y = 32;   cv::Vec3b intensity = img_rgb.at< cv::Vec3b >(y, x);   uchar blue  = intensity.val[0]; // We could write img_rgb.at< cv::Vec3b >(x,y)[0]   uchar green = intensity.val[1];   uchar red   = intensity.val[2];   std::cout << "At (x,y) = (" << x << ", " << y <<             "): (blue, green, red) = (" <<             (unsigned int)blue <<             ", " << (unsigned int)green << ", " <<             (unsigned int)red << ")" << std::endl;    std::cout << "Gray pixel there is: " <<                 (unsigned int)img_gry.at<uchar>(x, y) << std::endl;    x /= 4; y /= 4;   std::cout << "Pyramid2 pixel there is: " <<                (unsigned int)img_pyr2.at<uchar>(x, y) << std::endl;    img_cny.at<uchar>(x, y) = 128; // Set the Canny pixel there to 128 Input from a Camera Vision can mean many things in the world of computers. In some cases, we are analyzing still frames loaded from elsewhere. In other cases, we are analyzing video that is being read from disk. In

pixel there is: " <<                 (unsigned int)img_gry.at<uchar>(x, y) << std::endl;    x /= 4; y /= 4;   std::cout << "Pyramid2 pixel there is: " <<                (unsigned int)img_pyr2.at<uchar>(x, y) << std::endl;    img_cny.at<uchar>(x, y) = 128; // Set the Canny pixel there to 128 Input from a Camera Vision can mean many things in the world of computers. In some cases, we are analyzing still frames loaded from elsewhere. In other cases, we are analyzing video that is being read from disk. In still other cases, we want to work with real-time data streaming in from some kind of camera device. OpenCV—more specifically, the HighGUI portion of the OpenCV library—provides us with an easy way to handle this situation. The method is analogous to how we read videos from disk since the cv::VideoCapture object works the same for files on disk or from camera. For the former, you give it a path/filename, and for the latter, you give it a camera ID number (typically “0” if only one camera is connected to the

other cases, we want to work with real-time data streaming in from some kind of camera device. OpenCV—more specifically, the HighGUI portion of the OpenCV library—provides us with an easy way to handle this situation. The method is analogous to how we read videos from disk since the cv::VideoCapture object works the same for files on disk or from camera. For the former, you give it a path/filename, and for the latter, you give it a camera ID number (typically “0” if only one camera is connected to the system).  The default value is –1, which means “just pick one”; naturally, this also works

quite well when there is only one camera to pick (see Chapter 4 for more details). Camera capture from file or from camera is demonstrated in Example 2-10. Example 2-10: The same object can load videos from a camera or a file #include <opencv2/opencv.hpp> #include <iostream>  int main( int argc, char** argv ) {    cv::namedWindow( "Example2_10", cv::WINDOW_AUTOSIZE );   cv::VideoCapture cap;   if (argc==1) {    cap.open(0);           // open the default camera   } else {     cap.open(argv[1]);   }   if( !cap.isOpened() ) {  // check if we succeeded     std::cerr << "Couldn't open capture." << std::endl;     return -1;   }   // The rest of program proceeds as in Example 2-3   …   In Example 2-10, if a filename is supplied, it opens that file just like in Example 2-3, and if no filename is given, it attempts to open camera zero (0).  We have added a check that something actually opened that will report an error if not. Writing to an AVI File In many applications, we will want to record streaming input or even

) {  // check if we succeeded     std::cerr << "Couldn't open capture." << std::endl;     return -1;   }   // The rest of program proceeds as in Example 2-3   …   In Example 2-10, if a filename is supplied, it opens that file just like in Example 2-3, and if no filename is given, it attempts to open camera zero (0).  We have added a check that something actually opened that will report an error if not. Writing to an AVI File In many applications, we will want to record streaming input or even disparate captured images to an output video stream, and OpenCV provides a straightforward method for doing this. Just as we are able to create a capture device that allows us to grab frames one at a time from a video stream, we are able to create a writer device that allows us to place frames one by one into a video file. The object that allows us to do this is cv::VideoWriter. Once this call has been made, we may stream each frame to the cv::VideoWriter object, and finally call its cv::VideoWriter.release() method

captured images to an output video stream, and OpenCV provides a straightforward method for doing this. Just as we are able to create a capture device that allows us to grab frames one at a time from a video stream, we are able to create a writer device that allows us to place frames one by one into a video file. The object that allows us to do this is cv::VideoWriter. Once this call has been made, we may stream each frame to the cv::VideoWriter object, and finally call its cv::VideoWriter.release() method when we are done. Just to make things more interesting, Example 2-11 describes a program that opens a video file, reads the contents, converts them to a log-polar format (something like what your eye actually sees, as described in Chapter 6), and writes out the log-polar image to a new video file. Example 2-11: A complete program to read in a color video and write out the log-polar transformed video #include <opencv2/opencv.hpp> #include <iostream>  int main( int argc, char* argv[] ) {   cv::namedWindow(

when we are done. Just to make things more interesting, Example 2-11 describes a program that opens a video file, reads the contents, converts them to a log-polar format (something like what your eye actually sees, as described in Chapter 6), and writes out the log-polar image to a new video file. Example 2-11: A complete program to read in a color video and write out the log-polar transformed video #include <opencv2/opencv.hpp> #include <iostream>  int main( int argc, char* argv[] ) {   cv::namedWindow( "Example2_10", cv::WINDOW_AUTOSIZE );   cv::namedWindow( "Log_Polar", cv::WINDOW_AUTOSIZE );   cv::VideoCapture capture;   double fps = capture.get( cv::CAP_PROP_FPS );   cv::Size size(     (int)capture.get( cv::CAP_PROP_FRAME_WIDTH ),     (int)capture.get( cv::CAP_PROP_FRAME_HEIGHT )   );   cv::VideoWriter writer;   writer.open( argv[2], CV_FOURCC('M','J','P','G'), fps, size );   cv::Mat logpolar_frame(size,CV::U8C3), bgr_frame;   for(;;) {     capture >> bgr_frame;     if( bgr_frame.empty() ) break; // end

cv::WINDOW_AUTOSIZE );   cv::namedWindow( "Log_Polar", cv::WINDOW_AUTOSIZE );   cv::VideoCapture capture;   double fps = capture.get( cv::CAP_PROP_FPS );   cv::Size size(     (int)capture.get( cv::CAP_PROP_FRAME_WIDTH ),     (int)capture.get( cv::CAP_PROP_FRAME_HEIGHT )   );   cv::VideoWriter writer;   writer.open( argv[2], CV_FOURCC('M','J','P','G'), fps, size );   cv::Mat logpolar_frame(size,CV::U8C3), bgr_frame;   for(;;) {     capture >> bgr_frame;     if( bgr_frame.empty() ) break; // end if done

cv::imshow( "Example2_10", bgr_frame );     cv::logPolar(        bgr_frame,                      // Input color frame       logpolar_frame,                 // Output log-polar frame       cv::Point2f(                    // Centerpoint for log-polar transformation         bgr_frame.cols/2,             //  x         bgr_frame.rows/2              //  y       ),       40,                             // Magnitude (scale parameter)       cv::WARP_FILL_OUTLIERS          // Fill outliers with ‘zero’     );     cv::imshow( "Log_Polar", logpolar_frame );     writer << logpolar_frame;     char c = cv::waitKey(10);     if( c == 27 ) break;        // allow the user to break out   }   capture.release(); } Looking over this program reveals mostly familiar elements. We open one video and read some properties (frames per second, image width and height) that we’ll need to open a file for the cv::VideoWriter object. We then read the video frame by frame from the cv::VideoReader object, convert the frame to log-polar format,

"Log_Polar", logpolar_frame );     writer << logpolar_frame;     char c = cv::waitKey(10);     if( c == 27 ) break;        // allow the user to break out   }   capture.release(); } Looking over this program reveals mostly familiar elements. We open one video and read some properties (frames per second, image width and height) that we’ll need to open a file for the cv::VideoWriter object. We then read the video frame by frame from the cv::VideoReader object, convert the frame to log-polar format, and write the log-polar frames to this new video file one at a time until there are none left or until the user quits by pressing ESC. Then we close up. The call to cv::VideoWriter object contains several parameters that we should understand. The first is just the filename for the new file. The second is the video codec with which the video stream will be compressed. There are countless such codecs in circulation, but whichever codec you choose must be available on your machine (codecs are installed separately from

the log-polar frames to this new video file one at a time until there are none left or until the user quits by pressing ESC. Then we close up. The call to cv::VideoWriter object contains several parameters that we should understand. The first is just the filename for the new file. The second is the video codec with which the video stream will be compressed. There are countless such codecs in circulation, but whichever codec you choose must be available on your machine (codecs are installed separately from OpenCV). In our case, we choose the relatively popular MJPG codec; this is indicated to OpenCV by using the macro CV_FOURCC(), which takes four characters as arguments. These characters constitute the “four-character code” of the codec, and every codec has such a code. The four-character code for motion jpeg is “MJPG,” so we specify that as CV_FOURCC('M','J','P','G'). The next two arguments are the replay frame rate and the size of the images we will be using. In our case, we set these to the values we got

OpenCV). In our case, we choose the relatively popular MJPG codec; this is indicated to OpenCV by using the macro CV_FOURCC(), which takes four characters as arguments. These characters constitute the “four-character code” of the codec, and every codec has such a code. The four-character code for motion jpeg is “MJPG,” so we specify that as CV_FOURCC('M','J','P','G'). The next two arguments are the replay frame rate and the size of the images we will be using. In our case, we set these to the values we got from the original (color) video. Summary Before moving on to the next chapter, we should take a moment to take stock of where we are and look ahead to what is coming. We have seen that the OpenCV API provides us with a variety of easy-to-use tools for reading and writing still images and videos from and to files along with capturing video from cameras. We have also seen that the library contains primitive functions for manipulating these images. What we have not yet seen are the powerful elements of the

from the original (color) video. Summary Before moving on to the next chapter, we should take a moment to take stock of where we are and look ahead to what is coming. We have seen that the OpenCV API provides us with a variety of easy-to-use tools for reading and writing still images and videos from and to files along with capturing video from cameras. We have also seen that the library contains primitive functions for manipulating these images. What we have not yet seen are the powerful elements of the library, which allow for more sophisticated manipulation of the entire set of abstract data types that are important to practical vision problem solving. In the next few chapters, we will delve more deeply into the basics and come to understand in greater detail both the interface-related functions and the image data types. We will investigate the primitive image manipulation operators and, later, some much more advanced ones. Thereafter, we will be ready to explore the many specialized services that the API

which allow for more sophisticated manipulation of the entire set of abstract data types that are important to practical vision problem solving. In the next few chapters, we will delve more deeply into the basics and come to understand in greater detail both the interface-related functions and the image data types. We will investigate the primitive image manipulation operators and, later, some much more advanced ones. Thereafter, we will be ready to explore the many specialized services that the API provides for tasks as diverse as camera calibration, tracking, and recognition. Ready? Let’s go! Exercises Download and install OpenCV if you have not already done so. Systematically go through the directory structure. Note in particular the docs directory, where you can load index.htm, which further links to the main documentation of the library. Further explore the main areas of the library. The core module contains the basic data structures and algorithms, imgproc contains the image processing and vision

for tasks as diverse as camera calibration, tracking, and recognition. Ready? Let’s go! Exercises Download and install OpenCV if you have not already done so. Systematically go through the directory structure. Note in particular the docs directory, where you can load index.htm, which further links to the main documentation of the library. Further explore the main areas of the library. The core module contains the basic data structures and algorithms, imgproc contains the image processing and vision algorithms, ml

includes algorithms for machine learning and clustering, and highgui contains the I/O functions. Check out the …/samples/cpp directory, where many useful examples are stored. 1. Using the install and build instructions in this book or on the website http://opencv.org, build the library in both the debug and the release versions. This may take some time, but you will need the resulting library and dll files. Make sure you set the cmake file to build the samples …/opencv/samples/ directory. 2. Go to where you built the …/opencv/samples/ directory (the authors build in …/trunk/eclipse_build/bin) and look for lkdemo.c (this is an example motion tracking program). Attach a camera to your system and run the code. With the display window selected, type “r” to initialize tracking. You can add points by clicking on video positions with the mouse. You can also switch to watching only the points (and not the image) by typing “n.” Typing “n” again will toggle between “night” and “day” views. 3. Use the capture and store

built the …/opencv/samples/ directory (the authors build in …/trunk/eclipse_build/bin) and look for lkdemo.c (this is an example motion tracking program). Attach a camera to your system and run the code. With the display window selected, type “r” to initialize tracking. You can add points by clicking on video positions with the mouse. You can also switch to watching only the points (and not the image) by typing “n.” Typing “n” again will toggle between “night” and “day” views. 3. Use the capture and store code in Example 2-11 together with the cv::PyrDown() code of Example 2-6 to create a program that reads from a camera and stores down-sampled color images to disk. 4. Modify the code in Exercise 3 and combine it with the window display code in Example 2-2 to display the frames as they are processed. 5. Modify the program of Exercise 4 with a slider control from Example 2-4 so that the user can dynamically vary the pyramid downsampling reduction level by factors of between 2 and 8. You may skip writing this

code in Example 2-11 together with the cv::PyrDown() code of Example 2-6 to create a program that reads from a camera and stores down-sampled color images to disk. 4. Modify the code in Exercise 3 and combine it with the window display code in Example 2-2 to display the frames as they are processed. 5. Modify the program of Exercise 4 with a slider control from Example 2-4 so that the user can dynamically vary the pyramid downsampling reduction level by factors of between 2 and 8. You may skip writing this to disk, but you should display the results.

3 Getting to Know OpenCV OpenCV Data Types OpenCV has many data types, which are designed to make the representation and handling of important concepts of computer vision relatively easy and intuitive. At the same time, many algorithm developers require a set of relatively powerful primitives that can be generalized or extended for their particular needs. OpenCV attempts to address both of these needs through the use of templates for fundamental data types, and specializations of those templates that make everyday operations easier.  From an organizational perspective, it is convenient to divide the data types into three major categories. First, the basic data types are those that are assembled directly from C++ primitives (int, float, etc.). These types include simple vectors and matrices, as well as representations of simple geometric concepts like points, rectangles, sizes, and the like. The second category contains helper objects. These objects represent more abstract concepts such as the garbage

make everyday operations easier.  From an organizational perspective, it is convenient to divide the data types into three major categories. First, the basic data types are those that are assembled directly from C++ primitives (int, float, etc.). These types include simple vectors and matrices, as well as representations of simple geometric concepts like points, rectangles, sizes, and the like. The second category contains helper objects. These objects represent more abstract concepts such as the garbage collecting pointer class, range objects used for slicing, and abstractions such as termination criteria. The third category is what might be called large array types. These are objects whose fundamental purpose is to contain arrays or other assemblies of primitives or, more often, the basic data types mentioned first. The star example of this latter group is the cv::Mat class, which is used to represent arbitrary-dimensional arrays containing arbitrary basic elements. Objects such as images are specialized

collecting pointer class, range objects used for slicing, and abstractions such as termination criteria. The third category is what might be called large array types. These are objects whose fundamental purpose is to contain arrays or other assemblies of primitives or, more often, the basic data types mentioned first. The star example of this latter group is the cv::Mat class, which is used to represent arbitrary-dimensional arrays containing arbitrary basic elements. Objects such as images are specialized uses of the cv::Mat class but, unlike in earlier versions of OpenCV (i.e., before version 2.1), such specific use does not require a different class or type. In addition to cv::Mat, this category contains related objects such as the sparse matrix cv::SparseMat class, which is more naturally suited to non-dense data such as histograms. In addition to these types, OpenCV also makes heavy use of the Standard Template Library (STL) [STL]. This vector class is particularly relied on by OpenCV, and many OpenCV

uses of the cv::Mat class but, unlike in earlier versions of OpenCV (i.e., before version 2.1), such specific use does not require a different class or type. In addition to cv::Mat, this category contains related objects such as the sparse matrix cv::SparseMat class, which is more naturally suited to non-dense data such as histograms. In addition to these types, OpenCV also makes heavy use of the Standard Template Library (STL) [STL]. This vector class is particularly relied on by OpenCV, and many OpenCV library functions now have vector template objects in their argument lists. We will not cover STL in this book, other than as necessary to explain relevant functionality. If you are already comfortable with STL, many of the template mechanisms used “under the hood” in OpenCV will be familiar to you. Overview of the Basic Types The most straightforward of the basic data types is the template class cv::Vec<>. cv::Vec<> is a container class for primitives,1 which we will refer to as “the fixed vector classes.”

functions now have vector template objects in their argument lists. We will not cover STL in this book, other than as necessary to explain relevant functionality. If you are already comfortable with STL, many of the template mechanisms used “under the hood” in OpenCV will be familiar to you. Overview of the Basic Types The most straightforward of the basic data types is the template class cv::Vec<>. cv::Vec<> is a container class for primitives,1 which we will refer to as “the fixed vector classes.” Why not just use                                                              1 Actually, this is an oversimplification that we will clear up a little later in the chapter. In fact, cv::Vec<> is a vector container for anything, and uses templating to create this functionality. As a result, cv::Vec<> can contain other

standard template library classes? The key difference is that the fixed vector classes are classes intended for small vectors whose dimensions are known at compile time. This allows for efficient code. What “small” means in practice is that if you have more than just a few elements, you are probably using the wrong class. (In fact, as of version 2.2, this number cannot exceed nine in any case). We will look shortly at the cv::Mat class, which is the right way to handle big arrays of any number of dimensions, but for now, think of the fixed vector classes as being handy and speedy for little guys.  Even though cv::Vec<> is a template, you will not tend to see or use it in that form most of the time. Instead, there are aliases (typedef’s) for common instantiations of the cv::Vec<> template. They have names like cv::Vec2i, cv::Vec3i, or cv::Vec4d (for a two-element integer vector, a three-element integer vector, or a four-element double-precision floating-point vector, respectively). In general, anything of the

but for now, think of the fixed vector classes as being handy and speedy for little guys.  Even though cv::Vec<> is a template, you will not tend to see or use it in that form most of the time. Instead, there are aliases (typedef’s) for common instantiations of the cv::Vec<> template. They have names like cv::Vec2i, cv::Vec3i, or cv::Vec4d (for a two-element integer vector, a three-element integer vector, or a four-element double-precision floating-point vector, respectively). In general, anything of the form cv::Vec{2,3,4,6}{b,s,i,f,d} is valid for any combination of two to six dimensions2 and the five data types.  In addition to the fixed vector class, there are also “fixed matrix classes.” They are associated with the template cv::Matx<>. Just like the fixed vector classes, cv::Matx<> is not intended to be used for large arrays, but rather is designed for the handling of certain specific small matrix operations. In computer vision, there are a lot of 2-by-2 or 3-by-3 matrices running around, and a few

form cv::Vec{2,3,4,6}{b,s,i,f,d} is valid for any combination of two to six dimensions2 and the five data types.  In addition to the fixed vector class, there are also “fixed matrix classes.” They are associated with the template cv::Matx<>. Just like the fixed vector classes, cv::Matx<> is not intended to be used for large arrays, but rather is designed for the handling of certain specific small matrix operations. In computer vision, there are a lot of 2-by-2 or 3-by-3 matrices running around, and a few 4-by-4, which are used for various transformations and the like. cv::Matx<> is designed to hold these sorts of objects. As with cv::Vec<>, cv::Matx<> is normally accessed through aliases of the form cv::Matx{1,2,3,4,6}{1,2,3,4,6}{f,d}. It is important to notice that with the fixed matrix classes (like the fixed vector classes, but unlike cv::Mat, which we will come to shortly), the dimensionality of the fixed matrix classes must be known at compile time. Of course, it is precisely this knowledge that makes

4-by-4, which are used for various transformations and the like. cv::Matx<> is designed to hold these sorts of objects. As with cv::Vec<>, cv::Matx<> is normally accessed through aliases of the form cv::Matx{1,2,3,4,6}{1,2,3,4,6}{f,d}. It is important to notice that with the fixed matrix classes (like the fixed vector classes, but unlike cv::Mat, which we will come to shortly), the dimensionality of the fixed matrix classes must be known at compile time. Of course, it is precisely this knowledge that makes it possible to make operations with the fixed matrix classes highly efficient. Closely related to the fixed vector classes are the point classes. The point classes are containers for two or three values of one of the primitive types. The point classes are derived from their own template, so they are not directly descended from the fixed vector classes, but they can be cast to and from them. The main difference between the point classes and the fixed vector classes is that their members are accessed by

it possible to make operations with the fixed matrix classes highly efficient. Closely related to the fixed vector classes are the point classes. The point classes are containers for two or three values of one of the primitive types. The point classes are derived from their own template, so they are not directly descended from the fixed vector classes, but they can be cast to and from them. The main difference between the point classes and the fixed vector classes is that their members are accessed by named variables (mypoint.x, mypoint.y, etc.) rather than by a vector index (myvec[0], myvec[1], etc.). As with cv::Vec<>, the point classes are typically invoked using aliases for the instantiation of an appropriate template. Those aliases have names like cv::Point2i, cv::Point2f, or cv::Point2d, or cv::Point3i, cv::Point3f, or cv::Point3d. The class cv::Scalar is essentially a four-dimensional point. As with the point classes, cv::Scalar is actually associated with a template that can generate an arbitrary

variables (mypoint.x, mypoint.y, etc.) rather than by a vector index (myvec[0], myvec[1], etc.). As with cv::Vec<>, the point classes are typically invoked using aliases for the instantiation of an appropriate template. Those aliases have names like cv::Point2i, cv::Point2f, or cv::Point2d, or cv::Point3i, cv::Point3f, or cv::Point3d. The class cv::Scalar is essentially a four-dimensional point. As with the point classes, cv::Scalar is actually associated with a template that can generate an arbitrary four-component vector, but the keyword cv::Scalar specifically is aliased to a four-component vector with double-precision components. Unlike the point classes, the elements of a cv::Scalar object are accessed with an integer index, the same as cv::Vec<>. This is because cv::Scalar is directly derived from an instantiation of cv::Vec<> (specifically, from cv::Vec<double,4>). Next on our tour are cv::Size and cv::Rect. As with the point classes, these two are derived from their own templates. cv::Size is mainly

vector, but the keyword cv::Scalar specifically is aliased to a four-component vector with double-precision components. Unlike the point classes, the elements of a cv::Scalar object are accessed with an integer index, the same as cv::Vec<>. This is because cv::Scalar is directly derived from an instantiation of cv::Vec<> (specifically, from cv::Vec<double,4>). Next on our tour are cv::Size and cv::Rect. As with the point classes, these two are derived from their own templates. cv::Size is mainly distinguished by having data members width and height rather than x and y, while cv::Rect has all four. The class cv::Size is actually an alias for cv::Size2i, which is itself an alias of a more general template in the case of width and height being integers. For floating-point values of width and height, use the alias cv::Size2f. Similarly, cv::Rect is an alias for the integer form of rectangle. There is also a class to represent a rectangle that is not axis-aligned. It is called cv::RotatedRect and contains a

by having data members width and height rather than x and y, while cv::Rect has all four. The class cv::Size is actually an alias for cv::Size2i, which is itself an alias of a more general template in the case of width and height being integers. For floating-point values of width and height, use the alias cv::Size2f. Similarly, cv::Rect is an alias for the integer form of rectangle. There is also a class to represent a rectangle that is not axis-aligned. It is called cv::RotatedRect and contains a cv::Point2f called center, a cv::Size2f called size, and one additional float called angle.                                                                                                                                                                                       class objects, either from OpenCV or elsewhere. In most usages, however, cv::Vec is used as a container for C primitive types like int or float. 2 Except five, for some reason five is not an option.

Basic Types: Getting Down to Details Each of the basic types is actually a relatively complicated object, supporting its own interface functions, overloaded operators, and the like. In this section, we will take a somewhat more encyclopedic look at what each type offers, and how some of the otherwise similar appearing types differ from one another.  As we go over these classes, we will try to hit on the high points of their interfaces, but not get into every gory detail. Instead, we will provide examples that should convey what you can and can’t do with these objects. For the lowest-level details, you should consult …/opencv2/core/core.hpp. The Point Classes Of the OpenCV basic types, the point classes are probably the simplest. As we mentioned earlier, these are implemented based on a template structure, such that there can be points of any type: integer, floating-point, and so on. There are actually two such templates, one for two-dimensional and one for three-dimensional points. The big advantage of the

that should convey what you can and can’t do with these objects. For the lowest-level details, you should consult …/opencv2/core/core.hpp. The Point Classes Of the OpenCV basic types, the point classes are probably the simplest. As we mentioned earlier, these are implemented based on a template structure, such that there can be points of any type: integer, floating-point, and so on. There are actually two such templates, one for two-dimensional and one for three-dimensional points. The big advantage of the point classes is that they are simple and have very little overhead. Natively, they do not have a lot of operations defined on them, but they can be cast to somewhat more generalized types, such as the fixed vector classes or the fixed matrix classes (discussed later), when needed. In most programs, the point classes are instantiated using aliases that take forms like cv::Point2i or cv::Point3f, with the last letter indicating the desired primitive from which the point is to be constructed. (Here, b is an

point classes is that they are simple and have very little overhead. Natively, they do not have a lot of operations defined on them, but they can be cast to somewhat more generalized types, such as the fixed vector classes or the fixed matrix classes (discussed later), when needed. In most programs, the point classes are instantiated using aliases that take forms like cv::Point2i or cv::Point3f, with the last letter indicating the desired primitive from which the point is to be constructed. (Here, b is an unsigned character, s is a short integer, i is a 32-bit integer, f is a 32-bit floating-point number, and d is a 64-bit floating-point number.)  Table 3-1 is the (relatively short) list of functions natively supported by the point classes. It is important to note that there are several  important operations that are supported, but they are supported indirectly through implicit casting to the fixed vector classes (see below). These operations notably contain all of the vector and singleton3 overloaded

is an unsigned character, s is a short integer, i is a 32-bit integer, f is a 32-bit floating-point number, and d is a 64-bit floating-point number.)  Table 3-1 is the (relatively short) list of functions natively supported by the point classes. It is important to note that there are several  important operations that are supported, but they are supported indirectly through implicit casting to the fixed vector classes (see below). These operations notably contain all of the vector and singleton3 overloaded algebraic operators and comparison operators. Table 3-1: Operations supported directly by the point classes Operation Examples Default constructors cv::Point2i p(); cv::Point3f p();  Copy constructor cv::Point3f p2( p1 ); Value constructors cv::Point2i p( x0, x1 );              cv::Point3d p( x0, x1, x2 );              Cast to the fixed vector classes (cv::Vec3f) p; Member access p.x; p.y; // and for three-dimensional            // point classes:  p.z                    Dot product float x = p1.dot( p2 )

algebraic operators and comparison operators. Table 3-1: Operations supported directly by the point classes Operation Examples Default constructors cv::Point2i p(); cv::Point3f p();  Copy constructor cv::Point3f p2( p1 ); Value constructors cv::Point2i p( x0, x1 );              cv::Point3d p( x0, x1, x2 );              Cast to the fixed vector classes (cv::Vec3f) p; Member access p.x; p.y; // and for three-dimensional            // point classes:  p.z                    Dot product float x = p1.dot( p2 ) Double-precision dot double x = p1.ddot( p2 )                                                              3 You might have expected us to use the word “scalar” here, but we avoided doing so because cv::Scalar is an existing class in the library. As you will see shortly, a cv::Scalar in OpenCV is (somewhat confusingly) an array of four numbers, approximately equivalent to a cv::Vec with four elements!  In this context, the word “singleton” can be understood to mean “a single object of whatever type the

) Double-precision dot double x = p1.ddot( p2 )                                                              3 You might have expected us to use the word “scalar” here, but we avoided doing so because cv::Scalar is an existing class in the library. As you will see shortly, a cv::Scalar in OpenCV is (somewhat confusingly) an array of four numbers, approximately equivalent to a cv::Vec with four elements!  In this context, the word “singleton” can be understood to mean “a single object of whatever type the vector is an array of.”

product Cross product p1.cross( p2 ) // (for three-dimensional point                 // classes only) Query if point p is inside of rectangle r p.inside( r )  // (for two-dimensional point                 // classes only) These types can be cast to and from the old C interface types CvPoint and CvPoint2D32f. In the case in which a floating-point-valued instance of one of the point classes is cast to CvPoint, the values will automatically be rounded. class cv::Scalar The class cv::Scalar is really a four-dimensional point class. Like the others, it is actually associated with a template class, but the alias for accessing it returns an instantiation of that template in which all of the members are double-precision floating-point numbers. The cv::Scalar class also has some special member functions associated with uses of four-component vectors in computer vision. Table 3-2: Operations supported directly by class cv::Scalar Operation Example Default constructors cv::Scalar s(); Copy constructor cv::Scalar s2( s1

point class. Like the others, it is actually associated with a template class, but the alias for accessing it returns an instantiation of that template in which all of the members are double-precision floating-point numbers. The cv::Scalar class also has some special member functions associated with uses of four-component vectors in computer vision. Table 3-2: Operations supported directly by class cv::Scalar Operation Example Default constructors cv::Scalar s(); Copy constructor cv::Scalar s2( s1 ); Value constructors cv::Scalar s( x0 );  cv::Scalar s( x0, x1, x2, x3 );              Element-wise multiplication s1.mul( s2 ); (Quaternion) conjugation s.conj();   // (returns cv::Scalar(s0,-s1,-s2,-s2)) (Quaternion) real test s.isReal(); // (returns true iff s1==s2==s3==0) You will notice that for cv::Scalar, the operation “cast to the fixed vector classes” does not appear in Table 3-2 (as it did in the case of the point classes). This is because, unlike the point classes, cv::Scalar inherits directly from an

constructors cv::Scalar s( x0 );  cv::Scalar s( x0, x1, x2, x3 );              Element-wise multiplication s1.mul( s2 ); (Quaternion) conjugation s.conj();   // (returns cv::Scalar(s0,-s1,-s2,-s2)) (Quaternion) real test s.isReal(); // (returns true iff s1==s2==s3==0) You will notice that for cv::Scalar, the operation “cast to the fixed vector classes” does not appear in Table 3-2 (as it did in the case of the point classes). This is because, unlike the point classes, cv::Scalar inherits directly from an instantiation of the fixed vector class template. As a result, it inherits all of the vector algebra operations, member access functions (i.e., operator[]), and other properties from the fixed vector classes. We will get to that class later, but for now, just keep in mind that cv::Scalar is shorthand for a four-dimensional double-precision vector that has a few special member functions attached that are useful for various kinds of four-vectors. The class cv::Scalar can be freely cast to and from the old C

instantiation of the fixed vector class template. As a result, it inherits all of the vector algebra operations, member access functions (i.e., operator[]), and other properties from the fixed vector classes. We will get to that class later, but for now, just keep in mind that cv::Scalar is shorthand for a four-dimensional double-precision vector that has a few special member functions attached that are useful for various kinds of four-vectors. The class cv::Scalar can be freely cast to and from the old C interface CvScalar type. The Size Classes The size classes are, in practice, similar to the corresponding point classes, and can be cast to and from them. The primary difference between the two is that the point class’ data members are named x and y, while the corresponding data members in the size classes are named width and height. The three aliases for the size classes are cv::Size, cv::Size2i, and cv::Size2f. The first two of these are equivalent and imply integer size, while the last is for 32-bit

C interface CvScalar type. The Size Classes The size classes are, in practice, similar to the corresponding point classes, and can be cast to and from them. The primary difference between the two is that the point class’ data members are named x and y, while the corresponding data members in the size classes are named width and height. The three aliases for the size classes are cv::Size, cv::Size2i, and cv::Size2f. The first two of these are equivalent and imply integer size, while the last is for 32-bit floating-point sizes. As with the point classes, the size classes can be cast to and from the corresponding old-style OpenCV classes (in this case, CvSize and CvSize2D32f). Table 3-3: Operations supported directly by the size classes

Operation Example Default constructors cv::Size sz(); cv::Size2i sz(); cv::Point2f sz(); Copy constructor cv::Size sz2( sz1 ); Value constructors cv::Size2f sz( w, h );              Member access sz.width; sz.height;                    Compute area sz.area(); Unlike the point classes, the size classes do not support casting to the fixed vector classes. This means that the size classes have more restricted utility. On the other hand, the point classes and the fixed vector classes can be cast to the size classes without any problem.  class cv::Rect The rectangle classes include the members x and y of the point class (representing the upper-left corner of the rectangle) and the members width and height of the size class (representing the extent of the rectangle). The rectangle classes, however, do not inherit from the point or size classes, and so in general they do not inherit operators from them.  Table 3-4: Operations supported directly by class cv::Rect* Operation Example Default constructors cv::Rect r();

without any problem.  class cv::Rect The rectangle classes include the members x and y of the point class (representing the upper-left corner of the rectangle) and the members width and height of the size class (representing the extent of the rectangle). The rectangle classes, however, do not inherit from the point or size classes, and so in general they do not inherit operators from them.  Table 3-4: Operations supported directly by class cv::Rect* Operation Example Default constructors cv::Rect r(); Copy constructor cv::Rect r2( r1 ); Value constructors cv::Rect( x, y, w, h );             Construct from origin and size cv::Rect( p, sz ); Construct from two corners cv::Rect( p1, p2 ); Member access r.x; r.y; r.width; r.height; Compute area r.area(); Extract upper-left corner r.tl(); Extract lower-right corner r.lr; Determine if point p is inside of rectangle r r.contains( p ); Cast operators and copy constructors exist to allow cv::Rect to be computed from or cast to the old-style cv::CvRect type as well.

constructor cv::Rect r2( r1 ); Value constructors cv::Rect( x, y, w, h );             Construct from origin and size cv::Rect( p, sz ); Construct from two corners cv::Rect( p1, p2 ); Member access r.x; r.y; r.width; r.height; Compute area r.area(); Extract upper-left corner r.tl(); Extract lower-right corner r.lr; Determine if point p is inside of rectangle r r.contains( p ); Cast operators and copy constructors exist to allow cv::Rect to be computed from or cast to the old-style cv::CvRect type as well. cv::Rect is actually an alias for a rectangle template instantiated with integer members. The class cv::Rect also supports a variety of overloaded operators that can be used for the computation of various geometrical properties of two rectangles or a rectangle and another object.

Table 3-5. Overloaded operators that take objects of type cv::Rect Operation Example Intersection of rectangles r1 and r2 cv::Rect r3 = r1 & r2; r1 &= r2; Minimum area rectangle containing rectangles r1 and r2 cv::Rect r3 = r1 | r2; r1 |= r2; Translate rectangle r by an amount x cv::Rect rx = r + v; // v is a cv::Point2i r += v; Enlarge a rectangle r by an amount given by size s cv::Rect rs = r + s; // s is a cv::Point2i r += s; Compare rectangles r1 and r2 for exact equality bool eq = (r1 == r2); Compare rectangles r1 and r2 for inequality bool ne = (r1 != r2); class cv::RotatedRect The cv::RotatedRect class is one of the few classes in the C++ OpenCV interface that is not a template underneath. It is a container, which holds a cv::Point2f called center, a cv::Size2f called size, and one additional float called angle, with the latter representing the rotation of the rectangle around center. One very important difference between cv::RotatedRect and cv::Rect is the convention that a cv::RotatedRect is located

r1 and r2 for inequality bool ne = (r1 != r2); class cv::RotatedRect The cv::RotatedRect class is one of the few classes in the C++ OpenCV interface that is not a template underneath. It is a container, which holds a cv::Point2f called center, a cv::Size2f called size, and one additional float called angle, with the latter representing the rotation of the rectangle around center. One very important difference between cv::RotatedRect and cv::Rect is the convention that a cv::RotatedRect is located in “space” relative to its center, while the cv::Rect is located relative to its upper-left corner. Table 3-6: Operations supported directly by class cv::RotatedRect Operation Example Default constructors cv::RotatedRect rr(); Copy constructor cv::RotatedRect rr2( rr1 ); Construct from two corners cv::RotatedRect( p1, p2 ); Value constructors; takes a point, a size, and an angle cv::RotatedRect rr( p, sz, theta ) ;             Member access rr.center; rr.size; rr.angle; Return a list of the corners rr.points( pts[4]

relative to its center, while the cv::Rect is located relative to its upper-left corner. Table 3-6: Operations supported directly by class cv::RotatedRect Operation Example Default constructors cv::RotatedRect rr(); Copy constructor cv::RotatedRect rr2( rr1 ); Construct from two corners cv::RotatedRect( p1, p2 ); Value constructors; takes a point, a size, and an angle cv::RotatedRect rr( p, sz, theta ) ;             Member access rr.center; rr.size; rr.angle; Return a list of the corners rr.points( pts[4] ); The Fixed Matrix Classes The fixed matrix classes are for matrices whose dimensions are known at compile time (hence “fixed”). As a result, all memory for their data is allocated on the stack, which means that they allocate and clean up quickly. Operations on them are fast and there are specially optimized implementations for small matrices (2-by-2, 3-by-3, etc.). The fixed matrix classes are also central to many of the other basic types in the C++ interface to OpenCV. The fixed vector class derives from

The Fixed Matrix Classes The fixed matrix classes are for matrices whose dimensions are known at compile time (hence “fixed”). As a result, all memory for their data is allocated on the stack, which means that they allocate and clean up quickly. Operations on them are fast and there are specially optimized implementations for small matrices (2-by-2, 3-by-3, etc.). The fixed matrix classes are also central to many of the other basic types in the C++ interface to OpenCV. The fixed vector class derives from the fixed matrix classes and other classes either derive from the fixed vector class (like cv::Scalar), or they rely on casting to the fixed vector class for many important operations. As usual, the fixed matrix classes are really a template. The template is called

cv::Matx<>, but individual matrices are usually allocated using aliases. The basic form of such an alias is cv::Matx{1,2,…}{1,2,…}{f,d}, where the numbers can be any number from one to six, and the trailing letter has the same meaning as with the previous types.4  In general, you should use the fixed matrix classes when you are representing something that is really a matrix with which you are going to do matrix algebra. If your object is really a big data array, like an image, or a huge list of points or something like that, the fixed matrix classes are not the correct solution; you should be using cv::Mat (which we will get to shortly). Fixed matrix classes are for small matrices where you know the size at compile time: (e.g., a camera matrix). Table 3-7: Operations supported by class cv::Matx Operation Example Default constructor cv::Matx33f m33f(); cv::Matx43d m43d(); Copy constructor cv::Matx22d m22d( n22d ); Value constructors cv::Matx21f m(x0,x1); cv::Matx44d

an image, or a huge list of points or something like that, the fixed matrix classes are not the correct solution; you should be using cv::Mat (which we will get to shortly). Fixed matrix classes are for small matrices where you know the size at compile time: (e.g., a camera matrix). Table 3-7: Operations supported by class cv::Matx Operation Example Default constructor cv::Matx33f m33f(); cv::Matx43d m43d(); Copy constructor cv::Matx22d m22d( n22d ); Value constructors cv::Matx21f m(x0,x1); cv::Matx44d m(x0,x1,x2,x3,x4,x5,x6,x7,x8,x9,x10,x11,x12,x13,x14,x15);         Matrix of identical elements m33f = cv::Matx33f::all( x ); Matrix of zeros m23d = cv::Matx23d::zeros(); Matrix of ones m16f = cv::Matx16f::ones(); Create a unit matrix m33f = cv::Matx33f::eye(); Create a matrix that can hold the diagonal of another m31f = cv::Matx33f::diag(); // create a matrix of                              // size 3-by-1 of floats Create a matrix with uniformly distributed entries m33f = cv::Matx33f::randu( min, max ); Create

Matrix of identical elements m33f = cv::Matx33f::all( x ); Matrix of zeros m23d = cv::Matx23d::zeros(); Matrix of ones m16f = cv::Matx16f::ones(); Create a unit matrix m33f = cv::Matx33f::eye(); Create a matrix that can hold the diagonal of another m31f = cv::Matx33f::diag(); // create a matrix of                              // size 3-by-1 of floats Create a matrix with uniformly distributed entries m33f = cv::Matx33f::randu( min, max ); Create a matrix with normally distributed entries m33f = cv::Matx33f::nrandn( mean, variance ); Member access m( i, j ), m( i ); // one argument for                     // one-dimensional matrices only                Matrix algebra m1 = m0; m0 * m1; m0 + m1; m0 – m1; Singleton algebra m * a; a * m; m / a; Comparison m1 == m2; m1 != m2; Dot product m1.dot( m2 );  // (sum of element-wise                                                               4 At the time of writing, the relevant header file called core.hpp does not actually contain every possible combination

mean, variance ); Member access m( i, j ), m( i ); // one argument for                     // one-dimensional matrices only                Matrix algebra m1 = m0; m0 * m1; m0 + m1; m0 – m1; Singleton algebra m * a; a * m; m / a; Comparison m1 == m2; m1 != m2; Dot product m1.dot( m2 );  // (sum of element-wise                                                               4 At the time of writing, the relevant header file called core.hpp does not actually contain every possible combination of these integers. For example, there is no 1-by-1 matrix alias, nor is there a 5-by-5. This may or may not change in later releases, but you will pretty much never want the missing ones anyway. If you really do want one of the odd ones, you can just instantiate the template yourself (e.g., cv::Matx<5,5,float>).

// multiplications, precision of m) Dot product m1.ddot( m2 ); // (sum of element-wise multiplications,                // double precision) Reshape a matrix m91f = m33f.reshape<9,1>(); Cast operators m44f = (Matx44f) m44d Extract i,j-th algebraic minor m44f.get_minor( i, j ); Extract row i m14f = m44f.row( i ); Extract column j m41f = m44f.col( j ); Extract matrix diagonal m41f = m44f.diag(); Compute transpose  n44f = m44f.t(); Invert matrix n44f = m44f.inv( method ); // (default method is                             // cv::DECOMP_LU) Solve linear system m31f = m33f.solve( rhs31f, method ) m32f = m33f.solve<2>( rhs32f, method ); // (template                 // form5; default method is DECOMP_LU)  Per-element multiplication m1.mul( m2 ); Note that many of the fixed matrix functions are static relative to the class (i.e., you access them directly as members of the class rather than as members of a particular object). For example, if you would like to construct a 3-by-3 identity matrix, you have a

// cv::DECOMP_LU) Solve linear system m31f = m33f.solve( rhs31f, method ) m32f = m33f.solve<2>( rhs32f, method ); // (template                 // form5; default method is DECOMP_LU)  Per-element multiplication m1.mul( m2 ); Note that many of the fixed matrix functions are static relative to the class (i.e., you access them directly as members of the class rather than as members of a particular object). For example, if you would like to construct a 3-by-3 identity matrix, you have a handy class function for it: cv::Mat33f::eye(). Note that, in this example, eye() does not need any arguments because it is a member of the class, and the class is already a specialization of the cv::Matx<> template to 3-by-3. The Fixed Vector Classes  The fixed vector classes are derived from the fixed matrix classes. They are really just convenience functions for cv::Matx<>. In the proper sense of C++ inheritance, it is correct to say that the fixed vector template cv::Vec<> is a cv::Matx<> whose number

handy class function for it: cv::Mat33f::eye(). Note that, in this example, eye() does not need any arguments because it is a member of the class, and the class is already a specialization of the cv::Matx<> template to 3-by-3. The Fixed Vector Classes  The fixed vector classes are derived from the fixed matrix classes. They are really just convenience functions for cv::Matx<>. In the proper sense of C++ inheritance, it is correct to say that the fixed vector template cv::Vec<> is a cv::Matx<> whose number of columns is one. The readily available aliases for specific instantiations of cv::Vec<> are of the form cv::Vec{2,3,4,6}{b,s,w,i,f,d}, where the last character has the usual meanings (with the addition of w, which indicates an unsigned short). Table 3-8: Operations supported by class cv::Vec Operation Example Default constructor Vec2s v2s(); Vec6f v6f();        // etc…                                                              5 The template form is used when the right hand side of the implied matrix

of columns is one. The readily available aliases for specific instantiations of cv::Vec<> are of the form cv::Vec{2,3,4,6}{b,s,w,i,f,d}, where the last character has the usual meanings (with the addition of w, which indicates an unsigned short). Table 3-8: Operations supported by class cv::Vec Operation Example Default constructor Vec2s v2s(); Vec6f v6f();        // etc…                                                              5 The template form is used when the right hand side of the implied matrix equation has multiple columns.  In this case, we are essentially solving for ‘k’ different systems at once.  This value of ‘k’ must be supplied as the template argument to solve<>().  It will also determine the number of columns in the result matrix.

Copy constructor Vec3f u3f( v3f ); Value constructors Vec2f v2f(x0,x1); Vec6d v6d(x0,x1,x2,x3,x4,x5);               Member access v4f[ i ]; v3w( j ); // (operator() and operator[]                      // both work) Vector cross-product v3f.cross( u3f ); The primary conveniences of the fixed vector classes are the ability to access elements with a single ordinal, and a few specific additional functions that would not make sense for a general matrix (e.g., cross product). We can see this in Table 3-8 by the relatively small number of novel methods added to the large number of methods inherited from the fixed matrix classes. The Complex Number Classes  One more class type should be included in the basic types: the complex number classes. The OpenCV complex number classes are not identical to, but are compatible with, and can be cast to and from, the classes associated with the STL complex number class template complex<>. The most substantial difference between the OpenCV and STL complex number classes is in

relatively small number of novel methods added to the large number of methods inherited from the fixed matrix classes. The Complex Number Classes  One more class type should be included in the basic types: the complex number classes. The OpenCV complex number classes are not identical to, but are compatible with, and can be cast to and from, the classes associated with the STL complex number class template complex<>. The most substantial difference between the OpenCV and STL complex number classes is in member access. In the STL classes, the real and imaginary parts are accessed through the member functions real() and imag(), while in the OpenCV class, they are directly accessible as (public) member variables re and im. Table 3-9: Operations supported by the OpenCV complex number classes Operation Example Default constructor cv::Complexf z1; cv::Complexd z2; Copy constructor cv::Complexf z2( z1 ); Value constructors cv::Complexd z1(re0); cv::Complexd(re0,im1) ;              Copy constructor cv::Complexf u2f(

access. In the STL classes, the real and imaginary parts are accessed through the member functions real() and imag(), while in the OpenCV class, they are directly accessible as (public) member variables re and im. Table 3-9: Operations supported by the OpenCV complex number classes Operation Example Default constructor cv::Complexf z1; cv::Complexd z2; Copy constructor cv::Complexf z2( z1 ); Value constructors cv::Complexd z1(re0); cv::Complexd(re0,im1) ;              Copy constructor cv::Complexf u2f( v2f ); Member access z1.re; z1.im; Complex conjugate z2 = z1.conj(); Like many basic types, the complex classes are aliases for underlying templates. cv::Complexf and cv::Complexd are aliases for single- and double-precision complex floating point numbers, respectively. Helper Objects In addition to the basic types and the big containers (which we will get to in the next section), there is a family of helper objects, which are important for controlling various algorithms (such as termination criteria) or for

); Member access z1.re; z1.im; Complex conjugate z2 = z1.conj(); Like many basic types, the complex classes are aliases for underlying templates. cv::Complexf and cv::Complexd are aliases for single- and double-precision complex floating point numbers, respectively. Helper Objects In addition to the basic types and the big containers (which we will get to in the next section), there is a family of helper objects, which are important for controlling various algorithms (such as termination criteria) or for doing various operations on the containers (such as “ranges” or “slices”). There is also one very important object, the “smart” pointer object cv::Ptr. Looking into cv::Ptr, we will examine the garbage collecting system, which is integral to the C++ interface to OpenCV. This system allows us to not have to worry about the details of allocation and deallocation of objects in the manner that was once so onerous in the earlier C-based OpenCV interface (i.e., before version 2.1).

class cv::TermCriteria Many algorithms require a stopping condition to know when to quit. Generally, stopping criteria take the form of either some finite number of iterations that are allowed (called COUNT or MAX_ITER) or some kind of error parameter that basically says “if you are this close, you can quit” (called EPS—short for “epsilon,” everyone’s favorite tiny number). In many cases, it is desirable to have both of these at once, so that if the algorithm never gets “close enough,” then it will still quit at some point. cv::TermCriteria objects encapsulate one, the other, or both of the stopping criteria so that they can be passed conveniently to an OpenCV algorithm function. They have three member variables, type, maxCount, and epsilon, which can be set directly (they are public) or, more often, are just set by the constructor with the form TermCriteria( int type, int maxCount, double epsilon ). The variable type is set to one of the following values: cv::TermCriteria::COUNT or TermCriteria::EPS. You can

at some point. cv::TermCriteria objects encapsulate one, the other, or both of the stopping criteria so that they can be passed conveniently to an OpenCV algorithm function. They have three member variables, type, maxCount, and epsilon, which can be set directly (they are public) or, more often, are just set by the constructor with the form TermCriteria( int type, int maxCount, double epsilon ). The variable type is set to one of the following values: cv::TermCriteria::COUNT or TermCriteria::EPS. You can also “or” (i.e., |) the two together. The value cv::TermCriteria::COUNT is a synonym for cv::TermCriteria::MAX_ITER, so you can use that if you prefer. If the termination criterion includes cv::TermCriteria::COUNT, then you are telling the algorithm to terminate after maxCount iterations. If the termination criterion includes cv::TermCriteria::EPS, then you are telling the algorithm to terminate after some metric associated with the algorithm’s convergence falls below epsilon6 The type argument has to be set

“or” (i.e., |) the two together. The value cv::TermCriteria::COUNT is a synonym for cv::TermCriteria::MAX_ITER, so you can use that if you prefer. If the termination criterion includes cv::TermCriteria::COUNT, then you are telling the algorithm to terminate after maxCount iterations. If the termination criterion includes cv::TermCriteria::EPS, then you are telling the algorithm to terminate after some metric associated with the algorithm’s convergence falls below epsilon6 The type argument has to be set accordingly for maxCount or epsilon to be used. class cv::Range The cv::Range class is used to specify a continuous sequence of integers. cv::Range objects have two elements, start and end, which—similar to cv::TermCriteria—are often set with the constructor cv::Range( int start, int end ). Ranges are inclusive of their start value, but not inclusive of their end value, so cv::Range rng( 0, 4 ) includes the values 0, 1, 2, 3, but not 4. Using size(), one can find the number of elements in a range. From the

accordingly for maxCount or epsilon to be used. class cv::Range The cv::Range class is used to specify a continuous sequence of integers. cv::Range objects have two elements, start and end, which—similar to cv::TermCriteria—are often set with the constructor cv::Range( int start, int end ). Ranges are inclusive of their start value, but not inclusive of their end value, so cv::Range rng( 0, 4 ) includes the values 0, 1, 2, 3, but not 4. Using size(), one can find the number of elements in a range. From the above, rng.size() would be equal to 4. There is also a member empty() that tests if a range has no elements. Finally, cv::Range::all() can be used anywhere a range is required to indicate whatever range the consumer has available. The cv::Ptr template, and Garbage Collection 101 One very useful object type in C++ is a “smart” pointer.7 This pointer allows us to create a reference to a thing, and then pass that around. You can create more references to that thing, and then all of those references will be

above, rng.size() would be equal to 4. There is also a member empty() that tests if a range has no elements. Finally, cv::Range::all() can be used anywhere a range is required to indicate whatever range the consumer has available. The cv::Ptr template, and Garbage Collection 101 One very useful object type in C++ is a “smart” pointer.7 This pointer allows us to create a reference to a thing, and then pass that around. You can create more references to that thing, and then all of those references will be counted. As references go out of scope, the reference count for the smart pointer is decremented. Once all of the references (instances of the pointer) are gone, the “thing” will automatically be cleaned up (deallocated). The programmer doesn’t have to do this bookkeeping anymore. The way this all works is as follows: first, you define an instance of the pointer template for the class object that you want to “wrap.” You do this with a call like cv::Ptr<Matx33f> p( new cv::Matx33f ). The constructor for the

counted. As references go out of scope, the reference count for the smart pointer is decremented. Once all of the references (instances of the pointer) are gone, the “thing” will automatically be cleaned up (deallocated). The programmer doesn’t have to do this bookkeeping anymore. The way this all works is as follows: first, you define an instance of the pointer template for the class object that you want to “wrap.” You do this with a call like cv::Ptr<Matx33f> p( new cv::Matx33f ). The constructor for the template object takes a pointer to the object to be pointed to. Once you do this, you have your smart pointer p, which is a sort of pointer-like object that you can pass around and use just like a normal pointer (i.e., it supports operators such as operator*() and operator->()). Once you have p, you can create other objects of the same type without passing them a pointer to a new object.  For example, you could create: Ptr<Mat33f> q, and when you assign the value of p to q, somewhere behind the scenes, the

template object takes a pointer to the object to be pointed to. Once you do this, you have your smart pointer p, which is a sort of pointer-like object that you can pass around and use just like a normal pointer (i.e., it supports operators such as operator*() and operator->()). Once you have p, you can create other objects of the same type without passing them a pointer to a new object.  For example, you could create: Ptr<Mat33f> q, and when you assign the value of p to q, somewhere behind the scenes, the “smart” action of the smart pointer comes into play. You see, just like a usual pointer, there is still only one actual cv::Mat33f object out there that p and q both point to. The                                                              6 The exact termination criteria are clearly algorithm dependent, but the documentation will always be clear how a particular algorithm interprets epsilon. 7 If you are familiar with some of the more recent additions to the C++ standard, you will recognize a similarity

“smart” action of the smart pointer comes into play. You see, just like a usual pointer, there is still only one actual cv::Mat33f object out there that p and q both point to. The                                                              6 The exact termination criteria are clearly algorithm dependent, but the documentation will always be clear how a particular algorithm interprets epsilon. 7 If you are familiar with some of the more recent additions to the C++ standard, you will recognize a similarity between the OpenCV cv::Ptr<> template and the smart_ptr<> template. Similarly, there is a smart pointer shared_ptr<> in the Boost library. Ultimately, they all function more or less the same.

difference is that both p and q know that they are each one of two pointers. Should p disappear (such as by falling out of scope), q knows that it is the only remaining reference to the original matrix. If q should then disappear and its destructor is called (implicitly), q will know that is the last one left, and that it should deallocate the original matrix. You can think of this like the last person out of a building being responsible for turning out the lights and locking the door (and in this case, burning the building to the ground as well). The cv::Ptr<> template class supports several additional functions in its interface related to the reference counting functionality of the smart pointer. Specifically, the functions addref() and release() increment and decrement the internal reference counter of the pointer. These are relatively dangerous functions to use, but are available in case you need to micromanage the reference counters yourself.  There is also a function empty(), which can be used to

burning the building to the ground as well). The cv::Ptr<> template class supports several additional functions in its interface related to the reference counting functionality of the smart pointer. Specifically, the functions addref() and release() increment and decrement the internal reference counter of the pointer. These are relatively dangerous functions to use, but are available in case you need to micromanage the reference counters yourself.  There is also a function empty(), which can be used to determine if a smart pointer is pointing to an object that has been deallocated. This could happen if you called release() on the object one or more times. In this case, you would still have a smart pointer around, but the object pointed to might already have been destroyed. There is a second application of empty(), which is to determine if the internal object pointer inside of the smart pointer object happens to be NULL for some other reason. This might occur if you assigned the smart pointer by calling a

determine if a smart pointer is pointing to an object that has been deallocated. This could happen if you called release() on the object one or more times. In this case, you would still have a smart pointer around, but the object pointed to might already have been destroyed. There is a second application of empty(), which is to determine if the internal object pointer inside of the smart pointer object happens to be NULL for some other reason. This might occur if you assigned the smart pointer by calling a function that might just return NULL in the first place (cvLoadImage(),fopen(), etc.)8 The final member of Ptr<> that you will want to know about is delete_obj(). This is a function that gets called automatically when the reference count gets to zero. By default, this function is defined but does nothing. It is there so that you can overload it in the case of instantiation of cv::Ptr<>, which points to a class that requires some specific operation in order to clean up the class to which it points. For

a function that might just return NULL in the first place (cvLoadImage(),fopen(), etc.)8 The final member of Ptr<> that you will want to know about is delete_obj(). This is a function that gets called automatically when the reference count gets to zero. By default, this function is defined but does nothing. It is there so that you can overload it in the case of instantiation of cv::Ptr<>, which points to a class that requires some specific operation in order to clean up the class to which it points. For example, let’s say that you are working with an old-style (pre-version 2.1) IplImage.9 In the old days, you might, for example, have called cvLoadImage() to load that image from disk. In the C interface, that would have looked like:  IplImage* img_p = cvLoadImage( … ); The modern version of this (while still using IplImage, rather than cv::Mat, which we are still working our way up to) would look like:  Ptr<IplImage> img_p = cvLoadImage( “an_image” ); or (if you prefer): Ptr<IplImage> img_p(

it points. For example, let’s say that you are working with an old-style (pre-version 2.1) IplImage.9 In the old days, you might, for example, have called cvLoadImage() to load that image from disk. In the C interface, that would have looked like:  IplImage* img_p = cvLoadImage( … ); The modern version of this (while still using IplImage, rather than cv::Mat, which we are still working our way up to) would look like:  Ptr<IplImage> img_p = cvLoadImage( “an_image” ); or (if you prefer): Ptr<IplImage> img_p( cvLoadImage(“an_image” ) ); Now you can use img_p in exactly the same way as a pointer (which is to say, for readers experienced with the pre-version 2.1 interface, “exactly as you would have back then”). Conveniently, this particular template instantiation is actually already defined for you somewhere in the vast sea of header files that make up OpenCV. If you were to go search it out, you would find the following template function defined:  template<> inline void cv::Ptr<IplImage>::delete_obj() {

cvLoadImage(“an_image” ) ); Now you can use img_p in exactly the same way as a pointer (which is to say, for readers experienced with the pre-version 2.1 interface, “exactly as you would have back then”). Conveniently, this particular template instantiation is actually already defined for you somewhere in the vast sea of header files that make up OpenCV. If you were to go search it out, you would find the following template function defined:  template<> inline void cv::Ptr<IplImage>::delete_obj() {     cvReleaseImage(&obj);  } (The variable obj is the name of the class member variable inside of Ptr<> that actually holds the pointer to the allocated object.) As a result of this definition, you will not need to deallocate the IplImage* pointer you got from cvLoadImage(). Instead, it will be automatically deallocated for you when img_p falls out of scope.                                                              8 For the purposes of this example, we will make reference to IplImage and cvLoadImage(), both

} (The variable obj is the name of the class member variable inside of Ptr<> that actually holds the pointer to the allocated object.) As a result of this definition, you will not need to deallocate the IplImage* pointer you got from cvLoadImage(). Instead, it will be automatically deallocated for you when img_p falls out of scope.                                                              8 For the purposes of this example, we will make reference to IplImage and cvLoadImage(), both constructs from the ancient pre-version 2.1 interface that are now deprecated. We won’t really cover them in detail in this book, but all you need to know for this example is that IplImage is the old data structure for images, and cvLoadImage() was the old function to get an image from disk and return a pointer to the resulting image structure. 9 This example might seem a bit artificial, but in fact, if you have a large body of pre-v2.1 code you are trying to modernize, you will likely find yourself doing an operation like

ancient pre-version 2.1 interface that are now deprecated. We won’t really cover them in detail in this book, but all you need to know for this example is that IplImage is the old data structure for images, and cvLoadImage() was the old function to get an image from disk and return a pointer to the resulting image structure. 9 This example might seem a bit artificial, but in fact, if you have a large body of pre-v2.1 code you are trying to modernize, you will likely find yourself doing an operation like this quite often.

This example was a somewhat special (though highly relevant) situation, in that the case of a smart pointer to IplImage is sufficiently salient that it was defined for you by the library. In a somewhat more typical case, when the clean-up function does not exist for what you want, you will have to define it yourself. Consider the example of creating a file handle using a smart pointer to FILE.10 In this case, we define our own overloaded version of delete_obj() for the cv::Ptr<FILE> template: template<> inline void cv::Ptr<FILE>::delete_obj() {     fclose(obj);  } Then you could go ahead and use that pointer to open a file, do whatever with it, and later, just let the pointer fall out of scope (at which time the file handle would automatically be closed for you): {   Ptr<FILE> f(fopen("myfile.txt", "r"));   if(f.empty())     throw …;  // Throw an exception, we will get to this later on…   fprintf(f, …);   … }   At the final brace, f falls out of scope, the internal reference count in f goes to zero,

inline void cv::Ptr<FILE>::delete_obj() {     fclose(obj);  } Then you could go ahead and use that pointer to open a file, do whatever with it, and later, just let the pointer fall out of scope (at which time the file handle would automatically be closed for you): {   Ptr<FILE> f(fopen("myfile.txt", "r"));   if(f.empty())     throw …;  // Throw an exception, we will get to this later on…   fprintf(f, …);   … }   At the final brace, f falls out of scope, the internal reference count in f goes to zero, delete_obj() is called by f’s destructor, and (thus) fclose() is called on the file handle pointer (stored in obj). A tip for gurus: a serious programmer might worry that the incrementing and decrementing of the reference count might not be sufficiently atomic for the Ptr<> template to be safe in multithreaded applications. This, however, is not the case, and Ptr<> is thread safe. Similarly, the other reference counting objects in OpenCV are all thread-safe in this same sense. class cv::Exception and Exception

is called by f’s destructor, and (thus) fclose() is called on the file handle pointer (stored in obj). A tip for gurus: a serious programmer might worry that the incrementing and decrementing of the reference count might not be sufficiently atomic for the Ptr<> template to be safe in multithreaded applications. This, however, is not the case, and Ptr<> is thread safe. Similarly, the other reference counting objects in OpenCV are all thread-safe in this same sense. class cv::Exception and Exception Handling  OpenCV uses exceptions to handle errors. OpenCV defines its own exception type cv::Exception, which is derived from the STL exception class std::exception. Really, this exception type has nothing special about it, other than being in the cv:: namespace and so distinguishable from other objects that are also derived from std::exception.  The type cv::Exception has members code, err, func, file, and line, which are (respectively) a numerical error code, a string indicating the nature of the error that

OpenCV uses exceptions to handle errors. OpenCV defines its own exception type cv::Exception, which is derived from the STL exception class std::exception. Really, this exception type has nothing special about it, other than being in the cv:: namespace and so distinguishable from other objects that are also derived from std::exception.  The type cv::Exception has members code, err, func, file, and line, which are (respectively) a numerical error code, a string indicating the nature of the error that generated the exception, the name of the function in which the error occurred, the file in which the error occurred, and an integer indicating the line on which the error occurred in that file. err, func, and file are all STL strings.  There are several built-in macros for generating exceptions yourself. CV_Error( errorcode, description ) will generate and throw an exception with a fixed text description. CV_Error_( errorcode, printf_fmt_str, [printf-args] ) works the same, but allows you to replace the fixed

the exception, the name of the function in which the error occurred, the file in which the error occurred, and an integer indicating the line on which the error occurred in that file. err, func, and file are all STL strings.  There are several built-in macros for generating exceptions yourself. CV_Error( errorcode, description ) will generate and throw an exception with a fixed text description. CV_Error_( errorcode, printf_fmt_str, [printf-args] ) works the same, but allows you to replace the fixed description with a printf-like format string and arguments. Finally, there is CV_Assert( condition ) and CV_DbgAssert( condition ). Both will test your condition and throw an exception if the condition is not met. The latter version, however, will only operate in debug builds. These macros are the strongly preferred method of throwing exceptions, as they will automatically take care of the fields func, file, and line for you. The cv::DataType<> Template  When OpenCV library functions need to communicate the

with a printf-like format string and arguments. Finally, there is CV_Assert( condition ) and CV_DbgAssert( condition ). Both will test your condition and throw an exception if the condition is not met. The latter version, however, will only operate in debug builds. These macros are the strongly preferred method of throwing exceptions, as they will automatically take care of the fields func, file, and line for you. The cv::DataType<> Template  When OpenCV library functions need to communicate the concept of a particular data type, they do so by creating an object of type cv::DataType<>. cv::DataType<> itself is a template, and so the actual objects passed around are specializations of this template. This is an example of what in C++ are generally called traits. This allows the cv::DataType<> object to both contain runtime information about the                                                              10 In this case, by FILE we mean struct FILE, as defined in the C standard library.

type, as well as to contain typedef statements in its own definition that allow it to refer to the same type at compile time. This might sound a bit confusing, and it is, but that is an inevitable consequence of trying to mix runtime information and compile-time information in C++.11 An example will help clarify. The template class definition for DataType is the following: template<typename _Tp> class DataType {   typedef _Tp        value_type;   typedef value_type work_type;    typedef value_type channel_type;   typedef value_type vec_type;    enum {      generic_type = 1,      depth        = -1,      channels     = 1,      fmt          = 0,     type         = CV_MAKETYPE(depth, channels)    }; }; Let’s try to understand what this means, and then follow it with an example. First, we can see that cv::DataType<> is a template, and expects to be specialized to a class called _Tp. It then has four typedef statements that allow the type of the cv::DataType<>, as well as some other related types, to be extracted

typedef value_type vec_type;    enum {      generic_type = 1,      depth        = -1,      channels     = 1,      fmt          = 0,     type         = CV_MAKETYPE(depth, channels)    }; }; Let’s try to understand what this means, and then follow it with an example. First, we can see that cv::DataType<> is a template, and expects to be specialized to a class called _Tp. It then has four typedef statements that allow the type of the cv::DataType<>, as well as some other related types, to be extracted from the cv::DataType<> instantiated object at compile time. In the template definition, these are all the same, but we will see in our example of a specialization of the template that they do not have to be (and often should not be). The next section is an enum that contains several components.12 These are the generic_type, the depth, the number of channels, the format fmt, and the type. To see what all of these things mean, we’ll look at two example specializations of cv::DataType<>, from core.hpp. The first

the cv::DataType<> instantiated object at compile time. In the template definition, these are all the same, but we will see in our example of a specialization of the template that they do not have to be (and often should not be). The next section is an enum that contains several components.12 These are the generic_type, the depth, the number of channels, the format fmt, and the type. To see what all of these things mean, we’ll look at two example specializations of cv::DataType<>, from core.hpp. The first is the cv::DataType<> definition for float: template<> class DataType<float> { public:   typedef float      value_type;   typedef value_type work_type;   typedef value_type channel_type;   typedef value_type vec_type;    enum {      generic_type = 0,      depth        = DataDepth<channel_type>::value,      channels     = 1,     fmt          = DataDepth<channel_type>::fmt,     type         = CV_MAKETYPE(depth, channels)    }; }; The first thing to notice is that this is a definition for a C++ built-in type.

is the cv::DataType<> definition for float: template<> class DataType<float> { public:   typedef float      value_type;   typedef value_type work_type;   typedef value_type channel_type;   typedef value_type vec_type;    enum {      generic_type = 0,      depth        = DataDepth<channel_type>::value,      channels     = 1,     fmt          = DataDepth<channel_type>::fmt,     type         = CV_MAKETYPE(depth, channels)    }; }; The first thing to notice is that this is a definition for a C++ built-in type. It is useful to have such definitions for the built-in types, but we can also make them for more complicated objects. In this case, the value_type is of course float, and the work_type, channel_type, and vec_type are all the same. We will see more clearly what these are for in the next example. For the constants in the enum, this                                                              11 You don’t have this sort of problem in languages (e.g., Python) that support variable introspection and have an

It is useful to have such definitions for the built-in types, but we can also make them for more complicated objects. In this case, the value_type is of course float, and the work_type, channel_type, and vec_type are all the same. We will see more clearly what these are for in the next example. For the constants in the enum, this                                                              11 You don’t have this sort of problem in languages (e.g., Python) that support variable introspection and have an intrinsic runtime concept of data types. 12 If this construct is awkward to you, remember that you can always assign integer values to the “options” in an enum declaration. In effect, this is a way of stashing a bunch of integer constants that will be fixed at compile time.

example will do just fine. The first variable generic_type is set to 0, as it is zero for all types defined in core.hpp. The depth variable is the data type identifier used by OpenCV. For example, cv::DataDepth<float>::value resolves to the constant cv::F32. The entry channels is one because float is just a single number; we will see an alternative to this in the next example. The variable fmt gives a single character representation of the format. In this case, cv::DataDepth<float>::fmt resolves to f. The last entry is type, which is a representation similar to depth, but which includes the number of channels (in this case, one). CV_MAKETYPE(cv::F32,1) resolves to cv::F32C1. The important thing about DataType<>, however, is to communicate the nature of more complicated constructs. This is essential, for example, for allowing algorithms to be implemented in a manner that is agnostic to the incoming data type (i.e., algorithms that use introspection to determine how to proceed with incoming data). Consider the

entry is type, which is a representation similar to depth, but which includes the number of channels (in this case, one). CV_MAKETYPE(cv::F32,1) resolves to cv::F32C1. The important thing about DataType<>, however, is to communicate the nature of more complicated constructs. This is essential, for example, for allowing algorithms to be implemented in a manner that is agnostic to the incoming data type (i.e., algorithms that use introspection to determine how to proceed with incoming data). Consider the example of an instantiation of cv::DataType<> for a cv::Rect<> (itself containing an as yet unspecialized type _Tp): template<typename _Tp> class DataType<Rect_<_Tp> > { public:   typedef Rect_<_Tp>                               value_type;   typedef Rect_<typename DataType<_Tp>::work_type> work_type;   typedef _Tp                                      channel_type;    typedef Vec<channel_type, channels>              vec_type;    enum {      generic_type = 0,      depth        = DataDepth<channel_type>::value,

of an instantiation of cv::DataType<> for a cv::Rect<> (itself containing an as yet unspecialized type _Tp): template<typename _Tp> class DataType<Rect_<_Tp> > { public:   typedef Rect_<_Tp>                               value_type;   typedef Rect_<typename DataType<_Tp>::work_type> work_type;   typedef _Tp                                      channel_type;    typedef Vec<channel_type, channels>              vec_type;    enum {      generic_type = 0,      depth        = DataDepth<channel_type>::value,      channels     = 4,     fmt          = ((channels-1)<<8) + DataDepth<channel_type>::fmt,     type         = CV_MAKETYPE(depth, channels)    }; }; This is a much more complicated example. First, notice that cv::Rect itself does not appear. You will recall that earlier we mentioned that cv::Rect was actually an alias for a template, and that template is called cv::Rect_<>. So this template could be specialized as cv::DataType<Rect> or, for example, cv::DataType< Rect_<float> >. For the case cv::DataType<Rect>,

channels     = 4,     fmt          = ((channels-1)<<8) + DataDepth<channel_type>::fmt,     type         = CV_MAKETYPE(depth, channels)    }; }; This is a much more complicated example. First, notice that cv::Rect itself does not appear. You will recall that earlier we mentioned that cv::Rect was actually an alias for a template, and that template is called cv::Rect_<>. So this template could be specialized as cv::DataType<Rect> or, for example, cv::DataType< Rect_<float> >. For the case cv::DataType<Rect>, we recall that all of the elements are integers, so if we consider that case, all of the instantiations of the template parameter _Tp resolve to int. We can see that the value_type is just the compile-time name of the thing that the cv::DataType<> is describing (namely Rect). The work_type, however, is defined to be the work_type of cv::DataType<int> (which, not surprisingly is int). What we see is that the work_type is telling us what kind of variables the cv::DataType<> is made of (i.e., what we “do

we recall that all of the elements are integers, so if we consider that case, all of the instantiations of the template parameter _Tp resolve to int. We can see that the value_type is just the compile-time name of the thing that the cv::DataType<> is describing (namely Rect). The work_type, however, is defined to be the work_type of cv::DataType<int> (which, not surprisingly is int). What we see is that the work_type is telling us what kind of variables the cv::DataType<> is made of (i.e., what we “do work” on). The channel type is also int. This means that if we want to represent this variable as a multichannel object, it should be represented as some number of int objects. Finally, just as channel_type tells us how to represent this cv::DataType<> as a multichannel object, vec_type tells us how to represent it as an object of type cv::Vec<>. The alias cv::DataType<Rect>::vec_type will resolve to cv::Vec<int,4>. Moving on to the runtime constants: generic_type is again 0, depth is CV::S32, channels is 4

on). The channel type is also int. This means that if we want to represent this variable as a multichannel object, it should be represented as some number of int objects. Finally, just as channel_type tells us how to represent this cv::DataType<> as a multichannel object, vec_type tells us how to represent it as an object of type cv::Vec<>. The alias cv::DataType<Rect>::vec_type will resolve to cv::Vec<int,4>. Moving on to the runtime constants: generic_type is again 0, depth is CV::S32, channels is 4 (because there are actually four values, the same reason the vec_type instantiated to a cv::Vec<> of size 4), fmt resolves to 0x3069 (since i is 0x69), and type resolves to cv::S32C4. class InputArray and class OutputArray  Many OpenCV functions take arrays as arguments and return arrays as return values, but in OpenCV, there are many kinds of arrays. We have already seen that OpenCV supports some small array types (cv::Scalar, cv::Vec, cv::Matx), and STL’s std::vector<> in addition to the large array types in

there are actually four values, the same reason the vec_type instantiated to a cv::Vec<> of size 4), fmt resolves to 0x3069 (since i is 0x69), and type resolves to cv::S32C4. class InputArray and class OutputArray  Many OpenCV functions take arrays as arguments and return arrays as return values, but in OpenCV, there are many kinds of arrays. We have already seen that OpenCV supports some small array types (cv::Scalar, cv::Vec, cv::Matx), and STL’s std::vector<> in addition to the large array types in the next section (cv::Mat and cv::SparseMat). In order to keep the interface from becoming onerously complicated (and repetitive), OpenCV defines the types cv::InputArray and

cv::OutputArray. In effect, these types mean “any of the above” with respect to the many array forms supported by the library. There is even a cv::InputOutputArray, specifying an array for in place computation.  The primary difference between cv::InputArray and cv::OutputArray is that the former is assumed to be const (i.e., read only). You will typically see these two types used in the definitions of library routines. You will not tend to use them yourself, but when being introduced to library functions, their presence means that you can use any array type, including a single cv::Scalar, and the result should be what you expect. Related to cv::InputArray is the special function cv::noArray() that returns cv::InputArray. The returned object can be passed to any input requiring cv::InputArray to indicate that this input is not being used. Large Array Types Finally, our journey brings us to the large array types. Chief among these is cv::Mat, which could be considered the epicenter of the entire C++

their presence means that you can use any array type, including a single cv::Scalar, and the result should be what you expect. Related to cv::InputArray is the special function cv::noArray() that returns cv::InputArray. The returned object can be passed to any input requiring cv::InputArray to indicate that this input is not being used. Large Array Types Finally, our journey brings us to the large array types. Chief among these is cv::Mat, which could be considered the epicenter of the entire C++ implementation of the OpenCV library. The overwhelming majority of functions in the OpenCV library are members of the cv::Mat class or take a cv::Mat as an argument or return cv::Mat as a return value.  If you are familiar with the C interface (pre-version 2.1 implementation) of the OpenCV library, you will remember the data types IplImage and CvMat. You might also recall CvArr. In the C++ implementation, these are all gone, replaced with cv::Mat. This means no more dubious casting of void* pointers in function

of the OpenCV library. The overwhelming majority of functions in the OpenCV library are members of the cv::Mat class or take a cv::Mat as an argument or return cv::Mat as a return value.  If you are familiar with the C interface (pre-version 2.1 implementation) of the OpenCV library, you will remember the data types IplImage and CvMat. You might also recall CvArr. In the C++ implementation, these are all gone, replaced with cv::Mat. This means no more dubious casting of void* pointers in function arguments, and in general a tremendous enhancement in the cleanliness of the library internally. The cv::Mat class is used to represent dense arrays of any number of dimensions. In this context, dense means that for every entry in the array, there is a data value stored in memory corresponding to that entry, even if that entry is zero. Most images, for example, are stored as dense arrays. The alternative would be a sparse array. In the case of a sparse array, only nonzero entries are typically stored. This can

and in general a tremendous enhancement in the cleanliness of the library internally. The cv::Mat class is used to represent dense arrays of any number of dimensions. In this context, dense means that for every entry in the array, there is a data value stored in memory corresponding to that entry, even if that entry is zero. Most images, for example, are stored as dense arrays. The alternative would be a sparse array. In the case of a sparse array, only nonzero entries are typically stored. This can result in a great savings of storage space if many of the entries are in fact zero, but can be very wasteful if the array is relatively dense. A common case for using a sparse array rather than a dense array would be a histogram. For many histograms, most of the entries are zero, and storing all those zeros is not necessary. For the case of sparse arrays, OpenCV has the alternative data structure cv::SparseMat. class cv::Mat: N-Dimensional Dense Arrays  The class cv::Mat can be used for arrays of any number of

in a great savings of storage space if many of the entries are in fact zero, but can be very wasteful if the array is relatively dense. A common case for using a sparse array rather than a dense array would be a histogram. For many histograms, most of the entries are zero, and storing all those zeros is not necessary. For the case of sparse arrays, OpenCV has the alternative data structure cv::SparseMat. class cv::Mat: N-Dimensional Dense Arrays  The class cv::Mat can be used for arrays of any number of dimensions. The data is stored in the array in what can be thought of as an n-dimensional analog of “raster scan order.” This means that in a one-dimensional array, the elements are sequential. In a two-dimensional array, the data is organized into rows, and each row appears one after the other. For three-dimensional arrays, each plane is filled out row by row, and then the planes are packed one after the other. Each matrix contains an element signaling the contents of the array called flags, an element that

The data is stored in the array in what can be thought of as an n-dimensional analog of “raster scan order.” This means that in a one-dimensional array, the elements are sequential. In a two-dimensional array, the data is organized into rows, and each row appears one after the other. For three-dimensional arrays, each plane is filled out row by row, and then the planes are packed one after the other. Each matrix contains an element signaling the contents of the array called flags, an element that indicates the number of dimensions called dims, two elements that indicate the number of rows and columns called rows and cols (these are not valid for dims>2), a pointer to where the array data is stored called data, and a reference counter analogous to the reference counter used by cv::Ptr<> called refcount. This latter member allows cv::Mat to behave very much like a smart pointer for the data contained in data. The memory layout in data is described by the array step[]. The data array is laid out such that the

the number of dimensions called dims, two elements that indicate the number of rows and columns called rows and cols (these are not valid for dims>2), a pointer to where the array data is stored called data, and a reference counter analogous to the reference counter used by cv::Ptr<> called refcount. This latter member allows cv::Mat to behave very much like a smart pointer for the data contained in data. The memory layout in data is described by the array step[]. The data array is laid out such that the address of an element whose indices are given by   i0,†ii,†…,iNd−1() is:    &mtxi0,i1,…,†iNd−1Nd()=mtx.data+mtx.step0⎡⎣⎤⎦*i0+mtx.step1⎡⎣⎤⎦*i1+…+mtx.stepNd−1⎡⎣⎤⎦*iNd−1

In the simple case of a two-dimensional array, this reduces to:    &mtxi,j()=mtx.data+mtx.step0⎡⎣⎤⎦*i+mtx.step1⎡⎣⎤⎦*j

The data itself contained in cv::Mat is not required to be simple primitives. Each element of the data in a cv::Mat can itself be either a single number, or multiple numbers. In the case of multiple numbers, this is what the library refers to as a multichannel array. In fact, an n-dimensional array and an (n-1)-dimensional multichannel array are actually very similar objects, but because of the frequency of occasions in which it is useful to be able to think of an array as a vector valued array, the library contains special provisions for such structures.13 One reason why this distinction is made is because of memory access. By definition, an element of an array is the part that may be vector-valued. For example, an array might be said to be a two-dimensional three-channel array of 32-bit floats; in this case, the element of the array is the three 32-bit floats with a size of 12 bytes. When laid out in memory, rows of an array may not be absolutely sequential; there may be small gaps that buffer each row

contains special provisions for such structures.13 One reason why this distinction is made is because of memory access. By definition, an element of an array is the part that may be vector-valued. For example, an array might be said to be a two-dimensional three-channel array of 32-bit floats; in this case, the element of the array is the three 32-bit floats with a size of 12 bytes. When laid out in memory, rows of an array may not be absolutely sequential; there may be small gaps that buffer each row before the next. The difference between an  n-dimensional single-channel array and an  n−1()-dimensional multichannel array is that this padding will always occur at the end of full rows (i.e., the channels in an element will always be sequential).  Creating an Array  An array can be created simply by instantiating a variable of type cv::Mat. An array created in this manner has no size and no data type. It can, however, later be asked to allocate data by using a member function such as create(). One variation

the next. The difference between an  n-dimensional single-channel array and an  n−1()-dimensional multichannel array is that this padding will always occur at the end of full rows (i.e., the channels in an element will always be sequential).  Creating an Array  An array can be created simply by instantiating a variable of type cv::Mat. An array created in this manner has no size and no data type. It can, however, later be asked to allocate data by using a member function such as create(). One variation of create() takes as arguments a number of rows, a number of columns, and a type, and configures the array to represent a two-dimensional object. The type of an array determines what kind of elements it has. Valid types in this context specify both the fundamental type of element as well as the number of channels. All such types are defined in the library header, and have the form cv::{U8,S16,U16,S32,F32,F64}C{1,2,3}.14 For example, cv::F32C3 would imply a 32-bit floating-point three-channel array. You can

of create() takes as arguments a number of rows, a number of columns, and a type, and configures the array to represent a two-dimensional object. The type of an array determines what kind of elements it has. Valid types in this context specify both the fundamental type of element as well as the number of channels. All such types are defined in the library header, and have the form cv::{U8,S16,U16,S32,F32,F64}C{1,2,3}.14 For example, cv::F32C3 would imply a 32-bit floating-point three-channel array. You can also specify these things when you first allocate the matrix if you prefer. There are many constructors for cv::Mat, one of which takes the same arguments as create() (and an optional fourth argument with which to initialize all of the elements in your new array).  cv::Mat m; m.create( 3, 10, cv::F32C3 );        // 3 rows, 10 columns of 3-channel 32-bit floats m.setTo( cv::Scalar( 1.0f, 0.0f, 1.0f ) );   // 1st channel is 1.0, 2nd 0.0, 3rd 1.0 is equivalent to  cv::Mat m( 3, 10, cv::F32C3, cv::Scalar(

can also specify these things when you first allocate the matrix if you prefer. There are many constructors for cv::Mat, one of which takes the same arguments as create() (and an optional fourth argument with which to initialize all of the elements in your new array).  cv::Mat m; m.create( 3, 10, cv::F32C3 );        // 3 rows, 10 columns of 3-channel 32-bit floats m.setTo( cv::Scalar( 1.0f, 0.0f, 1.0f ) );   // 1st channel is 1.0, 2nd 0.0, 3rd 1.0 is equivalent to  cv::Mat m( 3, 10, cv::F32C3, cv::Scalar( 1.0f, 0.0f, 1.0f ) ); It is important to understand that the data in an array is not attached rigidly to the array object. The cv::Mat object is really a header for a data area, which—in principle—is an entirely separate thing. For example, it is possible to assign one matrix n to another matrix m (i.e., m=n). In this case, the data pointer inside of m will be changed to point to the same data as n. The data pointed to previously by the data                                                              13

1.0f, 0.0f, 1.0f ) ); It is important to understand that the data in an array is not attached rigidly to the array object. The cv::Mat object is really a header for a data area, which—in principle—is an entirely separate thing. For example, it is possible to assign one matrix n to another matrix m (i.e., m=n). In this case, the data pointer inside of m will be changed to point to the same data as n. The data pointed to previously by the data                                                              13 Pre-2.1 OpenCV array types had an explicit element IplImage::nChannels, which indicated the number of channels. Because of the more general way in which such concepts are captured in the cv::Mat object, this information is no longer directly stored in a class variable. Rather, it is returned by a member function cv::channels(). 14 OpenCV allows for arrays with more than three channels, but to construct one of these, you will have to call one of the functions cv::{U8,S16,U16,S32,F32,F64}C(). These functions

Pre-2.1 OpenCV array types had an explicit element IplImage::nChannels, which indicated the number of channels. Because of the more general way in which such concepts are captured in the cv::Mat object, this information is no longer directly stored in a class variable. Rather, it is returned by a member function cv::channels(). 14 OpenCV allows for arrays with more than three channels, but to construct one of these, you will have to call one of the functions cv::{U8,S16,U16,S32,F32,F64}C(). These functions take a single argument, which is the number of channels. So cv::U8C(3) is equivalent to cv::U8C3, but since there is no macro for cv::U8C7, to get this you would have to call cv::U8C(7).

element of m (if any) will be deallocated15. At the same time, the reference counter for the data area that they both now share will be incremented. Last, but not least, the members of m that characterize the data in m (such as rows, cols, and flags) will be updated to accurately describe the data now pointed to by data in m. This all results in a very convenient behavior, in which arrays can be assigned to one another, and the work necessary to do this is done automatically behind the scenes to give the correct result. The following is a complete list of the constructors available for cv::Mat. The list appears rather unwieldy, but in fact you will only use a small fraction of these most of the time. Having said that, when you need one of the more obscure ones, you will probably find that you are glad it is there. Table 3-10: cv::Mat constructors that do not copy data Constructor Description cv::Mat(); Default constructor cv::Mat( int rows, int cols, int type ); Two-dimensional arrays by type cv::Mat( int

correct result. The following is a complete list of the constructors available for cv::Mat. The list appears rather unwieldy, but in fact you will only use a small fraction of these most of the time. Having said that, when you need one of the more obscure ones, you will probably find that you are glad it is there. Table 3-10: cv::Mat constructors that do not copy data Constructor Description cv::Mat(); Default constructor cv::Mat( int rows, int cols, int type ); Two-dimensional arrays by type cv::Mat( int rows, int cols, int type,      const Scalar& s ); Two-dimensional arrays by type with initialization value cv::Mat( int rows, int cols, int type,      void* data, size_t step=AUTO_STEP ); Two-dimensional arrays by type with preexisting data cv::Mat( cv::Size sz, int type ); Two-dimensional arrays by type (size in sz) cv::Mat( cv::Size sz,       int type, const Scalar& s ); Two-dimensional arrays by type with initializtion value (size in sz) cv::Mat( cv::Size sz, int type,      void* data, size_t

int rows, int cols, int type,      const Scalar& s ); Two-dimensional arrays by type with initialization value cv::Mat( int rows, int cols, int type,      void* data, size_t step=AUTO_STEP ); Two-dimensional arrays by type with preexisting data cv::Mat( cv::Size sz, int type ); Two-dimensional arrays by type (size in sz) cv::Mat( cv::Size sz,       int type, const Scalar& s ); Two-dimensional arrays by type with initializtion value (size in sz) cv::Mat( cv::Size sz, int type,      void* data, size_t step=AUTO_STEP ); Two-dimensional arrays by type with preexisting data (size in sz) cv::Mat( int ndims, const int* sizes,      int type ); Multidimensional arrays by type  cv::Mat( int ndims, const int* sizes,      int type, const Scalar& s ); Multidimensional arrays by type with initialization value cv::Mat( int ndims, const int* sizes,      int type,       void* data, size_t step=AUTO_STEP ); Multidimensional arrays by type with preexisting data Table 3-10 lists the basic constructors for the cv::Mat object.

); Two-dimensional arrays by type with preexisting data (size in sz) cv::Mat( int ndims, const int* sizes,      int type ); Multidimensional arrays by type  cv::Mat( int ndims, const int* sizes,      int type, const Scalar& s ); Multidimensional arrays by type with initialization value cv::Mat( int ndims, const int* sizes,      int type,       void* data, size_t step=AUTO_STEP ); Multidimensional arrays by type with preexisting data Table 3-10 lists the basic constructors for the cv::Mat object. Other than the default constructor, these fall into three basic categories: those that take a number of rows and a number of columns to create a two-dimensional array, those that use a cv::Size object to create a two-dimensional array, and those that construct  n-dimensional arrays and require you to specify the number of dimensions and pass in an array of integers specifying the size of each of the dimensions.                                                              15 Technically, it will only be deallocated if

the default constructor, these fall into three basic categories: those that take a number of rows and a number of columns to create a two-dimensional array, those that use a cv::Size object to create a two-dimensional array, and those that construct  n-dimensional arrays and require you to specify the number of dimensions and pass in an array of integers specifying the size of each of the dimensions.                                                              15 Technically, it will only be deallocated if m was the last cv::Mat around which pointed to that particular data.

In addition, some of these allow you to initialize the data, either by providing a cv::Scalar (in which case, the entire array will be initialized to that value), or by providing a pointer to an appropriate data block that can be used by the array. In this latter case, you are essentially just creating a header to the existing data (i.e., no data is copied; the data member is set to point to the data indicated by the data argument). Table 3-11: cv::Mat constructors that copy data from other cv::Mat’s Constructor Description cv::Mat( const Mat& mat ); Copy constructor cv::Mat( const Mat& mat,      const cv::Range& rows,       const cv::Range& cols ); Copy constructor that copies only a subset of rows and columns cv::Mat( const Mat& mat,      const cv::Rect& roi ); Copy constructor that copies only a subset of rows and columns specified by a region of interest cv::Mat( const Mat& mat,      const cv::Range* ranges ); Generalized region of interest copy constructor that uses an array of ranges to select from an

Description cv::Mat( const Mat& mat ); Copy constructor cv::Mat( const Mat& mat,      const cv::Range& rows,       const cv::Range& cols ); Copy constructor that copies only a subset of rows and columns cv::Mat( const Mat& mat,      const cv::Rect& roi ); Copy constructor that copies only a subset of rows and columns specified by a region of interest cv::Mat( const Mat& mat,      const cv::Range* ranges ); Generalized region of interest copy constructor that uses an array of ranges to select from an n-dimensional array cv::Mat( const cv::MatExpr& expr ); Copy constructor that initializes m with the result of an algebraic expression of other matrices The copy constructors (Table 3-11) show how to create an array from another array. In addition to the basic copy constructor, there are three methods for constructing an array from a sub-region of an existing array and one constructor that initializes the new matrix using the result of some matrix expression. The sub-region (also known as “region of interest”)

array cv::Mat( const cv::MatExpr& expr ); Copy constructor that initializes m with the result of an algebraic expression of other matrices The copy constructors (Table 3-11) show how to create an array from another array. In addition to the basic copy constructor, there are three methods for constructing an array from a sub-region of an existing array and one constructor that initializes the new matrix using the result of some matrix expression. The sub-region (also known as “region of interest”) constructors come in three flavors: one that takes a range of rows and a range of columns (this works only on a two-dimensional matrix), one that uses a cv::Rect to specify a rectangular sub-region (also works only on a two-dimensional matrix), and a final one that takes an array of ranges. In this latter case, the number of valid ranges pointed to by the pointer argument ranges is required to be equal to the number of dimensions of the array mat. It is this third option that you must use if mat is a

constructors come in three flavors: one that takes a range of rows and a range of columns (this works only on a two-dimensional matrix), one that uses a cv::Rect to specify a rectangular sub-region (also works only on a two-dimensional matrix), and a final one that takes an array of ranges. In this latter case, the number of valid ranges pointed to by the pointer argument ranges is required to be equal to the number of dimensions of the array mat. It is this third option that you must use if mat is a multidimensional array with ndim greater than 2.  Table 3-12: cv::Mat constructors for pre-version 2.1 data types Constructor Description cv::Mat( const CvMat* old,       bool copyData=false ); Constructor for m that creates m from an old-style CvMat, with optional data copy cv::Mat( const IplImage* old,       bool copyData=false ); Constructor for m that creates m from an old-style IplImage, with optional data copy If you are modernizing or maintaining pre-version 2.1 code that still contains the C-style data

array with ndim greater than 2.  Table 3-12: cv::Mat constructors for pre-version 2.1 data types Constructor Description cv::Mat( const CvMat* old,       bool copyData=false ); Constructor for m that creates m from an old-style CvMat, with optional data copy cv::Mat( const IplImage* old,       bool copyData=false ); Constructor for m that creates m from an old-style IplImage, with optional data copy If you are modernizing or maintaining pre-version 2.1 code that still contains the C-style data structures, you may want to create a new C++-style cv::Mat structure from an existing CvMat or IplImage structure. In this case, you have two options. You can construct a header on the existing data (by setting copyData to false) or you can set copyData to true (in which case, new memory will be allocated for m and all of the data from old will be copied into m).

These objects do a lot more for you than you might realize at first. In particular, they allow for expressions that mix the C++ and C data-types by functioning as implicit constructors for the C++ data types on demand. Thus, it is possible to simply use a pointer to one of the C structures wherever a cv::Mat is expected and have a reasonable expectation that things will work out correctly. (This is why the copyData member defaults to false.) In addition to these constructors, there are corresponding cast operators that will convert a cv::Mat into CvMat or IplImage on demand. These also do not copy data. Table 3-13: cv::Mat template constructors Constructor Description cv::Mat( const cv::Vec<T,n>& vec,       bool copyData=true ); Construct a one-dimensional array of type T and size n from a cv::Vec of the same type cv::Mat(      const cv::Matx<T,m,n>& vec,       bool copyData=true ); Construct a two-dimensional array of type T and size m-by-n from a cv::Matx of the same type cv::Mat( const std::vector<T>& vec,

that will convert a cv::Mat into CvMat or IplImage on demand. These also do not copy data. Table 3-13: cv::Mat template constructors Constructor Description cv::Mat( const cv::Vec<T,n>& vec,       bool copyData=true ); Construct a one-dimensional array of type T and size n from a cv::Vec of the same type cv::Mat(      const cv::Matx<T,m,n>& vec,       bool copyData=true ); Construct a two-dimensional array of type T and size m-by-n from a cv::Matx of the same type cv::Mat( const std::vector<T>& vec,      bool copyData=true ); Construct an one-dimensional array of type T from an STL vector containing elements of the same type The last set of constructors is the template constructors. These constructors are template not because they construct a template form of cv::Mat, but because they construct an instance of cv::Mat from something that is itself template. These constructors allow either an arbitrary cv::Vec<> or cv::Matx<> to be used to create a cv::Mat array of corresponding dimension and type, or to use

bool copyData=true ); Construct an one-dimensional array of type T from an STL vector containing elements of the same type The last set of constructors is the template constructors. These constructors are template not because they construct a template form of cv::Mat, but because they construct an instance of cv::Mat from something that is itself template. These constructors allow either an arbitrary cv::Vec<> or cv::Matx<> to be used to create a cv::Mat array of corresponding dimension and type, or to use an STL vector<> object of arbitrary type to construct an array of that same type. Table 3-14: static functions that create cv::Mat Function Description cv::Mat::zeros( rows, cols, type ); Create a cv::Mat of size rows-by-cols, which is full of zeros, with type type (cv::F32, etc.) cv::Mat::ones( rows, cols, type ); Create a cv::Mat of size rows-by-cols, which is full of ones, with type type (cv::F32, etc.) cv::Mat::eye( rows, cols, type ); Create a cv::Mat of size rows-by-cols, which is an identity matrix,

an STL vector<> object of arbitrary type to construct an array of that same type. Table 3-14: static functions that create cv::Mat Function Description cv::Mat::zeros( rows, cols, type ); Create a cv::Mat of size rows-by-cols, which is full of zeros, with type type (cv::F32, etc.) cv::Mat::ones( rows, cols, type ); Create a cv::Mat of size rows-by-cols, which is full of ones, with type type (cv::F32, etc.) cv::Mat::eye( rows, cols, type ); Create a cv::Mat of size rows-by-cols, which is an identity matrix, with type type (cv::F32, etc.)

The class cv::Mat also provides a number of static member functions to create certain kinds of commonly used arrays. These include functions like zeros(), ones(), and eye(), which construct a matrix full of zeros, a matrix full of ones, or an identity matrix, respectively.16 Accessing Array Elements Individually   There are several ways to access a matrix, which are designed to be convenient in different contexts. In recent versions of OpenCV, however, a great deal of effort has been invested to make them all comparably, if not identically, efficient. The two primary options for accessing individual elements are to access them by location, or to access them through iteration. The basic means of direct access is the (template) member function at<>()17. There are many variations of this function that take different arguments for arrays of different numbers of dimensions. The way this function works is that you specialize the at<>() template to the type of data that the matrix contains, then access that element

comparably, if not identically, efficient. The two primary options for accessing individual elements are to access them by location, or to access them through iteration. The basic means of direct access is the (template) member function at<>()17. There are many variations of this function that take different arguments for arrays of different numbers of dimensions. The way this function works is that you specialize the at<>() template to the type of data that the matrix contains, then access that element using the row and column locations of the data you want. Here is a simple example: cv::Mat m = cv::Mat::eye( 10, 10, 32FC1 ); printf(   “Element (3,3) is %f\n”,    m.at<float>(3,3) //(row, col) ); For a multichannel array, the analogous example would look like this: cv::Mat m = cv::Mat::eye( 10, 10, 32FC2 ); printf(   “Element (3,3) is (%f,%f)\n”,    m.at<cv::Vec2f>(3,3)[0],   m.at<cv::Vec2f>(3,3)[1] ); Note that when you want to specify a template function like at<>() to operate on a multichannel array, the

the row and column locations of the data you want. Here is a simple example: cv::Mat m = cv::Mat::eye( 10, 10, 32FC1 ); printf(   “Element (3,3) is %f\n”,    m.at<float>(3,3) //(row, col) ); For a multichannel array, the analogous example would look like this: cv::Mat m = cv::Mat::eye( 10, 10, 32FC2 ); printf(   “Element (3,3) is (%f,%f)\n”,    m.at<cv::Vec2f>(3,3)[0],   m.at<cv::Vec2f>(3,3)[1] ); Note that when you want to specify a template function like at<>() to operate on a multichannel array, the best way to do this is to use a cv::Vec<> object (either a premade alias or the template form).  Similar to the vector case, you can create an array made of a more sophisticated type, such as complex numbers:  cv::Mat m = cv::Mat::eye( 10, 10, cv::DataType<cv::Complexf>::type ); printf(   “Element (3,3) is %f + i%f\n”,    m.at<cv::Complexf>(3,3).re,    m.at<cv::Complexf>(3,3).im,  ); It is also worth noting the use of the cv::DataType<> template here. The matrix constructor requires a runtime value that is a

way to do this is to use a cv::Vec<> object (either a premade alias or the template form).  Similar to the vector case, you can create an array made of a more sophisticated type, such as complex numbers:  cv::Mat m = cv::Mat::eye( 10, 10, cv::DataType<cv::Complexf>::type ); printf(   “Element (3,3) is %f + i%f\n”,    m.at<cv::Complexf>(3,3).re,    m.at<cv::Complexf>(3,3).im,  ); It is also worth noting the use of the cv::DataType<> template here. The matrix constructor requires a runtime value that is a variable of type int that happens to take on some “magic” values that the constructor understands. By contrast, cv::Complexf is an actual object type, a purely compile-time construct. The need to generate one of these representations (runtime) from the other (compile time) is precisely why the cv::DataType<> template exists. Table 3-15 lists the available variations of the at<>() template. Table 3-15: Variations of the at<>() accessor function Example Description M.at<int>( i ); Element i from integer array M

of type int that happens to take on some “magic” values that the constructor understands. By contrast, cv::Complexf is an actual object type, a purely compile-time construct. The need to generate one of these representations (runtime) from the other (compile time) is precisely why the cv::DataType<> template exists. Table 3-15 lists the available variations of the at<>() template. Table 3-15: Variations of the at<>() accessor function Example Description M.at<int>( i ); Element i from integer array M                                                               16 In the case of cv::Mat::eye() and cv::Mat::ones(), if the array created is multi-channel, only the first channel will be set 1.0 while the other channels will be 0.0. 17 For two channels, the order is (row, col),  at<>(row, col).

M.at<float>( i, j ); Element ( i, j ) (row, col) from float array M M.at<int>( pt ); Element at location (pt.x, pt.y) in integer matrix M M.at<float>( i, j, k ); Element at location ( i, j, k ) in three-dimensional float array M M.at<uchar>( idx ); Element at n-dimensional location indicated by idx[] in array M of unsigned characters  To access a two-dimensional array, you can also extract a C-style pointer to a specific row of the array. This is done with the ptr<>() template member function of cv::Mat. (Recall that the data in the array is contiguous by row, thus accessing a specific column in this way would not make sense; we will see the right way to do that shortly.) As with at<>(), ptr<>() is a template function instantiated with a type name. It takes an integer argument indicating the row you wish a pointer to. The function returns a pointer to the primitive type of which the array is constructed (i.e., if the array type is F32C3, the return value will be of type float*). Thus, given a three-channel

(Recall that the data in the array is contiguous by row, thus accessing a specific column in this way would not make sense; we will see the right way to do that shortly.) As with at<>(), ptr<>() is a template function instantiated with a type name. It takes an integer argument indicating the row you wish a pointer to. The function returns a pointer to the primitive type of which the array is constructed (i.e., if the array type is F32C3, the return value will be of type float*). Thus, given a three-channel matrix mtx of type float, the construction mtx.ptr<Vec3f>(3) would return a pointer to the first (floating-point) channel of the first element in row 3 of mtx. This is generally the fastest way to access elements of an array,18 because once you have the pointer, you are right down there with the data.  There are thus two ways to get a pointer to the data in a matrix mtx. One way is to go about it with the ptr<>() member function. The other is to directly use the member pointer data, and to use the member

matrix mtx of type float, the construction mtx.ptr<Vec3f>(3) would return a pointer to the first (floating-point) channel of the first element in row 3 of mtx. This is generally the fastest way to access elements of an array,18 because once you have the pointer, you are right down there with the data.  There are thus two ways to get a pointer to the data in a matrix mtx. One way is to go about it with the ptr<>() member function. The other is to directly use the member pointer data, and to use the member array step[] to compute addresses. The latter option is similar to what one tended to do in the C interface, but is generally no longer preferred over access methods such as at<>(), ptr<>(), and the iterators. Having said this, direct address computation may still be most efficient, particularly when dealing with arrays of dimension greater than two. There is one last important point to keep in mind about C-style pointer access. If you want to access everything in an array, you will likely iterate one row at

step[] to compute addresses. The latter option is similar to what one tended to do in the C interface, but is generally no longer preferred over access methods such as at<>(), ptr<>(), and the iterators. Having said this, direct address computation may still be most efficient, particularly when dealing with arrays of dimension greater than two. There is one last important point to keep in mind about C-style pointer access. If you want to access everything in an array, you will likely iterate one row at a time, because the rows may or may not be packed continuously in the array. However, there is a member function isContinuous() that will tell you if the members are continuously packed. If they are, you can just grab the pointer to the very first element of the first row and cruise through the entire array as it if were a giant one-dimensional array.  The other form of sequential access is to use the iterator mechanism built into cv::Mat. This mechanism is based on, and works more or less identically to, the

because the rows may or may not be packed continuously in the array. However, there is a member function isContinuous() that will tell you if the members are continuously packed. If they are, you can just grab the pointer to the very first element of the first row and cruise through the entire array as it if were a giant one-dimensional array.  The other form of sequential access is to use the iterator mechanism built into cv::Mat. This mechanism is based on, and works more or less identically to, the analogous mechanism provided by the STL containers. The basic idea is that OpenCV provides a pair of iterator templates, one for const and one for non-const arrays. These iterators are named cv::MatIterator<> and cv::MatConstIterator<>, respectively. The cv::Mat methods begin() and end() return objects of this type. This method of iteration is convenient because the iterators are smart enough to handle the continuous packing as well as the non-continuous packing cases automatically, as well as handling any

analogous mechanism provided by the STL containers. The basic idea is that OpenCV provides a pair of iterator templates, one for const and one for non-const arrays. These iterators are named cv::MatIterator<> and cv::MatConstIterator<>, respectively. The cv::Mat methods begin() and end() return objects of this type. This method of iteration is convenient because the iterators are smart enough to handle the continuous packing as well as the non-continuous packing cases automatically, as well as handling any number of dimensions in the array.                                                               18 The difference in performance between using at<>() and direct pointer access depends on compiler optimization. Access through at<>() will tend to be comparable to (though slightly slower than) direct pointer access in code with a good optimizer, but may be more than order of magnitude slower if that optimizer is turned off (e.g., when you do a debug build). Access through iterators is almost always slower

number of dimensions in the array.                                                               18 The difference in performance between using at<>() and direct pointer access depends on compiler optimization. Access through at<>() will tend to be comparable to (though slightly slower than) direct pointer access in code with a good optimizer, but may be more than order of magnitude slower if that optimizer is turned off (e.g., when you do a debug build). Access through iterators is almost always slower than either of these. In almost all cases, however, using built-in OpenCV functions will be faster than any loop you write regardless of the direct access methods described here, so avoid that kind of construct wherever possible.

Each iterator must be declared and specified to the type of object from which the array is constructed. Here is a simple example of the iterators being used to compute the “longest” element in a three-dimensional array of three-channel elements (a three-dimensional vector field): int sz[3] = { 4, 4, 4 }; cv::Mat m( 3, sz, cv::F32C3 );  // A three-dimensional array of size 4-by-4-by-4 cv::randu( m, -1.0f, 1.0f );   // fill with random numbers from -1.0 to 1.0    float max = 0.0f;              // minimum possible value of L2 norm cv::MatConstIterator<cv::Vec3f> it = m.begin(); while( it != m.end() ) {   len2 = (*it)[0]*(*it)[0]+(*it)[1]*(*it)[1]+(*it)[2]*(*it)[2];   if( len2 > max ) max = len2;   it++; } Iterator-based access is typically used when doing operations over an entire array, or element-wise across multiple arrays. Consider the case of adding two arrays, or of converting an array from the RGB color space to the HSV color space. In such cases, the same exact operation will be done at every pixel

possible value of L2 norm cv::MatConstIterator<cv::Vec3f> it = m.begin(); while( it != m.end() ) {   len2 = (*it)[0]*(*it)[0]+(*it)[1]*(*it)[1]+(*it)[2]*(*it)[2];   if( len2 > max ) max = len2;   it++; } Iterator-based access is typically used when doing operations over an entire array, or element-wise across multiple arrays. Consider the case of adding two arrays, or of converting an array from the RGB color space to the HSV color space. In such cases, the same exact operation will be done at every pixel location.  The N-ary Array Iterator: NAryMatIterator There is another form of iteration which, though it does not handle discontinuities in the packing of the arrays in the manner of cv::MatIterator<>, allows us to handle iteration over many arrays at once. This iterator is called cv::NAryMatIterator, and requires only that all of the arrays that are being iterated over be of the same geometry (number of dimensions and extent in each dimension). Instead of returning single elements of the arrays being

location.  The N-ary Array Iterator: NAryMatIterator There is another form of iteration which, though it does not handle discontinuities in the packing of the arrays in the manner of cv::MatIterator<>, allows us to handle iteration over many arrays at once. This iterator is called cv::NAryMatIterator, and requires only that all of the arrays that are being iterated over be of the same geometry (number of dimensions and extent in each dimension). Instead of returning single elements of the arrays being iterated over, the  N-ary iterator operates by returning chunks of those arrays, called planes.  A plane is a portion (typically one- or two-dimensional slice) of the input array in which the data is guaranteed to be contiguous in memory.19 This is how discontinuity is handled; you are given the contiguous chunks one by one.  For each such plane, you can either operate on it using array operations, or iterate trivially over it yourself. (In this case, “trivially” means to iterate over it in a way which does not

over, the  N-ary iterator operates by returning chunks of those arrays, called planes.  A plane is a portion (typically one- or two-dimensional slice) of the input array in which the data is guaranteed to be contiguous in memory.19 This is how discontinuity is handled; you are given the contiguous chunks one by one.  For each such plane, you can either operate on it using array operations, or iterate trivially over it yourself. (In this case, “trivially” means to iterate over it in a way which does not need to check for discontinuities inside of the chunk.) The concept of the plane is entirely separate from the concept of multiple arrays being iterated over simultaneously. Consider the following code example, in which we will sum just a single multidimensional array plane by plane:   const int n_mat_size = 5;   const int n_mat_sz[] = { n_mat_size, n_mat_size, n_mat_size };   cv::Mat n_mat( 3, n_mat_sz, cv::F32C1 );    cv::RNG rng;   rng.fill( n_mat, cv::RNG::UNIFORM, 0.f, 1.f );      const cv::Mat* arrays[]

to check for discontinuities inside of the chunk.) The concept of the plane is entirely separate from the concept of multiple arrays being iterated over simultaneously. Consider the following code example, in which we will sum just a single multidimensional array plane by plane:   const int n_mat_size = 5;   const int n_mat_sz[] = { n_mat_size, n_mat_size, n_mat_size };   cv::Mat n_mat( 3, n_mat_sz, cv::F32C1 );    cv::RNG rng;   rng.fill( n_mat, cv::RNG::UNIFORM, 0.f, 1.f );      const cv::Mat* arrays[] = { &n_mat, 0 };   cv::Mat my_planes[1];   cv::NAryMatIterator it( arrays, my_planes ); At this point, you have your  N-ary iterator. Continuing our example, we will compute the sum of m0 and m1, and place the result in m2. We will do this plane by plane, however:  // On each iteration, it.planes[i] will be the current plane of the // i-th array from ‘arrays’. //    float s = 0.f;                               // Total sum over all planes   int   n = 0;                                 // Total number of

= { &n_mat, 0 };   cv::Mat my_planes[1];   cv::NAryMatIterator it( arrays, my_planes ); At this point, you have your  N-ary iterator. Continuing our example, we will compute the sum of m0 and m1, and place the result in m2. We will do this plane by plane, however:  // On each iteration, it.planes[i] will be the current plane of the // i-th array from ‘arrays’. //    float s = 0.f;                               // Total sum over all planes   int   n = 0;                                 // Total number of planes                                                              19 In fact the dimensionality of the “plane” is not limited to 2; it can be larger. What is always the case is that the planes will be contiguous in memory.

for (int p = 0; p < it.nplanes; p++, ++it) {     s += cv::sum(it.planes[0])[0];                n++;                                        }    In this example, we first create the three-dimensional array n_mat and fill it with 125 random floating point numbers between 0.0 and 1.0.  To initialize the cv::NAryMatIterator object, we need to have two things. First, we need a C-style array containing pointers to all of the cv::Mat’s we wish to iterate over (in this example, there is just one). This array must always be terminated with a 0 or NULL. Next, we need another C-style array of cv::Mat’s that can be used to refer to the individual planes as we iterate over them (in this case, there is also just one). Once we have created the N-ary iterator, we can iterate over it.  Recall that this iteration is over the planes that make up the arrays we gave to the iterator. The number of planes (the same for each array, because they have the same geometry) will always be given by it.nplanes.  The N-ary iterator

must always be terminated with a 0 or NULL. Next, we need another C-style array of cv::Mat’s that can be used to refer to the individual planes as we iterate over them (in this case, there is also just one). Once we have created the N-ary iterator, we can iterate over it.  Recall that this iteration is over the planes that make up the arrays we gave to the iterator. The number of planes (the same for each array, because they have the same geometry) will always be given by it.nplanes.  The N-ary iterator contains a C-style array called planes that holds headers for the current plane in each input array.  In our example, there is only one array being iterated over, so we need only refer to it.planes[0] (the current plane in the one and only array).  In this example, we then call cv::sum() on each plane and accumulate the final result. To see the real utility of the  N-ary iterator, consider a slightly expanded version of this example in which there are two arrays we would like to sum over:   const int

contains a C-style array called planes that holds headers for the current plane in each input array.  In our example, there is only one array being iterated over, so we need only refer to it.planes[0] (the current plane in the one and only array).  In this example, we then call cv::sum() on each plane and accumulate the final result. To see the real utility of the  N-ary iterator, consider a slightly expanded version of this example in which there are two arrays we would like to sum over:   const int n_mat_size = 5;   const int n_mat_sz[] = { n_mat_size, n_mat_size, n_mat_size };   cv::Mat n_mat0( 3, n_mat_sz, cv::F32C1 );   cv::Mat n_mat1( 3, n_mat_sz, cv::F32C1 );    cv::RNG rng;    rng.fill( n_mat0, cv::RNG::UNIFORM, 0.f, 1.f );   rng.fill( n_mat1, cv::RNG::UNIFORM, 0.f, 1.f );      const cv::Mat* arrays[] = { &n_mat0, &n_mat1, 0 };   cv::Mat my_planes[2];   cv::NAryMatIterator it( arrays, my_planes );    float s = 0.f;                    // Total sum over all planes in both arrays   int   n = 0;

= 5;   const int n_mat_sz[] = { n_mat_size, n_mat_size, n_mat_size };   cv::Mat n_mat0( 3, n_mat_sz, cv::F32C1 );   cv::Mat n_mat1( 3, n_mat_sz, cv::F32C1 );    cv::RNG rng;    rng.fill( n_mat0, cv::RNG::UNIFORM, 0.f, 1.f );   rng.fill( n_mat1, cv::RNG::UNIFORM, 0.f, 1.f );      const cv::Mat* arrays[] = { &n_mat0, &n_mat1, 0 };   cv::Mat my_planes[2];   cv::NAryMatIterator it( arrays, my_planes );    float s = 0.f;                    // Total sum over all planes in both arrays   int   n = 0;                      // Total number of planes   for(int p = 0; p < it.nplanes; p++, ++it) {     s += cv::sum(it.planes[0])[0];                s += cv::sum(it.planes[1])[0];                n++;                                        }    In this second example, you can see that the C-style array called arrays is given pointers to both input arrays, and two matrices are supplied in the my_planes array.  When it is time to iterate over the planes, at each step, planes[0] contains a plane in n_mat0 and planes[1] contains

// Total number of planes   for(int p = 0; p < it.nplanes; p++, ++it) {     s += cv::sum(it.planes[0])[0];                s += cv::sum(it.planes[1])[0];                n++;                                        }    In this second example, you can see that the C-style array called arrays is given pointers to both input arrays, and two matrices are supplied in the my_planes array.  When it is time to iterate over the planes, at each step, planes[0] contains a plane in n_mat0 and planes[1] contains the corresponding plane in n_mat1.  In this simple example, we just sum the two planes and add them to our accumulator.  In an only slightly extended case, we could use element-wise addition to sum these two planes and place the result into the corresponding plane in a third array. Not shown in the example, but also important, is the member it.size, which indicates the size of each plane. The size reported is the number of elements in the plane, so it does not include a factor for the number of channels.

the corresponding plane in n_mat1.  In this simple example, we just sum the two planes and add them to our accumulator.  In an only slightly extended case, we could use element-wise addition to sum these two planes and place the result into the corresponding plane in a third array. Not shown in the example, but also important, is the member it.size, which indicates the size of each plane. The size reported is the number of elements in the plane, so it does not include a factor for the number of channels. In our previous example, if it.nplanes was four, then it.size would have been sixteen: /////////// compute dst[*] = pow(src1[*], src2[*]) ////////////// const Mat* arrays[] = { src1, src2, dst, 0 }; float* ptrs[3];  NAryMatIterator it(arrays, (uchar**)ptrs); for( size_t i = 0; i < it.nplanes; i++, ++it )

{       for( size_t j = 0; j < it.size; j++ )       {              ptrs[2][j] = std::pow(ptrs[0][j], ptrs[1][j]);       } } Accessing Array Elements by Block  In the previous section, we saw ways to access individual elements of an array, either singularly or by iterating sequentially through them all. Another common situation that arises is when you need to access a subset of an array as another array. This might be to select out a row or a column, or any sub-region of the original array. There are many methods that do this for us in one way or another; all of them are member functions of the cv::Mat class and return a subsection of the array on which they are called. The simplest of these methods are row() and col(), which take a single integer and return the indicated row or column of the array whose member we are calling. Clearly these make sense only for a two-dimensional array; we will get to the more complicated case momentarily. When you use m.row() or m.col() (for some array m), or any of the other

that do this for us in one way or another; all of them are member functions of the cv::Mat class and return a subsection of the array on which they are called. The simplest of these methods are row() and col(), which take a single integer and return the indicated row or column of the array whose member we are calling. Clearly these make sense only for a two-dimensional array; we will get to the more complicated case momentarily. When you use m.row() or m.col() (for some array m), or any of the other functions we are about to discuss, it is important to understand that the data in m is not copied to the new arrays. Consider an expression like m2 = m.row(3). This expression means to create a new array header m2, and to arrange its data pointer, step array, and so on, such that it will access the data in row 3 in m. If you modify the data in m2, you will be modifying the data in m. Later, we will visit the copyTo() method, which actually will copy data. The main advantage of the way this is handled in OpenCV is

we are about to discuss, it is important to understand that the data in m is not copied to the new arrays. Consider an expression like m2 = m.row(3). This expression means to create a new array header m2, and to arrange its data pointer, step array, and so on, such that it will access the data in row 3 in m. If you modify the data in m2, you will be modifying the data in m. Later, we will visit the copyTo() method, which actually will copy data. The main advantage of the way this is handled in OpenCV is that the amount of time required to create a new array that accesses part of an existing array is not only very small, but also independent of the size of either the old or the new array. Closely related to row() and col() are rowRange() and colRange(). These functions do essentially the same thing as their simpler cousins, except that they will extract an array with multiple contiguous rows (or columns). Both functions can be called in one of two ways, either by specifying an integer start and end row (or

that the amount of time required to create a new array that accesses part of an existing array is not only very small, but also independent of the size of either the old or the new array. Closely related to row() and col() are rowRange() and colRange(). These functions do essentially the same thing as their simpler cousins, except that they will extract an array with multiple contiguous rows (or columns). Both functions can be called in one of two ways, either by specifying an integer start and end row (or column), or by passing a cv::Range object that indicates the desired rows (or columns). In the case of the two-integer method, the range is inclusive of the start index but exclusive of the end index (you may recall that cv::Range uses a similar convention).  The member function diag() works the same as row() or col(), except that the array returned from m.diag() references the diagonal elements of a matrix. m.diag() expects an integer argument that indicates which diagonal is to be extracted. If that

(or column), or by passing a cv::Range object that indicates the desired rows (or columns). In the case of the two-integer method, the range is inclusive of the start index but exclusive of the end index (you may recall that cv::Range uses a similar convention).  The member function diag() works the same as row() or col(), except that the array returned from m.diag() references the diagonal elements of a matrix. m.diag() expects an integer argument that indicates which diagonal is to be extracted. If that argument is zero, then it will be the main diagonal. If it is positive, it will be offset from the main diagonal by that distance in the upper half of the array. If it is negative, then it will be from the lower half of the array. The last way to extract a sub-matrix is with operator(). Using this operator, you can pass either a pair of ranges (a cv::Range for rows and a cv::Range for columns) or a cv::Rect to specify the region you want. This is the only method of access that will allow you to extract a

argument is zero, then it will be the main diagonal. If it is positive, it will be offset from the main diagonal by that distance in the upper half of the array. If it is negative, then it will be from the lower half of the array. The last way to extract a sub-matrix is with operator(). Using this operator, you can pass either a pair of ranges (a cv::Range for rows and a cv::Range for columns) or a cv::Rect to specify the region you want. This is the only method of access that will allow you to extract a sub-volume from a higher-dimensional array. In this case, a pointer to a C-style array of ranges is expected and that array must have as many elements as the number of dimensions of the array. Table 3-16: Block access methods of cv::Mat Example Description m.row( i ); Array corresponding to row i of m m.col( j ); Array corresponding to column j of m m.rowRange( i0, i1 ); Array corresponding to rows i0 through i1-1 of matrix m

m.rowRange( cv::Range( i0, i1 ) ); Array corresponding to rows i0 through i1-1 of matrix m m.colRange( j0, j1 ); Array corresponding to columns j0 through j1-1 of matrix m m.colRange( cv::Range( j0, j1 ) ); Array corresponding to columns j0 through j1-1 of matrix m m.diag( d ); Array corresponding to the d-offset diagonal of matrix m m( cv::Range(i0,i1), cv::Range(j0,j1) ); Array corresponding to the subrectangle of matrix m with one corner at i0, j0 and the opposite corner at (i1-1, j1-1) m( cv::Rect(i0,i1,w,h) ); Array corresponding to the subrectangle of matrix m with one corner at i0, j0 and the opposite corner at (i0+w-1, j0+h-1) m( ranges ); Array extracted from m corresponding to the subvolume that is the intersection of the ranges given by ranges[0]-ranges[ndim-1]  Matrix Expressions: Algebra and cv::Mat One of the things enabled by the move to C++ in version 2.1 is the overloading of operators and the ability to create algebraic expressions consisting of matrix arrays20 and singletons. The primary

); Array corresponding to the subrectangle of matrix m with one corner at i0, j0 and the opposite corner at (i0+w-1, j0+h-1) m( ranges ); Array extracted from m corresponding to the subvolume that is the intersection of the ranges given by ranges[0]-ranges[ndim-1]  Matrix Expressions: Algebra and cv::Mat One of the things enabled by the move to C++ in version 2.1 is the overloading of operators and the ability to create algebraic expressions consisting of matrix arrays20 and singletons. The primary advantage of this is code clarity, as many operations can be combined into one expression that is both more compact and often more meaningful. In the background, many important features of OpenCV’s array class are being used to make these operations work. For example, matrix headers are created automatically as needed and workspace data areas are allocated (only) as required. When no longer needed, data areas are deallocated invisibly and automatically. The result of the computation is finally placed in the

of this is code clarity, as many operations can be combined into one expression that is both more compact and often more meaningful. In the background, many important features of OpenCV’s array class are being used to make these operations work. For example, matrix headers are created automatically as needed and workspace data areas are allocated (only) as required. When no longer needed, data areas are deallocated invisibly and automatically. The result of the computation is finally placed in the destination array by operator=(). However, one important distinction, is that this form of operator=() is not assigning a cv::Mat or a cv::Mat (as it might appear), but rather a cv::MatExpr (the expression itself21) to a cv::Mat. This distinction is important because data is always copied into the result (left-hand side) array. Recall that though m2=m1 is legal, it means something slightly different. In this first case, m2 would be another reference to the data in m1. By contrast, m2=m1+m0 means something different

array by operator=(). However, one important distinction, is that this form of operator=() is not assigning a cv::Mat or a cv::Mat (as it might appear), but rather a cv::MatExpr (the expression itself21) to a cv::Mat. This distinction is important because data is always copied into the result (left-hand side) array. Recall that though m2=m1 is legal, it means something slightly different. In this first case, m2 would be another reference to the data in m1. By contrast, m2=m1+m0 means something different again. Because m1+m0 is a                                                              20 For clarity, we use the word “array” when referring to a general object of type cv::Mat, and use the word “matrix” for those situations in which the manner in which the array is being used indicates that it is representing a mathematical object that would be called a matrix. The distinction is a purely semantic one, and not manifest in the actual design of OpenCV. 21 The underlying machinery of cv::MatExpr is more detail

Because m1+m0 is a                                                              20 For clarity, we use the word “array” when referring to a general object of type cv::Mat, and use the word “matrix” for those situations in which the manner in which the array is being used indicates that it is representing a mathematical object that would be called a matrix. The distinction is a purely semantic one, and not manifest in the actual design of OpenCV. 21 The underlying machinery of cv::MatExpr is more detail than we need, but you can think of cv::MatExpr as being a symbolic representation of the algebraic form of the right-hand side. The great advantage of cv::MatExpr is that when it is time to evaluate an expression, it is often clear that some operations can be removed or simplified without evaluation (such as computing the transpose of the transpose of a matrix, adding zero, or multiplying a matrix by its own inverse).

matrix expression, it will be evaluated and a pointer to the results will be assigned in m2. The results will reside in a newly allocated data area.22 Table 3-17 lists examples of the algebraic operations available. Note that in addition to simple algebra, there are comparison operators, operators for constructing matrices (such as cv::Mat::eye(), which we encountered earlier), and higher level operations for computing transposes and inversions. The key idea here is that you should be able to take the sorts of relatively nontrivial matrix expressions that occur when doing computer vision and express them on a single line in a clear and concise way. Table 3-17: Operations available for matrix expressions Example Description m0 + m1, m0 – m1; Addition or subtraction of matrices m0 + s; m0 – s; s + m0, s – m1; Addition or subtraction between a matrix and a singleton -m0; Negation of a matrix s * m0; m0 * s; Scaling of a matrix by a singleton m0.mul( m1 ); m0/m1; Per element multiplication of m0 and m1,

the sorts of relatively nontrivial matrix expressions that occur when doing computer vision and express them on a single line in a clear and concise way. Table 3-17: Operations available for matrix expressions Example Description m0 + m1, m0 – m1; Addition or subtraction of matrices m0 + s; m0 – s; s + m0, s – m1; Addition or subtraction between a matrix and a singleton -m0; Negation of a matrix s * m0; m0 * s; Scaling of a matrix by a singleton m0.mul( m1 ); m0/m1; Per element multiplication of m0 and m1, per-element division of m0 by m1 m0 * m1; Matrix multiplication of m0 and m1 m0.inv( method ); Matrix inversion of m0 (default value of method is DECOMP_LU) m0.t(); Matrix transpose of m0 (no copy is done) m0>m1; m0>=m1; m0==m1; m0<=m1; m0<m1; Per element comparison, returns uchar matrix with elements 0 or 255 m0&m1; m0|m1; m0^m1; ~m0;  m0&s; s&m0; m0|s; s|m0; m0^s; s^m0; Bitwise logical operators between matrices or matrix and a singleton min(m0,m1); max(m0,m1); min(m0,s);  min(s,m0); max(m0,s);

and m1, per-element division of m0 by m1 m0 * m1; Matrix multiplication of m0 and m1 m0.inv( method ); Matrix inversion of m0 (default value of method is DECOMP_LU) m0.t(); Matrix transpose of m0 (no copy is done) m0>m1; m0>=m1; m0==m1; m0<=m1; m0<m1; Per element comparison, returns uchar matrix with elements 0 or 255 m0&m1; m0|m1; m0^m1; ~m0;  m0&s; s&m0; m0|s; s|m0; m0^s; s^m0; Bitwise logical operators between matrices or matrix and a singleton min(m0,m1); max(m0,m1); min(m0,s);  min(s,m0); max(m0,s); max(s,m0); Per element minimum and maximum between two matrices or a matrix and a singleton cv::abs( m0 ); Per element absolute value of m0 m0.cross( m1 ); m0.dot( m1 ); Vector cross and dot product (vector cross product is only defined for 3-by-1 matrices) cv::Mat::eye( Nr, Nc, type );  Class static matrix initializers that return                                                              22 If you are a real expert, this will not surprise you. Clearly a temporary array must be created to store the result

Per element minimum and maximum between two matrices or a matrix and a singleton cv::abs( m0 ); Per element absolute value of m0 m0.cross( m1 ); m0.dot( m1 ); Vector cross and dot product (vector cross product is only defined for 3-by-1 matrices) cv::Mat::eye( Nr, Nc, type );  Class static matrix initializers that return                                                              22 If you are a real expert, this will not surprise you. Clearly a temporary array must be created to store the result of m1+m0. Then m2 really is just another reference, but it is another reference to that temporary array. When operator+() exits, its reference to the temporary array is discarded, but the reference count is not zero. m2 is left holding the one and only reference to that array.

cv::Mat::zeros( Nr, Nc, type );   cv::Mat::ones( Nr, Nc, type ); fixed Nr-by-Nc matrices of type type The matrix inversion operator inv() is actually a frontend to a variety of algorithms for matrix inversion. There are currently three options. The first option is cv::DECOMP_LU, which means LU decomposition, and which works for any nonsingular matrix. The second is cv::DECOMP_CHOLESKY, which solves the inversion by Cholesky decomposition. Cholesky decomposition only works for symmetric, positive definite matrices, but is much faster than LU decomposition for large matrices. The last option is cv::DECOMP_SVD, which solves the inversion by singular value decomposition. SVD is the only workable option for matrices that are singular or not even squares (the pseudo-inverse is then computed). Not included in Table 3-17 are all of the functions like cv::norm(), cv::mean(), cv::sum(), and so on (some of which we have not gotten to yet, but you can probably guess what they do) that convert matrices to other matrices

but is much faster than LU decomposition for large matrices. The last option is cv::DECOMP_SVD, which solves the inversion by singular value decomposition. SVD is the only workable option for matrices that are singular or not even squares (the pseudo-inverse is then computed). Not included in Table 3-17 are all of the functions like cv::norm(), cv::mean(), cv::sum(), and so on (some of which we have not gotten to yet, but you can probably guess what they do) that convert matrices to other matrices or to scalars. Any such object can still be used in a matrix expression. Saturation Casting  In OpenCV, you will often do operations that risk overflowing or underflowing the available values in the destination of some computation. This is particularly common when one is doing operations on unsigned types that involve subtraction, but it can happen anywhere. To deal with this problem, OpenCV relies on a construct called saturation casting. What this means is that OpenCV arithmetic and other operations that act on

Any such object can still be used in a matrix expression. Saturation Casting  In OpenCV, you will often do operations that risk overflowing or underflowing the available values in the destination of some computation. This is particularly common when one is doing operations on unsigned types that involve subtraction, but it can happen anywhere. To deal with this problem, OpenCV relies on a construct called saturation casting. What this means is that OpenCV arithmetic and other operations that act on arrays will check for underflows and overflows automatically; in these cases, the library functions will replace the resulting value of an operation with the lowest or highest available value, respectively. Note that this is not what C language operations normally and natively do. You may want to implement this particular behavior in your own functions as well. OpenCV provides some handy templated casting operators to make this easy for you. These are implemented as a template function called

that act on arrays will check for underflows and overflows automatically; in these cases, the library functions will replace the resulting value of an operation with the lowest or highest available value, respectively. Note that this is not what C language operations normally and natively do. You may want to implement this particular behavior in your own functions as well. OpenCV provides some handy templated casting operators to make this easy for you. These are implemented as a template function called cv::saturate_cast<>(), which allows you to specify the type to which you would like to cast the argument. Here is an example: uchar& Vxy = m0.at<uchar>( y, x ); Vxy = cv::saturate_cast<uchar>((Vxy-128)*2 + 128);} In this example code, the variable Vxy is first assigned to be a reference to an element of an 8-bit array m0. This array then has 128 subtracted from it, multiply that by two (scale that up), and add 128 (so the result is twice as far from 128 as the original). The usual C arithmetic rules would

cv::saturate_cast<>(), which allows you to specify the type to which you would like to cast the argument. Here is an example: uchar& Vxy = m0.at<uchar>( y, x ); Vxy = cv::saturate_cast<uchar>((Vxy-128)*2 + 128);} In this example code, the variable Vxy is first assigned to be a reference to an element of an 8-bit array m0. This array then has 128 subtracted from it, multiply that by two (scale that up), and add 128 (so the result is twice as far from 128 as the original). The usual C arithmetic rules would assign Vxy-128 to a (32-bit) signed integer; followed by integer multiplication by 2 and integer addition of 128. Notice, however, that if the original value of Vxy were (for example) 10, then Vxy-128 would be -118. The value of the expression would then be -108. This number will not fit into the 8-bit unsigned variable Vxy. This is where cv::saturation_cast<uchar>() comes to the rescue. It takes the value of -108 and, recognizing that it is too low for an unsigned char, converts it to 0. More Things an

assign Vxy-128 to a (32-bit) signed integer; followed by integer multiplication by 2 and integer addition of 128. Notice, however, that if the original value of Vxy were (for example) 10, then Vxy-128 would be -118. The value of the expression would then be -108. This number will not fit into the 8-bit unsigned variable Vxy. This is where cv::saturation_cast<uchar>() comes to the rescue. It takes the value of -108 and, recognizing that it is too low for an unsigned char, converts it to 0. More Things an Array Can Do  At this point, we have touched on most of the members of the cv::Mat class. Of course, there are a few things that were missed, as they did not fall into any specific category that was discussed so far. In this section we will review the leftovers that you will need in your daily life of OpenCV programming. Table 3-18: More class member functions of cv::Mat Example Description m1 = m0.clone(); Make a complete copy of m0, copying all data elements as well; cloned array will be continuous

an Array Can Do  At this point, we have touched on most of the members of the cv::Mat class. Of course, there are a few things that were missed, as they did not fall into any specific category that was discussed so far. In this section we will review the leftovers that you will need in your daily life of OpenCV programming. Table 3-18: More class member functions of cv::Mat Example Description m1 = m0.clone(); Make a complete copy of m0, copying all data elements as well; cloned array will be continuous m0.copyTo( m1 ); Copy contents of m0 onto m1, reallocating m1 if necessary (equivalent to m1=m0.clone())

m0.copyTo( m1, mask ); As m0.copyTo(m1) except only entries indicated in the array mask are copied m0.convertTo(     m1, type, scale, offset  ); Convert elements of m0 to type (i.eg., cv::F32) and write to m1 after scaling by scale (default 1.0) and adding offset (default 0.0) m0.assignTo( m1, type ); internal use only (resembles convertTo) m0.setTo( s, mask ); Set all entries in m0 to singleton value s; if mask is present, only set those value corresponding to nonzero elements in mask m0.reshape( chan, rows ); Changes effective shape of a two-dimensional matrix; chan or rows may be zero, which implies “no change”; data is not copied m0.push_back( s ); Extend a M-by-1 matrix and insert the singleton s at the end m0.push_back( m1 ); Extend an M-by-N by K rows and copy m1 into that rows; m1 must be K-by-N m0.pop_back( n ); Remove n rows from the end of an M-by-N (default value of n is one)23 m0.locateROI( size, offset ); Write whole size of m0 to cv::Size size; if m0 is a “view” of a larger matrix, write

rows ); Changes effective shape of a two-dimensional matrix; chan or rows may be zero, which implies “no change”; data is not copied m0.push_back( s ); Extend a M-by-1 matrix and insert the singleton s at the end m0.push_back( m1 ); Extend an M-by-N by K rows and copy m1 into that rows; m1 must be K-by-N m0.pop_back( n ); Remove n rows from the end of an M-by-N (default value of n is one)23 m0.locateROI( size, offset ); Write whole size of m0 to cv::Size size; if m0 is a “view” of a larger matrix, write location of starting corner to Point& offset m0.adjustROI( t, b, l, r ); Increase the size of a view by t pixels above, b pixels below, l pixels to the left, and r pixels to the right m0.total(); Compute the total number of array elements (does not include channels) m0.isContinuous(); Return true only if the rows in m0 are packed without space between them in memory m0.elemSize(); Return the size of the elements of m0 in bytes (eE.g., a three-channel float matrix would return 12 bytes)  m0.elemSize1(); Return

of starting corner to Point& offset m0.adjustROI( t, b, l, r ); Increase the size of a view by t pixels above, b pixels below, l pixels to the left, and r pixels to the right m0.total(); Compute the total number of array elements (does not include channels) m0.isContinuous(); Return true only if the rows in m0 are packed without space between them in memory m0.elemSize(); Return the size of the elements of m0 in bytes (eE.g., a three-channel float matrix would return 12 bytes)  m0.elemSize1(); Return the size of the subelements of m0 in bytes (eE.g., a three-channel float matrix would return 4 bytes) m0.type(); Return a valid type identifier for the elements of m0 (e.g., cv::F32C3) m0.depth(); Return a valid type identifier for the individial channels of                                                              23 Many implementations of “pop” functionality return the popped element.  This one does not; its return type is void.

m0 (e.g., cv::F32) m0.channels(); Return the number of channels in the elements of m0. m0.size(); Return the size of the m0 as a cv::Size object. m0.empty(); Return true only if the array has no elements (i.e., m0.total==0 or m0.data==NULL) class cv::SparseMat: Sparse Arrays  The class cv::SparseMat is used when an array is likely to be very large compared to the number of nonzero entries. This situation often arises in linear algebra with sparse matrices, but it also comes up when one wishes to represent data, particularly histograms, in higher-dimensional arrays, since most of space will be empty. A sparse representation only stores data that is actually present and so can save a great deal of memory. In practice, many sparse objects would be too huge to represent at all in a dense format. The disadvantage of sparse representations is that computation with them is slower (on a per-element basis). This last point is important, in that computation with sparse matrices is not categorically slower, as there can

particularly histograms, in higher-dimensional arrays, since most of space will be empty. A sparse representation only stores data that is actually present and so can save a great deal of memory. In practice, many sparse objects would be too huge to represent at all in a dense format. The disadvantage of sparse representations is that computation with them is slower (on a per-element basis). This last point is important, in that computation with sparse matrices is not categorically slower, as there can be a great economy in knowing in advance that many operations need not be done at all.  The OpenCV sparse matrix class cv::SparseMat functions analogously to the dense matrix class cv::Mat in most ways. It is defined similarly, supports most of the same operations, and can contain the same data types. Internally, the way data is organized is quite different. While cv::Mat uses a data array closely related to a C data array (one in which the data is sequentially packed and addresses are directly computable from

great economy in knowing in advance that many operations need not be done at all.  The OpenCV sparse matrix class cv::SparseMat functions analogously to the dense matrix class cv::Mat in most ways. It is defined similarly, supports most of the same operations, and can contain the same data types. Internally, the way data is organized is quite different. While cv::Mat uses a data array closely related to a C data array (one in which the data is sequentially packed and addresses are directly computable from the indices of the element), cv::SparseMat uses a hash table to store just the nonzero elements.24 That hash table is maintained automatically, so when the number of (nonzero) elements in the array becomes too large for efficient look-up, the table grows automatically. Accessing Sparse Array Elements The most important difference between sparse and dense arrays is how elements are accessed. Sparse arrays provide four different access mechanisms: cv::SparseMat::ptr(), cv::SparseMat::ref(),

computable from the indices of the element), cv::SparseMat uses a hash table to store just the nonzero elements.24 That hash table is maintained automatically, so when the number of (nonzero) elements in the array becomes too large for efficient look-up, the table grows automatically. Accessing Sparse Array Elements The most important difference between sparse and dense arrays is how elements are accessed. Sparse arrays provide four different access mechanisms: cv::SparseMat::ptr(), cv::SparseMat::ref(), cv::SparseMat::value(), and cv::SparseMat::find(). The cv::SparseMat::ptr() method has several variations, the simplest of which has the template:  uchar* cv::SparseMat::ptr( int i0, bool createMissing, size_t* hashval=0 ) This particular version is for accessing a one-dimensional array. The first argument i0 is the index of the requested element. The next argument createMissing indicates whether the element should be created if it is not already present in the array. When cv::SparseMat::ptr() is called, it

and cv::SparseMat::find(). The cv::SparseMat::ptr() method has several variations, the simplest of which has the template:  uchar* cv::SparseMat::ptr( int i0, bool createMissing, size_t* hashval=0 ) This particular version is for accessing a one-dimensional array. The first argument i0 is the index of the requested element. The next argument createMissing indicates whether the element should be created if it is not already present in the array. When cv::SparseMat::ptr() is called, it will return a pointer to the element if that element is already defined in the array, but NULL if that element is not defined. However, if the createMissing argument is true, however, that element will be created and a valid non-NULL pointer will be returned to that new element. To understand the final argument hashval, it is necessary to recall that the underlying data representation of a cv::SparseMat is as a hash table. Looking up objects in a hash table requires two steps: the first being the computation of the hash key (in

to the element if that element is already defined in the array, but NULL if that element is not defined. However, if the createMissing argument is true, however, that element will be created and a valid non-NULL pointer will be returned to that new element. To understand the final argument hashval, it is necessary to recall that the underlying data representation of a cv::SparseMat is as a hash table. Looking up objects in a hash table requires two steps: the first being the computation of the hash key (in this case, from the indices), and the second being the searching of a list associated with that key. Normally, that list will be short (ideally only one element), so the primary computational cost in a lookup is the computation of the hash key. If this key has already been computed (as with cv::SparseMat::hash(), which will be covered in the next section), then time can be saved by not recomputing it. In the case of cv::SparseMat::ptr(), if the argument hashval is left with its default argument of NULL,

this case, from the indices), and the second being the searching of a list associated with that key. Normally, that list will be short (ideally only one element), so the primary computational cost in a lookup is the computation of the hash key. If this key has already been computed (as with cv::SparseMat::hash(), which will be covered in the next section), then time can be saved by not recomputing it. In the case of cv::SparseMat::ptr(), if the argument hashval is left with its default argument of NULL, the hash key will be computed. If, however, a key is provided, it will be used.                                                              24 Actually, zero elements may be stored, if those elements have become zero as a result of computation on the array. If you want to clean up such elements, you must do so yourself. This is the function of the method SparseMat::erase(), which we will visit shortly.

There are also variations of cv::SparseMat::ptr() that allow for two or three indices, as well as a version whose first argument is a pointer to an array of integers (i.e., const int* idx), which is required to have the same number of entries as the dimension of the array being accessed. In all cases, the function cv::SparseMat::ptr() returns a pointer to an unsigned character (i.e., uchar*), which will typically need to be recast to the correct type for the array. The accessor template function SparseMat::ref<>() is used to return a reference to a particular element of the array. This function, like SparseMat::ptr(), can take one, two, or three indices, or a pointer to an array of indices, and also supports an optional pointer to the hash value to use in the lookup. Because it is a template function, you must specify the type of object being referenced. So, for example, if your array were of type cv::F32, then you might call SparseMat::ref<>() like this: a_sparse_mat.ref<float>( i0, i1 ) += 1.0f; The

is used to return a reference to a particular element of the array. This function, like SparseMat::ptr(), can take one, two, or three indices, or a pointer to an array of indices, and also supports an optional pointer to the hash value to use in the lookup. Because it is a template function, you must specify the type of object being referenced. So, for example, if your array were of type cv::F32, then you might call SparseMat::ref<>() like this: a_sparse_mat.ref<float>( i0, i1 ) += 1.0f; The template method cv::SparseMat::value<>() is identical to SparseMat::ref<>(), except that it returns the value and not a reference to the value. As such, this method is itself a “const method.”25 The final accessor function is cv::SparseMat::find<>(), which works similarly to cv::SparseMat::ref<>() and cv::SparseMat::value<>(), but returns a pointer to the requested object. Unlike cv::SparseMat::ptr(), however, this pointer is of the type specified by the template instantiation of cv::SparseMat::find<>(), and so does not

cv::SparseMat::value<>() is identical to SparseMat::ref<>(), except that it returns the value and not a reference to the value. As such, this method is itself a “const method.”25 The final accessor function is cv::SparseMat::find<>(), which works similarly to cv::SparseMat::ref<>() and cv::SparseMat::value<>(), but returns a pointer to the requested object. Unlike cv::SparseMat::ptr(), however, this pointer is of the type specified by the template instantiation of cv::SparseMat::find<>(), and so does not need to be recast. For purposes of code clarity, cv::SparseMat::find<>() is preferred over cv::SparseMat::ptr() wherever possible. cv::SparseMat::find<>(), however, is a const method, and returns a const pointer, so the two are not always interchangeable.  In addition to direct access through the four functions just outlined, it is also possible to access the elements of a sparse matrix through iterators. As with the dense array types, the iterators are normally templated. The templated iterators are

not need to be recast. For purposes of code clarity, cv::SparseMat::find<>() is preferred over cv::SparseMat::ptr() wherever possible. cv::SparseMat::find<>(), however, is a const method, and returns a const pointer, so the two are not always interchangeable.  In addition to direct access through the four functions just outlined, it is also possible to access the elements of a sparse matrix through iterators. As with the dense array types, the iterators are normally templated. The templated iterators are cv::SparseMatIterator_<> and cv::SparseMatConstIterator_<>, together with their corresponding cv::SparseMat::begin<>() and cv::SparseMat::end<>() routines. (The const forms of the begin() and end() routines return the const iterators.) There are also non-template iterators cv::SparseMatIterator and cv::SparseMatConstIterator, which are returned by the non-template SparseMat::begin() and SparseMat::end() routines. Here is an example in which we print out all of the nonzero elements of a sparse array: //

are cv::SparseMatIterator_<> and cv::SparseMatConstIterator_<>, together with their corresponding cv::SparseMat::begin<>() and cv::SparseMat::end<>() routines. (The const forms of the begin() and end() routines return the const iterators.) There are also non-template iterators cv::SparseMatIterator and cv::SparseMatConstIterator, which are returned by the non-template SparseMat::begin() and SparseMat::end() routines. Here is an example in which we print out all of the nonzero elements of a sparse array: // Create a 10x10 sparse matrix with a few nonzero elements //  int size[] = {10,10}; cv::SparseMat sm( 2, size, cv::F32 ); for( int i=0; i<10; i++ ) { // Fill the array   int idx[2];   idx[0] = size[0] * rand();   idx[1] = size[1] * rand();   sm.ref<float>( idx ) += 1.0f; }  // Print out the nonzero elements // cv::SparseMatConstIterator_<float> it     = sm.begin<float>(); cv::SparseMatConstIterator_<float> it_end = sm.end<float>();  for(; it != it_end; ++it) {   const cv::SparseMat::Node* node = it.node();

Create a 10x10 sparse matrix with a few nonzero elements //  int size[] = {10,10}; cv::SparseMat sm( 2, size, cv::F32 ); for( int i=0; i<10; i++ ) { // Fill the array   int idx[2];   idx[0] = size[0] * rand();   idx[1] = size[1] * rand();   sm.ref<float>( idx ) += 1.0f; }  // Print out the nonzero elements // cv::SparseMatConstIterator_<float> it     = sm.begin<float>(); cv::SparseMatConstIterator_<float> it_end = sm.end<float>();  for(; it != it_end; ++it) {   const cv::SparseMat::Node* node = it.node();   printf(" (%3d,%3d) %f\n", node->idx[0], node->idx[1], *it );                                                              25 For those of you not familiar with “const correctness,””, this means that the method is declared in its prototype such that the this pointer passed to SparseMat::value<>() is guaranteed to be a constant pointer, and thus SparseMat::value<>() can be called on const objects, while functions like SparseMat::ref<>() cannot. The next function SparseMat::find<>() is also a const function.

printf(" (%3d,%3d) %f\n", node->idx[0], node->idx[1], *it );                                                              25 For those of you not familiar with “const correctness,””, this means that the method is declared in its prototype such that the this pointer passed to SparseMat::value<>() is guaranteed to be a constant pointer, and thus SparseMat::value<>() can be called on const objects, while functions like SparseMat::ref<>() cannot. The next function SparseMat::find<>() is also a const function.

} In this example, we also slipped in the method node(), which is defined for the iterators. node() returns a pointer to the internal data node in the sparse matrix that is indicated by the iterator. The returned object of type cv::SparseMat::Node has the following definition: struct Node {   size_t hashval;   size_t next;   int idx[cv::MAX_DIM]; }; This structure contains both the index of the associated element (note that element idx is of type int[]), as well as the hash value associated with that node (element hashval is the same hash value as can be used with SparseMat::ptr(), SparseMat::ref(), SparseMat::value(), and SparseMat::find().) Functions Unique to Sparse Arrays As stated earlier, sparse matrices support many of the same operations as dense matrices. In addition, there are several methods that are unique to sparse matrices. These are listed in Table 3-19, and include the functions mentioned in the previous sections. Table 3-19: Additional class member functions of cv::SparseMat Example

node (element hashval is the same hash value as can be used with SparseMat::ptr(), SparseMat::ref(), SparseMat::value(), and SparseMat::find().) Functions Unique to Sparse Arrays As stated earlier, sparse matrices support many of the same operations as dense matrices. In addition, there are several methods that are unique to sparse matrices. These are listed in Table 3-19, and include the functions mentioned in the previous sections. Table 3-19: Additional class member functions of cv::SparseMat Example Description cv::SparseMat sm(); Create a sparse matrix without initialization cv::SparseMat sm( 3, sz, cv::F32 ); Create a three-dimensional sparse matrix with dimensions given by the array sz of type float cv::SparseMat sm( sm0 ); Create a new sparse matrix which is a copy of existing sparse matrix sm0 cv::SparseMat( m0, try1d ); Create a sparse matrix from an existing dense matrix m0; if the bool try1d is true, convert m0 to a one-dimensional sparse matrix if the dense matrix was N-by-1 or 1-by-N

Example Description cv::SparseMat sm(); Create a sparse matrix without initialization cv::SparseMat sm( 3, sz, cv::F32 ); Create a three-dimensional sparse matrix with dimensions given by the array sz of type float cv::SparseMat sm( sm0 ); Create a new sparse matrix which is a copy of existing sparse matrix sm0 cv::SparseMat( m0, try1d ); Create a sparse matrix from an existing dense matrix m0; if the bool try1d is true, convert m0 to a one-dimensional sparse matrix if the dense matrix was N-by-1 or 1-by-N cv::SparseMat( &old_sparse_mat ); Create a new sparse matrix from a pointer to a pre-version 2.1 C-style sparse matrix of type CvSparseMat  CvSparseMat* old_sm =    (cv::SparseMat*) sm; Cast operator creates a pointer to a pre-version 2.1 C-style sparse matrix; that CvSparseMat object is created and all data is copied into it, then its pointer is returned size_t n = sm.nzcount(); Return the number of nonzero elements in sm size_t h = sm.hash( i0 ); size_t h = sm.hash( i0, i1 ); size_t h = sm.hash( i0, i1,

cv::SparseMat( &old_sparse_mat ); Create a new sparse matrix from a pointer to a pre-version 2.1 C-style sparse matrix of type CvSparseMat  CvSparseMat* old_sm =    (cv::SparseMat*) sm; Cast operator creates a pointer to a pre-version 2.1 C-style sparse matrix; that CvSparseMat object is created and all data is copied into it, then its pointer is returned size_t n = sm.nzcount(); Return the number of nonzero elements in sm size_t h = sm.hash( i0 ); size_t h = sm.hash( i0, i1 ); size_t h = sm.hash( i0, i1, i2 ); size_t h = sm.hash( idx ); Return the hash value for element i0 in a one-dimensional sparse matrix, i0, i1 in a two-dimensional sparse matrix, i0, i1, i2 in a three-dimensional sparse matrix, or the element indicated by the array of integers idx in an n-dimensional sparse matrix

sm.ref<float>( i0 )         = f0; sm.ref<float>( i0, i1 )     = f0; sm.ref<float>( i0, i1, i2 ) = f0; sm.ref<float>( idx )        = f0; Assign the value f0 to element i0 in a one-dimensional sparse matrix, i0, i1 in a two-dimensional sparse matrix, i0, i1, i2 in a three-dimensional sparse matrix, or the element indicated by the array of integers idx in an n-dimensional sparse matrix f0 = sm.value<float>( i0 ); f0 = sm.value<float>( i0, i1 ); f0 = sm.value<float>( i0, i1, i2 ); f0 = sm.value<float>( idx ); Assign the value to f0 from element i0 in a one-dimensional sparse matrix, i0, i1 in a two-dimensional sparse matrix, i0, i1, i2 in a three-dimensional sparse matrix, or the element indicated by the array of integers idx in an n-dimensional sparse matrix p0 = sm.find<float>( i0 ); p0 = sm.find<float>( i0, i1 ); p0 = sm.find<float>( i0, i1, i2 ); p0 = sm.find<float>( idx ); Assign to p0 the address of element i0 in a one-dimensional sparse matrix, i0, i1 in a two-dimensional sparse matrix, i0, i1, i2 in a

Assign the value to f0 from element i0 in a one-dimensional sparse matrix, i0, i1 in a two-dimensional sparse matrix, i0, i1, i2 in a three-dimensional sparse matrix, or the element indicated by the array of integers idx in an n-dimensional sparse matrix p0 = sm.find<float>( i0 ); p0 = sm.find<float>( i0, i1 ); p0 = sm.find<float>( i0, i1, i2 ); p0 = sm.find<float>( idx ); Assign to p0 the address of element i0 in a one-dimensional sparse matrix, i0, i1 in a two-dimensional sparse matrix, i0, i1, i2 in a three-dimensional sparse matrix, or the element indicated by the array of integers idx in an n-dimensional sparse matrix sm.erase( i0, &hashval ); sm.erase( i0, i1, &hashval ); sm.erase( idx, &hashval );  Remove the element i0, i1 in a two-dimensional sparse matrix, i0, i1, i2 in a three-dimensional sparse matrix, or the element indicated by the array of integers idx in an n-dimensional sparse matrix. If hashval is not NULL, use the provided value instead of computing it cv::SparseMatIterator<float> it    =

sparse matrix, or the element indicated by the array of integers idx in an n-dimensional sparse matrix sm.erase( i0, &hashval ); sm.erase( i0, i1, &hashval ); sm.erase( idx, &hashval );  Remove the element i0, i1 in a two-dimensional sparse matrix, i0, i1, i2 in a three-dimensional sparse matrix, or the element indicated by the array of integers idx in an n-dimensional sparse matrix. If hashval is not NULL, use the provided value instead of computing it cv::SparseMatIterator<float> it    = sm.begin<float>(); Create a sparse matrix iterator it and point it at the first value of the floating-point array sm cv::SparseMatIterator<uchar> it_end    = sm.end<uchar>(); Create a sparse matrix iterator it_end and initialize it to the value succeeding the final value in the byte array sm The Template Structure Thus far in this chapter, we have regularly eluded to the existence of template forms for almost all of the basic types. In fact, most programmers can get quite far into OpenCV without ever digging down into the

Create a sparse matrix iterator it and point it at the first value of the floating-point array sm cv::SparseMatIterator<uchar> it_end    = sm.end<uchar>(); Create a sparse matrix iterator it_end and initialize it to the value succeeding the final value in the byte array sm The Template Structure Thus far in this chapter, we have regularly eluded to the existence of template forms for almost all of the basic types. In fact, most programmers can get quite far into OpenCV without ever digging down into the templates. However, knowing how to use the templates directly can be of great help in getting things done. In this section, we will look at how that all works. If your C++ programming skills are not entirely up to par, you can probably just skim or skip over this section entirely.  OpenCV version 2.1 and later is built on a template meta-programming style similar to STL, Boost, and similar libraries. This sort of library design can be extremely powerful, both in terms of the quality and speed of the final

templates. However, knowing how to use the templates directly can be of great help in getting things done. In this section, we will look at how that all works. If your C++ programming skills are not entirely up to par, you can probably just skim or skip over this section entirely.  OpenCV version 2.1 and later is built on a template meta-programming style similar to STL, Boost, and similar libraries. This sort of library design can be extremely powerful, both in terms of the quality and speed of the final code, as well as the flexibility it allows the developer. In particular, template structures of the kind used in OpenCV allow for algorithms to be implemented in an abstracted way that does not specifically rely on the primitive types that are native to C++ or even native to OpenCV.  As an example, consider the cv::Point class. When you instantiate an object of type cv::Point, you are actually instantiating a template object of type cv::Point_<int>. (Note the trailing underscore—this is the general

final code, as well as the flexibility it allows the developer. In particular, template structures of the kind used in OpenCV allow for algorithms to be implemented in an abstracted way that does not specifically rely on the primitive types that are native to C++ or even native to OpenCV.  As an example, consider the cv::Point class. When you instantiate an object of type cv::Point, you are actually instantiating a template object of type cv::Point_<int>. (Note the trailing underscore—this is the general convention in the library used to indicate a template.) This template could have been instantiated with a different type than int, obviously. In fact, it could have been instantiated with any type

that supports the same basic set of operators as int (i.e., addition, subtraction, multiplication, etc.). For example, OpenCV provides a type cv::Complex that you could have used. You also could have used the STL complex type std::complex, which has nothing to do with OpenCV at all. The same is true for some other types of your own construction. This same concept generalizes to other type templates such as cv::Scalar_<> and cv::Rect_<>, as well as cv::Matx_<> and cv::Vec_<>.  cv::Mat_<> and cv::SparseMat_<> Are a Little Bit Different This concept also generalizes to cv::Mat_<> and cv::SparseMat_<>, but in a somewhat nontrivial way. When you use cv::Point2i, recall that this is nothing more or less than an alias (typedef) for cv::Point_<int>. In the case of the template cv::Mat and cv::Mat_<>, their relationship is not so simple. Recall that cv::Mat already has the capability of representing essentially any type, but that this is done at construction time by explicitly specifying the base type. In the case of

Are a Little Bit Different This concept also generalizes to cv::Mat_<> and cv::SparseMat_<>, but in a somewhat nontrivial way. When you use cv::Point2i, recall that this is nothing more or less than an alias (typedef) for cv::Point_<int>. In the case of the template cv::Mat and cv::Mat_<>, their relationship is not so simple. Recall that cv::Mat already has the capability of representing essentially any type, but that this is done at construction time by explicitly specifying the base type. In the case of cv::Mat_<>, the instantiated template is actually derived from the cv::Mat class, and in effect specializes that class. This simplifies access and other member functions that would otherwise need to be templated.  This is worth reiterating. The purpose of using the template forms cv::Mat_<> and cv::SparseMat_<> are so you don’t have to use the template forms of their member functions. Consider this example, where we have a matrix defined by:   cv::Mat m( 10, 10, cv::F32C2 ); Individual element accesses to

cv::Mat_<>, the instantiated template is actually derived from the cv::Mat class, and in effect specializes that class. This simplifies access and other member functions that would otherwise need to be templated.  This is worth reiterating. The purpose of using the template forms cv::Mat_<> and cv::SparseMat_<> are so you don’t have to use the template forms of their member functions. Consider this example, where we have a matrix defined by:   cv::Mat m( 10, 10, cv::F32C2 ); Individual element accesses to this matrix would need to specify the type of the matrix, as in the following:   m.at< Vec2f >( i0, i1 ) = cv::Vec2f( x, y ); Alternatively, if you had defined the matrix m using the template class, you could use at() without specialization, or even just use operator():   cv::Mat_<Vec2f> m( 10, 10 );   m.at( i0, i1 ) = cv::Vec2f( x, y );   // or…   m( i0, i1 ) = cv::Vec2f( x, y ); There is a great deal of simplification in your code that results from using these template definitions and because if this, it

this matrix would need to specify the type of the matrix, as in the following:   m.at< Vec2f >( i0, i1 ) = cv::Vec2f( x, y ); Alternatively, if you had defined the matrix m using the template class, you could use at() without specialization, or even just use operator():   cv::Mat_<Vec2f> m( 10, 10 );   m.at( i0, i1 ) = cv::Vec2f( x, y );   // or…   m( i0, i1 ) = cv::Vec2f( x, y ); There is a great deal of simplification in your code that results from using these template definitions and because if this, it is the preferred way to code.  These two ways of declaring a matrix and their associated .at methods are equivalent in efficiency. The second method is more “correct” because it allows the compiler to detect type mismatches when m is passed into a function that requires a certain type of matrix. If cv::Mat m(10, 10, cv::F32C2 ); is passed into  void foo((cv::Mat_<char> *)myMat); failure would occur during runtime in perhaps nonobvious ways. If you instead used cv::Mat_<Vec2f> m( 10, 10 ); failure would be

is the preferred way to code.  These two ways of declaring a matrix and their associated .at methods are equivalent in efficiency. The second method is more “correct” because it allows the compiler to detect type mismatches when m is passed into a function that requires a certain type of matrix. If cv::Mat m(10, 10, cv::F32C2 ); is passed into  void foo((cv::Mat_<char> *)myMat); failure would occur during runtime in perhaps nonobvious ways. If you instead used cv::Mat_<Vec2f> m( 10, 10 ); failure would be detected at compile time. Template forms can be used to create template functions that operate on an array of a particular type. Consider our example from the previous section, where we created a small sparse matrix and then printed out its nonzero elements. We might try writing a function to achieve this as follows: void print_matrix( const cv::SparseMat* sm ) {    cv::SparseMatConstIterator_<float> it     = sm.begin<float>();   cv::SparseMatConstIterator_<float> it_end = sm.end<float>();    for(; it !=

detected at compile time. Template forms can be used to create template functions that operate on an array of a particular type. Consider our example from the previous section, where we created a small sparse matrix and then printed out its nonzero elements. We might try writing a function to achieve this as follows: void print_matrix( const cv::SparseMat* sm ) {    cv::SparseMatConstIterator_<float> it     = sm.begin<float>();   cv::SparseMatConstIterator_<float> it_end = sm.end<float>();    for(; it != it_end; ++it) {     const cv::SparseMat::Node* node = it.node();     printf(" (%3d,%3d) %f\n", node->idx[0], node->idx[1], *it );

} } Though this function would compile and work when it is passed a two-dimensional matrix of type cv::F32, it would fail when a matrix of unexpected type was passed in. Let’s look at how we could make this function more general. The first thing we would want to address is the issue of the underlying data type. We could explicitly use the cv::SparseMat_<float> template, but it would be better still to make the function a template function. We would also need to get rid of the use of printf(), as it makes an explicit assumption that *it is a float. A better function might look like this:  template <class T> void print_matrix( const cv::SparseMat_<T>* sm ) {    cv::SparseMatConstIterator_<T> it     = sm->begin();   cv::SparseMatConstIterator_<T> it_end = sm->end();    for(; it != it_end; ++it) {     const typename cv::SparseMat_<T>::Node* node = it.node();     cout <<”( “ <<node->idx[0] <<”, “ <<node->idx[1]       <<” ) = “ <<*it <<endl;   } }  void calling_function1( void ) {   …   cv::SparseMat_<float> sm(

an explicit assumption that *it is a float. A better function might look like this:  template <class T> void print_matrix( const cv::SparseMat_<T>* sm ) {    cv::SparseMatConstIterator_<T> it     = sm->begin();   cv::SparseMatConstIterator_<T> it_end = sm->end();    for(; it != it_end; ++it) {     const typename cv::SparseMat_<T>::Node* node = it.node();     cout <<”( “ <<node->idx[0] <<”, “ <<node->idx[1]       <<” ) = “ <<*it <<endl;   } }  void calling_function1( void ) {   …   cv::SparseMat_<float> sm( ndim, size );   …   print_matrix<float>( &sm ); }  void calling_function2( void ) {   …   cv::SparseMat sm( ndim, size, cv::F32 );   …   print_matrix<float>( (cv::SparseMat_<float>*) &sm ); } It is worth picking apart these changes. First though, before looking at changes, notice that the template for our function takes a pointer of type const cv::SparseMat_<t>*, a pointer to a sparse matrix template object. There is a good reason to use a pointer and not a reference here, because the caller may have a

sm( ndim, size );   …   print_matrix<float>( &sm ); }  void calling_function2( void ) {   …   cv::SparseMat sm( ndim, size, cv::F32 );   …   print_matrix<float>( (cv::SparseMat_<float>*) &sm ); } It is worth picking apart these changes. First though, before looking at changes, notice that the template for our function takes a pointer of type const cv::SparseMat_<t>*, a pointer to a sparse matrix template object. There is a good reason to use a pointer and not a reference here, because the caller may have a cv::Mat object (as is done in calling_function2()) and not a cv::Mat_<> template object (as used in calling_function1()). The cv::Mat can be dereferenced and then explicitly cast to a pointer to the sparse matrix template object type. In the templated prototype, we have promoted the function to a template of class T, and now expect a cv::SparseMat_<T>* pointer as argument. In the next two lines, we declare our iterators using the template type, but begin() and end() no longer have templated instantiations.

cv::Mat object (as is done in calling_function2()) and not a cv::Mat_<> template object (as used in calling_function1()). The cv::Mat can be dereferenced and then explicitly cast to a pointer to the sparse matrix template object type. In the templated prototype, we have promoted the function to a template of class T, and now expect a cv::SparseMat_<T>* pointer as argument. In the next two lines, we declare our iterators using the template type, but begin() and end() no longer have templated instantiations. The reason for this is that sm is now an instantiated template, and because of that explicit instantiation, sm “knows” what sort of matrix it is, and thus specialization of begin() and end() is unnecessary. The declaration of the Node is similarly changed so that the Node we are using is explicitly taken from the cv::SparseMat_<T> instantiated template class.26 Finally, we change the printf() statement to use stream output to cout. This has the advantage that the printing is now agnostic to the type of

The reason for this is that sm is now an instantiated template, and because of that explicit instantiation, sm “knows” what sort of matrix it is, and thus specialization of begin() and end() is unnecessary. The declaration of the Node is similarly changed so that the Node we are using is explicitly taken from the cv::SparseMat_<T> instantiated template class.26 Finally, we change the printf() statement to use stream output to cout. This has the advantage that the printing is now agnostic to the type of *it.                                                              26 The appearance of the typename keyword here is probably somewhat mysterious to most readers. It is a result of the dependent scoping rules in C++. If you should forget it, however, most modern compilers (e.g., g++) will throw you a friendly message reminding you to add it.

Array Operators As we have seen so far in this chapter, there are many basic operations on arrays that are now handled by member functions of the array classes. In addition to those, however, there are many more operations that are most naturally represented as “friend” functions that either take array types as arguments, have array types as return values, or both. The functions, together with their arguments, will be covered in more detail after the table. For a shortcut, it is best to download the cheat sheet at http://docs.opencv.org/trunk/opencv_cheatsheet.pdf. Table 3-20: Basic matrix and image operators Function Description cv::abs()  Absolute value of all elements in an array cv::absdiff()   Absolute value of differences between two arrays cv::add()  Element-wise addition of two arrays cv::addWeighted()       Element-wise weighted addition of two arrays (alpha blending) cv::bitwise_and() Element-wise bit-level AND of two arrays cv::bitwise_not() Element-wise bit-level NOT of two arrays cv::bitwise_or()

at http://docs.opencv.org/trunk/opencv_cheatsheet.pdf. Table 3-20: Basic matrix and image operators Function Description cv::abs()  Absolute value of all elements in an array cv::absdiff()   Absolute value of differences between two arrays cv::add()  Element-wise addition of two arrays cv::addWeighted()       Element-wise weighted addition of two arrays (alpha blending) cv::bitwise_and() Element-wise bit-level AND of two arrays cv::bitwise_not() Element-wise bit-level NOT of two arrays cv::bitwise_or() Element-wise bit-level OR of two arrays cv::bitwise_xor() Element-wise bit-level XOR of two arrays cv::calcCovarMatrix() Compute covariance of a set of n-dimensional vectors cv::cartToPolar() Compute angle and magnitude from a two-dimensional vector field cv::checkRange() Check array for invalid values cv::compare()                Apply selected comparison operator to all elements in two arrays cv::completeSymm()                Symmetrize matrix by copying elements from one half to the other

Element-wise bit-level OR of two arrays cv::bitwise_xor() Element-wise bit-level XOR of two arrays cv::calcCovarMatrix() Compute covariance of a set of n-dimensional vectors cv::cartToPolar() Compute angle and magnitude from a two-dimensional vector field cv::checkRange() Check array for invalid values cv::compare()                Apply selected comparison operator to all elements in two arrays cv::completeSymm()                Symmetrize matrix by copying elements from one half to the other cv::convertScaleAbs()  Scale array, take absolute value, then convert to 8-bit unsigned cv::countNonZero()                Count nonzero elements in an array cv::arrToMat() Converts pre-version 2.1 array types to cv::Mat cv::dct() Compute discrete cosine transform of array cv::determinant() Compute determinant of a square matrix cv::dft()                Compute discrete Fourier transform of array cv::divide() Element-wise division of one array by another cv::eigen() Compute eigenvalues and eigenvectors of a square matrix

cv::exp() Element-wise exponentiation of array cv::extractImageCOI() Extract single channel from pre-version 2.1 array type cv::flip() Flip an array about a selected axis cv::gemm() Generalized matrix multiplication cv::getConvertElem() Get a single pixel type conversion function cv::getConvertScaleElem()                Get a single pixel type conversion and scale function cv::idct() Compute inverse discrete cosine transform of array cv::idft()  Compute inverse discrete Fourier transform of array cv::inRange() Test if elements of an array are within values of two other arrays cv::invert() Invert a square matrix cv::log() Element-wise natural log of array cv::magnitude() Compute magnitudes from a  two-dimensional vector field cv::LUT() Converts array to indices of a look-up table cv::Mahalanobis() Compute Mahalanobis distance between two vectors cv::max() Compute element-wise maxima between two arrays cv::mean() Compute the average of the array elements cv::meanStdDev() Compute the average and standard

Test if elements of an array are within values of two other arrays cv::invert() Invert a square matrix cv::log() Element-wise natural log of array cv::magnitude() Compute magnitudes from a  two-dimensional vector field cv::LUT() Converts array to indices of a look-up table cv::Mahalanobis() Compute Mahalanobis distance between two vectors cv::max() Compute element-wise maxima between two arrays cv::mean() Compute the average of the array elements cv::meanStdDev() Compute the average and standard deviation of the array elements cv::merge() Merge several single-channel arrays into one multichannel arrays cv::min() Compute element-wise minima between two arrays cv::minMaxLoc() Find minimum and maximum values in an array cv::mixChannels() Shuffle channels from input arrays to output arrays cv::mulSpectrums() Compute element-wise multiplication of two Fourier spectra cv::multiply() Element-wise multiplication of two arrays cv::mulTransposed() Calculate matrix product of one array  cv::norm() Compute normalized

of the array elements cv::merge() Merge several single-channel arrays into one multichannel arrays cv::min() Compute element-wise minima between two arrays cv::minMaxLoc() Find minimum and maximum values in an array cv::mixChannels() Shuffle channels from input arrays to output arrays cv::mulSpectrums() Compute element-wise multiplication of two Fourier spectra cv::multiply() Element-wise multiplication of two arrays cv::mulTransposed() Calculate matrix product of one array  cv::norm() Compute normalized correlations between two arrays cv::normalize() Normalize elements in an array to some value cv::perspectiveTransform()            Perform perspective matrix transform of a list of vectors

cv::phase() Compute orientations from a two-dimensional vector field cv::polarToCart()   Compute two-dimensional vector field from angles and magnitudes cv::pow()  Raise every element of an array to a given power cv::randu()  Fill a given array with uniformly distributed random numbers cv::randn()  Fill a given array with normally distributed random numbers cv::randShuffle()  Randomly shuffle array elements cv::reduce() Reduce a two-dimensional array to a vector by a given operation cv::repeat() Tile the contents of one array into another cv::saturate_cast<>()  Template function for guarding against overflow  cv::scaleAdd() Element-wise sum of two arrays with optional scaling of the first cv::setIdentity() Set all elements of an array to 1 for the diagonal and 0 otherwise cv::solve() Solve a system of linear equations cv::solveCubic() Find the (only) real roots of a cubic equation  cv::solvePoly() Find the complex roots of a polynomial equation cv::sort() Sort elements in either the rows or columns in an

contents of one array into another cv::saturate_cast<>()  Template function for guarding against overflow  cv::scaleAdd() Element-wise sum of two arrays with optional scaling of the first cv::setIdentity() Set all elements of an array to 1 for the diagonal and 0 otherwise cv::solve() Solve a system of linear equations cv::solveCubic() Find the (only) real roots of a cubic equation  cv::solvePoly() Find the complex roots of a polynomial equation cv::sort() Sort elements in either the rows or columns in an array cv::sortIdx() Same as cv::sort() except array is unmodified and indices are returned cv::split() Split a multichannel array into multiple single-channel arrays cv::sqrt() Compute element-wise square root of an array cv::subtract() Element-wise subtraction of one array from another cv::sum() Sum all elements of an array cv::theRNG() Return a random number generator cv::trace()  Compute the trace of an array cv::transform() Apply matrix transformation on every element of an array cv::transpose()

in an array cv::sortIdx() Same as cv::sort() except array is unmodified and indices are returned cv::split() Split a multichannel array into multiple single-channel arrays cv::sqrt() Compute element-wise square root of an array cv::subtract() Element-wise subtraction of one array from another cv::sum() Sum all elements of an array cv::theRNG() Return a random number generator cv::trace()  Compute the trace of an array cv::transform() Apply matrix transformation on every element of an array cv::transpose() Transpose all elements of an array across the diagonal In these functions, some general rules are followed. To the extent that any exceptions exist, they are noted in the function descriptions. Because one or more of these rules applies to just about every function described in this section, they are listed here for convenience: Saturation Outputs of calculations are saturation casted to the type of the output array.

Output The output array will be created with cv::Mat::create() if its type and size do not match the inputs. Scalars Many functions such as cv::add() allow for addition of two arrays or an array and a scalar. Where the prototypes make the option clear, the result of providing a scalar argument is the same as if a second array had been provided with the same scalar value in every element. Masks Whenever a mask argument is present for a function, the output will only be computed for those elements where the mask value corresponding to that element in the output array is nonzero. dtype Many arithmetic and similar functions do not require the types of the input arrays to be the same, and even if they are the same, the output array may be of a different type than the inputs. In these cases, the output array must have its depth explicitly specified. This is done with the dtype argument. When present, dtype can be set to any of the basic types (cv::F32 etc.) and the result array will be of that type. If the input

mask value corresponding to that element in the output array is nonzero. dtype Many arithmetic and similar functions do not require the types of the input arrays to be the same, and even if they are the same, the output array may be of a different type than the inputs. In these cases, the output array must have its depth explicitly specified. This is done with the dtype argument. When present, dtype can be set to any of the basic types (cv::F32 etc.) and the result array will be of that type. If the input arrays have the same type, then dtype can be set to its default value of -1, and the resulting type will be the same as the types of the input arrays. In Place Operation Unless otherwise specified, any operation with both an array input and an array output that are of the same size and type can use the same array for both (i.e., it is allowable to write the output on top of an input). Multichannel For those operations that do not naturally make use of multiple channels, if given multichannel arguments, each

have the same type, then dtype can be set to its default value of -1, and the resulting type will be the same as the types of the input arrays. In Place Operation Unless otherwise specified, any operation with both an array input and an array output that are of the same size and type can use the same array for both (i.e., it is allowable to write the output on top of an input). Multichannel For those operations that do not naturally make use of multiple channels, if given multichannel arguments, each channel is processed separately. cv::abs() cv::MatExpr cv::abs( cv::InputArray src ); cv::MatExpr cv::abs( const cv::MatExpr& src                  // Matrix expresion ); These functions compute the absolute value of an array or of some expression of arrays. The most common usage computes the absolute value of every element in an array. Because cv::abs() can take a matrix expression, it is able to recognize certain special cases and handle them appropriately. In fact, calls to cv::abs() are actually converted to

is processed separately. cv::abs() cv::MatExpr cv::abs( cv::InputArray src ); cv::MatExpr cv::abs( const cv::MatExpr& src                  // Matrix expresion ); These functions compute the absolute value of an array or of some expression of arrays. The most common usage computes the absolute value of every element in an array. Because cv::abs() can take a matrix expression, it is able to recognize certain special cases and handle them appropriately. In fact, calls to cv::abs() are actually converted to calls to cv::absDiff() or other functions, and handled by those functions. In particular, the following special cases are implemented: • m2 = cv::abs( m0 - m1 ) is converted to cv::absDiff( m0, m1, m2 ) • m2 = cv::abs( m0 ) is converted to m2 = cv::absDiff( m0, cv::Scalar::all(0), m2 )  • m2 = cv::Mat_<Vec<uchar,n> >( cv::abs( alpha*m0 + beta ) ) (for alpha, beta real numbers) is converted to cv::convertScaleAbs( m0, m2, alpha, beta ) The third case might seem obscure, but this is just the case of computing a

to cv::absDiff() or other functions, and handled by those functions. In particular, the following special cases are implemented: • m2 = cv::abs( m0 - m1 ) is converted to cv::absDiff( m0, m1, m2 ) • m2 = cv::abs( m0 ) is converted to m2 = cv::absDiff( m0, cv::Scalar::all(0), m2 )  • m2 = cv::Mat_<Vec<uchar,n> >( cv::abs( alpha*m0 + beta ) ) (for alpha, beta real numbers) is converted to cv::convertScaleAbs( m0, m2, alpha, beta ) The third case might seem obscure, but this is just the case of computing a scale and offset (either of which could be trivial) to an n-channel array. This is typical of what one might do when computing a contrast correction for an image, for example. In the cases that are implemented by cv::absDiff(), the result array will have the same size and type as the input array. In the case implemented by cv::convertScaleAbs(), however, the result type of the return array will always be cv::U8.

cv::absdiff() void cv::absdiff(   cv::InputArray  src1,                   // First input array    cv::InputArray  src2,                   // Second input array    cv::OutputArray dst                     // Result array  )    dsti=saturate†src1i−src2i() Given two arrays, cv::absdiff() computes the difference between each pair of corresponding elements in those arrays, and places the absolute value of that difference into the corresponding element of the destination array.  cv::add()  void cv::add(   cv::InputArray  src1,                   // First input array    cv::InputArray  src2,                   // Second input array    cv::OutputArray dst,                    // Result array    cv::InputArray  mask  = cv::noArray(),  // Optional mask, compute only where nonzero   int             dtype = -1              // Output type for result array )      dsti=saturate†src1i+src2i() cv::add() is a simple addition function: it adds all of the elements in src1 to the corresponding elements in src2 and puts the results in

src1,                   // First input array    cv::InputArray  src2,                   // Second input array    cv::OutputArray dst,                    // Result array    cv::InputArray  mask  = cv::noArray(),  // Optional mask, compute only where nonzero   int             dtype = -1              // Output type for result array )      dsti=saturate†src1i+src2i() cv::add() is a simple addition function: it adds all of the elements in src1 to the corresponding elements in src2 and puts the results in dst.  For simple cases, the same result can be achieved with the matrix operation: dst = src1 + src2; Accumulation is also supported:  dst += src1;  cv::addWeighted()  void cv::addWeighted(   cv::InputArray  src1,                   // First input array    double          alpha,                  // Weight for first input array   cv::InputArray  src2,                   // Second input array    double          beta,                   // Weight for second input array    double          gamma,                  //

For simple cases, the same result can be achieved with the matrix operation: dst = src1 + src2; Accumulation is also supported:  dst += src1;  cv::addWeighted()  void cv::addWeighted(   cv::InputArray  src1,                   // First input array    double          alpha,                  // Weight for first input array   cv::InputArray  src2,                   // Second input array    double          beta,                   // Weight for second input array    double          gamma,                  // Offset added to weighted sum   cv::OutputArray dst,                    // Result array    int             dtype = -1              // Output type for result array ) The function cv::addWeighted() is similar to cvAdd() except that the result written to dst is computed according to the following formula:    dsti=saturate†src1i*αα†+src2i*ββ†+γ()†src1i*αα†+src2i*ββ†+γ().

The two source images, src1 and src2 may be of any pixel type as long as both are of the same type. They may also have any number of channels (grayscale, color, etc.), as long as they agree.  This function can be used to implement alpha blending [Smith79; Porter84]; that is, it can be used to blend one image with another. In this case, the parameter alpha is the blending strength of src1, and beta is the blending strength of src2. You can convert to the standard alpha blend equation by choosing α between 0 and 1, setting  β=1−α, and setting γ to 0; this yields:

However, cv::addWeighted() gives us a little more flexibility—both in how we weight the blended images and in the additional parameter γ, which allows for an additive offset to the resulting destination image. For the general form, you will probably want to keep alpha and beta at 0 or above, and their sum at no more than 1; gamma may be set depending on average or max image value to scale the pixels up. A program showing the use of alpha blending is shown in Example 3-1. Example 3-1. Complete program to alpha blend the ROI starting at  0,0() in src2 with the ROI starting at   x,y() in src1 // alphablend <imageA> <image B> <x> <y> <width> <height> alpha> <beta> // #include <cv.h> #include <highgui.h>  int main(int argc, char** argv) {    cv::Mat src1 = cv::imread(argv[1],1);   cv::Mat src2 = cv::imread(argv[2],1);    if( argc==9 && !src1.empty() && !src2.empty() ) {      int    x     = atoi(argv[3]);     int    y     = atoi(argv[4]);     int    w     = atoi(argv[5]);     int    h     = atoi(argv[6]);

alpha blend the ROI starting at  0,0() in src2 with the ROI starting at   x,y() in src1 // alphablend <imageA> <image B> <x> <y> <width> <height> alpha> <beta> // #include <cv.h> #include <highgui.h>  int main(int argc, char** argv) {    cv::Mat src1 = cv::imread(argv[1],1);   cv::Mat src2 = cv::imread(argv[2],1);    if( argc==9 && !src1.empty() && !src2.empty() ) {      int    x     = atoi(argv[3]);     int    y     = atoi(argv[4]);     int    w     = atoi(argv[5]);     int    h     = atoi(argv[6]);     double alpha = (double)atof(argv[7]);     double beta  = (double)atof(argv[8]);      cv::Mat roi1( src1, cv::Rect(x,y,w,h) );     cv::Mat roi2( src2, cv::Rect(0,0,w,h) );      cv::addWeighted( roi1, alpha, roi2, beta, 0.0, roi1 );      cv::namedWindow( "Alpha Blend", 1 );     cv::imshow( "Alpha Blend", src2 );     cv::waitKey( 0 );   }    return 0; } The code in Example 3-1 takes two source images: the primary one (src1) and the one to blend (src2). It reads in a rectangle ROI for src1 and applies an ROI of

alpha = (double)atof(argv[7]);     double beta  = (double)atof(argv[8]);      cv::Mat roi1( src1, cv::Rect(x,y,w,h) );     cv::Mat roi2( src2, cv::Rect(0,0,w,h) );      cv::addWeighted( roi1, alpha, roi2, beta, 0.0, roi1 );      cv::namedWindow( "Alpha Blend", 1 );     cv::imshow( "Alpha Blend", src2 );     cv::waitKey( 0 );   }    return 0; } The code in Example 3-1 takes two source images: the primary one (src1) and the one to blend (src2). It reads in a rectangle ROI for src1 and applies an ROI of the same size to src2, but located at the origin. It reads in alpha and beta levels but sets gamma to 0. Alpha blending is applied using cv::addWeighted(), and the results are put into src1 and displayed. Example output is shown in Figure 3-1, where the face of a child is blended onto a cat. Note that the code took the same ROI as in the ROI addition example in Figure 3-1. This time we used the ROI as the target blending region.

Figure 3-1: The face of a child is alpha blended onto the face of a cat cv::bitwise_and() void cv::bitwise_and(   cv::InputArray  src1,                   // First input array    cv::InputArray  src2,                   // Second input array    cv::OutputArray dst,                    // Result array    cv::InputArray  mask  = cv::noArray(),  // Optional mask, compute only where nonzero )    dsti=src1i∧src2i
 cv::bitwise_and() is a per-element bitwise conjunction operation. For every element in src1, the bitwise AND is computed with the corresponding element in src2 and put into the corresponding element of dst. If you are not using a mask, the same result can be achieved with the matrix operation: dst = src1 & src2; cv::bitwise_not() void cv::bitwise_not(   cv::InputArray  src,                    // Input array    cv::OutputArray dst,                    // Result array    cv::InputArray  mask  = cv::noArray(),  // Optional mask, compute only where nonzero   )    dsti=∼src1i

cv::bitwise_not() is a per-element bitwise inversion operation. For every element in src1 the logical inversion is computed and placed into the corresponding element of dst. If you are not using a mask, the same result can be achieved with the matrix operation: dst = !src1;

cv::bitwise_or() void cv::bitwise_and(   cv::InputArray  src1,                   // First input array    cv::InputArray  src2,                   // Second input array    cv::OutputArray dst,                    // Result array    cv::InputArray  mask  = cv::noArray(),  // Optional mask, compute only where nonzero )    dsti=src1i∨src2i
    dsti=src1i∨sc
 cv::bitwise_or() is a per-element bitwise disjunction operation. For every element in src1, the bitwise OR is computed with the corresponding element in src2 and put into the corresponding element of dst. If you are not using a mask, the same result can be achieved with the matrix operation: dst = src1 | src2; cv::bitwise_xor() void cv::bitwise_and(   cv::InputArray  src1,                   // First input array    cv::InputArray  src2,                   // Second input array    cv::OutputArray dst,                    // Result array    cv::InputArray  mask  = cv::noArray(),  // Optional mask, compute only where nonzero )    dsti=src1i⊕src2i
     dsti=src1i!sc

cv::bitwise_and() is a per-element bitwise “exclusive or” operation. For every element in src1, the bitwise XOR is computed with the corresponding element in src2 and put into the corresponding element of dst. If you are not using a mask, the same result can be achieved with the matrix operation: dst = src1 ^ src2; cv::calcCovarMatrix() void cv::calcCovarMatrix(   const cv::Mat* samples,                // C-array of n-by-1 or 1-by-n matrices   int            nsamples,               // number of matrices pointed to by ‘samples’   cv::Mat&       covar,                  // reference to return array for covariance   cv::Mat&       mean,                   // reference to return array for mean   int            flags,                  // special variations, see Table 3-21   int            ctype = cv::F64          // output matrix type for covar );  void cv::calcCovarMatrix(   cv::InputArray samples,                // n-by-m matrix, use ‘flags’ for which is which   cv::Mat&       covar,                  //

pointed to by ‘samples’   cv::Mat&       covar,                  // reference to return array for covariance   cv::Mat&       mean,                   // reference to return array for mean   int            flags,                  // special variations, see Table 3-21   int            ctype = cv::F64          // output matrix type for covar );  void cv::calcCovarMatrix(   cv::InputArray samples,                // n-by-m matrix, use ‘flags’ for which is which   cv::Mat&       covar,                  // reference to return array for covariance   cv::Mat&       mean,                   // reference to return array for mean   int            flags,                  // special variations, see Table 3-21   int            ctype = cv::F64          // output matrix type for covar ); Given any number of vectors, cv::calcCovarMatrix() will compute the mean and covariance matrix for the Gaussian approximation to the distribution of those points. This can be used in many ways,

of course, and OpenCV has some additional flags that will help in particular contexts (see Table 3-21). These flags may be combined by the standard use of the Boolean OR operator. Table 3-21: Possible components of flags argument to cv::calcCovarMatrix() Flag in flags argument Meaning cv::COVAR_NORMAL  Compute mean and covariance cv::COVAR_SCRAMBLED  Fast PCA “scrambled” covariance cv::COVAR_USE_AVERAGE                    Use mean as input instead of computing it cv::COVAR_SCALE   Rescale output covariance matrix cv::COVAR_ROWS   Use rows of samples for input vectors cv::COVAR_COLS   Use columns of samples for input vectors There are two basic calling conventions for cv::calcCovarMatrix(). In the first, a pointer to an array of cv::Mat objects is passed along with nsamples, the number of matrices in that array. In this case, the arrays may be n-by-1 or 1-by-n. The second calling convention is to pass a single array that is n-by-m. In this case, either the flag cv::COVAR_ROWS should be supplied, to indicate

matrix cv::COVAR_ROWS   Use rows of samples for input vectors cv::COVAR_COLS   Use columns of samples for input vectors There are two basic calling conventions for cv::calcCovarMatrix(). In the first, a pointer to an array of cv::Mat objects is passed along with nsamples, the number of matrices in that array. In this case, the arrays may be n-by-1 or 1-by-n. The second calling convention is to pass a single array that is n-by-m. In this case, either the flag cv::COVAR_ROWS should be supplied, to indicate that there are n (row) vectors of length m, or cv::COVAR_COLS should be supplied, to indicate that there are m (column) vectors of length n. The results will be placed in covar in all cases, but the exact meaning of avg depends on the flag values (see Table 3-21). The flags cv::COVAR_NORMAL and cv::COVAR_SCRAMBLED are mutually exclusive; you should use one or the other but not both. In the case of cv::COVAR_NORMAL, the function will simply compute the mean and covariance of the points provided:   Thus the

that there are n (row) vectors of length m, or cv::COVAR_COLS should be supplied, to indicate that there are m (column) vectors of length n. The results will be placed in covar in all cases, but the exact meaning of avg depends on the flag values (see Table 3-21). The flags cv::COVAR_NORMAL and cv::COVAR_SCRAMBLED are mutually exclusive; you should use one or the other but not both. In the case of cv::COVAR_NORMAL, the function will simply compute the mean and covariance of the points provided:   Thus the normal covariance   Σnormal2 is computed from the  m vectors of length  n, where  vn

is defined as the  nth element of the average vector:  v. The resulting covariance matrix is an  n-by- n matrix. The factor  z is an optional scale factor; it will be set to 1 unless the cv::COVAR_SCALE flag is used. In the case of cv::COVAR_SCRAMBLED, cv::calcCovarMatrix() will compute the following:   This matrix is not the usual covariance matrix (note the location of the transpose operator). This matrix is computed from the same m vectors of length n, but the resulting scrambled covariance matrix is an m-by-m matrix. This matrix is used in some specific algorithms such as fast PCA for very large vectors (as in the eigenfaces technique for face recognition). The flag cv::COVAR_USE_AVG is used when the mean of the input vectors is already known. In this case, the argument avg is used as an input rather than an output, which reduces computation time. Finally, the flag cv::COVAR_SCALE is used to apply a uniform scale to the covariance matrix calculated. This is the factor z in the preceding equations. When

m-by-m matrix. This matrix is used in some specific algorithms such as fast PCA for very large vectors (as in the eigenfaces technique for face recognition). The flag cv::COVAR_USE_AVG is used when the mean of the input vectors is already known. In this case, the argument avg is used as an input rather than an output, which reduces computation time. Finally, the flag cv::COVAR_SCALE is used to apply a uniform scale to the covariance matrix calculated. This is the factor z in the preceding equations. When used in conjunction with the cv::COVAR_NORMAL flag, the applied scale factor will be   1/m (or, equivalently, 1.0/nsamples). If instead cv::COVAR_SCRAMBLED is used, then the value of z will be   1/n (the inverse of the length of the vectors). The input and output arrays to cv::calcCovarMatrix() should all be of the same floating-point type. The size of the resulting matrix covar will be either  n-by- n or  m-by- m depending on whether the standard or scrambled covariance is being computed. It should be noted

in conjunction with the cv::COVAR_NORMAL flag, the applied scale factor will be   1/m (or, equivalently, 1.0/nsamples). If instead cv::COVAR_SCRAMBLED is used, then the value of z will be   1/n (the inverse of the length of the vectors). The input and output arrays to cv::calcCovarMatrix() should all be of the same floating-point type. The size of the resulting matrix covar will be either  n-by- n or  m-by- m depending on whether the standard or scrambled covariance is being computed. It should be noted that when using the cv::Mat* form, the “vectors” input in samples do not actually have to be one-dimensional; they can be two-dimensional objects (e.g., images) as well.

cv::cartToPolar() void cv::cartToPolar(   cv::InputArray  x,   cv::InputArray  y,   cv::OutputArray magnitude,   cv::OutputArray angle,   bool            angleInDegrees = false );    magnitudei=†xi2+yi2    anglei=atan2yi,†xi() This function cv::cartToPolar() takes two input arrays x and y, which are taken to be the  x and  y components of a vector field (note that this is not a single two-channel array, but two separate arrays). The arrays x and y must be of the same size. cv::cartToPolar() then computes the polar representation of each of those vectors. The magnitude of each vector is placed into the corresponding location in magnitude, and the orientation of each vector is placed into the corresponding location in angle. The returned angles are in radians unless the Boolean variable angleInDegrees is set to true. cv::checkRange() bool cv::checkRange(   cv::InputArray src,   bool           quiet  = true,   Point*     pos    = 0,            // if non-Null, location of first outlier   double         minVal =

polar representation of each of those vectors. The magnitude of each vector is placed into the corresponding location in magnitude, and the orientation of each vector is placed into the corresponding location in angle. The returned angles are in radians unless the Boolean variable angleInDegrees is set to true. cv::checkRange() bool cv::checkRange(   cv::InputArray src,   bool           quiet  = true,   Point*     pos    = 0,            // if non-Null, location of first outlier   double         minVal = -DBL_MAX,     // Lower check bound (inclusive)   double         maxVal =  DBL_MAX      // Upper check bound (exclusive) ); This function cv::checkRange() tests every element of the input array src and determines if that element is in a given range. The range is set by minVal and maxVal, but any NaN or inf value is also considered out of range. If an out-of-range value is found, an exception will be thrown unless quiet is set to true, in which case the return value of cv::checkRange() will be true if all

= -DBL_MAX,     // Lower check bound (inclusive)   double         maxVal =  DBL_MAX      // Upper check bound (exclusive) ); This function cv::checkRange() tests every element of the input array src and determines if that element is in a given range. The range is set by minVal and maxVal, but any NaN or inf value is also considered out of range. If an out-of-range value is found, an exception will be thrown unless quiet is set to true, in which case the return value of cv::checkRange() will be true if all values are in range and false if any value is out of range. If the pointer pos is not NULL, then the location of the first outlier will be stored in pos. cv::compare() bool cv::compare(   cv::InputArray  src1,                   // First input array    cv::InputArray  src2,                   // Second input array    cv::OutputArray dst,                    // Result array    int             cmpop                   // Comparison operator, see Table 3-22 ) This function makes element-wise comparisons between

values are in range and false if any value is out of range. If the pointer pos is not NULL, then the location of the first outlier will be stored in pos. cv::compare() bool cv::compare(   cv::InputArray  src1,                   // First input array    cv::InputArray  src2,                   // Second input array    cv::OutputArray dst,                    // Result array    int             cmpop                   // Comparison operator, see Table 3-22 ) This function makes element-wise comparisons between corresponding pixels in two arrays, src1 and src2, and places the results in the image dst. cv::compare() takes as its last argument a comparison operator, which may be any of the types listed in Table 3-22. In each case, the result dst will be an 8-bit array where pixels that match are marked with 255 and mismatches are set to 0. Table 3-22 Values of cmpop used by cv::compare() and the resulting comparison operation performed Value of cmp_op Comparison cv::CMP_EQ                    (src1i == src2i)

corresponding pixels in two arrays, src1 and src2, and places the results in the image dst. cv::compare() takes as its last argument a comparison operator, which may be any of the types listed in Table 3-22. In each case, the result dst will be an 8-bit array where pixels that match are marked with 255 and mismatches are set to 0. Table 3-22 Values of cmpop used by cv::compare() and the resulting comparison operation performed Value of cmp_op Comparison cv::CMP_EQ                    (src1i == src2i)  cv::CMP_GT                    (src1i  > src2i)  cv::CMP_GE                    (src1i >= src2i)

cv::CMP_LT                    (src1i  < src2i)  cv::CMP_LE                    (src1i <= src2i)  cv::CMP_NE                    (src1i != src2i)  All the listed comparisons are done with the same functions; you just pass in the appropriate argument to indicate what you would like done.  These comparison functions are useful, for example, in background subtraction to create a mask of the changed pixels (e.g., from a security camera) such that only novel information is pulled out of the image. These same results can be achieved with the matrix operations: dst = src1 == src2; dst = src1  > src2; dst = src1 >= src2; dst = src1  < src2; dst = src1 <= src2; dst = src1 != src2; cv::completeSymm() bool cv::completeSymm(    cv::InputArray mtx,    bool           lowerToUpper = false  )    !mtxij=mtxji!†i>j!
     (lowerToUpper = false)    !mtxij=mtxji!†j>i!

(lowerToUpper = false) Given a matrix (an array of dimension two) mtx, cv::completeSymm() symmetrizes the matrix by copying.27 Specifically, all of the elements from the lower triangle are copied to their transpose position on the upper triangle of the matrix. The diagonal elements of the mtx are left unchanged. If the flag lowerToUpper is set to true, then the elements from the upper triangle are copied into the lower triangle instead. cv::convertScaleAbs() void cv::convertScaleAbs(   cv::InputArray  src,                    // Input array    cv::OutputArray dst,                    // Result array    double          alpha = 1.0,            // Multiplicative scale factor   double          beta  = 0.0             // Additive offset factor );    dsti=saturateucharα*srci+β() The cv::convertScaleAbs() function is actually several functions rolled into one; it will perform four operations in sequence. The first operation is to rescale the source image by the factor alpha, the second is to offset by (add) the

src,                    // Input array    cv::OutputArray dst,                    // Result array    double          alpha = 1.0,            // Multiplicative scale factor   double          beta  = 0.0             // Additive offset factor );    dsti=saturateucharα*srci+β() The cv::convertScaleAbs() function is actually several functions rolled into one; it will perform four operations in sequence. The first operation is to rescale the source image by the factor alpha, the second is to offset by (add) the factor beta, the third is to compute the absolute value of that sum, and the fourth operation is to cast that result (with saturation) to unsigned char (8-bit).                                                              27 Mathematically inclined readers will realize that there are other symmetrizing processes for matrices that are more “natural” than this operation, but this particular operation is useful in its own right, for example to complete a matrix when only half of it was computed, and so is

the factor beta, the third is to compute the absolute value of that sum, and the fourth operation is to cast that result (with saturation) to unsigned char (8-bit).                                                              27 Mathematically inclined readers will realize that there are other symmetrizing processes for matrices that are more “natural” than this operation, but this particular operation is useful in its own right, for example to complete a matrix when only half of it was computed, and so is exposed in the library.

When you simply pass the default values (alpha = 1.0 or beta = 0.0), you need not have performance fears; OpenCV is smart enough to recognize these cases and not waste processor time on useless operations.  A similar result can be achieved, with somewhat greater generality using matrix algebra: Mat_<uchar> B = saturate_cast<uchar>(    cv::abs( src * alpha + beta )  ) This method will also generalize to types other than uchar: Mat_<unsigned short> B = saturate_cast<unsigned short>(    cv::abs( src * alpha + beta )  ) cv::countNonZero() int cv::countNonZero(              // Return number of non-zero elements in mtx   cv::InputArray mtx,              // Input array );    count=mtxi≠0∑1

cv::countNonZero() returns the number of nonzero pixels in the array mtx. cv::cvarrToMat() cv::Mat cv::cvarrToMat(   const CvArr* src,              // Input array: CvMat, IplImage, or CvMatND   bool         copyData = false, // if false just make new header, else copy data   bool         allowND  = true,  // if true, and possible, convert CvMatND to Mat   int          coiMode  = 0      // if 0: error if COI set, if 1: ignore COI ); cv::cvarrToMat() is used when you have an “old-style” (pre-version 2.1) image or matrix type and you want to convert it to a “new-style” (version 2.1 or later, which uses C++) cv::Mat object. By default, only the header for the new array is constructed without copying the data. Instead, the data pointers in the new header point to the existing data array (so do not deallocate it while the cv::Mat header is in use). If you want the data copied, just set copyData to true, and then you can freely do away with the original data object. cv::cvarrToMat() can also take CvMatND

2.1) image or matrix type and you want to convert it to a “new-style” (version 2.1 or later, which uses C++) cv::Mat object. By default, only the header for the new array is constructed without copying the data. Instead, the data pointers in the new header point to the existing data array (so do not deallocate it while the cv::Mat header is in use). If you want the data copied, just set copyData to true, and then you can freely do away with the original data object. cv::cvarrToMat() can also take CvMatND structures, but it cannot handle all cases. The key requirement for the conversion is that the matrix should be continuous, or at least it should be representable as a sequence of continuous matrices. Specifically, A.dim[i].size*A.dim.step[i] should be equal to A.dim.step[i-1] for all i, or at worst all but one. If allowND is set to true (default), cv::cvarrToMat() will attempt the conversion when it encounters a CvMatND structure, and throw an exception if that conversion is not possible (the condition

structures, but it cannot handle all cases. The key requirement for the conversion is that the matrix should be continuous, or at least it should be representable as a sequence of continuous matrices. Specifically, A.dim[i].size*A.dim.step[i] should be equal to A.dim.step[i-1] for all i, or at worst all but one. If allowND is set to true (default), cv::cvarrToMat() will attempt the conversion when it encounters a CvMatND structure, and throw an exception if that conversion is not possible (the condition above). If allowND is set to false, then an exception will be thrown whenever a CvMatND structure is encountered. Because the concept of COI28 is handled differently in the post-version 2.1 library (which is to say, it no longer exists), COI has to be handled during the conversion. If the argument coiMode is 0, then an exception will be thrown when src contains an active COI. If coiMode is nonzero, then no error will be reported, and instead a cv::Mat header, which corresponds to the entire image, will be

above). If allowND is set to false, then an exception will be thrown whenever a CvMatND structure is encountered. Because the concept of COI28 is handled differently in the post-version 2.1 library (which is to say, it no longer exists), COI has to be handled during the conversion. If the argument coiMode is 0, then an exception will be thrown when src contains an active COI. If coiMode is nonzero, then no error will be reported, and instead a cv::Mat header, which corresponds to the entire image, will be returned, ignoring                                                              28 “COI” is an old concept from the pre-v2 library which meant “Channel of interest”.  In the old IplImage class, this COI was analogous to ROI (Region of interest), and could be set to cause certain functions to act only on the indicated channel.

the COI. (If you want to handle COI properly, you will have to check yourself if the image has the COI set, and if so, use cv::extractImageCOI() to create a header for just that channel.) Most of the time, this function is used to help migrate old-style code to the new. In such case, you will probably need to both convert old CvArr* style structures to cv::Mat, as well as the reverse operation. The reverse operation is done using cast operators. If, for example, you have a matrix you defined as cv::Mat A, you can convert that to an IplImage* pointer simply with:  Mat A( 640, 480, cv::U8C3 ); IplImage img = A; // casting is implicit in assignment  cv::dct() void cv::dct(   cv::InputArray  src,                    // Input array   cv::OutputArray dst,                    // Result array    int             flags,                  // for inverse transform or row-by-row ); This function performs both the discrete cosine transform and the inverse transform depending on the flags argument. The source array src must be

can convert that to an IplImage* pointer simply with:  Mat A( 640, 480, cv::U8C3 ); IplImage img = A; // casting is implicit in assignment  cv::dct() void cv::dct(   cv::InputArray  src,                    // Input array   cv::OutputArray dst,                    // Result array    int             flags,                  // for inverse transform or row-by-row ); This function performs both the discrete cosine transform and the inverse transform depending on the flags argument. The source array src must be either one- or two-dimensional, and the size should be an even number (you can pad the array if necessary). The result array dst will have the same type and size as src. The argument flags is a bit-field and can be set to one or both of DCT_INVERSE or DCT_ROWS. If DCT_INVERSE is set, then the inverse transform is done instead of the forward transform. If the flag DCT_ROWS is set, then a two-dimensional n-by-m input is treated as n distinct one-dimensional vectors of length m. In this case, each such vector

either one- or two-dimensional, and the size should be an even number (you can pad the array if necessary). The result array dst will have the same type and size as src. The argument flags is a bit-field and can be set to one or both of DCT_INVERSE or DCT_ROWS. If DCT_INVERSE is set, then the inverse transform is done instead of the forward transform. If the flag DCT_ROWS is set, then a two-dimensional n-by-m input is treated as n distinct one-dimensional vectors of length m. In this case, each such vector will be transformed independently. The performance of cv::dct() depends strongly on the exact size of the arrays passed to it, and this relationship is not monotonic. There are just some sizes that work better than others. It is recommended that when passing an array to cv::dct(), you should first determine the most optimal size that is larger than your array, and extend your array to that size. OpenCV provides a convenient routine to compute such values for you, called cv::getOptimalDFTSize(). As

vector will be transformed independently. The performance of cv::dct() depends strongly on the exact size of the arrays passed to it, and this relationship is not monotonic. There are just some sizes that work better than others. It is recommended that when passing an array to cv::dct(), you should first determine the most optimal size that is larger than your array, and extend your array to that size. OpenCV provides a convenient routine to compute such values for you, called cv::getOptimalDFTSize(). As implemented, the discrete cosine transform of a vector of length n is computed using the discrete Fourier transform (cv::dft()) on a vector of length n/2. This means that to get the optimal size for a call to cv::dct(), you should compute it like this: size_t optimal_dct_size = 2 * getOptimalDFTSize( (N+1) / 2 ); This function (and discrete transforms in general) is covered in much greater detail in Chapter 6, General Image Transforms. In that section, we will discuss the details of how to pack and unpack

implemented, the discrete cosine transform of a vector of length n is computed using the discrete Fourier transform (cv::dft()) on a vector of length n/2. This means that to get the optimal size for a call to cv::dct(), you should compute it like this: size_t optimal_dct_size = 2 * getOptimalDFTSize( (N+1) / 2 ); This function (and discrete transforms in general) is covered in much greater detail in Chapter 6, General Image Transforms. In that section, we will discuss the details of how to pack and unpack the input and output, as well as information on when and why you might want to use the discrete cosine transform. cv::dft() void cv::dft(   cv::InputArray  src,                    // Input array   cv::OutputArray dst,                    // Result array    int             flags       = 0,        // for inverse transform or row-by-row, etc.   int             nonzeroRows = 0         // assume only this many entries are nonzero ); This function performs both the discrete Fourier transform as well as the inverse

input and output, as well as information on when and why you might want to use the discrete cosine transform. cv::dft() void cv::dft(   cv::InputArray  src,                    // Input array   cv::OutputArray dst,                    // Result array    int             flags       = 0,        // for inverse transform or row-by-row, etc.   int             nonzeroRows = 0         // assume only this many entries are nonzero ); This function performs both the discrete Fourier transform as well as the inverse transform (depending on the flags argument). The source array src must be either one- or two--dimensional. The result array dst will have the same type and size as src. The argument flags is a bit-field and can be set to one or more of DFT_INVERSE, DFT_ROWS, DFT_SCALE, DFT_COMPLEX_OUTPUT, or DFT_REAL_OUTPUT. If DFT_INVERSE is set, then the inverse transform is done. If the flag DFT_ROWS is set, then a two-dimensional  n-by- m input is treated as  n distinct one-dimensional vectors of length  m

and each such vector will be transformed independently. The flag DFT_SCALE normalizes the results by the number of elements in the array. This is normally done for DFT_INVERSE, as it guarantees that the inverse of the inverse will have the correct normalization. The flags DFT_COMPLEX_OUTPUT and DFT_REAL_OUTPUT are useful because when the Fourier transform of a real array is computed, the result will have a complex-conjugate symmetry. So, even though the result is complex, the number of array elements that result will be equal to the number of elements in the real input array rather than double that number. Such a packing is the default behavior of cv::dft(). To force the output to be in complex form, set the flag DFT_COMPLEX_OUTPUT. In the case of the inverse transform, the input is (in general) complex, and the output will be as well. However, if the input array (to the inverse transform) has complex-conjugate symmetry (for example, if it was itself the result of a Fourier transform of a real array), then

result will be equal to the number of elements in the real input array rather than double that number. Such a packing is the default behavior of cv::dft(). To force the output to be in complex form, set the flag DFT_COMPLEX_OUTPUT. In the case of the inverse transform, the input is (in general) complex, and the output will be as well. However, if the input array (to the inverse transform) has complex-conjugate symmetry (for example, if it was itself the result of a Fourier transform of a real array), then the inverse transform will be a real array. If you know this to be the case, and you would like the result array represented as a real array (and so use half the amount of memory), you can set the DFT_REAL_OUTPUT flag. (Note that if you do set this flag, cv::dft() does not check that the input array has the necessary symmetry, it simply assumes that it does.) The last parameter to cv::dft() is nonzeroRows. This defaults to 0, but if set to any nonzero value, will cause cv::dft() to assume that only the

the inverse transform will be a real array. If you know this to be the case, and you would like the result array represented as a real array (and so use half the amount of memory), you can set the DFT_REAL_OUTPUT flag. (Note that if you do set this flag, cv::dft() does not check that the input array has the necessary symmetry, it simply assumes that it does.) The last parameter to cv::dft() is nonzeroRows. This defaults to 0, but if set to any nonzero value, will cause cv::dft() to assume that only the first nonzeroRows of the input array are actually meaningful. (If DFT_INVERSE is set, then it is only the first nonzeroRows of the output array that are assumed to be nonzero.)  This flag is particularly handy when computing cross-correlations of convolutions using cv::dft(). Like the performance of cv::dct(), cv::dft() depends strongly on the exact size of the arrays passed to it, and this relationship is not monotonic. There are just some sizes that work better than others. It is recommended that when

first nonzeroRows of the input array are actually meaningful. (If DFT_INVERSE is set, then it is only the first nonzeroRows of the output array that are assumed to be nonzero.)  This flag is particularly handy when computing cross-correlations of convolutions using cv::dft(). Like the performance of cv::dct(), cv::dft() depends strongly on the exact size of the arrays passed to it, and this relationship is not monotonic. There are just some sizes that work better than others. It is recommended that when passing an array to cv::dft(), you should first determine the most optimal size larger than your array, and extend your array to that size. OpenCV provides a convenient routine to compute such values for you, called cv::getOptimalDFTSize(). Again, this function (and discrete transforms in general) is covered in much greater detail in Chapter 6, General Image Transforms. In that section, we will discuss the details of how to pack and unpack the input and output, as well as information on when and why you might

an array to cv::dft(), you should first determine the most optimal size larger than your array, and extend your array to that size. OpenCV provides a convenient routine to compute such values for you, called cv::getOptimalDFTSize(). Again, this function (and discrete transforms in general) is covered in much greater detail in Chapter 6, General Image Transforms. In that section, we will discuss the details of how to pack and unpack the input and output, as well as information on when and why you might want to use the discrete Fourier transform. cv::cvtColor() void cv::cvtColor(   cv::InputArray  src,                    // Input array   cv::OutputArray dst,                    // Result array    int             code,                   // color mapping code, see Table 3-23   int             dstCn = 0               // channels in output (0 for ‘automatic’) ); cv::cvtColor() is used to convert from one color space (number of channels) to another [Wharton71] while retaining the same data type. The input array src

to use the discrete Fourier transform. cv::cvtColor() void cv::cvtColor(   cv::InputArray  src,                    // Input array   cv::OutputArray dst,                    // Result array    int             code,                   // color mapping code, see Table 3-23   int             dstCn = 0               // channels in output (0 for ‘automatic’) ); cv::cvtColor() is used to convert from one color space (number of channels) to another [Wharton71] while retaining the same data type. The input array src can be an 8-bit array, a 16-bit unsigned array, or a 32-bit floating-point array. The output array dst will have the same size and depth as the input array. The conversion operation to be done is specified by the code argument, with possible values shown in Table 3-23.29  The final parameter dstCn is the desired number of channels in the destination image. If the default value (of 0) is given, then the number of channels is determined by the number of channels in src and the conversion code. Table 3-23:

can be an 8-bit array, a 16-bit unsigned array, or a 32-bit floating-point array. The output array dst will have the same size and depth as the input array. The conversion operation to be done is specified by the code argument, with possible values shown in Table 3-23.29  The final parameter dstCn is the desired number of channels in the destination image. If the default value (of 0) is given, then the number of channels is determined by the number of channels in src and the conversion code. Table 3-23: Conversions available by means of cv::cvtColor() Conversion code Meaning                                                              29 Long-time users of IPL should note that the function cvCvtColor() ignores the colorModel and channelSeq fields of the IplImage header. The conversions are done exactly as implied by the code argument.

cv::BGR2RGB cv::RGB2BGR cv::RGBA2BGRA cv::BGRA2RGBA Convert between RGB and BGR color spaces (with or without alpha channel) cv::RGB2RGBA cv::BGR2BGRA Add alpha channel to RGB or BGR image cv::RGBA2RGB cv::BGRA2BGR Remove alpha channel from RGB or BGR image cv::RGB2BGRA cv::RGBA2BGR cv::BGRA2RGB cv::BGR2RGBA Convert RGB to BGR color spaces while adding or removing alpha channel cv::RGB2GRAY cv::BGR2GRAY Convert RGB or BGR color spaces to grayscale cv::GRAY2RGB cv::GRAY2BGR cv::RGBA2GRAY cv::BGRA2GRAY Convert grayscale to RGB or BGR color spaces (optionally removing alpha channel in the process) cv::GRAY2RGBA cv::GRAY2BGRA Convert grayscale to RGB or BGR color spaces and add alpha channel cv::RGB2BGR565 cv::BGR2BGR565 cv::BGR5652RGB cv::BGR5652BGR cv::RGBA2BGR565 cv::BGRA2BGR565 cv::BGR5652RGBA cv::BGR5652BGRA Convert from RGB or BGR color space to BGR565 color representation with optional addition or removal of alpha channel (16-bit images) cv::GRAY2BGR565 cv::BGR5652GRAY Convert grayscale to BGR565 color

grayscale to RGB or BGR color spaces (optionally removing alpha channel in the process) cv::GRAY2RGBA cv::GRAY2BGRA Convert grayscale to RGB or BGR color spaces and add alpha channel cv::RGB2BGR565 cv::BGR2BGR565 cv::BGR5652RGB cv::BGR5652BGR cv::RGBA2BGR565 cv::BGRA2BGR565 cv::BGR5652RGBA cv::BGR5652BGRA Convert from RGB or BGR color space to BGR565 color representation with optional addition or removal of alpha channel (16-bit images) cv::GRAY2BGR565 cv::BGR5652GRAY Convert grayscale to BGR565 color representation or vice versa (16-bit images) cv::RGB2BGR555 cv::BGR2BGR555 cv::BGR5552RGB cv::BGR5552BGR cv::RGBA2BGR555 cv::BGRA2BGR555 cv::BGR5552RGBA cv::BGR5552BGRA Convert from RGB or BGR color space to BGR555 color representation with optional addition or removal of alpha channel (16-bit images) cv::GRAY2BGR555 cv::BGR5552GRAY Convert grayscale to BGR555 color representation or vice versa (16-bit images) cv::RGB2XYZ cv::BGR2XYZ cv::XYZ2RGB cv::XYZ2BGR Convert RGB or BGR image to CIE XYZ representation or

or vice versa (16-bit images) cv::RGB2BGR555 cv::BGR2BGR555 cv::BGR5552RGB cv::BGR5552BGR cv::RGBA2BGR555 cv::BGRA2BGR555 cv::BGR5552RGBA cv::BGR5552BGRA Convert from RGB or BGR color space to BGR555 color representation with optional addition or removal of alpha channel (16-bit images) cv::GRAY2BGR555 cv::BGR5552GRAY Convert grayscale to BGR555 color representation or vice versa (16-bit images) cv::RGB2XYZ cv::BGR2XYZ cv::XYZ2RGB cv::XYZ2BGR Convert RGB or BGR image to CIE XYZ representation or vice versa (Rec 709 with D65 white point) cv::RGB2YCrCb cv::BGR2YCrCb cv::YCrCb2RGB cv::YCrCb2BGR Convert RGB or BGR image to luma-chroma (aka YCC) color representation or vice versa cv::RGB2HSV cv::BGR2HSV cv::HSV2RGB cv::HSV2BGR Convert RGB or BGR image to HSV (hue saturation value) color representation or vice versa

cv::RGB2HLS cv::BGR2HLS cv::HLS2RGB cv::HLS2BGR Convert RGB or BGR image to HLS (hue lightness saturation) color representation or vice versa cv::RGB2Lab cv::BGR2Lab cv::Lab2RGB cv::Lab2BGR Convert RGB or BGR image to CIE Lab color representation or vice versa cv::RGB2Luv cv::BGR2Luv cv::Luv2RGB cv::Luv2BGR Convert RGB or BGR image to CIE Luv color representation or vice versa cv::BayerBG2RGB cv::BayerGB2RGB cv::BayerRG2RGB cv::BayerGR2RGB cv::BayerBG2BGR cv::BayerGB2BGR cv::BayerRG2BGR cv::BayerGR2BGR Convert from Bayer pattern (single-channel) to RGB or BGR image We will not go into the details of these conversions nor the subtleties of some of the representations (particularly the Bayer and the CIE color spaces) here. Instead, we will just note that OpenCV contains tools to convert to and from these various color spaces, which are of importance to various classes of users. The color-space conversions all use the conventions: 8-bit images are in the range 0 to 255, 16-bit images are in the range 0 to 65536,

from Bayer pattern (single-channel) to RGB or BGR image We will not go into the details of these conversions nor the subtleties of some of the representations (particularly the Bayer and the CIE color spaces) here. Instead, we will just note that OpenCV contains tools to convert to and from these various color spaces, which are of importance to various classes of users. The color-space conversions all use the conventions: 8-bit images are in the range 0 to 255, 16-bit images are in the range 0 to 65536, and floating-point numbers are in the range 0.0 to 1.0. When grayscale images are converted to color images, all components of the resulting image are taken to be equal; but for the reverse transformation (e.g., RGB or BGR to grayscale), the gray value is computed using the perceptually weighted formula:    Y=0.299()R+0.587()G+0.114()B

In the case of HSV or HLS representations, hue is normally represented as a value from 0 to 360.30 This can cause trouble in 8-bit representations and so, when converting to HSV, the hue is divided by 2 when the output image is an 8-bit image. cv::determinant() double cv::determinant(   cv::InputArray mat );    d=detmat() cv::determinant() computes the determinant of a square array. The array must be of one of the floating-point data types and must be single-channel. If the matrix is small, then the determinant is directly computed by the standard formula. For large matrices, this is not efficient and so the determinant is computed by Gaussian elimination. It is worth noting that if you know that a matrix has a symmetric and positive determinant, you can use the trick of solving via singular value decomposition (SVD). For more information, see the upcoming section on cv::SVD(), but the trick is to set                                                              30 Excluding 360, of course.

both U and V to NULL and then just take the products of the matrix W to obtain the determinant. cv::divide() void cv::divide(   cv::InputArray  src1,                  // First input array (numerators)   cv::InputArray  src2,                  // Second input array (denominators)   cv::OutputArray dst,                   // Results array (scale*src1/src2)   double          scale = 1.0,           // Multiplicative scale factor   int             dtype = -1             // Data type for dst, -1 to get from src2 )  void cv::divide(    double          scale,                 // Numerator for all divisions   cv::InputArray  src2,                  // Input array (denominators)    cv::OutputArray dst,                   // Results array (scale/src2)    int             dtype = -1             // Data type for dst, -1 to get from src2 )    dsti=saturatescale*†src1i/src2i()    dsti=saturatescale/src2i() cv::divide() is a simple division function; it divides all of the elements in src1 by the corresponding elements in src2 and

void cv::divide(    double          scale,                 // Numerator for all divisions   cv::InputArray  src2,                  // Input array (denominators)    cv::OutputArray dst,                   // Results array (scale/src2)    int             dtype = -1             // Data type for dst, -1 to get from src2 )    dsti=saturatescale*†src1i/src2i()    dsti=saturatescale/src2i() cv::divide() is a simple division function; it divides all of the elements in src1 by the corresponding elements in src2 and puts the results in dst.  cv::eigen() bool cv::eigen(   cv::InputArray  src,   cv::OutputArray eigenvalues,    int             lowindex     = -1,   int             highindex    = -1 );    bool cv::eigen(   cv::InputArray  src,   cv::OutputArray eigenvalues,   cv::OutputArray eigenvectors,    int             lowindex     = -1,   int             highindex    = -1 ); Given a symmetric matrix mat, cv::eigen() will compute the eigenvectors and eigenvalues of that matrix. The matrix must be of one of the

and puts the results in dst.  cv::eigen() bool cv::eigen(   cv::InputArray  src,   cv::OutputArray eigenvalues,    int             lowindex     = -1,   int             highindex    = -1 );    bool cv::eigen(   cv::InputArray  src,   cv::OutputArray eigenvalues,   cv::OutputArray eigenvectors,    int             lowindex     = -1,   int             highindex    = -1 ); Given a symmetric matrix mat, cv::eigen() will compute the eigenvectors and eigenvalues of that matrix. The matrix must be of one of the floating-point types. The results array eigenvalues will contain the eigenvalues of mat in descending order. If the array eigenvectors was provided, the eigenvectors will be stored as the rows of that array in the same order as the corresponding eigenvalues in eigenvalues. The additional parameters lowindex and highindex allow you to request only some of the eigenvalues to be computed (both must be used together). For example, if lowindex=0 and highindex=1, then only the largest two eigenvectors will be

floating-point types. The results array eigenvalues will contain the eigenvalues of mat in descending order. If the array eigenvectors was provided, the eigenvectors will be stored as the rows of that array in the same order as the corresponding eigenvalues in eigenvalues. The additional parameters lowindex and highindex allow you to request only some of the eigenvalues to be computed (both must be used together). For example, if lowindex=0 and highindex=1, then only the largest two eigenvectors will be computed. Regardless of the number of eigenvalues (and eigenvectors) requested, the result arrays will be of the same size (the requested values will always appear in the initial rows).

Similar to cv::det() (described previously), if the matrix in question is symmetric and positive definite,31 then it is better to use SVD to find the eigenvalues and eigenvectors of mat. cv::exp() void cv::exp(   cv::InputArray  src,   cv::OutputArray dst );     dsti=esrci cv::exp() exponentiates all of the elements in src1 and puts the results in dst. cv::extractImageCOI() bool cv::extractImageCOI(   cv::InputArray  mat,   cv::OutputArray dst,   int             coi = -1 );   The function cv::extractImageCOI() extracts the indicated COI from a legacy-style (pre-version 2.1) array, such as an IplImage or CvMat given by src and puts the result in dst. If the argument coi is provided, then that particular COI will be extracted. If not, then the COI field in src will be checked to determine which channel to extract. This method here specifically works with legacy arrays. If you need to extract a single channel from a modern cv::Mat object, use cv::mixChannels() or cv::split(). cv::flip() void cv::flip(

extracts the indicated COI from a legacy-style (pre-version 2.1) array, such as an IplImage or CvMat given by src and puts the result in dst. If the argument coi is provided, then that particular COI will be extracted. If not, then the COI field in src will be checked to determine which channel to extract. This method here specifically works with legacy arrays. If you need to extract a single channel from a modern cv::Mat object, use cv::mixChannels() or cv::split(). cv::flip() void cv::flip(   cv::InputArray  src,                    // Input array    cv::OutputArray dst,                    // Results array, size and type of ‘src’    int             flipCode = 0            // >0: y-flip, 0: x-flip, <0: both ); This function flips an image around the  x-axis, the  y-axis, or both. By default, flipCode is set to 0, which flips around the x-axis. If flipCode is set greater than zero (e.g., +1), the image will be flipped around the  y-axis, and if set to a negative value (e.g., –1), the image will be flipped

src,                    // Input array    cv::OutputArray dst,                    // Results array, size and type of ‘src’    int             flipCode = 0            // >0: y-flip, 0: x-flip, <0: both ); This function flips an image around the  x-axis, the  y-axis, or both. By default, flipCode is set to 0, which flips around the x-axis. If flipCode is set greater than zero (e.g., +1), the image will be flipped around the  y-axis, and if set to a negative value (e.g., –1), the image will be flipped about both axes. When doing video processing on Win32 systems, you will find yourself using this function often to switch between image formats with their origins at the upper-left and lower-left of the image. cv::gemm() void cv::gemm(   cv::InputArray  src1,                   // First input array   cv::InputArray  src2,                   // Second input array   double          alpha,                  // Weight for ‘src1’ * ‘src2’ product   cv::InputArray  src3,                   // Third (offset) input array

both axes. When doing video processing on Win32 systems, you will find yourself using this function often to switch between image formats with their origins at the upper-left and lower-left of the image. cv::gemm() void cv::gemm(   cv::InputArray  src1,                   // First input array   cv::InputArray  src2,                   // Second input array   double          alpha,                  // Weight for ‘src1’ * ‘src2’ product   cv::InputArray  src3,                   // Third (offset) input array   double          beta,                   // Weight for ‘src3’ array   cv::OutputArray dst,                    // Results array   int             flags = 0               // Use to transpose source arrays                                                              31 This is, for example, always the case for covariance matrices. See cvCalcCovarMatrixSee cv::calcCovarMatrixcvCalcCovarMatrix().

); Generalized matrix multiplication (GEMM) in OpenCV is performed by cv::gemm(), which performs matrix multiplication, multiplication by a transpose, scaled multiplication, and so on. In its most general form, cv::gemm() computes the following:      D=†α!opsrc1()*†opsrc2()+†β!opsrc3() where src1, src2, and src3 are matrices, α and β are numerical coefficients, and op() is an optional transposition of the matrix enclosed. The transpositions are controlled by the optional argument flags, which may be 0 or any combination (by means of Boolean OR) of cv::GEMM_1_T, cv::GEMM_2_T, and cv::GEMM_3_T (with each flag indicating a transposition of the corresponding matrix). All matrices must be of the appropriate size for the (matrix) multiplication, and all should be of floating-point types. The cv::gemm() function also supports two-channel matrices that will be treated as two components of a single complex number. A similar result can be achieved using the matrix algebra operators. For example:

which may be 0 or any combination (by means of Boolean OR) of cv::GEMM_1_T, cv::GEMM_2_T, and cv::GEMM_3_T (with each flag indicating a transposition of the corresponding matrix). All matrices must be of the appropriate size for the (matrix) multiplication, and all should be of floating-point types. The cv::gemm() function also supports two-channel matrices that will be treated as two components of a single complex number. A similar result can be achieved using the matrix algebra operators. For example:  cv::gemm(src1,src2,alpha,src3,bets,dst,cv::GEMM_1_T|cv::GEMM_3_T) would be equivalent to:  dst = alpha * src1.T() * src2 + beta * src3.T() cv::getConvertElem() and cv::getConvertScaleElem() cv::convertData cv::getConvertElem(       // Returns a conversion function (below)   int fromType,                           // Input pixel type (e.g., cv::U8)                           int toType                              // Output pixel type (e.g., cv::F32)   ); cv::convertScaleData cv::getConvertScaleElem(  //

cv::gemm(src1,src2,alpha,src3,bets,dst,cv::GEMM_1_T|cv::GEMM_3_T) would be equivalent to:  dst = alpha * src1.T() * src2 + beta * src3.T() cv::getConvertElem() and cv::getConvertScaleElem() cv::convertData cv::getConvertElem(       // Returns a conversion function (below)   int fromType,                           // Input pixel type (e.g., cv::U8)                           int toType                              // Output pixel type (e.g., cv::F32)   ); cv::convertScaleData cv::getConvertScaleElem(  // Returns a conversion function    int fromType,                           // Input pixel type (e.g., cv::U8)                           int toType                              // Output pixel type (e.g., cv::F32) );  // Conversion functions are of these forms: // typedef void (*ConvertData)(    const void* from,                       // Pointer to the input pixel location          void*       to,                         // Pointer to the result pixel location          int         cn                          //

a conversion function    int fromType,                           // Input pixel type (e.g., cv::U8)                           int toType                              // Output pixel type (e.g., cv::F32) );  // Conversion functions are of these forms: // typedef void (*ConvertData)(    const void* from,                       // Pointer to the input pixel location          void*       to,                         // Pointer to the result pixel location          int         cn                          // number of channels ); typedef void (*ConvertScaleData)(    const void* from,                       // Pointer to the input pixel location          void*       to,                         // Pointer to the result pixel location          int         cn,                         // number of channels   double      alpha,                      // scale factor    double      beta                        // offset factor ); The functions cv::getConvertElem() and cv::getConvertScaleElem() return function pointers to the

of channels ); typedef void (*ConvertScaleData)(    const void* from,                       // Pointer to the input pixel location          void*       to,                         // Pointer to the result pixel location          int         cn,                         // number of channels   double      alpha,                      // scale factor    double      beta                        // offset factor ); The functions cv::getConvertElem() and cv::getConvertScaleElem() return function pointers to the functions that are used for specific type conversions in OpenCV. The function returned by cv::getConvertElem() is defined (via typedef) to the type cv::ConvertData, which can be passed a pointer to two data areas and a number of “channels.” The number of channels is given by the argument cn of the conversion function, which is really the number of contiguous-in-memory objects of fromType to convert. This means that if you wanted, you could convert an entire (contiguous in memory) array by simply setting the

that are used for specific type conversions in OpenCV. The function returned by cv::getConvertElem() is defined (via typedef) to the type cv::ConvertData, which can be passed a pointer to two data areas and a number of “channels.” The number of channels is given by the argument cn of the conversion function, which is really the number of contiguous-in-memory objects of fromType to convert. This means that if you wanted, you could convert an entire (contiguous in memory) array by simply setting the number of channels equal to the total number of elements in the array.

Both cv::getConvertElem() and cv::getConvertScaleElem() take as arguments two types: fromType and toType. These types are specified using the integer constants (cv::F32, etc.). In the case of cv::getConvertScaleElem(), the returned function takes two additional arguments, alpha and beta. These values are used by the converter function to rescale (alpha) and offset (beta) the input value before conversion to the desired type. cv::idct() void cv::idct(   cv::InputArray  src,                    // Input array   cv::OutputArray dst,                    // Result array    int             flags,                  // for row-by-row ); cv::idct() is just a convenient shorthand for the inverse discrete cosine transform. A call to cv::idct() is exactly equivalent to a call to cv::dct() with the arguments: cv::dct( src, dst, flags | cv::DCT_INVERSE ); cv::idft() void cv::idft(   cv::InputArray  src,                    // Input array   cv::OutputArray dst,                    // Result array    int             flags       =

cv::OutputArray dst,                    // Result array    int             flags,                  // for row-by-row ); cv::idct() is just a convenient shorthand for the inverse discrete cosine transform. A call to cv::idct() is exactly equivalent to a call to cv::dct() with the arguments: cv::dct( src, dst, flags | cv::DCT_INVERSE ); cv::idft() void cv::idft(   cv::InputArray  src,                    // Input array   cv::OutputArray dst,                    // Result array    int             flags       = 0,        // for row-by-row, etc.   int             nonzeroRows = 0         // assume only this many entries are nonzero ); cv::idft() is just a convenient shorthand for the inverse discrete Fourier transform. A call to cv::idft() is exactly equivalent to a call to cv::dft() with the arguments: cv::dft( src, dst, flags | cv::DCT_INVERSE, outputRows ); It is noteworthy that neither cv::dft() nor cv::idft() scales the output by default. So you will probably want to call cv::idft() with the cv::DFT_SCALE

= 0,        // for row-by-row, etc.   int             nonzeroRows = 0         // assume only this many entries are nonzero ); cv::idft() is just a convenient shorthand for the inverse discrete Fourier transform. A call to cv::idft() is exactly equivalent to a call to cv::dft() with the arguments: cv::dft( src, dst, flags | cv::DCT_INVERSE, outputRows ); It is noteworthy that neither cv::dft() nor cv::idft() scales the output by default. So you will probably want to call cv::idft() with the cv::DFT_SCALE argument, that way, the  transform and its “inverse” will be true inverse operations. cv::inRange()  void cv::inRange(   cv::InputArray  src,                    // Input Array           cv::InputArray  upperb,                 // Array of upper bounds (inclusive)   cv::InputArray  lowerb,                 // Array of lower bounds (inclusive)   cv::OutputArray dst                     // Result array, cv::U8C1 type );       dsti=lowerbi≤srci≤upperbi

When applied to a one-dimensional array, each element of src is checked against the corresponding elements of upperb and lowerb. The corresponding element of dst is set to 255 if the element in src is between the values given by upperb and lowerb;; otherwise, it is set to 0. However, in the case of multichannel arrays for src, upperb, and lowerb, however, the output is still a single channel. The output value for element i will be set to 255 if and only if the values for the corresponding entry in src all lay inside of the intervals implied for the corresponding channel in upperb and lowerb. In this sense, upperb and lowerb define an  n-dimensional hypercube for each pixel and the corresponding value in dst is only set to true (255) if the pixel in src lies inside that hypercube.  cv::invert() double cv::invert(                        // Return 0 if ‘src’ is singular   cv::InputArray  src,                    // Input Array, m-by-n           cv::OutputArray dst                     // Result array, n-by-m

lay inside of the intervals implied for the corresponding channel in upperb and lowerb. In this sense, upperb and lowerb define an  n-dimensional hypercube for each pixel and the corresponding value in dst is only set to true (255) if the pixel in src lies inside that hypercube.  cv::invert() double cv::invert(                        // Return 0 if ‘src’ is singular   cv::InputArray  src,                    // Input Array, m-by-n           cv::OutputArray dst                     // Result array, n-by-m   int             method = cv::DECOMP_LU  // Method for computing (pseudo) inverse

); cv::invert() inverts the matrix in src and places the result in dst. The input array must be a floating-point type, and the result array will be of the same type. Because cv::invert() includes the possibility of computing pseudo-inverses, the input array need not be square. If the input array is  n-by- m, then the result array will be  m-by- n. This function supports several methods of computing the inverse matrix (see Table 3-24), but the default is Gaussian elimination. The return value depends on the method used. Table 3-24: Possible values of method argument to cv::invert() Value of method argument Meaning cv::DECOMP_LU                    Gaussian elimination (LU decomposition) cv::DECOMP_SVD                    Singular value decomposition (SVD) cv::DECOMP_CHOLESKY                    Only for symmetric positive matrices In the case of Gaussian elimination (cv::DECOMP_LU), the determinant of src is returned when the function is complete. If the determinant is 0, inversion failed and the array dst is set

used. Table 3-24: Possible values of method argument to cv::invert() Value of method argument Meaning cv::DECOMP_LU                    Gaussian elimination (LU decomposition) cv::DECOMP_SVD                    Singular value decomposition (SVD) cv::DECOMP_CHOLESKY                    Only for symmetric positive matrices In the case of Gaussian elimination (cv::DECOMP_LU), the determinant of src is returned when the function is complete. If the determinant is 0, inversion failed and the array dst is set to 0s. In the case of cv::DECOMP_SVD, the return value is the inverse condition number for the matrix (the ratio of the smallest to the largest eigenvalues). If the matrix src is singular, then cv::invert() in SVD mode will compute the pseudo-inverse. The other two methods (LU and Cholesky decomposition) require the source matrix to be square, nonsingular and positive. cv::log() void cv::log(   cv::InputArray  src,   cv::OutputArray dst );      dsti=logsrcisrci≠0−Celse⎧⎨⎪⎩⎪

cv::log() computes the natural log of the elements in src1 and puts the results in dst. Source pixels that are less than or equal to zero are marked with destination pixels set to a large negative value.  cv::LUT() void cv::LUT(   cv::InputArray  src,    cv::InputArray  lut,   cv::OutputArray dst );    dsti=lutsrci() The function cv::LUT() performs a “lookup table transform” on the input in src. cv::LUT() requires the source array src to be 8-bit index values. The lut array holds the lookup table. This lookup table array should have exactly 256 elements, and may have either a single channel or, in the case of a multichannel src array, the same number of channels as the source array. The function cv::LUT() then fills the destination array dst with values taken from the lookup table lut using the corresponding value from src as an index into that table. In the case where the values in src are signed 8-bit numbers, they are automatically offset by +128 so that their range will index the lookup table in a

lookup table array should have exactly 256 elements, and may have either a single channel or, in the case of a multichannel src array, the same number of channels as the source array. The function cv::LUT() then fills the destination array dst with values taken from the lookup table lut using the corresponding value from src as an index into that table. In the case where the values in src are signed 8-bit numbers, they are automatically offset by +128 so that their range will index the lookup table in a meaningful way. If the lookup table is multichannel (and the indices are as well), then the value in src is used as a multidimensional index into lut, and the result array dst will be single channel. If lut is one-dimensional, then the result array will be multichannel,

with each channel being separately computed from the corresponding index from src and the one-dimensional lookup table. cv::magnitude() void cv::magnitude(   cv::InputArray  x,    cv::InputArray  y,   cv::OutputArray dst );    dsti=xi2+yi2 cv::magnitude() essentially computes the radial part of a Cartesian-to-polar conversion on a two-dimensional vector field. In the case of cv::magnitude(), this vector field is expected to be in the form of two separate single channel arrays. These two input array should have the same size. (If you have a single two-channel array, cv::split() will give you separate channels.)  Each element in dst is computed from the corresponding elements of x and y as the Euclidian norm of the two (i.e., the square root of the sum of the squares of the corresponding values). cv::Mahalanobis() cv::Size cv::mahalanobis(     cv::InputArray  vec1,     cv::InputArray  vec2,     cv::OutputArray icovar ); cvMahalanobis() computes the value:     rmahalonobis=!x−!µ()TΣ−1!x−!µ()

The Mahalanobis distance is defined as the vector distance measured between a point and the center of a Gaussian distribution; it is computed using the inverse covariance of that distribution as a metric. See Figure 3-2. Intuitively, this is analogous to the z-score in basic statistics, where the distance from the center of a distribution is measured in units of the variance of that distribution. The Mahalanobis distance is just a multivariable generalization of the same idea.

Figure 3-2: A distribution of points in two dimensions with superimposed ellipsoids representing Mahalanobis distances of 1.0, 2.0, and 3.0 from the distribution’s mean The vector vec1 is presumed to be the point x, and the vector vec2 is taken to be the distribution’s mean.32 That matrix icovar is the inverse covariance. This covariance matrix will usually have been computed with cv::calcCovarMatrix() (described previously) and then inverted with cv::invert(). It is good programming practice to use the cv::DECOMP_SVD method for this inversion because someday you will encounter a distribution for which one of the eigenvalues is 0! cv::max() cv::MatExpr cv::max(  const cv::Mat& src1,                    // First input Array   const cv::Mat& src2                     // Second input array  ); MatExpr cv::max(                          // Return a matrix expression, not a matrix   const cv::Mat&  src1,                   // First input array (first position)                double          value                   //

method for this inversion because someday you will encounter a distribution for which one of the eigenvalues is 0! cv::max() cv::MatExpr cv::max(  const cv::Mat& src1,                    // First input Array   const cv::Mat& src2                     // Second input array  ); MatExpr cv::max(                          // Return a matrix expression, not a matrix   const cv::Mat&  src1,                   // First input array (first position)                double          value                   // Scalar in second position ); MatExpr cv::max(                          // Return a matrix expression, not a matrix   double          value,                  // Scalar in first position   const cv::Mat&  src1                    // Input Array (second position) );  void cv::max(    cv::InputArray  src1,                   // First input Array    cv::InputArray  src2,                   // Second input Array    cv::OutputArray dst                     // Result array ); void cv::max(    const Mat&      src1,

position ); MatExpr cv::max(                          // Return a matrix expression, not a matrix   double          value,                  // Scalar in first position   const cv::Mat&  src1                    // Input Array (second position) );  void cv::max(    cv::InputArray  src1,                   // First input Array    cv::InputArray  src2,                   // Second input Array    cv::OutputArray dst                     // Result array ); void cv::max(    const Mat&      src1,                   // First input Array   const Mat&      src2,                   // Second input Array   Mat&            dst                     // Result array ); void cv::max(    const Mat&      src1,                   // Input Array   double          value,                  // Scalar input   Mat&            dst                     // Result array );    dsti=max†src1,i,†src2,i() cv::max() computes the maximum value of each corresponding pair of pixels in the arrays src1 and src2. It has two basic forms: those that return a

First input Array   const Mat&      src2,                   // Second input Array   Mat&            dst                     // Result array ); void cv::max(    const Mat&      src1,                   // Input Array   double          value,                  // Scalar input   Mat&            dst                     // Result array );    dsti=max†src1,i,†src2,i() cv::max() computes the maximum value of each corresponding pair of pixels in the arrays src1 and src2. It has two basic forms: those that return a matrix expression and those that compute a result and put it someplace you have indicated. In the three-argument form, in the case where one of the operands is a cv::Scalar, comparison with a multichannel array is done on a per-channel basis with the appropriate component of the cv::Scalar. cv::mean() cv::Scalar cv::mean(   cv::InputArray  src,                                                              32 Actually, the Mahalanobis distance is more generally defined as the distance between any two vectors;

expression and those that compute a result and put it someplace you have indicated. In the three-argument form, in the case where one of the operands is a cv::Scalar, comparison with a multichannel array is done on a per-channel basis with the appropriate component of the cv::Scalar. cv::mean() cv::Scalar cv::mean(   cv::InputArray  src,                                                              32 Actually, the Mahalanobis distance is more generally defined as the distance between any two vectors; in any case, the vector vec2 is subtracted from the vector vec1. Neither is there any fundamental connection between mat in cvMahalanobiscv::Mahalanobis() and the inverse covariance; any metric can be imposed here as appropriate.

cv::InputArray  mask = cv::noArray(),   // Optional mask, compute only where nonzero );    N=†i,†maski≠0∑1
    meanc=†1Ni,†maski≠0∑srci
 The function cv::mean() computes average value of all of the pixels in the input array src that are not masked out. The result is computed on a per-channel basis if src is multichannel. cv::meanStdDev() void cv::meanStdDev(   cv::InputArray  src,   cv::OutputArray mean,   cv::OutputArray stddev,   cv::InputArray  mask = cv::noArray(),   // Optional mask, compute only where nonzero   );    N=†i,†maski≠0∑1

stddevc=†i,†mask≠0∑srcc,i−meanc()2 The function cv::meanStdDev() computes the average value of the pixels in the input array src not masked out, as well as their standard deviation. The mean and standard deviation are computed on a per-channel basis if src is multichannel. It is important to know that the standard deviation computed here is not the same as the covariance matrix. In fact, the standard deviation computed here is only the diagonal elements of the full covariance matrix. If you want to compute the full covariance matrix, you will have to use cv::calcCovarMatrix(). cv::merge() void cv::merge(    const cv::Mat*  mv,                   // C-style array of arrays   size_t          count,                // Number of arrays pointed to by ‘mv’   cv::OutputArray dst                   // Result array, contains all channels in ‘mv’  ); void merge(   const vector<cv::Mat>& mv,            // STL-style array of arrays   cv::OutputArray dst                   // Result array, contains all channels in ‘mv’

the full covariance matrix, you will have to use cv::calcCovarMatrix(). cv::merge() void cv::merge(    const cv::Mat*  mv,                   // C-style array of arrays   size_t          count,                // Number of arrays pointed to by ‘mv’   cv::OutputArray dst                   // Result array, contains all channels in ‘mv’  ); void merge(   const vector<cv::Mat>& mv,            // STL-style array of arrays   cv::OutputArray dst                   // Result array, contains all channels in ‘mv’  ); cv::merge() is the inverse operation of cv::split(). The arrays contained in mv are combined into the output array dst. In the case in which mv is a pointer to a C-style array of cv::Mat objects, the additional size parameter count must also be supplied.

cv::min()  cv::MatExpr cv::min(                      // Return a matrix expression, not a matrix   const cv::Mat&  src1,                   // First input array                    const cv::Mat&  src2                    // Second input array  ); MatExpr cv::min(                          // Return a matrix expression, not a matrix   const cv::Mat&  src1,                   // First input array (first position)                double          value                   // Scalar in second position ); MatExpr cv::min(                          // Return a matrix expression, not a matrix   double          value,                  // Scalar in first position   const cv::Mat&  src1                    // Input Array (second position) );  void cv::min(    cv::InputArray  src1,                   // First input Array    cv::InputArray  src2,                   // Second input Array    cv::OutputArray dst                     // Result array ); void cv::min(    const Mat&      src1,                   // First input Array   const

// Return a matrix expression, not a matrix   double          value,                  // Scalar in first position   const cv::Mat&  src1                    // Input Array (second position) );  void cv::min(    cv::InputArray  src1,                   // First input Array    cv::InputArray  src2,                   // Second input Array    cv::OutputArray dst                     // Result array ); void cv::min(    const Mat&      src1,                   // First input Array   const Mat&      src2,                   // Second input Array   Mat&            dst                     // Result array ); void cv::min(    const Mat&      src1,                   // Input Array   double          value,                  // Scalar input   Mat&            dst                     // Result array );    dsti=min†src1,i,†src2,i() cv::min() computes the minimum value of each corresponding pair of pixels in the arrays src1 and src2 (or one source matrix and a single value). Note that the variants of

Mat&      src2,                   // Second input Array   Mat&            dst                     // Result array ); void cv::min(    const Mat&      src1,                   // Input Array   double          value,                  // Scalar input   Mat&            dst                     // Result array );    dsti=min†src1,i,†src2,i() cv::min() computes the minimum value of each corresponding pair of pixels in the arrays src1 and src2 (or one source matrix and a single value). Note that the variants of cv::min() that return a value or return a matrix expression that can then be manipulated by OpenCV’s matrix expression machinery.  In the three-argument form, in the case where one of the operands is a cv::Scalar, comparison with a multichannel array is done on a per-channel basis with the appropriate component of the cv::Scalar. cv::minMaxIdx() void cv::minMaxLoc(   cv::InputArray src,                     // Input array, 1-dimensional only   double*        minVal,                  // min value goes here (in

that return a value or return a matrix expression that can then be manipulated by OpenCV’s matrix expression machinery.  In the three-argument form, in the case where one of the operands is a cv::Scalar, comparison with a multichannel array is done on a per-channel basis with the appropriate component of the cv::Scalar. cv::minMaxIdx() void cv::minMaxLoc(   cv::InputArray src,                     // Input array, 1-dimensional only   double*        minVal,                  // min value goes here (in not NULL)    double*        maxVal,                  // min value goes here (in not NULL)    cv::Point*     minLoc,                  // location of min goes here (if not NULL)   cv::Point*     maxLoc,                  // location of max goes here (if not NULL)   cv::InputArray mask = cv::noArray()     // search only non-zero values (if not NULL)  void cv::minMaxLoc(   const SparseMat& src,                   // Input sparse array   double*        minVal,                  // min value goes here (in not NULL)

NULL)    double*        maxVal,                  // min value goes here (in not NULL)    cv::Point*     minLoc,                  // location of min goes here (if not NULL)   cv::Point*     maxLoc,                  // location of max goes here (if not NULL)   cv::InputArray mask = cv::noArray()     // search only non-zero values (if not NULL)  void cv::minMaxLoc(   const SparseMat& src,                   // Input sparse array   double*        minVal,                  // min value goes here (in not NULL)    double*        maxVal,                  // min value goes here (in not NULL)    cv::Point*     minIdx,                  // C-style array, indices of min location   cv::Point*     maxIdx,                  // C-style array, indices of min location ); This routine finds the minimal and maximal values in the array src and (optionally) returns their locations. The computed minimum and maximum values are placed in minVal and maxVal. Optionally, the locations of those extrema can also be returned, but only if the

double*        maxVal,                  // min value goes here (in not NULL)    cv::Point*     minIdx,                  // C-style array, indices of min location   cv::Point*     maxIdx,                  // C-style array, indices of min location ); This routine finds the minimal and maximal values in the array src and (optionally) returns their locations. The computed minimum and maximum values are placed in minVal and maxVal. Optionally, the locations of those extrema can also be returned, but only if the src array is two-dimensional. These locations will be written to the addresses given by minLoc and maxLoc (provided that these arguments

are non-NULL). Because these locations are of type cv::Point, this form of the function should only be used on two-dimensional arrays (i.e., matrices or images).  cv::minMaxLoc() void cv::minMaxLoc(   cv::InputArray src,                     // Input array   double*        minVal,                  // min value goes here (in not NULL)    double*        maxVal,                  // min value goes here (in not NULL)    cv::Point*     minLoc,                  // location of min goes here (if not NULL)   cv::Point*     maxLoc,                  // location of max goes here (if not NULL)   cv::InputArray mask = cv::noArray()     // search only non-zero values (if present)  void cv::minMaxLoc(   const SparseMat& src,                   // Input sparse array   double*        minVal,                  // min value goes here (in not NULL)    double*        maxVal,                  // min value goes here (in not NULL)    cv::Point*     minIdx,                  // C-style array, indices of min location   cv::Point*

maxLoc,                  // location of max goes here (if not NULL)   cv::InputArray mask = cv::noArray()     // search only non-zero values (if present)  void cv::minMaxLoc(   const SparseMat& src,                   // Input sparse array   double*        minVal,                  // min value goes here (in not NULL)    double*        maxVal,                  // min value goes here (in not NULL)    cv::Point*     minIdx,                  // C-style array, indices of min location   cv::Point*     maxIdx,                  // C-style array, indices of min location ); This routine finds the minimal and maximal values in the array src and (optionally) returns their locations. The computed minimum and maximum values are placed in minVal and maxVal. Optionally, the locations of those extrema can also be returned, but only if the src array is two-dimensional. These locations will be written to the addresses given by minLoc and maxLoc (provided that these arguments are non-NULL). Because these locations are of

// C-style array, indices of min location ); This routine finds the minimal and maximal values in the array src and (optionally) returns their locations. The computed minimum and maximum values are placed in minVal and maxVal. Optionally, the locations of those extrema can also be returned, but only if the src array is two-dimensional. These locations will be written to the addresses given by minLoc and maxLoc (provided that these arguments are non-NULL). Because these locations are of type cv::Point, this form of the function should only be used on two-dimensional arrays (i.e., matrices or images). cv::minMaxLoc() can also be called with a cv::SparseMat for the src array. In this case, the array can be of any number of dimensions and the minimum and maximum will be computed and their location returned. In this case, the locations of the extrema will be returned and placed in the C-style arrays minIdx and maxIdx. Both of those arrays, if provided, should have the same number of elements as

cv::Point, this form of the function should only be used on two-dimensional arrays (i.e., matrices or images). cv::minMaxLoc() can also be called with a cv::SparseMat for the src array. In this case, the array can be of any number of dimensions and the minimum and maximum will be computed and their location returned. In this case, the locations of the extrema will be returned and placed in the C-style arrays minIdx and maxIdx. Both of those arrays, if provided, should have the same number of elements as the number of dimensions in the src array. In the case of cv::SparseMat, the minimum and maximum are only computed for what are generally referred to as nonzero elements in the source code. However, it is important to note that this terminology is slightly misleading, as what is really meant is: elements that exist in the sparse matrix representation in memory. In fact, there may, as a result of how the sparse matrix came into being and what has been done with it in the past, be elements that exist and are

the number of dimensions in the src array. In the case of cv::SparseMat, the minimum and maximum are only computed for what are generally referred to as nonzero elements in the source code. However, it is important to note that this terminology is slightly misleading, as what is really meant is: elements that exist in the sparse matrix representation in memory. In fact, there may, as a result of how the sparse matrix came into being and what has been done with it in the past, be elements that exist and are also zero. Such elements will be included in the computation of the minimum and maximum. When working with multichannel arrays, there are several options. Natively, cv::minMaxLoc() does not support multichannel input. Primarily this is because this operation is ambiguous.  If you want the minimum and maximum across all channels, you can use cv::reshape() to reshape the multichannel array into one giant single-channel array. If you would like the minimum and maximum for each channel separately, you can use

also zero. Such elements will be included in the computation of the minimum and maximum. When working with multichannel arrays, there are several options. Natively, cv::minMaxLoc() does not support multichannel input. Primarily this is because this operation is ambiguous.  If you want the minimum and maximum across all channels, you can use cv::reshape() to reshape the multichannel array into one giant single-channel array. If you would like the minimum and maximum for each channel separately, you can use cv::split() or cv::mixChannels() to separate the channels out and analyze them separately. In both forms of cv::minMaxLoc, the arguments for the minimum or maximum value or location may be set to NULL, which turns off the computation for that argument. cv::mixChannels() void cv::mixChannels(    const cv::Mat*         srcv,            // C-style array of matrices   int                    nsrc,            // Number of elements in ‘srcv’   cv::Mat*               dstv,            // C-style array of target

use cv::split() or cv::mixChannels() to separate the channels out and analyze them separately. In both forms of cv::minMaxLoc, the arguments for the minimum or maximum value or location may be set to NULL, which turns off the computation for that argument. cv::mixChannels() void cv::mixChannels(    const cv::Mat*         srcv,            // C-style array of matrices   int                    nsrc,            // Number of elements in ‘srcv’   cv::Mat*               dstv,            // C-style array of target matrices   int                    ndst,            // Number of elements in ‘dstv’   const int*             fromTo,          // C-style array of pairs, …from,to…   size_t                 n_pairs          // Number of pairs in ‘fromTo’

);  void cv::mixChannels(    const vector<cv::Mat>& srcv,            // STL-style vector of matrices   vector<cv::Mat>&       dstv,            // STL-style vector of target matrices   const int*             fromTo,          // C-style array of pairs, …from,to…   size_t                 n_pairs          // Number of pairs in ‘fromTo’ );   There are many operations in OpenCV that are special cases of the general problem of rearranging channels from one or more images in the input, and sorting them into particular channels in one or more images in the output. Functions like cv::split(), cv::merge(), and (at least some cases of) cv::cvtColor() all make use of such functionality. Those methods do what they need to do by calling the much more general cv::mixChannels(). This function allows you to supply multiple arrays, each with potentially multiple channels, for the input, and the same for the output, and to map the channels from the input arrays into the channels in the output arrays in any manner you choose. The

channels in one or more images in the output. Functions like cv::split(), cv::merge(), and (at least some cases of) cv::cvtColor() all make use of such functionality. Those methods do what they need to do by calling the much more general cv::mixChannels(). This function allows you to supply multiple arrays, each with potentially multiple channels, for the input, and the same for the output, and to map the channels from the input arrays into the channels in the output arrays in any manner you choose. The input and output arrays can either be specified as C-style arrays of cv::Mat objects with an accompanying integer indicating the number of cv::Mats, or as an STL vector<> of cv::Mat objects. Output arrays must be pre-allocated with their size and number of dimensions matching those of the input arrays.  Figure 3-3: A single four-channel RGBA image is converted to one BGR and one Alpha-only image. The mapping is controlled by the C-style integer array fromTo. This array can contain any number of integer pairs

and output arrays can either be specified as C-style arrays of cv::Mat objects with an accompanying integer indicating the number of cv::Mats, or as an STL vector<> of cv::Mat objects. Output arrays must be pre-allocated with their size and number of dimensions matching those of the input arrays.  Figure 3-3: A single four-channel RGBA image is converted to one BGR and one Alpha-only image. The mapping is controlled by the C-style integer array fromTo. This array can contain any number of integer pairs in sequence, with each pair indicating with its first value the source channel and with its second value the destination channel to which that should be copied. The channels are sequentially numbered starting at zero for the first image, then through the second image, and so on. (Figure 3-3). The total number of pairs is supplied by the argument n_pairs. Unlike most other functions in the post-version 2.1 library, cv::mixChannels() does not allocate the output arrays. They must be pre-allocated and have the

sequence, with each pair indicating with its first value the source channel and with its second value the destination channel to which that should be copied. The channels are sequentially numbered starting at zero for the first image, then through the second image, and so on. (Figure 3-3). The total number of pairs is supplied by the argument n_pairs. Unlike most other functions in the post-version 2.1 library, cv::mixChannels() does not allocate the output arrays. They must be pre-allocated and have the same size and dimensionality as the input arrays.

cv::mulSpectrums() doublevoid cv::mulSpectrums(                   cv::InputArray  arr1,                 // First input array     cv::InputArray  arr2,                 // Second input array, same size as ‘arr1’     cv::OutputArray dst,                  // Result array, same size as ‘arr1’     int             flags,                // used to indicate that rows are independent     bool            conj  = false         // If true, conjugate arr2 first  );   In many operations involving spectra (i.e., the results from cv::dft() or cv::idft()), one wishes to do a per-element multiplication that respects the packing of the spectra (real arrays), or their nature as complex variables. (See the description of cv::dft() for more details.)  The input arrays may be one- or two-dimensional, with the second the same size and type as the first. If the input array is two-dimensional, it may either be taken to be a true two-dimensional spectrum, or an array of one-dimensional spectra (one per row). In the latter case, flags

from cv::dft() or cv::idft()), one wishes to do a per-element multiplication that respects the packing of the spectra (real arrays), or their nature as complex variables. (See the description of cv::dft() for more details.)  The input arrays may be one- or two-dimensional, with the second the same size and type as the first. If the input array is two-dimensional, it may either be taken to be a true two-dimensional spectrum, or an array of one-dimensional spectra (one per row). In the latter case, flags should be set to cv::DFT_ROWS; otherwise it can be set to 0. When the two arrays are complex, they are simply multiplied on an element-wise basis, but cv::mulSpectrums() provides an option to conjugate the second array elements before multiplication. For example, you would use this option to perform correlation (using the Fourier transform), but for convolution, you would use conj=false.  cv::multiply() void cv::multiply(   cv::InputArray  src1,                   // First input array    cv::InputArray  src2,

be set to cv::DFT_ROWS; otherwise it can be set to 0. When the two arrays are complex, they are simply multiplied on an element-wise basis, but cv::mulSpectrums() provides an option to conjugate the second array elements before multiplication. For example, you would use this option to perform correlation (using the Fourier transform), but for convolution, you would use conj=false.  cv::multiply() void cv::multiply(   cv::InputArray  src1,                   // First input array    cv::InputArray  src2,                   // Second input array   cv::OutputArray dst,                    // Result array    double          scale = 1.0,            // overall scale factor   int             dtype = -1              // Output type for result array  );    dsti=saturatescale*†src1i*src2i() cv::multiply() is a simple multiplication function; it multiplies the elements in src1 by the corresponding elements in src2 and puts the results in dst.  cv::mulTransposed() void cv::mulTransposed(   cv::InputArray  src1,

// Second input array   cv::OutputArray dst,                    // Result array    double          scale = 1.0,            // overall scale factor   int             dtype = -1              // Output type for result array  );    dsti=saturatescale*†src1i*src2i() cv::multiply() is a simple multiplication function; it multiplies the elements in src1 by the corresponding elements in src2 and puts the results in dst.  cv::mulTransposed() void cv::mulTransposed(   cv::InputArray  src1,                   // Input matrix    cv::OutputArray dst,                    // Result array    bool            aTa,                    // If true, transpose then multiply   cv::InputArray  delta = cv::noArray(),  // subtract from ‘src1’ before multiplication   double          scale = 1.0,            // overall scale factor    int             dtype = -1              // Output type for result array   )    dst=scale*src−delta()Tsrc−delta()aTa=truescale*src−delta()src−delta()TaTa=false⎧⎨⎪⎩⎪

cv::mulTransposed() is used to compute the product of a matrix and its own transpose—useful, for example, in computing covariance. The matrix src should be two-dimensional and single-channel, but unlike cv::GEMM(), it is not restricted to the floating-point types. The result matrix will be the same type as the source matrix unless specified otherwise by dtype. If dtype is not negative (default), it should be either cv::F32 or cv::F64; the output array dst will then be of the indicated type. If a second input matrix delta is provided, that matrix will be subtracted from src before the multiplication. If no matrix is provided (i.e., delta=cv::noArray()), then no subtraction is done. The

array delta need not be the same size as src; if delta is smaller than src, delta is repeated (also called tiling, see cv::repeat()) in order to produce an array whose size matches the size of src. The argument scale is applied to the matrix after the multiplication is done. Finally, the argument aTa is used to select either the multiplication in which the transposed version of src is multiplied from the left (aTa=true) or from the right (aTa=false). cv::norm() double cv::norm(                          // Return computed norm in double precision   cv::InputArray src1,                    // Input matrix    int            normType = cv::NORM_L2,  // Type of norm to compute   cv::InputArray mask     = cv::noArray() // include only non-zero values (if present) ); double cv::norm(                          // Return computed norm of difference         cv::InputArray src1,                    // Input matrix   cv::InputArray src2,                    // Second input matrix   int            normType = cv::NORM_L2,  //

computed norm in double precision   cv::InputArray src1,                    // Input matrix    int            normType = cv::NORM_L2,  // Type of norm to compute   cv::InputArray mask     = cv::noArray() // include only non-zero values (if present) ); double cv::norm(                          // Return computed norm of difference         cv::InputArray src1,                    // Input matrix   cv::InputArray src2,                    // Second input matrix   int            normType = cv::NORM_L2,  // Type of norm to compute   cv::InputArray mask     = cv::noArray() // include only non-zero values (if present) ); double cv::norm(   const cv::SparseMat& src,               // Input sparse matrix    int                  normType = cv::NORM_L2, // Type of norm to compute );    src1∞,L1,L2

This function is used to compute the norm of an array (see Table 3-25) or a variety of distance norms (Table 3-26) between two arrays if two arrays are provided (see Table 3-26). The norm of a cv::SparseMat can also be computed, in which case zero-entries are ignored in the computation of the norm.  Table 3-25: Norm computed by cv:norm() for different values of normType when arr2=NULL normType Result  cv::NORM_INF                       src1∞=maxiabssrc1i()  cv::NORM_L1                       src1L1=i∑abssrc1i()  cv::NORM_L2                       src1L2=i∑src1i2i∑src1i2 If the second array argument src2 is non-NULL, then the norm computed is a difference norm—that is, something like the distance between the two arrays.33 In the first three cases (shown in Table 3-26) the norm is absolute; in the latter three cases it is rescaled by the magnitude of the second array src2. Table 3-26: Norm computed by cv::norm() for different values of normType when arr2 is non-NULL normType Result cv::NORM_INF

cv::NORM_L2                       src1L2=i∑src1i2i∑src1i2 If the second array argument src2 is non-NULL, then the norm computed is a difference norm—that is, something like the distance between the two arrays.33 In the first three cases (shown in Table 3-26) the norm is absolute; in the latter three cases it is rescaled by the magnitude of the second array src2. Table 3-26: Norm computed by cv::norm() for different values of normType when arr2 is non-NULL normType Result cv::NORM_INF                       src1−src2∞=maxiabssrc1i−src2i()                                                              33 At least in the case of the L2 norm, there is an intuitive interpretation of the difference norm as a Euclidean distance in a space of dimension equal to the number of pixels in the images.

cv::NORM_L1                       src1−src2L1=i∑abssrc1i−src2i() cv::NORM_L2                       src1−src2L2=i∑src1i−src2i()2 cv::NORM_RELATIVE_INF     src1−src2∞src2∞
 cv::NORM_RELATIVE_L1     src1−src2L1src2L1
 cv::NORM_RELATIVE_L2     src1−src2L2src2L2

In all cases, src1 and src2 must have the same size and number of channels. When there is more than one channel, the norm is computed over all of the channels together (i.e., the sums in Table  and Table 3-26 are not only over x and y but also over the channels). cv::normalize() void cv::normalize(   cv::InputArray  src1,                   // Input matrix    cv::OutputArray dst,                    // Result matrix   double          alpha    = 1,           // first parameter (see Table 3-27)   double          beta     = 0,           // second parameter (see Table 3-27)   int             normType = cv::NORM_L2, // Type of norm to compute   int             dtype = -1              // Output type for result array    cv::InputArray  mask     = cv::noArray()// include only non-zero values (if present) ); void cv::normalize(   const cv::SparseMat& src,               // Input sparse matrix   cv::SparseMat&       dst,               // Result sparse matrix   double               alpha    = 1,      // first parameter

beta     = 0,           // second parameter (see Table 3-27)   int             normType = cv::NORM_L2, // Type of norm to compute   int             dtype = -1              // Output type for result array    cv::InputArray  mask     = cv::noArray()// include only non-zero values (if present) ); void cv::normalize(   const cv::SparseMat& src,               // Input sparse matrix   cv::SparseMat&       dst,               // Result sparse matrix   double               alpha    = 1,      // first parameter (see Table 3-27)    int                  normType = cv::NORM_L2, // Type of norm to compute );    dst∞,L1,L2=α

mindst()=α,††††maxdst()=β
 As with so many OpenCV functions, cv::normalize() does more than it might at first appear. Depending on the value of normType, image src is normalized or otherwise mapped into a particular range in dst. The array dst will be the same size as src, and will have the same data type, unless the rtype argument is used. Optionally, rtype can be set to one of the OpenCV fundamental types (cv::F32, etc.) and the output array will be of that type. The exact meaning of this operation is dependent on the normType argument. The possible values of normType are shown in Table 3-27.  Table 3-27: Possible values of normType argument to cv::normalize() norm_type Result cv::NORM_INF                       dst∞=maxiabsdsti()=α

cv::NORM_L1                       dstL1=i∑absdsti()=α
 cv::NORM_L2                       dstL2=i∑dsti2=i∑dsti2=α

cv::NORM_MINMAX                    Map into range [α, β] In the case of the infinity norm, the array src is rescaled such that the magnitude of the absolute value of the largest entry is equal to alpha. In the case of the L1 or L2 norm, the array is rescaled so that the norm equals the value of alpha. If normType is set to cv::MINMAX, then the values of the array are rescaled and translated so that they are linearly mapped into the interval between alpha and beta (inclusive). As before, if mask is non-NULL then only those pixels corresponding to nonzero values of the mask image will contribute to the computation of the norm—and only those pixels will be altered by cv::normalize(). Note that if the operation dtype=cv::MINMAX is used, the source array may not be cv::SparseMat. The reason for this is that the cv::MIN_MAX operation can apply an overall offset, and this would affect the sparsity of the array (specifically, a sparse array would become non-sparse as all of the zero elements became non-zero as a

then only those pixels corresponding to nonzero values of the mask image will contribute to the computation of the norm—and only those pixels will be altered by cv::normalize(). Note that if the operation dtype=cv::MINMAX is used, the source array may not be cv::SparseMat. The reason for this is that the cv::MIN_MAX operation can apply an overall offset, and this would affect the sparsity of the array (specifically, a sparse array would become non-sparse as all of the zero elements became non-zero as a result of this operation). cv::perspectiveTransform() void cv::perspectiveTransform(   cv::InputArray  src,                    // Input array, 2 or 3 channels   cv::OutputArray dst,                    // Result array, size, type, as src1   cv::InputArray  mtx                     // 3-by-3 or 4-by-4 transoform matrix );    xyz⎡⎣⎢⎢⎢⎤⎦⎥⎥⎥→′x/w'′y/w'′z/w'⎡⎣⎢⎢⎢⎤⎦⎥⎥⎥

The cv::perspectiveTransform() function performs a plane-plane projective transform of a list of points (not pixels). The input array should be a two- or three-channel array, and the matrix mtx should be 3-by-3 or 4-by-4, respectively, in the two cases. cv::perspectiveTransform() thus transforms each element of src by first regarding it as a vector of length src.channels() + 1, with the additional dimension (the projective dimension) set initially to 1.0. This is also known as homogeneous coordinates. Each extended vector is then multiplied by mtx and the result is rescaled by the value of the (new) projective coordinate34 (which is then thrown away as it is always 1.0 after this operation). Note again that this routine is for transforming a list of points, not an image as such. If you want to apply a perspective transform to an image, you are actually asking not to                                                              34 Technically, it is possible that after multiplying by mtx, the value of w’ will

extended vector is then multiplied by mtx and the result is rescaled by the value of the (new) projective coordinate34 (which is then thrown away as it is always 1.0 after this operation). Note again that this routine is for transforming a list of points, not an image as such. If you want to apply a perspective transform to an image, you are actually asking not to                                                              34 Technically, it is possible that after multiplying by mtx, the value of w’ will be zero, corresponding to points projected to infinity. In this case, rather than dividing by zero, a value of 0 is assigned to the ratio.

transform the individual pixels, but rather to move them from one place in the image to another. This is the job of cv::warpPerspective().  If you want to solve the inverse problem to find the most probable perspective transformation given many pairs of corresponding points, use cv::getPerspectiveTransform() or cv::findHomography(). cv::phase() void cv::phase(   cv::InputArray  x,                      // Input array of x-components   cv::InputArray  y,                      // Input array of y-components   cv::OutputArray dst,                    // Output array of angles (radians) );    dsti=atan2yi,†xi()†

cv::phase() computes the azimuthal (angle) part of a Cartesian-to-polar conversion on a two-dimensional vector field. This vector field is expected to be in the form of two separate single--channel arrays. These two input arrays should, of course, be of the same size. (If you happen to have a single two-- channel array, a quick call to cv::split() will do just what you need.)  Each element in dst is then computed from the corresponding elements of x and y as the arctangent of the ratio of the two. cv::polarToCart() void cv::polarToCart(   cv::InputArray  magnitude,              // Input array of magnitudes   cv::InputArray  angle,                  // Input array of angles   cv::OutputArray x,                      // Output array of x-components   cv::OutputArray y,                      // Output array of y-components   bool            angleInDegrees = false  // degrees (if true) radians (if false) );    xi=magnitudei*cosanglei()    yi=magnitudei*sinanglei() cv::polarToCart() computes a vector field in

void cv::polarToCart(   cv::InputArray  magnitude,              // Input array of magnitudes   cv::InputArray  angle,                  // Input array of angles   cv::OutputArray x,                      // Output array of x-components   cv::OutputArray y,                      // Output array of y-components   bool            angleInDegrees = false  // degrees (if true) radians (if false) );    xi=magnitudei*cosanglei()    yi=magnitudei*sinanglei() cv::polarToCart() computes a vector field in Cartesian   x,†y() coordinates from polar coordinates. The input is in two arrays, magnitude and angle, of the same size and type, specifying the magnitude and angle of the field at every point. The output is similarly two arrays that will be of the same size and type as the inputs, and which will contain the x and y projections of the vector at each point. The additional flag angleInDegrees will cause the angle array to be interpreted as angles in degrees, rather than in radians. cv::pow() void cv::pow(   cv::InputArray

coordinates from polar coordinates. The input is in two arrays, magnitude and angle, of the same size and type, specifying the magnitude and angle of the field at every point. The output is similarly two arrays that will be of the same size and type as the inputs, and which will contain the x and y projections of the vector at each point. The additional flag angleInDegrees will cause the angle array to be interpreted as angles in degrees, rather than in radians. cv::pow() void cv::pow(   cv::InputArray  src,                    // Input array    double          p,                      // power for exponentiation    cv::OutputArray dst                     // Result array );    dsti=srcipp∈Zsrcipelse⎧⎨⎪⎩⎪

The function cv::pow() computes the element-wise exponentiation of an array by a given power p. In the case in which p is an integer, the power is computed directly. For non-integer p, the absolute value of

the source value is computed first, and then raised to the power p (so only real values are returned). For some special values of p, such as integer values, or †±12

, special algorithms are used resulting in faster computation. cv::randu() template<typename _Tp> _Tp randu();       // Return random number of specific type   void cv::randu(   cv::InputOutArray mtx,                  // All values will be randomized   cv::InputArray    low,                  // minimum, 1-by-1 (Nc=1,4), or 1-by-4 (Nc=1)   cv::InputArray    high                  // maximum, 1-by-1 (Nc=1,4), or 1-by-4 (Nc=1) );    mtxi∈lowi,†highi⎡⎣) There are two ways to call cv::randu(). The first method is to call the template form of randu<>(), which will return a random value of the appropriate type. Random numbers generated in this way are uniformly distributed35 in the range from zero to the maximum value available for that type (for integers), and in the interval from 0.0 to 1.0 (not inclusive of 1.0) for floating-point types.  This template form generates only single numbers36. The second way to call cv::randu() is to provide a matrix mtx that  you wish to have filled with values, and two additional

is to call the template form of randu<>(), which will return a random value of the appropriate type. Random numbers generated in this way are uniformly distributed35 in the range from zero to the maximum value available for that type (for integers), and in the interval from 0.0 to 1.0 (not inclusive of 1.0) for floating-point types.  This template form generates only single numbers36. The second way to call cv::randu() is to provide a matrix mtx that  you wish to have filled with values, and two additional arrays that  specify the minimum and maximum values for the range from which you would like a random number drawn for each particular array element.  These two additional values, low and high, should be 1-by-1 with 1 or 4 channels, or 1-by-4 with a single channel; they may also be of type cv::Scalar.  In any case, they are not the size of mtx, but rather the size of individual entries in mtx.  The array mtx is both an input and an output, in the sense that you must allocate the matrix so that cv::randu()

arrays that  specify the minimum and maximum values for the range from which you would like a random number drawn for each particular array element.  These two additional values, low and high, should be 1-by-1 with 1 or 4 channels, or 1-by-4 with a single channel; they may also be of type cv::Scalar.  In any case, they are not the size of mtx, but rather the size of individual entries in mtx.  The array mtx is both an input and an output, in the sense that you must allocate the matrix so that cv::randu() will know the number of random values you need, and how they are to be arranged in terms of rows, columns, and channels. cv::randn() void cv::randn(   cv::InputOutArray mtx,                  // All values will be randomized   cv::InputArray    mean,                 // means, array is in channel-space    cv::InputArray    stddev                // standard deviations, channel-space );    mtxi~Nmeani,†stddevi() The function cv::randn() fills a matrix mtx with random normally distributed values37. The parameters

know the number of random values you need, and how they are to be arranged in terms of rows, columns, and channels. cv::randn() void cv::randn(   cv::InputOutArray mtx,                  // All values will be randomized   cv::InputArray    mean,                 // means, array is in channel-space    cv::InputArray    stddev                // standard deviations, channel-space );    mtxi~Nmeani,†stddevi() The function cv::randn() fills a matrix mtx with random normally distributed values37. The parameters from which these values are drawn are taken from two additional arrays (mean and stddev) that specify the mean and standard deviation for the distribution from which you would like a random number drawn for each particular array element.                                                              35 Uniform distribution random numbers are generated using the Multiply-With-Carry algorithm [G. Marsaglia]. 36 In particular, this means that if you call the template form with a vector argument, such as:

from which these values are drawn are taken from two additional arrays (mean and stddev) that specify the mean and standard deviation for the distribution from which you would like a random number drawn for each particular array element.                                                              35 Uniform distribution random numbers are generated using the Multiply-With-Carry algorithm [G. Marsaglia]. 36 In particular, this means that if you call the template form with a vector argument, such as: cv::randu<Vec4f>, the return value, though it will be of vector type, will be all zeros except for the first element. 37 Gaussian-distribution random numbers are generated using the Ziggurat algorithm [G. Marsaglia, W. W. Tsang].

As with the array form of cv::randu(), every element of mtx is computed separately, and the arrays mean and stddev are in the channel-space for individual entries of mtx.  Thus, if mtx were four channels, then mean and stddev would be 1-by-4 or 1-by-1 with 4 channels (or equivalently of type cv::Scalar).38 cv::randShuffle() void cv::randShuffle(   cv::InputOutArray mtx,                  // All values will be shuffled   double            iterFactor = 1,       // Number of times to repeat shuffle   cv::RNG*          rng        = NULL     // your own generator, if you like ); cv::randShuffle() attempts to randomize the entries in a one-dimensional array by selecting random pairs of elements and interchanging their position. The number of such swaps is equal to the size of the array mtx multiplied by the optional factor iterFactor. Optionally, a random number generator can be supplied (for more on this, see “Random Number Generator (cv::RNG)”). If none is supplied, the default random number generator theRNG()

rng        = NULL     // your own generator, if you like ); cv::randShuffle() attempts to randomize the entries in a one-dimensional array by selecting random pairs of elements and interchanging their position. The number of such swaps is equal to the size of the array mtx multiplied by the optional factor iterFactor. Optionally, a random number generator can be supplied (for more on this, see “Random Number Generator (cv::RNG)”). If none is supplied, the default random number generator theRNG() will be used automatically. cv::reduce() void cv::reduce(   cv::InputArray  src,                    // Input array, n-by-m must be 2-dimensional   cv::OutputArray vec,                    // Output array, 1-by-m or n-by-1   int             dim,                    // Reduction direction (1) to row, (0) to col   int             reduceOp = cv::REDUCE_SUM,  // Reduce operation (see Table 3-)    int             dtype = -1              // Output type for result array  ); Reduction is the systematic transformation

will be used automatically. cv::reduce() void cv::reduce(   cv::InputArray  src,                    // Input array, n-by-m must be 2-dimensional   cv::OutputArray vec,                    // Output array, 1-by-m or n-by-1   int             dim,                    // Reduction direction (1) to row, (0) to col   int             reduceOp = cv::REDUCE_SUM,  // Reduce operation (see Table 3-)    int             dtype = -1              // Output type for result array  ); Reduction is the systematic transformation of the input matrix src into a vector vec by applying some combination rule reduceOp on each row (or column) and its neighbor until only one row (or column) remains (see Table 3-28).39 The argument dim controls how the reduction is done, as summarized in Table 3-29. Table 3-28: Argument reduceOp in cv::reduce() selects the reduction operator Value of op Result  cv::REDUCE_SUM  Compute sum across vectors  cv::REDUCE_AVG  Compute average across vectors  cv::REDUCE_MAX  Compute maximum across vectors

of the input matrix src into a vector vec by applying some combination rule reduceOp on each row (or column) and its neighbor until only one row (or column) remains (see Table 3-28).39 The argument dim controls how the reduction is done, as summarized in Table 3-29. Table 3-28: Argument reduceOp in cv::reduce() selects the reduction operator Value of op Result  cv::REDUCE_SUM  Compute sum across vectors  cv::REDUCE_AVG  Compute average across vectors  cv::REDUCE_MAX  Compute maximum across vectors  cv::REDUCE_MIN  Compute minimum across vectors Table 3-29: Argument dim in cv::reduce() controls the direction of the reduction Value of dim Result 1                    Collapse to a single row 0                    Collapse to a single column cv::reduce() supports multichannel arrays of any type. Using dtype, you can specify an alternative type for dst.                                                               38 Note that stddev is not a square matrix; correlated number generation is not supported by

cv::REDUCE_MIN  Compute minimum across vectors Table 3-29: Argument dim in cv::reduce() controls the direction of the reduction Value of dim Result 1                    Collapse to a single row 0                    Collapse to a single column cv::reduce() supports multichannel arrays of any type. Using dtype, you can specify an alternative type for dst.                                                               38 Note that stddev is not a square matrix; correlated number generation is not supported by cv::randn(). 39 Purists will note that averaging is not technically a proper fold in the sense implied here. OpenCV has a more practical view of reductions and so includes this useful operation in cvReduce.

Using the dtype argument to specify a higher-precision format for dst is particularly important for cv::REDUCE_SUM and cv::REDUCE_AVG, where overflows and summation problems are possible. cv::repeat() void cv::repeat(   cv::InputArray  src,                    // Input 2-dimensional array   int             nx,                     // Copies in x-direction   int             ny,                     // Copies in y-direction   cv::OutputArray dst                     // Resukt array ); cv::Mat cv::repeat(                       // Return result array   cv::InputArray  src,                    // Input 2-dimensional array   int             nx,                     // Copies in x-direction   int             ny                      // Copies in y-direction );    dsti,j=srci%src.rows,†j%src.cols†

This function copies the contents of src into dst, repeating as many times as necessary to fill dst. In particular, dst can be of any size relative to src. It may be larger or smaller, and it need not have an integer relationship between any of its dimensions and the corresponding dimensions of src. cv::repeat() has two calling conventions. The first is the old-style convention, in which the output array is passed as a reference to cv::repeat(). The second actually creates and returns a cv::Mat, and is much more convenient when working with matrix expressions. cv::scaleAdd() void cv::scaleAdd(   cv::InputArray  src1,                   // First input array    double          scale,                  // Scale factor applied to first array   cv::InputArray  src2,                   // Second input array    cv::OutputArray dst,                    // Result array  );    dsti=scale*src1i+src2i

cv::scaleAdd() is used to compute the sum of two arrays src1 and src2 with a scale factor scale applied to the first before the sum is done. The results are placed in the array dst. The same result can be achieved with the matrix algebra operation: dst = scale * src1 + src2; cv::setIdentity() void cv::setIdentity(    cv::InputOutputArray dst,               // Array to reset values              const cv::Scalar&    value = cv::Scalar(1.0)  // Value to apply to diagonal elems );    dsti,j=valuei=j0else⎧⎨⎪⎩⎪
 cv::setIdentity() sets all elements of the array to 0 except for elements whose row and column are equal; those elements are set to 1 (or to value if provided). cv::setIdentity() supports all data types and does not require the array to be square.

This can also be done using the eye() member function of the cv::Mat class. The use of eye() is often more convenient when one is working with matrix expressions. Mat A( 3, 3, cv::F32 ); cv::setIdentity( A, s ); C = A + B; For some other arrays B and C, and some scalar s, this is equivalent to:           C = s * cv::Mat::eye( 3, 3, cv::F32 ) + B; cv::solve() int cv::solve(   cv::InputArray  lhs,                    // Left-hand side of system, n-by-n   cv::InputArray  rhs,                    // Right-hand side of system, n-by-1     cv::OutputArray dst,                    // Results array, will be n-by-1    int             method = cv::DECOMP_LU  // Method for solver ); The function cv::solve() provides a fast way to solve linear systems based on cv::invert(). It computes the solution to     C=argminX†A!X−B

where A is a square matrix given by lhs, B is the vector rhs, and C is the solution computed by cv::solve() for the best vector X it could find. That best vector X is returned in dst. The actual method used to solve this system is determined by the value of the method argument (Table 3-30). Only floating-point data types are supported. The function returns an integer value where a nonzero return indicates that it could find a solution. Table 3-30: Possible values of method argument to cv::solve() Value of method argument Meaning cv::DECOMP_LU                    Gaussian elimination (LU decomposition) cv::DECOMP_SVD                    Singular value decomposition (SVD) cv::DECOMP_CHOLESKY                    For symmetric positive matrices cv::DECOMP_EIG                    Eigenvalue decomposition, symmetric matrices only cv::DECOMP_QR                    QR Factorization cv::DECOMP_NORMAL                    Optional additional flag, indicates that the normal equations are to be solved instead) The methods

of method argument Meaning cv::DECOMP_LU                    Gaussian elimination (LU decomposition) cv::DECOMP_SVD                    Singular value decomposition (SVD) cv::DECOMP_CHOLESKY                    For symmetric positive matrices cv::DECOMP_EIG                    Eigenvalue decomposition, symmetric matrices only cv::DECOMP_QR                    QR Factorization cv::DECOMP_NORMAL                    Optional additional flag, indicates that the normal equations are to be solved instead) The methods cv::DECOMP_LU and cv::DECOMP_CHOLESKY cannot be used on singular matrices. If a singular src1 is provided, both methods will exit and return 0 (a 1 will be returned if src1 is nonsingular). cv::solve() can be used to solve over-determined linear systems using either QR decomposition (cv::DECOMP_QR) or singular value decomposition (cv::DECOMP_SVD) methods to find the least-squares solution for the given system of equations. Both of these methods can be used in case the matrix src1 is singular. Though the

cv::DECOMP_LU and cv::DECOMP_CHOLESKY cannot be used on singular matrices. If a singular src1 is provided, both methods will exit and return 0 (a 1 will be returned if src1 is nonsingular). cv::solve() can be used to solve over-determined linear systems using either QR decomposition (cv::DECOMP_QR) or singular value decomposition (cv::DECOMP_SVD) methods to find the least-squares solution for the given system of equations. Both of these methods can be used in case the matrix src1 is singular. Though the first five arguments in Table 3-30 are mutually exclusive, the last option cv::DECOMP_NORMAL may be combined with any of the first five (e.g., by logical-or: cv_DECOMP_LU | cv::DECOMP_NORMAL). If provided, then cv::solve() will attempt to solve the normal equations:    src1T!src1!dst=src1T!src2

instead of the usual system    src1!dst=src2. cv::solveCubic() int cv::solveCubic(   cv::InputArray  coeffs,   cv::InputArray  roots

); Given a cubic polynomial in the form of a three- or four-element vector coeffs, cv::solveCubic() will compute the real roots of that polynomial. If coeffs has four elements, the roots of the following polynomial are computed:    coeffs0x3+coeffs1x2+coeffs2x+coeffs3=0
 If coeffs has only three elements, the roots of the following polynomial are computed:    x3+coeffs0x2+coeffs1x+coeffs2=0

The results are stored in the array roots, which will have either one or three elements, depending on how may real roots the polynomial has. A word of warning about cv::solveCubic() and cv::solvePoly(): the order of the coefficients in the seemingly analogous input arrays coeffs is opposite in the two routines. In cv::solveCubic(), the highest order coefficients come first, while in cv::solvePoly() the highest order coefficients come last. cv::solvePoly() int cv::solvePoly (   cv::InputArray  coeffs,   cv::OutputArray roots                   // n complex roots (2-channels)   int             maxIters = 300          // maximum iterations for solver ); Given a polynomial of any order in the form of a vector of coefficients coeffs, cv::solvePoly() will attempt to compute the roots of that polynomial. Given the array of coefficients coeffs, the roots of the following polynomial are computed:     coeffsnxn+coeffsn−1xn−1+!+coeffs1x+coeffs0=0.

These roots are not guaranteed to be real. For an order-n polynomial (i.e., coeffs having   n+1 elements), there will be  n roots. As a result, the array roots will be returned in a two-channel (real, imaginary) matrix of doubles.   cv::sort() void cv::sort(   cv::InputArray  src,   cv::InputArray  dst,   int             flags ); The OpenCV sort function is used for two-dimensional arrays. Only single-channel source arrays are supported. You should not think of this like sorting rows or columns in a spreadsheet; cv::sort() sorts every row or column separately. The result of the sort operation will be a new array dst, which is of the same size and type as the source array. Sorting can be done on every row or on every column by supplying either the cv::SORT_EVERY_ROW or cv::SORT_EVERY_COLUMN flag. Sort can be in ascending or descending order, which is indicated by the cv::SORT_ASCENDING or cv::SORT_DESCENDING flags respectively. One flag from each of the two groups is required. cv::sortIdx() void cv::sortIdx(

cv::sort() sorts every row or column separately. The result of the sort operation will be a new array dst, which is of the same size and type as the source array. Sorting can be done on every row or on every column by supplying either the cv::SORT_EVERY_ROW or cv::SORT_EVERY_COLUMN flag. Sort can be in ascending or descending order, which is indicated by the cv::SORT_ASCENDING or cv::SORT_DESCENDING flags respectively. One flag from each of the two groups is required. cv::sortIdx() void cv::sortIdx(   cv::InputArray  src,   cv::InputArray  dst,   int             flags

); Similar to cv::sort(), cv::sortIdx()is used only for single-channel two-dimensional arrays. cv::sortIdx() sorts every row or column separately. The result of the sort operation is a new array dst of the same size as the source array, but which contains the integer indices of the sorted elements. For example, given an array A, a call to cv::sortIdx( A, B, cv::SORT_EVERY_ROW | cv::SORT_DESCENDING ) would produce:  A =  0.00.10.21.01.11.22.02.12.2⎡⎣⎢⎢⎢⎤⎦⎥⎥⎥
 B =  210210210⎡⎣⎢⎢⎢⎤⎦⎥⎥⎥

In this toy case, every row was previously ordered from lowest to highest and sorting has indicated that this should be reversed.  cv::split() void cv::split(   const cv::Mat&   mtx,   cv::Mat*         mv ); void cv::split(   const cv::Mat&   mtx,   vector<Mat>&     mv                     // STL-style vector of n 1-channel cv::Mat’s ); The function cv::split() is a special, simpler case of cv::mixChannels(). cv::split() separates the channels in a multichannel array into multiple single-channel arrays. There are two ways for doing this: in the first, you supply a pointer to a C-style array of pointers to cv::Mat objects that cv::split() will use for the results of the split operation. In the second option, you supply an STL vector full of cv::Mat objects. If you use the C-style array, you need to make sure that the number of cv::Mat objects available is (at least) equal to the number of channels in mtx. If you use the STL vector form, cv::split() will handle the allocation of the result arrays for you.

There are two ways for doing this: in the first, you supply a pointer to a C-style array of pointers to cv::Mat objects that cv::split() will use for the results of the split operation. In the second option, you supply an STL vector full of cv::Mat objects. If you use the C-style array, you need to make sure that the number of cv::Mat objects available is (at least) equal to the number of channels in mtx. If you use the STL vector form, cv::split() will handle the allocation of the result arrays for you.  cv::sqrt() void cv::sqrt(   cv::InputArray  src,   cv::InputArray  dst ); As a special case of cv::pow(), cv::sqrt() will compute the element-wise square root of an array. Multiple channels are processed separately.  There is such thing as a square root of a matrix, i.e., a matrix  B whose relationship with some matrix  A is that   BB=A. If  A is square and positive definite, then if  B exists, it is unique. If  A can be diagonalized, then there is a matrix  V (made from the eigenvectors of  A as columns)

cv::sqrt() void cv::sqrt(   cv::InputArray  src,   cv::InputArray  dst ); As a special case of cv::pow(), cv::sqrt() will compute the element-wise square root of an array. Multiple channels are processed separately.  There is such thing as a square root of a matrix, i.e., a matrix  B whose relationship with some matrix  A is that   BB=A. If  A is square and positive definite, then if  B exists, it is unique. If  A can be diagonalized, then there is a matrix  V (made from the eigenvectors of  A as columns) such that  †A†=VDV−1, where  D is a diagonal matrix. The square root of a diagonal matrix  D is just the square roots of the elements of  †D. So to compute   A12

, we simply use the matrix  V and get:    A12=VD12V−1
 Math fans can easily verify that this expression is correct by explicitly squaring it:

In code, this would look something like40: void matrix_square_root( const cv::Mat& A, cv::Mat& sqrtA ) {   cv::Mat U, V, Vi, E;   cv::eigen( A, E, U );   V = U.T();   cv::transpose( V, Vi ); // inverse of the orthogonal V   cv::sqrt(E, E); // assume that A is positively-defined, otherwise its square root is complex-valued   sqrtA = V * Mat::diag(E) * Vi; } cv::subtract()  void cv::subtract(   cv::InputArray  src1,                   // First input array    cv::InputArray  src2,                   // Second input array    cv::OutputArray dst,                    // Result array    cv::InputArray  mask  = cv::noArray(),  // Optional mask, compute only where nonzero   int             dtype = -1              // Output type for result array )      dsti=saturate†src1i−src2i() cv::subtract() is a simple subtraction function: it subtracts all of the elements in src2 from the corresponding elements in src1 and puts the results in dst.  For simple cases the same result can be achieved with the matrix operation: dst =

input array    cv::OutputArray dst,                    // Result array    cv::InputArray  mask  = cv::noArray(),  // Optional mask, compute only where nonzero   int             dtype = -1              // Output type for result array )      dsti=saturate†src1i−src2i() cv::subtract() is a simple subtraction function: it subtracts all of the elements in src2 from the corresponding elements in src1 and puts the results in dst.  For simple cases the same result can be achieved with the matrix operation: dst = src1 - src2; Accumulation is also supported:  dst -= src1;  cv::sum() cv::Scalar cv::sum(   cv::InputArray  arr );    sumc=i,j∑arrci,j

cv::sum() sums all of the pixels in each of the channels of the array arr. The return value is of type cv::Scalar, so cv::sum() can accommodate multichannel arrays, but only up to four channels. The sum for each channel is placed in the corresponding component of the cv::scalar return value. cv::trace() cv::Scalar cv::trace(   cv::InputArray  mat                                                              40 Here “something like” means that if you were really writing a responsible piece of code, you would do a lot of checking to make sure that the matrix you were handed was in fact what you thought it was (i.e., square). You would also probably want to check the return values of cv::eigen() and cv::invert() and think more carefully about the actual methods used for the decomposition and inversion, make sure the eigenvalues are positive before blindly calling sqrt() on them.

);    Trmat()c=i∑matci,i
 The trace of a matrix is the sum of all of the diagonal elements. The trace in OpenCV is implemented on top of cv::diag(), so it does not require the array passed in to be square. Multichannel arrays are supported, but the trace is computed as a scalar so each component of the scalar will be the sum over each corresponding channel for up to four channels. cv::transform() void cv::transform(   cv::InputArray  src,   cv::OutputArray dst,   cv::InputArray  mtx );    dstc,i,j=′c∑mtxc,′csrc′ci,j

The function cv::transform() can be used to compute arbitrary linear image transforms. It treats a multichannel input array src as a collection of vectors in what you could think of as “channel space.” Those vectors are then each multiplied by the “small” matrix mtx, to affect a transformation in this channel space.  The matrix mtx must have as many rows as there are channels in src, or that number plus one. In the second case, the channel space vectors in src are automatically extended by one and the value 1.0 is assigned to the extended element. The exact meaning of this transformation depends on what you are using the different channels for. If you are using the channels as color channels, then this transformation can be thought of as a linear color space transformation. (Transformation between RGB and YUV color spaces is an example of such a transformation. If you are using the channels to represent the   x,y() or   x,y,z() coordinates of points, then these transformations can be thought of as rotations

1.0 is assigned to the extended element. The exact meaning of this transformation depends on what you are using the different channels for. If you are using the channels as color channels, then this transformation can be thought of as a linear color space transformation. (Transformation between RGB and YUV color spaces is an example of such a transformation. If you are using the channels to represent the   x,y() or   x,y,z() coordinates of points, then these transformations can be thought of as rotations (or other geometrical transformations) of those points.  cv::transpose() void cv::transpose(   cv::InputArray  src,                    // Input array, 2-dimensional, n-by-m   cv::OutputArray dst,                    // Result array, 2-dimensional, m-by-n ); cv::transpose() copies every element of src into the location in dst indicated by reversing the row and column indices. This function does support multichannel arrays; however, if you are using multiple channels to represent complex numbers, remember that

other geometrical transformations) of those points.  cv::transpose() void cv::transpose(   cv::InputArray  src,                    // Input array, 2-dimensional, n-by-m   cv::OutputArray dst,                    // Result array, 2-dimensional, m-by-n ); cv::transpose() copies every element of src into the location in dst indicated by reversing the row and column indices. This function does support multichannel arrays; however, if you are using multiple channels to represent complex numbers, remember that cv::transpose() does not perform complex conjugation. This same result can be achieved with the matrix member function cv::Mat::t(). The member function has the advantage that it can be used in matrix expressions like: A = B + B.t(); Utility Functions There are several operations provided by OpenCV that operate on single values (integers or floating-point numbers). These functions either serve some special function, or are fast implementations of particular

things that come up often enough in vision processing to justify implementing in a custom way for OpenCV. Table 3-31 Utility and System Functions Function Description cv::alignPtr() Align pointer to given number of bytes cv::alignSize() Align buffer size to given number of bytes cv::allocate() Allocate a C-style array of objects cvCeil()41  Round float number x to nearest integer not smaller than x cv::cubeRoot()                Compute the cube root of a number cv::CV_Assert() Throw an exception if a given condition is not true CV_Error() Macro to build a cv::Exception (from a fixed string) and throw it CV_Error_() Macro to build a cv::Exception (from a formatted string) and throw it cv::deallocate() Deallocate a C-style array of objects cv::error() Indicate an error and throw an exception cv::fastAtan2()                Calculate two-dimensional angle of a vector in degrees cv::fastFree() Deallocate a memory buffer cv::fastMalloc() Allocate an aligned memory buffer cvFloor()41 Round float number x to nearest

condition is not true CV_Error() Macro to build a cv::Exception (from a fixed string) and throw it CV_Error_() Macro to build a cv::Exception (from a formatted string) and throw it cv::deallocate() Deallocate a C-style array of objects cv::error() Indicate an error and throw an exception cv::fastAtan2()                Calculate two-dimensional angle of a vector in degrees cv::fastFree() Deallocate a memory buffer cv::fastMalloc() Allocate an aligned memory buffer cvFloor()41 Round float number x to nearest integer not larger than  x cv::format() Create an STL string using sprinf-like formatting cv::getCPUTickCount() Get tick count from internal CPU timer cv::getNumThreads() Count number of threads currently used by OpenCV cv::getOptimalDFTSize()                Compute the best size for an array that you plan to pass to cv::DFT() cv::getThreadNum() Get index of the current thread cv::getTickCount() Get tick count from generic source cv::getTickFrequency() Get number or ticks per second (see

x to nearest integer not larger than  x cv::format() Create an STL string using sprinf-like formatting cv::getCPUTickCount() Get tick count from internal CPU timer cv::getNumThreads() Count number of threads currently used by OpenCV cv::getOptimalDFTSize()                Compute the best size for an array that you plan to pass to cv::DFT() cv::getThreadNum() Get index of the current thread cv::getTickCount() Get tick count from generic source cv::getTickFrequency() Get number or ticks per second (see cv::getTickCount())                                                              41 This function has something of a legacy interface.  It is a C definition, not C++ (see core …/types_c.h) where it is defined as an inline function.  There are several others with a similar interface.

cvIsInf()41 Check if a floating-point number x is infinity cvIsNaN()41 Check if a floating-point number x is “Not a Number” cvRound()  Round float number x to the nearest integer cv::setNumThreads() Set number of threads used by OpenCV cv::setUseOptimized() Enables or disables the use of optimized code (SSE2, etc.) cv::useOptimized() Indicates status of optimized code enabling (see cv::setUseOptimized()) cv::alignPtr() template<T> T* cv::alignPtr(              // Return aligned pointer of type T*   T*  ptr,                                // pointer, unaligned   int n   = sizeof(T)                     // align to block size, a power of 2 ); Given a pointer of any type, this function computes an aligned pointer of the same type according to the following computation: (T*)(((size_t)ptr + n+1) & -n) On some architectures, it is not even possible to read a multi-byte object from an address that is not evenly divisible by the size of the object (i.e.,.,, by 4 for a 32-bit integer). On architectures such as x86,

// pointer, unaligned   int n   = sizeof(T)                     // align to block size, a power of 2 ); Given a pointer of any type, this function computes an aligned pointer of the same type according to the following computation: (T*)(((size_t)ptr + n+1) & -n) On some architectures, it is not even possible to read a multi-byte object from an address that is not evenly divisible by the size of the object (i.e.,.,, by 4 for a 32-bit integer). On architectures such as x86, this is handled for you automatically by the CPU by using multiple reads and assembling your value from those reads at the cost of a substantial penalty in performance. cv::alignSize() size_t cv::alignSize(                     // minimum size >=’sz’ divisible by ‘n’   size_t sz,                              // size of buffer   int n   = sizeof(T)                     // align to block size, a power of 2 ); Given a number n (typically a return value from sizeof()), and a size for a buffer sz, cv::alignSize()

this is handled for you automatically by the CPU by using multiple reads and assembling your value from those reads at the cost of a substantial penalty in performance. cv::alignSize() size_t cv::alignSize(                     // minimum size >=’sz’ divisible by ‘n’   size_t sz,                              // size of buffer   int n   = sizeof(T)                     // align to block size, a power of 2 ); Given a number n (typically a return value from sizeof()), and a size for a buffer sz, cv::alignSize() computes the size that this buffer should be in order to contain an integer number of objects of size n, i.e., the minimum number that is greater or equal to sz yet divisible by n. The following formula is used:  (sz + n-1) & -n cv::allocate() template<T> T* cv::allocate(              // Return pointer to allocated buffer   size_t sz                               // size of buffer, multiples of sizeof(T) );  The function cv::allocate() functions similarly to the array form of new, in that it allocates a

computes the size that this buffer should be in order to contain an integer number of objects of size n, i.e., the minimum number that is greater or equal to sz yet divisible by n. The following formula is used:  (sz + n-1) & -n cv::allocate() template<T> T* cv::allocate(              // Return pointer to allocated buffer   size_t sz                               // size of buffer, multiples of sizeof(T) );  The function cv::allocate() functions similarly to the array form of new, in that it allocates a C-style array of n objects of type T, calls the default constructor for each object, and returns a pointer to the first object in the array.  cv::deallocate() template<T> void cv::deallocate(   T*     ptr,                             // Pointer to buffer to free   size_t sz                               // size of buffer, multiples of sizeof(T) );

The function cv::deallocate() functions similarly to the array form of delete, in that it deallocates a C-style array of n objects of type T, and calls the destructor for each object. cv::deallocate() is used to deallocate objects allocated with cv::allocate(). The number of elements n passed to cv::deallocate() must be the same as the number of objects originally allocated with cv::allocate(). cv::fastAtan2() float cv::fastAtan2(                      // Return value is 32-bit float   float y,                                // y input value (32-bit float)   float x                                 // x input value (32-bit float) ); This function computes the arctangent of an x,y pair and returns the angle from the origin to the indicated point. The result is reported in degrees ranging from 0.0 to 360.0, inclusive of 0.0 but not inclusive of 360.0. cvCeil() int cvCeil(                               // Return the smallest int >= x   float x                                 // input value (32-bit float) ); Given

// y input value (32-bit float)   float x                                 // x input value (32-bit float) ); This function computes the arctangent of an x,y pair and returns the angle from the origin to the indicated point. The result is reported in degrees ranging from 0.0 to 360.0, inclusive of 0.0 but not inclusive of 360.0. cvCeil() int cvCeil(                               // Return the smallest int >= x   float x                                 // input value (32-bit float) ); Given a floating-point number x, cvCeil() computes the smallest integer not smaller than x. If the input value is outside of the range representable by a 32-bit integer, the result is undefined. cv::cubeRoot()  float cv::cubeRoot(                       // Return value is 32-bit float   float x                                 // input value (32-bit float) ); This function computes the cubed root of the argument x. Negative values of x are handled correctly (i.e., the return value is negative). cv::CV_Assert() and

a floating-point number x, cvCeil() computes the smallest integer not smaller than x. If the input value is outside of the range representable by a 32-bit integer, the result is undefined. cv::cubeRoot()  float cv::cubeRoot(                       // Return value is 32-bit float   float x                                 // input value (32-bit float) ); This function computes the cubed root of the argument x. Negative values of x are handled correctly (i.e., the return value is negative). cv::CV_Assert() and CV_DbgAssert() // example CV_Assert( x!=0 ) CV_Assert() is a macro that will test the expression passed to it and, if that expression evaluates to False (or 0), it will throw an exception. The CV_Assert() macro is always tested. Alternatively, you can use CV_DbgAssert(), which will only be tested in debug compilations. cv::CV_Error() and CV_Error_() // example CV_Error( ecode, estring )  CV_Error_( ecode, fmt, ... ) The macro CV_Error() allows you to pass in an error code ecode and a fixed C-style

and CV_DbgAssert() // example CV_Assert( x!=0 ) CV_Assert() is a macro that will test the expression passed to it and, if that expression evaluates to False (or 0), it will throw an exception. The CV_Assert() macro is always tested. Alternatively, you can use CV_DbgAssert(), which will only be tested in debug compilations. cv::CV_Error() and CV_Error_() // example CV_Error( ecode, estring )  CV_Error_( ecode, fmt, ... ) The macro CV_Error() allows you to pass in an error code ecode and a fixed C-style character string estring, which it then packages up into a cv::Exception and passes that to cv::error() to be handled. The variant macro CV_Error_() is used if you need to construct the message string on the fly. CV_Error_() accepts the same ecode as CV_Error(), but then expects a sprintf() style format string followed by a variable number of arguments as would be expected by sprintf(). cv::error()  void cv::error(   const cv::Exception& ex                 // Exception to be thrown ); This function is mostly

string estring, which it then packages up into a cv::Exception and passes that to cv::error() to be handled. The variant macro CV_Error_() is used if you need to construct the message string on the fly. CV_Error_() accepts the same ecode as CV_Error(), but then expects a sprintf() style format string followed by a variable number of arguments as would be expected by sprintf(). cv::error()  void cv::error(   const cv::Exception& ex                 // Exception to be thrown ); This function is mostly called from CV_Error() and CV_Error_(). If your code is compiled in a non-debug build, it will throw the exception ex. If your code is compiled in a debug build, it will deliberately provoke a memory access violation so that the execution stack and all of the parameters will be available for whatever debugger you are running.

You will probably not call cv::error() directly, but rather rely on the macros CV_Error() and CV_Error_() to throw the error for you. These macros take the information you want displayed in the exception and package it up for you and pass the resulting exception to cv::error().  cv::fastFree()  void cv::fastFree(   void* ptr                               // Pointer to buffer to be freed ); This routine deallocates buffers that were allocated with cv::fastMalloc() (covered next). cv::fastMalloc()  void* cv::fastMalloc(                     // Pointer to allocated buffer   size_t size                             // Size of buffer to allocate ); cv::FastMalloc() works just like the malloc() you are familiar with, except that it is often faster, and it does buffer size alignment for you. This means that if the buffer size passed is more than 16 bytes, the returned buffer will be aligned to a 16 byte boundary. cvFloor()  int cvFloor(                              // Return the largest int <= x   float x

// Pointer to allocated buffer   size_t size                             // Size of buffer to allocate ); cv::FastMalloc() works just like the malloc() you are familiar with, except that it is often faster, and it does buffer size alignment for you. This means that if the buffer size passed is more than 16 bytes, the returned buffer will be aligned to a 16 byte boundary. cvFloor()  int cvFloor(                              // Return the largest int <= x   float x                                 // input value (32-bit float) }; Given a floating-point number x, cv::Floor() computes the largest integer not larger than x. If the input value is outside of the range representable by a 32-bit integer, the result is undefined. cv::format()  string cv::format(                        // Return STL-string   const char* fmt,                        // formatting string, as sprintf()   ...                                     // vargs, as sprintf() ); This function is essentially the same as sprintf()

// input value (32-bit float) }; Given a floating-point number x, cv::Floor() computes the largest integer not larger than x. If the input value is outside of the range representable by a 32-bit integer, the result is undefined. cv::format()  string cv::format(                        // Return STL-string   const char* fmt,                        // formatting string, as sprintf()   ...                                     // vargs, as sprintf() ); This function is essentially the same as sprintf() from the standard library, but rather than requiring a character buffer from the caller, it constructs an STL string object and returns that. It is particularly handy for formatting error messages for the Exception() constructor (which expects STL strings as argument). cv::getCPUTickCount() int64 cv::getCPUTickCount( void );        // long int CPU for tick count This function reports the number of CPU ticks on those architectures that have such a construct (including, but not limited to, x86

as sprintf() from the standard library, but rather than requiring a character buffer from the caller, it constructs an STL string object and returns that. It is particularly handy for formatting error messages for the Exception() constructor (which expects STL strings as argument). cv::getCPUTickCount() int64 cv::getCPUTickCount( void );        // long int CPU for tick count This function reports the number of CPU ticks on those architectures that have such a construct (including, but not limited to, x86 architectures). It is important to know, however, that the return value of this function can be very difficult to interpret on many architectures. In particular, because on a multi-core system a thread can be put to sleep on one core and wake up on another, the difference between the results to two subsequent calls to cv::getCPUTickCount() can be misleading, even completely meaningless. Therefore, unless you are certain you know what you are doing, it is best to use cv::getTickCount() for timing

to, x86 architectures). It is important to know, however, that the return value of this function can be very difficult to interpret on many architectures. In particular, because on a multi-core system a thread can be put to sleep on one core and wake up on another, the difference between the results to two subsequent calls to cv::getCPUTickCount() can be misleading, even completely meaningless. Therefore, unless you are certain you know what you are doing, it is best to use cv::getTickCount() for timing measurements.42 This function is best for tasks like initializing random number generators. cv::getNumThreads() int cv::getNumThreads( void );            // number of threads allocated to OpenCV    Return the current number of threads being used by OpenCV.                                                              42 Of course, if you really do know what you are doing, then there is no more accurate way to get detailed timing information than from the CPU timers themselves.

cv::getOptimalDFTSize()                int cv::getOptimalDFTSize( int n );       // best size array to use for dft, >= n When making calls to cv::dft(), the algorithm used by OpenCV to compute the transform is extremely sensitive to the size of the array passed to cv::dft(). The preferred sizes do obey a rule for their generation, but that rule is sufficiently complicated that it is (at best) an annoyance to compute the correct size to which to pad your array every time. The function cv::getOptimalDFTSize() takes as argument the size of the array you would have passed to cv::dft(), and returns the size of the array you should pass to cv::dft(). The usage of this information is to just create a larger array into which you can copy your data and pad out the rest with zeros. cv::getThreadNum() int cv::getThreadNum( void );             // int, id of this particular thread If your OpenCV library was compiled with OpenMP support, it will return the index (starting from zero) of the currently executing thread.

takes as argument the size of the array you would have passed to cv::dft(), and returns the size of the array you should pass to cv::dft(). The usage of this information is to just create a larger array into which you can copy your data and pad out the rest with zeros. cv::getThreadNum() int cv::getThreadNum( void );             // int, id of this particular thread If your OpenCV library was compiled with OpenMP support, it will return the index (starting from zero) of the currently executing thread. cv::getTickCount() int64 cv::getTickCount( void );           // long int CPU for tick count This function returns a tick count relative to some architecture-dependent time. The rate of ticks is also architecture and operating system dependent, however; the time per tick can be computed by cv::getTickFrequency() (reviewed next).below). This function is preferable to cv::getCPUTickCount() for most timing applications, as it is not affected by low-level issues such as which core your thread is running on and

cv::getTickCount() int64 cv::getTickCount( void );           // long int CPU for tick count This function returns a tick count relative to some architecture-dependent time. The rate of ticks is also architecture and operating system dependent, however; the time per tick can be computed by cv::getTickFrequency() (reviewed next).below). This function is preferable to cv::getCPUTickCount() for most timing applications, as it is not affected by low-level issues such as which core your thread is running on and automatic throttling of CPU frequency (which most modern processors do for power management reasons). cv::getTickFrequency() double cv::getTickFrequency( void );      // Tick frequency in seconds as 64-bit When using cv::getTickCount() for timing analysis, the exact meaning of a tick is, in general, architecture dependent. The function cv::getTickFrequency() computes the conversion between clock time (i.e., seconds) and abstract “ticks.””.  Thus to compute the time required for some specific thing to happen

throttling of CPU frequency (which most modern processors do for power management reasons). cv::getTickFrequency() double cv::getTickFrequency( void );      // Tick frequency in seconds as 64-bit When using cv::getTickCount() for timing analysis, the exact meaning of a tick is, in general, architecture dependent. The function cv::getTickFrequency() computes the conversion between clock time (i.e., seconds) and abstract “ticks.””.  Thus to compute the time required for some specific thing to happen (such as a function to execute), one need only call cv::getTickCount() before and after the function call, subtract the results, and divide by the value of cv::getTickFrequency(). cvIsInf() int cvIsInf( double x );              // return 1 if x is IEEE754 “infinity”          The return value of cvIsInf() is one if x is plus or minus infinity and zero otherwise. The infinity test is the test implied by the IEEE754 standard. cvIsNaN() int cvIsNan( double x );              // return 1 if x is IEEE754 “Not a number”

a function to execute), one need only call cv::getTickCount() before and after the function call, subtract the results, and divide by the value of cv::getTickFrequency(). cvIsInf() int cvIsInf( double x );              // return 1 if x is IEEE754 “infinity”          The return value of cvIsInf() is one if x is plus or minus infinity and zero otherwise. The infinity test is the test implied by the IEEE754 standard. cvIsNaN() int cvIsNan( double x );              // return 1 if x is IEEE754 “Not a number”  The return value of cvIsNaN() is one if x is “not a number” and zero otherwise. The NaN test is the test implied by the IEEE754 standard. cvRound()  int cvRound( double x );              // Return integer nearest to ‘x’ Given a floating-point number x, cvRound() computes the integer closest to x. If the input value is outside of the range representable by a 32-bit integer, the result is undefined. cv::setNumThreads() void cv::setNumThreads( int nthreads );   // Set number of threads OpenCV can use

When OpenCV is compiled with OpenMP support, this function sets the number of threads that OpenCV will use in parallel OpenMP regions. The default value for the number of threads is the number of logical cores on the CPU (i.e., if we have four cores each with two hyper-threads, there will be eight threads by default). If nthreads is set to 0, the number of threads will be returned to this default value. cv::setUseOptimized() void cv::setUseOptimized( bool on_off );  // If false, turn off optimized routines Though early versions of OpenCV relied on outside libraries (such as IPP, the Intel Performance Primitives library) for access to high performance optimizations such as SSE2 instructions, later versions have increasingly moved to containing that code in the OpenCV itself. By default the use of these optimized routines is enabled, unless you specifically disabled it when you built your installation of the library. However, you can turn the use of these optimizations on and off at any time with

routines Though early versions of OpenCV relied on outside libraries (such as IPP, the Intel Performance Primitives library) for access to high performance optimizations such as SSE2 instructions, later versions have increasingly moved to containing that code in the OpenCV itself. By default the use of these optimized routines is enabled, unless you specifically disabled it when you built your installation of the library. However, you can turn the use of these optimizations on and off at any time with cv::setUseOptimized(). The test of the global flag for optimizations usage is done at a relatively high level inside of the OpenCV library functions. The implication of this is that you should not call cv::setUseOptimized() while any other routines might be running (on any threads). You should make sure to call this routine when you can be certain you know what is and what is not running, preferably from the very top level of your application. cv::useOptimized() bool cv::useOptimized( void );            //

cv::setUseOptimized(). The test of the global flag for optimizations usage is done at a relatively high level inside of the OpenCV library functions. The implication of this is that you should not call cv::setUseOptimized() while any other routines might be running (on any threads). You should make sure to call this routine when you can be certain you know what is and what is not running, preferably from the very top level of your application. cv::useOptimized() bool cv::useOptimized( void );            // return true if optimizations are enabled At any time, you can check the state of the global flag, which enables the use of high performance optimizations (see cv::setUseOptimized()) by calling cv::useOptimized(). True will be returned only if these optimizations are currently enabled; otherwise, this function will return False. Objects That Do Stuff As the OpenCV library has evolved, it has become increasingly common to introduce new objects which encapsulate functionality that is too complicated to be

// return true if optimizations are enabled At any time, you can check the state of the global flag, which enables the use of high performance optimizations (see cv::setUseOptimized()) by calling cv::useOptimized(). True will be returned only if these optimizations are currently enabled; otherwise, this function will return False. Objects That Do Stuff As the OpenCV library has evolved, it has become increasingly common to introduce new objects which encapsulate functionality that is too complicated to be associated with a single function and which, if implemented as a set of functions, would cause the overall function space of the library to become too cluttered. As a result, new functionality is often represented by an associated new object type, which can be thought of as a “machine” that does whatever this function is. Most of these machines have an overloaded operator(), which officially makes them function objects or functors. If you are not familiar with this programming idiom, the important idea is

associated with a single function and which, if implemented as a set of functions, would cause the overall function space of the library to become too cluttered. As a result, new functionality is often represented by an associated new object type, which can be thought of as a “machine” that does whatever this function is. Most of these machines have an overloaded operator(), which officially makes them function objects or functors. If you are not familiar with this programming idiom, the important idea is that unlike “normal” functions, function objects are created, and can maintain state information inside them. As a result, they can be set up with whatever data or configuration they need, and they are “asked” to perform services through either common member functions, or by calling them as functions (usually via the overloaded operator()43). Principal Component Analysis (cv::PCA) Principal component analysis is the process of analyzing a distribution in many dimensions and extracting from that distribution

unlike “normal” functions, function objects are created, and can maintain state information inside them. As a result, they can be set up with whatever data or configuration they need, and they are “asked” to perform services through either common member functions, or by calling them as functions (usually via the overloaded operator()43). Principal Component Analysis (cv::PCA) Principal component analysis is the process of analyzing a distribution in many dimensions and extracting from that distribution the particular subset of dimensions that carry the most information. The dimensions computed by PCA are not necessarily the basis dimensions in which the distribution was originally specified. Indeed, one of the most important aspects of PCA is the ability to generate a new basis in which                                                              43 Here the word “usually” means “usually when people program function objects,” but does not turn out to mean “usually” for the OpenCV library. There is a

the particular subset of dimensions that carry the most information. The dimensions computed by PCA are not necessarily the basis dimensions in which the distribution was originally specified. Indeed, one of the most important aspects of PCA is the ability to generate a new basis in which                                                              43 Here the word “usually” means “usually when people program function objects,” but does not turn out to mean “usually” for the OpenCV library. There is a competing convention in the OpenCV library which uses the overloaded operator() to load the configuration, and a named member to provide the fundamental service of the object. This convention is substantially less canonical in general, but quite common in the OpenCV library.

the axes of the new basis can be ordered by their importance44. These basis vectors will turn out to be the eigenvectors of the covariance matrix for the distribution as a whole, and the corresponding eigenvalues will tell us about the extent of the distribution in that dimension. We are now in a position to explain why PCA is handled by one of these function objects. Given a distribution once, the PCA object can compute and retain this new basis. The big advantage of the new basis is that the basis vectors that correspond to the large eigenvalues carry most of the information about the objects in the distribution. Thus, without losing much accuracy, we can throw away the less informative dimensions. This dimension reduction is called a KLT Transform.45 Once you have loaded a sample distribution and the principal components are computed, you might want to use that information to do various things, such as applying the KLT transform to new vectors. By making the PCA functionality a function object, it can

that correspond to the large eigenvalues carry most of the information about the objects in the distribution. Thus, without losing much accuracy, we can throw away the less informative dimensions. This dimension reduction is called a KLT Transform.45 Once you have loaded a sample distribution and the principal components are computed, you might want to use that information to do various things, such as applying the KLT transform to new vectors. By making the PCA functionality a function object, it can “remember” what it needs to know about the distribution you gave it, and thereafter use that information to provide the “service” of transforming new vectors on demand.  Figure 3-4. (a) Input data is characterized by a Gaussian approximation; (b) the data is projected into the space implied by the eigenvectors of the covariance of that approximation; (c)the data is projected by the KLT projection to a space defined only by the most “useful” of the eigenvectors; superimposed: a new data point (white diamond) is

what it needs to know about the distribution you gave it, and thereafter use that information to provide the “service” of transforming new vectors on demand.  Figure 3-4. (a) Input data is characterized by a Gaussian approximation; (b) the data is projected into the space implied by the eigenvectors of the covariance of that approximation; (c)the data is projected by the KLT projection to a space defined only by the most “useful” of the eigenvectors; superimposed: a new data point (white diamond) is projected to the reduced dimension space by cv::PCA::project(); that same point is brought back to the original space (black diamond) by cv::PCA::backProject() cv::PCA::PCA() PCA::PCA();                                                              44 You might be thinking to yourself “hey, this sounds like machine learning, what is it doing in this chapter?”  This is not a bad question. In modern computer vision, machine learning is becoming increasingly intrinsic to an ever -growing list of algorithms. For this

to the reduced dimension space by cv::PCA::project(); that same point is brought back to the original space (black diamond) by cv::PCA::backProject() cv::PCA::PCA() PCA::PCA();                                                              44 You might be thinking to yourself “hey, this sounds like machine learning, what is it doing in this chapter?”  This is not a bad question. In modern computer vision, machine learning is becoming increasingly intrinsic to an ever -growing list of algorithms. For this reason, component capabilities, such as PCA and SVD, are increasingly considered “building blocks.” 45 KLT stands for “Karhunen-Loeve Transform,””, so that phrase “KLT Transformation” is a bit of a malapropism. It is however at least as often said one way as the other.

PCA::PCA(   cv::InputArray data,                    // Data, as rows or cols in 2d array   cv::InputArray mean,                    // average, if known, 1-by-n or n-by-1   int            flags,                   // Are vectors rows or cols of ‘data’   int            maxComponents = 0        // Max dimensions to retain );  The PCA object has a default constructor, cv::PCA(), which simply builds the PCA object and initializes the empty structure. The second form executes the default construction, then immediately proceeds to pass its arguments to PCA::operator()() (discussed next). cv::PCA::operator()() PCA::operator()(   cv::InputArray data,                    // Data, as rows or cols in 2d array   cv::InputArray mean,                    // average, if known, 1-by-n or n-by-1   int            flags,                   // Are vectors rows or cols of ‘data’   int            maxComponents = 0        // Max dimensions to retain );  The overloaded operator()() for PCA builds the model of the distribution inside of

proceeds to pass its arguments to PCA::operator()() (discussed next). cv::PCA::operator()() PCA::operator()(   cv::InputArray data,                    // Data, as rows or cols in 2d array   cv::InputArray mean,                    // average, if known, 1-by-n or n-by-1   int            flags,                   // Are vectors rows or cols of ‘data’   int            maxComponents = 0        // Max dimensions to retain );  The overloaded operator()() for PCA builds the model of the distribution inside of the PCA object. The argument data is an array containing all of the samples that constitute the distribution. Optionally, mean, a second array that contains the mean value in each dimension, can be supplied. (mean can either be  n-by-1 or 1-by- n.) The data can be arranged as an  n-by- D (n rows of samples each of  D dimensions) or  D-by- n array ( n columns of samples each of  D dimensions). The argument flags is currently used only to specify the arrangement of the data in data and mean. In particular, flags

PCA object. The argument data is an array containing all of the samples that constitute the distribution. Optionally, mean, a second array that contains the mean value in each dimension, can be supplied. (mean can either be  n-by-1 or 1-by- n.) The data can be arranged as an  n-by- D (n rows of samples each of  D dimensions) or  D-by- n array ( n columns of samples each of  D dimensions). The argument flags is currently used only to specify the arrangement of the data in data and mean. In particular, flags can be set either to cv::PCA_DATA_AS_ROW or cv::PCA_DATA_AS_COL, to indicate that either data is n-by- D and mean is  n-by- l or data is  D-by- n and mean is  l-by- n respectively. The final argument maxComponents specifies the maximum number of components (dimensions) that PCA should retain. By default, all of the components are retained. Any subsequent call to cv::PCA::operator()() will overwrite the internal representations of the eigenvectors and eigenvalues, so you can recycle a PCA object whenever

can be set either to cv::PCA_DATA_AS_ROW or cv::PCA_DATA_AS_COL, to indicate that either data is n-by- D and mean is  n-by- l or data is  D-by- n and mean is  l-by- n respectively. The final argument maxComponents specifies the maximum number of components (dimensions) that PCA should retain. By default, all of the components are retained. Any subsequent call to cv::PCA::operator()() will overwrite the internal representations of the eigenvectors and eigenvalues, so you can recycle a PCA object whenever you need to (i.e., you don’t have to reallocate a new one for each new distribution you want to handle if you no longer need the information about the previous distribution). cv::PCA::project() cv::Mat PCA::project(                     // Return results, as a 2d matrix   cv::InputArray  vec                     // points to project, rows or cols, 2d ) const;  void PCA::project(   cv::InputArray  vec                     // points to project, rows or cols, 2d   cv::OutputArray result                  // Result

need to (i.e., you don’t have to reallocate a new one for each new distribution you want to handle if you no longer need the information about the previous distribution). cv::PCA::project() cv::Mat PCA::project(                     // Return results, as a 2d matrix   cv::InputArray  vec                     // points to project, rows or cols, 2d ) const;  void PCA::project(   cv::InputArray  vec                     // points to project, rows or cols, 2d   cv::OutputArray result                  // Result of projection, reduced space ) const;  Once you have loaded your reference distribution with cv::PCA::operator()(), you can start asking the PCA object to do useful things for you like compute the KLT projection of some set of vectors onto the basis vectors computed by the principal component analysis. The function cv::PCA::project() has two forms, the first returns a matrix containing the results of the projections, while the second writes the results to a matrix you provide. The advantage of the first form

projection, reduced space ) const;  Once you have loaded your reference distribution with cv::PCA::operator()(), you can start asking the PCA object to do useful things for you like compute the KLT projection of some set of vectors onto the basis vectors computed by the principal component analysis. The function cv::PCA::project() has two forms, the first returns a matrix containing the results of the projections, while the second writes the results to a matrix you provide. The advantage of the first form is that you can use it in matrix expressions. The argument vec contains the input vectors. vecvec is required to have the same number of dimensions and the same “orientation” as the data array that was passed to PCA when the distribution was first analyzed (i.e., if your data was columns when you called cv::PCA::operator()(), vec should also have the data arranged into columns).

The returned array will have the same number of objects as vec with the same orientation, but the dimensionality of each object will be whatever was passed to maxComponents when the PCA object was first configured with cv::PCA::operator()(). cv::PCA::backProject() cv::Mat PCA::backProject(                 // Return results, as a 2d matrix   cv::InputArray  vec                     // Result of projection, reduced space } const;  void PCA::backProject(   cv::InputArray  vec                     // Result of projection, reduced space   cv::OutputArray result                  // “reconstructed” vectors, full dimension ) const;  The cv::PCA::backProject() function performs the reverse operation of cv::PCA::project(), with the analogous restrictions on the input and output arrays. The argument vec contains the input vectors, which this time are from the projected space. They will have the same number of dimensions as you specified with maxComponents when you configured the PCA object and the same “orientation” as

projection, reduced space   cv::OutputArray result                  // “reconstructed” vectors, full dimension ) const;  The cv::PCA::backProject() function performs the reverse operation of cv::PCA::project(), with the analogous restrictions on the input and output arrays. The argument vec contains the input vectors, which this time are from the projected space. They will have the same number of dimensions as you specified with maxComponents when you configured the PCA object and the same “orientation” as the data array that was passed to PCA when the distribution was first analyzed (i.e., if your data was columns when you called cv::PCA::operator(), vec should also have the data arranged into columns). The returned array will have the same number of objects as vec with the same orientation, but the dimensionality of each object will be the dimensionality of the original data you gave to the PCA object when the PCA object was first configured with cv::PCA::operator()()).  If you did not retain all of the

as the data array that was passed to PCA when the distribution was first analyzed (i.e., if your data was columns when you called cv::PCA::operator(), vec should also have the data arranged into columns). The returned array will have the same number of objects as vec with the same orientation, but the dimensionality of each object will be the dimensionality of the original data you gave to the PCA object when the PCA object was first configured with cv::PCA::operator()()).  If you did not retain all of the dimensions when you configured the PCA object in the beginning, the result of back projecting vectors, which are themselves projections of some vector   !x from the original data space, will not be equal to   !x. Of course, the difference should be small, even if the number of components retained was much smaller than the original dimension of   †!x, as this is the very point of using PCA in the first place. Singular Value Decomposition (cv::SVD) The class cv::SVD is similar to cv::PCA above in that it is

dimensions when you configured the PCA object in the beginning, the result of back projecting vectors, which are themselves projections of some vector   !x from the original data space, will not be equal to   !x. Of course, the difference should be small, even if the number of components retained was much smaller than the original dimension of   †!x, as this is the very point of using PCA in the first place. Singular Value Decomposition (cv::SVD) The class cv::SVD is similar to cv::PCA above in that it is the same kind of function object. Its purpose, however, is quite different. The singular value decomposition is essentially a tool for working with non-square, ill-conditioned, or otherwise poorly-behaved matrices such those encountered when solving under-determined linear systems. Mathematically, the singular value decomposition (SVD) is the decomposition of an  m-by- n matrix A into the form:    A=U!W!VT where W is a diagonal matrix and U and V are  m-by- n and  n-by- n (unitary) matrices. Of course, the

the same kind of function object. Its purpose, however, is quite different. The singular value decomposition is essentially a tool for working with non-square, ill-conditioned, or otherwise poorly-behaved matrices such those encountered when solving under-determined linear systems. Mathematically, the singular value decomposition (SVD) is the decomposition of an  m-by- n matrix A into the form:    A=U!W!VT where W is a diagonal matrix and U and V are  m-by- n and  n-by- n (unitary) matrices. Of course, the matrix W is also an  m-by- n matrix, so here “diagonal” means that any element whose row and column numbers are not equal is necessarily 0.  cv::SVD() SVD::SVD(); SVD::SVD(   cv::InputArray A,                       // Linear system, array to be decomposed   int            flags = 0                // what to construct, can A can scratch );

The SVD object has a default constructor, cv::SVD(), which simply builds the SVD object and initializes the empty structure. The second form basically executes the default construction, then immediately proceeds to pass its arguments to cv::SVD::operator()() (discussed next). cv::SVD::operator()() SVD::& SVD::operator() (   cv::InputArray A,                       // Linear system, array to be decomposed   int            flags = 0                // what to construct, can A be scratch );  The operator cv::SVD::operator()() passes to the cv::SVD object the matrix to be decomposed. The matrix A, as described earlier, is decomposed into a matrix U, a matrix V (actually the transpose of V, which we will call Vt), and a set of singular values (which are the diagonal elements of the matrix W). The flags can be any one of cv::SVD::MODIFY_A, cv::SVD::NO_UV, or cv::SVD::FULL_UV. The latter two are mutually exclusive, but either can be combined with the first. The flag cv::SVD::MODIFY_A indicates that it is OK to modify

passes to the cv::SVD object the matrix to be decomposed. The matrix A, as described earlier, is decomposed into a matrix U, a matrix V (actually the transpose of V, which we will call Vt), and a set of singular values (which are the diagonal elements of the matrix W). The flags can be any one of cv::SVD::MODIFY_A, cv::SVD::NO_UV, or cv::SVD::FULL_UV. The latter two are mutually exclusive, but either can be combined with the first. The flag cv::SVD::MODIFY_A indicates that it is OK to modify the matrix A when computing. This speeds up computation a bit and saves some memory. It is more important when the input matrix is already very large. The flag cv::SVD::NO_UV tells cv::SVD to not explicitly compute the matrices U and Vt, while the flag cv::SVD::FULL_UV indicates that not only would you like U and Vt computed, but that you would like them to be represented as full size square orthogonal matrices. cv::SVD::compute() void SVD::compute(   cv::InputArray  A,                      // Linear system, array to be

computing. This speeds up computation a bit and saves some memory. It is more important when the input matrix is already very large. The flag cv::SVD::NO_UV tells cv::SVD to not explicitly compute the matrices U and Vt, while the flag cv::SVD::FULL_UV indicates that not only would you like U and Vt computed, but that you would like them to be represented as full size square orthogonal matrices. cv::SVD::compute() void SVD::compute(   cv::InputArray  A,                      // Linear system, array to be decomposed   cv::OutputArray W,                      // Output array ‘W’, singular values   cv::OurputArray U,                      // Output array ‘U’, left singular vectors   cv::OutputArray Vt,                     // Output array ‘Vt’, right singular vectors   int             flags = 0               // what to construct, and if A can be scratch );  This function is an alternative to using cv::SVD::operator()() to decompose the matrix A. The primary difference is that the matrices W, U, and Vt are stored in

cv::OutputArray W,                      // Output array ‘W’, singular values   cv::OurputArray U,                      // Output array ‘U’, left singular vectors   cv::OutputArray Vt,                     // Output array ‘Vt’, right singular vectors   int             flags = 0               // what to construct, and if A can be scratch );  This function is an alternative to using cv::SVD::operator()() to decompose the matrix A. The primary difference is that the matrices W, U, and Vt are stored in the user-supplied arrays, rather than being kept internally. The flags supported are exactly those supported by cv::SVD::operator()(). cv::SVD::solveZ() void SVD::solveZ(   cv::InputArray  A,                      // Linear system, array to be decomposed   cv::OutputArray z                       // One possible solution (unit length) );        †!z=†argmin!x:!x=1A"!x

Given an under-determined (singular) linear system, cv::SVD::solveZ() will (attempt to) find a unit length solution of     A!"x=0 and place the solution in the array y. Because the linear system is singular, however, it may have no solution, or it may have an infinite family of solutions. cv::SVD::solveZ() will find a solution, if one exists. If no solution exists, then the return value   !y will be a vector that minimizes    †A!"x, even if this is not, in fact, zero. cv::SVD::backSubst() void SVD::backSubst(   cv::InputArray  b,                      // Right-hand side of linear system   cv::OutputArray x                       // Found solution to linear system );   void SVD::backSubst(   cv::InputArray  W,                      // Output array ‘W’, singular values   cv::InputArray  U,                      // Output array ‘U’, left singular vectors

cv::InputArray  Vt,                     // Output array ‘Vt’, right singular vectors   cv::InputArray  b,                      // Right hand side of linear system   cv::OutputArray x                       // Found solution to linear system );  Assuming that the matrix A has been previously passed to the cv::SVD object (and thus decomposed into U, W, and Vt), the first form of cv::SVD::backSubst() attempts to solve the system :    UWVT()!"x="b
. The second form does the same thing, but expects the matrices W, U, and Vt to be passed to it as arguments. The actual method of computing dst is to evaluate the following expression:      x†!"!=VtT#diagW()−1#UT#b†!"!~A−1#b†!"!.

This method produces a pseudo-solution for an over-determined system, which is the best solution in the sense of minimizing the least-squares error.46 Of course, it will also exactly solve a correctly determined linear system. In practice, it is relatively rare that you would want to use cv::SVD::backSubst() directly. This is because you can do precisely the same thing by calling cv::solve() and passing the cv::DECOMP_SVD method flag—which is a lot easier. It is only in the less common case in which you need to solve many different systems with the same left-hand side (x) that you would be better off calling cv::SVD::backSubst() directly. (As opposed to solving the same system many times with different righthand sides (b), which you might just as well do with cv::solve(). Random Number Generator (cv::RNG) The random number generator object RNG holds the state of a pseudorandom sequence that generates random numbers. The benefit of doing things this way is that you can conveniently maintain multiple streams

to solve many different systems with the same left-hand side (x) that you would be better off calling cv::SVD::backSubst() directly. (As opposed to solving the same system many times with different righthand sides (b), which you might just as well do with cv::solve(). Random Number Generator (cv::RNG) The random number generator object RNG holds the state of a pseudorandom sequence that generates random numbers. The benefit of doing things this way is that you can conveniently maintain multiple streams of pseudorandom numbers.  When programming large systems, it is a good programming practice to use separate random number streams in different modules of the code. In this way, the removal of one module does not change the behavior of the streams in the other modules. Once created, the random number generator provides the “service” of generating random numbers on demand, drawn from either a uniform or a Gaussian distribution. The generator uses the Multiply with Carry algorithm (G. Marsaglia) for uniform

of pseudorandom numbers.  When programming large systems, it is a good programming practice to use separate random number streams in different modules of the code. In this way, the removal of one module does not change the behavior of the streams in the other modules. Once created, the random number generator provides the “service” of generating random numbers on demand, drawn from either a uniform or a Gaussian distribution. The generator uses the Multiply with Carry algorithm (G. Marsaglia) for uniform distributions and the Ziggurat algorithm (G. Marsaglia and W. W. Tsang) for the generation of numbers from a Gaussian distribution. cv::theRNG() cv::RNG& theRNG( void );                  // Return a random number generator  The function cv::theRNG() returns the default random number generator for the thread from which it was called. OpenCV automatically creates one instance of cv::RNG for each thread in execution. This is the same random number generator that is implicitly accessed by functions like

distributions and the Ziggurat algorithm (G. Marsaglia and W. W. Tsang) for the generation of numbers from a Gaussian distribution. cv::theRNG() cv::RNG& theRNG( void );                  // Return a random number generator  The function cv::theRNG() returns the default random number generator for the thread from which it was called. OpenCV automatically creates one instance of cv::RNG for each thread in execution. This is the same random number generator that is implicitly accessed by functions like cv::randu() or cv::randn(). Those functions are convenient and just as quick if you want a single number, or to initialize an array, but if you have a loop of your own that needs to generate a lot of random numbers, you are better off grabbing a reference to a random number generator (in this case, the default generator, but                                                              46 The object 𝑑𝑖𝑎𝑔𝑾!𝟏 is a matrix whose diagonal elements 𝜆!∗  are defined in terms of the diagonal elements 𝜆!  of 𝑾 by 𝜆!∗=𝜆!!!

or cv::randn(). Those functions are convenient and just as quick if you want a single number, or to initialize an array, but if you have a loop of your own that needs to generate a lot of random numbers, you are better off grabbing a reference to a random number generator (in this case, the default generator, but                                                              46 The object 𝑑𝑖𝑎𝑔𝑾!𝟏 is a matrix whose diagonal elements 𝜆!∗  are defined in terms of the diagonal elements 𝜆!  of 𝑾 by 𝜆!∗=𝜆!!! for  𝜆!≥𝜀. This value ε is the singularity threshold, a very small number that is typically proportional to the sum of the diagonal elements of 𝑾 (i.e.,  𝜀!𝜆!!).

you could use your own instead) and using RNG::operator T() to get your random numbers (more on that operator follows). cv::RNG() cv::RNG::RNG( void ); cv::RNG::RNG( uint64 state );             // create using the seed ‘state’  You can create an RNG object with either the default constructor, or you can pass it a 64-bit unsigned integer that it will use as the seed of the random number sequence. If you call the default constructor (or pass 0 to the second variation) the generator will initialize with a standardized value.47 cv::RNG::operator T() cv::RNG::operator uchar(); cv::RNG::operator schar(); cv::RNG::operator ushort(); cv::RNG::operator short int(); cv::RNG::operator int(); cv::RNG::operator unsigned(); cv::RNG::operator float(); cv::RNG::operator double(); This is really a set of different methods that return a new random number from cv::RNG of some specific type. Each of these is an overloaded cast operator, so in effect you cast the RNG object to whatever type you want: Example 3-2: Using the

a standardized value.47 cv::RNG::operator T() cv::RNG::operator uchar(); cv::RNG::operator schar(); cv::RNG::operator ushort(); cv::RNG::operator short int(); cv::RNG::operator int(); cv::RNG::operator unsigned(); cv::RNG::operator float(); cv::RNG::operator double(); This is really a set of different methods that return a new random number from cv::RNG of some specific type. Each of these is an overloaded cast operator, so in effect you cast the RNG object to whatever type you want: Example 3-2: Using the default random number generator we generate a pair of integers and a pair of floating-point numbers; the style of the cast operation is up to you, this example shows both the int(x) and the (int)x forms cv::RNG rng = cv::theRNG(); cout << ”An integer:      “ << (int)rng   << endl; cout << ”Another integer: “ << int(rng)   << endl; cout << ”A float:         “ << (float)rng << endl; cout << ”Another float:   “ << float(rng) << endl; When integer types are generated, they will be generated across the entire

default random number generator we generate a pair of integers and a pair of floating-point numbers; the style of the cast operation is up to you, this example shows both the int(x) and the (int)x forms cv::RNG rng = cv::theRNG(); cout << ”An integer:      “ << (int)rng   << endl; cout << ”Another integer: “ << int(rng)   << endl; cout << ”A float:         “ << (float)rng << endl; cout << ”Another float:   “ << float(rng) << endl; When integer types are generated, they will be generated across the entire range of available values (using the MWC algorithm described earlier and thus uniformly). When floating-point types are generated, they will always be in the range from the interval [0.0, 1.0).48 cv::RNG::operator() unsigned int cv::RNG::operator()();       // Return random value from 0-UINT_MAX unsigned int cv::RNG::operator()( unsigned int N );  // Return value from 0-(N-1) When generating integer random numbers, the overloaded operator()() allows a convenient way to just grab another one. In essence,

range of available values (using the MWC algorithm described earlier and thus uniformly). When floating-point types are generated, they will always be in the range from the interval [0.0, 1.0).48 cv::RNG::operator() unsigned int cv::RNG::operator()();       // Return random value from 0-UINT_MAX unsigned int cv::RNG::operator()( unsigned int N );  // Return value from 0-(N-1) When generating integer random numbers, the overloaded operator()() allows a convenient way to just grab another one. In essence, calling my_rng() is equivalent to calling (unsigned int)my_rng. The somewhat more interesting form of cv::RNG::operator()() takes an integer argument N. This form returns a random unsigned integer modulo N (using the MWC algorithm described earlier and thus uniformly). Thus, the range of integers returned by my_rng( N ) is then the range of integers from 0 to N-1. cv::RNG::uniform() int        cv::RNG::uniform( int a,    int b    );       // Return value from a-(b-1)

my_rng() is equivalent to calling (unsigned int)my_rng. The somewhat more interesting form of cv::RNG::operator()() takes an integer argument N. This form returns a random unsigned integer modulo N (using the MWC algorithm described earlier and thus uniformly). Thus, the range of integers returned by my_rng( N ) is then the range of integers from 0 to N-1. cv::RNG::uniform() int        cv::RNG::uniform( int a,    int b    );       // Return value from a-(b-1)                                                               47 This “standard value” is not zero because, for that value, many random number generators, including the ones used by RNG, will return nothing but zeros thereafter. Currently, this standard value is  2!"−1. 48 In case this notation is not familiar, the designation of an interval using square brackets, [], ‘[‘ indicates that this limit is inclusive, and the designation using parentheses, (), ‘(‘ indicates that this limit is noninclusiveinclusive. Thus the notation [0.0,1.0) means an interval

47 This “standard value” is not zero because, for that value, many random number generators, including the ones used by RNG, will return nothing but zeros thereafter. Currently, this standard value is  2!"−1. 48 In case this notation is not familiar, the designation of an interval using square brackets, [], ‘[‘ indicates that this limit is inclusive, and the designation using parentheses, (), ‘(‘ indicates that this limit is noninclusiveinclusive. Thus the notation [0.0,1.0) means an interval from 0.0 to 1.0 inclusive of 0.0 but not inclusive of 1.0.

float  cv::RNG::uniform( float a,  float b  );       // Return value in range [a,b)  double cv::RNG::uniform( double a, double b );       // Return value in range [a,b)  This function allows you to generate a random number uniformly (using the MWC algorithm) in the interval [a, b). It is important to note that the C++ compiler does not consider the return value of a function when determining which of multiple similar forms to use, only the arguments. As a result, if you call: float x = my_rng.uniform(0,1) you are going to get 0.f, because 0 and 1 are integers and the only integer in the interval [0, 1) is 0. If you want a floating-point number, you should use something like my_rng.uniform(0.f,1.f), and for a double, use my_rng.uniform(0.,1.). Of course, explicit casting of the arguments also works. cv::RNG::gaussian() double  cv::RNG::gaussian( double sigma ); // Gaussian number, zero mean, std-dev=’sigma’ This function allows you to generate a random number from a zero-mean Gaussian distribution (using the

you are going to get 0.f, because 0 and 1 are integers and the only integer in the interval [0, 1) is 0. If you want a floating-point number, you should use something like my_rng.uniform(0.f,1.f), and for a double, use my_rng.uniform(0.,1.). Of course, explicit casting of the arguments also works. cv::RNG::gaussian() double  cv::RNG::gaussian( double sigma ); // Gaussian number, zero mean, std-dev=’sigma’ This function allows you to generate a random number from a zero-mean Gaussian distribution (using the Ziggurat algorithm) with standard deviation sigma. cv::RNG::fill() void  cv::RNG::fill(    InputOutputArray mat,                   // Input array, values will be overwritten   int              distType,              // Type of distribution (Gaussian or uniform)   InputArray       a,                     // min (uniform) or mean (Gaussian)   InputArray       b                      // max (uniform) or std-deviation (Gaussian)  ); The cv::RNG::fill() algorithm fills a matrix mat of up to four channels with

the Ziggurat algorithm) with standard deviation sigma. cv::RNG::fill() void  cv::RNG::fill(    InputOutputArray mat,                   // Input array, values will be overwritten   int              distType,              // Type of distribution (Gaussian or uniform)   InputArray       a,                     // min (uniform) or mean (Gaussian)   InputArray       b                      // max (uniform) or std-deviation (Gaussian)  ); The cv::RNG::fill() algorithm fills a matrix mat of up to four channels with random numbers drawn from a specific distribution. That distribution is selected by the distType argument, which can be either cv::RNG::UNIFORM or cv::RNG::NORMAL. In the case of the uniform distribution, each element of mat will be filled with a random value generated from the interval  †matc,i∈ac,bc⎡⎣). In the case of the Gaussian (cv::RNG::NORMAL) distribution, each element is generated from a distribution with mean taken from a and standard deviation taken from b:  †matc,i∈Nac,bc(). It is important to

random numbers drawn from a specific distribution. That distribution is selected by the distType argument, which can be either cv::RNG::UNIFORM or cv::RNG::NORMAL. In the case of the uniform distribution, each element of mat will be filled with a random value generated from the interval  †matc,i∈ac,bc⎡⎣). In the case of the Gaussian (cv::RNG::NORMAL) distribution, each element is generated from a distribution with mean taken from a and standard deviation taken from b:  †matc,i∈Nac,bc(). It is important to note that the arrays a and b are not of the dimension of mat, they are  nc

is the number of channels in mat (i.e., there is not a separate distribution for each element of mat, a and b specify one distribution, not one distribution for every element of mat.) If you have a multichannel array, then you can generate individual entries in “channel-space” from a multivariate distribution simply by giving the appropriate mean and standard deviation for each channel in the input arrays a and b. This distribution, however, will be drawn from a distribution with only zero entries in the off-diagonal elements of its covariance matrix. (This is because each element is generated completely independently of the others.) If you need to draw from a more general distribution, the easiest method is to generate zero-mean values using an identity covariance matrix with cv::RNG::fill(), and then transform (translate and rotate) back to your original basis using cv::transform().  Summary In this chapter, we introduced some frequently encountered basic data structures. In particular, we looked at the

off-diagonal elements of its covariance matrix. (This is because each element is generated completely independently of the others.) If you need to draw from a more general distribution, the easiest method is to generate zero-mean values using an identity covariance matrix with cv::RNG::fill(), and then transform (translate and rotate) back to your original basis using cv::transform().  Summary In this chapter, we introduced some frequently encountered basic data structures. In particular, we looked at the all-important OpenCV array structure cv::Mat, which can contain matrices, images, and multidimensional arrays.

Exercises In the following exercises, you may need to refer to the manual at http://docs.opencv.org/ . 1. Find and open …/opencv/modules/core/include/opencv2/core/core.hpp. Read through and find the many conversion helper functions. a) Choose a negative floating-point number. Take its absolute value, round it, and then take its ceiling and floor.49 b) Generate some random numbers. c) Create a floating-point cv::Point2f and convert it to an integer cv::Point2i. d) Convert a cv::Point2i to a CvPoint2f. 2. This exercise will accustom you to the idea of many functions taking matrix types. Create a two-dimensional matrix with three channels of type byte with data size 100-by-100. Set all the values to 0. a) Draw a circle in the matrix using the cv::circle() function: void circle(    cv::Mat&          img,                  // Image to be drawn on   cv::Point         center,               // Location of circle center   int               radius,               // Radius of circle   const cv::Scalar& color,

exercise will accustom you to the idea of many functions taking matrix types. Create a two-dimensional matrix with three channels of type byte with data size 100-by-100. Set all the values to 0. a) Draw a circle in the matrix using the cv::circle() function: void circle(    cv::Mat&          img,                  // Image to be drawn on   cv::Point         center,               // Location of circle center   int               radius,               // Radius of circle   const cv::Scalar& color,                // Color, RGB form   int               thickness = 1,        // Thickness of line    int               lineType  = 8,        // Connectedness, 4 or 8   int               shift     = 0         // Bits of radius to treat as fraction ); b) Display this image using methods described in Chapter 2. 1. Create a two-dimensional matrix with three channels of type byte with data size 100-by-100, and set all the values to 0. Use the element access member: m.at<cv::Vec3f> to point to the middle (“green”) channel.

// Color, RGB form   int               thickness = 1,        // Thickness of line    int               lineType  = 8,        // Connectedness, 4 or 8   int               shift     = 0         // Bits of radius to treat as fraction ); b) Display this image using methods described in Chapter 2. 1. Create a two-dimensional matrix with three channels of type byte with data size 100-by-100, and set all the values to 0. Use the element access member: m.at<cv::Vec3f> to point to the middle (“green”) channel. Draw a green rectangle between (20, 5) and (40, 20). 2. Create a three-channel RGB image of size 100-by-100. Clear it. Use pointer arithmetic to draw a green square between (20, 5) and (40, 20). 3. Practice using the block access methods (Table 3-16). Create a 210-by-210 single-channel byte image and zero it. Within the image, build a pyramid of increasing values using the submatrix constructor and the cv::Range object. That is: the outer border should be 0, the next inner border should be 20, the next

Draw a green rectangle between (20, 5) and (40, 20). 2. Create a three-channel RGB image of size 100-by-100. Clear it. Use pointer arithmetic to draw a green square between (20, 5) and (40, 20). 3. Practice using the block access methods (Table 3-16). Create a 210-by-210 single-channel byte image and zero it. Within the image, build a pyramid of increasing values using the submatrix constructor and the cv::Range object. That is: the outer border should be 0, the next inner border should be 20, the next inner border should be 40, and so on until the final innermost square is set to value 200; all borders should be 10 pixels wide. Display the image. 4. Use multiple image objects for one image. Load an image that is at least 100-by-100. Create two additional image objects using the first object and the submatrix constructor. Create the new images with width at 20 and the height at 30, but with their origins located at pixel at (5, 10) and (50, 60), respectively. Logically invert the two images using the ‘not’

border should be 40, and so on until the final innermost square is set to value 200; all borders should be 10 pixels wide. Display the image. 4. Use multiple image objects for one image. Load an image that is at least 100-by-100. Create two additional image objects using the first object and the submatrix constructor. Create the new images with width at 20 and the height at 30, but with their origins located at pixel at (5, 10) and (50, 60), respectively. Logically invert the two images using the ‘not’ logical inversion operator. Display the loaded image, which should have two inverted rectangles within the larger image. 5. Add an CV_DbgAssert( condition ) to the code of question 4 that will be triggered by a condition in the program. Build the code in debug, run it and see the assert being triggered. Now build it in release mode and see that the condition is not triggered. 6. Create a mask using cv::compare(). Load a real image. Use cv::split() to split the image into red, green, and blue images. a) Find

inversion operator. Display the loaded image, which should have two inverted rectangles within the larger image. 5. Add an CV_DbgAssert( condition ) to the code of question 4 that will be triggered by a condition in the program. Build the code in debug, run it and see the assert being triggered. Now build it in release mode and see that the condition is not triggered. 6. Create a mask using cv::compare(). Load a real image. Use cv::split() to split the image into red, green, and blue images. a) Find and display the green image. b) Clone this green plane image twice (call these clone1 and clone2).                                                              49 Remember that cvFloor() and cvCeil() are legacy functions and are found in …/opencv/modules/core/include/opencv2/core/types_c.h.

c) Find the green plane’s minimum and maximum value. d) Set clone1’s values to thresh = (unsigned char)((maximum - minimum)/2.0). e) Set clone2 to 0 and use cv::compare(green_image, clone1, clone2, cv::CMP_GE). Now clone2 will have a mask of where the value exceeds thresh in the green image. f) Finally, compute the value:  green_image = green_image - thresh/2 and display the results. (Bonus: assign this value to green_image only where clone2 is non-zero.)

4 Graphical User Interface HighGUI: Portable Graphics Toolkit The OpenCV functions that allow us to interact with the operating system, the filesystem1, and hardware such as cameras are collected into a sub-library called HighGUI (which stands for “high-level graphical user interface”). HighGUI allows us to open windows, to display images, to read and write graphics-related files (both images and video), and to handle simple mouse, pointer, and keyboard events. We can also use it to create other useful doodads—like sliders, for example—and then add them to our windows. If you are a GUI guru in your window environment of choice, then you might find that much of what HighGUI offers is redundant. Yet, even so, you might find that the benefit of cross-platform portability is itself a tempting morsel. From our initial perspective, the HighGUI library in OpenCV can be divided into three parts: the hardware part, the filesystem part, and the GUI part. We will take a moment to overview what is in each part before

sliders, for example—and then add them to our windows. If you are a GUI guru in your window environment of choice, then you might find that much of what HighGUI offers is redundant. Yet, even so, you might find that the benefit of cross-platform portability is itself a tempting morsel. From our initial perspective, the HighGUI library in OpenCV can be divided into three parts: the hardware part, the filesystem part, and the GUI part. We will take a moment to overview what is in each part before we really dive in. The hardware part is primarily concerned with the operation of cameras. In most operating systems, interaction with a camera is a tedious and painful task. HighGUI allows an easy way to query a camera and retrieve its latest image. It hides all of the nasty stuff, and that keeps us happy. The filesystem part is concerned primarily with loading and saving images. One nice feature of the library is that it allows us to read video using the same methods we would use to read a camera. We can therefore

dive in. The hardware part is primarily concerned with the operation of cameras. In most operating systems, interaction with a camera is a tedious and painful task. HighGUI allows an easy way to query a camera and retrieve its latest image. It hides all of the nasty stuff, and that keeps us happy. The filesystem part is concerned primarily with loading and saving images. One nice feature of the library is that it allows us to read video using the same methods we would use to read a camera. We can therefore abstract ourselves away from the particular device we’re using and get on with writing interesting code. In a similar spirit, HighGUI provides us with a (relatively) universal pair of functions to load and save still images. These functions simply rely on the filename extension and automatically handle all of the decoding or encoding that is necessary. The third part of HighGUI is the window system (or GUI). The library provides some simple functions that allow us to open a window and throw an image into

abstract ourselves away from the particular device we’re using and get on with writing interesting code. In a similar spirit, HighGUI provides us with a (relatively) universal pair of functions to load and save still images. These functions simply rely on the filename extension and automatically handle all of the decoding or encoding that is necessary. The third part of HighGUI is the window system (or GUI). The library provides some simple functions that allow us to open a window and throw an image into that window. It also allows us to register and respond to mouse and keyboard events on that window. These features are most useful when trying to get off of the ground with a simple application. Tossing in some slider bars, we find ourselves able to prototype a surprising variety of applications using only the HighGUI library. If we want to link to Qt, we can even get a little more functionality.2                                                              1 Some lower level file system operations are

that window. It also allows us to register and respond to mouse and keyboard events on that window. These features are most useful when trying to get off of the ground with a simple application. Tossing in some slider bars, we find ourselves able to prototype a surprising variety of applications using only the HighGUI library. If we want to link to Qt, we can even get a little more functionality.2                                                              1 Some lower level file system operations are located in the core module as well. 2 This is Qt the cross-platform widget toolkit. We will talk more about how this works later in this chapter.

As we proceed in this chapter, we will not treat these three segments separately; rather, we will start with some functions of highest immediate utility and work our way to the subtler points thereafter. In this way, you will learn what you need to get going as soon as possible.  At the end of the chapter, we will consider additional functions that will allow us to mark up images by drawing lines or other shapes on them. These functions are primarily provided for debugging and validating code, but will turn out to be broadly useful. Finally, we will look at more advanced data persistence for OpenCV data types that will allow us to store and reload additional data types other than just images. Working with Image Files OpenCV provides special functions for the loading and saving of images that deal, either directly or implicitly, with the complexities associated with compression and decompression of that image data. These functions are different from the more universal XML/YML-based functions discussed in the

out to be broadly useful. Finally, we will look at more advanced data persistence for OpenCV data types that will allow us to store and reload additional data types other than just images. Working with Image Files OpenCV provides special functions for the loading and saving of images that deal, either directly or implicitly, with the complexities associated with compression and decompression of that image data. These functions are different from the more universal XML/YML-based functions discussed in the previous chapter in several respects. The primary distinction is that because these functions are designed for actual images, as opposed to general arrays of data, they rely heavily on existing backends for compression and decompression, handling all of the common file formats, each in the manner required by that file type. Most of these compression and decompression schemes have been developed with the idea that it is possible to lose some information without degrading the visual experience of the image.

previous chapter in several respects. The primary distinction is that because these functions are designed for actual images, as opposed to general arrays of data, they rely heavily on existing backends for compression and decompression, handling all of the common file formats, each in the manner required by that file type. Most of these compression and decompression schemes have been developed with the idea that it is possible to lose some information without degrading the visual experience of the image. Clearly such lossy compression schemes are not a good idea for arrays of nonimage data. Less obviously, artifacts introduced by lossy compressions schemes can also cause headaches for computer vision algorithms as well. In many cases, algorithms will find and respond to compression artifacts that are completely invisible to us humans. The key difference to remember is that the loading and saving functions we will discuss here are really an interface to the resources for handling image files that are already

such lossy compression schemes are not a good idea for arrays of nonimage data. Less obviously, artifacts introduced by lossy compressions schemes can also cause headaches for computer vision algorithms as well. In many cases, algorithms will find and respond to compression artifacts that are completely invisible to us humans. The key difference to remember is that the loading and saving functions we will discuss here are really an interface to the resources for handling image files that are already present in your operating system or its available libraries. The XML/YML data persistence mentioned here is entirely intrinsic to OpenCV. Loading and Saving Images The most common tasks we will need to accomplish are the loading and the saving of files from disk. The easiest way to do this is with the high-level functions cv::imread() and cv::imwrite(). These functions handle the complete task of decompression and compression as well as the actual interaction with the filesystem. Reading Files with cv::imread()

in your operating system or its available libraries. The XML/YML data persistence mentioned here is entirely intrinsic to OpenCV. Loading and Saving Images The most common tasks we will need to accomplish are the loading and the saving of files from disk. The easiest way to do this is with the high-level functions cv::imread() and cv::imwrite(). These functions handle the complete task of decompression and compression as well as the actual interaction with the filesystem. Reading Files with cv::imread() Obviously the first thing to do is to learn how to get an image out of the filesystem and into our program. The function that does this is cv::imread(): cv::Mat cv::imread(   const string& filename,                        // Input filename   int           flags   = cv::LOAD_IMAGE_COLOR   // Flags set how to interpret file ); When opening an image, cv::imread() does not look at the file extension. Instead, cv::imread() analyzes the first few bytes of the file (aka its signature or “magic sequence”) and

Obviously the first thing to do is to learn how to get an image out of the filesystem and into our program. The function that does this is cv::imread(): cv::Mat cv::imread(   const string& filename,                        // Input filename   int           flags   = cv::LOAD_IMAGE_COLOR   // Flags set how to interpret file ); When opening an image, cv::imread() does not look at the file extension. Instead, cv::imread() analyzes the first few bytes of the file (aka its signature or “magic sequence”) and determines the appropriate codec using that. The second argument flags can be set to one of several values. By default, flags is set to cv::IMREAD_COLOR. This value indicates that images are to be loaded as three-channel images with 8 bits per channel. In this case, even if the image is actually grayscale in the file, the resulting image in memory will still have three channels, with all of the channels containing identical information. Alternatively, if flags is set to cv::IMREAD_GRAYSCALE, the image will be

the appropriate codec using that. The second argument flags can be set to one of several values. By default, flags is set to cv::IMREAD_COLOR. This value indicates that images are to be loaded as three-channel images with 8 bits per channel. In this case, even if the image is actually grayscale in the file, the resulting image in memory will still have three channels, with all of the channels containing identical information. Alternatively, if flags is set to cv::IMREAD_GRAYSCALE, the image will be loaded as grayscale, regardless of the number of channels in the file. The final option is to set flags to

cv::IMREAD_ANYCOLOR. In this case, the image will be loaded “as is,” with the result being three-channel if the file is color, and one-channel if the file is grayscale.3  In addition to the color-related flags, cv::imread() supports the flag cv::IMREAD_ANYDEPTH, which indicates that if an input image channels have more than 8 bits, that it should be loaded without conversion (i.e., the allocated array will be of the type indicated in the file.) Table 4-1: Parameters accepted by cv::imread() Parameter ID Meaning Default  cv::IMREAD_COLOR Always load to three-channel array.  yes  cv::IMREAD_GRAYSCALE Always load to single-channel array.  no  cv::IMREAD_ANYCOLOR Channels as indicated by file (up to three).            no  cv::IMREAD_ANYDEPTH Allow loading of more than 8-bit depth.  no  cv::IMREAD_UNCHANGED Equivalent to combining: cv::LOAD_IMAGE_ANYCOLOR |  cv::LOAD_IMAGE_UNCHANGED   no cv::imread() does not give a runtime error when it fails to load an image; it simply returns an empty cv::Mat (i.e.,

ID Meaning Default  cv::IMREAD_COLOR Always load to three-channel array.  yes  cv::IMREAD_GRAYSCALE Always load to single-channel array.  no  cv::IMREAD_ANYCOLOR Channels as indicated by file (up to three).            no  cv::IMREAD_ANYDEPTH Allow loading of more than 8-bit depth.  no  cv::IMREAD_UNCHANGED Equivalent to combining: cv::LOAD_IMAGE_ANYCOLOR |  cv::LOAD_IMAGE_UNCHANGED   no cv::imread() does not give a runtime error when it fails to load an image; it simply returns an empty cv::Mat (i.e., empty()==true).   Writing Files with cv::imwrite() The obvious complementary function to cv::imread() is cv::imwrite(), which takes three arguments: int cv::imwrite(   const string&      filename,            // Input filename   cv::InputArray     image,               // Image to write to file   const vector<int>& params  = vector<int>() // (Optional) for parameterized formats ); The first argument gives the filename, whose extension is used to determine the format in which the file will be stored. The second

Writing Files with cv::imwrite() The obvious complementary function to cv::imread() is cv::imwrite(), which takes three arguments: int cv::imwrite(   const string&      filename,            // Input filename   cv::InputArray     image,               // Image to write to file   const vector<int>& params  = vector<int>() // (Optional) for parameterized formats ); The first argument gives the filename, whose extension is used to determine the format in which the file will be stored. The second argument is the image to be stored. The third argument is used for parameters that are accepted by the particular file type being used for the write operation. The params argument expects an STL vector of integers, with those integers being a sequence of parameter IDs followed by the value to be assigned to that parameter (i.e., alternating between the parameter ID and the parameter value). For the parameter IDs, there are aliases provided by OpenCV, as listed in Table 4-2. Table 4-2: Parameters accepted by

is the image to be stored. The third argument is used for parameters that are accepted by the particular file type being used for the write operation. The params argument expects an STL vector of integers, with those integers being a sequence of parameter IDs followed by the value to be assigned to that parameter (i.e., alternating between the parameter ID and the parameter value). For the parameter IDs, there are aliases provided by OpenCV, as listed in Table 4-2. Table 4-2: Parameters accepted by cv::imwrite() Parameter ID Meaning Range Default cv::IMWRITE_JPG_QUALITY JPEG quality  0–100  95                                                              3 At the time of writing, “as is” does not support the loading of a fourth channel for those file types that support alpha channels. In this case, the fourth channel will currently just be ignored and the file will be treated as if it had only three channels.

cv::IMWRITE_PNG_COMPRESSION PNG compression (higher values mean more compression)  0-9  3 cv::IMWRITE_PXM_BINARY Use binary format for PPM, PGM, or PBM files              0 or 1  1 The cv::imwrite() function will store only 8-bit single- or three-channel images for most file formats. Backends for flexible image formats like PNG, TIFF, or JPEG 2000 allow storing 16-bit or even float formats and some allow four-channel images (BGR plus alpha) as well. The return value will be 1 if the save was successful and should be 0 if the save was not.4   A Note about Codecs Remember, however, that cv::imwrite() is intended for images, and relies heavily on software libraries intended for handling image file types. These libraries are generically referred to as codecs (“co-mpression and dec-ompression librarie-s”). Your operating system will likely have many codecs available, with (at least) one being available for each of the different common file types. OpenCV comes with the codecs you will need for some file formats

should be 0 if the save was not.4   A Note about Codecs Remember, however, that cv::imwrite() is intended for images, and relies heavily on software libraries intended for handling image file types. These libraries are generically referred to as codecs (“co-mpression and dec-ompression librarie-s”). Your operating system will likely have many codecs available, with (at least) one being available for each of the different common file types. OpenCV comes with the codecs you will need for some file formats (JPEG, PNG, and TIFF), and on Microsoft Windows as well as on Apple Mac OS X, these should be available on installation. On Linux and other Unix-like operating systems, OpenCV will try to use codecs supplied with the OS image. In this case, you will need to ensure you have installed the relevant packages with their development files (i.e., install libjpeg-dev in addition to libjpeg on Linux). In Apple’s Mac OS X, it is also possible to use the native image readers from Mac OS X. If you do this, however, you

PNG, and TIFF), and on Microsoft Windows as well as on Apple Mac OS X, these should be available on installation. On Linux and other Unix-like operating systems, OpenCV will try to use codecs supplied with the OS image. In this case, you will need to ensure you have installed the relevant packages with their development files (i.e., install libjpeg-dev in addition to libjpeg on Linux). In Apple’s Mac OS X, it is also possible to use the native image readers from Mac OS X. If you do this, however, you should be aware that Mac OS X has an embedded native color management, and so the colors that appear in the loaded file may be different than if you had used the codecs included with the OpenCV package. This is a good example of how the reliance on external elements by cv::imread() and cv::imwrite() can have important and sometimes unexpected consequences.  Compression and Decompression As already mentioned, the cv::imread() and cv::imwrite functions are high-level tools that handle a number of separate things

be aware that Mac OS X has an embedded native color management, and so the colors that appear in the loaded file may be different than if you had used the codecs included with the OpenCV package. This is a good example of how the reliance on external elements by cv::imread() and cv::imwrite() can have important and sometimes unexpected consequences.  Compression and Decompression As already mentioned, the cv::imread() and cv::imwrite functions are high-level tools that handle a number of separate things necessary to ultimately get your image written to or read from the disk. In practice, it is often useful to be able to handle some of those subcomponents individually and, in particular, to be able to compress or decompress an image in memory (using the codecs we just reviewed). Compressing Files with cv::imencode() Images can be compressed directly from OpenCV’s array types. In this case, the result will not be an array type, but rather a simple character buffer. This should not be surprising, as the

necessary to ultimately get your image written to or read from the disk. In practice, it is often useful to be able to handle some of those subcomponents individually and, in particular, to be able to compress or decompress an image in memory (using the codecs we just reviewed). Compressing Files with cv::imencode() Images can be compressed directly from OpenCV’s array types. In this case, the result will not be an array type, but rather a simple character buffer. This should not be surprising, as the resulting object is now in some format that is meaningful only to the codec that compressed it, and will (by construction) not be the same size as the original image. void cv::imencode(   const string&      ext,                 // Extension specifies codec   cv::InputArray     img,                 // Image to be encoded   vector<uchar>&     buf,                 // Encoded file bytes go here   const vector<int>& params  = vector<int>() // (Optional) for parameterized formats

object is now in some format that is meaningful only to the codec that compressed it, and will (by construction) not be the same size as the original image. void cv::imencode(   const string&      ext,                 // Extension specifies codec   cv::InputArray     img,                 // Image to be encoded   vector<uchar>&     buf,                 // Encoded file bytes go here   const vector<int>& params  = vector<int>() // (Optional) for parameterized formats                                                              4 The reason we say “should” is that, in some OS environments, it is possible to issue save commands that will actually cause the operating system to throw an exception. Normally, however, a zero value will be returned to indicate failure.

); The first argument to cv::imencode() is the file extension ext, represented as a string, which is associated with the desired compression. Of course, no file is actually written, but the extension is not only an intuitive way to refer to the desired format, it is the actual key used by most operating systems to index the available codecs. The next argument img is the image to be compressed, and the argument following that buf is the character array into which the compressed image will be placed. The final argument params is used to specify any parameters which may be required (or desired) for the specific compression codec to be used. The possible values for params are those listed earlier in Table 4-2 in the cv::imwrite(). Uncompressing Files with cv::imdecode() cv::Mat cv::imdecode(   cv::InputArray buf,                     // Encoded file bytes are here   int            flags = cv::LOAD_IMAGE_COLOR // Flags set how to interpret file ); Just as cv::imencode() allows us to compress an image into a

final argument params is used to specify any parameters which may be required (or desired) for the specific compression codec to be used. The possible values for params are those listed earlier in Table 4-2 in the cv::imwrite(). Uncompressing Files with cv::imdecode() cv::Mat cv::imdecode(   cv::InputArray buf,                     // Encoded file bytes are here   int            flags = cv::LOAD_IMAGE_COLOR // Flags set how to interpret file ); Just as cv::imencode() allows us to compress an image into a character buffer, cv::imdecode() allows us to decompress from a character buffer into an image array. cv::imdecode() takes just two arguments, the first being the buffer5 buf and the second being the flags arguments, which takes the same options as the flags used by cv::imread() (see Table 4-1). As was the case with cv::imread(), cv::imdecode() does not need a file extension (as cv::imencode() did) because it can deduce the correct codec to use from the first bytes of the compressed image in the buffer. Just

buffer, cv::imdecode() allows us to decompress from a character buffer into an image array. cv::imdecode() takes just two arguments, the first being the buffer5 buf and the second being the flags arguments, which takes the same options as the flags used by cv::imread() (see Table 4-1). As was the case with cv::imread(), cv::imdecode() does not need a file extension (as cv::imencode() did) because it can deduce the correct codec to use from the first bytes of the compressed image in the buffer. Just as cv::imread() returns an empty (cv::Mat::empty()==true) array if it cannot read the file it is given, cv::imdecode() returns an empty array if the buffer it is given is empty, contains invalid or unusable data, and so on. Working with Video When working with video, we must consider several functions, including (of course) how to read and write video files. You will also probably find yourself thinking shortly thereafter about how to actually play back such files on the screen—either for debugging, or as the

cv::imread() returns an empty (cv::Mat::empty()==true) array if it cannot read the file it is given, cv::imdecode() returns an empty array if the buffer it is given is empty, contains invalid or unusable data, and so on. Working with Video When working with video, we must consider several functions, including (of course) how to read and write video files. You will also probably find yourself thinking shortly thereafter about how to actually play back such files on the screen—either for debugging, or as the final output of our program; we’ll get to that next, Reading Video with the cv::VideoCapture Object The first thing we need is the cv::VideoCapture object. This is another one of those “objects that do stuff” we encountered in the previous chapter. This object contains the information needed for reading frames from a camera or video file. Depending on the source, we use one of three different calls to create a cv::VideoCapture object: cv::VideoCapture::VideoCapture(   const string&      filename,

final output of our program; we’ll get to that next, Reading Video with the cv::VideoCapture Object The first thing we need is the cv::VideoCapture object. This is another one of those “objects that do stuff” we encountered in the previous chapter. This object contains the information needed for reading frames from a camera or video file. Depending on the source, we use one of three different calls to create a cv::VideoCapture object: cv::VideoCapture::VideoCapture(   const string&      filename,            // Input filename ); cv::VideoCapture::VideoCapture(    int device                              // Video Capture device id ); cv::VideoCapture::VideoCapture(); In the case of the first constructor, we can simply give a filename for a video file (.MPG, .AVI, etc.) and OpenCV will open the file and prepare to read it. If the open is successful and we are able to start reading frames, the member function cv::VideoCapture::isOpen() will return true.

// Input filename ); cv::VideoCapture::VideoCapture(    int device                              // Video Capture device id ); cv::VideoCapture::VideoCapture(); In the case of the first constructor, we can simply give a filename for a video file (.MPG, .AVI, etc.) and OpenCV will open the file and prepare to read it. If the open is successful and we are able to start reading frames, the member function cv::VideoCapture::isOpen() will return true.                                                              5 You should not be surprised that the input buf is not type vector<int>&, as it was with cv::imencode(). Recall that the type cv::InputArray covers many possibilities and vector<> is one of them.

A lot of people don’t always check these sorts of things, thinking that nothing will go wrong. Don’t do that here. The returned value of cv::VideoCapture::isOpen() will be false if for some reason the file could not be opened (e.g., if the file does not exist), but that is not the only possible cause. The constructed object will also not be ready to be used if the codec with which the video is compressed is not known. As with the image codecs, you will need to have the appropriate library already residing on your computer in order to successfully read the video file. This is why it is always important for your code to check the return value of cv::VideoCapture::isOpen(), because even if it works on one machine (where the needed codec DLL or shared library is available), it might not work on another machine (where that codec is missing). Once we have an open cv::VideoCapture object, we can begin reading frames and do a number of other things. But before we get into that, let’s take a look at how to capture

on your computer in order to successfully read the video file. This is why it is always important for your code to check the return value of cv::VideoCapture::isOpen(), because even if it works on one machine (where the needed codec DLL or shared library is available), it might not work on another machine (where that codec is missing). Once we have an open cv::VideoCapture object, we can begin reading frames and do a number of other things. But before we get into that, let’s take a look at how to capture images from a camera. The variant of cv::VideoCapture::VideoCapture() that takes an integer device argument works very much like the string version we just discussed, except without the headache from the codecs.6 In this case, we give an identifier that indicates a camera we would like to access and how we expect the operating system to talk to that camera. For the camera, this is just an identification number that is zero (0) when we only have one camera and increments upward when there are multiple cameras

from a camera. The variant of cv::VideoCapture::VideoCapture() that takes an integer device argument works very much like the string version we just discussed, except without the headache from the codecs.6 In this case, we give an identifier that indicates a camera we would like to access and how we expect the operating system to talk to that camera. For the camera, this is just an identification number that is zero (0) when we only have one camera and increments upward when there are multiple cameras on the same system. The other part of the identifier is called the domain of the camera and indicates (in essence) what type of camera we have. The domain can be any of the predefined constants shown in Table 4-3. Table 4-3: Camera “domain” indicates where HighGUI should look for your camera Camera capture constant Numerical value  cv::CAP_ANY                0  cv::CAP_MIL                100  cv::CAP_VFW                200  cv::CAP_V4L                200  cv::CAP_V4L2                200  cv::CAP_FIREWIRE

same system. The other part of the identifier is called the domain of the camera and indicates (in essence) what type of camera we have. The domain can be any of the predefined constants shown in Table 4-3. Table 4-3: Camera “domain” indicates where HighGUI should look for your camera Camera capture constant Numerical value  cv::CAP_ANY                0  cv::CAP_MIL                100  cv::CAP_VFW                200  cv::CAP_V4L                200  cv::CAP_V4L2                200  cv::CAP_FIREWIRE                300  cv::CAP_IEEE1394                300  cv::CAP_DC1394                300  cv::CAP_CMU1394                300  cv::CAP_QT 500  cv::CAP_DSHOW 700  cv::CAP_PVAPI 800  cv::CAP_OPENNI 900  cv::CAP_ANDROID 1000 …  When we construct the device argument for cv::VideoCapture::VideoCapture(), we pass in an identifier that is just the sum of the domain index and the camera index. For example: cv::VideoCapture capture( cv::CAP_FIREWIRE ); In this example, cv::VideoCapture::VideoCapture() will attempt to open

300  cv::CAP_IEEE1394                300  cv::CAP_DC1394                300  cv::CAP_CMU1394                300  cv::CAP_QT 500  cv::CAP_DSHOW 700  cv::CAP_PVAPI 800  cv::CAP_OPENNI 900  cv::CAP_ANDROID 1000 …  When we construct the device argument for cv::VideoCapture::VideoCapture(), we pass in an identifier that is just the sum of the domain index and the camera index. For example: cv::VideoCapture capture( cv::CAP_FIREWIRE ); In this example, cv::VideoCapture::VideoCapture() will attempt to open the first (i.e., number-zero) FireWire camera. In most cases, the domain is unnecessary when we have only one camera; it is sufficient to use cv::CAP_ANY (which is conveniently equal to 0, so we don’t even have to type that in). One last useful hint before we move on: on some platforms, you can pass -1 to                                                              6 Of course, to be completely fair, we should probably confess that the headache caused by different codecs has been replaced by the analogous

the first (i.e., number-zero) FireWire camera. In most cases, the domain is unnecessary when we have only one camera; it is sufficient to use cv::CAP_ANY (which is conveniently equal to 0, so we don’t even have to type that in). One last useful hint before we move on: on some platforms, you can pass -1 to                                                              6 Of course, to be completely fair, we should probably confess that the headache caused by different codecs has been replaced by the analogous headache of determining which cameras are (or are not) supported on our system.

cv::VideoCapture::VideoCapture(), which will cause OpenCV to open a window that allows you to select the desired camera. Your last option is to create the capture object without providing any information about what is to be opened. cv::VideoCapture cap(); cap.open( “my_video.avi” ); In this case, the capture object will be there, but not ready for use until you explicitly open the source you want to read from. This is done with the cv::VideoCapture::open() method which, like the cv::VideoCapture constructor, can take either an STL string or a device ID as arguments. In either case, cv::VideoCapture::open() will have exactly the same effect as calling the cv::VideoCapture constructor with the same argument. Reading Frames with cv::VideoCapture::read() bool cv::VideoCapture::read(   cv::Mat& image                          // Image into which to read data ); Once you have a cv::VideoCapture object, you can start reading frames. The easiest way to do this is to call cv::VideoCapture::read(), which will simply

can take either an STL string or a device ID as arguments. In either case, cv::VideoCapture::open() will have exactly the same effect as calling the cv::VideoCapture constructor with the same argument. Reading Frames with cv::VideoCapture::read() bool cv::VideoCapture::read(   cv::Mat& image                          // Image into which to read data ); Once you have a cv::VideoCapture object, you can start reading frames. The easiest way to do this is to call cv::VideoCapture::read(), which will simply go to the open file represented by cv::VideoCapture and get the next frame, inserting that frame into the provided array image. This action will automatically “advance” the video capture object such that a subsequent call to cv::VideoCapture::read() will return the next frame, and so on. If the read was not successful (e.g., if you have reached the end of your file), then this function call will return false (otherwise it will return true). Similarly, the array object you supplied to the function will also be

to the open file represented by cv::VideoCapture and get the next frame, inserting that frame into the provided array image. This action will automatically “advance” the video capture object such that a subsequent call to cv::VideoCapture::read() will return the next frame, and so on. If the read was not successful (e.g., if you have reached the end of your file), then this function call will return false (otherwise it will return true). Similarly, the array object you supplied to the function will also be empty. Reading Frames with cv::VideoCapture::operator>>() cv::VideoCapture& cv::VideoCapture::operator>>(   cv::Mat& image                          // Image into which to read data ); In addition to using the read method of cv::VideoCapture, you can use the overloaded function cv::VideoCapture::operator>>() (i.e., the “stream read” operator) to read the next frame from your video capture object. In this case, cv::VideoCapture::operator>>() behaves exactly the same as cv::VideoCapture::read(), except that

empty. Reading Frames with cv::VideoCapture::operator>>() cv::VideoCapture& cv::VideoCapture::operator>>(   cv::Mat& image                          // Image into which to read data ); In addition to using the read method of cv::VideoCapture, you can use the overloaded function cv::VideoCapture::operator>>() (i.e., the “stream read” operator) to read the next frame from your video capture object. In this case, cv::VideoCapture::operator>>() behaves exactly the same as cv::VideoCapture::read(), except that because it is a stream operator, it returns a reference to the original cv::VideoCapture::read() object, whether or not it succeeded. In this case, you must check that the return array is not empty.  Reading Frames with cv::VideoCapture::grab() and cv::VideoCapture::retrieve() Instead of taking images one at a time from your camera or video source and decoding them as you read them, it is possible to break this process down into a grab phase, which is a little more than a memory copy, and a retrieve phase,

because it is a stream operator, it returns a reference to the original cv::VideoCapture::read() object, whether or not it succeeded. In this case, you must check that the return array is not empty.  Reading Frames with cv::VideoCapture::grab() and cv::VideoCapture::retrieve() Instead of taking images one at a time from your camera or video source and decoding them as you read them, it is possible to break this process down into a grab phase, which is a little more than a memory copy, and a retrieve phase, which handles the actual decoding of the grabbed data. bool cv::VideoCapture::grab( void ); bool cv::VideoCapture::retrieve(   cv::Mat& image,                         // Image into which to read data   int      channel = 0                    // Used for multihead devices ); The cv::VideoCapture::grab() function copies the currently available image to an internal buffer that is invisible to the user. Why would you want OpenCV to put the frame somewhere you can’t access it? The answer is that this grabbed

which handles the actual decoding of the grabbed data. bool cv::VideoCapture::grab( void ); bool cv::VideoCapture::retrieve(   cv::Mat& image,                         // Image into which to read data   int      channel = 0                    // Used for multihead devices ); The cv::VideoCapture::grab() function copies the currently available image to an internal buffer that is invisible to the user. Why would you want OpenCV to put the frame somewhere you can’t access it? The answer is that this grabbed frame is unprocessed, and grab() is designed simply to get it onto the computer (typically from a camera) as quickly as possible.  There are many reasons to grab and retrieve separately rather than together as would be the case in calling cv::VideoCapture::read(). The most common situation

arises when there are multiple cameras (e.g., with stereo imaging). In this case, it is important to have frames that are separated in time by the minimum amount possible (ideally they would be simultaneous for stereo imaging). Therefore, it makes the most sense to first grab all the frames and then come back and decode them after you have them all safely in your buffers.  As was the case with cv::VideoCapture::read(), cv::VideoCapture::grab()returns true only if the grab was successful. Once you have grabbed your frame, you can call cv::VideoCapture::retrieve(), which handles the decoding and the allocation and copying necessary to return the frame to you as a cv::Mat array. The function cv::VideoCapture::retrieve() functions analogously to cv::VideoCapture::read(), except that it operates from the internal buffer to which cv::VideoCapture::grab() copies frames. The other important difference between cv::VideoCapture::read() and cv::VideoCapture::retrieve() is the additional argument channel. The channel

your frame, you can call cv::VideoCapture::retrieve(), which handles the decoding and the allocation and copying necessary to return the frame to you as a cv::Mat array. The function cv::VideoCapture::retrieve() functions analogously to cv::VideoCapture::read(), except that it operates from the internal buffer to which cv::VideoCapture::grab() copies frames. The other important difference between cv::VideoCapture::read() and cv::VideoCapture::retrieve() is the additional argument channel. The channel argument is used when the device being accessed natively has multiple “heads” (i.e., multiple imagers). This is typically the case for devices designed specifically to be stereo imagers, as well as slightly more exotic devices such as the Kinect.7 In these cases, the value of channel will indicate which image from the device is to be retrieved. In these cases, you would call cv::VideoCapture::grab() just once and then call cv::VideoCapture::retrieve() as many times as needed to retrieve all of the images in the

is used when the device being accessed natively has multiple “heads” (i.e., multiple imagers). This is typically the case for devices designed specifically to be stereo imagers, as well as slightly more exotic devices such as the Kinect.7 In these cases, the value of channel will indicate which image from the device is to be retrieved. In these cases, you would call cv::VideoCapture::grab() just once and then call cv::VideoCapture::retrieve() as many times as needed to retrieve all of the images in the camera (each time with a different value for channel). Camera Properties: cv::VideoCapture::get() and cv::VideoCapture::set() Video files contain not only the video frames themselves, but also important metadata, which can be essential for handling the files correctly. When a video file is opened, that information is copied into the cv::VideoCapture object’s internal data area. It is very common to want to read that information from the cv::VideoCapture object, and sometimes also useful to write to that data

(each time with a different value for channel). Camera Properties: cv::VideoCapture::get() and cv::VideoCapture::set() Video files contain not only the video frames themselves, but also important metadata, which can be essential for handling the files correctly. When a video file is opened, that information is copied into the cv::VideoCapture object’s internal data area. It is very common to want to read that information from the cv::VideoCapture object, and sometimes also useful to write to that data area ourselves. The cv::VideoCapture::get() and cv::VideoCapture::set() functions allow us to perform these two operations: double cv::VideoCapture::get(   int    propid                           // Property identifier (see Table 4-4) );  bool cv::VideoCapture::set(   int    propid                           // Property identifier (see Table 4-4)   double value                            // Value to which to set the property ); The routine cv::VideoCapture::get() accepts any of the property IDs shown in Table

ourselves. The cv::VideoCapture::get() and cv::VideoCapture::set() functions allow us to perform these two operations: double cv::VideoCapture::get(   int    propid                           // Property identifier (see Table 4-4) );  bool cv::VideoCapture::set(   int    propid                           // Property identifier (see Table 4-4)   double value                            // Value to which to set the property ); The routine cv::VideoCapture::get() accepts any of the property IDs shown in Table 4-48. Table 4-4: Video capture properties used by cv::VideoCapture::get() and cv::VideoCapture::set() Video capture property Camera Only Meaning cv::CAP_PROP_POS_MSEC   Current position in video file (milliseconds) or video                                                              7 The currently supported multihead cameras are Kinect and Videre; others may be added later. 8 It should be understood that not all of the properties recognized by OpenCV will be recognized or handled by the “backend” behind the

Table 4-4: Video capture properties used by cv::VideoCapture::get() and cv::VideoCapture::set() Video capture property Camera Only Meaning cv::CAP_PROP_POS_MSEC   Current position in video file (milliseconds) or video                                                              7 The currently supported multihead cameras are Kinect and Videre; others may be added later. 8 It should be understood that not all of the properties recognized by OpenCV will be recognized or handled by the “backend” behind the capture. For example, the capture mechanisms operating behind the scenes on Android, Firewire on Linux (via dc1394), Quicktime, or a Kinect (via OpenNI) are all going to be very different, and not all of them will offer all of the services implied by this long list of options. Expect this list to grow as well, as new system types make new options possible.

capture timestamp cv::CAP_PROP_POS_FRAMES                   Zero-based index of next frame  cv::CAP_PROP_POS_AVI_RATIO                   Relative position in the video (range is 0.0 to 1.0) cv::CAP_PROP_FRAME_WIDTH                   Width of frames in the video cv::CAP_PROP_FRAME_HEIGHT                   Height of frames in the video cv::CAP_PROP_FPS   Frame rate at which the video was recorded cv::CAP_PROP_FOURCC   Four character code indicating codec cv::CAP_PROP_FRAME_COUNT                   Total number of frames in a video file cv::CAP_PROP_FORMAT  Format of the Mat objects returned (e.g., CV::U8C3) cv::CAP_PROP_MODE  Indicates capture mode, values are specific to video backend being used (i.e., DC1394, etc.) cv::CAP_PROP_BRIGHTNESS ü Brightness setting for camera (when supported) cv::CAP_PROP_CONTRAST ü Contrast setting for camera (when supported) cv::CAP_PROP_SATURATION ü Saturation setting for camera (when supported) cv::CAP_PROP_HUE ü Hue setting for camera (when supported) cv::CAP_PROP_GAIN ü

of frames in a video file cv::CAP_PROP_FORMAT  Format of the Mat objects returned (e.g., CV::U8C3) cv::CAP_PROP_MODE  Indicates capture mode, values are specific to video backend being used (i.e., DC1394, etc.) cv::CAP_PROP_BRIGHTNESS ü Brightness setting for camera (when supported) cv::CAP_PROP_CONTRAST ü Contrast setting for camera (when supported) cv::CAP_PROP_SATURATION ü Saturation setting for camera (when supported) cv::CAP_PROP_HUE ü Hue setting for camera (when supported) cv::CAP_PROP_GAIN ü Gain setting for camera (when supported) cv::CAP_PROP_EXPOSURE ü Exposure setting for camera (when supported) cv::CAP_PROP_CONVERT_RGB ü If nonzero, captured images will be converted to have three channels cv::CAP_PROP_WHITE_BALANCE ü White balance setting for camera (when supported) cv::CAP_PROP_RECTIFICATION ü Rectification flag for stereo cameras (DC1394-2.x only) Most of these properties are self-explanatory. POS_MSEC is the current position in a video file, measured in milliseconds. POS_FRAME is the

setting for camera (when supported) cv::CAP_PROP_EXPOSURE ü Exposure setting for camera (when supported) cv::CAP_PROP_CONVERT_RGB ü If nonzero, captured images will be converted to have three channels cv::CAP_PROP_WHITE_BALANCE ü White balance setting for camera (when supported) cv::CAP_PROP_RECTIFICATION ü Rectification flag for stereo cameras (DC1394-2.x only) Most of these properties are self-explanatory. POS_MSEC is the current position in a video file, measured in milliseconds. POS_FRAME is the current position in frame number. POS_AVI_RATIO is the position given as a number between 0.0 and 1.0. (This is actually quite useful when you want to position a trackbar to allow folks to navigate around your video.) FRAME_WIDTH and FRAME_HEIGHT are the dimensions of the individual frames of the video to be read (or to be captured at the camera’s current settings). FPS is specific to video files and indicates the number of frames per second at which the video was captured; you will need to know this if you

position in frame number. POS_AVI_RATIO is the position given as a number between 0.0 and 1.0. (This is actually quite useful when you want to position a trackbar to allow folks to navigate around your video.) FRAME_WIDTH and FRAME_HEIGHT are the dimensions of the individual frames of the video to be read (or to be captured at the camera’s current settings). FPS is specific to video files and indicates the number of frames per second at which the video was captured; you will need to know this if you want to play back your video and have it come out at the right speed. FOURCC is the four-character code for the compression codec to be used for the video you are currently reading (more on these shortly). FRAME_COUNT should be the total number of frames in the video, but this figure is not entirely reliable. All of these values are returned as type double, which is perfectly reasonable except for the case of FOURCC (FourCC) [FourCC85]. Here you will have to recast the result in order to interpret it, as shown in

play back your video and have it come out at the right speed. FOURCC is the four-character code for the compression codec to be used for the video you are currently reading (more on these shortly). FRAME_COUNT should be the total number of frames in the video, but this figure is not entirely reliable. All of these values are returned as type double, which is perfectly reasonable except for the case of FOURCC (FourCC) [FourCC85]. Here you will have to recast the result in order to interpret it, as shown in Example 4-1. Example 4-1. Unpacking a four-character code to identify a video codec cv::VideoCapture cap( "my_video.avi" ); double           f      = cap.get( cv::CAP_PROP_FOURCC ); char*            fourcc = (char*) (&f);  For each of these video capture properties, there is a corresponding cv::VideoCapture::set() function that will attempt to set the property. These are not all meaningful things to do; for example, you should not be setting the FOURCC of a video you are currently reading. Attempting to

in Example 4-1. Example 4-1. Unpacking a four-character code to identify a video codec cv::VideoCapture cap( "my_video.avi" ); double           f      = cap.get( cv::CAP_PROP_FOURCC ); char*            fourcc = (char*) (&f);  For each of these video capture properties, there is a corresponding cv::VideoCapture::set() function that will attempt to set the property. These are not all meaningful things to do; for example, you should not be setting the FOURCC of a video you are currently reading. Attempting to move around the video by setting one of the position properties will sometimes work, but only for some video codecs (we’ll have more to say about video codecs in the next section).

Writing Video with the cv::VideoWriter Object The other thing we might want to do with video is writing it out to disk. OpenCV makes this easy; it is essentially the same as reading video but with a few extra details. Just as we did with the cv::VideoCapture device for reading, we must first create a cv::VideoWriter device before we can write out our video. The video writer has two constructors; one is a simple default constructor that just creates an uninitialized video object that we will have to open later, and the other has all of the arguments necessary to actually set up the writer. cv::VideoWriter::VideoWriter(   const string& filename,                 // Input filename   int           fourcc,                   // codec, use CV_FOUR_CC() macro   double        fps,                      // Frame rate (storred in output file)   cv::Size      frame_size,               // Size of individual images   bool          is_color  = true          // if false, you can pass gray frames   ); You will notice that the

and the other has all of the arguments necessary to actually set up the writer. cv::VideoWriter::VideoWriter(   const string& filename,                 // Input filename   int           fourcc,                   // codec, use CV_FOUR_CC() macro   double        fps,                      // Frame rate (storred in output file)   cv::Size      frame_size,               // Size of individual images   bool          is_color  = true          // if false, you can pass gray frames   ); You will notice that the video writer requires a few extra arguments relative to the video reader. In addition to the filename, we have to tell the writer what codec to use, what the frame rate is, and how big the frames will be. Optionally, we can tell OpenCV that the frames are already in color (i.e., three-channel). If you set isColor to false, you can pass grayscale frames and the video writer will handle them correctly. As with the video reader, you can also create the video writer with a default constructor, and then configure

writer requires a few extra arguments relative to the video reader. In addition to the filename, we have to tell the writer what codec to use, what the frame rate is, and how big the frames will be. Optionally, we can tell OpenCV that the frames are already in color (i.e., three-channel). If you set isColor to false, you can pass grayscale frames and the video writer will handle them correctly. As with the video reader, you can also create the video writer with a default constructor, and then configure the writer with the cv::VideoWriter::open() method, which takes the same arguments as the full constructor. cv::VideoWriter out(); out.open(    "my_video.mpg",   CV_FOUR_CC(‘D’,’I’,’V’,’X’),  // MPEG-4 codec   30.0,                         // fps   cv::Size( 640, 480 ),   true                          // expect only color frames ); Here, the codec is indicated by its four-character code. (For those of you who are not experts in compression codecs, they all have a unique four-character identifier associated

the writer with the cv::VideoWriter::open() method, which takes the same arguments as the full constructor. cv::VideoWriter out(); out.open(    "my_video.mpg",   CV_FOUR_CC(‘D’,’I’,’V’,’X’),  // MPEG-4 codec   30.0,                         // fps   cv::Size( 640, 480 ),   true                          // expect only color frames ); Here, the codec is indicated by its four-character code. (For those of you who are not experts in compression codecs, they all have a unique four-character identifier associated with them). In this case, the int that is named fourcc in the argument list for cv::VideoWriter::VideoWriter() is actually the four characters of the fourcc packed together. Since this comes up relatively often, OpenCV provides a convenient macro CV_FOURCC(c0,c1,c2,c3) that will do the bit packing for you. Once you have given your video writer all the information it needs to set itself up, it is always a good idea to ask it if it is ready to go. This is done with the cv::VideoWriter::isOpened() method,

with them). In this case, the int that is named fourcc in the argument list for cv::VideoWriter::VideoWriter() is actually the four characters of the fourcc packed together. Since this comes up relatively often, OpenCV provides a convenient macro CV_FOURCC(c0,c1,c2,c3) that will do the bit packing for you. Once you have given your video writer all the information it needs to set itself up, it is always a good idea to ask it if it is ready to go. This is done with the cv::VideoWriter::isOpened() method, which will return true if you are good to go. If it returns false, this could mean that you don’t have write permission to the directory for the file you indicated, or (most often) that the codec you specified is not available. The codecs available to you will depend on your operating system installation and the additional libraries you have installed. For portable code, it is especially important to be able to gracefully handle the case in which the desired codec is not available on any particular machine.

will return true if you are good to go. If it returns false, this could mean that you don’t have write permission to the directory for the file you indicated, or (most often) that the codec you specified is not available. The codecs available to you will depend on your operating system installation and the additional libraries you have installed. For portable code, it is especially important to be able to gracefully handle the case in which the desired codec is not available on any particular machine. Writing Frames with cv::VideoWriter::write() Once you have verified that your video writer is ready to write, you can write frames by simply passing your array to the writer: cv::VideoWriter::write(   const Mat& image                        // Image to write as next frame );

This image must have the same size as the size you gave to the writer when you configured it in the first place. If you told the writer that the frames would be in color, this must also be a three-channel image. If you indicated to the writer (via isColor) that the images is just one channel, you should supply a single-channel (grayscale) image. Writing Frames with cv::VideoWriter::operator<<() The video writer also supports the idiom of the overloaded output stream operator (operator<<()). In this case, once you have your video writer open, you can write images to the video stream in the same manner you would write to cout or a file ofstream object: my_video_writer << my_frame; Working with Windows The HighGUI toolkit provides some rudimentary built-in features for creating windows, displaying images in those windows, and for making some user interaction possible with those windows. The native OpenCV graphical user interface (GUI) functions have been part of the library for a very long time, and have the

once you have your video writer open, you can write images to the video stream in the same manner you would write to cout or a file ofstream object: my_video_writer << my_frame; Working with Windows The HighGUI toolkit provides some rudimentary built-in features for creating windows, displaying images in those windows, and for making some user interaction possible with those windows. The native OpenCV graphical user interface (GUI) functions have been part of the library for a very long time, and have the advantage of being stable, portable,9 and very easy to use.  On the other hand, the native GUI features have the disadvantage of being not particularly complete. As a result, there has been an effort to modernize the GUI portion of HighGUI, and to add a number of useful new features. This was done by converting from “native” interfaces to the use of Qt. Qt is a cross-platform toolkit, and so new features can be implemented only once in the library, rather than once each for each of the native platforms.

advantage of being stable, portable,9 and very easy to use.  On the other hand, the native GUI features have the disadvantage of being not particularly complete. As a result, there has been an effort to modernize the GUI portion of HighGUI, and to add a number of useful new features. This was done by converting from “native” interfaces to the use of Qt. Qt is a cross-platform toolkit, and so new features can be implemented only once in the library, rather than once each for each of the native platforms. Needless to say, this has the result of making development of the Qt interface more attractive, and so it does more stuff, and will probably grow in the future, leaving the features of the native interface to become static legacy code. In this section, we will first take a look at the native functions, and then move on to look at the differences, and particularly the new features, offered by the Qt-based interface. HighGUI Native Graphical User Interface This section describes the core interface functions

Needless to say, this has the result of making development of the Qt interface more attractive, and so it does more stuff, and will probably grow in the future, leaving the features of the native interface to become static legacy code. In this section, we will first take a look at the native functions, and then move on to look at the differences, and particularly the new features, offered by the Qt-based interface. HighGUI Native Graphical User Interface This section describes the core interface functions that are part of OpenCV and require no external toolkit support. Some of them, when you are using the Qt backend, will also behave somewhat differently or have some additional options, but we will put those details off until the next section. The native HighGUI user input tools support only two basic interactions. Specifically, mouse clicks on the image area can be responded to, and a simple track bar can be added. These basic functions are usually sufficient for basic mock-ups and debugging, but hardly

that are part of OpenCV and require no external toolkit support. Some of them, when you are using the Qt backend, will also behave somewhat differently or have some additional options, but we will put those details off until the next section. The native HighGUI user input tools support only two basic interactions. Specifically, mouse clicks on the image area can be responded to, and a simple track bar can be added. These basic functions are usually sufficient for basic mock-ups and debugging, but hardly ideal for end-user facing applications. For that, you will (at least) want to use the Qt-based interface or some other UI toolkit. The main advantages of the native tools are that they are fast and easy to use, and don’t require you to install any additional libraries.  Creating a Window with cv::namedWindow() First, we want to be able to create a window and show an image on the screen using HighGUI. The function that does the first part for us is cv::namedWindow(). The function expects a name for the new

ideal for end-user facing applications. For that, you will (at least) want to use the Qt-based interface or some other UI toolkit. The main advantages of the native tools are that they are fast and easy to use, and don’t require you to install any additional libraries.  Creating a Window with cv::namedWindow() First, we want to be able to create a window and show an image on the screen using HighGUI. The function that does the first part for us is cv::namedWindow(). The function expects a name for the new window and one flag. The name appears at the top of the window, and the name is also used as a handle for                                                              9 They are “portable” because they make use of native window GUI tools on various platforms. This means X11 on Linux, Cocoa on MacOSX, and the raw Win32 API on Microsoft Windows machines. However, this portability only extends to those platforms for which there is an implementation in the library. There exist platforms on which OpenCV can be

window and one flag. The name appears at the top of the window, and the name is also used as a handle for                                                              9 They are “portable” because they make use of native window GUI tools on various platforms. This means X11 on Linux, Cocoa on MacOSX, and the raw Win32 API on Microsoft Windows machines. However, this portability only extends to those platforms for which there is an implementation in the library. There exist platforms on which OpenCV can be used for which there are no available implementations of the HighGUI library (e.g., Android).

the window that can be passed to other HighGUI functions.10 The flag argument indicates if the window should autosize itself to fit an image we put into it. Here is the full prototype: int cv::namedWindow(   const string&  name,                    // Handle used to identify window   int            flags = 0                // Used to tell window to autosize ); For now, the only valid options available for flags are to set it to 0 (it’s the default value), which indicates that users are to be able (and required) to resize the window, or to set it to cv::WINDOW_AUTOSIZE.11 If cv::WINDOW_AUTOSIZE is set, then HighGUI resizes the window to fit automatically whenever a new image is loaded, but users cannot resize the window. Once we create a window, we usually want to put something inside it. But before we do that, let’s see how to get rid of the window when it is no longer needed. For this, we use cv::destroyWindow(), a function whose only argument is a string: the name given to the window when it was created.

to resize the window, or to set it to cv::WINDOW_AUTOSIZE.11 If cv::WINDOW_AUTOSIZE is set, then HighGUI resizes the window to fit automatically whenever a new image is loaded, but users cannot resize the window. Once we create a window, we usually want to put something inside it. But before we do that, let’s see how to get rid of the window when it is no longer needed. For this, we use cv::destroyWindow(), a function whose only argument is a string: the name given to the window when it was created.  int cv::destroyWindow(   const string&  name,                    // Handle used to identify window ); Drawing an Image with cv::imshow() Now we are ready for what we really want to do, and that is to load an image and to put it into the window where we can view it and appreciate its profundity. We do this via one simple function, cv::imshow(): void cv::imshow(   const string&  name,                    // Handle used to identify window   cv::InputArray image                    // Image to display in window ); The

const string&  name,                    // Handle used to identify window ); Drawing an Image with cv::imshow() Now we are ready for what we really want to do, and that is to load an image and to put it into the window where we can view it and appreciate its profundity. We do this via one simple function, cv::imshow(): void cv::imshow(   const string&  name,                    // Handle used to identify window   cv::InputArray image                    // Image to display in window ); The first argument is the name of the window within which we intend to draw. The second argument is the image to be drawn. Updating a Window and cv::waitKey() The function cv::waitKey() has two functions. The first is the one from which it derives its name, which is to wait for some specified (possibly indefinite) amount of time for a key-press on the keyboard, and to return that key value when it is received. cv::waitKey() accepts a key-press from any open OpenCV window (but will not function if no such window exists). int

is the name of the window within which we intend to draw. The second argument is the image to be drawn. Updating a Window and cv::waitKey() The function cv::waitKey() has two functions. The first is the one from which it derives its name, which is to wait for some specified (possibly indefinite) amount of time for a key-press on the keyboard, and to return that key value when it is received. cv::waitKey() accepts a key-press from any open OpenCV window (but will not function if no such window exists). int cv::waitKey(   int delay = 0                           // Milliseconds until giving up (0=’never’) ); cv::waitKey() takes a single argument delay, which is the amount of time (in milliseconds) which it will wait for a key-press before returning automatically. If the delay is set to 0, cv::waitKey() will wait indefinitely for a key-press. If no key-press comes before delay milliseconds has passed, cv::waitKey() will return –1. The second, less obvious function of cv::waitKey() is that it provides an

int cv::waitKey(   int delay = 0                           // Milliseconds until giving up (0=’never’) ); cv::waitKey() takes a single argument delay, which is the amount of time (in milliseconds) which it will wait for a key-press before returning automatically. If the delay is set to 0, cv::waitKey() will wait indefinitely for a key-press. If no key-press comes before delay milliseconds has passed, cv::waitKey() will return –1. The second, less obvious function of cv::waitKey() is that it provides an opportunity for any open OpenCV window to be updated. This means that if you do not call cv::waitKey(), your image may                                                              10 In OpenCV, windows are referenced by name instead of by some unfriendly (and invariably OS-dependent) “handle.” Conversion between handles and names happens under the hood of HighGUI, so you needn’t worry about it. 11 Later in this chapter, we will look at the (optional) Qt-based backend for HighGUI. If you are using that backend,

for any open OpenCV window to be updated. This means that if you do not call cv::waitKey(), your image may                                                              10 In OpenCV, windows are referenced by name instead of by some unfriendly (and invariably OS-dependent) “handle.” Conversion between handles and names happens under the hood of HighGUI, so you needn’t worry about it. 11 Later in this chapter, we will look at the (optional) Qt-based backend for HighGUI. If you are using that backend, there are more options available for cv::namedWindow() as well as other functions.

never be drawn in your window, or your window may behave strangely (and badly) when moved, resized, or uncovered.12  An Example Displaying an Image Let’s now put together a simple program that will display an image on the screen. We can read a filename from the command line, create a window, and put our image in the window in 15 lines (including comments!). This program will display our image as long as we want to sit and appreciate it, and then exit when the ESC-key (ASCII value of 27) is pressed. Example 4-2. A simple example of creating a window and displaying an image in that window int main( int argc, char** argv ) {    // Create a named window with the name of the file   cv::namedWindow( argv[1], 1 );    // Load the image from the given filename   cv::Mat = cv::imread( argv[1] );    // Show the image in the named window   cv::imshow( argv[1], img );    // Idle until the user hits the "Esc" key   while( true ) {     if( cv::waitKey( 100 ) == 27 ) break;   }    // Clean up and don't be piggies

Example 4-2. A simple example of creating a window and displaying an image in that window int main( int argc, char** argv ) {    // Create a named window with the name of the file   cv::namedWindow( argv[1], 1 );    // Load the image from the given filename   cv::Mat = cv::imread( argv[1] );    // Show the image in the named window   cv::imshow( argv[1], img );    // Idle until the user hits the "Esc" key   while( true ) {     if( cv::waitKey( 100 ) == 27 ) break;   }    // Clean up and don't be piggies   cv::destroyWindow( argv[1] );    exit(0); } For convenience, we have used the filename as the window name. This is nice because OpenCV automatically puts the window name at the top of the window, so we can tell which file we are viewing (see Figure 4-1). Easy as cake.                                                              12 What this sentence really means is that cv::waitKey() is the only function in HighGUI that can fetch and handle events. This means that if it is not called periodically, no normal

argv[1] );    exit(0); } For convenience, we have used the filename as the window name. This is nice because OpenCV automatically puts the window name at the top of the window, so we can tell which file we are viewing (see Figure 4-1). Easy as cake.                                                              12 What this sentence really means is that cv::waitKey() is the only function in HighGUI that can fetch and handle events. This means that if it is not called periodically, no normal event processing will take place. As a corollary to this, if HighGUI is being used within an environment that takes care of event processing, then you may not need to call cv::waitKey().

Figure 4-1: A simple image displayed with cv::imshow() Before we move on, there are a few other window-related functions you ought to know about. They are: void cv::moveWindow( const char* name, int x, int y ); void cv::destroyAllWindows( void ); int  cv::startWindowThread( void ); cv::moveWindow() simply moves a window on the screen so that its upper-left corner is positioned at x,y. cv::destroyAllWindows() is a useful cleanup function that closes all of the windows and de-allocates the associated memory. On Linux and MacOS, cv::startWindowThread() tries to start a thread that updates the window automatically and handles resizing and so forth. A return value of 0 indicates that no thread could be started—for example, because there is no support for this feature in the version of OpenCV that you are using. Note that, if you do not start a separate window thread, OpenCV can react to user interface actions only when it is explicitly given time to do so (this happens when your program invokes cv::waitKey()).

On Linux and MacOS, cv::startWindowThread() tries to start a thread that updates the window automatically and handles resizing and so forth. A return value of 0 indicates that no thread could be started—for example, because there is no support for this feature in the version of OpenCV that you are using. Note that, if you do not start a separate window thread, OpenCV can react to user interface actions only when it is explicitly given time to do so (this happens when your program invokes cv::waitKey()). Mouse Events Now that we can display an image to a user, we might also want to allow the user to interact with the image we have created. Since we are working in a window environment and since we already learned how to capture single keystrokes with cv::waitKey(), the next logical thing to consider is how to “listen to” and respond to mouse events. Unlike keyboard events, mouse events are handled by a more typical callback mechanism. This means that, to enable response to mouse clicks, we must first write a

Events Now that we can display an image to a user, we might also want to allow the user to interact with the image we have created. Since we are working in a window environment and since we already learned how to capture single keystrokes with cv::waitKey(), the next logical thing to consider is how to “listen to” and respond to mouse events. Unlike keyboard events, mouse events are handled by a more typical callback mechanism. This means that, to enable response to mouse clicks, we must first write a callback routine that OpenCV can call whenever a mouse event occurs. Once we have done that, we must register the callback with OpenCV, thereby informing OpenCV that this is the correct function to use whenever the user does something with the mouse over a particular window. Let’s start with the callback. For those of you who are a little rusty on your event-driven program lingo, the callback can be any function that takes the correct set of arguments and returns the correct type. Here, we must be able to tell

routine that OpenCV can call whenever a mouse event occurs. Once we have done that, we must register the callback with OpenCV, thereby informing OpenCV that this is the correct function to use whenever the user does something with the mouse over a particular window. Let’s start with the callback. For those of you who are a little rusty on your event-driven program lingo, the callback can be any function that takes the correct set of arguments and returns the correct type. Here, we must be able to tell the function to be used as a callback exactly what kind of event occurred and where it occurred. The function must also be told if the user was pressing such keys as Shift or Alt when the mouse event occurred. A pointer to such a function is called a cv::MouseCallback. Here is the exact prototype that your callback function must match: void your_mouse_callback(   int   event,                            // Event type (see Table 4-5)   int   x,                                // x-location of mouse event   int

function to be used as a callback exactly what kind of event occurred and where it occurred. The function must also be told if the user was pressing such keys as Shift or Alt when the mouse event occurred. A pointer to such a function is called a cv::MouseCallback. Here is the exact prototype that your callback function must match: void your_mouse_callback(   int   event,                            // Event type (see Table 4-5)   int   x,                                // x-location of mouse event   int   y,                                // y-location of mouse event   int   flags,                            // More details on event (see Table 4-6)

void* param                             // Parameters from cv::setMouseCallback() ); Now, whenever your function is called, OpenCV will fill in the arguments with their appropriate values. The first argument, called the event, will have one of the values shown in Table 4-5. Table 4-5: Mouse event types Event Numerical value  cv::EVENT_MOUSEMOVE  0  cv::EVENT_LBUTTONDOWN  1  cv::EVENT_RBUTTONDOWN  2  cv::EVENT_MBUTTONDOWN  3  cv::EVENT_LBUTTONUP  4  cv::EVENT_RBUTTONUP  5  cv::EVENT_MBUTTONUP  6  cv::EVENT_LBUTTONDBLCLK                  7  cv::EVENT_RBUTTONDBLCLK                  8  cv::EVENT_MBUTTONDBLCLK                  9  The second and third arguments will be set to the x and y coordinates of the mouse event. It is worth noting that these coordinates represent the pixel coordinates in the image independent of the size of the window.13 The fourth argument, called flags, is a bit field in which individual bits indicate special conditions present at the time of the event. For example,

6  cv::EVENT_LBUTTONDBLCLK                  7  cv::EVENT_RBUTTONDBLCLK                  8  cv::EVENT_MBUTTONDBLCLK                  9  The second and third arguments will be set to the x and y coordinates of the mouse event. It is worth noting that these coordinates represent the pixel coordinates in the image independent of the size of the window.13 The fourth argument, called flags, is a bit field in which individual bits indicate special conditions present at the time of the event. For example, cv::EVENT_FLAG_SHIFTKEY has a numerical value of 16 (i.e., the fifth bit, or 1<<4) and so, if we wanted to test whether the shift key were down, we could simply compute the bitwise AND of flags & cv::EVENT_FLAG_SHIFTKEY. Table 4-6 shows a complete list of the flags. Table 4-6: Mouse event flags Flag Numerical value  cv::EVENT_FLAG_LBUTTON                  1  cv::EVENT_FLAG_RBUTTON                  2  cv::EVENT_FLAG_MBUTTON                  4  cv::EVENT_FLAG_CTRLKEY                  8  cv::EVENT_FLAG_SHIFTKEY

has a numerical value of 16 (i.e., the fifth bit, or 1<<4) and so, if we wanted to test whether the shift key were down, we could simply compute the bitwise AND of flags & cv::EVENT_FLAG_SHIFTKEY. Table 4-6 shows a complete list of the flags. Table 4-6: Mouse event flags Flag Numerical value  cv::EVENT_FLAG_LBUTTON                  1  cv::EVENT_FLAG_RBUTTON                  2  cv::EVENT_FLAG_MBUTTON                  4  cv::EVENT_FLAG_CTRLKEY                  8  cv::EVENT_FLAG_SHIFTKEY                  16  cv::EVENT_FLAG_ALTKEY  32 The final argument is a void pointer that can be used to have OpenCV pass additional information in the form of a pointer to whatever kind of structure you need.14                                                              13 In general, this is not the same as the pixel coordinates of the event that would be returned by the OS. This is because OpenCV is concerned with telling you where in the image the event happened, not in the window (to which the OS typically references mouse

32 The final argument is a void pointer that can be used to have OpenCV pass additional information in the form of a pointer to whatever kind of structure you need.14                                                              13 In general, this is not the same as the pixel coordinates of the event that would be returned by the OS. This is because OpenCV is concerned with telling you where in the image the event happened, not in the window (to which the OS typically references mouse event coordinates). 14 A common situation in which you will want to use the param argument is when the callback itself is a static member function of a class. In this case, you will probably find yourself wanting to pass the this pointer and so indicate which class object instance the callback is intended to affect.

Next, we need the function that registers the callback. That function is called cv::setMouseCallback(), and it requires three arguments. void cv::setMouseCallback(   const string&     windowName,           // Handle used to identify window   cv::MouseCallback on_mouse,             // Callback function   void*             param     = NULL      // Additional parameters for callback fn. ); The first argument is the name of the window to which the callback will be attached. Only events in that particular window will trigger this specific callback. The second argument is your callback function. Finally, the third param argument allows us to specify the param information that should be given to the callback whenever it is executed. This is, of course, the same param we were just discussing with the callback prototype. In Example 4-3, we write a small program to draw boxes on the screen with the mouse. The function my_mouse_callback() responds to mouse events, and it uses the event to determine what to do when it

will trigger this specific callback. The second argument is your callback function. Finally, the third param argument allows us to specify the param information that should be given to the callback whenever it is executed. This is, of course, the same param we were just discussing with the callback prototype. In Example 4-3, we write a small program to draw boxes on the screen with the mouse. The function my_mouse_callback() responds to mouse events, and it uses the event to determine what to do when it is called. Example 4-3. Toy program for using a mouse to draw boxes on the screen #include <opencv2/opencv.hpp>  // Define our callback which we will install for // mouse events // void my_mouse_callback(    int event, int x, int y, int flags, void* param  );   Rect box; bool drawing_box = false;   // A litte subroutine to draw a box onto an image // void draw_box( cv::Mat& img, cv::Rect box ) {   cv::rectangle(     img,      box.tl(),     box.br(),     cv::Scalar(0x00,0x00,0xff)    /* red */   ); }   void

is called. Example 4-3. Toy program for using a mouse to draw boxes on the screen #include <opencv2/opencv.hpp>  // Define our callback which we will install for // mouse events // void my_mouse_callback(    int event, int x, int y, int flags, void* param  );   Rect box; bool drawing_box = false;   // A litte subroutine to draw a box onto an image // void draw_box( cv::Mat& img, cv::Rect box ) {   cv::rectangle(     img,      box.tl(),     box.br(),     cv::Scalar(0x00,0x00,0xff)    /* red */   ); }   void help() {   std::cout << "Call: ./ch4_ex4_1\n" <<     " shows how to use a mouse to draw regions in an image." << std::endl; }  int main( int argc, char** argv ) {    help();   box = cv::Rect(-1,-1,0,0);   cv::Mat image(200, 200, CV::U8C3), temp;   image.copyTo(temp);      box   = cv::Rect(-1,-1,0,0);   image = cv::Scalar::all(0);      cv::namedWindow( "Box Example" );     // Here is the crucial moment where we actually install   // the callback. Note that we set the value of ‘params’ to

// be the image we are working with so that the callback   // will have the image to edit.   //   cv::setMouseCallback(      "Box Example",      my_mouse_callback,      (void*)&image    );     // The main program loop. Here we copy the working image   // to the temp image, and if the user is drawing, then   // put the currently contemplated box onto that temp image.   // Display the temp image, and wait 15ms for a keystroke,   // then repeat.   //   for(;;) {    image.copyTo(temp);     if( drawing_box ) draw_box( temp, box );      cv::imshow( "Box Example", temp );       if( cv::waitKey( 15 ) == 27 ) break;   }     return 0; }   // This is our mouse callback. If the user // presses the left button, we start a box. // when the user releases that button, then we // add the box to the current image. When the // mouse is dragged (with the button down) we  // resize the box. // void my_mouse_callback(    int event, int x, int y, int flags, void* param  ) {     cv::Mat& image = *(cv::Mat*) param;    switch(

) draw_box( temp, box );      cv::imshow( "Box Example", temp );       if( cv::waitKey( 15 ) == 27 ) break;   }     return 0; }   // This is our mouse callback. If the user // presses the left button, we start a box. // when the user releases that button, then we // add the box to the current image. When the // mouse is dragged (with the button down) we  // resize the box. // void my_mouse_callback(    int event, int x, int y, int flags, void* param  ) {     cv::Mat& image = *(cv::Mat*) param;    switch( event ) {     case cv::EVENT_MOUSEMOVE: {       if( drawing_box ) {         box.width  = x-box.x;         box.height = y-box.y;       }     }     break;     case cv::EVENT_LBUTTONDOWN: {       drawing_box = true;       box = cv::Rect( x, y, 0, 0 );     }     break;        case cv::EVENT_LBUTTONUP: {       drawing_box = false;        if( box.width < 0  ) {          box.x += box.width;           box.width *= -1;        }       if( box.height < 0 ) {          box.y += box.height;          box.height *= -1;

) {     case cv::EVENT_MOUSEMOVE: {       if( drawing_box ) {         box.width  = x-box.x;         box.height = y-box.y;       }     }     break;     case cv::EVENT_LBUTTONDOWN: {       drawing_box = true;       box = cv::Rect( x, y, 0, 0 );     }     break;        case cv::EVENT_LBUTTONUP: {       drawing_box = false;        if( box.width < 0  ) {          box.x += box.width;           box.width *= -1;        }       if( box.height < 0 ) {          box.y += box.height;          box.height *= -1;        }

draw_box( image, box );     }     break;      } } Sliders, Trackbars, and Switches HighGUI provides a convenient slider element. In HighGUI, sliders are called trackbars. This is because their original (historical) intent was for selecting a particular frame in the playback of a video. Of course, once added to HighGUI, people began to use trackbars for all of the usual things one might do with a slider as well as many unusual ones (we’ll discuss some of these in the next section, “No Buttons”). As with the parent window, the slider is given a unique name (in the form of a character string) and is thereafter always referred to by that name. The HighGUI routine for creating a trackbar is: int cv::createTrackbar(   const string&        trackbarName,      // Handle used to identify trackbar, label   const string&        windowName,        // Handle used to identify window   int*                 value,             // Slider position gets put here   int                  count,             // Total counts

with the parent window, the slider is given a unique name (in the form of a character string) and is thereafter always referred to by that name. The HighGUI routine for creating a trackbar is: int cv::createTrackbar(   const string&        trackbarName,      // Handle used to identify trackbar, label   const string&        windowName,        // Handle used to identify window   int*                 value,             // Slider position gets put here   int                  count,             // Total counts for slider at far right   cv::TrackbarCallback onChange    = NULL,// Callback function (optional)   void*                param       = NULL // Additional parameters for callback fn. ); The first two arguments are the name for the trackbar itself and the name of the parent window to which the trackbar will be attached. When the trackbar is created, it is added to either the top or the bottom of the parent window;15 it will not occlude any image that is already in the window, rather it will make the window

for slider at far right   cv::TrackbarCallback onChange    = NULL,// Callback function (optional)   void*                param       = NULL // Additional parameters for callback fn. ); The first two arguments are the name for the trackbar itself and the name of the parent window to which the trackbar will be attached. When the trackbar is created, it is added to either the top or the bottom of the parent window;15 it will not occlude any image that is already in the window, rather it will make the window slightly bigger. The name of the trackbar will appear as a “label” for the trackbar. As with the location of the trackbar itself, the exact location of this label depends on the operating system, but most often it is immediately to the left.  Figure 4-2: A simple application displaying an image; this window has two trackbars attached named "Trackbar0" and "Trackbar1"                                                              15 Whether it is added to the top or bottom depends on the operating system, but

slightly bigger. The name of the trackbar will appear as a “label” for the trackbar. As with the location of the trackbar itself, the exact location of this label depends on the operating system, but most often it is immediately to the left.  Figure 4-2: A simple application displaying an image; this window has two trackbars attached named "Trackbar0" and "Trackbar1"                                                              15 Whether it is added to the top or bottom depends on the operating system, but it will always appear in the same place on any given platform.

The next two arguments are value, a pointer to an integer that will be set automatically to the value to which the slider has been moved, and count, a numerical value for the maximum value of the slider. The last argument is a pointer to a callback function that will be automatically called whenever the slider is moved. This is exactly analogous to the callback for mouse events. If used, the callback function must have the form specified by cv::TrackbarCallback, which means that it must match the following prototype: void your_trackbar_callback(    int   pos,                              // Trackbar slider position   void* param = NULL                      // Parameters from cv::setTrackbarCallback() ); This callback is not actually required, so if you don’t want a callback, then you can simply set this value to NULL. Without a callback, the only effect of the user moving the slider will be the value of *value being updated when the slider is moved. (Of course, if you don’t have a callback, you will be

prototype: void your_trackbar_callback(    int   pos,                              // Trackbar slider position   void* param = NULL                      // Parameters from cv::setTrackbarCallback() ); This callback is not actually required, so if you don’t want a callback, then you can simply set this value to NULL. Without a callback, the only effect of the user moving the slider will be the value of *value being updated when the slider is moved. (Of course, if you don’t have a callback, you will be responsible for polling this value if you are going to respond to it being changed.) The final argument to cv::createTrackbar() is params, which can be any pointer. This pointer will be passed to your callback as its params argument whenever the callback is executed. This is very helpful, among other things, for allowing you to handle trackbar events without having to introduce global variables. Finally, here are two more routines that will allow you to programmatically read or set the value of a trackbar just

for polling this value if you are going to respond to it being changed.) The final argument to cv::createTrackbar() is params, which can be any pointer. This pointer will be passed to your callback as its params argument whenever the callback is executed. This is very helpful, among other things, for allowing you to handle trackbar events without having to introduce global variables. Finally, here are two more routines that will allow you to programmatically read or set the value of a trackbar just by using its name: int cv::getTrackbarPos(   const string& trackbarName,             // Handle used to identify trackbar, label   const string& windowName,               // Handle used to identify window );  void cv::setTrackbarPos(   const string& trackbarName,             // Handle used to identify trackbar, label   const string& windowName,               // Handle used to identify window   int   pos                               // Trackbar slider position ); These functions allow you to read or set the value

its name: int cv::getTrackbarPos(   const string& trackbarName,             // Handle used to identify trackbar, label   const string& windowName,               // Handle used to identify window );  void cv::setTrackbarPos(   const string& trackbarName,             // Handle used to identify trackbar, label   const string& windowName,               // Handle used to identify window   int   pos                               // Trackbar slider position ); These functions allow you to read or set the value of a trackbar from anywhere in your program. No Buttons Unfortunately, the native interface in HighGUI does not provide any explicit support for buttons. It is thus common practice, among the particularly lazy,16 to instead use sliders with only two positions. Another option that occurs often in the OpenCV samples in …/opencv/samples/c/ is to use keyboard shortcuts instead of buttons (see, e.g., the floodfill demo in the OpenCV source-code bundle). Switches are just sliders (trackbars) that have only two

of a trackbar from anywhere in your program. No Buttons Unfortunately, the native interface in HighGUI does not provide any explicit support for buttons. It is thus common practice, among the particularly lazy,16 to instead use sliders with only two positions. Another option that occurs often in the OpenCV samples in …/opencv/samples/c/ is to use keyboard shortcuts instead of buttons (see, e.g., the floodfill demo in the OpenCV source-code bundle). Switches are just sliders (trackbars) that have only two positions, “on” (1) and “off” (0) (i.e., count has been set to 1). You can see how this is an easy way to obtain the functionality of a button using only the available trackbar tools. Depending on exactly how you want the switch to behave, you can use the trackbar callback to automatically reset the button back to 0 (as in Example 4-4; this is something like the standard behavior of most GUI “buttons”) or to automatically set other switches to 0 (which gives the effect of a “radio button”).

“on” (1) and “off” (0) (i.e., count has been set to 1). You can see how this is an easy way to obtain the functionality of a button using only the available trackbar tools. Depending on exactly how you want the switch to behave, you can use the trackbar callback to automatically reset the button back to 0 (as in Example 4-4; this is something like the standard behavior of most GUI “buttons”) or to automatically set other switches to 0 (which gives the effect of a “radio button”).                                                              16 For the less lazy, the common practice was to compose the image you are displaying with a “control panel” you have drawn and then use the mouse event callback to test for the mouse’s location when the event occurs. When the (x, y) location is within the area of a button you have drawn on your control panel, the callback is set to perform the button action. In this way, all “buttons” are internal to the mouse event callback routine associated with the parent window.

16 For the less lazy, the common practice was to compose the image you are displaying with a “control panel” you have drawn and then use the mouse event callback to test for the mouse’s location when the event occurs. When the (x, y) location is within the area of a button you have drawn on your control panel, the callback is set to perform the button action. In this way, all “buttons” are internal to the mouse event callback routine associated with the parent window. Really however, if you need this kind of functionality now, it is probably just best to use the Qt backend.

Example 4-4. Using a trackbar to create a “switch” that the user can turn on and off; this program plays a video, and uses the switch to create a pause functionality // An example program in which the user can draw boxes on the screen. // #include <opencv2/opencv.hpp> #include <iostream>  using namespace std;  // // Using a trackbar to create a "switch" that the user can turn on and off. // We make this value global so everyone can see it. // int g_switch_value = 1; void switch_off_function() { cout << "Pause\n"; }; //YOU COULD DO MORE WITH THESE FUNCTIONS void switch_on_function()  { cout << "Run\n"; };   // This will be the callback that we give to the trackbar. // void switch_callback( int position, void* ) {   if( position == 0 ) {     switch_off_function();   } else {     switch_on_function();   } }  void help() {  cout << "Call: my.avi" << endl;  cout << "Shows putting a pause button in a video." << endl; }  int main( int argc, char** argv ) {    cv::Mat frame; // To hold movie images

{ cout << "Pause\n"; }; //YOU COULD DO MORE WITH THESE FUNCTIONS void switch_on_function()  { cout << "Run\n"; };   // This will be the callback that we give to the trackbar. // void switch_callback( int position, void* ) {   if( position == 0 ) {     switch_off_function();   } else {     switch_on_function();   } }  void help() {  cout << "Call: my.avi" << endl;  cout << "Shows putting a pause button in a video." << endl; }  int main( int argc, char** argv ) {    cv::Mat frame; // To hold movie images   cv::VideoCapture g_capture;   help();   if( argc < 2 || !g_capture.open( argv[1] ) ){     cout << "Failed to open " << argv[1] << " video file\n" << endl;     return -1;   }    // Name the main window   //   cv::namedWindow( "Example", 1 );     // Create the trackbar. We give it a name,   // and tell it the name of the parent window.   //   cv::createTrackbar(     "Switch",     "Example",     &g_switch_value,     1,     switch_callback   );     // This will just cause OpenCV to idle until    // someone hits

g_capture;   help();   if( argc < 2 || !g_capture.open( argv[1] ) ){     cout << "Failed to open " << argv[1] << " video file\n" << endl;     return -1;   }    // Name the main window   //   cv::namedWindow( "Example", 1 );     // Create the trackbar. We give it a name,   // and tell it the name of the parent window.   //   cv::createTrackbar(     "Switch",     "Example",     &g_switch_value,     1,     switch_callback   );     // This will just cause OpenCV to idle until    // someone hits the "Escape" key.   //   for(;;) {    if( g_switch_value ) {

g_capture >> frame;     if( frame.empty() ) break;     cv::imshow( "Example", frame);    }    if( cv::waitKey(10)==27 ) break;   }      return 0; } You can see that this will turn on and off just like a light switch. In our example, whenever the trackbar “switch” is set to 0, the callback executes the function switch_off_function(), and whenever it is switched on, the switch_on_function() is called. Working with the Qt Backend As we described earlier, the trend in the development of the HighGUI portion of OpenCV is to rely increasingly on a separate library for GUI functionality. This makes sense, as it is not OpenCV’s purpose to reinvent that particular wheel, and there are plenty of great GUI toolkits out there that are well maintained and which evolve and adapt to the changing times under their own development teams. From the perspective of the OpenCV library, there is a great advantage to using such an outside toolkit. Functionality is gained, and at the same time, development time is reduced (which

of OpenCV is to rely increasingly on a separate library for GUI functionality. This makes sense, as it is not OpenCV’s purpose to reinvent that particular wheel, and there are plenty of great GUI toolkits out there that are well maintained and which evolve and adapt to the changing times under their own development teams. From the perspective of the OpenCV library, there is a great advantage to using such an outside toolkit. Functionality is gained, and at the same time, development time is reduced (which would otherwise be taken away from the core goal of the library).  However, an important clarification is that using HighGUI with the Qt backend is not the same as using Qt directly (we will explore this possibility briefly at the end of this chapter). The HighGUI interface is still the HighGUI interface; it simply uses Qt behind the scenes in place of the various native libraries. One side effect of this is that it is not that convenient to extend the Qt interface. If you want more than HighGUI gives you,

would otherwise be taken away from the core goal of the library).  However, an important clarification is that using HighGUI with the Qt backend is not the same as using Qt directly (we will explore this possibility briefly at the end of this chapter). The HighGUI interface is still the HighGUI interface; it simply uses Qt behind the scenes in place of the various native libraries. One side effect of this is that it is not that convenient to extend the Qt interface. If you want more than HighGUI gives you, you are still pretty stuck with writing your own window layer. On the bright side, the Qt interface gives you a lot more to work with, and so perhaps you will not find that extra level of complexity necessary as often. Getting Started If you have built your OpenCV installation with Qt support on,17 opening a window will automatically open a window with two new features. These are the Toolbar and the Status Bar (see Figure 4-3). These objects come up complete, with all of the contents you see in the figure.

you are still pretty stuck with writing your own window layer. On the bright side, the Qt interface gives you a lot more to work with, and so perhaps you will not find that extra level of complexity necessary as often. Getting Started If you have built your OpenCV installation with Qt support on,17 opening a window will automatically open a window with two new features. These are the Toolbar and the Status Bar (see Figure 4-3). These objects come up complete, with all of the contents you see in the figure. In particular, the Toolbar contains buttons which will allow you to pan (the first four arrow buttons), zoom (the next four buttons), save the current image (the ninth button), and a final button which can be used to pop up a properties window (more on this last one a little later).                                                              17 This means that when you configured the build with cmake, you used the –D WITH_QT=ON option.

Figure 4-3: This image is displayed with the Qt interface enabled; it shows the Toolbar, the Status Bar, and a text Overlay (in this case, containing the name of the image) The lower Status Bar in Figure 4-3 contains information about what is under your mouse at any given moment. The x-y location is displayed, as well as the RGB value of the pixel currently pointed at. All of this you get “for free” just for compiling your code with the Qt interface enabled. If you have compiled with Qt and you do not want these decorations, then you can simply add the cv::GUI_NORMAL flag when you call cv::namedWindow() and they will disappear.18 The Actions Menu As you can see, when you create the window with cv::GUI_EXTENDED, you will see a range of buttons in the Toolbar. An alternative to the toolbar, which will always be available whether you use cv::GUI_EXTENDED or not, is the pop-up menu. This pop-up menu contains the same options as the toolbar and can be made to appear at any time by right-clicking on the image.

these decorations, then you can simply add the cv::GUI_NORMAL flag when you call cv::namedWindow() and they will disappear.18 The Actions Menu As you can see, when you create the window with cv::GUI_EXTENDED, you will see a range of buttons in the Toolbar. An alternative to the toolbar, which will always be available whether you use cv::GUI_EXTENDED or not, is the pop-up menu. This pop-up menu contains the same options as the toolbar and can be made to appear at any time by right-clicking on the image.                                                              18 There is also a flag CV_GUI_EXTENDED, which (in theory) creates these decorations, but its numerical value is 0x00, so it is the default behavior anyhow.

Figure 4-4. Here the Qt extended UI window is shown with the pop-up menu, which provides the same options as the Toolbar (along with an explanation of the buttons and their keyboard shortcuts) The Text Overlay Another option provided by the Qt GUI is the ability to put a short-lived banner across the top of the image you are displaying. This banner is called the Overlay, and appears with a shaded box around it for easy reading. This is an exceptionally handy feature if you just want to throw some simple information like the frame number or frame rate on a video, or a filename of the image you are looking at. You can display an overlay on any window, even if you are using cv::GUI_NORMAL. int cv::displayOverlay(   const string& name,                     // Handle used to identify window   const string& text,                     // Text you want to display   int           delay                     // Milliseconds to show text (0=’forever’) ); The function cv::displayOverlay() takes three arguments. The first

information like the frame number or frame rate on a video, or a filename of the image you are looking at. You can display an overlay on any window, even if you are using cv::GUI_NORMAL. int cv::displayOverlay(   const string& name,                     // Handle used to identify window   const string& text,                     // Text you want to display   int           delay                     // Milliseconds to show text (0=’forever’) ); The function cv::displayOverlay() takes three arguments. The first is the name of the window you want the overlay to appear on. The second argument is whatever text you would like to appear in the window. One word of warning here, the text has a fixed size, so if you try to cram too much stuff in there, it will just overflow.19 By default, the text is always center justified. The third and final argument, delay, is the amount of time (in milliseconds) that the overlay will stay in place. If delay is set to 0, then the overlay will stay in place indefinitely (or at least

is the name of the window you want the overlay to appear on. The second argument is whatever text you would like to appear in the window. One word of warning here, the text has a fixed size, so if you try to cram too much stuff in there, it will just overflow.19 By default, the text is always center justified. The third and final argument, delay, is the amount of time (in milliseconds) that the overlay will stay in place. If delay is set to 0, then the overlay will stay in place indefinitely (or at least until you write over it with another call to cv::displayOverlay()). In general, if you call cv::displayOverlay() before the delay timer for a previous call is expired, the previous text is removed and replaced with the new text, and the timer is reset to the new delay value regardless of what is left in the timer before the new call. Writing Your Own Text into the Status Bar In addition to the Overlay, you can also write text into the Status Bar. By default, the Status Bar contains information about the

until you write over it with another call to cv::displayOverlay()). In general, if you call cv::displayOverlay() before the delay timer for a previous call is expired, the previous text is removed and replaced with the new text, and the timer is reset to the new delay value regardless of what is left in the timer before the new call. Writing Your Own Text into the Status Bar In addition to the Overlay, you can also write text into the Status Bar. By default, the Status Bar contains information about the pixel over which your mouse pointer is located (if any). You can see in Figure 4-3                                                              19 You can, however, insert new lines. So, for example, if you were to give the text string “Hello\nWorld”, then the word “Hello” would appear on the first (top) line, and the word “World” would appear on a second line right below “Hello.”

that the status bar contains an x-y location and the RGB color value of the pixel that was under the pointer when the figure was made. You can replace this text with your own text with the cv::displayStatusBar() method: int cv::displayStatusBar(   const string& name,                     // Handle used to identify window   const string& text,                     // Text you want to display   int           delay                     // Milliseconds to show text (0=’forever’) ); Unlike cv::displayOverlay(), cv::displayStatusBar() can only be used on windows that were created with the cv::GUI_EXTENDED flag (i.e., ones that have a Status Bar in the first place). When the delay timer is expired (if you didn’t set it to 0), then the default x-y and RGB text will re-appear in the Status Bar. The Control Panel You may have noticed in the earlier figures that there is a last button on the Toolbar, corresponding to a last option (which is “darkened” in Figure 4-4) on the pop-up menu which we have not really discussed

cv::displayStatusBar() can only be used on windows that were created with the cv::GUI_EXTENDED flag (i.e., ones that have a Status Bar in the first place). When the delay timer is expired (if you didn’t set it to 0), then the default x-y and RGB text will re-appear in the Status Bar. The Control Panel You may have noticed in the earlier figures that there is a last button on the Toolbar, corresponding to a last option (which is “darkened” in Figure 4-4) on the pop-up menu which we have not really discussed yet. This option opens up an entirely new window called the Control Panel. The control panel is a convenient place to put trackbars and buttons (the Qt GUI does support buttons) that you don’t want in your face all of the time. It is important to remember, however, that there is just one Control Panel per application, and you do not really create it, you just configure it.  Figure 4-5: In this image, we have added two trackbars to the main window; we also show the Control Panel with three pushbuttons, a

yet. This option opens up an entirely new window called the Control Panel. The control panel is a convenient place to put trackbars and buttons (the Qt GUI does support buttons) that you don’t want in your face all of the time. It is important to remember, however, that there is just one Control Panel per application, and you do not really create it, you just configure it.  Figure 4-5: In this image, we have added two trackbars to the main window; we also show the Control Panel with three pushbuttons, a trackbar, two radio buttons, and a checkbox The Control Panel will not be available unless some trackbars or buttons have been assigned to it (more on how to do this momentarily). If it is available, then the Control Panel can be made to appear by pressing the “Display Properties Window” button on the Toolbar (the one on the far right), the identical button on the Action Menu, or pressing Ctrl+P while your mouse is over any window. Trackbars Revisited In the previous section on the HighGUI native interface,

two radio buttons, and a checkbox The Control Panel will not be available unless some trackbars or buttons have been assigned to it (more on how to do this momentarily). If it is available, then the Control Panel can be made to appear by pressing the “Display Properties Window” button on the Toolbar (the one on the far right), the identical button on the Action Menu, or pressing Ctrl+P while your mouse is over any window. Trackbars Revisited In the previous section on the HighGUI native interface, we saw that it was possible to add trackbars to windows. The trackbars in 4 were created using the same cv::createTrackbar() command we saw

earlier. The only real difference between the trackbars in the figure, and the ones we created earlier, is that they are prettier (compare with the trackbars using the non-Qt interface in Figure 4-2). The important new concept in the Qt interface, however, is that we can also create trackbars in the Control Panel. This is done simply by creating the trackbar as you normally would, but by specifying an empty string as the window name to which the trackbar is to be attached.  int contrast = 128; cv::createTrackbar( "Contrast:", "", &contrast, 255, on_change_contrast ); For example, this fragment would create a trackbar in the Control Panel that would be labeled “Contrast:”, and whose value would start out at 128, with a maximum value of 255. Whenever the slider is adjusted, the callback on_change_contrast() will be called. Creating Buttons with cv::createButton() One of the most helpful new capabilities provided by the Qt interface is the ability to create buttons. This includes normal push-buttons, radio

"Contrast:", "", &contrast, 255, on_change_contrast ); For example, this fragment would create a trackbar in the Control Panel that would be labeled “Contrast:”, and whose value would start out at 128, with a maximum value of 255. Whenever the slider is adjusted, the callback on_change_contrast() will be called. Creating Buttons with cv::createButton() One of the most helpful new capabilities provided by the Qt interface is the ability to create buttons. This includes normal push-buttons, radio style (mutually exclusive) buttons, and checkboxes. All buttons created are always located in the Control Panel. All three styles of buttons are created with the same method: int cv::createButton(   const string&      buttonName,          // Handle used to identify trackbar, label   cv::ButtonCallback onChange     = NULL, // Callback for button event   void*              params,              // (Optional) parameters for button event   int                buttonType   = cv::PUSH_BUTTON,  // PUSH_BUTTON or RADIOBOX   int

exclusive) buttons, and checkboxes. All buttons created are always located in the Control Panel. All three styles of buttons are created with the same method: int cv::createButton(   const string&      buttonName,          // Handle used to identify trackbar, label   cv::ButtonCallback onChange     = NULL, // Callback for button event   void*              params,              // (Optional) parameters for button event   int                buttonType   = cv::PUSH_BUTTON,  // PUSH_BUTTON or RADIOBOX   int                initialState = 0     // Start state of the button ); The button expects a name buttonName that will appear on or next to the button. If you like, you can neglect this and simply provide an empty string, in which case the button name will be automatically generated in a serialized manner (e.g., “button 0,” “button 1,” etc.).  The second argument onChange is a callback that will be called whenever the button is pressed. The prototype for such a callback must match the declaration for

int                initialState = 0     // Start state of the button ); The button expects a name buttonName that will appear on or next to the button. If you like, you can neglect this and simply provide an empty string, in which case the button name will be automatically generated in a serialized manner (e.g., “button 0,” “button 1,” etc.).  The second argument onChange is a callback that will be called whenever the button is pressed. The prototype for such a callback must match the declaration for cv::ButtonCallback, which is: void your_button_callback(   int   state,                            // Identifies button event type   void* params                            // Paramaters from cv::createButton() );  When your callback is called as a result of someone pressing a button, it will be given the value state, which is derived from what just happened to the button. The pointer param that you gave to cv::createButton() will also be passed to your callback, filling its param argument.  The buttonType

cv::ButtonCallback, which is: void your_button_callback(   int   state,                            // Identifies button event type   void* params                            // Paramaters from cv::createButton() );  When your callback is called as a result of someone pressing a button, it will be given the value state, which is derived from what just happened to the button. The pointer param that you gave to cv::createButton() will also be passed to your callback, filling its param argument.  The buttonType argument can take one of three values: cv::PUSH_BUTTON, cv::RADIOBOX, or cv::CHECKBOX. The first corresponds to your standard button—you press it, it calls your callback. In the case of the checkbox, the value will be 1 or 0 depending on whether the box was checked or unchecked. The same is true for a radio button, except that when you click a radio button, the callback is called both for the button you just clicked, and for the button that is now unclicked (as a result of the mutex nature of radio

argument can take one of three values: cv::PUSH_BUTTON, cv::RADIOBOX, or cv::CHECKBOX. The first corresponds to your standard button—you press it, it calls your callback. In the case of the checkbox, the value will be 1 or 0 depending on whether the box was checked or unchecked. The same is true for a radio button, except that when you click a radio button, the callback is called both for the button you just clicked, and for the button that is now unclicked (as a result of the mutex nature of radio buttons). All buttons in the same row (see button bars below) are assumed to be part of the same mutex group.  When buttons are created, they are automatically organized into button bars. A button bar is a group of buttons that occupies a “row” in the Control Panel. Consider the following code, which generated the Control Panel you see in Figure 4-5.   cv::namedWindow( "Image", cv::GUI_EXPANDED );   cv::displayOverlay( "Image", file_name, 0 );   cv::createTrackbar( "Trackbar0", "Image", &mybar0, 255 );

buttons). All buttons in the same row (see button bars below) are assumed to be part of the same mutex group.  When buttons are created, they are automatically organized into button bars. A button bar is a group of buttons that occupies a “row” in the Control Panel. Consider the following code, which generated the Control Panel you see in Figure 4-5.   cv::namedWindow( "Image", cv::GUI_EXPANDED );   cv::displayOverlay( "Image", file_name, 0 );   cv::createTrackbar( "Trackbar0", "Image", &mybar0, 255 );   cv::createTrackbar( "Trackbar1", "Image", &mybar1, 255 );    cv::createButton( "", NULL, NULL, cv::PUSH_BUTTON );   cv::createButton( "", NULL, NULL, cv::PUSH_BUTTON );

cv::createButton( "", NULL, NULL, cv::PUSH_BUTTON );   cv::createTrackbar( "Trackbar2", "", &mybar1, 255 );   cv::createButton( "Button3", NULL, NULL, cv::RADIOBOX, 1 );   cv::createButton( "Button4", NULL, NULL, cv::RADIOBOX, 0 );   cv::createButton( "Button5", NULL, NULL, cv::CHECKBOX, 0 ); You will notice that Trackbar0 and Trackbar1 are created in the window called “Image,” while Trackbar2 is created in an unnamed window (the Control Panel). The first three cv::createButton() calls are not given a name for the button, and you can see in Figure 4-5 the automatically assigned names are placed onto the buttons. You will also notice in Figure 4-5 that the first three buttons are in one row, while the second group of three is on another. This is because of the trackbar. Buttons are created one after another, each to the right of its predecessor, until (unless) a trackbar is created. Because a trackbar consumes an entire row, it is given its own row below the buttons. If more buttons are created, they will

for the button, and you can see in Figure 4-5 the automatically assigned names are placed onto the buttons. You will also notice in Figure 4-5 that the first three buttons are in one row, while the second group of three is on another. This is because of the trackbar. Buttons are created one after another, each to the right of its predecessor, until (unless) a trackbar is created. Because a trackbar consumes an entire row, it is given its own row below the buttons. If more buttons are created, they will appear on a new row thereafter.20 Text and Fonts Just as the Qt interface allowed for much prettier trackbars and other elements, Qt also allows for much prettier and more versatile text. To write text using the Qt interface, you must first create a CvFont object,21 which you then use whenever you want to put some text on the screen. Fonts are created using the cv::fontQt() function: CvFont fontQt(                            // Return OpenCV font characerization struct   const string& fontName,

on a new row thereafter.20 Text and Fonts Just as the Qt interface allowed for much prettier trackbars and other elements, Qt also allows for much prettier and more versatile text. To write text using the Qt interface, you must first create a CvFont object,21 which you then use whenever you want to put some text on the screen. Fonts are created using the cv::fontQt() function: CvFont fontQt(                            // Return OpenCV font characerization struct   const string& fontName,                 // e.g., “Times”   int           pointSize,                // Size of font, using “point” system.   cv::Scalar    color    = cv::Scalar::all(0), // BGR color as a scalar (no alpha)   int           weight   = cv::FONT_NORMAL,    // Font weights, 1-100 or see Table 4-7   int           spacing  = 0              // Space between individual characters );    The first argument to cv::fontQt() is the system font name. This might be something like “Times.”  If your system does not have an available font with this

// e.g., “Times”   int           pointSize,                // Size of font, using “point” system.   cv::Scalar    color    = cv::Scalar::all(0), // BGR color as a scalar (no alpha)   int           weight   = cv::FONT_NORMAL,    // Font weights, 1-100 or see Table 4-7   int           spacing  = 0              // Space between individual characters );    The first argument to cv::fontQt() is the system font name. This might be something like “Times.”  If your system does not have an available font with this name, then a default font will be chosen for you. The second argument pointSize is the size of the font (i.e., 12=”12 point”, 14=”14 point”, etc.)  You may set this to 0, in which case a default font size (typically 12 point) will be selected for you. The argument color can be set to any cv::Scalar and will set the color in which the font is drawn; the default value is black. weight can take one of several pre-defined values, or any integer you like between 1 and 100. The pre-defined aliases and their

this name, then a default font will be chosen for you. The second argument pointSize is the size of the font (i.e., 12=”12 point”, 14=”14 point”, etc.)  You may set this to 0, in which case a default font size (typically 12 point) will be selected for you. The argument color can be set to any cv::Scalar and will set the color in which the font is drawn; the default value is black. weight can take one of several pre-defined values, or any integer you like between 1 and 100. The pre-defined aliases and their values are shown in Table 4-7. Table 4-7: Pre-defined aliases for Qt-font weight and their associated values Camera capture constan Numerical value  cv::FONT_LIGHT                25  cv::FONT_NORMAL 50  cv::FONT_DEMIBOLD 63  cv::FONT_BOLD 75  cv::FONT_BLACK 87 The final argument is spacing, which controls the spacing between individual characters. It can be negative or positive.                                                              20 Unfortunately, there is no “carriage return” for button

and their values are shown in Table 4-7. Table 4-7: Pre-defined aliases for Qt-font weight and their associated values Camera capture constan Numerical value  cv::FONT_LIGHT                25  cv::FONT_NORMAL 50  cv::FONT_DEMIBOLD 63  cv::FONT_BOLD 75  cv::FONT_BLACK 87 The final argument is spacing, which controls the spacing between individual characters. It can be negative or positive.                                                              20 Unfortunately, there is no “carriage return” for button placement. 21 You will notice that the name of this object is CvFont rather than what you might expect: cv::Font. This is a legacy to the old pre-C++ interface. CvFont is a struct, and is not in the cv:: namespace.

Once you have your font, you can put text on an image (and thus on the screen)22 with cv::addText(). void cv::addText(   cv::Mat&      image,                    // Image onto which to write   const string& text,                     // Text to write   cv::Point     location,                 // Location of lower-left corner of text   CvFont*       font                      // OpenCV font characerization struct ); The arguments to cv::addText() are just what you would expect, the image to write on, the text to write, where to write it, and the font to use—with the latter being a font you defined using cv::fontQt. The location argument corresponds to the lower-left corner of the first character in text (or, more precisely, the beginning of the baseline for that character). Setting and Getting Window Properties Many of the state properties of a window set at creation are queryable, and many of those can be changed (set) even after the window’s creation. void   cv::setWindowProperty(   const string& name,

write, where to write it, and the font to use—with the latter being a font you defined using cv::fontQt. The location argument corresponds to the lower-left corner of the first character in text (or, more precisely, the beginning of the baseline for that character). Setting and Getting Window Properties Many of the state properties of a window set at creation are queryable, and many of those can be changed (set) even after the window’s creation. void   cv::setWindowProperty(   const string& name,                     // Handle used to identify window   int           prop_id,                  // Identifies window property (see Table 4-8)   double        prop_value                // Value to which to set property );  double cv::getWindowProperty(   const string& name,                     // Handle used to identify window   int           prop_id                   // Identifies window property (see Table 4-8) ); To get a window property, you need only call cv::getWindowProperty() and supply the name of the window

// Handle used to identify window   int           prop_id,                  // Identifies window property (see Table 4-8)   double        prop_value                // Value to which to set property );  double cv::getWindowProperty(   const string& name,                     // Handle used to identify window   int           prop_id                   // Identifies window property (see Table 4-8) ); To get a window property, you need only call cv::getWindowProperty() and supply the name of the window and the property ID (prop_id argument) of the property you are interested in (see Table 4-8). Table 4-8: Gettable and settable window properties Property name Description cv::WIND_PROP_FULL_SIZE  Set to either cv::WINDOW_FULLSCREEN for full screen window, or to cv::WINDOW_NORMAL for regular window. cv::WIND_PROP_AUTOSIZE Set to either cv::WINDOW_AUTOSIZE to have the window automatically size to the displayed image, or cv::WINDOW_NORMAL to have the image size to the window.  cv::WIND_PROP_ASPECTRATIO Set to

and the property ID (prop_id argument) of the property you are interested in (see Table 4-8). Table 4-8: Gettable and settable window properties Property name Description cv::WIND_PROP_FULL_SIZE  Set to either cv::WINDOW_FULLSCREEN for full screen window, or to cv::WINDOW_NORMAL for regular window. cv::WIND_PROP_AUTOSIZE Set to either cv::WINDOW_AUTOSIZE to have the window automatically size to the displayed image, or cv::WINDOW_NORMAL to have the image size to the window.  cv::WIND_PROP_ASPECTRATIO Set to either cv::WINDOW_FREERATIO to allow the window to have any aspect ratio (as a result of user resizing) or cv::WINDOW_KEEPRATIO to only allow user resizing to affect absolute size (and not aspect ratio).                                                              22 It is important to notice here that cv::addText() is somewhat unlike all of the rest of the functions in the Qt interface (though not inconsistent with the behavior of its non-Qt analog cv::putText()). Specifically, cv::addText() does not put

either cv::WINDOW_FREERATIO to allow the window to have any aspect ratio (as a result of user resizing) or cv::WINDOW_KEEPRATIO to only allow user resizing to affect absolute size (and not aspect ratio).                                                              22 It is important to notice here that cv::addText() is somewhat unlike all of the rest of the functions in the Qt interface (though not inconsistent with the behavior of its non-Qt analog cv::putText()). Specifically, cv::addText() does not put text in or on a window, but rather in an image. This means that you are actually changing the pixel values of the image—which is different than what would happen if, for example, you were to use cv::displayOverlay().

Saving and Recovering Window State The Qt interface also allows the state of windows to be saved and restored. This can be very convenient, as it includes not only the location and size of your windows, but also the state of all of the trackbars and buttons. The interface state is saved with the cv::saveWindowParameters() function, which takes a single argument indicating the window to be saved: void cv::saveWindowParameters(   const string& name                      // Handle used to identify window ); Once the state of the window is saved, it can be restored with the complementary cv::loadWindowParameters() function: void cv::loadWindowParameters(   const string& name                      // Handle used to identify window ); The real magic here is that the load command will work correctly even if you have quit and restarted your program. The nature of how this is done is not important to us here, but one detail you should know is that the state information, wherever it is saved, is saved under a key that

the state of the window is saved, it can be restored with the complementary cv::loadWindowParameters() function: void cv::loadWindowParameters(   const string& name                      // Handle used to identify window ); The real magic here is that the load command will work correctly even if you have quit and restarted your program. The nature of how this is done is not important to us here, but one detail you should know is that the state information, wherever it is saved, is saved under a key that is constructed from the executable name. So if you should change the name of the executable (though you can change its location), the state will not restore.

Interacting with OpenGL On many contexts, it is very useful to be able to use OpenGL to render synthetic images and display them on top of camera of other processed images. The HighGUI and Qt GUI’s provide a convenient way to use OpenGL to perform that rendering right on top of the whatever image you are already showing.23 This can be extremely effective for visualizing and debugging robotic or augmented-reality applications, or anywhere in which you are trying to generate a three-dimensional model from your image and want to see the resulting model visualized on top of the original incoming image.  Figure 4-6. Here OpenGL is used to render a cube on top of our previous image The basic concept is very simple; you create a callback that is an OpenGL draw function, and register it with the interface. From there, OpenCV takes care of the rest. The callback is then called every time the window is drawn (which includes whenever you call cv::imshow(), as you would with successive frames of a video stream). Your

your image and want to see the resulting model visualized on top of the original incoming image.  Figure 4-6. Here OpenGL is used to render a cube on top of our previous image The basic concept is very simple; you create a callback that is an OpenGL draw function, and register it with the interface. From there, OpenCV takes care of the rest. The callback is then called every time the window is drawn (which includes whenever you call cv::imshow(), as you would with successive frames of a video stream). Your callback should match the prototype for cv::OpenGLCallback(), which means that it should be something like the following: void your_opengl_callback(   void* params             // (Optional) Parameters from cv::createOpenGLCallback() );  Once you have your callback, you can configure the OpenGL interface with cv::setOpenGlDrawCallback(): void cv::setOpenGlDrawCallback(   const string&      windowName,          // Handle used to identify window   cv::OpenGLCallback callback,            // OpenGL callback

callback should match the prototype for cv::OpenGLCallback(), which means that it should be something like the following: void your_opengl_callback(   void* params             // (Optional) Parameters from cv::createOpenGLCallback() );  Once you have your callback, you can configure the OpenGL interface with cv::setOpenGlDrawCallback(): void cv::setOpenGlDrawCallback(   const string&      windowName,          // Handle used to identify window   cv::OpenGLCallback callback,            // OpenGL callback routine   void*              params    = NULL     // (Optional) parameters for callback );  As you can see, there is not much one needs to really set things up. In addition to specifying the name of the window on which the drawing will be done and supplying the callback function, you have a third                                                              23 By way of reminder, in order to use these commands, you will need to have built OpenCV with the cmake flag –D WITH_OPENGL=ON.

argument params, which allows you to specify a pointer that will be passed to callback whenever it is called.  It is probably worth calling out here explicitly that none of this sets up the camera, lighting, or other aspects of your OpenGL activities. Internally there is a wrapper around your OpenGL callback that will set up the projection matrix using a call to gluPerspective(). If you want anything different (which you almost certainly will), you will have to clear and configure the projection matrix at the beginning of your callback.  In Figure 4-6, we have taken a simple example code from the OpenCV documentation which draws a cube in OpenGL, but we have replaced the fixed rotation angles in that cube with variables (rotx and roty) which we have made the values of the two sliders in our earlier examples. Now the user can rotate the cube with the sliders while enjoying the beautiful scenery behind. Example 4-5: Slightly modified code from the OpenCV documentation that draws a cube every frame; this

at the beginning of your callback.  In Figure 4-6, we have taken a simple example code from the OpenCV documentation which draws a cube in OpenGL, but we have replaced the fixed rotation angles in that cube with variables (rotx and roty) which we have made the values of the two sliders in our earlier examples. Now the user can rotate the cube with the sliders while enjoying the beautiful scenery behind. Example 4-5: Slightly modified code from the OpenCV documentation that draws a cube every frame; this modified version uses the global variables rotx and roty that are connected to the sliders in Figure 4-6 void on_opengl( void* param ) {   glMatrixModel( GL_MODELVIEW );   glLoadIdentity();    glTranslated( 0.0, 0.0, -1.0 );    glRotatef( rotx, 1, 0, 0 );   glRotatef( roty, 0, 1, 0 );   glRotatef( 0, 0, 0, 1 );    static const int coords[6][4][3] = {     { { +1, -1, -1 }, { -1, -1, -1 }, { -1, +1, -1 }, { +1, +1, -1 } },     { { +1, +1, -1 }, { -1, +1, -1 }, { -1, +1, +1 }, { +1, +1, +1 } },     { { +1, -1,

version uses the global variables rotx and roty that are connected to the sliders in Figure 4-6 void on_opengl( void* param ) {   glMatrixModel( GL_MODELVIEW );   glLoadIdentity();    glTranslated( 0.0, 0.0, -1.0 );    glRotatef( rotx, 1, 0, 0 );   glRotatef( roty, 0, 1, 0 );   glRotatef( 0, 0, 0, 1 );    static const int coords[6][4][3] = {     { { +1, -1, -1 }, { -1, -1, -1 }, { -1, +1, -1 }, { +1, +1, -1 } },     { { +1, +1, -1 }, { -1, +1, -1 }, { -1, +1, +1 }, { +1, +1, +1 } },     { { +1, -1, +1 }, { +1, -1, -1 }, { +1, +1, -1 }, { +1, +1, +1 } },     { { -1, -1, -1 }, { -1, -1, +1 }, { -1, +1, +1 }, { -1, +1, -1 } },     { { +1, -1, +1 }, { -1, -1, +1 }, { -1, -1, -1 }, { +1, -1, -1 } },     { { -1, -1, +1 }, { +1, -1, +1 }, { +1, +1, +1 }, { -1, +1, +1 } }   };    for (int i = 0; i < 6; ++i) {     glColor3ub( i*20, 100+i*10, i*42 );     glBegin(GL_QUADS);     for (int j = 0; j < 4; ++j) {       glVertex3d(0.2 * coords[i][j][0], 0.2 * coords[i][j][1], 0.2 * coords[i][j][2]);     }     glEnd();   } }

{ +1, -1, -1 }, { +1, +1, -1 }, { +1, +1, +1 } },     { { -1, -1, -1 }, { -1, -1, +1 }, { -1, +1, +1 }, { -1, +1, -1 } },     { { +1, -1, +1 }, { -1, -1, +1 }, { -1, -1, -1 }, { +1, -1, -1 } },     { { -1, -1, +1 }, { +1, -1, +1 }, { +1, +1, +1 }, { -1, +1, +1 } }   };    for (int i = 0; i < 6; ++i) {     glColor3ub( i*20, 100+i*10, i*42 );     glBegin(GL_QUADS);     for (int j = 0; j < 4; ++j) {       glVertex3d(0.2 * coords[i][j][0], 0.2 * coords[i][j][1], 0.2 * coords[i][j][2]);     }     glEnd();   } } As was mentioned above, whenever you call cv::imshow(), a call will be generated to your OpenGL callback. You can also induce a call directly to that callback by calling cv::updateWindow().24 Integrating OpenCV with Full GUI Toolkits Even OpenCV’s built-in Qt interface is still just a handy way of accomplishing some simple tasks that come up often while developing code or exploring algorithms. When it comes time to actually build an end                                                              24 You

} As was mentioned above, whenever you call cv::imshow(), a call will be generated to your OpenGL callback. You can also induce a call directly to that callback by calling cv::updateWindow().24 Integrating OpenCV with Full GUI Toolkits Even OpenCV’s built-in Qt interface is still just a handy way of accomplishing some simple tasks that come up often while developing code or exploring algorithms. When it comes time to actually build an end                                                              24 You will really use cv::updateWindow() only when you are using OpenGL. The utility of cv::updateWindow() is precisely that it induces an OpenGL draw event (and thus calls your callback which you would have set up with cv::setOpenGLDrawCallback()).

user facing application, neither the native UI nor the Qt-based interface are going to do it. In this section, we will (very) briefly explore some of the issues and techniques for working with OpenCV and two existing toolkits: wxWidgets and Qt.  Clearly there are countless UI toolkits out there, and we would not want to waste time digging into each of them. Having said that, it is useful to explore how to handle the issues which will arise if you want to use OpenCV with a more fully featured toolkit. The primary issue is how to convert OpenCV images to the form that the toolkit expects for graphics, and to know which widget or component in the toolkit is going to do that display work for you. From there, you don’t need much else that is specific to OpenCV. Notably, you will not need or want the features of the UI toolkits we have covered in this chapter. An Example of OpenCV and Qt Here we will show an example of using the actual Qt toolkit to write a program that reads a video file and displays it on the

primary issue is how to convert OpenCV images to the form that the toolkit expects for graphics, and to know which widget or component in the toolkit is going to do that display work for you. From there, you don’t need much else that is specific to OpenCV. Notably, you will not need or want the features of the UI toolkits we have covered in this chapter. An Example of OpenCV and Qt Here we will show an example of using the actual Qt toolkit to write a program that reads a video file and displays it on the screen. There are several subtleties, some of which have to do with how one uses Qt, and others to do with OpenCV. Of course, we will focus on the latter, but it is worth taking a moment to notice how the former affect our current goal. Below is the top-level code for our example; it just creates a Qt application and adds our QMoviePlayer widget. Everything interesting will happen inside that object. Example 4-6. An example program ch4_qt.cpp, which takes a single argument indicating a video file; that

the screen. There are several subtleties, some of which have to do with how one uses Qt, and others to do with OpenCV. Of course, we will focus on the latter, but it is worth taking a moment to notice how the former affect our current goal. Below is the top-level code for our example; it just creates a Qt application and adds our QMoviePlayer widget. Everything interesting will happen inside that object. Example 4-6. An example program ch4_qt.cpp, which takes a single argument indicating a video file; that video file will be replayed inside of a Qt object which we will define, called QMoviePlayer #include <QApplication> #include <QLabel> #include <QMoviePlayer.hpp> int main( int argc, char* argv[] ) {    QApplication app( argc, argv );    QMoviePlayer mp;   mp.open( argv[1] );   mp.show();    return app.exec(); } The interesting stuff is in the QMoviePlayer object. Let’s take a look at the header file which defines that object: 4-6 cont. The QMoviePlayer object header file QMoviePlayer.hpp #include

that video file will be replayed inside of a Qt object which we will define, called QMoviePlayer #include <QApplication> #include <QLabel> #include <QMoviePlayer.hpp> int main( int argc, char* argv[] ) {    QApplication app( argc, argv );    QMoviePlayer mp;   mp.open( argv[1] );   mp.show();    return app.exec(); } The interesting stuff is in the QMoviePlayer object. Let’s take a look at the header file which defines that object: 4-6 cont. The QMoviePlayer object header file QMoviePlayer.hpp #include "ui_QMoviePlayer.h" #include <opencv2/opencv.hpp> #include <string>  using namespace std;  class QMoviePlayer : public QWidget {      Q_OBJECT;    public:   QMoviePlayer( QWidget *parent = NULL );   virtual ~QMoviePlayer() {;}    bool open( string file );    private:   Ui::QMoviePlayer  ui;   cv::VideoCapture m_cap;

QImage  m_qt_img;   cv::Mat m_cv_img;   QTimer* m_timer;    void paintEvent( QPaintEvent* q );   void _copyImage( void );    public slots:   void nextFrame();  }; There is a lot going on here. The first thing that happens is the inclusion of the file ui_QMoviePlayer.h. This file was automatically generated by the Qt Designer. What matters here is that it is just a QWidget that contains nothing but a QFrame called frame. The member ui::QMoviePlayer is that interface object which is defined in ui_QMoviePlayer.h.  In this file, there is also a QImage called m_qt_img and a cv::Mat called m_cv_img. These will contain the Qt and OpenCV representations of the image we are getting from our video. Finally, there is a QTimer, which is what will take the place of cv::waitKey(), allowing us to replay the video frames at the correct rate. The remaining functions will become clear as we look at their actual definitions in QMoviePlayer.cpp. 4-6 cont. The QMoviePlayer object source file: QMoviePlayer.cpp #include

In this file, there is also a QImage called m_qt_img and a cv::Mat called m_cv_img. These will contain the Qt and OpenCV representations of the image we are getting from our video. Finally, there is a QTimer, which is what will take the place of cv::waitKey(), allowing us to replay the video frames at the correct rate. The remaining functions will become clear as we look at their actual definitions in QMoviePlayer.cpp. 4-6 cont. The QMoviePlayer object source file: QMoviePlayer.cpp #include "QMoviePlayer.hpp" #include <QTimer> #include <QPainter>  QMoviePlayer::QMoviePlayer( QWidget *parent )   : QWidget( parent ) {   ui.setupUi( this ); } The top-level constructor for QMoviePlayer just calls the setup function, which was automatically built for us for the UI member. 4-6 cont. Open. bool QMoviePlayer::open( string file ) {    if( !m_cap.open( file ) ) return false;    // If we opened the file, set up everything now:   //   m_cap.read( m_cv_img );   m_qt_img = QImage(      QSize( m_cv_img.cols, m_cv_img.rows

#include <QTimer> #include <QPainter>  QMoviePlayer::QMoviePlayer( QWidget *parent )   : QWidget( parent ) {   ui.setupUi( this ); } The top-level constructor for QMoviePlayer just calls the setup function, which was automatically built for us for the UI member. 4-6 cont. Open. bool QMoviePlayer::open( string file ) {    if( !m_cap.open( file ) ) return false;    // If we opened the file, set up everything now:   //   m_cap.read( m_cv_img );   m_qt_img = QImage(      QSize( m_cv_img.cols, m_cv_img.rows ),      QImage::Format_RGB888    );   ui.frame->setMinimumSize( m_qt_img.width(), m_qt_img.height() );   ui.frame->setMaximumSize( m_qt_img.width(), m_qt_img.height() );   _copyImage();    m_timer = new QTimer( this );   connect(     m_timer,     SIGNAL( timeout() ),     this,     SLOT( nextFrame() )   );   m_timer->start( 1000. / m_cap.get( cv::CAP_PROP_FPS ) );

return true;  } When an open() call is made on the QMoviePlayer, several things have to happen. The first is that the cv::VideoCapture member object m_cap needs to be opened. If that fails, we just return. Next we read the first frame into our OpenCV image member m_cv_img. Once we have this, we can set up the Qt image object m_qt_img, giving it the same size as the OpenCV image. Next we resize the frame object in the UI element to be the same size as the incoming images as well. We will look at the call to QMoviePlayer::_copyImage() in a moment; this is going to handle the very important process of converting the image we have already captured into m_cv_img onto the Qt image m_qt_img, which we are actually going to have Qt paint onto the screen for us. The last thing we do in QMoviePlayer::open() is to set up a QTimer which, when it “goes off,” will call the function QMoviePlayer::nextFrame() (which will, not surprisingly, get the next frame). The call to m_timer->start() is how we both start the timer

to QMoviePlayer::_copyImage() in a moment; this is going to handle the very important process of converting the image we have already captured into m_cv_img onto the Qt image m_qt_img, which we are actually going to have Qt paint onto the screen for us. The last thing we do in QMoviePlayer::open() is to set up a QTimer which, when it “goes off,” will call the function QMoviePlayer::nextFrame() (which will, not surprisingly, get the next frame). The call to m_timer->start() is how we both start the timer running, and indicate that it should go off at the correct rate implied by cv::CAP_PROP_FPS (i.e., 1,000 milliseconds divided by the frame rate). 4-6 cont. cv::Mat to QImage. void QMoviePlayer::_copyImage( void ) {    // Copy the image data into the Qt QImage   //   cv::Mat cv_header_to_qt_image(     cv::Size(        m_qt_img.width(),       m_qt_img.height()     ),     cv::U8C3,     m_qt_img.bits()   );   cv::cvtColor( m_cv_img, cv_header_to_qt_image, cv::BGR2RGB ); } The function QMoviePlayer::_copyImage()

and indicate that it should go off at the correct rate implied by cv::CAP_PROP_FPS (i.e., 1,000 milliseconds divided by the frame rate). 4-6 cont. cv::Mat to QImage. void QMoviePlayer::_copyImage( void ) {    // Copy the image data into the Qt QImage   //   cv::Mat cv_header_to_qt_image(     cv::Size(        m_qt_img.width(),       m_qt_img.height()     ),     cv::U8C3,     m_qt_img.bits()   );   cv::cvtColor( m_cv_img, cv_header_to_qt_image, cv::BGR2RGB ); } The function QMoviePlayer::_copyImage() is responsible for copying the image from the buffer m_cv_img into the Qt image buffer m_qt_img. The way we do this shows off a nice feature of the cv::Mat object. We first define a cv::Mat object called cv_hreader_to_qt_image. When we define that object, we actually tell it what area to use for its data area, and hand it the data area for the Qt QImage object m_qt_img.bits(). We then call cv::cvtColor to do the copying, which handles the subtlety that OpenCV prefers BGR ordering, while Qt prefers RGB. 4-6 cont.

for copying the image from the buffer m_cv_img into the Qt image buffer m_qt_img. The way we do this shows off a nice feature of the cv::Mat object. We first define a cv::Mat object called cv_hreader_to_qt_image. When we define that object, we actually tell it what area to use for its data area, and hand it the data area for the Qt QImage object m_qt_img.bits(). We then call cv::cvtColor to do the copying, which handles the subtlety that OpenCV prefers BGR ordering, while Qt prefers RGB. 4-6 cont. nextFrame. void QMoviePlayer::nextFrame() {      // Nothing to do if capture object is not open   //   if( !m_cap.isOpened() ) return;    m_cap.read( m_cv_img );   _copyImage();    this->update(); } The QMoviePlayer::nextFrame() function actually handles the reading of subsequent frames. Recall that this routine is called whenever the QTimer expires. It reads the new image into the OpenCV buffer, calls QMoviePlayer::_copyImage() to copy it into the Qt buffer, and then makes an update call on the QWidget that this

void QMoviePlayer::nextFrame() {      // Nothing to do if capture object is not open   //   if( !m_cap.isOpened() ) return;    m_cap.read( m_cv_img );   _copyImage();    this->update(); } The QMoviePlayer::nextFrame() function actually handles the reading of subsequent frames. Recall that this routine is called whenever the QTimer expires. It reads the new image into the OpenCV buffer, calls QMoviePlayer::_copyImage() to copy it into the Qt buffer, and then makes an update call on the QWidget that this is all part of (so that Qt knows that something has changed).

4-6 cont. paintEvent. void QMoviePlayer::paintEvent( QPaintEvent* e ) {      QPainter painter( this );   painter.drawImage( QPoint( ui.frame->x(), ui.frame->y()), m_qt_img );  } Last, but not least, is the function QMoviePlayer::paintEvent(). This is the function that is called by Qt whenever it is necessary to actually draw the QMoviePlayer widget. This function just creates a QPainter and tells it to draw the current Qt image m_qt_img (starting at the corner of the screen). An Example of OpenCV and wxWidgets In this example, we will use a different cross-platform toolkit, wxWidgets. The wxWidgets toolkit is similar in many ways to Qt in terms of its GUI components, but of course, it is in the details that things tend to become difficult. As with the Qt example, we will have one top-level file that basically puts everything in place and a code and header file pair that define an object which encapsulates our example task of playing a video. This time our object will be called WxMoviePlayer, and we will

In this example, we will use a different cross-platform toolkit, wxWidgets. The wxWidgets toolkit is similar in many ways to Qt in terms of its GUI components, but of course, it is in the details that things tend to become difficult. As with the Qt example, we will have one top-level file that basically puts everything in place and a code and header file pair that define an object which encapsulates our example task of playing a video. This time our object will be called WxMoviePlayer, and we will build it based on the UI classes provided by wxWidgets.  Example 4-7: An example program ch4_wx.cpp, which takes a single argument indicating a video file; that video file will be replayed inside of a wxWidgets object which we will define called WxMoviePlayer #include "wx/wx.h" #include "WxMoviePlayer.hpp"  // Application class, the top level object in wxWidgets // class MyApp : public wxApp {   public:      virtual bool OnInit(); };  // Behind the scenes stuff to create a main() function and attach MyApp //

it based on the UI classes provided by wxWidgets.  Example 4-7: An example program ch4_wx.cpp, which takes a single argument indicating a video file; that video file will be replayed inside of a wxWidgets object which we will define called WxMoviePlayer #include "wx/wx.h" #include "WxMoviePlayer.hpp"  // Application class, the top level object in wxWidgets // class MyApp : public wxApp {   public:      virtual bool OnInit(); };  // Behind the scenes stuff to create a main() function and attach MyApp // DECLARE_APP( MyApp ); IMPLEMENT_APP( MyApp );  // When MyApp is initialized, do these things. // bool MyApp::OnInit() {    wxFrame* frame = new wxFrame( NULL, wxID_ANY, wxT("ch4_wx") );   frame->Show( true );   WxMoviePlayer* mp = new WxMoviePlayer(     frame,     wxPoint( -1, -1 ),     wxSize( 640, 480 )   );   mp->open( wxString(argv[1]) );   mp->Show( true );    return true;  } The structure here is a little more complicated in appearance than with the Qt example, but the content is very similar. The first

MyApp ); IMPLEMENT_APP( MyApp );  // When MyApp is initialized, do these things. // bool MyApp::OnInit() {    wxFrame* frame = new wxFrame( NULL, wxID_ANY, wxT("ch4_wx") );   frame->Show( true );   WxMoviePlayer* mp = new WxMoviePlayer(     frame,     wxPoint( -1, -1 ),     wxSize( 640, 480 )   );   mp->open( wxString(argv[1]) );   mp->Show( true );    return true;  } The structure here is a little more complicated in appearance than with the Qt example, but the content is very similar. The first thing we do is create a class definition for our application, which we derive from the library class wxApp. The only thing different about our class is that it will overload the MyApp::OnInit() function with our own content. After declaring class MyApp,  we call two macros DECLARE_APP() and IMPLEMENT_APP(). In short, these are creating the main() function,

and installing an instance of MyApp as “the application.”  The last thing we do in our main program is to actually fill out the function MyApp::OnInit(). This is going to get called when our program starts: this functions largely as the equivalant to main() from our point of view. This function creates the window (called a “frame” in wxWidgets), and an instance of our WxMoviePlayer object in that frame. It then calls the open method on the WxMoviePlayer and hands it the name of the movie file we want to open. Of course, all of the interesting stuff is happening inside of the WxMoviePlayer object. Here is the header file for that object: The WxMoviePlayer object header file WxMoviePlayer.hpp #include "opencv2/opencv.hpp"  #include "wx/wx.h" #include <string>  #define TIMER_ID 0  using namespace std;  class WxMoviePlayer : public wxWindow {    public:     WxMoviePlayer(       wxWindow*      parent,        const wxPoint& pos,       const wxSize&  size     );     virtual ~WxMoviePlayer() {};     bool open(

to open. Of course, all of the interesting stuff is happening inside of the WxMoviePlayer object. Here is the header file for that object: The WxMoviePlayer object header file WxMoviePlayer.hpp #include "opencv2/opencv.hpp"  #include "wx/wx.h" #include <string>  #define TIMER_ID 0  using namespace std;  class WxMoviePlayer : public wxWindow {    public:     WxMoviePlayer(       wxWindow*      parent,        const wxPoint& pos,       const wxSize&  size     );     virtual ~WxMoviePlayer() {};     bool open( wxString file );    private:       cv::VideoCapture m_cap;     cv::Mat          m_cv_img;     wxImage          m_wx_img;     wxBitmap         m_wx_bmp;     wxTimer*         m_timer;     wxWindow*        m_parent;      void _copyImage( void );      void OnPaint( wxPaintEvent& e );     void OnTimer( wxTimerEvent& e );     void OnKey(   wxKeyEvent&   e );    protected:     DECLARE_EVENT_TABLE(); }; The important things to notice in the above declaration are the following. The WxMoviePlayer object is derived

wxString file );    private:       cv::VideoCapture m_cap;     cv::Mat          m_cv_img;     wxImage          m_wx_img;     wxBitmap         m_wx_bmp;     wxTimer*         m_timer;     wxWindow*        m_parent;      void _copyImage( void );      void OnPaint( wxPaintEvent& e );     void OnTimer( wxTimerEvent& e );     void OnKey(   wxKeyEvent&   e );    protected:     DECLARE_EVENT_TABLE(); }; The important things to notice in the above declaration are the following. The WxMoviePlayer object is derived from wxWindow, which is the generic class used by wxWidgets for just about anything that will be visible on the screen. We have three event-handling methods OnPaint(), onTimer(), and OnKey(). These will handle drawing, getting a new image from the video, and closing the file with the ESC key, respectively. Finally, you will notice that there is an object of type wxImage and an object of type wxBitmap, in addition to the OpenCV cv:Mat type image. In wxWidgets, bitmaps (which are operating system dependant)

from wxWindow, which is the generic class used by wxWidgets for just about anything that will be visible on the screen. We have three event-handling methods OnPaint(), onTimer(), and OnKey(). These will handle drawing, getting a new image from the video, and closing the file with the ESC key, respectively. Finally, you will notice that there is an object of type wxImage and an object of type wxBitmap, in addition to the OpenCV cv:Mat type image. In wxWidgets, bitmaps (which are operating system dependant) are distinguished from “images” (which are device independent representations of image data). The exact role of these two will be clear shortly as we look at the code file WxMoviePlayer.cpp. The WxMoviePlayer object source file WxMoviePlayer.cpp

#include "WxMoviePlayer.hpp"  BEGIN_EVENT_TABLE( WxMoviePlayer, wxWindow )   EVT_PAINT( WxMoviePlayer::OnPaint )   EVT_TIMER( TIMER_ID, WxMoviePlayer::OnTimer )   EVT_CHAR( WxMoviePlayer::OnKey ) END_EVENT_TABLE() The first thing we do is to set up the callbacks that will be associated with individual events. This is done by macros provided by the wxWidgets framework.25 WxMoviePlayer::WxMoviePlayer(   wxWindow*      parent,   const wxPoint& pos,   const wxSize&  size ) : wxWindow( parent, -1, pos, size, wxSIMPLE_BORDER ) {   m_timer         = NULL;   m_parent        = parent; } When the movie player is created, its timer element is NULL (we will set that up when we actually have a video open). We do take note of the parent of the player, however. (In this case, that parent will be the wxFrame we created to put it in.) We will need to know who the parent frame is when it comes time to closing the application in response to the ESC key. void WxMoviePlayer::OnPaint( wxPaintEvent& event ) {   wxPaintDC dc( this

) {   m_timer         = NULL;   m_parent        = parent; } When the movie player is created, its timer element is NULL (we will set that up when we actually have a video open). We do take note of the parent of the player, however. (In this case, that parent will be the wxFrame we created to put it in.) We will need to know who the parent frame is when it comes time to closing the application in response to the ESC key. void WxMoviePlayer::OnPaint( wxPaintEvent& event ) {   wxPaintDC dc( this );    if( !dc.Ok() ) return;    int x,y,w,h;   dc.BeginDrawing();     dc.GetClippingBox( &x, &y, &w, &h );     dc.DrawBitmap( m_wx_bmp, x, y );   dc.EndDrawing();    return; } The WxMoviePlayer::OnPaint() routine is called whenever the window needs to be repainted on screen. You will notice that when we execute WxMoviePlayer::OnPaint(), the information we need to actually do the painting is assumed to be in m_wx_bpm, the wxBitmap object. Because the wxBitmap is the system dependant representation, it is already prepared

) return;    int x,y,w,h;   dc.BeginDrawing();     dc.GetClippingBox( &x, &y, &w, &h );     dc.DrawBitmap( m_wx_bmp, x, y );   dc.EndDrawing();    return; } The WxMoviePlayer::OnPaint() routine is called whenever the window needs to be repainted on screen. You will notice that when we execute WxMoviePlayer::OnPaint(), the information we need to actually do the painting is assumed to be in m_wx_bpm, the wxBitmap object. Because the wxBitmap is the system dependant representation, it is already prepared to be copied to the screen. The next two methods, WxMoviePlayer::_copyImage()and WxMoviePlayer::open() will show how it got created in the first place. void WxMoviePlayer::_copyImage( void ) {   m_wx_bmp = wxBitmap( m_wx_img );    Refresh( FALSE ); // indicate that the object is dirty   Update(); }                                                                25 The astute reader will notice that the keyboard event is “hooked up” to the WxMoviePlayer widget and not to the top-level application or the frame (as

copied to the screen. The next two methods, WxMoviePlayer::_copyImage()and WxMoviePlayer::open() will show how it got created in the first place. void WxMoviePlayer::_copyImage( void ) {   m_wx_bmp = wxBitmap( m_wx_img );    Refresh( FALSE ); // indicate that the object is dirty   Update(); }                                                                25 The astute reader will notice that the keyboard event is “hooked up” to the WxMoviePlayer widget and not to the top-level application or the frame (as was the case for the Qt example, and is the case for HighGUI). There are various ways to accomplish this, but wxWidgets really prefers your keyboard events to be bound locally to visible objects in your UI, rather than globally. Since this is a simple example, we chose to just do the easiest thing and bind the keyboard events directly to the movie player.

The WxMoviePlayer::_copyImage() method is what is going to get called whenever a new image is read from the cv::VideoCapture object. It does not appear to do much, but actually a lot is going on in its short body. First and foremost is the construction of the wxBitmap m_wx_bmp from the wxImage m_wx_img. The constructor is handling the conversion from the abstract representation used by wxImage (which, we will see, looks very much like the representation used by OpenCV) to the device and system specific representation used by your particular machine. Once that copy is done, a call to Refresh() indicates that the widget is “dirty” and needs redrawing, and the subsequent call to Update() actually indicates that the time for that redrawing is now. bool WxMoviePlayer::open( wxString file ) {   if( !m_cap.open( std::string( file.mb_str() ) )) {     return false;   }    // If we opened the file, set up everything now:   //   m_cap.read( m_cv_img );    m_wx_img = wxImage(      m_cv_img.cols,     m_cv_img.rows,

used by your particular machine. Once that copy is done, a call to Refresh() indicates that the widget is “dirty” and needs redrawing, and the subsequent call to Update() actually indicates that the time for that redrawing is now. bool WxMoviePlayer::open( wxString file ) {   if( !m_cap.open( std::string( file.mb_str() ) )) {     return false;   }    // If we opened the file, set up everything now:   //   m_cap.read( m_cv_img );    m_wx_img = wxImage(      m_cv_img.cols,     m_cv_img.rows,     m_cv_img.data,     TRUE  // static data, do not free on delete()   );    _copyImage();    m_timer = new wxTimer( this, TIMER_ID );   m_timer->Start( 1000. / m_cap.get( cv::CAP_PROP_FPS ) );    return true; } The WxMoviePlayer::open() method also does several important things. The first is to actually open the cv::VideoCapture object, but there is a lot more to be done. Next, an image is read off of the player, and it is used to create a wxImage object which “points at” the OpenCV cv::Mat image. This is the opposite

TRUE  // static data, do not free on delete()   );    _copyImage();    m_timer = new wxTimer( this, TIMER_ID );   m_timer->Start( 1000. / m_cap.get( cv::CAP_PROP_FPS ) );    return true; } The WxMoviePlayer::open() method also does several important things. The first is to actually open the cv::VideoCapture object, but there is a lot more to be done. Next, an image is read off of the player, and it is used to create a wxImage object which “points at” the OpenCV cv::Mat image. This is the opposite philosophy to the one we used in the Qt example: in this case, it turns out to be a little more convenient to create the cv::Mat first and have it own the data, and the GUI toolkit’s image object second and have it be just a header to that existing data. Next, we call WxMoviePlayer::_copyImage(), and that function converts the OpenCV image m_cv_img into the native bitmap for us. Finally, we create a wxTimer object and tell it to wake us up every few milliseconds—with that number being computed from the FPS

philosophy to the one we used in the Qt example: in this case, it turns out to be a little more convenient to create the cv::Mat first and have it own the data, and the GUI toolkit’s image object second and have it be just a header to that existing data. Next, we call WxMoviePlayer::_copyImage(), and that function converts the OpenCV image m_cv_img into the native bitmap for us. Finally, we create a wxTimer object and tell it to wake us up every few milliseconds—with that number being computed from the FPS reported by the cv::VideoCapture object. Whenever that timer expires, a wxTimerEvent is generated and passed to WxMoviePlayer::OnTimer(), which you will recall to be the handler of such events. void WxMoviePlayer::OnTimer( wxTimerEvent& event ) {    if( !m_cap.isOpened() ) return;   m_cap.read( m_cv_img );   cv::cvtColor( m_cv_img, m_cv_img, cv::BGR2RGB );   _copyImage(); } That handler does not do too much; primarily it just iterates the reading of a new frame from the video and the conversion of that

FPS reported by the cv::VideoCapture object. Whenever that timer expires, a wxTimerEvent is generated and passed to WxMoviePlayer::OnTimer(), which you will recall to be the handler of such events. void WxMoviePlayer::OnTimer( wxTimerEvent& event ) {    if( !m_cap.isOpened() ) return;   m_cap.read( m_cv_img );   cv::cvtColor( m_cv_img, m_cv_img, cv::BGR2RGB );   _copyImage(); } That handler does not do too much; primarily it just iterates the reading of a new frame from the video and the conversion of that frame from BGR to RGB for display, and then calls our WxMoviePlayer::_copyImage(), which makes the next bitmap for us. void WxMoviePlayer::OnKey( wxKeyEvent& e ) {   if( e.GetKeyCode() == WXK_ESCAPE ) m_parent->Close(); }

Finally, we have our handler for any key-presses. It simply checks to see if that key was the Escape key, and if so, closes the program. It is worth noting that we do not close the WxMoviePlayer object, but rather the parent frame. Closing the frame is the same as closing the window any other way; it shuts down the application. An Example of OpenCV and the Windows Template Library In this example, we will use the native windows GUI API.26 The Windows Template Library, WTL, is a very thin C++ wrapper around the raw Win32 API. WTL applications are structured similarly to MFC, in that there is an application/document-view structure. For the purposes of this sample, we will start by running the WTL Application Wizard from within Visual Studio, and creating a new “SDI Application” and under “User Interface Features” ensuring “Use a View Window” is selected (it should be, by default).  Figure 4-7: The WTL Application Wizard The exact file names generated will depend on the name you give your project. For this

the raw Win32 API. WTL applications are structured similarly to MFC, in that there is an application/document-view structure. For the purposes of this sample, we will start by running the WTL Application Wizard from within Visual Studio, and creating a new “SDI Application” and under “User Interface Features” ensuring “Use a View Window” is selected (it should be, by default).  Figure 4-7: The WTL Application Wizard The exact file names generated will depend on the name you give your project. For this example, the project is named OpenCVTest, and we will mostly be working in the COpenCVTestView class. Example 4-8: An example header file for our custom View class class COpenCVTestView : public CWindowImpl<COpenCVTestView> {  public:   DECLARE_WND_CLASS(NULL)    bool OpenFile(std::string file);   void _copyImage();     BOOL PreTranslateMessage(MSG* pMsg);    BEGIN_MSG_MAP(COpenCVTestView)     MESSAGE_HANDLER(WM_ERASEBKGND, OnEraseBkgnd)     MESSAGE_HANDLER(WM_PAINT, OnPaint)

the project is named OpenCVTest, and we will mostly be working in the COpenCVTestView class. Example 4-8: An example header file for our custom View class class COpenCVTestView : public CWindowImpl<COpenCVTestView> {  public:   DECLARE_WND_CLASS(NULL)    bool OpenFile(std::string file);   void _copyImage();     BOOL PreTranslateMessage(MSG* pMsg);    BEGIN_MSG_MAP(COpenCVTestView)     MESSAGE_HANDLER(WM_ERASEBKGND, OnEraseBkgnd)     MESSAGE_HANDLER(WM_PAINT, OnPaint)                                                              26 Special thanks to Sam Leventer, who is the original author of this WTL example code.

MESSAGE_HANDLER(WM_TIMER, OnTimer)   END_MSG_MAP()  // Handler prototypes (uncomment arguments if needed): // LRESULT MessageHandler( //    UINT    /*uMsg*/,  //    WPARAM  /*wParam*/,  //    LPARAM  /*lParam*/,  //    BOOL&   /*bHandled*/ //  ); // LRESULT CommandHandler( //    WORD    /*wNotifyCode*/,  //    WORD    /*wID*/,  //    HWND    /*hWndCtl*/,  //    BOOL&   /*bHandled*/ //  ); // LRESULT NotifyHandler( //    int     /*idCtrl*/,  //    LPNMHDR /*pnmh*/,  //    BOOL&   /*bHandled*/ //  );    LRESULT OnPaint(     UINT    /*uMsg*/,      WPARAM  /*wParam*/,      LPARAM  /*lParam*/,      BOOL&   /*bHandled*/   );   LRESULT OnTimer(     UINT    /*uMsg*/,      WPARAM  /*wParam*/,      LPARAM  /*lParam*/,      BOOL&   /*bHandled*/   );   LRESULT OnEraseBkgnd(     UINT    /*uMsg*/,      WPARAM  /*wParam*/,      LPARAM  /*lParam*/,      BOOL&   /*bHandled*/   );  private:   cv::VideoCapture m_cap;   cv::Mat          m_cv_img;    RGBTRIPLE*       m_bitmapBits; }; The structure here is very similar to

LRESULT OnPaint(     UINT    /*uMsg*/,      WPARAM  /*wParam*/,      LPARAM  /*lParam*/,      BOOL&   /*bHandled*/   );   LRESULT OnTimer(     UINT    /*uMsg*/,      WPARAM  /*wParam*/,      LPARAM  /*lParam*/,      BOOL&   /*bHandled*/   );   LRESULT OnEraseBkgnd(     UINT    /*uMsg*/,      WPARAM  /*wParam*/,      LPARAM  /*lParam*/,      BOOL&   /*bHandled*/   );  private:   cv::VideoCapture m_cap;   cv::Mat          m_cv_img;    RGBTRIPLE*       m_bitmapBits; }; The structure here is very similar to the preceding wx example. The only change outside of the view code is for the Open menu item handler, which will be in your CMainFrame class. It will need to call into the view class to open the video:  LRESULT CMainFrame::OnFileOpen(   WORD /*wNotifyCode*/,    WORD /*wID*/,    HWND /*hWndCtl*/,    BOOL& /*bHandled*/ ) {     WTL::CFileDialog dlg(TRUE);   if (IDOK == dlg.DoModal(m_hWnd)) {     m_view.OpenFile(dlg.m_szFileName);   }

return 0; }  bool COpenCVTestView::OpenFile(std::string file) {    if( !m_cap.open( file ) ) return false;    // If we opened the file, set up everything now:   //   m_cap.read( m_cv_img );    // could create a DIBSection here, but lets just allocate memory for the raw bits   m_bitmapBits = new RGBTRIPLE[m_cv_img.cols * m_cv_img.rows];    _copyImage();    SetTimer(0, 1000.0f / m_cap.get( cv::CAP_PROP_FPS ) );    return true; }   void COpenCVTestView::_copyImage() {    // Copy the image data into the bitmap   //   cv::Mat cv_header_to_qt_image(     cv::Size(       m_cv_img.cols,       m_cv_img.rows     ),     CV::U8C3,     m_bitmapBits   );   cv::cvtColor( m_cv_img, cv_header_to_qt_image, cv::BGR2RGB ); }  LRESULT COpenCVTestView::OnPaint(   UINT   /*uMsg*/,    WPARAM /*wParam*/,    LPARAM /*lParam*/,    BOOL&  /*bHandled*/ ) {   CPaintDC dc(m_hWnd);    WTL::CRect rect;   GetClientRect(&rect);    if( m_cap.isOpened() ) {     BITMAPINFO bmi = {0};     bmi.bmiHeader.biSize = sizeof(bmi.bmiHeader);

bitmap   //   cv::Mat cv_header_to_qt_image(     cv::Size(       m_cv_img.cols,       m_cv_img.rows     ),     CV::U8C3,     m_bitmapBits   );   cv::cvtColor( m_cv_img, cv_header_to_qt_image, cv::BGR2RGB ); }  LRESULT COpenCVTestView::OnPaint(   UINT   /*uMsg*/,    WPARAM /*wParam*/,    LPARAM /*lParam*/,    BOOL&  /*bHandled*/ ) {   CPaintDC dc(m_hWnd);    WTL::CRect rect;   GetClientRect(&rect);    if( m_cap.isOpened() ) {     BITMAPINFO bmi = {0};     bmi.bmiHeader.biSize = sizeof(bmi.bmiHeader);     bmi.bmiHeader.biCompression = BI_RGB;     bmi.bmiHeader.biWidth = m_cv_img.cols;      // note that bitmaps default to bottom-up, use negative height to      // represent top-down     bmi.bmiHeader.biHeight = m_cv_img.rows * -1;          bmi.bmiHeader.biPlanes = 1;     bmi.bmiHeader.biBitCount = 24;  // 32 if you use RGBQUADs for the bits      dc.StretchDIBits(       0,                     0,        rect.Width(),          rect.Height(),

0,                     0,        bmi.bmiHeader.biWidth, abs(bmi.bmiHeader.biHeight),        m_bitmapBits,        &bmi,        DIB_RGB_COLORS,        SRCCOPY     );   } else {     dc.FillRect(rect, COLOR_WINDOW);   }    return 0; }  LRESULT COpenCVTestView::OnTimer(   UINT   /*uMsg*/,    WPARAM /*wParam*/,    LPARAM /*lParam*/,    BOOL&  /*bHandled*/ ) {   // Nothing to do if capture object is not open   //   if( !m_cap.isOpened() ) return 0;    m_cap.read( m_cv_img );   _copyImage();    Invalidate();    return 0; }  LRESULT COpenCVTestView::OnEraseBkgnd(   UINT   /*uMsg*/,    WPARAM /*wParam*/,    LPARAM /*lParam*/,    BOOL&  /*bHandled*/ ) {   // since we completely paint our window in the OnPaint handler, use    // an empty background handler   return 0; } This code illustrates how to use bitmap-based drawing in a C++ application under Windows. This method is simpler, but less efficient than using DirectShow to handle the video stream. If you are using the .NET Runtime (either through C#, VB.NET or

return 0; }  LRESULT COpenCVTestView::OnEraseBkgnd(   UINT   /*uMsg*/,    WPARAM /*wParam*/,    LPARAM /*lParam*/,    BOOL&  /*bHandled*/ ) {   // since we completely paint our window in the OnPaint handler, use    // an empty background handler   return 0; } This code illustrates how to use bitmap-based drawing in a C++ application under Windows. This method is simpler, but less efficient than using DirectShow to handle the video stream. If you are using the .NET Runtime (either through C#, VB.NET or Managed C++), then you may want to look into a package that completely wraps OpenCV, such as Emgu (http://emgu.com).  Drawing Things We often want to draw some kind of picture, or to draw something on top of an image obtained from somewhere else. Toward this end, OpenCV provides a menagerie of functions that will allow us to make lines, squares, circles, and the like. The drawing functions available will work with images of any depth, but most of them only affect the first three channels—defaulting to only

Managed C++), then you may want to look into a package that completely wraps OpenCV, such as Emgu (http://emgu.com).  Drawing Things We often want to draw some kind of picture, or to draw something on top of an image obtained from somewhere else. Toward this end, OpenCV provides a menagerie of functions that will allow us to make lines, squares, circles, and the like. The drawing functions available will work with images of any depth, but most of them only affect the first three channels—defaulting to only the first channel in the case of single-channel images. Most of the drawing functions support a color, a thickness, what is called a “line type” (which really means whether or not to anti-alias lines), and subpixel alignment of objects.

When specifying colors, the convention is to use the cv::Scalar object, even though only the first three values are used most of the time. (It is sometimes convenient to be able to use the fourth value in a cv::Scalar to represent an alpha-channel, but the drawing functions do not currently support alpha blending.)  Also, by convention, OpenCV uses BGR ordering27 for converting multichannel images to color renderings (this is what is used by the draw functions imshow(), which actually paints images onto your screen for viewing). Of course, you don’t have to use this convention, and it might not be ideal if you are using data from some other library with OpenCV headers on top of it. In any case, the core functions of the library are always agnostic to any “meaning” you might assign to a channel. Line Art and Filled Polygons Functions that draw lines of one kind or another (segments, circles, rectangles, etc.) will usually accept a thickness and lineType parameter. Both are integers, but the only accepted

your screen for viewing). Of course, you don’t have to use this convention, and it might not be ideal if you are using data from some other library with OpenCV headers on top of it. In any case, the core functions of the library are always agnostic to any “meaning” you might assign to a channel. Line Art and Filled Polygons Functions that draw lines of one kind or another (segments, circles, rectangles, etc.) will usually accept a thickness and lineType parameter. Both are integers, but the only accepted values for the latter are 4, 8, or cv::AA. thickness will be the thickness of the line measured in pixels. For circles, rectangles, and all of the other closed shapes, the thickness argument can also be set to cv::FILL (which is an alias for −1). In that case, the result is that the drawn figure will be filled in the same color as the edges. The lineType argument indicates whether the lines should be “4-connected,” “8-connected,” or anti-aliased. For the first two, the Bresenham algorithm is used, while for

for the latter are 4, 8, or cv::AA. thickness will be the thickness of the line measured in pixels. For circles, rectangles, and all of the other closed shapes, the thickness argument can also be set to cv::FILL (which is an alias for −1). In that case, the result is that the drawn figure will be filled in the same color as the edges. The lineType argument indicates whether the lines should be “4-connected,” “8-connected,” or anti-aliased. For the first two, the Bresenham algorithm is used, while for the anti-aliased lines, Gaussian filtering is used. Wide lines are always drawn with rounded ends.   Figure 4-8: The same line as it would be rendered using the 4-connected (a), 8-connected (b), and anti-aliased (c) line types For the drawing algorithms, endpoints (lines), center-points (circles), corners (rectangles), and so on are typically specified as integers. However, these algorithms support subpixel alignment, through the shift argument. Where shift is available, it is interpreted as the number of bits

anti-aliased lines, Gaussian filtering is used. Wide lines are always drawn with rounded ends.   Figure 4-8: The same line as it would be rendered using the 4-connected (a), 8-connected (b), and anti-aliased (c) line types For the drawing algorithms, endpoints (lines), center-points (circles), corners (rectangles), and so on are typically specified as integers. However, these algorithms support subpixel alignment, through the shift argument. Where shift is available, it is interpreted as the number of bits in the integer arguments to treat as fractional bits. For example, if you say you want a circle centered at (5,5), but set shift to 1, then the circle will be drawn at (2.5,2.5). The effect of this will typically be quite subtle, and depend on the line type used. The effect is most noticeable for anti-aliased lines. Table 4-9:Drawing Functions Function Description cv::circle() Draw a simple circle                                                              27 There is a slightly confusing point here,

bits in the integer arguments to treat as fractional bits. For example, if you say you want a circle centered at (5,5), but set shift to 1, then the circle will be drawn at (2.5,2.5). The effect of this will typically be quite subtle, and depend on the line type used. The effect is most noticeable for anti-aliased lines. Table 4-9:Drawing Functions Function Description cv::circle() Draw a simple circle                                                              27 There is a slightly confusing point here, which is mostly due to legacy in origin. That is that the macro CV_RGB(r,g,b) produces a cv::Scalar s with value s.val[] = { b, g, r, 0 }. This is as it should be, as general OpenCV functions only know what is red, green, or blue by the order, and the ordering convention for image data is BGR as stated in the text.

cv::clipLine() Determine if a line is inside a given box cv::ellipse() Draw an ellipse, which may be tilted or an elliptical arc cv::ellipse2Poly()  Compute a polygon approximation to an elliptical arc cv::fillConvexPoly()                Fast algorithm for drawing filled versions of simple polygons cv::fillPoly() General algorithm for drawing filled versions of arbitrary polygons cv::line() Draw a simple line cv::rectangle()                Draw a simple rectangle cv::polyLines() Draw multiple polygonal curves cv::circle() void circle(    cv::Mat&          img,                  // Image to be drawn on   cv::Point         center,               // Location of circle center   int               radius,               // Radius of circle   const cv::Scalar& color,                // Color, RGB form   int               thickness = 1,        // Thickness of line    int               lineType  = 8,        // Connectedness, 4 or 8   int               shift     = 0         // Bits of radius to treat as fraction ); The

cv::circle() void circle(    cv::Mat&          img,                  // Image to be drawn on   cv::Point         center,               // Location of circle center   int               radius,               // Radius of circle   const cv::Scalar& color,                // Color, RGB form   int               thickness = 1,        // Thickness of line    int               lineType  = 8,        // Connectedness, 4 or 8   int               shift     = 0         // Bits of radius to treat as fraction ); The first argument to cv::circle() is just your image img. Next, are the center, a two-dimensional point, and the radius. The remaining arguments are the standard color, thickness, lineType, and shift. The shift is applied to both the radius and the center location. cv::clipLine() bool clipLine(                            // True if any part of line in ‘imgRect’   cv::Rect          imgRect,              // Rectangle to clip to   cv::Point&        pt1,                  // First end-point of line, overwritten

first argument to cv::circle() is just your image img. Next, are the center, a two-dimensional point, and the radius. The remaining arguments are the standard color, thickness, lineType, and shift. The shift is applied to both the radius and the center location. cv::clipLine() bool clipLine(                            // True if any part of line in ‘imgRect’   cv::Rect          imgRect,              // Rectangle to clip to   cv::Point&        pt1,                  // First end-point of line, overwritten   cv::Point&        pt2                   // Second end-point of line, overwritten );   bool clipLine(                            // True if any part of line in image size   cv::Size          imgSize,              // Size of image, implies rectangle at 0,0   cv::Point&        pt1,                  // First end-point of line, overwritten   cv::Point&        pt2                   // Second end-point of line, overwritten ); This function is used to determine if a line, specified by the two points pt1 and pt2

cv::Point&        pt2                   // Second end-point of line, overwritten );   bool clipLine(                            // True if any part of line in image size   cv::Size          imgSize,              // Size of image, implies rectangle at 0,0   cv::Point&        pt1,                  // First end-point of line, overwritten   cv::Point&        pt2                   // Second end-point of line, overwritten ); This function is used to determine if a line, specified by the two points pt1 and pt2 lies inside of a rectangular boundary. In the first version, a cv::Rect is supplied and the line is compared to that rectangle. cv::clipLine() will return False only if the line is entirely outside of the specified rectangular region. The second version is the same, except that it takes a cv::Size argument. Calling this second version is equivalent to calling the first version with a rectangle whose (𝑥,𝑦) location is (0,0). cv::ellipse() bool ellipse(   cv::Mat&               img,             // Image to be

inside of a rectangular boundary. In the first version, a cv::Rect is supplied and the line is compared to that rectangle. cv::clipLine() will return False only if the line is entirely outside of the specified rectangular region. The second version is the same, except that it takes a cv::Size argument. Calling this second version is equivalent to calling the first version with a rectangle whose (𝑥,𝑦) location is (0,0). cv::ellipse() bool ellipse(   cv::Mat&               img,             // Image to be drawn on   cv::Point              center,          // Location of ellipse center   cv::Size               axes,            // Length of major and minor axes   double                 angle,           // Tilt angle of major axis   double                 startAngle,      // Start angle for arc drawing   double                 endAngle,        // End angle for arc drawing

const cv::Scalar&      color,           // Color, BGR form   int                    thickness = 1,   // Thickness of line    int                    lineType  = 8,   // Connectedness, 4 or 8   int                    shift     = 0    // Bits of radius to treat as fraction );  bool ellipse(   cv::Mat&               img,             // Image to be drawn on   const cv::RotatedRect& rect,            // Rotated rectangle bounds ellipse    const cv::Scalar&      color,           // Color, BGR form   int                    thickness = 1,   // Thickness of line    int                    lineType  = 8,   // Connectedness, 4 or 8   int                    shift     = 0    // Bits of radius to treat as fraction ); The cv::ellipse() function is very similar to the cv::circle() function, with the primary difference being the axes argument, which is of type cv::Size. In this case, the height and width arguments represent the length of the ellipse’s major and minor axes. The angle is the angle (in degrees) of the major

thickness = 1,   // Thickness of line    int                    lineType  = 8,   // Connectedness, 4 or 8   int                    shift     = 0    // Bits of radius to treat as fraction ); The cv::ellipse() function is very similar to the cv::circle() function, with the primary difference being the axes argument, which is of type cv::Size. In this case, the height and width arguments represent the length of the ellipse’s major and minor axes. The angle is the angle (in degrees) of the major axis, which is measured counterclockwise from horizontal (i.e., from the x-axis). Similarly, the startAngle and endAngle indicate (also in degrees) the angle for the arc to start and for it to finish. Thus, for a complete ellipse, you must set these values to 0 and 360, respectively.   Figure 4-8: An elliptical arc specified by the major and minor axes with tilt angle (left); a similar ellipse specified using a cv::RotatedRect (right). The alternate way to specify the drawing of an ellipse is to use a

axis, which is measured counterclockwise from horizontal (i.e., from the x-axis). Similarly, the startAngle and endAngle indicate (also in degrees) the angle for the arc to start and for it to finish. Thus, for a complete ellipse, you must set these values to 0 and 360, respectively.   Figure 4-8: An elliptical arc specified by the major and minor axes with tilt angle (left); a similar ellipse specified using a cv::RotatedRect (right). The alternate way to specify the drawing of an ellipse is to use a bounding box. In this case, the argument box of type cv::RotatedRect completely specifies both the size and the orientation of the ellipse. Both methods of specifying an ellipse are illustrated in Figure 4-8. cv::ellipse2Poly() void ellipse2Poly(   cv::Point              center,          // Location of ellipse center   cv::Size               axes,            // Length of major and minor axes   double                 angle,           // Tilt angle of major axis   double                 startAngle,      // Start

box. In this case, the argument box of type cv::RotatedRect completely specifies both the size and the orientation of the ellipse. Both methods of specifying an ellipse are illustrated in Figure 4-8. cv::ellipse2Poly() void ellipse2Poly(   cv::Point              center,          // Location of ellipse center   cv::Size               axes,            // Length of major and minor axes   double                 angle,           // Tilt angle of major axis   double                 startAngle,      // Start angle for arc drawing   double                 endAngle,        // End angle for arc drawing   int                    delta,           // Angle between sequential vertices   vector<cv::Point>&     pts              // Result, STL-vector of points );

The cv::ellipse2Poly() function is used by cv::ellipse() internally to compute elliptical arcs, but you can call it yourself as well. Given information about an elliptical arc (center, axes, angle, startAngle, and endAngle—all as defined in cv::ellipse()) and a parameter delta, which specifies the angle between subsequent points you want to sample, cv::ellipse2Poly() computes a sequence of points that form a polygonal approximation to the elliptical arc you specified. The computed points are returned in the vector<> pts. cv::fillConvexPoly() void fillConvexPoly(   cv::Mat&          img,                  // Image to be drawn on   const cv::Point*  pts,                  // C-style array of points   int               npts,                 // Number of points in ‘pts’   const cv::Scalar& color,                // Color, BGR form    int               lineType  = 8,        // Connectedness, 4 or 8   int               shift     = 0         // Bits of radius to treat as fraction ); This function draws a filled

in the vector<> pts. cv::fillConvexPoly() void fillConvexPoly(   cv::Mat&          img,                  // Image to be drawn on   const cv::Point*  pts,                  // C-style array of points   int               npts,                 // Number of points in ‘pts’   const cv::Scalar& color,                // Color, BGR form    int               lineType  = 8,        // Connectedness, 4 or 8   int               shift     = 0         // Bits of radius to treat as fraction ); This function draws a filled polygon. It is much faster than cv::fillPoly() because it uses a much simpler algorithm. This algorithm, however, will not work correctly if the polygon you pass to it has self-intersections.28 The points in pts are treated as sequential, and a segment from the last point in pts and the first point is implied (i.e., the polygon is assumed to be closed). cv::fillPoly() void fillPoly(   cv::Mat&          img,                  // Image to be drawn on   const cv::Point*  pts,                  // C-style array

polygon. It is much faster than cv::fillPoly() because it uses a much simpler algorithm. This algorithm, however, will not work correctly if the polygon you pass to it has self-intersections.28 The points in pts are treated as sequential, and a segment from the last point in pts and the first point is implied (i.e., the polygon is assumed to be closed). cv::fillPoly() void fillPoly(   cv::Mat&          img,                  // Image to be drawn on   const cv::Point*  pts,                  // C-style array of arrays of points   int               npts,                 // Number of points in ‘pts[i]’   int               ncontours,            // Number of arrays in ‘pts’   const cv::Scalar& color,                // Color, BGR form    int               lineType = 8,         // Connectedness, 4 or 8   int               shift    = 0,         // Bits of radius to treat as fraction   cv::Point         offset   = Point()    // Uniform offset applied to all points ); This function draws any number of filled polygons.

of arrays of points   int               npts,                 // Number of points in ‘pts[i]’   int               ncontours,            // Number of arrays in ‘pts’   const cv::Scalar& color,                // Color, BGR form    int               lineType = 8,         // Connectedness, 4 or 8   int               shift    = 0,         // Bits of radius to treat as fraction   cv::Point         offset   = Point()    // Uniform offset applied to all points ); This function draws any number of filled polygons. Unlike cv::fillConvexPoly(), it can handle polygons with self-intersections. The argument ncontours specifies how many different polygon contours there will be, and the argument npts is a C-style array that indicates how many points there are in each contour (i.e., npts[i] indicates how many points there are in polygon i). pts is a C-style array of C-style arrays containing all of the points in those polygons (i.e., pts[i][j] contains the 𝑗!! point in the 𝑖!! polygon.)  cv::fillPoly() also has one

Unlike cv::fillConvexPoly(), it can handle polygons with self-intersections. The argument ncontours specifies how many different polygon contours there will be, and the argument npts is a C-style array that indicates how many points there are in each contour (i.e., npts[i] indicates how many points there are in polygon i). pts is a C-style array of C-style arrays containing all of the points in those polygons (i.e., pts[i][j] contains the 𝑗!! point in the 𝑖!! polygon.)  cv::fillPoly() also has one additional argument offset, which is a pixel offset that will be applied to all vertex locations when the polygons are drawn. The polygons are assumed to be closed (i.e., a segment from the last element of pts[i][] to the first element will be assumed).  cv::line() void line(   cv::Mat&          img,                  // Image to be drawn on   cv::Point         pt1,                  // First end-point of line   cv::Point         pt2                   // Second end-point of line   const cv::Scalar& color,

argument offset, which is a pixel offset that will be applied to all vertex locations when the polygons are drawn. The polygons are assumed to be closed (i.e., a segment from the last element of pts[i][] to the first element will be assumed).  cv::line() void line(   cv::Mat&          img,                  // Image to be drawn on   cv::Point         pt1,                  // First end-point of line   cv::Point         pt2                   // Second end-point of line   const cv::Scalar& color,                // Color, BGR form    int               lineType = 8,         // Connectedness, 4 or 8   int               shift    = 0          // Bits of radius to treat as fraction );                                                              28 The algorithm use by cv::fillComvexPolyfillConvexPoly() is actually somewhat more general than implied here. It will correctly draw any polygon whose contour intersects every horizontal line at most twice (though it is allowed for the top or bottom of the polygon to be flat

// Color, BGR form    int               lineType = 8,         // Connectedness, 4 or 8   int               shift    = 0          // Bits of radius to treat as fraction );                                                              28 The algorithm use by cv::fillComvexPolyfillConvexPoly() is actually somewhat more general than implied here. It will correctly draw any polygon whose contour intersects every horizontal line at most twice (though it is allowed for the top or bottom of the polygon to be flat with respect to the horizontal). Such a polygon is said to be “monotone with respect to the horizontal.”

The function cv::line() draws a straight line from pt1 to pt2 in the image img. Lines are automatically clipped by the image boundaries. cv::rectangle() void rectangle(   cv::Mat&          img,                  // Image to be drawn on   cv::Point         pt1,                  // First corner of rectangle   cv::Point         pt2                   // Opposite corner of rectangle   const cv::Scalar& color,                // Color, BGR form    int               lineType = 8,         // Connectedness, 4 or 8   int               shift    = 0          // Bits of radius to treat as fraction );  void rectangle(   cv::Mat&          img,                  // Image to be drawn on   cv::Rect          r,                    // Rectangle to draw   const cv::Scalar& color,                // Color, BGR form    int               lineType = 8,         // Connectedness, 4 or 8   int               shift    = 0          // Bits of radius to treat as fraction ); The function cv::rectangle() draws a rectangle with corners pt1 to pt2

int               shift    = 0          // Bits of radius to treat as fraction );  void rectangle(   cv::Mat&          img,                  // Image to be drawn on   cv::Rect          r,                    // Rectangle to draw   const cv::Scalar& color,                // Color, BGR form    int               lineType = 8,         // Connectedness, 4 or 8   int               shift    = 0          // Bits of radius to treat as fraction ); The function cv::rectangle() draws a rectangle with corners pt1 to pt2 in the image img. An alternate form of cv::rectangle() allows the rectangle’s location and size to be specified by a single cv::Rect argument r. cv::polyLines() void polyLines(   cv::Mat&          img,                  // Image to be drawn on   const cv::Point*  pts,                  // C-style array of arrays of points   int               npts,                 // Number of points in ‘pts[i]’   int               ncontours,            // Number of arrays in ‘pts’   bool              isClosed,             //

in the image img. An alternate form of cv::rectangle() allows the rectangle’s location and size to be specified by a single cv::Rect argument r. cv::polyLines() void polyLines(   cv::Mat&          img,                  // Image to be drawn on   const cv::Point*  pts,                  // C-style array of arrays of points   int               npts,                 // Number of points in ‘pts[i]’   int               ncontours,            // Number of arrays in ‘pts’   bool              isClosed,             // If true, connect last and first points   const cv::Scalar& color,                // Color, BGR form    int               lineType = 8,         // Connectedness, 4 or 8   int               shift    = 0          // Bits of radius to treat as fraction ); This function draws any number of unfilled polygons. It can handle general polygons including polygons with self-intersections. The argument ncontours specifies how many different polygon contours there will be, and the argument npts is a C-style array that

If true, connect last and first points   const cv::Scalar& color,                // Color, BGR form    int               lineType = 8,         // Connectedness, 4 or 8   int               shift    = 0          // Bits of radius to treat as fraction ); This function draws any number of unfilled polygons. It can handle general polygons including polygons with self-intersections. The argument ncontours specifies how many different polygon contours there will be, and the argument npts is a C-style array that indicates how many points there are in each contour (i.e., npts[i] indicates how many points there are in polygon i). pts is a C-style array of C-style arrays containing all of the points in those polygons (i.e., pts[i][j] contains the 𝑗!! point in the 𝑖!! polygon.)  Polygons are not assumed to be closed. If the argument isClosed is true, then a segment from the last element of pts[i][] to the first element will be assumed. Otherwise the contour is taken to be an open contour containing only npts[i]-1

that indicates how many points there are in each contour (i.e., npts[i] indicates how many points there are in polygon i). pts is a C-style array of C-style arrays containing all of the points in those polygons (i.e., pts[i][j] contains the 𝑗!! point in the 𝑖!! polygon.)  Polygons are not assumed to be closed. If the argument isClosed is true, then a segment from the last element of pts[i][] to the first element will be assumed. Otherwise the contour is taken to be an open contour containing only npts[i]-1 segments between the npts[i] points listed.  cv::LineIterator() The cv::LineIterator object is an iterator that is used to get each pixel of a raster line in sequence. It is another one of those “objects that do stuff.” The constructor for the line iterator takes the two endpoints for the line as well as a line type specifier and an additional Boolean that indicates which direction the line should be traversed. LineIterator::LineIterator(   cv::Mat&          img,                  // Image to be drawn on

segments between the npts[i] points listed.  cv::LineIterator() The cv::LineIterator object is an iterator that is used to get each pixel of a raster line in sequence. It is another one of those “objects that do stuff.” The constructor for the line iterator takes the two endpoints for the line as well as a line type specifier and an additional Boolean that indicates which direction the line should be traversed. LineIterator::LineIterator(   cv::Mat&          img,                  // Image to be drawn on   cv::Point         pt1,                  // First end-point of line   cv::Point         pt2                   // Second end-point of line   int               lineType = 8,         // Connectedness, 4 or 8   bool              leftToRight = false   // If true, always start steps on the left );

Once initialized, the number of pixels in the line is stored in the member integer cv::LineIterator::count. The overloaded dereferencing operator cv::LineIterator::operator*() returns a pointer of type uchar*, which points to the “current” pixel. The current pixel starts at one end of the line, and is incremented by means of the overloaded increment operator cv::LineIterator::operator++(). The actual traversal is done according to the Bresenham algorithm mentioned earlier. The purpose of the cv::LineIterator is to make it possible for you to take some specific action on each pixel along the line. This is particularly handy when creating special effects such as switching the color of a pixel from black to white and white to black (i.e., an XOR operation on a binary image). When accessing an individual “pixel,” remember that this pixel may have one or many channels and it might be any kind of image depth. The return value from the dereferencing operator is always uchar*, so you are responsible for casting

is to make it possible for you to take some specific action on each pixel along the line. This is particularly handy when creating special effects such as switching the color of a pixel from black to white and white to black (i.e., an XOR operation on a binary image). When accessing an individual “pixel,” remember that this pixel may have one or many channels and it might be any kind of image depth. The return value from the dereferencing operator is always uchar*, so you are responsible for casting that pointer to the correct type. For example, if your image were a three-channel image of 32-bit floating-point numbers, and your iterator were called iter, then you would want to cast the return (pointer) value of the dereferencing operator like this: (Vec3f*)*iter. The style of the overloaded dereferencing operator cv::LineIterator::operator*() is slightly different than what you are probably used to from libraries like STL. The difference is that the return value from the iterator is itself a pointer, so the

to the correct type. For example, if your image were a three-channel image of 32-bit floating-point numbers, and your iterator were called iter, then you would want to cast the return (pointer) value of the dereferencing operator like this: (Vec3f*)*iter. The style of the overloaded dereferencing operator cv::LineIterator::operator*() is slightly different than what you are probably used to from libraries like STL. The difference is that the return value from the iterator is itself a pointer, so the iterator itself behaves not like a pointer, but like a pointer to a pointer. Fonts and Text One additional form of drawing is to draw text. Of course, text creates its own set of complexities, but—as always with this sort of thing—OpenCV is more concerned with providing a simple “down and dirty” solution that will work for simple cases than a robust, complex solution (which would be redundant anyway given the capabilities of other libraries). Table 4-10: Text drawing functions Function Description cv::putText()

itself behaves not like a pointer, but like a pointer to a pointer. Fonts and Text One additional form of drawing is to draw text. Of course, text creates its own set of complexities, but—as always with this sort of thing—OpenCV is more concerned with providing a simple “down and dirty” solution that will work for simple cases than a robust, complex solution (which would be redundant anyway given the capabilities of other libraries). Table 4-10: Text drawing functions Function Description cv::putText() Draw the specified text in an image cv::getTextSize() Determine the width and height of a text string cv::putText() void cv::putText(   cv::Mat&      img,                      // Image to be drawn on   const string& text,                     // String to write (often from cv::format)   cv::Point     origin,                   // Upper-left corner of text box   int           fontFace,                 // Font to use (e.g., cv::FONT_HERSHEY_PLAIN)   double        fontScale,                // Scale of font (not

Draw the specified text in an image cv::getTextSize() Determine the width and height of a text string cv::putText() void cv::putText(   cv::Mat&      img,                      // Image to be drawn on   const string& text,                     // String to write (often from cv::format)   cv::Point     origin,                   // Upper-left corner of text box   int           fontFace,                 // Font to use (e.g., cv::FONT_HERSHEY_PLAIN)   double        fontScale,                // Scale of font (not “point”, but multiple)   cv::Scalar    color,                    // Color, RGB form   int           thickness = 1,            // Thickness of line    int           lineType  = 8,            // Connectedness, 4 or 8   bool          bottomLeftOrigin = false  // If true, measure ‘origin’ from lower left ); OpenCV has one main routine, called cv::putText() that just throws some text onto an image. The text indicated by text is printed with its upper-left corner of the text box at origin and in the color

(not “point”, but multiple)   cv::Scalar    color,                    // Color, RGB form   int           thickness = 1,            // Thickness of line    int           lineType  = 8,            // Connectedness, 4 or 8   bool          bottomLeftOrigin = false  // If true, measure ‘origin’ from lower left ); OpenCV has one main routine, called cv::putText() that just throws some text onto an image. The text indicated by text is printed with its upper-left corner of the text box at origin and in the color indicated by color, unless the bottomLeftOrigin flag is true, in which case the lower-left corner of the text box is located at origin. The font used is selected by the fontFace argument, which can be any of those listed in Error! Reference source not found..

Table 4-11: Available fonts (all are variations of Hershey) Identifier Description cv::FONT_HERSHEY_SIMPLEX                  Normal size sans-serif cv::FONT_HERSHEY_PLAIN                  Small size sans-serif cv::FONT_HERSHEY_DUPLEX                  Normal size sans-serif, more complex than cv::FONT_HERSHEY_SIMPLEX cv::FONT_HERSHEY_COMPLEX                  Normal size serif, more complex than cv::FONT_HERSHEY_DUPLEX cv::FONT_HERSHEY_TRIPLEX                  Normal size serif, more complex than cv::FONT_HERSHEY_COMPLEX cv::FONT_HERSHEY_COMPLEX_SMALL                  Smaller version of cv::FONT_HERSHEY_COMPLEX cv::FONT_HERSHEY_SCRIPT_SIMPLEX                  Handwriting style cv::FONT_HERSHEY_SCRIPT_COMPLEX                  More complex variant of cv::FONT_HERSHEY_SCRIPT_SIMPLEX Any of the font names listed in Error! Reference source not found. can also be combined (with an OR operator) with cv::FONT_HERSHEY_ITALIC to render the indicated font in italics. Each font has a “natural” size. When fontScale is not

cv::FONT_HERSHEY_COMPLEX_SMALL                  Smaller version of cv::FONT_HERSHEY_COMPLEX cv::FONT_HERSHEY_SCRIPT_SIMPLEX                  Handwriting style cv::FONT_HERSHEY_SCRIPT_COMPLEX                  More complex variant of cv::FONT_HERSHEY_SCRIPT_SIMPLEX Any of the font names listed in Error! Reference source not found. can also be combined (with an OR operator) with cv::FONT_HERSHEY_ITALIC to render the indicated font in italics. Each font has a “natural” size. When fontScale is not 1.0, then the font size is scaled by this number before drawing.   Figure 4-9: The eight fonts of Table 4-11, with the origin of each line separated from the vertical by 30 pixels cv::getTextSize() cv::Size cv::getTextSize(   const string& text,   cv::Point     origin,   int           fontFace,   double        fontScale,   int           thickness,   int*          baseLine ); The function cv::getTextSize() answers the question of how big some text would be if you were to draw it (with some set of parameters), without

the font size is scaled by this number before drawing.   Figure 4-9: The eight fonts of Table 4-11, with the origin of each line separated from the vertical by 30 pixels cv::getTextSize() cv::Size cv::getTextSize(   const string& text,   cv::Point     origin,   int           fontFace,   double        fontScale,   int           thickness,   int*          baseLine ); The function cv::getTextSize() answers the question of how big some text would be if you were to draw it (with some set of parameters), without actually drawing it on an image. The only novel argument to

cv::getTextSize() is baseLine, which is actually an output parameter. baseLine is the y-coordinate of the text baseline relative to the bottom-most point in the text.29  Data Persistence OpenCV provides a mechanism for serializing and de-serializing its various data types to and from disk in either YAML or XML format, which can load and store any number of OpenCV data objects (including basic types like int, float, etc.) in a single file. There functions are separate from the special functions we saw earlier in the chapter that handle the more specialized situation of loading and saving image files and video data. In this section, we will focus on general object persistence: reading and writing matrices, OpenCV structures, configuration, and log files. The basic mechanism for reading and writing files is the cv::FileStorage object. This object essentially represents a file on disk, but does so in a manner that makes accessing the data represented in the file easy and natural. Writing to a cv::FileStorage

in the chapter that handle the more specialized situation of loading and saving image files and video data. In this section, we will focus on general object persistence: reading and writing matrices, OpenCV structures, configuration, and log files. The basic mechanism for reading and writing files is the cv::FileStorage object. This object essentially represents a file on disk, but does so in a manner that makes accessing the data represented in the file easy and natural. Writing to a cv::FileStorage FileStorage::FileStorage(); FileStorage::FileStorage( string fileName, int flag ); The cv::FileStorage object is a representation of an XML or YML data file. You can create it and pass a file name to the constructor, or you can just create an unopened storage object with the default constructor and open the file later with cv::FileStorage::open() where the flag argument should be either cv::FileStorage::WRITE or cv::FileStorage::APPEND. FileStorage::open( string fileName, int flag ); Once you have opened the

FileStorage::FileStorage( string fileName, int flag ); The cv::FileStorage object is a representation of an XML or YML data file. You can create it and pass a file name to the constructor, or you can just create an unopened storage object with the default constructor and open the file later with cv::FileStorage::open() where the flag argument should be either cv::FileStorage::WRITE or cv::FileStorage::APPEND. FileStorage::open( string fileName, int flag ); Once you have opened the file you want to write to, you can write using the operator cv::FileStorage::operator<<() in the same manner you might write to stdout with an STL stream. Internally, however, there is quite a bit more going on when you write in this manner.  Data inside of the cv::FileStorage is stored in one of two forms, either as a “mapping” (i.e., key-value pairs) or a “sequence” (which is a series of unnamed entries). At the top level, the data you write to the file storage is all a mapping, and inside of that mapping you can place other

to, you can write using the operator cv::FileStorage::operator<<() in the same manner you might write to stdout with an STL stream. Internally, however, there is quite a bit more going on when you write in this manner.  Data inside of the cv::FileStorage is stored in one of two forms, either as a “mapping” (i.e., key-value pairs) or a “sequence” (which is a series of unnamed entries). At the top level, the data you write to the file storage is all a mapping, and inside of that mapping you can place other mappings or sequences, and mappings or sequences inside of those as deep as you like. myFileStorage << ”someInteger” << 27;                    // save an integer myFileStorage << ”anArray” << cv::Mat::eye(3,3,CV::F32);  // save an array To create a sequence entry, you first provide the string name for the entry, and then the entry itself. The entry can be a number (integer, float, etc.), a string, or any OpenCV data type.  If you would like to create a new mapping or sequence you can do so with the special

mappings or sequences, and mappings or sequences inside of those as deep as you like. myFileStorage << ”someInteger” << 27;                    // save an integer myFileStorage << ”anArray” << cv::Mat::eye(3,3,CV::F32);  // save an array To create a sequence entry, you first provide the string name for the entry, and then the entry itself. The entry can be a number (integer, float, etc.), a string, or any OpenCV data type.  If you would like to create a new mapping or sequence you can do so with the special characters “{“ (for a mapping) or “[“ (for a sequence). Once you have started the mapping or sequence, you can add new elements and then finally close the mapping or sequence with “}” or “]” (respectively). myFileStorage << ”theCat” << ”{“; myFileStorage << ”fur” << ”gray” << "eyes” << ”green” << ”weightLbs” << 16; myFileStorage << ”}”; Once you have created a mapping, you enter each element with a name and the data following, just as you did for the top-level mapping. If you create a sequence, you simply

characters “{“ (for a mapping) or “[“ (for a sequence). Once you have started the mapping or sequence, you can add new elements and then finally close the mapping or sequence with “}” or “]” (respectively). myFileStorage << ”theCat” << ”{“; myFileStorage << ”fur” << ”gray” << "eyes” << ”green” << ”weightLbs” << 16; myFileStorage << ”}”; Once you have created a mapping, you enter each element with a name and the data following, just as you did for the top-level mapping. If you create a sequence, you simply enter the new data one item after another until you close the sequence. myFileStorage << ”theTeam” << ”[“;                                                              29 The “baseline” is the line on which the bottoms of characters such as a and b are aligned. Characters such as y and g hang below the baseline.

myFileStorage << ”eddie” << ”tom” << ”scott”; myFileStorage << ”]”; Once you are completely done writing, you close the file with the cv::FileStorage::release() member function. Here is an explicit code sample from the OpenCV documentation: #include "opencv2/opencv.hpp" #include <time.h>  int main(int, char** argv) {     cv::FileStorage fs("test.yml", cv::FileStorage::WRITE);      fs << "frameCount" << 5;     time_t rawtime; time(&rawtime);     fs << "calibrationDate" << asctime(localtime(&rawtime));     cv::Mat cameraMatrix = (       cv::Mat_<double>(3,3)        << 1000, 0, 320, 0, 1000, 240, 0, 0, 1     );     cv::Mat distCoeffs = (       cv::Mat_<double>(5,1)        << 0.1, 0.01, -0.001, 0, 0     );     fs << "cameraMatrix" << cameraMatrix << "distCoeffs" << distCoeffs;     fs << "features" << "[";     for( int i = 0; i < 3; i++ )     {         int x = rand() % 640;         int y = rand() % 480;         uchar lbp = rand() % 256;          fs << "{:" << "x" << x << "y" << y << "lbp" << "[:";         for(

cv::Mat cameraMatrix = (       cv::Mat_<double>(3,3)        << 1000, 0, 320, 0, 1000, 240, 0, 0, 1     );     cv::Mat distCoeffs = (       cv::Mat_<double>(5,1)        << 0.1, 0.01, -0.001, 0, 0     );     fs << "cameraMatrix" << cameraMatrix << "distCoeffs" << distCoeffs;     fs << "features" << "[";     for( int i = 0; i < 3; i++ )     {         int x = rand() % 640;         int y = rand() % 480;         uchar lbp = rand() % 256;          fs << "{:" << "x" << x << "y" << y << "lbp" << "[:";         for( int j = 0; j < 8; j++ )             fs << ((lbp >> j) & 1);         fs << "]" << "}";     }     fs << "]";     fs.release();     return 0; } The result of running this program would be a YML file with the following contents: %YAML:1.0 frameCount: 5 calibrationDate: "Fri Jun 17 14:09:29 2011\n" cameraMatrix: !!opencv-matrix    rows: 3    cols: 3    dt: d    data: [ 1000., 0., 320., 0., 1000., 240., 0., 0., 1. ] distCoeffs: !!opencv-matrix    rows: 5    cols: 1    dt: d    data: [ 1.0000000000000001e-01,

int j = 0; j < 8; j++ )             fs << ((lbp >> j) & 1);         fs << "]" << "}";     }     fs << "]";     fs.release();     return 0; } The result of running this program would be a YML file with the following contents: %YAML:1.0 frameCount: 5 calibrationDate: "Fri Jun 17 14:09:29 2011\n" cameraMatrix: !!opencv-matrix    rows: 3    cols: 3    dt: d    data: [ 1000., 0., 320., 0., 1000., 240., 0., 0., 1. ] distCoeffs: !!opencv-matrix    rows: 5    cols: 1    dt: d    data: [ 1.0000000000000001e-01, 1.0000000000000000e-02,        -1.0000000000000000e-03, 0., 0. ] features:    - { x:167, y:49, lbp:[ 1, 0, 0, 1, 1, 0, 1, 1 ] }    - { x:298, y:130, lbp:[ 0, 0, 0, 1, 0, 0, 1, 1 ] }    - { x:344, y:158, lbp:[ 1, 1, 0, 0, 0, 0, 1, 0 ] }

In the example code, you will notice that sometimes all of the data in a mapping or sequence is stored on a single line and other times it is stored with one element per line. This is not an automatic formatting behavior. Instead, it is created by a variant of the mapping and sequence creation strings: “{:” and “:}” mappings, and “[:” and “:]” for sequences. This feature is only meaningful for YML output; if the output file is XML, this nuance is ignored and the mapping or sequence is stored as it would have been without the variant.  Reading from a cv::FileStorage FileStorage::FileStorage( string fileName, int flag ); The cv::FileStorage can be opened for reading the same way it is opened for writing, except that the flag argument should be set to cv::FileStorage::READ. As with writing, you can also create an unopened file storage object with the default constructor and open it later with cv::FileStorage::open(). FileStorage::open( string fileName, int flag ); Once the file has been opened, the data can be

been without the variant.  Reading from a cv::FileStorage FileStorage::FileStorage( string fileName, int flag ); The cv::FileStorage can be opened for reading the same way it is opened for writing, except that the flag argument should be set to cv::FileStorage::READ. As with writing, you can also create an unopened file storage object with the default constructor and open it later with cv::FileStorage::open(). FileStorage::open( string fileName, int flag ); Once the file has been opened, the data can be read with either the overloaded array operator cv::FileStorage::operator[]() or with the iterator cv::FileNodeIterator. Once you are completely done reading, you then close the file with the cv::FileStorage::release() member function. To read from a mapping, the cv::FileStorage::operator[]() is passed the string key associated with the desired object. To read from a sequence, the same operator can be called with an integer argument instead. The return value of this operator is not the desired object, however;

with either the overloaded array operator cv::FileStorage::operator[]() or with the iterator cv::FileNodeIterator. Once you are completely done reading, you then close the file with the cv::FileStorage::release() member function. To read from a mapping, the cv::FileStorage::operator[]() is passed the string key associated with the desired object. To read from a sequence, the same operator can be called with an integer argument instead. The return value of this operator is not the desired object, however; it is an object of type cv::FileNode, which represents the value that goes with the given key in an abstract form. cv::FileNode Once you have a cv::FileNode object, you can do one of several things with it. If it represents an object (or a number or a string) you can just load it into a variable of the appropriate type with the overloaded extraction operator cv::FileNode::operator>>(). cv::Mat anArray; myFileStorage[“calibrationMatrix”] >> anArray; The cv::FileNode object also supports direct casting to any

is an object of type cv::FileNode, which represents the value that goes with the given key in an abstract form. cv::FileNode Once you have a cv::FileNode object, you can do one of several things with it. If it represents an object (or a number or a string) you can just load it into a variable of the appropriate type with the overloaded extraction operator cv::FileNode::operator>>(). cv::Mat anArray; myFileStorage[“calibrationMatrix”] >> anArray; The cv::FileNode object also supports direct casting to any of the basic data types. int aNumber; myFileStorage[“someInteger”] >> aNumber; is equivalent to: int aNumber; aNumber = (int)myFileStorage[“someInteger”]; As mentioned earlier, there is also an iterator for moving through file nodes that can be used as well. Given a cv::FileNode object, the member functions cv::FileNode::begin() and cv::FileNode::end() have their usual interpretations as providing the first and “after last” iterator for either a mapping or a sequence. The iterator, on dereferencing with the

the basic data types. int aNumber; myFileStorage[“someInteger”] >> aNumber; is equivalent to: int aNumber; aNumber = (int)myFileStorage[“someInteger”]; As mentioned earlier, there is also an iterator for moving through file nodes that can be used as well. Given a cv::FileNode object, the member functions cv::FileNode::begin() and cv::FileNode::end() have their usual interpretations as providing the first and “after last” iterator for either a mapping or a sequence. The iterator, on dereferencing with the usual overloaded dereferencing operator cv::FileNodeIterator::operator*(), will return another cv::FileNode object. Such iterators support the usual incrementing and decrementing operators. If the iterator was iterating through a mapping, then the returned cv::FileNode object will have a name that can be retrieved with cv::FileNode::name(). Table 4-12: Member functions of cv::FileNode Example Description cv::FileNode fn() File node object default constructor

cv::FileNode fn1( fn0 ) File node object copy constructor, creates a node fn1 from a node fn0 cv::FileNode fn1( fs, node ) File node constructor that creates a C++ style cv::FileNode object from a C-style CvFileStorage* pointer fs and a C-style CvFileNode* pointer node fn[ (string)key ] fn[ (char*)key ] STL string or C-string accessor for named child (of mapping node), converts key to the appropriate child node fn[ (int)id ] Accessor for numbered child (of sequence node), converts ID to the appropriate child node fn.type()  Returns node type enum  fn.empty()                Determine if node is empty fn.isNone() Determine if node has value “None” fn.isSeq() Determine if node is a sequence fn.isMap() Determine if node is a mapping fn.isInt() fn.isReal() fn.isString() Determine if node is an integer, a floating-point number, or a string (respectively) fn.name() Return nodes name if node is a child of a mapping size_t sz=fn.size() Return size of node (in bytes) (int)fn (float)fn (double)fn (string)fn Extract

child node fn.type()  Returns node type enum  fn.empty()                Determine if node is empty fn.isNone() Determine if node has value “None” fn.isSeq() Determine if node is a sequence fn.isMap() Determine if node is a mapping fn.isInt() fn.isReal() fn.isString() Determine if node is an integer, a floating-point number, or a string (respectively) fn.name() Return nodes name if node is a child of a mapping size_t sz=fn.size() Return size of node (in bytes) (int)fn (float)fn (double)fn (string)fn Extract the value from a node containing an integer, 32-bit float, 64-bit float, or string (respectively) Of the methods in Table 4-12, one requires special clarification: cv::FileNode::type(). The returned value is an enumerated type defined in the class cv::FileNode. The possible values are given in Table 4-13.  Table 4-13: Possible return values for cv::FileNode::type() Example Description cv::FileNode::NONE      = 0 Node is of type “None” cv::FileNode::INT       = 1 Node contains an integer

cv::FileNode::REAL      = 2 cv::FileNode::FLOAT     = 2 Node contains a floating-point number30 cv::FileNode::STR       = 3 cv::FileNode::STRING    = 3 Node contains a string cv::FileNode::REF       = 4 Node contains a reference (i.e., a compound object) cv::FileNode::SEQ       = 5 Node is itself a sequence of other nodes cv::FileNode::MAP       = 6 Node is itself a mapping of other nodes cv::FileNode::FLOW      = 8 Node is a Compact representation of a sequence or mapping  cv::FileNode::USER      = 16 Registered object (e.g., a Matrix) cv::FileNode::EMPTY     = 32 Node has no value assigned to it cv::FileNode::NAMED     = 64 Node is a child of a mapping (i.e., it has a name) It is worth noting that the last four enum values are powers of two starting at 8. This is because a node may have any or all of these properties in addition to one of the first eight listed types. The following code example (also from the OpenCV documentation) shows how we could read the file we wrote previously: cv::FileStorage

16 Registered object (e.g., a Matrix) cv::FileNode::EMPTY     = 32 Node has no value assigned to it cv::FileNode::NAMED     = 64 Node is a child of a mapping (i.e., it has a name) It is worth noting that the last four enum values are powers of two starting at 8. This is because a node may have any or all of these properties in addition to one of the first eight listed types. The following code example (also from the OpenCV documentation) shows how we could read the file we wrote previously: cv::FileStorage fs2("test.yml", cv::FileStorage::READ);  // first method: use (type) operator on FileNode. int frameCount = (int)fs2["frameCount"];  std::string date; // second method: use cv::FileNode::operator >> fs2["calibrationDate"] >> date;  cv::Mat cameraMatrix2, distCoeffs2; fs2["cameraMatrix"] >> cameraMatrix2; fs2["distCoeffs"] >> distCoeffs2;  cout << "frameCount: " << frameCount << endl      << "calibration date: " << date << endl      << "camera matrix: " << cameraMatrix2 << endl      << "distortion coeffs: "

fs2("test.yml", cv::FileStorage::READ);  // first method: use (type) operator on FileNode. int frameCount = (int)fs2["frameCount"];  std::string date; // second method: use cv::FileNode::operator >> fs2["calibrationDate"] >> date;  cv::Mat cameraMatrix2, distCoeffs2; fs2["cameraMatrix"] >> cameraMatrix2; fs2["distCoeffs"] >> distCoeffs2;  cout << "frameCount: " << frameCount << endl      << "calibration date: " << date << endl      << "camera matrix: " << cameraMatrix2 << endl      << "distortion coeffs: " << distCoeffs2 << endl;  cv::FileNode features = fs2["features"]; cv::FileNodeIterator it = features.begin(), it_end = features.end(); int idx = 0; std::vector<uchar> lbpval;  // iterate through a sequence using FileNodeIterator for( ; it != it_end; ++it, idx++ )                                                              30 Note that the floating-point types are not distinguished. This is a somewhat subtle point. Recall that XML and YML are ASCII text file formats. As a result, all floating-point numbers

<< distCoeffs2 << endl;  cv::FileNode features = fs2["features"]; cv::FileNodeIterator it = features.begin(), it_end = features.end(); int idx = 0; std::vector<uchar> lbpval;  // iterate through a sequence using FileNodeIterator for( ; it != it_end; ++it, idx++ )                                                              30 Note that the floating-point types are not distinguished. This is a somewhat subtle point. Recall that XML and YML are ASCII text file formats. As a result, all floating-point numbers are of no specific precision until cast to an internal machine variable type. So, at the time of parsing, all floating-point numbers are represented only as an abstract floating-point type.

{     cout << "feature #" << idx << ": ";     cout << "x=" << (int)(*it)["x"] << ", y=" << (int)(*it)["y"] << ", lbp: (";     // you can also easily read numerical arrays using FileNode >> std::vector operator.     (*it)["lbp"] >> lbpval;     for( int i = 0; i < (int)lbpval.size(); i++ )         cout << " " << (int)lbpval[i];     cout << ")" << endl; } fs.release(); Summary We have seen that OpenCV provides a number of ways to bring computer vision programs to the screen. The native HighGUI tools are convenient and easy to use, but not so great for functionality or final polish.  For a little more capability, the Qt-based HighGUI tools add buttons and some nice gadgets for manipulating your image on the screen—which is very helpful for debugging, parameter tuning, and for studying the subtle effects of changes in your program. Because those methods lack extensibility and are likely unsuitable for the production of professional applications, we went on to look at a few examples of how you might combine

convenient and easy to use, but not so great for functionality or final polish.  For a little more capability, the Qt-based HighGUI tools add buttons and some nice gadgets for manipulating your image on the screen—which is very helpful for debugging, parameter tuning, and for studying the subtle effects of changes in your program. Because those methods lack extensibility and are likely unsuitable for the production of professional applications, we went on to look at a few examples of how you might combine OpenCV with existing fully featured GUI toolkits.  We then touched on some other important functions you will need for manipulating arrays in basic ways and went on to drawing functions. Finally, we reviewed how OpenCV can store and retrieve objects from files. Exercises 1. This chapter completes our introduction to basic I/O programming and data structures in OpenCV. The following exercises build on this knowledge and create useful utilities for later use. a) Create a program that (1) reads frames from a

OpenCV with existing fully featured GUI toolkits.  We then touched on some other important functions you will need for manipulating arrays in basic ways and went on to drawing functions. Finally, we reviewed how OpenCV can store and retrieve objects from files. Exercises 1. This chapter completes our introduction to basic I/O programming and data structures in OpenCV. The following exercises build on this knowledge and create useful utilities for later use. a) Create a program that (1) reads frames from a video, (2) turns the result to grayscale, and (3) performs Canny edge detection on the image. Display all three stages of processing in three different windows, with each window appropriately named for its function. b) Display all three stages of processing in one image. Hint: Create another image of the same height but three times the width as the video frame. Copy the images into this, either by using pointers or (more cleverly) by creating three new cv::Mat objects that reference into the beginning of,

video, (2) turns the result to grayscale, and (3) performs Canny edge detection on the image. Display all three stages of processing in three different windows, with each window appropriately named for its function. b) Display all three stages of processing in one image. Hint: Create another image of the same height but three times the width as the video frame. Copy the images into this, either by using pointers or (more cleverly) by creating three new cv::Mat objects that reference into the beginning of, to one-third, and to two-thirds of the way into, the image data and then copying directly into these (i.e. with the copyTo() function). c) Write appropriate text labels describing the processing in each of the three slots. 2. Create a program that reads in and displays an image. When the user’s mouse clicks on the image, read in the corresponding pixel (blue, green, red) values and write those values as text to the screen at the mouse location. a) For the program of Exercise 1b, display the mouse

of, to one-third, and to two-thirds of the way into, the image data and then copying directly into these (i.e. with the copyTo() function). c) Write appropriate text labels describing the processing in each of the three slots. 2. Create a program that reads in and displays an image. When the user’s mouse clicks on the image, read in the corresponding pixel (blue, green, red) values and write those values as text to the screen at the mouse location. a) For the program of Exercise 1b, display the mouse coordinates of the individual image when clicking anywhere within the three-image display. 3. Create a program that reads in and displays an image. a) Allow the user to select a rectangular region in the image by drawing a rectangle with the mouse button held down, and highlight the region when the mouse button is released. Be careful to save an image copy in memory so that your drawing into the image does not destroy the original values there. The next mouse click should start the process all over again from

of the individual image when clicking anywhere within the three-image display. 3. Create a program that reads in and displays an image. a) Allow the user to select a rectangular region in the image by drawing a rectangle with the mouse button held down, and highlight the region when the mouse button is released. Be careful to save an image copy in memory so that your drawing into the image does not destroy the original values there. The next mouse click should start the process all over again from the original image. b) In a separate window, use the drawing functions to draw a graph in blue, green, and red for how many pixels of each value were found in the selected box. This is the color histogram of that color region. The x-axis should be eight bins that represent pixel values falling within the ranges 0–31,

32–63,…, 223–255. The y-axis should be counts of the number of pixels that were found in that bin range. Do this for each color channel, BGR. 4. Make an application that reads and displays a video and is controlled by sliders. One slider will control the position within the video from start to end in 10 increments; another binary slider should control pause/unpause. Label both sliders appropriately. a) Do this using the built in HighGUI native toolkit functions. b) Do this using the Qt Backend. 5. Create your own simple paint program. a) Write a program that creates an image, sets it to 0, and then displays it. Allow the user to draw lines, circles, ellipses, and polygons on the image using the left mouse button. Create an eraser function when the right mouse button is held down. b) Allow “logical drawing” by allowing the user to set a slider setting to AND, OR, and XOR. That is, if the setting is AND then the drawing will appear only when it crosses pixels greater than 0 (and so on for the other logical

your own simple paint program. a) Write a program that creates an image, sets it to 0, and then displays it. Allow the user to draw lines, circles, ellipses, and polygons on the image using the left mouse button. Create an eraser function when the right mouse button is held down. b) Allow “logical drawing” by allowing the user to set a slider setting to AND, OR, and XOR. That is, if the setting is AND then the drawing will appear only when it crosses pixels greater than 0 (and so on for the other logical functions). 6. Write a program that creates an image, sets it to 0, and then displays it. Hitting Enter should fix the label at the spot it was typed. a) When the user clicks on a location, he or she can type in a label there.  b) Enable Backspace during editing and provide for an abort key.  7. Perspective transform. a) Write a program that reads in an image and uses the numbers 1–9 on the keypad to control a perspective transformation matrix (refer to our discussion of the cv::warpPerspective() in the

functions). 6. Write a program that creates an image, sets it to 0, and then displays it. Hitting Enter should fix the label at the spot it was typed. a) When the user clicks on a location, he or she can type in a label there.  b) Enable Backspace during editing and provide for an abort key.  7. Perspective transform. a) Write a program that reads in an image and uses the numbers 1–9 on the keypad to control a perspective transformation matrix (refer to our discussion of the cv::warpPerspective() in the Dense Perspective Transform section of Chapter 6). Tapping any number should increment the corresponding cell in the perspective transform matrix; tapping with the Shift key depressed should decrement the number associated with that cell (stopping at 0). Each time a number is changed, display the results in two images: the raw image and the transformed image. b) Add functionality to zoom in or out. c) Add functionality to rotate the image. 8. Face fun. Go to the …/samples/cpp/ directory and build the

the Dense Perspective Transform section of Chapter 6). Tapping any number should increment the corresponding cell in the perspective transform matrix; tapping with the Shift key depressed should decrement the number associated with that cell (stopping at 0). Each time a number is changed, display the results in two images: the raw image and the transformed image. b) Add functionality to zoom in or out. c) Add functionality to rotate the image. 8. Face fun. Go to the …/samples/cpp/ directory and build the facedetect.cpp code. Draw a skull image (or find one on the Web) and store it to disk. Modify the facedetect program to load in the image of the skull. a) When a face rectangle is detected, draw the skull in that rectangle. Hint: cv::convertImage() can convert the size of the image, or you could look up the cv::resize() function. One may then set the ROI to the rectangle and use copyTo() to copy the properly resized image there. a) Add a slider with 10 settings corresponding to 0.0 to 1.0. Use this slider to

code. Draw a skull image (or find one on the Web) and store it to disk. Modify the facedetect program to load in the image of the skull. a) When a face rectangle is detected, draw the skull in that rectangle. Hint: cv::convertImage() can convert the size of the image, or you could look up the cv::resize() function. One may then set the ROI to the rectangle and use copyTo() to copy the properly resized image there. a) Add a slider with 10 settings corresponding to 0.0 to 1.0. Use this slider to alpha blend the skull over the face rectangle using the cv::addWeighted() function. 9. Image stabilization. Go to the .../samples/cpp/ directory and build the lkdemo.cpp code (this code does motion tracking, also known as optical flow). Create and display a video image in a much larger window image. Move the camera slightly but use the optical flow vectors to display the image in the same place within the larger window. This is a rudimentary image stabilization technique. 10. Create a structure of an integer, a

blend the skull over the face rectangle using the cv::addWeighted() function. 9. Image stabilization. Go to the .../samples/cpp/ directory and build the lkdemo.cpp code (this code does motion tracking, also known as optical flow). Create and display a video image in a much larger window image. Move the camera slightly but use the optical flow vectors to display the image in the same place within the larger window. This is a rudimentary image stabilization technique. 10. Create a structure of an integer, a cv::Point2i and a cv::Rect; call it my_struct. a) Write two functions: void write_my_struct(    cv::FileStorage* fs,    const char*      name,    my_struct*       ms );  and:

void read_my_struct(    cv::FileStorage* fs,    CvFileNode*      ms_node,    my_struct*       ms ); Use them to write and read my_struct. b) Write and read an array of 10 my_struct structures.

5 Filters and Convolution Overview At this point, we have all of the basics at our disposal. We understand the structure of the library as well as the basic data structures it uses to represent images. We understand the HighGUI interface and can actually run a program and display our results on the screen. Now that we understand these primitive methods required to manipulate image structures, we are ready to learn some more sophisticated operations. We will now move on to higher-level methods that treat the images as images, and not just as arrays of colored (or grayscale) values. When we say “image processing,” we mean just that: using higher-level operators that are defined on image structures in order to accomplish tasks whose meaning is naturally defined in the context of graphical, visual images. Before We Begin There are a couple of important concepts we will need throughout this chapter, so it is worth taking a moment to review these ideas before we dig into the specific image processing functions

the images as images, and not just as arrays of colored (or grayscale) values. When we say “image processing,” we mean just that: using higher-level operators that are defined on image structures in order to accomplish tasks whose meaning is naturally defined in the context of graphical, visual images. Before We Begin There are a couple of important concepts we will need throughout this chapter, so it is worth taking a moment to review these ideas before we dig into the specific image processing functions that make up the bulk of this chapter. We will need to learn two main concepts: first, we’ll need to understand filters (also called kernels) and how they are handled in OpenCV. Next, we’ll take a look at how boundaries are handled, and what happens when OpenCV needs to compute something that is a function of the area around a pixel if that area spills off of the edge of the image.  Filters, Kernels, and Convolution Most of the functions we will discuss in this chapter are special cases of a general concept

make up the bulk of this chapter. We will need to learn two main concepts: first, we’ll need to understand filters (also called kernels) and how they are handled in OpenCV. Next, we’ll take a look at how boundaries are handled, and what happens when OpenCV needs to compute something that is a function of the area around a pixel if that area spills off of the edge of the image.  Filters, Kernels, and Convolution Most of the functions we will discuss in this chapter are special cases of a general concept called image filtering. A filter is any algorithm that starts with some image 𝐼(𝑥,𝑦) and computes a new image 𝐼’(𝑥,𝑦) by computing for each pixel location 𝑥,𝑦 in 𝐼’ some function of the pixels in 𝐼 that are in some area around that point. The template that defines both this area’s shape, as well as how the elements of that area are combined, is called a filter or a kernel.1 In this chapter, many of the important kernels we encounter will be                                                              1 These

image filtering. A filter is any algorithm that starts with some image 𝐼(𝑥,𝑦) and computes a new image 𝐼’(𝑥,𝑦) by computing for each pixel location 𝑥,𝑦 in 𝐼’ some function of the pixels in 𝐼 that are in some area around that point. The template that defines both this area’s shape, as well as how the elements of that area are combined, is called a filter or a kernel.1 In this chapter, many of the important kernels we encounter will be                                                              1 These two terms can be considered essentially interchangeable for our purposes. The signal processing community typically prefers the word filter, while the mathematical community tends to prefer kernel.

linear kernels. This means that the value assigned to point 𝑥,𝑦 in 𝐼’ can be expressed as a weighted sum of the points around (and usually including) 𝑥,𝑦 in 𝐼2. If you like equations, this can be written as: 𝐼!𝑥,𝑦=𝑘!,!∙𝐼(𝑥+𝑖,𝑦+𝑗)!,!∈!"#$"%. This basically says that for some kernel of whatever size (e.g., 5-by-5), we should sum over the area of the kernel, and for each pair 𝑖,𝑗 (representing one point in the kernel), we should add a contribution equal to some value  𝑘!,! multiplied by the value of the pixel in I that is offset from 𝑥,𝑦 by 𝑖,𝑗. The size of the array 𝑘!,! is called the support of the kernel.3  Any filter which can be expressed in this way (i.e., with a linear kernel) is also known as convolutions. It is often convenient (and more intuitive) to represent the kernel graphically as an array of the values of  𝑘!,! (Figure 5-1). We will typically use this representation throughout the book when it is necessary to represent a kernel.  1 1 1 1 1     1 1 1 1 1          1 4 7 4 1 1 1 1 1 1     1 1 1 1

in I that is offset from 𝑥,𝑦 by 𝑖,𝑗. The size of the array 𝑘!,! is called the support of the kernel.3  Any filter which can be expressed in this way (i.e., with a linear kernel) is also known as convolutions. It is often convenient (and more intuitive) to represent the kernel graphically as an array of the values of  𝑘!,! (Figure 5-1). We will typically use this representation throughout the book when it is necessary to represent a kernel.  1 1 1 1 1     1 1 1 1 1          1 4 7 4 1 1 1 1 1 1     1 1 1 1 1   -1 0 1     4 16 26 16 4 1 1 1 1 1   1/25  x 1 1 1 1 1   -2 0 2   1/273 x 7 26 41 26 7 1 1 1 1 1     1 1 1 1 1   -1 0 1     4 16 26 16 4 1 1 1 1 1     1 1 1 1 1          1 4 7 4 1   A)         B)      C)       D)     Figure 5-1: (a) A 5-by-5 box kernel; (b) a normalized 5-by-5 box kernel; (c) a 3-by-3 Sobel “x-derivative” kernel; and (d) a 5-by-5 normalized Gaussian kernel; in each case, the “anchor” is represented in bold. Anchor Points Each kernel shown in Figure 5-1 has one value depicted in bold. This

-1 0 1     4 16 26 16 4 1 1 1 1 1   1/25  x 1 1 1 1 1   -2 0 2   1/273 x 7 26 41 26 7 1 1 1 1 1     1 1 1 1 1   -1 0 1     4 16 26 16 4 1 1 1 1 1     1 1 1 1 1          1 4 7 4 1   A)         B)      C)       D)     Figure 5-1: (a) A 5-by-5 box kernel; (b) a normalized 5-by-5 box kernel; (c) a 3-by-3 Sobel “x-derivative” kernel; and (d) a 5-by-5 normalized Gaussian kernel; in each case, the “anchor” is represented in bold. Anchor Points Each kernel shown in Figure 5-1 has one value depicted in bold. This is the anchor point of the kernel. This indicates how the kernel is to be aligned with the source image. For example, in Figure 5-1 (d), the number 41 appears in bold. This means that in the summation used to compute  𝐼’(𝑥,𝑦), it is 𝐼(𝑥,𝑦) that is multiplied by 41/273 (and similarly, the terms corresponding to 𝐼(𝑥−1,𝑦) and 𝐼(𝑥+1,𝑦) are multiplied by 26/273). Border Extrapolation and Boundary Conditions Something that will come up with some frequency as we look at how images are processed in OpenCV is the

is the anchor point of the kernel. This indicates how the kernel is to be aligned with the source image. For example, in Figure 5-1 (d), the number 41 appears in bold. This means that in the summation used to compute  𝐼’(𝑥,𝑦), it is 𝐼(𝑥,𝑦) that is multiplied by 41/273 (and similarly, the terms corresponding to 𝐼(𝑥−1,𝑦) and 𝐼(𝑥+1,𝑦) are multiplied by 26/273). Border Extrapolation and Boundary Conditions Something that will come up with some frequency as we look at how images are processed in OpenCV is the issue of how borders are handled. Unlike some other image handling libraries,4 the filtering operations in OpenCV (e.g., cv::blur(), cv::erode(), cv::dilate(), etc.) produce output images of the same size as the input. To achieve that result, OpenCV creates “virtual” pixels outside of the image at the borders. You can see this would be necessary for operation like cv::blur(), which is going to take all of the pixels in a neighborhood of some point and average them to determine a new value for that point. How

of how borders are handled. Unlike some other image handling libraries,4 the filtering operations in OpenCV (e.g., cv::blur(), cv::erode(), cv::dilate(), etc.) produce output images of the same size as the input. To achieve that result, OpenCV creates “virtual” pixels outside of the image at the borders. You can see this would be necessary for operation like cv::blur(), which is going to take all of the pixels in a neighborhood of some point and average them to determine a new value for that point. How could a meaningful result be computed for an edge pixel that does not have the correct number of neighbors? In fact, it will turn out that in the absence of any clearly “right” way of handling this, we will often find ourselves explicitly asserting how this issue is to be resolved in any given context.                                                              2 An example of a nonlinear kernel that comes up relatively often is the median filter, which replaces the pixel at 𝑥,𝑦 with the median value inside

a meaningful result be computed for an edge pixel that does not have the correct number of neighbors? In fact, it will turn out that in the absence of any clearly “right” way of handling this, we will often find ourselves explicitly asserting how this issue is to be resolved in any given context.                                                              2 An example of a nonlinear kernel that comes up relatively often is the median filter, which replaces the pixel at 𝑥,𝑦 with the median value inside of the kernel area. 3 For technical purists, the “support” of the kernel actually consists of only the nonzero portion of the kernel array. 4 E.g., MATLAB.

Making Borders Yourself Most of the library functions you will use will create these virtual pixels for you. In that context, you will only need to tell the particular function how you would like those pixels created.5 Just the same, in order to know what the options you have mean, it is best to take a look at the function that allows you to explicitly create “padded” images that use one method or another.  The function that does this is cv::copyMakeBorder(). Given an image you want to pad out, and a second image that is somewhat larger, you can ask cv::copyMakeBorder() to fill all of the pixels in the larger image in one way or another. void cv::copyMakeBorder(   cv::InputArray    src,                       // Input Image   cv::OutputArray   dst,                       // Result image   int               top,                       // Top side padding (pixels)   int               bottom,                    // Bottom side padding (pixels)   int               left,                      // Left side padding

image that is somewhat larger, you can ask cv::copyMakeBorder() to fill all of the pixels in the larger image in one way or another. void cv::copyMakeBorder(   cv::InputArray    src,                       // Input Image   cv::OutputArray   dst,                       // Result image   int               top,                       // Top side padding (pixels)   int               bottom,                    // Bottom side padding (pixels)   int               left,                      // Left side padding (pixels)   int               right,                     // Right side padding (pixels)   int               borderType,                // Pixel extrapolation method   const cv::Scalar& value = cv::Scalar()       // Used for constant borders  ); The first two arguments to cv::copyMakeBorder() are the smaller source image and the larger destination image. The next four arguments specify how many pixels of padding are to be added to the source image on the top, bottom, left, and right edges. The next argument

(pixels)   int               right,                     // Right side padding (pixels)   int               borderType,                // Pixel extrapolation method   const cv::Scalar& value = cv::Scalar()       // Used for constant borders  ); The first two arguments to cv::copyMakeBorder() are the smaller source image and the larger destination image. The next four arguments specify how many pixels of padding are to be added to the source image on the top, bottom, left, and right edges. The next argument borderType actually tells cv::copyMakeBorder() how to determine the correct values to assign to the padded pixels (as shown in Figure 5-2).  Figure 5-2: The same image is shown padded using each of the six different borderType options available to cv::copyMakeBorder() (the “NO BORDER” image in the upper-left is the original for comparison) To understand what each option does in detail, it is useful to consider an extremely zoomed in section at the edge of each image (Figure 5-3).

actually tells cv::copyMakeBorder() how to determine the correct values to assign to the padded pixels (as shown in Figure 5-2).  Figure 5-2: The same image is shown padded using each of the six different borderType options available to cv::copyMakeBorder() (the “NO BORDER” image in the upper-left is the original for comparison) To understand what each option does in detail, it is useful to consider an extremely zoomed in section at the edge of each image (Figure 5-3).                                                              5 Actually, the pixels are usually not even really created, but rather they are just “effectively created” by the generation of the correct boundary conditions in the evaluation of the particular function in question.

Figure 5-3:. An extreme zoom in at the left side of each image; for each case, the actual pixel values are shown, as well as a schematic representation; the vertical dotted line in the schematic represents the edge of the original image As you can see by inspecting the figures, some of the available options are quite different. The first option, a constant border cv::BORDER_CONSTANT sets all of the pixels in the border region to some fixed value. This value is set by the value argument to cv::copyMakeBorder(). (In Figure 5-2 and Figure 5-3, this value happens to be cv::Scalar(0,0,0)) The next option is to wrap around cv::BORDER_WRAP, assigning each pixel that is a distance 𝑛 off of the edge of the image the value of the pixel that is a distance 𝑛 in from the opposite edge. The replicate option cv::BORDER_REPLICATE assigns every pixel off of the edge the same value as the pixel on that edge. Finally, there are two slightly different forms of reflection available: cv::BORDER_REFLECT and

to cv::copyMakeBorder(). (In Figure 5-2 and Figure 5-3, this value happens to be cv::Scalar(0,0,0)) The next option is to wrap around cv::BORDER_WRAP, assigning each pixel that is a distance 𝑛 off of the edge of the image the value of the pixel that is a distance 𝑛 in from the opposite edge. The replicate option cv::BORDER_REPLICATE assigns every pixel off of the edge the same value as the pixel on that edge. Finally, there are two slightly different forms of reflection available: cv::BORDER_REFLECT and cv::BORDER_REFLECT_101. The first assigns each pixel that is a distance n off of the edge of the image the value of the pixel that is a distance 𝑛 in from that same edge. In contrast, cv::BORDER_REFLECT_101 assigns each pixel that is a distance 𝑛 off of the edge of the image the value of the pixel that is a distance 𝑛+1 in from that same edge (with the result that the very edge pixel is not replicated). In most cases, cv::BORDER_REFLECT_101 is the default behavior for OpenCV methods. The value of

and cv::BORDER_REFLECT_101. The first assigns each pixel that is a distance n off of the edge of the image the value of the pixel that is a distance 𝑛 in from that same edge. In contrast, cv::BORDER_REFLECT_101 assigns each pixel that is a distance 𝑛 off of the edge of the image the value of the pixel that is a distance 𝑛+1 in from that same edge (with the result that the very edge pixel is not replicated). In most cases, cv::BORDER_REFLECT_101 is the default behavior for OpenCV methods. The value of cv::BORDER_DEFAULT resolves to cv::BORDER_REFLECT_101. Table 5-1 summarizes these options. Table 5-1: borderType options available to cv::copyMakeBorder(), as well as many other functions that need to implicitly create boundary conditions Border Type Effect cv::BORDER_CONSTANT Extend pixels by using a supplied (constant) value cv::BORDER_WRAP Extend pixels by replicating from opposite side cv::BORDER_REPLICATE Extend pixels by copying edge pixel cv::BORDER_REFLECT Extend pixels by reflection

The value of cv::BORDER_DEFAULT resolves to cv::BORDER_REFLECT_101. Table 5-1 summarizes these options. Table 5-1: borderType options available to cv::copyMakeBorder(), as well as many other functions that need to implicitly create boundary conditions Border Type Effect cv::BORDER_CONSTANT Extend pixels by using a supplied (constant) value cv::BORDER_WRAP Extend pixels by replicating from opposite side cv::BORDER_REPLICATE Extend pixels by copying edge pixel cv::BORDER_REFLECT Extend pixels by reflection cv::BORDER_REFLECT_101 Extend pixels by reflection, edge pixel is not “doubled” cv::BORDER_DEFAULT Alias for cv::BORDER_REFLECT_101

Manual Extrapolation On some occasions, you will want to compute the location of the reference pixel to which a particular off-the-edge pixel is referred. For example, given an image of width 𝑤 and height ℎ, you might want to know what pixel in that image is being used to assign a value to virtual pixel (w+dx,h+dy). Though this operation is essentially extrapolation, the function that computes such a result for you is (somewhat confusingly) called cv::borderInterpolate(): int cv::borderInterpolate(             // Returns coordinate of “donor” pixel   int p,                               // 0-based coordinate of extrapolated pixel   int len,                             // Length of array (on relevant axis)     int borderType                       // Pixel extrapolation method );       The function cv::borderInterpolate() computes the extrapolation for one dimension at a time. It takes a coordinate p, a length len (which is the actual size of the image in the associated direction), and a borderType value. So,

// Returns coordinate of “donor” pixel   int p,                               // 0-based coordinate of extrapolated pixel   int len,                             // Length of array (on relevant axis)     int borderType                       // Pixel extrapolation method );       The function cv::borderInterpolate() computes the extrapolation for one dimension at a time. It takes a coordinate p, a length len (which is the actual size of the image in the associated direction), and a borderType value. So, for example, you could compute the value of a particular pixel in an image under a mixed set of boundary conditions, using BORDER_REFLECT_101 in one dimension, and BORDER_WRAP in another: float val = img.at<float>(   cv::borderInterpolate( 100, img.rows, BORDER_REFLECT_101 ),   cv::borderInterpolate(  -5, img.cols, BORDER_WRAP ) ); This function is typically used internally to OpenCV, for example inside of cv::copyMakeBorder or the cv::FilterEngine class (more on that later), but it can come in handy in

for example, you could compute the value of a particular pixel in an image under a mixed set of boundary conditions, using BORDER_REFLECT_101 in one dimension, and BORDER_WRAP in another: float val = img.at<float>(   cv::borderInterpolate( 100, img.rows, BORDER_REFLECT_101 ),   cv::borderInterpolate(  -5, img.cols, BORDER_WRAP ) ); This function is typically used internally to OpenCV, for example inside of cv::copyMakeBorder or the cv::FilterEngine class (more on that later), but it can come in handy in your own algorithms as well. The possible values for borderType are exactly the same as those used by cv::copyMakeBorder. Throughout this chapter, we will encounter functions that take a borderType argument; in all of those cases, they take the same list of argument. Threshold Operations Frequently we have done many layers of processing steps and want either to make a final decision about the pixels in an image or to categorically reject those pixels below or above some value while keeping the others. The

your own algorithms as well. The possible values for borderType are exactly the same as those used by cv::copyMakeBorder. Throughout this chapter, we will encounter functions that take a borderType argument; in all of those cases, they take the same list of argument. Threshold Operations Frequently we have done many layers of processing steps and want either to make a final decision about the pixels in an image or to categorically reject those pixels below or above some value while keeping the others. The OpenCV function cv::threshold() accomplishes these tasks (see survey [Sezgin04]). The basic idea is that an array is given, along with a threshold, and then something happens to every element of the array depending on whether it is below or above the threshold. If you like, you can think of threshold as a very simple convolution operation that uses a 1-by-1 kernel and performs one of several nonlinear operations on that one pixel:6 double cv::threshold(   cv::InputArray    src,                       //

The OpenCV function cv::threshold() accomplishes these tasks (see survey [Sezgin04]). The basic idea is that an array is given, along with a threshold, and then something happens to every element of the array depending on whether it is below or above the threshold. If you like, you can think of threshold as a very simple convolution operation that uses a 1-by-1 kernel and performs one of several nonlinear operations on that one pixel:6 double cv::threshold(   cv::InputArray    src,                       // Input Image   cv::OutputArray   dst,                       // Result image   double            thresh,                    // Threshold value   double            maxValue,                  // Max value for upward operations   int               thresholdType              // Threshold type to use (see Example 5-2) ); As shown in Table 5-2, each threshold type corresponds to a particular comparison operation between the i!" source pixel (srci) and the threshold thresh. Depending on the relationship between the

Input Image   cv::OutputArray   dst,                       // Result image   double            thresh,                    // Threshold value   double            maxValue,                  // Max value for upward operations   int               thresholdType              // Threshold type to use (see Example 5-2) ); As shown in Table 5-2, each threshold type corresponds to a particular comparison operation between the i!" source pixel (srci) and the threshold thresh. Depending on the relationship between the source pixel and the threshold, the destination pixel dsti may be set to 0, the srci, or the given maximum value maxValue.                                                              6 The utility of this point of view will become clearer as we proceed through this chapter, and look at other more complex convolutions. Many useful operations in computer vision can be expressed as a sequence of common convolutions, and more often than not, the last one of those convolutions is a threshold operation.

Table 5-2: thresholdType options for cv::threshold() Threshold type Operation cv::THRESH_BINARY  DST!=SRC!>THRESH    ?  MAXVALUE∶0 cv::THRESH_BINARY_INV  DST!=SRC!>THRESH    ?  0∶MAXVALUE cv::THRESH_TRUNC                DST!=SRC!>THRESH    ?  THRESH∶SRC! cv::THRESH_TOZERO  DST!=SRC!>THRESH    ?  SRC!∶0 cv::THRESH_TOZERO_INV  DST!=SRC!>THRESH    ?  0∶SRC!   Figure 5-4 should help to clarify the exact implications of each of the available values for thresholdType, the thresholding operation.

Figure 5-4: Results of varying the threshold type in cv::threshold(); the horizontal line through each chart represents a particular threshold level applied to the top chart and its effect for each of the five types of threshold operations below Let’s look at a simple example. In Example 5-1, we sum all three channels of an image and then clip the result at 100. Example 5-1: Example code making use of cv::threshold() #include <opencv2/opencv.hpp> #include <iostream> using namespace std;  void sum_rgb( const cv::Mat& src, cv::Mat& dst ) {      // Split image onto the color planes.    vector< cv::Mat> planes;    cv::split(src, planes);      cv::Mat b = planes[0], g = planes[1], r = planes[2], s;    // Add equally weighted rgb values.   cv::addWeighted( r, 1./3., g, 1./3., 0.0, s );   cv::addWeighted( s, 1., b, 1./3., 0.0, s );    // Truncate values above 100.   cv::threshold( s, dst, 100, 100, cv::THRESH_TRUNC ); }  void help() {  cout << "Call: ./ch5_ex5_2 faceScene.jpg" << endl;  cout << "Shows use of

const cv::Mat& src, cv::Mat& dst ) {      // Split image onto the color planes.    vector< cv::Mat> planes;    cv::split(src, planes);      cv::Mat b = planes[0], g = planes[1], r = planes[2], s;    // Add equally weighted rgb values.   cv::addWeighted( r, 1./3., g, 1./3., 0.0, s );   cv::addWeighted( s, 1., b, 1./3., 0.0, s );    // Truncate values above 100.   cv::threshold( s, dst, 100, 100, cv::THRESH_TRUNC ); }  void help() {  cout << "Call: ./ch5_ex5_2 faceScene.jpg" << endl;  cout << "Shows use of alpha blending (addWeighted) and threshold" << endl; }  int main(int argc, char** argv)

{  help();  if(argc < 2) { cout << "specify input image" << endl; return -1; }   // Load the image from the given file name.  cv::Mat src = cv::imread( argv[1] ), dst;  if( src.empty() ) { cout << "can not load " << argv[1] << endl; return -1; }  sum_rgb( src, dst);   // Create a named window with the name of the file and  // show the image in the window  cv::imshow( argv[1], dst );   // Idle until the user hits any key.  cv::waitKey(0);   return 0; } Some important ideas are shown here. One thing is that we don’t want to add directly into an 8-bit array (with the idea of normalizing next) because the higher bits will overflow. Instead, we use equally weighted addition of the three color channels (cv::addWeighted()); then the sum is truncated to saturate at the value of 100 for the return. Had we used a floating-point temporary image for s in Example 5-1, we could have substituted the code shown in Example 5-2 instead. Note that cv::accumulate() can accumulate 8-bit integer image types into a floating-point

we don’t want to add directly into an 8-bit array (with the idea of normalizing next) because the higher bits will overflow. Instead, we use equally weighted addition of the three color channels (cv::addWeighted()); then the sum is truncated to saturate at the value of 100 for the return. Had we used a floating-point temporary image for s in Example 5-1, we could have substituted the code shown in Example 5-2 instead. Note that cv::accumulate() can accumulate 8-bit integer image types into a floating-point image. Example 5-2: Alternative method to combine and threshold image planes void sum_rgb( const cv::Mat& src, cv::Mat& dst ) {    // Split image onto the color planes.   vector<cv::Mat> planes;   cv::split( src, planes );     cv::Mat b = planes[0], g = planes[1], r = planes[2];      // Accumulate separate planes, combine and threshold   cv::Mat s = cv::Mat::zeros( b.size(), cv::F32 );   cv::accumulate( b, s );   cv::accumulate( g, s );   cv::accumulate( r, s );     // Truncate values above 100 and rescale

image. Example 5-2: Alternative method to combine and threshold image planes void sum_rgb( const cv::Mat& src, cv::Mat& dst ) {    // Split image onto the color planes.   vector<cv::Mat> planes;   cv::split( src, planes );     cv::Mat b = planes[0], g = planes[1], r = planes[2];      // Accumulate separate planes, combine and threshold   cv::Mat s = cv::Mat::zeros( b.size(), cv::F32 );   cv::accumulate( b, s );   cv::accumulate( g, s );   cv::accumulate( r, s );     // Truncate values above 100 and rescale into dst   cv::threshold( s, s, 100, 100, cv::THRESH_TRUNC );   s.convertTo(dst, b.type()); } Otsu’s Algorithm It is also possible to have cv::threshold() attempt to determine the optimal value of the threshold for you. This is done by passing the special value cv::THRESH_OTSU as the value of thresh. Briefly, Otsu’s algorithm is to consider all possible thresholds, and to compute the variance 𝜎!! for each of the two classes of pixels (i.e., the class below the threshold and the class above it). Otsu’s

into dst   cv::threshold( s, s, 100, 100, cv::THRESH_TRUNC );   s.convertTo(dst, b.type()); } Otsu’s Algorithm It is also possible to have cv::threshold() attempt to determine the optimal value of the threshold for you. This is done by passing the special value cv::THRESH_OTSU as the value of thresh. Briefly, Otsu’s algorithm is to consider all possible thresholds, and to compute the variance 𝜎!! for each of the two classes of pixels (i.e., the class below the threshold and the class above it). Otsu’s algorithm minimizes:  𝜎!!≡𝑤!𝑡∙𝜎!!+𝑤!𝑡∙𝜎!!, where 𝑤!𝑡 and 𝑤!𝑡 are the relative weights for the two classes given by the relative (normalized) number of pixels in each class (that is, their probability) and 𝜎!! and 𝜎!! are the variances in each class. Some thought (think of an image with 2 colors) will convince you that minimizing the variance over both

classes is the same as maximizing the variance between the two classes. Because an exhaustive search of the space of possible thresholds is required, this is not a particularly fast process. Adaptive Threshold There is a modified threshold technique in which the threshold level is itself variable (across the image). In OpenCV, this method is implemented in the cv::adaptiveThreshold() [Jain86] function: void cv::adaptiveThreshold(   cv::InputArray    src,                       // Input Image   cv::OutputArray   dst,                       // Result image   double            maxValue,                  // Max value for upward operations   int               adaptiveMethod,            // Method to weight pixels in block by    int               thresholdType              // Threshold type to use (see Example 5-2)   int               blockSize,                 // Block size   double            C                          // Constant to offset sum over block by  ); cv::adaptiveThreshold() allows for two different

dst,                       // Result image   double            maxValue,                  // Max value for upward operations   int               adaptiveMethod,            // Method to weight pixels in block by    int               thresholdType              // Threshold type to use (see Example 5-2)   int               blockSize,                 // Block size   double            C                          // Constant to offset sum over block by  ); cv::adaptiveThreshold() allows for two different adaptive threshold types depending on the settings of adaptiveMethod. In both cases, the adaptive threshold T(x, y) is set on a pixel-by-pixel basis by computing a weighted average of the b-by-b region around each pixel location minus a constant, where 𝑏 is given by blockSize and the constant is given by C. If the method is set to cv::ADAPTIVE_THRESH_MEAN_C, then all pixels in the area are weighted equally. If it is set to cv::ADAPTIVE_THRESH_GAUSSIAN_C, then the pixels in the region around (𝑥,𝑦) are weighted

threshold types depending on the settings of adaptiveMethod. In both cases, the adaptive threshold T(x, y) is set on a pixel-by-pixel basis by computing a weighted average of the b-by-b region around each pixel location minus a constant, where 𝑏 is given by blockSize and the constant is given by C. If the method is set to cv::ADAPTIVE_THRESH_MEAN_C, then all pixels in the area are weighted equally. If it is set to cv::ADAPTIVE_THRESH_GAUSSIAN_C, then the pixels in the region around (𝑥,𝑦) are weighted according to a Gaussian function of their distance from that center point. Finally, the parameter thresholdType is the same as for cv::threshold() shown in Table 5-2. The adaptive threshold technique is useful when there are strong illumination or reflectance gradients that you need to threshold relative to the general intensity gradient. This function handles only single-channel 8-bit or floating-point images, and it requires that the source and destination images be distinct. Example 5-3 shows source code for

to a Gaussian function of their distance from that center point. Finally, the parameter thresholdType is the same as for cv::threshold() shown in Table 5-2. The adaptive threshold technique is useful when there are strong illumination or reflectance gradients that you need to threshold relative to the general intensity gradient. This function handles only single-channel 8-bit or floating-point images, and it requires that the source and destination images be distinct. Example 5-3 shows source code for comparing cv::adaptiveThreshold() and cv::threshold().

Figure 5-5 illustrates the result of processing an image that has a strong lighting gradient across it with both functions. The lower-left portion of the figure shows the result of using a single global threshold as in cv::threshold(); the lower-right portion shows the result of adaptive local threshold using cv::adaptiveThreshold().  We see that we get the whole checkerboard via adaptive threshold, a result that is impossible to achieve when using a single threshold. Note the calling-convention comments at the top of the code in Example 5-3; the parameters used for Figure 5-5 were: ./adaptThresh 15 1 1 71 15 ../Data/cal3-L.bmp   Figure 5-5: Binary threshold versus adaptive binary threshold: the input image (top) was turned into a Boolean image using a global threshold (lower-left) and an adaptive threshold (lower-right); raw image courtesy of Kurt Konolige Example 5-3: Threshold versus adaptive threshold #include <iostream>  using namespace std;  int main( int argc, char** argv ) {   if(argc != 7) { cout

at the top of the code in Example 5-3; the parameters used for Figure 5-5 were: ./adaptThresh 15 1 1 71 15 ../Data/cal3-L.bmp   Figure 5-5: Binary threshold versus adaptive binary threshold: the input image (top) was turned into a Boolean image using a global threshold (lower-left) and an adaptive threshold (lower-right); raw image courtesy of Kurt Konolige Example 5-3: Threshold versus adaptive threshold #include <iostream>  using namespace std;  int main( int argc, char** argv ) {   if(argc != 7) { cout <<    "Usage: ch5_ex5_3 fixed_threshold invert(0=off|1=on) "    "adaptive_type(0=mean|1=gaussian) block_size offset image\n"    "Example: ch5_ex5_3 100 1 0 15 10 fruits.jpg\n"; return -1; }    // Command line   double fixed_threshold = (double)atof(argv[1]);   int threshold_type  = atoi(argv[2]) ? cv::THRESH_BINARY : cv::THRESH_BINARY_INV;   int adaptive_method = atoi(argv[3]) ? cv::ADAPTIVE_THRESH_MEAN_C                                        : cv::ADAPTIVE_THRESH_GAUSSIAN_C;   int block_size =

7) { cout <<    "Usage: ch5_ex5_3 fixed_threshold invert(0=off|1=on) "    "adaptive_type(0=mean|1=gaussian) block_size offset image\n"    "Example: ch5_ex5_3 100 1 0 15 10 fruits.jpg\n"; return -1; }    // Command line   double fixed_threshold = (double)atof(argv[1]);   int threshold_type  = atoi(argv[2]) ? cv::THRESH_BINARY : cv::THRESH_BINARY_INV;   int adaptive_method = atoi(argv[3]) ? cv::ADAPTIVE_THRESH_MEAN_C                                        : cv::ADAPTIVE_THRESH_GAUSSIAN_C;   int block_size = atoi(argv[4]);   double offset = (double)atof(argv[5]);   cv::Mat Igray = cv::imread(argv[6], cv::LOAD_IMAGE_GRAYSCALE);    // Read in gray image

if( Igray.empty() ){ cout << "Can not load " << argv[6] << endl; return -1; }    // Declare the output images   cv::Mat It, Iat;    // Thresholds   cv::threshold(     Igray,     It,     fixed_threshold,     255,     threshold_type);   cv::adaptiveThreshold(     Igray,      Iat,      255,      adaptive_method,     threshold_type,      block_size,      offset   );    // Show the results   cv::imshow("Raw",Igray);   cv::imshow("Threshold",It);   cv::imshow("Adaptive Threshold",Iat);   cv::waitKey(0);   return 0; } Smoothing  Figure 5-6: Gaussian blur on 1D pixel array Smoothing, also called blurring, is a simple and frequently used image processing operation. There are many reasons for smoothing, but it is often done to reduce noise or camera artifacts. Smoothing is also important when we wish to reduce the resolution of an image in a principled way (we will discuss this in more detail in the “Image Pyramids” section of this chapter). OpenCV offers five different smoothing operations, each with its own

return 0; } Smoothing  Figure 5-6: Gaussian blur on 1D pixel array Smoothing, also called blurring, is a simple and frequently used image processing operation. There are many reasons for smoothing, but it is often done to reduce noise or camera artifacts. Smoothing is also important when we wish to reduce the resolution of an image in a principled way (we will discuss this in more detail in the “Image Pyramids” section of this chapter). OpenCV offers five different smoothing operations, each with its own associated library function, which each accomplish slightly different kinds of smoothing.

The src and dst arguments in all of these functions are the usual source and destination arrays. After that, each smoothing operation has parameters that are specific to the associated operation. Of these, the only common parameter is the last, borderType. This argument tells the smoothing operation how to handle pixels at the edge of the image. Simple Blur and the Box Filter  Figure 5-7: Image smoothing by block averaging: on the left are the input images; on the right, the output images void cv::blur(   cv::InputArray  src,                         // Input Image   cv::OutputArray dst,                         // Result image   cv::Size        ksize,                       // Kernel size      cv::Point       anchor     = cv::Point(-1,-1),  // Location of anchor point   int             borderType = cv::BORDER_DEFAULT // Border extrapolation to use );  The simple blur operation is provided by cv::blur(). Each pixel in the output is the simple mean of all of the pixels in a window, usually called a kernel,

cv::InputArray  src,                         // Input Image   cv::OutputArray dst,                         // Result image   cv::Size        ksize,                       // Kernel size      cv::Point       anchor     = cv::Point(-1,-1),  // Location of anchor point   int             borderType = cv::BORDER_DEFAULT // Border extrapolation to use );  The simple blur operation is provided by cv::blur(). Each pixel in the output is the simple mean of all of the pixels in a window, usually called a kernel, around the corresponding pixel in the input. The size of this window is specified by the argument ksize. The argument anchor can be used to specify how the kernel is aligned with the pixel being computed. By default, the value of anchor is cv::Point(-1,-1), which indicates that the kernel should be centered relative to the filter. In the case of multichannel images, each channel will be computed separately.    1 1 1 1 1   1 1 1 1 1 1/25  x 1 1 1 1 1   1 1 1 1 1   1 1 1 1 1  Figure 5-8: A 5-by-5 blur filter,

the corresponding pixel in the input. The size of this window is specified by the argument ksize. The argument anchor can be used to specify how the kernel is aligned with the pixel being computed. By default, the value of anchor is cv::Point(-1,-1), which indicates that the kernel should be centered relative to the filter. In the case of multichannel images, each channel will be computed separately.    1 1 1 1 1   1 1 1 1 1 1/25  x 1 1 1 1 1   1 1 1 1 1   1 1 1 1 1  Figure 5-8: A 5-by-5 blur filter, also called a normalized box filter.

The simple blur is a specialized version of the box filter. A box filter is any filter that has a rectangular profile and for which the values 𝑘!,! are all equal. In most cases, 𝑘!,!=1 for all 𝑖,𝑗, or 𝑘!,!=1𝐴, where 𝐴 is the area of the filter. The latter case is called a normalized box filter. void cv::boxFilter(   cv::InputArray  src,                 // Input Image   cv::OutputArray dst,                 // Result image   int             ddepth,              // Pixel depth of output image (e.g., cv::U8)   cv::Size        ksize,               // Kernel size      cv::Point       anchor     = cv::Point(-1,-1),  // Location of anchor point   bool            normalize  = true,              // If true, divide by box area   int             borderType = cv::BORDER_DEFAULT // Border extrapolation to use );  The OpenCV function cv::boxFilter() is the somewhat more general form of which cv::blur() is essentially a special case. The main difference between cv::boxFilter() and cv::blur() is that the former can be run

cv::Size        ksize,               // Kernel size      cv::Point       anchor     = cv::Point(-1,-1),  // Location of anchor point   bool            normalize  = true,              // If true, divide by box area   int             borderType = cv::BORDER_DEFAULT // Border extrapolation to use );  The OpenCV function cv::boxFilter() is the somewhat more general form of which cv::blur() is essentially a special case. The main difference between cv::boxFilter() and cv::blur() is that the former can be run in an un-normalized mode (normalize = false), and that the depth of the output image dst can be controlled. (In the case of cv::blur(), the depth of dst will always equal the depth of src.) If the value of ddepth is set to -1, then the destination image will have the same depth as the source; otherwise, you can use any of the usual aliases (e.g., cv::F32).  Median Filter  Figure 5-9: Image blurring by taking the median of surrounding pixels

The median filter [Bardyn84] replaces each pixel by the median or “middle-valued” pixel (as opposed to the mean pixel) in a rectangular neighborhood around the center pixel.7 Results of median filtering are shown in  Figure 5-9. Simple blurring by averaging can be sensitive to noisy images, especially images with large isolated outlier values (sometimes called “shot noise”). Large differences in even a small number of points can cause a noticeable movement in the average value. Median filtering is able to ignore the outliers by selecting the middle points, though at a cost in speed. void cv::medianBlur(   cv::InputArray  src,                 // Input Image   cv::OutputArray dst,                 // Result image   cv::Size        ksize                // Kernel size    ); The arguments to cv::medianBlur are essentially the same as with previous filters, the source array src, the destination array dst, and the kernel size ksize. For cv::medianBlur(), the anchor point is always assumed to be at the center of the

ignore the outliers by selecting the middle points, though at a cost in speed. void cv::medianBlur(   cv::InputArray  src,                 // Input Image   cv::OutputArray dst,                 // Result image   cv::Size        ksize                // Kernel size    ); The arguments to cv::medianBlur are essentially the same as with previous filters, the source array src, the destination array dst, and the kernel size ksize. For cv::medianBlur(), the anchor point is always assumed to be at the center of the kernel.                                                               7 Note that the median filter is an example of a nonlinear kernel, which cannot be represented in the pictorial style shown in Figure 5-1.

Gaussian Filter  Figure 5-10: Gaussian filtering (blurring) The next smoothing filter, the Gaussian filter, is probably the most useful. Gaussian filtering is done by convolving each point in the input array with a Gaussian kernel and then summing to produce the output array: void cv::GaussianBlur(   cv::InputArray  src,                 // Input Image   cv::OutputArray dst,                 // Result image   cv::Size        ksize,               // Kernel size      double          sigmaX,              // Gaussian half-width in x-direction   double          sigmaY      = 0.0,   // Gaussian half-width in y-direction   int             borderType = cv::BORDER_DEFAULT // Border extrapolation to use ); For the Gaussian blur (Figure 5-11), the parameter ksize gives the width and height of the filter window. The next parameter indicates the sigma value (half width at half max) of the Gaussian kernel in the x-dimension. The fourth parameter similarly indicates the sigma value in the y-dimension. If you specify only

Gaussian half-width in x-direction   double          sigmaY      = 0.0,   // Gaussian half-width in y-direction   int             borderType = cv::BORDER_DEFAULT // Border extrapolation to use ); For the Gaussian blur (Figure 5-11), the parameter ksize gives the width and height of the filter window. The next parameter indicates the sigma value (half width at half max) of the Gaussian kernel in the x-dimension. The fourth parameter similarly indicates the sigma value in the y-dimension. If you specify only the x value, and set the y value to zero (its default value), then the y and x values will be taken to be equal. If you set them both to zero, then the Gaussian’s parameters will be automatically determined from the window size using the following formulae: 𝜎!=𝑛!−12∙0.30+0.80,𝑛!=𝑘𝑠𝑖𝑧𝑒.𝑤𝑖𝑑𝑡ℎ−1, 𝜎!=𝑛!−12∙0.30+0.80,𝑛!=𝑘𝑠𝑖𝑧𝑒.ℎ𝑒𝑖𝑔ℎ𝑡−1. Finally, cv::GaussianBlur() takes the usual borderType argument.   1 4 7 4 1 1/1411 x 7 26 41 26 7   1 4 7 4 1        Figure 5-11: An example Gaussian kernel. Here ksize=(5,3),

the x value, and set the y value to zero (its default value), then the y and x values will be taken to be equal. If you set them both to zero, then the Gaussian’s parameters will be automatically determined from the window size using the following formulae: 𝜎!=𝑛!−12∙0.30+0.80,𝑛!=𝑘𝑠𝑖𝑧𝑒.𝑤𝑖𝑑𝑡ℎ−1, 𝜎!=𝑛!−12∙0.30+0.80,𝑛!=𝑘𝑠𝑖𝑧𝑒.ℎ𝑒𝑖𝑔ℎ𝑡−1. Finally, cv::GaussianBlur() takes the usual borderType argument.   1 4 7 4 1 1/1411 x 7 26 41 26 7   1 4 7 4 1        Figure 5-11: An example Gaussian kernel. Here ksize=(5,3), sigmaX=1, and sigmaY=0.5 Gaussian smoothing is faster than one might expect. Because convolution by Gaussians is linearly combinable, we may separate the convolution along the x and the y axis converting the operation from

being n2 in an n-by-n kernel size to being 2n for each pixel. The OpenCV implementation of Gaussian smoothing also provides even higher performance for several common kernels. 3-by-3, 5-by-5, and 7-by-7 with the “standard” sigma (i.e., sigmaX = 0.0) give better performance than other kernels. Gaussian blur supports single- or three-channel images in either 8-bit or 32-bit floating-point formats, and it can be done in place. Results of Gaussian blurring are shown in Figure 5-10. Bilateral Filter  Figure 5-12: Results of bilateral smoothing void cv::bilateralFilter(   cv::InputArray  src,                 // Input Image   cv::OutputArray dst,                 // Result image   int             d,                   // Pixel neighborhood size (max distance)   double          sigmaColor,          // Width parameter for color weighting function    double          sigmaSpace,          // Width parameter for spatial weighting function   int             borderType = cv::BORDER_DEFAULT // Border extrapolation to use );

Results of bilateral smoothing void cv::bilateralFilter(   cv::InputArray  src,                 // Input Image   cv::OutputArray dst,                 // Result image   int             d,                   // Pixel neighborhood size (max distance)   double          sigmaColor,          // Width parameter for color weighting function    double          sigmaSpace,          // Width parameter for spatial weighting function   int             borderType = cv::BORDER_DEFAULT // Border extrapolation to use ); The fifth and final form of smoothing supported by OpenCV is called bilateral filtering [Tomasi98], an example of which is shown in Figure 5-12. Bilateral filtering is one operation from a somewhat larger class of image analysis operators known as edge-preserving smoothing. Bilateral filtering is most easily understood when contrasted to Gaussian smoothing. A typical motivation for Gaussian smoothing is that pixels in a real image should vary slowly over space and thus be correlated to their neighbors, whereas

fifth and final form of smoothing supported by OpenCV is called bilateral filtering [Tomasi98], an example of which is shown in Figure 5-12. Bilateral filtering is one operation from a somewhat larger class of image analysis operators known as edge-preserving smoothing. Bilateral filtering is most easily understood when contrasted to Gaussian smoothing. A typical motivation for Gaussian smoothing is that pixels in a real image should vary slowly over space and thus be correlated to their neighbors, whereas random noise can be expected to vary greatly from one pixel to the next (i.e., noise is not spatially correlated). It is in this sense that Gaussian smoothing reduces noise while preserving signal. Unfortunately, this method breaks down near edges, where you do expect pixels to be uncorrelated with their neighbors across the edge. As a result, Gaussian smoothing blurs away edges. At the cost of what is unfortunately substantially more processing time, bilateral filtering provides a means of smoothing an

random noise can be expected to vary greatly from one pixel to the next (i.e., noise is not spatially correlated). It is in this sense that Gaussian smoothing reduces noise while preserving signal. Unfortunately, this method breaks down near edges, where you do expect pixels to be uncorrelated with their neighbors across the edge. As a result, Gaussian smoothing blurs away edges. At the cost of what is unfortunately substantially more processing time, bilateral filtering provides a means of smoothing an image without smoothing away the edges. Like Gaussian smoothing, bilateral filtering constructs a weighted average of each pixel and its neighboring components. The weighting has two components, the first of which is the same weighting used by Gaussian smoothing. The second component is also a Gaussian weighting but is based not on the spatial distance from the center pixel but rather on the difference in intensity8 from the center pixel.9 When intensities are

without smoothing away the edges. Like Gaussian smoothing, bilateral filtering constructs a weighted average of each pixel and its neighboring components. The weighting has two components, the first of which is the same weighting used by Gaussian smoothing. The second component is also a Gaussian weighting but is based not on the spatial distance from the center pixel but rather on the difference in intensity8 from the center pixel.9 When intensities are                                                              8 In the case of multichannel (i.e., color) images, the difference in intensity is replaced with a weighted sum over colors. This weighting is chosen to enforce a Euclidean distance in the CIE Lab color space.

close, the intensity weighting is nearly 1.0 and we get Gaussian smoothing, but when there are large differences in intensity, the weight is nearly zero and almost no averaging is done, keeping high-contrast edges sharp. The effect of this filter is typically to turn an image into what appears to be a watercolor painting of the same scene.10 This can be useful as an aid to segmenting the image. Bilateral filtering takes three parameters (other than the source and destination). The first is the diameter d of the pixel neighborhood that is considered when filtering. The second is the width of the Gaussian kernel used in the color domain called sigmaColor, which is analogous to the sigma parameters in the Gaussian filter. The third is the width of the Gaussian kernel in the spatial domain called sigmaSpace. The larger this second parameter, the broader the range of intensities (or colors) that will be included in the smoothing (and thus the more extreme a discontinuity must be in order to be preserved). The

d of the pixel neighborhood that is considered when filtering. The second is the width of the Gaussian kernel used in the color domain called sigmaColor, which is analogous to the sigma parameters in the Gaussian filter. The third is the width of the Gaussian kernel in the spatial domain called sigmaSpace. The larger this second parameter, the broader the range of intensities (or colors) that will be included in the smoothing (and thus the more extreme a discontinuity must be in order to be preserved). The filter size has a strong effect (as you might expect) on the speed of the algorithm. Typical values are less than or equal to five for video processing, but might be as high as nine for non-real-time applications. As an alternative to specifying d explicitly, it can be set to -1, in which case, it will be automatically computed from sigmaSpace. In practice, small values (e.g., 10) give a very light, but noticeable effect, while large values (e.g., 150) have a very strong effect and tend to render the image

filter size has a strong effect (as you might expect) on the speed of the algorithm. Typical values are less than or equal to five for video processing, but might be as high as nine for non-real-time applications. As an alternative to specifying d explicitly, it can be set to -1, in which case, it will be automatically computed from sigmaSpace. In practice, small values (e.g., 10) give a very light, but noticeable effect, while large values (e.g., 150) have a very strong effect and tend to render the image into a somewhat “cartoonish” appearance.  Derivatives and Gradients One of the most basic and important convolutions is the computation of derivatives (or approximations to them). There are many ways to do this, but only a few are well suited to a given situation. The Sobel Derivative In general, the most common operator used to represent differentiation is the Sobel derivative [Sobel68] operator (see Figure 5-13 and Figure 5-14). Sobel operators exist for any order of derivative as well as for mixed

image into a somewhat “cartoonish” appearance.  Derivatives and Gradients One of the most basic and important convolutions is the computation of derivatives (or approximations to them). There are many ways to do this, but only a few are well suited to a given situation. The Sobel Derivative In general, the most common operator used to represent differentiation is the Sobel derivative [Sobel68] operator (see Figure 5-13 and Figure 5-14). Sobel operators exist for any order of derivative as well as for mixed partial derivatives (e.g., 𝜕!/𝜕𝑥𝜕𝑦).                                                                                                                                                                                      9 Technically, the use of Gaussian distribution functions is not a necessary feature of bilateral filtering. The implementation in OpenCV uses Gaussian weighting even though the method allows many possible weighting functions. 10 This effect is particularly pronounced after multiple

for mixed partial derivatives (e.g., 𝜕!/𝜕𝑥𝜕𝑦).                                                                                                                                                                                      9 Technically, the use of Gaussian distribution functions is not a necessary feature of bilateral filtering. The implementation in OpenCV uses Gaussian weighting even though the method allows many possible weighting functions. 10 This effect is particularly pronounced after multiple iterations of bilateral filtering.

Figure 5-13: The effect of the Sobel operator when used to approximate a first derivative in the x-dimension. void cv::Sobel(    cv::InputArray  src,                 // Input Image   cv::OutputArray dst,                 // Result image   int             ddepth,              // Pixel depth of output image (e.g., cv::U8)   int             xorder,              // order of corresponding derivative in x   int             yorder,              // order of corresponding derivative in y   cv::Size        ksize = 3,           // Kernel size      double          scale = 1,           // Scale applied before assignment to dst   double          delta = 0,           // Offset applied before assignment to dst   int             borderType = cv::BORDER_DEFAULT // Border extrapolation to use ); Here, src and dst are your image input and output. The argument ddepth allows you to select the depth (type) of the generated output (e.g., cv::F32). As a good example of how to use ddepth, if src is an 8-bit image, then the dst

= 3,           // Kernel size      double          scale = 1,           // Scale applied before assignment to dst   double          delta = 0,           // Offset applied before assignment to dst   int             borderType = cv::BORDER_DEFAULT // Border extrapolation to use ); Here, src and dst are your image input and output. The argument ddepth allows you to select the depth (type) of the generated output (e.g., cv::F32). As a good example of how to use ddepth, if src is an 8-bit image, then the dst should have a depth of at least cv::S16 to avoid overflow. xorder and yorder are the orders of the derivative. Typically, you’ll use 0, 1, or at most 2; a 0 value indicates no derivative in that direction.11 The ksize parameter should be odd and is the width (and the height) of the filter used. Currently, aperture sizes up to 31 are supported.12 The scale factor and delta are applied to the derivative before storing in dst. This can be useful when you want to actually visualize a derivative in an 8-bit image

have a depth of at least cv::S16 to avoid overflow. xorder and yorder are the orders of the derivative. Typically, you’ll use 0, 1, or at most 2; a 0 value indicates no derivative in that direction.11 The ksize parameter should be odd and is the width (and the height) of the filter used. Currently, aperture sizes up to 31 are supported.12 The scale factor and delta are applied to the derivative before storing in dst. This can be useful when you want to actually visualize a derivative in an 8-bit image you can show on the screen: 𝑑𝑠𝑡!=𝑠𝑐𝑎𝑙𝑒∙𝑘!,!∗𝐼𝑥+𝑖,𝑦+𝑗!,!"#$%&!!"#$"%+𝑑𝑒𝑙𝑡𝑎. The borderType argument functions exactly as described for other convolution operations. Sobel computations are fast for the same reason Gaussian blurring is fast: the kernel can be separated and so, rather than a 2D x,y convolution over the area of the kernel, we reduce to a 1D x convolution combined with a 1D y convolution (see the exercises below).                                                              11 Either xorder or yorder

show on the screen: 𝑑𝑠𝑡!=𝑠𝑐𝑎𝑙𝑒∙𝑘!,!∗𝐼𝑥+𝑖,𝑦+𝑗!,!"#$%&!!"#$"%+𝑑𝑒𝑙𝑡𝑎. The borderType argument functions exactly as described for other convolution operations. Sobel computations are fast for the same reason Gaussian blurring is fast: the kernel can be separated and so, rather than a 2D x,y convolution over the area of the kernel, we reduce to a 1D x convolution combined with a 1D y convolution (see the exercises below).                                                              11 Either xorder or yorder must be nonzero. 12 In practice, it really only makes sense to set the kernel size to 3 or greater.  If you set ksize to 1, then the kernel size will automatically be adjusted up to 3.

Figure 5-14: The effect of the Sobel operator when used to approximate a first derivative in the y-dimension. Sobel derivatives have the nice property that they can be defined for kernels of any size, and those kernels can be constructed quickly and iteratively. Up to the limit of not spanning key image structures, larger kernels give a better approximation to the derivative because they span more area and are thus less sensitive to noise. To understand this more exactly, we must realize that a Sobel derivative is not really a derivative as it is defined on a discrete space. What the Sobel operator actually represents is a fit to a polynomial. That is, the Sobel derivative of second order in the x-direction is not really a second derivative; it is a local fit to a parabolic function. This explains why one might want to use a larger kernel: that larger kernel is computing the fit over a larger number of pixels. Scharr Filter In fact, there are many ways to approximate a derivative in the case of a discrete

is not really a derivative as it is defined on a discrete space. What the Sobel operator actually represents is a fit to a polynomial. That is, the Sobel derivative of second order in the x-direction is not really a second derivative; it is a local fit to a parabolic function. This explains why one might want to use a larger kernel: that larger kernel is computing the fit over a larger number of pixels. Scharr Filter In fact, there are many ways to approximate a derivative in the case of a discrete grid. The downside of the approximation used for the Sobel operator is that it is less accurate for small kernels. For large kernels, where more points are used in the approximation, this problem is less significant. This inaccuracy does not show up directly for the x and y filters used in cv::Sobel(), because they are exactly aligned with the x- and y-axes. The difficulty arises when you want to make image measurements that are approximations of

directional derivatives (i.e., direction of the image gradient by using the arctangent of the 𝑦/𝑥 filter responses)13. To put this in context, a concrete example of where you may want image measurements of this kind would be in the process of collecting shape information from an object by assembling a histogram of gradient angles around the object. Such a histogram is the basis on which many common shape classifiers are trained and operated. In this case, inaccurate measures of gradient angle will decrease the recognition performance of the classifier because the data to be learned will vary depending on the rotation of the object. For a 3-by-3 Sobel filter, the inaccuracies are more apparent the further the gradient angle is from horizontal or vertical. OpenCV addresses this inaccuracy for small (but fast) 3-by-3 Sobel derivative filters by a somewhat obscure use of the special ksize value cv::SCHARR in the cv::Sobel() function. The Scharr filter is just as fast but more accurate than the Sobel filter, so

the recognition performance of the classifier because the data to be learned will vary depending on the rotation of the object. For a 3-by-3 Sobel filter, the inaccuracies are more apparent the further the gradient angle is from horizontal or vertical. OpenCV addresses this inaccuracy for small (but fast) 3-by-3 Sobel derivative filters by a somewhat obscure use of the special ksize value cv::SCHARR in the cv::Sobel() function. The Scharr filter is just as fast but more accurate than the Sobel filter, so it should always be used if you want to make image measurements using a 3-by-3 filter. The filter coefficients for the Scharr filter are shown in Figure 5-15 [Scharr00]. -3 0 +3      -3 -10 -3 -10 0 +10      0 0 0 -3 0 +3      +3 +10 +3  A)        B)  Figure 5-15: The 3-by-3 Scharr filter using flag cv::SCHARR The Laplacian The OpenCV Laplacian function (first used in vision by Marr [Marr82]) implements a discrete approximation to the Laplacian operator14: 𝐿𝑎𝑝𝑙𝑎𝑐𝑒𝑓=  𝜕!𝑓𝜕𝑥!+𝜕!𝑓𝜕𝑦! Because the Laplacian

so it should always be used if you want to make image measurements using a 3-by-3 filter. The filter coefficients for the Scharr filter are shown in Figure 5-15 [Scharr00]. -3 0 +3      -3 -10 -3 -10 0 +10      0 0 0 -3 0 +3      +3 +10 +3  A)        B)  Figure 5-15: The 3-by-3 Scharr filter using flag cv::SCHARR The Laplacian The OpenCV Laplacian function (first used in vision by Marr [Marr82]) implements a discrete approximation to the Laplacian operator14: 𝐿𝑎𝑝𝑙𝑎𝑐𝑒𝑓=  𝜕!𝑓𝜕𝑥!+𝜕!𝑓𝜕𝑦! Because the Laplacian operator can be defined in terms of second derivatives, you might well suppose that the discrete implementation works something like the second-order Sobel derivative. Indeed it does, and in fact, the OpenCV implementation of the Laplacian operator uses the Sobel operators directly in its computation: void cv::Laplacian(     cv::InputArray  src,                 // Input Image   cv::OutputArray dst,                 // Result image   int             ddepth,              // Pixel depth of output image (e.g.,

operator can be defined in terms of second derivatives, you might well suppose that the discrete implementation works something like the second-order Sobel derivative. Indeed it does, and in fact, the OpenCV implementation of the Laplacian operator uses the Sobel operators directly in its computation: void cv::Laplacian(     cv::InputArray  src,                 // Input Image   cv::OutputArray dst,                 // Result image   int             ddepth,              // Pixel depth of output image (e.g., cv::U8)   cv::Size        ksize = 3,           // Kernel size      double          scale = 1,           // Scale applied before assignment to dst   double          delta = 0,           // Offset applied before assignment to dst   int             borderType = cv::BORDER_DEFAULT // Border extrapolation to use ); The cv::Laplacian() function takes the same arguments as the cv::Sobel() function, with the exception that the orders of the derivatives are not needed. This aperture ksize is precisely the same as

cv::U8)   cv::Size        ksize = 3,           // Kernel size      double          scale = 1,           // Scale applied before assignment to dst   double          delta = 0,           // Offset applied before assignment to dst   int             borderType = cv::BORDER_DEFAULT // Border extrapolation to use ); The cv::Laplacian() function takes the same arguments as the cv::Sobel() function, with the exception that the orders of the derivatives are not needed. This aperture ksize is precisely the same as the aperture appearing in the Sobel derivatives and, in effect, gives the size of the region over which the pixels are sampled in the computation of the second derivatives. In the actual implementation, for ksize anything other than 1, the Laplacian is computed directly from the sum of the corresponding Sobel                                                              13 As you might recall, there are functions cv::cartToPolar() and cv::polarToCart() that implement exactly this transformation. If you find

the aperture appearing in the Sobel derivatives and, in effect, gives the size of the region over which the pixels are sampled in the computation of the second derivatives. In the actual implementation, for ksize anything other than 1, the Laplacian is computed directly from the sum of the corresponding Sobel                                                              13 As you might recall, there are functions cv::cartToPolar() and cv::polarToCart() that implement exactly this transformation. If you find yourself wanting to call cv::cartToPolar() on a pair of 𝑥- and y-derivative images, you should probably be using CV_SCHARR to compute those images. 14 Note that the Laplacian operator is distinct from the Laplacian pyramid, which we will discuss in Chapter 6.

operators. In the special case of ksize=1, the Laplacian is computed by convolution with the following single kernel: 0 1 0 1 -4 1 0 1 0    Figure 5-16. The single kernel used by cv::Laplacian() when ksize=1  Figure 5-17: Laplace transform (upper right) of the racecar image: zooming in on the tire (circled in white) and considering only the x-dimension, we show a (qualitative) representation of the brightness as well as the first and second derivatives (lower three cells); the 0s in the second derivative correspond to edges, and the 0 corresponding to a large first derivative is a strong edge The Laplace operator can be used in a variety of contexts. A common application is to detect “blobs.” Recall that the form of the Laplacian operator is a sum of second derivatives along the x-axis and y-axis. This means that a single point or any small blob (smaller than the aperture) that is surrounded by higher values will tend to maximize this function. Conversely, a point or small blob that is surrounded by lower

correspond to edges, and the 0 corresponding to a large first derivative is a strong edge The Laplace operator can be used in a variety of contexts. A common application is to detect “blobs.” Recall that the form of the Laplacian operator is a sum of second derivatives along the x-axis and y-axis. This means that a single point or any small blob (smaller than the aperture) that is surrounded by higher values will tend to maximize this function. Conversely, a point or small blob that is surrounded by lower values will tend to maximize the negative of this function. With this in mind, the Laplace operator can also be used as a kind of edge detector. To see how this is done, consider the first derivative of a function, which will (of course) be large wherever the function is changing rapidly. Equally important, it will grow rapidly as we approach an edge-like discontinuity and shrink rapidly as we move past the discontinuity. Hence, the derivative will be at a local maximum somewhere within this range.

lower values will tend to maximize the negative of this function. With this in mind, the Laplace operator can also be used as a kind of edge detector. To see how this is done, consider the first derivative of a function, which will (of course) be large wherever the function is changing rapidly. Equally important, it will grow rapidly as we approach an edge-like discontinuity and shrink rapidly as we move past the discontinuity. Hence, the derivative will be at a local maximum somewhere within this range. Therefore, we can look to the 0s of the second derivative for locations of such local maxima. Got that? Edges in the original image will be 0s of the Laplacian. Unfortunately, both substantial and less meaningful edges will be 0s of the Laplacian, but this is not a problem because we can simply filter out those pixels that also have larger values of the first (Sobel) derivative. Figure 5-17 shows an example of using a Laplacian on an image together with details of the first and second derivatives and their

Therefore, we can look to the 0s of the second derivative for locations of such local maxima. Got that? Edges in the original image will be 0s of the Laplacian. Unfortunately, both substantial and less meaningful edges will be 0s of the Laplacian, but this is not a problem because we can simply filter out those pixels that also have larger values of the first (Sobel) derivative. Figure 5-17 shows an example of using a Laplacian on an image together with details of the first and second derivatives and their zero crossings.

Image Morphology  Figure 5-18: Summary results for all morphology operators OpenCV provides a fast, convenient interface for doing morphological transformations [Serra83] on an image. Image morphology is its own topic, and over the years, especially in the early days of computer vision, a great number of morphological operations were developed. Most were developed for one specific purpose or another, and some of those found broader utility over the years. Essentially, all morphology operations are based on just two primitive operations. We will start with those, and then move on to the more complex operations, each of which is typically defined in terms of its simpler predecessors. Dilation and Erosion  Figure 5-19: Morphological dilation: take the maximum under a square kernel The basic morphological transformations are called dilation and erosion, and they arise in a wide variety of contexts such as removing noise, isolating individual elements, and joining disparate elements in an image. More

are based on just two primitive operations. We will start with those, and then move on to the more complex operations, each of which is typically defined in terms of its simpler predecessors. Dilation and Erosion  Figure 5-19: Morphological dilation: take the maximum under a square kernel The basic morphological transformations are called dilation and erosion, and they arise in a wide variety of contexts such as removing noise, isolating individual elements, and joining disparate elements in an image. More sophisticated morphology operations, based on these two basic operations, can also be used to find intensity peaks (or holes) in an image and to define a particular form of image gradient.

Figure 5-20: Results of the dilation, or “max,” operator: bright regions are expanded and often joined Dilation is a convolution of some image with a kernel in which any given pixel is replaced with the local maximum of all of the pixel values covered by the kernel. As we mentioned earlier, this is an example of a nonlinear operation, so the kernel cannot be expressed in the form shown in Figure 5-1. Most often, the kernel used for Dilation is a “solid” square kernel, or sometimes a disk, with the anchor point at the center. The effect of dilation is to cause filled15 regions within an image to grow as diagrammed in  Figure 5-19.                                                              15 Here the term “filled” actually means those pixels whose value is nonzero. You could read this as “bright,” since the local maximum actually takes the pixel with the highest intensity value under the template (kernel). It is worth mentioning that the diagrams that appear in this chapter to illustrate morphological

point at the center. The effect of dilation is to cause filled15 regions within an image to grow as diagrammed in  Figure 5-19.                                                              15 Here the term “filled” actually means those pixels whose value is nonzero. You could read this as “bright,” since the local maximum actually takes the pixel with the highest intensity value under the template (kernel). It is worth mentioning that the diagrams that appear in this chapter to illustrate morphological operators are in this sense inverted relative to what would happen on your screen (because books write with dark ink on light paper instead of light pixels on a dark screen).

Erosion is the converse operation. The action of the erosion operator is equivalent to computing a local minimum over the area of the kernel.16 Erosion is diagrammed in  Figure 5-21. Image morphology is often done on Boolean17 images that result from a threshold operation. However, because dilation is just a max operator and erosion is just a min operator, morphology may be used on intensity images as well. In general, whereas dilation expands a bright region 𝐴, erosion ( Figure 5-21) reduces such a bright region. Moreover, dilation will tend to fill concavities and erosion will tend to remove protrusions. Of course, the exact result will depend on the kernel, but these statements are generally true so long as the kernel is both convex and filled. In OpenCV, we effect these transformations using the cv::erode() and cv::dilate() functions: void cv::erode(   cv::InputArray    src,               // Input Image   cv::OutputArray   dst,               // Result image   cv::InputArray    element,           //

region. Moreover, dilation will tend to fill concavities and erosion will tend to remove protrusions. Of course, the exact result will depend on the kernel, but these statements are generally true so long as the kernel is both convex and filled. In OpenCV, we effect these transformations using the cv::erode() and cv::dilate() functions: void cv::erode(   cv::InputArray    src,               // Input Image   cv::OutputArray   dst,               // Result image   cv::InputArray    element,           // Structuring element, can be cv::Mat()      cv::Point         anchor     = cv::Point(-1,-1),  // Location of anchor point   int               iterations = 1,                 // Number of times to apply   int               borderType = cv::BORDER_DEFAULT // Border extrapolation to use   const cv::Scalar& borderValue = cv::morphologyDefaultBorderValue()                                                               16 To be precise, the pixel in the destination image is set to the value equal to the minimal value of

element, can be cv::Mat()      cv::Point         anchor     = cv::Point(-1,-1),  // Location of anchor point   int               iterations = 1,                 // Number of times to apply   int               borderType = cv::BORDER_DEFAULT // Border extrapolation to use   const cv::Scalar& borderValue = cv::morphologyDefaultBorderValue()                                                               16 To be precise, the pixel in the destination image is set to the value equal to the minimal value of the pixels under the kernel in the source image. 17 It should be noted that OpenCV does not actually have a Boolean image data type. The minimum size representation is 8-bit characters. Those functions which interpret an image as Boolean do so by classifying all pixels as either zero (“False” or 0) or nonzero (“True” or 1).

);  Figure 5-21: Morphological erosion: take the minimum under a square kernel void cv::dilate(   cv::InputArray    src,               // Input Image   cv::OutputArray   dst,               // Result image   cv::InputArray    element,           // Structuring element, can be cv::Mat()      cv::Point         anchor     = cv::Point(-1,-1),  // Location of anchor point   int               iterations = 1,                 // Number of times to apply   int               borderType = cv::BORDER_CONSTANT // Border extrapolation to use   const cv::Scalar& borderValue = cv::morphologyDefaultBorderValue()  ); Both cv::erode() and cv::dilate() take a source and destination image, and both support “in place” calls (in which the source and destination are the same image). The third argument is the kernel, to which you may pass an uninitialized array cv::Mat(), which will cause it to default to using a 3-by-3 kernel with the anchor at its center (we will discuss how to create your own kernels later). The fourth argument is

extrapolation to use   const cv::Scalar& borderValue = cv::morphologyDefaultBorderValue()  ); Both cv::erode() and cv::dilate() take a source and destination image, and both support “in place” calls (in which the source and destination are the same image). The third argument is the kernel, to which you may pass an uninitialized array cv::Mat(), which will cause it to default to using a 3-by-3 kernel with the anchor at its center (we will discuss how to create your own kernels later). The fourth argument is the number of iterations. If not set to the default value of 1, the operation will be applied multiple times during the single call to the function. The borderType argument is the usual border type, and the borderValue is the value that will be used for off-the-edge pixels when the borderType is set to cv::BORDER_CONSTANT.  The results of an erode operation on a sample image are shown in

Figure 5-22 and those of a dilation operation on the same image are shown in  Figure 5-20. The erode operation is often used to eliminate “speckle” noise in an image. The idea here is that the speckles are eroded to nothing while larger regions that contain visually significant content are not affected. The dilate operation is often used when attempting to find connected components (i.e., large discrete regions of similar pixel color or intensity). The utility of dilation arises because in many cases a large region might otherwise be broken apart into multiple components as a result of noise, shadows, or some other similar effect. A small dilation will cause such components to “melt” together into one. To recap: when OpenCV processes the cv::erode() function, what happens beneath the hood is that the value of some point p is set to the minimum value of all of the points covered by the kernel when aligned at p; for the dilation operator, the equation is the same except that max is considered rather than min:

region might otherwise be broken apart into multiple components as a result of noise, shadows, or some other similar effect. A small dilation will cause such components to “melt” together into one. To recap: when OpenCV processes the cv::erode() function, what happens beneath the hood is that the value of some point p is set to the minimum value of all of the points covered by the kernel when aligned at p; for the dilation operator, the equation is the same except that max is considered rather than min:  Figure 5-22: Results of the erosion, or “min,” operator: bright regions are isolated and shrunk 𝑒𝑟𝑜𝑑𝑒𝑥,𝑦=min!,!∈!"#$"%𝑠𝑟𝑐𝑥+𝑖,𝑦+𝑗 𝑑𝑖𝑙𝑎𝑡𝑒𝑥,𝑦=max(!,!)∈!"#$"%𝑠𝑟𝑐(𝑥+𝑖,𝑦+𝑗) You might be wondering why we need a complicated formula when the earlier heuristic description was perfectly sufficient. Some users actually prefer such formulas but, more importantly, the formulas capture

some generality that isn’t apparent in the qualitative description. Observe that if the image is not Boolean, then the min and max operators play a less trivial role. Take another look at  Figure 5-22 and  Figure 5-20, which show the erosion and dilation operators applied to two real images. The General Morphology Function When working with Boolean images and image masks, the basic erode and dilate operations are usually sufficient. When working with grayscale or color images, however, a number of additional operations are often helpful. Several of the more useful operations can be handled by the multipurpose cv::morphologyEx() function. void cv::morphologyEx(   cv::InputArray    src,               // Input Image   cv::OutputArray   dst,               // Result image   int               op,                // Morphology operator to use e.g. cv::MOP_OPEN   cv::InputArray    element,           // Structuring element, can be cv::Mat()      cv::Point         anchor     = cv::Point(-1,-1),  // Location of anchor

operations are often helpful. Several of the more useful operations can be handled by the multipurpose cv::morphologyEx() function. void cv::morphologyEx(   cv::InputArray    src,               // Input Image   cv::OutputArray   dst,               // Result image   int               op,                // Morphology operator to use e.g. cv::MOP_OPEN   cv::InputArray    element,           // Structuring element, can be cv::Mat()      cv::Point         anchor     = cv::Point(-1,-1),  // Location of anchor point   int               iterations = 1,                 // Number of times to apply   int               borderType = cv::BORDER_DEFAULT // Border extrapolation to use   const cv::Scalar& borderValue = cv::morphologyDefaultBorderValue()

); In addition to the arguments that we saw with the cv::dilate() and cv::erode() functions, cv::morphologyEx() has one new—and very important—parameter. This new argument, called op, is the specific operation to be done; the possible values of this argument are listed on Table 5-3. Table 5-3: cv::morphologyEx() operation options Value of operation Morphological operator Requires temp image? cv::MOP_OPEN                  Opening No cv::MOP_CLOSE                  Closing No cv::MOP_GRADIENT  Morphological gradient Always cv::MOP_TOPHAT                  Top Hat For in-place only (src = dst) cv::MOP_BLACKHAT  Black Hat For in-place only (src = dst) Opening and closing  Figure 5-23: Morphological opening applied to a simple Boolean image. The first two operations, opening and closing, are actually simple combinations of the erosion and dilation operators. In the case of opening, we erode first and then dilate (Figure 5-23). Opening is often used to count regions in a Boolean image. For example, if we have

gradient Always cv::MOP_TOPHAT                  Top Hat For in-place only (src = dst) cv::MOP_BLACKHAT  Black Hat For in-place only (src = dst) Opening and closing  Figure 5-23: Morphological opening applied to a simple Boolean image. The first two operations, opening and closing, are actually simple combinations of the erosion and dilation operators. In the case of opening, we erode first and then dilate (Figure 5-23). Opening is often used to count regions in a Boolean image. For example, if we have thresholded an image of cells on a microscope slide, we might use opening to separate out cells that are near each other before counting the regions.   Figure 5-24: Morphological closing applied to a simple Boolean image. In the case of closing, we dilate first and then erode (Figure 5-24). Closing is used in most of the more sophisticated connected-component algorithms to reduce unwanted or noise-driven segments. For connected components, usually an erosion or closing operation is performed first to eliminate

an image of cells on a microscope slide, we might use opening to separate out cells that are near each other before counting the regions.   Figure 5-24: Morphological closing applied to a simple Boolean image. In the case of closing, we dilate first and then erode (Figure 5-24). Closing is used in most of the more sophisticated connected-component algorithms to reduce unwanted or noise-driven segments. For connected components, usually an erosion or closing operation is performed first to eliminate elements that

arise purely from noise and then an opening operation is used to connect nearby large regions. (Notice that, although the end result of using open or close is similar to using erode or dilate, these new operations tend to preserve the area of connected regions more accurately.)   Figure 5-25: Morphological opening operation applied to a (one-dimensional) non-Boolean image: the upward outliers are eliminated as a result   Figure 5-26: Morphological closing operation applied to a (one-dimensional) non-Boolean image: the downward outliers are eliminated as a result When used on non-Boolean images, the most prominent effect of closing is to eliminate lone outliers that are are lower in value than their neighbors, whereas the effect of opening is to eliminate lone outliers that are higher than their neighbors. Results of using the opening operator are shown in  Figure 5-25, and of the closing operator in   Figure 5-26.

Figure 5-27: Results of morphological opening on an image: small bright regions are removed, and the remaining bright regions are isolated but retain their size One last note on the opening and closing operators concerns how the iterations argument is interpreted. You might expect that asking for two iterations of closing would yield something like dilate-erode-dilate-erode. It turns out that this would not be particularly useful. What you usually want (and what you get) is dilate-dilate-erode-erode. In this way, not only the single outliers but also neighboring pairs of outliers will disappear. Figure 5-23 (c) and   Figure 5-26 (c) illustrate the effect of calling open and close (respectively) with an iteration count of two.  Figure 5-28: Results of morphological closing on an image: bright regions are joined but retain their basic size

Morphological gradient  Figure 5-29: Morphological gradient applied to a simple Boolean image. Our next available operator is the morphological gradient. For this one, it is probably easier to start with a formula and then figure out what it means: 𝑔𝑟𝑎𝑑𝑖𝑒𝑛𝑡𝑠𝑟𝑐=  𝑑𝑖𝑙𝑎𝑡𝑒𝑠𝑟𝑐–𝑒𝑟𝑜𝑑𝑒𝑠𝑟𝑐.  As we can see in Figure 5-29, the effect of subtracting the eroded (slightly reduced) image from the dilated (slightly enlarged) image is to leave behind a representation of the edges of objects in the original image.   Figure 5-30: Morphological gradient applied to (one-dimensional) non-Boolean image: as expected, the operator has its highest values where the grayscale image is changing most rapidly. With a grayscale image ( Figure 5-30), we see that the value of the operator is telling us something about how fast the image brightness is changing; this is why the name “morphological gradient” is justified. Morphological gradient is often used when we want to isolate the perimeters of bright regions so we can treat them as whole

5-30: Morphological gradient applied to (one-dimensional) non-Boolean image: as expected, the operator has its highest values where the grayscale image is changing most rapidly. With a grayscale image ( Figure 5-30), we see that the value of the operator is telling us something about how fast the image brightness is changing; this is why the name “morphological gradient” is justified. Morphological gradient is often used when we want to isolate the perimeters of bright regions so we can treat them as whole objects (or as whole parts of objects). The complete perimeter of a region tends to be found because a contracted

version is subtracted from an expanded version of the region, leaving a complete perimeter edge. This differs from calculating a gradient, which is much less likely to work around the full perimeter of an object.  Figure 5-31: Results of the morphological gradient operator: bright perimeter edges are identified Top Hat and Black Hat  Figure 5-32: The Top Hat (left) and Black Hat (right) morphology operators The last two operators are called Top Hat and Black Hat [Meyer78]. These operators are used to isolate patches that are, respectively, brighter or dimmer than their immediate neighbors. You would use these when trying to isolate parts of an object that exhibit brightness changes relative only to the object to which they are attached. This often occurs with microscope images of organisms or cells, for example. Both operations are defined in terms of the more primitive operators, as follows: 𝑇𝑜𝑝𝐻𝑎𝑡𝑠𝑟𝑐=  𝑠𝑟𝑐–𝑜𝑝𝑒𝑛𝑠𝑟𝑐,  𝐵𝑙𝑎𝑐𝑘𝐻𝑎𝑡𝑠𝑟𝑐=  𝑐𝑙𝑜𝑠𝑒𝑠𝑟𝑐–𝑠𝑟𝑐.

Figure 5-33: Results of morphological Top Hat operation: bright local peaks are isolated As you can see, the Top Hat operator subtracts the opened form of A from A. Recall that the effect of the open operation was to exaggerate small cracks or local drops. Thus, subtracting 𝑜𝑝𝑒𝑛(𝐴) from 𝐴 should reveal areas that are lighter than the surrounding region of 𝐴, relative to the size of the kernel (see Figure 5-33); conversely, the Black Hat operator reveals areas that are darker than the surrounding region of 𝐴 (Figure 5-34). Summary results for all the morphological operators discussed in this chapter are assembled in Figure 5-18.18  Figure 5-34: Results of morphological Black Hat operation: dark holes are isolated Making Your Own Kernel In the morphological operations we have looked at so far, the kernels considered were always square and 3-by-3. If you need something a little more general than that, OpenCV allows you to create your own kernel.                                                              18

𝐴 (Figure 5-34). Summary results for all the morphological operators discussed in this chapter are assembled in Figure 5-18.18  Figure 5-34: Results of morphological Black Hat operation: dark holes are isolated Making Your Own Kernel In the morphological operations we have looked at so far, the kernels considered were always square and 3-by-3. If you need something a little more general than that, OpenCV allows you to create your own kernel.                                                              18 Both of these operations (Top Hat and Black Hat) make more sense in grayscale morphology, where the structuring element is a matrix of real numbers (not just a Boolean mask) and the matrix is added to the current pixel neighborhood before taking a minimum or maximum. Unfortunately, this is not yet implemented in OpenCV.

In the case of morphology, the kernel is often called a structuring element, so the routine that allows you to create your own morphology kernels is called cv::getStructuringElement(). In fact, you can just create any array you like and use it as a structuring element in functions like cv::dilate(), cv::erode(), or cv::morphologyEx(), but this is often more work than is necessary. Often what you need is a nonsquare kernel of an otherwise common shape. This is what cv::getStructuringElement() is for: cv::Mat cv::getStructuringElement(   int       shape,                     // Element shape, e.g. cv::MORPH_RECT    cv::Size  ksize,                     // Size of structuring element (should be odd)    cv::Point anchor = cv::Point(-1,-1)  // Location of anchor point ); The first argument shape controls which basic shape will be used to create the element (Table 5-4), while ksize and anchor specify the size of the element and the location of the anchor point, respectively. As usual, if the anchor argument is left

cv::getStructuringElement(   int       shape,                     // Element shape, e.g. cv::MORPH_RECT    cv::Size  ksize,                     // Size of structuring element (should be odd)    cv::Point anchor = cv::Point(-1,-1)  // Location of anchor point ); The first argument shape controls which basic shape will be used to create the element (Table 5-4), while ksize and anchor specify the size of the element and the location of the anchor point, respectively. As usual, if the anchor argument is left with its default value of cv::Point(-1,-1), then cv::getStructuringElement() will take this to mean that the anchor should automatically be placed at the center of the element. Table 5-4 cv::getStructuringElement() element shapes Value of shape Element Description cv::MORPH_RECT                  Rectangular  𝐸!,!=1,∀𝑖,𝑗 cv::MORPH_ELLIPSE                  Elliptic  Ellipse with axes ksize.width and ksize.height. cv::MORPH_CROSS  Cross-shaped  𝐸!,!=1, iff 𝑖==𝑎𝑛𝑐ℎ𝑜𝑟.𝑦 or 𝑗==𝑎𝑛𝑐ℎ𝑜𝑟.𝑥 Of the options for shape

with its default value of cv::Point(-1,-1), then cv::getStructuringElement() will take this to mean that the anchor should automatically be placed at the center of the element. Table 5-4 cv::getStructuringElement() element shapes Value of shape Element Description cv::MORPH_RECT                  Rectangular  𝐸!,!=1,∀𝑖,𝑗 cv::MORPH_ELLIPSE                  Elliptic  Ellipse with axes ksize.width and ksize.height. cv::MORPH_CROSS  Cross-shaped  𝐸!,!=1, iff 𝑖==𝑎𝑛𝑐ℎ𝑜𝑟.𝑦 or 𝑗==𝑎𝑛𝑐ℎ𝑜𝑟.𝑥 Of the options for shape shown in Table 5-4, the last is there only for legacy compatibility. In the old C API (v1.x), there was a separate structure used for the purpose of expressing convolution kernels. There is no need to use this functionality now, as you can simply pass any cv::Mat to the morphological operators as a structuring element if you need something more complicated than the basic shape-based elements created by cv::getStructuringElement(). Convolution with an Arbitrary Linear Filter In the functions we have seen so

shown in Table 5-4, the last is there only for legacy compatibility. In the old C API (v1.x), there was a separate structure used for the purpose of expressing convolution kernels. There is no need to use this functionality now, as you can simply pass any cv::Mat to the morphological operators as a structuring element if you need something more complicated than the basic shape-based elements created by cv::getStructuringElement(). Convolution with an Arbitrary Linear Filter In the functions we have seen so far, the basic mechanics of the convolution were happening deep down below the level of the OpenCV API. We took some time to understand the basics of convolution, then we went on to look at a long list of functions that implemented different kinds of useful convolutions. In essentially every case, there was a kernel that was implied by the function we chose, and we just passed that function a little extra information that parameterized that particular filter type. For linear filters, however, it is

seen so far, the basic mechanics of the convolution were happening deep down below the level of the OpenCV API. We took some time to understand the basics of convolution, then we went on to look at a long list of functions that implemented different kinds of useful convolutions. In essentially every case, there was a kernel that was implied by the function we chose, and we just passed that function a little extra information that parameterized that particular filter type. For linear filters, however, it is possible to just provide the entire kernel and let OpenCV handle the convolution for us. From an abstract point of view, this is very straightforward: we just need a function that takes an array argument to describe the kernel and we are done. At a practical level, there is an important subtlety that strongly affects performance. That subtlety is that some kernels are separable, and others are not.   -1 0 +1    1           0 1 0 -2 0 +2  =  2  ⨂  -1 0 +1     1 1 1 -1 0 +1    1           0 1 0  (a)     (b)

possible to just provide the entire kernel and let OpenCV handle the convolution for us. From an abstract point of view, this is very straightforward: we just need a function that takes an array argument to describe the kernel and we are done. At a practical level, there is an important subtlety that strongly affects performance. That subtlety is that some kernels are separable, and others are not.   -1 0 +1    1           0 1 0 -2 0 +2  =  2  ⨂  -1 0 +1     1 1 1 -1 0 +1    1           0 1 0  (a)     (b)     (c)       (d)

Figure 5-35: The Sobel kernel (a) is separable; it can be expressed as two one-dimensional convolutions (b and c); (d) is an example of a non-separable kernel A separable kernel is one that can be thought of as two one-dimensional kernels, the application of which is done by first convolving with the 𝑥-kernel and then with the 𝑦-kernel. The benefit of this decomposition is that the computational cost of a kernel convolution is approximately the image area multiplied by the kernel area19. This means that convolving your image of area 𝐴 by an 𝑛-by-𝑛 kernel takes time proportional to 𝐴𝑛!, while convolving your image once by an 𝑛-by-1 kernel and then by a 1-by-𝑛 kernel takes time proportional to 𝐴𝑛+𝐴𝑛=2𝐴𝑛. For even n as small as 3 there is a benefit, and the benefit grows with 𝑛. Applying a general filter with cv::filter2D() Given that the number of operations required for an image convolution, at least at first glance20, seems to be the number of pixels in the image multiplied by the number of pixels in the

convolving your image of area 𝐴 by an 𝑛-by-𝑛 kernel takes time proportional to 𝐴𝑛!, while convolving your image once by an 𝑛-by-1 kernel and then by a 1-by-𝑛 kernel takes time proportional to 𝐴𝑛+𝐴𝑛=2𝐴𝑛. For even n as small as 3 there is a benefit, and the benefit grows with 𝑛. Applying a general filter with cv::filter2D() Given that the number of operations required for an image convolution, at least at first glance20, seems to be the number of pixels in the image multiplied by the number of pixels in the kernel, this can be a lot of computation and so is not something you want to do with some for loop and a lot of pointer de-referencing. In situations like this, it is better to let OpenCV do the work for you and take advantage of the internal optimizations. The OpenCV way to do this is with cv::filter2D(): cv::filter2D(   cv::InputArray  src,                 // Input Image   cv::OutputArray dst,                 // Result image   int             ddepth,              // Pixel depth of output image (e.g.,

the kernel, this can be a lot of computation and so is not something you want to do with some for loop and a lot of pointer de-referencing. In situations like this, it is better to let OpenCV do the work for you and take advantage of the internal optimizations. The OpenCV way to do this is with cv::filter2D(): cv::filter2D(   cv::InputArray  src,                 // Input Image   cv::OutputArray dst,                 // Result image   int             ddepth,              // Pixel depth of output image (e.g., cv::U8)    cv::InputArray  kernel,              // Your own kernel   cv::Point       anchor     = cv::Point(-1,-1),  // Location of anchor point   double          delta = 0,                      // Offset before assignment to dst   int             borderType = cv::BORDER_DEFAULT // Border extrapolation to use ); Here we create an array of the appropriate size, fill it with the coefficients of our linear filter, and then pass it together with the source and destination images into cv::filter2D(). As usual,

cv::U8)    cv::InputArray  kernel,              // Your own kernel   cv::Point       anchor     = cv::Point(-1,-1),  // Location of anchor point   double          delta = 0,                      // Offset before assignment to dst   int             borderType = cv::BORDER_DEFAULT // Border extrapolation to use ); Here we create an array of the appropriate size, fill it with the coefficients of our linear filter, and then pass it together with the source and destination images into cv::filter2D(). As usual, we can specify the depth of the resulting image with ddepth, the anchor point for the filter with anchor, and the border extrapolation method with borderType. The kernel can be of even size if its anchor point is defined; otherwise, it should be of odd size. If you want an overall offset applied to the result after application of the linear filter, you can use the argument delta. Applying a general separable filter with cv::sepFilter2D In the case where your kernel is separable, you will get the best

we can specify the depth of the resulting image with ddepth, the anchor point for the filter with anchor, and the border extrapolation method with borderType. The kernel can be of even size if its anchor point is defined; otherwise, it should be of odd size. If you want an overall offset applied to the result after application of the linear filter, you can use the argument delta. Applying a general separable filter with cv::sepFilter2D In the case where your kernel is separable, you will get the best performance from OpenCV by expressing it in its separated form and passing those one-dimensional kernels to OpenCV (e.g., passing the kernels shown in Figure 5-35b and Figure 5-35c instead of the one shown in Figure 5-35a.) The OpenCV function cv::sepFilter2D() is like cv::filter2D(), except that it expects these two one-dimensional kernels instead of one two-dimensional kernel. cv::sepFilter2D(   cv::InputArray  src,                 // Input Image   cv::OutputArray dst,                 // Result image   int

from OpenCV by expressing it in its separated form and passing those one-dimensional kernels to OpenCV (e.g., passing the kernels shown in Figure 5-35b and Figure 5-35c instead of the one shown in Figure 5-35a.) The OpenCV function cv::sepFilter2D() is like cv::filter2D(), except that it expects these two one-dimensional kernels instead of one two-dimensional kernel. cv::sepFilter2D(   cv::InputArray  src,                 // Input Image   cv::OutputArray dst,                 // Result image   int             ddepth,              // Pixel depth of output image (e.g., cv::U8)    cv::InputArray  rowKernel,           // 1-by-N row Kernel   cv::InputArray  columnKernel,        // M-by-1 column kernel   cv::Point       anchor     = cv::Point(-1,-1),  // Location of anchor point                                                              19 This statement is only exactly true for convolution in the spatial domain, which is how OpenCV handles only small kernels. 20 We say “at first glance” because it is also

ddepth,              // Pixel depth of output image (e.g., cv::U8)    cv::InputArray  rowKernel,           // 1-by-N row Kernel   cv::InputArray  columnKernel,        // M-by-1 column kernel   cv::Point       anchor     = cv::Point(-1,-1),  // Location of anchor point                                                              19 This statement is only exactly true for convolution in the spatial domain, which is how OpenCV handles only small kernels. 20 We say “at first glance” because it is also possible to perform convolutions in the frequency domain. In this case, for an n-by-n image and an 𝑚-by-𝑚 kernel with  𝑛  ≫𝑚, the computational time will be proportional to 𝑛!logn and not to the  𝑛!𝑚! that is expected for computations in the spatial domain. Because the frequency domain computation is independent of the size of the kernel, it is more efficient for large kernels. OpenCV automatically decides whether to do the convolution in the frequency domain based on the size of the kernel.

double          delta      = 0,                 // Offset before assignment to dst   int             borderType = cv::BORDER_DEFAULT // Border extrapolation to use ); All the arguments of cv::sepFilter2D() are the same as those of cv::filter2D(), with the exception of the replacement of the kernel argument with the rowKernel and columnKernel arguments. The latter two are expected to be 𝑛!-by-1 and 1-by-𝑛! arrays (with 𝑛! not necessarily equal to 𝑛!). The Filter Architecture Almost all of the functions described in this chapter are based on a unified Filter Architecture. The Filter Architecture is a class system that allows you (or the OpenCV developers) to create a new filter and automatically reuse all of the things that every filter has in common (the convolution steps, handling of the borders, etc.)  The Big Picture The process of doing convolutions is complex because these are always very computationally intensive operations. As a result, there is a great benefit to doing things very efficiently. On

described in this chapter are based on a unified Filter Architecture. The Filter Architecture is a class system that allows you (or the OpenCV developers) to create a new filter and automatically reuse all of the things that every filter has in common (the convolution steps, handling of the borders, etc.)  The Big Picture The process of doing convolutions is complex because these are always very computationally intensive operations. As a result, there is a great benefit to doing things very efficiently. On the other hand, writing very efficient code is often very difficult to do in a modular way. As a result, OpenCV has an entire architecture for handling just this problem. This architecture begins at the top with high-level operations like cv::blur(). This function takes your image, smoothes it, and gives it back. Sounds simple, but this is actually the result of many different layers operating together. When you want to do your own thing, you will have to figure out exactly which level you want to tinker

the other hand, writing very efficient code is often very difficult to do in a modular way. As a result, OpenCV has an entire architecture for handling just this problem. This architecture begins at the top with high-level operations like cv::blur(). This function takes your image, smoothes it, and gives it back. Sounds simple, but this is actually the result of many different layers operating together. When you want to do your own thing, you will have to figure out exactly which level you want to tinker with. Tinker at too high a level, and you won’t have the control you need. Tinker at too low a level, and you will find yourself reinventing every sort of wheel before you are done. To get the most out of OpenCV’s filter architecture, it is important to keep the big picture in mind. The big picture is best understood by working through an example. The “top” layer of the filter architecture is a collection of high-level service routines, like cv::blur() mentioned above. Inside of cv::blur(), however, your

with. Tinker at too high a level, and you won’t have the control you need. Tinker at too low a level, and you will find yourself reinventing every sort of wheel before you are done. To get the most out of OpenCV’s filter architecture, it is important to keep the big picture in mind. The big picture is best understood by working through an example. The “top” layer of the filter architecture is a collection of high-level service routines, like cv::blur() mentioned above. Inside of cv::blur(), however, your image is given to an object called a filter engine. The filter engine is a class that handles all of the general aspects of applying a filter kernel to your image, like iterating over the image, operating on each segment of the image, and handling boundary conditions. As the filter engine moves through the image, it will pass chunks of the image down to the next level that contains the actual filters. These filters, called base filters in OpenCV, do the actual work of computing individual destination image

image is given to an object called a filter engine. The filter engine is a class that handles all of the general aspects of applying a filter kernel to your image, like iterating over the image, operating on each segment of the image, and handling boundary conditions. As the filter engine moves through the image, it will pass chunks of the image down to the next level that contains the actual filters. These filters, called base filters in OpenCV, do the actual work of computing individual destination image pixels from small windows of the source image and the kernel. At the very bottom of this hierarchy are the kernel builders. These are routines that actually compute individual kernels from some specific set of parameters (i.e., the variance of the Gaussian used in the cv::blur() code). Filter Engines You can use the Filter Engine layer to implement your own basic filters. The filters you implement need not be linear filters as was the case with cv::filter2D(). For performance reasons, you may also want to

pixels from small windows of the source image and the kernel. At the very bottom of this hierarchy are the kernel builders. These are routines that actually compute individual kernels from some specific set of parameters (i.e., the variance of the Gaussian used in the cv::blur() code). Filter Engines You can use the Filter Engine layer to implement your own basic filters. The filters you implement need not be linear filters as was the case with cv::filter2D(). For performance reasons, you may also want to create composite filters, which—though they may accomplish something similar or identical to some sequence of filters provided for you by OpenCV—will run much faster. This is because combining the operations at a local level will mean that your data will stay in the processor cache, get handled, and get sent back to main memory, in just one transaction (as opposed to one for each filter you used in the composition). The Basics of How Filter Engines Work The basic objects that are used to represent filters

create composite filters, which—though they may accomplish something similar or identical to some sequence of filters provided for you by OpenCV—will run much faster. This is because combining the operations at a local level will mean that your data will stay in the processor cache, get handled, and get sent back to main memory, in just one transaction (as opposed to one for each filter you used in the composition). The Basics of How Filter Engines Work The basic objects that are used to represent filters in two dimensions are all derived from the cv::BaseFilter class. This class essentially defines the kernel operation. For linear kernels, this can

be thought of as the kernel weights. For nonlinear kernels, in principle, this could be just about any function that maps the weights in the kernel support area to a single number that is the result of the filter. The object responsible for applying a filter is a cv::FilterEngine. The cv::FilterEngine class is one of those “classes that do things.” When you want to use a cv::BaseFilter-derived kernel object, you have to create a cv::FilterEngine object to do the work, and pass it a pointer to your cv::BaseFilter object when you create it. The cv::FilterEngine Class The filter engine class is both a container for the base filters, as well as a device for actually applying the filter to an image. When constructed, the filter engine constructor will require either a two-dimensional filter (non-separable case) or two one-dimensional filters (separable case). In addition, you will have to tell it what kinds of images it will operate on, what kind it will return, and what data type to use for its intermediate

object when you create it. The cv::FilterEngine Class The filter engine class is both a container for the base filters, as well as a device for actually applying the filter to an image. When constructed, the filter engine constructor will require either a two-dimensional filter (non-separable case) or two one-dimensional filters (separable case). In addition, you will have to tell it what kinds of images it will operate on, what kind it will return, and what data type to use for its intermediate buffer (more on that in a moment). The way the borders are handled is also a property of the filter engine, as it is responsible for the action of the more atomic filter object it contains on the images provided to the engine. Here is the constructor for the cv::FilterEngine class: cv::FilterEngine(   const cv::Ptr< cv::BaseFilter>&       _filter2D,   const cv::Ptr< cv::BaseRowFilter>&    _rowFilter,   const cv::Ptr< cv::BaseColumnFilter>& _columnFilter,   int                                   srcType,    int

on that in a moment). The way the borders are handled is also a property of the filter engine, as it is responsible for the action of the more atomic filter object it contains on the images provided to the engine. Here is the constructor for the cv::FilterEngine class: cv::FilterEngine(   const cv::Ptr< cv::BaseFilter>&       _filter2D,   const cv::Ptr< cv::BaseRowFilter>&    _rowFilter,   const cv::Ptr< cv::BaseColumnFilter>& _columnFilter,   int                                   srcType,    int                                   dstType,    int                                   bufType,   int                                   _rowBorderType    = cv::BORDER_REPLICATE,   int                                   _columnBorderType = -1,   const Scalar&                         _borderValue      = cv::Scalar()  ); The first three arguments are used to pass the base filter to the filter engine. We will cover the details of how to make those shortly. What is important at the moment is that you will only use either the

dstType,    int                                   bufType,   int                                   _rowBorderType    = cv::BORDER_REPLICATE,   int                                   _columnBorderType = -1,   const Scalar&                         _borderValue      = cv::Scalar()  ); The first three arguments are used to pass the base filter to the filter engine. We will cover the details of how to make those shortly. What is important at the moment is that you will only use either the first option, _filter2D, or the next two, _rowFilter and _columnFilter. If you are using _filter2D, then you should pass cv::Mat() (empty matrix constructor) to _rowFilter and _columnFilter; conversely if you wish to use _rowFilter and _columnFilter, you should pass cv::Mat() to _filter2D. The next three arguments, srcType, dstType, and bufType should all be depth types (e.g., cv::F32 etc.) All data passed to the filter engine must be of type srcType. All final returned data will be of type dstType. The

first option, _filter2D, or the next two, _rowFilter and _columnFilter. If you are using _filter2D, then you should pass cv::Mat() (empty matrix constructor) to _rowFilter and _columnFilter; conversely if you wish to use _rowFilter and _columnFilter, you should pass cv::Mat() to _filter2D. The next three arguments, srcType, dstType, and bufType should all be depth types (e.g., cv::F32 etc.) All data passed to the filter engine must be of type srcType. All final returned data will be of type dstType. The buffer type bufType must be the same as the source type for non-separable filters. In the case of separable filters, the row filter will be processed first, and its output will be written to a buffer of type bufType. The column filter will then be applied on that buffer and the results written to the output area, which will be of the destination type. Thus the row filter must be one that takes arrays of type srcType and returns bufType, while the column filter must take arrays of type bufType and return

buffer type bufType must be the same as the source type for non-separable filters. In the case of separable filters, the row filter will be processed first, and its output will be written to a buffer of type bufType. The column filter will then be applied on that buffer and the results written to the output area, which will be of the destination type. Thus the row filter must be one that takes arrays of type srcType and returns bufType, while the column filter must take arrays of type bufType and return arrays of type dstType.  Row and column borders can be handled differently by the filter engine if you like, by passing different values to _rowBorderType and _columnBorderType. If you wish, you may pass -1 (the default value) to _columnBorderType to make it the same as the row border type. The border value _borderValue is only used for cv::BORDERTYPE_CONSTANT, in which case extrapolated pixels will have value _borderValue. You may create a filter engine with the constructor described above, or you may create

of type dstType.  Row and column borders can be handled differently by the filter engine if you like, by passing different values to _rowBorderType and _columnBorderType. If you wish, you may pass -1 (the default value) to _columnBorderType to make it the same as the row border type. The border value _borderValue is only used for cv::BORDERTYPE_CONSTANT, in which case extrapolated pixels will have value _borderValue. You may create a filter engine with the constructor described above, or you may create a filter engine using the default constructor cv::FilterEngine(), and then initialize the engine with its init() method: cv::FilterEngine::init(   const cv::Ptr< cv::BaseFilter>&       _filter2D,   const cv::Ptr< cv::BaseRowFilter>&    _rowFilter,   const cv::Ptr< cv::BaseColumnFilter>& _columnFilter,   int                                   srcType,

int                                   dstType,    int                                   bufType,   int                                   _rowBorderType    = cv::BORDER_REPLICATE,   int                                   _columnBorderType = -1,   const Scalar&                         _borderValue      = cv::Scalar()  ); Once you have the filter engine, there are several methods that allow you to use the engine to actually do a convolution. They are the methods called cv::FilterEngine::start(), cv::FilterEngine::proceed(), and cv::FilterEngine::apply(). The start() methods initializes convolution for processing pixels in a set region of the source image, though it does not actually process the pixels. The proceed() method does the processing of pixels in that section of the image after start() or a previous call to proceed(). The apply() method is essentially a higher-level call that processes the entire image at once. virtual int cv::FilterEngine::start(   // Return starting y-position in source image

and cv::FilterEngine::apply(). The start() methods initializes convolution for processing pixels in a set region of the source image, though it does not actually process the pixels. The proceed() method does the processing of pixels in that section of the image after start() or a previous call to proceed(). The apply() method is essentially a higher-level call that processes the entire image at once. virtual int cv::FilterEngine::start(   // Return starting y-position in source image   cv::Size        wholeSize,           // Size of entire source image   Rect            roi,                 // Region of interest inside of source image   int             maxBufRows = -1      // Leave this at -1, or see text below ); virtual int cv::FilterEngine::start(   // Return starting y-position in source image   const cv::Mat&  src,                 // Source image   const cv::Rect& srcRoi     = cv::Rect(0,0,-1,-1), // Region of interest in source    bool            isolated   = false,  // if true, use edge extrapolation

// Size of entire source image   Rect            roi,                 // Region of interest inside of source image   int             maxBufRows = -1      // Leave this at -1, or see text below ); virtual int cv::FilterEngine::start(   // Return starting y-position in source image   const cv::Mat&  src,                 // Source image   const cv::Rect& srcRoi     = cv::Rect(0,0,-1,-1), // Region of interest in source    bool            isolated   = false,  // if true, use edge extrapolation   int             maxBufRows = -1      // Leave this at -1, or see text below ); Both forms of cv::FilterEngine::start() support a region of interest, typically abbreviated ROI. This is the portion of the incoming source image that will be acted on by the filter engine. The first form of start() requires to be told both the size of the image that is going to be processed, called wholeSize, and the size and location of the ROI, which are together in the cv::Rect called roi. The argument maxBufRows should be left

maxBufRows = -1      // Leave this at -1, or see text below ); Both forms of cv::FilterEngine::start() support a region of interest, typically abbreviated ROI. This is the portion of the incoming source image that will be acted on by the filter engine. The first form of start() requires to be told both the size of the image that is going to be processed, called wholeSize, and the size and location of the ROI, which are together in the cv::Rect called roi. The argument maxBufRows should be left at its default value of -1.21 The second form is the one you will probably use most often; it takes the source image (and computes the equivalent of wholeSize from that), the region of interest, and a new argument called isolated. The parameter isolated deals with the way off-the-edge pixels are treated for an ROI that is smaller than the entire image. If isolated is true, those pixels will be extrapolated the same as would be off-the-edge pixels of the image as a whole. If isolated is false, however,

left at its default value of -1.21 The second form is the one you will probably use most often; it takes the source image (and computes the equivalent of wholeSize from that), the region of interest, and a new argument called isolated. The parameter isolated deals with the way off-the-edge pixels are treated for an ROI that is smaller than the entire image. If isolated is true, those pixels will be extrapolated the same as would be off-the-edge pixels of the image as a whole. If isolated is false, however, off-the-edge pixels will not be extrapolated. Instead they will be taken from the image outside of the ROI. cv::FilterEngine::start() returns an integer indicating the y-position in the image at which processing should begin (as computed from the ROI). Once you have called cv::FilterEngine::start(), you can begin processing your data with cv::FilterEngine::proceed().                                                              21 This parameter is the size of internal ring buffer used for filtering and

off-the-edge pixels will not be extrapolated. Instead they will be taken from the image outside of the ROI. cv::FilterEngine::start() returns an integer indicating the y-position in the image at which processing should begin (as computed from the ROI). Once you have called cv::FilterEngine::start(), you can begin processing your data with cv::FilterEngine::proceed().                                                              21 This parameter is the size of internal ring buffer used for filtering and affects only the processing speed. As maxBufRows increases from the minimal value (i.e., kernel_aperture_height), the speed will increase until, after a certain value, it will decrease because of the less efficient cache use. The measured difference in speed should be quite small, but you can play with it if filter performance is critical to your application. In the case of non-separable filters with large apertures however, keep in mind that the function cv::filter2D() checks the aperture size and, if it's

only the processing speed. As maxBufRows increases from the minimal value (i.e., kernel_aperture_height), the speed will increase until, after a certain value, it will decrease because of the less efficient cache use. The measured difference in speed should be quite small, but you can play with it if filter performance is critical to your application. In the case of non-separable filters with large apertures however, keep in mind that the function cv::filter2D() checks the aperture size and, if it's large enough (~11-by-11 or larger), it uses the fast DFT-based algorithm. This method filters each pixel in ~𝑂(𝑙𝑜𝑔  𝑛) time (i.e. time almost does not depend on the aperture size). By contrast, FilterEngine always uses the direct algorithm that takes  𝑂(𝑛!) operations for each pixel (in the case of an 𝑛-by-𝑛 filter). In other words, if you want the highest performance in linear non-separable filtering operations, where aperture size is sufficiently large, it's better to use cv::filter2D(), rather than to use

enough (~11-by-11 or larger), it uses the fast DFT-based algorithm. This method filters each pixel in ~𝑂(𝑙𝑜𝑔  𝑛) time (i.e. time almost does not depend on the aperture size). By contrast, FilterEngine always uses the direct algorithm that takes  𝑂(𝑛!) operations for each pixel (in the case of an 𝑛-by-𝑛 filter). In other words, if you want the highest performance in linear non-separable filtering operations, where aperture size is sufficiently large, it's better to use cv::filter2D(), rather than to use FilterEngine and try to tune maxBufRows. Thank Vadim Pisarevsky for this important insight.

virtual int cv::FilterEngine::proceed( // Returns the number of produced rows   const uchar* src,                    // Points into source image   int          srcStep,                // Source image step size   int          srcCount,               // Number of row steps to process   uchar*       dst,                    // Points into destination image   int          dstStep                 // Destination image step size ); When called, cv::FilterEngine::proceed() requires to be passed some detailed bookkeeping information. The first argument src is the location in memory of the current row being operated on. This means, for example, that the first time you call cv::FilterEngine::proceed(), you will need to compute src from the y-value returned to you by cv::FilterEngine::start(). You will also have to pass the step size from the source image as srcStep. The argument srcCount is the number or row steps you would like to process in this call to cv::FilterEngine::proceed(). Finally, you must give the data

information. The first argument src is the location in memory of the current row being operated on. This means, for example, that the first time you call cv::FilterEngine::proceed(), you will need to compute src from the y-value returned to you by cv::FilterEngine::start(). You will also have to pass the step size from the source image as srcStep. The argument srcCount is the number or row steps you would like to process in this call to cv::FilterEngine::proceed(). Finally, you must give the data pointer and step size for the area to which the results of the filter will be written. These two arguments are dst and dstStep, respectively. The way in which both cv::FilterEngine::start() and cv::FilterEngine::proceed() are used is best understood by example. Below is an example from the OpenCV source code, which is a (slightly simplified) implementation of the Laplace operator: Example 5-4: An example implementation of a Laplace operator using the cv::FilterEngine() class void laplace_f(   const cv::Mat& src,

step size for the area to which the results of the filter will be written. These two arguments are dst and dstStep, respectively. The way in which both cv::FilterEngine::start() and cv::FilterEngine::proceed() are used is best understood by example. Below is an example from the OpenCV source code, which is a (slightly simplified) implementation of the Laplace operator: Example 5-4: An example implementation of a Laplace operator using the cv::FilterEngine() class void laplace_f(   const cv::Mat& src,                  // Input image   cv::Mat&       dst                   // Result image ) {   CV_Assert( src.type() == cv::F32 );   dst.create( src.size(), src.type() );    // get the derivative and smooth kernels for d2I/dx2   // for d2I/dy2 use the same kernels, just swapped   cv::Mat kd, ks;   cv::getSobelKernels( kd, ks, 2, 0, ksize, false, ktype );    // process 10 source rows at once   int DELTA = std::min(10, src.rows);   cv::Ptr< cv::FilterEngine> Fxx = cv::createSeparableLinearFilter(     src.type(),

// Input image   cv::Mat&       dst                   // Result image ) {   CV_Assert( src.type() == cv::F32 );   dst.create( src.size(), src.type() );    // get the derivative and smooth kernels for d2I/dx2   // for d2I/dy2 use the same kernels, just swapped   cv::Mat kd, ks;   cv::getSobelKernels( kd, ks, 2, 0, ksize, false, ktype );    // process 10 source rows at once   int DELTA = std::min(10, src.rows);   cv::Ptr< cv::FilterEngine> Fxx = cv::createSeparableLinearFilter(     src.type(),     dst.type(),      kd, ks,      Point(-1,-1),      0,      borderType, borderType,      Scalar()    );   cv::Ptr< cv::FilterEngine> Fyy = cv::createSeparableLinearFilter(     src.type(),     dst.type(),      ks, kd,      Point(-1,-1),      0,      borderType, borderType,      Scalar()    );    int y = Fxx->start(src), dsty = 0, dy = 0;   Fyy->start(src);   const uchar* sptr = src.data + y*src.step;    // allocate the buffers for the spatial image derivatives;   // the buffers need to have more than DELTA

dst.type(),      kd, ks,      Point(-1,-1),      0,      borderType, borderType,      Scalar()    );   cv::Ptr< cv::FilterEngine> Fyy = cv::createSeparableLinearFilter(     src.type(),     dst.type(),      ks, kd,      Point(-1,-1),      0,      borderType, borderType,      Scalar()    );    int y = Fxx->start(src), dsty = 0, dy = 0;   Fyy->start(src);   const uchar* sptr = src.data + y*src.step;    // allocate the buffers for the spatial image derivatives;   // the buffers need to have more than DELTA rows, because at the   // last iteration the output may take max(kd.rows-1,ks.rows-1)

// rows more than the input.   cv::Mat Ixx( DELTA + kd.rows - 1, src.cols, dst.type() );   cv::Mat Iyy( DELTA + kd.rows - 1, src.cols, dst.type() );    // inside the loop always pass DELTA rows to the filter   // (note that the "proceed" method takes care of possibe overflow, since   // it was given the actual image height in the "start" method)   // on output you can get:   //  * < DELTA rows (initial buffer accumulation stage)   //  * = DELTA rows (settled state in the middle)   //  * > DELTA rows (when the input image is over, generate   //                  "virtual" rows using the border mode and filter them)   // this variable number of output rows is dy   // dsty is the current output row   // sptr is the pointer to the first input row in the portion to process   //   for( ; dsty < dst.rows; sptr += DELTA*src.step, dsty += dy )   {     Fxx->proceed(        sptr,        (int) src.step,        DELTA,       Ixx.data,        (int) Ixx.step      );     dy = Fyy->proceed(        sptr,        (int)

(when the input image is over, generate   //                  "virtual" rows using the border mode and filter them)   // this variable number of output rows is dy   // dsty is the current output row   // sptr is the pointer to the first input row in the portion to process   //   for( ; dsty < dst.rows; sptr += DELTA*src.step, dsty += dy )   {     Fxx->proceed(        sptr,        (int) src.step,        DELTA,       Ixx.data,        (int) Ixx.step      );     dy = Fyy->proceed(        sptr,        (int) src.step,        DELTA,        Iyy.data,        (int) Iyy.step      );     if( dy > 0 )     {       Mat dstripe = dst.rowRange( dsty, dsty + dy );       add( Ixx.rowRange(0, dy), Iyy.rowRange(0, dy), dstripe );     }   } } We will cover shortly how functions like cv::createSeparableLinearFilter() work, but what is important here is that these functions just return instances of classes derived from the cv::FilterEngine base class. Also, notice how the start() and proceed() methods are used. As you can see, the

DELTA,        Iyy.data,        (int) Iyy.step      );     if( dy > 0 )     {       Mat dstripe = dst.rowRange( dsty, dsty + dy );       add( Ixx.rowRange(0, dy), Iyy.rowRange(0, dy), dstripe );     }   } } We will cover shortly how functions like cv::createSeparableLinearFilter() work, but what is important here is that these functions just return instances of classes derived from the cv::FilterEngine base class. Also, notice how the start() and proceed() methods are used. As you can see, the implementation laplace_f() loops through the incoming image in chunks of size DELTA and processes each chunk before moving on. Now that you know how to use start() and proceed(), you might wonder why it is worth the effort. In fact, the next method we will look at called cv::FilterEngine::apply() just does the whole thing in one go.22 The reason is performance. If you have several things you want to do, like applying the two kernels (or more!) in sequence in our example, it is much better to do everything you

laplace_f() loops through the incoming image in chunks of size DELTA and processes each chunk before moving on. Now that you know how to use start() and proceed(), you might wonder why it is worth the effort. In fact, the next method we will look at called cv::FilterEngine::apply() just does the whole thing in one go.22 The reason is performance. If you have several things you want to do, like applying the two kernels (or more!) in sequence in our example, it is much better to do everything you want to do to a bit of data and then write it out to the destination image. In this way, you can keep all of the working data in processor cache and your filter will run much faster.  If this kind of performance detail is not critical to your application, you can simply tell a filter engine to just go through an entire image and output the result. This is done with the cv::FilterEngine::apply() method.                                                              22 One additional consequence of this is that when

do to a bit of data and then write it out to the destination image. In this way, you can keep all of the working data in processor cache and your filter will run much faster.  If this kind of performance detail is not critical to your application, you can simply tell a filter engine to just go through an entire image and output the result. This is done with the cv::FilterEngine::apply() method.                                                              22 One additional consequence of this is that when apply() calls start(), it always passes maxBufRows the default value of -1.

virtual void apply(   const cv::Mat&  src,                 // Input image   cv::Mat&        dst                  // Result image   const cv::Rect& srcRoi   = cv::Rect(0,0,-1,-1), // Region of interest to he processed   cv::Point       dstOffs  = cv::Point(0,0),      // Offset to write to in destination image   bool            isolated   = false,             // if true, use edge extrapolation ); Internally, cv::FilterEngine::apply() just wraps up a call to cv::FilterEngine::start() and then one big call to cv::FilterEngine::proceed(), which then goes through the entire image. The only new detail is the argument dstOffs, which is the destination offset. The destination offset is a point that indicates where in the destination image to write the result of the processed ROI. If left at its default value of cv::Point(0,0), then the result data will be written into the upper-left corner of the destination image. Otherwise, it will be offset appropriately. This is particularly useful when you wish to process some

cv::FilterEngine::proceed(), which then goes through the entire image. The only new detail is the argument dstOffs, which is the destination offset. The destination offset is a point that indicates where in the destination image to write the result of the processed ROI. If left at its default value of cv::Point(0,0), then the result data will be written into the upper-left corner of the destination image. Otherwise, it will be offset appropriately. This is particularly useful when you wish to process some ROI in the source image and write the results to the corresponding location in the destination image. Filter Engine Builders OpenCV contains built in functions that will create filter engines for you. It is worth taking a moment to appreciate what this really means. Creating a filter engine means both creating the filters that the engine will apply (i.e., a box filter of a Gaussian filter, etc.), as well as actually implementing the start(), proceed(), and apply() methods of that filter for you. Each of the

in the source image and write the results to the corresponding location in the destination image. Filter Engine Builders OpenCV contains built in functions that will create filter engines for you. It is worth taking a moment to appreciate what this really means. Creating a filter engine means both creating the filters that the engine will apply (i.e., a box filter of a Gaussian filter, etc.), as well as actually implementing the start(), proceed(), and apply() methods of that filter for you. Each of the library routines we will discuss in this section creates an object of a different type, but all of these objects are derived from cv::FilterEngine, and so inherit the interface we discussed in the previous section, as well as provide the implementation of that interface. For each of these cv::FilterEngine factories, there is also a corresponding higher-level routine we have already encountered, which (unbeknownst to you) creates the appropriate engine, applies it to your image, and cleans it up, returning you

routines we will discuss in this section creates an object of a different type, but all of these objects are derived from cv::FilterEngine, and so inherit the interface we discussed in the previous section, as well as provide the implementation of that interface. For each of these cv::FilterEngine factories, there is also a corresponding higher-level routine we have already encountered, which (unbeknownst to you) creates the appropriate engine, applies it to your image, and cleans it up, returning you only the result of that computation. For example, cv::boxFilter() is really just a higher-level wrapper around cv::createBoxFilter() and the subsequent application of the created filter to your image.23 You will notice that all of these routines return cv::Ptr<cv::FilterEngine>. It is probably worth taking a moment to appreciate what this means. Notice first that this is a good example of where OpenCV uses its smart pointer construct cv::Ptr<>. This way you will not have to deallocate the filter engine you

only the result of that computation. For example, cv::boxFilter() is really just a higher-level wrapper around cv::createBoxFilter() and the subsequent application of the created filter to your image.23 You will notice that all of these routines return cv::Ptr<cv::FilterEngine>. It is probably worth taking a moment to appreciate what this means. Notice first that this is a good example of where OpenCV uses its smart pointer construct cv::Ptr<>. This way you will not have to deallocate the filter engine you create. You can pass the smart pointer around, do what you need to with it, and trust that at the end of whatever you are doing, when there are no more references to the engine around, the engine itself will be deallocated. It is also worth noting that smart pointer, just like the traditional pointer, can be a base class pointer, and can be used to operate on derived class objects. Thus, just because all of these filter builders return a (smart) pointer of type cv::FilterEngine, it is not correct to

you create. You can pass the smart pointer around, do what you need to with it, and trust that at the end of whatever you are doing, when there are no more references to the engine around, the engine itself will be deallocated. It is also worth noting that smart pointer, just like the traditional pointer, can be a base class pointer, and can be used to operate on derived class objects. Thus, just because all of these filter builders return a (smart) pointer of type cv::FilterEngine, it is not correct to conclude that they return objects of type cv::FilterEngine. Exactly on the contrary, these functions all return their own specific filter engine types that are all derived from cv::FilterEngine. cv::createBoxFilter() You can directly create a box filter using cv::createBoxFilter(): cv::Ptr<cv::FilterEngine> cv::createBoxFilter(   int       srcType,                   // Source image type (e.g., cv::U8)    int       dstType,                   // Destination image type (e.g., cv::U8)   cv::Size  ksize,

that they return objects of type cv::FilterEngine. Exactly on the contrary, these functions all return their own specific filter engine types that are all derived from cv::FilterEngine. cv::createBoxFilter() You can directly create a box filter using cv::createBoxFilter(): cv::Ptr<cv::FilterEngine> cv::createBoxFilter(   int       srcType,                   // Source image type (e.g., cv::U8)    int       dstType,                   // Destination image type (e.g., cv::U8)   cv::Size  ksize,                     // Kernel size      cv::Point anchor     = cv::Point(-1,-1),  // Location of anchor point   bool      normalize  = true,              // If true, divide by box area                                                              23 Actually, cv::boxFilter() does not literally call cv::createBoxFilter() explicitly, but it essentially implements the latter’s functionality internally before calling the further subroutines shared by both routines.

int       borderType = cv::BORDER_DEFAULT // Border extrapolation to use ); As you might have expected, the arguments to cv::createBoxFilter() are essentially the same as those for cv::boxFilter(), with the exception that instead of providing the source and destination images, only the types of those images are required to define the filter. The ksize and anchor arguments define the size of the kernel and the (optional) anchor point as with cv::boxFilter, and the filter can be normalized or not by setting the normalize argument to true or false. The borderType argument has the usual effect. cv::createDerivFilter() You can create a derivative filter using cv::createBoxFilter(): cv::Ptr<cv::FilterEngine> cv::createDerivFilter(   int srcType,                         // Source image type (e.g., cv::U8)    int dstType,                         // Destination image type (e.g., cv::U8)    int dx,                              // order of corresponding derivative in x   int dy,                              // order

the normalize argument to true or false. The borderType argument has the usual effect. cv::createDerivFilter() You can create a derivative filter using cv::createBoxFilter(): cv::Ptr<cv::FilterEngine> cv::createDerivFilter(   int srcType,                         // Source image type (e.g., cv::U8)    int dstType,                         // Destination image type (e.g., cv::U8)    int dx,                              // order of corresponding derivative in x   int dy,                              // order of corresponding derivative in y   int ksize,                           // Kernel size      int borderType = cv::BORDER_DEFAULT  // Border extrapolation to use );  The arguments to cv::createDerivFilter() are essentially the same as those for cv::Sobel(), with the exception that instead of providing the source and destination images, only the types of those images are required to define the filter. The dx and dy arguments define the order of the derivative to be approximated, and ksize gives the size of the

corresponding derivative in y   int ksize,                           // Kernel size      int borderType = cv::BORDER_DEFAULT  // Border extrapolation to use );  The arguments to cv::createDerivFilter() are essentially the same as those for cv::Sobel(), with the exception that instead of providing the source and destination images, only the types of those images are required to define the filter. The dx and dy arguments define the order of the derivative to be approximated, and ksize gives the size of the kernel to be used24. The borderType argument has the usual effect. cv::createGaussianFilter() You can directly create a Gaussian filter using cv::createGaussianFilter(): cv::Ptr<cv::FilterEngine> cv::createGaussianFilter(   int      type,                       // Source and destination type (e.g., cv::U8)   cv::Size ksize,                      // Kernel size      double   sigmaX,                     // Gaussian half-width in x-direction   double   sigmaY      = 0.0,          // Gaussian half-width in

the kernel to be used24. The borderType argument has the usual effect. cv::createGaussianFilter() You can directly create a Gaussian filter using cv::createGaussianFilter(): cv::Ptr<cv::FilterEngine> cv::createGaussianFilter(   int      type,                       // Source and destination type (e.g., cv::U8)   cv::Size ksize,                      // Kernel size      double   sigmaX,                     // Gaussian half-width in x-direction   double   sigmaY      = 0.0,          // Gaussian half-width in y-direction   int      borderType = cv::BORDER_DEFAULT // Border extrapolation to use ); As you might expect by now, the arguments to cv::createGaussianFilter() are essentially the same as those for cv::GaussianBlur(). In this case, however, there is only one type argument for the source and destination; this filter requires both to be of the same type25. sigmaX and sigmaY indicate the variance of the Gaussian in the 𝑥- and 𝑦-direction respectively, with sigmaY being optional (if set to zero sigmaY will be

y-direction   int      borderType = cv::BORDER_DEFAULT // Border extrapolation to use ); As you might expect by now, the arguments to cv::createGaussianFilter() are essentially the same as those for cv::GaussianBlur(). In this case, however, there is only one type argument for the source and destination; this filter requires both to be of the same type25. sigmaX and sigmaY indicate the variance of the Gaussian in the 𝑥- and 𝑦-direction respectively, with sigmaY being optional (if set to zero sigmaY will be equal to sigmaX). The borderType argument has the usual effect. cv::createLinearFilter() You can directly create a general linear filter using cv::createLinearFilter(): cv::Ptr<cv::FilterEngine> cv::createLinearFilter(   int               srcType,           // Source image type (e.g., cv::U8)    int               dstType,           // Destination image type (e.g., cv::U8)    cv::InputArray    kernel,            // Your own kernel                                                              24 Note that the

equal to sigmaX). The borderType argument has the usual effect. cv::createLinearFilter() You can directly create a general linear filter using cv::createLinearFilter(): cv::Ptr<cv::FilterEngine> cv::createLinearFilter(   int               srcType,           // Source image type (e.g., cv::U8)    int               dstType,           // Destination image type (e.g., cv::U8)    cv::InputArray    kernel,            // Your own kernel                                                              24 Note that the ksize argument for cv::createDerivFilter() is an integer, while the ksize argument for cv::createBoxFilter() was of type cv::Size. This is natural because the kernel of a derivative filter will always be square. 25 If you need to work around this, you can call cv::getGaussianKernel() directly, and then cv::createSeparableLinearFilter(). (We will get to both of those functions shortly.)

cv::Point         anchor     = cv::Point(-1,-1),  // Location of anchor point   double            delta = 0,                      // Offset before assignment to dst    int               rowBorderType    = cv::BORDER_DEFAULT, // Row extrapolation    int               columnBorderType = -1,                 // Column extrapolation    const cv::Scalar& borderValue      = cv::Scalar()        // Value for constant borders ); If you have your own linear kernel, you can have OpenCV make it into a filter engine for you with cv::createLinearFilter(); this is the filter engine analog of cv::filter2D(). This function should only be used when you have a non-separable kernel, as you will get much better performance from cv::createSeparableLinearFilter() if your kernel is separable. The arguments are the source type, srcType, and destination type, dstType, for the input and output images, the kernel you want to use, an optional anchor point (the default is to use the center of your kernel), an optional offset delta, and

you with cv::createLinearFilter(); this is the filter engine analog of cv::filter2D(). This function should only be used when you have a non-separable kernel, as you will get much better performance from cv::createSeparableLinearFilter() if your kernel is separable. The arguments are the source type, srcType, and destination type, dstType, for the input and output images, the kernel you want to use, an optional anchor point (the default is to use the center of your kernel), an optional offset delta, and the border extrapolation parameters. Unlike most of the other similar functions, cv::createLinearFilter() allows you to specify the border extrapolation method for the vertical, rowBorderType, and the horizontal, columnBorderType, separately. If you omit the latter (or set it to -1) it will default to being the same as the former. The final borderValue argument is used to define the off-the-edge pixel values when either rowBorderType or columnBorderType is set to cv:: BORDER_CONSTANT.

offset delta, and the border extrapolation parameters. Unlike most of the other similar functions, cv::createLinearFilter() allows you to specify the border extrapolation method for the vertical, rowBorderType, and the horizontal, columnBorderType, separately. If you omit the latter (or set it to -1) it will default to being the same as the former. The final borderValue argument is used to define the off-the-edge pixel values when either rowBorderType or columnBorderType is set to cv:: BORDER_CONSTANT. cv::createMorphologyFilter() You can create a filter which implements the morphological operations using cv::createMorphologyFilter(): cv::Ptr<cv::FilterEngine> cv::createMorphologyFilter(   int               op,    int               type,    cv::InputArray    element,    cv::Point         anchor           = cv::Point(-1,-1),   // Location of anchor point   int               rowBorderType    = cv::BORDER_DEFAULT, // Row extrapolation    int               columnBorderType = -1,                 // Column

cv::createMorphologyFilter() You can create a filter which implements the morphological operations using cv::createMorphologyFilter(): cv::Ptr<cv::FilterEngine> cv::createMorphologyFilter(   int               op,    int               type,    cv::InputArray    element,    cv::Point         anchor           = cv::Point(-1,-1),   // Location of anchor point   int               rowBorderType    = cv::BORDER_DEFAULT, // Row extrapolation    int               columnBorderType = -1,                 // Column extrapolation    const cv::Scalar& borderValue      = cv::Scalar()        // Value for constant borders ) By now you know the pattern. The arguments to cv::createMorphologyFilter() are essentially the same as those for cv::morphologyEx(). The op argument must be one of the operations from Table 5-3, and the type is the type of the source and destination images, which must be the same. The element is your structuring element and the optional anchor lets you locate the anchor point somewhere other than the

extrapolation    const cv::Scalar& borderValue      = cv::Scalar()        // Value for constant borders ) By now you know the pattern. The arguments to cv::createMorphologyFilter() are essentially the same as those for cv::morphologyEx(). The op argument must be one of the operations from Table 5-3, and the type is the type of the source and destination images, which must be the same. The element is your structuring element and the optional anchor lets you locate the anchor point somewhere other than the (default) center of your structuring element. rowBorderType and columBorderType let you specify the vertical and horizontal border extrapolation methods, and borderValue is the constant used in the case that one of the extrapolation methods is cv::BORDER_CONSTANT. cv::createSeparableLinearFilter() In the case of a separable linear kernel, you can directly create a filter using cv::createSeparableLinearFilter(): cv::Ptr<cv::FilterEngine> cv::createSeparableLinearFilter(   int srcType,

center of your structuring element. rowBorderType and columBorderType let you specify the vertical and horizontal border extrapolation methods, and borderValue is the constant used in the case that one of the extrapolation methods is cv::BORDER_CONSTANT. cv::createSeparableLinearFilter() In the case of a separable linear kernel, you can directly create a filter using cv::createSeparableLinearFilter(): cv::Ptr<cv::FilterEngine> cv::createSeparableLinearFilter(   int srcType,                         // Source image type (e.g., cv::U8)    int dstType,                         // Destination image type (e.g., cv::U8)    cv::InputArray    rowKernel,         // 1-by-N row Kernel   cv::InputArray    columnKernel,      // M-by-1 column kernel   cv::Point         anchor           = cv::Point(-1,-1),   // Location of anchor point   double            delta            = 0,                  // Offset before dst   int               rowBorderType    = cv::BORDER_DEFAULT, // Row extrapolation    int

// Source image type (e.g., cv::U8)    int dstType,                         // Destination image type (e.g., cv::U8)    cv::InputArray    rowKernel,         // 1-by-N row Kernel   cv::InputArray    columnKernel,      // M-by-1 column kernel   cv::Point         anchor           = cv::Point(-1,-1),   // Location of anchor point   double            delta            = 0,                  // Offset before dst   int               rowBorderType    = cv::BORDER_DEFAULT, // Row extrapolation    int               columnBorderType = -1,                 // Column extrapolation    const cv::Scalar& borderValue      = cv::Scalar()        // Value for constant borders ); The cv::createSeparableLinearFilter() function is the filter engine analog of the cv::sepFilter2D() function. It requires the source type, srcType, and the destination type,

dstType. The row and column kernels are each specified separately with rowKernel and columnKernel. anchor, delta, and the border extrapolation arguments all have their usual meanings. Base Filters The filter builders we looked at in the last section created the low-level filter object you needed for you, and then wrapped it up in a cv::FilterEngine object that you could then use. Of course, if you were always going to rely on these functions, it would not have been necessary to learn so much about how the filter engine itself actually works or gets constructed.  The real reason to use this is when you are going to create your own filter entirely, and then use that to build a filter engine that does your own special thing. In this section, we will look at how you create a low-level filter object of your own. Because separability plays such an important role in the performance of filters, and because you would not be making your own base-level filters if you were not concerned about performance, the base

the filter engine itself actually works or gets constructed.  The real reason to use this is when you are going to create your own filter entirely, and then use that to build a filter engine that does your own special thing. In this section, we will look at how you create a low-level filter object of your own. Because separability plays such an important role in the performance of filters, and because you would not be making your own base-level filters if you were not concerned about performance, the base filter classes are divided into the non-separable cv::BaseFilter class, and the two classes used for separable filters, cv::BaseRowFilter and cv::BaseColumnFilter. The cv::BaseFilter Class Down at the bottom of the image filtering class hierarchy lives cv::BaseFilter. It implements the actual operator that combines some kernel with the appropriate window of an input image to compute the corresponding output image pixel. This mapping need not be linear at all, and can be any general function of the input

filter classes are divided into the non-separable cv::BaseFilter class, and the two classes used for separable filters, cv::BaseRowFilter and cv::BaseColumnFilter. The cv::BaseFilter Class Down at the bottom of the image filtering class hierarchy lives cv::BaseFilter. It implements the actual operator that combines some kernel with the appropriate window of an input image to compute the corresponding output image pixel. This mapping need not be linear at all, and can be any general function of the input image pixels and the kernel pixels. The only restriction is that this function must be computable on a row-by-row basis26.  The definition of the cv::BaseFilter class looks like the following: class cv::BaseFilter {  public:    virtual ~BaseFilter();    // To be overriden by the user.   //   // runs a filtering operation on the set of rows,   // "dstcount + ksize.height - 1" rows on input,   // "dstcount" rows on output,   // each input row has "(width + ksize.width-1)*cn" elements   // each output row has

image pixels and the kernel pixels. The only restriction is that this function must be computable on a row-by-row basis26.  The definition of the cv::BaseFilter class looks like the following: class cv::BaseFilter {  public:    virtual ~BaseFilter();    // To be overriden by the user.   //   // runs a filtering operation on the set of rows,   // "dstcount + ksize.height - 1" rows on input,   // "dstcount" rows on output,   // each input row has "(width + ksize.width-1)*cn" elements   // each output row has "width*cn" elements.   // the filtered rows are written into "dst" buffer.   //   virtual void operator() (     const uchar** src,      uchar* dst,      int dststep,     int dstcount,      int width,      int cn   ) = 0;    // resets the filter state (may be needed for IIR filters)   //                                                              26 This means that the function 𝐹, which maps the pixels 𝑥!,! of the image and the values {𝑘!,!} of the kernel may contain terms which are functions of multiple

"width*cn" elements.   // the filtered rows are written into "dst" buffer.   //   virtual void operator() (     const uchar** src,      uchar* dst,      int dststep,     int dstcount,      int width,      int cn   ) = 0;    // resets the filter state (may be needed for IIR filters)   //                                                              26 This means that the function 𝐹, which maps the pixels 𝑥!,! of the image and the values {𝑘!,!} of the kernel may contain terms which are functions of multiple values of 𝑗—the column index—but may not contain terms which are functions of multiple values of 𝑖—the row index.

virtual void reset();    Size ksize;   Point anchor; }; As you can see, the method operator() is declared pure virtual, and of course this means that you cannot (directly) instantiate objects of type cv::BaseFilter. This should not surprise you, however; the intent is for you to be able to derive your own base filter objects off of cv::BaseFilter, and to implement the operator() and reset() functions yourself as appropriate for your own needs. The cv::BaseFilter::operator() is called by the filter engine (into which you have installed your base filter), so the filter engine is going to supply all of the arguments. The src and dst pointers will point to the source and destination data respectively, but they will point into those images at the proper place. The src pointer actually points to an array of pointers, each of which points to one of the rows that you will need to process the kernel (given by the ksize member variable). Similarly, the dst pointer points to the row in which the destination pixels

(into which you have installed your base filter), so the filter engine is going to supply all of the arguments. The src and dst pointers will point to the source and destination data respectively, but they will point into those images at the proper place. The src pointer actually points to an array of pointers, each of which points to one of the rows that you will need to process the kernel (given by the ksize member variable). Similarly, the dst pointer points to the row in which the destination pixels are located. At the time when the call to cv::BaseFilter::operator() is made, these pointers will be “aligned,” in the sense that the dst points to the first pixel you will be computing, and the pointers in src point to the first pixels you will need for the kernel you have. All of this was set up for you by the filter engine. At this level, you do not need to worry about boundary conditions either, as the src pointers actually point to a place where the necessary boundary values are already computed. dststep

located. At the time when the call to cv::BaseFilter::operator() is made, these pointers will be “aligned,” in the sense that the dst points to the first pixel you will be computing, and the pointers in src point to the first pixels you will need for the kernel you have. All of this was set up for you by the filter engine. At this level, you do not need to worry about boundary conditions either, as the src pointers actually point to a place where the necessary boundary values are already computed. dststep is the step size you will need in order to increment the dst pointer to compute other destination values, while dstcount is the total number of destination values you will need to compute. width and cn are the width of the image rows to be processed and the number of channels respectively. The cv::BaseFilter::reset() method is only used in cases where the filter maintains an internal state. The method is automatically called before any new image is processed27.  Once you have created your base filter, you

is the step size you will need in order to increment the dst pointer to compute other destination values, while dstcount is the total number of destination values you will need to compute. width and cn are the width of the image rows to be processed and the number of channels respectively. The cv::BaseFilter::reset() method is only used in cases where the filter maintains an internal state. The method is automatically called before any new image is processed27.  Once you have created your base filter, you can construct a filter engine to run it with the cv::FilterEngine() constructor and you are ready for business. The cv::BaseRowFilter and cv::BaseColumnFilter Classes As we saw when we looked at cv::FilterEngine, it can be given either a single non-separable base filter, or a pair of base filters corresponding to the row and column portions of a separable filter. The cv::BaseRowFilter and cv::BaseColumnFilter classes are how we implement those components of separable filters. Here are the class definitions

can construct a filter engine to run it with the cv::FilterEngine() constructor and you are ready for business. The cv::BaseRowFilter and cv::BaseColumnFilter Classes As we saw when we looked at cv::FilterEngine, it can be given either a single non-separable base filter, or a pair of base filters corresponding to the row and column portions of a separable filter. The cv::BaseRowFilter and cv::BaseColumnFilter classes are how we implement those components of separable filters. Here are the class definitions for these two (virtual base) classes. class cv::BaseRowFilter {  public:   virtual ~BaseRowFilter();    // To be overriden by the user.   //   // runs filtering operation on the single input row   // of "width" element, each element is has "cn" channels.   // the filtered row is written into "dst" buffer.   //   virtual void operator()(     const uchar* src,      uchar* dst,     int width,      int cn   ) = 0;                                                              27 As an example of where the reset

for these two (virtual base) classes. class cv::BaseRowFilter {  public:   virtual ~BaseRowFilter();    // To be overriden by the user.   //   // runs filtering operation on the single input row   // of "width" element, each element is has "cn" channels.   // the filtered row is written into "dst" buffer.   //   virtual void operator()(     const uchar* src,      uchar* dst,     int width,      int cn   ) = 0;                                                              27 As an example of where the reset method might be used, the current implementation of the box filter uses a sliding sum buffer, which needs to be cleared before each new call.

int ksize, anchor; };  class cv::BaseColumnFilter {  public:   virtual ~BaseColumnFilter();    // To be overriden by the user.   //   // runs a filtering operation on the set of rows,   // "dstcount + ksize - 1" rows on input,   // "dstcount" rows on output,   // each input and output row has "width" elements   // the filtered rows are written into "dst" buffer.   //   virtual void operator()(     const uchar** src,      uchar* dst,      int dststep,     int dstcount,      int width   ) = 0;    // resets the filter state (may be needed for IIR filters)   //   virtual void reset();    int ksize, anchor; }; The cv::BaseRowFilter class is the simplest because row data is sequential in memory, and so there is a little less bookkeeping. The associated cv::BaseRowFilter::operator() inherits this relative simplicity. The arguments are just the pointers to the source and destination rows, the width of the rows, and the number of channels, cn. As with cv::BaseFilter::operator(), the source and destination

// resets the filter state (may be needed for IIR filters)   //   virtual void reset();    int ksize, anchor; }; The cv::BaseRowFilter class is the simplest because row data is sequential in memory, and so there is a little less bookkeeping. The associated cv::BaseRowFilter::operator() inherits this relative simplicity. The arguments are just the pointers to the source and destination rows, the width of the rows, and the number of channels, cn. As with cv::BaseFilter::operator(), the source and destination pointers are of type uchar*, and so it will be your responsibility to cast them to the correct type.  The cv::BaseColumnFilter class needs to handle the non-sequential nature of the required data. As a result, there are the additional arguments dststep and dstcount, which the row filter did not require28. These have the same meaning as they did for the cv::BaseFilter() class. Base Filter Builders There will be times when you want to build your own filter engine, but will want to use existing filters to

pointers are of type uchar*, and so it will be your responsibility to cast them to the correct type.  The cv::BaseColumnFilter class needs to handle the non-sequential nature of the required data. As a result, there are the additional arguments dststep and dstcount, which the row filter did not require28. These have the same meaning as they did for the cv::BaseFilter() class. Base Filter Builders There will be times when you want to build your own filter engine, but will want to use existing filters to build it. In these cases, it is not necessary to go and re-implement the low-level filters you want. Instead, you can ask OpenCV to just give you the base filter object it normally uses for a particular task.  getLinearFilter(), getLinearRowFilter(), and getLinearColumnFilter() cv::Ptr<cv::BaseFilter> cv::getLinearFilter( // Return filter object   int            srcType,              // Source image type (e.g., cv::U8)    int            dstType,              // Destination image type (e.g., cv::U8)

to build it. In these cases, it is not necessary to go and re-implement the low-level filters you want. Instead, you can ask OpenCV to just give you the base filter object it normally uses for a particular task.  getLinearFilter(), getLinearRowFilter(), and getLinearColumnFilter() cv::Ptr<cv::BaseFilter> cv::getLinearFilter( // Return filter object   int            srcType,              // Source image type (e.g., cv::U8)    int            dstType,              // Destination image type (e.g., cv::U8)    cv::InputArray kernel,               // Your own kernel   cv::Point      anchor = cv::Point(-1,-1),  // Location of anchor point   double         delta  = 0,                 // Offset before assignment to dst    int            bits   = 0                  // For fixed precision integer kernels                                                              28 You will also notice that the number of channels is not present. This is because it is, in effect, now absorbed into dststep.

);  cv::Ptr<cv::BaseColumnFilter> cv::getLinearColumnFilter( // Return filter object    int            bufType,              // Buffer image type (e.g., cv::U8)   int            dstType,              // Destination image type (e.g., cv::U8)    cv::InputArray columnKernel,         // M-by-1 column kernel   int            anchor,               // 1-d Location of anchor point   int            symmetryType,         // See Table 5-5   double         delta  = 0,           // Offset before assignment to dst    int            bits   = 0            // For fixed precision integer kernels );  cv::Ptr<cv::BaseRowFilter> cv::getLinearRowFilter(      // Return filter object   int            srcType,              // Source image type (e.g., cv::U8)    int            bufType,              // Buffer image type (e.g., cv::U8)    cv::InputArray rowKernel,            // 1-by-N row Kernel   int            anchor,               // 1-d Location of anchor point   int            symmetryType,         // See Table 5-5 ; The function

bits   = 0            // For fixed precision integer kernels );  cv::Ptr<cv::BaseRowFilter> cv::getLinearRowFilter(      // Return filter object   int            srcType,              // Source image type (e.g., cv::U8)    int            bufType,              // Buffer image type (e.g., cv::U8)    cv::InputArray rowKernel,            // 1-by-N row Kernel   int            anchor,               // 1-d Location of anchor point   int            symmetryType,         // See Table 5-5 ; The function cv::getLinearFilter() returns a (smart) pointer to a cv::BaseFilter-derived object that implements a general two-dimensional non-separable linear filter based on the kernel (and anchor point) you provide. You can provide an optional output offset delta, which will be applied after your kernel and before the result is written to the destination array. The last argument, bits, is used when your kernel is an integer matrix, but you want to use it to represent fixed (but nonzero) precision numbers. The

cv::getLinearFilter() returns a (smart) pointer to a cv::BaseFilter-derived object that implements a general two-dimensional non-separable linear filter based on the kernel (and anchor point) you provide. You can provide an optional output offset delta, which will be applied after your kernel and before the result is written to the destination array. The last argument, bits, is used when your kernel is an integer matrix, but you want to use it to represent fixed (but nonzero) precision numbers. The cv::getLinearRowFilter() and cv::getLinearColumnFilter() functions return (smart) pointers to cv::BaseRowFilter and cv::BaseColumnFilter objects, respectively. The one additional argument that is interesting in these two is the symmetryType argument. The symmetry type may be any of the options shown in Table 5-5, including combinations of the types where meaningful (e.g., a kernel might be both smooth and symmetrical, in which case symmetryType would be cv::KERNEL_SMOOTH | cv::KERNEL_SYMMETRICAL). Table 5-5:

and cv::getLinearColumnFilter() functions return (smart) pointers to cv::BaseRowFilter and cv::BaseColumnFilter objects, respectively. The one additional argument that is interesting in these two is the symmetryType argument. The symmetry type may be any of the options shown in Table 5-5, including combinations of the types where meaningful (e.g., a kernel might be both smooth and symmetrical, in which case symmetryType would be cv::KERNEL_SMOOTH | cv::KERNEL_SYMMETRICAL). Table 5-5: Filter kernel symmetry types Kernel Symmetry Meaning  cv::KERNEL_GENERAL                  Generic type, means no special symmetries.  cv::KERNEL_SYMMETRICAL                  Kernel is symmetrical and anchor is at the center.  cv::KERNEL_ASYMMETRICAL                  Kernel is asymmetrical and anchor is at the center.  cv::KERNEL_SMOOTH                  Kernel is greater than or equal to zero everywhere and normalized.  cv::KERNEL_INTEGER                  All kernel elements are integers. You should make sure that what you

types Kernel Symmetry Meaning  cv::KERNEL_GENERAL                  Generic type, means no special symmetries.  cv::KERNEL_SYMMETRICAL                  Kernel is symmetrical and anchor is at the center.  cv::KERNEL_ASYMMETRICAL                  Kernel is asymmetrical and anchor is at the center.  cv::KERNEL_SMOOTH                  Kernel is greater than or equal to zero everywhere and normalized.  cv::KERNEL_INTEGER                  All kernel elements are integers. You should make sure that what you indicate to be the symmetry properties of your kernel are in fact correct. The benefit of this information is that it can lead to a much faster operation of the filter. Alternatively, you can ask OpenCV to automatically inspect your kernel for you, by calling the function cv::getKernelType( cv::InputArray kernel, cv::Point anchor ), which will determine the kernel type for you. This is not really the best way to do things if you can compute the answer yourself at coding time—because of the runtime required to

to be the symmetry properties of your kernel are in fact correct. The benefit of this information is that it can lead to a much faster operation of the filter. Alternatively, you can ask OpenCV to automatically inspect your kernel for you, by calling the function cv::getKernelType( cv::InputArray kernel, cv::Point anchor ), which will determine the kernel type for you. This is not really the best way to do things if you can compute the answer yourself at coding time—because of the runtime required to inspect the kernel—but can be handy if the kernel itself is being dynamically generated at runtime.

getMorphologyFilter(),  getMorphologyRowFilter(), and getMorphologyColumnFilter() These functions are the analogs of the linear filter generators above, only for morphological filters. As with the linear filters, there is a function for creating a non-separable two-dimensional filter, and two methods for creating the two one-dimensional filter types needed to define a separable kernel: cv::Ptr<cv::BaseFilter> cv::getMorphologyFilter(   int            op,                   // Morphology operator to use e.g. cv::MOP_OPEN   int            type,                 // Source and destination type (e.g., cv::U8)    cv::InputArray element,              // Structuring element      cv::Point      anchor = cv::Point(-1,-1),  // Location of anchor point );  cv::Ptr<cv::BaseRowFilter> cv::getMorphologyRowFilter(   int            op,                   // Morphology operator to use e.g. cv::MOP_OPEN   int            type,                 // Source and destination type (e.g., cv::U8)   int            esize,                //

cv::MOP_OPEN   int            type,                 // Source and destination type (e.g., cv::U8)    cv::InputArray element,              // Structuring element      cv::Point      anchor = cv::Point(-1,-1),  // Location of anchor point );  cv::Ptr<cv::BaseRowFilter> cv::getMorphologyRowFilter(   int            op,                   // Morphology operator to use e.g. cv::MOP_OPEN   int            type,                 // Source and destination type (e.g., cv::U8)   int            esize,                // Element size (width)   int            anchor = -1           // 1-d Location of anchor point );  cv::Ptr<cv::BaseColumnFilter> cv::getMorphologyColumnFilter(   int            op,                   // Morphology operator to use e.g. cv::MOP_OPEN   int            type,                 // Source and destination type (e.g., cv::U8)   int            esize,                // Element size (height)   int            anchor = -1           // 1-d Location of anchor point ); The first function, cv::getMorphologyFilter(),

size (width)   int            anchor = -1           // 1-d Location of anchor point );  cv::Ptr<cv::BaseColumnFilter> cv::getMorphologyColumnFilter(   int            op,                   // Morphology operator to use e.g. cv::MOP_OPEN   int            type,                 // Source and destination type (e.g., cv::U8)   int            esize,                // Element size (height)   int            anchor = -1           // 1-d Location of anchor point ); The first function, cv::getMorphologyFilter(), which is used for non-separable kernels expects a morphological operator type op (from Table 5-3), the input and output image type given by type, a structuring element and an optional anchor point. The row and column filter constructors require the same arguments, except that the element size, esize, is just an integer that indicates the extent of the row (or column) dimension of the kernel. Similarly, the anchor point, anchor, is an integer that, as usual, can be set to -1 to indicate that the center point of

is used for non-separable kernels expects a morphological operator type op (from Table 5-3), the input and output image type given by type, a structuring element and an optional anchor point. The row and column filter constructors require the same arguments, except that the element size, esize, is just an integer that indicates the extent of the row (or column) dimension of the kernel. Similarly, the anchor point, anchor, is an integer that, as usual, can be set to -1 to indicate that the center point of the kernel is to be used. Kernel Builders At this point, you have found your way to the absolute bottom of the food chain: the OpenCV filter stack. High-level functions call filter engines. Filter engines call base filter generators. Base filter generators go to the kernel builders to get the actual arrays that represent individual kernels. The two kernel builders that you may want to access yourself are cv::getDerivKernel(), which constructs the Sobel and Scharr kernels, and cv::getGaussianKernel(), which

the kernel is to be used. Kernel Builders At this point, you have found your way to the absolute bottom of the food chain: the OpenCV filter stack. High-level functions call filter engines. Filter engines call base filter generators. Base filter generators go to the kernel builders to get the actual arrays that represent individual kernels. The two kernel builders that you may want to access yourself are cv::getDerivKernel(), which constructs the Sobel and Scharr kernels, and cv::getGaussianKernel(), which constructs Gaussian kernels. cv::getDerivKernel() The actual kernel array for a derivative filter is generated by cv::getDerivKernel().   void cv::getDerivKernels(   cv::OutputArray kx,    cv::OutputArray ky,    int             dx,                  // order of corresponding derivative in x   int             dy,                  // order of corresponding derivative in y   int             ksize,               // Kernel size      bool            normalize = true,    // If true, divide by box area   int

constructs Gaussian kernels. cv::getDerivKernel() The actual kernel array for a derivative filter is generated by cv::getDerivKernel().   void cv::getDerivKernels(   cv::OutputArray kx,    cv::OutputArray ky,    int             dx,                  // order of corresponding derivative in x   int             dy,                  // order of corresponding derivative in y   int             ksize,               // Kernel size      bool            normalize = true,    // If true, divide by box area   int             ktype     = cv::F32   // Type for filter coefficients ); The result of cv::getDerivKernel() is placed in the kx and ky array arguments. You might recall that the derivative type kernels (Sobel and Scharr) are separable kernels. For this reason, you will get back two arrays, one that is 1-by-ksize (row coefficients, kx) and another that is ksize-by-1 (column

coefficients, ky). These are computed from the x- and y-derivative orders dx and dy. The derivative kernels are always square, so the size argument ksize is an integer. ksize can be any of 1, 3, 5, 7, or cv::SCHARR. The normalize argument tells cv::getDerivKernels() if it should normalize the kernel elements “correctly.”  For situations where you are operating on floating point images, there is no reason not to set normalize to true, but when you are doing operations on integer arrays, it is often more sensible to not normalize the arrays until some later point in your processing, so that you will not throw away precision that you will later need29. The final argument ktype indicates the type of the filter coefficients (or equivalently the type of the arrays kx and ky). The value of ktype can be either cv::F32 or cv::F64. cv::getGaussianKernel() The actual kernel array for a Gaussian filter is generated by cv::getGaussianKernel(). cv::Mat cv::getGaussianKernel(   int             ksize,               //

sensible to not normalize the arrays until some later point in your processing, so that you will not throw away precision that you will later need29. The final argument ktype indicates the type of the filter coefficients (or equivalently the type of the arrays kx and ky). The value of ktype can be either cv::F32 or cv::F64. cv::getGaussianKernel() The actual kernel array for a Gaussian filter is generated by cv::getGaussianKernel(). cv::Mat cv::getGaussianKernel(   int             ksize,               // Kernel size      double          sigmaX,              // Gaussian half-width   int             ktype     = cv::F32   // Type for filter coefficients ); As with the derivative kernel, the Gaussian kernel is separable. For this reason cv::getGaussianKernel() computes only a ksize-by-1 array of coefficients. The value of ksize can be any odd positive number. The argument sigma sets the standard deviation of the approximated Gaussian distribution. The coefficients are computed from sigma according to the

// Kernel size      double          sigmaX,              // Gaussian half-width   int             ktype     = cv::F32   // Type for filter coefficients ); As with the derivative kernel, the Gaussian kernel is separable. For this reason cv::getGaussianKernel() computes only a ksize-by-1 array of coefficients. The value of ksize can be any odd positive number. The argument sigma sets the standard deviation of the approximated Gaussian distribution. The coefficients are computed from sigma according to the following function: 𝑘!=𝛼∙𝑒!!!!"#$%!!!!!!! That is, the coefficient alpha is computed such that the filter overall is normalized. sigma may be set to -1, in which case the value of sigma will be automatically computed from the size ksize.30   The Filter Architecture up Close (Advanced) An architecture as nuanced as the filter architecture in OpenCV is something like an automobile. You can choose to not understand it, and hope it works right (in which case, you could pretty much have skipped this entire

the following function: 𝑘!=𝛼∙𝑒!!!!"#$%!!!!!!! That is, the coefficient alpha is computed such that the filter overall is normalized. sigma may be set to -1, in which case the value of sigma will be automatically computed from the size ksize.30   The Filter Architecture up Close (Advanced) An architecture as nuanced as the filter architecture in OpenCV is something like an automobile. You can choose to not understand it, and hope it works right (in which case, you could pretty much have skipped this entire section of the chapter), you can understand it at a user level, and be content that this will help you not do terribly unwise things that bite you later (like knowing that the oil does, in fact, have to be changed every few thousand miles), or you can really get down and understand the thing, with the mindset of maintaining it, modifying it, and bending it to your will. This above section of the chapter was mainly for that kind of intermediate users, who want to understand and use wisely, but maybe not

section of the chapter), you can understand it at a user level, and be content that this will help you not do terribly unwise things that bite you later (like knowing that the oil does, in fact, have to be changed every few thousand miles), or you can really get down and understand the thing, with the mindset of maintaining it, modifying it, and bending it to your will. This above section of the chapter was mainly for that kind of intermediate users, who want to understand and use wisely, but maybe not seriously get down into it. This little subsection, however, is for the benefit of those of you who really want to go do some work yourself in the filter architecture. To this end, let’s consider a simple high-level call to cv::boxFilter(). This call will manage to use just about every nasty detail of the architecture before it is done. Example 5-5 shows what really happens when you call cv::boxFilter() and supply it with a source array, a destination array, and the kernel parameters.  Example 5-5: A detailed

get down into it. This little subsection, however, is for the benefit of those of you who really want to go do some work yourself in the filter architecture. To this end, let’s consider a simple high-level call to cv::boxFilter(). This call will manage to use just about every nasty detail of the architecture before it is done. Example 5-5 shows what really happens when you call cv::boxFilter() and supply it with a source array, a destination array, and the kernel parameters.  Example 5-5: A detailed walk-through of the operation of a high-level filter function: cv::boxFilter() cv::boxFilter calls cv::createBoxFilter, and passes the kernel parameters.                                                               29 If you are, in fact, going to do this, you will find yourself needing that normalization coefficient at some point. The normalization coefficient you will need is: 2!"#$%∗!!!"!!"!!. 30 In this case, 𝜎=0.3∙!"#$%!!!−1+0.8.

cv::createBoxFilter calls cv::getRowSumFilter() and gives it type information and kernel parameters cv::getRowSumFilter uses the type information to call the template RowSum<>() constructor RowSum<> constructor returns a type-specific object derived from BaseRowFilter. class cv::RowSum<> implements operator() for row summation. cv::getRowSumFilter returns the cv::RowSum<> object cv::createBoxFilter calls cv::getColumnSumFilter() and gives it type information and kernel parameters cv::getColumnSumFilter uses the type information to call the template ColumnSum<>() constructor ColumnSum<>() constructor returns a type-specific object derived from BaseColumnFilter. class cv::ColumnSum<> implements operator() for column summation. cv::getColumnSumFilter returns the cv::ColumnSum<> object cv::createBoxFilter creates a new cv::FilterEngine object, and passes it the cv::RowSum<> and cv::ColumnSum<> filter objects. cv::createBoxFilter returns a pointer to the new cv::FilterEngine object cv::boxFilter() calls the

uses the type information to call the template ColumnSum<>() constructor ColumnSum<>() constructor returns a type-specific object derived from BaseColumnFilter. class cv::ColumnSum<> implements operator() for column summation. cv::getColumnSumFilter returns the cv::ColumnSum<> object cv::createBoxFilter creates a new cv::FilterEngine object, and passes it the cv::RowSum<> and cv::ColumnSum<> filter objects. cv::createBoxFilter returns a pointer to the new cv::FilterEngine object cv::boxFilter() calls the apply method on the new cv::FilterEngine object it was given. The source and destination images given to cv::boxFilter are passed to the apply() method. (note that apply() would accept an ROI, but in this context one is not generated). The (generic) cv::FilterEngine::apply() method calls the (generic) cv::FilterEngine::start() method. It supplies the source image and any source image ROI. If an ROI was passed in, that is recorded in a cv::FilterEngine member variable. cv::FilterEngine::start() does some

apply method on the new cv::FilterEngine object it was given. The source and destination images given to cv::boxFilter are passed to the apply() method. (note that apply() would accept an ROI, but in this context one is not generated). The (generic) cv::FilterEngine::apply() method calls the (generic) cv::FilterEngine::start() method. It supplies the source image and any source image ROI. If an ROI was passed in, that is recorded in a cv::FilterEngine member variable. cv::FilterEngine::start() does some book-keeping that has to do with whether or not the source image is actually a subwindow of a larger image, we’ll ignore this for now. cv::FilterEngine::start() calls another (different) cv::FilterEngine::start() method. It passes this start() method the size of the “true” source image (i.e., if source is a view on some parent, this is the parent), and the location of the source ROI relative to that parent image. This cv::FilterEngine::start() allocates the internal ‘rows’ buffer of the FilterEngine. This is

that has to do with whether or not the source image is actually a subwindow of a larger image, we’ll ignore this for now. cv::FilterEngine::start() calls another (different) cv::FilterEngine::start() method. It passes this start() method the size of the “true” source image (i.e., if source is a view on some parent, this is the parent), and the location of the source ROI relative to that parent image. This cv::FilterEngine::start() allocates the internal ‘rows’ buffer of the FilterEngine. This is a vector of N char* pointers, where N is set either by the maxBufRows argument to apply (which nobody ever uses) or the kernel height. cv::FilterEngine::start() handles the borders cv::FilterEngine::start() computes the first source row that will be used in the computation of all results. Similarly, cv::FilterEngine::start() computes the last source row that will be used in the computation of all results. These are stored in in a cv::FilterEngine member variable cv::FilterEngine::start() calls the reset() member of

of N char* pointers, where N is set either by the maxBufRows argument to apply (which nobody ever uses) or the kernel height. cv::FilterEngine::start() handles the borders cv::FilterEngine::start() computes the first source row that will be used in the computation of all results. Similarly, cv::FilterEngine::start() computes the last source row that will be used in the computation of all results. These are stored in in a cv::FilterEngine member variable cv::FilterEngine::start() calls the reset() member of the columnFilter member (which is the cv::ColumnSum object constructed above).

cv::FilterEngine::start() (somewhat unnecessarily) returns the computed start source row.  The first cv::FilterEngine::start() method returns the computed first row, but adjusted into the “coordinates” of the parent view (if the source image is actually a view on a larger array.) The cv::FilterEngine::apply() method computes the destination pointer for the first destination pixel. This is just the data area for the source image offset by the “true” start row multiplied by the source step. The cv::FilterEngine::apply() method computes the total number of rows that will be analyzed (using the difference of the two row value member variables computed earlier by cv::FilterEngine::start()). The cv::FilterEngine::apply() method computes the address of the first pixel in the destination image to be computed. cv::FilterEngine::apply() method calls the (generic) cv::FilterEngine::proceed() method, passing it the locations of the first source pixel, the first destination pixel, the source step, the destination step,

method computes the total number of rows that will be analyzed (using the difference of the two row value member variables computed earlier by cv::FilterEngine::start()). The cv::FilterEngine::apply() method computes the address of the first pixel in the destination image to be computed. cv::FilterEngine::apply() method calls the (generic) cv::FilterEngine::proceed() method, passing it the locations of the first source pixel, the first destination pixel, the source step, the destination step, the total number of rows to be “used” to compute all destination pixels. For each destination pixel, cv::FilterEngine proceed() calls the member operator() of the rowFilter member of the cv::FilterEngine (our cv::RowSum<> object), passing it the source pointer, the destination pointer, and the width of the given to apply (which in this case is null) or the width of the image. cv::RowSum<>::operator() then computes a sum across ‘width’ source pixels accumulating them into the indicated destination pixel. All subsequent

of rows to be “used” to compute all destination pixels. For each destination pixel, cv::FilterEngine proceed() calls the member operator() of the rowFilter member of the cv::FilterEngine (our cv::RowSum<> object), passing it the source pointer, the destination pointer, and the width of the given to apply (which in this case is null) or the width of the image. cv::RowSum<>::operator() then computes a sum across ‘width’ source pixels accumulating them into the indicated destination pixel. All subsequent destination pixels in the row are then computed from that. For each destination pixel, proceed() calls the member operator() of the cv::columnFilter member of the cv::FilterEngine (our cv::ColumnSum<> object), passing it an array of source pointers corresponding to the relevant rows of the source image, a pointer to the destination pixel, the destination step, the number of columns that will be summed, and the width of the image (or of any ROI that was given to the filter when it was constructed). The

destination pixels in the row are then computed from that. For each destination pixel, proceed() calls the member operator() of the cv::columnFilter member of the cv::FilterEngine (our cv::ColumnSum<> object), passing it an array of source pointers corresponding to the relevant rows of the source image, a pointer to the destination pixel, the destination step, the number of columns that will be summed, and the width of the image (or of any ROI that was given to the filter when it was constructed). The cv::ColumnSum<>::operator() then computes the sum across count columns and accumulates the result in the destination pixel. Compute the sum for width pixels. cv::FilterEngine::proceed() returns cv::FilterEngine::apply() returns cv::boxFilter() returns In words, this is what happens. At the top level, the user calls cv::boxFilter(), and gives it a source and destination image and the size of the kernel. cv::boxFilter(), however, is really just a pretty thin wrapper; its purpose is to create a cv::FilterEngine

then computes the sum across count columns and accumulates the result in the destination pixel. Compute the sum for width pixels. cv::FilterEngine::proceed() returns cv::FilterEngine::apply() returns cv::boxFilter() returns In words, this is what happens. At the top level, the user calls cv::boxFilter(), and gives it a source and destination image and the size of the kernel. cv::boxFilter(), however, is really just a pretty thin wrapper; its purpose is to create a cv::FilterEngine that will handle the box filtering process. In order to do this, however, it will need the actual base filter objects. Because the box filter is a separable filter, two base filter objects are required, one for the horizontal (rows) and one for the vertical (columns).  To create a base filter object, cv::boxFilter() will use the cv::getRowSumFilter() and cv::getColumnSumFilter() methods31. What these do is they instantiate and return template base                                                              31 It is worth

the box filtering process. In order to do this, however, it will need the actual base filter objects. Because the box filter is a separable filter, two base filter objects are required, one for the horizontal (rows) and one for the vertical (columns).  To create a base filter object, cv::boxFilter() will use the cv::getRowSumFilter() and cv::getColumnSumFilter() methods31. What these do is they instantiate and return template base                                                              31 It is worth mentioning here that these latter two functions are not officially part of the OpenCV API; they are below the API functions. This means that you should probably not be calling them yourself, as they are not guaranteed to

filters called cv::RowSum<> and cv::ColumnSum<>. These objects are templates because each filter needs to know what types are being used for the source and destination images in order to implement their functionality in the most efficient way possible. Recall that it is the operator() members of these objects that do the low-level work (which we will get to momentarily). The cv::getRowSumFilter() and cv::getColumnSumFilter() methods return cv::Ptr<> smart pointers to the base filters, which are then passed to the cv::FilterEngine constructor. The constructed filter engine is then returned to cv::boxFilter() in the form of another cv::Ptr<> smart pointer. Once this pointer to the box filter engine is returned, cv::boxFilter() then just calls the apply() method of the filter engine. Of course, a lot gets done inside of this method. The cv::FilterEngine::apply() method takes as argument the source and destination images. It would also accept a region of interest (ROI), but none is generated in this context (so

to the cv::FilterEngine constructor. The constructed filter engine is then returned to cv::boxFilter() in the form of another cv::Ptr<> smart pointer. Once this pointer to the box filter engine is returned, cv::boxFilter() then just calls the apply() method of the filter engine. Of course, a lot gets done inside of this method. The cv::FilterEngine::apply() method takes as argument the source and destination images. It would also accept a region of interest (ROI), but none is generated in this context (so the default “whole image” is assumed). This apply() method is a generic member of cv::FilterEngine, in the sense that it is not overloaded in any way. The filter engine object is not a derived object, but the generic base class element. This is the way the architecture is designed; you should not have to overload cv::FilterEngine. (This is in contrast to the base filters, which are interfaces only and always need to be overloaded.) The cv::FilterEngine::apply() then calls cv::FilterEngine::start(). There is

default “whole image” is assumed). This apply() method is a generic member of cv::FilterEngine, in the sense that it is not overloaded in any way. The filter engine object is not a derived object, but the generic base class element. This is the way the architecture is designed; you should not have to overload cv::FilterEngine. (This is in contrast to the base filters, which are interfaces only and always need to be overloaded.) The cv::FilterEngine::apply() then calls cv::FilterEngine::start(). There is some opportunity for confusion here, because there are two start() methods associated with cv::FilterEnging(). The first start method takes the source image and the source ROI (and some other stuff) as arguments. It is responsible for some minor checking and housekeeping, and then it calls the second start() method. The main piece of housekeeping it does is to resolve the possibility that the source image is actually a view32 on a larger array, which has implications for the isolated argument, for example.

some opportunity for confusion here, because there are two start() methods associated with cv::FilterEnging(). The first start method takes the source image and the source ROI (and some other stuff) as arguments. It is responsible for some minor checking and housekeeping, and then it calls the second start() method. The main piece of housekeeping it does is to resolve the possibility that the source image is actually a view32 on a larger array, which has implications for the isolated argument, for example. Once this is done, the second start() method can be called, and given the parent array (i.e., if the current source is a view of some other array, it is that other larger array that is passed), and a corrected ROI (which is the given ROI converted to the coordinates of the larger parent array). The second start() routine now does some real work. The main responsibility of this start() method is to set up all of the data structures that are going to be needed for the base filters to operate. This is more

Once this is done, the second start() method can be called, and given the parent array (i.e., if the current source is a view of some other array, it is that other larger array that is passed), and a corrected ROI (which is the given ROI converted to the coordinates of the larger parent array). The second start() routine now does some real work. The main responsibility of this start() method is to set up all of the data structures that are going to be needed for the base filters to operate. This is more effort than it might sound like, because the base filters will not be made aware of things like the boundary type, or if they are in an ROI. They will just be handed data to chew on and they will spit out results. In order to achieve this, there will have to be areas for “off the edge” pixels to be placed alongside the real image pixel values so that the base filters will not have to be aware of the difference. This data goes in a filter engine member variable called rows which is an array of char* pointers,

than it might sound like, because the base filters will not be made aware of things like the boundary type, or if they are in an ROI. They will just be handed data to chew on and they will spit out results. In order to achieve this, there will have to be areas for “off the edge” pixels to be placed alongside the real image pixel values so that the base filters will not have to be aware of the difference. This data goes in a filter engine member variable called rows which is an array of char* pointers, one for each row of scratch space that will be needed (these actually form a circular buffer so that data only needs to be copied in once, and can then be used until it is no longer needed). start() is also responsible for computing the (integer ordinal for the) first and last rows that will be used. Again, these are the first and last row of the image in our case, because there is no ROI, but generally, if start() were not being called from inside of cv::boxFilter(), we might have gotten one from

one for each row of scratch space that will be needed (these actually form a circular buffer so that data only needs to be copied in once, and can then be used until it is no longer needed). start() is also responsible for computing the (integer ordinal for the) first and last rows that will be used. Again, these are the first and last row of the image in our case, because there is no ROI, but generally, if start() were not being called from inside of cv::boxFilter(), we might have gotten one from cv::FilterEngine::apply() or directly by whomever else might have called cv::FilterEngine::start(). A final responsibility that cv::FilterEngine::start() has is to call the reset() method on the base column filter. (We will understand why shortly when we get into the base column filter’s internal details.) The starting row (in parent image coordinates) is then returned to the first start() function, which in turn returns it to cv::FilterEngine::apply() in coordinates local to the source image.

or directly by whomever else might have called cv::FilterEngine::start(). A final responsibility that cv::FilterEngine::start() has is to call the reset() method on the base column filter. (We will understand why shortly when we get into the base column filter’s internal details.) The starting row (in parent image coordinates) is then returned to the first start() function, which in turn returns it to cv::FilterEngine::apply() in coordinates local to the source image.                                                                                                                                                                                      remain in existence or have stable interfaces in future versions of the library. There will be several things in this current exposition which are of this kind. 32 Recall that a ‘view’ refers to the case in which you have a cv::Mat A, and then generate a second cv::Mat B with a function like cv::Mat::getSubRect(). In this case B does not have its own data, it has a

remain in existence or have stable interfaces in future versions of the library. There will be several things in this current exposition which are of this kind. 32 Recall that a ‘view’ refers to the case in which you have a cv::Mat A, and then generate a second cv::Mat B with a function like cv::Mat::getSubRect(). In this case B does not have its own data, it has a pointer to the data area in A and offsets and strides which make accesses to B appear as if B were a separate array.

Next, cv::FilterEngine::apply() calls cv::FilterEngine::proceed(), and passes it the source image data, the source image step, the destination image data,  the destination step, and the number of rows from the source image that will be required to compute all destination pixels. It is important not to miss here that, in fact, there is a lot more information provided to proceed(), because when start() was running, it was computing various things that it stored in member variables of the filter engine (like those scratch buffers for the data rows). The source data and destination data areas given to proceed() are the location of the first data needed for actual computation of a required destination pixel. This is a little subtle, because all of the implications of ROIs are handled in cv::FilterEngine::apply() before calling proceed(). These pointers are not just pointers to the data in the images, they are pointers to the beginning or the relevant data. cv::FilterEngine::proceed() is essentially a big for

those scratch buffers for the data rows). The source data and destination data areas given to proceed() are the location of the first data needed for actual computation of a required destination pixel. This is a little subtle, because all of the implications of ROIs are handled in cv::FilterEngine::apply() before calling proceed(). These pointers are not just pointers to the data in the images, they are pointers to the beginning or the relevant data. cv::FilterEngine::proceed() is essentially a big for loop over all of the desired destination pixels. It actually handles them on a row by row basis. It counts through the destination rows, handles actual copy of data into the buffers (the ones that start() allocated for us), as well as updates the circular buffer, and calls the base filters once the data areas are ready. First it calls the operator() member of the base row filter for any given destination pixel, then it calls the operator() member of the base column filter for those pixels. (At least, that is

over all of the desired destination pixels. It actually handles them on a row by row basis. It counts through the destination rows, handles actual copy of data into the buffers (the ones that start() allocated for us), as well as updates the circular buffer, and calls the base filters once the data areas are ready. First it calls the operator() member of the base row filter for any given destination pixel, then it calls the operator() member of the base column filter for those pixels. (At least, that is what it does in this example. If the filter engine were a non-separable filter, then obviously it would just make a single call to the operator() member of the installed two-dimensional filter.) Inside of the base row filter, which in our example is cv::RowSum<>, each pixel in the destination row is computed as a sum over the neighboring pixels. The first such destination pixel is a straight sum. All subsequent pixels are computed from the previous pixel, by subtracting the one contributing pixel that is now

it does in this example. If the filter engine were a non-separable filter, then obviously it would just make a single call to the operator() member of the installed two-dimensional filter.) Inside of the base row filter, which in our example is cv::RowSum<>, each pixel in the destination row is computed as a sum over the neighboring pixels. The first such destination pixel is a straight sum. All subsequent pixels are computed from the previous pixel, by subtracting the one contributing pixel that is now out of the window and adding the one pixel that has just entered the window. Inside of the base column filter cv::ColumnSum<>, the summation over vertical columns (of the results from the row filter summation) is performed. One important difference between the row and column sum filters is that the column filters need to maintain some internal state—this is to enable a similar trick to what the row filter uses for summing, except that the row filter is given an entire row at a time, while the column filter

out of the window and adding the one pixel that has just entered the window. Inside of the base column filter cv::ColumnSum<>, the summation over vertical columns (of the results from the row filter summation) is performed. One important difference between the row and column sum filters is that the column filters need to maintain some internal state—this is to enable a similar trick to what the row filter uses for summing, except that the row filter is given an entire row at a time, while the column filter needs to be called many times in order to complete an entire column.  This call to cv::FilterEngine::proceed() finishes up all of its work and returns. cv::FilterEngine::apply() returns, and then cv::boxFilter() returns. At this point, the results of your computation are sitting in your destination array and you are all done. For each destination pixel, cv::FilterEngine proceed() calls the member operator() of the rowFilter member of the cv::FilterEngine (our cv::RowSum<> object), passing it the source

needs to be called many times in order to complete an entire column.  This call to cv::FilterEngine::proceed() finishes up all of its work and returns. cv::FilterEngine::apply() returns, and then cv::boxFilter() returns. At this point, the results of your computation are sitting in your destination array and you are all done. For each destination pixel, cv::FilterEngine proceed() calls the member operator() of the rowFilter member of the cv::FilterEngine (our cv::RowSum<> object), passing it the source pointer, the destination pointer, and the width of the given to apply (which in this case is null) or the width of the image. cv::RowSum<>::operator() then computes a sum across ‘width’ source pixels accumulating them into the indicated destination pixel. All subsequent destination pixels in the row are then computed from that. For each destination pixel, proceed() calls the member operator() of the cv::columnFilter member of the cv::FilterEngine (our cv::ColumnSum<> object), passing it an array of source

pointer, the destination pointer, and the width of the given to apply (which in this case is null) or the width of the image. cv::RowSum<>::operator() then computes a sum across ‘width’ source pixels accumulating them into the indicated destination pixel. All subsequent destination pixels in the row are then computed from that. For each destination pixel, proceed() calls the member operator() of the cv::columnFilter member of the cv::FilterEngine (our cv::ColumnSum<> object), passing it an array of source pointers corresponding to the relevant rows of the source image, a pointer to the destination pixel, the destination step, the number of columns that will be summed, and the ‘width’ of the image (or of any ROI that was given to the filter when it was constructed). The cv::ColumnSum<>::operator() then computes the sum across ‘count’ columns and accumulates the result in the destination pixel. Compute the sum for ‘width’ pixels.

Summary In this chapter, we learned about general image convolution, including the importance of how boundaries are handled in convolutions. We learned about image kernels, and the difference between linear and nonlinear kernels. We learned how OpenCV implements a number of common image filters, and learned what those filters do to different kinds of input data.  Finally, we learned how OpenCV actually implements the functions that do convolutions. We saw how the general filter architecture is designed, and learned how we could make our own components for that architecture and thus how to “micromanage” the process to tune custom filters for speed. Exercises 1. Load an image of an interesting or sufficiently “rich” scene. Using cv::threshold(), set the threshold to 128. Use each setting type in Table 5-5 on the image and display the results. You should familiarize yourself with thresholding functions because they will prove quite useful. a) Repeat the exercise but use cv::adaptiveThreshold() instead. Set

and learned how we could make our own components for that architecture and thus how to “micromanage” the process to tune custom filters for speed. Exercises 1. Load an image of an interesting or sufficiently “rich” scene. Using cv::threshold(), set the threshold to 128. Use each setting type in Table 5-5 on the image and display the results. You should familiarize yourself with thresholding functions because they will prove quite useful. a) Repeat the exercise but use cv::adaptiveThreshold() instead. Set param1=5. b) Repeat part a using param1=0 and then param1=-5.  2. Load an image with interesting textures. Smooth the image in several ways using cv::smooth() with smoothtype=cv::GAUSSIAN. a) Use a symmetric 3-by-3, 5-by-5, 9-by-9 and 11-by-11 smoothing window size and display the results. b) Are the output results nearly the same by smoothing the image twice with a 5-by-5 Gaussian filter as when you smooth once with two 11-by-11 filters? Why or why not? 3. Display the filter, creating a 100-by-100

Set param1=5. b) Repeat part a using param1=0 and then param1=-5.  2. Load an image with interesting textures. Smooth the image in several ways using cv::smooth() with smoothtype=cv::GAUSSIAN. a) Use a symmetric 3-by-3, 5-by-5, 9-by-9 and 11-by-11 smoothing window size and display the results. b) Are the output results nearly the same by smoothing the image twice with a 5-by-5 Gaussian filter as when you smooth once with two 11-by-11 filters? Why or why not? 3. Display the filter, creating a 100-by-100 single-channel image. Clear it and set the center pixel equal to 255. a) Smooth this image with a 5-by-5 Gaussian filter and display the results. What did you find? b) Do this again but now with a 9-by-9 Gaussian filter. c) What does it look like if you start over and smooth the image twice with the 5-by-5 filter? Compare this with the 9-by-9 results. Are they nearly the same? Why or why not? 4. Load an interesting image. Again, blur it with cv::smooth() using a Gaussian filter. a) Set param1=param2=9. Try

single-channel image. Clear it and set the center pixel equal to 255. a) Smooth this image with a 5-by-5 Gaussian filter and display the results. What did you find? b) Do this again but now with a 9-by-9 Gaussian filter. c) What does it look like if you start over and smooth the image twice with the 5-by-5 filter? Compare this with the 9-by-9 results. Are they nearly the same? Why or why not? 4. Load an interesting image. Again, blur it with cv::smooth() using a Gaussian filter. a) Set param1=param2=9. Try several settings of param3 (e.g., 1, 4, and 6). Display the results. b) This time, set param1=param2=0 before setting param3 to 1, 4, and 6. Display the results. Are they different? Why? c) Again use param1=param2=0 but now set param3=1 and param4=9. Smooth the picture and display the results. d) Repeat part c but with param3=9 and param4=1. Display the results. e) Now smooth the image once with the settings of part c and once with the settings of part d. Display the results. f) Compare the results in part

several settings of param3 (e.g., 1, 4, and 6). Display the results. b) This time, set param1=param2=0 before setting param3 to 1, 4, and 6. Display the results. Are they different? Why? c) Again use param1=param2=0 but now set param3=1 and param4=9. Smooth the picture and display the results. d) Repeat part c but with param3=9 and param4=1. Display the results. e) Now smooth the image once with the settings of part c and once with the settings of part d. Display the results. f) Compare the results in part e with smoothings that use param3=param4=9 and param3=param4=0 (i.e., a 9-by-9 filter). Are the results the same? Why or why not? 5. Create a low-variance random image (use a random number call such that the numbers don’t differ by much more than 3 and most numbers are near 0). Load the image into a drawing program such as PowerPoint and then draw a wheel of lines meeting at a single point. Use bilateral filtering on the resulting image and explain the results. 6. In a drawing program such as PowerPoint,

e with smoothings that use param3=param4=9 and param3=param4=0 (i.e., a 9-by-9 filter). Are the results the same? Why or why not? 5. Create a low-variance random image (use a random number call such that the numbers don’t differ by much more than 3 and most numbers are near 0). Load the image into a drawing program such as PowerPoint and then draw a wheel of lines meeting at a single point. Use bilateral filtering on the resulting image and explain the results. 6. In a drawing program such as PowerPoint, draw a series of concentric circles forming a bull’s-eye.

a) Make a series of lines going into the bull’s-eye. Save the image. b) Using a 3-by-3 aperture size, take and display the first-order x- and y-derivatives of your picture. Then increase the aperture size to 5-by-5, 9-by-9, and 13-by-13. Describe the results. 7. Create a new image that is just a 45 degree line, white on black. For a given series of aperture sizes, we will take the image’s first-order x-derivative (𝑑𝑥) and first-order y-derivative (𝑑𝑦). We will then take measurements of this line as follows. The (𝑑𝑥) and (𝑑𝑦) images constitute the gradient of the input image. The magnitude at location 𝑖,𝑗 is 𝑚𝑎𝑔𝑖,𝑗=𝑑𝑥!𝑖,𝑗+𝑑𝑦!𝑖,𝑗 and the angle is  Θ𝑖,𝑗=𝑎𝑡𝑎𝑛2(𝑑𝑦𝑖,𝑗,𝑑𝑥𝑖,𝑗). Scan across the image and find places where the magnitude is at or near maximum. Record the angle at these places. Average the angles and report that as the measured line angle. a) Do this for a 3-by-3 aperture Sobel filter. b) Do this for a 5-by-5 filter. c) Do this for a 9-by-9 filter. d) Do the results change? If so, why? 8. Find and load

The (𝑑𝑥) and (𝑑𝑦) images constitute the gradient of the input image. The magnitude at location 𝑖,𝑗 is 𝑚𝑎𝑔𝑖,𝑗=𝑑𝑥!𝑖,𝑗+𝑑𝑦!𝑖,𝑗 and the angle is  Θ𝑖,𝑗=𝑎𝑡𝑎𝑛2(𝑑𝑦𝑖,𝑗,𝑑𝑥𝑖,𝑗). Scan across the image and find places where the magnitude is at or near maximum. Record the angle at these places. Average the angles and report that as the measured line angle. a) Do this for a 3-by-3 aperture Sobel filter. b) Do this for a 5-by-5 filter. c) Do this for a 9-by-9 filter. d) Do the results change? If so, why? 8. Find and load a picture of a face where the face is frontal, has eyes open, and takes up most or all of the image area. Write code to find the pupils of the eyes. A Laplacian “likes” a bright central point surrounded by dark. Pupils are just the opposite. Invert and convolve with a sufficiently large Laplacian. 9. Use a camera to take two pictures of the same scene while moving the camera as little as possible. Load these images into the computer as src1 and src1. a) Take the absolute value of src1 minus src1 (subtract

a picture of a face where the face is frontal, has eyes open, and takes up most or all of the image area. Write code to find the pupils of the eyes. A Laplacian “likes” a bright central point surrounded by dark. Pupils are just the opposite. Invert and convolve with a sufficiently large Laplacian. 9. Use a camera to take two pictures of the same scene while moving the camera as little as possible. Load these images into the computer as src1 and src1. a) Take the absolute value of src1 minus src1 (subtract the images); call it diff12 and display. If this were done perfectly, diff12 would be black. Why isn’t it? b) Create cleandiff by using cv::erode() and then cv::dilate() on diff12. Display the results. c) Create dirtydiff by using cv::dilate() and then cv::erode() on diff12 and then display. d) Explain the difference between cleandiff and dirtydiff. 10. Take a picture of a scene. Then, without moving the camera, put a coffee cup in the scene and take a second picture. Load these images and convert both to

the images); call it diff12 and display. If this were done perfectly, diff12 would be black. Why isn’t it? b) Create cleandiff by using cv::erode() and then cv::dilate() on diff12. Display the results. c) Create dirtydiff by using cv::dilate() and then cv::erode() on diff12 and then display. d) Explain the difference between cleandiff and dirtydiff. 10. Take a picture of a scene. Then, without moving the camera, put a coffee cup in the scene and take a second picture. Load these images and convert both to 8-bit grayscale images. a) Take the absolute value of their difference. Display the result, which should look like a noisy mask of a coffee mug. b) Do a binary threshold of the resulting image using a level that preserves most of the coffee mug but removes some of the noise. Display the result. The “on” values should be set to 255. c) Do a cv::MOP_OPEN on the image to further clean up noise. 11. Create a clean mask from noise. After completing exercise 10, continue by keeping only the largest remaining

to 8-bit grayscale images. a) Take the absolute value of their difference. Display the result, which should look like a noisy mask of a coffee mug. b) Do a binary threshold of the resulting image using a level that preserves most of the coffee mug but removes some of the noise. Display the result. The “on” values should be set to 255. c) Do a cv::MOP_OPEN on the image to further clean up noise. 11. Create a clean mask from noise. After completing exercise 10, continue by keeping only the largest remaining shape in the image. Set a pointer to the upper-left of the image and then traverse the image. When you find a pixel of value 255 (“on”), store the location and then flood fill (Chapter 6, cv::floodFill()) it using a value of 100. Read the connected component returned from flood fill and record the area of filled region. If there is another larger region in the image, then flood fill the smaller region using a value of 0 and delete its recorded area. If the new region is larger than the previous region, then

in the image. Set a pointer to the upper-left of the image and then traverse the image. When you find a pixel of value 255 (“on”), store the location and then flood fill (Chapter 6, cv::floodFill()) it using a value of 100. Read the connected component returned from flood fill and record the area of filled region. If there is another larger region in the image, then flood fill the smaller region using a value of 0 and delete its recorded area. If the new region is larger than the previous region, then flood fill the previous region using the value 0 and delete its location. Finally, fill the remaining largest region with 255. Display the results. We now have a single, solid mask for the coffee mug. 12. For this exercise, use the mask created in exercise 10 or create another mask of your own (perhaps by drawing a digital picture, or simply use a square). Load an outdoor scene. Now use this mask with copyTo(), to copy an image of a mug into the scene. 13. Load an image of a scene and convert it to grayscale.

fill the previous region using the value 0 and delete its location. Finally, fill the remaining largest region with 255. Display the results. We now have a single, solid mask for the coffee mug. 12. For this exercise, use the mask created in exercise 10 or create another mask of your own (perhaps by drawing a digital picture, or simply use a square). Load an outdoor scene. Now use this mask with copyTo(), to copy an image of a mug into the scene. 13. Load an image of a scene and convert it to grayscale. a) Run the morphological Top Hat operation on your image and display the results.

b) Convert the resulting image into an 8-bit mask. c) Copy a grayscale value into the original image where the Top Hat mask (from b) is nonzero. Display the results. 14. Use cv::filter2D() to create a filter that detects only 60 degree lines in an image. Display the results on a sufficiently interesting image scene. 15. Refer to the Sobel derivative filter shown in Figure 5-15. Make a kernel of this figure and convolve an image with it. Then, make two kernels of the separable parts. Take the image again, and convolve with first the x and then the y kernels. Subtract these two images to convince yourself that the separable operation produces the exact same result. 16. Separable kernels. Create a 3-by-3 Gaussian kernel using rows [(1/16, 2/16, 1/16), (2/16, 4/16, 2/16), (1/16, 2/16, 1/16)] and with anchor point in the middle. a) Run this kernel on an image and display the results. b) Now create two one-dimensional kernels with anchors in the center: one going “across” (1/4, 2/4, 1/4), and one going down (1/4,

convolve with first the x and then the y kernels. Subtract these two images to convince yourself that the separable operation produces the exact same result. 16. Separable kernels. Create a 3-by-3 Gaussian kernel using rows [(1/16, 2/16, 1/16), (2/16, 4/16, 2/16), (1/16, 2/16, 1/16)] and with anchor point in the middle. a) Run this kernel on an image and display the results. b) Now create two one-dimensional kernels with anchors in the center: one going “across” (1/4, 2/4, 1/4), and one going down (1/4, 2/4, 1/4). Load the same original image and use cv::filter2D() to convolve the image twice, once with the first 1D kernel and once with the second 1D kernel. Describe the results. c) Describe the order of complexity (number of operations) for the kernel in part a) and for the kernels in part b). The difference is the advantage of being able to use separable kernels and the entire Gaussian class of filters—or any linearly decomposable filter that is separable, since convolution is a linear operation. 17. Can

1/4). Load the same original image and use cv::filter2D() to convolve the image twice, once with the first 1D kernel and once with the second 1D kernel. Describe the results. c) Describe the order of complexity (number of operations) for the kernel in part a) and for the kernels in part b). The difference is the advantage of being able to use separable kernels and the entire Gaussian class of filters—or any linearly decomposable filter that is separable, since convolution is a linear operation. 17. Can you make a separable kernel from the filter shown in Figure 5-15? If so, show what it looks like.

6 General Image Transforms Overview In the previous chapter, we covered the class of image transformations that can be understood specifically in terms of convolution. Of course, there are a lot of useful operations that cannot be expressed in this way (i.e., as a little window scanning over the image doing one thing or another). In general, transformations that can be expressed as convolutions are local, meaning that even though they may change the entire image, the effect on any particular pixel is determined by only a small number of pixels around that one. The transforms we will look in this chapter generally will not have this property.  Some very useful image transforms are very simple, and you will use them all of the time—resize for example. Others are somewhat more special purpose. In many cases, the image transforms we will look at in this chapter have the purpose of converting an image into some entirely different representation. This different representation will usually still be an array of

by only a small number of pixels around that one. The transforms we will look in this chapter generally will not have this property.  Some very useful image transforms are very simple, and you will use them all of the time—resize for example. Others are somewhat more special purpose. In many cases, the image transforms we will look at in this chapter have the purpose of converting an image into some entirely different representation. This different representation will usually still be an array of values, but those values might be quite different in meaning than the intensity values in the input image. An example of this would be the frequency representation resulting from a Fourier transform. In a few cases, the result of the transformation will be something like a list of components, as would be the case for the Hough Line Transform. There are a number of useful transforms that arise repeatedly in computer vision. OpenCV provides complete implementations of some of the more common ones as well as building

but those values might be quite different in meaning than the intensity values in the input image. An example of this would be the frequency representation resulting from a Fourier transform. In a few cases, the result of the transformation will be something like a list of components, as would be the case for the Hough Line Transform. There are a number of useful transforms that arise repeatedly in computer vision. OpenCV provides complete implementations of some of the more common ones as well as building blocks to help you implement your own transformations. Stretch, Shrink, Warp, and Rotate The simplest image transforms we will encounter are those that resize an image, either to make it larger or smaller. These operations are a little less trivial than you might think, because resizing immediately implies questions about how pixels are interpolated (for enlargement) or merged (for reduction). Uniform Resize We often encounter an image of some size that we would like to convert to an image of some other

blocks to help you implement your own transformations. Stretch, Shrink, Warp, and Rotate The simplest image transforms we will encounter are those that resize an image, either to make it larger or smaller. These operations are a little less trivial than you might think, because resizing immediately implies questions about how pixels are interpolated (for enlargement) or merged (for reduction). Uniform Resize We often encounter an image of some size that we would like to convert to an image of some other size. We may want to upsize (zoom in) or downsize (zoom out) the image; both of these tasks are accomplished by the same function. cv::resize() The cv::resize() function handles all of our resizing needs. We provide our input image, and the size we would like it be converted to, and it will generate a new image of exactly the desired size.

void cv::resize(   cv::InputArray  src,                         // Input Image   cv::OutputArray dst,                         // Result image   cv::Size        dsize,                       // New Size   double          fx            = 0,           // x-rescale   double          fy            = 0,           // y-rescale   int             interpolation = cv::INTER_LINEAR // interpolation method ); We can specify the size of the output image in two ways. One way is to use absolute sizing; in this case, the dsize argument directly sets the size we would like the result image dst to be. The other option is to use relative sizing; in this case, we set dsize to cv::Size(0,0), and instead set fx and fy to the scale factors we would like to apply to the x- and y-axes respectively.1 The last argument is the interpolation method, which defaults to linear interpolation. The other available options are shown in Table 6-1. Table 6-1: cv::resize() interpolation options Interpolation Meaning  cv::INTER_NEAREST

argument directly sets the size we would like the result image dst to be. The other option is to use relative sizing; in this case, we set dsize to cv::Size(0,0), and instead set fx and fy to the scale factors we would like to apply to the x- and y-axes respectively.1 The last argument is the interpolation method, which defaults to linear interpolation. The other available options are shown in Table 6-1. Table 6-1: cv::resize() interpolation options Interpolation Meaning  cv::INTER_NEAREST                Nearest neighbor  cv::INTER_LINEAR                Bilinear  cv::INTER_AREA                Pixel area re-sampling  cv::INTER_CUBIC                 Bicubic interpolation  cv::INTER_LANCZOS4                 Lanczos interpolation over 8-by-8 neighborhood. Interpolation is an important issue here. Pixels in the source image sit on an integer grid; for example, we can refer to a pixel at location (20, 17). When these integer locations are mapped to a new image, there can be gaps—either because the integer source

Nearest neighbor  cv::INTER_LINEAR                Bilinear  cv::INTER_AREA                Pixel area re-sampling  cv::INTER_CUBIC                 Bicubic interpolation  cv::INTER_LANCZOS4                 Lanczos interpolation over 8-by-8 neighborhood. Interpolation is an important issue here. Pixels in the source image sit on an integer grid; for example, we can refer to a pixel at location (20, 17). When these integer locations are mapped to a new image, there can be gaps—either because the integer source pixel locations are mapped to float locations in the destination image and must be rounded to the nearest integer pixel location, or because there are some locations to which no pixels at all are mapped (think about doubling the image size by stretching it; then every other destination pixel would be left blank). These problems are generally referred to as forward projection problems. To deal with such rounding problems and destination gaps, we actually solve the problem backwards: we step through each

pixel locations are mapped to float locations in the destination image and must be rounded to the nearest integer pixel location, or because there are some locations to which no pixels at all are mapped (think about doubling the image size by stretching it; then every other destination pixel would be left blank). These problems are generally referred to as forward projection problems. To deal with such rounding problems and destination gaps, we actually solve the problem backwards: we step through each pixel of the destination image and ask, “Which pixels in the source are needed to fill in this destination pixel?” These source pixels will almost always be on fractional pixel locations, so we must interpolate the source pixels to derive the correct value for our destination value. The default method is bilinear interpolation, but you may choose other methods (as shown in Table 6-1). The easiest approach is to take the resized pixel’s value from its closest pixel in the source image; this is the effect of

pixel of the destination image and ask, “Which pixels in the source are needed to fill in this destination pixel?” These source pixels will almost always be on fractional pixel locations, so we must interpolate the source pixels to derive the correct value for our destination value. The default method is bilinear interpolation, but you may choose other methods (as shown in Table 6-1). The easiest approach is to take the resized pixel’s value from its closest pixel in the source image; this is the effect of choosing the interpolation value cv::INTER_NEAREST. Alternatively, we can linearly weight the 2-by-2 surrounding source pixel values according to how close they are to the destination pixel, which is what cv::INTER_LINEAR does. We can also virtually place the new resized pixel over the old pixels and then average the covered pixel values, as done with cv::INTER_AREA.2 For yet smoother interpolation, we have the option of fitting a cubic spline between the 4-by-4 surrounding pixels in the source image and

choosing the interpolation value cv::INTER_NEAREST. Alternatively, we can linearly weight the 2-by-2 surrounding source pixel values according to how close they are to the destination pixel, which is what cv::INTER_LINEAR does. We can also virtually place the new resized pixel over the old pixels and then average the covered pixel values, as done with cv::INTER_AREA.2 For yet smoother interpolation, we have the option of fitting a cubic spline between the 4-by-4 surrounding pixels in the source image and then reading off the corresponding destination value from the fitted spline; this is the result of choosing the cv::INTER_CUBIC interpolation method. Finally, we have the Lanczos interpolation, which is similar to the cubic method, but uses information from an 8-by-8 area around the pixel.3                                                              1 Either dsize must be cv::Size(0,0) or fx and fy must both be zero. 2 At least that’s what happens when cv::resize() shrinks an image. When it expands an

and then reading off the corresponding destination value from the fitted spline; this is the result of choosing the cv::INTER_CUBIC interpolation method. Finally, we have the Lanczos interpolation, which is similar to the cubic method, but uses information from an 8-by-8 area around the pixel.3                                                              1 Either dsize must be cv::Size(0,0) or fx and fy must both be zero. 2 At least that’s what happens when cv::resize() shrinks an image. When it expands an image, cv::INTER_AREA amounts to the same thing as cv::INTER_NEAREST. 3 The subtleties of the Lanczos filter are beyond the scope of this book, but this filter is commonly used in processing digital images because it has the effect of increasing the perceived sharpness of the image.

It is important to notice the difference between cv::resize() and the similarly named cv::Mat::resize() member function of the cv::Mat class. cv::resize() creates a new image, of a different size, over which the original pixels are mapped. The cv::Mat::resize() member function resizes the image whose member you are calling, and it crops that image to the new size. Pixels are not interpolated (or extrapolated) in the case of cv::Mat::resize(). Image Pyramids Image pyramids [Adelson84] are heavily used in a wide variety of vision applications. An image pyramid is a collection of images—all arising from a single original image—that are successively downsampled until some desired stopping point is reached. (Of course, this stopping point could be a single-pixel image!) There are two kinds of image pyramids that arise often in the literature and in applications: the Gaussian [Rosenfeld80] and Laplacian [Burt83] pyramids [Adelson84]. The Gaussian pyramid is used to downsample images, and the Laplacian pyramid (to

wide variety of vision applications. An image pyramid is a collection of images—all arising from a single original image—that are successively downsampled until some desired stopping point is reached. (Of course, this stopping point could be a single-pixel image!) There are two kinds of image pyramids that arise often in the literature and in applications: the Gaussian [Rosenfeld80] and Laplacian [Burt83] pyramids [Adelson84]. The Gaussian pyramid is used to downsample images, and the Laplacian pyramid (to be discussed shortly) is required when we want to reconstruct an upsampled image from an image lower in the pyramid. cv::pyrDown() Normally, we produce layer (𝑖+1) in the Gaussian pyramid (we denote this layer 𝐺!!!) from layer 𝐺i of the pyramid, by first convolving 𝐺i with a Gaussian kernel and then removing every even-numbered row and column. Of course, in this case, it follows immediately that each image is exactly one-quarter the area of its predecessor. Iterating this process on the input image 𝐺!

(to be discussed shortly) is required when we want to reconstruct an upsampled image from an image lower in the pyramid. cv::pyrDown() Normally, we produce layer (𝑖+1) in the Gaussian pyramid (we denote this layer 𝐺!!!) from layer 𝐺i of the pyramid, by first convolving 𝐺i with a Gaussian kernel and then removing every even-numbered row and column. Of course, in this case, it follows immediately that each image is exactly one-quarter the area of its predecessor. Iterating this process on the input image 𝐺! produces the entire pyramid. OpenCV provides us with a method for generating each pyramid stage from its predecessor: void cv::pyrDown(   cv::InputArray  src,                         // Input Image   cv::OutputArray dst,                         // Result image   const cv::Size& dstsize = cv::Size()         // Output image size ); The cv::pyrDown() method will do exactly this for us if we leave the destination size argument dstsize set to its default value of cv::Size(). To be even a little more specific,

produces the entire pyramid. OpenCV provides us with a method for generating each pyramid stage from its predecessor: void cv::pyrDown(   cv::InputArray  src,                         // Input Image   cv::OutputArray dst,                         // Result image   const cv::Size& dstsize = cv::Size()         // Output image size ); The cv::pyrDown() method will do exactly this for us if we leave the destination size argument dstsize set to its default value of cv::Size(). To be even a little more specific, the default size of the output image is ( (src.cols+1)/2, (src.rows+1)/2 ).4 Alternatively, we can supply a dstsize, which will indicate the size we would like for the output image; dstsize, however, must obey some very strict constraints. Specifically: 𝑑𝑠𝑡𝑠𝑖𝑧𝑒.𝑤𝑖𝑑𝑡ℎ  ∗2−𝑠𝑟𝑐.𝑐𝑜𝑙𝑠  ≤2 𝑑𝑠𝑡𝑠𝑖𝑧𝑒.ℎ𝑒𝑖𝑔ℎ𝑡  ∗2−𝑠𝑟𝑐.𝑟𝑜𝑤𝑠  ≤2 This restriction means that the destination image is very close to half the size of the source image. The use of the dstsize argument is only for handling somewhat esoteric cases in which very

the default size of the output image is ( (src.cols+1)/2, (src.rows+1)/2 ).4 Alternatively, we can supply a dstsize, which will indicate the size we would like for the output image; dstsize, however, must obey some very strict constraints. Specifically: 𝑑𝑠𝑡𝑠𝑖𝑧𝑒.𝑤𝑖𝑑𝑡ℎ  ∗2−𝑠𝑟𝑐.𝑐𝑜𝑙𝑠  ≤2 𝑑𝑠𝑡𝑠𝑖𝑧𝑒.ℎ𝑒𝑖𝑔ℎ𝑡  ∗2−𝑠𝑟𝑐.𝑟𝑜𝑤𝑠  ≤2 This restriction means that the destination image is very close to half the size of the source image. The use of the dstsize argument is only for handling somewhat esoteric cases in which very tight control is needed on how the pyramid is constructed. cv::buildPyramid() It is a relatively common situation that you have an image, and wish to build a sequence of new images that are each downscaled from their predecessor. The function cv::buildPyramid() creates such a stack of images for you in a single call. void cv::buildPyramid(   cv::InputArray          src,                 // Input Image   cv::OutputArrayOfArrays dst,                 // Output Images from pyramid   int

very tight control is needed on how the pyramid is constructed. cv::buildPyramid() It is a relatively common situation that you have an image, and wish to build a sequence of new images that are each downscaled from their predecessor. The function cv::buildPyramid() creates such a stack of images for you in a single call. void cv::buildPyramid(   cv::InputArray          src,                 // Input Image   cv::OutputArrayOfArrays dst,                 // Output Images from pyramid   int                     maxlevel             // Number of pyramid levels );                                                              4 The +1s are there to make sure odd sized images are handled correctly. They have no effect if the image was even sized to begin with.

The argument src is the source image. The argument dst is of a somewhat unusual looking type cv::OutputArrayOfArrays, but you can think of this as just being an STL vector<> or objects of type cv::OutputArray. The most common example of this would be vector<cv::Mat>. The argument maxlevel indicates how many pyramid levels are to be constructed.  Figure 6-1: An image pyramid generated with maxlevel=3 (left); two pyramids interleaved together to create a 2 pyramid (right) The argument maxlevel is any integer greater than or equal to zero, and indicates the number of pyramid images to be generated. When cv::buildPyramid() runs, it will return a vector in dst that is of length maxlevel+1. The first entry in dst will be identical to src. The second will be half as large (i.e., as would result from calling cv::pyrDown()). The third will be half the size of the second, and so on (left-hand panel of Figure 6-1). In practice, one often wants a pyramid with a finer logarithmic scaling than factors of two. One way to

greater than or equal to zero, and indicates the number of pyramid images to be generated. When cv::buildPyramid() runs, it will return a vector in dst that is of length maxlevel+1. The first entry in dst will be identical to src. The second will be half as large (i.e., as would result from calling cv::pyrDown()). The third will be half the size of the second, and so on (left-hand panel of Figure 6-1). In practice, one often wants a pyramid with a finer logarithmic scaling than factors of two. One way to achieve this is to simply call cv::resize() yourself as many times as needed for whatever scale factor you want to use—but this can be quite slow. An alternative (for some common scale factors) is to call cv::resize() only once for each interleaved set of images you want, and then call cv::buildPyramid() on each of those resized “bases.”  You can then interleave these results together for one large finer-grained pyramid. Figure 6-1 (right) shows an example in which two pyramids are generated. The original

achieve this is to simply call cv::resize() yourself as many times as needed for whatever scale factor you want to use—but this can be quite slow. An alternative (for some common scale factors) is to call cv::resize() only once for each interleaved set of images you want, and then call cv::buildPyramid() on each of those resized “bases.”  You can then interleave these results together for one large finer-grained pyramid. Figure 6-1 (right) shows an example in which two pyramids are generated. The original image is first rescaled by a factor of  2, and then cv::buildPyramid() is called on that one image to make a second pyramid of four intermediate images. Once combined with the original pyramid, the result is a finer pyramid with scale factor of 2 across the entire pyramid. cv::pyrUp() Similarly, we can convert an existing image to an image that is twice as large in each direction by the following analogous (but not inverse!) operation: void cv::pyrUp(   cv::InputArray  src,                         // Input

image is first rescaled by a factor of  2, and then cv::buildPyramid() is called on that one image to make a second pyramid of four intermediate images. Once combined with the original pyramid, the result is a finer pyramid with scale factor of 2 across the entire pyramid. cv::pyrUp() Similarly, we can convert an existing image to an image that is twice as large in each direction by the following analogous (but not inverse!) operation: void cv::pyrUp(   cv::InputArray  src,                         // Input Image   cv::OutputArray dst,                         // Result image   const cv::Size& dstsize = cv::Size()         // Output image size ); In this case, the image is first upsized to twice the original in each dimension, with the new (even) rows filled with 0s. Thereafter, a convolution is performed with the Gaussian filter5 to approximate the values of the “missing” pixels.                                                              5 This filter is also normalized to four, rather than to one. This is

Image   cv::OutputArray dst,                         // Result image   const cv::Size& dstsize = cv::Size()         // Output image size ); In this case, the image is first upsized to twice the original in each dimension, with the new (even) rows filled with 0s. Thereafter, a convolution is performed with the Gaussian filter5 to approximate the values of the “missing” pixels.                                                              5 This filter is also normalized to four, rather than to one. This is appropriate because the inserted rows have 0s in all of their pixels before the convolution. (Normally, the sum of  Gaussian kernel elements would be 1, but in case of 2x

Analogous to cv::PyrDown(), if dstsize is set to its default value of cv::Size(), the resulting image will be exactly twice the size (in each dimension) as src. Again, we can supply a dstsize that will indicate the size we would like for the output image dstsize, but it must again obey some very strict constraints. Specifically: 𝑑𝑠𝑡𝑠𝑖𝑧𝑒.𝑤𝑖𝑑𝑡ℎ∗2−𝑠𝑟𝑐.𝑐𝑜𝑙𝑠  ≤(𝑑𝑠𝑡𝑠𝑖𝑧𝑒.𝑤𝑖𝑑𝑡ℎ  %  2)    𝑑𝑠𝑡𝑠𝑖𝑧𝑒.ℎ𝑒𝑖𝑔ℎ𝑡  ∗2−𝑠𝑟𝑐.𝑟𝑜𝑤𝑠  ≤(𝑑𝑠𝑡𝑠𝑖𝑧𝑒.ℎ𝑒𝑖𝑔ℎ𝑡  %  2) This restriction means that the destination image is very close to double the size of the source image. As before, the use of the dstsize argument is only for handling somewhat esoteric cases in which very tight control is needed on how the pyramid is constructed. The Laplacian Pyramid We noted previously that the operator cv::pyrUp() is not the inverse of cv::pyrDown(). This should be evident because cv::pyrDown() is an operator that loses information. In order to restore the original (higher-resolution) image, we would require access to the information that was discarded by the

size of the source image. As before, the use of the dstsize argument is only for handling somewhat esoteric cases in which very tight control is needed on how the pyramid is constructed. The Laplacian Pyramid We noted previously that the operator cv::pyrUp() is not the inverse of cv::pyrDown(). This should be evident because cv::pyrDown() is an operator that loses information. In order to restore the original (higher-resolution) image, we would require access to the information that was discarded by the downsampling. This data forms the Laplacian pyramid. The 𝑖!! layer of the Laplacian pyramid is defined by the relation: 𝐿!=𝐺!−𝑈𝑃(𝐺!!!)⨂ℊ!!! Here the operator 𝑈𝑃() upsizes by mapping each pixel in location (x, y) in the original image to pixel (2𝑥+1,2𝑦+1) in the destination image; the ⨂ symbol denotes convolution; and ℊ!!! is a 5-by-5 Gaussian kernel. Of course, 𝑈𝑃(𝐺!!!)⨂ℊ!!! is the definition of the cv::pyrUp() operator provided by OpenCV. Hence, we can use OpenCV to compute the Laplacian operator directly

downsampling. This data forms the Laplacian pyramid. The 𝑖!! layer of the Laplacian pyramid is defined by the relation: 𝐿!=𝐺!−𝑈𝑃(𝐺!!!)⨂ℊ!!! Here the operator 𝑈𝑃() upsizes by mapping each pixel in location (x, y) in the original image to pixel (2𝑥+1,2𝑦+1) in the destination image; the ⨂ symbol denotes convolution; and ℊ!!! is a 5-by-5 Gaussian kernel. Of course, 𝑈𝑃(𝐺!!!)⨂ℊ!!! is the definition of the cv::pyrUp() operator provided by OpenCV. Hence, we can use OpenCV to compute the Laplacian operator directly as: 𝐿!=𝐺!−𝑝𝑦𝑟𝑈𝑝(𝐺!!!) The Gaussian and Laplacian pyramids are shown diagrammatically in Table 6-2, which also shows the inverse process for recovering the original image from the sub-images. Note how the Laplacian is really an approximation that uses the difference of Gaussians, as revealed in the preceding equation and diagrammed in the figure.   Figure 6-2: The Gaussian pyramid and its inverse, the Laplacian pyramid

as: 𝐿!=𝐺!−𝑝𝑦𝑟𝑈𝑝(𝐺!!!) The Gaussian and Laplacian pyramids are shown diagrammatically in Table 6-2, which also shows the inverse process for recovering the original image from the sub-images. Note how the Laplacian is really an approximation that uses the difference of Gaussians, as revealed in the preceding equation and diagrammed in the figure.   Figure 6-2: The Gaussian pyramid and its inverse, the Laplacian pyramid                                                                                                                                                                                      pyramid up-sampling—in the 2D case—all the kernel elements are multiplied by 4 to recover the average brightness after inserting zero rows and columns.)

Nonuniform Mappings In this section, we turn to geometric manipulations of images, i.e., those transformations that have their origin at the intersection of three-dimensional geometry and projective geometry.6 Such manipulations include both uniform and nonuniform resizing (the latter is known as warping). There are many reasons to perform these operations: for example, warping and rotating an image so that it can be superimposed on a wall in an existing scene, or artificially enlarging a set of training images used for object recognition. 7 The functions that can stretch, shrink, warp, and/or rotate an image are called geometric transforms (for an early exposition, see [Semple79]). For planar areas, there are two flavors of geometric transforms: transforms that use a 2-by-3 matrix, which are called affine transforms; and transforms based on a 3-by-3 matrix, which are called perspective transforms or homographies. You can think of the latter transformation as a method for computing the way in which a plane in

used for object recognition. 7 The functions that can stretch, shrink, warp, and/or rotate an image are called geometric transforms (for an early exposition, see [Semple79]). For planar areas, there are two flavors of geometric transforms: transforms that use a 2-by-3 matrix, which are called affine transforms; and transforms based on a 3-by-3 matrix, which are called perspective transforms or homographies. You can think of the latter transformation as a method for computing the way in which a plane in three dimensions is perceived by a particular observer, who might not be looking straight on at that plane. An affine transformation is any transformation that can be expressed in the form of a matrix multiplication followed by a vector addition. In OpenCV, the standard style of representing such a transformation is as a 2-by-3 matrix. We define: 𝐴≡𝑎!!𝑎!"𝑎!"𝑎!!        𝐵≡  𝑏!𝑏!     𝑇≡𝐴𝐵     𝑋≡𝑥𝑦       𝑋!≡𝑥𝑦1 It is easily seen that the effect of the affine transformation 𝐴  ·  𝑋  +  𝐵 is exactly equivalent to

dimensions is perceived by a particular observer, who might not be looking straight on at that plane. An affine transformation is any transformation that can be expressed in the form of a matrix multiplication followed by a vector addition. In OpenCV, the standard style of representing such a transformation is as a 2-by-3 matrix. We define: 𝐴≡𝑎!!𝑎!"𝑎!"𝑎!!        𝐵≡  𝑏!𝑏!     𝑇≡𝐴𝐵     𝑋≡𝑥𝑦       𝑋!≡𝑥𝑦1 It is easily seen that the effect of the affine transformation 𝐴  ·  𝑋  +  𝐵 is exactly equivalent to extending the vector 𝑋 into the vector 𝑋´ and simply left-multiplying 𝑋´ by 𝑇. Affine transformations can be visualized as follows. Any parallelogram 𝐴𝐵𝐶𝐷 in a plane can be mapped to any other parallelogram 𝐴’𝐵’𝐶’𝐷’  by some affine transformation. If the areas of these parallelograms are nonzero, then the implied affine transformation is defined uniquely by (three vertices of) the two parallelograms. If you like, you can think of an affine transformation as drawing your image into a big rubber sheet and then

the vector 𝑋 into the vector 𝑋´ and simply left-multiplying 𝑋´ by 𝑇. Affine transformations can be visualized as follows. Any parallelogram 𝐴𝐵𝐶𝐷 in a plane can be mapped to any other parallelogram 𝐴’𝐵’𝐶’𝐷’  by some affine transformation. If the areas of these parallelograms are nonzero, then the implied affine transformation is defined uniquely by (three vertices of) the two parallelograms. If you like, you can think of an affine transformation as drawing your image into a big rubber sheet and then deforming the sheet by pushing or pulling8 on the corners to make different kinds of parallelograms.  Figure 6-3: Affine and perspective transformations                                                              6 We will cover these transformations in detail here, and will return to them in Chapter 11 when we discuss how they can be used in the context of three-dimensional vision techniques. 7 This activity might seem a bit dodgy; after all, wouldn’t it be better to just use a recognition method that’s

deforming the sheet by pushing or pulling8 on the corners to make different kinds of parallelograms.  Figure 6-3: Affine and perspective transformations                                                              6 We will cover these transformations in detail here, and will return to them in Chapter 11 when we discuss how they can be used in the context of three-dimensional vision techniques. 7 This activity might seem a bit dodgy; after all, wouldn’t it be better to just use a recognition method that’s invariant to local affine distortions? Nonetheless, this method has a long history and is quite useful in practice. 8 One can even pull in such a manner as to invert the parallelogram.

When we have multiple images that we know to be slightly different views of the same object, we might want to compute the actual transforms that relate the different views. In this case, affine transformations are often used to model the views because, having fewer parameters, they are easier to solve for. The downside is that true perspective distortions can only be modeled by a homography,9 so affine transforms yield a representation that cannot accommodate all possible relationships between the views. On the other hand, for small changes in viewpoint the resulting distortion is affine, so in some circumstances, an affine transformation may be sufficient. Affine transforms can convert rectangles to parallelograms. They can squash the shape but must keep the sides parallel; they can rotate it and/or scale it. Perspective transformations offer more flexibility; a perspective transform can turn a rectangle into a trapezoid or any general quadrilateral. Of course, since parallelograms are also trapezoids,

On the other hand, for small changes in viewpoint the resulting distortion is affine, so in some circumstances, an affine transformation may be sufficient. Affine transforms can convert rectangles to parallelograms. They can squash the shape but must keep the sides parallel; they can rotate it and/or scale it. Perspective transformations offer more flexibility; a perspective transform can turn a rectangle into a trapezoid or any general quadrilateral. Of course, since parallelograms are also trapezoids, affine transformations are a subset of perspective transformations. Figure 6-3 shows examples of various affine and perspective transformations. Affine Transformation There are two situations that arise when working with affine transformations. In the first case, we have an image (or a region of interest) we’d like to transform; in the second case, we have a list of points for which we’d like to compute the result of a transformation. Each of these cases if very similar in concept, but quite different in

affine transformations are a subset of perspective transformations. Figure 6-3 shows examples of various affine and perspective transformations. Affine Transformation There are two situations that arise when working with affine transformations. In the first case, we have an image (or a region of interest) we’d like to transform; in the second case, we have a list of points for which we’d like to compute the result of a transformation. Each of these cases if very similar in concept, but quite different in terms of practical implementation. As a result, OpenCV has two different functions for these situations. cv::warpAffine(), Dense affine transformations In the first case, the obvious input and output formats are images, and the implicit requirement is that the warping assumes the pixels are a dense representation of the underlying image. This means that image warping must necessarily handle interpolations so that the output images are smooth and look natural. The affine transformation function provided by

terms of practical implementation. As a result, OpenCV has two different functions for these situations. cv::warpAffine(), Dense affine transformations In the first case, the obvious input and output formats are images, and the implicit requirement is that the warping assumes the pixels are a dense representation of the underlying image. This means that image warping must necessarily handle interpolations so that the output images are smooth and look natural. The affine transformation function provided by OpenCV for dense transformations is cv::warpAffine(): void cv::warpAffine(   cv::InputArray    src,                       // Input Image   cv::OutputArray   dst,                       // Result image   cv::InputArray    M,                         // 2-by-3 transformation matrix   cv::Size          dsize,                     // Destination image size   int               flags       = cv::INTER_LINEAR,    // Interpolation, and inverse   int               borderMode  = cv::BORDER_CONSTANT, // Pixel

by OpenCV for dense transformations is cv::warpAffine(): void cv::warpAffine(   cv::InputArray    src,                       // Input Image   cv::OutputArray   dst,                       // Result image   cv::InputArray    M,                         // 2-by-3 transformation matrix   cv::Size          dsize,                     // Destination image size   int               flags       = cv::INTER_LINEAR,    // Interpolation, and inverse   int               borderMode  = cv::BORDER_CONSTANT, // Pixel extrapolation method   const cv::Scalar& borderValue = cv::Scalar() // Used for constant borders ); Here src and dst are your source and destination arrays. The input M is the 2-by-3 matrix we introduced earlier that quantifies the desired transformation. Each element in the destination array is computed from the element of the source array at the location given by: 𝑑𝑠𝑡𝑥,𝑦=𝑠𝑟𝑐𝑀!!𝑥+𝑀!"𝑦+𝑀!",𝑀!"𝑥+𝑀!!𝑦+𝑀!" In general, however, the location indicated by the right-hand side of this equation will not be an integer

method   const cv::Scalar& borderValue = cv::Scalar() // Used for constant borders ); Here src and dst are your source and destination arrays. The input M is the 2-by-3 matrix we introduced earlier that quantifies the desired transformation. Each element in the destination array is computed from the element of the source array at the location given by: 𝑑𝑠𝑡𝑥,𝑦=𝑠𝑟𝑐𝑀!!𝑥+𝑀!"𝑦+𝑀!",𝑀!"𝑥+𝑀!!𝑦+𝑀!" In general, however, the location indicated by the right-hand side of this equation will not be an integer pixel. In this case, it is necessary to use interpolation to find an appropriate value for 𝑑𝑠𝑡(𝑥,𝑦). The next argument, flags, selects the interpolation method. The available interpolation methods are those in Table 6-1, the same as cv::resize(), plus one additional option, cv::WARP_INVERSE_MAP (which may be added with the usual Boolean OR). This option is a convenience that allows for inverse warping from dst to src instead of from src to dst. The final two arguments are for border extrapolation, and have the same

this case, it is necessary to use interpolation to find an appropriate value for 𝑑𝑠𝑡(𝑥,𝑦). The next argument, flags, selects the interpolation method. The available interpolation methods are those in Table 6-1, the same as cv::resize(), plus one additional option, cv::WARP_INVERSE_MAP (which may be added with the usual Boolean OR). This option is a convenience that allows for inverse warping from dst to src instead of from src to dst. The final two arguments are for border extrapolation, and have the same meaning as similar arguments in image convolutions (See Chapter 5).                                                              9 “Homography” is the mathematical term for mapping points on one surface to points on another. In this sense, it is a more general term than used here. In the context of computer vision, homography almost always refers to mapping between points on two image planes that correspond to the same location on a planar object in the real world. It can be shown that such a mapping is

meaning as similar arguments in image convolutions (See Chapter 5).                                                              9 “Homography” is the mathematical term for mapping points on one surface to points on another. In this sense, it is a more general term than used here. In the context of computer vision, homography almost always refers to mapping between points on two image planes that correspond to the same location on a planar object in the real world. It can be shown that such a mapping is representable by a single 3-by-3 orthogonal matrix (more on this in Chapter 11).

cv::getAffineTransform(), Computing an Affine Map Matrix OpenCV provides two functions to help you generate the map matrix M. The first is used when you already have two images that you know to be related by an affine transformation or that you’d like to approximate in that way: cv::Mat cv::getAffineTransform(              // Return 2-by-3 matrix   const cv::Point2f* src,                    // Coordinates three of vertices   const cv::Point2f* dst                     // Target coordinates three of vertices ); Here src and dst are arrays containing three two-dimensional (𝑥,𝑦) points. The return value is an array that is the affine transform computed from those points.  The src and dst in cv::getAffineTransform() are just arrays of three points10 defining two parallelograms. The simplest way to define an affine transform is thus to set src to three corners in the source image—for example, the upper- and lower-left together with the upper-right of the source image. The mapping from the source to destination

); Here src and dst are arrays containing three two-dimensional (𝑥,𝑦) points. The return value is an array that is the affine transform computed from those points.  The src and dst in cv::getAffineTransform() are just arrays of three points10 defining two parallelograms. The simplest way to define an affine transform is thus to set src to three corners in the source image—for example, the upper- and lower-left together with the upper-right of the source image. The mapping from the source to destination image is then entirely defined by specifying dst, the locations to which these three points will be mapped in that destination image. Once the mapping of these three independent corners (which, in effect, specify a “representative” parallelogram) is established, all the other points can be warped accordingly. Example 6-1 shows some code that uses these functions. In the example, we obtain the cv::warpAffine() matrix parameters by first constructing two three-component arrays of points (the corners of our

image is then entirely defined by specifying dst, the locations to which these three points will be mapped in that destination image. Once the mapping of these three independent corners (which, in effect, specify a “representative” parallelogram) is established, all the other points can be warped accordingly. Example 6-1 shows some code that uses these functions. In the example, we obtain the cv::warpAffine() matrix parameters by first constructing two three-component arrays of points (the corners of our representative parallelogram) and then convert that to the actual transformation matrix using cv::getAffineTransform(). We then do an affine warp followed by a rotation of the image. For our array of representative points in the source image, called srcTri[], we take the three points: (0,0), (0,height-1), and (width-1,0). We then specify the locations to which these points will be mapped in the corresponding array dstTri[]. Example 6-1: An affine transformation #include <opencv2/opencv.hpp> #include

of our representative parallelogram) and then convert that to the actual transformation matrix using cv::getAffineTransform(). We then do an affine warp followed by a rotation of the image. For our array of representative points in the source image, called srcTri[], we take the three points: (0,0), (0,height-1), and (width-1,0). We then specify the locations to which these points will be mapped in the corresponding array dstTri[]. Example 6-1: An affine transformation #include <opencv2/opencv.hpp> #include <iostream>  using namespace std;  int main(int argc, char** argv) {    if(argc != 2) {      cout << "Warp affine\nUsage: ch6_ex6_2 <imagename>\n" << endl;      return -1;   }       cv::Mat src = cv::imread(argv[1],1);   if( src.empty() ) { cout << "Can not load " << argv[1] << endl; return -1; }        cv::Point3f srcTri[] = {      cv::Point2f(0,0),           // src Top left      cv::Point2f(src.cols-1, 0), // src Top right      cv::Point2f(0, src.rows-1)  // src Bottom left   };       cv::Point2f dstTri[]

<iostream>  using namespace std;  int main(int argc, char** argv) {    if(argc != 2) {      cout << "Warp affine\nUsage: ch6_ex6_2 <imagename>\n" << endl;      return -1;   }       cv::Mat src = cv::imread(argv[1],1);   if( src.empty() ) { cout << "Can not load " << argv[1] << endl; return -1; }        cv::Point3f srcTri[] = {      cv::Point2f(0,0),           // src Top left      cv::Point2f(src.cols-1, 0), // src Top right      cv::Point2f(0, src.rows-1)  // src Bottom left   };       cv::Point2f dstTri[] = {      cv::Point2f(src.cols*0.f,   src.rows*0.33f), // dst Top left      cv::Point2f(src.cols*0.85f, src.rows*0.25f), // dst Top right      cv::Point2f(src.cols*0.15f, src.rows*0.7f)   // dst Bottom left   };                                                                   10 We need just three points because, for an affine transformation, we are only representing a parallelogram. We will need four points to represent a general trapezoid when we address perspective transformations.

// COMPUTE AFFINE MATRIX   //     cv::Mat warp_mat = cv::getAffineTransform(srcTri, dstTri);   cv::Mat dst, dst2;   cv::warpAffine(     src,      dst,        warp_mat,     src.size(),     cv::INTER_LINEAR,      cv::BORDER_CONSTANT,      cv::Scalar()   );   for( int i = 0; i < 3; ++i )     cv::circle(dst, dstTri[i], 5, cv::Scalar(255, 0, 255), -1, cv::AA);       cv::imshow("Affine Transform Test", dst);   cv::waitKey();       for(int frame=0;;++frame) {     // COMPUTE ROTATION MATRIX     cv::Point2f center(src.cols*0.5f, src.rows*0.5f);      double angle = frame*3 % 360, scale = (cos((angle - 60)* cv::PI/180) + 1.05)*0.8;            cv::Mat rot_mat = cv::getRotationMatrix2D(center, angle, scale);      cv::warpAffine(       src,        dst,        rot_mat,        src.size(),       cv::INTER_LINEAR,        cv::BORDER_CONSTANT,        cv::Scalar()     );     cv::imshow("Rotated Image", dst);     if(cv::waitKey(30) >= 0 )       break;   }   return 0; }  The second way to compute the map matrix M is to use

center(src.cols*0.5f, src.rows*0.5f);      double angle = frame*3 % 360, scale = (cos((angle - 60)* cv::PI/180) + 1.05)*0.8;            cv::Mat rot_mat = cv::getRotationMatrix2D(center, angle, scale);      cv::warpAffine(       src,        dst,        rot_mat,        src.size(),       cv::INTER_LINEAR,        cv::BORDER_CONSTANT,        cv::Scalar()     );     cv::imshow("Rotated Image", dst);     if(cv::waitKey(30) >= 0 )       break;   }   return 0; }  The second way to compute the map matrix M is to use cv::getRotationMatrix2D(), which computes the map matrix for a rotation around some arbitrary point, combined with an optional rescaling. This is just one possible kind of affine transformation, but it represents an important subset that has an alternative (and more intuitive) representation that’s easier to work with in your head: cv::Mat cv::getRotationMatrix2D(               // Return 2-by-3 matrix   cv::Point2f  center                          // Center of rotation    double       angle,

cv::getRotationMatrix2D(), which computes the map matrix for a rotation around some arbitrary point, combined with an optional rescaling. This is just one possible kind of affine transformation, but it represents an important subset that has an alternative (and more intuitive) representation that’s easier to work with in your head: cv::Mat cv::getRotationMatrix2D(               // Return 2-by-3 matrix   cv::Point2f  center                          // Center of rotation    double       angle,                          // Angle of rotation   double       scale                           // Rescale after rotation ); The first argument, center, is the center point of the rotation. The next two arguments give the magnitude of the rotation and the overall rescaling. The function returns the map matrix M, which (as always) is a 2-by-3 matrix of floating-point numbers). If we define 𝛼=scale∗cos(angle) and  𝛽=scale∗sin(angle), then this function computes the matrix M to be:

𝛼𝛽1−𝛼∙𝑐𝑒𝑛𝑡𝑒𝑟!−𝛽∙𝑐𝑒𝑛𝑡𝑒𝑟!−𝛽𝛼𝛽∙𝑐𝑒𝑛𝑡𝑒𝑟!−1−𝛼∙𝑐𝑒𝑛𝑡𝑒𝑟! You can combine these methods of setting the map_matrix to obtain, for example, an image that is rotated, scaled, and warped. cv::transform() for Sparse Affine Transformations  We have explained that cv::warpAffine() is the right way to handle dense mappings. For sparse mappings (i.e., mappings of lists of individual points), it is best to use cv::transform(). You will recall from Chapter 3 that the transform method has the following prototype: void cv::transform(   cv::InputArray  src,                         // Input N-by-1 array (Ds channele)   cv::OutputArray dst,                         // Output N-by-1 array (Dd channels)   cv::InputArray  mtx                          // Transfor matrux (Ds-by-Dd) ); In general, src is an 𝑁-by-1 array with 𝐷! channels, where 𝑁 is the number of points to be transformed and 𝐷! is the dimension of those source points. The output array dst will be the same size but may have a different number of channels, 𝐷!. The

cv::transform(   cv::InputArray  src,                         // Input N-by-1 array (Ds channele)   cv::OutputArray dst,                         // Output N-by-1 array (Dd channels)   cv::InputArray  mtx                          // Transfor matrux (Ds-by-Dd) ); In general, src is an 𝑁-by-1 array with 𝐷! channels, where 𝑁 is the number of points to be transformed and 𝐷! is the dimension of those source points. The output array dst will be the same size but may have a different number of channels, 𝐷!. The transformation matrix mtx is a 𝐷!-by-𝐷! matrix that is then applied to every element of src, after which the results are placed into dst. Note that cv::transform() acts on the channel indices of every point in an array. For the current problem, we assume that the array is essentially a large vector (𝑁-by-1 or 1-by-𝑁) of these multichannel objects. The important thing to remember is that the index that the transformation matrix is relative to is the channel index, not the “vector” index of the large array.  In

matrix mtx is a 𝐷!-by-𝐷! matrix that is then applied to every element of src, after which the results are placed into dst. Note that cv::transform() acts on the channel indices of every point in an array. For the current problem, we assume that the array is essentially a large vector (𝑁-by-1 or 1-by-𝑁) of these multichannel objects. The important thing to remember is that the index that the transformation matrix is relative to is the channel index, not the “vector” index of the large array.  In the case of transformations that are simple rotations, our transformation matrix mtx will be a 2-by-2 matrix only, and it can be applied directly to the two-channel indices of src. In fact this is true for rotations, stretch, and warp as well in some simple cases. Usually, however, to do a general affine transformation, including translations and rotations about arbitrary centers, and so on, it is necessary to extend the number of channels in src to three, so that the action of the more usual 2-by-3 affine

the case of transformations that are simple rotations, our transformation matrix mtx will be a 2-by-2 matrix only, and it can be applied directly to the two-channel indices of src. In fact this is true for rotations, stretch, and warp as well in some simple cases. Usually, however, to do a general affine transformation, including translations and rotations about arbitrary centers, and so on, it is necessary to extend the number of channels in src to three, so that the action of the more usual 2-by-3 affine transformation matrix is defined. In this case, all of the third-channel entries must be set to 1.0 (i.e., the points must be supplied in homogeneous coordinates). Of course, the output array will still be a two-channel array. cv::invertAffineTransform(), Inverting an Affine Transformation  Given an affine transformation represented as a 2-by-3 matrix, it is often desirable to be able to compute the inverse transformation, which will “put back” all of the transformed points to where they came from. This is

transformation matrix is defined. In this case, all of the third-channel entries must be set to 1.0 (i.e., the points must be supplied in homogeneous coordinates). Of course, the output array will still be a two-channel array. cv::invertAffineTransform(), Inverting an Affine Transformation  Given an affine transformation represented as a 2-by-3 matrix, it is often desirable to be able to compute the inverse transformation, which will “put back” all of the transformed points to where they came from. This is done with cv::invertAffineTransform(): void cv::invertAffineTransform(   cv::InputArray  M,                           // Input 2-by-3 matrix   cv::OutputArray iM                           // Output also a 2-by-3 matrix ); This function takes a 2-by-3 array M and returns another 2-by-3 array iM that inverts M. Note that cv::invertAffineTransform() does not actually act on any image, it just supplies the inverse transform. Once you have iM, you can use it as you would have used M, with either

from. This is done with cv::invertAffineTransform(): void cv::invertAffineTransform(   cv::InputArray  M,                           // Input 2-by-3 matrix   cv::OutputArray iM                           // Output also a 2-by-3 matrix ); This function takes a 2-by-3 array M and returns another 2-by-3 array iM that inverts M. Note that cv::invertAffineTransform() does not actually act on any image, it just supplies the inverse transform. Once you have iM, you can use it as you would have used M, with either cv::warpAffine() or cv::transform(). Perspective Transformation To gain the greater flexibility offered by perspective transforms (homographies), we need a new function that will allow us to express this broader class of transformations. First we remark that, even though a perspective projection is specified completely by a single matrix, the projection is not actually a linear

transformation. This is because the transformation requires division by the final dimension (usually 𝑍; see Chapter 11) and thus loses a dimension in the process. As with affine transformations, image operations (dense transformations) are handled by different functions than transformations on point sets (sparse transformations). cv::warpPerspective(), Dense perspective transform The dense perspective transform uses an OpenCV function that is analogous to the one provided for dense affine transformations. Specifically, cv::warpPerspective() has all of the same arguments as cv::warpAffine(), except with the small, but crucial, distinction that the map matrix must now be 3-by-3. void cv::warpPerspective(   cv::InputArray    src,                       // Input Image   cv::OutputArray   dst,                       // Result image   cv::InputArray    M,                         // 3-by-3 transformation matrix   cv::Size          dsize,                     // Destination image size   int               flags       =

Specifically, cv::warpPerspective() has all of the same arguments as cv::warpAffine(), except with the small, but crucial, distinction that the map matrix must now be 3-by-3. void cv::warpPerspective(   cv::InputArray    src,                       // Input Image   cv::OutputArray   dst,                       // Result image   cv::InputArray    M,                         // 3-by-3 transformation matrix   cv::Size          dsize,                     // Destination image size   int               flags       = cv::INTER_LINEAR,    // Interpolation, and inverse   int               borderMode  = cv::BORDER_CONSTANT, // Pixel extrapolation method   const cv::Scalar& borderValue = cv::Scalar() // Used for constant borders ); Each element in the destination array is computed from the element of the source array at the location given by: 𝑑𝑠𝑡𝑥,𝑦=𝑠𝑟𝑐𝑀!!𝑥+𝑀!"𝑦+𝑀!"𝑀!"𝑥+𝑀!"𝑦+𝑀!!,𝑀!"𝑥+𝑀!!𝑦+𝑀!"𝑀!"𝑥+𝑀!"𝑦+𝑀!! As with the affine transformation, the location indicated by the right side of this equation will not (generally) be an

cv::INTER_LINEAR,    // Interpolation, and inverse   int               borderMode  = cv::BORDER_CONSTANT, // Pixel extrapolation method   const cv::Scalar& borderValue = cv::Scalar() // Used for constant borders ); Each element in the destination array is computed from the element of the source array at the location given by: 𝑑𝑠𝑡𝑥,𝑦=𝑠𝑟𝑐𝑀!!𝑥+𝑀!"𝑦+𝑀!"𝑀!"𝑥+𝑀!"𝑦+𝑀!!,𝑀!"𝑥+𝑀!!𝑦+𝑀!"𝑀!"𝑥+𝑀!"𝑦+𝑀!! As with the affine transformation, the location indicated by the right side of this equation will not (generally) be an integer location. Again the flags argument is used to select the desired interpolation method, and has the same possible values as the corresponding argument to cv::warpAffine(). cv::getPerspectiveTransform(), Computing the perspective map matrix As with the affine transformation, for filling the map_matrix in the preceding code we have a convenience function that can compute the transformation matrix from a list of point correspondences: cv::Mat cv::getPerspectiveTransform(           // Return 3-by-3

an integer location. Again the flags argument is used to select the desired interpolation method, and has the same possible values as the corresponding argument to cv::warpAffine(). cv::getPerspectiveTransform(), Computing the perspective map matrix As with the affine transformation, for filling the map_matrix in the preceding code we have a convenience function that can compute the transformation matrix from a list of point correspondences: cv::Mat cv::getPerspectiveTransform(           // Return 3-by-3 matrix   const cv::Point2f* src,                      // Coordinates of four vertices   const cv::Point2f* dst                       // Target coordinates of four vertices ); The src and dst argument are now arrays of four (not three) points, so we can independently control how the corners of (typically) a rectangle in src are mapped to (generally) some rhombus in dst. Our transformation is completely defined by the specified destinations of the four source points. As mentioned earlier, for perspective

matrix   const cv::Point2f* src,                      // Coordinates of four vertices   const cv::Point2f* dst                       // Target coordinates of four vertices ); The src and dst argument are now arrays of four (not three) points, so we can independently control how the corners of (typically) a rectangle in src are mapped to (generally) some rhombus in dst. Our transformation is completely defined by the specified destinations of the four source points. As mentioned earlier, for perspective transformations, the return value will be a 3-by-3 array; see Example 6-2 for sample code. Other than the 3-by-3 matrix and the shift from three to four control points, the perspective transformation is otherwise exactly analogous to the affine transformation we already introduced. Example 6-2: Code for perspective transformation #include <opencv2/opencv.hpp> #include <iostream>  using namespace std;  int main(int argc, char** argv) {    if(argc != 2) {      cout << "Perspective Warp\nUsage: ch6_ex6_3

transformations, the return value will be a 3-by-3 array; see Example 6-2 for sample code. Other than the 3-by-3 matrix and the shift from three to four control points, the perspective transformation is otherwise exactly analogous to the affine transformation we already introduced. Example 6-2: Code for perspective transformation #include <opencv2/opencv.hpp> #include <iostream>  using namespace std;  int main(int argc, char** argv) {    if(argc != 2) {      cout << "Perspective Warp\nUsage: ch6_ex6_3 <imagename>\n" << endl;      return -1;   }

Mat src = cv::imread(argv[1],1);   if( src.empty() ) { cout << "Can not load " << argv[1] << endl; return -1; }         cv::Point2f srcQuad[] = {     cv::Point2f(0,          0),          // src Top left     cv::Point2f(src.cols-1, 0),          // src Top right     cv::Point2f(src.cols-1, src.rows-1), // src Bottom right     cv::Point2f(0,          src.rows-1)  // src Bottom left   };        cv::Point2f dstQuad[] = {     cv::Point2f(src.cols*0.05f, src.rows*0.33f),     cv::Point2f(src.cols*0.9f,  src.rows*0.25f),     cv::Point2f(src.cols*0.8f,  src.rows*0.9f),     cv::Point2f(src.cols*0.2f,  src.rows*0.7f)   };        // COMPUTE PERSPECTIVE MATRIX     cv::Mat warp_mat = cv::getPerspectiveTransform(srcQuad, dstQuad);   cv::Mat dst;   cv::warpPerspective(src, dst, warp_mat, src.size(), cv::INTER_LINEAR,                        cv::BORDER_CONSTANT, cv::Scalar());   for( int i = 0; i < 4; i++ )     cv::circle(dst, dstQuad[i], 5, cv::Scalar(255, 0, 255), -1, cv::AA);        cv::imshow("Perspective Transform Test",

cv::Point2f(src.cols*0.8f,  src.rows*0.9f),     cv::Point2f(src.cols*0.2f,  src.rows*0.7f)   };        // COMPUTE PERSPECTIVE MATRIX     cv::Mat warp_mat = cv::getPerspectiveTransform(srcQuad, dstQuad);   cv::Mat dst;   cv::warpPerspective(src, dst, warp_mat, src.size(), cv::INTER_LINEAR,                        cv::BORDER_CONSTANT, cv::Scalar());   for( int i = 0; i < 4; i++ )     cv::circle(dst, dstQuad[i], 5, cv::Scalar(255, 0, 255), -1, cv::AA);        cv::imshow("Perspective Transform Test", dst);   cv::waitKey();   return 0; }  cv::perspectiveTransform(), Sparse perspective transformations There is a special function, cv::perspectiveTransform(), that performs perspective transformations on lists of points. Because cv::transform() is limited to linear operations, it does not conveniently handle perspective transforms. This is because such transformations require division by the third coordinate of the homogeneous representation  (𝑥  =  𝑓  ∗𝑋/𝑍,𝑦  =  𝑓  ∗𝑌/𝑍). The special function

Test", dst);   cv::waitKey();   return 0; }  cv::perspectiveTransform(), Sparse perspective transformations There is a special function, cv::perspectiveTransform(), that performs perspective transformations on lists of points. Because cv::transform() is limited to linear operations, it does not conveniently handle perspective transforms. This is because such transformations require division by the third coordinate of the homogeneous representation  (𝑥  =  𝑓  ∗𝑋/𝑍,𝑦  =  𝑓  ∗𝑌/𝑍). The special function cv::perspectiveTransform() takes care of this for us: void cv::perspectiveTransform(   cv::InputArray  src,                       // Input N-by-1 array (2 or 3 channele)   cv::OutputArray dst,                       // Output N-by-1 array (2 or 3 channels)   cv::InputArray  mtx                        // Transfor matrux (3-by-3 or 4-by-4) ); As usual, the src and dst arguments are, respectively, the array of source points to be transformed and the array of destination points resulting from the transformation. These

takes care of this for us: void cv::perspectiveTransform(   cv::InputArray  src,                       // Input N-by-1 array (2 or 3 channele)   cv::OutputArray dst,                       // Output N-by-1 array (2 or 3 channels)   cv::InputArray  mtx                        // Transfor matrux (3-by-3 or 4-by-4) ); As usual, the src and dst arguments are, respectively, the array of source points to be transformed and the array of destination points resulting from the transformation. These arrays should be two- or three-channel arrays. The matrix mat can be either a 3-by-3 or a 4-by-4 matrix. If it is 3-by-3, then the projection is from two dimensions to two; if the matrix is 4-by-4, then the projection is from three dimensions to three. In the current context, we are transforming a set of points in an image to another set of points in an image, which sounds like a mapping from two dimensions to two dimensions. This is not exactly correct, however, because the perspective transformation is actually mapping

two- or three-channel arrays. The matrix mat can be either a 3-by-3 or a 4-by-4 matrix. If it is 3-by-3, then the projection is from two dimensions to two; if the matrix is 4-by-4, then the projection is from three dimensions to three. In the current context, we are transforming a set of points in an image to another set of points in an image, which sounds like a mapping from two dimensions to two dimensions. This is not exactly correct, however, because the perspective transformation is actually mapping points on a two-dimensional plane embedded in a three-dimensional space back down to a (different) two-dimensional subspace. Think of this as being just what a camera does (we will return to this topic in greater detail when discussing cameras in later chapters). The camera takes points in three dimensions and maps them to the two dimensions of the camera imager. This is essentially what is meant when the source points are taken to be in “homogeneous coordinates.” We are adding a dimension to those points by

on a two-dimensional plane embedded in a three-dimensional space back down to a (different) two-dimensional subspace. Think of this as being just what a camera does (we will return to this topic in greater detail when discussing cameras in later chapters). The camera takes points in three dimensions and maps them to the two dimensions of the camera imager. This is essentially what is meant when the source points are taken to be in “homogeneous coordinates.” We are adding a dimension to those points by introducing the 𝑍 dimension and then setting all of the 𝑍 values to 1. The projective transformation is then projecting back out of that space onto the two-dimensional space of our output. This is a rather long-winded way of explaining why, when mapping points in one image to points in another, you will need a 3-by-3 matrix.

Outputs of the code in Example 6-1 and Example 6-2 are shown in Figure 6-4 for affine and perspective transformations. In these examples, we transform actual images; you can compare these with the simple diagrams of Figure 6-3.    Figure 6-4: Perspective and affine mapping of an image General Remappings The affine and perspective transformations we have seen so far are actually specific cases of a more general process. Under the hood, those two both have the same basic behavior. They take pixels from one place in the source image and map them to another place in the destination image. In fact, there are other useful operations that have the same structure. In this section, we will look at another few transformations of this kind, and then look at how OpenCV makes it possible to implement your own general mapping transformations. Polar Mappings In Chapter 3, we briefly encountered two functions, cv::cartToPolar() and cv::polarToCart(), which could be used to convert arrays of points in an 𝑥-𝑦 Cartesian

one place in the source image and map them to another place in the destination image. In fact, there are other useful operations that have the same structure. In this section, we will look at another few transformations of this kind, and then look at how OpenCV makes it possible to implement your own general mapping transformations. Polar Mappings In Chapter 3, we briefly encountered two functions, cv::cartToPolar() and cv::polarToCart(), which could be used to convert arrays of points in an 𝑥-𝑦 Cartesian representation to (or from) arrays of points in an 𝑟-𝜃 polar representation.  There is a slight style inconsistency here between the polar mapping functions and the perspective and affine transformation functions. The polar mapping functions expect pairs of single-channel arrays, rather than double-channel arrays as their way of representing two-dimensional vectors. This difference has its origin in the way the two functions are traditionally used, rather than any intrinsic difference between what they are

representation to (or from) arrays of points in an 𝑟-𝜃 polar representation.  There is a slight style inconsistency here between the polar mapping functions and the perspective and affine transformation functions. The polar mapping functions expect pairs of single-channel arrays, rather than double-channel arrays as their way of representing two-dimensional vectors. This difference has its origin in the way the two functions are traditionally used, rather than any intrinsic difference between what they are doing.

The functions cv::cartToPolar() and cv::polarToCart() are employed by more complex routines such as cv::logPolar() (described later) but are also useful in their own right. cv::cartToPolar(), Converting from Cartesian to Polar Coordinates For the case of mapping from Cartesian coordinates to polar coordinates, we have the function cv::cartToPolar(): void cv::cartToPolar(   cv::InputArray  x,                           // Input single channel x-array    cv::InputArray  y,                           // Input single channel y-array   cv::OutputArray magnitude,                   // Output single channel mag-array   cv::OutputArray angle,                       // Output single channel angle-array   bool            angleInDegrees = false       // Set true for degrees, else radians ); The first two arguments x, and y, are single-channel arrays. Conceptually, what is being represented here is not just a list of points, but a vector field11—with the x-component of the vector field at any given point being represented by

channel y-array   cv::OutputArray magnitude,                   // Output single channel mag-array   cv::OutputArray angle,                       // Output single channel angle-array   bool            angleInDegrees = false       // Set true for degrees, else radians ); The first two arguments x, and y, are single-channel arrays. Conceptually, what is being represented here is not just a list of points, but a vector field11—with the x-component of the vector field at any given point being represented by the value of the array x at that point, and the y-component of the vector field at any given point being represented by the value of the array y at that point. Similarly, the result of this function appears in the arrays magnitude and angle, with each point in magnitude representing the length of the vector at that point in x and y, and each point in angle representing the orientation of that vector. The angles recorded in angle will, by default, be in radians, i.e.,  [0,2𝜋). If the argument angleInDegrees is

value of the array x at that point, and the y-component of the vector field at any given point being represented by the value of the array y at that point. Similarly, the result of this function appears in the arrays magnitude and angle, with each point in magnitude representing the length of the vector at that point in x and y, and each point in angle representing the orientation of that vector. The angles recorded in angle will, by default, be in radians, i.e.,  [0,2𝜋). If the argument angleInDegrees is set to true, however, then the angles array will be recorded in degrees [0,360). As an example of where you might use this function, suppose you have already taken the x- and y-derivatives of an image, either by using cv::Sobel() or by using convolution functions via cv::DFT() or cv::filter2D(). If you stored the x-derivatives in an image dx_img and the y-derivatives in dy_img, you could now create an edge-angle recognition histogram. That is, you could then collect all the angles provided the magnitude or

set to true, however, then the angles array will be recorded in degrees [0,360). As an example of where you might use this function, suppose you have already taken the x- and y-derivatives of an image, either by using cv::Sobel() or by using convolution functions via cv::DFT() or cv::filter2D(). If you stored the x-derivatives in an image dx_img and the y-derivatives in dy_img, you could now create an edge-angle recognition histogram. That is, you could then collect all the angles provided the magnitude or strength of the edge pixel is above some desired threshold. To calculate this, we would first create two new destination images (and call them img_mag and img_angle, for example) for the directional derivatives and then use the function cvCartToPolar( dx_img, dy_img, img_mag, img_angle, 1 ). We would then fill a histogram from img_angle as long as the corresponding “pixel” in img_mag is above our desired threshold. In Chapter 13, we will discuss image recognition and image features. This process is

or strength of the edge pixel is above some desired threshold. To calculate this, we would first create two new destination images (and call them img_mag and img_angle, for example) for the directional derivatives and then use the function cvCartToPolar( dx_img, dy_img, img_mag, img_angle, 1 ). We would then fill a histogram from img_angle as long as the corresponding “pixel” in img_mag is above our desired threshold. In Chapter 13, we will discuss image recognition and image features. This process is actually the basis of how an important image feature used in object recognition, called HOG (histogram of oriented gradients), is calculated. cv::polarToCart(), Converting from Polar to Cartesian Coordinates The function cv::cartToPolar() performs the reverse mapping from polar coordinates to Cartesian coordinates. void cv::polarToCart(   cv::InputArray  magnitude,                   // Output single channel mag-array   cv::InputArray  angle,                       // Output single channel angle-array

is actually the basis of how an important image feature used in object recognition, called HOG (histogram of oriented gradients), is calculated. cv::polarToCart(), Converting from Polar to Cartesian Coordinates The function cv::cartToPolar() performs the reverse mapping from polar coordinates to Cartesian coordinates. void cv::polarToCart(   cv::InputArray  magnitude,                   // Output single channel mag-array   cv::InputArray  angle,                       // Output single channel angle-array   cv::OutputArray x,                           // Input single channel x-array    cv::OutputArray y,                           // Input single channel y-array   bool            angleInDegrees = false       // Set true for degrees, else radians ); The inverse operation is also often useful, allowing us to convert from polar back to Cartesian coordinates. It takes essentially the same arguments as cv::cartToPolar(), with the exception that magnitude and angle are now inputs, and x and y are now the results.

x,                           // Input single channel x-array    cv::OutputArray y,                           // Input single channel y-array   bool            angleInDegrees = false       // Set true for degrees, else radians ); The inverse operation is also often useful, allowing us to convert from polar back to Cartesian coordinates. It takes essentially the same arguments as cv::cartToPolar(), with the exception that magnitude and angle are now inputs, and x and y are now the results.                                                              11 If you are not familiar with the concept of a vector field, it is sufficient for our purposes to just think of this as a two component vector associated with every point in “image.”

LogPolar For two-dimensional images, the log-polar transform [Schwartz80] is a change from Cartesian to log-polar coordinates:  (𝑥,𝑦)⟷𝑟𝑒!", where 𝑟=𝑥!+𝑦! and  𝜃=𝑎𝑡𝑎𝑛2𝑦,𝑥. Next, to separate out the polar coordinates into a (𝜌,𝜃) space that is relative to some center point (𝑥𝑐,𝑦𝑐); we take the log so that 𝜌=log  ((𝑥−𝑥!)!+(𝑦−𝑦!)!) and  𝜃=𝑎𝑡𝑎𝑛2𝑦−𝑦!,𝑥−𝑥!. For image purposes—when we need to “fit” the interesting stuff into the available image memory—we typically apply a scaling factor 𝑚 to  𝜌. Figure 6-5 shows a square object on the left and its encoding in log-polar space.  Figure 6-5: The log-polar transform maps (𝑥,𝑦) into (𝑙𝑜𝑔(𝑟),𝜃). Here, a square is displayed in the log-polar coordinate system The next question is, of course, “Why bother?” The log-polar transform takes its inspiration from the human visual system. Your eye has a small but dense center of photoreceptors in its center (the fovea), and the density of receptors falls off rapidly (exponentially) from there. Try staring at a spot on the wall and

a square object on the left and its encoding in log-polar space.  Figure 6-5: The log-polar transform maps (𝑥,𝑦) into (𝑙𝑜𝑔(𝑟),𝜃). Here, a square is displayed in the log-polar coordinate system The next question is, of course, “Why bother?” The log-polar transform takes its inspiration from the human visual system. Your eye has a small but dense center of photoreceptors in its center (the fovea), and the density of receptors falls off rapidly (exponentially) from there. Try staring at a spot on the wall and holding your finger at arm’s length in your line of sight. Then, keep staring at the spot and move your finger slowly away; note how the detail rapidly decreases as the image of your finger moves away from your fovea. This structure also has certain nice mathematical properties (beyond the scope of this book) that concern preserving the angles of line intersections. More important for us is that the log-polar transform can be used to create two-dimensional invariant representations of object views by

and holding your finger at arm’s length in your line of sight. Then, keep staring at the spot and move your finger slowly away; note how the detail rapidly decreases as the image of your finger moves away from your fovea. This structure also has certain nice mathematical properties (beyond the scope of this book) that concern preserving the angles of line intersections. More important for us is that the log-polar transform can be used to create two-dimensional invariant representations of object views by shifting the transformed image’s center of mass to a fixed point in the log-polar plane; see

Figure 6-6. On the left are three shapes that we want to recognize as “square.” The problem is, they look very different. One is much larger than the others and another is rotated. The log-polar transform appears on the right in   Figure 6-6. Observe that size differences in the (𝑥,𝑦) plane are converted to shifts along the log  (𝑟) axis of the log-polar plane and that the rotation differences are converted to shifts along the 𝜃-axis in the log-polar plane. If we take the transformed center of each transformed square in the log-polar plane and then re-center that point to a certain fixed position, then all the squares will show up identically in the log-polar plane. This yields a type of invariance to two-dimensional rotation and scaling.12   Figure 6-6: Log-polar transform of rotated and scaled squares: size goes to a shift on the 𝑙𝑜𝑔𝑟 axis and rotation to a shift on the 𝜃-axis cv::logPolar() The OpenCV function for a log-polar transform is cv::logPolar(): void cv::logPolar(   cv::InputArray  src,

square in the log-polar plane and then re-center that point to a certain fixed position, then all the squares will show up identically in the log-polar plane. This yields a type of invariance to two-dimensional rotation and scaling.12   Figure 6-6: Log-polar transform of rotated and scaled squares: size goes to a shift on the 𝑙𝑜𝑔𝑟 axis and rotation to a shift on the 𝜃-axis cv::logPolar() The OpenCV function for a log-polar transform is cv::logPolar(): void cv::logPolar(   cv::InputArray  src,                           // Input image   cv::OutputArray dst,                           // Output image   cv::Point2f     center,                        // Center of transform   double          m,                             // Scale factor   int             flags = cv::INTER_LINEAR                               | cv::WARP_FILL_OUTLIERS // interpolation and fill modes );                                                              12 In Chapter 13, we‘ll learn about recognition. For now, simply note that it wouldn‘t

// Input image   cv::OutputArray dst,                           // Output image   cv::Point2f     center,                        // Center of transform   double          m,                             // Scale factor   int             flags = cv::INTER_LINEAR                               | cv::WARP_FILL_OUTLIERS // interpolation and fill modes );                                                              12 In Chapter 13, we‘ll learn about recognition. For now, simply note that it wouldn‘t be a good idea to derive a log-polar transform for a whole object because such transforms are quite sensitive to the exact location of their center points. What is more likely to work for object recognition is to detect a collection of key points (such as corners or blob locations) around an object, truncate the extent of such views, and then use the centers of those key points as log-polar centers. These local log-polar transforms could then be used to create local features that are (partially) scale- and

be a good idea to derive a log-polar transform for a whole object because such transforms are quite sensitive to the exact location of their center points. What is more likely to work for object recognition is to detect a collection of key points (such as corners or blob locations) around an object, truncate the extent of such views, and then use the centers of those key points as log-polar centers. These local log-polar transforms could then be used to create local features that are (partially) scale- and rotation-invariant and that can be associated with a visual object.

The src and dst are the usual input and output images. The parameter center is the center point (𝑥!,𝑦!) of the log-polar transform; m is the scale factor, which should be set so that the features of interest dominate the available image area. The flags parameter allows for different interpolation methods. The interpolation methods are the same set of standard interpolations available in OpenCV (Table 6-1). The interpolation methods can be combined with either or both of the flags cv::WARP_FILL_OUTLIERS (to fill points that would otherwise be undefined) or cv::WARP_INVERSE_MAP (to compute the reverse mapping from log-polar to Cartesian coordinates). Sample log-polar coding is given in Example 6-4, which demonstrates the forward and backward (inverse) log-polar transform. The results on a photographic image are shown in Figure 6-7.  Figure 6-7: Log-polar example on an elk with transform centered at the white circle on the left; the output is on the right Example 6-3: Log-polar transform example #include

(to fill points that would otherwise be undefined) or cv::WARP_INVERSE_MAP (to compute the reverse mapping from log-polar to Cartesian coordinates). Sample log-polar coding is given in Example 6-4, which demonstrates the forward and backward (inverse) log-polar transform. The results on a photographic image are shown in Figure 6-7.  Figure 6-7: Log-polar example on an elk with transform centered at the white circle on the left; the output is on the right Example 6-3: Log-polar transform example #include <opencv2/opencv.hpp> #include <iostream>  using namespace std;  int main(int argc, char** argv) {    if(argc != 3) {      cout << "LogPolar\nUsage: ch6_ex6_4 <imagename> <M value>\n”       <<”<M value>~30 is usually good enough\n";      return -1;    }        cv::Mat src = cv::imread(argv[1],1);        if( src.empty() ) { cout << "Can not load " << argv[1] << endl; return -1; }         double M = atof(argv[2]);   cv::Mat dst(src.size(), src.type()), src2(src.size(), src.type());        cv::logPolar(      src,

#include <iostream>  using namespace std;  int main(int argc, char** argv) {    if(argc != 3) {      cout << "LogPolar\nUsage: ch6_ex6_4 <imagename> <M value>\n”       <<”<M value>~30 is usually good enough\n";      return -1;    }        cv::Mat src = cv::imread(argv[1],1);        if( src.empty() ) { cout << "Can not load " << argv[1] << endl; return -1; }         double M = atof(argv[2]);   cv::Mat dst(src.size(), src.type()), src2(src.size(), src.type());        cv::logPolar(      src,       dst,      cv::Point2f(src.cols*0.5f, src.rows*0.5f),     M,       cv::INTER_LINEAR | cv::WARP_FILL_OUTLIERS    );   cv::logPolar(      dst,      src2,      cv::Point2f(src.cols*0.5f, src.rows*0.5f),     M,

cv::INTER_LINEAR | cv::WARP_INVERSE_MAP    );   cv::imshow( "log-polar", dst );   cv::imshow( "inverse log-polar", src2 );   cv::waitKey();   return 0; } Arbitrary Mappings We sometimes want to accomplish interpolation programmatically; that is, we’d like to apply some known algorithm that will determine the mapping. In other cases, however, we’d like to do this mapping ourselves. Before diving into some methods that will compute (and apply) these mappings for us, let’s take a moment to look at the function responsible for applying the mappings that these other methods rely upon.  One common use of cv::remap() is to rectify (correct distortions in) calibrated and stereo images. We will see functions in Chapters 11 and 12 that convert calculated camera distortions and alignments into mapx and mapy parameters.  The OpenCV function we want is called cv:remap(): cv::remap(), General Image Remapping void cv::remap(    cv::InputArray    src,                        // Input image    cv::OutputArray   dst,

responsible for applying the mappings that these other methods rely upon.  One common use of cv::remap() is to rectify (correct distortions in) calibrated and stereo images. We will see functions in Chapters 11 and 12 that convert calculated camera distortions and alignments into mapx and mapy parameters.  The OpenCV function we want is called cv:remap(): cv::remap(), General Image Remapping void cv::remap(    cv::InputArray    src,                        // Input image    cv::OutputArray   dst,                        // Output image   cv::InputArray    map1,                       // target x-location for src pixels   cv::InputArray    map2,                       // target y-location for src pixels   int               interpolation = cv::INTER_LINEAR,    // Interpolation, and inverse   int               borderMode    = cv::BORDER_CONSTANT, // Pixel extrapolation method   const cv::Scalar& borderValue   = cv::Scalar()         // Used for constant borders ); The first two arguments of cv::remap() are the

// Output image   cv::InputArray    map1,                       // target x-location for src pixels   cv::InputArray    map2,                       // target y-location for src pixels   int               interpolation = cv::INTER_LINEAR,    // Interpolation, and inverse   int               borderMode    = cv::BORDER_CONSTANT, // Pixel extrapolation method   const cv::Scalar& borderValue   = cv::Scalar()         // Used for constant borders ); The first two arguments of cv::remap() are the source and destination images, respectively. The next two arguments, map1 and map2, indicate where any particular pixel is to be relocated. This is how you specify your own general mapping. These should be the same size as the source and destination images, and must be one of the following data types: cv::S16C2, cv::F32C1, or cv::F32C2. Non-integer mappings are allowed: cv::remap() will do the interpolation calculations for you automatically.  The next argument, interpolation, contains flags that tell

the source and destination images, respectively. The next two arguments, map1 and map2, indicate where any particular pixel is to be relocated. This is how you specify your own general mapping. These should be the same size as the source and destination images, and must be one of the following data types: cv::S16C2, cv::F32C1, or cv::F32C2. Non-integer mappings are allowed: cv::remap() will do the interpolation calculations for you automatically.  The next argument, interpolation, contains flags that tell cv::remap() exactly how that interpolation is to be done. Any one of the values listed in Table 6-1 will work – except for cv::INTER_AREA, which is not implemented for cv::remap().  Discrete Fourier Transform For any set of values that are indexed by a discrete (integer) parameter, it is possible to define a discrete Fourier transform (DFT)13 in a manner analogous to the Fourier transform of a continuous function. For 𝑁 complex numbers 𝑥!,𝑥!,𝑥!,…,𝑥!!!, the one-dimensional DFT is defined by the following

cv::remap() exactly how that interpolation is to be done. Any one of the values listed in Table 6-1 will work – except for cv::INTER_AREA, which is not implemented for cv::remap().  Discrete Fourier Transform For any set of values that are indexed by a discrete (integer) parameter, it is possible to define a discrete Fourier transform (DFT)13 in a manner analogous to the Fourier transform of a continuous function. For 𝑁 complex numbers 𝑥!,𝑥!,𝑥!,…,𝑥!!!, the one-dimensional DFT is defined by the following formula (where 𝑖=−1): 𝑔!=𝑓!𝑒!!!"!!"!!!!!!                                                              13 Joseph Fourier [Fourier] was the first to find that some functions can be decomposed into an infinite series of other functions, and doing so became a field known as Fourier analysis. Some key text on methods of decomposing functions into their Fourier series are Morse for physics [Morse53] and Papoulis in general [Papoulis62]. The fast Fourier transform was invented by Cooley and Tukey in 1965 [Cooley65]

(where 𝑖=−1): 𝑔!=𝑓!𝑒!!!"!!"!!!!!!                                                              13 Joseph Fourier [Fourier] was the first to find that some functions can be decomposed into an infinite series of other functions, and doing so became a field known as Fourier analysis. Some key text on methods of decomposing functions into their Fourier series are Morse for physics [Morse53] and Papoulis in general [Papoulis62]. The fast Fourier transform was invented by Cooley and Tukey in 1965 [Cooley65] though Carl Gauss worked out the key steps as early as 1805 [Johnson84]. Early use in computer vision is described by Ballard and Brown [Ballard82].

A similar transform can be defined for a two-dimensional array of numbers (of course higher-dimensional analogues exist also): 𝑔!!,!!=𝑓!!,!!𝑒!!!"!(!!!!!!!!!)!!!!!!!!!!!!!!!! In general, one might expect that the computation of the 𝑁 different terms 𝑔𝑘 would require 𝑂(𝑁2) operations. In fact, there are several fast Fourier transform (FFT) algorithms capable of computing these values in 𝑂(𝑁  log  𝑁) time.  cv::dft(), the Discrete Fourier Transform The OpenCV function cv::dft() implements one such FFT algorithm. The function cv::dft() can compute FFTs for one- and two-dimensional arrays of inputs. In the latter case, the two-dimensional transform can be computed or, if desired, only the one-dimensional transforms of each individual row can be computed (this operation is much faster than calling cv::dft() several times): void cv::dft(   cv::InputArray    src,                        // Input array (real or complex)    cv::OutputArray   dst,                        // Output array   int               flags       =

The function cv::dft() can compute FFTs for one- and two-dimensional arrays of inputs. In the latter case, the two-dimensional transform can be computed or, if desired, only the one-dimensional transforms of each individual row can be computed (this operation is much faster than calling cv::dft() several times): void cv::dft(   cv::InputArray    src,                        // Input array (real or complex)    cv::OutputArray   dst,                        // Output array   int               flags       = 0,            // for inverse, or other variations   int               nonzeroRows = 0             // number of rows to not ignore ); The input array must be of floating-point type and may be single- or double-channel. In the single-channel case, the entries are assumed to be real numbers and the output will be packed in a special space-saving format called CCS or Complex Conjugate Symmetrical.14 If the source and channel are two-channel matrices or images, then the two channels will be interpreted as the real

// for inverse, or other variations   int               nonzeroRows = 0             // number of rows to not ignore ); The input array must be of floating-point type and may be single- or double-channel. In the single-channel case, the entries are assumed to be real numbers and the output will be packed in a special space-saving format called CCS or Complex Conjugate Symmetrical.14 If the source and channel are two-channel matrices or images, then the two channels will be interpreted as the real and imaginary components of the input data. In this case, there will be no special packing of the results, and some space will be wasted with a lot of 0s in both the input and output arrays.15 The special packing of result values that is used with single-channel CCS output is as follows. For a one-dimensional array: 𝑅𝑒  𝑌! 𝑅𝑒  𝑌! 𝐼𝑚  𝑌! 𝑅𝑒  𝑌! 𝐼𝑚  𝑌! ⋯ 𝑅𝑒  𝑌!!!! 𝐼𝑚  𝑌!!!! 𝑅𝑒  𝑌!! For a two-dimensional array: 𝑅𝑒  𝑌!! 𝑅𝑒  𝑌!" 𝐼𝑚  𝑌!" 𝑅𝑒  𝑌!" 𝐼𝑚  𝑌!" ⋯ 𝑅𝑒  𝑌!,!!!!! 𝐼𝑚  𝑌!,!!!!! 𝑅𝑒  𝑌!,!!! 𝑅𝑒  𝑌!" 𝑅𝑒  𝑌!! 𝐼𝑚

and imaginary components of the input data. In this case, there will be no special packing of the results, and some space will be wasted with a lot of 0s in both the input and output arrays.15 The special packing of result values that is used with single-channel CCS output is as follows. For a one-dimensional array: 𝑅𝑒  𝑌! 𝑅𝑒  𝑌! 𝐼𝑚  𝑌! 𝑅𝑒  𝑌! 𝐼𝑚  𝑌! ⋯ 𝑅𝑒  𝑌!!!! 𝐼𝑚  𝑌!!!! 𝑅𝑒  𝑌!! For a two-dimensional array: 𝑅𝑒  𝑌!! 𝑅𝑒  𝑌!" 𝐼𝑚  𝑌!" 𝑅𝑒  𝑌!" 𝐼𝑚  𝑌!" ⋯ 𝑅𝑒  𝑌!,!!!!! 𝐼𝑚  𝑌!,!!!!! 𝑅𝑒  𝑌!,!!! 𝑅𝑒  𝑌!" 𝑅𝑒  𝑌!! 𝐼𝑚  𝑌!! 𝑅𝑒  𝑌!" 𝐼𝑚  𝑌!" ⋯ 𝑅𝑒  𝑌!,!!!!! 𝐼𝑚  𝑌!,!!!!! 𝑅𝑒  𝑌!,!!! 𝑅𝑒  𝑌!" 𝑅𝑒  𝑌!" 𝐼𝑚  𝑌!" 𝑅𝑒  𝑌!! 𝐼𝑚  𝑌!! ⋯ 𝑅𝑒  𝑌!,!!!!! 𝐼𝑚  𝑌!,!!!!! 𝑅𝑒  𝑌!,!!! ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ 𝑅𝑒  𝑌!!!!!,! 𝑅𝑒  𝑌!!!!,! 𝐼𝑚  𝑌!!!!,! 𝑅𝑒  𝑌!!!!,! 𝐼𝑚  𝑌!!!!,! ⋯ 𝑅𝑒  𝑌!!!!,!!!!! 𝐼𝑚  𝑌!!!!,!!!!! 𝑅𝑒  𝑌!!!!,!!!                                                              14 As a result of this compact representation, the size of the output array for a single-channel image is the same as the size of the input array because the elements that are provably

𝑌!! 𝑅𝑒  𝑌!" 𝐼𝑚  𝑌!" ⋯ 𝑅𝑒  𝑌!,!!!!! 𝐼𝑚  𝑌!,!!!!! 𝑅𝑒  𝑌!,!!! 𝑅𝑒  𝑌!" 𝑅𝑒  𝑌!" 𝐼𝑚  𝑌!" 𝑅𝑒  𝑌!! 𝐼𝑚  𝑌!! ⋯ 𝑅𝑒  𝑌!,!!!!! 𝐼𝑚  𝑌!,!!!!! 𝑅𝑒  𝑌!,!!! ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ 𝑅𝑒  𝑌!!!!!,! 𝑅𝑒  𝑌!!!!,! 𝐼𝑚  𝑌!!!!,! 𝑅𝑒  𝑌!!!!,! 𝐼𝑚  𝑌!!!!,! ⋯ 𝑅𝑒  𝑌!!!!,!!!!! 𝐼𝑚  𝑌!!!!,!!!!! 𝑅𝑒  𝑌!!!!,!!!                                                              14 As a result of this compact representation, the size of the output array for a single-channel image is the same as the size of the input array because the elements that are provably zero are omitted. In the case of the two-channel (complex) array, the output size will, of course, also be equal to the input size. 15 When using this method, you must be sure to explicitly set the imaginary components to 0 in the two-channel representation. An easy way to do this is to create a matrix full of 0s using cv::Mat::zeros() for the imaginary part and then to call cv::merge() with a real-valued matrix to form a temporary complex array on which to run cv::dft() (possibly in-place). This procedure

zero are omitted. In the case of the two-channel (complex) array, the output size will, of course, also be equal to the input size. 15 When using this method, you must be sure to explicitly set the imaginary components to 0 in the two-channel representation. An easy way to do this is to create a matrix full of 0s using cv::Mat::zeros() for the imaginary part and then to call cv::merge() with a real-valued matrix to form a temporary complex array on which to run cv::dft() (possibly in-place). This procedure will result in full-size, unpacked, complex matrix of the spectrum.

𝐼𝑚  𝑌!!!!!,! 𝑅𝑒  𝑌!!!!,! 𝐼𝑚  𝑌!!!!,! 𝑅𝑒  𝑌!!!!,! 𝐼𝑚  𝑌!!!!,! ⋯ 𝑅𝑒  𝑌!!!!,!!!!! 𝐼𝑚  𝑌!!!!,!!!!! 𝑅𝑒  𝑌!!!!,!!! 𝑅𝑒  𝑌!!!,! 𝑅𝑒  𝑌!!!!,! 𝐼𝑚  𝑌!!!!,! 𝑅𝑒  𝑌!!!!,! 𝐼𝑚  𝑌!!!!,! ⋯ 𝑅𝑒  𝑌!!!!,!!!!! 𝐼𝑚  𝑌!!!!,!!!!! 𝑅𝑒  𝑌!!!!,!!! It is worth taking a moment to look closely at the indices of these arrays. Certain values in the array are guaranteed to be 0 (more accurately, certain values of 𝑓𝑘 are guaranteed to be real). It should also be noted that the last row listed in the table will be present only if 𝑁! is even and that the last column will be present only if 𝑁! is even. In the case of the two-dimensional array being treated as 𝑁! separate one-dimensional arrays rather than a full two-dimensional transform (we’ll take a look at how to do this), all of the result rows will be analogous to the single row listed for the output of the one-dimensional array. The third argument, called flags, indicates exactly what operation is to be done. As usual, flags is treated as a bit array, so you can combine any flags you need with

the last column will be present only if 𝑁! is even. In the case of the two-dimensional array being treated as 𝑁! separate one-dimensional arrays rather than a full two-dimensional transform (we’ll take a look at how to do this), all of the result rows will be analogous to the single row listed for the output of the one-dimensional array. The third argument, called flags, indicates exactly what operation is to be done. As usual, flags is treated as a bit array, so you can combine any flags you need with Boolean OR. The transformation we started with is known as a forward transform and is selected by default. The inverse transform16 is defined in exactly the same way except for a change of sign in the exponential and a scale factor. To perform the inverse transform without the scale factor, use the flag cv::DFT_INVERSE. The flag for the scale factor is cv::DFT_SCALE, which results in all of the output being scaled by a factor of 𝑁!! (or 𝑁!𝑁!!! for a two-dimensional transform). This scaling is necessary if the

OR. The transformation we started with is known as a forward transform and is selected by default. The inverse transform16 is defined in exactly the same way except for a change of sign in the exponential and a scale factor. To perform the inverse transform without the scale factor, use the flag cv::DFT_INVERSE. The flag for the scale factor is cv::DFT_SCALE, which results in all of the output being scaled by a factor of 𝑁!! (or 𝑁!𝑁!!! for a two-dimensional transform). This scaling is necessary if the sequential application of the forward transform and the inverse transform is to bring us back to where we started. Because one often wants to combine cv::DFT_INVERSE with cv::DFT_SCALE, there are several shorthand notations for this kind of operation. In addition to just combining the two operations, you can use cv::DFT_INV_SCALE (or cv::DFT_INVERSE_SCALE if you’re not into that brevity thing). The last flag you may want to have handy is cv::DFT_ROWS, which allows you to tell cv::dft() to treat a

if the sequential application of the forward transform and the inverse transform is to bring us back to where we started. Because one often wants to combine cv::DFT_INVERSE with cv::DFT_SCALE, there are several shorthand notations for this kind of operation. In addition to just combining the two operations, you can use cv::DFT_INV_SCALE (or cv::DFT_INVERSE_SCALE if you’re not into that brevity thing). The last flag you may want to have handy is cv::DFT_ROWS, which allows you to tell cv::dft() to treat a two-dimensional array as a collection of one-dimensional arrays that should each be transformed separately as if they were 𝑁!  distinct vectors of length 𝑁!. This can significantly reduce overhead when doing many transformations at a time. By using cv::DFT_ROWS it is also possible to implement three-dimensional (and higher) DFT. Though the default behavior of the forward transform is to produce results in CCS format (which results in an output array exactly the same size as the input array), you can

a two-dimensional array as a collection of one-dimensional arrays that should each be transformed separately as if they were 𝑁!  distinct vectors of length 𝑁!. This can significantly reduce overhead when doing many transformations at a time. By using cv::DFT_ROWS it is also possible to implement three-dimensional (and higher) DFT. Though the default behavior of the forward transform is to produce results in CCS format (which results in an output array exactly the same size as the input array), you can explicitly ask OpenCV to not do this with the flag cv::DFT_COMPLEX_OUTPUT. The result will be the full complex array (with all of the zeros in it). Conversely, when performing an inverse transformation on a complex array, the result is normally also a complex array. If the source array had complex conjugate symmetry,17 you can ask OpenCV to produce a purely real array (which will be smaller than the input array) by passing the cv::DFT_REAL_OUTPUT flag.  In order to understand the last argument, nonzero_rows, we

ask OpenCV to not do this with the flag cv::DFT_COMPLEX_OUTPUT. The result will be the full complex array (with all of the zeros in it). Conversely, when performing an inverse transformation on a complex array, the result is normally also a complex array. If the source array had complex conjugate symmetry,17 you can ask OpenCV to produce a purely real array (which will be smaller than the input array) by passing the cv::DFT_REAL_OUTPUT flag.  In order to understand the last argument, nonzero_rows, we must digress for a moment to explain that in general, DFT algorithms strongly prefer input vectors of some lengths over input vectors of other lengths; similarly for arrays of some sizes over arrays of other sizes. In most DFT algorithms, the preferred sizes are powers of 2 (i.e., 2! for some integer 𝑛). In the case of the algorithm used by OpenCV, the preference is that the vector lengths, or array dimensions, be 2!3!5!, for some integers 𝑝, 𝑞, and 𝑟. Hence the usual procedure is to create a somewhat larger

digress for a moment to explain that in general, DFT algorithms strongly prefer input vectors of some lengths over input vectors of other lengths; similarly for arrays of some sizes over arrays of other sizes. In most DFT algorithms, the preferred sizes are powers of 2 (i.e., 2! for some integer 𝑛). In the case of the algorithm used by OpenCV, the preference is that the vector lengths, or array dimensions, be 2!3!5!, for some integers 𝑝, 𝑞, and 𝑟. Hence the usual procedure is to create a somewhat larger array and then to copy your array into that somewhat roomier zero-padded array. For convenience, there is a handy utility function, cv::getOptimalDFTSize(), which takes the (integer) length of your vector and returns the first equal or larger size that can be expressed in the form given (i.e., 2!3!5!). Despite the need for this padding, it is possible to indicate to cv::dft() that you really do not care about the transform of those rows that you had to add down below your actual data (or, if you are doing an

and then to copy your array into that somewhat roomier zero-padded array. For convenience, there is a handy utility function, cv::getOptimalDFTSize(), which takes the (integer) length of your vector and returns the first equal or larger size that can be expressed in the form given (i.e., 2!3!5!). Despite the need for this padding, it is possible to indicate to cv::dft() that you really do not care about the transform of those rows that you had to add down below your actual data (or, if you are doing an inverse transform, which                                                              16 With the inverse transform, the input is packed in the special format described previously. This makes sense because, if we first called the forward DFT and then ran the inverse DFT on the results, we would expect to wind up with the original data—that is, of course, if we remember to use the cv::DFT_SCALE flag! 17 This is not to say that it is in CCS format, only that it possesses the symmetry, as it would if (for

an inverse transform, which                                                              16 With the inverse transform, the input is packed in the special format described previously. This makes sense because, if we first called the forward DFT and then ran the inverse DFT on the results, we would expect to wind up with the original data—that is, of course, if we remember to use the cv::DFT_SCALE flag! 17 This is not to say that it is in CCS format, only that it possesses the symmetry, as it would if (for example) it was the result of a forward transform of a purely real array in the first place. Also, take note that you are telling OpenCV that the input array has this symmetry—it will trust you. It does not actually check to verify that this symmetry is present.

rows in the result you do not care about). In either case, you can use nonzero_rows to indicate how many rows contain meaningful data. This will provide some savings in computation time. cv::idft(), the Inverse Discrete Fourier Transform As we saw earlier, the function cv::dft() can be made to implement not only the discrete Fourier transform, but also the inverse operation (with the provision of the correct flags argument). It is often preferable, if only for code readability, to have a separate function that does this inverse operation by default. void cv::idft(   cv::InputArray  src,                          // Input array (real or complex)   cv::OutputArray dst,                          // Output array   int             flags       = 0,              // for variations   int             nonzeroRows = 0               // number of rows to not ignore ); Calling cv::idft() is exactly equivalent to calling cv::dft() with the cv::DFT_INVERSE flag (in addition to whatever flags you supply to cv::idft(), of

that does this inverse operation by default. void cv::idft(   cv::InputArray  src,                          // Input array (real or complex)   cv::OutputArray dst,                          // Output array   int             flags       = 0,              // for variations   int             nonzeroRows = 0               // number of rows to not ignore ); Calling cv::idft() is exactly equivalent to calling cv::dft() with the cv::DFT_INVERSE flag (in addition to whatever flags you supply to cv::idft(), of course.) cv::mulSpectrums(), Spectrum Multiplication In many applications that involve computing DFTs, one must also compute the per-element multiplication of the two resulting spectra. Because such results are complex numbers, typically packed in their special high-density CCS format, it would be tedious to unpack them and handle the multiplication via the “usual” matrix operations. Fortunately, OpenCV provides the handy cv::mulSpectrums() routine, which performs exactly this function for us: void

of course.) cv::mulSpectrums(), Spectrum Multiplication In many applications that involve computing DFTs, one must also compute the per-element multiplication of the two resulting spectra. Because such results are complex numbers, typically packed in their special high-density CCS format, it would be tedious to unpack them and handle the multiplication via the “usual” matrix operations. Fortunately, OpenCV provides the handy cv::mulSpectrums() routine, which performs exactly this function for us: void cv::mulSpectrums(   cv::InputArray  src1,                         // First input array (ccs or complex)   cv::InputArray  src2,                         // Second input array (ccs or complex)   cv::OutputArray dst,                          // Result array                   int             flags,                        // for row-by-row computation   bool            conj = false                  // true to conjugate src2 ); Note that the first two arguments are arrays, which may be either CCS packed

us: void cv::mulSpectrums(   cv::InputArray  src1,                         // First input array (ccs or complex)   cv::InputArray  src2,                         // Second input array (ccs or complex)   cv::OutputArray dst,                          // Result array                   int             flags,                        // for row-by-row computation   bool            conj = false                  // true to conjugate src2 ); Note that the first two arguments are arrays, which may be either CCS packed single-channel spectra or two-channel complex spectra (as you would get from calls to cv::dft()). The third argument is the destination array, which will be of the same size and type as the source arrays. The final argument, conj, tells cv::mulSpectrums() exactly what you want done. In particular, it may be set to false for implementing the above pair multiplication or set to true if the element from the first array is to be multiplied by the complex conjugate of the corresponding element of the second

single-channel spectra or two-channel complex spectra (as you would get from calls to cv::dft()). The third argument is the destination array, which will be of the same size and type as the source arrays. The final argument, conj, tells cv::mulSpectrums() exactly what you want done. In particular, it may be set to false for implementing the above pair multiplication or set to true if the element from the first array is to be multiplied by the complex conjugate of the corresponding element of the second array18.  Convolution Using Discrete Fourier Transforms It is possible to greatly increase the speed of a convolution by using DFT via the convolution theorem [Titchmarsh26] that relates convolution in the spatial domain to multiplication in the Fourier domain [Morse53; Bracewell65; Arfken85].19 To accomplish this, one first computes the Fourier transform of the image and then the Fourier transform of the convolution filter. Once this is done, the convolution can be performed in the transform space in linear

Convolution Using Discrete Fourier Transforms It is possible to greatly increase the speed of a convolution by using DFT via the convolution theorem [Titchmarsh26] that relates convolution in the spatial domain to multiplication in the Fourier domain [Morse53; Bracewell65; Arfken85].19 To accomplish this, one first computes the Fourier transform of the image and then the Fourier transform of the convolution filter. Once this is done, the convolution can be performed in the transform space in linear time with respect to the number of pixels in the image. It is worthwhile to look at the source code for computing such a convolution, as it will also provide us with many good examples of using cv::dft(). The code is shown in Example 6-4, which is taken directly from the OpenCV reference.                                                              18 The primary usage of this argument is the implementation of a correlation in Fourier space. It turns out that the only difference between convolution (which we will

respect to the number of pixels in the image. It is worthwhile to look at the source code for computing such a convolution, as it will also provide us with many good examples of using cv::dft(). The code is shown in Example 6-4, which is taken directly from the OpenCV reference.                                                              18 The primary usage of this argument is the implementation of a correlation in Fourier space. It turns out that the only difference between convolution (which we will discuss in the next section), and correlation, is the conjugation of the second array in the spectrum multiplication. 19 Recall that OpenCV’s DFT algorithm implements the FFT whenever the data size makes the FFT faster.

Example 6-4: Use of cv::dft() and cv::idft() to accelerate the computation of convolutions #include <opencv2/opencv.hpp> #include <iostream>  using namespace std;  int main(int argc, char** argv) {      if(argc != 2) {      cout << "Fourier Transform\nUsage: ch6_ex6_5 <imagename>" << endl;      return -1;    }        cv::Mat A = cv::imread(argv[1],0);        if( A.empty() ) { cout << "Can not load " << argv[1] << endl; return -1; }         cv::Size patchSize( 100, 100 );   cv::Point topleft( A.cols/2, A.rows/2 );   cv::Rect roi( topleft.x, topleft.y, patchSize.width, patchSize.height );   cv::Mat B = A( roi );        int dft_M = cv::getOptimalDFTSize( A.rows+B.rows-1 );   int dft_N = cv::getOptimalDFTSize( A.cols+B.cols-1 );    cv::Mat dft_A = cv::Mat::zeros( dft_M, dft_N, cv::F32 );   cv::Mat dft_B = cv::Mat::zeros( dft_M, dft_N, cv::F32 );        cv::Mat dft_A_part = dft_A( Rect(0, 0, A.cols,A.rows) );   cv::Mat dft_B_part = dft_B( Rect(0, 0, B.cols,B.rows) );     A.convertTo( dft_A_part, dft_A_part.type(),

);   cv::Rect roi( topleft.x, topleft.y, patchSize.width, patchSize.height );   cv::Mat B = A( roi );        int dft_M = cv::getOptimalDFTSize( A.rows+B.rows-1 );   int dft_N = cv::getOptimalDFTSize( A.cols+B.cols-1 );    cv::Mat dft_A = cv::Mat::zeros( dft_M, dft_N, cv::F32 );   cv::Mat dft_B = cv::Mat::zeros( dft_M, dft_N, cv::F32 );        cv::Mat dft_A_part = dft_A( Rect(0, 0, A.cols,A.rows) );   cv::Mat dft_B_part = dft_B( Rect(0, 0, B.cols,B.rows) );     A.convertTo( dft_A_part, dft_A_part.type(), 1, -mean(A)[0] );   B.convertTo( dft_B_part, dft_B_part.type(), 1, -mean(B)[0] );        cv::dft( dft_A, dft_A, 0, A.rows );   cv::dft( dft_B, dft_B, 0, B.rows );        // set the last parameter to false to compute convolution instead of correlation   cv::mulSpectrums( dft_A, dft_B, dft_A, 0, true );   cv::idft( dft_A, dft_A, DFT_SCALE, A.rows + B.rows - 1 );        cv::Mat corr = dft_A( Rect(0, 0, A.cols + B.cols - 1, A.rows + B.rows - 1) );   cv::normalize( corr, corr, 0, 1, NORM_MINMAX, corr.type() );

-mean(A)[0] );   B.convertTo( dft_B_part, dft_B_part.type(), 1, -mean(B)[0] );        cv::dft( dft_A, dft_A, 0, A.rows );   cv::dft( dft_B, dft_B, 0, B.rows );        // set the last parameter to false to compute convolution instead of correlation   cv::mulSpectrums( dft_A, dft_B, dft_A, 0, true );   cv::idft( dft_A, dft_A, DFT_SCALE, A.rows + B.rows - 1 );        cv::Mat corr = dft_A( Rect(0, 0, A.cols + B.cols - 1, A.rows + B.rows - 1) );   cv::normalize( corr, corr, 0, 1, NORM_MINMAX, corr.type() );   cv::pow( corr, 3., corr );        cv::B ^= cv::Scalar::all( 255 );    cv::imshow( "Image", A );   cv::imshow( "Correlation", corr );   cv::waitKey();   return 0; } In Example 6-4, we can see that the input arrays are first created and then initialized. Next, two new arrays are created whose dimensions are optimal for the DFT algorithm. The original arrays are copied into these new arrays and the transforms are then computed. Finally, the spectra are multiplied together and the inverse transform is applied to

corr, 3., corr );        cv::B ^= cv::Scalar::all( 255 );    cv::imshow( "Image", A );   cv::imshow( "Correlation", corr );   cv::waitKey();   return 0; } In Example 6-4, we can see that the input arrays are first created and then initialized. Next, two new arrays are created whose dimensions are optimal for the DFT algorithm. The original arrays are copied into these new arrays and the transforms are then computed. Finally, the spectra are multiplied together and the inverse transform is applied to the product. The transforms are the slowest20 part of this operation; an 𝑁-by-                                                             20 By “slowest” we mean “asymptotically slowest”—in other words, that this portion of the algorithm takes the most time for very large 𝑁. This is an important distinction. In practice, as we saw in the earlier section on convolutions, it is

𝑁 image takes 𝑂(𝑁!  log  𝑁) time and so the entire process is also completed in that time (assuming that 𝑁>𝑀 for an 𝑀-by-𝑀 convolution kernel). This time is much faster than 𝑂(𝑁!𝑀!), the non-DFT convolution time required by the more naïve method. cv::dct(), the Discrete Cosine Transform For real-valued data, it is often sufficient to compute what is, in effect, only half of the discrete Fourier transform. The discrete cosine transform (DCT) [Ahmed74; Jain77] is defined analogously to the full DFT by the following formula: 𝑐!=1𝑁!!𝑥!+2𝑁!!𝑥!cos(𝑘+12)𝑛𝑁𝜋!!!!!! Of course, there is a similar transform for higher dimensions. Note that, by convention, the normalization factor is applied to both the cosine transform and its inverse (which is not the convention for the discrete Fourier transform).  The basic ideas of the DFT apply also to the DCT, but now all the coefficients are real-valued. Astute readers might object that the cosine transform is being applied to a vector that is not a manifestly even function.

following formula: 𝑐!=1𝑁!!𝑥!+2𝑁!!𝑥!cos(𝑘+12)𝑛𝑁𝜋!!!!!! Of course, there is a similar transform for higher dimensions. Note that, by convention, the normalization factor is applied to both the cosine transform and its inverse (which is not the convention for the discrete Fourier transform).  The basic ideas of the DFT apply also to the DCT, but now all the coefficients are real-valued. Astute readers might object that the cosine transform is being applied to a vector that is not a manifestly even function. However, with cv::dct(), the algorithm simply treats the vector as if it were extended to negative indices in a mirrored manner. The actual OpenCV call is: void cv::dct(   cv::InputArray  src,                          // Input array (even size)   cv::OutputArray dst,                          // Output array   int             flags       = 0               // for row-by-row or inverse ); The cv::dct() function expects arguments like those for cv::dft() except that, because the results are real-valued, there is

with cv::dct(), the algorithm simply treats the vector as if it were extended to negative indices in a mirrored manner. The actual OpenCV call is: void cv::dct(   cv::InputArray  src,                          // Input array (even size)   cv::OutputArray dst,                          // Output array   int             flags       = 0               // for row-by-row or inverse ); The cv::dct() function expects arguments like those for cv::dft() except that, because the results are real-valued, there is no need for any special packing of the result array (or of the input array in the case of an inverse transform). Unlike cv::dft(), however, the input array must have an even number of elements (you can pad the last element with a zero if necessary to achieve this). The flags argument can be set to cv::DCT_INVERSE to generate the inverse transformation, and either may be combined with cv::DCT_ROWS with the same effect as with cv::dft(). Because of the different normalization convention, both the forward and

no need for any special packing of the result array (or of the input array in the case of an inverse transform). Unlike cv::dft(), however, the input array must have an even number of elements (you can pad the last element with a zero if necessary to achieve this). The flags argument can be set to cv::DCT_INVERSE to generate the inverse transformation, and either may be combined with cv::DCT_ROWS with the same effect as with cv::dft(). Because of the different normalization convention, both the forward and inverse cosine transforms always contain their respective contribution to the overall normalization of the transform; hence there is no analog to cv::DFT_SCALE for cv::dct(). As with cv::dft(), there is a strong dependence of performance on array size. In fact, deep down, the implementation of cv::dct() actually calls cv::dft() on an array exactly half the size of your input array. For this reason, the optimal size of an array to pass to cv::dct() is exactly double the size of the optimal array you would

inverse cosine transforms always contain their respective contribution to the overall normalization of the transform; hence there is no analog to cv::DFT_SCALE for cv::dct(). As with cv::dft(), there is a strong dependence of performance on array size. In fact, deep down, the implementation of cv::dct() actually calls cv::dft() on an array exactly half the size of your input array. For this reason, the optimal size of an array to pass to cv::dct() is exactly double the size of the optimal array you would pass to cv::dft(). Putting everything together, the best way to get an optimal size for cv::dct() is to compute: size_t optimal_dct_size = 2 * cv::getOptimalDFTSize( (N+1)/2 ); where N is the actual size of your data that you want to transform. cv::idct(), the Inverse Discrete Cosine Transform Just as with cv::idft() and cv::dft(), cv::dct() can be asked to compute the inverse cosine transform using the flags argument. As before, code readability is often improved with the use of a separate function that

pass to cv::dft(). Putting everything together, the best way to get an optimal size for cv::dct() is to compute: size_t optimal_dct_size = 2 * cv::getOptimalDFTSize( (N+1)/2 ); where N is the actual size of your data that you want to transform. cv::idct(), the Inverse Discrete Cosine Transform Just as with cv::idft() and cv::dft(), cv::dct() can be asked to compute the inverse cosine transform using the flags argument. As before, code readability is often improved with the use of a separate function that does this inverse operation by default. void cv::idct(   cv::InputArray  src,                          // Input array                                                                                                                                                                                      not always optimal to pay the overhead for conversion to Fourier space. In general, when convolving with a small kernel it will not be worth the trouble to make this transformation.

cv::OutputArray dst,                          // Output array    int             flags = 0,                    // for row-by-row computation ); Calling cv::idct() is exactly equivalent to calling cv::dct() with the cv::DCT_INVERSE flag (in addition to any other flags you supply to cv::idct()). Integral Images OpenCV allows you to calculate an integral image easily with the appropriately named cv::integral() function. An integral image [Viola04] is a data structure that allows rapid summing of sub-regions21. Such summations are useful in many applications; a notable one is the computation of Haar wavelets, which are used in face detection and similar algorithms. There are three variations of the integral image that are supported by OpenCV. They are the sum, the square-sum, and the tilted-sum. A standard integral image sum has the form: 𝑠𝑢𝑚𝑥,𝑦=𝑖𝑚𝑎𝑔𝑒𝑥′,𝑦′!!!!!!!! The square-sum image is the sum of squares: 𝑠𝑢𝑚!"#$%&𝑥,𝑦=𝑖𝑚𝑎𝑔𝑒𝑥′,𝑦′!!!!!!!!! The tilted-sum is like the sum except that it is for the image rotated

Such summations are useful in many applications; a notable one is the computation of Haar wavelets, which are used in face detection and similar algorithms. There are three variations of the integral image that are supported by OpenCV. They are the sum, the square-sum, and the tilted-sum. A standard integral image sum has the form: 𝑠𝑢𝑚𝑥,𝑦=𝑖𝑚𝑎𝑔𝑒𝑥′,𝑦′!!!!!!!! The square-sum image is the sum of squares: 𝑠𝑢𝑚!"#$%&𝑥,𝑦=𝑖𝑚𝑎𝑔𝑒𝑥′,𝑦′!!!!!!!!! The tilted-sum is like the sum except that it is for the image rotated by 45 degrees: 𝑠𝑢𝑚!"#!$%𝑥,𝑦=  𝑖𝑚𝑎𝑔𝑒𝑥′,𝑦′!"#(!!!!)!!!!!! Using these integral images, one may calculate sums, means, and standard deviations over arbitrary upright or “tilted” rectangular regions of the image. As a simple example, to sum over a simple rectangular region described by the corner points (𝑥!,𝑦!)  and (𝑥!,𝑦!), where 𝑥!>𝑥!  and 𝑦!>𝑦!, we’d compute:  𝑖𝑚𝑎𝑔𝑒𝑥,𝑦!!!!<!!!!!!<!!=[𝑠𝑢𝑚𝑥!,𝑦!−𝑠𝑢𝑚𝑥!,𝑦!−𝑠𝑢𝑚𝑥!,𝑦!+𝑠𝑢𝑚𝑥!,𝑦!]  In this way, it is possible to do fast blurring, approximate gradients, compute means and

45 degrees: 𝑠𝑢𝑚!"#!$%𝑥,𝑦=  𝑖𝑚𝑎𝑔𝑒𝑥′,𝑦′!"#(!!!!)!!!!!! Using these integral images, one may calculate sums, means, and standard deviations over arbitrary upright or “tilted” rectangular regions of the image. As a simple example, to sum over a simple rectangular region described by the corner points (𝑥!,𝑦!)  and (𝑥!,𝑦!), where 𝑥!>𝑥!  and 𝑦!>𝑦!, we’d compute:  𝑖𝑚𝑎𝑔𝑒𝑥,𝑦!!!!<!!!!!!<!!=[𝑠𝑢𝑚𝑥!,𝑦!−𝑠𝑢𝑚𝑥!,𝑦!−𝑠𝑢𝑚𝑥!,𝑦!+𝑠𝑢𝑚𝑥!,𝑦!]  In this way, it is possible to do fast blurring, approximate gradients, compute means and standard deviations, and perform fast block correlations even for variable window sizes.                                                              21 The citation above is the best for more details on the method, but it was actually introduced in computer vision in 2001 in a paper Robust Real-time Object Detection by the same authors. The method was previously used as early as 1984 in computer graphics, where the integral image is known as a Summed Area Table.

To make this all a little more clear, consider the 7-by-5 image shown in   Figure 6-8; the region is shown as a bar chart in which the height associated with the pixels represents the brightness of those pixel values. The same information is shown in Figure 6-9, numerically on the left and in integral form on the right. Integral images 𝐼! are computed by going across rows, proceeding row by row using the previously computed integral image values together with the current raw image 𝐼 pixel value 𝐼(𝑥,𝑦) to calculate the next integral image value as follows: 𝐼!𝑥,𝑦=[𝐼𝑥,𝑦−𝐼𝑥−1,𝑦−𝐼𝑥,𝑦−1+𝐼𝑥−1,𝑦−1]   Figure 6-8: Simple 7-by-5 image shown as a bar chart with x, y, and height equal to pixel value The last term is subtracted off because this value is double-counted when adding the second and third terms. You can verify that this works by testing some values in Figure 6-9.

When using the integral image to compute a region, we can see by Figure 6-9 that, in order to compute the central rectangular area bounded by the 20s in the original image, we’d calculate 398  –  9  –  10  +  1  =  380. Thus, a rectangle of any size can be computed using four measurements (resulting in 𝑂(1) computational complexity).

1 2 5 1 2  0 0 0 0 0 0 2 20 50 20 5  0 1 3 8 9 11 5 50 100 50 2  0 3 25 80 101 108 2 20 50 20 1  0 8 80 235 306 315 1 5 25 1 2  0 10 102 307 398 408 5 2 25 2 5  0 11 108 338 430 442 2 1 5 2 1  0 16 115 370 464 481       0 18 118 378 474 492 Figure 6-9: The 7-by-5 image of   Figure 6-8 shown numerically at left (with the origin assumed to be the upper-left ) and converted to an (8-by-6) integral image at right cv::integral(), for Standard Summation Integral The different forms of integral are (somewhat confusingly) distinguished in the C++ API only by their arguments. The form that computes the basic sum has only three. void cv::integral(   cv::InputArray  image,                        // Input array   cv::OutputArray sum,                          // Output sum results   int             sdepth = -1                   // Depth for results (e.g., cv::F32) ); The first and second are the input and result images. If the input image is of size 𝑊-by-  𝐻, then the output image will be of size (𝑊+1)-by-(𝐻+1).22  The

distinguished in the C++ API only by their arguments. The form that computes the basic sum has only three. void cv::integral(   cv::InputArray  image,                        // Input array   cv::OutputArray sum,                          // Output sum results   int             sdepth = -1                   // Depth for results (e.g., cv::F32) ); The first and second are the input and result images. If the input image is of size 𝑊-by-  𝐻, then the output image will be of size (𝑊+1)-by-(𝐻+1).22  The third argument sdepth specifies the desired depth of the sum (destination) image. sdepth can be any of cv::S32, cv::F32, or cv::F64.23                                                               22 This allows for the rows of zeros which are implied by the fact that summing zero terms results in a sum of zero. 23 It is worth noting that even though sum and tilted_sum allow 32-bit float as output for input images of 32-bit float type, it is recommended to use 64-bit float, particularly for larger images. After all,

sdepth specifies the desired depth of the sum (destination) image. sdepth can be any of cv::S32, cv::F32, or cv::F64.23                                                               22 This allows for the rows of zeros which are implied by the fact that summing zero terms results in a sum of zero. 23 It is worth noting that even though sum and tilted_sum allow 32-bit float as output for input images of 32-bit float type, it is recommended to use 64-bit float, particularly for larger images. After all, a modern large image can be many millions of pixels.

cv::integral(), for Squared Summation Integral The squared sum is computed with the same function as the regular sum, except that the provision of an additional output argument for the squared sum. void cv::integral(   cv::InputArray  image,                        // Input array   cv::OutputArray sum,                          // Output sum results   cv::OutputArray sqsum,                        // Output sum of squares results   int             sdepth = -1                   // Depth for results (e.g., cv::F32) ); The cv::OutputArray argument sqsum indicates to cv::integral() that the square sum should be computed in addition to the regular sum. As before, sdepth specifies the desired depth of the resulting images. sdepth can be any of cv::S32, cv::F32, or cv::F64. cv::integral(), for Standard Summation Integral Similar to the squared sum, the tilted sum integral is essentially the same function, with an additional argument for the additional result. void cv::integral(   cv::InputArray  image,

); The cv::OutputArray argument sqsum indicates to cv::integral() that the square sum should be computed in addition to the regular sum. As before, sdepth specifies the desired depth of the resulting images. sdepth can be any of cv::S32, cv::F32, or cv::F64. cv::integral(), for Standard Summation Integral Similar to the squared sum, the tilted sum integral is essentially the same function, with an additional argument for the additional result. void cv::integral(   cv::InputArray  image,                        // Input array   cv::OutputArray sum,                          // Output sum results   cv::OutputArray sqsum,                        // Output sum of squares results   cv::OutputArray tilted,                       // Output tilted sum results   int             sdepth = -1                   // Depth for results (e.g., cv::F32) ); The additional cv::OutputArray argument tilted is computed by this form of cv::integral(), in addition to the other sums, thus all of the other arguments are the same.  The

// Input array   cv::OutputArray sum,                          // Output sum results   cv::OutputArray sqsum,                        // Output sum of squares results   cv::OutputArray tilted,                       // Output tilted sum results   int             sdepth = -1                   // Depth for results (e.g., cv::F32) ); The additional cv::OutputArray argument tilted is computed by this form of cv::integral(), in addition to the other sums, thus all of the other arguments are the same.  The Canny Edge Detector Though it is possible to expose edges in images with simple filters such as the Laplace filter, it is possible to improve on this method substantially. The simple Laplace filter method was refined by J. Canny in 1986 into what is now commonly called the Canny edge detector [Canny86]. One of the differences between the Canny algorithm and the simpler, Laplace-based algorithm in the previous chapter is that, in the Canny algorithm, the first derivatives are computed in 𝑥 and 𝑦 and then

Canny Edge Detector Though it is possible to expose edges in images with simple filters such as the Laplace filter, it is possible to improve on this method substantially. The simple Laplace filter method was refined by J. Canny in 1986 into what is now commonly called the Canny edge detector [Canny86]. One of the differences between the Canny algorithm and the simpler, Laplace-based algorithm in the previous chapter is that, in the Canny algorithm, the first derivatives are computed in 𝑥 and 𝑦 and then combined into four directional derivatives. The points where these directional derivatives are local maxima are then candidates for assembling into edges.

Figure 6-10: Results of Canny edge detection for two different images when the high and low thresholds are set to 50 and 10, respectively. However, the most significant new dimension to the Canny algorithm is that it tries to assemble the individual edge candidate pixels into contours.24 These contours are formed by applying a hysteresis threshold to the pixels. This means that there are two thresholds, an upper and a lower. If a pixel has a gradient larger than the upper threshold, then it is accepted as an edge pixel; if a pixel is below the lower threshold, it is rejected. If the pixel’s gradient is between the thresholds, then it will be accepted only if it is connected to a pixel that is above the high threshold. Canny recommended a ratio of high:low threshold between 2:1 and 3:1.  Figure 6-10 and  Figure 6-11 show the results of applying cv::Canny() to a test pattern and a photograph using high:low hysteresis threshold ratios of 5:1 and 3:2, respectively.

edge pixel; if a pixel is below the lower threshold, it is rejected. If the pixel’s gradient is between the thresholds, then it will be accepted only if it is connected to a pixel that is above the high threshold. Canny recommended a ratio of high:low threshold between 2:1 and 3:1.  Figure 6-10 and  Figure 6-11 show the results of applying cv::Canny() to a test pattern and a photograph using high:low hysteresis threshold ratios of 5:1 and 3:2, respectively.                                                              24 We’ll have much more to say about contours later. As you await those revelations, keep in mind that the cv::Canny() routine does not actually return objects of a contour type; we will have to build those from the output of cv::Canny() if we want them by using cv::findContours(). Everything you ever wanted to know about contours will be covered in Chapter 8.

Figure 6-11: Results of Canny edge detection for two different images when the high and low thresholds are set to 150 and 100, respectively cv::Canny() The OpenCV implementation of the Canny edge detection algorithm converts an input image into an “edge image”. void cv::Canny(   cv::InputArray  image,                        // Input single channel image   cv::OutputArray edges,                        // Output edge image   double          threshold1,                   // “lower” threshold   double          threshold2,                   // “upper” threshold   int             apertureSize = 3,             // Sobel aperture   bool            L2gradient   = false          // true for more accurate L2-norm ); The cv::Canny() function expects an input image, which must be single-channel, and an output image, which will also be grayscale (but which will actually be a Boolean image). The next two arguments are the low and high thresholds. The next to last argument apertureSize is the aperture used by the Sobel

threshold2,                   // “upper” threshold   int             apertureSize = 3,             // Sobel aperture   bool            L2gradient   = false          // true for more accurate L2-norm ); The cv::Canny() function expects an input image, which must be single-channel, and an output image, which will also be grayscale (but which will actually be a Boolean image). The next two arguments are the low and high thresholds. The next to last argument apertureSize is the aperture used by the Sobel derivative operators that are called inside of the implementation of cv::Canny(). The final argument L2gradient is used to select between computing the gradient “correctly” using the proper 𝐿!-norm, or if a faster less accurate 𝐿!-norm based method should be used. If the argument L2gradient is set to true, the more accurate form is used: 𝑔𝑟𝑎𝑑𝑥,𝑦!!=𝑑𝐼𝑑𝑥!+𝑑𝐼𝑑𝑦! If L2gradient is set to false, the faster form is used: 𝑔𝑟𝑎𝑑𝑥,𝑦!!=𝑑𝐼𝑑𝑥+𝑑𝐼𝑑𝑦 Line Segment Detection In a great number of practical image processing

derivative operators that are called inside of the implementation of cv::Canny(). The final argument L2gradient is used to select between computing the gradient “correctly” using the proper 𝐿!-norm, or if a faster less accurate 𝐿!-norm based method should be used. If the argument L2gradient is set to true, the more accurate form is used: 𝑔𝑟𝑎𝑑𝑥,𝑦!!=𝑑𝐼𝑑𝑥!+𝑑𝐼𝑑𝑦! If L2gradient is set to false, the faster form is used: 𝑔𝑟𝑎𝑑𝑥,𝑦!!=𝑑𝐼𝑑𝑥+𝑑𝐼𝑑𝑦 Line Segment Detection In a great number of practical image processing applications, it is useful to be able to find either the dominant line segments, or all of the lines in an image. In particular, straight lines often provide valuable clues about perspective and the structure of objects. OpenCV provides several different algorithms which address this problem in different ways, each with their own distinct strengths and weaknesses.

The Hough transform25 is a method for finding lines, circles, or other simple forms in an image. The original Hough transform was a line transform, which is a relatively fast way of searching a binary image for straight lines. The transform can be further generalized to cases other than just simple lines (we will return to this in the next section). In addition to the Hough Line Transform, another more recent algorithm called LSD (for Line Segment Detector) provides a fast technique which is, in general, more robust than the Hough Line Transform. Hough Line Transform The basic theory of the Hough line transform is that any point in a binary image could be part of some set of possible lines. If we parameterize each line by, for example, a slope 𝑎 and an intercept 𝑏, then a point in the original image is transformed to a locus of points in the (𝑎,𝑏) plane corresponding to all of the lines passing through that point (see  Figure 6-12). If we convert every nonzero pixel in the input image into such a set of

more robust than the Hough Line Transform. Hough Line Transform The basic theory of the Hough line transform is that any point in a binary image could be part of some set of possible lines. If we parameterize each line by, for example, a slope 𝑎 and an intercept 𝑏, then a point in the original image is transformed to a locus of points in the (𝑎,𝑏) plane corresponding to all of the lines passing through that point (see  Figure 6-12). If we convert every nonzero pixel in the input image into such a set of points in the output image and sum over all such contributions, then lines that appear in the input (i.e., (𝑥,𝑦) plane) image will appear as local maxima in the output (i.e., (𝑎,𝑏) plane) image. Because we are summing the contributions from each point, the (𝑎,𝑏) plane is commonly called the accumulator plane.                                                               25 Hough developed the transform for use in physics experiments [Hough59]; its use in vision was introduced by Duda and Hart [Duda72].

Figure 6-12: The Hough line transform finds many lines in each image; some of the lines found are expected, but others may not be. It might occur to you that the slope-intercept form is not really the best way to represent all of the lines passing through a point (because of the considerably different density of lines as a function of the slope, and the related fact that the interval of possible slopes goes from –∞ to +∞). It is for this reason that the actual parameterization of the transform image used in numerical computation is somewhat different. The preferred parameterization represents each line as a point in polar coordinates (𝜌,𝜃), with the implied line being the line passing through the indicated point but perpendicular to the radial from the origin to that point (see  Figure 6-13). The equation for such a line is: 𝜌=𝑥cos𝜃+𝑦sin𝜃 The OpenCV Hough transform algorithm does not make this computation explicit to the user. Instead, it simply returns the local maxima in the (𝜌,𝜃) plane. However, you will

numerical computation is somewhat different. The preferred parameterization represents each line as a point in polar coordinates (𝜌,𝜃), with the implied line being the line passing through the indicated point but perpendicular to the radial from the origin to that point (see  Figure 6-13). The equation for such a line is: 𝜌=𝑥cos𝜃+𝑦sin𝜃 The OpenCV Hough transform algorithm does not make this computation explicit to the user. Instead, it simply returns the local maxima in the (𝜌,𝜃) plane. However, you will need to understand this process in order to understand the arguments to the OpenCV Hough line transform function.  Figure 6-13: A point 𝑥!,𝑦!  in the image plane (a) implies many lines each parameterized by a different 𝜌 and 𝜃 (b); these lines in the 𝜌,𝜃 plane, which taken together form a curve of characteristic shape (c). OpenCV supports three different kinds of Hough line transform: the standard Hough transform (SHT) [Duda72], the multiscale Hough transform (MHT), and the progressive probabilistic Hough

need to understand this process in order to understand the arguments to the OpenCV Hough line transform function.  Figure 6-13: A point 𝑥!,𝑦!  in the image plane (a) implies many lines each parameterized by a different 𝜌 and 𝜃 (b); these lines in the 𝜌,𝜃 plane, which taken together form a curve of characteristic shape (c). OpenCV supports three different kinds of Hough line transform: the standard Hough transform (SHT) [Duda72], the multiscale Hough transform (MHT), and the progressive probabilistic Hough transform (PPHT).26 The SHT is the algorithm we just looked at. The MHT algorithm is a slight refinement that gives                                                              26 The “probabilistic Hough transform” (PHT) was introduced by Kiryati, Eldar, and Bruckshtein in 1991 [Kiryati91]; the PPHT was introduced by Matas, Galambosy, and Kittler in 1999 [Matas99].

more accurate values for the matched lines. The PPHT is a variation of this algorithm that, among other things, computes an extent for individual lines in addition to the orientation (as shown in   Figure 6-14). It is “probabilistic” because, rather than accumulating every possible point in the accumulator plane, it accumulates only a fraction of them. The idea is that if the peak is going to be high enough anyhow, then hitting it only a fraction of the time will be enough to find it; the result of this conjecture can be a substantial reduction in computation time.  cv::HoughLines(), the Standard and Multi-scale Hough Transforms The standard and multi-scale Hough transforms are both implemented in a single function: cv::HoughLines() – with the distinction being in the use (or nonuse) of two optional parameters. void cv::HoughLines(   cv::InputArray  image,                        // Input single channel image   cv::OutputArray lines,                        // N-by-1 2-channel array   double          rho,

can be a substantial reduction in computation time.  cv::HoughLines(), the Standard and Multi-scale Hough Transforms The standard and multi-scale Hough transforms are both implemented in a single function: cv::HoughLines() – with the distinction being in the use (or nonuse) of two optional parameters. void cv::HoughLines(   cv::InputArray  image,                        // Input single channel image   cv::OutputArray lines,                        // N-by-1 2-channel array   double          rho,                          // rho resolution (pixels)   double          theta,                        // theta resolution (radians)   int             threshold,                    // Unnormalized accumulator threshold   double          srn      = 0,                 // rho refinement (for MHT)   double          stn      = 0                  // theta refinement (for MHT) ); The first argument is the input image. It must be an 8-bit image, but the input is treated as binary information (i.e., all nonzero pixels are

// rho resolution (pixels)   double          theta,                        // theta resolution (radians)   int             threshold,                    // Unnormalized accumulator threshold   double          srn      = 0,                 // rho refinement (for MHT)   double          stn      = 0                  // theta refinement (for MHT) ); The first argument is the input image. It must be an 8-bit image, but the input is treated as binary information (i.e., all nonzero pixels are considered to be equivalent). The second argument is the place where the found lines will be stored. It will be an 𝑁-by-1 two-channel array of floating point type (the number of columns 𝑁, will be the number of lines returned).27 The two channels will contain the 𝜌 and 𝜃 values for each found line. The next two arguments, rho and theta, set the resolution desired for the lines (i.e., the resolution of the accumulator plane). The units of rho are pixels and the units of theta are radians; thus, the

are considered to be equivalent). The second argument is the place where the found lines will be stored. It will be an 𝑁-by-1 two-channel array of floating point type (the number of columns 𝑁, will be the number of lines returned).27 The two channels will contain the 𝜌 and 𝜃 values for each found line. The next two arguments, rho and theta, set the resolution desired for the lines (i.e., the resolution of the accumulator plane). The units of rho are pixels and the units of theta are radians; thus, the accumulator plane can be thought of as a two-dimensional histogram with cells of dimension rho pixels by theta radians. The threshold value is the value in the accumulator plane that must be reached for the algorithm to report a line. This last argument is a bit tricky in practice; it is not normalized, so you should expect to scale it up with the image size for SHT. Remember that this argument is, in effect, indicating the number of points (in the edge image) that must support the line for the line to be

accumulator plane can be thought of as a two-dimensional histogram with cells of dimension rho pixels by theta radians. The threshold value is the value in the accumulator plane that must be reached for the algorithm to report a line. This last argument is a bit tricky in practice; it is not normalized, so you should expect to scale it up with the image size for SHT. Remember that this argument is, in effect, indicating the number of points (in the edge image) that must support the line for the line to be returned.                                                              27 As usual, depending on the object type you pass to lines, this could be either a 1-by-𝑁 array with two channels, or if you like, a std::vector<> with 𝑁 entries, with each of those entries being of type Vec2f.

Figure 6-14: The Canny edge detector (param1=50, param2=150) is run first, with the results shown in gray, and the progressive probabilistic Hough transform (param1=50, param2=10) is run next, with the results overlaid in white; you can see that the strong lines are generally picked up by the Hough transform The parameters srn and stn are not used by the standard Hough transform; they control an extension of the SHT algorithm called the multi-scale Hough transform (MHT). For MHT, these two parameters indicate higher resolutions to which the parameters for the lines should be computed. MHT first computes the locations of the lines to the accuracy given by the rho and theta parameters and then goes on to refine those results by a factor of srn and stn, respectively (i.e., the final resolution in rho is rho divided by srn and the final resolution in theta is theta divided by stn). Leaving these parameters set to zero causes the SHT algorithm to be run. cv::HoughLinesP(), the Progressive Probabilistic Hough

higher resolutions to which the parameters for the lines should be computed. MHT first computes the locations of the lines to the accuracy given by the rho and theta parameters and then goes on to refine those results by a factor of srn and stn, respectively (i.e., the final resolution in rho is rho divided by srn and the final resolution in theta is theta divided by stn). Leaving these parameters set to zero causes the SHT algorithm to be run. cv::HoughLinesP(), the Progressive Probabilistic Hough Transform void cv::HoughLinesP(   cv::InputArray  image,                        // Input single channel image   cv::OutputArray lines,                        // N-by-1 4-channel array   double          rho,                          // rho resolution (pixels)   double          theta,                        // theta resolution (radians)   int             threshold,                    // Un-normalized accumulator threshold   double          minLineLength = 0,            // required line length   double

void cv::HoughLinesP(   cv::InputArray  image,                        // Input single channel image   cv::OutputArray lines,                        // N-by-1 4-channel array   double          rho,                          // rho resolution (pixels)   double          theta,                        // theta resolution (radians)   int             threshold,                    // Un-normalized accumulator threshold   double          minLineLength = 0,            // required line length   double          maxLineGap    = 0             // required line separation ); The cv::HoughLinesP() function works very much like cv::HoughLines(), with two important differences. The first is that the lines argument will be a four-channel array (or a vector of objects all of type Vec4i). The four channels will be the x!,y! and x!,y! (in that order), the x,y locations of the two endpoints of the found line segment. The second important difference is the meaning of the two parameters. For the PPHT, the minLineLength and maxLineGap

= 0             // required line separation ); The cv::HoughLinesP() function works very much like cv::HoughLines(), with two important differences. The first is that the lines argument will be a four-channel array (or a vector of objects all of type Vec4i). The four channels will be the x!,y! and x!,y! (in that order), the x,y locations of the two endpoints of the found line segment. The second important difference is the meaning of the two parameters. For the PPHT, the minLineLength and maxLineGap arguments set the minimum length of a line segment that will be returned, and the separation between collinear segments required for the algorithm not to join them into a single longer segment.  Line Segment Detection (LSD): cv::LineSegmentDetector The LSD algorithm, originally developed by Rafael Grompone von Gioi, Jérémie Jakubowicz, Jean-Michel Morel, and Gregory Randall [vonGioi10][vonGioi12], is a linear time line segment finding algorithm. The LSD algorithm works by first analyzing the image in a local

set the minimum length of a line segment that will be returned, and the separation between collinear segments required for the algorithm not to join them into a single longer segment.  Line Segment Detection (LSD): cv::LineSegmentDetector The LSD algorithm, originally developed by Rafael Grompone von Gioi, Jérémie Jakubowicz, Jean-Michel Morel, and Gregory Randall [vonGioi10][vonGioi12], is a linear time line segment finding algorithm. The LSD algorithm works by first analyzing the image in a local way and determining what kind of line might be supported by the pixels in these small patches. The patches are then agglomerated into larger hypotheses which are then validated before being accepted.

Figure 6-15: Beginning with an image (a), the LSD algorithm first computes the local level line field (b). Level lines associated with weak gradients are rejected (c). Line support regions are associated with rectangles (d) whose width W is the approximate width of the line. In some cases, the widths are reduced to produce lines with lower NFA scores (e). The final resulting lines are shown here on top of the original image (f). The local procedure is based on what is called a level line (or, collectively across the entire image, the level line field Figure 6-15b). The level line field is computed by first computing the gradient of the image at every point, the level lines are perpendicular to those local gradients.  Once the level line field has been computed, connected regions can be generated which contain level lines of the same or similar orientation (i.e. within some tolerance  𝜏). This is a very powerful idea, because it does not require the level lines to be associated with equally strong

across the entire image, the level line field Figure 6-15b). The level line field is computed by first computing the gradient of the image at every point, the level lines are perpendicular to those local gradients.  Once the level line field has been computed, connected regions can be generated which contain level lines of the same or similar orientation (i.e. within some tolerance  𝜏). This is a very powerful idea, because it does not require the level lines to be associated with equally strong gradients28. In this way, aliasing effects can be naturally accounted for, as even in an aliased line, the orientation of the local level lines will be approximately parallel.  Collectively, the regions containing similar level lines are called line support regions. A rectangle is fit to each such line support region; this rectangle represents a hypothesis for a line  Figure 6-15d. The LSD algorithm does not double count individual level lines. To avoid doing so, the level lines are first sorted based on the

In this way, aliasing effects can be naturally accounted for, as even in an aliased line, the orientation of the local level lines will be approximately parallel.  Collectively, the regions containing similar level lines are called line support regions. A rectangle is fit to each such line support region; this rectangle represents a hypothesis for a line  Figure 6-15d. The LSD algorithm does not double count individual level lines. To avoid doing so, the level lines are first sorted based on the magnitudes of their associated gradients, on the assumption that the strongest gradients go with the most important lines. The line support regions are then constructed by building out from those strong gradient points in descending order of their strength. Because an exact ordering is not necessary for this process, the gradient magnitudes are instead binned into 𝑁!"#$ different bins in between zero and the largest observed gradient. The advantage of this pseudo-ordering is that it takes 𝑂(𝑁) time, rather than

of their associated gradients, on the assumption that the strongest gradients go with the most important lines. The line support regions are then constructed by building out from those strong gradient points in descending order of their strength. Because an exact ordering is not necessary for this process, the gradient magnitudes are instead binned into 𝑁!"#$ different bins in between zero and the largest observed gradient. The advantage of this pseudo-ordering is that it takes 𝑂(𝑁) time, rather than 𝑂(𝑁log𝑁) time (as would be expected of a full sorting). Once the rectangle hypotheses have been formed, each can be tested; if a large number of the points contained are consistent with the hypothesis, then a line segment has probably been found. If, on the other hand, this bounding rectangle contains many points which are not part of the original line support region, the proposed segment is probably not real.  The actual test is based on what is called the number of false alarms (NFA) for a particular

𝑂(𝑁log𝑁) time (as would be expected of a full sorting). Once the rectangle hypotheses have been formed, each can be tested; if a large number of the points contained are consistent with the hypothesis, then a line segment has probably been found. If, on the other hand, this bounding rectangle contains many points which are not part of the original line support region, the proposed segment is probably not real.  The actual test is based on what is called the number of false alarms (NFA) for a particular rectangle. The NFA is a function of the total number of pixels in the rectangle 𝑛 as well as the number of aligned pixels in that rectangle  𝑘. Conceptually, the NFA is the number of rectangles we would expect to find in a random                                                              28 It is however, common practice to exclude level lines whose associated gradient is below some threshold (as shown in Figure 6-15c).

image with similar significance to the hypothesis rectangle (i.e. similar 𝑛 and  𝑘)29. Rectangle hypotheses are rejected if their NFA is below some threshold: 𝑁𝐹𝐴!,!𝑛,𝑘<𝜖. Optionally, a refinement pass can attempt to tune the rectangle hypotheses to minimize their NFA Figure 6-15e. Using the cv::lineSegmentDetector Object The LSD algorithm is implemented in OpenCV using the more contemporary style in which features of the library are implemented as objects which hold their various parameters and thereafter provide an algorithmic service by means of their member functions. The object which implements the LSD algorithm is called cv::lineSegmentDetector. The following function creates one of these objects and returns you a cv::Ptr<> to it: cv::Ptr<LineSegmentDetector> cv::createLineSegmentDetector( int    refine      = cv::LSD_REFINE_STD,  double scale       = 0.8,  double sigma_scale = 0.6,  double quant       = 2.0,  double ang_th      = 22.5,  double log_eps     = 0,  double density_th  = 0.7,  int    n_bins

an algorithmic service by means of their member functions. The object which implements the LSD algorithm is called cv::lineSegmentDetector. The following function creates one of these objects and returns you a cv::Ptr<> to it: cv::Ptr<LineSegmentDetector> cv::createLineSegmentDetector( int    refine      = cv::LSD_REFINE_STD,  double scale       = 0.8,  double sigma_scale = 0.6,  double quant       = 2.0,  double ang_th      = 22.5,  double log_eps     = 0,  double density_th  = 0.7,  int    n_bins      = 1024 ); You should first notice that every argument to this function has a default value. This is not an accident. These values were determined by the designers of the LSD algorithm to be broadly optimal. As a result, you are unlikely to want to change them. For experts (and the experimentally minded) however, we describe them here. The first argument is refine, which controls how the algorithm will attempt to refine (though not necessarily reduce) its initial findings to the best final output. The three

= 1024 ); You should first notice that every argument to this function has a default value. This is not an accident. These values were determined by the designers of the LSD algorithm to be broadly optimal. As a result, you are unlikely to want to change them. For experts (and the experimentally minded) however, we describe them here. The first argument is refine, which controls how the algorithm will attempt to refine (though not necessarily reduce) its initial findings to the best final output. The three refinement options are to do nothing (cv::LSD_REFINE_NONE), to attempt to break long arcs into multiple shorted but straighter segments (cv::LSD_REFINE_STD), and to both do arc decomposition as well as automatic parameter tuning, which should give better results at the expense of longer runtime (cv::LSD_REFINE_ADV).  The second argument is the scale argument. Any incoming image to the LSD algorithm is automatically rescaled in both dimensions by this factor. The utility of this is that, just as with human

refinement options are to do nothing (cv::LSD_REFINE_NONE), to attempt to break long arcs into multiple shorted but straighter segments (cv::LSD_REFINE_STD), and to both do arc decomposition as well as automatic parameter tuning, which should give better results at the expense of longer runtime (cv::LSD_REFINE_ADV).  The second argument is the scale argument. Any incoming image to the LSD algorithm is automatically rescaled in both dimensions by this factor. The utility of this is that, just as with human perception, lines which are apparent at large scales will often be invisible when the image is viewed at high resolution; similarly, at reduced resolution, lines in fine details will be lost. As part of the reduction produced by scale, the image is first convolved with a Gaussian kernel. The size of this kernel is specified by sigma_scale; the size in pixels is actually the ratio of sigma_scale divided by scale. The default sigma value of 0.8 along with the default value for sigma_scale of 0.6 are chosen to

lines which are apparent at large scales will often be invisible when the image is viewed at high resolution; similarly, at reduced resolution, lines in fine details will be lost. As part of the reduction produced by scale, the image is first convolved with a Gaussian kernel. The size of this kernel is specified by sigma_scale; the size in pixels is actually the ratio of sigma_scale divided by scale. The default sigma value of 0.8 along with the default value for sigma_scale of 0.6 are chosen to give good results even in images with substantial aliasing effects on lines, without substantially altering the return from what would be expected at full scale.  The argument quant is a bound on the possible error in the gradient value due to quantization effects. It is used to determine which gradients are too small to use for the computation of level lines. This value would naturally be 1.0 (because the pixel values are integer values), but it has been found empirically that one gets better results by doubling

results even in images with substantial aliasing effects on lines, without substantially altering the return from what would be expected at full scale.  The argument quant is a bound on the possible error in the gradient value due to quantization effects. It is used to determine which gradients are too small to use for the computation of level lines. This value would naturally be 1.0 (because the pixel values are integer values), but it has been found empirically that one gets better results by doubling this to 2.0 (the default).  The ang_th argument sets the angular tolerance within which level lines can be considered to have the same orientation as their bounding rectangle. The default value for this parameter is 22.5 (degrees). The log_eps argument is only used with advanced refinement (cv::LSD_REFINE_ADV). In this case, it is used to set the log of the cutoff on the number of false alarms (NFA). This is the (log of the) parameter                                                              29 We are hand

to 2.0 (the default).  The ang_th argument sets the angular tolerance within which level lines can be considered to have the same orientation as their bounding rectangle. The default value for this parameter is 22.5 (degrees). The log_eps argument is only used with advanced refinement (cv::LSD_REFINE_ADV). In this case, it is used to set the log of the cutoff on the number of false alarms (NFA). This is the (log of the) parameter                                                              29 We are hand waving away some math here. The actual concept of significance is based on the probability of generating a similar rectangle on a noise image. As such, “similar 𝑛 and  𝑘” means “those 𝑛 and  𝑘 which would give a similar probability”, which in fact means that they give the same value for 𝐵𝑛,𝑘,𝑝=𝑛𝑗!!!!𝑝!1−𝑝!!!. (Here 𝑝=𝜏/𝜋.)

𝜖 we encountered earlier. The default value for log  (𝜖) is 0, which corresponds to one false alarm (per image). You can set this to -1 or -2 to prune out some of the more marginal lines. The next argument, density_th is straight forward; it is simply the fraction of aligned points in a particular rectangle required for that rectangle to be considered. The default for this parameter is 0.7.  The final argument, n_bins, is the number of bins in the pseudo-ordering of the gradient modulus. The default number of bins is 1024. Once you have constructed your cv::lineSegmentDetector argument, you can use it to perform detections on images: void cv::LineSegmentDetector::detect( const InputArray image,  cv::OutputArray  lines,  cv::OutputArray  width = cv::noArray(),  cv::OutputArray  prec  = cv::noArray(),  cv::OutputArray  nfa   = cv::noArray() ); The input image should be a grayscale image (cv::U8C1). From this image, the resulting lines will be computed and returned. The lines output array will be filled with

of bins is 1024. Once you have constructed your cv::lineSegmentDetector argument, you can use it to perform detections on images: void cv::LineSegmentDetector::detect( const InputArray image,  cv::OutputArray  lines,  cv::OutputArray  width = cv::noArray(),  cv::OutputArray  prec  = cv::noArray(),  cv::OutputArray  nfa   = cv::noArray() ); The input image should be a grayscale image (cv::U8C1). From this image, the resulting lines will be computed and returned. The lines output array will be filled with cv::Vec4i objects, each of which will contain the x and y location of the point at one end of the line followed by the x and y location of the point at the other end. The optional parameters width, prec, and nfa return the widths, precisions, and NFA’s of each of the found lines. Each of these will be a single column array with the same number of rows as lines. The width is the transverse width of the associated rectangle, and thus an approximation of the width of the line. The precision (angular tolerance)

objects, each of which will contain the x and y location of the point at one end of the line followed by the x and y location of the point at the other end. The optional parameters width, prec, and nfa return the widths, precisions, and NFA’s of each of the found lines. Each of these will be a single column array with the same number of rows as lines. The width is the transverse width of the associated rectangle, and thus an approximation of the width of the line. The precision (angular tolerance) is initially set to  𝜏/𝜋, and so starts out proportional to ang_th. However, the LSD algorithm will attempt to reduce both the width and the precision if doing so will improve the NFA.30 The final NFA’s associated with each line can be returned in the array nfa, but only when advanced refinement (cv::LSD_REFINE_ADV) is used. The values in nfa are the negative log of the NFA for that rectangle (e.g. +1 corresponds to an NFA of 0.1). Once you have your lines, there is a convenient method for drawing them on top of

set to  𝜏/𝜋, and so starts out proportional to ang_th. However, the LSD algorithm will attempt to reduce both the width and the precision if doing so will improve the NFA.30 The final NFA’s associated with each line can be returned in the array nfa, but only when advanced refinement (cv::LSD_REFINE_ADV) is used. The values in nfa are the negative log of the NFA for that rectangle (e.g. +1 corresponds to an NFA of 0.1). Once you have your lines, there is a convenient method for drawing them on top of your image:  void cv::LineSegmentDetector::drawSegments( cv::InputOutputArray image,  cv::InputArray       lines );  Given an image, and the lines you computed with cv::LineSegmentDetector::detect(), an image is produced by adding the lines on top of image (in red if image is in color). If you should have two sets of lines, you can also compare them using the method cv::LineSegmentDetector::compareSegments(), which has the following prototype: int cv::LineSegmentDetector::compareSegments( const cv::Size&

image:  void cv::LineSegmentDetector::drawSegments( cv::InputOutputArray image,  cv::InputArray       lines );  Given an image, and the lines you computed with cv::LineSegmentDetector::detect(), an image is produced by adding the lines on top of image (in red if image is in color). If you should have two sets of lines, you can also compare them using the method cv::LineSegmentDetector::compareSegments(), which has the following prototype: int cv::LineSegmentDetector::compareSegments( const cv::Size&      size,  cv::InputArray       lines1,  cv::InputArray       lines2,  cv::InputOutputArray image = noArray() ); In order to use the comparison method, you must provide size, which is the size of the original image you found the two sets of lines in (they must come from an image of the same size). You can then provide two sets of lines: lines1 and lines2, in the same format (array of cv::Vec4i) that you them from the detector. If you provide an image array, the line segment detector will draw both sets of lines

cv::InputArray       lines1,  cv::InputArray       lines2,  cv::InputOutputArray image = noArray() ); In order to use the comparison method, you must provide size, which is the size of the original image you found the two sets of lines in (they must come from an image of the same size). You can then provide two sets of lines: lines1 and lines2, in the same format (array of cv::Vec4i) that you them from the detector. If you provide an image array, the line segment detector will draw both sets of lines (one in blue,                                                              30 If you think about a line without a smooth edge, reducing the width of the rectangle might well reduce the area of the rectangle much faster than it reduces the number of aligned points. In a similar idea, reducing the precision will reduce the number of aligned points, but may reduce the number of false alarms even faster.

the other in red) on your image. Even if you do not supply an image for visualization, the return value of the segment comparison method tells you how many pixels were not matching in the two line images (i.e. a return value of 0 corresponds to a perfect match). The Circle Detection Algorithms such as the Hough Transform can be generalized to other contexts as well. One particularly useful example is circle detection which, as we will see, employs a very similar mechanism to the Hough Line Transform to find circles in an image.  Hough Circle Transform The Hough circle transform [Kimme75] (see  Figure 6-16) works in a manner roughly analogous to the Hough line transforms just described. The reason it is only “roughly” is that—if one were to try doing the exactly analogous thing—the accumulator plane would have to be replaced with an accumulator volume with three dimensions: one for 𝑥 and one for 𝑦 (the location of the circle center), and another for the circle radius 𝑟. This would mean far greater memory

find circles in an image.  Hough Circle Transform The Hough circle transform [Kimme75] (see  Figure 6-16) works in a manner roughly analogous to the Hough line transforms just described. The reason it is only “roughly” is that—if one were to try doing the exactly analogous thing—the accumulator plane would have to be replaced with an accumulator volume with three dimensions: one for 𝑥 and one for 𝑦 (the location of the circle center), and another for the circle radius 𝑟. This would mean far greater memory requirements and much slower speed. The implementation of the circle transform in OpenCV avoids this problem by using a somewhat more tricky method called the Hough gradient method. The Hough gradient method works as follows. First, the image is passed through an edge detection phase (in this case, cv::Canny()). Next, for every nonzero point in the edge image, the local gradient is considered (the gradient is computed by first computing the first-order Sobel x- and y-derivatives via cv::Sobel()). Using this

and much slower speed. The implementation of the circle transform in OpenCV avoids this problem by using a somewhat more tricky method called the Hough gradient method. The Hough gradient method works as follows. First, the image is passed through an edge detection phase (in this case, cv::Canny()). Next, for every nonzero point in the edge image, the local gradient is considered (the gradient is computed by first computing the first-order Sobel x- and y-derivatives via cv::Sobel()). Using this gradient, every point along the line indicated by this slope—from a specified minimum to a specified maximum distance—is incremented in the accumulator. At the same time, the location of every one of these nonzero pixels in the edge image is noted. The candidate centers are then selected from those points in this (two-dimensional) accumulator that are both above some given threshold and larger than all of their immediate neighbors. These candidate centers are sorted in descending order of their accumulator values, so

point along the line indicated by this slope—from a specified minimum to a specified maximum distance—is incremented in the accumulator. At the same time, the location of every one of these nonzero pixels in the edge image is noted. The candidate centers are then selected from those points in this (two-dimensional) accumulator that are both above some given threshold and larger than all of their immediate neighbors. These candidate centers are sorted in descending order of their accumulator values, so that the centers with the most supporting pixels appear first. Next, for each center, all of the nonzero pixels (recall that this list was built earlier) are considered. These pixels are sorted according to their distance from the center. Working out from the smallest distances to the maximum radius, a single radius is selected that is best supported by the nonzero pixels. A center is kept if it has sufficient support from the nonzero pixels in the edge image and if it is a sufficient distance from any

so that the centers with the most supporting pixels appear first. Next, for each center, all of the nonzero pixels (recall that this list was built earlier) are considered. These pixels are sorted according to their distance from the center. Working out from the smallest distances to the maximum radius, a single radius is selected that is best supported by the nonzero pixels. A center is kept if it has sufficient support from the nonzero pixels in the edge image and if it is a sufficient distance from any previously selected center. This implementation enables the algorithm to run much faster and, perhaps more importantly, helps overcome the problem of the otherwise sparse population of a three-dimensional accumulator, which would

lead to a lot of noise and render the results unstable. On the other hand, this algorithm has several shortcomings that you should be aware of.  Figure 6-16: The Hough circle transform finds some of the circles in the test pattern and (correctly) finds none in the photograph First, the use of the Sobel derivatives to compute the local gradient—and the attendant assumption that this can be considered equivalent to a local tangent—is not a numerically stable proposition. It might be true “most of the time,” but you should expect this to generate some noise in the output. Second, the entire set of nonzero pixels in the edge image is considered for every candidate center; hence, if you make the accumulator threshold too low, the algorithm will take a long time to run. Third, because only one circle is selected for every center, if there are concentric circles then you will get only one of them. Finally, because centers are considered in ascending order of their associated accumulator value and because new centers

you should expect this to generate some noise in the output. Second, the entire set of nonzero pixels in the edge image is considered for every candidate center; hence, if you make the accumulator threshold too low, the algorithm will take a long time to run. Third, because only one circle is selected for every center, if there are concentric circles then you will get only one of them. Finally, because centers are considered in ascending order of their associated accumulator value and because new centers are not kept if they are too close to previously accepted centers, there is a bias toward keeping the larger circles when multiple circles are concentric or approximately concentric. (It is only a “bias” because of the noise arising from the Sobel derivatives; in a smooth image at infinite resolution, it would be a certainty.) cv::HoughCircles(), the Hough Circle Transform The Hough circle transform function cv::HoughCircles() has similar arguments to the line transform. void cv::HoughCircles(

centers are not kept if they are too close to previously accepted centers, there is a bias toward keeping the larger circles when multiple circles are concentric or approximately concentric. (It is only a “bias” because of the noise arising from the Sobel derivatives; in a smooth image at infinite resolution, it would be a certainty.) cv::HoughCircles(), the Hough Circle Transform The Hough circle transform function cv::HoughCircles() has similar arguments to the line transform. void cv::HoughCircles(   cv::InputArray  image,                        // Input single channel image   cv::OutputArray circles,                      // N-by-1 3-channel or vector of Vec3f   int             method,                       // Always cv::HOUGH_GRADIENT   double          dp,                           // Accumulator resolution (ratio)   double          minDist,                      // Required separation (between lines)   double          param1    = 100,              // Upper Canny threshold   double          param2    =

cv::InputArray  image,                        // Input single channel image   cv::OutputArray circles,                      // N-by-1 3-channel or vector of Vec3f   int             method,                       // Always cv::HOUGH_GRADIENT   double          dp,                           // Accumulator resolution (ratio)   double          minDist,                      // Required separation (between lines)   double          param1    = 100,              // Upper Canny threshold   double          param2    = 100,              // Un-normalized accumulator threshold   int             minRadius = 0,                // Smallest radius to consider   int             maxRadius = 0                 // Largest radius to consider ); The input image is again an 8-bit image. One significant difference between cv::HoughCircles() and cv::HoughLines() is that the latter requires a binary image. The cv::HoughCircles()

function will internally (automatically) call cv::Sobel()31 for you, so you can provide a more general grayscale image. The result array circles will be either a matrix-array or a vector, depending on what you pass to cv::HoughCircles(). If a matrix is used, it will be a one-dimensional array of type cv::F32C3; the three channels will be used to encode the location of the circle and its radius. If a vector is used, it must be of type std::vector<Vec3f>. The method argument must always be set to cv::HOUGH_GRADIENT. The parameter dp is the resolution of the accumulator image used. This parameter allows us to create an accumulator of a lower resolution than the input image. It makes sense to do this because there is no reason to expect the circles that exist in the image to fall naturally into the same number of bins as the width or height of the image itself. If dp is set to 1 then the resolutions will be the same; if set to a larger number (e.g., 2), then the accumulator resolution will be smaller by that

The parameter dp is the resolution of the accumulator image used. This parameter allows us to create an accumulator of a lower resolution than the input image. It makes sense to do this because there is no reason to expect the circles that exist in the image to fall naturally into the same number of bins as the width or height of the image itself. If dp is set to 1 then the resolutions will be the same; if set to a larger number (e.g., 2), then the accumulator resolution will be smaller by that factor (in this case, half). The value of dp cannot be less than 1. The parameter minDist is the minimum distance that must exist between two circles in order for the algorithm to consider them distinct circles. For the (currently required) case of the method being set to cv::HOUGH_GRADIENT, the next two arguments, param1 and param2, are the edge (Canny) threshold and the accumulator threshold, respectively. You may recall that the Canny edge detector actually takes two different thresholds itself. When cv::Canny() is

case, half). The value of dp cannot be less than 1. The parameter minDist is the minimum distance that must exist between two circles in order for the algorithm to consider them distinct circles. For the (currently required) case of the method being set to cv::HOUGH_GRADIENT, the next two arguments, param1 and param2, are the edge (Canny) threshold and the accumulator threshold, respectively. You may recall that the Canny edge detector actually takes two different thresholds itself. When cv::Canny() is called internally, the first (higher) threshold is set to the value of param1 passed into cv::HoughCircles(), and the second (lower) threshold is set to exactly half that value. The parameter param2 is the one used to threshold the accumulator and is exactly analogous to the threshold argument of cv::HoughLines(). The final two parameters are the minimum and maximum radius of circles that can be found. This means that these are the radii of circles for which the accumulator has a representation. Example 6-5

internally, the first (higher) threshold is set to the value of param1 passed into cv::HoughCircles(), and the second (lower) threshold is set to exactly half that value. The parameter param2 is the one used to threshold the accumulator and is exactly analogous to the threshold argument of cv::HoughLines(). The final two parameters are the minimum and maximum radius of circles that can be found. This means that these are the radii of circles for which the accumulator has a representation. Example 6-5 shows an example program using cv::HoughCircles(). Example 6-5: Using cv::HoughCircles() to return a sequence of circles found in a grayscale image #include <opencv2/opencv.hpp> #include <iostream> #include <math.h>  using namespace cv; using namespace std;  int main(int argc, char** argv) {    if(argc != 2) {      cout << "Hough Circle detect\nUsage: ch6_ex6_1 <imagename>\n" << endl;      return -1;    }        cv::Mat src = cv::imread(argv[1], 1), image;   if( src.empty() ) { cout << "Can not load " << argv[1]

an example program using cv::HoughCircles(). Example 6-5: Using cv::HoughCircles() to return a sequence of circles found in a grayscale image #include <opencv2/opencv.hpp> #include <iostream> #include <math.h>  using namespace cv; using namespace std;  int main(int argc, char** argv) {    if(argc != 2) {      cout << "Hough Circle detect\nUsage: ch6_ex6_1 <imagename>\n" << endl;      return -1;    }        cv::Mat src = cv::imread(argv[1], 1), image;   if( src.empty() ) { cout << "Can not load " << argv[1] << endl; return -1; }   cv::cvtColor(src, image, cv::BGR2GRAY);        cv::GaussianBlur(image, image, Size(5,5), 0, 0);      vector<cv::Vec3f> circles;   cv::HoughCircles(image, circles, cv::HOUGH_GRADIENT, 2, image.cols/10);        for( size_t i = 0; i < circles.size(); ++i ) {     cv::circle(                                                              31 The function cv::Sobel(), not cv::Canny(), is called internally. The reason is that cv::HoughCircles() needs to estimate the orientation of a gradient

<< endl; return -1; }   cv::cvtColor(src, image, cv::BGR2GRAY);        cv::GaussianBlur(image, image, Size(5,5), 0, 0);      vector<cv::Vec3f> circles;   cv::HoughCircles(image, circles, cv::HOUGH_GRADIENT, 2, image.cols/10);        for( size_t i = 0; i < circles.size(); ++i ) {     cv::circle(                                                              31 The function cv::Sobel(), not cv::Canny(), is called internally. The reason is that cv::HoughCircles() needs to estimate the orientation of a gradient at each pixel, and this is difficult to do with binary edge map.

src,        cv::Point(cvRound(circles[i][0]), cvRound(circles[i][1])),       cvRound(circles[i][2]),        cv::Scalar(0,0,255),        2,        cv::AA     );   }   cv::imshow( "Hough Circles", src);   cv::waitKey(0);   return 0;   } It is worth reflecting momentarily on the fact that, no matter what tricks we employ, there is no getting around the requirement that circles be described by three degrees of freedom (𝑥, 𝑦, and 𝑟), in contrast to only two degrees of freedom (𝜌 and 𝜃) for lines. The result will invariably be that any circle-finding algorithm requires more memory and computation time than the line-finding algorithms we looked at previously. With this in mind, it’s a good idea to bound the radius parameter as tightly as circumstances allow in order to keep these costs under control.32 The Hough transform was extended to arbitrary shapes by Ballard in 1981 [Ballard81] basically by considering objects as collections of gradient edges. Distance Transformation The distance transform of an image

will invariably be that any circle-finding algorithm requires more memory and computation time than the line-finding algorithms we looked at previously. With this in mind, it’s a good idea to bound the radius parameter as tightly as circumstances allow in order to keep these costs under control.32 The Hough transform was extended to arbitrary shapes by Ballard in 1981 [Ballard81] basically by considering objects as collections of gradient edges. Distance Transformation The distance transform of an image is defined as a new image in which every output pixel is set to a value equal to the distance to the nearest zero pixel in the input image—according to some specific distance metric. It should be immediately obvious that the typical input to a distance transform should be some kind of edge image. In most applications the input to the distance transform is an output of an edge detector such as the Canny edge detector that has been inverted (so that the edges have value zero and the non-edges are nonzero).

is defined as a new image in which every output pixel is set to a value equal to the distance to the nearest zero pixel in the input image—according to some specific distance metric. It should be immediately obvious that the typical input to a distance transform should be some kind of edge image. In most applications the input to the distance transform is an output of an edge detector such as the Canny edge detector that has been inverted (so that the edges have value zero and the non-edges are nonzero). There are two methods available to compute the distance transform. The first method uses a mask that is typically a 3-by-3 or 5-by-5 array. Each point in the array defines the “distance” to be associated with a point in that particular position relative to the center of the mask. Larger distances are built up (and thus approximated) as sequences of “moves” defined by the entries in the mask. This means that using a larger mask will yield more accurate distances. When using this method, given a specific

There are two methods available to compute the distance transform. The first method uses a mask that is typically a 3-by-3 or 5-by-5 array. Each point in the array defines the “distance” to be associated with a point in that particular position relative to the center of the mask. Larger distances are built up (and thus approximated) as sequences of “moves” defined by the entries in the mask. This means that using a larger mask will yield more accurate distances. When using this method, given a specific distance metric, the appropriate mask is automatically selected from a set known to OpenCV. This is the “original” method developed by Borgefors (1986) [Borgefors86]. The second method computes exact distances, and is due to Felzenszwalb [Felzenszwalb04]. Both methods run in time linear in the total number of pixels, but the exact algorithm is still slower.                                                              32 Although cv::HoughCircles() catches centers of the circles quite well, it sometimes fails

metric, the appropriate mask is automatically selected from a set known to OpenCV. This is the “original” method developed by Borgefors (1986) [Borgefors86]. The second method computes exact distances, and is due to Felzenszwalb [Felzenszwalb04]. Both methods run in time linear in the total number of pixels, but the exact algorithm is still slower.                                                              32 Although cv::HoughCircles() catches centers of the circles quite well, it sometimes fails to find the correct radius. Therefore, in an application where only a center must be found (or where some different technique can be used to find the actual radius), the radius returned by cv::HoughCircles() can be ignored.

Figure 6-17: First a Canny edge detector was run with param1=100 and param2=200; then the distance transform was run with the output scaled by a factor of 5 to increase visibility The distance metric can be any of several different types, including the classic L2 (Cartesian) distance metric. cv::distanceTransform(), for Unlabeled Distance Transform When calling the OpenCV distance transform function, the output image will be a 32-bit floating-point image (i.e., cv::F32). void cv::distanceTransform(   cv::InputArray  src,                         // Input Image   cv::OutputArray dst,                         // Result image   int             distanceType,                // Distance metric to use   int             maskSize                     // Mask to use (3, 5, or see below) ); cv::distanceTransform() has some additional parameters. The first is distanceType, which indicates the distance metric to be used. Your choices here are cv::DIST_C, cv::DIST_L1 and cv::DIST_L2. These methods compute the distance to

src,                         // Input Image   cv::OutputArray dst,                         // Result image   int             distanceType,                // Distance metric to use   int             maskSize                     // Mask to use (3, 5, or see below) ); cv::distanceTransform() has some additional parameters. The first is distanceType, which indicates the distance metric to be used. Your choices here are cv::DIST_C, cv::DIST_L1 and cv::DIST_L2. These methods compute the distance to the nearest zero based on integer steps along a grid. The difference between the methods is that cv::DIST_C is the distance when the steps are counted on a 4-connected grid (i.e., diagonal moves are not allowed), and cv::DIST_L1 gives the number of steps on an 8-connected grid (i.e., diagonal moves are allowed). When distanceType is set to cv::DIST_L2, cv::distanceTransform() attempts to compute the exact Euclidian distances. After the distance type is the maskSize, which may be 3, 5, or  cv::DIST_MASK_PRECISE. In the

zero based on integer steps along a grid. The difference between the methods is that cv::DIST_C is the distance when the steps are counted on a 4-connected grid (i.e., diagonal moves are not allowed), and cv::DIST_L1 gives the number of steps on an 8-connected grid (i.e., diagonal moves are allowed). When distanceType is set to cv::DIST_L2, cv::distanceTransform() attempts to compute the exact Euclidian distances. After the distance type is the maskSize, which may be 3, 5, or  cv::DIST_MASK_PRECISE. In the case of 3 or 5, this argument indicates that a 3-by-3 or 5-by-5 mask should be used with the Borgefors method. If you are using cv::DIST_L1 or cv::DIST_C, you can always use a 3-by-3 mask and you will get exact results. If you are using cv::DIST_L2, the Borgefors method is always approximate, and using the larger 5-by-5 mask will result in a better approximation to the 𝐿! distance, at the cost of a slightly slower computation. Alternatively, cv::DIST_MASK_PRECISE can be used to indicate the Felzenszwalb

case of 3 or 5, this argument indicates that a 3-by-3 or 5-by-5 mask should be used with the Borgefors method. If you are using cv::DIST_L1 or cv::DIST_C, you can always use a 3-by-3 mask and you will get exact results. If you are using cv::DIST_L2, the Borgefors method is always approximate, and using the larger 5-by-5 mask will result in a better approximation to the 𝐿! distance, at the cost of a slightly slower computation. Alternatively, cv::DIST_MASK_PRECISE can be used to indicate the Felzenszwalb algorithm (when used with cv::DIST_L2).  cv::distanceTransform(), for Labeled Distance Transform It is also possible to ask the distance transform algorithm to not only calculate the distances, but to also report which object that minimum distance is to. These “objects” are called connected components. We will have a lot more to say about connected components in Chapter 8, but for now, you can think of them as exactly what they sound like; they are structures made of continuously connected groups of zeros in

(when used with cv::DIST_L2).  cv::distanceTransform(), for Labeled Distance Transform It is also possible to ask the distance transform algorithm to not only calculate the distances, but to also report which object that minimum distance is to. These “objects” are called connected components. We will have a lot more to say about connected components in Chapter 8, but for now, you can think of them as exactly what they sound like; they are structures made of continuously connected groups of zeros in the source image.

void cv::distanceTransform(   cv::InputArray  src,                         // Input Image   cv::OutputArray dst,                         // Result image   cv::OutputArray labels,                      // Output connected componet id’s   int             distanceType,                // Distance metric to use   int             maskSize,                    // Mask to use (3, 5, or see below)   int             labelType = cv::DIST_LABEL_CCOMP  // How to label ); If a labels array is provided, then as a result of running cv::distanceTransform() it will be of the same size as dst. In this case, the computation of connected components will be done automatically, and the label associated with the nearest33 such component will be placed in each pixel of labels. If you are wondering how to differentiate labels, consider that for any pixel that is zero in src, then the corresponding distance must also be zero. In addition to this, the label for that pixel must be the label of the connected component it is part of. As a

running cv::distanceTransform() it will be of the same size as dst. In this case, the computation of connected components will be done automatically, and the label associated with the nearest33 such component will be placed in each pixel of labels. If you are wondering how to differentiate labels, consider that for any pixel that is zero in src, then the corresponding distance must also be zero. In addition to this, the label for that pixel must be the label of the connected component it is part of. As a result, if you want to know what label was given to any particular zero pixel, you need only look up that pixel in labels. The argument labelType can be set either to cv::DIST_LABEL_CCOMP or cv::DIST_LABEL_PIXEL.  In the former case, the function automatically finds connected components of zero pixels in the input image and gives each one a unique label.  In the latter case, all zero pixels are given distinct labels. Histogram Equalization Cameras and image sensors must usually deal not only with the

As a result, if you want to know what label was given to any particular zero pixel, you need only look up that pixel in labels. The argument labelType can be set either to cv::DIST_LABEL_CCOMP or cv::DIST_LABEL_PIXEL.  In the former case, the function automatically finds connected components of zero pixels in the input image and gives each one a unique label.  In the latter case, all zero pixels are given distinct labels. Histogram Equalization Cameras and image sensors must usually deal not only with the contrast in a scene but also with the image sensors’ exposure to the resulting light in that scene. In a standard camera, the shutter and lens aperture settings juggle between exposing the sensors to too much or too little light. Often the range of contrasts is too much for the sensors to deal with; hence, there is a trade-off between capturing the dark areas (e.g., shadows), which requires a longer exposure time, and the bright areas, which require shorter exposure to avoid saturating “whiteouts.”   Figure

in a scene but also with the image sensors’ exposure to the resulting light in that scene. In a standard camera, the shutter and lens aperture settings juggle between exposing the sensors to too much or too little light. Often the range of contrasts is too much for the sensors to deal with; hence, there is a trade-off between capturing the dark areas (e.g., shadows), which requires a longer exposure time, and the bright areas, which require shorter exposure to avoid saturating “whiteouts.”   Figure 6-18: The image on the left has poor contrast, as is confirmed by the histogram of its intensity values on the right After the picture has been taken, there’s nothing we can do about what the sensor recorded; however, we can still take what’s there and try to expand the dynamic range of the image to increase its contrast. The                                                              33 The output “labels” array will basically be the discrete Voronoi diagram.

most commonly used technique for this is histogram equalization.34 35 In   Figure 6-18 we can see that the image on the left is poor because there’s not much variation of the range of values. This is evident from the histogram of its intensity values on the right. Because we are dealing with an 8-bit image, its intensity values can range from 0 to 255, but the histogram shows that the actual intensity values are all clustered near the middle of the available range. Histogram equalization is a method for stretching this range out.  Figure 6-19: Result of cumulative distribution function (left) computed for a Gaussian distribution (right) The underlying math behind histogram equalization involves mapping one distribution (the given histogram of intensity values) to another distribution (a wider and, ideally, uniform distribution of intensity values). That is, we want to spread out the y-values of the original distribution as evenly as possible in the new distribution. It turns out that there is a good answer to

this range out.  Figure 6-19: Result of cumulative distribution function (left) computed for a Gaussian distribution (right) The underlying math behind histogram equalization involves mapping one distribution (the given histogram of intensity values) to another distribution (a wider and, ideally, uniform distribution of intensity values). That is, we want to spread out the y-values of the original distribution as evenly as possible in the new distribution. It turns out that there is a good answer to the problem of spreading out distribution values: the remapping function should be the cumulative distribution function. An example of the cumulative density function is shown in Error! Reference source not found. for the somewhat idealized case of a distribution that was originally pure Gaussian. However, cumulative density can be applied to any distribution; it is just the running sum of the original distribution from its negative to its positive bounds.

of spreading out distribution values: the remapping function should be the cumulative distribution function. An example of the cumulative density function is shown in Error! Reference source not found. for the somewhat idealized case of a distribution that was originally pure Gaussian. However, cumulative density can be applied to any distribution; it is just the running sum of the original distribution from its negative to its positive bounds.                                                              34 If you are wondering why histogram equalization is not in the chapter on histograms (Chapter 7), the reason is that histogram equalization makes no explicit use of any histogram data types. Although histograms are used internally, the function (from the user’s perspective) requires no histograms at all. 35 Histogram equalization is an old mathematical technique; its use in image processing is described in various textbooks [Jain86; Russ02; Acharya05], conference papers [Schwarz78], and even in biological

If you are wondering why histogram equalization is not in the chapter on histograms (Chapter 7), the reason is that histogram equalization makes no explicit use of any histogram data types. Although histograms are used internally, the function (from the user’s perspective) requires no histograms at all. 35 Histogram equalization is an old mathematical technique; its use in image processing is described in various textbooks [Jain86; Russ02; Acharya05], conference papers [Schwarz78], and even in biological vision [Laughlin81].

Figure 6-20: Using the cumulative density function to equalize a Gaussian distribution We may use the cumulative distribution function to remap the original distribution to an equally spread distribution (see  Figure 6-20) simply by looking up each y-value in the original distribution and seeing where it should go in the equalized distribution.  For continuous distributions the result will be an exact equalization, but for digitized/discrete distributions the results may be far from uniform.

Applying this equalization process to   Figure 6-18 yields the equalized intensity distribution histogram and resulting image in Error! Reference source not found..   Figure 6-21: Histogram equalized results: the spectrum has been spread out cv::equalizeHist(), Contrast Equalization OpenCV wraps this whole process up in one neat function. void cv::equalizeHist(   const cv::InputArray  src,                   // Input Image   cv::OutputArray dst                          // Result image ); In cv::equalizeHist(), the source src must be a single-channel, 8-bit image. The destination image dst will be the same. For color images you will have to separate the channels and process them one by one. Segmentation The topic of image segmentation is a large one, which we have touched on in several places already, and will return to in more sophisticated contexts later in the book as well. Here, we will focus on several methods of the library that specifically implement techniques that are either segmentation methods in

the source src must be a single-channel, 8-bit image. The destination image dst will be the same. For color images you will have to separate the channels and process them one by one. Segmentation The topic of image segmentation is a large one, which we have touched on in several places already, and will return to in more sophisticated contexts later in the book as well. Here, we will focus on several methods of the library that specifically implement techniques that are either segmentation methods in themselves, or primitives that will be used later by more sophisticated tactics. It should be noted that at this time, there is no general “magic” solution for image segmentation, and it remains a very active area in

computer vision research. Despite this, many good techniques have been developed that are reliable at least in some specific domain, and in practice can yield very good results. Flood Fill Flood fill [Heckbert00; Shaw04; Vandevenne04] is an extremely useful function that is often used to mark or isolate portions of an image for further processing or analysis. Flood fill can also be used to derive, from an input image, masks that can be used for subsequent routines to speed or restrict processing to only those pixels indicated by the mask. The function cv::floodFill() itself takes an optional mask that can be further used to control where filling is done (e.g., when doing multiple fills of the same image). In OpenCV, flood fill is a more general version of the sort of fill functionality that you probably already associate with typical computer painting programs. For both, a seed point is selected from an image and then all similar neighboring points are colored with a uniform color. The difference is that the

pixels indicated by the mask. The function cv::floodFill() itself takes an optional mask that can be further used to control where filling is done (e.g., when doing multiple fills of the same image). In OpenCV, flood fill is a more general version of the sort of fill functionality that you probably already associate with typical computer painting programs. For both, a seed point is selected from an image and then all similar neighboring points are colored with a uniform color. The difference is that the neighboring pixels need not all be identical in color.36 The result of a flood fill operation will always be a single contiguous region. The cv::floodFill() function will color a neighboring pixel either if it is within a specified range (loDiff to upDiff) of the current pixel or if (depending on the settings of flags) the neighboring pixel is within a specified range of the original seed value.  The cv::floodFill() function will color a neighboring pixel if it is within a specified range (loDiff to upDiff)

pixels need not all be identical in color.36 The result of a flood fill operation will always be a single contiguous region. The cv::floodFill() function will color a neighboring pixel either if it is within a specified range (loDiff to upDiff) of the current pixel or if (depending on the settings of flags) the neighboring pixel is within a specified range of the original seed value.  The cv::floodFill() function will color a neighboring pixel if it is within a specified range (loDiff to upDiff) of either the current pixel or if (depending on the settings of flags) the neighboring pixel is within a specified range of the original seed value. Flood filling can also be constrained by an optional mask argument. There are two different prototypes for the cv::floodFill() routine, one that accepts an explicit mask parameter, and one that does not. int cv::floodFill(   cv::InputOutputArray image,                  // Input image, 1 or 3 channels   cv::Point            seed,                   // Start point for flood

current pixel or if (depending on the settings of flags) the neighboring pixel is within a specified range of the original seed value. Flood filling can also be constrained by an optional mask argument. There are two different prototypes for the cv::floodFill() routine, one that accepts an explicit mask parameter, and one that does not. int cv::floodFill(   cv::InputOutputArray image,                  // Input image, 1 or 3 channels   cv::Point            seed,                   // Start point for flood   cv::Scalar           newVal,                 // Value for painted pixels   cv::Rect*            rect,                   // Output bounds painted domain   cv::Scalar           lowDiff  = cv::Scalar(),// Maximum down color distance   cv::Scalar           highDiff = cv::Scalar(),// Maximum up color distance   int                  flags                   // Local vs. global, and mask-only    );  int cv::floodFill(   cv::InputOutputArray image,                  // Input w-by-h image, 1 or 3 channels

flood   cv::Scalar           newVal,                 // Value for painted pixels   cv::Rect*            rect,                   // Output bounds painted domain   cv::Scalar           lowDiff  = cv::Scalar(),// Maximum down color distance   cv::Scalar           highDiff = cv::Scalar(),// Maximum up color distance   int                  flags                   // Local vs. global, and mask-only    );  int cv::floodFill(   cv::InputOutputArray image,                  // Input w-by-h image, 1 or 3 channels   cv::InputOutputArray mask,                   // Singl-channel 8-bit, w+2-by-h+2   cv::Point            seed,                   // Start point for flood   cv::Scalar           newVal,                 // Value for painted pixels   cv::Rect*            rect,                   // Output bounds painted domain   cv::Scalar           lowDiff  = cv::Scalar(),// Maximum down color distance   cv::Scalar           highDiff = cv::Scalar(),// Maximum up color distance   int                  flags                   //

cv::InputOutputArray mask,                   // Singl-channel 8-bit, w+2-by-h+2   cv::Point            seed,                   // Start point for flood   cv::Scalar           newVal,                 // Value for painted pixels   cv::Rect*            rect,                   // Output bounds painted domain   cv::Scalar           lowDiff  = cv::Scalar(),// Maximum down color distance   cv::Scalar           highDiff = cv::Scalar(),// Maximum up color distance   int                  flags                   // Local vs. global, and mask-only    ); The parameter image is the input image, which can be 8-bit or a floating-point type, and must either have one or three channels. In general, this image array will be modified by cv::floodFill(). The flood fill process begins at the location seed. The seed will be set to value newVal, as will all subsequent pixels colored by the algorithm. A pixel will be colorized if its intensity is not less than a colorized neighbor’s intensity minus loDiff and not greater than the

Local vs. global, and mask-only    ); The parameter image is the input image, which can be 8-bit or a floating-point type, and must either have one or three channels. In general, this image array will be modified by cv::floodFill(). The flood fill process begins at the location seed. The seed will be set to value newVal, as will all subsequent pixels colored by the algorithm. A pixel will be colorized if its intensity is not less than a colorized neighbor’s intensity minus loDiff and not greater than the colorized neighbor’s intensity plus upDiff. If the flags argument includes cv::FLOODFILL_FIXED_RANGE, then a pixel will be compared to the original seed point rather than to its neighbors. Generally, the flags argument controls the connectivity of the fill, what the fill is relative to, whether we are filling only a mask, and what values are used to fill the                                                              36 Users of contemporary painting and drawing programs should note that most of them now

colorized neighbor’s intensity plus upDiff. If the flags argument includes cv::FLOODFILL_FIXED_RANGE, then a pixel will be compared to the original seed point rather than to its neighbors. Generally, the flags argument controls the connectivity of the fill, what the fill is relative to, whether we are filling only a mask, and what values are used to fill the                                                              36 Users of contemporary painting and drawing programs should note that most of them now employ a filling algorithm very much like cv::floodFill().

mask. Our first example of flood fill is shown in  Figure 6-22. The argument mask indicates a mask that can function both as input to cv::floodFill() (in which case, it constrains the regions that can be filled) and as output from cv::floodFill() (in which case, it will indicate the regions that actually were filled). mask must be a single-channel 8-bit image whose size is exactly two pixels larger in width and height than the source image37.  In the sense that mask is an input to cv::floodFill(), the algorithm will not flood across nonzero pixels in the mask. As a result you should zero it before use if you don’t want masking to block the flooding operation.  When the mask is present, it will also be used as an output. When the algorithm runs, every “filled” pixel will be set to a nonzero value in the mask. You have the option of adding the value cv::FLOODFILL_MASK_ONLY to flags (using the usual Boolean OR operator). In this case, the input image will not be modified at all. Instead, only mask will be

algorithm will not flood across nonzero pixels in the mask. As a result you should zero it before use if you don’t want masking to block the flooding operation.  When the mask is present, it will also be used as an output. When the algorithm runs, every “filled” pixel will be set to a nonzero value in the mask. You have the option of adding the value cv::FLOODFILL_MASK_ONLY to flags (using the usual Boolean OR operator). In this case, the input image will not be modified at all. Instead, only mask will be modified.                                                               37 This is done to make processing easier and faster for the internal algorithm. Note that since the mask is larger than the original image, pixel (𝑥,𝑦) in image corresponds to pixel (𝑥+1,𝑦+1) in mask. This is, however, an excellent opportunity to use cv::Mat::getSubRect().

Figure 6-22: Results of flood fill (top image is filled with gray, bottom image with white) from the dark circle located just off center in both images; in this case, the upDiff and loDiff parameters were each set to 7.0 If the flood-fill mask is used, then the mask pixels corresponding to the repainted image pixels are set to 1. Don’t be confused if you fill the mask and see nothing but black upon display; the filled values are there, but the mask image needs to be rescaled if you want to display it in a way you can actually see it on the screen. After all, the difference between 0 and 1 is pretty small on an intensity scale of 0 to 255. Two possible values for flags have already been mentioned: cv::FLOODFILL_FIXED_RANGE and cv::FLOODFILL_MASK_ONLY. In addition to these, you can also add the numerical values 4 or 8.38  In this case, you are specifying whether the flood fill algorithm should consider the pixel array to be four-connected or eight-connected. In the former case, a four-connected array is one in

you can actually see it on the screen. After all, the difference between 0 and 1 is pretty small on an intensity scale of 0 to 255. Two possible values for flags have already been mentioned: cv::FLOODFILL_FIXED_RANGE and cv::FLOODFILL_MASK_ONLY. In addition to these, you can also add the numerical values 4 or 8.38  In this case, you are specifying whether the flood fill algorithm should consider the pixel array to be four-connected or eight-connected. In the former case, a four-connected array is one in which pixels are only connected to their four nearest neighbors (left, right, above, and below). In the latter eight-connected case, pixels are considered to be connected to diagonally neighboring pixels as well. It’s time to clarify the flags argument, which is tricky because it has three parts. The low 8 bits (0–7) can be set to 4 or 8. This controls the connectivity considered by the filling algorithm. If set to 4, only horizontal and vertical neighbors to the current pixel are considered in the filling

which pixels are only connected to their four nearest neighbors (left, right, above, and below). In the latter eight-connected case, pixels are considered to be connected to diagonally neighboring pixels as well. It’s time to clarify the flags argument, which is tricky because it has three parts. The low 8 bits (0–7) can be set to 4 or 8. This controls the connectivity considered by the filling algorithm. If set to 4, only horizontal and vertical neighbors to the current pixel are considered in the filling process; if set to 8, flood fill will additionally include diagonal neighbors. The high 8 bits (16–23) can be set with the flags cv::FLOODFILL_FIXED_RANGE (fill relative to the seed point pixel value; otherwise, fill relative to the neighbor’s value), and/or cv::FLOODFILL_MASK_ONLY (fill the mask location instead of the source image location). Obviously, you must supply an appropriate mask if cv::FLOODFILL_MASK_ONLY is set. The middle bits (8–15) of flags can be set to the value with which you want the

process; if set to 8, flood fill will additionally include diagonal neighbors. The high 8 bits (16–23) can be set with the flags cv::FLOODFILL_FIXED_RANGE (fill relative to the seed point pixel value; otherwise, fill relative to the neighbor’s value), and/or cv::FLOODFILL_MASK_ONLY (fill the mask location instead of the source image location). Obviously, you must supply an appropriate mask if cv::FLOODFILL_MASK_ONLY is set. The middle bits (8–15) of flags can be set to the value with which you want the mask to be filled. If the middle bits of flags are 0s, the mask will be filled with 1s. All these flags may be linked together via OR. For example, if you want an 8-way connectivity fill, filling only a fixed range, filling the mask not the image, and filling using a value of 47, then the parameter to pass in would be: flags = 8       | cv::FLOODFILL_MASK_ONLY       | cv::FLOODFILL_FIXED_RANGE       | (47<<8);                                                               38 The text here reads “add,” but

mask to be filled. If the middle bits of flags are 0s, the mask will be filled with 1s. All these flags may be linked together via OR. For example, if you want an 8-way connectivity fill, filling only a fixed range, filling the mask not the image, and filling using a value of 47, then the parameter to pass in would be: flags = 8       | cv::FLOODFILL_MASK_ONLY       | cv::FLOODFILL_FIXED_RANGE       | (47<<8);                                                               38 The text here reads “add,” but recall that flags is really a bit-field argument. Conveniently, however, 4 and 8 are single bits. So you can use “add” or “OR,” whichever you prefer (e.g., flags = 8 | cv::FLOODFILL_MASK_ONLY).

Figure 6-22 shows flood fill in action on a sample image. Using cv::FLOODFILL_FIXED_RANGE with a wide range resulted in most of the image being filled (starting at the center). We should note that newVal, loDiff, and upDiff are prototyped as type cv::Scalar so they can be set for three channels at once. For example, loDiff = cv::Scalar(20,30,40) will set lo_iff thresholds of 20 for red, 30 for green, and 40 for blue.  Figure 6-23: Results of flood fill (top image is filled with gray, bottom image with white) from the dark circle located just off center in both images; in this case, flood fill was done with a fixed range and with a high and low difference of 25.0 Watershed Algorithm In many practical contexts, we would like to segment an image but do not have the benefit of a separate background image. One technique that is often effective in this context is the watershed algorithm [Meyer92]. This algorithm converts lines in an image into “mountains” and uniform regions into “valleys” that can be used to help

from the dark circle located just off center in both images; in this case, flood fill was done with a fixed range and with a high and low difference of 25.0 Watershed Algorithm In many practical contexts, we would like to segment an image but do not have the benefit of a separate background image. One technique that is often effective in this context is the watershed algorithm [Meyer92]. This algorithm converts lines in an image into “mountains” and uniform regions into “valleys” that can be used to help segment objects. The watershed algorithm first takes the gradient of the intensity image; this has the effect of forming valleys or basins (the low points) where there is no texture and of forming mountains or ranges (high ridges corresponding to edges) where there are dominant lines in the image. It then successively floods basins starting from user-specified (or algorithm-specified) points until these regions meet. Regions that merge across the marks so generated are segmented as belonging together as the

objects. The watershed algorithm first takes the gradient of the intensity image; this has the effect of forming valleys or basins (the low points) where there is no texture and of forming mountains or ranges (high ridges corresponding to edges) where there are dominant lines in the image. It then successively floods basins starting from user-specified (or algorithm-specified) points until these regions meet. Regions that merge across the marks so generated are segmented as belonging together as the image “fills up.” In this way, the basins connected to the marker point become “owned” by that marker. We then segment the image into the corresponding marked regions. More specifically, the watershed algorithm allows a user (or another algorithm!) to mark parts of an object or background that are known to be part of the object or background. The user or algorithm can draw a simple line that effectively tells the watershed algorithm to “group points like these together.” The watershed algorithm then segments the

“fills up.” In this way, the basins connected to the marker point become “owned” by that marker. We then segment the image into the corresponding marked regions. More specifically, the watershed algorithm allows a user (or another algorithm!) to mark parts of an object or background that are known to be part of the object or background. The user or algorithm can draw a simple line that effectively tells the watershed algorithm to “group points like these together.” The watershed algorithm then segments the image by allowing marked regions to “own” the edge-defined

valleys in the gradient image that are connected with the segments.  Figure 6-24 clarifies this process. The function specification of the watershed segmentation algorithm is: void cv::watershed(   cv::InputArray image,                       // Input 8-bit, 3 channels   cv::InputOutputArray markers                // I/O 32-bit float, single channel ); Here, image must be an 8-bit, 3-channel (color) image and markers is a single-channel integer (cv::S32) image of the same (𝑥,𝑦) dimensions. On input, the value of markers is 0 except where the user (or an algorithm) has indicated by using positive numbers that some regions belong together. For example, in the left panel of  Figure 6-24, the orange might have been marked with a 1, the lemon with a 2, the lime with 3, the upper background with 4, and so on.  After the algorithm has run, all of the former zero value pixels in markers will be set to one of the given markers (i.e., all of the pixels of the orange are hoped to come out with a 1 on them, the pixels of

of markers is 0 except where the user (or an algorithm) has indicated by using positive numbers that some regions belong together. For example, in the left panel of  Figure 6-24, the orange might have been marked with a 1, the lemon with a 2, the lime with 3, the upper background with 4, and so on.  After the algorithm has run, all of the former zero value pixels in markers will be set to one of the given markers (i.e., all of the pixels of the orange are hoped to come out with a 1 on them, the pixels of the lemon with a 2, etc.), except the boundary pixels between regions, which will be set to -1.  Figure 6-24 (right) shows an example of such a segmentation.

Figure 6-24: Watershed algorithm: after a user has marked objects that belong together (left), the algorithm then merges the marked area into segments (right) One small word of warning is in order here. It is tempting to think that all regions will be separated by pixels with marker value -1 at their boundaries. This is not actually the case, however. Notably, if two neighboring pixels were input originally with nonzero but distinct values, they will remain touching and not separated by a -1 pixel on output. Grabcuts  The Grabcuts algorithm, introduced by Rother, Kolmogorov, and Blake [Rother04], extends the Graphcuts algorithm [Boykov01] for use in user-directed image segmentation.  The grabcuts algorithm is capable of obtaining excellent segmentations, often with no more than a bounding rectangle around the foreground object to be segmented. This Graphcuts algorithm uses user-labeled foreground and background regions to establish distribution histograms for those two classes of image regions.  It then

Grabcuts  The Grabcuts algorithm, introduced by Rother, Kolmogorov, and Blake [Rother04], extends the Graphcuts algorithm [Boykov01] for use in user-directed image segmentation.  The grabcuts algorithm is capable of obtaining excellent segmentations, often with no more than a bounding rectangle around the foreground object to be segmented. This Graphcuts algorithm uses user-labeled foreground and background regions to establish distribution histograms for those two classes of image regions.  It then combines the assertion that unlabeled foreground or background should conform to similar distributions, with the idea that these regions should tend to be smooth and connected (i.e., a bunch of blobs).  These assertions are combined into an energy functional which gives a low energy (i.e., cost) to solutions which conform to these assertions and a high energy to solutions which violate them.  The final result is obtained by minimizing this energy function39.   The Grabcuts algorithm extends Graphcuts in several

the assertion that unlabeled foreground or background should conform to similar distributions, with the idea that these regions should tend to be smooth and connected (i.e., a bunch of blobs).  These assertions are combined into an energy functional which gives a low energy (i.e., cost) to solutions which conform to these assertions and a high energy to solutions which violate them.  The final result is obtained by minimizing this energy function39.   The Grabcuts algorithm extends Graphcuts in several important ways.  The first is that it replaces the histogram models with a Gaussian mixture model, enabling the algorithm to work on color images.  In addition, it solves the energy functional minimization problem in an iterative manner, which provides better overall results, and allows much greater flexibility in the labeling provided by the user.  Notably, this latter point allows even one-sided labelings which identify either background or foreground pixels (while Graphcuts required both). The

several important ways.  The first is that it replaces the histogram models with a Gaussian mixture model, enabling the algorithm to work on color images.  In addition, it solves the energy functional minimization problem in an iterative manner, which provides better overall results, and allows much greater flexibility in the labeling provided by the user.  Notably, this latter point allows even one-sided labelings which identify either background or foreground pixels (while Graphcuts required both). The implementation in OpenCV allows the caller to either just provide a rectangle around the object to be segmented, in which case the pixels “under” the rectangle’s boundary are taken to be background and no foreground is specified.  Alternatively, the caller can specify an overall mask in which pixels are categorized as being either definitely foreground, definitely background, probably foreground, and probably background.  In this case, the definite regions will be used to classify the other regions, with the

in OpenCV allows the caller to either just provide a rectangle around the object to be segmented, in which case the pixels “under” the rectangle’s boundary are taken to be background and no foreground is specified.  Alternatively, the caller can specify an overall mask in which pixels are categorized as being either definitely foreground, definitely background, probably foreground, and probably background.  In this case, the definite regions will be used to classify the other regions, with the latter being classified into the definite categories by the algorithm. The OpenCV implementation of Grabcuts is implemented by the cv::Grabcuts() function: void cv::grabCut(   cv::InputArray       img,    cv::InputOutputArray mask,    cv::Rect             rect,                                                               39 This minimization is a non-trivial problem. In practice it is performed using a technique called Mincut, which is how both the Graphcuts and Grabcuts algorithms get their respective names.

cv::InputOutputArray bgdModel,     cv::InputOutputArray fgdModel,    int                  iterCount,    int                  mode     = cv::GC_EVAL  ); Given an input image img, the resulting labeling will be computed by cv::grabCut() and placed in the output array mask.  This mask array can also be used as an input.  This is determined by the mode variable.  If mode contains the flag cv::GC_INIT_WITH_MASK40, then the values currently in mask when it is called will be used to initialize the labeling of the image. The mask is expected to be a single channel image of cv::U8 type in which every value is one of the following enumerations:     The argument rect is only used when you are not using mask initialization.  When the mode contains the flag cv::GC_INIT_WITH_RECT, the entire region outside of the provided rectangle is taken to be “definitely background”, while the rest is automatically set to “probably foreground”. The next two arrays are essentially temporary buffers.  When you first call cv::grabCut(),

image. The mask is expected to be a single channel image of cv::U8 type in which every value is one of the following enumerations:     The argument rect is only used when you are not using mask initialization.  When the mode contains the flag cv::GC_INIT_WITH_RECT, the entire region outside of the provided rectangle is taken to be “definitely background”, while the rest is automatically set to “probably foreground”. The next two arrays are essentially temporary buffers.  When you first call cv::grabCut(), they can be empty arrays.  However, if for some reason you should run the Grabcuts algorithm for some number of iterations and then want to restart the algorithm for more iterations (possibly after allowing a user to provide additional “definite” pixels to guide the algorithm) you will need to pass in the same (unmodified) buffers that were filled by the previous run (in addition to using the mask you got back from the previous run as input for the next run). The Grabcuts algorithm essentially runs the

they can be empty arrays.  However, if for some reason you should run the Grabcuts algorithm for some number of iterations and then want to restart the algorithm for more iterations (possibly after allowing a user to provide additional “definite” pixels to guide the algorithm) you will need to pass in the same (unmodified) buffers that were filled by the previous run (in addition to using the mask you got back from the previous run as input for the next run). The Grabcuts algorithm essentially runs the Graphcuts algorithm some number of times.  Between each  such run, the mixture models are recomputed.  The itercount argument determines how many such iterations will be applied.  Typical values for itercount are 10 or 12, though the number required may depend on the size and nature of the image being processed. Mean-Shift Segmentation Mean-shift segmentation finds the peaks of color distributions over space [Comaniciu99]. It is related to the “Mean Shift Algorithm,” which we will discuss in Chapter 10, when

algorithm some number of times.  Between each  such run, the mixture models are recomputed.  The itercount argument determines how many such iterations will be applied.  Typical values for itercount are 10 or 12, though the number required may depend on the size and nature of the image being processed. Mean-Shift Segmentation Mean-shift segmentation finds the peaks of color distributions over space [Comaniciu99]. It is related to the “Mean Shift Algorithm,” which we will discuss in Chapter 10, when we discuss tracking and motion. The main difference between the two is that the former looks at spatial distributions of color (and is thus related to our current topic of segmentation), while the latter tracks those distributions through time in successive                                                              40 Actually, you do not need to explicitly provide the cv::GC_INIT_WITH_MASK flag, because mask initialization is the default behavior.  So as long as you do not provide the cv::GC_INIT_WITH_RECT

discuss tracking and motion. The main difference between the two is that the former looks at spatial distributions of color (and is thus related to our current topic of segmentation), while the latter tracks those distributions through time in successive                                                              40 Actually, you do not need to explicitly provide the cv::GC_INIT_WITH_MASK flag, because mask initialization is the default behavior.  So as long as you do not provide the cv::GC_INIT_WITH_RECT flag, you will get mask initialization.  However, this is not implemented as a default argument, but rather a default in the procedural logic of the function, and is therefore not guaranteed to remain unchanged in future releases of the library.  It is best to either use the cv::GC_INIT_WITH_MASK flag or the cv::GC_INIT_WITH_RECT flag explicitly, both for future proofing, and for general clarity. Enumerated value Numerical value Meaning cv::GC_BGD 0 Background (definitely) cv::GC_FGD 1 Foreground

flag, you will get mask initialization.  However, this is not implemented as a default argument, but rather a default in the procedural logic of the function, and is therefore not guaranteed to remain unchanged in future releases of the library.  It is best to either use the cv::GC_INIT_WITH_MASK flag or the cv::GC_INIT_WITH_RECT flag explicitly, both for future proofing, and for general clarity. Enumerated value Numerical value Meaning cv::GC_BGD 0 Background (definitely) cv::GC_FGD 1 Foreground (definitely) cv::PR_GC_BGD 2 Probably Background cv::PR_GC_FGD 3 Probably Foreground

frames. The function that does this segmentation based on the color distribution peaks is cv::pyrMeanShiftFiltering(). Given a set of multidimensional data points whose dimensions are (𝑥,𝑦,𝐼!"#$,𝐼!"##$,𝐼!"#), mean shift can find the highest density “clumps” of data in this space by scanning a window over the space. Notice, however, that the spatial variables (𝑥,𝑦) can have very different ranges from the color magnitude ranges (𝐼!"#$,𝐼!"##$,𝐼!"#). Therefore, mean shift needs to allow for different window radii in different dimensions. In this case we should have one radius for the spatial variables (spatialRadius) and one radius for the color magnitudes (colorRadius). As mean-shift windows move, all the points traversed by the windows that converge at a peak in the data become connected to or “owned” by that peak. This ownership, radiating out from the densest peaks, forms the segmentation of the image. The segmentation is actually done over a scale pyramid (cv::pyrUp(), cv::pyrDown()), as described in Chapter

in different dimensions. In this case we should have one radius for the spatial variables (spatialRadius) and one radius for the color magnitudes (colorRadius). As mean-shift windows move, all the points traversed by the windows that converge at a peak in the data become connected to or “owned” by that peak. This ownership, radiating out from the densest peaks, forms the segmentation of the image. The segmentation is actually done over a scale pyramid (cv::pyrUp(), cv::pyrDown()), as described in Chapter 5, so that color clusters at a high level in the pyramid (shrunken image) have their boundaries refined at lower pyramid levels in the pyramid.  The output of the mean-shift segmentation algorithm is a new image that has been “posterized,” meaning that the fine texture is removed and the gradients in color are mostly flattened. Such images can then be further segmented using whatever algorithm is appropriate for your needs (e.g., cv::Canny() combined with cv::findContours(), if in fact a contour segmentation

so that color clusters at a high level in the pyramid (shrunken image) have their boundaries refined at lower pyramid levels in the pyramid.  The output of the mean-shift segmentation algorithm is a new image that has been “posterized,” meaning that the fine texture is removed and the gradients in color are mostly flattened. Such images can then be further segmented using whatever algorithm is appropriate for your needs (e.g., cv::Canny() combined with cv::findContours(), if in fact a contour segmentation is what you ultimately want). The function prototype for cv::pyrMeanShiftFiltering() looks like this: void cv::pyrMeanShiftFiltering(   cv::InputArray   src,                       // 8-bit, 3-channel image   cv::OutputArray  dst,                       // 8-bit, 3-channels, same size as src   cv::double       sp,                        // spatial window radius   cv::double       sr,                        // color window radius   int              maxLevel = 1,              // max pyramid level

is what you ultimately want). The function prototype for cv::pyrMeanShiftFiltering() looks like this: void cv::pyrMeanShiftFiltering(   cv::InputArray   src,                       // 8-bit, 3-channel image   cv::OutputArray  dst,                       // 8-bit, 3-channels, same size as src   cv::double       sp,                        // spatial window radius   cv::double       sr,                        // color window radius   int              maxLevel = 1,              // max pyramid level   cv::TermCriteria termcrit = TermCriteria(     cv::TermCriteria::MAX_ITER | cv::TermCriteria::EPS,      5,      1   )   ); In cv::pyrMeanShiftFiltering() we have an input image src and an output image dst. Both must be 8-bit, three-channel color images of the same width and height. The spatialRadius and colorRadius define how the mean-shift algorithm averages color and space together to form a segmentation. For a 640-by-480 color image, it works well to set spatialRadius equal to 2 and colorRadius equal to 40. The next

termcrit = TermCriteria(     cv::TermCriteria::MAX_ITER | cv::TermCriteria::EPS,      5,      1   )   ); In cv::pyrMeanShiftFiltering() we have an input image src and an output image dst. Both must be 8-bit, three-channel color images of the same width and height. The spatialRadius and colorRadius define how the mean-shift algorithm averages color and space together to form a segmentation. For a 640-by-480 color image, it works well to set spatialRadius equal to 2 and colorRadius equal to 40. The next parameter of this algorithm is max_level, which describes how many levels of scale pyramid you want used for segmentation. A max_level of 2 or 3 works well for a 640-by-480 color image. The final parameter is cv::TermCriteria, which we have seen in some previous algorithms. cv::TermCriteria is used for all iterative algorithms in OpenCV. The mean-shift segmentation function comes with good defaults if you just want to leave this parameter blank. Otherwise, cv::TermCriteria has the following constructor:

parameter of this algorithm is max_level, which describes how many levels of scale pyramid you want used for segmentation. A max_level of 2 or 3 works well for a 640-by-480 color image. The final parameter is cv::TermCriteria, which we have seen in some previous algorithms. cv::TermCriteria is used for all iterative algorithms in OpenCV. The mean-shift segmentation function comes with good defaults if you just want to leave this parameter blank. Otherwise, cv::TermCriteria has the following constructor: cv::TermCriteria(     int    type;       // cv::TermCriteria::MAX_ITER, cv::TermCriteria::EPS, or both     int    max_iter,   // maximum iterations when cv::TermCriteria::MAX_ITER is used      double epsilon     // convergence value when cv::TermCriteria::EPS is used ); Typically, we just use the cv::TermCriteria() constructor inline to generate the cv::TermCriteria object that we need. The first argument is either cv::TermCriteria::MAX_ITER or cv::TermCriteria::EPS, which tells the algorithm that we want to

int    type;       // cv::TermCriteria::MAX_ITER, cv::TermCriteria::EPS, or both     int    max_iter,   // maximum iterations when cv::TermCriteria::MAX_ITER is used      double epsilon     // convergence value when cv::TermCriteria::EPS is used ); Typically, we just use the cv::TermCriteria() constructor inline to generate the cv::TermCriteria object that we need. The first argument is either cv::TermCriteria::MAX_ITER or cv::TermCriteria::EPS, which tells the algorithm that we want to terminate either after some fixed number of iterations or when the convergence metric reaches some small value, respectively. The next two arguments set the values at which these criteria should terminate the algorithm. The reason we have both options is because we can set the type to

cv::TermCriteria::MAX_ITER | cv::TermCriteria::EPS to stop when either limit is reached. The parameter max_iter limits the number of iterations if cv::TermCriteria::MAX_ITER is set, whereas epsilon sets the error limit if cv::TermCriteria::EPS is set. Of course the exact meaning of epsilon depends on the algorithm.  Figure 6-25 shows an example of mean-shift segmentation using the following values: cv::pyrMeanShiftFiltering( src, dst, 20, 40, 2);   Figure 6-25: Mean-shift segmentation over scale using cv::pyrMeanShiftFiltering() with parameters max_level=2, spatialRadius=20, and colorRadius=40; similar areas now have similar values and so can be treated as super pixels, which can speed up subsequent processing significantly Image Repair  Images are often corrupted by noise. There may be dust or water spots on the lens, scratches on the older images, or parts of an image that were vandalized. Inpainting [Telea04] is a method for removing such damage by taking the color and texture at the border of the damaged

with parameters max_level=2, spatialRadius=20, and colorRadius=40; similar areas now have similar values and so can be treated as super pixels, which can speed up subsequent processing significantly Image Repair  Images are often corrupted by noise. There may be dust or water spots on the lens, scratches on the older images, or parts of an image that were vandalized. Inpainting [Telea04] is a method for removing such damage by taking the color and texture at the border of the damaged area and propagating and mixing it inside the damaged area. See  Figure 6-26 for an application that involves the removal of writing from an image.

Inpainting  Inpainting works provided the damaged area is not too “thick” and enough of the original texture and color remains around the boundaries of the damage.  Figure 6-27 shows what happens when the damaged area is too large.  Figure 6-26: Inpainting: an image damaged by overwritten text (left) is restored by inpainting (right) The prototype for cv::inpaint() is void cv::inpaint(   cv::InputArray  src,                         // Input Image: 8-bit, 1 or 3 channels   cv::InputArray  inpaintMask,                 // 8-bit, 1 channel. Inpaint non-zeros   cv::OutputArray dst,                         // Result image   double          inpaintRadius,               // Range to consider around pixel   int             flags                        // Select NS or TELEA );    Figure 6-27: Inpainting cannot magically restore textures that are completely removed: the navel of the orange has been completely blotted out (left); inpainting fills it back in with mostly orangelike texture (right)

Here src is an 8-bit single-channel grayscale image or a three-channel color image to be repaired, and inpaintMask is an 8-bit single-channel image of the same size as src in which the damaged areas (e.g., the writing seen in the left panel of  Figure 6-26) have been marked by nonzero pixels; all other pixels are set to 0 in inpaintMask. The output image will be written to dst, which must have the same size and number of channels as src. The inpaintRadius is the area around each inpainted pixel that will be factored into the resulting output color of that pixel. As in  Figure 6-27, interior pixels within a thick enough inpainted region may take their color entirely from other inpainted pixels closer to the boundaries. Almost always, one uses a small radius such as 3 because too large a radius will result in a noticeable blur. Finally, the flags parameter allows you to experiment with two different methods of inpainting: cv::INPAINT_NS (Navier-Stokes method), and cv::INPAINT_TELEA (A. Telea’s method).

will be factored into the resulting output color of that pixel. As in  Figure 6-27, interior pixels within a thick enough inpainted region may take their color entirely from other inpainted pixels closer to the boundaries. Almost always, one uses a small radius such as 3 because too large a radius will result in a noticeable blur. Finally, the flags parameter allows you to experiment with two different methods of inpainting: cv::INPAINT_NS (Navier-Stokes method), and cv::INPAINT_TELEA (A. Telea’s method). Denoising  Another important problem that arises is the problem of noise in the image.  In many applications, the most important source of noise is low-light conditions.  In low light, the gain on the imager must be increased and the result is that noise is also amplified.  The character of this kind of noise is typically random isolated pixels which appear either too bright or too dark, but discoloration is also possible in color images. The denoising algorithm implemented in OpenCV is called Fast

Denoising  Another important problem that arises is the problem of noise in the image.  In many applications, the most important source of noise is low-light conditions.  In low light, the gain on the imager must be increased and the result is that noise is also amplified.  The character of this kind of noise is typically random isolated pixels which appear either too bright or too dark, but discoloration is also possible in color images. The denoising algorithm implemented in OpenCV is called Fast Non-local Means Denoising (FNLMD), and is based on work by Antoni Buades, Bartomeu Coll, and Jean-Michel Morel [Buades05]. While simple denoising algorithms essentially rely on averaging individual pixels with their neighbors, the central concept of FNLMD is to look for similar pixels elsewhere in the image, and average the among those.  In this context a pixel is considered to be a similar pixel not because it is similar in color or intensity, but because it is similar in environment.  The key logic here is that

Means Denoising (FNLMD), and is based on work by Antoni Buades, Bartomeu Coll, and Jean-Michel Morel [Buades05]. While simple denoising algorithms essentially rely on averaging individual pixels with their neighbors, the central concept of FNLMD is to look for similar pixels elsewhere in the image, and average the among those.  In this context a pixel is considered to be a similar pixel not because it is similar in color or intensity, but because it is similar in environment.  The key logic here is that many images contain repetitive structures, and so even if your pixel is corrupted by noise, there will be many other pixels similar to it which are not. The identification of similar pixels proceeds based on a window  𝐵(𝑝,𝑠), centered on pixel 𝑝 and of size  𝑠.  Given such a window around the point we wish to update, we can compare that window with an analogous window around some other pixel  𝑞.  We define the square-distance between 𝐵(𝑝,𝑠) and 𝐵(𝑞,𝑠) to be: 𝑑!𝐵𝑝,𝑠,𝐵𝑞,𝑠=13𝑠!𝐼!𝑝+𝑗−𝐼!𝑞+𝑗!!"#(!,!)!!!!,

where 𝑐 is the color index, 𝐼!(𝑝)  is the intensity of the image in channel 𝑐 at point  𝑝, and the summation over 𝑗 is over the elements of the patch.   From this square-distance, a weight can be assigned to every other pixel relative to the pixel currently being updated.  This weight is given by the formula: 𝑤𝑝,𝑞=𝑒!!"#!!!!!!,!.!!!. In this weight function, 𝜎 is the standard deviation expected in the noise (in intensity units), and ℎ is a generic filtering parameter which determines how quickly patches will become irrelevant as their square-distance grows from the patch we are updating. In general, increasing the value of ℎ will increase the noise removed but at the expense of some of the image detail.  Decreasing the value if ℎ will preserve detail, but also more of the noise. Typically, there is a decreasing return in considering patches very far away (in pixel units) from the pixel being updated, as the number of such patches increases quadratically with the distance allowed.  For this reason, a search

become irrelevant as their square-distance grows from the patch we are updating. In general, increasing the value of ℎ will increase the noise removed but at the expense of some of the image detail.  Decreasing the value if ℎ will preserve detail, but also more of the noise. Typically, there is a decreasing return in considering patches very far away (in pixel units) from the pixel being updated, as the number of such patches increases quadratically with the distance allowed.  For this reason, a search window is defined and only patches in the search window contribute to the update. The update of the current pixel is then given by a simple weighted average of all other pixels in the search window using the given exponentially decaying weight function41.  It is for this reason that the algorithm is called “non-local”; the patches which contribute to the repair of a given pixel are only loosely correlated to the location of the pixel being recomputed. The OpenCV implementation of FNLMD has several different

is defined and only patches in the search window contribute to the update. The update of the current pixel is then given by a simple weighted average of all other pixels in the search window using the given exponentially decaying weight function41.  It is for this reason that the algorithm is called “non-local”; the patches which contribute to the repair of a given pixel are only loosely correlated to the location of the pixel being recomputed. The OpenCV implementation of FNLMD has several different functions in it, each of which applies to slightly different circumstances. Basic FNLMD with cv::fastNlMeansDenoising() void cv::fastNlMeansDenoising(   cv::InputArray         src,                    // Input image   cv::OutputArray        dst,                    // Output image   float                  h                  = 3, // Weight decay parameter   int                    templateWindowSize = 7, // Size of patches used in comparison   int                    searchWindowSize   = 21 // Max distance to patch

in it, each of which applies to slightly different circumstances. Basic FNLMD with cv::fastNlMeansDenoising() void cv::fastNlMeansDenoising(   cv::InputArray         src,                    // Input image   cv::OutputArray        dst,                    // Output image   float                  h                  = 3, // Weight decay parameter   int                    templateWindowSize = 7, // Size of patches used in comparison   int                    searchWindowSize   = 21 // Max distance to patch considered ); The first of these four functions, cv::fastNlMeansDenoising(), implements the algorithm as described exactly.  The result array dst is computed from the input array src using a patch area of templateWindowSize, a decay parameter of h, and patches inside of searchWindowSize distance are considered. The image may be 1, 2, or 3 channel, but must be or type cv::U8.42  Table 6-2 lists some values, provided from the authors of the algorithm, which can be used to help set the decay parameter,  ℎ. Table

); The first of these four functions, cv::fastNlMeansDenoising(), implements the algorithm as described exactly.  The result array dst is computed from the input array src using a patch area of templateWindowSize, a decay parameter of h, and patches inside of searchWindowSize distance are considered. The image may be 1, 2, or 3 channel, but must be or type cv::U8.42  Table 6-2 lists some values, provided from the authors of the algorithm, which can be used to help set the decay parameter,  ℎ. Table 6-2: Recommended values for cv::fastNlMeansDenoising() for grayscale images. Noise: 𝝈 Patch Size: 𝒔 Search Window Decay Parameter: 𝒉 0<𝜎≤15 3 x 3 21 x 21 0.40∙𝜎 15<𝜎≤30 5 x 5 21 x 21 0.40∙𝜎 30<𝜎≤45 7 x 7 35 x 35 0.35∙𝜎 45<𝜎≤75 9 x 9 35 x 35 0.35∙𝜎                                                              41 There is one subtlety here, which is that the weight of the contribution of the pixel p in its own recalculation would be  𝑤𝑝,𝑝=𝑒!=1. In general this results in too high a weight relative to other similar

Recommended values for cv::fastNlMeansDenoising() for grayscale images. Noise: 𝝈 Patch Size: 𝒔 Search Window Decay Parameter: 𝒉 0<𝜎≤15 3 x 3 21 x 21 0.40∙𝜎 15<𝜎≤30 5 x 5 21 x 21 0.40∙𝜎 30<𝜎≤45 7 x 7 35 x 35 0.35∙𝜎 45<𝜎≤75 9 x 9 35 x 35 0.35∙𝜎                                                              41 There is one subtlety here, which is that the weight of the contribution of the pixel p in its own recalculation would be  𝑤𝑝,𝑝=𝑒!=1. In general this results in too high a weight relative to other similar pixels and very little change occurs in the value at  𝑝. For this reason, the weight at p is normally chosen to be the maximum of the weights of the pixels within the area  𝐵(𝑝,𝑠). 42 Note that though this image allows for multiple channels, it is not the best way to handle color images.  For color images, it is better to use cv::fastNlMeansDenoisingColored().

75<𝜎≤100 11 x 11 35 x 35 0.30∙𝜎 FNLMD on Color Images with cv::fastNlMeansDenoisingColored()  void cv::fastNlMeansDenoisingColored(   cv::InputArray         src,                    // Input image   cv::OutputArray        dst,                    // Output image   float                  h                  = 3, // Weight decay parameter   float                  hColor             = 3, // Weight decay parameter for color    int                    templateWindowSize = 7, // Size of patches used in comparison   int                    searchWindowSize   = 21 // Max distance to patch considered ); The second variation of the FNLMD algorithm is used for color images. It accepts only images of type cv::U8C3.  Though it would be possible in principle to apply the algorithm more or less directly to an RGB image, in practice it is better to convert the image to a different color space for the computation.  The function cv::fastNlMeansDenoisingColored() first converts the image to the LAB color space, then applies the

int                    searchWindowSize   = 21 // Max distance to patch considered ); The second variation of the FNLMD algorithm is used for color images. It accepts only images of type cv::U8C3.  Though it would be possible in principle to apply the algorithm more or less directly to an RGB image, in practice it is better to convert the image to a different color space for the computation.  The function cv::fastNlMeansDenoisingColored() first converts the image to the LAB color space, then applies the FNLMD algorithm, then converts the result back to RGB.    The primary advantage of doing this is that in color there are, in effect, three decay parameters. In an RGB representation however it would be unlikely that you would want to set any of them to different values.  In the LAB space however, it is natural to assign a different decay parameter to the luminosity component than to the color components. The function cv::fastNlMeansDenoisingColored() allows you to do just that.  The parameter h is used for

FNLMD algorithm, then converts the result back to RGB.    The primary advantage of doing this is that in color there are, in effect, three decay parameters. In an RGB representation however it would be unlikely that you would want to set any of them to different values.  In the LAB space however, it is natural to assign a different decay parameter to the luminosity component than to the color components. The function cv::fastNlMeansDenoisingColored() allows you to do just that.  The parameter h is used for the luminosity decay parameter, while the new parameter hColor is used for the color channels.  In general the value of hColor will be quite a bit smaller than h.  In most contexts, 10 is a suitable value. Table 6-3: Recommended values for cv::fastNlMeansDenoising() for color images. Noise: 𝝈 Patch Size: 𝒔 Search Window Decay Parameter: 𝒉 0<𝜎≤25 3 x 3 21 x 21 0.55∙𝜎 25<𝜎≤55 5 x 5 35 x 35 0.40∙𝜎 55<𝜎≤100 7 x 7 35 x 35 0.35∙𝜎 FNLMD on Video with cv::fastNlMeansDenoisingMulti() and

The parameter h is used for the luminosity decay parameter, while the new parameter hColor is used for the color channels.  In general the value of hColor will be quite a bit smaller than h.  In most contexts, 10 is a suitable value. Table 6-3: Recommended values for cv::fastNlMeansDenoising() for color images. Noise: 𝝈 Patch Size: 𝒔 Search Window Decay Parameter: 𝒉 0<𝜎≤25 3 x 3 21 x 21 0.55∙𝜎 25<𝜎≤55 5 x 5 35 x 35 0.40∙𝜎 55<𝜎≤100 7 x 7 35 x 35 0.35∙𝜎 FNLMD on Video with cv::fastNlMeansDenoisingMulti() and cv::fastNlMeansDenoisingColoredMulti()  void cv::fastNlMeansDenoisingMulti(   cv::InputArrayOfArrays srcImgs,                // Sequence of several images   cv::OutputArray        dst,                    // Output image   int                    imgToDenoiseIndex,      // Index of image from srcImgs to denoise   int                    temporalWindowSize,     // Number of images to use in denoising (odd)   float                  h                  = 3, // Weight decay parameter   int

and cv::fastNlMeansDenoisingColoredMulti()  void cv::fastNlMeansDenoisingMulti(   cv::InputArrayOfArrays srcImgs,                // Sequence of several images   cv::OutputArray        dst,                    // Output image   int                    imgToDenoiseIndex,      // Index of image from srcImgs to denoise   int                    temporalWindowSize,     // Number of images to use in denoising (odd)   float                  h                  = 3, // Weight decay parameter   int                    templateWindowSize = 7, // Size of patches used for comparison   int                    searchWindowSize   = 21 // Maximum distance to patch considered ); void cv::fastNlMeansDenoisingColoredMulti(   cv::InputArrayOfArrays srcImgs,                // Sequence of several images   cv::OutputArray        dst,                    // Output image   int                    imgToDenoiseIndex,      // Index of image from srcImgs to denoise   int                    temporalWindowSize,     // Number of images to use in

templateWindowSize = 7, // Size of patches used for comparison   int                    searchWindowSize   = 21 // Maximum distance to patch considered ); void cv::fastNlMeansDenoisingColoredMulti(   cv::InputArrayOfArrays srcImgs,                // Sequence of several images   cv::OutputArray        dst,                    // Output image   int                    imgToDenoiseIndex,      // Index of image from srcImgs to denoise   int                    temporalWindowSize,     // Number of images to use in denoising (odd)   float                  h                  = 3, // Weight decay parameter   float                  hColor             = 3, // Weight decay parameter for color dimensions   int                    templateWindowSize = 7, // Size of patches used for comparison   int                    searchWindowSize   = 21 // Maximum distance to patch considered );

The third and fourth variations are used for sequential images, such as those which might be captured from video.  In the case of sequential images, it is natural to imagine that other frames than just the current one might contain useful information for denoising a pixel.  In most applications the noise will not be constant between images, while the signal will likely be similar or even identical.  The functions cv::fastNlMeansDenoisingMulti() and cv::fastNlMeansDenoisingColoredMulti() expect an array of images, srcImgs, rather than a single image.  Additionally, they must be told which image in the sequence is actually to be denoised; this is done with the parameter imgToDenoiseIndex.  Finally, a temporal window must be provided which indicates the number of images from the sequence to be used in the denoising.  This parameter must be odd, and the implied window is always centered on imgToDenoiseIndex. (Thus, if you were to set imgToDenoiseIndex to 4, and temporalWindowSize to 5, the images which would be

images, srcImgs, rather than a single image.  Additionally, they must be told which image in the sequence is actually to be denoised; this is done with the parameter imgToDenoiseIndex.  Finally, a temporal window must be provided which indicates the number of images from the sequence to be used in the denoising.  This parameter must be odd, and the implied window is always centered on imgToDenoiseIndex. (Thus, if you were to set imgToDenoiseIndex to 4, and temporalWindowSize to 5, the images which would be used in the denoising would be 2, 3, 4, 5, and 6.) Summary In this chapter, we learned a variety of methods that can be used to transform images.  These transformations included scale transformations, as well as affine and perspective transformations.  We learned how to remap vector functions from Cartesian to polar representations as well as how to perform discrete Fourier transformations.  Finally, we learned how to perform edge transformations, including both low-level routines to extract edges on a

be used in the denoising would be 2, 3, 4, 5, and 6.) Summary In this chapter, we learned a variety of methods that can be used to transform images.  These transformations included scale transformations, as well as affine and perspective transformations.  We learned how to remap vector functions from Cartesian to polar representations as well as how to perform discrete Fourier transformations.  Finally, we learned how to perform edge transformations, including both low-level routines to extract edges on a pixel-by-pixel level, as well as higher-level routines which would integrate those edges into contours. What all of these functions have in common is their conversion of one image into another through a global operation on the entire image. Exercises 1. In this exercise, we learn to experiment with parameters by setting good lowThresh and highThresh values in cv::Canny(). Load an image with suitably interesting line structures. We’ll use three different high:low threshold settings of 1.5:1, 2.75:1, and 4:1.

level, as well as higher-level routines which would integrate those edges into contours. What all of these functions have in common is their conversion of one image into another through a global operation on the entire image. Exercises 1. In this exercise, we learn to experiment with parameters by setting good lowThresh and highThresh values in cv::Canny(). Load an image with suitably interesting line structures. We’ll use three different high:low threshold settings of 1.5:1, 2.75:1, and 4:1. a) Report what you see with a high setting of less than 50. b) Report what you see with high settings between 50 and 100. c) Report what you see with high settings between 100 and 150. d) Report what you see with high settings between 150 and 200. e) Report what you see with high settings between 200 and 250. f) Summarize your results and explain what happens as best you can. 2. Load an image containing clear lines and circles such as a side view of a bicycle. Use the Hough line and Hough circle calls and see how they

you see with a high setting of less than 50. b) Report what you see with high settings between 50 and 100. c) Report what you see with high settings between 100 and 150. d) Report what you see with high settings between 150 and 200. e) Report what you see with high settings between 200 and 250. f) Summarize your results and explain what happens as best you can. 2. Load an image containing clear lines and circles such as a side view of a bicycle. Use the Hough line and Hough circle calls and see how they respond to your image. 3. Can you think of a way to use the Hough transform to identify any kind of shape with a distinct perimeter? Explain how. 4. Get a product or print out a 1D barcode (having a series of parallel vertical lines in a row) from the web. Use the Line Segment Detector algorithm to create a 1D barcode detector. 5. Look at the diagrams of how the log-polar function transforms a square into a wavy line. a) Draw the log-polar results if the log-polar center point were sitting on one of the

respond to your image. 3. Can you think of a way to use the Hough transform to identify any kind of shape with a distinct perimeter? Explain how. 4. Get a product or print out a 1D barcode (having a series of parallel vertical lines in a row) from the web. Use the Line Segment Detector algorithm to create a 1D barcode detector. 5. Look at the diagrams of how the log-polar function transforms a square into a wavy line. a) Draw the log-polar results if the log-polar center point were sitting on one of the corners of the square. b) What would a circle look like in a log-polar transform if the center point were inside the circle and close to the edge? c) Draw what the transform would look like if the center point were sitting just outside of the circle.

6. A log-polar transform takes shapes of different rotations and sizes into a space where these correspond to shifts in the 𝜃-axis and log  (𝑟) axis. The Fourier transform is translation invariant. How can we use these facts to force shapes of different sizes and rotations to automatically give equivalent representations in the log-polar domain? 7. Draw separate pictures of large, small, large rotated, and small rotated squares. Take the log-polar transform of these each separately. Code up a two-dimensional shifter that takes the center point in the resulting log-polar domain and shifts the shapes to be as identical as possible. 8. Take the Fourier transform of a small Gaussian distribution and the Fourier transform of an image. Multiply them and take the inverse Fourier transform of the results. What have you achieved? As the filters get bigger, you will find that working in the Fourier space is much faster than in the normal space. 9. Load an interesting image, convert it to grayscale, and then take an

shifter that takes the center point in the resulting log-polar domain and shifts the shapes to be as identical as possible. 8. Take the Fourier transform of a small Gaussian distribution and the Fourier transform of an image. Multiply them and take the inverse Fourier transform of the results. What have you achieved? As the filters get bigger, you will find that working in the Fourier space is much faster than in the normal space. 9. Load an interesting image, convert it to grayscale, and then take an integral image of it. Now find vertical and horizontal edges in the image by using the properties of an integral image. Use long skinny rectangles; subtract and add them in place. 10. A good computer vision programming interview question is to code up an integral image. Let’s do that: • Write a routine that takes an image and returns an integral image. • Write a routine that takes an image and returns a 45 degree rotated integral image. • Can you write an integral image for 22.5 degree rotated images? If so,

image of it. Now find vertical and horizontal edges in the image by using the properties of an integral image. Use long skinny rectangles; subtract and add them in place. 10. A good computer vision programming interview question is to code up an integral image. Let’s do that: • Write a routine that takes an image and returns an integral image. • Write a routine that takes an image and returns a 45 degree rotated integral image. • Can you write an integral image for 22.5 degree rotated images? If so, how? 11. Explain how you could use the distance transform to automatically align a known shape with a test shape when the scale is known and held fixed. How would this be done over multiple scales? 12. Practice histogram equalization on images that you load in, and report the results. 13. Load an image, take a perspective transform, and then rotate it. Can this transform be done in one step? In the 2005 DARPA Grand Challenge robot race, the authors on the Stanford team used a kind of color clustering algorithm to

Explain how you could use the distance transform to automatically align a known shape with a test shape when the scale is known and held fixed. How would this be done over multiple scales? 12. Practice histogram equalization on images that you load in, and report the results. 13. Load an image, take a perspective transform, and then rotate it. Can this transform be done in one step? In the 2005 DARPA Grand Challenge robot race, the authors on the Stanford team used a kind of color clustering algorithm to separate road from non-road. The colors were sampled from a laser-defined trapezoid of road patch in front of the car. Other colors in the scene that were close in color to this patch—and whose connected component connected to the original trapezoid—were labeled as road. See Figure 6-28, where the watershed algorithm was used to segment the road after using a trapezoid mark inside the road and an inverted “U” mark outside the road. Suppose we could automatically generate these marks. What could go wrong with

road from non-road. The colors were sampled from a laser-defined trapezoid of road patch in front of the car. Other colors in the scene that were close in color to this patch—and whose connected component connected to the original trapezoid—were labeled as road. See Figure 6-28, where the watershed algorithm was used to segment the road after using a trapezoid mark inside the road and an inverted “U” mark outside the road. Suppose we could automatically generate these marks. What could go wrong with this method of segmenting the road? Hint: Look carefully at Figure 6-28 and then consider that we are trying to extend the road trapezoid by using things that look like what’s in the trapezoid.  Figure 6-28: Using the watershed algorithm to identify a road: markers are put in the original image (left), and the algorithm yields the segmented road (right) 14. In-painting works pretty well for the repair of writing over textured regions. What would happen if the writing obscured a real object edge in a picture? Try

of segmenting the road? Hint: Look carefully at Figure 6-28 and then consider that we are trying to extend the road trapezoid by using things that look like what’s in the trapezoid.  Figure 6-28: Using the watershed algorithm to identify a road: markers are put in the original image (left), and the algorithm yields the segmented road (right) 14. In-painting works pretty well for the repair of writing over textured regions. What would happen if the writing obscured a real object edge in a picture? Try it. 15. How well does inpainting work at fixing up writing drawn over a mean-shift segmented image? Try it for various settings and show the results. 16. Take an image and add noise to it in the following ways:

• Randomly add uniform noise of successively higher ranges. Run the function cv::fastNlMeansDenoisingColored() on it. Report the results. • Do the same, but use Gaussian noise instead. • Do the same, but add shot noise. Change the value of every “𝑁!!” pixel. For example, every 1000th pixel, 500th, 100th, 50th, and 10th.

7 Histograms and Matching In the course of analyzing images, objects, and video information, we frequently want to represent what we are looking at as a histogram. Histograms can be used to represent such diverse things as the color distribution of an object, an edge gradient template of an object [Freeman95], and the distribution of probabilities representing our current hypothesis about an object’s location. Figure 7-1 shows the use of histograms for rapid gesture recognition. Edge gradients were collected from “up,” “right,” “left,” “stop” and “OK” hand gestures. A webcam was then set up to watch a person who used these gestures to control web videos. In each frame, color interest regions were detected from the incoming video; then edge gradient directions were computed around these interest regions, and these directions were collected into orientation bins within a histogram. The histograms were then matched against the gesture models to recognize the gesture. The vertical bars in Figure 7-1 show the

from “up,” “right,” “left,” “stop” and “OK” hand gestures. A webcam was then set up to watch a person who used these gestures to control web videos. In each frame, color interest regions were detected from the incoming video; then edge gradient directions were computed around these interest regions, and these directions were collected into orientation bins within a histogram. The histograms were then matched against the gesture models to recognize the gesture. The vertical bars in Figure 7-1 show the match levels of the different gestures. The gray horizontal line represents the threshold for acceptance of the “winning” vertical bar corresponding to a gesture model. Histograms find uses in many computer vision applications. Histograms are used to detect scene transitions in videos by marking when the edge and color statistics markedly change from frame to frame. They are used to identify interest points in images by assigning each interest point a “tag” consisting of histograms of nearby features. Histograms

levels of the different gestures. The gray horizontal line represents the threshold for acceptance of the “winning” vertical bar corresponding to a gesture model. Histograms find uses in many computer vision applications. Histograms are used to detect scene transitions in videos by marking when the edge and color statistics markedly change from frame to frame. They are used to identify interest points in images by assigning each interest point a “tag” consisting of histograms of nearby features. Histograms of edges, colors, corners, and so on form a general feature type that is passed to classifiers for object recognition. Sequences of color or edge histograms are used to identify whether videos have been copied on the web, where scenes change in a movie, in image retrieval from massive databases, and the list goes on. Histograms are one of the classic tools of computer vision. Histograms are simply collected counts of the underlying data organized into a set of predefined bins. They can be populated by

of edges, colors, corners, and so on form a general feature type that is passed to classifiers for object recognition. Sequences of color or edge histograms are used to identify whether videos have been copied on the web, where scenes change in a movie, in image retrieval from massive databases, and the list goes on. Histograms are one of the classic tools of computer vision. Histograms are simply collected counts of the underlying data organized into a set of predefined bins. They can be populated by counts of features computed from the data, such as gradient magnitudes and directions, color, or just about any other characteristic. In any case, they are used to obtain a statistical picture of the underlying distribution of data. The histogram usually has fewer dimensions than the source data. Figure 7-2 depicts a typical situation. The figure shows a two-dimensional distribution of points (upper-left); we impose a grid (upper-right) and count the data points in each grid cell, yielding a one-dimensional

of features computed from the data, such as gradient magnitudes and directions, color, or just about any other characteristic. In any case, they are used to obtain a statistical picture of the underlying distribution of data. The histogram usually has fewer dimensions than the source data. Figure 7-2 depicts a typical situation. The figure shows a two-dimensional distribution of points (upper-left); we impose a grid (upper-right) and count the data points in each grid cell, yielding a one-dimensional histogram (lower-right). Because the raw data points can represent just about anything, the histogram is a handy way of representing whatever it is that you have learned from your image.

Figure 7-1: Local histograms of gradient orientations are used to find the hand and its gesture; here the “winning” gesture (longest vertical bar) is a correct recognition of “L” (move left) Histograms that represent continuous distributions do so by quantizing the points into each grid cell.1 This is where problems can arise, as shown in Figure 7-3. If the grid is too wide (upper-left), then the output is too coarse and we lose the structure of the distribution. If the grid is too narrow (upper-right), then there is not enough averaging to represent the distribution accurately and we get small, “spiky” cells.  Figure 7-2: Typical histogram example: starting with a cloud of points (upper-left), a counting grid is imposed (upper-right) that yields a one-dimensional histogram of point counts (lower-right) OpenCV has a data type for representing histograms. The histogram data structure is capable of representing histograms in one or many dimensions, and it contains all the data necessary to track bins of both

there is not enough averaging to represent the distribution accurately and we get small, “spiky” cells.  Figure 7-2: Typical histogram example: starting with a cloud of points (upper-left), a counting grid is imposed (upper-right) that yields a one-dimensional histogram of point counts (lower-right) OpenCV has a data type for representing histograms. The histogram data structure is capable of representing histograms in one or many dimensions, and it contains all the data necessary to track bins of both uniform and non-uniform sizes. And, as you might expect, it comes equipped with a variety of useful functions that allow us to easily perform common operations on our histograms.                                                              1 This is also true of histograms representing information that falls naturally into discrete groups when the histogram uses fewer bins than the natural description would suggest or require. An example of this is representing 8-bit intensity values in a 10-bin histogram:

uniform and non-uniform sizes. And, as you might expect, it comes equipped with a variety of useful functions that allow us to easily perform common operations on our histograms.                                                              1 This is also true of histograms representing information that falls naturally into discrete groups when the histogram uses fewer bins than the natural description would suggest or require. An example of this is representing 8-bit intensity values in a 10-bin histogram: each bin would then combine the points associated with approximately 25 different intensities, (erroneously) treating them all as equivalent.

Figure 7-3: A histogram’s accuracy depends on its grid size: a grid that is too wide yields too coarse quantization in the histogram values (left); a grid that is too small yields “spiky” and singleton results from too small samples (right).  Histogram Representation in OpenCV Histograms are represented in OpenCV as arrays, using the same array structures as are used for other data.2 This means that you can use cv::Mat, if you have a one- or two-dimensional array (with the array being 𝑁-by-1 or 1-by-𝑁 in the former case), vector<> types, or sparse matrices. Of course, the interpretation of the array is different, even if the underlying code is identical. For an n-dimensional array, the interpretation is an n-dimensional array of histogram bins, in which the value for any particular element represents the number of counts in that particular bin. This distinction is important in the sense that bins, being just indices into an array of some dimensionality, are simple integers. The identity of the bin, i.e.,

the former case), vector<> types, or sparse matrices. Of course, the interpretation of the array is different, even if the underlying code is identical. For an n-dimensional array, the interpretation is an n-dimensional array of histogram bins, in which the value for any particular element represents the number of counts in that particular bin. This distinction is important in the sense that bins, being just indices into an array of some dimensionality, are simple integers. The identity of the bin, i.e., what it represents, is separate from the bin’s integer index. Whenever you are working with histograms, you will find yourself needing to handle the conversion between measured values and histogram bin indices. Many OpenCV functions will do this task, or some part of this task, for you. When working with higher-dimensional histograms, often most of the entries in that histogram are zero. The cv::SparseMat class is very good for representing such cases. In fact, histograms are the primary reason for the

what it represents, is separate from the bin’s integer index. Whenever you are working with histograms, you will find yourself needing to handle the conversion between measured values and histogram bin indices. Many OpenCV functions will do this task, or some part of this task, for you. When working with higher-dimensional histograms, often most of the entries in that histogram are zero. The cv::SparseMat class is very good for representing such cases. In fact, histograms are the primary reason for the existence of the cv::SparseMat class. Most functions that work on dense arrays will also work on sparse arrays, but we will touch on a few important exceptions in the next section. cv::calcHist(), Creating a Histogram from Data The function cv::calcHist() computes the bin values for a histogram from one or more arrays of data. Recall that the dimensions of the histogram are not related to the dimensionality of the input arrays, or their size, but rather to their number. Each dimension of the histogram

the existence of the cv::SparseMat class. Most functions that work on dense arrays will also work on sparse arrays, but we will touch on a few important exceptions in the next section. cv::calcHist(), Creating a Histogram from Data The function cv::calcHist() computes the bin values for a histogram from one or more arrays of data. Recall that the dimensions of the histogram are not related to the dimensionality of the input arrays, or their size, but rather to their number. Each dimension of the histogram represents counting (and binning) across all pixels values in one of the channels of one of the input arrays. You are not required to use every channel in every image, however; you can pick whatever subset you would like of the channels of the arrays passed to cv::calcHist().                                                              2 This is a substantial change in the C++ API relative to the C API. In the latter, there is a specific structure called a CvHistogram for representing histogram data. The

represents counting (and binning) across all pixels values in one of the channels of one of the input arrays. You are not required to use every channel in every image, however; you can pick whatever subset you would like of the channels of the arrays passed to cv::calcHist().                                                              2 This is a substantial change in the C++ API relative to the C API. In the latter, there is a specific structure called a CvHistogram for representing histogram data. The elimination of this structure in the C++ interface creates a much simpler more unified library.

void cv::calcHist(   const cv::Mat*  images,            // C-style array of images, 8U or 32F   int             nimages,           // number of images in ‘images’ array   const int*      channels,          // C-style list of int’s identifying channels   cv::InputArray  mask,              // pixels in ‘images’ count iff ‘mask’ nonzero   cv::OutputArray hist,              // output histogram array   int             dims,              // histogram dimensionality < cv::MAX_DIMS (32)   const int*      histSize,          // C-style array, histogram sizes in each dim   const float**   ranges,            // C-style array of ‘dims’ pairs set bin sizes   bool            uniform    = true, // true for uniform binning   bool            accumulate = false // if true, add to ‘hist’ rather than replacing );                           void cv::calcHist(    const cv::Mat*  images,            // C-style array of images, 8U or 32F   int             nimages,           // number of images in ‘images’ array   const int*

// C-style array, histogram sizes in each dim   const float**   ranges,            // C-style array of ‘dims’ pairs set bin sizes   bool            uniform    = true, // true for uniform binning   bool            accumulate = false // if true, add to ‘hist’ rather than replacing );                           void cv::calcHist(    const cv::Mat*  images,            // C-style array of images, 8U or 32F   int             nimages,           // number of images in ‘images’ array   const int*      channels,          // C-style list of int’s identifying channels   cv::InputArray  mask,              // pixels in ‘images’ count iff ‘mask’ nonzero   cv::SparseMat&  hist,              // output histogram (sparse) array   int             dims,              // histogram dimensionality < cv::MAX_DIMS (32)   const int*      histSize,          // C-style array, histogram sizes in each dim   const float**   ranges,            // C-style array of ‘dims’ pairs set bin sizes   bool            uniform    = true, // true

// C-style list of int’s identifying channels   cv::InputArray  mask,              // pixels in ‘images’ count iff ‘mask’ nonzero   cv::SparseMat&  hist,              // output histogram (sparse) array   int             dims,              // histogram dimensionality < cv::MAX_DIMS (32)   const int*      histSize,          // C-style array, histogram sizes in each dim   const float**   ranges,            // C-style array of ‘dims’ pairs set bin sizes   bool            uniform    = true, // true for uniform binning   bool            accumulate = false // if true, add to ‘hist’ rather than replacing ); There are three forms of the cv::calcHist() function, two of which use “old-fashioned” C-style arrays, while the third uses the now preferred STL vector template type arguments. The primary distinction between the first two is whether the computed results are to be organized into a dense or a sparse array. The first arguments are the array data, with images being either a pointer to a C-style list of

for uniform binning   bool            accumulate = false // if true, add to ‘hist’ rather than replacing ); There are three forms of the cv::calcHist() function, two of which use “old-fashioned” C-style arrays, while the third uses the now preferred STL vector template type arguments. The primary distinction between the first two is whether the computed results are to be organized into a dense or a sparse array. The first arguments are the array data, with images being either a pointer to a C-style list of arrays or the more modern cv::InputArrayOfArrays. In either case, the role of images is to contain one or more arrays from which the histogram will be constructed. All of these arrays must be the same size, but each can have any number of channels. These arrays may be 8-bit integers or of 32-bit floating point type, but the type of all of the arrays must match. In the case of the C-style array input, the additional argument narrays indicates the number of arrays pointed to by images. The argument channels

arrays or the more modern cv::InputArrayOfArrays. In either case, the role of images is to contain one or more arrays from which the histogram will be constructed. All of these arrays must be the same size, but each can have any number of channels. These arrays may be 8-bit integers or of 32-bit floating point type, but the type of all of the arrays must match. In the case of the C-style array input, the additional argument narrays indicates the number of arrays pointed to by images. The argument channels indicates which channels to consider when creating the histogram. Once again, channels may be a C-style array or an STL vector of integers. These integers identify which channels from the input arrays are to be used for the output histogram. The channels are numbered sequentially, so the first 𝑁!! channels in images[0] are numbered zero through  𝑁!!−1, while the next 𝑁!! channels in images[1] are numbered 𝑁!! through  𝑁!!+𝑁!!−1, and so on. Of course, the number of entries in channels is equal to the number

indicates which channels to consider when creating the histogram. Once again, channels may be a C-style array or an STL vector of integers. These integers identify which channels from the input arrays are to be used for the output histogram. The channels are numbered sequentially, so the first 𝑁!! channels in images[0] are numbered zero through  𝑁!!−1, while the next 𝑁!! channels in images[1] are numbered 𝑁!! through  𝑁!!+𝑁!!−1, and so on. Of course, the number of entries in channels is equal to the number of dimensions of the histogram you are creating. The array mask is an optional mask which, if present, will be used to select which pixels of the arrays in images will contribute to the histogram. mask must be an 8-bit array and the same size as the input arrays. Any pixel in images corresponding to a nonzero pixel in mask will be counted. If you do not wish to use a mask, you can pass cv::noArray() instead. The argument hist is the output histogram you would like to fill. The argument dims is the number

of dimensions of the histogram you are creating. The array mask is an optional mask which, if present, will be used to select which pixels of the arrays in images will contribute to the histogram. mask must be an 8-bit array and the same size as the input arrays. Any pixel in images corresponding to a nonzero pixel in mask will be counted. If you do not wish to use a mask, you can pass cv::noArray() instead. The argument hist is the output histogram you would like to fill. The argument dims is the number of dimensions that histogram will have. Recall that dims is also the number of entries in the channels array, indicating how each dimension is to be computed. The argument histSize may be a C-style array or an STL style vector of integers and indicates the number of bins that should be allocated in each dimension of hist. The number of entries in histSize must also be equal to dims.

Figure 7-4: The ranges argument may be either a C-style array of arrays, or a single STL style vector of floating point numbers. In the case of a uniform histogram, only the minimum and maximum bin edges must be supplied. For a non-uniform histogram, the lower edge of each bin in each dimension must be supplied, as well as the maximum value.  While histSize indicates the number of bins in each dimension, ranges indicates the values that correspond to each bin in each dimension. ranges also can be a C-style array or an STL vector. In the C-style array case, each entry ranges[i] is another array and the length of ranges must be equal to dims. In this case, entry ranges[i] indicates the bin structure of the corresponding 𝑖!! dimension. How ranges[i] is interpreted depends on the value of the argument uniform. If uniform is true, then all of the bins in the 𝑖!! dimension are of the same size, and all that is needed is to specify the (inclusive) lower bound of the lowest bin and the (non-inclusive) upper bound of

array or an STL vector. In the C-style array case, each entry ranges[i] is another array and the length of ranges must be equal to dims. In this case, entry ranges[i] indicates the bin structure of the corresponding 𝑖!! dimension. How ranges[i] is interpreted depends on the value of the argument uniform. If uniform is true, then all of the bins in the 𝑖!! dimension are of the same size, and all that is needed is to specify the (inclusive) lower bound of the lowest bin and the (non-inclusive) upper bound of the highest bin (e.g., ranges[i] = {0,100.0}). If, on the other hand, uniform is false, then if there are 𝑁! bins in the 𝑖!! dimension, there must be 𝑁!+1 entries in ranges[i]. Thus, the 𝑗!! entry will be interpreted as the (inclusive) lower bound of bin j and the (non-inclusive) upper bound of bin  (𝑗−1). In the case in which ranges is of type vector<float>, the entries having the same meaning as the C-style array values, but here they are “flattened” into one single-level array (i.e., for the uniform

of the highest bin (e.g., ranges[i] = {0,100.0}). If, on the other hand, uniform is false, then if there are 𝑁! bins in the 𝑖!! dimension, there must be 𝑁!+1 entries in ranges[i]. Thus, the 𝑗!! entry will be interpreted as the (inclusive) lower bound of bin j and the (non-inclusive) upper bound of bin  (𝑗−1). In the case in which ranges is of type vector<float>, the entries having the same meaning as the C-style array values, but here they are “flattened” into one single-level array (i.e., for the uniform case, there are just two entries in ranges per histogram dimension and they are in the order of the dimensions, while for the non-uniform case, there will be  𝑁!+1 entries per dimension, and they are again all in the order of the dimensions). The final argument accumulate is used to tell OpenCV that the array hist is not to be deleted, reallocated, or otherwise set to zero before adding new counts from the arrays in images. Basic Manipulations with Histograms Even though the data structure for the histogram

there are just two entries in ranges per histogram dimension and they are in the order of the dimensions, while for the non-uniform case, there will be  𝑁!+1 entries per dimension, and they are again all in the order of the dimensions). The final argument accumulate is used to tell OpenCV that the array hist is not to be deleted, reallocated, or otherwise set to zero before adding new counts from the arrays in images. Basic Manipulations with Histograms Even though the data structure for the histogram is the same as the data structure used for matrices and image arrays, this particular interpretation of the data structure invites new operations on these arrays that accomplish tasks specific to histograms. In this section, we will touch on some simple operations that are specific to histograms, as well as review how some important histogram manipulations can be performed with array operations we have already discussed in prior chapters.

Histogram Normalization When dealing with a histogram, we first need to accumulate information into its various bins. Once we have done this, however, it is often desirable to work with the histogram in normalized form, so that individual bins will represent the fraction of the total number of events assigned to the entire histogram. In the C++ API, this can be accomplished simply using the array algebra operators and operations: cv::Mat normalized = my_hist / my_hist.Sum(); or: my_hist /= my_hist.Sum() Histogram Threshold It is also common that you wish to threshold a histogram, and (for example) throw away all bins whose entries contain less than some minimum number of counts. Like normalization, this operation can be accomplished without the use of any particular special histogram routine. Instead, you can use the standard array threshold function: cv::threshold(   my_hist,                // input histogram   my_thresholded_hist,               // result with all values<threshold set to zero   threshold,

Threshold It is also common that you wish to threshold a histogram, and (for example) throw away all bins whose entries contain less than some minimum number of counts. Like normalization, this operation can be accomplished without the use of any particular special histogram routine. Instead, you can use the standard array threshold function: cv::threshold(   my_hist,                // input histogram   my_thresholded_hist,               // result with all values<threshold set to zero   threshold,                         // cutoff value   0,                    // value does not matter in this case   cv::THRESH_TOZERO                  // threshold type ); Finding the Most Populated Bin  In some cases, you would like to find the bins that are above some threshold, and throw away the others. In other cases, you would like to simply find the one bin that has the most weight in it. This is particularly common when the histogram is being used to represent a probability distribution. In this case, the function

// cutoff value   0,                    // value does not matter in this case   cv::THRESH_TOZERO                  // threshold type ); Finding the Most Populated Bin  In some cases, you would like to find the bins that are above some threshold, and throw away the others. In other cases, you would like to simply find the one bin that has the most weight in it. This is particularly common when the histogram is being used to represent a probability distribution. In this case, the function cv::minMaxLoc() will give you what you want. In the case of a two-dimensional array, you can use the cv::InputArray form of cv::minMaxLoc(): void cv::minMaxLoc(   cv::InputArray src,                   // Input array   double*        minVal,                // fill with minimum value (if not NULL)   double*        maxVal = 0,            // fill with maximum value (if not NULL)   cv::Point*     minLoc = 0,            // fill with minimum location (if not NULL)   cv::Point*     maxLoc = 0,            // fill

cv::minMaxLoc() will give you what you want. In the case of a two-dimensional array, you can use the cv::InputArray form of cv::minMaxLoc(): void cv::minMaxLoc(   cv::InputArray src,                   // Input array   double*        minVal,                // fill with minimum value (if not NULL)   double*        maxVal = 0,            // fill with maximum value (if not NULL)   cv::Point*     minLoc = 0,            // fill with minimum location (if not NULL)   cv::Point*     maxLoc = 0,            // fill with maximum location (if not NULL)    cv::InputArray mask   = cv::noArray() // ignore points for which mask is zero ); The arguments minVal and maxVal are pointers to locations you provide for cv::minMaxLoc() to store the minimum and maximum values that have been identified. Similarly, minLoc and maxLoc are pointers to variables (of type cv::Point, in this case) where the actual locations of the minimum and maximum can be stored. If you do not want one or more of these four results to be computed, you can

with maximum location (if not NULL)    cv::InputArray mask   = cv::noArray() // ignore points for which mask is zero ); The arguments minVal and maxVal are pointers to locations you provide for cv::minMaxLoc() to store the minimum and maximum values that have been identified. Similarly, minLoc and maxLoc are pointers to variables (of type cv::Point, in this case) where the actual locations of the minimum and maximum can be stored. If you do not want one or more of these four results to be computed, you can simply pass NULL for that (pointer) variable and that information will not be computed:  double    max_val; cv::Point max_pt;  cv::minMaxLoc(   my_hist,    // input histogram   NULL,  // don’t care about the min value   &max_val,   // place to put the maximum value   NULL,       // don’t care about the location of the min value   &max_pt     // place to put the maximum value location (a cv::Point) );

In this example, though, the histogram would need to be two-dimensional.3 If your histogram is of sparse array type, then there is no problem. Recall that there is an alternate form of cv::minMaxLoc()for sparse arrays: void cv::minMaxLoc(   const cv::SparseMat& src,                   // Input (sparse) array   double*              minVal,                // fill minimum value (if not NULL)   double*              maxVal = 0,            // fill maximum value (if not NULL)   cv::Point*           minLoc = 0,            // fill minimum location (if not NULL)   cv::Point*           maxLoc = 0,            // fill maximum location (if not NULL)    cv::InputArray       mask   = cv::noArray() // ignore points if mask is zero ); It should be noted that this form of cv::minMaxLoc() actually differs from the previous form in several ways. In addition to taking a sparse matrix as the source, it also takes type int* for the minIdx and maxIdx variables instead of cv::Point* for the analogous minLoc and maxLoc variables. This

// fill minimum location (if not NULL)   cv::Point*           maxLoc = 0,            // fill maximum location (if not NULL)    cv::InputArray       mask   = cv::noArray() // ignore points if mask is zero ); It should be noted that this form of cv::minMaxLoc() actually differs from the previous form in several ways. In addition to taking a sparse matrix as the source, it also takes type int* for the minIdx and maxIdx variables instead of cv::Point* for the analogous minLoc and maxLoc variables. This is because the sparse matrix form of cv::minMaxLoc() supports arrays of any dimensionality. Therefore, you need to allocate the location variables yourself and make sure that they have the correct amount of space available for the n-dimensional index associated with a point in the (n-dimensional) sparse histogram: double maxval; int*   max_pt = new int[ my_hist.dims() ];  cv::minMaxLoc(   my_hist,                 // input sparse histogram   NULL,               // don’t care about the min value   &max_val,

is because the sparse matrix form of cv::minMaxLoc() supports arrays of any dimensionality. Therefore, you need to allocate the location variables yourself and make sure that they have the correct amount of space available for the n-dimensional index associated with a point in the (n-dimensional) sparse histogram: double maxval; int*   max_pt = new int[ my_hist.dims() ];  cv::minMaxLoc(   my_hist,                 // input sparse histogram   NULL,               // don’t care about the min value   &max_val,                // place to put the maximum value   NULL,                    // don’t care about the location of the min value   &max_pt                  // place to put the maximum value location (a cv::Point) ); It turns out that if you want to find the minimum or maximum of an n-dimensional array that is not sparse, you need to use another function. This function works essentially the same as cv::minMaxLoc(), and has a similar name, but is not quite the same creature: void cv::minMaxIdx(   cv::InputArray

// place to put the maximum value   NULL,                    // don’t care about the location of the min value   &max_pt                  // place to put the maximum value location (a cv::Point) ); It turns out that if you want to find the minimum or maximum of an n-dimensional array that is not sparse, you need to use another function. This function works essentially the same as cv::minMaxLoc(), and has a similar name, but is not quite the same creature: void cv::minMaxIdx(   cv::InputArray src,    double*        minVal,                // will put minimum value (if not NULL)   double*        maxVal = 0,            // will put maximum value (if not NULL)   int*           minLoc = 0,            // will put min location indices (if not NULL)   int*           maxLoc = 0,            // will put max location indices (if not NULL)    cv::InputArray mask   = cv::noArray() // ignore points if mask is zero ); In this case, the arguments have the same meanings as the corresponding arguments in the two

src,    double*        minVal,                // will put minimum value (if not NULL)   double*        maxVal = 0,            // will put maximum value (if not NULL)   int*           minLoc = 0,            // will put min location indices (if not NULL)   int*           maxLoc = 0,            // will put max location indices (if not NULL)    cv::InputArray mask   = cv::noArray() // ignore points if mask is zero ); In this case, the arguments have the same meanings as the corresponding arguments in the two forms of cv::minMaxLoc(). You must allocate minIdx and maxIdx to C-style arrays of the correct size yourself (as before). One word of warning is in order here, however. If the input array src is one dimensional, you should allocate minIdx and maxIdx to be of dimension two. The reason for this is that cv::minMaxIdx() treats a one-dimensional array as a two-dimensional array internally. As a result, if the maximum is found at position 𝑘, the return value for maxIdx will be (k,0) for a single-column matrix and

of cv::minMaxLoc(). You must allocate minIdx and maxIdx to C-style arrays of the correct size yourself (as before). One word of warning is in order here, however. If the input array src is one dimensional, you should allocate minIdx and maxIdx to be of dimension two. The reason for this is that cv::minMaxIdx() treats a one-dimensional array as a two-dimensional array internally. As a result, if the maximum is found at position 𝑘, the return value for maxIdx will be (k,0) for a single-column matrix and (0,k) for a single-row matrix. Comparing Two Histograms Yet another indispensable tool for working with histograms, first introduced by Swain and Ballard [Swain91] and further generalized by Schiele and Crowley [Schiele96], is the ability to compare two                                                              3 If you have a one-dimensional vector<> array, you can just use cv::Mat( vec ).reshape(1) to make it a 𝑁-by-1 array in two dimensions.

histograms in terms of some specific criteria for similarity. The function cv::compareHist() does just this. double cv::compareHist(   cv::InputArray       H1,        // First histogram to be compared   cv::InputArray       H2,        // Second histogram to be compared   int                  method     // method for the comparison (see options below) );  double cv::compareHist(   const cv::SparseMat& H1,        // First histogram to be compared   const cv::SparseMat& H2,        // Second histogram to be compared   int                  method     // method for the comparison (see options below) ); The first two arguments are the histograms to be compared, which should be of the same size. The third argument is where we select our desired distance metric. The four available options are as follows.  Correlation (method = cv::COMP_CORREL) The first comparison is based on statistical correlation; it implements the Pearson Correlation Coefficient, and is typically appropriate when 𝐻! and 𝐻! can be interpreted as

int                  method     // method for the comparison (see options below) ); The first two arguments are the histograms to be compared, which should be of the same size. The third argument is where we select our desired distance metric. The four available options are as follows.  Correlation (method = cv::COMP_CORREL) The first comparison is based on statistical correlation; it implements the Pearson Correlation Coefficient, and is typically appropriate when 𝐻! and 𝐻! can be interpreted as probability distributions. 𝑑!"##$%𝐻!,𝐻!=𝐻!!(𝑖)∙𝐻!!(𝑖)!𝐻!!!(𝑖)∙𝐻!!!(𝑖)!! Here, 𝐻!!𝑖=𝐻!𝑖−𝑁!!ℎ!(𝑗)! and 𝑁 is equal to the number of bins in the histogram. For correlation, a high score represents a better match than a low score. A perfect match is 1 and a maximal mismatch is –1; a value of 0 indicates no correlation (random association). Chi-square (method = cv::COMP_CHISQR) For chi-square,4 distance metric is based on the chi-squared test statistic, which is a test as to whether or not two distributions are in fact

distributions. 𝑑!"##$%𝐻!,𝐻!=𝐻!!(𝑖)∙𝐻!!(𝑖)!𝐻!!!(𝑖)∙𝐻!!!(𝑖)!! Here, 𝐻!!𝑖=𝐻!𝑖−𝑁!!ℎ!(𝑗)! and 𝑁 is equal to the number of bins in the histogram. For correlation, a high score represents a better match than a low score. A perfect match is 1 and a maximal mismatch is –1; a value of 0 indicates no correlation (random association). Chi-square (method = cv::COMP_CHISQR) For chi-square,4 distance metric is based on the chi-squared test statistic, which is a test as to whether or not two distributions are in fact correlated. 𝑑!!!!!"#$%&𝐻!,𝐻!=𝐻!𝑖−𝐻!𝑖!𝐻!𝑖+𝐻!𝑖! For this test, a low score represents a better match than a high score. A perfect match is 0 and a total mismatch is unbounded (depending on the size of the histogram). Intersection (method = cv::COMP_INTERSECT) The histogram intersection method is based on a simple intersection of the two histograms.  This means that it ask, in effect, what do these two have in common, and sums over all of the bins of the histograms. 𝑑!"#$%&$'#!("𝐻!,𝐻!=min  (𝐻!𝑖,𝐻!𝑖!) For this

correlated. 𝑑!!!!!"#$%&𝐻!,𝐻!=𝐻!𝑖−𝐻!𝑖!𝐻!𝑖+𝐻!𝑖! For this test, a low score represents a better match than a high score. A perfect match is 0 and a total mismatch is unbounded (depending on the size of the histogram). Intersection (method = cv::COMP_INTERSECT) The histogram intersection method is based on a simple intersection of the two histograms.  This means that it ask, in effect, what do these two have in common, and sums over all of the bins of the histograms. 𝑑!"#$%&$'#!("𝐻!,𝐻!=min  (𝐻!𝑖,𝐻!𝑖!) For this metric, high scores indicate good matches and low scores indicate bad matches. If both histograms are normalized to 1.0, then a perfect match is 1.0 and a total mismatch is 0.0. Bhattacharyya distance (method = cv::COMP_BHATTACHARYYA) The last option, called the Bhattacharyya distance [Bhattacharyya43], is also a measure of overlap of two distributions.                                                              4 The chi-square test was invented by Karl Pearson [Pearson] who founded the field of

For this metric, high scores indicate good matches and low scores indicate bad matches. If both histograms are normalized to 1.0, then a perfect match is 1.0 and a total mismatch is 0.0. Bhattacharyya distance (method = cv::COMP_BHATTACHARYYA) The last option, called the Bhattacharyya distance [Bhattacharyya43], is also a measure of overlap of two distributions.                                                              4 The chi-square test was invented by Karl Pearson [Pearson] who founded the field of mathematical statistics.

𝑑!"##$%𝐻!,𝐻!=1−𝐻!𝑖∙𝐻!𝑖!𝐻!(𝑖)!𝐻!(𝑖)! In this case, low scores indicate good matches and high scores indicate bad matches. A perfect match is 0.0 and a total mismatch is a 1.0. With cv::COMP_BHATTACHARYYA, a special factor in the code is used to normalize the input histograms. In general, however, you should normalize histograms before comparing them, because concepts like histogram intersection make little sense (even if allowed) without normalization. The simple case depicted in Figure 7-5 should help clarify matters using about the simplest case imaginable: a one-dimensional histogram with only two bins. The model histogram has a 1.0 value in the left bin and a 0.0 value in the right bin. The last three rows show the comparison histograms and the values generated for them by the various metrics (the EMD metric will be explained shortly).  Figure 7-5: Histogram matching measures Figure 7-5 provides a quick reference for the behavior of different matching types. Close inspection of these matching algorithms in

matters using about the simplest case imaginable: a one-dimensional histogram with only two bins. The model histogram has a 1.0 value in the left bin and a 0.0 value in the right bin. The last three rows show the comparison histograms and the values generated for them by the various metrics (the EMD metric will be explained shortly).  Figure 7-5: Histogram matching measures Figure 7-5 provides a quick reference for the behavior of different matching types. Close inspection of these matching algorithms in the figure will reveal a disconcerting observation. If histogram bins shift by just one slot, as with the chart’s first and third comparison histograms, then all these matching methods (except EMD) yield a maximal mismatch even though these two histograms have a similar “shape.” The rightmost column in Figure 7-5 reports values returned by EMD, a type of distance measure. In comparing the third to the model histogram, the EMD measure quantifies the situation precisely: the third histogram has moved to the

the figure will reveal a disconcerting observation. If histogram bins shift by just one slot, as with the chart’s first and third comparison histograms, then all these matching methods (except EMD) yield a maximal mismatch even though these two histograms have a similar “shape.” The rightmost column in Figure 7-5 reports values returned by EMD, a type of distance measure. In comparing the third to the model histogram, the EMD measure quantifies the situation precisely: the third histogram has moved to the right by one unit. We shall explore this measure further in the “Earth Mover’s Distance” section to follow. In the authors’ experience, intersection works well for quick-and-dirty matching and chi-square or Bhattacharyya work best for slower but more accurate matches. The EMD measure gives the most intuitive matches but is much slower. Histogram Usage Examples It’s probably time for some helpful examples. The program in Example 7-1 (adapted from the OpenCV code bundle) shows how we can use some of the

the right by one unit. We shall explore this measure further in the “Earth Mover’s Distance” section to follow. In the authors’ experience, intersection works well for quick-and-dirty matching and chi-square or Bhattacharyya work best for slower but more accurate matches. The EMD measure gives the most intuitive matches but is much slower. Histogram Usage Examples It’s probably time for some helpful examples. The program in Example 7-1 (adapted from the OpenCV code bundle) shows how we can use some of the functions just discussed. This program computes a hue-saturation histogram from an incoming image and then draws that histogram as an illuminated grid.

Example 7-1: Histogram computation and display #include <opencv2/opencv.hpp> #include <iostream>  using namespace std;  int main( int argc, char** argv ){    if(argc != 2) {      cout << "Computer Color Histogram\nUsage: ch7_ex7_1 <imagename>" << endl;      return -1;    }   cv::Mat src = cv::imread(argv[1],1);    if( src.empty() ) { cout << "Can not load " << argv[1] << endl; return -1; }     // Compute the HSV image, and decompose it into separate planes.   //   cv::Mat hsv;    cv::cvtColor(src, hsv, cv::BGR2HSV);    float h_ranges[]      = {0, 180}; // hue is [0, 180]   float s_ranges[]      = {0, 255};   const float* ranges[] = {h_ranges, s_ranges};   int histSize[]        = {20, 2}, ch[] = {0, 1};    cv::Mat hist;    // Compute the histogram   cv::calcHist(&hsv, 1, ch, cv::noArray(), hist, 2, histSize, ranges, true);   cv::normalize(hist, hist, 0, 255, cv::NORM_MINMAX);    int scale = 10;   cv::Mat hist_img(histSize[0]*scale, histSize[1]*scale, CV::U8C3);    // Draw our histogram.   for( int h = 0; h <

float h_ranges[]      = {0, 180}; // hue is [0, 180]   float s_ranges[]      = {0, 255};   const float* ranges[] = {h_ranges, s_ranges};   int histSize[]        = {20, 2}, ch[] = {0, 1};    cv::Mat hist;    // Compute the histogram   cv::calcHist(&hsv, 1, ch, cv::noArray(), hist, 2, histSize, ranges, true);   cv::normalize(hist, hist, 0, 255, cv::NORM_MINMAX);    int scale = 10;   cv::Mat hist_img(histSize[0]*scale, histSize[1]*scale, CV::U8C3);    // Draw our histogram.   for( int h = 0; h < histSize[0]; h++ ) {     for( int s = 0; s < histSize[1]; s++ ){       float hval = hist.at<float>(h, s);       cv::rectangle(         hist_img,          cv::Rect(h*scale,s*scale,scale,scale),         cv::Scalar::all(hval),          -1       );     }   }    cv::imshow("image", src);   cv::imshow("H-S histogram", hist_img);   cv::waitKey();   return 0; } In this example, we have spent a fair amount of time preparing the arguments for cv::calcHist(), which is not uncommon.  In many practical applications, it is useful

h++ ) {     for( int s = 0; s < histSize[1]; s++ ){       float hval = hist.at<float>(h, s);       cv::rectangle(         hist_img,          cv::Rect(h*scale,s*scale,scale,scale),         cv::Scalar::all(hval),          -1       );     }   }    cv::imshow("image", src);   cv::imshow("H-S histogram", hist_img);   cv::waitKey();   return 0; } In this example, we have spent a fair amount of time preparing the arguments for cv::calcHist(), which is not uncommon.  In many practical applications, it is useful to consider the color histograms associated with human skin tone. By way of example, we have histograms taken from a human hand under various lighting conditions in Figure 7-6. The left column shows images of a hand in an indoor environment, a shaded outdoor environment, and a sunlit outdoor environment. In the middle column are the blue, green, and red (BGR) histograms corresponding to the observed flesh tone of the hand. In the right column are the corresponding

HSV histograms, where the vertical axis is 𝑉 (value), the radius is 𝑆 (saturation) and the angle is 𝐻 (hue). Notice that indoors is the darkest, outdoors in shadow in is a bit brighter, and outdoors in the sun is the brightest. Note also that the colors shift around somewhat as a result of the changing color of the illuminating light.  Figure 7-6: Histogram of flesh colors under indoor (upper-left), shadowed outdoor (middle left), and direct sun outdoor (lower-left) lighting conditions; the middle and right-hand columns display the associated BGR and HSV histograms, respectively As a test of histogram comparison, we could take a portion of one palm (e.g., the top half of the indoor palm), and compare the histogram representation of the colors in that image either with the histogram representation of the colors in the remainder of that image or with the histogram representations of the other two hand images. To make a lower dimensional comparison, we use only hue (𝐻) and saturation (𝑆) from an HSV color space.

columns display the associated BGR and HSV histograms, respectively As a test of histogram comparison, we could take a portion of one palm (e.g., the top half of the indoor palm), and compare the histogram representation of the colors in that image either with the histogram representation of the colors in the remainder of that image or with the histogram representations of the other two hand images. To make a lower dimensional comparison, we use only hue (𝐻) and saturation (𝑆) from an HSV color space.  Table 7-1: Histogram comparison, via four matching methods, of palm-flesh colors in upper half of indoor palm with listed variant palm-flesh color.  We used 30 bins for hue, and 32 for saturation. For reference, the expected score for a perfect match is provided in the first row. Comparison CORREL CHISQR INTERSECT BHATTACHARYYA (Perfect Match) (1.0) (0.0) (1.0) (0.0) Indoor lower half 0.96 0.14 0.82 0.2 Outdoor shade 0.09 1.57 0.13 0.8 Outdoor sun –0.0 1.98 0.01 0.99 To put this experiment into practice (see

7-1: Histogram comparison, via four matching methods, of palm-flesh colors in upper half of indoor palm with listed variant palm-flesh color.  We used 30 bins for hue, and 32 for saturation. For reference, the expected score for a perfect match is provided in the first row. Comparison CORREL CHISQR INTERSECT BHATTACHARYYA (Perfect Match) (1.0) (0.0) (1.0) (0.0) Indoor lower half 0.96 0.14 0.82 0.2 Outdoor shade 0.09 1.57 0.13 0.8 Outdoor sun –0.0 1.98 0.01 0.99 To put this experiment into practice (see code later in Example 7-2), we take three images of a hand under different lighting conditions (Figure 7-6). First we construct a histogram from the hand portion of the top image (the dark one), which we will use as our reference. We then compare that histogram to a histogram taken from the hand in the bottom half of that same image, and then to the hands that appear in the next two (whole) images. The first image is an indoor image, the latter two are outdoors. Using 30 bins for hue and 32 for saturation, the

later in Example 7-2), we take three images of a hand under different lighting conditions (Figure 7-6). First we construct a histogram from the hand portion of the top image (the dark one), which we will use as our reference. We then compare that histogram to a histogram taken from the hand in the bottom half of that same image, and then to the hands that appear in the next two (whole) images. The first image is an indoor image, the latter two are outdoors. Using 30 bins for hue and 32 for saturation, the matching results are shown in Table 7-1, in which several things are apparent. First, the distance metrics even with the lower half of the same indoor image are hardly perfect. Second, some of the distance metrics return a small number when the distance is small, and a larger number when it is high, while other metrics do the opposite. This is as we should have expected from the simple analysis of the matching measures shown in Figure 7-5. The recognition results in Example 7-1 are not good however because,

results are shown in Table 7-1, in which several things are apparent. First, the distance metrics even with the lower half of the same indoor image are hardly perfect. Second, some of the distance metrics return a small number when the distance is small, and a larger number when it is high, while other metrics do the opposite. This is as we should have expected from the simple analysis of the matching measures shown in Figure 7-5. The recognition results in Example 7-1 are not good however because, referencing Figure 7-6, we can see that the distribution in H-S space shifts location, especially in

saturation and so, with fine grain bins, the bins will shift (similar to the last row of Figure 7-5). When we use only 2 bins for saturation and 20 for hue, we get better results as shown in Table 7-2. Table 7-2: Histogram comparison, same images as Table 7-1, but 2 bins for saturation, 20 for hue. Comparison CORREL CHISQR INTERSECT BHATTACHARYYA (Perfect Match) (1.0) (0.0) (1.0) (0.0) Indoor lower half 0.99 0.06 0.93 0.08 Outdoor shade 0.43 19.12 0.41 0.56 Outdoor sun 0.7 147.36 0.98 0.55  Some More Complicated Stuff Everything we’ve discussed so far was reasonably basic. Each of the functions provided for a relatively obvious need. Collectively, they form a good foundation for much of what you might want to do with histograms in the context of computer vision (and probably in other contexts as well). At this point we want to look at some more complicated routines available within OpenCV that are extremely useful in certain applications. These routines include a more sophisticated method of comparing two

Stuff Everything we’ve discussed so far was reasonably basic. Each of the functions provided for a relatively obvious need. Collectively, they form a good foundation for much of what you might want to do with histograms in the context of computer vision (and probably in other contexts as well). At this point we want to look at some more complicated routines available within OpenCV that are extremely useful in certain applications. These routines include a more sophisticated method of comparing two histograms as well as tools for computing and/or visualizing which portions of an image contribute to a given portion of a histogram. Earth Mover’s Distance We saw earlier that lighting changes can cause significant shifts in color values (see  Figure 7-6), although such shifts tend not to change the shape of the histogram of color values, but shift the color value locations and thus cause the histogram-matching schemes we’ve learned about to fail. The difficulty with histogram match measures is that they can

as well as tools for computing and/or visualizing which portions of an image contribute to a given portion of a histogram. Earth Mover’s Distance We saw earlier that lighting changes can cause significant shifts in color values (see  Figure 7-6), although such shifts tend not to change the shape of the histogram of color values, but shift the color value locations and thus cause the histogram-matching schemes we’ve learned about to fail. The difficulty with histogram match measures is that they can return a large difference in the case where two histograms are similarly shaped, but only displaced relative to one another. It is often desirable to have a distance measure that performs like a match, but is less sensitive to such displacements. Earth mover’s distance (EMD) [Rubner00] is such a metric; it essentially measures how much work it would take to “shovel” one histogram shape into another, including moving part (or all) of the histogram to a new location. It works in any number of dimensions.

Return again to Figure 7-5; we see the “earth shoveling” nature of EMD’s distance measure in the right-most column. An exact match is a distance of 0.0. Half a match is half a “shovel full,” the amount it would take to spread half of the left histogram into the next slot. Finally, moving the entire histogram one step to the right would require an entire unit of distance (i.e., to change the model histogram into the “totally mismatched” histogram). The EMD algorithm itself is quite general; it allows users to set their own distance metric or their own cost-of-moving matrix. One can record where the histogram “material” flowed from one histogram to another, and one can employ nonlinear distance metrics derived from prior information about the data. The EMD function in OpenCV is cv::EMD(): float cv::EMD(   cv::InputArray  signature1,             // size1-by-(dims+1) float array   cv::InputArray  signature2,             // size2-by-(dims+1) float array   int             distType,               // distance type

to set their own distance metric or their own cost-of-moving matrix. One can record where the histogram “material” flowed from one histogram to another, and one can employ nonlinear distance metrics derived from prior information about the data. The EMD function in OpenCV is cv::EMD(): float cv::EMD(   cv::InputArray  signature1,             // size1-by-(dims+1) float array   cv::InputArray  signature2,             // size2-by-(dims+1) float array   int             distType,               // distance type (e.g., ‘cv::DIST_L1’)   cv::InputArray  cost       = noArray(), // size1-by-size2 array (for cv::DIST_USER)   float*          lowerBound = 0,         // input/output low bound on sig. distance   cv::OutputArray flow       = noArray()  // output, size1-by-size2, from sig1 to sig2 ); Although we’re applying the EMD to histograms, the interface prefers that we talk to it in terms of what the algorithm calls signatures for the first two array parameters. These signatures are arrays that are always of type float

‘cv::DIST_L1’)   cv::InputArray  cost       = noArray(), // size1-by-size2 array (for cv::DIST_USER)   float*          lowerBound = 0,         // input/output low bound on sig. distance   cv::OutputArray flow       = noArray()  // output, size1-by-size2, from sig1 to sig2 ); Although we’re applying the EMD to histograms, the interface prefers that we talk to it in terms of what the algorithm calls signatures for the first two array parameters. These signatures are arrays that are always of type float and consist of rows containing the histogram bin count followed by its coordinates. For the one-dimensional histogram of  Figure 7-5, the signatures (listed array rows) for the left-hand column of histograms (skipping the model) would be as follows: top, [[1,0],[0,1]]; middle, [[0.5,0],[0.5,1]]; bottom, [[0,0],[1,1]]. If we had a bin in a three-dimensional histogram with a bin count of 537 at (x,y,z) index (7, 43, 11), then the signature row for that bin would be [537, 7, 43, 11]. This is how we perform the

consist of rows containing the histogram bin count followed by its coordinates. For the one-dimensional histogram of  Figure 7-5, the signatures (listed array rows) for the left-hand column of histograms (skipping the model) would be as follows: top, [[1,0],[0,1]]; middle, [[0.5,0],[0.5,1]]; bottom, [[0,0],[1,1]]. If we had a bin in a three-dimensional histogram with a bin count of 537 at (x,y,z) index (7, 43, 11), then the signature row for that bin would be [537, 7, 43, 11]. This is how we perform the necessary step of converting histograms into signatures. (We will go through this in a little more detail in Example 7-2.) The parameter distType should be any of: Manhattan distance (cv::DIST_L1), Euclidean distance (cv::DIST_L2), Checkerboard Distance (cv::DIST_C), or a user-defined distance metric (cv::DIST_USER). In the case of the user-defined distance metric, the user supplies this information in the form of a (pre-calculated) cost matrix via the cost argument. (In this case, cost is an n1-by-n2 matrix,

step of converting histograms into signatures. (We will go through this in a little more detail in Example 7-2.) The parameter distType should be any of: Manhattan distance (cv::DIST_L1), Euclidean distance (cv::DIST_L2), Checkerboard Distance (cv::DIST_C), or a user-defined distance metric (cv::DIST_USER). In the case of the user-defined distance metric, the user supplies this information in the form of a (pre-calculated) cost matrix via the cost argument. (In this case, cost is an n1-by-n2 matrix, with n1 and n2 the sizes of signature1 and signature2, respectively.) The argument lowerBound has two functions (one as input, the other as output). As a return value, it is a lower bound on the distance between the centers of mass of the two histograms. In order for this lower

bound to be computed, one of the standard distance metrics must be in use (i.e., not cv::DIST_USER), and the total weights of the two signatures must be the same (as would be the case for normalized histograms). If you supply a lower-bound argument, you must also initialize that variable to a meaningful value. This value is used as the lower bound on separations for which the EMD will be computed at all.5 Of course, if you want the EMD computed no matter what the distance is, you can always initialize lowerBound to zero. The next argument, flow, is an optional 𝑛!-by-𝑛! matrix that can be used to record the flow of mass from the 𝑖!! point of signature1 to the 𝑗!! point of signature2. In essence, this tells you how the mass was rearranged to give the computed total EMD. As an example, suppose we have two histograms, hist1 and hist2, which we want to convert into two signatures, sig1 and sig2. Just to make things more difficult, let’s suppose that these are two-dimensional histograms (as in the preceding code

to zero. The next argument, flow, is an optional 𝑛!-by-𝑛! matrix that can be used to record the flow of mass from the 𝑖!! point of signature1 to the 𝑗!! point of signature2. In essence, this tells you how the mass was rearranged to give the computed total EMD. As an example, suppose we have two histograms, hist1 and hist2, which we want to convert into two signatures, sig1 and sig2. Just to make things more difficult, let’s suppose that these are two-dimensional histograms (as in the preceding code examples) of dimension h_bins by s_bins. Example 7-2 shows how to convert these two histograms into two signatures. Example 7-2: Creating signatures from histograms for EMD; note that this code is the source of the data in Table 7-1, in which the hand histogram is compared in different lighting conditions #include <opencv2/opencv.hpp> #include <iostream>  using namespace std;  void help(){   cout << "\nCall is:\n"    << "./ch7_ex7_3_expanded modelImage0 testImage1 testImage2 badImage3\n\n"    << "for example: "

of dimension h_bins by s_bins. Example 7-2 shows how to convert these two histograms into two signatures. Example 7-2: Creating signatures from histograms for EMD; note that this code is the source of the data in Table 7-1, in which the hand histogram is compared in different lighting conditions #include <opencv2/opencv.hpp> #include <iostream>  using namespace std;  void help(){   cout << "\nCall is:\n"    << "./ch7_ex7_3_expanded modelImage0 testImage1 testImage2 badImage3\n\n"    << "for example: "   << "  ./ch7_ex7_3_expanded HandIndoorColor.jpg HandOutdoorColor.jpg "   << "HandOutdoorSunColor.jpg fruits.jpg\n"    << "\n"; }  // Compare 3 images’ histograms  // int main( int argc, char** argv ) {    if( argc != 5 ) { help(); return -1; }    vector<cv::Mat> src(5);   cv::Mat         tmp;   int             i;    tmp = cv::imread(argv[1], 1);   if(tmp.empty() ) {     cerr << "Error on reading image 1," << argv[1] << "\n" << endl;     help();     return(-1);   }    // Parse the first image into two image

<< "  ./ch7_ex7_3_expanded HandIndoorColor.jpg HandOutdoorColor.jpg "   << "HandOutdoorSunColor.jpg fruits.jpg\n"    << "\n"; }  // Compare 3 images’ histograms  // int main( int argc, char** argv ) {    if( argc != 5 ) { help(); return -1; }    vector<cv::Mat> src(5);   cv::Mat         tmp;   int             i;    tmp = cv::imread(argv[1], 1);   if(tmp.empty() ) {     cerr << "Error on reading image 1," << argv[1] << "\n" << endl;     help();     return(-1);   }    // Parse the first image into two image halves divided halfway on y   //   cv::Size size  = tmp.size();   int width      = size.width;   int height     = size.height;                                                              5 This is important because it is typically possible to compute the lower bound for a pair of histograms much more quickly than the actual EMD. As a result, in many practical cases, if the EMD is above some bound, you probably do not care about the actual value of the EMD, only that it is “too big” (i.e., the things you

halves divided halfway on y   //   cv::Size size  = tmp.size();   int width      = size.width;   int height     = size.height;                                                              5 This is important because it is typically possible to compute the lower bound for a pair of histograms much more quickly than the actual EMD. As a result, in many practical cases, if the EMD is above some bound, you probably do not care about the actual value of the EMD, only that it is “too big” (i.e., the things you are comparing are “not similar”). In this case, it is quite helpful to have cv::EMD() exit once it is known that the EMD value will be big enough that you do not care for the exact value.

int halfheight = height >> 1;   cout << "Getting size [[" << tmp.cols << "] [" << tmp.rows << "]]\n" << endl;   cout << "Got size (w,h): (" << size.width << "," << size.height << ")" << endl;    src[0] = cv::Mat(cv::Size(width,halfheight), CV::U8C3);   src[1] = cv::Mat(cv::Size(width,halfheight), CV::U8C3);    // Divide the first image into top and bottom halfs into src[0] and src[1]   //   cv::Mat_<cv::Vec3b>::iterator tmpit = tmp.begin<cv::Vec3b>();    // top half   cv::Mat_<cv::Vec3b>::iterator s0it = src[0].begin<cv::Vec3b>();   for(i = 0; i < width*halfheight; ++i, ++tmpit, ++s0it) *s0it = *tmpit;    // Bottom half   cv::Mat_<cv::Vec3b>::iterator s1it = src[1].begin<cv::Vec3b>();   for(i = 0; i < width*halfheight; ++i, ++tmpit, ++s1it) *s1it = *tmpit;    // Load the other three images   //   for(i = 2; i<5; ++i){     src[i] = cv::imread(argv[i], 1);     if(src[i].empty()) {       cerr << "Error on reading image " << i << ": " << argv[i] << "\n" << endl;       help();       return(-1);     }   }    //

src[0].begin<cv::Vec3b>();   for(i = 0; i < width*halfheight; ++i, ++tmpit, ++s0it) *s0it = *tmpit;    // Bottom half   cv::Mat_<cv::Vec3b>::iterator s1it = src[1].begin<cv::Vec3b>();   for(i = 0; i < width*halfheight; ++i, ++tmpit, ++s1it) *s1it = *tmpit;    // Load the other three images   //   for(i = 2; i<5; ++i){     src[i] = cv::imread(argv[i], 1);     if(src[i].empty()) {       cerr << "Error on reading image " << i << ": " << argv[i] << "\n" << endl;       help();       return(-1);     }   }    // Compute the HSV image, and decompose it into separate planes.   //   vector<cv::Mat> hsv(5), hist(5), hist_img(5);   int          h_bins      = 8;   int          s_bins      = 8;   int          hist_size[] = { h_bins, s_bins }, ch[] = {0, 1};   float        h_ranges[]  = { 0, 180 };     // hue range is [0,180]   float        s_ranges[]  = { 0, 255 };   const float* ranges[]    = { h_ranges, s_ranges };   int          scale       = 10;    for(i = 0; i<5; ++i) {     cv::cvtColor( src[i], hsv[i], cv::BGR2HSV

Compute the HSV image, and decompose it into separate planes.   //   vector<cv::Mat> hsv(5), hist(5), hist_img(5);   int          h_bins      = 8;   int          s_bins      = 8;   int          hist_size[] = { h_bins, s_bins }, ch[] = {0, 1};   float        h_ranges[]  = { 0, 180 };     // hue range is [0,180]   float        s_ranges[]  = { 0, 255 };   const float* ranges[]    = { h_ranges, s_ranges };   int          scale       = 10;    for(i = 0; i<5; ++i) {     cv::cvtColor( src[i], hsv[i], cv::BGR2HSV );     cv::calcHist( &hsv[i], 1, ch, noArray(), hist[i], 2, hist_size, ranges, true );     cv::normalize( hist[i], hist[i], 0, 255, cv::NORM_MINMAX );     hist_img[i] = cv::Mat::zeros( hist_size[0]*scale, hist_size[1]*scale, CV::U8C3 );      // Draw our histogram For the 5 images     //     for( int h = 0; h < hist_size[0]; h++ )       for( int s = 0; s < hist_size[1]; s++ ){         float hval = hist[i].at<float>(h, s);         cv::rectangle(           hist_img[i],            cv::Rect(h*scale, s*scale,

);     cv::calcHist( &hsv[i], 1, ch, noArray(), hist[i], 2, hist_size, ranges, true );     cv::normalize( hist[i], hist[i], 0, 255, cv::NORM_MINMAX );     hist_img[i] = cv::Mat::zeros( hist_size[0]*scale, hist_size[1]*scale, CV::U8C3 );      // Draw our histogram For the 5 images     //     for( int h = 0; h < hist_size[0]; h++ )       for( int s = 0; s < hist_size[1]; s++ ){         float hval = hist[i].at<float>(h, s);         cv::rectangle(           hist_img[i],            cv::Rect(h*scale, s*scale, scale, scale),           cv::Scalar::all(hval),            -1         );       }   }    // Display   cv::namedWindow( "Source0", 1 );cv::imshow( "Source0", src[0] );

cv::namedWindow( "HS Histogram0", 1 );cv::imshow( "HS Histogram0", hist_img[0] );    cv::namedWindow( "Source1", 1 );cv::imshow( "Source1", src[1] );   cv::namedWindow( "HS Histogram1", 1 ); cv::imshow( "HS Histogram1", hist_img[1] );    cv::namedWindow( "Source2", 1 ); cv::imshow( "Source2", src[2] );   cv::namedWindow( "HS Histogram2", 1 ); cv::imshow( "HS Histogram2", hist_img[2] );    cv::namedWindow( "Source3", 1 ); cv::imshow( "Source3", src[3] );   cv::namedWindow( "HS Histogram3", 1 ); cv::imshow( "HS Histogram3", hist_img[3] );    cv::namedWindow( "Source4", 1 ); cv::imshow( "Source4", src[4] );   cv::namedWindow( "HS Histogram4", 1 ); cv::imshow( "HS Histogram4", hist_img[4] );    // Compare the histogram src0 vs 1, vs 2, vs 3, vs 4   //   cout << "Comparison:\n"     << "Corr                 Chi                 Intersect          Bhat\n"      << endl;    for(i=1; i<5; ++i) {  // For each histogram     cout << "Hist[0] vs Hist[" << i << "]: " << endl;;     for(int j=0; j<4; ++j) { // For each

"HS Histogram3", hist_img[3] );    cv::namedWindow( "Source4", 1 ); cv::imshow( "Source4", src[4] );   cv::namedWindow( "HS Histogram4", 1 ); cv::imshow( "HS Histogram4", hist_img[4] );    // Compare the histogram src0 vs 1, vs 2, vs 3, vs 4   //   cout << "Comparison:\n"     << "Corr                 Chi                 Intersect          Bhat\n"      << endl;    for(i=1; i<5; ++i) {  // For each histogram     cout << "Hist[0] vs Hist[" << i << "]: " << endl;;     for(int j=0; j<4; ++j) { // For each comparison type       cout << "method[" << j << "]: " << cv::compareHist(hist[0],hist[i],j) << "  ";     }     cout << endl;   }    //Do EMD and report   //   vector<cv::Mat> sig(5);   cout << "\nEMD: " << endl;    // Oi Vey, parse histograms to earth movers signatures   //   for( i=0; i<5; ++i) {      vector<cv::Vec3f> sigv;      // (re)normalize histogram to make the bin weights sum to 1.     //     cv::normalize(hist[i], hist[i], 1, 0, cv::NORM_L1);     for( int h = 0; h < h_bins; h++ )       for( int s = 0;

type       cout << "method[" << j << "]: " << cv::compareHist(hist[0],hist[i],j) << "  ";     }     cout << endl;   }    //Do EMD and report   //   vector<cv::Mat> sig(5);   cout << "\nEMD: " << endl;    // Oi Vey, parse histograms to earth movers signatures   //   for( i=0; i<5; ++i) {      vector<cv::Vec3f> sigv;      // (re)normalize histogram to make the bin weights sum to 1.     //     cv::normalize(hist[i], hist[i], 1, 0, cv::NORM_L1);     for( int h = 0; h < h_bins; h++ )       for( int s = 0; s < s_bins; s++ ) {         float bin_val = hist[i].at<float>(h, s);         if( bin_val != 0 )           sigv.push_back( cv::Vec3f(bin_val, (float)h, (float)s));       }      // make Nx3 32fC1 matrix, where N is the number of nonzero histogram bins     //     sig[i] = cv::Mat(sigv).clone().reshape(1);     if( i > 0 )       cout << "Hist[0] vs Hist[" << i << "]: "             << EMD(sig[0], sig[i], cv::DIST_L2) << endl;   }   cv::waitKey(0); }

Back Projection Back projection is a way of recording how well the pixels fit the distribution of pixels in a histogram model. For example, if we have a histogram of flesh color, then we can use back projection to find flesh color areas in an image. The function for doing this kind of lookup has two variations, one for dense arrays, and one for sparse arrays.  Basic Back Projection: cv::calcBackProject() Back-projection computes a vector from the selected channels of the input images just like cv::calcHist(), but instead of accumulating events in the output histogram it reads the output histogram and reports the bin value already present.  In the context of statistics, if you think of the input histogram as a (prior) probability distribution for the particular vector (color) on some object, then back projection is computing the probability that any particular part of the image is in fact drawn from that prior distribution (e.g., part of the object). void cv::calcBackProject(   const cv::Mat*  images,

but instead of accumulating events in the output histogram it reads the output histogram and reports the bin value already present.  In the context of statistics, if you think of the input histogram as a (prior) probability distribution for the particular vector (color) on some object, then back projection is computing the probability that any particular part of the image is in fact drawn from that prior distribution (e.g., part of the object). void cv::calcBackProject(   const cv::Mat*  images,            // C-style array of images, 8U or 32F   int             nimages,           // number of images in ‘images’ array   const int*      channels,          // C-style list of ints identifying channels   cv::InputArray  hist,              // input histogram array   cv::OutputArray backProject,       // output single channel array    const float**   ranges,            // C-style array of ‘dims’ pairs set bin sizes   double          scale      = 1,    // Optional scale factor for output   bool            uniform

// C-style array of images, 8U or 32F   int             nimages,           // number of images in ‘images’ array   const int*      channels,          // C-style list of ints identifying channels   cv::InputArray  hist,              // input histogram array   cv::OutputArray backProject,       // output single channel array    const float**   ranges,            // C-style array of ‘dims’ pairs set bin sizes   double          scale      = 1,    // Optional scale factor for output   bool            uniform    = true  // true for uniform binning );  void cv::calcBackProject(   const cv::Mat*  images,            // C-style array of images, 8U or 32F   int             nimages,           // number of images in ‘images’ array   const int*      channels,          // C-style list of ints identifying channels   const cv::SparseMat& hist,         // input (sparse) histogram array   cv::OutputArray backProject,       // output single channel array    const float**   ranges,            // C-style array of ‘dims’ pairs set

= true  // true for uniform binning );  void cv::calcBackProject(   const cv::Mat*  images,            // C-style array of images, 8U or 32F   int             nimages,           // number of images in ‘images’ array   const int*      channels,          // C-style list of ints identifying channels   const cv::SparseMat& hist,         // input (sparse) histogram array   cv::OutputArray backProject,       // output single channel array    const float**   ranges,            // C-style array of ‘dims’ pairs set bin sizes   double          scale      = 1,    // Optional scale factor for output   bool            uniform    = true  // true for uniform binning );  void cv::calcBackProject(   cv::InputArrayOfArrays images,            // STL-vector of images, 8U or 32F   const vector<int>&     channels,          // STL-vector, channels indices to use   cv::InputArray         hist,              // input histogram array   cv::OutputArray        backProject,       // output single channel array    const vector<float>&

bin sizes   double          scale      = 1,    // Optional scale factor for output   bool            uniform    = true  // true for uniform binning );  void cv::calcBackProject(   cv::InputArrayOfArrays images,            // STL-vector of images, 8U or 32F   const vector<int>&     channels,          // STL-vector, channels indices to use   cv::InputArray         hist,              // input histogram array   cv::OutputArray        backProject,       // output single channel array    const vector<float>&   ranges,            // STL-style vector of range boundaries    double                 scale      = 1,    // Optional scale factor for output   bool                   uniform    = true  // true for uniform binning ); There are three versions of cv::calcBackProject(). The first two use C-style arrays for their inputs. One of these supports dense histograms and one supports sparse histograms. The third version uses the newer template-based inputs rather than C-style pointers6. In both cases, the image is

ranges,            // STL-style vector of range boundaries    double                 scale      = 1,    // Optional scale factor for output   bool                   uniform    = true  // true for uniform binning ); There are three versions of cv::calcBackProject(). The first two use C-style arrays for their inputs. One of these supports dense histograms and one supports sparse histograms. The third version uses the newer template-based inputs rather than C-style pointers6. In both cases, the image is provided in the form of a set of individual single or multichannel arrays (the images variable), while the histogram is precisely the form of histogram that is produced by cv::calcHist() (the hist variable). The set of single-channel arrays is exactly the same form as what you would have used when you called cv::calcHist() in the first place, only this time, it is the image you want to compare your histogram to. If the argument images is a C-style array (type cv::Mat*), you will also need to tell

image is provided in the form of a set of individual single or multichannel arrays (the images variable), while the histogram is precisely the form of histogram that is produced by cv::calcHist() (the hist variable). The set of single-channel arrays is exactly the same form as what you would have used when you called cv::calcHist() in the first place, only this time, it is the image you want to compare your histogram to. If the argument images is a C-style array (type cv::Mat*), you will also need to tell  cv::calcBackProject() how many elements it has; this is the function of the nimages argument.                                                               6 Of these three, the third is the generally preferred form in modern code (i.e., the use of the C-style arrays for input is considered “old-fashioned” in most modern OpenCV code.)

The argument channels is a list of the channels that will actually be used in the back projection. Once again, the form of this argument is the same as the form of the corresponding argument used by cv::calcHist(). Each integer entry in the array channels is related to a channel in the input arrays by enumerating the channels in order, starting with the first array (arrays[0]), then for the second array (images[1]), and so on (e.g., if there were three matrices pointed to by images, with three channels each, their channels would correspond to the values 0, 1, and 2 for the first array, 3, 4, and 5 for the second array, and 6, 7, and 8 for the third array). As you can see, though the number of entries in channels must be the same as the dimensionality of the histogram hist, that number need not be the same as the number of arrays in (or the total number of channels represented by) images.  The results of the back projection computation will be placed in the array backProject. backProject will be the same size

their channels would correspond to the values 0, 1, and 2 for the first array, 3, 4, and 5 for the second array, and 6, 7, and 8 for the third array). As you can see, though the number of entries in channels must be the same as the dimensionality of the histogram hist, that number need not be the same as the number of arrays in (or the total number of channels represented by) images.  The results of the back projection computation will be placed in the array backProject. backProject will be the same size and type as images[0], and have a single channel. Because histogram data is stored in the same matrix structures used for other data, there is no place to record the bin information that was used in the original construction of the histogram. In this sense, to really comprehend a histogram completely, the associated cv::Mat (or cv::SparseMat or whatever) is needed, as well as the original ranges data structure that was used when the histogram was created by cv::calcHist().7 It is for this reason that this

and type as images[0], and have a single channel. Because histogram data is stored in the same matrix structures used for other data, there is no place to record the bin information that was used in the original construction of the histogram. In this sense, to really comprehend a histogram completely, the associated cv::Mat (or cv::SparseMat or whatever) is needed, as well as the original ranges data structure that was used when the histogram was created by cv::calcHist().7 It is for this reason that this range of information needs to be supplied to cv::calcBackProject() in the ranges argument. Finally, there are two optional arguments, scale and uniform. scale is an optional scale factor that is applied to the return values placed in backProject. (This is particularly useful if you want to visualize the results.)  uniform is used to indicate whether or not the input histogram is a uniform histogram (in the sense of cv::calcHist()). Because uniform defaults to true, this argument is only needed for

this range of information needs to be supplied to cv::calcBackProject() in the ranges argument. Finally, there are two optional arguments, scale and uniform. scale is an optional scale factor that is applied to the return values placed in backProject. (This is particularly useful if you want to visualize the results.)  uniform is used to indicate whether or not the input histogram is a uniform histogram (in the sense of cv::calcHist()). Because uniform defaults to true, this argument is only needed for non-uniform histograms. Example 7-1 showed how to convert an image into single-channel planes and then make an array of them. As described above, the values in backProject are set to the values in the associated bin in hist. If the histogram is normalized, then this value can be associated with a conditional probability value (i.e., the                                                              7 An alternative approach would have been to define another data-type for histograms which inherited from cv::Mat,

histograms. Example 7-1 showed how to convert an image into single-channel planes and then make an array of them. As described above, the values in backProject are set to the values in the associated bin in hist. If the histogram is normalized, then this value can be associated with a conditional probability value (i.e., the                                                              7 An alternative approach would have been to define another data-type for histograms which inherited from cv::Mat, but which also contained this bin information. The authors of the library chose not to take this route in the 2.0 (and later) version of the library in favor of simplifying the library.

probability that a pixel in image is a member of the type characterized by the histogram in hist).8 In  Figure 7-7, we use a flesh-color histogram to derive a probability of flesh image.                                                               8 Specifically, in the case of our flesh-tone H-S histogram, if 𝐶 is the color of the pixel and 𝐹 is the probability that a pixel is flesh, then this probability map gives us 𝑝(𝐶|𝐹), the probability of drawing that color if the pixel actually is flesh. This is not quite the same as 𝑝(𝐹|𝐶), the probability that the pixel is flesh given its color. However, these two probabilities are related by Bayes' theorem [Bayes1763] and so, if we know the overall probability of encountering a flesh-colored object in a scene as well as the total probability of encountering of the range of flesh colors, then we can compute 𝑝(𝐹|𝐶) from 𝑝(𝐶|𝐹). Specifically, Bayes' theorem establishes the following relation: 𝑝𝐹𝐶=𝑝𝐹𝑝𝐶𝑝(𝐶|𝐹)

Figure 7-7: Back projection of histogram values onto each pixel based on its color: the HS (Hue and Saturation planes of an HSV representation of the image) flesh-color histogram (upper-left) is used to convert the hand image (upper-right) into the flesh-color probability image (lower-right); the lower-left panel is the histogram of the hand image When backProject is a byte image rather than a float image, you should either not normalize the histogram or else scale it up before use. The reason is that the highest possible value in a normalized histogram is 1, so anything less than that will be rounded down to 0 in the 8-bit image. You might also need to scale backProject in order to see the values with your eyes, depending on how high the values are in your histogram. Template Matching Template matching via cv::matchTemplate() is not based on histograms; rather, the function matches an actual image patch against an input image by “sliding” the patch over the input image using one of the matching methods

the highest possible value in a normalized histogram is 1, so anything less than that will be rounded down to 0 in the 8-bit image. You might also need to scale backProject in order to see the values with your eyes, depending on how high the values are in your histogram. Template Matching Template matching via cv::matchTemplate() is not based on histograms; rather, the function matches an actual image patch against an input image by “sliding” the patch over the input image using one of the matching methods described in this section.9  If, as in Figure 7-8, we have an image patch containing a face, then we can slide that face over an input image looking for strong matches that would indicate another face is present.  void cv::matchTemplate   cv::InputArray  image,        // Input image to be searched, 8U or 32F, size W-by-H   cv::InputArray  templ,        // Template to use, same type as ‘image’, size w-by-h   cv::OutputArray result,       // Result image, type 32F, size (W-w+1)-by(H-h+1)   int

described in this section.9  If, as in Figure 7-8, we have an image patch containing a face, then we can slide that face over an input image looking for strong matches that would indicate another face is present.  void cv::matchTemplate   cv::InputArray  image,        // Input image to be searched, 8U or 32F, size W-by-H   cv::InputArray  templ,        // Template to use, same type as ‘image’, size w-by-h   cv::OutputArray result,       // Result image, type 32F, size (W-w+1)-by(H-h+1)   int             method        // Comparison method to use );  The input to cv::matchTemplate() starts with a single 8-bit or floating-point plane or color image. The matching model in templ is just a patch from another (presumably similar) image containing the object for which you are searching. The computed output will be put in the result image, which should                                                              9 In prior versions of the library, there was a function called: cvCalcBackProjectPatch(). This function

// Comparison method to use );  The input to cv::matchTemplate() starts with a single 8-bit or floating-point plane or color image. The matching model in templ is just a patch from another (presumably similar) image containing the object for which you are searching. The computed output will be put in the result image, which should                                                              9 In prior versions of the library, there was a function called: cvCalcBackProjectPatch(). This function was not ported to the 2.x library as it was considered to be very slow, and the similar results can be achieved through use of cv::matchTemplate().

be a single-channel byte or floating-point image of size (image.width – templ.width + 1, image.height – templ.height + 1). The matching method is chosen from one of the options listed below (we use I to denote the input image, T the template, and R the result image in the definitions). For each of these, there is also a normalized version10:   Figure 7-8: cv::matchTemplate() sweeps a template image patch across another image looking for matches Square difference matching method (method = cv::TM_SQDIFF) These methods match the squared difference, so a perfect match will be 0 and bad matches will be large: 𝑅!"_!"##=𝑇𝑥!,𝑦!−𝐼𝑥+𝑥!,𝑦+𝑦!!!!,!! Normalized square difference matching method (method = cv::TM_SQDIFF_NORMED) 𝑅!"_!"##_!"#$%&=𝑇𝑥!,𝑦!−𝐼𝑥+𝑥!,𝑦+𝑦!!!!,!!𝑇𝑥!,𝑦!!!!,!!∙𝐼𝑥+𝑥!,𝑦+𝑦!!!!,!! Correlation matching methods (method = cv::TM_CCORR) These methods multiplicatively match the template against the image, so a perfect match will be large and bad matches will be small or zero. 𝑅!!"##=𝑇𝑥!,𝑦!∙𝐼𝑥+𝑥!,𝑦+𝑦!!!,!!

These methods match the squared difference, so a perfect match will be 0 and bad matches will be large: 𝑅!"_!"##=𝑇𝑥!,𝑦!−𝐼𝑥+𝑥!,𝑦+𝑦!!!!,!! Normalized square difference matching method (method = cv::TM_SQDIFF_NORMED) 𝑅!"_!"##_!"#$%&=𝑇𝑥!,𝑦!−𝐼𝑥+𝑥!,𝑦+𝑦!!!!,!!𝑇𝑥!,𝑦!!!!,!!∙𝐼𝑥+𝑥!,𝑦+𝑦!!!!,!! Correlation matching methods (method = cv::TM_CCORR) These methods multiplicatively match the template against the image, so a perfect match will be large and bad matches will be small or zero. 𝑅!!"##=𝑇𝑥!,𝑦!∙𝐼𝑥+𝑥!,𝑦+𝑦!!!,!! Normalized cross-correlation matching method (method = cv::TM_SQDIFF_NORMED) 𝑅!!"##_!"#$%&=𝑇𝑥!,𝑦!∙𝐼𝑥+𝑥!,𝑦+𝑦!!!,!!𝑇𝑥!,𝑦!!!!,!!∙𝐼𝑥+𝑥!,𝑦+𝑦!!!!,!!                                                              10 The normalized versions were first developed by Galton [Galton] as described by Rodgers [Rodgers88]. The normalized methods are useful, as they can help reduce the effects of lighting differences between the template and the image. In each case, the normalization coefficient is the same.

Correlation coefficient matching methods (method = cv::TM_CCOEFF) These methods match a template relative to its mean against the image relative to its mean, so a perfect match will be 1.0 and a perfect mismatch will be -1.0; a value of 0.0 simply means that there is no correlation (random alignments). 𝑅!!"#$$=𝑇!!!,!!∙𝐼!!!!!,!!!!!!,!! 𝑇!𝑥!,𝑦!=𝑇𝑥!,𝑦!−𝑇(𝑥!!,𝑦!!)!!!,!!!(𝑤−ℎ) 𝐼!𝑥+𝑥!,𝑦+𝑦!=𝐼𝑥+𝑥!,𝑦+𝑦!−𝐼(𝑥!!,𝑦!!)!!!,!!!(𝑤−ℎ) Normalized correlation coefficient matching method (method = cv::TM_CCOEFF_NORMED) 𝑅!!"#$$_!"#$%&=𝑇′𝑥!,𝑦!∙𝐼′𝑥+𝑥!,𝑦+𝑦!!!,!!𝑇′𝑥!,𝑦!!!!,!!∙𝐼′𝑥+𝑥!,𝑦+𝑦!!!!,!! Here T’ and I’ are as defined for cv::TM_CCOEFF. As usual, we obtain more accurate matches (at the cost of more computations) as we move from simpler measures (square difference) to the more sophisticated ones (correlation coefficient). It’s best to do some test trials of all these settings and then choose the one that best trades off accuracy for speed in your application. Be careful when interpreting your results. The square-difference

Here T’ and I’ are as defined for cv::TM_CCOEFF. As usual, we obtain more accurate matches (at the cost of more computations) as we move from simpler measures (square difference) to the more sophisticated ones (correlation coefficient). It’s best to do some test trials of all these settings and then choose the one that best trades off accuracy for speed in your application. Be careful when interpreting your results. The square-difference methods show best matches with a minimum, whereas the correlation and correlation-coefficient methods show best matches at maximum points. Once we use cv::matchTemplate() to obtain a matching result image, we can then use cv::minMaxLoc() or cv::minMaxIdx() to find the location of the best match. Again, we want to ensure there’s an area of good match around that point in order to avoid random template alignments that just happen to work well. A good match should have good matches nearby, because slight misalignments of the template shouldn’t vary the results too much for real

correlation-coefficient methods show best matches at maximum points. Once we use cv::matchTemplate() to obtain a matching result image, we can then use cv::minMaxLoc() or cv::minMaxIdx() to find the location of the best match. Again, we want to ensure there’s an area of good match around that point in order to avoid random template alignments that just happen to work well. A good match should have good matches nearby, because slight misalignments of the template shouldn’t vary the results too much for real matches. Looking for the best matching “hill” can be done by slightly smoothing the result image before seeking the maximum (for correlation or correlation-coefficient) or minimum (for square-difference matching methods). The morphological operators (for example) can be helpful in this context.

Figure 7-9: Match results of six matching methods for the template search depicted in Figure 7-8: the best match for square difference is zero and for the other methods it’s the maximum point; thus, matches are indicated by dark areas in the left column and by bright spots in the other two columns Example 7-3 should give you a good idea of how the different template matching techniques behave. This program first reads in a template and image to be matched and then performs the matching via the methods we’ve discussed here. Example 7-3: Template matching #include <opencv2/opencv.hpp> #include <iostream>  using namespace std;  void help(){    cout << "\n"     "Example of using matchTemplate(). The call is:\n"     "\n"     "ch7_ex7_5 template image_to_be_searched\n"     "\n"     "   This routine will search using all methods:\n"     "         cv::TM_SQDIFF        0\n"     "         cv::TM_SQDIFF_NORMED 1\n"     "         cv::TM_CCORR         2\n"     "         cv::TM_CCORR_NORMED  3\n"     "

we’ve discussed here. Example 7-3: Template matching #include <opencv2/opencv.hpp> #include <iostream>  using namespace std;  void help(){    cout << "\n"     "Example of using matchTemplate(). The call is:\n"     "\n"     "ch7_ex7_5 template image_to_be_searched\n"     "\n"     "   This routine will search using all methods:\n"     "         cv::TM_SQDIFF        0\n"     "         cv::TM_SQDIFF_NORMED 1\n"     "         cv::TM_CCORR         2\n"     "         cv::TM_CCORR_NORMED  3\n"     "         cv::TM_CCOEFF        4\n"     "         cv::TM_CCOEFF_NORMED 5\n"     "\n"; }   // Display the results of the matches //  int main( int argc, char** argv ) {    if( argc != 3) {     help();

return -1;   }    cv::Mat src, templ, ftmp[6]; // ftmp is what to display on        // Read in the template to be used for matching:   //   if((templ=cv::imread(argv[1], 1)).empty()) {     cout << "Error on reading template " << argv[1] << endl;     help(); return -1;   }    // Read in the source image to be searched:   //   if((src=cv::imread(argv[2], 1)).empty()) {     cout << "Error on reading src image " << argv[2] << endl;     help(); return -1;   }    // Do the matching of the template with the image   for(int i=0; i<6; ++i){     cv::matchTemplate( src, templ, ftmp[i], i);      cv::normalize(ftmp[i],ftmp[i],1,0,cv::MINMAX);   }    // Display   //   cv::imshow( "Template", templ );   cv::imshow( "Image", src );   cv::imshow( "SQDIFF", ftmp[0] );   cv::imshow( "SQDIFF_NORMED", ftmp[1] );   cv::imshow( "CCORR", ftmp[2] );   cv::imshow( "CCORR_NORMED", ftmp[3] );   cv::imshow( "CCOEFF", ftmp[4] );   cv::imshow( "CCOEFF_NORMED", ftmp[5] );    // Let user view results:   //   cv::waitKey(0); } Note the

image   for(int i=0; i<6; ++i){     cv::matchTemplate( src, templ, ftmp[i], i);      cv::normalize(ftmp[i],ftmp[i],1,0,cv::MINMAX);   }    // Display   //   cv::imshow( "Template", templ );   cv::imshow( "Image", src );   cv::imshow( "SQDIFF", ftmp[0] );   cv::imshow( "SQDIFF_NORMED", ftmp[1] );   cv::imshow( "CCORR", ftmp[2] );   cv::imshow( "CCORR_NORMED", ftmp[3] );   cv::imshow( "CCOEFF", ftmp[4] );   cv::imshow( "CCOEFF_NORMED", ftmp[5] );    // Let user view results:   //   cv::waitKey(0); } Note the use of cv::normalize() in this code, which allows us to display the results in a consistent way. (Recall that some of the matching methods can return negative-valued results.) We use the cv::MINMAX flag when normalizing; this tells the function to shift and scale the floating-point images so that all returned values are between 0.0 and 1.0. Figure 7-9 shows the results of sweeping the face template over the source image (shown in Figure 7-9) using each of cv::matchTemplate()’s available matching methods.

use of cv::normalize() in this code, which allows us to display the results in a consistent way. (Recall that some of the matching methods can return negative-valued results.) We use the cv::MINMAX flag when normalizing; this tells the function to shift and scale the floating-point images so that all returned values are between 0.0 and 1.0. Figure 7-9 shows the results of sweeping the face template over the source image (shown in Figure 7-9) using each of cv::matchTemplate()’s available matching methods. In outdoor imagery especially, it’s almost always better to use one of the normalized methods. Among those, correlation coefficient gives the most clearly delineated match—but, as expected, at a greater computational cost. For a specific application, such as automatic parts inspection or tracking features in a video, you should try all the methods and find the speed and accuracy trade-off that best serves your needs. Summary In this chapter, we learned how OpenCV represents histograms at dense or sparse

In outdoor imagery especially, it’s almost always better to use one of the normalized methods. Among those, correlation coefficient gives the most clearly delineated match—but, as expected, at a greater computational cost. For a specific application, such as automatic parts inspection or tracking features in a video, you should try all the methods and find the speed and accuracy trade-off that best serves your needs. Summary In this chapter, we learned how OpenCV represents histograms at dense or sparse matrix objects. In practice, such histograms are typically used to represent probability density functions, which associate probability amplitude to every element of an array of some number of dimensions. We learned how to do basic operations on arrays, which are useful when interpreting arrays as probability distributions—such as normalization and comparison with other distributions.

Exercises 1. Generate 1,000 random numbers 𝑟i between 0.0 and 1.0. Decide on a bin size and then take a histogram of 1/𝑟i. a) Are there similar numbers of entries (i.e., within a factor of ±10) in each histogram bin? b) Propose a way of dealing with distributions that are highly nonlinear so that each bin has, within a factor of 10, the same amount of data. 2. Take three images of a hand in each of the three lighting conditions discussed in the text. Use cv::calcHist() to make an RGB histogram of the flesh color of one of the hands photographed indoors. a) Try using just a few large bins (e.g., 2 per dimension), a medium number of bins (16 per dimension) and many bins (256 per dimension). Then run a matching routine (using all histogram matching methods) against the other indoor lighting images of hands. Describe what you find. b) Now add 8 and then 32 bins per dimension and try matching across lighting conditions (train on indoor, test on outdoor). Describe the results. 3. As in exercise 2, gather RGB

flesh color of one of the hands photographed indoors. a) Try using just a few large bins (e.g., 2 per dimension), a medium number of bins (16 per dimension) and many bins (256 per dimension). Then run a matching routine (using all histogram matching methods) against the other indoor lighting images of hands. Describe what you find. b) Now add 8 and then 32 bins per dimension and try matching across lighting conditions (train on indoor, test on outdoor). Describe the results. 3. As in exercise 2, gather RGB histograms of hand flesh color. Take one of the indoor histogram samples as your model and measure EMD (earth mover’s distance) against the second indoor histogram and against the first outdoor shaded and first outdoor sunlit histograms. Use these measurements to set a distance threshold. a) Using this EMD threshold, see how well you detect the flesh histogram of the third indoor histogram, the second outdoor shaded, and the second outdoor sunlit histograms. Report your results. b) Take histograms of

RGB histograms of hand flesh color. Take one of the indoor histogram samples as your model and measure EMD (earth mover’s distance) against the second indoor histogram and against the first outdoor shaded and first outdoor sunlit histograms. Use these measurements to set a distance threshold. a) Using this EMD threshold, see how well you detect the flesh histogram of the third indoor histogram, the second outdoor shaded, and the second outdoor sunlit histograms. Report your results. b) Take histograms of randomly chosen nonflesh background patches to see how well your EMD discriminates. Can it reject the background while matching the true flesh histograms? 4. Using your collection of hand images, design a histogram that can determine under which of the three lighting conditions a given image was captured. Toward this end, you should create features—perhaps sampling from parts of the whole scene, sampling brightness values, and/or sampling relative brightness (e.g., from top to bottom patches in the frame) or

chosen nonflesh background patches to see how well your EMD discriminates. Can it reject the background while matching the true flesh histograms? 4. Using your collection of hand images, design a histogram that can determine under which of the three lighting conditions a given image was captured. Toward this end, you should create features—perhaps sampling from parts of the whole scene, sampling brightness values, and/or sampling relative brightness (e.g., from top to bottom patches in the frame) or gradients from center to edges. 5. Assemble three histograms of flesh models from each of our three lighting conditions. a) Use the first histograms from indoor, outdoor shaded, and outdoor sunlit as your models. Test each one of these against the second images in each respective class to see how well the flesh-matching score works. Report matches. b) Use the “scene detector” you devised in part a, to create a “switching histogram” model. First use the scene detector to determine which histogram model to use:

from center to edges. 5. Assemble three histograms of flesh models from each of our three lighting conditions. a) Use the first histograms from indoor, outdoor shaded, and outdoor sunlit as your models. Test each one of these against the second images in each respective class to see how well the flesh-matching score works. Report matches. b) Use the “scene detector” you devised in part a, to create a “switching histogram” model. First use the scene detector to determine which histogram model to use: indoor, outdoor shaded, or outdoor sunlit. Then use the corresponding flesh model to accept or reject the second flesh patch under all three conditions. How well does this switching model work? 6. Create a flesh-region interest (or “attention”) detector. a) Just indoors for now, use several samples of hand and face flesh to create an RGB histogram. b) Use cv::calcBackProject() to find areas of flesh. c) Use cv::erode() from Chapter 5 to clean up noise and then cv::floodFill() (from the same chapter) to find large

outdoor shaded, or outdoor sunlit. Then use the corresponding flesh model to accept or reject the second flesh patch under all three conditions. How well does this switching model work? 6. Create a flesh-region interest (or “attention”) detector. a) Just indoors for now, use several samples of hand and face flesh to create an RGB histogram. b) Use cv::calcBackProject() to find areas of flesh. c) Use cv::erode() from Chapter 5 to clean up noise and then cv::floodFill() (from the same chapter) to find large areas of flesh in an image. These are your “attention” regions. 7. Try some hand-gesture recognition. Photograph a hand about two feet from the camera; create some (nonmoving) hand gestures: thumb up, thumb left, and thumb right. a) Using your attention detector from exercise 6, take image gradients in the area of detected flesh around the hand and create a histogram model for each of the three gestures. Also create a histogram of the face (if there’s a face in the image) so that you’ll have a (nongesture)

areas of flesh in an image. These are your “attention” regions. 7. Try some hand-gesture recognition. Photograph a hand about two feet from the camera; create some (nonmoving) hand gestures: thumb up, thumb left, and thumb right. a) Using your attention detector from exercise 6, take image gradients in the area of detected flesh around the hand and create a histogram model for each of the three gestures. Also create a histogram of the face (if there’s a face in the image) so that you’ll have a (nongesture) model of that large flesh region. You might also take histograms of some similar but nongesture hand positions, just so they won’t be confused with the actual gestures.

b) Test for recognition using a webcam: use the flesh interest regions to find “potential hands”; take gradients in each flesh region; use histogram matching above a threshold to detect the gesture. If two models are above threshold, take the better match as the winner. c) Move your hand one to two feet further back and see if the gradient histogram can still recognize the gestures. Report. 8. Repeat exercise 7 but with EMD for the matching. What happens to EMD as you move your hand back? 9. With the same images as before but with captured image patches instead of histograms of the flesh around the hand, use cv::matchTemplate() instead of histogram matching. What happens to template matching when you move your hand backwards so that its size is smaller in the image? 10. With your hands facing a camera, take the gradient direction of several pictures of your open hand, a closed fist and a “thumbs up” gesture. Collect histograms of the gradient direction in a window around your hands. This becomes your trained

as before but with captured image patches instead of histograms of the flesh around the hand, use cv::matchTemplate() instead of histogram matching. What happens to template matching when you move your hand backwards so that its size is smaller in the image? 10. With your hands facing a camera, take the gradient direction of several pictures of your open hand, a closed fist and a “thumbs up” gesture. Collect histograms of the gradient direction in a window around your hands. This becomes your trained “model”.  Now run live and use the various histogram matching techniques to see how well they can recognize your gestures.

8 
Contours 
Although algorithms like the Canny edge detector can be used to find the edge pixe ls that separate different 
segments in an image, they do not tell you anything about those edges as entities in themselves. The next 
step is to be able to assemble those edge pixels into contours. By now you have probably come to expect 
that there is a convenient function in OpenCV that will do exactly this for you, and indeed there is: 
cv::findContours() . We will start out this chapter with some basics that we will need in order to use 
this function.. With those concepts in hand, we will ge t into contour finding in some detail. Thereafter, we 
will move on to the many things we can do with contours after they’ve been computed. 
Contour Finding 
A contour is a list of points that represent, in one way or another, a curve in an image. This representation 
can be different depending on the circumstance at hand. There are many ways to represent a curve.

can be different depending on the circumstance at hand. There are many ways to represent a curve. 
Contours are represented in OpenCV by STL style vector<>  template objects in which every entry in the 
vector encodes information about the location of the next point on the curve. It should be noted that though a sequence of 2d points (vector<cv::Point>  or vector<cv::Point2f> ) is the most common 
representation, there are other ways to represent contou rs as well. One example of such a construct is the 
Freeman Chain, in which each point is represented as a particular “step” in a given direction from the prior

Freeman Chain, in which each point is represented as a particular “step” in a given direction from the prior 
point. We will get into such variations in more detail as we encounter them. For now, the important thing to know is that contours are almost always STL vectors,  but are not necessarily lim ited to the obvious vectors 
of cv::Point  objects. 
The function cv::findContours()  computes contours from binary im ages. It can take images created 
by cv::Canny() , which have edge pixels in them, or images created by functions like 
cv::threshold()  or cv::adaptiveThreshold() , in which the edges are implicit as boundaries 
between positive and negative regions.
1 
Contour Hierarchies

1 
Contour Hierarchies 
Before getting down to exactly how to extract contours, it is worth taking a moment to understand exactly what a contour is, and how groups of contours can be related to one another. Of particular interest is the 
concept of a contour tree, which is important for understanding one of the most useful ways 
                                                          
1 There are some subtle differences between passing edge images and binary images to cvFindContours();  we 
will discuss those shortly.

cv::findContours()  (retrieval methods derive from Suzuki [Suzuki85]) can communicate its results 
to us. 
Take a moment to look at 
 
Figure 8-1 , which depicts the functionality of cv::findContours() . The upper part of the figure 
shows a test image containing a number of “colored” (here, gray) region s (labeled A through E) on a light 
background. The lower portion of the figure depicts the same image along with the contours that will be 
located by cv::findContours() . Those contours are labeled cX or hX, where “c” stands for 
“contour,” “h” stands for “hole,” an d “X” is some number. OpenCV and cv::findContours()  
distinguishes between the exterior boundaries of non-zero regions which are labeled contours and the 
interior 
 boundaries  which are labeled holes .

F s 
co t 
(th ed 
(th ur 
 
The concept of containment here is important in many applications. For this reason, OpenCV can be asked 
that encodes the containment relationships in its 
The
of v
node an 
entr
part
the d ause 
it ha rent). 
mponent in the four-ele ment vector representation of each node in a contour igure 8-1: A test image (left side) passed to cv::findContours() . There are five colored region
(labeled A, B, C, D, and E), but c ontours are formed from both the exte rior and interior edges of each 
lored region. The result is nine contours in total. Ea ch contour is identified and appears in an output lis
e contours  argument—upper-right). Optionally, a hierarchic al representation can also be generat
e hierarchy  argument—lower-right). In the graph shown (corresponding to the constructed conto
tree), each node is a contour, the links  in the graph are labeled with the index in the four-element data
structure associated with each node in the hierarchy  array.

structure associated with each node in the hierarchy  array. 
to assemble the found contours into a contour tree2 
structure. A contour tree corresponding to this test image would have the contour called c0 at the root node, 
with the holes h00 and h01 as its children. Those would in turn have as children the contours that they 
directly contain, and so on. 
re are many possible ways to repres ent such a tree. OpenCV represents such trees with arrays (typically 
ntry  or ectors) in which each entry in the array represents one particular contour. In that array, each e

 contains a set of four integers (typi cally represented as an element of type cv::Vec4i , just like 
y in a four-channel array). Each of the four components  indicates another node in the hierarchy with a 
hat element of icular relationship to the current node. Where a particular relationship does not exist, t
ata structure is set to -1 (e.g., element 3, the parent id for the root node would have value -1 bec
s no pa
Table 8-1: Meaning of each co
hierarchy list 
Index Meaning

contains a set of four integers (typi cally represented as an element of type cv::Vec4i , just like 
s no pa
Index Meaning 
                                                           
2 Contour trees first appeared in Reeb [Reeb46] and were  further developed by [Bajaj97], [Kreveld97], [Pascucci02], 
and [Carr04].

0 Next Contour (same level) 
1 Previous Contour (same level) 
2 First Child (next level down) 
3 Parent (next level up) 
By way of example, consider the contours in 
 
Figure 8-1 . The five colored regions result in a total of nine total contours (counting both the exterior and 
ch region). If  a contour tree is constructed from these nine contours, each node will the interior edges of ea

have as children  those contours that are contained with in it. The resulting tree is visualized in the lower-
right of 
  
re 8-1 , For each node, those links that are valid are also visualized, and the links are labeled with the 
x associated with that link in the fo ur-element data structure for that node ( Error! Reference source 
found. ). Figu
inde
not 
It is interesting to note the c sequences of using cv::findContou on rs()  on an image 
generated by cv::canny()  or a similar edge detector relative to what happens with a 
binary image such as the test imag e shown in Figure 8-1. Deep down, 
cv::findContours()  does not really know anything about edge images. This means 
that, to cv::findContours(), an “edge” is just a very thin “on” area. As a result, for 
every exterior contour, there will be a hole contour that almost exactly coincides with it. 
This hole is actually just inside of the exte rior boundary. You can think of it as the on-to-
off transition that marks the interior edge of the edge.

off transition that marks the interior edge of the edge. 
Finding Contours with    cv::findContours()
trees in hand, we can look at the cv::findContours()  function itself and With this concept of contour 
see exactly how we tell it what we want  and how we interpret its response: 
void cv::findContours( 
  cv::InputOutputArray    image,               // Input “binary”  8-bit single channel 
  cv::OutputArrayOfArrays co ntours,            // Vector of vectors or points 
  cv::OutputArray         hierarchy,           // (optional) topology informati on

  cv::OutputArray         hierarchy,           // (optional) topology informati on 
  int                     mode,                // Contour retrieval mode (Figure 8 -2) 
  int                     method,              // Approximation method 
  cv::Point               offset = cv::Point() // (optional) Offset ever y point 
); 
 
  cv::InputOutputArray    image,               // Input “binary” 8-bit single channel 
  cv::OutputArrayOfArrays contours,            // Vector of vectors or points

int                     mode,                // Contour retrieval mode (Figure 8 -2) 
  cv::Point               offset = cv::Point() // (optional) Offset every point 
); 
The  be 
inte ted as binary (i.e., as if all nonzero pixels were equivalent to one another). When it runs, 
g one of the two forms of  fir
rprest argument is the input image; this image should be an 8-bit single-channel image and will
cv::findContours()  will actually use this image as scratch sp ace for computation, so if you need that 
image for anything later, you should make a copy and pass that to cv::findContours() . The second 
argument is an array of arrays, which in most practical  cases will mean an STL vector of STL vectors. This 
will be filled with the list of contours found (i.e., it will be a vector of contours, where contours[i]  will 
be a specific contour and thus contours[i][j]  would refer to a specific vertex in contour[i] ).

be a specific contour and thus contours[i][j]  would refer to a specific vertex in contour[i] ). 
The next argument  can be either supplied or not supplied (by usin hierarchy
the function shown above). If supplied, hierarchy  is the output that describe s the tree structure of the 
contours. The output hierarchy  will be an array (again typically an  STL vector) with one entry for each 
contour in contours . Each such entry will contain an array of  four elements, each indicating the node to 
which a particular link from th e current node is connected ( Error! Reference source not found. ).

Figure 8-2: The way in which  the tree node variables ar e used to “hook up” all of the contours located by 
findContours() . The contour nodes are the same as in cv::
 
Figure 8-1 . 
The  argument tells OpenCV how you would like th e contours extracted. There are four possible 
TERNAL , cv::RETR_LIST , cv::RETR_CCOMP , and 
. Each mode is described below: mode
values for mode: cv::RETR_EX
cv::RETR_TREE

cv:
e extreme outer contours. In :RETR_EXTERNAL  
Retrieves only th
 
Figure 8-1 , there is only one exterior contour, so 
 
Figure 8-2  indicates that the first contour points to th at outermost sequence and that there are no 
further connections.

cv::RETR_LIST  
Retrieves all the contours and puts them in the list. 
 
Figure 8-2  depicts the “hierarchy” resulting from the test image in

Figure 8-1 . In this case, nine contours are found an d they are all connected to one another by 
hierarchy[i][0]  and hierarchy[i][1] (hierarchy[i][2]  and hierarchy[i][3]  
are not used here).3 
cv::RETR_CCOMP  
Retrieves all the contours and organizes them into a tw o-level hierarchy, where the top-level boundaries are 
external boundaries of the components and the second-level boundaries are boundaries of the holes. 
Referring to 
  
Figure 8-2 , we can see that there are five exterior bound aries, of which three c ontain holes. The holes 
are connected to their corresp onding exterior boundaries by hierarchy[i][2]  and 
hierarchy[i][3] . The outermost boundary c0 contains two holes. Because hierarchy[i][2]  
can contain only one value, the node can only have one child. All of the holes inside of c0 are 
connected to one another by the hierarchy[i][0]  and hierarchy[i][1]  pointers. 
                                                           
3 Yo  the OpenCV

3 Yo  the OpenCV 
librar  which the contours return value was not automatically organized into a list as the vector<>  type now 
implies. u are
y in not very likely to use this opti on. This organization primarily made sens e in previous versions of

cv::RETR_TREE  
Retrieves all the contours and reconstructs the full hierarchy of nested contours. In our example 
(
  
Figure 8-1  and 
  
Figure 8-2 ), this means that the root node is the outermost contour c0. Below c0 is the hole h00, which 
is connected to the other hole h01 at the same leve l. Each of those holes in turn has children (the 
contours c000 and c010, respectively), which are connected to their parents by vertical links. Thi
h become the leaf nodes in the tree. s 
ontours in the image, whic
The next five values pertain to the method  (i.e., how the contours are represented):  continues down to the most-interior c

cv:
ce a large number 
point. No attempt is made to 
ny 
ction of the number of points returned. The extreme 
that is or iented along the x-y axes. In this case, only four 
AIN_APPROX_TC89_L1  or cv::CHAIN_APPROX_TC89_KCOS  
 returned. 
T-C algorithm requires no additional parameters to run. 
t. The offset  argument is optional. If 
void  cv::drawContours( :CHAIN_APPROX_NONE  
Translates all the points from the contour code in to points. This operation
oints, as each point w ill be one of the eight neighbors of the previous  will produ
of p
reduce the number of vertices returned. 
cv::CHAIN_APPROX_SIMPLE  
Compresses horizontal, vertical, and diagonal segments, leaving only their ending points. For ma
special cases, this can result in a substantial redu
example would be a rectangle (of any size) 
points would be returned. 
cv::CH
Applies one of the flavors of the Te h-Chin chain approximation algorithm.4 The Teh-Chin algorithm is

of p
cv::CH
Applies one of the flavors of the Te h-Chin chain approximation algorithm.4 The Teh-Chin algorithm is 
a more sophisticated (and more compute-intensive) method for reducing the number of points
The 
The final argument to cv::findContours()  is offse
present, every point in the returned contour will be shifted by this amount. This is particularly useful when 
either the contours are extracted from a region of interest, but you would like them represented in the parent image’s coordinate system, or the re verse case, where you are extracting the contours in the coordinates of

The 
either the contours are extracted from a region of interest, but you would like them represented in the parent image’s coordinate system, or the re verse case, where you are extracting the contours in the coordinates of 
a larger image, but would like to express them relative to some subregion of the image.  
Drawing Contours 
One of the most straightforward things you might want to  do with a list of contours , once you have it, is to 
draw the contours on the screen. For this we have cv::drawContours() : 
  cv::InputOutputArray   image,                 // Will draw on input image 
  cv::InputArrayOfArrays contours,              // Vector of vectors or points 
  int                    contourIdx,            // Contour to draw (-1 is “all”)

  int                    contourIdx,            // Contour to draw (-1 is “all”) 
  const cv::Scalar&      color,                 // Color for contours 
  int                    thickness = 1,         // Thickness for contour lines 
  int                    lineType  = 8,         // Connectedness (‘4’ or ‘8’) 
  cv::InputArray         hierarchy = noArray(), // optional (from findContours) 
  int                    maxLevel  = INT_MAX,   // Max level in hierarchy to descend 
  cv::Point              offset = cv::Point()   // (optional) Offset every point 
)

const cv::Scalar&      color,                 // Color for contours 
) 
The first argument image is simple: it is the image on which to draw the contours. The next argument, 
contour , is the list of contours to be drawn. This is in the same form as the contour  output of 
cv::findContours() ; it is a list of lists of points. The contourIdx  argument can be used to select 
either a single contour to draw or to tell cv::drawContours() to draw all of the contours on the list 
provid n. If 
contourIdx  
e corresponding arguments in other

) 
e corresponding arguments in other 
                                                          ed in the contours  argument. If contourIdx  is a positive number, all contours will be draw
is negative (usually this is just set to -1), all contours are drawn. 
The color, thickness , and lineType  arguments are similar to th
the color argument draw functions such as cv::Line() . As usual, is a four-component cv::Scalar , 
the thickness is an integer indicating the thickness of the lines to be drawn in pixels, and the lineType

ed in the contours  argument. If contourIdx  is a positive number, all contours will be draw
the thickness is an integer indicating the thickness of the lines to be drawn in pixels, and the lineType  
may be either 4 or 8 indicating whether the line is to be drawn as a 4-connected (ugly), 8-connected (not 
too ugly), or cv::AA  (pretty) line. 
 
eper details of the algorithm. 4 If you are interested in the details of how this algor ithm works, you can consult Teh, C.H. and Chin, R.T., On the 
Detection of Dominant Points on Digital Curve . PAMI 11 8, pp. 859–872 (1989). Because the algorithm requires no 
tuning parameters, however, you can get quite far w ithout knowing the de

The hierarchy  argument corresponds to th e hierarchy output from cv::findContours() . The 
hierarchy  works with the maxLevel  argument. The latter limits the depth in the hierarchy to which 
contours will be drawn in your image. Setting maxLev  indicates that the highest 
level) in the hierarchy should be drawn; higher nu mbers indicate that numbe of layers down from the 
highest level which should be included. Looking at el to zero  only “level 0” (
r 
 
Figure 8-2 , you can see that this is useful for contour trees; it is also potentially useful for connected 
component MP) in case you would like only to visualize exterior contours (but not 
“holes”—in
Finally, we ca  to the draw routine so that the cont our will be drawn elsewhere than at the 
absolute co t was defined. This feature is par cularly useful when he contour has 
already been converted to center-o f-mass or other local coordinates. offset  is particularly helpful in the 
case in whi indContours()  one or more times in different im regions

case in whi indContours()  one or more times in different im regions 
(ROIs) but e results within the original lar e image. Conversely, we could use 
offset  if we’d extracted a contour from a large image and then wanted to form a small mask for this 
contour. 
A Contour Example 
 an image in it. A trackbar 
sets nd the contours in the thresholded image are drawn. The image is updated 
whe
ever the s (cv::RETR_CCO
terior contours). 
n give an offset
ordinates by which i ti  t
ch you have used cv::f
now want to display all thage sub-
g
Example 8-1  is drawn from the OpenCV package. Here we cr eate a window with
a simple threshold, a

contour. 
whe
g
never the trackbar is adjusted. 
Example 8-1: Finding contours based on a trackbar’s location; the contours are updated when
trackbar is moved 
#include <opencv2/opencv.hpp> 
#include <iostream> 
 
using namespace std; 
 
cv::Mat g_gray, g_binary; 
int g_thresh = 100; 
 
void on_trackbar(int, void*) { 
  cv::threshold( g_gray, g_binary, g_thresh, 255, cv::THRESH_BINARY ); 
  vector< vector< cv::Point> > contours; 
  cv::findContours( 
    g_binary,

contours,  
    cv::noArray(), 
    cv::RETR_LIST,  
    cv::CHAIN_APPROX_SIMPLE 
  ); 
  g_binary = cv alar::all(0); ::Sc
     
  cv::drawContours( g_binary, contours, -1, cv::Scalar::all(255)); 
  cv::imshow( "Contours", g_binary ); 
} 
 
int main( int argc, char** argv ) 
{ 
  if( argc != 2 || ( g_gray = cv::imread(argv[1], 0)).empty() ) { 
 cout << "Find threshold dependent contours\nUsage: ./ch8_ex8_1 fruits.jpg"  
      << endl; 
    return -1; 
  } 
  cv::namedWindow( "Contours", 1 ); 
  cv::createTrackbar(  
    "Threshold",  
    "Contours",  
    &g_thresh,  
    255,  
    on_trackbar 
  ); 
  on_trackbar(0, 0); 
  cv::waitKey(); 
  return 0; 
} 
Here, everything of interest to us is happening inside of the function on_trackbar() . The image 
g_gray  is thresholded such that only those pixels brighter than g_thresh  remain nzero. The 
cv::findContours()  function is then called on this thresholded image.  is 
then ca
In th  
good
(cv
you d 
by   no
cvDrawContours()

cv::createTrackbar(  
    255,  
  ); 
  return 0; 
} 
then ca
In th  
good
(cv
you d 
by   no
lled and the contours are drawn (in white) onto the grayscale image.  
Another Contour Example 
is example, we find contours on an input image and then proceed to draw them one by one. This is a
 example to tinker with on your o n to explore the effects of changing either the contour finding mod w e 
If ::RETR_LIST  in the code) or the max_depth  that is used to draw the contours ( 0 in the code). 
 set max_depth  to a larger number, notice that the exampl e code steps through the contours returne

:findContours()  by means of hierarchy[i][1] . Thus, for some topologies cv:
(cv::RETR_TREE, cv::RETR_CCOMP , etc.), you may see the same contour more than once as you 
step through. See Example 8-2 . 
Example 8-2: Finding and drawing contours on an input image 
#include <algorithm> 
 
using name space std; 
 
struct AreaCmp 
{ 
    AreaCmp(const vector<float>& _areas) : areas(&_areas) {}

{ 
    AreaCmp(const vector<float>& _areas) : areas(&_areas) {} 
    bool operator()(int a, int b) const { return (*areas)[a] > (*areas)[b]; } 
    const vector<float>* areas; 
};

int main(int argc, cha r* argv[]) { 
 
  cv::Mat img, img_edge, img_color; 
   
  // load image or show help if no image was provided 
  if( argc != 2 || (img = cv::imread( argv[1], cv::LOAD_IMAGE_GRAYSCALE )).empty() ){ 
      cout << "\nExample 8_2 Drawing Contours\nCall is:\n./ch8_ex8_2 image\n\n"; 
      return -1; 
  } 
   
  cv::threshold(img, img_edge, 128, 255, cv::THRESH_BINARY); 
  cv::imshow("Image after threshold", img_edge); 
  vector< vector< cv::Point > > contours; 
  vector< cv::Vec4i > hierar ;   chy
     
    img_edge,  
    contours,  
    hierarchy,  
  ); 
  cout << "\n\nHit any key to draw the next contour, ESC to quit\n\n"; 
  cout << "Total Contours Detected: " << contours.size() << endl; 
  vector<int> sortIdx(contours.size());   
  vector<float> areas(contours.size()); 
  for( int n = 0; n < (int) ntours.size(); n++ ) { co
      sortIdx[n] = n; 
      areas[n] = contourArea(contours[n], false); 
  }

cv::findContours( 
    img_edge,  
    contours,  
  ); 
  } 
  // sort contours so that the largest contours go first 
  std::sort( sortIdx.begin(), sortIdx.end(), AreaCmp(areas )); 
     
  for( int n = 0; n < (int)sortIdx.size(); n++ ) { 
     int idx = sortIdx[n];  
     cv::cvtColor( img, img_ olor, cv::GRAY2BGR ); c
     cv::drawContours(img_color, contours, idx , 
                   cv::Scalar(0,0,255), 2, 8, hierarchy, 
                   0 // Try different values of max_level, and see what happens 
                   );

} 
     
                   );  
     cout << "Contour #" << idx << ": area=" << areas[idx] << 
        ", nvertices=" << contours[idx].size() << endl; 
     cv::imshow(argv[0], img_color); 
     int k; 
     if((k = cv::waitKey()&255) == 27) 
       break; 
  } 
  cout << "Finished  all contours\n"; 
  return 0; 
} 
More to Do with Contours 
When analyzing an image, there are many different things we might want to do with contours. After all, 
most contours are—or are candidates to be—things that we are interested in identifying or manipulating.

", nvertices=" << contours[idx].size() << endl; 
     int k; 
       break; 
  } 
  return 0; 
} 
most contours are—or are candidates to be—things that we are interested in identifying or manipulating. 
The various relevant tasks include characterizing the contours in various ways, simplifying or 
approximating them, matching them to templates, and so on. 
In this section, we will examine some of these co mmon tasks and visit the various functions built into 
OpenCV that will either do these things for us or at  least make it easier for us to perform our own tasks.

Polygon Approximations 
cv::approxPolyDP()
The routine cv::approxPolyDP()  is an implementation of one of these two algorithms5: 
void cv::approxPolyDP( If we are drawing a contour or are engaged in shap e analysis, it is common to approximate a contour 
representing a polygon with another contour having fewer vertices. There are many different ways to do 
this; OpenCV offers implementations of two of them.  
Polygon Approximation with   
  cv::InputArray  curve,    // Array or vector of 2-dimensional points 
  cv::OutputArray approxCurve, // Result, type is same as ‘curve’ 
  double          epsilo      // Max distance from original ‘curve’ to ‘approxCurve’ n,
  bool            closed      // If true, assume link from last to first vertex   
); 
The cv::approxPolyDP()  function acts on one poly at a time, which is given in the input curve. 
The output of  will be placed in the  output array. As usual, 
resented

cv::InputArray  curve,    // Array or vector of 2-dimensional points 
); 
resented
arrays of size N-by-1 (but having two channels). Whicheve representation you choose, the input and 
output arrays used for curve and approxCurve  should be of the same type. 
The parameter epsilon  is the accuracy of approximation you  require. The meaning of the epsilon 
inal lygon
closed ot th
indicated by rve uld be considered a closed polygon. If set to true, the curve will be assumed to be 
The Douglas-Peucker Algorithm Explained

resented
inal lygon
The Douglas-Peucker Algorithm Explained 
In order to help understand how to set the epsilon  parameter, as well as to better understand the output 
of cv::approxPolyDP() , it is worth taking a moment to understand exactly how the algorithm works. gon 
cv:: approxPolyDP() approxCurve
these polygons can be rep  as either STL vectors of cv::Point  objects or as OpenCV cv::Mat  
r

inal lygon
r 
parameter is that this is the largest deviation you will allow between the orig po  and the final approximated polygon. , the last argument, indicates whether or n e sequence of points 
 cu sho
closed (i.e., that the last point is to be considered connected to the first point). 
In 
  
5 For aficionados, the method we are di scussing here is the Douglas-Peucker  (DP) approximation [Douglas73]. Other

r 
 cu sho
In 
  
5 For aficionados, the method we are di scussing here is the Douglas-Peucker  (DP) approximation [Douglas73]. Other 
popular methods are the Rosenfeld-Johnson [Rosenfeld73] a nd Teh-Chin [Teh89] algorithms. Of those two, the Teh-
Chin algorithm is not available in OpenCV as a reduction method, but is available at the time of the extraction of the 
polygon (see “Finding Contours with cv::findContours() ”).

Figure 8-3 , starting with a contour (panel b), the algorithm begins by picking two extremal points and 
connecting them with a line (panel c). Then the original polygon is searched to find the point farthest from 
the line just drawn, and that point is added to the approximation. 
The process is iterated (panel d), ad ding the next most distant point to  the accumulated approximation, until 
all of the points are less than the distance indicated by the precision parameter (panel f). This means that 
good arameter are some fraction of the contour’s length, or of the length of its bounding 
box, of the contour’s overall size.  candidates for the p
r measure  or a simila
 
Figure 8-3: Visualization of the DP algorithm used by cv:: approxPolyDP() : the original image (a) is 
approximated by a contour (b) and then, starting from th e first two maximally separated vertices (c), the 
vertices are iteratively se lected from that contour (d–f) 
s we will discuss work equally well for any collection of

s we will discuss work equally well for any collection of 
urve between those point s). We will mention along 
the way which methods make sense only for curves (such as computing arc length) and which make sense 
ing additional 
Geometry and Summary Characteristics 
Another task that one often faces with  contours is computing their various summary characteristics. These 
might include length or some other form of size measure of the overall contour. Other useful characteristics

might include length or some other form of size measure of the overall contour. Other useful characteristics 
are the contour moments , which can be used to summarize the gross shape characteristics of a contour 
discussed in the next section. Some of the methodpoints (i.e., even those that do not imply a piecewise c
for any general set of points (such as bound boxes). 
Length using cv::arcLength() 
The subroutine cv::arcLength()  will take a contour and return its length.  
double  cv::arcLength( 
  cv::InputArray  points,    // Array or vector of 2-dimensional points

  cv::InputArray  points,    // Array or vector of 2-dimensional points 
  bool            closed       // If true, assume link from last to first vertex 
); 
The first argument of cv::arcLength()  is the contour itself, whose form may be any of the usual 
representations of a curve (i.e., STL vector of poin ts or array of two-chan nel elements). The second 
points closed argument indicates whether the contour should be treated as closed. If the contour is considered 
closed, the distance from the last point in to the first contributes to the overall arc length.

bool            closed       // If true, assume link from last to first vertex 
); 
closed, the distance from the last point in to the first contributes to the overall arc length.  
cv::arcLength()  is an example of a case where the points argument is implicitly assumed to represent 
a curve, and so is not particularly meaningful for a general set of points.

Upright Bounding Box with cv::boundingRect()  
Of course, the length and area are simple characteriza tions of a contour. One of the simplest ways to 
characterize a contour is to report a bounding box for that contour. The simplest version of that would be to 
simply compute the upright bound ing rectangle. This is what cv::boundingRect()  does for us: 
cv::Rect cv::boundingRect(     // Return upright rectangle bounding the points 
); 
The cv::boundingRect()  function just takes one argument, which is the curve whose bounding box 
you would like computed. The function returns a v f type cv::Rect , which is the bounding box you 
are looking for. 
The bounding box computation is meaningful for any set of points, whether or not those p s r sent a 
int
A Minimum Area Rectangle with cv::minAreaRect()  
e b g ralue o
oint epre
curve or are just some arbitrary constellation of po s.

); 
int
e b g ralue o
oint epre
curve or are just some arbitrary constellation of po s. 
One problem with th oundin ectangle from cv::boundingRect() is that it returns a cv::Rect  
and so can only represent a rectangle whose sides are oriented horizontally and ver tically. In contrast, the 
routine cv::minAreaRect()  returns the minimal rectangle that  will bound your contour, and this 
rectangle may be inclined rela tive to the vertical; see 
 
Figure 8-4 . The arguments are otherwise similar to cv::boundingRect() . The OpenCV  data type 
cv::RotatedRect  is just what is needed to represent such a rectangle. Recall that it has the following

cv::RotatedRect  is just what is needed to represent such a rectangle. Recall that it has the following 
definition: 
class cv:: tatedRect { Ro
  cv::Point2f center;      // Exact center point (around which to rotate) 
  cv::Size2f  size;        // Size of rectangle (centered on ‘center’) 
  float       angle;       // degrees 
}; 
So, in order to get a little tighter fit, you can call cv::minAreaRect() : 
cv::RotatedRect cv::minAreaRect( // Return rectangle bounding the points 
  cv::InputArray  points,      // Array or vector of 2-dimensional points 
);

}; 
); 
As usual, points  can be any of the standard representations for a sequence of points, and is equally 
meaningful for curves as we ll as arbitrary point sets.

Figure 8-4: cv::Rect  can represent only upright rectangles, but cv::Box2D  can handle rectangles of 
any inclination 
nclosing Circle Using cv::minEnclosingCircle()  
Next, we have cv::minEnclosingCircle() .6 This routine works pretty much he same way as the 
bounding box routines, with the exception that there is no convenient data type for the return value. As a 
result, you must pass in references to variables you would like set by cv::minEnclosingCircle() : 
v::minEnclosingCircle( A Minimal E
 t
void  c
   cv::  2 InputArray points,      // Array or vector of -dimensional points 
   cv::Point2f&   center,        // Result location of circle center 
   float&         radius         // Result radius of circle 
); 
The input curve is just the usual sequence of points representation. The center and radius variables are 
variables you will have to allocate and which will be set for you by cv::minEnclosingCircle() .

 t
void  c
); 
variables you will have to allocate and which will be set for you by cv::minEnclosingCircle() . 
The cv::minEnclosingCircle  function is equally meaningful for curves as for general point sets. 
Fitting an Ellipse with cv::fitEllipse() 
As with the minimal enclosing circle, OpenCV also provides a method for fitting an ellipse to a set of 
point
cv::RotatedRect cv::fitEllipse(  // ngle bounding the ellipse (Figure 8- 5)  s: 
 Return recta
 oints  cv::InputArray  points      // Array or vector of 2-dimensional p
); 
cv::fitE e()  tak st a po nt.  
At first g it m ppear cv:: lips s just elliptical analog of

point
); 
At first g it m ppear cv:: lips s just elliptical analog of 
cv::minEnclosingCircle() . e is, however sub re n 
cv::minEnclosingCircle()  an:fitE se() ,  is that rmer si omputes 
the smalles  that c ely enc e giv ts, wh he latt  a fittin tion and 
re s  th est ati poi m t no s in the 
                  llips
lance, 
t circlees ju
ight a
ompletints array as argu
that 
Ther
d cv:
loses thme
fitEl
llip
en poine() i
, a 
 which
ereas t the 
tle diffe
 the fo
er usesnce betwee
mply c
g func
turn the ellipse at is the b  approxim on to the nt set. This eans tha t all point

                  llips
lance, 
ight a
that 
Ther
d cv:
loses thme
fitEl
llip
, a 
 which
ereas t the 
tle diffe
 the fo
mply c
g func
                                         
6 For more information on the inner workings of these fitting techniques, see Fitzgi bbon and Fisher [Fitzgibbon95] and 
Zhang [Zhang96].

contour will even be enclosed in the ellipse returned by cv::fitEllipse() .7 The fitting is done using 
a least-squares fitness function. 
The results of the fit are returned in a cv::RotatedRect  structure. The indicated box exactly encloses 
the ellipse (
  
Figure 8- 5). 
 
Figure 8- 5: Ten-point contour with the minimal enclosing ci rcle superimposed (a) and wi
b). A “rotated rect angle” is used by OpenCV to represent that ellith the best fitting 
ellipsoid ( psoid (c). 
Finding the B r Contour with 
In many cases,  set of points which you believe is approximately a straight 
line—or, more  be a no isy sample whose underlying origin is a straight 
line. In such a situation, the problem is to determine what line would be the best explanation for the points 
you observe. In fact, there are many reasons why one might want to find best line fit, and as a result, there 
are many variations of how to actually do that fitting. 
This fitting is d , which is defined to b  est Line Fit to You cv::fitLine()

This fitting is d , which is defined to b  est Line Fit to You cv::fitLine()  
 your “contour” will actually be a
 accurately, which you believe to
 a 
e: one by minimizing a cost function
, where 
  
Here 
  is the set of parameters that define the line, 
  is the ith point in the contour, and 
  is the distance 
betw n that p ee oint and the line defined by 
 . Thus e function , it is th  
  that fundamen distinguishes 
                                                        tally 
   
7th ciently small (or certai n other degenerate cases—including all of the points

your “contour” will actually be a
 a 
, where 
  
Here 
                                                        tally 
   
7th ciently small (or certai n other degenerate cases—including all of the points 
), points to lie on th e ellipse. In general, however, some points will be inside, 
some will be outside, and few if any will actually lie on the ellipse itself.  Of course, if e number of points is suffi
being collinear  it is possible for all of the

the different available fittin g methods. In the case of , the cost function will become the 
familiar least-squares fitting procedure that is probably st readers from elementary statistics. 
The more complex distan useful when m methods  are needed (i.e., fitting 
u a orfamiliar to mo
ce functions are ore robust fitting 
 
 methods that handle o tlier d ta points more gracefully). Table 8-2  shows the available forms f  and 
ST_L2 the associated OpenCV enum values used by cv::fitLine() .  
Table 8-2: Available distance metrics for the distType  parameter in cv::fitLine()  
distType Distance Metric   
Least-squares 
method 
 cv::DI
cv::DIST_L1 
  
cv::DIST_L12 
  
cv::DIST_FAIR 
  
 cv::DIST_WELSCH 
 
cv::DIST_HUBER 
 
The OpenCV function cv::fitLine()  has the followi ng function p rototype: 
void cv::fitLine( 
  cv::InputArray  points,      // Array  2-dimensional points  or vector of
  cv::OutputArray line,          // Vector of Vec4f (2d), or Vec6f (3d)

method 
 cv::DI
  
  cv::OutputArray line,          // Vector of Vec4f (2d), or Vec6f (3d) 
  int             distType,      // Distance type (Table 8-2) 
  double          param,         // Parameter for distance metric (Table 8-2 ) 
  double          reps,          // Radius accuracy parameter 
  double          aeps           // Angle accuracy parameter 
); 
The argument points  is mostly what you have come to expect, a representation of a set of points either
a cv::Mat  array or an STL vector. One very important difference, however, between  as 
c  
 
t, 
r 
his 
 cv::fitLine() matically select the optim e for

double          param,         // Parameter for distance metric (Table 8-2 ) 
); 
c  
 
t, 
r 
his 
 cv::fitLine() matically select the optim e for 
 
r v::fitLine()
al valuand many of the other functions we are looking at in this section is that cv::fitLine()  accepts both
two- and three-dimensional points. The output line is a little strange. That entry should be of type
cv::Vec4f  (for a two-dimensional line), or cv::Vec6f  (for a three-dimensional line) where the first 
half of the values gives the line direction and the s econd half a point on the line. The third argumen
distType , allows us to select the distance metric we  would like to use. The possible values fo

distType  are shown in Table 8-2 . The argument param is used to supply a value for the parameters 
used by some of the distance metrics (these parameters appear as the variables C in Table 8-2 ). T
parameter can be set to 0, in which case  will auto
the selected distance metric. The parameters reps and aeps represent your requ ired accuracy for the
origin of the fitted line (the x, y, z pa rts) and for the angle of the line (the v x, vy, vz parts). Typical values fo
these parameters are  
   for both of them.

Finding the Convex Hull of a Contour Using cv::convexHull()  
ere are many situations in which we need to simpTh lify a polygon by finding its convex hull . The convex 
hull of a polygon or contour is the polygon which completely contains the original, is made only of 
from the original, and is everywpoints 
here convex (i.e., that the internal angle between any three sequential 
poin s less than 180 degrees). An example of a convex hull can be seen in Figure 8-6 . There are many 
reasons to compute convex hulls. One particularly common reason is that testing if a point is inside of a 
convex polygon can be very fast, and it is often worthwhile to test first if a point is inside of the convex hull of a complicated polygon before even bothering to test if it is in the true polygon. ts i
 
Figure 8-6: An image (a) is converted to a contour (b ). The Convex hull of that contour (c) has far fewer 
points and is a much simpler piece of geometry.

points and is a much simpler piece of geometry. 
In order to compute the convex hull of a contour, OpenCV provides the function cv::convexHull() : 
void cv::convexHull( 
  cv::InputArray  points,             // Array or vector of 2-dimensional points 
  cv::OutputArray hull,                 // Can be array of points or of (int) indices 
  bool            clockwise    = false, // if true, output points will be clocwise 
  bool            returnPoints = true   // true for points in ‘hull’, else indices 
);

); 
The points  input to cv::convexHull()  can be any of the usual representations of a contour. The 
argument hull is where the resulting convex hull will appear. For this argument, you have two options: if 
you like, you can provide the usual contour type structure and cv::convexHull()  will fill that up with 
the points in the resulting convex hull. The other option is to provide not an array of points but an array of

bool            returnPoints = true   // true for points in ‘hull’, else indices 
); 
the points in the resulting convex hull. The other option is to provide not an array of points but an array of 
integers. In this case, cv::convexHull()  will associate an index with each point that is to appear in the 
hull and place those indices in hull. In this case, the indexes will begin at zero and the index value i will 
refer to the point points[i] . 
The clockwise  argument indicates how you would like to have cv::convexHull()  express the hull 
it computes. If clockwise  is set to true, then hull will be in a clockwise order, otherwise it will be in a

it computes. If clockwise  is set to true, then hull will be in a clockwise order, otherwise it will be in a 
counter-clockwise order.  The final argument, returnPoints , is associated with the option to return 
point indices rather than point values. If points  is an STL vector object, this argument is ignored, 
because the type of the vector template (int vs. cv::Point ) can be used to infer what you want. If, 
however, points  is a cv::Mat  type array, returnPoints  must be set to true if you are expecting 
point coordinates, and false if you are expecting indices. 
Geometrical Tests

Geometrical Tests 
When dealing with bounding boxes and other summary representations of polygon contours, it is often 
desirable to perform such simple geometrical checks as polygon overlap or a fast overlap check between 
bounding boxes. OpenCV provides a small but handy set of routines for this sort of geometrical checking.

Many important tests that apply to rectangles are su pported through interfaces pr ovided by those rectangle 
types. For example, the contains()  method of type cv::Rect  can be passed a point and it will 
determine if that point is inside the rectangle. 
Similarly, the minimal rectangle cont aining two rectangles can be comput ed with the logical “or” operator 
(e.g., rect1 | rect2 ), while the intersection of two rectangles can be computed with the logical “and” 
operator (e.g., rect1 & rect2 ). For cv::RotatedRect , we have no such functions since the result 
is mostly not a rectangle.  
For operations on general curves, however, there are library functions that can be used.  
25BTesting If a Contour Is Convex with cv::isContourConvex()  
A common thing to want to know about a contour is whet her or not it is convex. Ther e are lots of reasons to 
do this, but one of the most common reasons is that th ere are a lot of algorithms for working with polygons

do this, but one of the most common reasons is that th ere are a lot of algorithms for working with polygons 
that either only work on convex polygons or that can be simplified dramatically for the case of convex 
polygons. To test if a polygon is convex, simply call cv::isContourConvex()  and pass it your 
contour in any of the usual representations. The cont our passed will always be assumed to be a closed 
polygon (i.e., a link between the last and first points in the contour is presumed to be implied): 
bool cv::isContourConvex(      // Return true if contour is convex

bool cv::isContourConvex(      // Return true if contour is convex 
  cv::InputArray  contour    // Array or vector of 2-dimensional points 
); 
Because of the nature of the implementation, cv::isContourConvex()  requires that the contour 
passed to it be a simple polygon . This means that the contour must not have any self-intersections.  
26BTesting If a Point Is Inside a Polygon with cv::pointPolygonTest()  
double cv::pointPolygonTest(   // Return distance to polygon boundary (or just side) 
  cv::InputArray contour,    // Array or vector of 2-dimensional points 
  cv::Point2f    pt,           // Test point

); 
  cv::Point2f    pt,           // Test point 
  bool           measureDist   // If true, return distance, otherwise, {0,+1,-1} only 
); 
This first geometry toolkit function is cv::pointPolygonTest() , which allows you to test whether a 
point is inside a polygon (indicated by the array contour ). In particular, if the argument measureDist  
is set to true , then the function returns the di stance to the nearest contour edge; that distance is 0 if the 
point is inside the contour and positive if the point is outside. If the measure_dist  argument is false

); 
point is inside the contour and positive if the point is outside. If the measure_dist  argument is false  
then the return values are simply + 1, – 1, or 0 depending on whether the point is inside, outside, or on an 
edge (or vertex), respectively. As always, the c ontour itself can be either an STL vector or an n-by-1 two-
channel array of points. 
2BMatching Contours and Images 
Now that we have a pretty good idea of what a contour is and of how to work with contours as objects in 
OpenCV, we would like to move to the topic of how to use them for some practical purposes. The most 
common task associated with contours is matching them in some way with one another. We may have two

common task associated with contours is matching them in some way with one another. We may have two 
computed contours that we’d like to compare or a computed contour and some abstract template with 
which we’d like to compare our contour. We will discuss both of these cases. 
12BMoments 
One of the simplest ways to compare two contours is to compute what are called contour moments . Contour 
moments represent certain high-level characteristics of a contour, an image, or a set of points. (The entire

moments represent certain high-level characteristics of a contour, an image, or a set of points. (The entire 
discussion that follows will apply equally well to contours, images, or point sets, so for convenience we will just refer to these options collectively as objects .) Numerically, the moments are defined by the 

In this expression, the moment 
  is defined as a sum over all of the pixels in the object, in which the 
value of the pixel at point x, y is multiplied by the factor 
 . For example, if the image is a binary image 
(i.e., one in which every pixel is either zero of one), then 
  is just the area of the nonzero pixels in the 
image. In the case of a contour, the result is the length of the contour, F8
F and in the case of a point set it is just 
the number of points. After a little t hought, you should be able to convince yourself that for the same binary 
image, the 
 and 
 moments, divided by the 
  moment, are the average x and y values across the 
object. The term “moments” relates to the how this term is used in statistics, and the higher order moments 
can be related to what are called the moments of a statis tical distribution (i.e., area, average, variance, etc.). 
In this sense, you can think of the moments of a non-binary image as being the moments of a binary image

image, the 
 and 
In this sense, you can think of the moments of a non-binary image as being the moments of a binary image 
in which individual pixels are individually weighted. 
27BComputing Moments with cv::moments()  
The function that computes these moments for us is 
cv::Moments cv::moments(             // Return structure contains moments 
  cv::InputArray points,             // 2-dimensional points or an “image” 
  bool           binaryImage = false // If false, interpret image values as “mass” 
) 
The first argument, points , is the contour we are interested in, and the second, binaryImage , tells 
OpenCV if the input image should be interpreted as a binary image. The points  argument can be either a

) 
OpenCV if the input image should be interpreted as a binary image. The points  argument can be either a 
two-dimensional array (in which case it will be understood to be an image) or a set of points represented as 
an N-by-1 or 1-by-N array (with two channels) or an STL vector of cv::Point  objects. In the latter cases 
(the sets of points), cv::moments  will interpret these points not as a discrete set of points, but as a 
contour with those points as vertices. F9
F The meaning of this second argument is that if true , all nonzero

F The meaning of this second argument is that if true , all nonzero 
pixels will be treated as having value one (1), rather than whatever actual value is stored there. This is 
particularly useful when the image is the output of a threshold operation, but which might, for example, 
have 255 as its nonzero values. The cv::moments()  function returns an instance of the cv::Moments  
object. That object is  defined as follows: 
class Moments { 
public: 
                                                           
8 Mathematical purist s might object that 
  should be not the contour’s length but rather its area. But because we are

public: 
                                                           
  should be not the contour’s length but rather its area. But because we are 
looking here at a contour and not a filled polygon, the length a nd the area are actually the same  in a discrete pixel space 
(at least for the relevant distance measure in our pixel space). There are also functions for computing moments of 
IplImage  images; in that case, 
  would actually be the area of nonzero pixe ls. Indeed the distinction is not entirely 
academic, however, a contour is actually represented as a set of vertex points, the formula used to compute the length

academic, however, a contour is actually represented as a set of vertex points, the formula used to compute the length 
will not give precisely the same area as would be co mputed by first rasterizing the contour (i.e., using 
cv::drawContours() ) and then computing the area of that rast erization—though the two should converge to the 
same value in the limit of infinite resolution. 
9 In such case as you may need to handle a set of points, ra ther than a contour, it is most convenient to simply create an 
image containing those points.

double m00;                    //   zero order moment               (x1) 
    double m10, m01;               //  first order moments              (x2) 
    double m20, m11, m02;          // second order moments              (x3) 
    double m30, m21, m12, m03;     //  third order moments              (x4) 
    double mu20, mu11, mu02;       // second order central moments      (x3) 
    double mu30, mu21, mu12, mu03; //  third order central moments      (x4) 
    double nu20, nu11, nu02;       // second order Hu invariant moments (x3) 
    double nu30, nu21, nu12, nu03; //  third order Hu invariant moments (x4) 
  Moments(); 
  Moments( 
    double m00,  
    double m10, double m01,  
    double m20, double m11, double m02,  
    double m30, double m21, double m12, double m03  
  ); 
  Moments( const CvMoments& moments );  // convert v1.x struct to C++ object 
  operator CvMoments() const;           // convert C++ object to v1.x struct 
}

double nu20, nu11, nu02;       // second order Hu invariant moments (x3) 
  Moments( 
  ); 
}  
A single call to cv::moments()  will compute all of the moments up to third order (i.e., moments for 
which 
 ). It will also compute what are called central moments  and normalized central moments . 
We will discuss those next. 
13BMore About Moments 
The moment computation just described gives some rudimentary characteristics of a contour that can be

operator CvMoments() const;           // convert C++ object to v1.x struct 
}  
which 
The moment computation just described gives some rudimentary characteristics of a contour that can be 
used to compare two contours. However, the moments resulting from that computation are not the best parameters for such comparisons in most practical cases. In general, the moments we have discussed so far 
will not be the same for two otherwise identical contou rs which are displaced relative to one another, of 
different size, or rotated relative to one another.  
28BCentral Moments Are Invariant Under Translation 
Given a particular contour or image, the 
 moment of that contour will clearly be the same no matter

 moment of that contour will clearly be the same no matter 
where that contour appears in an image. The higher or der moments, however, clearly will not be. Consider 
the 
  moment, which we identified earlier with the aver age x-position of a pixel in the object. Clearly 
given two otherwise identical objects in different places, the average x-pos ition is different. It may be less 
obvious on casual inspection, but the second order moments, which tell us something about the spread of 
the object, are also not in variant under translation F10
F. This is not particularly convenient, as we would 
certainly like (in most cases) to be able to use thes e moments to compare an ob ject that might appear

certainly like (in most cases) to be able to use thes e moments to compare an ob ject that might appear 
anywhere in an image to a refere nce object that appeared  somewhere (probably somewhere else) in some 
reference image. 
The solution to this is to compute what are called central moments, which are usually denoted 
 and 
defined by the following relation: 
 
where: 
                                                           
10 For those who are not into this sort of math jargon, th e phrase “invariant under translati on” means that some quantity 
computed for some object is unchanged if that entire object is moved (i.e., “tra nslated”) from one place to another in

 and 
 
where: 
                                                           
computed for some object is unchanged if that entire object is moved (i.e., “tra nslated”) from one place to another in 
the image. The phrase “invariant under ro tation” similarly means that the quanti ty being computed is unchanged of the 
object is rotated in the image.

, and 
  
Of course, it should be immediately clear that 
 =
  (because the terms involving p and q vanish 
anyhow), and that the 
  and 
  central moments are both equal to zero as well. The higher order 
moments are thus the same as the non-central moment s but measured with respect  to the “center of mass” 
of (or in the coordinates of the center of mass of) th e object as a whole. Because these measurements are 
relative to this center, they do not change if the object appears in any arbitrary location in the image. 
You will notice that there are no elements mu00 , mu10 , or mu01  in the object 
cv::Moments. This is simply because these values are “trivial” (i.e., mu00=m00 , and 
mu10=mu01=0 ). The same is true for the normalized central moments (except that 
nu00=1 , while nu10  and nu01  are both zero). For this reas on they are not included in 
the structure, as they would just wast e memory storing redundant information. 
29BNormalized Central Moments Are Also Invariant Under Scaling

29BNormalized Central Moments Are Also Invariant Under Scaling 
Just as the central moments allow for us to compare tw o different objects that are in different locations in 
our image, it is also often important to be able to co mpare two different objects that are the same except for 
being different sizes. (This sometimes happens because we are looking for an object of a type that appears 
in nature of different sizes—e.g., bears—but more of ten it is simply because we do not necessarily know 
how far the object will be from the imager that generated our image in the first place.)

how far the object will be from the imager that generated our image in the first place.) 
Just as the central moments are defined based on th e original moments by subtracting out the average to 
achieve translational invariance, the normalized central moments  achieve scale invariance by factoring out 
the overall size of the object. The formula for the normalized central moments is the following: 
 
This marginally intimidating formula just says that the normalized central moments are equal to the central 
moments up to a normalization factor that is itself ju st some power of the area of the object  (with that 
power being greater for higher order moments).

power being greater for higher order moments). 
There is no specific function for computing normali zed moments in OpenCV, as they are computed 
automatically by cv::Moments()  when the standard and central moments are computed. 
30BHu Invariant Moments are Invariant Under Rotation 
Finally, the Hu invariant moments  are linear combinations of the norm alized central moments. The idea 
here is that, by combining the different normalized central moments, it is possible to create functions

here is that, by combining the different normalized central moments, it is possible to create functions 
representing different aspects of the im age in a way that is invariant to scale, rotation, and (for all but the 
one called h1) reflection. 
For the sake of completeness, we show here the actual definitions of the Hu moments:

-
  
Looking at X
  
Figure 8-7 X and XTable 8-3 X, we can gain a sense of how the Hu moments behave. Observe first that the 
moments tend to be smaller as we move to higher orders. This should be no surprise in that, by their 
definition, higher Hu moments have more powers of va rious normalized factors. Since each of those factors 
is less than 1, the products of more and more of them will tend to be smaller numbers. 
 
Figure 8-7: Images of five simple c haracters; looking at their Hu moment s yields some intuition concerning 
their behavior. 
Table 8-3: Values of the Hu moments fo r the five simple characters shown in 
X
  
Figure 8-7 X.       
 
h1 h 2 h 3 h 4 h 5 h 6 h 7 
A 2.837e–1 1.961e–3 1.484e–2 2.265e –4 –4.152e–7 1.003e–5 –7.941e–9 
I 4.578e–1 1.820e–1 0.000 0.000 0.000 0.000 0.000 
O 3.791e–1 2.623e–4 4.501e–7 5.858e –7 1.529e–13 7.775e–9 –2.591e–13 
M 2.465e–1 4.775e–4 7.263e–5 2.617e –6 –3.607e–11 –5.718e–8 –7.218e–24

X
  
M 2.465e–1 4.775e–4 7.263e–5 2.617e –6 –3.607e–11 –5.718e–8 –7.218e–24 
F 3.186e–1 2.914e–2 9.397e–3 8.221e –4 3.872e–8 2.019e–5 2.285e–6

Other factors of particular interest are that the “I ”, which is symmetric under 180 degree rotations and 
reflection, has a value of exactly 0 for h3 through h7 and that the “O,” which has similar symmetries, has 
all nonzero moments. We leave it to the reader to l ook at the figures, compare the various moments, and 
build a basic intuition for what those moments represent. 
31BComputing Hu Invariant Moments with cv::HuMoments()   
While the other moments were all computed with the same function cv::moments() , the Hu invariant 
moments are computed with a second function that takes the cv::Moments object you got from 
cv::moments()  and returns a list of numbers for the seven invariant moments:  
void cv::HuMoments( 
  const cv::Moments& moments, // Input is result from cv::moments() function 
  double*            hu       // Return is C-style array of 7 Hu moments 
); 
The function cv::HuMoments()  expects a cv::Moments  object and a pointer to a C-style array you

); 
The function cv::HuMoments()  expects a cv::Moments  object and a pointer to a C-style array you 
should have already allo cated with room for the seven invariant moments. 
14BMatching and Hu Moments 
double  cv::MatchShapes( 
  cv::InputArray object1,      // First array of 2-d points or cv:U8C1 image 
  cv::InputArray object2,      // Second array of 2-d points or cv:U8C1 image   
  int            method,       // Comparison method ( XTable 8-4 X) 
  double         parameter = 0 // Method-specific parameter (for later extensions) 
); 
Naturally, with Hu moments we would like to compare two objects and determine wh ether they are similar.

); 
Naturally, with Hu moments we would like to compare two objects and determine wh ether they are similar. 
Of course, there are many possible definitions of “s imilar.” To make this process somewhat easier, the 
OpenCV function cv::matchShapes()  allows us to simply provide two objects and have their 
moments computed and compared accordi ng to a criterion that we provide. 
These objects can be either grayscale images or contours. In either case, cv::matchShapes()  will 
compute the moments for you before proceeding with the comparison. The method  used in 
cv::matchShapes()  is one of the three listed in XTable 8-4 X.

cv::matchShapes()  is one of the three listed in XTable 8-4 X. 
Table 8-4: Matching methods used by cv::matchShapes()  
Value of method cv::matchShapes()  return value 
cv::CONTOURS_MATCH_I1 
 
cv::CONTOURS_MATCH_I2  
 
cv::CONTOURS_MATCH_I3 
 
In the table, 
  and 
 are defined as:

In these expressions, 
  and 
  are the Hu invariant moments of images A and B, respectively. 
Each of the three values defined in XTable 8-4 X has a different meaning in terms of how the comparison 
metric is computed. This metric determines the value ultimately returned by cv::matchShapes() . The 
final parameter  argument is not currently used, so we can safely leave it at the default value of 0 (it is 
there for future comparison metrics that may re quire an additional user provided parameter). 
3BSummary 
In this chapter we learned about contours, sequences of points in two dimensions.  These sequences could 
be represented as STL vectors of two-dimensional point objects (e.g., cv::Vec2f ), as N-by-1 dual 
channel arrays, or as N-by-2 single channel arrays.  Such  sequences can be used to represent contours in in 
image plane, and there are many features built into th e library to help us construct and manipulate these 
contours.

3BSummary 
contours. 
Contours are generally useful for representing spatial pa rtitions of an image.  In this context, the OpenCV 
library provides us with tools for comparing such par titions to one another, as well as for testing properties 
of these partitions, such as convexity, moments, or the relationship of an arbitrary point with such a 
contour. 
4BExercises 
1. Finding the extremal points (i.e., the two points that are farthest apart) in a closed contour of N points 
can be accomplished by comparing the dist ance of each point to every other point.

contour. 
can be accomplished by comparing the dist ance of each point to every other point. 
a) What is the complexity of such an algorithm? 
b) Explain how you can do this faster. What is the maximal closed contour length that could fit into a 
4-by-4 image? What is its contour area? 
2. Using PowerPoint or a similar program, draw a white circle of radius 20 on a black background (the 
circle’s circumference will thus be 2 π 20 ≈ 125.7. Save your drawing as an image. 
a) Read the image in, turn it into grayscale, threshold, and find the contour. What is the contour 
length? Is it the same (within rounding) or different from the calculated length?

length? Is it the same (within rounding) or different from the calculated length? 
b) Using 125.7 as a base length of the contour, run cv::approxPolyDP()  using as parameters 
the following fractions of the base length: 90%, 66%, 33%, 10%. Find the contour length and 
draw the results. 
3. Suppose we are building a bottle detector and wish to  create a “bottle” feature. We have many images 
of bottles that are easy to segment and find the c ontours of, but the bottles are rotated and come in

of bottles that are easy to segment and find the c ontours of, but the bottles are rotated and come in 
various sizes. We can draw the contours and then find the Hu moments to yield an invariant bottle-feature vector. So far, so good—but should we draw  filled-in contours or just line contours? Explain 
your answer. 
4. When using cv::moments()  to extract bottle contour moments in exercise 
X3 X, how should we set 
isBinary ? Explain your answer. 
5. Take the letter shapes used in the discussion of Hu  moments. Produce variant images of the shapes by

5. Take the letter shapes used in the discussion of Hu  moments. Produce variant images of the shapes by 
rotating to several different angles , scaling larger and smaller, an d combining these transformations. 
Describe which Hu features respond to rotation, which to scale, and which to both. 
6. Make a shape in PowerPoint (or another drawing prog ram) and save it as an image. Make a scaled, a 
rotated, and a rotated and scaled version of the ob ject and then store these as images. Compare them

using cv::matchShapes(). How do the match scores compare for Hu 
moments verses ordinary moments?  
7. Use a depth sensor such as the kinect camera to  segment your hands. Use moments to attempt to 
recognize various gestures.

9 
Background Subtraction 
0BOverview of Background Subtraction 
Because of its simplicity and because camer a locations are fixed in many contexts, background subtraction  
(aka background differencing ) is a fundamental image processing oper ation for video security applications. 
Toyama, Krumm, Brumitt, and Meyers give a good overview and comparison with many techniques 
[Toyama99]. In order to perform background subtraction, we first must “learn” a model of the background. 
Once learned, this background model  is compared against the curr ent image and then the known 
background parts are subtracted away . The objects left after subtractio n are presumably new foreground 
objects. 
Of course, “background” is an ill-defined concept that  varies by application. For example, if you are 
watching a highway, perhaps averag e traffic flow should be considered  background. Normally, background 
is considered to be any static or periodically moving parts of a scene that remain static or periodic over the

objects. 
is considered to be any static or periodically moving parts of a scene that remain static or periodic over the 
period of interest. The whole ensemble may have time-varying components, such as trees waving in 
morning and evening wind but standing still at noon. Two common but substantially  distinct environment 
categories that are likely to be encountered are indoor and outdoor sc enes. We are interested in tools that 
will help us in both of these environments. First we will discuss the weaknesses of typical background

will help us in both of these environments. First we will discuss the weaknesses of typical background 
models and then will move on to discuss higher-level scene models. In that context, we present a quick method that is mostly good for indoor static background scenes whose lighting doesn’t change much. We 
then follow this by a “codebook” method that is slightly slower but can work in both outdoor and indoor 
scenes; it allows for periodic movements (such as the trees waving in the wind) and for lighting to change slowly or periodically. This method is also tolerant to learning the background even when there are

scenes; it allows for periodic movements (such as the trees waving in the wind) and for lighting to change slowly or periodically. This method is also tolerant to learning the background even when there are 
occasional foreground objects moving by. We’ll t op this off by another discussion of connected 
components (first seen in Chapter 5) in the context of  cleaning up foreground object detection. We will 
then compare the quick background method against the codebook background method. This chapter will 
conclude with a discussion of the implementations available in the OpenCV library of two modern 
algorithms for background subtraction.  These algorithms use the principles discussed in the chapter, but 
also include both extensions and implementation details which make them more suitable for real-world 
application. 
1BWeaknesses of Background Subtraction

1BWeaknesses of Background Subtraction 
Although the background modeling methods mentioned here work fairly well for simple scenes, they suffer 
from an assumption that is often violated: namely that the behavior of all of the pixels in the image is 
statistically independent from the behavior of all of th e others. Notably, the methods we describe here learn

a model for the variations a pixel experiences without considering any of its neighboring pixels. In order to 
take surrounding pixels into account, we could lear n a multipart model, a simple example of which would 
be an extension of our basic independent pixel model to include a rudimentary sense of the brightness of 
neighboring pixels. In this case, we use the brightness of neighboring pixels to distinguish when neighboring pixel values are relatively bright or dim. We then learn effectively two models for the 
individual pixel: one for when the surrounding pixels are bright and one for when the surrounding pixels 
are dim. In this way, we have a model that takes into account the surrounding context . But this comes at the 
cost of twice as much memory use and more computati on, since we now need diff erent values for when the 
surrounding pixels are bright or dim. We also need tw ice as much data to fill out this two-state model. We

surrounding pixels are bright or dim. We also need tw ice as much data to fill out this two-state model. We 
can generalize the idea of “high” and “low” contexts to a multidimensional histogram of single and surrounding pixel intensities as well as make it even more complex by doing all this over a few time steps. 
Of course, this richer model over space and time w ould require still more memory, more collected data 
samples, and more computational resources.
F1 
Because of these extra costs, the mo re complex models are usually avoide d. We can often more efficiently 
invest our resources in cleaning up the false positive  pixels that result when the independent pixel

F1 
invest our resources in cleaning up the false positive  pixels that result when the independent pixel 
assumption is violated. This cleanup usually takes the form of image processing operations 
(cv::erode(), cv::dilate() , and cv::floodFill() , mostly) that eliminate stray patches of 
pixels. We’ve discussed these routines previously (Chapter 5) in the context of finding large and compact F2
F 
connected components  within noisy data. We will employ connected components again in this chapter and 
so, for now, will restrict our discussion to approaches that assume pixels vary independently. 
2BScene Modeling 
How do we define background and foreground? If we’re watching a parking lot and a car comes in to park,

F 
How do we define background and foreground? If we’re watching a parking lot and a car comes in to park, 
then this car is a new foreground object. But should it stay foreground forever? How about a trash can that 
was moved? It will show up as foreground in two places: the place it was moved to and the “hole” it was 
moved from. How do we tell the difference? And again, how long should the trash can (and its hole) remain 
foreground? If we are modeling a dark room and suddenly someone turns on a light, should the whole room 
become foreground? To answer these questions, we need  a higher-level “scene” model, in which we define

become foreground? To answer these questions, we need  a higher-level “scene” model, in which we define 
multiple levels between foreground and background states, and a timing-based method of slowly relegating unmoving foreground patches to background patches. We will also have to detect and create a new model 
when there is a global change in a scene. 
In general, a scene model might contain multiple layers, from “new foreground” to older foreground on 
down to background. There might also be some motion detection so that, when an object is moved, we can 
identify both its “positive” aspect (its new location) and its “negative” aspect (its old location, the “hole”).

identify both its “positive” aspect (its new location) and its “negative” aspect (its old location, the “hole”). 
In this way, a new foreground object would be put in  the “new foreground” object level and marked as a 
positive object or a hole. In areas where there was no foreground object, we could continue updating our 
background model. If a foreground object does not move for a given time, it is demoted to “older foreground,” where its pixel statistics are provisi onally learned until its learned model joins the learned 
background model. 
For global change detection such as turning on a light in a room, we might use global frame differencing. 
For example, if many pixels change at once, then we could classify it as a global  rather than local change

For example, if many pixels change at once, then we could classify it as a global  rather than local change 
and then switch to using a different model for the new situation. 
                                                          
1 In cases in which a computer is expected to “learn” something from data, it is often the case that the primary practical 
obstacle to success turns out to be having enough data. The more complex your model becomes, the easier it is to get 
yourself into a situation in which the expressive power of  your model vastly exceeds your capability to generate

yourself into a situation in which the expressive power of  your model vastly exceeds your capability to generate 
training data for that model. We will re-encounter this issu e in more detail in the later chapter on machine learning. 
2 Here we are using mathematicia n’s definition of “compact,” wh ich has nothing to do with size.

10BA Slice of Pixels 
Before we go on to modeling pixel changes, let’s get an idea of what pixels in an image can look like over 
time. Consider a camera looking out a window to a scene of a tree blowing in the wind. Error! Reference 
source not found.  shows what the pixels in a given line segmen t of the image look like over 60 frames. We 
wish to model these kinds of fluctuations. Before doing  so, however, we make a small digression to discuss 
how we sampled this line because it’s a generally useful trick for creating features and for debugging. 
 
Figure 9-1: Fluctuations of a line of pixels in a sc ene of a tree moving in the wind over 60 frames: some 
dark areas (upper-left) are quite stable, whereas moving branches (upper-center) can vary widely 
Because this comes up quite often in various contexts, OpenCV makes it easy to samp le an arbitrary line of 
pixels. This is done with the object called the line iterator, which we encountered way back in Chapter 3.

pixels. This is done with the object called the line iterator, which we encountered way back in Chapter 3. 
The line iterator, cv::LineIterator , is an object which, once instantiated, can be queried to give us 
information about all of the points along a line in sequence. 
The first thing we need to do is to instantiate a line iterator object. We do this with the 
cv::LineIterator  constructor: 
cv::LineIterator::LineIterator( 
  const cv::Mat& image,                 // Image to iterate over 
  cv::Point      pt1,                   // Start point for iterator 
  cv::Point      pt2,                   // End point for iterator 
  int            connectivity  = 8,     // Connectivity, either 4 or 8

  int            connectivity  = 8,     // Connectivity, either 4 or 8 
  int            left_to_right = 0      // Iteration direction, 0 (false) or 1 (true) 
); 
Here, the input image  may be of any type or number of channels. The points pt1 and pt2 are the ends of 
the line segment. The connectivity  can be 4 (the line can step right , left, up, or down) or 8 (the line 
can additionally step along the diagonals). Finally, if left_to_right  is set to 0 (false), then 
line_iterator  scans from pt1 to pt2; otherwise, it will go from the leftmost to the rightmost point. F3
F

int            left_to_right = 0      // Iteration direction, 0 (false) or 1 (true) 
); 
F  
The iterator can then just be increm ented through, pointing to  each of the pixels along the line between the 
given endpoints points. We increment the iterator with the usual cv::LineIterator 
::operator++() . All the channels are available at once. If , for example, our line iterator is called 
line_iterator , then we can access the current point by dereferencing the iterator (e.g., 
*line_iterator ). One word of warning is in order here, ho wever, which is that the return type of

F  
*line_iterator ). One word of warning is in order here, ho wever, which is that the return type of 
cv::LineIterator::operator*()  is not a pointer to a built-in OpenCV vector type (i.e., 
cv::Vec<>  or some instantiation of it), but rather it is a uchar*  pointer. This means that you will 
                                                           
3 The left_to_right  flag was introduced because a discrete line drawn from pt1 to pt2 does not always match 
the line from pt2 to pt1. Therefore, setting this flag gives the user a consistent rasterization regardless of the pt1, 
pt2 order.

typically want to cast this value yourself to something like cv::Vec3f*  (or whatever is appropriate for 
the array image ). F4
F  
With this convenient tool in hand, we can take look at how it can be used to extract data from a scene. The 
program in XExample 9-1 X reads a movie file and generates the sort of data seen in XFigure 9-1 X. 
Example 9-1: Reads out the RGB valu es of all pixels in one row of a video and accumulates those values 
into three separate files 
#include <fstream> 
 
void help() { 
  cout << "\n" 
    << "Read out RGB pixel values and store them to disk\nCall:\n"  
    << "./ch9_ex9_1 avi_file\n"  
    << "\n This will store to files blines.csv, glines.csv and rlines.csv\n\n"  
    << endl; 
} 
 
int main( int argc, char** argv  ) { 
 
  if(argc != 2) { help(); return -1; } 
  cv::namedWindow( "Example9_1", CV_WINDOW_AUTOSIZE ); 
 
  cv::VideoCapture cap; 
  if((argc < 2)|| !cap.open(argv[1])) 
  {

    << endl; 
} 
 
  { 
  cerr << "Couldn't open video file" << endl; 
   help(); 
    cap.open(0); 
    return -1; 
  } 
 
  cv::Point pt1(10,10), pt2(30,30); 
  int max_buffer; 
  cv::Mat rawImage; 
  ofstream b,g,r; 
  b.open("blines.csv"); 
  g.open("glines.csv"); 
  r.open("rlines.csv"); 
 
  // MAIN PROCESSING LOOP: 
  // 
  for(;;) { 
   cap >> rawImage;  
    if( !rawImage.data ) break; 
 
    cv::LineIterator it( rawImage, pt1, pt2, 8); 
    for( int j=0; j<it.count; ++j,++it ) { 
      b << (int)(*it)[0] << ", ";

{ 
   help(); 
    return -1; 
  } 
 
  // 
  for(;;) { 
 
      b << (int)(*it)[0] << ", "; 
                                                           
4 In some cases, you can get away with being a little slopp y here. Specifically, when the image is already of unsigned 
character type, you can just access the elements directly with constructions like (*line_iterator)[0] , or 
(*line_iterator)[1] , and so on. On close inspection, these are actually dereferencing the iterator to get a

for( int j=0; j<it.count; ++j,++it ) { 
                                                           
(*line_iterator)[1] , and so on. On close inspection, these are actually dereferencing the iterator to get a 
character pointer, then using the built-in C offset dereferen ce bracket operator, rather than casting the dereferenced 
iterator to an OpenCV vector type like Vec3f and accessi ng the channel through the overloaded dereferencing operator 
of that class. In the end, for the special case of Vec3b (or any number of channels), it happens to all come to the same 
thing.

g << (int)(*it)[1] << ", "; 
      r << (int)(*it)[2] << ", "; 
      (*it)[2] = 255;               // Mark this sample in red 
    } 
    cv::imshow( "Example9_1", rawImage ); 
    int c = cv::waitKey(10); 
    b << "\n"; g << "\n"; r << "\n"; 
  } 
 
  // CLEAN UP: 
  // 
  b << endl; g << endl; r << endl; 
  b.close(); g.close(); r.close(); 
    << "Data stored to files: blines.csv, glines.csv and rlines.csv\n\n"  
    << endl; 
} 
In Example 9-1, we stepped through the points, one at a time, and processed each one. Another common 
and useful way to approach the problem is to create a buffer (of the appropriate type) and copy the entire 
line into that buffer before processi ng the buffer. In that ca se, the buffer copy would have looked something 
like the following: 
LineIterator it(rawImage, pt1, pt2, 8); 
vector<Vec3b> buf(it.count); 
for( int i=0; i < it.count; i++, ++it ) buf[i] = (const Vec3b)*it;

<< endl; 
} 
for( int i=0; i < it.count; i++, ++it ) buf[i] = (const Vec3b)*it; 
The primary advantage of this ap proach is that if the image rawImage  were not of an unsigned character 
type, this method is a cleaner way of casting to the appropriate type. 
We are now ready to move on to some methods for modeling the kinds of pixel fluctuations seen in XFigure 
9-1 XError! Reference source not found. . As we move from simple to increasingly complex models, we 
shall restrict our attention to those models that will run in real time and within reasonable memory 
constraints. 
11BFrame Differencing

11BFrame Differencing 
The very simplest background subtraction method is to subtract one frame from another (possibly several 
frames later) and then label any difference that is “big  enough” the foreground. This process tends to catch 
the edges of moving objects. For simplicity, le t’s say we have three single-channel images: 
frameTime1, frameTime2 , and frameForeground . The image frameTime1  is filled with an 
older grayscale image, and frameTime2  is filled with the current grayscale image. We could then use the 
following code to detect the magnitude (absolute value) of foreground differences in

following code to detect the magnitude (absolute value) of foreground differences in 
frameForeground : 
cv::absdiff( 
  frameTime1,                    // First input array 
  frameTime2,                    // Second input array 
  frameForeground                // Result array  
); 
Because pixel values always exhibit noise and fluctuatio ns, we should ignore (set  to 0) small differences 
(say, less than 15), and mark the rest as big differences (set to 255): 
cv::threshold( 
  frameForeground,                // Input Image 
  frameForeground,                // Result image

); 
  frameForeground,                // Result image 
  15,                             // Threshold value 
  255,                            // Max value for upward operations 
  cv::THRESH_BINARY               // Threshold type to use 
);

The image frameForeground  then marks candidate foreground objects as 255 and background pixels 
as 0. We need to clean up small noise areas as discussed earlier; we might do this with cv::erode()  
followed by cv::dilate() or by using connected components. For color images, we could use the same 
code for each color channel and then combine the channels with the cv::max()  function. This method is 
much too simple for most applications  other than merely indicating regions  of motion. For a more effective 
background model we need to keep some statistics about the means and average differe nces of pixels in the 
scene. You can look ahead to the section entitled “ XA Quick X” to see examples of frame differencing in 
X
  
Figure 9-6 X and X
  
Figure 9-7 X. 
3BAveraging Background Method 
The averaging method basically learns the average and standard deviation (or similarly, but 
computationally faster, the average difference) of each pixel as its model of the background.

X
  
computationally faster, the average difference) of each pixel as its model of the background. 
Consider the pixel line from XFigure 9-1 X. Instead of plotting one sequence of values for each frame (as we 
did in that figure), we can represent the variations of  each pixel throughout the video in terms of an average 
value and a pixel’s associated average differences ( XFigure 9-2 X). In the same video, a foreground object

(which is, in fact, a hand) passes in front of the camer a. The resulting change in statistics of an associated 
pixel is shown in the figure. 
 
Figure 9-2: Data from XFigure 9-1 X presented in terms of average differences: an object (a hand) passes in 
front of the camera causing the sta tistics of a pixel’s value to change. 
The averaging method makes use of four OpenCV routines: cv::Mat::operator+=() , to accumulate 
images over time; cv::absdiff() , to accumulate frame-to-frame image differences over time; 
cv::inRange() , to segment the image (once a background mode l has been learned) into foreground and 
background regions; and cv::max() , to compile segmentations from different color channels into a 
single mask image. Because this is a rather long code  example, we will break it in to pieces and discuss each 
piece in turn. 
First, we create pointers for the vari ous scratch and statistics-keeping images we will need along the way. It

First, we create pointers for the vari ous scratch and statistics-keeping images we will need along the way. It 
will prove helpful to sort these pointers according  to the type of images they will later hold. 
 
// Global storage 
// 
// Float, 3-channel images 
// 
cv::Mat IavgF, IdiffF, IprevF, IhiF, IlowF; 
cv::Mat tmp, tmp2;  
 
// Float, 1-channel images 
// 
vector<cv::Mat> Igray(3); 
vector<cv::Mat> Ilow(3); 
vector<cv::Mat> Ihi(3); 
 
// Byte, 1-channel image 
// 
cv::Mat Imaskt; 
 
// Counts number of images learned for averaging later 
// 
float Icount;

// 
// 
 
// 
 
// 
 
// 
float Icount; 
Next, we create a single call to allocate all the necessary intermediate images. For convenience, we pass in 
a single image (from our video) that can be used as a reference for sizing the intermediate images: 
// I is just a sample image for allocation purposes 
// (passed in for sizing) 
//

void AllocateImages( IplImage* I ){ 
 
  CvSize sz = cvGetSize( I ); 
 
  IavgF     = cv::Mat::zeros( sz, CV::F32C3 ); 
  IdiffF    = cv::Mat::zeros( sz, CV::F32C3 ); 
  IprevF    = cv::Mat::zeros( sz, CV::F32C3 ); 
  IhiF      = cv::Mat::zeros( sz, CV::F32C3 ); 
  IlowF     = cv::Mat::zeros( sz, CV::F32C3 );  
  Icount    = 0.00001; // Protect against divide by zero 
 
  tmp       = cv::Mat::zeros( sz, CV::F32C3 ); 
  tmp2      = cv::Mat::zeros( sz, CV::F32C3 ); 
  Imaskt    = cv::Mat( sz, CV::F32C1 ); 
} 
In the next piece of code, we learn the accumulated  background image and the accumulated absolute value 
of frame-to-frame image differences (a computationally quicker proxy F5
F for learning the standard deviation 
of the image pixels). This is typically called for 30 to 1,000 frames, sometimes taking just a few frames from each second or sometimes taking all available frames. The routine will be called with a three-color-
channel image of depth 8 bits:

Imaskt    = cv::Mat( sz, CV::F32C1 ); 
} 
channel image of depth 8 bits: 
// Learn the background statistics for one more frame 
// I is a color sample of the background, 3-channel, 8u 
// 
void accumulateBackground( cv::Mat& I ){ 
 
  static int first = 1;                 // nb. Not thread safe 
  I.convertTo( tmp, CV::F32 );          // convert to float 
  if( !first ){ 
    IavgF += tmp; 
    cv::absdiff( tmp, IprevF, tmp2 ); 
    IdiffF += tmp2; 
    Icount += 1.0; 
  } 
  first = 0; 
  IprevF = tmp; 
}

// 
 
  } 
  first = 0; 
} 
We first use cv::Mat::convertTo()  to turn the raw background 8-bit-per-channel, three-color-
channel image into a floating-point three-channel imag e. We then accumulate the raw floating-point images 
into IavgF . Next, we calculate the frame-to-fra me absolute difference image using cv::absdiff()  
and accumulate th at into image IdiffF . Each time we accumulate these images, we increment the image 
count Icount , a global, to use for averaging later.

} 
  first = 0; 
} 
count Icount , a global, to use for averaging later. 
Once we have accumulated enough frames, we convert th em into a statistical m odel of the background. 
That is, we compute the means and deviation measures  (the average absolute di fferences) of each pixel: 
void createModelsfromStats() { 
 
    IavgF  *= (1.0/Icount); 
    IdiffF *= (1.0/Icount); 
 
    // Make sure diff is always something 
    // 
    IdiffF += cv::Scalar( 1.0, 1.0, 1.0 );

    // 
    IdiffF += cv::Scalar( 1.0, 1.0, 1.0 ); 
                                                           
5 Notice our use of the word “proxy.” Average difference is not  mathematically equivalent to  standard deviation, but in 
this context it is close enough to yield re sults of similar quality. The advantage of average difference is that it is faster 
to compute than the standard deviati on. With only a tiny modification of the code example you can use standard 
deviations instead and compare the quality of the final results for yourself; we’ll discuss this more explicitly later in 
this section.

setHighThreshold( 7.0 ); 
    setLowThreshold( 6.0 ); 
} 
In this section, we use cv::Mat::operator*=()  to calculate the average ra w and absolute difference 
images by dividing by the number of input images accumulated. As a precautio n, we ensure that the 
average difference image is at leas t 1; we’ll need to scale this f actor when calculating a foreground-
background threshold and would like to avoid the degenerate case in which these two thresholds could 
become equal. 
The next two routines, setHighThreshold()  and setLowThreshold() , are utility functions that 
set a threshold based on the frame-to-frame averag e absolute differences (FFAAD). The FFAAD can be 
thought of as the basic metric against which we compar e observed changes in order to determine if they are 
significant. The call setHighThreshold(7.0) , for example, fixes a threshold such that any value that 
is 7 times the FFAAD above average for that pixel is considered foreground; likewise,

is 7 times the FFAAD above average for that pixel is considered foreground; likewise, 
setLowThreshold(6.0)  sets a threshold bound that is 6 times the FFAAD below the average for that 
pixel. Within this range around the pixel’s average va lue, objects are considered  to be background. These 
threshold functions are: 
void setHighThreshold( float scale ) { 
  IhiF = IavgF + (IdiffF * scale); 
  cv::split( IhiF, Ihi ); 
} 
void setLowThreshold( float scale ) { 
  IlowF = IavgF - (IdiffF * scale); 
  cv::split( IlowF, Ilow ); 
}

} 
} 
In setLowThreshold()  and setHighThreshold() , we first scale the difference image (the 
FFAAD) prior to adding or subtr acting these ranges relative to IavgF . This action sets the IhiF  and 
IlowF  range for each channel in the image via cv::split() . 
Once we have our background model, complete with high and low thresholds, we use it to segment the 
image into foreground (things not “explained” by the background image) and the background (anything 
that fits within the high and low thresholds of our background model). Segmentation is done by calling:

that fits within the high and low thresholds of our background model). Segmentation is done by calling: 
// Create a binary: 0,255 mask where 255 means foreground pixel 
// I      Input image, 3-channel, 8u 
// Imask  Mask image to be created, 1-channel 8u 
// 
void backgroundDiff( 
  cv::Mat& I, 
  cv::Mat& Imask 
) { 
  I.convertTo( tmp, CV::F32 );    // To float 
  cv::split( tmp, Igray ); 
 
  // Channel 1 
  // 
  cv::inRange( Igray[0], Ilow[0], Ihi[0], Imask ); 
 
  // Channel 2 
  // 
  cv::inRange( Igray[1], Ilow[1], Ihi[1], Imaskt ); 
  Imask = cv::min( Imask, Imaskt ); 
 
  // Channel 3 
  //

// 
) { 
 
  // 
 
  // 
 
  // 
  cv::inRange( Igray[2], Ilow[2], Ihi[2], Imaskt );  
 
  // Finally, invert the results 
  //

Imask = 255 – Imask; 
} 
This function first converts the input image I (the image to be segmented) into a floating-point image by 
calling cv::Mat::convertTo() . We then convert the three-channe l image into separate one-channel 
image planes using cv::split() . These color channel planes are then checked to see if they are within 
the high and low range of the average background pixel via the cv::inRange()  function, which sets the 
grayscale 8-bit depth image Imaskt  to max (255) when it’s in range and to 0 otherwise. For each color 
channel, we logically AND F6
F the segmentation results into a mask image Imask , since strong differences in 
any color channel are considered  evidence of a foreground pixel here. Finally, we invert Imask  using 
cv::operator-() , because foreground should be the values out of range, not in range. The mask 
image is the output result. 
By way of putting it all together, we can define a function main() , which reads in a video and builds a

By way of putting it all together, we can define a function main() , which reads in a video and builds a 
background model. For our example, we run the video in a training mode  until the user hits the space bar, 
after which the video runs in a mode in which any foreground objects detected are highlighted in red: 
    << "Train a background model on incoming video, then run the model\n"  
    << "./ch9_ex9_2 avi_file\n"    
    << endl; 
} 
 
  cv::namedWindow( "Example9_2", cv::WINDOW_AUTOSIZE ); 
 
  if((argc < 2)|| !cap.open(argv[1])) {

    << endl; 
} 
 
  if((argc < 2)|| !cap.open(argv[1])) { 
   help(); 
    return -1; 
  } 
 
  // FIRST PROCESSING LOOP (TRAINING): 
  // 
  while(1) { 
   cap >> image;  
    if( !image.data ) exit(0); 
   
    accumulateBackground( image ); 
 
    cv::imshow( "Example9_2", rawImage ); 
    if( cv::waitKey(7) == 0x20 ) break; 
  } 
 
  // We have all of our data, so create the models 
  // 
  createModelsfromStats(); 
   
  // SECOND PROCESSING LOOP (TESTING): 
  // 
  cv::Mat mask; 
  while(1) {

cerr << "Couldn't open video file" << endl; 
   help(); 
    return -1; 
  } 
 
  // 
  while(1) { 
   
  } 
 
  // 
   
  // 
  while(1) { 
                                                           
6 In this circumstance, you could have used the bitwise OR operator as well, because the images being OR’ed are 
unsigned character images  and only the values 0x00  and 0xff  are relevant. In general, however, the cv::max()  
operation is a good way to get a “fuzzy” OR wh ich responds sensibly to a range of values.

cap >> image;  
   
    backgroundDiff( image, mask ); 
   
    // A simple visualization is to write to the red channel 
    // 
    cv::split( image, Igray ); 
    Igray[2] = cv::max( mask, Igray[2] ); 
    cv::merge( Igray, image ); 
 
    cv::imshow( "Example9_2", image ); 
  } 
 
  exit(0); 
} 
We’ve just seen a simple method of learning background scenes and se gmenting foreground objects. It will 
work well only with scenes that do not contain movi ng background components (it would fail with a waving 
curtain or waving trees). It also assumes that the lighting remains fairly constant (as in indoor static 
scenes). You can look ahead to X
  
Figure 9-6 X to check the performance of this averaging method. 
12BAccumulating Means, Variances, and Covariances 
The averaging background method just described made use the accumulation operator,

The averaging background method just described made use the accumulation operator, 
cv::Mat::operator+=()  to do what was essentially the simplest possible thing: to sum up a bunch of 
data that we could then normalize into an average. The average is a convenient statistical quantity for a lot 
of reasons, of course, but one often overlooked advantag e of the average is the fact that it can be computed 
incrementally in this way. F7
F This means that we can do processing on line  without needing to accumulate all 
of the data before analyzing. We will now consider a slightly more sophisticated model, which can also be 
computed on line in this way.

computed on line in this way. 
Our next model will represent the intensity (or color) variation within a pixel by computing a Gaussian 
model  for that variation. A one-dimensional Gaussian mode l is characterized by a si ngle mean (or average) 
                                                           
7 For purists, our implementation above is not exactly a purely incremen tal computation, as we divide by the number of 
samples at the end. There does, howev er, exist a purely incremental method fo r updating the average when a new data

samples at the end. There does, howev er, exist a purely incremental method fo r updating the average when a new data 
point is introduced, but the “nearly increm ental” version used is s ubstantially more computati onally efficient. We will 
continue throughout the chapter to refer to methods as “inc remental” if they can be co mputed from purely cumulative 
functions of the data combined with factors associated with overall normalization.

and a single variance  (which tells us something about the expected spread of measured values about the 
mean).  In the case of a d-dimensional model (e.g., a three-color model), there will be a d-dimensional 
vector for the mean, and a 
 -element matrix that represents not only the individual variances of the d-
dimensions, but also the covariances, which repres ent correlations between each of the individual 
As promised, each of these quantities— the means, the variances, and the covariances—can be computed in 
an incremental manner. Given a stream of incoming images, we can define three functions that will 
accumulate the necessary data, and three functions th at will actually convert th ose accumulations into the 
model parameters. 
The code which follows assumes the ex istence of a few global variables: 
cv::Mat sum; 
cv::Mat sqsum; 
int     image_count = 0; 
23BComputing the Mean with cv::Mat::operator +=()

23BComputing the Mean with cv::Mat::operator +=()  
As we saw in our previous example, the best method to compute the pixel means is to add them all up using 
cv::Mat::operator+=()  and then divide by the total number of images to obtain the mean: 
void accumulateMean( 
  cv::Mat& I 
) { 
  if( sum.empty ) { 
    sum = cv::Mat::zeros( I.size(), CV::F32C(I.channels()) ); 
  } 
  I.convertTo( scratch, sum.type() ); 
  sum += scratch; 
  image_count++; 
} 
The function above, accumulateMean() , is then called on each inco ming image. Once all of the 
images that are going to be used for the backgro und model have been computed, you can call the next

  cv::Mat& I 
) { 
  } 
} 
images that are going to be used for the backgro und model have been computed, you can call the next 
function, computeMean()  to get a single “image” that contains the averages for ever y pixel across your 
entire input set: 
cv::Mat& computeMean(  
  cv::Mat& mean 
) { 
  mean = sum / image_count; 
} 
24BComputing the mean with cv::accumulate()  
OpenCV provides another function, which is essentially similar to just using the 
cv::Mat::operator+=()  operator, but with two important dis tinctions. The first is that it will 
automatically handle the cv::Mat::convertTo()  functionality (and thus remove the need for a

) { 
} 
automatically handle the cv::Mat::convertTo()  functionality (and thus remove the need for a 
scratch image), and the second is that it allows  the use of an image mask. This function is 
cv::accumulate() . The ability to use an image mask when computing a background model is a very 
useful thing, as one often has some other information that some part of the image should not be included in 
the background model. For example, one might be building a background model of a highway or other 
uniformly colored area, and be able to immediately dete rmine from color that some objects are not part of

uniformly colored area, and be able to immediately dete rmine from color that some objects are not part of 
the background. This sort of thing can be very helpful in a real-world situation in which there is little or no 
opportunity to get access to the scene in th e complete absence of  foreground objects. 
The accumulate function has the following prototype: 
void accumulate( 
 
  cv::InputArray       src,                 // Input, 1 or 3 channels, U8 or F32 
  cv::InputOutputArray dst,                 // Result image, F32 or F64

cv::InputArray       mask = cv::noArray() // Use src pixel if mask pixel != 0  
); 
Here the array dst is the array in which the accumulation is happening, and src is the new image which 
will be added. cv::accumulate()  admits an optional mask. If present, only the pixels in dst which 
correspond to nonzero elements in mask will be updated. 
With cv::accumulate() , the function accumulateMean()  above can be simplified to: 
  cv::Mat& I 
) { 
  } 
  cv::accumulate( I, sum ); 
} 
25BVariation: Computing the mean with cv::accumulateWeighted()  
Another alternative that is often useful is to use a running average . The running average is given by the 
following formula: 
 
For a constant value of α, running averages are not equivalent to the result of summing with 
cv::Mat::operator+=() , or cv::accumulate() . To see this, simply consider adding three

sum = cv::Mat::zeros( I.size(), CV::F32C(I.channels()) ); 
  } 
} 
 
cv::Mat::operator+=() , or cv::accumulate() . To see this, simply consider adding three 
numbers (2, 3, and 4) with α set to 0.5. If we were to accumulate them with cv::accumulate() , then 
the sum would be 9 and the average 3. If we were to accumulate them with 
cv::accumulateWeighted() , the first sum would give 
 , and then adding 
the third term would give 
 . The reason the second number  is larger is that the 
most recent contributions are given mo re weight than those from farther in the past. Such a running average

most recent contributions are given mo re weight than those from farther in the past. Such a running average 
is thus also called a tracker . The parameter α can be thought of as setting th e amount of time necessary for 
the influence of a previous frame to fade. 
To accumulate running averages across en tire images, we use the OpenCV function 
cv::accumulateWeighted() : 
void accumulateWeighted( 
  cv::InputOutputArray dst,                 // Result image, F32 or F64  
  double               alpha,               // Weight factor applied to src

  double               alpha,               // Weight factor applied to src 
  cv::InputArray       mask = cv::noArray() // Use src pixel if mask pixel != 0  
); 
Here the array dst is the array in which the accumulation is happening, and src is the new image that will 
be added. The value alpha is the weighting parameter. Like cv::accumulate() ,

cv::accumulateWeighted()  admits an optional mask. If present, only the pixels in dst that 
26BFinding the variance with the help of cv::accumulateSquare()  
We can also accumulate squared images, which will allo w us to compute quickly th e variance of individual 
pixels. You may recall from your last class in statistics that the variance of a finite population is defined by the formula: 
 
where 
  is the mean of x for all N samples. The problem with this formula is that it entails making one pass 
through the images to compute 
  and then a second pass to compute 
 . A little algebra should allow you 
to convince yourself that the following formula will work just as well: 
 
Using this form, we can accumulate both the pixel valu es and their squares in a single pass. Then, the 
variance of a single pixel is just the average of the sq uare minus the square of the average. With this in

where 
 
variance of a single pixel is just the average of the sq uare minus the square of the average. With this in 
mind, we can define an accumulati on function and a computa tion function as we did with the mean. As 
with the mean, one option would be to first do an elem ent by element squaring of  the incoming image, and 
then to accumulate that  with something like sqsum += I.mul(I) . This, however, has several 
disadvantages, the most significant of which is that I.mul(I)  does not do any kind of implicit type 
conversion (as we saw that += did not do either). As a result, elements of (for example) an 8-bit array,

conversion (as we saw that += did not do either). As a result, elements of (for example) an 8-bit array, 
when squared, will almost inevitably cause overflows. As with cv::accumulate() , however, OpenCV 
provides us with a convenient function that does what  we need all in a single convenient package; it is 
called cv::accumulateSquare() : 
void accumulateSquare( 
);

); 
With the help of cv::accumulateSquare() , we can write a function to  accumulate the information 
we need for our variance computation: 
void accumulateVariance( 
  cv::Mat& I 
) { 
    sum   = cv::Mat::zeros( I.size(), CV::F32C(I.channels()) ); 
    sqsum = cv::Mat::zeros( I.size(), CV::F32C(I.channels()) ); 
  } 
  cv::accumulateSquare( I, sqsum ); 
} 
The associated computatio n function would then be: 
// note that ‘variance’ is sigma^2 
//

  cv::Mat& I 
) { 
  } 
} 
// 
void computeVariance(

cv::Mat& variance 
) {   
  double one_by_N = 1.0 / image_count; 
  variance = one_by_N  * sqsum – (one_by_N * one_by_N) * sum.mul(sum); 
} 
27BFinding the Covariance with cv::accumulateWeighted()  
The variance of the individual channels in a mult ichannel image captures so me important information 
about how similar we expect  background pixels in future images to be to our observed average. This, 
however, is still a very simplistic model for both the background and our “expectations.”  One important additional concept which we could introduce is the concept of covariance. Covariance captures inter-
relations between the variations in individual channels. For example, our background might be an ocean 
scene in which we expect very little variation in the red channel, but quite a bit in the green and blue 
channels. If we follow our intuition that  the ocean is just one color really , and that the variation we see is

channels. If we follow our intuition that  the ocean is just one color really , and that the variation we see is 
primarily a result of lighting effects, we might conclude th at if there were a gain or loss of intensity in the 
green channel, there should be a corresponding  gain or loss of intensity in the blue channel. The corollary 
of this is that if there were a subs tantial gain in the blue channel with out an accompanying gain in the green 
channel, we might not want to consider this background. This intuition is captured by the concept of covariance. 
 
Figure 9-3: The same data set is visualized on the left and  the right. On the left (a) the (square root of the) 
variance of the data in the x and y dimensions is show n, and the resulting model for the data is visualized.

variance of the data in the x and y dimensions is show n, and the resulting model for the data is visualized. 
On the right (b) the covariance of the data is captured  in the visualized model. The model has become an 
ellipsoid which is narrower in one dimension and wide r in the other than the more simplistic model on the 
left. 
In XFigure 9-3 X, we visualize what might be the blue and green channel data for a particular pixel in our 
ocean background exam ple. On the left, the variance only has been  computed. On the right, the covariance 
between the two channels has also been computed, and the resulting model is able to fit the data much more 
tightly.

left. 
tightly. 
Mathematically, the covariance between any two different observables is given by the formula: 
 
As you can see, the covariance between any observable x and itself: 
  is equal to the variance of 
that same observable
 . In a d-dimensional space (such as the RGB values for a pixel, for which d=3), it

is convenient to talk about the covariance matrix
 , whose components include all of the covariances 
between the variables as well as the variances of th e variables individually. As can be seen from the 
formula above, the covariance  matrix is symmetric, i.e.,
 . 
In a much earlier chapter, we encountered a function which could be used when dealing with individual 
vectors of data (as opposed to whole arrays of individual vectors). That function was 
cv::calcCovarMatrix() , which will allow us to provide N vectors of dimension d and it will spit out 
the d-by-d covariance matrix. Our problem now, however, is that we would like to compute such a matrix 
for every point in an array (or at least, in the case of a 3-dimensional RGB image, we would like to 
compute the 6 unique entries in that matrix). 
In practice, the best way to do this is simply to co mpute the variances using the code we already developed, 
and to compute the three new object s (the off-diagonal elements of
 ) separately. Looking at the

 ) separately. Looking at the 
formula for the covariance, we see that cv::accumulateSquare()  will not quite work here, as we 
need to accumulate the 
  terms (i.e., the product of two differ ent channel values from a particular 
pixel in each image). 
The function which does this for us in OpenCV is cv::accumulateProduct() . 
void accumulateProduct( 
  cv::InputArray       src1,                // Input, 1 or 3 channels, U8 or F32 
  cv::InputArray       src2,                // Input, 1 or 3 channels, U8 or F32 

  cv::InputOutputArray dst,                 // Result image, F32 or F64 
); 
This function works exactly like cv::accumulateSquare() , except that rather  than squaring the 
individual elements of src, it multiplies the corresponding elements of src1  and src2 . What it does not 
do (unfortunately) is allow us to pluck individual ch annels out of those incoming arrays. In the case of 
multichannel arrays in src1  and src2 , the computed result is done on a per-channel basis.

); 
multichannel arrays in src1  and src2 , the computed result is done on a per-channel basis.  
For our current need to compute the of f-diagonal elements of a covariance model, this is not really what we 
want. What we want are different channels of the same  image. To do this, we will have to split our 
incoming image apart using cv::split() . 
vector<cv::Mat> planes(3);  
vector<cv::Mat> sums(3);  
vector<cv::Mat> xysums(6); 
 
int image_count = 0; 
 
void accumulateCovariance( 
  cv::Mat& I 
) { 
  int i, j, n; 
    for( i=0; i<3; i++ ) {    // the r, g, and b sums

  cv::Mat& I 
) { 
  int i, j, n; 
    for( i=0; i<3; i++ ) {    // the r, g, and b sums 
      sums[i]   = cv::Mat::zeros( I.size(), CV::F32C1 ); 
    } 
    for( n=0; n<6; n++ ) {    // the rr, rg, rb, gg, gb, and bb elements 
      xysums[n] = cv::Mat::zeros( I.size(), CV::F32C1 ) ); 
    } 
  }

cv::split( I, rgb ); 
  for( i=0; i<3; i++ ) { 
    cv::accumulate( rgb[i], sums[i] );  
  } 
  n = 0; 
  for( i=0; i<3; i++ ) {      // "row" of Sigma 
    for( j=i; j<3; j++ ) {    // "column" of Sigma 
      n++; 
      cv::accumulateProduct( rgb[i], rgb[j], xysums[n] ); 
    } 
  } 
} 
The corresponding compute function is also just a slight extension of the compute function for the 
variances we saw earlier. 
// 
void computeVariance( 
  cv::Mat& covariance      // a six-channel array, channels are the 
                           // rr, rg, rb, gg, gb, and bb elements of Sigma_xy 
) {   
 
  // reuse the xysum arrays as storage for individual entries 
  // 
  int n = 0; 
  for( int i=0; i<3; i++ ) {     // "row" of Sigma 
    for( int j=i; j<3; j++ ) {   // "column" of Sigma 
      n++; 
      xysums[n] = one_by_N  * xysums[n]  
        – (one_by_N * one_by_N) * sums[i].mul(sums[j]); 
    } 
  }

// rr, rg, rb, gg, gb, and bb elements of Sigma_xy 
) {   
 
  // 
  int n = 0; 
      n++; 
    } 
  } 
   
  // reassemble the six individual elements into a six-channel array 
  //  
  cv::merge( xysums, covariance ); 
} 
28BA Brief Note on Model Testing and cv::Mahalanobis()  
In this section, we introduced some slightly more complicated models, but did not discuss how to test if a 
particular pixel in a new image is in the predicted domain of variation for the background model. In the case of the variance-only model (Gaussian models on a ll channels with an implicit assumption of statistical

} 
  } 
   
  //  
} 
particular pixel in a new image is in the predicted domain of variation for the background model. In the case of the variance-only model (Gaussian models on a ll channels with an implicit assumption of statistical 
independence between the channels) the problem is made  more complicated by the fact that the variances 
for the individual dimensions  will not necessarily be equal. In this  case, however, it is common to compute 
what is called a z-score for each dimension separately (the z-score is the distance from the mean divided by 
the standard deviation:
). The z-score tells us something about the probability of the individual

). The z-score tells us something about the probability of the individual 
pixel originating from the distribution in question. The z-scores for multiple dimensions are then 
summarized as the square root  of sum of squares (e.g., 
 ). 
In the case of the full covariance matrix, the analog of the z- score is called the Mahalanobis distance . This 
distance is essentially the distance from the mean to the point in question measured in constant-probability 
contours such as that shown in XFigure 9-3 X. Looking back at XFigure 9-3 X, a point up and to the left of the 
mean in the model (a) will appear to have a low Mahalanobis distance by that model. The same point

 ). 
mean in the model (a) will appear to have a low Mahalanobis distance by that model. The same point 
would have a much higher Mahalanobis distance by the model in (b). It is worth noting that the z-score 
formula for the simplified model in the previous paragraph is precisely the Mahalanobis distance under the model in 
XFigure 9-3 X (a), as one would expect.

OpenCV provides a function for computing Mahalanobis distances: 
double cv::Mahalanobis(                     // Return distance as F64 
  cv::InputArray vec1,                      // First vector (1-dimensional, length n) 
  cv::InputArray vec2,                      // Second vector (1-dimensional, length n) 
  cv::InputArray icovar                     // Inverse covariance matrix, n-by-n 
); 
The cv::Mahalanobis()  function expects vector objects for vec1  and vec2  of dimension d and a d-
by-d matrix for the inverse  covariance icovar . (The inverse covariance is used because inverting this 
matrix is costly, and in most cases you have many  vectors you would like to  compare with the same 
covariance—so the assumption is that you will invert it once and pass the inverse covariance to 
cv::Mahalanobis()  many times for each such inversion.) 
In our context of background subtraction, this is not entirely convenient, as 
cv::Mahalanobis()  wants to be called on a per-element basis. Unfortunately, there

cv::Mahalanobis()  wants to be called on a per-element basis. Unfortunately, there 
is no array-sized version of this capability in  OpenCV. As a result, you will have to loop 
through each pixel, creat e the covariance matrix from the i ndividual elements, invert that 
matrix, and store the inverse somewhere. Th en, when you want to make a comparison, 
you will need to loop through the pixels in your image, retrieve the inverse covariance 
you need, and call cv::Mahalanobis()  for each pixel. 
4BA More Advanced Background Subtraction Method 
Many background scenes contain co mplicated moving objects such as trees waving in the wind, fans

Many background scenes contain co mplicated moving objects such as trees waving in the wind, fans 
turning, curtains fluttering, and so on. Often, such scenes also contain varying lighting, such as clouds passing by or doors and windows letting in different light. 
A nice method to deal with this would be to fit a time-series model to each pixel or group of pixels. This 
kind of model deals with the temporal fluctuations well, but its disadvantage is the need for a great deal of memory [Toyama99]. If we use 2 seconds of previous input at 30 Hz, this means we need 60 samples for

kind of model deals with the temporal fluctuations well, but its disadvantage is the need for a great deal of memory [Toyama99]. If we use 2 seconds of previous input at 30 Hz, this means we need 60 samples for 
each pixel. The resulting model for each pixel would then  encode what it had lear ned in the form of 60 
different adapted weights . Often we’d need to gather background statistics for much longer than 2 seconds, 
which means that such methods are typically impractical on present-day hardware. 
To get fairly close to the performance of adaptive filtering, we take inspiration from the techniques of video 
compression and attempt to form a codebook
F8
F to represent significant states in the background. F9
F The

F8
F The 
simplest way to do this would be to compare a new value observed for a pixel with prior observed values. If 
the value is close to a prior value, then it is modeled as a perturbation on that color. If it is not close, then it 
can seed a new group of colors to be associated with that pixel. The result could be envisioned as a bunch 
                                                           
8 The method OpenCV implements is de rived from Kim, Chalidabhongse, Harw ood, and Davis [Kim05], but rather

F The 
                                                           
8 The method OpenCV implements is de rived from Kim, Chalidabhongse, Harw ood, and Davis [Kim05], but rather 
than learning oriented cylinders in RGB space, for sp eed, the authors use axis-aligned boxes in YUV space. Fast 
methods for cleaning up the resulting background  image can be found in Martins [Martins99]. 
9 There is a large literature for bac kground modeling and segmentation. OpenCV’s  implementation is intended to be 
fast and robust enough that you can use it to collect foreground objects mainly for the purposes  of collecting data sets

fast and robust enough that you can use it to collect foreground objects mainly for the purposes  of collecting data sets 
to train classifiers on. Recent work in  background subtraction allo ws arbitrary camera motion [Farin04; Colombari07] 
and dynamic background models using the mean-shift algorithm [Liu07].

of blobs floating in RGB space, each blob represen ting a separate volume considered likely to be 
background. 
In practice, the choice of RGB is not particularly op timal. It is almost always better to use a color space 
whose axis is aligned with brightness, such as the YUV color space. (YUV is the most common choice, but 
spaces such as HSV, where V is essentially brightness , would work as well.) The reason for this is that, 
empirically, most of the natural variation in the background tends to be along the brightness axis, not the 
color axis. 
The next detail is how to model these “blobs.” We ha ve essentially the same choices as before with our 
simpler model. We could, for exampl e, choose to model the blobs as Ga ussian clusters with a mean and a 
covariance. It turns out that the simplest case, in which the “blobs” are simply boxes with a learned extent in each of the three axes of our color space, works out  quite well. It is the simplest in terms of memory

covariance. It turns out that the simplest case, in which the “blobs” are simply boxes with a learned extent in each of the three axes of our color space, works out  quite well. It is the simplest in terms of memory 
required and in terms of the computational cost of determining whether a newly observed pixel is inside 
any of the learned boxes. 
Let’s explain what a codebook is  by using a simple example 
( X

Figure 9-4 X). A codebook is made up of boxes that grow to cover the common values seen over time. The 
upper panel of X
  
Figure 9-4 X shows a waveform over time; you could think of this as the brightness of an individual pixel. In 
the lower panel, boxes form to cover a new value and then slowly grow to cover nearby values. If a value is 
too far away, then a new box forms to cover it and likewise grows slowly toward new values. 
 
Figure 9-4: Codebooks are just “boxes” delimiting intens ity values: a box is form ed to cover a new value 
and slowly grows to cover nearby va lues; if values are too far away th en a new box is formed (see text)

In the case of our background model, we will learn a codebook of boxes that cover three dimensions: the 
three channels that make up our image at each pixel. 
X
  
Figure 9-5 X visualizes the (intensity dimens ion of the) codebooks for six different pixels learned from the data 
in XFigure 9-1 X. F10
F This codebook method can deal wi th pixels that change levels dram atically (e.g., pixels in a 
windblown tree, which might alternately be one of many colors of leaves, or  the blue sky beyond that tree). 
With this more precise method of modeling, we can detect a foreground object that  has values between the 
pixel values. Compare this with XFigure 9-2 X, where the averaging method cannot distinguish the hand value 
(shown as a dotted line) from the pixel fluctuations. Pe eking ahead to the next section, we see the better 
performance of the codebook method versu s the averaging method shown later in 
X
  
Figure 9-8 X.

X
  
Figure 9-8 X. 
 
10 In this case, we have chosen several pixels at random from the scan line to avoid excessive clutter. Of course, there is 
actually a codebook for every pixel.

Figure 9-5: Intensity portion of learned codebook entries for fluctuati ons of six chosen pixels (shown as 
vertical boxes): codebook boxes accomm odate pixels that take on multiple di screte values and so can better 
model discontinuous distributions; thus they can de tect a foreground hand (value at dotted line) whose 
average value is between the values  that background pixels can assume. In this case the codebooks are one-
dimensional and only represen t variations in intensity 
In the codebook method of learning a background model, each box is defined by two thresholds ( max and 
min) over each of the three-color axes. These box boundary thresholds will expand ( max getting larger, 
min getting smaller) if new background samples fall within a learning threshold ( learnHigh  and 
learnLow ) above max or below min, respectively. If new background samples fall outside of the box 
and its learning thresholds, then a new box will be started. In the background difference  mode, there are

and its learning thresholds, then a new box will be started. In the background difference  mode, there are 
acceptance thresholds maxMod  and minMod ; using these threshold values, we say that if a pixel is “close 
enough” to a max or a min box boundary then we count it as if it were inside the box. At runtime, the 
threshold for inclusion in a “box” can be set to a different value than was used in the construction of the 
boxes; often this threshold is simply set to zero in all three dimensions. 
A situation we will not cover is a pan-tilt camera surveying a large scene. When working 
with a large scene, it is necessary to stitch together learned models indexed by the pan 
and tilt angles. 
13BStructures

13BStructures 
It’s time to look at all of this in more detail, so let’s create an implementation of the codebook algorithm. 
First, we need our codebook stru cture, which will simply point to  a bunch of boxes in YUV space: 
class CodeBook : public vector<CodeElement> { 
public: 
  int t;                                              // count every access 
  CodeBook() { t=0; }                                 // Default is an empty book 
  CodeBook( int n ) : vector<CodeElement>(n) { t=0; } // Construct book of size n 
};

public: 
}; 
The codebook is derived from an STL vector of CodeElement  objects (see below). The variable t counts 
the number of points we’ve accumulated since the start or the last clear operation. Here’s how the actual 
codebook elements are described: 
#define CHANNELS 3 
class CodeElement { 
public: 
  uchar learnHigh[CHANNELS];   // High side threshold for learning 
  uchar learnLow[CHANNELS];    // Low side threshold for learning 
  uchar max[CHANNELS];         // High side of box boundary 
  uchar min[CHANNELS];         // Low side of box boundary 
  int   t_last_update;         // Allow us to kill stale entries

public: 
  int   t_last_update;         // Allow us to kill stale entries 
  int   stale;                 // max negative run (longest period of inactivity) 
 
  CodeElement& operator=( CodeElement& ce ) { 
    for( i=0; i<CHANNELS; i++ ) { 
      learnHigh[i] = ce.learnHigh[i]; 
      learnLow[i]  = ce.learnLow[i]; 
      min[i]       = ce.min[i]; 
      max[i]       = ce.max[i]; 
    } 
    t_last_update = ce.t_last_update; 
    stale         = ce.stale; 
  } 
  CodeElement( CodeElement& ce ) { *this = ce; } 
}; 
Each codebook entry consumes four bytes per channel plus two integers, or (4 * CHANNELS  + 4 + 4) bytes

CodeElement& operator=( CodeElement& ce ) { 
    } 
  } 
}; 
Each codebook entry consumes four bytes per channel plus two integers, or (4 * CHANNELS  + 4 + 4) bytes 
(20 bytes when we use three channels). We may set CHANNELS  to any positive number equal to or less

than the number of color channels in an image, but it is  usually set to either 1 (“Y,” or brightness only) or 3 
(YUV, HSV). In this structure, for each channel, max and min are the boundaries of  the codebook box. 
The parameters learnHigh[]  and learnLow[]  are the thresholds that trigger generation of a new 
code element. Specifically, a new code element will be generated if a new pixel is encountered whose 
values do not lie between min – learnLow  and max + learnHigh  in each of the channels. The 
time to last update ( t_last_update ) and stale  are used to enable the deletion of seldom-used 
codebook entries created during learning. Now we can proceed to investigate the functions that use this 
structure to learn dynamic backgrounds. 
14BLearning the background 
We will have one CodeBook  of CodeElements  for each pixel. We will need an array of such 
codebooks that is equal in length to the number of pixels in the images  we’ll be learning. For each pixel,

codebooks that is equal in length to the number of pixels in the images  we’ll be learning. For each pixel, 
updateCodebook()  is called for as many images as are suffici ent to capture the relevant changes in the 
background. Learning may be updated periodically throughout, and clearStaleEntries()  can be 
used to learn the background in the presence of (small numbers of) moving foreground objects. This is 
possible because the seldom-used “stale” entries induc ed by a moving foreground will be deleted. The 
interface to updateCodebook()  is as follows: 
// Updates the codebook entry with a new data point  
// NOTES:

// NOTES: 
//      cbBounds must be of length equal to numChannels 
// 
int updateCodebook(       // return CodeBook index 
  cv::Vec3b& p,           // incoming YUV pixel 
  CodeBook&  c,           // CodeBook for the pixel 
  unsigned*  cbBounds,    // Learning bounds for codebook (Rule of thumb: {10,10,10}) 
  int        numChannels   // Number of color channels we're learning 
) { 
  unsigned int high[3], low[3], n; 
  for( n=0; n<numChannels; n++ ) { 
    high[n] = p[n] + *(cbBounds+n);    if( high[n] > 255 ) high[n] = 255; 
    low[n]  = p[n] - *(cbBounds+n);    if( low[n]  < 0   ) low[n]  = 0; 
  }

cv::Vec3b& p,           // incoming YUV pixel 
) { 
  } 
  int matchChannel; 
  // SEE IF THIS FITS AN EXISTING CODEWORD 
  // 
  int i; 
  for( i=0; i<c.size(); i++ ) { 
    matchChannel = 0; 
    for( n=0; n<numChannels; n++ ) { 
      if( // Found an entry for this channel 
       ( c[i]->learnLow[n] <= p[n] ) && ( p[n] <= c[i]->learnHigh[n]) 
      ) 
      matchChannel++; 
    } 
    if( matchChannel == numChannels ) {     // If an entry was found 
      c[i]->t_last_update = c.t; 
 
      // adjust this codeword for the first channel 
      //

} 
  // 
  int i; 
      ) 
    } 
 
      // 
      for( n=0; n<numChannels; n++ ) { 
        if( c[i]->max[n] < p[n] )       c[i]->max[n] = p[n]; 
        else if( c[i]->min[n] > p[n] )  c[i]->min[n] = p[n]; 
      } 
      break; 
    } 
  } 
. . .continued below

This function grows or adds a codebook entry when the pixel p falls outside the existing codebook boxes. 
Boxes grow when the pixel is within cbBounds  of an existing box. If a pixel is outside the cbBounds  
distance from a box, a new codebook box  is created. The routine first sets high  and low levels to be used 
later. It then goes through each codebook entry to check whether the pixel value p is inside the learning 
bounds of the codebook “box.” If the pixel is w ithin the learning bounds for all channels, then the 
appropriate max or min level is adjusted to include this pixel and the time of last update is set to the 
current tim ed count c.t. Next, the updateCodebook()  routine keeps statistics on how often each 
codebook entry is hit: 
. . . continued from above  
 
  // OVERHEAD TO TRACK POTENTIAL STALE ENTRIES 
  // 
  for( int s=0; s<c.size(); s++ ) { 
    // Track which codebook entries are going stale: 
    // 
    int negRun = c.t - c[s]->t_last_update;

  // 
    // 
    int negRun = c.t - c[s]->t_last_update; 
    if( c[s]->stale < negRun ) c[s]->stale = negRun; 
  } 
 
. . .continued below  
Here, the variable stale  contains the largest negative runtime  (i.e., the longest span of time during which 
that code was not accessed by the data ). Tracking stale entries allows us to delete codebooks that were 
formed from noise or moving foreground objects and hence tend to become stale over time. In the next 
stage of learning the background, updateCodebook()  adds a new codebook if needed: 
 
  // ENTER A NEW CODEWORD IF NEEDED 
  //

} 
 
  // 
  if( i == c.size() ) {          // if no existing codeword found, make one 
    CodeElement ce; 
      ce->learnHigh[n] = high[n]; 
      ce->learnLow[n]  = low[n]; 
      ce->max[n]       = p[n]; 
      ce->min[n]       = p[n]; 
    } 
    ce->t_last_update = c.t; 
    ce->stale = 0; 
    c.push_back( ce ); 
  } 
Finally, updateCodebook()  slowly adjusts (by adding 1) the learnHigh  and learnLow  learning

// 
    } 
  } 
Finally, updateCodebook()  slowly adjusts (by adding 1) the learnHigh  and learnLow  learning 
boundaries if pixels were found outside of the box thresholds but still within the high  and low bounds: 
 
  // SLOWLY ADJUST LEARNING BOUNDS 
  // 
    if( c[i]->learnHigh[n] < high[n]) c[i]->learnHigh[n] += 1; 
    if( c[i]->learnLow[n]  > low[n] ) c[i]->learnLow[n]  -= 1; 
  } 
  return i; 
} 
The routine concludes by returning the index of the modified codebook. We’ve now seen how codebooks

  // 
  } 
  return i; 
} 
The routine concludes by returning the index of the modified codebook. We’ve now seen how codebooks 
are learned. In order to learn in the presence of moving foreground objects and to avoid learning codes for 
spurious noise, we need a way to delete entries that were accessed only rarely during learning.

15BLearning with moving foreground objects 
The following routine, clearStaleEntries() , allows us to learn the b ackground even if there are 
moving foreground objects: 
// During learning, after you've learned for some period of time, 
// periodically call this to clear out stale codebook entries 
//  
//   
int clearStaleEntries(     // return number of entries cleared 
  CodeBook &c              // Codebook to clean up 
){ 
  int staleThresh = c.t>>1; 
  int *keep = new int[c.size()]; 
  int keepCnt = 0; 
 
  // SEE WHICH CODEBOOK ENTRIES ARE TOO STALE 
  // 
  for( int i=0; i<c.size(); i++ ){ 
    if(c[i]->stale > staleThresh) 
      keep[i] = 0; // Mark for destruction 
    else 
    { 
      keep[i] = 1; // Mark to keep 
      keepCnt += 1; 
    } 
  } 
 
  // move the entries we want to keep to the front of the vector and then 
  // truncate to the correct length once all of the good stuff is saved. 
  // 
  int k = 0; 
  int numCleared = 0 
  for( int ii=0; ii<c.size(); ii++ ) {

int keepCnt = 0; 
 
  // 
    else 
    { 
    } 
  } 
 
  // 
  int k = 0; 
  for( int ii=0; ii<c.size(); ii++ ) { 
    if( keep[ii] ) { 
      c[k] = c[ii];  
      // We have to refresh these entries for next clearStale 
      cc[k]->t_last_update = 0; 
      k++; 
    } else { 
      numCleared++; 
    } 
  } 
  c.resize( keepCnt ); 
  delete[] keep; 
 
  return numCleared; 
} 
The routine begins by defining the parameter staleThresh , which is hardcoded (by a rule of thumb) to 
be half the total running time count, c.t. This means that, during backgr ound learning, if codebook entry

if( keep[ii] ) { 
      k++; 
    } else { 
    } 
  } 
 
} 
be half the total running time count, c.t. This means that, during backgr ound learning, if codebook entry 
i is not accessed for a period of time equal to half the total learning time, then i is marked for deletion 
(keep[i] = 0 ). The vector keep[]  is allocated so that we can mark each codebook entry; hence it is 
c.size()  long. The variable keepCnt  counts how many entries we will keep. After recording which 
codebook entries to keep, we go through the entries and mo ve the ones we want to th e front of the vector in 
the codebook. Finally, we resize that vector so that all of the stuff hanging off of the end is chopped off.

16BBackground differencing: Finding foreground objects 
We’ve seen how to create a backgro und codebook model and how to clear it of seldom-used entries. Next 
we turn to background_diff() , where we use the learned model to segment foreground pixels from 
the previously learned background: 
// Given a pixel and a codebook, determine if the pixel is 
// covered by the codebook 
//  
// NOTES: 
// minMod and maxMod must have length numChannels, 
// e.g. 3 channels => minMod[3], maxMod[3]. There is one min and 
//      one max threshold per channel. 
// 
uchar backgroundDiff(     // return 0 => background, 255 => foreground 
  cv::Vec3b& p,            // Pixel (YUV) 
  CodeBook&  c,            // Codebook 
  int        numChannels,  // Number of channels we are testing 
  int*       minMod,       // Add this (possibly negative) number onto max level 
                           //   when determining if new pixel is foreground 
  int*       maxMod        // Subtract this (possibly negative) number from min level

// 
  int*       maxMod        // Subtract this (possibly negative) number from min level 
) { 
 
  // 
  for( int i=0; i<c.size(); i++ ) { 
    for( int n=0; n<numChannels; n++ ) { 
      if(  
        (c[i]->min[n] - minMod[n] <= p[n] ) && (p[n] <= c[i]->max[n] + maxMod[n]) 
      ) { 
        matchChannel++; // Found an entry for this channel 
      } else { 
        break; 
      } 
    } 
    if(matchChannel == numChannels) {

//   when determining if new pixel is foreground 
) { 
 
  // 
      if(  
      ) { 
      } else { 
        break; 
      } 
    } 
    if(matchChannel == numChannels) { 
      break; // Found an entry that matched all channels 
    } 
  } 
  if( i >= c.size() ) return 0; 
  return 255; 
} 
The background differencing function has an inner loop similar to the learning routine 
updateCodebook , except here we look within the learned max and min bounds plus an offset threshold, 
maxMod  and minMod , of each codebook box. If the pixel is within the box plus maxMod  on the high side

} else { 
        break; 
      } 
    } 
    } 
  } 
} 
maxMod  and minMod , of each codebook box. If the pixel is within the box plus maxMod  on the high side 
or minus minMod  on the low side for each channel, then the matchChannel  count is incremented. When 
matchChannel  equals the number of channels, we’ve search ed each dimension and know that we have a 
match. If the pixel is not within a learned box, 2 55 is returned (a positive detection of foreground); 
otherwise, 0 is returned (the pixel is background). 
The three functions updateCodebook(), clearStaleEntries() , and backgroundDiff()

The three functions updateCodebook(), clearStaleEntries() , and backgroundDiff()  
constitute a codebook method of segmenti ng foreground from learned background. 
17BUsing the Codebook Background Model 
To use the codebook background segmentation technique, typically we take the following steps:

1. Learn a basic model of the background over a few seconds or minutes using updateCodebook() . 
2. Clean out stale entries with clearStaleEntries() . 
3. Adjust the thresholds minMod  and maxMod  to best segment the known foreground. 
4. Maintain a higher-level scene model (as discussed previously). 
5. Use the learned model to segment the foreground from the background via backgroundDiff() . 
6. Periodically update the learned background pixels. 
7. At a much slower frequency, periodically  clean out stale codebook entries with 
clearStaleEntries() . 
18BA Few More Thoughts on Codebook Models 
In general, the codebook method works quite well across a wide number of conditions, and it is relatively 
quick to train and to run. It doesn’t deal well with varying patterns of light—such as morning, noon, and

quick to train and to run. It doesn’t deal well with varying patterns of light—such as morning, noon, and 
evening sunshine—or with someone turning lights on or off indoors. This type of global variability can be taken into account by using severa l different codebook models, one for each condition, and then allowing 
the condition to control which model is active. 
5BConnected Components for Foreground Cleanup 
Before comparing the averaging method to the codebook method, we should pause to discuss ways to clean 
up the raw segmented image using connected-components analysis. This form of analysis is useful for

up the raw segmented image using connected-components analysis. This form of analysis is useful for 
noisy input mask images, and such noise is more the rule than the exception. The basic idea behind the 
method is to use the morphological operation open  to shrink areas of small noise to 0 followed by the 
morphological operation close  to rebuild the area of surviving components that was lost in opening. 
Thereafter, we find the “large enough”  contours of the surviving segments and can take statistics of all such 
segments. Finally, we retrieve either the largest contour or all contours of size above some threshold. In the routine that follows, we implement most of the f unctions that you would want for this connected 
component analysis:

component analysis: 
• Whether to approximate the surviving component contours by polygons or by convex hulls 
• Setting how large a component contour must be in order not to be deleted 
• Returning the bounding boxes of the surviving component contours 
• Returning the centers of the surviving component contours 
The connected components header that implements these operations is as follows: 
// This cleans up the foreground segmentation mask derived from calls 
// to backgroundDiff 
// 
void findConnectedComponents(

// 
void findConnectedComponents( 
  cv::Mat&   mask,               // Is a grayscale (8-bit depth) "raw" mask image that 
                                 // will be cleaned up 
  int        poly1_hull0 = 1,    // If set, approximate connected component by 
                                 //  (DEFAULT) polygon, or else convex hull (0) 
  float      perimScale  = 4,    // Len = image (width+height)/perimScale. If contour 
                                 //  len < this, delete that contour (DEFAULT: 4)

// 
                                 //  len < this, delete that contour (DEFAULT: 4) 
  vector<cv::Rect>&  bbs         // Reference to bounding box rectangle return vector 
  vector<cv::Point>& centers     // Reference to contour centers return vector  
 ); 
The function body is listed below. First, we do morphological opening and closing in order to clear out 
small pixel noise, after which we rebuild the eroded areas that survive the erosion of the opening operation. 
The routine takes two additional parameters, which here are hardcoded via #define . The defined values

vector<cv::Rect>&  bbs         // Reference to bounding box rectangle return vector 
 ); 
The routine takes two additional parameters, which here are hardcoded via #define . The defined values 
work well, and you are unlikely to want to change them. These additional parameters control how simple

the boundary of a foreground region should be (higher numbers are more simple) and how many iterations 
the morphological operators should perform; the higher the number of iterations, the more erosion takes 
place in opening before dilation in closing. F11
F More erosion eliminates larger regions of blotchy noise at the 
cost of eroding the boundaries of larger regions. Again, the parameters used in this sample code work well, but there’s no harm in experimenting with them if you like: 
// polygons will be simplified using DP algorithm with ‘epsilon’ a fixed  
// fraction of the polygons length. This number is that divisor. 
// 
#define DP_EPSILON_DENOMINATOR 20.0 
 
// How many iterations of erosion and/or dilation there should be 
// 
#define CVCLOSE_ITR 1 
We now discuss the connected-component algorithm itself. The first part of the routine performs the 
morphological open and closing operations: 
  cv::Mat&   mask, 
  int        poly1_hull0, 
  float      perimScale,

// 
 
// 
  float      perimScale, 
  vector<cv::Rect>&  bbs, 
  vector<cv::Point>& centers 
) { 
 
  // CLEAN UP RAW MASK 
  // 
  cv::morphologyEx(  
    mask, mask, cv::MOP_OPEN,  cv::Mat(), cv::Point(-1,-1), CVCLOSE_ITR  
  ); 
    mask, mask, cv::MOP_CLOSE, cv::Mat(), cv::Point(-1,-1), CVCLOSE_ITR  
  ); 
Now that the noise has been removed fr om the mask, we find all contours: 
  // FIND CONTOURS AROUND ONLY BIGGER REGIONS 
  // 
  vector< vector<cv::Point> > contours_all; // all contours found 
  vector< vector<cv::Point> > contours;     // just the ones we want to keep

// CLEAN UP RAW MASK 
  // 
  ); 
  ); 
  // 
  vector< vector<cv::Point> > contours;     // just the ones we want to keep 
  cv::findContours(   
    mask, 
    contours_all,  
    CV_RETR_EXTERNAL,  
    CV_CHAIN_APPROX_SIMPLE 
  ); 
Next, we toss out contours that are too small and approximate the rest with polygons or convex hulls: 
  for(  
    vector< vector<cv::Point> >::iterator c = contours_all.begin(); 
    c != contours.end(); 
    ++c  
  ) { 
 
    // length of this contour 
    // 
    int len = cv::arcLength( *c, true ); 
 
    // length threshold a fraction of image perimeter

cv::findContours(   
    mask, 
  ); 
  for(  
    ++c  
  ) { 
 
    // 
 
    // length threshold a fraction of image perimeter 
                                                           
11 Observe that the value CVCLOSE_ITR  is actually dependent on the resolution. For images of extremely high 
resolution, leaving this value set to 1 is  not likely to yield satisfactory results.

//    
    double q = (mask->height + mask->width) / DP_EPSILON_DENOMINATOR; 
 
    if( len >= q ) {        // If the contour is long enough to keep... 
      vector<cv::Point> c_new; 
      if( poly1_hull0 ) {   // If the caller wants results as reduced polygons... 
        cv::approxPolyDP( *c, c_new, len/20.0, true ); 
      } else {              // Convex Hull of the segmentation 
        Cv::convexHull( *c, c_new ); 
      } 
      contours.push_back(c_new ); 
    } 
  } 
In the preceding code, we use the Douglas-Peucker approximation algorithm to reduce polygons (if the user 
has not asked us to just return convex hulls). All this processing yields a new list of contours. Before 
drawing the contours back into the mask, we define some simple colors to draw: 
  // Just some convenience variables 
  const cv::Scalar CVX_WHITE   = cv::RGB(0xff,0xff,0xff); 
  const cv::Scalar CVX_BLACK   = cv::RGB(0x00,0x00,0x00);

} 
    } 
  } 
  const cv::Scalar CVX_BLACK   = cv::RGB(0x00,0x00,0x00); 
We use these definitions in the following code, where we  first analyze each of the contours separately, then 
zero out the mask and draw the whole set of clean contours back into the mask: 
 
  // CALC CENTER OF MASS AND/OR BOUNDING RECTANGLES 
  // 
  int idx = 0; 
  cv::Moments moments;  
  cv::Mat scratch = mask.clone(); 
  for(  
    vector< vector<cv::Point> >::iterator c = contours.begin();  
    c != contours.end;  
    c++, idx++  
  ) {

  // 
  int idx = 0; 
  for(  
    c++, idx++  
  ) { 
 
    cv::drawContours( scratch, contours, idx, CVX_WHITE, CV_FILLED ); 
     
    // Find the center of each contour 
    // 
    moments = cv::moments( scratch, true ); 
    cv::Point p; 
    p.x = (int)( moments.m10 / moments.m00 ); 
    p.y = (int)( moments.m01 / moments.m00 ); 
    centers.push_back(p); 
 
    bbs.push_back( cv::boundingRect(c) ); 
 
    Scratch.setTo( 0 ); 
  } 
 
  // PAINT THE FOUND REGIONS BACK INTO THE IMAGE 
  // 
  mask.setTo( 0 ); 
  cv::drawContours( mask, contours, -1, CVX_WHITE );  
}

// Find the center of each contour 
    // 
 
  } 
 
  // 
} 
That concludes a useful routine for creating clean masks out of noisy raw masks 
19BA Quick Test 
We start with an example to see how this really works in an actual video. Let’s stick with our video of the 
tree outside of the window. Recall ( XFigure 9-1 X) that at some point in time, a hand passes through the scene.

One might expect that we could find this hand relativ ely easily with a technique such as frame differencing 
(discussed previously in its own section). The basic id ea of frame differencing was to subtract the current 
frame from a “lagged” frame and then threshold the difference. 
Sequential frames in a video tend to be quite similar. Hence one might expect that, if we take a simple 
difference of the original frame and the lagged fra me, we’ll not see too much unless there is some 
foreground object moving through the scene. F12
F But what does “not see too much” mean in this context? 
Really, it means “just noise.” Thus, in practice the problem is sorting out that noise from the signal when a 
foreground object does come along. 
To understand this noise a little better, first consider a pair of frames from the vi deo in which there is no 
foreground object—just the background and the resulting noise. 
X

X
  
Figure 9-6 X shows a typical frame from such a video (upper-left) and the previous frame (upper-right). The 
figure also shows the results of frame differencing with a threshold value of 15 (lower-left). You can see 
substantial noise from the moving leaves of the tree. Nevertheless, the method of connected components is able to clean up this scattered noise quite well
F13
F (lower-right). This is not su rprising, because there is no

X
  
F13
F (lower-right). This is not su rprising, because there is no 
                                                           
12 In the context of frame differencing, an object is identifie d as “foreground” mainly by its velocity. This is reasonable 
in scenes that are generally static or in which foreground objects are expected to be much  closer to the camera than 
background objects (and thus appear to move faster by virtue of the projective geometry of cameras). 
13 The size threshold for the connected components has been tuned to give zero response in these empty

13 The size threshold for the connected components has been tuned to give zero response in these empty 
frames. The real question then is whether or not th e foreground object of interest (the hand) survives

reason to expect much spatial correlation in this noise  and so its signal is characterized by a large number 
of very small regions. 
 
Figure 9-6: Frame differencing: a tree is waving in the background in the current (upper-left) and previous 
(upper-right) frame images; the difference image (lower-l eft) is completely cleaned up (lower-right) by the 
connected-components method 
                                                                                                                                                                             
pruning at this size th reshold. We will see 
(
  
Figure 9-7) that it does so nicely.

Now consider the situation in which a foreground obj ect (our ubiquitous hand) passes through the view of 
the imager. X
  
Figure 9-7 X shows two frames that are similar to those in 
X
  
Figure 9-6 X except that now there is a hand moving across fro m left to right. As be fore, the current frame 
(upper-left) and the previous frame (upper-right) are shown along with the response to frame differencing 
(lower-left) and the fairly good results of the connected-component cleanup (lower-right).

Figure 9-7: Frame difference method of detecting a hand, which is moving left to right as the foreground 
object (upper two panels); the difference image (lower-l eft) shows the “hole” (where the hand used to be) 
toward the left and its leading edge toward the right, and the connected-com ponent image (lower-right) 
shows the cleaned-up difference 
We can also clearly see one of the deficiencies of  frame differencing: it cannot distinguish between the 
region from where the object moved (the “hole”) and where the object is now. Furthermore, in the overlap region, there is often a gap because “flesh minus flesh” is 0 (or at least below threshold). 
Thus, we see that using connected components for cleanup is a powerful technique for rejecting noise in 
background subtraction. As a bonus, we were also able to glimpse some of the strengths and weaknesses of frame differencing. 
6BComparing Two Background Methods

6BComparing Two Background Methods 
We have discussed two classes of background modeling techniques so far in this chapter: the average 
distance method (and its variants) and the codebook method. You might be wondering which method is better, or, at least, when you can ge t away with using the easy one. In these situations, it’s always best to 
just do a straight bake off
F14
F between the available methods. 
We will continue with the same tree video that we’ve be en using throughout the chapter. In addition to the

F14
We will continue with the same tree video that we’ve be en using throughout the chapter. In addition to the 
moving tree, this film has a lot of glare coming off a building to the right and off portions of the inside wall 
on the left. It is a fairly challenging background to model. 
                                                           
14 For the uninitiated, “bake off” is actually a bona fide term  used to describe any challenge or comparison of multiple 
algorithms on a predetermined data set.

In X
  
XFigure 9-8 X, we compare the average difference method at  top against the codebook method at bottom; on 
the left are the raw foreground images and on the right are the cleaned-up connect ed components. You can 
see that the average difference method leaves be hind a sloppier mask and breaks the hand into two 
components. This is not so surprising; in XFigure 9-2 X, we saw that using the average difference from the 
mean as a background model often incl uded pixel values associ ated with the hand va lue (shown as a dotted 
line in that figure). Compare this with 
X
  
Figure 9-5 X, where codebooks can more accurately model the fluctuations of the leaves and branches and so 
more precisely identify foreground hand pixels  (dotted line) from background pixels. 
X

Figure 9-8 X confirms not only that the background mode l yields less noise but also that connected 
components can generate a fairly accurate object outline. 
 
Figure 9-8: With the averaging method (top row), the connected-components cleanu p knocks out the fingers 
(upper-right); the codebook method (bottom row) does much better at segmentation and creates a clean 
connected-component mask (lower-right) 
7BOpenCV Background Subtraction Encapsulation 
Thus far, we have looked in detail at how you might implement your own basic background subtraction 
algorithms. The advantage of that approach is that  it is much more clear what is going on and how 
everything is working. The disadvantage is that as time progresses, newer and better methods are developed 
which, though rooted in the same fundamental ideas, become sufficiently complicated so that you would 
like to be able to regard them as “black boxes” and just use them without getting too deep in the gory details.

like to be able to regard them as “black boxes” and just use them without getting too deep in the gory details. 
To this end, OpenCV now provides a genericized cla ss-based interface to background subtraction. At this 
time, there are two implementations which use this interf ace, but as time progresses, there are expected to 
be more. In this section we will first look at the interface in its generic form, then investigate the two 
implementations which are available. Both implementations are based on a mixture of gaussians  (MOG) 
approach, which essentially takes the statistical modeling concept we introduced for our simplest 
background modeling scheme (see “

XAccumulating Means, Variances, and Covariances X”) and marries it 
with the multimodal capability of the codebook scheme (the one developed in “ XA More Advanced 
Background Subtraction Method X”). Both of these MOG methods are 21st century algorithms suitable for 
many practical day-to-day situations. 
20BThe cv::BackgroundSubtractor  Base Class 
The cv::BackgroundSubtractor  (abstract) base class F15
F specifies only the minimal number of 
necessary methods. It has the following definition: 
class cv::BackgroundSubtractor : public Algorithm {

class cv::BackgroundSubtractor : public Algorithm { 
 
15 Actually, as shown below, this base class is not literally abstract (i.e., it does not cont ain any pure virtual functions). 
However, it is always used in the library  as if it were abstract; meaning that t hough the compiler will let you instantiate 
an instance of cv::BackgroundSubtractor  there is no purpose in, nor meaning to, doing so. We considered 
coining the phrase “relatively abstract” for the circumstance, but later thought better of it.

public: 
  virtual ~BackgroundSubtractor(); 
  virtual void apply( 
    cv::InputArray  image,  
    cv::OutputArray fgmask,  
    double          learningRate = 0 
  ); 
  virtual void getBackgroundImage( 
    cv::OutputArray backgroundImage 
  ) const; 
}; 
As you can see, after the destructor, there are only two methods defined F16
F. The first is the apply()  
function, which in this context is used to both ingest a new image and to produce the calculated foreground 
mask for that image. The second function produces an image representation of the background. This image 
is primarily for visualization and debugging; after all there is much more inform ation associated with any 
single pixel in the background than just a color. As a result the image produced by 
getBackgroundImage()  can only be a partial presentation of the information that exists in the 
One thing that might seem to be  a glaring omission is the absenc e of a method that accumulates

One thing that might seem to be  a glaring omission is the absenc e of a method that accumulates 
background images for training. The reason for this is  that there came to be (relative) consensus in the 
academic literature that any backgr ound subtraction algorithm that was not essentially continuously 
training was an undesirable algorithm. The reasons for this are many, with the most obvious of which being 
the effect of gradual illumination change on a scene (e.g ., as the sun rises and sets outside the window). The

the effect of gradual illumination change on a scene (e.g ., as the sun rises and sets outside the window). The 
more subtle issues arise from the f act that in many practical scenes th ere is not opportunity to expose the 
algorithm to a prolonged period in which no foreground objects are present. Similarly, in many cases, 
things that seem to be background for an extended period (such as a parked car) might finally move, 
leaving a permanent foregro und “hole” at the location of their absenc e. For these reasons, essentially all 
modern background subtraction algorithms do not distinguish between training and running modes; rather,

modern background subtraction algorithms do not distinguish between training and running modes; rather, 
they continuously train and build models in which those things that are seen rarely (and can thus be 
understood to be foreground) are removed and those thin gs that are seen a major ity of the time (which are 
understood to be the background) are retained. 
21BKadewTraKuPong and Bowden Method 
The first of the available algorithms brings us seve ral new capabilities which address real-world challenges 
in background subtraction. These are: a multimodal model, continuous online training, two separate

in background subtraction. These are: a multimodal model, continuous online training, two separate 
(automatic) training modes that improve initialization performance, and explicit detection and rejection of shadows [KaewTraKulPong2001]. All of this is largely invisible to you, the user. Not unexpectedly, 
however, this algorithm does have some parameters which you may want to tune to your particular 
application. They are the history , the number of Gaussian mixtures , the background ratio , and the noise 
strength .
F17

strength .
F17 
                                                           
16 You will also note that there is no constructor at all (other  than the implied default constructor). We will see that the 
construction of the subclasses of cv::BackgroundSubtractor  will be the things we actually want to create, so 
they provide their own construction scheme. 
17 If you find yourself looking up the citation give n for this algorithm, the first three parameters— history , number of 
Gaussian mixtures , and background ratio —are referred to in the paper as: L, K, and T respectively. The last, and noise

Gaussian mixtures , and background ratio —are referred to in the paper as: L, K, and T respectively. The last, and noise 
strength , can be thought of as th e initialization value of 
  for a newly created component.

The first of these, the history , is the point at which the algorithm will switch out of the initialization mode 
and into its nominal run mode. The default value for this parameter is 200 frames. The number of Gaussian 
mixtures  is the number of Gaussian components to the overall mixture model that is used to approximate 
the background in any given pixel. The default value for this parameter is 5.  
Given some number of Gaussian components to the model, each will have a weight . This weight indicates 
the portion of the observed values of  a pixel that are explained by that particular component of the model. 
They are not all necessarily “background,” some are likely to be foreground objects which have passed by 
at one point or another. Ordering the components by weight, the ones which are included as true 
background are the first b of them, where b is the minimum number required to “explain” some fixed

background are the first b of them, where b is the minimum number required to “explain” some fixed 
percentage of the total model. This percentage is  called the background ratio, and its default value is 0.7 (or 
70% ). Thus, by way of example, if there are five components, with weights 0.40, 0.25, 0.20, 0.10, and 
0.05, then b would be three, because it required the first three 
  to exceed the required 
background ratio of 0.70.  
The last parameter is the noise strength . This parameter sets the uncertainty assigned to a new Gaussian 
component when it is created. New components are created whenever new unexplained pixels appear,

component when it is created. New components are created whenever new unexplained pixels appear, 
either because not all components have been assigned yet, or because a new pixel value has been observed 
which is not explained by any existing component (in which case the least valuable existing component is recycled to make room for this new information). In pr actice, the effect of increas ing the noise strength is 
to allow the given number of Gaussian components to “explain” more. Of course, the tradeoff is that they 
will tend to explain perhaps even more than has been observed. The default value for the noise strength is 
15 (measured in units of 0‐255  pixel intensities).

29Bcv::createBackgroundSubtractorMOG()  and cv::BackgroundSubtractorMOG 
When we would like to construct an algorithm obj ect which actually implemen ts a specific form of 
background subtraction, we rely on a creator function to generate a cv::Ptr<>  smart pointer to an 
instance of the algorithm obj ect. The algorithm object cv::BackgroundSubtractorMOG   is a 
subclass of the cv::BackgroundSubtractor  base class. In the case of 
cv::BackgroundSubtractorMOG  that function is 
cv::createBackgroundSubtractorMOG() : 
cv::Ptr<cv::BackgroundSubtractorMOG> cv::createBackgroundSubtractorMOG( 
  int    history         = 200,      // Length of initialization history 
  int    nmixtures       = 5,        // Number of Gaussian components in mixture 
  double backgroundRatio = 0.7,      // Keep components which explain this fraction 
  double noiseSigma      = 0         // Start uncertainty for new components 
); 
Once you have your background subtractor object, you can then proceed to make use of its apply()

); 
Once you have your background subtractor object, you can then proceed to make use of its apply()  
method.  The default values used by cv::createBackgroundSubtractorMOG()  should serve for 
the majority of cases.  The last value is actually the one you are most likely to want to experiment with, the 
value of noiseSigma, in most cases, should be set to a larger value, 5, 10, or even 15.  
22BZivkovic Method 
This second background subtraction method is in many ways similar to the first, in that it also uses a 
Gaussian mixture model  to model the distribution of colors observed in any particular pixel. One

Gaussian mixture model  to model the distribution of colors observed in any particular pixel. One 
particularly notable distinction between the two algorithms is that the Zivkovic method does not use a fixed 
number of Gaussian components; rather, it adapts the number dynamically to give the best overall 
explanation of the observed distribution [Zivkovic04, Zivkovic06]. This has the downside that the more components there are, the more compute resources are consumed updating and comparing with the model. 
On the other hand, it has the upside that the model is capable of potentially much higher fidelity.

On the other hand, it has the upside that the model is capable of potentially much higher fidelity. 
This algorithm has some parameters in common with the KB method, but introduces many new parameters 
as well. Fortunately, only two of the parameters are especially important, while the others are ones which 
we can mostly leave at their default values. Th e two particularly critical parameters are the history  (also 
called the decay parameter ) and the variance threshold .  
The first of these, the history , sets the amount of time over which some “experience” of a pixel color will 
last. Essentially, it is the time it takes for the influence of that pixel to decay away  to nothing. The default

last. Essentially, it is the time it takes for the influence of that pixel to decay away  to nothing. The default 
value for this period is 500 frames. That value is approximately the time before a measurement is 
“forgotten.”  Internally  to the algorithm, however, it is slightly  more accurate to think of this as an 
exponential decay parameter  whose value is 
  (i.e., the influence of a measurement 
decays like 
 . 
The second parameter, the variance threshold  sets the confidence level with which a new pixel 
measurement must be within, relative to an existing Gaussian mixture component, to be considered part of

 . 
measurement must be within, relative to an existing Gaussian mixture component, to be considered part of 
that component. The units of the variance threshold are in squared-Mahalanobis distance. This means 
essentially that if you would include a pixel that is three sigma from the center of a component into that

component, then you would set the variance threshold to 
 . F18
F  The default value for this 
parameter is actually 
 . 
30Bcv::createBackgroundSubtractorMOG2()  and cv::BackgroundSubtractorMOG2 
Analogous to the previous cv::BackgroundSubtractorMOG  case, the Zivkovic method is 
implemented by the object: cv::BackgroundSubtractorMOG2 , which is another subclass of the 
cv::BackgroundSubtractor  base class. As before, these objects are generated by an associated 
creator function: cv::createBackgroundSubtractorMOG2() , which not only allocates the 
algorithm object, but also returns a smart pointer to the allocated instance. 
cv::Ptr<cv::BackgroundSubtractorMOG2> cv::createBackgroundSubtractorMOG2( 
  int   history          = 500,    // Length of history 
  float varThreshold     = 16,     // Threshold decides if new pixel is “explained” 
  bool  bShadowDetection = true    // true if MOG2 should try to detect shadows 
);

); 
The history and variance threshold parameters history  and varThreshold  are just as described 
above. The new parameter bShadowDetection  allows optional shadow detection and removal to be 
turned on. When operational, it functions much like the similar functionality in the KB algorithm but, as 
you would expect, slows the algorithm down slightly. 
 If you want to modify any of the values you set when you called 
cv::createBackgroundSubtractorMOG2() , there are getter/setter methods which can be used to

cv::createBackgroundSubtractorMOG2() , there are getter/setter methods which can be used to 
change not only these values, but a number of the more subtle para meters of the algorithm as well. 
int    cv::createBackgroundSubtractorMOG2::getHistory();                      // Get 
void   cv::createBackgroundSubtractorMOG2::setHistory( int val );             // Set 
 
int    cv::createBackgroundSubtractorMOG2::getNMixtures();                    // Get 
void   cv::createBackgroundSubtractorMOG2::setNMixtures( int val );           // Set

void   cv::createBackgroundSubtractorMOG2::setNMixtures( int val );           // Set 
 
double cv::createBackgroundSubtractorMOG2::getBackgroundRatio();              // Get 
void   cv::createBackgroundSubtractorMOG2::setBackgroundRatio( double val );  // Set 
 
double cv::createBackgroundSubtractorMOG2::getVarThresholdGen();              // Get 
void   cv::createBackgroundSubtractorMOG2::setVarThresholdGen( double val );  // Set 
 
double cv::createBackgroundSubtractorMOG2::getVarInt();                       // Get 
void   cv::createBackgroundSubtractorMOG2::setVarInt(double val );            // Set

void   cv::createBackgroundSubtractorMOG2::setVarInt(double val );            // Set 
 
double cv::createBackgroundSubtractorMOG2::getComplexityReductionThreshold(); // Get 
void   cv::createBackgroundSubtractorMOG2::setComplexityReductionThreshold(  
                                             double val  
                                           );                                 // Set 
 
bool   cv::createBackgroundSubtractorMOG2::getDetectShadows();                // Get 
void   cv::createBackgroundSubtractorMOG2::setDetectShadows( bool val );      // Set 
 
double cv::createBackgroundSubtractorMOG2::getShadowThreshold();              // Get

                                             double val  
                                           );                                 // Set 
 
double cv::createBackgroundSubtractorMOG2::getShadowThreshold();              // Get 
void   cv::createBackgroundSubtractorMOG2::setShadowThreshold( double val );  // Set  
 
18 Recall that the Mahalanobis distance is essentially a z- score (i.e., a measurement of how far you are from the center 
of a Gaussian distribution—measured in units of that distribution’s uncertainty) which takes into account the 
complexities of a distribution in an arbitrary numbe r of dimensions with arbitrary covariance matrix 
 .

 . 
 
You can also see why computing the squa red Mahalanobis distance is more na tural, which is why you provide the 
threshold as 
  rather than z.

int    cv::createBackgroundSubtractorMOG2::getShadowValue();                  // Get 
void   cv::createBackgroundSubtractorMOG2::setShadowValue( int val );         // Set 
The meaning of these functions is as follows: setNMixtures()  resets the length of the history that you 
assigned with the constructor. setNMixtures()  sets the maximum number of Gaussian components 
any pixel model can have (the default is 5). Increasing this improves model fidelity at the cost of runtime. 
setBackgroundRatio()  sets the background ratio, which has the same meaning as in the KB 
algorithm F19
F (default for this algorithm is 0.90).  
The function setVarThresholdGen()  controls when a new componen t of the muli-Gaussian model 
will be created.  If the squared Mahalanobis distance fr om the center of the nearest component of the model 
exceeds this threshold for generation, then a new com ponent will be added centered on the new pixel value. 
The default value for this parameter is

 . Similarly, the function setVarInit()  controls the 
variance threshold  (described earlier in this section), whic h is the initial variance assigned to a new 
Gaussian component of a model. Don’t forget that both the threshold for generation and the new model 
variance are squared distances, so typical values will be 9, 16, 25, etc. (not 3, 4, or 5). 
The setComplexityReductionThreshold()  function controls what Zivkovic et al. call the

The setComplexityReductionThreshold()  function controls what Zivkovic et al. call the 
complexity reduction prior . It is related to the number of samples needed to accept that a component 
actually exists. The default value for this parameter is 0.05. Probably the most important thing to know about this value is that if you set it to 0.00 , then the entire algorithm simplifies substantially
F20
F (both in 
terms of speed and result quality). 
The remaining functions set variables associated with how shadows are handled.  The

F20
F (both in 
The remaining functions set variables associated with how shadows are handled.  The 
setDetectShadows()  function simply allows you to turn on and off the shadow detection behavior of 
the algorithm (effectively overridi ng whatever value you gave to bShadowDetection  when you called 
the algorithm constructor).  If shadow detection is tu rned on, you can set the threshold that is used to 
determine if a pixel is a shadow using the setShadowThreshold()  function. The interpretation of the 
shadow threshold is that it is the relative brightness threshold for a pixel to be considered a shadow relative

shadow threshold is that it is the relative brightness threshold for a pixel to be considered a shadow relative 
to something which is already in the model (e.g., if the shadow threshold is 0.60, then any pixel which has 
the same color as an existing component and is between 0.60 and 1.0 times as bright is considered a 
                                                           
19 As a good rule of thumb, you can expect  that a pixel whose value is not desc ribed by the existing model and which 
stays approximately constant for a number of frames equal to  the history times the background ratio, will be updated in 
the model to become part of the background.

the model to become part of the background. 
20 For a more technical definition of “simplifies substantially,”  what really happens is that Zivkovic’s algorithm 
simplifies into something very similar to the algorithm of St auffer and Grimson. We do not discuss that algorithm here 
in detail, but it is cited in Zivkovic’s paper and was a re latively standard benchmark relative to which Zivkovic’s 
algorithm was an improvement.

shadow). The default value for this parameter is 0.50. Finally, the setShadowValue()  function is used 
if you want to change the numerical value assigned to shadow pixels in the foreground mask.  By default, 
background will be assigned the value of 0, foreground the value of 255, and shadow pixels the value of 
127.  You can change the value assigned to shadow pixels using setShadowValue()  to any value 
(except 0 or 255). 
8BSummary  
In this chapter, we looked at the specific problem of background subtraction.  This problem plays a major 
role in a vast array of practical computer vision applications, ranging from industrial automation, to 
security, to robotics.  Starting with the basic theory  of background subtraction, we developed two basic 
models of how such subtraction could be accomplished based on simple statistical methods.  From there we 
showed how connected component analysis could be us ed to increase the utility of  background subtraction

showed how connected component analysis could be us ed to increase the utility of  background subtraction 
results and compared the two basic methods we had developed. 
We concluded the chapter by looking at the more advanced background subtraction methods supplied by 
the OpenCV library as complete implementations.  These methods are similar in spirit to the simpler methods we developed in detail at the beginning of the chapter, but contain improvements which make 
them suitable for more challengi ng real-world applications. 
9BExercises 
1. Using cv::accumulateWeighted() , re-implement the averaging method of background

1. Using cv::accumulateWeighted() , re-implement the averaging method of background 
subtraction. In order to do so, learn the running average of the pixel values in the scene to find the 
mean and the running average of the absolute difference ( cv::absdiff() ) as a proxy for the 
standard deviation of the image. 
2. Shadows are often a problem in background subtraction because they can show up as a foreground 
object. Use the averaging or codebook method of background subtraction to learn the background. Have a person then walk in the foreground. Shadows will “emanate” from the bottom of the 
foreground object.

foreground object. 
a) Outdoors, shadows are darker and bluer than thei r surround; use this fact to eliminate them. 
b) Indoors, shadows are darker than their su rround; use this fact to eliminate them. 
3. The simple background models presented in this chapter are often quite sensitive to their threshold 
parameters. In Chapter 10, we’ll see how to track moti on, and this can be used as a reality check on the 
background model and its thresholds. You can also use it when a known person is doing a “calibration

background model and its thresholds. You can also use it when a known person is doing a “calibration 
walk” in front of the camera: find the moving object and adjust the parameters until the foreground 
object corresponds to the motion boundaries. We can also use distinct patterns on a calibration object 
itself (or on the background) for a reality check and tuning guide when we know that a portion of the 
background has been occluded. 
a) Modify the code to include an auto-calibration mode. Learn a background model and then put a

a) Modify the code to include an auto-calibration mode. Learn a background model and then put a 
brightly colored object in the scen e. Use color to find the colored object and then use that object to 
automatically set the thresholds in the background routine so that it segments the object. Note that 
you can leave this object in the scene for continuous tuning. 
b) Use your revised code to address the shadow-removal problem of exercise 
X2 X. 
4. Use background segmentation to segment a person with arms held out. Investigate the effects of the 
different parameters and defaults in the cv::findContours()  routine. Show your results for

X2 X. 
different parameters and defaults in the cv::findContours()  routine. Show your results for 
different settings of the co ntour approximation method:

a) cv::CHAIN_APPROX_NONE   
b) cv::CHAIN_APPROX_SIMPLE   
c) cv::CHAIN_APPROX_TC89_L1  
d) cv::CHAIN_APPROX_TC89_KCOS  
5. Although it might be a little slow, try running backgr ound segmentation when the video input is first 
pre-segmented by using cv::pyrMeanShiftFiltering() . That is, the input stream is first 
mean-shift segmented and then passed for background learning—and later testing for foreground—by 
the codebook background segmentation routine. 
a) Show the results compared to not ru nning the mean-shift segmentation. 
b) Try systematically varying the maximum pyramid level ( max_level ), spatial radius ( sp), and 
color radius ( cr) of the mean-shift segmentation. Compare those results. 
6. Set up a camera in your room or looking out over a scene. Use 
cv::BackgroundSubtractorMOG  to “watch” your room or scene over several days.  
a) Detect lights on and off by looking at very instantaneous change in brightness.

a) Detect lights on and off by looking at very instantaneous change in brightness. 
b) Segment out (save instances to a file) of fast ch anging objects (for example people) from medium 
changing objects (for  example chairs).

Reinfo rcement 
Learning
An Introduction
second edition
Richard S. Sutton and Andrew G. Barto

Adaptive Computation and Machine Learning
Francis Bach
A complete list of books published in the Adaptive Computation and Machine Learning
series appears at the back of this book.
The cover design is based on the trajectories of a simulated bicycle controlled by a
reinforcement learning system developed by Jette Randløv.

Reinforcement Learning:
Cambridge, Massachusetts
London, England

c2018 Richard S. Sutton and Andrew G. Barto
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivs
2.0 Generic License. To view a copy of this license, visit http://creativecommons.
org/licenses/by-nc-nd/2.0/ or send a letter to Creative Commons, PO Box 1866,
Mountain View, CA 94042, USA.
This book was set in —— by ——. Printed and bound in the United States of America.
Comp: Please ﬁll in font, compositor, and location
Library of Congress Cataloging-in-Publication Data is available.
ISBN: 978-0-262-19398-6

Contents
Preface to the Second Edition xiii
Preface to the First Edition xvii
Summary of Notation xix
1.1 Reinforcement Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 Elements of Reinforcement Learning . . . . . . . . . . . . . . . . . . . . . 6
1.4 Limitations and Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.5 An Extended Example: Tic-Tac-Toe . . . . . . . . . . . . . . . . . . . . . 8
1.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.7 Early History of Reinforcement Learning . . . . . . . . . . . . . . . . . . . 13
I Tabular Solution Methods 23
2 Multi-armed Bandits 25
2.1 A k-armed Bandit Problem . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.2 Action-value Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.3 The 10-armed Testbed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

2.4 Incremental Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.5 Tracking a Nonstationary Problem . . . . . . . . . . . . . . . . . . . . . . 32
2.6 Optimistic Initial Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
2.7 Upper-Conﬁdence-Bound Action Selection . . . . . . . . . . . . . . . . . . 35
2.8 Gradient Bandit Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.9 Associative Search (Contextual Bandits) . . . . . . . . . . . . . . . . . . . 41

2.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
vii

viii Contents
3 Finite Markov Decision Processes 47
3.1 The Agent–Environment Interface . . . . . . . . . . . . . . . . . . . . . . 47
3.2 Goals and Rewards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
3.3 Returns and Episodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
3.4 Uniﬁed Notation for Episodic and Continuing Tasks . . . . . . . . . . . . 57
3.5 Policies and Value Functions . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.6 Optimal Policies and Optimal Value Functions . . . . . . . . . . . . . . . 62
3.7 Optimality and Approximation . . . . . . . . . . . . . . . . . . . . . . . . 67
3.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
4 Dynamic Programming 73
4.1 Policy Evaluation (Prediction) . . . . . . . . . . . . . . . . . . . . . . . . 74
4.2 Policy Improvement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
4.3 Policy Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

4.4 Value Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
4.5 Asynchronous Dynamic Programming . . . . . . . . . . . . . . . . . . . . 85
4.6 Generalized Policy Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . 86
4.7 Eciency of Dynamic Programming . . . . . . . . . . . . . . . . . . . . . 87
4.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
5 Monte Carlo Methods 91
5.1 Monte Carlo Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

5.2 Monte Carlo Estimation of Action Values . . . . . . . . . . . . . . . . . . 96
5.3 Monte Carlo Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
5.4 Monte Carlo Control without Exploring Starts . . . . . . . . . . . . . . . 100
5.5 O↵-policy Prediction via Importance Sampling . . . . . . . . . . . . . . . 103
5.6 Incremental Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . 109
5.7 O↵-policy Monte Carlo Control . . . . . . . . . . . . . . . . . . . . . . . . 110

5.8 *Discounting-aware Importance Sampling . . . . . . . . . . . . . . . . . . 112
5.9 *Per-decision Importance Sampling . . . . . . . . . . . . . . . . . . . . . . 114
5.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
6 Temporal-Di↵erence Learning 119
6.1 TD Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
6.2 Advantages of TD Prediction Methods . . . . . . . . . . . . . . . . . . . . 124

6.3 Optimality of TD(0) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
6.4 Sarsa: On-policy TD Control . . . . . . . . . . . . . . . . . . . . . . . . . 129
6.5 Q-learning: O↵-policy TD Control . . . . . . . . . . . . . . . . . . . . . . 131
6.6 Expected Sarsa . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
6.7 Maximization Bias and Double Learning . . . . . . . . . . . . . . . . . . . 134
6.8 Games, Afterstates, and Other Special Cases . . . . . . . . . . . . . . . . 136

6.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

Contents ix
7n-step Bootstrapping 141
7.1 n-step TD Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
7.2 n-step Sarsa . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
7.3 n-step O↵-policy Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
7.4 *Per-decision Methods with Control Variates . . . . . . . . . . . . . . . . 150
7.5 O↵-policy Learning Without Importance Sampling:
The n-step Tree Backup Algorithm . . . . . . . . . . . . . . . . . . . . . . 152
7.6 *A Unifying Algorithm: n-step Q()..................... 154
7.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
8 Planning and Learning with Tabular Methods 159
8.1 Models and Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
8.2 Dyna: Integrated Planning, Acting, and Learning . . . . . . . . . . . . . . 161
8.3 When the Model Is Wrong . . . . . . . . . . . . . . . . . . . . . . . . . . . 166

8.4 Prioritized Sweeping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
8.5 Expected vs. Sample Updates . . . . . . . . . . . . . . . . . . . . . . . . . 172
8.6 Trajectory Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
8.7 Real-time Dynamic Programming . . . . . . . . . . . . . . . . . . . . . . . 177
8.8 Planning at Decision Time . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
8.9 Heuristic Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181

8.10 Rollout Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
8.11 Monte Carlo Tree Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
8.12 Summary of the Chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
8.13 Summary of Part I: Dimensions . . . . . . . . . . . . . . . . . . . . . . . . 189
II Approximate Solution Methods 195
9 On-policy Prediction with Approximation 197
9.1 Value-function Approximation . . . . . . . . . . . . . . . . . . . . . . . . . 198

9.2 The Prediction Objective ( VE) . . . . . . . . . . . . . . . . . . . . . . . . 199
9.3 Stochastic-gradient and Semi-gradient Methods . . . . . . . . . . . . . . . 200
9.4 Linear Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
9.5 Feature Construction for Linear Methods . . . . . . . . . . . . . . . . . . 210
9.5.1 Polynomials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
9.5.2 Fourier Basis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211

9.5.3 Coarse Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
9.5.4 Tile Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
9.5.5 Radial Basis Functions . . . . . . . . . . . . . . . . . . . . . . . . . 221
9.6 Selecting Step-Size Parameters Manually . . . . . . . . . . . . . . . . . . . 222
9.7 Nonlinear Function Approximation: Artiﬁcial Neural Networks . . . . . . 223
9.8 Least-Squares TD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228

x Contents
9.9 Memory-based Function Approximation . . . . . . . . . . . . . . . . . . . 230
9.10 Kernel-based Function Approximation . . . . . . . . . . . . . . . . . . . . 232
9.11 Looking Deeper at On-policy Learning: Interest and Emphasis . . . . . . 234
9.12 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
10 On-policy Control with Approximation 243
10.1 Episodic Semi-gradient Control . . . . . . . . . . . . . . . . . . . . . . . . 243
10.2 Semi-gradient n-step Sarsa . . . . . . . . . . . . . . . . . . . . . . . . . . 247
10.3 Average Reward: A New Problem Setting for Continuing Tasks . . . . . . 249
10.4 Deprecating the Discounted Setting . . . . . . . . . . . . . . . . . . . . . . 253
10.5 Di↵erential Semi-gradient n-step Sarsa . . . . . . . . . . . . . . . . . . . . 255
10.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
11 *O↵-policy Methods with Approximation 257

11.1 Semi-gradient Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
11.2 Examples of O↵-policy Divergence . . . . . . . . . . . . . . . . . . . . . . 260
11.3 The Deadly Triad . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
11.4 Linear Value-function Geometry . . . . . . . . . . . . . . . . . . . . . . . 266
11.5 Gradient Descent in the Bellman Error . . . . . . . . . . . . . . . . . . . . 269
11.6 The Bellman Error is Not Learnable . . . . . . . . . . . . . . . . . . . . . 274

11.7 Gradient-TD Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
11.8 Emphatic-TD Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281
11.9 Reducing Variance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
11.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
12 Eligibility Traces 287
12.1 The -return . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288
12.2 TD( )...................................... 292

13 Policy Gradient Methods 321
13.1 Policy Approximation and its Advantages . . . . . . . . . . . . . . . . . . 322
13.2 The Policy Gradient Theorem . . . . . . . . . . . . . . . . . . . . . . . . . 324
13.3 REINFORCE: Monte Carlo Policy Gradient . . . . . . . . . . . . . . . . . 326
13.4 REINFORCE with Baseline . . . . . . . . . . . . . . . . . . . . . . . . . . 329
13.5 Actor–Critic Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331
13.6 Policy Gradient for Continuing Problems . . . . . . . . . . . . . . . . . . 333
13.7 Policy Parameterization for Continuous Actions . . . . . . . . . . . . . . . 335
13.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337
III Looking Deeper 339
14 Psychology 341
14.1 Prediction and Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342
14.2 Classical Conditioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343

14.2.1 Blocking and Higher-order Conditioning . . . . . . . . . . . . . . . 345
14.2.2 The Rescorla–Wagner Model . . . . . . . . . . . . . . . . . . . . . 346
14.2.3 The TD Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349
14.2.4 TD Model Simulations . . . . . . . . . . . . . . . . . . . . . . . . . 350
14.3 Instrumental Conditioning . . . . . . . . . . . . . . . . . . . . . . . . . . . 357
14.4 Delayed Reinforcement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361

14.5 Cognitive Maps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363
14.6 Habitual and Goal-directed Behavior . . . . . . . . . . . . . . . . . . . . . 364
14.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 368
15 Neuroscience 377
15.1 Neuroscience Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 378
15.2 Reward Signals, Reinforcement Signals, Values, and Prediction Errors . . 380

15.3 The Reward Prediction Error Hypothesis . . . . . . . . . . . . . . . . . . 381
15.4 Dopamine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383
15.5 Experimental Support for the Reward Prediction Error Hypothesis . . . . 387
15.6 TD Error/Dopamine Correspondence . . . . . . . . . . . . . . . . . . . . . 390
15.7 Neural Actor–Critic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395
15.8 Actor and Critic Learning Rules . . . . . . . . . . . . . . . . . . . . . . . 398

15.9 Hedonistic Neurons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402
15.10 Collective Reinforcement Learning . . . . . . . . . . . . . . . . . . . . . . 404
15.11 Model-based Methods in the Brain . . . . . . . . . . . . . . . . . . . . . . 407
15.12 Addiction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409
15.13 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410

16 Applications and Case Studies 421
16.1 TD-Gammon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 421
16.2 Samuel’s Checkers Player . . . . . . . . . . . . . . . . . . . . . . . . . . . 426
16.3 Watson’s Daily-Double Wagering . . . . . . . . . . . . . . . . . . . . . . . 429
16.4 Optimizing Memory Control . . . . . . . . . . . . . . . . . . . . . . . . . . 432
16.5 Human-level Video Game Play . . . . . . . . . . . . . . . . . . . . . . . . 436
16.6 Mastering the Game of Go . . . . . . . . . . . . . . . . . . . . . . . . . . . 441
16.6.1 AlphaGo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444
16.6.2 AlphaGo Zero . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 447
16.7 Personalized Web Services . . . . . . . . . . . . . . . . . . . . . . . . . . . 450
16.8 Thermal Soaring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 453
17 Frontiers 459
17.1 General Value Functions and Auxiliary Tasks . . . . . . . . . . . . . . . . 459

17.2 Temporal Abstraction via Options . . . . . . . . . . . . . . . . . . . . . . 461
17.3 Observations and State . . . . . . . . . . . . . . . . . . . . . . . . . . . . 464
17.4 Designing Reward Signals . . . . . . . . . . . . . . . . . . . . . . . . . . . 469
17.5 Remaining Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 472
17.6 The Future of Artiﬁcial Intelligence . . . . . . . . . . . . . . . . . . . . . . 475
References 481
Index 519

Preface to the Second Edition
The twenty years since the publication of the ﬁrst edition of this book have seen tremendous
progress in artiﬁcial intelligence, propelled in large part by advances in machine learning,
including advances in reinforcement learning. Although the impressive computational
power that became available is responsible for some of these advances, new developments
in theory and algorithms have been driving forces as well. In the face of this progress, a
second edition of our 1998 book was long overdue, and we ﬁnally began the project in
2012. Our goal for the second edition was the same as our goal for the ﬁrst: to provide a
clear and simple account of the key ideas and algorithms of reinforcement learning that
is accessible to readers in all the related disciplines. The edition remains an introduction,
and we retain a focus on core, online learning algorithms. This edition includes some new
topics that rose to importance over the intervening years, and we expanded coverage of

topics that we now understand better. But we made no attempt to provide comprehensive
coverage of the ﬁeld, which has exploded in many di↵erent directions. We apologize for
having to leave out all but a handful of these contributions.
As in the ﬁrst edition, we chose not to produce a rigorous formal treatment of
reinforcement learning, or to formulate it in the most general terms. However, our deeper
understanding of some topics since the ﬁrst edition required a bit more mathematics
to explain; we have set o↵ the more mathematical parts in shaded boxes that the non-

mathematically-inclined may choose to skip. We also use a slightly di↵erent notation
than was used in the ﬁrst edition. In teaching, we have found that the new notation
helps to address some common points of confusion. It emphasizes the di↵erence between
random variables, denoted with capital letters, and their instantiations, denoted in lower
case. For example, the state, action, and reward at time step tare denoted St,At,
andRt, while their possible values might be denoted s,a, and r. Along with this, it is

natural to use lower case for value functions (e.g., v⇡) and restrict capitals to their tabular
estimates (e.g., Qt(s, a)). Approximate value functions are deterministic functions of
random parameters and are thus also in lower case (e.g., ˆv(s,wt)⇡v⇡(s)). Vectors, such
as the weight vector wt(formerly ✓t) and the feature vector xt(formerly t), are bold
and written in lowercase even if they are random variables. Uppercase bold is reserved for
matrices. In the ﬁrst edition we used special notations, Pa
ss0andRa
ss0, for the transition

ss0andRa
probabilities and expected rewards. One weakness of that notation is that it still did not
fully characterize the dynamics of the rewards, giving only their expectations, which is
sucient for dynamic programming but not for reinforcement learning. Another weakness
xiii

xiv Preface to the Second Edition
is the excess of subscripts and superscripts. In this edition we use the explicit notation of
p(s0,r|s, a) for the joint probability for the next state and reward given the current state
and action. All the changes in notation are summarized in a table on page xix.
The second edition is signiﬁcantly expanded, and its top-level organization has been
changed. After the introductory ﬁrst chapter, the second edition is divided into three new
parts. The ﬁrst part (Chapters 2–8) treats as much of reinforcement learning as possible
without going beyond the tabular case for which exact solutions can be found. We cover
both learning and planning methods for the tabular case, as well as their uniﬁcation
inn-step methods and in Dyna. Many algorithms presented in this part are new to
the second edition, including UCB, Expected Sarsa, Double learning, tree-backup, Q(),
RTDP, and MCTS. Doing the tabular case ﬁrst, and thoroughly, enables core ideas to be

developed in the simplest possible setting. The second part of the book (Chapters 9–13)
is then devoted to extending the ideas to function approximation. It has new sections on
artiﬁcial neural networks, the fourier basis, LSTD, kernel-based methods, Gradient-TD
and Emphatic-TD methods, average-reward methods, true online TD( ), and policy-
gradient methods. The second edition signiﬁcantly expands the treatment of o↵-policy
learning, ﬁrst for the tabular case in Chapters 5–7, then with function approximation in

Chapters 11 and 12. Another change is that the second edition separates the forward-view
idea of n-step bootstrapping (now treated more fully in Chapter 7) from the backward-
view idea of eligibility traces (now treated independently in Chapter 12). The third part
of the book has large new chapters on reinforcement learning’s relationships to psychology
(Chapter 14) and neuroscience (Chapter 15), as well as an updated case-studies chapter
including Atari game playing, Watson’s wagering strategy, and the Go playing programs

AlphaGo and AlphaGo Zero (Chapter 16). Still, out of necessity we have included only a
small subset of all that has been done in the ﬁeld. Our choices reﬂect our long-standing
interests in inexpensive model-free methods that should scale well to large applications.
The ﬁnal chapter now includes a discussion of the future societal impacts of reinforcement
learning. For better or worse, the second edition is about twice as large as the ﬁrst.
This book is designed to be used as the primary text for a one- or two-semester

course on reinforcement learning. For a one-semester course, the ﬁrst ten chapters should
be covered in order and form a good core, to which can be added material from the
other chapters, from other books such as Bertsekas and Tsitsiklis (1996), Wiering and
van Otterlo (2012), and Szepesv´ ari (2010), or from the literature, according to taste.
Depending of the students’ background, some additional material on online supervised
learning may be helpful. The ideas of options and option models are a natural addition

(Sutton, Precup and Singh, 1999). A two-semester course can cover all the chapters as
well as supplementary material. The book can also be used as part of broader courses
on machine learning, artiﬁcial intelligence, or neural networks. In this case, it may be
desirable to cover only a subset of the material. We recommend covering Chapter 1 for a
brief overview, Chapter 2 through Section 2.4, Chapter 3, and then selecting sections
from the remaining chapters according to time and interests. Chapter 6 is the most

important for the subject and for the rest of the book. A course focusing on machine
learning or neural networks should cover Chapters 9 and 10, and a course focusing on
artiﬁcial intelligence or planning should cover Chapter 8. Throughout the book, sections
and chapters that are more dicult and not essential to the rest of the book are marked

Preface to the Second Edition xv
with a ⇤. These can be omitted on ﬁrst reading without creating problems later on. Some
exercises are also marked with a ⇤to indicate that they are more advanced and not
essential to understanding the basic material of the chapter.
Most chapters end with a section entitled “Bibliographical and Historical Remarks,”
wherein we credit the sources of the ideas presented in that chapter, provide pointers to
further reading and ongoing research, and describe relevant historical background. Despite
our attempts to make these sections authoritative and complete, we have undoubtedly left
out some important prior work. For that we again apologize, and we welcome corrections
and extensions for incorporation into the electronic version of the book.
Like the ﬁrst edition, this edition of the book is dedicated to the memory of A. Harry
Klopf. It was Harry who introduced us to each other, and it was his ideas about the brain

and artiﬁcial intelligence that launched our long excursion into reinforcement learning.
Trained in neurophysiology and long interested in machine intelligence, Harry was a
senior scientist aliated with the Avionics Directorate of the Air Force Oce of Scientiﬁc
Research (AFOSR) at Wright-Patterson Air Force Base, Ohio. He was dissatisﬁed with
the great importance attributed to equilibrium-seeking processes, including homeostasis
and error-correcting pattern classiﬁcation methods, in explaining natural intelligence

and in providing a basis for machine intelligence. He noted that systems that try to
maximize something (whatever that might be) are qualitatively di↵erent from equilibrium-
seeking systems, and he argued that maximizing systems hold the key to understanding
important aspects of natural intelligence and for building artiﬁcial intelligences. Harry was
instrumental in obtaining funding from AFOSR for a project to assess the scientiﬁc merit
of these and related ideas. This project was conducted in the late 1970s at the University

of Massachusetts Amherst (UMass Amherst), initially under the direction of Michael
Arbib, William Kilmer, and Nico Spinelli, professors in the Department of Computer
and Information Science at UMass Amherst, and founding members of the Cybernetics
Center for Systems Neuroscience at the University, a farsighted group focusing on the
intersection of neuroscience and artiﬁcial intelligence. Barto, a recent Ph.D. from the
University of Michigan, was hired as post doctoral researcher on the project. Meanwhile,

Sutton, an undergraduate studying computer science and psychology at Stanford, had
been corresponding with Harry regarding their mutual interest in the role of stimulus
timing in classical conditioning. Harry suggested to the UMass group that Sutton would
be a great addition to the project. Thus, Sutton became a UMass graduate student,
whose Ph.D. was directed by Barto, who had become an Associate Professor. The study
of reinforcement learning as presented in this book is rightfully an outcome of that

project instigated by Harry and inspired by his ideas. Further, Harry was responsible
for bringing us, the authors, together in what has been a long and enjoyable interaction.
By dedicating this book to Harry we honor his essential contributions, not only to the
ﬁeld of reinforcement learning, but also to our collaboration. We also thank Professors
Arbib, Kilmer, and Spinelli for the opportunity they provided to us to begin exploring

these ideas. Finally, we thank AFOSR for generous support over the early years of our
research, and the NSF for its generous support over many of the following years.
We have very many people to thank for their inspiration and help with this second
edition. Everyone we acknowledged for their inspiration and help with the ﬁrst edition

xvi Preface to the Second Edition
deserve our deepest gratitude for this edition as well, which would not exist were it not
for their contributions to edition number one. To that long list we must add many others
who contributed speciﬁcally to the second edition. Our students over the many years that
we have taught this material contributed in countless ways: exposing errors, o↵ering ﬁxes,
and—not the least—being confused in places where we could have explained things better.
We especially thank Martha Steenstrup for reading and providing detailed comments
throughout. The chapters on psychology and neuroscience could not have been written
without the help of many experts in those ﬁelds. We thank John Moore for his patient
tutoring over many many years on animal learning experiments, theory, and neuroscience,
and for his careful reading of multiple drafts of Chapters 14 and 15. We also thank Matt
Botvinick, Nathaniel Daw, Peter Dayan, and Yael Niv for their penetrating comments on

drafts of these chapter, their essential guidance through the massive literature, and their
interception of many of our errors in early drafts. Of course, the remaining errors in these
chapters—and there must still be some—are totally our own. We thank Phil Thomas for
helping us make these chapters accessible to non-psychologists and non-neuroscientists,
and we thank Peter Sterling for helping us improve the exposition. We are grateful to Jim
Houk for introducing us to the subject of information processing in the basal ganglia and

for alerting us to other relevant aspects of neuroscience. Jos´ e Mart´ ınez, Terry Sejnowski,
David Silver, Gerry Tesauro, Georgios Theocharous, and Phil Thomas generously helped
us understand details of their reinforcement learning applications for inclusion in the
case-studies chapter, and they provided helpful comments on drafts of these sections.
Special thanks are owed to David Silver for helping us better understand Monte Carlo
Tree Search and the DeepMind Go-playing programs. We thank George Konidaris for his

help with the section on the Fourier basis. Emilio Cartoni, Thomas Cederborg, Stefan
Dernbach, Clemens Rosenbaum, Patrick Taylor, Thomas Colin, and Pierre-Luc Bacon
helped us in a number important ways for which we are most grateful.
Sutton would also like to thank the members of the Reinforcement Learning and
Artiﬁcial Intelligence laboratory at the University of Alberta for contributions to the
second edition. He owes a particular debt to Rupam Mahmood for essential contributions
to the treatment of o↵-policy Monte Carlo methods in Chapter 5, to Hamid Maei for

helping develop the perspective on o↵-policy learning presented in Chapter 11, to Eric
Graves for conducting the experiments in Chapter 13, to Shangtong Zhang for replicating
and thus verifying almost all the experimental results, to Kris De Asis for improving
the new technical content of Chapters 7 and 12, and to Harm van Seijen for insights
that led to the separation of n-step methods from eligibility traces and (along with Hado
van Hasselt) for the ideas involving exact equivalence of forward and backward views of

eligibility traces presented in Chapter 12. Sutton also gratefully acknowledges the support
and freedom he was granted by the Government of Alberta and the National Science and
Engineering Research Council of Canada throughout the period during which the second
edition was conceived and written. In particular, he would like to thank Randy Goebel
for creating a supportive and far-sighted environment for research in Alberta. He would
also like to thank DeepMind their support in the last six months of writing the book.

Finally, we owe thanks to the many careful readers of drafts of the second edition that
we posted on the internet. They found many errors that we had missed and alerted us to
potential points of confusion.

Preface to the First Edition
We ﬁrst came to focus on what is now known as reinforcement learning in late 1979. We
were both at the University of Massachusetts, working on one of the earliest projects to
revive the idea that networks of neuronlike adaptive elements might prove to be a promising
approach to artiﬁcial adaptive intelligence. The project explored the “heterostatic theory
of adaptive systems” developed by A. Harry Klopf. Harry’s work was a rich source of
ideas, and we were permitted to explore them critically and compare them with the long
history of prior work in adaptive systems. Our task became one of teasing the ideas apart
and understanding their relationships and relative importance. This continues today,
but in 1979 we came to realize that perhaps the simplest of the ideas, which had long
been taken for granted, had received surprisingly little attention from a computational
perspective. This was simply the idea of a learning system that wants something, that

adapts its behavior in order to maximize a special signal from its environment. This
was the idea of a “hedonistic” learning system, or, as we would say now, the idea of
reinforcement learning.
Like others, we had a sense that reinforcement learning had been thoroughly explored
in the early days of cybernetics and artiﬁcial intelligence. On closer inspection, though,
we found that it had been explored only slightly. While reinforcement learning had clearly
motivated some of the earliest computational studies of learning, most of these researchers

had gone on to other things, such as pattern classiﬁcation, supervised learning, and
adaptive control, or they had abandoned the study of learning altogether. As a result, the
special issues involved in learning how to get something from the environment received
relatively little attention. In retrospect, focusing on this idea was the critical step that
set this branch of research in motion. Little progress could be made in the computational
study of reinforcement learning until it was recognized that such a fundamental idea had

not yet been thoroughly explored.
The ﬁeld has come a long way since then, evolving and maturing in several directions.
Reinforcement learning has gradually become one of the most active research areas in ma-
chine learning, artiﬁcial intelligence, and neural network research. The ﬁeld has developed
strong mathematical foundations and impressive applications. The computational study
of reinforcement learning is now a large ﬁeld, with hundreds of active researchers around
the world in diverse disciplines such as psychology, control theory, artiﬁcial intelligence,

and neuroscience. Particularly important have been the contributions establishing and
developing the relationships to the theory of optimal control and dynamic programming.
xvii

xviii Preface to the First Edition
The overall problem of learning from interaction to achieve goals is still far from being
solved, but our understanding of it has improved signiﬁcantly. We can now place compo-
nent ideas, such as temporal-di↵erence learning, dynamic programming, and function
approximation, within a coherent perspective with respect to the overall problem.
Our goal in writing this book was to provide a clear and simple account of the key
ideas and algorithms of reinforcement learning. We wanted our treatment to be accessible
to readers in all of the related disciplines, but we could not cover all of these perspectives
in detail. For the most part, our treatment takes the point of view of artiﬁcial intelligence
and engineering. Coverage of connections to other ﬁelds we leave to others or to another
time. We also chose not to produce a rigorous formal treatment of reinforcement learning.
We did not reach for the highest possible level of mathematical abstraction and did not

rely on a theorem–proof format. We tried to choose a level of mathematical detail that
points the mathematically inclined in the right directions without distracting from the
simplicity and potential generality of the underlying ideas.
...
In some sense we have been working toward this book for thirty years, and we have lots
of people to thank. First, we thank those who have personally helped us develop the overall
view presented in this book: Harry Klopf, for helping us recognize that reinforcement

...
learning needed to be revived; Chris Watkins, Dimitri Bertsekas, John Tsitsiklis, and
Paul Werbos, for helping us see the value of the relationships to dynamic programming;
John Moore and Jim Kehoe, for insights and inspirations from animal learning theory;
Oliver Selfridge, for emphasizing the breadth and importance of adaptation; and, more
generally, our colleagues and students who have contributed in countless ways: Ron
Williams, Charles Anderson, Satinder Singh, Sridhar Mahadevan, Steve Bradtke, Bob

Crites, Peter Dayan, and Leemon Baird. Our view of reinforcement learning has been
signiﬁcantly enriched by discussions with Paul Cohen, Paul Utgo↵, Martha Steenstrup,
Gerry Tesauro, Mike Jordan, Leslie Kaelbling, Andrew Moore, Chris Atkeson, Tom
Mitchell, Nils Nilsson, Stuart Russell, Tom Dietterich, Tom Dean, and Bob Narendra.
We thank Michael Littman, Gerry Tesauro, Bob Crites, Satinder Singh, and Wei Zhang
for providing speciﬁcs of Sections 4.7, 15.1, 15.4, 15.4, and 15.6 respectively. We thank

the Air Force Oce of Scientiﬁc Research, the National Science Foundation, and GTE
Laboratories for their long and farsighted support.
We also wish to thank the many people who have read drafts of this book and
provided valuable comments, including Tom Kalt, John Tsitsiklis, Pawel Cichosz, Olle
G¨ allmo, Chuck Anderson, Stuart Russell, Ben Van Roy, Paul Steenstrup, Paul Cohen,
Sridhar Mahadevan, Jette Randlov, Brian Sheppard, Thomas O’Connell, Richard Coggins,

Cristina Versino, John H. Hiett, Andreas Badelt, Jay Ponte, Joe Beck, Justus Piater,
Martha Steenstrup, Satinder Singh, Tommi Jaakkola, Dimitri Bertsekas, Torbj¨ orn Ekman,
Christina Bj¨ orkman, Jakob Carlstr¨ om, and Olle Palmgren. Finally, we thank Gwyn
Mitchell for helping in many ways, and Harry Stanton and Bob Prior for being our
champions at MIT Press.

Summary of Notation
Capital letters are used for random variables, whereas lower case letters are used for
the values of random variables and for scalar functions. Quantities that are required to
be real-valued vectors are written in bold and in lower case (even if random variables).
Matrices are bold capitals.
.= equality relationship that is true by deﬁnition
⇡approximately equal
/proportional to
Pr{X=x} probability that a random variable Xtakes on the value x
X⇠p random variable Xselected from distribution p(x).=P r {X=x}
E[X] expectation of a random variable X,i . e . , E[X].=P
xp(x)x
argmaxaf(a) a value of aat which f(a) takes its maximal value
lnx natural logarithm of x
exthe base of the natural logarithm, e⇡2.71828, carried to power x;elnx=x
R set of real numbers
f:X!Y function ffrom elements of set Xto elements of set Y
 assignment
(a, b] the real interval between aandbincluding bbut not including a
" probability of taking a random action in an "-greedy policy
↵,  step-size parameters

xp(x)x
 discount-rate parameter
 decay-rate parameter for eligibility traces
predicate indicator function (
 predicate.=1i ft h e predicate is true, else 0)
In a multi-arm bandit problem:
k number of actions (arms)
t discrete time step or play number
q⇤(a) true value (expected reward) of action a
Qt(a) estimate at time tofq⇤(a)
Nt(a) number of times action ahas been selected up prior to time t
Ht(a) learned preference for selecting action aat time t
⇡t(a) probability of selecting action aat time t

¯Rt estimate at time tof the expected reward given ⇡t
xix

xx Summary of Notation
In a Markov Decision Process:
s, s0states
a an action
r a reward
S set of all nonterminal states
S+set of all states, including the terminal state
A(s) set of all actions available in state s
R set of all possible rewards, a ﬁnite subset of R
⇢subset of; e.g., R⇢R
2is an element of; e.g., s2S,r2R
|S| number of elements in set S
t discrete time step
T,T(t) ﬁnal time step of an episode, or of the episode including time step t
At action at time t
St state at time t, typically due, stochastically, to St1andAt1
Rt reward at time t, typically due, stochastically, to St1andAt1
⇡ policy (decision-making rule)
⇡(s) action taken in state sunder deterministic policy ⇡
⇡(a|s) probability of taking action ain state sunder stochastic policy ⇡
Gt return following time t
h horizon, the time step one looks up to in a forward view
Gt:t+n,Gt:hn-step return from t+1t o t+n, or to h(discounted and corrected)
¯Gt:h ﬂat return (undiscounted and uncorrected) from t+1t o h(Section 5.8)
G

Summary of Notation xxi
t temporal-di↵erence (TD) error at t(a random variable) (Section 6.1)
s
t,a
t state- and action-speciﬁc forms of the TD error (Section 12.9)
n inn-step methods, nis the number of steps of bootstrapping
d dimensionality—the number of components of w
d0alternate dimensionality—the number of components of ✓
w,wt d-vector of weights underlying an approximate value function
wi,wt,i ith component of learnable weight vector
ˆv(s,w) approximate value of state sgiven weight vector w
vw(s) alternate notation for ˆ v(s,w)
ˆq(s, a,w) approximate value of state–action pair s, agiven weight vector w
rˆv(s,w) column vector of partial derivatives of ˆ v(s,w)w i t hr e s p e c tt o w
rˆq(s, a,w) column vector of partial derivatives of ˆ q(s, a,w)w i t hr e s p e c tt o w
x(s) vector of features visible when in state s
x(s, a) vector of features visible when in state staking action a
xi(s),xi(s, a)ith component of vector x(s) orx(s, a)
xt shorthand for x(St) orx(St,At)

w>x inner product of vectors, w>x.=P
iwixi; e.g., ˆ v(s,w).=w>x(s)
v,vt secondary d-vector of weights, used to learn w(Chapter 11)
zt d-vector of eligibility traces at time t(Chapter 12)
✓,✓t parameter vector of target policy (Chapter 13)
⇡(a|s,✓) probability of taking action ain state sgiven parameter vector ✓
⇡✓ policy corresponding to parameter ✓
r⇡(a|s,✓) column vector of partial derivatives of ⇡(a|s,✓)w i t hr e s p e c tt o ✓
J(✓) performance measure for the policy ⇡✓

rJ(✓) column vector of partial derivatives of J(✓)w i t hr e s p e c tt o ✓
h(s, a,✓) preference for selecting action ain state sbased on ✓
b(a|s) behavior policy used to select actions while learning about target policy ⇡
b(s) a baseline function b:S7!Rfor policy-gradient methods
b branching factor for an MDP or search tree
⇢t:h importance sampling ratio for time tthrough time h(Section 5.5)
⇢t importance sampling ratio for time talone, ⇢t.=⇢t:t
r(⇡) average reward (reward rate) for policy ⇡(Section 10.3)
¯Rt estimate of r(⇡) at time t

µ(s) on-policy distribution over states (Section 9.2)
µ |S|-vector of the µ(s) for all s2S
kvk2
µµ-weighted squared norm of value function v,i . e . , kvk2
µ.=P
s2Sµ(s)v(s)2
⌘(s) expected number of visits to state sper episode (page 199)
⇧ projection operator for value functions (page 268)
B⇡ Bellman operator for value functions (Section 11.4)

Chapter 1
The idea that we learn by interacting with our environment is probably the ﬁrst to occur
to us when we think about the nature of learning. When an infant plays, waves its arms,
or looks about, it has no explicit teacher, but it does have a direct sensorimotor connection
to its environment. Exercising this connection produces a wealth of information about
cause and e↵ect, about the consequences of actions, and about what to do in order to
achieve goals. Throughout our lives, such interactions are undoubtedly a major source
of knowledge about our environment and ourselves. Whether we are learning to drive
a car or to hold a conversation, we are acutely aware of how our environment responds
to what we do, and we seek to inﬂuence what happens through our behavior. Learning
from interaction is a foundational idea underlying nearly all theories of learning and
In this book we explore a computational approach to learning from interaction. Rather

than directly theorizing about how people or animals learn, we primarily explore idealized
learning situations and evaluate the e↵ectiveness of various learning methods.1That
is, we adopt the perspective of an artiﬁcial intelligence researcher or engineer. We
explore designs for machines that are e↵ective in solving learning problems of scientiﬁc or
economic interest, evaluating the designs through mathematical analysis or computational
experiments. The approach we explore, called reinforcement learning , is much more

focused on goal-directed learning from interaction than are other approaches to machine
learning.
1.1 Reinforcement Learning
Reinforcement learning is learning what to do—how to map situations to actions—so
as to maximize a numerical reward signal. The learner is not told which actions to
take, but instead must discover which actions yield the most reward by trying them. In
the most interesting and challenging cases, actions may a↵ect not only the immediate
1The relationships to psychology and neuroscience are summarized in Chapters 14 and 15.
1

2C h a p t e r 1 : I n t r o d u c t i o n
reward but also the next situation and, through that, all subsequent rewards. These two
characteristics—trial-and-error search and delayed reward—are the two most important
distinguishing features of reinforcement learning.
Reinforcement learning, like many topics whose names end with “ing,” such as machine
learning and mountaineering, is simultaneously a problem, a class of solution methods
that work well on the problem, and the ﬁeld that studies this problem and its solution
methods. It is convenient to use a single name for all three things, but at the same time
essential to keep the three conceptually separate. In particular, the distinction between
problems and solution methods is very important in reinforcement learning; failing to
make this distinction is the source of many confusions.
We formalize the problem of reinforcement learning using ideas from dynamical sys-
tems theory, speciﬁcally, as the optimal control of incompletely-known Markov decision

processes. The details of this formalization must wait until Chapter 3, but the basic idea
is simply to capture the most important aspects of the real problem facing a learning
agent interacting over time with its environment to achieve a goal. A learning agent
must be able to sense the state of its environment to some extent and must be able to
take actions that a↵ect the state. The agent also must have a goal or goals relating to
the state of the environment. Markov decision processes are intended to include just

these three aspects—sensation, action, and goal—in their simplest possible forms without
trivializing any of them. Any method that is well suited to solving such problems we
consider to be a reinforcement learning method.
Reinforcement learning is di↵erent from supervised learning , the kind of learning studied
in most current research in the ﬁeld of machine learning. Supervised learning is learning
from a training set of labeled examples provided by a knowledgable external supervisor.
Each example is a description of a situation together with a speciﬁcation—the label—of

the correct action the system should take to that situation, which is often to identify a
category to which the situation belongs. The object of this kind of learning is for the
system to extrapolate, or generalize, its responses so that it acts correctly in situations
not present in the training set. This is an important kind of learning, but alone it is not
adequate for learning from interaction. In interactive problems it is often impractical to

obtain examples of desired behavior that are both correct and representative of all the
situations in which the agent has to act. In uncharted territory—where one would expect
learning to be most beneﬁcial—an agent must be able to learn from its own experience.
Reinforcement learning is also di↵erent from what machine learning researchers call
unsupervised learning , which is typically about ﬁnding structure hidden in collections of
unlabeled data. The terms supervised learning and unsupervised learning would seem

to exhaustively classify machine learning paradigms, but they do not. Although one
might be tempted to think of reinforcement learning as a kind of unsupervised learning
because it does not rely on examples of correct behavior, reinforcement learning is trying
to maximize a reward signal instead of trying to ﬁnd hidden structure. Uncovering
structure in an agent’s experience can certainly be useful in reinforcement learning, but by
itself does not address the reinforcement learning problem of maximizing a reward signal.

We therefore consider reinforcement learning to be a third machine learning paradigm,
alongside supervised learning and unsupervised learning and perhaps other paradigms.

1.1. Reinforcement Learning 3
One of the challenges that arise in reinforcement learning, and not in other kinds
of learning, is the trade-o↵ between exploration and exploitation. To obtain a lot of
reward, a reinforcement learning agent must prefer actions that it has tried in the past
and found to be e↵ective in producing reward. But to discover such actions, it has to
try actions that it has not selected before. The agent has to exploit what it has already
experienced in order to obtain reward, but it also has to explore in order to make better
action selections in the future. The dilemma is that neither exploration nor exploitation
can be pursued exclusively without failing at the task. The agent must try a variety of
actions andprogressively favor those that appear to be best. On a stochastic task, each
action must be tried many times to gain a reliable estimate of its expected reward. The
exploration–exploitation dilemma has been intensively studied by mathematicians for

many decades, yet remains unresolved. For now, we simply note that the entire issue of
balancing exploration and exploitation does not even arise in supervised and unsupervised
learning, at least in the purest forms of these paradigms.
Another key feature of reinforcement learning is that it explicitly considers the whole
problem of a goal-directed agent interacting with an uncertain environment. This is in
contrast to many approaches that consider subproblems without addressing how they
might ﬁt into a larger picture. For example, we have mentioned that much of machine

learning research is concerned with supervised learning without explicitly specifying how
such an ability would ﬁnally be useful. Other researchers have developed theories of
planning with general goals, but without considering planning’s role in real-time decision
making, or the question of where the predictive models necessary for planning would
come from. Although these approaches have yielded many useful results, their focus on
isolated subproblems is a signiﬁcant limitation.

Reinforcement learning takes the opposite tack, starting with a complete, interactive,
goal-seeking agent. All reinforcement learning agents have explicit goals, can sense
aspects of their environments, and can choose actions to inﬂuence their environments.
Moreover, it is usually assumed from the beginning that the agent has to operate despite
signiﬁcant uncertainty about the environment it faces. When reinforcement learning
involves planning, it has to address the interplay between planning and real-time action

selection, as well as the question of how environment models are acquired and improved.
When reinforcement learning involves supervised learning, it does so for speciﬁc reasons
that determine which capabilities are critical and which are not. For learning research to
make progress, important subproblems have to be isolated and studied, but they should
be subproblems that play clear roles in complete, interactive, goal-seeking agents, even if
all the details of the complete agent cannot yet be ﬁlled in.
By a complete, interactive, goal-seeking agent we do not always mean something like

a complete organism or robot. These are clearly examples, but a complete, interactive,
goal-seeking agent can also be a component of a larger behaving system. In this case,
the agent directly interacts with the rest of the larger system and indirectly interacts
with the larger system’s environment. A simple example is an agent that monitors the
charge level of robot’s battery and sends commands to the robot’s control architecture.

This agent’s environment is the rest of the robot together with the robot’s environment.
One must look beyond the most obvious examples of agents and their environments to

4C h a p t e r 1 : I n t r o d u c t i o n
appreciate the generality of the reinforcement learning framework.
One of the most exciting aspects of modern reinforcement learning is its substantive
and fruitful interactions with other engineering and scientiﬁc disciplines. Reinforcement
learning is part of a decades-long trend within artiﬁcial intelligence and machine learning
toward greater integration with statistics, optimization, and other mathematical subjects.
For example, the ability of some reinforcement learning methods to learn with parameter-
ized approximators addresses the classical “curse of dimensionality” in operations research
and control theory. More distinctively, reinforcement learning has also interacted strongly
with psychology and neuroscience, with substantial beneﬁts going both ways. Of all the
forms of machine learning, reinforcement learning is the closest to the kind of learning
that humans and other animals do, and many of the core algorithms of reinforcement

learning were originally inspired by biological learning systems. Reinforcement learning
has also given back, both through a psychological model of animal learning that better
matches some of the empirical data, and through an inﬂuential model of parts of the
brain’s reward system. The body of this book develops the ideas of reinforcement learning
that pertain to engineering and artiﬁcial intelligence, with connections to psychology and
neuroscience summarized in Chapters 14 and 15.
Finally, reinforcement learning is also part of a larger trend in artiﬁcial intelligence

back toward simple general principles. Since the late 1960’s, many artiﬁcial intelligence
researchers presumed that there are no general principles to be discovered, that intelligence
is instead due to the possession of a vast number of special purpose tricks, procedures,
and heuristics. It was sometimes said that if we could just get enough relevant facts into a
machine, say one million, or one billion, then it would become intelligent. Methods based

on general principles, such as search or learning, were characterized as “weak methods,”
whereas those based on speciﬁc knowledge were called “strong methods.” This view is
still common today, but not dominant. From our point of view, it was simply premature:
too little e↵ort had been put into the search for general principles to conclude that there
were none. Modern artiﬁcial intelligence now includes much research looking for general
principles of learning, search, and decision making. It is not clear how far back the

pendulum will swing, but reinforcement learning research is certainly part of the swing
back toward simpler and fewer general principles of artiﬁcial intelligence.
1.2 Examples
A good way to understand reinforcement learning is to consider some of the examples
and possible applications that have guided its development.
•A master chess player makes a move. The choice is informed both by planning—
anticipating possible replies and counterreplies—and by immediate, intuitive judg-
ments of the desirability of particular positions and moves.

•An adaptive controller adjusts parameters of a petroleum reﬁnery’s operation in
real time. The controller optimizes the yield/cost/quality trade-o↵ on the basis
of speciﬁed marginal costs without sticking strictly to the set points originally
suggested by engineers.

1.3. Elements of Reinforcement Learning 5
•A gazelle calf struggles to its feet minutes after being born. Half an hour later it is
running at 20 miles per hour.
•A mobile robot decides whether it should enter a new room in search of more trash
to collect or start trying to ﬁnd its way back to its battery recharging station. It
makes its decision based on the current charge level of its battery and how quickly
and easily it has been able to ﬁnd the recharger in the past.
•Phil prepares his breakfast. Closely examined, even this apparently mundane
activity reveals a complex web of conditional behavior and interlocking goal–subgoal
relationships: walking to the cupboard, opening it, selecting a cereal box, then
reaching for, grasping, and retrieving the box. Other complex, tuned, interactive
sequences of behavior are required to obtain a bowl, spoon, and milk carton. Each
step involves a series of eye movements to obtain information and to guide reaching

and locomotion. Rapid judgments are continually made about how to carry the
objects or whether it is better to ferry some of them to the dining table before
obtaining others. Each step is guided by goals, such as grasping a spoon or getting
to the refrigerator, and is in service of other goals, such as having the spoon to eat
with once the cereal is prepared and ultimately obtaining nourishment. Whether
he is aware of it or not, Phil is accessing information about the state of his body

that determines his nutritional needs, level of hunger, and food preferences.
These examples share features that are so basic that they are easy to overlook. All
involve interaction between an active decision-making agent and its environment, within
which the agent seeks to achieve a goal despite uncertainty about its environment. The
agent’s actions are permitted to a↵ect the future state of the environment (e.g., the
next chess position, the level of reservoirs of the reﬁnery, the robot’s next location and

the future charge level of its battery), thereby a↵ecting the actions and opportunities
available to the agent at later times. Correct choice requires taking into account indirect,
delayed consequences of actions, and thus may require foresight or planning.
At the same time, in all of these examples the e↵ects of actions cannot be fully predicted;
thus the agent must monitor its environment frequently and react appropriately. For
example, Phil must watch the milk he pours into his cereal bowl to keep it from overﬂowing.

All these examples involve goals that are explicit in the sense that the agent can judge
progress toward its goal based on what it can sense directly. The chess player knows
whether or not he wins, the reﬁnery controller knows how much petroleum is being
produced, the gazelle calf knows when it falls, the mobile robot knows when its batteries
run down, and Phil knows whether or not he is enjoying his breakfast.
In all of these examples the agent can use its experience to improve its performance

over time. The chess player reﬁnes the intuition he uses to evaluate positions, thereby
improving his play; the gazelle calf improves the eciency with which it can run; Phil
learns to streamline making his breakfast. The knowledge the agent brings to the task at
the start—either from previous experience with related tasks or built into it by design or
evolution—inﬂuences what is useful or easy to learn, but interaction with the environment
is essential for adjusting behavior to exploit speciﬁc features of the task.

6C h a p t e r 1 : I n t r o d u c t i o n
1.3 Elements of Reinforcement Learning
Beyond the agent and the environment, one can identify four main subelements of a
reinforcement learning system: a policy ,areward signal ,avalue function , and, optionally,
amodel of the environment.
Apolicy deﬁnes the learning agent’s way of behaving at a given time. Roughly speaking,
a policy is a mapping from perceived states of the environment to actions to be taken
when in those states. It corresponds to what in psychology would be called a set of
stimulus–response rules or associations. In some cases the policy may be a simple function
or lookup table, whereas in others it may involve extensive computation such as a search
process. The policy is the core of a reinforcement learning agent in the sense that it alone
is sucient to determine behavior. In general, policies may be stochastic, specifying
probabilities for each action.
Areward signal deﬁnes the goal of a reinforcement learning problem. On each time

step, the environment sends to the reinforcement learning agent a single number called
thereward . The agent’s sole objective is to maximize the total reward it receives over
the long run. The reward signal thus deﬁnes what are the good and bad events for the
agent. In a biological system, we might think of rewards as analogous to the experiences
of pleasure or pain. They are the immediate and deﬁning features of the problem faced
by the agent. The reward signal is the primary basis for altering the policy; if an action

selected by the policy is followed by low reward, then the policy may be changed to
select some other action in that situation in the future. In general, reward signals may
be stochastic functions of the state of the environment and the actions taken.
Whereas the reward signal indicates what is good in an immediate sense, a value
function speciﬁes what is good in the long run. Roughly speaking, the value of a state is
the total amount of reward an agent can expect to accumulate over the future, starting

from that state. Whereas rewards determine the immediate, intrinsic desirability of
environmental states, values indicate the long-term desirability of states after taking into
account the states that are likely to follow and the rewards available in those states. For
example, a state might always yield a low immediate reward but still have a high value
because it is regularly followed by other states that yield high rewards. Or the reverse

could be true. To make a human analogy, rewards are somewhat like pleasure (if high)
and pain (if low), whereas values correspond to a more reﬁned and farsighted judgment
of how pleased or displeased we are that our environment is in a particular state.
Rewards are in a sense primary, whereas values, as predictions of rewards, are secondary.
Without rewards there could be no values, and the only purpose of estimating values is to
achieve more reward. Nevertheless, it is values with which we are most concerned when

making and evaluating decisions. Action choices are made based on value judgments. We
seek actions that bring about states of highest value, not highest reward, because these
actions obtain the greatest amount of reward for us over the long run. Unfortunately, it
is much harder to determine values than it is to determine rewards. Rewards are basically
given directly by the environment, but values must be estimated and re-estimated from
the sequences of observations an agent makes over its entire lifetime. In fact, the most

important component of almost all reinforcement learning algorithms we consider is a

1.4. Limitations and Scope 7
method for eciently estimating values. The central role of value estimation is arguably
the most important thing that has been learned about reinforcement learning over the
last six decades.
The fourth and ﬁnal element of some reinforcement learning systems is a model of
the environment. This is something that mimics the behavior of the environment, or
more generally, that allows inferences to be made about how the environment will behave.
For example, given a state and action, the model might predict the resultant next state
and next reward. Models are used for planning , by which we mean any way of deciding
on a course of action by considering possible future situations before they are actually
experienced. Methods for solving reinforcement learning problems that use models and
planning are called model-based methods, as opposed to simpler model-free methods that
are explicitly trial-and-error learners—viewed as almost the opposite of planning. In

Chapter 8 we explore reinforcement learning systems that simultaneously learn by trial
and error, learn a model of the environment, and use the model for planning. Modern
reinforcement learning spans the spectrum from low-level, trial-and-error learning to
high-level, deliberative planning.
1.4 Limitations and Scope
Reinforcement learning relies heavily on the concept of state—as input to the policy and
value function, and as both input to and output from the model. Informally, we can
think of the state as a signal conveying to the agent some sense of “how the environment

is” at a particular time. The formal deﬁnition of state as we use it here is given by
the framework of Markov decision processes presented in Chapter 3. More generally,
however, we encourage the reader to follow the informal meaning and think of the state
as whatever information is available to the agent about its environment. In e↵ect, we
assume that the state signal is produced by some preprocessing system that is nominally
part of the agent’s environment. We do not address the issues of constructing, changing,

or learning the state signal in this book (other than brieﬂy in Section 17.3). We take this
approach not because we consider state representation to be unimportant, but in order
to focus fully on the decision-making issues. In other words, our concern in this book is
not with designing the state signal, but with deciding what action to take as a function
of whatever state signal is available.
Most of the reinforcement learning methods we consider in this book are structured
around estimating value functions, but it is not strictly necessary to do this to solve

reinforcement learning problems. For example, solution methods such as genetic algo-
rithms, genetic programming, simulated annealing, and other optimization methods never
estimate value functions. These methods apply multiple static policies each interacting
over an extended period of time with a separate instance of the environment. The policies
that obtain the most reward, and random variations of them, are carried over to the
next generation of policies, and the process repeats. We call these evolutionary methods

because their operation is analogous to the way biological evolution produces organisms
with skilled behavior even if they do not learn during their individual lifetimes. If the
space of policies is suciently small, or can be structured so that good policies are

8C h a p t e r 1 : I n t r o d u c t i o n
common or easy to ﬁnd—or if a lot of time is available for the search—then evolutionary
methods can be e↵ective. In addition, evolutionary methods have advantages on problems
in which the learning agent cannot sense the complete state of its environment.
Our focus is on reinforcement learning methods that learn while interacting with the
environment, which evolutionary methods do not do. Methods able to take advantage
of the details of individual behavioral interactions can be much more ecient than
evolutionary methods in many cases. Evolutionary methods ignore much of the useful
structure of the reinforcement learning problem: they do not use the fact that the policy
they are searching for is a function from states to actions; they do not notice which states
an individual passes through during its lifetime, or which actions it selects. In some cases
this information can be misleading (e.g., when states are misperceived), but more often it

should enable more ecient search. Although evolution and learning share many features
and naturally work together, we do not consider evolutionary methods by themselves to
be especially well suited to reinforcement learning problems and, accordingly, we do not
cover them in this book.
1.5 An Extended Example: Tic-Tac-Toe
To illustrate the general idea of reinforcement learning and contrast it with other ap-
proaches, we next consider a single example in more detail.
XXXOOXOConsider the familiar child’s game of tic-tac-toe. Two players

take turns playing on a three-by-three board. One player plays
Xs and the other Os until one player wins by placing three marks
in a row, horizontally, vertically, or diagonally, as the X player
has in the game shown to the right. If the board ﬁlls up with
neither player getting three in a row, then the game is a draw.
Because a skilled player can play so as never to lose, let us assume
that we are playing against an imperfect player, one whose play
is sometimes incorrect and allows us to win. For the moment, in

fact, let us consider draws and losses to be equally bad for us. How might we construct a
player that will ﬁnd the imperfections in its opponent’s play and learn to maximize its
chances of winning?
Although this is a simple problem, it cannot readily be solved in a satisfactory way
through classical techniques. For example, the classical “minimax” solution from game
theory is not correct here because it assumes a particular way of playing by the opponent.
For example, a minimax player would never reach a game state from which it could

lose, even if in fact it always won from that state because of incorrect play by the
opponent. Classical optimization methods for sequential decision problems, such as
dynamic programming, can compute an optimal solution for any opponent, but require
as input a complete speciﬁcation of that opponent, including the probabilities with which
the opponent makes each move in each board state. Let us assume that this information
is not available a priori for this problem, as it is not for the vast majority of problems of

practical interest. On the other hand, such information can be estimated from experience,
in this case by playing many games against the opponent. About the best one can do

1.5. An Extended Example: Tic-Tac-Toe 9
on this problem is ﬁrst to learn a model of the opponent’s behavior, up to some level of
conﬁdence, and then apply dynamic programming to compute an optimal solution given
the approximate opponent model. In the end, this is not that di↵erent from some of the
reinforcement learning methods we examine later in this book.
An evolutionary method applied to this problem would directly search the space
of possible policies for one with a high probability of winning against the opponent.
Here, a policy is a rule that tells the player what move to make for every state of the
game—every possible conﬁguration of Xs and Os on the three-by-three board. For each
policy considered, an estimate of its winning probability would be obtained by playing
some number of games against the opponent. This evaluation would then direct which
policy or policies were considered next. A typical evolutionary method would hill-climb

in policy space, successively generating and evaluating policies in an attempt to obtain
incremental improvements. Or, perhaps, a genetic-style algorithm could be used that
would maintain and evaluate a population of policies. Literally hundreds of di↵erent
optimization methods could be applied.
Here is how the tic-tac-toe problem would be approached with a method making use
of a value function. First we would set up a table of numbers, one for each possible state
of the game. Each number will be the latest estimate of the probability of our winning

from that state. We treat this estimate as the state’s value , and the whole table is the
learned value function. State Ahas higher value than state B, or is considered “better”
than state B, if the current estimate of the probability of our winning from Ais higher
than it is from B. Assuming we always play Xs, then for all states with three Xs in a row
the probability of winning is 1, because we have already won. Similarly, for all states
with three Os in a row, or that are ﬁlled up, the correct probability is 0, as we cannot

win from them. We set the initial values of all the other states to 0.5, representing a
guess that we have a 50% chance of winning.
We then play many games against the opponent. To select our moves we examine the
states that would result from each of our possible moves (one for each blank space on the
board) and look up their current values in the table. Most of the time we move greedily ,
selecting the move that leads to the state with greatest value, that is, with the highest
estimated probability of winning. Occasionally, however, we select randomly from among

the other moves instead. These are called exploratory moves because they cause us to
experience states that we might otherwise never see. A sequence of moves made and
considered during a game can be diagrammed as in Figure 1.1.
While we are playing, we change the values of the states in which we ﬁnd ourselves
during the game. We attempt to make them more accurate estimates of the probabilities
of winning. To do this, we “back up” the value of the state after each greedy move to

the state before the move, as suggested by the arrows in Figure 1.1. More precisely, the
current value of the earlier state is updated to be closer to the value of the later state.
This can be done by moving the earlier state’s value a fraction of the way toward the
value of the later state. If we let Stdenote the state before the greedy move, and St+1
the state after the move, then the update to the estimated value of St, denoted V(St),
V(St) V(St)+↵h
V(St+1)V(St)i
,

10 Chapter 1: Introduction
..•our move{opponent's move{our move{starting position•••abc*dee*opponent's move{c•f•g*gopponent's move{our move{.•ag*cstarting positionbc*de*efg…Figure 1.1: A sequence of tic-tac-toe moves. The solid black lines represent the moves taken
during a game; the dashed lines represent moves that we (our reinforcement learning player)
considered but did not make. Our second move was an exploratory move, meaning that it was
taken even though another sibling move, the one leading to e⇤, was ranked higher. Exploratory
moves do not result in any learning, but each of our other moves does, causing updates as
suggested by the red arrows in which estimated values are moved up the tree from later nodes
to earlier nodes as detailed in the text.
where ↵is a small positive fraction called the step-size parameter ,w h i c hi n ﬂ u e n c e s
the rate of learning. This update rule is an example of a temporal-di↵erence learning

method, so called because its changes are based on a di↵erence, V(St+1)V(St), between
estimates at two successive times.
The method described above performs quite well on this task. For example, if the
step-size parameter is reduced properly over time, then this method converges, for any
ﬁxed opponent, to the true probabilities of winning from each state given optimal play
by our player. Furthermore, the moves then taken (except on exploratory moves) are in

fact the optimal moves against this (imperfect) opponent. In other words, the method
converges to an optimal policy for playing the game against this opponent. If the step-size
parameter is not reduced all the way to zero over time, then this player also plays well
against opponents that slowly change their way of playing.
This example illustrates the di↵erences between evolutionary methods and methods
that learn value functions. To evaluate a policy an evolutionary method holds the policy

ﬁxed and plays many games against the opponent, or simulates many games using a model
of the opponent. The frequency of wins gives an unbiased estimate of the probability

1.5. An Extended Example: Tic-Tac-Toe 11
of winning with that policy, and can be used to direct the next policy selection. But
each policy change is made only after many games, and only the ﬁnal outcome of each
game is used: what happens during the games is ignored. For example, if the player wins,
then allof its behavior in the game is given credit, independently of how speciﬁc moves
might have been critical to the win. Credit is even given to moves that never occurred!
Value function methods, in contrast, allow individual states to be evaluated. In the end,
evolutionary and value function methods both search the space of policies, but learning a
value function takes advantage of information available during the course of play.
This simple example illustrates some of the key features of reinforcement learning
methods. First, there is the emphasis on learning while interacting with an environment,
in this case with an opponent player. Second, there is a clear goal, and correct behavior

requires planning or foresight that takes into account delayed e↵ects of one’s choices. For
example, the simple reinforcement learning player would learn to set up multi-move traps
for a shortsighted opponent. It is a striking feature of the reinforcement learning solution
that it can achieve the e↵ects of planning and lookahead without using a model of the
opponent and without conducting an explicit search over possible sequences of future
states and actions.
While this example illustrates some of the key features of reinforcement learning, it is

so simple that it might give the impression that reinforcement learning is more limited
than it really is. Although tic-tac-toe is a two-person game, reinforcement learning
also applies in the case in which there is no external adversary, that is, in the case of
a “game against nature.” Reinforcement learning also is not restricted to problems in
which behavior breaks down into separate episodes, like the separate games of tic-tac-toe,
with reward only at the end of each episode. It is just as applicable when behavior

continues indeﬁnitely and when rewards of various magnitudes can be received at any
time. Reinforcement learning is also applicable to problems that do not even break down
into discrete time steps like the plays of tic-tac-toe. The general principles apply to
continuous-time problems as well, although the theory gets more complicated and we
omit it from this introductory treatment.
Tic-tac-toe has a relatively small, ﬁnite state set, whereas reinforcement learning can
be used when the state set is very large, or even inﬁnite. For example, Gerry Tesauro

(1992, 1995) combined the algorithm described above with an artiﬁcial neural network
to learn to play backgammon, which has approximately 1020states. With this many
states it is impossible ever to experience more than a small fraction of them. Tesauro’s
program learned to play far better than any previous program and eventually better than
the world’s best human players (Section 16.1). The artiﬁcial neural network provides the
program with the ability to generalize from its experience, so that in new states it selects

moves based on information saved from similar states faced in the past, as determined
by its network. How well a reinforcement learning system can work in problems with
such large state sets is intimately tied to how appropriately it can generalize from past
experience. It is in this role that we have the greatest need for supervised learning methods
with reinforcement learning. Artiﬁcial neural networks and deep learning (Section 9.6)
are not the only, or necessarily the best, way to do this.
In this tic-tac-toe example, learning started with no prior knowledge beyond the

12 Chapter 1: Introduction
rules of the game, but reinforcement learning by no means entails a tabula rasa view of
learning and intelligence. On the contrary, prior information can be incorporated into
reinforcement learning in a variety of ways that can be critical for ecient learning (e.g.,
see Sections 9.5, 17.4, and 13.1). We also have access to the true state in the tic-tac-toe
example, whereas reinforcement learning can also be applied when part of the state is
hidden, or when di↵erent states appear to the learner to be the same.
Finally, the tic-tac-toe player was able to look ahead and know the states that would
result from each of its possible moves. To do this, it had to have a model of the game
that allowed it to foresee how its environment would change in response to moves that it
might never make. Many problems are like this, but in others even a short-term model
of the e↵ects of actions is lacking. Reinforcement learning can be applied in either case.

A model is not required, but models can easily be used if they are available or can be
learned (Chapter 8).
On the other hand, there are reinforcement learning methods that do not need any
kind of environment model at all. Model-free systems cannot even think about how
their environments will change in response to a single action. The tic-tac-toe player is
model-free in this sense with respect to its opponent: it has no model of its opponent

of any kind. Because models have to be reasonably accurate to be useful, model-free
methods can have advantages over more complex methods when the real bottleneck in
solving a problem is the diculty of constructing a suciently accurate environment
model. Model-free methods are also important building blocks for model-based methods.
In this book we devote several chapters to model-free methods before we discuss how
they can be used as components of more complex model-based methods.
Reinforcement learning can be used at both high and low levels in a system. Although

the tic-tac-toe player learned only about the basic moves of the game, nothing prevents
reinforcement learning from working at higher levels where each of the “actions” may
itself be the application of a possibly elaborate problem-solving method. In hierarchical
learning systems, reinforcement learning can work simultaneously on several levels.
Exercise 1.1: Self-Play Suppose, instead of playing against a random opponent, the
reinforcement learning algorithm described above played against itself, with both sides

learning. What do you think would happen in this case? Would it learn a di↵erent policy
for selecting moves? ⇤
Exercise 1.2: Symmetries Many tic-tac-toe positions appear di↵erent but are really
the same because of symmetries. How might we amend the learning process described
above to take advantage of this? In what ways would this change improve the learning
process? Now think again. Suppose the opponent did not take advantage of symmetries.
In that case, should we? Is it true, then, that symmetrically equivalent positions should
necessarily have the same value? ⇤

Exercise 1.3: Greedy Play Suppose the reinforcement learning player was greedy , that is,
it always played the move that brought it to the position that it rated the best. Might it
learn to play better, or worse, than a nongreedy player? What problems might occur? ⇤
Exercise 1.4: Learning from Exploration Suppose learning updates occurred after all
moves, including exploratory moves. If the step-size parameter is appropriately reduced

1.7. Early History of Reinforcement Learning 13
over time (but not the tendency to explore), then the state values would converge to
a di↵erent set of probabilities. What (conceptually) are the two sets of probabilities
computed when we do, and when we do not, learn from exploratory moves? Assuming
that we do continue to make exploratory moves, which set of probabilities might be better
to learn? Which would result in more wins? ⇤
Exercise 1.5: Other Improvements Can you think of other ways to improve the reinforce-
ment learning player? Can you think of any better way to solve the tic-tac-toe problem
as posed? ⇤
1.6 Summary
Reinforcement learning is a computational approach to understanding and automating
goal-directed learning and decision making. It is distinguished from other computational
approaches by its emphasis on learning by an agent from direct interaction with its
environment, without requiring exemplary supervision or complete models of the envi-

as posed? ⇤
ronment. In our opinion, reinforcement learning is the ﬁrst ﬁeld to seriously address the
computational issues that arise when learning from interaction with an environment in
order to achieve long-term goals.
Reinforcement learning uses the formal framework of Markov decision processes to
deﬁne the interaction between a learning agent and its environment in terms of states,
actions, and rewards. This framework is intended to be a simple way of representing
essential features of the artiﬁcial intelligence problem. These features include a sense of

cause and e↵ect, a sense of uncertainty and nondeterminism, and the existence of explicit
goals.
The concepts of value and value function are key to most of the reinforcement learning
methods that we consider in this book. We take the position that value functions
are important for ecient search in the space of policies. The use of value functions
distinguishes reinforcement learning methods from evolutionary methods that search
directly in policy space guided by evaluations of entire policies.
1.7 Early History of Reinforcement Learning

goals.
The early history of reinforcement learning has two main threads, both long and rich, that
were pursued independently before intertwining in modern reinforcement learning. One
thread concerns learning by trial and error, and originated in the psychology of animal
learning. This thread runs through some of the earliest work in artiﬁcial intelligence
and led to the revival of reinforcement learning in the early 1980s. The second thread
concerns the problem of optimal control and its solution using value functions and

dynamic programming. For the most part, this thread did not involve learning. The
two threads were mostly independent, but became interrelated to some extent around a
third, less distinct thread concerning temporal-di↵erence methods such as that used in
the tic-tac-toe example in this chapter. All three threads came together in the late 1980s
to produce the modern ﬁeld of reinforcement learning as we present it in this book.

14 Chapter 1: Introduction
The thread focusing on trial-and-error learning is the one with which we are most
familiar and about which we have the most to say in this brief history. Before doing that,
however, we brieﬂy discuss the optimal control thread.
The term “optimal control” came into use in the late 1950s to describe the problem of
designing a controller to minimize or maximize a measure of a dynamical system’s behavior
over time. One of the approaches to this problem was developed in the mid-1950s by
Richard Bellman and others through extending a nineteenth century theory of Hamilton
and Jacobi. This approach uses the concepts of a dynamical system’s state and of a
value function, or “optimal return function,” to deﬁne a functional equation, now often
called the Bellman equation. The class of methods for solving optimal control problems
by solving this equation came to be known as dynamic programming (Bellman, 1957a).

Bellman (1957b) also introduced the discrete stochastic version of the optimal control
problem known as Markov decision processes (MDPs). Ronald Howard (1960) devised
the policy iteration method for MDPs. All of these are essential elements underlying the
theory and algorithms of modern reinforcement learning.
Dynamic programming is widely considered the only feasible way of solving general
stochastic optimal control problems. It su↵ers from what Bellman called “the curse of

dimensionality,” meaning that its computational requirements grow exponentially with the
number of state variables, but it is still far more ecient and more widely applicable than
any other general method. Dynamic programming has been extensively developed since
the late 1950s, including extensions to partially observable MDPs (surveyed by Lovejoy,
1991), many applications (surveyed by White, 1985, 1988, 1993), approximation methods
(surveyed by Rust, 1996), and asynchronous methods (Bertsekas, 1982, 1983). Many

excellent modern treatments of dynamic programming are available (e.g., Bertsekas, 2005,
2012; Puterman, 1994; Ross, 1983; and Whittle, 1982, 1983). Bryson (1996) provides an
authoritative history of optimal control.
Connections between optimal control and dynamic programming, on the one hand,
and learning, on the other, were slow to be recognized. We cannot be sure about what
accounted for this separation, but its main cause was likely the separation between
the disciplines involved and their di↵erent goals. Also contributing may have been the

prevalent view of dynamic programming as an o✏ine computation depending essentially
on accurate system models and analytic solutions to the Bellman equation. Further,
the simplest form of dynamic programming is a computation that proceeds backwards
in time, making it dicult to see how it could be involved in a learning process that
must proceed in a forward direction. Some of the earliest work in dynamic programming,
such as that by Bellman and Dreyfus (1959), might now be classiﬁed as following

a learning approach. Witten’s (1977) work (discussed below) certainly qualiﬁes as a
combination of learning and dynamic-programming ideas. Werbos (1987) argued explicitly
for greater interrelation of dynamic programming and learning methods and for dynamic
programming’s relevance to understanding neural and cognitive mechanisms. For us the
full integration of dynamic programming methods with online learning did not occur
until the work of Chris Watkins in 1989, whose treatment of reinforcement learning

using the MDP formalism has been widely adopted. Since then these relationships have
been extensively developed by many researchers, most particularly by Dimitri Bertsekas

1.7. Early History of Reinforcement Learning 15
and John Tsitsiklis (1996), who coined the term “neurodynamic programming” to refer
to the combination of dynamic programming and artiﬁcial neural networks. Another
term currently in use is “approximate dynamic programming.” These various approaches
emphasize di↵erent aspects of the subject, but they all share with reinforcement learning
an interest in circumventing the classical shortcomings of dynamic programming.
We consider all of the work in optimal control also to be, in a sense, work in reinforce-
ment learning. We deﬁne a reinforcement learning method as any e↵ective way of solving
reinforcement learning problems, and it is now clear that these problems are closely
related to optimal control problems, particularly stochastic optimal control problems
such as those formulated as MDPs. Accordingly, we must consider the solution methods
of optimal control, such as dynamic programming, also to be reinforcement learning

methods. Because almost all of the conventional methods require complete knowledge
of the system to be controlled, it feels a little unnatural to say that they are part of
reinforcement learning . On the other hand, many dynamic programming algorithms are
incremental and iterative. Like learning methods, they gradually reach the correct answer
through successive approximations. As we show in the rest of this book, these similarities
are far more than superﬁcial. The theories and solution methods for the cases of complete

and incomplete knowledge are so closely related that we feel they must be considered
together as part of the same subject matter.
Let us return now to the other major thread leading to the modern ﬁeld of reinforcement
learning, the thread centered on the idea of trial-and-error learning. We only touch on
the major points of contact here, taking up this topic in more detail in Section 14.3.
According to American psychologist R. S. Woodworth (1938) the idea of trial-and-error
learning goes as far back as the 1850s to Alexander Bain’s discussion of learning by

“groping and experiment” and more explicitly to the British ethologist and psychologist
Conway Lloyd Morgan’s 1894 use of the term to describe his observations of animal
behavior. Perhaps the ﬁrst to succinctly express the essence of trial-and-error learning as
a principle of learning was Edward Thorndike:
Of several responses made to the same situation, those which are accompanied
or closely followed by satisfaction to the animal will, other things being
equal, be more ﬁrmly connected with the situation, so that, when it recurs,

they will be more likely to recur; those which are accompanied or closely
followed by discomfort to the animal will, other things being equal, have their
connections with that situation weakened, so that, when it recurs, they will
be less likely to occur. The greater the satisfaction or discomfort, the greater
the strengthening or weakening of the bond. (Thorndike, 1911, p. 244)
Thorndike called this the “Law of E↵ect” because it describes the e↵ect of reinforcing
events on the tendency to select actions. Thorndike later modiﬁed the law to better

account for subsequent data on animal learning (such as di↵erences between the e↵ects
of reward and punishment), and the law in its various forms has generated considerable
controversy among learning theorists (e.g., see Gallistel, 2005; Herrnstein, 1970; Kimble,
1961, 1967; Mazur, 1994). Despite this, the Law of E↵ect—in one form or another—is
widely regarded as a basic principle underlying much behavior (e.g., Hilgard and Bower,
1975; Dennett, 1978; Campbell, 1960; Cziko, 1995). It is the basis of the inﬂuential

16 Chapter 1: Introduction
learning theories of Clark Hull (1943, 1952) and the inﬂuential experimental methods of
B. F. Skinner (1938).
The term “reinforcement” in the context of animal learning came into use well after
Thorndike’s expression of the Law of E↵ect, ﬁrst appearing in this context (to the best of
our knowledge) in the 1927 English translation of Pavlov’s monograph on conditioned
reﬂexes. Pavlov described reinforcement as the strengthening of a pattern of behavior due
to an animal receiving a stimulus—a reinforcer—in an appropriate temporal relationship
with another stimulus or with a response. Some psychologists extended the idea of
reinforcement to include weakening as well as strengthening of behavior, and extended
the idea of a reinforcer to include possibly the omission or termination of stimulus. To
be considered reinforcer, the strengthening or weakening must persist after the reinforcer
is withdrawn; a stimulus that merely attracts an animal’s attention or that energizes its

behavior without producing lasting changes would not be considered a reinforcer.
The idea of implementing trial-and-error learning in a computer appeared among the
earliest thoughts about the possibility of artiﬁcial intelligence. In a 1948 report, Alan
Turing described a design for a “pleasure-pain system” that worked along the lines of the
Law of E↵ect:
When a conﬁguration is reached for which the action is undetermined, a
random choice for the missing data is made and the appropriate entry is made
in the description, tentatively, and is applied. When a pain stimulus occurs

all tentative entries are cancelled, and when a pleasure stimulus occurs they
are all made permanent. (Turing, 1948)
Many ingenious electro-mechanical machines were constructed that demonstrated trial-
and-error learning. The earliest may have been a machine built by Thomas Ross (1933)
that was able to ﬁnd its way through a simple maze and remember the path through
the settings of switches. In 1951 W. Grey Walter built a version of his “mechanical

tortoise” (Walter, 1950) capable of a simple form of learning. In 1952 Claude Shannon
demonstrated a maze-running mouse named Theseus that used trial and error to ﬁnd
its way through a maze, with the maze itself remembering the successful directions
via magnets and relays under its ﬂoor (see also Shannon, 1951). J. A. Deutsch (1954)
described a maze-solving machine based on his behavior theory (Deutsch, 1953) that
has some properties in common with model-based reinforcement learning (Chapter 8).

In his Ph.D. dissertation, Marvin Minsky (1954) discussed computational models of
reinforcement learning and described his construction of an analog machine composed of
components he called SNARCs (Stochastic Neural-Analog Reinforcement Calculators)
meant to resemble modiﬁable synaptic connections in the brain (Chapter 15). The
web site cyberneticzoo.com contains a wealth of information on these and many other
electro-mechanical learning machines.

Building electro-mechanical learning machines gave way to programming digital com-
puters to perform various types of learning, some of which implemented trial-and-error
learning. Farley and Clark (1954) described a digital simulation of a neural-network
learning machine that learned by trial and error. But their interests soon shifted from
trial-and-error learning to generalization and pattern recognition, that is, from reinforce-
ment learning to supervised learning (Clark and Farley, 1955). This began a pattern

1.7. Early History of Reinforcement Learning 17
of confusion about the relationship between these types of learning. Many researchers
seemed to believe that they were studying reinforcement learning when they were actually
studying supervised learning. For example, artiﬁcial neural network pioneers such as
Rosenblatt (1962) and Widrow and Ho↵ (1960) were clearly motivated by reinforcement
learning—they used the language of rewards and punishments—but the systems they
studied were supervised learning systems suitable for pattern recognition and perceptual
learning. Even today, some researchers and textbooks minimize or blur the distinction
between these types of learning. For example, some artiﬁcial neural network textbooks
have used the term “trial-and-error” to describe networks that learn from training exam-
ples. This is an understandable confusion because these networks use error information
to update connection weights, but this misses the essential character of trial-and-error

learning as selecting actions on the basis of evaluative feedback that does not rely on
knowledge of what the correct action should be.
Partly as a result of these confusions, research into genuine trial-and-error learning
became rare in the 1960s and 1970s, although there were notable exceptions. In the 1960s
the terms “reinforcement” and “reinforcement learning” were used in the engineering
literature for the ﬁrst time to describe engineering uses of trial-and-error learning (e.g.,
Waltz and Fu, 1965; Mendel, 1966; Fu, 1970; Mendel and McClaren, 1970). Particularly

inﬂuential was Minsky’s paper “Steps Toward Artiﬁcial Intelligence” (Minsky, 1961),
which discussed several issues relevant to trial-and-error learning, including prediction,
expectation, and what he called the basic credit-assignment problem for complex rein-
forcement learning systems : How do you distribute credit for success among the many
decisions that may have been involved in producing it? All of the methods we discuss in
this book are, in a sense, directed toward solving this problem. Minsky’s paper is well

worth reading today.
In the next few paragraphs we discuss some of the other exceptions and partial
exceptions to the relative neglect of computational and theoretical study of genuine
trial-and-error learning in the 1960s and 1970s.
One exception was the work of the New Zealand researcher John Andreae, who
developed a system called STeLLA that learned by trial and error in interaction with
its environment. This system included an internal model of the world and, later, an
“internal monologue” to deal with problems of hidden state (Andreae, 1963, 1969a,b).

Andreae’s later work (1977) placed more emphasis on learning from a teacher, but still
included learning by trial and error, with the generation of novel events being one of
the system’s goals. A feature of this work was a “leakback process,” elaborated more
fully in Andreae (1998), that implemented a credit-assignment mechanism similar to the
backing-up update operations that we describe. Unfortunately, his pioneering research
was not well known and did not greatly impact subsequent reinforcement learning research.

Recent summaries are available (Andreae, 2017a,b).
More inﬂuential was the work of Donald Michie. In 1961 and 1963 he described a
simple trial-and-error learning system for learning how to play tic-tac-toe (or naughts
and crosses) called MENACE (for Matchbox Educable Naughts and Crosses Engine). It
consisted of a matchbox for each possible game position, each matchbox containing a
number of colored beads, a di↵erent color for each possible move from that position. By

18 Chapter 1: Introduction
drawing a bead at random from the matchbox corresponding to the current game position,
one could determine MENACE’s move. When a game was over, beads were added to
or removed from the boxes used during play to reward or punish MENACE’s decisions.
Michie and Chambers (1968) described another tic-tac-toe reinforcement learner called
GLEE (Game Learning Expectimaxing Engine) and a reinforcement learning controller
called BOXES. They applied BOXES to the task of learning to balance a pole hinged to
a movable cart on the basis of a failure signal occurring only when the pole fell or the
cart reached the end of a track. This task was adapted from the earlier work of Widrow
and Smith (1964), who used supervised learning methods, assuming instruction from a
teacher already able to balance the pole. Michie and Chambers’s version of pole-balancing
is one of the best early examples of a reinforcement learning task under conditions of

incomplete knowledge. It inﬂuenced much later work in reinforcement learning, beginning
with some of our own studies (Barto, Sutton, and Anderson, 1983; Sutton, 1984). Michie
consistently emphasized the role of trial and error and learning as essential aspects of
artiﬁcial intelligence (Michie, 1974).
Widrow, Gupta, and Maitra (1973) modiﬁed the Least-Mean-Square (LMS) algorithm
of Widrow and Ho↵ (1960) to produce a reinforcement learning rule that could learn
from success and failure signals instead of from training examples. They called this form

of learning “selective bootstrap adaptation” and described it as “learning with a critic”
instead of “learning with a teacher.” They analyzed this rule and showed how it could
learn to play blackjack. This was an isolated foray into reinforcement learning by Widrow,
whose contributions to supervised learning were much more inﬂuential. Our use of the
term “critic” is derived from Widrow, Gupta, and Maitra’s paper. Buchanan, Mitchell,
Smith, and Johnson (1978) independently used the term critic in the context of machine

learning (see also Dietterich and Buchanan, 1984), but for them a critic is an expert
system able to do more than evaluate performance.
Research on learning automata had a more direct inﬂuence on the trial-and-error
thread leading to modern reinforcement learning research. These are methods for solving
a nonassociative, purely selectional learning problem known as the k-armed bandit by
analogy to a slot machine, or “one-armed bandit,” except with klevers (see Chapter 2).
Learning automata are simple, low-memory machines for improving the probability

of reward in these problems. Learning automata originated with work in the 1960s
of the Russian mathematician and physicist M. L. Tsetlin and colleagues (published
posthumously in Tsetlin, 1973) and has been extensively developed since then within
engineering (see Narendra and Thathachar, 1974, 1989). These developments included the
study of stochastic learning automata , which are methods for updating action probabilities
on the basis of reward signals. Although not developed in the tradition of stochastic

learning automata, Harth and Tzanakou’s (1974) Alopex algorithm (for Algorithm of
pattern extraction) is a stochastic method for detecting correlations between actions and
reinforcement that inﬂuenced some of our early research (Barto, Sutton, and Brouwer,
1981). Stochastic learning automata were foreshadowed by earlier work in psychology,
beginning with William Estes’ (1950) e↵ort toward a statistical theory of learning and
further developed by others (e.g., Bush and Mosteller, 1955; Sternberg, 1963).

The statistical learning theories developed in psychology were adopted by researchers in

1.7. Early History of Reinforcement Learning 19
economics, leading to a thread of research in that ﬁeld devoted to reinforcement learning.
This work began in 1973 with the application of Bush and Mosteller’s learning theory to
a collection of classical economic models (Cross, 1973). One goal of this research was to
study artiﬁcial agents that act more like real people than do traditional idealized economic
agents (Arthur, 1991). This approach expanded to the study of reinforcement learning
in the context of game theory. Reinforcement learning in economics developed largely
independently of the early work in reinforcement learning in artiﬁcial intelligence, though
game theory remains a topic of interest in both ﬁelds (beyond the scope of this book).
Camerer (2011) discusses the reinforcement learning tradition in economics, and Now´ e,
Vrancx, and De Hauwere (2012) provide an overview of the subject from the point of view
of multi-agent extensions to the approach that we introduce in this book. Reinforcement

in the context of game theory is a much di↵erent subject than reinforcement learning
used in programs to play tic-tac-toe, checkers, and other recreational games. See, for
example, Szita (2012) for an overview of this aspect of reinforcement learning and games.
John Holland (1975) outlined a general theory of adaptive systems based on selectional
principles. His early work concerned trial and error primarily in its nonassociative
form, as in evolutionary methods and the k-armed bandit. In 1976 and more fully in

1986, he introduced classiﬁer systems , true reinforcement learning systems including
association and value functions. A key component of Holland’s classiﬁer systems was
the “bucket-brigade algorithm” for credit assignment, which is closely related to the
temporal di↵erence algorithm used in our tic-tac-toe example and discussed in Chapter 6.
Another key component was a genetic algorithm , an evolutionary method whose role was
to evolve useful representations. Classiﬁer systems have been extensively developed by

many researchers to form a major branch of reinforcement learning research (reviewed by
Urbanowicz and Moore, 2009), but genetic algorithms—which we do not consider to be
reinforcement learning systems by themselves—have received much more attention, as
have other approaches to evolutionary computation (e.g., Fogel, Owens and Walsh, 1966,
and Koza, 1992).
The individual most responsible for reviving the trial-and-error thread to reinforcement
learning within artiﬁcial intelligence was Harry Klopf (1972, 1975, 1982). Klopf recognized

that essential aspects of adaptive behavior were being lost as learning researchers came
to focus almost exclusively on supervised learning. What was missing, according to
Klopf, were the hedonic aspects of behavior, the drive to achieve some result from the
environment, to control the environment toward desired ends and away from undesired
ends (see Section 15.9). This is the essential idea of trial-and-error learning. Klopf’s
ideas were especially inﬂuential on the authors because our assessment of them (Barto

and Sutton, 1981a) led to our appreciation of the distinction between supervised and
reinforcement learning, and to our eventual focus on reinforcement learning. Much of
the early work that we and colleagues accomplished was directed toward showing that
reinforcement learning and supervised learning were indeed di↵erent (Barto, Sutton, and
Brouwer, 1981; Barto and Sutton, 1981b; Barto and Anandan, 1985). Other studies
showed how reinforcement learning could address important problems in artiﬁcial neural

network learning, in particular, how it could produce learning algorithms for multilayer
networks (Barto, Anderson, and Sutton, 1982; Barto and Anderson, 1985; Barto, 1985,
1986; Barto and Jordan, 1987; see Section 15.10).

20 Chapter 1: Introduction
We turn now to the third thread to the history of reinforcement learning, that concerning
temporal-di↵erence learning. Temporal-di↵erence learning methods are distinctive in
being driven by the di↵erence between temporally successive estimates of the same
quantity—for example, of the probability of winning in the tic-tac-toe example. This
thread is smaller and less distinct than the other two, but it has played a particularly
important role in the ﬁeld, in part because temporal-di↵erence methods seem to be new
and unique to reinforcement learning.
The origins of temporal-di↵erence learning are in part in animal learning psychology,
in particular, in the notion of secondary reinforcers . A secondary reinforcer is a stimulus
that has been paired with a primary reinforcer such as food or pain and, as a result, has
come to take on similar reinforcing properties. Minsky (1954) may have been the ﬁrst to

realize that this psychological principle could be important for artiﬁcial learning systems.
Arthur Samuel (1959) was the ﬁrst to propose and implement a learning method that
included temporal-di↵erence ideas, as part of his celebrated checkers-playing program
(Section 16.2).
Samuel made no reference to Minsky’s work or to possible connections to animal
learning. His inspiration apparently came from Claude Shannon’s (1950) suggestion that
a computer could be programmed to use an evaluation function to play chess, and that it

might be able to improve its play by modifying this function online. (It is possible that
these ideas of Shannon’s also inﬂuenced Bellman, but we know of no evidence for this.)
Minsky (1961) extensively discussed Samuel’s work in his “Steps” paper, suggesting the
connection to secondary reinforcement theories, both natural and artiﬁcial.
As we have discussed, in the decade following the work of Minsky and Samuel, little
computational work was done on trial-and-error learning, and apparently no computational

work at all was done on temporal-di↵erence learning. In 1972, Klopf brought trial-and-
error learning together with an important component of temporal-di↵erence learning.
Klopf was interested in principles that would scale to learning in large systems, and thus
was intrigued by notions of local reinforcement, whereby subcomponents of an overall
learning system could reinforce one another. He developed the idea of “generalized
reinforcement,” whereby every component (nominally, every neuron) views all of its
inputs in reinforcement terms: excitatory inputs as rewards and inhibitory inputs as

punishments. This is not the same idea as what we now know as temporal-di↵erence
learning, and in retrospect it is farther from it than was Samuel’s work. On the other
hand, Klopf linked the idea with trial-and-error learning and related it to the massive
empirical database of animal learning psychology.
Sutton (1978a,b,c) developed Klopf’s ideas further, particularly the links to animal
learning theories, describing learning rules driven by changes in temporally successive

predictions. He and Barto reﬁned these ideas and developed a psychological model of
classical conditioning based on temporal-di↵erence learning (Sutton and Barto, 1981a;
Barto and Sutton, 1982). There followed several other inﬂuential psychological models of
classical conditioning based on temporal-di↵erence learning (e.g., Klopf, 1988; Moore et
al., 1986; Sutton and Barto, 1987, 1990). Some neuroscience models developed at this
time are well interpreted in terms of temporal-di↵erence learning (Hawkins and Kandel,

1984; Byrne, Gingrich, and Baxter, 1990; Gelperin, Hopﬁeld, and Tank, 1985; Tesauro,

Bibliographical Remarks 21
1986; Friston et al., 1994), although in most cases there was no historical connection.
Our early work on temporal-di↵erence learning was strongly inﬂuenced by animal
learning theories and by Klopf’s work. Relationships to Minsky’s “Steps” paper and to
Samuel’s checkers players were recognized only afterward. By 1981, however, we were
fully aware of all the prior work mentioned above as part of the temporal-di↵erence and
trial-and-error threads. At this time we developed a method for using temporal-di↵erence
learning combined with trial-and-error learning, known as the actor–critic architecture ,
and applied this method to Michie and Chambers’s pole-balancing problem (Barto, Sutton,
and Anderson, 1983). This method was extensively studied in Sutton’s (1984) Ph.D.
dissertation and extended to use backpropagation neural networks in Anderson’s (1986)
Ph.D. dissertation. Around this time, Holland (1986) incorporated temporal-di↵erence

ideas explicitly into his classiﬁer systems in the form of his bucket-brigade algorithm.
A key step was taken by Sutton (1988) by separating temporal-di↵erence learning from
control, treating it as a general prediction method. That paper also introduced the TD( )
algorithm and proved some of its convergence properties.
As we were ﬁnalizing our work on the actor–critic architecture in 1981, we discovered
a paper by Ian Witten (1977, 1976a) which appears to be the earliest publication of a
temporal-di↵erence learning rule. He proposed the method that we now call tabular TD(0)

for use as part of an adaptive controller for solving MDPs. This work was ﬁrst submitted
for journal publication in 1974 and also appeared in Witten’s 1976 PhD dissertation.
Witten’s work was a descendant of Andreae’s early experiments with STeLLA and other
trial-and-error learning systems. Thus, Witten’s 1977 paper spanned both major threads
of reinforcement learning research—trial-and-error learning and optimal control—while
making a distinct early contribution to temporal-di↵erence learning.

The temporal-di↵erence and optimal control threads were fully brought together
in 1989 with Chris Watkins’s development of Q-learning. This work extended and
integrated prior work in all three threads of reinforcement learning research. Paul Werbos
(1987) contributed to this integration by arguing for the convergence of trial-and-error
learning and dynamic programming since 1977. By the time of Watkins’s work there had
been tremendous growth in reinforcement learning research, primarily in the machine

learning subﬁeld of artiﬁcial intelligence, but also in artiﬁcial neural networks and artiﬁcial
intelligence more broadly. In 1992, the remarkable success of Gerry Tesauro’s backgammon
playing program, TD-Gammon, brought additional attention to the ﬁeld.
In the time since publication of the ﬁrst edition of this book, a ﬂourishing subﬁeld of
neuroscience developed that focuses on the relationship between reinforcement learning

algorithms and reinforcement learning in the nervous system. Most responsible for this is
an uncanny similarity between the behavior of temporal-di↵erence algorithms and the
activity of dopamine producing neurons in the brain, as pointed out by a number of
researchers (Friston et al., 1994; Barto, 1995a; Houk, Adams, and Barto, 1995; Montague,
Dayan, and Sejnowski, 1996; and Schultz, Dayan, and Montague, 1997). Chapter 15
provides an introduction to this exciting aspect of reinforcement learning. Other important

contributions made in the recent history of reinforcement learning are too numerous to
mention in this brief account; we cite many of these at the end of the individual chapters
in which they arise.

22 Chapter 1: Introduction
Bibliographical Remarks
For additional general coverage of reinforcement learning, we refer the reader to the
books by Szepesv´ ari (2010), Bertsekas and Tsitsiklis (1996), Kaelbling (1993a), and
Sugiyama, Hachiya, and Morimura (2013). Books that take a control or operations research
perspective include those of Si, Barto, Powell, and Wunsch (2004), Powell (2011), Lewis
and Liu (2012), and Bertsekas (2012). Cao’s (2009) review places reinforcement learning
in the context of other approaches to learning and optimization of stochastic dynamic
systems. Three special issues of the journal Machine Learning focus on reinforcement
learning: Sutton (1992a), Kaelbling (1996), and Singh (2002). Useful surveys are provided
by Barto (1995b); Kaelbling, Littman, and Moore (1996); and Keerthi and Ravindran
(1997). The volume edited by Weiring and van Otterlo (2012) provides an excellent
overview of recent developments.

1.2 The example of Phil’s breakfast in this chapter was inspired by Agre (1988).
1.5 The temporal-di↵erence method used in the tic-tac-toe example is developed in
Chapter 6.

Part I: Tabular Solution Methods
In this part of the book we describe almost all the core ideas of reinforcement learning
algorithms in their simplest forms: that in which the state and action spaces are small
enough for the approximate value functions to be represented as arrays, or tables .I n
this case, the methods can often ﬁnd exact solutions, that is, they can often ﬁnd exactly
the optimal value function and the optimal policy. This contrasts with the approximate
methods described in the next part of the book, which only ﬁnd approximate solutions,
but which in return can be applied e↵ectively to much larger problems.
The ﬁrst chapter of this part of the book describes solution methods for the special
case of the reinforcement learning problem in which there is only a single state, called
bandit problems. The second chapter describes the general problem formulation that we
treat throughout the rest of the book—ﬁnite Markov decision processes—and its main

ideas including Bellman equations and value functions.
The next three chapters describe three fundamental classes of methods for solving ﬁnite
Markov decision problems: dynamic programming, Monte Carlo methods, and temporal-
di↵erence learning. Each class of methods has its strengths and weaknesses. Dynamic
programming methods are well developed mathematically, but require a complete and
accurate model of the environment. Monte Carlo methods don’t require a model and are

conceptually simple, but are not well suited for step-by-step incremental computation.
Finally, temporal-di↵erence methods require no model and are fully incremental, but are
more complex to analyze. The methods also di↵er in several ways with respect to their
eciency and speed of convergence.
The remaining two chapters describe how these three classes of methods can be
combined to obtain the best features of each of them. In one chapter we describe how
the strengths of Monte Carlo methods can be combined with the strengths of temporal-

di↵erence methods via multi-step bootstrapping methods. In the ﬁnal chapter of this part
of the book we show how temporal-di↵erence learning methods can be combined with
model learning and planning methods (such as dynamic programming) for a complete
and uniﬁed solution to the tabular reinforcement learning problem.
23

Chapter 2
Multi-armed Bandits
The most important feature distinguishing reinforcement learning from other types of
learning is that it uses training information that evaluates the actions taken rather
than instructs by giving correct actions. This is what creates the need for active
exploration, for an explicit search for good behavior. Purely evaluative feedback indicates
how good the action taken was, but not whether it was the best or the worst action
possible. Purely instructive feedback, on the other hand, indicates the correct action to
take, independently of the action actually taken. This kind of feedback is the basis of
supervised learning, which includes large parts of pattern classiﬁcation, artiﬁcial neural
networks, and system identiﬁcation. In their pure forms, these two kinds of feedback
are quite distinct: evaluative feedback depends entirely on the action taken, whereas
instructive feedback is independent of the action taken.

In this chapter we study the evaluative aspect of reinforcement learning in a simpliﬁed
setting, one that does not involve learning to act in more than one situation. This
nonassociative setting is the one in which most prior work involving evaluative feedback
has been done, and it avoids much of the complexity of the full reinforcement learning
problem. Studying this case enables us to see most clearly how evaluative feedback di↵ers
from, and yet can be combined with, instructive feedback.

The particular nonassociative, evaluative feedback problem that we explore is a simple
version of the k-armed bandit problem. We use this problem to introduce a number
of basic learning methods which we extend in later chapters to apply to the full rein-
forcement learning problem. At the end of this chapter, we take a step closer to the full
reinforcement learning problem by discussing what happens when the bandit problem
becomes associative, that is, when actions are taken in more than one situation.

2.1 A k-armed Bandit Problem
Consider the following learning problem. You are faced repeatedly with a choice among
kdi↵erent options, or actions. After each choice you receive a numerical reward chosen
from a stationary probability distribution that depends on the action you selected. Your
25

26 Chapter 2: Multi-armed Bandits
objective is to maximize the expected total reward over some time period, for example,
over 1000 action selections, or time steps .
This is the original form of the k-armed bandit problem , so named by analogy to a slot
machine, or “one-armed bandit,” except that it has klevers instead of one. Each action
selection is like a play of one of the slot machine’s levers, and the rewards are the payo↵s
for hitting the jackpot. Through repeated action selections you are to maximize your
winnings by concentrating your actions on the best levers. Another analogy is that of
a doctor choosing between experimental treatments for a series of seriously ill patients.
Each action is the selection of a treatment, and each reward is the survival or well-being
of the patient. Today the term “bandit problem” is sometimes used for a generalization
of the problem described above, but in this book we use it to refer just to this simple
case.

case.
In our k-armed bandit problem, each of the kactions has an expected or mean reward
given that that action is selected; let us call this the value of that action. We denote the
action selected on time step tasAt, and the corresponding reward as Rt. The value then
of an arbitrary action a, denoted q⇤(a), is the expected reward given that ais selected:
q⇤(a).=E[Rt|At=a].
If you knew the value of each action, then it would be trivial to solve the k-armed bandit
problem: you would always select the action with highest value. We assume that you do

not know the action values with certainty, although you may have estimates. We denote
the estimated value of action aat time step tasQt(a). We would like Qt(a) to be close
toq⇤(a).
If you maintain estimates of the action values, then at any time step there is at least
one action whose estimated value is greatest. We call these the greedy actions. When you
select one of these actions, we say that you are exploiting your current knowledge of the
values of the actions. If instead you select one of the nongreedy actions, then we say you

toq⇤(a).
areexploring , because this enables you to improve your estimate of the nongreedy action’s
value. Exploitation is the right thing to do to maximize the expected reward on the one
step, but exploration may produce the greater total reward in the long run. For example,
suppose a greedy action’s value is known with certainty, while several other actions are
estimated to be nearly as good but with substantial uncertainty. The uncertainty is
such that at least one of these other actions probably is actually better than the greedy

action, but you don’t know which one. If you have many time steps ahead on which
to make action selections, then it may be better to explore the nongreedy actions and
discover which of them are better than the greedy action. Reward is lower in the short
run, during exploration, but higher in the long run because after you have discovered
the better actions, you can exploit them many times. Because it is not possible both to
explore and to exploit with any single action selection, one often refers to the “conﬂict”
between exploration and exploitation.

In any speciﬁc case, whether it is better to explore or exploit depends in a complex
way on the precise values of the estimates, uncertainties, and the number of remaining
steps. There are many sophisticated methods for balancing exploration and exploitation
for particular mathematical formulations of the k-armed bandit and related problems.

2.2. Action-value Methods 27
However, most of these methods make strong assumptions about stationarity and prior
knowledge that are either violated or impossible to verify in applications and in the full
reinforcement learning problem that we consider in subsequent chapters. The guarantees
of optimality or bounded loss for these methods are of little comfort when the assumptions
of their theory do not apply.
In this book we do not worry about balancing exploration and exploitation in a
sophisticated way; we worry only about balancing them at all. In this chapter we present
several simple balancing methods for the k-armed bandit problem and show that they
work much better than methods that always exploit. The need to balance exploration
and exploitation is a distinctive challenge that arises in reinforcement learning; the
simplicity of our version of the k-armed bandit problem enables us to show this in a
particularly clear form.
2.2 Action-value Methods

We begin by looking more closely at methods for estimating the values of actions and
for using the estimates to make action selection decisions, which we collectively call
action-value methods . Recall that the true value of an action is the mean reward when
that action is selected. One natural way to estimate this is by averaging the rewards
actually received:
Qt(a).=sum of rewards when ataken prior to t
number of times ataken prior to t=Pt1
i=1Ri·
Ai=aPt1
i=1
 Ai=a, (2.1)
where

i=1Ri·
Ai=aPt1
i=1
where
 predicate denotes the random variable that is 1 if predicate is true and 0 if it is not.
If the denominator is zero, then we instead deﬁne Qt(a) as some default value, such as
0. As the denominator goes to inﬁnity, by the law of large numbers, Qt(a) converges to
q⇤(a). We call this the sample-average method for estimating action values because each
estimate is an average of the sample of relevant rewards. Of course this is just one way

i=1Ri·
Ai=aPt1
i=1
where
to estimate action values, and not necessarily the best one. Nevertheless, for now let us
stay with this simple estimation method and turn to the question of how the estimates
might be used to select actions.
The simplest action selection rule is to select one of the actions with the highest
estimated value, that is, one of the greedy actions as deﬁned in the previous section.
If there is more than one greedy action, then a selection is made among them in some

arbitrary way, perhaps randomly. We write this greedy action selection method as
At.= argmax
aQt(a), (2.2)
where argmaxadenotes the action afor which the expression that follows is maximized
(again, with ties broken arbitrarily). Greedy action selection always exploits current
knowledge to maximize immediate reward; it spends no time at all sampling apparently
inferior actions to see if they might really be better. A simple alternative is to behave
greedily most of the time, but every once in a while, say with small probability ", instead

28 Chapter 2: Multi-armed Bandits
select randomly from among all the actions with equal probability, independently of
the action-value estimates. We call methods using this near-greedy action selection rule
"-greedy methods. An advantage of these methods is that, in the limit as the number of
steps increases, every action will be sampled an inﬁnite number of times, thus ensuring
that all the Qt(a) converge to q⇤(a). This of course implies that the probability of selecting
the optimal action converges to greater than 1 ", that is, to near certainty. These are
just asymptotic guarantees, however, and say little about the practical e↵ectiveness of
the methods.
Exercise 2.1 In"-greedy action selection, for the case of two actions and "=0.5, what is
the probability that the greedy action is selected? ⇤
2.3 The 10-armed Testbed
To roughly assess the relative e↵ectiveness of the greedy and "-greedy action-value
methods, we compared them numerically on a suite of test problems. This was a set

of 2000 randomly generated k-armed bandit problems with k= 10. For each bandit
problem, such as the one shown in Figure 2.1, the action values, q⇤(a),a=1,...,10,0123-3-2-1q⇤(1)q⇤(2)q⇤(3)q⇤(4)q⇤(5)q⇤(6)q⇤(7)q⇤(8)q⇤(9)q⇤(10)Rewarddistribution12635478910ActionFigure 2.1: An example bandit problem from the 10-armed testbed. The true value q⇤(a)o f
each of the ten actions was selected according to a normal distribution with mean zero and unit
variance, and then the actual rewards were selected according to a mean q⇤(a) unit variance

normal distribution, as suggested by these gray distributions.

2.3. The 10-armed Testbed 29
were selected according to a normal (Gaussian) distribution with mean 0 and variance 1.
Then, when a learning method applied to that problem selected action Atat time step t,
the actual reward, Rt, was selected from a normal distribution with mean q⇤(At) and
variance 1. These distributions are shown in gray in Figure 2.1. We call this suite of test
tasks the 10-armed testbed . For any learning method, we can measure its performance
and behavior as it improves with experience over 1000 time steps when applied to one of
the bandit problems. This makes up one run. Repeating this for 2000 independent runs,
each with a di↵erent bandit problem, we obtained measures of the learning algorithm’s
average behavior.
Figure 2.2 compares a greedy method with two "-greedy methods ( "=0.01 and "=0.1),
as described above, on the 10-armed testbed. All the methods formed their action-value
estimates using the sample-average technique. The upper graph shows the increase in

expected reward with experience. The greedy method improved slightly faster than the
other methods at the very beginning, but then leveled o↵ at a lower level. It achieved a
reward-per-step of only about 1, compared with the best possible of about 1.55 on this
testbed. The greedy method performed signiﬁcantly worse in the long run because it (greedy)00.511.5Averagereward02505007501000Steps0%20%40%60%80%100%%Optimalaction02505007501000Steps  11"=0.1"=0.01"=0.1"=0.01"=0 (greedy)"=0
Figure 2.2: Average performance of "-greedy action-value methods on the 10-armed testbed.

These data are averages over 2000 runs with di↵erent bandit problems. All methods used sample
averages as their action-value estimates.

30 Chapter 2: Multi-armed Bandits
often got stuck performing suboptimal actions. The lower graph shows that the greedy
method found the optimal action in only approximately one-third of the tasks. In the
other two-thirds, its initial samples of the optimal action were disappointing, and it never
returned to it. The "-greedy methods eventually performed better because they continued
to explore and to improve their chances of recognizing the optimal action. The "=0.1
method explored more, and usually found the optimal action earlier, but it never selected
that action more than 91% of the time. The "=0.01 method improved more slowly, but
eventually would perform better than the "=0.1 method on both performance measures
shown in the ﬁgure. It is also possible to reduce "over time to try to get the best of both
high and low values.
The advantage of "-greedy over greedy methods depends on the task. For example,
suppose the reward variance had been larger, say 10 instead of 1. With noisier rewards

it takes more exploration to ﬁnd the optimal action, and "-greedy methods should fare
even better relative to the greedy method. On the other hand, if the reward variances
were zero, then the greedy method would know the true value of each action after trying
it once. In this case the greedy method might actually perform best because it would
soon ﬁnd the optimal action and then never explore. But even in the deterministic case
there is a large advantage to exploring if we weaken some of the other assumptions. For

example, suppose the bandit task were nonstationary, that is, the true values of the
actions changed over time. In this case exploration is needed even in the deterministic
case to make sure one of the nongreedy actions has not changed to become better than
the greedy one. As we shall see in the next few chapters, nonstationarity is the case
most commonly encountered in reinforcement learning. Even if the underlying task is
stationary and deterministic, the learner faces a set of banditlike decision tasks each of

which changes over time as learning proceeds and the agent’s decision-making policy
changes. Reinforcement learning requires a balance between exploration and exploitation.
Exercise 2.2: Bandit example Consider a k-armed bandit problem with k= 4 actions,
denoted 1, 2, 3, and 4. Consider applying to this problem a bandit algorithm using
"-greedy action selection, sample-average action-value estimates, and initial estimates
ofQ1(a) = 0, for all a. Suppose the initial sequence of actions and rewards is A1= 1,
R1=1,A2= 2, R2= 1, A3= 2, R3=2,A4= 2, R4= 2, A5= 3, R5= 0. On some

of these time steps the "case may have occurred, causing an action to be selected at
random. On which time steps did this deﬁnitely occur? On which time steps could this
possibly have occurred? ⇤
Exercise 2.3 In the comparison shown in Figure 2.2, which method will perform best in
the long run in terms of cumulative reward and probability of selecting the best action?
How much better will it be? Express your answer quantitatively. ⇤
2.4 Incremental Implementation

The action-value methods we have discussed so far all estimate action values as sample
averages of observed rewards. We now turn to the question of how these averages can be
computed in a computationally ecient manner, in particular, with constant memory

2.4. Incremental Implementation 31
and constant per-time-step computation.
To simplify notation we concentrate on a single action. Let Rinow denote the reward
received after the ith selection of this action , and let Qndenote the estimate of its action
value after it has been selected n1 times, which we can now write simply as
Qn.=R1+R2+···+Rn1
n1.
The obvious implementation would be to maintain a record of all the rewards and then
perform this computation whenever the estimated value was needed. However, if this is
done, then the memory and computational requirements would grow over time as more
rewards are seen. Each additional reward would require additional memory to store it
and additional computation to compute the sum in the numerator.
As you might suspect, this is not really necessary. It is easy to devise incremental
formulas for updating averages with small, constant computation required to process
each new reward. Given Qnand the nth reward, Rn, the new average of all nrewards
can be computed by

32 Chapter 2: Multi-armed Bandits
method uses the step-size parameter1
n. In this book we denote the step-size parameter
by↵or, more generally, by ↵t(a).
Pseudocode for a complete bandit algorithm using incrementally computed sample
averages and "-greedy action selection is shown in the box below. The function bandit (a)
is assumed to take an action and return a corresponding reward.
A simple bandit algorithm
Initialize, for a=1t o k:
Q(a) 0
N(a) 0
Loop forever:
A ⇢argmaxaQ(a) with probability 1 "(breaking ties randomly)
a random action with probability "
R bandit (A)
N(A) N(A)+1
Q(A) Q(A)+1
N(A)⇥
RQ(A)⇤
2.5 Tracking a Nonstationary Problem
The averaging methods discussed so far are appropriate for stationary bandit problems,
that is, for bandit problems in which the reward probabilities do not change over time.
As noted earlier, we often encounter reinforcement learning problems that are e↵ectively
nonstationary. In such cases it makes sense to give more weight to recent rewards than

of the convention that 00= 1.) Accordingly, this is sometimes called an exponential
recency-weighted average .
Sometimes it is convenient to vary the step-size parameter from step to step. Let ↵n(a)
denote the step-size parameter used to process the reward received after the nth selection
of action a. As we have noted, the choice ↵n(a)=1
nresults in the sample-average method,
which is guaranteed to converge to the true action values by the law of large numbers.
But of course convergence is not guaranteed for all choices of the sequence {↵n(a)}.A
well-known result in stochastic approximation theory gives us the conditions required to
assure convergence with probability 1:
1X
n=1↵n(a)=1and1X
n=1↵2
n(a)<1. (2.7)
The ﬁrst condition is required to guarantee that the steps are large enough to eventually
overcome any initial conditions or random ﬂuctuations. The second condition guarantees
that eventually the steps become small enough to assure convergence.

1X
n=1↵2
Note that both convergence conditions are met for the sample-average case, ↵n(a)=1
n,
but not for the case of constant step-size parameter, ↵n(a)=↵. In the latter case, the
second condition is not met, indicating that the estimates never completely converge but
continue to vary in response to the most recently received rewards. As we mentioned
above, this is actually desirable in a nonstationary environment, and problems that are
e↵ectively nonstationary are the most common in reinforcement learning. In addition,

n,
sequences of step-size parameters that meet the conditions (2.7) often converge very slowly
or need considerable tuning in order to obtain a satisfactory convergence rate. Although
sequences of step-size parameters that meet these convergence conditions are often used
in theoretical work, they are seldom used in applications and empirical research.
Exercise 2.4 If the step-size parameters, ↵n, are not constant, then the estimate Qnis
a weighted average of previously received rewards with a weighting di↵erent from that

given by (2.6). What is the weighting on each prior reward for the general case, analogous
to (2.6), in terms of the sequence of step-size parameters? ⇤
Exercise 2.5 (programming) Design and conduct an experiment to demonstrate the
diculties that sample-average methods have for nonstationary problems. Use a modiﬁed
version of the 10-armed testbed in which all the q⇤(a) start out equal and then take
independent random walks (say by adding a normally distributed increment with mean
zero and standard deviation 0.01 to all the q⇤(a) on each step). Prepare plots like

Figure 2.2 for an action-value method using sample averages, incrementally computed,
and another action-value method using a constant step-size parameter, ↵=0.1. Use
"=0.1 and longer runs, say of 10,000 steps. ⇤

34 Chapter 2: Multi-armed Bandits
2.6 Optimistic Initial Values
All the methods we have discussed so far are dependent to some extent on the initial
action-value estimates, Q1(a). In the language of statistics, these methods are biased
by their initial estimates. For the sample-average methods, the bias disappears once all
actions have been selected at least once, but for methods with constant ↵, the bias is
permanent, though decreasing over time as given by (2.6). In practice, this kind of bias
is usually not a problem and can sometimes be very helpful. The downside is that the
initial estimates become, in e↵ect, a set of parameters that must be picked by the user, if
only to set them all to zero. The upside is that they provide an easy way to supply some
prior knowledge about what level of rewards can be expected.
Initial action values can also be used as a simple way to encourage exploration. Suppose
that instead of setting the initial action values to zero, as we did in the 10-armed testbed,

we set them all to +5. Recall that the q⇤(a) in this problem are selected from a normal
distribution with mean 0 and variance 1. An initial estimate of +5 is thus wildly optimistic.
But this optimism encourages action-value methods to explore. Whichever actions are
initially selected, the reward is less than the starting estimates; the learner switches to
other actions, being “disappointed” with the rewards it is receiving. The result is that all

actions are tried several times before the value estimates converge. The system does a
fair amount of exploration even if greedy actions are selected all the time.
Figure 2.3 shows the performance on the 10-armed bandit testbed of a greedy method
using Q1(a) = +5, for all a. For comparison, also shown is an "-greedy method with
Q1(a) = 0. Initially, the optimistic method performs worse because it explores more,
but eventually it performs better because its exploration decreases with time. We call

this technique for encouraging exploration optimistic initial values . We regard it as
a simple trick that can be quite e↵ective on stationary problems, but it is far from
being a generally useful approach to encouraging exploration. For example, it is not
well suited to nonstationary problems because its drive for exploration is inherently
0%20%40%60%80%100%%Optimalaction02004006008001000Playsoptimistic, greedyQ0 = 5,  !!= 0realistic, !-greedyQ0 = 0,  !!= 0.111

Steps1Optimistic, greedyQ1=5,"=0Realistic,   -greedy"Q1=0,"=0.1
Figure 2.3: The e↵ect of optimistic initial action-value estimates on the 10-armed testbed.
Both methods used a constant step-size parameter, ↵=0.1.

2.7. Upper-Conﬁdence-Bound Action Selection 35
temporary. If the task changes, creating a renewed need for exploration, this method
cannot help. Indeed, any method that focuses on the initial conditions in any special way
is unlikely to help with the general nonstationary case. The beginning of time occurs
only once, and thus we should not focus on it too much. This criticism applies as well to
the sample-average methods, which also treat the beginning of time as a special event,
averaging all subsequent rewards with equal weights. Nevertheless, all of these methods
are very simple, and one of them—or some simple combination of them—is often adequate
in practice. In the rest of this book we make frequent use of several of these simple
exploration techniques.
Exercise 2.6: Mysterious Spikes The results shown in Figure 2.3 should be quite reliable
because they are averages over 2000 individual, randomly chosen 10-armed bandit tasks.

Why, then, are there oscillations and spikes in the early part of the curve for the optimistic
method? In other words, what might make this method perform particularly better or
worse, on average, on particular early steps? ⇤
Exercise 2.7: Unbiased Constant-Step-Size Trick In most of this chapter we have used
sample averages to estimate action values because sample averages do not produce the
initial bias that constant step sizes do (see the analysis leading to (2.6) ). However, sample

average without initial bias . ⇤
2.7 Upper-Conﬁdence-Bound Action Selection
Exploration is needed because there is always uncertainty about the accuracy of the
action-value estimates. The greedy actions are those that look best at present, but some of
the other actions may actually be better. "-greedy action selection forces the non-greedy
actions to be tried, but indiscriminately, with no preference for those that are nearly
greedy or particularly uncertain. It would be better to select among the non-greedy
actions according to their potential for actually being optimal, taking into account both
how close their estimates are to being maximal and the uncertainties in those estimates.
One e↵ective way of doing this is to select actions according to
a"
Qt(a)+cs
lnt
Nt(a)#
, (2.10)

36 Chapter 2: Multi-armed Bandits
where lntdenotes the natural logarithm of t(the number that e⇡2.71828 would have
to be raised to in order to equal t),Nt(a) denotes the number of times that action ahas
been selected prior to time t(the denominator in (2.1) ), and the number c>0 controls
the degree of exploration. If Nt(a) = 0, then ais considered to be a maximizing action.
The idea of this upper conﬁdence bound (UCB) action selection is that the square-root
term is a measure of the uncertainty or variance in the estimate of a’s value. The quantity
being max’ed over is thus a sort of upper bound on the possible true value of action a,w i t h
cdetermining the conﬁdence level. Each time ais selected the uncertainty is presumably
reduced: Nt(a) increments, and, as it appears in the denominator, the uncertainty term
decreases. On the other hand, each time an action other than ais selected, tincreases but
Nt(a) does not; because tappears in the numerator, the uncertainty estimate increases.

The use of the natural logarithm means that the increases get smaller over time, but are
unbounded; all actions will eventually be selected, but actions with lower value estimates,
or that have already been selected frequently, will be selected with decreasing frequency
over time.
Results with UCB on the 10-armed testbed are shown in Figure 2.4. UCB often
performs well, as shown here, but is more dicult than "-greedy to extend beyond bandits
to the more general reinforcement learning settings considered in the rest of this book.

those presented in Section 2.5 would be needed. Another diculty is dealing with large
state spaces, particularly when using function approximation as developed in Part II of
this book. In these more advanced settings the idea of UCB action selection is usually
not practical.
1250500750100000.511.5-greedy   = 0.1UCB  c = 2AveragerewardStepsFigure 2.4: Average performance of UCB action selection on the 10-armed testbed. As shown,
UCB generally performs better than "-greedy action selection, except in the ﬁrst ksteps, when
it selects randomly among the as-yet-untried actions.
Exercise 2.8: UCB Spikes In Figure 2.4 the UCB algorithm shows a distinct spike
in performance on the 11th step. Why is this? Note that for your answer to be fully
satisfactory it must explain both why the reward increases on the 11th step and why it
decreases on the subsequent steps. Hint: if c= 1, then the spike is less prominent. ⇤

2.8. Gradient Bandit Algorithms 37
2.8 Gradient Bandit Algorithms
So far in this chapter we have considered methods that estimate action values and use
those estimates to select actions. This is often a good approach, but it is not the only
one possible. In this section we consider learning a numerical preference for each action
a, which we denote Ht(a). The larger the preference, the more often that action is taken,
but the preference has no interpretation in terms of reward. Only the relative preference
of one action over another is important; if we add 1000 to all the action preferences there
is no e↵ect on the action probabilities, which are determined according to a soft-max
distribution (i.e., Gibbs or Boltzmann distribution) as follows:
Pr{At=a}.=eHt(a)
Pk
b=1eHt(b).=⇡t(a), (2.11)
where here we have also introduced a useful new notation, ⇡t(a), for the probability of
taking action aat time t. Initially all action preferences are the same (e.g., H1(a) = 0,

where ↵>0 is a step-size parameter, and ¯Rt2Ris the average of all the rewards up
through and including time t, which can be computed incrementally as described in
Section 2.4 (or Section 2.5 if the problem is nonstationary). The ¯Rtterm serves as a
baseline with which the reward is compared. If the reward is higher than the baseline,
then the probability of taking Atin the future is increased, and if the reward is below
baseline, then probability is decreased. The non-selected actions move in the opposite
Figure 2.5 shows results with the gradient bandit algorithm on a variant of the 10-
armed testbed in which the true expected rewards were selected according to a normal
distribution with a mean of +4 instead of zero (and with unit variance as before). This
shifting up of all the rewards has absolutely no e↵ect on the gradient bandit algorithm
because of the reward baseline term, which instantaneously adapts to the new level. But

if the baseline were omitted (that is, if ¯Rtwas taken to be constant zero in (2.12) ), then
performance would be signiﬁcantly degraded, as shown in the ﬁgure.

38 Chapter 2: Multi-armed Bandits
%OptimalactionStepsα = 0.1100%80%60%40%20%0%α = 0.4α = 0.1α = 0.4without baselinewith baseline12505007501000Figure 2.5: Average performance of the gradient bandit algorithm with and without a reward
baseline on the 10-armed testbed when the q⇤(a) are chosen to be near +4 rather than near zero.
The Bandit Gradient Algorithm as Stochastic Gradient Ascent
One can gain a deeper insight into the gradient bandit algorithm by understanding
it as a stochastic approximation to gradient ascent. In exact gradient ascent , each
action preference Ht(a) would be incremented proportional to the increment’s e↵ect
on performance:
Ht+1(a).=Ht(a)+↵@E[Rt]
@Ht(a), (2.13)
where the measure of performance here is the expected reward:
E[Rt]=X
x⇡t(x)q⇤(x),
and the measure of the increment’s e↵ect is the partial derivative of this performance
measure with respect to the action preference. Of course, it is not possible to
implement gradient ascent exactly in our case because by assumption we do not

E[Rt]=X
know the q⇤(x), but in fact the updates of our algorithm (2.12) are equal to (2.13)
in expected value, making the algorithm an instance of stochastic gradient ascent .
The calculations showing this require only beginning calculus, but take several

⇣Pk
y=1eHt(y)⌘2
=
 a=x⇡t(x)⇡t(x)⇡t(a)
=⇡t(x)
a=x⇡t(a)
. Q.E.D.
We have just shown that the expected update of the gradient bandit algorithm
is equal to the gradient of expected reward, and thus that the algorithm is an
instance of stochastic gradient ascent. This assures us that the algorithm has robust
convergence properties.
Note that we did not require any properties of the reward baseline other than
that it does not depend on the selected action. For example, we could have set
it to zero, or to 1000, and the algorithm would still be an instance of stochastic
gradient ascent. The choice of the baseline does not a↵ect the expected update
of the algorithm, but it does a↵ect the variance of the update and thus the rate
of convergence (as shown, e.g., in Figure 2.5). Choosing it as the average of the
rewards may not be the very best, but it is simple and works well in practice.

2.9. Associative Search (Contextual Bandits) 41
2.9 Associative Search (Contextual Bandits)
So far in this chapter we have considered only nonassociative tasks, that is, tasks in which
there is no need to associate di↵erent actions with di↵erent situations. In these tasks
the learner either tries to ﬁnd a single best action when the task is stationary, or tries to
track the best action as it changes over time when the task is nonstationary. However,
in a general reinforcement learning task there is more than one situation, and the goal
is to learn a policy: a mapping from situations to the actions that are best in those
situations. To set the stage for the full problem, we brieﬂy discuss the simplest way in
which nonassociative tasks extend to the associative setting.
As an example, suppose there are several di↵erent k-armed bandit tasks, and that on
each step you confront one of these chosen at random. Thus, the bandit task changes

randomly from step to step. This would appear to you as a single, nonstationary k-armed
bandit task whose true action values change randomly from step to step. You could
try using one of the methods described in this chapter that can handle nonstationarity,
but unless the true action values change slowly, these methods will not work very well.
Now suppose, however, that when a bandit task is selected for you, you are given some
distinctive clue about its identity (but not its action values). Maybe you are facing an

actual slot machine that changes the color of its display as it changes its action values.
Now you can learn a policy associating each task, signaled by the color you see, with
the best action to take when facing that task—for instance, if red, select arm 1; if green,
select arm 2. With the right policy you can usually do much better than you could in
the absence of any information distinguishing one bandit task from another.
This is an example of an associative search task, so called because it involves both

trial-and-error learning to search for the best actions, and association of these actions
with the situations in which they are best. Associative search tasks are often now called
contextual bandits in the literature. Associative search tasks are intermediate between
thek-armed bandit problem and the full reinforcement learning problem. They are like
the full reinforcement learning problem in that they involve learning a policy, but like
our version of the k-armed bandit problem in that each action a↵ects only the immediate

reward. If actions are allowed to a↵ect the next situation as well as the reward, then
we have the full reinforcement learning problem. We present this problem in the next
chapter and consider its ramiﬁcations throughout the rest of the book.
Exercise 2.10 Suppose you face a 2-armed bandit task whose true action values change
randomly from time step to time step. Speciﬁcally, suppose that, for any time step, the
true values of actions 1 and 2 are respectively 0.1 and 0.2 with probability 0.5 (case A),

and 0.9 and 0.8 with probability 0.5 (case B). If you are not able to tell which case you
face at any step, what is the best expectation of success you can achieve and how should
you behave to achieve it? Now suppose that on each step you are told whether you are
facing case A or case B (although you still don’t know the true action values). This is an
associative search task. What is the best expectation of success you can achieve in this
task, and how should you behave to achieve it? ⇤

42 Chapter 2: Multi-armed Bandits
2.10 Summary
We have presented in this chapter several simple ways of balancing exploration and
exploitation. The "-greedy methods choose randomly a small fraction of the time, whereas
UCB methods choose deterministically but achieve exploration by subtly favoring at each
step the actions that have so far received fewer samples. Gradient bandit algorithms
estimate not action values, but action preferences, and favor the more preferred actions
in a graded, probabilistic manner using a soft-max distribution. The simple expedient of
initializing estimates optimistically causes even greedy methods to explore signiﬁcantly.
It is natural to ask which of these methods is best. Although this is a dicult question
to answer in general, we can certainly run them all on the 10-armed testbed that we
have used throughout this chapter and compare their performances. A complication is
that they all have a parameter; to get a meaningful comparison we have to consider

their performance as a function of their parameter. Our graphs so far have shown the
course of learning over time for each algorithm and parameter setting, to produce a
learning curve for that algorithm and parameter setting. If we plotted learning curves
for all algorithms and all parameter settings, then the graph would be too complex and
crowded to make clear comparisons. Instead we summarize a complete learning curve
by its average value over the 1000 steps; this value is proportional to the area under the

learning curve. Figure 2.6 shows this measure for the various bandit algorithms from
this chapter, each as a function of its own parameter shown on a single scale on the
x-axis. This kind of graph is called a parameter study . Note that the parameter values
are varied by factors of two and presented on a log scale. Note also the characteristic
inverted-U shapes of each algorithm’s performance; all the algorithms perform best at

an intermediate value of their parameter, neither too large nor too small. In assessingAveragerewardover ﬁrst 1000 steps1.51.41.31.21.11-greedyUCBgradientbanditgreedy withoptimisticinitializationα = 0.11241/21/41/81/161/321/641/128"↵cQ0
Figure 2.6: A parameter study of the various bandit algorithms presented in this chapter.
Each point is the average reward obtained over 1000 steps with a particular algorithm at a
particular setting of its parameter.

2.10. Summary 43
a method, we should attend not just to how well it does at its best parameter setting,
but also to how sensitive it is to its parameter value. All of these algorithms are fairly
insensitive, performing well over a range of parameter values varying by about an order
of magnitude. Overall, on this problem, UCB seems to perform best.
Despite their simplicity, in our opinion the methods presented in this chapter can
fairly be considered the state of the art. There are more sophisticated methods, but their
complexity and assumptions make them impractical for the full reinforcement learning
problem that is our real focus. Starting in Chapter 5 we present learning methods for
solving the full reinforcement learning problem that use in part the simple methods
explored in this chapter.
Although the simple methods explored in this chapter may be the best we can do
at present, they are far from a fully satisfactory solution to the problem of balancing
exploration and exploitation.

One well-studied approach to balancing exploration and exploitation in k-armed bandit
problems is to compute a special kind of action value called a Gittins index . In certain
important special cases, this computation is tractable and leads directly to optimal
solutions, although it does require complete knowledge of the prior distribution of possible
problems, which we generally assume is not available. In addition, neither the theory
nor the computational tractability of this approach appear to generalize to the full

reinforcement learning problem that we consider in the rest of the book.
The Gittins-index approach is an instance of Bayesian methods, which assume a known
initial distribution over the action values and then update the distribution exactly after
each step (assuming that the true action values are stationary). In general, the update
computations can be very complex, but for certain special distributions (called conjugate
priors ) they are easy. One possibility is to then select actions at each step according

to their posterior probability of being the best action. This method, sometimes called
posterior sampling orThompson sampling , often performs similarly to the best of the
distribution-free methods we have presented in this chapter.
In the Bayesian setting it is even conceivable to compute the optimal balance between
exploration and exploitation. One can compute for any possible action the probability
of each possible immediate reward and the resultant posterior distributions over action
values. This evolving distribution becomes the information state of the problem. Given

a horizon, say of 1000 steps, one can consider all possible actions, all possible resulting
rewards, all possible next actions, all next rewards, and so on for all 1000 steps. Given
the assumptions, the rewards and probabilities of each possible chain of events can be
determined, and one need only pick the best. But the tree of possibilities grows extremely
rapidly; even if there were only two actions and two rewards, the tree would have 22000

leaves. It is generally not feasible to perform this immense computation exactly, but
perhaps it could be approximated eciently. This approach would e↵ectively turn the
bandit problem into an instance of the full reinforcement learning problem. In the end, we
may be able to use approximate reinforcement learning methods such as those presented
in Part II of this book to approach this optimal solution. But that is a topic for research
and beyond the scope of this introductory book.

44 Chapter 2: Multi-armed Bandits
Exercise 2.11 (programming) Make a ﬁgure analogous to Figure 2.6 for the nonstationary
case outlined in Exercise 2.5. Include the constant-step-size "-greedy algorithm with
↵=0.1. Use runs of 200,000 steps and, as a performance measure for each algorithm and
parameter setting, use the average reward over the last 100,000 steps. ⇤
Bibliographical and Historical Remarks
2.1 Bandit problems have been studied in statistics, engineering, and psychology. In
statistics, bandit problems fall under the heading “sequential design of experi-
ments,” introduced by Thompson (1933, 1934) and Robbins (1952), and studied
by Bellman (1956). Berry and Fristedt (1985) provide an extensive treatment of
bandit problems from the perspective of statistics. Narendra and Thathachar
(1989) treat bandit problems from the engineering perspective, providing a good
discussion of the various theoretical traditions that have focused on them. In

psychology, bandit problems have played roles in statistical learning theory (e.g.,
Bush and Mosteller, 1955; Estes, 1950).
The term greedy is often used in the heuristic search literature (e.g., Pearl, 1984).
The conﬂict between exploration and exploitation is known in control engineering
as the conﬂict between identiﬁcation (or estimation) and control (e.g., Witten,
1976b). Feldbaum (1965) called it the dual control problem, referring to the
need to solve the two problems of identiﬁcation and control simultaneously when

trying to control a system under uncertainty. In discussing aspects of genetic
algorithms, Holland (1975) emphasized the importance of this conﬂict, referring
to it as the conﬂict between the need to exploit and the need for new information.
2.2 Action-value methods for our k-armed bandit problem were ﬁrst proposed by
Thathachar and Sastry (1985). These are often called estimator algorithms in the
learning automata literature. The term action value is due to Watkins (1989).
The ﬁrst to use "-greedy methods may also have been Watkins (1989, p. 187),

but the idea is so simple that some earlier use seems likely.
2.4–5 This material falls under the general heading of stochastic iterative algorithms,
which is well covered by Bertsekas and Tsitsiklis (1996).
2.6 Optimistic initialization was used in reinforcement learning by Sutton (1996).
2.7 Early work on using estimates of the upper conﬁdence bound to select actions
was done by Lai and Robbins (1985), Kaelbling (1993b), and Agrawal (1995).
The UCB algorithm we present here is called UCB1 in the literature and was

ﬁrst developed by Auer, Cesa-Bianchi and Fischer (2002).
2.8 Gradient bandit algorithms are a special case of the gradient-based reinforcement
learning algorithms introduced by Williams (1992), and that later developed into
the actor–critic and policy-gradient algorithms that we treat later in this book.
Our development here was inﬂuenced by that by Balaraman Ravindran (personal

Bibliographical and Historical Remarks 45
communication). Further discussion of the choice of baseline is provided there
and by Greensmith, Bartlett, and Baxter (2002, 2004) and Dick (2015). Early
systematic studies of algorithms like this were done by Sutton (1984).
The term soft-max for the action selection rule (2.11) is due to Bridle (1990).
This rule appears to have been ﬁrst proposed by Luce (1959).
2.9 The term associative search and the corresponding problem were introduced by
Barto, Sutton, and Brouwer (1981). The term associative reinforcement learning
has also been used for associative search (Barto and Anandan, 1985), but we
prefer to reserve that term as a synonym for the full reinforcement learning
problem (as in Sutton, 1984). (And, as we noted, the modern literature also
uses the term “contextual bandits” for this problem.) We note that Thorndike’s
Law of E↵ect (quoted in Chapter 1) describes associative search by referring

to the formation of associative links between situations (states) and actions.
According to the terminology of operant, or instrumental, conditioning (e.g.,
Skinner, 1938), a discriminative stimulus is a stimulus that signals the presence
of a particular reinforcement contingency. In our terms, di↵erent discriminative
stimuli correspond to di↵erent states.
2.10 Bellman (1956) was the ﬁrst to show how dynamic programming could be used
to compute the optimal balance between exploration and exploitation within a

Bayesian formulation of the problem. The Gittins index approach is due to Gittins
and Jones (1974). Du↵ (1995) showed how it is possible to learn Gittins indices
for bandit problems through reinforcement learning. The survey by Kumar (1985)
provides a good discussion of Bayesian and non-Bayesian approaches to these
problems. The term information state comes from the literature on partially
observable MDPs; see, e.g., Lovejoy (1991).
Other theoretical research focuses on the eciency of exploration, usually ex-

pressed as how quickly an algorithm can approach an optimal decision-making
policy. One way to formalize exploration eciency is by adapting to reinforcement
learning the notion of sample complexity for a supervised learning algorithm,
which is the number of training examples the algorithm needs to attain a desired
degree of accuracy in learning the target function. A deﬁnition of the sample
complexity of exploration for a reinforcement learning algorithm is the number of
time steps in which the algorithm does not select near-optimal actions (Kakade,

2003). Li (2012) discusses this and several other approaches in a survey of
theoretical approaches to exploration eciency in reinforcement learning. A
thorough modern treatment of Thompson sampling is provided by Russo, Van
Roy, Kazerouni, Osband, and Wen (2018).

Chapter 3
Finite Markov Decision
Processes
In this chapter we introduce the formal problem of ﬁnite Markov decision processes, or
ﬁnite MDPs, which we try to solve in the rest of the book. This problem involves evaluative
feedback, as in bandits, but also an associative aspect—choosing di↵erent actions in
di↵erent situations. MDPs are a classical formalization of sequential decision making,
where actions inﬂuence not just immediate rewards, but also subsequent situations, or
states, and through those future rewards. Thus MDPs involve delayed reward and the
need to tradeo↵ immediate and delayed reward. Whereas in bandit problems we estimated
the value q⇤(a) of each action a, in MDPs we estimate the value q⇤(s, a) of each action a
in each state s, or we estimate the value v⇤(s) of each state given optimal action selections.
These state-dependent quantities are essential to accurately assigning credit for long-term
consequences to individual action selections.

MDPs are a mathematically idealized form of the reinforcement learning problem
for which precise theoretical statements can be made. We introduce key elements of
the problem’s mathematical structure, such as returns, value functions, and Bellman
equations. We try to convey the wide range of applications that can be formulated as
ﬁnite MDPs. As in all of artiﬁcial intelligence, there is a tension between breadth of
applicability and mathematical tractability. In this chapter we introduce this tension

and discuss some of the trade-o↵s and challenges that it implies. Some ways in which
reinforcement learning can be taken beyond MDPs are treated in Chapter 17.
3.1 The Agent–Environment Interface
MDPs are meant to be a straightforward framing of the problem of learning from
interaction to achieve a goal. The learner and decision maker is called the agent .T h e
thing it interacts with, comprising everything outside the agent, is called the environment .

These interact continually, the agent selecting actions and the environment responding to
47

48 Chapter 3: Finite Markov Decision Processes
these actions and presenting new situations to the agent.1The environment also gives
rise to rewards, special numerical values that the agent seeks to maximize over time
through its choice of actions.
AgentEnvironmentactionAtrewardRtstateStRt+1St+1Figure 3.1: The agent–environment interaction in a Markov decision process.
More speciﬁcally, the agent and environment interact at each of a sequence of discrete
time steps, t=0,1,2,3,....2At each time step t, the agent receives some representation
of the environment’s state ,St2S, and on that basis selects an action ,At2A(s).3One
time step later, in part as a consequence of its action, the agent receives a numerical
reward ,Rt+12R⇢R, and ﬁnds itself in a new state, St+1.4The MDP and agent
together thereby give rise to a sequence or trajectory that begins like this:
S0,A0,R1,S1,A1,R2,S2,A2,R3,... (3.1)
In a ﬁnite MDP, the sets of states, actions, and rewards ( S,A, and R) all have a ﬁnite

number of elements. In this case, the random variables Rtand Sthave well deﬁned
discrete probability distributions dependent only on the preceding state and action. That
is, for particular values of these random variables, s02Sandr2R, there is a probability
of those values occurring at time t, given particular values of the preceding state and
action:
p(s0,r|s, a).=P r {St=s0,Rt=r|St1=s, A t1=a}, (3.2)
for all s0,s2S,r2R, and a2A(s). The function pdeﬁnes the dynamics of the MDP.
The dot over the equals sign in the equation reminds us that it is a deﬁnition (in this

action:
case of the function p) rather than a fact that follows from previous deﬁnitions. The
dynamics function p:S⇥R⇥S⇥A![0,1] is an ordinary deterministic function of four
arguments. The ‘ |’ in the middle of it comes from the notation for conditional probability,
1We use the terms agent ,environment ,a n d action instead of the engineers’ terms controller ,controlled
system (orplant ), and control signal because they are meaningful to a wider audience.

2We restrict attention to discrete time to keep things as simple as possible, even though many of the
ideas can be extended to the continuous-time case (e.g., see Bertsekas and Tsitsiklis, 1996; Doya, 1996).
3To simplify notation, we sometimes assume the special case in which the action set is the same in all
states and write it simply as A.
4We use Rt+1instead of Rtto denote the reward due to Atbecause it emphasizes that the next
reward and next state, Rt+1andSt+1,a r ej o i n t l yd e t e r m i n e d . U n f o r t u n a t e l y ,b o t hc o n v e n t i o n sa r e

widely used in the literature.

3.1. The Agent–Environment Interface 49
but here it just reminds us that pspeciﬁes a probability distribution for each choice of s
anda, that is, that
X
s02SX
r2Rp(s0,r|s, a)=1 ,for all s2S,a2A(s). (3.3)
In a Markov decision process, the probabilities given by pcompletely characterize the
environment’s dynamics. That is, the probability of each possible value for StandRt
depends only on the immediately preceding state and action, St1andAt1, and, given
them, not at all on earlier states and actions. This is best viewed a restriction not on the
decision process, but on the state . The state must include information about all aspects
of the past agent–environment interaction that make a di↵erence for the future. If it
does, then the state is said to have the Markov property . We will assume the Markov
property throughout this book, though starting in Part II we will consider approximation
methods that do not rely on it, and in Chapter 17 we consider how a Markov state can

be learned and constructed from non-Markov observations.
From the four-argument dynamics function, p, one can compute anything else one might
want to know about the environment, such as the state-transition probabilities (which we
denote, with a slight abuse of notation, as a three-argument function p:S⇥S⇥A![0,1]),
p(s0|s, a).=P r {St=s0|St1=s, A t1=a}=X
r2Rp(s0,r|s, a). (3.4)
We can also compute the expected rewards for state–action pairs as a two-argument
function r:S⇥A!R:
r(s, a).=E[Rt|St1=s, A t1=a]=X
r2RrX
s02Sp(s0,r|s, a), (3.5)

r2RrX
and the expected rewards for state–action–next-state triples as a three-argument function
r:S⇥A⇥S!R,
r(s, a, s0).=E[Rt|St1=s, A t1=a, S t=s0]=X
r2Rrp(s0,r|s, a)
p(s0|s, a). (3.6)
In this book, we usually use the four-argument pfunction (3.2) , but each of these other
notations are also occasionally convenient.
The MDP framework is abstract and ﬂexible and can be applied to many di↵erent
problems in many di↵erent ways. For example, the time steps need not refer to ﬁxed
intervals of real time; they can refer to arbitrary successive stages of decision making
and acting. The actions can be low-level controls, such as the voltages applied to the
motors of a robot arm, or high-level decisions, such as whether or not to have lunch or
to go to graduate school. Similarly, the states can take a wide variety of forms. They
can be completely determined by low-level sensations, such as direct sensor readings, or

they can be more high-level and abstract, such as symbolic descriptions of objects in a
room. Some of what makes up a state could be based on memory of past sensations or

50 Chapter 3: Finite Markov Decision Processes
even be entirely mental or subjective. For example, an agent could be in the state of not
being sure where an object is, or of having just been surprised in some clearly deﬁned
sense. Similarly, some actions might be totally mental or computational. For example,
some actions might control what an agent chooses to think about, or where it focuses its
attention. In general, actions can be any decisions we want to learn how to make, and
the states can be anything we can know that might be useful in making them.
In particular, the boundary between agent and environment is typically not the same
as the physical boundary of a robot’s or animal’s body. Usually, the boundary is drawn
closer to the agent than that. For example, the motors and mechanical linkages of a robot
and its sensing hardware should usually be considered parts of the environment rather
than parts of the agent. Similarly, if we apply the MDP framework to a person or animal,

the muscles, skeleton, and sensory organs should be considered part of the environment.
Rewards, too, presumably are computed inside the physical bodies of natural and artiﬁcial
learning systems, but are considered external to the agent.
The general rule we follow is that anything that cannot be changed arbitrarily by
the agent is considered to be outside of it and thus part of its environment. We do
not assume that everything in the environment is unknown to the agent. For example,

the agent often knows quite a bit about how its rewards are computed as a function of
its actions and the states in which they are taken. But we always consider the reward
computation to be external to the agent because it deﬁnes the task facing the agent and
thus must be beyond its ability to change arbitrarily. In fact, in some cases the agent may
know everything about how its environment works and still face a dicult reinforcement
learning task, just as we may know exactly how a puzzle like Rubik’s cube works, but

still be unable to solve it. The agent–environment boundary represents the limit of the
agent’s absolute control , not of its knowledge.
The agent–environment boundary can be located at di↵erent places for di↵erent
purposes. In a complicated robot, many di↵erent agents may be operating at once, each
with its own boundary. For example, one agent may make high-level decisions which form
part of the states faced by a lower-level agent that implements the high-level decisions. In
practice, the agent–environment boundary is determined once one has selected particular

states, actions, and rewards, and thus has identiﬁed a speciﬁc decision making task of
interest.
The MDP framework is a considerable abstraction of the problem of goal-directed
learning from interaction. It proposes that whatever the details of the sensory, memory,
and control apparatus, and whatever objective one is trying to achieve, any problem of
learning goal-directed behavior can be reduced to three signals passing back and forth
between an agent and its environment: one signal to represent the choices made by the

interest.
agent (the actions), one signal to represent the basis on which the choices are made (the
states), and one signal to deﬁne the agent’s goal (the rewards). This framework may not
be sucient to represent all decision-learning problems usefully, but it has proved to be
widely useful and applicable.
Of course, the particular states and actions vary greatly from task to task, and how
they are represented can strongly a↵ect performance. In reinforcement learning, as in
other kinds of learning, such representational choices are at present more art than science.

3.1. The Agent–Environment Interface 51
In this book we o↵er some advice and examples regarding good ways of representing
states and actions, but our primary focus is on general principles for learning how to
behave once the representations have been selected.
Example 3.1: Bioreactor Suppose reinforcement learning is being applied to determine
moment-by-moment temperatures and stirring rates for a bioreactor (a large vat of
nutrients and bacteria used to produce useful chemicals). The actions in such an
application might be target temperatures and target stirring rates that are passed to
lower-level control systems that, in turn, directly activate heating elements and motors to
attain the targets. The states are likely to be thermocouple and other sensory readings,
perhaps ﬁltered and delayed, plus symbolic inputs representing the ingredients in the
vat and the target chemical. The rewards might be moment-by-moment measures of the

rate at which the useful chemical is produced by the bioreactor. Notice that here each
state is a list, or vector, of sensor readings and symbolic inputs, and each action is a
vector consisting of a target temperature and a stirring rate. It is typical of reinforcement
learning tasks to have states and actions with such structured representations. Rewards,
on the other hand, are always single numbers.
Example 3.2: Pick-and-Place Robot Consider using reinforcement learning to
control the motion of a robot arm in a repetitive pick-and-place task. If we want to learn

movements that are fast and smooth, the learning agent will have to control the motors
directly and have low-latency information about the current positions and velocities of the
mechanical linkages. The actions in this case might be the voltages applied to each motor
at each joint, and the states might be the latest readings of joint angles and velocities.
The reward might be +1 for each object successfully picked up and placed. To encourage
smooth movements, on each time step a small, negative reward can be given as a function

of the moment-to-moment “jerkiness” of the motion.
Exercise 3.1 Devise three example tasks of your own that ﬁt into the MDP framework,
identifying for each its states, actions, and rewards. Make the three examples as di↵erent
from each other as possible. The framework is abstract and ﬂexible and can be applied in
many di↵erent ways. Stretch its limits in some way in at least one of your examples. ⇤
Exercise 3.2 Is the MDP framework adequate to usefully represent allgoal-directed
learning tasks? Can you think of any clear exceptions? ⇤

Exercise 3.3 Consider the problem of driving. You could deﬁne the actions in terms of
the accelerator, steering wheel, and brake, that is, where your body meets the machine.
Or you could deﬁne them farther out—say, where the rubber meets the road, considering
your actions to be tire torques. Or you could deﬁne them farther in—say, where your
brain meets your body, the actions being muscle twitches to control your limbs. Or you
could go to a really high level and say that your actions are your choices of where to drive.

What is the right level, the right place to draw the line between agent and environment?
On what basis is one location of the line to be preferred over another? Is there any
fundamental reason for preferring one location over another, or is it a free choice? ⇤

52 Chapter 3: Finite Markov Decision Processes
Example 3.3 Recycling Robot
A mobile robot has the job of collecting empty soda cans in an oce environment. It
has sensors for detecting cans, and an arm and gripper that can pick them up and place
them in an onboard bin; it runs on a rechargeable battery. The robot’s control system
has components for interpreting sensory information, for navigating, and for controlling
the arm and gripper. High-level decisions about how to search for cans are made by a
reinforcement learning agent based on the current charge level of the battery. To make a
simple example, we assume that only two charge levels can be distinguished, comprising
as m a l ls t a t es e t S={high ,low}.I n e a c h s t a t e , t h e a g e n t c a n d e c i d e w h e t h e r t o ( 1 )
actively search for a can for a certain period of time, (2) remain stationary and wait
for someone to bring it a can, or (3) head back to its home base to recharge its battery.

When the energy level is high , recharging would always be foolish, so we do not include it
in the action set for this state. The action sets are then A(high )={search ,wait }and
A(low)={search ,wait ,recharge }.
The rewards are zero most of the time, but become positive when the robot secures an
empty can, or large and negative if the battery runs all the way down. The best way to
ﬁnd cans is to actively search for them, but this runs down the robot’s battery, whereas

waiting does not. Whenever the robot is searching, the possibility exists that its battery
will become depleted. In this case the robot must shut down and wait to be rescued
(producing a low reward). If the energy level is high , then a period of active search can
always be completed without risk of depleting the battery. A period of searching that
begins with a high energy level leaves the energy level high with probability ↵and reduces
it to lowwith probability 1 ↵. On the other hand, a period of searching undertaken

3.2. Goals and Rewards 53
summarizing the dynamics of a ﬁnite MDP, as a transition graph . There are two kinds of
nodes: state nodes andaction nodes . There is a state node for each possible state (a large
open circle labeled by the name of the state), and an action node for each state–action
pair (a small solid circle labeled by the name of the action and connected by a line to the
state node). Starting in state sand taking action amoves you along the line from state
node sto action node ( s, a). Then the environment responds with a transition to the next
state’s node via one of the arrows leaving action node ( s, a). Each arrow corresponds to
a triple ( s, s0,a), where s0is the next state, and we label the arrow with the transition
probability, p(s0|s, a), and the expected reward for that transition, r(s, a, s0). Note that
the transition probabilities labeling the arrows leaving an action node always sum to 1.
Exercise 3.4 Give a table analogous to that in Example 3.3, but for p(s0,r|s, a). It

should have columns for s,a,s0,r, and p(s0,r|s, a), and a row for every 4-tuple for which
p(s0,r|s, a)>0. ⇤
3.2 Goals and Rewards
In reinforcement learning, the purpose or goal of the agent is formalized in terms of a
special signal, called the reward , passing from the environment to the agent. At each time
step, the reward is a simple number, Rt2R. Informally, the agent’s goal is to maximize
the total amount of reward it receives. This means maximizing not immediate reward,
but cumulative reward in the long run. We can clearly state this informal idea as the

reward hypothesis :
That all of what we mean by goals and purposes can be well thought of as
the maximization of the expected value of the cumulative sum of a received
scalar signal (called reward).
The use of a reward signal to formalize the idea of a goal is one of the most distinctive
features of reinforcement learning.
Although formulating goals in terms of reward signals might at ﬁrst appear limiting,
in practice it has proved to be ﬂexible and widely applicable. The best way to see this is

to consider examples of how it has been, or could be, used. For example, to make a robot
learn to walk, researchers have provided reward on each time step proportional to the
robot’s forward motion. In making a robot learn how to escape from a maze, the reward
is often 1 for every time step that passes prior to escape; this encourages the agent to
escape as quickly as possible. To make a robot learn to ﬁnd and collect empty soda cans

for recycling, one might give it a reward of zero most of the time, and then a reward of
+1 for each can collected. One might also want to give the robot negative rewards when
it bumps into things or when somebody yells at it. For an agent to learn to play checkers
or chess, the natural rewards are +1 for winning, 1 for losing, and 0 for drawing and
for all nonterminal positions.
You can see what is happening in all of these examples. The agent always learns to
maximize its reward. If we want it to do something for us, we must provide rewards

to it in such a way that in maximizing them the agent will also achieve our goals. It

54 Chapter 3: Finite Markov Decision Processes
is thus critical that the rewards we set up truly indicate what we want accomplished.
In particular, the reward signal is not the place to impart to the agent prior knowledge
about how to achieve what we want it to do.5For example, a chess-playing agent should
be rewarded only for actually winning, not for achieving subgoals such as taking its
opponent’s pieces or gaining control of the center of the board. If achieving these sorts
of subgoals were rewarded, then the agent might ﬁnd a way to achieve them without
achieving the real goal. For example, it might ﬁnd a way to take the opponent’s pieces
even at the cost of losing the game. The reward signal is your way of communicating to
the robot what you want it to achieve, not how you want it achieved.6
3.3 Returns and Episodes
So far we have discussed the objective of learning informally. We have said that the
agent’s goal is to maximize the cumulative reward it receives in the long run. How might

this be deﬁned formally? If the sequence of rewards received after time step tis denoted
Rt+1,Rt+2,Rt+3,..., then what precise aspect of this sequence do we wish to maximize?
In general, we seek to maximize the expected return , where the return, denoted Gt,i s
deﬁned as some speciﬁc function of the reward sequence. In the simplest case the return
is the sum of the rewards:
Gt.=Rt+1+Rt+2+Rt+3+···+RT, (3.7)
where Tis a ﬁnal time step. This approach makes sense in applications in which there

is a natural notion of ﬁnal time step, that is, when the agent–environment interaction
breaks naturally into subsequences, which we call episodes ,7such as plays of a game,
trips through a maze, or any sort of repeated interaction. Each episode ends in a special
state called the terminal state , followed by a reset to a standard starting state or to a
sample from a standard distribution of starting states. Even if you think of episodes as
ending in di↵erent ways, such as winning and losing a game, the next episode begins

independently of how the previous one ended. Thus the episodes can all be considered to
end in the same terminal state, with di↵erent rewards for the di↵erent outcomes. Tasks
with episodes of this kind are called episodic tasks . In episodic tasks we sometimes need
to distinguish the set of all nonterminal states, denoted S, from the set of all states plus
the terminal state, denoted S+. The time of termination, T, is a random variable that
normally varies from episode to episode.
On the other hand, in many cases the agent–environment interaction does not break

naturally into identiﬁable episodes, but goes on continually without limit. For example,
this would be the natural way to formulate an on-going process-control task, or an
application to a robot with a long life span. We call these continuing tasks .T h er e t u r n
formulation (3.7) is problematic for continuing tasks because the ﬁnal time step would
5Better places for imparting this kind of prior knowledge are the initial policy or initial value function,
or in inﬂuences on these.

6Section 17.4 delves further into the issue of designing e↵ective reward signals.
7Episodes are sometimes called “trials” in the literature.

56 Chapter 3: Finite Markov Decision ProcessesExample 3.4: Pole-Balancing
The objective in this task is to apply
forces to a cart moving along a track
so as to keep a pole hinged to the cart
from falling over: A failure is said to
occur if the pole falls past a given angle
from vertical or if the cart runs o↵ the
track. The pole is reset to vertical
after each failure. This task could be
treated as episodic, where the natural
episodes are the repeated attempts to balance the pole. The reward in this case could be
+1 for every time step on which failure did not occur, so that the return at each time
would be the number of steps until failure. In this case, successful balancing forever would
mean a return of inﬁnity. Alternatively, we could treat pole-balancing as a continuing
task, using discounting. In this case the reward would be 1 on each failure and zero at
all other times. The return at each time would then be related to K,w h e r e Kis the

number of time steps before failure. In either case, the return is maximized by keeping
the pole balanced for as long as possible.
Exercise 3.6 Suppose you treated pole-balancing as an episodic task but also used
discounting, with all rewards zero except for 1 upon failure. What then would the
return be at each time? How does this return di↵er from that in the discounted, continuing
formulation of this task? ⇤
Exercise 3.7 Imagine that you are designing a robot to run a maze. You decide to give it a

reward of +1 for escaping from the maze and a reward of zero at all other times. The task
seems to break down naturally into episodes—the successive runs through the maze—so
you decide to treat it as an episodic task, where the goal is to maximize expected total
reward (3.7). After running the learning agent for a while, you ﬁnd that it is showing
no improvement in escaping from the maze. What is going wrong? Have you e↵ectively
communicated to the agent what you want it to achieve? ⇤

Exercise 3.8 Suppose =0.5 and the following sequence of rewards is received R1=1,
R2= 2, R3= 6, R4= 3, and R5= 2, with T= 5. What are G0,G1,...,G5? Hint:
Work backwards. ⇤
Exercise 3.9 Suppose =0.9 and the reward sequence is R1= 2 followed by an inﬁnite
sequence of 7s. What are G1andG0? ⇤
Exercise 3.10 Prove the second equality in (3.10). ⇤

3.4. Uniﬁed Notation for Episodic and Continuing Tasks 57
3.4 Uniﬁed Notation for Episodic and Continuing Tasks
In the preceding section we described two kinds of reinforcement learning tasks, one
in which the agent–environment interaction naturally breaks down into a sequence of
separate episodes (episodic tasks), and one in which it does not (continuing tasks). The
former case is mathematically easier because each action a↵ects only the ﬁnite number of
rewards subsequently received during the episode. In this book we consider sometimes
one kind of problem and sometimes the other, but often both. It is therefore useful to
establish one notation that enables us to talk precisely about both cases simultaneously.
To be precise about episodic tasks requires some additional notation. Rather than one
long sequence of time steps, we need to consider a series of episodes, each of which consists
of a ﬁnite sequence of time steps. We number the time steps of each episode starting

anew from zero. Therefore, we have to refer not just to St, the state representation at
time t,b u tt o St,i, the state representation at time tof episode i(and similarly for At,i,
Rt,i,⇡t,i,Ti, etc.). However, it turns out that when we discuss episodic tasks we almost
never have to distinguish between di↵erent episodes. We are almost always considering a
particular single episode, or stating something that is true for all episodes. Accordingly,
in practice we almost always abuse notation slightly by dropping the explicit reference to

episode number. That is, we write Stto refer to St,i, and so on.
We need one other convention to obtain a single notation that covers both episodic
and continuing tasks. We have deﬁned the return as a sum over a ﬁnite number of terms
in one case (3.7) and as a sum over an inﬁnite number of terms in the other (3.8). These
two can be uniﬁed by considering episode termination to be the entering of a special
absorbing state that transitions only to itself and that generates only rewards of zero. For

example, consider the state transition diagram:R1 = +1S0S1R2 = +1S2R3 = +1R4 = 0R5 = 0. . .Here the solid square represents the special absorbing state corresponding to the end of an
episode. Starting from S0, we get the reward sequence +1 ,+1,+1,0,0,0,....S u m m i n g
these, we get the same return whether we sum over the ﬁrst Trewards (here T= 3) or
over the full inﬁnite sequence. This remains true even if we introduce discounting. Thus,

58 Chapter 3: Finite Markov Decision Processes
3.5 Policies and Value Functions
Almost all reinforcement learning algorithms involve estimating value functions —functions
of states (or of state–action pairs) that estimate how good it is for the agent to be in a
given state (or how good it is to perform a given action in a given state). The notion
of “how good” here is deﬁned in terms of future rewards that can be expected, or, to
be precise, in terms of expected return. Of course the rewards the agent can expect to
receive in the future depend on what actions it will take. Accordingly, value functions
are deﬁned with respect to particular ways of acting, called policies.
Formally, a policy is a mapping from states to probabilities of selecting each possible
action. If the agent is following policy ⇡at time t,t h e n ⇡(a|s) is the probability that
At=aifSt=s. Like p,⇡is an ordinary function; the “ |” in the middle of ⇡(a|s)
merely reminds that it deﬁnes a probability distribution over a2A(s) for each s2S.

3.5. Policies and Value Functions 59
These kinds of methods are presented in Chapter 5. Of course, if there are very many
states, then it may not be practical to keep separate averages for each state individually.
Instead, the agent would have to maintain v⇡andq⇡as parameterized functions (with
fewer parameters than states) and adjust the parameters to better match the observed
returns. This can also produce accurate estimates, although much depends on the nature
of the parameterized function approximator. These possibilities are discussed in Part II
of the book.
A fundamental property of value functions used throughout reinforcement learning and
dynamic programming is that they satisfy recursive relationships similar to that which
we have already established for the return (3.9) . For any policy ⇡and any state s,t h e
following consistency condition holds between the value of sand the value of its possible
successor states:
v⇡(s).=E⇡[Gt|St=s]
=E⇡[Rt+1+Gt+1|St=s] (by (3.9))
=X
a⇡(a|s)X
s0X
rp(s0,r|s, a)h

=X
a⇡(a|s)X
s0X
r+E⇡[Gt+1|St+1=s0]i
=X
a⇡(a|s)X
s0,rp(s0,r|s, a)h
r+v⇡(s0)i
,for all s2S, (3.14)
where it is implicit that the actions, a, are taken from the set A(s), that the next states,
s0, are taken from the set S(or from S+in the case of an episodic problem), and that
the rewards, r, are taken from the set R. Note also how in the last equation we have
merged the two sums, one over all the values of s0and the other over all the values of r,
into one sum over all the possible values of both. We use this kind of merged sum often

=X
a⇡(a|s)X
r+v⇡(s0)i
to simplify formulas. Note how the ﬁnal expression can be read easily as an expected
value. It is really a sum over all values of the three variables, a,s0, and r. For each triple,
we compute its probability, ⇡(a|s)p(s0,r|s, a), weight the quantity in brackets by that
probability, then sum over all possibilities to get an expected value.⇡ss0⇡rpa
Backup diagram for v⇡Equation (3.14) is the Bellman equation for v⇡.I te x p r e s s e s
a relationship between the value of a state and the values of

its successor states. Think of looking ahead from a state to its
possible successor states, as suggested by the diagram to the
right. Each open circle represents a state and each solid circle
represents a state–action pair. Starting from state s, the root
node at the top, the agent could take any of some set of actions—
three are shown in the diagram—based on its policy ⇡. From
each of these, the environment could respond with one of several next states, s0(two are

shown in the ﬁgure), along with a reward, r, depending on its dynamics given by the
function p. The Bellman equation (3.14) averages over all the possibilities, weighting each
by its probability of occurring. It states that the value of the start state must equal the
(discounted) value of the expected next state, plus the reward expected along the way.
The value function v⇡is the unique solution to its Bellman equation. We show in
subsequent chapters how this Bellman equation forms the basis of a number of ways to

60 Chapter 3: Finite Markov Decision Processes
compute, approximate, and learn v⇡. We call diagrams like that above backup diagrams
because they diagram relationships that form the basis of the update or backup operations
that are at the heart of reinforcement learning methods. These operations transfer
value information back to a state (or a state–action pair) from its successor states (or
state–action pairs). We use backup diagrams throughout the book to provide graphical
summaries of the algorithms we discuss. (Note that, unlike transition graphs, the state
nodes of backup diagrams do not necessarily represent distinct states; for example, a
state might be its own successor.)
Example 3.5: Gridworld Figure 3.2 (left) shows a rectangular gridworld representation
of a simple ﬁnite MDP. The cells of the grid correspond to the states of the environment. At
each cell, four actions are possible: north ,south ,east , and west , which deterministically

cause the agent to move one cell in the respective direction on the grid. Actions that
would take the agent o↵ the grid leave its location unchanged, but also result in a reward
of1. Other actions result in a reward of 0, except those that move the agent out of the
special states AandB. From state A, all four actions yield a reward of +10 and take the
agent to A0. From state B, all actions yield a reward of +5 and take the agent to B0.3.7. VALUE FUNCTIONS63

s,asas'ra's'r(b)(a)Figure 3.4: Backup diagrams for (a)v⇡and (b)q⇡.the states of the environment. At each cell, four actions are possible:north,south,east,a n dwest, which deterministically cause the agent to move onecell in the respective direction on the grid. Actions that would take the agentothe grid leave its location unchanged, but also result in a reward of1.Other actions result in a reward of 0, except those that move the agent outof the special states A and B. From state A, all four actions yield a reward of+10 and take the agent to A. From state B, all actions yield a reward of +5and take the agent to B.Suppose the agent selects all four actions with equal probability in allstates. Figure 3.5b shows the value function,v⇡, for this policy, for the dis-counted reward case with=0.9. This value function was computed by solv-ing the system of equations (3.10). Notice the negative values near the loweredge; these are the result of the high probability of hitting the edge of the gridthere under the

3.38.84.45.31.51.53.02.31.90.50.10.70.70.4-0.4-1.0-0.4-0.4-0.6-1.2-1.9-1.3-1.2-1.4-2.0ABA'B'+10+5Actions(a)(b)Figure 3.5: Grid example: (a) exceptional reward dynamics; (b) state-valuefunction for the equiprobable random policy.Figure 3.2: Gridworld example: exceptional reward dynamics (left) and state-value function
for the equiprobable random policy (right).
Suppose the agent selects all four actions with equal probability in all states. Figure 3.2
(right) shows the value function, v⇡, for this policy, for the discounted reward case with
=0.9. This value function was computed by solving the system of linear equations
(3.14). Notice the negative values near the lower edge; these are the result of the high
probability of hitting the edge of the grid there under the random policy. State Ais the
best state to be in under this policy, but its expected return is less than 10, its immediate
reward, because from Athe agent is taken to A0, from which it is likely to run into the

edge of the grid. State B, on the other hand, is valued more than 5, its immediate reward,
because from Bthe agent is taken to B0, which has a positive value. From B0the expected
penalty (negative reward) for possibly running into an edge is more than compensated
for by the expected gain for possibly stumbling onto AorB.
Exercise 3.14 The Bellman equation (3.14) must hold for each state for the value function
v⇡shown in Figure 3.2 (right) of Example 3.5. Show numerically that this equation holds

for the center state, valued at +0 .7, with respect to its four neighboring states, valued at
+2.3, +0 .4,0.4, and +0 .7. (These numbers are accurate only to one decimal place.) ⇤
Exercise 3.15 In the gridworld example, rewards are positive for goals, negative for
running into the edge of the world, and zero the rest of the time. Are the signs of these

3.5. Policies and Value Functions 61
rewards important, or only the intervals between them? Prove, using (3.8), that adding a
constant cto all the rewards adds a constant, vc, to the values of all states, and thus
does not a↵ect the relative values of any states under any policies. What is vcin terms
ofcand? ⇤
Exercise 3.16 Now consider adding a constant cto all the rewards in an episodic task,
such as maze running. Would this have any e↵ect, or would it leave the task unchanged
as in the continuing task above? Why or why not? Give an example. ⇤
Example 3.6: Golf To formulate playing a hole of golf as a reinforcement learning
task, we count a penalty (negative reward) of 1 for each stroke until we hit the ball
into the hole. The state is the location of the ball. The value of a state is the negative of
the number of strokes to the hole from that location. Our actions are how we aim and
swing at the ball, of course, and which club we select. Let us take the former as given

and consider just the choice of club, which we assume is either a putter or a driver. The
upper part of Figure 3.3 shows a possible state-value function, vputt(s), for the policy thatQ*(s,driver)Vputtsandgreen!1sand!2!2!3!4!1!5!6!4!3
!3!2!4
sandgreen!1sand!2!3!20
0!"!"vputtq⇤(s,driver)
Figure 3.3: A golf example: the state-value func-
tion for putting (upper) and the optimal action-
value function for using the driver (lower).always uses the putter. The terminal
state in-the-hole has a value of 0. From

!3!2!4
anywhere on the green we assume we can
make a putt; these states have value 1.
O↵ the green we cannot reach the hole
by putting, and the value is greater. If
we can reach the green from a state by
putting, then that state must have value
one less than the green’s value, that is,
2. For simplicity, let us assume we can
putt very precisely and deterministically,
but with a limited range. This gives us
the sharp contour line labeled 2i nt h e
ﬁgure; all locations between that line and

complete the hole. Similarly, any location
within putting range of the 2 contour
line must have a value of 3, and so
on to get all the contour lines shown in
the ﬁgure. Putting doesn’t get us out of
sand traps, so they have a value of 1.
Overall, it takes us six strokes to get from
the tee to the hole by putting.rs0s, aa0⇡pq⇡backup diagram
Exercise 3.17 What is the Bellman equation for action values, that
is, for q⇡? It must give the action value q⇡(s, a) in terms of the action
values, q⇡(s0,a0), of possible successors to the state–action pair ( s, a).
Hint: the backup diagram to the right corresponds to this equation.
Show the sequence of equations analogous to (3.14), but for action
values. ⇤

62 Chapter 3: Finite Markov Decision Processes
Exercise 3.18 The value of a state depends on the values of the actions possible in that
state and on how likely each action is to be taken under the current policy. We can
think of this in terms of a small backup diagram rooted at the state and considering each
possible action:staken withprobability⇡(a|s)v⇡(s)q⇡(s, a)a1a2a3
Give the equation corresponding to this intuition and diagram for the value at the root
node, v⇡(s), in terms of the value at the expected leaf node, q⇡(s, a), given St=s.T h i s
equation should include an expectation conditioned on following the policy, ⇡. Then give
a second equation in which the expected value is written out explicitly in terms of ⇡(a|s)
such that no expected value notation appears in the equation. ⇤
Exercise 3.19 The value of an action, q⇡(s, a), depends on the expected next reward and
the expected sum of the remaining rewards. Again we can think of this in terms of a

small backup diagram, this one rooted at an action (state–action pair) and branching to
the possible next states:s, aq⇡(s, a)s03s02s01r1r2r3expectedrewardsv⇡(s0)
Give the equation corresponding to this intuition and diagram for the action value,
q⇡(s, a), in terms of the expected next reward, Rt+1, and the expected next state value,
v⇡(St+1), given that St=sandAt=a. This equation should include an expectation but
notone conditioned on following the policy. Then give a second equation, writing out the

expected value explicitly in terms of p(s0,r|s, a)d e ﬁ n e db y (3.2) , such that no expected
value notation appears in the equation. ⇤
3.6 Optimal Policies and Optimal Value Functions
Solving a reinforcement learning task means, roughly, ﬁnding a policy that achieves a lot
of reward over the long run. For ﬁnite MDPs, we can precisely deﬁne an optimal policy
in the following way. Value functions deﬁne a partial ordering over policies. A policy ⇡is

deﬁned to be better than or equal to a policy ⇡0if its expected return is greater than
or equal to that of ⇡0for all states. In other words, ⇡⇡0if and only if v⇡(s)v⇡0(s)
for all s2S. There is always at least one policy that is better than or equal to all other
policies. This is an optimal policy . Although there may be more than one, we denote all
the optimal policies by ⇡⇤. They share the same state-value function, called the optimal
state-value function , denoted v⇤, and deﬁned as
v⇤(s).= max
⇡v⇡(s), (3.15)
for all s2S.

3.6. Optimal Policies and Optimal Value Functions 63
Optimal policies also share the same optimal action-value function , denoted q⇤, and
deﬁned as
q⇤(s, a).= max
⇡q⇡(s, a), (3.16)
for all s2Sand a2A(s). For the state–action pair ( s, a), this function gives the
expected return for taking action ain state sand thereafter following an optimal policy.
Thus, we can write q⇤in terms of v⇤as follows:
q⇤(s, a)=E[Rt+1+v⇤(St+1)|St=s, A t=a]. (3.17)
Example 3.7: Optimal Value Functions for Golf The lower part of Figure 3.3
shows the contours of a possible optimal action-value function q⇤(s,driver ). These are
the values of each state if we ﬁrst play a stroke with the driver and afterward select either
the driver or the putter, whichever is better. The driver enables us to hit the ball farther,
but with less accuracy. We can reach the hole in one shot using the driver only if we
are already very close; thus the 1 contour for q⇤(s,driver ) covers only a small portion

of the green. If we have two strokes, however, then we can reach the hole from much
farther away, as shown by the 2 contour. In this case we don’t have to drive all the way
to within the small 1 contour, but only to anywhere on the green; from there we can
use the putter. The optimal action-value function gives the values after committing to a
particular ﬁrst action, in this case, to the driver, but afterward using whichever actions
are best. The 3 contour is still farther out and includes the starting tee. From the tee,

the best sequence of actions is two drives and one putt, sinking the ball in three strokes.
Because v⇤is the value function for a policy, it must satisfy the self-consistency
condition given by the Bellman equation for state values (3.14). Because it is the optimal
value function, however, v⇤’s consistency condition can be written in a special form
without reference to any speciﬁc policy. This is the Bellman equation for v⇤, or the
Bellman optimality equation . Intuitively, the Bellman optimality equation expresses the

64 Chapter 3: Finite Markov Decision Processes
The backup diagrams in the ﬁgure below show graphically the spans of future states
and actions considered in the Bellman optimality equations for v⇤andq⇤. These are the
same as the backup diagrams for v⇡andq⇡presented earlier except that arcs have been
added at the agent’s choice points to represent that the maximum over that choice is
taken rather than the expected value given some policy. The backup diagram on the left
graphically represents the Bellman optimality equation (3.19) and the backup diagram
on the right graphically represents (3.20).ss0arrs0s, aa0maxmax(v⇤)(q⇤)
Figure 3.4: Backup diagrams for v⇤andq⇤
For ﬁnite MDPs, the Bellman optimality equation for v⇡(3.19) has a unique solution
independent of the policy. The Bellman optimality equation is actually a system of
equations, one for each state, so if there are nstates, then there are nequations in n
unknowns. If the dynamics pof the environment are known, then in principle one can

solve this system of equations for v⇤using any one of a variety of methods for solving
systems of nonlinear equations. One can solve a related set of equations for q⇤.
Once one has v⇤, it is relatively easy to determine an optimal policy. For each state
s, there will be one or more actions at which the maximum is obtained in the Bellman
optimality equation. Any policy that assigns nonzero probability only to these actions is
an optimal policy. You can think of this as a one-step search. If you have the optimal

value function, v⇤, then the actions that appear best after a one-step search will be optimal
actions. Another way of saying this is that any policy that is greedy with respect to the
optimal evaluation function v⇤is an optimal policy. The term greedy is used in computer
science to describe any search or decision procedure that selects alternatives based only
on local or immediate considerations, without considering the possibility that such a
selection may prevent future access to even better alternatives. Consequently, it describes

policies that select actions based only on their short-term consequences. The beauty of v⇤
is that if one uses it to evaluate the short-term consequences of actions—speciﬁcally, the
one-step consequences—then a greedy policy is actually optimal in the long-term sense in
which we are interested because v⇤already takes into account the reward consequences of
all possible future behavior. By means of v⇤, the optimal expected long-term return is
turned into a quantity that is locally and immediately available for each state. Hence, a

one-step-ahead search yields the long-term optimal actions.
Having q⇤makes choosing optimal actions even easier. With q⇤, the agent does not
even have to do a one-step-ahead search: for any state s, it can simply ﬁnd any action
that maximizes q⇤(s, a). The action-value function e↵ectively caches the results of all
one-step-ahead searches. It provides the optimal expected long-term return as a value
that is locally and immediately available for each state–action pair. Hence, at the cost of

3.6. Optimal Policies and Optimal Value Functions 65
representing a function of state–action pairs, instead of just of states, the optimal action-
value function allows optimal actions to be selected without having to know anything
about possible successor states and their values, that is, without having to know anything
about the environment’s dynamics.
Example 3.8: Solving the Gridworld Suppose we solve the Bellman equation for v⇤
for the simple grid task introduced in Example 3.5 and shown again in Figure 3.5 (left).
Recall that state Ais followed by a reward of +10 and transition to state A0, while state
Bis followed by a reward of +5 and transition to state B0. Figure 3.5 (middle) shows the
optimal value function, and Figure 3.5 (right) shows the corresponding optimal policies.
Where there are multiple arrows in a cell, all of the corresponding actions are optimal.

a) gridworldb) V*c) !*22.024.422.019.417.519.822.019.817.816.017.819.817.816.014.416.017.816.014.413.014.416.014.413.011.7ABA'B'+10+5v*π*Gridworldv⇤⇡⇤
Figure 3.5: Optimal solutions to the gridworld example.
Example 3.9: Bellman Optimality Equations for the Recycling Robot Using
(3.19), we can explicitly give the Bellman optimality equation for the recycling robot
example. To make things more compact, we abbreviate the states high andlow, and the
actions search ,wait , and recharge respectively by h,l,s,w, and re. Because there are

66 Chapter 3: Finite Markov Decision Processes
Explicitly solving the Bellman optimality equation provides one route to ﬁnding an
optimal policy, and thus to solving the reinforcement learning problem. However, this
solution is rarely directly useful. It is akin to an exhaustive search, looking ahead at
all possibilities, computing their probabilities of occurrence and their desirabilities in
terms of expected rewards. This solution relies on at least three assumptions that are
rarely true in practice: (1) we accurately know the dynamics of the environment; (2)
we have enough computational resources to complete the computation of the solution;
and (3) the Markov property. For the kinds of tasks in which we are interested, one is
generally not able to implement this solution exactly because various combinations of
these assumptions are violated. For example, although the ﬁrst and third assumptions
present no problems for the game of backgammon, the second is a major impediment.

Because the game has about 1020states, it would take thousands of years on today’s
fastest computers to solve the Bellman equation for v⇤, and the same is true for ﬁnding
q⇤. In reinforcement learning one typically has to settle for approximate solutions.
Many di↵erent decision-making methods can be viewed as ways of approximately
solving the Bellman optimality equation. For example, heuristic search methods can be
viewed as expanding the right-hand side of (3.19) several times, up to some depth, forming

a “tree” of possibilities, and then using a heuristic evaluation function to approximate
v⇤at the “leaf” nodes. (Heuristic search methods such as A⇤are almost always based
on the episodic case.) The methods of dynamic programming can be related even more
closely to the Bellman optimality equation. Many reinforcement learning methods can
be clearly understood as approximately solving the Bellman optimality equation, using
actual experienced transitions in place of knowledge of the expected transitions. We

consider a variety of such methods in the following chapters.
Exercise 3.20 Draw or describe the optimal state-value function for the golf example. ⇤
Exercise 3.21 Draw or describe the contours of the optimal action-value function for
putting, q⇤(s,putter ), for the golf example. ⇤+200+1leftrightExercise 3.22 Consider the continuing MDP shown on to the
right. The only decision to be made is that in the top state,
where two actions are available, leftandright .T h e n u m b e r s

show the rewards that are received deterministically after
each action. There are exactly two deterministic policies,
⇡leftand⇡right . What policy is optimal if = 0? If =0.9?
If=0.5? ⇤
Exercise 3.23 Give the Bellman equation for q⇤for the recycling robot. ⇤
Exercise 3.24 Figure 3.5 gives the optimal value of the best state of the gridworld as
24.4, to one decimal place. Use your knowledge of the optimal policy and (3.8) to express
this value symbolically, and then to compute it to three decimal places. ⇤

3.7. Optimality and Approximation 67
Exercise 3.27 Give an equation for ⇡⇤in terms of q⇤. ⇤
Exercise 3.28 Give an equation for ⇡⇤in terms of v⇤and the four-argument p. ⇤
Exercise 3.29 Rewrite the four Bellman equations for the four value functions ( v⇡,v⇤,q⇡,
andq⇤) in terms of the three argument function p(3.4) and the two-argument function r
(3.5). ⇤
3.7 Optimality and Approximation
We have deﬁned optimal value functions and optimal policies. Clearly, an agent that
learns an optimal policy has done very well, but in practice this rarely happens. For
the kinds of tasks in which we are interested, optimal policies can be generated only
with extreme computational cost. A well-deﬁned notion of optimality organizes the
approach to learning we describe in this book and provides a way to understand the
theoretical properties of various learning algorithms, but it is an ideal that agents can
only approximate to varying degrees. As we discussed above, even if we have a complete

and accurate model of the environment’s dynamics, it is usually not possible to simply
compute an optimal policy by solving the Bellman optimality equation. For example,
board games such as chess are a tiny fraction of human experience, yet large, custom-
designed computers still cannot compute the optimal moves. A critical aspect of the
problem facing the agent is always the computational power available to it, in particular,
the amount of computation it can perform in a single time step.
The memory available is also an important constraint. A large amount of memory

is often required to build up approximations of value functions, policies, and models.
In tasks with small, ﬁnite state sets, it is possible to form these approximations using
arrays or tables with one entry for each state (or state–action pair). This we call the
tabular case, and the corresponding methods we call tabular methods. In many cases
of practical interest, however, there are far more states than could possibly be entries
in a table. In these cases the functions must be approximated, using some sort of more

compact parameterized function representation.
Our framing of the reinforcement learning problem forces us to settle for approxi-
mations. However, it also presents us with some unique opportunities for achieving
useful approximations. For example, in approximating optimal behavior, there may be
many states that the agent faces with such a low probability that selecting suboptimal
actions for them has little impact on the amount of reward the agent receives. Tesauro’s
backgammon player, for example, plays with exceptional skill even though it might make

very bad decisions on board conﬁgurations that never occur in games against experts. In
fact, it is possible that TD-Gammon makes bad decisions for a large fraction of the game’s
state set. The online nature of reinforcement learning makes it possible to approximate
optimal policies in ways that put more e↵ort into learning to make good decisions for
frequently encountered states, at the expense of less e↵ort for infrequently encountered
states. This is one key property that distinguishes reinforcement learning from other
approaches to approximately solving MDPs.

68 Chapter 3: Finite Markov Decision Processes
3.8 Summary
Let us summarize the elements of the reinforcement learning problem that we have
presented in this chapter. Reinforcement learning is about learning from interaction
how to behave in order to achieve a goal. The reinforcement learning agent and its
environment interact over a sequence of discrete time steps. The speciﬁcation of their
interface deﬁnes a particular task: the actions are the choices made by the agent; the
states are the basis for making the choices; and the rewards are the basis for evaluating
the choices. Everything inside the agent is completely known and controllable by the
agent; everything outside is incompletely controllable but may or may not be completely
known. A policy is a stochastic rule by which the agent selects actions as a function of
states. The agent’s objective is to maximize the amount of reward it receives over time.
When the reinforcement learning setup described above is formulated with well deﬁned

transition probabilities it constitutes a Markov decision process (MDP). A ﬁnite MDP is
an MDP with ﬁnite state, action, and (as we formulate it here) reward sets. Much of the
current theory of reinforcement learning is restricted to ﬁnite MDPs, but the methods
and ideas apply more generally.
The return is the function of future rewards that the agent seeks to maximize (in
expected value). It has several di↵erent deﬁnitions depending upon the nature of the
task and whether one wishes to discount delayed reward. The undiscounted formulation

is appropriate for episodic tasks , in which the agent–environment interaction breaks
naturally into episodes ; the discounted formulation is appropriate for continuing tasks ,i n
which the interaction does not naturally break into episodes but continues without limit.
We try to deﬁne the returns for the two kinds of tasks such that one set of equations can
apply to both the episodic and continuing cases.
A policy’s value functions assign to each state, or state–action pair, the expected return

from that state, or state–action pair, given that the agent uses the policy. The optimal
value functions assign to each state, or state–action pair, the largest expected return
achievable by any policy. A policy whose value functions are optimal is an optimal policy .
Whereas the optimal value functions for states and state–action pairs are unique for a
given MDP, there can be many optimal policies. Any policy that is greedy with respect to

the optimal value functions must be an optimal policy. The Bellman optimality equations
are special consistency conditions that the optimal value functions must satisfy and that
can, in principle, be solved for the optimal value functions, from which an optimal policy
can be determined with relative ease.
A reinforcement learning problem can be posed in a variety of di↵erent ways depending
on assumptions about the level of knowledge initially available to the agent. In problems
ofcomplete knowledge , the agent has a complete and accurate model of the environment’s

dynamics. If the environment is an MDP, then such a model consists of the complete four-
argument dynamics function p(3.2) . In problems of incomplete knowledge , a complete
and perfect model of the environment is not available.
Even if the agent has a complete and accurate environment model, the agent is
typically unable to perform enough computation per time step to fully use it. The
memory available is also an important constraint. Memory may be required to build

Bibliographical and Historical Remarks 69
up accurate approximations of value functions, policies, and models. In most cases of
practical interest there are far more states than could possibly be entries in a table, and
approximations must be made.
A well-deﬁned notion of optimality organizes the approach to learning we describe in
this book and provides a way to understand the theoretical properties of various learning
algorithms, but it is an ideal that reinforcement learning agents can only approximate
to varying degrees. In reinforcement learning we are very much concerned with cases in
which optimal solutions cannot be found but must be approximated in some way.
The reinforcement learning problem is deeply indebted to the idea of Markov decision
processes (MDPs) from the ﬁeld of optimal control. These historical inﬂuences and other
major inﬂuences from psychology are described in the brief history given in Chapter 1.

Reinforcement learning adds to MDPs a focus on approximation and incomplete infor-
mation for realistically large problems. MDPs and the reinforcement learning problem
are only weakly linked to traditional learning and decision-making problems in artiﬁcial
intelligence. However, artiﬁcial intelligence is now vigorously exploring MDP formulations
for planning and decision making from a variety of perspectives. MDPs are more general
than previous formulations used in artiﬁcial intelligence in that they permit more general
kinds of goals and uncertainty.

The theory of MDPs is treated by, for example, Bertsekas (2005), White (1969), Whittle
(1982, 1983), and Puterman (1994). A particularly compact treatment of the ﬁnite case
is given by Ross (1983). MDPs are also studied under the heading of stochastic optimal
control, where adaptive optimal control methods are most closely related to reinforcement
learning (e.g., Kumar, 1985; Kumar and Varaiya, 1986).
The theory of MDPs evolved from e↵orts to understand the problem of making sequences

of decisions under uncertainty, where each decision can depend on the previous decisions
and their outcomes. It is sometimes called the theory of multistage decision processes,
or sequential decision processes, and has roots in the statistical literature on sequential
sampling beginning with the papers by Thompson (1933, 1934) and Robbins (1952) that
we cited in Chapter 2 in connection with bandit problems (which are prototypical MDPs
if formulated as multiple-situation problems).

The earliest instance of which we are aware in which reinforcement learning was
discussed using the MDP formalism is Andreae’s (1969b) description of a uniﬁed view of
learning machines. Witten and Corbin (1973) experimented with a reinforcement learning
system later analyzed by Witten (1977, 1976a) using the MDP formalism. Although
he did not explicitly mention MDPs, Werbos (1977) suggested approximate solution
methods for stochastic optimal control problems that are related to modern reinforcement

learning methods (see also Werbos, 1982, 1987, 1988, 1989, 1992). Although Werbos’s
ideas were not widely recognized at the time, they were prescient in emphasizing the
importance of approximately solving optimal control problems in a variety of domains,
including artiﬁcial intelligence. The most inﬂuential integration of reinforcement learning
and MDPs is due to Watkins (1989).

70 Chapter 3: Finite Markov Decision Processes
3.1 Our characterization of the dynamics of an MDP in terms of p(s0,r|s, a)i s
slightly unusual. It is more common in the MDP literature to describe the
dynamics in terms of the state transition probabilities p(s0|s, a) and expected
next rewards r(s, a). In reinforcement learning, however, we more often have
to refer to individual actual or sample rewards (rather than just their expected
values). Our notation also makes it plainer that StandRtare in general jointly
determined, and thus must have the same time index. In teaching reinforcement
learning, we have found our notation to be more straightforward conceptually
and easier to understand.
For a good intuitive discussion of the system-theoretic concept of state, see
Minsky (1967).
The bioreactor example is based on the work of Ungar (1990) and Miller and
Williams (1992). The recycling robot example was inspired by the can-collecting
robot built by Jonathan Connell (1989). Kober and Peters (2012) present a

collection of robotics applications of reinforcement learning.
3.2 The reward hypothesis was suggested by Michael Littman (personal communica-
tion).
3.3–4 The terminology of episodic andcontinuing tasks is di↵erent from that usually
used in the MDP literature. In that literature it is common to distinguish
three types of tasks: (1) ﬁnite-horizon tasks, in which interaction terminates
after a particular ﬁxed number of time steps; (2) indeﬁnite-horizon tasks, in

tion).
which interaction can last arbitrarily long but must eventually terminate; and
(3) inﬁnite-horizon tasks, in which interaction does not terminate. Our episodic
and continuing tasks are similar to indeﬁnite-horizon and inﬁnite-horizon tasks,
respectively, but we prefer to emphasize the di↵erence in the nature of the
interaction. This di↵erence seems more fundamental than the di↵erence in the
objective functions emphasized by the usual terms. Often episodic tasks use
an indeﬁnite-horizon objective function and continuing tasks an inﬁnite-horizon

objective function, but we see this as a common coincidence rather than a
fundamental di↵erence.
The pole-balancing example is from Michie and Chambers (1968) and Barto,
Sutton, and Anderson (1983).
3.5–6 Assigning value on the basis of what is good or bad in the long run has ancient
roots. In control theory, mapping states to numerical values representing the
long-term consequences of control decisions is a key part of optimal control theory,
which was developed in the 1950s by extending nineteenth century state-function

theories of classical mechanics (see, e.g., Schultz and Melsa, 1967). In describing
how a computer could be programmed to play chess, Shannon (1950) suggested
using an evaluation function that took into account the long-term advantages
and disadvantages of chess positions.
Watkins’s (1989) Q-learning algorithm for estimating q⇤(Chapter 6) made action-
value functions an important part of reinforcement learning, and consequently

Bibliographical and Historical Remarks 71
these functions are often called “Q-functions.” But the idea of an action-value
function is much older than this. Shannon (1950) suggested that a function
h(P,M) could be used by a chess-playing program to decide whether a move M
in position Pis worth exploring. Michie’s (1961, 1963) MENACE system and
Michie and Chambers’s (1968) BOXES system can be understood as estimating
action-value functions. In classical physics, Hamilton’s principal function is
an action-value function; Newtonian dynamics are greedy with respect to this
function (e.g., Goldstein, 1957). Action-value functions also played a central role
in Denardo’s (1967) theoretical treatment of dynamic programming in terms of
contraction mappings.
The Bellman optimality equation (for v⇤) was popularized by Richard Bellman
(1957a), who called it the “basic functional equation.” The counterpart of the
Bellman optimality equation for continuous time and state problems is known

as the Hamilton–Jacobi–Bellman equation (or often just the Hamilton–Jacobi
equation), indicating its roots in classical physics (e.g., Schultz and Melsa, 1967).
The golf example was suggested by Chris Watkins.

Chapter 4
Dynamic Programming
The term dynamic programming (DP) refers to a collection of algorithms that can be
used to compute optimal policies given a perfect model of the environment as a Markov
decision process (MDP). Classical DP algorithms are of limited utility in reinforcement
learning both because of their assumption of a perfect model and because of their great
computational expense, but they are still important theoretically. DP provides an essential
foundation for the understanding of the methods presented in the rest of this book. In
fact, all of these methods can be viewed as attempts to achieve much the same e↵ect as
DP, only with less computation and without assuming a perfect model of the environment.
We usually assume that the environment is a ﬁnite MDP. That is, we assume that its
state, action, and reward sets, S,A, and R, are ﬁnite, and that its dynamics are given by a
set of probabilities p(s0,r|s, a), for all s2S,a2A(s),r2R, and s02S+(S+isSplus a

terminal state if the problem is episodic). Although DP ideas can be applied to problems
with continuous state and action spaces, exact solutions are possible only in special cases.
A common way of obtaining approximate solutions for tasks with continuous states and
actions is to quantize the state and action spaces and then apply ﬁnite-state DP methods.
The methods we explore in Chapter 9 are applicable to continuous problems and are a
signiﬁcant extension of that approach.
The key idea of DP, and of reinforcement learning generally, is the use of value functions

74 Chapter 4: Dynamic Programming
for all s2S,a2A(s), and s02S+. As we shall see, DP algorithms are obtained by
turning Bellman equations such as these into assignments, that is, into update rules for
improving approximations of the desired value functions.
4.1 Policy Evaluation (Prediction)
First we consider how to compute the state-value function v⇡for an arbitrary policy ⇡.
This is called policy evaluation in the DP literature. We also refer to it as the prediction
problem . Recall from Chapter 3 that, for all s2S,
=E⇡[Rt+1+Gt+1|St=s] (from (3.9))
=E⇡[Rt+1+v⇡(St+1)|St=s] (4.3)
=X
a⇡(a|s)X
r+v⇡(s0)i
, (4.4)
where ⇡(a|s) is the probability of taking action ain state sunder policy ⇡, and the
expectations are subscripted by ⇡to indicate that they are conditional on ⇡being followed.
The existence and uniqueness of v⇡are guaranteed as long as either <1 or eventual
termination is guaranteed from all states under the policy ⇡.

=X
a⇡(a|s)X
r+v⇡(s0)i
, (4.4)
If the environment’s dynamics are completely known, then (4.4) is a system of |S|
simultaneous linear equations in |S|unknowns (the v⇡(s),s2S). In principle, its solution
is a straightforward, if tedious, computation. For our purposes, iterative solution methods
are most suitable. Consider a sequence of approximate value functions v0,v1,v2,..., each
mapping S+toR(the real numbers). The initial approximation, v0, is chosen arbitrarily

(except that the terminal state, if any, must be given value 0), and each successive
approximation is obtained by using the Bellman equation for v⇡(4.4) as an update rule:
vk+1(s).=E⇡[Rt+1+vk(St+1)|St=s]
=X
a⇡(a|s)X
r+vk(s0)i
, (4.5)
for all s2S. Clearly, vk=v⇡is a ﬁxed point for this update rule because the Bellman
equation for v⇡assures us of equality in this case. Indeed, the sequence {vk}can be
shown in general to converge to v⇡ask!1under the same conditions that guarantee

=X
a⇡(a|s)X
r+vk(s0)i
, (4.5)
the existence of v⇡. This algorithm is called iterative policy evaluation .
To produce each successive approximation, vk+1from vk, iterative policy evaluation
applies the same operation to each state s: it replaces the old value of swith a new value
obtained from the old values of the successor states of s, and the expected immediate
rewards, along all the one-step transitions possible under the policy being evaluated. We
call this kind of operation an expected update . Each iteration of iterative policy evalu-

ation updates the value of every state once to produce the new approximate value function

4.1. Policy Evaluation (Prediction) 75
vk+1. There are several di↵erent kinds of expected updates, depending on whether a
state (as here) or a state–action pair is being updated, and depending on the precise way
the estimated values of the successor states are combined. All the updates done in DP
algorithms are called expected updates because they are based on an expectation over all
possible next states rather than on a sample next state. The nature of an update can
be expressed in an equation, as above, or in a backup diagram like those introduced in
Chapter 3. For example, the backup diagram corresponding to the expected update used
in iterative policy evaluation is shown on page 59.
To write a sequential computer program to implement iterative policy evaluation as
given by (4.5) you would have to use two arrays, one for the old values, vk(s), and one
for the new values, vk+1(s). With two arrays, the new values can be computed one by

one from the old values without the old values being changed. Of course it is easier to
use one array and update the values “in place,” that is, with each new value immediately
overwriting the old one. Then, depending on the order in which the states are updated,
sometimes new values are used instead of old ones on the right-hand side of (4.5). This
in-place algorithm also converges to v⇡; in fact, it usually converges faster than the
two-array version, as you might expect, because it uses new data as soon as they are

available. We think of the updates as being done in a sweep through the state space. For
the in-place algorithm, the order in which states have their values updated during the
sweep has a signiﬁcant inﬂuence on the rate of convergence. We usually have the in-place
version in mind when we think of DP algorithms.
A complete in-place version of iterative policy evaluation is shown in pseudocode in
the box below. Note how it handles termination. Formally, iterative policy evaluation
converges only in the limit, but in practice it must be halted short of this. The pseudocode

76 Chapter 4: Dynamic Programming
Example 4.1 Consider the 4 ⇥4 gridworld shown below.
actionsr  =  !1on all transitions1234567891011121314Rt=1
The nonterminal states are S={1,2,...,14}. There are four actions possible in each
state, A={up,down ,right ,left }, which deterministically cause the corresponding
state transitions, except that actions that would take the agent o↵ the grid in fact leave
the state unchanged. Thus, for instance, p(6,1|5,right ) = 1, p(7,1|7,right ) = 1,
and p(10,r|5,right ) = 0 for all r2R. This is an undiscounted, episodic task. The
reward is 1 on all transitions until the terminal state is reached. The terminal state is
shaded in the ﬁgure (although it is shown in two places, it is formally one state). The
expected reward function is thus r(s, a, s0)=1 for all states s, s0and actions a. Suppose
the agent follows the equiprobable random policy (all actions equally likely). The left side
of Figure 4.1 shows the sequence of value functions {vk}computed by iterative policy

evaluation. The ﬁnal estimate is in fact v⇡, which in this case gives for each state the
negation of the expected number of steps from that state until termination.
Exercise 4.1 In Example 4.1, if ⇡is the equiprobable random policy, what is q⇡(11,down )?
What is q⇡(7,down )? ⇤
Exercise 4.2 In Example 4.1, suppose a new state 15 is added to the gridworld just below
state 13, and its actions, left ,up,right , and down , take the agent to states 12, 13, 14,
and 15, respectively. Assume that the transitions from the original states are unchanged.

What, then, is v⇡(15) for the equiprobable random policy? Now suppose the dynamics of
state 13 are also changed, such that action down from state 13 takes the agent to the new
state 15. What is v⇡(15) for the equiprobable random policy in this case? ⇤
Exercise 4.3 What are the equations analogous to (4.3), (4.4), and (4.5) for the action-
value function q⇡and its successive approximation by a sequence of functions q0,q1,q2,...?
⇤
4.2 Policy Improvement
Our reason for computing the value function for a policy is to help ﬁnd better policies.

⇤
Suppose we have determined the value function v⇡for an arbitrary deterministic policy
⇡. For some state swe would like to know whether or not we should change the policy
to deterministically choose an action a6=⇡(s). We know how good it is to follow the
current policy from s—that is v⇡(s)—but would it be better or worse to change to the
new policy? One way to answer this question is to consider selecting ainsand thereafter

4.2. Policy Improvement 77 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.7-2.0-2.0-1.7-2.0-2.0-2.0-2.0-2.0-2.0-1.7-2.0-2.0-1.7-2.4-2.9-3.0-2.4-2.9-3.0-2.9-2.9-3.0-2.9-2.4-3.0-2.9-2.4-6.1-8.4-9.0-6.1-7.7-8.4-8.4-8.4-8.4-7.7-6.1-9.0-8.4-6.1-14.-20.-22.-14.-18.-20.-20.-20.-20.-18.-14.-22.-20.-14.Vk  for theRandom PolicyGreedy Policyw.r.t. Vkk = 0k = 1k = 2
k = 10k = !k = 3optimal policyrandom policy 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0vk      for therandom policyvkgreedy policy    w.r.t.vk
Figure 4.1: Convergence of iterative policy evaluation on a small gridworld. The left column is
the sequence of approximations of the state-value function for the random policy (all actions
equally likely). The right column is the sequence of greedy policies corresponding to the value
function estimates (arrows are shown for all actions achieving the maximum, and the numbers

shown are rounded to two signiﬁcant digits). The last policy is guaranteed only to be an
improvement over the random policy, but in this case it, and all policies after the third iteration,
are optimal.

78 Chapter 4: Dynamic Programming
following the existing policy, ⇡. The value of this way of behaving is
q⇡(s, a).=E[Rt+1+v⇡(St+1)|St=s, A t=a] (4.6)
=X
r+v⇡(s0)i
.
The key criterion is whether this is greater than or less than v⇡(s). If it is greater—that
is, if it is better to select aonce in sand thereafter follow ⇡than it would be to follow
⇡all the time—then one would expect it to be better still to select aevery time sis
encountered, and that the new policy would in fact be a better one overall.
That this is true is a special case of a general result called the policy improvement
theorem . Let ⇡and⇡0be any pair of deterministic policies such that, for all s2S,
q⇡(s, ⇡0(s))v⇡(s). (4.7)
Then the policy ⇡0must be as good as, or better than, ⇡. That is, it must obtain greater
or equal expected return from all states s2S:
v⇡0(s)v⇡(s). (4.8)
Moreover, if there is strict inequality of (4.7) at any state, then there must be strict

inequality of (4.8) at that state. This result applies in particular to the two policies
that we considered in the previous paragraph, an original deterministic policy, ⇡, and a
changed policy, ⇡0, that is identical to ⇡except that ⇡0(s)=a6=⇡(s). Obviously, (4.7)
holds at all states other than s.T h u s , i f q⇡(s, a)>v ⇡(s), then the changed policy is
indeed better than ⇡.
The idea behind the proof of the policy improvement theorem is easy to understand.
Starting from (4.7), we keep expanding the q⇡side with (4.6) and reapplying (4.7) until
we get v⇡0(s):
v⇡(s)q⇡(s, ⇡0(s))

4.2. Policy Improvement 79
to consider changes at allstates and to allpossible actions, selecting at each state the
action that appears best according to q⇡(s, a). In other words, to consider the new greedy
policy, ⇡0, given by
⇡0(s).= argmax
aq⇡(s, a)
= argmax
aE[Rt+1+v⇡(St+1)|St=s, A t=a] (4.9)
= argmax
aX
r+v⇡(s0)i
,
where argmaxadenotes the value of aat which the expression that follows is maximized
(with ties broken arbitrarily). The greedy policy takes the action that looks best in the
short term—after one step of lookahead—according to v⇡. By construction, the greedy
policy meets the conditions of the policy improvement theorem (4.7), so we know that it
is as good as, or better than, the original policy. The process of making a new policy that
improves on an original policy, by making it greedy with respect to the value function of
the original policy, is called policy improvement .
Suppose the new greedy policy, ⇡0, is as good as, but not better than, the old policy ⇡.

Then v⇡=v⇡0, and from (4.9) it follows that for all s2S:
v⇡0(s) = max
aE[Rt+1+v⇡0(St+1)|St=s, A t=a]
= max
aX
r+v⇡0(s0)i
.
But this is the same as the Bellman optimality equation (4.1), and therefore, v⇡0must be
v⇤, and both ⇡and⇡0must be optimal policies. Policy improvement thus must give us a
strictly better policy except when the original policy is already optimal.
So far in this section we have considered the special case of deterministic policies.

= max
aX
.
In the general case, a stochastic policy ⇡speciﬁes probabilities, ⇡(a|s), for taking each
action, a, in each state, s. We will not go through the details, but in fact all the ideas of
this section extend easily to stochastic policies. In particular, the policy improvement
theorem carries through as stated for the stochastic case. In addition, if there are ties in
policy improvement steps such as (4.9)—that is, if there are several actions at which the
maximum is achieved—then in the stochastic case we need not select a single action from

among them. Instead, each maximizing action can be given a portion of the probability
of being selected in the new greedy policy. Any apportioning scheme is allowed as long
as all submaximal actions are given zero probability.
The last row of Figure 4.1 shows an example of policy improvement for stochastic
policies. Here the original policy, ⇡, is the equiprobable random policy, and the new
policy, ⇡0, is greedy with respect to v⇡. The value function v⇡is shown in the bottom-left
diagram and the set of possible ⇡0is shown in the bottom-right diagram. The states

maximum of ﬁve cars can be moved from one location to the other in one night. We take
the discount rate to be =0.9 and formulate this as a continuing ﬁnite MDP, where
the time steps are days, the state is the number of cars at each location at the end of
the day, and the actions are the net numbers of cars moved between the two locations
overnight. Figure 4.2 shows the sequence of policies found by policy iteration starting
from the policy that never moves any cars.
4V612#Cars at second location042020020#Cars at first location115
!1!2-4432432!3005
!1!2!3!412340"1"0"2
!3!4!201234
!1"32
!4!3!201345
!1"4
#Cars at second location#Cars at first location5
200020v⇡4⇡0⇡1⇡2⇡3⇡4
Figure 4.2: The sequence of policies found by policy iteration on Jack’s car rental problem,
and the ﬁnal state-value function. The ﬁrst ﬁve diagrams show, for each number of cars at
each location at the end of the day, the number of cars to be moved from the ﬁrst location to

!1"32
!1"4
the second (negative numbers indicate transfers from the second location to the ﬁrst). Each
successive policy is a strict improvement over the previous policy, and the last policy is optimal.

82 Chapter 4: Dynamic Programming
Policy iteration often converges in surprisingly few iterations, as the example of Jack’s
car rental illustrates, and as is also illustrated by the example in Figure 4.1. The bottom-
left diagram of Figure 4.1 shows the value function for the equiprobable random policy,
and the bottom-right diagram shows a greedy policy for this value function. The policy
improvement theorem assures us that these policies are better than the original random
policy. In this case, however, these policies are not just better, but optimal, proceeding
to the terminal states in the minimum number of steps. In this example, policy iteration
would ﬁnd the optimal policy after just one iteration.
Exercise 4.4 The policy iteration algorithm on page 80 has a subtle bug in that it may
never terminate if the policy continually switches between two or more policies that are
equally good. This is ok for pedagogy, but not for actual use. Modify the pseudocode so
that convergence is guaranteed. ⇤

Exercise 4.5 How would policy iteration be deﬁned for action values? Give a complete
algorithm for computing q⇤, analogous to that on page 80 for computing v⇤. Please pay
special attention to this exercise, because the ideas involved will be used throughout the
rest of the book. ⇤
Exercise 4.6 Suppose you are restricted to considering only policies that are "-soft,
meaning that the probability of selecting each action in each state, s, is at least "/|A(s)|.
Describe qualitatively the changes that would be required in each of the steps 3, 2, and 1,

in that order, of the policy iteration algorithm for v⇤on page 80. ⇤
Exercise 4.7 (programming) Write a program for policy iteration and re-solve Jack’s car
rental problem with the following changes. One of Jack’s employees at the ﬁrst location
rides a bus home each night and lives near the second location. She is happy to shuttle
one car to the second location for free. Each additional car still costs $2, as do all cars
moved in the other direction. In addition, Jack has limited parking space at each location.

If more than 10 cars are kept overnight at a location (after any moving of cars), then an
additional cost of $4 must be incurred to use a second parking lot (independent of how
many cars are kept there). These sorts of nonlinearities and arbitrary dynamics often
occur in real problems and cannot easily be handled by optimization methods other than
dynamic programming. To check your program, ﬁrst replicate the results given for the
original problem. ⇤
4.4 Value Iteration
One drawback to policy iteration is that each of its iterations involves policy evaluation,

which may itself be a protracted iterative computation requiring multiple sweeps through
the state set. If policy evaluation is done iteratively, then convergence exactly to v⇡
occurs only in the limit. Must we wait for exact convergence, or can we stop short of
that? The example in Figure 4.1 certainly suggests that it may be possible to truncate
policy evaluation. In that example, policy evaluation iterations beyond the ﬁrst three
have no e↵ect on the corresponding greedy policy.

In fact, the policy evaluation step of policy iteration can be truncated in several ways
without losing the convergence guarantees of policy iteration. One important special

4.4. Value Iteration 83
case is when policy evaluation is stopped after just one sweep (one update of each state).
This algorithm is called value iteration . It can be written as a particularly simple update
operation that combines the policy improvement and truncated policy evaluation steps:
vk+1(s).= max
aE[Rt+1+vk(St+1)|St=s, A t=a]
= max
aX
r+vk(s0)i
, (4.10)
for all s2S. For arbitrary v0,t h es e q u e n c e {vk}can be shown to converge to v⇤under
the same conditions that guarantee the existence of v⇤.
Another way of understanding value iteration is by reference to the Bellman optimality
equation (4.1). Note that value iteration is obtained simply by turning the Bellman
optimality equation into an update rule. Also note how the value iteration update is
identical to the policy evaluation update (4.5) except that it requires the maximum to be
taken over all actions. Another way of seeing this close relationship is to compare the

backup diagrams for these algorithms on page 59 (policy evaluation) and on the left of
Figure 3.4 (value iteration). These two are the natural backup operations for computing
v⇡andv⇤.
Finally, let us consider how value iteration terminates. Like policy evaluation, value
iteration formally requires an inﬁnite number of iterations to converge exactly to v⇤.I n
practice, we stop once the value function changes by only a small amount in a sweep.
The box below shows a complete algorithm with this kind of termination condition.

84 Chapter 4: Dynamic Programming
these updates, this just means that the max operation is added to some sweeps of policy
evaluation. All of these algorithms converge to an optimal policy for discounted ﬁnite
MDPs.
Example 4.3: Gambler’s Problem A gambler has the opportunity to make bets on
the outcomes of a sequence of coin ﬂips. If the coin comes up heads, he wins as many
dollars as he has staked on that ﬂip; if it is tails, he loses his stake. The game ends
when the gambler wins by reaching his goal of $100, or loses by running out of money.
On each ﬂip, the gambler must decide what portion of his capital to stake, in integer
numbers of dollars. This problem can be formulated as an undiscounted, episodic, ﬁnite
99755025111020304050100.20.40.60.81
25507599
CapitalCapitalValueestimates
Finalpolicy(stake)sweep 1sweep 2sweep 3sweep 32Final valuefunction
Figure 4.3: The solution to the gambler’s problem
forph=0.4. The upper graph shows the value func-
tion found by successive sweeps of value iteration. The

25507599
lower graph shows the ﬁnal policy.MDP. The state is the gambler’s capi-
tal,s2{1,2,..., 99}and the actions
are stakes, a2{0,1,...,min(s,100
s)}. The reward is zero on all transi-
tions except those on which the gam-
bler reaches his goal, when it is +1.
The state-value function then gives
the probability of winning from each
state. A policy is a mapping from
levels of capital to stakes. The opti-
mal policy maximizes the probability
of reaching the goal. Let phdenote
the probability of the coin coming up
heads. If phis known, then the en-

tire problem is known and it can be
solved, for instance, by value iteration.
Figure 4.3 shows the change in the
value function over successive sweeps
of value iteration, and the ﬁnal policy
found, for the case of ph=0.4. This
policy is optimal, but not unique. In
fact, there is a whole family of opti-
mal policies, all corresponding to ties
for the argmax action selection with
respect to the optimal value function.
Can you guess what the entire family
looks like?

Exercise 4.8 Why does the optimal policy for the gambler’s problem have such a curious
form? In particular, for capital of 50 it bets it all on one ﬂip, but for capital of 51 it does
not. Why is this a good policy? ⇤
Exercise 4.9 (programming) Implement value iteration for the gambler’s problem and
solve it for ph=0.25 and ph=0.55. In programming, you may ﬁnd it convenient to
introduce two dummy states corresponding to termination with capital of 0 and 100,

giving them values of 0 and 1 respectively. Show your results graphically, as in Figure 4.3.

4.5. Asynchronous Dynamic Programming 85
Are your results stable as ✓!0? ⇤
Exercise 4.10 What is the analog of the value iteration update (4.10) for action values,
qk+1(s, a)? ⇤
4.5 Asynchronous Dynamic Programming
A major drawback to the DP methods that we have discussed so far is that they involve
operations over the entire state set of the MDP, that is, they require sweeps of the state
set. If the state set is very large, then even a single sweep can be prohibitively expensive.
For example, the game of backgammon has over 1020states. Even if we could perform
the value iteration update on a million states per second, it would take over a thousand
years to complete a single sweep.
Asynchronous DP algorithms are in-place iterative DP algorithms that are not organized
in terms of systematic sweeps of the state set. These algorithms update the values of
states in any order whatsoever, using whatever values of other states happen to be

available. The values of some states may be updated several times before the values of
others are updated once. To converge correctly, however, an asynchronous algorithm
must continue to update the values of all the states: it can’t ignore any state after some
point in the computation. Asynchronous DP algorithms allow great ﬂexibility in selecting
states to update.
For example, one version of asynchronous value iteration updates the value, in place, of
only one state, sk, on each step, k, using the value iteration update (4.10). If 0 <1,

asymptotic convergence to v⇤is guaranteed given only that all states occur in the
sequence {sk}an inﬁnite number of times (the sequence could even be stochastic). (In
the undiscounted episodic case, it is possible that there are some orderings of updates
that do not result in convergence, but it is relatively easy to avoid these.) Similarly, it
is possible to intermix policy evaluation and value iteration updates to produce a kind
of asynchronous truncated policy iteration. Although the details of this and other more

unusual DP algorithms are beyond the scope of this book, it is clear that a few di↵erent
updates form building blocks that can be used ﬂexibly in a wide variety of sweepless DP
Of course, avoiding sweeps does not necessarily mean that we can get away with less
computation. It just means that an algorithm does not need to get locked into any
hopelessly long sweep before it can make progress improving a policy. We can try to
take advantage of this ﬂexibility by selecting the states to which we apply updates so

as to improve the algorithm’s rate of progress. We can try to order the updates to let
value information propagate from state to state in an ecient way. Some states may not
need their values updated as often as others. We might even try to skip updating some
states entirely if they are not relevant to optimal behavior. Some ideas for doing this are
discussed in Chapter 8.
Asynchronous algorithms also make it easier to intermix computation with real-time
interaction. To solve a given MDP, we can run an iterative DP algorithm at the same

time that an agent is actually experiencing the MDP . The agent’s experience can be used

86 Chapter 4: Dynamic Programming
to determine the states to which the DP algorithm applies its updates. At the same time,
the latest value and policy information from the DP algorithm can guide the agent’s
decision making. For example, we can apply updates to states as the agent visits them.
This makes it possible to focus the DP algorithm’s updates onto parts of the state set that
are most relevant to the agent. This kind of focusing is a repeated theme in reinforcement
learning.
4.6 Generalized Policy Iteration
Policy iteration consists of two simultaneous, interacting processes, one making the value
function consistent with the current policy (policy evaluation), and the other making
the policy greedy with respect to the current value function (policy improvement). In
policy iteration, these two processes alternate, each completing before the other begins,
but this is not really necessary. In value iteration, for example, only a single iteration of

learning.
policy evaluation is performed in between each policy improvement. In asynchronous DP
methods, the evaluation and improvement processes are interleaved at an even ﬁner grain.
In some cases a single state is updated in one process before returning to the other. As
long as both processes continue to update all states, the ultimate result is typically the
same—convergence to the optimal value function and an optimal policy.evaluationimprovement⇡greedy(V)V⇡Vv⇡v⇤⇡⇤
We use the term generalized policy iteration (GPI) to re-

fer to the general idea of letting policy-evaluation and policy-
improvement processes interact, independent of the granularity
and other details of the two processes. Almost all reinforcement
learning methods are well described as GPI. That is, all have
identiﬁable policies and value functions, with the policy always
being improved with respect to the value function and the value
function always being driven toward the value function for the
policy, as suggested by the diagram to the right. If both the
evaluation process and the improvement process stabilize, that

is, no longer produce changes, then the value function and policy
must be optimal. The value function stabilizes only when it
is consistent with the current policy, and the policy stabilizes
only when it is greedy with respect to the current value function.
Thus, both processes stabilize only when a policy has been found that is greedy with
respect to its own evaluation function. This implies that the Bellman optimality equation
(4.1) holds, and thus that the policy and the value function are optimal.

The evaluation and improvement processes in GPI can be viewed as both competing
and cooperating. They compete in the sense that they pull in opposing directions. Making
the policy greedy with respect to the value function typically makes the value function
incorrect for the changed policy, and making the value function consistent with the policy
typically causes that policy no longer to be greedy. In the long run, however, these

two processes interact to ﬁnd a single joint solution: the optimal value function and an
optimal policy.

4.7. Eciency of Dynamic Programming 87v⇤,⇡⇤⇡= greedy(v)v,⇡v=v⇡One might also think of the interaction between
the evaluation and improvement processes in GPI
in terms of two constraints or goals—for example,
as two lines in two-dimensional space as suggested
by the diagram to the right. Although the real
geometry is much more complicated than this, the
diagram suggests what happens in the real case.
Each process drives the value function or policy
toward one of the lines representing a solution to
one of the two goals. The goals interact because the two lines are not orthogonal. Driving
directly toward one goal causes some movement away from the other goal. Inevitably,
however, the joint process is brought closer to the overall goal of optimality. The arrows
in this diagram correspond to the behavior of policy iteration in that each takes the
system all the way to achieving one of the two goals completely. In GPI one could also

take smaller, incomplete steps toward each goal. In either case, the two processes together
achieve the overall goal of optimality even though neither is attempting to achieve it
directly.
4.7 Eciency of Dynamic Programming
DP may not be practical for very large problems, but compared with other methods for
solving MDPs, DP methods are actually quite ecient. If we ignore a few technical details,
then the (worst case) time DP methods take to ﬁnd an optimal policy is polynomial in

directly.
the number of states and actions. If nandkdenote the number of states and actions, this
means that a DP method takes a number of computational operations that is less than
some polynomial function of nandk. A DP method is guaranteed to ﬁnd an optimal
policy in polynomial time even though the total number of (deterministic) policies is kn.
In this sense, DP is exponentially faster than any direct search in policy space could
be, because direct search would have to exhaustively examine each policy to provide the

same guarantee. Linear programming methods can also be used to solve MDPs, and in
some cases their worst-case convergence guarantees are better than those of DP methods.
But linear programming methods become impractical at a much smaller number of states
than do DP methods (by a factor of about 100). For the largest problems, only DP
methods are feasible.
DP is sometimes thought to be of limited applicability because of the curse of dimen-
sionality , the fact that the number of states often grows exponentially with the number

of state variables. Large state sets do create diculties, but these are inherent diculties
of the problem, not of DP as a solution method. In fact, DP is comparatively better
suited to handling large state spaces than competing methods such as direct search and
linear programming.
In practice, DP methods can be used with today’s computers to solve MDPs with
millions of states. Both policy iteration and value iteration are widely used, and it is not
clear which, if either, is better in general. In practice, these methods usually converge

much faster than their theoretical worst-case run times, particularly if they are started

88 Chapter 4: Dynamic Programming
with good initial value functions or policies.
On problems with large state spaces, asynchronous DP methods are often preferred. To
complete even one sweep of a synchronous method requires computation and memory for
every state. For some problems, even this much memory and computation is impractical,
yet the problem is still potentially solvable because relatively few states occur along
optimal solution trajectories. Asynchronous methods and other variations of GPI can be
applied in such cases and may ﬁnd good or optimal policies much faster than synchronous
methods can.
4.8 Summary
In this chapter we have become familiar with the basic ideas and algorithms of dynamic
programming as they relate to solving ﬁnite MDPs. Policy evaluation refers to the (typi-
cally) iterative computation of the value functions for a given policy. Policy improvement
refers to the computation of an improved policy given the value function for that policy.

Putting these two computations together, we obtain policy iteration andvalue iteration ,
the two most popular DP methods. Either of these can be used to reliably compute
optimal policies and value functions for ﬁnite MDPs given complete knowledge of the
MDP.
Classical DP methods operate in sweeps through the state set, performing an expected
update operation on each state. Each such operation updates the value of one state
based on the values of all possible successor states and their probabilities of occurring.

MDP.
Expected updates are closely related to Bellman equations: they are little more than
these equations turned into assignment statements. When the updates no longer result in
any changes in value, convergence has occurred to values that satisfy the corresponding
Bellman equation. Just as there are four primary value functions ( v⇡,v⇤,q⇡, and q⇤),
there are four corresponding Bellman equations and four corresponding expected updates.
An intuitive view of the operation of DP updates is given by their backup diagrams .

Insight into DP methods and, in fact, into almost all reinforcement learning methods,
can be gained by viewing them as generalized policy iteration (GPI). GPI is the general idea
of two interacting processes revolving around an approximate policy and an approximate
value function. One process takes the policy as given and performs some form of policy
evaluation, changing the value function to be more like the true value function for the
policy. The other process takes the value function as given and performs some form

of policy improvement, changing the policy to make it better, assuming that the value
function is its value function. Although each process changes the basis for the other,
overall they work together to ﬁnd a joint solution: a policy and value function that are
unchanged by either process and, consequently, are optimal. In some cases, GPI can be
proved to converge, most notably for the classical DP methods that we have presented in
this chapter. In other cases convergence has not been proved, but still the idea of GPI
improves our understanding of the methods.

It is not necessary to perform DP methods in complete sweeps through the state
set.Asynchronous DP methods are in-place iterative methods that update states in an
arbitrary order, perhaps stochastically determined and using out-of-date information.

Bibliographical and Historical Remarks 89
Many of these methods can be viewed as ﬁne-grained forms of GPI.
Finally, we note one last special property of DP methods. All of them update estimates
of the values of states based on estimates of the values of successor states. That is, they
update estimates on the basis of other estimates. We call this general idea bootstrapping .
Many reinforcement learning methods perform bootstrapping, even those that do not
require, as DP requires, a complete and accurate model of the environment. In the next
chapter we explore reinforcement learning methods that do not require a model and do
not bootstrap. In the chapter after that we explore methods that do not require a model
but do bootstrap. These key features and properties are separable, yet can be mixed in
interesting combinations.
The term “dynamic programming” is due to Bellman (1957a), who showed how these

methods could be applied to a wide range of problems. Extensive treatments of DP can
be found in many texts, including Bertsekas (2005, 2012), Bertsekas and Tsitsiklis (1996),
Dreyfus and Law (1977), Ross (1983), White (1969), and Whittle (1982, 1983). Our
interest in DP is restricted to its use in solving MDPs, but DP also applies to other types
of problems. Kumar and Kanal (1988) provide a more general look at DP.
To the best of our knowledge, the ﬁrst connection between DP and reinforcement

learning was made by Minsky (1961) in commenting on Samuel’s checkers player. In
a footnote, Minsky mentioned that it is possible to apply DP to problems in which
Samuel’s backing-up process can be handled in closed analytic form. This remark may
have misled artiﬁcial intelligence researchers into believing that DP was restricted to
analytically tractable problems and therefore largely irrelevant to artiﬁcial intelligence.
Andreae (1969b) mentioned DP in the context of reinforcement learning, speciﬁcally

policy iteration, although he did not make speciﬁc connections between DP and learning
algorithms. Werbos (1977) suggested an approach to approximating DP called “heuristic
dynamic programming” that emphasizes gradient-descent methods for continuous-state
problems (Werbos, 1982, 1987, 1988, 1989, 1992). These methods are closely related to
the reinforcement learning algorithms that we discuss in this book. Watkins (1989) was

explicit in connecting reinforcement learning to DP, characterizing a class of reinforcement
learning methods as “incremental dynamic programming.”
4.1–4 These sections describe well-established DP algorithms that are covered in any of
the general DP references cited above. The policy improvement theorem and the
policy iteration algorithm are due to Bellman (1957a) and Howard (1960). Our
presentation was inﬂuenced by the local view of policy improvement taken by

Watkins (1989). Our discussion of value iteration as a form of truncated policy
iteration is based on the approach of Puterman and Shin (1978), who presented a
class of algorithms called modiﬁed policy iteration , which includes policy iteration
and value iteration as special cases. An analysis showing how value iteration can
be made to ﬁnd an optimal policy in ﬁnite time is given by Bertsekas (1987).
Iterative policy evaluation is an example of a classical successive approximation

90 Chapter 4: Dynamic Programming
algorithm for solving a system of linear equations. The version of the algorithm
that uses two arrays, one holding the old values while the other is updated, is
often called a Jacobi-style algorithm, after Jacobi’s classical use of this method.
It is also sometimes called a synchronous algorithm because the e↵ect is as if all
the values are updated at the same time. The second array is needed to simulate
this parallel computation sequentially. The in-place version of the algorithm
is often called a Gauss–Seidel-style algorithm after the classical Gauss–Seidel
algorithm for solving systems of linear equations. In addition to iterative policy
evaluation, other DP algorithms can be implemented in these di↵erent versions.
Bertsekas and Tsitsiklis (1989) provide excellent coverage of these variations and
their performance di↵erences.
4.5 Asynchronous DP algorithms are due to Bertsekas (1982, 1983), who also called

them distributed DP algorithms. The original motivation for asynchronous
DP was its implementation on a multiprocessor system with communication
delays between processors and no global synchronizing clock. These algorithms
are extensively discussed by Bertsekas and Tsitsiklis (1989). Jacobi-style and
Gauss–Seidel-style DP algorithms are special cases of the asynchronous version.
Williams and Baird (1990) presented DP algorithms that are asynchronous at a
ﬁner grain than the ones we have discussed: the update operations themselves

are broken into steps that can be performed asynchronously.
4.7 This section, written with the help of Michael Littman, is based on Littman,
Dean, and Kaelbling (1995). The phrase “curse of dimensionality” is due to
Bellman (1957a).
Foundational work on the linear programming approach to reinforcement learning
was done by Daniela de Farias (de Farias, 2002; de Farias and Van Roy, 2003).

Chapter 5
Monte Carlo Methods
In this chapter we consider our ﬁrst learning methods for estimating value functions and
discovering optimal policies. Unlike the previous chapter, here we do not assume complete
knowledge of the environment. Monte Carlo methods require only experience —sample
sequences of states, actions, and rewards from actual or simulated interaction with an
environment. Learning from actual experience is striking because it requires no prior
knowledge of the environment’s dynamics, yet can still attain optimal behavior. Learning
from simulated experience is also powerful. Although a model is required, the model need
only generate sample transitions, not the complete probability distributions of all possible
transitions that is required for dynamic programming (DP). In surprisingly many cases it
is easy to generate experience sampled according to the desired probability distributions,
but infeasible to obtain the distributions in explicit form.

Monte Carlo methods are ways of solving the reinforcement learning problem based on
averaging sample returns. To ensure that well-deﬁned returns are available, here we deﬁne
Monte Carlo methods only for episodic tasks. That is, we assume experience is divided
into episodes, and that all episodes eventually terminate no matter what actions are
selected. Only on the completion of an episode are value estimates and policies changed.

Monte Carlo methods can thus be incremental in an episode-by-episode sense, but not in
a step-by-step (online) sense. The term “Monte Carlo” is often used more broadly for
any estimation method whose operation involves a signiﬁcant random component. Here
we use it speciﬁcally for methods based on averaging complete returns (as opposed to
methods that learn from partial returns, considered in the next chapter).
Monte Carlo methods sample and average returns for each state–action pair much like

the bandit methods we explored in Chapter 2 sample and average rewards for each action.
The main di↵erence is that now there are multiple states, each acting like a di↵erent
bandit problem (like an associative-search or contextual bandit) and the di↵erent bandit
problems are interrelated. That is, the return after taking an action in one state depends
on the actions taken in later states in the same episode. Because all the action selections

are undergoing learning, the problem becomes nonstationary from the point of view of
the earlier state.
91

92 Chapter 5: Monte Carlo Methods
To handle the nonstationarity, we adapt the idea of general policy iteration (GPI)
developed in Chapter 4 for DP. Whereas there we computed value functions from knowledge
of the MDP, here we learn value functions from sample returns with the MDP. The value
functions and corresponding policies still interact to attain optimality in essentially the
same way (GPI). As in the DP chapter, ﬁrst we consider the prediction problem (the
computation of v⇡andq⇡for a ﬁxed arbitrary policy ⇡) then policy improvement, and,
ﬁnally, the control problem and its solution by GPI. Each of these ideas taken from DP
is extended to the Monte Carlo case in which only sample experience is available.
5.1 Monte Carlo Prediction
We begin by considering Monte Carlo methods for learning the state-value function for a
given policy. Recall that the value of a state is the expected return—expected cumulative
future discounted reward—starting from that state. An obvious way to estimate it from

experience, then, is simply to average the returns observed after visits to that state. As
more returns are observed, the average should converge to the expected value. This idea
underlies all Monte Carlo methods.
In particular, suppose we wish to estimate v⇡(s), the value of a state sunder policy ⇡,
given a set of episodes obtained by following ⇡and passing through s. Each occurrence
of state sin an episode is called a visit tos. Of course, smay be visited multiple times

in the same episode; let us call the ﬁrst time it is visited in an episode the ﬁrst visit
tos.T h e ﬁrst-visit MC method estimates v⇡(s) as the average of the returns following
ﬁrst visits to s, whereas the every-visit MC method averages the returns following all
visits to s. These two Monte Carlo (MC) methods are very similar but have slightly
di↵erent theoretical properties. First-visit MC has been most widely studied, dating back
to the 1940s, and is the one we focus on in this chapter. Every-visit MC extends more

naturally to function approximation and eligibility traces, as discussed in Chapters 9 and
12. First-visit MC is shown in procedural form in the box. Every-visit MC would be the
same except without the check for Sthaving occurred earlier in the episode.
First-visit MC prediction, for estimating V⇡v⇡
Input: a policy ⇡to be evaluated
Initialize:
V(s)2R, arbitrarily, for all s2S
Returns (s) an empty list, for all s2S
Loop forever (for each episode):
Generate an episode following ⇡:S0,A0,R1,S1,A1,R2,...,S T1,AT1,RT
G 0
Loop for each step of episode, t=T1,T2,..., 0:
G G+Rt+1

5.1. Monte Carlo Prediction 93
Both ﬁrst-visit MC and every-visit MC converge to v⇡(s) as the number of visits (or
ﬁrst visits) to sgoes to inﬁnity. This is easy to see for the case of ﬁrst-visit MC. In
this case each return is an independent, identically distributed estimate of v⇡(s)w i t h
ﬁnite variance. By the law of large numbers the sequence of averages of these estimates
converges to their expected value. Each average is itself an unbiased estimate, and the
standard deviation of its error falls as 1 /pn,w h e r e nis the number of returns averaged.
Every-visit MC is less straightforward, but its estimates also converge quadratically to
v⇡(s) (Singh and Sutton, 1996).
The use of Monte Carlo methods is best illustrated through an example.
Example 5.1: Blackjack The object of the popular casino card game of blackjack is to
obtain cards the sum of whose numerical values is as great as possible without exceeding
21. All face cards count as 10, and an ace can count either as 1 or as 11. We consider

the version in which each player competes independently against the dealer. The game
begins with two cards dealt to both dealer and player. One of the dealer’s cards is face
up and the other is face down. If the player has 21 immediately (an ace and a 10-card),
it is called a natural . He then wins unless the dealer also has a natural, in which case the
game is a draw. If the player does not have a natural, then he can request additional
cards, one by one ( hits), until he either stops ( sticks ) or exceeds 21 ( goes bust ). If he goes

bust, he loses; if he sticks, then it becomes the dealer’s turn. The dealer hits or sticks
according to a ﬁxed strategy without choice: he sticks on any sum of 17 or greater, and
hits otherwise. If the dealer goes bust, then the player wins; otherwise, the outcome—win,
lose, or draw—is determined by whose ﬁnal sum is closer to 21.
Playing blackjack is naturally formulated as an episodic ﬁnite MDP. Each game of
blackjack is an episode. Rewards of +1, 1, and 0 are given for winning, losing, and

drawing, respectively. All rewards within a game are zero, and we do not discount ( = 1);
therefore these terminal rewards are also the returns. The player’s actions are to hit or
to stick. The states depend on the player’s cards and the dealer’s showing card. We
assume that cards are dealt from an inﬁnite deck (i.e., with replacement) so that there is
no advantage to keeping track of the cards already dealt. If the player holds an ace that

he could count as 11 without going bust, then the ace is said to be usable . In this case
it is always counted as 11 because counting it as 1 would make the sum 11 or less, in
which case there is no decision to be made because, obviously, the player should always
hit. Thus, the player makes decisions on the basis of three variables: his current sum
(12–21), the dealer’s one showing card (ace–10), and whether or not he holds a usable
ace. This makes for a total of 200 states.
Consider the policy that sticks if the player’s sum is 20 or 21, and otherwise hits. To

ﬁnd the state-value function for this policy by a Monte Carlo approach, one simulates
many blackjack games using the policy and averages the returns following each state.
In this way, we obtained the estimates of the state-value function shown in Figure 5.1.
The estimates for states with a usable ace are less certain and less regular because these
states are less common. In any event, after 500,000 games the value function is very well
approximated.

94 Chapter 5: Monte Carlo Methods
+1!1
ADealer showing1012Player sum21After 500,000 episodesAfter 10,000 episodesUsableaceNousableaceFigure 5.1: Approximate state-value functions for the blackjack policy that sticks only on 20
or 21, computed by Monte Carlo policy evaluation.
Exercise 5.1 Consider the diagrams on the right in Figure 5.1. Why does the estimated
value function jump up for the last two rows in the rear? Why does it drop o↵ for the
whole last row on the left? Why are the frontmost values higher in the upper diagrams
than in the lower? ⇤
Exercise 5.2 Suppose every-visit MC was used instead of ﬁrst-visit MC on the blackjack
task. Would you expect the results to be very di↵erent? Why or why not? ⇤
Although we have complete knowledge of the environment in the blackjack task, it
would not be easy to apply DP methods to compute the value function. DP methods
require the distribution of next events—in particular, they require the environments

dynamics as given by the four-argument function p—and it is not easy to determine
this for blackjack. For example, suppose the player’s sum is 14 and he chooses to stick.
What is his probability of terminating with a reward of +1 as a function of the dealer’s
showing card? All of the probabilities must be computed before DP can be applied, and
such computations are often complex and error-prone. In contrast, generating the sample
games required by Monte Carlo methods is easy. This is the case surprisingly often; the

ability of Monte Carlo methods to work with sample episodes alone can be a signiﬁcant
advantage even when one has complete knowledge of the environment’s dynamics.
Can we generalize the idea of backup diagrams to Monte Carlo algorithms? The
general idea of a backup diagram is to show at the top the root node to be updated and
to show below all the transitions and leaf nodes whose rewards and estimated values
contribute to the update. For Monte Carlo estimation of v⇡, the root is a state node, and

below it is the entire trajectory of transitions along a particular single episode, ending

5.1. Monte Carlo Prediction 95at the terminal state, as shown to the right. Whereas the DP diagram (page 59)
shows all possible transitions, the Monte Carlo diagram shows only those sampled
on the one episode. Whereas the DP diagram includes only one-step transitions,
the Monte Carlo diagram goes all the way to the end of the episode. These
di↵erences in the diagrams accurately reﬂect the fundamental di↵erences between
the algorithms.
An important fact about Monte Carlo methods is that the estimates for each
state are independent. The estimate for one state does not build upon the estimate
of any other state, as is the case in DP. In other words, Monte Carlo methods do
notbootstrap as we deﬁned it in the previous chapter.
In particular, note that the computational expense of estimating the value of
a single state is independent of the number of states. This can make Monte Carlo
methods particularly attractive when one requires the value of only one or a subset

of states. One can generate many sample episodes starting from the states of interest,
averaging returns from only these states, ignoring all others. This is a third advantage
Monte Carlo methods can have over DP methods (after the ability to learn from actual
experience and from simulated experience).
A bubble on a wire loop.
From Hersh and Griego (1969). Reproduced with
permission. c1969 Scientiﬁc American, a divi-
sion of Nature America, Inc. All rights reserved.Example 5.2: Soap Bubble Suppose a wire
frame forming a closed loop is dunked in soapy

water to form a soap surface or bubble conform-
ing at its edges to the wire frame. If the geom-
etry of the wire frame is irregular but known,
how can you compute the shape of the surface?
The shape has the property that the total force
on each point exerted by neighboring points is
zero (or else the shape would change). This
means that the surface’s height at any point is
the average of its heights at points in a small
circle around that point. In addition, the sur-
face must meet at its boundaries with the wire

frame. The usual approach to problems of this
kind is to put a grid over the area covered by
the surface and solve for its height at the grid points by an iterative computation. Grid
points at the boundary are forced to the wire frame, and all others are adjusted toward
the average of the heights of their four nearest neighbors. This process then iterates, much
like DP’s iterative policy evaluation, and ultimately converges to a close approximation
to the desired surface.

This is similar to the kind of problem for which Monte Carlo methods were originally
designed. Instead of the iterative computation described above, imagine standing on the
surface and taking a random walk, stepping randomly from grid point to neighboring
grid point, with equal probability, until you reach the boundary. It turns out that the
expected value of the height at the boundary is a close approximation to the height of
the desired surface at the starting point (in fact, it is exactly the value computed by the

iterative method described above). Thus, one can closely approximate the height of the

96 Chapter 5: Monte Carlo Methods
surface at a point by simply averaging the boundary heights of many walks started at
the point. If one is interested in only the value at one point, or any ﬁxed small set of
points, then this Monte Carlo method can be far more ecient than the iterative method
based on local consistency.
5.2 Monte Carlo Estimation of Action Values
If a model is not available, then it is particularly useful to estimate action values (the
values of state–action pairs) rather than state values. With a model, state values alone are
sucient to determine a policy; one simply looks ahead one step and chooses whichever
action leads to the best combination of reward and next state, as we did in the chapter on
DP. Without a model, however, state values alone are not sucient. One must explicitly
estimate the value of each action in order for the values to be useful in suggesting a policy.
Thus, one of our primary goals for Monte Carlo methods is to estimate q⇤. To achieve

this, we ﬁrst consider the policy evaluation problem for action values.
The policy evaluation problem for action values is to estimate q⇡(s, a), the expected
return when starting in state s, taking action a, and thereafter following policy ⇡.T h e
Monte Carlo methods for this are essentially the same as just presented for state values,
except now we talk about visits to a state–action pair rather than to a state. A state–
action pair s, ais said to be visited in an episode if ever the state sis visited and action

ais taken in it. The every-visit MC method estimates the value of a state–action pair
as the average of the returns that have followed all the visits to it. The ﬁrst-visit MC
method averages the returns following the ﬁrst time in each episode that the state was
visited and the action was selected. These methods converge quadratically, as before, to
the true expected values as the number of visits to each state–action pair approaches
inﬁnity.
The only complication is that many state–action pairs may never be visited. If ⇡is

inﬁnity.
a deterministic policy, then in following ⇡one will observe returns only for one of the
actions from each state. With no returns to average, the Monte Carlo estimates of the
other actions will not improve with experience. This is a serious problem because the
purpose of learning action values is to help in choosing among the actions available in
each state. To compare alternatives we need to estimate the value of allthe actions from
each state, not just the one we currently favor.
This is the general problem of maintaining exploration , as discussed in the context

of the k-armed bandit problem in Chapter 2. For policy evaluation to work for action
values, we must assure continual exploration. One way to do this is by specifying that
the episodes start in a state–action pair , and that every pair has a nonzero probability of
being selected as the start. This guarantees that all state–action pairs will be visited an
inﬁnite number of times in the limit of an inﬁnite number of episodes. We call this the
assumption of exploring starts .

The assumption of exploring starts is sometimes useful, but of course it cannot be
relied upon in general, particularly when learning directly from actual interaction with an
environment. In that case the starting conditions are unlikely to be so helpful. The most
common alternative approach to assuring that all state–action pairs are encountered is

5.3. Monte Carlo Control 97
to consider only policies that are stochastic with a nonzero probability of selecting all
actions in each state. We discuss two important variants of this approach in later sections.
For now, we retain the assumption of exploring starts and complete the presentation of a
full Monte Carlo control method.
Exercise 5.3 What is the backup diagram for Monte Carlo estimation of q⇡? ⇤
5.3 Monte Carlo Control
We are now ready to consider how Monte Carlo estimation can be used in control, that
is, to approximate optimal policies. The overall idea is to proceed according to the same
pattern as in the DP chapter, that is, according to the idea of generalized policy iterationevaluationimprovement⇡Q⇡greedy(Q)Qq⇡
(GPI). In GPI one maintains both an approximate policy and
an approximate value function. The value function is repeatedly
altered to more closely approximate the value function for the
current policy, and the policy is repeatedly improved with respect

to the current value function, as suggested by the diagram to
the right. These two kinds of changes work against each other to
some extent, as each creates a moving target for the other, but
together they cause both policy and value function to approach
optimality.
To begin, let us consider a Monte Carlo version of classical policy iteration. In
this method, we perform alternating complete steps of policy evaluation and policy
improvement, beginning with an arbitrary policy ⇡0and ending with the optimal policy

each q⇡kexactly, for arbitrary ⇡k.
Policy improvement is done by making the policy greedy with respect to the current
value function. In this case we have an action -value function, and therefore no model is
needed to construct the greedy policy. For any action-value function q, the corresponding
greedy policy is the one that, for each s2S, deterministically chooses an action with
maximal action-value:
⇡(s).= arg max
aq(s, a). (5.1)
Policy improvement then can be done by constructing each ⇡k+1as the greedy policy
with respect to q⇡k. The policy improvement theorem (Section 4.2) then applies to ⇡k

98 Chapter 5: Monte Carlo Methods
and⇡k+1because, for all s2S,
q⇡k(s, ⇡ k+1(s)) = q⇡k(s,argmax
aq⇡k(s, a))
= max
aq⇡k(s, a)
q⇡k(s, ⇡ k(s))
v⇡k(s).
As we discussed in the previous chapter, the theorem assures us that each ⇡k+1is uniformly
better than ⇡k, or just as good as ⇡k, in which case they are both optimal policies. This
in turn assures us that the overall process converges to the optimal policy and optimal
value function. In this way Monte Carlo methods can be used to ﬁnd optimal policies
given only sample episodes and no other knowledge of the environment’s dynamics.
We made two unlikely assumptions above in order to easily obtain this guarantee of
convergence for the Monte Carlo method. One was that the episodes have exploring
starts, and the other was that policy evaluation could be done with an inﬁnite number of
episodes. To obtain a practical algorithm we will have to remove both assumptions. We
postpone consideration of the ﬁrst assumption until later in this chapter.

For now we focus on the assumption that policy evaluation operates on an inﬁnite
number of episodes. This assumption is relatively easy to remove. In fact, the same issue
arises even in classical DP methods such as iterative policy evaluation, which also converge
only asymptotically to the true value function. In both DP and Monte Carlo cases there
are two ways to solve the problem. One is to hold ﬁrm to the idea of approximating q⇡k
in each policy evaluation. Measurements and assumptions are made to obtain bounds

on the magnitude and probability of error in the estimates, and then sucient steps are
taken during each policy evaluation to assure that these bounds are suciently small.
This approach can probably be made completely satisfactory in the sense of guaranteeing
correct convergence up to some level of approximation. However, it is also likely to require
far too many episodes to be useful in practice on any but the smallest problems.
There is a second approach to avoiding the inﬁnite number of episodes nominally

required for policy evaluation, in which we give up trying to complete policy evaluation
before returning to policy improvement. On each evaluation step we move the value
function toward q⇡k, but we do not expect to actually get close except over many steps.
We used this idea when we ﬁrst introduced the idea of GPI in Section 4.6. One extreme
form of the idea is value iteration, in which only one iteration of iterative policy evaluation
is performed between each step of policy improvement. The in-place version of value

iteration is even more extreme; there we alternate between improvement and evaluation
steps for single states.
For Monte Carlo policy evaluation it is natural to alternate between evaluation and
improvement on an episode-by-episode basis. After each episode, the observed returns
are used for policy evaluation, and then the policy is improved at all the states visited in
the episode. A complete simple algorithm along these lines, which we call Monte Carlo
ES, for Monte Carlo with Exploring Starts, is given in pseudocode in the box on the next
page.

Append GtoReturns (St,At)
Q(St,At) average( Returns (St,At))
⇡(St) argmaxaQ(St,a)
Exercise 5.4 The pseudocode for Monte Carlo ES is inecient because, for each state–
action pair, it maintains a list of all returns and repeatedly calculates their mean. It would
be more ecient to use techniques similar to those explained in Section 2.4 to maintain
just the mean and a count (for each state–action pair) and update them incrementally.
Describe how the pseudocode would be altered to achieve this. ⇤
In Monte Carlo ES, all the returns for each state–action pair are accumulated and
averaged, irrespective of what policy was in force when they were observed. It is easy
to see that Monte Carlo ES cannot converge to any suboptimal policy. If it did, then
the value function would eventually converge to the value function for that policy, and
that in turn would cause the policy to change. Stability is achieved only when both
the policy and the value function are optimal. Convergence to this optimal ﬁxed point

seems inevitable as the changes to the action-value function decrease over time, but has
not yet been formally proved. In our opinion, this is one of the most fundamental open
theoretical questions in reinforcement learning (for a partial solution, see Tsitsiklis, 2002).
Example 5.3: Solving Blackjack It is straightforward to apply Monte Carlo ES to
blackjack. Because the episodes are all simulated games, it is easy to arrange for exploring

starts that include all possibilities. In this case one simply picks the dealer’s cards, the
player’s sum, and whether or not the player has a usable ace, all at random with equal
probability. As the initial policy we use the policy evaluated in the previous blackjack
example, that which sticks only on 20 or 21. The initial action-value function can be zero
for all state–action pairs. Figure 5.2 shows the optimal policy for blackjack found by
Monte Carlo ES. This policy is the same as the “basic” strategy of Thorp (1966) with the

sole exception of the leftmost notch in the policy for a usable ace, which is not present
in Thorp’s strategy. We are uncertain of the reason for this discrepancy, but conﬁdent
that what is shown here is indeed the optimal policy for the version of blackjack we have
described.

100 Chapter 5: Monte Carlo Methods
UsableaceNousableace20
10A23456789Dealer showingPlayer sumHITSTICK1921
1112131415161718!*
10A23456789HITSTICK201921
1112131415161718V*211012A
Dealer showingPlayer sum10A1221+1"1v*UsableaceNousableace20
Dealer showingPlayer sum10A1221+1"1v*Dealer showingPlayer sum**Figure 5.2: The optimal policy and state-value function for blackjack, found by Monte Carlo
ES. The state-value function shown was computed from the action-value function found by
Monte Carlo ES.
5.4 Monte Carlo Control without Exploring Starts
How can we avoid the unlikely assumption of exploring starts? The only general way to
ensure that all actions are selected inﬁnitely often is for the agent to continue to select

them. There are two approaches to ensuring this, resulting in what we call on-policy
methods and o↵-policy methods. On-policy methods attempt to evaluate or improve the
policy that is used to make decisions, whereas o↵-policy methods evaluate or improve
a policy di↵erent from that used to generate the data. The Monte Carlo ES method
developed above is an example of an on-policy method. In this section we show how an
on-policy Monte Carlo control method can be designed that does not use the unrealistic

assumption of exploring starts. O↵-policy methods are considered in the next section.
In on-policy control methods the policy is generally soft, meaning that ⇡(a|s)>0
for all s2Sand all a2A(s), but gradually shifted closer and closer to a deterministic
optimal policy. Many of the methods discussed in Chapter 2 provide mechanisms for
this. The on-policy method we present in this section uses "-greedy policies, meaning
that most of the time they choose an action that has maximal estimated action value,

but with probability "they instead select an action at random. That is, all nongreedy
actions are given the minimal probability of selection,"
|A(s)|, and the remaining bulk of
the probability, 1 "+"
|A(s)|, is given to the greedy action. The "-greedy policies are
examples of "-soft policies, deﬁned as policies for which ⇡(a|s)"
|A(s)|for all states and
actions, for some ">0. Among "-soft policies, "-greedy policies are in some sense those

5.4. Monte Carlo Control without Exploring Starts 101
that are closest to greedy.
The overall idea of on-policy Monte Carlo control is still that of GPI. As in Monte
Carlo ES, we use ﬁrst-visit MC methods to estimate the action-value function for the
current policy. Without the assumption of exploring starts, however, we cannot simply
improve the policy by making it greedy with respect to the current value function, because
that would prevent further exploration of nongreedy actions. Fortunately, GPI does not
require that the policy be taken all the way to a greedy policy, only that it be moved
toward a greedy policy. In our on-policy method we will move it only to an "-greedy
policy. For any "-soft policy, ⇡, any "-greedy policy with respect to q⇡is guaranteed to
be better than or equal to ⇡. The complete algorithm is given in the box below.
On-policy ﬁrst-visit MC control (for "-soft policies), estimates ⇡⇡⇡⇤
Algorithm parameter: small ">0
⇡ an arbitrary "-soft policy

102 Chapter 5: Monte Carlo Methods
must be less than or equal to the largest number averaged)
="
|A(s)|X
aq⇡(s, a)"
|A(s)|X
aq⇡(s, a)+X
a⇡(a|s)q⇡(s, a)
=v⇡(s).
Thus, by the policy improvement theorem, ⇡0⇡(i.e., v⇡0(s)v⇡(s), for all s2S). We
now prove that equality can hold only when both ⇡0and⇡are optimal among the "-soft
policies, that is, when they are better than or equal to all other "-soft policies.
Consider a new environment that is just like the original environment, except with the
requirement that policies be "-soft “moved inside” the environment. The new environment
has the same action and state set as the original and behaves as follows. If in state s
and taking action a, then with probability 1 "the new environment behaves exactly
like the old environment. With probability "it repicks the action at random, with equal
probabilities, and then behaves like the old environment with the new, random action.
The best one can do in this new environment with general policies is the same as the

=( 1 ") max
aX
r+v⇡(s0)i
+"
|A(s)|X
aX
r+v⇡(s0)i
.
However, this equation is the same as the previous one, except for the substitution of v⇡
forev⇤. Because ev⇤is the unique solution, it must be that v⇡=ev⇤.
In essence, we have shown in the last few pages that policy iteration works for "-soft
policies. Using the natural notion of greedy policy for "-soft policies, one is assured of
improvement on every step, except when the best policy has been found among the "-soft
policies. This analysis is independent of how the action-value functions are determined
at each stage, but it does assume that they are computed exactly. This brings us to

5.5. O↵-policy Prediction via Importance Sampling 103
roughly the same point as in the previous section. Now we only achieve the best policy
among the "-soft policies, but on the other hand, we have eliminated the assumption of
exploring starts.
5.5 O↵-policy Prediction via Importance Sampling
All learning control methods face a dilemma: They seek to learn action values conditional
on subsequent optimal behavior, but they need to behave non-optimally in order to
explore all actions (to ﬁnd the optimal actions). How can they learn about the optimal
policy while behaving according to an exploratory policy? The on-policy approach in the
preceding section is actually a compromise—it learns action values not for the optimal
policy, but for a near-optimal policy that still explores. A more straightforward approach
is to use two policies, one that is learned about and that becomes the optimal policy, and
one that is more exploratory and is used to generate behavior. The policy being learned

about is called the target policy , and the policy used to generate behavior is called the
behavior policy . In this case we say that learning is from data “o↵” the target policy, and
the overall process is termed o↵-policy learning .
Throughout the rest of this book we consider both on-policy and o↵-policy methods.
On-policy methods are generally simpler and are considered ﬁrst. O↵-policy methods
require additional concepts and notation, and because the data is due to a di↵erent policy,

o↵-policy methods are often of greater variance and are slower to converge. On the other
hand, o↵-policy methods are more powerful and general. They include on-policy methods
as the special case in which the target and behavior policies are the same. O↵-policy
methods also have a variety of additional uses in applications. For example, they can
often be applied to learn from data generated by a conventional non-learning controller,
or from a human expert. O↵-policy learning is also seen by some as key to learning

multi-step predictive models of the world’s dynamics (see Section 17.2; Sutton, 2009;
Sutton et al., 2011).
In this section we begin the study of o↵-policy methods by considering the prediction
problem, in which both target and behavior policies are ﬁxed. That is, suppose we wish
to estimate v⇡orq⇡, but all we have are episodes following another policy b,w h e r e
b6=⇡. In this case, ⇡is the target policy, bis the behavior policy, and both policies are
considered ﬁxed and given.
In order to use episodes from bto estimate values for ⇡, we require that every action

taken under ⇡is also taken, at least occasionally, under b. That is, we require that
⇡(a|s)>0i m p l i e s b(a|s)>0. This is called the assumption of coverage . It follows
from coverage that bmust be stochastic in states where it is not identical to ⇡.T h e
target policy ⇡, on the other hand, may be deterministic, and, in fact, this is a case
of particular interest in control applications. In control, the target policy is typically
the deterministic greedy policy with respect to the current estimate of the action-value

function. This policy becomes a deterministic optimal policy while the behavior policy
remains stochastic and more exploratory, for example, an "-greedy policy. In this section,
however, we consider the prediction problem, in which ⇡is unchanging and given.
Almost all o↵-policy methods utilize importance sampling , a general technique for

are generally unknown, they appear identically in both the numerator and denominator,
and thus cancel. The importance sampling ratio ends up depending only on the two
policies and the sequence, not on the MDP.
Recall that we wish to estimate the expected returns (values) under the target policy,
but all we have are returns Gtdue to the behavior policy. These returns have the wrong
expectation E[Gt|St=s]=vb(s) and so cannot be averaged to obtain v⇡.T h i si sw h e r e
importance sampling comes in. The ratio ⇢t:T1transforms the returns to have the right
expected value:
E[⇢t:T1Gt|St=s]=v⇡(s). (5.4)
Now we are ready to give a Monte Carlo algorithm that averages returns from a batch
of observed episodes following policy bto estimate v⇡(s). It is convenient here to number
time steps in a way that increases across episode boundaries. That is, if the ﬁrst episode
of the batch ends in a terminal state at time 100, then the next episode begins at time

t= 101. This enables us to use time-step numbers to refer to particular steps in particular
episodes. In particular, we can deﬁne the set of all time steps in which state sis visited,
denoted T(s). This is for an every-visit method; for a ﬁrst-visit method, T(s) would only
include time steps that were ﬁrst visits to swithin their episodes. Also, let T(t) denote
the ﬁrst time of termination following time t, and Gtdenote the return after tup through
T(t). Then {Gt}t2T(s)are the returns that pertain to state s, and
⇢t:T(t)1

⇢t:T(t)1 
t2T(s)are
the corresponding importance-sampling ratios. To estimate v⇡(s), we simply scale the
returns by the ratios and average the results:
V(s).=P
t2T(s)⇢t:T(t)1Gt
|T(s)|. (5.5)

5.5. O↵-policy Prediction via Importance Sampling 105
When importance sampling is done as a simple average in this way it is called ordinary
importance sampling .
An important alternative is weighted importance sampling ,w h i c hu s e sa weighted
average, deﬁned as
V(s).=P
t2T(s)⇢t:T(t)1GtP
t2T(s)⇢t:T(t)1, (5.6)
or zero if the denominator is zero. To understand these two varieties of importance
sampling, consider the estimates of their ﬁrst-visit methods after observing a single return
from state s. In the weighted-average estimate, the ratio ⇢t:T(t)1for the single return
cancels in the numerator and denominator, so that the estimate is equal to the observed
return independent of the ratio (assuming the ratio is nonzero). Given that this return
was the only one observed, this is a reasonable estimate, but its expectation is vb(s) rather
than v⇡(s), and in this statistical sense it is biased. In contrast, the ﬁrst-visit version

of the ordinary importance-sampling estimator (5.5) is always v⇡(s) in expectation (it
is unbiased), but it can be extreme. Suppose the ratio were ten, indicating that the
trajectory observed is ten times as likely under the target policy as under the behavior
policy. In this case the ordinary importance-sampling estimate would be ten times the
observed return. That is, it would be quite far from the observed return even though the
episode’s trajectory is considered very representative of the target policy.

Formally, the di↵erence between the ﬁrst-visit methods of the two kinds of importance
sampling is expressed in their biases and variances. Ordinary importance sampling is
unbiased whereas weighted importance sampling is biased (though the bias converges
asymptotically to zero). On the other hand, the variance of ordinary importance sampling
is in general unbounded because the variance of the ratios can be unbounded, whereas in
the weighted estimator the largest weight on any single return is one. In fact, assuming

bounded returns, the variance of the weighted importance-sampling estimator converges
to zero even if the variance of the ratios themselves is inﬁnite (Precup, Sutton, and
Dasgupta 2001). In practice, the weighted estimator usually has dramatically lower
variance and is strongly preferred. Nevertheless, we will not totally abandon ordinary
importance sampling as it is easier to extend to the approximate methods using function
approximation that we explore in the second part of this book.
The every-visit methods for ordinary and weighed importance sampling are both biased,

though, again, the bias falls asymptotically to zero as the number of samples increases.
In practice, every-visit methods are often preferred because they remove the need to keep
track of which states have been visited and because they are much easier to extend to
approximations. A complete every-visit MC algorithm for o↵-policy policy evaluation
using weighted importance sampling is given in the next section on page 110.
Exercise 5.5 Consider an MDP with a single nonterminal state and a single action

that transitions back to the nonterminal state with probability pand transitions to the
terminal state with probability 1 p. Let the reward be +1 on all transitions, and let
=1. Suppose you observe one episode that lasts 10 steps, with a return of 10. What
are the ﬁrst-visit and every-visit estimators of the value of the nonterminal state? ⇤

106 Chapter 5: Monte Carlo Methods
Example 5.4: O↵-policy Estimation of a Blackjack State Value We applied
both ordinary and weighted importance-sampling methods to estimate the value of a single
blackjack state (Example 5.1) from o↵-policy data. Recall that one of the advantages
of Monte Carlo methods is that they can be used to evaluate a single state without
forming estimates for any other states. In this example, we evaluated the state in which
the dealer is showing a deuce, the sum of the player’s cards is 13, and the player has
a usable ace (that is, the player holds an ace and a deuce, or equivalently three aces).
The data was generated by starting in this state then choosing to hit or stick at random
with equal probability (the behavior policy). The target policy was to stick only on
a sum of 20 or 21, as in Example 5.1. The value of this state under the target policy
is approximately 0.27726 (this was determined by separately generating one-hundred

million episodes using the target policy and averaging their returns). Both o↵-policy
methods closely approximated this value after 1000 o↵-policy episodes using the random
policy. To make sure they did this reliably, we performed 100 independent runs, each
starting from estimates of zero and learning for 10,000 episodes. Figure 5.3 shows the
resultant learning curves—the squared error of the estimates of each method as a function
of number of episodes, averaged over the 100 runs. The error approaches zero for both

algorithms, but the weighted importance-sampling method has much lower error at the
beginning, as is typical in practice.
Ordinary importance samplingWeighted importance samplingEpisodes (log scale)010100100010,000Meansquareerror(average over100 runs)05Figure 5.3: Weighted importance sampling produces lower error estimates of the value of a
single blackjack state from o↵-policy episodes.
Example 5.5: Inﬁnite Variance The estimates of ordinary importance sampling will
typically have inﬁnite variance, and thus unsatisfactory convergence properties, whenever

the scaled returns have inﬁnite variance—and this can easily happen in o↵-policy learning
when trajectories contain loops. A simple example is shown inset in Figure 5.4. There is
only one nonterminal state sand two actions, right andleft.T h e right action causes a
deterministic transition to termination, whereas the leftaction transitions, with probability
0.9, back to sor, with probability 0.1, on to termination. The rewards are +1 on the
latter transition and otherwise zero. Consider the target policy that always selects left.

All episodes under this policy consist of some number (possibly zero) of transitions back

5.5. O↵-policy Prediction via Importance Sampling 107
tosfollowed by termination with a reward and return of +1. Thus the value of sunder
the target policy is 1 ( = 1). Suppose we are estimating this value from o↵-policy data
using the behavior policy that selects right andleftwith equal probability.
1100,0001,000,00010,000,000100,000,00020.10.9R=+ 1s⇡(left|s)=1leftrightR=0R=0b(left|s)=12v⇡(s)Monte-Carlo estimate of           with ordinaryimportance sampling(ten runs)Episodes (log scale)110100100010,0000Figure 5.4: Ordinary importance sampling produces surprisingly unstable estimates on the
one-state MDP shown inset (Example 5.5). The correct estimate here is 1 ( =1 ) ,a n d ,e v e n
though this is the expected value of a sample return (after importance sampling), the variance
of the samples is inﬁnite, and the estimates do not converge to this value. These results are for
o↵-policy ﬁrst-visit MC.
The lower part of Figure 5.4 shows ten independent runs of the ﬁrst-visit MC algorithm

using ordinary importance sampling. Even after millions of episodes, the estimates fail
to converge to the correct value of 1. In contrast, the weighted importance-sampling
algorithm would give an estimate of exactly 1 forever after the ﬁrst episode that ended
with the leftaction. All returns not equal to 1 (that is, ending with the right action)
would be inconsistent with the target policy and thus would have a ⇢t:T(t)1of zero and
contribute neither to the numerator nor denominator of (5.6) . The weighted importance-
sampling algorithm produces a weighted average of only the returns consistent with the

108 Chapter 5: Monte Carlo Methods
expectation of the square of the random variable is inﬁnite. Thus, we need only show
that the expected square of the importance-sampling-scaled return is inﬁnite:
Eb2
4 T1Y
t=0⇡(At|St)
b(At|St)G0!23
5.
To compute this expectation, we break it down into cases based on episode length and
termination. First note that, for any episode ending with the right action, the importance
sampling ratio is zero, because the target policy would never take this action; these
episodes thus contribute nothing to the expectation (the quantity in parenthesis will be
zero) and can be ignored. We need only consider episodes that involve some number
(possibly zero) of leftactions that transition back to the nonterminal state, followed by a
leftaction transitioning to termination. All of these episodes have a return of 1, so the
G0factor can be ignored. To get the expected square we need only consider each length

of episode, multiplying the probability of the episode’s occurrence by the square of its
importance-sampling ratio, and add these up:
=1
2·0.1✓1
0.5◆2
(the length 1 episode)
+1
2·0.9·1
2·0.1✓1
0.51
0.5◆2
(the length 2 episode)
+1
2·0.9·1
2·0.9·1
2·0.1✓1
0.51
0.51
0.5◆2
(the length 3 episode)
+···
=0.11X
k=00.9k·2k·2= 0 .21X
k=01.8k=1.
Exercise 5.6 What is the equation analogous to (5.6) foraction values Q(s, a) instead of
state values V(s), again given returns generated using b? ⇤
Exercise 5.7 In learning curves such as those shown in Figure 5.3 error generally decreases

=1
2·0.1✓1
0.5◆2
+1
2·0.9·1
2·0.1✓1
0.51
0.5◆2
+1
2·0.9·1
2·0.9·1
2·0.1✓1
0.51
0.51
0.5◆2
+···
=0.11X
with training, as indeed happened for the ordinary importance-sampling method. But for
the weighted importance-sampling method error ﬁrst increased and then decreased. Why
do you think this happened? ⇤
Exercise 5.8 The results with Example 5.5 and shown in Figure 5.4 used a ﬁrst-visit MC
method. Suppose that instead an every-visit MC method was used on the same problem.
Would the variance of the estimator still be inﬁnite? Why or why not? ⇤

5.7. O↵-policy Monte Carlo Control 109
5.6 Incremental Implementation
Monte Carlo prediction methods can be implemented incrementally, on an episode-by-
episode basis, using extensions of the techniques described in Chapter 2 (Section 2.4).
Whereas in Chapter 2 we averaged rewards , in Monte Carlo methods we average returns .
In all other respects exactly the same methods as used in Chapter 2 can be used for on-
policy Monte Carlo methods. For o↵-policy Monte Carlo methods, we need to separately
consider those that use ordinary importance sampling and those that use weighted
importance sampling.
In ordinary importance sampling, the returns are scaled by the importance sampling
ratio ⇢t:T(t)1(5.3) , then simply averaged, as in (5.5) . For these methods we can again
use the incremental methods of Chapter 2, but using the scaled returns in place of
the rewards of that chapter. This leaves the case of o↵-policy methods using weighted

Vn+1.=Vn+Wn
Cnh
GnVni
,n 1, (5.8)
and
Cn+1.=Cn+Wn+1,
where C0.=0 (and V1is arbitrary and thus need not be speciﬁed). The box on the
next page contains a complete episode-by-episode incremental algorithm for Monte Carlo
policy evaluation. The algorithm is nominally for the o↵-policy case, using weighted
importance sampling, but applies as well to the on-policy case just by choosing the
target and behavior policies as the same (in which case ( ⇡=b),Wis always 1). The
approximation Qconverges to q⇡(for all encountered state–action pairs) while actions
are selected according to a potentially di↵erent policy, b.
Exercise 5.9 Modify the algorithm for ﬁrst-visit MC policy evaluation (Section 5.1) to
use the incremental implementation for sample averages described in Section 2.4. ⇤
Exercise 5.10 Derive the weighted-average update rule (5.8) from (5.7). Follow the
pattern of the derivation of the unweighted rule (2.3). ⇤

C(St,At)[GQ(St,At)]
W W⇡(At|St)
b(At|St)
5.7 O↵-policy Monte Carlo Control
We are now ready to present an example of the second class of learning control methods
we consider in this book: o↵-policy methods. Recall that the distinguishing feature of
on-policy methods is that they estimate the value of a policy while using it for control.
In o↵-policy methods these two functions are separated. The policy used to generate
behavior, called the behavior policy, may in fact be unrelated to the policy that is
evaluated and improved, called the target policy. An advantage of this separation is
that the target policy may be deterministic (e.g., greedy), while the behavior policy can
continue to sample all possible actions.
O↵-policy Monte Carlo control methods use one of the techniques presented in the
preceding two sections. They follow the behavior policy while learning about and
improving the target policy. These techniques require that the behavior policy has a

nonzero probability of selecting all actions that might be selected by the target policy
(coverage). To explore all possibilities, we require that the behavior policy be soft (i.e.,
that it select all actions in all states with nonzero probability).
The box on the next page shows an o↵-policy Monte Carlo control method, based on
GPI and weighted importance sampling, for estimating ⇡⇤andq⇤. The target policy
⇡⇡⇡⇤is the greedy policy with respect to Q, which is an estimate of q⇡. The behavior

policy bcan be anything, but in order to assure convergence of ⇡to the optimal policy, an
inﬁnite number of returns must be obtained for each pair of state and action. This can be
assured by choosing bto be "-soft. The policy ⇡converges to optimal at all encountered
states even though actions are selected according to a di↵erent soft policy b,w h i c hm a y
change between or even within episodes.

IfAt6=⇡(St) then exit inner Loop (proceed to next episode)
W W1
b(At|St)
A potential problem is that this method learns only from the tails of episodes, when
all of the remaining actions in the episode are greedy. If nongreedy actions are common,
then learning will be slow, particularly for states appearing in the early portions of
long episodes. Potentially, this could greatly slow learning. There has been insucient
experience with o↵-policy Monte Carlo methods to assess how serious this problem is. If
it is serious, the most important way to address it is probably by incorporating temporal-
di↵erence learning, the algorithmic idea developed in the next chapter. Alternatively, if 
is less than 1, then the idea developed in the next section may also help signiﬁcantly.
Exercise 5.11 In the boxed algorithm for o↵-policy MC control, you may have been
expecting the Wupdate to have involved the importance-sampling ratio⇡(At|St)
b(At|St),b u t
instead it involves1
b(At|St). Why is this nevertheless correct? ⇤

Exercise 5.12: Racetrack (programming) Consider driving a race car around a turn
like those shown in Figure 5.5. You want to go as fast as possible, but not so fast as
to run o↵ the track. In our simpliﬁed racetrack, the car is at one of a discrete set of
grid positions, the cells in the diagram. The velocity is also discrete, a number of grid
cells moved horizontally and vertically per time step. The actions are increments to the

velocity components. Each may be changed by +1, 1, or 0 in each step, for a total of
nine (3 ⇥3) actions. Both velocity components are restricted to be nonnegative and less
than 5, and they cannot both be zero except at the starting line. Each episode begins
in one of the randomly selected start states with both velocity components zero and
ends when the car crosses the ﬁnish line. The rewards are 1 for each step until the car
crosses the ﬁnish line. If the car hits the track boundary, it is moved back to a random

position on the starting line, both velocity components are reduced to zero, and the

112 Chapter 5: Monte Carlo Methods
Starting lineFinishline
Starting lineFinishlineFigure 5.5: A couple of right turns for the racetrack task.
episode continues. Before updating the car’s location at each time step, check to see if
the projected path of the car intersects the track boundary. If it intersects the ﬁnish line,
the episode ends; if it intersects anywhere else, the car is considered to have hit the track
boundary and is sent back to the starting line. To make the task more challenging, with
probability 0.1 at each time step the velocity increments are both zero, independently of
the intended increments. Apply a Monte Carlo control method to this task to compute
the optimal policy from each starting state. Exhibit several trajectories following the
optimal policy (but turn the noise o↵ for these trajectories). ⇤
5.8 *Discounting-aware Importance Sampling
The o↵-policy methods that we have considered so far are based on forming importance-

sampling weights for returns considered as unitary wholes, without taking into account
the returns’ internal structures as sums of discounted rewards. We now brieﬂy consider
cutting-edge research ideas for using this structure to signiﬁcantly reduce the variance of
o↵-policy estimators.
For example, consider the case where episodes are long and is signiﬁcantly less than
1. For concreteness, say that episodes last 100 steps and that = 0. The return from
time 0 will then be just G0=R1, but its importance sampling ratio will be a product of
100 factors,⇡(A0|S0)

b(A0|S0)⇡(A1|S1)
b(A1|S1)···⇡(A99|S99)
b(A99|S99). In ordinary importance sampling, the return
will be scaled by the entire product, but it is really only necessary to scale by the ﬁrst
factor, by⇡(A0|S0)
b(A0|S0). The other 99 factors⇡(A1|S1)
b(A99|S99)are irrelevant because
after the ﬁrst reward the return has already been determined. These later factors are
all independent of the return and of expected value 1; they do not change the expected

update, but they add enormously to its variance. In some cases they could even make the

5.10. Summary 115
which we might expect to sometimes be of lower variance.
Is there a per-decision version of weighted importance sampling? This is less clear. So
far, all the estimators that have been proposed for this that we know of are not consistent
(that is, they do not converge to the true value with inﬁnite data).
⇤Exercise 5.13 Show the steps to derive (5.14) from (5.12). ⇤
⇤Exercise 5.14 Modify the algorithm for o↵-policy Monte Carlo control (page 111) to use
the idea of the truncated weighted-average estimator (5.10) . Note that you will ﬁrst need
to convert this equation to action values. ⇤
The Monte Carlo methods presented in this chapter learn value functions and optimal
policies from experience in the form of sample episodes . This gives them at least three
kinds of advantages over DP methods. First, they can be used to learn optimal behavior
directly from interaction with the environment, with no model of the environment’s

dynamics. Second, they can be used with simulation or sample models . For surprisingly
many applications it is easy to simulate sample episodes even though it is dicult to
construct the kind of explicit model of transition probabilities required by DP methods.
Third, it is easy and ecient to focus Monte Carlo methods on a small subset of the states.
A region of special interest can be accurately evaluated without going to the expense of
accurately evaluating the rest of the state set (we explore this further in Chapter 8).

A fourth advantage of Monte Carlo methods, which we discuss later in the book, is
that they may be less harmed by violations of the Markov property. This is because they
do not update their value estimates on the basis of the value estimates of successor states.
In other words, it is because they do not bootstrap.
In designing Monte Carlo control methods we have followed the overall schema of
generalized policy iteration (GPI) introduced in Chapter 4. GPI involves interacting
processes of policy evaluation and policy improvement. Monte Carlo methods provide an

alternative policy evaluation process. Rather than use a model to compute the value of
each state, they simply average many returns that start in the state. Because a state’s
value is the expected return, this average can become a good approximation to the
value. In control methods we are particularly interested in approximating action-value
functions, because these can be used to improve the policy without requiring a model of
the environment’s transition dynamics. Monte Carlo methods intermix policy evaluation

and policy improvement steps on an episode-by-episode basis, and can be incrementally
implemented on an episode-by-episode basis.
Maintaining sucient exploration is an issue in Monte Carlo control methods. It is
not enough just to select the actions currently estimated to be best, because then no
returns will be obtained for alternative actions, and it may never be learned that they
are actually better. One approach is to ignore this problem by assuming that episodes
begin with state–action pairs randomly selected to cover all possibilities. Such exploring

starts can sometimes be arranged in applications with simulated episodes, but are unlikely
in learning from real experience. In on-policy methods, the agent commits to always

116 Chapter 5: Monte Carlo Methods
exploring and tries to ﬁnd the best policy that still explores. In o↵-policy methods, the
agent also explores, but learns a deterministic optimal policy that may be unrelated to
the policy followed.
O↵-policy prediction refers to learning the value function of a target policy from data
generated by a di↵erent behavior policy . Such learning methods are based on some form
ofimportance sampling , that is, on weighting returns by the ratio of the probabilities of
taking the observed actions under the two policies, thereby transforming their expectations
from the behavior policy to the target policy. Ordinary importance sampling uses a
simple average of the weighted returns, whereas weighted importance sampling uses a
weighted average. Ordinary importance sampling produces unbiased estimates, but has
larger, possibly inﬁnite, variance, whereas weighted importance sampling always has
ﬁnite variance and is preferred in practice. Despite their conceptual simplicity, o↵-policy

Monte Carlo methods for both prediction and control remain unsettled and are a subject
of ongoing research.
The Monte Carlo methods treated in this chapter di↵er from the DP methods treated
in the previous chapter in two major ways. First, they operate on sample experience,
and thus can be used for direct learning without a model. Second, they do not bootstrap.
That is, they do not update their value estimates on the basis of other value estimates.
These two di↵erences are not tightly linked, and can be separated. In the next chapter

we consider methods that learn from experience, like Monte Carlo methods, but also
bootstrap, like DP methods.
The term “Monte Carlo” dates from the 1940s, when physicists at Los Alamos devised
games of chance that they could study to help understand complex physical phenomena
relating to the atom bomb. Coverage of Monte Carlo methods in this sense can be found
in several textbooks (e.g., Kalos and Whitlock, 1986; Rubinstein, 1981).
5.1–2 Singh and Sutton (1996) distinguished between every-visit and ﬁrst-visit MC

methods and proved results relating these methods to reinforcement learning
algorithms. The blackjack example is based on an example used by Widrow,
Gupta, and Maitra (1973). The soap bubble example is a classical Dirichlet
problem whose Monte Carlo solution was ﬁrst proposed by Kakutani (1945; see
Hersh and Griego, 1969; Doyle and Snell, 1984).
Barto and Du↵ (1994) discussed policy evaluation in the context of classical
Monte Carlo algorithms for solving systems of linear equations. They used the

analysis of Curtiss (1954) to point out the computational advantages of Monte
Carlo policy evaluation for large problems.
5.3–4 Monte Carlo ES was introduced in the 1998 edition of this book. That may have
been the ﬁrst explicit connection between Monte Carlo estimation and control
methods based on policy iteration. An early use of Monte Carlo methods to
estimate action values in a reinforcement learning context was by Michie and

Bibliographical and Historical Remarks 117
Chambers (1968). In pole balancing (page 56), they used averages of episode
durations to assess the worth (expected balancing “life”) of each possible action
in each state, and then used these assessments to control action selections. Their
method is similar in spirit to Monte Carlo ES with every-visit MC estimates.
Narendra and Wheeler (1986) studied a Monte Carlo method for ergodic ﬁnite
Markov chains that used the return accumulated between successive visits to the
same state as a reward for adjusting a learning automaton’s action probabilities.
5.5 Ecient o↵-policy learning has become recognized as an important challenge
that arises in several ﬁelds. For example, it is closely related to the idea of
“interventions” and “counterfactuals” in probabalistic graphical (Bayesian) models
(e.g., Pearl, 1995; Balke and Pearl, 1994). O↵-policy methods using importance
sampling have a long history and yet still are not well understood. Weighted

importance sampling, which is also sometimes called normalized importance
sampling (e.g., Koller and Friedman, 2009), is discussed by Rubinstein (1981),
Hesterberg (1988), Shelton (2001), and Liu (2001) among others.
The target policy in o↵-policy learning is sometimes referred to in the literature
as the “estimation” policy, as it was in the ﬁrst edition of this book.
5.7 The racetrack exercise is adapted from Barto, Bradtke, and Singh (1995), and
from Gardner (1973).

5.8 Our treatment of the idea of discounting-aware importance sampling is based on
the analysis of Sutton, Mahmood, Precup, and van Hasselt (2014). It has been
worked out most fully to date by Mahmood (2017; Mahmood, van Hasselt, and
Sutton, 2014).
5.9 Per-decision importance sampling was introduced by Precup, Sutton, and Singh
(2000). They also combined o↵-policy learning with temporal-di↵erence learning,
eligibility traces, and approximation methods, introducing subtle issues that we
consider in later chapters.

Chapter 6
Temporal-Di↵erence Learning
If one had to identify one idea as central and novel to reinforcement learning, it would
undoubtedly be temporal-di↵erence (TD) learning. TD learning is a combination of
Monte Carlo ideas and dynamic programming (DP) ideas. Like Monte Carlo methods,
TD methods can learn directly from raw experience without a model of the environment’s
dynamics. Like DP, TD methods update estimates based in part on other learned
estimates, without waiting for a ﬁnal outcome (they bootstrap). The relationship between
TD, DP, and Monte Carlo methods is a recurring theme in the theory of reinforcement
learning; this chapter is the beginning of our exploration of it. Before we are done, we
will see that these ideas and methods blend into each other and can be combined in many
ways. In particular, in Chapter 7 we introduce n-step algorithms, which provide a bridge
from TD to Monte Carlo methods, and in Chapter 12 we introduce the TD( ) algorithm,
which seamlessly uniﬁes them.

As usual, we start by focusing on the policy evaluation or prediction problem, the
problem of estimating the value function v⇡for a given policy ⇡. For the control problem
(ﬁnding an optimal policy), DP, TD, and Monte Carlo methods all use some variation of
generalized policy iteration (GPI). The di↵erences in the methods are primarily di↵erences
in their approaches to the prediction problem.
6.1 TD Prediction
Both TD and Monte Carlo methods use experience to solve the prediction problem. Given

some experience following a policy ⇡, both methods update their estimate Vofv⇡for
the nonterminal states Stoccurring in that experience. Roughly speaking, Monte Carlo
methods wait until the return following the visit is known, then use that return as a
target for V(St). A simple every-visit Monte Carlo method suitable for nonstationary
environments is
GtV(St)i
, (6.1)
119

120 Chapter 6: Temporal-Di↵erence Learning
where Gtis the actual return following time t, and ↵is a constant step-size parameter (c.f.,
Equation 2.4). Let us call this method constant- ↵MC. Whereas Monte Carlo methods
must wait until the end of the episode to determine the increment to V(St) (only then is
Gtknown), TD methods need to wait only until the next time step. At time t+1t h e y
immediately form a target and make a useful update using the observed reward Rt+1and
the estimate V(St+1). The simplest TD method makes the update
Rt+1+V(St+1)V(St)i
(6.2)
immediately on transition to St+1and receiving Rt+1. In e↵ect, the target for the Monte
Carlo update is Gt, whereas the target for the TD update is Rt+1+V(St+1). This TD
method is called TD(0) , orone-step TD , because it is a special case of the TD( ) and
n-step TD methods developed in Chapter 12 and Chapter 7. The box below speciﬁes
TD(0) completely in procedural form.
Tabular TD(0) for estimating v⇡
Input: the policy ⇡to be evaluated

Initialize V(s), for all s2S+, arbitrarily except that V(terminal )=0
Loop for each episode:
Initialize S
Loop for each step of episode:
A action given by ⇡forS
Take action A, observe R,S0
V(S) V(S)+↵⇥
R+V(S0)V(S)⇤
S S0
until Sis terminal
Because TD(0) bases its update in part on an existing estimate, we say that it is a
bootstrapping method, like DP. We know from Chapter 3 that
v⇡(s).=E⇡[Gt|St=s] (6.3)
=E⇡[Rt+1+v⇡(St+1)|St=s]. (6.4)
Roughly speaking, Monte Carlo methods use an estimate of (6.3) as a target, whereas
DP methods use an estimate of (6.4) as a target. The Monte Carlo target is an estimate
because the expected value in (6.3) is not known; a sample return is used in place of the
real expected return. The DP target is an estimate not because of the expected values,
which are assumed to be completely provided by a model of the environment, but because
v⇡(St+1) is not known and the current estimate, V(St+1), is used instead. The TD target

is an estimate for both reasons: it samples the expected values in (6.4) andit uses the
current estimate Vinstead of the true v⇡. Thus, TD methods combine the sampling of

6.1. TD Prediction 121
Monte Carlo with the bootstrapping of DP. As we shall see, with care and imagination
this can take us a long way toward obtaining the advantages of both Monte Carlo and
DP methods.TD(0)Shown to the right is the backup diagram for tabular TD(0). The value
estimate for the state node at the top of the backup diagram is updated on
the basis of the one sample transition from it to the immediately following
state. We refer to TD and Monte Carlo updates as sample updates because
they involve looking ahead to a sample successor state (or state–action pair),
using the value of the successor and the reward along the way to compute a
backed-up value, and then updating the value of the original state (or state–
action pair) accordingly. Sample updates di↵er from the expected updates
of DP methods in that they are based on a single sample successor rather than on a
complete distribution of all possible successors.
Finally, note that the quantity in brackets in the TD(0) update is a sort of error,

measuring the di↵erence between the estimated value of Stand the better estimate
Rt+1+V(St+1). This quantity, called the TD error , arises in various forms throughout
reinforcement learning:
t.=Rt+1+V(St+1)V(St). (6.5)
Notice that the TD error at each time is the error in the estimate made at that time .
Because the TD error depends on the next state and next reward, it is not actually
available until one time step later. That is, tis the error in V(St), available at time
t+ 1. Also note that if the array Vdoes not change during the episode (as it does not in

122 Chapter 6: Temporal-Di↵erence Learning
Example 6.1: Driving Home Each day as you drive home from work, you try to
predict how long it will take to get home. When you leave your oce, you note the time,
the day of week, the weather, and anything else that might be relevant. Say on this
Friday you are leaving at exactly 6 o’clock, and you estimate that it will take 30 minutes
to get home. As you reach your car it is 6:05, and you notice it is starting to rain. Trac
is often slower in the rain, so you reestimate that it will take 35 minutes from then, or a
total of 40 minutes. Fifteen minutes later you have completed the highway portion of
your journey in good time. As you exit onto a secondary road you cut your estimate of
total travel time to 35 minutes. Unfortunately, at this point you get stuck behind a slow
truck, and the road is too narrow to pass. You end up having to follow the truck until
you turn onto the side street where you live at 6:40. Three minutes later you are home.

The sequence of states, times, and predictions is thus as follows:
Elapsed Time Predicted Predicted
State (minutes) Time to Go Total Time
leaving oce, friday at 6 0 30 30
reach car, raining 5 35 40
exiting highway 20 15 35
2ndary road, behind truck 30 10 40
entering home street 40 3 43
arrive home 43 0 43
The rewards in this example are the elapsed times on each leg of the journey.1We are
not discounting ( = 1), and thus the return for each state is the actual time to go from
that state. The value of each state is the expected time to go. The second column of

numbers gives the current estimated value for each state encountered.
A simple way to view the operation of Monte Carlo methods is to plot the predicted
total time (the last column) over the sequence, as in Figure 6.1 (left). The red arrows
show the changes in predictions recommended by the constant- ↵MC method (6.1) , for
↵= 1. These are exactly the errors between the estimated value (predicted time to go)
in each state and the actual return (actual time to go). For example, when you exited

the highway you thought it would take only 15 minutes more to get home, but in fact it
took 23 minutes. Equation 6.1 applies at this point and determines an increment in the
estimate of time to go after exiting the highway. The error, GtV(St), at this time is
eight minutes. Suppose the step-size parameter, ↵,i s1 /2. Then the predicted time to go
after exiting the highway would be revised upward by four minutes as a result of this
experience. This is probably too large a change in this case; the truck was probably just

an unlucky break. In any event, the change can only be made o✏ine, that is, after you
have reached home. Only at this point do you know any of the actual returns.
Is it necessary to wait until the ﬁnal outcome is known before learning can begin?
Suppose on another day you again estimate when leaving your oce that it will take 30
minutes to drive home, but then you become stuck in a massive trac jam. Twenty-ﬁve
minutes after leaving the oce you are still bumper-to-bumper on the highway. You now

1If this were a control problem with the objective of minimizing travel time, then we would of course
make the rewards the negative of the elapsed time. But because we are concerned here only with
prediction (policy evaluation), we can keep things simple by using positive numbers.

6.1. TD Prediction 123
road30354045Predictedtotaltraveltimeleavingofficeexitinghighway2ndaryhomearriveSituationactual outcome
reachcarstreethomeactualoutcome
Situation30354045Predictedtotaltraveltimeroadleavingofficeexitinghighway2ndaryhomearrivereachcarstreethomeFigure 6.1: Changes recommended in the driving home example by Monte Carlo methods (left)
and TD methods (right).
estimate that it will take another 25 minutes to get home, for a total of 50 minutes. As
you wait in trac, you already know that your initial estimate of 30 minutes was too
optimistic. Must you wait until you get home before increasing your estimate for the
initial state? According to the Monte Carlo approach you must, because you don’t yet
know the true return.
According to a TD approach, on the other hand, you would learn immediately, shifting
your initial estimate from 30 minutes toward 50. In fact, each estimate would be shifted
toward the estimate that immediately follows it. Returning to our ﬁrst day of driving,

Figure 6.1 (right) shows the changes in the predictions recommended by the TD rule
(6.2) (these are the changes made by the rule if ↵= 1). Each error is proportional to the
change over time of the prediction, that is, to the temporal di↵erences in predictions.
Besides giving you something to do while waiting in trac, there are several computa-
tional reasons why it is advantageous to learn based on your current predictions rather
than waiting until termination when you know the actual return. We brieﬂy discuss some
of these in the next section.

Exercise 6.2 This is an exercise to help develop your intuition about why TD methods
are often more ecient than Monte Carlo methods. Consider the driving home example
and how it is addressed by TD and Monte Carlo methods. Can you imagine a scenario
in which a TD update would be better on average than a Monte Carlo update? Give
an example scenario—a description of past experience and a current state—in which
you would expect the TD update to be better. Here’s a hint: Suppose you have lots of

experience driving home from work. Then you move to a new building and a new parking
lot (but you still enter the highway at the same place). Now you are starting to learn
predictions for the new building. Can you see why TD updates are likely to be much
better, at least initially, in this case? Might the same sort of thing happen in the original
scenario? ⇤

124 Chapter 6: Temporal-Di↵erence Learning
6.2 Advantages of TD Prediction Methods
TD methods update their estimates based in part on other estimates. They learn a
guess from a guess—they bootstrap . Is this a good thing to do? What advantages do
TD methods have over Monte Carlo and DP methods? Developing and answering such
questions will take the rest of this book and more. In this section we brieﬂy anticipate
some of the answers.
Obviously, TD methods have an advantage over DP methods in that they do not
require a model of the environment, of its reward and next-state probability distributions.
The next most obvious advantage of TD methods over Monte Carlo methods is that
they are naturally implemented in an online, fully incremental fashion. With Monte
Carlo methods one must wait until the end of an episode, because only then is the return
known, whereas with TD methods one need wait only one time step. Surprisingly often

this turns out to be a critical consideration. Some applications have very long episodes, so
that delaying all learning until the end of the episode is too slow. Other applications are
continuing tasks and have no episodes at all. Finally, as we noted in the previous chapter,
some Monte Carlo methods must ignore or discount episodes on which experimental
actions are taken, which can greatly slow learning. TD methods are much less susceptible

to these problems because they learn from each transition regardless of what subsequent
actions are taken.
But are TD methods sound? Certainly it is convenient to learn one guess from the
next, without waiting for an actual outcome, but can we still guarantee convergence
to the correct answer? Happily, the answer is yes. For any ﬁxed policy ⇡, TD(0) has
been proved to converge to v⇡, in the mean for a constant step-size parameter if it is
suciently small, and with probability 1 if the step-size parameter decreases according to

the usual stochastic approximation conditions (2.7). Most convergence proofs apply only
to the table-based case of the algorithm presented above (6.2), but some also apply to
the case of general linear function approximation. These results are discussed in a more
general setting in Chapter 9.
If both TD and Monte Carlo methods converge asymptotically to the correct predictions,
then a natural next question is “Which gets there ﬁrst?” In other words, which method
learns faster? Which makes the more ecient use of limited data? At the current time

this is an open question in the sense that no one has been able to prove mathematically
that one method converges faster than the other. In fact, it is not even clear what is the
most appropriate formal way to phrase this question! In practice, however, TD methods
have usually been found to converge faster than constant- ↵MC methods on stochastic
tasks, as illustrated in Example 6.2.

6.2. Advantages of TD Prediction Methods 125
Example 6.2 Random Walk
In this example we empirically compare the prediction abilities of TD(0) and
constant- ↵MC when applied to the following Markov reward process:
ABCDE100000startAMarkov reward process , or MRP, is a Markov decision process without actions.
We will often use MRPs when focusing on the prediction problem, in which there is
no need to distinguish the dynamics due to the environment from those due to the
agent. In this MRP, all episodes start in the center state, C,t h e np r o c e e de i t h e rl e f t
or right by one state on each step, with equal probability. Episodes terminate either
on the extreme left or the extreme right. When an episode terminates on the right,
a reward of +1 occurs; all other rewards are zero. For example, a typical episode
might consist of the following state-and-reward sequence: C,0,B,0,C,0,D,0,E,1.
Because this task is undiscounted, the true value of each state is the probability of

terminating on the right if starting from that state. Thus, the true value of the
center state is v⇡(C)=0 .5. The true values of all the states, Athrough E, are
1
6,2
6,3
6,4
6, and5
6.
0.8
00.20.40.6
ABCDE0101100
StateEstimatedvalueTrue valuesEstimatedvalue00.050.10.150.20.25
0255075100Walks / EpisodesTDMCRMS error,averagedover statesα=.01
α=.1α=.02α=.03α=.04α=.15α=.05Empirical RMS error, averaged over statesThe left graph above shows the values learned after various numbers of episodes
on a single run of TD(0). The estimates after 100 episodes are about as close as

1
6,2
6,3
6,4
6, and5
6.
0.8
they ever come to the true values—with a constant step-size parameter ( ↵=0.1
in this example), the values ﬂuctuate indeﬁnitely in response to the outcomes
of the most recent episodes. The right graph shows learning curves for the two
methods for various values of ↵. The performance measure shown is the root
mean-squared (RMS) error between the value function learned and the true value
function, averaged over the ﬁve states, then averaged over 100 runs. In all cases the

approximate value function was initialized to the intermediate value V(s)=0 .5, for
alls. The TD method was consistently better than the MC method on this task.

126 Chapter 6: Temporal-Di↵erence Learning
Exercise 6.3 From the results shown in the left graph of the random walk example it
appears that the ﬁrst episode results in a change in only V(A). What does this tell you
about what happened on the ﬁrst episode? Why was only the estimate for this one state
changed? By exactly how much was it changed? ⇤
Exercise 6.4 The speciﬁc results shown in the right graph of the random walk example
are dependent on the value of the step-size parameter, ↵. Do you think the conclusions
about which algorithm is better would be a↵ected if a wider range of ↵values were used?
Is there a di↵erent, ﬁxed value of ↵at which either algorithm would have performed
signiﬁcantly better than shown? Why or why not? ⇤
⇤Exercise 6.5 In the right graph of the random walk example, the RMS error of the
TD method seems to go down and then up again, particularly at high ↵’s. What could
have caused this? Do you think this always occurs, or might it be a function of how the

approximate value function was initialized? ⇤
Exercise 6.6 In Example 6.2 we stated that the true values for the random walk example
are1
6,2
6,3
6,4
6, and5
6, for states Athrough E. Describe at least two di↵erent ways that
these could have been computed. Which would you guess we actually used? Why? ⇤
6.3 Optimality of TD(0)
Suppose there is available only a ﬁnite amount of experience, say 10 episodes or 100
time steps. In this case, a common approach with incremental learning methods is to

are1
6,2
6,3
6,4
6, and5
present the experience repeatedly until the method converges upon an answer. Given an
approximate value function, V, the increments speciﬁed by (6.1) or (6.2) are computed
for every time step tat which a nonterminal state is visited, but the value function is
changed only once, by the sum of all the increments. Then all the available experience is
processed again with the new value function to produce a new overall increment, and so
on, until the value function converges. We call this batch updating because updates are

made only after processing each complete batch of training data.
Under batch updating, TD(0) converges deterministically to a single answer independent
of the step-size parameter, ↵, as long as ↵is chosen to be suciently small. The constant-
↵MC method also converges deterministically under the same conditions, but to a
di↵erent answer. Understanding these two answers will help us understand the di↵erence
between the two methods. Under normal updating the methods do not move all the way
to their respective batch answers, but in some sense they take steps in these directions.

Before trying to understand the two answers in general, for all possible tasks, we ﬁrst
look at a few examples.
Example 6.3: Random walk under batch updating Batch-updating versions of
TD(0) and constant- ↵MC were applied as follows to the random walk prediction example
(Example 6.2). After each new episode, all episodes seen so far were treated as a batch.
They were repeatedly presented to the algorithm, either TD(0) or constant- ↵MC, with
↵suciently small that the value function converged. The resulting value function was

then compared with v⇡, and the average root mean-squared error across the ﬁve states
(and across 100 independent repetitions of the whole experiment) was plotted to obtain

6.3. Optimality of TD(0) 127
the learning curves shown in Figure 6.2. Note that the batch TD method was consistently
better than the batch Monte Carlo method.
.0.05.1.15.2.25
0255075100TDMCBATCH TRAINING
Walks / EpisodesRMS error,averagedover statesFigure 6.2: Performance of TD(0) and constant- ↵
MC under batch training on the random walk task.Under batch training, constant- ↵
MC converges to values, V(s), that
are sample averages of the actual re-
turns experienced after visiting each
state s. These are optimal estimates
in the sense that they minimize the
mean-squared error from the actual
returns in the training set. In this
sense it is surprising that the batch
TD method was able to perform
better according to the root mean-
squared error measure shown in the
ﬁgure to the right. How is it that
batch TD was able to perform better
than this optimal method? The an-
swer is that the Monte Carlo method
is optimal only in a limited way, and

that TD is optimal in a way that is more relevant to predicting returns.
Example 6.4: You are the Predictor Place yourself now in the role of the predictor
of returns for an unknown Markov reward process. Suppose you observe the following
eight episodes:
A,0,B,0 B,1
B,1 B,1
B,1 B,1
B,1 B,0
This means that the ﬁrst episode started in state A, transitioned to Bwith a reward of
0, and then terminated from Bwith a reward of 0. The other seven episodes were even

B,1 B,1
B,1 B,1
B,1 B,0
shorter, starting from Band terminating immediately. Given this batch of data, what
would you say are the optimal predictions, the best values for the estimates V(A) and
V(B)? Everyone would probably agree that the optimal value for V(B)i s3
4, because six
out of the eight times in state Bthe process terminated immediately with a return of 1,
and the other two times in Bthe process terminated immediately with a return of 0.

But what is the optimal value for the estimate V(A) given this data? Here there areABr = 1100%75%25%r = 0r = 0two reasonable answers. One is to observe that 100% of the
times the process was in state Ait traversed immediately to
B(with a reward of 0); and because we have already decided
that Bhas value3
4, therefore Amust have value3
4as well.
One way of viewing this answer is that it is based on ﬁrst
modeling the Markov process, in this case as shown to the

4as well.
right, and then computing the correct estimates given the
model, which indeed in this case gives V(A)=3
4.T h i s i s

128 Chapter 6: Temporal-Di↵erence Learning
also the answer that batch TD(0) gives.
The other reasonable answer is simply to observe that we have seen Aonce and the
return that followed it was 0; we therefore estimate V(A) as 0. This is the answer that
batch Monte Carlo methods give. Notice that it is also the answer that gives minimum
squared error on the training data. In fact, it gives zero error on the data. But still we
expect the ﬁrst answer to be better. If the process is Markov, we expect that the ﬁrst
answer will produce lower error on future data, even though the Monte Carlo answer is
better on the existing data.
Example 6.4 illustrates a general di↵erence between the estimates found by batch
TD(0) and batch Monte Carlo methods. Batch Monte Carlo methods always ﬁnd the
estimates that minimize mean-squared error on the training set, whereas batch TD(0)
always ﬁnds the estimates that would be exactly correct for the maximum-likelihood

model of the Markov process. In general, the maximum-likelihood estimate of a parameter
is the parameter value whose probability of generating the data is greatest. In this case,
the maximum-likelihood estimate is the model of the Markov process formed in the
obvious way from the observed episodes: the estimated transition probability from itoj
is the fraction of observed transitions from ithat went to j, and the associated expected
reward is the average of the rewards observed on those transitions. Given this model,

we can compute the estimate of the value function that would be exactly correct if the
model were exactly correct. This is called the certainty-equivalence estimate because it
is equivalent to assuming that the estimate of the underlying process was known with
certainty rather than being approximated. In general, batch TD(0) converges to the
certainty-equivalence estimate.
This helps explain why TD methods converge more quickly than Monte Carlo methods.
In batch form, TD(0) is faster than Monte Carlo methods because it computes the

true certainty-equivalence estimate. This explains the advantage of TD(0) shown in the
batch results on the random walk task (Figure 6.2). The relationship to the certainty-
equivalence estimate may also explain in part the speed advantage of nonbatch TD(0)
(e.g., Example 6.2, page 125, right graph). Although the nonbatch methods do not achieve
either the certainty-equivalence or the minimum squared-error estimates, they can be
understood as moving roughly in these directions. Nonbatch TD(0) may be faster than

constant- ↵MC because it is moving toward a better estimate, even though it is not
getting all the way there. At the current time nothing more deﬁnite can be said about
the relative eciency of online TD and Monte Carlo methods.
Finally, it is worth noting that although the certainty-equivalence estimate is in some
sense an optimal solution, it is almost never feasible to compute it directly. If n=|S|is
the number of states, then just forming the maximum-likelihood estimate of the process
may require on the order of n2memory, and computing the corresponding value function

requires on the order of n3computational steps if done conventionally. In these terms it
is indeed striking that TD methods can approximate the same solution using memory
no more than order nand repeated computations over the training set. On tasks with
large state spaces, TD methods may be the only feasible way of approximating the
certainty-equivalence solution.
⇤Exercise 6.7 Design an o↵-policy version of the TD(0) update that can be used with

6.4. Sarsa: On-policy TD Control 129
arbitrary target policy ⇡and covering behavior policy b, using at each step tthe importance
sampling ratio ⇢t:t(5.3). ⇤
6.4 Sarsa: On-policy TD Control
We turn now to the use of TD prediction methods for the control problem. As usual, we
follow the pattern of generalized policy iteration (GPI), only this time using TD methods
for the evaluation or prediction part. As with Monte Carlo methods, we face the need to
trade o↵ exploration and exploitation, and again approaches fall into two main classes:
on-policy and o↵-policy. In this section we present an on-policy TD control method.
The ﬁrst step is to learn an action-value function rather than a state-value function.
In particular, for an on-policy method we must estimate q⇡(s, a) for the current behavior
policy ⇡and for all states sand actions a. This can be done using essentially the same TD
method described above for learning v⇡. Recall that an episode consists of an alternating

sequence of states and state–action pairs:AtRt+1StAt+1Rt+2St+1At+2Rt+3St+2At+3St+3. . .. . .In the previous section we considered transitions from state to state and learned the
values of states. Now we consider transitions from state–action pair to state–action pair,
and learn the values of state–action pairs. Formally these cases are identical: they are
both Markov chains with a reward process. The theorems assuring the convergence of
state values under TD(0) also apply to the corresponding algorithm for action values:
Q(St,At) Q(St,At)+↵h
Rt+1+Q(St+1,At+1)Q(St,At)i

. (6.7)SarsaThis update is done after every transition from a nonterminal state St.I f
St+1is terminal, then Q(St+1,At+1) is deﬁned as zero. This rule uses every
element of the quintuple of events, ( St,At,Rt+1,St+1,At+1), that make up a
transition from one state–action pair to the next. This quintuple gives rise to
the name Sarsa for the algorithm. The backup diagram for Sarsa is as shown
to the right.
It is straightforward to design an on-policy control algorithm based on the Sarsa
prediction method. As in all on-policy methods, we continually estimate q⇡for the

behavior policy ⇡, and at the same time change ⇡toward greediness with respect to q⇡.
The general form of the Sarsa control algorithm is given in the box on the next page.
The convergence properties of the Sarsa algorithm depend on the nature of the policy’s
dependence on Q. For example, one could use "-greedy or "-soft policies. Sarsa converges
with probability 1 to an optimal policy and action-value function as long as all state–action
pairs are visited an inﬁnite number of times and the policy converges in the limit to

the greedy policy (which can be arranged, for example, with "-greedy policies by setting
"=1/t).
Exercise 6.8 Show that an action-value version of (6.6) holds for the action-value form
of the TD error t=Rt+1+Q(St+1,At+1)Q(St,At), again assuming that the values
don’t change from step to step. ⇤

130 Chapter 6: Temporal-Di↵erence Learning
Sarsa (on-policy TD control) for estimating Q⇡q⇤
Algorithm parameters: step size ↵2(0,1], small ">0
Initialize Q(s, a), for all s2S+,a2A(s),arbitrarily except that Q(terminal ,·)=0
Choose Afrom Susing policy derived from Q(e.g., "-greedy)
Choose A0from S0using policy derived from Q(e.g., "-greedy)
Q(S, A) Q(S, A)+↵⇥
R+Q(S0,A0)Q(S, A)⇤
S S0;A A0;
Example 6.5: Windy Gridworld Shown inset below is a standard gridworld, with
start and goal states, but with one di↵erence: there is a crosswind running upward
through the middle of the grid. The actions are the standard four— up, down, right ,
andleft —but in the middle region the resultant next states are shifted upward by a
“wind,” the strength of which varies from column to column. The strength of the wind
010002000300040005000600070008000050100150170

S S0;A A0;
Time stepsSG0000111122ActionsEpisodesis given below each column, in num-
ber of cells shifted upward. For ex-
ample, if you are one cell to the
right of the goal, then the action
left takes you to the cell just above
the goal. This is an undiscounted
episodic task, with constant rewards
of1 until the goal state is reached.
The graph to the right shows the
results of applying "-greedy Sarsa to
this task, with "=0.1,↵=0.5,
and the initial values Q(s, a)=0
for all s, a. The increasing slope of

the graph shows that the goal was
reached more quickly over time. By
8000 time steps, the greedy policy was long since optimal (a trajectory from it is shown
inset); continued "-greedy exploration kept the average episode length at about 17 steps,
two more than the minimum of 15. Note that Monte Carlo methods cannot easily be used
here because termination is not guaranteed for all policies. If a policy was ever found
that caused the agent to stay in the same state, then the next episode would never end.

Online learning methods such as Sarsa do not have this problem because they quickly
learn during the episode that such policies are poor, and switch to something else.
Exercise 6.9: Windy Gridworld with King’s Moves (programming) Re-solve the windy
gridworld assuming eight possible actions, including the diagonal moves, rather than the

6.5. Q-learning: O↵-policy TD Control 131
usual four. How much better can you do with the extra actions? Can you do even better
by including a ninth action that causes no movement at all other than that caused by
the wind? ⇤
Exercise 6.10: Stochastic Wind (programming) Re-solve the windy gridworld task with
King’s moves, assuming that the e↵ect of the wind, if there is any, is stochastic, sometimes
varying by 1 from the mean values given for each column. That is, a third of the time
you move exactly according to these values, as in the previous exercise, but also a third
of the time you move one cell above that, and another third of the time you move one
cell below that. For example, if you are one cell to the right of the goal and you move
left , then one-third of the time you move one cell above the goal, one-third of the time
you move two cells above the goal, and one-third of the time you move to the goal. ⇤
6.5 Q-learning: O↵-policy TD Control

One of the early breakthroughs in reinforcement learning was the development of an
o↵-policy TD control algorithm known as Q-learning (Watkins, 1989), deﬁned by
Rt+1+max
aQ(St+1,a)Q(St,At)i
. (6.8)
In this case, the learned action-value function, Q, directly approximates q⇤, the optimal
action-value function, independent of the policy being followed. This dramatically
simpliﬁes the analysis of the algorithm and enabled early convergence proofs. The policy

Rt+1+max
. (6.8)
still has an e↵ect in that it determines which state–action pairs are visited and updated.
However, all that is required for correct convergence is that all pairs continue to be
updated. As we observed in Chapter 5, this is a minimal requirement in the sense that
any method guaranteed to ﬁnd optimal behavior in the general case must require it.
Under this assumption and a variant of the usual stochastic approximation conditions on
the sequence of step-size parameters, Qhas been shown to converge with probability 1 to

q⇤. The Q-learning algorithm is shown below in procedural form.
Q-learning (o↵-policy TD control) for estimating ⇡⇡⇡⇤
R+max aQ(S0,a)Q(S, A)⇤
S S0

132 Chapter 6: Temporal-Di↵erence Learning
What is the backup diagram for Q-learning? The rule (6.8) updates a state–action
pair, so the top node, the root of the update, must be a small, ﬁlled action node. The
update is also from action nodes, maximizing over all those actions possible in the next
state. Thus the bottom nodes of the backup diagram should be all these action nodes.
Finally, remember that we indicate taking the maximum of these “next action” nodes
with an arc across them (Figure 3.4-right). Can you guess now what the diagram is? If
so, please do make a guess before turning to the answer in Figure 6.4 on page 134.
Example 6.6: Cli↵ Walking This gridworld example compares Sarsa and Q-learning,
highlighting the di↵erence between on-policy (Sarsa) and o↵-policy (Q-learning) methods.
Rewardperepsiode!!∀∀!#∃!∃∀!%∃
∀!∀∀%∀∀&∀∀∍∀∀∃∀∀EpisodesSarsaQ-learningSG  The Cliff R
Sum of rewardsduringepisodeR = -1Safer pathOptimal pathR = -100
EpisodesSarsaQ-learningSGr = !!∀∀The Cliffr =(!!saopR

RSum of rewardsduringepisodeR = -1safe pathoptimal pathR = -100
Episodes-25-50-75-1000100200300400500Consider the gridworld shown to the
right. This is a standard undis-
counted, episodic task, with start
and goal states, and the usual ac-
tions causing movement up, down,
right, and left. Reward is 1 on all
transitions except those into the re-
gion marked “The Cli↵.” Stepping
into this region incurs a reward of
100 and sends the agent instantly
back to the start.
performance of the Sarsa and Q-

learning methods with "-greedy ac-
tion selection, "=0.1. After an
initial transient, Q-learning learns
values for the optimal policy, that
which travels right along the edge
of the cli↵. Unfortunately, this re-
sults in its occasionally falling o↵
the cli↵ because of the "-greedy ac-
tion selection. Sarsa, on the other
hand, takes the action selection into
account and learns the longer but
safer path through the upper part
of the grid. Although Q-learning ac-
tually learns the values of the opti-
mal policy, its online performance

is worse than that of Sarsa, which
learns the roundabout policy. Of course, if "were gradually reduced, then both methods
would asymptotically converge to the optimal policy.
Exercise 6.11 Why is Q-learning considered an o↵-policy control method? ⇤
Exercise 6.12 Suppose action selection is greedy. Is Q-learning then exactly the same
algorithm as Sarsa? Will they make exactly the same action selections and weight
updates? ⇤

6.6. Expected Sarsa 133
6.6 Expected Sarsa
Consider the learning algorithm that is just like Q-learning except that instead of the
maximum over next state–action pairs it uses the expected value, taking into account
how likely each action is under the current policy. That is, consider the algorithm with
the update rule
Rt+1+E⇡[Q(St+1,At+1)|St+1]Q(St,At)i
 Q(St,At)+↵h
Rt+1+X
a⇡(a|St+1)Q(St+1,a)Q(St,At)i
,(6.9)
but that otherwise follows the schema of Q-learning. Given the next state, St+1,t h i s
algorithm moves deterministically in the same direction as Sarsa moves in expectation ,
and accordingly it is called Expected Sarsa . Its backup diagram is shown on the right in
Figure 6.4.
Expected Sarsa is more complex computationally than Sarsa but, in return, it eliminates
the variance due to the random selection of At+1. Given the same amount of experience
we might expect it to perform slightly better than Sarsa, and indeed it generally does.

Figure 6.3 shows summary results on the cli↵-walking task with Expected Sarsa compared
to Sarsa and Q-learning. Expected Sarsa retains the signiﬁcant advantage of Sarsa over
Q-learning on this problem. In addition, Expected Sarsa shows a signiﬁcant improvement

We then present results on two versions of the windygrid world problem, one with a deterministic environmentand one with a stochastic environment. We do so in orderto evaluate the inﬂuence of environment stochasticity onthe performance difference between Expected Sarsa andSarsa and conﬁrm the ﬁrst part of Hypothesis 2. We thenpresent results for different amounts of policy stochasticityto conﬁrm the second part of Hypothesis 2. For completeness,we also show the performance of Q-learning on this problem.Finally, we present results in other domains verifying theadvantages of Expected Sarsa in a broader setting. All resultspresented below are averaged over numerous independenttrials such that the standard error becomes negligible.A. Cliff WalkingWe begin by testing Hypothesis 1 using the cliff walkingtask, an undiscounted, episodic navigation task in which theagent has to ﬁnd its way from start to goal in a deterministicgrid world. Along the edge of the grid world is a cliff (seeFigure 1). The agent can take

we present results in other domains verifying theadvantages of Expected Sarsa in a broader setting. All resultspresented below are averaged over numerous independenttrials such that the standard error becomes negligible.A. Cliff WalkingWe begin by testing Hypothesis 1 using the cliff walkingtask, an undiscounted, episodic navigation task in which theagent has to ﬁnd its way from start to goal in a deterministicgrid world. Along the edge of the grid world is a cliff (seeFigure 1). The agent can take any of four movement actions:up,down,leftandright, each of which moves the agent onesquare in the corresponding direction. Each step results in areward of -1, except when the agent steps into the cliff area,which results in a reward of -100 and an immediate returnto the start state. The episode ends upon reaching the goalstate.SG

Fig. 1. The cliff walking task. The agent has to move from the start [S]to the goal [G], while avoiding stepping into the cliff (grey area).We evaluated the performance over the ﬁrst n episodes asa function of the learning rate↵using an-greedy policywith= 0.1. Figure 2 shows the result forn= 100andn= 100,000. We averaged the results over 50,000 runs and10 runs, respectively.Discussion.Expected Sarsa outperforms Q-learning andSarsa for all learning rate values, conﬁrming Hypothesis 1and providing some evidence for Hypothesis 2. The optimal↵value of Expected Sarsa forn= 100is 1, while forSarsa it is lower, as expected for a deterministic problem.That the optimal value of Q-learning is also lower than 1 issurprising, since Q-learning also has no stochasticity in itsupdates in a deterministic environment. Our explanation isthat Q-learning ﬁrst learns policies that are sub-optimal inthe greedy sense, i.e. walking towards the goal with a detourfurther from the cliff. Q-learning iteratively optimizes theseearly

for Hypothesis 2. The optimal↵value of Expected Sarsa forn= 100is 1, while forSarsa it is lower, as expected for a deterministic problem.That the optimal value of Q-learning is also lower than 1 issurprising, since Q-learning also has no stochasticity in itsupdates in a deterministic environment. Our explanation isthat Q-learning ﬁrst learns policies that are sub-optimal inthe greedy sense, i.e. walking towards the goal with a detourfurther from the cliff. Q-learning iteratively optimizes theseearly policies, resulting in a path more closely along the cliff.However, although this path is better in the off-line sense, interms of on-line performance it is worse. A large value of↵ensures the goal is reached quickly, but a value somewhatlower than 1 ensures that the agent does not try to walk righton the edge of the cliff immediately, resulting in a slightlybetter on-line performance.Forn= 100,000, the average return is equal for all↵values in case of Expected Sarsa and Q-learning. Thisindicates that the

policies, resulting in a path more closely along the cliff.However, although this path is better in the off-line sense, interms of on-line performance it is worse. A large value of↵ensures the goal is reached quickly, but a value somewhatlower than 1 ensures that the agent does not try to walk righton the edge of the cliff immediately, resulting in a slightlybetter on-line performance.Forn= 100,000, the average return is equal for all↵values in case of Expected Sarsa and Q-learning. Thisindicates that the algorithms have converged long before theend of the run for all↵values, since we do not see anyeffect of the initial learning phase. For Sarsa the performancecomes close to the performance of Expected Sarsa only for↵=0.1, while for large↵, the performance forn= 100,000even drops below the performance forn= 100. The reasonis that for large values of↵the Q values of Sarsa diverge.Although the policy is still improved over the initial randompolicy during the early stages of learning, divergence causesthe

the algorithms have converged long before theend of the run for all↵values, since we do not see anyeffect of the initial learning phase. For Sarsa the performancecomes close to the performance of Expected Sarsa only for↵=0.1, while for large↵, the performance forn= 100,000even drops below the performance forn= 100. The reasonis that for large values of↵the Q values of Sarsa diverge.Although the policy is still improved over the initial randompolicy during the early stages of learning, divergence causesthe policy to get worse in the long run.0.10.20.30.40.50.60.70.80.91−160−140−120−100−80−60−40−200

alphaaverage return
  
n = 100, Sarsan = 100, Q−learningn = 100, Expected Sarsan = 1E5, Sarsan = 1E5, Q−learningn = 1E5, Expected Sarsa

Fig. 2. Average return on the cliff walking task over the ﬁrstnepisodesforn=1 0 0andn=1 0 0,000using an-greedy policy with= 0.1. Thebig dots indicate the maximal values.B. Windy Grid WorldWe turn to the windy grid world task to further test Hy-pothesis 2. The windy grid world task is another navigationtask, where the agent has to ﬁnd its way from start to goal.The grid has a height of 7 and a width of 10 squares. Thereis a wind blowing in the ’up’ direction in the middle part ofthe grid, with a strength of 1 or 2 depending on the column.Figure 3 shows the grid world with a number below eachcolumn indicating the wind strength. Again, the agent canchoose between four movement actions:up,down,leftandright, each resulting in a reward of -1. The result of an actionis a movement of 1 square in the corresponding direction plusan additional movement in the ’up’ direction, correspondingwith the wind strength. For example, when the agent is inthe square right of the goal and takes a ’left’ action, it endsup in the

of 1 or 2 depending on the column.Figure 3 shows the grid world with a number below eachcolumn indicating the wind strength. Again, the agent canchoose between four movement actions:up,down,leftandright, each resulting in a reward of -1. The result of an actionis a movement of 1 square in the corresponding direction plusan additional movement in the ’up’ direction, correspondingwith the wind strength. For example, when the agent is inthe square right of the goal and takes a ’left’ action, it endsup in the square just above the goal.1) Deterministic Environment:We ﬁrst consider a de-terministic environment. As in the cliff walking task, weuse an-greedy policy with=0.1. Figure 4 shows theperformance as a function of the learning rate↵over theﬁrstnepisodes forn= 100andn= 100,000. Forn= 100

Expected SarsaSarsaQ-learningAsymptotic PerformanceInterim PerformanceQ-learningSum of rewardsper episode↵10.10.20.40.60.80.30.50.70.90-40-80-120Figure 6.3: Interim and asymptotic performance of TD control methods on the cli↵-walking
task as a function of ↵. All algorithms used an "-greedy policy with "=0.1. Asymptotic
performance is an average over 100,000 episodes whereas interim performance is an average
over the ﬁrst 100 episodes. These data are averages of over 50,000 and 10 runs for the interim
and asymptotic cases respectively. The solid circles mark the best interim performance of each
method. Adapted from van Seijen et al. (2009).

134 Chapter 6: Temporal-Di↵erence LearningQ-learning Expected SarsaFigure 6.4:The backup diagrams for Q-learning and Expected Sarsa.over Sarsa over a wide range of values for the step-size parameter↵. In cli↵ walkingthe state transitions are all deterministic and all randomness comes from the policy. Insuch cases, Expected Sarsa can safely set↵= 1 without su↵ering any degradation ofasymptotic performance, whereas Sarsa can only perform well in the long run at a smallvalue of↵, at which short-term performance is poor. In this and other examples there isa consistent empirical advantage of Expected Sarsa over Sarsa.In these cli↵ walking results Expected Sarsa was used on-policy, but in general itmight use a policy di↵erent from the target policy⇡to generate behavior, in which caseit becomes an o↵-policy algorithm. For example, suppose⇡is the greedy policy whilebehavior is more exploratory; then Expected Sarsa is exactly Q-learning. In this senseExpected Sarsa subsumes and generalizes Q-learning while reliably

is poor. In this and other examples there isa consistent empirical advantage of Expected Sarsa over Sarsa.In these cli↵ walking results Expected Sarsa was used on-policy, but in general itmight use a policy di↵erent from the target policy⇡to generate behavior, in which caseit becomes an o↵-policy algorithm. For example, suppose⇡is the greedy policy whilebehavior is more exploratory; then Expected Sarsa is exactly Q-learning. In this senseExpected Sarsa subsumes and generalizes Q-learning while reliably improving over Sarsa.Except for the small additional computational cost, Expected Sarsa may completelydominate both of the other more-well-known TD control algorithms.6.7 Maximization Bias and Double LearningAll the control algorithms that we have discussed so far involve maximization in theconstruction of their target policies. For example, in Q-learning the target policy isthe greedy policy given the current action values, which is deﬁned with a max, and inSarsa the policy is often"-greedy, which also

improving over Sarsa.Except for the small additional computational cost, Expected Sarsa may completelydominate both of the other more-well-known TD control algorithms.6.7 Maximization Bias and Double LearningAll the control algorithms that we have discussed so far involve maximization in theconstruction of their target policies. For example, in Q-learning the target policy isthe greedy policy given the current action values, which is deﬁned with a max, and inSarsa the policy is often"-greedy, which also involves a maximization operation. In thesealgorithms, a maximum over estimated values is used implicitly as an estimate of themaximum value, which can lead to a signiﬁcant positive bias. To see why, consider asingle stateswhere there are many actionsawhose true values,q(s, a), are all zero butwhose estimated values,Q(s, a), are uncertain and thus distributed some above and somebelow zero. The maximum of the true values is zero, but the maximum of the estimatesis positive, a positive bias. We call

also involves a maximization operation. In thesealgorithms, a maximum over estimated values is used implicitly as an estimate of themaximum value, which can lead to a signiﬁcant positive bias. To see why, consider asingle stateswhere there are many actionsawhose true values,q(s, a), are all zero butwhose estimated values,Q(s, a), are uncertain and thus distributed some above and somebelow zero. The maximum of the true values is zero, but the maximum of the estimatesis positive, a positive bias. We call thismaximization bias.Example 6.7: Maximization Bias ExampleThe small MDP shown inset inFigure 6.5 provides a simple example of how maximization bias can harm the performanceof TD control algorithms. The MDP has two non-terminal statesAandB.E p i s o d e salways start inAwith a choice between two actions,leftandright.T h erightactiontransitions immediately to the terminal state with a reward and return of zero. Theleftaction transitions toB, also with a reward of zero, from which there are manypossible actions

bias.Example 6.7: Maximization Bias ExampleThe small MDP shown inset inFigure 6.5 provides a simple example of how maximization bias can harm the performanceof TD control algorithms. The MDP has two non-terminal statesAandB.E p i s o d e salways start inAwith a choice between two actions,leftandright.T h erightactiontransitions immediately to the terminal state with a reward and return of zero. Theleftaction transitions toB, also with a reward of zero, from which there are manypossible actions all of which cause immediate termination with a reward drawn from anormal distribution with mean0.1 and variance 1.0. Thus, the expected return forany trajectory starting withleftis0.1, and thus takingleftin stateAis always a

6.7. Maximization Bias and Double Learning 135
BArightleft0. . .N(0.1,1)0Q-learningDoubleQ-learningEpisodes1001200300% leftactionsfrom A100%75%50%25%5%0optimalFigure 6.5: Comparison of Q-learning and Double Q-learning on a simple episodic MDP (shown
inset). Q-learning initially learns to take the left action much more often than the right action,
and always takes it signiﬁcantly more often than the 5% minimum probability enforced by
"-greedy action selection with "=0.1. In contrast, Double Q-learning is essentially una↵ected by
maximization bias. These data are averaged over 10,000 runs. The initial action-value estimates
were zero. Any ties in "-greedy action selection were broken randomly.
mistake. Nevertheless, our control methods may favor leftbecause of maximization bias
making Bappear to have a positive value. Figure 6.5 shows that Q-learning with "-greedy
action selection initially learns to strongly favor the leftaction on this example. Even at

asymptote, Q-learning takes the leftaction about 5% more often than is optimal at our
parameter settings ( "=0.1,↵=0.1, and = 1).
Are there algorithms that avoid maximization bias? To start, consider a bandit case in
which we have noisy estimates of the value of each of many actions, obtained as sample
averages of the rewards received on all the plays with each action. As we discussed above,
there will be a positive maximization bias if we use the maximum of the estimates as
an estimate of the maximum of the true values. One way to view the problem is that

it is due to using the same samples (plays) both to determine the maximizing action
and to estimate its value. Suppose we divided the plays in two sets and used them to
learn two independent estimates, call them Q1(a) and Q2(a), each an estimate of the
true value q(a), for all a2A. We could then use one estimate, say Q1,t od e t e r m i n e
the maximizing action A⇤=argmaxaQ1(a), and the other, Q2, to provide the estimate
of its value, Q2(A⇤)=Q2(argmaxaQ1(a)). This estimate will then be unbiased in the

sense that E[Q2(A⇤)]=q(A⇤). We can also repeat the process with the role of the two
estimates reversed to yield a second unbiased estimate Q1(argmaxaQ2(a)). This is the
idea of double learning . Note that although we learn two estimates, only one estimate is
updated on each play; double learning doubles the memory requirements, but does not
increase the amount of computation per step.
The idea of double learning extends naturally to algorithms for full MDPs. For example,

the double learning algorithm analogous to Q-learning, called Double Q-learning, divides
the time steps in two, perhaps by ﬂipping a coin on each step. If the coin comes up heads,

136 Chapter 6: Temporal-Di↵erence Learning
the update is
Q1(St,At) Q1(St,At)+↵h
Rt+1+Q2
St+1,argmax
aQ1(St+1,a)
Q1(St,At)i
.(6.10)
If the coin comes up tails, then the same update is done with Q1and Q2switched,
so that Q2is updated. The two approximate value functions are treated completely
symmetrically. The behavior policy can use both action-value estimates. For example, an
"-greedy policy for Double Q-learning could be based on the average (or sum) of the two
action-value estimates. A complete algorithm for Double Q-learning is given in the box
below. This is the algorithm used to produce the results in Figure 6.5. In that example,
double learning seems to eliminate the harm caused by maximization bias. Of course
there are also double versions of Sarsa and Expected Sarsa.
Double Q-learning, for estimating Q1⇡Q2⇡q⇤
Initialize Q1(s, a) and Q2(s, a), for all s2S+,a2A(s),such that Q(terminal ,·)=0

example, our general approach involves learning an action -value function, but in Chapter 1
we presented a TD method for learning to play tic-tac-toe that learned something much
more like a state -value function. If we look closely at that example, it becomes apparent
that the function learned there is neither an action-value function nor a state-value
function in the usual sense. A conventional state-value function evaluates states in which
the agent has the option of selecting an action, but the state-value function used in

6.8. Games, Afterstates, and Other Special Cases 137
tic-tac-toe evaluates board positions after the agent has made its move. Let us call these
afterstates , and value functions over these, afterstate value functions . Afterstates are
useful when we have knowledge of an initial part of the environment’s dynamics but not
necessarily of the full dynamics. For example, in games we typically know the immediate
e↵ects of our moves. We know for each possible chess move what the resulting position
will be, but not how our opponent will reply. Afterstate value functions are a natural
way to take advantage of this kind of knowledge and thereby produce a more ecient
learning method.
The reason it is more ecient to design algorithms in terms of afterstates is apparent
from the tic-tac-toe example. A conventional action-value function would map from
positions andmoves to an estimate of the value. But many position–move pairs produce
the same resulting position, as in the example below:

XOXXO+XO+XXIn such cases the position–move pairs are di↵erent but produce the same “afterposition,”
and thus must have the same value. A conventional action-value function would have to
separately assess both pairs, whereas an afterstate value function would immediately assess
both equally. Any learning about the position–move pair on the left would immediately
transfer to the pair on the right.
Afterstates arise in many tasks, not just games. For example, in queuing tasks there

are actions such as assigning customers to servers, rejecting customers, or discarding
information. In such cases the actions are in fact deﬁned in terms of their immediate
e↵ects, which are completely known.
It is impossible to describe all the possible kinds of specialized problems and corre-
sponding specialized learning algorithms. However, the principles developed in this book
should apply widely. For example, afterstate methods are still aptly described in terms

of generalized policy iteration, with a policy and (afterstate) value function interacting in
essentially the same way. In many cases one will still face the choice between on-policy
and o↵-policy methods for managing the need for persistent exploration.
Exercise 6.14 Describe how the task of Jack’s Car Rental (Example 4.2) could be
reformulated in terms of afterstates. Why, in terms of this speciﬁc task, would such a
reformulation be likely to speed convergence? ⇤

138 Chapter 6: Temporal-Di↵erence Learning
In this chapter we introduced a new kind of learning method, temporal-di↵erence (TD)
learning, and showed how it can be applied to the reinforcement learning problem. As
usual, we divided the overall problem into a prediction problem and a control problem.
TD methods are alternatives to Monte Carlo methods for solving the prediction problem.
In both cases, the extension to the control problem is via the idea of generalized policy
iteration (GPI) that we abstracted from dynamic programming. This is the idea that
approximate policy and value functions should interact in such a way that they both
move toward their optimal values.
One of the two processes making up GPI drives the value function to accurately predict
returns for the current policy; this is the prediction problem. The other process drives
the policy to improve locally (e.g., to be "-greedy) with respect to the current value

function. When the ﬁrst process is based on experience, a complication arises concerning
maintaining sucient exploration. We can classify TD control methods according to
whether they deal with this complication by using an on-policy or o↵-policy approach.
Sarsa is an on-policy method, and Q-learning is an o↵-policy method. Expected Sarsa
is also an o↵-policy method as we present it here. There is a third way in which TD
methods can be extended to control which we did not include in this chapter, called

actor–critic methods. These methods are covered in full in Chapter 13.
The methods presented in this chapter are today the most widely used reinforcement
learning methods. This is probably due to their great simplicity: they can be applied
online, with a minimal amount of computation, to experience generated from interaction
with an environment; they can be expressed nearly completely by single equations that
can be implemented with small computer programs. In the next few chapters we extend

these algorithms, making them slightly more complicated and signiﬁcantly more powerful.
All the new algorithms will retain the essence of those introduced here: they will be able
to process experience online, with relatively little computation, and they will be driven
by TD errors. The special cases of TD methods introduced in the present chapter should
rightly be called one-step, tabular, model-free TD methods. In the next two chapters we
extend them to n-step forms (a link to Monte Carlo methods) and forms that include

a model of the environment (a link to planning and dynamic programming). Then, in
the second part of the book we extend them to various forms of function approximation
rather than tables (a link to deep learning and artiﬁcial neural networks).
Finally, in this chapter we have discussed TD methods entirely within the context of
reinforcement learning problems, but TD methods are actually more general than this.
They are general methods for learning to make long-term predictions about dynamical
systems. For example, TD methods may be relevant to predicting ﬁnancial data, life

spans, election outcomes, weather patterns, animal behavior, demands on power stations,
or customer purchases. It was only when TD methods were analyzed as pure prediction
methods, independent of their use in reinforcement learning, that their theoretical
properties ﬁrst came to be well understood. Even so, these other potential applications
of TD learning methods have not yet been extensively explored.

Bibliographical and Historical Remarks 139
As we outlined in Chapter 1, the idea of TD learning has its early roots in animal learning
psychology and artiﬁcial intelligence, most notably the work of Samuel (1959) and Klopf
(1972). Samuel’s work is described as a case study in Section 16.2. Also related to TD
learning are Holland’s (1975, 1976) early ideas about consistency among value predictions.
These inﬂuenced one of the authors (Barto), who was a graduate student from 1970 to
1975 at the University of Michigan, where Holland was teaching. Holland’s ideas led to
a number of TD-related systems, including the work of Booker (1982) and the bucket
brigade of Holland (1986), which is related to Sarsa as discussed below.
6.1–2 Most of the speciﬁc material from these sections is from Sutton (1988), includ-
ing the TD(0) algorithm, the random walk example, and the term “temporal-
di↵erence learning.” The characterization of the relationship to dynamic pro-

gramming and Monte Carlo methods was inﬂuenced by Watkins (1989), Werbos
(1987), and others. The use of backup diagrams was new to the ﬁrst edition of
this book.
Tabular TD(0) was proved to converge in the mean by Sutton (1988) and with
probability 1 by Dayan (1992), based on the work of Watkins and Dayan (1992).
These results were extended and strengthened by Jaakkola, Jordan, and Singh
(1994) and Tsitsiklis (1994) by using extensions of the powerful existing theory

this book.
of stochastic approximation. Other extensions and generalizations are covered in
6.3 The optimality of the TD algorithm under batch training was established by
Sutton (1988). Illuminating this result is Barnard’s (1993) derivation of the TD
algorithm as a combination of one step of an incremental method for learning a
model of the Markov chain and one step of a method for computing predictions
from the model. The term certainty equivalence is from the adaptive control
literature (e.g., Goodwin and Sin, 1984).

6.4 The Sarsa algorithm was introduced by Rummery and Niranjan (1994). They
explored it in conjunction with artiﬁcial neural networks and called it “Modiﬁed
Connectionist Q-learning”. The name “Sarsa” was introduced by Sutton (1996).
The convergence of one-step tabular Sarsa (the form treated in this chapter) has
been proved by Singh, Jaakkola, Littman, and Szepesv´ ari (2000). The “windy
gridworld” example was suggested by Tom Kalt.
Holland’s (1986) bucket brigade idea evolved into an algorithm closely related to

Sarsa. The original idea of the bucket brigade involved chains of rules triggering
each other; it focused on passing credit back from the current rule to the rules
that triggered it. Over time, the bucket brigade came to be more like TD learning
in passing credit back to any temporally preceding rule, not just to the ones
that triggered the current rule. The modern form of the bucket brigade, when
simpliﬁed in various natural ways, is nearly identical to one-step Sarsa, as detailed
by Wilson (1994).

140 Chapter 6: Temporal-Di↵erence Learning
6.5 Q-learning was introduced by Watkins (1989), whose outline of a convergence
proof was made rigorous by Watkins and Dayan (1992). More general convergence
results were proved by Jaakkola, Jordan, and Singh (1994) and Tsitsiklis (1994).
6.6 The Expected Sarsa algorithm was introduced by George John (1994), who
called it “ Q-learning” and stressed its advantages over Q-learning as an o↵-policy
algorithm. John’s work was not known to us when we presented Expected
Sarsa in the ﬁrst edition of this book as an exercise, or to van Seijen, van
Hasselt, Whiteson, and Weiring (2009) when they established Expected Sarsa’s
convergence properties and conditions under which it will outperform regular
Sarsa and Q-learning. Our Figure 6.3 is adapted from their results. Van Seijen
et al. deﬁned “Expected Sarsa” to be an on-policy method exclusively (as we
did in the ﬁrst edition), whereas now we use this name for the general algorithm

in which the target and behavior policies may di↵er. The general o↵-policy
view of Expected Sarsa was noted by van Hasselt (2011), who called it “General
Q-learning.”
6.7 Maximization bias and double learning were introduced and extensively investi-
gated by van Hasselt (2010, 2011). The example MDP in Figure 6.5 was adapted
from that in his Figure 4.1 (van Hasselt, 2011).
6.8 The notion of an afterstate is the same as that of a “post-decision state” (Van
Roy, Bertsekas, Lee, and Tsitsiklis, 1997; Powell, 2011).

Chapter 7
n-step Bootstrapping
In this chapter we unify the Monte Carlo (MC) methods and the one-step temporal-
di↵erence (TD) methods presented in the previous two chapters. Neither MC methods nor
one-step TD methods are always the best. In this chapter we present n-step TD methods
that generalize both methods so that one can shift from one to the other smoothly as
needed to meet the demands of a particular task. n-step methods span a spectrum with
MC methods at one end and one-step TD methods at the other. The best methods are
often intermediate between the two extremes.
Another way of looking at the beneﬁts of n-step methods is that they free you from
the tyranny of the time step. With one-step TD methods the same time step determines
how often the action can be changed and the time interval over which bootstrapping
is done. In many applications one wants to be able to update the action very fast to
take into account anything that has changed, but bootstrapping works best if it is over a

length of time in which a signiﬁcant and recognizable state change has occurred. With
one-step TD methods, these time intervals are the same, and so a compromise must be
made. n-step methods enable bootstrapping to occur over multiple steps, freeing us from
the tyranny of the single time step.
The idea of n-step methods is usually used as an introduction to the algorithmic
idea of eligibility traces (Chapter 12), which enable bootstrapping over multiple time
intervals simultaneously. Here we instead consider the n-step bootstrapping idea on its

own, postponing the treatment of eligibility-trace mechanisms until later. This allows us
to separate the issues better, dealing with as many of them as possible in the simpler
n-step setting.
As usual, we ﬁrst consider the prediction problem and then the control problem. That
is, we ﬁrst consider how n-step methods can help in predicting returns as a function of
state for a ﬁxed policy (i.e., in estimating v⇡). Then we extend the ideas to action values
and control methods.
141

142 Chapter 7: n-step Bootstrapping
7.1 n-step TD Prediction
What is the space of methods lying between Monte Carlo and TD methods? Consider
estimating v⇡from sample episodes generated using ⇡. Monte Carlo methods perform
an update for each state based on the entire sequence of observed rewards from that
state until the end of the episode. The update of one-step TD methods, on the other
hand, is based on just the one next reward, bootstrapping from the value of the state
one step later as a proxy for the remaining rewards. One kind of intermediate method,
then, would perform an update based on an intermediate number of rewards: more than
one, but less than all of them until termination. For example, a two-step update would
be based on the ﬁrst two rewards and the estimated value of the state two steps later.
Similarly, we could have three-step updates, four-step updates, and so on. Figure 7.1
shows the backup diagrams of the spectrum of n-step updates forv⇡, with the one-step

TD update on the left and the up-until-termination Monte Carlo update on the right.1-step TDand TD(0)2-step TD3-step TDn-step TD∞-step TDand Monte Carlo············
Figure 7.1: The backup diagrams of n-step methods. These methods form a spectrum ranging
from one-step TD methods to Monte Carlo methods.
The methods that use n-step updates are still TD methods because they still change
an earlier estimate based on how it di↵ers from a later estimate. Now the later estimate
is not one step later, but nsteps later. Methods in which the temporal di↵erence extends

over nsteps are called n-step TD methods . The TD methods introduced in the previous
chapter all used one-step updates, which is why we called them one-step TD methods.
More formally, consider the update of the estimated value of state Stas a result of the
state–reward sequence, St,Rt+1,St+1,Rt+2,...,R T,ST(omitting the actions). We know
that in Monte Carlo updates the estimate of v⇡(St) is updated in the direction of the

while the values of all other states remain unchanged: Vt+n(s)=Vt+n1(s), for all s6=St.
We call this algorithm n-step TD . Note that no changes at all are made during the ﬁrst
n1 steps of each episode. To make up for that, an equal number of additional updates
are made at the end of the episode, after termination and before starting the next episode.
Complete pseudocode is given in the box on the next page.
Exercise 7.1 In Chapter 6 we noted that the Monte Carlo error can be written as the
sum of TD errors (6.6) if the value estimates don’t change from step to step. Show that
then-step error used in (7.2) can also be written as a sum TD errors (again if the value
estimates don’t change) generalizing the earlier result. ⇤
Exercise 7.2 (programming) With an n-step method, the value estimates dochange from
step to step, so an algorithm that used the sum of TD errors (see previous exercise) in

the correct predictions under appropriate technical conditions. The n-step TD methods
thus form a family of sound methods, with one-step TD methods and Monte Carlo
methods as extreme members.
Example 7.1: n-step TD Methods on the Random Walk Consider using n-step
TD methods on the 5-state random walk task described in Example 6.2 (page 125).
Suppose the ﬁrst episode progressed directly from the center state, C, to the right,
through DandE, and then terminated on the right with a return of 1. Recall that the
estimated values of all the states started at an intermediate value, V(s)=0 .5. As a result
of this experience, a one-step method would change only the estimate for the last state,

7.2. n-step Sarsa 145
V(E), which would be incremented toward 1, the observed return. A two-step method,
on the other hand, would increment the values of the two states preceding termination:
V(D) and V(E) both would be incremented toward 1. A three-step method, or any n-step
method for n>2, would increment the values of all three of the visited states toward 1,
all by the same amount.
Which value of nis better? Figure 7.2 shows the results of a simple empirical test for
a larger random walk process, with 19 states instead of 5 (and with a 1 outcome on the
left, all values initialized to 0), which we use as a running example in this chapter. Results
are shown for n-step TD methods with a range of values for nand↵. The performance
measure for each parameter setting, shown on the vertical axis, is the square-root of
the average squared error between the predictions at the end of the episode for the 19
states and their true values, then averaged over the ﬁrst 10 episodes and 100 repetitions

of the whole experiment (the same sets of walks were used for all parameter settings).
Note that methods with an intermediate value of nworked best. This illustrates how
the generalization of TD and Monte Carlo methods to n-step methods can potentially
perform better than either of the two extreme methods.
↵AverageRMS errorover 19 statesand ﬁrst 10 episodesn=1n=2n=4n=8n=16n=32n=32n=641285122560.550.50.450.350.30.250.40.40.200.80.61Figure 7.2: Performance of n-step TD methods as a function of ↵, for various values of n,o n
a 19-state random walk task (Example 7.1).

Exercise 7.3 Why do you think a larger random walk task (19 states instead of 5) was
used in the examples of this chapter? Would a smaller walk have shifted the advantage
to a di↵erent value of n? How about the change in left-side outcome from 0 to 1 made
in the larger walk? Do you think that made any di↵erence in the best value of n? ⇤
7.2 n-step Sarsa
How can n-step methods be used not just for prediction, but for control? In this section
we show how n-step methods can be combined with Sarsa in a straightforward way to

compared to one-step methods is given in Figure 7.4.1-step Sarsaaka Sarsa(0)2-step Sarsa3-step Sarsan-step Sarsa∞-step Sarsaaka Monte Carlon-step Expected SarsaFigure 7.3: The backup diagrams for the spectrum of n-step methods for state–action values.
They range from the one-step update of Sarsa(0) to the up-until-termination update of the
Monte Carlo method. In between are the n-step updates, based on nsteps of real rewards and
the estimated value of the nth next state–action pair, all appropriately discounted. On the far
right is the backup diagram for n-step Expected Sarsa.

Until ⌧=T1
Path takenAction values increasedby one-step SarsaAction values increased by 10-step SarsaGGGFigure 7.4: Gridworld example of the speedup of policy learning due to the use of n-step
methods. The ﬁrst panel shows the path taken by an agent in a single episode, ending at a
location of high reward, marked by the G. In this example the values were all initially 0, and all
rewards were zero except for a positive reward at G. The arrows in the other two panels show
which action values were strengthened as a result of this path by one-step and n-step Sarsa
methods. The one-step method strengthens only the last action of the sequence of actions that
led to the high reward, whereas the n-step method strengthens the last nactions of the sequence,
so that much more is learned from the one episode.

all action possibilities weighted, as always, by their probability under ⇡. This algorithm
can be described by the same equation as n-step Sarsa (above) except with the n-step
return redeﬁned as
Gt:t+n.=Rt+1+···+n1Rt+n+n¯Vt+n1(St+n),t +n<T , (7.7)
(with Gt:t+n.=Gtfort+nT)w h e r e ¯Vt(s)i st h e expected approximate value of state s,
using the estimated action values at time t, under the target policy:
¯Vt(s).=X
a⇡(a|s)Qt(s, a), for all s2S. (7.8)
Expected approximate values are used in developing many of the action-value methods
in the rest of this book. If sis terminal, then its expected approximate value is deﬁned
to be 0.
7.3 n-step O↵-policy Learning
Recall that o↵-policy learning is learning the value function for one policy, ⇡,w h i l e
following another policy, b.O f t e n , ⇡is the greedy policy for the current action-value-
function estimate, and bis a more exploratory policy, perhaps "-greedy. In order to
use the data from bwe must take into account the di↵erence between the two policies,

7.3. n-step O↵-policy Learning 149
For example, if any one of the actions would never be taken by ⇡(i.e., ⇡(Ak|Sk) = 0) then
then-step return should be given zero weight and be totally ignored. On the other hand,
if by chance an action is taken that ⇡would take with much greater probability than b
does, then this will increase the weight that would otherwise be given to the return. This
makes sense because that action is characteristic of ⇡(and therefore we want to learn
about it) but is selected only rarely by band thus rarely appears in the data. To make
up for this we have to over-weight it when it does occur. Note that if the two policies
are actually the same (the on-policy case) then the importance sampling ratio is always
1. Thus our new update (7.9) generalizes and can completely replace our earlier n-step
TD update. Similarly, our previous n-step Sarsa update can be completely replaced by a
simple o↵-policy form:
Qt+n(St,At).=Qt+n1(St,At)+↵⇢t+1:t+n[Gt:t+nQt+n1(St,At)], (7.11)

for 0 t<T . Note that the importance sampling ratio here starts and ends one step
later than for n-step TD (7.9) . This is because here we are updating a state–action
pair. We do not have to care how likely we were to select the action; now that we have
selected it we want to learn fully from what happens, with importance sampling only for
subsequent actions. Pseudocode for the full algorithm is shown in the box below.
O↵-policy n-step Sarsa for estimating Q⇡q⇤orq⇡
Input: an arbitrary behavior policy bsuch that b(a|s)>0, for all s2S,a2A

Initialize Q(s, a) arbitrarily, for all s2S,a2A
Initialize ⇡to be greedy with respect to Q,o ra saﬁ x e dg i v e np o l i c y
Algorithm parameters: step size ↵2(0,1], a positive integer n
All store and access operations (for St,At,a n d Rt)c a nt a k et h e i ri n d e xm o d n+1
Initialize and store S06= terminal
Select and store an action A0⇠b(·|S0)
T 1
Loop for t=0,1,2,...:
|Ift<T ,t h e n :
| Take action At
| Observe and store the next reward as Rt+1and the next state as St+1
| IfSt+1is terminal, then:
| T t+1
| else:

150 Chapter 7: n-step Bootstrapping
The o↵-policy version of n-step Expected Sarsa would use the same update as above
forn-step Sarsa except that the importance sampling ratio would have one less factor in
it. That is, the above equation would use ⇢t+1:t+n1instead of ⇢t+1:t+n, and of course
it would use the Expected Sarsa version of the n-step return (7.7) . This is because in
Expected Sarsa all possible actions are taken into account in the last state; the one
actually taken has no e↵ect and does not have to be corrected for.
7.4 *Per-decision Methods with Control Variates
The multi-step o↵-policy methods presented in the previous section are simple and
conceptually clear, but are probably not the most ecient. A more sophisticated approach
would use per-decision importance sampling ideas such as were introduced in Section 5.9.
To understand this approach, ﬁrst note that the ordinary n-step return (7.1) , like all
returns, can be written recursively. For the nsteps ending at horizon h,t h e n-step return

Gt:h=Rt+1+Gt+1:h,t < h < T , (7.12)
where Gh:h.=Vh1(Sh). (Recall that this return is used at time h, previously denoted
t+n.) Now consider the e↵ect of following a behavior policy bthat is not the same
as the target policy ⇡. All of the resulting experience, including the ﬁrst reward Rt+1
and the next state St+1must be weighted by the importance sampling ratio for time t,
⇢t=⇡(At|St)
b(At|St). One might be tempted to simply weight the righthand side of the above

⇡, so that ⇢tis zero. Then a simple weighting would result in the n-step return being
zero, which could result in high variance when it was used as a target. Instead, in this
more sophisticated approach, one uses an alternate, o↵-policy deﬁnition of the n-step
return ending at horizon h, as
Gt:h.=⇢t(Rt+1+Gt+1:h)+( 1 ⇢t)Vh1(St),t < h < T , (7.13)
where again Gh:h.=Vh1(Sh). In this approach, if ⇢tis zero, then instead of the target
being zero and causing the estimate to shrink, the target is the same as the estimate and
causes no change. The importance sampling ratio being zero means we should ignore the
sample, so leaving the estimate unchanged seems appropriate. The second, additional
term in (7.13) is called a control variate (for obscure reasons). Notice that the control
variate does not change the expected update; the importance sampling ratio has expected
value one (Section 5.9) and is uncorrelated with the estimate, so the expected value

of the control variate is zero. Also note that the o↵-policy deﬁnition (7.13) is a strict
generalization of the earlier on-policy deﬁnition of the n-step return (7.1) , as the two are
identical in the on-policy case, in which ⇢tis always 1.
For a conventional n-step method, the learning rule to use in conjunction with (7.13)
is the n-step TD update (7.2) , which has no explicit importance sampling ratios other
than those embedded in the return.
Exercise 7.5 Write the pseudocode for the o↵-policy state-value prediction algorithm
described above. ⇤

the recursion ends with and GT1:h.=RT. The resultant prediction algorithm (after
combining with (7.5)) is analogous to Expected Sarsa.
Exercise 7.6 Prove that the control variate in the above equations does not change the
expected value of the return. ⇤
⇤Exercise 7.7 Write the pseudocode for the o↵-policy action-value prediction algorithm
described immediately above. Pay particular attention to the termination conditions for
the recursion upon hitting the horizon or the end of episode. ⇤
Exercise 7.8 Show that the general (o↵-policy) version of the n-step return (7.13) can
still be written exactly and compactly as the sum of state-based TD errors (6.5) if the
approximate state value function does not change. ⇤
Exercise 7.9 Repeat the above exercise for the action version of the o↵-policy n-step
return (7.14) and the Expected Sarsa TD error (the quantity in brackets in Equation 6.9).
⇤
Exercise 7.10 (programming) Devise a small o↵-policy prediction problem and use it to

⇤
show that the o↵-policy learning algorithm using (7.13) and(7.2) is more data ecient
than the simpler algorithm using (7.1) and (7.9). ⇤
The importance sampling that we have used in this section, the previous section, and
in Chapter 5, enables sound o↵-policy learning, but also results in high variance updates,
forcing the use of a small step-size parameter and thereby causing learning to be slow. It
is probably inevitable that o↵-policy training is slower than on-policy training—after all,

the data is less relevant to what is being learned. However, it is probably also true that
these methods can be improved on. The control variates are one way of reducing the
variance. Another is to rapidly adapt the step sizes to the observed variance, as in the
Autostep method (Mahmood, Sutton, Degris and Pilarski, 2012). Yet another promising
approach is the invariant updates of Karampatziakis and Langford (2010) as extended
to TD by Tian (in preparation). The usage technique of Mahmood (2017; Mahmood

152 Chapter 7: n-step Bootstrapping
and Sutton, 2015) may also be part of the solution. In the next section we consider an
o↵-policy learning method that does not use importance sampling.
Then-step Tree Backup Algorithm
Is o↵-policy learning possible without importance sampling? Q-learning and Expected
Sarsa from Chapter 6 do this for the one-step case, but is there a corresponding multi-step
algorithm? In this section we present just such an n-step method, called the tree-backup
algorithm .St,AtAt+1Rt+1St+1St+2Rt+2At+2Rt+3St+3
the 3-step
tree-backup
updateThe idea of the algorithm is suggested by the 3-step tree-backup backup
diagram shown to the right. Down the central spine and labeled in the
diagram are three sample states and rewards, and two sample actions.
These are the random variables representing the events occurring after the
initial state–action pair St,At. Hanging o↵ to the sides of each state are

the 3-step
the actions that were notselected. (For the last state, all the actions are
considered to have not (yet) been selected.) Because we have no sample
data for the unselected actions, we bootstrap and use the estimates of
their values in forming the target for the update. This slightly extends the
idea of a backup diagram. So far we have always updated the estimated
value of the node at the top of the diagram toward a target combining
the rewards along the way (appropriately discounted) and the estimated
values of the nodes at the bottom. In the tree-backup update, the target

includes all these things plus the estimated values of the dangling action
nodes hanging o↵ the sides, at all levels. This is why it is called a tree-
backup update; it is an update from the entire tree of estimated action
values.
More precisely, the update is from the estimated action values of the
leaf nodes of the tree. The action nodes in the interior, corresponding to
the actual actions taken, do not participate. Each leaf node contributes to the target

values.
with a weight proportional to its probability of occurring under the target policy ⇡.T h u s
each ﬁrst-level action acontributes with a weight of ⇡(a|St+1), except that the action
actually taken, At+1, does not contribute at all. Its probability, ⇡(At+1|St+1), is used
to weight all the second-level action values. Thus, each non-selected second-level action
a0contributes with weight ⇡(At+1|St+1)⇡(a0|St+2). Each third-level action contributes
with weight ⇡(At+1|St+1)⇡(At+2|St+2)⇡(a00|St+3), and so on. It is as if each arrow to an

action node in the diagram is weighted by the action’s probability of being selected under
the target policy and, if there is a tree below the action, then that weight applies to all
the leaf nodes in the tree.

So far in this chapter we have considered three di↵erent kinds of action-value algorithms,
corresponding to the ﬁrst three backup diagrams shown in Figure 7.5. n-step Sarsa has
all sample transitions, the tree-backup algorithm has all state-to-action transitions fully
branched without sampling, and n-step Expected Sarsa has all sample transitions except
for the last state-to-action one, which is fully branched with an expected value. To what
extent can these algorithms be uniﬁed?
One idea for uniﬁcation is suggested by the fourth backup diagram in Figure 7.5. This
is the idea that one might decide on a step-by-step basis whether one wanted to take the
action as a sample, as in Sarsa, or consider the expectation over all actions instead, as in
the tree-backup update. Then, if one chose always to sample, one would obtain Sarsa,
whereas if one chose never to sample, one would get the tree-backup algorithm. Expected
Sarsa would be the case where one chose to sample for all steps except for the last one.

look ahead to the next nrewards, states, and actions. The two
4-step backup diagrams to the right together summarize most of the
methods introduced. The state-value update shown is for n-step
TD with importance sampling, and the action-value update is for
n-step Q(), which generalizes Expected Sarsa and Q-learning. All
n-step methods involve a delay of ntime steps before updating,
as only then are all the required future events known. A further
drawback is that they involve more computation per time step
than previous methods. Compared to one-step methods, n-step
methods also require more memory to record the states, actions,
rewards, and sometimes other variables over the last ntime steps.
Eventually, in Chapter 12, we will see how multi-step TD methods
can be implemented with minimal memory and computational
complexity using eligibility traces, but there will always be some
additional computation beyond one-step methods. Such costs can
be well worth paying to escape the tyranny of the single time step.

Although n-step methods are more complex than those using
eligibility traces, they have the great beneﬁt of being conceptually
clear. We have sought to take advantage of this by developing two
approaches to o↵-policy learning in the n-step case. One, based on
importance sampling is conceptually simple but can be of high variance. If the target and
behavior policies are very di↵erent it probably needs some new algorithmic ideas before

it can be ecient and practical. The other, based on tree-backup updates, is the natural
extension of Q-learning to the multi-step case with stochastic target policies. It involves
no importance sampling but, again if the target and behavior policies are substantially
di↵erent, the bootstrapping may span only a few steps even if nis large.

158 Chapter 7: n-step Bootstrapping
The notion of n-step returns is due to Watkins (1989), who also ﬁrst discussed their error
reduction property. n-step algorithms were explored in the ﬁrst edition of this book,
in which they were treated as of conceptual interest, but not feasible in practice. The
work of Cichosz (1995) and particularly van Seijen (2016) showed that they are actually
completely practical algorithms. Given this, and their conceptual clarity and simplicity,
we have chosen to highlight them here in the second edition. In particular, we now
postpone all discussion of the backward view and of eligibility traces until Chapter 12.
7.1–2 The results in the random walk examples were made for this text based on work
of Sutton (1988) and Singh and Sutton (1996). The use of backup diagrams to
describe these and other algorithms in this chapter is new.
7.3–5 The developments in these sections are based on the work of Precup, Sutton,

and Singh (2000), Precup, Sutton, and Dasgupta (2001), and Sutton, Mahmood,
Precup, and van Hasselt (2014).
The tree-backup algorithm is due to Precup, Sutton, and Singh (2000), but the
presentation of it here is new.
7.6 TheQ() algorithm is new to this text, but closely related algorithms have been
explored further by De Asis, Hernandez-Garcia, Holland, and Sutton (2017).

Chapter 8
Planning and Learning with
Tabular Methods
In this chapter we develop a uniﬁed view of reinforcement learning methods that require
a model of the environment, such as dynamic programming and heuristic search, and
methods that can be used without a model, such as Monte Carlo and temporal-di↵erence
methods. These are respectively called model-based andmodel-free reinforcement learning
methods. Model-based methods rely on planning as their primary component, while
model-free methods primarily rely on learning . Although there are real di↵erences between
these two kinds of methods, there are also great similarities. In particular, the heart of
both kinds of methods is the computation of value functions. Moreover, all the methods
are based on looking ahead to future events, computing a backed-up value, and then
using it as an update target for an approximate value function. Earlier in this book we
presented Monte Carlo and temporal-di↵erence methods as distinct alternatives, then

showed how they can be uniﬁed by n-step methods. Our goal in this chapter is a similar
integration of model-based and model-free methods. Having established these as distinct
in earlier chapters, we now explore the extent to which they can be intermixed.
8.1 Models and Planning
By a model of the environment we mean anything that an agent can use to predict how the
environment will respond to its actions. Given a state and an action, a model produces a
prediction of the resultant next state and next reward. If the model is stochastic, then

there are several possible next states and next rewards, each with some probability of
occurring. Some models produce a description of all possibilities and their probabilities;
these we call distribution models . Other models produce just one of the possibilities,
sampled according to the probabilities; these we call sample models . For example, consider
modeling the sum of a dozen dice. A distribution model would produce all possible sums
and their probabilities of occurring, whereas a sample model would produce an individual
159

160 Chapter 8: Planning and Learning with Tabular Methods
sum drawn according to this probability distribution. The kind of model assumed in
dynamic programming—estimates of the MDP’s dynamics, p(s0,r|s, a)—is a distribution
model. The kind of model used in the blackjack example in Chapter 5 is a sample model.
Distribution models are stronger than sample models in that they can always be used
to produce samples. However, in many applications it is much easier to obtain sample
models than distribution models. The dozen dice are a simple example of this. It would
be easy to write a computer program to simulate the dice rolls and return the sum, but
harder and more error-prone to ﬁgure out all the possible sums and their probabilities.
Models can be used to mimic or simulate experience. Given a starting state and action,
a sample model produces a possible transition, and a distribution model generates all
possible transitions weighted by their probabilities of occurring. Given a starting state

and a policy, a sample model could produce an entire episode, and a distribution model
could generate all possible episodes and their probabilities. In either case, we say the
model is used to simulate the environment and produce simulated experience .
The word planning is used in several di↵erent ways in di↵erent ﬁelds. We use the
term to refer to any computational process that takes a model as input and produces or
improves a policy for interacting with the modeled environment:

planningmodelpolicyIn artiﬁcial intelligence, there are two distinct approaches to planning according to our
deﬁnition. State-space planning , which includes the approach we take in this book,
is viewed primarily as a search through the state space for an optimal policy or an
optimal path to a goal. Actions cause transitions from state to state, and value functions
are computed over states. In what we call plan-space planning , planning is instead a
search through the space of plans. Operators transform one plan into another, and

value functions, if any, are deﬁned over the space of plans. Plan-space planning includes
evolutionary methods and “partial-order planning,” a common kind of planning in artiﬁcial
intelligence in which the ordering of steps is not completely determined at all stages of
planning. Plan-space methods are dicult to apply eciently to the stochastic sequential
decision problems that are the focus in reinforcement learning, and we do not consider
them further (but see, e.g., Russell and Norvig, 2010).
The uniﬁed view we present in this chapter is that all state-space planning methods

share a common structure, a structure that is also present in the learning methods
presented in this book. It takes the rest of the chapter to develop this view, but there are
two basic ideas: (1) all state-space planning methods involve computing value functions
as a key intermediate step toward improving the policy, and (2) they compute value
functions by updates or backup operations applied to simulated experience. This common

structure can be diagrammed as follows:valuesbackupsmodelsimulatedexperiencepolicyupdatesbackupsDynamic programming methods clearly ﬁt this structure: they make sweeps through the
space of states, generating for each state the distribution of possible transitions. Each
distribution is then used to compute a backed-up value (update target) and update the

8.2. Dyna: Integrated Planning, Acting, and Learning 161
state’s estimated value. In this chapter we argue that various other state-space planning
methods also ﬁt this structure, with individual methods di↵ering only in the kinds of
updates they do, the order in which they do them, and in how long the backed-up
information is retained.
Viewing planning methods in this way emphasizes their relationship to the learning
methods that we have described in this book. The heart of both learning and planning
methods is the estimation of value functions by backing-up update operations. The
di↵erence is that whereas planning uses simulated experience generated by a model,
learning methods use real experience generated by the environment. Of course this
di↵erence leads to a number of other di↵erences, for example, in how performance is
assessed and in how ﬂexibly experience can be generated. But the common structure
means that many ideas and algorithms can be transferred between planning and learning.

In particular, in many cases a learning algorithm can be substituted for the key update
step of a planning method. Learning methods require only experience as input, and in
many cases they can be applied to simulated experience just as well as to real experience.
The box below shows a simple example of a planning method based on one-step tabular
Q-learning and on random samples from a sample model. This method, which we call
random-sample one-step tabular Q-planning , converges to the optimal policy for the model

under the same conditions that one-step tabular Q-learning converges to the optimal
policy for the real environment (each state–action pair must be selected an inﬁnite number
of times in Step 1, and ↵must decrease appropriately over time).
Random-sample one-step tabular Q-planning
1. Select a state, S2S, and an action, A2A(S), at random
2. Send S, Ato a sample model, and obtain
a sample next reward, R, and a sample next state, S0
3. Apply one-step tabular Q-learning to S, A, R, S0:

In addition to the uniﬁed view of planning and learning methods, a second theme in
this chapter is the beneﬁts of planning in small, incremental steps. This enables planning
to be interrupted or redirected at any time with little wasted computation, which appears
to be a key requirement for eciently intermixing planning with acting and with learning
of the model. Planning in very small steps may be the most ecient approach even on
pure planning problems if the problem is too large to be solved exactly.
8.2 Dyna: Integrated Planning, Acting, and Learning

When planning is done online, while interacting with the environment, a number of
interesting issues arise. New information gained from the interaction may change the
model and thereby interact with planning. It may be desirable to customize the planning
process in some way to the states or decisions currently under consideration, or expected

162 Chapter 8: Planning and Learning with Tabular Methods
in the near future. If decision making and model learning are both computation-intensive
processes, then the available computational resources may need to be divided between
them. To begin exploring these issues, in this section we present Dyna-Q, a simple
architecture integrating the major functions needed in an online planning agent. Each
function appears in Dyna-Q in a simple, almost trivial, form. In subsequent sections we
elaborate some of the alternate ways of achieving each function and the trade-o↵s between
them. For now, we seek merely to illustrate the ideas and stimulate your intuition.
Within a planning agent, there are at least two roles for real experience: it can be
used to improve the model (to make it more accurately match the real environment)
and it can be used to directly improve the value function and policy using the kinds of

planningvalue/policyexperiencemodelmodellearningactingdirectRLreinforcement learning methods we have discussed
in previous chapters. The former we call model-
learning , and the latter we call direct reinforcement
learning (direct RL). The possible relationships
between experience, model, values, and policy are
summarized in the diagram to the right. Each ar-
row shows a relationship of inﬂuence and presumed
improvement. Note how experience can improve
value functions and policies either directly or in-
directly via the model. It is the latter, which is

sometimes called indirect reinforcement learning ,
that is involved in planning.
Both direct and indirect methods have advantages and disadvantages. Indirect methods
often make fuller use of a limited amount of experience and thus achieve a better policy
with fewer environmental interactions. On the other hand, direct methods are much
simpler and are not a↵ected by biases in the design of the model. Some have argued
that indirect methods are always superior to direct ones, while others have argued that

direct methods are responsible for most human and animal learning. Related debates
in psychology and artiﬁcial intelligence concern the relative importance of cognition as
opposed to trial-and-error learning, and of deliberative planning as opposed to reactive
decision making (see Chapter 14 for discussion of some of these issues from the perspective
of psychology). Our view is that the contrast between the alternatives in all these debates

has been exaggerated, that more insight can be gained by recognizing the similarities
between these two sides than by opposing them. For example, in this book we have
emphasized the deep similarities between dynamic programming and temporal-di↵erence
methods, even though one was designed for planning and the other for model-free learning.
Dyna-Q includes all of the processes shown in the diagram above—planning, acting,
model-learning, and direct RL—all occurring continually. The planning method is the

random-sample one-step tabular Q-planning method on page 161. The direct RL method
is one-step tabular Q-learning. The model-learning method is also table-based and assumes
the environment is deterministic. After each transition St,At!Rt+1,St+1,t h em o d e l
records in its table entry for St,Atthe prediction that Rt+1,St+1will deterministically
follow. Thus, if the model is queried with a state–action pair that has been experienced

before, it simply returns the last-observed next state and next reward as its prediction.

8.2. Dyna: Integrated Planning, Acting, and Learning 163
During planning, the Q-planning algorithm randomly samples only from state–action
pairs that have previously been experienced (in Step 1), so the model is never queried
with a pair about which it has no information.
The overall architecture of Dyna agents, of which the Dyna-Q algorithm is one example,
is shown in Figure 8.1. The central column represents the basic interaction between
agent and environment, giving rise to a trajectory of real experience. The arrow on the
left of the ﬁgure represents direct reinforcement learning operating on real experience to
improve the value function and the policy. On the right are model-based processes. The
model is learned from real experience and gives rise to simulated experience. We use the
term search control to refer to the process that selects the starting states and actions
for the simulated experiences generated by the model. Finally, planning is achieved by

applying reinforcement learning methods to the simulated experiences just as if they had
really happened. Typically, as in Dyna-Q, the same reinforcement learning method is
used both for learning from real experience and for planning from simulated experience.
The reinforcement learning method is thus the “ﬁnal common path” for both learning
and planning. Learning and planning are deeply integrated in the sense that they share
almost all the same machinery, di↵ering only in the source of their experience.

realdirect RLupdateModelplanning updatesearchcontrolPolicy/value functionsexperiencemodellearningEnvironmentsimulatedexperience
Figure 8.1: The general Dyna Architecture. Real experience, passing back and forth between
the environment and the policy, a↵ects policy and value functions in much the same way as does
simulated experience generated by the model of the environment.
Conceptually, planning, acting, model-learning, and direct RL occur simultaneously

and in parallel in Dyna agents. For concreteness and implementation on a serial computer,
however, we fully specify the order in which they occur within a time step. In Dyna-Q,
the acting, model-learning, and direct RL processes require little computation, and we
assume they consume just a fraction of the time. The remaining time in each step can be
devoted to the planning process, which is inherently computation-intensive. Let us assume
that there is time in each step, after acting, model-learning, and direct RL, to complete

164 Chapter 8: Planning and Learning with Tabular Methods
niterations (Steps 1–3) of the Q-planning algorithm. In the pseudocode algorithm for
Dyna-Q in the box below, Model (s, a) denotes the contents of the (predicted next state
and reward) for state–action pair ( s, a). Direct reinforcement learning, model-learning,
and planning are implemented by steps (d), (e), and (f), respectively. If (e) and (f) were
omitted, the remaining algorithm would be one-step tabular Q-learning.
Tabular Dyna-Q
Initialize Q(s, a) and Model (s, a) for all s2Sanda2A(s)
(a)S current (nonterminal) state
(b)A "-greedy( S, Q)
(c) Take action A; observe resultant reward, R, and state, S0
(d)Q(S, A) Q(S, A)+↵⇥
(e)Model (S, A) R, S0(assuming deterministic environment)
(f) Loop repeat ntimes:
S random previously observed state
A random action previously taken in S
R, S0 Model (S, A)

Example 8.1: Dyna Maze Consider the simple maze shown inset in Figure 8.2. In
each of the 47 states there are four actions, up,down ,right , and left , which take the
agent deterministically to the corresponding neighboring states, except when movement
is blocked by an obstacle or the edge of the maze, in which case the agent remains where
it is. Reward is zero on all transitions, except those into the goal state, on which it is +1.
After reaching the goal state ( G), the agent returns to the start state ( S) to begin a new

episode. This is a discounted, episodic task with =0.95.
The main part of Figure 8.2 shows average learning curves from an experiment in
which Dyna-Q agents were applied to the maze task. The initial action values were zero,
the step-size parameter was ↵=0.1, and the exploration parameter was "=0.1. When
selecting greedily among actions, ties were broken randomly. The agents varied in the
number of planning steps, n, they performed per real step. For each n,t h ec u r v e ss h o w

the number of steps taken by the agent to reach the goal in each episode, averaged over 30
repetitions of the experiment. In each repetition, the initial seed for the random number
generator was held constant across algorithms. Because of this, the ﬁrst episode was
exactly the same (about 1700 steps) for all values of n, and its data are not shown in
the ﬁgure. After the ﬁrst episode, performance improved for all values of n,b u tm u c h
more rapidly for larger values. Recall that the n= 0 agent is a nonplanning agent, using

only direct reinforcement learning (one-step tabular Q-learning). This was by far the
slowest agent on this problem, despite the fact that the parameter values ( ↵and")w e r e
optimized for it. The nonplanning agent took about 25 episodes to reach ( "-)optimal
performance, whereas the n= 5 agent took about ﬁve episodes, and the n= 50 agent
took only three episodes.

8.2. Dyna: Integrated Planning, Acting, and Learning 165
2800600400200142010304050  0planningsteps(direct RL only)
EpisodesStepsperepisode5 planning steps  50planningstepsSGactionsFigure 8.2: A simple maze (inset) and the average learning curves for Dyna-Q agents varying
in their number of planning steps ( n) per real step. The task is to travel from StoGas quickly
Figure 8.3 shows why the planning agents found the solution so much faster than
the nonplanning agent. Shown are the policies found by the n= 0 and n= 50 agents
halfway through the second episode. Without planning ( n= 0), each episode adds only
one additional step to the policy, and so only one step (the last) has been learned so far.
With planning, again only one step is learned during the ﬁrst episode, but here during
the second episode an extensive policy has been developed that by the end of the episode
will reach almost back to the start state. This policy is built by the planning process

while the agent is still wandering near the start state. By the end of the third episode a
complete optimal policy will have been found and perfect performance attained.SGSGWITHOUT PLANNING (=0)WITH PLANNING (=50)nnFigure 8.3: Policies found by planning and nonplanning Dyna-Q agents halfway through the
second episode. The arrows indicate the greedy action in each state; if no arrow is shown for a
state, then all of its action values were equal. The black square indicates the location of the
agent.

166 Chapter 8: Planning and Learning with Tabular Methods
In Dyna-Q, learning and planning are accomplished by exactly the same algorithm,
operating on real experience for learning and on simulated experience for planning.
Because planning proceeds incrementally, it is trivial to intermix planning and acting.
Both proceed as fast as they can. The agent is always reactive and always deliberative,
responding instantly to the latest sensory information and yet always planning in the
background. Also ongoing in the background is the model-learning process. As new
information is gained, the model is updated to better match reality. As the model changes,
the ongoing planning process will gradually compute a di↵erent way of behaving to match
the new model.
Exercise 8.1 The nonplanning method looks particularly poor in Figure 8.3 because it is
a one-step method; a method using multi-step bootstrapping would do better. Do you
think one of the multi-step bootstrapping methods from Chapter 7 could do as well as

the Dyna method? Explain why or why not. ⇤
8.3 When the Model Is Wrong
In the maze example presented in the previous section, the changes in the model were
relatively modest. The model started out empty, and was then ﬁlled only with exactly
correct information. In general, we cannot expect to be so fortunate. Models may be
incorrect because the environment is stochastic and only a limited number of samples
have been observed, or because the model was learned using function approximation that

has generalized imperfectly, or simply because the environment has changed and its new
behavior has not yet been observed. When the model is incorrect, the planning process is
likely to compute a suboptimal policy.
In some cases, the suboptimal policy computed by planning quickly leads to the
discovery and correction of the modeling error. This tends to happen when the model
is optimistic in the sense of predicting greater reward or better state transitions than

are actually possible. The planned policy attempts to exploit these opportunities and in
doing so discovers that they do not exist.
Example 8.2: Blocking Maze A maze example illustrating this relatively minor
kind of modeling error and recovery from it is shown in Figure 8.4. Initially, there is a
short path from start to goal, to the right of the barrier, as shown in the upper left of the
ﬁgure. After 1000 time steps, the short path is “blocked,” and a longer path is opened up

along the left-hand side of the barrier, as shown in upper right of the ﬁgure. The graph
shows average cumulative reward for a Dyna-Q agent and an enhanced Dyna-Q+ agent
to be described shortly. The ﬁrst part of the graph shows that both Dyna agents found
the short path within 1000 steps. When the environment changed, the graphs become
ﬂat, indicating a period during which the agents obtained no reward because they were
wandering around behind the barrier. After a while, however, they were able to ﬁnd the

new opening and the new optimal behavior.
Greater diculties arise when the environment changes to become better than it was
before, and yet the formerly correct policy does not reveal the improvement. In these
cases the modeling error may not be detected for a long time, if ever.

8.3. When the Model Is Wrong 167
Cumulativereward0100020003000Time steps150
0Dyna-Q+SGGS
Dyna-QFigure 8.4: Average performance of Dyna agents on a blocking task. The left environment
was used for the ﬁrst 1000 steps, the right environment for the rest. Dyna-Q+ is Dyna-Q with
an exploration bonus that encourages exploration.
CumulativerewardSGGS
030006000Time steps400
0Dyna-Q+Dyna-QFigure 8.5: Average performance of Dyna agents on
a shortcut task. The left environment was used for the
ﬁrst 3000 steps, the right environment for the rest.Example 8.3: Shortcut Maze
The problem caused by this kind of
environmental change is illustrated
by the maze example shown in Fig-
ure 8.5. Initially, the optimal path is
to go around the left side of the bar-
rier (upper left). After 3000 steps,
however, a shorter path is opened up
along the right side, without disturb-
ing the longer path (upper right).
The graph shows that the regular
Dyna-Q agent never switched to the
shortcut. In fact, it never realized

that it existed. Its model said that
there was no shortcut, so the more it
planned, the less likely it was to step
to the right and discover it. Even
with an "-greedy policy, it is very
unlikely that an agent will take so
many exploratory actions as to dis-
cover the shortcut.
The general problem here is another version of the conﬂict between exploration and
exploitation. In a planning context, exploration means trying actions that improve the
model, whereas exploitation means behaving in the optimal way given the current model.

168 Chapter 8: Planning and Learning with Tabular Methods
We want the agent to explore to ﬁnd changes in the environment, but not so much that
performance is greatly degraded. As in the earlier exploration/exploitation conﬂict, there
probably is no solution that is both perfect and practical, but simple heuristics are often
e↵ective.
The Dyna-Q+ agent that did solve the shortcut maze uses one such heuristic. This
agent keeps track for each state–action pair of how many time steps have elapsed since
the pair was last tried in a real interaction with the environment. The more time that
has elapsed, the greater (we might presume) the chance that the dynamics of this pair
has changed and that the model of it is incorrect. To encourage behavior that tests
long-untried actions, a special “bonus reward” is given on simulated experiences involving
these actions. In particular, if the modeled reward for a transition is r, and the transition

has not been tried in ⌧time steps, then planning updates are done as if that transition
produced a reward of r+p⌧, for some small . This encourages the agent to keep
testing all accessible state transitions and even to ﬁnd long sequences of actions in order
to carry out such tests.1Of course all this testing has its cost, but in many cases, as in the
shortcut maze, this kind of computational curiosity is well worth the extra exploration.
Exercise 8.2 Why did the Dyna agent with exploration bonus, Dyna-Q+, perform

better in the ﬁrst phase as well as in the second phase of the blocking and shortcut
experiments? ⇤
Exercise 8.3 Careful inspection of Figure 8.5 reveals that the di↵erence between Dyna-Q+
and Dyna-Q narrowed slightly over the ﬁrst part of the experiment. What is the reason
for this? ⇤
Exercise 8.4 (programming) The exploration bonus described above actually changes
the estimated values of states and actions. Is this necessary? Suppose the bonus p⌧
was used not in updates, but solely in action selection. That is, suppose the action
selected was always that for which Q(St,a)+p

for this? ⇤
⌧(St,a)was maximal. Carry out a
gridworld experiment that tests and illustrates the strengths and weaknesses of this
alternate approach. ⇤
Exercise 8.5 How might the tabular Dyna-Q algorithm shown on page 164 be modiﬁed
to handle stochastic environments? How might this modiﬁcation perform poorly on
changing environments such as considered in this section? How could the algorithm be
modiﬁed to handle stochastic environments andchanging environments? ⇤
8.4 Prioritized Sweeping

In the Dyna agents presented in the preceding sections, simulated transitions are started in
state–action pairs selected uniformly at random from all previously experienced pairs. But
a uniform selection is usually not the best; planning can be much more ecient if simulated
transitions and updates are focused on particular state–action pairs. For example, consider
1The Dyna-Q+ agent was changed in two other ways as well. First, actions that had never been

tried before from a state were allowed to be considered in the planning step (f) of the Tabular Dyna-Q
algorithm in the box above. Second, the initial model for such actions was that they would lead back to
the same state with a reward of zero.

8.4. Prioritized Sweeping 169
what happens during the second episode of the ﬁrst maze task (Figure 8.3). At the
beginning of the second episode, only the state–action pair leading directly into the goal
has a positive value; the values of all other pairs are still zero. This means that it is
pointless to perform updates along almost all transitions, because they take the agent
from one zero-valued state to another, and thus the updates would have no e↵ect. Only
an update along a transition into the state just prior to the goal, or from it, will change
any values. If simulated transitions are generated uniformly, then many wasteful updates
will be made before stumbling onto one of these useful ones. As planning progresses, the
region of useful updates grows, but planning is still far less ecient than it would be if
focused where it would do the most good. In the much larger problems that are our real
objective, the number of states is so large that an unfocused search would be extremely
inecient.

inecient.
This example suggests that search might be usefully focused by working backward from
goal states. Of course, we do not really want to use any methods speciﬁc to the idea of
“goal state.” We want methods that work for general reward functions. Goal states are
just a special case, convenient for stimulating intuition. In general, we want to work back
not just from goal states but from any state whose value has changed. Suppose that the
values are initially correct given the model, as they were in the maze example prior to

discovering the goal. Suppose now that the agent discovers a change in the environment
and changes its estimated value of one state, either up or down. Typically, this will imply
that the values of many other states should also be changed, but the only useful one-step
updates are those of actions that lead directly into the one state whose value has been
changed. If the values of these actions are updated, then the values of the predecessor
states may change in turn. If so, then actions leading into them need to be updated, and

then their predecessor states may have changed. In this way one can work backward
from arbitrary states that have changed in value, either performing useful updates or
terminating the propagation. This general idea might be termed backward focusing of
planning computations.
As the frontier of useful updates propagates backward, it often grows rapidly, producing
many state–action pairs that could usefully be updated. But not all of these will be
equally useful. The values of some states may have changed a lot, whereas others may

have changed little. The predecessor pairs of those that have changed a lot are more
likely to also change a lot. In a stochastic environment, variations in estimated transition
probabilities also contribute to variations in the sizes of changes and in the urgency with
which pairs need to be updated. It is natural to prioritize the updates according to a
measure of their urgency, and perform them in order of priority. This is the idea behind
prioritized sweeping . A queue is maintained of every state–action pair whose estimated

value would change nontrivially if updated , prioritized by the size of the change. When
the top pair in the queue is updated, the e↵ect on each of its predecessor pairs is computed.
If the e↵ect is greater than some small threshold, then the pair is inserted in the queue
with the new priority (if there is a previous entry of the pair in the queue, then insertion
results in only the higher priority entry remaining in the queue). In this way the e↵ects
of changes are eciently propagated backward until quiescence. The full algorithm for

the case of deterministic environments is given in the box on the next page.

10204794186376752150430086016Gridworld size (#states)Dyna-QPrioritizedsweepingUpdatesUpdatesuntiloptimalsolutionExample 8.4: Prioritized Sweepingon MazesPrioritized sweeping has beenfound to dramatically increase the speedat which optimal solutions are found inmaze tasks, often by a factor of 5 to 10.A typical example is shown to the right.These data are for a sequence of mazetasks of exactly the same structure as theone shown in Figure 8.2, except that theyvary in the grid resolution. Prioritizedsweeping maintained a decisive advantageover unprioritized Dyna-Q. Both systemsmade at mostn= 5 updates per environ-mental interaction. Adapted from Pengand Williams (1993).Extensions of prioritized sweeping to stochastic environments are straightforward. Themodel is maintained by keeping counts of the number of times each state–action pair hasbeen experienced and of what the next states were. It is natural then to update each pairnot with a sample update, as we have been using so far, but with an expected

maintained a decisive advantageover unprioritized Dyna-Q. Both systemsmade at mostn= 5 updates per environ-mental interaction. Adapted from Pengand Williams (1993).Extensions of prioritized sweeping to stochastic environments are straightforward. Themodel is maintained by keeping counts of the number of times each state–action pair hasbeen experienced and of what the next states were. It is natural then to update each pairnot with a sample update, as we have been using so far, but with an expected update,taking into account all possible next states and their probabilities of occurring.Prioritized sweeping is just one way of distributing computations to improve planningeciency, and probably not the best way. One of prioritized sweeping’s limitations is thatit usesexpectedupdates, which in stochastic environments may waste lots of computationon low-probability transitions. As we show in the following section, sample updates

8.4. Prioritized Sweeping 171
Example 8.5 Prioritized Sweeping for Rod Maneuvering
StartGoal

StartGoal
The objective in this task is tomaneuver a rod around some awk-wardly placed obstacles within alimited rectangular work space to agoal position in the fewest numberof steps. The rod can be translatedalong its long axis or perpendicu-lar to that axis, or it can be ro-tated in either direction around itscenter. The distance of each move-ment is approximately 1/20 of thework space, and the rotation incre-ment is 10 degrees. Translationsare deterministic and quantized toone of 20⇥20 positions. To theright is shown the obstacles and theshortest solution from start to goal,found by prioritized sweeping. This problem is deterministic, but has four actionsand 14,400 potential states (some of these are unreachable because of the obstacles).This problem is probably too large to be solved with unprioritized methods. Figurereprinted from Moore and Atkeson (1993).can in many cases get closer to the true value function with less computation despite

the variance introduced by sampling. Sample updates can win because they break the
overall backing-up computation into smaller pieces—those corresponding to individual
transitions—which then enables it to be focused more narrowly on the pieces that will
have the largest impact. This idea was taken to what may be its logical limit in the “small
backups” introduced by van Seijen and Sutton (2013). These are updates along a single
transition, like a sample update, but based on the probability of the transition without
sampling, as in an expected update. By selecting the order in which small updates
are done it is possible to greatly improve planning eciency beyond that possible with
prioritized sweeping.
We have suggested in this chapter that all kinds of state-space planning can be viewed
as sequences of value updates, varying only in the type of update, expected or sample,
large or small, and in the order in which the updates are done. In this section we have

emphasized backward focusing, but this is just one strategy. For example, another would
be to focus on states according to how easily they can be reached from the states that
are visited frequently under the current policy, which might be called forward focusing .
Peng and Williams (1993) and Barto, Bradtke and Singh (1995) have explored versions
of forward focusing, and the methods introduced in the next few sections take it to an
extreme form.

172 Chapter 8: Planning and Learning with Tabular Methods
8.5 Expected vs. Sample Updates
The examples in the previous sections give some idea of the range of possibilities for
combining methods of learning and planning. In the rest of this chapter, we analyze some
of the component ideas involved, starting with the relative advantages of expected and
sample updates.
Much of this book has been about di↵erent kinds of value-function updates, and we
have considered a great many varieties. Focusing for the moment on one-step updates,
they vary primarily along three binary dimensions. The ﬁrst two dimensions are whether
they update state values or action values and whether they estimate the value for the
optimal policy or for an arbitrary given policy. These two dimensions give rise to four

classes of updates for approximating the four value functions, q⇤,v⇤,q⇡, and v⇡.T h eValueestimatedExpected updates(DP)Sample updates (one-step TD)⇡ss0⇡rpaq⇡(s, a)q⇤(s, a)v⇡(s)v⇤(s)ss0rmaxappolicy evaluationvalue iterationrs0s, aa0⇡pq-policy evaluationrs0s, aa0maxpq-value iterationsAS0RRS0s, aA0RS0s, amaxTD(0)SarsaQ-learninga0
Figure 8.6: Backup diagrams for all the one-step
updates considered in this book.other binary dimension is whether the
updates are expected updates, consider-
ing all possible events that might hap-
pen, or sample updates, considering a

single sample of what might happen.
These three binary dimensions give rise
to eight cases, seven of which corre-
spond to speciﬁc algorithms, as shown
in the ﬁgure to the right. (The eighth
case does not seem to correspond to
any useful update.) Any of these one-
step updates can be used in planning
methods. The Dyna-Q agents discussed
earlier use q⇤sample updates, but they
could just as well use q⇤expected up-
dates, or either expected or sample q⇡
updates. The Dyna-AC system uses v⇡
sample updates together with a learning
policy structure (as in Chapter 13). For
stochastic problems, prioritized sweep-
ing is always done using one of the ex-
pected updates.
When we introduced one-step sam-
ple updates in Chapter 6, we presented
them as substitutes for expected up-

dates. In the absence of a distribution
model, expected updates are not pos-
sible, but sample updates can be done
using sample transitions from the envi-
ronment or a sample model. Implicit in
that point of view is that expected up-
dates, if possible, are preferable to sam-
ple updates. But are they? Expected

8.5. Expected vs. Sample Updates 173
updates certainly yield a better estimate because they are uncorrupted by sampling error,
but they also require more computation, and computation is often the limiting resource
in planning. To properly assess the relative merits of expected and sample updates for
planning we must control for their di↵erent computational requirements.
For concreteness, consider the expected and sample updates for approximating q⇤,
and the special case of discrete states and actions, a table-lookup representation of
the approximate value function, Q, and a model in the form of estimated dynamics,
ˆp(s0,r|s, a). The expected update for a state–action pair, s, a,i s :
Q(s, a) X
s0,rˆp(s0,r|s, a)h
r+max
a0Q(s0,a0)i
. (8.1)
The corresponding sample update for s, a, given a sample next state and reward, S0and
R(from the model), is the Q-learning-like update:
Q(s, a) Q(s, a)+↵h
R+max
a0Q(S0,a0)Q(s, a)i
, (8.2)
where ↵is the usual positive step-size parameter.

Q(s, a) X
r+max
. (8.1)
R+max
, (8.2)
The di↵erence between these expected and sample updates is signiﬁcant to the extent
that the environment is stochastic, speciﬁcally, to the extent that, given a state and
action, many possible next states may occur with various probabilities. If only one next
state is possible, then the expected and sample updates given above are identical (taking
↵= 1). If there are many possible next states, then there may be signiﬁcant di↵erences.
In favor of the expected update is that it is an exact computation, resulting in a new

Q(s, a) whose correctness is limited only by the correctness of the Q(s0,a0) at successor
states. The sample update is in addition a↵ected by sampling error. On the other hand,
the sample update is cheaper computationally because it considers only one next state,
not all possible next states. In practice, the computation required by update operations
is usually dominated by the number of state–action pairs at which Qis evaluated. For a
particular starting pair, s, a,l e t bbe the branching factor (i.e., the number of possible

next states, s0, for which ˆp(s0|s, a)>0). Then an expected update of this pair requires
roughly btimes as much computation as a sample update.
If there is enough time to complete an expected update, then the resulting estimate is
generally better than that of bsample updates because of the absence of sampling error.
But if there is insucient time to complete an expected update, then sample updates are
always preferable because they at least make some improvement in the value estimate
with fewer than bupdates. In a large problem with many state–action pairs, we are often

in the latter situation. With so many state–action pairs, expected updates of all of them
would take a very long time. Before that we may be much better o↵ with a few sample
updates at many state–action pairs than with expected updates at a few pairs. Given a
unit of computational e↵ort, is it better devoted to a few expected updates or to btimes
as many sample updates?
Figure 8.7 shows the results of an analysis that suggests an answer to this question. It

shows the estimation error as a function of computation time for expected and sample
updates for a variety of branching factors, b. The case considered is that in which all

174 Chapter 8: Planning and Learning with Tabular Methodsb = 2 (branching factor)b =10b =100b =1000b =10,000sampleupdatesexpectedupdates1001b2bRMS errorin valueestimateNumber of                         computationsmaxa0Q(s0,a0)
Figure 8.7: Comparison of eciency of expected and sample updates.
bsuccessor states are equally likely and in which the error in the initial estimate is
1. The values at the next states are assumed correct, so the expected update reduces
the error to zero upon its completion. In this case, sample updates reduce the error
according toq
b1
btwhere tis the number of sample updates that have been performed
(assuming sample averages, i.e., ↵=1/t). The key observation is that for moderately
large bthe error falls dramatically with a tiny fraction of bupdates. For these cases,
many state–action pairs could have their values improved dramatically, to within a few
percent of the e↵ect of an expected update, in the same time that a single state–action
pair could undergo an expected update.

b1
The advantage of sample updates shown in Figure 8.7 is probably an underestimate of
the real e↵ect. In a real problem, the values of the successor states would be estimates
that are themselves updated. By causing estimates to be more accurate sooner, sample
updates will have a second advantage in that the values backed up from the successor
states will be more accurate. These results suggest that sample updates are likely to be
superior to expected updates on problems with large stochastic branching factors and
too many states to be solved exactly.

Exercise 8.6 The analysis above assumed that all of the bpossible next states were
equally likely to occur. Suppose instead that the distribution was highly skewed, that
some of the bstates were much more likely to occur than most. Would this strengthen or
weaken the case for sample updates over expected updates? Support your answer. ⇤
8.6 Trajectory Sampling
In this section we compare two ways of distributing updates. The classical approach, from
dynamic programming, is to perform sweeps through the entire state (or state–action)

space, updating each state (or state–action pair) once per sweep. This is problematic

8.6. Trajectory Sampling 175
on large tasks because there may not be time to complete even one sweep. In many
tasks the vast majority of the states are irrelevant because they are visited only under
very poor policies or with very low probability. Exhaustive sweeps implicitly devote
equal time to all parts of the state space rather than focusing where it is needed. As we
discussed in Chapter 4, exhaustive sweeps and the equal treatment of all states that they
imply are not necessary properties of dynamic programming. In principle, updates can
be distributed any way one likes (to assure convergence, all states or state–action pairs
must be visited in the limit an inﬁnite number of times; although an exception to this is
discussed in Section 8.7 below), but in practice exhaustive sweeps are often used.
The second approach is to sample from the state or state–action space according
to some distribution. One could sample uniformly, as in the Dyna-Q agent, but this

would su↵er from some of the same problems as exhaustive sweeps. More appealing
is to distribute updates according to the on-policy distribution, that is, according to
the distribution observed when following the current policy. One advantage of this
distribution is that it is easily generated; one simply interacts with the model, following
the current policy. In an episodic task, one starts in a start state (or according to the

starting-state distribution) and simulates until the terminal state. In a continuing task,
one starts anywhere and just keeps simulating. In either case, sample state transitions
and rewards are given by the model, and sample actions are given by the current policy.
In other words, one simulates explicit individual trajectories and performs updates at the
state or state–action pairs encountered along the way. We call this way of generating
experience and updates trajectory sampling .
It is hard to imagine any ecient way of distributing updates according to the on-policy

distribution other than by trajectory sampling. If one had an explicit representation
of the on-policy distribution, then one could sweep through all states, weighting the
update of each according to the on-policy distribution, but this leaves us again with all
the computational costs of exhaustive sweeps. Possibly one could sample and update
individual state–action pairs from the distribution, but even if this could be done eciently,
what beneﬁt would this provide over simulating trajectories? Even knowing the on-policy

distribution in an explicit form is unlikely. The distribution changes whenever the policy
changes, and computing the distribution requires computation comparable to a complete
policy evaluation. Consideration of such other possibilities makes trajectory sampling
seem both ecient and elegant.
Is the on-policy distribution of updates a good one? Intuitively it seems like a good
choice, at least better than the uniform distribution. For example, if you are learning to
play chess, you study positions that might arise in real games, not random positions of

chess pieces. The latter may be valid states, but to be able to accurately value them is a
di↵erent skill from evaluating positions in real games. We will also see in Part II that the
on-policy distribution has signiﬁcant advantages when function approximation is used.
Whether or not function approximation is used, one might expect on-policy focusing to
signiﬁcantly improve the speed of planning.
Focusing on the on-policy distribution could be beneﬁcial because it causes vast,

uninteresting parts of the space to be ignored, or it could be detrimental because it causes
the same old parts of the space to be updated over and over. We conducted a small

176 Chapter 8: Planning and Learning with Tabular Methods
experiment to assess the e↵ect empirically. To isolate the e↵ect of the update distribution,
we used entirely one-step expected tabular updates, as deﬁned by (8.1). In the uniform
case, we cycled through all state–action pairs, updating each in place, and in the on-policy
case we simulated episodes, all starting in the same state, updating each state–action pair
that occurred under the current "-greedy policy ( "=0.1). The tasks were undiscounted
episodic tasks, generated randomly as follows. From each of the |S|states, two actions
were possible, each of which resulted in one of bnext states, all equally likely, with a
di↵erent random selection of bstates for each state–action pair. The branching factor, b,
was the same for all state–action pairs. In addition, on all transitions there was a 0.1
probability of transition to the terminal state, ending the episode. The expected reward

on each transition was selected from a Gaussian distribution with mean 0 and variance 1.b=10b=3b=1
b=1ion-polcy
ion-polcyuniform
uniform 
0123Value ofstart stateundergreedypolicy05,00010,00015,00020,000Computation time, in full backups
0123Value ofstart stateundergreedypolicy050,000100,000150,000200,000Computation time, in full backupsuniformuniformon-policyon-policyexpected updates
expected updates1,000 STATES
10,000 STATESFigure 8.8: Relative eciency of updates dis-
tributed uniformly across the state space versus
focused on simulated on-policy trajectories, each

uniform 
starting in the same state. Results are for randomly
generated tasks of two sizes and various branching
factors, b.At any point in the planning process
one can stop and exhaustively compute
v˜⇡(s0), the true value of the start state
under the greedy policy, ˜⇡, given the cur-
rent action-value function Q, as an indi-
cation of how well the agent would do on
a new episode on which it acted greed-
ily (all the while assuming the model is
correct).
The upper part of the ﬁgure to
the right shows results averaged over

correct).
200 sample tasks with 1000 states and
branching factors of 1, 3, and 10. The
quality of the policies found is plotted as
a function of the number of expected up-
dates completed. In all cases, sampling
according to the on-policy distribution
resulted in faster planning initially and
retarded planning in the long run. The
e↵ect was stronger, and the initial pe-
riod of faster planning was longer, at
smaller branching factors. In other ex-
periments, we found that these e↵ects
also became stronger as the number of

states increased. For example, the lower
part of the ﬁgure shows results for a
branching factor of 1 for tasks with
10,000 states. In this case the advan-
tage of on-policy focusing is large and
long-lasting.
All of these results make sense. In the
short term, sampling according to the
on-policy distribution helps by focusing
on states that are near descendants of

8.7. Real-time Dynamic Programming 177
the start state. If there are many states and a small branching factor, this e↵ect will be
large and long-lasting. In the long run, focusing on the on-policy distribution may hurt
because the commonly occurring states all already have their correct values. Sampling
them is useless, whereas sampling other states may actually perform some useful work.
This presumably is why the exhaustive, unfocused approach does better in the long run,
at least for small problems. These results are not conclusive because they are only for
problems generated in a particular, random way, but they do suggest that sampling
according to the on-policy distribution can be a great advantage for large problems, in
particular for problems in which a small subset of the state–action space is visited under
the on-policy distribution.
Exercise 8.7 Some of the graphs in Figure 8.8 seem to be scalloped in their early portions,

particularly the upper graph for b= 1 and the uniform distribution. Why do you think
this is? What aspects of the data shown support your hypothesis? ⇤
Exercise 8.8 (programming) Replicate the experiment whose results are shown in the
lower part of Figure 8.8, then try the same experiment but with b= 3. Discuss the
meaning of your results. ⇤
8.7 Real-time Dynamic Programming
Real-time dynamic programming , or RTDP, is an on-policy trajectory-sampling version of
the value-iteration algorithm of dynamic programming (DP). Because it is closely related

to conventional sweep-based policy iteration, RTDP illustrates in a particularly clear way
some of the advantages that on-policy trajectory sampling can provide. RTDP updates
the values of states visited in actual or simulated trajectories by means of expected
tabular value-iteration updates as deﬁned by (4.10). It is basically the algorithm that
produced the on-policy results shown in Figure 8.8.
The close connection between RTDP and conventional DP makes it possible to derive

some theoretical results by adapting existing theory. RTDP is an example of an asyn-
chronous DP algorithm as described in Section 4.5. Asynchronous DP algorithms are
not organized in terms of systematic sweeps of the state set; they update state values in
any order whatsoever, using whatever values of other states happen to be available. In
RTDP, the update order is dictated by the order states are visited in real or simulated

trajectories.Start StatesIrrelevant States: unreachable  from any start stateunder any optimal policyRelevant Statesreachable from some start state under some optimal policyIf trajectories can start only from a designated
set of start states, and if you are interested in
the prediction problem for a given policy, then on-
policy trajectory sampling allows the algorithm to
completely skip states that cannot be reached by
the given policy from any of the start states: such
states are irrelevant to the prediction problem.
For a control problem, where the goal is to ﬁnd

an optimal policy instead of evaluating a given
policy, there might well be states that cannot be

178 Chapter 8: Planning and Learning with Tabular Methods
reached by any optimal policy from any of the start states, and there is no need to specify
optimal actions for these irrelevant states. What is needed is an optimal partial policy ,
meaning a policy that is optimal for the relevant states but can specify arbitrary actions,
or even be undeﬁned, for the irrelevant states.
But ﬁnding such an optimal partial policy with an on-policy trajectory-sampling
control method, such as Sarsa (Section 6.4), in general requires visiting all state–action
pairs—even those that will turn out to be irrelevant—an inﬁnite number of times. This
can be done, for example, by using exploring starts (Section 5.3). This is true for RTDP
as well: for episodic tasks with exploring starts, RTDP is an asynchronous value-iteration
algorithm that converges to optimal polices for discounted ﬁnite MDPs (and for the
undiscounted case under certain conditions). Unlike the situation for a prediction problem,

it is generally not possible to stop updating any state or state–action pair if convergence
to an optimal policy is important.
The most interesting result for RTDP is that for certain types of problems satisfying
reasonable conditions, RTDP is guaranteed to ﬁnd a policy that is optimal on the relevant
states without visiting every state inﬁnitely often, or even without visiting some states at
all. Indeed, in some problems, only a small fraction of the states need to be visited. This
can be a great advantage for problems with very large state sets, where even a single

sweep may not be feasible.
The tasks for which this result holds are undiscounted episodic tasks for MDPs with
absorbing goal states that generate zero rewards, as described in Section 3.4. At every step
of a real or simulated trajectory, RTDP selects a greedy action (breaking ties randomly)
and applies the expected value-iteration update operation to the current state. It can
also update the values of an arbitrary collection of other states at each step; for example,

it can update the values of states visited in a limited-horizon look-ahead search from the
current state.
For these problems, with each episode beginning in a state randomly chosen from the
set of start states and ending at a goal state, RTDP converges with probability one to a
policy that is optimal for all the relevant states provided: 1) the initial value of every
goal state is zero, 2) there exists at least one policy that guarantees that a goal state
will be reached with probability one from any start state, 3) all rewards for transitions

from non-goal states are strictly negative, and 4) all the initial values are equal to, or
greater than, their optimal values (which can be satisﬁed by simply setting the initial
values of all states to zero). This result was proved by Barto, Bradtke, and Singh (1995)
by combining results for asynchronous DP with results about a heuristic search algorithm
known as learning real-time A* due to Korf (1990).
Tasks having these properties are examples of stochastic optimal path problems ,w h i c h

are usually stated in terms of cost minimization instead of as reward maximization as
we do here. Maximizing the negative returns in our version is equivalent to minimizing
the costs of paths from a start state to a goal state. Examples of this kind of task are
minimum-time control tasks, where each time step required to reach a goal produces a
reward of 1, or problems like the Golf example in Section 3.5, whose objective is to hit
the hole with the fewest strokes.

8.7. Real-time Dynamic Programming 179
Example 8.6: RTDP on the Racetrack The racetrack problem of Exercise 5.12
(page 111) is a stochastic optimal path problem. Comparing RTDP and the conventional
DP value iteration algorithm on an example racetrack problem illustrates some of the
advantages of on-policy trajectory sampling.
Recall from the exercise that an agent has to learn how to drive a car around a turn
like those shown in Figure 5.5 and cross the ﬁnish line as quickly as possible while staying
on the track. Start states are all the zero-speed states on the starting line; the goal states
are all the states that can be reached in one time step by crossing the ﬁnish line from
inside the track. Unlike Exercise 5.12, here there is no limit on the car’s speed, so the
state set is potentially inﬁnite. However, the set of states that can be reached from the
set of start states via any policy is ﬁnite and can be considered to be the state set of the

problem. Each episode begins in a randomly selected start state and ends when the car
crosses the ﬁnish line. The rewards are 1 for each step until the car crosses the ﬁnish
line. If the car hits the track boundary, it is moved back to a random start state, and the
episode continues.
A racetrack similar to the small racetrack on the left of Figure 5.5 has 9,115 states
reachable from start states by any policy, only 599 of which are relevant, meaning that
they are reachable from some start state via some optimal policy. (The number of relevant

states was estimated by counting the states visited while executing optimal actions for
107episodes.)
The table below compares solving this task by conventional DP and by RTDP. These
results are averages over 25 runs, each begun with a di↵erent random number seed.
Conventional DP in this case is value iteration using exhaustive sweeps of the state set,
with values updated one state at a time in place, meaning that the update for each state
uses the most recent values of the other states (This is the Gauss-Seidel version of value

iteration, which was found to be approximately twice as fast as the Jacobi version on
this problem. See Section 4.8.) No special attention was paid to the ordering of the
updates; other orderings could have produced faster convergence. Initial values were all
zero for each run of both methods. DP was judged to have converged when the maximum
change in a state value over a sweep was less than 104, and RTDP was judged to have
converged when the average time to cross the ﬁnish line over 20 episodes appeared to

stabilize at an asymptotic number of steps. This version of RTDP updated only the value
of the current state on each step.
DP RTDP
Average computation to convergence 28 sweeps 4000 episodes
Average number of updates to convergence 252,784 127,600
Average number of updates per episode — 31.9
% of states updated 100 times — 98.45
% of states updated 10 times — 80.51
% of states updated 0 times — 3.18
Both methods produced policies averaging between 14 and 15 steps to cross the ﬁnish
line, but RTDP required only roughly half of the updates that DP did. This is the result

DP RTDP
of RTDP’s on-policy trajectory sampling. Whereas the value of every state was updated

180 Chapter 8: Planning and Learning with Tabular Methods
in each sweep of DP, RTDP focused updates on fewer states. In an average run, RTDP
updated the values of 98.45% of the states no more than 100 times and 80.51% of the
states no more than 10 times; the values of about 290 states were not updated at all in
an average run.
Another advantage of RTDP is that as the value function approaches the optimal
value function v⇤, the policy used by the agent to generate trajectories approaches an
optimal policy because it is always greedy with respect to the current value function.
This is in contrast to the situation in conventional value iteration. In practice, value
iteration terminates when the value function changes by only a small amount in a sweep,
which is how we terminated it to obtain the results in the table above. At this point,
the value function closely approximates v⇤, and a greedy policy is close to an optimal
policy. However, it is possible that policies that are greedy with respect to the latest

value function were optimal, or nearly so, long before value iteration terminates. (Recall
from Chapter 4 that optimal policies can be greedy with respect to many di↵erent
value functions, not just v⇤.) Checking for the emergence of an optimal policy before
value iteration converges is not a part of the conventional DP algorithm and requires
considerable additional computation.
In the racetrack example, by running many test episodes after each DP sweep, with
actions selected greedily according to the result of that sweep, it was possible to estimate

the earliest point in the DP computation at which the approximated optimal evaluation
function was good enough so that the corresponding greedy policy was nearly optimal.
For this racetrack, a close-to-optimal policy emerged after 15 sweeps of value iteration, or
after 136,725 value-iteration updates. This is considerably less than the 252,784 updates
DP needed to converge to v⇤, but sill more than the 127,600 updates RTDP required.
Although these simulations are certainly not deﬁnitive comparisons of the RTDP with

conventional sweep-based value iteration, they illustrate some of advantages of on-policy
trajectory sampling. Whereas conventional value iteration continued to update the value
of all the states, RTDP strongly focused on subsets of the states that were relevant to
the problem’s objective. This focus became increasingly narrow as learning continued.
Because the convergence theorem for RTDP applies to the simulations, we know that
RTDP eventually would have focused only on relevant states, i.e., on states making up

optimal paths. RTDP achieved nearly optimal control with about 50% of the computation
required by sweep-based value iteration.
8.8 Planning at Decision Time
Planning can be used in at least two ways. The one we have considered so far in this
chapter, typiﬁed by dynamic programming and Dyna, is to use planning to gradually
improve a policy or value function on the basis of simulated experience obtained from
a model (either a sample or a distribution model). Selecting actions is then a matter
of comparing the current state’s action values obtained from a table in the tabular

case we have thus far considered, or by evaluating a mathematical expression in the
approximate methods we consider in Part II below. Well before an action is selected for
any current state St, planning has played a part in improving the table entries, or the

8.9. Heuristic Search 181
mathematical expression, needed to select the action for many states, including St. Used
this way, planning is not focussed on the current state. We call planning used in this way
background planning .
The other way to use planning is to begin and complete it after encountering each
new state St, as a computation whose output is the selection of a single action At; on
the next step planning begins anew with St+1to produce At+1, and so on. The simplest,
and almost degenerate, example of this use of planning is when only state values are
available, and an action is selected by comparing the values of model-predicted next states
for each action (or by comparing the values of afterstates as in the tic-tac-toe example
in Chapter 1). More generally, planning used in this way can look much deeper than
one-step-ahead and evaluate action choices leading to many di↵erent predicted state and
reward trajectories. Unlike the ﬁrst use of planning, here planning focuses on a particular

state. We call this decision-time planning .
These two ways of thinking about planning—using simulated experience to gradually
improve a policy or value function, or using simulated experience to select an action for
the current state—can blend together in natural and interesting ways, but they have
tended to be studied separately, and that is a good way to ﬁrst understand them. Let us
now take a closer look at decision-time planning.
Even when planning is only done at decision time, we can still view it, as we did

in Section 8.1, as proceeding from simulated experience to updates and values, and
ultimately to a policy. It is just that now the values and policy are speciﬁc to the current
state and the action choices available there, so much so that the values and policy created
by the planning process are typically discarded after being used to select the current
action. In many applications this is not a great loss because there are very many states
and we are unlikely to return to the same state for a long time. In general, one may

want to do a mix of both: focus planning on the current state and store the results
of planning so as to be that much farther along should one return to the same state
later. Decision-time planning is most useful in applications in which fast responses are
not required. In chess playing programs, for example, one may be permitted seconds or
minutes of computation for each move, and strong programs may plan dozens of moves
ahead within this time. On the other hand, if low latency action selection is the priority,

then one is generally better o↵ doing planning in the background to compute a policy
that can then be rapidly applied to each newly encountered state.
8.9 Heuristic Search
The classical state-space planning methods in artiﬁcial intelligence are decision-time
planning methods collectively known as heuristic search . In heuristic search, for each
state encountered, a large tree of possible continuations is considered. The approximate
value function is applied to the leaf nodes and then backed up toward the current state

at the root. The backing up within the search tree is just the same as in the expected
updates with maxes (those for v⇤andq⇤) discussed throughout this book. The backing
up stops at the state–action nodes for the current state. Once the backed-up values of
these nodes are computed, the best of them is chosen as the current action, and then all

182 Chapter 8: Planning and Learning with Tabular Methods
backed-up values are discarded.
In conventional heuristic search no e↵ort is made to save the backed-up values by
changing the approximate value function. In fact, the value function is generally designed
by people and never changed as a result of search. However, it is natural to consider
allowing the value function to be improved over time, using either the backed-up values
computed during heuristic search or any of the other methods presented throughout
this book. In a sense we have taken this approach all along. Our greedy, "-greedy, and
UCB (Section 2.7) action-selection methods are not unlike heuristic search, albeit on a
smaller scale. For example, to compute the greedy action given a model and a state-value
function, we must look ahead from each possible action to each possible next state, take
into account the rewards and estimated values, and then pick the best action. Just as

in conventional heuristic search, this process computes backed-up values of the possible
actions, but does not attempt to save them. Thus, heuristic search can be viewed as an
extension of the idea of a greedy policy beyond a single step.
The point of searching deeper than one step is to obtain better action selections. If one
has a perfect model and an imperfect action-value function, then in fact deeper search
will usually yield better policies.2Certainly, if the search is all the way to the end of

the episode, then the e↵ect of the imperfect value function is eliminated, and the action
determined in this way must be optimal. If the search is of sucient depth ksuch that k
is very small, then the actions will be correspondingly near optimal. On the other hand,
the deeper the search, the more computation is required, usually resulting in a slower
response time. A good example is provided by Tesauro’s grandmaster-level backgammon

player, TD-Gammon (Section 16.1). This system used TD learning to learn an afterstate
value function through many games of self-play, using a form of heuristic search to make
its moves. As a model, TD-Gammon used a priori knowledge of the probabilities of dice
rolls and the assumption that the opponent always selected the actions that TD-Gammon
rated as best for it. Tesauro found that the deeper the heuristic search, the better the
moves made by TD-Gammon, but the longer it took to make each move. Backgammon

has a large branching factor, yet moves must be made within a few seconds. It was
only feasible to search ahead selectively a few steps, but even so the search resulted in
signiﬁcantly better action selections.
We should not overlook the most obvious way in which heuristic search focuses updates:
on the current state. Much of the e↵ectiveness of heuristic search is due to its search tree
being tightly focused on the states and actions that might immediately follow the current
state. You may spend more of your life playing chess than checkers, but when you play

checkers, it pays to think about checkers and about your particular checkers position,
your likely next moves, and successor positions. No matter how you select actions, it
is these states and actions that are of highest priority for updates and where you most
urgently want your approximate value function to be accurate. Not only should your
computation be preferentially devoted to imminent events, but so should your limited
memory resources. In chess, for example, there are far too many possible positions to

store distinct value estimates for each of them, but chess programs based on heuristic
search can easily store distinct estimates for the millions of positions they encounter
2There are interesting exceptions to this (see, e.g., Pearl, 1984).

8.10. Rollout Algorithms 183
looking ahead from a single position. This great focusing of memory and computational
resources on the current decision is presumably the reason why heuristic search can be so
e↵ective.
The distribution of updates can be altered in similar ways to focus on the current
state and its likely successors. As a limiting case we might use exactly the methods of
heuristic search to construct a search tree, and then perform the individual, one-step
updates from bottom up, as suggested by Figure 8.9. If the updates are ordered in this
way and a tabular representation is used, then exactly the same overall update would
be achieved as in depth-ﬁrst heuristic search. Any state-space search can be viewed in
this way as the piecing together of a large number of individual one-step updates. Thus,
the performance improvement observed with deeper searches is not due to the use of
multistep updates as such. Instead, it is due to the focus and concentration of updates

on states and actions immediately downstream from the current state. By devoting a
large amount of computation speciﬁcally relevant to the candidate actions, decision-time
planning can produce better decisions than can be produced by relying on unfocused
updates.
123
45678910Figure 8.9: Heuristic search can be implemented as a sequence of one-step updates (shown
here outlined in blue) backing up values from the leaf nodes toward the root. The ordering
shown here is for a selective depth-ﬁrst search.
8.10 Rollout Algorithms

updates.
123
Rollout algorithms are decision-time planning algorithms based on Monte Carlo control
applied to simulated trajectories that all begin at the current environment state. They
estimate action values for a given policy by averaging the returns of many simulated
trajectories that start with each possible action and then follow the given policy. When
the action-value estimates are considered to be accurate enough, the action (or one of the

184 Chapter 8: Planning and Learning with Tabular Methods
actions) having the highest estimated value is executed, after which the process is carried
out anew from the resulting next state. As explained by Tesauro and Galperin (1997),
who experimented with rollout algorithms for playing backgammon, the term “rollout”
comes from estimating the value of a backgammon position by playing out, i.e., “rolling
out,” the position many times to the game’s end with randomly generated sequences of
dice rolls, where the moves of both players are made by some ﬁxed policy.
Unlike the Monte Carlo control algorithms described in Chapter 5, the goal of a
rollout algorithm is not to estimate a complete optimal action-value function, q⇤, or a
complete action-value function, q⇡, for a given policy ⇡. Instead, they produce Monte
Carlo estimates of action values only for each current state and for a given policy usually
called the rollout policy . As decision-time planning algorithms, rollout algorithms make

immediate use of these action-value estimates, then discard them. This makes rollout
algorithms relatively simple to implement because there is no need to sample outcomes
for every state-action pair, and there is no need to approximate a function over either
the state space or the state-action space.
What then do rollout algorithms accomplish? The policy improvement theorem
described in Section 4.2 tells us that given any two policies ⇡and⇡0that are identical

except that ⇡0(s)=a6=⇡(s) for some state s,i fq⇡(s, a)v⇡(s), then policy ⇡0is as good
as, or better, than ⇡. Moreover, if the inequality is strict, then ⇡0is in fact better than ⇡.
This applies to rollout algorithms where sis the current state and ⇡is the rollout policy.
Averaging the returns of the simulated trajectories produces estimates of q⇡(s, a0) for
each action a02A(s). Then the policy that selects an action in sthat maximizes these
estimates and thereafter follows ⇡is a good candidate for a policy that improves over

⇡. The result is like one step of the policy-iteration algorithm of dynamic programming
discussed in Section 4.3 (though it is more like one step of asynchronous value iteration
described in Section 4.5 because it changes the action for just the current state).
In other words, the aim of a rollout algorithm is to improve upon the rollout policy;
not to ﬁnd an optimal policy. Experience has shown that rollout algorithms can be
surprisingly e↵ective. For example, Tesauro and Galperin (1997) were surprised by the

dramatic improvements in backgammon playing ability produced by the rollout method.
In some applications, a rollout algorithm can produce good performance even if the
rollout policy is completely random. But the performance of the improved policy depends
on properties of the rollout policy and the ranking of actions produced by the Monte
Carlo value estimates. Intuition suggests that the better the rollout policy and the more
accurate the value estimates, the better the policy produced by a rollout algorithm is
likely be (but see Gelly and Silver, 2007).

This involves important tradeo↵s because better rollout policies typically mean that
more time is needed to simulate enough trajectories to obtain good value estimates.
As decision-time planning methods, rollout algorithms usually have to meet strict time
constraints. The computation time needed by a rollout algorithm depends on the number
of actions that have to be evaluated for each decision, the number of time steps in the
simulated trajectories needed to obtain useful sample returns, the time it takes the rollout

policy to make decisions, and the number of simulated trajectories needed to obtain good
Monte Carlo action-value estimates.

8.11. Monte Carlo Tree Search 185
Balancing these factors is important in any application of rollout methods, though there
are several ways to ease the challenge. Because the Monte Carlo trials are independent of
one another, it is possible to run many trials in parallel on separate processors. Another
approach is to truncate the simulated trajectories short of complete episodes, correcting
the truncated returns by means of a stored evaluation function (which brings into play
all that we have said about truncated returns and updates in the preceding chapters).
It is also possible, as Tesauro and Galperin (1997) suggest, to monitor the Monte Carlo
simulations and prune away candidate actions that are unlikely to turn out to be the
best, or whose values are close enough to that of the current best that choosing them
instead would make no real di↵erence (though Tesauro and Galperin point out that this
would complicate a parallel implementation).

We do not ordinarily think of rollout algorithms as learning algorithms because they
do not maintain long-term memories of values or policies. However, these algorithms take
advantage of some of the features of reinforcement learning that we have emphasized
in this book. As instances of Monte Carlo control, they estimate action values by
averaging the returns of a collection of sample trajectories, in this case trajectories of
simulated interactions with a sample model of the environment. In this way they are

like reinforcement learning algorithms in avoiding the exhaustive sweeps of dynamic
programming by trajectory sampling, and in avoiding the need for distribution models
by relying on sample, instead of expected, updates. Finally, rollout algorithms take
advantage of the policy improvement property by acting greedily with respect to the
estimated action values.
8.11 Monte Carlo Tree Search
Monte Carlo Tree Search (MCTS) is a recent and strikingly successful example of decision-
time planning. At its base, MCTS is a rollout algorithm as described above, but enhanced

by the addition of a means for accumulating value estimates obtained from the Monte
Carlo simulations in order to successively direct simulations toward more highly-rewarding
trajectories. MCTS is largely responsible for the improvement in computer Go from
a weak amateur level in 2005 to a grandmaster level (6 dan or more) in 2015. Many
variations of the basic algorithm have been developed, including a variant that we discuss
in Section 16.6 that was critical for the stunning 2016 victories of the program AlphaGo

over an 18-time world champion Go player. MCTS has proved to be e↵ective in a wide
variety of competitive settings, including general game playing (e.g., see Finnsson and
Bj¨ ornsson, 2008; Genesereth and Thielscher, 2014), but it is not limited to games; it can
be e↵ective for single-agent sequential decision problems if there is an environment model
simple enough for fast multistep simulation.
MCTS is executed after encountering each new state to select the agent’s action for
that state; it is executed again to select the action for the next state, and so on. As in a

rollout algorithm, each execution is an iterative process that simulates many trajectories
starting from the current state and running to a terminal state (or until discounting
makes any further reward negligible as a contribution to the return). The core idea
of MCTS is to successively focus multiple simulations starting at the current state by

186 Chapter 8: Planning and Learning with Tabular Methods
extending the initial portions of trajectories that have received high evaluations from
earlier simulations. MCTS does not have to retain approximate value functions or policies
from one action selection to the next, though in many implementations it retains selected
action values likely to be useful for its next execution.
For the most part, the actions in the simulated trajectories are generated using a simple
policy, usually called a rollout policy as it is for simpler rollout algorithms. When both
the rollout policy and the model do not require a lot of computation, many simulated
trajectories can be generated in a short period of time. As in any tabular Monte Carlo
method, the value of a state–action pair is estimated as the average of the (simulated)
returns from that pair. Monte Carlo value estimates are maintained only for the subset
of state–action pairs that are most likely to be reached in a few steps, which form a tree

rooted at the current state, as illustrated in Figure 8.10. MCTS incrementally extends
the tree by adding nodes representing states that look promising based on the results of
the simulated trajectories. Any simulated trajectory will pass through the tree and then
exit it at some leaf node. Outside the tree and at the leaf nodes the rollout policy is used
for action selections, but at the states inside the tree something better is possible. For
these states we have value estimates for of at least some of the actions, so we can pick

among them using an informed policy, called the tree policy , that balances explorationSelectionSimulationExpansionBackupRepeat while time remains Tree PolicyRolloutPolicyFigure 8.10: Monte Carlo Tree Search. When the environment changes to a new state, MCTS
executes as many iterations as possible before an action needs to be selected, incrementally
building a tree whose root node represents the current state. Each iteration consists of the four
operations Selection ,Expansion (though possibly skipped on some iterations), Simulation ,

andBackup , as explained in the text and illustrated by the bold arrows in the trees. Adapted
from Chaslot, Bakkes, Szita, and Spronck (2008).

8.11. Monte Carlo Tree Search 187
and exploitation. For example, the tree policy could select actions using an "-greedy or
UCB selection rule (Chapter 2).
In more detail, each iteration of a basic version of MCTS consists of the following four
steps as illustrated in Figure 8.10:
1.Selection. Starting at the root node, a tree policy based on the action values
attached to the edges of the tree traverses the tree to select a leaf node.
2.Expansion. On some iterations (depending on details of the application), the tree
is expanded from the selected leaf node by adding one or more child nodes reached
from the selected node via unexplored actions.
3.Simulation. From the selected node, or from one of its newly-added child nodes
(if any), simulation of a complete episode is run with actions selected by the rollout
policy. The result is a Monte Carlo trial with actions selected ﬁrst by the tree
policy and beyond the tree by the rollout policy.

4.Backup. The return generated by the simulated episode is backed up to update,
or to initialize, the action values attached to the edges of the tree traversed by
the tree policy in this iteration of MCTS. No values are saved for the states and
actions visited by the rollout policy beyond the tree. Figure 8.10 illustrates this by
showing a backup from the terminal state of the simulated trajectory directly to the
state–action node in the tree where the rollout policy began (though in general, the

entire return over the simulated trajectory is backed up to this state–action node).
MCTS continues executing these four steps, starting each time at the tree’s root node,
until no more time is left, or some other computational resource is exhausted. Then,
ﬁnally, an action from the root node (which still represents the current state of the
environment) is selected according to some mechanism that depends on the accumulated
statistics in the tree; for example, it may be an action having the largest action value

of all the actions available from the root state, or perhaps the action with the largest
visit count to avoid selecting outliers. This is the action MCTS actually selects. After
the environment transitions to a new state, MCTS is run again, sometimes starting
with a tree of a single root node representing the new state, but often starting with a
tree containing any descendants of this node left over from the tree constructed by the
previous execution of MCTS; all the remaining nodes are discarded, along with the action
values associated with them.

MCTS was ﬁrst proposed to select moves in programs playing two-person competitive
games, such as Go. For game playing, each simulated episode is one complete play of the
game in which both players select actions by the tree and rollout policies. Section 16.6
describes an extension of MCTS used in the AlphaGo program that combines the Monte
Carlo evaluations of MCTS with action values learned by a deep artiﬁcial neural network
via self-play reinforcement learning.
Relating MCTS to the reinforcement learning principles we describe in this book

provides some insight into how it achieves such impressive results. At its base, MCTS is
a decision-time planning algorithm based on Monte Carlo control applied to simulations

188 Chapter 8: Planning and Learning with Tabular Methods
that start from the root state; that is, it is a kind of rollout algorithm as described in
the previous section. It therefore beneﬁts from online, incremental, sample-based value
estimation and policy improvement. Beyond this, it saves action-value estimates attached
to the tree edges and updates them using reinforcement learning’s sample updates. This
has the e↵ect of focusing the Monte Carlo trials on trajectories whose initial segments
are common to high-return trajectories previously simulated. Further, by incrementally
expanding the tree, MCTS e↵ectively grows a lookup table to store a partial action-value
function, with memory allocated to the estimated values of state–action pairs visited in
the initial segments of high-yielding sample trajectories. MCTS thus avoids the problem
of globally approximating an action-value function while it retains the beneﬁt of using
past experience to guide exploration.

The striking success of decision-time planning by MCTS has deeply inﬂuenced artiﬁcial
intelligence, and many researchers are studying modiﬁcations and extensions of the basic
procedure for use in both games and single-agent applications.
8.12 Summary of the Chapter
Planning requires a model of the environment. A distribution model consists of the
probabilities of next states and rewards for possible actions; a sample model produces
single transitions and rewards generated according to these probabilities. Dynamic

programming requires a distribution model because it uses expected updates , which involve
computing expectations over all the possible next states and rewards. A sample model ,
on the other hand, is what is needed to simulate interacting with the environment during
which sample updates , like those used by many reinforcement learning algorithms, can be
used. Sample models are generally much easier to obtain than distribution models.
We have presented a perspective emphasizing the surprisingly close relationships be-

tween planning optimal behavior and learning optimal behavior. Both involve estimating
the same value functions, and in both cases it is natural to update the estimates incre-
mentally, in a long series of small backing-up operations. This makes it straightforward
to integrate learning and planning processes simply by allowing both to update the same
estimated value function. In addition, any of the learning methods can be converted into
planning methods simply by applying them to simulated (model-generated) experience

rather than to real experience. In this case learning and planning become even more
similar; they are possibly identical algorithms operating on two di↵erent sources of
It is straightforward to integrate incremental planning methods with acting and model-
learning. Planning, acting, and model-learning interact in a circular fashion (as in
the diagram on page 162), each producing what the other needs to improve; no other
interaction among them is either required or prohibited. The most natural approach

is for all processes to proceed asynchronously and in parallel. If the processes must
share computational resources, then the division can be handled almost arbitrarily—by
whatever organization is most convenient and ecient for the task at hand.
In this chapter we have touched upon a number of dimensions of variation among
state-space planning methods. One dimension is the variation in the size of updates. The

8.13. Summary of Part I: Dimensions 189
smaller the updates, the more incremental the planning methods can be. Among the
smallest updates are one-step sample updates, as in Dyna. Another important dimension
is the distribution of updates, that is, of the focus of search. Prioritized sweeping focuses
backward on the predecessors of states whose values have recently changed. On-policy
trajectory sampling focuses on states or state–action pairs that the agent is likely to
encounter when controlling its environment. This can allow computation to skip over
parts of the state space that are irrelevant to the prediction or control problem. Real-
time dynamic programming, an on-policy trajectory sampling version of value iteration,
illustrates some of the advantages this strategy has over conventional sweep-based policy
iteration.
Planning can also focus forward from pertinent states, such as states actually encoun-
tered during an agent-environment interaction. The most important form of this is when

planning is done at decision time, that is, as part of the action-selection process. Classical
heuristic search as studied in artiﬁcial intelligence is an example of this. Other examples
are rollout algorithms and Monte Carlo Tree Search that beneﬁt from online, incremental,
sample-based value estimation and policy improvement.
8.13 Summary of Part I: Dimensions
This chapter concludes Part I of this book. In it we have tried to present reinforcement
learning not as a collection of individual methods, but as a coherent set of ideas cutting

across methods. Each idea can be viewed as a dimension along which methods vary. The
set of such dimensions spans a large space of possible methods. By exploring this space
at the level of dimensions we hope to obtain the broadest and most lasting understanding.
In this section we use the concept of dimensions in method space to recapitulate the view
of reinforcement learning developed so far in this book.
All of the methods we have explored so far in this book have three key ideas in common:

ﬁrst, they all seek to estimate value functions; second, they all operate by backing up
values along actual or possible state trajectories; and third, they all follow the general
strategy of generalized policy iteration (GPI), meaning that they maintain an approximate
value function and an approximate policy, and they continually try to improve each on the
basis of the other. These three ideas are central to the subjects covered in this book. We

suggest that value functions, backing up value updates, and GPI are powerful organizing
principles potentially relevant to any model of intelligence, whether artiﬁcial or natural.
Two of the most important dimensions along which the methods vary are shown in
Figure 8.11. These dimensions have to do with the kind of update used to improve the
value function. The horizontal dimension is whether they are sample updates (based on a
sample trajectory) or expected updates (based on a distribution of possible trajectories).

Expected updates require a distribution model, whereas sample updates need only a
sample model, or can be done from actual experience with no model at all (another
dimension of variation). The vertical dimension of Figure 8.11 corresponds to the depth
of updates, that is, to the degree of bootstrapping. At three of the four corners of the
space are the three primary methods for estimating values: dynamic programming, TD,
and Monte Carlo. Along the left edge of the space are the sample-update methods,

190 Chapter 8: Planning and Learning with Tabular Methodswidthof updatedepth(length)of update
Temporal-differencelearning
Dynamicprogramming
MonteCarlo...
ExhaustivesearchFigure 8.11: A slice through the space of reinforcement learning methods, highlighting the
two of the most important dimensions explored in Part I of this book: the depth and width of
the updates.
ranging from one-step TD updates to full-return Monte Carlo updates. Between these
is a spectrum including methods based on n-step updates (and in Chapter 12 we will
extend this to mixtures of n-step updates such as the -updates implemented by eligibility
traces).
Dynamic programming methods are shown in the extreme upper-right corner of the
space because they involve one-step expected updates. The lower-right corner is the
extreme case of expected updates so deep that they run all the way to terminal states
(or, in a continuing task, until discounting has reduced the contribution of any further

traces).
rewards to a negligible level). This is the case of exhaustive search. Intermediate methods
along this dimension include heuristic search and related methods that search and update
up to a limited depth, perhaps selectively. There are also methods that are intermediate
along the horizontal dimension. These include methods that mix expected and sample
updates, as well as the possibility of methods that mix samples and distributions within
a single update. The interior of the square is ﬁlled in to represent the space of all such
intermediate methods.

A third dimension that we have emphasized in this book is the binary distinction
between on-policy and o↵-policy methods. In the former case, the agent learns the value
function for the policy it is currently following, whereas in the latter case it learns the

Bibliographical and Historical Remarks 191
value function for the policy for a di↵erent policy, often the one that the agent currently
thinks is best. The policy generating behavior is typically di↵erent from what is currently
thought best because of the need to explore. This third dimension might be visualized as
perpendicular to the plane of the page in Figure 8.11.
In addition to the three dimensions just discussed, we have identiﬁed a number of
others throughout the book:
Deﬁnition of return Is the task episodic or continuing, discounted or undiscounted?
Action values vs. state values vs. afterstate values What kind of values should
be estimated? If only state values are estimated, then either a model or a separate
policy (as in actor–critic methods) is required for action selection.
Action selection/exploration How are actions selected to ensure a suitable trade-o↵
between exploration and exploitation? We have considered only the simplest ways to

do this: "-greedy, optimistic initialization of values, soft-max, and upper conﬁdence
bound.
Synchronous vs. asynchronous Are the updates for all states performed simultane-
ously or one by one in some order?
Real vs. simulated Should one update based on real experience or simulated experi-
ence? If both, how much of each?
Location of updates What states or state–action pairs should be updated? Model-
free methods can choose only among the states and state–action pairs actually

bound.
encountered, but model-based methods can choose arbitrarily. There are many
possibilities here.
Timing of updates Should updates be done as part of selecting actions, or only after-
ward?
Memory for updates How long should updated values be retained? Should they be
retained permanently, or only while computing an action selection, as in heuristic
search?
Of course, these dimensions are neither exhaustive nor mutually exclusive. Individual
algorithms di↵er in many other ways as well, and many algorithms lie in several places

ward?
search?
along several dimensions. For example, Dyna methods use both real and simulated
experience to a↵ect the same value function. It is also perfectly sensible to maintain
multiple value functions computed in di↵erent ways or over di↵erent state and action
representations. These dimensions do, however, constitute a coherent set of ideas for
describing and exploring a wide space of possible methods.
The most important dimension not mentioned here, and not covered in Part I of
this book, is that of function approximation. Function approximation can be viewed as

an orthogonal spectrum of possibilities ranging from tabular methods at one extreme
through state aggregation, a variety of linear methods, and then a diverse set of nonlinear
methods. This dimension is explored in Part II.

192 Chapter 8: Planning and Learning with Tabular Methods
8.1 The overall view of planning and learning presented here has developed gradually
over a number of years, in part by the authors (Sutton, 1990, 1991a, 1991b;
Barto, Bradtke, and Singh, 1991, 1995; Sutton and Pinette, 1985; Sutton and
Barto, 1981b); it has been strongly inﬂuenced by Agre and Chapman (1990; Agre
1988), Bertsekas and Tsitsiklis (1989), Singh (1993), and others. The authors
were also strongly inﬂuenced by psychological studies of latent learning (Tolman,
1932) and by psychological views of the nature of thought (e.g., Galanter and
Gerstenhaber, 1956; Craik, 1943; Campbell, 1960; Dennett, 1978). In Part
III of the book, Section 14.6 relates model-based and model-free methods to
psychological theories of learning and behavior, and Section 15.11 discusses ideas
about how the brain might implement these types of methods.
8.2 The terms direct and indirect , which we use to describe di↵erent kinds of

reinforcement learning, are from the adaptive control literature (e.g., Goodwin
and Sin, 1984), where they are used to make the same kind of distinction. The
term system identiﬁcation is used in adaptive control for what we call model-
learning (e.g., Goodwin and Sin, 1984; Ljung and S¨ oderstrom, 1983; Young,
1984). The Dyna architecture is due to Sutton (1990), and the results in this
and the next section are based on results reported there. Barto and Singh
(1990) consider some of the issues in comparing direct and indirect reinforcement

learning methods. Early work extending Dyna to linear function approximation
(Chapter 9) was done by Sutton, Szepesv´ ari, Geramifard, and Bowling (2008)
and by Parr, Li, Taylor, Painter-Wakeﬁeld, and Littman (2008).
8.3 There have been several works with model-based reinforcement learning that take
the idea of exploration bonuses and optimistic initialization to its logical extreme,
in which all incompletely explored choices are assumed maximally rewarding
and optimal paths are computed to test them. The E3algorithm of Kearns and

Singh (2002) and the R-max algorithm of Brafman and Tennenholtz (2003) are
guaranteed to ﬁnd a near-optimal solution in time polynomial in the number
of states and actions. This is usually too slow for practical algorithms but is
probably the best that can be done in the worst case.
8.4 Prioritized sweeping was developed simultaneously and independently by Moore
and Atkeson (1993) and Peng and Williams (1993). The results in the box on
page 170 are due to Peng and Williams (1993). The results in the box on page 171

are due to Moore and Atkeson. Key subsequent work in this area includes that
by McMahan and Gordon (2005) and by van Seijen and Sutton (2013).
8.5 This section was strongly inﬂuenced by the experiments of Singh (1993).
8.6–7 Trajectory sampling has implicitly been a part of reinforcement learning from
the outset, but it was most explicitly emphasized by Barto, Bradtke, and Singh
(1995) in their introduction of RTDP. They recognized that Korf’s (1990) learning

Bibliographical and Historical Remarks 193
real-time A* (LRTA*) algorithm is an asynchronous DP algorithm that applies
to stochastic problems as well as the deterministic problems on which Korf
focused. Beyond LRTA*, RTDP includes the option of updating the values of
many states in the time intervals between the execution of actions. Barto et
al. (1995) proved the convergence result described here by combining Korf’s (1990)
convergence proof for LRTA* with the result of Bertsekas (1982) (also Bertsekas
and Tsitsiklis, 1989) ensuring convergence of asynchronous DP for stochastic
shortest path problems in the undiscounted case. Combining model-learning
with RTDP is called Adaptive RTDP, also presented by Barto et al. (1995) and
discussed by Barto (2011).
8.9 For further reading on heuristic search, the reader is encouraged to consult texts
and surveys such as those by Russell and Norvig (2009) and Korf (1988). Peng
and Williams (1993) explored a forward focusing of updates much as is suggested
in this section.

8.10 Abramson’s (1990) expected-outcome model is a rollout algorithm applied to two-
person games in which the play of both simulated players is random. He argued
that even with random play, it is a “powerful heuristic” that is “precise, accurate,
easily estimable, eciently calculable, and domain-independent.” Tesauro and
Galperin (1997) demonstrated the e↵ectiveness of rollout algorithms for improving
the play of backgammon programs, adopting the term “rollout” from its use
in evaluating backgammon positions by playing out positions with di↵erent

randomly generating sequences of dice rolls. Bertsekas, Tsitsiklis, and Wu (1997)
examine rollout algorithms applied to combinatorial optimization problems, and
Bertsekas (2013) surveys their use in discrete deterministic optimization problems,
remarking that they are “often surprisingly e↵ective.”
8.11 The central ideas of MCTS were introduced by Coulom (2006) and by Kocsis
and Szepesv´ ari (2006). They built upon previous research with Monte Carlo
planning algorithms as reviewed by these authors. Browne, Powley, Whitehouse,

Lucas, Cowling, Rohlfshagen, Tavener, Perez, Samothrakis, and Colton (2012)
is an excellent survey of MCTS methods and their applications. David Silver
contributed to the ideas and presentation in this section.

Part II:
Approximate Solution Methods
In the second part of the book we extend the tabular methods presented in the ﬁrst part
to apply to problems with arbitrarily large state spaces. In many of the tasks to which we
would like to apply reinforcement learning the state space is combinatorial and enormous;
the number of possible camera images, for example, is much larger than the number of
atoms in the universe. In such cases we cannot expect to ﬁnd an optimal policy or the
optimal value function even in the limit of inﬁnite time and data; our goal instead is to
ﬁnd a good approximate solution using limited computational resources. In this part of
the book we explore such approximate solution methods.
The problem with large state spaces is not just the memory needed for large tables,
but the time and data needed to ﬁll them accurately. In many of our target tasks, almost
every state encountered will never have been seen before. To make sensible decisions in

such states it is necessary to generalize from previous encounters with di↵erent states
that are in some sense similar to the current one. In other words, the key issue is that of
generalization . How can experience with a limited subset of the state space be usefully
generalized to produce a good approximation over a much larger subset?
Fortunately, generalization from examples has already been extensively studied, and
we do not need to invent totally new methods for use in reinforcement learning. To some

extent we need only combine reinforcement learning methods with existing generalization
methods. The kind of generalization we require is often called function approximation
because it takes examples from a desired function (e.g., a value function) and attempts
to generalize from them to construct an approximation of the entire function. Function
approximation is an instance of supervised learning , the primary topic studied in machine

learning, artiﬁcial neural networks, pattern recognition, and statistical curve ﬁtting. In
theory, any of the methods studied in these ﬁelds can be used in the role of function
approximator within reinforcement learning algorithms, although in practice some ﬁt
more easily into this role than others.
Reinforcement learning with function approximation involves a number of new issues
that do not normally arise in conventional supervised learning, such as nonstationarity,
bootstrapping, and delayed targets. We introduce these and other issues successively over

the ﬁve chapters of this part. Initially we restrict attention to on-policy training, treating
in Chapter 9 the prediction case, in which the policy is given and only its value function
is approximated, and then in Chapter 10 the control case, in which an approximation to
the optimal policy is found. The challenging problem of o↵-policy learning with function
195

approximation is treated in Chapter 11. In each of these three chapters we will have
to return to ﬁrst principles and re-examine the objectives of the learning to take into
account function approximation. Chapter 12 introduces and analyzes the algorithmic
mechanism of eligibility traces , which dramatically improves the computational properties
of multi-step reinforcement learning methods in many cases. The ﬁnal chapter of this
part explores a di↵erent approach to control, policy-gradient methods , which approximate
the optimal policy directly and need never form an approximate value function (although
they may be much more ecient if they do approximate a value function as well the
policy).

Chapter 9
On-policy Prediction with
Approximation
In this chapter, we begin our study of function approximation in reinforcement learning
by considering its use in estimating the state-value function from on-policy data, that is,
in approximating v⇡from experience generated using a known policy ⇡.T h en o v e l t yi n
this chapter is that the approximate value function is represented not as a table but as a
parameterized functional form with weight vector w2Rd.W ew i l lw r i t e ˆv(s,w)⇡v⇡(s)
for the approximate value of state sgiven weight vector w. For example, ˆvmight be
a linear function in features of the state, with wthe vector of feature weights. More
generally, ˆvmight be the function computed by a multi-layer artiﬁcial neural network,
with wthe vector of connection weights in all the layers. By adjusting the weights, any
of a wide range of di↵erent functions can be implemented by the network. Or ˆvmight be
the function computed by a decision tree, where wis all the numbers deﬁning the split

points and leaf values of the tree. Typically, the number of weights (the dimensionality of
w) is much less than the number of states ( d⌧|S|), and changing one weight changes the
estimated value of many states. Consequently, when a single state is updated, the change
generalizes from that state to a↵ect the values of many other states. Such generalization
makes the learning potentially more powerful but also potentially more dicult to manage
and understand.
Perhaps surprisingly, extending reinforcement learning to function approximation also

makes it applicable to partially observable problems, in which the full state is not available
to the agent. If the parameterized function form for ˆvdoes not allow the estimated
value to depend on certain aspects of the state, then it is just as if those aspects are
unobservable. In fact, all the theoretical results for methods using function approximation
presented in this part of the book apply equally well to cases of partial observability.
What function approximation can’t do, however, is augment the state representation

with memories of past observations. Some such possible further extensions are discussed
brieﬂy in Section 17.3.
197

198 Chapter 9: On-policy Prediction with Approximation
9.1 Value-function Approximation
All of the prediction methods covered in this book have been described as updates to an
estimated value function that shift its value at particular states toward a “backed-up value,”
orupdate target , for that state. Let us refer to an individual update by the notation s7!u,
where sis the state updated and uis the update target that s’s estimated value is shifted
toward. For example, the Monte Carlo update for value prediction is St7!Gt, the TD(0)
update is St7!Rt+1+ˆv(St+1,wt), and the n-step TD update is St7!Gt:t+n.I n t h e D P
(dynamic programming) policy-evaluation update, s7!E⇡[Rt+1+ˆv(St+1,wt)|St=s],
an arbitrary state sis updated, whereas in the other cases the state encountered in actual
experience, St, is updated.
It is natural to interpret each update as specifying an example of the desired input–
output behavior of the value function. In a sense, the update s7!umeans that the

estimated value for state sshould be more like the update target u. Up to now, the
actual update has been trivial: the table entry for s’s estimated value has simply been
shifted a fraction of the way toward u, and the estimated values of all other states
were left unchanged. Now we permit arbitrarily complex and sophisticated methods to
implement the update, and updating at sgeneralizes so that the estimated values of
many other states are changed as well. Machine learning methods that learn to mimic

input–output examples in this way are called supervised learning methods, and when the
outputs are numbers, like u, the process is often called function approximation . Function
approximation methods expect to receive examples of the desired input–output behavior
of the function they are trying to approximate. We use these methods for value prediction
simply by passing to them the s7!gof each update as a training example. We then
interpret the approximate function they produce as an estimated value function.

Viewing each update as a conventional training example in this way enables us to use
any of a wide range of existing function approximation methods for value prediction. In
principle, we can use any method for supervised learning from examples, including artiﬁcial
neural networks, decision trees, and various kinds of multivariate regression. However,
not all function approximation methods are equally well suited for use in reinforcement
learning. The most sophisticated artiﬁcial neural network and statistical methods all

assume a static training set over which multiple passes are made. In reinforcement
learning, however, it is important that learning be able to occur online, while the agent
interacts with its environment or with a model of its environment. To do this requires
methods that are able to learn eciently from incrementally acquired data. In addition,
reinforcement learning generally requires function approximation methods able to handle
nonstationary target functions (target functions that change over time). For example,

in control methods based on GPI (generalized policy iteration) we often seek to learn
q⇡while ⇡changes. Even if the policy remains the same, the target values of training
examples are nonstationary if they are generated by bootstrapping methods (DP and TD
learning). Methods that cannot easily handle such nonstationarity are less suitable for

9.2. The Prediction Objective ( VE) 199
9.2 The Prediction Objective ( VE)
Up to now we have not speciﬁed an explicit objective for prediction. In the tabular case
a continuous measure of prediction quality was not necessary because the learned value
function could come to equal the true value function exactly. Moreover, the learned
values at each state were decoupled—an update at one state a↵ected no other. But with
genuine approximation, an update at one state a↵ects many others, and it is not possible
to get the values of all states exactly correct. By assumption we have far more states
than weights, so making one state’s estimate more accurate invariably means making
others’ less accurate. We are obligated then to say which states we care most about. We
must specify a state distribution µ(s)0,P
sµ(s) = 1, representing how much we care
about the error in each state s. By the error in a state swe mean the square of the
di↵erence between the approximate value ˆv(s,w) and the true value v⇡(s). Weighting

this over the state space by µ, we obtain a natural objective function, the Mean Squared
Value Error , denoted VE:
VE(w).=X
s2Sµ(s)h
v⇡(s)ˆv(s,w)i2
. (9.1)
The square root of this measure, the root VE, gives a rough measure of how much the
approximate values di↵er from the true values and is often used in plots. Often µ(s)i s
chosen to be the fraction of time spent in s. Under on-policy training this is called the
on-policy distribution ; we focus entirely on this case in this chapter. In continuing tasks,

VE(w).=X
s2Sµ(s)h
. (9.1)
the on-policy distribution is the stationary distribution under ⇡.
The on-policy distribution in episodic tasks
In an episodic task, the on-policy distribution is a little di↵erent in that it depends
on how the initial states of episodes are chosen. Let h(s) denote the probability
that an episode begins in each state s, and let ⌘(s) denote the number of time
steps spent, on average, in state sin a single episode. Time is spent in a state s
if episodes start in s, or if transitions are made into sfrom a preceding state ¯sin
which time is spent:
⌘(s)=h(s)+X
¯s⌘(¯s)X

¯s⌘(¯s)X
a⇡(a|¯s)p(s|¯s, a),for all s2S. (9.2)
This system of equations can be solved for the expected number of visits ⌘(s). The
on-policy distribution is then the fraction of time spent in each state normalized to
sum to one:
µ(s)=⌘(s)P
s0⌘(s0),for all s2S. (9.3)
This is the natural choice without discounting. If there is discounting ( <1) it
should be treated as a form of termination, which can be done simply by including
a factor of in the second term of (9.2).

200 Chapter 9: On-policy Prediction with Approximation
The two cases, continuing and episodic, behave similarly, but with approximation they
must be treated separately in formal analyses, as we will see repeatedly in this part of
the book. This completes the speciﬁcation of the learning objective.
But it is not completely clear that the VEis the right performance objective for
reinforcement learning. Remember that our ultimate purpose—the reason we are learning
a value function—is to ﬁnd a better policy. The best value function for this purpose is
not necessarily the best for minimizing VE. Nevertheless, it is not yet clear what a more
useful alternative goal for value prediction might be. For now, we will focus on VE.
An ideal goal in terms of VEwould be to ﬁnd a global optimum , a weight vector w⇤
for which VE(w⇤)VE(w) for all possible w. Reaching this goal is sometimes possible
for simple function approximators such as linear ones, but is rarely possible for complex

function approximators such as artiﬁcial neural networks and decision trees. Short of
this, complex function approximators may seek to converge instead to a local optimum ,
a weight vector w⇤for which VE(w⇤)VE(w) for all win some neighborhood of w⇤.
Although this guarantee is only slightly reassuring, it is typically the best that can be
said for nonlinear function approximators, and often it is enough. Still, for many cases of
interest in reinforcement learning there is no guarantee of convergence to an optimum, or

even to within a bounded distance of an optimum. Some methods may in fact diverge,
with their VE approaching inﬁnity in the limit.
In the last two sections we outlined a framework for combining a wide range of
reinforcement learning methods for value prediction with a wide range of function
approximation methods, using the updates of the former to generate training examples
for the latter. We also described a VEperformance measure which these methods may
aspire to minimize. The range of possible function approximation methods is far too

large to cover all, and anyway too little is known about most of them to make a reliable
evaluation or recommendation. Of necessity, we consider only a few possibilities. In
the rest of this chapter we focus on function approximation methods based on gradient
principles, and on linear gradient-descent methods in particular. We focus on these
methods in part because we consider them to be particularly promising and because they
reveal key theoretical issues, but also because they are simple and our space is limited.

9.3 Stochastic-gradient and Semi-gradient Methods
We now develop in detail one class of learning methods for function approximation in
value prediction, those based on stochastic gradient descent (SGD). SGD methods are
among the most widely used of all function approximation methods and are particularly
well suited to online reinforcement learning.
In gradient-descent methods, the weight vector is a column vector with a ﬁxed number
of real valued components, w.=(w1,w2,...,w d)>,1and the approximate value function

ˆv(s,w) is a di↵erentiable function of wfor all s2S. We will be updating wat each of
a series of discrete time steps, t=0,1,2,3,..., so we will need a notation wtfor the
1The>denotes transpose, needed here to turn the horizontal row vector in the text into a vertical
column vector; in this book vectors are generally taken to be column vectors unless explicitly written out
horizontally or transposed.

9.3. Stochastic-gradient and Semi-gradient Methods 201
weight vector at each step. For now, let us assume that, on each step, we observe a new
example St7!v⇡(St) consisting of a (possibly randomly selected) state Stand its true
value under the policy. These states might be successive states from an interaction with
the environment, but for now we do not assume so. Even though we are given the exact,
correct values, v⇡(St) for each St, there is still a dicult problem because our function
approximator has limited resources and thus limited resolution. In particular, there is
generally no wthat gets all the states, or even all the examples, exactly correct. In
addition, we must generalize to all the other states that have not appeared in examples.
We assume that states appear in examples with the same distribution, µ,o v e rw h i c h
we are trying to minimize the VEas given by (9.1). A good strategy in this case is
to try to minimize error on the observed examples. Stochastic gradient-descent (SGD)

methods do this by adjusting the weight vector after each example by a small amount in
the direction that would most reduce the error on that example:
wt+1.=wt1
2↵rh
v⇡(St)ˆv(St,wt)i2
(9.4)
=wt+↵h
v⇡(St)ˆv(St,wt)i
rˆv(St,wt), (9.5)
where ↵is a positive step-size parameter, and rf(w), for any scalar expression f(w)
that is a function of a vector (here w), denotes the column vector of partial derivatives
of the expression with respect to the components of the vector:
rf(w).=✓@f(w)
@w1,@f(w)
@w2,...,@f(w)
@wd◆>
. (9.6)

wt+1.=wt1
2↵rh
(9.4)
=wt+↵h
@w1,@f(w)
@wd◆>
. (9.6)
This derivative vector is the gradient offwith respect to w. SGD methods are “gradient
descent” methods because the overall step in wtis proportional to the negative gradient
of the example’s squared error (9.4) . This is the direction in which the error falls most
rapidly. Gradient descent methods are called “stochastic” when the update is done, as
here, on only a single example, which might have been selected stochastically. Over many
examples, making small steps, the overall e↵ect is to minimize an average performance
measure such as the VE.

It may not be immediately apparent why SGD takes only a small step in the direction
of the gradient. Could we not move all the way in this direction and completely eliminate
the error on the example? In many cases this could be done, but usually it is not desirable.
Remember that we do not seek or expect to ﬁnd a value function that has zero error for
all states, but only an approximation that balances the errors in di↵erent states. If we
completely corrected each example in one step, then we would not ﬁnd such a balance.

In fact, the convergence results for SGD methods assume that ↵decreases over time. If
it decreases in such a way as to satisfy the standard stochastic approximation conditions
(2.7), then the SGD method (9.5) is guaranteed to converge to a local optimum.
We turn now to the case in which the target output, here denoted Ut2R, of the tth
training example, St7!Ut, is not the true value, v⇡(St), but some, possibly random,
approximation to it. For example, Utmight be a noise-corrupted version of v⇡(St), or it

might be one of the bootstrapping targets using ˆvmentioned in the previous section. In

202 Chapter 9: On-policy Prediction with Approximation
these cases we cannot perform the exact update (9.5) because v⇡(St)i su n k n o w n ,b u t
we can approximate it by substituting Utin place of v⇡(St). This yields the following
general SGD method for state-value prediction:
wt+1.=wt+↵h
Utˆv(St,wt)i
rˆv(St,wt). (9.7)
IfUtis an unbiased estimate, that is, if E[Ut|St=s]=v⇡(St), for each t,t h e n wtis
guaranteed to converge to a local optimum under the usual stochastic approximation
conditions (2.7) for decreasing ↵.
For example, suppose the states in the examples are the states generated by interaction
(or simulated interaction) with the environment using policy ⇡. Because the true value of
a state is the expected value of the return following it, the Monte Carlo target Ut.=Gtis
by deﬁnition an unbiased estimate of v⇡(St). With this choice, the general SGD method
(9.7) converges to a locally optimal approximation to v⇡(St). Thus, the gradient-descent

version of Monte Carlo state-value prediction is guaranteed to ﬁnd a locally optimal
solution. Pseudocode for a complete algorithm is shown in the box below.
Gradient Monte Carlo Algorithm for Estimating ˆv⇡v⇡
Input: a di↵erentiable function ˆ v:S⇥Rd!R
Algorithm parameter: step size ↵>0
Initialize value-function weights w2Rdarbitrarily (e.g., w=0)
Generate an episode S0,A0,R1,S1,A1,...,R T,STusing ⇡
Loop for each step of episode, t=0,1,...,T 1:
w w+↵⇥
Gtˆv(St,w)⇤

w w+↵⇥
rˆv(St,w)
One does not obtain the same guarantees if a bootstrapping estimate of v⇡(St)i su s e d
as the target Utin(9.7) . Bootstrapping targets such as n-step returns Gt:t+nor the DP
targetP
a,s0,r⇡(a|St)p(s0,r|St,a)[r+ˆv(s0,wt)] all depend on the current value of the
weight vector wt, which implies that they will be biased and that they will not produce a
true gradient-descent method. One way to look at this is that the key step from (9.4)
to(9.5) relies on the target being independent of wt. This step would not be valid if

targetP
a bootstrapping estimate were used in place of v⇡(St). Bootstrapping methods are not
in fact instances of true gradient descent (Barnard, 1993). They take into account the
e↵ect of changing the weight vector wton the estimate, but ignore its e↵ect on the target.
They include only a part of the gradient and, accordingly, we call them semi-gradient
methods .
Although semi-gradient (bootstrapping) methods do not converge as robustly as
gradient methods, they do converge reliably in important cases such as the linear case

methods .
discussed in the next section. Moreover, they o↵er important advantages that make them
often clearly preferred. One reason for this is that they typically enable signiﬁcantly faster
learning, as we have seen in Chapters 6 and 7. Another is that they enable learning to

9.3. Stochastic-gradient and Semi-gradient Methods 203
be continual and online, without waiting for the end of an episode. This enables them to
be used on continuing problems and provides computational advantages. A prototypical
semi-gradient method is semi-gradient TD(0), which uses Ut.=Rt+1+ˆv(St+1,w) as its
target. Complete pseudocode for this method is given in the box below.
Semi-gradient TD(0) for estimating ˆv⇡v⇡
Input: a di↵erentiable function ˆ v:S+⇥Rd!Rsuch that ˆ v(terminal ,·)=0
Choose A⇠⇡(·|S)
Take action A, observe R, S0
w w+↵⇥
R+ˆv(S0,w)ˆv(S,w)⇤
rˆv(S,w)
S S0
State aggregation is a simple form of generalizing function approximation in which
states are grouped together, with one estimated value (one component of the weight

w w+↵⇥
rˆv(S,w)
S S0
vector w) for each group. The value of a state is estimated as its group’s component,
and when the state is updated, that component alone is updated. State aggregation
is a special case of SGD (9.7) in which the gradient, rˆv(St,wt), is 1 for St’s group’s
component and 0 for the other components.
Example 9.1: State Aggregation on the 1000-state Random Walk Consider a
1000-state version of the random walk task (Examples 6.2 and 7.1 on pages 125 and

144). The states are numbered from 1 to 1000, left to right, and all episodes begin near
the center, in state 500. State transitions are from the current state to one of the 100
neighboring states to its left, or to one of the 100 neighboring states to its right, all with
equal probability. Of course, if the current state is near an edge, then there may be fewer
than 100 neighbors on that side of it. In this case, all the probability that would have
gone into those missing neighbors goes into the probability of terminating on that side

(thus, state 1 has a 0.5 chance of terminating on the left, and state 950 has a 0.25 chance
of terminating on the right). As usual, termination on the left produces a reward of
1, and termination on the right produces a reward of +1. All other transitions have a
reward of zero. We use this task as a running example throughout this section.
Figure 9.1 shows the true value function v⇡for this task. It is nearly a straight line,
but curving slightly toward the horizontal for the last 100 states at each end. Also shown

is the ﬁnal approximate value function learned by the gradient Monte-Carlo algorithm
with state aggregation after 100,000 episodes with a step size of ↵=2⇥105. For the
state aggregation, the 1000 states were partitioned into 10 groups of 100 states each (i.e.,
states 1–100 were one group, states 101–200 were another, and so on). The staircase e↵ect

204 Chapter 9: On-policy Prediction with Approximation
0StateValuescale    True valuev⇡    Approximate MC valueˆv    State distribution         0.00170.0137Distributionscale100010-11µ
Figure 9.1: Function approximation by state aggregation on the 1000-state random walk task,
using the gradient Monte Carlo algorithm (page 202).
shown in the ﬁgure is typical of state aggregation; within each group, the approximate
value is constant, and it changes abruptly from one group to the next. These approximate
values are close to the global minimum of the VE (9.1).
Some of the details of the approximate values are best appreciated by reference to
the state distribution µfor this task, shown in the lower portion of the ﬁgure with a
right-side scale. State 500, in the center, is the ﬁrst state of every episode, but is rarely
visited again. On average, about 1.37% of the time steps are spent in the start state.
The states reachable in one step from the start state are the second most visited, with

about 0.17% of the time steps being spent in each of them. From there µfalls o↵ almost
linearly, reaching about 0.0147% at the extreme states 1 and 1000. The most visible
e↵ect of the distribution is on the leftmost groups, whose values are clearly shifted higher
than the unweighted average of the true values of states within the group, and on the
rightmost groups, whose values are clearly shifted lower. This is due to the states in
these areas having the greatest asymmetry in their weightings by µ. For example, in the

leftmost group, state 100 is weighted more than 3 times more strongly than state 1. Thus
the estimate for the group is biased toward the true value of state 100, which is higher
than the true value of state 1.
9.4 Linear Methods
One of the most important special cases of function approximation is that in which the
approximate function, ˆv(·,w), is a linear function of the weight vector, w. Corresponding
to every state s, there is a real-valued vector x(s).=(x1(s),x2(s),...,x d(s))>,w i t ht h e
same number of components as w. Linear methods approximate state-value function by

9.4. Linear Methods 205
the inner product between wandx(s):
ˆv(s,w).=w>x(s).=dX
i=1wixi(s). (9.8)
In this case the approximate value function is said to be linear in the weights , or simply
linear .
The vector x(s) is called a feature vector representing state s. Each component xi(s)
ofx(s) is the value of a function xi:S!R. We think of a feature as the entirety of one
of these functions, and we call its value for a state safeature of s. For linear methods,
features are basis functions because they form a linear basis for the set of approximate
functions. Constructing d-dimensional feature vectors to represent states is the same as
selecting a set of dbasis functions. Features may be deﬁned in many di↵erent ways; we
cover a few possibilities in the next sections.
It is natural to use SGD updates with linear function approximation. The gradient of
the approximate value function with respect to win this case is
rˆv(s,w)=x(s).
Thus, in the linear case the general SGD update (9.7) reduces to a particularly simple

form:
x(St).
Because it is so simple, the linear SGD case is one of the most favorable for mathematical
analysis. Almost all useful convergence results for learning systems of all kinds are for
linear (or simpler) function approximation methods.
In particular, in the linear case there is only one optimum (or, in degenerate cases,
one set of equally good optima), and thus any method that is guaranteed to converge to
or near a local optimum is automatically guaranteed to converge to or near the global

x(St).
optimum. For example, the gradient Monte Carlo algorithm presented in the previous
section converges to the global optimum of the VEunder linear function approximation
if↵is reduced over time according to the usual conditions.
The semi-gradient TD(0) algorithm presented in the previous section also converges
under linear function approximation, but this does not follow from general results on
SGD; a separate theorem is necessary. The weight vector converged to is also not the

Proof of Convergence of Linear TD(0)
What properties assure convergence of the linear TD(0) algorithm (9.9) ? Some
insight can be gained by rewriting (9.10) as
E[wt+1|wt]=(I↵A)wt+↵b. (9.13)
Note that the matrix Amultiplies the weight vector wtand not b; only Ais
important to convergence. To develop intuition, consider the special case in which
Ais a diagonal matrix. If any of the diagonal elements are negative, then the
corresponding diagonal element of I↵Awill be greater than one, and the
corresponding component of wtwill be ampliﬁed, which will lead to divergence if
continued. On the other hand, if the diagonal elements of Aare all positive, then
↵can be chosen smaller than one over the largest of them, such that I↵Ais
diagonal with all diagonal elements between 0 and 1. In this case the ﬁrst term
of the update tends to shrink wt, and stability is assured. In general, wtwill be
reduced toward zero whenever Aispositive deﬁnite , meaning y>Ay>0 for any

9.4. Linear Methods 207
transition from stos0under policy ⇡,Pis the |S|⇥|S|matrix of these probabilities,
Dis the |S|⇥|S|diagonal matrix with the µ(s) on its diagonal, and Xis the |S|⇥d
matrix with x(s) as its rows. From here it is clear that the inner matrix D(IP)
is key to determining the positive deﬁniteness of A.
For a key matrix of this form, positive deﬁniteness is assured if all of its columns
sum to a nonnegative number. This was shown by Sutton (1988, p. 27) based
on two previously established theorems. One theorem says that any matrix M
is positive deﬁnite if and only if the symmetric matrix S=M+M>is positive
deﬁnite (Sutton 1988, appendix). The second theorem says that any symmetric
real matrix Sis positive deﬁnite if all of its diagonal entries are positive and greater
than the sum of the absolute values of the corresponding o↵-diagonal entries (Varga
1962, p. 23). For our key matrix, D(IP), the diagonal entries are positive

and the o↵-diagonal entries are negative, so all we have to show is that each row
sum plus the corresponding column sum is positive. The row sums are all positive
because Pis a stochastic matrix and <1. Thus it only remains to show that
the column sums are nonnegative. Note that the row vector of the column sums
of any matrix Mcan be written as 1>M,w h e r e 1is the column vector with all
components equal to 1. Let µdenote the |S|-vector of the µ(s), where µ=P>µby

208 Chapter 9: On-policy Prediction with Approximation
A bound analogous to (9.14) applies to other on-policy bootstrapping methods as well.
For example, linear semi-gradient DP (Eq. 9.7 with Ut.=P
a⇡(a|St)P
s0,rp(s0,r|St,a)[r+
ˆv(s0,wt)]) with updates according to the on-policy distribution will also converge to
the TD ﬁxed point. One-step semi-gradient action-value methods, such as semi-gradient
Sarsa(0) covered in the next chapter converge to an analogous ﬁxed point and an analogous
bound. For episodic tasks, there is a slightly di↵erent but related bound (see Bertsekas
and Tsitsiklis, 1996). There are also a few technical conditions on the rewards, features,
and decrease in the step-size parameter, which we have omitted here. The full details
can be found in the original paper (Tsitsiklis and Van Roy, 1997).
Critical to the these convergence results is that states are updated according to the
on-policy distribution. For other update distributions, bootstrapping methods using

function approximation may actually diverge to inﬁnity. Examples of this and a discussion
of possible solution methods are given in Chapter 11.
Example 9.2: Bootstrapping on the 1000-state Random Walk State aggregation
is a special case of linear function approximation, so let’s return to the 1000-state random
walk to illustrate some of the observations made in this chapter. The left panel of
Figure 9.2 shows the ﬁnal value function learned by the semi-gradient TD(0) algorithm

(page 203) using the same state aggregation as in Example 9.1. We see that the near-
asymptotic TD approximation is indeed farther from the true values than the Monte
Carlo approximation shown in Figure 9.1.
Nevertheless, TD methods retain large potential advantages in learning rate, and
generalize Monte Carlo methods, as we investigated fully with n-step TD methods in
Chapter 7. The right panel of Figure 9.2 shows results with an n-step semi-gradient
TD method using state aggregation on the 1000-state random walk that are strikingly

similar to those we obtained earlier with tabular methods and the 19-state random
walk (Figure 7.2). To obtain such quantitatively similar results we switched the state
aggregation to 20 groups of 50 states each. The 20 groups were then quantitatively close
0.550.50.450.350.30.250.40.40.200.80.61
↵AverageRMS errorover 1000 statesand ﬁrst 10 episodesn=1n=2n=4n=8n=16n=32n=64128512256State    True valuev⇡    Approximate TD value10-111000ˆv
Figure 9.2: Bootstrapping with state aggregation on the 1000-state random walk task. Left:

Asymptotic values of semi-gradient TD are worse than the asymptotic Monte Carlo values in
Figure 9.1. Right : Performance of n-step methods with state-aggregation are strikingly similar
to those with tabular representations (cf. Figure 7.2). These data are averages over 100 runs.

9.5. Feature Construction for Linear Methods 209
to the 19 states of the tabular problem. In particular, recall that state transitions were
up to 100 states to the left or right. A typical transition would then be of 50 states to
the right or left, which is quantitatively analogous to the single-state state transitions of
the 19-state tabular system. To complete the match, we use here the same performance
measure—an unweighted average of the RMS error over all states and over the ﬁrst
10 episodes—rather than a VEobjective as is otherwise more appropriate when using
function approximation.
The semi-gradient n-step TD algorithm used in the example above is the natural
extension of the tabular n-step TD algorithm presented in Chapter 7 to semi-gradient
function approximation. Pseudocode is given in the box below.
n-step semi-gradient TD for estimating ˆv⇡v⇡

Algorithm parameters: step size ↵>0, a positive integer n
Initialize value-function weights warbitrarily (e.g., w=0)
All store and access operations ( StandRt) can take their index mod n+1
T 1
| Take an action according to ⇡(·|St)
| IfSt+1is terminal, then T t+1
|⌧ tn+1 ( ⌧is the time whose state’s estimate is being updated)
|If⌧0:
| G Pmin( ⌧+n,T)
i=⌧+1i⌧1Ri

210 Chapter 9: On-policy Prediction with Approximation
9.5 Feature Construction for Linear Methods
Linear methods are interesting because of their convergence guarantees, but also because
in practice they can be very ecient in terms of both data and computation. Whether or
not this is so depends critically on how the states are represented in terms of features,
which we investigate in this large section. Choosing features appropriate to the task is
an important way of adding prior domain knowledge to reinforcement learning systems.
Intuitively, the features should correspond to the aspects of the state space along which
generalization may be appropriate. If we are valuing geometric objects, for example,
we might want to have features for each possible shape, color, size, or function. If we
are valuing states of a mobile robot, then we might want to have features for locations,
degrees of remaining battery power, recent sonar readings, and so on.

A limitation of the linear form is that it cannot take into account any interactions
between features, such as the presence of feature ibeing good only in the absence of
feature j. For example, in the pole-balancing task (Example 3.4) high angular velocity
can be either good or bad depending on the angle. If the angle is high, then high angular
velocity means an imminent danger of falling—a bad state—whereas if the angle is low,

then high angular velocity means the pole is righting itself—a good state. A linear value
function could not represent this if its features coded separately for the angle and the
angular velocity. It needs instead, or in addition, features for combinations of these two
underlying state dimensions. In the following subsections we consider a variety of general
ways of doing this.
9.5.1 Polynomials
The states of many problems are initially expressed as numbers, such as positions and

velocities in the pole-balancing task (Example 3.4), the number of cars in each lot in the
Jack’s car rental problem (Example 4.2), or the gambler’s capital in the gambler problem
(Example 4.3). In these types of problems, function approximation for reinforcement
learning has much in common with the familiar tasks of interpolation and regression.
Various families of features commonly used for interpolation and regression can also be
used in reinforcement learning. Polynomials make up one of the simplest families of

features used for interpolation and regression. While the basic polynomial features we
discuss here do not work as well as other types of features in reinforcement learning, they
serve as a good introduction because they are simple and familiar.
As an example, suppose a reinforcement learning problem has states with two numerical
dimensions. For a single representative state s,l e ti t st w on u m b e r sb e s12Rands22R.
You might choose to represent ssimply by its two state dimensions, so that x(s)=
(s1,s2)>, but then you would not be able to take into account any interactions between

these dimensions. In addition, if both s1ands2were zero, then the approximate value
would have to also be zero. Both limitations can be overcome by instead representing sby
the four-dimensional feature vector x(s)=( 1 ,s1,s2,s1s2)>. The initial 1 feature allows
the representation of ane functions in the original state numbers, and the ﬁnal product
feature, s1s2, enables interactions to be taken into account. Or you might choose to use
higher-dimensional feature vectors like x(s)=( 1 ,s1,s2,s1s2,s2
1,s2
2,s1s2

1,s2
2,s1s2
2,s2
1s2,s2
1s2
2)>to

9.5. Feature Construction for Linear Methods 211
take more complex interactions into account. Such feature vectors enable approximations
as arbitrary quadratic functions of the state numbers—even though the approximation is
still linear in the weights that have to be learned. Generalizing this example from two
toknumbers, we can represent highly-complex interactions among a problem’s state
Suppose each state scorresponds to knumbers, s1,s2,. . . , sk, with each si2R.
For this k-dimensional state space, each order- npolynomial-basis feature xican be
written as
xi(s)=⇧k
j=1sci,j
j, (9.17)
where each ci,jis an integer in the set {0,1,...,n }for an integer n0. These
features make up the order- npolynomial basis for dimension k, which contains
(n+ 1)kdi↵erent features.
Higher-order polynomial bases allow for more accurate approximations of more compli-
cated functions. But because the number of features in an order- npolynomial basis grows

written as
xi(s)=⇧k
j=1sci,j
j, (9.17)
exponentially with the dimension kof the natural state space (if n>0), it is generally
necessary to select a subset of them for function approximation. This can be done using
prior beliefs about the nature of the function to be approximated, and some automated
selection methods developed for polynomial regression can be adapted to deal with the
incremental and nonstationary nature of reinforcement learning.
Exercise 9.2 Why does (9.17) deﬁne ( n+ 1)kdistinct features for dimension k? ⇤

Exercise 9.3 What nandci,jproduce the feature vectors x(s)=( 1 ,s1,s2,s1s2,s2
1,s2
2,
s1s2
2,s2
1s2,s2
1s2
2)>? ⇤
9.5.2 Fourier Basis
Another linear function approximation method is based on the time-honored Fourier
series, which expresses periodic functions as weighted sums of sine and cosine basis
functions (features) of di↵erent frequencies. (A function fis periodic if f(x)=f(x+⌧)
for all xand some period ⌧.) The Fourier series and the more general Fourier transform

1,s2
2,
s1s2
2,s2
1s2,s2
1s2
2)>? ⇤
are widely used in applied sciences in part because if a function to be approximated is
known, then the basis function weights are given by simple formulae and, further, with
enough basis functions essentially any function can be approximated as accurately as
desired. In reinforcement learning, where the functions to be approximated are unknown,
Fourier basis functions are of interest because they are easy to use and can perform well
in a range of reinforcement learning problems.

First consider the one-dimensional case. The usual Fourier series representation of a
function of one dimension having period ⌧represents the function as a linear combination
of sine and cosine functions that are each periodic with periods that evenly divide ⌧(in
other words, whose frequencies are integer multiples of a fundamental frequency 1 /⌧).
But if you are interested in approximating an aperiodic function deﬁned over a bounded
interval, then you can use these Fourier basis features with ⌧set to the length the interval.

212 Chapter 9: On-policy Prediction with Approximation
The function of interest is then just one period of the periodic linear combination of the
sine and cosine features.
Furthermore, if you set ⌧to twice the length of the interval of interest and restrict
attention to the approximation over the half interval [0 ,⌧/2], then you can use just the
cosine features. This is possible because you can represent any even function, that is,
any function that is symmetric about the origin, with just the cosine basis. So any
function over the half-period [0 ,⌧/2] can be approximated as closely as desired with
enough cosine features. (Saying “any function” is not exactly correct because the function
has to be mathematically well-behaved, but we skip this technicality here.) Alternatively,
it is possible to use just sine features, linear combinations of which are always odd
functions, that is functions that are anti-symmetric about the origin. But it is generally

better to keep just the cosine features because “half-even” functions tend to be easier to
approximate than “half-odd” functions because the latter are often discontinuous at the
origin. Of course, this does not rule out using both sine and cosine features to approximate
over the interval [0 ,⌧/2], which might be advantageous in some circumstances.
Following this logic and letting ⌧= 2 so that the features are deﬁned over the half- ⌧
interval [0 ,1], the one-dimensional order- nFourier cosine basis consists of the n+1
features
xi(s) = cos( i⇡s),s2[0,1],

features
fori=0,...,n . Figure 9.3 shows one-dimensional Fourier cosine features xi, for i=
1,2,3,4;x0is a constant function.

features
00.10.20.30.40.50.60.70.80.91−1−0.8−0.6−0.4−0.200.20.40.60.81Univariate Fourier Basis Function k=11-11000.10.20.30.40.50.60.70.80.91−1−0.8−0.6−0.4−0.200.20.40.60.81Univariate Fourier Basis Function k=21-11000.10.20.30.40.50.60.70.80.91−1−0.8−0.6−0.4−0.200.20.40.60.81Univariate Fourier Basis Function k=31-11000.10.20.30.40.50.60.70.80.91−1−0.8−0.6−0.4−0.200.20.40.60.81Univariate Fourier Basis Function k=41-110Figure 9.3: One-dimensional Fourier cosine-basis features xi,i=1,2,3,4, for approximating

functions over the interval [0 ,1]. After Konidaris et al. (2011).
This same reasoning applies to the Fourier cosine series approximation in the multi-
dimensional case as described in the box below.
Suppose each state scorresponds to a vector of knumbers, s=(s1,s2,. . . ,s k)>,
with each si2[0,1]. The ith feature in the order- nFourier cosine basis can then
be written
xi(s) = cos
⇡s>ci
, (9.18)
where ci=(ci
1,...,ci
k)>,w i t h ci
j2{0,...,n }forj=1,...,k andi=0,..., (n+1)k.

be written
xi(s) = cos
⇡s>ci
, (9.18)
1,...,ci
This deﬁnes a feature for each of the ( n+ 1)kpossible integer vectors ci.T h ei n n e r

9.5. Feature Construction for Linear Methods 213
product s>cihas the e↵ect of assigning an integer in {0,...,n }to each dimension
ofs. As in the one-dimensional case, this integer determines the feature’s frequency
along that dimension. The features can of course be shifted and scaled to suit the
bounded state space of a particular application.
As an example, consider the k= 2 case in which s=(s1,s2)>, where each ci=(ci
1,ci
2)>.
Figure 9.4 shows a selection of six Fourier cosine features, each labeled by the vector ci
that deﬁnes it ( s1is the horizontal axis and ciis shown as a row vector with the index i
omitted). Any zero in cmeans the feature is constant along that state dimension. So if
c=( 0,0)>, the feature is constant over both dimensions; if c=(c1,0)>the feature is
constant over the second dimension and varies over the ﬁrst with frequency depending
onc1; and similarly, for c=( 0 ,c2)>.W h e n c=(c1,c2)>with neither cj= 0, the

feature varies along both dimensions and represents an interaction between the two state
variables. The values of c1andc2determine the frequency along each dimension, and
their ratio gives the direction of the interaction.
c = (0, 1)11001100c=( 0,1)>
c = (1, 0)11001100c=( 1,0)>
c = (1, 1)11001100c=( 1,1)>
c = (1, 5)11001100c=( 0,5)>
c = (2, 5)11001100c=( 2,5)>
1100c=( 5,2)>

Figure 9.4:A selection of six two-dimensional Fourier cosine features, each labeled by thevectorcithat deﬁnes it (s1is the horizontal axis, andciis shown with the indexiomitted).After Konidaris et al. (2011).When using Fourier cosine features with a learning algorithm such as (9.7), semi-gradient TD(0), or semi-gradient Sarsa, it may be helpful to use a di↵erent step-sizeparameter for each feature. If↵is the basic step-size parameter, then Konidaris, Osentoski,and Thomas (2011) suggest setting the step-size parameter for featurexito↵i=↵/p(ci1)2+···+(cik)2(except when eachcij= 0, in which case↵i=↵).Fourier cosine features with Sarsa can produce good performance compared to several

214 Chapter 9: On-policy Prediction with Approximation
other collections of basis functions, including polynomial and radial basis functions. Not
surprisingly, however, Fourier features have trouble with discontinuities because it is
dicult to avoid “ringing” around points of discontinuity unless very high frequency basis
functions are included.
The number of features in the order- nFourier basis grows exponentially with the
dimension of the state space, but if that dimension is small enough (e.g., k5), then
one can select nso that all of the order- nFourier features can be used. This makes the
selection of features more-or-less automatic. For high dimension state spaces, however, it
is necessary to select a subset of these features. This can be done using prior beliefs about
the nature of the function to be approximated, and some automated selection methods
can be adapted to deal with the incremental and nonstationary nature of reinforcement

learning. An advantage of Fourier basis features in this regard is that it is easy to select
features by setting the civectors to account for suspected interactions among the state
variables and by limiting the values in the cjvectors so that the approximation can
ﬁlter out high frequency components considered to be noise. On the other hand, because
Fourier features are non-zero over the entire state space (with the few zeros excepted),
they represent global properties of states, which can make it dicult to ﬁnd good ways
to represent local properties.

Figure 9.5 shows learning curves comparing the Fourier and polynomial bases on the
1000-state random walk example. In general, we do not recommend using polynomials
for online learning.2
.4.3.2.1005000EpisodesPolynomial basisFourier basispVE averagedover 30 runsFigure 9.5: Fourier basis vs polynomials on the 1000-state random walk. Shown are learning
curves for the gradient Monte Carlo method with Fourier and polynomial bases of order 5, 10,
and 20. The step-size parameters were roughly optimized for each case: ↵=0.0001 for the

polynomial basis and ↵=0.00005 for the Fourier basis. The performance measure (y-axis) is
the root mean squared value error (9.1).
2There are families of polynomials more complicated than those we have discussed, for example,
di↵erent families of orthogonal polynomials, and these might work better, but at present there is little
experience with them in reinforcement learning.

9.5. Feature Construction for Linear Methods 215
9.5.3 Coarse Codings0s
Figure 9.6: Coarse coding. Generaliza-
tion from state sto state s0depends on
the number of their features whose recep-
tive ﬁelds (in this case, circles) overlap.
These states have one feature in common,
so there will be slight generalization be-
tween them.Consider a task in which the natural repre-
sentation of the state set is a continuous two-
dimensional space. One kind of representation for
this case is made up of features corresponding to
circles in state space, as shown to the right. If
the state is inside a circle, then the corresponding
feature has the value 1 and is said to be present ;
otherwise the feature is 0 and is said to be absent .
This kind of 1–0-valued feature is called a binary
feature . Given a state, which binary features are
present indicate within which circles the state lies,
and thus coarsely code for its location. Represent-
ing a state with features that overlap in this way

(although they need not be circles or binary) is
known as coarse coding .
Assuming linear gradient-descent function ap-
proximation, consider the e↵ect of the size and
density of the circles. Corresponding to each cir-
cle is a single weight (a component of w) that is
a↵ected by learning. If we train at one state, a
point in the space, then the weights of all circles
intersecting that state will be a↵ected. Thus, by (9.8), the approximate value function
will be a↵ected at all states within the union of the circles, with a greater e↵ect the more

circles a point has “in common” with the state, as shown in Figure 9.6. If the circles are
small, then the generalization will be over a short distance, as in Figure 9.7 (left), whereas
if they are large, it will be over a large distance, as in Figure 9.7 (middle). Moreover,
a) Narrow generalizationb) Broad generalizationc) Asymmetric generalizationFigure 9.7: Generalization in linear function approximation methods is determined by the

sizes and shapes of the features’ receptive ﬁelds. All three of these cases have roughly the same
number and density of features.

216 Chapter 9: On-policy Prediction with Approximation
the shape of the features will determine the nature of the generalization. For example, if
they are not strictly circular, but are elongated in one direction, then generalization will
be similarly a↵ected, as in Figure 9.7 (right).
Features with large receptive ﬁelds give broad generalization, but might also seem to
limit the learned function to a coarse approximation, unable to make discriminations
much ﬁner than the width of the receptive ﬁelds. Happily, this is not the case. Initial
generalization from one point to another is indeed controlled by the size and shape of
the receptive ﬁelds, but acuity, the ﬁnest discrimination ultimately possible, is controlled
more by the total number of features.
Example 9.3: Coarseness of Coarse Coding This example illustrates the e↵ect on
learning of the size of the receptive ﬁelds in coarse coding. Linear function approximation
based on coarse coding and (9.7) was used to learn a one-dimensional square-wave function

(shown at the top of Figure 9.8). The values of this function were used as the targets, Ut.
With just one dimension, the receptive ﬁelds were intervals rather than circles. Learning
was repeated with three di↵erent sizes of the intervals: narrow, medium, and broad, as
shown at the bottom of the ﬁgure. All three cases had the same density of features,
about 50 over the extent of the function being learned. Training examples were generated
uniformly at random over this extent. The step-size parameter was ↵=0.2
n,w h e r e nis

the number of features that were present at one time. Figure 9.8 shows the functions
learned in all three cases over the course of learning. Note that the width of the features
had a strong e↵ect early in learning. With broad features, the generalization tended to be
broad; with narrow features, only the close neighbors of each trained point were changed,
causing the function learned to be more bumpy. However, the ﬁnal function learned was
a↵ected only slightly by the width of the features. Receptive ﬁeld shape tends to have a

strong e↵ect on generalization but little e↵ect on asymptotic solution quality.1040160640256010240Narrowfeaturesdesiredfunction
MediumfeaturesBroadfeatures#Examplesapprox-imation
featurewidthFigure 9.8: Example of feature width’s strong e↵ect on initial generalization (ﬁrst row) and
weak e↵ect on asymptotic accuracy (last row).

9.5. Feature Construction for Linear Methods 217
9.5.4 Tile Coding
Tile coding is a form of coarse coding for multi-dimensional continuous spaces that is
ﬂexible and computationally ecient. It may be the most practical feature representation
for modern sequential digital computers.
In tile coding the receptive ﬁelds of the features are grouped into partitions of the state
space. Each such partition is called a tiling , and each element of the partition is called a
tile. For example, the simplest tiling of a two-dimensional state space is a uniform grid
such as that shown on the left side of Figure 9.9. The tiles or receptive ﬁeld here are
squares rather than the circles in Figure 9.6. If just this single tiling were used, then the
state indicated by the white spot would be represented by the single feature whose tile
it falls within; generalization would be complete to all states within the same tile and
nonexistent to states outside it. With just one tiling, we would not have coarse coding

but just a case of state aggregation.Point in state spaceto berepresentedTiling 1Tiling 2Tiling 3Tiling 4Continuous 2D state spaceFour activetiles/features overlap the pointand are used to represent itFigure 9.9: Multiple, overlapping grid-tilings on a limited two-dimensional space. These tilings
are o↵set from one another by a uniform amount in each dimension.
To get the strengths of coarse coding requires overlapping receptive ﬁelds, and by
deﬁnition the tiles of a partition do not overlap. To get true coarse coding with tile coding,

multiple tilings are used, each o↵set by a fraction of a tile width. A simple case with
four tilings is shown on the right side of Figure 9.9. Every state, such as that indicated
by the white spot, falls in exactly one tile in each of the four tilings. These four tiles
correspond to four features that become active when the state occurs. Speciﬁcally, the
feature vector x(s) has one component for each tile in each tiling. In this example there
are 4 ⇥4⇥4 = 64 components, all of which will be 0 except for the four corresponding to
the tiles that sfalls within. Figure 9.10 shows the advantage of multiple o↵set tilings
(coarse coding) over a single tiling on the 1000-state random walk example.

An immediate practical advantage of tile coding is that, because it works with partitions,
the overall number of features that are active at one time is the same for any state.
Exactly one feature is present in each tiling, so the total number of features present is
always the same as the number of tilings. This allows the step-size parameter, ↵,t o
be set in an easy, intuitive way. For example, choosing ↵=1
n,w h e r e nis the number

218 Chapter 9: On-policy Prediction with Approximation
.4.3.2.10 averagedover 30 runs05000EpisodesState aggregation(one tiling)Tile coding (50 tilings)pVE
Figure 9.10: Why we use coarse coding. Shown are learning curves on the 1000-state random
walk example for the gradient Monte Carlo algorithm with a single tiling and with multiple
tilings. The space of 1000 states was treated as a single continuous dimension, covered with tiles
each 200 states wide. The multiple tilings were o↵set from each other by 4 states. The step-size
parameter was set so that the initial learning rate in the two cases was the same, ↵=0.0001 for
the single tiling and ↵=0.0001/50 for the 50 tilings.
of tilings, results in exact one-trial learning. If the example s7!vis trained on, then
whatever the prior estimate, ˆv(s,wt), the new estimate will be ˆv(s,wt+1)=v. Usually
one wishes to change more slowly than this, to allow for generalization and stochastic
variation in target outputs. For example, one might choose ↵=1

10n, in which case the
estimate for the trained state would move one-tenth of the way to the target in one
update, and neighboring states will be moved less, proportional to the number of tiles
they have in common.
Tile coding also gains computational advantages from its use of binary feature vectors.
Because each component is either 0 or 1, the weighted sum making up the approximate
value function (9.8) is almost trivial to compute. Rather than performing dmultiplications

and additions, one simply computes the indices of the n⌧dactive features and then
adds up the ncorresponding components of the weight vector.
Generalization occurs to states other than the one trained if those states fall within
any of the same tiles, proportional to the number of tiles in common. Even the choice of
how to o↵set the tilings from each other a↵ects generalization. If they are o↵set uniformly
in each dimension, as they were in Figure 9.9, then di↵erent states can generalize in

qualitatively di↵erent ways, as shown in the upper half of Figure 9.11. Each of the eight
subﬁgures show the pattern of generalization from a trained state to nearby points. In this
example there are eight tilings, thus 64 subregions within a tile that generalize distinctly,
but all according to one of these eight patterns. Note how uniform o↵sets result in a
strong e↵ect along the diagonal in many patterns. These artifacts can be avoided if the

tilings are o↵set asymmetrically, as shown in the lower half of the ﬁgure. These lower
generalization patterns are better because they are all well centered on the trained state
with no obvious asymmetries.

9.5. Feature Construction for Linear Methods 219Possible generalizations for uniformly oﬀset tilingsPossible generalizationsfor asymmetrically oﬀset tilingsFigure 9.11: Why tile asymmetrical o↵sets are preferred in tile coding. Shown is the strength
of generalization from a trained state, indicated by the small black plus, to nearby states, for the
case of eight tilings. If the tilings are uniformly o↵set (above), then there are diagonal artifacts
and substantial variations in the generalization, whereas with asymmetrically o↵set tilings the
generalization is more spherical and homogeneous.
Tilings in all cases are o↵set from each other by a fraction of a tile width in each
dimension. If wdenotes the tile width and nthe number of tilings, thenw
nis a fundamental
unit. Within small squaresw
non a side, all states activate the same tiles, have the same
feature representation, and the same approximated value. If a state is moved byw
n

n
in any cartesian direction, the feature representation changes by one component/tile.
Uniformly o↵set tilings are o↵set from each other by exactly this unit distance. For a
two-dimensional space, we say that each tiling is o↵set by the displacement vector (1 ,1),
meaning that it is o↵set from the previous tiling byw
ntimes this vector. In these terms,
the asymmetrically o↵set tilings shown in the lower part of Figure 9.11 are o↵set by a
displacement vector of (1 ,3).

n
Extensive studies have been made of the e↵ect of di↵erent displacement vectors on the
generalization of tile coding (Parks and Militzer, 1991; An, 1991; An, Miller and Parks,

220 Chapter 9: On-policy Prediction with Approximation
1991; Miller, An, Glanz and Carter, 1990), assessing their homegeneity and tendency
toward diagonal artifacts like those seen for the (1 ,1) displacement vectors. Based on this
work, Miller and Glanz (1996) recommend using displacement vectors consisting of the
ﬁrst odd integers. In particular, for a continuous space of dimension k, a good choice is
to use the ﬁrst odd integers (1 ,3,5,7,..., 2k1), with n(the number of tilings) set to an
integer power of 2 greater than or equal to 4 k. This is what we have done to produce the
tilings in the lower half of Figure 9.11, in which k= 2, n=234k, and the displacement
vector is (1 ,3). In a three-dimensional case, the ﬁrst four tilings would be o↵set in total
from a base position by (0 ,0,0), (1 ,3,5), (2 ,6,10), and (3 ,9,15). Open-source software
that can eciently make tilings like this for any kis readily available.
In choosing a tiling strategy, one has to pick the number of the tilings and the shape of

the tiles. The number of tilings, along with the size of the tiles, determines the resolution
or ﬁneness of the asymptotic approximation, as in general coarse coding and illustrated
in Figure 9.8. The shape of the tiles will determine the nature of generalization as in
Figure 9.7. Square tiles will generalize roughly equally in each dimension as indicated in
Figure 9.11 (lower). Tiles that are elongated along one dimension, such as the stripe tilings
in Figure 9.12 (middle), will promote generalization along that dimension. The tilings in

Figure 9.12 (middle) are also denser and thinner on the left, promoting discrimination
along the horizonal dimension at lower values along that dimension. The diagonal stripe
tiling in Figure 9.12 (right) will promote generalization along one diagonal. In higher
dimensions, axis-aligned stripes correspond to ignoring some of the dimensions in some
of the tilings, that is, to hyperplanar slices. Irregular tilings such as shown in Figure 9.12
(left) are also possible, though rare in practice and beyond the standard software.

a) Irregularb) Log stripesc) Diagonal stripesFigure 9.12: Tilings need not be grids. They can be arbitrarily shaped and non-uniform, while
still in many cases being computationally ecient to compute.
In practice, it is often desirable to use di↵erent shaped tiles in di↵erent tilings. For
example, one might use some vertical stripe tilings and some horizontal stripe tilings.
This would encourage generalization along either dimension. However, with stripe tilings
alone it is not possible to learn that a particular conjunction of horizontal and vertical

coordinates has a distinctive value (whatever is learned for it will bleed into states with the
same horizontal and vertical coordinates). For this one needs the conjunctive rectangular
tiles such as originally shown in Figure 9.9. With multiple tilings—some horizontal, some
vertical, and some conjunctive—one can get everything: a preference for generalizing
along each dimension, yet the ability to learn speciﬁc values for conjunctions (see Sutton,

9.5. Feature Construction for Linear Methods 221
1996 for examples). The choice of tilings determines generalization, and until this choice
can be e↵ectively automated, it is important that tile coding enables the choice to be
made ﬂexibly and in a way that makes sense to people.
Another useful trick for reducing memory requirements is hashing —a consistent pseudo-
random collapsing of a large tiling into a much smaller set of tiles. Hashing produces
tiles consisting of noncontiguous, disjoint regions randomly spread throughout the state
onetilespace, but that still form an exhaustive partition. For example,
one tile might consist of the four subtiles shown to the right.
Through hashing, memory requirements are often reduced by
large factors with little loss of performance. This is possible
because high resolution is needed in only a small fraction of the
state space. Hashing frees us from the curse of dimensionality
in the sense that memory requirements need not be exponential

in the number of dimensions, but need merely match the real
demands of the task. Open-source implementations of tile
coding commonly include ecient hashing.
Exercise 9.4 Suppose we believe that one of two state dimensions is more likely to have
an e↵ect on the value function than is the other, that generalization should be primarily
across this dimension rather than along it. What kind of tilings could be used to take
advantage of this prior knowledge? ⇤
9.5.5 Radial Basis Functions

222 Chapter 9: On-policy Prediction with Approximation
The primary advantage of RBFs over binary features is that they produce approximate
functions that vary smoothly and are di↵erentiable. Although this is appealing, in most
cases it has no practical signiﬁcance. Nevertheless, extensive studies have been made of
graded response functions such as RBFs in the context of tile coding (An, 1991; Miller et
al., 1991; An et al., 1991; Lane, Handelman and Gelfand, 1992). All of these methods
require substantial additional computational complexity (over tile coding) and often
reduce performance when there are more than two state dimensions. In high dimensions
the edges of tiles are much more important, and it has proven dicult to obtain well
controlled graded tile activations near the edges.
AnRBF network is a linear function approximator using RBFs for its features. Learning
is deﬁned by equations (9.7) and (9.8), exactly as in other linear function approximators.

In addition, some learning methods for RBF networks change the centers and widths of
the features as well, bringing them into the realm of nonlinear function approximators.
Nonlinear methods may be able to ﬁt target functions much more precisely. The downside
to RBF networks, and to nonlinear RBF networks especially, is greater computational
complexity and, often, more manual tuning before learning is robust and ecient.
9.6 Selecting Step-Size Parameters Manually

Most SGD methods require the designer to select an appropriate step-size parameter ↵.
Ideally this selection would be automated, and in some cases it has been, but for most
cases it is still common practice to set it manually. To do this, and to better understand
the algorithms, it is useful to develop some intuitive sense of the role of the step-size
parameter. Can we say in general how it should be set?
Theoretical considerations are unfortunately of little help. The theory of stochastic

approximation gives us conditions (2.7) on a slowly decreasing step-size sequence that are
sucient to guarantee convergence, but these tend to result in learning that is too slow.
The classical choice ↵t=1/t, which produces sample averages in tabular MC methods, is
not appropriate for TD methods, for nonstationary problems, or for any method using
function approximation. For linear methods, there are recursive least-squares methods
that set an optimal matrix step size, and these methods can be extended to temporal-

di↵erence learning as in the LSTD method described in Section 9.8, but these require
O(d2) step-size parameters, or dtimes more parameters than we are learning. For this
reason we rule them out for use on large problems where function approximation is most
needed.
To get some intuitive feel for how to set the step-size parameter manually, it is best
to go back momentarily to the tabular case. There we can understand that a step size
of↵= 1 will result in a complete elimination of the sample error after one target (see

needed.
(2.4) with a step size of one). As discussed on page 201, we usually want to learn slower
than this. In the tabular case, a step size of ↵=1
10would take about 10 experiences to
converge approximately to their mean target, and if we wanted to learn in 100 experiences
we would use ↵=1
100. In general, if ↵=1
⌧, then the tabular estimate for a state will
approach the mean of its targets, with the most recent targets having the greatest e↵ect,
after about ⌧experiences with the state.

9.7. Nonlinear Function Approximation: Artiﬁcial Neural Networks 223
With general function approximation there is not such a clear notion of number of
experiences with a state, as each state may be similar to and dissimilar from all the others
to various degrees. However, there is a similar rule that gives similar behavior in the case
of linear function approximation. Suppose you wanted to learn in about ⌧experiences
with substantially the same feature vector. A good rule of thumb for setting the step-size
parameter of linear SGD methods is then
↵.=
⌧E⇥
x>x⇤1, (9.19)
where xis a random feature vector chosen from the same distribution as input vectors
will be in the SGD. This method works best if the feature vectors do not vary greatly in
length; ideally x>xis a constant.
Exercise 9.5 Suppose you are using tile coding to transform a seven-dimensional continuous
state space into binary feature vectors to estimate a state value function ˆv(s,w)⇡v⇡(s).

of each dimension separately (stripe tilings), for 7 ⇥8 = 56 tilings. In addition, in case
there are some pairwise interactions between the dimensions, you also take all7
2
= 21
pairs of dimensions and tile each pair conjunctively with rectangular tiles. You make
two tilings for each pair of dimensions, making a grand total of 21 ⇥2 + 56 = 98 tilings.
Given these feature vectors, you suspect that you still have to average out some noise,
so you decide that you want learning to be gradual, taking about 10 presentations with
the same feature vector before learning nears its asymptote. What step-size parameter ↵
should you use? Why? ⇤
9.7 Nonlinear Function Approximation:
Artiﬁcial Neural Networks
Artiﬁcial neural networks (ANNs) are widely used for nonlinear function approximation.
An ANN is a network of interconnected units that have some of the properties of neurons,
the main components of nervous systems. ANNs have a long history, with the latest

advances in training deeply-layered ANNs (deep learning) being responsible for some
of the most impressive abilities of machine learning systems, including reinforcement
learning systems. In Chapter 16 we describe several impressive examples of reinforcement
learning systems that use ANN function approximation.
Figure 9.14 shows a generic feedforward ANN, meaning that there are no loops in the
network, that is, there are no paths within the network by which a unit’s output can
inﬂuence its input. The network in the ﬁgure has an output layer consisting of two output

units, an input layer with four input units, and two “hidden layers”: layers that are neither
input nor output layers. A real-valued weight is associated with each link. A weight
roughly corresponds to the ecacy of a synaptic connection in a real neural network (see
Section 15.1). If an ANN has at least one loop in its connections, it is a recurrent rather
than a feedforward ANN. Although both feedforward and recurrent ANNs have been
used in reinforcement learning, here we look only at the simpler feedforward case.

224 Chapter 9: On-policy Prediction with Approximation
Figure 9.14: A generic feedforward ANN with four input units, two output units, and two
hidden layers.
The units (the circles in Figure 9.14) are typically semi-linear units, meaning that they
compute a weighted sum of their input signals and then apply to the result a nonlinear
function, called the activation function , to produce the unit’s output, or activation.
Di↵erent activation functions are used, but they are typically S-shaped, or sigmoid,
functions such as the logistic function f(x)=1 /(1 + ex), though sometimes the rectiﬁer
nonlinearity f(x)=max(0,x) is used. A step function like f(x)=1i f x✓, and 0
otherwise, results in a binary unit with threshold ✓. The units in a network’s input layer
are somewhat di↵erent in having their activations set to externally-supplied values that
are the inputs to the function the network is approximating.
The activation of each output unit of a feedforward ANN is a nonlinear function of the

activation patterns over the network’s input units. The functions are parameterized by
the network’s connection weights. An ANN with no hidden layers can represent only a
very small fraction of the possible input-output functions. However an ANN with a single
hidden layer containing a large enough ﬁnite number of sigmoid units can approximate
any continuous function on a compact region of the network’s input space to any degree
of accuracy (Cybenko, 1989). This is also true for other nonlinear activation functions

that satisfy mild conditions, but nonlinearity is essential: if all the units in a multi-layer
feedforward ANN have linear activation functions, the entire network is equivalent to a
network with no hidden layers (because linear functions of linear functions are themselves
linear).
Despite this “universal approximation” property of one-hidden-layer ANNs, both
experience and theory show that approximating the complex functions needed for many
artiﬁcial intelligence tasks is made easier—indeed may require—abstractions that are

linear).
hierarchical compositions of many layers of lower-level abstractions, that is, abstractions
produced by deep architectures such as ANNs with many hidden layers. (See Bengio,
2009, for a thorough review.) The successive layers of a deep ANN compute increasingly

9.7. Nonlinear Function Approximation: Artiﬁcial Neural Networks 225
abstract representations of the network’s “raw” input, with each unit providing a feature
contributing to a hierarchical representation of the overall input-output function of the
network.
Training the hidden layers of an ANN is therefore a way to automatically create
features appropriate for a given problem so that hierarchical representations can be
produced without relying exclusively on hand-crafted features. This has been an enduring
challenge for artiﬁcial intelligence and explains why learning algorithms for ANNs with
hidden layers have received so much attention over the years. ANNs typically learn by a
stochastic gradient method (Section 9.3). Each weight is adjusted in a direction aimed at
improving the network’s overall performance as measured by an objective function to
be either minimized or maximized. In the most common supervised learning case, the

objective function is the expected error, or loss, over a set of labeled training examples. In
reinforcement learning, ANNs can use TD errors to learn value functions, or they can aim
to maximize expected reward as in a gradient bandit (Section 2.8) or a policy-gradient
algorithm (Chapter 13). In all of these cases it is necessary to estimate how a change
in each connection weight would inﬂuence the network’s overall performance, in other
words, to estimate the partial derivative of an objective function with respect to each

weight, given the current values of all the network’s weights. The gradient is the vector
of these partial derivatives.
The most successful way to do this for ANNs with hidden layers (provided the units
have di↵erentiable activation functions) is the backpropagation algorithm, which consists
of alternating forward and backward passes through the network. Each forward pass
computes the activation of each unit given the current activations of the network’s input
units. After each forward pass, a backward pass eciently computes a partial derivative

for each weight. (As in other stochastic gradient learning algorithms, the vector of these
partial derivatives is an estimate of the true gradient.) In Section 15.10 we discuss
methods for training ANNs with hidden layers that use reinforcement learning principles
instead of backpropagation. These methods are less ecient than the backpropagation
algorithm, but they may be closer to how real neural networks learn.
The backpropagation algorithm can produce good results for shallow networks having

1 or 2 hidden layers, but it may not work well for deeper ANNs. In fact, training a
network with k+ 1 hidden layers can actually result in poorer performance than training
a network with khidden layers, even though the deeper network can represent all the
functions that the shallower network can (Bengio, 2009). Explaining results like these
is not easy, but several factors are important. First, the large number of weights in
a typical deep ANN makes it dicult to avoid the problem of overﬁtting, that is, the

problem of failing to generalize correctly to cases on which the network has not been
trained. Second, backpropagation does not work well for deep ANNs because the partial
derivatives computed by its backward passes either decay rapidly toward the input side
of the network, making learning by deep layers extremely slow, or the partial derivatives
grow rapidly toward the input side of the network, making learning unstable. Methods
for dealing with these problems are largely responsible for many impressive recent results
achieved by systems that use deep ANNs.

Overﬁtting is a problem for any function approximation method that adjusts functions

226 Chapter 9: On-policy Prediction with Approximation
with many degrees of freedom on the basis of limited training data. It is less of a
problem for online reinforcement learning that does not rely on limited training sets, but
generalizing e↵ectively is still an important issue. Overﬁtting is a problem for ANNs in
general, but especially so for deep ANNs because they tend to have very large numbers
of weights. Many methods have been developed for reducing overﬁtting. These include
stopping training when performance begins to decrease on validation data di↵erent
from the training data (cross validation), modifying the objective function to discourage
complexity of the approximation (regularization), and introducing dependencies among
the weights to reduce the number of degrees of freedom (e.g., weight sharing).
A particularly e↵ective method for reducing overﬁtting by deep ANNs is the dropout
method introduced by Srivastava, Hinton, Krizhevsky, Sutskever, and Salakhutdinov

(2014). During training, units are randomly removed from the network (dropped out)
along with their connections. This can be thought of as training a large number of
“thinned” networks. Combining the results of these thinned networks at test time is a way
to improve generalization performance. The dropout method eciently approximates this
combination by multiplying each outgoing weight of a unit by the probability that that
unit was retained during training. Srivastava et al. found that this method signiﬁcantly

improves generalization performance. It encourages individual hidden units to learn
features that work well with random collections of other features. This increases the
versatility of the features formed by the hidden units so that the network does not overly
specialize to rarely-occurring cases.
Hinton, Osindero, and Teh (2006) took a major step toward solving the problem of
training the deep layers of a deep ANN in their work with deep belief networks, layered
networks closely related to the deep ANNs discussed here. In their method, the deepest

layers are trained one at a time using an unsupervised learning algorithm. Without
relying on the overall objective function, unsupervised learning can extract features that
capture statistical regularities of the input stream. The deepest layer is trained ﬁrst, then
with input provided by this trained layer, the next deepest layer is trained, and so on,
until the weights in all, or many, of the network’s layers are set to values that now act as
initial values for supervised learning. The network is then ﬁne-tuned by backpropagation

with respect to the overall objective function. Studies show that this approach generally
works much better than backpropagation with weights initialized with random values.
The better performance of networks trained with weights initialized this way could be
due to many factors, but one idea is that this method places the network in a region of
weight space from which a gradient-based algorithm can make good progress.
Batch normalization (Io↵e and Szegedy, 2015) is another technique that makes it easier

to train deep ANNs. It has long been known that ANN learning is easier if the network
input is normalized, for example, by adjusting each input variable to have zero mean and
unit variance. Batch normalization for training deep ANNs normalizes the output of deep
layers before they feed into the following layer. Io↵e and Szegedy (2015) used statistics
from subsets, or “mini-batches,” of training examples to normalize these between-layer
signals to improve the learning rate of deep ANNs.

Another technique useful for training deep ANNs is deep residual learning (He, Zhang,
Ren, and Sun, 2016). Sometimes it is easier to learn how a function di↵ers from the

9.7. Nonlinear Function Approximation: Artiﬁcial Neural Networks 227
identity function than to learn the function itself. Then adding this di↵erence, or residual
function, to the input produces the desired function. In deep ANNs, a block of layers
can be made to learn a residual function simply by adding shortcut, or skip, connections
around the block. These connections add the input to the block to its output, and
no additional weights are needed. He et al. (2016) evaluated this method using deep
convolutional networks with skip connections around every pair of adjacent layers, ﬁnding
substantial improvement over networks without the skip connections on benchmark image
classiﬁcation tasks. Both batch normalization and deep residual learning were used in
the reinforcement learning application to the game of Go that we describe in Chapter 16.
A type of deep ANN that has proven to be very successful in applications, including
impressive reinforcement learning applications (Chapter 16), is the deep convolutional

network . This type of network is specialized for processing high-dimensional data arranged
in spatial arrays, such as images. It was inspired by how early visual processing works in
the brain (LeCun, Bottou, Bengio and Ha↵ner, 1998). Because of its special architecture,
a deep convolutional network can be trained by backpropagation without resorting to
methods like those described above to train the deep layers.
Figure 9.15 illustrates the architecture of a deep convolutional network. This instance,

from LeCun et al. (1998), was designed to recognize hand-written characters. It consists
of alternating convolutional and subsampling layers, followed by several fully connected
ﬁnal layers. Each convolutional layer produces a number of feature maps. A feature
map is a pattern of activity over an array of units, where each unit performs the same
operation on data in its receptive ﬁeld, which is the part of the data it “sees” from the

preceding layer (or from the external input in the case of the ﬁrst convolutional layer).
The units of a feature map are identical to one another except that their receptive ﬁelds,
which are all the same size and shape, are shifted to di↵erent locations on the arrays
of incoming data. Units in the same feature map share the same weights. This means
that a feature map detects the same feature no matter where it is located in the input
array. In the network in Figure 9.15, for example, the ﬁrst convolutional layer produces 6

feature maps, each consisting of 28 ⇥28 units. Each unit in each feature map has a 5 ⇥
5 receptive ﬁeld, and these receptive ﬁelds overlap (in this case by four columns and four
Figure 9.15: Deep Convolutional Network. Republished with permission of Proceedings of the
IEEE, from Gradient-based learning applied to document recognition, LeCun, Bottou, Bengio,
and Ha↵ner, volume 86, 1998; permission conveyed through Copyright Clearance Center, Inc.

228 Chapter 9: On-policy Prediction with Approximation
rows). Consequently, each of the 6 feature maps is speciﬁed by just 25 adjustable weights.
The subsampling layers of a deep convolutional network reduce the spatial resolution of
the feature maps. Each feature map in a subsampling layer consists of units that average
over a receptive ﬁeld of units in the feature maps of the preceding convolutional layer.
For example, each unit in each of the 6 feature maps in the ﬁrst subsampling layer of the
network of Figure 9.15 averages over a 2 ⇥2 non-overlapping receptive ﬁeld over one of
the feature maps produced by the ﬁrst convolutional layer, resulting in six 14 ⇥14 feature
maps. Subsampling layers reduce the network’s sensitivity to the spatial locations of the
features detected, that is, they help make the network’s responses spatially invariant.
This is useful because a feature detected at one place in an image is likely to be useful at
other places as well.

Advances in the design and training of ANNs—of which we have only mentioned a
few—all contribute to reinforcement learning. Although current reinforcement learning
theory is mostly limited to methods using tabular or linear function approximation
methods, the impressive performances of notable reinforcement learning applications owe
much of their success to nonlinear function approximation by multi-layer ANNs. We
discuss several of these applications in Chapter 16.
9.8 Least-Squares TD

All the methods we have discussed so far in this chapter have required computation per
time step proportional to the number of parameters. With more computation, however,
one can do better. In this section we present a method for linear function approximation
that is arguably the best that can be done for this case.
As we established in Section 9.4 TD(0) with linear function approximation converges
asymptotically (for appropriately decreasing step sizes) to the TD ﬁxed point:
wTD=A1b,
where
A.=E⇥
xt(xtxt+1)>⇤

9.8. Least-Squares TD 229
However, the extra tfactors cancel out when LSTD uses these estimates to estimate the
TD ﬁxed point as
wt.=bA1
tbbt. (9.21)
This algorithm is the most data ecient form of linear TD(0), but it is also more
expensive computationally. Recall that semi-gradient TD(0) requires memory and per-
step computation that is only O(d).
How complex is LSTD? As it is written above the complexity seems to increase with
t, but the two approximations in (9.20) could be implemented incrementally using the
techniques we have covered earlier (e.g., in Chapter 2) so that they can be done in
constant time per step. Even so, the update for bAtwould involve an outer product (a
column vector times a row vector) and thus would be a matrix update; its computational
complexity would be O(d2), and of course the memory required to hold the bAtmatrix
would be O(d2).
A potentially greater problem is that our ﬁnal computation (9.21) uses the inverse

multiplications and thus is only O(d2). Thus we can store the inverse matrix bA1
t,
maintain it with (9.22) , and then use it in (9.21) , all with only O(d2) memory and
per-step computation. The complete algorithm is given in the box on the next page.
Of course, O(d2) is still signiﬁcantly more expensive than the O(d) of semi-gradient
TD. Whether the greater data eciency of LSTD is worth this computational expense
depends on how large dis, how important it is to learn quickly, and the expense of other
parts of the system. The fact that LSTD requires no step-size parameter is sometimes
also touted, but the advantage of this is probably overstated. LSTD does not require a
step size, but it does requires ";i f"is chosen too small the sequence of inverses can vary
wildly, and if "is chosen too large then learning is slowed. In addition, LSTD’s lack of a
step-size parameter means that it never forgets. This is sometimes desirable, but it is

problematic if the target policy ⇡changes as it does in reinforcement learning and GPI.
In control applications, LSTD typically has to be combined with some other mechanism
to induce forgeting, mooting any initial advantage of not requiring a step-size parameter.

this approach, a learning algorithm adjusts the parameters of a functional form intended
to approximate the value function over a problem’s entire state space. Each update,
s7!g, is a training example used by the learning algorithm to change the parameters
with the aim of reducing the approximation error. After the update, the training example
can be discarded (although it might be saved to be used again). When an approximate
value of a state (which we will call the query state ) is needed, the function is simply
evaluated at that state using the latest parameters produced by the learning algorithm.
Memory-based function approximation methods are very di↵erent. They simply save
training examples in memory as they arrive (or at least save a subset of the examples)
without updating any parameters. Then, whenever a query state’s value estimate is
needed, a set of examples is retrieved from memory and used to compute a value estimate

for the query state. This approach is sometimes called lazy learning because processing
training examples is postponed until the system is queried to provide an output.
Memory-based function approximation methods are prime examples of nonparametric
methods. Unlike parametric methods, the approximating function’s form is not limited
to a ﬁxed parameterized class of functions, such as linear functions or polynomials, but is
instead determined by the training examples themselves, together with some means for
combining them to output estimated values for query states. As more training examples

accumulate in memory, one expects nonparametric methods to produce increasingly
accurate approximations of any target function.

9.9. Memory-based Function Approximation 231
There are many di↵erent memory-based methods depending on how the stored training
examples are selected and how they are used to respond to a query. Here, we focus on
local-learning methods that approximate a value function only locally in the neighborhood
of the current query state. These methods retrieve a set of training examples from memory
whose states are judged to be the most relevant to the query state, where relevance
usually depends on the distance between states: the closer a training example’s state is
to the query state, the more relevant it is considered to be, where distance can be deﬁned
in many di↵erent ways. After the query state is given a value, the local approximation is
discarded.
The simplest example of the memory-based approach is the nearest neighbor method,
which simply ﬁnds the example in memory whose state is closest to the query state and
returns that example’s value as the approximate value of the query state. In other words,

if the query state is s, and s07!gis the example in memory in which s0is the closest
state to s,t h e n gis returned as the approximate value of s. Slightly more complicated
areweighted average methods that retrieve a set of nearest neighbor examples and return
a weighted average of their target values, where the weights generally decrease with
increasing distance between their states and the query state. Locally weighted regression is
similar, but it ﬁts a surface to the values of a set of nearest states by means of a parametric

approximation method that minimizes a weighted error measure like (9.1), where the
weights depend on distances from the query state. The value returned is the evaluation of
the locally-ﬁtted surface at the query state, after which the local approximation surface
is discarded.
Being nonparametric, memory-based methods have the advantage over parametric
methods of not limiting approximations to pre-speciﬁed functional forms. This allows
accuracy to improve as more data accumulates. Memory-based local approximation

methods have other properties that make them well suited for reinforcement learning.
Because trajectory sampling is of such importance in reinforcement learning, as discussed
in Section 8.6, memory-based local methods can focus function approximation on local
neighborhoods of states (or state–action pairs) visited in real or simulated trajectories.
There may be no need for global approximation because many areas of the state space will
never (or almost never) be reached. In addition, memory-based methods allow an agent’s

experience to have a relatively immediate a↵ect on value estimates in the neighborhood
of the current state, in contrast with a parametric method’s need to incrementally adjust
parameters of a global approximation.
Avoiding global approximation is also a way to address the curse of dimensionality.
For example, for a state space with kdimensions, a tabular method storing a global
approximation requires memory exponential in k. On the other hand, in storing examples
for a memory-based method, each example requires memory proportional to k, and the

memory required to store, say, nexamples is linear in n. Nothing is exponential in kor
n. Of course, the critical remaining issue is whether a memory-based method can answer
queries quickly enough to be useful to an agent. A related concern is how speed degrades
as the size of the memory grows. Finding nearest neighbors in a large database can take
too long to be practical in many applications.

232 Chapter 9: On-policy Prediction with Approximation
Proponents of memory-based methods have developed ways to accelerate the nearest
neighbor search. Using parallel computers or special purpose hardware is one approach;
another is the use of special multi-dimensional data structures to store the training data.
One data structure studied for this application is the k-d tree (short for k-dimensional
tree), which recursively splits a k-dimensional space into regions arranged as nodes of a
binary tree. Depending on the amount of data and how it is distributed over the state
space, nearest-neighbor search using k-d trees can quickly eliminate large regions of the
space in the search for neighbors, making the searches feasible in some problems where
naive searches would take too long.
Locally weighted regression additionally requires fast ways to do the local regression
computations which have to be repeated to answer each query. Researchers have developed

many ways to address these problems, including methods for forgetting entries in order to
keep the size of the database within bounds. The Bibliographic and Historical Comments
section at the end of this chapter points to some of the relevant literature, including a
selection of papers describing applications of memory-based learning to reinforcement
learning.
9.10 Kernel-based Function Approximation
Memory-based methods such as the weighted average and locally weighted regression

learning.
methods described above depend on assigning weights to examples s07!gin the database
depending on the distance between s0and a query states s. The function that assigns
these weights is called a kernel function , or simply a kernel . In the weighted average and
locally weighted regressions methods, for example, a kernel function k:R!Rassigns
weights to distances between states. More generally, weights do not have to depend on
distances; they can depend on some other measure of similarity between states. In this

case, k:S⇥S!R, so that k(s, s0) is the weight given to data about s0in its inﬂuence
on answering queries about s.
Viewed slightly di↵erently, k(s, s0) is a measure of the strength of generalization from
s0tos. Kernel functions numerically express how relevant knowledge about any state
is to any other state. As an example, the strengths of generalization for tile coding
shown in Figure 9.11 correspond to di↵erent kernel functions resulting from uniform and
asymmetrical tile o↵sets. Although tile coding does not explicitly use a kernel function

in its operation, it generalizes according to one. In fact, as we discuss more below, the
strength of generalization resulting from linear parametric function approximation can
always be described by a kernel function.
Kernel regression is the memory-based method that computes a kernel weighted average
of the targets of allexamples stored in memory, assigning the result to the query state.
IfDis the set of stored examples, and g(s0) denotes the target for state s0in a stored

example, then kernel regression approximates the target function, in this case a value
function depending on D, as
ˆv(s,D)=X
s02Dk(s, s0)g(s0). (9.23)

9.10. Kernel-based Function Approximation 233
The weighted average method described above is a special case in which k(s, s0) is non-zero
only when sands0are close to one another so that the sum need not be computed over
all of D.
A common kernel is the Gaussian radial basis function (RBF) used in RBF function
approximation as described in Section 9.5.5. In the method described there, RBFs are
features whose centers and widths are either ﬁxed from the start, with centers presumably
concentrated in areas where many examples are expected to fall, or are adjusted in some
way during learning. Barring methods that adjust centers and widths, this is a linear
parametric method whose parameters are the weights of each RBF, which are typically
learned by stochastic gradient, or semi-gradient, descent. The form of the approximation
is a linear combination of the pre-determined RBFs. Kernel regression with an RBF
kernel di↵ers from this in two ways. First, it is memory-based: the RBFs are centered on

the states of the stored examples. Second, it is nonparametric: there are no parameters
to learn; the response to a query is given by (9.23).
Of course, many issues have to be addressed for practical implementation of kernel
regression, issues that are beyond the scope of our brief discussion. However, it turns out
that any linear parametric regression method like those we described in Section 9.4, with
states represented by feature vectors x(s)=( x1(s),x2(s),...,x d(s))>, can be recast as
kernel regression where k(s, s0) is the inner product of the feature vector representations

ofsands0; that is
k(s, s0)=x(s)>x(s0). (9.24)
Kernel regression with this kernel function produces the same approximation that a linear
parametric method would if it used these feature vectors and learned with the same
We skip the mathematical justiﬁcation for this, which can be found in any modern
machine learning text, such as Bishop (2006), and simply point out an important
implication. Instead of constructing features for linear parametric function approximators,

one can instead construct kernel functions directly without referring at all to feature
vectors. Not all kernel functions can be expressed as inner products of feature vectors
as in (9.24), but a kernel function that can be expressed like this can o↵er signiﬁcant
advantages over the equivalent parametric method. For many sets of feature vectors,
(9.24) has a compact functional form that can be evaluated without any computation
taking place in the d-dimensional feature space. In these cases, kernel regression is much

less complex than directly using a linear parametric method with states represented by
these feature vectors. This is the so-called “kernel trick” that allows e↵ectively working
in the high-dimension of an expansive feature space while actually working only with the
set of stored training examples. The kernel trick is the basis of many machine learning
methods, and researchers have shown how it can sometimes beneﬁt reinforcement learning.

234 Chapter 9: On-policy Prediction with Approximation
9.11 Looking Deeper at On-policy Learning: Interest
and Emphasis
The algorithms we have considered so far in this chapter have treated all the states
encountered equally, as if they were all equally important. In some cases, however, we
are more interested in some states than others. In discounted episodic problems, for
example, we may be more interested in accurately valuing early states in the episode
than in later states where discounting may have made the rewards much less important
to the value of the start state. Or, if an action-value function is being learned, it may be
less important to accurately value poor actions whose value is much less than the greedy
action. Function approximation resources are always limited, and if they were used in a
more targeted way, then performance could be improved.
One reason we have treated all states encountered equally is that then we are updating

according to the on-policy distribution, for which stronger theoretical results are available
for semi-gradient methods. Recall that the on-policy distribution was deﬁned as the
distribution of states encountered in an MDP while following the target policy. Now we
will generalize this concept signiﬁcantly. Rather than having one on-policy distribution
for the MDP, we will have many. All of them will have in common that they are a
distribution of states encountered in trajectories while following the target policy, but

they will vary in how the trajectories are, in a sense, initiated.
We now introduce some new concepts. First we introduce a non-negative scalar measure,
a random variable Itcalled interest , indicating the degree to which we are interested in
accurately valuing the state (or state–action pair) at time t. If we don’t care at all about
the state, then the interest should be zero; if we fully care, it might be one, though it is
formally allowed to take any non-negative value. The interest can be set in any causal

way; for example, it may depend on the trajectory up to time tor the learned parameters
at time t. The distribution µin the VE (9.1) is then deﬁned as the distribution of
states encountered while following the target policy, weighted by the interest. Second, we
introduce another non-negative scalar random variable, the emphasis Mt. This scalar
multiplies the learning update and thus emphasizes or de-emphasizes the learning done
at time t. The general n-step learning rule, replacing (9.15), is
wt+n.=wt+n1+↵M t[Gt:t+nˆv(St,wt+n1)]rˆv(St,wt+n1),0t<T , (9.25)

9.11. Looking Deeper at On-policy Learning: Interest and Emphasis 235
Example 9.4: Interest and Emphasis
To see the potential beneﬁts of using interest and emphasis, consider the four-state
Markov reward process shown below:+1+1+1+1v⇡=4v⇡=3v⇡=2v⇡=1i=1i=0i=0i=0w1w1w2w2
Episodes start in the leftmost state, then transition one state to the right, with a
reward of +1, on each step until the terminal state is reached. The true value of
the ﬁrst state is thus 4, of the second state 3, and so on as shown below each state.
These are the true values; the estimated values can only approximate these because
they are constrained by the parameterization. There are two components to the
parameter vector w=(w1,w2)>, and the parameterization is as written inside
each state. The estimated values of the ﬁrst two states are given by w1alone and
thus must be the same even though their true values are di↵erent. Similarly, the
estimated values of the third and fourth states are given by w2alone and must be

the same even though their true values are di↵erent. Suppose that we are interested
in accurately valuing only the leftmost state; we assign it an interest of 1 while all
the other states are assigned an interest of 0, as indicated above the states.
First consider applying gradient Monte Carlo algorithms to this problem. The
algorithms presented earlier in this chapter that do not take into account interest
and emphasis (in (9.7) and the box on page 202) will converge (for decreasing step

sizes) to the parameter vector w1=( 3.5,1.5), which gives the ﬁrst state—the only
one we are interested in—a value of 3 .5 (i.e., intermediate between the true values
of the ﬁrst and second states). The methods presented in this section that do use
interest and emphasis, on the other hand, will learn the value of the ﬁrst state
exactly correctly; w1will converge to 4 while w2will never be updated because the
emphasis is zero in all states save the leftmost.

Now consider applying two-step semi-gradient TD methods. The methods from
earlier in this chapter without interest and emphasis (in (9.15) and (9.16) and
the box on page 209) will again converge to w1=( 3.5,1.5), while the methods
with interest and emphasis converge to w1=( 4 ,2). The latter produces the
exactly correct values for the ﬁrst state and for the third state (which the ﬁrst state
bootstraps from) while never making any updates corresponding to the second or
fourth states.

236 Chapter 9: On-policy Prediction with Approximation
9.12 Summary
Reinforcement learning systems must be capable of generalization if they are to be
applicable to artiﬁcial intelligence or to large engineering applications. To achieve this,
any of a broad range of existing methods for supervised-learning function approximation
can be used simply by treating each update as a training example.
Perhaps the most suitable supervised learning methods are those using parameterized
function approximation , in which the policy is parameterized by a weight vector w.
Although the weight vector has many components, the state space is much larger still,
and we must settle for an approximate solution. We deﬁned the mean squared value error ,
VE(w), as a measure of the error in the values v⇡w(s) for a weight vector wunder the
on-policy distribution ,µ.T h e VEgives us a clear way to rank di↵erent value-function
approximations in the on-policy case.

To ﬁnd a good weight vector, the most popular methods are variations of stochastic
gradient descent (SGD). In this chapter we have focused on the on-policy case with a ﬁxed
policy , also known as policy evaluation or prediction; a natural learning algorithm for this
case is n-step semi-gradient TD , which includes gradient Monte Carlo and semi-gradient
TD(0) algorithms as the special cases when n=1andn=1 respectively. Semi-gradient
TD methods are not true gradient methods. In such bootstrapping methods (including

DP), the weight vector appears in the update target, yet this is not taken into account in
computing the gradient—thus they are semi -gradient methods. As such, they cannot
rely on classical SGD results.
Nevertheless, good results can be obtained for semi-gradient methods in the special case
oflinear function approximation, in which the value estimates are sums of features times
corresponding weights. The linear case is the most well understood theoretically and
works well in practice when provided with appropriate features. Choosing the features

is one of the most important ways of adding prior domain knowledge to reinforcement
learning systems. They can be chosen as polynomials, but this case generalizes poorly in
the online learning setting typically considered in reinforcement learning. Better is to
choose features according the Fourier basis, or according to some form of coarse coding
with sparse overlapping receptive ﬁelds. Tile coding is a form of coarse coding that
is particularly computationally ecient and ﬂexible. Radial basis functions are useful

for one- or two-dimensional tasks in which a smoothly varying response is important.
LSTD is the most data-ecient linear TD prediction method, but requires computation
proportional to the square of the number of weights, whereas all the other methods are of
complexity linear in the number of weights. Nonlinear methods include artiﬁcial neural
networks trained by backpropagation and variations of SGD; these methods have become
very popular in recent years under the name deep reinforcement learning .

Linear semi-gradient n-step TD is guaranteed to converge under standard conditions,
for all n,t oa VEthat is within a bound of the optimal error (achieved asymptotically
by Monte Carlo methods). This bound is always tighter for higher nand approaches
zero as n!1. However, in practice very high nresults in very slow learning, and some
degree of bootstrapping ( n<1) is usually preferrable, just as we saw in comparisons of
tabular n-step methods in Chapter 7 and in comparisons of tabular TD and Monte Carlo

methods in Chapter 6.

Bibliographical and Historical Remarks 237
Generalization and function approximation have always been an integral part of rein-
forcement learning. Bertsekas and Tsitsiklis (1996), Bertsekas (2012), and Sugiyama et
al. (2013) present the state of the art in function approximation in reinforcement learning.
Some of the early work with function approximation in reinforcement learning is discussed
at the end of this section.
9.3 Gradient-descent methods for minimizing mean-squared error in supervised
learning are well known. Widrow and Ho↵ (1960) introduced the least-mean-
square (LMS) algorithm, which is the prototypical incremental gradient-descent
algorithm. Details of this and related algorithms are provided in many texts
(e.g., Widrow and Stearns, 1985; Bishop, 1995; Duda and Hart, 1973).
Semi-gradient TD(0) was ﬁrst explored by Sutton (1984, 1988), as part of the
linear TD( ) algorithm that we will treat in Chapter 12. The term “semi-gradient”

to describe these bootstrapping methods is new to the second edition of this
book.
The earliest use of state aggregation in reinforcement learning may have been
Michie and Chambers’s BOXES system (1968). The theory of state aggregation
in reinforcement learning has been developed by Singh, Jaakkola, and Jordan
(1995) and Tsitsiklis and Van Roy (1996). State aggregation has been used in
dynamic programming from its earliest days (e.g., Bellman, 1957a).
9.4 Sutton (1988) proved convergence of linear TD(0) in the mean to the minimal

book.
VEsolution for the case in which the feature vectors, {x(s):s2S}, are linearly
independent. Convergence with probability 1 was proved by several researchers
at about the same time (Peng, 1993; Dayan and Sejnowski, 1994; Tsitsiklis,
1994; Gurvits, Lin, and Hanson, 1994). In addition, Jaakkola, Jordan, and Singh
(1994) proved convergence under online updating. All of these results assumed
linearly independent feature vectors, which implies at least as many components
towtas there are states. Convergence for the more important case of general

(dependent) feature vectors was ﬁrst shown by Dayan (1992). A signiﬁcant
generalization and strengthening of Dayan’s result was proved by Tsitsiklis and
Van Roy (1997). They proved the main result presented in this section, the
bound on the asymptotic error of linear bootstrapping methods.
9.5 Our presentation of the range of possibilities for linear function approximation is
based on that by Barto (1990).
9.5.2 Konidaris, Osentoski, and Thomas (2011) introduced the Fourier basis in a

simple form suitable for reinforcement learning problems with multi-dimensional
continuous state spaces and functions that do not have to be periodic.
9.5.3 The term coarse coding is due to Hinton (1984), and our Figure 9.6 is based on
one of his ﬁgures. Waltz and Fu (1965) provide an early example of this type of
function approximation in a reinforcement learning system.

238 Chapter 9: On-policy Prediction with Approximation
9.5.4 Tile coding, including hashing, was introduced by Albus (1971, 1981). He de-
scribed it in terms of his “cerebellar model articulator controller,” or CMAC, as
tile coding is sometimes known in the literature. The term “tile coding” was new
to the ﬁrst edition of this book, though the idea of describing CMAC in these
terms is taken from Watkins (1989). Tile coding has been used in many rein-
forcement learning systems (e.g., Shewchuk and Dean, 1990; Lin and Kim, 1991;
Miller, Scalera, and Kim, 1994; Sofge and White, 1992; Tham, 1994; Sutton, 1996;
Watkins, 1989) as well as in other types of learning control systems (e.g., Kraft and
Campagna, 1990; Kraft, Miller, and Dietz, 1992). This section draws heavily on
the work of Miller and Glanz (1996). General software for tile coding is available in
several languages (e.g., see http://incompleteideas.net/tiles/tiles3.html ).

9.5.5 Function approximation using radial basis functions has received wide attention
ever since being related to ANNs by Broomhead and Lowe (1988). Powell (1987)
reviewed earlier uses of RBFs, and Poggio and Girosi (1989, 1990) extensively
developed and applied this approach.
9.6 Automatic methods for adapting the step-size parameter include RMSprop (Tiele-
man and Hinton, 2012), Adam (Kingma and Ba, 2015), stochastic meta-descent
methods such as Delta-Bar-Delta (Jacobs, 1988), its incremental generaliza-

tion (Sutton, 1992b, c; Mahmood et al., 2012), and nonlinear generalizations
(Schraudolph, 1999, 2002). Methods explicitly designed for reinforcement learn-
ing include AlphaBound (Dabney and Barto, 2012), SID and NOSID (Dabney,
2014), TIDBD (Kearney et al., in preparation) and the application of stochastic
meta-descent to policy gradient learning (Schraudolph, Yu, and Aberdeen, 2006).
9.6 The introduction of the threshold logic unit as an abstract model neuron by

McCulloch and Pitts (1943) was the beginning of ANNs. The history of ANNs as
learning methods for classiﬁcation or regression has passed through several stages:
roughly, the Perceptron (Rosenblatt, 1962) and ADALINE (ADAptive LINear
Element) (Widrow and Ho↵, 1960) stage of learning by single-layer ANNs, the
error-backpropagation stage (LeCun, 1985; Rumelhart, Hinton, and Williams,
1986) of learning by multi-layer ANNs, and the current deep-learning stage with
its emphasis on representation learning (e.g., Bengio, Courville, and Vincent,

2012; Goodfellow, Bengio, and Courville, 2016). Examples of the many books on
ANNs are Haykin (1994), Bishop (1995), and Ripley (2007).
ANNs as function approximation for reinforcement learning goes back to the early
work of Farley and Clark (1954), who used reinforcement-like learning to modify
the weights of linear threshold functions representing policies. Widrow, Gupta,
and Maitra (1973) presented a neuron-like linear threshold unit implementing a
learning process they called learning with a critic orselective bootstrap adaptation ,

a reinforcement-learning variant of the ADALINE algorithm. Werbos (1987,
1994) developed an approach to prediction and control that uses ANNs trained by
error backpropation to learn policies and value functions using TD-like algorithms.
Barto, Sutton, and Brouwer (1981) and Barto and Sutton (1981b) extended the

Bibliographical and Historical Remarks 239
idea of an associative memory network (e.g., Kohonen, 1977; Anderson, Silverstein,
Ritz, and Jones, 1977) to reinforcement learning. Barto, Anderson, and Sutton
(1982) used a two-layer ANN to learn a nonlinear control policy, and emphasized
the ﬁrst layer’s role of learning a suitable representation. Hampson (1983, 1989)
was an early proponent of multilayer ANNs for learning value functions. Barto,
Sutton, and Anderson (1983) presented an actor–critic algorithm in the form of an
ANN learning to balance a simulated pole (see Sections 15.7 and 15.8). Barto and
Anandan (1985) introduced a stochastic version of Widrow et al.’s (1973) selective
bootstrap algorithm called the associative reward-penalty (ARP)algorithm .
Barto (1985, 1986) and Barto and Jordan (1987) described multi-layer ANNs
consisting of A RPunits trained with a globally-broadcast reinforcement signal
to learn classiﬁcation rules that are not linearly separable. Barto (1985) discussed

this approach to ANNs and how this type of learning rule is related to others in
the literature at that time. (See Section 15.10 for additional discussion of this
approach to training multi-layer ANNs.) Anderson (1986, 1987, 1989) evaluated
numerous methods for training multilayer ANNs and showed that an actor–critic
algorithm in which both the actor and critic were implemented by two-layer
ANNs trained by error backpropagation outperformed single-layer ANNs in the

pole-balancing and tower of Hanoi tasks. Williams (1988) described several ways
that backpropagation and reinforcement learning can be combined for training
ANNs. Gullapalli (1990) and Williams (1992) devised reinforcement learning
algorithms for neuron-like units having continuous, rather than binary, outputs.
Barto, Sutton, and Watkins (1990) argued that ANNs can play signiﬁcant roles
for approximating functions required for solving sequential decision problems.
Williams (1992) related REINFORCE learning rules (Section 13.3) to the error

backpropagation method for training multi-layer ANNs. Tesauro’s TD-Gammon
(Tesauro 1992, 1994; Section 16.1) inﬂuentially demonstrated the learning abilities
of TD( ) algorithm with function approximation by multi-layer ANNs in learning
to play backgammon. The AlphaGo ,AlphaGo Zero , and AlphaZero programs
of Silver et al. (2016, 2017a, b; Section 16.6) used reinforcement learning with
deep convolutional ANNs in achieving impressive results with the game of Go.
Schmidhuber (2015) reviews applications of ANNs in reinforcement learning,

including applications of recurrent ANNs.
9.8 LSTD is due to Bradtke and Barto (see Bradtke, 1993, 1994; Bradtke and Barto,
1996; Bradtke, Ydstie, and Barto, 1994), and was further developed by Boyan
(1999, 2002), Nedi´ c and Bertsekas (2003), and Yu (2010). The incremental update
of the inverse matrix has been known at least since 1949 (Sherman and Morrison,
1949). An extension of least-squares methods to control was introduced by
Lagoudakis and Parr (2003; Bu¸ soniu, Lazaric, Ghavamzadeh, Munos, Babu˘ ska,
and De Schutter, 2012).

240 Chapter 9: On-policy Prediction with Approximation
9.9 Our discussion of memory-based function approximation is largely based on
the review of locally weighted learning by Atkeson, Moore, and Schaal (1997).
Atkeson (1992) discussed the use of locally weighted regression in memory-based
robot learning and supplied an extensive bibliography covering the history of
the idea. Stanﬁll and Waltz (1986) inﬂuentially argued for the importance of
memory based methods in artiﬁcial intelligence, especially in light of parallel
architectures then becoming available, such as the Connection Machine. Baird
and Klopf (1993) introduced a novel memory-based approach and used it as the
function approximation method for Q-learning applied to the pole-balancing task.
Schaal and Atkeson (1994) applied locally weighted regression to a robot juggling
control problem, where it was used to learn a system model. Peng (1995) used
the pole-balancing task to experiment with several nearest-neighbor methods

for approximating value functions, policies, and environment models. Tadepalli
and Ok (1996) obtained promising results with locally-weighted linear regression
to learn a value function for a simulated automatic guided vehicle task. Bottou
and Vapnik (1992) demonstrated surprising eciency of several local learning
algorithms compared to non-local algorithms in some pattern recognition tasks,
discussing the impact of local learning on generalization.
Bentley (1975) introduced k-d trees and reported observing average running

time of O(log n) for nearest neighbor search over nrecords. Friedman, Bentley,
and Finkel (1977) clariﬁed the algorithm for nearest neighbor search with k-d
trees. Omohundro (1987) discussed eciency gains possible with hierarchical
data structures such as k-d-trees. Moore, Schneider, and Deng (1997) introduced
the use of k-d trees for ecient locally weighted regression.
9.10 The origin of kernel regression is the method of potential functions of Aizerman,
Braverman, and Rozonoer (1964). They likened the data to point electric charges

of various signs and magnitudes distributed over space. The resulting electric
potential over space produced by summing the potentials of the point charges
corresponded to the interpolated surface. In this analogy, the kernel function is
the potential of a point charge, which falls o↵ as the reciprocal of the distance
from the charge. Connell and Utgo↵ (1987) applied an actor–critic method
to the pole-balancing task in which the critic approximated the value function
using kernel regression with an inverse-distance weighting. Predating widespread

interest in kernel regression in machine learning, these authors did not use the
term kernel, but referred to “Shepard’s method” (Shepard, 1968). Other kernel-
based approaches to reinforcement learning include those of Ormoneit and Sen
(2002), Dietterich and Wang (2002), Xu, Xie, Hu, and Lu (2005), Taylor and Parr
(2009), Barreto, Precup, and Pineau (2011), and Bhat, Farias, and Moallemi
(2012).
9.11 For Emphatic-TD methods, see the bibliographical notes to Section 11.8.

Bibliographical and Historical Remarks 241
The earliest example we know of in which function approximation methods were
used for learning value functions was Samuel’s checkers player (1959, 1967). Samuel
followed Shannon’s (1950) suggestion that a value function did not have to be exact to
be a useful guide to selecting moves in a game and that it might be approximated by
linear combination of features. In addition to linear function approximation, Samuel
experimented with lookup tables and hierarchical lookup tables called signature tables
(Grith, 1966, 1974; Page, 1977; Biermann, Fairﬁeld, and Beres, 1982).
At about the same time as Samuel’s work, Bellman and Dreyfus (1959) proposed using
function approximation methods with DP. (It is tempting to think that Bellman and
Samuel had some inﬂuence on one another, but we know of no reference to the other in
the work of either.) There is now a fairly extensive literature on function approximation

methods and DP, such as multigrid methods and methods using splines and orthogonal
polynomials (e.g., Bellman and Dreyfus, 1959; Bellman, Kalaba, and Kotkin, 1973; Daniel,
1976; Whitt, 1978; Reetz, 1977; Schweitzer and Seidmann, 1985; Chow and Tsitsiklis,
1991; Kushner and Dupuis, 1992; Rust, 1996).
Holland’s (1986) classiﬁer system used a selective feature-match technique to generalize
evaluation information across state–action pairs. Each classiﬁer matched a subset of states

having speciﬁed values for a subset of features, with the remaining features having arbitrary
values (“wild cards”). These subsets were then used in a conventional state-aggregation
approach to function approximation. Holland’s idea was to use a genetic algorithm
to evolve a set of classiﬁers that collectively would implement a useful action-value
function. Holland’s ideas inﬂuenced the early research of the authors on reinforcement
learning, but we focused on di↵erent approaches to function approximation. As function

approximators, classiﬁers are limited in several ways. First, they are state-aggregation
methods, with concomitant limitations in scaling and in representing smooth functions
eciently. In addition, the matching rules of classiﬁers can implement only aggregation
boundaries that are parallel to the feature axes. Perhaps the most important limitation of
conventional classiﬁer systems is that the classiﬁers are learned via the genetic algorithm,
an evolutionary method. As we discussed in Chapter 1, there is available during learning

much more detailed information about how to learn than can be used by evolutionary
methods. This perspective led us to instead adapt supervised learning methods for
use in reinforcement learning, speciﬁcally gradient-descent and ANN methods. These
di↵erences between Holland’s approach and ours are not surprising because Holland’s
ideas were developed during a period when ANNs were generally regarded as being too
weak in computational power to be useful, whereas our work was at the beginning of

the period that saw widespread questioning of that conventional wisdom. There remain
many opportunities for combining aspects of these di↵erent approaches.
Christensen and Korf (1986) experimented with regression methods for modifying
coecients of linear value function approximations in the game of chess. Chapman
and Kaelbling (1991) and Tan (1991) adapted decision-tree methods for learning value
functions. Explanation-based learning methods have also been adapted for learning

value functions, yielding compact representations (Yee, Saxena, Utgo↵, and Barto, 1990;
Dietterich and Flann, 1995).

Chapter 10
On-policy Control with
In this chapter we return to the control problem, now with parametric approximation of
the action-value function ˆq(s, a,w)⇡q⇤(s, a), where w2Rdis a ﬁnite-dimensional weight
vector. We continue to restrict attention to the on-policy case, leaving o↵-policy methods
to Chapter 11. The present chapter features the semi-gradient Sarsa algorithm, the
natural extension of semi-gradient TD(0) (last chapter) to action values and to on-policy
control. In the episodic case, the extension is straightforward, but in the continuing case
we have to take a few steps backward and re-examine how we have used discounting to
deﬁne an optimal policy. Surprisingly, once we have genuine function approximation we
have to give up discounting and switch to a new “average-reward” formulation of the
control problem, with new “di↵erential” value functions.
Starting ﬁrst in the episodic case, we extend the function approximation ideas presented

in the last chapter from state values to action values. Then we extend them to control
following the general pattern of on-policy GPI, using "-greedy for action selection. We
show results for n-step linear Sarsa on the Mountain Car problem. Then we turn to the
continuing case and repeat the development of these ideas for the average-reward case
with di↵erential values.
10.1 Episodic Semi-gradient Control
The extension of the semi-gradient prediction methods of Chapter 9 to action values is

straightforward. In this case it is the approximate action-value function, ˆ q⇡q⇡, that is
represented as a parameterized functional form with weight vector w. Whereas before we
considered random training examples of the form St7!Ut, now we consider examples
of the form St,At7!Ut. The update target Utcan be any approximation of q⇡(St,At),
including the usual backed-up values such as the full Monte Carlo return ( Gt) or any
of the n-step Sarsa returns (7.4) . The general gradient-descent update for action-value
243

244 Chapter 10: On-policy Control with Approximation
prediction is
Utˆq(St,At,wt)i
rˆq(St,At,wt). (10.1)
For example, the update for the one-step Sarsa method is
Rt+1+ˆq(St+1,At+1,wt)ˆq(St,At,wt)i
rˆq(St,At,wt).(10.2)
We call this method episodic semi-gradient one-step Sarsa . For a constant policy, this
method converges in the same way that TD(0) does, with the same kind of error bound
(9.14).
To form control methods, we need to couple such action-value prediction methods with
techniques for policy improvement and action selection. Suitable techniques applicable to
continuous actions, or to actions from large discrete sets, are a topic of ongoing research
with as yet no clear resolution. On the other hand, if the action set is discrete and not
too large, then we can use the techniques already developed in previous chapters. That
is, for each possible action aavailable in the current state St, we can compute ˆq(St,a ,wt)
and then ﬁnd the greedy action A⇤

t=argmaxaˆq(St,a ,wt). Policy improvement is then
done (in the on-policy case treated in this chapter) by changing the estimation policy to a
soft approximation of the greedy policy such as the "-greedy policy. Actions are selected
according to this same policy. Pseudocode for the complete algorithm is given in the box.
Episodic Semi-gradient Sarsa for Estimating ˆq⇡q⇤
Input: a di↵erentiable action-value function parameterization ˆ q:S⇥A⇥Rd!R
Algorithm parameters: step size ↵>0, small ">0

S, A initial state and action of episode (e.g., "-greedy)
IfS0is terminal:
w w+↵⇥
Rˆq(S, A, w)⇤
rˆq(S, A, w)
Go to next episode
Choose A0as a function of ˆ q(S0,·,w) (e.g., "-greedy)
w w+↵⇥
R+ˆq(S0,A0,w)ˆq(S, A, w)⇤
S S0
A A0
Example 10.1: Mountain Car Task Consider the task of driving an underpowered

w w+↵⇥
w w+↵⇥
S S0
A A0
car up a steep mountain road, as suggested by the diagram in the upper left of Figure 10.1.
The diculty is that gravity is stronger than the car’s engine, and even at full throttle
the car cannot accelerate up the steep slope. The only solution is to ﬁrst move away from
the goal and up the opposite slope on the left. Then, by applying full throttle the car

10.1. Episodic Semi-gradient Control 245
!1.2Position0.6Step 428Goal
Position40!.07.07VelocityVelocityVelocity
VelocityVelocityVelocityPositionPosition
Position027
01200104046Episode 12
Episode 104Episode 1000Episode 9000MOUNTAIN   CARGoalFigure 10.1: The Mountain Car task (upper left panel) and the cost-to-go function
(max aˆq(s, a,w)) learned during one run.
can build up enough inertia to carry it up the steep slope even though it is slowing down
the whole way. This is a simple example of a continuous control task where things have
to get worse in a sense (farther from the goal) before they can get better. Many control
methodologies have great diculties with tasks of this kind unless explicitly aided by a
human designer.
The reward in this problem is 1 on all time steps until the car moves past its goal
position at the top of the mountain, which ends the episode. There are three possible
actions: full throttle forward (+1), full throttle reverse ( 1), and zero throttle (0). The

246 Chapter 10: On-policy Control with Approximation
and asymmetrical o↵sets as described in Section 9.5.4.1The feature vectors x(s, a) created
by tile coding were then combined linearly with the parameter vector to approximate the
action-value function:
ˆq(s, a,w).=w>x(s, a)=dX
i=1wi·xi(s, a), (10.3)
for each pair of state, s, and action, a.
Figure 10.1 shows what typically happens while learning to solve this task with this
form of function approximation.2Shown is the negative of the value function (the cost-
to-go function) learned on a single run. The initial action values were all zero, which was
optimistic (all true values are negative in this task), causing extensive exploration to occur
even though the exploration parameter, ", was 0. This can be seen in the middle-top panel
of the ﬁgure, labeled “Step 428”. At this time not even one episode had been completed,
but the car has oscillated back and forth in the valley, following circular trajectories in

state space. All the states visited frequently are valued worse than unexplored states,
because the actual rewards have been worse than what was (unrealistically) expected.
This continually drives the agent away from wherever it has been, to explore new states,
until a solution is found.
Figure 10.2 shows several learning curves for semi-gradient Sarsa on this problem, with
various step sizes.
10020040010000Mountain CarSteps per episodelog scaleaveraged over 100 runsEpisode500↵=0.5/8↵=0.1/8↵=0.2/8

Figure 10.2: Mountain Car learning curves for the semi-gradient Sarsa method with tile-coding
function approximation and "-greedy action selection.
1In particular, we used the tile-coding software, available at http://incompleteideas.net/tiles/
tiles3.html ,w i t h iht=IHT(4096) andtiles(iht,8,[8*x/(0.5+1.2),8*xdot/(0.07+0.07)],A) to get
the indices of the ones in the feature vector for state ( x,xdot )a n da c t i o n A.

2This data is actually from the “semi-gradient Sarsa( )” algorithm that we will not meet until
Chapter 12, but semi-gradient Sarsa would behave similarly.

248 Chapter 10: On-policy Control with Approximation
10020040010000Mountain CarSteps per episodelog scaleaveraged over 100 runsEpisode500n=1n=8Figure 10.3: Performance of one-step vs 8-step semi-gradient Sarsa on the Mountain Car task.
Good step sizes were used: ↵=0.5/8 for n=1a n d ↵=0.3/8 for n=8 .
22024026030000.511.5Mountain CarSteps per episodeaveraged overﬁrst 50 episodesand 100 runs↵×  number of tilings (8)280n=1n=2n=4n=8n=16n=8n=4n=2n=16n=1Figure 10.4: E↵ect of the ↵andnon early performance of n-step semi-gradient Sarsa and
tile-coding function approximation on the Mountain Car task. As usual, an intermediate level of
bootstrapping ( n= 4) performed best. These results are for selected ↵values, on a log scale,
and then connected by straight lines. The standard errors ranged from 0.5 (less than the line
width) for n= 1 to about 4 for n= 16, so the main e↵ects are all statistically signiﬁcant.
Exercise 10.1 We have not explicitly considered or given pseudocode for any Monte Carlo

methods or in this chapter. What would they be like? Why is it reasonable not to give
pseudocode for them? How would they perform on the Mountain Car task? ⇤
Exercise 10.2 Give pseudocode for semi-gradient one-step Expected Sarsa for control. ⇤
Exercise 10.3 Why do the results shown in Figure 10.4 have higher standard errors at
large nthan at small n? ⇤

10.3. Average Reward: A New Problem Setting for Continuing Tasks 249
10.3 Average Reward: A New Problem Setting for
Continuing Tasks
We now introduce a third classical setting—alongside the episodic and discounted settings—
for formulating the goal in Markov decision problems (MDPs). Like the discounted
setting, the average reward setting applies to continuing problems, problems for which the
interaction between agent and environment goes on and on forever without termination
or start states. Unlike that setting, however, there is no discounting—the agent cares just
as much about delayed rewards as it does about immediate reward. The average-reward
setting is one of the major settings commonly considered in the classical theory of dynamic
programming and less-commonly in reinforcement learning. As we discuss in the next
section, the discounted setting is problematic with function approximation, and thus the
average-reward setting is needed to replace it.

In the average-reward setting, the quality of a policy ⇡is deﬁned as the average rate of
reward, or simply average reward , while following that policy, which we denote as r(⇡):
r(⇡).=l i m
h!11
hhX
t=1E[Rt|S0,A0:t1⇠⇡] (10.6)
=l i m
t!1E[Rt|S0,A0:t1⇠⇡], (10.7)
=X
sµ⇡(s)X
a⇡(a|s)X
s0,rp(s0,r|s, a)r,
where the expectations are conditioned on the initial state, S0, and on the subsequent
actions, A0,A1,...,A t1, being taken according to ⇡.µ⇡is the steady-state distribution,

µ⇡(s).=l i m t!1Pr{St=s|A0:t1⇠⇡}, which is assumed to exist for any ⇡and to be
independent of S0. This assumption about the MDP is known as ergodicity . It means
that where the MDP starts or any early decision made by the agent can have only a
temporary e↵ect; in the long run the expectation of being in a state depends only on the
policy and the MDP transition probabilities. Ergodicity is sucient to guarantee the
existence of the limits in the equations above.
There are subtle distinctions that can be drawn between di↵erent kinds of optimality
in the undiscounted continuing case. Nevertheless, for most practical purposes it may
be adequate simply to order policies according to their average reward per time step,
in other words, according to their r(⇡). This quantity is essentially the average reward
under ⇡, as suggested by (10.7) . In particular, we consider all policies that attain the
maximal value of r(⇡) to be optimal.

Note that the steady state distribution is the special distribution under which, if you
select actions according to ⇡, you remain in the same distribution. That is, for which
X
sµ⇡(s)X
a⇡(a|s)p(s0|s, a)=µ⇡(s0). (10.8)
In the average-reward setting, returns are deﬁned in terms of di↵erences between

single queue. If given access to a server, the customers pay a reward of 1, 2, 4, or 8 to
the server, depending on their priority, with higher priority customers paying more. In
each time step, the customer at the head of the queue is either accepted (assigned to one
of the servers) or rejected (removed from the queue, with a reward of zero). In either
case, on the next time step the next customer in the queue is considered. The queue
never empties, and the priorities of the customers in the queue are equally randomly
distributed. Of course a customer cannot be served if there is no free server; the customer
is always rejected in this case. Each busy server becomes free with probability p=0.06
on each time step. Although we have just described them for deﬁniteness, let us assume
the statistics of arrivals and departures are unknown. The task is to decide on each step
whether to accept or reject the next customer, on the basis of his priority and the number

of free servers, so as to maximize long-term reward without discounting.
In this example we consider a tabular solution to this problem. Although there is no
generalization between states, we can still consider it in the general function approximation
setting as this setting generalizes the tabular setting. Thus we have a di↵erential action-
value estimate for each pair of state (number of free servers and priority of the customer
at the head of the queue) and action (accept or reject). Figure 10.5 shows the solution

found by di↵erential semi-gradient Sarsa with parameters ↵=0.01,=0.01, and "=0.1.
The initial action values and ¯Rwere zero.

252 Chapter 10: On-policy Control with Approximation
-10-50100Diﬀerentialvalue of best actionNumber of free servers123456789100!15!10!5057priority 8priority 4priority 2priority 1
Number of free servers428ACCEPTREJECT12345678910Number of free serversPriority1POLICY
Value ofbest actionVALUEFUNCTION512345678910priority 8priority 4priority 2priority 1POLICYVALUEFUNCTIONFigure 10.5: The policy and value function found by di↵erential semi-gradient one-step Sarsa
on the access-control queuing task after 2 million steps. The drop on the right of the graph
is probably due to insucient data; many of these states were never experienced. The value
learned for ¯Rwas about 2 .31.
Exercise 10.7 Suppose there is an MDP that under any policy produces the deterministic
sequence of rewards +1 ,0,+1,0,+1,0,...going on forever. Technically, this is not allowed
because it violates ergodicity; there is no stationary limiting distribution µ⇡and the limit

(10.7) does not exist. Nevertheless, the average reward (10.6) is well deﬁned; What is
it? Now consider two states in this MDP. From A, the reward sequence is exactly as
described above, starting with a +1, whereas, from B, the reward sequence starts with
a 0 and then continues with +1 ,0,+1,0,.... The di↵erential return (10.9) is not well
deﬁned for this case as the limit does not exist. To repair this, one could alternately
deﬁne the value of a state as
v⇡(s).=l i m!1lim
h!1hX
t=0t⇣
E⇡[Rt+1|S0=s]r(⇡)⌘
. (10.13)

10.4. Deprecating the Discounted Setting 253
stuck there. What would the sequence of Rt¯Rterrors be? What would the sequence of
terrors be (using (10.10) )? Which error sequence would produce a more stable estimate
of the average reward if the estimate were allowed to change in response to the errors?
Why? ⇤
10.4 Deprecating the Discounted Setting
The continuing, discounted problem formulation has been very useful in the tabular case,
in which the returns from each state can be separately identiﬁed and averaged. But in the
approximate case it is questionable whether one should ever use this problem formulation.
To see why, consider an inﬁnite sequence of returns with no beginning or end, and no
clearly identiﬁed states. The states might be represented only by feature vectors, which
may do little to distinguish the states from each other. As a special case, all of the feature
vectors may be the same. Thus one really has only the reward sequence (and the actions),

and performance has to be assessed purely from these. How could it be done? One way
is by averaging the rewards over a long interval—this is the idea of the average-reward
setting. How could discounting be used? Well, for each time step we could measure
the discounted return. Some returns would be small and some big, so again we would
have to average them over a suciently large time interval. In the continuing setting
there are no starts and ends, and no special time steps, so there is nothing else that

could be done. However, if you do this, it turns out that the average of the discounted
returns is proportional to the average reward. In fact, for policy ⇡, the average of the
discounted returns is always r(⇡)/(1), that is, it is essentially the average reward,
r(⇡). In particular, the ordering of all policies in the average discounted return setting
would be exactly the same as in the average-reward setting. The discount rate thus has

they are all weighted by this, and thus the average of the returns will be this times the
average reward, or r(⇡)/(1).
This example and the more general argument in the box show that if we optimized
discounted value over the on-policy distribution, then the e↵ect would be identical to
optimizing undiscounted average reward; the actual value of would have no e↵ect. This
strongly suggests that discounting has no role to play in the deﬁnition of the control
problem with function approximation. One can nevertheless go ahead and use discounting
in solution methods. The discounting parameter changes from a problem parameter
to a solution method parameter! However, in this case we unfortunately would not
be guaranteed to optimize average reward (or the equivalent discounted value over the
on-policy distribution).

no longer true that if we change the policy to improve the discounted value of one state
then we are guaranteed to have improved the overall policy in any useful sense. That
guarantee was key to the theory of our reinforcement learning control methods. With
function approximation we have lost it!
In fact, the lack of a policy improvement theorem is also a theoretical lacuna for the
total-episodic and average-reward settings. Once we introduce function approximation
we can no longer guarantee improvement for any setting. In Chapter 13 we introduce an
alternative class of reinforcement learning algorithms based on parameterized policies,
and there we have a theoretical guarantee called the “policy-gradient theorem” which
plays a similar role as the policy improvement theorem. But for methods that learn
action values we seem to be currently without a local improvement guarantee (possibly
the approach taken by Perkins and Precup (2003) may provide a part of the answer). We

do know that "-greediﬁcation may sometimes result in an inferior policy, as policies may
chatter among good policies rather than converge (Gordon, 1996a). This is an area with
multiple open theoretical questions.

long-term estimate of the average reward. Unfortunately, ¯Rwill then be biased by its
initial value for many steps, which may make learning inecient. Alternatively, one could
use a sample average of the observed rewards for ¯R. That would initially adapt rapidly
but in the long run would also adapt slowly. As the policy slowly changed, ¯Rwould also
change; the potential for such long-term nonstationarity makes sample-average methods
ill-suited. In fact, the step-size parameter on the average reward is a perfect place to use
the unbiased constant-step-size trick from Exercise 2.7. Describe the speciﬁc changes
needed to the boxed algorithm for di↵erential semi-gradient n-step Sarsa to use this
trick. ⇤

256 Chapter 10: On-policy Control with Approximation
10.6 Summary
In this chapter we have extended the ideas of parameterized function approximation and
semi-gradient descent, introduced in the previous chapter, to control. The extension is
immediate for the episodic case, but for the continuing case we have to introduce a whole
new problem formulation based on maximizing the average reward setting per time step.
Surprisingly, the discounted formulation cannot be carried over to control in the presence
of approximations. In the approximate case most policies cannot be represented by a
value function. The arbitrary policies that remain need to be ranked, and the scalar
average reward r(⇡) provides an e↵ective way to do this.
The average reward formulation involves new di↵erential versions of value functions,
Bellman equations, and TD errors, but all of these parallel the old ones, and the
conceptual changes are small. There is also a new parallel set of di↵erential algorithms
for the average-reward case.

10.1 Semi-gradient Sarsa with function approximation was ﬁrst explored by Rummery
and Niranjan (1994). Linear semi-gradient Sarsa with "-greedy action selection
does not converge in the usual sense, but does enter a bounded region near
the best solution (Gordon, 1996a, 2001). Precup and Perkins (2003) showed
convergence in a di↵erentiable action selection setting. See also Perkins and
Pendrith (2002) and Melo, Meyn, and Ribeiro (2008). The mountain–car example

is based on a similar task studied by Moore (1990), but the exact form used here
is from Sutton (1996).
10.2 Episodic n-step semi-gradient Sarsa is based on the forward Sarsa( ) algorithm
of van Seijen (2016). The empirical results shown here are new to the second
edition of this text.
10.3 The average-reward formulation has been described for dynamic programming
(e.g., Puterman, 1994) and from the point of view of reinforcement learning (Ma-

hadevan, 1996; Tadepalli and Ok, 1994; Bertsekas and Tsitiklis, 1996; Tsitsiklis
and Van Roy, 1999). The algorithm described here is the on-policy analog of the
“R-learning” algorithm introduced by Schwartz (1993). The name R-learning was
probably meant to be the alphabetic successor to Q-learning, but we prefer to
think of it as a reference to the learning of di↵erential or relative values. The
access-control queuing example was suggested by the work of Carlstr¨ om and
Nordstr¨ om (1997).

10.4 The recognition of the limitations of discounting as a formulation of the rein-
forcement learning problem with function approximation became apparent to
the authors shortly after the publication of the ﬁrst edition of this text. Singh,
Jaakkola, and Jordan (1994) may have been the ﬁrst to observe it in print.

Chapter 11
*O↵-policy Methods with
This book has treated on-policy and o↵-policy learning methods since Chapter 5 primarily
as two alternative ways of handling the conﬂict between exploitation and exploration
inherent in learning forms of generalized policy iteration. The two chapters preceding this
have treated the on-policy case with function approximation, and in this chapter we treat
theo↵-policy case with function approximation. The extension to function approximation
turns out to be signiﬁcantly di↵erent and harder for o↵-policy learning than it is for
on-policy learning. The tabular o↵-policy methods developed in Chapters 6 and 7 readily
extend to semi-gradient algorithms, but these algorithms do not converge as robustly as
they do under on-policy training. In this chapter we explore the convergence problems,
take a closer look at the theory of linear function approximation, introduce a notion of
learnability, and then discuss new algorithms with stronger convergence guarantees for the

o↵-policy case. In the end we will have improved methods, but the theoretical results will
not be as strong, nor the empirical results as satisfying, as they are for on-policy learning.
Along the way, we will gain a deeper understanding of approximation in reinforcement
learning for on-policy learning as well as o↵-policy learning.
Recall that in o↵-policy learning we seek to learn a value function for a target policy
⇡, given data due to a di↵erent behavior policy b. In the prediction case, both policies

are static and given, and we seek to learn either state values ˆv⇡v⇡or action values
ˆq⇡q⇡. In the control case, action values are learned, and both policies typically change
during learning— ⇡being the greedy policy with respect to ˆq, and bbeing something
more exploratory such as the "-greedy policy with respect to ˆ q.
The challenge of o↵-policy learning can be divided into two parts, one that arises in
the tabular case and one that arises only with function approximation. The ﬁrst part

of the challenge has to do with the target of the update (not to be confused with the
target policy), and the second part has to do with the distribution of the updates. The
techniques related to importance sampling developed in Chapters 5 and 7 deal with
the ﬁrst part; these may increase variance but are needed in all successful algorithms,
257

258 Chapter 11: O↵-policy Methods with Approximation
tabular and approximate. The extension of these techniques to function approximation
are quickly dealt with in the ﬁrst section of this chapter.
Something more is needed for the second part of the challenge of o↵-policy learning
with function approximation because the distribution of updates in the o↵-policy case is
not according to the on-policy distribution. The on-policy distribution is important to
the stability of semi-gradient methods. Two general approaches have been explored to
deal with this. One is to use importance sampling methods again, this time to warp the
update distribution back to the on-policy distribution, so that semi-gradient methods
are guaranteed to converge (in the linear case). The other is to develop true gradient
methods that do not rely on any special distribution for stability. We present methods
based on both approaches. This is a cutting-edge research area, and it is not clear which

of these approaches is most e↵ective in practice.
11.1 Semi-gradient Methods
We begin by describing how the methods developed in earlier chapters for the o↵-
policy case extend readily to function approximation as semi-gradient methods. These
methods address the ﬁrst part of the challenge of o↵-policy learning (changing the update
targets) but not the second part (changing the update distribution). Accordingly, these
methods may diverge in some cases, and in that sense are not sound, but still they
are often successfully used. Remember that these methods areguaranteed stable and

asymptotically unbiased for the tabular case, which corresponds to a special case of
function approximation. So it may still be possible to combine them with feature selection
methods in such a way that the combined system could be assured stable. In any event,
these methods are simple and thus a good place to start.
In Chapter 7 we described a variety of tabular o↵-policy algorithms. To convert them
to semi-gradient form, we simply replace the update to an array ( VorQ) to an update

to a weight vector ( w), using the approximate value function ( ˆvorˆq) and its gradient.
Many of these algorithms use the per-step importance sampling ratio:
⇢t.=⇢t:t=⇡(At|St)
b(At|St). (11.1)
For example, the one-step, state-value algorithm is semi-gradient o↵-policy TD(0), which
is just like the corresponding on-policy algorithm (page 203) except for the addition of
⇢t:
wt+1.=wt+↵⇢ttrˆv(St,wt), (11.2)
where tis deﬁned appropriately depending on whether the problem is episodic and

260 Chapter 11: O↵-policy Methods with Approximation
11.2 Examples of O↵-policy Divergence
In this section we begin to discuss the second part of the challenge of o↵-policy learning
with function approximation—that the distribution of updates does not match the on-
policy distribution. We describe some instructive counterexamples to o↵-policy learning—
cases where semi-gradient and other simple algorithms are unstable and diverge.
To establish intuitions, it is best to consider ﬁrst a very simple example. Suppose,
perhaps as part of a larger MDP, there are two states whose estimated values are of
the functional form wand 2 w, where the parameter vector wconsists of only a single
component w. This occurs under linear function approximation if the feature vectors
for the two states are each simple numbers (single-component vectors), in this case 1
and 2. In the ﬁrst state, only one action is available, and it results deterministically in a
transition to the second state with a reward of 0:2w02w

where the expressions inside the two circles indicate the two state’s values.
Suppose initially w= 10. The transition will then be from a state of estimated value
10 to a state of estimated value 20. It will look like a good transition, and wwill be
increased to raise the ﬁrst state’s estimated value. If is nearly 1, then the TD error will
be nearly 10, and, if ↵=0.1, then wwill be increased to nearly 11 in trying to reduce the
TD error. However, the second state’s estimated value will also be increased, to nearly

11.2. Examples of O↵-policy Divergence 261
behavior policy might select actions on those other transitions which the target policy
never would. For these transitions, ⇢twould be zero and no update would be made.
Under on-policy training, however, ⇢tis always one. Each time there is a transition from
thewstate to the 2 wstate, increasing w, there would also have to be a transition out
of the 2 wstate. That transition would reduce w, unless it were to a state whose value
was higher (because <1) than 2 w, and then that state would have to be followed by a
state of even higher value, or else again wwould be reduced. Each state can support the
one before only by creating a higher expectation. Eventually the piper must be paid. In
the on-policy case the promise of future reward must be kept and the system is kept in
check. But in the o↵-policy case, a promise can be made and then, after taking an action
that the target policy never would, forgotten and forgiven.

This simple example communicates much of the reason why o↵-policy training can lead
to divergence, but it is not completely convincing because it is not complete—it is just a
fragment of a complete MDP. Can there really be a complete system with instability? A
simple complete example of divergence is Baird’s counterexample . Consider the episodic
seven-state, two-action MDP shown in Figure 11.1. The dashed action takes the system
to one of the six upper states with equal probability, whereas the solid action takes the

system to the seventh state. The behavior policy bselects the dashed andsolid actions
with probabilities6
7and1
7, so that the next-state distribution under it is uniform (the
same for all nonterminal states), which is also the starting distribution for each episode.
The target policy ⇡always takes the solid action, and so the on-policy distribution (for ⇡)
is concentrated in the seventh state. The reward is zero on all transitions. The discount
rate is =0.99.
Consider estimating the state-value under the linear parameterization indicated by

7and1
the expression shown in each state circle. For example, the estimated value of the
leftmost state is 2 w1+w8, where the subscript corresponds to the component of the2w2+w82w1+w82w3+w82w4+w82w5+w82w6+w8w7+2w8b(dashed|·)=6/7b(solid|·)=1/7⇡(solid|·)=1=0.99
Figure 11.1: Baird’s counterexample. The approximate state-value function for this Markov
process is of the form shown by the linear expressions inside each state. The solid action usually
results in the seventh state, and the dashed action usually results in one of the other six states,

each with equal probability. The reward is always zero.

262 Chapter 11: O↵-policy Methods with Approximation
overall weight vector w2R8; this corresponds to a feature vector for the ﬁrst state
being x(1) = (2 ,0,0,0,0,0,0,1)>. The reward is zero on all transitions, so the true value
function is v⇡(s) = 0, for all s, which can be exactly approximated if w=0. In fact,
there are many solutions, as there are more components to the weight vector (8) than
there are nonterminal states (7). Moreover, the set of feature vectors, {x(s):s2S},i s
a linearly independent set. In all these ways this task seems a favorable case for linear
If we apply semi-gradient TD(0) to this problem (11.2) , then the weights diverge
to inﬁnity, as shown in Figure 11.2 (left). The instability occurs for any positive step
size, no matter how small. In fact, it even occurs if an expected update is done as in
dynamic programming (DP), as shown in Figure 11.2 (right). That is, if the weight

vector, wk, is updated for all states at the same time in a semi-gradient way, using the
DP (expectation-based) target:
wk+1.=wk+↵
|S|X
s⇣
E⇡[Rt+1+ˆv(St+1,wk)|St=s]ˆv(s,wk)⌘
rˆv(s,wk).(11.9)
In this case, there is no randomness and no asynchrony, just as in a classical DP update.
The method is conventional except in its use of semi-gradient function approximation.
Yet still the system is unstable.
If we alter just the distribution of DP updates in Baird’s counterexample, from the

|S|X
s⇣
uniform distribution to the on-policy distribution (which generally requires asynchronous
updating), then convergence is guaranteed to a solution with error bounded by (9.14).
This example is striking because the TD and DP methods used are arguably the simplest
w8w83002001001011000010000w1–w6Stepsw7SweepsSemi-gradient Off-policy TDSemi-gradient DPw1–w6w7
Figure 11.2: Demonstration of instability on Baird’s counterexample. Shown are the evolution

of the components of the parameter vector wof the two semi-gradient algorithms. The step size
was↵=0.01, and the initial weights were w=( 1,1,1,1,1,1,10,1)>.

11.2. Examples of O↵-policy Divergence 263
and best-understood bootstrapping methods, and the linear, semi-descent method used is
arguably the simplest and best-understood kind of function approximation. The example
shows that even the simplest combination of bootstrapping and function approximation
can be unstable if the updates are not done according to the on-policy distribution.
There are also counterexamples similar to Baird’s showing divergence for Q-learning.
This is cause for concern because otherwise Q-learning has the best convergence guarantees
of all control methods. Considerable e↵ort has gone into trying to ﬁnd a remedy to
this problem or to obtain some weaker, but still workable, guarantee. For example, it
may be possible to guarantee convergence of Q-learning as long as the behavior policy is
suciently close to the target policy, for example, when it is the "-greedy policy. To the
best of our knowledge, Q-learning has never been found to diverge in this case, but there

has been no theoretical analysis. In the rest of this section we present several other ideas
that have been explored.
Suppose that instead of taking just a step toward the expected one-step return on each
iteration, as in Baird’s counterexample, we actually change the value function all the way
to the best, least-squares approximation. Would this solve the instability problem? Of
course it would if the feature vectors, {x(s):s2S}, formed a linearly independent set,
as they do in Baird’s counterexample, because then exact approximation is possible on

each iteration and the method reduces to standard tabular DP. But of course the point
here is to consider the case when an exact solution is notpossible. In this case stability
is not guaranteed even when forming the best approximation at each iteration, as shown
in the example.
Example 11.1: Tsitsiklis and Van Roy’s Counterexample This example shows
that linear function approximation would not work with DP even if the least-squares1""w2w
solution was found at each step. The counterexample is formed

by extending the w-to-2wexample (from earlier in this section)
with a terminal state, as shown to the right. As before, the
estimated value of the ﬁrst state is w, and the estimated value
of the second state is 2 w. The reward is zero on all transitions,
so the true values are zero at both states, which is exactly
representable with w= 0. If we set wk+1at each step so
as to minimize the VEbetween the estimated value and the
expected one-step return, then we have
wk+1= argmin
w2RX
s2S⇣
ˆv(s,w)E⇡⇥

264 Chapter 11: O↵-policy Methods with Approximation
Another way to try to prevent instability is to use special methods for function
approximation. In particular, stability is guaranteed for function approximation methods
that do not extrapolate from the observed targets. These methods, called averagers ,
include nearest neighbor methods and locally weighted regression, but not popular
methods such as tile coding and artiﬁcial neural networks (ANNs).
Exercise 11.3 (programming) Apply one-step semi-gradient Q-learning to Baird’s coun-
terexample and show empirically that its weights diverge. ⇤
11.3 The Deadly Triad
Our discussion so far can be summarized by saying that the danger of instability and
divergence arises whenever we combine all of the following three elements, making up
what we call the deadly triad :
Function approximation A powerful, scalable way of generalizing from a state space
much larger than the memory and computational resources (e.g., linear function
approximation or ANNs).

Bootstrapping Update targets that include existing estimates (as in dynamic pro-
gramming or TD methods) rather than relying exclusively on actual rewards and
complete returns (as in MC methods).
O↵-policy training Training on a distribution of transitions other than that produced
by the target policy. Sweeping through the state space and updating all states
uniformly, as in dynamic programming, does not respect the target policy and is
an example of o↵-policy training.

In particular, note that the danger is notdue to control or to generalized policy iteration.
Those cases are more complex to analyze, but the instability arises in the simpler prediction
case whenever it includes all three elements of the deadly triad. The danger is also not
due to learning or to uncertainties about the environment, because it occurs just as
strongly in planning methods, such as dynamic programming, in which the environment
is completely known.

If any two elements of the deadly triad are present, but not all three, then instability
can be avoided. It is natural, then, to go through the three and see if there is any one
that can be given up.
Of the three, function approximation most clearly cannot be given up. We need
methods that scale to large problems and to great expressive power. We need at least
linear function approximation with many features and parameters. State aggregation or

nonparametric methods whose complexity grows with data are too weak or too expensive.
Least-squares methods such as LSTD are of quadratic complexity and are therefore too
expensive for large problems.
Doing without bootstrapping is possible, at the cost of computational and data eciency.
Perhaps most important are the losses in computational eciency. Monte Carlo (non-
bootstrapping) methods require memory to save everything that happens between making

11.3. The Deadly Triad 265
each prediction and obtaining the ﬁnal return, and all their computation is done once the
ﬁnal return is obtained. The cost of these computational issues is not apparent on serial
von Neumann computers, but would be on specialized hardware. With bootstrapping and
eligibility traces (Chapter 12), data can be dealt with when and where it is generated,
then need never be used again. The savings in communication and memory made possible
by bootstrapping are great.
The losses in data eciency by giving up bootstrapping are also signiﬁcant. We
have seen this repeatedly, such as in Chapters 7 (Figure 7.2) and 9 (Figure 9.2), where
some degree of bootstrapping performed much better than Monte Carlo methods on
the random-walk prediction task, and in Chapter 10 where the same was seen on the
Mountain-Car control task (Figure 10.4). Many other problems show much faster learning
with bootstrapping (e.g., see Figure 12.14). Bootstrapping often results in faster learning

because it allows learning to take advantage of the state property, the ability to recognize
a state upon returning to it. On the other hand, bootstrapping can impair learning on
problems where the state representation is poor and causes poor generalization (e.g.,
this seems to be the case on Tetris, see S¸im¸ sek, Alg´ orta, and Kothiyal, 2016). A poor
state representation can also result in bias; this is the reason for the poorer bound on

the asymptotic approximation quality of bootstrapping methods (Equation 9.14). On
balance, the ability to bootstrap has to be considered extremely valuable. One may
sometimes choose not to use it by selecting long n-step updates (or a large bootstrapping
parameter, ⇡1; see Chapter 12) but often bootstrapping greatly increases eciency. It
is an ability that we would very much like to keep in our toolkit.
Finally, there is o↵-policy learning ; can we give that up? On-policy methods are often

adequate. For model-free reinforcement learning, one can simply use Sarsa rather than
Q-learning. O↵-policy methods free behavior from the target policy. This could be
considered an appealing convenience but not a necessity. However, o↵-policy learning
is essential to other anticipated use cases, cases that we have not yet mentioned in this
book but may be important to the larger goal of creating a powerful intelligent agent.
In these use cases, the agent learns not just a single value function and single policy,

but large numbers of them in parallel. There is extensive psychological evidence that
people and animals learn to predict many di↵erent sensory events, not just rewards. We
can be surprised by unusual events, and correct our predictions about them, even if
they are of neutral valence (neither good nor bad). This kind of prediction presumably
underlies predictive models of the world such as are used in planning. We predict what
we will see after eye movements, how long it will take to walk home, the probability of

making a jump shot in basketball, and the satisfaction we will get from taking on a new
project. In all these cases, the events we would like to predict depend on our acting in
a certain way. To learn them all, in parallel, requires learning from the one stream of
experience. There are many target policies, and thus the one behavior policy cannot
equal all of them. Yet parallel learning is conceptually possible because the behavior
policy may overlap in part with many of the target policies. To take full advantage of
this requires o↵-policy learning.

266 Chapter 11: O↵-policy Methods with Approximation
11.4 Linear Value-function Geometry
To better understand the stability challenge of o↵-policy learning, it is helpful to think
about value function approximation more abstractly and independently of how learning
is done. We can imagine the space of all possible state-value functions—all functions
from states to real numbers v:S!R. Most of these value functions do not correspond
to any policy. More important for our purposes is that most are not representable by the
function approximator, which by design has far fewer parameters than there are states.
Given an enumeration of the state space S={s1,s2,...,s |S|}, any value function v
corresponds to a vector listing the value of each state in order [ v(s1),v(s2),...,v (s|S|)]>.
This vector representation of a value function has as many components as there are
states. In most cases where we want to use function approximation, this would be far

too many components to represent the vector explicitly. Nevertheless, the idea of this
vector is conceptually useful. In the following, we treat a value function and its vector
representation interchangeably.
To develop intuitions, consider the case with three states S={s1,s2,s3}and two
parameters w=(w1,w2)>. We can then view all value functions/vectors as points in
a three-dimensional space. The parameters provide an alternative coordinate system
over a two-dimensional subspace. Any weight vector w=(w1,w2)>is a point in the

two-dimensional subspace and thus also a complete value function vwthat assigns values
to all three states. With general function approximation the relationship between the
full space and the subspace of representable functions could be complex, but in the case
oflinear value-function approximation the subspace is a simple plane, as suggested by
Figure 11.3.
Now consider a single ﬁxed policy ⇡. We assume that its true value function, v⇡,i st o o
complex to be represented exactly as an approximation. Thus v⇡is not in the subspace;

in the ﬁgure it is depicted as being above the planar subspace of representable functions.
Ifv⇡cannot be represented exactly, what representable value function is closest to
it? This turns out to be a subtle question with multiple answers. To begin, we need
a measure of the distance between two value functions. Given two value functions v1
andv2, we can talk about the vector di↵erence between them, v=v1v2.I fvis small,
then the two value functions are close to each other. But how are we to measure the size

of this di↵erence vector? The conventional Euclidean norm is not appropriate because,
as discussed in Section 9.2, some states are more important than others because they
occur more frequently or because we are more interested in them (Section 9.11). As
in Section 9.2, let us use the distribution µ:S![0,1] to specify the degree to which
we care about di↵erent states being accurately valued (often taken to be the on-policy
distribution). We can then deﬁne the distance between value functions using the norm
kvk2
µ.=X
s2Sµ(s)v(s)2. (11.11)

kvk2
µ.=X
Note that the VEfrom Section 9.2 can be written simply using this norm as VE(w)=
kvwv⇡k2
µ. For any value function v, the operation of ﬁnding its closest value function
in the subspace of representable value functions is a projection operation. We deﬁne a

used to select actions.The functionv⇡is called thestate-value functionfor policy⇡.A key subproblem underlying almost all ecient solution strategies for MDPs ispolicyevaluation, the computation or estimation ofv⇡for a given policy⇡. For example, thepopular DP algorithm known as policy iteration involves computing the value function fora sequence of policies, each of which is better than the previous, until an optimal policy isfound. In TDL, algorithms such as TD() are used to approximate the value function forthe current policy, for example as part of actor–critic methods.If the state space is ﬁnite, then the estimated value function may be represented in acomputer as a large array with one entry for each state and the entries directly updated toform the estimate. Suchtabularmethods can handle large state spaces, even continuousones, through discretization, state aggregation, and interpolation, but as the dimensionalityof the state space increases, these methods rapidly become computationally infeasible

forthe current policy, for example as part of actor–critic methods.If the state space is ﬁnite, then the estimated value function may be represented in acomputer as a large array with one entry for each state and the entries directly updated toform the estimate. Suchtabularmethods can handle large state spaces, even continuousones, through discretization, state aggregation, and interpolation, but as the dimensionalityof the state space increases, these methods rapidly become computationally infeasible orine↵ective. This is the e↵ect which gave rise to the phrase “the curse of dimensionality.”A more general and ﬂexible approach is to represent the value function by a functionalform of ﬁxed size and ﬁxed structure with many variable parameters or weights. The weightsare then changed to reshape the approximate value function to better match the true valuefunction. We denote the parameterized value function approximator asv(s)⇡v⇡(s),s2S,(2)where✓2Rn,w i t hn⌧|S|, is the weight/parameter vector. The approximate

This is the e↵ect which gave rise to the phrase “the curse of dimensionality.”A more general and ﬂexible approach is to represent the value function by a functionalform of ﬁxed size and ﬁxed structure with many variable parameters or weights. The weightsare then changed to reshape the approximate value function to better match the true valuefunction. We denote the parameterized value function approximator asv(s)⇡v⇡(s),s2S,(2)where✓2Rn,w i t hn⌧|S|, is the weight/parameter vector. The approximate valuefunction can have arbitrary form as long as it is everywhere di↵erentiable with respect tothe weights. For example, it could be a cubic spline, or it could implemented by a multi-layer neural network where✓is the concatenation of all the connection weights. Henceforthrefer to✓exclusively as the weights, or weight vector, and reserve the word “parameter”for things like the discount-rate parameter,, and step-size parameters.An important special case is that in which the approximate value function is linear

general di↵erent from that of minimum VE.The BE was ﬁrst proposed as an objective function for DP by Schweitzer and Seidmann(1985). Baird (1995, 1999) extended it to TDL based on stochastic gradient descent, andEngel, Mannor, and Meir (2003) extended it to least squares (O(n2)) methods known asGaussian Process TDL. In the literature, BE minimization is often referred to as Bellmanresidual minimization.2.3 Projected Bellman errorThe third goal for approximation is to approximately solve theprojectedBellman equation:v=⇧B⇡v.(11)Unlike the original Bellman equation, for most function approximators (e.g., linear ones) theprojected Bellman equation can be solved exactly. The original TDL methods (Sutton 1988,Dayan 1992) converge to this solution, as does least-squares TDL (Bradke & Barto 1996,Boyan 1999). The goal of achieving (11) exactly is common; less common is to considerapproximating it as an objective. The early work on gradient-TD (e.g., Sutton et al. 2009)appears to be ﬁrst to have explicitly proposed

dimensional space of all value functions over three states, while shown as a plane is the subspace of
all value functions representable by a linear function approximator with parameter w=(w1,w2)>.
The true value function v⇡is in the larger space and can be projected down (into the subspace,
using a projection operator ⇧) to its best approximation in the value error (VE) sense. The
best approximators in the Bellman error (BE), projected Bellman error (PBE), and temporal
di↵erence error (TDE) senses are all potentially di↵erent and are shown in the lower right. (VE,
BE, and PBE are all treated as the corresponding vectors in this ﬁgure.) The Bellman operator
takes a value function in the plane to one outside, which can then be projected back. If you
iteratively applied the Bellman operator outside the space (shown in gray above) you would
reach the true value function, as in conventional dynamic programming. If instead you kept

projecting back into the subspace at each step, as in the lower step shown in gray, then the ﬁxed
point would be the point of vector-zero PBE.
projection operator ⇧ that takes an arbitrary value function to the representable function
that is closest in our norm:
⇧v.=vwwhere w= argmin
w2Rdkvvwk2
µ. (11.12)
The representable value function that is closest to the true value function v⇡is thus its
projection, ⇧ v⇡, as suggested in Figure 11.3. This is the solution asymptotically found

µ. (11.12)
by Monte Carlo methods, albeit often very slowly. The projection operation is discussed
more fully in the box on the next page.
TD methods ﬁnd di↵erent solutions. To understand their rationale, recall that the
Bellman equation for value function v⇡is
v⇡(s)=X
a⇡(a|s)X
s0,rp(s0,r|s, a)[r+v⇡(s0)], for all s2S. (11.13)

268 Chapter 11: O↵-policy Methods with Approximation
The projection matrix
For a linear function approximator, the projection operation is linear, which implies
that it can be represented as an |S|⇥| S|matrix:
⇧.=X
X>DX1X>D, (11.14)
where, as in Section 9.4, Ddenotes the |S|⇥| S|diagonal matrix with the µ(s)
on the diagonal, and Xdenotes the |S|⇥dmatrix whose rows are the feature
vectors x(s)>, one for each state s.I ft h ei n v e r s ei n (11.14) does not exist, then the
pseudoinverse is substituted. Using these matrices, the squared norm of a vector
kvk2
µ=v>Dv, (11.15)
and the approximate linear value function can be written
vw=Xw. (11.16)
The true value function v⇡is the only value function that solves (11.13) exactly. If an
approximate value function vwwere substituted for v⇡, the di↵erence between the right
and left sides of the modiﬁed equation could be used as a measure of how far o↵ vwis
from v⇡. We call this the Bellman error at state s:
¯w(s).=0
@X
a⇡(a|s)X

µ. (11.19)
It is not possible in general to reduce the BEto zero (at which point vw=v⇡), but for
linear function approximation there is a unique value of wfor which the BEis minimized.
This point in the representable-function subspace (labeled minBEin Figure 11.3) is
di↵erent in general from that which minimizes the VE(shown as ⇧ v⇡). Methods that
seek to minimize the BE are discussed in the next two sections.
The Bellman error vector is shown in Figure 11.3 as the result of applying the Bellman
operator B⇡:R|S|!R|S|to the approximate value function. The Bellman operator is

11.5. Gradient Descent in the Bellman Error 269
deﬁned by
(B⇡v)(s).=X
a⇡(a|s)X
s0,rp(s0,r|s, a)[r+v(s0)], (11.20)
for all s2Sandv:S!R. The Bellman error vector for vcan be written ¯w=B⇡vwvw.
If the Bellman operator is applied to a value function in the representable subspace,
then, in general, it will produce a new value function that is outside the subspace, as
suggested in the ﬁgure. In dynamic programming (without function approximation), this
operator is applied repeatedly to the points outside the representable space, as suggested
by the gray arrows in the top of Figure 11.3. Eventually that process converges to the
true value function v⇡, the only ﬁxed point for the Bellman operator, the only value
function for which
v⇡=B⇡v⇡, (11.21)
which is just another way of writing the Bellman equation for ⇡(11.13).
With function approximation, however, the intermediate value functions lying outside
the subspace cannot be represented. The gray arrows in the upper part of Figure 11.3

cannot be followed because after the ﬁrst update (dark line) the value function must
be projected back into something representable. The next iteration then begins within
the subspace; the value function is again taken outside of the subspace by the Bellman
operator and then mapped back by the projection operator, as suggested by the lower
gray arrow and line. Following these arrows is a DP-like process with approximation.
In this case we are interested in the projection of the Bellman error vector back into

Section 9.4. As we have seen, this point is not always stable under semi-gradient TD
methods and o↵-policy training. As shown in the ﬁgure, this value function is generally
di↵erent from those minimizing VEorBE. Methods that are guaranteed to converge to
it are discussed in Sections 11.7 and 11.8.
11.5 Gradient Descent in the Bellman Error
Armed with a better understanding of value function approximation and its various
objectives, we return now to the challenge of stability in o↵-policy learning. We would
like to apply the approach of stochastic gradient descent (SGD, Section 9.3), in which
updates are made that in expectation are equal to the negative gradient of an objective

270 Chapter 11: O↵-policy Methods with Approximation
function. These methods always go downhill (in expectation) in the objective and because
of this are typically stable with excellent convergence properties. Among the algorithms
investigated so far in this book, only the Monte Carlo methods are true SGD methods.
These methods converge robustly under both on-policy and o↵-policy training as well
as for general nonlinear (di↵erentiable) function approximators, though they are often
slower than semi-gradient methods with bootstrapping, which are not SGD methods.
Semi-gradient methods may diverge under o↵-policy training, as we have seen earlier in
this chapter, and under contrived cases of nonlinear function approximation (Tsitsiklis
and Van Roy, 1997). With a true SGD method such divergence would not be possible.
The appeal of SGD is so strong that great e↵ort has gone into ﬁnding a practical
way of harnessing it for reinforcement learning. The starting place of all such e↵orts is

the choice of an error or objective function to optimize. In this and the next section
we explore the origins and limits of the most popular proposed objective function, that
based on the Bellman error introduced in the previous section. Although this has been a
popular and inﬂuential approach, the conclusion that we reach here is that it is a misstep
and yields no good learning algorithms. On the other hand, this approach fails in an

11.5. Gradient Descent in the Bellman Error 271
residual-gradient algorithm (after Baird, 1995). Although the naive residual-gradient
algorithm converges robustly, it does not necessarily converge to a desirable place.
Example 11.2: A-split example,
showing the naivet´ e of the naive residual-gradient algorithmABC0001Consider the three-state episodic MRP shown to the right.
Episodes begin in state Aand then ‘split’ stochastically, half
the time going to B(and then invariably going on to terminate
with a reward of 1) and half the time going to state C(and
then invariably terminating with a reward of zero). Reward for
the ﬁrst transition, out of A, is always zero whichever way the
episode goes. As this is an episodic problem, we can take to
be 1. We also assume on-policy training, so that ⇢tis always
1, and tabular function approximation, so that the learning algorithms are free to
give arbitrary, independent values to all three states. Thus, this should be an easy
problem.

problem.
What should the values be? From A, half the time the return is 1, and half the
time the return is 0; Ashould have value1
2. From Bthe return is always 1, so its
value should be 1, and similarly from Cthe return is always 0, so its value should
be 0. These are the true values and, as this is a tabular problem, all the methods
presented previously converge to them exactly.
However, the naive residual-gradient algorithm ﬁnds di↵erent values for Band
C. It converges with Bhaving a value of3
4andChaving a value of1
4(Aconverges

correctly to1
2). These are in fact the values that minimize the TDE.
Let us compute the TDE for these values. The ﬁrst transition of each episode is
either up from A’s1
2toB’s3
4, a change of1
4, or down from A’s1
2toC’s1
4, a change
of1
4. Because the reward is zero on these transitions, and = 1, these changes are
the TD errors, and thus the squared TD error is always1
16on the ﬁrst transition.
The second transition is similar; it is either up from B’s3
4to a reward of 1 (and a
terminal state of value 0), or down from C’s1
4to a reward of 0 (again with a terminal

2toB’s3
2toC’s1
4, a change
of1
state of value 0). Thus, the TD error is always ±1
4, for a squared error of1
16on the
second step. Thus, for this set of values, the TDE on both steps is1
16.
Now let’s compute the TDE for the true values ( Bat 1, Cat 0, and Aat1
2). In this
case the ﬁrst transition is either from1
2up to 1, at B, or from1
2down to 0, at C;i n
either case the absolute error is1
2and the squared error is1
4. The second transition
has zero error because the starting value, either 1 or 0 depending on whether the

16on the
16.
2). In this
transition is from BorC, always exactly matches the immediate reward and return.
Thus the squared TD error is1
4on the ﬁrst transition and 0 on the second, for a
mean reward over the two transitions of1
8. As1
8is bigger that1
16, this solution is
worse according to the TDE . On this simple problem, the true values do not have
the smallest TDE.

272 Chapter 11: O↵-policy Methods with Approximation
A tabular representation is used in the A-split example, so the true state values can
be exactly represented, yet the naive residual-gradient algorithm ﬁnds di↵erent values,
and these values have lower TDE than do the true values. Minimizing the TDE is naive;
by penalizing all TD errors it achieves something more like temporal smoothing than
accurate prediction.
A better idea would seem to be minimizing the Bellman error. If the exact values
are learned, the Bellman error is zero everywhere. Thus, a Bellman-error-minimizing
algorithm should have no trouble with the A-split example. We cannot expect to achieve
zero Bellman error in general, as it would involve ﬁnding the true value function, which
we presume is outside the space of representable value functions. But getting close to
this ideal is a natural-seeming goal. As we have seen, the Bellman error is also closely
related to the TD error. The Bellman error for a state is the expected TD error in that

above reduces almost exactly to (11.23) , the naive residual-gradient algorithm.1But
this is naive, because the equation above involves the next state, St+1, appearing in two
expectations that are multiplied together. To get an unbiased sample of the product,
two independent samples of the next state are required, but during normal interaction
with an external environment only one is obtained. One expectation or the other can be
sampled, but not both.
There are two ways to make the residual-gradient algorithm work. One is in the case
of deterministic environments. If the transition to the next state is deterministic, then
the two samples will necessarily be the same, and the naive algorithm is valid. The
other way is to obtain twoindependent samples of the next state, St+1, from St, one for
the ﬁrst expectation and another for the second expectation. In real interaction with
an environment, this would not seem possible, but when interacting with a simulated

environment, it is. One simply rolls back to the previous state and obtains an alternate
next state before proceeding forward from the ﬁrst next state. In either of these cases the
residual-gradient algorithm is guaranteed to converge to a minimum of the BEunder the
usual conditions on the step-size parameter. As a true SGD method, this convergence is
1For state values there remains a small di↵erence in the treatment of the importance sampling ratio

⇢t.I nt h ea n a l a g o u sa c t i o n - v a l u ec a s e( w h i c hi st h em o s ti m p o r t a n tc a s ef o rc o n t r o la l g o r i t h m s ) ,t h e
residual-gradient algorithm would reduce exactly to the naive version.

11.5. Gradient Descent in the Bellman Error 273
robust, applying to both linear and nonlinear function approximators. In the linear case,
convergence is always to the unique wthat minimizes the BE.
However, there remain at least three ways in which the convergence of the residual-
gradient method is unsatisfactory. The ﬁrst of these is that empirically it is slow, much
slower that semi-gradient methods. Indeed, proponents of this method have proposed
increasing its speed by combining it with faster semi-gradient methods initially, then
gradually switching over to residual gradient for the convergence guarantee (Baird and
Moore, 1999). The second way in which the residual-gradient algorithm is unsatisfactory
is that it still seems to converge to the wrong values. It does get the right values in all
tabular cases, such as the A-split example, as for those an exact solution to the Bellman

Example 11.3: A-presplit example, a counterexample for the BEA1BC0001A2AConsider the three-state episodic MRP shown to theright: Episodes start in eitherA1orA2, with equalprobability. These two states look exactly the same tothe function approximator, like a single stateAwhosefeature representation is distinct from and unrelated tothe feature representation of the other two states,BandC, which are also distinct from each other. Speciﬁcally,the parameter of the function approximator has threecomponents, one giving the value of stateB, one giving the value of stateC, and onegiving the value of both statesA1andA2. Other than the selection of the initialstate, the system is deterministic. If it starts inA1, then it transitions toBwith areward of 0 and then on to termination with a reward of 1. If it starts inA2,t h e ni ttransitions toC, and then to termination, with both rewards zero.To a learning algorithm, seeing only the features, the system looks identical tothe A-split example. The system seems to always

giving the value of stateB, one giving the value of stateC, and onegiving the value of both statesA1andA2. Other than the selection of the initialstate, the system is deterministic. If it starts inA1, then it transitions toBwith areward of 0 and then on to termination with a reward of 1. If it starts inA2,t h e ni ttransitions toC, and then to termination, with both rewards zero.To a learning algorithm, seeing only the features, the system looks identical tothe A-split example. The system seems to always start inA, followed by eitherBorCwith equal probability, and then terminating with a 1 or a 0 dependingdeterministically on the previous state. As in the A-split example, the true valuesofBandCare 1 and 0, and the best shared value ofA1andA2is12,b ys y m m e t r y .Because this problem appears externally identical to the A-split example, wealready know what values will be found by the algorithms. Semi-gradient TDconverges to the ideal values just mentioned, while the naive residual-gradientalgorithm

always start inA, followed by eitherBorCwith equal probability, and then terminating with a 1 or a 0 dependingdeterministically on the previous state. As in the A-split example, the true valuesofBandCare 1 and 0, and the best shared value ofA1andA2is12,b ys y m m e t r y .Because this problem appears externally identical to the A-split example, wealready know what values will be found by the algorithms. Semi-gradient TDconverges to the ideal values just mentioned, while the naive residual-gradientalgorithm converges to values of34and14forBandCrespectively. All statetransitions are deterministic, so the non-naive residual-gradient algorithm will alsoconverge to these values (it is the same algorithm in this case). It follows then thatthis ‘naive’ solution must also be the one that minimizes theBE, and so it is. On adeterministic problem, the Bellman errors and TD errors are all the same, so theBEis always the same as theTDE.O p t i m i z i n gt h eBEon this example gives rise tothe same failure mode as with

converges to values of34and14forBandCrespectively. All statetransitions are deterministic, so the non-naive residual-gradient algorithm will alsoconverge to these values (it is the same algorithm in this case). It follows then thatthis ‘naive’ solution must also be the one that minimizes theBE, and so it is. On adeterministic problem, the Bellman errors and TD errors are all the same, so theBEis always the same as theTDE.O p t i m i z i n gt h eBEon this example gives rise tothe same failure mode as with the naive residual-gradient algorithm on the A-splitexample.

274 Chapter 11: O↵-policy Methods with Approximation
equation is possible. But if we examine examples with genuine function approximation,
then the residual-gradient algorithm, and indeed the BEobjective, seem to ﬁnd the
wrong value functions. One of the most telling such examples is the variation on the
A-split example known as the A- presplit example, shown on the preceding page, in which
the residual-gradient algorithm ﬁnds the same poor solution as its naive version. This
example shows intuitively that minimizing the BE(which the residual-gradient algorithm
surely does) may not be a desirable goal.
The third way in which the convergence of the residual-gradient algorithm is not
satisfactory is explained in the next section. Like the second way, the third way is also
a problem with the BEobjective itself rather than with any particular algorithm for
achieving it.
11.6 The Bellman Error is Not Learnable
The concept of learnability that we introduce in this section is di↵erent from that

commonly used in machine learning. There, a hypothesis is said to be “learnable” if
it is eciently learnable, meaning that it can be learned within a polynomial rather
than an exponential number of examples. Here we use the term in a more basic way,
to mean learnable at all, with any amount of experience. It turns out many quantities
of apparent interest in reinforcement learning cannot be learned even from an inﬁnite
amount of experiential data. These quantities are well deﬁned and can be computed

given knowledge of the internal structure of the environment, but cannot be computed
or estimated from the observed sequence of feature vectors, actions, and rewards.2We
say that they are not learnable . It will turn out that the Bellman error objective ( BE)
introduced in the last two sections is not learnable in this sense. That the Bellman error
objective cannot be learned from the observable data is probably the strongest reason
not to seek it.

To make the concept of learnability clear, let’s start with some simple examples.
Consider the two Markov reward processes3(MRPs) diagrammed below:020220www
Where two edges leave a state, both transitions are assumed to occur with equal probability,
and the numbers indicate the reward received. All the states appear the same; they all
produce the same single-component feature vector x= 1 and have approximated value
w. Thus, the only varying part of the data trajectory is the reward sequence. The left

MRP stays in the same state and emits an endless stream of 0s and 2s at random, each
with 0.5 probability. The right MRP, on every step, either stays in its current state or
2They would of course be estimated if the state sequence were observed rather than only the
corresponding feature vectors.
3All MRPs can be considered MDPs with a single action in all states; what we conclude about MRPs
here applies as well to MDPs.

11.6. The Bellman Error is Not Learnable 275
switches to the other, with equal probability. The reward is deterministic in this MRP,
always a 0 from one state and always a 2 from the other, but because the each state is
equally likely on each step, the observable data is again an endless stream of 0s and 2s at
random, identical to that produced by the left MRP. (We can assume the right MRP
starts in one of two states at random with equal probability.) Thus, even given even
an inﬁnite amount of data, it would not be possible to tell which of these two MRPs
was generating it. In particular, we could not tell if the MRP has one state or two, is
stochastic or deterministic. These things are not learnable.
This pair of MRPs also illustrates that the VEobjective (9.1) is not learnable. If
= 0, then the true values of the three states (in both MRPs), left to right, are 1, 0,
and 2. Suppose w= 1. Then the VEis 0 for the left MRP and 1 for the right MRP.

Because the VEis di↵erent in the two problems, yet the data generated has the same
distribution, the VEcannot be learned. The VEis not a unique function of the data
distribution. And if it cannot be learned, then how could the VEpossibly be useful as
an objective for learning?
If an objective cannot be learned, it does indeed draw its utility into question. In
the case of the VE, however, there is a way out. Note that the same solution, w= 1,
is optimal for both MRPs above (assuming µis the same for the two indistinguishable

states in the right MRP). Is this a coincidence, or could it be generally true that all
MDPs with the same data distribution also have the same optimal parameter vector? If
this is true—and we will show next that it is—then the VEremains a usable objective.
The VE is not learnable, but the parameter that optimizes it is!
To understand this, it is useful to bring in another natural objective function, this time
one that is clearly learnable. One error that is always observable is that between the value

sof the expectation of the squared error given that St=s. Then add and subtract the
true value of state sfrom the error (before squaring), grouping the subtracted true value
with the return and the added true value with the estimated value. Then, if you expand
the square, the most complex term will end up being zero, leaving you with (11.24). ⇤
Now let us return to the BE.T h e BEis like the VEin that it can be computed from
knowledge of the MDP but is not learnable from data. But it is not like the VEin that its
minimum solution is not learnable. The box on the next page presents a counterexample—
two MRPs that generate the same data distribution but whose minimizing parameter
vector is di↵erent, proving that the optimal parameter vector is not a function of the

276 Chapter 11: O↵-policy Methods with Approximation
Example 11.4: Counterexample to the learnability of the Bellman error
To show the full range of possibilities we need a slightly more complex pair of Markov
reward processes (MRPs) than those considered earlier. Consider the following two
MRPs:BA10-1BA0-1B01-1Where two edges leave a state, both transitions are assumed to occur with equal
probability, and the numbers indicate the reward received. The MRP on the left has
two states that are represented distinctly. The MRP on the right has three states,
two of which, BandB0, appear the same and must be given the same approximate
value. Speciﬁcally, whas two components and the value of state Ais given by the ﬁrst
component and the value of BandB0is given by the second. The second MRP has
been designed so that equal time is spent in all three states, so we can take µ(s)=1
3,
for all s.
Note that the observable data distribution is identical for the two MRPs. In both

3,
for all s.
cases the agent will see single occurrences of Afollowed by a 0, then some number
of apparent Bs, each followed by a 1 except the last, which is followed by a 1, then
we start all over again with a single Aand a 0, etc. All the statistical details are the
same as well; in both MRPs, the probability of a string of kBsi s2k.
Now suppose w=0. In the ﬁrst MRP, this is an exact solution, and the BEis
zero. In the second MRP, this solution produces a squared error in both BandB0of
1, such that BE=µ(B)1 + µ(B0)1 =2

Moreover (and unlike the earlier example for the VE) the minimizing value of w
is di↵erent for the two MRPs. For the ﬁrst MRP, w=0minimizes the BEfor any
. For the second MRP, the minimizing wis a complicated function of ,b u ti n
the limit, as !1, it is ( 1
2,0)>. Thus the solution that minimizes BEcannot be
estimated from data alone; knowledge of the MRP beyond what is revealed in the
data is required. In this sense, it is impossible in principle to pursue the BEas an
objective for learning.
It may be surprising that in the second MRP the BE-minimizing value of Ais so far
from zero. Recall that Ahas a dedicated weight and thus its value is unconstrained
by function approximation. Ais followed by a reward of 0 and transition to a state
with a value of nearly 0, which suggests vw(A) should be 0; why is its optimal
value substantially negative rather than 0? The answer is that making vw(A) negative
reduces the error upon arriving in Afrom B. The reward on this deterministic transition

is 1, which implies that Bshould have a value 1 more than A. Because B’s value is
approximately zero, A’s value is driven toward 1. The BE-minimizing value of ⇡1
2
forAis a compromise between reducing the errors on leaving and on entering A.

a source of data trajectories. To knowPis to knoweverything about the statistics of the data, but it is still less than knowing the MDP. Inparticular, the VE and BE objectives are readily computed from the MDP as described inSection 3, but these cannot be determined fromPalone.✓1✓2✓3✓4BE1BE2MDP1MDP2✓1✓2✓3✓4BE1BE2MDP1MDP2TDE RE VEThe problem can be seen in very simple, POMDP-like examples, in which the observabledata produced by two di↵erent MDPs is identical in every respect, yet the BE is di↵erent.In such a case the BE is literally not a function of the data, and thus there is no way toestimate it from data. One of the simplest examples is the pair of MDPs shown below:BA10-1BA0-1B01-1These MDPs have only one action (or, equivalently, no actions), so they are in e↵ect Markovchains. Where two edges leave a state, both possibilities are assumed to occur with equalprobability. The numbers on the edges indicate the reward emitted if that edge is traversed.The MDP on the left has two states that are

such a case the BE is literally not a function of the data, and thus there is no way toestimate it from data. One of the simplest examples is the pair of MDPs shown below:BA10-1BA0-1B01-1These MDPs have only one action (or, equivalently, no actions), so they are in e↵ect Markovchains. Where two edges leave a state, both possibilities are assumed to occur with equalprobability. The numbers on the edges indicate the reward emitted if that edge is traversed.The MDP on the left has two states that are represented distinctly; each has a separateweight so that they can take on any value. The MDP on the right has three states, twoof which,BandB, are represented identically and must be given the same approximatevalue. We can imagine that the value of stateAis given by the ﬁrst component of✓andthe value ofBandBis given by the second. Notice that the observable data is identicalfor the two MDPs. In both cases the agent will see single occurrences ofAfollowed by a0, then some number ofBs each followed by a1, except

is a complete characterization of a source of data trajectories. To knowPis to knoweverything about the statistics of the data, but it is still less than knowing the MDP. Inparticular, the VE and BE objectives are readily computed from the MDP as described inSection 3, but these cannot be determined fromPalone.✓1✓2✓3✓4BE1BE2MDP1MDP2✓1✓2✓3✓4BE1BE2MDP1MDP2TDE RE VEThe problem can be seen in very simple, POMDP-like examples, in which the observabledata produced by two di↵erent MDPs is identical in every respect, yet the BE is di↵erent.In such a case the BE is literally not a function of the data, and thus there is no way toestimate it from data. One of the simplest examples is the pair of MDPs shown below:BA10-1BA0-1B01-1These MDPs have only one action (or, equivalently, no actions), so they are in e↵ect Markovchains. Where two edges leave a state, both possibilities are assumed to occur with equalprobability. The numbers on the edges indicate the reward emitted if that edge is traversed.The MDP on the

respect, yet the BE is di↵erent.In such a case the BE is literally not a function of the data, and thus there is no way toestimate it from data. One of the simplest examples is the pair of MDPs shown below:BA10-1BA0-1B01-1These MDPs have only one action (or, equivalently, no actions), so they are in e↵ect Markovchains. Where two edges leave a state, both possibilities are assumed to occur with equalprobability. The numbers on the edges indicate the reward emitted if that edge is traversed.The MDP on the left has two states that are represented distinctly; each has a separateweight so that they can take on any value. The MDP on the right has three states, twoof which,BandB, are represented identically and must be given the same approximatevalue. We can imagine that the value of stateAis given by the ﬁrst component of✓andthe value ofBandBis given by the second. Notice that the observable data is identicalfor the two MDPs. In both cases the agent will see single occurrences ofAfollowed by a0, then some

which the observabledata produced by two di↵erent MDPs is identical in every respect, yet the BE is di↵erent.In such a case the BE is literally not a function of the data, and thus there is no way toestimate it from data. One of the simplest examples is the pair of MDPs shown below:BA10-1BA0-1B01-1These MDPs have only one action (or, equivalently, no actions), so they are in e↵ect Markovchains. Where two edges leave a state, both possibilities are assumed to occur with equalprobability. The numbers on the edges indicate the reward emitted if that edge is traversed.The MDP on the left has two states that are represented distinctly; each has a separateweight so that they can take on any value. The MDP on the right has three states, twoof which,BandB, are represented identically and must be given the same approximatevalue. We can imagine that the value of stateAis given by the ﬁrst component of✓andthe value ofBandBis given by the second. Notice that the observable data is identicalfor the two MDPs. In both

very simple, POMDP-like examples, in which the observabledata produced by two di↵erent MDPs is identical in every respect, yet the BE is di↵erent.In such a case the BE is literally not a function of the data, and thus there is no way toestimate it from data. One of the simplest examples is the pair of MDPs shown below:BA10-1BA0-1B01-1These MDPs have only one action (or, equivalently, no actions), so they are in e↵ect Markovchains. Where two edges leave a state, both possibilities are assumed to occur with equalprobability. The numbers on the edges indicate the reward emitted if that edge is traversed.The MDP on the left has two states that are represented distinctly; each has a separateweight so that they can take on any value. The MDP on the right has three states, twoof which,BandB, are represented identically and must be given the same approximatevalue. We can imagine that the value of stateAis given by the ﬁrst component of✓andthe value ofBandBis given by the second. Notice that the observable data is

1, for an overall BEofpd(B)+d(B), orp2/3 if the three states are equally weighted byd.T h e t w o M D P s ,which generate the same data, have di↵erent BEs. Thus, the BE cannot be estimated fromdata alone; knowledge of the MDP beyond what is revealed in the data is required.Moreover, the two MDPs have di↵erent minimal-BE value functions.2For the ﬁrst MDP,the minimal-BE value function is the exact solutionv=0 for any. For the second MDP,2. This is a critical observation, as it is possible for an error function to be unobservable and yet still beperfectly satisfactory for use in learning settings because the value that minimizes itcanbe determinedfrom data. For example, this is what happens with the VE. The VE is not observable from data, but its20DatadistributionMDP1MDP2MSVE1MSVE2MSREDatadistributionpolicy together completely determine the probability distribution over data trajectories.Assume for the moment that the state, action, and reward sets are all ﬁnite. Then,for any ﬁnite

is an exact solution, and the overall BE is zero. Inthe second MDP, this solution produces an error in bothBandBof 1, for an overall BEofpd(B)+d(B), orp2/3 if the three states are equally weighted byd.T h e t w o M D P s ,which generate the same data, have di↵erent BEs. Thus, the BE cannot be estimated fromdata alone; knowledge of the MDP beyond what is revealed in the data is required.Moreover, the two MDPs have di↵erent minimal-BE value functions.2For the ﬁrst MDP,the minimal-BE value function is the exact solutionv=0 for any. For the second MDP,2. This is a critical observation, as it is possible for an error function to be unobservable and yet still beperfectly satisfactory for use in learning settings because the value that minimizes itcanbe determinedfrom data. For example, this is what happens with the VE. The VE is not observable from data, but its20w⇤w⇤1w⇤2w⇤3w⇤4VE1VE2REBE2BE1PBETDEMonte CarloobjectivesBootstrappingobjectivesFigure 11.4: Causal relationships among the data distribution, MDPs,

Left, Monte Carlo objectives: Two di↵erent MDPs can produce the same data distribution
yet also produce di↵erent VEs, proving that the VEobjective cannot be determined from data
and is not learnable. However, all such VEs must have the same optimal parameter vector, w⇤!
Moreover, this same w⇤can be determined from another objective, the RE,w h i c h isuniquely
determined from the data distribution. Thus w⇤and the REare learnable even though the VEs
are not. Right, Bootstrapping objectives: Two di↵erent MDPs can produce the same data
distribution yet also produce di↵erent BEsand have di↵erent minimizing parameter vectors;
these are not learnable from the data distribution. The PBE andTDE objectives and their
(di↵erent) minima can be directly determined from data and thus are learnable.
data and thus cannot be learned from it. The other bootstrapping objectives that we
have considered, the PBE andTDE , can be determined from data (are learnable) and

determine optimal solutions that are in general di↵erent from each other and the BE
minimums. The general case is summarized in the right side of Figure 11.4.
Thus, the BEis not learnable; it cannot be estimated from feature vectors and other
observable data. This limits the BEto model-based settings. There can be no algorithm
that minimizes the BEwithout access to the underlying MDP states beyond the feature
vectors. The residual-gradient algorithm is only able to minimize BEbecause it is allowed

to double sample from the same state—not a state that has the same feature vector,
but one that is guaranteed to be the same underlying state. We can see now that there
is no way around this. Minimizing the BErequires some such access to the nominal,
underlying MDP. This is an important limitation of the BEbeyond that identiﬁed in the
A-presplit example on page 273. All this directs more attention toward the PBE.

11.7. Gradient-TD Methods 279
Substituting these expectations for the three factors in our expression for the gradient of
thePBE, we get
rPBE( w)=2 E⇥
⇢t(xt+1xt)x>
t⇤
E⇥
xtx>
t⇤1E[⇢ttxt]. (11.27)
It might not be obvious that we have made any progress by writing the gradient in this
form. It is a product of three expressions and the ﬁrst and last are not independent.
They both depend on the next feature vector xt+1; we cannot simply sample both of
these expectations and then multiply the samples. This would give us a biased estimate
of the gradient just as in the naive residual-gradient algorithm.
Another idea would be to estimate the three expectations separately and then combine
them to produce an unbiased estimate of the gradient. This would work, but would
require a lot of computational resources, particularly to store the ﬁrst two expectations,
which are d⇥dmatrices, and to compute the inverse of the second. This idea can be

improved. If two of the three expectations are estimated and stored, then the third could
be sampled and used in conjunction with the two stored quantities. For example, you
could store estimates of the second two quantities (using the increment inverse-updating
techniques in Section 9.8) and then sample the ﬁrst expression. Unfortunately, the overall
algorithm would still be of quadratic complexity (of order O(d2)).
The idea of storing some estimates separately and then combining them with samples

is a good one and is also used in Gradient-TD methods. Gradient-TD methods estimate
and store the product of the second two factors in (11.27) . These factors are a d⇥d
matrix and a d-vector, so their product is just a d-vector, like witself. We denote this
second learned vector as v:
v⇡E⇥
xtx>
t⇤1E[⇢ttxt]. (11.28)
This form is familiar to students of linear supervised learning. It is the solution to a linear
least-squares problem that tries to approximate ⇢ttfrom the features. The standard

11.8. Emphatic-TD Methods 281
an optimal solution, ˆv(s) = 0, for all s, for which wwould have to be proportional to
(1,1,1,1,1,1,4,2)>. After 1000 iterations we are still far from an optimal solution, as
we can see from the VE, which remains almost 2. The system is actually converging to
an optimal solution, but progress is extremely slow because the PBE is already so close
to zero.
GTD2 and TDC both involve two learning processes, a primary one for wand a
secondary one for v. The logic of the primary learning process relies on the secondary
learning process having ﬁnished, at least approximately, whereas the secondary learning
process proceeds without being inﬂuenced by the ﬁrst. We call this sort of asymmetrical
dependence a cascade . In cascades we often assume that the secondary learning process
is proceeding faster and thus is always at its asymptotic value, ready and accurate to
assist the primary learning process. The convergence proofs for these methods often make

this assumption explicitly. These are called two-time-scale proofs. The fast time scale is
that of the secondary learning process, and the slower time scale is that of the primary
learning process. If ↵is the step size of the primary learning process, and is the step
size of the secondary learning process, then these convergence proofs will typically require
that in the limit !0 and↵
!0.
Gradient-TD methods are currently the most well understood and widely used stable
o↵-policy methods. There are extensions to action values and control (GQ, Maei et al.,

to nonlinear function approximation (Maei et al., 2009). There have also been proposed
hybrid algorithms midway between semi-gradient TD and gradient TD (Hackman, 2012;
White and White, 2016). Hybrid-TD algorithms behave like Gradient-TD algorithms in
states where the target and behavior policies are very di↵erent, and behave like semi-
gradient algorithms in states where the target and behavior policies are the same. Finally,
the Gradient-TD idea has been combined with the ideas of proximal methods and control
variates to produce more ecient methods (Mahadevan et al., 2014; Du et al., 2017).
11.8 Emphatic-TD Methods
We turn now to the second major strategy that has been extensively explored for obtaining
a cheap and ecient o↵-policy learning method with function approximation. Recall
that linear semi-gradient TD methods are ecient and stable when trained under the
on-policy distribution, and that we showed in Section 9.4 that this has to do with the

positive deﬁniteness of the matrix A(9.11)4and the match between the on-policy state
distribution µ⇡and the state-transition probabilities p(s|s, a) under the target policy. In
o↵-policy learning, we reweight the state transitions using importance sampling so that
they become appropriate for learning about the target policy, but the state distribution
is still that of the behavior policy. There is a mismatch. A natural idea is to somehow
reweight the states, emphasizing some and de-emphasizing others, so as to return the

distribution of updates to the on-policy distribution. There would then be a match,
and stability and convergence would follow from existing results. This is the idea of
4In the o↵-policy case, the matrix Ais generally deﬁned as Es⇠b⇥
x(s)E⇥
x(St+1)>St=s, A t⇠⇡⇤⇤
.

282 Chapter 11: O↵-policy Methods with Approximation
Emphatic-TD methods, ﬁrst introduced for on-policy training in Section 9.11.
Actually, the notion of “the on-policy distribution” is not quite right, as there are many
on-policy distributions, and any one of these is sucient to guarantee stability. Consider
an undiscounted episodic problem. The way episodes terminate is fully determined by the
transition probabilities, but there may be several di↵erent ways the episodes might begin.
However the episodes start, if all state transitions are due to the target policy, then the
state distribution that results is an on-policy distribution. You might start close to the
terminal state and visit only a few states with high probability before ending the episode.
Or you might start far away and pass through many states before terminating. Both are
on-policy distributions, and training on both with a linear semi-gradient method would

be guaranteed to be stable. However the process starts, an on-policy distribution results
as long as all states encountered are updated up until termination.
If there is discounting, it can be treated as partial or probabilistic termination for these
purposes. If =0.9, then we can consider that with probability 0 .1 the process terminates
on every time step and then immediately restarts in the state that is transitioned to. A
discounted problem is one that is continually terminating and restarting with probability

1on every step. This way of thinking about discounting is an example of a more
general notion of pseudo termination —termination that does not a↵ect the sequence of
state transitions, but does a↵ect the learning process and the quantities being learned.
This kind of pseudo termination is important to o↵-policy learning because the restarting
is optional—remember we can start any way we want to—and the termination relieves
the need to keep including encountered states within the on-policy distribution. That is,

and the VEgoes to zero. These trajectories are obtained by iteratively computing the
expectation of the parameter vector trajectory without any of the variance due to sampling
of transitions and rewards. We do not show the results of applying the Emphatic-TD
algorithm directly because its variance on Baird’s counterexample is so high that it is
nigh impossible to get consistent results in computational experiments. The algorithm
converges to the optimal solution in theory on this problem, but in practice it does not.
We turn to the topic of reducing the variance of all these algorithms in the next section.

11.9. Reducing Variance 283
Sweepsw1–w6w7w8pVE10520-501000Figure 11.6: The behavior of the one-step Emphatic-TD algorithm in expectation on Baird’s
counterexample. The step size was ↵=0.03.
11.9 Reducing Variance
O↵-policy learning is inherently of greater variance than on-policy learning. This is not
surprising; if you receive data less closely related to a policy, you should expect to learn
less about the policy’s values. In the extreme, one may be able to learn nothing. You
can’t expect to learn how to drive by cooking dinner, for example. Only if the target and
behavior policies are related, if they visit similar states and take similar actions, should
one be able to make signiﬁcant progress in o↵-policy training.
On the other hand, any policy has many neighbors, many similar policies with con-
siderable overlap in states visited and actions chosen, and yet which are not identical.
The raison d’ˆ etre of o↵-policy learning is to enable generalization to this vast number

of related-but-not-identical policies. The problem remains of how to make the best use
of the experience. Now that we have some methods that are stable in expected value
(if the step sizes are set right), attention naturally turns to reducing the variance of the
estimates. There are many possible ideas, and we can just touch on a few of them in this
introductory text.
Why is controlling variance especially critical in o↵-policy methods based on importance

sampling? As we have seen, importance sampling often involves products of policy ratios.
The ratios are always one in expectation (5.13) , but their actual values may be very high
or as low as zero. Successive ratios are uncorrelated, so their products are also always one
in expected value, but they can be of very high variance. Recall that these ratios multiply
the step size in SGD methods, so high variance means taking steps that vary greatly in
their sizes. This is problematic for SGD because of the occasional very large steps. They

must not be so large as to take the parameter to a part of the space with a very di↵erent
gradient. SGD methods rely on averaging over multiple steps to get a good sense of
the gradient, and if they make large moves from single samples they become unreliable.

284 Chapter 11: O↵-policy Methods with Approximation
If the step-size parameter is set small enough to prevent this, then the expected step
can end up being very small, resulting in very slow learning. The notions of momentum
(Derthick, 1984), of Polyak-Ruppert averaging (Polyak, 1990; Ruppert, 1988; Polyak and
Juditsky, 1992), or further extensions of these ideas may signiﬁcantly help. Methods for
adaptively setting separate step sizes for di↵erent components of the parameter vector
are also pertinent (e.g., Jacobs, 1988; Sutton, 1992b, c), as are the “importance weight
aware” updates of Karampatziakis and Langford (2010).
In Chapter 5 we saw how weighted importance sampling is signiﬁcantly better behaved,
with lower variance updates, than ordinary importance sampling. However, adapting
weighted importance sampling to function approximation is challenging and can probably
only be done approximately with O(d) complexity (Mahmood and Sutton, 2015).

The Tree Backup algorithm (Section 7.5) shows that it is possible to perform some
o↵-policy learning without using importance sampling. This idea has been extended to
the o↵-policy case to produce stable and more ecient methods by Munos, Stepleton,
Harutyunyan, and Bellemare (2016) and by Mahmood, Yu and Sutton (2017).
Another, complementary strategy is to allow the target policy to be determined in
part by the behavior policy, in such a way that it never can be so di↵erent from it to

create large importance sampling ratios. For example, the target policy can be deﬁned by
reference to the behavior policy, as in the “recognizers” proposed by Precup et al. (2006).
11.10 Summary
O↵-policy learning is a tempting challenge, testing our ingenuity in designing stable and
ecient learning algorithms. Tabular Q-learning makes o↵-policy learning seem easy,
and it has natural generalizations to Expected Sarsa and to the Tree Backup algorithm.

But as we have seen in this chapter, the extension of these ideas to signiﬁcant function
approximation, even linear function approximation, involves new challenges and forces us
to deepen our understanding of reinforcement learning algorithms.
Why go to such lengths? One reason to seek o↵-policy algorithms is to give ﬂexibility
in dealing with the tradeo↵ between exploration and exploitation. Another is to free
behavior from learning, and avoid the tyranny of the target policy. TD learning appears

to hold out the possibility of learning about multiple things in parallel, of using one
stream of experience to solve many tasks simultaneously. We can certainly do this in
special cases, just not in every case that we would like to or as eciently as we would
like to.
In this chapter we divided the challenge of o↵-policy learning into two parts. The
ﬁrst part, correcting the targets of learning for the behavior policy, is straightforwardly

like to.
dealt with using the techniques devised earlier for the tabular case, albeit at the cost of
increasing the variance of the updates and thereby slowing learning. High variance will
probably always remains a challenge for o↵-policy learning.
The second part of the challenge of o↵-policy learning emerges as the instability
of semi-gradient TD methods that involve bootstrapping. We seek powerful function

Bibliographical and Historical Remarks 285
approximation, o↵-policy learning, and the eciency and ﬂexibility of bootstrapping
TD methods, but it is challenging to combine all three aspects of this deadly triad in
one algorithm without introducing the potential for instability. There have been several
attempts. The most popular has been to seek to perform true stochastic gradient descent
(SGD) in the Bellman error (a.k.a. the Bellman residual). However, our analysis concludes
that this is not an appealing goal in many cases, and that anyway it is impossible to
achieve with a learning algorithm—the gradient of the BEis not learnable from experience
that reveals only feature vectors and not underlying states. Another approach, Gradient-
TD methods, performs SGD in the projected Bellman error. The gradient of the PBE
is learnable with O(d) complexity, but at the cost of a second parameter vector with a
second step size. The newest family of methods, Emphatic-TD methods, reﬁne an old idea

for reweighting updates, emphasizing some and de-emphasizing others. In this way they
restore the special properties that make on-policy learning stable with computationally
simple semi-gradient methods.
The whole area of o↵-policy learning is relatively new and unsettled. Which methods
are best or even adequate is not yet clear. Are the complexities of the new methods
introduced at the end of this chapter really necessary? Which of them can be combined
e↵ectively with variance reduction methods? The potential for o↵-policy learning remains

tantalizing, the best way to achieve it still a mystery.
11.1 The ﬁrst semi-gradient method was linear TD( ) (Sutton, 1988). The name
“semi-gradient” is more recent (Sutton, 2015a). Semi-gradient o↵-policy TD(0)
with general importance-sampling ratio may not have been explicitly stated until
Sutton, Mahmood, and White (2016), but the action-value forms were introduced
by Precup, Sutton, and Singh (2000), who also did eligibility trace forms of these

algorithms (see Chapter 12). Their continuing, undiscounted forms have not
been signiﬁcantly explored. The n-step forms given here are new.
11.2 The earliest w-to-2 wexample was given by Tsitsiklis and Van Roy (1996), who
also introduced the speciﬁc counterexample in the box on page 263. Baird’s
counterexample is due to Baird (1995), though the version we present here is
slightly modiﬁed. Averaging methods for function approximation were developed
by Gordon (1995, 1996b). Other examples of instability with o↵-policy DP

methods and more complex methods of function approximation are given by
Boyan and Moore (1995). Bradtke (1993) gives an example in which Q-learning
using linear function approximation in a linear quadratic regulation problem
converges to a destabilizing policy.
11.3 The deadly triad was ﬁrst identiﬁed by Sutton (1995b) and thoroughly analyzed
by Tsitsiklis and Van Roy (1997). The name “deadly triad” is due to Sutton
(2015a).
11.4 This kind of linear analysis was pioneered by Tsitsiklis and Van Roy (1996; 1997),

286 Chapter 11: O↵-policy Methods with Approximation
including the dynamic programming operator. Diagrams like Figure 11.3 were
introduced by Lagoudakis and Parr (2003).
What we have called the Bellman operator, and denoted B⇡, is more commonly
denoted T⇡and called a “dynamic programming operator,” while a generalized
form, denoted T(), is called the “TD( ) operator” (Tsitsiklis and Van Roy, 1996,
1997).
11.5 TheBEwas ﬁrst proposed as an objective function for dynamic programming by
Schweitzer and Seidmann (1985). Baird (1995, 1999) extended it to TD learning
based on stochastic gradient descent. In the literature, BEminimization is often
referred to as Bellman residual minimization.
The earliest A-split example is due to Dayan (1992). The two forms given here
were introduced by Sutton et al. (2009a).
11.6 The contents of this section are new to this text.
11.7 Gradient-TD methods were introduced by Sutton, Szepesv´ ari, and Maei (2009b).

The methods highlighted in this section were introduced by Sutton et al. (2009a)
and Mahmood et al. (2014). A major extension to proximal TD methods was
developed by Mahadeval et al. (2014). The most sensitive empirical investigations
to date of Gradient-TD and related methods are given by Geist and Scherrer
(2014), Dann, Neumann, and Peters (2014), White (2015), and Ghiassian, White,
White, and Sutton (in preparation). Recent developments in the theory of
Gradient-TD methods are presented by Yu (2017).

11.8 Emphatic-TD methods were introduced by Sutton, Mahmood, and White (2016).
Full convergence proofs and other theory were later established by Yu (2015;
2016; Yu, Mahmood, and Sutton, 2017), Hallak, Tamar, and Mannor (2015), and
Hallak, Tamar, Munos, and Mannor (2016).

methods are augmented with eligibility traces, they produce a family of methods spanning
a spectrum that has Monte Carlo methods at one end ( =1) and one-step TD methods
at the other ( = 0). In between are intermediate methods that are often better than
either extreme method. Eligibility traces also provide a way of implementing Monte Carlo
methods online and on continuing problems without episodes.
Of course, we have already seen one way of unifying TD and Monte Carlo methods: the
n-step TD methods of Chapter 7. What eligibility traces o↵er beyond these is an elegant
algorithmic mechanism with signiﬁcant computational advantages. The mechanism is
a short-term memory vector, the eligibility trace zt2Rd, that parallels the long-term
weight vector wt2Rd. The rough idea is that when a component of wtparticipates in
producing an estimated value, then the corresponding component of ztis bumped up and
then begins to fade away. Learning will then occur in that component of wtif a nonzero

TD error occurs before the trace falls back to zero. The trace-decay parameter 2[0,1]
determines the rate at which the trace falls.
The primary computational advantage of eligibility traces over n-step methods is that
only a single trace vector is required rather than a store of the last nfeature vectors.
Learning also occurs continually and uniformly in time rather than being delayed and
then catching up at the end of the episode. In addition learning can occur and a↵ect

behavior immediately after a state is encountered rather than being delayed nsteps.
Eligibility traces illustrate that a learning algorithm can sometimes be implemented in
a di↵erent way to obtain computational advantages. Many algorithms are most naturally
formulated and understood as an update of a state’s value based on events that follow
that state over multiple future time steps. For example, Monte Carlo methods (Chapter 5)
update a state based on all the future rewards, and n-step TD methods (Chapter 7)
287

288 Chapter 12: Eligibility Traces
update based on the next nrewards and state nsteps in the future. Such formulations,
based on looking forward from the updated state, are called forward views . Forward views
are always somewhat complex to implement because the update depends on later things
that are not available at the time. However, as we show in this chapter it is often possible
to achieve nearly the same updates—and sometimes exactly the same updates—with an
algorithm that uses the current TD error, looking backward to recently visited states
using an eligibility trace. These alternate ways of looking at and implementing learning
algorithms are called backward views . Backward views, transformations between forward
views and backward views, and equivalences between them, date back to the introduction
of temporal di↵erence learning but have become much more powerful and sophisticated
since 2014. Here we present the basics of the modern view.

As usual, ﬁrst we fully develop the ideas for state values and prediction, then extend
them to action values and control. We develop them ﬁrst for the on-policy case then
extend them to o↵-policy learning. Our treatment pays special attention to the case of
linear function approximation, for which the results with eligibility traces are stronger.
All these results apply also to the tabular and state aggregation cases because these are
special cases of linear function approximation.
12.1 The -return

return, for n1, is a valid update target for a tabular learning update, just as it is for
an approximate SGD learning update such as (9.7).
Now we note that a valid update can be done not just toward any n-step return, but
toward any average ofn-step returns for di↵erent ns. For example, an update can be
done toward a target that is half of a two-step return and half of a four-step return:
1
2Gt:t+2+1
2Gt:t+4. Any set of n-step returns can be averaged in this way, even an inﬁnite
set, as long as the weights on the component returns are positive and sum to 1. The
composite return possesses an error reduction property similar to that of individual n-step
returns (7.3) and thus can be used to construct updates with guaranteed convergence
properties. Averaging produces a substantial new range of algorithms. For example, one
could average one-step and inﬁnite-step returns to obtain another way of interrelating TD
and Monte Carlo methods. In principle, one could even average experience-based updates

with DP updates to get a simple combination of experience-based and model-based
methods (cf. Chapter 8).
An update that averages simpler component updates is called a compound update .T h e
backup diagram for a compound update consists of the backup diagrams for each of the
component updates with a horizontal line above them and the weighting fractions below.

algorithm alongside that of the n-step TD methods. In both case, intermediate values of the
bootstrapping parameter ( orn) performed best. The results with the o✏ine -return algorithm
are slightly better at the best values of ↵and,a n da th i g h ↵.
The approach that we have been taking so far is what we call the theoretical, or
forward , view of a learning algorithm. For each state visited, we look forward in time to
all the future rewards and decide how best to combine them. We might imagine ourselves
riding the stream of states, looking forward from each state to determine its update, as
suggested by Figure 12.4. After looking forward from and updating one state, we move
on to the next and never have to work with the preceding state again. Future states,
on the other hand, are viewed and processed repeatedly, once from each vantage point
preceding them.

eligibility traces. Here we will show empirically that it approximates the o✏ine -return
algorithm presented in the previous section.
TD() improves over the o✏ine -return algorithm in three ways. First it updates
the weight vector on every step of an episode rather than only at the end, and thus
its estimates may be better sooner. Second, its computations are equally distributed
in time rather than all at the end of the episode. And third, it can be applied to
continuing problems rather than just to episodic problems. In this section we present the
semi-gradient version of TD( ) with function approximation.
With function approximation, the eligibility trace is a vector zt2Rdwith the same
number of components as the weight vector wt. Whereas the weight vector is a long-term
memory, accumulating over the lifetime of the system, the eligibility trace is a short-term
memory, typically lasting less time than the length of an episode. Eligibility traces assist

294 Chapter 12: Eligibility Traces
TD() is oriented backward in time. At each moment we look at the current TD error
and assign it backward to each prior state according to how much that state contributed
to the current eligibility trace at that time. We might imagine ourselves riding along the
stream of states, computing TD errors, and shouting them back to the previously visited
states, as suggested by Figure 12.5. Where the TD error and traces come together, we
get the update given by (12.7), changing the values of those past states for when they
occur again in the future.
To better understand the backward view of TD( ), consider what happens at various
values of .I f = 0, then by (12.5) the trace at tis exactly the value gradient
corresponding to St.T h u st h eT D ( ) update (12.7) reduces to the one-step semi-gradient
TD update treated in Chapter 9 (and, in the tabular case, to the simple TD rule (6.2)).
This is why that algorithm was called TD(0). In terms of Figure 12.5, TD(0) is the case

steps it needs to be discounted, like any reward in a return, by k, which is just what
the falling eligibility trace achieves. If = 1 and = 1, then the eligibility traces do not
decay at all with time. In this case the method behaves like a Monte Carlo method for
an undiscounted, episodic task. If = 1, the algorithm is also known as TD(1).
TD(1) is a way of implementing Monte Carlo algorithms that is more general than those
presented earlier and that signiﬁcantly increases their range of applicability. Whereas
the earlier Monte Carlo methods were limited to episodic tasks, TD(1) can be applied to
discounted continuing tasks as well. Moreover, TD(1) can be performed incrementally
and online. One disadvantage of Monte Carlo methods is that they learn nothing from
an episode until it is over. For example, if a Monte Carlo control method takes an action
that produces a very poor reward but does not end the episode, then the agent’s tendency

to repeat the action will be undiminished during the episode. Online TD(1), on the other
hand, learns in an n-step TD way from the incomplete ongoing episode, where the n
steps are all the way up to the current step. If something unusually good or bad happens
during an episode, control methods based on TD(1) can learn immediately and alter their
behavior on that same episode.
It is revealing to revisit the 19-state random walk example (Example 7.1) to see how
well TD( ) does in approximating the o✏ine -return algorithm. The results for both

298 Chapter 12: Eligibility Traces
ˆv(S1,w0). Note that this is exactly what G
0:1is, with the sum in the ﬁrst term of the
equation degenerating to zero. Using this update target, we construct w1. Then, after
advancing the data horizon to step 2, what do we do? We have new data in the form of
R2andS2, as well as the new w1, so now we can construct a better update target G
0:2
for the ﬁrst update from S0as well as a better update target G
1:2for the second update
from S1. Using these improved targets, we redo the updates at S1andS2, starting again
from w0,t op r o d u c e w2. Now we advance the horizon to step 3 and repeat, going all the
way back to produce three new targets, redoing all updates starting from the original w0
to produce w3, and so on. Each time the horizon is advanced, all the updates are redone
starting from w0using the weight vector from the preceding horizon.
This conceptual algorithm involves multiple passes over the episode, one at each

horizon, each generating a di↵erent sequence of weight vectors. To describe it clearly we
have to distinguish between the weight vectors computed at the di↵erent horizons. Let us
usewh
tto denote the weights used to generate the value at time tin the sequence up to
horizon h. The ﬁrst weight vector wh
0in each sequence is that inherited from the previous
episode (so they are the same for all h), and the last weight vector wh
hin each sequence

component of the feature vector was 1 or 0:
zi,t.=⇢
1i f xi,t=1
z i,t1otherwise .(12.12)
Nowadays, we see replacing traces as crude approximations to dutch traces, which largely
supercede them. Dutch traces usually perform better than replacing traces and have a
clearer theoretical basis. Accumulating traces remain of interest for nonlinear function
approximations where dutch traces are not available.
12.6 *Dutch Traces in Monte Carlo Learning
Although eligibility traces are closely associated historically with TD learning, in fact
they have nothing to do with it. In fact, eligibility traces arise even in Monte Carlo
learning, as we show in this section. We show that the linear MC algorithm (Chapter 9),
taken as a forward view, can be used to derive an equivalent yet computationally cheaper
backward-view algorithm using dutch traces. This is the only equivalence of forward- and
backward-views that we explicitly demonstrate in this book. It gives some of the ﬂavor

of the proof of equivalence of true online TD( ) and the online -return algorithm, but is
The linear version of the gradient Monte Carlo prediction algorithm (page 202) makes
the following sequence of updates, one for each time step of the episode:
wt+1.=wt+↵⇥
Gw>
txt⇤
xt,0t<T . (12.13)
To simplify the example, we assume here that the return Gis a single reward received at
the end of the episode (this is why Gis not subscripted by time) and that there is no
discounting. In this case the update is also known as the Least Mean Square (LMS) rule.

Gw>
txt⇤
As a Monte Carlo algorithm, all the updates depend on the ﬁnal reward/return, so none
can be made until the end of the episode. The MC algorithm is an o✏ine algorithm and
we do not seek to improve this aspect of it. Rather we seek merely an implementation of
this algorithm with computational advantages. We will still update the weight vector
only at the end of the episode, but we will do some computation during each step of the
episode and less at its end. This will give a more equal distribution of computation— O(d)

per step—and also remove the need to store the feature vectors at each step for use later
at the end of each episode. Instead, we will introduce an additional vector memory, the
eligibility trace, keeping in it a summary of all the feature vectors seen so far. This will
be sucient to eciently recreate exactly the same overall update as the sequence of MC

12.7. Sarsa( ) 303
The auxiliary vectors, atandzt, are updated on each time step t<T and then, at time
Twhen Gis observed, they are used in (12.14) to compute wT. In this way we achieve
exactly the same ﬁnal result as the MC/LMS algorithm that has poor computational
properties (12.13) , but now with an incremental algorithm whose time and memory
complexity per step is O(d). This is surprising and intriguing because the notion of
an eligibility trace (and the dutch trace in particular) has arisen in a setting without
temporal-di↵erence (TD) learning (in contrast to van Seijen and Sutton, 2014). It seems
eligibility traces are not speciﬁc to TD learning at all; they are more fundamental than
that. The need for eligibility traces seems to arise whenever one tries to learn long-term
predictions in an ecient manner.
12.7 Sarsa( )
Very few changes in the ideas already presented in this chapter are required in order to
extend eligibility-traces to action-value methods. To learn approximate action values,

and either accumulating or replacing traces is given in the box on the next page. This
pseudocode highlights a few optimizations possible in the special case of binary features
(features are either active (=1) or inactive (=0).
Example 12.1: Traces in Gridworld The use of eligibility traces can substantially
increase the eciency of control algorithms over one-step methods and even over n-step
methods. The reason for this is illustrated by the gridworld example below.
Path takenAction values increasedby one-step SarsaAction values increasedby Sarsa() with =0.9GGGPath takenAction values increasedby one-step SarsaAction values increasedby Sarsa(!) with !=0.9by 10-step SarsaGGGPath takenAction values increasedby one-step SarsaAction values increased by 10-step SarsaGGGλλThe ﬁrst panel shows the path taken by an agent in a single episode. The initial estimated
values were zero, and all rewards were zero except for a positive reward at the goal

location marked by G. The arrows in the other panels show, for various algorithms, which
action-values would be increased, and by how much, upon reaching the goal. A one-step
method would increment only the last action value, whereas an n-step method would
equally increment the last nactions’ values, and an eligibility trace method would update
all the action values up to the beginning of the episode, to di↵erent degrees, fading with
recency. The fading strategy is often the best.

of state–action feature vectors xt=x(St,At) instead of state feature vectors xt=x(St).
Pseudocode for the resulting ecient algorithm, called true online Sarsa( )is given in
the box on the next page. The ﬁgure below compares the performance of various versions
of Sarsa( ) on the Mountain Car example.True Online TD()
00.511.500.20.40.60.81
step−sizeRMS           error         TD(λ), accumulating traces − Task 1λ = 0λ = 0.975λ = 1λ = 0.95λ = 0.2λ = 0.1
step−sizeRMS           error         TD(λ), replacing traces − Task 1λ = 0λ = 100.511.500.20.40.60.81
step−sizeRMS           error         true online TD(λ) − Task 1λ = 0λ = 100.511.500.20.40.60.81
step−sizeRMS           error         TD(λ), accumulating traces − Task 2λ = 1λ = 0
step−sizeRMS          error         TD(λ), replacing traces − Task 2
step−sizeRMS           error         true online TD(λ) − Task 2

α0return
  Sarsa(λ), replacing, clearingSarsa(λ), replacing, no clearingSarsa(λ), accumulatingtrue online Sarsa(λ)
Figure 4.Average return over ﬁrst 20 episodes on mountain cartask for=0.9and different↵0. Results are averaged over 100independent runs.AcknowledgementsThe authors thank Hado van Hasselt and Rupam Mahmoodfor extensive discussions leading to the reﬁnement of theseideas. This work was supported by grants from AlbertaInnovates – Technology Futures and the National Scienceand Engineering Research Council of Canada.Mountain CarReward per episodeaveraged overﬁrst 20 episodesand 100 runs×  number of tilings (8)↵True online Sarsa(λ)Sarsa(λ) with replacing tracesSarsa(λ) with replacing tracesand clearing the traces of other actionsSarsa(λ) with accumulating tracesFigure 12.11: Summary comparison of Sarsa( ) algorithms on the Mountain Car task. True
online Sarsa( ) performed better than regular Sarsa( ) with both accumulating and replacing

312 Chapter 12: Eligibility Traces
make irrevocable updates as a trajectory unfolds, whereas true Monte Carlo methods
would make no update for a trajectory if any action within it has zero probability under
the target policy. In particular, all of these methods, even at = 1, still bootstrap in
the sense that their targets depend on the current value estimates—it’s just that the
dependence cancels out in expected value. Whether this is a good or bad property in
practice is another question. Recently, methods have been proposed that do achieve an
exact equivalence (Sutton, Mahmood, Precup and van Hasselt, 2014). These methods
require an additional vector of “provisional weights” that keep track of updates which
have been made but may need to be retracted (or emphasized) depending on the actions
taken later. The state and state–action versions of these methods are called PTD( ) and
PQ() respectively, where the ‘P’ stands for Provisional.

The practical consequences of all these new o↵-policy methods have not yet been
established. Undoubtedly, issues of high variance will arise as they do in all o↵-policy
methods using importance sampling (Section 11.9).
If<1, then all these o↵-policy algorithms involve bootstrapping and the deadly
triad applies (Section 11.3), meaning that they can be guaranteed stable only for the
tabular case, for state aggregation, and for other limited forms of function approximation.

For linear and more-general forms of function approximation the parameter vector may
diverge to inﬁnity as in the examples in Chapter 11. As we discussed there, the challenge
of o↵-policy learning has two parts. O↵-policy eligibility traces deal e↵ectively with the
ﬁrst part of the challenge, correcting for the expected value of the targets, but not at
all with the second part of the challenge, having to do with the distribution of updates.
Algorithmic strategies for meeting the second part of the challenge of o↵-policy learning

with eligibility traces are summarized in Section 12.11.
Exercise 12.13 What are the dutch-trace and replacing-trace versions of o↵-policy
eligibility traces for state-value and action-value methods? ⇤
12.10 Watkins’s Q( ) to Tree-Backup( )
Several methods have been proposed over the years to extend Q-learning to eligibility
traces. The original is Watkins’s Q( ), which decays its eligibility traces in the usual way
as long as a greedy action was taken, then cuts the traces to zero after the ﬁrst non-greedy

action. The backup diagram for Watkins’s Q( ) is shown in Figure 12.12. In Chapter 6,
we uniﬁed Q-learning and Expected Sarsa in the o↵-policy version of the latter, which
includes Q-learning as a special case, and generalizes it to arbitrary target policies, and
in the previous section of this chapter we completed our treatment of Expected Sarsa by
generalizing it to o↵-policy eligibility traces. In Chapter 7, however, we distinguished
n-step Expected Sarsa from n-step Tree Backup, where the latter retained the property

of not using importance sampling. It remains then to present the eligibility trace version
of Tree Backup, which we will call Tree-Backup( ), orTB()for short. This is arguably
the true successor to Q-learning because it retains its appealing absence of importance
sampling even though it can be applied to o↵-policy data.

TB() would have to be combined with one of the methods presented in the next section.
⇤Exercise 12.14 How might Double Expected Sarsa be extended to eligibility traces? ⇤
12.11 Stable O↵-policy Methods with Traces
Several methods using eligibility traces have been proposed that achieve guarantees
of stability under o↵-policy training, and here we present four of the most important
using this book’s standard notation, including general bootstrapping and discounting
functions. All are based on either the Gradient-TD or the Emphatic-TD ideas presented
in Sections 11.7 and 11.8. All the algorithms assume linear function approximation,
though extensions to nonlinear function approximation can also be found in the literature.
GTD( )is the eligibility-trace algorithm analogous to TDC, the better of the two
state-value Gradient-TD prediction algorithms discussed in Section 11.7. Its goal is to
learn a parameter wtsuch that ˆv(s,w).=w>
tx(s)⇡v⇡(s), even from data that is due to

Sutton, 2016).
12.12 Implementation Issues
It might at ﬁrst appear that tabular methods using eligibility traces are much more
complex than one-step methods. A naive implementation would require every state (or
state–action pair) to update both its value estimate and its eligibility trace on every time
step. This would not be a problem for implementations on single-instruction, multiple-
data, parallel computers or in plausible artiﬁcial neural network (ANN) implementations,
but it is a problem for implementations on conventional serial computers. Fortunately,
for typical values of andthe eligibility traces of almost all states are almost always
nearly zero; only those states that have recently been visited will have traces signiﬁcantly
greater than zero and only these few states need to be updated to closely approximate
these algorithms.
In practice, then, implementations on conventional computers may keep track of and

update only the few traces that are signiﬁcantly greater than zero. Using this trick, the
computational expense of using traces in tabular methods is typically just a few times
that of a one-step method. The exact multiple of course depends on andand on the
expense of the other computations. Note that the tabular case is in some sense the worst
case for the computational complexity of eligibility traces. When function approximation
is used, the computational advantages of not using traces generally decrease. For example,

12.13. Conclusions 317
12.13 Conclusions
Eligibility traces in conjunction with TD errors provide an ecient, incremental way of
shifting and choosing between Monte Carlo and TD methods. The n-step methods of
Chapter 7 also enabled this, but eligibility trace methods are more general, often faster to
learn, and o↵er di↵erent computational complexity tradeo↵s. This chapter has o↵ered an
introduction to the elegant, emerging theoretical understanding of eligibility traces for on-
and o↵-policy learning and for variable bootstrapping and discounting. One aspect of this
elegant theory is true online methods, which exactly reproduce the behavior of expensive
ideal methods while retaining the computational congeniality of conventional TD methods.
Another aspect is the possibility of derivations that automatically convert from intuitive
forward-view methods to more ecient incremental backward-view algorithms. We
illustrated this general idea in a derivation that started with a classical, expensive Monte

Carlo algorithm and ended with a cheap incremental non-TD implementation using the
same novel eligibility trace used in true online TD methods.
As we mentioned in Chapter 5, Monte Carlo methods may have advantages in non-
Markov tasks because they do not bootstrap. Because eligibility traces make TD methods
more like Monte Carlo methods, they also can have advantages in these cases. If one
wants to use TD methods because of their other advantages, but the task is at least
partially non-Markov, then the use of an eligibility trace method is indicated. Eligibility

traces are the ﬁrst line of defense against both long-delayed rewards and non-Markov
tasks.
By adjusting , we can place eligibility trace methods anywhere along a continuum
from Monte Carlo to one-step TD methods. Where shall we place them? We do not yet
have a good theoretical answer to this question, but a clear empirical answer appears to
be emerging. On tasks with many steps per episode, or many steps within the half-life of
discounting, it appears signiﬁcantly better to use eligibility traces than not to (e.g., see

tasks.
Figure 12.14). On the other hand, if the traces are so long as to produce a pure Monte
Carlo method, or nearly so, then performance degrades sharply. An intermediate mixture
appears to be the best choice. Eligibility traces should be used to bring us toward Monte
Carlo methods, but not all the way there. In the future it may be possible to more ﬁnely
vary the trade-o↵ between TD and Monte Carlo methods by using variable , but at
present it is not clear how this can be done reliably and usefully.

Methods using eligibility traces require more computation than one-step methods, but
in return they o↵er signiﬁcantly faster learning, particularly when rewards are delayed by
many steps. Thus it often makes sense to use eligibility traces when data are scarce and
cannot be repeatedly processed, as is often the case in online applications. On the other
hand, in o✏ine applications in which data can be generated cheaply, perhaps from an

inexpensive simulation, then it often does not pay to use eligibility traces. In these cases
the objective is not to get more out of a limited amount of data, but simply to process as
much data as possible as quickly as possible. In these cases the speedup per datum due to
traces is typically not worth their computational cost, and one-step methods are favored.

318 Chapter 12: Eligibility Traces
accumulatingtraces  
0.20.30.40.5
00.20.40.60.81!RANDOM WALK
50100150200250300Failures per100,000 steps
00.20.40.60.81!CART AND POLE400450500550600650700 Steps perepisode
00.20.40.60.81!MOUNTAIN CAR
replacingtraces
150160170180190200210220230240Cost perepisode00.20.40.60.81!PUDDLE WORLDreplacingtracesaccumulatingtraces replacingtraces
accumulatingtracesRMS errorFigure 12.14: The e↵ect of on reinforcement learning performance in four di↵erent test
problems. In all cases, performance is generally best (a lower number in the graph) at an
intermediate value of . The two left panels are applications to simple continuous-state control
tasks using the Sarsa( ) algorithm and tile coding, with either replacing or accumulating traces
(Sutton, 1996). The upper-right panel is for policy evaluation on a random walk task using TD( )
(Singh and Sutton, 1996). The lower right panel is unpublished data for the pole-balancing task
(Example 3.4) from an earlier study (Sutton, 1984).

Bibliographical and Historical Remarks 319
Eligibility traces came into reinforcement learning via the fecund ideas of Klopf (1972).
Our use of eligibility traces is based on Klopf’s work (Sutton, 1978a, 1978b, 1978c; Barto
and Sutton, 1981a, 1981b; Sutton and Barto, 1981a; Barto, Sutton, and Anderson, 1983;
Sutton, 1984). We may have been the ﬁrst to use the term “eligibility trace” (Sutton
and Barto, 1981a). The idea that stimuli produce after e↵ects in the nervous system
that are important for learning is very old (see Chapter 14). Some of the earliest uses of
eligibility traces were in the actor–critic methods discussed in Chapter 13 (Barto, Sutton,
and Anderson, 1983; Sutton, 1984).
12.1 Compound updates were called “complex backups” in the ﬁrst edition of this
book.
The-return and its error-reduction properties were introduced by Watkins (1989)
and further developed by Jaakkola, Jordan, and Singh (1994). The random walk

book.
results in this and subsequent sections are new to this text, as are the terms
“forward view” and “backward view.” The notion of a -return algorithm was
introduced in the ﬁrst edition of this text. The more reﬁned treatment presented
here was developed in conjunction with Harm van Seijen (e.g., van Seijen and
12.2 TD() with accumulating traces was introduced by Sutton (1988, 1984). Con-
vergence in the mean was proved by Dayan (1992), and with probability 1 by
many researchers, including Peng (1993), Dayan and Sejnowski (1994), Tsitsiklis

(1994), and Gurvits, Lin, and Hanson (1994). The bound on the error of the
asymptotic -dependent solution of linear TD( ) is due to Tsitsiklis and Van
Roy (1997).
12.3 Truncated TD methods were developed by Cichosz (1995) and van Seijen (2016).
12.4 The idea of redoing updates was extensively developed by van Seijen, originally
under the name “best-match learning” (van Seijen, 2011; van Seijen, Whiteson,
van Hasselt, and Weiring, 2011).
12.5 True online TD( ) is primarily due to Harm van Seijen (van Seijen and Sutton,

2014; van Seijen et al., 2016) though some of its key ideas were discovered
independently by Hado van Hasselt (personal communication). The name “dutch
traces” is in recognition of the contributions of both scientists.
Replacing traces are due to Singh and Sutton (1996).
12.6 The material in this section is from van Hasselt and Sutton (2015).
12.7 Sarsa( ) with accumulating traces was ﬁrst explored as a control method by
Rummery and Niranjan (1994; Rummery, 1995). True online Sarsa( ) was
introduced by van Seijen and Sutton (2014). The algorithm on page 307 was

Chapter 13
Policy Gradient Methods
In this chapter we consider something new. So far in this book almost all the methods
have been action-value methods ; they learned the values of actions and then selected
actions based on their estimated action values1; their policies would not even exist without
the action-value estimates. In this chapter we consider methods that instead learn a
parameterized policy that can select actions without consulting a value function. A value
function may still be used to learn the policy parameter, but is not required for action
selection. We use the notation ✓2Rd0for the policy’s parameter vector. Thus we write
⇡(a|s,✓)=Pr{At=a|St=s,✓t=✓}for the probability that action ais taken at time t
given that the environment is in state sat time twith parameter ✓.I fam e t h o du s e sa
learned value function as well, then the value function’s weight vector is denoted w2Rd
as usual, as in ˆ v(s,w).
In this chapter we consider methods for learning the policy parameter based on the

gradient of some scalar performance measure J(✓) with respect to the policy parameter.
These methods seek to maximize performance, so their updates approximate gradient
ascent inJ:
✓t+1=✓t+↵\rJ(✓t), (13.1)
where \rJ(✓t)2Rdis a stochastic estimate whose expectation approximates the gradient
of the performance measure with respect to its argument ✓t. All methods that follow
this general schema we call policy gradient methods , whether or not they also learn an
approximate value function. Methods that learn approximations to both policy and value

functions are often called actor–critic methods , where ‘actor’ is a reference to the learned
policy, and ‘critic’ refers to the learned value function, usually a state-value function.
First we treat the episodic case, in which performance is deﬁned as the value of the start
state under the parameterized policy, before going on to consider the continuing case, in
which performance is deﬁned as the average reward rate, as in Section 10.3. In the end,
we are able to express the algorithms for both cases in very similar terms.

1The lone exception is the gradient bandit algorithms of Section 2.8. In fact, that section goes through
many of the same steps, in the single-state bandit case, as we go through here for full MDPs. Reviewing
that section would be good preparation for fully understanding this chapter.
321

322 Chapter 13: Policy Gradient Methods
13.1 Policy Approximation and its Advantages
In policy gradient methods, the policy can be parameterized in any way, as long as
⇡(a|s,✓) is di↵erentiable with respect to its parameters, that is, as long as r⇡(a|s,✓)( t h e
column vector of partial derivatives of ⇡(a|s,✓) with respect to the components of ✓)e x i s t s
and is ﬁnite for all s2S,a2A(s), and ✓2Rd0. In practice, to ensure exploration we
generally require that the policy never becomes deterministic (i.e., that ⇡(a|s,✓)2(0,1),
for all s, a,✓). In this section we introduce the most common parameterization for
discrete action spaces and point out the advantages it o↵ers over action-value methods.
Policy-based methods also o↵er useful ways of dealing with continuous action spaces, as
we describe later in Section 13.7.
If the action space is discrete and not too large, then a natural and common kind of
parameterization is to form parameterized numerical preferences h(s, a,✓)2Rfor each

state–action pair. The actions with the highest preferences in each state are given the
highest probabilities of being selected, for example, according to an exponential soft-max
⇡(a|s,✓).=eh(s,a,✓)
P
beh(s,b,✓), (13.2)
where e⇡2.71828 is the base of the natural logarithm. Note that the denominator here
is just what is required so that the action probabilities in each state sum to one. We call
this kind of policy parameterization soft-max in action preferences .

P
The action preferences themselves can be parameterized arbitrarily. For example, they
might be computed by a deep artiﬁcial neural network (ANN), where ✓is the vector
of all the connection weights of the network (as in the AlphaGo system described in
Section 16.6). Or the preferences could simply be linear in features,
h(s, a,✓)=✓>x(s, a), (13.3)
using feature vectors x(s, a)2Rd0constructed by any of the methods described in
Chapter 9.
One advantage of parameterizing policies according to the soft-max in action preferences

Chapter 9.
is that the approximate policy can approach a deterministic policy, whereas with "-greedy
action selection over action values there is always an "probability of selecting a random
action. Of course, one could select according to a soft-max distribution based on action
values, but this alone would not allow the policy to approach a deterministic policy.
Instead, the action-value estimates would converge to their corresponding true values,
which would di↵er by a ﬁnite amount, translating to speciﬁc probabilities other than 0 and

1. If the soft-max distribution included a temperature parameter, then the temperature
could be reduced over time to approach determinism, but in practice it would be dicult
to choose the reduction schedule, or even the initial temperature, without more prior
knowledge of the true action values than we would like to assume. Action preferences
are di↵erent because they do not approach speciﬁc values; instead they are driven to
produce the optimal stochastic policy. If the optimal policy is deterministic, then the

preferences of the optimal actions will be driven inﬁnitely higher than all suboptimal
actions (if permitted by the parameterization).

13.1. Policy Approximation and its Advantages 323
A second advantage of parameterizing policies according to the soft-max in action
preferences is that it enables the selection of actions with arbitrary probabilities. In
problems with signiﬁcant function approximation, the best approximate policy may be
stochastic. For example, in card games with imperfect information the optimal play is
often to do two di↵erent things with speciﬁc probabilities, such as when blung in Poker.
Action-value methods have no natural way of ﬁnding stochastic optimal policies, whereas
policy approximating methods can, as shown in Example 13.1.
Example 13.1 Short corridor with switched actions
Consider the small corridor gridworld shown inset in the graph below. The reward
is1 per step, as usual. In each of the three nonterminal states there are only
two actions, right andleft. These actions have their usual consequences in the ﬁrst
and third states ( leftcauses no movement in the ﬁrst state), but in the second

state they are reversed, so that right moves to the left and leftmoves to the right.
The problem is dicult because all the states appear identical under the function
approximation. In particular, we deﬁne x(s,right )=[ 1 ,0]>andx(s,left)=[ 0 ,1]>,
for all s. An action-value method with "-greedy action selection is forced to choose
between just two policies: choosing right with high probability 1 "/2 on all steps
or choosing leftwith the same high probability on all time steps. If "=0.1, then
these two policies achieve a value (at the start state) of less than 44 and 82,

learn a speciﬁc probability with which to select right . The best probability is about
0.59, which achieves a value of about 11.6.
probability of right action-11.60.10.2-20-40-60-80-1000.30.400.60.70.80.90.51-greedy left -greedy right optimalstochasticpolicy J(✓)=v⇡✓(S)
GSPerhaps the simplest advantage that policy parameterization may have over action-
value parameterization is that the policy may be a simpler function to approximate.
Problems vary in the complexity of their policies and action-value functions. For some,
the action-value function is simpler and thus easier to approximate. For others, the policy
is simpler. In the latter case a policy-based method will typically learn faster and yield a
superior asymptotic policy (as in Tetris; see S ¸im¸ sek, Alg´ orta, and Kothiyal, 2016).

324 Chapter 13: Policy Gradient Methods
Finally, we note that the choice of policy parameterization is sometimes a good way
of injecting prior knowledge about the desired form of the policy into the reinforcement
learning system. This is often the most important reason for using a policy-based learning
method.
Exercise 13.1 Use your knowledge of the gridworld and its dynamics to determine an
exact symbolic expression for the optimal probability of selecting the right action in
Example 13.1. ⇤
13.2 The Policy Gradient Theorem
In addition to the practical advantages of policy parameterization over "-greedy action
selection, there is also an important theoretical advantage. With continuous policy
parameterization the action probabilities change smoothly as a function of the learned
parameter, whereas in "-greedy selection the action probabilities may change dramatically
for an arbitrarily small change in the estimated action values, if that change results in a

di↵erent action having the maximal value. Largely because of this stronger convergence
guarantees are available for policy-gradient methods than for action-value methods. In
particular, it is the continuity of the policy dependence on the parameters that enables
policy-gradient methods to approximate gradient ascent (13.1).
The episodic and continuing cases deﬁne the performance measure, J(✓), di↵erently
and thus have to be treated separately to some extent. Nevertheless, we will try to

present both cases uniformly, and we develop a notation so that the major theoretical
results can be described with a single set of equations.
In this section we treat the episodic case, for which we deﬁne the performance measure
as the value of the start state of the episode. We can simplify the notation without
losing any meaningful generality by assuming that every episode starts in some particular
(non-random) state s0. Then, in the episodic case we deﬁne performance as
J(✓).=v⇡✓(s0), (13.4)

where v⇡✓is the true value function for ⇡✓, the policy determined by ✓. From here on in
our discussion we will assume no discounting ( = 1) for the episodic case, although for
completeness we do include the possibility of discounting in the boxed algorithms.
With function approximation, it may seem challenging to change the policy parameter
in a way that ensures improvement. The problem is that performance depends on both
the action selections and the distribution of states in which those selections are made,

and that both of these are a↵ected by the policy parameter. Given a state, the e↵ect of
the policy parameter on the actions, and thus on reward, can be computed in a relatively
straightforward way from knowledge of the parameterization. But the e↵ect of the policy
on the state distribution is a function of the environment and is typically unknown. How
can we estimate the performance gradient with respect to the policy parameter when the
gradient depends on the unknown e↵ect of policy changes on the state distribution?

Fortunately, there is an excellent theoretical answer to this challenge in the form of
thepolicy gradient theorem , which provides an analytic expression for the gradient of

13.2. The Policy Gradient Theorem 325
Proof of the Policy Gradient Theorem (episodic case)
With just elementary calculus and re-arranging of terms, we can prove the policy
gradient theorem from ﬁrst principles. To keep the notation simple, we leave it
implicit in all cases that ⇡is a function of ✓, and all gradients are also implicitly
with respect to ✓. First note that the gradient of the state-value function can be
written in terms of the action-value function as
rv⇡(s)=r"X
a⇡(a|s)q⇡(s, a)#
,for all s2S (Exercise 3.18)
=X
ah
r⇡(a|s)q⇡(s, a)+⇡(a|s)rq⇡(s, a)i
(product rule of calculus)
=X
ah
r⇡(a|s)q⇡(s, a)+⇡(a|s)rX
s0,rp(s0,r|s, a)
r+v⇡(s0)i
(Exercise 3.19 and Equation 3.2)
=X
ah
r⇡(a|s)q⇡(s, a)+⇡(a|s)X
s0p(s0|s, a)rv⇡(s0)i
(Eq. 3.4)
=X
ah
s0p(s0|s, a) (unrolling)
X
a0⇥
r⇡(a0|s0)q⇡(s0,a0)+⇡(a0|s0)X
s00p(s00|s0,a0)rv⇡(s00)⇤i
=X
x2S1X
k=0Pr(s!x, k, ⇡ )X
ar⇡(a|x)q⇡(x, a),
after repeated unrolling, where Pr(s!x, k, ⇡ ) is the probability of transitioning

=X
ah
=X
ah
r+v⇡(s0)i
=X
ah
(Eq. 3.4)
=X
ah
X
a0⇥
=X
x2S1X
from state sto state xinksteps under policy ⇡. It is then immediate that
rJ(✓)=rv⇡(s0)
=X
s 1X
k=0Pr(s0!s, k, ⇡ )!X
ar⇡(a|s)q⇡(s, a)
=X
s⌘(s)X
ar⇡(a|s)q⇡(s, a) (box page 199)
=X
s0⌘(s0)X
s⌘(s)P
s0⌘(s0)X
=X
s0⌘(s0)X
sµ(s)X
ar⇡(a|s)q⇡(s, a) (Eq. 9.3)
/X
sµ(s)X
ar⇡(a|s)q⇡(s, a)( Q . E . D . )

326 Chapter 13: Policy Gradient Methods
performance with respect to the policy parameter (which is what we need to approximate
for gradient ascent (13.1) ) that does notinvolve the derivative of the state distribution.
The policy gradient theorem for the episodic case establishes that
rJ(✓)/X
sµ(s)X
aq⇡(s, a)r⇡(a|s,✓), (13.5)
where the gradients are column vectors of partial derivatives with respect to the compo-
nents of ✓, and ⇡denotes the policy corresponding to parameter vector ✓. The symbol /
here means “proportional to”. In the episodic case, the constant of proportionality is the
average length of an episode, and in the continuing case it is 1, so that the relationship is
actually an equality. The distribution µhere (as in Chapters 9 and 10) is the on-policy
distribution under ⇡(see page 199). The policy gradient theorem is proved for the
episodic case in the box on the previous page.
13.3 REINFORCE: Monte Carlo Policy Gradient

We are now ready to derive our ﬁrst policy-gradient learning algorithm. Recall our overall
strategy of stochastic gradient ascent (13.1) , which requires a way to obtain samples such
that the expectation of the sample gradient is proportional to the actual gradient of the
performance measure as a function of the parameter. The sample gradients need only be
proportional to the gradient because any constant of proportionality can be absorbed
into the step size ↵, which is otherwise arbitrary. The policy gradient theorem gives an

exact expression proportional to the gradient; all that is needed is some way of sampling
whose expectation equals or approximates this expression. Notice that the right-hand
side of the policy gradient theorem is a sum over states weighted by how often the states
occur under the target policy ⇡;i f⇡is followed, then states will be encountered in these
proportions. Thus
rJ(✓)/X
sµ(s)X
aq⇡(s, a)r⇡(a|s,✓)
=E⇡"X
aq⇡(St,a)r⇡(a|St,✓)#
. (13.6)
We could stop here and instantiate our stochastic gradient-ascent algorithm (13.1) as
✓t+1.=✓t+↵X
aˆq(St,a ,w)r⇡(a|St,✓), (13.7)

rJ(✓)/X
sµ(s)X
=E⇡"X
. (13.6)
where ˆqis some learned approximation to q⇡. This algorithm, which has been called
anall-actions method because its update involves all of the actions, is promising and
deserving of further study, but our current interest is the classical REINFORCE algorithm
(Willams, 1992) whose update at time tinvolves just At, the one action actually taken at
time t.
We continue our derivation of REINFORCE by introducing Atin the same way as we
introduced Stin(13.6) —by replacing a sum over the random variable’s possible values

13.3. REINFORCE: Monte Carlo Policy Gradient 327
by an expectation under ⇡, and then sampling the expectation. Equation (13.6) involves
an appropriate sum over actions, but each term is not weighted by ⇡(a|St,✓) as is needed
for an expectation under ⇡. So we introduce such a weighting, without changing the
equality, by multiplying and then dividing the summed terms by ⇡(a|St,✓). Continuing
from (13.6), we have
rJ(✓)=E⇡"X
a⇡(a|St,✓)q⇡(St,a)r⇡(a|St,✓)
⇡(a|St,✓)#
=E⇡
q⇡(St,At)r⇡(At|St,✓)
⇡(At|St,✓)
(replacing aby the sample At⇠⇡)
=E⇡
Gtr⇡(At|St,✓)
, (because E⇡[Gt|St,At]=q⇡(St,At))
where Gtis the return as usual. The ﬁnal expression in brackets is exactly what is
needed, a quantity that can be sampled on each time step whose expectation is equal
to the gradient. Using this sample to instantiate our generic stochastic gradient ascent
algorithm (13.1) yields the REINFORCE update:
✓t+1.=✓t+↵Gtr⇡(At|St,✓t)
⇡(At|St,✓t). (13.8)

=E⇡
=E⇡
This update has an intuitive appeal. Each increment is proportional to the product of a
return Gtand a vector, the gradient of the probability of taking the action actually taken
divided by the probability of taking that action. The vector is the direction in parameter
space that most increases the probability of repeating the action Aton future visits
to state St. The update increases the parameter vector in this direction proportional
to the return, and inversely proportional to the action probability. The former makes

sense because it causes the parameter to move most in the directions that favor actions
that yield the highest return. The latter makes sense because otherwise actions that are
selected frequently are at an advantage (the updates will be more often in their direction)
and might win out even if they do not yield the highest return.
Note that REINFORCE uses the complete return from time t, which includes all
future rewards up until the end of the episode. In this sense REINFORCE is a Monte
Carlo algorithm and is well deﬁned only for the episodic case with all updates made in

retrospect after the episode is completed (like the Monte Carlo algorithms in Chapter 5).
This is shown explicitly in the boxed on the next page.
Notice that the update in the last line of pseudocode appears rather di↵erent from
the REINFORCE update rule (13.8) . One di↵erence is that the pseudocode uses the
compact expression rln⇡(At|St,✓t) for the fractional vectorr⇡(At|St,✓t)
⇡(At|St,✓t)in(13.8) . That
these two expressions for the vector are equivalent follows from the identity rlnx=rx
x.

x.
This vector has been given several names and notations in the literature; we will refer
to it simply as the eligibility vector . Note that it is the only place that the policy
parameterization appears in the algorithm.

The second di↵erence between the pseudocode update and the REINFORCE update
equation (13.8) is that the former includes a factor of t. This is because, as mentioned
earlier, in the text we are treating the non-discounted case ( = 1) while in the boxed
algorithms we are giving the algorithms for the general discounted case. All of the ideas
go through in the discounted case with appropriate adjustments (including to the box on
page 199) but involve additional complexity that distracts from the main ideas.
⇤Exercise 13.2 Generalize the box on page 199, the policy gradient theorem (13.5) ,t h e
proof of the policy gradient theorem (page 325), and the steps leading to the REINFORCE
update equation (13.8) , so that (13.8) ends up with a factor of tand thus aligns with
the general algorithm given in the pseudocode. ⇤
Figure 13.1 shows the performance of REINFORCE on the short-corridor gridworld
from Example 13.1.

13.4. REINFORCE with Baseline 329
As a stochastic gradient method, REINFORCE has good theoretical convergence
properties. By construction, the expected update over an episode is in the same direction
as the performance gradient. This assures an improvement in expected performance for
suciently small ↵, and convergence to a local optimum under standard stochastic approx-
imation conditions for decreasing ↵. However, as a Monte Carlo method REINFORCE
may be of high variance and thus produce slow learning.
Exercise 13.3 In Section 13.1 we considered policy parameterizations using the soft-max in
action preferences (13.2) with linear action preferences (13.3) . For this parameterization,
prove that the eligibility vector is
rln⇡(a|s,✓)=x(s, a)X
b⇡(b|s,✓)x(s, b), (13.9)
using the deﬁnitions and elementary calculus. ⇤
13.4 REINFORCE with Baseline
The policy gradient theorem (13.5) can be generalized to include a comparison of the
action value to an arbitrary baseline b(s):
rJ(✓)/X
sµ(s)X
a⇣
q⇡(s, a)b(s)⌘

rJ(✓)/X
sµ(s)X
a⇣
r⇡(a|s,✓). (13.10)
The baseline can be any function, even a random variable, as long as it does not vary
with a; the equation remains valid because the subtracted quantity is zero:
X
ab(s)r⇡(a|s,✓)= b(s)rX
a⇡(a|s,✓)= b(s)r1=0 .
The policy gradient theorem with baseline (13.10) can be used to derive an update
rule using similar steps as in the previous section. The update rule that we end up with
is a new version of REINFORCE that includes a general baseline:
✓t+1.=✓t+↵⇣
Gtb(St)⌘r⇡(At|St,✓t)

X
⇡(At|St,✓t). (13.11)
Because the baseline could be uniformly zero, this update is a strict generalization of
REINFORCE. In general, the baseline leaves the expected value of the update unchanged,
but it can have a large e↵ect on its variance. For example, we saw in Section 2.8 that an
analogous baseline can signiﬁcantly reduce the variance (and thus speed the learning) of
gradient bandit algorithms. In the bandit algorithms the baseline was just a number (the

average of the rewards seen so far), but for MDPs the baseline should vary with state.
In some states all actions have high values and we need a high baseline to di↵erentiate
the higher valued actions from the less highly valued ones; in other states all actions will
have low values and a low baseline is appropriate.
One natural choice for the baseline is an estimate of the state value, ˆv(St,w), where
w2Rmis a weight vector learned by one of the methods presented in previous chapters.

13.5. Actor–Critic Methods 331
Figure 13.2 compares the behavior of REINFORCE with and without a baseline on
the short-corridor gridword (Example 13.1). Here the approximate state-value function
used in the baseline is ˆ v(s,w)=w. That is, wis a single component, w.
13.5 Actor–Critic Methods
Although the REINFORCE-with-baseline method learns both a policy and a state-value
function, we do not consider it to be an actor–critic method because its state-value function
is used only as a baseline, not as a critic. That is, it is not used for bootstrapping (updating
the value estimate for a state from the estimated values of subsequent states), but only
as a baseline for the state whose estimate is being updated. This is a useful distinction,
for only through bootstrapping do we introduce bias and an asymptotic dependence
on the quality of the function approximation. As we have seen, the bias introduced
through bootstrapping and reliance on the state representation is often beneﬁcial because

it reduces variance and accelerates learning. REINFORCE with baseline is unbiased
and will converge asymptotically to a local minimum, but like all Monte Carlo methods
it tends to learn slowly (produce estimates of high variance) and to be inconvenient
to implement online or for continuing problems. As we have seen earlier in this book,
with temporal-di↵erence methods we can eliminate these inconveniences, and through
multi-step methods we can ﬂexibly choose the degree of bootstrapping. In order to gain

these advantages in the case of policy gradient methods we use actor–critic methods with
a bootstrapping critic.
First consider one-step actor–critic methods, the analog of the TD methods introduced
in Chapter 6 such as TD(0), Sarsa(0), and Q-learning. The main appeal of one-step
methods is that they are fully online and incremental, yet avoid the complexities of
eligibility traces. They are a special case of the eligibility trace methods, and not as

general, but easier to understand. One-step actor–critic methods replace the full return
of REINFORCE (13.11) with the one-step return (and use a learned state-value function
as the baseline) as follows:
Gt:t+1ˆv(St,w)⌘r⇡(At|St,✓t)
⇡(At|St,✓t)(13.12)
=✓t+↵⇣
Rt+1+ˆv(St+1,w)ˆv(St,w)⌘r⇡(At|St,✓t)
⇡(At|St,✓t)(13.13)
=✓t+↵tr⇡(At|St,✓t)
⇡(At|St,✓t). (13.14)
The natural state-value-function learning method to pair with this is semi-gradient TD(0).
Pseudocode for the complete algorithm is given in the box at the top of the next page.

13.6. Policy Gradient for Continuing Problems 333
13.6 Policy Gradient for Continuing Problems
As discussed in Section 10.3, for continuing problems without episode boundaries we need
to deﬁne performance in terms of the average rate of reward per time step:
J(✓).=r(⇡).=l i m
h!11
hhX
t=1E[Rt|S0,A0:t1⇠⇡] (13.15)
=l i m
t!1E[Rt|S0,A0:t1⇠⇡]
=X
sµ(s)X
a⇡(a|s)X
where µis the steady-state distribution under ⇡,µ(s).=l i m t!1Pr{St=s|A0:t⇠⇡},
which is assumed to exist and to be independent of S0(an ergodicity assumption).
Remember that this is the special distribution under which, if you select actions according
to⇡, you remain in the same distribution:
X
sµ(s)X
a⇡(a|s,✓)p(s0|s, a)=µ(s0),for all s02S. (13.16)
Complete pseudocode for the actor–critic algorithm in the continuing case (backward
view) is given in the box below.
Actor–Critic with Eligibility Traces (continuing), for estimating ⇡✓⇡⇡⇤
Input: a di↵erentiable policy parameterization ⇡(a|s,✓)

334 Chapter 13: Policy Gradient Methods
Naturally, in the continuing case, we deﬁne values, v⇡(s).=E⇡[Gt|St=s]andq⇡(s, a).=
E⇡[Gt|St=s, A t=a], with respect to the di↵erential return:
Gt.=Rt+1r(⇡)+ Rt+2r(⇡)+ Rt+3r(⇡)+ ···. (13.17)
With these alternate deﬁnitions, the policy gradient theorem as given for the episodic
case (13.5) remains true for the continuing case. A proof is given in the box on the next
page. The forward and backward view equations also remain the same.
Proof of the Policy Gradient Theorem (continuing case)
The proof of the policy gradient theorem for the continuing case begins similarly
to the episodic case. Again we leave it implicit in all cases that ⇡is a function
of✓and that the gradients are with respect to ✓. Recall that in the continuing
caseJ(✓)=r(⇡)(13.15) and that v⇡andq⇡denote values with respect to the
di↵erential return (13.17) . The gradient of the state-value function can be written,
for any s2S, as
=X
ah

=X
ah
=X
ah
rr(✓)+v⇡(s0)i
=X
ah
r⇡(a|s)q⇡(s, a)+⇡(a|s)⇥
rr(✓)+X
s0p(s0|s, a)rv⇡(s0)⇤i
.
After re-arranging terms, we obtain
rr(✓)=X
ah
rv⇡(s).
Notice that the left-hand side can be written rJ(✓), and that it does not depend
ons. Thus the right-hand side does not depend on seither, and we can safely sum
it over all s2S, weighted by µ(s), without changing it (becauseP
sµ(s) = 1):
rJ(✓)=X
sµ(s) X

action preferences is given by a weighted sum of the unit’s input vector, that is, assume
that h(s,1,✓)h(s,0,✓)=✓>x(s), where ✓is the unit’s weight vector.
(a)Show that if the exponential soft-max distribution (13.2) is used to convert action
preferences to policies, then Pt=⇡(1|St,✓t)=1 /(1+e x p ( ✓>
tx(St)))(the logistic
function).
(b)What is the Monte-Carlo REINFORCE update of ✓tto✓t+1upon receipt of return
Gt?
(c)Express the eligibility rln⇡(a|s,✓) for a Bernoulli-logistic unit, in terms of a,x(s),
and⇡(a|s,✓) by calculating the gradient.
Hint: separately for each action compute the derivative of the logarithm ﬁrst with respect
toPt=⇡(a|s,✓t), combine the two results into one expression that depends on aandPt,
and then use the chain rule, noting that the derivative of the logistic function f(x)i s
f(x)(1f(x)). ⇤

Bibliographical and Historical Remarks 337
13.8 Summary
Prior to this chapter, this book focused on action-value methods —meaning methods that
learn action values and then use them to determine action selections. In this chapter, on
the other hand, we considered methods that learn a parameterized policy that enables
actions to be taken without consulting action-value estimates. In particular, we have
considered policy-gradient methods —meaning methods that update the policy parameter
on each step in the direction of an estimate of the gradient of performance with respect
to the policy parameter.
Methods that learn and store a policy parameter have many advantages. They can
learn speciﬁc probabilities for taking the actions. They can learn appropriate levels
of exploration and approach deterministic policies asymptotically. They can naturally
handle continuous action spaces. All these things are easy for policy-based methods but

awkward or impossible for "-greedy methods and for action-value methods in general. In
addition, on some problems the policy is just simpler to represent parametrically than
the value function; these problems are more suited to parameterized policy methods.
Parameterized policy methods also have an important theoretical advantage over
action-value methods in the form of the policy gradient theorem , which gives an exact
formula for how performance is a↵ected by the policy parameter that does not involve

derivatives of the state distribution. This theorem provides a theoretical foundation for
all policy gradient methods.
The REINFORCE method follows directly from the policy gradient theorem. Adding
a state-value function as a baseline reduces REINFORCE’s variance without introducing
bias. Using the state-value function for bootstrapping introduces bias but is often
desirable for the same reason that bootstrapping TD methods are often superior to Monte

Carlo methods (substantially reduced variance). The state-value function assigns credit
to—critizes—the policy’s action selections, and accordingly the former is termed the critic
and the latter the actor , and these overall methods are termed actor–critic methods.
Overall, policy-gradient methods provide a signiﬁcantly di↵erent set of strengths and
weaknesses than action-value methods. Today they are less well understood in some
respects, but a subject of excitement and ongoing research.

Methods that we now see as related to policy gradients were actually some of the earliest
to be studied in reinforcement learning (Witten, 1977; Barto, Sutton, and Anderson, 1983;
Sutton, 1984; Williams, 1987, 1992) and in predecessor ﬁelds (Phansalkar and Thathachar,
1995). They were largely supplanted in the 1990s by the action-value methods that are
the focus of the other chapters of this book. In recent years, however, attention has
returned to actor–critic methods and to policy-gradient methods in general. Among the

further developments beyond what we cover here are natural-gradient methods (Amari,
1998; Kakade, 2002, Peters, Vijayakumar and Schaal, 2005; Peters and Schall, 2008; Park,
Kim and Kang, 2005; Bhatnagar, Sutton, Ghavamzadeh and Lee, 2009; see Grondman,
Busoniu, Lopes and Babuska, 2012), deterministic policy gradient methods (Silver et al.,

338 Chapter 13: Policy Gradient Methods
2014), o↵-policy policy-gradient methods (Degris, White, and Sutton, 2012; Maei, 2018),
and entropy regularization (see Schulman, Chen, and Abbeel, 2017). Major applications
include acrobatic helicopter autopilots and AlphaGo (Section 16.6).
Our presentation in this chapter is based primarily on that by Sutton, McAllester,
Singh, and Mansour (2000), who introduced the term “policy gradient methods.” A useful
overview is provided by Bhatnagar et al. (2009). One of the earliest related works is
by Aleksandrov, Sysoyev, and Shemeneva (1968). Thomas (2014) ﬁrst realized that the
factor of t, as speciﬁed in the boxed algorithms of this chapter, was needed in the case
of discounted episodic problems.
13.1 Example 13.1 and the results with it in this chapter were developed with Eric
Graves.
13.2 The policy gradient theorem here and on page 334 was ﬁrst obtained by Marbach
and Tsitsiklis (1998, 2001) and then independently by Sutton et al. (2000). A

Graves.
similar expression was obtained by Cao and Chen (1997). Other early results
are due to Konda and Tsitsiklis (2000, 2003), Baxter and Bartlett (2001), and
Baxter, Bartlett, and Weaver (2001). Some additional results are developed by
Sutton, Singh, and McAllester (2000).
13.3 REINFORCE is due to Williams (1987, 1992). Phansalkar and Thathachar
(1995) proved both local and global convergence theorems for modiﬁed versions
of REINFORCE algorithms.
The all-actions algorithm was ﬁrst presented in an unpublished but widely

circulated incomplete paper (Sutton, Singh, and McAllester, 2000) and then
developed further by Ciosek and Whiteson (2017, 2018), who termed it “expected
policy gradients,” and by Asadi, Allen, Roderick, Mohamed, Konidaris, and
Littman (2017) who called it “mean actor critic.”
13.4 The baseline was introduced in Williams’s (1987, 1992) original work. Greensmith,
Bartlett, and Baxter (2004) analyzed an arguably better baseline (see Dick, 2015).
Thomas and Brunskill (2017) argue that an action-dependent baseline can be
used without incurring bias.

13.5–6 Actor–critic methods were among the earliest to be investigated in reinforcement
learning (Witten, 1977; Barto, Sutton, and Anderson, 1983; Sutton, 1984). The
algorithms presented here are based on the work of Degris, White, and Sutton
(2012). Actor–critic methods are sometimes referred to as advantage actor–critic
methods in the literature.
13.7 The ﬁrst to show how continuous actions could be handled this way appears
to have been Williams (1987, 1992). The ﬁgure on page 335 is adapted from
Wikipedia.

Part III: Looking Deeper
In this last part of the book we look beyond the standard reinforcement learning ideas
presented in the ﬁrst two parts of the book to brieﬂy survey their relationships with
psychology and neuroscience, a sampling of reinforcement learning applications, and some
of the active frontiers for future reinforcement learning research.
339

Chapter 14
In previous chapters we developed ideas for algorithms based on computational con-
siderations alone. In this chapter we look at some of these algorithms from another
perspective: the perspective of psychology and its study of how animals learn. The goals
of this chapter are, ﬁrst, to discuss ways that reinforcement learning ideas and algorithms
correspond to what psychologists have discovered about animal learning, and second, to
explain the inﬂuence reinforcement learning is having on the study of animal learning.
The clear formalism provided by reinforcement learning that systemizes tasks, returns,
and algorithms is proving to be enormously useful in making sense of experimental data,
in suggesting new kinds of experiments, and in pointing to factors that may be critical to
manipulate and to measure. The idea of optimizing return over the long term that is
at the core of reinforcement learning is contributing to our understanding of otherwise

puzzling features of animal learning and behavior.
Some of the correspondences between reinforcement learning and psychological theories
are not surprising because the development of reinforcement learning drew inspiration
from psychological learning theories. However, as developed in this book, reinforcement
learning explores idealized situations from the perspective of an artiﬁcial intelligence
researcher or engineer, with the goal of solving computational problems with ecient
algorithms, rather than to replicate or explain in detail how animals learn. As a result,

some of the correspondences we describe connect ideas that arose independently in their
respective ﬁelds. We believe these points of contact are specially meaningful because they
expose computational principles important to learning, whether it is learning by artiﬁcial
or by natural systems.
For the most part, we describe correspondences between reinforcement learning and
learning theories developed to explain how animals like rats, pigeons, and rabbits learn
in controlled laboratory experiments. Thousands of these experiments were conducted

throughout the 20th century, and many are still being conducted today. Although
sometimes dismissed as irrelevant to wider issues in psychology, these experiments probe
subtle properties of animal learning, often motivated by precise theoretical questions.
As psychology shifted its focus to more cognitive aspects of behavior, that is, to mental
processes such as thought and reasoning, animal learning experiments came to play
341

342 Chapter 14: Psychology
less of a role in psychology than they once did. But this experimentation led to the
discovery of learning principles that are elemental and widespread throughout the animal
kingdom, principles that should not be neglected in designing artiﬁcial learning systems.
In addition, as we shall see, some aspects of cognitive processing connect naturally to the
computational perspective provided by reinforcement learning.
This chapter’s ﬁnal section includes references relevant to the connections we discuss as
well as to connections we neglect. We hope this chapter encourages readers to probe all
of these connections more deeply. Also included in this ﬁnal section is a discussion of how
the terminology used in reinforcement learning relates to that of psychology. Many of
the terms and phrases used in reinforcement learning are borrowed from animal learning
theories, but the computational/engineering meanings of these terms and phrases do not

always coincide with their meanings in psychology.
14.1 Prediction and Control
The algorithms we describe in this book fall into two broad categories: algorithms
forprediction and algorithms for control .1These categories arise naturally in solution
methods for the reinforcement learning problem presented in Chapter 3. In many ways
these categories respectively correspond to categories of learning extensively studied
by psychologists: classical , or Pavlovian ,conditioning and instrumental , or operant ,

conditioning . These correspondences are not completely accidental because of psychology’s
inﬂuence on reinforcement learning, but they are nevertheless striking because they connect
ideas arising from di↵erent objectives.
The prediction algorithms presented in this book estimate quantities that depend
on how features of an agent’s environment are expected to unfold over the future. We
speciﬁcally focus on estimating the amount of reward an agent can expect to receive over

the future while it interacts with its environment. In this role, prediction algorithms are
policy evaluation algorithms , which are integral components of algorithms for improving
policies. But prediction algorithms are not limited to predicting future reward; they can
predict any feature of the environment (see, for example, Modayil, White, and Sutton,
2014). The correspondence between prediction algorithms and classical conditioning rests
on their common property of predicting upcoming stimuli, whether or not those stimuli

are rewarding (or punishing).
The situation in an instrumental, or operant, conditioning experiment is di↵erent.
Here, the experimental apparatus is set up so that an animal is given something it likes
(a reward) or something it dislikes (a penalty) depending on what the animal did. The
animal learns to increase its tendency to produce rewarded behavior and to decrease its
tendency to produce penalized behavior. The reinforcing stimulus is said to be contingent
on the animal’s behavior, whereas in classical conditioning it is not (although it is dicult

to remove all behavior contingencies in a classical conditioning experiment). Instrumental
conditioning experiments are like those that inspired Thorndike’s Law of E↵ect that
1What control means for us is di↵erent from what it typically means in animal learning theories; there
the environment controls the agent instead of the other way around. See our comments on terminology
at the end of this chapter.

14.2. Classical Conditioning 343
we brieﬂy discuss in Chapter 1. Control is at the core of this form of learning, which
corresponds to the operation of reinforcement learning’s policy-improvement algorithms.
Thinking of classical conditioning in terms of prediction, and instrumental conditioning
in terms of control, is a starting point for connecting our computational view of rein-
forcement learning to animal learning, but in reality, the situation is more complicated
than this. There is more to classical conditioning than prediction; it also involves action,
and so is a mode of control, sometimes called Pavlovian control . Further, classical and
instrumental conditioning interact in interesting ways, with both sorts of learning likely
being engaged in most experimental situations. Despite these complications, aligning the
classical/instrumental distinction with the prediction/control distinction is a convenient
ﬁrst approximation in connecting reinforcement learning to animal learning.

In psychology, the term reinforcement is used to describe learning in both classical and
instrumental conditioning. Originally referring only to the strengthening of a pattern of
behavior, it is frequently also used for the weakening of a pattern of behavior. A stimulus
considered to be the cause of the change in behavior is called a reinforcer, whether or
not it is contingent on the animal’s previous behavior. At the end of this chapter we
discuss this terminology in more detail and how it relates to terminology used in machine
learning.
14.2 Classical Conditioning

learning.
While studying the activity of the digestive system, the celebrated Russian physiologist
Ivan Pavlov found that an animal’s innate responses to certain triggering stimuli can
come to be triggered by other stimuli that are quite unrelated to the inborn triggers. His
experimental subjects were dogs that had undergone minor surgery to allow the intensity
of their salivary reﬂex to be accurately measured. In one case he describes, the dog did
not salivate under most circumstances, but about 5 seconds after being presented with

food it produced about six drops of saliva over the next several seconds. After several
repetitions of presenting another stimulus, one not related to food, in this case the sound
of a metronome, shortly before the introduction of food, the dog salivated in response to
the sound of the metronome in the same way it did to the food. “The activity of the
salivary gland has thus been called into play by impulses of sound—a stimulus quite
alien to food” (Pavlov, 1927, p. 22). Summarizing the signiﬁcance of this ﬁnding, Pavlov
wrote:

wrote:
It is pretty evident that under natural conditions the normal animal must
respond not only to stimuli which themselves bring immediate beneﬁt or
harm, but also to other physical or chemical agencies—waves of sound, light,
and the like—which in themselves only signal the approach of these stimuli;
though it is not the sight and sound of the beast of prey which is in itself
harmful to the smaller animal, but its teeth and claws. (Pavlov, 1927, p. 14)
Connecting new stimuli to innate reﬂexes in this way is now called classical, or Pavlovian,

conditioning. Pavlov (or more exactly, his translators) called inborn responses (e.g.,
salivation in his demonstration described above) “unconditioned responses” (URs), their

344 Chapter 14: Psychology
natural triggering stimuli (e.g., food) “unconditioned stimuli” (USs), and new responses
triggered by predictive stimuli (e.g., here also salivation) “conditioned responses” (CRs).
A stimulus that is initially neutral, meaning that it does not normally elicit strong
responses (e.g., the metronome sound), becomes a “conditioned stimulus” (CS) as the
animal learns that it predicts the US and so comes to produce a CR in response to the CS.
These terms are still used in describing classical conditioning experiments (though better
translations would have been “conditional” and “unconditional” instead of conditioned
and unconditioned). The US is called a reinforcer because it reinforces producing a CR
in response to the CS.tTrace ConditioningDelay ConditioningCSUSCSUSISIThe arrangement of stimuli in two
common types of classical condition-
ing experiments is shown to the right.
Indelay conditioning ,t h eC Se x t e n d s
throughout the interstimulus interval, or

ISI, which is the time interval between
the CS onset and the US onset (with
the CS ending when the US ends in a
common version shown here). In trace
conditioning , the US begins after the CS
ends, and the time interval between CS
o↵set and US onset is called the trace
interval.
The salivation of Pavlov’s dogs to the
sound of a metronome is just one exam-
ple of classical conditioning, which has
been intensively studied across many response systems of many species of animals. URs
are often preparatory in some way, like the salivation of Pavlov’s dog, or protective in

interval.
some way, like an eye blink in response to something irritating to the eye, or freezing
in response to seeing a predator. Experiencing the CS-US predictive relationship over
a series of trials causes the animal to learn that the CS predicts the US so that the
animal can respond to the CS with a CR that prepares the animal for, or protects it
from, the predicted US. Some CRs are similar to the UR but begin earlier and di↵er in
ways that increase their e↵ectiveness. In one intensively studied type of experiment, for

example, a tone CS reliably predicts a pu↵ of air (the US) to a rabbit’s eye, triggering a
UR consisting of the closure of a protective inner eyelid called the nictitating membrane.
After one or more trials, the tone comes to trigger a CR consisting of membrane closure
that begins before the air pu↵ and eventually becomes timed so that peak closure occurs
just when the air pu↵ is likely to occur. This CR, being initiated in anticipation of the
air pu↵ and appropriately timed, o↵ers better protection than simply initiating closure

as a reaction to the irritating US. The ability to act in anticipation of important events
by learning about predictive relationships among stimuli is so beneﬁcial that it is widely
present across the animal kingdom.

14.2. Classical Conditioning 345
14.2.1 Blocking and Higher-order Conditioning
Many interesting properties of classical conditioning have been observed in experiments.
Beyond the anticipatory nature of CRs, two widely observed properties ﬁgured prominently
in the development of classical conditioning models: blocking andhigher-order conditioning .
Blocking occurs when an animal fails to learn a CR when a potential CS is presented along
with another CS that had been used previously to condition the animal to produce that
CR. For example, in the ﬁrst stage of a blocking experiment involving rabbit nictitating
membrane conditioning, a rabbit is ﬁrst conditioned with a tone CS and an air pu↵ US
to produce the CR of closing its nictitating membrane in anticipation of the air pu↵. The
experiment’s second stage consists of additional trials in which a second stimulus, say
a light, is added to the tone to form a compound tone/light CS followed by the same

air pu↵ US. In the experiment’s third phase, the second stimulus alone—the light—is
presented to the rabbit to see if the rabbit has learned to respond to it with a CR. It
turns out that the rabbit produces very few, or no, CRs in response to the light: learning
to the light had been blocked by the previous learning to the tone.2Blocking results like
this challenged the idea that conditioning depends only on simple temporal contiguity,
that is, that a necessary and sucient condition for conditioning is that a US frequently

follows a CS closely in time. In the next section we describe the Rescorla–Wagner model
(Rescorla and Wagner, 1972) that o↵ered an inﬂuential explanation for blocking.
Higher-order conditioning occurs when a previously-conditioned CS acts as a US
in conditioning another initially neutral stimulus. Pavlov described an experiment in
which his assistant ﬁrst conditioned a dog to salivate to the sound of a metronome that
predicted a food US, as described above. After this stage of conditioning, a number of

trials were conducted in which a black square, to which the dog was initially indi↵erent,
was placed in the dog’s line of vision followed by the sound of the metronome—and
this was notfollowed by food. In just ten trials, the dog began to salivate merely upon
seeing the black square, despite the fact that the sight of it had never been followed by
food. The sound of the metronome itself acted as a US in conditioning a salivation CR

to the black square CS. This was second-order conditioning. If the black square had
been used as a US to establish salivation CRs to another otherwise neutral CS, it would
have been third-order conditioning, and so on. Higher-order conditioning is dicult to
demonstrate, especially above the second order, in part because a higher-order reinforcer
loses its reinforcing value due to not being repeatedly followed by the original US during
higher-order conditioning trials. But under the right conditions, such as intermixing

ﬁrst-order trials with higher-order trials or by providing a general energizing stimulus,
higher-order conditioning beyond the second order can be demonstrated. As we describe
below, the TD model of classical conditioning uses the bootstrapping idea that is central
to our approach to extend the Rescorla–Wagner model’s account of blocking to include
both the anticipatory nature of CRs and higher-order conditioning.
Higher-order instrumental conditioning occurs as well. In this case, a stimulus that

2Comparison with a control group is necessary to show that the previous conditioning to the tone is
responsible for blocking learning to the light. This is done by trials with the tone/light CS but with no
prior conditioning to the tone. Learning to the light in this case is unimpaired. Moore and Schmajuk
(2008) give a full account of this procedure.

346 Chapter 14: Psychology
consistently predicts primary reinforcement becomes a reinforcer itself, where reinforce-
ment is primary if its rewarding or penalizing quality has been built into the animal by
evolution. The predicting stimulus becomes a secondary reinforcer , or more generally, a
higher-order orconditioned reinforcer —the latter being a better term when the predicted
reinforcing stimulus is itself a secondary, or an even higher-order, reinforcer. A condi-
tioned reinforcer delivers conditioned reinforcement : conditioned reward or conditioned
penalty. Conditioned reinforcement acts like primary reinforcement in increasing an
animal’s tendency to produce behavior that leads to conditioned reward, and to decrease
an animal’s tendency to produce behavior that leads to conditioned penalty. (See our
comments at the end of this chapter that explain how our terminology sometimes di↵ers,
as it does here, from terminology used in psychology.)

Conditioned reinforcement is a key phenomenon that explains, for instance, why we
work for the conditioned reinforcer money, whose worth derives solely from what is
predicted by having it. In actor–critic methods described in Section 13.5 (and discussed
in the context of neuroscience in Sections 15.7 and 15.8), the critic uses a TD method
to evaluate the actor’s policy, and its value estimates provide conditioned reinforcement
to the actor, allowing the actor to improve its policy. This analog of higher-order

instrumental conditioning helps address the credit-assignment problem mentioned in
Section 1.7 because the critic gives moment-by-moment reinforcement to the actor when
the primary reward signal is delayed. We discuss this more below in Section 14.4.
14.2.2 The Rescorla–Wagner Model
Rescorla and Wagner created their model mainly to account for blocking. The core
idea of the Rescorla–Wagner model is that an animal only learns when events violate
its expectations, in other words, only when the animal is surprised (although without

necessarily implying any conscious expectation or emotion). We ﬁrst present Rescorla and
Wagner’s model using their terminology and notation before shifting to the terminology
and notation we use to describe the TD model.
Here is how Rescorla and Wagner described their model. The model adjusts the
“associative strength” of each component stimulus of a compound CS, which is a number
representing how strongly or reliably that component is predictive of a US. When
a compound CS consisting of several component stimuli is presented in a classical

conditioning trial, the associative strength of each component stimulus changes in a way
that depends on an associative strength associated with the entire stimulus compound,
called the “aggregate associative strength,” and not just on the associative strength of
each component itself.
Rescorla and Wagner considered a compound CS AX, consisting of component stimuli
A and X, where the animal may have already experienced stimulus A, and stimulus X
might be new to the animal. Let VA,VX, and VAXrespectively denote the associative

strengths of stimuli A, X, and the compound AX. Suppose that on a trial the compound
CS AX is followed by a US, which we label stimulus Y. Then the associative strengths of

magnitude of a reward signal, with the caveat that the US in classical conditioning is not
necessarily rewarding or penalizing.) A key assumption of the model is that the aggregate
associative strength VAXis equal to VA+VX. The associative strengths as changed by
these s become the associative strengths at the beginning of the next trial.
To be complete, the model needs a response-generation mechanism, which is a way
of mapping values of Vs to CRs. Because this mapping would depend on details of
the experimental situation, Rescorla and Wagner did not specify a mapping but simply
assumed that larger Vs would produce stronger or more likely CRs, and that negative
Vs would mean that there would be no CRs.
The Rescorla–Wagner model accounts for the acquisition of CRs in a way that explains
blocking. As long as the aggregate associative strength, VAX, of the stimulus compound
is below the asymptotic level of associative strength, RY, that the US Y can support, the

prediction error RYVAXis positive. This means that over successive trials the associative
strengths VAandVXof the component stimuli increase until the aggregate associative
strength VAXequals RY, at which point the associative strengths stop changing (unless the
US changes). When a new component is added to a compound CS to which the animal has
already been conditioned, further conditioning with the augmented compound produces
little or no increase in the associative strength of the added CS component because the

error has already been reduced to zero, or to a low value. The occurrence of the US is
already predicted nearly perfectly, so little or no error—or surprise—is introduced by the
new CS component. Prior learning blocks learning to the new component.
To transition from Rescorla and Wagner’s model to the TD model of classical condi-
tioning (which we just call the TD model), we ﬁrst recast their model in terms of the
concepts that we are using throughout this book. Speciﬁcally, we match the notation
we use for learning with linear function approximation (Section 9.4), and we think of

the conditioning process as one of learning to predict the “magnitude of the US” on a
trial on the basis of the compound CS presented on that trial, where the magnitude of
a US Y is the RYof the Rescorla–Wagner model as given above. We also introduce
states. Because the Rescorla–Wagner model is a trial-level model, meaning that it deals
with how associative strengths change from trial to trial without considering any details
about what happens within and between trials, we do not have to consider how states

change during a trial until we present the full TD model in the following section. Instead,
here we simply think of a state as a way of labeling a trial in terms of the collection of
component CSs that are present on the trial.
Therefore, assume that trial-type, or state, sis described by a real-valued vector of
features x(s)=( x1(s),x2(s),...,x d(s))>where xi(s)=1i f CSi,t h e ithcomponent of a

348 Chapter 14: Psychology
compound CS, is present on the trial and 0 otherwise. Then if the d-dimensional vector
of associative strengths is w, the aggregate associative strength for trial-type sis
ˆv(s,w)=w>x(s). (14.1)
This corresponds to a value estimate in reinforcement learning, and we think of it as the
US prediction .
Now temporally let tdenote the number of a complete trial and not its usual meaning
as a time step (we revert to t’s usual meaning when we extend this to the TD model
below), and assume that Stis the state corresponding to trial t. Conditioning trial t
updates the associative strength vector wttowt+1as follows:
wt+1=wt+↵tx(St), (14.2)
where ↵is the step-size parameter, and—because here we are describing the Rescorla–
Wagner model— tis the prediction error
t=Rtˆv(St,wt). (14.3)
Rtis the target of the prediction on trial t, that is, the magnitude of the US, or in Rescorla
and Wagner’s terms, the associative strength that the US on the trial can support. Note

that because of the factor x(St) in (14.2), only the associative strengths of CS components
present on a trial are adjusted as a result of that trial. You can think of the prediction
error as a measure of surprise, and the aggregate associative strength as the animal’s
expectation that is violated when it does not match the target US magnitude.
From the perspective of machine learning, the Rescorla–Wagner model is an error-
correction supervised learning rule. It is essentially the same as the Least Mean Square

(LMS), or Widrow-Ho↵, learning rule (Widrow and Ho↵, 1960) that ﬁnds the weights—
here the associative strengths—that make the average of the squares of all the errors as
close to zero as possible. It is a “curve-ﬁtting,” or regression, algorithm that is widely
used in engineering and scientiﬁc applications (see Section 9.4).3
The Rescorla–Wagner model was very inﬂuential in the history of animal learning
theory because it showed that a “mechanistic” theory could account for the main facts
about blocking without resorting to more complex cognitive theories involving, for

example, an animal’s explicit recognition that another stimulus component had been
added and then scanning its short-term memory backward to reassess the predictive
relationships involving the US. The Rescorla–Wagner model showed how traditional
contiguity theories of conditioning—that temporal contiguity of stimuli was a necessary
and sucient condition for learning—could be adjusted in a simple way to account for
blocking (Moore and Schmajuk, 2008).

The Rescorla–Wagner model provides a simple account of blocking and some other
features of classical conditioning, but it is not a complete or perfect model of classical
3The only di↵erences between the LMS rule and the Rescorla–Wagner model are that for LMS the
input vectors xtcan have any real numbers as components, and—at least in the simplest version of the
LMS rule—the step-size parameter ↵does not depend on the input vector or the identity of the stimulus
setting the prediction target.

14.2. Classical Conditioning 349
conditioning. Di↵erent ideas account for a variety of other observed e↵ects, and progress
is still being made toward understanding the many subtleties of classical conditioning.
The TD model, which we describe next, though also not a complete or perfect model
model of classical conditioning, extends the Rescorla–Wagner model to address how
within-trial and between-trial timing relationships among stimuli can inﬂuence learning
and how higher-order conditioning might arise.
14.2.3 The TD Model
The TD model is a real-time model, as opposed to a trial-level model like the Rescorla–
Wagner model. A single step tin the Rescorla–Wagner model represents an entire
conditioning trial. The model does not apply to details about what happens during the
time a trial is taking place, or what might happen between trials. Within each trial an
animal might experience various stimuli whose onsets occur at particular times and that

have particular durations. These timing relationships strongly inﬂuence learning. The
Rescorla–Wagner model also does not include a mechanism for higher-order conditioning,
whereas for the TD model, higher-order conditioning is a natural consequence of the
bootstrapping idea that is at the base of TD algorithms.
To describe the TD model we begin with the formulation of the Rescorla–Wagner
model above, but tnow labels time steps within or between trials instead of complete

trials. Think of the time between tandt+ 1 as a small time interval, say .01 second, and
think of a trial as a sequences of states, one associated with each time step, where the
state at step tnow represents details of how stimuli are represented at tinstead of just
a label for the CS components present on a trial. In fact, we can completely abandon
the idea of trials. From the point of view of the animal, a trial is just a fragment of its
continuing experience interacting with its world. Following our usual view of an agent

interacting with its environment, imagine that the animal is experiencing an endless
sequence of states s, each represented by a feature vector x(s). That said, it is still often
convenient to refer to trials as fragments of time during which patterns of stimuli repeat
in an experiment.
State features are not restricted to describing the external stimuli that an animal
experiences; they can describe neural activity patterns that external stimuli produce in
an animal’s brain, and these patterns can be history-dependent, meaning that they can

be persistent patterns produced by sequences of external stimuli. Of course, we do not
know exactly what these neural activity patterns are, but a real-time model like the TD
model allows one to explore the consequences on learning of di↵erent hypotheses about
the internal representations of external stimuli. For these reasons, the TD model does
not commit to any particular state representation. In addition, because the TD model
includes discounting and eligibility traces that span time intervals between stimuli, the

model also makes it possible to explore how discounting and eligibility traces interact with
stimulus representations in making predictions about the results of classical conditioning
Below we describe some of the state representations that have been used with the
TD model and some of their implications, but for the moment we stay agnostic about

backward view of the semi-gradient TD( ) algorithm with linear function approximation
(Chapter 12), except that Rtin the model does not have to be a reward signal as it does
when the TD algorithm is used to learn a value function for policy-improvement.
14.2.4 TD Model Simulations
Real-time conditioning models like the TD model are interesting primarily because they
make predictions for a wide range of situations that cannot be represented by trial-level
models. These situations involve the timing and durations of conditionable stimuli, the
timing of these stimuli in relation to the timing of the US, and the timing and shapes
of CRs. For example, the US generally must begin after the onset of a neutral stimulus
for conditioning to occur, with the rate and e↵ectiveness of learning depending on the
inter-stimulus interval, or ISI, the interval between the onsets of the CS and the US. When
CRs appear, they generally begin before the appearance of the US and their temporal

proﬁles change during learning. In conditioning with compound CSs, the component
stimuli of the compound CSs may not all begin and end at the same time, sometimes
forming what is called a serial compound in which the component stimuli occur in a
sequence over time. Timing considerations like these make it important to consider how
stimuli are represented, how these representations unfold over time during and between
trials, and how they interact with discounting and eligibility traces.

14.2. Classical Conditioning 351PresenceComplete Serial CompoundStimulus RepresentationMicrostimuliCSUS
Figure 14.1: Three stimulus representations (in columns) sometimes used with the TD model.
Each row represents one element of the stimulus representation. The three representations vary
along a temporal generalization gradient, with no generalization between nearby time points in
the complete serial compound (left column) and complete generalization between nearby time
points in the presence representation (right column). The microstimulus representation occupies
a middle ground. The degree of temporal generalization determines the temporal granularity
with which US predictions are learned. Adapted with minor changes from Learning & Behavior ,
Evaluating the TD Model of Classical Conditioning, volume 40, 2012, p. 311, E. A. Ludvig, R. S.
Sutton, E. J. Kehoe. With permission of Springer.
Figure 14.1 shows three of the stimulus representations that have been used in exploring

the behavior of the TD model: the complete serial compound (CSC), the microstimulus
(MS), and the presence representations (Ludvig, Sutton, and Kehoe, 2012). These
representations di↵er in the degree to which they force generalization among nearby time
points during which a stimulus is present.
The simplest of the representations shown in Figure 14.1 is the presence representation
in the ﬁgure’s right column. This representation has a single feature for each component
CS present on a trial, where the feature has value 1 whenever that component is present,

and 0 otherwise.4The presence representation is not a realistic hypothesis about how
stimuli are represented in an animal’s brain, but as we describe below, the TD model
with this representation can produce many of the timing phenomena seen in classical
conditioning.
For the CSC representation (left column of Figure 14.1), the onset of each external
stimulus initiates a sequence of precisely-timed short-duration internal signals that

4In our formalism, there is a di↵erent state, St,f o re a c ht i m es t e p tduring a trial, and for a trial
in which a compound CS consists of ncomponent CSs of various durations occurring at various times
throughout the trial, there is a feature, xi,f o re a c hc o m p o n e n t CSi,i=1,...,n ,w h e r e xi(St)=1f o r
all times twhen the CS iis present, and equals zero otherwise.

352 Chapter 14: Psychology
continues until the external stimulus ends.5This is like assuming the animal’s nervous
system has a clock that keeps precise track of time during stimulus presentations; it is
what engineers call a “tapped delay line.” Like the presence representation, the CSC
representation is unrealistic as a hypothesis about how the brain internally represents
stimuli, but Ludvig et al. (2012) call it a “useful ﬁction” because it can reveal details of
how the TD model works when relatively unconstrained by the stimulus representation.
The CSC representation is also used in most TD models of dopamine-producing neurons
in the brain, a topic we take up in Chapter 15. The CSC representation is often viewed
as an essential part of the TD model, although this view is mistaken.
The MS representation (center column of Figure 14.1) is like the CSC representation
in that each external stimulus initiates a cascade of internal stimuli, but in this case the

internal stimuli—the microstimuli—are not of such limited and non-overlapping form;
they are extended over time and overlap. As time elapses from stimulus onset, di↵erent
sets of microstimuli become more or less active, and each subsequent microstimulus
becomes progressively wider in time and reaches a lower maximal level. Of course, there
are many MS representations depending on the nature of the microstimuli, and a number
of examples of MS representations have been studied in the literature, in some cases along

with proposals for how an animal’s brain might generate them (see the Bibliographic and
Historical Comments at the end of this chapter). MS representations are more realistic
than the presence or CSC representations as hypotheses about neural representations of
stimuli, and they allow the behavior of the TD model to be related to a broader collection
of phenomena observed in animal experiments. In particular, by assuming that cascades
of microstimuli are initiated by USs as well as by CSs, and by studying the signiﬁcant

e↵ects on learning of interactions between microstimuli, eligibility traces, and discounting,
the TD model is helping to frame hypotheses to account for many of the subtle phenomena
of classical conditioning and how an animal’s brain might produce them. We say more
about this below, particularly in Chapter 15 where we discuss reinforcement learning and
neuroscience.
Even with the simple presence representation, however, the TD model produces all the
basic properties of classical conditioning that are accounted for by the Rescorla–Wagner

model, plus features of conditioning that are beyond the scope of trial-level models. For
example, as we have already mentioned, a conspicuous feature of classical conditioning is
that the US generally must begin after the onset of a neutral stimulus for conditioning
to occur, and that after conditioning, the CR begins before the appearance of the US.
In other words, conditioning generally requires a positive ISI, and the CR generally
anticipates the US. How the strength of conditioning (e.g., the percentage of CRs elicited

by a CS) depends on the ISI varies substantially across species and response systems, but
it typically has the following properties: it is negligible for a zero or negative ISI, i.e., when
the US onset occurs simultaneously with, or earlier than, the CS onset (although research
has found that associative strengths sometimes increase slightly or become negative with
5In our formalism, for each CS component CSipresent on a trial, and for each time step tduring a
trial, there is a separate feature xt
i,w h e r e xt
i(St0)=1i f t=t0for any t0at which CSiis present, and

i,w h e r e xt
equals 0 otherwise. This is di↵erent from the CSC representation in Sutton and Barto (1990) in which
there are the same distinct features for each time step but no reference to external stimuli; hence the
name complete serial compound.

14.2. Classical Conditioning 353
negative ISIs); it increases to a maximum at a positive ISI where conditioning is most
e↵ective; and it then decreases to zero after an interval that varies widely with response
systems. The precise shape of this dependency for the TD model depends on the values
of its parameters and details of the stimulus representation, but these basic features of
ISI-dependency are core properties of the TD model.
wCSA
wCSA
Facilitation of remote associations in the TD modelOne of the theoretical issues arising
with serial-compound conditioning, that
is, conditioning with a compound CS
whose components occur in a sequence,
concerns the facilitation of remote asso-
ciations. It has been found that if the
empty trace interval between a ﬁrst CS
(CSA) and the US is ﬁlled with a second
CS (CSB) to form a serial-compound
stimulus, then conditioning to CSA is
facilitated. Shown to the right is the
behavior of the TD model with the pres-
ence representation in a simulation of

such an experiment whose timing details
are shown above. Consistent with the
experimental results (Kehoe, 1982), the
model shows facilitation of both the rate
of conditioning and the asymptotic level
of conditioning of the ﬁrst CS due to the
presence of the second CS.
wCSB
The Egger-Miller e↵ect in the TD modelA well-known demonstration of the
e↵ects on conditioning of temporal re-
lationships among stimuli within a trial
is an experiment by Egger and Miller
(1962) that involved two overlapping
CSs in a delay conﬁguration as shown

wCSB
to the right (top). Although CSB was
in a better temporal relationship with
the US, the presence of CSA substan-
tially reduced conditioning to CSB as
compared to controls in which CSA was
absent. Directly to the right is shown
the same result being generated by the
TD model in a simulation of this exper-
iment with the presence representation.
The TD model accounts for blocking
because it is an error-correcting learn-
ing rule like the Rescorla–Wagner model.
Beyond accounting for basic blocking re-

354 Chapter 14: Psychology
sults, however, the TD model predicts (with the presence representation and more complex
representations as well) that blocking is reversed if the blocked stimulus is moved earlier
wCSB
Figure 14.2: Temporal primacy overriding
blocking in the TD model.in time so that its onset occurs before the
onset of the blocking stimulus (like CSA in
the diagram to the right). This feature of
the TD model’s behavior deserves attention
because it had not been observed at the
time of the model’s introduction. Recall
that in blocking, if an animal has already
learned that one CS predicts a US, then
learning that a newly-added second CS also
predicts the US is much reduced, i.e., is
blocked. But if the newly-added second CS
begins earlier than the pretrained CS, then—
according to the TD model—learning to the
newly-added CS is not blocked. In fact, as
training continues and the newly-added CS
gains associative strength, the pretrained
CS loses associative strength. The behavior

of the TD model under these conditions
is shown in the lower part of Figure 14.2.
This simulation experiment di↵ered from the Egger-Miller experiment (bottom of the
preceding page) in that the shorter CS with the later onset was given prior training
until it was fully associated with the US. This surprising prediction led Kehoe, Schreurs,
and Graham (1987) to conduct the experiment using the well-studied rabbit nictitating
membrane preparation. Their results conﬁrmed the model’s prediction, and they noted

that non-TD models have considerable diculty explaining their data.
With the TD model, an earlier predictive stimulus takes precedence over a later
predictive stimulus because, like all the prediction methods described in this book, the
TD model is based on the backing-up or bootstrapping idea: updates to associative
strengths shift the strengths at a particular state toward the strength at later states.
Another consequence of bootstrapping is that the TD model provides an account of higher-

order conditioning, a feature of classical conditioning that is beyond the scope of the
Rescoral-Wagner and similar models. As we described above, higher-order conditioning
is the phenomenon in which a previously-conditioned CS can act as a US in conditioning
another initially neutral stimulus. Figure 14.3 shows the behavior of the TD model (again
with the presence representation) in a higher-order conditioning experiment—in this case

it is second-order conditioning. In the ﬁrst phase (not shown in the ﬁgure), CSB is trained
to predict a US so that its associative strength increases, here to 1.65. In the second
phase, CSA is paired with CSB in the absence of the US, in the sequential arrangement
shown at the top of the ﬁgure. CSA acquires associative strength even though it is never
paired with the US. With continued training, CSA’s associative strength reaches a peak
and then decreases because the associative strength of CSB, the secondary reinforcer,

decreases so that it loses its ability to provide secondary reinforcement. CSB’s associative

unless the original predictive relationships
are periodically refreshed by occasionally
inserting ﬁrst-order trials.
The TD model produces an analog of
second- and higher-order conditioning be-
cause ˆv(St+1,wt)ˆv(St,wt) appears in
the TD error t(14.5). This means that as
a result of previous learning, ˆv(St+1,wt)
can di↵er from ˆv(St,wt), making tnon-zero (a temporal di↵erence). This di↵erence has
the same status as Rt+1in (14.5), implying that as far as learning is concerned there
is no di↵erence between a temporal di↵erence and the occurrence of a US. In fact, this
feature of the TD algorithm is one of the major reasons for its development, which we now
understand through its connection to dynamic programming as described in Chapter 6.
Bootstrapping values is intimately related to second-order, and higher-order, conditioning.
In the examples of the TD model’s behavior described above, we examined only the
changes in the associative strengths of the CS components; we did not look at what

the model predicts about properties of an animal’s conditioned responses (CRs): their
timing, shape, and how they develop over conditioning trials. These properties depend
on the species, the response system being observed, and parameters of the conditioning
trials, but in many experiments with di↵erent animals and di↵erent response systems, the
magnitude of the CR, or the probability of a CR, increases as the expected time of the
US approaches. For example, in classical conditioning of a rabbit’s nictitating membrane

response that we mentioned above, over conditioning trials the delay from CS onset to
when the nictitating membrane begins to move across the eye decreases over trials, and
the amplitude of this anticipatory closure gradually increases over the interval between
the CS and the US until the membrane reaches maximal closure at the expected time of
the US. The timing and shape of this CR is critical to its adaptive signiﬁcance—covering
the eye too early reduces vision (even though the nictitating membrane is translucent),

while covering it too late is of little protective value. Capturing CR features like these is
challenging for models of classical conditioning.
The TD model does not include as part of its deﬁnition any mechanism for translating
the time course of the US prediction, ˆv(St,wt), into a proﬁle that can be compared
with the properties of an animal’s CR. The simplest choice is to let the time course of

356 Chapter 14: Psychology
a simulated CR equal the time course of the US prediction. In this case, features of
simulated CRs and how they change over trials depend only on the stimulus representation
chosen and the values of the model’s parameters ↵,, and .
Figure 14.4 shows the time courses of US predictions at di↵erent points during learning
with the three representations shown in Figure 14.1. For these simulations the US
occurred 25 time steps after the onset of the CS, and ↵=.05,=.95 and =.97.
With the CSC representation (Figure 14.4 left), the curve of the US prediction formed
by the TD model increases exponentially throughout the interval between the CS and
the US until it reaches a maximum exactly when the US occurs (at time step 25). This
exponential increase is the result of discounting in the TD model learning rule. With the
presence representation (Figure 14.4 middle), the US prediction is nearly constant while

the stimulus is present because there is only one weight, or associative strength, to be
learned for each stimulus. Consequently, the TD model with the presence representation
cannot recreate many features of CR timing. With an MS representation (Figure 14.4
right), the development of the TD model’s US prediction is more complicated. After 200
trials the prediction’s proﬁle is a reasonable approximation of the US prediction curve
produced with the CSC representation.
ˆvˆvˆv

ˆvˆvˆv
Figure 14.4: Time course of US prediction over the course of acquisition for the TD model
with three di↵erent stimulus representations. Left: With the complete serial compound (CSC),
the US prediction increases exponentially through the interval, peaking at the time of the US.
At asymptote (trial 200), the US prediction peaks at the US intensity (1 in these simulations).
Middle: With the presence representation, the US prediction converges to an almost constant

ˆvˆvˆv
level. This constant level is determined by the US intensity and the length of the CS–US interval.
Right: With the microstimulus representation, at asymptote, the TD model approximates the
exponentially increasing time course depicted with the CSC through a linear combination of the
di↵erent microstimuli. Adapted with minor changes from Learning & Behavior ,E v a l u a t i n gt h e
TD Model of Classical Conditioning, volume 40, 2012, E. A. Ludvig, R. S. Sutton, E. J. Kehoe.
With permission of Springer.

The US prediction curves shown in Figure 14.4 were not intended to precisely match
proﬁles of CRs as they develop during conditioning in any particular animal experiment,
but they illustrate the strong inﬂuence that the stimulus representation has on predictions
derived from the TD model. Further, although we can only mention it here, how the
stimulus representation interacts with discounting and eligibility traces is important

14.3. Instrumental Conditioning 357
in determining properties of the US prediction proﬁles produced by the TD model.
Another dimension beyond what we can discuss here is the inﬂuence of di↵erent response-
generation mechanisms that translate US predictions into CR proﬁles; the proﬁles shown
in Figure 14.4 are “raw” US prediction proﬁles. Even without any special assumption
about how an animal’s brain might produce overt responses from US predictions, however,
the proﬁles in Figure 14.4 for the CSC and MS representations increase as the time of the
US approaches and reach a maximum at the time of the US, as is seen in many animal
conditioning experiments.
The TD model, when combined with particular stimulus representations and response-
generation mechanisms, is able to account for a surprisingly wide range of phenomena
observed in animal classical conditioning experiments, but it is far from being a perfect
model. To generate other details of classical conditioning the model needs to be extended,

perhaps by adding model-based elements and mechanisms for adaptively altering some of
its parameters. Other approaches to modeling classical conditioning depart signiﬁcantly
from the Rescorla–Wagner-style error-correction process. Bayesian models, for example,
work within a probabilistic framework in which experience revises probability estimates.
All of these models usefully contribute to our understanding of classical conditioning.
Perhaps the most notable feature of the TD model is that it is based on a theory—the

theory we have described in this book—that suggests an account of what an animal’s
nervous system is trying to do while undergoing conditioning: it is trying to form accurate
long-term predictions , consistent with the limitations imposed by the way stimuli are
represented and how the nervous system works. In other words, it suggests a normative
account of classical conditioning in which long-term, instead of immediate, prediction is a
key feature.
The development of the TD model of classical conditioning is one instance in which the

explicit goal was to model some of the details of animal learning behavior. In addition to
its standing as an algorithm , then, TD learning is also the basis of this model of aspects
of biological learning. As we discuss in Chapter 15, TD learning has also turned out
to underlie an inﬂuential model of the activity of neurons that produce dopamine, a
chemical in the brain of mammals that is deeply involved in reward processing. These
are instances in which reinforcement learning theory makes detailed contact with animal
behavioral and neural data.

We now turn to considering correspondences between reinforcement learning and animal
behavior in instrumental conditioning experiments, the other major type of laboratory
experiment studied by animal learning psychologists.
14.3 Instrumental Conditioning
Ininstrumental conditioning experiments learning depends on the consequences of be-
havior: the delivery of a reinforcing stimulus is contingent on what the animal does.
In classical conditioning experiments, in contrast, the reinforcing stimulus—the US—is

delivered independently of the animal’s behavior. Instrumental conditioning is usually
considered to be the same as operant conditioning , the term B. F. Skinner (1938, 1963)
introduced for experiments with behavior-contingent reinforcement, though the experi-

358 Chapter 14: Psychology
ments and theories of those who use these two terms di↵er in a number of ways, some of
which we touch on below. We will exclusively use the term instrumental conditioning for
experiments in which reinforcement is contingent upon behavior. The roots of instrumen-
tal conditioning go back to experiments performed by the American psychologist Edward
Thorndike one hundred years before publication of the ﬁrst edition of this book.
One of Thorndike’s puzzle boxes.
Reprinted from Thorndike, Animal Intelligence: An
Experimental Study of the Associative Processes in
Animals, The Psychological Review, Series of Mono-
graph Supplements II(4), Macmillan, New York, 1898.Thorndike observed the behavior of cats
when they were placed in “puzzle boxes,”
such as the one at the right, from which
they could escape by appropriate actions.
For example, a cat could open the door
of one box by performing a sequence of
three separate actions: depressing a plat-
form at the back of the box, pulling a string

by clawing at it, and pushing a bar up or
down. When ﬁrst placed in a puzzle box,
with food visible outside, all but a few of
Thorndike’s cats displayed “evident signs
of discomfort” and extraordinarily vigorous
activity “to strive instinctively to escape
from conﬁnement” (Thorndike, 1898).
In experiments with di↵erent cats and
boxes with di↵erent escape mechanisms, Thorndike recorded the amounts of time each
cat took to escape over multiple experiences in each box. He observed that the time

almost invariably decreased with successive experiences, for example, from 300 seconds
to 6 or 7 seconds. He described cats’ behavior in a puzzle box like this:
The cat that is clawing all over the box in her impulsive struggle will probably
claw the string or loop or button so as to open the door. And gradually all the
other non-successful impulses will be stamped out and the particular impulse
leading to the successful act will be stamped in by the resulting pleasure,

until, after many trials, the cat will, when put in the box, immediately claw
the button or loop in a deﬁnite way. (Thorndike 1898, p. 13)
These and other experiments (some with dogs, chicks, monkeys, and even ﬁsh) led
Thorndike to formulate a number of “laws” of learning, the most inﬂuential being the
Law of E↵ect , a version of which we quoted in Chapter 1 (page 15). This law describes
what is generally known as learning by trial and error. As mentioned in Chapter 1,

many aspects of the Law of E↵ect have generated controversy, and its details have been
modiﬁed over the years. Still the law—in one form or another—expresses an enduring
principle of learning.
Essential features of reinforcement learning algorithms correspond to features of animal
learning described by the Law of E↵ect. First, reinforcement learning algorithms are
selectional , meaning that they try alternatives and select among them by comparing their
consequences. Second, reinforcement learning algorithms are associative , meaning that

the alternatives found by selection are associated with particular situations, or states,
to form the agent’s policy. Like learning described by the Law of E↵ect, reinforcement

14.3. Instrumental Conditioning 359
learning is not just the process of ﬁnding actions that produce a lot of reward, but also
ofconnecting these actions to situations or states. Thorndike used the phrase learning
by “selecting and connecting” (Hilgard, 1956). Natural selection in evolution is a prime
example of a selectional process, but it is not associative (at least as it is commonly
understood); supervised learning is associative, but it is not selectional because it relies
on instructions that directly tell the agent how to change its behavior.
In computational terms, the Law of E↵ect describes an elementary way of combining
search andmemory : search in the form of trying and selecting among many actions
in each situation, and memory in the form of associations linking situations with the
actions found—so far—to work best in those situations. Search and memory are essential
components of all reinforcement learning algorithms, whether memory takes the form of

an agent’s policy, value function, or environment model.
A reinforcement learning algorithm’s need to search means that it has to explore in
some way. Animals clearly explore as well, and early animal learning researchers disagreed
about the degree of guidance an animal uses in selecting its actions in situations like
Thorndike’s puzzle boxes. Are actions the result of “absolutely random, blind groping”
(Woodworth, 1938, p. 777), or is there some degree of guidance, either from prior learning,

reasoning, or other means? Although some thinkers, including Thorndike, seem to have
taken the former position, others favored more deliberate exploration. Reinforcement
learning algorithms allow wide latitude for how much guidance an agent can employ in
selecting actions. The forms of exploration we have used in the algorithms presented
in this book, such as "-greedy and upper-conﬁdence-bound action selection, are merely
among the simplest. More sophisticated methods are possible, with the only stipulation

being that there has to be some form of exploration for the algorithms to work e↵ectively.
The feature of our treatment of reinforcement learning allowing the set of actions
available at any time to depend on the environment’s current state echoes something
Thorndike observed in his cats’ puzzle-box behaviors. The cats selected actions from
those that they instinctively perform in their current situation, which Thorndike called
their “instinctual impulses.” First placed in a puzzle box, a cat instinctively scratches,

claws, and bites with great energy: a cat’s instinctual responses to ﬁnding itself in a
conﬁned space. Successful actions are selected from these and not from every possible
action or activity. This is like the feature of our formalism where the action selected
from a state sbelongs to a set of admissible actions, A(s). Specifying these sets is an
important aspect of reinforcement learning because it can radically simplify learning.
They are like an animal’s instinctual impulses. On the other hand, Thorndike’s cats might

have been exploring according to an instinctual context-speciﬁc ordering over actions
rather than by just selecting from a set of instinctual impulses. This is another way to
make reinforcement learning easier.
Among the most prominent animal learning researchers inﬂuenced by the Law of E↵ect
were Clark Hull (e.g., Hull, 1943) and B. F. Skinner (e.g., Skinner, 1938). At the center
of their research was the idea of selecting behavior on the basis of its consequences.
Reinforcement learning has features in common with Hull’s theory, which included

eligibility-like mechanisms and secondary reinforcement to account for the ability to learn
when there is a signiﬁcant time interval between an action and the consequent reinforcing

360 Chapter 14: Psychology
stimulus (see Section 14.4). Randomness also played a role in Hull’s theory through what
he called “behavioral oscillation” to introduce exploratory behavior.
Skinner did not fully subscribe to the memory aspect of the Law of E↵ect. Being averse
to the idea of associative linkages, he instead emphasized selection from spontaneously-
emitted behavior. He introduced the term “operant” to emphasize the key role of an
action’s e↵ects on an animal’s environment. Unlike the experiments of Thorndike and
others, which consisted of sequences of separate trials, Skinner’s operant conditioning
experiments allowed animal subjects to behave for extended periods of time without
interruption. He invented the operant conditioning chamber, now called a “Skinner box,”
the most basic version of which contains a lever or key that an animal can press to obtain
a reward, such as food or water, which would be delivered according to a well-deﬁned rule,

called a reinforcement schedule. By recording the cumulative number of lever presses
as a function of time, Skinner and his followers could investigate the e↵ect of di↵erent
reinforcement schedules on the animal’s rate of lever-pressing. Modeling results from
experiments likes these using the reinforcement learning principles we present in this
book is not well developed, but we mention some exceptions in the Bibliographic and
Historical Remarks section at the end of this chapter.
Another of Skinner’s contributions resulted from his recognition of the e↵ectiveness of

training an animal by reinforcing successive approximations of the desired behavior, a
process he called shaping . Although this technique had been used by others, including
Skinner himself, its signiﬁcance was impressed upon him when he and colleagues were
attempting to train a pigeon to bowl by swiping a wooden ball with its beak. After
waiting for a long time without seeing any swipe that they could reinforce, they
... decided to reinforce any response that had the slightest resemblance to

a swipe—perhaps, at ﬁrst, merely the behavior of looking at the ball—and
then to select responses which more closely approximated the ﬁnal form. The
result amazed us. In a few minutes, the ball was caroming o↵ the walls of
the box as if the pigeon had been a champion squash player. (Skinner, 1958,
p. 94)
Not only did the pigeon learn a behavior that is unusual for pigeons, it learned quickly
through an interactive process in which its behavior and the reinforcement contingencies

p. 94)
changed in response to each other. Skinner compared the process of altering reinforcement
contingencies to the work of a sculptor shaping clay into a desired form. Shaping is a
powerful technique for computational reinforcement learning systems as well. When it is
dicult for an agent to receive any non-zero reward signal at all, either due to sparseness
of rewarding situations or their inaccessibility given initial behavior, starting with an
easier problem and incrementally increasing its diculty as the agent learns can be an

e↵ective, and sometimes indispensable, strategy.
A concept from psychology that is especially relevant in the context of instrumental
conditioning is motivation , which refers to processes that inﬂuence the direction and
strength, or vigor, of behavior. Thorndike’s cats, for example, were motivated to escape
from puzzle boxes because they wanted the food that was sitting just outside. Obtaining
this goal was rewarding to them and reinforced the actions allowing them to escape. It
is dicult to link the concept of motivation, which has many dimensions, in a precise

14.4. Delayed Reinforcement 361
way to reinforcement learning’s computational perspective, but there are clear links with
some of its dimensions.
In one sense, a reinforcement learning agent’s reward signal is at the base of its
motivation: the agent is motivated to maximize the total reward it receives over the long
run. A key facet of motivation, then, is what makes an agent’s experience rewarding. In
reinforcement learning, reward signals depend on the state of the reinforcement learning
agent’s environment and the agent’s actions. Further, as pointed out in Chapter 1, the
state of the agent’s environment not only includes information about what is external to
the machine, like an organism or a robot, that houses the agent, but also what is internal
to this machine. Some internal state components correspond to what psychologists call
an animal’s motivational state , which inﬂuences what is rewarding to the animal. For
example, an animal will be more rewarded by eating when it is hungry than when it has

just ﬁnished a satisfying meal. The concept of state dependence is broad enough to allow
for many types of modulating inﬂuences on the generation of reward signals.
Value functions provide a further link to psychologists’ concept of motivation. If the
most basic motive for selecting an action is to obtain as much reward as possible, for a
reinforcement learning agent that selects actions using a value function, a more proximal
motive is to ascend the gradient of its value function , that is, to select actions expected

to lead to the most highly-valued next states (or what is essentially the same thing, to
select actions with the greatest action-values). For these agents, value functions are the
main driving force determining the direction of their behavior.
Another dimension of motivation is that an animal’s motivational state not only
inﬂuences learning, but also inﬂuences the strength, or vigor, of the animal’s behavior
after learning. For example, after learning to ﬁnd food in the goal box of a maze, a hungry

rat will run faster to the goal box than one that is not hungry. This aspect of motivation
does not link so cleanly to the reinforcement learning framework we present here, but
in the Bibliographical and Historical Remarks section at the end of this chapter we cite
several publications that propose theories of behavioral vigor based on reinforcement
learning.
We turn now to the subject of learning when reinforcing stimuli occur well after the

learning.
events they reinforce. The mechanisms used by reinforcement learning algorithms to
enable learning with delayed reinforcement—eligibility traces and TD learning—closely
correspond to psychologists’ hypotheses about how animals can learn under these condi-
tions.
14.4 Delayed Reinforcement
The Law of E↵ect requires a backward e↵ect on connections, and some early critics of the
law could not conceive of how the present could a↵ect something that was in the past. This
concern was ampliﬁed by the fact that learning can even occur when there is a considerable

tions.
delay between an action and the consequent reward or penalty. Similarly, in classical
conditioning, learning can occur when US onset follows CS o↵set by a non-negligible time
interval. We call this the problem of delayed reinforcement, which is related to what
Minsky (1961) called the “credit-assignment problem for learning systems”: how do you

362 Chapter 14: Psychology
distribute credit for success among the many decisions that may have been involved in
producing it? The reinforcement learning algorithms presented in this book include two
basic mechanisms for addressing this problem. The ﬁrst is the use of eligibility traces,
and the second is the use of TD methods to learn value functions that provide nearly
immediate evaluations of actions (in tasks like instrumental conditioning experiments) or
that provide immediate prediction targets (in tasks like classical conditioning experiments).
Both of these methods correspond to similar mechanisms proposed in theories of animal
learning.
Pavlov (1927) pointed out that every stimulus must leave a trace in the nervous system
that persists for some time after the stimulus ends, and he proposed that stimulus traces
make learning possible when there is a temporal gap between the CS o↵set and the
US onset. To this day, conditioning under these conditions is called trace conditioning

learning.
(page 344). Assuming a trace of the CS remains when the US arrives, learning occurs
through the simultaneous presence of the trace and the US. We discuss some proposals
for trace mechanisms in the nervous system in Chapter 15.
Stimulus traces were also proposed as a means for bridging the time interval between
actions and consequent rewards or penalties in instrumental conditioning. In Hull’s
inﬂuential learning theory, for example, “molar stimulus traces” accounted for what
he called an animal’s goal gradient , a description of how the maximum strength of an

instrumentally-conditioned response decreases with increasing delay of reinforcement
(Hull, 1932, 1943). Hull hypothesized that an animal’s actions leave internal stimuli whose
traces decay exponentially as functions of time since an action was taken. Looking at the
animal learning data available at the time, he hypothesized that the traces e↵ectively
reach zero after 30 to 40 seconds.
The eligibility traces used in the algorithms described in this book are like Hull’s

traces: they are decaying traces of past state visitations, or of past state–action pairs.
Eligibility traces were introduced by Klopf (1972) in his neuronal theory in which they
are temporally-extended traces of past activity at synapses, the connections between
neurons. Klopf’s traces are more complex than the exponentially-decaying traces our
algorithms use, and we discuss this more when we take up his theory in Section 15.9.
To account for goal gradients that extend over longer time periods than spanned

by stimulus traces, Hull (1943) proposed that longer gradients result from conditioned
reinforcement passing backwards from the goal, a process acting in conjunction with
his molar stimulus traces. Animal experiments showed that if conditions favor the
development of conditioned reinforcement during a delay period, learning does not
decrease with increased delay as much as it does under conditions that obstruct secondary
reinforcement. Conditioned reinforcement is favored if there are stimuli that regularly
occur during the delay interval. Then it is as if reward is not actually delayed because

there is more immediate conditioned reinforcement. Hull therefore envisioned that there
is a primary gradient based on the delay of the primary reinforcement mediated by
stimulus traces, and that this is progressively modiﬁed, and lengthened, by conditioned
reinforcement.
Algorithms presented in this book that use both eligibility traces and value functions
to enable learning with delayed reinforcement correspond to Hull’s hypothesis about how

14.5. Cognitive Maps 363
animals are able to learn under these conditions. The actor–critic architecture discussed
in Sections 13.5, 15.7, and 15.8 illustrates this correspondence most clearly. The critic uses
a TD algorithm to learn a value function associated with the system’s current behavior,
that is, to predict the current policy’s return. The actor updates the current policy based
on the critic’s predictions, or more exactly, on changes in the critic’s predictions. The
TD error produced by the critic acts as a conditioned reinforcement signal for the actor,
providing an immediate evaluation of performance even when the primary reward signal
itself is considerably delayed. Algorithms that estimate action-value functions, such as
Q-learning and Sarsa, similarly use TD learning principles to enable learning with delayed
reinforcement by means of conditioned reinforcement. The close parallel between TD
learning and the activity of dopamine producing neurons that we discuss in Chapter 15

lends additional support to links between reinforcement learning algorithms and this
aspect of Hull’s learning theory.
14.5 Cognitive Maps
Model-based reinforcement learning algorithms use environment models that have elements
in common with what psychologists call cognitive maps . Recall from our discussion of
planning and learning in Chapter 8 that by an environment model we mean anything
an agent can use to predict how its environment will respond to its actions in terms of
state transitions and rewards, and by planning we mean any process that computes a

policy from such a model. Environment models consist of two parts: the state-transition
part encodes knowledge about the e↵ect of actions on state changes, and the reward-
model part encodes knowledge about the reward signals expected for each state or each
state–action pair. A model-based algorithm selects actions by using a model to predict
the consequences of possible courses of action in terms of future states and the reward
signals expected to arise from those states. The simplest kind of planning is to compare

the predicted consequences of collections of “imagined” sequences of decisions.
Questions about whether or not animals use environment models, and if so, what are the
models like and how are they learned, have played inﬂuential roles in the history of animal
learning research. Some researchers challenged the then-prevailing stimulus-response
(S–R) view of learning and behavior, which corresponds to the simplest model-free way
of learning policies, by demonstrating latent learning . In the earliest latent learning

experiment, two groups of rats were run in a maze. For the experimental group, there
was no reward during the ﬁrst stage of the experiment, but food was suddenly introduced
into the goal box of the maze at the start of the second stage. For the control group, food
was in the goal box throughout both stages. The question was whether or not rats in the
experimental group would have learned anything during the ﬁrst stage in the absence
of food reward. Although the experimental rats did not appear to learn much during

the ﬁrst, unrewarded, stage, as soon as they discovered the food that was introduced
in the second stage, they rapidly caught up with the rats in the control group. It was
concluded that “during the non-reward period, the rats [in the experimental group] were
developing a latent learning of the maze which they were able to utilize as soon as reward
was introduced” (Blodgett, 1929).

364 Chapter 14: Psychology
Latent learning is most closely associated with the psychologist Edward Tolman, who
interpreted this result, and others like it, as showing that animals could learn a “cognitive
map of the environment” in the absence of rewards or penalties, and that they could use
the map later when they were motivated to reach a goal (Tolman, 1948). A cognitive map
could also allow a rat to plan a route to the goal that was di↵erent from the route the rat
had used in its initial exploration. Explanations of results like these led to the enduring
controversy lying at the heart of the behaviorist/cognitive dichotomy in psychology. In
modern terms, cognitive maps are not restricted to models of spatial layouts but are
more generally environment models, or models of an animal’s “task space” (e.g., Wilson,
Takahashi, Schoenbaum, and Niv, 2014). The cognitive map explanation of latent learning
experiments is analogous to the claim that animals use model-based algorithms, and that

environment models can be learned even without explicit rewards or penalties. Models
are then used for planning when the animal is motivated by the appearance of rewards or
penalties.
Tolman’s account of how animals learn cognitive maps was that they learn stimulus-
stimulus, or S–S, associations by experiencing successions of stimuli as they explore an
environment. In psychology this is called expectancy theory : given S–S associations, the
occurrence of a stimulus generates an expectation about the stimulus to come next. This

is much like what control engineers call system identiﬁcation , in which a model of a
system with unknown dynamics is learned from labeled training examples. In the simplest
discrete-time versions, training examples are S–S0pairs, where S is a state and S0,t h e
subsequent state, is the label. When S is observed, the model creates the “expectation”
that S0will be observed next. Models more useful for planning involve actions as well,
so that examples look like SA–S0,w h e r eS0is expected when action A is executed in

state S. It is also useful to learn how the environment generates rewards. In this case,
examples are of the form S– Ror SA– R,w h e r e Ris a reward signal associated with S or
the SA pair. These are all forms of supervised learning by which an agent can acquire
cognitive-like maps whether or not it receives any non-zero reward signals while exploring
its environment.
14.6 Habitual and Goal-directed Behavior
The distinction between model-free and model-based reinforcement learning algorithms
corresponds to the distinction psychologists make between habitual andgoal-directed

control of learned behavioral patterns. Habits are behavior patterns triggered by appro-
priate stimuli and then performed more-or-less automatically. Goal-directed behavior,
according to how psychologists use the phrase, is purposeful in the sense that it is con-
trolled by knowledge of the value of goals and the relationship between actions and their
consequences. Habits are sometimes said to be controlled by antecedent stimuli, whereas
goal-directed behavior is said to be controlled by its consequences (Dickinson, 1980,

1985). Goal-directed control has the advantage that it can rapidly change an animal’s
behavior when the environment changes its way of reacting to the animal’s actions. While
habitual behavior responds quickly to input from an accustomed environment, it is unable
to quickly adjust to changes in the environment. The development of goal-directed

14.6. Habitual and Goal-directed Behavior 365
behavioral control was likely a major advance in the evolution of animal intelligence.
Figure 14.5 illustrates the di↵erence between model-free and model-based decision
strategies in a hypothetical task in which a rat has to navigate a maze that has distinctive
goal boxes, each delivering an associated reward of the magnitude shown (Figure 14.5
top). Starting at S 1, the rat has to ﬁrst select left (L) or right (R) and then has to select
L or R again at S 2or S 3to reach one of the goal boxes. The goal boxes are the terminal
states of each episode of the rat’s episodic task. A model-free strategy (Figure 14.5 lower
left) relies on stored values for state–action pairs. These action values are estimates of
the highest return the rat can expect for each action taken from each (nonterminal) state.
They are obtained over many trials of running the maze from start to ﬁnish. When the
action values have become good enough estimates of the optimal returns, the rat just has

to select at each state the action with the largest action value in order to make optimal
S1, LS2, LS2, RS3, LS3, RS1, R430432Model-Free= 4= 0= 2= 3RewardModel-BasedS1LRS2S3LRLR(4)S2S3S14023Figure 14.5: Model-based and model-free strategies to solve a hypothetical sequential action-
selection problem. Top: a rat navigates a maze with distinctive goal boxes, each associated
with a reward having the value shown. Lower left: a model-free strategy relies on stored action
values for all the state–action pairs obtained over many learning trials. To make decisions the

rat just has to select at each state the action with the largest action value for that state. Lower
right: in a model-based strategy, the rat learns an environment model, consisting of knowledge
of state–action-next-state transitions and a reward model consisting of knowledge of the reward
associated with each distinctive goal box. The rat can decide which way to turn at each state
by using the model to simulate sequences of action choices to ﬁnd a path yielding the highest

return. Adapted from Trends in Cognitive Science , volume 10, number 8, Y. Niv, D. Joel, and P.
Dayan, A Normative Perspective on Motivation, p. 376, 2006, with permission from Elsevier.

366 Chapter 14: Psychology
decisions. In this case, when the action-value estimates become accurate enough, the
rat selects L from S 1and R from S 2to obtain the maximum return of 4. A di↵erent
model-free strategy might simply rely on a cached policy instead of action values, making
direct links from S 1to L and from S 2to R. In neither of these strategies do decisions
rely on an environment model. There is no need to consult a state-transition model, and
no connection is required between the features of the goal boxes and the rewards they
deliver.
Figure 14.5 (lower right) illustrates a model-based strategy. It uses an environment
model consisting of a state-transition model and a reward model. The state-transition
model is shown as a decision tree, and the reward model associates the distinctive features
of the goal boxes with the rewards to be found in each. (The rewards associated with
states S 1,S2, and S 3are also part of the reward model, but here they are zero and are

deliver.
not shown.) A model-based agent can decide which way to turn at each state by using
the model to simulate sequences of action choices to ﬁnd a path yielding the highest
return. In this case the return is the reward obtained from the outcome at the end of
the path. Here, with a suciently accurate model, the rat would select L and then R to
obtain reward of 4. Comparing the predicted returns of simulated paths is a simple form
of planning, which can be done in a variety of ways as discussed in Chapter 8.

When the environment of a model-free agent changes the way it reacts to the agent’s
actions, the agent has to acquire new experience in the changed environment during
which it can update its policy and/or value function. In the model-free strategy shown
in Figure 14.5 (lower left), for example, if one of the goal boxes were to somehow shift
to delivering a di↵erent reward, the rat would have to traverse the maze, possibly many

times, to experience the new reward upon reaching that goal box, all the while updating
either its policy or its action-value function (or both) based on this experience. The key
point is that for a model-free agent to change the action its policy speciﬁes for a state, or
to change an action value associated with a state, it has to move to that state, act from
it, possibly many times, and experience the consequences of its actions.

A model-based agent can accommodate changes in its environment without this kind
of ‘personal experience’ with the states and actions a↵ected by the change. A change in
its model automatically (through planning) changes its policy. Planning can determine
the consequences of changes in the environment that have never been linked together in
the agent’s own experience. For example, again referring to the maze task of Figure 14.5,
imagine that a rat with a previously learned transition and reward model is placed directly

in the goal box to the right of S 2to ﬁnd that the reward available there now has value 1
instead of 4. The rat’s reward model will change even though the action choices required
to ﬁnd that goal box in the maze were not involved. The planning process will bring
knowledge of the new reward to bear on maze running without the need for additional
experience in the maze; in this case changing the policy to right turns at both S 1and S 3
to obtain a return of 3.
Exactly this logic is the basis of outcome-devaluation experiments with animals. Results

from these experiments provide insight into whether an animal has learned a habit or if
its behavior is under goal-directed control. Outcome-devaluation experiments are like
latent-learning experiments in that the reward changes from one stage to the next. After

14.6. Habitual and Goal-directed Behavior 367
an initial rewarded stage of learning, the reward value of an outcome is changed, including
being shifted to zero or even to a negative value.
An early important experiment of this type was conducted by Adams and Dickinson
(1981). They trained rats via instrumental conditioning until the rats energetically pressed
a lever for sucrose pellets in a training chamber. The rats were then placed in the same
chamber with the lever retracted and allowed non-contingent food, meaning that pellets
were made available to them independently of their actions. After 15-minutes of this
free-access to the pellets, rats in one group were injected with the nausea-inducing poison
lithium chloride. This was repeated for three sessions, in the last of which none of the
injected rats consumed any of the non-contingent pellets, indicating that the reward
value of the pellets had been decreased—the pellets had been devalued. In the next stage

taking place a day later, the rats were again placed in the chamber and given a session of
extinction training, meaning that the response lever was back in place but disconnected
from the pellet dispenser so that pressing it did not release pellets. The question was
whether the rats that had the reward value of the pellets decreased would lever-press
less than rats that did not have the reward value of the pellets decreased, even without
experiencing the devalued reward as a result of lever-pressing. It turned out that the

injected rats had signiﬁcantly lower response rates than the non-injected rats right from
the start of the extinction trials .
Adams and Dickinson concluded that the injected rats associated lever pressing with
consequent nausea by means of a cognitive map linking lever pressing to pellets, and
pellets to nausea. Hence, in the extinction trials, the rats “knew” that the consequences
of pressing the lever would be something they did not want, and so they reduced their
lever-pressing right from the start. The important point is that they reduced lever-pressing

without ever having experienced lever-pressing directly followed by being sick: no lever
was present when they were made sick. They seemed able to combine knowledge of the
outcome of a behavioral choice (pressing the lever will be followed by getting a pellet)
with the reward value of the outcome (pellets are to be avoided) and hence could alter
their behavior accordingly. Not every psychologist agrees with this “cognitive” account
of this kind of experiment, and it is not the only possible way to explain these results,

but the model-based planning explanation is widely accepted.
Nothing prevents an agent from using both model-free and model-based algorithms, and
there are good reasons for using both. We know from our own experience that with enough
repetition, goal-directed behavior tends to turn into habitual behavior. Experiments show
that this happens for rats too. Adams (1982) conducted an experiment to see if extended
training would convert goal-directed behavior into habitual behavior. He did this by
comparing the e↵ect of outcome devaluation on rats that experienced di↵erent amounts

of training. If extended training made the rats less sensitive to devaluation compared to
rats that received less training, this would be evidence that extended training made the
behavior more habitual. Adams’ experiment closely followed the Adams and Dickinson
(1981) experiment just described. Simplifying a bit, rats in one group were trained until
they made 100 rewarded lever-presses, and rats in the other group—the overtrained
group—were trained until they made 500 rewarded lever-presses. After this training,

the reward value of the pellets was decreased (using lithium chloride injections) for rats

368 Chapter 14: Psychology
in both groups. Then both groups of rats were given a session of extinction training.
Adams’ question was whether devaluation would e↵ect the rate of lever-pressing for the
overtrained rats less than it would for the non-overtrained rats, which would be evidence
that extended training reduces sensitivity to outcome devaluation. It turned out that
devaluation strongly decreased the lever-pressing rate of the non-overtrained rats. For
the overtrained rats, in contrast, devaluation had little e↵ect on their lever-pressing; in
fact, if anything, it made it more vigorous. (The full experiment included control groups
showing that the di↵erent amounts of training did not by themselves signiﬁcantly e↵ect
lever-pressing rates after learning.) This result suggested that while the non-overtrained
rats were acting in a goal-directed manner sensitive to their knowledge of the outcome of
their actions, the overtrained rats had developed a lever-pressing habit.

Viewing this and other results like it from a computational perspective provides insight
as to why one might expect animals to behave habitually in some circumstances, in a
goal-directed way in others, and why they shift from one mode of control to another as
they continue to learn. While animals undoubtedly use algorithms that do not exactly
match those we have presented in this book, one can gain insight into animal behavior by
considering the tradeo↵s that various reinforcement learning algorithms imply. An idea

developed by computational neuroscientists Daw, Niv, and Dayan (2005) is that animals
use both model-free and model-based processes. Each process proposes an action, and
the action chosen for execution is the one proposed by the process judged to be the more
trustworthy of the two as determined by measures of conﬁdence that are maintained
throughout learning. Early in learning the planning process of a model-based system is
more trustworthy because it chains together short-term predictions which can become

accurate with less experience than long-term predictions of the model-free process. But
with continued experience, the model-free process becomes more trustworthy because
planning is prone to making mistakes due to model inaccuracies and short-cuts necessary
to make planning feasible, such as various forms of “tree-pruning”: the removal of
unpromising search tree branches. According to this idea one would expect a shift from

goal-directed behavior to habitual behavior as more experience accumulates. Other ideas
have been proposed for how animals arbitrate between goal-directed and habitual control,
and both behavioral and neuroscience research continues to examine this and related
questions.
The distinction between model-free and model-based algorithms is proving to be useful
for this research. One can examine the computational implications of these types of
algorithms in abstract settings that expose basic advantages and limitations of each

type. This serves both to suggest and to sharpen questions that guide the design
of experiments necessary for increasing psychologists’ understanding of habitual and
goal-directed behavioral control.
14.7 Summary
Our goal in this chapter has been to discuss correspondences between reinforcement
learning and the experimental study of animal learning in psychology. We emphasized
at the outset that reinforcement learning as described in this book is not intended

14.7. Summary 369
to model details of animal behavior. It is an abstract computational framework that
explores idealized situations from the perspective of artiﬁcial intelligence and engineering.
But many of the basic reinforcement learning algorithms were inspired by psychological
theories, and in some cases, these algorithms have contributed to the development of
new animal learning models. This chapter described the most conspicuous of these
correspondences.
The distinction in reinforcement learning between algorithms for prediction and al-
gorithms for control parallels animal learning theory’s distinction between classical, or
Pavlovian, conditioning and instrumental conditioning. The key di↵erence between
instrumental and classical conditioning experiments is that in the former the reinforcing
stimulus is contingent upon the animal’s behavior, whereas in the latter it is not. Learning
to predict via a TD algorithm corresponds to classical conditioning, and we described

theTD model of classical conditioning as one instance in which reinforcement learning
principles account for some details of animal learning behavior. This model generalizes
the inﬂuential Rescorla–Wagner model by including the temporal dimension where events
within individual trials inﬂuence learning, and it provides an account of second-order
conditioning, where predictors of reinforcing stimuli become reinforcing themselves. It
also is the basis of an inﬂuential view of the activity of dopamine neurons in the brain,
something we take up in Chapter 15.

Learning by trial and error is at the base of the control aspect of reinforcement learning.
We presented some details about Thorndike’s experiments with cats and other animals that
led to his Law of E↵ect , which we discussed here and in Chapter 1 (page 15). We pointed
out that in reinforcement learning, exploration does not have to be limited to “blind
groping”; trials can be generated by sophisticated methods using innate and previously
learned knowledge as long as there is some exploration. We discussed the training method

B. F. Skinner called shaping in which reward contingencies are progressively altered to
train an animal to successively approximate a desired behavior. Shaping is not only
indispensable for animal training, it is also an e↵ective tool for training reinforcement
learning agents. There is also a connection to the idea of an animal’s motivational state,
which inﬂuences what an animal will approach or avoid and what events are rewarding
or punishing for the animal.
The reinforcement learning algorithms presented in this book include two basic mecha-

nisms for addressing the problem of delayed reinforcement: eligibility traces and value
functions learned via TD algorithms. Both mechanisms have antecedents in theories of
animal learning. Eligibility traces are similar to stimulus traces of early theories, and
value functions correspond to the role of secondary reinforcement in providing nearly
immediate evaluative feedback.
The next correspondence the chapter addressed is that between reinforcement learning’s
environment models and what psychologists call cognitive maps . Experiments conducted

in the mid 20th century purported to demonstrate the ability of animals to learn cognitive
maps as alternatives to, or as additions to, state–action associations, and later use them
to guide behavior, especially when the environment changes unexpectedly. Environment
models in reinforcement learning are like cognitive maps in that they can be learned by
supervised learning methods without relying on reward signals, and then they can be

370 Chapter 14: Psychology
used later to plan behavior.
Reinforcement learning’s distinction between model-free andmodel-based algorithms
corresponds to the distinction in psychology between habitual andgoal-directed behavior.
Model-free algorithms make decisions by accessing information that has been stored in a
policy or an action-value function, whereas model-based methods select actions as the
result of planning ahead using a model of the agent’s environment. Outcome-devaluation
experiments provide information about whether an animal’s behavior is habitual or under
goal-directed control. Reinforcement learning theory has helped clarify thinking about
these issues.
Animal learning clearly informs reinforcement learning, but as a type of machine
learning, reinforcement learning is directed toward designing and understanding e↵ective
learning algorithms, not toward replicating or explaining details of animal behavior.
We focused on aspects of animal learning that relate in clear ways to methods for

solving prediction and control problems, highlighting the fruitful two-way ﬂow of ideas
between reinforcement learning and psychology without venturing deeply into many of the
behavioral details and controversies that have occupied the attention of animal learning
researchers. Future development of reinforcement learning theory and algorithms will
likely exploit links to many other features of animal learning as the computational utility
of these features becomes better appreciated. We expect that a ﬂow of ideas between

reinforcement learning and psychology will continue to bear fruit for both disciplines.
Many connections between reinforcement learning and areas of psychology and other
behavioral sciences are beyond the scope of this chapter. We largely omit discussing
links to the psychology of decision making, which focuses on how actions are selected,
or how decisions are made, after learning has taken place. We also do not discuss links
to ecological and evolutionary aspects of behavior studied by ethologists and behavioral

ecologists: how animals relate to one another and to their physical surroundings, and how
their behavior contributes to evolutionary ﬁtness. Optimization, MDPs, and dynamic
programming ﬁgure prominently in these ﬁelds, and our emphasis on agent interaction
with dynamic environments connects to the study of agent behavior in complex “ecologies.”
Multi-agent reinforcement learning, omitted in this book, has connections to social aspects
of behavior. Despite the lack of treatment here, reinforcement learning should by no means

be interpreted as dismissing evolutionary perspectives. Nothing about reinforcement
learning implies a tabula rasa view of learning and behavior. Indeed, experience with
engineering applications has highlighted the importance of building into reinforcement
learning systems knowledge that is analogous to what evolution provides to animals.

Bibliographical and Historical Remarks 371
Ludvig, Bellemare, and Pearson (2011) and Shah (2012) review reinforcement learning in
the contexts of psychology and neuroscience. These publications are useful companions
to this chapter and the following chapter on reinforcement learning and neuroscience.
14.1 Dayan, Niv, Seymour, and Daw (2006) focused on interactions between clas-
sical and instrumental conditioning, particularly situations where classically-
conditioned and instrumental responses are in conﬂict. They proposed a Q-
learning framework for modeling aspects of this interaction. Modayil and Sutton
(2014) used a mobile robot to demonstrate the e↵ectiveness of a control method
combining a ﬁxed response with online prediction learning. Calling this Pavlo-
vian control , they emphasized that it di↵ers from the usual control methods of
reinforcement learning, being based on predictively executing ﬁxed responses

and not on reward maximization. The electro-mechanical machine of Ross (1933)
and especially the learning version of Walter’s turtle (Walter, 1951) were very
early illustrations of Pavlovian control.
14.2.1 Kamin (1968) ﬁrst reported blocking, now commonly known as Kamin blocking,
in classical conditioning. Moore and Schmajuk (2008) provide an excellent
summary of the blocking phenomenon, the research it stimulated, and its lasting
inﬂuence on animal learning theory. Gibbs, Cool, Land, Kehoe, and Gormezano

(1991) describe second-order conditioning of the rabbit’s nictitating membrane
response and its relationship to conditioning with serial-compound stimuli. Finch
and Culler (1934) reported obtaining ﬁfth-order conditioning of a dog’s foreleg
withdrawal “when the motivation of the animal is maintained through the various
orders.”
14.2.2 The idea built into the Rescorla–Wagner model that learning occurs when animals
are surprised is derived from Kamin (1969). Models of classical conditioning

orders.”
other than Rescorla and Wagner’s include the models of Klopf (1988), Grossberg
(1975), Mackintosh (1975), Moore and Stickney (1980), Pearce and Hall (1980),
and Courville, Daw, and Touretzky (2006). Schmajuk (2008) review models of
classical conditioning. Wagner (2008) provides a modern psychological perspective
on the Rescorla-Wagner model and similar elemental theories of learning.
14.2.3 An early version of the TD model of classical conditioning appeared in Sutton and

Barto (1981a), which also included the early model’s prediction that temporal
primacy overrides blocking, later shown by Kehoe, Schreurs, and Graham (1987)
to occur in the rabbit nictitating membrane preparation. Sutton and Barto
(1981a) contains the earliest recognition of the near identity between the Rescorla–
Wagner model and the Least-Mean-Square (LMS), or Widrow-Ho↵, learning
rule (Widrow and Ho↵, 1960). This early model was revised following Sutton’s
development of the TD algorithm (Sutton, 1984, 1988) and was ﬁrst presented as

the TD model in Sutton and Barto (1987) and more completely in Sutton and
Barto (1990), upon which this section is largely based. Additional exploration

372 Chapter 14: Psychology
of the TD model and its possible neural implementation was conducted by
Moore and colleagues (Moore, Desmond, Berthier, Blazis, Sutton, and Barto,
1986; Moore and Blazis, 1989; Moore, Choi, and Brunzell, 1998; Moore, Marks,
Castagna, and Polewan, 2001). Klopf’s (1988) drive-reinforcement theory of
classical conditioning extends the TD model to address additional experimental
details, such as the S-shape of acquisition curves. In some of these publications
TD is taken to mean Time Derivative instead of Temporal Di↵erence.
14.2.4 Ludvig, Sutton, and Kehoe (2012) evaluated the performance of the TD model
in previously unexplored tasks involving classical conditioning and examined
the inﬂuence of various stimulus representations, including the microstimulus
representation that they introduced earlier (Ludvig, Sutton, and Kehoe, 2008).
Earlier investigations of the inﬂuence of various stimulus representations and their

possible neural implementations on response timing and topography in the context
of the TD model are those of Moore and colleagues cited above. Although not in
the context of the TD model, representations like the microstimulus representation
of Ludvig et al. (2012) have been proposed and studied by Grossberg and
Schmajuk (1989), Brown, Bullock, and Grossberg (1999), Buhusi and Schmajuk
(1999), and Machado (1997). The ﬁgures on pages 353–355 are adapted from
Sutton and Barto (1990).

14.3 Section 1.7 includes comments on the history of trial-and-error learning and
the Law of E↵ect. The idea that Thorndike’s cats might have been exploring
according to an instinctual context-speciﬁc ordering over actions rather than by
just selecting from a set of instinctual impulses was suggested by Peter Dayan
(personal communication). Selfridge, Sutton, and Barto (1985) illustrated the
e↵ectiveness of shaping in a pole-balancing reinforcement learning task. Other

examples of shaping in reinforcement learning are Gullapalli and Barto (1992),
Mahadevan and Connell (1992), Mataric (1994), Dorigo and Colombette (1994),
Saksida, Raymond, and Touretzky (1997), and Randløv and Alstrøm (1998). Ng
(2003) and Ng, Harada, and Russell (1999) used the term shaping in a sense
somewhat di↵erent from Skinner’s, focussing on the problem of how to alter the
reward signal without altering the set of optimal policies.
Dickinson and Balleine (2002) discuss the complexity of the interaction between

learning and motivation. Wise (2004) provides an overview of reinforcement
learning and its relation to motivation. Daw and Shohamy (2008) link motivation
and learning to aspects of reinforcement learning theory. See also McClure,
Daw, and Montague (2003), Niv, Joel, and Dayan (2006), Rangel, Camerer, and
Montague (2008), and Dayan and Berridge (2014). McClure et al. (2003), Niv,
Daw, and Dayan (2006), and Niv, Daw, Joel, and Dayan (2007) present theories
of behavioral vigor related to the reinforcement learning framework.

14.4 Spence, Hull’s student and collaborator at Yale, elaborated the role of higher-
order reinforcement in addressing the problem of delayed reinforcement (Spence,
1947). Learning over very long delays, as in taste-aversion conditioning with

Bibliographical and Historical Remarks 373
delays up to several hours, led to interference theories as alternatives to decaying-
trace theories (e.g., Revusky and Garcia, 1970; Boakes and Costa, 2014). Other
views of learning under delayed reinforcement invoke roles for awareness and
working memory (e.g., Clark and Squire, 1998; Seo, Barraclough, and Lee, 2007).
14.5 Thistlethwaite (1951) provides an extensive review of latent learning experiments
up to the time of its publication. Ljung (1998) provides an overview of model
learning, or system identiﬁcation, techniques in engineering. Gopnik, Glymour,
Sobel, Schulz, Kushnir, and Danks (2004) present a Bayesian theory about how
children learn models.
14.6 Connections between habitual and goal-directed behavior and model-free and
model-based reinforcement learning were ﬁrst proposed by Daw, Niv, and Dayan
(2005). The hypothetical maze task used to explain habitual and goal-directed
behavioral control is based on the explanation of Niv, Joel, and Dayan (2006).

Dolan and Dayan (2013) review four generations of experimental research related
to this issue and discuss how it can move forward on the basis of reinforcement
learning’s model-free/model-based distinction. Dickinson (1980, 1985) and Dick-
inson and Balleine (2002) discuss experimental evidence related to this distinction.
Donahoe and Burgos (2000) alternatively argue that model-free processes can
account for the results of outcome-devaluation experiments. Dayan and Berridge

(2014) argue that classical conditioning involves model-based processes. Rangel,
Camerer, and Montague (2008) review many of the outstanding issues involving
habitual, goal-directed, and Pavlovian modes of control.
Comments on Terminology — The traditional meaning of reinforcement in psychol-
ogy is the strengthening of a pattern of behavior (by increasing either its intensity or
frequency) as a result of an animal receiving a stimulus (or experiencing the omission

of a stimulus) in an appropriate temporal relationship with another stimulus or with a
response. Reinforcement produces changes that remain in future behavior. Sometimes in
psychology reinforcement refers to the process of producing lasting changes in behavior,
whether the changes strengthen or weaken a behavior pattern (Mackintosh, 1983). Letting
reinforcement refer to weakening in addition to strengthening is at odds with the everyday
meaning of reinforce, and its traditional use in psychology, but it is a useful extension

that we have adopted here. In either case, a stimulus considered to be the cause of the
behavioral change is called a reinforcer .
Psychologists do not generally use the speciﬁc phrase reinforcement learning as we
do. Animal learning pioneers probably regarded reinforcement and learning as being
synonymous, so it would be redundant to use both words. Our use of the phrase follows
its use in computational and engineering research, inﬂuenced mostly by Minsky (1961).
But the phrase is lately gaining currency in psychology and neuroscience, likely because

strong parallels have surfaced between reinforcement learning algorithms and animal
learning—parallels described in this chapter and the next.
According to common usage, a reward is an object or event that an animal will
approach and work for. A reward may be given to an animal in recognition of its ‘good’

374 Chapter 14: Psychology
behavior, or given in order to make the animal’s behavior ‘better.’ Similarly, a penalty is
an object or event that the animal usually avoids and that is given as a consequence of
‘bad’ behavior, usually in order to change that behavior. Primary reward is reward due
to machinery built into an animal’s nervous system by evolution to improve its chances
of survival and reproduction, for example, reward produced by the taste of nourishing
food, sexual contact, successful escape, and many other stimuli and events that predicted
reproductive success over the animal’s ancestral history. As explained in Section 14.2.1,
higher-order reward is reward delivered by stimuli that predict primary reward, either
directly or indirectly by predicting other stimuli that predict primary reward. Reward is
secondary if its rewarding quality is the result of directly predicting primary reward.
In this book we call Rtthe ‘reward signal at time t’ or sometimes just the ‘reward

at time t,’ but we do not think of it as an object or event in the agent’s environment.
Because Rtis a number—not an object or an event—it is more like a reward signal in
neuroscience, which is a signal internal to the brain, like the activity of neurons, that
inﬂuences decision making and learning. This signal might be triggered when the animal
perceives an attractive (or an aversive) object, but it can also be triggered by things that
do not physically exist in the animal’s external environment, such as memories, ideas, or

hallucinations. Because our Rtcan be positive, negative, or zero, it might be better to
call a negative Rta penalty, and an Rtequal to zero a neutral signal, but for simplicity
we generally avoid these terms.
In reinforcement learning, the process that generates all the Rts deﬁnes the problem
the agent is trying to solve. The agent’s objective is to keep the magnitude of Rtas large
as possible over time. In this respect, Rtis like primary reward for an animal if we think
of the problem the animal faces as the problem of obtaining as much primary reward as

possible over its lifetime (and thereby, through the prospective “wisdom” of evolution,
improve its chances of solving its real problem, which is to pass its genes on to future
generations). However, as we suggest in Chapter 15, it is unlikely that there is a single
“master” reward signal like Rtin an animal’s brain.
Not all reinforcers are rewards or penalties. Sometimes reinforcement is not the result
of an animal receiving a stimulus that evaluates its behavior by labeling the behavior

good or bad. A behavior pattern can be reinforced by a stimulus that arrives to an animal
no matter how the animal behaved. As described in Section 14.1, whether the delivery of
reinforcer depends, or does not depend, on preceding behavior is the deﬁning di↵erence
between instrumental, or operant, conditioning experiments and classical, or Pavlovian,
conditioning experiments. Reinforcement is at work in both types of experiments, but
only in the former is it feedback that evaluates past behavior. (Though it has often been

pointed out that even when the reinforcing US in a classical conditioning experiment is
not contingent on the subject’s preceding behavior, its reinforcing value can be inﬂuenced
by this behavior, an example being that a closed eye makes an air pu↵ to the eye less
aversive.)
The distinction between reward signals and reinforcement signals is a crucial point
when we discuss neural correlates of these signals in the next chapter. Like a reward signal,
for us, the reinforcement signal at any speciﬁc time is a positive or negative number, or

zero. A reinforcement signal is the major factor directing changes a learning algorithm

Bibliographical and Historical Remarks 375
makes in an agent’s policy, value estimates, or environment models. The deﬁnition that
makes the most sense to us is that a reinforcement signal at any time is a number that
multiplies (possibly along with some constants) a vector to determine parameter updates
in some learning algorithm.
For some algorithms, the reward signal alone is the critical multiplier in the parameter-
update equation. For these algorithms the reinforcement signal is the same as the
reward signal. But for most of the algorithms we discuss in this book, reinforcement
signals include terms in addition to the reward signal, an example being a TD error
t=Rt+1+V(St+1)V(St), which is the reinforcement signal for TD state-value
learning (and analogous TD errors for action-value learning). In this reinforcement signal,
Rt+1is the primary reinforcement contribution, and the temporal di↵erence in predicted
values, V(St+1)V(St) (or an analogous temporal di↵erence for action values), is

B. F. Skinner and his followers. For Skinner, positive reinforcement occurs when the
consequences of an animal’s behavior increase the frequency of that behavior; punishment
occurs when the behavior’s consequences decrease that behavior’s frequency. Negative
reinforcement occurs when behavior leads to the removal of an aversive stimulus (that is,
a stimulus the animal does not like), thereby increasing the frequency of that behavior.
Negative punishment, on the other hand, occurs when behavior leads to the removal of an
appetitive stimulus (that is, a stimulus the animal likes), thereby decreasing the frequency
of that behavior. We ﬁnd no critical need for these distinctions because our approach
is more abstract than this, with both reward and reinforcement signals allowed to take
on both positive and negative values. (But note especially that when our reinforcement
signal is negative, it is not the same as Skinner’s negative reinforcement.)

On the other hand, it has often been pointed out that using a single number as a
reward or a penalty signal, depending only on its sign, is at odds with the fact that
animals’ appetitive and aversive systems have qualitatively di↵erent properties and
involve di↵erent brain mechanisms. This points to a direction in which the reinforcement
learning framework might be developed in the future to exploit computational advantages
of separate appetitive and aversive systems, but for now we are passing over these
possibilities.

Another discrepancy in terminology is how we use the word action . To many cognitive
scientists, an action is purposeful in the sense of being the result of an animal’s knowledge
about the relationship between the behavior in question and the consequences of that
behavior. An action is goal-directed and the result of a decision, in contrast to a response,
which is triggered by a stimulus; the result of a reﬂex or a habit. We use the word
action without di↵erentiating among what others call actions, decisions, and responses.

These are important distinctions, but for us they are encompassed by di↵erences between

376 Chapter 14: Psychology
model-free and model-based reinforcement learning algorithms, which we discussed above
in relation to habitual and goal-directed behavior in Section 14.6. Dickinson (1985)
discusses the distinction between responses and actions.
A term used a lot in this book is control . What we mean by control is entirely di↵erent
from what it means to animal learning psychologists. By control we mean that an agent
inﬂuences its environment to bring about states or events that the agent prefers: the agent
exerts control over its environment. This is the sense of control used by control engineers.
In psychology, on the other hand, control typically means that an animal’s behavior is
inﬂuenced by—is controlled by—the stimuli the animal receives (stimulus control) or
the reinforcement schedule it experiences. Here the environment is controlling the agent.
Control in this sense is the basis of behavior modiﬁcation therapy. Of course, both of

these directions of control are at play when an agent interacts with its environment,
but our focus is on the agent as controller; not the environment as controller. A view
equivalent to ours, and perhaps more illuminating, is that the agent is actually controlling
the input it receives from its environment (Powers, 1973). This is notwhat psychologists
mean by stimulus control.
Sometimes reinforcement learning is understood to refer solely to learning policies
directly from rewards (and penalties) without the involvement of value functions or

environment models. This is what psychologists call stimulus-response, or S-R, learning.
But for us, along with most of today’s psychologists, reinforcement learning is much
broader than this, including in addition to S-R learning, methods involving value functions,
environment models, planning, and other processes that are commonly thought to belong
to the more cognitive side of mental functioning.

Chapter 15
Neuroscience is the multidisciplinary study of nervous systems: how they regulate bodily
functions; control behavior; change over time as a result of development, learning, and
aging; and how cellular and molecular mechanisms make these functions possible. One
of the most exciting aspects of reinforcement learning is the mounting evidence from
neuroscience that the nervous systems of humans and many other animals implement
algorithms that correspond in striking ways to reinforcement learning algorithms. The
main objective of this chapter is to explain these parallels and what they suggest about
the neural basis of reward-related learning in animals.
The most remarkable point of contact between reinforcement learning and neuroscience
involves dopamine, a chemical deeply involved in reward processing in the brains of mam-
mals. Dopamine appears to convey temporal-di↵erence (TD) errors to brain structures

where learning and decision making take place. This parallel is expressed by the reward
prediction error hypothesis of dopamine neuron activity , a hypothesis that resulted from
the convergence of computational reinforcement learning and results of neuroscience
experiments. In this chapter we discuss this hypothesis, the neuroscience ﬁndings that
led to it, and why it is a signiﬁcant contribution to understanding brain reward systems.
We also discuss parallels between reinforcement learning and neuroscience that are less

striking than this dopamine/TD-error parallel but that provide useful conceptual tools
for thinking about reward-based learning in animals. Other elements of reinforcement
learning have the potential to impact the study of nervous systems, but their connections
to neuroscience are still relatively undeveloped. We discuss several of these evolving
connections that we think will grow in importance over time.
As we outlined in the history section of this book’s introductory chapter (Section 1.7),
many aspects of reinforcement learning were inﬂuenced by neuroscience. A second

objective of this chapter is to acquaint readers with ideas about brain function that have
contributed to our approach to reinforcement learning. Some elements of reinforcement
learning are easier to understand when seen in light of theories of brain function. This
is particularly true for the idea of the eligibility trace, one of the basic mechanisms
of reinforcement learning, that originated as a conjectured property of synapses, the
structures by which nerve cells—neurons—communicate with one another.
377

378 Chapter 15: Neuroscience
In this chapter we do not delve very deeply into the enormous complexity of the neural
systems underlying reward-based learning in animals: this chapter is too short, and we are
not neuroscientists. We do not try to describe—or even to name—the very many brain
structures and pathways, or any of the molecular mechanisms, believed to be involved in
these processes. We also do not do justice to hypotheses and models that are alternatives
to those that align so well with reinforcement learning. It should not be surprising that
there are di↵ering views among experts in the ﬁeld. We can only provide a glimpse into
this fascinating and developing story. We hope, though, that this chapter convinces
you that a very fruitful channel has emerged connecting reinforcement learning and its
theoretical underpinnings to the neuroscience of reward-based learning in animals.
Many excellent publications cover links between reinforcement learning and neuro-

science, some of which we cite in this chapter’s ﬁnal section. Our treatment di↵ers from
most of these because we assume familiarity with reinforcement learning as presented
in the earlier chapters of this book, but we do not assume knowledge of neuroscience.
We begin with a brief introduction to the neuroscience concepts needed for a basic
understanding of what is to follow.
15.1 Neuroscience Basics
Some basic information about nervous systems is helpful for following what we cover in
this chapter. Terms that we refer to later are italicized. Skipping this section will not be

a problem if you already have an elementary knowledge of neuroscience.
Neurons , the main components of nervous systems, are cells specialized for processing
and transmitting information using electrical and chemical signals. They come in many
forms, but a neuron typically has a cell body, dendrites , and a single axon .D e n d r i t e s
are structures that branch from the cell body to receive input from other neurons (or to
also receive external signals in the case of sensory neurons). A neuron’s axon is a ﬁber

that carries the neuron’s output to other neurons (or to muscles or glands). A neuron’s
output consists of sequences of electrical pulses called action potentials that travel along
the axon. Action potentials are also called spikes , and a neuron is said to ﬁrewhen it
generates a spike. In models of neural networks it is common to use real numbers to
represent a neuron’s ﬁring rate , the average number of spikes per some unit of time.
A neuron’s axon can branch widely so that the neuron’s action potentials reach

many targets. The branching structure of a neuron’s axon is called the neuron’s axonal
arbor . Because the conduction of an action potential is an active process, not unlike the
burning of a fuse, when an action potential reaches an axonal branch point it “lights
up” action potentials on all of the outgoing branches (although propagation to a branch
can sometimes fail). As a result, the activity of a neuron with a large axonal arbor can
inﬂuence many target sites.

15.1. Neuroscience Basics 379
Asynapse is a structure generally at the termination of an axon branch that mediates
the communication of one neuron to another. A synapse transmits information from
thepresynaptic neuron’s axon to a dendrite or cell body of the postsynaptic neuron.
With a few exceptions, synapses release a chemical neurotransmitter upon the arrival
of an action potential from the presynaptic neuron. (The exceptions are cases of direct
electric coupling between neurons, but these will not concern us here.) Neurotransmitter
molecules released from the presynaptic side of the synapse di↵use across the synaptic
cleft, the very small space between the presynaptic ending and the postsynaptic neuron,
and then bind to receptors on the surface of the postsynaptic neuron to excite or inhibit
its spike-generating activity, or to modulate its behavior in other ways. A particular
neurotransmitter may bind to several di↵erent types of receptors, with each producing a

di↵erent e↵ect on the postsynaptic neuron. For example, there are at least ﬁve di↵erent
receptor types by which the neurotransmitter dopamine can a↵ect a postsynaptic neuron.
Many di↵erent chemicals have been identiﬁed as neurotransmitters in animal nervous
systems.
A neuron’s background activity is its level of activity, usually its ﬁring rate, when the
neuron does not appear to be driven by synaptic input related to the task of interest
to the experimenter, for example, when the neuron’s activity is not correlated with a

systems.
stimulus delivered to a subject as part of an experiment. Background activity can be
irregular due to input from the wider network, or due to noise within the neuron or its
synapses. Sometimes background activity is the result of dynamic processes intrinsic to
the neuron. A neuron’s phasic activity, in contrast to its background activity, consists of
bursts of spiking activity usually caused by synaptic input. Activity that varies slowly
and often in a graded manner, whether as background activity or not, is called a neuron’s
tonic activity.

The strength or e↵ectiveness by which the neurotransmitter released at a synapse
inﬂuences the postsynaptic neuron is the synapse’s ecacy . One way a nervous system
can change through experience is through changes in synaptic ecacies as a result of
combinations of the activities of the presynaptic and postsynaptic neurons, and sometimes
by the presence of a neuromodulator , which is a neurotransmitter having e↵ects other
than, or in addition to, direct fast excitation or inhibition.

Brains contain several di↵erent neuromodulation systems consisting of clusters of
neurons with widely branching axonal arbors, with each system using a di↵erent neuro-
transmitter. Neuromodulation can alter the function of neural circuits, mediate motivation,
arousal, attention, memory, mood, emotion, sleep, and body temperature. Important
here is that a neuromodulatory system can distribute something like a scalar signal, such

as a reinforcement signal, to alter the operation of synapses in widely distributed sites
critical for learning.
The ability of synaptic ecacies to change is called synaptic plasticity . It is one of the
primary mechanisms responsible for learning. The parameters, or weights, adjusted by
learning algorithms correspond to synaptic ecacies. As we detail below, modulation of
synaptic plasticity via the neuromodulator dopamine is a plausible mechanism for how

the brain might implement learning algorithms like many of those described in this book.

380 Chapter 15: Neuroscience
15.2 Reward Signals, Reinforcement Signals, Values,
and Prediction Errors
Links between neuroscience and computational reinforcement learning begin as parallels
between signals in the brain and signals playing prominent roles in reinforcement learning
theory and algorithms. In Chapter 3 we said that any problem of learning goal-directed
behavior can be reduced to the three signals representing actions, states, and rewards.
However, to explain links that have been made between neuroscience and reinforcement
learning, we have to be less abstract than this and consider other reinforcement learning
signals that correspond, in certain ways, to signals in the brain. In addition to reward
signals, these include reinforcement signals (which we argue are di↵erent from reward
signals), value signals, and signals conveying prediction errors. When we label a signal by
its function in this way, we are doing it in the context of reinforcement learning theory

in which the signal corresponds to a term in an equation or an algorithm. On the other
hand, when we refer to a signal in the brain, we mean a physiological event such as
a burst of action potentials or the secretion of a neurotransmitter. Labeling a neural
signal by its function, for example calling the phasic activity of a dopamine neuron a
reinforcement signal, means that the neural signal behaves like, and is conjectured to
function like, the corresponding theoretical signal.
Uncovering evidence for these correspondences involves many challenges. Neural

activity related to reward processing can be found in nearly every part of the brain,
and it is dicult to interpret results unambiguously because representations of di↵erent
reward-related signals tend to be highly correlated with one another. Experiments need to
be carefully designed to allow one type of reward-related signal to be distinguished with
any degree of certainty from others—or from an abundance of other signals not related to
reward processing. Despite these diculties, many experiments have been conducted with

the aim of reconciling aspects of reinforcement learning theory and algorithms with neural
signals, and some compelling links have been established. To prepare for examining these
links, in the rest of this section we remind the reader of what various reward-related
signals mean according to reinforcement learning theory.
In our Comments on Terminology at the end of the previous chapter, we said that
Rtis like a reward signal in an animal’s brain and not an object or event in the
animal’s environment. In reinforcement learning, the reward signal (along with an agent’s

environment) deﬁnes the problem a reinforcement learning agent is trying to solve. In
this respect, Rtis like a signal in an animal’s brain that distributes primary reward to
sites throughout the brain. But it is unlikely that a unitary master reward signal like Rt
exists in an animal’s brain. It is best to think of Rtas an abstraction summarizing the
overall e↵ect of a multitude of neural signals generated by many systems in the brain
that assess the rewarding or punishing qualities of sensations and states.

15.3. The Reward Prediction Error Hypothesis 381
reinforcement signal for some algorithms could be just the reward signal, but for most
of the algorithms we consider the reinforcement signal is the reward signal adjusted by
other information, such as the value estimates in TD errors.
Estimates of state values or of action values, that is, VorQ, specify what is good or
bad for the agent over the long run. They are predictions of the total reward an agent can
expect to accumulate over the future. Agents make good decisions by selecting actions
leading to states with the largest estimated state values, or by selecting actions with the
largest estimated action values.
Prediction errors measure discrepancies between expected and actual signals or sensa-
tions. Reward prediction errors (RPEs) speciﬁcally measure discrepancies between the
expected and the received reward signal, being positive when the reward signal is greater
than expected, and negative otherwise. TD errors like (6.5) are special kinds of RPEs

that signal discrepancies between current and earlier expectations of reward over the
long-term. When neuroscientists refer to RPEs they generally (though not always) mean
TD RPEs, which we simply call TD errors throughout this chapter. Also in this chapter,
a TD error is generally one that does not depend on actions, as opposed to TD errors
used in learning action-values by algorithms like Sarsa and Q-learning. This is because
the most well-known links to neuroscience are stated in terms of action-free TD errors,

but we do not mean to rule out possible similar links involving action-dependent TD
errors. (TD errors for predicting signals other than rewards are useful too, but that case
will not concern us here. See, for example, Modayil, White, and Sutton, 2014.)
One can ask many questions about links between neuroscience data and these theoretically-
deﬁned signals. Is an observed signal more like a reward signal, a value signal, a prediction
error, a reinforcement signal, or something altogether di↵erent? And if it is an error

signal, is it an RPE, a TD error, or a simpler error like the Rescorla–Wagner error (14.3)?
And if it is a TD error, does it depend on actions like the TD error of Q-learning or
Sarsa? As indicated above, probing the brain to answer questions like these is extremely
dicult. But experimental evidence suggests that one neurotransmitter, speciﬁcally
the neurotransmitter dopamine, signals RPEs, and further, that the phasic activity of
dopamine-producing neurons in fact conveys TD errors (see Section 15.1 for a deﬁnition of

phasic activity). This evidence led to the reward prediction error hypothesis of dopamine
neuron activity ,w h i c hw ed e s c r i b en e x t .
15.3 The Reward Prediction Error Hypothesis
The reward prediction error hypothesis of dopamine neuron activity proposes that one
of the functions of the phasic activity of dopamine-producing neurons in mammals is to
deliver an error between an old and a new estimate of expected future reward to target
areas throughout the brain. This hypothesis (though not in these exact words) was

382 Chapter 15: Neuroscience
activity of dopamine neurons in mammals. The experiments that led to this hypothesis
were performed in the 1980s and early 1990s in the laboratory of neuroscientist Wolfram
Schultz. Section 15.4 describes these inﬂuential experiments, Section 15.6 explains how the
results of these experiments align with TD errors, and the Bibliographical and Historical
Remarks section at the end of this chapter includes a guide to the literature surrounding
the development of this inﬂuential hypothesis.
Montague et al. (1996) compared the TD errors of the TD model of classical conditioning
with the phasic activity of dopamine-producing neurons during classical conditioning
experiments. Recall from Section 14.2 that the TD model of classical conditioning is
basically the semi-gradient-descent TD( ) algorithm with linear function approximation.
Montague et al. made several assumptions to set up this comparison. First, because a

TD error can be negative but neurons cannot have a negative ﬁring rate, they assumed
that the quantity corresponding to dopamine neuron activity is t1+bt,w h e r e btis the
background ﬁring rate of the neuron. A negative TD error corresponds to a drop in a
dopamine neuron’s ﬁring rate below its background rate.2
A second assumption was needed about the states visited in each classical conditioning
trial and how they are represented as inputs to the learning algorithm. This is the same

issue we discussed in Section 14.2.4 for the TD model. Montague et al. chose a complete
serial compound (CSC) representation as shown in the left column of Figure 14.1, but
where the sequence of short-duration internal signals continues until the onset of the US,
which here is the arrival of a non-zero reward signal. This representation allows the TD
error to mimic the fact that dopamine neuron activity not only predicts a future reward,
but that it is also sensitive to when after a predictive cue that reward is expected to

arrive. There has to be some way to keep track of the time between sensory cues and
the arrival of reward. If a stimulus initiates a sequence of internal signals that continues
after the stimulus ends, and if there is a di↵erent signal for each time step following the
stimulus, then each time step after the stimulus is represented by a distinct state. Thus,
the TD error, being state-dependent, can be sensitive to the timing of events within a
trial.
In simulated trials with these assumptions about background ﬁring rate and input

trial.
representation, TD errors of the TD model are remarkably similar to dopamine neuron
phasic activity. Previewing our description of details about these similarities in Section 15.4
below, the TD errors parallel the following features of dopamine neuron activity: 1) the
phasic response of a dopamine neuron only occurs when a rewarding event is unpredicted;
2) early in learning, neutral cues that precede a reward do not cause substantial phasic
dopamine responses, but with continued learning these cues gain predictive value and

come to elicit phasic dopamine responses; 3) if an even earlier cue reliably precedes a
cue that has already acquired predictive value, the phasic dopamine response shifts to
the earlier cue, ceasing for the later cue; and 4) if after learning, the predicted rewarding
event is omitted, a dopamine neuron’s response decreases below its baseline level shortly
after the expected time of the rewarding event.
Although not every dopamine neuron monitored in the experiments of Schultz and

15.4. Dopamine 383
colleagues behaved in all of these ways, the striking correspondence between the ac-
tivities of most of the monitored neurons and TD errors lends strong support to the
reward prediction error hypothesis. There are situations, however, in which predictions
based on the hypothesis do not match what is observed in experiments. The choice
of input representation is critical to how closely TD errors match some of the details
of dopamine neuron activity, particularly details about the timing of dopamine neuron
responses. Di↵erent ideas, some of which we discuss below, have been proposed about
input representations and other features of TD learning to make the TD errors ﬁt the data
better, though the main parallels appear with the CSC representation that Montague et
al. used. Overall, the reward prediction error hypothesis has received wide acceptance
among neuroscientists studying reward-based learning, and it has proven to be remarkably

resilient in the face of accumulating results from neuroscience experiments.
To prepare for our description of the neuroscience experiments supporting the reward
prediction error hypothesis, and to provide some context so that the signiﬁcance of the
hypothesis can be appreciated, we next present some of what is known about dopamine,
the brain structures it inﬂuences, and how it is involved in reward-based learning.
15.4 Dopamine
Dopamine is produced as a neurotransmitter by neurons whose cell bodies lie mainly

in two clusters of neurons in the midbrain of mammals: the substantia nigra pars
compacta (SNpc) and the ventral tegmental area (VTA). Dopamine plays essential roles
in many processes in the mammalian brain. Prominent among these are motivation,
learning, action-selection, most forms of addiction, and the disorders schizophrenia and
Parkinson’s disease. Dopamine is called a neuromodulator because it performs many
functions other than direct fast excitation or inhibition of targeted neurons. Although

much remains unknown about dopamine’s functions and details of its cellular e↵ects, it is
clear that it is fundamental to reward processing in the mammalian brain. Dopamine
is not the only neuromodulator involved in reward processing, and its role in aversive
situations—punishment—remains controversial. Dopamine also can function di↵erently in
non-mammals. But no one doubts that dopamine is essential for reward-related processes
in mammals, including humans.

An early, traditional view is that dopamine neurons broadcast a reward signal to
multiple brain regions implicated in learning and motivation. This view followed from a
famous 1954 paper by James Olds and Peter Milner that described the e↵ects of electrical
stimulation on certain areas of a rat’s brain. They found that electrical stimulation to
particular regions acted as a very powerful reward in controlling the rat’s behavior: “...the
control exercised over the animal’s behavior by means of this reward is extreme, possibly

exceeding that exercised by any other reward previously used in animal experimentation”
(Olds and Milner, 1954). Later research revealed that the sites at which stimulation
was most e↵ective in producing this rewarding e↵ect excited dopamine pathways, either
directly or indirectly, that ordinarily are excited by natural rewarding stimuli. E↵ects
similar to these were also observed with human subjects. These observations strongly
suggested that dopamine neuron activity signals reward.

about the reinforcing e↵ect of electrical stimulation in an instrumental conditioning task.
Electrical stimulation not only energized the rats’ behavior—through dopamine’s e↵ect on
motivation—it also led to the rats quickly learning to stimulate themselves by pressing a
lever, which they would do frequently for long periods of time. The activity of dopamine
neurons triggered by electrical stimulation reinforced the rats’ lever pressing.
More recent experiments using optogenetic methods clinch the role of phasic responses
of dopamine neurons as reinforcement signals. These methods allow neuroscientists to
precisely control the activity of selected neuron types at a millisecond timescale in awake
behaving animals. Optogenetic methods introduce light-sensitive proteins into selected
neuron types so that these neurons can be activated or silenced by means of ﬂashes of
laser light. The ﬁrst experiment using optogenetic methods to study dopamine neurons

showed that optogenetic stimulation producing phasic activation of dopamine neurons
in mice was enough to condition the mice to prefer the side of a chamber where they
received this stimulation as compared to the chamber’s other side where they received
no, or lower-frequency, stimulation (Tsai et al. 2009). In another example, Steinberg
et al. (2013) used optogenetic activation of dopamine neurons to create artiﬁcial bursts
of dopamine neuron activity in rats at the times when rewarding stimuli were expected

but omitted—times when dopamine neuron activity normally pauses. With these pauses
replaced by artiﬁcial bursts, responding was sustained when it would ordinarily decrease
due to lack of reinforcement (in extinction trials), and learning was enabled when it would
ordinarily be blocked due to the reward being already predicted (the blocking paradigm;
Section 14.2.1).
Additional evidence for the reinforcing function of dopamine comes from optogenetic
experiments with fruit ﬂies, except in these animals dopamine’s e↵ect is the opposite of

its e↵ect in mammals: optically triggered bursts of dopamine neuron activity act just
like electric foot shock in reinforcing avoidance behavior, at least for the population

15.4. Dopamine 385
of dopamine neurons activated (Claridge-Chang et al. 2009). Although none of these
optogenetic experiments showed that phasic dopamine neuron activity is speciﬁcally
like a TD error, they convincingly demonstrated that phasic dopamine neuron activity
acts just like acts (or perhaps like minus acts in fruit ﬂies) as the reinforcement
signal in algorithms for both prediction (classical conditioning) and control (instrumental
conditioning).
Axonal arbor of a single neuron producing
dopamine as a neurotransmitter. These
axons make synaptic contacts with a huge
number of dendrites of neurons in targeted
brain areas.
Adapted from The Journal of Neuroscience ,
Matsuda, Furuta, Nakamura, Hioki, Fujiyama,
Arai, and Kaneko, volume 29, 2009, page 451.Dopamine neurons are particularly well suited
to broadcasting a reinforcement signal to many
areas of the brain. These neurons have huge
axonal arbors, each releasing dopamine at 100 to
1,000 times more synaptic sites than reached by

the axons of typical neurons. Shown to the right
is the axonal arbor of a single dopamine neuron
whose cell body is in the SNpc of a rat’s brain.
Each axon of a SNpc or VTA dopamine neuron
makes roughly 500,000 synaptic contacts on the
dendrites of neurons in targeted brain areas.
If dopamine neurons broadcast a reinforce-
ment signal like reinforcement learning’s ,t h e n
because this is a scalar signal, i.e., a single num-
ber, all dopamine neurons in both the SNpc
and VTA would be expected to activate more-

or-less identically so that they would act in near
synchrony to send the same signal to all of the
sites their axons target. Although it has been
a common belief that dopamine neurons do act
together like this, modern evidence is pointing
to the more complicated picture that di↵erent
subpopulations of dopamine neurons respond to
input di↵erently depending on the structures to
which they send their signals and the di↵erent
ways these signals act on their target structures. Dopamine has functions other than

signaling RPEs, and even for dopamine neurons that do signal RPEs, it can make sense
to send di↵erent RPEs to di↵erent structures depending on the roles these structures
play in producing reinforced behavior. This is beyond what we treat in any detail in this
book, but vector-valued RPE signals make sense from the perspective of reinforcement
learning when decisions can be decomposed into separate sub-decisions, or more generally,

as a way to address the structural version of the credit assignment problem: How do
you distribute credit for success (or blame for failure) of a decision among the many
component structures that could have been involved in producing it? We say a bit more
about this in Section 15.10 below.
The axons of most dopamine neurons make synaptic contact with neurons in the frontal
cortex and the basal ganglia, areas of the brain involved in voluntary movement, decision
making, learning, and cognitive functions such as planning. Because most ideas relating

386 Chapter 15: Neuroscience
dopamine to reinforcement learning focus on the basal ganglia, and the connections from
dopamine neurons are particularly dense there, we focus on the basal ganglia here. The
basal ganglia are a collection of neuron groups, or nuclei, lying at the base of the forebrain.
The main input structure of the basal ganglia is called the striatum. Essentially all of the
cerebral cortex, among other structures, provides input to the striatum. The activity of
cortical neurons conveys a wealth of information about sensory input, internal states, and
motor activity. The axons of cortical neurons make synaptic contacts on the dendrites of
the main input/output neurons of the striatum, called medium spiny neurons. Output
from the striatum loops back via other basal ganglia nuclei and the thalamus to frontal
areas of cortex, and to motor areas, making it possible for the striatum to inﬂuence
movement, abstract decision processes, and reward processing. Two main subdivisions

of the striatum are important for reinforcement learning: the dorsal striatum, primarily
implicated in inﬂuencing action selection, and the ventral striatum, thought to be critical
for di↵erent aspects of reward processing, including the assignment of a↵ective value to
sensations.
The dendrites of medium spiny neurons are covered with spines on whose tips the
axons of neurons in the cortex make synaptic contact. Also making synaptic contact with
these spines—in this case contacting the spine stems—are axons of dopamine neurons

(Figure 15.1). This arrangement brings together presynaptic activity of cortical neurons,
Figure 15.1: Spine of a striatal neuron showing input from both cortical and dopamine neurons.
Axons of cortical neurons inﬂuence striatal neurons via corticostriatal synapses releasing the
neurotransmitter glutamate at the tips of spines covering the dendrites of striatal neurons.
An axon of a VTA or SNpc dopamine neuron is shown passing by the spine (from the lower
right). “Dopamine varicosities” on this axon release dopamine at or near the spine stem, in an

arrangement that brings together presynaptic input from cortex, postsynaptic activity of the
striatal neuron, and dopamine, making it possible that several types of learning rules govern the
plasticity of corticostriatal synapses. Each axon of a dopamine neuron makes synaptic contact
with the stems of roughly 500,000 spines. Some of the complexity omitted from our discussion
is shown here by other neurotransmitter pathways and multiple receptor types, such as D1 an
D2 dopamine receptors by which dopamine can produce di↵erent e↵ects at spines and other

postsynaptic sites. From Journal of Neurophysiology ,W .S c h u l t z ,v o l .8 0 ,1 9 9 8 ,p a g e1 0 .

15.5. Experimental Support for the Reward Prediction Error Hypothesis 387
postsynaptic activity of medium spiny neurons, and input from dopamine neurons. What
actually occurs at these spines is complex and not completely understood. Figure 15.1
hints at the complexity by showing two types of receptors for dopamine, receptors for
glutamate—the neurotransmitter of the cortical inputs—and multiple ways that the
various signals can interact. But evidence is mounting that changes in the ecacies of
the synapses on the pathway from the cortex to the striatum, which neuroscientists call
corticostriatal synapses , depend critically on appropriately-timed dopamine signals.
15.5 Experimental Support for the Reward
Prediction Error Hypothesis
Dopamine neurons respond with bursts of activity to intense, novel, or unexpected visual
and auditory stimuli that trigger eye and body movements, but very little of their activity is
related to the movements themselves. This is surprising because degeneration of dopamine

neurons is a cause of Parkinson’s disease, whose symptoms include motor disorders,
particularly deﬁcits in self-initiated movement. Motivated by the weak relationship
between dopamine neuron activity and stimulus-triggered eye and body movements,
Romo and Schultz (1990) and Schultz and Romo (1990) took the ﬁrst steps toward the
reward prediction error hypothesis by recording the activity of dopamine neurons and
muscle activity while monkeys moved their arms.
They trained two monkeys to reach from a resting hand position into a bin containing

a bit of apple, a piece of cookie, or a raisin, when the monkey saw and heard the bin’s
door open. The monkey could then grab and bring the food to its mouth. After a monkey
became good at this, it was trained on two additional tasks. The purpose of the ﬁrst
task was to see what dopamine neurons do when movements are self-initiated. The bin
was left open but covered from above so that the monkey could not see inside but could
reach in from below. No triggering stimuli were presented, and after the monkey reached

for and ate the food morsel, the experimenter usually (though not always), silently and
unseen by the monkey, replaced food in the bin by sticking it onto a rigid wire. Here too,
the activity of the dopamine neurons Romo and Schultz monitored was not related to the
monkey’s movements, but a large percentage of these neurons produced phasic responses
whenever the monkey ﬁrst touched a food morsel. These neurons did not respond when
the monkey touched just the wire or explored the bin when no food was there. This was

good evidence that the neurons were responding to the food and not to other aspects of
the task.
The purpose of Romo and Schultz’s second task was to see what happens when
movements are triggered by stimuli. This task used a di↵erent bin with a movable cover.
The sight and sound of the bin opening triggered reaching movements to the bin. In this
case, Romo and Schultz found that after some period of training, the dopamine neurons
no longer responded to the touch of the food but instead responded to the sight and sound

the task.
of the opening cover of the food bin. The phasic responses of these neurons had shifted
from the reward itself to stimuli predicting the availability of the reward. In a followup
study, Romo and Schultz found that most of the dopamine neurons whose activity they

388 Chapter 15: Neuroscience
monitored did not respond to the sight and sound of the bin opening outside the context
of the behavioral task. These observations suggested that the dopamine neurons were
responding neither to the initiation of a movement nor to the sensory properties of the
stimuli, but were rather signaling an expectation of reward.
Schultz’s group conducted many additional studies involving both SNpc and VTA
dopamine neurons. A particular series of experiments was inﬂuential in suggesting that
the phasic responses of dopamine neurons correspond to TD errors and not to simpler
errors like those in the Rescorla–Wagner model (14.3). In the ﬁrst of these experiments
(Ljungberg, Apicella, and Schultz, 1992), monkeys were trained to depress a lever after
a light was illuminated as a ‘trigger cue’ to obtain a drop of apple juice. As Romo
and Schultz had observed earlier, many dopamine neurons initially responded to the

reward—the drop of juice (Figure 15.2, top panel). But many of these neurons lost that
reward response as training continued and developed responses instead to the illumination
of the light that predicted the reward (Figure 15.2, middle panel). With continued
training, lever pressing became faster while the number of dopamine neurons responding
to the trigger cue decreased.
Following this study, the same monkeys were trained on a new task (Schultz, Apicella,
and Ljungberg, 1993). Here the monkeys faced two levers, each with a light above it.

Illuminating one of these lights was an ‘instruction cue’ indicating which of the two levers
Figure 15.2: The response of dopamine neurons shifts from initial responses to primary reward
to earlier predictive stimuli. These are plots of the number of action potentials produced
by monitored dopamine neurons within small time intervals, averaged over all the monitored
dopamine neurons (ranging from 23 to 44 neurons for these data). Top: dopamine neurons are
activated by the unpredicted delivery of drop of apple juice. Middle: with learning, dopamine

neurons developed responses to the reward-predicting trigger cue and lost responsiveness to the
delivery of reward. Bottom: with the addition of an instruction cue preceding the trigger cue by
1 second, dopamine neurons shifted their responses from the trigger cue to the earlier instruction
cue. From Schultz et al. (1995), MIT Press.

15.5. Experimental Support for the Reward Prediction Error Hypothesis 389
Figure 15.3: The response of dopamine neurons drops
below baseline shortly after the time when an expected
reward fails to occur. Top: dopamine neurons are
activated by the unpredicted delivery of a drop of
apple juice. Middle: dopamine neurons respond to a
conditioned stimulus (CS) that predicts reward and
do not respond to the reward itself. Bottom: when
the reward predicted by the CS fails to occur, the
activity of dopamine neurons drops below baseline
shortly after the time the reward is expected to occur.
At the top of each of these panels is shown the average
number of action potentials produced by monitored
dopamine neurons within small time intervals around
the indicated times. The raster plots below show the
activity patterns of the individual dopamine neurons
that were monitored; each dot represents an action
potential. From Schultz, Dayan, and Montague, A
Neural Substrate of Prediction and Reward, Science ,

vol. 275, issue 5306, pages 1593-1598, March 14, 1997.
Reprinted with permission from AAAS.would produce a drop of apple juice.
In this task, the instruction cue pre-
ceded the trigger cue of the previ-
ous task by a ﬁxed interval of 1 sec-
ond. The monkeys learned to with-
hold reaching until seeing the trig-
ger cue, and dopamine neuron activ-
ity increased, but now the responses
of the monitored dopamine neurons
occurred almost exclusively to the
earlier instruction cue and not to
the trigger cue (Figure 15.2, bot-

tom panel). Here again the num-
ber of dopamine neurons responding
to the instruction cue was much re-
duced when the task was well learned.
During learning across these tasks,
dopamine neuron activity shifted
from initially responding to the re-
ward to responding to the earlier
predictive stimuli, ﬁrst progressing
to the trigger stimulus then to the
still earlier instruction cue. As re-
sponding moved earlier in time it
disappeared from the later stimuli.
This shifting of responses to earlier
reward predictors, while losing re-

sponses to later predictors is a hall-
mark of TD learning (see, for exam-
ple, Figure 14.2).
The task just described revealed
another property of dopamine neu-
ron activity shared with TD learn-
ing. The monkeys sometimes pressed
the wrong key, that is, the key other
than the instructed one, and conse-
quently received no reward. In these
trials, many of the dopamine neu-
rons showed a sharp decrease in their
ﬁring rates below baseline shortly af-
ter the reward’s usual time of deliv-

ery, and this happened without the
availability of any external cue to
mark the usual time of reward de-
livery (Figure 15.3). Somehow the

390 Chapter 15: Neuroscience
monkeys were internally keeping track of the timing of the reward. (Response timing is
one area where the simplest version of TD learning needs to be modiﬁed to account for
some of the details of the timing of dopamine neuron responses. We consider this issue in
the following section.)
The observations from the studies described above led Schultz and his group to conclude
that dopamine neurons respond to unpredicted rewards, to the earliest predictors of
reward, and that dopamine neuron activity decreases below baseline if a reward, or a
predictor of reward, does not occur at its expected time. Researchers familiar with
reinforcement learning were quick to recognize that these results are strikingly similar
to how the TD error behaves as the reinforcement signal in a TD algorithm. The next
section explores this similarity by working through a speciﬁc example in detail.
15.6 TD Error/Dopamine Correspondence

This section explains the correspondence between the TD error and the phasic responses
of dopamine neurons observed in the experiments just described. We examine how 
changes over the course of learning in a task something like the one described above
where a monkey ﬁrst sees an instruction cue and then a ﬁxed time later has to respond
correctly to a trigger cue in order to obtain reward. We use a simple idealized version of
this task, but we go into a lot more detail than is usual because we want to emphasize

the theoretical basis of the parallel between TD errors and dopamine neuron activity.
The ﬁrst simplifying assumption is that the agent has already learned the actions
required to obtain reward. Then its task is just to learn accurate predictions of future
reward for the sequence of states it experiences. This is then a prediction task, or
more technically, a policy-evaluation task: learning the value function for a ﬁxed policy
(Sections 4.1 and 6.1). The value function to be learned assigns to each state a value that

predicts the return that will follow that state if the agent selects actions according to the
given policy, where the return is the (possibly discounted) sum of all the future rewards.
This is unrealistic as a model of the monkey’s situation because the monkey would likely
learn these predictions at the same time that it is learning to act correctly (as would a
reinforcement learning algorithm that learns policies as well as value functions, such as
an actor–critic algorithm), but this scenario is simpler to describe than one in which a

policy and a value function are learned simultaneously.
Now imagine that the agent’s experience divides into multiple trials, in each of which
the same sequence of states repeats, with a distinct state occurring on each time step
during the trial. Further imagine that the return being predicted is limited to the return
over a trial, which makes a trial analogous to a reinforcement learning episode as we have
deﬁned it. In reality, of course, the returns being predicted are not conﬁned to single

trials, and the time interval between trials is an important factor in determining what an
animal learns. This is true for TD learning as well, but here we assume that returns do
not accumulate over multiple trials. Given this, then, a trial in experiments like those
conducted by Schultz and colleagues is equivalent to an episode of reinforcement learning.
(Though in this discussion, we will use the term trial instead of episode to relate better
to the experiments.)

15.6. TD Error/Dopamine Correspondence 391
As usual, we also need to make an assumption about how states are represented as
inputs to the learning algorithm, an assumption that inﬂuences how closely the TD error
corresponds to dopamine neuron activity. We discuss this issue later, but for now we
assume the same CSC representation used by Montague et al. (1996) in which there is a
separate internal stimulus for each state visited at each time step in a trial. This reduces
the process to the tabular case covered in the ﬁrst part of this book. Finally, we assume
that the agent uses TD(0) to learn a value function, V, stored in a lookup table initialized
to be zero for all the states. We also assume that this is a deterministic task and that
the discount factor, , is very nearly one so that we can ignore it.
Figure 15.4 shows the time courses of R,V, and at several stages of learning in this
policy-evaluation task. The time axes represent the time interval over which a sequence

of states is visited in a trial (where for clarity we omit showing individual states). The
reward signal is zero throughout each trial except when the agent reaches the rewarding
state, shown near the right end of the time line, when the reward signal becomes some
positive number, say R?. The goal of TD learning is to predict the return for each
state visited in a trial, which in this undiscounted case and given our assumption that
predictions are conﬁned to individual trials, is simply R?for each state.

392 Chapter 15: Neuroscience
Preceding the rewarding state is a sequence of reward-predicting states, with the
earliest reward-predicting state shown near the left end of the time line. This is like
the state near the start of a trial, for example like the state marked by the instruction
cue in a trial of the monkey experiment of Schultz et al. (1993) described above. It is
the ﬁrst state in a trial that reliably predicts that trial’s reward. (Of course, in reality
states visited on preceding trials are even earlier reward-predicting states, but because
we are conﬁning predictions to individual trials, these do not qualify as predictors of this
trial’s reward. Below we give a more satisfactory, though more abstract, description of an
earliest reward-predicting state.) The latest reward-predicting state in a trial is the state
immediately preceding the trial’s rewarding state. This is the state near the far right end
of the time line in Figure 15.4. Note that the rewarding state of a trial does not predict

the estimated values of the states visited at times tandt1 in a trial. The TD error at
this stage of learning is analogous to a dopamine neuron responding to an unpredicted
reward (e.g., a drop of apple juice) at the start of training.
Throughout this ﬁrst trial and all successive trials, TD(0) updates occur at each
state transition as described in Chapter 6. This successively increases the values of the
reward-predicting states, with the increases spreading backwards from the rewarding
state, until the values converge to the correct return predictions. In this case (because
we are assuming no discounting) the correct predictions are equal to R?for all the
reward-predicting states. This can be seen in Figure 15.4 as the graph of Vlabeled
‘learning complete’ where the values of all the states from the earliest to the latest
reward-predicting states all equal R?. The values of the states preceding the earliest
reward-predicting state remain low (which Figure 15.4 shows as zero) because they are

Schultz et al. (1993) described above and shown in Figure 15.3.
The idea of an earliest reward-predicting state deserves more attention. In the scenario
described above, because experience is divided into trials, and we assumed that predictions
are conﬁned to individual trials, the earliest reward-predicting state is always the ﬁrst
state of a trial. Clearly this is artiﬁcial. A more general way to think of an earliest
reward-predicting state is that it is an unpredicted predictor of reward, and there can
be many such states. In an animal’s life, many di↵erent states may precede an earliest
reward-predicting state. However, because these states are more often followed by other
states that do not predict reward, their reward-predicting powers, that is, their values,
remain low. A TD algorithm, if operating throughout the animal’s life, would update the
values of these states too, but the updates would not consistently accumulate because, by

assumption, none of these states reliably precedes an earliest reward-predicting state. If
any of them did, they would be reward-predicting states as well. This might explain why
with overtraining, dopamine responses decrease to even the earliest reward-predicting
stimulus in a trial. With overtraining one would expect that even a formerly-unpredicted
predictor state would become predicted by stimuli associated with earlier states: the
animal’s interaction with its environment both inside and outside of an experimental

task would become commonplace. Upon breaking this routine with the introduction of a
new task, however, one would see TD errors reappear, as indeed is observed in dopamine
neuron activity.
The example described above explains why the TD error shares key features with
the phasic activity of dopamine neurons when the animal is learning in a task similar
to the idealized task of our example. But not every property of the phasic activity of
dopamine neurons coincides so neatly with properties of . One of the most troubling

discrepancies involves what happens when a reward occurs earlier than expected. We
have seen that the omission of an expected reward produces a negative prediction error
at the reward’s expected time, which corresponds to the activity of dopamine neurons
decreasing below baseline when this happens. If the reward arrives later than expected,
it is then an unexpected reward and generates a positive prediction error. This happens
with both TD errors and dopamine neuron responses. But when reward arrives earlier

than expected, dopamine neurons do not do what the TD error does—at least with the
CSC representation used by Montague et al. (1996) and by us in our example. Dopamine

394 Chapter 15: Neuroscience
neurons do respond to the early reward, which is consistent with a positive TD error
because the reward is not predicted to occur then. However, at the later time when the
reward is expected but omitted, the TD error is negative whereas, in contrast to this
prediction, dopamine neuron activity does not drop below baseline in the way the TD
model predicts (Hollerman and Schultz, 1998). Something more complicated is going on
in the animal’s brain than simply TD learning with a CSC representation.
Some of the mismatches between the TD error and dopamine neuron activity can
be addressed by selecting suitable parameter values for the TD algorithm and by using
stimulus representations other than the CSC representation. For instance, to address
the early-reward mismatch just described, Suri and Schultz (1999) proposed a CSC
representation in which the sequences of internal signals initiated by earlier stimuli

are cancelled by the occurrence of a reward. Another proposal by Daw, Courville,
and Touretzky (2006) is that the brain’s TD system uses representations produced by
statistical modeling carried out in sensory cortex rather than simpler representations
based on raw sensory input. Ludvig, Sutton, and Kehoe (2008) found that TD learning
with a microstimulus (MS) representation (Figure 14.1) ﬁts the activity of dopamine
neurons in the early-reward and other situations better than when a CSC representation

is used. Pan, Schmidt, Wickens, and Hyland (2005) found that even with the CSC
representation, prolonged eligibility traces improve the ﬁt of the TD error to some aspects
of dopamine neuron activity. In general, many ﬁne details of TD-error behavior depend
on subtle interactions between eligibility traces, discounting, and stimulus representations.
Findings like these elaborate and reﬁne the reward prediction error hypothesis without

refuting its core claim that the phasic activity of dopamine neurons is well characterized
as signaling TD errors.
On the other hand, there are other discrepancies between the TD theory and exper-
imental data that are not so easily accommodated by selecting parameter values and
stimulus representations (we mention some of these discrepancies in the Bibliographical
and Historical Remarks section at the end of this chapter), and more mismatches are
likely to be discovered as neuroscientists conduct ever more reﬁned experiments. But the

reward prediction error hypothesis has been functioning very e↵ectively as a catalyst for
improving our understanding of how the brain’s reward system works. Intricate experi-
ments have been designed to validate or refute predictions derived from the hypothesis,
and experimental results have, in turn, led to reﬁnement and elaboration of the TD
error/dopamine hypothesis.
A remarkable aspect of these developments is that the reinforcement learning algo-
rithms and theory that connect so well with properties of the dopamine system were

developed from a computational perspective in total absence of any knowledge about the
relevant properties of dopamine neurons—remember, TD learning and its connections
to optimal control and dynamic programming were developed many years before any of
the experiments were conducted that revealed the TD-like nature of dopamine neuron
activity. This unplanned correspondence, despite not being perfect, suggests that the TD
error/dopamine parallel captures something signiﬁcant about brain reward processes.

In addition to accounting for many features of the phasic activity of dopamine neurons,
the reward prediction error hypothesis links neuroscience to other aspects of reinforcement

15.7. Neural Actor–Critic 395
learning, in particular, to learning algorithms that use TD errors as reinforcement signals.
Neuroscience is still far from reaching complete understanding of the circuits, molecular
mechanisms, and functions of the phasic activity of dopamine neurons, but evidence
supporting the reward prediction error hypothesis, along with evidence that phasic
dopamine responses are reinforcement signals for learning, suggest that the brain might
implement something like an actor–critic algorithm in which TD errors play critical roles.
Other reinforcement learning algorithms are plausible candidates too, but actor–critic
algorithms ﬁt the anatomy and physiology of the mammalian brain particularly well, as
we describe in the following two sections.
15.7 Neural Actor–Critic
Actor–critic algorithms learn both policies and value functions. The ‘actor’ is the
component that learns policies, and the ‘critic’ is the component that learns about

whatever policy is currently being followed by the actor in order to ‘criticize’ the actor’s
action choices. The critic uses a TD algorithm to learn the state-value function for the
actor’s current policy. The value function allows the critic to critique the actor’s action
choices by sending TD errors, , to the actor. A positive means that the action was
‘good’ because it led to a state with a better-than-expected value; a negative means

that the action was ‘bad’ because it led to a state with a worse-than-expected value.
Based on these critiques, the actor continually updates its policy.
Two distinctive features of actor–critic algorithms are responsible for thinking that the
brain might implement an algorithm like this. First, the two components of an actor–critic
algorithm—the actor and the critic—suggest that two parts of the striatum—the dorsal
and ventral subdivisions (Section 15.4), both critical for reward-based learning—may

function respectively something like an actor and a critic. A second property of actor–
critic algorithms that suggests a brain implementation is that the TD error has the dual
role of being the reinforcement signal for both the actor and the critic, though it has a
di↵erent inﬂuence on learning in each of these components. This ﬁts well with several
properties of the neural circuitry: axons of dopamine neurons target both the dorsal

and ventral subdivisions of the striatum; dopamine appears to be critical for modulating
synaptic plasticity in both structures; and how a neuromodulator such as dopamine
acts on a target structure depends on properties of the target structure and not just on
properties of the neuromodulator.
Section 13.5 presents actor–critic algorithms as policy gradient methods, but the actor–
critic algorithm of Barto, Sutton, and Anderson (1983) was simpler and was presented as
an artiﬁcial neural network (ANN). Here we describe an ANN implementation something

like that of Barto et al., and we follow Takahashi, Schoenbaum, and Niv (2008) in giving
a schematic proposal for how this ANN might be implemented by real neural networks in
the brain. We postpone discussion of the actor and critic learning rules until Section 15.8,
where we present them as special cases of the policy-gradient formulation and discuss
what they suggest about how dopamine might modulate synaptic plasticity.

396 Chapter 15: Neuroscience
Figure 15.5a shows an implementation of an actor–critic algorithm as an ANN with
component networks implementing the actor and the critic. The critic consists of a single
neuron-like unit, V, whose output activity represents state values, and a component
shown as the diamond labeled TD that computes TD errors by combining V’s output
with reward signals and with previous state values (as suggested by the loop from the
TD diamond to itself). The actor network has a single layer of kactor units labeled Ai,
i=1,...,k . The output of each actor unit is a component of a k-dimensional action
vector. An alternative is that there are kseparate actions, one commanded by each actor
unit, that compete with one another to be executed, but here we will think of the entire
A-vector as an action.
Reward
ActorTD. . .. . .Critic. . .δ ActionsStates/StimuliTD error
Environment

action. b) Hypothetical neural implementation of an actor–critic algorithm. The actor and the
value-learning part of the critic are respectively placed in the dorsal and ventral subdivisions
of the striatum. The TD error is transmitted by dopamine neurons located in the VTA and
SNpc to modulate changes in synaptic ecacies of input from cortical areas to the ventral and
dorsal striatum. Adapted from Frontiers in Neuroscience, vol. 2(1), 2008, Y. Takahashi, G.
Schoenbaum, and Y. Niv, Silencing the critics: Understanding the e↵ects of cocaine sensitization
on dorsolateral and ventral striatum in the context of an Actor/Critic model.

15.7. Neural Actor–Critic 397
Both the critic and actor networks receive input consisting of multiple features repre-
senting the state of the agent’s environment. (Recall from Chapter 1 that the environment
of a reinforcement learning agent includes components both inside and outside of the
‘organism’ containing the agent.) The ﬁgure shows these features as the circles labeled
x1,x2,...,x n, shown twice just to keep the ﬁgure simple. A weight representing the
ecacy of a synapse is associated with each connection from each feature xito the
critic unit, V, and to each of the action units, Ai. The weights in the critic network
parameterize the value function, and the weights in the actor network parameterize the
policy. The networks learn as these weights change according to the critic and actor
learning rules that we describe in the following section.
The TD error produced by circuitry in the critic is the reinforcement signal for changing

the weights in both the critic and the actor networks. This is shown in Figure 15.5a by
the line labeled ‘TD error ’ extending across all of the connections in the critic and
actor networks. This aspect of the network implementation, together with the reward
prediction error hypothesis and the fact that the activity of dopamine neurons is so
widely distributed by the extensive axonal arbors of these neurons, suggests that an
actor–critic network something like this may not be too farfetched as a hypothesis about

how reward-related learning might happen in the brain.
Figure 15.5b suggests—very schematically—how the ANN on the ﬁgure’s left might
map onto structures in the brain according to the hypothesis of Takahashi et al. (2008).
The hypothesis puts the actor and the value-learning part of the critic respectively in the
dorsal and ventral subdivisions of the striatum, the input structure of the basal ganglia.
Recall from Section 15.4 that the dorsal striatum is primarily implicated in inﬂuencing
action selection, and the ventral striatum is thought to be critical for di↵erent aspects of

reward processing, including the assignment of a↵ective value to sensations. The cerebral
cortex, along with other structures, sends input to the striatum conveying information
about stimuli, internal states, and motor activity.
In this hypothetical actor–critic brain implementation, the ventral striatum sends value
information to the VTA and SNpc, where dopamine neurons in these nuclei combine it
with information about reward to generate activity corresponding to TD errors (though

exactly how dopaminergic neurons calculate these errors is not yet understood). The ‘TD
error ’ line in Figure 15.5a becomes the line labeled ‘Dopamine’ in Figure 15.5b, which
represents the widely branching axons of dopamine neurons whose cell bodies are in the
VTA and SNpc. Referring back to Figure 15.1, these axons make synaptic contact with
the spines on the dendrites of medium spiny neurons, the main input/output neurons of
both the dorsal and ventral divisions of the striatum. Axons of the cortical neurons that

send input to the striatum make synaptic contact on the tips of these spines. According
to the hypothesis, it is at these spines where changes in the ecacies of the synapses
from cortical regions to the striatum are governed by learning rules that critically depend
on a reinforcement signal supplied by dopamine.
An important implication of the hypothesis illustrated in Figure 15.5b is that the
dopamine signal is not the ‘master’ reward signal like the scalar Rtof reinforcement
learning. In fact, the hypothesis implies that one should not necessarily be able to

probe the brain and record any signal like Rtin the activity of any single neuron.

398 Chapter 15: Neuroscience
Many interconnected neural systems generate reward-related information, with di↵erent
structures being recruited depending on di↵erent types of rewards. Dopamine neurons
receive information from many di↵erent brain areas, so the input to the SNpc and
VTA labeled ‘Reward’ in Figure 15.5b should be thought of as vector of reward-related
information arriving to neurons in these nuclei along multiple input channels. What the
theoretical scalar reward signal Rtmight correspond to, then, is the net contribution of
all reward-related information to dopamine neuron activity. It is the result of a pattern
of activity across many neurons in di↵erent areas of the brain.
Although the actor–critic neural implementation illustrated in Figure 15.5b may be
correct on some counts, it clearly needs to be reﬁned, extended, and modiﬁed to qualify
as a full-ﬂedged model of the function of the phasic activity of dopamine neurons. The

Historical and Bibliographic Remarks section at the end of this chapter cites publications
that discuss in more detail both empirical support for this hypothesis and places where it
falls short. We now look in detail at what the actor and critic learning algorithms suggest
about the rules governing changes in synaptic ecacies of corticostriatal synapses.
15.8 Actor and Critic Learning Rules
If the brain does implement something like the actor–critic algorithm—and assuming

populations of dopamine neurons broadcast a common reinforcement signal to the corti-
costriatal synapses of both the dorsal and ventral striatum as illustrated in Figure 15.5b
(which is likely an oversimpliﬁcation as we mentioned above)—then this reinforcement
signal a↵ects the synapses of these two structures in di↵erent ways. The learning rules for
the critic and the actor use the same reinforcement signal, the TD error ,b u ti t se ↵ e c t
on learning is di↵erent for these two components. The TD error (combined with eligibility

traces) tells the actor how to update action probabilities in order to reach higher-valued
states. Learning by the actor is like instrumental conditioning using a Law-of-E↵ect-type
learning rule (Section 1.7): the actor works to keep as positive as possible. On the
other hand, the TD error (when combined with eligibility traces) tells the critic the
direction and magnitude in which to change the parameters of the value function in order
to improve its predictive accuracy. The critic works to reduce ’s magnitude to be as

close to zero as possible using a learning rule like the TD model of classical conditioning
(Section 14.2). The di↵erence between the critic and actor learning rules is relatively
simple, but this di↵erence has a profound e↵ect on learning and is essential to how the
actor–critic algorithm works. The di↵erence lies solely in the eligibility traces each type
of learning rule uses.
More than one set of learning rules can be used in actor–critic neural networks like
those in Figure 15.5b but, to be speciﬁc, here we focus on the actor–critic algorithm for

continuing problems with eligibility traces presented in Section 13.6. On each transition
from state Stto state St+1, taking action Atand receiving action Rt+1, that algorithm
computes the TD error ( ) and then updates the eligibility trace vectors ( zw
tandz✓
t) and

linear function of the feature-vector representation of state s,x(s)=( x1(s),...,x n(s))>,
parameterized by a weight vector w=(w1,...,w n)>:
ˆv(s,w)=w>x(s). (15.1)
Each xi(s) is like the presynaptic signal to a neuron’s synapse whose ecacy is wi.T h e
weights of the critic are incremented according to the rule above by ↵wtzw
t,w h e r et h e
reinforcement signal, t, corresponds to a dopamine signal being broadcast to all of the
critic unit’s synapses. The eligibility trace vector, zw
t, for the critic unit is a trace (average
of recent values) of rˆv(St,w). Because ˆv(s,w) is linear in the weights, rˆv(St,w)=x(St).
In neural terms, this means that each synapse has its own eligibility trace, which is
one component of the vector zw
t. A synapse’s eligibility trace accumulates according to
the level of activity arriving at that synapse, that is, the level of presynaptic activity,
represented here by the component of the feature vector x(St) arriving at that synapse.

The trace otherwise decays toward zero at a rate governed by the fraction w. A synapse
iseligible for modiﬁcation as long as its eligibility trace is non-zero. How the synapse’s
ecacy is actually modiﬁed depends on the reinforcement signals that arrive while the
synapse is eligible. We call eligibility traces like these of the critic unit’s synapses non-
contingent eligibility traces because they only depend on presynaptic activity and are not
contingent in any way on postsynaptic activity.

The non-contingent eligibility traces of the critic unit’s synapses mean that the critic
unit’s learning rule is essentially the TD model of classical conditioning described in
Section 14.2. With the deﬁnition we have given above of the critic unit and its learning
rule, the critic in Figure 15.5a is the same as the critic in the ANN actor–critic of Barto
et al. (1983). Clearly, a critic like this consisting of just one linear neuron-like unit is the

simplest starting point; this critic unit is a proxy for a more complicated neural network
able to learn value functions of greater complexity.
The actor in Figure 15.5a is a one-layer network of kneuron-like actor units, each
receiving at time tthe same feature vector, x(St), that the critic unit receives. Each
actor unit j,j=1,...,k , has its own weight vector, ✓j, but because the actor units are
all identical, we describe just one of the units and omit the subscript. One way for these

400 Chapter 15: Neuroscience
units to follow the actor–critic algorithm given in the equations above is for each to be a
Bernoulli-logistic unit . This means that the output of each actor unit at each time is
a random variable, At, taking value 0 or 1. Think of value 1 as the neuron ﬁring, that
is, emitting an action potential. The weighted sum, ✓>x(St), of a unit’s input vector
determines the unit’s action probabilities via the exponential soft-max distribution (13.2),
which for two actions is the logistic function:
⇡(1|s,✓)=1 ⇡(0|s,✓)=1
1+e x p ( ✓>x(s)). (15.2)
The weights of each actor unit are incremented, as above, by: ✓ ✓+↵✓tz✓
t,w h e r e
again corresponds to the dopamine signal: the same reinforcement signal that is sent to
all the critic unit’s synapses. Figure 15.5a shows tbeing broadcast to all the synapses
of all the actor units (which makes this actor network a team of reinforcement learning
agents, something we discuss in Section 15.10 below). The actor eligibility trace vector
z✓

z✓
tis a trace (average of recent values) of rln⇡(At|St,✓). To understand this eligibility
trace refer to Exercise 13.5, which deﬁnes this kind of unit and asks you to give a learning
rule for it. That exercise asked you to express rln⇡(a|s,✓) in terms of a,x(s), and
⇡(a|s,✓) (for arbitrary state sand action a) by calculating the gradient. For the action
and state actually occurring at time t, the answer is
r⇡(At|St,✓)=
At⇡(At|St,✓)
x(St). (15.3)
Unlike the non-contingent eligibility trace of a critic synapse that only accumulates
the presynaptic activity x(St), the eligibility trace of an actor unit’s synapse in addition
depends on the activity of the actor unit itself. We call this a contingent eligibility
trace because it is contingent on this postsynaptic activity. The eligibility trace at each
synapse continually decays, but increments or decrements depending on the activity of
the presynaptic neuron and whether or not the postsynaptic neuron ﬁres. The factor

At⇡(At|St,✓)i n(15.3) is positive when At= 1 and negative otherwise. The postsynaptic
contingency in the eligibility traces of actor units is the only di↵erence between the critic
and actor learning rules. By keeping information about what actions were taken in
what states, contingent eligibility traces allow credit for reward (positive ), or blame for
punishment (negative ), to be apportioned among the policy parameters (the ecacies
of the actor units’ synapses) according to the contributions these parameters made to the

15.8. Actor and Critic Learning Rules 401
quantities. The actor learning rule, on the other hand, is a three-factor learning rule
because, in addition to depending on , its eligibility traces depend on both presynaptic
and postsynaptic activity. Unlike Hebb’s proposal, however, the relative timing of the
factors is critical to how synaptic ecacies change, with eligibility traces intervening to
allow the reinforcement signal to a↵ect synapses that were active in the recent past.
Some subtleties about signal timing for the actor and critic learning rules deserve closer
attention. In deﬁning the neuron-like actor and critic units, we ignored the small amount
of time it takes synaptic input to e↵ect the ﬁring of a real neuron. When an action
potential from the presynaptic neuron arrives at a synapse, neurotransmitter molecules
are released that di↵use across the synaptic cleft to the postsynaptic neuron, where
they bind to receptors on the postsynaptic neuron’s surface; this activates molecular

machinery that causes the postsynaptic neuron to ﬁre (or to inhibit its ﬁring in the case of
inhibitory synaptic input). This process can take several tens of milliseconds. According
to (15.1) and (15.2), though, the input to a critic and actor unit instantaneously produces
the unit’s output. Ignoring activation time like this is common in abstract models of
Hebbian-style plasticity in which synaptic ecacies change according to a simple product
of simultaneous pre- and postsynaptic activity. More realistic models must take activation
time into account.

. To assign credit for reinforce-
ment correctly, the presynaptic factor deﬁning the eligibility trace must be a cause of the
postsynaptic factor that also deﬁnes the trace. Contingent eligibility traces for a more
realistic actor unit would have to take activation time into account. (Activation time
should not be confused with the time required for a neuron to receive a reinforcement
signal inﬂuenced by that neuron’s activity. The function of eligibility traces is to span
this time interval which is generally much longer than the activation time. We discuss
this further in the following section.)
There are hints from neuroscience for how this process might work in the brain.
Neuroscientists have discovered a form of Hebbian plasticity called spike-timing-dependent
plasticity (STDP) that lends plausibility to the existence of actor-like synaptic plasticity
in the brain. STDP is a Hebbian-style plasticity, but changes in a synapse’s ecacy

depend on the relative timing of presynaptic and postsynaptic action potentials. The
dependence can take di↵erent forms, but in the one most studied, a synapse increases in
strength if spikes incoming via that synapse arrive shortly before the postsynaptic neuron
ﬁres. If the timing relation is reversed, with a presynaptic spike arriving shortly after the
postsynaptic neuron ﬁres, then the strength of the synapse decreases. STDP is a type of
Hebbian plasticity that takes the activation time of a neuron into account, which is one

of the ingredients needed for actor-like learning.

402 Chapter 15: Neuroscience
The discovery of STDP has led neuroscientists to investigate the possibility of a three-
factor form of STDP in which neuromodulatory input must follow appropriately-timed
pre- and postsynaptic spikes. This form of synaptic plasticity, called reward-modulated
STDP , is much like the actor learning rule discussed here. Synaptic changes that would
be produced by regular STDP only occur if there is neuromodulatory input within a time
window after a presynaptic spike is closely followed by a postsynaptic spike. Evidence
is accumulating that reward-modulated STDP occurs at the spines of medium spiny
neurons of the dorsal striatum, with dopamine providing the neuromodulatory factor—
the sites where actor learning takes place in the hypothetical neural implementation of
an actor–critic algorithm illustrated in Figure 15.5b. Experiments have demonstrated
reward-modulated STDP in which lasting changes in the ecacies of corticostriatal

synapses occur only if a neuromodulatory pulse arrives within a time window that can
last up to 10 seconds after a presynaptic spike is closely followed by a postsynaptic
spike (Yagishita et al. 2014). Although the evidence is indirect, these experiments point
to the existence of contingent eligibility traces having prolonged time courses. The
molecular mechanisms producing these traces, as well as the much shorter traces that
likely underly STDP, are not yet understood, but research focusing on time-dependent

and neuromodulator-dependent synaptic plasticity is continuing.
The neuron-like actor unit that we have described here, with its Law-of-E↵ect-style
learning rule, appeared in somewhat simpler form in the actor–critic network of Barto et
al. (1983). That network was inspired by the “hedonistic neuron” hypothesis proposed
by physiologist A. H. Klopf (1972, 1982). Not all the details of Klopf’s hypothesis are
consistent with what has been learned about synaptic plasticity, but the discovery of
STDP and the growing evidence for a reward-modulated form of STDP suggest that

Klopf’s ideas may not have been far o↵ the mark. We discuss Klopf’s hedonistic neuron
hypothesis next.
15.9 Hedonistic Neurons
In his hedonistic neuron hypothesis, Klopf (1972, 1982) conjectured that individual
neurons seek to maximize the di↵erence between synaptic input treated as rewarding
and synaptic input treated as punishing by adjusting the ecacies of their synapses
on the basis of rewarding or punishing consequences of their own action potentials. In

other words, individual neurons can be trained with response-contingent reinforcement
like an animal can be trained in an instrumental conditioning task. His hypothesis
included the idea that rewards and punishments are conveyed to a neuron via the same
synaptic input that excites or inhibits the neuron’s spike-generating activity. (Had Klopf
known what we know today about neuromodulatory systems, he might have assigned the
reinforcing role to neuromodulatory input, but he wanted to avoid any centralized source

of training information.) Synaptically-local traces of past pre- and postsynaptic activity
had the key function in Klopf’s hypothesis of making synapses eligible —the term he
introduced—for modiﬁcation by later reward or punishment. He conjectured that these
traces are implemented by molecular mechanisms local to each synapse and therefore
di↵erent from the electrical activity of both the pre- and the postsynaptic neurons. In

15.9. Hedonistic Neurons 403
the Bibliographical and Historical Remarks section of this chapter we bring attention to
some similar proposals made by others.
Klopf speciﬁcally conjectured that synaptic ecacies change in the following way. When
a neuron ﬁres an action potential, all of its synapses that were active in contributing to
that action potential become eligible to undergo changes in their ecacies. If the action
potential is followed within an appropriate time period by an increase of reward, the
ecacies of all the eligible synapses increase. Symmetrically, if the action potential is
followed within an appropriate time period by an increase of punishment, the ecacies
of eligible synapses decrease. This is implemented by triggering an eligibility trace at a
synapse upon a coincidence of presynaptic and postsynaptic activity (or more exactly,
upon pairing of presynaptic activity with the postsynaptic activity that that presynaptic

activity participates in causing)—what we call a contingent eligibility trace. This is
essentially the three-factor learning rule of an actor unit described in the previous section.
The shape and time course of an eligibility trace in Klopf’s theory reﬂects the durations
of the many feedback loops in which the neuron is embedded, some of which lie entirely
within the brain and body of the organism, while others extend out through the organism’s
external environment as mediated by its motor and sensory systems. His idea was that

the shape of a synaptic eligibility trace is like a histogram of the durations of the feedback
loops in which the neuron is embedded. The peak of an eligibility trace would then occur
at the duration of the most prevalent feedback loops in which that neuron participates.
The eligibility traces used by algorithms described in this book are simpliﬁed versions
of Klopf’s original idea, being exponentially (or geometrically) decreasing functions
controlled by the parameters and. This simpliﬁes simulations as well as theory, but

we regard these simple eligibility traces as a placeholders for traces closer to Klopf’s
original conception, which would have computational advantages in complex reinforcement
learning systems by reﬁning the credit-assignment process.
Klopf’s hedonistic neuron hypothesis is not as implausible as it may at ﬁrst appear.
A well-studied example of a single cell that seeks some stimuli and avoids others is the
bacterium Escherichia coli . The movement of this single-cell organism is inﬂuenced by
chemical stimuli in its environment, behavior known as chemotaxis. It swims in its liquid

environment by rotating hairlike structures called ﬂagella attached to its surface. (Yes,
it rotates them!) Molecules in the bacterium’s environment bind to receptors on its
surface. Binding events modulate the frequency with which the bacterium reverses ﬂagellar
rotation. Each reversal causes the bacterium to tumble in place and then head o↵ in a
random new direction. A little chemical memory and computation causes the frequency
of ﬂagellar reversal to decrease when the bacterium swims toward higher concentrations

of molecules it needs to survive (attractants) and increase when the bacterium swims
toward higher concentrations of molecules that are harmful (repellants). The result is
that the bacterium tends to persist in swimming up attractant gradients and tends to
avoid swimming up repellant gradients.
The chemotactic behavior just described is called klinokinesis. It is a kind of trial-
and-error behavior, although it is unlikely that learning is involved: the bacterium needs
a modicum of short-term memory to detect molecular concentration gradients, but it

probably does not maintain long-term memories. Artiﬁcial intelligence pioneer Oliver

404 Chapter 15: Neuroscience
Selfridge called this strategy “run and twiddle,” pointing out its utility as a basic adaptive
strategy: “keep going in the same way if things are getting better, and otherwise move
around” (Selfridge, 1978, 1984). Similarly, one might think of a neuron “swimming” (not
literally of course) in a medium composed of the complex collection of feedback loops
in which it is embedded, acting to obtain one type of input signal and to avoid others.
Unlike the bacterium, however, the neuron’s synaptic strengths retain information about
its past trial-and-error behavior. If this view of the behavior of a neuron (or just one type
of neuron) is plausible, then the closed-loop nature of how the neuron interacts with its
environment is important for understanding its behavior, where the neuron’s environment
consists of the rest of the animal together with the environment with which the animal
as a whole interacts.

Klopf’s hedonistic neuron hypothesis extended beyond the idea that individual neurons
are reinforcement learning agents. He argued that many aspects of intelligent behavior
can be understood as the result of the collective behavior of a population of self-interested
hedonistic neurons interacting with one another in an immense society or economic system
making up an animal’s nervous system. Whether or not this view of nervous systems
is useful, the collective behavior of reinforcement learning agents has implications for

neuroscience. We take up this subject next.
15.10 Collective Reinforcement Learning
The behavior of populations of reinforcement learning agents is deeply relevant to the
study of social and economic systems, and if anything like Klopf’s hedonistic neuron
hypothesis is correct, to neuroscience as well. The hypothesis described above about how
an actor–critic algorithm might be implemented in the brain only narrowly addresses
the implications of the fact that the dorsal and ventral subdivisions of the striatum, the

respective locations of the actor and the critic according to the hypothesis, each contain
millions of medium spiny neurons whose synapses undergo change modulated by phasic
bursts of dopamine neuron activity.
The actor in Figure 15.5a is a single-layer network of kactor units. The actions
produced by this network are vectors ( A1,A2,···,Ak)>presumed to drive the animal’s
behavior. Changes in the ecacies of the synapses of all of these units depend on the
reinforcement signal . Because actor units attempt to make as large as possible, 

e↵ectively acts as a reward signal for them (so in this case reinforcement is the same
as reward). Thus, each actor unit is itself a reinforcement learning agent—a hedonistic
neuron if you will. Now, to make the situation as simple as possible, assume that each
of these units receives the same reward signal at the same time (although, as indicated
above, the assumption that dopamine is released at all the corticostriatal synapses under
the same conditions and at the same times is likely an oversimpliﬁcation).

What can reinforcement learning theory tell us about what happens when all members
of a population of reinforcement learning agents learn according to a common reward
signal? The ﬁeld of multi-agent reinforcement learning considers many aspects of learning
by populations of reinforcement learning agents. Although this ﬁeld is beyond the scope
of this book, we believe that some of its basic concepts and results are relevant to thinking

15.10. Collective Reinforcement Learning 405
about the brain’s di↵use neuromodulatory systems. In multi-agent reinforcement learning
(and in game theory), the scenario in which all the agents try to maximize a common
reward signal that they simultaneously receive is known as a cooperative game or a team
problem .
What makes a team problem interesting and challenging is that the common reward
signal sent to each agent evaluates the pattern of activity produced by the entire population,
that is, it evaluates the collective action of the team members. This means that any
individual agent has only limited ability to a↵ect the reward signal because any single
agent contributes just one component of the collective action evaluated by the common
reward signal. E↵ective learning in this scenario requires addressing a structural credit
assignment problem : which team members, or groups of team members, deserve credit for
a favorable reward signal, or blame for an unfavorable reward signal? It is a cooperative

game, or a team problem, because the agents are united in seeking to increase the same
reward signal: there are no conﬂicts of interest among the agents. The scenario would be
acompetitive game if di↵erent agents receive di↵erent reward signals, where each reward
signal again evaluates the collective action of the population, and the objective of each
agent is to increase its own reward signal. In this case there might be conﬂicts of interest
among the agents, meaning that actions that are good for some agents are bad for others.

Even deciding what the best collective action should be is a non-trivial aspect of game
theory. This competitive setting might be relevant to neuroscience too (for example, to
account for heterogeneity of dopamine neuron activity), but here we focus only on the
cooperative, or team, case.
How can each reinforcement learning agent in a team learn to “do the right thing” so
that the collective action of the team is highly rewarded? An interesting result is that
if each agent can learn e↵ectively despite its reward signal being corrupted by a large

amount of noise, and despite its lack of access to complete state information, then the
population as a whole will learn to produce collective actions that improve as evaluated by
the common reward signal, even when the agents cannot communicate with one another.
Each agent faces its own reinforcement learning task in which its inﬂuence on the reward
signal is deeply buried in the noise created by the inﬂuences of other agents. In fact, for
any agent, all the other agents are part of its environment because its input, both the part

conveying state information and the reward part, depends on how all the other agents
are behaving. Furthermore, lacking access to the actions of the other agents, indeed
lacking access to the parameters determining their policies, each agent can only partially
observe the state of its environment. This makes each team member’s learning task very
dicult, but if each uses a reinforcement learning algorithm able to increase a reward
signal even under these dicult conditions, teams of reinforcement learning agents can

learn to produce collective actions that improve over time as evaluated by the team’s
common reward signal.
If the team members are neuron-like units, then each unit has to have the goal of
increasing the amount of reward it receives over time, as the actor unit does that we
described in Section 15.8. Each unit’s learning algorithm has to have two essential features.
First, it has to use contingent eligibility traces. Recall that a contingent eligibility trace,
in neural terms, is initiated (or increased) at a synapse when its presynaptic input

406 Chapter 15: Neuroscience
participates in causing the postsynaptic neuron to ﬁre. A non-contingent eligibility trace,
in contrast, is initiated or increased by presynaptic input independently of what the
postsynaptic neuron does. As explained in Section 15.8, by keeping information about
what actions were taken in what states, contingent eligibility traces allow credit for
reward, or blame for punishment, to be apportioned to an agent’s policy parameters
according to the contribution the values of these parameters made in determining the
agent’s action. By similar reasoning, a team member must remember its recent action so
that it can either increase or decrease the likelihood of producing that action according
to the reward signal that is subsequently received. The action component of a contingent
eligibility trace implements this action memory. Because of the complexity of the learning
task, however, contingent eligibility is merely a preliminary step in the credit assignment

process: the relationship between a single team member’s action and changes in the
team’s reward signal is a statistical correlation that has to be estimated over many trials.
Contingent eligibility is an essential but preliminary step in this process.
Learning with non-contingent eligibility traces does not work at all in the team setting
because it does not provide a way to correlate actions with consequent changes in the
reward signal. Non-contingent eligibility traces are adequate for learning to predict, as

the critic component of the actor–critic algorithm does, but they do not support learning
to control, as the actor component must do. The members of a population of critic-like
agents may still receive a common reinforcement signal, but they would all learn to
predict the same quantity (which in the case of an actor–critic method, would be the
expected return for the current policy). How successful each member of the population
would be in learning to predict the expected return would depend on the information it

receives, which could be very di↵erent for di↵erent members of the population. There
would be no need for the population to produce di↵erentiated patterns of activity. This
is not a team problem as deﬁned here.
A second requirement for collective learning in a team problem is that there has to be
variability in the actions of the team members in order for the team to explore the space
of collective actions. The simplest way for a team of reinforcement learning agents to do
this is for each member to independently explore its own action space through persistent

variability in its output. This will cause the team as a whole to vary its collective actions.
For example, a team of the actor units described in Section 15.8 explores the space
of collective actions because the output of each unit, being a Bernoulli-logistic unit,
probabilistically depends on the weighted sum of its input vector’s components. The
weighted sum biases ﬁring probability up or down, but there is always variability. Because
each unit uses a REINFORCE policy gradient algorithm (Chapter 13), each unit adjusts

its weights with the goal of maximizing the average reward rate it experiences while
stochastically exploring its own action space. One can show, as Williams (1992) did, that
a team of Bernoulli-logistic REINFORCE units implements a policy gradient algorithm
as a whole with respect to average rate of the team’s common reward signal, where the
actions are the collective actions of the team.
Further, Williams (1992) showed that a team of Bernoulli-logistic units using REIN-
FORCE ascends the average reward gradient when the units in the team are interconnected

to form a multilayer ANN. In this case, the reward signal is broadcast to all the units in

15.11. Model-based Methods in the Brain 407
the network, though reward may depend only on the collective actions of the network’s
output units. This means that a multilayer team of Bernoulli-logistic REINFORCE
units learns like a multilayer network trained by the widely-used error backpropagation
method, but in this case the backpropagation process is replaced by the broadcasted
reward signal. In practice, the error backpropagation method is considerably faster,
but the reinforcement learning team method is more plausible as a neural mechanism,
especially in light of what is being learned about reward-modulated STDP as discussed
in Section 15.8.
Exploration through independent exploration by team members is only the simplest
way for a team to explore; more sophisticated methods are possible if the team members
coordinate their actions to focus on particular parts of the collective action space, either
by communicating with one another or by responding to common inputs. There are also

mechanisms more sophisticated than contingent eligibility traces for addressing structural
credit assignment, which is easier in a team problem when the set of possible collective
actions is restricted in some way. An extreme case is a winner-take-all arrangement (for
example, the result of lateral inhibition in the brain) that restricts collective actions to
those to which only one, or a few, team members contribute. In this case the winners get
the credit or blame for resulting reward or punishment.

Details of learning in cooperative games (or team problems) and non-cooperative
game problems are beyond the scope of this book. The Bibliographical and Historical
Remarks section at the end of this chapter cites a selection of the relevant publications,
including extensive references to research on implications for neuroscience of collective
15.11 Model-based Methods in the Brain
Reinforcement learning’s distinction between model-free and model-based algorithms is

proving to be useful for thinking about animal learning and decision processes. Section 14.6
discusses how this distinction aligns with that between habitual and goal-directed animal
behavior. The hypothesis discussed above about how the brain might implement an
actor–critic algorithm is relevant only to an animal’s habitual mode of behavior because
the basic actor–critic method is model-free. What neural mechanisms are responsible
for producing goal-directed behavior, and how do they interact with those underlying

habitual behavior?
One way to investigate questions about the brain structures involved in these modes
of behavior is to inactivate an area of a rat’s brain and then observe what the rat does in
an outcome-devaluation experiment (Section 14.6). Results from experiments like these
indicate that the actor–critic hypothesis described above is too simple in placing the
actor in the dorsal striatum. Inactivating one part of the dorsal striatum, the dorsolateral
striatum (DLS), impairs habit learning, causing the animal to rely more on goal-directed

processes. On the other hand, inactivating the dorsomedial striatum (DMS) impairs
goal-directed processes, requiring the animal to rely more on habit learning. Results
like these support the view that the DLS in rodents is more involved in model-free
processes, whereas their DMS is more involved in model-based processes. Results of

408 Chapter 15: Neuroscience
studies with human subjects in similar experiments using functional neuroimaging, and
with non-human primates, support the view that the analogous structures in the primate
brain are di↵erentially involved in habitual and goal-directed modes of behavior.
Other studies identify activity associated with model-based processes in the prefrontal
cortex of the human brain, the front-most part of the frontal cortex implicated in
executive function, including planning and decision making. Speciﬁcally implicated is the
orbitofrontal cortex (OFC), the part of the prefrontal cortex immediately above the eyes.
Functional neuroimaging in humans, and also recordings of the activities of single neurons
in monkeys, reveals strong activity in the OFC related to the subjective reward value
of biologically signiﬁcant stimuli, as well as activity related to the reward expected as a
consequence of actions. Although not free of controversy, these results suggest signiﬁcant

involvement of the OFC in goal-directed choice. It may be critical for the reward part of
an animal’s environment model.
Another structure involved in model-based behavior is the hippocampus, a structure
critical for memory and spatial navigation. A rat’s hippocampus plays a critical role
in the rat’s ability to navigate a maze in the goal-directed manner that led Tolman to
the idea that animals use models, or cognitive maps, in selecting actions (Section 14.5).
The hippocampus may also be a critical component of our human ability to imagine

new experiences (Hassabis and Maguire, 2007; ´Olafsd´ ottir, Barry, Saleem, Hassabis, and
Spiers, 2015).
The ﬁndings that most directly implicate the hippocampus in planning—the process
needed to enlist an environment model in making decisions—come from experiments
that decode the activity of neurons in the hippocampus to determine what part of space
hippocampal activity is representing on a moment-to-moment basis. When a rat pauses
at a choice point in a maze, the representation of space in the hippocampus sweeps

forward (and not backwards) along the possible paths the animal can take from that
point (Johnson and Redish, 2007). Furthermore, the spatial trajectories represented by
these sweeps closely correspond to the rat’s subsequent navigational behavior (Pfei↵er
and Foster, 2013). These results suggest that the hippocampus is critical for the state-
transition part of an animal’s environment model, and that it is part of a system that
uses the model to simulate possible future state sequences to assess the consequences of
possible courses of action: a form of planning.

The results described above add to a voluminous literature on neural mechanisms
underlying goal-directed, or model-based, learning and decision making, but many
questions remain unanswered. For example, how can areas as structurally similar as the
DLS and DMS be essential components of modes of learning and behavior that are as
di↵erent as model-free and model-based algorithms? Are separate structures responsible
for (what we call) the transition and reward components of an environment model? Is all

planning conducted at decision time via simulations of possible future courses of action
as the forward sweeping activity in the hippocampus suggests? In other words, is all
planning something like a rollout algorithm (Section 8.10)? Or are models sometimes
engaged in the background to reﬁne or recompute value information as illustrated by the
Dyna architecture (Section 8.2)? How does the brain arbitrate between the use of the

habit and goal-directed systems? Is there, in fact, a clear separation between the neural

15.12. Addiction 409
substrates of these systems?
The evidence is not pointing to a positive answer to this last question. Summarizing
the situation, Doll, Simon, and Daw (2012) wrote that “model-based inﬂuences appear
ubiquitous more or less wherever the brain processes reward information,” and this is
true even in the regions thought to be critical for model-free learning. This includes the
dopamine signals themselves, which can exhibit the inﬂuence of model-based information
in addition to the reward prediction errors thought to be the basis of model-free processes.
Continuing neuroscience research informed by reinforcement learning’s model-free and
model-based distinction has the potential to sharpen our understanding of habitual and
goal-directed processes in the brain. A better grasp of these neural mechanisms may lead
to algorithms combining model-free and model-based methods in ways that have not yet
been explored in computational reinforcement learning.
15.12 Addiction

Understanding the neural basis of drug abuse is a high-priority goal of neuroscience with
the potential to produce new treatments for this serious public health problem. One
view is that drug craving is the result of the same motivation and learning processes that
lead us to seek natural rewarding experiences that serve our biological needs. Addictive
substances, by being intensely reinforcing, e↵ectively co-opt our natural mechanisms

of learning and decision making. This is plausible given that many—though not all—
drugs of abuse increase levels of dopamine either directly or indirectly in regions around
terminals of dopamine neuron axons in the striatum, a brain structure ﬁrmly implicated in
normal reward-based learning (Section 15.7). But the self-destructive behavior associated
with drug addiction is not characteristic of normal learning. What is di↵erent about

dopamine-mediated learning when the reward is the result of an addictive drug? Is
addiction the result of normal learning in response to substances that were largely
unavailable throughout our evolutionary history, so that evolution could not select against
their damaging e↵ects? Or do addictive substances somehow interfere with normal
dopamine-mediated learning?
The reward prediction error hypothesis of dopamine neuron activity and its connection
to TD learning are the basis of a model due to Redish (2004) of some—but certainly not

all—features of addiction. The model is based on the observation that administration of
cocaine and some other addictive drugs produces a transient increase in dopamine. In the
model, this dopamine surge is assumed to increase the TD error, , in a way that cannot
be cancelled out by changes in the value function. In other words, whereas is reduced
to the degree that a normal reward is predicted by antecedent events (Section 15.6), the
contribution to due to an addictive stimulus does not decrease as the reward signal

becomes predicted: drug rewards cannot be “predicted away.” The model does this by
preventing from ever becoming negative when the reward signal is due to an addictive
drug, thus eliminating the error-correcting feature of TD learning for states associated
with administration of the drug. The result is that the values of these states increase
without bound, making actions leading to these states preferred above all others.

410 Chapter 15: Neuroscience
Addictive behavior is much more complicated than this result from Redish’s model, but
the model’s main idea may be a piece of the puzzle. Or the model might be misleading.
Dopamine appears not to play a critical role in all forms of addiction, and not everyone
is equally susceptible to developing addictive behavior. Moreover, the model does not
include the changes in many circuits and brain regions that accompany chronic drug
taking, for example, changes that lead to a drug’s diminishing e↵ect with repeated use.
It is also likely that addiction involves model-based processes. Still, Redish’s model
illustrates how reinforcement learning theory can be enlisted in the e↵ort to understand
a major health problem. In a similar manner, reinforcement learning theory has been
inﬂuential in the development of the new ﬁeld of computational psychiatry, which aims
to improve understanding of mental disorders through mathematical and computational
methods.
15.13 Summary

methods.
The neural pathways involved in the brain’s reward system are complex and incompletely
understood, but neuroscience research directed toward understanding these pathways
and their roles in behavior is progressing rapidly. This research is revealing striking
correspondences between the brain’s reward system and the theory of reinforcement
learning as presented in this book.
The reward prediction error hypothesis of dopamine neuron activity was proposed by
scientists who recognized striking parallels between the behavior of TD errors and the

activity of neurons that produce dopamine, a neurotransmitter essential in mammals
for reward-related learning and behavior. Experiments conducted in the late 1980s and
1990s in the laboratory of neuroscientist Wolfram Schultz showed that dopamine neurons
respond to rewarding events with substantial bursts of activity, called phasic responses,
only if the animal does not expect those events, suggesting that dopamine neurons are
signaling reward prediction errors instead of reward itself. Further, these experiments

showed that as an animal learns to predict a rewarding event on the basis of preceding
sensory cues, the phasic activity of dopamine neurons shifts to earlier predictive cues
while decreasing to later predictive cues. This parallels the backing-up e↵ect of the TD
error as a reinforcement learning agent learns to predict reward.
Other experimental results ﬁrmly establish that the phasic activity of dopamine neurons
is a reinforcement signal for learning that reaches multiple areas of the brain by means of

profusely branching axons of dopamine producing neurons. These results are consistent
with the distinction we make between a reward signal, Rt, and a reinforcement signal,
which is the TD error tin most of the algorithms we present. Phasic responses of
dopamine neurons are reinforcement signals, not reward signals.
A prominent hypothesis is that the brain implements something like an actor–critic
algorithm. Two structures in the brain (the dorsal and ventral subdivisions of the

striatum), both of which play critical roles in reward-based learning, may function
respectively like an actor and a critic. That the TD error is the reinforcement signal for
both the actor and the critic ﬁts well with the facts that dopamine neuron axons target
both the dorsal and ventral subdivisions of the striatum; that dopamine appears to be

15.13. Summary 411
critical for modulating synaptic plasticity in both structures; and that the e↵ect on a
target structure of a neuromodulator such as dopamine depends on properties of the
target structure and not just on properties of the neuromodulator.
The actor and the critic can be implemented by ANNs consisting of neuron-like units
having learning rules based on the policy-gradient actor–critic method described in
Section 13.5. Each connection in these networks is like a synapse between neurons in
the brain, and the learning rules correspond to rules governing how synaptic ecacies
change as functions of the activities of the presynaptic and the postsynaptic neurons,
together with neuromodulatory input corresponding to input from dopamine neurons. In
this setting, each synapse has its own eligibility trace that records past activity involving
that synapse. The only di↵erence between the actor and critic learning rules is that they

use di↵erent kinds of eligibility traces: the critic unit’s traces are non-contingent because
they do not involve the critic unit’s output, whereas the actor unit’s traces are contingent
because in addition to the actor unit’s input, they depend on the actor unit’s output. In
the hypothetical implementation of an actor–critic system in the brain, these learning
rules respectively correspond to rules governing plasticity of corticostriatal synapses that
convey signals from the cortex to the principal neurons in the dorsal and ventral striatal

subdivisions, synapses that also receive inputs from dopamine neurons.
The learning rule of an actor unit in the actor–critic network closely corresponds to
reward-modulated spike-timing-dependent plasticity . In spike-timing-dependent plasticity
(STDP), the relative timing of pre- and postsynaptic activity determines the direction of
synaptic change. In reward-modulated STDP, changes in synapses in addition depend
on a neuromodulator, such as dopamine, arriving within a time window that can last

up to 10 seconds after the conditions for STDP are met. Evidence is accumulating that
reward-modulated STDP occurs at corticostriatal synapses, where the actor’s learning
takes place in the hypothetical neural implementation of an actor–critic system, adds to
the plausibility of the hypothesis that something like an actor–critic system exists in the
brains of some animals.
The idea of synaptic eligibility and basic features of the actor learning rule derive

from Klopf’s hypothesis of the “hedonistic neuron” (Klopf, 1972, 1981). He conjectured
that individual neurons seek to obtain reward and to avoid punishment by adjusting the
ecacies of their synapses on the basis of rewarding or punishing consequences of their
action potentials. A neuron’s activity can a↵ect its later input because the neuron is
embedded in many feedback loops, some within the animal’s nervous system and body
and others passing through the animal’s external environment. Klopf’s idea of eligibility

is that synapses are temporarily marked as eligible for modiﬁcation if they participated
in the neuron’s ﬁring (making this the contingent form of eligibility trace). A synapse’s
ecacy is modiﬁed if a reinforcing signal arrives while the synapse is eligible. We alluded
to the chemotactic behavior of a bacterium as an example of a single cell that directs its
movements in order to seek some molecules and to avoid others.
A conspicuous feature of the dopamine system is that ﬁbers releasing dopamine project

widely to multiple parts of the brain. Although it is likely that only some populations
of dopamine neurons broadcast the same reinforcement signal, if this signal reaches
the synapses of many neurons involved in actor-type learning, then the situation can

412 Chapter 15: Neuroscience
be modeled as a team problem . In this type of problem, each agent in a collection of
reinforcement learning agents receives the same reinforcement signal, where that signal
depends on the activities of all members of the collection, or team. If each team member
uses a suciently capable learning algorithm, the team can learn collectively to improve
performance of the entire team as evaluated by the globally-broadcast reinforcement
signal, even if the team members do not directly communicate with one another. This
is consistent with the wide dispersion of dopamine signals in the brain and provides
a neurally plausible alternative to the widely-used error-backpropagation method for
training multilayer networks.
The distinction between model-free and model-based reinforcement learning is helping
neuroscientists investigate the neural bases of habitual and goal-directed learning and
decision making. Research so far points to their being some brain regions more involved

in one type of process than the other, but the picture remains unclear because model-free
and model-based processes do not appear to be neatly separated in the brain. Many
questions remain unanswered. Perhaps most intriguing is evidence that the hippocampus,
a structure traditionally associated with spatial navigation and memory, appears to be
involved in simulating possible future courses of action as part of an animal’s decision-
making process. This suggests that it is part of a system that uses an environment model
for planning.

Reinforcement learning theory is also inﬂuencing thinking about neural processes
underlying drug abuse. A model of some features of drug addiction is based on the reward
prediction error hypothesis. It proposes that an addicting stimulant, such as cocaine,
destabilizes TD learning to produce unbounded growth in the values of actions associated
with drug intake. This is far from a complete model of addiction, but it illustrates how
a computational perspective suggests theories that can be tested with further research.

The new ﬁeld of computational psychiatry similarly focuses on the use of computational
models, some derived from reinforcement learning, to better understand mental disorders.
This chapter only touched the surface of how the neuroscience of reinforcement learning
and the development of reinforcement learning in computer science and engineering
have inﬂuenced one another. Most features of reinforcement learning algorithms owe
their design to purely computational considerations, but some have been inﬂuenced by

hypotheses about neural learning mechanisms. Remarkably, as experimental data has
accumulated about the brain’s reward processes, many of the purely computationally-
motivated features of reinforcement learning algorithms are turning out to be consistent
with neuroscience data. Other features of computational reinforcement learning, such as
eligibility traces and the ability of teams of reinforcement learning agents to learn to act
collectively under the inﬂuence of a globally-broadcast reinforcement signal, may also

turn out to parallel experimental data as neuroscientists continue to unravel the neural
basis of reward-based animal learning and behavior.

Bibliographical and Historical Remarks 413
The number of publications treating parallels between the neuroscience of learning and
decision making and the approach to reinforcement learning presented in this book is
enormous. We can cite only a small selection. Niv (2009), Dayan and Niv (2008), Gimcher
(2011), Ludvig, Bellemare, and Pearson (2011), and Shah (2012) are good places to start.
Together with economics, evolutionary biology, and mathematical psychology, reinforce-
ment learning theory is helping to formulate quantitative models of the neural mechanisms
of choice in humans and non-human primates. With its focus on learning, this chapter
only lightly touches upon the neuroscience of decision making. Glimcher (2003) introduced
the ﬁeld of “neuroeconomics,” in which reinforcement learning contributes to the study
of the neural basis of decision making from an economics perspective. See also Glimcher

and Fehr (2013). The text on computational and mathematical modeling in neuroscience
by Dayan and Abbott (2001) includes reinforcement learning’s role in these approaches.
Sterling and Laughlin (2015) examined the neural basis of learning in terms of general
design principles that enable ecient adaptive behavior.
15.1 There are many good expositions of basic neuroscience. Kandel, Schwartz, Jessell,
Siegelbaum, and Hudspeth (2013) is an authoritative and very comprehensive
source.
15.2 Berridge and Kringelbach (2008) reviewed the neural basis of reward and pleasure,

source.
pointing out that reward processing has many dimensions and involves many
neural systems. Space prevents discussion of the inﬂuential research of Berridge
and Robinson (1998), who distinguish between the hedonic impact of a stimulus,
which they call “liking,” and the motivational e↵ect, which they call “wanting.”
Hare, O’Doherty, Camerer, Schultz, and Rangel (2008) examined the neural basis
of value-related signals from an economic perspective, distinguishing between

goal values, decision values, and prediction errors. Decision value is goal value
minus action cost. See also Rangel, Camerer, and Montague (2008), Rangel and
Hare (2010), and Peters and B¨ uchel (2010).
15.3 The reward prediction error hypothesis of dopamine neuron activity is most
prominently discussed by Schultz, Dayan, and Montague (1997). The hypothesis
was ﬁrst explicitly put forward by Montague, Dayan, and Sejnowski (1996). As
they stated the hypothesis, it referred to reward prediction errors (RPEs) but

not speciﬁcally to TD errors; however, their development of the hypothesis made
it clear that they were referring to TD errors. The earliest recognition of the
TD-error/dopamine connection of which we are aware is that of Montague, Dayan,
Nowlan, Pouget, and Sejnowski (1993), who proposed a TD-error-modulated
Hebbian learning rule motivated by results on dopamine signaling from Schultz’s
group. The connection was also pointed out in an abstract by Quartz, Dayan,
Montague, and Sejnowski (1992). Montague and Sejnowski (1994) emphasized

the importance of prediction in the brain and outlined how predictive Hebbian
learning modulated by TD errors could be implemented via a di↵use neuromod-
ulatory system, such as the dopamine system. Friston, Tononi, Reeke, Sporns,

414 Chapter 15: Neuroscience
and Edelman (1994) presented a model of value-dependent learning in the brain
in which synaptic changes are mediated by a TD-like error provided by a global
neuromodulatory signal (although they did not single out dopamine). Mon-
tague, Dayan, Person, and Sejnowski (1995) presented a model of honeybee
foraging using the TD error. The model is based on research by Hammer, Men-
zel, and colleagues (Hammer and Menzel, 1995; Hammer, 1997) showing that
the neuromodulator octopamine acts as a reinforcement signal in the honeybee.
Montague et al. (1995) pointed out that dopamine likely plays a similar role
in the vertebrate brain. Barto (1995a) related the actor–critic architecture to
basal-ganglionic circuits and discussed the relationship between TD learning
and the main results from Schultz’s group. Houk, Adams, and Barto (1995)
suggested how TD learning and the actor–critic architecture might map onto the
anatomy, physiology, and molecular mechanism of the basal ganglia. Doya and

Sejnowski (1998) extended their earlier paper on a model of birdsong learning
(Doya and Sejnowski, 1995) by including a TD-like error identiﬁed with dopamine
to reinforce the selection of auditory input to be memorized. O’Reilly and Frank
(2006) and O’Reilly, Frank, Hazy, and Watz (2007) argued that phasic dopamine
signals are RPEs but not TD errors. In support of their theory they cited results
with variable interstimulus intervals that do not match predictions of a simple
TD model, as well as the observation that higher-order conditioning beyond

second-order conditioning is rarely observed, while TD learning is not so limited.
Dayan and Niv (2008) discussed “the good, the bad, and the ugly” of how
reinforcement learning theory and the reward prediction error hypothesis align
with experimental data. Glimcher (2011) reviewed the empirical ﬁndings that
support the reward prediction error hypothesis and emphasized the signiﬁcance
of the hypothesis for contemporary neuroscience.
15.4 Graybiel (2000) is a brief primer on the basal ganglia. The experiments mentioned

that involve optogenetic activation of dopamine neurons were conducted by Tsai,
Zhang, Adamantidis, Stuber, Bonci, de Lecea, and Deisseroth (2009), Steinberg,
Keiﬂin, Boivin, Witten, Deisseroth, and Janak (2013), and Claridge-Chang,
Roorda, Vrontou, Sjulson, Li, Hirsh, and Miesenb¨ ock (2009). Fiorillo, Yun, and
Song (2013), Lammel, Lim, and Malenka (2014), and Saddoris, Cacciapaglia,
Wightmman, and Carelli (2015) are among studies showing that the signaling
properties of dopamine neurons are specialized for di↵erent target regions. RPE-

signaling neurons may belong to one among multiple populations of dopamine
neurons having di↵erent targets and subserving di↵erent functions. Eshel, Tian,
Bukwich, and Uchida (2016) found homogeneity of reward prediction error
responses of dopamine neurons in the lateral VTA during classical conditioning
in mice, though their results do not rule out response diversity across wider areas.
Gershman, Pesaran, and Daw (2009) studied reinforcement learning tasks that
can be decomposed into independent components with separate reward signals,

ﬁnding evidence in human neuroimaging data suggesting that the brain exploits
this kind of structure.

Bibliographical and Historical Remarks 415
15.5 Schultz’s 1998 survey article is a good entr´ ee into the very extensive literature
on reward predicting signaling of dopamine neurons. Berns, McClure, Pagnoni,
and Montague (2001), Breiter, Aharon, Kahneman, Dale, and Shizgal (2001),
Pagnoni, Zink, Montague, and Berns (2002), and O’Doherty, Dayan, Friston,
Critchley, and Dolan (2003) described functional brain imaging studies supporting
the existence of signals like TD errors in the human brain.
15.6 This section roughly follows Barto (1995a) in explaining how TD errors mimic the
main results from Schultz’s group on the phasic responses of dopamine neurons.
15.7 This section is largely based on Takahashi, Schoenbaum, and Niv (2008) and Niv
(2009). To the best of our knowledge, Barto (1995a) and Houk, Adams, and Barto
(1995) ﬁrst speculated about possible implementations of actor–critic algorithms
in the basal ganglia. On the basis of functional magnetic resonance imaging of

human subjects while engaged in instrumental conditioning, O’Doherty, Dayan,
Schultz, Deichmann, Friston, and Dolan (2004) suggested that the actor and
the critic are most likely located respectively in the dorsal and ventral striatum.
Gershman, Moustafa, and Ludvig (2014) focused on how time is represented in
reinforcement learning models of the basal ganglia, discussing evidence for, and
implications of, various computational approaches to time representation.

The hypothetical neural implementation of the actor–critic architecture described
in this section includes very little detail about known basal ganglia anatomy and
physiology. In addition to the more detailed hypothesis of Houk, Adams, and
Barto (1995), a number of other hypotheses include more speciﬁc connections to
anatomy and physiology and are claimed to explain additional data. These include
hypotheses proposed by Suri and Schultz (1998, 1999), Brown, Bullock, and
Grossberg (1999), Contreras-Vidal and Schultz (1999), Suri, Bargas, and Arbib

(2001), O’Reilly and Frank (2006), and O’Reilly, Frank, Hazy, and Watz (2007).
Joel, Niv, and Ruppin (2002) critically evaluated the anatomical plausibility of
several of these models and present an alternative intended to accommodate
some neglected features of basal ganglionic circuitry.
15.8 The actor learning rule discussed here is more complicated than the one in
the early actor–critic network of Barto et al. (1983). Actor-unit eligibility
traces in that network were traces of just At⇥x(St) instead of the full ( At

⇡(At|St,✓))x(St). That work did not beneﬁt from the policy-gradient theory
presented in Chapter 13 or the contributions of Williams (1986, 1992), who
showed how an ANN of Bernoulli-logistic units could implement a policy-gradient
method.
Reynolds and Wickens (2002) proposed a three-factor rule for synaptic plasticity
in the corticostriatal pathway in which dopamine modulates changes in corti-
costriatal synaptic ecacy. They discussed the experimental support for this
kind of learning rule and its possible molecular basis. The deﬁnitive demon-

method.
stration of spike-timing-dependent plasticity (STDP) is attributed to Markram,
L¨ ubke, Frotscher, and Sakmann (1997), with evidence from earlier experiments

416 Chapter 15: Neuroscience
by Levy and Steward (1983) and others that the relative timing of pre- and
postsynaptic spikes is critical for inducing changes in synaptic ecacy. Rao
and Sejnowski (2001) suggested how STDP could be the result of a TD-like
mechanism at synapses with non-contingent eligibility traces lasting about 10
milliseconds. Dayan (2002) commented that this would require an error as
in Sutton and Barto’s (1981a) early model of classical conditioning and not a
true TD error. Representative publications from the extensive literature on
reward-modulated STDP are Wickens (1990), Reynolds and Wickens (2002), and
Calabresi, Picconi, Tozzi and Di Filippo (2007). Pawlak and Kerr (2008) showed
that dopamine is necessary to induce STDP at the corticostriatal synapses of
medium spiny neurons. See also Pawlak, Wickens, Kirkwood, and Kerr (2010).
Yagishita, Hayashi-Takagi, Ellis-Davies, Urakubo, Ishii, and Kasai (2014) found
that dopamine promotes spine enlargement of the medium spiny neurons of mice

only during a time window of from 0.3 to 2 seconds after STDP stimulation.
Izhikevich (2007) proposed and explored the idea of using STDP timing condi-
tions to trigger contingent eligibility traces. Fr´ emaux, Sprekeler, and Gerstner
(2010) proposed theoretical conditions for successful learning by rules based on
reward-modulated STDP.
15.9 Klopf’s hedonistic neuron hypothesis (Klopf 1972, 1982) inspired our actor–critic
algorithm implemented as an ANN with a single neuron-like unit, called the

actor unit, implementing a Law-of-E↵ect-like learning rule (Barto, Sutton, and
Anderson, 1983). Ideas related to Klopf’s synaptically-local eligibility have been
proposed by others. Crow (1968) proposed that changes in the synapses of
cortical neurons are sensitive to the consequences of neural activity. Emphasizing
the need to address the time delay between neural activity and its consequences
in a reward-modulated form of synaptic plasticity, he proposed a contingent form

of eligibility, but associated with entire neurons instead of individual synapses.
According to his hypothesis, a wave of neuronal activity
leads to a short-term change in the cells involved in the wave such that they
are picked out from a background of cells not so activated. ... such cells are
rendered sensitive by the short-term change to a reward signal ... in such
a way that if such a signal occurs before the end of the decay time of the
change the synaptic connexions between the cells are made more e↵ective.
(Crow, 1968)

Crow argued against previous proposals that reverberating neural circuits play
this role by pointing out that the e↵ect of a reward signal on such a circuit would
“...establish the synaptic connexions leading to the reverberation (that is to say,
those involved in activity at the time of the reward signal) and not those on the
path which led to the adaptive motor output.” Crow further postulated that
reward signals are delivered via a “distinct neural ﬁber system,” presumably the
one into which Olds and Milner (1954) tapped, that would transform synaptic

connections “from a short into a long-term form.”

Bibliographical and Historical Remarks 417
In another farsighted hypothesis, Miller proposed a Law-of-E↵ect-like learning
rule that includes synaptically-local contingent eligibility traces:
... it is envisaged that in a particular sensory situation neurone B, by chance,
ﬁres a ‘meaningful burst’ of activity, which is then translated into motor acts,
which then change the situation. It must be supposed that the meaningful
burst has an inﬂuence, at the neuronal level , on all of its own synapses which
are active at the time ... thereby making a preliminary selection of the
synapses to be strengthened, though not yet actually strengthening them.
...The strengthening signal ... makes the ﬁnal selection ... and accomplishes
the deﬁnitive change in the appropriate synapses. (Miller, 1981, p. 81)
Miller’s hypothesis also included a critic-like mechanism, which he called a “sen-
sory analyzer unit,” that worked according to classical conditioning principles to

provide reinforcement signals to neurons so that they would learn to move from
lower- to higher-valued states, thus anticipating the use of the TD error as a rein-
forcement signal in the actor–critic architecture. Miller’s idea not only parallels
Klopf’s (with the exception of its explicit invocation of a distinct “strengthening
signal”), it also anticipated the general features of reward-modulated STDP.
A related though di↵erent idea, which Seung (2003) called the “hedonistic

synapse,” is that synapses individually adjust the probability that they release
neurotransmitter in the manner of the Law of E↵ect: if reward follows release,
the release probability increases, and decreases if reward follows failure to release.
This is essentially the same as the learning scheme Minsky used in his 1954
Princeton Ph.D. dissertation, where he called the synapse-like learning element
a SNARC (Stochastic Neural-Analog Reinforcement Calculator). Contingent

eligibility is involved in these ideas too, although it is contingent on the activity
of an individual synapse instead of the postsynaptic neuron. Also related is the
proposal of Unnikrishnan and Venugopal (1994) that uses the correlation-based
method of Harth and Tzanakou (1974) to adjust ANN weights.
Frey and Morris (1997) proposed the idea of a “synaptic tag” for the induction
of long-lasting strengthening of synaptic ecacy. Though not unlike Klopf’s
eligibility, their tag was hypothesized to consist of a temporary strengthening of a

synapse that could be transformed into a long-lasting strengthening by subsequent
neuron activation. The model of O’Reilly and Frank (2006) and O’Reilly, Frank,
Hazy, and Watz (2007) uses working memory to bridge temporal intervals instead
of eligibility traces. Wickens and Kotter (1995) discuss possible mechanisms
for synaptic eligibility. He, Huertas, Hong, Tie, Hell, Shouval, Kirkwood (2015)
provide evidence supporting the existence of contingent eligibility traces in
synapses of cortical neurons with time courses like those of the eligibility traces

Klopf postulated.
The metaphor of a neuron using a learning rule related to bacterial chemotaxis was
discussed by Barto (1989). Koshland’s extensive study of bacterial chemotaxis
was in part motivated by similarities between features of bacteria and features of
neurons (Koshland, 1980). See also Berg (1975). Shimansky (2009) proposed a

418 Chapter 15: Neuroscience
synaptic learning rule somewhat similar to Seung’s mentioned above in which
each synapse individually acts like a chemotactic bacterium. In this case a
collection of synapses “swims” toward attractants in the high-dimensional space
of synaptic weight values. Montague, Dayan, Person, and Sejnowski (1995)
proposed a chemotactic-like model of the bee’s foraging behavior involving the
neuromodulator octopamine.
15.10 Research on the behavior of reinforcement learning agents in team and game
problems has a long history roughly occurring in three phases. To the best
of our knowledge, the ﬁrst phase began with investigations by the Russian
mathematician and physicist M. L. Tsetlin. A collection of his work was published
as Tsetlin (1973) after his death in 1966. Our Sections 1.7 and 4.8 refer to his
study of learning automata in connection to bandit problems. The Tsetlin
collection also includes studies of learning automata in team and game problems,

which led to later work in this area using stochastic learning automata as
described by Narendra and Thathachar (1974, 1989), Viswanathan and Narendra
(1974), Lakshmivarahan and Narendra (1982), Narendra and Wheeler (1983), and
Thathachar and Sastry (2002). Thathachar and Sastry (2011) is a more recent
comprehensive account. These studies were mostly restricted to non-associative
learning automata, meaning that they did not address associative, or contextual,
bandit problems (Section 2.9).

The second phase began with the extension of learning automata to the associative,
or contextual, case. Barto, Sutton, and Brouwer (1981) and Barto and Sutton
(1981b) experimented with associative stochastic learning automata in single-
layer ANNs to which a global reinforcement signal was broadcast. The learning
algorithm was an associative extension of the Alopex algorithm of Harth and
Tzanakou (1974). Barto et al. called neuron-like elements implementing this

kind of learning associative search elements (ASEs). Barto and Anandan (1985)
introduced an associative reinforcement learning algorithm called the associative
reward-penalty (ARP) algorithm. They proved a convergence result by combining
theory of stochastic learning automata with theory of pattern classiﬁcation. Barto
(1985, 1986) and Barto and Jordan (1987) described results with teams of A RP
units connected into multi-layer ANNs, showing that they could learn nonlinear

functions, such as XOR and others, with a globally-broadcast reinforcement signal.
Barto (1985) extensively discussed this approach to ANNs and how this type of
learning rule is related to others in the literature at that time. Williams (1992)
mathematically analyzed and broadened this class of learning rules and related
their use to the error backpropagation method for training multilayer ANNs.
Williams (1988) described several ways that backpropagation and reinforcement

learning can be combined for training ANNs. Williams (1992) showed that a
special case of the A RPalgorithm is a REINFORCE algorithm, although better
results were obtained with the general A RPalgorithm (Barto, 1985).

Bibliographical and Historical Remarks 419
The third phase of interest in teams of reinforcement learning agents was inﬂu-
enced by increased understanding of the role of dopamine as a widely broadcast
neuromodulator and speculation about the existence of reward-modulated STDP.
Much more so than earlier research, this research considers details of synaptic
plasticity and other constraints from neuroscience. Publications include the
following (chronologically and alphabetically): Bartlett and Baxter (1999, 2000),
Xie and Seung (2004), Baras and Meir (2007), Farries and Fairhall (2007), Florian
(2007), Izhikevich (2007), Pecevski, Maass, and Legenstein (2008), Legenstein,
Pecevski, and Maass (2008), Kolodziejski, Porr, and W¨ org¨ otter (2009), Urbanczik
and Senn (2009), and Vasilaki, Fr´ emaux, Urbanczik, Senn, and Gerstner (2009).
Now´ e, Vrancx, and De Hauwere (2012) reviewed more recent developments in
the wider ﬁeld of multi-agent reinforcement learning

15.11 Yin and Knowlton (2006) reviewed ﬁndings from outcome-devaluation experi-
ments with rodents supporting the view that habitual and goal-directed behavior
(as psychologists use the phrase) are respectively most associated with process-
ing in the dorsolateral striatum (DLS) and the dorsomedial striatum (DMS).
Results of functional imaging experiments with human subjects in the outcome-
devaluation setting by Valentin, Dickinson, and O’Doherty (2007) suggest that
the orbitofrontal cortex (OFC) is an important component of goal-directed choice.

Single unit recordings in monkeys by Padoa-Schioppa and Assad (2006) support
the role of the OFC in encoding values guiding choice behavior. Rangel, Camerer,
and Montague (2008) and Rangel and Hare (2010) reviewed ﬁndings from the
perspective of neuroeconomics about how the brain makes goal-directed decisions.
Pezzulo, van der Meer, Lansink, and Pennartz (2014) reviewed the neuroscience
of internally generated sequences and presented a model of how these mecha-
nisms might be components of model-based planning. Daw and Shohamy (2008)

proposed that while dopamine signaling connects well to habitual, or model-free,
behavior, other processes are involved in goal-directed, or model-based, behavior.
Data from experiments by Bromberg-Martin, Matsumoto, Hong, and Hikosaka
(2010) indicate that dopamine signals contain information pertinent to both
habitual and goal-directed behavior. Doll, Simon, and Daw (2012) argued that
there may not a clear separation in the brain between mechanisms that subserve
habitual and goal-directed learning and choice.

15.12 Keiﬂin and Janak (2015) reviewed connections between TD errors and addiction.
Nutt, Lingford-Hughes, Erritzoe, and Stokes (2015) critically evaluated the
hypothesis that addiction is due to a disorder of the dopamine system. Montague,
Dolan, Friston, and Dayan (2012) outlined the goals and early e↵orts in the ﬁeld
of computational psychiatry, and Adams, Huys, and Roiser (2015) reviewed more
recent progress.

Chapter 16
Applications and Case Studies
In this chapter we present a few case studies of reinforcement learning. Several of these
are substantial applications of potential economic signiﬁcance. One, Samuel’s checkers
player, is primarily of historical interest. Our presentations are intended to illustrate some
of the trade-o↵s and issues that arise in real applications. For example, we emphasize how
domain knowledge is incorporated into the formulation and solution of the problem. We
also highlight the representation issues that are so often critical to successful applications.
The algorithms used in some of these case studies are substantially more complex than
those we have presented in the rest of the book. Applications of reinforcement learning
are still far from routine and typically require as much art as science. Making applications
easier and more straightforward is one of the goals of current research in reinforcement
learning.
16.1 TD-Gammon

learning.
One of the most impressive applications of reinforcement learning to date is that by
Gerald Tesauro to the game of backgammon (Tesauro, 1992, 1994, 1995, 2002). Tesauro’s
program, TD-Gammon , required little backgammon knowledge, yet learned to play
extremely well, near the level of the world’s strongest grandmasters. The learning
algorithm in TD-Gammon was a straightforward combination of the TD( ) algorithm
and nonlinear function approximation using a multilayer artiﬁcial neural network (ANN)
trained by backpropagating TD errors.

Backgammon is a major game in the sense that it is played throughout the world,
with numerous tournaments and regular world championship matches. It is in part a
game of chance, and it is a popular vehicle for waging signiﬁcant sums of money. There
are probably more professional backgammon players than there are professional chess
players. The game is played with 15 white and 15 black pieces on a board of 24 locations,
called points . To the right on the next page is shown a typical position early in the game,

seen from the perspective of the white player. White here has just rolled the dice and
obtained a 5 and a 2. This means that he can move one of his pieces 5 steps and one
421

422 Chapter 16: Applications and Case Studies
white pieces move    counterclockwise
123456789101112181716151413192021222324
    black pieces move clockwiseA backgammon position(possibly the same piece) 2 steps. For
example, he could move two pieces from
the 12 point, one to the 17 point, and
one to the 14 point. White’s objective is
to advance all of his pieces into the last
quadrant (points 19–24) and then o↵
the board. The ﬁrst player to remove
all his pieces wins. One complication
is that the pieces interact as they pass
each other going in di↵erent directions.
For example, if it were black’s move, he
could use the dice roll of 2 to move a
piece from the 24 point to the 22 point,
“hitting” the white piece there. Pieces that have been hit are placed on the “bar” in the
middle of the board (where we already see one previously hit black piece), from whence
they reenter the race from the start. However, if there are two pieces on a point, then

the opponent cannot move to that point; the pieces are protected from being hit. Thus,
white cannot use his 5–2 dice roll to move either of his pieces on the 1 point, because
their possible resulting points are occupied by groups of black pieces. Forming contiguous
blocks of occupied points to block the opponent is one of the elementary strategies of the
game.
Backgammon involves several further complications, but the above description gives

game.
the basic idea. With 30 pieces and 24 possible locations (26, counting the bar and
o↵-the-board) it should be clear that the number of possible backgammon positions is
enormous, far more than the number of memory elements one could have in any physically
realizable computer. The number of moves possible from each position is also large. For a
typical dice roll there might be 20 di↵erent ways of playing. In considering future moves,
such as the response of the opponent, one must consider the possible dice rolls as well.

The result is that the game tree has an e↵ective branching factor of about 400. This is
far too large to permit e↵ective use of the conventional heuristic search methods that
have proved so e↵ective in games like chess and checkers.
On the other hand, the game is a good match to the capabilities of TD learning
methods. Although the game is highly stochastic, a complete description of the game’s
state is available at all times. The game evolves over a sequence of moves and positions
until ﬁnally ending in a win for one player or the other, ending the game. The outcome

can be interpreted as a ﬁnal reward to be predicted. On the other hand, the theoretical
results we have described so far cannot be usefully applied to this task. The number of
states is so large that a lookup table cannot be used, and the opponent is a source of
uncertainty and time variation.
TD-Gammon used a nonlinear form of TD( ). The estimated value, ˆv(s,w), of any
state (board position) swas meant to estimate the probability of winning starting from

state s. To achieve this, rewards were deﬁned as zero for all time steps except those
on which the game is won. To implement the value function, TD-Gammon used a
standard multilayer ANN, much like that shown to the right on the next page. (The real

The gradient in this equation can be computed eciently by thebackpropagation procedure. For the backgammon application, in which=1and the reward is always zero except upon winning, the TD error portion of thelearning rule is usually just ˆv(St+1,w)ˆv(St,w), as suggested in Figure 15.2.To apply the learning rule we need a source of backgammon games. Tesauroobtained an unending sequence of games by playing his learning backgammonplayer against itself. To choose its moves, TD-Gammon considered each of the20 or so ways it could play its dice roll and the corresponding positions thatwould result. The resulting positions areafterstatesas discussed in Section 6.8.The network was consulted to estimate each of their values. The move wasthen selected that would lead to the position with the highest estimated value.Continuing in this way, with TD-Gammon making the moves for both sides,it was possible to easily generate large numbers of backgammon games. Eachgame was treated as an episode, with the sequence of

the20 or so ways it could play its dice roll and the corresponding positions thatwould result. The resulting positions areafterstatesas discussed in Section 6.8.The network was consulted to estimate each of their values. The move wasthen selected that would lead to the position with the highest estimated value.Continuing in this way, with TD-Gammon making the moves for both sides,it was possible to easily generate large numbers of backgammon games. Eachgame was treated as an episode, with the sequence of positions acting asthe states,S0,S1,S2,.... Tesauro applied the nonlinear TD rule (15.1) fullyincrementally, that is, after each individual move.The weights of the network were set initially to small random values. Theinitial evaluations were thus entirely arbitrary. Since the moves were selectedon the basis of these evaluations, the initial moves were inevitably poor, andthe initial games often lasted hundreds or thousands of moves before one sideor the other won, almost by accident. After a few dozen games

acting asthe states,S0,S1,S2,.... Tesauro applied the nonlinear TD rule (15.1) fullyincrementally, that is, after each individual move.The weights of the network were set initially to small random values. Theinitial evaluations were thus entirely arbitrary. Since the moves were selectedon the basis of these evaluations, the initial moves were inevitably poor, andthe initial games often lasted hundreds or thousands of moves before one sideor the other won, almost by accident. After a few dozen games however,performance improved rapidly.After playing about 300,000 games against itself, TD-Gammon 0.0 as de-scribed above learned to play approximately as well as the best previousbackgammon computer programs. This was a striking result because all theprevious high-performance computer programs had used extensive backgam-mon knowledge. For example, the reigning champion program at the timewas, arguably,Neurogammon, another program written by Tesauro that useda neural network but not TD learning. Neurogammon’s

improved rapidly.After playing about 300,000 games against itself, TD-Gammon 0.0 as de-scribed above learned to play approximately as well as the best previousbackgammon computer programs. This was a striking result because all theprevious high-performance computer programs had used extensive backgam-mon knowledge. For example, the reigning champion program at the timewas, arguably,Neurogammon, another program written by Tesauro that useda neural network but not TD learning. Neurogammon’s network was trainedon a large training corpus of exemplary moves provided by backgammon ex-perts, and, in addition, started with a set of features specially crafted forTD error15.1. TD-GAMMON263gation algorithm (Rumelhart, Hinton, and Williams, 1986). Recall that thegeneral update rule for this case iswt+1=wt+hRt+1+ˆv(St+1,wt)ˆv(St,wt)iet,(15.1)wherewtis the vector of all modiﬁable parameters (in this case, the weightsof the network) andetis a vector of eligibility traces, one for each componentofwt, updated

improved rapidly.After playing about 300,000 games against itself, TD-Gammon 0.0 as de-scribed above learned to play approximately as well as the best previousbackgammon computer programs. This was a striking result because all theprevious high-performance computer programs had used extensive backgam-mon knowledge. For example, the reigning champion program at the timewas, arguably,Neurogammon, another program written by Tesauro that useda neural network but not TD learning. Neurogammon’s network was trainedon a large training corpus of exemplary moves provided by backgammon ex-perts, and, in addition, started with a set of features specially crafted forhidden units(40-80)Figure 16.1: The TD-Gammon ANNnetwork had two additional units in its

ﬁnal layer to estimate the probability of
each player’s winning in a special way
called a “gammon” or “backgammon.”)
The network consisted of a layer of input
units, a layer of hidden units, and a ﬁnal
output unit. The input to the network
was a representation of a backgammon
position, and the output was an estimate
of the value of that position.
In the ﬁrst version of TD-Gammon,
TD-Gammon 0.0, backgammon posi-
tions were represented to the network
in a relatively direct way that involved
little backgammon knowledge. It did,
however, involve substantial knowledge of how ANNs work and how information is best
presented to them. It is instructive to note the exact representation Tesauro chose. There
were a total of 198 input units to the network. For each point on the backgammon board,
four units indicated the number of white pieces on the point. If there were no white
pieces, then all four units took on the value zero. If there was one piece, then the ﬁrst

unit took on the value 1. This encoded the elementary concept of a “blot,” i.e., a piece
that can be hit by the opponent. If there were two or more pieces, then the second unit
was set to 1. This encoded the basic concept of a “made point” on which the opponent
cannot land. If there were exactly three pieces on the point, then the third unit was set
to 1. This encoded the basic concept of a “single spare,” i.e., an extra piece in addition
to the two pieces that made the point. Finally, if there were more than three pieces, the

fourth unit was set to a value proportionate to the number of additional pieces beyond
three. Letting ndenote the total number of pieces on the point, if n>3, then the fourth
unit took on the value ( n3)/2. This encoded a linear representation of “multiple spares”
at the given point.
With four units for white and four for black at each of the 24 points, that made a
total of 192 units. Two additional units encoded the number of white and black pieces on
the bar (each took the value n/2, where nis the number of pieces on the bar), and two

more encoded the number of black and white pieces already successfully removed from
the board (these took the value n/15, where nis the number of pieces already borne
o↵). Finally, two units indicated in a binary fashion whether it was white’s or black’s
turn to move. The general logic behind these choices should be clear. Basically, Tesauro
tried to represent the position in a straightforward way, while keeping the number of
units relatively small. He provided one unit for each conceptually distinct possibility that

seemed likely to be relevant, and he scaled them to roughly the same range, in this case
between 0 and 1.
Given a representation of a backgammon position, the network computed its estimated
value in the standard way. Corresponding to each connection from an input unit to a
hidden unit was a real-valued weight. Signals from each input unit were multiplied by

424 Chapter 16: Applications and Case Studies
their corresponding weights and summed at the hidden unit. The output, h(j), of hidden
unit jwas a nonlinear sigmoid function of the weighted sum:
h(j)= X
iwijxi!
=1
1+eP
iwijxi,
where xiis the value of the ith input unit and wijis the weight of its connection to the
jth hidden unit (all the weights in the network together make up the parameter vector w).
The output of the sigmoid is always between 0 and 1, and has a natural interpretation as
a probability based on a summation of evidence. The computation from hidden units
to the output unit was entirely analogous. Each connection from a hidden unit to the
output unit had a separate weight. The output unit formed the weighted sum and then
passed it through the same sigmoid nonlinearity.
TD-Gammon used the semi-gradient form of the TD( ) algorithm described in Sec-
tion 12.2, with the gradients computed by the error backpropagation algorithm (Rumel-

always zero except upon winning, the TD error portion of the learning rule is usually
just ˆv(St+1,w)ˆv(St,w), as suggested in Figure 16.1.
To apply the learning rule we need a source of backgammon games. Tesauro obtained
an unending sequence of games by playing his learning backgammon player against itself.
To choose its moves, TD-Gammon considered each of the 20 or so ways it could play
its dice roll and the corresponding positions that would result. The resulting positions
areafterstates as discussed in Section 6.8. The network was consulted to estimate each
of their values. The move was then selected that would lead to the position with the
highest estimated value. Continuing in this way, with TD-Gammon making the moves
for both sides, it was possible to easily generate large numbers of backgammon games.
Each game was treated as an episode, with the sequence of positions acting as the states,
S0,S1,S2,.... Tesauro applied the nonlinear TD rule (16.1) fully incrementally, that is,

after each individual move.
The weights of the network were set initially to small random values. The initial
evaluations were thus entirely arbitrary. Because the moves were selected on the basis
of these evaluations, the initial moves were inevitably poor, and the initial games often
lasted hundreds or thousands of moves before one side or the other won, almost by
accident. After a few dozen games however, performance improved rapidly.
After playing about 300,000 games against itself, TD-Gammon 0.0 as described above

learned to play approximately as well as the best previous backgammon computer

16.1. TD-Gammon 425
programs. This was a striking result because all the previous high-performance computer
programs had used extensive backgammon knowledge. For example, the reigning champion
program at the time was, arguably, Neurogammon , another program written by Tesauro
that used an ANN but not TD learning. Neurogammon’s network was trained on a
large training corpus of exemplary moves provided by backgammon experts, and, in
addition, started with a set of features specially crafted for backgammon. Neurogammon
was a highly tuned, highly e↵ective backgammon program that decisively won the
World Backgammon Olympiad in 1989. TD-Gammon 0.0, on the other hand, was
constructed with essentially zero backgammon knowledge. That it was able to do as
well as Neurogammon and all other approaches is striking testimony to the potential of
self-play learning methods.
The tournament success of TD-Gammon 0.0 with zero expert backgammon knowledge

suggested an obvious modiﬁcation: add the specialized backgammon features but keep
the self-play TD learning method. This produced TD-Gammon 1.0. TD-Gammon 1.0 was
clearly substantially better than all previous backgammon programs and found serious
competition only among human experts. Later versions of the program, TD-Gammon 2.0
(40 hidden units) and TD-Gammon 2.1 (80 hidden units), were augmented with a selective
two-ply search procedure. To select moves, these programs looked ahead not just to the
positions that would immediately result, but also to the opponent’s possible dice rolls

and moves. Assuming the opponent always took the move that appeared immediately
best for him, the expected value of each candidate move was computed and the best
was selected. To save computer time, the second ply of search was conducted only for
candidate moves that were ranked highly after the ﬁrst ply, about four or ﬁve moves on
average. Two-ply search a↵ected only the moves selected; the learning process proceeded
exactly as before. The ﬁnal versions of the program, TD-Gammon 3.0 and 3.1, used 160

hidden units and a selective three-ply search. TD-Gammon illustrates the combination
of learned value functions and decision-time search as in heuristic search and MCTS
methods. In follow-on work, Tesauro and Galperin (1997) explored trajectory sampling
methods as an alternative to full-width search, which reduced the error rate of live play
by large numerical factors (4x–6x) while keeping the think time reasonable at ⇠5–10
seconds per move.

During the 1990s, Tesauro was able to play his programs in a signiﬁcant number of
games against world-class human players. A summary of the results is given in Table 16.1.
Program Hidden Training Opponents Results
Units Games
TD-Gammon 0.0 40 300,000 other programs tied for best
TD-Gammon 1.0 80 300,000 Robertie, Magriel, ... 13 pts / 51 games
TD-Gammon 2.0 40 800,000 various Grandmasters 7 pts / 38 games
TD-Gammon 2.1 80 1,500,000 Robertie 1 pt / 40 games
TD-Gammon 3.0 80 1,500,000 Kazaros +6 pts / 20 games
Table 16.1: Summary of TD-Gammon Results

426 Chapter 16: Applications and Case Studies
Based on these results and analyses by backgammon grandmasters (Robertie, 1992; see
Tesauro, 1995), TD-Gammon 3.0 appeared to play at close to, or possibly better than,
the playing strength of the best human players in the world. Tesauro reported in a
subsequent article (Tesauro, 2002) the results of an extensive rollout analysis of the move
decisions and doubling decisions of TD-Gammon relative to top human players. The
conclusion was that TD-Gammon 3.1 had a “lopsided advantage” in piece-movement
decisions, and a “slight edge” in doubling decisions, over top humans.
TD-Gammon had a signiﬁcant impact on the way the best human players play the
game. For example, it learned to play certain opening positions di↵erently than was
the convention among the best human players. Based on TD-Gammon’s success and
further analysis, the best human players now play these positions as TD-Gammon does

(Tesauro, 1995). The impact on human play was greatly accelerated when several other
self-teaching ANN backgammon programs inspired by TD-Gammon, such as Jellyﬁsh,
Snowie, and GNUBackgammon, became widely available. These programs enabled wide
dissemination of new knowledge generated by the ANNs, resulting in great improvements
in the overall caliber of human tournament play (Tesauro, 2002).
16.2 Samuel’s Checkers Player
An important precursor to Tesauro’s TD-Gammon was the seminal work of Arthur Samuel

(1959, 1967) in constructing programs for learning to play checkers. Samuel was one of
the ﬁrst to make e↵ective use of heuristic search methods and of what we would now call
temporal-di↵erence learning. His checkers players are instructive case studies in addition
to being of historical interest. We emphasize the relationship of Samuel’s methods to
modern reinforcement learning methods and try to convey some of Samuel’s motivation
for using them.

Samuel ﬁrst wrote a checkers-playing program for the IBM 701 in 1952. His ﬁrst
learning program was completed in 1955 and was demonstrated on television in 1956.
Later versions of the program achieved good, though not expert, playing skill. Samuel
was attracted to game-playing as a domain for studying machine learning because games
are less complicated than problems “taken from life” while still allowing fruitful study of
how heuristic procedures and learning can be used together. He chose to study checkers

instead of chess because its relative simplicity made it possible to focus more strongly on
learning.
Samuel’s programs played by performing a lookahead search from each current position.
They used what we now call heuristic search methods to determine how to expand the
search tree and when to stop searching. The terminal board positions of each search were
evaluated, or “scored,” by a value function, or “scoring polynomial,” using linear function
approximation. In this and other respects Samuel’s work seems to have been inspired

learning.
by the suggestions of Shannon (1950). In particular, Samuel’s program was based on
Shannon’s minimax procedure to ﬁnd the best move from the current position. Working
backward through the search tree from the scored terminal positions, each position was
given the score of the position that would result from the best move, assuming that the
machine would always try to maximize the score, while the opponent would always try to

16.2. Samuel’s Checkers Player 427
minimize it. Samuel called this the “backed-up score” of the position. When the minimax
procedure reached the search tree’s root—the current position—it yielded the best move
under the assumption that the opponent would be using the same evaluation criterion,
shifted to its point of view. Some versions of Samuel’s programs used sophisticated search
control methods analogous to what are known as “alpha-beta” cuto↵s (e.g., see Pearl,
1984).
Samuel used two main learning methods, the simplest of which he called rote learning .
It consisted simply of saving a description of each board position encountered during play
together with its backed-up value determined by the minimax procedure. The result was
that if a position that had already been encountered were to occur again as a terminal
position of a search tree, the depth of the search was e↵ectively ampliﬁed because this
position’s stored value cached the results of one or more searches conducted earlier. One

initial problem was that the program was not encouraged to move along the most direct
path to a win. Samuel gave it a “a sense of direction” by decreasing a position’s value
a small amount each time it was backed up a level (called a ply) during the minimax
analysis. “If the program is now faced with a choice of board positions whose scores
di↵er only by the ply number, it will automatically make the most advantageous choice,
choosing a low-ply alternative if winning and a high-ply alternative if losing” (Samuel,

1959, p. 80). Samuel found this discounting-like technique essential to successful learning.
Rote learning produced slow but continual improvement that was most e↵ective for
opening and endgame play. His program became a “better-than-average novice” after
learning from many games against itself, a variety of human opponents, and from book
games in a supervised learning mode.
Rote learning and other aspects of Samuel’s work strongly suggest the essential idea
of temporal-di↵erence learning—that the value of a state should equal the value of

likely following states. Samuel came closest to this idea in his second learning method,
his “learning by generalization” procedure for modifying the parameters of the value
function. Samuel’s method was the same in concept as that used much later by Tesauro
in TD-Gammon. He played his program many games against another version of itself and
performed an update after each move. The idea of Samuel’s update is suggested by the
backup diagram in Figure 16.2. Each open circle represents a position where the program

moves next, an on-move position, and each solid circle represents a position where the
opponent moves next. An update was made to the value of each on-move position after a
move by each side, resulting in a second on-move position. The update was toward the
minimax value of a search launched from the second on-move position. Thus, the overall
e↵ect was that of a backing-up over one full move of real events and then a search over
possible events, as suggested by Figure 16.2. Samuel’s actual algorithm was signiﬁcantly

more complex than this for computational reasons, but this was the basic idea.
Samuel did not include explicit rewards. Instead, he ﬁxed the weight of the most
important feature, the piece advantage feature, which measured the number of pieces
the program had relative to how many its opponent had, giving higher weight to kings,
and including reﬁnements so that it was better to trade pieces when winning than when
losing. Thus, the goal of Samuel’s program was to improve its piece advantage, which in
checkers is highly correlated with winning.

428 Chapter 16: Applications and Case Studies
hypothetical eventsactual eventsbackupFigure 16.2: The backup diagram for Samuel’s checkers player.
However, Samuel’s learning method may have been missing an essential part of a sound
temporal-di↵erence algorithm. Temporal-di↵erence learning can be viewed as a way of
making a value function consistent with itself, and this we can clearly see in Samuel’s
method. But also needed is a way of tying the value function to the true value of the
states. We have enforced this via rewards and by discounting or giving a ﬁxed value to
the terminal state. But Samuel’s method included no rewards and no special treatment of
the terminal positions of games. As Samuel himself pointed out, his value function could
have become consistent merely by giving a constant value to all positions. He hoped to
discourage such solutions by giving his piece-advantage term a large, nonmodiﬁable weight.
But although this may decrease the likelihood of ﬁnding useless evaluation functions,

it does not prohibit them. For example, a constant function could still be attained by
setting the modiﬁable weights so as to cancel the e↵ect of the nonmodiﬁable one.
Because Samuel’s learning procedure was not constrained to ﬁnd useful evaluation
functions, it should have been possible for it to become worse with experience. In fact,
Samuel reported observing this during extensive self-play training sessions. To get the
program improving again, Samuel had to intervene and set the weight with the largest

absolute value back to zero. His interpretation was that this drastic intervention jarred
the program out of local optima, but another possibility is that it jarred the program out
of evaluation functions that were consistent but had little to do with winning or losing
the game.
Despite these potential problems, Samuel’s checkers player using the generalization
learning method approached “better-than-average” play. Fairly good amateur opponents

the game.
characterized it as “tricky but beatable” (Samuel, 1959). In contrast to the rote-learning
version, this version was able to develop a good middle game but remained weak in
opening and endgame play. This program also included an ability to search through sets of
features to ﬁnd those that were most useful in forming the value function. A later version
(Samuel, 1967) included reﬁnements in its search procedure, such as alpha-beta pruning,

16.3. Watson’s Daily-Double Wagering 429
extensive use of a supervised learning mode called “book learning,” and hierarchical
lookup tables called signature tables (Grith, 1966) to represent the value function
instead of linear function approximation. This version learned to play much better than
the 1959 program, though still not at a master level. Samuel’s checkers-playing program
was widely recognized as a signiﬁcant achievement in artiﬁcial intelligence and machine
learning.
16.3 Watson’s Daily-Double Wagering
IBM W atson1is the system developed by a team of IBM researchers to play the popular
TV quiz show Jeopardy! .2It gained fame in 2011 by winning ﬁrst prize in an exhibition
match against human champions. Although the main technical achievement demonstrated
by W atson was its ability to quickly and accurately answer natural language questions
over broad areas of general knowledge, its winning Jeopardy! performance also relied on

learning.
sophisticated decision-making strategies for critical parts of the game. Tesauro, Gondek,
Lechner, Fan, and Prager (2012, 2013) adapted Tesauro’s TD-Gammon system described
above to create the strategy used by W atson in “Daily-Double” (DD) wagering in its
celebrated winning performance against human champions. These authors report that the
e↵ectiveness of this wagering strategy went well beyond what human players are able to
do in live game play, and that it, along with other advanced strategies, was an important

contributor to W atson ’s impressive winning performance. Here we focus only on DD
wagering because it is the component of W atson that owes the most to reinforcement
learning.
Jeopardy! is played by three contestants who face a board showing 30 squares, each of
which hides a clue and has a dollar value. The squares are arranged in six columns, each
corresponding to a di↵erent category. A contestant selects a square, the host reads the
square’s clue, and each contestant may choose to respond to the clue by sounding a buzzer

learning.
(“buzzing in”). The ﬁrst contestant to buzz in gets to try responding to the clue. If this
contestant’s response is correct, their score increases by the dollar value of the square; if
their response is not correct, or if they do not respond within ﬁve seconds, their score
decreases by that amount, and the other contestants get a chance to buzz in to respond
to the same clue. One or two squares (depending on the game’s current round) are
special DD squares. A contestant who selects one of these gets an exclusive opportunity

to respond to the square’s clue and has to decide—before the clue is revealed—on how
much to wager, or bet. The bet has to be greater than ﬁve dollars but not greater than
the contestant’s current score. If the contestant responds correctly to the DD clue, their
score increases by the bet amount; otherwise it decreases by the bet amount. At the end
of each game is a “Final Jeopardy” (FJ) round in which each contestant writes down
a sealed bet and then writes an answer after the clue is read. The contestant with the

highest score after three rounds of play (where a round consists of revealing all 30 clues)
is the winner. The game has many other details, but these are enough to appreciate
1Registered trademark of IBM Corp.
2Registered trademark of Jeopardy Productions Inc.

430 Chapter 16: Applications and Case Studies
the importance of DD wagering. Winning or losing often depends on a contestant’s DD
wagering strategy.
Whenever W atson selected a DD square, it chose its bet by comparing action values,
ˆq(s, bet), that estimated the probability of a win from the current game state, s, for
each round-dollar legal bet. Except for some risk-abatement measures described below,
Watson selected the bet with the maximum action value. Action values were computed
whenever a betting decision was needed by using two types of estimates that were learned
before any live game play took place. The ﬁrst were estimated values of the afterstates
(Section 6.8) that would result from selecting each legal bet. These estimates were obtained
from a state-value function, ˆv(·,w), deﬁned by parameters w, that gave estimates of the
probability of a win for W atson from any game state. The second estimates used to
compute action values gave the “in-category DD conﬁdence,” pDD, which estimated the

likelihood that W atson would respond correctly to the as-yet unrevealed DD clue.
Tesauro et al. used the reinforcement learning approach of TD-Gammon described above
to learn ˆv(·,w): a straightforward combination of nonlinear TD( )u s i n gam u l t i l a y e r
ANN with weights wtrained by backpropagating TD errors during many simulated
games. States were represented to the network by feature vectors speciﬁcally designed
forJeopardy! . Features included the current scores of the three players, how many DDs

remained, the total dollar value of the remaining clues, and other information related to
the amount of play left in the game. Unlike TD-Gammon, which learned by self-play,
Watson ’sˆvwas learned over millions of simulated games against carefully-crafted models
of human players. In-category conﬁdence estimates were conditioned on the number of
right responses rand wrong responses wthat W atson gave in previously-played clues in
the current category. The dependencies on ( r, w) were estimated from W atson ’s actual
accuracies over many thousands of historical categories.

With the previously learned value function ˆvand in-category DD conﬁdence pDD,
Watson computed ˆ q(s, bet) for each legal round-dollar bet as follows:
ˆq(s, bet)=pDD⇥ˆv(SW+bet,. . . )+( 1 pDD)⇥ˆv(SWbet,. . . ), (16.2)
where SWis Watson ’s current score, and ˆvgives the estimated value for the game state
after W atson ’s response to the DD clue, which is either correct or incorrect. Computing
an action value this way corresponds to the insight from Exercise 3.19 that an action

value is the expected next state value given the action (except that here it is the expected
next afterstate value because the full next state of the entire game depends on the next
square selection).
Tesauro et al. found that selecting bets by maximizing action values incurred “a
frightening amount of risk,” meaning that if W atson ’s response to the clue happened to
be wrong, the loss could be disastrous for its chances of winning. To decrease the downside

risk of a wrong answer, Tesauro et al. adjusted (16.2) by subtracting a small fraction of
the standard deviation over W atson ’s correct/incorrect afterstate evaluations. They
further reduced risk by prohibiting bets that would cause the wrong-answer afterstate
value to decrease below a certain limit. These measures slightly reduced W atson ’s
expectation of winning, but they signiﬁcantly reduced downside risk, not only in terms of
average risk per DD bet, but even more so in extreme-risk scenarios where a risk-neutral

Watson would bet most or all of its bankroll.

16.3. Watson’s Daily-Double Wagering 431
Why was the TD-Gammon method of self-play not used to learn the critical value
function ˆv? Learning from self-play in Jeopardy! would not have worked very well
because W atson was so di↵erent from any human contestant. Self-play would have led to
exploration of state space regions that are not typical for play against human opponents,
particularly human champions. In addition, unlike backgammon, Jeopardy! is a game
of imperfect information because contestants do not have access to all the information
inﬂuencing their opponents’ play. In particular, Jeopardy! contestants do not know how
much conﬁdence their opponents have for responding to clues in the various categories.
Self-play would have been something like playing poker with someone who is holding the
same cards that you hold.
As a result of these complications, much of the e↵ort in developing W atson ’s DD-
wagering strategy was devoted to creating good models of human opponents. The models

did not address the natural language aspect of the game, but were instead stochastic
process models of events that can occur during play. Statistics were extracted from an
extensive fan-created archive of game information from the beginning of the show to the
present day. The archive includes information such as the ordering of the clues, right and
wrong contestant answers, DD locations, and DD and FJ bets for nearly 300,000 clues.
Three models were constructed: an Average Contestant model (based on all the data), a

Champion model (based on statistics from games with the 100 best players), and a Grand
Champion model (based on statistics from games with the 10 best players). In addition
to serving as opponents during learning, the models were used to assess the beneﬁts
produced by the learned DD-wagering strategy. W atson ’s win rate in simulation when it
used a baseline heuristic DD-wagering strategy was 61%; when it used the learned values
and a default conﬁdence value, its win rate increased to 64%; and with live in-category

conﬁdence, it was 67%. Tesauro et al. regarded this as a signiﬁcant improvement, given
that the DD wagering was needed only about 1.5 to 2 times in each game.
Because W atson had only a few seconds to bet, as well as to select squares and decide
whether or not to buzz in, the computation time needed to make these decisions was
a critical factor. The ANN implementation of ˆvallowed DD bets to be made quickly
enough to meet the time constraints of live play. However, once games could be simulated
fast enough through improvements in the simulation software, near the end of a game

it was feasible to estimate the value of bets by averaging over many Monte-Carlo trials
in which the consequence of each bet was determined by simulating play to the game’s
end. Selecting endgame DD bets in live play based on Monte-Carlo trials instead of the
ANN signiﬁcantly improved W atson ’s performance because errors in value estimates
in endgames could seriously a↵ect its chances of winning. Making all the decisions via
Monte-Carlo trials might have led to better wagering decisions, but this was simply

impossible given the complexity of the game and the time constraints of live play.
Although its ability to quickly and accurately answer natural language questions
stands out as W atson ’s major achievement, all of its sophisticated decision strategies
contributed to its impressive defeat of human champions. According to Tesauro et al.
(2012):
... it is plainly evident that our strategy algorithms achieve a level of quanti-
tative precision and real-time performance that exceeds human capabilities.

432 Chapter 16: Applications and Case Studies
This is particularly true in the cases of DD wagering and endgame buzzing,
where humans simply cannot come close to matching the precise equity and
conﬁdence estimates and complex decision calculations performed by Watson.
16.4 Optimizing Memory Control
Most computers use dynamic random access memory (DRAM) as their main memory
because of its low cost and high capacity. The job of a DRAM memory controller is to
eciently use the interface between the processor chip and an o↵-chip DRAM system
to provide the high-bandwidth and low-latency data transfer necessary for high-speed
program execution. A memory controller needs to deal with dynamically changing
patterns of read/write requests while adhering to a large number of timing and resource
constraints required by the hardware. This is a formidable scheduling problem, especially
with modern processors with multiple cores sharing the same DRAM.

˙Ipek, Mutlu, Mart´ ınez, and Caruana (2008) (also Mart´ ınez and ˙Ipek, 2009) designed
a reinforcement learning memory controller and demonstrated that it can signiﬁcantly
improve the speed of program execution over what was possible with conventional
controllers at the time of their research. They were motivated by limitations of existing
state-of-the-art controllers that used policies that did not take advantage of past scheduling
experience and did not account for long-term consequences of scheduling decisions. ˙Ipek

et al.’s project was carried out by means of simulation, but they designed the controller
at the detailed level of the hardware needed to implement it—including the learning
algorithm—directly on a processor chip.
Accessing DRAM involves a number of steps that have to be done according to strict
time constraints. DRAM systems consist of multiple DRAM chips, each containing
multiple rectangular arrays of storage cells arranged in rows and columns. Each cell
stores a bit as the charge on a capacitor. Because the charge decreases over time, each

DRAM cell needs to be recharged—refreshed—every few milliseconds to prevent memory
content from being lost. This need to refresh the cells is why DRAM is called “dynamic.”
Each cell array has a row bu↵er that holds a row of bits that can be transferred into
or out of one of the array’s rows. An activate command “opens a row,” which means
moving the contents of the row whose address is indicated by the command into the
row bu↵er. With a row open, the controller can issue read andwrite commands to the

cell array. Each read command transfers a word (a short sequence of consecutive bits)
in the row bu↵er to the external data bus, and each write command transfers a word
in the external data bus to the row bu↵er. Before a di↵erent row can be opened, a
precharge command must be issued which transfers the (possibly updated) data in the
row bu↵er back into the addressed row of the cell array. After this, another activate
command can open a new row to be accessed. Read and write commands are column

commands because they sequentially transfer bits into or out of columns of the row bu↵er;
multiple bits can be transferred without re-opening the row. Read and write commands
to the currently-open row can be carried out more quickly than accessing a di↵erent row,
which would involve additional row commands : precharge and activate; this is sometimes

16.4. Optimizing Memory Control 433
referred to as “row locality.” A memory controller maintains a memory transaction queue
that stores memory-access requests from the processors sharing the memory system. The
controller has to process requests by issuing commands to the memory system while
adhering to a large number of timing constraints.
A controller’s policy for scheduling access requests can have a large e↵ect on the
performance of the memory system, such as the average latency with which requests
can be satisﬁed and the throughput the system is capable of achieving. The simplest
scheduling strategy handles access requests in the order in which they arrive by issuing
all the commands required by the request before beginning to service the next one. But if
the system is not ready for one of these commands, or executing a command would result
in resources being underutilized (e.g., due to timing constraints arising from servicing

that one command), it makes sense to begin servicing a newer request before ﬁnishing
the older one. Policies can gain eciency by reordering requests, for example, by giving
priority to read requests over write requests, or by giving priority to read/write commands
to already open rows. The policy called First-Ready, First-Come-First-Serve (FR-FCFS),
gives priority to column commands (read and write) over row commands (activate and
precharge), and in case of a tie gives priority to the oldest command. FR-FCFS was

shown to outperform other scheduling policies in terms of average memory-access latency
under conditions commonly encountered (Rixner, 2004).
Figure 16.3 is a high-level view of ˙Ipek et al.’s reinforcement learning memory controller.
They modeled the DRAM access process as an MDP whose states are the contents of the
transaction queue and whose actions are commands to the DRAM system: precharge ,
activate ,read,write , and NoOp . The reward signal is 1 whenever the action is read or
write , and otherwise it is 0. State transitions were considered to be stochastic because

the next state of the system not only depends on the scheduler’s command, but also on
aspects of the system’s behavior that the scheduler cannot control, such as the workloads
of the processor cores accessing the DRAM system.
Figure 16.3: High-level view of the reinforcement learning DRAM controller. The scheduler is
the reinforcement learning agent. Its environment is represented by features of the transaction
queue, and its actions are commands to the DRAM system. c2009 IEEE. Reprinted, with

permission, from J. F. Mart´ ınez and E. ˙Ipek, Dynamic multicore resource management: A
machine learning approach, Micro, IEEE, 29 (5), p. 12.

434 Chapter 16: Applications and Case Studies
Critical to this MDP are constraints on the actions available in each state. Recall from
Chapter 3 that the set of available actions can depend on the state: At2A(St), where
Atis the action at time step tandA(St) is the set of actions available in state St.I n
this application, the integrity of the DRAM system was assured by not allowing actions
that would violate timing or resource constraints. Although ˙Ipek et al. did not make it
explicit, they e↵ectively accomplished this by pre-deﬁning the sets A(St) for all possible
states St.
These constraints explain why the MDP has a NoOp action and why the reward signal
is 0 except when a read orwrite command is issued. NoOp is issued when it is the sole
legal action in a state. To maximize utilization of the memory system, the controller’s
task is to drive the system to states in which either a read or a write action can be
selected: only these actions result in sending data over the external data bus, so it is only

states St.
these that contribute to the throughput of the system. Although precharge andactivate
produce no immediate reward, the agent needs to select these actions to make it possible
to later select the rewarded read andwrite actions.
The scheduling agent used Sarsa (Section 6.4) to learn an action-value function. States
were represented by six integer-valued features. To approximate the action-value function,
the algorithm used linear function approximation implemented by tile coding with hashing

(Section 9.5.4). The tile coding had 32 tilings, each storing 256 action values as 16-bit
ﬁxed point numbers. Exploration was "-greedy with "=0.05.
State features included the number of read requests in the transaction queue, the
number of write requests in the transaction queue, the number of write requests in the
transaction queue waiting for their row to be opened, and the number of read requests in
the transaction queue waiting for their row to be opened that are the oldest issued by

their requesting processors. (The other features depended on how the DRAM interacts
with cache memory, details we omit here.) The selection of the state features was based
on˙Ipek et al.’s understanding of factors that impact DRAM performance. For example,
balancing the rate of servicing reads and writes based on how many of each are in the
transaction queue can help avoid stalling the DRAM system’s interaction with cache
memory. The authors in fact generated a relatively long list of potential features, and

then pared them down to a handful using simulations guided by stepwise feature selection.
An interesting aspect of this formulation of the scheduling problem as an MDP is
that the features input to the tile coding for deﬁning the action-value function were
di↵erent from the features used to specify the action-constraint sets A(St). Whereas the
tile coding input was derived from the contents of the transaction queue, the constraint
sets depended on a host of other features related to timing and resource constraints that

had to be satisﬁed by the hardware implementation of the entire system. In this way, the
action constraints ensured that the learning algorithm’s exploration could not endanger
the integrity of the physical system, while learning was e↵ectively limited to a “safe”
region of the much larger state space of the hardware implementation.
Because an objective of this work was that the learning controller could be implemented
on a chip so that learning could occur online while a computer is running, hardware

implementation details were important considerations. The design included two ﬁve-stage
pipelines to calculate and compare two action values at every processor clock cycle, and

16.4. Optimizing Memory Control 435
to update the appropriate action value. This included accessing the tile coding which
was stored on-chip in static RAM. For the conﬁguration ˙Ipek et al. simulated, which was
a 4GHz 4-core chip typical of high-end workstations at the time of their research, there
were 10 processor cycles for every DRAM cycle. Considering the cycles needed to ﬁll
the pipes, up to 12 actions could be evaluated in each DRAM cycle. ˙Ipek et al. found
that the number of legal commands for any state was rarely greater than this, and that
performance loss was negligible if enough time was not always available to consider all
legal commands. These and other clever design details made it feasible to implement the
complete controller and learning algorithm on a multi-processor chip.
˙Ipek et al. evaluated their learning controller in simulation by comparing it with
three other controllers: 1) the FR-FCFS controller mentioned above that produces the

best on-average performance, 2) a conventional controller that processes each request
in order, and 3) an unrealizable ideal controller, called the Optimistic controller, able
to sustain 100% DRAM throughput if given enough demand by ignoring all timing
and resource constraints, but otherwise modeling DRAM latency (as row bu↵er hits)
and bandwidth. They simulated nine memory-intensive parallel workloads consisting of
scientiﬁc and data-mining applications. Figure 16.4 shows the performance (the inverse

of execution time normalized to the performance of FR-FCFS) of each controller for
the nine applications, together with the geometric mean of their performances over the
applications. The learning controller, labeled RL in the ﬁgure, improved over that of
FR-FCFS by from 7% to 33% over the nine applications, with an average improvement of
19%. Of course, no realizable controller can match the performance of Optimistic, which
ignores all timing and resource constraints, but the learning controller’s performance

closed the gap with Optimistic’s upper bound by an impressive 27%.
Because the rationale for on-chip implementation of the learning algorithm was to
allow the scheduling policy to adapt online to changing workloads, ˙Ipek et al. analyzed
the impact of online learning compared to a previously-learned ﬁxed policy. They trained
Figure 16.4: Performances of four controllers over a suite of 9 simulated benchmark applications.
The controllers are: the simplest ‘in-order’ controller, FR-FCFS, the learning controller RL,

and the unrealizable Optimistic controller which ignores all timing and resource constraints to
provide a performance upper bound. Performance, normalized to that of FR-FCFS, is the inverse
of execution time. At far right is the geometric mean of performances over the 9 benchmark
applications for each controller. Controller RL comes closest to the ideal performance. c2009
IEEE. Reprinted, with permission, from J. F. Mart´ ınez and E. ˙Ipek, Dynamic multicore resource
management: A machine learning approach, Micro, IEEE, 29 (5), p. 13.

436 Chapter 16: Applications and Case Studies
their controller with data from all nine benchmark applications and then held the resulting
action values ﬁxed throughout the simulated execution of the applications. They found
that the average performance of the controller that learned online was 8% better than
that of the controller using the ﬁxed policy, leading them to conclude that online learning
is an important feature of their approach.
This learning memory controller was never committed to physical hardware because of
the large cost of fabrication. Nevertheless, ˙Ipek et al. could convincingly argue on the basis
of their simulation results that a memory controller that learns online via reinforcement
learning has the potential to improve performance to levels that would otherwise require
more complex and more expensive memory systems, while removing from human designers
some of the burden required to manually design ecient scheduling policies. Mukundan

and Mart´ ınez (2012) took this project forward by investigating learning controllers with
additional actions, other performance criteria, and more complex reward functions derived
using genetic algorithms. They considered additional performance criteria related to
energy eciency. The results of these studies surpassed the earlier results described
above and signiﬁcantly surpassed the 2012 state-of-the-art for all of the performance
criteria they considered. The approach is especially promising for developing sophisticated
power-aware DRAM interfaces.

16.5 Human-level Video Game Play
One of the greatest challenges in applying reinforcement learning to real-world problems
is deciding how to represent and store value functions and/or policies. Unless the state
set is ﬁnite and small enough to allow exhaustive representation by a lookup table—as in
many of our illustrative examples—one must use a parameterized function approximation
scheme. Whether linear or nonlinear, function approximation relies on features that

have to be readily accessible to the learning system and able to convey the information
necessary for skilled performance. Most successful applications of reinforcement learning
owe much to sets of features carefully handcrafted based on human knowledge and
intuition about the speciﬁc problem to be tackled.
A team of researchers at Google DeepMind developed an impressive demonstration that
a deep multi-layer ANN can automate the feature design process (Mnih et al., 2013, 2015).

Multi-layer ANNs have been used for function approximation in reinforcement learning
ever since the 1986 popularization of the backpropagation algorithm as a method for
learning internal representations (Rumelhart, Hinton, and Williams, 1986; see Section 9.6).
Striking results have been obtained by coupling reinforcement learning with backpropa-
gation. The results obtained by Tesauro and colleages with TD-Gammon and W atson
discussed above are notable examples. These and other applications beneﬁted from the

ability of multi-layer ANNs to learn task-relevant features. However, in all the examples
of which we are aware, the most impressive demonstrations required the network’s input
to be represented in terms of specialized features handcrafted for the given problem.
This is vividly apparent in the TD-Gammon results. TD-Gammon 0.0, whose network
input was essentially a “raw” representation of the backgammon board, meaning that it
involved very little knowledge of backgammon, learned to play approximately as well as

16.5. Human-level Video Game Play 437
the best previous backgammon computer programs. Adding specialized backgammon
features produced TD-Gammon 1.0 which was substantially better than all previous
backgammon programs and competed well against human experts.
Mnih et al. developed a reinforcement learning agent called deep Q-network (DQN)
that combined Q-learning with a deep convolutional ANN, a many-layered, or deep,
ANN specialized for processing spatial arrays of data such as images. We describe deep
convolutional ANNs in Section 9.6. By the time of Mnih et al.’s work with DQN, deep
ANNs, including deep convolutional ANNs, had produced impressive results in many
applications, but they had not been widely used in reinforcement learning.
Mnih et al. used DQN to show how a reinforcement learning agent can achieve a high
level of performance on any of a collection of di↵erent problems without having to use
di↵erent problem-speciﬁc feature sets. To demonstrate this, they let DQN learn to play

49 di↵erent Atari 2600 video games by interacting with a game emulator. DQN learned a
di↵erent policy for each of the 49 games (because the weights of its ANN were reset to
random values before learning on each game), but it used the same raw input, network
architecture, and parameter values (e.g., step size, discount rate, exploration parameters,
and many more speciﬁc to the implementation) for all the games. DQN achieved levels
of play at or beyond human level on a large fraction of these games. Although the games

were alike in being played by watching streams of video images, they varied widely in other
respects. Their actions had di↵erent e↵ects, they had di↵erent state-transition dynamics,
and they needed di↵erent policies for learning high scores. The deep convolutional ANN
learned to transform the raw input common to all the games into features specialized for
representing the action values required for playing at the high level DQN achieved for
most of the games.
The Atari 2600 is a home video game console that was sold in various versions by Atari

Inc. from 1977 to 1992. It introduced or popularized many arcade video games that are
now considered classics, such as Pong, Breakout, Space Invaders, and Asteroids. Although
much simpler than modern video games, Atari 2600 games are still entertaining and
challenging for human players, and they have been attractive as testbeds for developing
and evaluating reinforcement learning methods (Diuk, Cohen, Littman, 2008; Naddaf,
2010; Cobo, Zang, Isbell, and Thomaz, 2011; Bellemare, Veness, and Bowling, 2013).

Bellemare, Naddaf, Veness, and Bowling (2012) developed the publicly available Arcade
Learning Environment (ALE) to encourage and simplify using Atari 2600 games to study
learning and planning algorithms.
These previous studies and the availability of ALE made the Atari 2600 game collection
a good choice for Mnih et al.’s demonstration, which was also inﬂuenced by the impressive
human-level performance that TD-Gammon was able to achieve in backgammon. DQN

is similar to TD-Gammon in using a multi-layer ANN as the function approximation
method for a semi-gradient form of a TD algorithm, with the gradients computed by
the backpropagation algorithm. However, instead of using TD( ) as TD-Gammon did,
DQN used the semi-gradient form of Q-learning. TD-Gammon estimated the values of
afterstates, which were easily obtained from the rules for making backgammon moves.
To use the same algorithm for the Atari games would have required generating the next

states for each possible action (which would not have been afterstates in that case).

438 Chapter 16: Applications and Case Studies
This could have been done by using the game emulator to run single-step simulations
for all the possible actions (which ALE makes possible). Or a model of each game’s
state-transition function could have been learned and used to predict next states (Oh,
Guo, Lee, Lewis, and Singh, 2015). While these methods might have produced results
comparable to DQN’s, they would have been more complicated to implement and would
have signiﬁcantly increased the time needed for learning. Another motivation for using
Q-learning was that DQN used the experience replay method, described below, which
requires an o↵-policy algorithm. Being model-free and o↵-policy made Q-learning a
natural choice.
Before describing the details of DQN and how the experiments were conducted, we look
at the skill levels DQN was able to achieve. Mnih et al. compared the scores of DQN with
the scores of the best performing learning system in the literature at the time, the scores

of a professional human games tester, and the scores of an agent that selected actions at
random. The best system from the literature used linear function approximation with
features designed using some knowledge about Atari 2600 games (Bellemare, Naddaf,
Veness, and Bowling, 2013). DQN learned on each game by interacting with the game
emulator for 50 million frames, which corresponds to about 38 days of experience with
the game. At the start of learning on each game, the weights of DQN’s network were reset

to random values. To evaluate DQN’s skill level after learning, its score was averaged
over 30 sessions on each game, each lasting up to 5 minutes and beginning with a random
initial game state. The professional human tester played using the same emulator (with
the sound turned o↵ to remove any possible advantage over DQN which did not process
audio). After 2 hours of practice, the human played about 20 episodes of each game for up
to 5 minutes each and was not allowed to take any break during this time. DQN learned

to play better than the best previous reinforcement learning systems on all but 6 of the
games, and played better than the human player on 22 of the games. By considering any
performance that scored at or above 75% of the human score to be comparable to, or
better than, human-level play, Mnih et al. concluded that the levels of play DQN learned
reached or exceeded human level on 29 of the 46 games. See Mnih et al. (2015) for a
more detailed account of these results.
For an artiﬁcial learning system to achieve these levels of play would be impressive

enough, but what makes these results remarkable—and what many at the time considered
to be breakthrough results for artiﬁcial intelligence—is that the very same learning system
achieved these levels of play on widely varying games without relying on any game-speciﬁc
modiﬁcations.
A human playing any of these 49 Atari games sees 210 ⇥160 pixel image frames with
128 colors at 60Hz. In principle, exactly these images could have formed the raw input to
DQN, but to reduce memory and processing requirements, Mnih et al. preprocessed each

frame to produce an 84 ⇥84 array of luminance values. Because the full states of many
of the Atari games are not completely observable from the image frames, Mnih et al.
“stacked” the four most recent frames so that the inputs to the network had dimension
84⇥84⇥4. This did not eliminate partial observability for all of the games, but it was
helpful in making many of them more Markovian.
An essential point here is that these preprocessing steps were exactly the same for all 46

16.5. Human-level Video Game Play 439
games. No game-speciﬁc prior knowledge was involved beyond the general understanding
that it should still be possible to learn good policies with this reduced dimension and
that stacking adjacent frames should help with the partial observability of some of the
games. Because no game-speciﬁc prior knowledge beyond this minimal amount was used
in preprocessing the image frames, we can think of the 84 ⇥84⇥4 input vectors as being
“raw” input to DQN.
The basic architecture of DQN is similar to the deep convolutional ANN illustrated in
Figure 9.15 (though unlike that network, subsampling in DQN is treated as part of each
convolutional layer, with feature maps consisting of units having only a selection of the
possible receptive ﬁelds). DQN has three hidden convolutional layers, followed by one
fully connected hidden layer, followed by the output layer. The three successive hidden
convolutional layers of DQN produce 32 20 ⇥20 feature maps, 64 9 ⇥9 feature maps,

and 64 7 ⇥7 feature maps. The activation function of the units of each feature map is a
rectiﬁer nonlinearity ( max(0,x)). The 3,136 (64 ⇥7⇥7) units in this third convolutional
layer all connect to each of 512 units in the fully connected hidden layer, which then each
connect to all 18 units in the output layer, one for each possible action in an Atari game.
The activation levels of DQN’s output units were the estimated optimal action values
of the corresponding state–action pairs, for the state represented by the network’s input.

The assignment of output units to a game’s actions varied from game to game, and
because the number of valid actions varied between 4 and 18 for the games, not all output
units had functional roles in all of the games. It helps to think of the network as if it
were 18 separate networks, one for estimating the optimal action value of each possible
action. In reality, these networks shared their initial layers, but the output units learned
to use the features extracted by these layers in di↵erent ways.

DQN’s reward signal indicated how a games’s score changed from one time step to
the next: +1 whenever it increased, 1 whenever it decreased, and 0 otherwise. This
standardized the reward signal across the games and made a single step-size parameter
work well for all the games despite their varying ranges of scores. DQN used an "-greedy
policy, with "decreasing linearly over the ﬁrst million frames and remaining at a low
value for the rest of the learning session. The values of the various other parameters,

such as the learning step size, discount rate, and others speciﬁc to the implementation,
were selected by performing informal searches to see which values worked best for a small
selection of the games. These values were then held ﬁxed for all of the games.
After DQN selected an action, the action was executed by the game emulator, which
returned a reward and the next video frame. The frame was preprocessed and added
to the four-frame stack that became the next input to the network. Skipping for the
moment the changes to the basic Q-learning procedure made by Mnih et al., DQN used

the following semi-gradient form of Q-learning to update the network’s weights:
wt+1=wt+↵h
Rt+1+max
aˆq(St+1,a ,wt)ˆq(St,At,wt)i
rˆq(St,At,wt),(16.3)
where wtis the vector of the network’s weights, Atis the action selected at time step t,
andStandSt+1are respectively the preprocessed image stacks input to the network at
time steps tandt+ 1.
The gradient in (16.3) was computed by backpropagation. Imagining again that there

440 Chapter 16: Applications and Case Studies
was a separate network for each action, for the update at time step t, backpropagation
was applied only to the network corresponding to At. Mnih et al. took advantage of
techniques shown to improve the basic backpropagation algorithm when applied to large
networks. They used a mini-batch method that updated weights only after accumulating
gradient information over a small batch of images (here after 32 images). This yielded
smoother sample gradients compared to the usual procedure that updates weights after
each action. They also used a gradient-ascent algorithm called RMSProp (Tieleman and
Hinton, 2012) that accelerates learning by adjusting the step-size parameter for each
weight based on a running average of the magnitudes of recent gradients for that weight.
Mnih et al. modiﬁed the basic Q-learning procedure in three ways. First, they used
a method called experience replay ﬁrst studied by Lin (1992). This method stores the

agent’s experience at each time step in a replay memory that is accessed to perform the
weight updates. It worked like this in DQN. After the game emulator executed action
Atin a state represented by the image stack St, and returned reward Rt+1and image
stack St+1, it added the tuple ( St,At,Rt+1,St+1) to the replay memory. This memory
accumulated experiences over many plays of the same game. At each time step multiple Q-
learning updates—a mini-batch—were performed based on experiences sampled uniformly
at random from the replay memory. Instead of St+1becoming the new Stfor the next

update as it would in the usual form of Q-learning, a new unconnected experience was
drawn from the replay memory to supply data for the next update. Because Q-learning
is an o↵-policy algorithm, it does not need to be applied along connected trajectories.
Q-learning with experience replay provided several advantages over the usual form of
Q-learning. The ability to use each stored experience for many updates allowed DQN to
learn more eciently from its experiences. Experience replay reduced the variance of the

updates because successive updates were not correlated with one another as they would
be with standard Q-learning. And by removing the dependence of successive experiences
on the current weights, experience replay eliminated one source of instability.
Mnih et al. modiﬁed standard Q-learning in a second way to improve its stability. As
in other methods that bootstrap, the target for a Q-learning update depends on the
current action-value function estimate. When a parameterized function approximation

method is used to represent action values, the target is a function of the same parameters
that are being updated. For example, the target in the update given by (16.3) is
max aˆq(St+1,a ,wt). Its dependence on wtcomplicates the process compared to the
simpler supervised-learning situation in which the targets do not depend on the parameters
being updated. As discussed in Chapter 11 this can lead to oscillations and/or divergence.
To address this problem Mnih et al. used a technique that brought Q-learning closer

to the simpler supervised-learning case while still allowing it to bootstrap. Whenever
a certain number, C, of updates had been done to the weights wof the action-value
network, they inserted the network’s current weights into another network and held
these duplicate weights ﬁxed for the next Cupdates of w. The outputs of this duplicate
network over the next Cupdates of wwere used as the Q-learning targets. Letting ˜q
denote the output of this duplicate network, then instead of (16.3) the update rule was:
Rt+1+max
a˜q(St+1,a ,wt)ˆq(St,At,wt)i
rˆq(St,At,wt).

16.6. Mastering the Game of Go 441
A ﬁnal modiﬁcation of standard Q-learning was also found to improve stability. They
clipped the error term Rt+1+max a˜q(St+1,a ,wt)ˆq(St,At,wt) so that it remained in
the interval [ 1,1].
Mnih et al. conducted a large number of learning runs on 5 of the games to gain insight
into the e↵ect that various of DQN’s design features had on its performance. They ran
DQN with the four combinations of experience replay and the duplicate target network
being included or not included. Although the results varied from game to game, each of
these features alone signiﬁcantly improved performance, and very dramatically improved
performance when used together. Mnih et al. also studied the role played by the deep
convolutional ANN in DQN’s learning ability by comparing the deep convolutional version
of DQN with a version having a network of just one linear layer, both receiving the same
stacked preprocessed video frames. Here, the improvement of the deep convolutional

version over the linear version was particularly striking across all 5 of the test games.
Creating artiﬁcial agents that excel over a diverse collection of challenging tasks has
been an enduring goal of artiﬁcial intelligence. The promise of machine learning as
a means for achieving this has been frustrated by the need to craft problem-speciﬁc
representations. DeepMind’s DQN stands as a major step forward by demonstrating
that a single agent can learn problem-speciﬁc features enabling it to acquire human-

competitive skills over a range of tasks. This demonstration did not produce one agent
that simultaneously excelled at all the tasks (because learning occurred separately for
each task), but it showed that deep learning can reduce, and possibly eliminate, the need
for problem-speciﬁc design and tuning. As Mnih et al. point out, however, DQN is not
a complete solution to the problem of task-independent learning. Although the skills

needed to excel on the Atari games were markedly diverse, all the games were played by
observing video images, which made a deep convolutional ANN a natural choice for this
collection of tasks. In addition, DQN’s performance on some of the Atari 2600 games
fell considerably short of human skill levels on these games. The games most dicult
for DQN—especially Montezuma’s Revenge on which DQN learned to perform about as
well as the random player—require deep planning beyond what DQN was designed to

do. Further, learning control skills through extensive practice, like DQN learned how to
play the Atari games, is just one of the types of learning humans routinely accomplish.
Despite these limitations, DQN advanced the state-of-the-art in machine learning by
impressively demonstrating the promise of combining reinforcement learning with modern
methods of deep learning.
16.6 Mastering the Game of Go
The ancient Chinese game of Go has challenged artiﬁcial intelligence researchers for many

decades. Methods that achieve human-level skill, or even superhuman-level skill, in other
games have not been successful in producing strong Go programs. Thanks to a very
active community of Go programmers and international competitions, the level of Go
program play has improved signiﬁcantly over the years. Until recently, however, no Go
program had been able to play anywhere near the level of a human Go master.
A team at DeepMind (Silver et al., 2016) developed the program AlphaGo that broke

442 Chapter 16: Applications and Case Studies
this barrier by combining deep ANNs (Section 9.6), supervised learning, Monte Carlo
tree search (MCTS, Section 8.11), and reinforcement learning. By the time of Silver et
al.’s 2016 publication, AlphaGo had been shown to be decisively stronger than other
current Go programs, and it had defeated the European Go champion Fan Hui 5 games
to 0. These were the ﬁrst victories of a Go program over a professional human Go player
without handicap in full Go games. Shortly thereafter, a similar version of AlphaGo won
stunning victories over the 18-time world champion Lee Sedol, winning 4 out of a 5 games
in a challenge match, making worldwide headline news. Artiﬁcial intelligence researchers
thought that it would be many more years, perhaps decades, before a program reached
this level of play.
Here we describe AlphaGo and a successor program called AlphaGo Zero (Silver et al.
2017a). Where in addition to reinforcement learning, AlphaGo relied on supervised learn-

ing from a large database of expert human moves, AlphaGo Zero used only reinforcement
learning and no human data or guidance beyond the basic rules of the game (hence the
Zero in its name). We ﬁrst describe AlphaGo in some detail in order to highlight the
relative simplicity of AlphaGo Zero , which is both higher-performing and more of a pure
reinforcement learning program.
In many ways, both AlphaGo andAlphaGo Zero are descendants of Tesauro’s TD-
Gammon (Section 16.1), itself a descendant of Samuel’s checkers player (Section 16.2).

All these programs included reinforcement learning over simulated games of self-play.
AlphaGo andAlphaGo Zero also built upon the progress made by DeepMind on playing
Atari games with the program DQN (Section 16.5) that used deep convolutional ANNs
to approximate optimal value functions.
A Go board conﬁgurationGo is a game between two players who alter-
nately place black and white ‘stones’ on unoccu-
pied intersections, or ‘points,’ on a board with
a grid of 19 horizontal and 19 vertical lines to
produce positions like that shown to the right.

The game’s goal is to capture an area of the
board larger than that captured by the oppo-
nent. Stones are captured according to simple
rules. A player’s stones are captured if they
are completely surrounded by the other player’s
stones, meaning that there is no horizontally
or vertically adjacent point that is unoccupied.
For example, Figure 16.5 shows on the left three
white stones with an unoccupied adjacent point
(labeled X). If player black places a stone on X,
the three white stones are captured and taken

o↵ the board (Figure 16.5 middle). However,
if player white were to place a stone on point X ﬁrst, then the possibility of this cap-
ture would be blocked (Figure 16.5 right). Other rules are needed to prevent inﬁnite
capturing/re-capturing loops. The game ends when neither player wishes to place another
stone. These rules are simple, but they produce a very complex game that has had wide

16.6. Mastering the Game of Go 443XFigure 16.5: Go capturing rule. Left: the three white stones are not surrounded because point
X is unoccupied. Middle: if black places a stone on X, the three white stones are captured and
removed from the board. Right: if white places a stone on point X ﬁrst, the capture is blocked.
appeal for thousands of years.
Methods that produce strong play for other games, such as chess, have not worked as
well for Go. The search space for Go is signiﬁcantly larger than that of chess because
Go has a larger number of legal moves per position than chess ( ⇡250 versus ⇡35) and
Go games tend to involve more moves than chess games ( ⇡150 versus ⇡80). But the
size of the search space is not the major factor that makes Go so dicult. Exhaustive
search is infeasible for both chess and Go, and Go on smaller boards (e.g., 9 ⇥9) has
proven to be exceedingly dicult as well. Experts agree that the major stumbling block

to creating stronger-than-amateur Go programs is the diculty of deﬁning an adequate
position evaluation function. A good evaluation function allows search to be truncated at
a feasible depth by providing relatively easy-to-compute predictions of what deeper search
would likely yield. According to M¨ uller (2002): “No simple yet reasonable evaluation
function will ever be found for Go.” A major step forward was the introduction of MCTS

to Go programs. The strongest programs at the time of AlphaGo ’s development all
included MCTS, but master-level skill remained elusive.
Recall from Section 8.11 that MCTS is a decision-time planning procedure that does
not attempt to learn and store a global evaluation function. Like a rollout algorithm
(Section 8.10), it runs many Monte Carlo simulations of entire episodes (here, entire
Go games) to select each action (here, each Go move: where to place a stone or to
resign). Unlike a simple rollout algorithm, however, MCTS is an iterative procedure that

incrementally extends a search tree whose root node represents the current environment
state. As illustrated in Figure 8.10, each iteration traverses the tree by simulating
actions guided by statistics associated with the tree’s edges. In its basic version, when
a simulation reaches a leaf node of the search tree, MCTS expands the tree by adding
some, or all, of the leaf node’s children to the tree. From the leaf node, or one of its
newly added child nodes, a rollout is executed: a simulation that typically proceeds all

the way to a terminal state, with actions selected by a rollout policy. When the rollout
completes, the statistics associated with the search tree’s edges that were traversed in
this iteration are updated by backing up the return produced by the rollout. MCTS
continues this process, starting each time at the search tree’s root at the current state, for
as many iterations as possible given the time constraints. Then, ﬁnally, an action from
the root node (which still represents the current environment state) is selected according

to statistics accumulated in the root node’s outgoing edges. This is the action the agent
takes. After the environment transitions to its next state, MCTS is executed again with
the root node set to represent the new current state. The search tree at the start of this

444 Chapter 16: Applications and Case Studies
next execution might be just this new root node, or it might include descendants of this
node left over from MCTS’s previous execution. The remainder of the tree is discarded.
16.6.1 AlphaGo
The main innovation that made AlphaGo such a strong player is that it selected moves by
a novel version of MCTS that was guided by both a policy and a value function learned
by reinforcement learning with function approximation provided by deep convolutional
ANNs. Another key feature is that instead of reinforcement learning starting from random
network weights, it started from weights that were the result of previous supervised
learning from a large collection of human expert moves.
The DeepMind team called AlphaGo ’s modiﬁcation of basic MCTS “asynchronous
policy and value MCTS,” or APV-MCTS. It selected actions via basic MCTS as described
above but with some twists in how it extended its search tree and how it evaluated action

edges. In contrast to basic MCTS, which expands its current search tree by using stored
action values to select an unexplored edge from a leaf node, APV-MCTS, as implemented
inAlphaGo , expanded its tree by choosing an edge according to probabilities supplied by
a 13-layer deep convolutional ANN, called the SL-policy network , trained previously by
supervised learning to predict moves contained in a database of nearly 30 million human
expert moves.
Then, also in contrast to basic MCTS, which evaluates the newly-added state node

solely by the return of a rollout initiated from it, APV-MCTS evaluated the node in two
ways: by this return of the rollout, but also by a value function, v✓, learned previously by
a reinforcement learning method. If swas the newly-added node, its value became
v(s)=( 1 ⌘)v✓(s)+⌘G, (16.4)
where Gwas the return of the rollout and ⌘controlled the mixing of the values resulting
from these two evaluation methods. In AlphaGo , these values were supplied by the
value network , another 13-layer deep convolutional ANN that was trained as we describe

below to output estimated values of board positions. APV-MCTS’s rollouts in AlphaGo
were simulated games with both players using a fast rollout policy provided by a simple
linear network, also trained by supervised learning before play. Throughout its execution,
APV-MCTS kept track of how many simulations passed through each edge of the search
tree, and when its execution completed, the most-visited edge from the root node was
selected as the action to take, here the move AlphaGo actually made in a game.

The value network had the same structure as the deep convolutional SL policy network
except that it had a single output unit that gave estimated values of game positions
instead of the SL policy network’s probability distributions over legal actions. Ideally,
the value network would output optimal state values, and it might have been possible to
approximate the optimal value function along the lines of TD-Gammon described above:

self-play with nonlinear TD( ) coupled to a deep convolutional ANN. But the DeepMind
team took a di↵erent approach that held more promise for a game as complex as Go.
They divided the process of training the value network into two stages. In the ﬁrst stage,
they created the best policy they could by using reinforcement learning to train an RL

16.6. Mastering the Game of Go 445
policy network . This was a deep convolutional ANN with the same structure as the SL
policy network. It was initialized with the ﬁnal weights of the SL policy network that
were learned via supervised learning, and then policy-gradient reinforcement learning was
used to improve upon the SL policy. In the second stage of training the value network,
the team used Monte Carlo policy evaluation on data obtained from a large number of
simulated self-play games with moves selected by the RL policy network.
Figure 16.6 illustrates the networks used by AlphaGo and the steps taken to train them
in what the DeepMind team called the “ AlphaGo pipeline.” All these networks were
trained before any live game play took place, and their weights remained ﬁxed throughout
live play.
28 JANUARY 2016 | VOL 529 | NATURE | 485ARTICLERESEARCH

sampled state-action pairs (s, a), using stochastic gradient ascent to maximize the likelihood of the human move a selected in state s∆σσ∝∂(|)∂σpaslogWe trained a 13-layer policy network, which we call the SL policy network, from 30 million positions from the KGS Go Server. The net-work predicted expert moves on a held out test set with an accuracy of 57.0% using all input features, and 55.7% using only raw board posi-tion and move history as inputs, compared to the state-of-the-art from other research groups of 44.4% at date of submission24 (full results in Extended Data Table 3). Small improvements in accuracy led to large improvements in playing strength (Fig. 2a); larger networks achieve better accuracy but are slower to evaluate during search. We also trained a faster but less accurate rollout policy pπ(a|s), using a linear softmax of small pattern features (see Extended Data Table 4) with weights π; this achieved an accuracy of 24.2%, using just 2 µs to select an action, rather than 3 ms for the policy

of 44.4% at date of submission24 (full results in Extended Data Table 3). Small improvements in accuracy led to large improvements in playing strength (Fig. 2a); larger networks achieve better accuracy but are slower to evaluate during search. We also trained a faster but less accurate rollout policy pπ(a|s), using a linear softmax of small pattern features (see Extended Data Table 4) with weights π; this achieved an accuracy of 24.2%, using just 2 µs to select an action, rather than 3 ms for the policy network.Reinforcement learning of policy networksThe second stage of the training pipeline aims at improving the policy network by policy gradient reinforcement learning (RL)25,26. The RL policy network pρ is identical in structure to the SL policy network, and its weights ρ are initialized to the same values, ρ = σ. We play games between the current policy network pρ and a randomly selected previous iteration of the policy network. Randomizing from a pool of opponents in this way stabilizes training by

network.Reinforcement learning of policy networksThe second stage of the training pipeline aims at improving the policy network by policy gradient reinforcement learning (RL)25,26. The RL policy network pρ is identical in structure to the SL policy network, and its weights ρ are initialized to the same values, ρ = σ. We play games between the current policy network pρ and a randomly selected previous iteration of the policy network. Randomizing from a pool of opponents in this way stabilizes training by preventing overfitting to the current policy. We use a reward function r(s) that is zero for all non-terminal time steps t < T. The outcome zt = ± r(sT) is the termi-nal reward at the end of the game from the perspective of the current player at time step t: +1 for winning and −1 for losing. Weights are then updated at each time step t by stochastic gradient ascent in the direction that maximizes expected outcome25∆ρρ∝∂(|)∂ρpaszlogtttWe evaluated the performance of the RL policy network in game  play,

by preventing overfitting to the current policy. We use a reward function r(s) that is zero for all non-terminal time steps t < T. The outcome zt = ± r(sT) is the termi-nal reward at the end of the game from the perspective of the current player at time step t: +1 for winning and −1 for losing. Weights are then updated at each time step t by stochastic gradient ascent in the direction that maximizes expected outcome25∆ρρ∝∂(|)∂ρpaszlogtttWe evaluated the performance of the RL policy network in game  play, sampling each move ∼(⋅|)ρapstt from its output probability  distribution over actions. When played head-to-head, the RL policy network won more than 80% of games against the SL policy network. We also tested against the strongest open-source Go program, Pachi14, a sophisticated Monte Carlo search program, ranked at 2 amateur dan on KGS, that executes 100,000 simulations per move. Using no search at all, the RL policy network won 85% of games against Pachi. In com-parison, the previous state-of-the-art, based

each move ∼(⋅|)ρapstt from its output probability  distribution over actions. When played head-to-head, the RL policy network won more than 80% of games against the SL policy network. We also tested against the strongest open-source Go program, Pachi14, a sophisticated Monte Carlo search program, ranked at 2 amateur dan on KGS, that executes 100,000 simulations per move. Using no search at all, the RL policy network won 85% of games against Pachi. In com-parison, the previous state-of-the-art, based only on supervised Figure 1 | Neural network training pipeline and architecture. a, A fast rollout policy pπ and supervised learning (SL) policy network pσ are trained to predict human expert moves in a data set of positions.  A reinforcement learning (RL) policy network pρ is initialized to the SL policy network, and is then improved by policy gradient learning to maximize the outcome (that is, winning more games) against previous versions of the policy network. A new data set is generated by playing games of

on supervised Figure 1 | Neural network training pipeline and architecture. a, A fast rollout policy pπ and supervised learning (SL) policy network pσ are trained to predict human expert moves in a data set of positions.  A reinforcement learning (RL) policy network pρ is initialized to the SL policy network, and is then improved by policy gradient learning to maximize the outcome (that is, winning more games) against previous versions of the policy network. A new data set is generated by playing games of self-play with the RL policy network. Finally, a value network vθ is trained by regression to predict the expected outcome (that is, whether the current player wins) in positions from the self-play data set.  b, Schematic representation of the neural network architecture used in AlphaGo. The policy network takes a representation of the board position s as its input, passes it through many convolutional layers with parameters σ (SL policy network) or ρ (RL policy network), and outputs a probability

games of self-play with the RL policy network. Finally, a value network vθ is trained by regression to predict the expected outcome (that is, whether the current player wins) in positions from the self-play data set.  b, Schematic representation of the neural network architecture used in AlphaGo. The policy network takes a representation of the board position s as its input, passes it through many convolutional layers with parameters σ (SL policy network) or ρ (RL policy network), and outputs a probability distribution (|)σpas or (|)ρpas over legal moves a, represented by a probability map over the board. The value network similarly uses many convolutional layers with parameters θ, but outputs a scalar value vθ(s′) that predicts the expected outcome in position s′.RegressionClassificationClassificationSelf PlayPolicy gradient

Figure 2 | Strength and accuracy of policy and value networks.  a, Plot showing the playing strength of policy networks as a function of their training accuracy. Policy networks with 128, 192, 256 and 384 convolutional filters per layer were evaluated periodically during training; the plot shows the winning rate of AlphaGo using that policy network against the match version of AlphaGo. b, Comparison of evaluation accuracy between the value network and rollouts with different policies. Positions and outcomes were sampled from human expert games. Each position was evaluated by a single forward pass of the value network vθ, or by the mean outcome of 100 rollouts, played out using either uniform random rollouts, the fast rollout policy pπ, the SL policy network pσ or the RL policy network pρ. The mean squared error between the predicted value and the actual game outcome is plotted against the stage of the game (how many moves had been played in the given position).154575105135165195225255>285Move

and outcomes were sampled from human expert games. Each position was evaluated by a single forward pass of the value network vθ, or by the mean outcome of 100 rollouts, played out using either uniform random rollouts, the fast rollout policy pπ, the SL policy network pσ or the RL policy network pρ. The mean squared error between the predicted value and the actual game outcome is plotted against the stage of the game (how many moves had been played in the given position).154575105135165195225255>285Move number0.100.150.200.250.300.350.400.450.50Mean squared erroron expert gamesUniform random rollout policyFast rollout policyValue networkSL policy networkRL policy network50515253545556575859Training accuracy on KGS dataset (%)010203040506070128 filters192 filters256 filters384 filtersAlphaGo win rate (%)ab

© 2016 Macmillan Publishers Limited. All rights reservedRollout Policy         SL policy network                       RL policy Network          Value NetworkPolicy gradientSupervised LearningMC Policy EvaluationSelf PlaySupervised Learning]]NetworksDataFigure 16.6: AlphaGo pipeline. Adapted with permission from Macmillan Publishers Ltd:
Nature ,v o l .5 2 9 ( 7 5 8 7 ) ,p .4 8 5 , c2016.
Here is some more detail about AlphaGo ’s ANNs and their training. The identically-
structured SL and RL policy networks were similar to DQN’s deep convolutional network
described in Section 16.5 for playing Atari games, except that they had 13 convolutional
layers with the ﬁnal layer consisting of a soft-max unit for each point on the 19 ⇥19
Go board. The networks’ input was a 19 ⇥19⇥48 image stack in which each point
on the Go board was represented by the values of 48 binary or integer-valued features.
For example, for each point, one feature indicated if the point was occupied by one of

AlphaGo ’s stones, one of its opponent’s stones, or was unoccupied, thus providing the
“raw” representation of the board conﬁguration. Other features were based on the rules
of Go, such as the number of adjacent points that were empty, the number of opponent
stones that would be captured by placing a stone there, the number of turns since a stone
was placed there, and other features that the design team considered to be important.
Training the SL policy network took approximately 3 weeks using a distributed

implementation of stochastic gradient ascent on 50 processors. The network achieved 57%
accuracy, where the best accuracy achieved by other groups at the time of publication

446 Chapter 16: Applications and Case Studies
was 44.4%. Training the RL policy network was done by policy gradient reinforcement
learning over simulated games between the RL policy network’s current policy and
opponents using policies randomly selected from policies produced by earlier iterations
of the learning algorithm. Playing against a randomly selected collection of opponents
prevented overﬁtting to the current policy. The reward signal was +1 if the current
policy won, 1 if it lost, and zero otherwise. These games directly pitted the two policies
against one another without involving MCTS. By simulating many games in parallel on
50 processors, the DeepMind team trained the RL policy network on a million games
in a single day. In testing the ﬁnal RL policy, they found that it won more than 80%
of games played against the SL policy, and it won 85% of games played against a Go
program using MCTS that simulated 100,000 games per move.

The value network, whose structure was similar to that of the SL and RL policy
networks except for its single output unit, received the same input as the SL and RL
policy networks with the exception that there was an additional binary feature giving
the current color to play. Monte Carlo policy evaluation was used to train the network
from data obtained from a large number of self-play games played using the RL policy.
To avoid overﬁtting and instability due to the strong correlations between positions

encountered in self-play, the DeepMind team constructed a data set of 30 million positions
each chosen randomly from a unique self-play game. Then training was done using 50
million mini-batches each of 32 positions drawn from this data set. Training took one
week on 50 GPUs.
The rollout policy was learned prior to play by a simple linear network trained by
supervised learning from a corpus of 8 million human moves. The rollout policy network

had to output actions quickly while still being reasonably accurate. In principle, the SL
or RL policy networks could have been used in the rollouts, but the forward propagation
through these deep networks took too much time for either of them to be used in rollout
simulations, a great many of which had to be carried out for each move decision during
live play. For this reason, the rollout policy network was less complex than the other
policy networks, and its input features could be computed more quickly than the features

used for the policy networks. The rollout policy network allowed approximately 1,000
complete game simulations per second to be run on each of the processing threads that
AlphaGo used.
One may wonder why the SL policy was used instead of the better RL policy to select
actions in the expansion phase of APV-MCTS. These policies took the same amount of
time to compute because they used the same network architecture. The team actually
found that AlphaGo played better against human opponents when APV-MCTS used as

the SL policy instead of the RL policy. They conjectured that the reason for this was
that the latter was tuned to respond to optimal moves rather than to the broader set
of moves characteristic of human play. Interestingly, the situation was reversed for the
value function used by APV-MCTS. They found that when APV-MCTS used the value
function derived from the RL policy, it performed better than if it used the value function
derived from the SL policy.
Several methods worked together to produce AlphaGo ’s impressive playing skill. The

DeepMind team evaluated di↵erent versions of AlphaGo in order to assess the contributions

16.6. Mastering the Game of Go 447
made by these various components. The parameter ⌘in (16.4) controlled the mixing
of game state evaluations produced by the value network and by rollouts. With ⌘= 0,
AlphaGo used just the value network without rollouts, and with ⌘= 1, evaluation
relied just on rollouts. They found that AlphaGo using just the value network played
better than the rollout-only AlphaGo , and in fact played better than the strongest of all
other Go programs existing at the time. The best play resulted from setting ⌘=0.5,
indicating that combining the value network with rollouts was particularly important
toAlphaGo ’s success. These evaluation methods complemented one another: the value
network evaluated the high-performance RL policy that was too slow to be used in live
play, while rollouts using the weaker but much faster rollout policy were able to add
precision to the value network’s evaluations for speciﬁc states that occurred during games.

Overall, AlphaGo ’s remarkable success fueled a new round of enthusiasm for the promise
of artiﬁcial intelligence, speciﬁcally for systems combining reinforcement learning with
deep ANNs, to address problems in other challenging domains.
16.6.2 AlphaGo Zero
Building upon the experience with AlphaGo , a DeepMind team developed AlphaGo Zero
(Silver et al. 2017a). In contrast to AlphaGo , this program used no human data or
guidance beyond the basic rules of the game (hence the Zero in its name). It learned

exclusively from self-play reinforcement learning, with input giving just “raw” descriptions
of the placements of stones on the Go board. AlphaGo Zero implemented a form of
policy iteration (Section 4.3), interleaving policy evaluation with policy improvement.
Figure 16.7 is an overview of AlphaGo Zero ’s algorithm. A signiﬁcant di↵erence between
AlphaGo Zero andAlphaGo is that AlphaGo Zero used MCTS to select moves throughout

self-play reinforcement learning, whereas AlphaGo used MCTS for live play after—but not
during—learning. Other di↵erences besides not using any human data or human-crafted
features are that AlphaGo Zero used only one deep convolutional ANN and used a simpler
version of MCTS.
AlphaGo Zero ’s MCTS was simpler than the version used by AlphaGo in that it did
not include rollouts of complete games, and therefore did not need a rollout policy. Each
iteration of AlphaGo Zero ’s MCTS ran a simulation that ended at a leaf node of the

current search tree instead of at the terminal position of a complete game simulation.
But as in AlphaGo , each iteration of MCTS in AlphaGo Zero was guided by the output
of a deep convolutional network, labeled f✓in Figure 16.7, were ✓is the network’s weight
vector. The input to the network, whose architecture we describe below, consisted of raw
representations of board positions, and its output had two parts: a scalar value, v, an
estimate of the probability that the current player will win from from the current board

position, and a vector, p, of move probabilities, one for each possible stone placement on
the current board, plus the pass, or resign, move.
Instead of selecting self-play actions according to the probabilities p,h o w e v e r , AlphaGo
Zero used these probabilities, together with the network’s value output, to direct each
execution of MCTS, which returned new move probabilities, shown in Figure 16.7 as the
policies ⇡i. These policies beneﬁtted from the many simulations that MCTS conducted

games against itself, one shown here as a sequence of board positions si,i=1,2,...,T ,w i t h
moves ai,i=1,2,...,T ,a n dw i n n e r z.E a c hm o v e aiwas determined by action probabilities ⇡i
returned by MCTS executed from root node siand guided by a deep convolutional network,
here labeled f✓,w i t hl a t e s tw e i g h t s ✓. Shown here for just one position sbut repeated for all
si, the network’s inputs were raw representations of board positions si(together with several
past positions, though not shown here), and its outputs were vectors pof move probabilities
that guided MCTS’s forward searches, and scalar values vthat estimated the probability of the
current player winning from each position si. b) Deep convolutional network training. Training
examples were randomly sampled steps from recent self-play games. Weights ✓were updated
to move the policy vector ptoward the probabilities ⇡returned by MCTS, and to include the

winners zin the estimated win probability v. Reprinted from draft of Silver et al. (2017a) with
permission of the authors and DeepMind.
each time it executed. The result was that the policy actually followed by AlphaGo
Zero was an improvement over the policy given by the network’s outputs p. Silver et al.
(2017a) wrote that “MCTS may therefore be viewed as a powerful policy improvement
operator.”
Here is more detail about AlphaGo Zero ’s ANN and how it was trained. The network

took as input a 19 ⇥19⇥17 image stack consisting of 17 binary feature planes. The ﬁrst
8 feature planes were raw representations of the positions of the current player’s stones in
the current and seven past board conﬁgurations: a feature value was 1 if a player’s stone
was on the corresponding point, and was 0 otherwise. The next 8 feature planes similarly
coded the positions of the opponent’s stones. A ﬁnal input feature plane had a constant
value indicating the color of the current play: 1 for black; 0 for white. Because repetition

is not allowed in Go and one player is given some number of “compensation points” for
not getting the ﬁrst move, the current board position is not a Markov state of Go. This

16.6. Mastering the Game of Go 449
is why features describing past board positions and the color feature were needed.
The network was “two-headed,” meaning that after a number of initial layers, the
network split into two separate “heads” of additional layers that separately fed into two
sets of output units. In this case, one head fed 362 output units producing 192+1m o v e
probabilities p, one for each possible stone placement plus pass; the other head fed just
one output unit producing the scalar v, an estimate of the probability that the current
player will win from the current board position. The network before the split consisted of
41 convolutional layers, each followed by batch normalization, and with skip connections
added to implement residual learning by pairs of layers (see Section 9.6). Overall, move
probabilities and values were computed by 43 and 44 layers respectively.
Starting with random weights, the network was trained by stochastic gradient descent

(with momentum, regularization, and step-size parameter decreasing as training continues)
using batches of examples sampled uniformly at random from all the steps of the most
recent 500,000 games of self-play with the current best policy. Extra noise was added
to the network’s output pto encourage exploration of all possible moves. At periodic
checkpoints during training, which Silver et al. (2017a) chose to be at every 1,000 training
steps, the policy output by the ANN with the latest weights was evaluated by simulating

400 games (using MCTS with 1,600 iterations to select each move) against the current
best policy. If the new policy won (by a margin set to reduce noise in the outcome), then
it became the best policy to be used in subsequent self-play. The network’s weights were
updated to make the network’s policy output pmore closely match the policy returned
by MCTS, and to make its value output, v, more closely match the probability that the
current best policy wins from the board position represented by the network’s input.

The DeepMind team trained AlphaGo Zero over 4.9 million games of self-play, which
took about 3 days. Each move of each game was selected by running MCTS for 1,600
iterations, taking approximately 0.4 second per move. Network weights were updated over
700,000 batches each consisting of 2,048 board conﬁgurations. They then ran tournaments
with the trained AlphaGo Zero playing against the version of AlphaGo that defeated Fan
Hui by 5 games to 0, and against the version that defeated Lee Sedol by 4 games to 1.

They used the Elo rating system to evaluate the relative performances of the programs.
The di↵erence between two Elo ratings is meant to predict the outcome of games between
the players. The Elo ratings of AlphaGo Zero , the version of AlphaGo that played against
Fan Hui, and the version that played against Lee Sedol were respectively 4,308, 3,144,
and 3,739. The gaps in these Elo ratings translate into predictions that AlphaGo Zero

would defeat these other programs with probabilities very close to one. In a match of 100
games between AlphaGo Zero , trained as described, and the exact version of AlphaGo
that defeated Lee Sedol held under the same conditions that were used in that match,
AlphaGo Zero defeated AlphaGo in all 100 games.
The DeepMind team also compared AlphaGo Zero with a program using an ANN with
the same architecture but trained by supervised learning to predict human moves in a
data set containing nearly 30 million positions from 160,000 games. They found that the

supervised-learning player initially played better than AlphaGo Zero , and was better at
predicting human expert moves, but played less well after AlphaGo Zero was trained for
a day. This suggested that AlphaGo Zero had discovered a strategy for playing that was

450 Chapter 16: Applications and Case Studies
di↵erent from how humans play. In fact, AlphaGo Zero discovered, and came to prefer,
some novel variations of classical move sequences.
Final tests of AlphaGo Zero ’s algorithm were conducted with a version having a larger
ANN and trained over 29 million self-play games, which took about 40 days, again starting
with random weights. This version achieved an Elo rating of 5,185. The team pitted
this version of AlphaGo Zero against a program called AlphaGo Master , the strongest
program at the time, that was identical to AlphaGo Zero but, like AlphaGo , used human
data and features. AlphaGo Master ’s Elo rating was 4,858, and it had defeated the
strongest human professional players 60 to 0 in online games. In a 100 game match,
AlphaGo Zero with the larger network and more extensive learning defeated AlphaGo
Master 89 games to 11, thus providing a convincing demonstration of the problem-solving
power of AlphaGo Zero ’s algorithm.

AlphaGo Zero soundly demonstrated that superhuman performance can be achieved
by pure reinforcement learning, augmented by a simple version of MCTS, and deep ANNs
with very minimal knowledge of the domain and no reliance on human data or guidance.
We will surely see systems inspired by the DeepMind accomplishments of both AlphaGo
andAlphaGo Zero applied to challenging problems in other domains.
Recently, yet a better program, AlphaZero , was described by Silver et al. (2017b) that

does not even incorporate knowledge of Go. AlphaZero is a general reinforcement learning
algorithm that improves over the world’s hitherto best programs in the diverse games of
Go, chess, and shogi.
16.7 Personalized Web Services
Personalizing web services such as the delivery of news articles or advertisements is one
approach to increasing users’ satisfaction with a website or to increase the yield of a
marketing campaign. A policy can recommend content considered to be the best for each

particular user based on a proﬁle of that user’s interests and preferences inferred from
their history of online activity. This is a natural domain for machine learning, and in
particular, for reinforcement learning. A reinforcement learning system can improve a
recommendation policy by making adjustments in response to user feedback. One way
to obtain user feedback is by means of website satisfaction surveys, but for acquiring
feedback in real time it is common to monitor user clicks as indicators of interest in a
link.

link.
A method long used in marketing called A/B testing is a simple type of reinforcement
learning used to decide which of two versions, A or B, of a website users prefer. Because
it is non-associative, like a two-armed bandit problem, this approach does not personalize
content delivery. Adding context consisting of features describing individual users and
the content to be delivered allows personalizing service. This has been formalized as a
contextual bandit problem (or an associative reinforcement learning problem, Section 2.9)

with the objective of maximizing the total number of user clicks. Li, Chu, Langford, and
Schapire (2010) applied a contextual bandit algorithm to the problem of personalizing
the Yahoo! Front Page Today webpage (one of the most visited pages on the internet at
the time of their research) by selecting the news story to feature. Their objective was to

16.7. Personalized Web Services 451
maximize the click-through rate (CTR), which is the ratio of the total number of clicks
all users make on a webpage to the total number of visits to the page. Their contextual
bandit algorithm improved over a standard non-associative bandit algorithm by 12.5%.
Theocharous, Thomas, and Ghavamzadeh (2015) argued that better results are possible
by formulating personalized recommendation as a Markov decision problem (MDP) with
the objective of maximizing the total number of clicks users make over repeated visits to
a website. Policies derived from the contextual bandit formulation are greedy in the sense
that they do not take long-term e↵ects of actions into account. These policies e↵ectively
treat each visit to a website as if it were made by a new visitor uniformly sampled from
the population of the website’s visitors. By not using the fact that many users repeatedly
visit the same websites, greedy policies do not take advantage of possibilities provided by

long-term interactions with individual users.
As an example of how a marketing strategy might take advantage of long-term user
interaction, Theocharous et al. contrasted a greedy policy with a longer-term policy for
displaying ads for buying a product, say a car. The ad displayed by the greedy policy
might o↵er a discount if the user buys the car immediately. A user either takes the o↵er
or leaves the website, and if they ever return to the site, they would likely see the same
o↵er. A longer-term policy, on the other hand, can transition the user “down a sales

funnel” before presenting the ﬁnal deal. It might start by describing the availability of
favorable ﬁnancing terms, then praise an excellent service department, and then, on the
next visit, o↵er the ﬁnal discount. This type of policy can result in more clicks by a user
over repeated visits to the site, and if the policy is suitably designed, more eventual sales.
Working at Adobe Systems Incorporated, Theocharous et al. conducted experiments
to see if policies designed to maximize clicks over the long term could in fact improve

over short-term greedy policies. The Adobe Marketing Cloud, a set of tools that many
companies use to run digital marketing campaigns, provides infrastructure for automating
user-targed advertising and fund-raising campaigns. Actually deploying novel policies
using these tools entails signiﬁcant risk because a new policy may end up performing
poorly. For this reason, the research team needed to assess what a policy’s performance
would be if it were to be actually deployed, but to do so on the basis of data collected

under the execution of other policies. A critical aspect of this research, then, was o↵-policy
evaluation. Further, the team wanted to do this with high conﬁdence to reduce the
risk of deploying a new policy. Although high conﬁdence o↵-policy evaluation was a
central component of this research (see also Thomas, 2015; Thomas, Theocharous, and
Ghavamzadeh, 2015), here we focus only on the algorithms and their results.
Theocharous et al. compared the results of two algorithms for learning ad recommen-

dation policies. The ﬁrst algorithm, which they called greedy optimization , had the goal
of maximizing only the probability of immediate clicks. As in the standard contextual
bandit formulation, this algorithm did not take the long-term e↵ects of recommendations
into account. The other algorithm, a reinforcement learning algorithm based on an MDP
formulation, aimed at improving the number of clicks users made over multiple visits to

a website. They called this latter algorithm life-time value (LTV) optimization. Both
algorithms faced challenging problems because the reward signal in this domain is very
sparse because users usually do not click on ads, and user clicking is very random so that

452 Chapter 16: Applications and Case Studies
returns have high variance.
Data sets from the banking industry were used for training and testing these algorithms.
The data sets consisted of many complete trajectories of customer interaction with a
bank’s website that showed each customer one out of a collection of possible o↵ers. If
a customer clicked, the reward was 1, and otherwise it was 0. One data set contained
approximately 200,000 interactions from a month of a bank’s campaign that randomly
o↵ered one of 7 o↵ers. The other data set from another bank’s campaign contained
4,000,000 interactions involving 12 possible o↵ers. All interactions included customer
features such as the time since the customer’s last visit to the website, the number of their
visits so far, the last time the customer clicked, geographic location, one of a collection of
interests, and features giving demographic information.
Greedy optimization was based on a mapping estimating the probability of a click

as a function of user features. The mapping was learned via supervised learning from
one of the data sets by means of a random forest (RF) algorithm (Breiman, 2001). RF
algorithms have been widely used for large-scale applications in industry because they are
e↵ective predictive tools that tend not to overﬁt and are relatively insensitive to outliers
and noise. Theocharous et al. then used the mapping to deﬁne an "-greedy policy that
selected with probability 1- "the o↵er predicted by the RF algorithm to have the highest

probability of producing a click, and otherwise selected from the other o↵ers uniformly at
random.
LTV optimization used a batch-mode reinforcement learning algorithm called ﬁtted
Q iteration (FQI). It is a variant of ﬁtted value iteration (Gordon, 1999) adapted to
Q-learning. Batch mode means that the entire data set for learning is available from
the start, as opposed to the online mode of the algorithms we focus on in this book in
which data are acquired sequentially while the learning algorithm executes. Batch-mode

random.
reinforcement learning algorithms are sometimes necessary when online learning is not
practical, and they can use any batch-mode supervised learning regression algorithm,
including algorithms known to scale well to high-dimensional spaces. The convergence of
FQI depends on properties of the function approximation algorithm (Gordon, 1999). For
their application to LTV optimization, Theocharous et al. used the same RF algorithm
they used for the greedy optimization approach. Because in this case FQI convergence

is not monotonic, Theocharous et al. kept track of the best FQI policy by o↵-policy
evaluation using a validation training set. The ﬁnal policy for testing the LTV approach
was the "-greedy policy based on the best policy produced by FQI with the initial
action-value function set to the mapping produced by the RF for the greedy optimization
approach.
To measure the performance of the policies produced by the greedy and LTV approaches,
Theocharous et al. used the CTR metric and a metric they called the LTV metric. These

approach.
metrics are similar, except that the LTV metric critically distinguishes between individual
website visitors:
CTR =Total # of Clicks
Total # of Visits,
LTV =Total # of Clicks
Total # of Visitors.

16.8. Thermal Soaring 453
Figure 16.8 illustrates how these metrics di↵er. Each circle represents a user visit to
the site; black circles are visits at which the user clicks. Each row represents visits by
a particular user. By not distinguishing between visitors, the CTR for these sequences
is 0.35, whereas the LTV is 1.5. Because LTV is larger than CTR to the extent that
individual users revisit the site, it is an indicator of how successful a policy is in encouraging
users to engage in extended interactions with the site.
Figure 16.8: Click through rate (CTR) versus life-time value (LTV). Each circle represents
a user visit; black circles are visits at which the user clicks. Adapted from Theocharous et
al. (2015).
Testing the policies produced by the greedy and LTV approaches was done using
a high conﬁdence o↵-policy evaluation method on a test data set consisting of real-
world interactions with a bank website served by a random policy. As expected, results

showed that greedy optimization performed best as measured by the CTR metric, while
LTV optimization performed best as measured by the LTV metric. Furthermore—
although we have omitted its details—the high conﬁdence o↵-policy evaluation method
provided probabilistic guarantees that the LTV optimization method would, with high
probability, produce policies that improve upon policies currently deployed. Assured by
these probabilistic guarantees, Adobe announced in 2016 that the new LTV algorithm

would be a standard component of the Adobe Marketing Cloud so that a retailer could
issue a sequence of o↵ers following a policy likely to yield higher return than a policy
that is insensitive to long-term results.
16.8 Thermal Soaring
Birds and gliders take advantage of upward air currents—thermals—to gain altitude in
order to maintain ﬂight while expending little, or no, energy. Thermal soaring, as this
behavior is called, is a complex skill requiring responding to subtle environmental cues

to increase altitude by exploiting a rising column of air for as long as possible. Reddy,
Celani, Sejnowski, and Vergassola (2016) used reinforcement learning to investigate
thermal soaring policies that are e↵ective in the strong atmospheric turbulence usually
accompanying rising air currents. Their primary goal was to provide insight into the
cues birds sense and how they use them to achieve their impressive thermal soaring
performance, but the results also contribute to technology relevant to autonomous gliders.

Reinforcement learning had previously been applied to the problem of navigating eciently

454 Chapter 16: Applications and Case Studies
to the vicinity of a thermal updraft (Woodbury, Dunn, and Valasek, 2014) but not to the
more challenging problem of soaring within the turbulence of the updraft itself.
Reddy et al. modeled the soaring problem as a continuing MDP with discounting.
The agent interacted with a detailed model of a glider ﬂying in turbulent air. They
devoted signiﬁcant e↵ort toward making the model generate realistic thermal soaring
conditions, including investigating several di↵erent approaches to atmospheric modeling.
For the learning experiments, air ﬂow in a three-dimensional box with one kilometer
sides, one of which was at ground level, was modeled by a sophisticated physics-based
set of partial di↵erential equations involving air velocity, temperature, and pressure.
Introducing small random perturbations into the numerical simulation caused the model
to produce analogs of thermal updrafts and accompanying turbulence (Figure 16.9 Left)

Glider ﬂight was modeled by aerodynamic equations involving velocity, lift, drag, and
other factors governing powerless ﬂight of a ﬁxed-wing aircraft. Maneuvering the glider
involved changing its angle of attack (the angle between the glider’s wing and the direction
of air ﬂow) and its bank angle (Figure 16.9 Right).

contribute significantly and more exploratory strategies arepreferred.The SARSA algorithm finds the optimal policy by estimatingfor every state–action pair itsQfunction defined as the expectedsum of future rewards given the current statesand the actiona.At each step, theQfunction is updated as follows:Qðs,aÞ→Qðs,aÞ+ηðr+βQðs′,a′Þ−Qðs,aÞÞ,[5]whereris the received reward andηis the learning rate. Theupdate is made online and does not require any prior model ofthe flow or the flight. This feature is particularly relevant inmodeling decision-making processes in animals. When the algo-rithm is close to convergence, theQfunction approaches thesolution to Bellman’s dynamic programming equations (12).The policyπas, which encodes the probability of choosing actionaat states, approaches the optimal oneπpand is obtained fromtheQfunction via a Boltzmann-like expression:πas∝exp/C0−^Qðs,aÞ/C14τtemp/C1,[6]^Qðs,aÞ=maxa′Qðs,a′Þ−Qðs,aÞmaxa′Qðs,a′Þ−mina′Qðs,a′Þ.[7]Here,τtempis an effective“temperature”: whenτtemp/C291, ac-tions

relevant inmodeling decision-making processes in animals. When the algo-rithm is close to convergence, theQfunction approaches thesolution to Bellman’s dynamic programming equations (12).The policyπas, which encodes the probability of choosing actionaat states, approaches the optimal oneπpand is obtained fromtheQfunction via a Boltzmann-like expression:πas∝exp/C0−^Qðs,aÞ/C14τtemp/C1,[6]^Qðs,aÞ=maxa′Qðs,a′Þ−Qðs,aÞmaxa′Qðs,a′Þ−mina′Qðs,a′Þ.[7]Here,τtempis an effective“temperature”: whenτtemp/C291, ac-tions are only weakly dependent on the associatedQfunction ;conversely, forτtempsmall, the policy greedily chooses the actionwith the largestQ. The temperature parameter is initially chosenlarge and lowered as training progresses to create an annealingeffect, thereby preventing the policy from getting stuck in localextrema. Parameters used in our simulations can be found inTable S1.In the sequel, we shall qualify the policy identified by SARSAas optimal. It should be understood, however, that the SARSAalgorithm

are only weakly dependent on the associatedQfunction ;conversely, forτtempsmall, the policy greedily chooses the actionwith the largestQ. The temperature parameter is initially chosenlarge and lowered as training progresses to create an annealingeffect, thereby preventing the policy from getting stuck in localextrema. Parameters used in our simulations can be found inTable S1.In the sequel, we shall qualify the policy identified by SARSAas optimal. It should be understood, however, that the SARSAalgorithm (as other reinforcement learning algorithms) typicallyidentifies an approximately optimal policy and“approximately”is skipped only for the sake of conciseness.ResultsSensorimotor Cues and Reward Function for Effective Learning.Keyaspects of the learning for the soaring problem are the sensori-motor cues that the glider can sense (state space) and the choiceof the reward used to train the glider to ascend quickly. As thestate and action spaces are continuous and high-dimensional, it isnecessary to discretize

other reinforcement learning algorithms) typicallyidentifies an approximately optimal policy and“approximately”is skipped only for the sake of conciseness.ResultsSensorimotor Cues and Reward Function for Effective Learning.Keyaspects of the learning for the soaring problem are the sensori-motor cues that the glider can sense (state space) and the choiceof the reward used to train the glider to ascend quickly. As thestate and action spaces are continuous and high-dimensional, it isnecessary to discretize them, which we realize here by a standardlookup table representation. The height ascended per trial,averaged over different realizations of the flow, serves as ourperformance criterion.The glider is allowed control over its angle of attack and itsbank angle (Fig. 1B). Control over the angle of attack featurestwo regimes: (i) at small angles of attack, the horizontal speed islarge and the climb rate is small (the glider sinks quickly); (ii)a tlarge angles of attack but below the stall angle, the

discretize them, which we realize here by a standardlookup table representation. The height ascended per trial,averaged over different realizations of the flow, serves as ourperformance criterion.The glider is allowed control over its angle of attack and itsbank angle (Fig. 1B). Control over the angle of attack featurestwo regimes: (i) at small angles of attack, the horizontal speed islarge and the climb rate is small (the glider sinks quickly); (ii)a tlarge angles of attack but below the stall angle, the horizontalspeed is small, whereas the climb rate is large. The bank anglecontrols the heading of the glider, and we allow for a range ofvariation between−15° and 15°. Exploring various possibilities,we found that three actions are minimally sufficient: increasing,decreasing, or preserving the angle of attack and the bank angle.The angle of attack and bank angle were incremented/decre-mented in steps of 2.5° and 5°, respectively. In summary, theglider can choose 32possible actions to control its navigation

horizontalspeed is small, whereas the climb rate is large. The bank anglecontrols the heading of the glider, and we allow for a range ofvariation between−15° and 15°. Exploring various possibilities,we found that three actions are minimally sufficient: increasing,decreasing, or preserving the angle of attack and the bank angle.The angle of attack and bank angle were incremented/decre-mented in steps of 2.5° and 5°, respectively. In summary, theglider can choose 32possible actions to control its navigation inresponse to the sensorimotor cues described hereafter.Our rationale in the choice of the state space was trying tominimize biological or electronic sensory devices necessary forcontrol. We tested different combinations of local sensorimotor

A
CzyLift L
zxLift LDrag Dvelocity directionwing directionbank angle
glide angleangle of attackB
D

Fig. 1.Snapshots of the vertical velocity (A)a n dt h et e m p e r a t u r ef i e l d s(B)i no u rn u m e r i c a ls i m u l a t i o n so f3 DR a y l e i g h–Bénard convection. For the verticalvelocity field, the red and blue colors indicate regions of large upward and downward flow, respectively. For the temperature field, the red and blue colorsindicate regions of high and low temperature, respectively. Notice that the hot and cold regions drive the upward and downward branches of the convectivecell, in agreement with the basic physics of convection. (C)T h ef o r c e - b o d yd i a g r a mo ff l i g h tw i t hn ot h r u s t ,t h a ti s ,w i t h o u ta n ye n g i n eo rf l a p p i n go fw i n g s .The figure also shows the bank angleμ(blue), the angle of attackα(green), and the glide angleγ(red). (D)T h er a n g eo fh o r i z o n t a ls p e e d sa n dc l i m br a t e saccessible by controlling the angle of attack. At small angles of attack, the glider moves fast but also sinks fast, whereas at larger

in agreement with the basic physics of convection. (C)T h ef o r c e - b o d yd i a g r a mo ff l i g h tw i t hn ot h r u s t ,t h a ti s ,w i t h o u ta n ye n g i n eo rf l a p p i n go fw i n g s .The figure also shows the bank angleμ(blue), the angle of attackα(green), and the glide angleγ(red). (D)T h er a n g eo fh o r i z o n t a ls p e e d sa n dc l i m br a t e saccessible by controlling the angle of attack. At small angles of attack, the glider moves fast but also sinks fast, whereas at larger angles, the glider moves andsinks more slowly. If the angle of attack is too high, at about 16°, the glider stalls, leading to a sudden drop in lift. The vertical black dashed line shows thefixed angle of attack for most of the simulations (Results,Control over the Angle of Attack).

Reddy et al.PNAS|Published online August 1, 2016|E4879NEUROSCIENCEPHYSICSPNAS PLUS

Reddy et al.PNAS|Published online August 1, 2016|E4879NEUROSCIENCEPHYSICSPNAS PLUS↵
Figure 16.9: Thermal soaring model: Left: snapshot of the vertical velocity ﬁeld of the
simulated cube of air: in red (blue) is a region of large upward (downward) ﬂow. Right: diagram
of powerless ﬂight showing bank angle µand angle of attack ↵. Adapted with permission From
PNAS vol. 113(22), p. E4879, 2016, Reddy, Celani, Sejnowski, and Vergassola, Learning to Soar
in Turbulent Environments.
The interface between the agent and the environment required deﬁning the agent’s
actions, the state information the agent receives from the environment, and the reward
signal. By experimenting with various possibilities, Reddy et al. decided that three actions
each for the angle of attack and the bank angle were enough for their purposes: increment
or decrement the current bank angle and angle of attack by 5and 2 .5, respectively, or
leave them unchanged. This resulted in 32possible actions. The bank angle was bounded

16.8. Thermal Soaring 455
the authors tried various sets of signals as input to the reinforcement learning agent.
They started by using state aggregation (Section 9.3) of a four-dimensional state space
with dimensions giving local vertical wind speed, local vertical wind acceleration, torque
depending on the di↵erence between the vertical wind velocities at the left and right wing
tips, and the local temperature. Each dimension was discretized into three bins: positive
high, negative high, and small. Results, described below, showed that only two of these
dimensions were critical for e↵ective soaring behavior.
The overall objective of thermal soaring is to gain as much altitude as possible from
each rising column of air. Reddy et al. tried a straightforward reward signal that rewarded
the agent at the end of each episode based on the altitude gained over the episode, a
large negative reward signal if the glider touched the ground, and zero otherwise. They

found that learning was not successful with this reward signal for episodes of realistic
duration and that eligibility traces did not help. By experimenting with various reward
signals, they found that learning was best with a reward signal that at each time step
linearly combined the vertical wind velocity and vertical wind acceleration observed on
the previous time step.
Learning was by one-step Sarsa, with actions selected according to a soft-max dis-

tribution based on normalized action values. Speciﬁcally, the action probabilities were
computed according to (13.2) with action preferences:
h(s, a,✓)=ˆq(s, a,✓)min bˆq(s, b,✓)
⌧
max bˆq(s, b,✓)min bˆq(s, b,✓),
where ✓is a parameter vector with one component for each action and aggregated group
of states, and ˆq(s, a,✓) merely returned the component corresponding to s, ain the usual
way for state aggregation methods. The above equation forms the action preferences
by normalizing the approximate action values to the interval [0 ,1] then dividing by ⌧,a

⌧
positive “temperature parameter.”3As⌧increases, the probability of selecting an action
becomes less dependent on its preference; as ⌧decreases toward zero, the probability of
selecting the most highly-preferred action approaches one, making the policy approach
the greedy policy. The temperature parameter ⌧was initialized to 2.0 and incrementally
decreased to 0.2 during learning. Action preferences were computed from the current
estimates of the action values: the action with the maximum estimated action value was

given preference 1 /⌧, the action with the minimum estimated action value was given
preference 0, and the preferences of the other actions were scaled between these extremes.
The step-size and discount-rate parameters were ﬁxed at 0.1 and 0.98 respectively.
Each learning episode took place with the agent controlling simulated ﬂight in an
independently generated period of simulated turbulent air currents. Each episode lasted
2.5 minutes simulated with a 1 second time step. Learning e↵ectively converged after a

few hundred episodes. The left panel of Figure 16.10 shows a sample trajectory before
learning where the agent selects actions randomly. Starting at the top of the volume
shown, the glider’s trajectory is in the direction indicated by the arrow and quickly loses
altitude. Figure 16.10’s right panel is a trajectory after learning. The glider starts at the
same place (here appearing at the bottom of the volume) and gains altitude by spiraling
3Reddy et al. described this slightly di↵erently, but our version is equivalent to theirs.

456 Chapter 16: Applications and Case Studies
(a)(b)Figure 16.10: Sample thermal soaring trajectories, with arrows showing the direction of
ﬂight from the same starting point (note that the altitude scales are shifted). Left: before
learning: the agent selects actions randomly and the glider descends. Right: after learning:
the glider gains altitude by following a spiral trajectory. Adapted with permission from PNAS
vol. 113(22), p. E4879, 2016, Reddy, Celani, Sejnowski, and Vergassola, Learning to Soar in
Turbulent Environments.
within the rising column of air. Although Reddy et al. found that performance varied
widely over di↵erent simulated periods of air ﬂow, the number of times the glider touched
the ground consistently decreased to nearly zero as learning progressed.
After experimenting with di↵erent sets of features available to the learning agent, it
turned out that the combination of just vertical wind acceleration and torques worked

best. The authors conjectured that because these features give information about the
gradient of vertical wind velocity in two di↵erent directions, they allow the controller to
select between turning by changing the bank angle or continuing along the same course
by leaving the bank angle alone. This allows the glider to stay within a rising column of
air. Vertical wind velocity is indicative of the strength of the thermal but does not help
in staying within the ﬂow. They found that sensitivity to temperature was of little help.

They also found that controlling the angle of attack is not helpful in staying within a
particular thermal, being useful instead for traveling between thermals when covering
large distances, as in cross-country gliding and bird migration.
Due to the fact that soaring in di↵erent levels of turbulence requires di↵erent policies,
training was done in conditions ranging from weak to strong turbulence. In strong
turbulence the rapidly changing wind and glider velocities allowed less time for the

controller to react. This reduced the amount of control possible compared to what
was possible for maneuvering when ﬂuctuations were weak. Reddy et al. examined the
policies Sarsa learned under these di↵erent conditions. Common to policies learned in all
regimes were these features: when sensing negative wind acceleration, bank sharply in the
direction of the wing with the higher lift; when sensing large positive wind acceleration
and no torque, do nothing. However, di↵erent levels of turbulence led to policy di↵erences.

16.8. Thermal Soaring 457
Policies learned in strong turbulence were more conservative in that they preferred small
bank angles, whereas in weak turbulence, the best action was to turn as much as possible
by banking sharply. Systematic study of the bank angles preferred by the policies learned
under the di↵erent conditions led the authors to suggest that by detecting when vertical
wind acceleration crosses a certain threshold the controller can adjust its policy to cope
with di↵erent turbulence regimes.
Reddy et al. also conducted experiments to investigate the e↵ect of the discount-rate
parameter on the performance of the learned policies. They found that the altitude
gained in an episode increased as increased, reaching a maximum for =.99, suggesting
that e↵ective thermal soaring requires taking into account long-term e↵ects of control
This computational study of thermal soaring illustrates how reinforcement learning

can further progress toward di↵erent kinds of objectives. Learning policies having
access to di↵erent sets of environmental cues and control actions contributes to both
the engineering objective of designing autonomous gliders and the scientiﬁc objective of
improving understanding of the soaring skills of birds. In both cases, hypotheses resulting
from the learning experiments can be tested in the ﬁeld by instrumenting real gliders and
by comparing predictions with observed bird soaring behavior.

Chapter 17
Frontiers
In this ﬁnal chapter we touch on some topics that are beyond the scope of this book but
that we see as particularly important for the future of reinforcement learning. Many of
these topics bring us beyond what is reliably known, and some bring us beyond the MDP
framework.
17.1 General Value Functions and Auxiliary Tasks
Over the course of this book, our notion of value function has become quite general.
With o↵-policy learning we allowed a value function to be conditional on an arbitrary
target policy. Then in Section 12.8 we generalized discounting to a termination function
:S7![0,1], so that a di↵erent discount rate could be applied at each time step in
determining the return (12.17) . This allowed us to express predictions about how much
reward we will get over an arbitrary, state-dependent horizon. The next, and perhaps
ﬁnal, step is to generalize beyond rewards to permit predictions about arbitrary signals.

460 Chapter 17: Frontiers
is in the form of a value function and thus can be learned in the usual ways using the
methods developed in this book for learning approximate value functions. Along with
the learned predictions, we might also learn policies to maximize the predictions in the
usual ways by Generalized Policy Iteration (Section 4.6) or by actor–critic methods. In
this way an agent could learn to predict and control great numbers of signals, not just
long-term reward.
Why might it be useful to predict and control signals other than long-term reward?
These are auxiliary tasks in that they are extra, in-addition-to, the main task of maximiz-
ing reward. One answer is that the ability to predict and control a diverse multitude of
signals can constitute a powerful kind of environmental model. As we saw in Chapter 8,
a good model can enable the agent to get reward more eciently. It takes a couple of
further concepts to develop this answer clearly, so we postpone it to the next section.

First let’s consider two simpler ways in which a multitude of diverse predictions can be
helpful to a reinforcement learning agent.
One simple way in which auxiliary tasks can help on the main task is that they may
require some of the same representations as are needed on the main task. Some of the
auxiliary tasks may be easier, with less delay and a clearer connection between actions
and outcomes. If good features can be found early on easy auxilary tasks, then those
features may signiﬁcantly speed learning on the main task. There is no necessary reason

why this has to be true, but in many cases it seems plausible. For example, if you learn
to predict and control your sensors over short time scales, say seconds, then you might
plausibly come up with part of the idea of objects, which would then greatly help with
the prediction and control of long-term reward.
We might imagine an artiﬁcial neural network (ANN) in which the last layer is split
into multiple parts, or heads , each working on a di↵erent task. One head might produce

the approximate value function for the main task (with reward as its cumulant) whereas
the others would produce solutions to various auxilary tasks. All heads could propagate
errors by stochastic gradient descent into the same body—the shared preceding part
of the network—which would then try to form representations, in its next-to-last layer,
to support all the heads. Researchers have experimented with auxiliary tasks such as
predicting change in pixels, predicting the next time step’s reward, and predicting the

distribution of the return. In many cases this approach has been shown to greatly
accelerate learning on the main task (Jaderberg et al., 2017). Multiple predictions
have similarly been repeatedly proposed as a way of directing the construction of state
estimates (see Section 17.3).
Another simple way in which the learning of auxiliary tasks can improve performance
is best explained by analogy to the psychological phenomena of classical conditioning
(Section 14.2). One way of understanding classical conditioning is that evolution has

built in a reﬂexive (non-learned) association to a particular action from the prediction
of a particular signal. For example, humans and many other animals appear to have a
built-in reﬂex to blink whenever their prediction of being poked in the eye exceeds some
threshold. The prediction is learned, but the association from prediction to eye closure
is built in, and thus the animal is saved many unprotected pokes in its eye. Similarly,
the association from fear to increased heart rate, or to freezing, can be built in. Agent

17.2. Temporal Abstraction via Options 461
designers can do something similar, connecting by design (without learning) predictions
of speciﬁc events to predetermined actions. For example, a self-driving car that learns to
predict whether going forward will produce a collision could be given a built-in reﬂex to
stop, or to turn away, whenever the prediction is above some threshold. Or consider a
vacuum-cleaning robot that learned to predict whether it might run out of battery power
before returning to the charger and that reﬂexively headed back to the charger whenever
the prediction became non-zero. The correct prediction would depend on the size of the
house, the room the robot was in, and the age of the battery, all of which would be hard
for the robot designer to know. It would be dicult for the designer to build in a reliable
algorithm for deciding whether to head back to the charger in sensory terms, but it might
be easy to do this in terms of the learned prediction. We foresee many possible ways

like this in which learned predictions might combine usefully with built-in algorithms for
controlling behavior.
Finally, perhaps the most important role for auxiliary tasks is in moving beyond the
assumption we have made throughout this book that the state representation is ﬁxed
and given to the agent. To explain this role, we ﬁrst have to take a few steps back to
appreciate the magnitude of this assumption and the implications of removing it. We do
that in Section 17.3.
17.2 Temporal Abstraction via Options

An appealing aspect of the MDP formalism is that it can be applied usefully to tasks
at many di↵erent time scales. One can use it to formalize the task of deciding which
muscles to twitch to grasp an object, which airplane ﬂight to take to arrive conveniently
at a distant city, and which job to take to lead a satisfying life. These tasks di↵er greatly
in their time scales, yet each can be usefully formulated as an MDP that can be solved
by planning or learning processes as described in this book. All involve interaction with

the world, sequential decision making, and a goal usefully conceived of as accumulating
rewards over time, and so all can be formulated as MDPs.
Although all these tasks can be formulated as MDPs, one might think that they cannot
be formulated as a single MDP. They involve such di↵erent time scales, such di↵erent
notions of choice and action! It would be no good, for example, to plan a ﬂight across a
continent at the level of muscle twitches. Yet for other tasks, grasping, throwing darts,

or hitting a baseball, low-level muscle twitches may be just the right level. People do
all these things seamlessly without appearing to switch between levels. Can the MDP
framework be stretched to cover all the levels simultaneously?
Perhaps it can. One popular idea is to formalize an MDP at a detailed level, with a
small time step, yet enable planning at higher levels using extended courses of action that
correspond to many base-level time steps. To do this we need a notion of course of action

that extends over many time steps and includes a notion of termination. A general way
to formulate these two ideas is as a policy, ⇡, and a state-dependent termination function,
, as in GVFs. We deﬁne a pair of these as a generalized notion of action termed an
option . To execute an option !=h⇡!,!iat time tis to obtain the action to take, At,
from ⇡!(·|St), then terminate at time t+ 1 with probability !(St+1). If the option does

462 Chapter 17: Frontiers
not terminate at t+ 1, then At+1is selected from ⇡!(·|St+1), and the option terminates
att+ 2 with probability !(St+2), and so on until eventual termination. It is convenient
to consider low-level actions to be special cases of options—each action acorresponds
to an option h⇡!,!iwhose policy picks the action ( ⇡!(s)=afor all s2S) and whose
termination function is zero ( !(s) = 0 for all s2S+). Options e↵ectively extend the
action space. The agent can either select a low-level action/option, terminating after one
time step, or select an extended option that might execute for many time steps before
terminating.
Options are designed so that they are interchangable with low-level actions. For
example, the notion of an action-value function q⇡naturally generalizes to an option -
value function that takes a state and option as input and returns the expected return
starting from that state, executing that option to termination, and thereafter following

the policy, ⇡. We can also generalize the notion of policy to a hierarchical policy that
selects from options rather than actions, where options, when selected, execute until
termination. With these ideas, many of the algorithms in this book can be generalized to
learn approximate option-value functions and hierarchical policies. In the simplest case,
the learning process ‘jumps’ from option initiation to option termination, with an update
only occurring when an option terminates. More subtly, updates can be made on each

time step, using “intra-option” learning algorithms, which in general require o↵-policy
learning.
Perhaps the most important generalization made possible by option ideas is that of the
environmental model as developed in Chapters 3, 4, and 8. The conventional model of an
action is the state-transition probabilities and the expected immediate reward for taking
the action in each state. How do conventional action models generalize to option models ?
For options, the appropriate model is again of two parts, one corresponding to the state

17.2. Temporal Abstraction via Options 463
The above deﬁnition of the transition part of an option model allows us to formulate
Bellman equations and dynamic programming algorithms that apply to all options,
including low-level actions as a special case. For example, the general Bellman equation
for the state values of a hierarchical policy ⇡is
v⇡(s)=X
!2⌦(s)⇡(!|s)"
r(s, !)+X
s0p(s0|s, !)v⇡(s0)#
, (17.4)
where ⌦( s) denotes the set of options available in state s.I f ⌦ ( s) includes only the
low-level actions, then this equation reduces to a version of the usual Bellman equation
(3.14) , except of course is included in the new p(17.3) and thus does not appear.
Similarly, the corresponding planning algorithms also have no . For example, the value
iteration algorithm with options, analogous to (4.10), is
!2⌦(s)"
r(s, !)+X
s0p(s0|s, !)vk(s0)#
,for all s2S.
If ⌦( s) includes all the low-level actions available in each s, then this algorithm converges

!2⌦(s)"
r(s, !)+X
to the conventional v⇤, from which the optimal policy can be computed. However, it
is particularly useful to plan with options when only a subset of the possible options
are considered (in ⌦( s)) in each state. Value iteration will then converge to the best
hierarchical policy limited to the restricted set of options. Although this policy may be
sub-optimal, convergence can be much faster because fewer options are considered and
because each option can jump over many time steps.

To plan with options, one must either be given the option models, or learn them. One
natural way to learn an option model is to formulate it as a collection of GVFs (as deﬁned
in the preceding section) and then learn the GVFs using the methods presented in this
book. It is not dicult to see how this could be done for the reward part of the option
model. One merely chooses one GVF’s cumulant to be the reward ( Ct=Rt), its policy to
be the the option’s policy ( ⇡=⇡!), and its termination function to be the discount rate

reward part of the option model, v⇡,,C(s)=r(s, !), and the learning methods described
in this book can be used to approximate it. The state-transition part of the option model
is only a little more complicated. One needs to allocate one GVF for each state that the
option might terminate in. We don’t want these GVFs to accumulate anything except
when the option terminates, and then only when the termination is in the appropriate
state. This can be achieved by choosing the cumulant of the GVF that predicts transition
to state s0to be Ct=(St)·
St=s0. The GVF’s policy and termination functions are
chosen the same as for the reward part of the option model. The true GVF then equals
thes0portion of the option’s state-transition model, v⇡,,C(s)=p(s0|s, !), and again this
book’s methods could be employed to learn it. Although each of these steps is seemingly
natural, putting them all together (including function approximation and other essential

components) is quite challenging and beyond the current state of the art.

464 Chapter 17: Frontiers
Exercise 17.1 This section has presented options for the discounted case, but discounting
is arguably inappropriate for control when using function approximation (Section 10.4).
What is the natural Bellman equation for a hierarchical policy, analogous to (17.4) ,b u t
for the average reward setting (Section 10.3)? What are the two parts of the option
model, analogous to (17.2) and (17.3), for the average reward setting? ⇤
17.3 Observations and State
Throughout this book we have written the learned approximate value functions (and
the policies in Chapter 13) as functions of the environment’s state. This is a signiﬁcant
limitation of the methods presented in Part I, in which the learned value function was
implemented as a table such that any value function could be exactly approximated;
that case is tantamount to assuming that the state of the environment is completely
observed by the agent. But in many cases of interest, and certainly in the lives of all

natural intelligences, the sensory input gives only partial information about the state of
the world. Some objects may be occluded by others, or behind the agent, or miles away.
In these cases, potentially important aspects of the environment’s state are not directly
observable, and it is a strong, unrealistic, and limiting assumption to assume that the
learned value function is implemented as a table over the environment’s state space.
The framework of parametric function approximation that we developed in Part II is far

less restrictive and, arguably, no limitation at all. In Part II we retained the assumption
that the learned value functions (and policies) are functions of the environment’s state,
but allowed these functions to be arbitrarily restricted by the parameterization. It is
somewhat surprising and not widely recognized that function approximation includes
important aspects of partial observability. For example, if there is a state variable that is
not observable, then the parameterization can be chosen such that the approximate value

does not depend on that state variable. The e↵ect is just as if the state variable were not
observable. Because of this, all the results obtained for the parameterized case apply to
partial observability without change. In this sense, the case of parameterized function
approximation includes the case of partial observability.
Nevertheless, there are many issues that cannot be investigated without a more explicit
treatment of partial observability. Although we cannot give them a full treatment here,

we can outline the changes that would be needed to do so. There are four steps.
First, we would change the problem. The environment would emit not its states, but
only observations —signals that depend on its state but, like a robot’s sensors, provide
only partial information about it. For convenience, without loss of generality, we assume
that the reward is a direct, known function of the observation (perhaps the observation is
a vector, and the reward is one of is components). The environmental interaction would

then have no explicit states or rewards, but would simply be an alternating sequence of
actions At2Aand observations Ot2O:
A0,O1,A1,O2,A2,O3,A3,O4,...,
going on forever (cf. Equation 3.1) or forming episodes each ending with a special terminal
observation.

17.3. Observations and State 465
Second, we can recover the idea of state as used in this book from the sequence of
observations and actions. Let us use the word history , and the notation Ht, for an initial
portion of the trajectory up to an observation: Ht.=A0,O1,...,A t1,Ot. The history
represents the most that we can know about the past without looking outside of the data
stream (because the history is the whole past data stream). Of course, the history grows
with tand can become large and unwieldy. The idea of state is that of some compact
summary of the history that is as useful as the actual history for predicting the future.
Let us be clear about exactly what this means. To be a summary of the history, the state
must be a function of history, St=f(Ht), and to be as useful for predicting the future
as the whole history, it must have what is known as the Markov property . Formally, this
is a property of the function f. A function fhas the Markov property if and only if any

two histories handh0that are mapped by fto the same state ( f(h)=f(h0)) also have
the same probabilities for their next observation,
f(h)=f(h0))Pr{Ot+1=o|Ht=h, A t=a}=P r {Ot+1=o|Ht=h0,At=a},(17.5)
for all o2Oanda2A.I ffis Markov, then St=f(Ht) is a state as we have used the
term in this book. Let us henceforth call it a Markov state to distinguish it from states
that are summaries of the history but fall short of the Markov property (which we will
consider shortly).
A Markov state is a good basis for predicting the next observation (17.5) but, more

importantly, it is also a good basis for predicting or controlling anything . For example,
let a testbe any speciﬁc sequence of alternating actions and observations that might
occur in the future. For example, a three-step test is denoted ⌧=a1o1a2,o2,a3,o3.T h e
probability of this test given a speciﬁc history his deﬁned as
p(⌧|h).=P r {Ot+1=o1,Ot+2=o2,Ot+3=o3|Ht=h, A t=a1,At+1=a2,At+2=a3}.
(17.6)
Iffis Markov and handh0are any two histories that map to the same state under f,

(17.6)
then for any test ⌧of any length, its probabilities given the two histories must also be
the same:
f(h)=f(h0))p(⌧|h)=p(⌧|h0). (17.7)
In other words, a Markov state summarizes all the information in the history necessary
for determining any test’s probability. In fact, it summarizes all that is necessary for
making any prediction , including any GVF, and for behaving optimally (if fis Markov,
then there is always a deterministic function ⇡such that choosing At.=⇡(f(Ht)) is
optimal).

the same:
optimal).
The third step in extending reinforcement learning to partial observability is to deal
with certain computational considerations. In particular, we want the state to be a
compact summary of the history. For example, the identity function completely satisﬁes
the conditions for a Markov f, but would nevertheless be of little use because the
corresponding state St=Htwould grow with time and become unwieldy, as mentioned
earlier, but more fundamentally because it would never recur; the agent would never

466 Chapter 17: Frontiers
encounter the same state twice (in a continuing task) and thus could never beneﬁt from
a tabular learning method. We want our states to be compact as well as Markov. There
is a similar issue regarding how state is obtained and updated. We don’t really want a
function fthat takes whole histories. Instead, for computational reasons we prefer to
obtain the same e↵ect as fwith an incremental, recursive update that computes St+1
from St, incorporating the next increment of data, AtandOt+1:
St+1.=u(St,At,Ot+1),for all t0, (17.8)
with the ﬁrst state S0given. The function uis called the state-update function. For
example, if fwere the identity ( St=Ht), then uwould merely extend Stby appending At
andOt+1to it. Given f, it is always possible to construct a corresponding u,b u ti tm a y
not be computationally convenient and, as in the identity example, it may not produce
a compact state. The state-update function is a central part of any agent architecture

that handles partial observability. It must be eciently computable, as no actions or
predictions can be made until the state is available. An overall diagram of such an agent
architecture is given in Figure 17.1.
An example of obtaining Markov states through a state-update function is provided
by the popular Bayesian approach known as Partially Observable MDPs , orPOMDPs .
In this approach the environment is assumed to have a well deﬁned latent state Xtthat

underlies and produces the environment’s observations, but is never available to the agent
(and is not to be confused with the state Stused by the agent to make predictions and
decisions). The natural Markov state, St, for a POMDP is the distribution over the latent
states given the history, called the belief state . For concreteness, assume the usual case in
which there are a ﬁnite number of hidden states, Xt2{1,2,...,d }. Then the belief state
is the vector St.=st2Rdwith components
st[i].=P r {Xt=i|Ht},for all possible latent states i2{1,2,...,d }.

The belief state remains the same size (same number of components) however tgrows. It
can also be incrementally updated by Bayes’ rule, assuming one has complete knowledge
of the internal workings of the environment. Speciﬁcally, the ith component of the
belief-state update function is
u(s,a ,o)[i].=Pd
x=1s[x]p(i, o|x, a)
Pd
x=1Pd
x0=1s[x]p(x0,o|x, a), (17.9)
for all a2A,o2O, and belief states s2Rdwith components s[x], where the four-
argument pfunction here is not the usual one for MDPs (as in Chapter 3), but the

Pd
x=1Pd
analogous one for POMDPs, in terms of the latent state: p(x0,o|x, a).=P r {Xt=x0,Ot=o
|Xt1=x, A t1=a}. This approach is popular in theoretical work and has many signiﬁ-
cant applications, but its assumptions and computational complexity scale poorly and we
do not recommend it as an approach to artiﬁcial intelligence.
Another example of Markov states is provided by Predictive State Representations , or
PSRs . PSRs address the weakness of the POMDP approach that the semantics of its
agent state Stare grounded in the environment state, Xt, which is never observed and

17.3. Observations and State 467
Policy, Value fn.WorldPlannerModelAORA, RSuA
Figure 17.1: A conceptual agent architecture including a model, a planner, and a state-update
function. The world in this case receives actions Aand emits observations O. The observations
and a copy of the action are used by the state-update function uto produce the new state. The
new state is input to the policy and value function, producing the next action, and is also input
to the planner (and to u). The information ﬂows most responsible for learning are shown by
dashed lines that pass diagonally across the boxes that they change. The reward Rdirectly
changes the policy and value function. The action, reward, and state change the model, which
works closely with the planner to also change the policy and value function. Note that the
operation of the planner can be decoupled from the agent–environment interaction, whereas the
other processes should operate in lock step with this interaction to keep up with the arrival of

new data. Also note that the model and planner do not deal with observations directly, but only
with the states produced by u, which can act as targets for model learning.
thus is dicult to learn about. In PSRs and related approaches, the semantics of the agent
state is instead grounded in predictions about future observations and actions, which are
readily observable. In PSRs, a Markov state is deﬁned as a d-vector of the probabilities
ofdspecially chosen “core” tests as deﬁned above (17.6) . The vector is then updated by

a state-update function uthat is analogous to Bayes rule, but with a semantics grounded
in observable data, which arguably makes it easier to learn. This approach has been
extended in many ways, including end-tests, compositional tests, powerful “spectral”
methods, and closed-loop and temporally abstract tests learned by TD methods. Some of
the best theoretical developments are for systems known as Observable Operator Models
(OOMs) and Sequential Systems (Thon, 2017).
The fourth and ﬁnal step in our brief outline of how to handle partial observability in

reinforcement learning is to re-introduce approximation. As discussed in the introduction
to Part II, to approach artiﬁcial intelligence ambitiously one must embrace approximation.
This is just as true for states as it is for value functions. We must accept and work with
an approximate notion of state. The approximate state will play the same role in our
algorithms as before, so we continue to use the notation Stfor the state used by the
agent, even though it may not be Markov.

468 Chapter 17: Frontiers
Perhaps the simplest example of an approximate state is just the latest observation,
St.=Ot. Of course this approach cannot handle any hidden state information. It would
be better to use the last kobservations and actions, St.=Ot,At1,Ot1,...,A tk, for
some k1, which can be achieved by a state-update function that just shifts the new
data in and the oldest data out. This kth-order history approach is still very simple,
but can greatly increase the agent’s capabilities compared to trying to use the single
immediate observation directly as the state.
What happens when the Markov property (17.5) is only approximately satisﬁed?
Unfortunately, long-term prediction performance can degrade dramatically when the
one-step predictions deﬁning the Markov property become even slightly inaccurate.
Longer-term tests, GVFs, and state-update functions may all approximate poorly. The
short-term and long-term approximation objectives are just di↵erent, and there are no

useful theoretical guarantees at present.
Nevertheless, there are still reasons to think that the general idea outlined in this
section applies to the approximate case. The general idea is that a state that is good for
some predictions is also good for others (in particular, that a Markov state, sucient for
one-step predictions, is also sucient for all others). If we step back from that speciﬁc
result for the Markov case, the general idea is similar to what we discussed in Section 17.1

with multi-headed learning and auxiliary tasks. We discussed how representations that
were good for the auxiliary tasks were often also good for the main task. Taken together,
these suggest an approach to both partial observability and representation learning in
which multiple predictions are pursued and used to direct the construction of state
features. The guarantee provided by the perfect-but-impractical Markov property is
replaced by the heuristic that what’s good for some predictions may be good for others.

This approach scales well with computational resources. With a large machine one could
experiment with large numbers of predictions, perhaps favoring those that are most
similar to the ones of ultimate interest or that are easiest to learn reliably, or according
to some other criteria. It is important here to move beyond selecting the predictions
manually. The agent should do it. This would require a general language for predictions,
so that the agent can systematically explore a large space of possible predictions, sifting
through them for the ones that are most useful.

In particular, both POMDP and PSR approaches can be applied with approximate
states. The semantics of the state is useful in forming the state-update function, as it
is in these two approaches and in the k-order approach. There is not a strong need for
the semantics to be correct in order to retain useful information in the state. Some
approaches to state augmentation, such as Echo state networks (Jaeger, 2002), keep
almost arbitrary information about the history and can nevertheless perform well. There

are many possibilities and we expect more work and ideas in this area. Learning the
state-update function for an approximate state is a major part of the representation
learning problem as it arises in reinforcement learning.

17.4. Designing Reward Signals 469
17.4 Designing Reward Signals
A major advantage of reinforcement learning over supervised learning is that reinforcement
learning does not rely on detailed instructional information: generating a reward signal
does not depend on knowledge of what the agent’s correct actions should be. But the
success of a reinforcement learning application strongly depends on how well the reward
signal frames the goal of the application’s designer and how well the signal assesses
progress in reaching that goal. For these reasons, designing a reward signal is a critical
part of any application of reinforcement learning.
By designing a reward signal we mean designing the part of an agent’s environment
that is responsible for computing each scalar reward Rtand sending it to the agent at
each time t. In our discussion of terminology at the end of Chapter 14, we said that Rt
is more like a signal generated inside an animal’s brain than it is like an object or event

in the animal’s external environment. The parts of our brains that generate these signals
for us evolved over millions of years to be well suited to the challenges our ancestors
had to face in their struggles to propagate their genes to future generations. We should
therefore not think that designing a good reward signal is always an easy thing to do!
One challenge is to design a reward signal so that as an agent learns, its behavior
approaches, and ideally eventually achieves, what the application’s designer actually

desires. This can be easy if the designer’s goal is simple and easy to identify, such as
ﬁnding the solution to a well-deﬁned problem or earning a high score in a well-deﬁned
game. In cases like these, it is usual to reward the agent according to its success in solving
the problem or its success in improving its score. But some problems involve goals that
are dicult to translate into reward signals. This is especially true when the problem
requires the agent to skillfully perform a complex task or set of tasks, such as would be

required of a useful household robotic assistant. Further, reinforcement learning agents
can discover unexpected ways to make their environments deliver reward, some of which
might be undesirable, or even dangerous. This is a longstanding and critical challenge for
any method, like reinforcement learning, that is based on optimization. We discuss this
issue more in Section 17.6, the ﬁnal section of this book.
Even when there is a simple and easily identiﬁable goal, the problem of sparse reward

often arises. Delivering non-zero reward frequently enough to allow the agent to achieve
the goal once, let alone to learn to achieve it eciently from multiple initial conditions,
can be a daunting challenge. State–action pairs that clearly deserve to trigger reward may
be few and far between, and rewards that mark progress toward a goal can be infrequent
because progress is dicult or even impossible to detect. The agent may wander aimlessly
for long periods of time (what Minsky, 1961, called the “plateau problem”).

In practice, designing a reward signal is often left to an informal trial-and-error search
for a signal that produces acceptable results. If the agent fails to learn, learns too slowly,
or learns the wrong thing, then the designer tweaks the reward signal and tries again.
To do this, the designer judges the agent’s performance by criteria that he or she is
attempting to translate into a reward signal so that the agent’s goal matches his or her
own. And if learning is too slow, the designer may try to design a non-sparse reward signal

that e↵ectively guides learning throughout the agent’s interaction with its environment.

470 Chapter 17: Frontiers
It is tempting to address the sparse reward problem by rewarding the agent for
achieving subgoals that the designer thinks are important way stations to the overall
goal. But augmenting the reward signal with well-intentioned supplemental rewards may
lead the agent to behave very di↵erently from what is intended; the agent may end up
not achieving the overall goal at all. A better way to provide such guidance is to leave
the reward signal alone and instead augment the value-function approximation with an
initial guess of what it should ultimately be, or augment it with initial guesses as to what
certain parts of it should be. For example, suppose one wants to o↵er v0:S!Ras an
initial guess at the true optimal value function v⇤, and that one is using linear function
approximation with features x:S!Rd. Then one would deﬁne the initial value function
approximation to be
ˆv(s,w).=w>x(s)+v0(s), (17.10)

and update the weights was usual. If the initial weight vector is 0, then the initial value
function will be v0, but the asymptotic solution quality will be determined by the feature
vectors as usual. This initialization can be done for arbitrary nonlinear approximators
and arbitrary forms of v0, though it is not guaranteed to always accelerate learning.
A particularly e↵ective approach to the sparse reward problem is the shaping tech-
nique introduced by the psychologist B. F. Skinner and described in Section 14.3. The

e↵ectiveness of this technique relies on the fact that sparse reward problems are not
just problems with the reward signal; they are also problems with an agent’s policy in
preventing the agent from frequently encountering rewarding states. Shaping involves
changing the reward signal as learning proceeds, starting from a reward signal that is not
sparse given the agent’s initial behavior, and gradually modifying it toward the reward
signal suited to problem of original interest. Each modiﬁcation is made so that the

agent is frequently rewarded given its current behavior. The agent faces a sequence of
increasingly-dicult reinforcement learning problems, where what is learned at each stage
makes the next-harder problem relatively easy because the agent now encounters reward
more frequently than it would if it did not have prior experience with easier problems.
This kind of shaping is an essential technique in training animals, and it is e↵ective in
computational reinforcement learning as well.
What if one has no idea what the rewards should be but there is another agent, perhaps

a person, who is already expert at the task and whose behavior can be observed? In
this case one can use methods known variously as “imitation learning,” “learning from
demonstration,” and “apprenticeship learning.” The idea here is to beneﬁt from the
expert agent but leave open the possibility of eventually performing better. Learning from
an expert’s behavior can be done either by learning directly by supervised learning or by
extracting a reward signal using what is known as “inverse reinforcement learning” and

then using a reinforcement learning algorithm with that reward signal to learn a policy.
The task of inverse reinforcement learning as explored by Ng and Russell (2000) is to try to
recover the expert’s reward signal from the expert’s behavior alone. This cannot be done
exactly because a policy can be optimal with respect to many di↵erent reward signals (for
example, any reward signal that gives the same reward for all states and actions), but it is
possible to ﬁnd plausible reward signal candidates. Unfortunately, strong assumptions are

required, including knowledge of the environment’s dynamics and of the feature vectors

17.4. Designing Reward Signals 471
in which the reward signal is linear. The method also requires completely solving the
problem (e.g., by dynamic programming methods) multiple times. These diculties
notwithstanding, Abbeel and Ng (2004) argue that the inverse reinforcement learning
approach can sometimes be more e↵ective than supervised learning for beneﬁting from
the behavior of an expert.
Another approach to ﬁnding a good reward signal is to automate the trial-and-error
search for a good signal that we mentioned above. From an application perspective, the
reward signal is a parameter of the learning algorithm. As is true for other algorithm
parameters, the search for a good reward signal can be automated by deﬁning a space of
feasible candidates and applying an optimization algorithm. The optimization algorithm
evaluates each candidate reward signal by running the reinforcement learning system with
that signal for some number of steps, and then scoring the overall result by a “high-level”

objective function intended to encode the designer’s true goal, ignoring the limitations
of the agent. Reward signals can even be improved via online gradient ascent, where
the gradient is that of the high-level objective function (Sorg, Lewis, and Singh, 2010).
Relating this approach to the natural world, the algorithm for optimizing the high-level
objective function is analogous to evolution, where the high-level objective function is an
animal’s evolutionary ﬁtness determined by the number of its o↵spring that survive to
reproductive age.

Computational experiments with this bilevel optimization approach—one level analo-
gous to evolution, and the other due to reinforcement learning by individual agents—have
conﬁrmed that intuition alone is not always adequate to devise a good reward signal
(Singh, Lewis, and Barto, 2009). The performance of a reinforcement learning agent as
evaluated by the high-level objective function can be very sensitive to details of the agent’s
reward signal in subtle ways determined by the agent’s limitations and the environments

in which it acts and learns. These experiments also demonstrated that an agent’s goal
should not always be the same as the goal of the agent’s designer.
At ﬁrst this seems counterintuitive, but it may be impossible for the agent to achieve the
designer’s goal no matter what its reward signal is. The agent has to learn under various
kinds of constraints, such as limited computational power, limited access to information
about its environment, or limited time to learn. When there are constraints like these,

learning to achieve a goal that is di↵erent from the designer’s goal can sometimes end up
getting closer to the designer’s goal than if that goal were pursued directly (Sorg, Singh,
and Lewis, 2010; Sorg, 2011). Examples of this in the natural world are easy to ﬁnd.
Because we cannot directly assess the nutritional value of most foods, evolution—the
designer of our reward signal—gave us a reward signal that makes us seek certain tastes.

Though certainly not infallible (indeed, possibly detrimental in environments that di↵er in
certain ways from ancestral environments), this compensates for many of our limitations:
our limited sensory abilities, the limited time over which we can learn, and the risks
involved in ﬁnding a healthy diet through personal experimentation. Similarly, because
an animal cannot observe its own evolutionary ﬁtness, that objective function does not
work as a reward signal for learning. Evolution instead provides reward signals that are

sensitive to observable predictors of evolutionary ﬁtness.
Finally, remember that a reinforcement learning agent is not necessarily like a complete

472 Chapter 17: Frontiers
organism or robot; it can be a component of a larger behaving system. This means
that reward signals may be inﬂuenced by things inside the larger behaving agent, such
as motivational states, memories, ideas, or even hallucinations. Reward signals may
also depend on properties of the learning process itself, such as measures of how much
progress learning is making. Making reward signals sensitive to information about internal
factors such as these makes it possible for an agent to learn how to control the “cognitive
architecture” of which it is a part, as well as to acquire knowledge and skills that would be
dicult to learn from a reward signal that depended only on external events. Possibilities
like these led to the idea of “intrinsically-motivated reinforcement learning” that we
brieﬂy discuss further at the end of the following section.
17.5 Remaining Issues
In this book we have presented the foundations of a reinforcement learning approach to

artiﬁcial intelligence. Roughly speaking, that approach is based on model-free and model-
based methods working together, as in the Dyna architecture of Chapter 8, combined
with function approximation as developed in Part II. The focus has been on online and
incremental algorithms, which we see as fundamental even to model-based methods, and
on how these can be applied in o↵-policy training situations. The full rationale for the
latter has been presented only in this last chapter. That is, we have all along presented o↵-

policy learning as an appealing way to deal with the explore/exploit dilemma, but only in
this chapter have we discussed learning about many diverse auxiliary tasks simultaneously
with GVFs and learning about the world hierarchically in terms of temporally-abstract
option models, both of which involve o↵-policy learning. Much remains to be worked
out, as we have indicated throughout the book and as evidenced by the directions for
additional research discussed in this chapter. But suppose we are generous and grant the

broad outlines of everything that we have done in the book andeverything that has been
outlined so far in this chapter. What would remain after that? Of course we can’t know
for sure what will be required, but we can make some guesses. In this section we highlight
six further issues which it seems to us will still need to be addressed by future research.
First, we still need powerful parametric function approximation methods that work well
in fully incremental and online settings. Methods based on deep learning and ANNs are

a major step in this direction but, still, only work well with batch training on large data
sets, with training from extensive o✏ine self play, or with learning from the interleaved
experience of multiple agents on the same task. These and other settings are ways of
working around a basic limitation of today’s deep learning methods, which struggle to
learn rapidly in the incremental, online settings that are most natural for the reinforcement
learning algorithms emphasized in this book. The problem is sometimes described as

one of “catastrophic interference” or “correlated data.” When something new is learned
it tends to replace what has previously been learned rather than adding to it, with the
result that the beneﬁt of the older learning is lost. Techniques such as “replay bu↵ers”
are often used to retain and replay old data so that its beneﬁts are not permanently lost.
An honest assessment has to be that current deep learning methods are not well suited to
online learning. We see no reason that this limitation is insurmountable, but algorithms

17.5. Remaining Issues 473
that address it, while at the same time retaining the advantages of deep learning, have
not yet been devised. Most current deep learning research is directed toward working
around this limitation rather than removing it.
Second (and perhaps closely related), we still need methods for learning features such
that subsequent learning generalizes well. This issue is an instance of a general problem
variously called “representation learning,” “constructive induction,” and “meta-learning”—
how can we use experience not just to learn a given desired function, but to learn inductive
biases such that future learning generalizes better and is thus faster? This is an old
problem, dating back to the origins of artiﬁcial intelligence and pattern recognition in
the 1950s and 1960s.1Such age should give one pause. Perhaps there is no solution. But
it is equally likely that the time for ﬁnding a solution and demonstrating its e↵ectiveness

has not yet arrived. Today machine learning is conducted at a far larger scale than it has
been in the past, and the potential beneﬁts of a good representation learning method have
become much more apparent. We note that a new annual conference—the International
Conference on Learning Representations—has been exploring this and related topics
every year since 2013. It is also less common to explore representation learning within
a reinforcement learning context. Reinforcement learning brings some new possibilities

to this old issue, such as the auxiliary tasks discussed in Section 17.1. In reinforcement
learning, the problem of representation learning can be identiﬁed with the problem of
learning the state-update function discussed in Section 17.3.
Third, we still need scalable methods for planning with learned environment models.
Planning methods have proven extremely e↵ective in applications such as AlphaGo Zero
and computer chess in which the model of the environment is known from the rules of
the game or can otherwise be supplied by human designers. But cases of full model-based

reinforcement learning, in which the environment model is learned from data and then
used for planning, are rare. The Dyna system described in Chapter 8 is one example, but
as described there and in most subsequent work it uses a tabular model without function
approximation, which greatly limits its applicability. Only a few studies have included
learned linear models, and even fewer have also explored including temporally-abstract
models using options as discussed in Section 17.2.

More work is needed before planning with learned models can be e↵ective. For example,
the learning of the model needs to be selective because the scope of a model strongly
a↵ects planning eciency. If a model focuses on the key consequences of the most
important options, then planning can be ecient and rapid, but if a model includes
details of unimportant consequences of options that are unlikely to be selected, then
planning may be almost useless. Environment models should be constructed judiciously

with regard to both their states and dynamics with the goal of optimizing the planning
process. The various parts of the model should be continually monitored as to the degree
to which they contribute to, or detract from, planning eciency. The ﬁeld has not
yet addressed this complex of issues or designed model-learning methods that take into
account their implications.
1Some would claim that deep learning solves this problem, for example, that DQN as described in

Section 16.5 illustrates a solution, but we are unconvinced. There is as yet little evidence that deep
learning alone solves the representation learning problem in a general and ecient way.

474 Chapter 17: Frontiers
A fourth issue that needs to be addressed in future research is that of automating the
choice of tasks on which an agent works and uses to structure its developing competence.
It is usual in machine learning for human designers to set the tasks that the learning
agent is expected to master. Because these tasks are known in advance and remain ﬁxed,
they can be built into the learning algorithm code. However, looking ahead, we will want
the agent to make its own choices about what tasks it should try to master. These might
be subtasks of a speciﬁc overall task that is already known, or they might be intended to
create building blocks that permit more ecient learning of many di↵erent tasks that the
agent is likely to face in the future but which are currently unknown.
These tasks may be like the auxiliary tasks or the GVFs discussed in Section 17.1, or
tasks solved by options as discussed in Section 17.2. In forming a GVF, for example, what

should the cumulant, the policy, and the termination function be? The current state of
the art is to select these manually, but far greater power and generality would come from
making these task choices automatically, particularly when they derive from what the
agent has previously constructed as a result of representation learning or experience with
previous subproblems. If GVF design is automated, then the design choices themselves
will have to be explicitly represented. Rather than the task choices being in the mind

of the designer and built into the code, they will have to be in the machine itself in
such a way that they can be set and changed, monitored, ﬁltered, and searched among
automatically. Tasks could then be built hierarchically upon others much like features are
in an ANN. The tasks are the questions, and the contents of the ANN are the answers to
those questions. We expect there will need to be a full hierarchy of questions to match
the hierarchy of answers provided by modern deep learning methods.
The ﬁfth issue that we would like to highlight for future research is that of the

interaction between behavior and learning via some computational analog of curiosity .
In this chapter we have been imagining a setting in which many tasks are being learned
simultaneously, using o↵-policy methods, from the same stream of experience. The actions
taken will of course inﬂuence this stream of experience, which in turn will determine how
much learning occurs and which tasks are learned. When reward is not available, or not
strongly inﬂuenced by behavior, the agent is free to choose actions that maximize in

some sense the learning on the tasks, that is, to use some measure of learning progress
as an internal or “intrinsic” reward, implementing a computational form of curiosity. In
addition to measuring learning progress, intrinsic reward can, among other possibilities,
signal the receipt of unexpected, novel, or otherwise interesting input, or can assess the
agent’s ability to cause changes in its environment. Intrinsic reward signals generated in
these ways can be used by an agent to pose tasks for itself by deﬁning auxiliary tasks,

GVFs, or options, as discussed above, so that skills learned in this way can contribute
to the agent’s ability to master future tasks. The result is a computational analog of
something like play. Many preliminary studies of such uses of intrinsic reward signals
have been conducted, and exciting topics for future research remain in this general area.
A ﬁnal issue that demands attention in future research is that of developing methods to
make it acceptably safe to embed reinforcement learning agents into physical environments.

This is one of the most pressing areas for future research, and we discuss it further in the
following section.

17.6. The Future of Artiﬁcial Intelligence 475
17.6 The Future of Artiﬁcial Intelligence
When we were writing the ﬁrst edition of this book in the mid-1990s, artiﬁcial intelligence
was making signiﬁcant progress and was having an impact on society, though it was
mostly still the promise of artiﬁcial intelligence that was inspiring developments. Machine
learning was part of that outlook, but it had not yet become indispensable to artiﬁcial
intelligence. By today that promise has transitioned to applications that are changing the
lives of millions of people, and machine learning has come into its own as a key technology.
As we write this second edition, some of the most remarkable developments in artiﬁcial
intelligence have involved reinforcement learning, most notably “deep reinforcement
learning”—reinforcement learning with function approximation by deep artiﬁcial neural
networks. We are at the beginning of a wave of real-world applications of artiﬁcial

intelligence, many of which will include reinforcement learning, deep and otherwise, that
will impact our lives in ways that are hard to predict.
But an abundance of successful real-world applications does not mean that true artiﬁcial
intelligence has arrived. Despite great progress in many areas, the gulf between artiﬁcial
intelligence and the intelligence of humans, and even of other animals, remains great.
Superhuman performance can be achieved in some domains, even formidable domains
like Go, but it remains a signiﬁcant challenge to develop systems that are like us in

being complete, interactive agents having general adaptability and problem-solving skills,
emotional sophistication, creativity, and the ability to learn quickly from experience.
With its focus on learning by interacting with dynamic environments, reinforcement
learning, as it develops over the future, will be a critical component of agents with these
abilities.
Reinforcement learning’s connections to psychology and neuroscience (Chapters 14
and 15) underscore its relevance to another longstanding goal of artiﬁcial intelligence:

shedding light on fundamental questions about the mind and how it emerges from the
brain. Reinforcement learning theory is already contributing to our understanding of
the brain’s reward, motivation, and decision-making processes, and there is good reason
to believe that through its links to computational psychiatry, reinforcement learning
theory will contribute to methods for treating mental disorders, including drug abuse
and addiction.
Another contribution that reinforcement learning can make over the future is as an

aid to human decision making. Policies derived by reinforcement learning in simulated
environments can advise human decision makers in such areas as education, healthcare,
transportation, energy, and public-sector resource allocation. Particularly relevant is the
key feature of reinforcement learning that it takes long-term consequences of decisions
into account. This is very clear in games like backgammon and Go, where some of
the most impressive results of reinforcement learning have been demonstrated, but it

is also a property of many high-stakes decisions that a↵ect our lives and our planet.
Reinforcement learning follows related methods for advising human decision making that
have been developed in the past by decision analysts in many disciplines. With advanced
function approximation methods and massive computational power, reinforcement learning
methods have the potential to overcome some of the diculties of scaling up traditional
decision-support methods to larger and more complex problems.

476 Chapter 17: Frontiers
The rapid pace of advances in artiﬁcial intelligence has led to warnings that artiﬁcial
intelligence poses serious threats to our societies, even to humanity itself. The renowned
scientist and artiﬁcial intelligence pioneer Herbert Simon anticipated the warnings we are
hearing today in a presentation at the Earthware Symposium at CMU in 2000 (Simon,
2000). He spoke of the eternal conﬂict between the promise and perils of any new
knowledge, reminding us of the Greek myths of Prometheus, the hero of modern science,
who stole ﬁre from the gods for the beneﬁt of mankind, and Pandora, whose box could
be opened by a small and innocent action to release untold perils on the world. While
accepting that this conﬂict is inevitable, Simon urged us to recognize that as designers of
our future and not mere spectators, the decisions wemake can tilt the scale in Prometheus’
favor. This is certainly true for reinforcement learning, which can beneﬁt society but

can also produce undesirable outcomes if it is carelessly deployed. Thus, the safety of
artiﬁcial intelligence applications involving reinforcement learning is a topic that deserves
careful attention.
A reinforcement learning agent can learn by interacting with either the real world or
with a simulation of some piece of the real world, or by a mixture of these two sources of
experience. Simulators provide safe environments in which an agent can explore and learn
without risking real damage to itself or to its environment. In most current applications,

policies are learned from simulated experience instead of direct interaction with the
real world. In addition to avoiding undesirable real-world consequences, learning from
simulated experience can make virtually unlimited data available for learning, generally
at less cost than needed to obtain real experience, and because simulations typically run
much faster than real time, learning can often occur more quickly than if it relied on real
Nevertheless, the full potential of reinforcement learning requires reinforcement learning

agents to be embedded into the ﬂow of real-world experience, where they act, explore,
and learn in ourworld, and not just in their worlds. After all, reinforcement learning
algorithms—at least those upon which we focus in this book—are designed to learn online,
and they emulate many aspects of how animals are able to survive in nonstationary and
hostile environments. Embedding reinforcement learning agents in the real world can be
transformative in realizing the promises of artiﬁcial intelligence to amplify and extend
human abilities.

A major reason for wanting a reinforcement learning agent to act and learn in the real
world is that it is often dicult, sometimes impossible, to simulate real-world experience
with enough ﬁdelity to make the resulting policies, whether derived by reinforcement
learning or by other methods, work well—and safely—when directing real actions. This
is especially true for environments whose dynamics depend on the behavior of humans,
such as in education, healthcare, transportation, and public policy, domains that can

surely beneﬁt from improved decision making. However, it is for real-world embedded
agents that warnings about potential dangers of artiﬁcial intelligence need to be heeded.
Some of these warnings are particularly relevant to reinforcement learning. Because
reinforcement learning is based on optimization, it inherits the pluses and minuses of all
optimization methods. On the minus side is the problem of devising objective functions,
or reward signals in the case of reinforcement learning, so that optimization produces

17.6. The Future of Artiﬁcial Intelligence 477
the desired results while avoiding undesirable results. We said in Section 17.4 that
reinforcement learning agents can discover unexpected ways to make their environments
deliver reward, some of which might be undesirable, or even dangerous. When we specify
what we want a system to learn only indirectly, as we do in designing a reinforcement
learning system’s reward signal, we will not know how closely the agent will fulﬁll our desire
until its learning is complete. This is hardly a new problem with reinforcement learning;
recognition of it has a long history in both literature and engineering. For example, in
Goethe’s poem “The Sorcerer’s Apprentice” (Goethe, 1878), the apprentice uses magic to
enchant a broom to do his job of fetching water, but the result is an unintended ﬂood due
to the apprentice’s inadequate knowledge of magic. In the engineering context, Norbert
Wiener, the founder of cybernetics, warned of this problem more than half a century ago

by relating the supernatural story of “The Monkey’s Paw” (Wiener, 1964): “... it grants
what you ask for, not what you should have asked for or what you intend” (p. 59). The
problem has also been discussed at length in a modern context by Nick Bostrom (2014).
Anyone having experience with reinforcement learning has likely seen their systems
discover unexpected ways to obtain a lot of reward. Sometimes the unexpected behavior
is good: it solves a problem in a nice new way. In other instances, what the agent

learns violates considerations that the system designer may never have thought about.
Careful design of reward signals is essential if an agent is to act in the real world with no
opportunity for human vetting of its actions or means to easily interrupt its behavior.
Despite the possibility of unintended negative consequences, optimization has been used
for hundreds of years by engineers, architects, and others whose designs have positively

impacted the world. We owe much that is good in our environment to the application
of optimization methods. Many approaches have been developed to mitigate the risk
of optimization, such as adding hard and soft constraints, restricting optimization to
robust and risk-sensitive policies, and optimizing with multiple objective functions. Some
of these approaches have been adapted to reinforcement learning, and more research is
needed to address these concerns. The problem of ensuring that a reinforcement learning
agent’s goal is attuned to our own remains a challenge.

Another challenge if reinforcement learning agents are to act and learn in the real world
is not just about what they might learn eventually , but about how they will behave while
they are learning. How do you make sure that an agent gets enough experience to learn a
high-performing policy, all the while not harming its environment, other agents, or itself
(or more realistically, while keeping the probability of harm acceptably low)? This problem

is also not novel or unique to reinforcement learning. Risk management and mitigation
for embedded reinforcement learning is similar to what control engineers have had to
confront from the beginning of using automatic control in situations where a controller’s
behavior can have unacceptable, possibly catastrophic, consequences, as in the control of
an aircraft or a delicate chemical process. Control applications rely on careful system

modeling, model validation, and extensive testing, and there is a highly-developed body
of theory aimed at ensuring convergence and stability of adaptive controllers designed for
use when the dynamics of the system to be controlled are not fully known. Theoretical
guarantees are never iron-clad because they depend on the validity of the assumptions
underlying the mathematics, but without this theory, combined with risk-management
and mitigation practices, automatic control—adaptive and otherwise—would not be as

478 Chapter 17: Frontiers
beneﬁcial as it is today in improving the quality, eciency, and cost-e↵ectiveness of
processes on which we have come to rely. One of the most pressing areas for future
reinforcement learning research is to adapt and extend methods developed in control
engineering with the goal of making it acceptably safe to fully embed reinforcement
learning agents into physical environments.
In closing, we return to Simon’s call for us to recognize that we are designers of our
future and not simply spectators. By decisions we make as individuals, and by the
inﬂuence we can exert on how our societies are governed, we can work toward ensuring
that the beneﬁts made possible by a new technology outweigh the harm it can cause.
There is ample opportunity to do this in the case of reinforcement learning, which can
help improve the quality, fairness, and sustainability of life on our planet, but which
can also release new perils. A threat already here is the displacement of jobs caused

by applications of artiﬁcial intelligence. Still there are good reasons to believe that the
beneﬁts of artiﬁcial intelligence can outweigh the disruption it causes. As to safety,
hazards possible with reinforcement learning are not completely di↵erent from those
that have been managed successfully for related applications of optimization and control
methods. As reinforcement learning moves out into the real world in future applications,

developers have an obligation to follow best practices that have evolved for similar
technologies, while at the same time extending them to make sure that Prometheus keeps
the upper hand.

Bibliographical and Historical Remarks 479
17.1 General value functions were ﬁrst explicitly identiﬁed by Sutton and colleagues
(Sutton, 1995a; Sutton et al., 2011; Modayil, White and Sutton, 2013). Ring (in
preparation) developed an extensive thought experiment with GVFs (“forecasts”)
that has been inﬂuential despite not yet having been published.
The ﬁrst demonstrations of multi-headed learning in reinforcement learning were
by Jaderberg et al. (2017). Bellemare, Dabney and Munos (2017) showed that
predicting more things about the distribution of reward could signiﬁcantly speed
learning to optimize its expectation, an instance of auxiliary tasks. Many others
have since taken up this line of research.
The general theory of classical conditioning as learned predictions together with
built-in, reﬂexive reactions to the predictions has not to our knowledge been
clearly articulated in the psychological literature. Modayil and Sutton (2014)

describe it as an approach to the engineering of robots and other agents, calling
it “Pavlovian control” to allude to its roots in classical conditioning.
17.2 The formalization of temporally abstract courses of action as options was intro-
duced by Sutton, Precup, and Singh (1999), building on prior work by Parr (1998)
and Sutton (1995a), and on classical work on Semi-MDPs (e.g., see Puterman,
1994). Precup’s (2000) PhD thesis developed option ideas fully. An important
limitation of these early works is that they did not treat the o↵-policy case

with function approximation. Intra-option learning in general requires o↵-policy
learning, which could not be done reliably with function approximation at that
time. Although now we have a variety of stable o↵-policy learning methods using
function approximation, their combination with option ideas had not been signif-
icantly explored at the time of publication of this book. Barto and Mahadevan
(2003) and Hengst (2012) review the options formalism and other approaches to
temporal abstraction.

Using GVFs to implement option models has not previously been described. Our
presentation uses the trick introduced by Modayil, White and Sutton (2014) for
predicting signals at the termination of policies.
Among the few works that have learned option models with function approxi-
mation are those by Sorg and Singh (2010), and by Bacon, Harb, and Precup
(2017).
The extension of options and option models to the average-reward setting has
not yet been developed in the literature.

(2017).
17.3 A good presentation of the POMDP approach is given by Monahan (1982). PSRs
and tests were introduced by Littman, Sutton and Singh (2002). OOMs were
introduced by Jaeger (1997, 1998, 2000). Sequential Systems, which unify PSRs,
OOMs, and many other works, were introduced in the PhD thesis of Michael Thon
(2017; Thon and Jaeger, 2015). Extensions to networks of temporal relationships

480 Chapter 17: Frontiers
were developed by Tanner (2006; Sutton and Tanner, 2005) and then extended
to options (Sutton, Rafols, and Koop, 2006).
The theory of reinforcement learning with a non-Markov state representation was
developed explicitly by Singh, Jaakkola, and Jordan (1994; Jaakkola, Singh, and
Jordan, 1995). Early reinforcement learning approaches to partial observability
were developed by Chrisman (1992), McCallum (1993, 1995), Parr and Russell
(1995), Littman, Cassandra, and Kaelbling (1995), and by Lin and Mitchell
(1992).
17.4 Early e↵orts to include advice and teaching in reinforcement learning include
those by Lin (1992), Maclin and Shavlik (1994), Clouse (1996), and Clouse and
Utgo↵ (1992).
Skinner’s shaping should not be confused with the “potential-based shaping”
technique introduced by Ng, Harada, and Russell (1999). Their technique has
been shown by Wiewiora (2003) to be equivalent to the simpler idea of providing
an initial approximation to the value function, as in (17.10).

(1992).
17.5 We recommend the book by Goodfellow, Bengio, and Courville (2016) for discus-
sion of today’s deep learning techniques. The problem of catastrophic interference
in ANNs was developed by McCloskey and Cohen (1989), Ratcli↵ (1990), and
French (1999). The idea of a replay bu↵er was introduced by Lin (1992) and used
prominently in deep learning in the Atari game playing system (Section 16.5,
Mnih et al., 2013, 2015).
Minsky (1961) was one of the ﬁrst to identify the problem of representation
learning.

learning.
Among the few works to consider planning with learned, approximate models
are those by Kuvayev and Sutton (1996), Sutton, Szepesvari, Geramifard, and
Bowling (2008), Nouri and Littman (2009), and Hester and Stone (2012).
The need to be selective in model construction to avoid slowing planning is well
known in artiﬁcial intelligence. Some of the classic work is by Minton (1990) and
Tambe, Newell, and Rosenbloom (1990). Hauskrecht, Meuleau, Kaelbling, Dean,

learning.
and Boutilier (1998) showed this e↵ect in MDPs with deterministic options.
Schmidhuber (1991a, b) proposed how something like curiosity would result if
reward signals were a function of how quickly an agent’s environment model
is improving. The empowerment function proposed by Klyubin, Polani, and
Nehaniv (2005) is an information-theoretic measure of an agent’s ability to control
its environment that can function as an intrinsic reward signal. Baldassarre and
Mirolli (2013) is a collection of contributions by researchers studying intrinsic

reward and motivation from both biological and computational perspectives,
including a perspective on “intrinsically-motivated reinforcement learning,” to
use the term introduced by Singh, Barto, and Chentenez (2004). See also Oudeyer
and Kaplan (2007), Oudeyer, Kaplan, and Hafner (2007), and Barto (2013).

Abbeel, P., Ng, A. Y. (2004). Apprenticeship learning via inverse reinforcement learning. In
Proceedings of the 21st International Conference on Machine Learning . ACM, New York.
Abramson, B. (1990). Expected-outcome: A general model of static evaluation. IEEE Transac-
tions on Pattern Analysis and Machine Intelligence, 12 (2):182–193.
Adams, C. D. (1982). Variations in the sensitivity of instrumental responding to reinforcer
devaluation. The Quarterly Journal of Experimental Psychology, 34 (2):77–98.
Adams, C. D., Dickinson, A. (1981). Instrumental responding following reinforcer devaluation.
The Quarterly Journal of Experimental Psychology, 33 (2):109–121.
Adams, R. A., Huys, Q. J. M., Roiser, J. P. (2015). Computational Psychiatry: towards a
mathematically informed understanding of mental illness. Journal of Neurology, Neurosurgery
&P s y c h i a t r y . doi:10.1136/jnnp-2015-310737
Agrawal, R. (1995). Sample mean based index policies with O(logn) regret for the multi-armed

bandit problem. Advances in Applied Probability, 27 (4):1054–1078.
Agre, P. E. (1988). The Dynamic Structure of Everyday Life . Ph.D. thesis, Massachusetts
Institute of Technology, Cambridge MA. AI-TR 1085, MIT Artiﬁcial Intelligence Laboratory.
Agre, P. E., Chapman, D. (1990). What are plans for? Robotics and Autonomous Systems,
6(1-2):17–34.
Aizerman, M. A., Braverman, E. ´I., Rozonoer, L. I. (1964). Probability problem of pattern
recognition learning and potential functions method. Avtomat. i Telemekh, 25 (9):1307–1323.

Albus, J. S. (1971). A theory of cerebellar function. Mathematical Biosciences, 10 (1-2):25–61.
Albus, J. S. (1981). Brain, Behavior, and Robotics . Byte Books, Peterborough, NH.
Aleksandrov, V. M., Sysoev, V. I., Shemeneva, V. V. (1968). Stochastic optimization of systems.
Izv. Akad. Nauk SSSR, Tekh. Kibernetika :14–19.
Amari, S. I. (1998). Natural gradient works eciently in learning. Neural Computation,
10(2):251–276.
An, P. C. E. (1991). An Improved Multi-dimensional CMAC Neural network: Receptive Field

Function and Placement . Ph.D. thesis, University of New Hampshire, Durham.
An, P. C. E., Miller, W. T., Parks, P. C. (1991). Design improvements in associative memories for
cerebellar model articulation controllers (CMAC). Artiﬁcial Neural Networks ,p p .1 2 0 7 – 1 2 1 0 ,
Elsevier North-Holland. http://www.incompleteideas.net/papers/AnMillerParks1991.pdf
Anderson, C. W. (1986). Learning and Problem Solving with Multilayer Connectionist Systems .
Ph.D. thesis, University of Massachusetts, Amherst.
Anderson, C. W. (1987). Strategy learning with multilayer connectionist representations. In

Proceedings of the 4th International Workshop on Machine Learning , pp. 103–114. Morgan
Kaufmann.
481

482 References
Anderson, C. W. (1989). Learning to control an inverted pendulum using neural networks. IEEE
Control Systems Magazine, 9 (3):31–37.
Anderson, J. A., Silverstein, J. W., Ritz, S. A., Jones, R. S. (1977). Distinctive features,
categorical perception, and probability learning: Some applications of a neural model.
Psychological Review, 84 (5):413–451.
Andreae, J. H. (1963). STELLA, A scheme for a learning machine. In Proceedings of the 2nd
IFAC Congress, Basle , pp. 497–502. Butterworths, London.
Andreae, J. H. (1969a). A learning machine with monologue. International Journal of
Man–Machine Studies, 1 (1):1–20.
Andreae, J. H. (1969b). Learning machines—a uniﬁed view. In A. R. Meetham and R. A. Hudson
(Eds.), Encyclopedia of Information, Linguistics, and Control , pp. 261–270. Pergamon,
Oxford.
Andreae, J. H. (1977). Thinking with the Teachable Machine . Academic Press, London.
Andreae, J. H. (2017a). A model of how the brain learns: A short introduction to multiple

Oxford.
context associative learning (MCAL) and the PP system. Unpublished report.
Andreae, J. H. (2017b). Working memory for the associative learning of language. Unpublished
report.
Arthur, W. B. (1991). Designing economic agents that act like human agents: A behavioral
approach to bounded rationality. The American Economic Review, 81 (2):353–359.
Asadi, K., Allen, C., Roderick, M., Mohamed, A. R., Konidaris, G., Littman, M. (2017). Mean
actor critic. ArXiv:1709.00503.

report.
Atkeson, C. G. (1992). Memory-based approaches to approximating continuous functions. In
Sante Fe Institute Studies in the Sciences of Complexity , Proceedings Vol. 12, pp. 521–521.
Atkeson, C. G., Moore, A. W., Schaal, S. (1997). Locally weighted learning. Artiﬁcial Intelligence
Review, 11 :11–73.
Auer, P., Cesa-Bianchi, N., Fischer, P. (2002). Finite-time analysis of the multiarmed bandit
problem. Machine learning, 47 (2-3):235–256.
Bacon, P. L., Harb, J., Precup, D. (2017). The Option-Critic Architecture. In Proceedings of the

Association for the Advancement of Artiﬁcial Intelligence ,p p .1 7 2 6 – 1 7 3 4 .
Baird, L. C. (1995). Residual algorithms: Reinforcement learning with function approximation.
InProceedings of the 12th International Conference on Machine Learning (ICML 1995) ,
pp. 30–37. Morgan Kaufmann.
Baird, L. C. (1999). Reinforcement Learning through Gradient Descent . Ph.D. thesis, Carnegie
Mellon University, Pittsburgh PA.
Baird, L. C., Klopf, A. H. (1993). Reinforcement learning with high-dimensional, continuous

actions. Wright Laboratory, Wright-Patterson Air Force Base, Tech. Rep. WL-TR-93-1147.
Baird, L., Moore, A. W. (1999). Gradient descent for general reinforcement learning. In
Advances in Neural Information Processing Systems 11 (NIPS 1998) ,p p .9 6 8 – 9 7 4 . M I T
Press, Cambridge MA.
Baldassarre, G., Mirolli, M. (Eds.) (2013). Intrinsically Motivated Learning in Natural and
Artiﬁcial Systems . Springer-Verlag, Berlin Heidelberg.

Balke, A., Pearl, J. (1994). Counterfactual probabilities: Computational methods, bounds
and applications. In Proceedings of the Tenth International Conference on Uncertainty in
Artiﬁcial Intelligence (UAI-1994 , pp. 46–54. Morgan Kaufmann.
Baras, D., Meir, R. (2007). Reinforcement learning, spike-time-dependent plasticity, and the
BCM rule. Neural Computation, 19 (8):2245–2279.

References 483
Barnard, E. (1993). Temporal-di↵erence methods and Markov models. IEEE Transactions on
Systems, Man, and Cybernetics, 23 (2):357–365.
Barreto, A. S., Precup, D., Pineau, J. (2011). Reinforcement learning using kernel-based
stochastic factorization. In Advances in Neural Information Processing Systems 24 (NIPS
2011) , pp. 720–728. Curran Associates, Inc.
Bartlett, P. L., Baxter, J. (1999). Hebbian synaptic modiﬁcations in spiking neurons that
learn. Technical report, Research School of Information Sciences and Engineering, Australian
National University.
Bartlett, P. L., Baxter, J. (2000). A biologically plausible and locally optimal learning algorithm
for spiking neurons. Rapport technique, Australian National University.
Barto, A. G. (1985). Learning by statistical cooperation of self-interested neuron-like computing
elements. Human Neurobiology, 4 (4):229–256.
Barto, A. G. (1986). Game-theoretic cooperativity in networks of self-interested units. In

J. S. Denker (Ed.), Neural Networks for Computing , pp. 41–46. American Institute of Physics,
New York.
Barto, A. G. (1989). From chemotaxis to cooperativity: Abstract exercises in neuronal learning
strategies. In R. Durbin, R. Maill and G. Mitchison (Eds.), The Computing Neuron ,p p .7 3 – 9 8 .
Addison-Wesley, Reading, MA.
Barto, A. G. (1990). Connectionist learning for control: An overview. In T. Miller, R. S. Sutton,
and P. J. Werbos (Eds.), Neural Networks for Control , pp. 5–58. MIT Press, Cambridge,
MA.

New York.
MA.
Barto, A. G. (1991). Some learning tasks from a control perspective. In L. Nadel and D. L. Stein
(Eds.), 1990 Lectures in Complex Systems ,p p .1 9 5 – 2 2 3 .A d d i s o n - W e s l e y ,R e d w o o dC i t y ,C A .
Barto, A. G. (1992). Reinforcement learning and adaptive critic methods. In D. A. White and
D. A. Sofge (Eds.), Handbook of Intelligent Control: Neural, Fuzzy, and Adaptive Approaches ,
pp. 469–491. Van Nostrand Reinhold, New York.
Barto, A. G. (1995a). Adaptive critics and the basal ganglia. In J. C. Houk, J. L. Davis, and

D. G. Beiser (Eds.), Models of Information Processing in the Basal Ganglia ,p p .2 1 5 – 2 3 2 .
MIT Press, Cambridge, MA.
Barto, A. G. (1995b). Reinforcement learning. In M. A. Arbib (Ed.), Handbook of Brain Theory
and Neural Networks , pp. 804–809. MIT Press, Cambridge, MA.
Barto, A. G. (2011). Adaptive real-time dynamic programming. In C. Sammut and G. I Webb
(Eds.), Encyclopedia of Machine Learning , pp. 19–22. Springer Science and Business Media.
Barto, A. G. (2013). Intrinsic motivation and reinforcement learning. In G. Baldassarre and M.

Mirolli (Eds.), Intrinsically Motivated Learning in Natural and Artiﬁcial Systems , pp. 17–47.
Springer-Verlag, Berlin Heidelberg.
Barto, A. G., Anandan, P. (1985). Pattern recognizing stochastic learning automata. IEEE
Transactions on Systems, Man, and Cybernetics, 15 (3):360–375.
Barto, A. G., Anderson, C. W. (1985). Structural learning in connectionist systems. In Program
of the Seventh Annual Conference of the Cognitive Science Society , pp. 43–54.
Barto, A. G., Anderson, C. W., Sutton, R. S. (1982). Synthesis of nonlinear control surfaces by

a layered associative search network. Biological Cybernetics, 43 (3):175–185.
Barto, A. G., Bradtke, S. J., Singh, S. P. (1991). Real-time learning and control using
asynchronous dynamic programming. Technical Report 91-57. Department of Computer
and Information Science, University of Massachusetts, Amherst.
Barto, A. G., Bradtke, S. J., Singh, S. P. (1995). Learning to act using real-time dynamic
programming. Artiﬁcial Intelligence, 72 (1-2):81–138.

484 References
Barto, A. G., Du↵, M. (1994). Monte Carlo matrix inversion and reinforcement learning. In
Advances in Neural Information Processing Systems 6 (NIPS 1993) , pp. 687–694. Morgan
Kaufmann, San Francisco.
Barto, A. G., Jordan, M. I. (1987). Gradient following without back-propagation in layered
networks. In M. Caudill and C. Butler (Eds.), Proceedings of the IEEE First Annual
Conference on Neural Networks , pp. II629–II636. SOS Printing, San Diego.
Barto, A. G., Mahadevan, S. (2003). Recent advances in hierarchical reinforcement learning.
Discrete Event Dynamic Systems, 13 (4):341–379.
Barto, A. G., Singh, S. P. (1990). On the computational economics of reinforcement learning. In
Connectionist Models: Proceedings of the 1990 Summer School . Morgan Kaufmann.
Barto, A. G., Sutton, R. S. (1981a). Goal seeking components for adaptive intelligence: An
initial assessment. Technical Report AFWAL-TR-81-1070. Air Force Wright Aeronautical
Laboratories/Avionics Laboratory, Wright-Patterson AFB, OH.

Barto, A. G., Sutton, R. S. (1981b). Landmark learning: An illustration of associative search.
Biological Cybernetics, 42 (1):1–8.
Barto, A. G., Sutton, R. S. (1982). Simulation of anticipatory responses in classical conditioning
by a neuron-like adaptive element. Behavioural Brain Research, 4 (3):221–235.
Barto, A. G., Sutton, R. S., Anderson, C. W. (1983). Neuronlike elements that can solve
dicult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics,

13(5):835–846. Reprinted in J. A. Anderson and E. Rosenfeld (Eds.), Neurocomputing:
Foundations of Research , pp. 535–549. MIT Press, Cambridge, MA, 1988.
Barto, A. G., Sutton, R. S., Brouwer, P. S. (1981). Associative search network: A reinforcement
learning associative memory. Biological Cybernetics, 40 (3):201–211.
Barto, A. G., Sutton, R. S., Watkins, C. J. C. H. (1990). Learning and sequential decision
making. In M. Gabriel and J. Moore (Eds.), Learning and Computational Neuroscience:

Foundations of Adaptive Networks , pp. 539–602. MIT Press, Cambridge, MA.
Baxter, J., Bartlett, P. L. (2001). Inﬁnite-horizon policy-gradient estimation. Journal of Artiﬁcial
Intelligence Research, 15 :319–350.
Baxter, J., Bartlett, P. L., Weaver, L. (2001). Experiments with inﬁnite-horizon, policy-gradient
estimation. Journal of Artiﬁcial Intelligence Research, 15 :351–381.
Bellemare, M. G., Dabney, W., Munos, R. (2017). A distributional perspective on reinforcement
learning. ArXiv:1707.06887.

Bellemare, M. G., Naddaf, Y., Veness, J., Bowling, M. (2013). The arcade learning environment:
An evaluation platform for general agents. Journal of Artiﬁcial Intelligence Research,
47:253–279.
Bellemare, M. G., Veness, J., Bowling, M. (2012). Investigating contingency awareness using
Atari 2600 games. In Proceedings of the Twenty-Sixth AAAI Conference on Artiﬁcial
Intelligence (AAAI-12) , pp. 864–871. AAAI Press, Menlo Park, CA.

Bellman, R. E. (1956). A problem in the sequential design of experiments. Sankhya, 16 :221–229.
Bellman, R. E. (1957a). Dynamic Programming . Princeton University Press, Princeton.
Bellman, R. E. (1957b). A Markov decision process. Journal of Mathematics and Mechanics,
6(5):679–684.
Bellman, R. E., Dreyfus, S. E. (1959). Functional approximations and dynamic programming.
Mathematical Tables and Other Aids to Computation, 13 :247–251.
Bellman, R. E., Kalaba, R., Kotkin, B. (1973). Polynomial approximation—A new computational

technique in dynamic programming: Allocation processes. Mathematical Computation,
17:155–161.

References 485
Bengio, Y. (2009). Learning deep architectures for AI. Foundations and Trends in Machine
Learning, 2 (1):1–27.
Bengio, Y., Courville, A. C., Vincent, P. (2012). Unsupervised feature learning and deep learning:
A review and new perspectives. CoRR 1 , ArXiv:1206.5538.
Bentley, J. L. (1975). Multidimensional binary search trees used for associative searching.
Communications of the ACM, 18 (9):509–517.
Berg, H. C. (1975). Chemotaxis in bacteria. Annual review of biophysics and bioengineering,
4(1):119–136.
Berns, G. S., McClure, S. M., Pagnoni, G., Montague, P. R. (2001). Predictability modulates
human brain response to reward. The journal of neuroscience, 21 (8):2793–2798.
Berridge, K. C., Kringelbach, M. L. (2008). A↵ective neuroscience of pleasure: reward in humans
and animals. Psychopharmacology, 199 (3):457–480.
Berridge, K. C., Robinson, T. E. (1998). What is the role of dopamine in reward: hedonic
impact, reward learning, or incentive salience? Brain Research Reviews, 28 (3):309–369.

Berry, D. A., Fristedt, B. (1985). Bandit Problems .C h a p m a n a n d H a l l , L o n d o n .
Bertsekas, D. P. (1982). Distributed dynamic programming. IEEE Transactions on Automatic
Control, 27 (3):610–616.
Bertsekas, D. P. (1983). Distributed asynchronous computation of ﬁxed points. Mathematical
Programming, 27 (1):107–120.
Bertsekas, D. P. (1987). Dynamic Programming: Deterministic and Stochastic Models . Prentice-
Hall, Englewood Cli↵s, NJ.

Bertsekas, D. P. (2005). Dynamic Programming and Optimal Control, Volume 1, third edition.
Athena Scientiﬁc, Belmont, MA.
Bertsekas, D. P. (2012). Dynamic Programming and Optimal Control, Volume 2: Approximate
Dynamic Programming , fourth edition. Athena Scientiﬁc, Belmont, MA.
Bertsekas, D. P. (2013). Rollout algorithms for discrete optimization: A survey. In Handbook of
Combinatorial Optimization , pp. 2989–3013. Springer, New York.
Bertsekas, D. P., Tsitsiklis, J. N. (1989). Parallel and Distributed Computation: Numerical

Methods . Prentice-Hall, Englewood Cli↵s, NJ.
Bertsekas, D. P., Tsitsiklis, J. N. (1996). Neuro-Dynamic Programming . Athena Scientiﬁc,
Belmont, MA.
Bertsekas, D. P., Tsitsiklis, J. N., Wu, C. (1997). Rollout algorithms for combinatorial optimiza-
tion. Journal of Heuristics, 3 (3):245–262.
Bertsekas, D. P., Yu, H. (2009). Projected equation methods for approximate solution of large
linear systems. Journal of Computational and Applied Mathematics, 227 (1):27–50.
Bhat, N., Farias, V., Moallemi, C. C. (2012). Non-parametric approximate dynamic programming

via the kernel method. In Advances in Neural Information Processing Systems 25 (NIPS
2012) , pp. 386–394. Curran Associates, Inc.
Bhatnagar, S., Sutton, R., Ghavamzadeh, M., Lee, M. (2009). Natural actor–critic algorithms.
Automatica, 45 (11).
Biermann, A. W., Fairﬁeld, J. R. C., Beres, T. R. (1982). Signature table systems and learning.
IEEE Transactions on Systems, Man, and Cybernetics, 12 (5):635–648.
Bishop, C. M. (1995). Neural Networks for Pattern Recognition . Clarendon, Oxford.
Bishop, C. M. (2006). Pattern Recognition and Machine Learning . Springer Science + Business

Media New York LLC.
Blodgett, H. C. (1929). The e↵ect of the introduction of reward upon the maze performance of
rats. University of California Publications in Psychology, 4 :113–134.

486 References
Boakes, R. A., Costa, D. S. J. (2014). Temporal contiguity in associative learning: Iinterference
and decay from an historical perspective. Journal of Experimental Psychology: Animal
Learning and Cognition, 40 (4):381–400.
Booker, L. B. (1982). Intelligent Behavior as an Adaptation to the Task Environment . Ph.D. the-
sis, University of Michigan, Ann Arbor.
Bostrom, N. (2014). Superintelligence: Paths, Dangers, Strategies . Oxford University Press.
Bottou, L., Vapnik, V. (1992). Local learning algorithms. Neural Computation, 4 (6):888–900.
Boyan, J. A. (1999). Least-squares temporal di↵erence learning. In Proceedings of the 16th
International Conference on Machine Learning (ICML 1999) ,p p .4 9 – 5 6 .
Boyan, J. A. (2002). Technical update: Least-squares temporal di↵erence learning. Machine
Learning, 49 (2):233–246.
Boyan, J. A., Moore, A. W. (1995). Generalization in reinforcement learning: Safely approxi-
mating the value function. In Advances in Neural Information Processing Systems 7 (NIPS

1994) , pp. 369–376. MIT Press, Cambridge, MA.
Bradtke, S. J. (1993). Reinforcement learning applied to linear quadratic regulation. In Advances
in Neural Information Processing Systems 5 (NIPS 1992) , pp. 295–302. Morgan Kaufmann.
Bradtke, S. J. (1994). Incremental Dynamic Programming for On-Line Adaptive Optimal Control .
Ph.D. thesis, University of Massachusetts, Amherst. Appeared as CMPSCI Technical Report
94-62.
Bradtke, S. J., Barto, A. G. (1996). Linear least–squares algorithms for temporal di↵erence
learning. Machine Learning, 22 :33–57.

94-62.
Bradtke, S. J., Ydstie, B. E., Barto, A. G. (1994). Adaptive linear quadratic control using policy
iteration. In Proceedings of the American Control Conference , pp. 3475–3479. American
Automatic Control Council, Evanston, IL.
Brafman, R. I., Tennenholtz, M. (2003). R-max – a general polynomial time algorithm for
near-optimal reinforcement learning. Journal of Machine Learning Research, 3 :213–231.
Breiman, L. (2001). Random forests. Machine Learning, 45 (1):5–32.

Breiter, H. C., Aharon, I., Kahneman, D., Dale, A., Shizgal, P. (2001). Functional imaging
of neural responses to expectancy and experience of monetary gains and losses. Neuron,
30(2):619–639.
Breland, K., Breland, M. (1961). The misbehavior of organisms. American Psychologist,
16(11):681–684.
Bridle, J. S. (1990). Training stochastic model recognition algorithms as networks can lead to
maximum mutual information estimates of parameters. In Advances in Neural Information

Processing Systems 2 (NIPS 1989) , pp. 211–217. Morgan Kaufmann, San Mateo, CA.
Broomhead, D. S., Lowe, D. (1988). Multivariable functional interpolation and adaptive networks.
Complex Systems, 2 :321–355.
Bromberg-Martin, E. S., Matsumoto, M., Hong, S., Hikosaka, O. (2010). A pallidus-habenula-
dopamine pathway signals inferred stimulus values. Journal of Neurophysiology, 104 (2):1068–
1076.
Browne, C.B., Powley, E., Whitehouse, D., Lucas, S.M., Cowling, P.I., Rohlfshagen, P., Tavener,

1076.
S., Perez, D., Samothrakis, S., Colton, S. (2012). A survey of monte carlo tree search methods.
IEEE Transactions on Computational Intelligence and AI in Games, 4 (1):1–43.
Brown, J., Bullock, D., Grossberg, S. (1999). How the basal ganglia use parallel excitatory
and inhibitory learning pathways to selectively respond to unexpected rewarding cues. The
Journal of Neuroscience, 19 (23):10502–10511.
Bryson, A. E., Jr. (1996). Optimal control—1950 to 1985. IEEE Control Systems, 13 (3):26–33.

References 487
Buchanan, B. G., Mitchell, T., Smith, R. G., Johnson, C. R., Jr. (1978). Models of learning
systems. Encyclopedia of Computer Science and technology, 11 .
Buhusi, C. V., Schmajuk, N. A. (1999). Timing in simple conditioning and occasion setting: A
neural network approach. Behavioural Processes, 45 (1):33–57.
Bu¸ soniu, L., Lazaric, A., Ghavamzadeh, M., Munos, R., Babu˘ ska, R., De Schutter, B. (2012).
Least-squares methods for policy iteration. In M. Wiering and M. van Otterlo (Eds.),
Reinforcement Learning: State-of-the-Art , pp. 75–109. Springer-Verlag Berlin Heidelberg.
Bush, R. R., Mosteller, F. (1955). Stochastic Models for Learning . Wiley, New York.
Byrne, J. H., Gingrich, K. J., Baxter, D. A. (1990). Computational capabilities of single
neurons: Relationship to simple forms of associative and nonassociative learning in aplysia .
In R. D. Hawkins and G. H. Bower (Eds.), Computational Models of Learning , pp. 31–63.
Academic Press, New York.

Calabresi, P., Picconi, B., Tozzi, A., Filippo, M. D. (2007). Dopamine-mediated regulation of
corticostriatal synaptic plasticity. Trends in Neuroscience, 30 (5):211–219.
Camerer, C. (2011). Behavioral Game Theory: Experiments in Strategic Interaction . Princeton
Campbell, D. T. (1960). Blind variation and selective survival as a general strategy in knowledge-
processes. In M. C. Yovits and S. Cameron (Eds.), Self-Organizing Systems ,p p .2 0 5 – 2 3 1 .
Pergamon, New York.

Cao, X. R. (2009). Stochastic learning and optimization—A sensitivity-based approach. Annual
Reviews in Control, 33 (1):11–24.
Cao, X. R., Chen, H. F. (1997). Perturbation realization, potentials, and sensitivity analysis of
Markov processes. IEEE Transactions on Automatic Control, 42 (10):1382–1393.
Carlstr¨ om, J., Nordstr¨ om, E. (1997). Control of self-similar ATM call trac by reinforcement
learning. In Proceedings of the International Workshop on Applications of Neural Networks

to Telecommunications 3 , pp. 54–62. Erlbaum, Hillsdale, NJ.
Chapman, D., Kaelbling, L. P. (1991). Input generalization in delayed reinforcement learning:
An algorithm and performance comparisons. In Proceedings of the Twelfth International
Conference on Artiﬁcial Intelligence (IJCAI-91) , pp. 726–731. Morgan Kaufmann, San
Mateo, CA.
Chaslot, G., Bakkes, S., Szita, I., Spronck, P. (2008). Monte-Carlo tree search: A new framework

Mateo, CA.
for game AI. In Proceedings of the Fourth AAAI Conference on Artiﬁcial Intelligence and
Interactive Digital Entertainment (AIDE-08) , pp. 216–217. AAAI Press, Menlo Park, CA.
Chow, C.-S., Tsitsiklis, J. N. (1991). An optimal one-way multigrid algorithm for discrete-time
stochastic control. IEEE Transactions on Automatic Control, 36 (8):898–914.
Chrisman, L. (1992). Reinforcement learning with perceptual aliasing: The perceptual distinc-
tions approach. In Proceedings of the Tenth National Conference on Artiﬁcial Intelligence
(AAAI-92) , pp. 183–188. AAAI/MIT Press, Menlo Park, CA.

Christensen, J., Korf, R. E. (1986). A uniﬁed theory of heuristic evaluation functions and
its application to learning. In Proceedings of the Fifth National Conference on Artiﬁcial
Intelligence , pp. 148–152. Morgan Kaufmann.
Cichosz, P. (1995). Truncating temporal di↵erences: On the ecient implementation of TD( )
for reinforcement learning. Journal of Artiﬁcial Intelligence Research, 2 :287–318.
Ciosek, K., Whiteson, S. (2017). Expected policy gradients. ArXiv:1706.05374v1. A revised ver-

sion appeared in Proceedings of the Annual Conference of the Association for the Advancement
of Artiﬁcial Intelligence (AAAI-2018) ,p p .2 8 6 8 – 2 8 7 5 .

488 References
Ciosek, K., Whiteson, S. (2018). Expected policy gradients for reinforcement learning. ArXiv:
1801.03326.
Claridge-Chang, A., Roorda, R. D., Vrontou, E., Sjulson, L., Li, H., Hirsh, J., Miesenb¨ ock, G.
(2009). Writing memories with light-addressable reinforcement circuitry. Cell, 139 (2):405–
415.
Clark, R. E., Squire, L. R. (1998). Classical conditioning and brain systems: the role of awareness.
Science, 280 (5360):77–81.
Clark, W. A., Farley, B. G. (1955). Generalization of pattern recognition in a self-organizing
system. In Proceedings of the 1955 Western Joint Computer Conference ,p p .8 6 – 9 1 .
Clouse, J. (1996). On Integrating Apprentice Learning and Reinforcement Learning TITLE2 .
96-026.
Clouse, J., Utgo↵, P. (1992). A teaching method for reinforcement learning systems. In
Proceedings of the 9th International Workshop on Machine Learning , pp. 92–101. Morgan
Kaufmann.

96-026.
Kaufmann.
Cobo, L. C., Zang, P., Isbell, C. L., Thomaz, A. L. (2011). Automatic state abstraction from
demonstration. In Proceedings of the Twenty-Second International Joint Conference on
Artiﬁcial Intelligence (IJCAI-11) , pp. 1243-1248. AAAI Press.
Connell, J. (1989). A colony architecture for an artiﬁcial creature. Technical Report AI-TR-1151.
MIT Artiﬁcial Intelligence Laboratory, Cambridge, MA.
Connell, M. E., Utgo↵, P. E. (1987). Learning to control a dynamic physical system. Computa-
tional intelligence, 3 (1):330–337.

Contreras-Vidal, J. L., Schultz, W. (1999). A predictive reinforcement model of dopamine neurons
for learning approach behavior. Journal of Computational Neuroscience, 6 (3):191–214.
Coulom, R. (2006). Ecient selectivity and backup operators in Monte-Carlo tree search. In
Proceedings of the 5th International Conference on Computers and Games (CG’06) ,p p .7 2 – 8 3 .
Springer-Verlag Berlin, Heidelberg.
Courville, A. C., Daw, N. D., Touretzky, D. S. (2006). Bayesian theories of conditioning in a
changing world. Trends in Cognitive Science, 10 (7):294–300.

Craik, K. J. W. (1943). The Nature of Explanation . Cambridge University Press, Cambridge.
Cross, J. G. (1973). A stochastic learning model of economic behavior. The Quarterly Journal
of Economics, 87 (2):239–266.
Crow, T. J. (1968). Cortical synapses and reinforcement: a hypothesis. Nature, 219 (5155):736–
737.
Curtiss, J. H. (1954). A theoretical comparison of the eciencies of two classical methods and a
Monte Carlo method for computing one component of the solution of a set of linear algebraic

737.
equations. In H. A. Meyer (Ed.), Symposium on Monte Carlo Methods ,p p .1 9 1 – 2 3 3 .W i l e y ,
New York.
Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. Mathematics of
control, signals and systems, 2 (4):303–314.
Cziko, G. (1995). Without Miracles: Universal Selection Theory and the Second Darvinian
Revolution . MIT Press, Cambridge, MA.
Dabney, W. (2014). Adaptive step-sizes for reinforcement learning . PhD thesis, University of
Massachusetts, Amherst.

New York.
Dabney, W., Barto, A. G. (2012). Adaptive step-size for online temporal di↵erence learning. In
Proceedings of the Annual Conference of the Association for the Advancement of Artiﬁcial
Intelligence (AAAI) .

References 489
Daniel, J. W. (1976). Splines and eciency in dynamic programming. Journal of Mathematical
Analysis and Applications, 54 :402–407.
Dann, C., Neumann, G., Peters, J. (2014). Policy evaluation with temporal di↵erences: A survey
and comparison. Journal of Machine Learning Research, 15 :809–883.
Daw, N. D., Courville, A. C., Touretzky, D. S. (2003). Timing and partial observability in the
dopamine system. In Advances in Neural Information Processing Systems 15 (NIPS 2002) ,
pp. 99–106. MIT Press, Cambridge, MA.
Daw, N. D., Courville, A. C., Touretzky, D. S. (2006). Representation and timing in theories of
the dopamine system. Neural Computation, 18 (7):1637–1677.
Daw, N. D., Niv, Y., Dayan, P. (2005). Uncertainty based competition between prefrontal and
dorsolateral striatal systems for behavioral control. Nature Neuroscience, 8 (12):1704–1711.
Daw, N. D., Shohamy, D. (2008). The cognitive neuroscience of motivation and learning. Social
Cognition, 26 (5):593–620.

Dayan, P. (1991). Reinforcement comparison. In D. S. Touretzky, J. L. Elman, T. J. Sejnowski,
and G. E. Hinton (Eds.), Connectionist Models: Proceedings of the 1990 Summer School ,
pp. 45–51. Morgan Kaufmann.
Dayan, P. (1992). The convergence of TD( ) for general .Machine Learning, 8 (3):341–362.
Dayan, P. (2002). Matters temporal. Trends in Cognitive Sciences, 6 (3):105–106.
Dayan, P., Abbott, L. F. (2001). Theoretical Neuroscience: Computational and Mathematical

Modeling of Neural Systems . MIT Press, Cambridge, MA.
Dayan, P., Berridge, K. C. (2014). Model-based and model-free Pavlovian reward learning:
Revaluation, revision, and revaluation. Cognitive, A↵ective, & Behavioral Neuroscience,
14(2):473–492.
Dayan, P., Niv, Y. (2008). Reinforcement learning: the good, the bad and the ugly. Current
Opinion in Neurobiology, 18 (2):185–196.
Dayan, P., Niv, Y., Seymour, B., Daw, N. D. (2006). The misbehavior of value and the discipline
of the will. Neural Networks, 19 (8):1153–1160.

Dayan, P., Sejnowski, T. (1994). TD( ) converges with probability 1. Machine Learning,
14(3):295–301.
De Asis, K., Hernandez-Garcia, J. F., Holland, G. Z., Sutton, R. S. (2017). Multi-step Rein-
forcement Learning: A Unifying Algorithm. ArXiv:1703.01327.
de Farias, D. P. (2002). The Linear Programming Approach to Approximate Dynamic Program-
ming: Theory and Application. Stanford University PhD thesis.
de Farias, D. P., Van Roy, B. (2003). The linear programming approach to approximate dynamic
programming. Operations Research 51 (6):850–865.

Dean, T., Lin, S.-H. (1995). Decomposition techniques for planning in stochastic domains.
InProceedings of the Fourteenth International Joint Conference on Artiﬁcial Intelligence
(IJCAI-95) , pp. 1121–1127. Morgan Kaufmann. See also Technical Report CS-95-10, Brown
University, Department of Computer Science, 1995.
Degris, T., White, M., Sutton, R. S. (2012). O↵-policy actor–critic. In Proceedings of the 29th
International Conference on Machine Learning (ICML 2012) . ArXiv:1205.4839, 2012.

Denardo, E. V. (1967). Contraction mappings in the theory underlying dynamic programming.
SIAM Review, 9 (2):165–177.
Dennett, D. C. (1978). Why the Law of E↵ect Will Not Go Away. Brainstorms ,p p .7 1 – 8 9 .
Bradford/MIT Press, Cambridge, MA.
Derthick, M. (1984). Variations on the Boltzmann machine learning algorithm. Carnegie-Mellon
University Department of Computer Science Technical Report No. CMU-CS-84-120.

490 References
Deutsch, J. A. (1953). A new type of behaviour theory. British Journal of Psychology. General
Section, 44 (4):304–317.
Deutsch, J. A. (1954). A machine with insight. Quarterly Journal of Experimental Psychology,
6(1):6–11.
Dick, T. (2015). Policy Gradient Reinforcement Learning Without Regret .M . S c .t h e s i s ,
University of Alberta.
Dickinson, A. (1980). Contemporary Animal Learning Theory . Cambridge University Press.
Dickinson, A. (1985). Actions and habits: the development of behavioral autonomy. Phil. Trans.
R. Soc. Lond. B, 308 (1135):67–78.
Dickinson, A., Balleine, B. W. (2002). The role of learning in motivation. In C. R. Gallistel
(Ed.), Stevens’ Handbook of Experimental Psychology ,v o l u m e3 ,p p .4 9 7 – 5 3 3 .W i l e y ,N Y .
Dietterich, T. G., Buchanan, B. G. (1984). The role of the critic in learning systems. In O. G.
Selfridge, E. L. Rissland, and M. A. Arbib (Eds.), Adaptive Control of Ill-Deﬁned Systems ,
pp. 127–147. Plenum Press, NY.

Dietterich, T. G., Flann, N. S. (1995). Explanation-based learning and reinforcement learning:
A uniﬁed view. In A. Prieditis and S. Russell (Eds.), Proceedings of the 12th International
Conference on Machine Learning , pp. 176–184. Morgan Kaufmann.
Dietterich, T. G., Wang, X. (2002). Batch value function approximation via support vectors.
InAdvances in Neural Information Processing Systems 14 , pp. 1491–1498. MIT Press,
Cambridge, MA.
Diuk, C., Cohen, A., Littman, M. L. (2008). An object-oriented representation for ecient

reinforcement learning. In Proceedings of the 25th International Conference on Machine
Learning , pp. 240–247. ACM, New York.
Dolan, R. J., Dayan, P. (2013). Goals and habits in the brain. Neuron, 80 (2):312–325.
Doll, B. B., Simon, D. A., Daw, N. D. (2012). The ubiquity of model-based reinforcement
learning. Current Opinion in Neurobiology, 22 (6):1–7.
Donahoe, J. W., Burgos, J. E. (2000). Behavior analysis and revaluation. Journal of the
Experimental Analysis of Behavior, 74 (3):331–346.
Dorigo, M., Colombetti, M. (1994). Robot shaping: Developing autonomous agents through

learning. Artiﬁcial Intelligence, 71 (2):321–370.
Doya, K. (1996). Temporal di↵erence learning in continuous time and space. In Advances
in Neural Information Processing Systems 8 (NIPS 1995) , pp. 1073–1079. MIT Press,
Doya, K., Sejnowski, T. J. (1995). A novel reinforcement model of birdsong vocalization learning.
InAdvances in Neural Information Processing Systems 7 (NIPS 1994) ,p p .1 0 1 – 1 0 8 .M I T
Press, Cambridge, MA.

Doya, K., Sejnowski, T. J. (1998). A computational model of birdsong learning by auditory
experience and auditory feedback. In P. W. F. Poon and J. F. Brugge (Eds.), Central
Auditory Processing and Neural Modeling , pp. 77–88. Springer, Boston, MA.
Doyle, P. G., Snell, J. L. (1984). Random Walks and Electric Networks .T h e M a t h e m a t i c a l
Association of America. Carus Mathematical Monograph 22.
Dreyfus, S. E., Law, A. M. (1977). The Art and Theory of Dynamic Programming .A c a d e m i c
Press, New York.

Du, S. S., Chen, J., Li, L., Xiao, L., Zhou, D. (2017). Stochastic variance reduction methods for
policy evaluation. Proceedings of the 34th International Conference on Machine Learning ,
pp. 1049–1058. ArXiv:1702.07944.
Duda, R. O., Hart, P. E. (1973). Pattern Classiﬁcation and Scene Analysis . Wiley, New York.

References 491
Du↵, M. O. (1995). Q-learning for bandit problems. In Proceedings of the 12th International
Conference on Machine Learning , pp. 209–217. Morgan Kaufmann.
Egger, D. M., Miller, N. E. (1962). Secondary reinforcement in rats as a function of information
value and reliability of the stimulus. Journal of Experimental Psychology, 64 :97–104.
Eshel, N., Tian, J., Bukwich, M., Uchida, N. (2016). Dopamine neurons share common response
function for reward prediction error. Nature Neuroscience, 19 (3):479–486.
Estes, W. K. (1943). Discriminative conditioning. I. A discriminative property of conditioned
anticipation. Journal of Experimental Psychology, 32 (2):150–155.
Estes, W. K. (1948). Discriminative conditioning. II. E↵ects of a Pavlovian conditioned stimulus
upon a subsequently established operant response. Journal of Experimental Psychology,
38(2):173–177.
Estes, W. K. (1950). Toward a statistical theory of learning. Psychololgical Review, 57 (2):94–107.

Farley, B. G., Clark, W. A. (1954). Simulation of self-organizing systems by digital computer.
IRE Transactions on Information Theory, 4 (4):76–84.
Farries, M. A., Fairhall, A. L. (2007). Reinforcement learning with modulated spike timingde-
pendent synaptic plasticity. Journal of Neurophysiology, 98 (6):3648–3665.
Feldbaum, A. A. (1965). Optimal Control Systems . Academic Press, New York.
Finch, G., Culler, E. (1934). Higher order conditioning with constant motivation. The American
Journal of Psychology :596–602.

Finnsson, H., Bj¨ ornsson, Y. (2008). Simulation-based approach to general game playing. In
Proceedings of the Association for the Advancement of Artiﬁcial Intelligence ,p p .2 5 9 – 2 6 4 .
Fiorillo, C. D., Yun, S. R., Song, M. R. (2013). Diversity and homogeneity in responses of
midbrain dopamine neurons. The Journal of Neuroscience, 33 (11):4693–4709.
Florian, R. V. (2007). Reinforcement learning through modulation of spike-timing-dependent
synaptic plasticity. Neural Computation, 19 (6):1468–1502.

Fogel, L. J., Owens, A. J., Walsh, M. J. (1966). Artiﬁcial Intelligence through Simulated Evolution .
John Wiley and Sons.
French, R. M. (1999). Catastrophic forgetting in connectionist networks. Trends in cognitive
sciences, 3 (4):128–135.
Frey, U., Morris, R. G. M. (1997). Synaptic tagging and long-term potentiation. Nature,
385(6616):533–536.
Fr´ emaux, N., Sprekeler, H., Gerstner, W. (2010). Functional requirements for reward-modulated

spike-timing-dependent plasticity. The Journal of Neuroscience, 30 (40): 13326–13337
Friedman, J. H., Bentley, J. L., Finkel, R. A. (1977). An algorithm for ﬁnding best matches in
logarithmic expected time. ACM Transactions on Mathematical Software, 3 (3):209–226.
Friston, K. J., Tononi, G., Reeke, G. N., Sporns, O., Edelman, G. M. (1994). Value-dependent
selection in the brain: Simulation in a synthetic neural model. Neuroscience, 59 (2):229–243.

Fu, K. S. (1970). Learning control systems—Review and outlook. IEEE Transactions on
Automatic Control, 15 (2):210–221.
Galanter, E., Gerstenhaber, M. (1956). On thought: The extrinsic theory. Psychological Review,
63(4):218–227.
Gallistel, C. R. (2005). Deconstructing the law of e↵ect. Games and Economic Behavior,
52(2):410–423.
Gardner, M. (1973). Mathematical games. Scientiﬁc American, 228 (1):108–115.
Geist, M., Scherrer, B. (2014). O↵-policy learning with eligibility traces: A survey. Journal of
Machine Learning Research, 15 (1):289–333.

Gelly, S., Silver, D. (2007). Combining online and o✏ine knowledge in UCT. Proceedings of the

492 References
24th International Conference on Machine Learning (ICML 2007 ) ,p p . 2 7 3 – 2 8 0 .
Gelperin, A., Hopﬁeld, J. J., Tank, D. W. (1985). The logic of limax learning. In A. Selverston
(Ed.), Model Neural Networks and Behavior , pp. 247–261. Plenum Press, New York.
Genesereth, M., Thielscher, M. (2014). General game playing. Synthesis Lectures on Artiﬁcial
Intelligence and Machine Learning, 8 (2):1–229.
Gershman, S. J., Moustafa, A. A., Ludvig, E. A. (2014). Time representation in reinforcement
learning models of the basal ganglia. Frontiers in Computational Neuroscience, 7 :194.
Gershman, S. J., Pesaran, B., Daw, N. D. (2009). Human reinforcement learning subdivides
structured action spaces by learning e↵ector-speciﬁc values. The Journal of Neuroscience,
29(43):13524–13531.
Ghiassian, S., Raﬁee, B., Sutton, R. S. (2016). A ﬁrst empirical study of emphatic temporal
di↵erence learning. Workshop on Continual Learning and Deep Learning at the Conference

on Neural Information Processing Systems (NIPS 2016). ArXiv:1705.04185.
Ghiassian, S., White, A., White, M., Sutton, R. S. (in preparation). A comparison of o↵-policy
policy evaluation methods.
Gibbs, C. M., Cool, V., Land, T., Kehoe, E. J., Gormezano, I. (1991). Second-order conditioning
of the rabbit’s nictitating membrane response. Integrative Physiological and Behavioral
Science, 26 (4):282–295.
Gittins, J. C., Jones, D. M. (1974). A dynamic allocation index for the sequential design of

experiments. In J. Gani, K. Sarkadi, and I. Vincze (Eds.), Progress in Statistics ,p p .2 4 1 – 2 6 6 .
North-Holland, Amsterdam–London.
Glimcher, P. W. (2011). Understanding dopamine and reinforcement learning: The dopamine
reward prediction error hypothesis. Proceedings of the National Academy of Sciences,
108(Supplement 3):15647–15654.
Glimcher, P. W. (2003). Decisions, Uncertainty, and the Brain: The science of Neuroeconomics .

Glimcher, P. W., Fehr, E. (Eds.) (2013). Neuroeconomics: Decision Making and the Brain,
Second Edition . Academic Press.
Goethe, J. W. V. (1878). The Sorcerer’s Apprentice. In The Permanent Goethe ,p .3 4 9 .T h e
Dial Press, Inc., New York.
Goldstein, H. (1957). Classical Mechanics .A d d i s o n - W e s l e y , R e a d i n g , M A .
Goodfellow, I., Bengio, Y., Courville, A. (2016). Deep Learning . MIT Press, Cambridge, MA.
Goodwin, G. C., Sin, K. S. (1984). Adaptive Filtering Prediction and Control . Prentice-Hall,
Englewood Cli↵s, NJ.

Gopnik, A., Glymour, C., Sobel, D., Schulz, L. E., Kushnir, T., Danks, D. (2004). A theory of
causal learning in children: Causal maps and Bayes nets. Psychological Review, 111 (1):3–32.
Gordon, G. J. (1995). Stable function approximation in dynamic programming. In A. Prieditis
and S. Russell (Eds.), Proceedings of the 12th International Conference on Machine Learning
(ICML 1995) , pp. 261–268. Morgan Kaufmann. An expanded version was published as
Technical Report CMU-CS-95-103. Carnegie Mellon University, Pittsburgh, PA, 1995.

Gordon, G. J. (1996a). Chattering in SARSA( ). CMU learning lab internal report.
Gordon, G. J. (1996b). Stable ﬁtted reinforcement learning. In Advances in Neural Information
Processing Systems 8 (NIPS 1995) , pp. 1052–1058. MIT Press, Cambridge, MA.
Gordon, G. J. (1999). Approximate Solutions to Markov Decision Processes . Ph.D. thesis,
Carnegie Mellon University, Pittsburgh PA. Pittsburgh, PA.

References 493
Gordon, G. J. (2001). Reinforcement learning with function approximation converges to a region.
InAdvances in Neural Information Processing Systems 13 (NIPS 2000) ,p p .1 0 4 0 – 1 0 4 6 .M I T
Graybiel, A. M. (2000). The basal ganglia. Current Biology, 10 (14):R509–R511.
Greensmith, E., Bartlett, P. L., Baxter, J. (2002). Variance reduction techniques for gradient
estimates in reinforcement learning. In Advances in Neural Information Processing Systems
14 (NIPS 2001) , pp. 1507–1514. MIT Press, Cambridge, MA.
Greensmith, E., Bartlett, P. L., Baxter, J. (2004). Variance reduction techniques for gradient
estimates in reinforcement learning. Journal of Machine Learning Research, 5 (Nov):1471–
1530.
Grith, A. K. (1966). A new machine learning technique applied to the game of checkers.
Technical Report Project MAC, Artiﬁcial Intelligence Memo 94. Massachusetts Institute of
Technology, Cambridge, MA.

1530.
Grith, A. K. (1974). A comparison and evaluation of three machine learning procedures as
applied to the game of checkers. Artiﬁcial Intelligence, 5 (2):137–148.
Grondman, I., Busoniu, L., Lopes, G. A., Babuska, R. (2012). A survey of actor–critic reinforce-
ment learning: Standard and natural policy gradients. IEEE Transactions on Systems, Man,
and Cybernetics, Part C (Applications and Reviews), 42 (6):1291–1307.
Grossberg, S. (1975). A neural model of attention, reinforcement, and discrimination learning.
International Review of Neurobiology, 18 :263–327.

Grossberg, S., Schmajuk, N. A. (1989). Neural dynamics of adaptive timing and temporal
discrimination during associative learning. Neural Networks, 2 (2):79–102.
Gullapalli, V. (1990). A stochastic reinforcement algorithm for learning real-valued functions.
Neural Networks, 3 (6): 671–692.
Gullapalli, V., Barto, A. G. (1992). Shaping as a method for accelerating reinforcement learning.
InProceedings of the 1992 IEEE International Symposium on Intelligent Control ,p p .5 5 4 – 5 5 9 .
IEEE.

IEEE.
Gurvits, L., Lin, L.-J., Hanson, S. J. (1994). Incremental learning of evaluation functions
for absorbing Markov chains: New methods and theorems. Siemans Corporate Research,
Princeton, NJ.
Hackman, L. (2012). Faster Gradient-TD Algorithms . M.Sc. thesis, University of Alberta,
Edmonton.
Hallak, A., Tamar, A., Mannor, S. (2015). Emphatic TD Bellman operator is a contraction.
ArXiv:1508.03411.
Hallak, A., Tamar, A., Munos, R., Mannor, S. (2016). Generalized emphatic temporal di↵erence

IEEE.
Edmonton.
learning: Bias-variance analysis. In Proceedings of the Thirtieth AAAI Conference on
Artiﬁcial Intelligence (AAAI-16) , pp. 1631–1637. AAAI Press, Menlo Park, CA.
Hammer, M. (1997). The neural basis of associative reward learning in honeybees. Trends in
Neuroscience, 20 (6):245–252.
Hammer, M., Menzel, R. (1995). Learning and memory in the honeybee. The Journal of
Neuroscience, 15 (3):1617–1630.
Hampson, S. E. (1983). AN e u r a lM o d e lo fA d a p t i v eB e h a v i o r . Ph.D. thesis, University of
California, Irvine.

Hampson, S. E. (1989). Connectionist Problem Solving: Computational Aspects of Biological
Learning . Birkhauser, Boston.
Hare, T. A., O’Doherty, J., Camerer, C. F., Schultz, W., Rangel, A. (2008). Dissociating the role
of the orbitofrontal cortex and the striatum in the computation of goal values and prediction

494 References
errors. The Journal of Neuroscience, 28 (22):5623–5630.
Harth, E., Tzanakou, E. (1974). Alopex: A stochastic method for determining visual receptive
ﬁelds. Vision Research, 14 (12):1475–1482.
Hassabis, D., Maguire, E. A. (2007). Deconstructing episodic memory with construction. Trends
in Cognitive Sciences, 11 (7):299–306.
Hauskrecht, M., Meuleau, N., Kaelbling, L. P., Dean, T., Boutilier, C. (1998). Hierarchical
solution of Markov decision processes using macro-actions. In Proceedings of the Fourteenth
Conference on Uncertainty in Artiﬁcial Intelligence , pp. 220–229. Morgan Kaufmann.
Hawkins, R. D., Kandel, E. R. (1984). Is there a cell-biological alphabet for simple forms of
learning? Psychological Review, 91 (3):375–391.
Haykin, S. (1994). Neural networks: A Comprehensive Foundation , Macmillan, New York.
He, K., Huertas, M., Hong, S. Z., Tie, X., Hell, J. W., Shouval, H., Kirkwood, A. (2015). Distinct
eligibility traces for LTP and LTD in cortical synapses. Neuron, 88 (3):528–538.

He, K., Zhang, X., Ren, S., Sun, J. (2016). Deep residual learning for image recognition. In
Proceedings of the 1992 IEEE Conference on Computer Vision and Pattern Recognition ,
pp. 770–778.
Hebb, D. O. (1949). The Organization of Behavior: A Neuropsychological Theory .J o h n W i l e y
and Sons Inc., New York. Reissued by Lawrence Erlbaum Associates Inc., Mahwah NJ, 2002.
Hengst, B. (2012). Hierarchical approaches. In M. Wiering and M. van Otterlo (Eds.), Rein-

forcement Learning: State-of-the-Art , pp. 293–323. Springer-Verlag Berlin Heidelberg.
Herrnstein, R. J. (1970). On the Law of E↵ect. Journal of the Experimental Analysis of Behavior,
13(2):243–266.
Hersh, R., Griego, R. J. (1969). Brownian motion and potential theory. Scientiﬁc American,
220(3):66–74.
Hester, T., Stone, P. (2012). Learning and using models. In M. Wiering and M. van Otterlo (Eds.),
Reinforcement Learning: State-of-the-Art , pp. 111–141. Springer-Verlag Berlin Heidelberg.

Hesterberg, T. C. (1988), Advances in Importance Sampling , Ph.D. thesis, Statistics Department,
Stanford University.
Hilgard, E. R. (1956). Theories of Learning, Second Edition . Appleton-Century-Cofts, Inc.,
New York.
Hilgard, E. R., Bower, G. H. (1975). Theories of Learning . Prentice-Hall, Englewood Cli↵s, NJ.
Hinton, G. E. (1984). Distributed representations. Technical Report CMU-CS-84-157. Depart-
ment of Computer Science, Carnegie-Mellon University, Pittsburgh, PA.

New York.
Hinton, G. E., Osindero, S., Teh, Y. (2006). A fast learning algorithm for deep belief nets.
Neural Computation, 18 (7):1527–1554.
Hochreiter, S., Schmidhuber, J. (1997). LTSM can solve hard time lag problems. In Advances in
Neural Information Processing Systems 9 (NIPS 1996) , pp. 473–479. MIT Press, Cambridge,
MA.
Holland, J. H. (1975). Adaptation in Natural and Artiﬁcial Systems . University of Michigan
Press, Ann Arbor.
Holland, J. H. (1976). Adaptation. In R. Rosen and F. M. Snell (Eds.), Progress in Theoretical

MA.
Biology , vol. 4, pp. 263–293. Academic Press, New York.
Holland, J. H. (1986). Escaping brittleness: The possibility of general-purpose learning algorithms
applied to rule-based systems. In R. S. Michalski, J. G. Carbonell, and T. M. Mitchell
(Eds.), Machine Learning: An Artiﬁcial Intelligence Approach , vol. 2, pp. 593–623. Morgan
Kaufmann.
Hollerman, J. R., Schultz, W. (1998). Dopmine neurons report an error in the temporal

References 495
prediction of reward during learning. Nature Neuroscience, 1 (4):304–309.
Houk, J. C., Adams, J. L., Barto, A. G. (1995). A model of how the basal ganglia generates and
uses neural signals that predict reinforcement. In J. C. Houk, J. L. Davis, and D. G. Beiser
(Eds.), Models of Information Processing in the Basal Ganglia , pp. 249–270. MIT Press,
Howard, R. (1960). Dynamic Programming and Markov Processes . MIT Press, Cambridge, MA.
Hull, C. L. (1932). The goal-gradient hypothesis and maze learning. Psychological Review,
39(1):25–43.
Hull, C. L. (1943). Principles of Behavior . Appleton-Century, New York.
Hull, C. L. (1952). AB e h a v i o rS y s t e m . Wiley, New York.
Io↵e, S., Szegedy, C. (2015). Batch normalization: Accelerating deep network training by
reducing internal covariate shift. ArXiv:1502.03167.
˙Ipek, E., Mutlu, O., Mart´ ınez, J. F., Caruana, R. (2008). Self-optimizing memory controllers: A

reinforcement learning approach. In ISCA’08:Proceedings of the 35th Annual International
Symposium on Computer Architecture , pp. 39–50. IEEE Computer Society Washington, DC.
Izhikevich, E. M. (2007). Solving the distal reward problem through linkage of STDP and
dopamine signaling. Cerebral Cortex, 17 (10):2443–2452.
Jaakkola, T., Jordan, M. I., Singh, S. P. (1994). On the convergence of stochastic iterative
dynamic programming algorithms. Neural Computation, 6 :1185–1201.

Jaakkola, T., Singh, S. P., Jordan, M. I. (1995). Reinforcement learning algorithm for partially
observable Markov decision problems. In Advances in Neural Information Processing Systems
7( N I P S1 9 9 4 ) , pp. 345–352. MIT Press, Cambridge, MA.
Jacobs, R. A. (1988). Increased rates of convergence through learning rate adaptation. Neural
Networks, 1 (4):295–307.
Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, T., Leibo, J. Z., Silver, D., Kavukcuoglu, K.

(2016). Reinforcement learning with unsupervised auxiliary tasks. ArXiv:1611.05397.
Jaeger, H. (1997). Observable operator models and conditioned continuation representations. Ar-
beitspapiere der GMD 1043, GMD Forschungszentrum Informationstechnik, Sankt Augustin,
Germany.
Jaeger, H. (1998). Discrete Time, Discrete Valued Observable Operator Models: A Tutorial .
GMD-Forschungszentrum Informationstechnik.
Jaeger, H. (2000). Observable operator models for discrete stochastic time series. Neural
Computation, 12 (6):1371–1398.

Germany.
Jaeger, H. (2002). Tutorial on training recurrent neural networks, covering BPPT, RTRL, EKF
and the ‘echo state network’ approach. German National Research Center for Information
Technology, Technical Report GMD report 159, 2002.
Joel, D., Niv, Y., Ruppin, E. (2002). Actor–critic models of the basal ganglia: New anatomical
and computational perspectives. Neural Networks, 15 (4):535–547.
Johnson, A., Redish, A. D. (2007). Neural ensembles in CA3 transiently encode paths forward
of the animal at a decision point. The Journal of Neuroscience, 27 (45):12176–12189.

Kaelbling, L. P. (1993a). Hierarchical learning in stochastic domains: Preliminary results.
InProceedings of the 10th International Conference on Machine Learning (ICML 1993) ,
pp. 167–173. Morgan Kaufmann.
Kaelbling, L. P. (1993b). Learning in Embedded Systems . MIT Press, Cambridge, MA.
Kaelbling, L. P. (Ed.) (1996). Special triple issue on reinforcement learning, Machine Learning,
22(1/2/3).
Kaelbling, L. P., Littman, M. L., Moore, A. W. (1996). Reinforcement learning: A survey.

496 References
Journal of Artiﬁcial Intelligence Research, 4 :237–285.
Kakade, S. M. (2002). A natural policy gradient. In Advances in Neural Information Processing
Systems 14 (NIPS 2001) , pp. 1531–1538. MIT Press, Cambridge, MA.
Kakade, S. M. (2003). On the Sample Complexity of Reinforcement Learning . Ph.D. thesis,
University of London.
Kakutani, S. (1945). Markov processes and the Dirichlet problem. Proceedings of the Japan
Academy, 21 (3-10):227–233.
Kalos, M. H., Whitlock, P. A. (1986). Monte Carlo Methods . Wiley, New York.
Kamin, L. J. (1968). “Attention-like” processes in classical conditioning. In M. R. Jones (Ed.),
Miami Symposium on the Prediction of Behavior, 1967: Aversive Stimulation ,p p .9 – 3 1 .
University of Miami Press, Coral Gables, Florida.
Kamin, L. J. (1969). Predictability, surprise, attention, and conditioning. In B. A. Campbell and
R. M. Church (Eds.), Punishment and Aversive Behavior , pp. 279–296. Appleton-Century-
Crofts, New York.

Kandel, E. R., Schwartz, J. H., Jessell, T. M., Siegelbaum, S. A., Hudspeth, A. J. (Eds.) (2013).
Principles of Neural Science, Fifth Edition . McGraw-Hill Companies, Inc.
Karampatziakis, N., Langford, J. (2010). Online importance weight aware updates. ArXiv:1011.1576.
Kashyap, R. L., Blaydon, C. C., Fu, K. S. (1970). Stochastic approximation. In J. M. Mendel
and K. S. Fu (Eds.), Adaptive, Learning, and Pattern Recognition Systems: Theory and
Applications , pp. 329–355. Academic Press, New York.

Kearney, A., Veeriah, V, Travnik, J, Sutton, R. S., Pilarski, P. M. (in preparation). TIDBD:
Adapting Temporal-di↵erence Step-sizes Through Stochastic Meta-descent.
Kearns, M., Singh, S. (2002). Near-optimal reinforcement learning in polynomial time. Machine
Learning, 49 (2-3):209–232.
Keerthi, S. S., Ravindran, B. (1997). Reinforcement learning. In E. Fieslerm and R. Beale
(Eds.), Handbook of Neural Computation , C3. Oxford University Press, New York.

Kehoe, E. J. (1982). Conditioning with serial compound stimuli: Theoretical and empirical
issues. Experimental Animal Behavior, 1 :30–65.
Kehoe, E. J., Schreurs, B. G., Graham, P. (1987). Temporal primacy overrides prior training
in serial compound conditioning of the rabbit’s nictitating membrane response. Animal
Learning & Behavior, 15 (4):455–464.
Keiﬂin, R., Janak, P. H. (2015). Dopamine prediction errors in reward learning and addiction:
Ffrom theory to neural circuitry. Neuron, 88 (2):247– 263.

Kimble, G. A. (1961). Hilgard and Marquis’ Conditioning and Learning . Appleton-Century-
Kimble, G. A. (1967). Foundations of Conditioning and Learning . Appleton-Century-Crofts,
New York.
Kingma, D., Ba, J. (2014). Adam: A method for stochastic optimization. ArXiv:1412.6980.
Klopf, A. H. (1972). Brain function and adaptive systems—A heterostatic theory. Technical
Report AFCRL-72-0164, Air Force Cambridge Research Laboratories, Bedford, MA. A

New York.
summary appears in Proceedings of the International Conference on Systems, Man, and
Cybernetics (1974) . IEEE Systems, Man, and Cybernetics Society, Dallas, TX.
Klopf, A. H. (1975). A comparison of natural and artiﬁcial intelligence. SIGART Newsletter,
53:11–13.
Klopf, A. H. (1982). The Hedonistic Neuron: A Theory of Memory, Learning, and Intelligence .
Hemisphere, Washington, DC.
Klopf, A. H. (1988). A neuronal model of classical conditioning. Psychobiology, 16 (2):85–125.

References 497
Klyubin, A. S., Polani, D., Nehaniv, C. L. (2005). Empowerment: A universal agent-centric
measure of control. In Proceedings of the 2005 IEEE Congress on Evolutionary Computation
(Vol. 1, pp. 128–135). IEEE.
Kober, J., Peters, J. (2012). Reinforcement learning in robotics: A survey. In M. Wiering, M.
van Otterlo (Eds.), Reinforcement Learning: State-of-the-Art , pp. 579–610. Springer-Verlag.
Kocsis, L., Szepesv´ ari, Cs. (2006). Bandit based Monte-Carlo planning. In Proceedings of the
European Conference on Machine Learning , pp. 282–293. Springer-Verlag Berlin Heidelberg.
Kohonen, T. (1977). Associative Memory: A System Theoretic Approach . Springer-Verlag,
Berlin.
Koller, D., Friedman, N. (2009). Probabilistic Graphical Models: Principles and Techniques .
MIT Press.
Kolodziejski, C., Porr, B., W¨ org¨ otter, F. (2009). On the asymptotic equivalence between
di↵erential Hebbian and temporal di↵erence learning. Neural Computation, 21 (4):1173–1202.

Berlin.
MIT Press.
Kolter, J. Z. (2011). The ﬁxed points of o↵-policy TD. In Advances in Neural Information
Processing Systems 24 (NIPS 2011) , pp. 2169–2177. Curran Associates, Inc.
Konda, V. R., Tsitsiklis, J. N. (2000). Actor-critic algorithms. In Advances in Neural Information
Processing Systems 12 (NIPS 1999) , pp. 1008–1014. MIT Press, Cambridge, MA.
Konda, V. R., Tsitsiklis, J. N. (2003). On actor-critic algorithms. SIAM Journal on Control
and Optimization, 42 (4):1143–1166.

Konidaris, G. D., Osentoski, S., Thomas, P. S. (2011). Value function approximation in
reinforcement learning using the Fourier basis . In Proceedings of the Twenty-Fifth Conference
of the Association for the Advancement of Artiﬁcial Intelligence ,p p .3 8 0 – 3 8 5 .
Korf, R. E. (1988). Optimal path ﬁnding algorithms. In L. N. Kanal and V. Kumar (Eds.),
Search in Artiﬁcial Intelligence , pp. 223–267. Springer-Verlag, Berlin.
Korf, R. E. (1990). Real-time heuristic search. Artiﬁcial Intelligence, 42 (2–3), 189–211.

Koshland, D. E. (1980). Bacterial Chemotaxis as a Model Behavioral System . Raven Press, New
York.
Koza, J. R. (1992). Genetic Programming: On the Programming of Computers by Means of
Natural Selection (Vol. 1). MIT Press., Cambridge, MA.
Kraft, L. G., Campagna, D. P. (1990). A summary comparison of CMAC neural network and
traditional adaptive control systems. In T. Miller, R. S. Sutton, and P. J. Werbos (Eds.),
Neural Networks for Control , pp. 143–169. MIT Press, Cambridge, MA.
Kraft, L. G., Miller, W. T., Dietz, D. (1992). Development and application of CMAC neural

York.
network-based control. In D. A. White and D. A. Sofge (Eds.), Handbook of Intelligent
Control: Neural, Fuzzy, and Adaptive Approaches , pp. 215–232. Van Nostrand Reinhold,
New York.
Kumar, P. R., Varaiya, P. (1986). Stochastic Systems: Estimation, Identiﬁcation, and Adaptive
Control . Prentice-Hall, Englewood Cli↵s, NJ.
Kumar, P. R. (1985). A survey of some results in stochastic adaptive control. SIAM Journal of
Control and Optimization, 23 (3):329–380.

New York.
Kumar, V., Kanal, L. N. (1988). The CDP, A unifying formulation for heuristic search, dynamic
programming, and branch-and-bound. In L. N. Kanal and V. Kumar (Eds.), Search in
Artiﬁcial Intelligence , pp. 1–37. Springer-Verlag, Berlin.
Kushner, H. J., Dupuis, P. (1992). Numerical Methods for Stochastic Control Problems in
Continuous Time . Springer-Verlag, New York.
Kuvayev, L., Sutton, R.S. (1996). Model-based reinforcement learning with an approximate,
learned model. Proceedings of the Ninth Yale Workshop on Adaptive and Learning Systems ,

498 References
pp. 101–105, Yale University, New Haven, CT.
Lagoudakis, M., Parr, R. (2003). Least squares policy iteration. Journal of Machine Learning
Research, 4 (Dec):1107–1149.
Lai, T. L., Robbins, H. (1985). Asymptotically ecient adaptive allocation rules. Advances in
Applied Mathematics, 6 (1):4–22.
Lakshmivarahan, S., Narendra, K. S. (1982). Learning algorithms for two-person zero-sum
stochastic games with incomplete information: A uniﬁed approach. SIAM Journal of Control
and Optimization, 20 (4):541–552.
Lammel, S., Lim, B. K., Malenka, R. C. (2014). Reward and aversion in a heterogeneous
midbrain dopamine system. Neuropharmacology, 76 :353–359.
Lane, S. H., Handelman, D. A., Gelfand, J. J. (1992). Theory and development of higher-order
CMAC neural networks. IEEE Control Systems, 12 (2):23–30.
LeCun, Y. (1985). Une procdure d’apprentissage pour rseau a seuil asymmetrique (a learning
scheme for asymmetric threshold networks). In Proceedings of Cognitiva 85 , Paris, France.

LeCun, Y., Bottou, L., Bengio, Y., Ha↵ner, P. (1998). Gradient-based learning applied to
document recognition. Proceedings of the IEEE, 86 (11):2278–2324.
Legenstein, R. W., Maass, D. P. (2008). A learning theory for reward-modulated spike-timing-
dependent plasticity with application to biofeedback. PLoS Computational Biology, 4 (10).
Levy, W. B., Steward, D. (1983). Temporal contiguity requirements for long-term associative
potentiation/depression in the hippocampus. Neuroscience, 8 (4):791–797.

Lewis, F. L., Liu, D. (Eds.) (2012). Reinforcement Learning and Approximate Dynamic
Programming for Feedback Control .J o h nW i l e ya n dS o n s .
Lewis, R. L., Howes, A., Singh, S. (2014). Computational rationality: Linking mechanism and
behavior through utility maximization. Topics in Cognitive Science, 6 (2):279–311.
Li, L. (2012). Sample complexity bounds of exploration. In M. Wiering and M. van Otterlo (Eds.),
Reinforcement Learning: State-of-the-Art , pp. 175–204. Springer-Verlag Berlin Heidelberg.

Li, L., Chu, W., Langford, J., Schapire, R. E. (2010). A contextual-bandit approach to personal-
ized news article recommendation. In Proceedings of the 19th International Conference on
World Wide Web , pp. 661–670. ACM, New York.
Lin, C.-S., Kim, H. (1991). CMAC-based adaptive critic self-learning control. IEEE Transactions
on Neural Networks, 2 (5):530–533.
Lin, L.-J. (1992). Self-improving reactive agents based on reinforcement learning, planning and
teaching. Machine Learning, 8 (3-4):293–321.

Lin, L.-J., Mitchell, T. (1992). Reinforcement learning with hidden states. In Proceedings of
the Second International Conference on Simulation of Adaptive Behavior: From Animals to
Animats , pp. 271–280. MIT Press, Cambridge, MA.
Littman, M. L., Cassandra, A. R., Kaelbling, L. P. (1995). Learning policies for partially
observable environments: Scaling up. In Proceedings of the 12th International Conference on
Machine Learning (ICML 1995) , pp. 362–370. Morgan Kaufmann.

Littman, M. L., Dean, T. L., Kaelbling, L. P. (1995). On the complexity of solving Markov
decision problems. In Proceedings of the Eleventh Annual Conference on Uncertainty in
Artiﬁcial Intelligence ,p p .3 9 4 – 4 0 2 .
Littman, M. L., Sutton, R. S., Singh (2002). Predictive representations of state. In Advances
in Neural Information Processing Systems 14 (NIPS 2001) , pp. 1555-1561. MIT Press,
Liu, J. S. (2001). Monte Carlo Strategies in Scientiﬁc Computing . Springer-Verlag, Berlin.

Ljung, L. (1998). System identiﬁcation. In A. Proch´ azka, J. Uhl´ ıˆ r, P. W. J. Rayner, and N. G.

References 499
Kingsbury (Eds.), Signal Analysis and Prediction , pp. 163–173. Springer Science + Business
Media New York, LLC.
Ljung, L., S¨ oderstrom, T. (1983). Theory and Practice of Recursive Identiﬁcation . MIT Press,
Ljungberg, T., Apicella, P., Schultz, W. (1992). Responses of monkey dopamine neurons during
learning of behavioral reactions. Journal of Neurophysiology, 67 (1):145–163.
Lovejoy, W. S. (1991). A survey of algorithmic methods for partially observed Markov decision
processes. Annals of Operations Research, 28 (1):47–66.
Luce, D. (1959). Individual Choice Behavior . Wiley, New York.
Ludvig, E. A., Bellemare, M. G., Pearson, K. G. (2011). A primer on reinforcement learning
in the brain: Psychological, computational, and neural perspectives. In E. Alonso and E.
Mondrag´ on (Eds.), Computational Neuroscience for Advancing Artiﬁcial Intelligence: Models,
Methods and Applications , pp. 111–44. Medical Information Science Reference, Hershey PA.

Ludvig, E. A., Sutton, R. S., Kehoe, E. J. (2008). Stimulus representation and the timing of
reward-prediction errors in models of the dopamine system. Neural Computation, 20 (12):3034–
3054.
Ludvig, E. A., Sutton, R. S., Kehoe, E. J. (2012). Evaluating the TD model of classical
conditioning. Learning & behavior, 40 (3):305–319.
Machado, A. (1997). Learning the temporal dynamics of behavior. Psychological Review,
104(2):241–265.
Mackintosh, N. J. (1975). A theory of attention: Variations in the associability of stimuli with

3054.
reinforcement. Psychological Review, 82 (4):276–298.
Mackintosh, N. J. (1983). Conditioning and Associative Learning . Clarendon Press, Oxford.
Maclin, R., Shavlik, J. W. (1994). Incorporating advice into agents that learn from reinforcements.
InProceedings of the Twelfth National Conference on Artiﬁcial Intelligence (AAAI-94) ,
pp. 694–699. AAAI Press, Menlo Park, CA.
Maei, H. R. (2011). Gradient Temporal-Di↵erence Learning Algorithms . Ph.D. thesis, University
of Alberta, Edmonton.
Maei, H. R. (2018). Convergent actor-critic algorithms under o↵-policy training and function

approximation. ArXiv:1802.07842.
Maei, H. R., Sutton, R. S. (2010). GQ( ): A general gradient algorithm for temporal-di↵erence
prediction learning with eligibility traces. In Proceedings of the Third Conference on Artiﬁcial
General Intelligence, pp. 91–96.
Maei, H. R., Szepesv´ ari, Cs., Bhatnagar, S., Precup, D., Silver, D., Sutton, R. S. (2009).
Convergent temporal-di↵erence learning with arbitrary smooth function approximation. In
Advances in Neural Information Processing Systems 22 (NIPS 2009) , pp. 1204–1212. Curran
Associates, Inc.

Maei, H. R., Szepesv´ ari, Cs., Bhatnagar, S., Sutton, R. S. (2010). Toward o↵-policy learning
control with function approximation. In Proceedings of the 27th International Conference on
Machine Learning (ICML 2010) ,p p .7 1 9 – 7 2 6 ) .
Mahadevan, S. (1996). Average reward reinforcement learning: Foundations, algorithms, and
empirical results. Machine Learning, 22 (1):159–196.
Mahadevan, S., Liu, B., Thomas, P., Dabney, W., Giguere, S., Jacek, N., Gemp, I., Liu, J. (2014).

Proximal reinforcement learning: A new theory of sequential decision making in primal-dual
spaces. ArXiv:1405.6757.
Mahadevan, S., Connell, J. (1992). Automatic programming of behavior-based robots using
reinforcement learning. Artiﬁcial Intelligence, 55 (2-3):311–365.

500 References
Mahmood, A. R. (2017). Incremental O↵-Policy Reinforcement Learning Algorithms. Ph.D.
thesis, University of Alberta, Edmonton.
Mahmood, A. R., Sutton, R. S. (2015). O↵-policy learning based on weighted importance
sampling with linear computational complexity. In Proceedings of the 31st Conference on
Uncertainty in Artiﬁcial Intelligence (UAI-2015) , pp. 552–561. AUAI Press Corvallis, Oregon.
Mahmood, A. R., Sutton, R. S., Degris, T., Pilarski, P. M. (2012). Tuning-free step-size adapta-
tion. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing
(ICASSP), Proceedings ,p p .2 1 2 1 – 2 1 2 4 .I E E E .
Mahmood, A. R., Yu, H, Sutton, R. S. (2017). Multi-step o↵-policy learning without importance
sampling ratios. ArXiv:1702.03006.
Mahmood, A. R., van Hasselt, H., Sutton, R. S. (2014). Weighted importance sampling for
o↵-policy learning with linear function approximation. Advances in Neural Information

Processing Systems 27 (NIPS 2014 ) , pp. 3014–3022. Curran Associates, Inc.
Marbach, P., Tsitsiklis, J. N. (1998). Simulation-based optimization of Markov reward processes.
MIT Technical Report LIDS-P-2411.
Marbach, P., Tsitsiklis, J. N. (2001). Simulation-based optimization of Markov reward processes.
IEEE Transactions on Automatic Control, 46 (2):191–209.
Markram, H., L¨ ubke, J., Frotscher, M., Sakmann, B. (1997). Regulation of synaptic ecacy by
coincidence of postsynaptic APs and EPSPs. Science, 275 (5297):213–215.

Mart´ ınez, J. F., ˙Ipek, E. (2009). Dynamic multicore resource management: A machine learning
approach. Micro, IEEE, 29 (5):8–17.
Mataric, M. J. (1994). Reward functions for accelerated learning. In Proceedings of the
11th International Conference on Machine Learning (ICML 1994) , pp. 181–189. Morgan
Kaufmann.
Matsuda, W., Furuta, T., Nakamura, K. C., Hioki, H., Fujiyama, F., Arai, R., Kaneko, T.
(2009). Single nigrostriatal dopaminergic neurons form widely spread and highly dense
axonal arborizations in the neostriatum. The Journal of Neuroscience, 29 (2):444–453.

Kaufmann.
Mazur, J. E. (1994). Learning and Behavior , 3rd ed. Prentice-Hall, Englewood Cli↵s, NJ.
McCallum, A. K. (1993). Overcoming incomplete perception with utile distinction memory.
pp. 190–196. Morgan Kaufmann.
McCallum, A. K. (1995). Reinforcement Learning with Selective Perception and Hidden State .
Ph.D. thesis, University of Rochester, Rochester NY.
McCloskey, M., Cohen, N. J. (1989). Catastrophic interference in connectionist networks: The

sequential learning problem. Psychology of Learning and Motivation, 24 :109–165.
McClure, S. M., Daw, N. D., Montague, P. R. (2003). A computational substrate for incentive
salience. Trends in Neurosciences, 26 (8):423–428.
McCulloch, W. S., Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity.
Bulletin of Mathematical Biophysics, 5 (4):115–133.
McMahan, H. B., Gordon, G. J. (2005). Fast Exact Planning in Markov Decision Processes.
InProceedings of the International Conference on Automated Planning and Scheduling ,
pp. 151-160.

Melo, F. S., Meyn, S. P., Ribeiro, M. I. (2008). An analysis of reinforcement learning with
function approximation. In Proceedings of the 25th International Conference on Machine
Learning (ICML 2008) ,p p .6 6 4 – 6 7 1 .
Mendel, J. M. (1966). A survey of learning control systems. ISA Transactions, 5 :297–303.
Mendel, J. M., McLaren, R. W. (1970). Reinforcement learning control and pattern recognition

References 501
systems. In J. M. Mendel and K. S. Fu (Eds.), Adaptive, Learning and Pattern Recognition
Systems: Theory and Applications , pp. 287–318. Academic Press, New York.
Michie, D. (1961). Trial and error. In S. A. Barnett and A. McLaren (Eds.), Science Survey,
Part 2 , pp. 129–145. Penguin, Harmondsworth.
Michie, D. (1963). Experiments on the mechanisation of game learning. 1. characterization of
the model and its parameters. The Computer Journal, 6 (3):232–263.
Michie, D. (1974). On Machine Intelligence . Edinburgh University Press, Edinburgh.
Michie, D., Chambers, R. A. (1968). BOXES, An experiment in adaptive control. In E. Dale
and D. Michie (Eds.), Machine Intelligence 2 , pp. 137–152. Oliver and Boyd, Edinburgh.
Miller, R. (1981). Meaning and Purpose in the Intact Brain: A Philosophical, Psychological,
and Biological Account of Conscious Process . Clarendon Press, Oxford.
Miller, W. T., An, E., Glanz, F., Carter, M. (1990). The design of CMAC neural networks for

control. Adaptive and Learning Systems, 1 :140–145.
Miller, W. T., Glanz, F. H. (1996). UNH CMAC verison 2.1: The University of New Hampshire
Implementation of the Cerebellar Model Arithmetic Computer - CMAC . Robotics Laboratory
Technical Report, University of New Hampshire, Durham.
Miller, S., Williams, R. J. (1992). Learning to control a bioreactor using a neural net Dyna-Q
system. In Proceedings of the Seventh Yale Workshop on Adaptive and Learning Systems ,
pp. 167–172. Center for Systems Science, Dunham Laboratory, Yale University, New Haven.

Miller, W. T., Scalera, S. M., Kim, A. (1994). Neural network control of dynamic balance for a
biped walking robot. In Proceedings of the Eighth Yale Workshop on Adaptive and Learning
Systems , pp. 156–161. Center for Systems Science, Dunham Laboratory, Yale University,
New Haven.
Minton, S. (1990). Quantitative results concerning the utility of explanation-based learning.
Artiﬁcial Intelligence, 42 (2-3):363–391.
Minsky, M. L. (1954). Theory of Neural-Analog Reinforcement Systems and Its Application to

New Haven.
the Brain-Model Problem . Ph.D. thesis, Princeton University.
Minsky, M. L. (1961). Steps toward artiﬁcial intelligence. Proceedings of the Institute of Radio
Engineers , 49:8–30. Reprinted in E. A. Feigenbaum and J. Feldman (Eds.), Computers and
Thought , pp. 406–450. McGraw-Hill, New York, 1963.
Minsky, M. L. (1967). Computation: Finite and Inﬁnite Machines . Prentice-Hall, Englewood
Cli↵s, NJ.
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.

Cli↵s, NJ.
(2013). Playing atari with deep reinforcement learning. ArXiv:1312.5602.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves,
A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A.,
Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., Hassabis, D. (2015). Human-
level control through deep reinforcement learning. Nature, 518 (7540):529–533.
Modayil, J., Sutton, R. S. (2014). Prediction driven behavior: Learning predictions that drive

ﬁxed responses. In AAAI-14 Workshop on Artiﬁcial Intelligence and Robotics ,Q u e b e cC i t y ,
Canada.
Modayil, J., White, A., Sutton, R. S. (2014). Multi-timescale nexting in a reinforcement learning
robot. Adaptive Behavior, 22 (2):146–160.
Monahan, G. E. (1982). State of the art—a survey of partially observable Markov decision
processes: theory, models, and algorithms. Management Science, 28 (1):1–16.
Montague, P. R., Dayan, P., Nowlan, S. J., Pouget, A., Sejnowski, T. J. (1993). Using aperiodic

Canada.
reinforcement for directed self-organization during development. In Advances in Neural

502 References
Information Processing Systems 5 (NIPS 1992) , pp. 969–976. Morgan Kaufmann.
Montague, P. R., Dayan, P., Person, C., Sejnowski, T. J. (1995). Bee foraging in uncertain
environments using predictive hebbian learning. Nature, 377 (6551):725–728.
Montague, P. R., Dayan, P., Sejnowski, T. J. (1996). A framework for mesencephalic dopamine
systems based on predictive Hebbian learning. The Journal of Neuroscience, 16 (5):1936–1947.
Montague, P. R., Dolan, R. J., Friston, K. J., Dayan, P. (2012). Computational psychiatry.
Trends in Cognitive Sciences, 16 (1):72–80.
Montague, P. R., Sejnowski, T. J. (1994). The predictive brain: Temporal coincidence and
temporal order in synaptic learningmechanisms. Learning & Memory, 1 (1):1–33.
Moore, A. W. (1990). Ecient Memory-Based Learning for Robot Control . Ph.D. thesis,
University of Cambridge.
Moore, A. W., Atkeson, C. G. (1993). Prioritized sweeping: Reinforcement learning with less
data and less real time. Machine Learning, 13 (1):103–130.

Moore, A. W., Schneider, J., Deng, K. (1997). Ecient locally weighted polynomial regression
predictions. In Proceedings of the 14th International Conference on Machine Learning (ICML
1997) . Morgan Kaufmann.
Moore, J. W., Blazis, D. E. J. (1989). Simulation of a classically conditioned response: A
cerebellar implementation of the sutton-barto-desmond model. In J. H. Byrne and W. O.
Berry (Eds.), Neural Models of Plasticity , pp. 187–207. Academic Press, San Diego, CA.

Moore, J. W., Choi, J.-S., Brunzell, D. H. (1998). Predictive timing under temporal uncertainty:
The time derivative model of the conditioned response. In D. A. Rosenbaum and C. E.
Collyer (Eds.), Timing of Behavior , pp. 3–34. MIT Press, Cambridge, MA.
Moore, J. W., Desmond, J. E., Berthier, N. E., Blazis, E. J., Sutton, R. S., Barto, A. G. (1986).
Simulation of the classically conditioned nictitating membrane response by a neuron-like
adaptive element: I. Response topography, neuronal ﬁring, and interstimulus intervals.

Behavioural Brain Research, 21 (2):143–154.
Moore, J. W., Marks, J. S., Castagna, V. E., Polewan, R. J. (2001). Parameter stability in the
TD model of complex CR topographies. In Society for Neuroscience Abstracts, 27 :642.
Moore, J. W., Schmajuk, N. A. (2008). Kamin blocking. Scholarpedia, 3 (5):3542.
Moore, J. W., Stickney, K. J. (1980). Formation of attentional-associative networks in real
time:Role of the hippocampus and implications for conditioning. Physiological Psychology,
8(2):207–217.
Mukundan, J., Mart´ ınez, J. F. (2012). MORSE, Multi-objective reconﬁgurable self-optimizing

memory scheduler. In IEEE 18th International Symposium on High Performance Computer
Architecture (HPCA) ,p p .1 – 1 2 .
M¨ uller, M. (2002). Computer Go. Artiﬁcial Intelligence, 134 (1):145–179.
Munos, R., Stepleton, T., Harutyunyan, A., Bellemare, M. (2016). Safe and ecient o↵-policy
reinforcement learning. In Advances in Neural Information Processing Systems 29 (NIPS
2016) , pp. 1046–1054. Curran Associates, Inc.
Naddaf, Y. (2010). Game-Independent AI Agents for Playing Atari 2600 Console Games . Ph.D.

Narendra, K. S., Thathachar, M. A. L. (1974). Learning automata—A survey. IEEE Transactions
on Systems, Man, and Cybernetics, 4 :323–334.
Narendra, K. S., Thathachar, M. A. L. (1989). Learning Automata: An Introduction . Prentice-
Narendra, K. S., Wheeler, R. M. (1983). An N-player sequential stochastic game with identical
payo↵s. IEEE Transactions on Systems, Man, and Cybernetics, 6 :1154–1158.

References 503
Narendra, K. S., Wheeler, R. M. (1986). Decentralized learning in ﬁnite Markov chains. IEEE
Transactions on Automatic Control, 31 (6):519–526.
Nedi´ c, A., Bertsekas, D. P. (2003). Least squares policy evaluation algorithms with linear
function approximation. Discrete Event Dynamic Systems, 13 (1-2):79–110.
Ng, A. Y. (2003). Shaping and Policy Search in Reinforcement Learning . Ph.D. thesis, University
of California, Berkeley.
Ng, A. Y., Harada, D., Russell, S. (1999). Policy invariance under reward transformations:
Theory and application to reward shaping. In I. Bratko and S. Dzeroski (Eds.), Proceedings
of the 16th International Conference on Machine Learning (ICML 1999) ,p p .2 7 8 – 2 8 7 .
Ng, A. Y., Russell, S. J. (2000). Algorithms for inverse reinforcement learning. In Proceedings of
the 17th International Conference on Machine Learning (ICML 2000) ,p p .6 6 3 – 6 7 0 .
Niv, Y. (2009). Reinforcement learning in the brain. Journal of Mathematical Psychology,
53(3):139–154.

Niv, Y., Daw, N. D., Dayan, P. (2006). How fast to work: Response vigor, motivation and
tonic dopamine. In Advances in Neural Information Processing Systems 18 (NIPS 2005) ,
pp. 1019–1026. MIT Press, Cambridge, MA.
Niv, Y., Daw, N. D., Joel, D., Dayan, P. (2007). Tonic dopamine: opportunity costs and the
control of response vigor. Psychopharmacology, 191 (3):507–520.
Niv, Y., Joel, D., Dayan, P. (2006). A normative perspective on motivation. Trends in Cognitive
Sciences, 10 (8):375–381.

Nouri, A., Littman, M. L. (2009). Multi-resolution exploration in continuous spaces. In Advances
in Neural Information Processing Systems 21 (NIPS 2008) , pp. 1209–1216. Curran Associates,
Inc.
Now´ e, A., Vrancx, P., Hauwere, Y.-M. D. (2012). Game theory and multi-agent reinforcement
learning. In M. Wiering and M. van Otterlo (Eds.), Reinforcement Learning: State-of-the-Art ,
pp. 441–467. Springer-Verlag Berlin Heidelberg.

Inc.
Nutt, D. J., Lingford-Hughes, A., Erritzoe, D., Stokes, P. R. A. (2015). The dopamine theory of
addiction: 40 years of highs and lows. Nature Reviews Neuroscience, 16 (5):305–312.
O’Doherty, J. P., Dayan, P., Friston, K., Critchley, H., Dolan, R. J. (2003). Temporal di↵erence
models and reward-related learning in the human brain. Neuron, 38 (2):329–337.
O’Doherty, J. P., Dayan, P., Schultz, J., Deichmann, R., Friston, K., Dolan, R. J. (2004).
Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science,
304(5669):452–454.

´Olafsd´ ottir, H. F., Barry, C., Saleem, A. B., Hassabis, D., Spiers, H. J. (2015). Hippocampal
place cells construct reward related sequences through unexplored space. Elife, 4 :e06063.
Oh, J., Guo, X., Lee, H., Lewis, R. L., Singh, S. (2015). Action-conditional video prediction
using deep networks in Atari games. In Advances in Neural Information Processing Systems
28 (NIPS 2015) , pp. 2845–2853. Curran Associates, Inc.
Olds, J., Milner, P. (1954). Positive reinforcement produced by electrical stimulation of the septal

area and other regions of rat brain. Journal of Comparative and Physiological Psychology,
47(6):419–427.
O’Reilly, R. C., Frank, M. J. (2006). Making working memory work: A computational model of
learning in the prefrontal cortex and basal ganglia. Neural Computation, 18 (2):283–328.
O’Reilly, R. C., Frank, M. J., Hazy, T. E., Watz, B. (2007). PVLV, the primary value and
learned value Pavlovian learning algorithm. Behavioral Neuroscience, 121 (1):31–49.
Omohundro, S. M. (1987). Ecient algorithms with neural network behavior. Technical Report,

Department of Computer Science, University of Illinois at Urbana-Champaign.

504 References
Ormoneit, D., Sen, ´S. (2002). Kernel-based reinforcement learning. Machine Learning, 49 (2-
3):161–178.
Oudeyer, P.-Y., Kaplan, F. (2007). What is intrinsic motivation? A typology of computational
approaches. Frontiers in Neurorobotics, 1 :6.
Oudeyer, P.-Y., Kaplan, F., Hafner, V. V. (2007). Intrinsic motivation systems for autonomous
mental development. IEEE Transactions on Evolutionary Computation, 11 (2):265–286.
Padoa-Schioppa, C., Assad, J. A. (2006). Neurons in the orbitofrontal cortex encode economic
value. Nature, 441 (7090):223–226.
Page, C. V. (1977). Heuristics for signature table analysis as a pattern recognition technique.
IEEE Transactions on Systems, Man, and Cybernetics, 7 (2):77–86.
Pagnoni, G., Zink, C. F., Montague, P. R., Berns, G. S. (2002). Activity in human ventral
striatum locked to errors of reward prediction. Nature Neuroscience, 5 (2):97–98.
Pan, W.-X., Schmidt, R., Wickens, J. R., Hyland, B. I. (2005). Dopamine cells respond to

predicted events during classical conditioning: Evidence for eligibility traces in the reward-
learning network. The Journal of Neuroscience, 25 (26):6235–6242.
Park, J., Kim, J., Kang, D. (2005). An RLS-based natural actor–critic algorithm for locomotion
of a two-linked robot arm. Computational Intelligence and Security :65–72.
Parks, P. C., Militzer, J. (1991). Improved allocation of weights for associative memory storage
in learning control systems. In IFAC Design Methods of Control Systems , Zurich, Switzerland,
pp. 507–512.

Parr, R. (1988). Hierarchical Control and Learning for Markov Decision Processes. Ph.D. thesis,
University of California, Berkeley.
Parr, R., Li, L., Taylor, G., Painter-Wakeﬁeld, C., Littman, M. L. (2008). An analysis of linear
models, linear value-function approximation, and feature selection for reinforcement learning.
InProceedings of the 25th international conference on Machine learning ,p p .7 5 2 – 7 5 9 ) .
Parr, R., Russell, S. (1995). Approximating optimal policies for partially observable stochastic

domains. In Proceedings of the Fourteenth International Joint Conference on Artiﬁcial
Intelligence , pp. 1088–1094. Morgan Kaufmann.
Pavlov, P. I. (1927). Conditioned Reﬂexes . Oxford University Press, London.
Pawlak, V., Kerr, J. N. D. (2008). Dopamine receptor activation is required for corticostriatal
spike-timing-dependent plasticity. The Journal of Neuroscience, 28 (10):2435–2446.
Pawlak, V., Wickens, J. R., Kirkwood, A., Kerr, J. N. D. (2010). Timing is not every-
thing: neuromodulation opens the STDP gate. Frontiers in Synaptic Neuroscience, 2 :146.
doi:10.3389/fnsyn.2010.00146.

Pearce, J. M., Hall, G. (1980). A model for Pavlovian learning: Variation in the e↵ectiveness of
conditioning but not unconditioned stimuli. Psychological Review, 87 (6):532–552.
Pearl, J. (1984). Heuristics: Intelligent Search Strategies for Computer Problem Solving .
Pearl, J. (1995). Causal diagrams for empirical research. Biometrika ,8 2 ( 4 ) : 6 6 9 - 6 8 8 .
Pecevski, D., Maass, W., Legenstein, R. A. (2008). Theoretical analysis of learning with

reward-modulated spike-timing-dependent plasticity. In Advances in Neural Information
Processing Systems 20 (NIPS 2007) , pp. 881–888. Curran Associates, Inc.
Peng, J. (1993). Ecient Dynamic Programming-Based Learning for Control . Ph.D. thesis,
Northeastern University, Boston MA.
Peng, J. (1995). Ecient memory-based dynamic programming. In Proceedings of the 12th
International Conference on Machine Learning (ICML 1995) ,p p .4 3 8 – 4 4 6 .

References 505
Adaptive Behavior, 1 (4):437–454.
Peng, J., Williams, R. J. (1994). Incremental multi-step Q-learning. In Proceedings of the
11th International Conference on Machine Learning (ICML 1994) , pp. 226–232. Morgan
Peng, J., Williams, R. J. (1996). Incremental multi-step Q-learning. Machine Learning,
22(1):283–290.
Perkins, T. J., Pendrith, M. D. (2002). On the existence of ﬁxed points for Q-learning and
Sarsa in partially observable domains. In Proceedings of the 19th International Conference
on Machine Learning (ICML 2002) ,p p .4 9 0 – 4 9 7 .
Perkins, T. J., Precup, D. (2003). A convergent form of approximate policy iteration. In
Advances in Neural Information Processing Systems 15 (NIPS 2002) ,p p .1 6 2 7 – 1 6 3 4 .M I T
Peters, J., B¨ uchel, C. (2010). Neural representations of subjective reward value. Behavioral
Brain Research, 213 (2):135–141.
Peters, J., Schaal, S. (2008). Natural actor–critic. Neurocomputing, 71 (7):1180–1190.

Peters, J., Vijayakumar, S., Schaal, S. (2005). Natural actor–critic. In European Conference on
Machine Learning , pp. 280–291. Springer Berlin Heidelberg.
Pezzulo, G., van der Meer, M. A. A., Lansink, C. S., Pennartz, C. M. A. (2014). Internally
generated sequences in learning and executing goal-directed behavior. Trends in Cognitive
Science, 18 (12):647–657.
Pfei↵er, B. E., Foster, D. J. (2013). Hippocampal place-cell sequences depict future paths to
remembered goals. Nature, 497 (7447):74–79.

Phansalkar, V. V., Thathachar, M. A. L. (1995). Local and global optimization algorithms for
generalized learning automata. Neural Computation, 7 (5):950–973.
Poggio, T., Girosi, F. (1989). A theory of networks for approximation and learning. A.I. Memo
1140. Artiﬁcial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge,
MA.
Poggio, T., Girosi, F. (1990). Regularization algorithms for learning that are equivalent to
multilayer networks. Science, 247 (4945):978–982.

MA.
Polyak, B. T. (1990). New stochastic approximation type procedures. Automat. i Telemekh,
7(98-107):2 (in Russian).
Polyak, B. T., Juditsky, A. B. (1992). Acceleration of stochastic approximation by averaging.
SIAM Journal on Control and Optimization, 30 (4):838–855.
Powell, M. J. D. (1987). Radial basis functions for multivariate interpolation: A review. In
J. C. Mason and M. G. Cox (Eds.), Algorithms for Approximation ,p p .1 4 3 – 1 6 7 .C l a r e n d o n
Press, Oxford.

Powell, W. B. (2011). Approximate Dynamic Programming: Solving the Curses of Dimensionality ,
Second edition. John Wiley and Sons.
Powers, W. T. (1973). Behavior: The Control of Perception . Aldine de Gruyter, Chicago. 2nd
expanded edition 2005.
Precup, D. (2000). Temporal Abstraction in Reinforcement Learning. Ph.D. thesis, University of
Precup, D., Sutton, R. S., Dasgupta, S. (2001). O↵-policy temporal-di↵erence learning with
function approximation. In Proceedings of the 18th International Conference on Machine

Learning (ICML 2001) ,p p .4 1 7 – 4 2 4 .
Precup, D., Sutton, R. S., Paduraru, C., Koop, A., Singh, S. (2006). O↵-policy learning with
options and recognizers. In Advances in Neural Information Processing Systems 18 (NIPS

506 References
2005) , pp. 1097–1104. MIT Press, Cambridge, MA.
Precup, D., Sutton, R. S., Singh, S. (2000). Eligibility traces for o↵-policy policy evaluation.
InProceedings of the 17th International Conference on Machine Learning (ICML 2000) ,
pp. 759–766. Morgan Kaufmann.
Puterman, M. L. (1994). Markov Decision Problems . Wiley, New York.
Puterman, M. L., Shin, M. C. (1978). Modiﬁed policy iteration algorithms for discounted
Markov decision problems. Management Science, 24 (11):1127–1137.
Quartz, S., Dayan, P., Montague, P. R., Sejnowski, T. J. (1992). Expectation learning in the
brain using di↵use ascending connections. In Society for Neuroscience Abstracts, 18 :1210.
Randløv, J., Alstrøm, P. (1998). Learning to drive a bicycle using reinforcement learning and
shaping. In Proceedings of the 15th International Conference on Machine Learning (ICML
1998) ,p p .4 6 3 – 4 7 1 .
Rangel, A., Camerer, C., Montague, P. R. (2008). A framework for studying the neurobiology of

value-based decision making. Nature Reviews Neuroscience, 9 (7):545–556.
Rangel, A., Hare, T. (2010). Neural computations associated with goal-directed choice. Current
Opinion in Neurobiology, 20 (2):262–270.
Rao, R. P., Sejnowski, T. J. (2001). Spike-timing-dependent Hebbian plasticity as temporal
di↵erence learning. Neural Computation, 13 (10):2221–2237.
Ratcli↵, R. (1990). Connectionist models of recognition memory: Constraints imposed by
learning and forgetting functions. Psychological Review, 97 (2):285–308.

Reddy, G., Celani, A., Sejnowski, T. J., Vergassola, M. (2016). Learning to soar in turbulent
environments. Proceedings of the National Academy of Sciences, 113 (33):E4877–E4884.
Redish, D. A. (2004). Addiction as a computational process gone awry. Science, 306 (5703):1944–
1947.
Reetz, D. (1977). Approximate solutions of a discounted Markovian decision process. Bonner
Mathematische Schriften, 98 :77–92.
Rescorla, R. A., Wagner, A. R. (1972). A theory of Pavlovian conditioning: Variations in the

1947.
e↵ectiveness of reinforcement and nonreinforcement. In A. H. Black and W. F. Prokasy
(Eds.), Classical Conditioning II , pp. 64–99. Appleton-Century-Crofts, New York.
Revusky, S., Garcia, J. (1970). Learned associations over long delays. In G. Bower (Ed.), The
Psychology of Learning and Motivation , v. 4, pp. 1–84. Academic Press, Inc., New York.
Reynolds, J. N. J., Wickens, J. R. (2002). Dopamine-dependent plasticity of corticostriatal
synapses. Neural Networks, 15 (4):507–521.

Ring, M. B. (in preparation). Representing knowledge as forecasts (and state as knowledge).
Ripley, B. D. (2007). Pattern Recognition and Neural Networks . Cambridge University Press.
Rixner, S. (2004). Memory controller optimizations for web servers. In Proceedings of the
37th annual IEEE/ACM International Symposium on Microarchitecture ,p .3 5 5 – 3 6 6 .I E E E
Computer Society.
Robbins, H. (1952). Some aspects of the sequential design of experiments. Bulletin of the
American Mathematical Society, 58 :527–535.

Robertie, B. (1992). Carbon versus silicon: Matching wits with TD-Gammon. Inside Backgam-
mon, 2 (2):14–22.
Romo, R., Schultz, W. (1990). Dopamine neurons of the monkey midbrain: Contingencies of
responses to active touch during self-initiated arm movements. Journal of Neurophysiology,
63(3):592–624.
Rosenblatt, F. (1962). Principles of Neurodynamics: Perceptrons and the Theory of Brain
Mechanisms . Spartan Books, Washington, DC.

References 507
Ross, S. (1983). Introduction to Stochastic Dynamic Programming . Academic Press, New York.
Ross, T. (1933). Machines that think. Scientiﬁc American, 148 (4):206–208.
Rubinstein, R. Y. (1981). Simulation and the Monte Carlo Method . Wiley, New York.
Rumelhart, D. E., Hinton, G. E., Williams, R. J. (1986). Learning internal representations by
error propagation. In D. E. Rumelhart and J. L. McClelland (Eds.), Parallel Distributed Pro-
cessing: Explorations in the Microstructure of Cognition ,v o l .I , Foundations . Bradford/MIT
Rummery, G. A. (1995). Problem Solving with Reinforcement Learning . Ph.D. thesis, University
of Cambridge.
Rummery, G. A., Niranjan, M. (1994). On-line Q-learning using connectionist systems. Technical
Report CUED/F-INFENG/TR 166. Engineering Department, Cambridge University.
Ruppert, D. (1988). Ecient estimations from a slowly convergent Robbins-Monro process.

Cornell University Operations Research and Industrial Engineering Technical Report No. 781.
Russell, S., Norvig, P. (2009). Artiﬁcial Intelligence: A Modern Approach , 3rd edition. Prentice-
Russo, D. J., Van Roy, B., Kazerouni, A., Osband, I., Wen, Z. (2018). A tutorial on Thompson
sampling, Foundations and Trends in Machine Learning . ArXiv:1707.02038.
Rust, J. (1996). Numerical dynamic programming in economics. In H. Amman, D. Kendrick, and

J. Rust (Eds.), Handbook of Computational Economics , pp. 614–722. Elsevier, Amsterdam.
Saddoris, M. P., Cacciapaglia, F., Wightmman, R. M., Carelli, R. M. (2015). Di↵erential
dopamine release dynamics in the nucleus accumbens core and shell reveal complemen-
tary signals for error prediction and incentive motivation. The Journal of Neuroscience,
35(33):11572–11582.
Saksida, L. M., Raymond, S. M., Touretzky, D. S. (1997). Shaping robot behavior using principles
from instrumental conditioning. Robotics and Autonomous Systems, 22 (3):231–249.

Samuel, A. L. (1959). Some studies in machine learning using the game of checkers. IBM
Journal on Research and Development, 3 (3), 210–229.
Samuel, A. L. (1967). Some studies in machine learning using the game of checkers. II—Recent
progress. IBM Journal on Research and Development, 11 (6):601–617.
Schaal, S., Atkeson, C. G. (1994). Robot juggling: Implementation of memory-based learning.
IEEE Control Systems, 14 (1):57–71.
Schmajuk, N. A. (2008). Computational models of classical conditioning. Scholarpedia,
3(3):1664.

Schmidhuber, J. (1991a). Curious model-building control systems. In Proceedings of the IEEE
International Joint Conference on Neural Networks ,p p .1 4 5 8 – 1 4 6 3 .I E E E .
Schmidhuber, J. (1991b). A possibility for implementing curiosity and boredom in model-building
neural controllers. In From Animals to Animats: Proceedings of the First International
Conference on Simulation of Adaptive Behavior , pp. 222–227. MIT Press, Cambridge, MA.
Schmidhuber, J. (2015). Deep learning in neural networks: An overview. Neural Networks,
6:85–117.

6:85–117.
Schmidhuber, J., Storck, J., Hochreiter, S. (1994). Reinforcement driven information acquisition
in nondeterministic environments. Technical report, Fakult¨ at f¨ ur Informatik, Technische
Universit¨ at M¨ unchen, M¨ unchen, Germany.
Schraudolph, N. N. (1999). Local gain adaptation in stochastic gradient descent. In Proceedings
of the International Conference on Artiﬁcial Neural Networks ,p p .5 6 9 – 5 7 4 .I E E E ,L o n d o n .
Schraudolph, N. N. (2002). Fast curvature matrix-vector products for second-order gradient

descent. Neural Computation, 14 (7):1723–1738.

508 References
Schraudolph, N. N., Yu, J., Aberdeen, D. (2006). Fast online policy gradient learning with SMD
gain vector adaptation. In Advances in Neural Information Processing Systems ,p p .1 1 8 5 –
1192.
Schulman, J., Chen, X., Abbeel, P. (2017). Equivalence between policy gradients and soft
Q-Learning. ArXiv:1704.06440.
Schultz, D. G., Melsa, J. L. (1967). State Functions and Linear Control Systems . McGraw-Hill,
New York.
Schultz, W. (1998). Predictive reward signal of dopamine neurons. Journal of Neurophysiology,
80(1):1–27.
Schultz, W., Apicella, P., Ljungberg, T. (1993). Responses of monkey dopamine neurons to
reward and conditioned stimuli during successive steps of learning a delayed response task.
The Journal of Neuroscience, 13 (3):900–913.
Schultz, W., Dayan, P., Montague, P. R. (1997). A neural substrate of prediction and reward.
Science, 275 (5306):1593–1598.
Schultz, W., Romo, R. (1990). Dopamine neurons of the monkey midbrain: contingencies of

responses to stimuli eliciting immediate behavioral reactions. Journal of Neurophysiology,
63(3):607–624.
Schultz, W., Romo, R., Ljungberg, T., Mirenowicz, J., Hollerman, J. R., Dickinson, A. (1995).
Reward-related signals carried by dopamine neurons. In J. C. Houk, J. L. Davis, and D. G.
Beiser (Eds.), Models of Information Processing in the Basal Ganglia ,p p .2 3 3 – 2 4 8 . M I T
Schwartz, A. (1993). A reinforcement learning method for maximizing undiscounted rewards.

pp. 298–305. Morgan Kaufmann.
Schweitzer, P. J., Seidmann, A. (1985). Generalized polynomial approximations in Markovian
decision processes. Journal of Mathematical Analysis and Applications, 110 (2):568–582.
Selfridge, O. G. (1978). Tracking and trailing: Adaptation in movement strategies. Technical
report, Bolt Beranek and Newman, Inc. Unpublished report.

Selfridge, O. G. (1984). Some themes and primitives in ill-deﬁned systems. In O. G. Selfridge,
E. L. Rissland, and M. A. Arbib (Eds.), Adaptive Control of Ill-Deﬁned Systems ,p p .2 1 – 2 6 .
Plenum Press, NY. Proceedings of the NATO Advanced Research Institute on Adaptive
Control of Ill-deﬁned Systems, NATO Conference Series II, Systems Science, Vol. 16.
Selfridge, O. J., Sutton, R. S., Barto, A. G. (1985). Training and tracking in robotics. In A. Joshi
(Ed.), Proceedings of the Ninth International Joint Conference on Artiﬁcial Intelligence ,

pp. 670–672. Morgan Kaufmann.
Seo, H., Barraclough, D., Lee, D. (2007). Dynamic signals related to choices and outcomes in
the dorsolateral prefrontal cortex. Cerebral Cortex, 17 (suppl 1):110–117.
Seung, H. S. (2003). Learning in spiking neural networks by reinforcement of stochastic synaptic
transmission. Neuron, 40 (6):1063–1073.
Shah, A. (2012). Psychological and neuroscientiﬁc connections with reinforcement learning. In M.
Wiering and M. van Otterlo (Eds.), Reinforcement Learning: State-of-the-Art ,p p .5 0 7 – 5 3 7 .
Springer-Verlag Berlin Heidelberg.

Shannon, C. E. (1950). Programming a computer for playing chess. Philosophical Magazine and
Journal of Science, 41 (314):256–275.
Shannon, C. E. (1951). Presentation of a maze-solving machine. In H. V. Forester (Ed.), Cyber-
netics. Transactions of the Eighth Conference , pp. 173–180. Josiah Macy Jr. Foundation.
Shannon, C. E. (1952). “Theseus” maze-solving mouse. http://cyberneticzoo.com/mazesolvers/1952-

References 509
-theseus-maze-solving-mouse--claude-shannon-american/.
Shelton, C. R. (2001). Importance Sampling for Reinforcement Learning with Multiple Objectives .
Ph.D. thesis, Massachusetts Institute of Technology, Cambridge MA.
Shepard, D. (1968). A two-dimensional interpolation function for irregularly-spaced data. In
Proceedings of the 23rd ACM National Conference , pp. 517–524. ACM, New York.
Sherman, J., Morrison, W. J. (1949). Adjustment of an inverse matrix corresponding to changes
in the elements of a given column or a given row of the original matrix (abstract). Annals of
Mathematical Statistics, 20 (4):621.
Shewchuk, J., Dean, T. (1990). Towards learning time-varying functions with high input
dimensionality. In Proceedings of the Fifth IEEE International Symposium on Intelligent
Control , pp. 383–388. IEEE Computer Society Press, Los Alamitos, CA.
Shimansky, Y. P. (2009). Biologically plausible learning in neural networks: a lesson from

bacterial chemotaxis. Biological Cybernetics, 101 (5-6):379–385.
Si, J., Barto, A., Powell, W., Wunsch, D. (Eds.) (2004). Handbook of Learning and Approximate
Dynamic Programming .J o h nW i l e ya n dS o n s .
Silver, D. (2009). Reinforcement Learning and Simulation Based Search in the Game of Go .
Ph.D. thesis, University of Alberta, Edmonton.
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser,
J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J.,

Kalchbrenner, N., Sutskever, I., Lillicrap, T., Leach, M., Kavukcuoglu, K., Graepel, T.,
Hassabis, D. (2016). Mastering the game of Go with deep neural networks and tree search.
Nature, 529 (7587):484–489.
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., Riedmiller, M. (2014). Deterministic
policy gradient algorithms. In Proceedings of the 31st International Conference on Machine
Learning (ICML 2014) ,p p .3 8 7 – 3 9 5 .
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T.,

Baker, L., Lai, M., Bolton, A., Chen, Y., Lillicrap, L., Hui, F., Sifre, L., van den Driessche,
G., Graepel, T., Hassibis, D. (2017a). Mastering the game of Go without human knowledge.
Nature, 550 (7676):354–359.
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L.,
Kumaran, D., Graepel, T., Lillicrap, T., Simoyan, K., Hassibis, D. (2017b). Mastering chess
and shogi by self-play with a general reinforcement learning algorithm. ArXiv:1712.01815.

S¸im¸ sek, ¨O., Alg´ orta, S., Kothiyal, A. (2016). Why most decisions are easy in tetris—And perhaps
in other sequential decision problems, as well. In Proceedings of the 33rd International
Conference on Machine Learning (ICML 2016) ,p p .1 7 5 7 - 1 7 6 5 .
Simon, H. (2000). Lecture at the Earthware Symposium, Carnegie Mellon University. https://www.youtube.com/watch?v=EZhyi-
8DBjc.
Singh, S. P. (1992a). Reinforcement learning with a hierarchy of abstract models. In Proceed-

8DBjc.
ings of the Tenth National Conference on Artiﬁcial Intelligence (AAAI-92) ,p p .2 0 2 – 2 0 7 .
AAAI/MIT Press, Menlo Park, CA.
Singh, S. P. (1992b). Scaling reinforcement learning algorithms by learning variable temporal
resolution models. In Proceedings of the 9th International Workshop on Machine Learning ,
pp. 406–415. Morgan Kaufmann.
Singh, S. P. (1993). Learning to Solve Markovian Decision Processes . Ph.D. thesis, University
of Massachusetts, Amherst.

Singh, S. P. (Ed.) (2002). Special double issue on reinforcement learning, Machine Learning,
49(2-3).

510 References
Singh, S., Barto, A. G., Chentanez, N. (2005). Intrinsically motivated reinforcement learning.
InAdvances in Neural Information Processing Systems 17 (NIPS 2004) ,p p .1 2 8 1 – 1 2 8 8 .M I T
Singh, S. P., Bertsekas, D. (1997). Reinforcement learning for dynamic channel allocation in
cellular telephone systems. In Advances in Neural Information Processing Systems 9 (NIPS
1996) , pp. 974–980. MIT Press, Cambridge, MA.
Singh, S. P., Jaakkola, T., Jordan, M. I. (1994). Learning without state-estimation in partially
observable Markovian decision problems. In Proceedings of the 11th International Conference
on Machine Learning (ICML 1994) , pp. 284–292. Morgan Kaufmann.
Singh, S., Jaakkola, T., Littman, M. L., Szepesvri, C. (2000). Convergence results for single-step
on-policy reinforcement-learning algorithms. Machine Learning, 38 (3):287–308.
Singh, S. P., Jaakkola, T., Jordan, M. I. (1995). Reinforcement learning with soft state

aggregation. In Advances in Neural Information Processing Systems 7 (NIPS 1994) ,p p .3 5 9 –
368. MIT Press, Cambridge, MA.
Singh, S., Lewis, R. L., Barto, A. G. (2009). Where do rewards come from? In N. Taatgen
and H. van Rijn (Eds.), Proceedings of the 31st Annual Conference of the Cognitive Science
Society ,p p .2 6 0 1 – 2 6 0 6 .C o g n i t i v eS c i e n c eS o c i e t y .
Singh, S., Lewis, R. L., Barto, A. G., Sorg, J. (2010). Intrinsically motivated reinforcement
learning: An evolutionary perspective. IEEE Transactions on Autonomous Mental Develop-

ment, 2 (2):70–82. Special issue on Active Learning and Intrinsically Motivated Exploration
in Robots: Advances and Challenges.
Singh, S. P., Sutton, R. S. (1996). Reinforcement learning with replacing eligibility traces.
Machine Learning, 22 (1-3):123–158.
Skinner, B. F. (1938). The Behavior of Organisms: An Experimental Analysis . Appleton-Century,
New York.
Skinner, B. F. (1958). Reinforcement today. American Psychologist, 13 (3):94–99.
Skinner, B. F. (1963). Operant behavior. American Psychologist, 18 (8):503–515.

New York.
Sofge, D. A., White, D. A. (1992). Applied learning: Optimal control for manufacturing. In
D. A. White and D. A. Sofge (Eds.), Handbook of Intelligent Control: Neural, Fuzzy, and
Adaptive Approaches , pp. 259–281. Van Nostrand Reinhold, New York.
Sorg, J. D. (2011). The Optimal Reward Problem:Designing E↵ective Reward for Bounded
Agents . Ph.D. thesis, University of Michigan, Ann Arbor.
Sorg, J., Lewis, R. L., Singh, S. P. (2010). Reward design via online gradient ascent. In Advances
in Neural Information Processing Systems 23 (NIPS 2010) , pp. 2190–2198. Curran Associates,
Inc.

Inc.
Sorg, J., Singh, S. (2010). Linear options. In Proceedings of the 9th International Conference on
Autonomous Agents and Multiagent Systems ,p p .3 1 – 3 8 .
Sorg, J., Singh, S., Lewis, R. (2010). Internal rewards mitigate agent boundedness. In Proceedings
of the 27th International Conference on Machine Learning (ICML 2010) ,p p .1 0 0 7 – 1 0 1 4 .
Spence, K. W. (1947). The role of secondary reinforcement in delayed reward learning. Psycho-
logical Review, 54 (1):1–8.

Inc.
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R. (2014). Dropout:
A simple way to prevent neural networks from overﬁtting. Journal of Machine Learning
Research, 15 (1):1929–1958.
Staddon, J. E. R. (1983). Adaptive Behavior and Learning . Cambridge University Press.
Stanﬁll, C., Waltz, D. (1986). Toward memory-based reasoning. Communications of the ACM,
29(12):1213–1228.

References 511
Steinberg, E. E., Keiﬂin, R., Boivin, J. R., Witten, I. B., Deisseroth, K., Janak, P. H. (2013). A
causal link between prediction errors, dopamine neurons and learning. Nature Neuroscience,
16(7):966–973.
Sterling, P., Laughlin, S. (2015). Principles of Neural Design . MIT Press, Cambridge, MA.
Sternberg, S. (1963). Stochastic learning theory. In: Handbook of Mathematical Psychology,
Volume II, R. D. Luce, R. R. Bush, and E. Galanter (Eds.). John Wiley & Sons.
Sugiyama, M., Hachiya, H., Morimura, T. (2013). Statistical Reinforcement Learning: Modern
Machine Learning Approaches .C h a p m a n&H a l l / C R C .
Suri, R. E., Bargas, J., Arbib, M. A. (2001). Modeling functions of striatal dopamine modulation
in learning and planning. Neuroscience, 103 (1):65–85.
Suri, R. E., Schultz, W. (1998). Learning of sequential movements by neural network model
with dopamine-like reinforcement signal. Experimental Brain Research, 121 (3):350–354.

Suri, R. E., Schultz, W. (1999). A neural network model with dopamine-like reinforcement
signal that learns a spatial delayed response task. Neuroscience, 91 (3):871–890.
Sutton, R. S. (1978a). Learning theory support for a single channel theory of the brain.
Unpublished report.
Sutton, R. S. (1978b). Single channel theory: A neuronal theory of learning. Brain Theory
Newsletter, 4 :72–75. Center for Systems Neuroscience, University of Massachusetts, Amherst,
MA.

MA.
Sutton, R. S. (1978c). Au n i ﬁ e dt h e o r yo fe x p e c t a t i o ni nc l a s s i c a la n di n s t r u m e n t a lc o n d i t i o n i n g .
Bachelors thesis, Stanford University.
Sutton, R. S. (1984). Temporal Credit Assignment in Reinforcement Learning . Ph.D. thesis,
University of Massachusetts, Amherst.
Sutton, R. S. (1988). Learning to predict by the method of temporal di↵erences. Machine
Learning, 3 (1):9–44 (important erratum p. 377).
Sutton, R. S. (1990). Integrated architectures for learning, planning, and reacting based on

approximating dynamic programming. In Proceedings of the 7th International Workshop on
Machine Learning , pp. 216–224. Morgan Kaufmann.
Sutton, R. S. (1991a). Dyna, an integrated architecture for learning, planning, and reacting.
SIGART Bulletin, 2 (4):160–163. ACM, New York.
Sutton, R. S. (1991b). Planning by incremental dynamic programming. In Proceedings of the
8th International Workshop on Machine Learning , pp. 353–357. Morgan Kaufmann.
Sutton, R. S. (Ed.) (1992a). Reinforcement Learning . Kluwer Academic Press. Reprinting of a
special double issue on reinforcement learning, Machine Learning, 8 (3-4).

Sutton, R.S. (1992b). Adapting bias by gradient descent: An incremental version of delta-bar-
delta. Proceedings of the Tenth National Conference on Artiﬁcial Intelligence ,p p .1 7 1 – 1 7 6 ,
MIT Press.
Sutton, R.S. (1992c). Gain adaptation beats least squares? Proceedings of the Seventh Yale
Workshop on Adaptive and Learning Systems , pp. 161–166, Yale University, New Haven, CT.
Sutton, R. S. (1995a). TD models: Modeling the world at a mixture of time scales. In Proceedings

MIT Press.
of the 12th International Conference on Machine Learning (ICML 1995) , pp. 531–539. Morgan
Kaufmann.
Sutton, R. S. (1995b). On the virtues of linear learning and trajectory distributions. In
Proceedings of the Workshop on Value Function Approximation atThe 12th International
Conference on Machine Learning (ICML 1995) .
Sutton, R. S. (1996). Generalization in reinforcement learning: Successful examples using
sparse coarse coding. In Advances in Neural Information Processing Systems 8 (NIPS 1995) ,

512 References
pp. 1038–1044. MIT Press, Cambridge, MA.
Sutton, R. S. (2009). The grand challenge of predictive empirical abstract knowledge. Working
Notes of the IJCAI-09 Workshop on Grand Challenges for Reasoning from Experiences .
Sutton, R. S. (2015a) Introduction to reinforcement learning with function approximation.
Tutorial at the Conference on Neural Information Processing Systems (NIPS), Montreal,
December 7, 2015.
Sutton, R. S. (2015b) True online Emphatic TD( ): Quick reference and implementation guide.
ArXiv:1507.07147. Code is available in Python and C++ by downloading the source ﬁles of
this arXiv paper as a zip archive.
Sutton, R. S., Barto, A. G. (1981a). Toward a modern theory of adaptive networks: Expectation
and prediction. Psychological Review, 88 (2):135–170.
Sutton, R. S., Barto, A. G. (1981b). An adaptive network that constructs and uses an internal
model of its world. Cognition and Brain Theory, 3 :217–246.

Sutton, R. S., Barto, A. G. (1987). A temporal-di↵erence model of classical conditioning. In
Proceedings of the Ninth Annual Conference of the Cognitive Science Society ,p p .3 5 5 - 3 7 8 .
Erlbaum, Hillsdale, NJ.
Sutton, R. S., Barto, A. G. (1990). Time-derivative models of Pavlovian reinforcement. In
M. Gabriel and J. Moore (Eds.), Learning and Computational Neuroscience: Foundations of
Adaptive Networks , pp. 497–537. MIT Press, Cambridge, MA.
Sutton, R. S., Maei, H. R., Precup, D., Bhatnagar, S., Silver, D., Szepesv´ ari, Cs., Wiewiora, E.

(2009a). Fast gradient-descent methods for temporal-di↵erence learning with linear function
approximation. In Proceedings of the 26th International Conference on Machine Learning
(ICML 2009) , pp. 993–1000. ACM, New York.
Sutton, R. S., Szepesv´ ari, Cs., Maei, H. R. (2009b). A convergent O(d2) temporal-di↵erence
algorithm for o↵-policy learning with linear function approximation. In Advances in Neural
Information Processing Systems 21 (NIPS 2008) , pp. 1609–1616. Curran Associates, Inc.

Sutton, R. S., Mahmood, A. R., Precup, D., van Hasselt, H. (2014). A new Q( ) with interim
forward view and Monte Carlo equivalence. In Proceedings of the International Conference
on Machine Learning, 31 .JMLR W&CP 32 (2).
Sutton, R. S., Mahmood, A. R., White, M. (2016). An emphatic approach to the problem of
o↵-policy temporal-di↵erence learning. Journal of Machine Learning Research, 17 (73):1–29.
Sutton, R. S., McAllester, D. A., Singh, S. P., Mansour, Y. (2000). Policy gradient methods

for reinforcement learning with function approximation. In Advances in Neural Information
Processing Systems 12 (NIPS 1999) , pp. 1057–1063. MIT Press, Cambridge, MA.
Sutton, R. S., Modayil, J., Delp, M., Degris, T., Pilarski, P. M., White, A., Precup, D. (2011).
Horde: A scalable real-time architecture for learning knowledge from unsupervised senso-
rimotor interaction. In Proceedings of the Tenth International Conference on Autonomous
Agents and Multiagent Systems ,p p .7 6 1 – 7 6 8 ,T a i p e i ,T a i w a n .

Sutton, R. S., Pinette, B. (1985). The learning of world models by connectionist networks. In
Proceedings of the Seventh Annual Conference of the Cognitive Science Society , pp. 54–64.
Sutton, R. S., Precup, D., Singh, S. (1999). Between MDPs and semi-MDPs: A framework for
temporal abstraction in reinforcement learning. Artiﬁcial Intelligence, 112 (1-2):181–211.
Sutton, R. S., Rafols, E., Koop, A. (2006). Temporal abstraction in temporal-di↵erence networks.
InAdvances in neural information processing systems ,p p .1 3 1 3 – 1 3 2 0 .

Sutton, R. S., Singh, S. P., McAllester, D. A. (2000). Comparing policy-gradient algorithms.
Sutton, R. S., Szepesv´ ari, Cs., Geramifard, A., Bowling, M., (2008). Dyna-style planning with

References 513
linear function approximation and prioritized sweeping. In Proceedings of the 24th Conference
on Uncertainty in Artiﬁcial Intelligence ,p p .5 2 8 – 5 3 6 .
Sutton, R. S., Tanner, B. (2005). Temporal-di↵erence networks. In Advances in Neural
Information Processing Systems 17 ,p .1 3 7 7 – 1 3 8 4 .
Szepesv´ ari, Cs. (2010). Algorithms for reinforcement learning. In Synthesis Lectures on Artiﬁcial
Intelligence and Machine Learning, 4 (1):1–103. Morgan and Claypool.
Szita, I. (2012). Reinforcement learning in games. In M. Wiering and M. van Otterlo (Eds.),
Reinforcement Learning: State-of-the-Art , pp. 539–577. Springer-Verlag Berlin Heidelberg.
Tadepalli, P., Ok, D. (1994). H-learning: A reinforcement learning method to optimize
undiscounted average reward. Technical Report 94-30-01. Oregon State University, Computer
Science Department, Corvallis.
Tadepalli, P., Ok, D. (1996). Scaling up average reward reinforcement learning by approximating

the domain models and the value function. In Proceedings of the 13th International Conference
on Machine Learning (ICML 1996) ,p p .4 7 1 – 4 7 9 .
Takahashi, Y., Schoenbaum, G., and Niv, Y. (2008). Silencing the critics: Understanding the
e↵ects of cocaine sensitization on dorsolateral and ventral striatum in the context of an
actor/critic model. Frontiers in Neuroscience, 2 (1):86–99.
Tambe, M., Newell, A., Rosenbloom, P. S. (1990). The problem of expensive chunks and its

solution by restricting expressiveness. Machine Learning, 5 (3):299–348.
Tan, M. (1991). Learning a cost-sensitive internal representation for reinforcement learning. In
L. A. Birnbaum and G. C. Collins (Eds.), Proceedings of the 8th International Workshop on
Machine Learning , pp. 358–362. Morgan Kaufmann.
Tanner, B. (2006). Temporal-Di↵erence Networks . MSc thesis, University of Alberta.
Taylor, G., Parr, R. (2009). Kernelized value function approximation for reinforcement learning.

InProceedings of the 26th International Conference on Machine Learning (ICML 2009) ,
pp. 1017–1024. ACM, New York.
Taylor, M. E., Stone, P. (2009). Transfer learning for reinforcement learning domains: A survey.
Journal of Machine Learning Research, 10 :1633–1685.
Tesauro, G. (1986). Simple neural models of classical conditioning. Biological Cybernetics,
55(2-3):187–200.
Tesauro, G. (1992). Practical issues in temporal di↵erence learning. Machine Learning,
8(3-4):257–277.

Tesauro, G. (1994). TD-Gammon, a self-teaching backgammon program, achieves master-level
play. Neural Computation, 6 (2):215–219.
Tesauro, G. (1995). Temporal di↵erence learning and TD-Gammon. Communications of the
ACM, 38 (3):58–68.
Tesauro, G. (2002). Programming backgammon using self-teaching neural nets. Artiﬁcial
Intelligence, 134 (1-2):181–199.
Tesauro, G., Galperin, G. R. (1997). On-line policy improvement using Monte-Carlo search. In
Advances in Neural Information Processing Systems 9 (NIPS 1996) ,p p .1 0 6 8 – 1 0 7 4 .M I T

Tesauro, G., Gondek, D. C., Lechner, J., Fan, J., Prager, J. M. (2012). Simulation, learning,
and optimization techniques in Watson’s game strategies. IBM Journal of Research and
Development, 56 (3-4):16–1–16–11.
Tesauro, G., Gondek, D. C., Lenchner, J., Fan, J., Prager, J. M. (2013). Analysis of W atson ’s
strategies for playing Jeopardy! Journal of Artiﬁcial Intelligence Research, 47 :205–251.
Tham, C. K. (1994). Modular On-Line Function Approximation for Scaling up Reinforcement

514 References
Learning . Ph.D. thesis, University of Cambridge.
Thathachar, M. A. L., Sastry, P. S. (1985). A new approach to the design of reinforcement
schemes for learning automata. IEEE Transactions on Systems, Man, and Cybernetics,
15(1):168–175.
Thathachar, M., Sastry, P. S. (2002). Varieties of learning automata: an overview. IEEE
Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, 36 (6):711–722.
Thathachar, M., Sastry, P. S. (2011). Networks of Learning Automata: Techniques for Online
Stochastic Optimization . Springer Science & Business Media.
Theocharous, G., Thomas, P. S., Ghavamzadeh, M. (2015). Personalized ad recommendation for
life-time value optimization guarantees. In Proceedings of the Twenty-Fourth International
Joint Conference on Artiﬁcial Intelligence . AAAI Press, Palo Alto, CA.
Thistlethwaite, D. (1951). A critical review of latent learning and related experiments. Psycho-
logical Bulletin, 48 (2):97–129.

Thomas, P. S. (2014). Bias in natural actor–critic algorithms. In Proceedings of the 31st
International Conference on Machine Learning ,JMLR W&CP 32 (1), pp. 441–448.
Thomas, P. S. (2015). Safe Reinforcement Learning . Ph.D. thesis, University of Massachusetts,
Amherst.
Thomas, P. S., Brunskill, E. (2017). Policy gradient methods for reinforcement learning with
function approximation and action-dependent baselines. ArXiv:1706.06643.
Thomas, P. S., Theocharous, G., Ghavamzadeh, M. (2015). High-conﬁdence o↵-policy evaluation.

Amherst.
InProceedings of the Twenty-Ninth AAAI Conference on Artiﬁcial Intelligence ,p p .3 0 0 0 – 3 0 0 6 .
AAAI Press, Menlo Park, CA.
Thompson, W. R. (1933). On the likelihood that one unknown probability exceeds another in
view of the evidence of two samples. Biometrika, 25 (3-4):285–294.
Thompson, W. R. (1934). On the theory of apportionment. American Journal of Mathematics,
57:4 5 0 – 4 5 7 .
Thon, M. (2017). Spectral Learning of Sequential Systems. Ph.D. thesis, Jacobs University
Bremen.

Bremen.
Thon, M., Jaeger, H. (2015). Links between multiplicity automata, observable operator models
and predictive state representations: a uniﬁed learning framework. The Journal of Machine
Learning Research, 16 (1):103–147.
Thorndike, E. L. (1898). Animal intelligence: An experimental study of the associative processes
in animals. The Psychological Review, Series of Monograph Supplements ,I I ( 4 ) .
Thorndike, E. L. (1911). Animal Intelligence . Hafner, Darien, CT.

Bremen.
Thorp, E. O. (1966). Beat the Dealer: A Winning Strategy for the Game of Twenty-One .
Random House, New York.
Tian, T. (in preparation) An Empirical Study of Sliding-Step Methods in Temporal Di↵erence
Learning. M.Sc thesis, University of Alberta, Edmonton.
Tieleman, T., Hinton, G. (2012). Lecture 6.5–RMSProp. COURSERA: Neural networks for
machine learning 4.2:26–31.
Tolman, E. C. (1932). Purposive Behavior in Animals and Men . Century, New York.
Tolman, E. C. (1948). Cognitive maps in rats and men. Psychological Review, 55 (4):189–208.

Tsai, H.-S., Zhang, F., Adamantidis, A., Stuber, G. D., Bonci, A., de Lecea, L., Deisseroth,
K. (2009). Phasic ﬁring in dopaminergic neurons is sucient for behavioral conditioning.
Science, 324 (5930):1080–1084.
Tsetlin, M. L. (1973). Automaton Theory and Modeling of Biological Systems . Academic Press,
New York.

References 515
Tsitsiklis, J. N. (1994). Asynchronous stochastic approximation and Q-learning. Machine
Learning, 16 (3):185–202.
Tsitsiklis, J. N. (2002). On the convergence of optimistic policy iteration. Journal of Machine
Learning Research, 3 :59–72.
Tsitsiklis, J. N., Van Roy, B. (1996). Feature-based methods for large scale dynamic programming.
Machine Learning, 22 (1-3):59–94.
Tsitsiklis, J. N., Van Roy, B. (1997). An analysis of temporal-di↵erence learning with function
approximation. IEEE Transactions on Automatic Control, 42 (5):674–690.
Tsitsiklis, J. N., Van Roy, B. (1999). Average cost temporal-di↵erence learning. Automatica,
35(11):1799–1808.
Turing, A. M. (1948). Intelligent machinery. In B. Jack Copeland (Ed.) (2004), The Essential
Turing , pp. 410–432. Oxford University Press, Oxford.
Ungar, L. H. (1990). A bioreactor benchmark for adaptive network-based process control. In
W. T. Miller, R. S. Sutton, and P. J. Werbos (Eds.), Neural Networks for Control ,p p .3 8 7 – 4 0 2 .

Unnikrishnan, K. P., Venugopal, K. P. (1994). Alopex: A correlation-based learning algorithm
for feedforward and recurrent neural networks. Neural Computation, 6(3): 469–490.
Urbanczik, R., Senn, W. (2009). Reinforcement learning in populations of spiking neurons.
Nature neuroscience, 12 (3):250–252.
Urbanowicz, R. J., Moore, J. H. (2009). Learning classiﬁer systems: A complete introduction,
review, and roadmap. Journal of Artiﬁcial Evolution and Applications .1 0 . 1 1 5 5 / 2 0 0 9 / 7 3 6 3 9 8 .

Valentin, V. V., Dickinson, A., O’Doherty, J. P. (2007). Determining the neural substrates of
goal-directed learning in the human brain. The Journal of Neuroscience, 27 (15):4019–4026.
van Hasselt, H. (2010). Double Q-learning. In Advances in Neural Information Processing
Systems 23 (NIPS 2010) , pp. 2613–2621. Curran Associates, Inc.
van Hasselt, H. (2011). Insights in Reinforcement Learning: Formal Analysis and Empirical
Evaluation of Temporal-di↵erence Learning . SIKS dissertation series number 2011-04.

van Hasselt, H. (2012). Reinforcement learning in continuous state and action spaces. In M.
Wiering and M. van Otterlo (Eds.), Reinforcement Learning: State-of-the-Art ,p p .2 0 7 – 2 5 1 .
van Hasselt, H., Sutton, R. S. (2015). Learning to predict independent of span. ArXiv:1508.04582.
Van Roy, B., Bertsekas, D. P., Lee, Y., Tsitsiklis, J. N. (1997). A neuro-dynamic programming
approach to retailer inventory management. In Proceedings of the 36th IEEE Conference on

Decision and Control ,V o l .4 ,p p .4 0 5 2 – 4 0 5 7 .
van Seijen, H. (2011). Reinforcement Learning under Space and Time Constraints. University of
Amsterdam PhD thesis. Hague: TNO.
van Seijen, H. (2016). E↵ective multi-step temporal-di↵erence learning for non-linear function
approximation. ArXiv:1608.05151.
van Seijen, H., Sutton, R. S. (2013). Ecient planning in MDPs by small backups. In: Proceedings
of the 30th International Conference on Machine Learning (ICML 2013) ,p p .3 6 1 – 3 6 9 .

van Seijen, H., Sutton, R. S. (2014). True online TD( ). In Proceedings of the 31st International
Conference on Machine Learning (ICML 2014) ,p p .6 9 2 – 7 0 0 .J M L RW & C P3 2 ( 1 ) ,
van Seijen, H., Mahmood, A. R., Pilarski, P. M., Machado, M. C., Sutton, R. S. (2016). True
online temporal-di↵erence learning. Journal of Machine Learning Research, 17 (145):1–40.
van Seijen, H., Van Hasselt, H., Whiteson, S., Wiering, M. (2009). A theoretical and empirical

analysis of Expected Sarsa. In IEEE Symposium on Adaptive Dynamic Programming and
Reinforcement Learning ,p p .1 7 7 – 1 8 4 .

516 References
van Seijen, H., Whiteson, S., van Hasselt, H., Wiering, M. (2011). Exploiting best-match
equations for ecient reinforcement learning. Journal of Machine Learning Research 12 :2045–
2094.
Varga, R. S. (1962). Matrix Iterative Analysis. Englewood Cli↵s, NJ: Prentice-Hall.
Vasilaki, E., Fr´ emaux, N., Urbanczik, R., Senn, W., Gerstner, W. (2009). Spike-based rein-
forcement learning in continuous state and action space: when policy gradient methods fail.
PLoS Computational Biology, 5 (12).
Viswanathan, R., Narendra, K. S. (1974). Games of stochastic automata. IEEE Transactions
on Systems, Man, and Cybernetics, 4 (1):131–135.
Wagner, A. R. (2008). Evolution of an elemental theory of Pavlovian conditioning. Learning &
Behavior, 36 (3):253–265.
Walter, W. G. (1950). An imitation of life. Scientiﬁc American, 182 (5):42–45.
Walter, W. G. (1951). A machine that learns. Scientiﬁc American, 185 (2):60–63.
Waltz, M. D., Fu, K. S. (1965). A heuristic approach to reinforcement learning control systems.

IEEE Transactions on Automatic Control, 10 (4):390–398.
Watkins, C. J. C. H. (1989). Learning from Delayed Rewards . Ph.D. thesis, University of
Watkins, C. J. C. H., Dayan, P. (1992). Q-learning. Machine Learning, 8 (3-4):279–292.
Werbos, P. J. (1977). Advanced forecasting methods for global crisis warning and models of
intelligence. General Systems Yearbook, 22 (12):25–38.
Werbos, P. J. (1982). Applications of advances in nonlinear sensitivity analysis. In R. F. Drenick

and F. Kozin (Eds.), System Modeling and Optimization , pp. 762–770. Springer-Verlag.
Werbos, P. J. (1987). Building and understanding adaptive systems: A statistical/numerical
approach to factory automation and brain research. IEEE Transactions on Systems, Man,
and Cybernetics, 17 (1):7–20.
Werbos, P. J. (1988). Generalization of back propagation with applications to a recurrent gas
market model. Neural Networks, 1 (4):339–356.
Werbos, P. J. (1989). Neural networks for control and system identiﬁcation. In Proceedings of

the 28th Conference on Decision and Control , pp. 260–265. IEEE Control Systems Society.
Werbos, P. J. (1992). Approximate dynamic programming for real-time control and neural
modeling. In D. A. White and D. A. Sofge (Eds.), Handbook of Intelligent Control: Neural,
Fuzzy, and Adaptive Approaches , pp. 493–525. Van Nostrand Reinhold, New York.
Werbos, P. J. (1994). The Roots of Backpropagation: From Ordered Derivatives to Neural
Networks and Political Forecasting (Vol. 1). John Wiley and Sons.

Wiering, M., Van Otterlo, M. (2012). Reinforcement Learning: State-of-the-Art . Springer-Verlag
Berlin Heidelberg.
White, A. (2015). Developing a Predictive Approach to Knowledge . Ph.D. thesis, University of
Alberta, Edmonton.
White, D. J. (1969). Dynamic Programming . Holden-Day, San Francisco.
White, D. J. (1985). Real applications of Markov decision processes. Interfaces, 15 (6):73–83.
White, D. J. (1988). Further real applications of Markov decision processes. Interfaces,
18(5):55–61.

White, D. J. (1993). A survey of applications of Markov decision processes. Journal of the
Operational Research Society, 44 (11):1073–1096.
White, A., White, M. (2016). Investigating practical linear temporal di↵erence learning. In
Proceedings of the 2016 International Conference on Autonomous Agents and Multiagent
Systems ,p p .4 9 4 – 5 0 2 .

References 517
Whitehead, S. D., Ballard, D. H. (1991). Learning to perceive and act by trial and error.
Machine Learning, 7 (1):45–83.
Whitt, W. (1978). Approximations of dynamic programs I. Mathematics of Operations Research,
3(3):231–243.
Whittle, P. (1982). Optimization over Time , vol. 1. Wiley, New York.
Whittle, P. (1983). Optimization over Time , vol. 2. Wiley, New York.
Wickens, J., K¨ otter, R. (1995). Cellular models of reinforcement. In J. C. Houk, J. L. Davis and
D. G. Beiser (Eds.), Models of Information Processing in the Basal Ganglia ,p p .1 8 7 – 2 1 4 .
Widrow, B., Gupta, N. K., Maitra, S. (1973). Punish/reward: Learning with a critic in adaptive
threshold systems. IEEE Transactions on Systems, Man, and Cybernetics, 3 (5):455–465.
Widrow, B., Ho↵, M. E. (1960). Adaptive switching circuits. In 1960 WESCON Convention
Record Part IV , pp. 96–104. Institute of Radio Engineers, New York. Reprinted in J. A.

Anderson and E. Rosenfeld, Neurocomputing: Foundations of Research ,p p .1 2 6 – 1 3 4 .M I T
Press, Cambridge, MA, 1988.
Widrow, B., Smith, F. W. (1964). Pattern-recognizing control systems. In J. T. Tou and
R. H. Wilcox (Eds.), Computer and Information Sciences , pp. 288–317. Spartan, Washington,
DC.
Widrow, B., Stearns, S. D. (1985). Adaptive Signal Processing . Prentice-Hall, Englewood Cli↵s,
NJ.
Wiener, N. (1964). God and Golem, Inc: A Comment on Certain Points where Cybernetics
Impinges on Religion . MIT Press, Cambridge, MA.

DC.
NJ.
Wiewiora, E. (2003). Potential-based shaping and Q-value initialization are equivalent. Journal
of Artiﬁcial Intelligence Research, 19 :205–208.
Williams, R. J. (1986). Reinforcement learning in connectionist networks: A mathematical
analysis. Technical Report ICS 8605. Institute for Cognitive Science, University of California
at San Diego, La Jolla.
Williams, R. J. (1987). Reinforcement-learning connectionist systems. Technical Report
NU-CCS-87-3. College of Computer Science, Northeastern University, Boston.

Williams, R. J. (1988). On the use of backpropagation in associative reinforcement learning. In
Proceedings of the IEEE International Conference on Neural Networks ,p p .I 2 6 3 – I 2 7 0 .I E E E
San Diego section and IEEE TAB Neural Network Committee.
Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist
reinforcement learning. Machine Learning, 8 (3-4):229–256.
Williams, R. J., Baird, L. C. (1990). A mathematical analysis of actor–critic architectures for
learning optimal controls through incremental dynamic programming. In Proceedings of the

Sixth Yale Workshop on Adaptive and Learning Systems , pp. 96–101. Center for Systems
Science, Dunham Laboratory, Yale University, New Haven.
Wilson, R. C., Takahashi, Y. K., Schoenbaum, G., Niv, Y. (2014). Orbitofrontal cortex as a
cognitive map of task space. Neuron, 81 (2):267–279.
Wilson, S. W. (1994). ZCS, A zeroth order classiﬁer system. Evolutionary Computation,
2(1):1–18.
Wise, R. A. (2004). Dopamine, learning, and motivation. Nature Reviews Neuroscience,
5(6):1–12.

Witten, I. H. (1976a). Learning to Control. University of Essex PhD thesis.
Witten, I. H. (1976b). The apparent conﬂict between estimation and control—A survey of the
two-armed problem. Journal of the Franklin Institute, 301 (1-2):161–189.

518 Index
Witten, I. H. (1977). An adaptive optimal controller for discrete-time Markov environments.
Information and Control, 34 (4):286–295.
Witten, I. H., Corbin, M. J. (1973). Human operators and automatic adaptive controllers: A
comparative study on a particular control task. International Journal of Man–Machine
Studies, 5 (1):75–104.
Woodbury, T., Dunn, C., and Valasek, J. (2014). Autonomous soaring using reinforcement
learning for trajectory generation. In 52nd Aerospace Sciences Meeting ,p .0 9 9 0 .
Woodworth, R. S. (1938). Experimental Psychology . New York: Henry Holt and Company.
Xie, X., Seung, H. S. (2004). Learning in neural networks by reinforcement of irregular spiking.
Physical Review E, 69 (4):041909.
Xu, X., Xie, T., Hu, D., Lu, X. (2005). Kernel least-squares temporal di↵erence learning.
International Journal of Information Technology, 11 (9):54–63.
Yagishita, S., Hayashi-Takagi, A., Ellis-Davies, G. C. R., Urakubo, H., Ishii, S., Kasai, H. (2014).

A critical time window for dopamine actions on the structural plasticity of dendritic spines.
Science, 345 (6204):1616–1619.
Yee, R. C., Saxena, S., Utgo↵, P. E., Barto, A. G. (1990). Explaining temporal di↵erences to
create useful concepts for evaluating states. In Proceedings of the Eighth National Conference
on Artiﬁcial Intelligence (AAAI-90) , pp. 882–888. AAAI Press, Menlo Park, CA.
Yin, H. H., Knowlton, B. J. (2006). The role of the basal ganglia in habit formation. Nature
Reviews Neuroscience, 7 (6):464–476.

Young, P. (1984). Recursive Estimation and Time-Series Analysis . Springer-Verlag, Berlin.
Yu, H. (2010). Convergence of least squares temporal di↵erence methods under general conditions.
International Conference on Machine Learning 27 ,p p .1 2 0 7 – 1 2 1 4 .
Yu, H. (2012). Least squares temporal di↵erence methods: An analysis under general conditions.
SIAM Journal on Control and Optimization, 50 (6):3310–3343.
Yu, H. (2015). On convergence of emphatic temporal-di↵erence learning. In Proceedings of the
28th Annual Conference on Learning Theory ,JMLR W&CP 40 . Also ArXiv:1506.02582.

Yu, H. (2016). Weak convergence properties of constrained emphatic temporal-di↵erence learning
with constant and slowly diminishing stepsize. Journal of Machine Learning Research,
17(220):1–58.
Yu, H. (2017). On convergence of some gradient-based temporal-di↵erences algorithms for
o↵-policy learning. Arxiv:1712.09652.
Yu, H., Mahmood, A. R., Sutton, R. S. (2017). On generalized bellman equations and temporal-
di↵erence learning. ArXiv:17041.04463. A summary appeared in Proceedings of the Canadian

Conference on Artiﬁcial Intelligence , pp. 3–14. Springer.

Index
Page numbers in italics are recommended to be consulted ﬁrst. Page numbers in bold contain
boxed algorithms.
k-armed bandits, 25–45
absorbing state, 57
access-control queuing example, 256
action preferences, 322,3 2 9 ,3 3 6 ,4 5 5
in bandit problems, 37,4 2
action-value function, seevalue function, action
action-value methods, 321
for bandit problems, 27
actor–critic, 21, 239, 321, 331–332 ,3 3 8 ,4 0 6
one-step (episodic), 332
with eligibility traces (episodic), 332
with eligibility traces (continuing), 333
neural, 395–415
addiction, 409–410
advantage actor–critic methods, 338
afterstates, 137,1 4 0 ,1 8 1 ,1 8 2 ,1 9 1 ,4 2 4 ,4 3 0
agent–environment interface, 47–57, 467
all-actions algorithm, 326
AlphaGo, AlphaGo Zero, AlphaZero, 441–450
Andreae, John, 17,2 1 ,6 9 ,8 9
ANN, seeartiﬁcial neural networks
applications and case studies, 421–457
approximate dynamic programming, 15
artiﬁcial intelligence, xvii, 1, 472, 475–478
artiﬁcial neural networks, 223–228 ,2 3 8 – 2 3 9 ,

395–398, 423, 430, 436–450, 472
associative reinforcement learning, 45,4 1 8
associative search, 41
asynchronous dynamic programming, 85,8 8
Atari video game play, 436–441
auxiliary tasks, 460–461 ,4 6 8 ,4 7 4
average reward setting, 249–255 ,2 5 8 ,4 6 4
averagers, 264
backgammon, 11, 21, 182, 184, 421–426
backpropagation, 21, 225–227 ,2 3 9 ,4 0 7 ,4 2 4 ,
436, 439
backup diagram, 60,1 3 9
for dynamic programming, 59, 61, 64, 172
for Monte Carlo methods, 94
for Q-learning, 134
for TD(0), 121for Sarsa, 129
for Expected Sarsa, 134

Bellman equation, 14
forv⇡,5 9
forq⇡,7 8
for optimal value functions: v⇤andq⇤,6 3
di↵erential, 250
for options, 463
Bellman error, 268,2 7 0 ,2 7 2 ,2 7 3
learnability of, 274–277
vector, 267–269
Bellman operator, 267–269, 286
Bellman residual, 286, seeBellman error
Bellman, Richard, 14,7 1 ,8 9 ,2 4 1
binary features, 215,2 2 2 ,2 4 5 ,3 0 4 ,3 0 5
bioreactor example, 51
blackjack example, 93–94 ,99,1 0 6
blocking maze example, 166
bootstrapping, 89,1 8 9 ,3 0 8 ,3 3 1
n-step, 141–158 ,2 5 5
and dynamic programming, 89
and function approximation, 208, 264–274
519

520 Index
and Monte Carlo methods, 95
and stability, 263–265
and TD learning, 120
assessment of, 124–128, 248, 264,2 9 1 ,3 1 8
in psychology, 345, 349, 354,3 5 5
parameter ( orn),291,3 0 7 ,3 9 9
BOXES, 18,7 1 ,2 3 7
branching factor, 173–177 ,4 2 2
breakfast example, 5,2 2
bucket-brigade algorithm, 19, 21, 139
catastrophic interference, 472
certainty-equivalence estimate, 128
chess, 4, 20, 54, 182, 450
classical conditioning, 20, 343–357
blocking, 371
and higher-order conditioning, 345–355
delay and trace conditioning, 344
Rescorla-Wagner model, 346–349
TD model, 349–357
classiﬁer systems, 19,2 1
cli↵ walking example, 132,1 3 3
CMAC, seetile coding
coarse coding, 215–220 ,2 3 7
cognitive maps, 363–364
collective reinforcement learning, 404–407
complex backups, seecompound update
compound stimulus, 345, 346–356 ,3 7 1 ,3 8 2
compound update/backup, 288, 319
conditioned/unconditioned stimulus, conditioned
response (CS/US, CR), 343
constant- ↵MC, 120
contextual bandits, 41

continuing tasks, 54,5 7 ,7 0 ,1 2 4 ,2 4 9 ,2 9 4
continuous action, 73, 244, 335–336
continuous state, 73,2 2 3 ,2 3 7
continuous time, 11,7 1
control and prediction, 342
control theory, 4,7 0
control variates, 150–152 ,1 5 5 ,2 8 1
and eligibility traces, 309–312
credit assignment, 11, 17,1 9 ,4 7 ,2 9 4 ,4 0 1
in psychology, 346, 361
structural, 385,4 0 5 ,4 0 7
critic, 18,2 3 8 ,3 4 6 ,4 1 7 , seeactor–critic
cumulant, 459
curiosity, 474
curse of dimensionality, 4, 14,2 2 1 ,2 3 1
cybernetics, xvii, 477deadly triad, 264

deep learning, 11, 223,4 4 1 ,4 7 2 – 4 7 4 ,4 8 0
deep reinforcement learning, 236
deep residual learning, 226
delayed reinforcement, 361–363
delayed reward, 2,4 7 ,2 4 9
dimensions of reinforcement learning methods,
189–191
direct and indirect RL, 162,1 6 4 ,1 9 2
discounting, 55,1 9 9 ,2 4 3 ,2 4 9 ,2 8 2 ,3 2 4 ,3 2 8 ,
427, 459
in pole balancing, 56
state dependent, 307
deprecated, 253,2 5 6
distribution models, 159,1 8 5
dopamine, 377, 381–387 ,4 1 3 – 4 1 9
and addiction, 409–410
double learning, 134–136 ,1 4 0

189–191
427, 459
DP,seedynamic programming
driving-home example, 122–123
Dyna architecture, 164,1 6 1 – 1 7 0
dynamic programming, 13–15, 73–90 ,1 7 4 ,2 6 2
and artiﬁcial intelligence, 89
and function approximation, 241
and options, 463
and the deadly triad, 264
computational eciency of, 87
eligibility traces, 287–320 ,3 5 0 ,3 6 2 ,3 9 8 – 4 0 3
accumulating, 301,3 0 6 ,3 1 0
replacing, 301,3 0 6
dutch, 300–303
contingent/non-contingent, 399–403, 411
o↵-policy, 309–316
with state-dependent and,3 0 9 – 3 1 6
Emphatic-TD methods, 234–235, 315

o↵-policy, 281–282
environment, 47–57
episodes, episodic tasks, 11, 54–57 ,9 1
error reduction property, 144,2 8 8
evaluative feedback, 17, 25,4 7
evolution, 7, 359, 374, 471
evolutionary methods, 7,8 – 1 0 ,1 9
expected approximate value, 148,1 5 5
Expected Sarsa, 133, see also Sarsa, Expected
expected update, 75,1 7 2 – 1 8 1 ,1 8 9
experience replay, 440–441
explore/exploit dilemma, 3,1 0 3 ,4 7 2
exploring starts, 96,9 8 – 1 0 0 ,1 7 8

Index 521
feature construction, 210–223
ﬁnal time step ( T), 54
Fourier basis, 211–214
function approximation, 195–200
gambler’s example, 84
game theory, 19
gazelle calf example, 5
general value functions (GVFs), 459–463 ,4 7 4
generalized policy iteration (GPI), 86–87 ,9 2 ,
97, 138, 189
genetic algorithms, 19
Gittins index, 43
gliding/soaring case study, 453–457
goal, seereward signal
golf example, 61,6 3 ,6 6
gradient, 201
gradient descent, seestochastic gradient de-
scent
Gradient-TD methods, 278–281 ,3 1 4 – 3 1 5
greedy or "-greedy
as exploiting, 26–28
as shortsighted, 64
"-greedy policies, 100
gridworld examples, 60, 65, 76, 147
cli↵ walking, 132
Dyna blocking maze, 166
Dyna maze, 164
Dyna shortcut maze, 167
windy, 130, 131
habitual and goal-directed control, 364–368
hedonistic neurons, 402–404
heuristic search, 181–183 ,1 9 0
as sequences of backups, 183
in Samuel’s checkers player, 426
in TD-Gammon, 425
history of reinforcement learning, 13–21
Holland, John, 19,2 1 ,4 4 ,1 3 9 ,2 4 1

Hull, Clark, 16, 359, 360, 362–363
importance sampling, 103–117 ,1 5 1 ,2 5 7
ratio, 104,1 4 8 ,2 5 8
weighted and ordinary, 105,1 0 6
and inﬁnite variance, 106
discounting aware, 112–113
incremental implementation, 109
per-decision, 114–115n-step, 148–156
incremental implementation
of averages, 30–33
of weighted averages, 109
instrumental conditioning, 357–361, see also
Law of E↵ect
and motivation, 360–361
Thorndike’s puzzle boxes, 358
interest and emphasis, 234–235 ,2 8 2 ,3 1 6

inverse reinforcement learning, 470
Jack’s car rental example, 81–82 ,1 3 7 ,2 1 0
kernel-based function approximation, 232–233
Klopf, A. Harry, xv,x v i i ,1 9 – 2 1 ,4 0 2 – 4 0 4 ,4 1 1
latent learning, 192, 363,3 6 6
Law of E↵ect, 15–16 ,4 5 ,3 4 2 ,3 5 8 – 3 6 1 ,4 1 7
learning automata, 18
Least Mean Square (LMS) algorithm, 279, 301
Least-Squares TD (LSTD), 228–229
linear function approx., 204–209 ,2 6 6 – 2 6 9
linear programming, 87, 90
local and global optima, 200
Markov decision process (MDP), 2, 14, 47–71

Markov property, 49,1 1 5 ,4 6 5 – 4 6 8
Markov reward process (MRP), 125
maximization bias, 134–136
maximum-likelihood estimate, 128
MC, seeMonte Carlo methods
Mean Squared
Bellman Error, BE, 268
Projected Bellman Error, PBE, 269
Return Error, RE, 275
TD Error, TDE, 270
Value Error, VE, 199–200
memory-based function approx., 230–232
Michie, Donald, 17,7 1 ,1 1 6
Minsky, Marvin, 16,1 7 ,2 0 ,8 9
model of the environment, 7, 159
model-based and model-free methods, 7, 159
in animal learning, 363–368

model-based reinforcement learning, 159–193
in neuroscience, 407–409
Monte Carlo methods, 91–117
ﬁrst- and every-visit MC, 92
ﬁrst-visit MC control, 101
ﬁrst-visit MC prediction, 92

on-policy distribution, 175,1 9 9 ,2 0 8 ,2 5 8 ,2 6 2 ,
281, 282
vs uniform distribution, 176
on-policy methods, 100
actor–critic, 332, 333approximate
control, 244,247,251,255
prediction, 202, 203 ,209
Monte Carlo, 101,1 0 0 – 1 0 3 , 328,330
n-step, 144,147
Sarsa, 130,1 2 9 – 1 3 1
TD(0), 120,1 1 9 – 1 2 8
with eligibility traces, 293,300,305,307
operant conditioning, seeinstrumental learning
optimal control, 2, 14–15 ,2 1
optimistic initial values, 34–35 ,1 9 2
optimizing memory control, 432–436
options, 461–464
models of, 462
pain and pleasure, 6,1 6 ,4 1 3
Partially Observable MDPs (POMDPs), 466
Pavlov, Ivan, 16, 343–345 ,3 6 2
Pavlovian
conditioning, seeclassical conditioning
control, 343, 371,3 7 3 ,4 7 9
personalizing web services, 450–453
planning, 3, 5, 7, 11, 138, 159–193
in psychology, 363, 364, 366
with learned models, 161–168 ,4 7 3
with options, 461, 463
policy, 6, 41, 58
hierarchical, 462
soft and "-soft, 100–103, 110
policy approximation, 321–324
policy evaluation, 74–76, see also prediction

Pavlovian
iterative, 75
policy gradient methods, 321–338
REINFORCE, 328,330
actor–critic, 332, 333
policy gradient theorem, 324–326
proof, episodic case, 325
proof, continuing case, 334
policy improvement, 76–80
theorem, 78,1 0 1
policy iteration, 14, 80,80–82
polynomial basis, 210–211
prediction, 74–76 ,see also policy evaluation
and control, 342
Monte Carlo, 92–97
o↵-policy, 103–108
TD, 119–126
with approximation, 197–241
prior knowledge, 11, 34, 54, 137, 236, 324, 470

Index 523
prioritized sweeping, 170,1 6 8 – 1 7 1
projected Bellman error, 285
vector, 267, 269
proximal TD methods, 286
pseudo termination, 282, 308
psychology, 4, 13, 18, 20, 341–376
Q(), Watkins’s, 312–314
Q-function, seeaction-value function
Q-learning, 21, 131,131–135
double, 136
Q-planning, 161
Q(),156,1 5 4 – 1 5 6
queuing example, 251
R-learning, 256
racetrack exercise, 111
radial basis functions (RBFs), 221–222
random walk, 95
5-state, 125,1 2 6 ,1 2 7
19-state, 144,2 9 1
TD() results on, 294, 295 ,2 9 9
1000-state, 203–209 ,2 1 7 ,2 1 8
Fourier and polynomial bases, 214
real-time dynamic programming, 177–180
recycling robot example, 52
REINFORCE, 328,3 2 6 – 3 3 1
with baseline, 330
reinforcement learning, 1–21
reinforcement signal, 380
representation learning, 473
residual-gradient algorithm, 272–274 ,2 7 7
naive, 270, 271
return, 54–57
n-step, 143
forQ(), 155
for action values, 146
for Expected Sarsa, 148
for Tree Backup, 153
with control variates, 150, 151
with function approximation, 209

di↵erential, 250, 255, 334
ﬂat partial, 113
with state-dependent termination, 308
-return, 288–291
truncated, 296
reward prediction error hypothesis, 381–383 ,
387–395
reward signal, 1, 6, 48,53,3 6 1 ,3 8 0 ,3 8 3 ,3 9 7and reinforcement, 373–375, 380–381
design of, 469–472 ,4 7 6
intrinsic, 474
sparse, 469–470
rod maneuvering example, 171
rollout algorithms, 183–185
root mean-squared (RMS) error, 125
safety, 434, 475–478
sample and expected updates, 121,1 7 0 – 1 7 4
sample or simulation model, 115

387–395
sample-average method, 27
Samuel’s checkers player, 20, 241, 426–429
Sarsa, 130,129–131 ,244
vs Q-learning, 132
di↵erential, one-step, 251
Expected, 133–134 ,1 4 0
n-step, 148
n-step o↵-policy, 150
n-step, 147,145–148 ,247
di↵erential, 255
o↵-policy, 149
Sarsa( ),305,3 0 3 – 3 0 7
true online, 307
Schultz, Wolfram, 387–395, 410
search control, 163
secondary reinforcement, 20,3 4 6 ,3 5 4 ,3 6 9
selective bootstrap adaptation, 239
semi-gradient methods, 202, 258–259

SGD, seestochastic gradient descent
Shannon, Claude, 16, 20,7 0 ,7 1 ,4 2 6
shaping, 360,4 7 0
Skinner, B. F., 359–360 ,3 7 5 ,4 7 0 ,4 8 0
soap bubble example, 95
soft and "-soft policies, 100–103, 110
soft-max, 322–323 ,3 2 9 ,3 3 6 ,4 0 0 ,4 4 5 ,4 5 5
for bandits, 37,4 5
spike-timing-dependent plasticity (STDP), 401
state, 7,4 8 ,4 9
kth-order history approach, 468
and observations, 464–468
Markov property, 465–468
belief, 466
latent, 466
observable operator models (OOMs), 467
partially observable MDPs, 14, 466

predictive state representations, 466
state-update function, 466

524 Index
state aggregation, 203–204
step-size parameter, 10, 31–33 ,1 2 0 ,1 2 5 ,1 2 6
automatic adaptation, 238
in DQN, 439, 440
in psychological models, 347, 348
selecting manually, 222–223
with coarse coding, 216
with Fourier features, 213
with tile coding, 217,2 2 3
stochastic approx. convergence conditions, 33
stochastic gradient descent (SGD), 200–204
in the Bellman error, 269–277
strong and weak methods, 4
supervised learning, xvii, 2,1 6 – 1 9 ,1 9 8
sweeps, 75,1 6 0 , see also prioritized sweeping
synaptic plasticity, 379
Hebbian, 400
two-factor and three factor, 400
system identiﬁcation, 364
tabular solution methods, 23
target
policy, 103,1 1 0
of update, 31, 143, 198
TD, seetemporal-di↵erence learning
TD error, 121
n-step, 255
with function approximation, 270
TD(),293,2 9 2 – 2 9 5
truncated, 295–297
true online, 300,2 9 9 – 3 0 1
TD-Gammon, 21, 421–426
temporal abstraction, 461–464
temporal-di↵erence learning, 10, 119–140
history of, 20–21

target
advantages of, 124–126
optimality of, 126–128
TD(0), 120,203
TD(1), 294
-return methods
o✏ine, 290
online, 297–299
n-step, 144,141–158 ,209
termination function, 307, 459
Thompson sampling, 43,4 5Thorndike, Edward, seeLaw of E↵ect
tic-tac-toe, 8–13,1 7 ,1 3 7
tile coding, 217–221 ,2 2 3 ,2 3 8 ,2 4 6 ,4 3 4 ,4 3 5
Tolman, Edward, 364,4 0 8
trace-decay parameter ( ), 287, 289,2 9 0 ,2 9 2
trajectory sampling, 174–177
transition probabilities, 49
Tree Backup

TD(1), 294
o✏ine, 290
n-step, 152–153 ,154
Tree-Backup( ), 312–314
trial-and-error, 2, 7, 15–21 ,4 0 3 ,4 0 4 , see also
instrumental conditioning
true online TD( ),300,2 9 9 – 3 0 1
Tsitsiklis and Van Roy’s Counterexample, 263
undiscounted continuing tasks, seeaverage re-
ward setting
unsupervised learning, 2,2 2 6
value, 6,2 6 ,4 7
value function, 6, 58–67
for a given policy: v⇡andq⇡,5 8
for an optimal policy: v⇤andq⇤,6 2
action, 58,6 3 ,6 4 ,7 1 ,1 2 9 ,1 3 1
approximate action values: ˆq(s, a,w), 243
approximate state values: ˆ v(s,w), 197

di↵erential, 243
vs evolutionary methods, 10
value iteration, 83,8 2 – 8 5
value-function approximation, 198
Watkins, Chris, 14,2 1 ,8 9 ,3 2 0
Watson ( Jeopardy! player), 429–432
Werbos, Paul, 14, 21, 69,8 9 ,1 3 9 ,2 3 8
Witten, Ian, 21,6 9

525
Francis Bach, Editor
Bioinformatics: The Machine Learning Approach , Pierre Baldi and Søren Brunak
Reinforcement Learning: An Introduction , Richard S. Sutton and Andrew G. Barto
Graphical Models for Machine Learning and Digital Communication , Brendan J. Frey
Learning in Graphical Models , Michael I. Jordan
Causation, Prediction, and Search , second edition, Peter Spirtes, Clark Glymour, and
Richard Scheines
Principles of Data Mining , David Hand, Heikki Mannila, and Padhraic Smyth
Bioinformatics: The Machine Learning Approach , second edition, Pierre Baldi and Søren
Brunak
Learning Kernel Classiﬁers: Theory and Algorithms , Ralf Herbrich
Learning with Kernels: Support Vector Machines, Regularization, Optimization, and
Beyond , Bernhard Sch¨ olkopf and Alexander J. Smola
Introduction to Machine Learning , Ethem Alpaydin
Gaussian Processes for Machine Learning , Carl Edward Rasmussen and Christopher K.I.
Williams

Brunak
Williams
Semi-Supervised Learning , Olivier Chapelle, Bernhard Sch¨ olkopf, and Alexander Zien,
Eds.
The Minimum Description Length Principle , Peter D. Gr¨ unwald
Introduction to Statistical Relational Learning , Lise Getoor and Ben Taskar, Eds.

526
Probabilistic Graphical Models: Principles and Techniques , Daphne Koller and Nir Fried-
man
Introduction to Machine Learning , second edition, Ethem Alpaydin
Machine Learning in Non-Stationary Environments: Introduction to Covariate Shift
Adaptation , Masashi Sugiyama and Motoaki Kawanabe
Boosting: Foundations and Algorithms , Robert E. Schapire and Yoav Freund
Machine Learning: A Probabilistic Perspective ,K e v i nP .M u r p h y
Foundations of Machine Learning , Mehryar Mohri, Afshin Rostami, and Ameet Talwalker
Introduction to Machine Learning , third edition, Ethem Alpaydin
Deep Learning , Ian Goodfellow, Yoshua Bengio, and Aaron Courville
Elements of Causal Inference , Jonas Peters, Dominik Janzing, and Bernhard Sch¨ olkopf
Machine Learning for Data Streams, with Practical Examples in MOA , Albert Bifet,
Ricard Gavald‘ a, Geo↵rey Holmes, Bernhard Pfahringer

ContentsWebsiteviiAcknowledgmentsviiiNotationxi1Introduction11.1WhoShouldReadThisBook?. . . . . .. . . . . . . . .. . . . .81.2HistoricalTrendsinDeepLearning. . . . . . . . .. . . . . . . .11IAppliedMathandMachineLearningBasics292LinearAlgebra312.1Scalars,Vectors,MatricesandTensors. . . . . . . . .. . . . . .312.2MultiplyingMatricesandVectors. . . . . . . . .. . . . . . . ..342.3IdentityandInverseMatrices. . . . . . . . .. . . . . . . .. . .362.4LinearDependenceandSpan. . . . . . . . .. . . . . . . .. . .372.5Norms. . . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .392.6SpecialKindsofMatricesandVectors. . . . . . . . . . . . . . .402.7Eigendecomposition. . . . . . . . . .. . . . . . . .. . . . . . . .422.8SingularValueDecomposition. . . . . . . .. . . . . . . .. . . .442.9TheMoore-PenrosePseudoinverse. . . . . . . . .. . . . . . . ..452.10TheTraceOperator. . . . . . . . .. . . . . . . .. . . . . . . .462.11TheDeterminant. .. . . . . . . .. . . . . . . .. . . . . . . .

. . . . .. . . . . . . .. . .372.5Norms. . . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .392.6SpecialKindsofMatricesandVectors. . . . . . . . . . . . . . .402.7Eigendecomposition. . . . . . . . . .. . . . . . . .. . . . . . . .422.8SingularValueDecomposition. . . . . . . .. . . . . . . .. . . .442.9TheMoore-PenrosePseudoinverse. . . . . . . . .. . . . . . . ..452.10TheTraceOperator. . . . . . . . .. . . . . . . .. . . . . . . .462.11TheDeterminant. .. . . . . . . .. . . . . . . .. . . . . . . . .472.12Example:PrincipalComponentsAnalysis. . . . . . . . .. . . .483ProbabilityandInformationTheory533.1WhyProbability?. . . . .. . . . . . . . .. . . . . . . .. . . . .54i

CONTENTS3.2RandomVariables. . . . .. . . . . . . .. . . . . . . . .. . . .563.3ProbabilityDistributions. . . . . . . . .. . . . . . . .. . . . . .563.4MarginalProbability. . . . . . . . .. . . . . . . . .. . . . . . .583.5ConditionalProbability. .. . . . . . . .. . . . . . . .. . . . .593.6TheChainRuleofConditionalProbabilities. . . . . . . . .. . .593.7IndependenceandConditionalIndependence. . . . . . . . .. . .603.8Expectation,VarianceandCovariance. . . . . . . . . .. . . . .603.9CommonProbabilityDistributions. . . . . . . . . . . . . . .. .623.10UsefulPropertiesofCommonFunctions. . .. . . . . . . . .. .673.11Bayes’Rule. . . . . . . . . .. . . . . . . .. . . . . . . .. . . .703.12TechnicalDetailsofContinuousVariables. . . . . .. . . . . . .713.13InformationTheory. . . . . . . . . .. . . . . . . .. . . . . . . .723.14StructuredProbabilisticModels. . . .. . . . . . . .. . . . . . .754NumericalComputation804.1OverﬂowandUnderﬂow. . . . . . . . .. . . . . . . .. . . . . .804.2PoorConditioning. . . . . . . . .. .

. . . . . . . . . . . . . .. .623.10UsefulPropertiesofCommonFunctions. . .. . . . . . . . .. .673.11Bayes’Rule. . . . . . . . . .. . . . . . . .. . . . . . . .. . . .703.12TechnicalDetailsofContinuousVariables. . . . . .. . . . . . .713.13InformationTheory. . . . . . . . . .. . . . . . . .. . . . . . . .723.14StructuredProbabilisticModels. . . .. . . . . . . .. . . . . . .754NumericalComputation804.1OverﬂowandUnderﬂow. . . . . . . . .. . . . . . . .. . . . . .804.2PoorConditioning. . . . . . . . .. . . . . . . .. . . . . . . . .824.3Gradient-BasedOptimization. . . . . . .. . . . . . . .. . . . .824.4ConstrainedOptimization. . . . . . . . . . . . .. . . . . . . ..934.5Example:LinearLeastSquares. . . . . . .. . . . . . . . .. . .965MachineLearningBasics985.1LearningAlgorithms. . . . . . . . . . .. . . . . . . .. . . . . .995.2Capacity,OverﬁttingandUnderﬁtting. .. . . . . . . .. . . . .1105.3HyperparametersandValidationSets.. . . . . . . .. . . . . . .1205.4Estimators,BiasandVariance. . . . . .. . . . . . . ..

. . .. . . . . . . . .824.3Gradient-BasedOptimization. . . . . . .. . . . . . . .. . . . .824.4ConstrainedOptimization. . . . . . . . . . . . .. . . . . . . ..934.5Example:LinearLeastSquares. . . . . . .. . . . . . . . .. . .965MachineLearningBasics985.1LearningAlgorithms. . . . . . . . . . .. . . . . . . .. . . . . .995.2Capacity,OverﬁttingandUnderﬁtting. .. . . . . . . .. . . . .1105.3HyperparametersandValidationSets.. . . . . . . .. . . . . . .1205.4Estimators,BiasandVariance. . . . . .. . . . . . . .. . . . . .1225.5MaximumLikelihoodEstimation. . . . . .. . . . . . . . .. . .1315.6BayesianStatistics. . . . . . . . . . .. . . . . . . .. . . . . . .1355.7SupervisedLearningAlgorithms. . .. . . . . . . .. . . . . . . .1395.8UnsupervisedLearningAlgorithms. . . . . . . . . . . . . . .. .1455.9StochasticGradientDescent. . . .. . . . . . . . .. . . . . . . .1505.10BuildingaMachineLearningAlgorithm. . . . . . . . . . . . ..1525.11ChallengesMotivatingDeepLearning. . . . .. . . . . . . . ..

. . . .. . . . . . . .. . . . . .1225.5MaximumLikelihoodEstimation. . . . . .. . . . . . . . .. . .1315.6BayesianStatistics. . . . . . . . . . .. . . . . . . .. . . . . . .1355.7SupervisedLearningAlgorithms. . .. . . . . . . .. . . . . . . .1395.8UnsupervisedLearningAlgorithms. . . . . . . . . . . . . . .. .1455.9StochasticGradientDescent. . . .. . . . . . . . .. . . . . . . .1505.10BuildingaMachineLearningAlgorithm. . . . . . . . . . . . ..1525.11ChallengesMotivatingDeepLearning. . . . .. . . . . . . . .. .154IIDeepNetworks:ModernPractices1656DeepFeedforwardNetworks1676.1Example:LearningXOR.. . . . . . . . .. . . . . . . .. . . . .1706.2Gradient-BasedLearning.. . . . . . . .. . . . . . . .. . . . . .176ii

CONTENTS6.3HiddenUnits. . . . . .. . . . . . . .. . . . . . . . .. . . . . .1906.4ArchitectureDesign. . . . . . . . .. . . . . . . .. . . . . . . ..1966.5Back-PropagationandOtherDiﬀerentiationAlgorithms. . . . .2036.6HistoricalNotes. . . . . . .. . . . . . . .. . . . . . . . .. . . .2247RegularizationforDeepLearning2287.1ParameterNormPenalties. . . . .. . . . . . . . .. . . . . . . .2307.2NormPenaltiesasConstrainedOptimization. . . . . . . .. . . .2377.3RegularizationandUnder-ConstrainedProblems. .. . . . . . .2397.4DatasetAugmentation. . . . . . . . . .. . . . . . . . .. . . . .2407.5NoiseRobustness. . . . . . . . .. . . . . . . .. . . . . . . .. .2427.6Semi-SupervisedLearning. . . . . . . . . . . . . . . .. . . . . .2437.7Multi-TaskLearning. . . . . . . . . . . . . .. . . . . . . . .. .2447.8EarlyStopping. . . . . . . . .. . . . . . . .. . . . . . . .. . .2467.9ParameterTyingandParameterSharing . . . . . . . . . . . . . .2537.10SparseRepresentations. . . . . . . . .. . . . . . . .. . . . . .

.. . . . . . .2397.4DatasetAugmentation. . . . . . . . . .. . . . . . . . .. . . . .2407.5NoiseRobustness. . . . . . . . .. . . . . . . .. . . . . . . .. .2427.6Semi-SupervisedLearning. . . . . . . . . . . . . . . .. . . . . .2437.7Multi-TaskLearning. . . . . . . . . . . . . .. . . . . . . . .. .2447.8EarlyStopping. . . . . . . . .. . . . . . . .. . . . . . . .. . .2467.9ParameterTyingandParameterSharing . . . . . . . . . . . . . .2537.10SparseRepresentations. . . . . . . . .. . . . . . . .. . . . . . .2547.11BaggingandOtherEnsembleMethods.. . . . . . . . .. . . . .2567.12Dropout. . . . . . . .. . . . . . . . .. . . . . . . .. . . . . . .2587.13AdversarialTraining. . . . . . . .. . . . . . . . .. . . . . . . .2687.14TangentDistance,TangentProp,andManifoldTangentClassiﬁer2708OptimizationforTrainingDeepModels2748.1HowLearningDiﬀersfromPureOptimization. . . . . . . . . . .2758.2ChallengesinNeuralNetworkOptimization. . . . .. . . . . . .2828.3BasicAlgorithms. . . . . . . . . . . . .. . . . . . . .. . . . .

.2547.11BaggingandOtherEnsembleMethods.. . . . . . . . .. . . . .2567.12Dropout. . . . . . . .. . . . . . . . .. . . . . . . .. . . . . . .2587.13AdversarialTraining. . . . . . . .. . . . . . . . .. . . . . . . .2687.14TangentDistance,TangentProp,andManifoldTangentClassiﬁer2708OptimizationforTrainingDeepModels2748.1HowLearningDiﬀersfromPureOptimization. . . . . . . . . . .2758.2ChallengesinNeuralNetworkOptimization. . . . .. . . . . . .2828.3BasicAlgorithms. . . . . . . . . . . . .. . . . . . . .. . . . . .2948.4ParameterInitializationStrategies.. . . . . . . . .. . . . . . .3018.5AlgorithmswithAdaptiveLearningRates. . . . . . .. . . . . .3068.6ApproximateSecond-OrderMethods. . . .. . . . . . . . .. . .3108.7OptimizationStrategiesandMeta-Algorithms. . . . .. . . . . .3179ConvolutionalNetworks3309.1TheConvolutionOperation. . . . . . . . . . . . . . . .. . . . .3319.2Motivation. .. . . . . . . .. . . . . . . . .. . . . . . . .. . . .3359.3Pooling. . . . . . . . . . . . .. . . . . . . .. . . . . . . . .. .

. . .2948.4ParameterInitializationStrategies.. . . . . . . . .. . . . . . .3018.5AlgorithmswithAdaptiveLearningRates. . . . . . .. . . . . .3068.6ApproximateSecond-OrderMethods. . . .. . . . . . . . .. . .3108.7OptimizationStrategiesandMeta-Algorithms. . . . .. . . . . .3179ConvolutionalNetworks3309.1TheConvolutionOperation. . . . . . . . . . . . . . . .. . . . .3319.2Motivation. .. . . . . . . .. . . . . . . . .. . . . . . . .. . . .3359.3Pooling. . . . . . . . . . . . .. . . . . . . .. . . . . . . . .. . .3399.4ConvolutionandPoolingasanInﬁnitelyStrongPrior. .. . . . .3459.5VariantsoftheBasicConvolutionFunction. . . . . . . . . . . .3479.6StructuredOutputs.. . . . . . . .. . . . . . . . .. . . . . . . .3589.7DataTypes. . . . . .. . . . . . . .. . . . . . . .. . . . . . . .3609.8EﬃcientConvolutionAlgorithms. . . . . . . .. . . . . . . .. .3629.9RandomorUnsupervisedFeatures. . . . . . . .. . . . . . . ..363iii

CONTENTS9.10TheNeuroscientiﬁcBasisforConvolutionalNetworks. . . . . ..3649.11ConvolutionalNetworksandtheHistoryofDeepLearning. . . .37110 SequenceModeling:RecurrentandRecursiveNets37310.1UnfoldingComputationalGraphs. . . . . . . . . . . . .. . . . .37510.2RecurrentNeuralNetworks. . .. . . . . . . . .. . . . . . . ..37810.3BidirectionalRNNs . . . . . . . . . . . . . .. . . . . . . . .. . .39510.4Encoder-DecoderSequence-to-SequenceArchitectures. . . . . ..39610.5DeepRecurrentNetworks. . . . . . . .. . . . . . . . .. . . . .39810.6RecursiveNeuralNetworks. . . . .. . . . . . . . .. . . . . . . .40010.7TheChallengeofLong-TermDependencies. . . . . . . . . .. . .40210.8EchoStateNetworks. . . . . . . . . .. . . . . . . . .. . . . . .40510.9LeakyUnitsandOtherStrategiesforMultipleTimeScales. . ..40810.10 TheLongShort-TermMemoryandOtherGatedRNNs. .. . . .41010.11 OptimizationforLong-TermDependencies. . . . . . . .. . . . .41410.12 ExplicitMemory. . . . . . . . . .. . . . . . . . .. . . . . . . .41811

. . . .. . . . . . . . .. . . . .39810.6RecursiveNeuralNetworks. . . . .. . . . . . . . .. . . . . . . .40010.7TheChallengeofLong-TermDependencies. . . . . . . . . .. . .40210.8EchoStateNetworks. . . . . . . . . .. . . . . . . . .. . . . . .40510.9LeakyUnitsandOtherStrategiesforMultipleTimeScales. . ..40810.10 TheLongShort-TermMemoryandOtherGatedRNNs. .. . . .41010.11 OptimizationforLong-TermDependencies. . . . . . . .. . . . .41410.12 ExplicitMemory. . . . . . . . . .. . . . . . . . .. . . . . . . .41811 PracticalMethodology42311.1PerformanceMetrics. . . . . . . . . .. . . . . . . .. . . . . . .42411.2DefaultBaselineModels. . . . . . . .. . . . . . . .. . . . . . .42711.3DeterminingWhethertoGatherMoreData. . . . . . . . . . . .42811.4SelectingHyperparameters. . . . . . . . .. . . . . . . .. . . . .42911.5DebuggingStrategies. . . . .. . . . . . . .. . . . . . . . .. . .43811.6Example:Multi-DigitNumberRecognition. . . . .. . . . . . . .44212 Applications44512.1LargeScaleDeepLearning.. . . . . . . .. . . . . .

. . . . . . . . .. . . . . . . .. . . . . . .42411.2DefaultBaselineModels. . . . . . . .. . . . . . . .. . . . . . .42711.3DeterminingWhethertoGatherMoreData. . . . . . . . . . . .42811.4SelectingHyperparameters. . . . . . . . .. . . . . . . .. . . . .42911.5DebuggingStrategies. . . . .. . . . . . . .. . . . . . . . .. . .43811.6Example:Multi-DigitNumberRecognition. . . . .. . . . . . . .44212 Applications44512.1LargeScaleDeepLearning.. . . . . . . .. . . . . . . . .. . . .44512.2ComputerVision. . . . . . . . .. . . . . . . .. . . . . . . .. .45412.3SpeechRecognition . . . . . .. . . . . . . .. . . . . . . . .. . .46012.4NaturalLanguageProcessing. . .. . . . . . . .. . . . . . . ..46312.5OtherApplications. . . . . . . . .. . . . . . . .. . . . . . . ..479IIIDeepLearningResearch48813 LinearFactorModels49113.1ProbabilisticPCAandFactorAnalysis. . . . . . .. . . . . . . .49213.2IndependentComponentAnalysis(ICA). . . . . . . . . . . .. .49313.3SlowFeatureAnalysis. . . . . .. . . . . . . . .. . . . . . .

. . . . . . .. . . . . . . .. . . . . . . .. .45412.3SpeechRecognition . . . . . .. . . . . . . .. . . . . . . . .. . .46012.4NaturalLanguageProcessing. . .. . . . . . . .. . . . . . . ..46312.5OtherApplications. . . . . . . . .. . . . . . . .. . . . . . . ..479IIIDeepLearningResearch48813 LinearFactorModels49113.1ProbabilisticPCAandFactorAnalysis. . . . . . .. . . . . . . .49213.2IndependentComponentAnalysis(ICA). . . . . . . . . . . .. .49313.3SlowFeatureAnalysis. . . . . .. . . . . . . . .. . . . . . . ..49513.4SparseCoding. . . . . .. . . . . . . .. . . . . . . . .. . . . . .498iv

CONTENTS13.5ManifoldInterpretationofPCA. . . . . . . . . . . . . . . . .. .50114 Autoencoders50414.1UndercompleteAutoencoders. . . . . . . . . .. . . . . . . .. .50514.2RegularizedAutoencoders. . . . . . . . .. . . . . . . .. . . . .50614.3RepresentationalPower,LayerSizeandDepth. . . . . .. . . . .51014.4StochasticEncodersandDecoders. . . . . . . . . . .. . . . . . .51114.5DenoisingAutoencoders. .. . . . . . . .. . . . . . . . .. . . .51214.6LearningManifoldswithAutoencoders. . . . . .. . . . . . . . .51714.7ContractiveAutoencoders.. . . . . . . .. . . . . . . .. . . . .52314.8PredictiveSparseDecomposition. . . . . . . .. . . . . . . . ..52514.9ApplicationsofAutoencoders. . . . .. . . . . . . . .. . . . . .52615 RepresentationLearning52815.1GreedyLayer-WiseUnsupervisedPretraining. . . . . .. . . . .53015.2TransferLearningandDomainAdaptation. . . .. . . . . . . ..53815.3Semi-SupervisedDisentanglingofCausalFactors. . . . .. . . .54315.4DistributedRepresentation. . . . . . . . . . . .. . . . . . . .

. . . .51714.7ContractiveAutoencoders.. . . . . . . .. . . . . . . .. . . . .52314.8PredictiveSparseDecomposition. . . . . . . .. . . . . . . . ..52514.9ApplicationsofAutoencoders. . . . .. . . . . . . . .. . . . . .52615 RepresentationLearning52815.1GreedyLayer-WiseUnsupervisedPretraining. . . . . .. . . . .53015.2TransferLearningandDomainAdaptation. . . .. . . . . . . ..53815.3Semi-SupervisedDisentanglingofCausalFactors. . . . .. . . .54315.4DistributedRepresentation. . . . . . . . . . . .. . . . . . . . ..54815.5ExponentialGainsfromDepth. . . . . . . . . .. . . . . . . ..55515.6ProvidingCluestoDiscoverUnderlyingCauses. . . .. . . . . .55616 StructuredProbabilisticModelsforDeepLearning56016.1TheChallengeofUnstructuredModeling.. . . . . . . .. . . . .56116.2UsingGraphstoDescribeModelStructure. .. . . . . . . .. . .56516.3SamplingfromGraphicalModels. . .. . . . . . . .. . . . . . .58216.4AdvantagesofStructuredModeling .. . . . . . . . .. . . . . . .58416.5LearningaboutDependencies. . . .. . . . . . . .. . .

..54815.5ExponentialGainsfromDepth. . . . . . . . . .. . . . . . . ..55515.6ProvidingCluestoDiscoverUnderlyingCauses. . . .. . . . . .55616 StructuredProbabilisticModelsforDeepLearning56016.1TheChallengeofUnstructuredModeling.. . . . . . . .. . . . .56116.2UsingGraphstoDescribeModelStructure. .. . . . . . . .. . .56516.3SamplingfromGraphicalModels. . .. . . . . . . .. . . . . . .58216.4AdvantagesofStructuredModeling .. . . . . . . . .. . . . . . .58416.5LearningaboutDependencies. . . .. . . . . . . .. . . . . . . .58416.6InferenceandApproximateInference. . . . . . . . .. . . . . . .58516.7TheDeepLearningApproachtoStructuredProbabilisticModels58617 MonteCarloMethods59217.1SamplingandMonteCarloMethods. . . . . . . .. . . . . . . .59217.2ImportanceSampling. . . . . . . . . . . .. . . . . . . .. . . . .59417.3MarkovChainMonteCarloMethods. . . . .. . . . . . . .. . .59717.4GibbsSampling . . . . . . .. . . . . . . . .. . . . . . . .. . . .60117.5TheChallengeofMixingbetweenSeparatedModes. . . . . . ..60118

.. . . . . . . .58416.6InferenceandApproximateInference. . . . . . . . .. . . . . . .58516.7TheDeepLearningApproachtoStructuredProbabilisticModels58617 MonteCarloMethods59217.1SamplingandMonteCarloMethods. . . . . . . .. . . . . . . .59217.2ImportanceSampling. . . . . . . . . . . .. . . . . . . .. . . . .59417.3MarkovChainMonteCarloMethods. . . . .. . . . . . . .. . .59717.4GibbsSampling . . . . . . .. . . . . . . . .. . . . . . . .. . . .60117.5TheChallengeofMixingbetweenSeparatedModes. . . . . . ..60118 ConfrontingthePartitionFunction60718.1TheLog-LikelihoodGradient.. . . . . . . .. . . . . . . . .. .60818.2StochasticMaximumLikelihoodandContrastiveDivergence. . .609v

CONTENTS18.3Pseudolikelihood. . . . . . . . . . .. . . . . . . . .. . . . . . .61718.4ScoreMatchingandRatioMatching. . . . . . . .. . . . . . . .61918.5DenoisingScoreMatching. . . . . . . . .. . . . . . . .. . . . .62118.6Noise-ContrastiveEstimation. . . . .. . . . . . . .. . . . . . .62218.7EstimatingthePartitionFunction. . . . . . . . . . .. . . . . . .62519 ApproximateInference63319.1InferenceasOptimization.. . . . . . . . .. . . . . . . .. . . .63519.2ExpectationMaximization. .. . . . . . . .. . . . . . . . .. . .63619.3MAPInferenceandSparseCoding.. . . . . . . . .. . . . . . .63719.4VariationalInferenceandLearning. . . . . . . . . . . . . . .. .64019.5LearnedApproximateInference. . .. . . . . . . . .. . . . . . .65320 DeepGenerativeModels65620.1BoltzmannMachines. . . . . . . . . . .. . . . . . . . .. . . . .65620.2RestrictedBoltzmannMachines. . . . . . .. . . . . . . . .. . .65820.3DeepBeliefNetworks.. . . . . . . .. . . . . . . .. . . . . . . .66220.4DeepBoltzmannMachines. . . . . . . . . .. . . . . . .

. . .. . .63619.3MAPInferenceandSparseCoding.. . . . . . . . .. . . . . . .63719.4VariationalInferenceandLearning. . . . . . . . . . . . . . .. .64019.5LearnedApproximateInference. . .. . . . . . . . .. . . . . . .65320 DeepGenerativeModels65620.1BoltzmannMachines. . . . . . . . . . .. . . . . . . . .. . . . .65620.2RestrictedBoltzmannMachines. . . . . . .. . . . . . . . .. . .65820.3DeepBeliefNetworks.. . . . . . . .. . . . . . . .. . . . . . . .66220.4DeepBoltzmannMachines. . . . . . . . . .. . . . . . . .. . . .66520.5BoltzmannMachinesforReal-ValuedData. . . . . . . . .. . . .67820.6ConvolutionalBoltzmannMachines . . . . . . . . . . . . . . . ..68520.7BoltzmannMachinesforStructuredorSequentialOutputs. . . .68720.8OtherBoltzmannMachines. . . . .. . . . . . . .. . . . . . . .68820.9Back-PropagationthroughRandomOperations. . . . . .. . . .68920.10 DirectedGenerativeNets. . . . . . . . . . . .. . . . . . . . .. .69420.11 DrawingSamplesfromAutoencoders. . . . .. . . . . . . . .. .71220.12

. . . .. . . . . . . .. . . .66520.5BoltzmannMachinesforReal-ValuedData. . . . . . . . .. . . .67820.6ConvolutionalBoltzmannMachines . . . . . . . . . . . . . . . ..68520.7BoltzmannMachinesforStructuredorSequentialOutputs. . . .68720.8OtherBoltzmannMachines. . . . .. . . . . . . .. . . . . . . .68820.9Back-PropagationthroughRandomOperations. . . . . .. . . .68920.10 DirectedGenerativeNets. . . . . . . . . . . .. . . . . . . . .. .69420.11 DrawingSamplesfromAutoencoders. . . . .. . . . . . . . .. .71220.12 GenerativeStochasticNetworks. . .. . . . . . . .. . . . . . . .71620.13 OtherGenerationSchemes. . . . . . . . . . . . .. . . . . . . . .71720.14 EvaluatingGenerativeModels . . . . . . . . . . . . .. . . . . . .71920.15 Conclusion. . . . . . . .. . . . . . . . .. . . . . . . .. . . . . .721Bibliography723Index780

CHAPTER1.INTRODUCTION



Visible layer(input pixels)1st hidden layer(edges)2nd hidden layer(corners andcontours)3rd hidden layer(object parts)CARPERSONANIMALOutput(object identity)

InputHand-designed programOutput
InputHand-designed featuresMapping from featuresOutput
InputFeaturesMapping from featuresOutput
InputSimple featuresMapping from featuresOutput
Additional layers of more abstract features
Rule-basedsystemsClassicmachinelearningRepresentationlearningDeeplearningFigure1.5: FlowchartsshowinghowthediﬀerentpartsofanAIsystemrelatetoeachotherwithindiﬀerentAIdisciplines.Shadedboxesindicatecomponentsthatareabletolearnfromdata.10

CHAPTER1.INTRODUCTION1. IntroductionPart I: Applied Math and Machine Learning Basics2. Linear Algebra3. Probability and Information Theory4. Numerical Computation5. Machine Learning BasicsPart II: Deep Networks: Modern Practices6. Deep Feedforward Networks7. Regularization8. Optimization9.  CNNs10.  RNNs11. Practical Methodology12. ApplicationsPart III: Deep Learning Research13. Linear Factor Models14. Autoencoders15. Representation Learning16. Structured Probabilistic Models17. Monte Carlo Methods18. Partition Function19. Inference20. Deep Generative ModelsFigure1.6:Thehigh-levelorganizationofthebook.Anarrowfromonechaptertoanotherindicatesthattheformerchapterisprerequisitematerialforunderstandingthelatter.12

CHAPTER1.INTRODUCTIONneuroscienceatall.Itis worth notingthat theeﬀorttounderstandhowthe brainworksonan algorithmic level is alive andwell.This endeavor isprimarily

x,= yzx= ) = (zp= xzy= )(zp= yz= )z.(3.8)We candenoteindependence andconditionalindependence withcompactnotation:xy⊥meansthatxandyareindependent,whilexyz⊥|meansthatxandyareconditionallyindependentgivenz.3.8Expectation,VarianceandCovarianceTheexpectationexpectedvalueorofsomefunctionf(x) withrespecttoaprobabilitydistributionP(x)istheaverageormeanvaluethatftakesonwhenxisdrawnfrom.Fordiscretevariablesthiscanbecomputedwithasummation:PEx∼P[()] =fxxPxfx,()()(3.9)whileforcontinuousvariables,itiscomputedwithanintegral:Ex∼p[()] =fxpxfxdx.()()(3.10)60

CHAPTER3.PROBABILITYANDINFORMATIONTHEORY
xProbability Densityq∗= argminqDKL()pqpx()q∗()x
xProbability Densityq∗= argminqDKL()qpp()xq∗()x

CHAPTER4.NUMERICALCOMPUTATION
xfx()Negativecurvature
xfx()Nocurvature
xfx()Positivecurvature

CHAPTER5.MACHINELEARNINGBASICS
CapacityBiasGeneralizationerrorVarianceOptimalcapacityOverﬁtting zoneUnderﬁtting zone

conditiondescribed by Eq.is5.55knownas consistency.Itissometimesreferredto asweakconsistency, withstrongconsistencyreferringtothealmostsureconvergenceofˆθtoθ.Almostsure130

CHAPTER5.MACHINELEARNINGBASICS5.7.1ProbabilisticSupervisedLearningMost supervised learning algorithms inthis book are based onestimating aprobabilitydistributionp(y|x).Wecandothissimplybyusingmaximumlikelihoodestimationtoﬁndthebestparametervectorθforaparametricfamilyofdistributions.py(|xθ;)Wehavealreadyseenthatlinearregressioncorrespondstothefamilypyy(|Nxθ;) =

dimensionsin thedata ishigh.This phenomenonis knownas thecurseofdimensionality. Ofparticularconcernisthatthenumberofpossibledistinctconﬁgurationsofasetofvariablesincreasesexponentiallyasthenumberofvariablesincreases.

CHAPTER6.DEEPFEEDFORWARDNETWORKS
01x101x2OriginalSpacex
012h101h2LearnedSpaceh

X XW(1)W(1)U(1)U(1)matmulH Hrelu
U(3)U(3)sqru(4)u(4)sumλ λu(7)u(7)W(2)W(2)U(2)U(2)matmuly yJMLEJMLEcross_entropy
U(5)U(5)sqru(6)u(6)sumu(8)u(8)J J+×+

didnot userectiﬁedlinearunits, butinsteadappliedrectiﬁcationto225

88First ensemble memberSecond ensemble memberOriginal datasetFirst resampled datasetSecond resampled

CHAPTER7.REGULARIZATIONFORDEEPLEARNING
y yh1h1h2h2x1x1x2x2y yh1h1h2h2x1x1x2x2y yh1h1h2h2x2x2y yh1h1h2h2x1x1y yh2h2x1x1x2x2y yh1h1x1x1x2x2y yh1h1h2h2y yx1x1x2x2y yh2h2x2x2y yh1h1x1x1y yh1h1x2x2y yh2h2x1x1y yx1x1y yx2x2y yh2h2y yh1h1y yBase network
Ensemble of Sub-NetworksFigure 7.6:Dropout trainsan ensemble consistingof allsub-networks that canbeconstructedbyremovingnon-outputunitsfromanunderlyingbasenetwork.Here,webeginwithabasenetworkwithtwovisibleunitsandtwohiddenunits.Therearesixteenpossiblesubsetsofthesefourunits.Weshowallsixteensubnetworksthatmaybeformedbydroppingoutdiﬀerentsubsetsofunitsfromtheoriginalnetwork.Inthissmallexample,alargeproportionoftheresultingnetworkshavenoinputunitsornopathconnectingtheinputtotheoutput.Thisproblembecomesinsigniﬁcantfornetworkswithwiderlayers,wheretheprobabilityofdroppingallpossiblepathsfrominputstooutputsbecomessmaller.260

CHAPTER8.OPTIMIZATIONFORTRAININGDEEPMODELSmostcommonlyusedpropertyisthegradient:∇θJ() = θEx,y∼ˆpdata∇θlogpmodel(;)x,yθ.(8.6)Computing thisexpectation exactly isvery expensive because it

CHAPTER9.CONVOLUTIONALNETWORKS
Input to layerConvolution stage:Ane transformﬃDetector stage:Nonlinearitye.g., rectiﬁed linearPooling stageNext layer
Input to layersConvolution layer:Ane transform ﬃDetector layer: Nonlinearitye.g., rectiﬁed linearPooling layerNext layerComplex layer terminologySimple layer terminology
Figure9.7:Thecomponentsofatypicalconvolutionalneuralnetworklayer.Therearetwocommonlyusedsetsofterminologyfordescribingtheselayers.(Left)Inthisterminology,theconvolutionalnetisviewedasasmallnumberofrelativelycomplexlayers,witheachlayerhavingmany“stages.”Inthisterminology,thereisaone-to-onemappingbetweenkerneltensorsandnetworklayers.Inthisbookwegenerallyusethisterminology.(Right)Inthisterminology,theconvolutionalnetisviewedasalargernumberofsimplelayers;everystepofprocessingisregardedasalayerinitsownright.Thismeansthatnotevery“layer”hasparameters.
341

CHAPTER9.CONVOLUTIONALNETWORKSLarge responsein pooling unitLarge responsein pooling unitLargeresponsein detectorunit 1Largeresponsein detectorunit 3

Input image: 256x256x3Output of convolution+ ReLU: 256x256x64Output of pooling with stride 4: 64x64x64Output of convolution+ReLU: 64x64x64Output of pooling with stride 4: 16x16x64Output of reshape to vector:16,384 unitsOutput of matrix multiply: 1,000 unitsOutput of softmax: 1,000 class probabilities
Input image: 256x256x3Output of convolution+ ReLU: 256x256x64Output of pooling with stride 4: 64x64x64Output of convolution+ReLU: 64x64x64Output of pooling to 3x3 grid: 3x3x64Output of reshape to vector:576 unitsOutput of matrix multiply: 1,000 unitsOutput of softmax: 1,000 class probabilities
Input image: 256x256x3Output of convolution+ ReLU: 256x256x64Output of pooling with stride 4: 64x64x64Output of convolution+ReLU: 64x64x64Output of convolution:16x16x1,000Output of average pooling: 1x1x1,000Output of softmax: 1,000 class probabilities
Output of pooling with stride 4: 16x16x64

x1x1x2x2x3x3s2s2s1s1s3s3x4x4s4s4x5x5s5s5
x1x1x2x2s1s1s3s3x5x5s5s5x1x1x2x2x3x3s2s2s1s1s3s3x4x4s4s4x5x5s5s5
a        ba        ba        ba        ba        a        bc      de     fg      h i

a        ba        ba        ba        ba        a        bc      de     fg      h i  
x4x4x3x3s4s4s2s2Figure9.14:Comparisonoflocalconnections,convolution,andfullconnections.(Top)Alocallyconnectedlayerwithapatchsizeoftwopixels.Eachedgeislabeledwithauniquelettertoshowthateachedgeisassociatedwithitsownweightparameter.(Center)Aconvolutionallayerwithakernelwidthoftwopixels.Thismodelhasexactlythesameconnectivityasthelocallyconnectedlayer.Thediﬀerenceliesnotinwhichunitsinteractwitheachother,butinhowtheparametersareshared.Thelocallyconnectedlayerhasnoparametersharing.Theconvolutionallayerusesthesametwoweightsrepeatedlyacrosstheentireinput,asindicatedbytherepetitionoftheletterslabelingeachedge.(Bottom)Afullyconnectedlayerresemblesalocallyconnectedlayerinthesensethateachedgehasitsownparameter(therearetoomanytolabelexplicitlywithlettersinthisdiagram).However,itdoesnothavetherestrictedconnectivityofthelocallyconnectedlayer.353

Input TensorOutput Tensor
Spatial coordinatesChannel coordinates
Figure9.15: Aconvolutionalnetworkwiththeﬁrsttwooutputchannelsconnectedtoonlytheﬁrsttwoinputchannels,andthesecondtwooutputchannelsconnectedtoonlythesecondtwoinputchannels.354

x1x1x2x2x3x3s2s2s1s1s3s3x4x4s4s4x5x5s5s5x1x1x2x2x3x3s2s2s1s1s3s3x4x4s4s4x5x5s5s5
a        ba        ba        ba        ba       a        bc      de     fg      h i  
x1x1x2x2x3x3s2s2s1s1s3s3x4x4s4s4x5x5s5s5a        bc        da        bc        da

CHAPTER10.SEQUENCEMODELING:RECURRENTANDRECURSIVENETSinformationﬂowforwardintime(computingoutputsandlosses)andbackwardintime(computinggradients)byexplicitlyshowingthepathalongwhichthisinformationﬂows.10.2RecurrentNeuralNetworksArmedwiththegraphunrollingandparametersharingideasofSec.,wecan10.1designawidevarietyofrecurrentneuralnetworks.
U UV VW Wo(t−1)o(t−1)h ho oy yL L
x xo()to()to(+1)to(+1)tL(t−1)L(t−1)L()tL()tL(+1)tL(+1)ty(t−1)y(t−1)y()ty()ty(+1)ty(+1)t
h(t−1)h(t−1)h()th()th(+1)th(+1)tx(t−1)x(t−1)x()tx()tx(+1)tx(+1)tW WW WW WW Wh()...h()...h()...h()...V VV VV VU UU UU UUnfold

CHAPTER10.SEQUENCEMODELING:RECURRENTANDRECURSIVENETS
UVWo(t−1)o(t−1)h ho oy yL L

long-rangeinputs, if the RNNis ableto learnto carry thatinformation.Comparedtoaconvolutionalnetwork,RNNsappliedtoimagesaretypicallymore395

CHAPTER12.APPLICATIONS
−150015...x0−15.00.15.x1Rawinput
−150015...x0GCN,= 10λ−2
−150015...x0GCN,= 0λ

thechoiceofoneoutof||VwordscanbeobtainedbydoingO(log||V)operations(oneforeachofthenodesonthepathfromtheroot).Inthisexample,computingtheprobabilityofawordycanbedonebymultiplyingthreeprobabilities,associatedwiththebinarydecisionstomoveleftorrightateachnodeonthepathfromtheroottoanodey.Letbi(y)bethei-thbinarydecisionwhentraversingthetreetowardsthevaluey.Theprobabilityofsamplinganoutputydecomposesintoaproductofconditionalprobabilities,usingthechainruleforconditionalprobabilities,witheachnodeindexedbythepreﬁxofthesebits.Forexample,node(1,0)correspondstothepreﬁx(b0(w4) = 1,b1(w4) = 0),andtheprobabilityofw4canbedecomposedasfollows:Pw(= y4) = (Pb0= 1,b1= 0,b2= 0)(12.12)= (Pb0= 1)(Pb1= 0 |b0= 1)(Pb2= 0 |b0= 1,b1= 0).(12.13)470

CHAPTER12.APPLICATIONS12.4.5.1UsinganAttentionMechanismandAligningPiecesofData
α(t−1)α(t−1)α()tα()tα(+1)tα(+1)th(t−1)h(t−1)h()th()th(+1)th(+1)tc c
× ×× ×× ×+

CHAPTER13.LINEARFACTORMODELSsamplefrom.Nextwesamplethereal-valuedobservablevariablesgiventhefactors:xWhb= ++noise(13.2)wherethenoiseistypicallyGaussiananddiagonal(independentacrossdimensions).ThisisillustratedinFig..13.1h1h1h2h2h3h3x1x1x2x2x3x3xhn  o i s  exhn  o i s

CHAPTER14.AUTOENCODERSbytrainingwiththesquarederrorcriterion||gf((˜xx))−||2(14.16)andcorruptionC(˜x=˜xx|) = (N˜xx;= µ,σΣ = 2I)(14.17)withnoisevarianceσ2.SeeFig.foranillustrationofhowthisworks.14.5

CHAPTER15.REPRESENTATIONLEARNINGillustratedinFig..15.2
Selection switchh(1)h(1)h(2)h(2)h(3)h(3)y yh(shared)h(shared)

CHAPTER15.REPRESENTATIONLEARNINGhx=fx()x
xtestytesthy=fy()y
y−space
Relationship between embedded points within one of the domainsMaps between representation spaces fxfyx−space

CHAPTER15.REPRESENTATIONLEARNING
h1h2h3
h= [1,,11]h= [0,,11]h= [1,,01]h= [1,,10]h= [0,,10]h= [0,,01]h= [1,,00]

CHAPTER16.STRUCTUREDPROBABILISTICMODELSFORDEEPLEARNING
h1h1h2h2h3h3
v1v1v2v2v3v3ab
ca
cb
ca

Themoststandard,genericguaranteesforMCMCtechniquesareonlyapplicablewhenthemodeldoesnotassignzeroprobabilitytoanystate.Therefore,itismostconvenientto present these techniques assampling froman energy-basedmodel

CHAPTER18.CONFRONTINGTHEPARTITIONFUNCTION18.1TheLog-LikelihoodGradientWhat makes learning undirectedmodels by maximum likelihood particularlydiﬃcultisthatthepartitionfunctiondependsontheparameters.Thegradientofthelog-likelihoodwithrespecttotheparametershasatermcorrespondingtothegradientofthepartitionfunction:∇θlog(;) = pxθ∇θlog

˜p())xfor˜p()x:x∇θexp(log ˜p())xZ(18.9)=xexp(log ˜p())x∇θlog ˜p()xZ(18.10)=x˜p()x∇θlog ˜p()xZ(18.11)=xp()x∇θlog ˜p()x(18.12)608

exp∝b()lh()l+b(1)l−h(1)l−+h(1)l−W()lh()l,(20.17)Ph(()ki= 1 |h(+1)k) = σb()ki+W(+1)k:,ih(+1)k∀∀∈−i,k1,...,l2,(20.18)Pv(i= 1 |h(1)) = σb(0)i+W(1):,ih(1)∀i.(20.19)Inthecaseofreal-valuedvisibleunits,substitutev∼Nvb;(0)+W(1)h(1),β−1(20.20)662

BIBLIOGRAPHYGuo,H.andGelfand,S.B.(1992).Classiﬁcationtreeswithneuralnetworkfeatureextraction.NeuralNetworks,IEEETransactionson,(6),923–933.3452Gupta,S.,Agrawal,A.,Gopalakrishnan,K.,andNarayanan,P.(2015).Deeplearningwithlimitednumericalprecision.,.CoRRa b s / 1 5 0 2 . 0 2 5 5 1454Gutmann,M.andHyvarinen,A.(2010).Noise-contrastiveestimation: Anewestima-tionprincipleforunnormalizedstatisticalmodels. InProceedingsofTheThirteenthInternationalConferenceonArtiﬁcialIntelligenceandStatistics(AISTATS’10).622Hadsell, R., Sermanet, P., Ben, J.,Erkan,A., Han, J., Muller,U., andLeCun, Y.(2007).Onlinelearningforoﬀroadrobots:Spatiallabelpropagationtolearnlong-rangetraversability.InProceedingsofRobotics:ScienceandSystems,Atlanta,GA,USA.455Hajnal,A.,Maass,W.,Pudlak,P.,Szegedy,M.,andTuran,G.(1993).Thresholdcircuitsofboundeddepth.,,129–154.J.Comput.System.Sci.4

S. (1976).Taylorexpansionofthe accumulated roundingerror.BITNumericalMathematics,(2),146–160.1

6(1),77–105.400Polyak,B.andJuditsky,A.(1992).Accelerationofstochasticapproximationbyaveraging.SIAMJ.ControlandOptimization,,838–855.3 0 ( 4 )322Polyak,B.T.(1964).Somemethodsofspeedinguptheconvergenceofiterationmethods.USSRComputationalMathematicsandMathematicalPhysics,(5),1–17.4296Poole,B.,Sohl-Dickstein,J.,andGanguli,S.(2014). Analyzingnoiseinautoencodersanddeepnetworks.,.CoRRa b s / 1 4 0 6 . 1 8 3

BIBLIOGRAPHYRumelhart, D., Hinton, G., andWilliams, R.(1986a).Learningrepresentationsbyback-propagatingerrors.Nature,,533–536.,,,,,,,3 2

b s / 1 4 1 2 . 6 5 5 8290303305404Sutskever,I.(2012).TrainingRecurrentNeuralNetworks.Ph.D.thesis,Departmentofcomputerscience,UniversityofToronto.,407414772

b s / 1 3 1 2 . 6 1 9 9.268271,Szegedy,C.,Vanhoucke,V.,Ioﬀe,S.,Shlens,J.,andWojna,Z.(2015).RethinkingtheInceptionArchitectureforComputerVision..,ArXive-prints243322Taigman,Y.,Yang,M.,Ranzato,M.,andWolf,L.(2014).DeepFace:Closingthegaptohuman-levelperformanceinfaceveriﬁcation.In.CVPR’2014100Tandy,D.W.(1997).WorksandDays:ATranslationandCommentaryfortheSocialSciences.UniversityofCaliforniaPress.1773